Zestylogic · ps-george · Sep 21, 2019 · Sep 21, 2019 · Sep 21, 2019
diff --git a/FMark/src/Common/Lexer/README.md b/FMark/src/Common/Lexer/README.md
@@ -1,13 +1,10 @@
 # Project Contribution
 
-The preprocessor and lexer are part of FMark, which a markdown parser in F#. This sub project contains the 
-lexer and the preprocessor for the markdown parser. The preprocessor is a completely separate parser
-which preprocesses the markdown before passing it to the lexer and finally the parser.
+The preprocessor and lexer are part of FMark, which is a markdown parser in F#. This sub project contains the lexer and the preprocessor for the markdown parser. The preprocessor is a completely separate parser which preprocesses the markdown before passing it to the lexer and finally the parser.
 
 # Preprocessor
 
-This project contains the Preprocessor for FMark. The preprocessor adds templating
-capabilities to FMark, which was inspired by [Liquid](https://shopify.github.io/liquid/).
+This project contains the Preprocessor for FMark. The preprocessor adds templating capabilities to FMark, which was inspired by [Liquid](https://shopify.github.io/liquid/).
 
 ## Specification
 
@@ -108,8 +105,7 @@ This is the second macro
 
 ```
 
-More complicated macros can also be created by writing html in the macros. Due to the 
-html passthrough in the lexer, the html will be copied over literally to the output html.
+More complicated macros can also be created by writing html in the macros. Due to the html passthrough in the lexer, the html will be copied over literally to the output html.
 
 ### Future improvements
 

diff --git a/FMark/src/Common/TOCite/README.md b/FMark/src/Common/TOCite/README.md
@@ -128,79 +128,11 @@ At the end of the document:
 > 
 > FMark Smith. 2017. "Not a real website." Accessed March 3, 2018. https://www.example.com/website
 
----
-# ALL INFORMATION AFTER THIS MAYBE OUTDATED
-## Submission Write-up
+# Tests
    
-### How will (or might) your code contribute to the group deliverable? What have you done to ensure interfaces etc will be compatible? What are your interfaces (enough information for your module to be used by someone else not in your team. Assessment here is based on best efforts while allowing independent development, not whether the code is actually useful. Typical length 1/4 page.
+The following may be out of date. Refer to the main documentation.
 
-* My part includes parsers for headers and footers. `Token list -> ParsedObj list`.
-  * These two will be run after the tokeniser but before the main parser to pick out the headers and footers in the document.
-  * These two should be collected and then will be used to build a table of contents and a citation list.
-  * Unique identifiers will be inserted to the token list after pulling out the headers and footers. These will be useful for relative linking in the final HTML document.
-
-```
-Overall Flowchart:
-
-            ┌───────────┐                      ┌──────────────┐
-Source ───> │ Tokeniser │ ───> Token list ───> │ First Parser │ ───> Token list with identifiers ┐
-            └───────────┘           │          └──────────────┘                                  │
-                                    │                 │                                          │
-                                    │                 └────────────> Header+Footer list ────>────┤
-                                    │                                                            │
-                                    │          ┌──────────────┐                                  │
-                                    └────────> │ Table Parser │────────── PreTable ─────────>────│
-                                               └──────────────┘                                  │
-                                                                                                 │
-                                   ┌─────────┐                            ┌─────────────┐        │
-             Final Document  <──── │ HTMLGen │ <──── ParsedObj list <──── │ Main Parser │ <──────┘
-                                   └─────────┘                            └─────────────┘
-```
-* First Parser cooresponds to my code. The other parts are simplified for easy understanding of where this parser fits in the whole program.
-
-* To ensure compatible interface, `Types.fs` created as a group, which defines `Token` and `ParsedObj`.
-  * This lets me know what to expect as inputs (`Token list`), and what to give as outputs (`ParsedObj`).
-  * I just had to make sure my code follow these two type definitions, and in case when it cannot, we had discussions on GitHub to see if `Types.fs` can be modified to fit someone's requirements.
-
-### What is the specification of your code? Detail differences from VisUAL (if doing standard project), and reasons for them. Detail any areas where spec was initially unclear and has been clarified. Typical length 1/2 page + Tables.
-_Your markdown file can refer to comments in code, or the code itself, for details of normal functionality. Your markdown file should contain a precise description of how much functionality has been implemented, and how much tested (tables of features are good for this). A precise specification document would be very long: your document should only detail issues not obvious from the initial spec that needed to be resolved. An example of this for the default project would be where upper/lower case is significant, and where not._
-
-* Headers are indicated with any number of `#` on a new line, followed by more than one `WHITESPACE`.
-* Footers are indicated by `[^1]`, and its cooresponding text will have `[^1]:` before it.
-  * Footer text is parsed as regular text, a new line with 4 `WHITESPACE` can be used to write the text in multiple lines.
-  * This text is tupled with its `ID :int`.
-
-`Types.fs`
-* The shared definition for interface.
-  * Under `type Token`,
-    * `HEADER of int` and `FOOTER of int` are unique identifiers to be inserted into the token list after the first parsing.
-  * Under `type ParsedObj`,
-    * `THeader` is the returned headers, consist of the header text of type `TLine`, and its level of type `int`.
-    * `Footnote` if the returned footnote, with its ID of type `int`, and the footer text of type `TLine`.
-
-`Parser.fs`
-* The first parser. Consist of three main parts.
-  * Text Parsers, this is primitive, and to be merged with others' work once group stage starts:
-    * `parseText (Token list -> Token list * InlineElement list)`, parse all `Literal`s until something different, then return the unparsed portion of the `Token list`, tupled with the list of `Literal`s.
-    * `parseLine (TLine -> Token list -> TLine * Token list)`, handles `Emphasis` and calls parseText to parse header and footer texts.
-    * `parseLine' (Token list -> InlineElement list)`, _call this when the rest of the token list is not needed._
-  * Header Parser.
-    * `tocParse (Token list -> int -> int -> THeader list * Token list)`, goes through the token list from the tokeniser, replace the headers with identifiers `HEADER of int`, and generate a list of these headers as `THeader`s for building the table of contents later. depth and index are needed for recusively tracking the level of header and the position of the header in the whole document.
-    * `tocGen' (Token list -> int -> THeader list * Token list`, takes in the maximum depth of the table of content, and calls tocParse to generate the desired lists. _Call this function when header parsing is needed._
-    * `tocGen (Token list -> int -> Ttoc)`, used when a ParsedObj is required as output.
-  * Footer Parser, two versions provided, distinguished by the `'` after the function names. The ones with `'` is more powerful.
-    * `citeParse' (Token list -> (int*TLine) list * Token list)`, goes through the token list from tokeniser, replacing in text footers with the identifier `FOOTER of int`, and builds a list of the footer texts by calling `citeParseIn'`.
-    * `citeParseIn' (InlineElement list -> Token list -> TLine * Token list)`, parses footer texts by calling `parseLine`, and feed the rest of the unparsed tokens back to `citeParse'`.
-    * `citeGen' (Token list -> ParsedObj list * Token list)`, builds the result from `citeParse'`.
-
-### A short description of your Test Plan. Typical length 1/2 page + tables. What you have tested will be clear from the feature specification which includes test status. How you have tested it must be itemised. Again a table is good (could be the same one as used for specification). Add any rationale for your test plan.
-
-`Parsertest.fs`
-* Header tests
-  * Each test in `testDataHd` is a tuple of three items, name, input, and output.
-  * Input is a `Token list`, output is a tuple of a `THeader list` and a `Token list`.
-  * The `THeader list` is the list of headers found, and the `Token list` is the original list with headers replaced by identifiers.
-
+## Header tests
 |Test|Rationale|Passed?|
 |----|---------|-------|
 |Basic Test|Basic functionality|Yes|
@@ -213,8 +145,7 @@ _Your markdown file can refer to comments in code, or the code itself, for detai
 |Emphasis in header text|`parseLine` in header parsing should be able to handle formats, more formats will be added once merged in group phase|Yes|
 |Multiple headers with emphasis|A general test with multiple test points|Yes|
 
-* Footer tests
-  * Stored similar to Header tests in `testDataFt`.
+## Footer tests
 
 |Test|Rationale|Passed?|
 |----|---------|-------|
@@ -224,8 +155,3 @@ _Your markdown file can refer to comments in code, or the code itself, for detai
 |Footer text continuation over multiple lines|Footer texts can be written in multiple lines in source, as long as the next line is started with a whitespace of more than 4|Yes|
 |Footer texts sorting|Footer texts can be written in any order, and they will be sorted before passed forward|Yes|
 |Emphasis in footer|Similar to Emphasis in header|Yes|
-
-### Anything notable learnt during testing.
-
-* Finding items to test can be difficult, but each new test greatly improves the confidence I had on my code.
-* Having a systematic testing system is useful to ensure, if changing the code to fulfill one test accidently break something else, it can be quickly spotted.
diff --git a/MACROS.md b/MACROS.md
@@ -0,0 +1,75 @@
+## Macros
+
+### Supported Constructs
+
+These are the supported constructs in the preprocessor.
+
+|Supported|Syntax|Description|Tested|
+|---|---|---|---|
+|Simple Macro|`{% macro name value %}`| Sets the Macro `name` equal to the string `value`|Unit Test|
+|Function Macro|`{% macro name(arg1; arg2) value %}`|Sets the Macro `name` equal to the string `value` with two parameters.|Unit Test|
+|Simple Evaluation|`{{ macro_name }}`|Evaluates the macro `macro_name` and replaces the evaluation with the evaluated body of the macro.|Unit Test|
+|Function Evaluation|`{{ macro_name(arg 1; arg 2) }}`|Evaluates the macro `macro_name` with the arguments `arg 1` and `arg 2` and replaces the evaluation with the evaluated body of the macro.|Unit Test|
+|File Include|`{{ include relative/path/to/file }}`|Includes and preprocesses a file using a relative or absolute path. The macros declared in that file will then be available in the current file|Unit Test|
+|Complex Macro Evaluation|`{{ x( {{ y( {{z}} ; Hello ) }} ; {{z}} ) }}`|Nested macro evaulations are supported. This way, default arguments can be created for other macros.|Unit Test|
+
+### Supported Features
+
+These are the features that are currently supported by the preprocessor.
+
+|Feature|Example|Description|Tested|
+|---|---|---|---|
+| Simple whitespace control|`{% macro x y %}` evaluates to `y` and not ` ` `y` ` `.|Removes whitespace and newlines in macros where one wouldn't expect them to be added to the macro body.|Unit Test|
+|Shadowing of macros through|`{% macro x x %} {% macro y(x) {{ x }} %}` with `{{ y(z) }}` will evaluate to `z` but `{{ x }}` outside of the macro will always evaluate to `x`.|Macros can be shadowed by arguments of other macros.|Unit Test|
+|Nested macros|`{% macro x {% macro y %} %}`|Macro y is only defined inside macro x and cannot be seen outside of the scope of x.|Unit Test|
+|Shadowing of macros through|`{% macro x x %} {% macro y {% macro x z %} {{x}} %} y: {{ y }}, x: {{ x }}` will evaluate to `y: z, x: x`|Macros can be shadowed by other macros which will be used instead for evaluation.|Unit Test|
+|Evaluation of large strings|`{{ x(This is the first argument; This is the second argument) }}`|One can pass large strings as arguments to the macros.|Unit Test|
+|Escaping of characters inside argument|`{{ x(arg 1 with a \); arg 2 with a \;) }}`|One can esape all the special characters inside macros and substitutions|Unit Test|
+|Escaping macros|`\{% macro x y %}`|This will escape the whole macro and not evaluate it|Unit Test|
+|Escaping Subsitutions|`\{{ x }}`| will not evaluate the substitution but instead output it literally|Unit Test|
+|Outputting unmatched subsituttion|`{{ x }}` -> `{{ x }}` if not in scope|If the subsitution is not matched, it will output it as it got it|Unit Test|
+|Nested Evaluations|`{{ x( {{y}} ) }}`|Arguments can now be evaluated inside them.|Unit Test|
+
+### Example
+
+In markdown using macros, one can write the following:
+
+```
+Text before macro
+{% macro Hello(arg1; arg2)
+This is text inside the macro, with semicolons;
+{% macro local(arg1; arg2)
+This is the second macro
+%}
+Now back in the first macro.
+{{ local(arg1; arg2) }}
+%}
+Outside both macros
+Should be printed as not in scope: {{ local(arg1; arg2) }}
+
+{{ Hello(arg1; arg2) }}
+```
+
+which then evaluates to
+
+```
+Text before macro
+Outside both macros
+Should be printed as not in scope: {{ local(arg1; arg2) }}
+
+
+This is text inside the macro, with semicolons;
+Now back in the first macro.
+
+This is the second macro 
+```
+
+The lexer supports HTML pass through, which can be used to display raw html in markdown. Using this with macros can give very useful and interesting functions.
+
+## Features
+
+Supports escaping of all the special characters defined in [Types](/FMark/FMark/src/Common/Types.fs). This is done by adding
+a `\` in front of the character that should be escaped.
+
+Tokens that match multiple characters can also be escaped by just putting a `\` before it. For example, 
+`***` can be escaped by writing `\***`.