8000 feat(developer): next generation KMN compiler - lexer 🤔 by markcsinclair · Pull Request #13350 · keymanapp/keyman · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

feat(developer): next generation KMN compiler - lexer 🤔 #13350

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 76 commits into
base: epic/ng-compiler
Choose a base branch
from
Draft

feat(developer): next generation KMN compiler - lexer 🤔 #13350

wants to merge 76 commits into from
+2,081 −0

Conversation

markcsinclair
Copy link
Contributor
@markcsinclair markcsinclair commented Feb 25, 2025 •
edited
Loading

A lexer for the next generation KMN compiler, based on a modified port of Fowler's Java regex lexer (see Domain Specific Languages by Martin Fowler and Rebecca Parsons, Ch.20)

see Next Generation KMN Compiler #13349

@keymanapp-test-bot skip

To do list

  • Add tokens for remaining keywords/symbols, including:
    • ALWAYS
    • BITMAPS
    • CAPS
    • COPYRIGHT
    • FREES
    • LAYOUT
    • OFF
    • OLDCHARPOSMATCHING
    • ON
    • ONLY
    • SHIFT
    • Named constants
    • Compile targets
    • Hangeul syllables
    • decimal character codes
    • hex character codes
    • octal character codes
  • Add errors for compile targets, decimal, hex and octal character codes
  • Add error callback for lexing failure
  • Add warnings for deprecated and downlevel tokens

Questions

  • Whitespace and Comments: are currently emitted as tokens to allow for careful control in the syntax analyser. This is at odds with many language compiler designs that do not emit whitespace and comments, but is believed necessary for Keyman language LSP support. Is this right?
  • Source Lines: are currently captured in the token._line field of NEWLINE and lexer generated EOF Tokens. This has been done to allow the parser to have access to the original sourcecode in the AST via included Tokens, allowing round-trip recreation of the .kmn soure file. Is this right/necessary?

@keymanapp-test-bot
Copy link
keymanapp-test-bot bot commented Feb 25, 2025 •
edited
Loading

User Test Results

Test specification and instructions

User tests are not required

Test Artifacts

@keymanapp-test-bot keymanapp-test-bot bot added this to the B18S2 milestone Feb 25, 2025
@markcsinclair markcsinclair marked this pull request as draft February 25, 2025 16:35
srl295
srl295 previously approved these changes Feb 26, 2025
View reviewed changes
@darcywong00 darcywong00 removed this from the B18S2 milestone Feb 28, 2025
@markcsinclair markcsinclair dismissed srl295’s stale review May 6, 2025 16:05

Still in draft

@darcywong00 darcywong00 modified the milestones: A19S2, A19S3 May 10, 2025
@keyman-server keyman-server modified the milestones: A19S3, A19S4 May 26, 2025
@keyman-server keyman-server modified the milestones: A19S4, A19S5 Jun 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mcdurdin mcdurdin Awaiting requested review from mcdurdin

@darcywong00 darcywong00 Awaiting requested review from darcywong00

@SabineSIL SabineSIL Awaiting requested review from SabineSIL

@srl295 srl295 Awaiting requested review from srl295

At least 1 approving review is required to merge this pull request.

Assignees

@markcsinclair markcsinclair

Labels
developer/compilers/ developer/ epic-ng-compiler feat
Projects
Keyman
Status: No status
Milestone
A19S5
Development

Successfully merging this pull request may close these issues.
5 participants
@markcsinclair @mcdurdin @srl295 @keyman-server @darcywong00
0