8000 regexp dont support ?i #43
Comments
|
Is anyone watching this？I got the same problem |
shibingli
added a commit
to shibingli/tokenizer
that referenced
this issue
Aug 28, 2024
|
@shibingli I have merged your repo and fixed some 'import/go.mod' error. Now it works. https://github.com/whitezhang/tokenizer |
|
not able to count tokens for gpt-4 and gpt-3.5turbo getting this same error, --- FAIL: TestGetTokenCountSugarMe (0.10s)
panic: regexp: Compile(`(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+`): error parsing regexp: invalid or unsupported Perl syntax: `(?!` [recovered]
panic: regexp: Compile(`(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+`): error parsing regexp: invalid or unsupported Perl syntax: `(?!`model used: "Xenova/gpt-3.5-turbo", "Xenova/gpt-4", |
|
If anyone can help? No bandwidth atm. Thanks |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
<
39D5
create-branch
data-default-repo="sugarme/tokenizer"
data-selected-nwo="sugarme/tokenizer"
data-default-source-branch="master"
data-sidebar-url="/sugarme/tokenizer/issues/closing_references/partials/sidebar?source_id=2450743386&source_type=ISSUE"
class="discussion-sidebar-item d-block">
panic: regexp: Compile("(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\\p{L}\\p{N}]?\p{L}+|\p{N}{1}| ?[^\\s\\p{L}\\p{N}\r\n]+|\s*[\r\n]+|\s+(?!\S)|\s+"): error parsing regexp: invalid or unsupported Perl syntax:
(?![recovered]panic: regexp: Compile("(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\\p{L}\\p{N}]?\p{L}+|\p{N}{1}| ?[^\\s\\p{L}\\p{N}\r\n]+|\s*[\r\n]+|\s+(?!\S)|\s+"): error parsing regexp: invalid or unsupported Perl syntax:
(?!The text was updated successfully, but these errors were encountered: