8000 Support async tokenizer? · Issue #22 · ttezel/bayes · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content
Support async tokenizer? #22
Open
Open
@Bin-Huang

Description

@Bin-Huang

It is difficult to segment a text into tokens in some languages(such as Chinese), there are many hard works need to do to implement better tokenizer. For this reason, sometimes the tokenizer is implemented in other programming language, even in other services( in microservices architecture). In this case, support async version of tokenizer to request tokens between services is required.

PR: #21

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      0