8000 hfst-tokenise --xerox output is not the same as hfst-lookup · Issue #500 · hfst/hfst · GitHub

More Web Proxy on the site http://driver.im/

hfst-tokenise --xerox output is not the same as hfst-lookup #500

< 89BD summary id="button-0194183a946eeeb4" class="btn btn-sm btn-primary m-0 ml-0 ml-md-2" > New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Open

reynoldsnlp opened this issue Mar 4, 2021 · 0 comments

reynoldsnlp commented

I assume that the output of hfst-tokenize --xerox should be exactly the same as the output of hfst-lookup (minus weights?), but it is not:

$ echo "Саша бларгвит." | hfst-tokenize --xerox tools/tokenisers/tokeniser-disamb-gt-desc.pmhfst
Саша	Саша N Prop Sem/Ant Msc Anim Sg Nom
Саша	Саша N Prop Sem/Ant Fem Inan Sg Nom
Саша

бларгвит

.	. CLB

$ echo "Саша бларгвит." | hfst-tokenize tools/tokenisers/tokeniser-disamb-gt-desc.pmhfst | hfst-lookup -q src/analyser-gt-desc.hfstol
Саша	Саша+N+Prop+Sem/Ant+Fem+Inan+Sg+Nom	0.000000
Саша	Саша+N+Prop+Sem/Ant+Msc+Anim+Sg+Nom	0.000000

бларгвит	бларгвит+?	inf

.	.+CLB	0.000000

There is an extra token after recognized tokens (Саша) with any reading, and there is no reading at all after unrecognized tokens (бларгвит).

The text was updated successfully, but these errors were encountered:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

0