8000 Neural tools integration by cgr71ii · Pull Request #235 · bitextor/bitextor · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Neural tools integration #235

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 54 commits into from
Jun 1, 2022
Merged

Neural tools integration #235

merged 54 commits into from
Jun 1, 2022

Conversation

cgr71ii
Copy link
Collaborator
@cgr71ii cgr71ii commented May 31, 2022 •
edited
Loading

Changes:

  • Neural tools have been integrated: Vecalign and Neural Document Aligner.
    • Neural tools have been adapted in order to work with the features which had not support before: headers, paragraph identification and deferred crawling.
  • Third party modules have been moved to a specific directory instead of being in the root directory.
  • The upload steps from the CI tests has been removed.
  • Tests optimized and updated.
  • Update documentation.

Fixes:

  • Hunalign was not working well with N * N alignments from docalign step when the trg index had been matched with another src index previously.
  • When Hunalign didn't return any result, the execution failed.
  • Dictionary docalign was not processing all the possible matches, what might lead to non deterministic results with an older version of the docalign utility script.

cgr71ii added 30 commits May 4, 2022 16:41
@cgr71ii
Add NDA and Vecalign rules
ec8bbee
@cgr71ii
Add NDA and Vecalign submodules
25e9de8
@cgr71ii
Refactor and more support to NDA and Vecalign
e9a18b7 
Documentation updated and installation support
@cgr71ii
Minor change
7ba4e83
@cgr71ii
Fix rule raw (input path)
f33786b
@cgr71ii
Minor fix
799111c
@cgr71ii
Add new tests
7f48a92 
Neural tests
@cgr71ii
Fix raw rule
8237191 
Update neural dependencies installation
@cgr71ii
Forgotten vecalign files
d8cfd0c
@cgr71ii
Minor changes
981b0da
@cgr71ii
Fix vecalign
0b1c024 
Add header, remove N-N alignments and only allow vecalign if NDA
and vice verse
@cgr71ii
Minor change
ad8881d
@cgr71ii
Update submodule vecalign
89670cf
@cgr71ii
Update vecalign scripts
3543762 
Vecalign optional installation removed due to new setup.py
installation
@cgr71ii
Add vecalign installation to conda and CI
e96a483
@cgr71ii
Fix setup.py
6d85b47
@cgr71ii
Update submodule vecalign
7a6076e
@cgr71ii
Update submodule NDA
ed24490
@cgr71ii
Support NDA header
dbb333c
@cgr71ii
Update submodule vecalign
a6c62d4
@cgr71ii
Vecalign paragraph id support
ce7227c
@cgr71ii
Update run-tests-min.sh
44ee007
@cgr71ii
Update submodule vecalign
1d0cf03
@cgr71ii
Update submodule vecalign
aeb9371
@cgr71ii
Update submodule vecalign
97dd3e9
@cgr71ii
Update submodule vecalign
1ddd19d 10000
@cgr71ii
Update submodule vecalign
8e2f75f
@cgr71ii
vecalign paragraphs support
bd91148
@cgr71ii
Add deferred support to vecalign
205f37b
@cgr71ii
Check simplified in rule split_segalign
6002b27
cgr71ii added 23 commits May 19, 2022 12:25
@cgr71ii
Minor fix
95402f0
@cgr71ii
Fix vecalign script
57c93d7
@cgr71ii
Fix vecalign script
0a99d4d
@cgr71ii
Update submodule vecalign
72fdf1d
@cgr71ii
Fix vecalign script
ea61509
@cgr71ii
Remove neural test from min tests
df1ab02
@cgr71ii
Merge branch 'master' into neural
362c4e5
@cgr71ii
Remove unnecessary code in tests and CI
86f27d1
@cgr71ii
Submodules moved to third_party directory
5da9f25
@cgr71ii
Fix submodules path in all files
acaf2e7
@cgr71ii
Fix deferred test
8586bba
@cgr71ii
Minor dic pipeline fixes
2c16492
@cgr71ii
Revert changes in structure distance script
b2f9bf8
@cgr71ii
Minor fix dic docalign
660e38a
@cgr71ii
Fix duplicate matches in dic docalign
a579329 
Variable was not being reseted and that might lead to duplicate
matches
@cgr71ii
Fix hunalign script
502ba1d 
Hunalign might not find any matches, and it was failing in that case
@cgr71ii
Remove unnecessary dependencies installation from CI
9de0850
@cgr71ii
Revert last changes about CI
44e2140
@cgr71ii
Minor fix hunalign
c03fe07 
N * N alignments were failing when trg idx had been matched before
@cgr71ii
NDA docalign integrated with hunalign
77050a7
@cgr71ii
Merge branch 'master' into neural
ca165d2
@cgr71ii
Specify embeddings model and update CONFIG.md
65624ea
@cgr71ii
Merge branch 'master' into neural
c84cc48
@cgr71ii cgr71ii marked this pull request as ready for review June 1, 2022 07:38
@cgr71ii cgr71ii requested a review from lpla June 1, 2022 07:38
@cgr71ii cgr71ii merged commit fdc7e08 into master Jun 1, 2022
@lpla lpla deleted the neural branch January 25, 2023 15:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lpla lpla lpla approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@cgr71ii @lpla
0