8000 clo: Add CLI command to extract a compressed file as IR. by haiqi96 · Pull Request #437 · y-scope/clp · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

clo: Add CLI command to extract a compressed file as IR. #437

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
10000
Merged
merged 106 commits into from
Jun 22, 2024
Merged

clo: Add CLI command to extract a compressed file as IR. #437

merged 106 commits into from
Jun 22, 2024

Conversation

haiqi96
Copy link
Contributor
@haiqi96 haiqi96 commented Jun 12, 2024 •
edited
Loading

Description

The change refactored clo so it takes in a command and can extract a file split into one or more IR chunks.
clo extracts the IRs to a local file system path, and sends metadata to a mongodb specified by the user input.

Validation performed

  1. Verified that CLP package still properly return search results.
    The following validation are all performed directly via commandline (as opposed to in a clp package)
  2. Compressed a hadoop64MB log into an archive, resulting in a single file split.
  3. Extract the file split as IR with target-size = 16MB (resulting in 5 IR chunks), with results sent to a mongodb running in container
  • Verified that metadata are properly written to the mongodb collection
  • Verified that is_last_ir_chunk is set properly in the metadata.
  • Verified that IR chunks are successfully written to the disk

haiqi96 added 30 commits May 15, 2024 19:16
@haiqi96
Add combined file msg offset to the global metadata databases
ad69313
@haiqi96
Add combined file message offset to reader side file and File metadata
d53afcd
@haiqi96
Linter fix
12d39a9
@haiqi96
Linter fix again
9f3c1cb
@haiqi96
Fix comment
87456f5
@haiqi96
address code review comments
73c4216
@haiqi96
linter
437ee87
@haiqi96
small fix
dc56550
@haiqi96
Update msg_idx to message_ix for consistency
0a5cde1
@haiqi96
Update msg_idx to message_ix for consistency
ad9bbd8
@haiqi96
Further clean up 10000
83eaaf1
@haiqi96
Linter
7cd1ff6
@haiqi96
Address code review comments
7700b8f
@haiqi96
Add partial support for file ID & msg_ix querying
06e3975
@haiqi96
Add support for global MySQL database
dd355ad 
� Conflicts:
�	components/core/src/clp/GlobalMySQLMetadataDB.cpp
@haiqi96
Replace file iterator with simpler function
8b09311
@haiqi96
Get previous code's review changes.
abd1940
@haiqi96
Linter
75fc71b
@haiqi96
rebase and linter
f17e955
@haiqi96
Update comments and function interfaces
58a28ce
@haiqi96
Remove extra empty line
201b336
@haiqi96
Resolve code review concerns
66fe18e
@haiqi96
Revert string view
3e0da95
@haiqi96
Merge branch 'main' into FileIDFilter
3246c61
@haiqi96
Replace magic number with constexpr to avoid confusion
2781f8a
@haiqi96
Update docstring
356736d
@haiqi96
Add Ir decompression api
8b2d2b1
@haiqi96
Linter
aa9377f
@haiqi96
10000
Retouch the core Archive to IR functions
85e928d
@haiqi96
Merge remote-tracking branch 'origin/main' into ArchiveToIR
323e80e
kirkrodrigues
kirkrodrigues requested changes Jun 15, 2024
View reviewed changes
components/core/src/clp/clo/utils.hpp Outdated
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this file only contains the exception class, I would name it CloOperationFailed.hpp

LinZhihao-723
LinZhihao-723 requested changes Jun 20, 2024
View reviewed changes
Copy link
Member
@LinZhihao-723 LinZhihao-723 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ask Haiqi asked, this is a review on CommandLinArguments.hpp/cpp and clo.cpp.
Note that the review is brief, mostly catching micro-level issues. Didn't review it from an architecture level, so the variable naming + API design review might be required (please let me know if I should do a further as well).

kirkrodrigues
kirkrodrigues approved these changes Jun 22, 2024
View reviewed changes
Copy link
Member
@kirkrodrigues kirkrodrigues left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the PR title, how about:

clo: Add CLI command to extract a compressed file as IR.

@haiqi96 haiqi96 changed the title core-clp: Add CLI command in clo to extract a compressed file as IR. clo: Add CLI command to extract a compressed file as IR. Jun 22, 2024
@haiqi96 haiqi96 merged commit ea99f50 into y-scope:main Jun 22, 2024
11 checks passed
jackluo923 pushed a commit to jackluo923/clp that referenced this pull request Dec 4, 2024
@haiqi96
clo: Add CLI command to extract a compressed file as IR. (y-scope#437)
4cc27e3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kirkrodrigues kirkrodrigues kirkrodrigues approved these changes

@LinZhihao-723 LinZhihao-723 LinZhihao-723 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@haiqi96 @kirkrodrigues @LinZhihao-723
0