8000 [tools/data_processing_v2] - Make Processing Data More User Friendly · Issue #368 · waterloo-rocketry/omnibus · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

[tools/data_processing_v2] - Make Processing Data More User Friendly #368

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
ChrisYx511 opened this issue Mar 8, 2025 · 0 comments · May be fixed by #382
Open

[tools/data_processing_v2] - Make Processing Data More User Friendly #368

ChrisYx511 opened this issue Mar 8, 2025 · 0 comments · May be fixed by #382
Assignees
@ChrisYx511 @Mustafa-Mian
Labels
enhancement New feature or request

Comments

@ChrisYx511
Copy link
Contributor
ChrisYx511 commented Mar 8, 2025 •
edited
Loading

Feature Description

  • Expected implementation date: 2025-03-29

  • As a: DAQ Ops,

  • Expected Behavior: I would want to be able to have a nice interface / CLI to use to process globallog log files into CSVs. Furthermore, when processing the msgpacked log files, the entire log should be processed line by line and shouldn't be read into memory before processing.

  • Because: A proper CLI / UI would make data processing a lot more user friendly while retaining the new features. Making the code not reliant on memory space reduces memory requirements and prevents most OOM issues. Therefore, it would allow for very large logs to processed without fuss (and then processed in chunks later in Pandas).

  • Links to related issues / PRs / documents / discussions (Slack): https://docs.google.com/document/d/1G8s8M530FyW4kbQcjkgZ2ImOGnDQj_MT3Omr9mVWqEw/edit?tab=t.0#heading=h.watkmfcldmtw

Related: #212

Requirements

Specific Requirements

  • The script MUST provide a basic CLI that allows the user to run it as python main.py daq [source file] and output to a CSV file in the same folder and with a default file name + a random hash at the end (example: omnibus-processed-daq-[date]-[hash].csv).
  • All DAQ functions MUST be in a daq subcommand (as seen above) to distinguish from future log parsing functionality.
  • The script SHOULD provide a CLI option -o [outfile] to specify the output file name
  • The script MUST process the msgpacked log line by line, do not read into a single list

Standard Requirements - Do Not Modify Without Approval

  • All changes made MUST be tested and reviewed by a code reviewer.
  • The conditions outlined in CONTRIBUTING.MD MUST be respected.
  • Deep consider of new library dependencies MUST be made before adding them to the project.
  • Keep the codebase clean, don't push non-code files (e.g. DATA, VirtualEnv, etc.) to the repository.
  • All new feature changes made SHOULD have associated unit tests (if applicable)
  • Keep well-documented code, and follow the style guide.

Satisfying the above requirements WILL close this issue.

Stakeholder / Product Owner

DAQ Ops

Mention: @ChrisYx511
@ChrisYx511 ChrisYx511 added the enhancement New feature or request label Mar 8, 2025
@ChrisYx511 ChrisYx511 mentioned this issue Mar 8, 2025
Merged
@ChrisYx511 ChrisYx511 changed the title [tools/data_processing_v2] - New Data Processing Script! [tools/data_processing_v2] - Make Processing Data More User Friendly Mar 18, 2025
@ChrisYx511 ChrisYx511 self-assigned this Mar 18, 2025
@Mustafa-Mian Mustafa-Mian linked a pull request May 6, 2025 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ChrisYx511 ChrisYx511

@Mustafa-Mian Mustafa-Mian

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

m2mian/368 make processing data more user friendly waterloo-rocketry/omnibus
2 participants
@ChrisYx511 @Mustafa-Mian
0