Basics for SAIL station data #126
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cb7c7f0 to
f774941
Compare
There was a problem hiding this comment.
Pull Request Overview
This PR implements enhancements for processing SAIL station data including new test cases, updated dependency requirements, and refactored data utilities and API wrappers.
- Introduces comprehensive tests for SAIL data retrieval.
- Updates dependencies and version numbers in setup and init files.
- Refactors SensorDescription formatting and implements SAILPointData methods with a variable validation check.
Reviewed Changes
Copilot reviewed 12 out of 13 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_sail.py | New tests for SAIL data methods with a minor spelling error in a comment. |
| tests/test_geosphere.py | Adjusted frequency pattern in date_range creation. |
| setup.py | Added new dependencies and bumped version. |
| metloom/variables.py | Reformatted SensorDescription definitions for clarity. |
| metloom/pointdata/sail.py | Added SAILPointData implementation; contains a variable validation bug. |
| metloom/dataframe_utils.py | Introduced a new resample_series function and updated resample_whole_df. |
| metloom/arm_utils.py | Added ARM data utility functions for file download and processing. |
| metloom/init.py | Bumped package version. |
| .github/workflows/testing.yml | Updated flake8 configuration (increased max line length, extended ignores). |
Files not reviewed (1)
- setup.cfg: Language not supported
|
@aeslaughter you crushed this fast! Looking good, just a couple tweaks |
|
FYI, I figured out how to do the file combination and conversion to CSV within the API. I am switching to it, it will get rid of the need for netcdf and simplify the download logic. |
micah-prime
reviewed
May 8, 2025
8f3dad6 to
98da762
Compare
I raised the line limit, most of the files have not been formatted and have lines much longer
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
7f7b922 to
9bfaf60
Compare
9bfaf60 to
4963b7a
Compare
There was a problem hiding this comment.
Pull Request Overview
This PR introduces the basic support for handling SAIL station data by adding new tests and implementing the SAILPointData functionality while updating sensor definitions and dependencies.
- Adds comprehensive tests for both hourly and daily SAIL data retrieval
- Refactors sensor description definitions and variable handling in multiple modules
- Implements a new SAILPointData class including geometry based lookup and ARM data utilities
Reviewed Changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/test_sail.py | New tests covering data retrieval and date validations |
| tests/test_geosphere.py | Minor update of frequency parameter in pd.date_range |
| setup.py | Added dependency on pydash |
| metloom/variables.py | Refactoring of SensorDescription instantiation and field list extensions |
| metloom/pointdata/sail.py | New implementation of SAILPointData and helper functions |
| metloom/pointdata/base.py | Updated points_from_geometry method as a class method |
| metloom/dataframe_utils.py | Introduced resample_series and preserved resample_whole_df functionality |
| metloom/arm_utils.py | New utility for interacting with ARM Live Data Webservice |
| .github/workflows/testing.yml | Adjusted flake8 configuration to allow longer lines during testing |
|
@micah-prime I believe I have addressed your comments as well as implemented the points_from_geometry function and tests. |
e9cc53e to
fd79717
Compare
fd79717 to
fe384cf
Compare
21cd18e to
2bcf797
Compare
33a9a60 to
d908ecf
Compare
07976bc to
f5280b5
Compare
f5280b5 to
7a234b6
Compare
There was a problem hiding this comment.
Pull Request Overview
This PR adds support for SAIL station data by implementing the SAILPointData class and defining SAIL-specific sensor variables while also introducing a new battery of tests to verify data retrieval and geometry queries.
- Introduces SAILPointData in metloom/pointdata/sail.py with methods to download hourly and daily data.
- Adds SAILStationVariables and updates sensor variable definitions in metloom/variables.py.
- Implements new tests in tests/test_sail.py and updates workflow and dependency configurations.
Reviewed Changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_sail.py | New tests to verify SAILPointData functionality and geometry-based point selection. |
| tests/test_geosphere.py | Minor update to frequency string in date range generation. |
| setup.py | Added new dependency (pydash) to support ARM data processing. |
| metloom/variables.py | Refactored SensorDescription instantiations and added SAILStationVariables. |
| metloom/pointdata/sail.py | Implements retrieval and processing of SAIL station data, along with location lookup. |
| metloom/pointdata/base.py | Adjusted points_from_geometry method to use class method semantics. |
| metloom/pointdata/init.py | SAILPointData added to public API. |
| metloom/dataframe_utils.py | Enhanced resampling functions with additional documentation. |
| metloom/arm_utils.py | Introduced ARM data retrieval tooling with concurrency support. |
| .github/workflows/testing.yml | Minor change to flake8 configuration to relax max line-length and ignore additional warnings. |
micah-prime
approved these changes
May 27, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://www.arm.gov/research/campaigns/amf2021sail
closes #109