python tools for adding clock data to a timescale db.
- Ensure you have Python 3.9 or higher installed
- Pip install the latest version of opensampl:
pip install opensampluv venv
uv sync --extra all
source .venv/bin/activateThis will create a virtual environment and install the development dependencies.
The tool requires several environment variables. Create a .env file in your project root:
When routing through a backend:
ROUTE_TO_BACKEND=true # Set to true if using backend service
BACKEND_URL=http://localhost:8000 # Only needed if ROUTE_TO_BACKEND is true
# Archive configuration
ARCHIVE_PATH=/path/to/archive # Where processed files are storedWhen directly accessing db:
# Database connection
DATABASE_URL=postgresql://<user>:<password>@<host>:<port>/<database>
# Archive configuration
ARCHIVE_PATH=/path/to/archive # Where processed files are storedThe CLI tool provides several commands. You can use opensampl --help (or, any deeper opensampl [command] --help) to get details
Load data from ADVA probes:
# Load single file
opensampl load probe adva path/to/file.txt.gz
# Load directory of files
opensampl load probe adva path/to/directory/ADVA probes have all their metadata and their time data in each file, so no need to use the -m or -t options, though if you want to skip loading one or the other it becomes useful!
options:
--metadata(-m): Only load probe metadata--time-data(-t): Only load time series data--no-archive(-n): Don't archive processed files--archive-path(-a): Override default archive directory--max-workers(-w): Maximum number of worker threads (default: 4)--chunk-size(-c): Number of time data entries per batch (default: 10000)
Load data directly into a database table. Format can be yaml or json. Can be a list of dictionaries or a single dictionary.
you do not have to specify schema, is assumed to be castdb.
The --if-exists option controls how to handle conflicts:
- update: Only update fields that are provided and non-default (default)
- error: Raise an error if entry exists
- replace: Replace all non-primary-key fields with new values
- ignore: Skip if entry exists
opensampl load table table_name path/to/data.yamlSo, you can do things like the following
opensampl load table locations --if-exists replace updated_location.yamlWhere this is the updated_location
name: EPB Chattanooga
lat: 35.9311256
lon: -84.3292469And it will overwrite the existing entry for EPB Chattanooga, or create a new one if it doesn't exist yet.
Display current environment configuration:
# Show all variables
poetry run opensampl config show
# Show with descriptions
poetry run opensampl config show --explain
# Show specific variable
poetry run opensampl config show --var DATABASE_URLUpdate environment variables:
poetry run opensampl config set VARIABLE_NAME valueThe tool currently supports ADVA probe data files with the following naming convention:
<ip_address>CLOCK_PROBE-<probe_id>-YYYY-MM-DD-HH-MM-SS.txt.gz
Example: 10.0.0.121CLOCK_PROBE-1-1-2024-01-02-18-24-56.txt.gz
Stores geographic locations with their coordinates and metadata. Supports both 2D and 3D point geometries.
name: "Lab A" # Unique name for the location
lat: 35.93 # Latitude coordinate
lon: -84.31 # Longitude coordinate
z: 100 # Optional elevation in meters
projection: 4326 # Optional SRID/projection (defaults to 4326/WGS84)
public: true # Optional boolean for public visibilityTracks testing periods and experiments with start and end timestamps.
name: "Holdover Test 1" # Unique name for the test
start_date: "2024-01-01T00:00:00" # Test start timestamp
end_date: "2024-01-07T00:00:00" # Test end timestampContains information about timing probes, including their network location and associated metadata. Insertion handled by opensampl load probe.
probe_id: "1-1" # Probe identifier
ip_address: "10.0.0.121" # IP address of the probe
vendor: "ADVA" # Vendor type
model: "OSA 5422" # Model number
name: "GMC1" # Human-readable name
public: true # Optional boolean for public visibility
location_uiid: "123e4567-e89b-12d3-a456-426614174000" # Optional reference to location
test_uiid: "123e4567-e89b-12d3-a456-426614174001" # Optional reference to testTime series data from probes, storing timestamps and measured values. Insertion handled by opensampl load probe.
time: "2024-01-01T00:00:00" # Timestamp of measurement
probe_uuid: "123e4567-e89b-12d3-a456-426614174000" # Reference to probe
value: 1.234e-09 # Measured valueADVA-specific configuration and status information for probes. Insertion handled by opensampl load probe.
probe_uuid: "123e4567-e89b-12d3-a456-426614174000" # Reference to probe
type: "Phase" # Measurement type
start: "2024-01-01T00:00:00" # Start timestamp
frequency: 1 # Sampling frequency
timemultiplier: 1 # Time multiplier
multiplier: 1 # Value multiplier
title: "ClockProbe1" # Probe title
adva_probe: "ClockProbe" # Probe type
adva_reference: "GPS" # Reference source
adva_reference_expected_ql: "QL-NONE" # Expected quality level
adva_source: "TimeClock" # Source type
adva_direction: "NA" # Direction
adva_version: 1.0 # Version number
adva_status: "RUNNING" # Operating status
adva_mtie_mask: "G823-PDH" # MTIE mask type
adva_mask_margin: 0 # Mask margin- All tables use UUIDs as primary keys which are automatically generated.
- Table relationships are maintained through UUID references
- Geographic coordinates use WGS84 projection (SRID 4326) by default
- Boolean fields (public) are optional and can be null