A comprehensive data processing and analytics pipeline for Mezo mainnet data, including bridge transactions, MUSD lending, and collateralization analysis.
This repository provides automated data collection, processing, and analysis tools for Mezo's mainnet operations. It fetches data from multiple sources including subgraphs, APIs, and Mezo's blockchain explorer, processes it, and stores results in Supabase.
- Bridge Transactions: Cross-chain bridge activity via Mezo Bridge subgraph
- MUSD Lending: Loan origination, adjustments, and liquidations via MUSD subgraphs
- Token Prices: Real-time pricing data from CoinGecko API
- MUSD Token Data: On-chain transfer and holder data from Mezo Explorer API
Data Sources → Processing Scripts → Supabase Database → Analytics/Visualization
↓ ↓ ↓ ↓
Subgraphs Python Pipeline Live Tables Notebooks
APIs Visual Progress Auto-Schema Dashboards
Explorer Error Handling Time-Series Reports
mezo-analytics/
├── .github/workflows/ # GitHub Actions for automation
│ ├── data-processing.yml # Main cron job (every 6 hours)
│ └── manual-data-processing.yml # Manual trigger workflow
├── mezo/ # Core library modules
│ ├── clients.py # Database and API clients
│ ├── currency_config.py # Token mappings and configurations
│ ├── currency_utils.py # Price fetching and formatting
│ ├── data_utils.py # Data processing utilities
│ ├── datetime_utils.py # Date/time formatting
│ ├── queries.py # GraphQL queries
│ └── visual_utils.py # Progress indicators and UI
├── scripts/ # Main processing scripts
│ ├── get_raw_data.py # Data fetching functions
│ ├── process_bridge_data.py # Bridge transaction processing
│ ├── process_musd_data.py # MUSD lending data processing
│ └── archive/ # Legacy scripts
├── tests/ # Test and debug utilities
├── notebooks/ # Jupyter analysis notebooks
├── data/ # Local data storage
└── requirements.txt # Python dependencies
- Python 3.13
- Supabase account
- CoinGecko API key
-
Clone the repository:
git clone <repository-url> cd mezo-analytics-mainnet
-
Install dependencies:
pip install -r requirements.txt
-
Configure environment variables: Create a
.envfile with:SUPABASE_URL_PROD=your_mezo_portal_prod_supabase_url SUPABASE_KEY_PROD=your_mezo_portal_prod_supabase_key SUPABASE_DATA_URL=your_data_analytics_supabase_url SUPABASE_DATA_KEY=your_data_analytics_supabase_key COINGECKO_KEY=your_coingecko_api_key
The repository includes GitHub Actions workflows that run automatically:
- Every 6 hours: Processes both bridge and MUSD data
- Manual trigger: On-demand processing via GitHub Actions UI
Run individual scripts locally:
# Process bridge transaction data
python scripts/process_bridge_data.py
# Process MUSD lending data
python scripts/process_musd_data.py- Bridge Analytics: Daily aggregations by token with USD values
- MUSD Metrics: Loan lifecycle, liquidations, and system health
- System Health: Collateralization ratios and risk metrics
- Time Series: Daily, rolling, and cumulative metrics
- Cohort Analysis: User behavior and loan patterns
- Risk Assessment: Collateralization and liquidation analysis
- Token Distribution: Cross-chain asset flow tracking
mainnet_daily_bridge_data- Daily bridge volume by tokenmainnet_bridge_by_token- Token-level bridge summariesmainnet_bridge_summary- Overall bridge statistics
mainnet_musd_daily- Daily lending activity and balancesmainnet_musd_borrow_summary- Cumulative borrowing metricsmainnet_musd_system_health- Collateralization and stabilitymainnet_musd_averages- Loan size and ratio statisticsmainnet_musd_token_summary- Token holder and transfer data
-
currency_config.py: Token mappings and configurationsTOKEN_MAP: Contract address → Symbol mappingTOKEN_TYPE_MAP: Symbol → Category (bitcoin, stablecoin, ethereum)TOKENS_ID_MAP: Symbol → CoinGecko ID for price fetching
-
subgraph_config.py: Centralized subgraph endpoint definitions- Portal, Market, Bridge, Borrower Operations subgraphs
- MUSD Token, Stability Pool, Trove Manager endpoints
-
currency_utils.py: Currency formatting and price fetchingformat_currency_columns(): Handles token decimal conversions (1e6, 1e8, 1e18)get_token_prices(): Fetches USD prices from CoinGecko APIreplace_token_labels(): Maps contract addresses to symbols
-
datetime_utils.py: Date and time processingconvert_unix_to_datetime(): Unix timestamp conversion with timezone handlingformat_datetimes(): Standardizes date columns to YYYY-MM-DD formatgroupby_date(): Date-based aggregation helper
-
data_utils.py: Analytics and transformation utilitiesadd_cumulative_columns(): Creates cumulative sum columnsadd_pct_change_columns(): Calculates percentage change metricsadd_rolling_values(): Generates rolling window averages
queries.py: GraphQL query templatesMUSDQueries: Loan data, liquidations, collateral snapshotsBridgeQueries: Bridge transactions, deposits, withdrawals
clients.py: Database and API communicationSupabaseClient: Dual-database architecture (production + data warehouse)SubgraphClient: GraphQL query execution with paginationAPIClient: Generic HTTP client for REST APIs
visual_utils.py: Progress indicators and error handlingProgressIndicators: Visual symbols (✅❌⚠️ ) and progress barsExceptionHandler: Retry logic and safe execution with visual feedback- Decorators:
@with_progress()and@safe_operation()
- Create fetch functions in
scripts/get_raw_data.py - Add processing logic following existing patterns and naming conventions
- Update configurations in
mezo/currency_config.py - Add tests in
tests/
- Follow existing code patterns and visual indicators
- Add comprehensive error handling
- Include progress indicators for long-running operations
- Write tests for new functionality
- Update documentation
The notebooks/ directory contains analysis examples:
- Bridge transaction flow analysis
- MUSD lending market dynamics
- Collateralization ratio distributions
- Cross-chain asset migration patterns
- Set up environment variables in
.env - Run initial processing:
python scripts/process_bridge_data.py
- Check Supabase for generated tables and data
- Enable GitHub Actions for automated processing
- Explore notebooks for analysis examples