Adapters facilitate the ingestion of data from various sources into TN, leveraging the SDK and Prefect for efficient data management.
Note: While adapters use a Python Framework, it's also possible to push data from other languages like Go or JavaScript. Refer to the respective SDKs for more information:
- Programming Knowledge:
- Proficiency in Python
- Prefect knowledge (https://www.prefect.io)
- Tools:
- Clone the Repository:
git clone https://github.com/trufnetwork/adapters.git
cd adapters- Configure Environment Variables:
- Duplicate the
.env.examplefile and rename it to.env. - Update the environment variables in the
.envfile as needed.
- Setup Virtual Environment:
We recommend using a virtual environment to manage dependencies. You can create one using the following commands:
python -m venv .venv
source .venv/bin/activate
pip install -e ."[dev]"- Launch Services with Docker Compose:
docker compose up -dThis command initializes the necessary services in the background.
- Browser:
If you prefer to see the flows and blocks in browser, you can access http://localhost:4200/dashboard, on your local machine.
The repository offers several reusable tasks to facilitate data ingestion into TN. However you can develop your own tasks to fit your use-case, these tasks can serve as example how to implement custom adapter to your data:
Reading Data:
read_repo_csv_file: Read a CSV file from a GitHub repository.task_read_gsheet: Retrieves data from a Google Sheet.
TN Operations:
task_insert_tsn_records: Inserts records into TN.task_get_all_tsn_records: Fetches all records from TN.task_deploy_primitive_if_needed: Deploys a primitive if it doesn't already exist.
Data Manipulation:
task_reconcile_data: Reconciles data between sources.task_normalize_source: Standardizes source data.task_filter_by_source_id: Filters data based on a source ID.task_prepare_records_for_tsn: Prepares records for insertion into TN.
The repository includes examples demonstrating data ingestion from Google Sheets:
- Direct Method: Specify the sheet ID and source ID directly.
- Dynamic Method: Retrieve the sheet ID and source ID from a CSV file in a GitHub repository.
Refer to the examples directory for detailed implementations.