From 24ee005406383c50ee6a6d8b1666046e0c9b62ee Mon Sep 17 00:00:00 2001 From: Jennie Gao Date: Wed, 16 Apr 2025 13:01:18 -0700 Subject: [PATCH 1/8] swap out default readme --- README.md | 485 +++++++++++++----- airbtye-local-cli-bash/README.md | 167 ++++++ .../airbyte-local.sh | 0 .../resources}/demo.gif | Bin .../test}/.shellspec | 0 .../test}/Dockerfile | 0 .../test}/spec/airbyte-local_spec.sh | 0 .../test}/spec/spec_helper.sh | 0 airbyte-local-cli-nodejs/README.md | 403 +-------------- 9 files changed, 535 insertions(+), 520 deletions(-) create mode 100644 airbtye-local-cli-bash/README.md rename airbyte-local.sh => airbtye-local-cli-bash/airbyte-local.sh (100%) rename {resources => airbtye-local-cli-bash/resources}/demo.gif (100%) rename {test => airbtye-local-cli-bash/test}/.shellspec (100%) rename {test => airbtye-local-cli-bash/test}/Dockerfile (100%) rename {test => airbtye-local-cli-bash/test}/spec/airbyte-local_spec.sh (100%) rename {test => airbtye-local-cli-bash/test}/spec/spec_helper.sh (100%) mode change 100644 => 120000 airbyte-local-cli-nodejs/README.md diff --git a/README.md b/README.md index 290ce087..70ac08a2 100644 --- a/README.md +++ b/README.md @@ -1,157 +1,406 @@ # Airbyte Local CLI [![CI](https://github.com/faros-ai/airbyte-local-cli/actions/workflows/ci.yaml/badge.svg)](https://github.com/faros-ai/airbyte-local-cli/actions/workflows/ci.yaml) -CLI for running Airbyte sources & destinations locally or on a Kubernetes cluster without an Airbyte server +CLI for running Airbyte sources & destinations locally. -![Alt Text](https://github.com/Faros-ai/airbyte-local-cli/raw/main/resources/demo.gif) +> **Note:** The previous Bash-based version of this CLI has been deprecated. +> You can find its documentation [here](./airbyte-local-cli-bash/README.md). +> For migration details, see the [Migration Guide](#migration-guide). -## Example Usage +## Table of Contents -**Requirements**: `bash`, `jq`, `tee`. Additionally, `docker` when running syncs locally, or `kubectl` when running on a Kubernetes cluster. +- [Getting Started](#getting-started) + - [Prerequisites](#prerequisites) + - [Step 1. Install](#step-1-install) + - [Step 2. Create an Airbyte Configuration File](#step-2-create-an-airbyte-configuration-file) + - [Step 2a. Auto Generate Configuration Templates (Experimental)](#step-2a-auto-generate-configuration-templates-experimental) + - [Step 2b. Craft your own configuration](#step-2b-craft-your-own-configuration) + - [Step 3. Run it!](#step-3-run-it) +- [Advanced Settings](#advanced-settings) + - [CLI Arguments](#cli-arguments) + - [Airbyte Configuration - Override Airbyte Catalog](#airbyte-configuration---override-airbyte-catalog) + - [Airbyte Configuration - Customize Docker Settings](#airbyte-configuration---customize-docker-settings) +- [FAQ](#faq) +- [Migration Guide](#migration-guide) + - [Old CLI v.s. New CLI](#old-cli-vs-new-cli) + - [New/Renamed Arguments](#newrenamed-arguments) + - [Unsupported Arguments](#unsupported-arguments) -Either [download the script manually](https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) or invoke the script directly with curl: +## Getting Started + +**Supported System** + +- Linux x86_64 +- MacOS arm64 (Apple chip) +- Windows x86_64 + +### Prerequisites + +- Docker +- Faros API key: check out the [instructions](https://docs.faros.ai/reference/getting-api-access). + +### Step 1. Install + +[All releases](https://github.com/faros-ai/airbyte-local-cli/releases.) + +#### Linux/MacOS + +Here is the steps of downloading the CLI on MacOS. Linux should have very similar steps. ```sh -bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) --help +# Download from the CLI +# Please swap out `v0.0.5` to the version you want +wget -O airbyte-local.zip https://github.com/faros-ai/airbyte-local-cli/releases/download/v0.0.5/airbyte-local-macos-arm64.zip | unzip -o airbyte-local.zip + +# run `--help` or `--version` to check if the CLI is installed correctly +./airbyte-local # this shows the help manual +./airbyte-local --help +./airbyte-local --version ``` -For example here is how you can sync ServiceNow source with [Faros Cloud](https://www.faros.ai) destination: +#### Windows (Powershell) + +```ps1 +# Download from the CLI +# Please swap out `v0.0.5` to the version you want +Invoke-WebRequest -Uri "https://github.com/faros-ai/airbyte-local-cli/releases/download/v0.0.5/airbyte-local-win-x64.zip" -OutFile "airbyte-local-win-x64.zip" +Expand-Archive -Path "airbyte-local-win-x64.zip" -DestinationPath . -Force + +# run `--help` or `--version` to check if the CLI is installed correctly +.\airbyte-local +.\airbyte-local --help +.\airbyte-local --version -```sh -./airbyte-local.sh \ - --src 'farosai/airbyte-servicenow-source' \ - --src.username '' \ - --src.password '' \ - --src.url '' \ - --dst 'farosai/airbyte-faros-destination' \ - --dst.edition_configs.edition 'cloud' \ - --dst.edition_configs.api_url '' \ - --dst.edition_configs.api_key '' \ - --dst.edition_configs.graph 'default' \ - --state state.json \ - --check-connection ``` -Or with [Faros Community Edition](https://github.com/faros-ai/faros-community-edition) as the destination: +### Step 2. Create an Airbyte Configuration File + +First of all, you will have to know where you want to pull data from and where you want to push data to. +For example, pulling data from Github and pushing data to Faros AI. + +Then, there're two options you can go to create the config file. + +a. Auto Generate Configuration Templates: +This is recommended for first-time users. It helps you to start with a template to update the Airbyte configs. (Go to Step 2a) + +b. Craft your own configuration: This is for users that are familiar with Airbyte configurations and are looking for a finer tune on the configs. (Go to Step 2b) + +#### Step 2a. Auto Generate Configuration Templates (Experimental) + +You can utilize the `generate-config` subcommand to bootstrap your Airbyte config. +It's required to provide the Airbyte source, which means you will have to know which source data you are pulling from, e.g. Github, Jira, etc. For the Airbtye destination, it is set to Faros by default, i.e. pushing the data to Faros. +For users convenience, the CLI will attempt to guess the source and destination from your inputs. It handles some typos and is case insensitive. +If the CLI cannot correctly guess the source/destination, or you want to use your own images that are not managed by Faros, you can use the option `--image` to specify your images. +The CLI will use your exact input image(s) to generate Airbyte configurations. +The default destination image is set to Faros, i.e. `farosai/airbyte-faros-destination`. + +By running this subcommand, it prints out both Airbyte source and deestination configuration tables in the terminal for your reference. +And it generates a template config file `faros_airbyte_cli_config.json` in the current directory. The template config file only includes requried configs. If you need additional configs, please refer to the configuration tables and update the config file. + +Run the command to generate the template ```sh -./airbyte-local.sh \ - --src 'farosai/airbyte-servicenow-source' \ - --src.username '' \ - --src.password '' \ - --src.url '' \ - --dst 'farosai/airbyte-faros-destination' \ - --dst.edition_configs.edition 'community' \ - --dst.edition_configs.hasura_admin_secret 'admin' \ - --dst.edition_configs.hasura_url 'http://host.docker.internal:8080/' \ - --state state.json \ - --check-connection +# ./airbyte-local generate-config [destination] + +# Ex: Pull data from Github and push to Faros +./airbyte-local generate-config github +./airbyte-local generate-config github faros + +# Ex: Pull data from Jira and push to Faros +./airbyte-local generate-config jira + +# Suppress printing out the configuration tables +./airbyte-local generate-config -s jira + +# Use your own custom images +./airbyte-local generate-config --image farosai/airbyte-github-custom-source +./airbyte-local generate-config -s --image farosai/airbyte-github-custom-source +./airbyte-local generate-config --image farosai/airbyte-github-custom-source farosai/airbyte-custom-destination ``` -**Note**: The `src.*` and `dst.*` arguments will differ depending on the source and destination being used. -Or on a Kubernetes cluster: +Note: Both source and destination inputs are case insensitive and tolerate a bit of typos. -```sh -./airbyte-local.sh \ - --src 'farosai/airbyte-servicenow-source' \ - --src.username '' \ - --src.password '' \ - --src.url '' \ - --dst 'farosai/airbyte-faros-destination' \ - --dst.edition_configs.edition 'cloud' \ - --dst.edition_configs.api_url '' \ - --dst.edition_configs.api_key '' \ - --dst.edition_configs.graph 'default' \ - --state state.json \ - --k8s-deployment \ - --k8s-namespace default \ - --max-cpus 0.5 \ - --max-mem 500Mi \ - --keep-containers +After running the command, you should see `🔹 **Next Steps:**` instructions showing up in your terminal. Follow the steps to complete the config. + +#### Step 2b. Craft your own configuration + +The CLI uses arguement `--config-file` (`-c` for short) to take the airybte configuration in a JSON file format. + +JSON Schema + +```json +{ + "src": { + "image": "", + "config": {...YOUR_SOURCE_CONFIG...} + }, + "dst": { + "image": "", + "config": {...YOUR_DESTINATION_CONFIG...} + } +} ``` -**Note**: The command assumes Kubernetes cluster context, and credentials are already configured. For more info, see [official docs](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/). +You can find all the available soure and destiantion images that are supported by Faros [here](https://hub.docker.com/u/farosai). \ +Here're some popular source image -## Configuring Faros source/destination using a wizard +- Github source: `farosai/airbyte-github-source` +- Jira source: `farosai/airbyte-jira-source` +- Faros destination: `farosai/airbyte-faros-destination` -**Note**: Faros Sources and/or Faros Destination only. Not supported with Kubernetes deployment. +#### Example of Airbyte Config File -Instead of passing `src.*` and `dst.*`, it is possible to invoke a configuration wizard for the Faros source -and/or destination: +Assuming you want to pull data from Github org `my-org` by using GitHub PAT and push data to Faros `default` workspace. This is what the JSON would look like +```json +{ + "src": { + "image": "farosai/airbyte-github-source", + "config": { + "api_key": "", <-- Faros API key + "graph": "default" <-- Faros workspace + "authentication": { + "type": "token", + "personal_access_token": "" + }, + "organizations": ["my-org"], + } + }, + "dst": { + "image": "farosai/airbyte-faros-destination", + "config": { + "edition_configs": { + "api_key": "", <-- Faros API key + "graph": "default" <-- Faros workspace + } + } + } +} ``` -./airbyte-local.sh \ - --src 'farosai/airbyte-servicenow-source' \ - --src-wizard \ - --dst 'farosai/airbyte-faros-destination' \ - --dst-wizard + +Save it as `faros_airbyte_cli_config.json`. \ +In most cases, you always have to provide Faros API key and workspace under `src.config` and `dst.config.edition_configs`. + +More resources you can find it in [Faros Documantation](https://docs.faros.ai/), e.g. instructions to create GitHub PAT and what permission you need for the PAT, etc. + +### Step 3. Run it! + +#### Linux/MacOS + +```sh +./airbyte-local --config-file 'faros_airbyte_cli_config.json' +./airbyte-local -c 'faros_airbyte_cli_config.json' +``` + +#### Windows (Powershell) + +```ps1 +.\airbyte-local --config-file 'faros_airbyte_cli_config.json' +.\airbyte-local -c 'faros_airbyte_cli_config.json' ``` -## Arguments - -| Argument | Required | Description | -| --------------------------------- | -------- | ------------------------------------------------------------------------------------------------- | -| --src \ | Yes | Airbyte source Docker image | -| --dst \ | Yes | Airbyte destination Docker image | -| --src.\ \ | | Append `"key": "value"` into the source config \* | -| --dst.\ \ | | Append `"key": "value"` into the destination config \* | -| --check-connection | | Validate the Airbyte source connection | -| --full-refresh | | Force source full_refresh and destination overwrite mode | -| --state \ | | Override state file path for incremental sync | -| --src-output-file \ | | Write source output as a file (handy for debugging) | -| --src-catalog-overrides \ | | JSON string of sync mode overrides. See [overriding default catalog](#overriding-default-catalog) | -| --src-config-file \ | | Source config file path | -| --src-config-json \ | | Source config as a JSON string | -| --src-catalog-file \ | | Source catalog file path | -| --src-catalog-json \ | | Source catalog as a JSON string | -| --dst-config-file \ | | Destination config file path | -| --dst-config-json \ | | Destination config as a JSON string | -| --dst-catalog-file \ | | Destination catalog file path | -| --dst-catalog-json \ | | Destination catalog as a JSON string | -| --dst-stream-prefix \ | | Destination stream prefix | -| --no-src-pull | | Skip pulling Airbyte source image | -| --no-dst-pull | | Skip pulling Airbyte destination image | -| --src-wizard | | Run the Airbyte source configuration wizard | -| --dst-wizard | | Run the Airbyte destination configuration wizard | -| --src-only | | Only run the Airbyte source | -| --dst-only \ | | Use a file for destination input instead of a source | -| --connection-name | | Connection name used in various places | -| --raw-messages | | Output raw Airbyte messages, i.e., without a log prefix or colors | -| --max-log-size \ | | Set Docker maximum log size | -| --max-mem \ | | Set the maximum amount of memory for Docker or Kubernetes container, e.g., `"1g"` or `"1024Mi"` | -| --max-cpus \ | | Set the maximum number of CPUs for each Docker or Kubernetes container, e.g, `"1"` or `"1000m"` | -| --src-docker-options "\" | | Set additional options to pass to the `docker run ` command, e.g `--src-docker-options "-e NODE_OPTIONS=--max_old_space_size=2000 -e NODE_TLS_REJECT_UNAUTHORIZED=0"` | -| --dst-docker-options "\" | | Set additional options to pass to the `docker run ` command, e.g `--dst-docker-options "-e NODE_OPTIONS=--max_old_space_size=2000"` | -| --k8s-deployment | | Deploy and run source/destination connectors as a pod on a Kubernetes cluster | -| --k8s-namespace \ | | Kubernetes namespace where the source/destination connectors pod is deployed to | -| --keep-containers | | Do not delete source and destination containers (or Kubernetes pod) after they exit | -| --debug | | Enable debug logging | - -**Note**: when passing an array value for a parameter specify it as a json array, for example: +The logs should indicate the process of the data sync. \ +You can also see the progress and logs in Faros App > Switch to the workspace you are running against > Admin Settings > Data Control > Sources. + +## Advanced Settings + +In some cases, you might want to customized the data sync more. \ +We provide some more CLI optional arguments and optional fields in the Airbyte configuration file. + +### CLI Arguments + +| Option | Required | Description | +| -------------------------- | -------- | ---------------------------------------------------------------------------------------------------------- | +| `-c, --config-file ` | Yes | Airbyte source and destination connector config JSON file path | +| `-h, --help` | | Display usage information | +| `-v, --version` | | Output the current version | +| `--full-refresh` | | Force full_refresh and overwrite mode. This overrides the mode in provided config file | +| `--state-file ` | | Override state file path for incremental sync | +| `--no-src-pull` | | Skip pulling Airbyte source image | +| `--no-dst-pull` | | Skip pulling Airbyte destination image | +| `--src-only` | | Only run the Airbyte source and write output in stdout. Use '--src-output-file' instead to write to a file | +| `--src-output-file ` | | Write source output as a file (requires a destination) | +| `--src-check-connection` | | Validate the Airbyte source connection | +| `--dst-only ` | | Use a file for destination input instead of a source | +| `--dst-use-host-network` | | Use the host network when running the Airbyte destination | +| `--log-level ` | | Set level of source and destination loggers (default: "info") | +| `--raw-messages` | | Output raw Airbyte messages | +| `--connection-name ` | | Connection name used in various places | +| `--keep-containers` | | Do not remove source and destination containers after they exit | +| `--debug` | | Enable debug logging | +| `--src ` | | [Deprecated] Airbyte source Docker image | +| `--dst ` | | [Deprecated] Airbyte destination Docker image | +| `--src. ` | | [Deprecated] Add "key": "value" into the source config | +| `--dst. ` | | [Deprecated] Add "key": "value" into the destination config | + +#### Example Usage + +##### Linux/MacOS + +```sh +# Turn on debug logs +./airbyte-local --config-file 'sample.json' --debug + +# Run source sync only +./airbyte-local \ + --config-file 'sample.json' \ + --src-only + +# Check source connection +./airbyte-local \ + --config-file 'sample.json' \ + --src-check-connection + +# Enforce full refreash +./airbyte-local \ + --config-file 'sample.json' \ + --full-refresh +# Use customized connection name +./airbyte-local \ + --config-file 'sample.json' \ + --connection-name 'test-connection' ``` ---src.projects '["project-1","project-2","project-3"]' + +##### Windows (Powershell) + +```ps1 +# Turn on debug logs +./airbyte-local --config-file 'sample.json' --debug + +# Run source sync only +./airbyte-local ` + --config-file 'sample.json' ` + --src-only + +# Check source connection +./airbyte-local ` + --config-file 'sample.json' ` + --src-check-connection + +# Enforce full refreash +./airbyte-local ` + --config-file 'sample.json' ` + --full-refresh + +# Use customized connection name +./airbyte-local ` + --config-file 'sample.json' ` + --connection-name 'test-connection' ``` -## Overriding Default Catalog +### Airbyte Configuration - Override Airbyte Catalog -To generate the Airbyte catalog needed for running the source and destination -connectors, the script runs the `discover` command on the source to get the list -of all supported streams. It then creates an Airbyte configured catalog, -enabling all of the streams and using "incremental" sync mode for all the -streams that support it. Each stream's destination sync mode defaults to -"append" for incremental streams and "overwrite" for full_refresh streams. To -disable or customize the sync mode or destination sync mode on any of the -streams, pass a `--src-catalog-overrides` option whose value is a JSON string in -the following format: +You can override the default Airbyte catalog in the Airbyte configuration file that passed via `--config-file`. \ +It should be defined under `catalog` and src and dst has the same schema. +You will have to know the stream name and sync mode you would like to run. ```json { - "": { "disabled": true }, - "": { - "sync_mode": "full_refresh", - "destination_sync_mode": "append" + "src": { + "image": ..., + "config": ..., + "catalog": { <-- define your catalog + "streams":[ + { + "stream":{"name":""}, <-- stream name + "sync_mode":"full_refresh" <-- sync mode: "full_refresh" or "incremental" + } + ] + } + }, + "dst": { + "image": ..., + "config": ..., + "catalog": { + ... + } } } ``` -You can also force full_refresh mode for all streams by setting the `--full-refresh` flag. +### Airbyte Configuration - Customize Docker Settings + +If you want to customize the Airbyte connectors docker settings, there're optional fields that you configure CPU, memory and log file size. This can be do so by specifying in the Airbyte configuration file. + +```json +{ + "src": { + "image": ..., + "dockerOptions": { + "maxCpus": 2, <-- unit: CPU (type: number) + "maxMemory": 256 , <-- unit: MB (type: number) + "maxLogSize": "10m" <-- unit: k/m/g (type: string) + } + }, + ... +} + +``` + +## FAQ + +- If you have customized your docker socket, please exports the docker socket in env var `DOCKER_HOST`. +- We only support reading Airbyte configuration file in encoding: `utf-8`, `utf-16le`, `utf-16be`. + +## Migration Guide + +As some users might come from our previous Airbyte CLI Bash version. \ +Here is some guide for you to upgrade to the new one. + +### Old CLI v.s. New CLI + +1. Update the CLI from Bash script to binary - open up to users that don't use Bash by default +1. Move Airbyte configuration into one JSON file - avoid syntax issues caused by running on different systems + +#### Example Usage + +```sh +# Older version +./airbyte-local.sh \ + --src 'farosai/airbyte-faros-graphql-source' \ + --src.api_url $FAROS_API_URL \ + --src.graph 'faros' \ + --src.result_model 'Flat' \ + --src.models_filter '["org_Team"]' \ + --dst 'farosai/airbyte-faros-destination' \ + --dst.edition_configs.graph 'default' \ + --dst.edition_configs.api_url $FAROS_API_URL + +# Newer version +./airbyte-local --config-file graph_copy.json +``` + +### New/Renamed Arguments + +| Old Argument | New Argument | Replacement/Notes | +| -------------------- | ------------------------ | ------------------------------------------ | +| | `--config-file` | New arugment to take Airbyte configuration | +| `--check-connection` | `--src-check-connection` | For naming consistency | +| `--state ` | `--state-file ` | For naming consistency | + +### Unsupported Arguments + +The following arguments are droppeed in the new CLI. Please update your command according. + +For arguments `--src ...` and `--dst ...`, they are still supported for user convenience. We strongly encourage users to use the new arugment `--config-file` to pass in Airbyte configuration in favor of the deprecated ones. Also, you can find a file named `faros_airbyte_cli_config.json` be automatically generated after running the CLI. It should covert your Airybte configuration to the new schema and next time you can just pass in this file with arugment `--config-file` and stop using the deprecated arguments! + +| Argument | Status | Replacement/Notes | +| -------------------------------- | ----------- | ------------------------------------------------------------------- | +| `--src ` | Deprecated | Image name is now defined in Aribyte configuration file | +| `--dst ` | Deprecated | Image name is now defined in Aribyte configuration file | +| `--src. ` | Deprecated | Airbyte config is now defined in Aribyte configuration file | +| `--dst. ` | Deprecated | Airbyte config is now defined in Aribyte configuration file | +| `--src-catalog-overrides ` | Unsupported | Airbyte catalog config is now defined in Aribyte configuration file | +| `--src-catalog-file ` | Unsupported | Airbyte catalog config is now defined in Aribyte configuration file | +| `--src-catalog-json ` | Unsupported | Airbyte catalog config is now defined in Aribyte configuration file | +| `--dst-catalog-file ` | Unsupported | Airbyte catalog config is now defined in Aribyte configuration file | +| `--dst-catalog-json ` | Unsupported | Airbyte catalog config is now defined in Aribyte configuration file | +| `--src-wizard` | Unsupported | Use `--generate-config` instead | +| `--dst-wizard` | Unsupported | Use `--generate-config` instead | +| `--max-log-size ` | Unsupported | Docker settings are now defined in Aribyte configuration file | +| `--max-mem ` | Unsupported | Docker settings are now defined in Aribyte configuration file | +| `--max-cpus ` | Unsupported | Docker settings are now defined in Aribyte configuration file | +| `--src-docker-options "` | Unsupported | Docker settings are now defined in Aribyte configuration file | +| `--dst-docker-options "` | Unsupported | Docker settings are now defined in Aribyte configuration file | +| `--k8s-deployment` | Unsupported | Stop surporting running on local kubernetes cluster | +| `--dst-stream-prefix ` | Unsupported | Use `--connection-name` instead | diff --git a/airbtye-local-cli-bash/README.md b/airbtye-local-cli-bash/README.md new file mode 100644 index 00000000..b905bf27 --- /dev/null +++ b/airbtye-local-cli-bash/README.md @@ -0,0 +1,167 @@ +# Airbyte Local CLI (Deprecated) [![CI](https://github.com/faros-ai/airbyte-local-cli/actions/workflows/ci.yaml/badge.svg)](https://github.com/faros-ai/airbyte-local-cli/actions/workflows/ci.yaml) [![Deprecated](https://img.shields.io/badge/status-deprecated-red)](../README.md) + +⚠️ **DEPRECATION NOTICE** ⚠️ +This project is deprecated and is no longer actively maintained. Please use the new [Airbyte Local CLI](../README.md) instead. + +## Migration Path + +This project has been replaced by the new Airbyte Local CLI (Node.js) project. +Please refer to [Migration Guide](../README.md#migration-guide) for installation and usage instructions. + +--- + +CLI for running Airbyte sources & destinations locally or on a Kubernetes cluster without an Airbyte server + +![Alt Text](https://github.com/Faros-ai/airbyte-local-cli/raw/main/resources/demo.gif) + +## Example Usage + +**Requirements**: `bash`, `jq`, `tee`. Additionally, `docker` when running syncs locally, or `kubectl` when running on a Kubernetes cluster. + +Either [download the script manually](https:/ /raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) or invoke the script directly with curl: + +```sh +bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) --help +``` + +For example here is how you can sync ServiceNow source with [Faros Cloud](https://www.faros.ai) destination: + +```sh +./airbyte-local.sh \ + --src 'farosai/airbyte-servicenow-source' \ + --src.username '' \ + --src.password '' \ + --src.url '' \ + --dst 'farosai/airbyte-faros-destination' \ + --dst.edition_configs.edition 'cloud' \ + --dst.edition_configs.api_url '' \ + --dst.edition_configs.api_key '' \ + --dst.edition_configs.graph 'default' \ + --state state.json \ + --check-connection +``` + +Or with [Faros Community Edition](https://github.com/faros-ai/faros-community-edition) as the destination: + +```sh +./airbyte-local.sh \ + --src 'farosai/airbyte-servicenow-source' \ + --src.username '' \ + --src.password '' \ + --src.url '' \ + --dst 'farosai/airbyte-faros-destination' \ + --dst.edition_configs.edition 'community' \ + --dst.edition_configs.hasura_admin_secret 'admin' \ + --dst.edition_configs.hasura_url 'http://host.docker.internal:8080/' \ + --state state.json \ + --check-connection +``` +**Note**: The `src.*` and `dst.*` arguments will differ depending on the source and destination being used. + +Or on a Kubernetes cluster: + +```sh +./airbyte-local.sh \ + --src 'farosai/airbyte-servicenow-source' \ + --src.username '' \ + --src.password '' \ + --src.url '' \ + --dst 'farosai/airbyte-faros-destination' \ + --dst.edition_configs.edition 'cloud' \ + --dst.edition_configs.api_url '' \ + --dst.edition_configs.api_key '' \ + --dst.edition_configs.graph 'default' \ + --state state.json \ + --k8s-deployment \ + --k8s-namespace default \ + --max-cpus 0.5 \ + --max-mem 500Mi \ + --keep-containers +``` +**Note**: The command assumes Kubernetes cluster context, and credentials are already configured. For more info, see [official docs](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/). + + +## Configuring Faros source/destination using a wizard + +**Note**: Faros Sources and/or Faros Destination only. Not supported with Kubernetes deployment. + +Instead of passing `src.*` and `dst.*`, it is possible to invoke a configuration wizard for the Faros source +and/or destination: + +``` +./airbyte-local.sh \ + --src 'farosai/airbyte-servicenow-source' \ + --src-wizard \ + --dst 'farosai/airbyte-faros-destination' \ + --dst-wizard +``` + +## Arguments + +| Argument | Required | Description | +| --------------------------------- | -------- | ------------------------------------------------------------------------------------------------- | +| --src \ | Yes | Airbyte source Docker image | +| --dst \ | Yes | Airbyte destination Docker image | +| --src.\ \ | | Append `"key": "value"` into the source config \* | +| --dst.\ \ | | Append `"key": "value"` into the destination config \* | +| --check-connection | | Validate the Airbyte source connection | +| --full-refresh | | Force source full_refresh and destination overwrite mode | +| --state \ | | Override state file path for incremental sync | +| --src-output-file \ | | Write source output as a file (handy for debugging) | +| --src-catalog-overrides \ | | JSON string of sync mode overrides. See [overriding default catalog](#overriding-default-catalog) | +| --src-config-file \ | | Source config file path | +| --src-config-json \ | | Source config as a JSON string | +| --src-catalog-file \ | | Source catalog file path | +| --src-catalog-json \ | | Source catalog as a JSON string | +| --dst-config-file \ | | Destination config file path | +| --dst-config-json \ | | Destination config as a JSON string | +| --dst-catalog-file \ | | Destination catalog file path | +| --dst-catalog-json \ | | Destination catalog as a JSON string | +| --dst-stream-prefix \ | | Destination stream prefix | +| --no-src-pull | | Skip pulling Airbyte source image | +| --no-dst-pull | | Skip pulling Airbyte destination image | +| --src-wizard | | Run the Airbyte source configuration wizard | +| --dst-wizard | | Run the Airbyte destination configuration wizard | +| --src-only | | Only run the Airbyte source | +| --dst-only \ | | Use a file for destination input instead of a source | +| --connection-name | | Connection name used in various places | +| --raw-messages | | Output raw Airbyte messages, i.e., without a log prefix or colors | +| --max-log-size \ | | Set Docker maximum log size | +| --max-mem \ | | Set the maximum amount of memory for Docker or Kubernetes container, e.g., `"1g"` or `"1024Mi"` | +| --max-cpus \ | | Set the maximum number of CPUs for each Docker or Kubernetes container, e.g, `"1"` or `"1000m"` | +| --src-docker-options "\" | | Set additional options to pass to the `docker run ` command, e.g `--src-docker-options "-e NODE_OPTIONS=--max_old_space_size=2000 -e NODE_TLS_REJECT_UNAUTHORIZED=0"` | +| --dst-docker-options "\" | | Set additional options to pass to the `docker run ` command, e.g `--dst-docker-options "-e NODE_OPTIONS=--max_old_space_size=2000"` | +| --k8s-deployment | | Deploy and run source/destination connectors as a pod on a Kubernetes cluster | +| --k8s-namespace \ | | Kubernetes namespace where the source/destination connectors pod is deployed to | +| --keep-containers | | Do not delete source and destination containers (or Kubernetes pod) after they exit | +| --debug | | Enable debug logging | + +**Note**: when passing an array value for a parameter specify it as a json array, for example: + +``` +--src.projects '["project-1","project-2","project-3"]' +``` + +## Overriding Default Catalog + +To generate the Airbyte catalog needed for running the source and destination +connectors, the script runs the `discover` command on the source to get the list +of all supported streams. It then creates an Airbyte configured catalog, +enabling all of the streams and using "incremental" sync mode for all the +streams that support it. Each stream's destination sync mode defaults to +"append" for incremental streams and "overwrite" for full_refresh streams. To +disable or customize the sync mode or destination sync mode on any of the +streams, pass a `--src-catalog-overrides` option whose value is a JSON string in +the following format: + +```json +{ + "": { "disabled": true }, + "": { + "sync_mode": "full_refresh", + "destination_sync_mode": "append" + } +} +``` + +You can also force full_refresh mode for all streams by setting the `--full-refresh` flag. diff --git a/airbyte-local.sh b/airbtye-local-cli-bash/airbyte-local.sh similarity index 100% rename from airbyte-local.sh rename to airbtye-local-cli-bash/airbyte-local.sh diff --git a/resources/demo.gif b/airbtye-local-cli-bash/resources/demo.gif similarity index 100% rename from resources/demo.gif rename to airbtye-local-cli-bash/resources/demo.gif diff --git a/test/.shellspec b/airbtye-local-cli-bash/test/.shellspec similarity index 100% rename from test/.shellspec rename to airbtye-local-cli-bash/test/.shellspec diff --git a/test/Dockerfile b/airbtye-local-cli-bash/test/Dockerfile similarity index 100% rename from test/Dockerfile rename to airbtye-local-cli-bash/test/Dockerfile diff --git a/test/spec/airbyte-local_spec.sh b/airbtye-local-cli-bash/test/spec/airbyte-local_spec.sh similarity index 100% rename from test/spec/airbyte-local_spec.sh rename to airbtye-local-cli-bash/test/spec/airbyte-local_spec.sh diff --git a/test/spec/spec_helper.sh b/airbtye-local-cli-bash/test/spec/spec_helper.sh similarity index 100% rename from test/spec/spec_helper.sh rename to airbtye-local-cli-bash/test/spec/spec_helper.sh diff --git a/airbyte-local-cli-nodejs/README.md b/airbyte-local-cli-nodejs/README.md deleted file mode 100644 index 2229c341..00000000 --- a/airbyte-local-cli-nodejs/README.md +++ /dev/null @@ -1,402 +0,0 @@ -# Airbyte Local CLI [![CI](https://github.com/faros-ai/airbyte-local-cli/actions/workflows/ci.yaml/badge.svg)](https://github.com/faros-ai/airbyte-local-cli/actions/workflows/ci.yaml) - -CLI for running Airbyte sources & destinations locally. - -## Table of Contents - -- [Getting Started](#getting-started) - - [Prerequisites](#prerequisites) - - [Step 1. Install](#step-1-install) - - [Step 2. Create an Airbyte Configuration File](#step-2-create-an-airbyte-configuration-file) - - [Step 2a. Auto Generate Configuration Templates (Experimental)](#step-2a-auto-generate-configuration-templates-experimental) - - [Step 2b. Craft your own configuration](#step-2b-craft-your-own-configuration) - - [Step 3. Run it!](#step-3-run-it) -- [Advanced Settings](#advanced-settings) - - [CLI Arguments](#cli-arguments) - - [Airbyte Configuration - Override Airbyte Catalog](#airbyte-configuration---override-airbyte-catalog) - - [Airbyte Configuration - Customize Docker Settings](#airbyte-configuration---customize-docker-settings) -- [FAQ](#faq) -- [Migration Guide](#migration-guide) - - [Old CLI v.s. New CLI](#old-cli-vs-new-cli) - - [New/Renamed Arguments](#newrenamed-arguments) - - [Unsupported Arguments](#unsupported-arguments) - -## Getting Started - -**Supported System** - -- Linux x86_64 -- MacOS arm64 (Apple chip) -- Windows x86_64 - -### Prerequisites - -- Docker -- Faros API key: check out the [instructions](https://docs.faros.ai/reference/getting-api-access). - -### Step 1. Install - -[All releases](https://github.com/faros-ai/airbyte-local-cli/releases.) - -#### Linux/MacOS - -Here is the steps of downloading the CLI on MacOS. Linux should have very similar steps. - -```sh -# Download from the CLI -# Please swap out `v0.0.5` to the version you want -wget -O airbyte-local.zip https://github.com/faros-ai/airbyte-local-cli/releases/download/v0.0.5/airbyte-local-macos-arm64.zip | unzip -o airbyte-local.zip - -# run `--help` or `--version` to check if the CLI is installed correctly -./airbyte-local # this shows the help manual -./airbyte-local --help -./airbyte-local --version -``` - -#### Windows (Powershell) - -```ps1 -# Download from the CLI -# Please swap out `v0.0.5` to the version you want -Invoke-WebRequest -Uri "https://github.com/faros-ai/airbyte-local-cli/releases/download/v0.0.5/airbyte-local-win-x64.zip" -OutFile "airbyte-local-win-x64.zip" -Expand-Archive -Path "airbyte-local-win-x64.zip" -DestinationPath . -Force - -# run `--help` or `--version` to check if the CLI is installed correctly -.\airbyte-local -.\airbyte-local --help -.\airbyte-local --version - -``` - -### Step 2. Create an Airbyte Configuration File - -First of all, you will have to know where you want to pull data from and where you want to push data to. -For example, pulling data from Github and pushing data to Faros AI. - -Then, there're two options you can go to create the config file. - -a. Auto Generate Configuration Templates: -This is recommended for first-time users. It helps you to start with a template to update the Airbyte configs. (Go to Step 2a) - -b. Craft your own configuration: This is for users that are familiar with Airbyte configurations and are looking for a finer tune on the configs. (Go to Step 2b) - -#### Step 2a. Auto Generate Configuration Templates (Experimental) - -You can utilize the `generate-config` subcommand to bootstrap your Airbyte config. -It's required to provide the Airbyte source, which means you will have to know which source data you are pulling from, e.g. Github, Jira, etc. For the Airbtye destination, it is set to Faros by default, i.e. pushing the data to Faros. -For users convenience, the CLI will attempt to guess the source and destination from your inputs. It handles some typos and is case insensitive. -If the CLI cannot correctly guess the source/destination, or you want to use your own images that are not managed by Faros, you can use the option `--image` to specify your images. -The CLI will use your exact input image(s) to generate Airbyte configurations. -The default destination image is set to Faros, i.e. `farosai/airbyte-faros-destination`. - -By running this subcommand, it prints out both Airbyte source and deestination configuration tables in the terminal for your reference. -And it generates a template config file `faros_airbyte_cli_config.json` in the current directory. The template config file only includes requried configs. If you need additional configs, please refer to the configuration tables and update the config file. - -Run the command to generate the template - -```sh -# ./airbyte-local generate-config [destination] - -# Ex: Pull data from Github and push to Faros -./airbyte-local generate-config github -./airbyte-local generate-config github faros - -# Ex: Pull data from Jira and push to Faros -./airbyte-local generate-config jira - -# Suppress printing out the configuration tables -./airbyte-local generate-config -s jira - -# Use your own custom images -./airbyte-local generate-config --image farosai/airbyte-github-custom-source -./airbyte-local generate-config -s --image farosai/airbyte-github-custom-source -./airbyte-local generate-config --image farosai/airbyte-github-custom-source farosai/airbyte-custom-destination -``` - -Note: Both source and destination inputs are case insensitive and tolerate a bit of typos. - -After running the command, you should see `🔹 **Next Steps:**` instructions showing up in your terminal. Follow the steps to complete the config. - -#### Step 2b. Craft your own configuration - -The CLI uses arguement `--config-file` (`-c` for short) to take the airybte configuration in a JSON file format. - -JSON Schema - -```json -{ - "src": { - "image": "", - "config": {...YOUR_SOURCE_CONFIG...} - }, - "dst": { - "image": "", - "config": {...YOUR_DESTINATION_CONFIG...} - } -} -``` - -You can find all the available soure and destiantion images that are supported by Faros [here](https://hub.docker.com/u/farosai). \ -Here're some popular source image - -- Github source: `farosai/airbyte-github-source` -- Jira source: `farosai/airbyte-jira-source` -- Faros destination: `farosai/airbyte-faros-destination` - -#### Example of Airbyte Config File - -Assuming you want to pull data from Github org `my-org` by using GitHub PAT and push data to Faros `default` workspace. This is what the JSON would look like - -```json -{ - "src": { - "image": "farosai/airbyte-github-source", - "config": { - "api_key": "", <-- Faros API key - "graph": "default" <-- Faros workspace - "authentication": { - "type": "token", - "personal_access_token": "" - }, - "organizations": ["my-org"], - } - }, - "dst": { - "image": "farosai/airbyte-faros-destination", - "config": { - "edition_configs": { - "api_key": "", <-- Faros API key - "graph": "default" <-- Faros workspace - } - } - } -} -``` - -Save it as `faros_airbyte_cli_config.json`. \ -In most cases, you always have to provide Faros API key and workspace under `src.config` and `dst.config.edition_configs`. - -More resources you can find it in [Faros Documantation](https://docs.faros.ai/), e.g. instructions to create GitHub PAT and what permission you need for the PAT, etc. - -### Step 3. Run it! - -#### Linux/MacOS - -```sh -./airbyte-local --config-file 'faros_airbyte_cli_config.json' -./airbyte-local -c 'faros_airbyte_cli_config.json' -``` - -#### Windows (Powershell) - -```ps1 -.\airbyte-local --config-file 'faros_airbyte_cli_config.json' -.\airbyte-local -c 'faros_airbyte_cli_config.json' -``` - -The logs should indicate the process of the data sync. \ -You can also see the progress and logs in Faros App > Switch to the workspace you are running against > Admin Settings > Data Control > Sources. - -## Advanced Settings - -In some cases, you might want to customized the data sync more. \ -We provide some more CLI optional arguments and optional fields in the Airbyte configuration file. - -### CLI Arguments - -| Option | Required | Description | -| -------------------------- | -------- | ---------------------------------------------------------------------------------------------------------- | -| `-c, --config-file ` | Yes | Airbyte source and destination connector config JSON file path | -| `-h, --help` | | Display usage information | -| `-v, --version` | | Output the current version | -| `--full-refresh` | | Force full_refresh and overwrite mode. This overrides the mode in provided config file | -| `--state-file ` | | Override state file path for incremental sync | -| `--no-src-pull` | | Skip pulling Airbyte source image | -| `--no-dst-pull` | | Skip pulling Airbyte destination image | -| `--src-only` | | Only run the Airbyte source and write output in stdout. Use '--src-output-file' instead to write to a file | -| `--src-output-file ` | | Write source output as a file (requires a destination) | -| `--src-check-connection` | | Validate the Airbyte source connection | -| `--dst-only ` | | Use a file for destination input instead of a source | -| `--dst-use-host-network` | | Use the host network when running the Airbyte destination | -| `--log-level ` | | Set level of source and destination loggers (default: "info") | -| `--raw-messages` | | Output raw Airbyte messages | -| `--connection-name ` | | Connection name used in various places | -| `--keep-containers` | | Do not remove source and destination containers after they exit | -| `--debug` | | Enable debug logging | -| `--src ` | | [Deprecated] Airbyte source Docker image | -| `--dst ` | | [Deprecated] Airbyte destination Docker image | -| `--src. ` | | [Deprecated] Add "key": "value" into the source config | -| `--dst. ` | | [Deprecated] Add "key": "value" into the destination config | - -#### Example Usage - -##### Linux/MacOS - -```sh -# Turn on debug logs -./airbyte-local --config-file 'sample.json' --debug - -# Run source sync only -./airbyte-local \ - --config-file 'sample.json' \ - --src-only - -# Check source connection -./airbyte-local \ - --config-file 'sample.json' \ - --src-check-connection - -# Enforce full refreash -./airbyte-local \ - --config-file 'sample.json' \ - --full-refresh - -# Use customized connection name -./airbyte-local \ - --config-file 'sample.json' \ - --connection-name 'test-connection' -``` - -##### Windows (Powershell) - -```ps1 -# Turn on debug logs -./airbyte-local --config-file 'sample.json' --debug - -# Run source sync only -./airbyte-local ` - --config-file 'sample.json' ` - --src-only - -# Check source connection -./airbyte-local ` - --config-file 'sample.json' ` - --src-check-connection - -# Enforce full refreash -./airbyte-local ` - --config-file 'sample.json' ` - --full-refresh - -# Use customized connection name -./airbyte-local ` - --config-file 'sample.json' ` - --connection-name 'test-connection' -``` - -### Airbyte Configuration - Override Airbyte Catalog - -You can override the default Airbyte catalog in the Airbyte configuration file that passed via `--config-file`. \ -It should be defined under `catalog` and src and dst has the same schema. -You will have to know the stream name and sync mode you would like to run. - -```json -{ - "src": { - "image": ..., - "config": ..., - "catalog": { <-- define your catalog - "streams":[ - { - "stream":{"name":""}, <-- stream name - "sync_mode":"full_refresh" <-- sync mode: "full_refresh" or "incremental" - } - ] - } - }, - "dst": { - "image": ..., - "config": ..., - "catalog": { - ... - } - } -} -``` - -### Airbyte Configuration - Customize Docker Settings - -If you want to customize the Airbyte connectors docker settings, there're optional fields that you configure CPU, memory and log file size. This can be do so by specifying in the Airbyte configuration file. - -```json -{ - "src": { - "image": ..., - "dockerOptions": { - "maxCpus": 2, <-- unit: CPU (type: number) - "maxMemory": 256 , <-- unit: MB (type: number) - "maxLogSize": "10m" <-- unit: k/m/g (type: string) - } - }, - ... -} - -``` - -## FAQ - -- If you have customized your docker socket, please exports the docker socket in env var `DOCKER_HOST`. -- We only support reading Airbyte configuration file in encoding: `utf-8`, `utf-16le`, `utf-16be`. - -## Migration Guide - -As some users might come from our previous Airbyte CLI Bash version. \ -Here is some guide for you to upgrade to the new one. - -### Old CLI v.s. New CLI - -1. Update the CLI from Bash script to binary - open up to users that don't use Bash by default -1. Move Airbyte configuration into one JSON file - avoid syntax issues caused by running on different systems - -#### Example Usage - -```sh -# Older version -./airbyte-local.sh \ - --src 'farosai/airbyte-faros-graphql-source' \ - --src.api_url $FAROS_API_URL \ - --src.graph 'faros' \ - --src.result_model 'Flat' \ - --src.models_filter '["org_Team"]' \ - --dst 'farosai/airbyte-faros-destination' \ - --dst.edition_configs.graph 'default' \ - --dst.edition_configs.api_url $FAROS_API_URL - -# Newer version -./airbyte-local --config-file graph_copy.json -``` - -### New/Renamed Arguments - -| Old Argument | New Argument | Replacement/Notes | -| -------------------- | ------------------------ | ------------------------------------------ | -| | `--config-file` | New arugment to take Airbyte configuration | -| `--check-connection` | `--src-check-connection` | For naming consistency | -| `--state ` | `--state-file ` | For naming consistency | - -### Unsupported Arguments - -The following arguments are droppeed in the new CLI. Please update your command according. - -For arguments `--src ...` and `--dst ...`, they are still supported for user convenience. We strongly encourage users to use the new arugment `--config-file` to pass in Airbyte configuration in favor of the deprecated ones. Also, you can find a file named `faros_airbyte_cli_config.json` be automatically generated after running the CLI. It should covert your Airybte configuration to the new schema and next time you can just pass in this file with arugment `--config-file` and stop using the deprecated arguments! - -| Argument | Status | Replacement/Notes | -| -------------------------------- | ----------- | ------------------------------------------------------------------- | -| `--src ` | Deprecated | Image name is now defined in Aribyte configuration file | -| `--dst ` | Deprecated | Image name is now defined in Aribyte configuration file | -| `--src. ` | Deprecated | Airbyte config is now defined in Aribyte configuration file | -| `--dst. ` | Deprecated | Airbyte config is now defined in Aribyte configuration file | -| `--src-catalog-overrides ` | Unsupported | Airbyte catalog config is now defined in Aribyte configuration file | -| `--src-catalog-file ` | Unsupported | Airbyte catalog config is now defined in Aribyte configuration file | -| `--src-catalog-json ` | Unsupported | Airbyte catalog config is now defined in Aribyte configuration file | -| `--dst-catalog-file ` | Unsupported | Airbyte catalog config is now defined in Aribyte configuration file | -| `--dst-catalog-json ` | Unsupported | Airbyte catalog config is now defined in Aribyte configuration file | -| `--src-wizard` | Unsupported | Use `--generate-config` instead | -| `--dst-wizard` | Unsupported | Use `--generate-config` instead | -| `--max-log-size ` | Unsupported | Docker settings are now defined in Aribyte configuration file | -| `--max-mem ` | Unsupported | Docker settings are now defined in Aribyte configuration file | -| `--max-cpus ` | Unsupported | Docker settings are now defined in Aribyte configuration file | -| `--src-docker-options "` | Unsupported | Docker settings are now defined in Aribyte configuration file | -| `--dst-docker-options "` | Unsupported | Docker settings are now defined in Aribyte configuration file | -| `--k8s-deployment` | Unsupported | Stop surporting running on local kubernetes cluster | -| `--dst-stream-prefix ` | Unsupported | Use `--connection-name` instead | diff --git a/airbyte-local-cli-nodejs/README.md b/airbyte-local-cli-nodejs/README.md new file mode 120000 index 00000000..32d46ee8 --- /dev/null +++ b/airbyte-local-cli-nodejs/README.md @@ -0,0 +1 @@ +../README.md \ No newline at end of file From 6535e534274f1860ffe6d30f23629a40ee598f46 Mon Sep 17 00:00:00 2001 From: Jennie Gao Date: Wed, 16 Apr 2025 14:00:15 -0700 Subject: [PATCH 2/8] update link --- README.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 70ac08a2..f5d51348 100644 --- a/README.md +++ b/README.md @@ -2,8 +2,8 @@ CLI for running Airbyte sources & destinations locally. -> **Note:** The previous Bash-based version of this CLI has been deprecated. -> You can find its documentation [here](./airbyte-local-cli-bash/README.md). +> 💡 **IMPORTANT: The previous Bash-based version of this CLI has been deprecated.** +> You can find its documentation [here](https://github.com/faros-ai/airbyte-local-cli/blob/main/airbyte-local-cli-bash/README.md). > For migration details, see the [Migration Guide](#migration-guide). ## Table of Contents @@ -40,7 +40,7 @@ CLI for running Airbyte sources & destinations locally. ### Step 1. Install -[All releases](https://github.com/faros-ai/airbyte-local-cli/releases.) +[All releases](https://github.com/faros-ai/airbyte-local-cli/releases) #### Linux/MacOS @@ -79,10 +79,10 @@ For example, pulling data from Github and pushing data to Faros AI. Then, there're two options you can go to create the config file. -a. Auto Generate Configuration Templates: -This is recommended for first-time users. It helps you to start with a template to update the Airbyte configs. (Go to Step 2a) +* a. Auto Generate Configuration Templates: +This is recommended for first-time users. It helps you to start with a template to update the Airbyte configs. (Go to [Step 2a](#step-2a-auto-generate-configuration-templates-experimental)) -b. Craft your own configuration: This is for users that are familiar with Airbyte configurations and are looking for a finer tune on the configs. (Go to Step 2b) +* b. Craft your own configuration: This is for users that are familiar with Airbyte configurations and are looking for a finer tune on the configs. (Go to [Step 2b](#step-2b-craft-your-own-configuration)) #### Step 2a. Auto Generate Configuration Templates (Experimental) @@ -395,8 +395,8 @@ For arguments `--src ...` and `--dst ...`, they are still supported for user con | `--src-catalog-json ` | Unsupported | Airbyte catalog config is now defined in Aribyte configuration file | | `--dst-catalog-file ` | Unsupported | Airbyte catalog config is now defined in Aribyte configuration file | | `--dst-catalog-json ` | Unsupported | Airbyte catalog config is now defined in Aribyte configuration file | -| `--src-wizard` | Unsupported | Use `--generate-config` instead | -| `--dst-wizard` | Unsupported | Use `--generate-config` instead | +| `--src-wizard` | Unsupported | Use `generate-config` instead | +| `--dst-wizard` | Unsupported | Use `generate-config` instead | | `--max-log-size ` | Unsupported | Docker settings are now defined in Aribyte configuration file | | `--max-mem ` | Unsupported | Docker settings are now defined in Aribyte configuration file | | `--max-cpus ` | Unsupported | Docker settings are now defined in Aribyte configuration file | From 5be8ea5a26d9d42d02f2843e38f2cf58c803ee31 Mon Sep 17 00:00:00 2001 From: Jennie Gao Date: Wed, 16 Apr 2025 14:32:29 -0700 Subject: [PATCH 3/8] update bash cli test in ci --- .github/workflows/ci.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index 6f3d4ef8..9c3504f0 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -49,6 +49,7 @@ jobs: # Run shellspec tests againgst airbyte-local.sh - name: Run ShellSpec tests + working-directory: airbyete-local-cli-bash run: | docker run --rm \ -v "$PWD/test:/src" \ From 8ce53799620b9a84cc5f7a117a82034976fb7175 Mon Sep 17 00:00:00 2001 From: Jennie Gao Date: Wed, 16 Apr 2025 14:39:18 -0700 Subject: [PATCH 4/8] fix working dir --- .github/workflows/ci.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index 9c3504f0..3f0d96c4 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -43,7 +43,7 @@ jobs: run: npm run test -- --coverage --color - name: Build Shellspec image - working-directory: test + working-directory: airbyete-local-cli-bash/test run: | docker build . -t 'farosai/shellspec:kcov' From c090fadb83631c8989935c5d885b83bca20e9c04 Mon Sep 17 00:00:00 2001 From: Jennie Gao Date: Wed, 16 Apr 2025 14:44:10 -0700 Subject: [PATCH 5/8] fix typo --- .github/workflows/ci.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index 3f0d96c4..3f9498cb 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -43,13 +43,13 @@ jobs: run: npm run test -- --coverage --color - name: Build Shellspec image - working-directory: airbyete-local-cli-bash/test + working-directory: airbtye-local-cli-bash/test run: | docker build . -t 'farosai/shellspec:kcov' # Run shellspec tests againgst airbyte-local.sh - name: Run ShellSpec tests - working-directory: airbyete-local-cli-bash + working-directory: airbtye-local-cli-bash run: | docker run --rm \ -v "$PWD/test:/src" \ From 58b34eef97cef72ef9e89fa70ca107a8d089b032 Mon Sep 17 00:00:00 2001 From: Jennie Gao Date: Wed, 16 Apr 2025 14:51:20 -0700 Subject: [PATCH 6/8] update sonar --- sonar-project.properties | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sonar-project.properties b/sonar-project.properties index eab32ca4..6928de95 100644 --- a/sonar-project.properties +++ b/sonar-project.properties @@ -1,5 +1,5 @@ sonar.projectKey=faros-ai_airbyte-local-cli sonar.organization=faros-ai -sonar.sources=airbyte-local.sh,airbyte-local-cli-nodejs/src -sonar.tests=test,airbyte-local-cli-nodejs/test +sonar.sources=airbyte-local-cli-bash/airbyte-local.sh,airbyte-local-cli-nodejs/src +sonar.tests=airbyte-local-cli-bash/test,airbyte-local-cli-nodejs/test sonar.javascript.lcov.reportPaths=airbyte-local-cli-nodejs/out/coverage/lcov.info From 0a641ea5ebb87b50ccef4db3083d71dfcf2c6ac9 Mon Sep 17 00:00:00 2001 From: Jennie Gao Date: Wed, 16 Apr 2025 14:58:17 -0700 Subject: [PATCH 7/8] fix typo :(( --- .github/workflows/ci.yaml | 4 ++-- .../README.md | 0 .../airbyte-local.sh | 0 .../resources/demo.gif | Bin .../test/.shellspec | 0 .../test/Dockerfile | 0 .../test/spec/airbyte-local_spec.sh | 0 .../test/spec/spec_helper.sh | 0 8 files changed, 2 insertions(+), 2 deletions(-) rename {airbtye-local-cli-bash => airbyte-local-cli-bash}/README.md (100%) rename {airbtye-local-cli-bash => airbyte-local-cli-bash}/airbyte-local.sh (100%) rename {airbtye-local-cli-bash => airbyte-local-cli-bash}/resources/demo.gif (100%) rename {airbtye-local-cli-bash => airbyte-local-cli-bash}/test/.shellspec (100%) rename {airbtye-local-cli-bash => airbyte-local-cli-bash}/test/Dockerfile (100%) rename {airbtye-local-cli-bash => airbyte-local-cli-bash}/test/spec/airbyte-local_spec.sh (100%) rename {airbtye-local-cli-bash => airbyte-local-cli-bash}/test/spec/spec_helper.sh (100%) diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index 3f9498cb..c05263ad 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -43,13 +43,13 @@ jobs: run: npm run test -- --coverage --color - name: Build Shellspec image - working-directory: airbtye-local-cli-bash/test + working-directory: airbyte-local-cli-bash/test run: | docker build . -t 'farosai/shellspec:kcov' # Run shellspec tests againgst airbyte-local.sh - name: Run ShellSpec tests - working-directory: airbtye-local-cli-bash + working-directory: airbyte-local-cli-bash run: | docker run --rm \ -v "$PWD/test:/src" \ diff --git a/airbtye-local-cli-bash/README.md b/airbyte-local-cli-bash/README.md similarity index 100% rename from airbtye-local-cli-bash/README.md rename to airbyte-local-cli-bash/README.md diff --git a/airbtye-local-cli-bash/airbyte-local.sh b/airbyte-local-cli-bash/airbyte-local.sh similarity index 100% rename from airbtye-local-cli-bash/airbyte-local.sh rename to airbyte-local-cli-bash/airbyte-local.sh diff --git a/airbtye-local-cli-bash/resources/demo.gif b/airbyte-local-cli-bash/resources/demo.gif similarity index 100% rename from airbtye-local-cli-bash/resources/demo.gif rename to airbyte-local-cli-bash/resources/demo.gif diff --git a/airbtye-local-cli-bash/test/.shellspec b/airbyte-local-cli-bash/test/.shellspec similarity index 100% rename from airbtye-local-cli-bash/test/.shellspec rename to airbyte-local-cli-bash/test/.shellspec diff --git a/airbtye-local-cli-bash/test/Dockerfile b/airbyte-local-cli-bash/test/Dockerfile similarity index 100% rename from airbtye-local-cli-bash/test/Dockerfile rename to airbyte-local-cli-bash/test/Dockerfile diff --git a/airbtye-local-cli-bash/test/spec/airbyte-local_spec.sh b/airbyte-local-cli-bash/test/spec/airbyte-local_spec.sh similarity index 100% rename from airbtye-local-cli-bash/test/spec/airbyte-local_spec.sh rename to airbyte-local-cli-bash/test/spec/airbyte-local_spec.sh diff --git a/airbtye-local-cli-bash/test/spec/spec_helper.sh b/airbyte-local-cli-bash/test/spec/spec_helper.sh similarity index 100% rename from airbtye-local-cli-bash/test/spec/spec_helper.sh rename to airbyte-local-cli-bash/test/spec/spec_helper.sh From 70acfff001b207c89ef04d372ddb19ac6e75c18e Mon Sep 17 00:00:00 2001 From: Jennie Gao Date: Fri, 18 Apr 2025 01:07:46 +0800 Subject: [PATCH 8/8] Update airbyte-local-cli-bash/README.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- airbyte-local-cli-bash/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/airbyte-local-cli-bash/README.md b/airbyte-local-cli-bash/README.md index b905bf27..d11291cc 100644 --- a/airbyte-local-cli-bash/README.md +++ b/airbyte-local-cli-bash/README.md @@ -18,7 +18,7 @@ CLI for running Airbyte sources & destinations locally or on a Kubernetes cluste **Requirements**: `bash`, `jq`, `tee`. Additionally, `docker` when running syncs locally, or `kubectl` when running on a Kubernetes cluster. -Either [download the script manually](https:/ /raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) or invoke the script directly with curl: +Either [download the script manually](https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) or invoke the script directly with curl: ```sh bash <(curl -s https://raw.githubusercontent.com/faros-ai/airbyte-local-cli/main/airbyte-local.sh) --help