feat: auto-stack compatible with upgraded node #907
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit updates various files to replace references from TSN to TN, reflecting the new naming convention. Changes include modifications in the CDK setup, README.md, and benchmark documentation. Additionally, new constructs for TN have been introduced, and existing configurations have been refined for clarity and consistency. The changes aim to ensure a cohesive transition to the TN naming scheme across the codebase.
…figuration This commit modifies the CDK context file to update hosted zone configurations, reflecting changes in domain names. Additionally, it adjusts asset paths in various Go files to ensure correct directory structures for building and deploying assets. The changes aim to enhance clarity and maintainability in the deployment setup.
This commit updates the CDK context file by removing the entry for the hosted zone associated with the domain `tsn.test.truflation.com`. The change aims to streamline the configuration and reflect the current deployment setup accurately.
…Taskfile.yml This commit introduces the `DB_OWNER` environment variable in the `compose` and `compose:down` tasks for improved configuration consistency. Additionally, it adds a new task `action-migrate-dev` to run SQL migrations in the development environment, enhancing the deployment workflow.
This commit introduces a new fronting mechanism for API routing and TLS termination using AWS API Gateway. It includes the implementation of the `ApiGatewayFronting` construct, which provisions an HTTP API and integrates it with the existing KGW and Indexer services. Additionally, it adds support for regional ACM certificates, enhancing the deployment flexibility. The README.md is updated to reflect the new fronting options and usage instructions, ensuring clarity for developers on how to configure and deploy the new fronting setup.
This commit adds a new `FrontingSelector` construct that dynamically determines the desired fronting type (API, CloudFront, or ALB) based on a CloudFormation parameter. It enhances the existing deployment logic in `cdk_main.go` by replacing hardcoded fronting type checks with a more flexible approach. Additionally, it updates the `AttachRoutes` methods in the ALB and CloudFront constructs to return a `FrontingResult`, indicating unimplemented features. New tests for the `ParseKind` function ensure robust validation of fronting types.
This commit refactors the `cdk_main.go` file to create the `CertStack` unconditionally, allowing stacks to utilize it based on their internal logic. It also updates the `FrontingSelector` to directly return the fronting kind without additional validation steps. Furthermore, it introduces a new `kwild-config.tmpl` file for configuration management and adds tests for the rendering of this template. The `go.mod` and `go.sum` files are updated to include new dependencies and versions, ensuring compatibility and improved functionality across the codebase.
… certificate management This commit introduces several enhancements across the fronting constructs. It updates the `fronting_selector.go` to utilize `ParseKind` for validating fronting types, ensuring robust error handling. The `domain_config.go` file adds a new method for generating fully-qualified subdomains. The `default_provider.go` now supports additional Subject Alternative Names (SANs) for certificates, improving flexibility in domain management. Furthermore, the `cert_manager.go` introduces a shared certificate management utility, allowing multiple services to share a single certificate efficiently. The `api_gateway.go` and `kwil_cluster.go` files are updated to reflect these changes, ensuring seamless integration of the new features. Additionally, the `ingress_rules` logic is centralized in a new utility, enhancing maintainability and clarity in security group configurations.
… S3 assets This commit adds the `goasset` package, which provides a standardized way to bundle Go applications and test binaries into AWS CDK S3 assets using local Go build commands. It includes the `Bundle` and `BundleDir` functions for flexible asset creation, along with comprehensive validation and logging features. Additionally, a test suite is introduced to ensure the functionality of the bundling process, covering various scenarios including successful builds, syntax error handling, and validation of input options. The README.md is also added to document usage and configuration options for developers.
This commit refines the startup scripts for the Kwil Gateway and Indexer by implementing error handling for Docker installation and configuration scripts. It introduces a new `WrapAsFile` utility to encapsulate script content into executable files, improving maintainability. Additionally, it integrates template rendering for various scripts, ensuring a more structured and readable approach to generating shell commands. The `go.mod` and `go.sum` files are updated to include new dependencies for template rendering, enhancing the overall functionality and robustness of the deployment scripts.
This commit introduces a new approach for managing the fronting type in the CDK setup. It adds the "frontingType" parameter to the CDK context, allowing for dynamic selection of fronting types (defaulting to "api"). The `GetFrontingKind` function is implemented to read and validate this parameter, enhancing error handling. Additionally, the `TnAutoStack` and `TnFromConfigStack` files are updated to utilize the new context helper for retrieving the selected fronting kind, streamlining the stack configuration process.
…eneration This commit introduces the `DbOwner` field to the `AutoStackEnvironmentVariables` and `GenerateGenesisFileInput` structures, ensuring that the database owner can be specified during genesis file generation. It updates the `GenerateGenesisFile` function to include the `DbOwner` parameter in the command execution for the kwild CLI. Additionally, a new test file is added to validate the genesis file generation process, along with a corresponding golden file for snapshot testing. The `KwilNetworkConfigAssetsFromNumberOfNodes` function is also updated to handle the new `DbOwner` and `GenesisFilePath` inputs, improving the overall configuration flexibility in the network setup.
This commit introduces the `SessionSecret` field to the `MainEnvironmentVariables` structure, improving the management of environment variables. It also refactors the `NewNode` function to accept a `NewNodeInput` struct, consolidating parameters for better readability and maintainability. Additionally, the `BuildTNAssets` function is updated to adjust the directory structure for Docker image assets, ensuring correct asset packaging. The changes enhance the overall configuration flexibility and clarity in the deployment process.
This commit updates the deployment configuration to read the `stage` and `devPrefix` values from the CDK context instead of CloudFormation parameters. It introduces new functions `GetStage` and `GetDevPrefix` for retrieving these values, enhancing type safety and validation. The changes streamline the deployment process and improve the overall clarity of the configuration management across various stacks and constructs.
This commit refactors the `KwilAssetOptions` structure to improve clarity by renaming fields for better understanding. It updates the `BuildKwilAssets` function to utilize the new field names. Additionally, it enhances the `NodeKeys` struct to reflect changes in key generation, including renaming and removing unnecessary fields. The integration tests for node key generation are introduced to validate the updated structure and functionality. Furthermore, various test files are updated to align with the new field names, ensuring consistency across the codebase.
This commit introduces automatic provisioning of EC2 instances for the Kwil Gateway and Indexer within the `TnAutoStack`, improving deployment efficiency. It also updates the `README.md` to clarify the deployment process, specifying the automatic provisioning behavior for different stacks. Additionally, a new context variable `stackSuffix` is added to enhance stack naming flexibility, and the `indexer_startup_scripts.go` file is refactored to remove unnecessary environment variable references, streamlining the configuration. These changes enhance clarity and maintainability across the deployment infrastructure.
This commit updates the `GetSharedCertProps` function in `cert_manager.go` to accept fully qualified domain names (FQDNs) instead of simple record names, improving clarity and error handling. It adds validation to ensure the input FQDNs end with the expected zone suffix. Additionally, it modifies the Docker configuration in `tn_startup_scripts.go` to correct the data root path and refines the template rendering in `configure_docker.sh.tmpl` for better formatting. The `validator_set.go` file is updated to grant pull access to the TN Docker image repository, enhancing asset management. These changes improve the overall robustness and maintainability of the deployment infrastructure.
…ndering This commit enhances the `AutoStackEnvironmentVariables` structure by adding comments to clarify the purpose of the `IncludeObserver` and `DbOwner` fields. Additionally, it modifies the `tn_db_startup.sh.tmpl` and `TestGoldenTnScript.golden` files to prevent quoting of environment variable values, ensuring compatibility with Docker Compose. These changes improve the clarity and functionality of the deployment scripts and environment variable management.
…ialization This commit modifies the `kwild-config.tmpl` file to streamline the configuration structure by removing redundant sections and adding a new P2P and consensus configuration block. It also introduces a temporary directory for initialization files in the `tn_instance.go` file, ensuring that configuration and genesis files are handled more effectively during the EC2 instance setup. These changes improve the clarity and organization of the deployment configuration, enhancing overall maintainability.
This commit introduces a new structure for managing node keys, allowing for the inclusion of private key hex and key type in the `NewNodeInput` struct. It updates the `newNode` function to generate a `nodekey.json` file containing these details, ensuring secure handling of node keys during EC2 instance initialization. Additionally, the genesis file generation process is improved by modifying the validator structure to include the key type, enhancing the overall configuration and security of the network setup. These changes streamline the deployment process and improve maintainability.
This commit modifies the `gateway-compose.yaml` file to change the build context and Dockerfile path for the gateway service, enhancing the deployment structure. It also adds a new volume binding for the `KGW_BINARY_PATH`, ensuring that the binary is correctly mapped to the container. These changes improve the clarity and maintainability of the Docker configuration.
… and indexer services This commit modifies the `dev-gateway-compose.yaml` and `dev-indexer-compose.yaml` files to change the build context and Dockerfile paths, enhancing the deployment structure. It also adds volume bindings for `KGW_BINARY_PATH` and `INDEXER_BINARY_PATH`, ensuring that the binaries are correctly mapped to the containers. Additionally, the `gateway.dockerfile` and `indexer.dockerfile` files are updated to remove unnecessary build stages, streamlining the Docker image creation process. These changes improve the clarity and maintainability of the deployment configuration.
<
10000
div data-view-component="true" class="TimelineItem-badge">
…ml instead of nodekey.json This commit modifies the configuration script in the Dockerfile to check for the existence of `config.toml` instead of `nodekey.json`. This change ensures that the correct configuration file is referenced during the initialization process, improving the accuracy of the deployment setup.
This commit modifies the `KGWInstance` and `IndexerInstance` structs to replace the `InstanceDnsName` field with `GatewayFqdn` and `IndexerFqdn`, respectively. It updates the A record creation logic to reflect these changes, ensuring that fully qualified domain names are used for the gateway and indexer instances. Additionally, the endpoint settings in the `tn_auto_stack.go` and `tn_from_config_stack.go` files are updated to utilize the new FQDN fields, enhancing clarity and maintainability in the deployment configuration.
This commit modifies the `dev-gateway-compose.yaml` file to change the source path for the `kgw-config.pkl` volume binding and updates the build context and Dockerfile path. These changes enhance the deployment structure and ensure that the correct configuration file is referenced, improving the clarity and maintainability of the Docker setup.
…y group This commit modifies the API Gateway integration in `api_gateway.go` to include parameter mapping for the request path, improving the routing capabilities. Additionally, it updates the security group name and description in `kgw_instance.go` to better reflect its purpose as the KGW instance security group. These changes enhance the clarity and maintainability of the deployment configuration.
…se configuration This commit updates the `.dockerignore` and `.gitignore` files to include additional patterns for ignoring environment files and IDE configurations. It also modifies the `compose.yaml` file to change the default value for the `DB_OWNER` environment variable and updates the port mapping for the TN RPC service, enhancing the clarity and maintainability of the Docker setup. Additionally, the `Taskfile.yml` is updated to include a new `PROVIDER` environment variable for migration tasks.
This commit updates the README.md file to correct references from TSN-DB to TN-DB, ensuring consistency in documentation. Additionally, it modifies the migrate.sh script to include synchronization and private key parameters when executing SQL files, enhancing the migration process's security and functionality.
This commit removes the TODO comments regarding field adjustments and the inclusion of the Stage field in the `ValidatorSetProps` struct within `validator_set.go`. This change enhances code clarity by eliminating unnecessary placeholders.
This commit removes unnecessary comment sections from the `GenerateGenesisFile` function in `generate_genesis.go`, enhancing code clarity by eliminating redundant annotations. The changes streamline the function's readability without altering its functionality.
Time Submission Status
|
MicBun
approved these changes
May 2, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary of Infrastructure Refactoring
I apologize for the size of this PR, but some of those changes were hard to test individually, but with this I could get to a working state
Many of these files being changed are about renames: tsn -> tn which was all over infrastructure
This PR introduces a major refactoring of the infrastructure deployment (
deployments/infra), focusing on abstraction, improved configuration management, and updated tooling alignment.Key Changes:
ValidatorSet,KwilCluster,Fronting,GoAsset,Renderer) for better maintainability. The heavy lifting done here should simplify future development, particularly for theTN-From-Config-Stack.stage,devPrefix,frontingType) to be more transparent to manage.scripts/renderer) for EC2 user-data andkwildconfig files, replacing hardcoded scripts/logic, addressing maintainability pain points.kwil-networkutilities to generate peer info and a single genesis asset, aligning with templated node configuration.Frontingconstruct).GoAssetconstruct bundles Go application binaries.kwild key gen) and genesis creation (kwild setup init) with kwild v0.10.2, includingdb_ownerandleaderfields.lib/observer) using templates and integrating with new constructs.Taskfile.yml,compose.yaml, and documentation. Added helper utilities (lib/utils). Removed outdated code/constructs related to previous deployment methods. Updated SQL migration (003-primitive-insertion.sql). Updated binary download script (scripts/download-binaries.sh).Related Problem
TsnAutoStackCDK incompatible with v0.10 #894How Has This Been Tested?
rendererpackage,config/node,config/domain, Go asset bundling, andfrontingkind parsing.kwil-networkutilities (node key generation, genesis file generation).TN-DB-Auto-dev-Stackusing the command:task action-migrate-dev.