8000 GitHub - D-Thatcher/eks_kafka_argo: Self-hosted Kafka cluster on EKS, integrated with ArgoCD and deployed as IaC
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

D-Thatcher/eks_kafka_argo

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

20 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Self-hosted Kafka Cluster using the Strimzi Operator integrated with ArgoCD

Getting started

6D67
  • ensure the following cli tools are up-to-date aws-cli, kubectl, terrarform
  • deploy an Elastic Kubernetes Cluster: make deploy_infrastructure
  • authenticate to your cluster: make auth_kubectl
  • confirm your access: kubectl get nodes
  • port-forward the Argo UI: make proxy_argo
  • in a sepeparate terminal, get your argo user password: make get_argo_secret
  • in a browser, visit https://localhost:8080 and login in with the user admin and your password
  • to deploy a test NGINX app: make deploy_example_app
    • it'll populate the app on the Argo UI, and you can view its exernal service's URL with: make get_example_app_url
    • remove the example app: make remove_example_app
  • deploy the Strimzi Kafka Operator: make deploy_kafka_operator
  • deploy an example Kafka cluster: make deploy_kafka_cluster

PVC Issue

There's a known Argo + Strimzi issue with incorrectly reporting the state of the PVCs. To mitigate this, we've disabled auto-pruning for this Application. Do not manually prune the PVC even though Argo says it's out-of-sync. Image of Kafka Cluster

Test Kafka Producer + Consumer

  • run make produce_messages to create a producer pod that will connect to the Kafka Cluster
  • type some messages in the terminal like this: Image of Producer
  • run make consume_messages to create a consumer pod that will connect to the Kafka Cluster
  • the previous messages will appear in the terminal: Image of Consumer

Clean up

  • clean up all resources, including the underlying kubernetes cluster: make clean

Links

About

Self-hosted Kafka cluster on EKS, integrated with ArgoCD and deployed as IaC

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

0