The Kafka config files are in k8s/clusters folder. The deployment should work as:
kubectl apply -f zookeeper.yaml
kubectl apply -f kafka.yaml
kubectl apply -f kafka-svc.yamlNOTE: Keep an eye out for KRAFT - Kafka with Raft instead of zookeeper.
This is in the java/consumer folder.
This is in the java/producer folder.
This is in the java/order folder.
The docker images are built with the Google jib tool and compiled with:
mvn compile jib:buildThe docker config file (~/.docker/config.json) has to be set up for this to work:
{
"auths": {
"https://index.docker.io/v1/": {
"auth": "YWRtaW46cGFzc3dvcmQ="
}
},
"HttpHeaders": {
"User-Agent": "Docker-Client/20.10.12 (linux)"
}
}YWRtaW46cGFzc3dvcmQ= is the base64 encoding of the docker user id and password - echo -n admin:password | base64.
sudo apt install maven -yEdit ~/.m2/settings.xml:
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
https://maven.apache.org/xsd/settings-1.0.0.xsd">
<localRepository>/mydata/.m2</localRepository>
</settings>mvn archetype:generate -DgroupId=com.github.ratnadeepb.eda -DartifactId=producer -DarchetypeArtifactId=maven-archet
8000
ype-quickstart -DinteractiveMode=falsemvn exec:java -Dexec.mainClass=com.github.ratnadeepb.kafka.OrderProducerIn the order directory,
mvn install # install into local maven repoIn the pom.xml of producer and consumer:
<dependency>
<groupId>com.github.ratnadeepb.kafka</groupId>
<artifactId>order</artifactId>
<version>1.0-SNAPSHOT</version>
</dependency>mvn clean install # clean previous install and reinstallmvn update project # update the projectmvn generate-sources # generate source if using avro and some suchcd k8s/db/influxdb
kubectl apply -f influxdb-k8.yaml kubectl port-forward svc/influxdb 8086:8086ssh -L 8086:localhost:8086 node0brew install kafkaThis installs Kafka in /usr/local/Cellar/kafka. Manually, start the services.
zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties
kafka-server-start /usr/local/etc/kafka/server.propertieskafka-topics --list --bootstrap-server localhost:9092kafka-topics --describe --bootstrap-server localhost:9092 --topic first-topickafka-topics --delete --bootstrap-server localhost:9092 --topic first-topickafka-topics --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 10 --topic first-topickafka-console-consumer --bootstrap-server localhost:9092 --topic first-topickafka-console-producer --broker-list localhost:9092 --topic first-topicFind Java 11 with, /usr/libexec/java_home -v 11
sudo apt-get install -y librdkafka-devJAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home confluent local services startconfluent local services stopkubectl exec -it kafka-0 -- kafka-topics.sh --zookeeper zk-cs.default.svc.cluster.local:2181 --listkubectl exec -it kafka-0 -- kafka-topics.sh --zookeeper zk-cs.default.svc.cluster.local:2181 --describe --topic OrderTopickubectl exec -it kafka-0 -- kafka-topics.sh --zookeeper zk-cs.default.svc.cluster.local:2181 --delete --topic OrderTopickubectl exec -it kafka-0 -- kafka-topics.sh --zookeeper zk-cs.default.svc.cluster.local:2181 --create --replication-factor 2 --partitions 10 --topic OrderTopicsudo docker build -t ratnadeepb/kafka-producer .Kafka and Zookeeper yaml files
Producer-Consumer Example using Kafka
Producer-Consumer Example using Nats
Rust producer-consumer
Kafka Consumer Group
Kafka Operating Modes
Java Producer Consumer
Weighted Consumer Group Assignor - Java
Lag based assignor
Install Influx
InfluxDB Java Client
Java Main
Java Influx