8000 Kafka_0_9_support by dmitrypekar · Pull Request #168 · mesos/kafka · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Kafka_0_9_support #168

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Dec 30, 2015
Merged

Kafka_0_9_support #168

merged 6 commits into from
Dec 30, 2015

Conversation

dmitrypekar
Copy link
Contributor

Pushed PR with changes required for 0.9 support.
Joe, please review&eval&merge when will have time.

@dmitrypekar dmitrypekar mentioned this pull request Dec 21, 2015
Closed
@JohnOmernik
Copy link

I got a Scheduler Test Failure when I tried to build this:

oad https://repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.10.6/scala-reflect-2.10.6.jar
:processResources UP-TO-DATE
:classes
:compileTestJava UP-TO-DATE
:compileTestScala
[ant:scalac] Element '/mapr/brewpot/mesos/prod/mesos-kafka/kafka/out/gradle/resources/main' does not exist.
:processTestResources UP-TO-DATE
:testClasses
:test

ly.stealth.mesos.kafka.SchedulerTest > syncBrokers FAILED
java.lang.NullPointerException at SchedulerTest.scala:88

ly.stealth.mesos.kafka.SchedulerTest > launchTask FAILED
java.lang.NullPointerException at SchedulerTest.scala:182

ly.stealth.mesos.kafka.SchedulerTest > acceptOffer FAILED
java.lang.NullPointerException at SchedulerTest.scala:112

ly.stealth.mesos.kafka.SchedulerTest > newTask FAILED
java.lang.NullPointerException at SchedulerTest.scala:54

ly.stealth.mesos.kafka.CliTest > connection_refused FAILED
java.lang.IllegalStateException at CliTest.scala:242
java.lang.IllegalStateException at CliTest.scala:44

@dmitrypekar
Copy link
Contributor Author

Could you please send a zip-ed test reports. It will contain a stack trace.
For me the test runs fine.

Also could you please build with ./gradlew jar -x test and try to run?

@Hylke1982
Copy link
Contributor

I have found the problem why the test were failing, I have created a pull request that fixes the test execution on the Kafka_0_9_support branch.

The problem that the version was missing in the temporary .tgz files, a placeholder version number is now added.

@dmitrypekar
Copy link
Contributor Author

Merged fix of @Hylke1982 .
Now tests should pass fine.

@joestein
Copy link
Contributor

stepped through upgrade procedure looks good, after stopping old scheduler and starting new one

./kafka-mesos.sh broker stop 0
./kafka-mesos.sh broker update 0 --options inter.broker.protocol.version=0.8.2.1
./kafka-mesos.sh broker start 0
./kafka-mesos.sh broker stop 1
./kafka-mesos.sh broker update 1 --options inter.broker.protocol.version=0.8.2.1
./kafka-mesos.sh broker start 1
./kafka-mesos.sh broker stop 2
./kafka-mesos.sh broker update 2 --options inter.broker.protocol.version=0.8.2.1
./kafka-mesos.sh broker start 2
kafka_2.11-0.9.0.0/bin/kafka-console-consumer.sh --zookeeper 192.168.3.5:2181/a --topic 09test --from-beginning
./kafka-mesos.sh broker stop 0
./kafka-mesos.sh broker update 0 --options inter.broker.protocol.version=0.9.0.0
./kafka-mesos.sh broker start 0
./kafka-mesos.sh broker stop 1
./kafka-mesos.sh broker update 1 --options inter.broker.protocol.version=0.9.0.0
./kafka-mesos.sh broker start 1
./kafka-mesos.sh broker stop 2
./kafka-mesos.sh broker update 2 --options inter.broker.protocol.version=0.9.0.0
./kafka-mesos.sh broker start 2

joestein added a commit that referenced this pull request Dec 30, 2015
@joestein
Merge pull request #168 from mesos/kafka_0_9_support
7c03486 
Kafka_0_9_support
@joestein joestein merged commit 7c03486 into master Dec 30, 2015
@keithchambers
Copy link

Nice work!

@mindscratch
Copy link
Contributor

@joestein are the steps you listed required to upgrade to kafka 0.9 and 0.9.4 of the framework? I'm currently on 0.9.2.0 of the framework.

@joestein
Copy link
Contributor
joestein commented Jan 2, 2016

Hey @mindscratch If you are running 0.9.4.0 scheduler and want to stick to running Kafka 0.8.X brokers that is fine too. To upgrade the scheduler stop your scheduler and start the new one (however you are scheduling HA). What makes the broker Kafka 0.8.X vs 0.9.X is the Kafka binary you download that is sitting in the directory with the jar when you start the scheduler. In the steps above before I had two directories one for 0.9.3.0 mesosKafka and kafka 0.8.2.2. That scheduler turned off and started 0.9.4.0 mesosKafka. At that point all the tasks/broker where still exactly like they were before 0.8.2.2 brokers still running happily along. When the stop/update/start occured for each task/broker when it started the new kafka_2.11-0.9.0.0.tgz is what the scheduler gave back as the URI and made available for download to the executer when it restarted back up

$ ls -l ~/github/mesos/kafka/run_0822/
total 69784
-rw-r--r--  1 joestein  staff  17481604 Dec 27 21:39 kafka-mesos-0.9.3.0.jar
-rw-r--r--  1 joestein  staff      1452 Dec 29 10:27 kafka-mesos.json
-rw-r--r--  1 joestein  staff       418 Dec 29 10:31 kafka-mesos.properties
-rwxr-xr-x  1 joestein  staff       307 Dec 27 21:39 kafka-mesos.sh
drwxr-xr-x  8 joestein  staff       272 Dec 29 23:02 kafka_2.10-0.8.2.2
-rw-r--r--  1 joestein  staff  16162559 Dec 27 21:39 kafka_2.10-0.8.2.2.tgz
-rw-r--r--  1 joestein  staff       374 Dec 27 21:40 me.err
-rw-r--r--  1 joestein  staff   2064554 Dec 28 07:29 me.log

$ ls -l ~/github/mesos/kafka/run_0900/
total 103712
-rw-r--r--  1 joestein  staff  17488887 Dec 29 09:09 kafka-mesos-0.9.4.0.jar
-rw-r--r--  1 joestein  staff       213 Dec 29 10:29 kafka-mesos.properties
-rwxr-xr-x  1 joestein  staff       307 Dec 27 21:50 kafka-mesos.sh
drwxr-xr-x  8 joestein  staff       272 Nov 20 20:02 kafka_2.11-0.9.0.0
-rw-r-----@
8000
 1 joestein  staff  35600524 Dec 29 09:09 kafka_2.11-0.9.0.0.tgz

@mindscratch
Copy link
Contributor

@joestein the upgrade steps you documented previously worked great, thanks!

@joestein joestein mentioned this pull request Apr 15, 2016
Closed
@joestein joestein mentioned this pull request May 23, 2016
Closed
@steveniemitz steveniemitz deleted the kafka_0_9_support branch January 25, 2017 15:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@dmitrypekar @JohnOmernik @Hylke1982 @joestein @keithchambers @mindscratch
0