TEZ-4631: Include an official script that installs hadoop and tez and runs a simple example DAG #414
Conversation
… runs a simple example DAG
This comment was marked as outdated.
This comment was marked as outdated.
There was a problem hiding this comment.
Do we need to specify: mapreduce.framework.name as yarn as well?
for me earlier it never use to work unless, I specify export HADOOP_USER_CLASSPATH_FIRST=true
does it work for you without that, even BigTop had to add that
https://github.com/apache/bigtop/pull/1246/files#diff-f68b85f9302907e466b58d438376afb074df98fdbe571d30c188cd1767ff11eeR18
dev-support/bin/tez_run_example.sh
Outdated
| #$HADOOP_HOME/sbin/stop-dfs.sh | ||
| #$HADOOP_HOME/sbin/stop-yarn.sh | ||
|
|
||
| hdfs namenode -format |
There was a problem hiding this comment.
did you run it twice, usually if there was previous installation & if you run namenode -format, it asks for a prompt, are you sure you want to delete & you need to give Y
2025-05-16 13:43:11,940 INFO snapshot.SnapshotManager: SkipList is disabled
2025-05-16 13:43:11,942 INFO util.GSet: Computing capacity for map cachedBlocks
2025-05-16 13:43:11,942 INFO util.GSet: VM type = 64-bit
2025-05-16 13:43:11,942 INFO util.GSet: 0.25% max memory 7.1 GB = 18.2 MB
2025-05-16 13:43:11,942 INFO util.GSet: capacity = 2^21 = 2097152 entries
2025-05-16 13:43:11,949 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
2025-05-16 13:43:11,949 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
2025-05-16 13:43:11,949 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
2025-05-16 13:43:12,062 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
2025-05-16 13:43:12,062 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
2025-05-16 13:43:12,064 INFO util.GSet: Computing capacity for map NameNodeRetryCache
2025-05-16 13:43:12,064 INFO util.GSet: VM type = 64-bit
2025-05-16 13:43:12,064 INFO util.GSet: 0.029999999329447746% max memory 7.1 GB = 2.2 MB
2025-05-16 13:43:12,064 INFO util.GSet: capacity = 2^18 = 262144 entries
Re-format filesystem in Storage Directory root= /tmp/hadoop-ayushsaxena/dfs/name; location= null ? (Y or N)
There was a problem hiding this comment.
there is a " -force" option of namenode format, let me try
dev-support/bin/tez_run_example.sh
Outdated
| hadoop fs -mkdir /apps/ | ||
| hadoop fs -mkdir /apps/tez-$TEZ_VERSION |
There was a problem hiding this comment.
hadoop fs -mkdir -p /apps/tez-$TEZ_VERSION
dev-support/bin/tez_run_example.sh
Outdated
| hadoop fs -copyFromLocal words.txt /words.txt | ||
|
|
||
| # finally run the example | ||
| hadoop jar $TEZ_HOME/tez-examples-$TEZ_VERSION.jar orderedwordcount /words.txt /words_out |
There was a problem hiding this comment.
I haven't used yarn executable so far, fine with changing, but for the record: what advantages does it have?
There was a problem hiding this comment.
AFAIK for YARN it should be yarn jar
https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop.cmd#L186-L190
if you have yarn opts defined and all, it would shooting a warning as well. hadoop jar was for MR job, though it doesn't fail for Tez job today
dev-support/bin/tez_run_example.sh
Outdated
|
|
||
| # configure this if needed, by default it will use the latest stable versions in the current directory | ||
| export TEZ_VERSION=$(curl -s "https://downloads.apache.org/tez/" | grep -oP '\K[0-9]+\.[0-9]+\.[0-9]+(?=/)' | sort -V | tail -1) # e.g. 0.10.4 | ||
| export HADOOP_VERSION=$(curl -s "https://downloads.apache.org/hadoop/common/" | grep -oP 'hadoop-\K[0-9]+\.[0-9]+\.[0-9]+(?=/)' | sort -V | tail -1) # e.g. 3.4.1 |
There was a problem hiding this comment.
shouldn't the hadoop version should be from the pom? not always the latest version is gonna work with Tez
There was a problem hiding this comment.
good question, depends on what we want to achieve with this script, here is what I can think of:
- get hadoop from the tez pom.xml as you adviced
- both HADOOP_VERSION and TEZ_VERSION could be used from env if already defined (making the user able to define any for random experience)
There was a problem hiding this comment.
I am like if the user defines it use it else get it from the POM, I believe that is what the Hive docker build script does that as well.
There was a problem hiding this comment.
yeah, makes sense, let me do the same
dev-support/bin/tez_run_example.sh
Outdated
| cd $HADOOP_STACK_HOME | ||
| wget -nc https://a 8000 rchive.apache.org/dist/hadoop/common/hadoop-$HADOOP_VERSION/hadoop-$HADOOP_VERSION.tar.gz | ||
| wget -nc https://archive.apache.org/dist/tez/$TEZ_VERSION/apache-tez-$TEZ_VERSION-bin.tar.gz | ||
|
|
There was a problem hiding this comment.
is there some caching possible? like if it is already present we don't download it again
There was a problem hiding this comment.
-nc (--no-clobber) is exactly what takes care of this
yeah, I can see this workaround happening everywhere, but here, it has just worked OOTB, maybe a certain state of defining ENV vars like HADOOP_CLASSPATH? I don't know
let me check mapreduce.framework.name as well, for me, the script simply ran a Tez DAG, so I haven't configured anything more...but this is really interesting, I'll discover |
wow, that's indeed needed, otherwise I get exotic exception like adding this export right before the Tez DAG submission maybe it ran successfully before because I ran the steps one-by-one and I had environment exports, so messed up something/anything |
|
🎊 +1 overall
This message was automatically generated. |
… runs a simple example DAG (apache#414) - addendum ASF license
… runs a simple example DAG (apache#414) - addendum ASF license
… runs a simple example DAG (apache#414) - addendum ASF license
… runs a simple example DAG (apache#414) - addendum ASF license
Running a simple Tez example from the terminal can be challenging, especially since the installation guide isn’t always up to date. In the long term, providing both a clear web presence and a convenience script would be essential for maintaining the project’s health and earning users’ trust.
The introduced dev-support/bin folder structure follows the hadoop's one: https://github.com/apache/hadoop/tree/trunk/dev-support/bin