2019 Pune Data Conference Software smoke testing is a preliminary level of testing. It makes certain that all of the primary components of a system are functioning correctly. For example, when installing a new secured Hadoop cluster, running a series of quick tests to make sure that things like HDFS and MapReduce are operational can save a lot of headache before enabling Kerberos. Smoke tests can also save you time and embarrassment by making sure that things work before you turn the cluster over to your customer. In this talk, Michael Arnold will explain the utility of testing Hadoop components after cluster builds and software upgrades. Michael will present code examples that you can use to confirm functionality of Spark, Kudu, HBase, Kafka, MapReduce, etc on your cluster.

1. Michael Arnold
Admins: Smoke Test Your
Hadoop Cluster!
2019 Pune Data Conferenceby

2. • Who is this guy?
• What is a smoke test?
• When should we smoke test?
• Why should we smoke test?
• How should we smoke test?
Agenda

3. 3
Who is Michael Arnold?
• Principal Systems Engineer / Consultant
• Automation geek
• 20+ years in IT - 9 years with Hadoop
• I help people deal with:
• Servers (physical and virtual)
• Networks
• Server operating systems
• Hadoop distributions
• Making it all run smoothly

4. 4
A smoke test is a quick, preliminary test to confirm that
basic functionality exists in the item being tested.
What is a Smoke Test?

5. 5
Addresses basic questions like:
• does the program run?
• does the user interface open?
• does clicking the main button do anything?
https://en.wikipedia.org/wiki/Smoke_testing_(software)
What is a Smoke Test?

6. 6
"The process of smoke testing aims to determine whether
the application is so badly broken as to make further
immediate testing unnecessary."
https://en.wikipedia.org/wiki/Smoke_testing_(software)
What is a Smoke Test?

7. 7
Software Testing Levels:
• Unit test
• Integration test
• System test
• Acceptance test
Are there other kinds of tests?

8. 8
Software Testing Types:
• Smoke test
• Functional test
• Usability test
• Security test
• Performance test
• Regression test
• Compliance test
Are there other kinds of tests?

9. 9
Integration Smoke Test
An integration smoke test is simply a smoke test of the parts
of the system.
In our case, a Hadoop subsystem like HDFS and
MapReduce.

10. 10
Performing a smoke test immediately following an
installation or upgrade acts as a way to ensure that the high-
level functionality of the system is working.
Even after minor configuration changes.
When should an integration smoke test be performed?

11. 11
It can catch regressions or misconfigurations before release
to customers.
It can mitigate the risk of outage.
Why integration smoke testing?

12. 12
Does not catch regressions within applications running on
top of the cluster services (ie custom Spark or MapReduce
code).
That would be a System test.
What does Hadoop integration smoke testing not do?

13. 13
Smoke Test Examples:
• ZooKeeper
• HDFS
• MapReduce
• Spark
• Pig
• Hive
• Impala
• Kudu
• HBase
• Accumulo
• Solr
• Kafka

14. 14
# Replace $ZOOKEEPER 'localhost' with the correct hostname.
ZOOKEEPER=localhost:2181
cat <<EOF | zookeeper-client -server $ZOOKEEPER 2>/dev/null | grep -v
INFO
create /zk_test my_data
ls /
get /zk_test
set /zk_test junk
get /zk_test
quit
EOF
ZooKeeper

15. 15
hdfs dfs -ls /
hdfs dfs -put /etc/hosts /tmp/hosts
hdfs dfs -get /tmp/hosts /tmp/hosts123
cat /tmp/hosts123
HDFS

16. 16
yarn jar /opt/cloudera/parcels/CDH/lib/hadoop-0.20-
mapreduce/hadoop-examples.jar pi 10 1000
MapReduce

17. 17
echo "this is the end. the only end. my friend." >/tmp/mrin
hdfs dfs -put /tmp/mrin /tmp/
yarn jar /opt/cloudera/parcels/CDH/lib/hadoop-0.20-
mapreduce/hadoop-examples.jar wordcount /tmp/mrin
/tmp/mrout
hdfs dfs -cat /tmp/mrout/part-*
MapReduce
(wordcount)

18. 18
MASTER=yarn /opt/cloudera/parcels/CDH/lib/spark/bin/run-
example SparkPi 100
Spark

19. 19
echo "this is the end. the only end. my friend." >/tmp/sparkin
hdfs dfs -put /tmp/sparkin /tmp/
cat <<EOF | spark-shell --master yarn-client
val file = sc.textFile("hdfs:///tmp/sparkin")
val counts = file.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey(_
+ _)
counts.saveAsTextFile("hdfs:///tmp/sparkout")
exit
EOF
hdfs dfs -cat /tmp/sparkout/part-*
Spark
(wordcount)

20. 20
hdfs dfs -copyFromLocal /etc/passwd /tmp/test.pig
cat <<EOF | pig /tmp/pig
A = LOAD '/tmp/test.pig' USING PigStorage(':');
B = FOREACH A GENERATE $0 AS id;
STORE B INTO '/tmp/test.pig.out';
EOF
hdfs dfs -cat /tmp/test.pig.out/part-m-00000
Pig

21. 21
# Replace $HIVESERVER2 with the correct hostname that is running the HS2
HIVESERVER2=
# Create table
beeline -n `whoami` -u "jdbc:hive2://${HIVESERVER2}:10000/" -e "CREATE
TABLE test_hive(id INT, name STRING);'
# Insert data
beeline -n `whoami` -u "jdbc:hive2://${HIVESERVER2}:10000/" -e 'INSERT
INTO TABLE test_hive VALUES (1, "justin"), (2, "michael"), (3, "scott");'
# Query table
beeline -n `whoami` -u "jdbc:hive2://${HIVESERVER2}:10000/" -e "SELECT *
FROM test_hive WHERE id=1;"
Hive

22. 22
# Replace $IMPALAD with the correct hostname that's running the Impala
Daemon
IMPALAD=
impala-shell -i $IMPALAD -q 'INVALIDATE METADATA;
SELECT * FROM test_hive;'
Impala (data from Hive)

23. 23
# Replace $IMPALAD with the correct hostname that's running the Impala Daemon
IMPALAD=
# Create table
impala-shell -i $IMPALAD -q 'CREATE TABLE test_impala(id INT,
name STRING);'
# Insert data
impala-shell -i $IMPALAD -q 'INSERT INTO TABLE test_impala
VALUES (1, "bohan"), (2, "shanlin"), (3, "xiaobing");'
# query table
impala-shell -i $IMPALAD -q 'SELECT * FROM test_impala WHERE
id=1;'
Impala

24. 24
# Replace $IMPALAD with the correct hostname that's running the Impala Daemon
IMPALAD=
# Create table
impala-shell -i $IMPALAD -q 'CREATE TABLE test_kudu(id BIGINT, name
STRING, PRIMARY KEY(id)) PARTITION BY HASH (id) PARTITIONS 3
STORED AS KUDU;'
# Insert data
impala-shell -i $IMPALAD -q 'INSERT INTO TABLE test_kudu VALUES
(1, "wasim"), (2, "ninad"), (3, "mohsin");'
# query table
impala-shell -i $IMPALAD -q 'SELECT * FROM test_kudu WHERE id=1;'
Kudu (via Impala)

25. 25
cat <<EOF >/tmp/hbase
create 'test', 'cf'
list 'test'
put 'test', 'row1', 'cf:a', 'value1'
scan 'test'
exit
EOF
hbase shell -n /tmp/hbase
HBase

26. 26
cat <<EOF >/tmp/accumulo
createtable test
insert row1 cf a value1
flush -w
scan -t test
exit
EOF
accumulo shell -u root -p secret -f /tmp/accumulo
Accumulo

27. 27
# Replace $SOLRSERVER with the correct hostname that's running a Solr
Server
SOLRSERVER=
solrctl instancedir --generate /tmp/solr
solrctl instancedir --create t_conf /tmp/solr
solrctl collection --create t_col -s 1 -c t_conf
Solr

28. 28
cd /opt/cloudera/parcels/CDH/share/doc/solr-
doc*/example/exampledocs
java -Durl=http://${SOLRSERVER}:8983/solr/t_col/update -
jar post.jar *.xml
curl
"http://${SOLRSERVER}:8983/solr/t_col_shard1_replica1/select?q
=*%3A*&wt=json&indent=true"
Solr (continued)

29. 29
# Replace $KAFKA 'localhost' with the correct hostname.
KAFKA=localhost:9092
# Replace $ZK_ROOT with the correct ZooKeeper root for Kafka (if you configured one).
ZK_ROOT=
kafka-topics --zookeeper ${ZOOKEEPER}${ZK_ROOT} --create --
topic test_kafka --partitions 1 --replication-factor 1 2>/dev/null
kafka-topics --zookeeper ${ZOOKEEPER}${ZK_ROOT} --list
2>/dev/null
Kafka

30. 30
# Run the consumer and producer in separate terminals.
# Send data to the producer and it appears in the consumer.
# ^C to quit.
kafka-console-consumer --zookeeper
${ZOOKEEPER}${ZK_ROOT} --topic test_kafka 2>/dev/null
cat /etc/passwd | kafka-console-producer --broker-list
$KAFKA --topic test_kafka 2>/dev/null
Kafka (continued)

31. 31
Make sure to remove your test data as part of the
integration smoke test.
Removal details:
https://github.com/teamclairvoyant/hadoop-smoke-tests
Clean Up

32. Questions?
https://github.com/teamclairvoyant/hadoop-smoke-tests

33. Michael Arnold
@hadoopgeek
https://www.linkedin.com/in/michaelarnold/
Thank you
http://clairvoyantsoft.com/insight/