Apache Hive facilitates querying and managing large datasets residing in distributed storage. Being used by a very wide community, Hive has been extended to support multiple distributed storage systems. It is now a common practice to have data in different storage systems within an organization. This presentation covers two important aspects of Apache Hive. First aspect covers how Hive makes it possible for organizations to run complex analytical queries across various storage systems or big data components. We recently added HiveKa, to support hive queries on Kafka, and will use it as an example. At Cloudera, we focus not only on providing solutions to help organizations answer bigger questions, but we also make sure that these solutions are robust. The second aspect of this presentation goes over advanced methods/ technologies, like, Random Query Generators, Dockers, Benchmarks, etc that we use at Cloudera to make sure Hive is ready to find right answers from that huge Volume, high Velocity and various Varieties of today’s data.

1. 1© Cloudera, Inc. All rights reserved.
Querying multiple distributed
storage systems with Apache
Hive robustly
Ashish Singh | Software Engineer, Cloudera

2. 2© Cloudera, Inc. All rights reserved. 2© Cloudera, Inc. All rights reserved.

3. 3© Cloudera, Inc. All rights reserved.
Programming
Model
SQL

4. 4© Cloudera, Inc. All rights reserved.

5. 5© Cloudera, Inc. All rights reserved.

6. 6© Cloudera, Inc. All rights reserved.
Storage Handler

7. 7© Cloudera, Inc. All rights reserved.

8. 8© Cloudera, Inc. All rights reserved.

9. 9© Cloudera, Inc. All rights reserved.

10. 10© Cloudera, Inc. All rights reserved.
+ = HiveKa

11. 11© Cloudera, Inc. All rights reserved.
+ = HiveKa
Project available on github (https://github.com/HiveKa)

12. 12© Cloudera, Inc. All rights reserved.

13. 13© Cloudera, Inc. All rights reserved.

14. 14© Cloudera, Inc. All rights reserved.

15. 15© Cloudera, Inc. All rights reserved.

16. 16© Cloudera, Inc. All rights reserved.
Demo Time
16© Cloudera, Inc. All rights reserved.

17. 17© Cloudera, Inc. All rights reserved.

18. 18© Cloudera, Inc. All rights reserved.

19. 19© Cloudera, Inc. All rights reserved.

20. 20© Cloudera, Inc. All rights reserved.

21. 21© Cloudera, Inc. All rights reserved.

22. 22© Cloudera, Inc. All rights reserved.

23. 23© Cloudera, Inc. All rights reserved.
• Strict code review policies

24. 24© Cloudera, Inc. All rights reserved.
• Strict code review policies
• ~7600 upstream tests

25. 25© Cloudera, Inc. All rights reserved.
• Strict code review policies
• ~7600 upstream tests
• End-to-end tests: qTests

26. 26© Cloudera, Inc. All rights reserved. 26© Cloudera, Inc. All rights reserved.

27. 27© Cloudera, Inc. All rights reserved.
@Cloudera

28. 28© Cloudera, Inc. All rights reserved.
@Cloudera
• Believe in Open source Community
• Invest heavily in improving upstream test inf
• Ptests to reduce turn around time

29. 29© Cloudera, Inc. All rights reserved.
@Cloudera
• Believe in Open source Community
• Invest heavily in improving upstream test inf
• Ptests to reduce turn around time
But, is that enough?

30. 30© Cloudera, Inc. All rights reserved. 30© Cloudera, Inc. All rights reserved.
Integration Testing

31. 31© Cloudera, Inc. All rights reserved. 31© Cloudera, Inc. All rights reserved.
Compatibility Testing

32. 32© Cloudera, Inc. All rights reserved. 32© Cloudera, Inc. All rights reserved.
Scale Testing

33. 33© Cloudera, Inc. All rights reserved. 33© Cloudera, Inc. All rights reserved.
Upgrade Testing

34. 34© Cloudera, Inc. All rights reserved. 34© Cloudera, Inc. All rights reserved.
Random Query Generator

35. 35© Cloudera, Inc. All rights reserved.
Thank you
Ashish Singh
asingh@cloudera.com
@singhasdev