Deploying Hadoop in the cloud

1. Scalable Hadoop
in the Cloud
Johan Gustavsson

2. WHO AM I?
• Johan Gustavsson（ヨハン）
• Some contributions in Hadoop, Hive…
• Software Engineer at Treasure Data, Inc.
• Hadoop Team

3. HIGH LEVEL OVERVIEW

4. CONTENT
• Basic architecture:
• Storage
• Replacing a Hadoop Cluster
• Basic job flow with Plazma
• Overview Basic job execution
• Isolation (JobClient)
• Isolation (In cluster)
• Architecture changes (PTD):
• What is PTD?
• Multiple Hadoop Versions
• Multiple Version Job submission

5. BASIC ARCHITECTURE

6. STORAGE (PLAZMA)
• Time indexed database (hourly partitioned)

7. STORAGE (PLAZMA)
• Metadata in Postgres
• Data in mpc1 files on S3 (columnar format file with schema on read)

8. STORAGE (PLAZMA)
• A write will create the files and write them to S3
• Then commit by writing metadata to Postgres

9. REPLACING A HADOOP CLUSTER

10. REPLACING A HADOOP CLUSTER

11. REPLACING A HADOOP CLUSTER

12. BASIC JOB FLOW WITH PLAZMA
• Job one runs reading from Plazma
• Shuffle uses local disk same as always

13. BASIC JOB FLOW WITH PLAZMA
• Output of the first job is written to HDFS

14. BASIC JOB FLOW WITH PLAZMA
• Second job reads from HDFS

15. BASIC JOB FLOW WITH PLAZMA
• Final job in the dag writes to HDFS, then the data is downloaded to a result bucket on S3

16. BASIC JOB FLOW WITH PLAZMA
• In case of INSTERT data is written directly to a table in plazma

17. OVERVIEW BASIC JOB EXECUTION

18. OVERVIEW BASIC JOB EXECUTION

19. OVERVIEW BASIC JOB EXECUTION

20. OVERVIEW BASIC JOB EXECUTION

21. OVERVIEW BASIC JOB EXECUTION

22. ISOLATION (JOBCLIENT)
• Worker builds command line options with java
properties
• Runs QueryRunner as a subprocess

23. ISOLATION (JOBCLIENT)
• UDFs used in query enabled
• Executing CREATE TEMPORARY FUNCTION
• Add databases/tables from PlazmaDB to Metastore
• Executing CREATE DATABASE/TABLE

24. ISOLATION (JOBCLIENT)

25. ISOLATION (JOBCLIENT)
• The good:
• High level of isolation
• OOM deals protect jobs from each other
• The bad:
• Job setup costs are a bit high

26. ISOLATION (IN CLUSTER)
• Using Hadoop resource pools:
• 1 account 1 resource pool (not counting sub-pools)
• Based on price plan max and min running containers are set
• Currently 6711 pools in production

27. ISOLATION (IN CLUSTER)
• The good part:
• Relatively low cost to guarantee minimum resources
• Jobs can still burst to max if resources are free in the cluster

28. ISOLATION (IN CLUSTER)
•The bad parts:
• Due to too many pools meaning cluster separation is needed
• The Resourcemanager tends to get slow with too many pools
• Some unsafe UDFs needs to be disabled
• java_method()
• reflect()

29. ARCHITECTURE CHANGES (PTD)

30. WHAT IS PTD?
• Patchset Treasure Data

31. WHAT IS PTD?
• Patchset Treasure Data
• Name first coined by these two
@frsyuki @tagomoris

32. WHAT IS PTD?
• Patchset Treasure Data
• Name first coined by these two
• Still in development @frsyuki @tagomoris

33. WHAT IS PTD?
• Patchset Treasure Data
• Name first coined by these two
• Still in development
• Original plan
@frsyuki @tagomoris

34. WHAT IS PTD?
• Patchset Treasure Data
• Name first coined by these two
• Still in development
• Original plan
• Base all internal Hadoop components on latest community edition
• Simplify releases to keep an as current version as possible
@frsyuki @tagomoris

35. WHAT IS PTD?
• Patchset Treasure Data
• Name first coined by these two
• Still in development
• Original plan
• Base all internal Hadoop components on latest community edition
• Simplify releases to keep an as current version as possible
• What it’s turning into
@frsyuki @tagomoris

36. WHAT IS PTD?
• Patchset Treasure Data
• Name first coined by these two
• Still in development
• Original plan
• Base all internal Hadoop components on latest community edition
• Simplify releases to keep an as current version as possible
• What it’s turning into
• A complete overhaul of most things related to Hadoop
@frsyuki @tagomoris

37. MULTIPLE HADOOP VERSIONS

38. MULTIPLE HADOOP VERSIONS

39. MULTIPLE HADOOP VERSIONS

40. MULTIPLE HADOOP VERSIONS
• By changing settings in a data bag default version is change

41. MULTIPLE VERSION JOB SUBMISSION

42. MULTIPLE VERSION JOB SUBMISSION

43. ELEPHANT SERVER
• Provides REST api for job submission and monitoring
• All Hive/Pig related code separated from the generic worker
• Distributed on memory queue managing job progress

44. ELEPHANT SERVER
• Built to support multiple versions of Hadoop/Hive/Pig…

45. ELEPHANT SERVER
• This could lead to the following longterm solution

46. JOB PRESERVING RESTARTS
• Worker is polling job status from local server

47. JOB PRESERVING RESTARTS
• A new instance of the server is started joining the Hazelcast cluster and repeatedly trying
to start REST server
• The old instance goes into shutdown mode (not starting new jobs but keep current ones
running)

48. JOB PRESERVING RESTARTS
• Newly submitted jobs popped and managed by the new instance

49. JOB PRESERVING RESTARTS
• Ones all jobs running on the old instance have finished one way or another it shuts down

50. JOB PRESERVING RESTARTS
• Since the port opens up, the new instance starts REST api

51. https://www.treasuredata.com/