Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Cloudbreak - Technical Deep Dive

1,188 views

Published on

Cloudbreak - Technical Deep Dive

Published in: Technology

Cloudbreak - Technical Deep Dive

  1. 1. Cloudbreak – Technical Deep Dive Janos Matyas & Krisztian Horvath Hortonworks
  2. 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Presenters Krisztian Horvath Senior Member of technical staff, Cloudbreak Former Co-Founder at SequenceIQ Janos Matyas Senior Director of Engineering, Cloudbreak Former Co-Founder and CTO for SequenceIQ
  3. 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda Goals and Motivations Technology Stack + Deep Dive Lessons Learned + Best Practices Demo + Q & A
  4. 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Goals and Motivations – What We Wanted to Do…  Declarative/full Hadoop stack provisioning in all major cloud providers  Automate and unify the process  Zero-configuration approach  Same process through a cluster lifecycle (Dev, QA, UAT, Prod)  Provide tooling - UI, REST API and CLI/shell  Secure and multi-tenant  SLA policy based autoscaling
  5. 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Goals and Motivations – What We Wanted to Do…  All cloud providers are fundamentally different…  Compute, network, security, performance  We want to share what we found, and how we made it work!
  6. 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda Goals and Motivations Technology Stack + Deep Dive Lessons Learned + Best Practices Demo + Q & A
  7. 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Technology Stack  Apache Ambari  Cloud provider API  Salt  Docker  Packer
  8. 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Deep Dive - Overview  Cloudbreak Deployer (CBD) – Tool to deploy the Cloudbreak application – Microservice architecture (using Docker) – DevOps friendly  Cloudbreak Application – Extensible, available through UI, CLI, REST API – SLA auto-scaling policy management  Cluster deployed with Cloudbreak
  9. 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Deep Dive – Cloudbreak Deployer  Installation – Single binary, written in Go – Requires Docker 1.9.1+ – DIY installation on any RHEL / CentOS / Oracle Linux 7 (64-bit) distro – Use one of the pre-built cloud images (AWS, Azure, GCP, OpenStack)  Operations – Easy upgrades/downgrades, automatic schema migration  Cloud provider support – AWS – generates IAM roles – Azure – ARM and DASH config  Utilities – Cloudbreak shell support - interactive, remote, automated execution, OAuth2 token generation – Local development environment setup
  10. 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Deep Dive – Cloudbreak Application  Installation – Done with Cloudbreak Deployer (CBD)  Operations – Consistent feature set through UI, CLI and secure REST API – Multi-tenant, ACL setup, usage reports – Custom stack repositories, failure actions – Event history, cluster management – SLA based auto-scaling policy configs, enforcement  Cloud provider support – Agnostic API – AWS, Azure, GCP, OpenStack, Mesos – SPI interface – bring your own provider, stack under Cloudbreak management
  11. 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Deep Dive – Cluster deployed with Cloudbreak  Installation – Managed by Cloudbreak using cloud provider API – Default (optimized) configs – specific to cloud provider  Operations – Default, custom configs for stacks, services, network, storage, security – Declarative Hadoop cluster – Custom instance types (heterogeneous clusters) – Different storage types – Configurable network – Security (access, Kerberos, SSSD, FreeIPA)  Utilities – Ambari Views – Metadata/shared clusters support
  12. 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda Goals and Motivations Technology Stack + Deep Dive Lessons Learned + Best Practices Demo + Q & A
  13. 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Lessons Learned  Not all cloud providers are the same – Difference in performance, storage and functionality  (Capacity) planning – Based on workload type (batch / interactive and ad-hoc / long running) – Use heterogeneous clusters – Trial and error – mistakes are cheap, iterate until you find your best fit – Leverage the cloud - scale your cluster on demand  Number one consideration – storage – Multiple choices (ephemeral, block storage and BLOB store) – Bring compute to storage – might not work (everywhere) – in cloud everything is as a service – Independently scale storage from compute, partition your data  Security – Consider using strict security rules (private subnets, access, etc) and use edge nodes
  14. 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Lessons Learned - AWS  Compute – Find your instance types for the workload, use heterogeneous clusters – Different instance types for transient (e.g. C4, M4) and long running (e.g. H2, D2) clusters – Dedicated instances (to avoid noise, regulations e.g. HIPPA)  Storage – Use latest version of Hadoop (Hortonworks contributed cloud specific optimizations) – Note that S3 gives you only eventual consistency – Different driver implementation: S3n (native, jets3t based), S3a (successor of n) , S3 (block based)  Network – Use enhanced networking (Amazon Linux by default, RHEL based – apply patch) – Placement groups – Not all instance types can use the 10Gbit network (e.g. use 8x)  Security – Use instance roles to access S3, deploy in a private subnet/VPC
  15. 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Lessons Learned - AWS * D28xlarge used as instance type
  16. 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Lessons Learned - AWS * D28xlarge used as instance type
  17. 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Lessons Learned - Azure  Compute – Find your instance types for the workload, use heterogeneous clusters – Different instance types for transient (e.g. A and D family) and long running (e.g. Dv2) clusters – Use ARM instead of old API  Storage – Use latest version of Hadoop (Hortonworks contributed cloud specific optimizations) – Storage account scaling limitations – Use WASB or WASB with DASH (default with Cloudbreak) – Azure Data Lake Store – soon – Ephemeral disk is faster than root disk – does not survive auto-updates  Network – No PTR record/reverse lookup support  Security – Integrate/sync with your corporate AD
  18. 18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Lessons Learned - Azure
  19. 19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Lessons Learned - GCP  Compute – Find your instance types for the workload, use heterogeneous clusters – No template based provisioning  Storage – Use latest version of Hadoop (Hortonworks contributed cloud specific optimizations) – Use Google Cloud Storage Connector  Network – Network isolation/DNS problem  Security
  20. 20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Lessons Learned - OpenStack  Compute – Find your instance types for the workload, use heterogeneous clusters – Use Heat templates instead of API calls (we support both)  Storage – Currently we support only Cinder volumes – Swift and Ceph is planned – Data locality through Cloudbreak – let us know your topology or rack/hypervisor mapping  Network – Configure DNS properly – Use multiple network (Neutron) nodes in case of a large cluster  Security – Use Keystone 3 (support for OAuth, Federation, introduction of groups/domains)
  21. 21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Lessons Learned - Mesos  In Tech Preview – come and talk to us after the talk – Or @Hortonworks boot
  22. 22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda Goals and Motivations Technology Stack + Deep Dive Lessons Learned + Best Practices Demo + Q & A
  23. 23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Thank You

×