Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Introducing Cloud Development with Mantl

347 views

Published on

Speaker Brian Hicks and Ryan Eschinger

Published in: Technology
  • Be the first to comment

Introducing Cloud Development with Mantl

  1. 1. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID We’re ready. Are you?
  2. 2. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Introducing Cloud Development with Mantl Brian Hicks (Mantl Core Committer) Ryan Eschinger (Mantl Core Committer)
  3. 3. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID • Introduction • Mesos • Frameworks • Docker • Consul • Mantl Agenda 3
  4. 4. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Introduction 4 Brian Hicks Ryan Eschinger
  5. 5. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Mesos What is it? Program against your datacenter like it’s a single pool of resources Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively. 5
  6. 6. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Mesos 2-Tier Scheduler Frameworks: • Receive resources offered by the master nodes • Schedule work on offered resources • Are notified if workloads fail via internal and external mechanisms 6 Mesos: • Determines resources available on nodes in the cluster • Offers those resources to frameworks • Keep track of node status (online, offline, remaining resources)
  7. 7. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Mesos Architecture Slaves (Agents in 1.0): • Tell the masters which resources are available • Run tasks on claimed resources 7 Masters: • Communicate with Schedulers • Keep track of Agents • Provide a UI for Ops / DevOps
  8. 8. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Mesos Architecture Executor: • A binary or script that runs on the slave to perform work • Runs in an isolated environment (can use Docker) 8 Framework (Scheduler): • Schedules work based on offers received • Domain specific
  9. 9. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Framework Demos 9
  10. 10. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Marathon What is it? 1 0 A cluster-wide init and control system for services in cgroups and Docker containers Marathon is an Apache Mesos framework for long- running applications. Given that you have Mesos running as the kernel for your datacenter, Marathon is the init or upstart daemon.
  11. 11. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Marathon Overview 1 1 • Developed by Mesosphere • Controls app resources, ports, environment variables, and scaling • Runs “long-running” tasks
  12. 12. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Cassandra What is it? Distributed database designed for scalability and high availability Apache Cassandra is an open source distributed database management system designed to handle large amounts of data across many commodity servers, providing high availability with no single point of failure. 1 2
  13. 13. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Cassandra Overview 1 3 • Designed to handle big data workloads across multiple nodes (ring) with no single point of failure • Data is distributed among all nodes in the cluster • Uses the Gossip protocol for peer-to-peer communication • Eventually consistent (C): prioritizes Availability and Partitioning tolerance (AP)
  14. 14. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Cassandra Mesos Framework 1 4 • Automates bootstrapping and operations of Cassandra clusters • Bootstraps initial seed nodes and expands cluster to desired number of nodes • Runs periodic operational tasks • Restart, remove, and replace nodes on failure • Self-contained: does not require specialized software installed on agent nodes
  15. 15. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Spark What is it? A fast and general engine for large-scale data processing Apache Spark is an open source, high performance big data processing framework. It provides high-level APIs in Java, Scala, Python, and R. Spark includes higher-level libraries, including support for SQL queries, streaming data, machine learning and graph processing. 1 5
  16. 16. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Spark Overview 1 6 • Originally developed at UC Berkeley in 2009 • Up to 100x faster than Hadoop for large scale data processing • Execution engine works both in-memory and on-disk • Architecture • Data storage: supports Hadoop-compatible data sources like HDFS, HBase, Cassandra, etc. • API: Scala, Java, Python, and R. • Resource Management: standalone, Mesos, Yarn
  17. 17. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Spark on Mesos 1 7 • Runs as a framework • Supports the Docker containerizer • 2-level scheduling: Spark can make decisions about the offers it receives from Mesos • Enables sophisticated scheduling scenarios
  18. 18. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Spark Run Modes 1 8 • Coarse-grained • Runs long-running Spark executors on every node • Fast startup, better for interactive sessions, but can be inefficient • Fine Grained • Launches a Spark executor per task • Slower startup but can utilize resources better
  19. 19. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Docker What is it? A shipping container system for applications A container virtualization platform that abstracts underlying Linux resource isolation technologies like cgroups and kernel namespaces. Allows developers to build, package, and ship applications that can run anywhere*. *anywhere that runs a 3.10+ Linux kernel 1 9
  20. 20. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Docker Overview 2 0 • Compared to virtual machines, containers are smaller, lighter weight, more portable, and easier to deploy • Package an application, along with all of its dependencies, in a single artifact • Use the same artifact throughout development, testing, and production • Reduces concerns about the compatibility in different environments
  21. 21. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Consul 2 1
  22. 22. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Consul Discovery (HTTP) 2 2 curl localhost:8500/v1/catalog/service/marathon [{ "Address": "10.0.113.214", "Node": "mantl-control-01", "ServiceAddress": "", "ServiceID": "marathon", "ServiceName": "marathon", "ServicePort": 18080, "ServiceTags": ["marathon"] }]
  23. 23. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Consul Discovery (DNS) 2 3 dig +short marathon.service.consul 10.0.113.214 10.0.220.54 10.0.131.25 dig +short marathon.service.consul SRV 1 1 18080 mantl-control-02.node.mantl.consul 1 1 18080 mantl-control-01.node.mantl.consul 1 1 18080 mantl-control-03.node.mantl-consul
  24. 24. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Consul Key/Value Store 2 4 curl -X PUT -d world http://localhost:8500/v1/kv/hello true
  25. 25. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Consul Key/Value Store 2 5 curl -X GET http://localhost:8500/v1/kv/hello [{ "CreateIndex": 223, "Flags": 0, "Key": "hello", "LockIndex": 0, "ModifyIndex": 223, "Value": "d29ybGQ=" }]
  26. 26. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Consul consul-template 2 6 global maxconn {{or (key "service/haproxy/maxconn") 256}} debug {{range services}} frontend {{.Name}} bind *:{{key (printf "ports/%s" .Name)}} {{range service .Name}} server {{.Node}} {{.Address}}:{{.Port}}{{end}} {{end}}
  27. 27. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Mantl 2 7
  28. 28. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID Thank you © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID 28
  29. 29. © 2015 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation ID

×