Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Lies We Tell Our Code (#seascale 2015 04-22)


Published on

We tell our code lies from development to deploy. The most common of these lies start with the simple act of launching a virtual machine. These lies are critical to our applications. Some of them protect applications from themselves and each other, some even improve performance. Some, however, decrease performance, and others create barriers to simply getting things done.

We lie about the systems, networks, storage, RAM, CPU and other resources our applications use, but how we tell those lies is critical to how the applications that depend on them perform. Joyent's Casey Bisson will explore the lies we tell our code and demonstrate examples of how they sometimes help and hurt us.

Slides as presented at Video from that meetup is on YouTube,

Published in: Software, Technology
  • Nice !! Download 100 % Free Ebooks, PPts, Study Notes, Novels, etc @
    Are you sure you want to  Yes  No
    Your message goes here
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website!
    Are you sure you want to  Yes  No
    Your message goes here

The Lies We Tell Our Code (#seascale 2015 04-22)

  1. 1. “the lies we tell our code”
  2. 2. Powering modern applications Your favorite code Container optimized infrastructure Your favorite tools
  3. 3. Our data center or yours Joyent Public Cloud Joyent Container Service. We run our customer’s mission critical applications on container native infrastructure. Private DataCenter SmartDataCenter is an on-premise, container run-time environment used by some of the world’s most recognizable companies.
  4. 4. Our data center or yours Joyent Public Cloud Joyent Container Service. We run our customer’s mission critical applications on container native infrastructure. Private DataCenter SmartDataCenter is an on-premise, container run-time environment used by some of the world’s most recognizable companies. …and open source too! Fork me, pull me:
  5. 5. Node.js enterprise support As the corporate steward of Node.js and one of the largest- scale production users, Joyent is uniquely equipped to deliver the highest level of enterprise support for this dynamic runtime. • Best practices guidance • Performance analysis • Core file analysis • Debugging support • Critical incident support
  6. 6. Docker container hosting The original container infrastructure company loves the new container packaging standard … • Portability From laptop to any public or private cloud • Productivity Faster code, test, and deploy • Devops for everyone Large community building tools for management, deployment, and scale
  7. 7. Proprietary & Confidential Information © 2015 Joyent, Inc ‹#›. The best place to run Docker containers, 
 making Ops simple and scalable. Triton Triton SecurityManagement Networking IntrospectionPerformance Utilization
  8. 8. breath for a moment
  9. 9. lying to our code is a practical choice
  10. 10. without moral consequence
  11. 11. without all consequence …but not
  12. 12. most importantly
  13. 13. most importantly never lie to yourself
  14. 14. The earliest common lie Virtual memory from
  15. 15. Virtual memory
 according to Poul-Henning Kamp Take Squid for instance, a 1975 program if I ever saw one: You tell it how much RAM it can use and how much disk it can use. It will then spend inordinate amounts of time keeping track of what HTTP objects are in RAM and which are on disk and it will move them forth and back depending on traffic patterns. Squid’s elaborate memory management…gets into fights with the kernel’s elaborate memory management, and like any civil war, that never gets anything done. from
  16. 16. Virtual memory
 according to Poul-Henning Kamp Varnish knows it is not running on the bare metal but under an operating system that provides a virtual-memory-based abstract machine. For example, Varnish does not ignore the fact that memory is virtual; it actively exploits it. A 300-GB backing store, memory mapped on a machine with no more than 16 GB of RAM, is quite typical. The user paid for 64 bits of address space, and I am not afraid to use it. from
  17. 17. vm.swappiness = 0
  18. 18. The harmless lie Hyperthreading from
  19. 19. Hyperthreading One physical core appears as two processors to the operating system, which can use each core to schedule two processes at once. It takes advantage of superscalar architecture in which multiple instructions operate on separate data in parallel. Hyper-threading can be properly utilized only with an OS specifically optimized for it. from
  20. 20. Faster, but not double the performance Hyperthreading from
  21. 21. The lie that built the cloud Hardware virtual machines from
  22. 22. HVM: call translation Say a virtual machine guest OS makes the call to flush the TLB (translation look-aside buffer) which is a physical component of a physical CPU. If the guest OS was allowed to clear the entire TLB on a physical processor, that would have negative performance effects for all the other VMs that were also sharing that same physical TLB. [Instead, the hypervisor must translate that call] so that only the section of the TLB that is relevant to that virtual machine is flushed. from
  23. 23. The lie that made VMware huge HVM: type 1 vs. type 2 from
  24. 24. Lies upon lies Paravirtualization from
  25. 25. HVM vs. clocksource… EC2 User: the kernel time will jump from 0 to 
 thousands of seconds. Kernel dev: for some reason it looks like the vcpu time info misses…without implementation details of the host code it is hard to say anything more. AWS: Ubuntu…uses the underlying hardware as a timesource, rather than sources native to the instance, leading to timestamps that are out of sync with the local instance time. from
  26. 26. HVM vs. CPU oversubscription An operating system requires synchronous progress on all its CPUs, and it might malfunction when it detects this requirement is not being met. For example, a watchdog timer might expect a response from its sibling vCPU within the specified time and would crash otherwise. When running these operating systems as a guest, ESXi must therefore maintain synchronous progress on the virtual CPUs. from
  27. 27. HVMs vs. network I/O Reality: interrupts are challenging in HVM with oversubscribed CPU. Consider these AWS network tuning recommendations: • Turn off tcp_slow_start_after_idle • Increased netdev_max_backlog from 1000 to 5000 • Maximize window size (rwnd, swnd, and cwnd) from reinvent-2013
  28. 28. HVMs vs. memory oversubscription [P]age sharing, ballooning, and compression are opportunistic techniques. They do not guarantee memory reclamation from VMs. For example, a VM may not have sharable content, the balloon driver may not be installed, or its memory pages may not yield good compression. Reclamation by swapping is a guaranteed method for reclaiming memory from VMs. from
  29. 29. HVM vs. performance Most successful AWS cluster deployments use more EC2 instances than they would the same number of physical nodes to compensate for the performance variability caused by shared, virtualized resources. Plan to have more EC2 instance based nodes than physical server nodes when estimating cluster size with respect to node count. from
  30. 30. Because lying about software is easier than lying about hardware OS-based virtualization from
  31. 31. OS-based virtualization Simple idea • The kernel is there to manage the relationship with hardware and isolate processes from each other • We’ve depended on secure memory protection, process isolation, privilege management in unix for a long time • Let’s leverage that and expand on it OS virt adds new requirements • Namespace lies (pid, uid, ipc, uts, net, mnt) • Polyinstantiation of resources • Virtualized network interfaces, etc Learn more about Linux, SmartOS
  32. 32. OS-based virtualization • Significantly reduced RAM requirements • Makes microservices possible • Shorter I/O chains • Kernel visibility across all processes • Coscheduled I/O and CPU tasks • Elastic use of memory and CPU across all containers • Allowing explicit resizing of containers (raising RAM, CPU, I/O limits) • Allowing bursting of containers (unused CPU cycles can claimed by whatever container wants them) • Allowing the kernel to use unused RAM as an FS cache across all containers • Greater tolerance of CPU oversubscription • Significantly higher workload density
  33. 33. Rendering from Bruce Irving Go container-native for Earth Day
  34. 34. OS-based virtualization: Linux Linux kernel support for namespaces is still very new. This note accompanying their introduction has proved prescient: “[T]he changes wrought by this work are subtle and wide ranging. Thus, it may happen that user namespaces have some as-yet unknown security issues that remain to be found and fixed in the future.” from
  35. 35. from
  36. 36. OS-based virtualization: Joyent • Kernel and facilities built for zones from the start • Process encapsulation separates processes, their data and the namespace • Processes cannot escape from zones. • Processes cannot observe other zones. • Processes cannot signal other zones. • Naming (such as user IDs or opening a port on an IP address) does not conflict with other zones • Zone processes have a privilege limit and no process in a zone ever has as much privilege as the global zone • Mature and tested: almost ten years in production at Joyent without incident • Coming up: filesystem and network virtualization contributions to container security
  37. 37. Playing charades: two syllables, sounds like… Syscall virtualization • The internet • Native Linux binaries • Linux syscall translation • SmartOS Kernel
  38. 38. Syscall virtualization • Branded zones provide a set of interposition points in the kernel that are only applied to processes executing in a branded zone. • These points are found in such paths as the syscall path, the process loading path, and the thread creation path. • At each of these points, a brand can choose to supplement or replace the standard behavior. from
  39. 39. The lie on which our massive media libraries were built Virtual block storage: RAID from
  40. 40. The lie that puts data in a separate cloud from compute Virtual block storage: SAN from and
  41. 41. SAN vs. app performance Riak's primary bottleneck will be disk and network I/O. [S]tandard EBS will incur too much latency and iowait. Riak's I/O pattern tends to operate on small blobs from many places on the disk, whereas EBS is best at bulk reads and writes. from
  42. 42. SAN vs. disaster [Some common solutions] force non-obvious single points of failure. [They are] a nice transition away from traditional storage, but at the end of the day it is just a different implementation of the same thing. SAN and Software Defined Storage are all single points of failure when used for virtual machine storage. from
  43. 43. More lies about where your data is Filesystem virtualization: links from 
 see also Busybox’s use of links,
  44. 44. The lie on which Docker containers are built Filesystem virtualization: copy-on-write from
  45. 45. Filesystem virtualization: AUFS ★ Works on top of other filesystems ★ File-based copy-on-write ★ Each layer is just a directory in the host filesystem; no user namespace mapping is applied ★ Original underlying filesystem for Docker containers ★ Read/write performance degrades with number of layers ★ Write performance degrades with filesize ★ In practice, dotCloud avoided these performance problems by adding secondary volumes to containers to store data separately from container layers See also and https://
  46. 46. True lies about filesystems and blockstores Filesystem virtualization: ZFS from
  47. 47. More lies for better performance Filesystem virtualization: ZFS hybrid pools from and http://
  48. 48. Filesystem virtualization: ZFS ★ Native block-based copy on write ★ No performance hit for CoW ★ Default thin provisioned filesystems backed by hybrid pools of real devices ★ Low provisioning cost ★ Native snapshots map to Docker layers ★ Native checksum validation used to detect device errors before the device reports them ★ Convenient, fast, and reliable by default ★ Native support for write-through SSD and big read caches to further improve performance
  49. 49. from
  50. 50. from
  51. 51. The lie that completes the cloud Network virtualization from (Wayback Machine)
  52. 52. Network virtualization: Weave A weave router captures Ethernet packets from its bridge-connected interface in promiscuous mode, using ‘pcap’. This typically excludes traffic between local containers, and between the host and local containers, all of which is routed straight over the bridge by the kernel. Captured packets are forwarded over UDP to weave router peers running on other hosts. On receipt of such a packet, a router injects the packet on its bridge interface using ‘pcap’ and/or forwards the packet to peers. from
  53. 53. The lie that completes the cloud Network virtualization: Crossbow from (Wayback Machine)
  54. 54. Network virtualization: Triton SDN • Extends Crossbow to add user-defined networks. • Every user gets a private layer 2 network with a unique IP. • All my containers have working interconnectivity, regardless of what physical hardware they’re on • …but your containers can’t see my containers. • When requested, containers also get a unique, publicly routable IP.
  55. 55. An exquisite collection of lies Docker from
  56. 56. Docker: Swarm • Aggregates any number of Docker Remote API endpoints and presents them as a single endpoint • Automatically distributes container workload among available APIs • Works in combination with Docker Compose to deploy and scale applications composed of multiple containers • Offers a direct path from building and testing on our laptops to deploying across a number of hosts • Downside: you pay for VMs, not containers
  57. 57. Docker: Triton • Exposes the entire data center as a single Docker Remote API endpoint • Automatically distributes container workload among available APIs • Works in combination with Docker Compose to deploy and scale applications composed of multiple containers (awaiting DOCKER-335 or compose/1317) • Offers a direct path from building and testing on our laptops to deploying across a number of hosts • You pay for containers, not VMs
  58. 58. breath for a moment
  59. 59. Lie about all the things • Containerize for better performance and workload density • Don't run containers in VMs, that's sad • Watch out for security issues • ...including at the filesystem level • Virtualize the network too, • give every container its own NICs and IPs • Don't stop lying at the edge of the compute node
  60. 60. Missy Elliot’s philosophy • Is it worth it? • Let me work it • I put my thing down, flip it, and reverse it • Get that cash • Ain't no shame • Do your thing • Just make sure you’re ahead of the game
  61. 61. Thank you
  62. 62. Remember Joyent for… • Proven container security Run containers securely on bare metal in multi-tenant environments • Bare metal container performance Eliminate the hardware hypervisor tax • Simplified container networking Each container has its own IP(s) in a user-defined network (SDN) • Simplified host management Eliminates Docker host proliferation • Hybrid: your data center or ours Private cloud, public cloud, hybrid cloud, and open source