This document provides an overview and agenda for a presentation on Nomad, an open source cluster scheduler created by HashiCorp. The presentation covers Nomad fundamentals like architecture, job configuration, and scheduling. It also demonstrates Nomad's capabilities at large scale by describing a "million container challenge" deployment of 1 million containers across 1,000 jobs and 5,000 hosts on Google Cloud Platform. The document promotes an upcoming HashiConf conference for further discussion.
Scaling Development Environments with DockerDocker, Inc.
This document discusses using Docker to create a scalable development environment. It outlines setting up containers for different development components like the build server, production servers, and tools. Templates are used to configure container dependencies and build processes. The goal is allowing developers to run all components locally for testing and to reproduce the production environment.
Building a Global-Scale Multi-Tenant Cloud Platform on AWS and Docker: Lesson...Felix Gessert
In this talk we share the lessons learned while building out the Baqend Cloud platform on AWS and Docker. Baqend’s AWS-hosted architecture consists of a caching CDN-Layer, global and local load balancing, a group of REST and Node.js servers and a database cluster with Redis and MongoDB. As customers have their own set of containerized REST and Node servers, we needed a cluster that on the one hand is horizontally scalable and on the other hand easily manageable and fault-tolerant from an operational perspective. Today there are at least 4 popular systems that claim to support this:
- Kubernetes
- Apache Mesos
- Docker Swarm
- AWS Elastic Container Service (ECS)
Thinking that ECS would certainly be the easiest option on AWS, we started building our cluster on it. We quickly came to realize that while ECS was astoundingly stable and easy to use there were inherent limitations that could not be worked around. An old Docker version, missing network isolation, no means of parameterizing task and forced memory constraints are major limitations of ECS we will talk about. Seeing the daunting operational overhead of running Kubernetes or Mesos in practice we turned to Docker’s native clustering solution Swarm. We will present how Swarm works with both Docker and AWS and highlight the advantages and downsides compared to Amazon’s ECS.
This document summarizes Mystery Machine, a tool that analyzes performance of complex systems from log files. It generates a causal model of the system from logs containing request IDs and timestamps. It then identifies critical paths that most impact performance and calculates slack to determine where improvements can be made without slowing the system. UberTrace is used to collect logs from all parts of the system to a central location for Mystery Machine to analyze end-to-end performance without requiring code instrumentation.
Building a Cloud Native Service - Docker Meetup Santa Clara (July 20, 2017)Yong Tang
In this talk, we share our experience of building up a cloud native service with Docker, Kubernetes, and CoreDNS. It is a customer-facing, multi-tenant, and globally available service that helps customers defending against various Internet attacks.
The global availability of the service is achieved through Anycast so that all customers only need to access one IP address across different regions. Deploying Anycast turns out to be a challenge because of the limitations on certain clouds. We overcome those limitations through containerization of different components with Docker.
We also share our experiences in container orchestration, container networking, load balancing, and service registration & discovery. We use a simplified architecture for container networking, and the service registration & discovery is done through CoreDNS. The overall design have helped our deployed service with improved elasticity, ease of use, and lowered maintenance cost.
Load Balancing Applications with NGINX in a CoreOS ClusterKevin Jones
The document discusses load balancing applications with NGINX in a CoreOS cluster. It provides an overview of using CoreOS, etcd, and fleet to deploy and manage containers across a cluster. Etcd is used for service discovery to track dynamic IP addresses and endpoints, while fleet is used as an application scheduler to deploy units and rebalance loads. NGINX can then be used as a software load balancer to distribute traffic to the backend services. The document demonstrates setting up this environment with CoreOS, etcd, fleet and NGINX to provide load balancing in a clustered deployment.
This document provides an overview and agenda for a presentation on Nomad, an open source cluster scheduler created by HashiCorp. The presentation covers Nomad fundamentals like architecture, job configuration, and scheduling. It also demonstrates Nomad's capabilities at large scale by describing a "million container challenge" deployment of 1 million containers across 1,000 jobs and 5,000 hosts on Google Cloud Platform. The document promotes an upcoming HashiConf conference for further discussion.
Scaling Development Environments with DockerDocker, Inc.
This document discusses using Docker to create a scalable development environment. It outlines setting up containers for different development components like the build server, production servers, and tools. Templates are used to configure container dependencies and build processes. The goal is allowing developers to run all components locally for testing and to reproduce the production environment.
Building a Global-Scale Multi-Tenant Cloud Platform on AWS and Docker: Lesson...Felix Gessert
In this talk we share the lessons learned while building out the Baqend Cloud platform on AWS and Docker. Baqend’s AWS-hosted architecture consists of a caching CDN-Layer, global and local load balancing, a group of REST and Node.js servers and a database cluster with Redis and MongoDB. As customers have their own set of containerized REST and Node servers, we needed a cluster that on the one hand is horizontally scalable and on the other hand easily manageable and fault-tolerant from an operational perspective. Today there are at least 4 popular systems that claim to support this:
- Kubernetes
- Apache Mesos
- Docker Swarm
- AWS Elastic Container Service (ECS)
Thinking that ECS would certainly be the easiest option on AWS, we started building our cluster on it. We quickly came to realize that while ECS was astoundingly stable and easy to use there were inherent limitations that could not be worked around. An old Docker version, missing network isolation, no means of parameterizing task and forced memory constraints are major limitations of ECS we will talk about. Seeing the daunting operational overhead of running Kubernetes or Mesos in practice we turned to Docker’s native clustering solution Swarm. We will present how Swarm works with both Docker and AWS and highlight the advantages and downsides compared to Amazon’s ECS.
This document summarizes Mystery Machine, a tool that analyzes performance of complex systems from log files. It generates a causal model of the system from logs containing request IDs and timestamps. It then identifies critical paths that most impact performance and calculates slack to determine where improvements can be made without slowing the system. UberTrace is used to collect logs from all parts of the system to a central location for Mystery Machine to analyze end-to-end performance without requiring code instrumentation.
Building a Cloud Native Service - Docker Meetup Santa Clara (July 20, 2017)Yong Tang
In this talk, we share our experience of building up a cloud native service with Docker, Kubernetes, and CoreDNS. It is a customer-facing, multi-tenant, and globally available service that helps customers defending against various Internet attacks.
The global availability of the service is achieved through Anycast so that all customers only need to access one IP address across different regions. Deploying Anycast turns out to be a challenge because of the limitations on certain clouds. We overcome those limitations through containerization of different components with Docker.
We also share our experiences in container orchestration, container networking, load balancing, and service registration & discovery. We use a simplified architecture for container networking, and the service registration & discovery is done through CoreDNS. The overall design have helped our deployed service with improved elasticity, ease of use, and lowered maintenance cost.
Load Balancing Applications with NGINX in a CoreOS ClusterKevin Jones
The document discusses load balancing applications with NGINX in a CoreOS cluster. It provides an overview of using CoreOS, etcd, and fleet to deploy and manage containers across a cluster. Etcd is used for service discovery to track dynamic IP addresses and endpoints, while fleet is used as an application scheduler to deploy units and rebalance loads. NGINX can then be used as a software load balancer to distribute traffic to the backend services. The document demonstrates setting up this environment with CoreOS, etcd, fleet and NGINX to provide load balancing in a clustered deployment.
Anatomy of the libvirt virtualization library
http://www.ibm.com/developerworks/library/l-libvirt/
libvirt
http://libvirt.org/index.html
Scheduling
http://docs.openstack.org/icehouse/config-reference/content/section_compute-scheduler.html
Openstack Zoning – Region/Availability Zone/Host Aggregate
https://kimizhang.wordpress.com/2013/08/26/openstack-zoning-regionavailability-zonehost-aggregate/
Availability Zones and Host Aggregates in OpenStack Compute (Nova)
http://blog.russellbryant.net/2013/05/21/availability-zones-and-host-aggregates-in-openstack-compute-nova/
An Introduction to Droplet Metadata
https://www.digitalocean.com/community/tutorials/an-introduction-to-droplet-metadata
HOW WE USE CLOUDINIT IN OPENSTACK HEAT
http://sdake.io/2013/03/03/how-we-use-cloudinit-in-openstack-heat/
How to inject file/meta/ssh key/root password/userdata/config drive to a VM during nova boot
https://kimizhang.wordpress.com/2014/03/18/how-to-inject-filemetassh-keyroot-passworduserdataconfig-drive-to-a-vm-during-nova-boot/
Cloud-init
https://cloudinit.readthedocs.org/en/latest/
Content caching is one of the most effective ways to dramatically improve the performance of a web site. In this webinar, we’ll deep-dive into NGINX’s caching abilities and investigate the architecture used, debugging techniques and advanced configuration. By the end of the webinar, you’ll be well equipped to configure NGINX to cache content exactly as you need.
View full webinar on demand at http://nginx.com/resources/webinars/content-caching-nginx/
The nova scheduler determines where to run virtual machine instances in OpenStack. It uses filters and weights to identify the best compute host from available information. An instance request is fulfilled by the scheduler selecting a host, informing the conductor, and having the compute node launch the instance. For large clouds, a horizontally scalable scheduler that uses flavor-based queues and avoids the database may improve performance. A Scheduler-as-a-Service project is also planned to provide a generic scheduler for other OpenStack components.
Wanting distributed volumes - Experiences with ceph-dockerEwout Prangsma
Slides of a docker meetup presentation in Cologne (april 28,2016)
The presentation talks about how to run ceph in docker containers and how to use the ceph filesystems for volumes of docker containers in need of persistent storage.
This document discusses implementing microservices using Docker Swarm and Consul. It recommends programming languages and tools for orchestration, databases, load balancing, monitoring, and other functions. Docker Swarm allows clustering Docker hosts into a pool of resources. Consul provides service discovery, configuration, and failure detection across multiple datacenters. Consul-Template listens for Consul updates and configures applications. Registrator automatically registers and deregisters Docker services with Consul. An example scenario shows how services scale across nodes with this architecture.
If you're running your container workloads on AWS EKS orchestration platform and you are trying to dynamically provision workload resources based on the current load, you might find yourself in a position where limitations and rules of node group scaling might feel a bit too rigid. This talk will focus on an interesting node lifecycle management solution from AWSlabs called Karpenter, which is an alternative approach to probably the most frequently used Cluster Autoscaler. Is this a better and more efficient way of allocating worker node resources? Would that get you around some of the node group constraints? The project hasn’t reached GA stage yet and still has to solve some goals from the roadmap, but I think it has a lot of potential. We will look into what the current release has to offer and how it is dealing with this challenge of improving efficient dynamic workload provisioning.
Distributed Logging Architecture in Container EraSATOSHI TAGOMORI
Distributed Logging Architecture in Container Era
The document discusses distributed logging architecture in the container era. It covers: 1) The difficulties of logging with microservices and containers due to their ephemeral and distributed nature, 2) The need to redesign logging to push logs from containers to destinations quickly without fixed addresses or mappings; 3) Common patterns for distributed logging architectures including source aggregation, destination aggregation, and scaling; and 4) A case study using Docker and Fluentd to implement source aggregation and scaling for logging. Open source solutions are important to keep the logging layer transparent, interoperable, and able to scale independently of applications and infrastructure.
NewSQL overview:
- History of RDBMs
- The reasons why NoSQL concept appeared
- Why NoSQL was not enough, the necessity of NewSQL
- Characteristics of NewSQL
- 7 DBs that belongs to NewSQL
- Overview Table with main properties
Uploading the presentation given at the OpenStack Summit, Austin in April, 2016. The video link is here ,
https://www.openstack.org/videos/video/multi-tenancy-for-docker-containers-with-keystone-and-adding-quota-limits
Nova compute and controller services provide virtual machines and coordinate nova services in OpenStack. Nova compute runs on each node and interacts with the hypervisor like KVM to launch VMs. The nova controller runs most nova services including the scheduler, which dispatches VM requests to nodes based on filters. Key components include the nova API, compute, and conductor services. Compute resources must be designed considering the number of processors, memory allocation, storage, resource pools, and over-commit ratios to optimize VM deployment in OpenStack.
This document provides an overview of Kubernetes including:
- Kubernetes is an open source system for managing containerized applications and services across clusters of hosts. It provides tools to deploy, maintain, and scale applications.
- Kubernetes objects include pods, services, deployments, jobs, and others to define application components and how they relate.
- The Kubernetes architecture consists of a control plane running on the master including the API server, scheduler and controller manager. Nodes run the kubelet and kube-proxy to manage pods and services.
- Kubernetes can be deployed on AWS using tools like CloudFormation templates to automate cluster creation and management for high availability and scalability.
Docker Engine 1.12 can be rightly called ” A Next Generation Docker Clustering & Distributed System”. Though Docker Engine 1.12 Final Release is around corner but the recent RC3 brings lots of improvements and exciting features. One of the major highlight of this release is Docker Swarm Mode which provides powerful yet optional ability to create coordinated groups of decentralized Docker Engines. Swarm Mode combines your engine in swarms of any scale. It’s self-organizing and self-healing. It enables infrastructure-agnostic topology.The newer version democratizes orchestration with out-of-box capabilities for multi-container on multi-host app deployments.
The document discusses Docker 1.5 and its new features including IPv6 support, read-only containers, Docker stats, and the Docker image specification. IPv6 can be enabled by running Docker with the --ipv6 flag. Read-only containers restrict writes to containers. Docker stats provides live metrics for containers. The Docker image specification defines the format for layered image files and metadata.
Fluentd is an open source data collector that allows for unified logging and data collection. It can be used to collect and parse logs from multiple sources like applications and servers running on multiple hosts. Fluentd works with Docker to provide a logging driver that routes container output to Fluentd. This allows Fluentd to collect logs from Docker containers and structure the data as JSON. Fluentd then reliably forwards the logs to a destination like Elasticsearch for storage and analysis. The document demonstrates how to set up Fluentd and Elasticsearch Docker containers to collect logs produced by other application containers running on the host.
Single tenant software to multi-tenant SaaS using K8SCloudLinux
This document discusses how Kubernetes can be used to convert single-tenant software applications into multi-tenant SaaS applications. Key points include:
1) Kubernetes can orchestrate each tenant as a separate pod or set of pods, providing isolation, easy scalability, and the ability to customize deployments for each tenant.
2) This approach simplifies many challenges of traditional SaaS like customer management, billing integration, high availability, upgrades and rollbacks by leveraging Kubernetes features.
3) An initial test project converted an existing PHP/MySQL billing application for 10,000+ companies into a multi-tenant SaaS deployment using Kubernetes, requiring under 40 hours of development.
Apache BookKeeper State Store: A Durable Key-Value Store - Pulsar Summit NA 2021StreamNative
Apache Pulsar is used for various streaming use cases. There is a strong requirement for storing checkpoints while processing the stream in Pulsar Functions, so that in case of any interruption stream processing engine could go back to the last checkpoint.
Pulsar uses Zookeeper not only for leader elections or service discovery like critical use cases but also for storing various metadata which puts unnecessary load on zookeeper which hampers mission critical use of Zookeeper.
A durable key value store based off of the Apache Pulsar ecosystem addresses the above mentioned use cases nicely.
This talk focuses on taking existing Apache Bookkeeper Table Service/State store implementation and taking it to production. Furthermore this talk also touches upon contributing all the features, bug fixes, tools and other improvements back to open source.
Docker Service Registration and Discoverym_richardson
This document discusses service registration and discovery with Consul. It begins with an overview of service registration and discovery and how existing approaches like DNS can struggle in Docker environments. Consul is presented as a tool that stores information about services and supports service discovery. Registrator is introduced as a tool that automatically registers and deregisters Docker services with Consul. The document demonstrates how to run Consul and Registrator as Docker containers and have Registrator register container services with Consul. Finally, Consul Template is discussed as a tool to query services registered with Consul and apply configuration templates.
How secure are your Terraform sensitive values?Marko Bevc
This talk will focus on how to securely manage sensitive values, such as secrets, passwords and keys, early on (shift left) in Terraform code. We will explore options how to tackle this and avoid the most common pitfalls we have observed at The Scale Factory while working with our clients. Using available mechanisms in the latest Terraform release I will also demo how we can better handle sensitive values in our infrastructure definitions.
MongoDB World 2019: Packing Up Your Data and Moving to MongoDB AtlasMongoDB
Moving to a new home is daunting. Packing up all your things, getting a vehicle to move it all, unpacking it, updating your mailing address, and making sure you did not leave anything behind. Well, the move to MongoDB Atlas is similar, but all the logistics are already figured out for you by MongoDB.
USENIX LISA15: How TubeMogul Handles over One Trillion HTTP Requests a MonthNicolas Brousse
TubeMogul grew from few servers to over two thousands servers and handling over one trillion http requests a month, processed in less than 50ms each. To keep up with the fast growth, the SRE team had to implement an efficient Continuous Delivery infrastructure that allowed to do over 10,000 puppet deployment and 8,500 application deployment in 2014. In this presentation, we will cover the nuts and bolts of the TubeMogul operations engineering team and how they overcome challenges.
Anatomy of the libvirt virtualization library
http://www.ibm.com/developerworks/library/l-libvirt/
libvirt
http://libvirt.org/index.html
Scheduling
http://docs.openstack.org/icehouse/config-reference/content/section_compute-scheduler.html
Openstack Zoning – Region/Availability Zone/Host Aggregate
https://kimizhang.wordpress.com/2013/08/26/openstack-zoning-regionavailability-zonehost-aggregate/
Availability Zones and Host Aggregates in OpenStack Compute (Nova)
http://blog.russellbryant.net/2013/05/21/availability-zones-and-host-aggregates-in-openstack-compute-nova/
An Introduction to Droplet Metadata
https://www.digitalocean.com/community/tutorials/an-introduction-to-droplet-metadata
HOW WE USE CLOUDINIT IN OPENSTACK HEAT
http://sdake.io/2013/03/03/how-we-use-cloudinit-in-openstack-heat/
How to inject file/meta/ssh key/root password/userdata/config drive to a VM during nova boot
https://kimizhang.wordpress.com/2014/03/18/how-to-inject-filemetassh-keyroot-passworduserdataconfig-drive-to-a-vm-during-nova-boot/
Cloud-init
https://cloudinit.readthedocs.org/en/latest/
Content caching is one of the most effective ways to dramatically improve the performance of a web site. In this webinar, we’ll deep-dive into NGINX’s caching abilities and investigate the architecture used, debugging techniques and advanced configuration. By the end of the webinar, you’ll be well equipped to configure NGINX to cache content exactly as you need.
View full webinar on demand at http://nginx.com/resources/webinars/content-caching-nginx/
The nova scheduler determines where to run virtual machine instances in OpenStack. It uses filters and weights to identify the best compute host from available information. An instance request is fulfilled by the scheduler selecting a host, informing the conductor, and having the compute node launch the instance. For large clouds, a horizontally scalable scheduler that uses flavor-based queues and avoids the database may improve performance. A Scheduler-as-a-Service project is also planned to provide a generic scheduler for other OpenStack components.
Wanting distributed volumes - Experiences with ceph-dockerEwout Prangsma
Slides of a docker meetup presentation in Cologne (april 28,2016)
The presentation talks about how to run ceph in docker containers and how to use the ceph filesystems for volumes of docker containers in need of persistent storage.
This document discusses implementing microservices using Docker Swarm and Consul. It recommends programming languages and tools for orchestration, databases, load balancing, monitoring, and other functions. Docker Swarm allows clustering Docker hosts into a pool of resources. Consul provides service discovery, configuration, and failure detection across multiple datacenters. Consul-Template listens for Consul updates and configures applications. Registrator automatically registers and deregisters Docker services with Consul. An example scenario shows how services scale across nodes with this architecture.
If you're running your container workloads on AWS EKS orchestration platform and you are trying to dynamically provision workload resources based on the current load, you might find yourself in a position where limitations and rules of node group scaling might feel a bit too rigid. This talk will focus on an interesting node lifecycle management solution from AWSlabs called Karpenter, which is an alternative approach to probably the most frequently used Cluster Autoscaler. Is this a better and more efficient way of allocating worker node resources? Would that get you around some of the node group constraints? The project hasn’t reached GA stage yet and still has to solve some goals from the roadmap, but I think it has a lot of potential. We will look into what the current release has to offer and how it is dealing with this challenge of improving efficient dynamic workload provisioning.
Distributed Logging Architecture in Container EraSATOSHI TAGOMORI
Distributed Logging Architecture in Container Era
The document discusses distributed logging architecture in the container era. It covers: 1) The difficulties of logging with microservices and containers due to their ephemeral and distributed nature, 2) The need to redesign logging to push logs from containers to destinations quickly without fixed addresses or mappings; 3) Common patterns for distributed logging architectures including source aggregation, destination aggregation, and scaling; and 4) A case study using Docker and Fluentd to implement source aggregation and scaling for logging. Open source solutions are important to keep the logging layer transparent, interoperable, and able to scale independently of applications and infrastructure.
NewSQL overview:
- History of RDBMs
- The reasons why NoSQL concept appeared
- Why NoSQL was not enough, the necessity of NewSQL
- Characteristics of NewSQL
- 7 DBs that belongs to NewSQL
- Overview Table with main properties
Uploading the presentation given at the OpenStack Summit, Austin in April, 2016. The video link is here ,
https://www.openstack.org/videos/video/multi-tenancy-for-docker-containers-with-keystone-and-adding-quota-limits
Nova compute and controller services provide virtual machines and coordinate nova services in OpenStack. Nova compute runs on each node and interacts with the hypervisor like KVM to launch VMs. The nova controller runs most nova services including the scheduler, which dispatches VM requests to nodes based on filters. Key components include the nova API, compute, and conductor services. Compute resources must be designed considering the number of processors, memory allocation, storage, resource pools, and over-commit ratios to optimize VM deployment in OpenStack.
This document provides an overview of Kubernetes including:
- Kubernetes is an open source system for managing containerized applications and services across clusters of hosts. It provides tools to deploy, maintain, and scale applications.
- Kubernetes objects include pods, services, deployments, jobs, and others to define application components and how they relate.
- The Kubernetes architecture consists of a control plane running on the master including the API server, scheduler and controller manager. Nodes run the kubelet and kube-proxy to manage pods and services.
- Kubernetes can be deployed on AWS using tools like CloudFormation templates to automate cluster creation and management for high availability and scalability.
Docker Engine 1.12 can be rightly called ” A Next Generation Docker Clustering & Distributed System”. Though Docker Engine 1.12 Final Release is around corner but the recent RC3 brings lots of improvements and exciting features. One of the major highlight of this release is Docker Swarm Mode which provides powerful yet optional ability to create coordinated groups of decentralized Docker Engines. Swarm Mode combines your engine in swarms of any scale. It’s self-organizing and self-healing. It enables infrastructure-agnostic topology.The newer version democratizes orchestration with out-of-box capabilities for multi-container on multi-host app deployments.
The document discusses Docker 1.5 and its new features including IPv6 support, read-only containers, Docker stats, and the Docker image specification. IPv6 can be enabled by running Docker with the --ipv6 flag. Read-only containers restrict writes to containers. Docker stats provides live metrics for containers. The Docker image specification defines the format for layered image files and metadata.
Fluentd is an open source data collector that allows for unified logging and data collection. It can be used to collect and parse logs from multiple sources like applications and servers running on multiple hosts. Fluentd works with Docker to provide a logging driver that routes container output to Fluentd. This allows Fluentd to collect logs from Docker containers and structure the data as JSON. Fluentd then reliably forwards the logs to a destination like Elasticsearch for storage and analysis. The document demonstrates how to set up Fluentd and Elasticsearch Docker containers to collect logs produced by other application containers running on the host.
Single tenant software to multi-tenant SaaS using K8SCloudLinux
This document discusses how Kubernetes can be used to convert single-tenant software applications into multi-tenant SaaS applications. Key points include:
1) Kubernetes can orchestrate each tenant as a separate pod or set of pods, providing isolation, easy scalability, and the ability to customize deployments for each tenant.
2) This approach simplifies many challenges of traditional SaaS like customer management, billing integration, high availability, upgrades and rollbacks by leveraging Kubernetes features.
3) An initial test project converted an existing PHP/MySQL billing application for 10,000+ companies into a multi-tenant SaaS deployment using Kubernetes, requiring under 40 hours of development.
Apache BookKeeper State Store: A Durable Key-Value Store - Pulsar Summit NA 2021StreamNative
Apache Pulsar is used for various streaming use cases. There is a strong requirement for storing checkpoints while processing the stream in Pulsar Functions, so that in case of any interruption stream processing engine could go back to the last checkpoint.
Pulsar uses Zookeeper not only for leader elections or service discovery like critical use cases but also for storing various metadata which puts unnecessary load on zookeeper which hampers mission critical use of Zookeeper.
A durable key value store based off of the Apache Pulsar ecosystem addresses the above mentioned use cases nicely.
This talk focuses on taking existing Apache Bookkeeper Table Service/State store implementation and taking it to production. Furthermore this talk also touches upon contributing all the features, bug fixes, tools and other improvements back to open source.
Docker Service Registration and Discoverym_richardson
This document discusses service registration and discovery with Consul. It begins with an overview of service registration and discovery and how existing approaches like DNS can struggle in Docker environments. Consul is presented as a tool that stores information about services and supports service discovery. Registrator is introduced as a tool that automatically registers and deregisters Docker services with Consul. The document demonstrates how to run Consul and Registrator as Docker containers and have Registrator register container services with Consul. Finally, Consul Template is discussed as a tool to query services registered with Consul and apply configuration templates.
How secure are your Terraform sensitive values?Marko Bevc
This talk will focus on how to securely manage sensitive values, such as secrets, passwords and keys, early on (shift left) in Terraform code. We will explore options how to tackle this and avoid the most common pitfalls we have observed at The Scale Factory while working with our clients. Using available mechanisms in the latest Terraform release I will also demo how we can better handle sensitive values in our infrastructure definitions.
MongoDB World 2019: Packing Up Your Data and Moving to MongoDB AtlasMongoDB
Moving to a new home is daunting. Packing up all your things, getting a vehicle to move it all, unpacking it, updating your mailing address, and making sure you did not leave anything behind. Well, the move to MongoDB Atlas is similar, but all the logistics are already figured out for you by MongoDB.
USENIX LISA15: How TubeMogul Handles over One Trillion HTTP Requests a MonthNicolas Brousse
TubeMogul grew from few servers to over two thousands servers and handling over one trillion http requests a month, processed in less than 50ms each. To keep up with the fast growth, the SRE team had to implement an efficient Continuous Delivery infrastructure that allowed to do over 10,000 puppet deployment and 8,500 application deployment in 2014. In this presentation, we will cover the nuts and bolts of the TubeMogul operations engineering team and how they overcome challenges.
This document summarizes a summer training seminar on BigData Hadoop that was attended. The training was provided by LinuxWorld Informatics Pvt Ltd, which offers open source and commercial training programs. The attendee learned about Hadoop, MapReduce, single and multi-node clusters, Docker, and Ansible. Big data challenges related to volume, variety, velocity, and veracity of data were also covered. Hadoop and its core components HDFS and MapReduce were explained as solutions for storing and processing large datasets in a distributed manner across commodity hardware. Docker containers were introduced as a lightweight alternative to virtual machines.
10,000 microservices are generated each month using JHipster!
During this in-depth session by the two JHipster lead developers, we’ll detail:
How to develop and deploy microservices easily
Scalability and failover of microservices
The JHipster Registry for scaling, configuring and monitoring microservices
Common architecture patterns and pitfalls
[WSO2Con Asia 2018] Architecting for Container-native EnvironmentsWSO2
Cloud native applications take advantage of cloud characteristics by using microservices architectures and containers. Microservices allow each service to have a single focus, be loosely coupled, lightweight, and highly scalable. Containers enable fast, immutable deployments and optimize resource usage. Orchestration tools like Kubernetes manage containers and provide additional capabilities like networking and scaling. Observability tools provide monitoring, logging, and distributed tracing to gain insights into application performance and issues.
What is Google Cloud Platform - GDG DevFest 18 DepokImre Nagi
This document provides an overview of Google Cloud Platform (GCP) services presented by Imre Nagi. It discusses:
1. What cloud computing is and how GCP provides infrastructure like virtual machines, networking, and storage in Google's data centers while handling scaling, migrations, and maintenance.
2. The main GCP services including Compute Engine, Kubernetes Engine, App Engine, and Cloud Functions for deploying applications, as well as storage, database, analytics, and machine learning services.
3. Options for deploying applications to GCP including using Compute Engine virtual machines, containers on Kubernetes Engine, or serverless functions on Cloud Functions. It notes advantages of managed services like App Engine over unmanaged infrastructure.
Pull, push, clone, it is all in your daily workflow. But what if this wasn't your source code or your container, but the state of your whole computer? Push your production database over to another machine? No problem!
This talk shows how you can use Dotmesh with LinuxKit to work with persistent data on your server as simply as you work with git. This workflow helps unleash new ways of working with servers and data. Immutable infrastructure from LinuxKit meets controlled and manageable data storage from Dotmesh. Combining these two open source projects allows new possibilities in how to manage your infrastructure.
Slides from the August 2021 St. Louis Big Data IDEA meeting from Sam Portillo. The presentation covers AWS EMR including comparisons to other similar projects and lessons learned. A recording is available in the comments for the meeting.
The document summarizes a meeting of the Accra MongoDB User Group held on November 10th, 2012. It provides information about MongoDB and 10gen, the company that develops MongoDB. It discusses 10gen's founders, management, offices, investors, and customer portfolio. It also summarizes why users should join the MongoDB User Group and covers topics from the meeting including MongoDB operations, what's new in version 2.2, aggregation framework, TTL collections, fragmentation, data center awareness, and developing an application to find nearby restaurants serving fufu using MongoDB.
Dimension Data Cloud Business Unit - Solution OfferingRifaHaryadi
Dimension Data - Cloud Business Unit Solution Offering. This presentation will take you through Dimension Data Solution Offering and Roadmap to the Future of Cloud Computing. Dimension Data Cloud Computing Solution are fully control by Manage Cloud Platform - Dimension Data Propretiary Orchestration and Automation Tools
MongoDB Stitch is a serverless platform designed to help you easily and securely build an application on top of MongoDB Atlas. It lets developers focus on building applications rather than on managing data manipulation code, service integration, or backend infrastructure. MongoDB Stitch also makes it simple to respond to backend changes immediately, allowing you to simplify client side code and build complex flows more easily. This talk will cover ways that MongoDB Stitch helps you respond to changes in your database and take your applications to the next level.
This document provides an introduction to Azure and describes several of its core services. It outlines compute, data, networking and other services available on Azure. Specifically it discusses virtual machines, websites, SQL databases, storage tables and blobs, import/export of large data sets, file services, virtual networks, ExpressRoute for private connections, Traffic Manager for routing, Automation for managing resources via PowerShell runbooks, API Management, backup services, messaging queues, Service Bus Relay for cross-firewall communication, Scheduler for scheduling jobs, caching with Cache service, Content Delivery Network for caching blobs globally, and HDInsight for Hadoop clusters.
Keep Your Cache Always Fresh With Debezium! With Gunnar Morling | Current 2022HostedbyConfluent
The saying goes that there are only two hard things in Computer Science: cache invalidation, and naming things. Well, turns out the first one is solved actually ;)
Join us for this session to learn how to keep read views of your data in distributed caches close to your users, always kept in sync with your primary data stores change data capture. You will learn how to
- Implement a low-latency data pipeline for cache updates based on Debezium, Apache Kafka, and Infinispan
- Create denormalized views of your data using Kafka Streams and make them accessible via plain key look-ups from a cache cluster close by
- Propagate updates between cache clusters using cross-site replication
We'll also touch on some advanced concepts, such as detecting and rejecting writes to the system of record which are derived from outdated cached state, and show in a demo how all the pieces come together, of course connected via Apache Kafka.
"NoSQL on the move" by Glynn Bird
Mobile-first app web development is a solved problem, but how can you websites and apps the continue to work with little or internet connectivity? Discover how Offline-first development allows apps to present an "always on" experience for their user
Similar to Devoxx 2016 talk: Going Global with Nomad and Google Cloud Platform (20)
Instagram has become one of the most popular social media platforms, allowing people to share photos, videos, and stories with their followers. Sometimes, though, you might want to view someone's story without them knowing.
Gen Z and the marketplaces - let's translate their needsLaura Szabó
The product workshop focused on exploring the requirements of Generation Z in relation to marketplace dynamics. We delved into their specific needs, examined the specifics in their shopping preferences, and analyzed their preferred methods for accessing information and making purchases within a marketplace. Through the study of real-life cases , we tried to gain valuable insights into enhancing the marketplace experience for Generation Z.
The workshop was held on the DMA Conference in Vienna June 2024.
APNIC Foundation, presented by Ellisha Heppner at the PNG DNS Forum 2024APNIC
Ellisha Heppner, Grant Management Lead, presented an update on APNIC Foundation to the PNG DNS Forum held from 6 to 10 May, 2024 in Port Moresby, Papua New Guinea.
3. @NakedN3rd#Devoxx #GoingGlobal
Latency costs
Roundtrip from Asia to Europe adds
500ms latency to every request.
In a Google experiment in 2006, increasing page load time from
0.4s to 0.9s resulted in 20% lower traffic and revenue.
12. @NakedN3rd#Devoxx #GoingGlobal
Helloworld
● A simple demo webapp.
● Written in Go, no OS dependencies.
● 12 factor.
● /hello Says hello and counts how many times.
● /health endpoint returns health
● /fs/... serves filesytem
https://github.com/bastiaanb/helloworld
16. @NakedN3rd#Devoxx #GoingGlobal
Nomad Datacenters
“Nomad models infrastructure as regions and datacenters. Regions may
contain multiple datacenters. Servers are assigned to regions and manage all
state for the region and make scheduling decisions within that region.
Requests that are made between regions are forwarded to the appropriate
servers.As an example, you may have a US region with the useast1 and
uswest1 datacenters, connected to the EU region with the eufr1 and
euuk1 datacenters.”
17. @NakedN3rd#Devoxx #GoingGlobal
Consul Datacenters
“we define a datacenter to be a networking environment that is
private, low latency, and high bandwidth.This excludes
communication that would traverse the public internet, but for our
purposes multiple availability zones within a single EC2
region would be considered part of a single datacenter.”
Our goal: Creating a global datacenter that is highly available, super scalable and responsive.
With ‘highly available’ we mean: can widthstand loss / failure of single processes, but also compute nodes, complete availability zones or even regions.
With ‘super scalable’ we mean that we can scale up capacity multiple orders of magnitude without having to change the architecture.
With ‘responsive’ we mean that we want requests to be served by a server that is ‘near’ the client. Ergo US users get serviced by a US datacenter, European users by a European one.