In this talk we share the lessons learned while building out the Baqend Cloud platform on AWS and Docker. Baqend’s AWS-hosted architecture consists of a caching CDN-Layer, global and local load balancing, a group of REST and Node.js servers and a database cluster with Redis and MongoDB. As customers have their own set of containerized REST and Node servers, we needed a cluster that on the one hand is horizontally scalable and on the other hand easily manageable and fault-tolerant from an operational perspective. Today there are at least 4 popular systems that claim to support this:
- Kubernetes
- Apache Mesos
- Docker Swarm
- AWS Elastic Container Service (ECS)
Thinking that ECS would certainly be the easiest option on AWS, we started building our cluster on it. We quickly came to realize that while ECS was astoundingly stable and easy to use there were inherent limitations that could not be worked around. An old Docker version, missing network isolation, no means of parameterizing task and forced memory constraints are major limitations of ECS we will talk about. Seeing the daunting operational overhead of running Kubernetes or Mesos in practice we turned to Docker’s native clustering solution Swarm. We will present how Swarm works with both Docker and AWS and highlight the advantages and downsides compared to Amazon’s ECS.
This talk demonstrates how to develop single page apps with the new angular2 framework and TypeScript. We show the new concepts of angular2 not only in theory, but using a real application. To this end, we develop a real-time angular2 website, for users to to ask and upvote questions during a talk identified by a hash tag. The session chair can ask the most popular questions at the end of the talk.
Dieser Vortrag zeigt, wie man mit dem neuen Angular2 Framework und TypeScript schnelle Single Page Apps entwickelt. Die neuen Konzepte von Angular2 zeigen wir dabei nicht nur in der Theorie, sondern ganz praktisch. Dazu entwickeln wir live eine Real-Time Angular2 App, mit der Zuhörer während eines Vortrags – identifiziert durch einen Hash-Tag - Fragen stellen und gegenseitig upvoten können. Der Session Chair kann so am Ende eines Vortrags die bestbewertesten Fragen an den Speaker stellen.
Building a multi-tenant cloud service from legacy code with Docker containersaslomibm
A reusable architectural pattern to migrate legacy application to a cloud service. The architecture pattern can be used by other legacy applications that need to migrate to cloud. The architecture was validated by the Beta release of the IBM Bluemix Workflow service and Docker containers were key capability to manage separate workflow engines for each tenant combined with cloud database for persistence layer and a content-based routing.
Uploading the presentation given at the OpenStack Summit, Austin in April, 2016. The video link is here ,
https://www.openstack.org/videos/video/multi-tenancy-for-docker-containers-with-keystone-and-adding-quota-limits
Lessons learned from writing over 300,000 lines of infrastructure codeYevgeniy Brikman
This talk is a concise masterclass on how to write infrastructure code. I share key lessons from the “Infrastructure Cookbook” we developed at Gruntwork while creating and maintaining a library of over 300,000 lines of infrastructure code that’s used in production by hundreds of companies. Come and hear our war stories, laugh about all the mistakes we’ve made along the way, and learn what Terraform, Packer, Docker, and Go look like in the wild.
Serverless and Servicefull Applications - Where Microservices complements Ser...Red Hat Developers
Serverless is a misnomer, your future cloud native applications will consist of both microservices and functions, wrapped as Linux containers, but in many cases where you the developer ignore the operational aspects of managing that infrastructure. In this session we get started using Function as a Service (FaaS) engine with Apache OpenWhisk deployed on Kubernetes and OpenShift. With Kubernetes/OpenShift being de-facto platform for Cloud Native Java Applications, we will explore on to see how to make Cloud Native Java Applications a.k.a Microservices can complement the serverless Functions. This technical intensive session will open up one of the possible serverless web application architecture where we deploy an API Gateway into the FaaS platform to draw the curtains up to see the microservices talking to the serverless functions. This session ends with us seeing how Event Sinks and Event Sources map in the Serverless World.
"[WORKSHOP] K8S for developers", Denis RomanukFwdays
"It seems that perfection is attained, not when there is nothing more to add, but when there is nothing more to take away." - Antoine de Saint Exupéry
We can talk a lot about Kubernetes. But does the whole info is really need for just begining? Let's take away everything, except the really needed, and show a way for ones, who still uses only docker.
Let's consider:
Why do developers need Kubernetes and what does it looks like?
An attempt to automate ops, or why does it was a reinvention of Kubernetes
What does kube consists of?
How to get a Kubernetes?
From docker-compose to c
What is a Helm and why it's difficult without one?
What is "local development on Kubernetes"?
CRD & Operators
This talk demonstrates how to develop single page apps with the new angular2 framework and TypeScript. We show the new concepts of angular2 not only in theory, but using a real application. To this end, we develop a real-time angular2 website, for users to to ask and upvote questions during a talk identified by a hash tag. The session chair can ask the most popular questions at the end of the talk.
Dieser Vortrag zeigt, wie man mit dem neuen Angular2 Framework und TypeScript schnelle Single Page Apps entwickelt. Die neuen Konzepte von Angular2 zeigen wir dabei nicht nur in der Theorie, sondern ganz praktisch. Dazu entwickeln wir live eine Real-Time Angular2 App, mit der Zuhörer während eines Vortrags – identifiziert durch einen Hash-Tag - Fragen stellen und gegenseitig upvoten können. Der Session Chair kann so am Ende eines Vortrags die bestbewertesten Fragen an den Speaker stellen.
Building a multi-tenant cloud service from legacy code with Docker containersaslomibm
A reusable architectural pattern to migrate legacy application to a cloud service. The architecture pattern can be used by other legacy applications that need to migrate to cloud. The architecture was validated by the Beta release of the IBM Bluemix Workflow service and Docker containers were key capability to manage separate workflow engines for each tenant combined with cloud database for persistence layer and a content-based routing.
Uploading the presentation given at the OpenStack Summit, Austin in April, 2016. The video link is here ,
https://www.openstack.org/videos/video/multi-tenancy-for-docker-containers-with-keystone-and-adding-quota-limits
Lessons learned from writing over 300,000 lines of infrastructure codeYevgeniy Brikman
This talk is a concise masterclass on how to write infrastructure code. I share key lessons from the “Infrastructure Cookbook” we developed at Gruntwork while creating and maintaining a library of over 300,000 lines of infrastructure code that’s used in production by hundreds of companies. Come and hear our war stories, laugh about all the mistakes we’ve made along the way, and learn what Terraform, Packer, Docker, and Go look like in the wild.
Serverless and Servicefull Applications - Where Microservices complements Ser...Red Hat Developers
Serverless is a misnomer, your future cloud native applications will consist of both microservices and functions, wrapped as Linux containers, but in many cases where you the developer ignore the operational aspects of managing that infrastructure. In this session we get started using Function as a Service (FaaS) engine with Apache OpenWhisk deployed on Kubernetes and OpenShift. With Kubernetes/OpenShift being de-facto platform for Cloud Native Java Applications, we will explore on to see how to make Cloud Native Java Applications a.k.a Microservices can complement the serverless Functions. This technical intensive session will open up one of the possible serverless web application architecture where we deploy an API Gateway into the FaaS platform to draw the curtains up to see the microservices talking to the serverless functions. This session ends with us seeing how Event Sinks and Event Sources map in the Serverless World.
"[WORKSHOP] K8S for developers", Denis RomanukFwdays
"It seems that perfection is attained, not when there is nothing more to add, but when there is nothing more to take away." - Antoine de Saint Exupéry
We can talk a lot about Kubernetes. But does the whole info is really need for just begining? Let's take away everything, except the really needed, and show a way for ones, who still uses only docker.
Let's consider:
Why do developers need Kubernetes and what does it looks like?
An attempt to automate ops, or why does it was a reinvention of Kubernetes
What does kube consists of?
How to get a Kubernetes?
From docker-compose to c
What is a Helm and why it's difficult without one?
What is "local development on Kubernetes"?
CRD & Operators
NATS: Simple, Secure and Scalable Messaging For the Cloud Native Erawallyqs
The majority of middleware and messaging systems in use were built in a time that did not have the concept of scale and real-time data that developers operate in today.
With the rise of Cloud Native and Microservices architectures as a design principle and the emphasis on simplicity, speed, and flexibility that come with it, developers need a messaging protocol to match.
Enter NATS. NATS is a remarkably lightweight messaging protocol, and extremely flexible and resilient. It is just a few MB in size, and can scale to publish tens of millions of message from a single server.
This is a presentation I held at "DevOps and Security" -meetup on 5th of April 2016 at RedHat.
Source is available at: https://github.com/jerryjj/devsec_050416
Microservices @ Work - A Practice Report of Developing MicroservicesQAware GmbH
Cloud Native Night October 2016, Mainz: Talk by Simon Bäumler (Technical Chief Designer at QAware).
Join our Meetup: www.meetup.com/cloud-native-night
Abstract: This talk takes a practice oriented approach to examine microservice oriented architecture. It will show two real systems, one build from scratch in a microservice architecture, the other migrated from a monolithic system to a microservice architecture.
With the example of these two systems the pittfalls, advantages and lessons learned using microservice oriented architectures will be discussed.
While both systems use the java stack, including spring boot and spring cloud many topics will be kept general and will be of interest for all developers.
OpenEBS is a container-native open source containerized storage project for containers – tightly integrated into Kubernetes.
You can find the full presentation here: https://www.facebook.com/VMTNcommunity/videos/2008142932762386/
Sf bay area Kubernetes meetup dec8 2016 - deployment modelsPeter Ss
I talk about deploying complex, multi-layer applications in Kuberentes.
I describe how Kubernetes AppController project (https://github.com/Mirantis/k8s-AppController) can be leveraged to enhance such deployments
Kubernetes is a container orchestration platform that provides a mechanism to manage the resources of containers in the cluster. That mechanism is known as "Requests and Limits".
Requests and limits play a key role not only in resource management but also in applications stability, capacity planning, scheduling the resources (i.e., on which node the pod will be running).
In this session we will cover:
- A quick review of Containers, Docker, and Kubernetes.
- Containers resource management in Kubernetes.
- Containers resource types in Kubernetes.
- 3 different ways to set requests and limits.
- The difference between capacity and allocatable resources.
- Tips and recap.
Slides used for Orchestructure May 2018 workshop.
Labs:
https://github.com/mrbobbytables/k8s-intro-tutorials
Event Information:
https://www.meetup.com/orchestructure/events/250189685/
Presented at AI NEXTCon Seattle 1/17-20, 2018
http://aisea18.xnextcon.com
join our free online AI group with 50,000+ tech engineers to learn and practice AI technology, including: latest AI news, tech articles/blogs, tech talks, tutorial videos, and hands-on workshop/codelabs, on machine learning, deep learning, data science, etc..
This presentation was made by Sathyajith Bhat, Senior DevOps Engineer, Adobe as part of Container Conference 2018.
Securing Containers: "Containers have almost become the ubiquitous method of packaging and deploying applications. While containers are perceived to be completed isolated and secure methods of running your application, the reality is that containers are not completely foolproof and are susceptible to many attack vectors. This session takes a look at the attack vectors and different ways to mitigate them."
www.containerconf.in
MongoDB .local Bengaluru 2019: Using MongoDB Services in Kubernetes: Any Plat...MongoDB
MongoDB Kubernetes operator is ready for prime-time. Learn about how MongoDB can be used with most popular orchestration platform, Kubernetes, and bring self-service, persistent storage to your containerized applications.
Infrastructure-as-code: bridging the gap between Devs and OpsMykyta Protsenko
Ops are overwhelmed with support. Devs are mad because their cannot deploy the changes as fast as they want. Sounds familiar?
Infrastructure-as-code can make your life easier by empowering developers and reducing operations' routine toil. It can cut down the lead time for infrastructure provisioning from hours or even days to minutes.
This talk reviews several IaC tools and approaches, showing how to integrate them into continuous delivery pipeline. It covers the problems and challenges that engineers may face while working with infrastructure-as-code tools and provides a few hands-on recipes to address them.
NATS: Simple, Secure and Scalable Messaging For the Cloud Native Erawallyqs
The majority of middleware and messaging systems in use were built in a time that did not have the concept of scale and real-time data that developers operate in today.
With the rise of Cloud Native and Microservices architectures as a design principle and the emphasis on simplicity, speed, and flexibility that come with it, developers need a messaging protocol to match.
Enter NATS. NATS is a remarkably lightweight messaging protocol, and extremely flexible and resilient. It is just a few MB in size, and can scale to publish tens of millions of message from a single server.
This is a presentation I held at "DevOps and Security" -meetup on 5th of April 2016 at RedHat.
Source is available at: https://github.com/jerryjj/devsec_050416
Microservices @ Work - A Practice Report of Developing MicroservicesQAware GmbH
Cloud Native Night October 2016, Mainz: Talk by Simon Bäumler (Technical Chief Designer at QAware).
Join our Meetup: www.meetup.com/cloud-native-night
Abstract: This talk takes a practice oriented approach to examine microservice oriented architecture. It will show two real systems, one build from scratch in a microservice architecture, the other migrated from a monolithic system to a microservice architecture.
With the example of these two systems the pittfalls, advantages and lessons learned using microservice oriented architectures will be discussed.
While both systems use the java stack, including spring boot and spring cloud many topics will be kept general and will be of interest for all developers.
OpenEBS is a container-native open source containerized storage project for containers – tightly integrated into Kubernetes.
You can find the full presentation here: https://www.facebook.com/VMTNcommunity/videos/2008142932762386/
Sf bay area Kubernetes meetup dec8 2016 - deployment modelsPeter Ss
I talk about deploying complex, multi-layer applications in Kuberentes.
I describe how Kubernetes AppController project (https://github.com/Mirantis/k8s-AppController) can be leveraged to enhance such deployments
Kubernetes is a container orchestration platform that provides a mechanism to manage the resources of containers in the cluster. That mechanism is known as "Requests and Limits".
Requests and limits play a key role not only in resource management but also in applications stability, capacity planning, scheduling the resources (i.e., on which node the pod will be running).
In this session we will cover:
- A quick review of Containers, Docker, and Kubernetes.
- Containers resource management in Kubernetes.
- Containers resource types in Kubernetes.
- 3 different ways to set requests and limits.
- The difference between capacity and allocatable resources.
- Tips and recap.
Slides used for Orchestructure May 2018 workshop.
Labs:
https://github.com/mrbobbytables/k8s-intro-tutorials
Event Information:
https://www.meetup.com/orchestructure/events/250189685/
Presented at AI NEXTCon Seattle 1/17-20, 2018
http://aisea18.xnextcon.com
join our free online AI group with 50,000+ tech engineers to learn and practice AI technology, including: latest AI news, tech articles/blogs, tech talks, tutorial videos, and hands-on workshop/codelabs, on machine learning, deep learning, data science, etc..
This presentation was made by Sathyajith Bhat, Senior DevOps Engineer, Adobe as part of Container Conference 2018.
Securing Containers: "Containers have almost become the ubiquitous method of packaging and deploying applications. While containers are perceived to be completed isolated and secure methods of running your application, the reality is that containers are not completely foolproof and are susceptible to many attack vectors. This session takes a look at the attack vectors and different ways to mitigate them."
www.containerconf.in
MongoDB .local Bengaluru 2019: Using MongoDB Services in Kubernetes: Any Plat...MongoDB
MongoDB Kubernetes operator is ready for prime-time. Learn about how MongoDB can be used with most popular orchestration platform, Kubernetes, and bring self-service, persistent storage to your containerized applications.
Infrastructure-as-code: bridging the gap between Devs and OpsMykyta Protsenko
Ops are overwhelmed with support. Devs are mad because their cannot deploy the changes as fast as they want. Sounds familiar?
Infrastructure-as-code can make your life easier by empowering developers and reducing operations' routine toil. It can cut down the lead time for infrastructure provisioning from hours or even days to minutes.
This talk reviews several IaC tools and approaches, showing how to integrate them into continuous delivery pipeline. It covers the problems and challenges that engineers may face while working with infrastructure-as-code tools and provides a few hands-on recipes to address them.
Docker and Cloud - Enables for DevOps - by ACA-ITStijn Wijndaele
DevOps is gericht op het tot stand brengen van een cultuur binnen organisaties waardoor het ontwikkelen, valideren en releasen van software sneller, meer betrouwbaar en frequenter kan verlopen. Om dit te realiseren staan het automatiseren van het 'software delivery process' en de bijhorende infrastructurele veranderingen centraal. Door de opkomst van 'Microservice Architecture' neemt het belang hiervan nog verder toe.
Sprekers: Stijn Van den Enden & Stijn Wijndaele (ACA IT-Solutions) DevOps is gericht op het tot stand brengen van een cultuur binnen organisaties waardoor het ontwikkelen, valideren en releasen van software sneller, meer betrouwbaar en frequenter kan verlopen. Om dit te realiseren staan het automatiseren van het 'software delivery process' en de bijhorende infrastructurele veranderingen centraal. Door de opkomst van 'Microservice Architecture' neemt het belang hiervan nog verder toe.
In deze avondconferentie werd, na een korte toelichting over DevOps, nagegaan wat Docker en de Cloud kunnen betekenen voor uw business, en hoe zij als enablers kunnen dienen voor het tot stand brengen van een DevOps-cultuur. Het container-landschap waarvan tools zoals Kubernetes, Docker Swarm, ...een belangrijk onderdeel vormen, wordt toegelicht en er wordt ingegaan op de wijze waarop deze tools aangewend kunnen worden om 'development' en 'operations' efficiënt te laten samenwerken.
DCEU 18: Use Cases and Practical Solutions for Docker Container Storage on Sw...Docker, Inc.
Mark Church - Product Manager, Docker
Don Stewart - Solutions Architect, Docker
Persistent storage has quickly advanced from something considered incompatible with containers to a mature set of solutions and patterns that have been thoroughly adopted by the industry. We’ll define the persistent characteristics of different use-cases and map these to some of the many solutions that exist for container storage. From this talk you’ll learn about the storage options available to users on Swarm, Kubernetes, on-premises, cloud, and how they work and compare to each other. You’ll also learn how to characterize different persistent application requirements and the solutions best for suited for them.
Hybrid and multicloud deployments are critical approaches for bridging the gap between legacy and modern architectures. Sandeep Parikh discusses common patterns for creating scalable cross-environment deployments using Kubernetes and explores best practices and repeatable patterns for leveraging Kubernetes as a consistent abstraction layer across multiple environments.
Cloud native applications are popular these days. They promise superior reliability and almost arbitrary scalability. They follow three key principles: they are built and composed as microservices. They are packaged and distributed in containers. The containers are executed dynamically in the cloud. But which technology is best to build this kind of application? This talk will be your guidebook.
In this hands-on session, we will briefly introduce the core concepts and some key technologies of the cloud native stack and then show how to build, package, containerize, compose and orchestrate a cloud native showcase application on top of a cluster operating system such as Kubernetes or OpenShift. Throughout the session we will be using an off-the-shelf MIDI controller to visualize the concepts and to remote control the cluster.
Container Days 2017 conference. @ConDaysEU #CDS17 #qaware #CloudNativeNerd @LeanderReimer
A hitchhiker‘s guide to the cloud native stackQAware GmbH
Container Days 2017, Hamburg: Vortrag von Mario-Leander Reimer (@LeanderReimer, Cheftechnologe bei QAware).
Abstract: Cloud-Größen wie Google, Twitter und Netflix haben die Kernbausteine ihrer Infrastruktur quelloffen verfügbar gemacht. Das Resultat aus vielen Jahren Cloud-Erfahrung ist nun frei zugänglich, und jeder kann seine eigenen Cloud-nativen Anwendungen entwickeln – Anwendungen, die in der Cloud zuverlässig laufen und fast beliebig skalieren. Die einzelnen Bausteine wachsen zu einem großen Ganzen zusammen, dem Cloud Native Stack.
In dieser Session stellen wir die wichtigsten Konzepte und Schlüsseltechnologien vor und bringen dann eine Spring-Cloud-basierte Beispielanwendung schrittweise auf Kubernetes und DC/OS zum Laufen. Dabei diskutieren wir verschiedene praktikable Architekturalternativen.
DEVNET-1140 InterCloud Mapreduce and Spark Workload Migration and Sharing: Fi...Cisco DevNet
Data gravity is a reality when dealing with massive amounts and globally distributed systems. Processing this data requires distributed analytics processing across InterCloud. In this presentation we will share our real world experience with storing, routing, and processing big data workloads on Cisco Cloud Services and Amazon Web Services clouds.
For our next ArcReady, we will explore a topic on everyone’s mind: Cloud computing. Several industry companies have announced cloud computing services . In October 2008 at the Professional Developers Conference, Microsoft announced the next phase of our Software + Services vision: the Azure Services Platform. The Azure Services Platforms provides a wide range of internet services that can be consumed from both on premises environments or the internet.
Session 1: Cloud Services
In our first session we will explore the current state of cloud services. We will then look at how applications should be architected for the cloud and explore a reference application deployed on Windows Azure. We will also look at the services that can be built for on premise application, using .NET Services. We will also address some of the concerns that enterprises have about cloud services, such as regulatory and compliance issues.
Session 2: The Azure Platform
In our second session we will take a slightly different look at cloud based services by exploring Live Mesh and Live Services. Live Mesh is a data synchronization client that has a rich API to build applications on. Live services are a collection of APIs that can be used to create rich applications for your customers. Live Services are based on internet standard protocols and data formats.
Join us for a deep dive into Windows Azure. We’ll start with a developer-focused overview of this brave new platform and the cloud computing services that can be used either together or independently to build amazing applications. As the day unfolds, we’ll explore data storage, SQL Azure™, and the basics of deployment with Windows Azure. Register today for these free, live sessions in your local area.
Cloud-native .NET Microservices mit KubernetesQAware GmbH
BASTA! 2017, Mainz: Talk von Mario-Leander Reimer (@LeanderReimer, Cheftechnologe bei QAware).
Cloud-Größen wie Google, Twitter und Netflix haben die Kernbausteine ihrer Infrastruktur quelloffen verfügbar gemacht. Das Resultat aus vielen Jahren Cloud-Erfahrung ist nun frei zugänglich, und jeder kann seine eigenen Cloud-nativen Anwendungen entwickeln – Anwendungen, die in der Cloud zuverlässig laufen und fast beliebig skalieren. Die einzelnen Bausteine wachsen zu einem großen Ganzen zusammen, dem Cloud-Native-Stack. In dieser Session stellen wir die wichtigsten Konzepte und aktuellen Schlüsseltechnologien kurz vor. Anschließend implementieren wir einen einfachen Microservice mit .NET Core und Steeltoe OSS und bringen ihn zusammen mit ausgewählten Bausteinen für Service-Discovery und Konfiguration schrittweise auf einem Kubernetes-Cluster zum Laufen.
Similar to Building a Global-Scale Multi-Tenant Cloud Platform on AWS and Docker: Lessons Learned (20)
Speed Kit: Getting Websites out of the Web Performance Stone AgeFelix Gessert
Page load time is money. This is not only true for companies like Amazon that lose more than $1.3B in revenue per year if their website is a 10th of a second slower. It is also true for publishers, whose business model depends on a user experience that facilitates consumption of as much content as possible. However, many publishers have heterogeneous and complex technology stacks that make it extremely hard to tackle performance, scalability and page load time.
Novel browser technologies now offer a means of making web performance as simple as including a script. The research spin-off Baqend has developed a "Speed Kit" that directly hooks into existing publisher, e-commerce and brand website and makes them 50-300% faster. In this pitch, we will explain how the technology works for large websites.
NoSQL Data Stores in Research and Practice - ICDE 2016 Tutorial - Extended Ve...Felix Gessert
The unprecedented scale at which data is consumed and generated today has shown a large demand for scalable data management and given rise to non-relational, distributed "NoSQL" database systems. Two central problems triggered this process: 1) vast amounts of user-generated content in modern applications and the resulting requests loads and data volumes 2) the desire of the developer community to employ problem-specific data models for storage and querying. To address these needs, various data stores have been developed by both industry and research, arguing that the era of one-size-fits-all database systems is over. The heterogeneity and sheer amount of these systems - now commonly referred to as NoSQL data stores - make it increasingly difficult to select the most appropriate system for a given application. Therefore, these systems are frequently combined in polyglot persistence architectures to leverage each system in its respective sweet spot. This tutorial gives an in-depth survey of the most relevant NoSQL databases to provide comparative classification and highlight open challenges. To this end, we analyze the approach of each system to derive its scalability, availability, consistency, data modeling and querying characteristics. We present how each system's design is governed by a central set of trade-offs over irreconcilable system properties. We then cover recent research results in distributed data management to illustrate that some shortcomings of NoSQL systems could already be solved in practice, whereas other NoSQL data management problems pose interesting and unsolved research challenges.
If you'd like to use these slides for e.g. teaching, contact us at gessert at informatik.uni-hamburg.de - we'll send you the PowerPoint.
Web Performance – die effektivsten Techniken aus der PraxisFelix Gessert
Eine durchschnittliche Webseite lädt 2299KB an Daten und macht dafür 100 HTTP Anfragen. Dass Ladezeiten einen immensen Einfluss auf User-Zufriedenheit und Business-Metriken haben, bezweifelt dieser Tage niemand mehr. Aber die Meinungen darüber, welche Techniken Ladezeiten effektiv minimieren, gehen weit auseinander. Dieser Vortrag gibt einen detaillierten Überblick zu den wichtigsten Techniken der Web Performance Optimierung vom Critical Rendering Path bis zu verteilten Caching-Infrastrukturen an einem Beispiel aus der Praxis.
Cache Sketches: Using Bloom Filters and Web Caching Against Slow Load TimesFelix Gessert
Nach aktuellem Stand (April 2016) lädt eine durchschnittliche Webseite 2299KB an Daten und macht dafür 100 HTTP Requests. Dass Ladezeiten einen immensen Einfluss auf User-Zufriedenheit und Business-Metriken haben, bezweifelt dieser Tage niemand mehr. Aber die Meinungen darüber mit welchen Techniken sich Ladezeiten effektiv minimieren lassen, gehen weit auseinander. Wir möchten einen völlig neuen Ansatz vorstellen, der in 5 Jahren Forschung im Fachbereich Informatik an der Uni Hamburg entwickelt wurde. Die Idee dahinter ist die wohl älteste Performance-Optimierung der Informatik überhaupt: Caching. Das neue an der Methode liegt darin, dass alle Arten von existierenden Web Caches vom Browser bis zum CDN durch ein paar algorithmischen Tricks dazu in der Lage versetzt werden, stets aktuelle Daten auszuliefern, anstatt mit über den Daumen gepeilten TTLs längst veralteten Content zu verteilen. Das auf Bloomfiltern, Real-Time Query Matching und Machine Learning basierende "Cache Sketch" Verfahren möchten wir im Detail diskutieren und zeigen, wie sich moderne Web-Anwendungen damit drastisch beschleunigen lassen.
Cloud Databases in Research and PracticeFelix Gessert
The combination of database systems and cloud computing is extremely attractive: unlimited storage capacities, elastic scalability and as-a-Service models seem to be within reach. This talk will give an in-depth survey of existing solutions for cloud databases that evolved in the last years and provide classification and comparison. This includes real-world systems (e.g. Azure Tables, DynamoDB and Parse) as well as research approaches (e.g. RelationalCloud and ElasTras). In practice however, there are some unsolved problems. Network latency, scalable transactions, SLAs, multi-tenancy, abstract data modelling, elastic scalability and polyglot persistence pose daunting tasks for many scenarios. Therefore, we conclude with „Orestes“ a research approach based on well-known techniques such as web caching, Bloom filters and optimistic concurrency control that demonstrates how existing cloud databases can be enhanced to suit specific applications.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Quantum Computing: Current Landscape and the Future Role of APIs
Building a Global-Scale Multi-Tenant Cloud Platform on AWS and Docker: Lessons Learned
1. Felix Gessert, Florian Bücklers
{fg,fb}@baqend.com
Building a Global-Scale Multi-
Tenant Cloud Platform on AWS and
Docker: Lessons Learned
@baqendcom
13. Problem: Netzwerklatenz
I. Grigorik, High performance browser networking.
O’Reilly Media, 2013.
2× Bandwidth = Same Load Time
½ Latency ≈ ½ Load Time
24. Innovation
Solution: Baqendproactivelyrevalidates data
F. Gessert, F. Bücklers, und N. Ritter, „ORESTES:a Scalable
Database-as-a-Service Architecture for Low Latency“, in
CloudDB 2014, 2014.
F. Gessert und F. Bücklers, „ORESTES: ein System für horizontal
skalierbaren Zugriff auf Cloud-Datenbanken“, in Informatiktage
2013, 2013.
F. Gessert, S. Friedrich, W. Wingerath, M. Schaarschmidt, und
N. Ritter, „Towards a Scalable and Unified REST API for Cloud
Data Stores“, in 44. Jahrestagung der GI, Bd. 232, S. 723–734.
F. Gessert, M. Schaarschmidt, W. Wingerath, S. Friedrich, und
N. Ritter, „The Cache Sketch:Revisiting Expiration-based
Caching in the Age of Cloud Data Management“, in BTW 2015.
F. Gessert und F. Bücklers, Performanz- und
Reaktivitätssteigerung von OODBMS vermittels der Web-
Caching-Hierarchie. Bachelorarbeit, 2010.
F. Gessert und F. Bücklers, KohärentesWeb-Caching von
Datenbankobjektenim Cloud Computing. Masterarbeit 2012.
W. Wingerath, S. Friedrich, und F. Gessert, „Who Watches the
Watchmen? On the Lack of Validation in NoSQL
Benchmarking“, in BTW 2015.
M. Schaarschmidt, F. Gessert, und N. Ritter, „Towards
Automated Polyglot Persistence“, in BTW 2015.
S. Friedrich, W. Wingerath, F. Gessert, und N. Ritter, „NoSQL
OLTP Benchmarking: A Survey“, in 44. Jahrestagungder
Gesellschaftfür Informatik, 2014, Bd. 232, S. 693–704.
F. Gessert, „Skalierbare NoSQL- und Cloud-Datenbanken in
Forschung und Praxis“, BTW 2015
38. Route 53, EC2, ASGs, IAM etc.
Elastic LoadBalancer: TCP Balancing for Logging
◦ Not suited for multi-tenant SSL termination: ELB cannot
dynamically route to an IP:port pair
RedisElastiCache: Metadata Storage
◦ Easy to use but very limited: no Redis cluster support, no
append-only files, bad snapshotting
What we don‘t use:
◦ Beanstalk: supports Docker but needs a dedicated EC2 instance
◦ Cloudfront: useless invalidations, expensive
◦ DynamoDB: difficult to scale, very limited queries
AWS Services
Serviceswe use
39. Every tenant needs a private JVM and Node.JS process
Containerization
Why we needcontainers& cluster management
Baqend
Server
Customer‘s
Business
Logic
40. Every tenant needs a private JVM and Node.JS process
Provisioning new instances needs to be fast & easy:
Containerization
Why we needcontainers& cluster management
Baqend
Server
Customer‘s
Business
Logic
Launch
App
BBQ
Manager
Start
Configure databases,
CDN, etc.
45. Docker Concepts
What is Docker?
Source: https://docs.docker.com/engine/introduction/understanding-docker/
The docker image can be hosted and transferred to different
hosts (DockerRegistry)
The docker image can be executedas a new container on any
machine that runs a Dockerdaemon
Updatesare handled by just stopping and starting a new
container
Docker typically
isolates a single
application
An applicationis built
into a Dockerimage
(including the OS)
46. Docker runs on all common Linux distributions
Docker can be installed from Docker’s own package
repository
The Docker daemon can be configured by editing
/etc/default/docker
The Docker daemon allows many useful configurations:
◦ Inter-container communication
◦ Docker remote REST API
◦ Labeling
◦ DNS configuration
◦ IP forwarding (disables internet for containers)
◦ SSL encryption for the Docker damon
Docker Architecture
Howto set up a Dockerhost
48. Filesystem: by using multiple read-only file systems and
mounting a read-write file system on top
Data volumes: mount additional physical volumes into
the container
CPU: by CPU shares and core limitation
Memory: by defining memory constraints
Network: by using virtual networks
Systemprivileges: such as port binding, execution
rights, inter process communication, etc.
Logging: by using docker logging capabilities or external
loggers (json, syslog, aws, etc...)
How a Docker container works
Isolation,performance, light-weight
49. Most constraints are set when the container is started
Docker Options
Imposingconstraints on containers
--add-host=[] Add a custom host-to-IP mapping (host:ip)
--cpu-shares=0 CPU shares (relative weight)
--cpu-quota=0 Limit CPU CFS (Completely Fair Scheduler) quota
-e, --env=[] Set environment variables
-l, --label=[] Set metadata on the container (e.g., --label=key=value)
--link=[] Add link to another container
-m, --memory="" Memory limit
--memory-swap="" Total memory (memory + swap), '-1' to disable swap
--name="" Assign a name to the container
--net="bridge" Connects a container to a network
'bridge': creates a new network stack on the docker bridge
'none': no networking for this container
'container:<name|id>': reuses another container network stack
'host': use the host network stack inside the container
'NETWORK': connects the container to user-created network
--oom-kill-disable=false Whether to disable OOM Killer for the container or not
-p, --publish=[] Publish a container's port(s) to the host
--read-only=false Mount the container's root filesystem as read only
--restart="no" Restart policy (no, on-failure[:max-retry], always)
-v, --volume=[] Bind mount a volume
50. Docker containers can talk to each other by default
Communication between containers can be restricted
by the daemon option: –-icc=false
Docker containers can discover other linked containers
by their names
Docker Networking
Making containerstalk to each other
EXPOSE 8080
docker run --name="orestes" orestes docker run --link="orestes" node
Can access orestes:8080
Port 8080 not published,
(can’t be accessed from host
or other containers)
51. Docker containers can talk to each other by default
Communication between containers can be restricted
by the daemon option: –-icc=false
Docker containers can discover other linked containers
by their names
Docker Networking
Making containerstalk to each other
EXPOSE 8080
docker run --name="orestes"
-p 0.0.0.0:80:8080 orestes
docker run --link="orestes" node
Can access orestes:8080
Port 8080 is published and
can be accessed on the host
port 80
52. AWS provides ECS-optimizedAMIs for simple deployment
ECS manages EC2 instances by running an ECS Agent on each instance
ECS can automatically deploy and scale new Docker containers
specified by a Task definition across the ECS Cluster
Elastic Container Service
HowAmazonECSworks
ECS Cluster
Docker Daemon
ECS Agent
Docker Daemon
ECS Agent
Docker Daemon
ECS Agent
53. ECS groups containers into Tasks and deploys them
together
A Task definitiondescribes:
◦ The Docker images
◦ Resource requirements
◦ Environment variables
◦ Network links
◦ Data Volumes
ECS Services can be used to keep a specified number of
Tasks running
ECS can autoscale a Service when it is used with an ELB
ECS: Tasks and Services
Defininggroupsof containers
54. ECS has used an outdated version of docker, now it’s 1.9, yeah!
Tasks can now be parametrized using commandline args
Previously only environment variables could be passed while
starting a Task
Environment variables are exposed to linked containers, this can
be a security issue!
Limitations that AWS fixed
Old Docker, Parameterization
docker run --name="orestes"
--env SECRET=7kekfjd9e
docker run --link="orestes" node
Can access env
ORESTES_SECRET
Untrusted ProcessSecured Process
https://docs.docker.com/engine/userguide/networking/default_network/dockerlinks/#environment-variables
55. ECS uses hard memory constraints (run –m) for Tasks to
schedule container placement
This allocates a fixed amountof memoryon the EC2 instance
and can’t be exceeded by the process
This is very ugly for shared, multi tenant applications:
◦ Setting the constraint too low causes Docker to kill the process on
memory peaks
◦ Setting the value too high limits the number of containers that can be
launched per EC2 instance
Neither Docker’s memory swapping nor unlimited memory
usage is allowed by ECS
Current Limitation: Memory Constraints
RestrictingRAM consumption
56. Docker has introduced a new network API, which allows
to create custom virtual networks
Current Limitation: Networking
Docker‘snew network API not supported
BridgeNetworksconnect
groups of containers together
and isolate them from other
groups on the same host
OverlayNetworksuse a key-
value store to connect
containers across different
host machines
Source: https://docs.docker.com/engine/userguide/networking/dockernetworks/
57. Very simplesetup,thanks to
the optimized ECS AMI
Task abstraction makes it
really comfortable to start
multiple containers together
Services ensures that the
desired count of tasks are
always up and running
Automatically starts newEC2
instancesif no capacity is left
for new containers
Can be combined with an ELB
for a high availability setup
Many Docker options aren’t
available
Service Tasks can’t be
parametrized
Runningthe sameServices for
different tenants on the same
EC2 instance is not possible
Only the legacy networking is
supported
New featureswill always be
delayed since they must first
be implemented in ECS
58. Docker Swarm is Docker’s nativesolution for cluster
management
Docker Swarm uses a discovery serviceto manage the
shared state of the cluster
The following backends for discovery are supported:
◦ Docker Hub (for development only)
◦ Static file
◦ etcd
◦ consul
◦ zookeeper
◦ IP list or a range pattern of IPs
Docker Swarm
A replacementfor ECS
60. The Swarm manager acts as a proxy of the Docker
Remote API
◦ All Docker run options are available in Swarm, too
Docker Swarm can be combined with overlaynetworks
◦ Containers can connect to others by just using the containers
name (servicediscovery)
◦ Works across Docker hosts, availability zones and external
hosts
Containers can use any other service without defining
them in a group (such as a Task)
Swarm is Docker
Fixing the shortcomingsof ECS
61. Docker hosts can be added and removed to the Swarm
Cluster silently
Swarm provides an API to gather CPU usage and
memory consumption of hosts or containers
Swarm provides no concept to scale services within
containers
Autoscaling in Swarm
Scale-out and scale-in
62. LabeledDocker daemons can be used by the manager
to run specific containers only on specific hosts
Containers can be launched:
◦ On the same host where other containers are running
◦ In a specific availability zone
◦ On hosts with special capabilities (RAM, CPU or SSD)
The Docker daemon can restartfailed containers using
a restart policy --restart="yes"
Containers will also be restarted if the docker host
restarts
Failed machines must be handled manually
High Availability in Swarm
Handling failures and outages
63. Swarm requires that the Docker daemon is exposed via
TCP
In most setups this will be a security issue since you can
easily get root permission on the Docker host
Also containers can access the exposed API by default
Therefore it is recommended to always secure the
Docker daemons on each host with SSL
Docker supports SSL client, server and both
authentication mechanisms
SSL server authentication is not very practical since it
requires a signed certificate for each host
Securing Swarm Hosts
Securitypitfalls
64. Securing Swarm Hosts
Securitypitfalls
Securing a Swarm cluster requires signed SSL certificates on all
docker hosts, on the swarm manager and the docker client
Docker Swarm Cluster
Docker Daemon
Expose 2375
Docker Daemon
Expose 2375
Docker Daemon
Expose 2375
Swarm Manager
Docker Client
Certificate
Authority
Client
Certificate
Server
Certificates
Server/Client
Certificate
65. Wrap-up: Docker Swarm
ProsandCons
Swarmis Docker, all Docker
options are available
LabelingDocker hosts,
allows to deploy containers
on specific hosts
OverlayNetworksallow
containers to communicate
across hosts
ServiceDiscoveryacross
containers is made really
simple
Complexsetupand many
componentsare required
for a completesetup
No built-in way for
autoscalingservices
Still many bugs
The Docker Swarm API
integrationinto Docker is
not yet completed
66. Conclusions
ECSvs Swarm
SimpleSetup
Taskand Servicedefinition
makes it easy to deploy
and update containers
Detect failuresand restart
failed tasks within services
Integratedinto other AWS
Services such as Elastic
Load Balancers and Auto
Scaling Groups
Complex Setup
Many configuration options
for deploying containers
Is compatibleto the Docker
API, allows to use all
Docker clients
Supports Docker’s network
API
No Vendor Lock-In
67. Ziel mit InnoRampUp
Want to try Baqend?
Download Community
Edition
Invited-Beta Cloud Instance
support@baqend.com
Baqend Cloud launching
this February