This document discusses Netflix's experience with Cassandra and database automation during a reboot event in 2014 across all availability zones on AWS. It describes the initial stack used including Cassandra, Priam, and Atlas for managing instances. Gaps were identified in this approach and alternatives were explored, leading Netflix to decide on an asynchronous, RESTful solution. Key principles learned included using asynchronous vs synchronous communications and REST vs SSH. The future direction of this work is also briefly discussed.
What's New in JHipsterLand - Devoxx US 2017Matt Raible
YouTube: https://youtu.be/N5bN-wvLZMg
JHipster is an application generator that allows you to create monoliths or microservices based on Spring Boot and Angular. It leverages Spring Cloud for microservices and contains best-of-breed JavaScript and CSS libraries for creating your UI. In this session, you’ll learn about what’s new in JHipster. Possible topics include Angular 2, Progressive Web Apps, HTTP/2, JUnit 5 and Spring 5.
Demo code: https://github.com/mraible/devoxxus-jhipster-microservices-demo
The Ultimate Getting Started with Angular Workshop - Devoxx France 2017Matt Raible
These slides are from a workshop I did at Devoxx France 2017. I showed how to set up an Angular development environment from scratch, develop a simple app, test it, integrating CSS frameworks (Angular Material and Bootstrap 4), secure it with OpenID Connect, and deploy it to the cloud.
Source code and tutorial: https://github.com/mraible/ng-demo
The basics you need to know to get up and running with Chaos Monkey in your Amazon Web Service's Cloud enviornment.
Links:
CloudFormation Template:
https://github.com/joehack3r/aws/blob/master/cloudformation/templates/chaosMonkey.json
Simian Army Quick Start Guide:
https://github.com/Netflix/SimianArmy/wiki/Quick-Start-Guide
Chaos Monkey Configuration:
https://github.com/Netflix/SimianArmy/wiki/Chaos-Settings
Chaos Monkey Army:
https://github.com/Netflix/SimianArmy/wiki/The-Chaos-Monkey-Army
What's New in JHipsterLand - Devoxx US 2017Matt Raible
YouTube: https://youtu.be/N5bN-wvLZMg
JHipster is an application generator that allows you to create monoliths or microservices based on Spring Boot and Angular. It leverages Spring Cloud for microservices and contains best-of-breed JavaScript and CSS libraries for creating your UI. In this session, you’ll learn about what’s new in JHipster. Possible topics include Angular 2, Progressive Web Apps, HTTP/2, JUnit 5 and Spring 5.
Demo code: https://github.com/mraible/devoxxus-jhipster-microservices-demo
The Ultimate Getting Started with Angular Workshop - Devoxx France 2017Matt Raible
These slides are from a workshop I did at Devoxx France 2017. I showed how to set up an Angular development environment from scratch, develop a simple app, test it, integrating CSS frameworks (Angular Material and Bootstrap 4), secure it with OpenID Connect, and deploy it to the cloud.
Source code and tutorial: https://github.com/mraible/ng-demo
The basics you need to know to get up and running with Chaos Monkey in your Amazon Web Service's Cloud enviornment.
Links:
CloudFormation Template:
https://github.com/joehack3r/aws/blob/master/cloudformation/templates/chaosMonkey.json
Simian Army Quick Start Guide:
https://github.com/Netflix/SimianArmy/wiki/Quick-Start-Guide
Chaos Monkey Configuration:
https://github.com/Netflix/SimianArmy/wiki/Chaos-Settings
Chaos Monkey Army:
https://github.com/Netflix/SimianArmy/wiki/The-Chaos-Monkey-Army
Talk given at DevTeach Montreal on RxJS - The Basics & The Future.
Example repo: https://github.com/ladyleet/rxjs-test
Have questions? Find me on twitter http://twitter.com/ladyleet
Automated Testing is a vital part of any healthy software development process. It ensures high code quality, architectural flexibility which in turn helps to keep a fast pace in developing new features. It even gets more important for open source projects. You don’t want to blindly depend on untested projects which you include as libraries in your next production release.
In this talk we’re not going to talk about TDD or unit testing. Rather, we’re diving into how automated, end-2-end testing is done in 2018, with Cypress.
Exactly once delivery is a harsh mistress - DevOps Days TLVNatan Silnitsky
In this talk I go over the basic theory of messaging in distributed systems, the different message delivery guarantees in Kafka and the to use them.
I focus on exactly once delivery guarantees and the way Kafka implements it with transaction based messaging protocol.
Including a discussion of the latency/throughput trade-offs, resource utilisation and its overall advantages and shortcomings.
Finally, I show a use-case at Wix where exactly once delivery helped us solve a big problem.
The Azure Portal is one of the biggest, and most complex Single Page Applications (SPA) in the World. Are you curious what’s under the cover? Let’s check what tools, frameworks, libraries and techniques are used to enable over 500 developers to work on one web application, and having possibility to release a new version anytime.
In this session I will uncover all details of end-to-end development process, performance tweaks, and how we implemented Continuous Integration, and Continuous Delivery.
The Power of RxJS in Nativescript + AngularTracy Lee
Learn the basics of use and power of RxJS in NativeScript & Angular in this presentation given at NativeScript Developer Days in New York City September 2017
Mobile is a strong focus at Etsy and we are working on bringing the lessons we learned from running a successful website to the mobile space. This talk will cover how we do Continuous Integration for our mobile apps and have build up an infrastructure around it that supports the workflow of mobile development.
Talk given at Swift Summit 2016, in San Francisco CA by Sommer Panage. Discusses how GraphQL is beneficial to mobile apps and how it can interface with Swift via up-and-coming code generation.
Talk given at DevTeach Montreal on RxJS - The Basics & The Future.
Example repo: https://github.com/ladyleet/rxjs-test
Have questions? Find me on twitter http://twitter.com/ladyleet
Automated Testing is a vital part of any healthy software development process. It ensures high code quality, architectural flexibility which in turn helps to keep a fast pace in developing new features. It even gets more important for open source projects. You don’t want to blindly depend on untested projects which you include as libraries in your next production release.
In this talk we’re not going to talk about TDD or unit testing. Rather, we’re diving into how automated, end-2-end testing is done in 2018, with Cypress.
Exactly once delivery is a harsh mistress - DevOps Days TLVNatan Silnitsky
In this talk I go over the basic theory of messaging in distributed systems, the different message delivery guarantees in Kafka and the to use them.
I focus on exactly once delivery guarantees and the way Kafka implements it with transaction based messaging protocol.
Including a discussion of the latency/throughput trade-offs, resource utilisation and its overall advantages and shortcomings.
Finally, I show a use-case at Wix where exactly once delivery helped us solve a big problem.
The Azure Portal is one of the biggest, and most complex Single Page Applications (SPA) in the World. Are you curious what’s under the cover? Let’s check what tools, frameworks, libraries and techniques are used to enable over 500 developers to work on one web application, and having possibility to release a new version anytime.
In this session I will uncover all details of end-to-end development process, performance tweaks, and how we implemented Continuous Integration, and Continuous Delivery.
The Power of RxJS in Nativescript + AngularTracy Lee
Learn the basics of use and power of RxJS in NativeScript & Angular in this presentation given at NativeScript Developer Days in New York City September 2017
Mobile is a strong focus at Etsy and we are working on bringing the lessons we learned from running a successful website to the mobile space. This talk will cover how we do Continuous Integration for our mobile apps and have build up an infrastructure around it that supports the workflow of mobile development.
Talk given at Swift Summit 2016, in San Francisco CA by Sommer Panage. Discusses how GraphQL is beneficial to mobile apps and how it can interface with Swift via up-and-coming code generation.
Architecting Container Infrastructure for Security and Compliance - CON406 - ...Amazon Web Services
While organizations gain agility and scalability when they migrate to containers and microservices, they also benefit from compliance and security, advantages that are often overlooked. In this session, Kelvin Zhu, lead software engineer at Okta, joins Mitch Beaumont, enterprise solutions architect at AWS, to discuss security best practices for containerized infrastructure. Learn how Okta built their development workflow with an emphasis on security through testing and automation. Dive deep into how containers enable automated security and compliance checks throughout the development lifecycle. Also understand best practices for implementing AWS security and secrets management services for any containerized service architecture.
by Omar Lari, Partner Solutions Architect, AWS
Amazon Elastic Container Service for Kubernetes (Amazon EKS) is a new managed service for running Kubernetes on AWS. This session will provide an overview of Amazon EKS, why we built it, and how it works.
As re:Invent draws to a close for another year, join us for a comprehensive recap of the most exciting AWS news with AWS EMEA Chief Technical Evangelist, Ian Massingham. On the webinar Ian will discuss new AWS service offerings in AI, Machine Learning, IoT, databases, and analytics. Ian will be discussing all of latest technology updates revealed during re:Invent 2017.
NEW LAUNCH! Introducing Amazon EKS - CON215 - re:Invent 2017Amazon Web Services
Amazon Elastic Container Service for Kubernetes (Amazon EKS) is a new managed service for running Kubernetes on AWS. This session will provide an overview of Amazon EKS, why we built it, and how it works.
Serverless in production, an experience report (CoDe-Conf)Yan Cui
AWS Lambda has changed the way we deploy and run software, but this new serverless paradigm has created new challenges to old problems - how do you test a cloud-hosted function locally? How do you monitor them? What about logging and config management? And how do we start migrating from existing architectures?
In this talk Yan and Scott will discuss solutions to these challenges by drawing from real-world experience running Lambda in production and migrating from an existing monolithic architecture.
How to make successful use of the cloud for your software startup. Based on 4 years of using various cloud services. Includes advice, war stories, and best practices.
Presented at CoderFaire Atlanta 2013.
Open Source at AWS: Code, Contributions, Collaboration, and CommunicationAmazon Web Services
At OSCON 2018, Adrian Cockcroft detailed the many ways AWS participates in open source: contributing to open source projects, reporting bugs, contributing fixes and enhancements to a wide spectrum of projects ranging from the Linux kernel to PostgreSQL and Kubernetes, and managing the hundreds of projects of its own.
Serverless in Production, an experience report (AWS UG South Wales)Yan Cui
AWS Lambda has changed the way we deploy and run software, but this new serverless paradigm has created new challenges to old problems - how do you test a cloud-hosted function locally? How do you monitor them? What about logging and config management? And how do we start migrating from existing architectures?
In this talk Yan and Scott will discuss solutions to these challenges by drawing from real-world experience running Lambda in production and migrating from an existing monolithic architecture.
ENT210-How to Get from Zero to Hundreds of AWS-Certified EngineersAmazon Web Services
For many organizations, a perceived lack of cloud skills in their staff can limit their move to the cloud. Proper training of your engineers and developers can speed the pace of adoption, cloud migration, and delivery of business benefits by effectively operating the AWS Cloud. In this session, we discuss field-proven, prescriptive steps for reskilling and scaling your technical teams so that you can use the AWS Cloud securely, efficiently, and effectively.
Introduction to AWS Fargate & Amazon Elastic Container Service for KubernetesAmazon Web Services
AWS Fargate makes running containerized workloads on AWS easier than ever before. This session will provide a technical background for using Fargate with your existing containerized services, including best practices for building images, configuring task definitions, task networking, secrets management, and monitoring.
About
Indigenized remote control interface card suitable for MAFI system CCR equipment. Compatible for IDM8000 CCR. Backplane mounted serial and TCP/Ethernet communication module for CCR remote access. IDM 8000 CCR remote control on serial and TCP protocol.
• Remote control: Parallel or serial interface.
• Compatible with MAFI CCR system.
• Compatible with IDM8000 CCR.
• Compatible with Backplane mount serial communication.
• Compatible with commercial and Defence aviation CCR system.
• Remote control system for accessing CCR and allied system over serial or TCP.
• Indigenized local Support/presence in India.
• Easy in configuration using DIP switches.
Technical Specifications
Indigenized remote control interface card suitable for MAFI system CCR equipment. Compatible for IDM8000 CCR. Backplane mounted serial and TCP/Ethernet communication module for CCR remote access. IDM 8000 CCR remote control on serial and TCP protocol.
Key Features
Indigenized remote control interface card suitable for MAFI system CCR equipment. Compatible for IDM8000 CCR. Backplane mounted serial and TCP/Ethernet communication module for CCR remote access. IDM 8000 CCR remote control on serial and TCP protocol.
• Remote control: Parallel or serial interface
• Compatible with MAFI CCR system
• Copatiable with IDM8000 CCR
• Compatible with Backplane mount serial communication.
• Compatible with commercial and Defence aviation CCR system.
• Remote control system for accessing CCR and allied system over serial or TCP.
• Indigenized local Support/presence in India.
Application
• Remote control: Parallel or serial interface.
• Compatible with MAFI CCR system.
• Compatible with IDM8000 CCR.
• Compatible with Backplane mount serial communication.
• Compatible with commercial and Defence aviation CCR system.
• Remote control system for accessing CCR and allied system over serial or TCP.
• Indigenized local Support/presence in India.
• Easy in configuration using DIP switches.
Water scarcity is the lack of fresh water resources to meet the standard water demand. There are two type of water scarcity. One is physical. The other is economic water scarcity.
Hierarchical Digital Twin of a Naval Power SystemKerry Sado
A hierarchical digital twin of a Naval DC power system has been developed and experimentally verified. Similar to other state-of-the-art digital twins, this technology creates a digital replica of the physical system executed in real-time or faster, which can modify hardware controls. However, its advantage stems from distributing computational efforts by utilizing a hierarchical structure composed of lower-level digital twin blocks and a higher-level system digital twin. Each digital twin block is associated with a physical subsystem of the hardware and communicates with a singular system digital twin, which creates a system-level response. By extracting information from each level of the hierarchy, power system controls of the hardware were reconfigured autonomously. This hierarchical digital twin development offers several advantages over other digital twins, particularly in the field of naval power systems. The hierarchical structure allows for greater computational efficiency and scalability while the ability to autonomously reconfigure hardware controls offers increased flexibility and responsiveness. The hierarchical decomposition and models utilized were well aligned with the physical twin, as indicated by the maximum deviations between the developed digital twin hierarchy and the hardware.
Saudi Arabia stands as a titan in the global energy landscape, renowned for its abundant oil and gas resources. It's the largest exporter of petroleum and holds some of the world's most significant reserves. Let's delve into the top 10 oil and gas projects shaping Saudi Arabia's energy future in 2024.
Explore the innovative world of trenchless pipe repair with our comprehensive guide, "The Benefits and Techniques of Trenchless Pipe Repair." This document delves into the modern methods of repairing underground pipes without the need for extensive excavation, highlighting the numerous advantages and the latest techniques used in the industry.
Learn about the cost savings, reduced environmental impact, and minimal disruption associated with trenchless technology. Discover detailed explanations of popular techniques such as pipe bursting, cured-in-place pipe (CIPP) lining, and directional drilling. Understand how these methods can be applied to various types of infrastructure, from residential plumbing to large-scale municipal systems.
Ideal for homeowners, contractors, engineers, and anyone interested in modern plumbing solutions, this guide provides valuable insights into why trenchless pipe repair is becoming the preferred choice for pipe rehabilitation. Stay informed about the latest advancements and best practices in the field.
NO1 Uk best vashikaran specialist in delhi vashikaran baba near me online vas...Amil Baba Dawood bangali
Contact with Dawood Bhai Just call on +92322-6382012 and we'll help you. We'll solve all your problems within 12 to 24 hours and with 101% guarantee and with astrology systematic. If you want to take any personal or professional advice then also you can call us on +92322-6382012 , ONLINE LOVE PROBLEM & Other all types of Daily Life Problem's.Then CALL or WHATSAPP us on +92322-6382012 and Get all these problems solutions here by Amil Baba DAWOOD BANGALI
#vashikaranspecialist #astrologer #palmistry #amliyaat #taweez #manpasandshadi #horoscope #spiritual #lovelife #lovespell #marriagespell#aamilbabainpakistan #amilbabainkarachi #powerfullblackmagicspell #kalajadumantarspecialist #realamilbaba #AmilbabainPakistan #astrologerincanada #astrologerindubai #lovespellsmaster #kalajaduspecialist #lovespellsthatwork #aamilbabainlahore#blackmagicformarriage #aamilbaba #kalajadu #kalailam #taweez #wazifaexpert #jadumantar #vashikaranspecialist #astrologer #palmistry #amliyaat #taweez #manpasandshadi #horoscope #spiritual #lovelife #lovespell #marriagespell#aamilbabainpakistan #amilbabainkarachi #powerfullblackmagicspell #kalajadumantarspecialist #realamilbaba #AmilbabainPakistan #astrologerincanada #astrologerindubai #lovespellsmaster #kalajaduspecialist #lovespellsthatwork #aamilbabainlahore #blackmagicforlove #blackmagicformarriage #aamilbaba #kalajadu #kalailam #taweez #wazifaexpert #jadumantar #vashikaranspecialist #astrologer #palmistry #amliyaat #taweez #manpasandshadi #horoscope #spiritual #lovelife #lovespell #marriagespell#aamilbabainpakistan #amilbabainkarachi #powerfullblackmagicspell #kalajadumantarspecialist #realamilbaba #AmilbabainPakistan #astrologerincanada #astrologerindubai #lovespellsmaster #kalajaduspecialist #lovespellsthatwork #aamilbabainlahore #Amilbabainuk #amilbabainspain #amilbabaindubai #Amilbabainnorway #amilbabainkrachi #amilbabainlahore #amilbabaingujranwalan #amilbabainislamabad
13. Our stack during Re:boot 2014
Disappearing
instance?
Launch new
instance
All good
Is the C* ring
healthy?
Are all instances
healthy?
All good
Can we fix
automatically?
Replace bad
instance
All good
Is there an
offline
maintenance?
First failure?
Sleep for X
minutes and
retry
PagerDuty
Is there an
offline
maintenance?
First failure?
All good
Every
30 min
14. Our stack during Re:boot 2014
AWS
Bootre:
September 2014, Every AZ
32. Check out our https://jobs.netflix.com page for current
openings
33. Who we are
Jean-Sebastien Jeannotte – JS
Senior Software Engineer
Platform Automation Engineering
jjeannotte@netflix.com
@jsjeannotte
http://www.linkedin.com/in/jsjeannotte
Nir Alfasi
Senior Software Engineer
Platform Automation Engineering
alfasi@netflix.com
@niralfasi
http://www.linkedin.com/in/alfasin
Christos Kalantzis
Director of Engineering
Cloud Database Engineering
Cassandra MVP
ckalantzis@netflix.com
@chriskalan
http://www.linkedin.com/in/christoskalantzis
Editor's Notes
Building a house of cards on a solid database foundation.
Lead Cloud database Engineering for Netflix.
Among other things, we offer C* as a service within Netflix.
Feel free to follow me on Twitter or link up on LinkedIn.
Talk about the Simian Army
- introduce simian army
Netflix LOVES chaos. We love it so much that we generate it.
Monkey - run in prod
Kong - Exercice
We run it on most of Netflix services, and even on C*
Talk about the Simian Army
- introduce simian army
Netflix LOVES chaos. We love it so much that we generate it.
Monkey - run in prod
Kong - Exercice
We run it on most of Netflix services, and even on C*
Talk about the Simian Army
- introduce simian army
Netflix LOVES chaos. We love it so much that we generate it.
Monkey - run in prod
Kong - Exercice
We run it on most of Netflix services, and even on C*
CDE has Chaos Monkey enabled on our C* clusters
Maximum 1 node per day, during business hours
Our Healthcheck dectects the missing instance and replaces it
218 C* nodes rebooted
22 nodes didn’t start and were automatically terminated by AWS internal healthcheck
Our heathcheck identified the missing nodes and automatically remediated the issue
0 downtime
- Bunch of Python/Shell scripts
- Jenkins as job scheduler (HC, node-replacements, repairs, upgrades and etc)
- On C* nodes: C* + Priam
- Is something missing? Monitoring? OpsCenter?
- Why not OpsCenter?
- Didn’t exist when Netflix started using C*
- Redundant in our stack
( continuation on why not OpsCenter)
- change slide according to christos's feedback
- Atlas is already a very powerful metrics and alerting tool, and our metric systems add non-C* related metrics (App metrics for example) that help in correlation. Alerts can be a combination of C* and App metrics.
- How it behaved during the Re:boot
- How did the healthcheck behave, how does it work and react to Chaos Monkey
( continuation on why not OpsCenter)
Atlas is already a very powerful metrics and alerting tool, and our metric systems add non-C* related metrics (App metrics for example) that help in correlation.
( continuation on why not OpsCenter)
Alerts can be a combination of C* and App metrics.
Healthcheck flow
2 scenarios are automatically remediated
How did the healthcheck behave during Re:boot
HC - Big monolith
About 100k lines of Python/Bash scripts
Hard to maintain
Lack of chaining (statefulness: if this job failed run that, else…)
Stateless
Lack of native support for TRIGGERING jobs based on events, like listening to SQS queues
High Availability: The Jenkins master node is a Single Point of Failure
Long running processes may crash due to a transient connection issue between the slave & the master
High Availability: The Jenkins master node is a Single Point of Failure
Long running processes may crash due to a transient connection issue between the slave & the master
What we learned, and what we decided to focus on (Principles)
What others are doing:
Facebook (FBAR) / LinkedIn (Nurse) / DropBox (Naoru)
Do our own or adopt existing solution?
We started with our own POC, then we decided to go with Stackstorm- event-driven automation platform
Facilitated Troubleshooting/Event handling
Automated remediation (Discovery example)
Do our own or adopt existing solution?
We started with our own POC, then we decided to go with Stackstorm- event-driven automation platform
Facilitated Troubleshooting/Event handling
Automated remediation (Discovery example)
What we decided to do:
new env
SackStorm-desc (rules/actions…)
Example of the Disk Space Alert
gap recap
Idempotence (make a stateless system feels like a stateful system)
Automation tools need to assure that you reach a certain state
Example:
Downloading the C* tarball: First, check the nodetool version
K.I.S.S. - “Simplicity is the ultimate sophistication” (Example: Resumable repairs - make more concise)
Prefer HTTP over SSH and Async over Sync
Retries with Timeouts and exponential back-off
Serving-fallbacks
Example:
Dynamic property service with hard-coded defaults
Netflix personalized recommendations falling back to default recommendations
Audit trail: use logstash to index data into Elasticsearch for Trend Analysis
- Talk about the fact that we already use LogStash @ Netflix, but we want to plug it into our automated remediation system
Metadata / Statistics / Long term metrics
Use Trend Analysis to be proactive instead of reactive:
Disk usage to predict when we need to increase the cluster size with automated resizing
Lead Cloud database Engineering for Netflix.
Among other things, we offer C* as a service within Netflix.
Feel free to follow me on Twitter or link up on LinkedIn.