Usually, companies have a clear separation between testing environments and development environments. But what if we take a different approach and combine everything into one environment and still survive? The speaker will discuss ways to organize cloud environments for testing and development, compare them, and discuss the pros and cons of the combined approach.
A deck from the first CDIsrael meetup, presenting our CD flow at Snyk, focusing on our testing framework. A day in a life of a developer - code, test, publish, deploy, monitor.
The document discusses MockServer-driven testing (MSDT), a methodology for developing automated UI tests using a mock server. The main ideas are to create a mock server that replaces the real backend and cover features supported by the mock server with UI tests. Key benefits include faster, independent tests and the ability to test more scenarios. Potential downsides include costs of maintaining the mock server and possibility of bugs. Examples of use cases and implementing a proof of concept mock server for chat messaging are provided.
Join this info-packed and hands-on workshop where we will cover:
Introduction to Kubernetes & GitOps talk:
We'll cover the most popular path that has brought success to many users already - GitOps as a natural evolution of Kubernetes. We'll give an overview of how you can benefit from Kubernetes and GitOps: greater security, reliability, velocity and more. Importantly, we cover definitions and principles standardized by the CNCF's OpenGitOps group and what it means for you.
Get Started with GitOps:
You'll have GitOps up and running in about 30 mins using our free and open source tools! We'll give a brief vision of where you want to be with those security, reliability, and velocity benefits, and then we'll support you while go through the getting started steps. During the workshop, you'll also experience in action and see demos for:
* an opinionated repo structure to minimize decision fatigue
* disaster recovery using GitOps
* Helm charts example
* Multi-cluster example
* all with free and open source tools mostly in the CNCF (eg. Flux and Helm).
If you have questions before or after the workshop, talk to us at #weave-gitops http://bit.ly/WeaveGitOpsSlack (If you need to invite yourself to the Slack, visit https://slack.weave.works/)
Vskills certification for The Grinder Testing Professional assesses the candidate as per the company’s need for load testing web applications. The certification tests the candidates on various areas in agents, workers, properties file, logging, console, TCPProxy, scripts, Jython, Clojure, instrumentation, script gallery, plug-ins, statistics, SSL and garbage collection.
Pluggable Infrastructure with CI/CD and DockerBob Killen
The docker cluster ecosystem is still young, and highly modular. This presentation covers some of the challenges we faced deciding on what infrastructure to deploy, and a few tips and tricks in making both applications and infrastructure easily adaptable.
The journey to Native Cloud Architecture & Microservices, tracing the footste...Mek Srunyu Stittri
The document discusses Netflix's adoption of microservices and continuous delivery to improve speed and agility. Key points include:
1) Netflix moved to microservices and continuous delivery on the cloud to dramatically speed up product development and deployment.
2) This allowed independent teams to deploy code frequently without coordination, with automated testing and deployment replacing handoffs and long release cycles.
3) Netflix's approach involved building stateless, independently deployable microservices; continuous monitoring; and other techniques to enable developers to deploy code safely and rapidly.
This document discusses automating development operations and introduces seven capabilities for effective DevOps: isolation, portability, automated configuration, simplified code reviews, continuous integration and testing, atomic deployments, and automated monitoring. For each capability, it outlines challenges and lists common automation options and tools to address those challenges, noting pros and cons of different approaches. It emphasizes that automation should adapt to existing workflows, scale gradually, and not reduce developer productivity.
Orchestrating Automated Tests in Different Systems - Please Download to watch...Kelvin Silva
This talk was presented in DevOpsDays Sao Paulo (June/2018) and in Daitan DevWeek (July/2018).
What should you do if your product must be tested on multiple operating systems before delivering? And what if it must be tested on different browsers? Now it is a lot of combinations. And how to do this in an agile way?
Here is a talk about how we can orchestrate automated tests on different systems, on different browsers. Assuring the coverage on all operating systems and browsers supported by a product is one of the toughest tasks commonly faced when transitioning to a DevOps setup.
Due to the high quantity of environment combinations, the complexity of organizing and executing tests increases and can lead to an ineffective pipeline. In this talk, we will evaluate an automated orchestration of complex test suites that delivers the effectiveness needed for proper DevOps.
A deck from the first CDIsrael meetup, presenting our CD flow at Snyk, focusing on our testing framework. A day in a life of a developer - code, test, publish, deploy, monitor.
The document discusses MockServer-driven testing (MSDT), a methodology for developing automated UI tests using a mock server. The main ideas are to create a mock server that replaces the real backend and cover features supported by the mock server with UI tests. Key benefits include faster, independent tests and the ability to test more scenarios. Potential downsides include costs of maintaining the mock server and possibility of bugs. Examples of use cases and implementing a proof of concept mock server for chat messaging are provided.
Join this info-packed and hands-on workshop where we will cover:
Introduction to Kubernetes & GitOps talk:
We'll cover the most popular path that has brought success to many users already - GitOps as a natural evolution of Kubernetes. We'll give an overview of how you can benefit from Kubernetes and GitOps: greater security, reliability, velocity and more. Importantly, we cover definitions and principles standardized by the CNCF's OpenGitOps group and what it means for you.
Get Started with GitOps:
You'll have GitOps up and running in about 30 mins using our free and open source tools! We'll give a brief vision of where you want to be with those security, reliability, and velocity benefits, and then we'll support you while go through the getting started steps. During the workshop, you'll also experience in action and see demos for:
* an opinionated repo structure to minimize decision fatigue
* disaster recovery using GitOps
* Helm charts example
* Multi-cluster example
* all with free and open source tools mostly in the CNCF (eg. Flux and Helm).
If you have questions before or after the workshop, talk to us at #weave-gitops http://bit.ly/WeaveGitOpsSlack (If you need to invite yourself to the Slack, visit https://slack.weave.works/)
Vskills certification for The Grinder Testing Professional assesses the candidate as per the company’s need for load testing web applications. The certification tests the candidates on various areas in agents, workers, properties file, logging, console, TCPProxy, scripts, Jython, Clojure, instrumentation, script gallery, plug-ins, statistics, SSL and garbage collection.
Pluggable Infrastructure with CI/CD and DockerBob Killen
The docker cluster ecosystem is still young, and highly modular. This presentation covers some of the challenges we faced deciding on what infrastructure to deploy, and a few tips and tricks in making both applications and infrastructure easily adaptable.
The journey to Native Cloud Architecture & Microservices, tracing the footste...Mek Srunyu Stittri
The document discusses Netflix's adoption of microservices and continuous delivery to improve speed and agility. Key points include:
1) Netflix moved to microservices and continuous delivery on the cloud to dramatically speed up product development and deployment.
2) This allowed independent teams to deploy code frequently without coordination, with automated testing and deployment replacing handoffs and long release cycles.
3) Netflix's approach involved building stateless, independently deployable microservices; continuous monitoring; and other techniques to enable developers to deploy code safely and rapidly.
This document discusses automating development operations and introduces seven capabilities for effective DevOps: isolation, portability, automated configuration, simplified code reviews, continuous integration and testing, atomic deployments, and automated monitoring. For each capability, it outlines challenges and lists common automation options and tools to address those challenges, noting pros and cons of different approaches. It emphasizes that automation should adapt to existing workflows, scale gradually, and not reduce developer productivity.
Orchestrating Automated Tests in Different Systems - Please Download to watch...Kelvin Silva
This talk was presented in DevOpsDays Sao Paulo (June/2018) and in Daitan DevWeek (July/2018).
What should you do if your product must be tested on multiple operating systems before delivering? And what if it must be tested on different browsers? Now it is a lot of combinations. And how to do this in an agile way?
Here is a talk about how we can orchestrate automated tests on different systems, on different browsers. Assuring the coverage on all operating systems and browsers supported by a product is one of the toughest tasks commonly faced when transitioning to a DevOps setup.
Due to the high quantity of environment combinations, the complexity of organizing and executing tests increases and can lead to an ineffective pipeline. In this talk, we will evaluate an automated orchestration of complex test suites that delivers the effectiveness needed for proper DevOps.
The document discusses options for testing software including manual testing with Vagrant and Robot Framework as well as automated testing. It recommends using OpenShift for automated testing as it provides a reliable solution with Kubernetes that is scalable and controllable while Docker provides speed and high density. Open vSwitch enables network connectivity in OpenShift. Workflow and reference details are also included.
Intro to GitOps with Weave GitOps, Flagger and LinkerdWeaveworks
This document provides an overview of GitOps, service meshes, Linkerd, Flux, Weave GitOps and progressive delivery. It introduces the speakers and outlines the agenda which includes explanations of GitOps, service meshes, Linkerd and Weave GitOps. It then demonstrates how Weave GitOps and Linkerd can be used together for progressive delivery and provides a Q&A section at the end.
Free GitOps Workshop (with Intro to Kubernetes & GitOps)Weaveworks
View this video on Youtube here: https://youtu.be/tK4S8y3j5TA
In this info-packed and hands-on workshop we covered:
Introduction to Kubernetes & GitOps talk:
We covered the most popular path that has brought success to many users already - GitOps as a natural evolution of Kubernetes. We'll give an overview of how you can benefit from Kubernetes and GitOps: greater security, reliability, velocity and more. Importantly, we cover definitions and principles standardized by the CNCF's OpenGitOps group and what it means for you.
Get Started with GitOps:
You'll have GitOps up and running in about 30 mins using our free and open source tools! We'll give a brief vision of where you want to be with those security, reliability, and velocity benefits, and then we'll support you while go through the getting started steps. During the workshop, you'll also experience in action and see demos for:
- an opinionated repo structure to minimize decision fatigue
- disaster recovery using GitOps
- Helm charts example
- Multi-cluster example
- all with free and open source tools mostly in the CNCF (eg. Flux and Helm).
If you have questions before or after the workshop, talk to us at #weave-gitops http://bit.ly/WeaveGitOpsSlack (If you need to invite yourself to the Slack, visit https://slack.weave.works/)
The DevOps methodology integrates development and operations so that system changes can get rolled out quickly without causing unplanned downtime. Industrial organizations that successfully implement DevOps will have a strong advantage, but knowing how to get started can be a real challenge.
The presentation which I was using during my talk at EPAM Lviv JS community about offline-first applications. Contains high-level review of tools and web platform to submerge folks in a world of offline-first thinking.
apidays LIVE New York - Navigating the Sea of Javascript Tools to Discover Sc...apidays
apidays LIVE New York - API for Legacy Industries: Banking, Insurance, Healthcare and Retail
Navigating the Sea of Javascript Tools to Discover Scalable Tools for Continuous Delivery
Menelaos Kotsollaris, Senior Software Engineer
Viki Green, Senior Software Developer at Trulioo
1. The document discusses building a small data center network for Dyn, focusing on lessons learned from two design iterations.
2. The first design used MPLS VPNs but had issues with routing and IPv6 support. The second design used virtual routing and forwarding with multiple routing tables to separate traffic and improve service mobility.
3. Key lessons included validating designs before deploying, automating network operations, and moving security policies to instances rather than the network to improve agility and isolate impacts.
Montreal Kubernetes Meetup: Developer-first workflows (for microservices) on ...Ambassador Labs
1. The document discusses developer-first workflows for building and operating microservices on Kubernetes.
2. It recommends creating self-sufficient, autonomous teams and using Kubernetes, Docker, and Envoy to provide the basic infrastructure primitives needed for distributed workflows.
3. The strategies suggested depend on the service maturity level and include using similar development and production environments for prototyping, implementing software redundancy for production services, and defining service level objectives and network observability for internal dependencies.
Dive deep into specific OSS packages to examine the top issues in the enterprise with two of our most qualified OSS architects, Bill Crowell and Vince Cox walkthrough: Their day-to-day work in OSS packages; ways to fix reported issues; why you can’t expect in-house developers to handle issues in OSS packages.
Things You MUST Know Before Deploying OpenStack: Bruno Lago, Catalyst ITOpenStack
Audience: Advanced
About: Real world lessons and war stories about Catalyst IT’s experience in rolling out an OpenStack based public cloud in New Zealand.
This presentation will provide tips and advice that may save you a lot of time, money and nights of sleep if you are planning to run OpenStack in the future. It may also bring some insights to people that are already running OpenStack in production.
Topics covered will include: selection of hardware for optimal costs, techniques that drive quality and service levels up, common deployment mistakes, in place upgrades, how to identify the maturity level of each project and decide what is ready for production, and much more!
Speaker Bio: Bruno Lago – Entrepreneur, Catalyst IT Limited
Bruno Lago is a solutions architect that has been involved with the Catalyst Cloud (New Zealand’s first public cloud based on OpenStack) from its inception. He is passionate about open source software, cloud computing and disruptive technologies.
OpenStack Australia Day - Sydney 2016
https://events.aptira.com/openstack-australia-day-sydney-2016/
Antifragility and testing for distributed systems failureDiUS
Failure is inevitable. In our modern world filled with continuously delivered and increasingly complex distributed architectures (looking at you micro-services), it is important to be able to test and improve our systems under a range of failure conditions.
In this talk, Matt discusses these complexities and the forces they exert on development teams, presenting some simple strategies and practical advice to deal with them.
Do you have a healthy CI/CD pipeline? Do releases simply flow through? CI, CD, PRs, Pipelines, Releases, Deployments and all that jazz.
Whether you're new to Continuous Delivery or a hardened traveller down that road, this session has something for you. We’ll start with an exploration of branching strategy (releaseflow.org) before walking through a healthy continuous delivery configuration.
We’ll watch a code change make it's way through a pipeline to production and discuss how we can apply such practices to our everyday work.
Weave GitOps 2022.09 Release: A Fast & Reliable Path to Production with Progr...Weaveworks
Weave GitOps 2022.09 Features Launch Event
The latest release of Weave GitOps introduces new features enabling progressive delivery, policy as code, and accelerated application onboarding.
Weave GitOps is the leading full-stack GitOps platform to automate trusted application delivery and secure infrastructure operations on premise, in the cloud and at the edge. Trusted by Customers, including Deutsche Telekom and The Department of Defense, Platform and Application Teams, Weave GitOps unlocks the benefits of increased efficiency and compliance, while boosting deployment velocity and confidence.
Join us where we’ll do a live demo of Weave GitOps showcasing:
- Advanced Deployment Patterns—Progressive Delivery has never been easier
- Multi-tenancy and Application Portability—More collaboration and control
- Strengthened GitOps Security—If you can code it, you can secure it.
The monolith to cloud-native, microservices evolution has driven a shift from monitoring to observability. OpenTelemetry, a merger of the OpenTracing and OpenCensus projects, is enabling Observability 2.0. This talk gives an overview of the OpenTelemetry project and then outlines some production-proven architectures for improving the observability of your applications and systems.
Learnings from the Field. Lessons from Working with Dozens of Small & Large D...HostedbyConfluent
- Upgrades should be done often to get bug fixes and improvements, following the upgrade guide carefully. Start with a healthy cluster and upgrade components outward from Zookeeper to Kafka brokers to clients. Don't rush the process or have any unresolved partition reassignments.
- Collect JMX metrics to monitor the cluster as outages can be prolonged without visibility. The Kafka defaults are suitable for single node deployments but replication factor, threads, and broker configuration should be tuned for larger clusters.
- Quotas like replication throttling and bandwidth/request limits per client or topic should be used to protect the cluster and clients. Log files should separate each component and be retained for a few days. Consider multiple clusters by SLA
This document summarizes Netflix's use of Kafka in their data pipeline. It discusses how Netflix evolved from using S3 and EMR to introducing Kafka and Kafka producers and consumers to handle 400 billion events per day. It covers challenges of scaling Kafka clusters and tuning Kafka clients and brokers. Finally, it outlines Netflix's roadmap which includes contributing to open source projects like Kafka and testing failure resilience.
This document summarizes Netflix's use of Kafka in their data pipeline. It discusses the evolution of Netflix's data pipeline to incorporate Kafka to handle 400 billion events per day. It describes how Netflix uses Kafka clusters with different priorities and configurations. It also outlines some of the challenges of using Kafka at Netflix's scale, such as Zookeeper client issues and cluster scaling, and the solutions Netflix developed to address these challenges.
There is a problem of finding the best instance of a service in distributed systems with dynamic configuration. Nowadays, there are many products for the configuration storage and service discovery. It should be mentioned at least Netflix Eureka, Consul, etc or good old Zookeeper. These products can keep and give configuration, manage service instances lifecycle and some of them even can be as dynamic DNS service. But main question is not about what instance may be called at the certain time. It is about what instance is better for call? This means that smart load balancing top on service discovery is required. Spring Cloud project allows to integrate these products to your project and provides powerful solutions for typical problems, that make cloud native services developing easier. This talk will review the internal structure of SpringCloud implementation of Client-Side Service Discovery and Client Load Balancing patterns. It also will include specific details of concrete implementations with examples from official libraries and the author’s own library.
Процесс разработки не начинается и не заканчивается на написании кода программного продукта. Мы пишем документацию, придумываем, как это всё оттестировать, и заботимся о том, чтобы доступность приложения была на высоком уровне.
Мы все делаем привычные вещи привычным для нас способом. Порой выполняя много ручной и неэффективной работы. Но что, если есть другой, радикальный подход. Можно ли формализовать свою деятельность и переложить её в код? Какие практики и инструменты для этого использовать?
В докладе будет представлен личный опыт автора по автоматизации различных элементов разработки ПО.
More Related Content
Similar to Cocktail of Environments. How to Mix Test and Development Environments and Stay Alive
The document discusses options for testing software including manual testing with Vagrant and Robot Framework as well as automated testing. It recommends using OpenShift for automated testing as it provides a reliable solution with Kubernetes that is scalable and controllable while Docker provides speed and high density. Open vSwitch enables network connectivity in OpenShift. Workflow and reference details are also included.
Intro to GitOps with Weave GitOps, Flagger and LinkerdWeaveworks
This document provides an overview of GitOps, service meshes, Linkerd, Flux, Weave GitOps and progressive delivery. It introduces the speakers and outlines the agenda which includes explanations of GitOps, service meshes, Linkerd and Weave GitOps. It then demonstrates how Weave GitOps and Linkerd can be used together for progressive delivery and provides a Q&A section at the end.
Free GitOps Workshop (with Intro to Kubernetes & GitOps)Weaveworks
View this video on Youtube here: https://youtu.be/tK4S8y3j5TA
In this info-packed and hands-on workshop we covered:
Introduction to Kubernetes & GitOps talk:
We covered the most popular path that has brought success to many users already - GitOps as a natural evolution of Kubernetes. We'll give an overview of how you can benefit from Kubernetes and GitOps: greater security, reliability, velocity and more. Importantly, we cover definitions and principles standardized by the CNCF's OpenGitOps group and what it means for you.
Get Started with GitOps:
You'll have GitOps up and running in about 30 mins using our free and open source tools! We'll give a brief vision of where you want to be with those security, reliability, and velocity benefits, and then we'll support you while go through the getting started steps. During the workshop, you'll also experience in action and see demos for:
- an opinionated repo structure to minimize decision fatigue
- disaster recovery using GitOps
- Helm charts example
- Multi-cluster example
- all with free and open source tools mostly in the CNCF (eg. Flux and Helm).
If you have questions before or after the workshop, talk to us at #weave-gitops http://bit.ly/WeaveGitOpsSlack (If you need to invite yourself to the Slack, visit https://slack.weave.works/)
The DevOps methodology integrates development and operations so that system changes can get rolled out quickly without causing unplanned downtime. Industrial organizations that successfully implement DevOps will have a strong advantage, but knowing how to get started can be a real challenge.
The presentation which I was using during my talk at EPAM Lviv JS community about offline-first applications. Contains high-level review of tools and web platform to submerge folks in a world of offline-first thinking.
apidays LIVE New York - Navigating the Sea of Javascript Tools to Discover Sc...apidays
apidays LIVE New York - API for Legacy Industries: Banking, Insurance, Healthcare and Retail
Navigating the Sea of Javascript Tools to Discover Scalable Tools for Continuous Delivery
Menelaos Kotsollaris, Senior Software Engineer
Viki Green, Senior Software Developer at Trulioo
1. The document discusses building a small data center network for Dyn, focusing on lessons learned from two design iterations.
2. The first design used MPLS VPNs but had issues with routing and IPv6 support. The second design used virtual routing and forwarding with multiple routing tables to separate traffic and improve service mobility.
3. Key lessons included validating designs before deploying, automating network operations, and moving security policies to instances rather than the network to improve agility and isolate impacts.
Montreal Kubernetes Meetup: Developer-first workflows (for microservices) on ...Ambassador Labs
1. The document discusses developer-first workflows for building and operating microservices on Kubernetes.
2. It recommends creating self-sufficient, autonomous teams and using Kubernetes, Docker, and Envoy to provide the basic infrastructure primitives needed for distributed workflows.
3. The strategies suggested depend on the service maturity level and include using similar development and production environments for prototyping, implementing software redundancy for production services, and defining service level objectives and network observability for internal dependencies.
Dive deep into specific OSS packages to examine the top issues in the enterprise with two of our most qualified OSS architects, Bill Crowell and Vince Cox walkthrough: Their day-to-day work in OSS packages; ways to fix reported issues; why you can’t expect in-house developers to handle issues in OSS packages.
Things You MUST Know Before Deploying OpenStack: Bruno Lago, Catalyst ITOpenStack
Audience: Advanced
About: Real world lessons and war stories about Catalyst IT’s experience in rolling out an OpenStack based public cloud in New Zealand.
This presentation will provide tips and advice that may save you a lot of time, money and nights of sleep if you are planning to run OpenStack in the future. It may also bring some insights to people that are already running OpenStack in production.
Topics covered will include: selection of hardware for optimal costs, techniques that drive quality and service levels up, common deployment mistakes, in place upgrades, how to identify the maturity level of each project and decide what is ready for production, and much more!
Speaker Bio: Bruno Lago – Entrepreneur, Catalyst IT Limited
Bruno Lago is a solutions architect that has been involved with the Catalyst Cloud (New Zealand’s first public cloud based on OpenStack) from its inception. He is passionate about open source software, cloud computing and disruptive technologies.
OpenStack Australia Day - Sydney 2016
https://events.aptira.com/openstack-australia-day-sydney-2016/
Antifragility and testing for distributed systems failureDiUS
Failure is inevitable. In our modern world filled with continuously delivered and increasingly complex distributed architectures (looking at you micro-services), it is important to be able to test and improve our systems under a range of failure conditions.
In this talk, Matt discusses these complexities and the forces they exert on development teams, presenting some simple strategies and practical advice to deal with them.
Do you have a healthy CI/CD pipeline? Do releases simply flow through? CI, CD, PRs, Pipelines, Releases, Deployments and all that jazz.
Whether you're new to Continuous Delivery or a hardened traveller down that road, this session has something for you. We’ll start with an exploration of branching strategy (releaseflow.org) before walking through a healthy continuous delivery configuration.
We’ll watch a code change make it's way through a pipeline to production and discuss how we can apply such practices to our everyday work.
Weave GitOps 2022.09 Release: A Fast & Reliable Path to Production with Progr...Weaveworks
Weave GitOps 2022.09 Features Launch Event
The latest release of Weave GitOps introduces new features enabling progressive delivery, policy as code, and accelerated application onboarding.
Weave GitOps is the leading full-stack GitOps platform to automate trusted application delivery and secure infrastructure operations on premise, in the cloud and at the edge. Trusted by Customers, including Deutsche Telekom and The Department of Defense, Platform and Application Teams, Weave GitOps unlocks the benefits of increased efficiency and compliance, while boosting deployment velocity and confidence.
Join us where we’ll do a live demo of Weave GitOps showcasing:
- Advanced Deployment Patterns—Progressive Delivery has never been easier
- Multi-tenancy and Application Portability—More collaboration and control
- Strengthened GitOps Security—If you can code it, you can secure it.
The monolith to cloud-native, microservices evolution has driven a shift from monitoring to observability. OpenTelemetry, a merger of the OpenTracing and OpenCensus projects, is enabling Observability 2.0. This talk gives an overview of the OpenTelemetry project and then outlines some production-proven architectures for improving the observability of your applications and systems.
Learnings from the Field. Lessons from Working with Dozens of Small & Large D...HostedbyConfluent
- Upgrades should be done often to get bug fixes and improvements, following the upgrade guide carefully. Start with a healthy cluster and upgrade components outward from Zookeeper to Kafka brokers to clients. Don't rush the process or have any unresolved partition reassignments.
- Collect JMX metrics to monitor the cluster as outages can be prolonged without visibility. The Kafka defaults are suitable for single node deployments but replication factor, threads, and broker configuration should be tuned for larger clusters.
- Quotas like replication throttling and bandwidth/request limits per client or topic should be used to protect the cluster and clients. Log files should separate each component and be retained for a few days. Consider multiple clusters by SLA
This document summarizes Netflix's use of Kafka in their data pipeline. It discusses how Netflix evolved from using S3 and EMR to introducing Kafka and Kafka producers and consumers to handle 400 billion events per day. It covers challenges of scaling Kafka clusters and tuning Kafka clients and brokers. Finally, it outlines Netflix's roadmap which includes contributing to open source projects like Kafka and testing failure resilience.
This document summarizes Netflix's use of Kafka in their data pipeline. It discusses the evolution of Netflix's data pipeline to incorporate Kafka to handle 400 billion events per day. It describes how Netflix uses Kafka clusters with different priorities and configurations. It also outlines some of the challenges of using Kafka at Netflix's scale, such as Zookeeper client issues and cluster scaling, and the solutions Netflix developed to address these challenges.
Similar to Cocktail of Environments. How to Mix Test and Development Environments and Stay Alive (20)
There is a problem of finding the best instance of a service in distributed systems with dynamic configuration. Nowadays, there are many products for the configuration storage and service discovery. It should be mentioned at least Netflix Eureka, Consul, etc or good old Zookeeper. These products can keep and give configuration, manage service instances lifecycle and some of them even can be as dynamic DNS service. But main question is not about what instance may be called at the certain time. It is about what instance is better for call? This means that smart load balancing top on service discovery is required. Spring Cloud project allows to integrate these products to your project and provides powerful solutions for typical problems, that make cloud native services developing easier. This talk will review the internal structure of SpringCloud implementation of Client-Side Service Discovery and Client Load Balancing patterns. It also will include specific details of concrete implementations with examples from official libraries and the author’s own library.
Процесс разработки не начинается и не заканчивается на написании кода программного продукта. Мы пишем документацию, придумываем, как это всё оттестировать, и заботимся о том, чтобы доступность приложения была на высоком уровне.
Мы все делаем привычные вещи привычным для нас способом. Порой выполняя много ручной и неэффективной работы. Но что, если есть другой, радикальный подход. Можно ли формализовать свою деятельность и переложить её в код? Какие практики и инструменты для этого использовать?
В докладе будет представлен личный опыт автора по автоматизации различных элементов разработки ПО.
There is a problem of finding the correct operating services In a distributed systems with dynamic configuration. Currently, there are designed one and more solutions for the ever-changing storage configuration. It should be mentioned at least Netflix Eureka, Consul, etcd or good old Zookeeper. Spring Cloud project allows to integrate some of these solutions to your project and provides powerful solutions for typical problems. However, on the way to unicorns not the most obvious subtleties of implementation and associated problems of use in real projects wait for a developer.
This talk will review the internal structure SpringCloud, implementation of Client-Side Service Discovery pattern, and specific details of concrete implementations on the example of the official libraries and the author's own library.
Сейчас только ленивый не говорит про DevOps, краеугольным камнем которого является организация потока непрерывной доставки ценности клиенту. Continuous Delivery перестаёт быть опцией и становится обязательным требованием.
В докладе будут рассмотрены:
- общие подходы к организации Continuous Delivery на базе Jenkins-а в совсем не тепличных условиях
- практики и подходы, которые позволяют быстро настраивать и собирать десятки микросервисов
- подводные камни, с которыми пришлось столкнуться, и способы борьбы с ними
Everyone has heard about microservices. Someone tries to implement them in practice. And only the bravests of us have already used them in production environment. But then why so many people hype around microservices, if the idea is not quite new? What are microservices? What is the difference between it and good old SOA approach? How can developers create modern enterprise applications in Java easily with the help of this approach?
Our guests should try to answer these questions and share their own experience related to live coding of imaginary startup ‘Hippos’.
The following topics will be covered:
What are microservices? Where is the theory, bro?
What kind of technologies should we choose? What have we chosen and why?
Why RPC is still competetive in REST-domination era?
How to pack and distribute microservices? How can SpringBoot and Docker help us to solve our problems?
Why is service discovery considered as one of the crucial components? How to cook Spring Cloud and what problems you may face in real life?
Security of microservices, API gateway and other.
Speakers will also cover other topics related to distributed system development.
Everyone has heard about microservices. Someone tries to implement them in practice. And only the bravests of us have already used them in production environment. But then why so many people hype around microservices, if the idea is not quite new? What are microservices? What is the difference between it and good old SOA approach? How can developers create modern enterprise applications in Java easily with the help of this approach?
Our guests should try to answer these questions and share their own experience related to live coding of imaginary startup 'Hippos'. The following topics will be covered:
What are microservices? Where is the theory, bro?
What kind of technologies should we choose? What have we chosen and why?
Why RPC is still competetive in REST-domination era?
How to pack and distribute microservices? How can SpringBoot and Docker help us to solve our problems?
Why is service discovery considered as one of the crucial components? How to cook Spring Cloud and what problems you may face in real life?
Security of microservices, API gateway and other.
Speakers will also cover other topics related to distributed system development.
Сколько раз мы собираем ПО каждый день? Сколько раз за день код компилируется, тестируется и превращается в артефакты? Какие разнообразные сценарии возможны в казалось бы простой цепочке compile-test-assemble? Билд-системы из узкоспециализованных утилит превратились в мощный инструмент сборки, валидации, дистрибуции и автоматизации наших программных продуктов.
Это будет тематический доклад о том как настроить инструмент сборки для себя, а не против себя. Мы рассмотрим основные принципы работы Gradle DSL и возможности его расширения. Как и на чём их писать, как тестировать и как делиться результатами своего труда со своими коллегами.
Вместе с вами мы напишем автоматизированный сценарий сборки для проекта и покажем как ловко можно вставлять костыли всю силу и мощь Gradle 2.x.
Любите ли вы велосипеды? Все разработчики любят свои ненаколеночныерешения велосипеды! И мы не исключение. В нашем докладе мы покажем как собирать, сколачивать, вылепливать собственный велосипед так, чтобы на нем потом могла ездить без слёз вся команда, компания, или может весь мир.
Что в докладе будет:
- много Spring Boot-а;
- live coding;
- создание собственного Spring Boot Starter-а;
- Apache Thrift в качестве подопытного кролика.
Чего не будет:
- бенчмарков и сравнений Thrift vs REST vs gRPC vs XXX.
Микросервисы, кто-то только слышал о них, кто-то пытался делать, кто-то уже использует в продакшене. Идеи, заложенные в концепцию микросервисов, не новы и основные постулаты уже звучали раньше. Так почему же в последнее время мы всё чаще слышим о микросервисах? Что такое микросервисы для нас и чем они отличаются от старого доброго подхода SOA? Как теперь разрабатывать enterprise-приложения с микросервисным подходом на нашем любимом языке программирования Java?
На эти и некоторые другие вопросы постараемся ответить во время встречи. Наши гости, Кирилл Толкачёв и Александр Тарасов, в режиме live coding попытаются создать небольшой стартап, попутно использовав новомодные подходы и инструменты.
На пути к релизу стартапа будут затронуты основные проблемы выбранных подходов в целом и технологий в частности:
Микросервис — что это, для чего и как с этим дальше жить. Где теория брат? ;)
На чём писать API: REST или RPC, и почему Thrift имеет право на жизнь в эпоху тотального распространения JSON-а. Упрощай и превозмогай с помощью Spring boot starter;
Какой стек выбрать для разработки, что выбрали мы и почему. Небольшое сравнение легковесных и не очень java фреймворков а так же сопутствующих инструментов;
Способы упаковки, дистрибуции и разворачивания микросервисов, как Spring Boot и Docker помогают нам в решении этих непростых для разработчика проблемах;
Как микросервисам найти друг друга, как готовить Spring Cloud и как обойти существующие проблемы и ограничения. Не доверяйте технологиям, доверяйте только себе;
API Gateway. Предохраняй и сохраняй свои микросервисы.
Так же речь пойдет о других распространенных проблемах распределенных систем и их решениях.
Docker on a local machine and Docker in production — are two big differences. It's easy to play with technology but it's hard to do something real for many customers.
Half a year ago inside of AlfaLab (division of Alfa-Bank) we've started building architecture with microservices approach for few our pilot projects. We've almost completely changed a stack of the used technologies on a frontend and significantly changed it on a middle layer. For package and distribution we have choosen Docker. Two months ago we've deployed project to production and have opened service for clients.
In the report the following topics will be covered:
- reasons of a choice Docker;
- why Docker without other tools is not enough for a production;
- what stack of technologies we used in our solution;
- what advantages we've got;
- what problems have been faced and how we've solved them.
Docker on a local machine and Docker in production — are two big differences. It's easy to play with technology but it's hard to do something real for many customers.
Half a year ago inside of Alpha Laboratory (division of Alfa-Bank) we've started building new microservices architecture for one of our pilot projects. We've almost completely changed a stack of the used technologies on a frontend and significantly changed it on a middle layer. For package and distribution we have choosen Docker. Two months ago we've deployed project to production and have opened service for clients.
In the report the following topics will be covered:
- reasons of a choice Docker;
- why Docker without other tools is not enough for a production;
- what stack of technologies we used in our solution;
- what advantages we've got;
- what problems have been faced and how we've solved them.
Conversational agents, or chatbots, are increasingly used to access all sorts of services using natural language. While open-domain chatbots - like ChatGPT - can converse on any topic, task-oriented chatbots - the focus of this paper - are designed for specific tasks, like booking a flight, obtaining customer support, or setting an appointment. Like any other software, task-oriented chatbots need to be properly tested, usually by defining and executing test scenarios (i.e., sequences of user-chatbot interactions). However, there is currently a lack of methods to quantify the completeness and strength of such test scenarios, which can lead to low-quality tests, and hence to buggy chatbots.
To fill this gap, we propose adapting mutation testing (MuT) for task-oriented chatbots. To this end, we introduce a set of mutation operators that emulate faults in chatbot designs, an architecture that enables MuT on chatbots built using heterogeneous technologies, and a practical realisation as an Eclipse plugin. Moreover, we evaluate the applicability, effectiveness and efficiency of our approach on open-source chatbots, with promising results.
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor IvaniukFwdays
At this talk we will discuss DDoS protection tools and best practices, discuss network architectures and what AWS has to offer. Also, we will look into one of the largest DDoS attacks on Ukrainian infrastructure that happened in February 2022. We'll see, what techniques helped to keep the web resources available for Ukrainians and how AWS improved DDoS protection for all customers based on Ukraine experience
Introduction of Cybersecurity with OSS at Code Europe 2024Hiroshi SHIBATA
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
Your One-Stop Shop for Python Success: Top 10 US Python Development Providersakankshawande
Simplify your search for a reliable Python development partner! This list presents the top 10 trusted US providers offering comprehensive Python development services, ensuring your project's success from conception to completion.
Ivanti’s Patch Tuesday breakdown goes beyond patching your applications and brings you the intelligence and guidance needed to prioritize where to focus your attention first. Catch early analysis on our Ivanti blog, then join industry expert Chris Goettl for the Patch Tuesday Webinar Event. There we’ll do a deep dive into each of the bulletins and give guidance on the risks associated with the newly-identified vulnerabilities.
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfChart Kalyan
A Mix Chart displays historical data of numbers in a graphical or tabular form. The Kalyan Rajdhani Mix Chart specifically shows the results of a sequence of numbers over different periods.
Programming Foundation Models with DSPy - Meetup SlidesZilliz
Prompting language models is hard, while programming language models is easy. In this talk, I will discuss the state-of-the-art framework DSPy for programming foundation models with its powerful optimizers and runtime constraint system.
Taking AI to the Next Level in Manufacturing.pdfssuserfac0301
Read Taking AI to the Next Level in Manufacturing to gain insights on AI adoption in the manufacturing industry, such as:
1. How quickly AI is being implemented in manufacturing.
2. Which barriers stand in the way of AI adoption.
3. How data quality and governance form the backbone of AI.
4. Organizational processes and structures that may inhibit effective AI adoption.
6. Ideas and approaches to help build your organization's AI strategy.
The Microsoft 365 Migration Tutorial For Beginner.pptxoperationspcvita
This presentation will help you understand the power of Microsoft 365. However, we have mentioned every productivity app included in Office 365. Additionally, we have suggested the migration situation related to Office 365 and how we can help you.
You can also read: https://www.systoolsgroup.com/updates/office-365-tenant-to-tenant-migration-step-by-step-complete-guide/
Monitoring and Managing Anomaly Detection on OpenShift.pdfTosin Akinosho
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...Alex Pruden
Folding is a recent technique for building efficient recursive SNARKs. Several elegant folding protocols have been proposed, such as Nova, Supernova, Hypernova, Protostar, and others. However, all of them rely on an additively homomorphic commitment scheme based on discrete log, and are therefore not post-quantum secure. In this work we present LatticeFold, the first lattice-based folding protocol based on the Module SIS problem. This folding protocol naturally leads to an efficient recursive lattice-based SNARK and an efficient PCD scheme. LatticeFold supports folding low-degree relations, such as R1CS, as well as high-degree relations, such as CCS. The key challenge is to construct a secure folding protocol that works with the Ajtai commitment scheme. The difficulty, is ensuring that extracted witnesses are low norm through many rounds of folding. We present a novel technique using the sumcheck protocol to ensure that extracted witnesses are always low norm no matter how many rounds of folding are used. Our evaluation of the final proof system suggests that it is as performant as Hypernova, while providing post-quantum security.
Paper Link: https://eprint.iacr.org/2024/257
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUpanagenda
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/
DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen!
Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell.
Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten.
Diese Themen werden behandelt
- Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten
- Wie funktionieren CCB- und CCX-Lizenzen wirklich?
- Verstehen des DLAU-Tools und wie man es am besten nutzt
- Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw.
- Praxisbeispiele und Best Practices zum sofortigen Umsetzen
Have you ever been confused by the myriad of choices offered by AWS for hosting a website or an API?
Lambda, Elastic Beanstalk, Lightsail, Amplify, S3 (and more!) can each host websites + APIs. But which one should we choose?
Which one is cheapest? Which one is fastest? Which one will scale to meet our needs?
Join me in this session as we dive into each AWS hosting service to determine which one is best for your scenario and explain why!
Main news related to the CCS TSI 2023 (2023/1695)Jakub Marek
An English 🇬🇧 translation of a presentation to the speech I gave about the main changes brought by CCS TSI 2023 at the biggest Czech conference on Communications and signalling systems on Railways, which was held in Clarion Hotel Olomouc from 7th to 9th November 2023 (konferenceszt.cz). Attended by around 500 participants and 200 on-line followers.
The original Czech 🇨🇿 version of the presentation can be found here: https://www.slideshare.net/slideshow/hlavni-novinky-souvisejici-s-ccs-tsi-2023-2023-1695/269688092 .
The videorecording (in Czech) from the presentation is available here: https://youtu.be/WzjJWm4IyPk?si=SImb06tuXGb30BEH .
21. Benefits
● Fast feedback loop
● Good for simple usecases
Drawbacks
● Hard to test complex scenarios
● Hard to collaborate
○ QA
○ other developers
Local Only
21
25. 25
Full Copy for each Developer
Own logical DB
Own topics and
subscriptions
26. Issues to
address
● Tool to establish “Vasya” env
○ 10-15 minutes max
● Handle the load
○ 10k workloads and much more
● Support separated infra components
○ dbs in containers
○ emulators for queues
26
27. Issues to
address
● Tool to establish “Vasya” env
○ 10-15 minutes max
● Handle the load
○ 10k workloads and much more
● Support separated infra components
○ dbs in containers
○ emulators for queues
27
28. Issues to
address
● Tool to establish “Vasya” env
○ 10-15 minutes max
● Handle the load
○ 10k workloads and much more
● Support separated infra components
○ dbs in containers
○ emulators for queues
28
29. Optimizations
● Part copy instead of full
○ core services first
○ specific services on-demand
● Shutdown every night
● Env per squad not developer
● Get rid of X
○ do not use service mesh
○ do not keep logs
29
30. Optimizations
● Part copy instead of full
○ core services first
○ specific services on-demand
● Shutdown every night
● Env per squad not developer
● Get rid of X
○ do not use service mesh
○ do not keep logs
30
31. Optimizations
● Part copy instead of full
○ core services first
○ specific services on-demand
● Shutdown every night
● Env per squad, not developer
● Get rid of X
○ do not use service mesh
○ do not keep logs
31
32. Optimizations
● Part copy instead of full
○ core services first
○ specific services on-demand
● Shutdown every night
● Env per squad, not developer
● Get rid of X
○ do not use service mesh
○ do not keep logs
32
33. Benefits
● Full isolation
● The cognitive load is low
Drawbacks
● Custom configuration
● High resources consumption
● You own - you troubleshooting
Full Copy for each Developer
33
57. What if not?
● No dogfooding
○ A silo between QA and developers
○ Developers are pushed to fix the stage
● We should keep stable two environments instead of one
○ Staging should be stable by design
○ The development environment should be stable too
■ If the authorization service doesn’t work -
developer cannot test their branch
57
58. What if not?
● No dogfooding
○ A silo between QA and developers
○ Developers are pushed to fix the stage
● We should keep stable two environments instead of one
○ Staging should be stable by design
○ The development environment should be stable too
■ If the authorization service doesn’t work -
developer cannot test their branch
58
129. Types of
Ephemeral
Environments
● One service branch
● Several services branches
○ under one x-source-route
○ Jira-based
● Custom environments
○ for squad (payment-dev)
○ for domain (warehouse)
129
130. Types of
Ephemeral
Environments
● One service branch
● Several services branches
○ under one x-source-route
○ Jira-based
● Custom environments
○ for squad (payment-dev)
○ for domain (warehouse)
130