Rodrigo Albani de Campos gave a presentation on capacity planning at the São Paulo Perl Workshop. He discussed typical performance metrics like load average and CPU usage, but emphasized that time series data alone is not sufficient for capacity planning. He covered concepts like arrival rate, service time, and queues. De Campos demonstrated using the PDQ queuing model tool to model an Apache web server and explore "what if" scenarios. He provided several references for further reading on performance analysis and capacity planning techniques.
Slides from my Planning to Fail talk given at PHP North East conference 2013. This is a slightly longer version of the same talk given at the PHP UK conference. The talk was on how you can build resilient systems by embracing failure.
Description of some of the elements that go in to creating a PostgreSQL-as-a-Service for organizations with many teams and a diverse ecosystem of applications and teams.
Slides from my Planning to Fail talk given at PHP North East conference 2013. This is a slightly longer version of the same talk given at the PHP UK conference. The talk was on how you can build resilient systems by embracing failure.
Description of some of the elements that go in to creating a PostgreSQL-as-a-Service for organizations with many teams and a diverse ecosystem of applications and teams.
Presented at the inaugural Kafka summit (2016) hosted by Confluent in San Francisco
Abstract:
Kafka is a backbone for various data pipelines and asynchronous messaging at LinkedIn and beyond. 2015 was an exciting year at LinkedIn in that we hit a new level of scale with Kafka: we now process more than 1 trillion published messages per day across nearly 1300 brokers. We run into some interesting production issues at this scale and I will dive into some of the most critical incidents that we encountered at LinkedIn in the past year:
Data loss: We have extremely stringent SLAs on latency and completeness that were violated on a few occasions. Some of these incidents were due to subtle configuration problems or even missing features.
Offset resets: As of early 2015, Kafka-based offset management was still a relatively new feature and we occasionally hit offset resets. Troubleshooting these incidents turned out to be extremely tricky and resulted in various fixes in offset management/log compaction as well as our monitoring.
Cluster unavailability due to high request/response latencies: Such incidents demonstrate how even subtle performance regressions and monitoring gaps can lead to an eventual cluster meltdown.
Power failures! What happens when an entire data center goes down? We experienced this first hand and it was not so pretty.
and more…
This talk will go over how we detected, investigated and remediated each of these issues and summarize some of the features in Kafka that we are working on that will help eliminate or mitigate such incidents in the future.
This is my noob recap of KubeCon 2019, which I transformed into a kubernetes bootcamp. I walked away with a bunch of learnings, so here they are for you :)
Integrating multiple CDN providers at Etsy - Velocity Europe (London) 2013Marcus Barczak
Relying on a single content delivery network for your site can impose a number of flexibility limitations. By diversifying your CDN providers you can put the power back in your hands, allowing you to get the best of both worlds in terms of performance, reliability and cost. In this talk Marcus and Laurie will present Etsy’s recent work integrating multiple CDN providers to their site delivery infrastructure.
This presentation was delivered at Velocity Europe, November 2013
(CMP303) ResearchCloud: CfnCluster and Internet2 for Enterprise HPCAmazon Web Services
Biogen built ResearchCloud for large-scale processing of research data. This extension of our infrastructure capability allows us to be more nimble, process more data, scale as needed, and collaborate with external organizations. In this session, learn about the design choices we took into account when building ResearchCloud. We cover our implementation of Internet2 and AWS Direct Connect, and the challenges we encountered when scaling to speeds of 10 gigabits. We also discuss the architecture of InstantHPC, which combines CfnCluster with GlusterFS using secure templates.
This is a talk that I gave at the San Francisco DevOps meetup on 9/29/15. I talk about how Yelp performs service discovery using SmartStack and Docker.
Easy Taxi está presente em mais de 30 países e tem milhões de usuários, entre passageiros e taxistas. Seu aplicativo roda em dezenas de plataformas móveis e suporta milhares de acessos simultâneos. A aplicação nasceu na nuvem da AWS e faz pleno uso de todos os seus recursos. Nesta apresentação avançada, exploramos a arquitetura da Easy Taxi e analisamos as estratégias de otimização disponíveis para os aplicativos implementados na nuvem AWS.
A brief introduction to YARN: how and why it came into existence and how it fits together with this thing called Hadoop.
Focus given to architecture, availability, resource management and scheduling, migration from MR1 to MR2, job history and logging, interfaces, and applications.
Go Reactive: Event-Driven, Scalable, Resilient & Responsive SystemsJonas Bonér
The demands and expectations for applications have changed dramatically in recent years. Applications today are deployed on a wide range of infrastructure; from mobile devices up to thousands of nodes running in the cloud—all powered by multi-core processors. They need to be rich and collaborative, have a real-time feel with millisecond response time and should never stop running. Additionally, modern applications are a mashup of external services that need to be consumed and composed to provide the features at hand.
We are seeing a new type of applications emerging to address these new challenges—these are being called Reactive Applications. In this talk we will discuss four key traits of Reactive; Event-Driven, Scalable, Resilient and Responsive—how they impact application design, how they interact, their supporting technologies and techniques, how to think when designing and building them—all to make it easier for you and your team to Go Reactive.
Breaking Spark: Top 5 mistakes to avoid when using Apache Spark in productionNeelesh Srinivas Salian
Spark has been growing in deployments for the past year. The increasing amount of data being analyzed and processed through the framework is massive and continues to push the boundaries of the engine. Drawing on experiences across 150+ production deployments, Neelesh Srinivas Salian explores common issues observed in a cluster environment setup with Apache Spark and offers guidelines to help setup a real-world environment when planning an Apache Spark deployment in a cluster. Attendees can use these observations to improve the usability and supportability of Apache Spark and avoid such issues in their projects.
Topics include:
Scaling the architecture
Memory configurations
End-user code
Incompatible dependencies
Administration- and operation-related issues
Apache Kafka's rise in popularity as a streaming platform has demanded a revisit of its traditional at-least-once message delivery semantics.
In this talk, we present the recent additions to Kafka to achieve exactly-once semantics (EoS) including support for idempotence and transactions in the Kafka clients. The main focus will be the specific semantics that Kafka distributed transactions enable and the underlying mechanics which allow them to scale efficiently.
This is Apache ZooKeeper session.
ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
By the end of this presentation you should be fairly clear about Apache ZooKeeper.
To watch the video or know more about the course, please visit
http://www.knowbigdata.com/page/big-data-and-hadoop-online-instructor-led-training
Infrastructure at Scale: Apache Kafka, Twitter Storm & Elastic Search (ARC303...Amazon Web Services
"This is a technical architect's case study of how Loggly has employed the latest social-media-scale technologies as the backbone ingestion processing for our multi-tenant, geo-distributed, and real-time log management system. This presentation describes design details of how we built a second-generation system fully leveraging AWS services including Amazon Route 53 DNS with heartbeat and latency-based routing, multi-region VPCs, Elastic Load Balancing, Amazon Relational Database Service, and a number of pro-active and re-active approaches to scaling computational and indexing capacity.
The talk includes lessons learned in our first generation release, validated by thousands of customers; speed bumps and the mistakes we made along the way; various data models and architectures previously considered; and success at scale: speeds, feeds, and an unmeltable log processing engine."
With more than 140 million users, KakaoTalk is the most popular mobile messaging platform in South Korea. The team at daumkakao has been using OpenStack with the intention for tranforming the current legacy infrastructure into scale out based cloud to build and offer new services for its users. In this session, we'd like to share our experiences with the OpenStack community, specifically in regards to meeting our needs for networking with Neutron.OpenStack Neutron offers a lot of methods to implement networking for VMs and containers. For production operations, VM migration can be a common activity to manage resources and improve uptime. It's not hard using shared storage like Ceph, but network settings, such as IP addresses need to be preserved. With a shared storage environment, an image can be attached anywhere inside of a data center, but a service IP for a virtual machine is different story. And when you don't use the floating IPs, keeping the same IP across a data center-wide set of VLANs is hard job.To maintain a virtual machine's IP settings and balance IPs between VLANS, we tried several options including overlay, SDN, and NFV technologies. In the end we came to use a route-only network for our virtual machine networks, leveraging technology like Quagga for RIP, OSPF BGP integrated with Neutron.
Netflix changed its data pipeline architecture recently to use Kafka as the gateway for data collection for all applications which processes hundreds of billions of messages daily. This session will discuss the motivation of moving to Kafka, the architecture and improvements we have added to make Kafka work in AWS. We will also share the lessons learned and future plans.
Presented at the inaugural Kafka summit (2016) hosted by Confluent in San Francisco
Abstract:
Kafka is a backbone for various data pipelines and asynchronous messaging at LinkedIn and beyond. 2015 was an exciting year at LinkedIn in that we hit a new level of scale with Kafka: we now process more than 1 trillion published messages per day across nearly 1300 brokers. We run into some interesting production issues at this scale and I will dive into some of the most critical incidents that we encountered at LinkedIn in the past year:
Data loss: We have extremely stringent SLAs on latency and completeness that were violated on a few occasions. Some of these incidents were due to subtle configuration problems or even missing features.
Offset resets: As of early 2015, Kafka-based offset management was still a relatively new feature and we occasionally hit offset resets. Troubleshooting these incidents turned out to be extremely tricky and resulted in various fixes in offset management/log compaction as well as our monitoring.
Cluster unavailability due to high request/response latencies: Such incidents demonstrate how even subtle performance regressions and monitoring gaps can lead to an eventual cluster meltdown.
Power failures! What happens when an entire data center goes down? We experienced this first hand and it was not so pretty.
and more…
This talk will go over how we detected, investigated and remediated each of these issues and summarize some of the features in Kafka that we are working on that will help eliminate or mitigate such incidents in the future.
This is my noob recap of KubeCon 2019, which I transformed into a kubernetes bootcamp. I walked away with a bunch of learnings, so here they are for you :)
Integrating multiple CDN providers at Etsy - Velocity Europe (London) 2013Marcus Barczak
Relying on a single content delivery network for your site can impose a number of flexibility limitations. By diversifying your CDN providers you can put the power back in your hands, allowing you to get the best of both worlds in terms of performance, reliability and cost. In this talk Marcus and Laurie will present Etsy’s recent work integrating multiple CDN providers to their site delivery infrastructure.
This presentation was delivered at Velocity Europe, November 2013
(CMP303) ResearchCloud: CfnCluster and Internet2 for Enterprise HPCAmazon Web Services
Biogen built ResearchCloud for large-scale processing of research data. This extension of our infrastructure capability allows us to be more nimble, process more data, scale as needed, and collaborate with external organizations. In this session, learn about the design choices we took into account when building ResearchCloud. We cover our implementation of Internet2 and AWS Direct Connect, and the challenges we encountered when scaling to speeds of 10 gigabits. We also discuss the architecture of InstantHPC, which combines CfnCluster with GlusterFS using secure templates.
This is a talk that I gave at the San Francisco DevOps meetup on 9/29/15. I talk about how Yelp performs service discovery using SmartStack and Docker.
Easy Taxi está presente em mais de 30 países e tem milhões de usuários, entre passageiros e taxistas. Seu aplicativo roda em dezenas de plataformas móveis e suporta milhares de acessos simultâneos. A aplicação nasceu na nuvem da AWS e faz pleno uso de todos os seus recursos. Nesta apresentação avançada, exploramos a arquitetura da Easy Taxi e analisamos as estratégias de otimização disponíveis para os aplicativos implementados na nuvem AWS.
A brief introduction to YARN: how and why it came into existence and how it fits together with this thing called Hadoop.
Focus given to architecture, availability, resource management and scheduling, migration from MR1 to MR2, job history and logging, interfaces, and applications.
Go Reactive: Event-Driven, Scalable, Resilient & Responsive SystemsJonas Bonér
The demands and expectations for applications have changed dramatically in recent years. Applications today are deployed on a wide range of infrastructure; from mobile devices up to thousands of nodes running in the cloud—all powered by multi-core processors. They need to be rich and collaborative, have a real-time feel with millisecond response time and should never stop running. Additionally, modern applications are a mashup of external services that need to be consumed and composed to provide the features at hand.
We are seeing a new type of applications emerging to address these new challenges—these are being called Reactive Applications. In this talk we will discuss four key traits of Reactive; Event-Driven, Scalable, Resilient and Responsive—how they impact application design, how they interact, their supporting technologies and techniques, how to think when designing and building them—all to make it easier for you and your team to Go Reactive.
Breaking Spark: Top 5 mistakes to avoid when using Apache Spark in productionNeelesh Srinivas Salian
Spark has been growing in deployments for the past year. The increasing amount of data being analyzed and processed through the framework is massive and continues to push the boundaries of the engine. Drawing on experiences across 150+ production deployments, Neelesh Srinivas Salian explores common issues observed in a cluster environment setup with Apache Spark and offers guidelines to help setup a real-world environment when planning an Apache Spark deployment in a cluster. Attendees can use these observations to improve the usability and supportability of Apache Spark and avoid such issues in their projects.
Topics include:
Scaling the architecture
Memory configurations
End-user code
Incompatible dependencies
Administration- and operation-related issues
Apache Kafka's rise in popularity as a streaming platform has demanded a revisit of its traditional at-least-once message delivery semantics.
In this talk, we present the recent additions to Kafka to achieve exactly-once semantics (EoS) including support for idempotence and transactions in the Kafka clients. The main focus will be the specific semantics that Kafka distributed transactions enable and the underlying mechanics which allow them to scale efficiently.
This is Apache ZooKeeper session.
ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
By the end of this presentation you should be fairly clear about Apache ZooKeeper.
To watch the video or know more about the course, please visit
http://www.knowbigdata.com/page/big-data-and-hadoop-online-instructor-led-training
Infrastructure at Scale: Apache Kafka, Twitter Storm & Elastic Search (ARC303...Amazon Web Services
"This is a technical architect's case study of how Loggly has employed the latest social-media-scale technologies as the backbone ingestion processing for our multi-tenant, geo-distributed, and real-time log management system. This presentation describes design details of how we built a second-generation system fully leveraging AWS services including Amazon Route 53 DNS with heartbeat and latency-based routing, multi-region VPCs, Elastic Load Balancing, Amazon Relational Database Service, and a number of pro-active and re-active approaches to scaling computational and indexing capacity.
The talk includes lessons learned in our first generation release, validated by thousands of customers; speed bumps and the mistakes we made along the way; various data models and architectures previously considered; and success at scale: speeds, feeds, and an unmeltable log processing engine."
With more than 140 million users, KakaoTalk is the most popular mobile messaging platform in South Korea. The team at daumkakao has been using OpenStack with the intention for tranforming the current legacy infrastructure into scale out based cloud to build and offer new services for its users. In this session, we'd like to share our experiences with the OpenStack community, specifically in regards to meeting our needs for networking with Neutron.OpenStack Neutron offers a lot of methods to implement networking for VMs and containers. For production operations, VM migration can be a common activity to manage resources and improve uptime. It's not hard using shared storage like Ceph, but network settings, such as IP addresses need to be preserved. With a shared storage environment, an image can be attached anywhere inside of a data center, but a service IP for a virtual machine is different story. And when you don't use the floating IPs, keeping the same IP across a data center-wide set of VLANs is hard job.To maintain a virtual machine's IP settings and balance IPs between VLANS, we tried several options including overlay, SDN, and NFV technologies. In the end we came to use a route-only network for our virtual machine networks, leveraging technology like Quagga for RIP, OSPF BGP integrated with Neutron.
Netflix changed its data pipeline architecture recently to use Kafka as the gateway for data collection for all applications which processes hundreds of billions of messages daily. This session will discuss the motivation of moving to Kafka, the architecture and improvements we have added to make Kafka work in AWS. We will also share the lessons learned and future plans.
Performance Benchmarking: Tips, Tricks, and Lessons LearnedTim Callaghan
Presentation covering 25 years worth of lessons learned while performance benchmarking applications and databases. Presented at Percona Live London in November 2014.
Monitoring and Scaling Redis at DataDog - Ilan Rabinovitch, DataDogRedis Labs
Think you have big data? What about high availability
requirements? At DataDog we process billions of data points every day including metrics and events, as we help the world
monitor the their applications and infrastructure. Being the world’s monitoring system is a big responsibility, and thanks to
Redis we are up to the task. Join us as we discuss how the DataDog team monitors and scales Redis to power our SaaS based monitoring offering. We will discuss our usage and deployment patterns, as well as dive into monitoring best practices for production Redis workloads
Librato's Joseph Ruscio at Heroku's 2013: Instrumenting 12-Factor AppsHeroku
Librato's CTO Joseph Ruscio took to the Waza 2013 stage to present "Instrumenting Twelve-Factor Apps". For more from Ruscio ping him at @josephruscio. For more on Waza visit http://waza.heroku.com/2013.
For Waza videos stay tuned at http://blog.heroku.com or visit http://vimeo.com/herokuwaza
Using Riak for Events storage and analysis at Booking.comDamien Krotkine
At Booking.com, we have a constant flow of events coming from various applications and internal subsystems. This critical data needs to be stored for real-time, medium and long term analysis. Events are schema-less, making it difficult to use standard analysis tools.This presentation will explain how we built a storage and analysis solution based on Riak. The talk will cover: data aggregation and serialization, Riak configuration, solutions for lowering the network usage, and finally, how Riak's advanced features are used to perform real-time data crunching on the cluster nodes.
Velocity Conference NYC 2014 - Real World DevOpsRodrigo Campos
In a world where agility has become a requirement, business and engineering demands have decreed the death of the “Department of No”. This talk will cover the journey of an IT Operations department from a single DevOps team to a business-wide cultural shift that has affected the way people interact and work with each other.
In order to make sure that our DevOps initiative would be successful, we needed to make changes to the corporate organization, rearrange teams and roles in several areas, and make sure that everyone fully understand where we were being headed to.
All these steps will be covered in this talk that will demonstrate some common pitfalls and misconceptions that jeopardize the DevOps adoption, particularly in large enterprises with several compliancy requirements and some outdated bureaucracy.
DevOps has become a broad term that is frequently misunderstood. This presentation, originally made to Walmart GeC Brazil Operations teams, aimed at explaining the basic aspects of DevOps and how its adoption can help us to become more agile.
Communications Mining Series - Zero to Hero - Session 1DianaGray10
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Maruthi Prithivirajan, Head of ASEAN & IN Solution Architecture, Neo4j
Get an inside look at the latest Neo4j innovations that enable relationship-driven intelligence at scale. Learn more about the newest cloud integrations and product enhancements that make Neo4j an essential choice for developers building apps with interconnected data and generative AI.
How to Get CNIC Information System with Paksim Ga.pptxdanishmna97
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
Building RAG with self-deployed Milvus vector database and Snowpark Container...Zilliz
This talk will give hands-on advice on building RAG applications with an open-source Milvus database deployed as a docker container. We will also introduce the integration of Milvus with Snowpark Container Services.
UiPath Test Automation using UiPath Test Suite series, part 6DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
20 Comprehensive Checklist of Designing and Developing a WebsitePixlogix Infotech
Dive into the world of Website Designing and Developing with Pixlogix! Looking to create a stunning online presence? Look no further! Our comprehensive checklist covers everything you need to know to craft a website that stands out. From user-friendly design to seamless functionality, we've got you covered. Don't miss out on this invaluable resource! Check out our checklist now at Pixlogix and start your journey towards a captivating online presence today.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank
Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.
3. Why Perl ?
• Main reason: I feel comfortable with it
• Ubiquitous and free
• Plenty of stable statistics modules available
at CPAN
• Ultimately, it gets the job done
4. Capacity Planning
• Is just like sex...
• Everyone wants to do it
• Many say they’re doing it
• You always exaggerate how much of it
you’re doing
• Most people aren’t actually doing it (despite
their best efforts)
• Everybody else seems to be doing more
than you
5. A tale of discovery
There once was a system administrator...
6. A tale of discovery
How many ?
Actual capacity ?
Servers do we need ?
How much memory ?
What’s the predicted growth ?
IO Capacity ?
7. Typical Performance
Metrics
• Load Average - uptime
• The single most misunderstood metric
• CPU - mpstat
• IO - iostat
• Memory Usage - vmstat
9. Time series charts
I’m looking at you cacti huggers !
• Time series performance data is useful for:
• Troubleshooting
• Simplistic forecasting
• Find trends
• Identify seasonal behavior
• This left alone is NOT Capacity Planning
10. Frustration
• Computer systems can be harsh
• Most systems will not scale linearly
• Diminishing returns and lock contention
will punch you in the face
• “Oh but I’ve checked cacti and the CPU
was 25% idle”
11. Let’s put it in the Cloud
• We are moving back to an utility computing
model
• You’re charged per usage
• Even more important to care about
capacity planning !!!
12. Call the experts
• Cost per MIPS
• IBM System/370 model 158-3 - 1.0 MIPS @
1.0 MHz -1972
• Average purchase price: $ 771,000*
• No disks or peripherals included
• $ 4,082,039 by 2011
• Need to squeeze every drop of processing
power
* Source: http://www-03.ibm.com/ibm/history/exhibits/mainframe/mainframe_PP3135.html
13. Queues
The not so typical performance metrics
• 1961 - CTSS was first
demonstrated at MIT
• 1965 - Allan Scherr used
machine repairman
problem to model a
time-shared system as
part of Project MAC
• Another offspring of
Project MAC is Multics
14. Queues
The not so typical performance metrics
Computer System
Disks
CPU
15. Queues
The not so typical performance metrics
(A) λ X (C)
S
Open/Closed W
Network R
A Arrival Count
λ Arrival Rate (A/T)
W Time spent in Queue
R Residence Time (W+S)
S Service Time
X System Throughput (C/T)
C Completed tasks count
16. Arrival Rate (λ)
• Pretty straightforward
• Requests per second/hour/day
• Not the same as throughput (X)
• Although in a steady state:
• A = C as T →∞
• λ=X
17. Service Time (S)
• Time spent in processing
• Web server response time
• Total query time
• IO operation time length
18. !"#$%&"'(%)"*+,'
Mythical Performance
!#)"
!#("
!#'"
• Not gonna happen...
!"#$%&"'(%)"'*+,'
!#&"
*+,-./+"0.1+234"
• Don’t believe vendor’s sales pitch
!#%"
• “In God we trust, all others must bring
data” - William Edwards Deming
!#$"
!"
!" (" $!" $(" %!" %(" &!" &(" '!" '("
-##%$./'0.1"'*2%1+3+,'
19. Mythical Performance
• Not gonna happen...
• Don’t believe vendor’s sales pitch
• “In God we trust, all others must bring
data” - William Edwards Deming
20. How to measure ?
• Apache: %D in mod_log_config
• nginx: $request_time in HttpLogModule
• use Benchmark;
• tcprstat - http://goo.gl/0cbYx
• collectd - http://goo.gl/OXKG7
• metrics - http://goo.gl/gQFVM
• sysstat - http://goo.gl/2aLul
21. How to measure ?
my ($date,$svctime) = (m/[(S+).+?s(d+)$/);
$arrivalRate{$date}++;
$serviceTimeAcc{$date} += $svctime;
[02/Jul/2010:14:00:18... 1863
Time to serve the
request,
in μseconds.
27. What to look for ?
• Stretch factor
• Method/Operation
• Geolocation
• Cookies
• Use mod_logio to measure inbound
traffic as well
28. Modeling
Prediction is very difficult, especially if it’s about
the future.
Niels Bohr
Capacity planning is about setting expectations.
Even wrong expectations are better than no
expectations!
Neil J. Gunther - The Guerrilla Manifesto
http://goo.gl/lZKWH
29. Modeling
• A model is an abstraction of a complex
system
• A model allows us to observe phenomena
that cannot be easily replicated
30. Modeling Methods
• Statistics / Trending / Forecasting
• Pros:
• Easy to understand
• Tools readily available
• Cons:
• Hard to create “What-if” scenarios
• Hard to predict contention and
bottlenecks
31. Modeling Methods
• Queuing Analisys
• Pros:
• Allows you to make predictions when no
production data is available
• Allows you to create “What-if” scenarios
• Cons:
• Sometimes it can be unintuitive
• The math behind it can be difficult
35. Queues as models
m1.small ? m1.large ? m1.xlarge ?
Virtual Cores X
EC2 CU
Memory Bus
36. use pdq;
• Available at http://goo.gl/s98wQ (not on
CPAN)
• PDQ is a queuing circuit solver by Neil J.
Gunther
• There’s a whole book about it
http://goo.gl/9MA2c
37. use pdq;
CreateNode() Define a queuing center
Define a traffic stream of an
CreateOpen()
open circuit
Define a traffic stream of a
CreateClosed()
closed circuit
Define the service demand for
SetDemand()
each of the queuing centers
38. use pdq;
Node Types
CEN Queuing Center
DLY Delay Center
39. use pdq;
Service Disciplines
FCFS First-come first-served
LCFS Last-come first-served
ISRV Infinite Server
PSHR Processor Sharing
40. use pdq;
• Apache Web Server
• Average Network RTD: 0.00921 seconds
• Added as a delay center in the circuit
• Average Arrival Rate: 65.142 hits/s
• Average Service time: 0.0159 seconds
• 128 worker threads
42. pdq::Report();
Metric Value Unit
------ ----- ----
Workload: "httpd"
Number in system 8.0279 Trans
Mean throughput 65.1420 Trans/Sec
Response time 0.1232 Sec
Stretch factor 1.0626
45. Resources and
References
• CMG Public Proceedings:
http://www.cmg.org/proceedings/
• Measure IT:
http://www.cmg.org/measureit/
• Guerrilla Capacity Planning
http://www.perfdynamics.com/Classes/
Outlines/guerilla.html
46. Resources and
References
• Performance by Design - Menasce, Dowdy,
Almeida - http://amzn.to/mpqfVO
• Capacity Planning for Web Performance:
Metrics, Models, and Methods - Daniel
Menasce,Virgilio Almeida - http://amzn.to/
lOATba
• Capacity Planning for Web Services: Metrics,
Models, and Methods - Daniel Menasce,
Virgilio Almeida - http://amzn.to/iClpsB
47. Resources and
References
• Guerrilla Capacity Planning: A Tactical
Approach to Planning for Highly Scalable
Applications and Services - Neil Gunther -
http://amzn.to/kfrfLK
• The Art of Computer Systems Performance
Analysis: Techniques for Experimental Design,
Measurement, Simulation, and Modeling - R.
K. Jain - http://amzn.to/jqud1I