A chaos experiment a day, keeping the outage away

•

0 likes•385 views

Presented at ServerlessDays Warsaw Recording: https://youtu.be/21HprKZQczs You might have heard about chaos engineering in the context of Netflix and Amazon, and how they kill EC2 servers in production at random to verify that their systems can stay up in the face of infrastructure failures. But did you know that the same ideas can be applied to serverless applications? Yes, despite not having access to the underlying servers, we can still apply principles of chaos engineering to uncover failure modes in our system (and there are plenty!) so we can build defence against them and make our serverless applications more robust and more resilient!

one chaos experiment a day.
keep the outages away.
yan cui,@theburningmonk

@theburningmonk theburningmonk.com
“the discipline of experimenting on a system in order to build conﬁdence in the
system’s capability to withstand turbulent conditions in production”
principlesofchaos.org

@theburningmonk theburningmonk.com
microservices death stars circa 2015

@theburningmonk theburningmonk.com
“the capacity to recover quickly from difﬁculties; toughness.”
resilience
/rɪˈzɪlɪəns/
noun

@theburningmonk theburningmonk.com
“the capacity to recover quickly from difﬁculties; toughness.”
resilience
/rɪˈzɪlɪəns/
noun
it’s not about
preventing failures!

@theburningmonk theburningmonk.com
“You don't choose the moment, the moment chooses you!
You only choose how prepared you are when it does.”
Fire Chief Mike Burtch

@theburningmonk theburningmonk.com
anything that can go wrong, will go wrong.
MURPHY’s LAW

@theburningmonk theburningmonk.com
identify weaknesses before they manifest in system-wide, aberrant behaviors
GOAL

@theburningmonk theburningmonk.com
learn about the system’s behavior by observing it during a controlled experiments
HOW

@theburningmonk theburningmonk.com
learn about the system’s behavior by observing it during a controlled experiments
HOW
game days
failure injection

Yan Cui
http://theburningmonk.com
@theburningmonk
AWS user for 10 years

Yan Cui
http://theburningmonk.com
@theburningmonk
http://bit.ly/yubl-serverless

Yan Cui
http://theburningmonk.com
@theburningmonk
Developer Advocate @

Yan Cui
http://theburningmonk.com
@theburningmonk
Independent Consultant
advisetraining delivery

“using serverless reduces the blast radius”
www.buzzsprout.com/877747/4615985

@theburningmonk theburningmonk.com
serverless improves resilience
as platform takes care of infrastructure failures

@theburningmonk theburningmonk.com
by Russ Miles @russmiles
source https://medium.com/russmiles/chaos-engineering-for-the-business-17b723f26361

@theburningmonk theburningmonk.com
Shared Responsibility Model

@theburningmonk theburningmonk.com
chaos monkey kills an
EC2 instance
latency monkey induces
artiﬁcial delay in APIs
chaos gorilla kills an AWS
Availability Zone
chaos kong kills an entire
AWS region

@theburningmonk theburningmonk.com
there are no servers to kill!
SERVERLESS

@theburningmonk theburningmonk.com
improperly tuned timeouts

@theburningmonk theburningmonk.com
missing error handling

@theburningmonk theburningmonk.com
missing fallbacks

@theburningmonk theburningmonk.com
STEP 1.
deﬁne steady state
i.e. “what does normal look like”

@theburningmonk theburningmonk.com
STEP 2.
hypothesis that steady state continues in control and experimental group
e.g. “the system stays up if a server dies”

@theburningmonk theburningmonk.com
STEP 3.
inject realistic failures
e.g. “slow response from 3rd-party service”

@theburningmonk theburningmonk.com
STEP 4.
try to disprove hypothesis
i.e. “look for difference between control and experimental group”

@theburningmonk theburningmonk.com
latency inject latency to function invocation

@theburningmonk theburningmonk.com
“what if service X has elevated latency?”

@theburningmonk theburningmonk.com
API Gateway Lambda API Gateway Lambda

@theburningmonk theburningmonk.com
hypothesis: API would timeout and our try-catch
would handle it and return default response

@theburningmonk theburningmonk.com
result: function times out after 6s
(hypothesis is disproved)

@theburningmonk theburningmonk.com
API Gateway Lambda API Gateway Lambda
502
200

@theburningmonk theburningmonk.com
API Gateway Lambda API Gateway Lambda
3s timeout
6s timeout

@theburningmonk theburningmonk.com
API Gateway Lambda API Gateway Lambda
max 29s integration
max 15 mins timeout

@theburningmonk theburningmonk.com
and then there’s
cold starts…

@theburningmonk theburningmonk.com
TIL: most HTTP client libraries have default timeout of 60s.
API Gateway has an integration timeout of 29s.
Most Lambda functions default to timeout of 3-6s.
Don’t forget about the cold starts!

@theburningmonk theburningmonk.com
https://bit.ly/2Wvfort

@theburningmonk theburningmonk.com
outcome: a more resilient system

@theburningmonk theburningmonk.com
latency
exception
inject latency to function invocation
throws exception

@theburningmonk theburningmonk.com
latency
exception
statuscode
inject latency to function invocation
throws exception
return HTTP status code

@theburningmonk theburningmonk.com
latency
exception
statuscode
diskspace
inject latency to function invocation
throws exception
return HTTP status code
ﬁlls up /tmp directory

@theburningmonk theburningmonk.com
latency
exception
statuscode
diskspace
denylist
inject latency to function invocation
throws exception
return HTTP status code
ﬁlls up /tmp directory
looses network connectivity

@theburningmonk theburningmonk.com
“what if DynamoDB has an elevated error rate?”

@theburningmonk theburningmonk.com
API Gateway Lambda DynamoDB

@theburningmonk theburningmonk.com
hypothesis: the AWS SDK retries would handle it

@theburningmonk theburningmonk.com
TIL: the js DynamoDB client defaults to 10 retries
with base delay of 50ms

@theburningmonk theburningmonk.com
TIL: the js DynamoDB client defaults to 10 retries
with base delay of 50ms
delay = Math.random() * (Math.pow(2, retryCount) * base)
this is Marc Brooker’s
fav formula!

@theburningmonk theburningmonk.com
action: set max retry count + fallback

https://theburningmonk.com/hire-me
AdviseTraining Delivery
“Fundamentally, Yan has improved our team by increasing our
ability to derive value from AWS and Lambda in particular.”
Nick Blair
Tech Lead

@theburningmonk
theburningmonk.com
github.com/theburningmonk

You might have heard about chaos engineering in the context of Netflix and Amazon, and how they kill EC2 servers in production at random to verify that their systems can stay up in the face of infrastructure failures. But did you know that the same ideas can be applied to serverless applications? Yes, despite not having access to the underlying servers, we can still apply principles of chaos engineering to uncover failure modes in our system (and there are plenty!) so we can build a defence against them and make our serverless applications more robust and more resilient!

Patterns and Practices for Building Resilient Serverless Applications

Yan Cui

Patterns and practices for building resilient serverless applications

Yan Cui

Lambda gives you multi-AZ out-of-the-box, but still, things can go wrong in production. There are region-wide outages, and performance degradation in services your function depends on can cause it to time out or error. And what if you're dealing with downstream systems that just aren't as scalable and can't handle the load you put on them? The bottom line is many things can go wrong and they often do at the worst of times. The goal of building resilient systems is not to prevent failures, but to build systems that can withstand these failures. In this talk, we will look at a number of practices and architectural patterns that can help you build more resilient serverless applications. Such as multi-region, active-active, employing DLQs and surge queues. We'll also see how we can use chaos experiments to help us identify failure modes before they manifest in production

Patterns and practices for building resilient serverless applications.pdf

Yan Cui

Looking in from the outside, serverless seems so simple! And yet, many companies are struggling on their journey to serverless. In this talk, I highlight a number of mistakes companies are making when they adopt serverless. Recording: https://www.youtube.com/watch?v=4hzNqxo0yTA Real-world serverless podcast: https://realworldserverless.com Learn Lambda best practices: https://lambdabestpractice.com Blog: https://theburningmonk.com Consulting services: https://theburningmonk.com/hire-me Production-Ready Serverless workshop: https://productionreadyserverless.com

Beware the potholes on the road to serverless

Yan Cui

Essential open source tools for serverless developers

Yan Cui

Patterns and practices for building resilient Serverless applications

Yan Cui

How to build observability into a serverless application

Yan Cui

Serverless introduces a number of challenges to existing tools for observability, we need to adapt our practices to fit this new paradigm. In this talk, we will discuss how we can build observability into a serverless application. We will see how you can implement log aggregation, distributed tracing and correlation IDs through both synchronous as well as asynchronous events. Recording of this talk is available at https://www.youtube.com/watch?v=AMKUKKGPJhQ

Dont try these at home

Yan Cui

Learning by doing is hands-down the best way to learn. With serverless, the cost of experimenting (and learning!) in the cloud has reduced to zero, almost. In this talk, Yan Cui discusses the value of experimenting in the cloud and doing so in a tasteful manner (i.e. don't put your experiments into production!), and shares a number of interesting experiments that the serverless community has done, and some ideas and learning that we can take from them. Recording of the talk is available here: https://www.youtube.com/watch?v=oeomaQR1IT4

Migrating existing monolith to serverless in 8 steps

Yan Cui

Stand back; I'm going to try Scientist!

Matt Eland

Erlang - Because s**t Happens by Mahesh Paolini-Subramanya

Hakka Labs

Mahesh talks about the buddha-nature of Erlang/OTP, pointing out how the various features of the language tie together into one seamless Fault Tolerant whole. Mahesh emphasizes that Erlang begins and ends with Fault Tolerance. Fault Tolerance is baked into the very genes of Erlang/OTP - something that ends up being amazingly useful when building any kind of system. Mahesh Paolini-Subramanya is the V.P. of R&D at Ubiquiti Networks - a manufacturer of disruptive technology platforms for emerging markets. He has spent the recent past building out Erlang-based massively concurrent Cloud Services and VoIP platforms. Mahesh was previously the CTO of Vocalocity after its merger with Aptela, where he was a founder and CTO.

Green Custard Friday Talk 19: Chaos Engineering

Green Custard

In Green Custard's 19th Friday talk, Zoltan explores the subject of Chaos Engineering Topics covered: - What is chaos engineering? - Why would anyone do this? - Availability - Chaos engineering in practice - The four golden signals - Chaos engineering in practice - Chaos Monkey - The Simian Army Green Custard is a custom software development consultancy. To discover more about their work and the team visit www.green-custard.com.

PagerDuty | OSCON 2016 Failure Testing

PagerDuty

testing for people who hate testing

Bram Vogelaar

Migrating existing monolith to serverless in 8 steps

Yan Cui

Migrating existing monolith to serverless in 8 steps

Yan Cui

Refactoring a monolith to serverless can be intimidating, but there are discrete steps that you can take to simplify the process. In this talk, AWS Serverless Hero Yan Cui outlines 8 steps to successfully refactor your monolith and highlight key decision points such as language and tooling choices. Recording: https://www.youtube.com/watch?v=-HQw-UK2bHc Real-world serverless podcast: https://realworldserverless.com Learn Lambda best practices: https://lambdabestpractice.com Blog: https://theburningmonk.com Consulting services: https://theburningmonk.com/hire-me Production-Ready Serverless workshop: https://productionreadyserverless.com

Serverless observability - a hero's perspective

Yan Cui

Serverless gives you wings

Yan Cui

re:Invent 2019 Highly Available ECS Spot Architecture: Save 50%-90%

Tung Nguyen

Introduction of Plasma Chamber at EDCON 2019

YurikoNishijima

Sangam 18 - Database Development: Return of the SQL Jedi

Connor McDonald

Applying Chaos Engineering to build Resilient Serverless Applications - Emrah...

PROIDEA

Serverless applications are the epitome of highly distributed, microservices applications. Execution happens everywhere - both inside and outside the serverless compute environment. For example, your functions could be triggered by an external service, then execute some code within AWS Lambda, then send a request over to a database, which *then* requires AWS Lambda to perform an update in a second data store. You might be able to predict and design for certain troublesome issues but there are many, many more that you probably will not be able to easily plan for. How do you build a resilient system under these highly distributed circumstances? The answer is chaos engineering. Join us as we walk through: The unique challenges of building a highly resilient serverless app Why you need to design for problems you cannot predict and cannot easily test for How you can use chaos engineering to build a resilient serverless application How you can take advantage of out of the box and third-party observability solutions to measure the impact of chaos experiments.

How to win the game of trade-offs

Yan Cui

How to choose the right messaging service

Yan Cui

At the heart of every event-driven architecture is a conduit for messages to flow through. AWS offers many services that can act as such conduit - EventBridge, SNS, SQS, Kinesis, DynamoDB streams, MSK, IOT Core and Amazon MQ just to name a few! These services have different characteristics and trade-offs around performance, scalability and cost. Picking the right service for your workload is not always easy. In this talk, let’s talk about how to pick the right messaging service to use in your event-driven architecture and play the game of trade-offs to your advantage.

Similar to A chaos experiment a day, keeping the outage away

Beware the potholes on the road to serverless

Yan Cui

Beware the potholes on the road to serverless

Yan Cui

Essential open source tools for serverless developers

Yan Cui

Patterns and practices for building resilient Serverless applications

Yan Cui

How to build observability into a serverless application

Yan Cui

Dont try these at home

Yan Cui

Migrating existing monolith to serverless in 8 steps

Yan Cui

Stand back; I'm going to try Scientist!

Matt Eland

Erlang - Because s**t Happens by Mahesh Paolini-Subramanya

Hakka Labs

Green Custard Friday Talk 19: Chaos Engineering

Green Custard

PagerDuty | OSCON 2016 Failure Testing

PagerDuty

testing for people who hate testing

Bram Vogelaar

Migrating existing monolith to serverless in 8 steps

Yan Cui

Migrating existing monolith to serverless in 8 steps

Yan Cui

Serverless observability - a hero's perspective

Yan Cui

Serverless gives you wings

Yan Cui

re:Invent 2019 Highly Available ECS Spot Architecture: Save 50%-90%

Tung Nguyen

Introduction of Plasma Chamber at EDCON 2019

YurikoNishijima

Sangam 18 - Database Development: Return of the SQL Jedi

Connor McDonald

Applying Chaos Engineering to build Resilient Serverless Applications - Emrah...

PROIDEA

Similar to A chaos experiment a day, keeping the outage away (20)

Beware the potholes on the road to serverless

Essential open source tools for serverless developers

Patterns and practices for building resilient Serverless applications

How to build observability into a serverless application

Dont try these at home

Migrating existing monolith to serverless in 8 steps

Stand back; I'm going to try Scientist!

Erlang - Because s**t Happens by Mahesh Paolini-Subramanya

Green Custard Friday Talk 19: Chaos Engineering

PagerDuty | OSCON 2016 Failure Testing

testing for people who hate testing

Migrating existing monolith to serverless in 8 steps

Serverless observability - a hero's perspective

Serverless gives you wings

re:Invent 2019 Highly Available ECS Spot Architecture: Save 50%-90%

Introduction of Plasma Chamber at EDCON 2019

Sangam 18 - Database Development: Return of the SQL Jedi

Applying Chaos Engineering to build Resilient Serverless Applications - Emrah...

More from Yan Cui

How to win the game of trade-offs

Yan Cui

How to choose the right messaging service

Yan Cui

How to choose the right messaging service for your workload

Yan Cui

Lambda and DynamoDB best practices

Yan Cui

Lessons from running AppSync in prod

Yan Cui

How to ship customer value faster with step functions

Yan Cui

Learn all about AWS Step Functions and how to use them to model business workflows and ship customer values quickly. In this session, we will talk about what is Step Functions, how to model business workflows as state machines, real-world case studies, and design patterns. By the end of this webinar, you should have a good idea of where Step Functions fit into your application and why you should use them (and why not!) to model workflows instead of building a custom solution yourself.

How serverless changes the cost paradigm

Yan Cui

One of the key characteristics of serverless components is the pay-per-use pricing model. For example, with AWS Lambda, you don’t pay for the uptime of the underlying infrastructure but for the no. of invocations and how long your code actually runs for. This important characteristic removes the need for many premature micro-optimizations as your cost is always tightly linked to usage and minimizes waste. As a result, many applications would run at a fraction of the cost if they were moved to serverless. The pay-per-use pricing model also enables more accurate cost prediction and monitoring based on your application’s throughput. This gives rise to the notion of FinDev, where finance and development can intersect and allows optimization to be targeted to give the optimal return-on-invest on the engineering efforts. And by building your application on serverless components, you can also leverage it as a business advantage and offer a more competitive, usage-based pricing to your customers. Which is going to be crucial at a time when businesses all around the world are affected by COVID and are looking for better efficiencies. In this webinar, we will cover topics such as: - How does the cost of serverless differ from serverful applications? - How to predict and monitor cost in serverless applications? - When should you optimize for cost? - How can you leverage usage-based pricing as a business advantage?

Why your next serverless project should use AWS AppSync

Yan Cui

Build social network in 4 weeks

Yan Cui

Serverless technologies drastically simplify the task of building modern, scalable APIs in the cloud, and GraphQL makes it easy for frontend teams to consume these APIs and to iterate quickly on your product idea. Together, they are a perfect combination for a product-focused, full-stack team to deliver customer values quickly. In this talk, see how we built a new social network mobile app in under 4 weeks using Lambda, AppSync, DynamoDB and Algolia. How we approached CI/CD, testing, authentication and lessons we learnt along the way. Real-world serverless podcast: https://realworldserverless.com Learn Lambda best practices: https://lambdabestpractice.com Blog: https://theburningmonk.com Consulting services: https://theburningmonk.com/hire-me Production-Ready Serverless workshop: https://productionreadyserverless.com

Building a social network in under 4 weeks with Serverless and GraphQL

Yan Cui

FinDev as a business advantage in the post covid19 economy

Yan Cui

How to improve lambda cold starts

Yan Cui

AWS has improved Lambda cold starts by leaps and bounds in the last year. But for performance-sensitive applications such as user-facing APIs, Lambda cold starts are still a thorn in one’s side, especially when working with languages such as Java and .Net Core. In this webinar, we will dive into strategies for improving cold start latency and how to mitigate them altogether with Provisioned Concurrency, and how Lumigo helps you optimize your use of Provisioned Concurrency.

What can you do with lambda in 2020

Yan Cui

How to debug slow lambda response times

Yan Cui

What can you do with lambda in 2020

Yan Cui

How to ship customer value faster with step functions

Yan Cui

In this talk, I'm gonna tell you all about AWS Step Functions - how it works, when to use it, and some tips on how to accelerate app development so you can ship customer values faster. Recording: coming soon Real-world serverless podcast: https://realworldserverless.com Learn Lambda best practices: https://lambdabestpractice.com Blog: https://theburningmonk.com Consulting services: https://theburningmonk.com/hire-me Production-Ready Serverless workshop: https://productionreadyserverless.com

Debugging Lambda timeouts

Yan Cui

When a Lambda function times out, it’s one of the trickier problems to debug, especially if the function performs multiple IO calls. What was it doing when it timed out? And how do I identify these timeout errors quickly when Lambda lumps them together with other unhandled exceptions? In this webinar, we will show you how to identify and debug timeout errors, and strategies for mitigating timeouts and degraded performance more gracefully when there are underlying issues.

Serverless a superpower for frontend developers

Yan Cui

Debugging AWS Lambda Performance Issues

Yan Cui

Serverless Security: Defence Against the Dark Arts

Yan Cui

Recording: https://www.youtube.com/watch?v=bnXp29kQIwU Real-world serverless podcast: https://realworldserverless.com Learn Lambda best practices: https://lambdabestpractice.com Blog: https://theburningmonk.com Consulting services: https://theburningmonk.com/hire-me Production-Ready Serverless workshop: https://productionreadyserverless.com AWS has taken over the responsibilities of patching the OS and securing the underlying physical infrastructure that runs your serverless application, so what’s left for you to secure? Quite a bit it turns out. The OWASP top 10 is as relevant to you as ever; DOS attacks are still a threat even if you can probably brute force your way through it as AWS auto-scales Lambda functions automatically; and did you know attackers can easily steal your AWS credentials via your application dependencies? In addition to the traditional threats, serverless applications have more granular deployment units and therefore there are more things to configure and secure, and the tools and practices are still catching up with this fast-changing world.

More from Yan Cui (20)

How to win the game of trade-offs

How to choose the right messaging service

How to choose the right messaging service for your workload

Lambda and DynamoDB best practices

Lessons from running AppSync in prod

How to ship customer value faster with step functions

How serverless changes the cost paradigm

Why your next serverless project should use AWS AppSync

Build social network in 4 weeks

Building a social network in under 4 weeks with Serverless and GraphQL

FinDev as a business advantage in the post covid19 economy

How to improve lambda cold starts

What can you do with lambda in 2020

How to debug slow lambda response times

What can you do with lambda in 2020

How to ship customer value faster with step functions

Debugging Lambda timeouts

Serverless a superpower for frontend developers

Debugging AWS Lambda Performance Issues

Serverless Security: Defence Against the Dark Arts

Recently uploaded

Essentials of Automations: Optimizing FME Workflows with Parameters

Safe Software

Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place. Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects. Here’s what you’ll gain: - Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows. - Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy. - Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency. - Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity. We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic. Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.

FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf

FIDO Alliance

From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...

Product School

Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...

Ramesh Iyer

In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.

Leading Change strategies and insights for effective change management pdf 1.pdf

OnBoard

Neuro-symbolic is not enough, we need neuro-*semantic*

Frank van Harmelen

Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”. All of this illustrated with link prediction over knowledge graphs, but the argument is general.

ODC, Data Fabric and Architecture User Group

CatarinaPereira64715

GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...

James Anderson

Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management. The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM). Speakers: Bob Boule Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle. Gopinath Rebala Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.

Search and Society: Reimagining Information Access for Radical Futures

Bhaskar Mitra

The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

BookNet Canada

The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more. Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/ Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.

Connector Corner: Automate dynamic content and events by pushing a button

DianaGray10

Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to: Create a campaign using Mailchimp with merge tags/fields Send an interactive Slack channel message (using buttons) Have the message received by managers and peers along with a test email for review But there’s more: In a second workflow supporting the same use case, you’ll see: Your campaign sent to target colleagues for approval If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team But—if the “Reject” button is pushed, colleagues will be alerted via Slack message Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors. And... Speakers: Akshay Agnihotri, Product Manager Charlie Greenberg, Host

Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...

Thierry Lestable

Accelerate your Kubernetes clusters with Varnish Caching

Thijs Feryn

When stars align: studies in data quality, knowledge graphs, and machine lear...

Elena Simperl

Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024

Tobias Schneck

As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other? Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

FIDO Alliance

GraphRAG is All You need? LLM & Knowledge Graph

Guy Korland

Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs. 1. Unifying Large Language Models and Knowledge Graphs: A Roadmap. https://arxiv.org/abs/2306.08302 2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs: https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/

Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...

Product School

DevOps and Testing slides at DASA Connect

Kari Kakkonen

The Future of Platform Engineering

Jemma Hussein Allen

Recently uploaded (20)

Essentials of Automations: Optimizing FME Workflows with Parameters

FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf

From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...

Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...

Leading Change strategies and insights for effective change management pdf 1.pdf

Neuro-symbolic is not enough, we need neuro-*semantic*

ODC, Data Fabric and Architecture User Group

GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...

Search and Society: Reimagining Information Access for Radical Futures

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

Connector Corner: Automate dynamic content and events by pushing a button

Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...

Accelerate your Kubernetes clusters with Varnish Caching

When stars align: studies in data quality, knowledge graphs, and machine lear...

Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

GraphRAG is All You need? LLM & Knowledge Graph

Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...

DevOps and Testing slides at DASA Connect

The Future of Platform Engineering

A chaos experiment a day, keeping the outage away

1. one chaos experiment a day. keep the outages away. yan cui,@theburningmonk

2. Chaos Engineering?

3. MUST KILL SERVERS! RAWR!! RAWR!!

4. @theburningmonk theburningmonk.com “the discipline of experimenting on a system in order to build conﬁdence in the system’s capability to withstand turbulent conditions in production” principlesofchaos.org

5. @theburningmonk theburningmonk.com microservices death stars circa 2015

6. @theburningmonk theburningmonk.com

7. @theburningmonk theburningmonk.com “the capacity to recover quickly from difﬁculties; toughness.” resilience /rɪˈzɪlɪəns/ noun

8. @theburningmonk theburningmonk.com “the capacity to recover quickly from difﬁculties; toughness.” resilience /rɪˈzɪlɪəns/ noun it’s not about preventing failures!

9. everything fails, all the time

10. @theburningmonk theburningmonk.com “You don't choose the moment, the moment chooses you! You only choose how prepared you are when it does.” Fire Chief Mike Burtch

11. @theburningmonk theburningmonk.com

12. @theburningmonk theburningmonk.com anything that can go wrong, will go wrong. MURPHY’s LAW

13. @theburningmonk theburningmonk.com identify weaknesses before they manifest in system-wide, aberrant behaviors GOAL

14. @theburningmonk theburningmonk.com learn about the system’s behavior by observing it during a controlled experiments HOW

15. @theburningmonk theburningmonk.com learn about the system’s behavior by observing it during a controlled experiments HOW game days failure injection

16. Yan Cui http://theburningmonk.com @theburningmonk AWS user for 10 years

17. Yan Cui http://theburningmonk.com @theburningmonk http://bit.ly/yubl-serverless

18. Yan Cui http://theburningmonk.com @theburningmonk Developer Advocate @

19.

20. Yan Cui http://theburningmonk.com @theburningmonk Independent Consultant advisetraining delivery

21. theburningmonk.com/courses

22. theburningmonk.com/courses

23. realworldserverless.com

24. “using serverless reduces the blast radius” www.buzzsprout.com/877747/4615985

25. @theburningmonk theburningmonk.com serverless improves resilience as platform takes care of infrastructure failures

26. @theburningmonk theburningmonk.com

27. @theburningmonk theburningmonk.com by Russ Miles @russmiles source https://medium.com/russmiles/chaos-engineering-for-the-business-17b723f26361

28. @theburningmonk theburningmonk.com by Russ Miles @russmiles source https://medium.com/russmiles/chaos-engineering-for-the-business-17b723f26361

29. @theburningmonk theburningmonk.com Shared Responsibility Model

30. @theburningmonk theburningmonk.com by Russ Miles @russmiles source https://medium.com/russmiles/chaos-engineering-for-the-business-17b723f26361

31. @theburningmonk theburningmonk.com chaos monkey kills an EC2 instance latency monkey induces artiﬁcial delay in APIs chaos gorilla kills an AWS Availability Zone chaos kong kills an entire AWS region

32. @theburningmonk theburningmonk.com

33. @theburningmonk theburningmonk.com

34. @theburningmonk theburningmonk.com there are no servers to kill! SERVERLESS

35. @theburningmonk theburningmonk.com improperly tuned timeouts

36. @theburningmonk theburningmonk.com missing error handling

37. @theburningmonk theburningmonk.com missing fallbacks

38. @theburningmonk theburningmonk.com

39. @theburningmonk theburningmonk.com STEP 1. deﬁne steady state i.e. “what does normal look like”

40. @theburningmonk theburningmonk.com STEP 2. hypothesis that steady state continues in control and experimental group e.g. “the system stays up if a server dies”

41. @theburningmonk theburningmonk.com STEP 3. inject realistic failures e.g. “slow response from 3rd-party service”

42. @theburningmonk theburningmonk.com STEP 4. try to disprove hypothesis i.e. “look for difference between control and experimental group”

43. @theburningmonk theburningmonk.com latency inject latency to function invocation

44. @theburningmonk theburningmonk.com “what if service X has elevated latency?”

45. @theburningmonk theburningmonk.com API Gateway Lambda API Gateway Lambda

46. @theburningmonk theburningmonk.com

47. @theburningmonk theburningmonk.com hypothesis: API would timeout and our try-catch would handle it and return default response

48. @theburningmonk theburningmonk.com

49. @theburningmonk theburningmonk.com

50. @theburningmonk theburningmonk.com result: function times out after 6s (hypothesis is disproved)

51. @theburningmonk theburningmonk.com

52. @theburningmonk theburningmonk.com API Gateway Lambda API Gateway Lambda 502 200

53. @theburningmonk theburningmonk.com API Gateway Lambda API Gateway Lambda 3s timeout 6s timeout

54. @theburningmonk theburningmonk.com API Gateway Lambda API Gateway Lambda max 29s integration max 15 mins timeout

55. @theburningmonk theburningmonk.com and then there’s cold starts…

56. @theburningmonk theburningmonk.com TIL: most HTTP client libraries have default timeout of 60s. API Gateway has an integration timeout of 29s. Most Lambda functions default to timeout of 3-6s. Don’t forget about the cold starts!

57. @theburningmonk theburningmonk.com

58. @theburningmonk theburningmonk.com

59. @theburningmonk theburningmonk.com https://bit.ly/2Wvfort

60. @theburningmonk theburningmonk.com

61. @theburningmonk theburningmonk.com

62. @theburningmonk theburningmonk.com

63. @theburningmonk theburningmonk.com

64. @theburningmonk theburningmonk.com

65. @theburningmonk theburningmonk.com

66. @theburningmonk theburningmonk.com

67. @theburningmonk theburningmonk.com outcome: a more resilient system

68. @theburningmonk theburningmonk.com latency exception inject latency to function invocation throws exception

69. @theburningmonk theburningmonk.com latency exception statuscode inject latency to function invocation throws exception return HTTP status code

70. @theburningmonk theburningmonk.com latency exception statuscode diskspace inject latency to function invocation throws exception return HTTP status code ﬁlls up /tmp directory

71. @theburningmonk theburningmonk.com latency exception statuscode diskspace denylist inject latency to function invocation throws exception return HTTP status code ﬁlls up /tmp directory looses network connectivity

72. @theburningmonk theburningmonk.com “what if DynamoDB has an elevated error rate?”

73. @theburningmonk theburningmonk.com API Gateway Lambda DynamoDB

74. @theburningmonk theburningmonk.com

75. @theburningmonk theburningmonk.com hypothesis: the AWS SDK retries would handle it

76. @theburningmonk theburningmonk.com result: function times out after 6s (hypothesis is disproved)

77. @theburningmonk theburningmonk.com TIL: the js DynamoDB client defaults to 10 retries with base delay of 50ms

78. @theburningmonk theburningmonk.com TIL: the js DynamoDB client defaults to 10 retries with base delay of 50ms delay = Math.random() * (Math.pow(2, retryCount) * base) this is Marc Brooker’s fav formula!

79. @theburningmonk theburningmonk.com

80. @theburningmonk theburningmonk.com

81. @theburningmonk theburningmonk.com action: set max retry count + fallback

82. @theburningmonk theburningmonk.com

83. @theburningmonk theburningmonk.com

84. @theburningmonk theburningmonk.com

85. @theburningmonk theburningmonk.com

86. @theburningmonk theburningmonk.com outcome: a more resilient system

87. @theburningmonk theburningmonk.com latency exception statuscode diskspace denylist inject latency to function invocation throws exception return HTTP status code ﬁlls up /tmp directory looses network connectivity

88. everything fails, all the time

89. @theburningmonk theburningmonk.com by Russ Miles @russmiles source https://medium.com/russmiles/chaos-engineering-for-the-business-17b723f26361

90. @theburningmonk theburningmonk.com

91. https://theburningmonk.com/hire-me AdviseTraining Delivery “Fundamentally, Yan has improved our team by increasing our ability to derive value from AWS and Lambda in particular.” Nick Blair Tech Lead

92. @theburningmonk theburningmonk.com github.com/theburningmonk

A chaos experiment a day, keeping the outage away

Recommended

Recommended

More Related Content

Similar to A chaos experiment a day, keeping the outage away

Similar to A chaos experiment a day, keeping the outage away (20)

More from Yan Cui

More from Yan Cui (20)

Recently uploaded

Recently uploaded (20)

A chaos experiment a day, keeping the outage away