Hopelessness and Confidence in Distributed Systems Design

•

3 likes•1,862 views

The slides from my Strangeloop 2015 keynote Video is available here: https://www.youtube.com/watch?v=TlU1opuCXB0&feature=youtu.be&a

Technology

Hopelessness and Confidence in
Distributed Systems Design
Camille Fournier, Senior Thinker and
Raconteur
@skamille

The easiest
way to scale!
Hassle-free
scaling!
Easiest.
Database.
Ever.
Sync state in a cinch at a
distributed system scale.
Real-time
data
Real-time
Analytics
Real-time
Decisions

Distributed Systems: Ugly, Hard,
and Here to Stay

A brief interlude: The goals of
Distributed Systems

"DSM
FIGURE"
by
Mehrnazzhian
-‐
Own
work.
Licensed
under
CC
BY-‐SA
4.0
via
Commons

GLOBAL SERVICE DISCOVERY USING
ZOOKEEPER

Naïve approach: Central Coordinator!
•  Should all data be in one system? That would be
easy!
•  But then…
DISTANCE PERFOMANCE/

•  Speed of light
•  Limitations of bandwidth and latency
DISTANCE PERFOMANCE/

Fallacies of Distributed Computing
Deutsch et al, Sun Microsystems
1.  The network is reliable.
2. Latency is zero.
3. Bandwidth is infinite.
4.  The network is secure.
5.  Topology doesn't change.
6.  There is one administrator.
7.  Transport cost is zero.
8. The network is homogeneous.

BEST PRACTICE: Remember that
clients are part of your system

Scaling (systems and humans)
and failure tolerance (??)
/

COMMON
UNDERSTANDING
THE PERFECT
TOOL FOR THE
JOB
/
@coda

BEST PRACTICE: Data Design (again!)
•  Silence comes via good design!
•  You can always design poorly and have to
overcommunicate/coordinate

COMBAT HOPE AND FEAR BY
UNDERSTANDING TRADE/OFFS

Thank you!
Special Thanks:
@randommood
@SeanTAllen
@chrisk5000
@HenryR
@aphyr
@caitie
@pbailis (aka
Littlefinger)
@giltene
@MikeM6174
@tsantero
?s: @skamille

Similar to Hopelessness and Confidence in Distributed Systems Design

Reactive Summit Keynote 2020: https://www.youtube.com/watch?v=e5kek8vx2ws Abstract: Building applications for the cloud means embracing a radically different architecture than that of a traditional single-machine monolith, requiring new tools, practices, and design patterns. The cloud’s distributed nature brings its own set of concerns–building a Cloud Native, Edge Native, or Internet of Things (IoT) application means building and running a distributed system on unreliable hardware and across unreliable networks. In this keynote session by Jonas Bonér, creator of Akka, founder/CTO of Lightbend, and Chair of the Reactive Foundation, we’ll review a set of Reactive Principles that enable the design and implementation of Cloud Native applications–applications that are highly concurrent, distributed, performant, scalable, and resilient, while at the same time conserving resources when deploying, operating, and maintaining them.

The Reactive Principles: Design Principles For Cloud Native Applications

Jonas Bonér

Big Data Berlin – Automating Decisions is the Next Frontier for Big Data

Lars Trieloff

The Reactive Principles: Eight Tenets For Building Cloud Native Applications

Lightbend

TAMING THE INFRASTRUCTURE GONE WILD

Colloquium

TAMING THE INFRASTRUCTURE GONE WILD

Isaac Christoffersen

Chaos engineering

Alberto Acerbis

Successfully Migrate Cisco Call Manager 4x To 7x With a Proven Framework

glamba

Title: Bloom & CALM: Disorderly Distributed Computing and Minimal Coordination Speaker: Joseph Hellerstein, UC Berkeley Date & Time: Thurs, Apr 2, 12:00pm Lunch will be served. Abstract: The rise of cloud computing puts pressure on the software community to find new approaches to the difficulties of distributed and parallel programming. Traditional constructs like transactions and linearizability can preserve programmer illusions of sequentiality, but the coordination overheads of these constructs are often unattractive or untenable at global scales. In the absence of such constructs, there are few general principles or tools to help programmers design and verify the correctness of their applications. In the BOOM group at Berkeley we have addressed this situation on two fronts. First, we have designed Bloom, a “disorderly”, data-centric language for distributed computing that encourages order-insensitive, easily-parallelized programming. Second, we proposed the CALM Theorem, which formally links Consistency And Logical Monotonicity of distributed programs, and precisely characterizes the class of programs for which coordination is required. Said differently, CALM explains the expressive power of time in computer programs. CALM and Bloom provide a principled foundation for design patterns found in the distributed systems developer community, and allowed us to build software engineering tools that address the hard, domain-specific debugging questions that distributed programmers often ask: Is my program consistent? Is it fault-tolerant? Does it block unnecessarily on coordination boundaries? Joint work with Peter Alvaro, Ras Bodik, Neil Conway, David Maier, Bill Marczak and Sriram Srinivasan.

Bloom plseminar-sp15

Joe Hellerstein

The case for chaos testing

Peter Lamar

To stay competitive in today's global economy, organizations need to harness data from multiple sources, extract information and then make real-time decisions. Depending on the industry and the organization, Big Data encompasses information from multiple internal and external sources. Capturing, integrating and preparing this data for downstream analysis is often time-consuming and presents a big challenge. Today, organizations are struggling to cobble together different open source software to build an effective data pipeline that can withstand the volume as well as the speed of data ingestion and analysis. Robin relieves customers from the pains of building and maintaining a data pipeline and helps enterprises to make the most of Big Data. In this presentation, we will focus on how Robin’s containerization platform can be used to: - Build an agile and elastic data pipeline - Deploy, scale, and manage the most complex big data applications with just a single click of a button - Deal with variety, velocity, and volume of enterprise Big Data

Build agile and elastic data pipeline

Deba Chatterjee

Big Data and Machine Learning on AWS

CloudHesive

Control and Security - A constant risk to the network, and ultimately XYZ Account, are unapproved applications and rogue devices that may appear on the network and either permit unauthorized access or interfere with other devices. A means to monitor all devices and applications that operate across the network is vital. Just as important are the audit and reporting capabilities necessary to report on who, what, where, when, and how XYZ Account data is accessed. Our centralized management system will leverage data in the network to understand application use. Maintaining high quality user experience (Single-Pane-of-Glass Control from BYOD to the Data Center). Minimizing risk from consumer products and mobile devices (automation of routine tasks). Identifying root-cause of service outage (Make decisions about your network based on analytics, not assumptions.). An experience is a personal and emotional event we remember. Every experience is established based upon pre-determined expectations we conceive and create in our minds. It’s personal, and therefore, remains a moving and evolving target in every scenario. When our experience concludes and the moment has passed, the outcome remains in our memory. Think about what makes you happy when connecting with your own device and then think about what makes you really upset when things are hard, complicated, and slow. If the user has a bad experience in anyone of these areas (simple, fast, and smart), they are likely to leave, share their negative experience, and potentially never return. Users might forget facts or details about their computing environment but they find it difficult to forgot the feeling behind a bad network experience. When something goes wrong with the network or an application, do you always get the blame? Business alignment - Over time, the proliferation of devices has created unnecessary complexity. Control Center delivers centralized visibility and granular control of network resources. One click can equal a thousand actions when you manage your network. Control Center can even manage beyond Extreme Networks switching, routing, and wireless hardware to deliver standards-based control of other vendors’ network equipment. Transform complex network data into actionable information (Gain visibility from data in your network). Centralize and simplify the definition, management, and enforcement of policies (Detect anomalies and get alerts based on real network behavior). Manage third-party devices to provide a complete picture of the entire infrastructure in a heterogeneous network environment (Balance CapEx and OpEx and decrease complexity).

1.) extreme networks (top 10 reasons)

Jeff Green

Introduction to Distributed Computing & Distributed Databases

Shankar Iyer

Resilience and Compliance at Speed and Scale

Jason Chan

iland and Veeam recently conducted a data protection survey of IT organizations worldwide. In this webinar, we summarize and analyze the survey responses so you canunderstand today’s data protection landscape. Then, we cover best practices that can help ensure thatyour organization, and its data,are properly protected Watch the webinar on-demand: https://www.iland.com/wb-data-protection-report/

The Great Disconnect of Data Protection: Perception, Reality and Best Practices

iland Cloud

Talk at the Boston Cloud Foundry Meetup June 2015

Chip Childers

Building computer systems that are reliable is hard. The functional programming community has invested a lot of time and energy into up-front-correctness guarantees: types and the like. Unfortunately, absolutely correct software is time-consuming to write and expensive as a result. Fault-tolerant systems achieve system-total reliability by accepting that sub-components will fail and planning for that failure as a first-class concern of the system. As companies embrace the wave of "as-a-service" architectures, failure of sub-systems become a more pressing concern. Using examples from heavy industry, aeronautics and telecom systems, this talk will explore how you can design for fault-tolerance and how functional programming techniques get us most of the way there.

Fault-tolerance on the Cheap: Making Systems That (Probably) Won't Fall Over

Brian Troutwine

Data centers of all sizes are looking for ways to increase the return on investment on their virtualized (desktop and server) infrastructures. Converged infrastructures propose to increase ROI by reducing the number of layers, essentially combining compute, storage and networking into a single tier. But are these architectures right for you? In this webinar, join experts from Storage Switzerland and Scale Computing to find out.

Webinar: Is Convergence right for you? – 4 questions to ask

Storage Switzerland

Inerview Quesion on Data Mining and Machine Learning

Yash Diwakar

Interactive Session on "Chaos engineering: Break it to make it" by Anupam Agarwal,Nagarro, Peeyush Girdhar, Cloud / DevOps Nagarro. at #ATAGTR2021. #ATAGTR2021 was the 6th Edition of Global Testing Retreat. The video recording of the session is now available on the following link: https://www.youtube.com/watch?v=4bM4f8xNp2A To know more about #ATAGTR2021, please visit:https://gtr.agiletestingalliance.org/

#ATAGTR2021 Presentation : "Chaos engineering: Break it to make it" by Anupa...

Agile Testing Alliance

Similar to Hopelessness and Confidence in Distributed Systems Design (20)

The Reactive Principles: Design Principles For Cloud Native Applications

Big Data Berlin – Automating Decisions is the Next Frontier for Big Data

The Reactive Principles: Eight Tenets For Building Cloud Native Applications

TAMING THE INFRASTRUCTURE GONE WILD

Chaos engineering

Successfully Migrate Cisco Call Manager 4x To 7x With a Proven Framework

Bloom plseminar-sp15

The case for chaos testing

Build agile and elastic data pipeline

Big Data and Machine Learning on AWS

1.) extreme networks (top 10 reasons)

Introduction to Distributed Computing & Distributed Databases

Resilience and Compliance at Speed and Scale

The Great Disconnect of Data Protection: Perception, Reality and Best Practices

Talk at the Boston Cloud Foundry Meetup June 2015

Fault-tolerance on the Cheap: Making Systems That (Probably) Won't Fall Over

Webinar: Is Convergence right for you? – 4 questions to ask

Inerview Quesion on Data Mining and Machine Learning

#ATAGTR2021 Presentation : "Chaos engineering: Break it to make it" by Anupa...

Recently uploaded

ICT role in 21st century education and its challenges

rafiqahmad00786416

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Edi Saputra

💉💊+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHABI}}+971581248768 +971581248768 Mtp-Kit (500MG) Prices » Dubai [(+971581248768**)] Abortion Pills For Sale In Dubai, UAE, Mifepristone and Misoprostol Tablets Available In Dubai, UAE CONTACT DR.Maya Whatsapp +971581248768 We Have Abortion Pills / Cytotec Tablets /Mifegest Kit Available in Dubai, Sharjah, Abudhabi, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE, Buy cytotec in Dubai +971581248768''''Abortion Pills near me DUBAI | ABU DHABI|UAE. Price of Misoprostol, Cytotec” +971581248768' Dr.DEEM ''BUY ABORTION PILLS MIFEGEST KIT, MISOPROTONE, CYTOTEC PILLS IN DUBAI, ABU DHABI,UAE'' Contact me now via What's App…… abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain Above all, Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg abortion pill in Dubai, UAE. Medication abortion offers an alternative to Surgical Abortion for women in the early weeks of pregnancy. We only offer abortion pills from 1 week-6 Months. We then advise you to use surgery if its beyond 6 months. Our Abu Dhabi, Ajman, Al Ain, Dubai, Fujairah, Ras Al Khaimah (RAK), Sharjah, Umm Al Quwain (UAQ) United Arab Emirates Abortion Clinic provides the safest and most advanced techniques for providing non-surgical, medical and surgical abortion methods for early through late second trimester, including the Abortion By Pill Procedure (RU 486, Mifeprex, Mifepristone, early options French Abortion Pill), Tamoxifen, Methotrexate and Cytotec (Misoprostol). The Abu Dhabi, United Arab Emirates Abortion Clinic performs Same Day Abortion Procedure using medications that are taken on the first day of the office visit and will cause the abortion to occur generally within 4 to 6 hours (as early as 30 minutes) for patients who are 3 to 12 weeks pregnant. When Mifepristone and Misoprostol are used, 50% of patients complete in 4 to 6 hours; 75% to 80% in 12 hours; and 90% in 24 hours. We use a regimen that allows for completion without the need for surgery 99% of the time. All advanced second trimester and late term pregnancies at our Tampa clinic (17 to 24 weeks or greater) can be completed within 24 hours or less 99% of the time without the need surgery. The procedure is completed with minimal to no complications. Our Women's Health Center located in Abu Dhabi, United Arab Emirates, uses the latest medications for medical abortions (RU-486, Mifeprex, Mifegyne, Mifepristone, early options French abortion pill), Methotrexate and Cytotec (Misoprostol). The safety standards of our Abu Dhabi, United Arab Emirates Abortion Doctors remain unparalleled. They consistently maintain the lowest complication rates throughout the nation. Our Physicians and staff are always available to answer questions and care for women in one of the most difficult times in their lives. The decision to have an abortion at the Abortion Cl

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@

FWD Group - Insurer Innovation Award 2024

The Digital Insurer

In this session, we will delve into strategic approaches for optimizing knowledge management within Microsoft 365, amidst the evolving landscape of Copilot. From leveraging automatic metadata classification and permission governance with SharePoint Premium, to unlocking Viva Engage for the cultivation of knowledge and communities, you will gain actionable insights to bolster your organization's knowledge-sharing initiatives. In this session, we will also explore how to facilitate solutions to enable your employees to find answers and expertise within Microsoft 365. You will leave equipped with practical techniques and a deeper understanding of how there is more to effective knowledge management than just enabling Copilot, but building actual solutions to prepare the knowledge that Copilot and your employees can use.

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Drew Madelung

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

DBX First Quarter 2024 Investor Presentation

Dropbox

AXA XL - Insurer Innovation Award Americas 2024

The Digital Insurer

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the deployment of external web forms using Jotform for Bonterra Impact Management. This solution can be customized to your organization’s needs and deployed to support the common use cases below: - Intake and consent - Assessments - Surveys - Applications - Program registration Interested in deploying web form automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Jeffrey Haguewood

Manulife - Insurer Transformation Award 2024

The Digital Insurer

MINDCTI Revenue Release Quarter One 2024

MIND CTI

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

apidays

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

GenAI Risks & Security Meetup 01052024.pdf

lior mazor

Modernizing Securities Finance: The cloud-native prime brokerage platform transforming capital markets. Madhu Subbu, Managing Director, Head of Securities Finance Engineering Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu

apidays

Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...

Zilliz

Corporate and higher education. Two industries that, in the past, have had a clear divide with very little crossover. The difference in goals, learning styles and objectives paved the way for differing learning technologies platforms to evolve. Now, those stark lines are blurring as both sides are discovering they have content that’s relevant to the other. Join Tammy Rutherford as she walks through the pros and cons of corporate and higher ed collaborating. And the challenges of these different technology platforms working together for a brighter future.

Corporate and higher education May webinar.pptx

Rustici Software

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

wesley chun

Recently uploaded (20)

ICT role in 21st century education and its challenges

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

FWD Group - Insurer Innovation Award 2024

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Automating Google Workspace (GWS) & more with Apps Script

Axa Assurance Maroc - Insurer Innovation Award 2024

DBX First Quarter 2024 Investor Presentation

AXA XL - Insurer Innovation Award Americas 2024

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Manulife - Insurer Transformation Award 2024

MINDCTI Revenue Release Quarter One 2024

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

presentation ICT roal in 21st century education

GenAI Risks & Security Meetup 01052024.pdf

Apidays Singapore 2024 - Modernizing Securities Finance by Madhu Subbu

Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...

Corporate and higher education May webinar.pptx

Boost Fertility New Invention Ups Success Rates.pdf

Powerful Google developer tools for immediate impact! (2023-24 C)

Hopelessness and Confidence in Distributed Systems Design

1. Hopelessness and Confidence in Distributed Systems Design Camille Fournier, Senior Thinker and Raconteur @skamille

2. The easiest way to scale! Hassle-free scaling! Easiest. Database. Ever. Sync state in a cinch at a distributed system scale. Real-time data Real-time Analytics Real-time Decisions

3. @aphyr

4. Distributed Systems: Ugly, Hard, and Here to Stay

7. TRADE/OFF

8. TRADE/OFF

9. Embrace Hopelessness Gain Confidence

10.

11. A brief interlude: The goals of Distributed Systems

12. SCALE

13. FAILURE TOLERANCE

14.

15. A DISTRIBUTED CACHE FOR RISK ANALYSIS

16.

17.

18. Scaling /

19. "DSM FIGURE" by Mehrnazzhian -‐ Own work. Licensed under CC BY-‐SA 4.0 via Commons

20. Azul

21.

22. SYNCHRONIZATION PERFORMANCE/

23. REWRITING MODIFICATION/

24. BEST practice: TESTING

25. GLOBAL SERVICE DISCOVERY USING ZOOKEEPER

26. GLOBAL service discovery system

27. Failure Tolerance Scaling /

28. Naïve approach: Central Coordinator! •  Should all data be in one system? That would be easy! •  But then… DISTANCE PERFOMANCE/

29. •  Speed of light •  Limitations of bandwidth and latency DISTANCE PERFOMANCE/

30. Fallacies of Distributed Computing Deutsch et al, Sun Microsystems 1.  The network is reliable. 2. Latency is zero. 3. Bandwidth is infinite. 4.  The network is secure. 5.  Topology doesn't change. 6.  There is one administrator. 7.  Transport cost is zero. 8. The network is homogeneous.

31. Wide Area Networks Add Complexity

32. FEATURES CONTROL/

33. BEST PRACTICE: Remember that clients are part of your system

34. A second brief interlude…

35.

36. MICROSERVICES AT RENT THE RUNWAY

37. Evolution off of a monolith

38. Scaling (systems and humans) and failure tolerance (??) /

39. COMMON UNDERSTANDING THE PERFECT TOOL FOR THE JOB / @coda

40. RATE OF CHANGE THOROUGHNESS/

41. @adrianco

42. BEST PRACTICE: DATA DESIGN (AGAIN!)