SlideShare a Scribd company logo
A concrete talk about abstraction.
Theo Schlossnagle @postwait 1
 Founder: Circonus, OmniTI, Sparkpost, Fontdeck
 Distinguish ACM Member
 ACM Member at Large
 Avid Open Source Contributor
 Gentleman Farmer
Theo Schlossnagle @postwait 2
Boundaries in the software/hardware and software/human interface.
Also in the software/software interface.
Theo Schlossnagle @postwait 3
Transistors
Hardware Function
Micro Architecture
Instruction Set Architecture
Machine Code
Assembly Languages
Programming Languages
Algorithms/Data Structures
Applications
Theo Schlossnagle @postwait 4
Hardware
Software
Abstractions boundaries are at a
significantly lower dimension
(typically one)
than the space they separate
(typically greater than four).
Theo Schlossnagle @postwait 5
Task /
Problem
Skills
Efficiency /
Resources
Safety /
Correctness
Availability
Security /
Compliance
Time Series Data
Histogram
RocksDB
Raw Data
Column Families
Rollups
Tiers
Time Shards
LMDB
ZFS
Files
Abstractions are divides,
and the world isn’t always clean.
Performance requirements are
often a witness.
Result:
“Leaky Abstractions” or
“Abstraction Violations”
Theo Schlossnagle @postwait 6
Time Series Data
(for rebuild)
Time Series Data
(for rollup ETL)
Constraints Design
Performanc
e
AudienceScale
Resources
Techniques
API
Boundary
Abstraction
Core Problem
Abstractions aim to be generically useful.
The fewer initial consumers, the less likely this will be satisfied.
Theo Schlossnagle @postwait 7
Abstractions interfere with holistic
systems thinking.
Problems span layers.
Therefore analysis spans layers.
Abstractions are loans.
If abstraction boundaries bisect
common analysis tasks, those end
up being high-interest loans.
Theo Schlossnagle @postwait 8
Layer 1
client
communications
Layer 2
distributed
coordination
Layer 3
storage
layer
Expensive analysis
Inexpensiveanalysis
Abstractions in real-life can win, can lose, but most often do both.
Theo Schlossnagle @postwait 9
 Time Series Database
 Distributed
 Scalable
 High performance
• HTTP
• Lua
• CRDTs
• Math
• Resource Pools
• RocksDB
• C/C++/Asm
Speed Scale
AccessibilityCorrectness
Theo Schlossnagle @postwait 10
A sordid love story of layer violations and compromise.
Theo Schlossnagle @postwait 11
 libmtev is a multi-threaded, event-driven framework for
building high-performance service applications in C
 It makes it “easy”… for C developers that understand event-driven
code.
 Lua is an elegant and accessible functional language
 LuaJIT (is lua, but fast)
 LuaJIT’s embedding API make violating boundaries
convenient.
Theo Schlossnagle @postwait 12
Mistakes layered on mistakes layered on mistakes.
Theo Schlossnagle @postwait 13
 updating, appending, deleting
 reading randomly, sequentially, bulk
 It’s basically impossible to excel at all without layering
violations.
Theo Schlossnagle @postwait 14
Concurrency limitations and fair scheduling.
Theo Schlossnagle @postwait 15
 Resources (CPU, memory, disk, L2 cache, network, etc.) are
limited.
 Concurrency can help until… it hurts.
 For performance, we abstract on resource usage
boundaries/workload characteristics.
Theo Schlossnagle @postwait 16
Concrete lessons about abstractions.
Theo Schlossnagle @postwait 17
No one would get anything done. Ever.
Theo Schlossnagle @postwait 18
Microservices (in particular) afford
isolation choices beyond code
complexity: security, compliance,
availability, ownership, etc.
The dimension on which the abstraction
boundary is drawn should be selected to
yield the most benefit.
Theo Schlossnagle @postwait 19
The more things (and ultimately people)
that use the abstraction, the better it will
be designed (or subsequently
redesigned).
Never build an API without a real
consumer (TDD doesn’t count). This
should apply to all simplifications.
Theo Schlossnagle @postwait 20
They are not a goal, but they are not
evil. It should be intentional.
Some things behind an abstraction
should intuitively be hidden. Intuition
isn’t always right. When these leaks
consistently benefit multiple
stakeholders, it simply means the line of
abstraction has been redrawn.
Theo Schlossnagle @postwait 21
They happen.
They happen all the time.
I am unapologetic about this.
Violation carries cost burden.
Assess it, accept it, and move on.
Theo Schlossnagle @postwait 22
https://irondb.io/
https://github.com/circonus-labs/libmtev
Theo Schlossnagle @postwait 23

More Related Content

Similar to Adding Simplicity to Complexity

FIWARE Wednesday Webinars - Performing Big Data Analysis Using Cosmos With Sp...
FIWARE Wednesday Webinars - Performing Big Data Analysis Using Cosmos With Sp...FIWARE Wednesday Webinars - Performing Big Data Analysis Using Cosmos With Sp...
FIWARE Wednesday Webinars - Performing Big Data Analysis Using Cosmos With Sp...
FIWARE
 
Beyond the RTOS: A Better Way to Design Real-Time Embedded Software
Beyond the RTOS: A Better Way to Design Real-Time Embedded SoftwareBeyond the RTOS: A Better Way to Design Real-Time Embedded Software
Beyond the RTOS: A Better Way to Design Real-Time Embedded Software
Quantum Leaps, LLC
 
Reactive Programming Models for IoT
Reactive Programming Models for IoTReactive Programming Models for IoT
Reactive Programming Models for IoT
Todd Montgomery
 
Concurrent Programming with Ruby and Tuple Spaces
Concurrent Programming with Ruby and Tuple SpacesConcurrent Programming with Ruby and Tuple Spaces
Concurrent Programming with Ruby and Tuple Spaces
luccastera
 
Overview of QP Frameworks and QM Modeling Tools (Notes)
Overview of QP Frameworks and QM Modeling Tools (Notes)Overview of QP Frameworks and QM Modeling Tools (Notes)
Overview of QP Frameworks and QM Modeling Tools (Notes)
Quantum Leaps, LLC
 
Need for Async: Hot pursuit for scalable applications
Need for Async: Hot pursuit for scalable applicationsNeed for Async: Hot pursuit for scalable applications
Need for Async: Hot pursuit for scalable applications
Konrad Malawski
 
Introduction to Multimodal LLMs with LLaVA
Introduction to Multimodal LLMs with LLaVAIntroduction to Multimodal LLMs with LLaVA
Introduction to Multimodal LLMs with LLaVA
Robert McDermott
 
Introduction to Multimodal LLMs with LLaVA
Introduction to Multimodal LLMs with LLaVAIntroduction to Multimodal LLMs with LLaVA
Introduction to Multimodal LLMs with LLaVA
Robert McDermott
 
Erlang
ErlangErlang
Erlang
ESUG
 
Beyond the RTOS: A Better Way to Design Real-Time Embedded Software
Beyond the RTOS: A Better Way to Design Real-Time Embedded SoftwareBeyond the RTOS: A Better Way to Design Real-Time Embedded Software
Beyond the RTOS: A Better Way to Design Real-Time Embedded Software
Miro Samek
 
Un actor (model) per amico - Alessandro Melchiori - Codemotion Milan 2016
Un actor (model) per amico - Alessandro Melchiori - Codemotion Milan 2016Un actor (model) per amico - Alessandro Melchiori - Codemotion Milan 2016
Un actor (model) per amico - Alessandro Melchiori - Codemotion Milan 2016
Codemotion
 
BUD17-104: Scripting Languages in IoT: Challenges and Approaches
BUD17-104: Scripting Languages in IoT: Challenges and ApproachesBUD17-104: Scripting Languages in IoT: Challenges and Approaches
BUD17-104: Scripting Languages in IoT: Challenges and Approaches
Linaro
 
Clair Tolan - Passwords for the clouds
Clair Tolan - Passwords for the cloudsClair Tolan - Passwords for the clouds
Clair Tolan - Passwords for the clouds
Open Labs Albania
 
Infrastructure as code might be literally impossible part 2
Infrastructure as code might be literally impossible part 2Infrastructure as code might be literally impossible part 2
Infrastructure as code might be literally impossible part 2
ice799
 
UTOPOLL白皮書.pdf
UTOPOLL白皮書.pdfUTOPOLL白皮書.pdf
UTOPOLL白皮書.pdf
aipaypoll
 
Modern Embedded Software
Modern Embedded SoftwareModern Embedded Software
Modern Embedded Software
Quantum Leaps, LLC
 
Extreme Network Performance with Hazelcast on Torusware
Extreme Network Performance with Hazelcast on ToruswareExtreme Network Performance with Hazelcast on Torusware
Extreme Network Performance with Hazelcast on Torusware
Hazelcast
 
Oscon keynote: Working hard to keep it simple
Oscon keynote: Working hard to keep it simpleOscon keynote: Working hard to keep it simple
Oscon keynote: Working hard to keep it simple
Martin Odersky
 
QCon NY 2014 - Evolving REST for an IoT World
QCon NY 2014 - Evolving REST for an IoT WorldQCon NY 2014 - Evolving REST for an IoT World
QCon NY 2014 - Evolving REST for an IoT WorldTodd Montgomery
 
Joe armstrong erlanga_languageforprogrammingreliablesystems
Joe armstrong erlanga_languageforprogrammingreliablesystemsJoe armstrong erlanga_languageforprogrammingreliablesystems
Joe armstrong erlanga_languageforprogrammingreliablesystems
Sentifi
 

Similar to Adding Simplicity to Complexity (20)

FIWARE Wednesday Webinars - Performing Big Data Analysis Using Cosmos With Sp...
FIWARE Wednesday Webinars - Performing Big Data Analysis Using Cosmos With Sp...FIWARE Wednesday Webinars - Performing Big Data Analysis Using Cosmos With Sp...
FIWARE Wednesday Webinars - Performing Big Data Analysis Using Cosmos With Sp...
 
Beyond the RTOS: A Better Way to Design Real-Time Embedded Software
Beyond the RTOS: A Better Way to Design Real-Time Embedded SoftwareBeyond the RTOS: A Better Way to Design Real-Time Embedded Software
Beyond the RTOS: A Better Way to Design Real-Time Embedded Software
 
Reactive Programming Models for IoT
Reactive Programming Models for IoTReactive Programming Models for IoT
Reactive Programming Models for IoT
 
Concurrent Programming with Ruby and Tuple Spaces
Concurrent Programming with Ruby and Tuple SpacesConcurrent Programming with Ruby and Tuple Spaces
Concurrent Programming with Ruby and Tuple Spaces
 
Overview of QP Frameworks and QM Modeling Tools (Notes)
Overview of QP Frameworks and QM Modeling Tools (Notes)Overview of QP Frameworks and QM Modeling Tools (Notes)
Overview of QP Frameworks and QM Modeling Tools (Notes)
 
Need for Async: Hot pursuit for scalable applications
Need for Async: Hot pursuit for scalable applicationsNeed for Async: Hot pursuit for scalable applications
Need for Async: Hot pursuit for scalable applications
 
Introduction to Multimodal LLMs with LLaVA
Introduction to Multimodal LLMs with LLaVAIntroduction to Multimodal LLMs with LLaVA
Introduction to Multimodal LLMs with LLaVA
 
Introduction to Multimodal LLMs with LLaVA
Introduction to Multimodal LLMs with LLaVAIntroduction to Multimodal LLMs with LLaVA
Introduction to Multimodal LLMs with LLaVA
 
Erlang
ErlangErlang
Erlang
 
Beyond the RTOS: A Better Way to Design Real-Time Embedded Software
Beyond the RTOS: A Better Way to Design Real-Time Embedded SoftwareBeyond the RTOS: A Better Way to Design Real-Time Embedded Software
Beyond the RTOS: A Better Way to Design Real-Time Embedded Software
 
Un actor (model) per amico - Alessandro Melchiori - Codemotion Milan 2016
Un actor (model) per amico - Alessandro Melchiori - Codemotion Milan 2016Un actor (model) per amico - Alessandro Melchiori - Codemotion Milan 2016
Un actor (model) per amico - Alessandro Melchiori - Codemotion Milan 2016
 
BUD17-104: Scripting Languages in IoT: Challenges and Approaches
BUD17-104: Scripting Languages in IoT: Challenges and ApproachesBUD17-104: Scripting Languages in IoT: Challenges and Approaches
BUD17-104: Scripting Languages in IoT: Challenges and Approaches
 
Clair Tolan - Passwords for the clouds
Clair Tolan - Passwords for the cloudsClair Tolan - Passwords for the clouds
Clair Tolan - Passwords for the clouds
 
Infrastructure as code might be literally impossible part 2
Infrastructure as code might be literally impossible part 2Infrastructure as code might be literally impossible part 2
Infrastructure as code might be literally impossible part 2
 
UTOPOLL白皮書.pdf
UTOPOLL白皮書.pdfUTOPOLL白皮書.pdf
UTOPOLL白皮書.pdf
 
Modern Embedded Software
Modern Embedded SoftwareModern Embedded Software
Modern Embedded Software
 
Extreme Network Performance with Hazelcast on Torusware
Extreme Network Performance with Hazelcast on ToruswareExtreme Network Performance with Hazelcast on Torusware
Extreme Network Performance with Hazelcast on Torusware
 
Oscon keynote: Working hard to keep it simple
Oscon keynote: Working hard to keep it simpleOscon keynote: Working hard to keep it simple
Oscon keynote: Working hard to keep it simple
 
QCon NY 2014 - Evolving REST for an IoT World
QCon NY 2014 - Evolving REST for an IoT WorldQCon NY 2014 - Evolving REST for an IoT World
QCon NY 2014 - Evolving REST for an IoT World
 
Joe armstrong erlanga_languageforprogrammingreliablesystems
Joe armstrong erlanga_languageforprogrammingreliablesystemsJoe armstrong erlanga_languageforprogrammingreliablesystems
Joe armstrong erlanga_languageforprogrammingreliablesystems
 

More from Theo Schlossnagle

Monitoring 101
Monitoring 101Monitoring 101
Monitoring 101
Theo Schlossnagle
 
Distributed Systems - Like It Or Not
Distributed Systems - Like It Or NotDistributed Systems - Like It Or Not
Distributed Systems - Like It Or Not
Theo Schlossnagle
 
Applying SRE techniques to micro service design
Applying SRE techniques to micro service designApplying SRE techniques to micro service design
Applying SRE techniques to micro service design
Theo Schlossnagle
 
Craftsmanship
CraftsmanshipCraftsmanship
Craftsmanship
Theo Schlossnagle
 
SRECon Coherent Performance
SRECon Coherent PerformanceSRECon Coherent Performance
SRECon Coherent Performance
Theo Schlossnagle
 
Commandments of scale
Commandments of scaleCommandments of scale
Commandments of scale
Theo Schlossnagle
 
Adaptive availability
Adaptive availabilityAdaptive availability
Adaptive availability
Theo Schlossnagle
 
Project reality
Project realityProject reality
Project reality
Theo Schlossnagle
 
Monitoring the #DevOps way
Monitoring the #DevOps wayMonitoring the #DevOps way
Monitoring the #DevOps way
Theo Schlossnagle
 
Operational Software Design
Operational Software DesignOperational Software Design
Operational Software Design
Theo Schlossnagle
 
A Coherent Discussion About Performance
A Coherent Discussion About PerformanceA Coherent Discussion About Performance
A Coherent Discussion About PerformanceTheo Schlossnagle
 
The math behind big systems analysis.
The math behind big systems analysis.The math behind big systems analysis.
The math behind big systems analysis.
Theo Schlossnagle
 
Understanding Slowness
Understanding SlownessUnderstanding Slowness
Understanding Slowness
Theo Schlossnagle
 
OmniOS Motivation and Design ~ LISA 2012
OmniOS Motivation and Design ~ LISA 2012OmniOS Motivation and Design ~ LISA 2012
OmniOS Motivation and Design ~ LISA 2012Theo Schlossnagle
 
Monitoring and observability
Monitoring and observabilityMonitoring and observability
Monitoring and observabilityTheo Schlossnagle
 
Monitoring and observability
Monitoring and observabilityMonitoring and observability
Monitoring and observability
Theo Schlossnagle
 
Xtreme Deployment
Xtreme DeploymentXtreme Deployment
Xtreme Deployment
Theo Schlossnagle
 
Atldevops
AtldevopsAtldevops
It's all about telemetry
It's all about telemetryIt's all about telemetry
It's all about telemetry
Theo Schlossnagle
 

More from Theo Schlossnagle (20)

Monitoring 101
Monitoring 101Monitoring 101
Monitoring 101
 
Distributed Systems - Like It Or Not
Distributed Systems - Like It Or NotDistributed Systems - Like It Or Not
Distributed Systems - Like It Or Not
 
Applying SRE techniques to micro service design
Applying SRE techniques to micro service designApplying SRE techniques to micro service design
Applying SRE techniques to micro service design
 
Craftsmanship
CraftsmanshipCraftsmanship
Craftsmanship
 
SRECon Coherent Performance
SRECon Coherent PerformanceSRECon Coherent Performance
SRECon Coherent Performance
 
Commandments of scale
Commandments of scaleCommandments of scale
Commandments of scale
 
Adaptive availability
Adaptive availabilityAdaptive availability
Adaptive availability
 
Project reality
Project realityProject reality
Project reality
 
Monitoring the #DevOps way
Monitoring the #DevOps wayMonitoring the #DevOps way
Monitoring the #DevOps way
 
Operational Software Design
Operational Software DesignOperational Software Design
Operational Software Design
 
A Coherent Discussion About Performance
A Coherent Discussion About PerformanceA Coherent Discussion About Performance
A Coherent Discussion About Performance
 
The math behind big systems analysis.
The math behind big systems analysis.The math behind big systems analysis.
The math behind big systems analysis.
 
Understanding Slowness
Understanding SlownessUnderstanding Slowness
Understanding Slowness
 
OmniOS Motivation and Design ~ LISA 2012
OmniOS Motivation and Design ~ LISA 2012OmniOS Motivation and Design ~ LISA 2012
OmniOS Motivation and Design ~ LISA 2012
 
Monitoring and observability
Monitoring and observabilityMonitoring and observability
Monitoring and observability
 
Omnios and unix
Omnios and unixOmnios and unix
Omnios and unix
 
Monitoring and observability
Monitoring and observabilityMonitoring and observability
Monitoring and observability
 
Xtreme Deployment
Xtreme DeploymentXtreme Deployment
Xtreme Deployment
 
Atldevops
AtldevopsAtldevops
Atldevops
 
It's all about telemetry
It's all about telemetryIt's all about telemetry
It's all about telemetry
 

Recently uploaded

Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
KatiaHIMEUR1
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Tobias Schneck
 
FIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdfFIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance
 
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
BookNet Canada
 
Assuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyesAssuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyes
ThousandEyes
 
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
Product School
 
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 previewState of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
Prayukth K V
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Product School
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
Elena Simperl
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Inflectra
 
UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4
DianaGray10
 
Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*
Frank van Harmelen
 
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdfFIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance
 
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdfFIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance
 
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
UiPathCommunity
 
Elevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object CalisthenicsElevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object Calisthenics
Dorra BARTAGUIZ
 
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Product School
 
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
Product School
 
Leading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdfLeading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdf
OnBoard
 
Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........
Alison B. Lowndes
 

Recently uploaded (20)

Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
 
FIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdfFIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdf
 
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
 
Assuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyesAssuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyes
 
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
 
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 previewState of ICS and IoT Cyber Threat Landscape Report 2024 preview
State of ICS and IoT Cyber Threat Landscape Report 2024 preview
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
 
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
 
UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4
 
Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*Neuro-symbolic is not enough, we need neuro-*semantic*
Neuro-symbolic is not enough, we need neuro-*semantic*
 
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdfFIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
 
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdfFIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
 
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...
 
Elevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object CalisthenicsElevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object Calisthenics
 
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...
 
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
 
Leading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdfLeading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdf
 
Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........
 

Adding Simplicity to Complexity

  • 1. A concrete talk about abstraction. Theo Schlossnagle @postwait 1
  • 2.  Founder: Circonus, OmniTI, Sparkpost, Fontdeck  Distinguish ACM Member  ACM Member at Large  Avid Open Source Contributor  Gentleman Farmer Theo Schlossnagle @postwait 2
  • 3. Boundaries in the software/hardware and software/human interface. Also in the software/software interface. Theo Schlossnagle @postwait 3
  • 4. Transistors Hardware Function Micro Architecture Instruction Set Architecture Machine Code Assembly Languages Programming Languages Algorithms/Data Structures Applications Theo Schlossnagle @postwait 4 Hardware Software
  • 5. Abstractions boundaries are at a significantly lower dimension (typically one) than the space they separate (typically greater than four). Theo Schlossnagle @postwait 5 Task / Problem Skills Efficiency / Resources Safety / Correctness Availability Security / Compliance
  • 6. Time Series Data Histogram RocksDB Raw Data Column Families Rollups Tiers Time Shards LMDB ZFS Files Abstractions are divides, and the world isn’t always clean. Performance requirements are often a witness. Result: “Leaky Abstractions” or “Abstraction Violations” Theo Schlossnagle @postwait 6 Time Series Data (for rebuild) Time Series Data (for rollup ETL)
  • 7. Constraints Design Performanc e AudienceScale Resources Techniques API Boundary Abstraction Core Problem Abstractions aim to be generically useful. The fewer initial consumers, the less likely this will be satisfied. Theo Schlossnagle @postwait 7
  • 8. Abstractions interfere with holistic systems thinking. Problems span layers. Therefore analysis spans layers. Abstractions are loans. If abstraction boundaries bisect common analysis tasks, those end up being high-interest loans. Theo Schlossnagle @postwait 8 Layer 1 client communications Layer 2 distributed coordination Layer 3 storage layer Expensive analysis Inexpensiveanalysis
  • 9. Abstractions in real-life can win, can lose, but most often do both. Theo Schlossnagle @postwait 9
  • 10.  Time Series Database  Distributed  Scalable  High performance • HTTP • Lua • CRDTs • Math • Resource Pools • RocksDB • C/C++/Asm Speed Scale AccessibilityCorrectness Theo Schlossnagle @postwait 10
  • 11. A sordid love story of layer violations and compromise. Theo Schlossnagle @postwait 11
  • 12.  libmtev is a multi-threaded, event-driven framework for building high-performance service applications in C  It makes it “easy”… for C developers that understand event-driven code.  Lua is an elegant and accessible functional language  LuaJIT (is lua, but fast)  LuaJIT’s embedding API make violating boundaries convenient. Theo Schlossnagle @postwait 12
  • 13. Mistakes layered on mistakes layered on mistakes. Theo Schlossnagle @postwait 13
  • 14.  updating, appending, deleting  reading randomly, sequentially, bulk  It’s basically impossible to excel at all without layering violations. Theo Schlossnagle @postwait 14
  • 15. Concurrency limitations and fair scheduling. Theo Schlossnagle @postwait 15
  • 16.  Resources (CPU, memory, disk, L2 cache, network, etc.) are limited.  Concurrency can help until… it hurts.  For performance, we abstract on resource usage boundaries/workload characteristics. Theo Schlossnagle @postwait 16
  • 17. Concrete lessons about abstractions. Theo Schlossnagle @postwait 17
  • 18. No one would get anything done. Ever. Theo Schlossnagle @postwait 18
  • 19. Microservices (in particular) afford isolation choices beyond code complexity: security, compliance, availability, ownership, etc. The dimension on which the abstraction boundary is drawn should be selected to yield the most benefit. Theo Schlossnagle @postwait 19
  • 20. The more things (and ultimately people) that use the abstraction, the better it will be designed (or subsequently redesigned). Never build an API without a real consumer (TDD doesn’t count). This should apply to all simplifications. Theo Schlossnagle @postwait 20
  • 21. They are not a goal, but they are not evil. It should be intentional. Some things behind an abstraction should intuitively be hidden. Intuition isn’t always right. When these leaks consistently benefit multiple stakeholders, it simply means the line of abstraction has been redrawn. Theo Schlossnagle @postwait 21
  • 22. They happen. They happen all the time. I am unapologetic about this. Violation carries cost burden. Assess it, accept it, and move on. Theo Schlossnagle @postwait 22