The document discusses lies that architects sometimes tell and truths they avoid. It provides examples of six common lies: 1) saying a system is real-time or has big data when it really has specific requirements, 2) claiming a microservices architecture exists when the goal is still to migrate, 3) saying hybrid/multi-cloud architectures don't exist when the architecture is just copy-pasted, 4) using "best of breed" when really using only one of everything, 5) claiming something can't be done at an organization due to its nature when other similar organizations succeeded, and 6) avoiding risk or change by safely interpreting things in a non-threatening way. The document advocates defining responsibilities clearly, embracing change, taking measured
Bringing Infosec Into The Devops Tribe: Q&A With Gene Kim and Pete CheslockThreat Stack
As we see more companies undertake cloud initiatives, deploying new projects into places like Amazon, Google and Azure, Infosec teams become new barriers to progress. We should instead be providing deep insight into services, users, and activities that these companies need, and provide this information to Devs, Ops and Infosec users.
Microservices: Why We Did It (and should you?) Outlyer
Mason will present a skeptical, humorous, and practical look at whether companies should consider microservices, and why/not. The story includes the reasons why Credit Karma did make the move, the approach we took, and shares some of our learnings so far.
The background on Creating a Big Data Platform in a private Cloud for HealthCare using OpenStack virtualization and Datastax with RightScale orchestration within a RESTful Service Oriented Architecture to deliver an ICD-10 Computer Assisted Coding solution.
Bringing Infosec Into The Devops Tribe: Q&A With Gene Kim and Pete CheslockThreat Stack
As we see more companies undertake cloud initiatives, deploying new projects into places like Amazon, Google and Azure, Infosec teams become new barriers to progress. We should instead be providing deep insight into services, users, and activities that these companies need, and provide this information to Devs, Ops and Infosec users.
Microservices: Why We Did It (and should you?) Outlyer
Mason will present a skeptical, humorous, and practical look at whether companies should consider microservices, and why/not. The story includes the reasons why Credit Karma did make the move, the approach we took, and shares some of our learnings so far.
The background on Creating a Big Data Platform in a private Cloud for HealthCare using OpenStack virtualization and Datastax with RightScale orchestration within a RESTful Service Oriented Architecture to deliver an ICD-10 Computer Assisted Coding solution.
These days, you can’t swing a dry erase marker without hitting someone talking about microservices. Developers are studying Eric Evans' prescient book, Domain-Driven Design. Teams are refactoring monolithic apps, looking for bounded contexts and defining a ubiquitous language. And although there have been countless articles, videos, and talks to help you convert to microservices, few have spent any appreciable time asking if a given application should be a microservice. In this talk, I‘ll show you a set of factors you can apply to help you decide if something deserves to be a microservice or not. We’ll also look at what we need to do to maintain a healthy micro(services)biome.
Cosa sono i microservizi? Perché li devo usare? Sono una moda? In alcuni dicono che siano una soluzione "standard", altri dicono che non si dovrebbero usare, altri ne negano l'esistenza... Ma chi sviluppa software e deve portare a casa un po' di software che funziona... cosa deve fare?
Proviamo a vedere e a capire da dove arrivano, cosa sono e quali caratteristiche hanno, in modo da fare in ogni contesto una scelta una consapevole.
Using Defensive Pessimism to Build Great Software at YMLAdam_Talcott
From The Silicon Valley iOS Developers' Meetup held at YML in Redwood City, CA on June 11, 2018.
Using “Defensive Pessimism” to Build Great Software at YML
Adam Talcott, Edward Cessna, and Ramsundar Shandilya, Y Media Labs
At YML, we take pride in creating great software as part of our mission to make digital products and experiences which have lasting impact. An important part of our process is anticipating the various scenarios our software may face and taking those scenarios into account from design through deployment and beyond. At YML, we refer to this concept as “defensive pessimism”.
After introducing YML, a customer experience design and technology agency with a vision of becoming our clients’ most valued partner, we will dive into defensive pessimism. We will discuss the ramifications for design and review examples of user experiences created with defensive pessimism in mind. We will also cover the resulting architecture considerations and the impact on reliability.
Slides for my keynote at incontrodevops.it, where I talked about distributed architectures, microservices, kubernetes and cloud native environments. All to get to the question: are microservices worth it?
Slides from my DevOpsExpo London talk "From oops to NoOps".
They tell you in these conferences that DevOps is not about tools, but about culture. And they are partially right. I am going to tell you that it’s not only about culture or tools but also abstractions.
It is a lot about how you see software and its value. About our mental model of what software is: how it runs, evolves, and interacts with the other facets of an enterprise.
We used to view software as code. As a state of code. Now we think about software as change, as a flow. A dynamic system where people, machines, and processes interact continuously.
At Platform.sh we spend a bunch of time asking ourselves not “How do you build?” - or even “How do you build consistently?” - but rather “What does it mean to consistently build in a world where change is good?” A world that lets you push security fixes into production as soon as they’re available because you don’t want to be an Equifax but you do want stability.
In this presentation, I will go over what we think software is and why having the right ideas about software will help you get your culture right and your tooling aligned, as well as gain in productivity, and general happiness and well-being.
devops, microservices, and platforms, oh my!Andrew Shafer
A story about a boy and his quest to build great software delivered at the Cloud Foundry Summit in Santa Clara May 2015. (https://www.youtube.com/watch?v=rX4mQHPWuUY) Walk through the history of my personal career, and the evolution of the industry highlighting themes like devops, microservices and platforms.
DevOps Patterns Distilled: Implementing The Needed Practices In Practical StepsCA Technologies
Learn from Gene Kim, one of the “DevOps Cookbook” authors, how to help accelerate DevOps adoption, increase the success of DevOps initiatives and lower the activation energy required for DevOps transformations to start and finish.
For more information on DevOps solutions from CA Technologies, please visit: http://bit.ly/1wbjjqX
#NoEstimates - Stop lying to yourself and your customers, and stop estimatinggerardbeckerleg
After his successful session last year on Agile Scrum, our resident Scrum White Robe Gerard Beckerleg is at it again, except this time he's taking on one of the most divisive topics in software development: Estimation.
In this video recorded at the Sydney SSW offices, Gerard Beckerleg takes a dive into the depths of this controversial topic and extracts the most interesting ideas and raises some very difficult questions about the big white elephant in the room that is Software Estimation.
After examining the pros and cons of estimation Gerard lays the blueprint for a better way to help you and your clients get what they are really looking for.
The Hardcore Stuff I Hack:
This talk is going to give a run through of some of the technical challenges paul and his team have overcome over the years - in as much hardcore detail as possible
These are my summarized notes from all the microservices session I attended at QCon 2015. These sessions had tons of learning around how to scale microservices and avoid common pitfalls
2017 Melbourne YOW! CTO Summit - Monolith to micro-services with CQRS & Event...Douglas English
These are the slides I presented at the Melbourne 2017 YOW! CTO Summit. The talk followed Culture Amp's steps and miss-steps on our journey to micro-services, and how we came to find CQRS and Event Sourcing fantastic tools.
I have spent some time working on a project, and built 8 micro services and 2 applications, and planned to carve out a few more. Deployment was carried out in a farm of 25 servers in production with a single click in less than 3 minutes.
This presentation is about the experiences with building a micro service based architecture - the good, the bad and the ugly.
- What are micro services?
- When/Why/How micro services?
- Why NOT micro services?
- Managing Continuous Integration and Continuous Delivery with micro services
- A few design principles that we followed and that worked for us
Exploring the problem of Microservices communication and how both Kafka and Service Mesh solutions address it. We then look at some approaches for combining both.
More Related Content
Similar to Lies Enterprise Architects Tell - Data Day Texas 2018 Keynote
These days, you can’t swing a dry erase marker without hitting someone talking about microservices. Developers are studying Eric Evans' prescient book, Domain-Driven Design. Teams are refactoring monolithic apps, looking for bounded contexts and defining a ubiquitous language. And although there have been countless articles, videos, and talks to help you convert to microservices, few have spent any appreciable time asking if a given application should be a microservice. In this talk, I‘ll show you a set of factors you can apply to help you decide if something deserves to be a microservice or not. We’ll also look at what we need to do to maintain a healthy micro(services)biome.
Cosa sono i microservizi? Perché li devo usare? Sono una moda? In alcuni dicono che siano una soluzione "standard", altri dicono che non si dovrebbero usare, altri ne negano l'esistenza... Ma chi sviluppa software e deve portare a casa un po' di software che funziona... cosa deve fare?
Proviamo a vedere e a capire da dove arrivano, cosa sono e quali caratteristiche hanno, in modo da fare in ogni contesto una scelta una consapevole.
Using Defensive Pessimism to Build Great Software at YMLAdam_Talcott
From The Silicon Valley iOS Developers' Meetup held at YML in Redwood City, CA on June 11, 2018.
Using “Defensive Pessimism” to Build Great Software at YML
Adam Talcott, Edward Cessna, and Ramsundar Shandilya, Y Media Labs
At YML, we take pride in creating great software as part of our mission to make digital products and experiences which have lasting impact. An important part of our process is anticipating the various scenarios our software may face and taking those scenarios into account from design through deployment and beyond. At YML, we refer to this concept as “defensive pessimism”.
After introducing YML, a customer experience design and technology agency with a vision of becoming our clients’ most valued partner, we will dive into defensive pessimism. We will discuss the ramifications for design and review examples of user experiences created with defensive pessimism in mind. We will also cover the resulting architecture considerations and the impact on reliability.
Slides for my keynote at incontrodevops.it, where I talked about distributed architectures, microservices, kubernetes and cloud native environments. All to get to the question: are microservices worth it?
Slides from my DevOpsExpo London talk "From oops to NoOps".
They tell you in these conferences that DevOps is not about tools, but about culture. And they are partially right. I am going to tell you that it’s not only about culture or tools but also abstractions.
It is a lot about how you see software and its value. About our mental model of what software is: how it runs, evolves, and interacts with the other facets of an enterprise.
We used to view software as code. As a state of code. Now we think about software as change, as a flow. A dynamic system where people, machines, and processes interact continuously.
At Platform.sh we spend a bunch of time asking ourselves not “How do you build?” - or even “How do you build consistently?” - but rather “What does it mean to consistently build in a world where change is good?” A world that lets you push security fixes into production as soon as they’re available because you don’t want to be an Equifax but you do want stability.
In this presentation, I will go over what we think software is and why having the right ideas about software will help you get your culture right and your tooling aligned, as well as gain in productivity, and general happiness and well-being.
devops, microservices, and platforms, oh my!Andrew Shafer
A story about a boy and his quest to build great software delivered at the Cloud Foundry Summit in Santa Clara May 2015. (https://www.youtube.com/watch?v=rX4mQHPWuUY) Walk through the history of my personal career, and the evolution of the industry highlighting themes like devops, microservices and platforms.
DevOps Patterns Distilled: Implementing The Needed Practices In Practical StepsCA Technologies
Learn from Gene Kim, one of the “DevOps Cookbook” authors, how to help accelerate DevOps adoption, increase the success of DevOps initiatives and lower the activation energy required for DevOps transformations to start and finish.
For more information on DevOps solutions from CA Technologies, please visit: http://bit.ly/1wbjjqX
#NoEstimates - Stop lying to yourself and your customers, and stop estimatinggerardbeckerleg
After his successful session last year on Agile Scrum, our resident Scrum White Robe Gerard Beckerleg is at it again, except this time he's taking on one of the most divisive topics in software development: Estimation.
In this video recorded at the Sydney SSW offices, Gerard Beckerleg takes a dive into the depths of this controversial topic and extracts the most interesting ideas and raises some very difficult questions about the big white elephant in the room that is Software Estimation.
After examining the pros and cons of estimation Gerard lays the blueprint for a better way to help you and your clients get what they are really looking for.
The Hardcore Stuff I Hack:
This talk is going to give a run through of some of the technical challenges paul and his team have overcome over the years - in as much hardcore detail as possible
These are my summarized notes from all the microservices session I attended at QCon 2015. These sessions had tons of learning around how to scale microservices and avoid common pitfalls
2017 Melbourne YOW! CTO Summit - Monolith to micro-services with CQRS & Event...Douglas English
These are the slides I presented at the Melbourne 2017 YOW! CTO Summit. The talk followed Culture Amp's steps and miss-steps on our journey to micro-services, and how we came to find CQRS and Event Sourcing fantastic tools.
I have spent some time working on a project, and built 8 micro services and 2 applications, and planned to carve out a few more. Deployment was carried out in a farm of 25 servers in production with a single click in less than 3 minutes.
This presentation is about the experiences with building a micro service based architecture - the good, the bad and the ugly.
- What are micro services?
- When/Why/How micro services?
- Why NOT micro services?
- Managing Continuous Integration and Continuous Delivery with micro services
- A few design principles that we followed and that worked for us
Exploring the problem of Microservices communication and how both Kafka and Service Mesh solutions address it. We then look at some approaches for combining both.
Presentation for Papers We Love at QCON NYC 17. I didn't write the paper, good people at Facebook did. But I sure enjoyed reading it and presenting it.
Streaming Data Integration - For Women in Big Data MeetupGwen (Chen) Shapira
A stream processing platform is not an island unto itself; it must be connected to all of your existing data systems, applications, and sources. In this talk, we will provide different options for integrating systems and applications with Apache Kafka, with a focus on the Kafka Connect framework and the ecosystem of Kafka connectors. We will discuss the intended use cases for Kafka Connect and share our experience and best practices for building large-scale data pipelines using Apache Kafka.
Modern data systems don't just process massive amounts of data, they need to do it very fast. Using fraud detection as a convenient example, this session will include best practices on how to build real-time data processing applications using Apache Kafka. We'll explain how Kafka makes real-time processing almost trivial, discuss the pros and cons of the famous lambda architecture, help you choose a stream processing framework and even talk about deployment options.
This session will go into best practices and detail on how to architect a near real-time application on Hadoop using an end-to-end fraud detection case study as an example. It will discuss various options available for ingest, schema design, processing frameworks, storage handlers and others, available for architecting this fraud detection application and walk through each of the architectural decisions among those choices.
Many architectures include both real-time and batch processing components. This often results in two separate pipelines performing similar tasks, which can be challenging to maintain and operate. We'll show how a single, well designed ingest pipeline can be used for both real-time and batch processing, making the desired architecture feasible for scalable production use cases.
"Analyzing Twitter Data with Hadoop - Live Demo", presented at Oracle Open World 2014. The repository for the slides is in https://github.com/cloudera/cdh-twitter-example
E-commerce Application Development Company.pdfHornet Dynamics
Your business can reach new heights with our assistance as we design solutions that are specifically appropriate for your goals and vision. Our eCommerce application solutions can digitally coordinate all retail operations processes to meet the demands of the marketplace while maintaining business continuity.
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...Crescat
Crescat is industry-trusted event management software, built by event professionals for event professionals. Founded in 2017, we have three key products tailored for the live event industry.
Crescat Event for concert promoters and event agencies. Crescat Venue for music venues, conference centers, wedding venues, concert halls and more. And Crescat Festival for festivals, conferences and complex events.
With a wide range of popular features such as event scheduling, shift management, volunteer and crew coordination, artist booking and much more, Crescat is designed for customisation and ease-of-use.
Over 125,000 events have been planned in Crescat and with hundreds of customers of all shapes and sizes, from boutique event agencies through to international concert promoters, Crescat is rigged for success. What's more, we highly value feedback from our users and we are constantly improving our software with updates, new features and improvements.
If you plan events, run a venue or produce festivals and you're looking for ways to make your life easier, then we have a solution for you. Try our software for free or schedule a no-obligation demo with one of our product specialists today at crescat.io
Transform Your Communication with Cloud-Based IVR SolutionsTheSMSPoint
Discover the power of Cloud-Based IVR Solutions to streamline communication processes. Embrace scalability and cost-efficiency while enhancing customer experiences with features like automated call routing and voice recognition. Accessible from anywhere, these solutions integrate seamlessly with existing systems, providing real-time analytics for continuous improvement. Revolutionize your communication strategy today with Cloud-Based IVR Solutions. Learn more at: https://thesmspoint.com/channel/cloud-telephony
AI Pilot Review: The World’s First Virtual Assistant Marketing SuiteGoogle
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
👉👉 Click Here To Get More Info 👇👇
https://sumonreview.com/ai-pilot-review/
AI Pilot Review: Key Features
✅Deploy AI expert bots in Any Niche With Just A Click
✅With one keyword, generate complete funnels, websites, landing pages, and more.
✅More than 85 AI features are included in the AI pilot.
✅No setup or configuration; use your voice (like Siri) to do whatever you want.
✅You Can Use AI Pilot To Create your version of AI Pilot And Charge People For It…
✅ZERO Manual Work With AI Pilot. Never write, Design, Or Code Again.
✅ZERO Limits On Features Or Usages
✅Use Our AI-powered Traffic To Get Hundreds Of Customers
✅No Complicated Setup: Get Up And Running In 2 Minutes
✅99.99% Up-Time Guaranteed
✅30 Days Money-Back Guarantee
✅ZERO Upfront Cost
See My Other Reviews Article:
(1) TubeTrivia AI Review: https://sumonreview.com/tubetrivia-ai-review
(2) SocioWave Review: https://sumonreview.com/sociowave-review
(3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review
(4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review
Software Engineering, Software Consulting, Tech Lead, Spring Boot, Spring Cloud, Spring Core, Spring JDBC, Spring Transaction, Spring MVC, OpenShift Cloud Platform, Kafka, REST, SOAP, LLD & HLD.
Do you want Software for your Business? Visit Deuglo
Deuglo has top Software Developers in India. They are experts in software development and help design and create custom Software solutions.
Deuglo follows seven steps methods for delivering their services to their customers. They called it the Software development life cycle process (SDLC).
Requirement — Collecting the Requirements is the first Phase in the SSLC process.
Feasibility Study — after completing the requirement process they move to the design phase.
Design — in this phase, they start designing the software.
Coding — when designing is completed, the developers start coding for the software.
Testing — in this phase when the coding of the software is done the testing team will start testing.
Installation — after completion of testing, the application opens to the live server and launches!
Maintenance — after completing the software development, customers start using the software.
Artificia Intellicence and XPath Extension FunctionsOctavian Nadolu
The purpose of this presentation is to provide an overview of how you can use AI from XSLT, XQuery, Schematron, or XML Refactoring operations, the potential benefits of using AI, and some of the challenges we face.
OpenMetadata Community Meeting - 5th June 2024OpenMetadata
The OpenMetadata Community Meeting was held on June 5th, 2024. In this meeting, we discussed about the data quality capabilities that are integrated with the Incident Manager, providing a complete solution to handle your data observability needs. Watch the end-to-end demo of the data quality features.
* How to run your own data quality framework
* What is the performance impact of running data quality frameworks
* How to run the test cases in your own ETL pipelines
* How the Incident Manager is integrated
* Get notified with alerts when test cases fail
Watch the meeting recording here - https://www.youtube.com/watch?v=UbNOje0kf6E
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI AppGoogle
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
👉👉 Click Here To Get More Info 👇👇
https://sumonreview.com/ai-fusion-buddy-review
AI Fusion Buddy Review: Key Features
✅Create Stunning AI App Suite Fully Powered By Google's Latest AI technology, Gemini
✅Use Gemini to Build high-converting Converting Sales Video Scripts, ad copies, Trending Articles, blogs, etc.100% unique!
✅Create Ultra-HD graphics with a single keyword or phrase that commands 10x eyeballs!
✅Fully automated AI articles bulk generation!
✅Auto-post or schedule stunning AI content across all your accounts at once—WordPress, Facebook, LinkedIn, Blogger, and more.
✅With one keyword or URL, generate complete websites, landing pages, and more…
✅Automatically create & sell AI content, graphics, websites, landing pages, & all that gets you paid non-stop 24*7.
✅Pre-built High-Converting 100+ website Templates and 2000+ graphic templates logos, banners, and thumbnail images in Trending Niches.
✅Say goodbye to wasting time logging into multiple Chat GPT & AI Apps once & for all!
✅Save over $5000 per year and kick out dependency on third parties completely!
✅Brand New App: Not available anywhere else!
✅ Beginner-friendly!
✅ZERO upfront cost or any extra expenses
✅Risk-Free: 30-Day Money-Back Guarantee!
✅Commercial License included!
See My Other Reviews Article:
(1) AI Genie Review: https://sumonreview.com/ai-genie-review
(2) SocioWave Review: https://sumonreview.com/sociowave-review
(3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review
(4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review
#AIFusionBuddyReview,
#AIFusionBuddyFeatures,
#AIFusionBuddyPricing,
#AIFusionBuddyProsandCons,
#AIFusionBuddyTutorial,
#AIFusionBuddyUserExperience
#AIFusionBuddyforBeginners,
#AIFusionBuddyBenefits,
#AIFusionBuddyComparison,
#AIFusionBuddyInstallation,
#AIFusionBuddyRefundPolicy,
#AIFusionBuddyDemo,
#AIFusionBuddyMaintenanceFees,
#AIFusionBuddyNewbieFriendly,
#WhatIsAIFusionBuddy?,
#HowDoesAIFusionBuddyWorks
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
Zoom is a comprehensive platform designed to connect individuals and teams efficiently. With its user-friendly interface and powerful features, Zoom has become a go-to solution for virtual communication and collaboration. It offers a range of tools, including virtual meetings, team chat, VoIP phone systems, online whiteboards, and AI companions, to streamline workflows and enhance productivity.
Need for Speed: Removing speed bumps from your Symfony projects ⚡️Łukasz Chruściel
No one wants their application to drag like a car stuck in the slow lane! Yet it’s all too common to encounter bumpy, pothole-filled solutions that slow the speed of any application. Symfony apps are not an exception.
In this talk, I will take you for a spin around the performance racetrack. We’ll explore common pitfalls - those hidden potholes on your application that can cause unexpected slowdowns. Learn how to spot these performance bumps early, and more importantly, how to navigate around them to keep your application running at top speed.
We will focus in particular on tuning your engine at the application level, making the right adjustments to ensure that your system responds like a well-oiled, high-performance race car.
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptxrickgrimesss22
Discover the essential features to incorporate in your Winzo clone app to boost business growth, enhance user engagement, and drive revenue. Learn how to create a compelling gaming experience that stands out in the competitive market.
10. Actual Requirements
• Throughput:
X MB/sec
• Latency:
X ms from A to B
• Storage: TB
• Size of working set: GB
• Availability
• Durability
• Data access patterns
11. Soft Requirements
• Tolerance for failure
• Tolerance for “cutting edge”
• Operational maturity
• Size of engineering team
• Culture
12. Buzzwords are Suspicious
• Cloud Native == Resilient & Elastic
• Serverless == Less ops & Pay per use
• Service mesh == Flexible infra & Central control
13. What I say:
What They Hear:
What I mean:
“Oh, 15 minute produce-to-consume latency
and 1 MB/sec latency is easy!”
“You are working on a crappy system”
“We can build something cheap, stable and
easy to maintain. YAY!”
19. Have an escape plan
Once the data is in Kafka, it’s so much more
portable… Portable data means you don’t need to
worry about vendor lock-in.”
— Chris Ricommini, WePay
25. Distributed Monolith?
• How many services do you need to modify
when making “a tiny change”?
• One of your microservices crashed.
How many microservices do you need to restart?
• Can you deploy a new version of any microservice
at any time and at any order?
39. Bad Architect Good Architect
Uses buzzwords to justify
decisions
Uses data to drive decisions
Forces rapid adoption of trendy
choices
Looks at goals, costs and
benefits
Copy-pastes from vendors Does own research and POC
Protects “play with tech” turf
Help engineers adopt
technologies
Says “we’ve always done it this
way”
Is a change agent
Avoids risk / ignores risk.
Takes risks that make a
difference.
40. — Rabbi Hanina bar Hama
"I have learned much from my teachers, more from
my colleagues, and the most from my students"
Editor's Notes
Hey, I’m Gwen Shapira - I’m a committer on Apache Kafka project and I work for Confluent, which is the company building a streaming event platform out of Kafka. I left my title out, because if I included it, you wouldn’t believe a single word I’m about to say.Before I start, I want to find out what percentage of the audience I’ll accidentally insult this morning. So… how many of you have the title of “enterprise architect”?Those of you working as enterprise architects probably know what you do. So it may surprise you to discover that to a large extent the rest of the world doesn’t.
Hey, I’m Gwen Shapira - I’m a committer on Apache Kafka project and I work for Confluent, which is the company building a streaming event platform out of Kafka. I left my title out, because if I included it, you wouldn’t believe a single word I’m about to say.
Apache Kafka is often used for what some people call “fast data” systems. Which means that the first thing I often hear when I meet a new customer is…
This is not a lie. Regardless of what you may think, we have real work to do.
Architects are there to force software engineers across the company to collaborate.
Software engineers want whatever they want.
And their management is incentivized to let them do it, because hiring engineers is hard and keeping them productive is harder.
You want to use Go? fantastic! Want to use Airflow? Go for it. Love Spark - you do you! Love REST APIs? We’ll do REST APIs. You prefer GRPC? Lets do GRPC!
But we all need to work together, and this is where it gets challenging. So enterprise architects are supposed to pick “standards” and get the entire company to use them. Standard languages, frameworks, tools, methods, architectures, design styles, governance, etc.
Engineers have tons of leverage. So you can’t force them to collaborate. You need to convince them to do things your way. Which is why no one knows what architects do: When we do a great job, it looks like we do nothing at all.
Convincing smart and opinionated people ain’t easy. You need to build trust, you need to have good arguments, you may need to have proof which means that you need to build convincing proof of concepts, you need to do research, you need to educate, you need to win debates.
This is difficult. So, sometimes we take shortcuts.
Not always intentionally. Usually the first victim is ourselves. We want something to be true, so we convince our selves it is true. And then we started repeating it to other people.
Our lies are a shortcut. A way to convince ourselves and everyone else that we made a good decision, instead of doing the hard work of research, proof and consensus building.
Lets look at some examples of lies that I’ve heard again and again and again, in 10 years of advising enterprise architects, in companies large and small, on how to build their data infrastructure. Lets see look at the lies and the truth behind them, because knowing the truth will free you to do a better job.
Since I work with Apache Kafka, which is loosely connected to something called “Fast data” or “speed layer”, I get this one a lot.
If you naively imagine they are building something like…
you are wrong.
I learned to ask: Which part of the system need to be how fast?
Because most of the time it is “15 minutes until the dashboard reflects changes to the DB”.
Except when it is “15 ms until mobile app shows that the action was accepted”.
Or even “15 microseconds until we trigger a trade”.
That lie has an older cousin. I’ve heard this non-stop between 2012 to 2016.
It was usually a reason for adopting an unusual, poorly understood and poorly tested data storage system. “Why are you re-building your system to use System-Z?”“Well, we have big data, you know”.Big data in that context was around 45GB. Not that it matters.
Buzzwords like “Real time” and “big data” are nearly meaningless. All those buzzwords are a shortcut for specific requirements. The things you actually need your system to do.
We take the shortcut because collecting requirements is difficult, and we can’t be really sure we nailed them. To make things worse, storage isn’t always as agile as other components - if we make a mistake, migration is difficult.
But: Picking a system based on a buzzword is worse - because the probability it is an actual fit is pretty low.
Talk to me in requirements. What’s your throughput? What is your latency tolerance. How long do you need to store your data? What’s the size of the working set? What are your data access patterns? What’s your reporting requirements?
I’ve seen an email from an engineer who decided to adopt a relatively new technology, and use one of the most cutting edge features in it. The email basically said “I’ve now lost all my credibility and I need to switch tracks or I’ll lose my job too”. This is heartbreaking to me.
Like lies, they are used as shortcuts. And often lose all meaning. You need to learn to translate them.
Always look for the costs and benefits.
I learned that I need to be very gentle when I give my opinion about those requirements. I definitely can’t laugh out loud. Architects can be sensitive. I say “Well, 15 minutes is trivial” and they hear “You are a crappy architect working on a crappy system”. What I mean is “Yay! We can build a system that is cheap, easy to maintain and low risk!”
Lots of architects act like they are allergic to simplicity. Simple is good.
Collect your requirements and if they lead you to a simple system - celebrate!
This lie has a close sibling. Now, you’d think that if “we have big data” is a lie, then “we don’t have big data” must be the truth. But you’d be wrong. Just like its sibling, this is a way of explaining a choice of data storage system made without proper consideration of requirements, capabilities, performance tests, any tests at all or even any data at all. An example would be:“Why are you using Oracle to search text documents?” “Well, we don’t need Elastic, right? We don’t have big data or anything”.
If you are using Oracle to do something that Elastic is really good at, there’s high probability that you are paying few million dollars too many for your text search. Same for using Oracle as a work queue.
On a side note, it is really funny how I show large prospects how they can use Kafka for their work queue and therefore save few millions of dollars and suddenly 20,000 dollar becomes “why should I pay so much for open source?”. I’ll never understand that.
Anyway, what’s the truth here?
“Big Data” and “Not Big Data” are nearly meaningless. Talk to me in requirements. What’s your throughput? What is your latency tolerance. How long do you need to store your data? What’s the size of the working set? What are your data access patterns? What’s your reporting requirements?
Lets talk about non-functional requirements too: What is your appetite for risk? Do you prefer a well-understood and stable system or a presentation with “lessons learned implementing System-Z” at a conference? What’s the outcome of failure like in your organization? How many other databases do you have? What is your operational culture like? Are you comfortable contributing to open source?
I’ve seen an email from an engineer who decided to adopt a relatively new technology, and use one of the most cutting edge features in it. The email basically said “I’ve now lost all my credibility and I need to switch tracks or I’ll lose my job too”. This is heartbreaking to me.
Given all the functional and non-functional factors, you should be able to make a strong, well-justified choice of a database. Without a single buzzword.
This is a particularly annoying lie, because it is so easy to disprove. I have maybe 20 counter-examples. From a very diverse set of companies. Early on, I only ever heard this from “Developer Advocates” working for a certain cloud provider. But now I also hear it from customers.
By now, if an architect tells me: “Mult-cloud architectures are not a thing”, you of know that they are very very attached to a specific cloud provider. You can tell for sure, if the next thing they say is:
You really can’t argue with someone who didn’t really think about the architecture in the first place.
But! If you get there first, you can get them to copy-paste YOUR architecture!
Seriously though. There are lots of benefits to using lots of cloud managed services. You just need to have an escape hatch. A plan on what to do if things don’t turn out as expected. And this is true for most architecture decisions.
https://riccomini.name/kafka-escape-hatch
I have to admit that this is a lie that I fell for many many times before I discovered the truth.
Actually, there are few versions of the truth.
Sometimes it isn’t a lie as much as it is an aspiration. They want to do microservices. They are moving in that direction. They are just not there yet. This is great.
It includes an implicit admission in what is really a grand universal truth: “Software architectures, like anything else, only exist in a state of change”. Like buddhist mandalas, they are great works of art, built in sand, to be swept away. In few month there will be somewhere else for you to migrate to. Maybe serverless.
We have 50,000 different microservices that we can’t keep track of. Changing a simple config requires weeks of stitching together a workflow across at least 6 different services. Troubleshooting is impossible.
How do you know if you have a distributed monolith? You try to do one of the things that microservices are supposed to make easy. Was it easy?
The worst offender is leak of responsibilities. Microservices are supposed to contain entire context. But you see cases where the customer profile service needs to call the insurance quotes service whenever someone changes address. Some of the logic around quotes leaked from the quotes service into the customer profile service, where it does not belong.
Now, if we had an event driven architecture, the customer profile service would publish changes in profile and the insurance quote service can decide how to use them.
Now, don’t copy-paste my architecture… but you may want to look into this.
As I said earlier, as a software architect… you have one job. Make sure your organization is standardized on a set of technologies and design principles. “Best of breed” can be a nice way to say “I’m not doing my job”.
There is a cost to having a technology. What makes additional technology costly? learning, deployment, monitoring, finding all the “unexpected behaviors”.
If a technology does not serve a purpose or “spark joy”, say “thank you” and figure out a plan to get rid of it.
Carefully assess the technical and cultural fit of new technology vs the costs of adding another technology to the stack.
Remember that the cost of adding new software is much higher if there are lots of unique integration points. Each integration adds its own risk, so definitely try to minimize those.
We used to call it “integration tax” - it ain’t bad at first, but system #20 has to integrate with the 19 older things. Unless you take steps to control it with something like Kafka as central integration point.
There is a tricky anti pattern here though…
Sometimes architects think that just because their job is to get the organization to standardize, they are the only ones who can use new technologies. This is terrible and if you work in such a place, leave.
As a developer, it robs you of your growth and joy. As an architect, it robs you of your chance to make an impact… because developers are unlikely to actually do what you tell them.
I learned this from my customers. The most successful enterprise architects know how to detect great bets and work with the engineers and managers to help the rest of the organization adopt the new methods. You do it with workgroups, hackathons, office-hours, tech-talks, etc, etc.
It is an immensely satisfying way to work - you get to see many engineers grow and your decisions take root and take the organization to the next level.
Lyft had a great talk at Qcon NYC couple of years back on how they migrated from REST to GRPC, and the main challenges were getting company-wide participation – and they wrote many tools, including a legendary proxy to help gradually increase comfort level with the new technology.
And please don’t tell me “Oh, this is Lyft. We can never do it here” because…
From “We know almost nothing and call a vendor for everything” through “We know lots, do almost everything alone and also have a deep expert on retainer” all the way to “We employ 2+ committers on the project”.
Remember that being an architect is all about changing how you think and approach problems, changing how others work and changing entire organizations.
This is your job. Assess capacity for risk. Small organizations can feel bolder. Large organizations sometimes prefer to play it safe, and other times want to use their resources to do very bold things. Assess what matters to the business. Find the best solutions from large teams of engineers - and with sure hand, steer the organization in the right direction at the right speed.
This can be a very benign lie. Agile manifesto has very reasonable ideas like “value people” and “value working software”. Why would anyone lie about something so reasonable?
There is a wonderful document called “the half-arsed agile manifesto”.https://www.halfarsedagilemanifesto.org/
I learned a lot preparing this presentation. Thank you for the opportunity to learn and share.