There's No Place Like Production

•

0 likes•51 views

There’s a reason “i test in prod” isn’t a cheeky take but a lived reality. And that reason is there is no place like production. Not local dev or staging or other environment. This became clear when I deployed a tiny config change that passed all checks, reviews and pre-production environments that triggered a SEV-1. Examining each step of the journey from PR to production I uncovered the snafu that had occurred (unsurprisingly it relates to overwriting key blocks on nested YAML files). I’ll share how difficult it was to reconstruct the chain of events in the system compared to the ideal case of a highly observable system and how to share your own incident learnings since we all test in prod!

Technology

@paigerduty
There’s No
Place Like
Production
Paige Cruz, Chronosphere

@paigerduty
pandemic
~80% dept attrition
K8s migration
CI/CD migration
personal life stuff

@paigerduty
A cannabis technology
platform providing
integrated solutions
for consumers and
businesses

@paigerduty
ACCEPTANCE STAGING PROD
Let’s Ship It

@paigerduty
Let’s Ship It
ACCEPTANCE STAGING PROD

@paigerduty
Hey…I think its your Traefik change

@paigerduty
Black
Is the color of ebony
and of outer space. It
has been the symbolic
color of elegance,
solemnity and
authority.

@paigerduty
Pssst. Hey I think
was really cool how
you handled
yourself during the
incident!

@paigerduty
To Company
From Paige
*no returns

@paigerduty
36
Get in betch, we’re learning from this incident

@paigerduty
Spot the Differences: HARD MODE

@paigerduty
Incident
Retro
Chapter
BE
Chapter
FE

@paigerduty
Incident
Retro
Chapter
BE
Chapter
FE
SRE
Study
Time

@paigerduty
Thanks!
You can ﬁnd me at:
➜ paigerduty
@chronosphere.io
➜ @hachyderm.io
➜ Template by SlidesCarnival
➜ Images by Unsplash, Pexels,
Pixabay

More from Paige Cruz

For years tech companies have chased the fabled “single pane of glass” , the one observability tool to understand your system from north to south and east to west. Leafing through promo materials promising instant insights and seamless turnkey integrations you’d think increasing system observability is as easy as assembling a Lego set. In my experience chasing the “single pane of glass” translates to “pain in the ass”. Survey data supports this revealing the majority of engineers cite tool sprawl as a minor or non-existent problem despite relying on several tools. As alluring as the siren call of “single pane of glass” is, let's be practical and examine how to best observe systems across a myriad of tooling. From the telemetry buffet of metrics, events, traces and logs learn when to reach for which type and ways to bridge the gaps with links and enhance with context to free yourself from the fool’s errand of a “single pane of glass”.

Pushing Observability Uphill - The Single “Pain” of Glass

Paige Cruz

Curious about containers? There’s a new generation of containers on the scene, Podman! Supporting secure, rootless containers for Kubernetes microservices, it was designed and built with the cloud in mind. Benefitting from the lessons learned out in the open from Docker, this next generation of containers will quickly become a trusted daily driver in your dev workflow. Covering what you need to know as an end-user from the UI to the backend, sharing a real world use case leveraging Podman for open source observability workshops https://o11y-workshops.gitlab.io. Paige will share how Podman and the adorable seal mascots Caitlín, Maighréad and Róisín have transformed her local development!

Power Up with Podman

Paige Cruz

* Who/What/Where/When/Why of openTelemetry * Demystifying observability terms (telemetry, instrumentation, cardinality, percentile, observability) * Nuts and Bolts of Instrumentation * Tour of the Sample App * Automatic instrumentation with openTelemetry * Explore OOTB response time metric on charts * Manually add a label to the metric and review new facets to query by * Explore an OOTB trace * Manually instrument and add a span and review new level of visibility

Intro to Instrumentation

Paige Cruz

Are you collecting just about every metric under the sun and the kitchen sink too? Understanding the cost of collecting metrics and the usefulness of those metrics is the only way to scale in a cloud native world. You can’t get away with just collecting everything as you grow. How can you make decisions about what to collect, what to drop, what to aggregate while still being able to alert, triage, remediate? Gain immediate insights into high cost data (DPPS), when to drop time series data, and how to determine when the value of that data is at its lowest.

From Cardinal(ity) Sins to Cost-Efficient Metrics Aggregation

Paige Cruz

99.9% of Your Traces are Trash

Paige Cruz

Today we are caught between the 2nd and 3rd waves of system observability - the proprietary approaches of the past are fading fast in favor of open source instrumentation. Pioneering projects like OpenTelemetry and Prometheus are leading the way providing a path for competitors to cooperate for the benefit of the community. Let's look back at the 3 waves of observability to understand how we got here, the vision for the future and ways you can join us for the journey to liberate telemetry for all!

3rd Wave Observability: Open or Bust

Paige Cruz

More from Paige Cruz (6)

Pushing Observability Uphill - The Single “Pain” of Glass

Power Up with Podman

Intro to Instrumentation

From Cardinal(ity) Sins to Cost-Efficient Metrics Aggregation

99.9% of Your Traces are Trash

3rd Wave Observability: Open or Bust

Recently uploaded

GenAI Risks & Security Meetup 01052024.pdf

lior mazor

What is a good lead in your organisation? Which leads are priority? What happens to leads? When sales and marketing give different answers to these questions, or perhaps aren't sure of the answers at all, frustrations build and opportunities are left on the table. Join us for an illuminating session with Cian McLoughlin, HubSpot Principal Customer Success Manager, as we look at that crucial piece of the customer journey in which leads are transferred from marketing to sales.

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

HampshireHUG

Tech Trends Report 2024 Future Today Institute.pdf

hans926745

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

Scaling API-first – The story of a global engineering organization

Radu Cotescu

BooK Now Call us at +918448380779 to hire a gorgeous and seductive call girl for sex. Take a Delhi Escort Service. The help of our escort agency is mostly meant for men who want sexual Indian Escorts In Delhi NCR. It should be noted that any impersonator will get 100 attention from our Young Girls Escorts in Delhi. They will assume the position of reliable allies. VIP Call Girl With Original Photos Book Tonight +918448380779 Our Cheap Price 1 Hour not available 2 Hours 5000 Full Night 8000 TAG: Call Girls in Delhi, Noida, Gurgaon, Ghaziabad, Connaught Place, Greater Kailash Delhi, Lajpat Nagar Delhi, Mayur Vihar Delhi, Chanakyapuri Delhi, New Friends Colony Delhi, Majnu Ka Tilla, Karol Bagh, Malviya Nagar, Saket, Khan Market, Noida Sector 18, Noida Sector 76, Noida Sector 51, Gurgaon Mg Road, Iffco Chowk Gurgaon, Rajiv Chowk Gurgaon All Delhi Ncr Free Home Deliver

08448380779 Call Girls In Friends Colony Women Seeking Men

Delhi Call girls

[2024]Digital Global Overview Report 2024 Meltwater.pdf

hans926745

Imagine a world where information flows as swiftly as thought itself, making decision-making as fluid as the data driving it. Every moment is critical, and the right tools can significantly boost your organization’s performance. The power of real-time data automation through FME can turn this vision into reality. Aimed at professionals eager to leverage real-time data for enhanced decision-making and efficiency, this webinar will cover the essentials of real-time data and its significance. We’ll explore: FME’s role in real-time event processing, from data intake and analysis to transformation and reporting An overview of leveraging streams vs. automations FME’s impact across various industries highlighted by real-life case studies Live demonstrations on setting up FME workflows for real-time data Practical advice on getting started, best practices, and tips for effective implementation Join us to enhance your skills in real-time data automation with FME, and take your operational capabilities to the next level.

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Safe Software

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

wesley chun

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

The presentation explores the development and application of artificial intelligence (AI) from its inception to its current status in the modern world. The term "artificial intelligence" was first coined by John McCarthy in 1956 to describe efforts to develop computer programs capable of performing tasks that typically require human intelligence. This concept was first introduced at a conference held at Dartmouth College, where programs demonstrated capabilities such as playing chess, proving theorems, and interpreting texts. In the early stages, Alan Turing contributed to the field by defining intelligence as the ability of a being to respond to certain questions intelligently, proposing what is now known as the Turing Test to evaluate the presence of intelligent behavior in machines. As the decades progressed, AI evolved significantly. The 1980s focused on machine learning, teaching computers to learn from data, leading to the development of models that could improve their performance based on their experiences. The 1990s and 2000s saw further advances in algorithms and computational power, which allowed for more sophisticated data analysis techniques, including data mining. By the 2010s, the proliferation of big data and the refinement of deep learning techniques enabled AI to become mainstream. Notable milestones included the success of Google's AlphaGo and advancements in autonomous vehicles by companies like Tesla and Waymo. A major theme of the presentation is the application of generative AI, which has been used for tasks such as natural language text generation, translation, and question answering. Generative AI uses large datasets to train models that can then produce new, coherent pieces of text or other media. The presentation also discusses the ethical implications and the need for regulation in AI, highlighting issues such as privacy, bias, and the potential for misuse. These concerns have prompted calls for comprehensive regulations to ensure the safe and equitable use of AI technologies. Artificial intelligence has also played a significant role in healthcare, particularly highlighted during the COVID-19 pandemic, where it was used in drug discovery, vaccine development, and analyzing the spread of the virus. The capabilities of AI in healthcare are vast, ranging from medical diagnostics to personalized medicine, demonstrating the technology's potential to revolutionize fields beyond just technical or consumer applications. In conclusion, AI continues to be a rapidly evolving field with significant implications for various aspects of society. The development from theoretical concepts to real-world applications illustrates both the potential benefits and the challenges that come with integrating advanced technologies into everyday life. The ongoing discussion about AI ethics and regulation underscores the importance of managing these technologies responsibly to maximize their their benefits while minimizing potential harms.

Artificial Intelligence: Facts and Myths

Joaquim Jorge

08448380779 Call Girls In Greater Kailash - I Women Seeking Men

Delhi Call girls

Heather Hedden, Senior Consultant at Enterprise Knowledge, presented “The Role of Taxonomy and Ontology in Semantic Layers” at a webinar hosted by Progress Semaphore on April 16, 2024. Taxonomies at their core enable effective tagging and retrieval of content, and combined with ontologies they extend to the management and understanding of related data. There are even greater benefits of taxonomies and ontologies to enhance your enterprise information architecture when applying them to a semantic layer. A survey by DBP-Institute found that enterprises using a semantic layer see their business outcomes improve by four times, while reducing their data and analytics costs. Extending taxonomies to a semantic layer can be a game-changing solution, allowing you to connect information silos, alleviate knowledge gaps, and derive new insights. Hedden, who specializes in taxonomy design and implementation, presented how the value of taxonomies shouldn’t reside in silos but be integrated with ontologies into a semantic layer. Learn about: - The essence and purpose of taxonomies and ontologies in information and knowledge management; - Advantages of semantic layers leveraging organizational taxonomies; and - Components and approaches to creating a semantic layer, including the integration of taxonomies and ontologies

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

Enterprise Knowledge

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Rafal Los

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Neo4j

In this session, we will delve into strategic approaches for optimizing knowledge management within Microsoft 365, amidst the evolving landscape of Copilot. From leveraging automatic metadata classification and permission governance with SharePoint Premium, to unlocking Viva Engage for the cultivation of knowledge and communities, you will gain actionable insights to bolster your organization's knowledge-sharing initiatives. In this session, we will also explore how to facilitate solutions to enable your employees to find answers and expertise within Microsoft 365. You will leave equipped with practical techniques and a deeper understanding of how there is more to effective knowledge management than just enabling Copilot, but building actual solutions to prepare the knowledge that Copilot and your employees can use.

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Drew Madelung

Finology Group – Insurtech Innovation Award 2024

The Digital Insurer

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

Data Cloud, More than a CDP by Matt Robison

Anna Loughnan Colquhoun

Recently uploaded (20)

GenAI Risks & Security Meetup 01052024.pdf

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

Tech Trends Report 2024 Future Today Institute.pdf

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Scaling API-first – The story of a global engineering organization

08448380779 Call Girls In Friends Colony Women Seeking Men

[2024]Digital Global Overview Report 2024 Meltwater.pdf

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Strategies for Landing an Oracle DBA Job as a Fresher

Powerful Google developer tools for immediate impact! (2023-24 C)

How to Troubleshoot Apps for the Modern Connected Worker

Artificial Intelligence: Facts and Myths

08448380779 Call Girls In Greater Kailash - I Women Seeking Men

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Finology Group – Insurtech Innovation Award 2024

How to Troubleshoot Apps for the Modern Connected Worker

Data Cloud, More than a CDP by Matt Robison