DevOps Oxford- DevOps + BigData @ RealTime
•
1 like•188 views
This document discusses using DevOps and big data techniques to analyze tweets from the 2014 World Cup in real time. It was predicted that 1.2 million tweets per hour would need to be analyzed, but the actual peak was 2.5 million tweets per hour. AWS was used to deploy multiple EC2 instances across availability zones load balanced with ELB. Infrastructure was provisioned automatically using CloudFormation and Puppet in only 2 minutes. The tournament averaged 17 million tweets per day with a peak of 33 million in one day, demonstrating the need for scalable real-time analytics of big data.
Report
Share
Report
Share
Download to read offline
Recommended
DevOps in the clouds
DevOps in the clouds covers the usage of DevOps methodology to make managing cloud environments easy for developers.
Houldridge real options 2000
This document discusses overcoming barriers to adopting real options as a decision-making tool within organizations. It provides practical advice on using real options, including allowing flexibility in resource allocation, incorporating uncertainty into objectives, and using both qualitative and quantitative analysis. The document also discusses how Smith & Nephew overcame barriers to adoption by communicating examples to key decision-makers and allowing autonomous adoption by different business units.
Actividad tema3
Este documento describe una actividad para analizar iniciativas educativas relacionadas con el fomento de las competencias STEM. Se recomienda seleccionar una experiencia como CORD en Egipto, que promueve la robótica educativa entre estudiantes, y realizar un análisis DAFO. El análisis deberá publicarse en línea con el hashtag #Msec_TeI_STEM.
BROCHURE
The document discusses the benefits of exercise for mental health. Regular physical activity can help reduce anxiety and depression and improve mood and cognitive function. Exercise causes chemical changes in the brain that may help protect against mental illness and improve symptoms.
La biodiversidad
El documento trata sobre la biodiversidad y la vida silvestre. Define la biodiversidad como la diversidad de especies vegetales y animales en un hábitat, e identifica varios tipos como la diversidad de especies, genética, ecosistemas y cultural. Define la vida silvestre como organismos no domesticados y explica que la conservación de la vida silvestre busca proteger especies salvajes para que continúen reproduciéndose sin amenazas, utilizando técnicas como prohibiciones, restauración de hábitats y santuarios.
storyshop_handbook_FINAL1
This document provides guidance on using digital storytelling as a training methodology for healthcare professionals to help manage work-related stress. It outlines the digital storytelling process, including choosing topics, writing scripts, storyboarding, recording voiceovers, editing videos, and facilitating workshops. Some potential topics addressed are balancing work and personal life, workload, interactions with clients and their families, ethical dilemmas, and cultural differences. The goal is to help professionals reflect on challenging experiences and identify ways to improve stress management and relationships through creating and sharing digital stories.
Rostros Edelson Parada
Rostros en acrílicos sobre espátulas es una obra de arte de Edelson Parada que representa rostros humanos pintados con acrílico sobre espátulas de madera. La artista utiliza espátulas como lienzo para explorar la identidad y la individualidad a través de retratos abstractos de rostros.
Recommended
DevOps in the clouds
DevOps in the clouds covers the usage of DevOps methodology to make managing cloud environments easy for developers.
Houldridge real options 2000
This document discusses overcoming barriers to adopting real options as a decision-making tool within organizations. It provides practical advice on using real options, including allowing flexibility in resource allocation, incorporating uncertainty into objectives, and using both qualitative and quantitative analysis. The document also discusses how Smith & Nephew overcame barriers to adoption by communicating examples to key decision-makers and allowing autonomous adoption by different business units.
Actividad tema3
Este documento describe una actividad para analizar iniciativas educativas relacionadas con el fomento de las competencias STEM. Se recomienda seleccionar una experiencia como CORD en Egipto, que promueve la robótica educativa entre estudiantes, y realizar un análisis DAFO. El análisis deberá publicarse en línea con el hashtag #Msec_TeI_STEM.
BROCHURE
The document discusses the benefits of exercise for mental health. Regular physical activity can help reduce anxiety and depression and improve mood and cognitive function. Exercise causes chemical changes in the brain that may help protect against mental illness and improve symptoms.
La biodiversidad
El documento trata sobre la biodiversidad y la vida silvestre. Define la biodiversidad como la diversidad de especies vegetales y animales en un hábitat, e identifica varios tipos como la diversidad de especies, genética, ecosistemas y cultural. Define la vida silvestre como organismos no domesticados y explica que la conservación de la vida silvestre busca proteger especies salvajes para que continúen reproduciéndose sin amenazas, utilizando técnicas como prohibiciones, restauración de hábitats y santuarios.
storyshop_handbook_FINAL1
This document provides guidance on using digital storytelling as a training methodology for healthcare professionals to help manage work-related stress. It outlines the digital storytelling process, including choosing topics, writing scripts, storyboarding, recording voiceovers, editing videos, and facilitating workshops. Some potential topics addressed are balancing work and personal life, workload, interactions with clients and their families, ethical dilemmas, and cultural differences. The goal is to help professionals reflect on challenging experiences and identify ways to improve stress management and relationships through creating and sharing digital stories.
Rostros Edelson Parada
Rostros en acrílicos sobre espátulas es una obra de arte de Edelson Parada que representa rostros humanos pintados con acrílico sobre espátulas de madera. La artista utiliza espátulas como lienzo para explorar la identidad y la individualidad a través de retratos abstractos de rostros.
R.ADAMS
This document is an advertisement for JCPenney that features various clothing items and accessories on sale. It includes:
- Descriptions and prices for women's tops, shorts, dresses, and other summer clothing items mostly from brands like Arizona and St. John's Bay.
- Additional coupons and deals offering extra percentages off other items storewide and with a JCPenney credit card.
- An explanation of a "mix and match" deal buying bras and underwear in sets at a discounted price.
Proceso Tecnológico
El documento explica las fases del proceso tecnológico para resolver un problema, que incluyen: 1) definir el problema, 2) buscar información, 3) diseñar una solución, 4) planificar cómo implementarla, 5) construir un prototipo, y 6) evaluar la solución. También describe cada fase en más detalle.
Comunicación interactiva
Este documento presenta conceptos clave sobre comunicación interactiva y digital. Explica que la comunicación interactiva implica una interacción directa entre emisor y receptor, donde el receptor puede regular el flujo de información. También describe características de Internet como hipertexto, multimedia e interactividad; y formatos digitales como HTML, blogs, redes sociales y la nube.
Actividad 2
Este documento presenta una actividad educativa sobre el uso de Twitter. El objetivo es que los estudiantes analicen las funcionalidades de comunicación de Twitter y descubran sus ventajas para la autoformación. Se propone crear un perfil de Twitter y participar en la red siguiendo cuentas educativas y publicando contenido con hashtag. Finalmente, los estudiantes deben idear un proyecto educativo que incorpore el uso de Twitter y publicarlo online con el hashtag #Msec_TeI_Twitter.
La castañeda
Esta presentación fue hecha para la materia Desarrollo de Habilidades en el uso de la Tecnología, la Información y la Comunicación,de acuerdo a los materiales didácticos de la plataforma :“Creación de presentaciones efectivas” de la web de Observatorio tecnológico y la presentación “Death by PowerPoint”. El tema consiste en las razones por las que se creo el Manicomio General La Castañeda y la influencia europea en el
La informacion
Este documento define la información y discute su importancia para las ciencias de la información. La información se compone de datos, recuperación de información y usuarios. Nicholas Belkin propuso que la información implica comunicación entre personas con un propósito. La información mejora la eficiencia y productividad y es fundamental para el desarrollo de las ciencias de la información ya que permite a las personas tomar mejores decisiones y difundir conocimiento.
tabletop
Rhonda Adams is a table top art director and stylist. Her experience includes 5 years working in table top art direction and styling. She has the skills to direct and style table top presentations.
Presentation 1 1
Este documento habla sobre la importancia de la claridad y coherencia en la redacción de textos. Menciona que cuando se escribe un texto, se debe tener en cuenta al posible lector u oyente y asegurarse de que la información sea completamente clara para ellos, ya que no necesariamente tendrán conocimiento previo sobre el tema. También destaca la necesidad de considerar el estilo, claridad, coherencia y cohesión al redactar. Finalmente, define el resumen como una pequeña idea sacada del texto leído que ofrece toda la información necesaria
Caso carlos
Basado en casos reales de Mobbing en el trabajo, enriquecido por experiencias personales y en re estructuración de empresas.
La Escuela y yo
El documento describe las opiniones de un estudiante sobre su escuela. A él más le gusta la enseñanza de los profesores, las aulas y las áreas de apoyo psicopedagógico y cafetería. Lo que menos le gusta son la falta de sillas suficientes y en mal estado, y los baños sucios y sin agua. Para mejorar la escuela, propone incrementar sillas nuevas, mejorar los salones, mantener la limpieza y siempre tener agua y mejor ventilación.
Trabalho vilmar slide
O documento discute os principais fatores que contribuem para a obesidade, como o consumo excessivo de alimentos industrializados, a falta de exercício físico e a influência da indústria de alimentos por meio de marketing direcionado a crianças. Também aborda a obesidade infantil como um problema grave do século XXI e os riscos de um desmame precoce.
OAHSPRING2016_RENOVATION ARTICLE
This document summarizes the renovation of a home in Ottawa's McKellar Park by its owners Simon and Dawn Melanson. They transformed the home into a warm, luxurious space decorated with their extensive collection of artifacts from over 40 countries. The renovation included designing rooms like the library and lounge in distinct styles, and showcasing treasures like pieces of the Berlin Wall. The goal was to create a home reflecting the owners' love of travel, family, and collecting meaningful objects from around the world.
KapustaBell
Team Kapusta Racing is a newly formed NASCAR Xfinity team owned by businessman Thom Kapusta. The team is based in Mooresville, NC and combines the owners' passion for racing with a successful business model to achieve success on and off the track. The team hopes to partner with sponsors to promote their brands through the team's national television exposure, marketing promotions, and exclusive access opportunities. The team's driver, TJ Bell, is an experienced racer who has had success in various racing series.
THE RISE AND FALL OF SERVERLESS COSTS - TAMING THE (SERVERLESS) BEAST
From talk given on June 10, 2020 at the DevTalks Reimagined conference.
ABSTRACT:
Serverless is an amazing beast of a technology. With it, you can quickly build and deploy incredible systems. You get out-of-the-box scalability and flexibility. Nevertheless, with great power comes great(er) cost! In this talk, you’ll learn about building a huge data pipeline using Serverless architecture, and how to tame the beast. After this session, you will understand the pitfalls to avoid and the great powers to exploit.
Big data @ Hootsuite analtyics
An overview over what the Hootsuite Bucharest office does to aggregate data into analytics and insights.
Childhood Dream to Pi World Record by Emma Haruka, Developer Advocate @Google
Childhood Dream to Pi World Record by Emma Haruka, Developer Advocate @GoogleParis Women in Machine Learning and Data Science
We have calculated 31.4 trillion digits of Pi in 2019 and broke the world record in the Pi computation. This talk will discuss the nature of the calculation, the architecture, challenges and techniques, and of course the brief history of Pi computation. Calculating Pi has been the speaker's childhood dream and this talk will also explain how the small dream grew to the new world record.Playlist preprogramming, from (almost) nothing to something cool, ASKIP
Retour d'expérience sur la mise en place de la publication différée des playlists éditoriales chez Deezer.
Challenges techniques et solutions mises en place et un bilan calmement.
Balogh gyorgy big_data
Big data is characterized by large volumes of data that cannot be handled by traditional databases. The amount of data produced globally each second includes 30 TB from digital media like YouTube. Factors like Moore's law and increasing data storage have led to an exponential growth of data. This paradigm shift requires distributed storage and processing of data across thousands of nodes. Techniques like data compression, clever aggregation, and cache-aware programming can improve efficiency and boost performance by orders of magnitude for big data problems. Simply using Hadoop may not always be the most cost effective solution.
stackconf 2021 | Help, My Datacenter is on Fire
Most people will claim that this never happens, others hope it never happens, but it happened on March 10, 2021, and it was not just the 1 datacenter that got impacted, but the whole campus of the provider that got powered down. This talk will explain how our customers survived this outage, how our culture, opensource tooling and automation saved the da(y,ta). A talk about disaster recovery, business continuity plans and building cloud agnostic stacks that survive disasters.
Help , My Datacenter is on fire
Most people will claim that this never happens, others hope it never happens, but it happened on March 10, 2021, and it was not just the 1 datacenter that got impacted, but the whole campus of the provider that got powered down. This talk will explain how our customers survived this outage, how our culture, opensource tooling and automation saved the da(y,ta). A talk about disaster recovery, business continuity plans and building cloud agnostic stacks that survive disasters.
More Related Content
Viewers also liked
R.ADAMS
This document is an advertisement for JCPenney that features various clothing items and accessories on sale. It includes:
- Descriptions and prices for women's tops, shorts, dresses, and other summer clothing items mostly from brands like Arizona and St. John's Bay.
- Additional coupons and deals offering extra percentages off other items storewide and with a JCPenney credit card.
- An explanation of a "mix and match" deal buying bras and underwear in sets at a discounted price.
Proceso Tecnológico
El documento explica las fases del proceso tecnológico para resolver un problema, que incluyen: 1) definir el problema, 2) buscar información, 3) diseñar una solución, 4) planificar cómo implementarla, 5) construir un prototipo, y 6) evaluar la solución. También describe cada fase en más detalle.
Comunicación interactiva
Este documento presenta conceptos clave sobre comunicación interactiva y digital. Explica que la comunicación interactiva implica una interacción directa entre emisor y receptor, donde el receptor puede regular el flujo de información. También describe características de Internet como hipertexto, multimedia e interactividad; y formatos digitales como HTML, blogs, redes sociales y la nube.
Actividad 2
Este documento presenta una actividad educativa sobre el uso de Twitter. El objetivo es que los estudiantes analicen las funcionalidades de comunicación de Twitter y descubran sus ventajas para la autoformación. Se propone crear un perfil de Twitter y participar en la red siguiendo cuentas educativas y publicando contenido con hashtag. Finalmente, los estudiantes deben idear un proyecto educativo que incorpore el uso de Twitter y publicarlo online con el hashtag #Msec_TeI_Twitter.
La castañeda
Esta presentación fue hecha para la materia Desarrollo de Habilidades en el uso de la Tecnología, la Información y la Comunicación,de acuerdo a los materiales didácticos de la plataforma :“Creación de presentaciones efectivas” de la web de Observatorio tecnológico y la presentación “Death by PowerPoint”. El tema consiste en las razones por las que se creo el Manicomio General La Castañeda y la influencia europea en el
La informacion
Este documento define la información y discute su importancia para las ciencias de la información. La información se compone de datos, recuperación de información y usuarios. Nicholas Belkin propuso que la información implica comunicación entre personas con un propósito. La información mejora la eficiencia y productividad y es fundamental para el desarrollo de las ciencias de la información ya que permite a las personas tomar mejores decisiones y difundir conocimiento.
tabletop
Rhonda Adams is a table top art director and stylist. Her experience includes 5 years working in table top art direction and styling. She has the skills to direct and style table top presentations.
Presentation 1 1
Este documento habla sobre la importancia de la claridad y coherencia en la redacción de textos. Menciona que cuando se escribe un texto, se debe tener en cuenta al posible lector u oyente y asegurarse de que la información sea completamente clara para ellos, ya que no necesariamente tendrán conocimiento previo sobre el tema. También destaca la necesidad de considerar el estilo, claridad, coherencia y cohesión al redactar. Finalmente, define el resumen como una pequeña idea sacada del texto leído que ofrece toda la información necesaria
Caso carlos
Basado en casos reales de Mobbing en el trabajo, enriquecido por experiencias personales y en re estructuración de empresas.
La Escuela y yo
El documento describe las opiniones de un estudiante sobre su escuela. A él más le gusta la enseñanza de los profesores, las aulas y las áreas de apoyo psicopedagógico y cafetería. Lo que menos le gusta son la falta de sillas suficientes y en mal estado, y los baños sucios y sin agua. Para mejorar la escuela, propone incrementar sillas nuevas, mejorar los salones, mantener la limpieza y siempre tener agua y mejor ventilación.
Trabalho vilmar slide
O documento discute os principais fatores que contribuem para a obesidade, como o consumo excessivo de alimentos industrializados, a falta de exercício físico e a influência da indústria de alimentos por meio de marketing direcionado a crianças. Também aborda a obesidade infantil como um problema grave do século XXI e os riscos de um desmame precoce.
OAHSPRING2016_RENOVATION ARTICLE
This document summarizes the renovation of a home in Ottawa's McKellar Park by its owners Simon and Dawn Melanson. They transformed the home into a warm, luxurious space decorated with their extensive collection of artifacts from over 40 countries. The renovation included designing rooms like the library and lounge in distinct styles, and showcasing treasures like pieces of the Berlin Wall. The goal was to create a home reflecting the owners' love of travel, family, and collecting meaningful objects from around the world.
KapustaBell
Team Kapusta Racing is a newly formed NASCAR Xfinity team owned by businessman Thom Kapusta. The team is based in Mooresville, NC and combines the owners' passion for racing with a successful business model to achieve success on and off the track. The team hopes to partner with sponsors to promote their brands through the team's national television exposure, marketing promotions, and exclusive access opportunities. The team's driver, TJ Bell, is an experienced racer who has had success in various racing series.
Viewers also liked (15)
Similar to DevOps Oxford- DevOps + BigData @ RealTime
THE RISE AND FALL OF SERVERLESS COSTS - TAMING THE (SERVERLESS) BEAST
From talk given on June 10, 2020 at the DevTalks Reimagined conference.
ABSTRACT:
Serverless is an amazing beast of a technology. With it, you can quickly build and deploy incredible systems. You get out-of-the-box scalability and flexibility. Nevertheless, with great power comes great(er) cost! In this talk, you’ll learn about building a huge data pipeline using Serverless architecture, and how to tame the beast. After this session, you will understand the pitfalls to avoid and the great powers to exploit.
Big data @ Hootsuite analtyics
An overview over what the Hootsuite Bucharest office does to aggregate data into analytics and insights.
Childhood Dream to Pi World Record by Emma Haruka, Developer Advocate @Google
Childhood Dream to Pi World Record by Emma Haruka, Developer Advocate @GoogleParis Women in Machine Learning and Data Science
We have calculated 31.4 trillion digits of Pi in 2019 and broke the world record in the Pi computation. This talk will discuss the nature of the calculation, the architecture, challenges and techniques, and of course the brief history of Pi computation. Calculating Pi has been the speaker's childhood dream and this talk will also explain how the small dream grew to the new world record.Playlist preprogramming, from (almost) nothing to something cool, ASKIP
Retour d'expérience sur la mise en place de la publication différée des playlists éditoriales chez Deezer.
Challenges techniques et solutions mises en place et un bilan calmement.
Balogh gyorgy big_data
Big data is characterized by large volumes of data that cannot be handled by traditional databases. The amount of data produced globally each second includes 30 TB from digital media like YouTube. Factors like Moore's law and increasing data storage have led to an exponential growth of data. This paradigm shift requires distributed storage and processing of data across thousands of nodes. Techniques like data compression, clever aggregation, and cache-aware programming can improve efficiency and boost performance by orders of magnitude for big data problems. Simply using Hadoop may not always be the most cost effective solution.
stackconf 2021 | Help, My Datacenter is on Fire
Most people will claim that this never happens, others hope it never happens, but it happened on March 10, 2021, and it was not just the 1 datacenter that got impacted, but the whole campus of the provider that got powered down. This talk will explain how our customers survived this outage, how our culture, opensource tooling and automation saved the da(y,ta). A talk about disaster recovery, business continuity plans and building cloud agnostic stacks that survive disasters.
Help , My Datacenter is on fire
Most people will claim that this never happens, others hope it never happens, but it happened on March 10, 2021, and it was not just the 1 datacenter that got impacted, but the whole campus of the provider that got powered down. This talk will explain how our customers survived this outage, how our culture, opensource tooling and automation saved the da(y,ta). A talk about disaster recovery, business continuity plans and building cloud agnostic stacks that survive disasters.
Continuous Infrastructure First
This document discusses continuous infrastructure and delivery. It describes how automation of build, deployment, and testing processes through improved collaboration between developers, operations, and testers allows delivery teams to release changes in hours or even minutes regardless of project size or code complexity. The document also discusses how treating infrastructure as code through configuration management tools like Puppet, Chef, and Ansible supports continuous delivery.
OSMC 2023 | Large-scale logging made easy by Alexandr Valialkin
Logging at scale is a common source of infrastructure expenses and frustration. While logging is something any organization does, there is still no silver bullet or just a simple and scalable solution without trade-offs. After studying the most popular logging systems, Aliaksandr came up with his design and vision of the problem. The proposed solution fits ideally for SRE, DevOps, and system engineers who need to provide logging solutions as a platform for the entire company or team. It accepts logs from existing logging agents, pipelines, and streams, efficiently stores them in a highly optimised log database and can be queried at lightning-fast speeds with perfect integration with tools like jq, awk, cut, etc.
Data Lessons Learned at Scale
Charlie Reverte, VP of Engineering at AddThis, discusses lessons learned from processing large-scale web data. AddThis processes data from 14 million domains, including 100 billion monthly page views and 50,000 events per second. Reverte outlines challenges around distributed ID generation, counting unique values, joining distributed data, sampling large datasets, and deploying systems that invalidate over 1.4 billion browser caches. He advocates for loose coupling between systems using approaches like Kafka for asynchronous event logging. Reverte also discusses techniques for columnar compression, tunable quality of service, and open sourcing Hydra, AddThis' custom processing system optimized for real-time data.
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
This presentation covers the following topics:
What is logging?
The purpose of logging: Debugging
The purpose of logging: Security
The purpose of logging: Stats & analytics
Traditional logging
Traditional logging: Advantages
Traditional logging: Disadvantages
The solution: Large-scale logging
Large-scale logging: Core principles
Large-scale logging: Solution types
Large-scale logging: Cloud vs on-prem
Large-scale logging: Operational complexity
Large-scale logging: Security
Large-scale logging: Costs
Large-scale logging: On-prem comparison
- Elasticsearch
- Grafana Loki
- VictoriaLogs
On-prem comparison: Setup and operation
On-prem comparison: Costs
On-prem comparison: Full-text search support
On-prem comparison: How to efficiently query 100TB of logs?
On-prem comparison: Integration with CLI tools
VictoriaLogs for large-scale logging
VictoriaLogs demo instance
- Ingestion rate: 3600 messages / minute
- The number of log messages: 1.1 billion
- Uncompressed log messages’ size: 1.5TB
- Compressed log messages’ size: 23GB
- Compression ratio: 47x
- Memory usage: 150MB
VictoriaLogs CLI integration demo
- Which errors have occurred in all the apps during the last hour?
- How many errors have occurred during the last hour?
- Which apps generated the most of errors during the last hour?
- The number of per-minute errors for the last 10 minutes
- Status codes for the last hour
- Non-200 status codes for the last week
- Top client IPs for the last 4 weeks with 404 and 500 response status codes
- Per-month stats for the given IP across all the logs
Large-scale logging solution
MUST provide
excellent CLI integration
VictoriaLogs: (temporary) drawbacks
VictoriaLogs: Recap
- Easy to setup and operate
- The lowest RAM usage and disk space usage (up to 30x less than Elasticsearch and Grafana Loki)
- Fast full-text search
- Excellent integration with traditional command-line tools for log analysis
- Accepts logs from popular log shippers (Filebeat, Fluentbit, Logstash, Vector, Promtail, Grafana Agent)
- Open source and free to use!
testmon for Python
testmon for Python concepts and IndieGoGo campaign. Slides for presentation on a local Ruby and Python meet-up.
Druid Summit 2023 : Changing Druid Ingestion from 3 hours to 5 minutes
This is a real world account from a Druid cluster in production. A story of 48 hours of debugging, learning and understanding Druid better, filing a couple of issues in Druid github and finally a stable production pipeline again thanks to the Druid community.
We will discuss the bottlenecks we had in overlord, slot issues for Peons in middle managers, coordinator bottlenecks, how we mitigated task and segment flooding, what configs we changed sprinkled with real world numbers and snapshots from our grafana dashboards.
Prometheus the future
This document provides an overview of the Prometheus monitoring system. It discusses that Prometheus was originally created by frustrated engineers at SoundCloud to address their monitoring needs. It gained popularity and was later maintained as an open source project. Major milestones included Prometheus 1.0 and 2.0 releases, joining the CNCF, and graduating as a CNCF project. The document outlines ongoing work to improve areas like data integrity, security, and community involvement through initiatives like OpenMetrics. The future of Prometheus is focused on further developing the project and monitoring ecosystem through an expanding team and community.
Data Science in the Cloud @StitchFix
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2lGNybu.
Stefan Krawczyk discusses how his team at StitchFix use the cloud to enable over 80 data scientists to be productive. He also talks about prototyping ideas, algorithms and analyses, how they set up & keep schemas in sync between Hive, Presto, Redshift & Spark and make access easy for their data scientists, etc. Filmed at qconsf.com..
Stefan Krawczyk is Algo Dev Platform Lead at StitchFix, where he’s leading development of the algorithm development platform. He spent formative years at Stanford, LinkedIn, Nextdoor & Idibon, working on everything from growth engineering, product engineering, data engineering, to recommendation systems, NLP, data science and business intelligence.
OSMC 2018 | Learnings, patterns and Uber’s metrics platform M3, open sourced ...
At Uber we use high cardinality monitoring to observe and detect issues with our 4,000 microservices running on Mesos and across our infrastructure systems and servers. We’ll cover how we put the resulting 6 billion plus time series to work in a variety of different ways, auto-discovering services and their usage of other systems at Uber, setting up and tearing down alerts automatically for services, sending smart alert notifications that rollup different failures into individual high level contextual alerts, and more. We’ll also talk about how we accomplish all this with a global view of our systems with M3, our open source metrics platform. We’ll take a deep dive look at how we use M3DB, now available as an open source Prometheus long term storage backend, to horizontally scale our metrics platform in a cost efficient manner with a system that’s still sane to operate with petabytes of metrics data.
AWS Techniques and lessons writing low cost autoscaling GitLab runners
After publishing my first terraform module I shared some details of how I used AWS services and leveraged serverless concepts and provided a demo to the Melbourne AWS User Group meetup https://www.meetup.com/aws-aus/events/291459709/
Serverless for High Performance Computing
Serverless is great for web applications and APIs, but this does not mean it cannot be used successfully for other use cases. In this talk, we will discuss a successful application of serverless in the field of High Performance Computing. Specifically we will discuss how Lambda, Fargate, Kinesis and other serverless technologies are being used to run sophisticated financial models at one of the major reinsurance companies in the World. We we learn about the architecture, the tradeoffs, some challenges and some unresolved pain points. Most importantly, we'll find out if serverless can be a great fit for HPC and if we can finally stop managing those boring EC2 instances!
What is Software Development by Thesys Tech Head of Development
From this presentation you will learn how the modern software development life-cycle works and the differences between different data stores from a non-technical perspective.
Embedded Recipes 2017 - Understanding SCHED_DEADLINE - Steven Rostedt
Starting in Linux version 3.14, a new scheduling class was introduced. This class is called SCHED_DEADLINE. It implements Earliest Deadline First (EDF) along with a Constant Bandwidth Scheduler (CBS) that is used to give applications a guaranteed amount of CPU for a periodic time frame. This type of scheduling is advantageous for robotics, media players and recorders, as well as virtual machine guest management. This talk will explain the history of SCHED_DEADLINE and compare it with various other methods to deal with periodic deadlines. It will also discuss some of the current issues with the current Linux implementation and some of the improvements that are currently being worked on.
Since its conception, SCHED_DEADLINE was not being used at all. I know this because as soon as I started using it, I discovered several bugs within the code (the fixes have now been upstreamed). But the fact that these bugs have been in the kernel for so long tells me that this wonderful feature is not well known. This talk is to help spread the word, and perhaps more people can take advantage of the SCHED_DEADLINE power.
Steven Rostedt, VMware
Similar to DevOps Oxford- DevOps + BigData @ RealTime (20)
THE RISE AND FALL OF SERVERLESS COSTS - TAMING THE (SERVERLESS) BEAST
THE RISE AND FALL OF SERVERLESS COSTS - TAMING THE (SERVERLESS) BEAST
Childhood Dream to Pi World Record by Emma Haruka, Developer Advocate @Google
Childhood Dream to Pi World Record by Emma Haruka, Developer Advocate @Google
Playlist preprogramming, from (almost) nothing to something cool, ASKIP
Playlist preprogramming, from (almost) nothing to something cool, ASKIP
OSMC 2023 | Large-scale logging made easy by Alexandr Valialkin
OSMC 2023 | Large-scale logging made easy by Alexandr Valialkin
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
Druid Summit 2023 : Changing Druid Ingestion from 3 hours to 5 minutes
Druid Summit 2023 : Changing Druid Ingestion from 3 hours to 5 minutes
OSMC 2018 | Learnings, patterns and Uber’s metrics platform M3, open sourced ...
OSMC 2018 | Learnings, patterns and Uber’s metrics platform M3, open sourced ...
AWS Techniques and lessons writing low cost autoscaling GitLab runners
AWS Techniques and lessons writing low cost autoscaling GitLab runners
What is Software Development by Thesys Tech Head of Development
What is Software Development by Thesys Tech Head of Development
Embedded Recipes 2017 - Understanding SCHED_DEADLINE - Steven Rostedt
Embedded Recipes 2017 - Understanding SCHED_DEADLINE - Steven Rostedt
DevOps Oxford- DevOps + BigData @ RealTime
- 1. DevOps + BigData @ RealTime
- 2. DevOps+ (DevOps in a startup)
- 3. DevOps+ ● Fixing the electrical outage
- 4. DevOps+ ● Fixing the electrical outage ● Cycling to Maplin for an extension lead ● Dealing with the sewage from the blocked toilet
- 5. BigData ● Also known as - Data ● BigData is writing Doom or Elite for a 32k machine:
- 6. BigData ● Or just because you can’t quite fit it on a 39” rotating disk:
- 7. @RealTime ● Can you survive this ?
- 8. The Challenge - The 2014 (Football) World Cup Analyse every Tweet issued during World Cup games in the 2014 competition
- 9. The Challenge - The 2014 (Football) World Cup
- 10. The Challenge - The 2014 (Football) World Cup How busy was it expected to be ?
- 12. The Challenge - The 2014 (Football) World Cup How busy was it expected to be ? ● Predicted 1.2M Tweets per hour ● Average time for our analysis is 100ms - lengthy texts up to 150ms ● Our analysis time increases logarithmically with sentence count and sentence length so needed to avoid pathological cases ● 150ms x 1.2M = 180k sec = 3000 minutes = between 33 and 50 hours
- 13. The Solution - AWS ● Lots of EC2 instances ● 3 Availability Zones ● ELB
- 14. The Solution - Unattended Build ● Cloud Formation ● Puppet ● Route53 ● ELB ● 2 min
- 15. The Challenge - The 2014 (Football) World Cup How busy was it ? ● Predicted peak 1.2M Tweets per hour ● Actual peak 2.5M Tweets per hour ● Whole tournament avg 17M per day for 30 days ● Peak 33M in one day
- 16. The Challenge - The 2014 (Football) World Cup How busy was it ? ● Predicted peak 1.2M Tweets per hour ● Actual peak 2.5M Tweets per hour ● Whole tournament avg 17M per day for 30 days ● Peak 33M in one day
- 17. The Challenge - The 2014 (Football) World Cup How busy was it ? ● Predicted peak 1.2M Tweets per hour ● Actual peak 2.5M Tweets per hour ● Whole tournament avg 17M per day for 30 days ● Peak 33M in one day
- 20. ● A Long Time Ago In A Galaxy Far Far Away ● Carhire3000 began - we had 6 servers ● Building a new server took a week ● 8 years later:- ○ 2 acquisitions ○ Booking.com ○ $996 million ○ 600+ servers ○ DevOps team of 5
- 21. ● Booking.com methodology - Unattended build in 5 minutes ● Control Room - Monitoring health of website
- 22. ● Control Room - Columnar Databases
- 23. ● Columnar Databases ● Fire & Forget UDP events ● What do you monitor ? ● Experiments & Z-score ● DB Queries < 2 seconds ● Hot event data in MySQL - Hands Off ! ● 5 minutes later moved to columnar storage database