IEEE ISSRE 2018 - Use of Self-Healing Techniques to Improve the Reliability of a Dynamic and Geo-Distributed Ad Delivery Service

•

2 likes•430 views

The advertising industry faces numerous challenges in achieving its goal of targeting a given audience dynamically and accurately in order to deliver a meaningful brand message. Near real-time, low latency delivery of dynamic content, the sheer volume of information processed, and the sparse geographic distribution of the intended eyeball traffic all drive the complexity of building a successful experience for the end user and the brand. Additionally, the competitiveness of the industry makes it critical to preserve low operational expenses while delivering reliably at scale. In attempting to address the above, we have found that a distributed infrastructure that leverages public cloud providers and a private cloud with open infrastructure technologies can deliver dynamic advertising content with low latency while preserving its high availability. But network or physical utility infrastructures can’t be relied on to ensure the service dependability. We show that the complexity of the networks, the sparse geographic distribution of eyeballs, the risk of data center failures, and the increase of encrypted transactions call for thoughtful architectures. The introduction of modern practices, failure injections, and self-healing mechanisms allowed us to improve the service fault tolerance while optimizing for latency and significantly improve our service reliability.

Technology

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Use of Self-Healing Techniques to Improve the Reliability of a Dynamic
and Geo-Distributed Ad Delivery Service
Nicolas Brousse and Oleksii Mykhailov

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Adobe Advertising Cloud
Serving All Media Content Across
Any Screens in Any Format
2

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 3
BEFORE
RFP, IO, human based orders
NOW
Programmatic Ad Buying with
Real Time Bidding

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 4
Latency
<50ms @ 95th percentile
High Traffic
300 billion requests a day
Huge Datasets
Billions of objects to store

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 5
Ad Content Delivered To Eyeball

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 6
Traditional Ad
Serving Implement
GeoDNS GSLB

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Inconsistent GeoDNS Routing
High Latency From Eyeball To Content Origin
Origin Failure Impact User Experience
Impact Campaign Performance and Revenue
7

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Relying on GeoDNS to Figure Out Eyeball Location is
UNRELIABLE
8
Optimal Route
Actual Route

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 9
TCP and TLS
Handshake Impact
Latency

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Datacenter Blackout
Network Outage
Human Errors
Natural Disaster
10
High Risks Of Origin Failures

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Service Unavailability
11

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 12
SOLUTION
Eyeball Traffic Access Content via Smart Edges

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 13
Smart Edges Are
Anycast POPs That
Manage Failover and
Self-Healing

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 14
Few Words About Anycast
§ Shortest Path Routing Means
§ Not Latency Aware
§ Not Congestion Aware / Packet Loss
§ Limited Control for Traffic Steering
§ Difficult Troubleshooting
§ Failover lead to packet RST for Active Sessions
§ Mitigation with a large and well distributed number of POPs

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 15
Anycast POP
Improve Latency
e.g. 3X Faster

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 16
Automate
Failover and Recovery

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 17
Inject Failures In
Production To Validate
Smart Edges Behavior

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Human Failover vs Self-Healing
18
> 1h Failure
< 15min
< 30min Region
Traffic Rerouted
Self Recover
Note: since the paper publication, we reduced automated failover time to be less than a few seconds. See demo.
Fig. 1 Human Failover with Manual Recovery steps Fig. 2 Automated Failover and Self-Healing Recovery

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 19
Injecting Complete
Data Center Failure
at the Regional Level
LIVE

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Recorded Demo
20

IEEE ISSRE 2018 - Use of Self-Healing Techniques to Improve the Reliability of a Dynamic and Geo-Distributed Ad Delivery Service

The document discusses several key facts about A2P messaging: 1) 95% of A2P traffic is generated by over-the-top (OTT) applications; 2) two-factor authentication (2FA) messages form the main part of monetizable traffic; 3) local A2P traffic should also be monitored, as large volumes may be leaking over unauthorized "grey routes"; 4) domestic A2P revenues can be lost if international grey route prices are lower than domestic rates, even though domestic volumes are often 10 times higher. The document promotes contacting experts at haud.com to learn more about A2P messaging solutions.

Visualizing Cellular Audience for Streaming KPI's

Akamai Developers & Admins

This document discusses visualizing streaming performance data for cellular audiences. It notes that more than 75% of viewers now stream from mobile networks. The document outlines how Akamai's Media Analytics and Network Experience Analytics tools can provide granular performance data and metrics on a cellular level, including by cell ID, operator, and location. This level of data insights allows content providers to optimize their streaming for mobile audiences and cellular networks.

Improving Adobe Experience Cloud Services Dependability with Machine Learning

Nicolas Brousse

Adobe Experience Cloud is a collection of best-in-class solutions for marketing, analytics, advertising, and commerce. All integrated on a cloud platform for a single experience system of record. The Adobe Experience Cloud's SRE team works hand-in-hand with the Product and Engineering teams to build dependable services. In this presentation, you will learn how the team leverage Adobe's artificial intelligence and machine learning engine to build predictive auto-scaling and self-healing services.

Using AEM in a customer global multi-channel program

Meryll Blanchet

This document discusses using Adobe Experience Manager (AEM) in a global multi-channel program. It begins with an overview and agenda. It then discusses a real-life example program involving CRM, web, email, mobile, and analytics systems. It focuses on how AEM was used, including user authentication via SAML, personalizing content, and integrating with Adobe Target and Analytics. Key challenges discussed are managing expectations, careful planning and estimation, and prioritizing analytics requirements.

When Content Meets Data, Big Things Happen - Peter Krmpotic, Adobe

NewsCred

The document discusses Adobe's content marketing solution. It describes how content explosion has created challenges for marketers. Adobe's solution addresses these challenges through a comprehensive content marketing process that includes audience segmentation, content strategy, curation, composition, delivery, and amplification across channels. It leverages Adobe Experience Manager for content optimization and integration with the Adobe Marketing Cloud.

Altitude San Francisco 2018: How Magento moved to the cloud while maintaining...

Fastly

Magento Commerce was first released by a small web development agency over ten years when they saw first-hand what a challenge it was for companies like them to build unique eCommerce sites. They created an open source platform that gives developers the flexibility to create meaningful shopping experiences while building a global community that drives down merchant costs and fosters innovation. Amid the rise of cloud-based software Magento needed to keep pace with more complex merchant needs and heightened shopper expectations. In this session learn how Magento, with the help of Partners like Fastly, evolved into a cloud-based platform without sacrificing their commitment to open software, flexibility, and the community.

12/6 Initiative

Rob Healy

The document proposes an initiative called the 12/6 Initiative to reduce wasted energy from always-on web servers. It suggests powering down servers outside of peak traffic hours, which could save 57% of wasted energy costs and reduce carbon emissions by 19.9 million tons per year if 10% of servers worldwide adopted this approach. In addition to energy savings, turning servers off during low traffic periods could also reduce the attack surface for hackers, lowering security risks. The document outlines the business and technical feasibility of automating backup and restore to enable safely powering down servers when not needed.

IMMERSE 2016 IST Mark Szulc Keynote

AdobeMarketingCloud

Two large enterprise AEM implementations were presented and compared. Anshul Chhabra from Symantec presented their implementation handling 3.3 billion requests per month. Anil Kalbag from Cisco presented their implementation handling 375 million monthly page views. Both implementations utilized multiple data centers for high availability and disaster recovery. Key architecture decisions around virtual/physical infrastructure, storage, caching, and multi-tenancy were discussed and compared between the two organizations.

Developer To Architect

Anurag Yadav

Design - Start Your API Journey Today

LaurenWendler

This document discusses how APIs can power the API economy and digital transformation. It describes how APIs can be used to expose functionality from various systems and data sources, and how API management is needed to securely manage APIs. The document promotes IBM API Connect as a way to create, manage and secure APIs and microservices across hybrid cloud environments to drive innovation. It provides links to learn more about API Connect and digital transformation workshops.

SAP TechEd 2010 Rich Internet Applications for the Enterprise

Anne Kathrine Petterøe

The document discusses rich internet applications for enterprises. It describes the challenges of enterprise application development and the importance of intuitive user experiences. Examples of different types of rich internet applications are provided. The document also discusses Adobe LiveCycle and its integration with SAP applications, allowing for improved user-centric experiences over traditional system-centric applications. A case study of a project called Hendrix is presented, which used these technologies to optimize processes and improve the user experience for an Adobe call center platform.

Adobe Flash Platform Summit 2010

Anne Kathrine Petterøe

The document discusses rich internet applications (RIAs) for enterprises. It covers the challenges of enterprise application development, how intuitive user experiences can help address those challenges, and how RIA technologies from Adobe like Flash, Flex and LiveCycle can be used to create more engaging user-centric applications. It also provides examples of how Adobe technologies have been used by companies like SAP, Orange and Verizon to build applications that integrate with backend systems and improve customer experiences. Specific integration options for building RIAs with SAP systems using technologies like Flash Islands and LiveCycle Data Services are also examined.

Adobe Advertising Cloud: The Reality of Cloud Bursting with OpenStack

Nicolas Brousse

After successfully moving a large workload from a Public Cloud to an OpenStack Private Cloud, the former TubeMogul Operations Engineering team tackled its next important step toward Cloud Bursting. While experiencing hyper-growth on the Adobe Advertising Cloud, the team had to figure out a simple way to quickly provision new compute resources. Our latency critical workload need our core private resources while some workload can safely leverage public cloud. Cloud Bursting helped to ensure rapid support of the business and provide a more flexible capacity planning strategy. Being able to burst some workload back to the Public Cloud allowed the team to leverage the best of public and private cloud. In this presentation, we will detail how the team dealt with the network complexity between clouds infrastructure, the cost effectiveness of cloud bursting, the impact on performance, and the hard reality of multi-cloud environments.

[NEW LAUNCH!] How to Architect for Multi-Region Redundancy Using Anycast IPs ...

Amazon Web Services

Deployed globally in multiple edge locations, AWS Global Accelerator helps you manage traffic destined to your multi-regional applications with further higher levels of availability and performance. This session comprises ways in which Ubiquity helps you build fault tolerant and highly performant systems across AWS regions using anycast static IP addresses. In this session, you will learn about Global Accelerator’s shuffle sharding technique used for its static IPs, benefits of anycast and more.

Marketing in the Age of Mobile

Adobe Experience Cloud

Mobile usage is growing rapidly, with people checking their phones hundreds of times per day. Many companies have rushed to create mobile apps but struggle with user acquisition, analytics, and iteration. Adobe's Project Fast Track created a unified solution across its mobile apps and Creative Cloud to gather usage data and enable data-driven product improvements. The project team integrated Adobe's Mobile SDK and Analytics to provide insights from nine apps in just nine weeks. This system allows Adobe to better understand user behavior and quickly iterate apps to increase engagement and subscriptions.

Where is cold fusion headed

ColdFusionConference

The document discusses the future direction of the ColdFusion product. It outlines plans to modernize the language with improved object oriented support, CFScript 2.0, support for null values, and multi-threading. It also discusses leveraging containerization with Docker, provisioning a microservices architecture, embracing devsecops practices like a built-in web application firewall, and accelerating digital transformation with enhanced API management capabilities. The goal is to implement a modern platform for building and deploying applications.

Design - Start Your API Journey Today

LaurenWendler

This document discusses the opportunities presented by the API economy. It describes how APIs can enable new experiences like pre-ordering food from a restaurant app before arriving. The document outlines challenges of managing APIs at scale across hybrid cloud environments. It introduces IBM API Connect as a way to create, manage and secure APIs throughout their lifecycle to power digital transformations. Finally, it provides links to learn more about API Connect and taking advantage of IBM's cloud offerings.

Accelerate Value from Big Data, AI, and IoT Initiatives with One-Tenth of the...

Amazon Web Services

Many organizations are awash in different types of data, yet they struggle to rapidly use these assets to improve operations and benefit customers. In this session, we explore best practices from deployments at leading global organizations that have unlocked value with C3 IoT and AWS. We also address how the C3 IoT platform has pre-integrated over 40 AWS services, enabling developers and data scientists to build and deploy enterprise scale big data, AI, and IoT applications in one-tenth of the time and cost. Session sponsored by C3 IoT, Inc.

JUMP13 Whitepapers Live: Mobile Innovation

Jamie Brighton

A new statistic or report seems to be released every day about device adoption, app downloads or the changing ways in which consumers are interacting on mobile channels. Behind these statistics are significant business challenges that also represent significant business opportunities for those brands and marketers that embrace mobile marketing. This session will explore those opportunities and look at some specific examples of companies that are innovating in the mobile channel - by providing uniquely mobile customer experiences that truly enhance the relationship that the customer has with the brand. It will also look at the tools that are available to help marketers address the mobile channel, from app creation and management to behavioural analytics and mobile content optimisation and targeting.

Value Added Services and WebRTC

Dialogic Inc.

The document discusses value-added services using WebRTC technology. It begins by outlining challenges currently facing enterprises and service providers, such as improving mobile experiences and reducing costs. It then examines how WebRTC could address these challenges by enabling new communication applications. Examples of potential WebRTC uses are presented across vertical industries like healthcare, IoT, and special needs, as well as for customer management, collaboration, and broadcasting. The document concludes by speculating on future directions such as using the data channel, moving beyond phone numbers with digital identity, and incorporating artificial intelligence into services.

Monitoring Serverless Applications (SRV303-S) - AWS re:Invent 2018

Amazon Web Services

Serverless brings many advantages to software development, but it introduces new monitoring challenges as well. Isolated telemetry on individual functions might not provide enough visibility, and instrumentation in a world where 100 ms of extra execution time could cost thousands of dollars might prove prohibitive. In this session, we explore how New Relic enables full observability of the serverless stack, including its executing context, with minimal impact in performance. Learn from customer case studies and real-world examples. This session is brought to you by AWS partner, New Relic.

iBeacons: Reality or Still a Work in Progress?

Ray Pun

AEM & Single Page Applications (SPAs) 101

Adobe

This document discusses single page applications (SPAs) and their relationship with content management systems (CMS) like Adobe Experience Manager (AEM). It provides background on the shift from traditional websites to SPAs, outlines benefits and challenges of SPAs for both IT and marketing teams, and describes how AEM 6.4 aims to support content delivery to various channels including SPAs through a decoupled architecture.

Automating Disaster Recovery for Faultless Service Delivery

CA Technologies

Accelerate Value from Big Data, AI, and IoT Initiatives with One-Tenth of the...

Amazon Web Services

Building Volkswagen Group's Digital Ecosystem (AMT304) - AWS re:Invent 2018

Amazon Web Services

Volkswagen has built a One Digital Platform (ODP) based on AWS to power its digital transformation and develop new mobility solutions. The ODP provides a flexible cloud-native architecture and will extend connectivity to vehicles. Volkswagen migrated over 100 applications to the ODP within 14 months to support new vehicle launches. Lessons learned included adopting an evolutionary approach, aligning platform development with application needs, investing in new skills, and gaining acceptance across functions. Next steps involve refactoring applications, modularizing code, and scaling the platform to support 15 million connected vehicles.

Automating the Modern Software Factory

CA Technologies

This session will provide an overview of how automation enables a Modern Software Factory to drive success in your digital transformation initiatives. From build to operate, automation from CA powers all stages of the development lifecycle, provides self-service dev and test environments while automating and optimizing operational processes. Listen to this presentation to get an understanding of how automation powers CA Test Data Manager, drives CA Service Virtualization, deploys APIs and can automatically remediate application performance issues with CA Application Performance Management. For more information on Automation, please visit: http://ow.ly/yVYf50fQDlB

<Programming> 2019 - ICW'19: The Issue of Monorepo and Polyrepo In Large Ente...

Nicolas Brousse

Paper presentation at ICW'19. Abstract: Product and engineering teams’ speed of producing high-quality results is critical to ensuring enterprise competitiveness. Additionally, one can observe an increase in IT systems complexity driven by the adoption of service-oriented architecture, micro-services, and serverless. Therefore, many large enterprises benefit from a mono-repository for source code management because of the improved team cognition that results from eroding barriers between teams and from influencing enhanced teamwork quality. This paper, first, reviews the characteristics of a multi-repositories structure, a monorepository structure, and a hybrid model. Second, it discusses why some manage source code in a multi-repositories structure, either by choice or because of the organic evolution of large enterprises. Third, it reviews how mono-repositories in large teams, beyond the technical arguments, can drive high efficiency and enhanced product quality through improved team cognition.

PuppetConf 2017 | Adobe Advertising Cloud: A Lean Puppet Workflow to Support ...

Nicolas Brousse

Building and scaling a multi-cloud solution that's enabled for cloud bursting is not a trivial task, and requires a lot of automation. While experiencing hyper-growth on the Adobe Advertising Cloud, our operations engineering team had to frequently update and improve its workflow in order to stay nimble and allow fast delivery of new infrastructure. At TubeMogul/Adobe Advertising Cloud, we implemented a lean Puppet workflow that enables the operations engineering team to deploy and support a broad range of services in a complex environment that supports hundreds of billions of requests a day. With over 150 changes released per day on its production infrastructure, the team had to adjust and tune its processes to enforce quality, standards, to review, and to prevent systems from breaking. In this talk, you will learn how we implemented our infrastructure as code by leveraging tools like Puppet, Gerrit, Terraform, and Jenkins, which together enable our private and public cloud infrastructures across 12 locations and four continents.

Similar to IEEE ISSRE 2018 - Use of Self-Healing Techniques to Improve the Reliability of a Dynamic and Geo-Distributed Ad Delivery Service

Adobe Ask the AEM Community Expert Session Oct 2016

AdobeMarketingCloud

Developer To Architect

Anurag Yadav

Design - Start Your API Journey Today

LaurenWendler

SAP TechEd 2010 Rich Internet Applications for the Enterprise

Anne Kathrine Petterøe

Adobe Flash Platform Summit 2010

Anne Kathrine Petterøe

Adobe Advertising Cloud: The Reality of Cloud Bursting with OpenStack

Nicolas Brousse

[NEW LAUNCH!] How to Architect for Multi-Region Redundancy Using Anycast IPs ...

Amazon Web Services

Marketing in the Age of Mobile

Adobe Experience Cloud

Where is cold fusion headed

ColdFusionConference

Design - Start Your API Journey Today

LaurenWendler

Accelerate Value from Big Data, AI, and IoT Initiatives with One-Tenth of the...

Amazon Web Services

JUMP13 Whitepapers Live: Mobile Innovation

Jamie Brighton

Value Added Services and WebRTC

Dialogic Inc.

Monitoring Serverless Applications (SRV303-S) - AWS re:Invent 2018

Amazon Web Services

iBeacons: Reality or Still a Work in Progress?

Ray Pun

AEM & Single Page Applications (SPAs) 101

Adobe

Automating Disaster Recovery for Faultless Service Delivery

CA Technologies

Accelerate Value from Big Data, AI, and IoT Initiatives with One-Tenth of the...

Amazon Web Services

Building Volkswagen Group's Digital Ecosystem (AMT304) - AWS re:Invent 2018

Amazon Web Services

Automating the Modern Software Factory

CA Technologies

Similar to IEEE ISSRE 2018 - Use of Self-Healing Techniques to Improve the Reliability of a Dynamic and Geo-Distributed Ad Delivery Service (20)

Adobe Ask the AEM Community Expert Session Oct 2016

Developer To Architect

Design - Start Your API Journey Today

SAP TechEd 2010 Rich Internet Applications for the Enterprise

Adobe Flash Platform Summit 2010

Adobe Advertising Cloud: The Reality of Cloud Bursting with OpenStack

[NEW LAUNCH!] How to Architect for Multi-Region Redundancy Using Anycast IPs ...

Marketing in the Age of Mobile

Where is cold fusion headed

Design - Start Your API Journey Today

Accelerate Value from Big Data, AI, and IoT Initiatives with One-Tenth of the...

JUMP13 Whitepapers Live: Mobile Innovation

Value Added Services and WebRTC

Monitoring Serverless Applications (SRV303-S) - AWS re:Invent 2018

iBeacons: Reality or Still a Work in Progress?

AEM & Single Page Applications (SPAs) 101

Automating Disaster Recovery for Faultless Service Delivery

Accelerate Value from Big Data, AI, and IoT Initiatives with One-Tenth of the...

Building Volkswagen Group's Digital Ecosystem (AMT304) - AWS re:Invent 2018

Automating the Modern Software Factory

More from Nicolas Brousse

<Programming> 2019 - ICW'19: The Issue of Monorepo and Polyrepo In Large Ente...

Nicolas Brousse

PuppetConf 2017 | Adobe Advertising Cloud: A Lean Puppet Workflow to Support ...

Nicolas Brousse

SuiteWorld16: Mega Volume - How TubeMogul Leverages NetSuite

Nicolas Brousse

TubeMogul is an enterprise software company for digital branding that leverages NetSuite to automate tracking and billing of its advertising campaigns. It exports campaign data hourly from its platform to NetSuite using custom APIs and Amazon SWF. At the end of each month, accounting verifies the ingested data and cuts invoices to enable teams to focus on growing revenue. Upaya developed custom RESTlet APIs to integrate NetSuite with TubeMogul's systems for customer, campaign, vendor and other data.

SRECon16: Moving Large Workloads from a Public Cloud to an OpenStack Private ...

Nicolas Brousse

It can be easy to come up with a TCO analysis that would challenge any public cloud and make you think, "let's go in-house!" What are the challenges and is it really worth it? The TubeMogul Operation team went thru the technical challenges at building a private cloud. In this presentation you will learn how the team went from a R&D to an automated deployment of a bare-metal servers to finally migrate a large workload from a Public Cloud to its own Private Cloud infrastructure. We will detail how the team dealt with unexpected issues and also how we chose the hardware, estimated capacity, stay cost effective, improve overall performance of the system, and bring better control and visibility. This talk will cover the technical detail of: * Evaluating OpenStack, Building and automating a CI environment for a mix of bare metal and cloud servers. * What are the network limitations of OpenStack and how we creatively leverage VLANs to handle large packet per seconds. * How to efficiently monitor your cloud infrastructure Find quickly your bottlenecks * What we missed and should be consider before moving in house Lesson Learned and Post Cost Analysis

USENIX LISA15: How TubeMogul Handles over One Trillion HTTP Requests a Month

Nicolas Brousse

TubeMogul grew from few servers to over two thousands servers and handling over one trillion http requests a month, processed in less than 50ms each. To keep up with the fast growth, the SRE team had to implement an efficient Continuous Delivery infrastructure that allowed to do over 10,000 puppet deployment and 8,500 application deployment in 2014. In this presentation, we will cover the nuts and bolts of the TubeMogul operations engineering team and how they overcome challenges.

Puppet Camp Silicon Valley 2015: How TubeMogul reached 10,000 Puppet Deployme...

Nicolas Brousse

Improving Operations Efficiency with Puppet

Nicolas Brousse

Scaling Bleeding Edge Technology in a Fast-paced Environment

Nicolas Brousse

The document discusses key principles for scaling bleeding edge technology in a fast-paced environment: be passionate about creating real customer value; be pragmatic and focus on solving problems where innovation is needed most; iterate often to refine solutions; and pay down technical debt through refactoring code as understanding improves. It provides quotes from entrepreneurs like Jeff Bezos and technologists like Paul Graham and Ward Cunningham supporting these principles. The document also briefly describes an automated video advertising software solution and its evolution through iterations and refactoring.

Scaling on EC2 in a fast-paced environment (LISA'11 - Full Paper)

Nicolas Brousse

Managing a server infrastructure in a fastpaced environment like a start-up is challenging. You have little time for provisioning, testing and planning but still you need to prepare for scaling when your product reaches the tipping point. Amazon EC2 is one of the cloud providers that we experimented with while growing our infrastructure from 20 servers to 500 servers. In this paper we will go over the pros and cons of managing EC2 instances with a mix of Bind, LDAP, SimpleDB and Python scripts; how we kept a smooth working process by using NFS, auto-mount and shell-scripting; why we switched from managing our instances based on tailor-made AMI/Shell-scripting to the official Ubuntu AMI, Cloud-init and puppet; and finally, we will go over some rules we had to follow carefully to be able to handle billions of daily non-static http request across multiple Amazon EC2 regions.

Bringing Business Awareness to Your Operation Team (Nagios World Conference 2...

Nicolas Brousse

Managing an infrastructure with a global footprint can be a challenging task for a centralized operation team. Bringing business awareness to your operation is key to success. In this talk you will learn how TubeMogul Operation Team easily manage its on-call rotation, how we centralize monitoring information from multiple datacenter for efficient on-call and how we define our business priority with a business focused dashboard.

Optimizing your Monitoring and Trending tools for the Cloud (Nagios World Con...

Nicolas Brousse

Nowadays most start-up are using cloud solution, while some will go from public cloud to hybrid solution, all have to deal with fast growing infrastructure. In this presentation, I will go over few solutions that we implemented at TubeMogul while growing from 20 servers to over 700 servers in 4 years and dealing with over 10 billions HTTP requests a day. With so many informations and data every day, it's hard to get a good read of what really matter and to alert the right person. You will learn how we integrated Nagios with Google Calendar for easy on-call rotation management, how we centralized our Nagios information from 5 different DC to a common dashboard, how we make daily maintenance report for pro-active action.

More from Nicolas Brousse (11)

<Programming> 2019 - ICW'19: The Issue of Monorepo and Polyrepo In Large Ente...

PuppetConf 2017 | Adobe Advertising Cloud: A Lean Puppet Workflow to Support ...

SuiteWorld16: Mega Volume - How TubeMogul Leverages NetSuite

SRECon16: Moving Large Workloads from a Public Cloud to an OpenStack Private ...

USENIX LISA15: How TubeMogul Handles over One Trillion HTTP Requests a Month

Puppet Camp Silicon Valley 2015: How TubeMogul reached 10,000 Puppet Deployme...

Improving Operations Efficiency with Puppet

Scaling Bleeding Edge Technology in a Fast-paced Environment

Scaling on EC2 in a fast-paced environment (LISA'11 - Full Paper)

Bringing Business Awareness to Your Operation Team (Nagios World Conference 2...

Optimizing your Monitoring and Trending tools for the Cloud (Nagios World Con...

Recently uploaded

Driving Business Innovation: Latest Generative AI Advancements & Success Story

Safe Software

Are you ready to revolutionize how you handle data? Join us for a webinar where we’ll bring you up to speed with the latest advancements in Generative AI technology and discover how leveraging FME with tools from giants like Google Gemini, Amazon, and Microsoft OpenAI can supercharge your workflow efficiency. During the hour, we’ll take you through: Guest Speaker Segment with Hannah Barrington: Dive into the world of dynamic real estate marketing with Hannah, the Marketing Manager at Workspace Group. Hear firsthand how their team generates engaging descriptions for thousands of office units by integrating diverse data sources—from PDF floorplans to web pages—using FME transformers, like OpenAIVisionConnector and AnthropicVisionConnector. This use case will show you how GenAI can streamline content creation for marketing across the board. Ollama Use Case: Learn how Scenario Specialist Dmitri Bagh has utilized Ollama within FME to input data, create custom models, and enhance security protocols. This segment will include demos to illustrate the full capabilities of FME in AI-driven processes. Custom AI Models: Discover how to leverage FME to build personalized AI models using your data. Whether it’s populating a model with local data for added security or integrating public AI tools, find out how FME facilitates a versatile and secure approach to AI. We’ll wrap up with a live Q&A session where you can engage with our experts on your specific use cases, and learn more about optimizing your data workflows with AI. This webinar is ideal for professionals seeking to harness the power of AI within their data management systems while ensuring high levels of customization and security. Whether you're a novice or an expert, gain actionable insights and strategies to elevate your data processes. Join us to see how FME and AI can revolutionize how you work with data!

Things to Consider When Choosing a Website Developer for your Website | FODUU

FODUU

Infrastructure Challenges in Scaling RAG with Custom AI models

Zilliz

Building Retrieval-Augmented Generation (RAG) systems with open-source and custom AI models is a complex task. This talk explores the challenges in productionizing RAG systems, including retrieval performance, response synthesis, and evaluation. We’ll discuss how to leverage open-source models like text embeddings, language models, and custom fine-tuned models to enhance RAG performance. Additionally, we’ll cover how BentoML can help orchestrate and scale these AI components efficiently, ensuring seamless deployment and management of RAG systems in the cloud.

Cosa hanno in comune un mattoncino Lego e la backdoor XZ?

Speck&Tech

ABSTRACT: A prima vista, un mattoncino Lego e la backdoor XZ potrebbero avere in comune il fatto di essere entrambi blocchi di costruzione, o dipendenze di progetti creativi e software. La realtà è che un mattoncino Lego e il caso della backdoor XZ hanno molto di più di tutto ciò in comune. Partecipate alla presentazione per immergervi in una storia di interoperabilità, standard e formati aperti, per poi discutere del ruolo importante che i contributori hanno in una comunità open source sostenibile. BIO: Sostenitrice del software libero e dei formati standard e aperti. È stata un membro attivo dei progetti Fedora e openSUSE e ha co-fondato l'Associazione LibreItalia dove è stata coinvolta in diversi eventi, migrazioni e formazione relativi a LibreOffice. In precedenza ha lavorato a migrazioni e corsi di formazione su LibreOffice per diverse amministrazioni pubbliche e privati. Da gennaio 2020 lavora in SUSE come Software Release Engineer per Uyuni e SUSE Manager e quando non segue la sua passione per i computer e per Geeko coltiva la sua curiosità per l'astronomia (da cui deriva il suo nickname deneb_alpha).

Your One-Stop Shop for Python Success: Top 10 US Python Development Providers

akankshawande

Programming Foundation Models with DSPy - Meetup Slides

Zilliz

AI-Powered Food Delivery Transforming App Development in Saudi Arabia.pdf

Techgropse Pvt.Ltd.

In this blog post, we'll delve into the intersection of AI and app development in Saudi Arabia, focusing on the food delivery sector. We'll explore how AI is revolutionizing the way Saudi consumers order food, how restaurants manage their operations, and how delivery partners navigate the bustling streets of cities like Riyadh, Jeddah, and Dammam. Through real-world case studies, we'll showcase how leading Saudi food delivery apps are leveraging AI to redefine convenience, personalization, and efficiency.

AI 101: An Introduction to the Basics and Impact of Artificial Intelligence

IndexBug

Monitoring and Managing Anomaly Detection on OpenShift.pdf

Tosin Akinosho

Monitoring and Managing Anomaly Detection on OpenShift Overview Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices. Key Topics Covered 1. Introduction to Anomaly Detection - Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems. 2. Understanding Edge (IoT) - Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source. 3. What is ArgoCD? - Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices. 4. Deployment Using ArgoCD for Edge Devices - Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD. 5. Introduction to Apache Kafka and S3 - Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions. 6. Viewing Kafka Messages in the Data Lake - Learn how to view and analyze Kafka messages stored in a data lake for better insights. 7. What is Prometheus? - Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices. 8. Monitoring Application Metrics with Prometheus - Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system. 9. What is Camel K? - Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes. 10. Configuring Camel K Integrations for Data Pipelines - Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow. 11. What is a Jupyter Notebook? - Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text. 12. Jupyter Notebooks with Code Examples - Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.

Climate Impact of Software Testing at Nordic Testing Days

Kari Kakkonen

My slides at Nordic Testing Days 6.6.2024 Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.

Full-RAG: A modern architecture for hyper-personalization

Zilliz

Mike Del Balso, CEO & Co-Founder at Tecton, presents "Full RAG," a novel approach to AI recommendation systems, aiming to push beyond the limitations of traditional models through a deep integration of contextual insights and real-time data, leveraging the Retrieval-Augmented Generation architecture. This talk will outline Full RAG's potential to significantly enhance personalization, address engineering challenges such as data management and model training, and introduce data enrichment with reranking as a key solution. Attendees will gain crucial insights into the importance of hyperpersonalization in AI, the capabilities of Full RAG for advanced personalization, and strategies for managing complex data integrations for deploying cutting-edge AI solutions.

Microsoft - Power Platform_G.Aspiotis.pdf

Uni Systems S.M.S.A.

Removing Uninteresting Bytes in Software Fuzzing

Aftab Hussain

Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process. In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds. - These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.

“I’m still / I’m still / Chaining from the Block”

Claudio Di Ciccio

Choosing The Best AWS Service For Your Website + API.pptx

Brandon Minnick, MBA

Have you ever been confused by the myriad of choices offered by AWS for hosting a website or an API? Lambda, Elastic Beanstalk, Lightsail, Amplify, S3 (and more!) can each host websites + APIs. But which one should we choose? Which one is cheapest? Which one is fastest? Which one will scale to meet our needs? Join me in this session as we dive into each AWS hosting service to determine which one is best for your scenario and explain why!

20240609 QFM020 Irresponsible AI Reading List May 2024

Matthew Sinclair

UI5 Controls simplified - UI5con2024 presentation

Wouter Lemaire

Video Streaming: Then, Now, and in the Future

Alpen-Adria-Universität

In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.

Fueling AI with Great Data with Airbyte Webinar

Zilliz

HCL Notes and Domino License Cost Reduction in the World of DLAU

panagenda

Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/ The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this! We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model. Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward. These topics will be covered - Reducing license cost by finding and fixing misconfigurations and superfluous accounts - How do CCB and CCX licenses really work? - Understanding the DLAU tool and how to best utilize it - Tips for common problem areas, like team mailboxes, functional/test users, etc - Practical examples and best practices to implement right away

Recently uploaded (20)

Driving Business Innovation: Latest Generative AI Advancements & Success Story

Things to Consider When Choosing a Website Developer for your Website | FODUU

Infrastructure Challenges in Scaling RAG with Custom AI models

Cosa hanno in comune un mattoncino Lego e la backdoor XZ?

Your One-Stop Shop for Python Success: Top 10 US Python Development Providers

Programming Foundation Models with DSPy - Meetup Slides

AI-Powered Food Delivery Transforming App Development in Saudi Arabia.pdf

AI 101: An Introduction to the Basics and Impact of Artificial Intelligence

Monitoring and Managing Anomaly Detection on OpenShift.pdf

Climate Impact of Software Testing at Nordic Testing Days

Full-RAG: A modern architecture for hyper-personalization

Microsoft - Power Platform_G.Aspiotis.pdf

Removing Uninteresting Bytes in Software Fuzzing

“I’m still / I’m still / Chaining from the Block”

Choosing The Best AWS Service For Your Website + API.pptx

20240609 QFM020 Irresponsible AI Reading List May 2024

UI5 Controls simplified - UI5con2024 presentation

Video Streaming: Then, Now, and in the Future

Fueling AI with Great Data with Airbyte Webinar

HCL Notes and Domino License Cost Reduction in the World of DLAU

IEEE ISSRE 2018 - Use of Self-Healing Techniques to Improve the Reliability of a Dynamic and Geo-Distributed Ad Delivery Service

1. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Use of Self-Healing Techniques to Improve the Reliability of a Dynamic and Geo-Distributed Ad Delivery Service Nicolas Brousse and Oleksii Mykhailov

7. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Inconsistent GeoDNS Routing High Latency From Eyeball To Content Origin Origin Failure Impact User Experience Impact Campaign Performance and Revenue 7

14. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 14 Few Words About Anycast § Shortest Path Routing Means § Not Latency Aware § Not Congestion Aware / Packet Loss § Limited Control for Traffic Steering § Difficult Troubleshooting § Failover lead to packet RST for Active Sessions § Mitigation with a large and well distributed number of POPs

18. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Human Failover vs Self-Healing 18 > 1h Failure < 15min < 30min Region Traffic Rerouted Self Recover Note: since the paper publication, we reduced automated failover time to be less than a few seconds. See demo. Fig. 1 Human Failover with Manual Recovery steps Fig. 2 Automated Failover and Self-Healing Recovery

IEEE ISSRE 2018 - Use of Self-Healing Techniques to Improve the Reliability of a Dynamic and Geo-Distributed Ad Delivery Service

Recommended

Recommended

More Related Content

Similar to IEEE ISSRE 2018 - Use of Self-Healing Techniques to Improve the Reliability of a Dynamic and Geo-Distributed Ad Delivery Service

Similar to IEEE ISSRE 2018 - Use of Self-Healing Techniques to Improve the Reliability of a Dynamic and Geo-Distributed Ad Delivery Service (20)

More from Nicolas Brousse

More from Nicolas Brousse (11)

Recently uploaded

Recently uploaded (20)

IEEE ISSRE 2018 - Use of Self-Healing Techniques to Improve the Reliability of a Dynamic and Geo-Distributed Ad Delivery Service