IEEE ISSRE 2018 - Use of Self-Healing Techniques to Improve the Reliability of a Dynamic and Geo-Distributed Ad Delivery Service

•

2 likes•430 views

The advertising industry faces numerous challenges in achieving its goal of targeting a given audience dynamically and accurately in order to deliver a meaningful brand message. Near real-time, low latency delivery of dynamic content, the sheer volume of information processed, and the sparse geographic distribution of the intended eyeball traffic all drive the complexity of building a successful experience for the end user and the brand. Additionally, the competitiveness of the industry makes it critical to preserve low operational expenses while delivering reliably at scale. In attempting to address the above, we have found that a distributed infrastructure that leverages public cloud providers and a private cloud with open infrastructure technologies can deliver dynamic advertising content with low latency while preserving its high availability. But network or physical utility infrastructures can’t be relied on to ensure the service dependability. We show that the complexity of the networks, the sparse geographic distribution of eyeballs, the risk of data center failures, and the increase of encrypted transactions call for thoughtful architectures. The introduction of modern practices, failure injections, and self-healing mechanisms allowed us to improve the service fault tolerance while optimizing for latency and significantly improve our service reliability.

Technology

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Use of Self-Healing Techniques to Improve the Reliability of a Dynamic
and Geo-Distributed Ad Delivery Service
Nicolas Brousse and Oleksii Mykhailov

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Adobe Advertising Cloud
Serving All Media Content Across
Any Screens in Any Format
2

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 3
BEFORE
RFP, IO, human based orders
NOW
Programmatic Ad Buying with
Real Time Bidding

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 4
Latency
<50ms @ 95th percentile
High Traffic
300 billion requests a day
Huge Datasets
Billions of objects to store

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 5
Ad Content Delivered To Eyeball

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 6
Traditional Ad
Serving Implement
GeoDNS GSLB

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Inconsistent GeoDNS Routing
High Latency From Eyeball To Content Origin
Origin Failure Impact User Experience
Impact Campaign Performance and Revenue
7

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Relying on GeoDNS to Figure Out Eyeball Location is
UNRELIABLE
8
Optimal Route
Actual Route

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 9
TCP and TLS
Handshake Impact
Latency

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Datacenter Blackout
Network Outage
Human Errors
Natural Disaster
10
High Risks Of Origin Failures

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Service Unavailability
11

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 12
SOLUTION
Eyeball Traffic Access Content via Smart Edges

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 13
Smart Edges Are
Anycast POPs That
Manage Failover and
Self-Healing

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 14
Few Words About Anycast
§ Shortest Path Routing Means
§ Not Latency Aware
§ Not Congestion Aware / Packet Loss
§ Limited Control for Traffic Steering
§ Difficult Troubleshooting
§ Failover lead to packet RST for Active Sessions
§ Mitigation with a large and well distributed number of POPs

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 15
Anycast POP
Improve Latency
e.g. 3X Faster

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 16
Automate
Failover and Recovery

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 17
Inject Failures In
Production To Validate
Smart Edges Behavior

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Human Failover vs Self-Healing
18
> 1h Failure
< 15min
< 30min Region
Traffic Rerouted
Self Recover
Note: since the paper publication, we reduced automated failover time to be less than a few seconds. See demo.
Fig. 1 Human Failover with Manual Recovery steps Fig. 2 Automated Failover and Self-Healing Recovery

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 19
Injecting Complete
Data Center Failure
at the Regional Level
LIVE

© 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Recorded Demo
20

IEEE ISSRE 2018 - Use of Self-Healing Techniques to Improve the Reliability of a Dynamic and Geo-Distributed Ad Delivery Service

The document discusses several key facts about A2P messaging: 1) 95% of A2P traffic is generated by over-the-top (OTT) applications; 2) two-factor authentication (2FA) messages form the main part of monetizable traffic; 3) local A2P traffic should also be monitored, as large volumes may be leaking over unauthorized "grey routes"; 4) domestic A2P revenues can be lost if international grey route prices are lower than domestic rates, even though domestic volumes are often 10 times higher. The document promotes contacting experts at haud.com to learn more about A2P messaging solutions.

Visualizing Cellular Audience for Streaming KPI's

Akamai Developers & Admins

This document discusses visualizing streaming performance data for cellular audiences. It notes that more than 75% of viewers now stream from mobile networks. The document outlines how Akamai's Media Analytics and Network Experience Analytics tools can provide granular performance data and metrics on a cellular level, including by cell ID, operator, and location. This level of data insights allows content providers to optimize their streaming for mobile audiences and cellular networks.

Improving Adobe Experience Cloud Services Dependability with Machine Learning

Nicolas Brousse

Adobe Experience Cloud is a collection of best-in-class solutions for marketing, analytics, advertising, and commerce. All integrated on a cloud platform for a single experience system of record. The Adobe Experience Cloud's SRE team works hand-in-hand with the Product and Engineering teams to build dependable services. In this presentation, you will learn how the team leverage Adobe's artificial intelligence and machine learning engine to build predictive auto-scaling and self-healing services.

Using AEM in a customer global multi-channel program

Meryll Blanchet

This document discusses using Adobe Experience Manager (AEM) in a global multi-channel program. It begins with an overview and agenda. It then discusses a real-life example program involving CRM, web, email, mobile, and analytics systems. It focuses on how AEM was used, including user authentication via SAML, personalizing content, and integrating with Adobe Target and Analytics. Key challenges discussed are managing expectations, careful planning and estimation, and prioritizing analytics requirements.

When Content Meets Data, Big Things Happen - Peter Krmpotic, Adobe

NewsCred

The document discusses Adobe's content marketing solution. It describes how content explosion has created challenges for marketers. Adobe's solution addresses these challenges through a comprehensive content marketing process that includes audience segmentation, content strategy, curation, composition, delivery, and amplification across channels. It leverages Adobe Experience Manager for content optimization and integration with the Adobe Marketing Cloud.

Altitude San Francisco 2018: How Magento moved to the cloud while maintaining...

Fastly

Magento Commerce was first released by a small web development agency over ten years when they saw first-hand what a challenge it was for companies like them to build unique eCommerce sites. They created an open source platform that gives developers the flexibility to create meaningful shopping experiences while building a global community that drives down merchant costs and fosters innovation. Amid the rise of cloud-based software Magento needed to keep pace with more complex merchant needs and heightened shopper expectations. In this session learn how Magento, with the help of Partners like Fastly, evolved into a cloud-based platform without sacrificing their commitment to open software, flexibility, and the community.

12/6 Initiative

Rob Healy

The document proposes an initiative called the 12/6 Initiative to reduce wasted energy from always-on web servers. It suggests powering down servers outside of peak traffic hours, which could save 57% of wasted energy costs and reduce carbon emissions by 19.9 million tons per year if 10% of servers worldwide adopted this approach. In addition to energy savings, turning servers off during low traffic periods could also reduce the attack surface for hackers, lowering security risks. The document outlines the business and technical feasibility of automating backup and restore to enable safely powering down servers when not needed.

IMMERSE 2016 IST Mark Szulc Keynote

AdobeMarketingCloud

Two large enterprise AEM implementations were presented and compared. Anshul Chhabra from Symantec presented their implementation handling 3.3 billion requests per month. Anil Kalbag from Cisco presented their implementation handling 375 million monthly page views. Both implementations utilized multiple data centers for high availability and disaster recovery. Key architecture decisions around virtual/physical infrastructure, storage, caching, and multi-tenancy were discussed and compared between the two organizations.

Developer To Architect

Anurag Yadav

Design - Start Your API Journey Today

LaurenWendler

This document discusses how APIs can power the API economy and digital transformation. It describes how APIs can be used to expose functionality from various systems and data sources, and how API management is needed to securely manage APIs. The document promotes IBM API Connect as a way to create, manage and secure APIs and microservices across hybrid cloud environments to drive innovation. It provides links to learn more about API Connect and digital transformation workshops.

SAP TechEd 2010 Rich Internet Applications for the Enterprise

Anne Kathrine Petterøe

The document discusses rich internet applications for enterprises. It describes the challenges of enterprise application development and the importance of intuitive user experiences. Examples of different types of rich internet applications are provided. The document also discusses Adobe LiveCycle and its integration with SAP applications, allowing for improved user-centric experiences over traditional system-centric applications. A case study of a project called Hendrix is presented, which used these technologies to optimize processes and improve the user experience for an Adobe call center platform.

Adobe Flash Platform Summit 2010

Anne Kathrine Petterøe

The document discusses rich internet applications (RIAs) for enterprises. It covers the challenges of enterprise application development, how intuitive user experiences can help address those challenges, and how RIA technologies from Adobe like Flash, Flex and LiveCycle can be used to create more engaging user-centric applications. It also provides examples of how Adobe technologies have been used by companies like SAP, Orange and Verizon to build applications that integrate with backend systems and improve customer experiences. Specific integration options for building RIAs with SAP systems using technologies like Flash Islands and LiveCycle Data Services are also examined.

Adobe Advertising Cloud: The Reality of Cloud Bursting with OpenStack

Nicolas Brousse

After successfully moving a large workload from a Public Cloud to an OpenStack Private Cloud, the former TubeMogul Operations Engineering team tackled its next important step toward Cloud Bursting. While experiencing hyper-growth on the Adobe Advertising Cloud, the team had to figure out a simple way to quickly provision new compute resources. Our latency critical workload need our core private resources while some workload can safely leverage public cloud. Cloud Bursting helped to ensure rapid support of the business and provide a more flexible capacity planning strategy. Being able to burst some workload back to the Public Cloud allowed the team to leverage the best of public and private cloud. In this presentation, we will detail how the team dealt with the network complexity between clouds infrastructure, the cost effectiveness of cloud bursting, the impact on performance, and the hard reality of multi-cloud environments.

[NEW LAUNCH!] How to Architect for Multi-Region Redundancy Using Anycast IPs ...

Amazon Web Services

Deployed globally in multiple edge locations, AWS Global Accelerator helps you manage traffic destined to your multi-regional applications with further higher levels of availability and performance. This session comprises ways in which Ubiquity helps you build fault tolerant and highly performant systems across AWS regions using anycast static IP addresses. In this session, you will learn about Global Accelerator’s shuffle sharding technique used for its static IPs, benefits of anycast and more.

Marketing in the Age of Mobile

Adobe Experience Cloud

Mobile usage is growing rapidly, with people checking their phones hundreds of times per day. Many companies have rushed to create mobile apps but struggle with user acquisition, analytics, and iteration. Adobe's Project Fast Track created a unified solution across its mobile apps and Creative Cloud to gather usage data and enable data-driven product improvements. The project team integrated Adobe's Mobile SDK and Analytics to provide insights from nine apps in just nine weeks. This system allows Adobe to better understand user behavior and quickly iterate apps to increase engagement and subscriptions.

Where is cold fusion headed

ColdFusionConference

The document discusses the future direction of the ColdFusion product. It outlines plans to modernize the language with improved object oriented support, CFScript 2.0, support for null values, and multi-threading. It also discusses leveraging containerization with Docker, provisioning a microservices architecture, embracing devsecops practices like a built-in web application firewall, and accelerating digital transformation with enhanced API management capabilities. The goal is to implement a modern platform for building and deploying applications.

Design - Start Your API Journey Today

LaurenWendler

This document discusses the opportunities presented by the API economy. It describes how APIs can enable new experiences like pre-ordering food from a restaurant app before arriving. The document outlines challenges of managing APIs at scale across hybrid cloud environments. It introduces IBM API Connect as a way to create, manage and secure APIs throughout their lifecycle to power digital transformations. Finally, it provides links to learn more about API Connect and taking advantage of IBM's cloud offerings.

Accelerate Value from Big Data, AI, and IoT Initiatives with One-Tenth of the...

Amazon Web Services

Many organizations are awash in different types of data, yet they struggle to rapidly use these assets to improve operations and benefit customers. In this session, we explore best practices from deployments at leading global organizations that have unlocked value with C3 IoT and AWS. We also address how the C3 IoT platform has pre-integrated over 40 AWS services, enabling developers and data scientists to build and deploy enterprise scale big data, AI, and IoT applications in one-tenth of the time and cost. Session sponsored by C3 IoT, Inc.

JUMP13 Whitepapers Live: Mobile Innovation

Jamie Brighton

A new statistic or report seems to be released every day about device adoption, app downloads or the changing ways in which consumers are interacting on mobile channels. Behind these statistics are significant business challenges that also represent significant business opportunities for those brands and marketers that embrace mobile marketing. This session will explore those opportunities and look at some specific examples of companies that are innovating in the mobile channel - by providing uniquely mobile customer experiences that truly enhance the relationship that the customer has with the brand. It will also look at the tools that are available to help marketers address the mobile channel, from app creation and management to behavioural analytics and mobile content optimisation and targeting.

Value Added Services and WebRTC

Dialogic Inc.

The document discusses value-added services using WebRTC technology. It begins by outlining challenges currently facing enterprises and service providers, such as improving mobile experiences and reducing costs. It then examines how WebRTC could address these challenges by enabling new communication applications. Examples of potential WebRTC uses are presented across vertical industries like healthcare, IoT, and special needs, as well as for customer management, collaboration, and broadcasting. The document concludes by speculating on future directions such as using the data channel, moving beyond phone numbers with digital identity, and incorporating artificial intelligence into services.

Monitoring Serverless Applications (SRV303-S) - AWS re:Invent 2018

Amazon Web Services

Serverless brings many advantages to software development, but it introduces new monitoring challenges as well. Isolated telemetry on individual functions might not provide enough visibility, and instrumentation in a world where 100 ms of extra execution time could cost thousands of dollars might prove prohibitive. In this session, we explore how New Relic enables full observability of the serverless stack, including its executing context, with minimal impact in performance. Learn from customer case studies and real-world examples. This session is brought to you by AWS partner, New Relic.

iBeacons: Reality or Still a Work in Progress?

Ray Pun

AEM & Single Page Applications (SPAs) 101

Adobe

This document discusses single page applications (SPAs) and their relationship with content management systems (CMS) like Adobe Experience Manager (AEM). It provides background on the shift from traditional websites to SPAs, outlines benefits and challenges of SPAs for both IT and marketing teams, and describes how AEM 6.4 aims to support content delivery to various channels including SPAs through a decoupled architecture.

Automating Disaster Recovery for Faultless Service Delivery

CA Technologies

Accelerate Value from Big Data, AI, and IoT Initiatives with One-Tenth of the...

Amazon Web Services

Building Volkswagen Group's Digital Ecosystem (AMT304) - AWS re:Invent 2018

Amazon Web Services

Volkswagen has built a One Digital Platform (ODP) based on AWS to power its digital transformation and develop new mobility solutions. The ODP provides a flexible cloud-native architecture and will extend connectivity to vehicles. Volkswagen migrated over 100 applications to the ODP within 14 months to support new vehicle launches. Lessons learned included adopting an evolutionary approach, aligning platform development with application needs, investing in new skills, and gaining acceptance across functions. Next steps involve refactoring applications, modularizing code, and scaling the platform to support 15 million connected vehicles.

Automating the Modern Software Factory

CA Technologies

This session will provide an overview of how automation enables a Modern Software Factory to drive success in your digital transformation initiatives. From build to operate, automation from CA powers all stages of the development lifecycle, provides self-service dev and test environments while automating and optimizing operational processes. Listen to this presentation to get an understanding of how automation powers CA Test Data Manager, drives CA Service Virtualization, deploys APIs and can automatically remediate application performance issues with CA Application Performance Management. For more information on Automation, please visit: http://ow.ly/yVYf50fQDlB

<Programming> 2019 - ICW'19: The Issue of Monorepo and Polyrepo In Large Ente...

Nicolas Brousse

Paper presentation at ICW'19. Abstract: Product and engineering teams’ speed of producing high-quality results is critical to ensuring enterprise competitiveness. Additionally, one can observe an increase in IT systems complexity driven by the adoption of service-oriented architecture, micro-services, and serverless. Therefore, many large enterprises benefit from a mono-repository for source code management because of the improved team cognition that results from eroding barriers between teams and from influencing enhanced teamwork quality. This paper, first, reviews the characteristics of a multi-repositories structure, a monorepository structure, and a hybrid model. Second, it discusses why some manage source code in a multi-repositories structure, either by choice or because of the organic evolution of large enterprises. Third, it reviews how mono-repositories in large teams, beyond the technical arguments, can drive high efficiency and enhanced product quality through improved team cognition.

PuppetConf 2017 | Adobe Advertising Cloud: A Lean Puppet Workflow to Support ...

Nicolas Brousse

Building and scaling a multi-cloud solution that's enabled for cloud bursting is not a trivial task, and requires a lot of automation. While experiencing hyper-growth on the Adobe Advertising Cloud, our operations engineering team had to frequently update and improve its workflow in order to stay nimble and allow fast delivery of new infrastructure. At TubeMogul/Adobe Advertising Cloud, we implemented a lean Puppet workflow that enables the operations engineering team to deploy and support a broad range of services in a complex environment that supports hundreds of billions of requests a day. With over 150 changes released per day on its production infrastructure, the team had to adjust and tune its processes to enforce quality, standards, to review, and to prevent systems from breaking. In this talk, you will learn how we implemented our infrastructure as code by leveraging tools like Puppet, Gerrit, Terraform, and Jenkins, which together enable our private and public cloud infrastructures across 12 locations and four continents.

Similar to IEEE ISSRE 2018 - Use of Self-Healing Techniques to Improve the Reliability of a Dynamic and Geo-Distributed Ad Delivery Service

Adobe Ask the AEM Community Expert Session Oct 2016

AdobeMarketingCloud

Developer To Architect

Anurag Yadav

Design - Start Your API Journey Today

LaurenWendler

SAP TechEd 2010 Rich Internet Applications for the Enterprise

Anne Kathrine Petterøe

Adobe Flash Platform Summit 2010

Anne Kathrine Petterøe

Adobe Advertising Cloud: The Reality of Cloud Bursting with OpenStack

Nicolas Brousse

[NEW LAUNCH!] How to Architect for Multi-Region Redundancy Using Anycast IPs ...

Amazon Web Services

Marketing in the Age of Mobile

Adobe Experience Cloud

Where is cold fusion headed

ColdFusionConference

Design - Start Your API Journey Today

LaurenWendler

Accelerate Value from Big Data, AI, and IoT Initiatives with One-Tenth of the...

Amazon Web Services

JUMP13 Whitepapers Live: Mobile Innovation

Jamie Brighton

Value Added Services and WebRTC

Dialogic Inc.

Monitoring Serverless Applications (SRV303-S) - AWS re:Invent 2018

Amazon Web Services

iBeacons: Reality or Still a Work in Progress?

Ray Pun

AEM & Single Page Applications (SPAs) 101

Adobe

Automating Disaster Recovery for Faultless Service Delivery

CA Technologies

Accelerate Value from Big Data, AI, and IoT Initiatives with One-Tenth of the...

Amazon Web Services

Building Volkswagen Group's Digital Ecosystem (AMT304) - AWS re:Invent 2018

Amazon Web Services

Automating the Modern Software Factory

CA Technologies

Similar to IEEE ISSRE 2018 - Use of Self-Healing Techniques to Improve the Reliability of a Dynamic and Geo-Distributed Ad Delivery Service (20)

Adobe Ask the AEM Community Expert Session Oct 2016

Developer To Architect

Design - Start Your API Journey Today

SAP TechEd 2010 Rich Internet Applications for the Enterprise

Adobe Flash Platform Summit 2010

Adobe Advertising Cloud: The Reality of Cloud Bursting with OpenStack

[NEW LAUNCH!] How to Architect for Multi-Region Redundancy Using Anycast IPs ...

Marketing in the Age of Mobile

Where is cold fusion headed

Design - Start Your API Journey Today

Accelerate Value from Big Data, AI, and IoT Initiatives with One-Tenth of the...

JUMP13 Whitepapers Live: Mobile Innovation

Value Added Services and WebRTC

Monitoring Serverless Applications (SRV303-S) - AWS re:Invent 2018

iBeacons: Reality or Still a Work in Progress?

AEM & Single Page Applications (SPAs) 101

Automating Disaster Recovery for Faultless Service Delivery

Accelerate Value from Big Data, AI, and IoT Initiatives with One-Tenth of the...

Building Volkswagen Group's Digital Ecosystem (AMT304) - AWS re:Invent 2018

Automating the Modern Software Factory

More from Nicolas Brousse

<Programming> 2019 - ICW'19: The Issue of Monorepo and Polyrepo In Large Ente...

Nicolas Brousse

PuppetConf 2017 | Adobe Advertising Cloud: A Lean Puppet Workflow to Support ...

Nicolas Brousse

SuiteWorld16: Mega Volume - How TubeMogul Leverages NetSuite

Nicolas Brousse

TubeMogul is an enterprise software company for digital branding that leverages NetSuite to automate tracking and billing of its advertising campaigns. It exports campaign data hourly from its platform to NetSuite using custom APIs and Amazon SWF. At the end of each month, accounting verifies the ingested data and cuts invoices to enable teams to focus on growing revenue. Upaya developed custom RESTlet APIs to integrate NetSuite with TubeMogul's systems for customer, campaign, vendor and other data.

SRECon16: Moving Large Workloads from a Public Cloud to an OpenStack Private ...

Nicolas Brousse

It can be easy to come up with a TCO analysis that would challenge any public cloud and make you think, "let's go in-house!" What are the challenges and is it really worth it? The TubeMogul Operation team went thru the technical challenges at building a private cloud. In this presentation you will learn how the team went from a R&D to an automated deployment of a bare-metal servers to finally migrate a large workload from a Public Cloud to its own Private Cloud infrastructure. We will detail how the team dealt with unexpected issues and also how we chose the hardware, estimated capacity, stay cost effective, improve overall performance of the system, and bring better control and visibility. This talk will cover the technical detail of: * Evaluating OpenStack, Building and automating a CI environment for a mix of bare metal and cloud servers. * What are the network limitations of OpenStack and how we creatively leverage VLANs to handle large packet per seconds. * How to efficiently monitor your cloud infrastructure Find quickly your bottlenecks * What we missed and should be consider before moving in house Lesson Learned and Post Cost Analysis

USENIX LISA15: How TubeMogul Handles over One Trillion HTTP Requests a Month

Nicolas Brousse

TubeMogul grew from few servers to over two thousands servers and handling over one trillion http requests a month, processed in less than 50ms each. To keep up with the fast growth, the SRE team had to implement an efficient Continuous Delivery infrastructure that allowed to do over 10,000 puppet deployment and 8,500 application deployment in 2014. In this presentation, we will cover the nuts and bolts of the TubeMogul operations engineering team and how they overcome challenges.

Puppet Camp Silicon Valley 2015: How TubeMogul reached 10,000 Puppet Deployme...

Nicolas Brousse

Improving Operations Efficiency with Puppet

Nicolas Brousse

Scaling Bleeding Edge Technology in a Fast-paced Environment

Nicolas Brousse

The document discusses key principles for scaling bleeding edge technology in a fast-paced environment: be passionate about creating real customer value; be pragmatic and focus on solving problems where innovation is needed most; iterate often to refine solutions; and pay down technical debt through refactoring code as understanding improves. It provides quotes from entrepreneurs like Jeff Bezos and technologists like Paul Graham and Ward Cunningham supporting these principles. The document also briefly describes an automated video advertising software solution and its evolution through iterations and refactoring.

Scaling on EC2 in a fast-paced environment (LISA'11 - Full Paper)

Nicolas Brousse

Managing a server infrastructure in a fastpaced environment like a start-up is challenging. You have little time for provisioning, testing and planning but still you need to prepare for scaling when your product reaches the tipping point. Amazon EC2 is one of the cloud providers that we experimented with while growing our infrastructure from 20 servers to 500 servers. In this paper we will go over the pros and cons of managing EC2 instances with a mix of Bind, LDAP, SimpleDB and Python scripts; how we kept a smooth working process by using NFS, auto-mount and shell-scripting; why we switched from managing our instances based on tailor-made AMI/Shell-scripting to the official Ubuntu AMI, Cloud-init and puppet; and finally, we will go over some rules we had to follow carefully to be able to handle billions of daily non-static http request across multiple Amazon EC2 regions.

Bringing Business Awareness to Your Operation Team (Nagios World Conference 2...

Nicolas Brousse

Managing an infrastructure with a global footprint can be a challenging task for a centralized operation team. Bringing business awareness to your operation is key to success. In this talk you will learn how TubeMogul Operation Team easily manage its on-call rotation, how we centralize monitoring information from multiple datacenter for efficient on-call and how we define our business priority with a business focused dashboard.

Optimizing your Monitoring and Trending tools for the Cloud (Nagios World Con...

Nicolas Brousse

Nowadays most start-up are using cloud solution, while some will go from public cloud to hybrid solution, all have to deal with fast growing infrastructure. In this presentation, I will go over few solutions that we implemented at TubeMogul while growing from 20 servers to over 700 servers in 4 years and dealing with over 10 billions HTTP requests a day. With so many informations and data every day, it's hard to get a good read of what really matter and to alert the right person. You will learn how we integrated Nagios with Google Calendar for easy on-call rotation management, how we centralized our Nagios information from 5 different DC to a common dashboard, how we make daily maintenance report for pro-active action.

More from Nicolas Brousse (11)

<Programming> 2019 - ICW'19: The Issue of Monorepo and Polyrepo In Large Ente...

PuppetConf 2017 | Adobe Advertising Cloud: A Lean Puppet Workflow to Support ...

SuiteWorld16: Mega Volume - How TubeMogul Leverages NetSuite

SRECon16: Moving Large Workloads from a Public Cloud to an OpenStack Private ...

USENIX LISA15: How TubeMogul Handles over One Trillion HTTP Requests a Month

Puppet Camp Silicon Valley 2015: How TubeMogul reached 10,000 Puppet Deployme...

Improving Operations Efficiency with Puppet

Scaling Bleeding Edge Technology in a Fast-paced Environment

Scaling on EC2 in a fast-paced environment (LISA'11 - Full Paper)

Bringing Business Awareness to Your Operation Team (Nagios World Conference 2...

Optimizing your Monitoring and Trending tools for the Cloud (Nagios World Con...

Recently uploaded

HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU

panagenda

Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/ DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen! Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell. Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten. Diese Themen werden behandelt - Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten - Wie funktionieren CCB- und CCX-Lizenzen wirklich? - Verstehen des DLAU-Tools und wie man es am besten nutzt - Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw. - Praxisbeispiele und Best Practices zum sofortigen Umsetzen

Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack

shyamraj55

Mind map of terminologies used in context of Generative AI

Kumud Singh

Cosa hanno in comune un mattoncino Lego e la backdoor XZ?

Speck&Tech

ABSTRACT: A prima vista, un mattoncino Lego e la backdoor XZ potrebbero avere in comune il fatto di essere entrambi blocchi di costruzione, o dipendenze di progetti creativi e software. La realtà è che un mattoncino Lego e il caso della backdoor XZ hanno molto di più di tutto ciò in comune. Partecipate alla presentazione per immergervi in una storia di interoperabilità, standard e formati aperti, per poi discutere del ruolo importante che i contributori hanno in una comunità open source sostenibile. BIO: Sostenitrice del software libero e dei formati standard e aperti. È stata un membro attivo dei progetti Fedora e openSUSE e ha co-fondato l'Associazione LibreItalia dove è stata coinvolta in diversi eventi, migrazioni e formazione relativi a LibreOffice. In precedenza ha lavorato a migrazioni e corsi di formazione su LibreOffice per diverse amministrazioni pubbliche e privati. Da gennaio 2020 lavora in SUSE come Software Release Engineer per Uyuni e SUSE Manager e quando non segue la sua passione per i computer e per Geeko coltiva la sua curiosità per l'astronomia (da cui deriva il suo nickname deneb_alpha).

Infrastructure Challenges in Scaling RAG with Custom AI models

Zilliz

Building Retrieval-Augmented Generation (RAG) systems with open-source and custom AI models is a complex task. This talk explores the challenges in productionizing RAG systems, including retrieval performance, response synthesis, and evaluation. We’ll discuss how to leverage open-source models like text embeddings, language models, and custom fine-tuned models to enhance RAG performance. Additionally, we’ll cover how BentoML can help orchestrate and scale these AI components efficiently, ensuring seamless deployment and management of RAG systems in the cloud.

Artificial Intelligence for XMLDevelopment

Octavian Nadolu

In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject. We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup. Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved. The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring. The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise. By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.

OpenID AuthZEN Interop Read Out - Authorization

David Brossard

Generating privacy-protected synthetic data using Secludy and Milvus

Zilliz

During this demo, the founders of Secludy will demonstrate how their system utilizes Milvus to store and manipulate embeddings for generating privacy-protected synthetic data. Their approach not only maintains the confidentiality of the original data but also enhances the utility and scalability of LLMs under privacy constraints. Attendees, including machine learning engineers, data scientists, and data managers, will witness first-hand how Secludy's integration with Milvus empowers organizations to harness the power of LLMs securely and efficiently.

Columbus Data & Analytics Wednesdays - June 2024

Jason Packer

“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...

Edge AI and Vision Alliance

For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/building-and-scaling-ai-applications-with-the-nx-ai-manager-a-presentation-from-network-optix/ Robin van Emden, Senior Director of Data Science at Network Optix, presents the “Building and Scaling AI Applications with the Nx AI Manager,” tutorial at the May 2024 Embedded Vision Summit. In this presentation, van Emden covers the basics of scaling edge AI solutions using the Nx tool kit. He emphasizes the process of developing AI models and deploying them globally. He also showcases the conversion of AI models and the creation of effective edge AI pipelines, with a focus on pre-processing, model conversion, selecting the appropriate inference engine for the target hardware and post-processing. van Emden shows how Nx can simplify the developer’s life and facilitate a rapid transition from concept to production-ready applications.He provides valuable insights into developing scalable and efficient edge AI solutions, with a strong focus on practical implementation.

GenAI Pilot Implementation in the organizations

kumardaparthi1024

GraphRAG for Life Science to increase LLM accuracy

Tomaz Bratanic

Serial Arm Control in Real Time Presentation

tolgahangng

Taking AI to the Next Level in Manufacturing.pdf

ssuserfac0301

Read Taking AI to the Next Level in Manufacturing to gain insights on AI adoption in the manufacturing industry, such as: 1. How quickly AI is being implemented in manufacturing. 2. Which barriers stand in the way of AI adoption. 3. How data quality and governance form the backbone of AI. 4. Organizational processes and structures that may inhibit effective AI adoption. 6. Ideas and approaches to help build your organization's AI strategy.

UiPath Test Automation using UiPath Test Suite series, part 6

DianaGray10

Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI. UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities. Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes. What will you get from this session? 1. Insights into integrating generative AI. 2. Understanding how this integration enhances test automation within the UiPath platform 3. Practical demonstrations 4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath Topics covered: What is generative AI Test Automation with generative AI and Open AI. UiPath integration with generative AI Speaker: Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP

Microsoft - Power Platform_G.Aspiotis.pdf

Uni Systems S.M.S.A.

Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf

Malak Abu Hammad

Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers: * What is Vector Search? * Importance and benefits of vector search * Practical use cases across various industries * Step-by-step implementation guide * Live demos with code snippets * Enhancing LLM capabilities with vector search * Best practices and optimization strategies Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications. #MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology

“I’m still / I’m still / Chaining from the Block”

Claudio Di Ciccio

AI 101: An Introduction to the Basics and Impact of Artificial Intelligence

IndexBug

Monitoring and Managing Anomaly Detection on OpenShift.pdf

Tosin Akinosho

Monitoring and Managing Anomaly Detection on OpenShift Overview Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices. Key Topics Covered 1. Introduction to Anomaly Detection - Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems. 2. Understanding Edge (IoT) - Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source. 3. What is ArgoCD? - Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices. 4. Deployment Using ArgoCD for Edge Devices - Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD. 5. Introduction to Apache Kafka and S3 - Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions. 6. Viewing Kafka Messages in the Data Lake - Learn how to view and analyze Kafka messages stored in a data lake for better insights. 7. What is Prometheus? - Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices. 8. Monitoring Application Metrics with Prometheus - Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system. 9. What is Camel K? - Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes. 10. Configuring Camel K Integrations for Data Pipelines - Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow. 11. What is a Jupyter Notebook? - Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text. 12. Jupyter Notebooks with Code Examples - Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.

Recently uploaded (20)

HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU

Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack

Mind map of terminologies used in context of Generative AI

Cosa hanno in comune un mattoncino Lego e la backdoor XZ?

Infrastructure Challenges in Scaling RAG with Custom AI models

Artificial Intelligence for XMLDevelopment

OpenID AuthZEN Interop Read Out - Authorization

Generating privacy-protected synthetic data using Secludy and Milvus

Columbus Data & Analytics Wednesdays - June 2024

“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...

GenAI Pilot Implementation in the organizations

GraphRAG for Life Science to increase LLM accuracy

Serial Arm Control in Real Time Presentation

Taking AI to the Next Level in Manufacturing.pdf

UiPath Test Automation using UiPath Test Suite series, part 6

Microsoft - Power Platform_G.Aspiotis.pdf

Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf

“I’m still / I’m still / Chaining from the Block”

AI 101: An Introduction to the Basics and Impact of Artificial Intelligence

Monitoring and Managing Anomaly Detection on OpenShift.pdf

IEEE ISSRE 2018 - Use of Self-Healing Techniques to Improve the Reliability of a Dynamic and Geo-Distributed Ad Delivery Service

1. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Use of Self-Healing Techniques to Improve the Reliability of a Dynamic and Geo-Distributed Ad Delivery Service Nicolas Brousse and Oleksii Mykhailov

7. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Inconsistent GeoDNS Routing High Latency From Eyeball To Content Origin Origin Failure Impact User Experience Impact Campaign Performance and Revenue 7

14. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 14 Few Words About Anycast § Shortest Path Routing Means § Not Latency Aware § Not Congestion Aware / Packet Loss § Limited Control for Traffic Steering § Difficult Troubleshooting § Failover lead to packet RST for Active Sessions § Mitigation with a large and well distributed number of POPs

18. © 2018 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Human Failover vs Self-Healing 18 > 1h Failure < 15min < 30min Region Traffic Rerouted Self Recover Note: since the paper publication, we reduced automated failover time to be less than a few seconds. See demo. Fig. 1 Human Failover with Manual Recovery steps Fig. 2 Automated Failover and Self-Healing Recovery

IEEE ISSRE 2018 - Use of Self-Healing Techniques to Improve the Reliability of a Dynamic and Geo-Distributed Ad Delivery Service

Recommended

Recommended

More Related Content

Similar to IEEE ISSRE 2018 - Use of Self-Healing Techniques to Improve the Reliability of a Dynamic and Geo-Distributed Ad Delivery Service

Similar to IEEE ISSRE 2018 - Use of Self-Healing Techniques to Improve the Reliability of a Dynamic and Geo-Distributed Ad Delivery Service (20)

More from Nicolas Brousse

More from Nicolas Brousse (11)

Recently uploaded

Recently uploaded (20)

IEEE ISSRE 2018 - Use of Self-Healing Techniques to Improve the Reliability of a Dynamic and Geo-Distributed Ad Delivery Service