Top 5 Things I've Messed Up in Live Streaming

•Download as PPTX, PDF•

3 likes•634 views

June 24, 2014 - SF Video Technology Meetup. Live streaming can be difficult and challenging, yet extremely rewarding. In the video gaming world, it has become the most powerful way of reaching your audience. It combines traditional web technologies with even more traditional broadcast video stacks in a "it's going live now" environment. In this presentation Lee Chen, head of product at Fastly, goes over the top 5 things that have lost him sleep, made his eyebrows go up, and sometimes even made him fail -- what he did to correct it.

Technology

Top 5 things I’ve F’d up (In Live
Streaming)

About me
• GotFrag.com
• Major League Gaming
• Product Manager for Streaming Media at
Fastly
• Head of Product at Fastly

What’s this all about?
• Live streaming (not live linear)
• The (comedic) errors:
5. Forest, Trees. The big picture.
4. Latency sucks.
3. Failover. Not Fall Over.
2. Redundancy is not redundant.
1. Death by a thousand cuts. Measure it!

Some assumptions
• What:
– Live streaming is hard
– It shouldn’t be.
– But it is.
– It’s real time.
– So it’s hard.
• Why:
– “Analog” broadcast to “Digital”
medium
– Every step of the signal path can
break it
– And then there’s the Internet.

#5: The whole picture is important
• Analog to Digital
• Wires to Code
• Complexity
• Troubleshooting and QA is therefore
hard
• Studio != Touring Show

#5: What got F’d.
• Audio out of sync
• Root cause was on the switcher <>
audio board
• Keyframes >1s on Encoder
compounded problem

#4: Spoilers suck
• Time Behind Live (TBL) is critical
• Broadcast standard is <15s TBL
• In other words, latency sucks.
– Latency = Buffering, jitters
– Latency = Spoilers
– Latency = WTF just happened.

Latency Sources (Digital side)
Source Ideal Avg Latency
Encoding <100ms
Uplink IP = <100ms, Sat = <1500ms
Ingest <10ms
Transcoding 50 - 500ms
CDN 150 – 1500 ms
Last Mile: Player / Browser / User
Internet
?????
Total: 3610ms (3.6sec) + Last mile

#4 What got F’d
• RTMP delivery sucks on a CDN
– 5min into broadcast = 5sec TBL
– 30min into broadcast = 30sec TBL
– 8hrs into broadcast = 2min TBL
• Trolls hate spoilers. So do legit fans.
• Separate stack from normal delivery, older
hardware
• HTTP streaming is the answer (HLS!!)

#3: What got F’d
• Just cause the vendor says so, doesn’t
mean it _IS_ so.
• Test it.
• No really, TEST IT.
• Can you fail back and forth?
• Does it cause disconnects at player?

#2: Redundancy is not redundant
• Analog
– Matrixed encoders
– Multiple uplinks
– Back up routers
• Digital
– Cloud transcoding
– Multi-CDN

#2: What got F’d
• Production Budget
– $10k/hr for sat truck
• Upstream ISP link
from venue died
• Fix: Microwave pt to
pt to the Luxor.

#1: Real time feedback. Flexible solutions
Digital feedback:
• Analytics.
– Disconnects
– Buffering
– Edge usage
– Load on systems
– Geo distro
– etc
Analog feedback
• Leverage the audience
– Twitter
• Watch the stream yourself
• Filter the noise
Flexible partners and solutions

#1: What got F’d
• Tendency is to have digital side isolated from the
analog side.
• Don’t. Analog thinks about the room. Digital thinks
about the “Internets” Put the two together.

Fastly Altitude - June 25, 2015. Catchpoint CEO Mehdi Daoudi talks about performance measuring and monitoring. Video from the talk: http://fastly.us/Altitude2015-Catchpoint-Video Mehdi's bio: Mehdi is the founder and CEO of Catchpoint, the fastest-growing Web Performance Monitoring Solution. The Catchpoint vision is to provide the best IT analytics & intelligence to help companies improve their end user performance and keep an eye on their infrastructure and vendors. Before Catchpoint, Mehdi spent 10+ years at DoubleClick and Google, where he was responsible for Quality of Services, buying, building, deploying and using various internal and external monitoring solutions to keep an eye on the DART infrastructure delivering billions of transactions a day using thousands of servers, routers, and various other web performance systems.

Interative Traffic Engineering in Changing Internet Economics - Tom Daly at L...

Fastly

The physical, logical, and economic topology of the Internet is ever changing due to new network to network interconnections, the planned or accidental disconnection of existing interconnections, and the ever growing needs for more network capacity. As a multi-Terabit traffic source, Fastly must regularly evaluate its needs for network capacity against performance and operating costs. This talk will examine the capacity planning models, scalability concerns, and economic drivers used to make key business decisions regarding connectivity, emphasizing the needs for metrics collection, analysis, and constant iterative tuning involving a distributed, international team.

Building and Scaling the Fastly Network - João Taveira at Fastly Altitude 2015

Fastly

Fastly Altitude - June 25, 2015. Fastly set out to build a highly performant, real-time CDN that is both reliable and scalable. This talk will cover how we chose to design our network and its unique routing and load balancing properties, as well as where we're going next. Video of the talk: http://fastly.us/Altitude2015_Networking João's bio: João Taveira is a network engineer at Fastly, where he is responsible for making dumb switches do clever things. In addition to writing software for network orchestration, João works on protocol design and performance, and holds a PhD from University College London or something to that effect.

Design & Performance - Steve Souders at Fastly Altitude 2015

Fastly

Fastly Altitude - June 25, 2015. Chief SpeedCurver Steve Souders explains how design and web performance are more interconnected than ever before. Users want a fast website with a rich design, but sometimes the interplay between design and performance feels like a fixed sum game: one side's gain is the other side's loss. Design and performance are indeed connected, but it's more like the yin and yang. They aren't opposing forces, but instead complement each other. Bringing these processes together produces experiences that are rich and fast. Video from the talk: http://fastly.us/Altitude2015_Design-Performance Steve's bio: Steve Souders is a co-founder at SpeedCurve, where he develops web performance services. His book, High Performance Web Sites, explains his best practices for performance; it was #1 in Amazon's Computer and Internet bestsellers. His follow-up book, Even Faster Web Sites, provides performance tips for today's Web 2.0 applications. Steve is the creator of many performance tools and services including YSlow, the HTTP Archive, Cuzillion, Jdrop, SpriteMe, ControlJS, and Browserscope. He serves as co-chair of Velocity, the web performance and operations conference from O'Reilly, and is co-founder of the Firebug Working Group.

We all want to move fast. We construct our systems in a rapidly-iterating and agile way. Ideally we design and build them to be efficient, robust, and have low latency. But sometimes in the search for speed we make mistakes that come back to haunt us. The talk is about the things most of us sacrifice when trying to iterate fast. Some of these are reasonable trade-offs — others can wreck your product. I hope you will walk away with a notion of what to pay attention to at various stages of software development, and how you can avoid common pitfalls.

Advanced VCL Workshop - Rogier Mulhuijzen and Stephen Basile at Fastly Altitu...

Fastly

Fastly Altitude - June 25, 2015. Want to integrate Fastly with a paywall, or learn how to efficiently do a large amount of redirects from VCL? This workshop with Fastly engineers Rogier Mulhuijzen and Stephen Basile is a deep dive into six advanced use cases and the VCL needed for them. Rogier's bio: Rogier “Doc” Mulhuijzen is a Varnish wizard and senior professional services engineer at Fastly where he focuses on performance tuning and troubleshooting. When he’s not helping customers, he sits on the Varnish Governance Board, where he helps give direction and solve issues for the Varnish open source project. Formerly an Oracle DBA and Unix admin, Rogier learned scaling while working for Ziggo, a major Dutch ISP.

The Fallacy of Fast - Ines Sombra at Fastly Altitude 2015

Fastly

Fastly Altitude - June 25, 2015. Ines Sombra, a Systems Engineer at Fastly, talks about lessons learned in rapid systems development. Video of the talk: http://fastly.us/Altitude2015_Fallacy-of-Fast Ines' bio: Ines Sombra is a Systems Engineer at Fastly, where she spends her time helping the Web go faster. Ines holds an M.S. in Computer Science and an M.S. in Information Management from Washington University in Saint Louis. Being a true Argentine, she has a fondness for steak, fernet, and a pug named Gordo.

Developing a Globally Distributed Purging System

Fastly

(Surge 2014) How do you build a distributed cache invalidation system that can invalidate content in 150 milliseconds across a global network of servers? Fastly CTO Tyler McMullen and engineer Bruce Spang will discuss the process of constructing a production-ready distributed system built on solid theoretical foundations. This talk will cover using research to design systems, the bimodal multicast algorithm, and the behavior of this system in production.

It Probably Works

Fastly

Probabilistic algorithms exist to solve problems that are either impossible or unrealistic (too expensive, too time consuming, etc.) to solve precisely. In an ideal world, you would never actually need to use probabilistic algorithms. For programmers who are not familiar with them, the concept can be positively nerve-racking: “How do I know it will actually work? What if it is inexplicably wrong? How can I debug it? Maybe we should just punt on this problem or buy a whole lot more servers. . .” However, for those who either deeply understand probability theory or at least have used and observed the behavior of probabilistic algorithms in large-scale production environments, these algorithms are not only acceptable but also worth using at any opportunity. This is because they can help solve problems, create systems that are less expensive and more predictable, and do things that could not be done otherwise.

Beyond Breakpoints: A Tour of Dynamic Analysis

Fastly

Despite advances in software design and static analysis techniques, software remains incredibly complicated and difficult to reason about. Understanding highly-concurrent, kernel-level, and intentionally-obfuscated programs are among the problem domains that spawned the field of dynamic program analysis. More than mere debuggers, the challenge of dynamic analysis tools is to be able record, analyze, and replay execution without sacrificing performance. This talk will provide an introduction to the dynamic analysis research space and hopefully inspire you to consider integrating these techniques into your own internal tools.

It Probably Works - QCon 2015

Fastly

From list sorting to network routing, and from hash tables to capacity planning, a programmer's daily work is filled with probability. We use probabilistic algorithms, data structures, and systems constantly often without even thinking about it. Experienced engineers reach for probabilistic algorithms frequently and intentionally, especially when building systems of serious scale. How do probabilistic algorithms actually work in practice? And how do we know they'll be safe and reliable in our critical production systems? We'll address those questions, explore a few algorithms, and see why "with high probability" is often better than "exactly".

Building a better web

Fastly

On the eve of what was hoped to be of the biggest traffic days for New York Magazine’s sites, the company was the target of a DDoS attack that caused their sites to go dark. New York quickly turned to Fastly to deflect and overcome the attack. Larry discusses how New York Mag went from zero page views per second to getting back online and recording one of their biggest traffic days of the year with the aid of Fastly’s team and tech. In addition he discusses how New York is leveraging Fastly as part of a larger strategy of performance improvements to deliver the build a better web and deliver the best premium content experience in the context of alternative distribution and consumption channels, such as Google Amp and FB Instant Article.

The future of the edge

Fastly

Over the last few years, Fastly has changed what’s possible at the edge of the internet. But what would it take to build complete, complex applications at the edge? What new tools and concepts would we need and what is the current state-of-the-art? In this talk, Fastly CTO Tyler McMullen discusses where the edge is heading over the next several years, some ideas for what it could look like, and how it could fundamentally change the way you build applications.

Rails Caching: Secrets From the Edge

Fastly

Migrating Target to Fastly - Eddie Roger at Fastly Altitude 2015

Fastly

Fastly Altitude - June 25, 2015. Eddie Rodger is a Senior Engineer at Target, who spoke about migrating Target to Fastly. Video of the talk: http://fastly.us/Altitude2015_Migrating-to-Fastly Eddie's bio: Eddie Roger is a Senior Engineer at Target Corporation. Over the course of his career, Eddie has experienced retail and marketing IT, developing solutions across varying languages and platforms. Currently, he works on api.target.com, focusing on creating and integrated technologies that drive data to Target teams, their partners, and guests. A born technologist, Eddie enjoys expanding his knowledge of all things tech and finding ways of creating digital works to solve problems.

From Zero to Capacity Planning

Fastly

Making ops life easier

Fastly

Carmen Sarlo Principal Operations Engineer, Vox Media at Fastly Altitude 2016 Fastly’s API has provided us the ability to create tooling for any member of the team to use and feel confident to roll out changes. Some of our tooling using Fastly’s API is done by chat-ops or integrated workflow through Jenkins. Making edge tasks easier for other team members, makes a happier engineering staff.

Debugging Your CDN - Austin Spires at Fastly Altitude 2015

Fastly

Fastly Altitude - June 25, 2015. Austin Spires, Lead Customer Engineer at Fastly, covers how to use curl tricks, Varnish logging, and web services to identify and debug issues. Video of the talk: http://fastly.us/Altitude2015_Debugging-Your-CDN Austin's bio: Austin Spires is a lead customer engineer at Fastly. He’s been working on developer tools and customer happiness for five years, and frequently speaks at conferences and meetups. Recently, he’s been blogging about how Fastly scales its support infrastructure. Before Fastly, Austin worked in sales and support at GitHub, where he helped lead customer onboarding. Originally from Texas, Austin plays a mean bass and likes drinking cheap beer.

Caching the Uncacheable [Long Version]

Fastly

(Surge 2014) This is a longer version of our Velocity 2014 slides around caching dynamic content. Topic: In the past, CDNs have been used to cache and distribute static objects. But issues around invalidation, staleness, and lack of visibility have prevented us from using CDNs to fully leverage the benefits of caching when it comes to dynamic content. Today, using a real-time, modern CDN that provides instant cache invalidation and real-time analytics allows for instantaneous control over dynamic content caching.

Resource Allocation in Computer Networks

Fastly

João tells us "Resource allocation in computer networks is notoriously hard - it took over twenty years to find something that works, and jury is out as to whether we'll ever find something that works properly. In a rollercoaster ride of historical revisionism, this talk will cover the evolution of resource allocation as applied to networking, and provide context for the mess we're currently in" [1] Paul Baran - On Distributed Communications - https://www.rand.org/content/dam/rand/pubs/research_memoranda/2006/RM3420.pdf [2] V. Cerf and R. Kahn - A protocol for packet network intercommunication - https://www.cs.princeton.edu/courses/archive/fall06/cos561/papers/cerf74.pdf [3] Van Jacobson - Congestion avoidance and control - http://ee.lbl.gov/papers/congavoid.pdf [4] Bob Briscoe - Flow Rate Fairness: Dismantling a Religion - http://pbg.cs.illinois.edu/courses/cs598fa09/readings/b07.pdf João's Bio João Taveira is a network engineer at Fastly, where he is responsible for making dumb switches do clever things. In addition to writing software for network orchestration, Joao works on protocol design and performance, and holds a PhD from University College London on something to that effect.

Load balancing is impossible

Fastly

Load balancing is something most of us assume is a solved problem. But the idea that load balancing is “solved” could not be further from the truth. If you use multiple load balancers, the problem is even worse. Most of us use “random” or “round-robin” techniques, which have certain advantages but are highly inefficient. Others use more complex algorithms like “least-conns,” which can be more efficient but have horrific edge cases. “Consistent hashing” is a very useful technique but only applies to certain problems. There are several factors that exist both in theory and practice that make efficient load balancing an exceptionally hard problem, including Poisson request arrival times, exponentially distributed response latency, and oscillations when sharing data between multiple load balancers. Luckily, there are techniques and algorithms that have been developed that can make life better. Tyler McMullen explains some of the ways that we can do better than “random,” “round-robin,” and naive “least-conns,” even with distributed load balancers.

Load testing, Lessons learnt and Loadzen - Martin Buhr at DevTank - 31st Janu...

Loadzen

Scratching the itch, making Scratch for the Raspberry Pie

ESUG

Title: Scratching the itch, making Scratch for the Raspberry Pie Speaker: Tim Rowledge Fri, August 22, 12:00pm – 12:30pm Abstract: Scratch was originally written in a Squeak 2.8 era image. Much has changed since then and to make the Raspberry Pi run Scratch as well as possible we have ported the code forward to a 4.5 image so it can run on a StackVM; and soon a Cog VM. A substantial amount of Smalltalk code has had to be rewritten to do this and yet we have to maintain complete compatibility with the original system to avoid overloading the teachers that use it in their classes. A new branch of Cog for the ARM cpu is being written as well. Bio: Tim Rowledge has almost 30 years of Smalltalk experience, and almost as much with ARM. Somehow the two have always gone together.

Viewers also liked

Applying Varnish

Fastly

Confident Refactoring - Ember SF Meetup

Fastly

What we can learn from CDNs about Web Development, Deployment, and Performance

Fastly

Fallacy of Fast

Fastly

Advanced VCL Workshop - Rogier Mulhuijzen and Stephen Basile at Fastly Altitu...

Fastly

The Fallacy of Fast - Ines Sombra at Fastly Altitude 2015

Fastly

Developing a Globally Distributed Purging System

Fastly

It Probably Works

Fastly

Beyond Breakpoints: A Tour of Dynamic Analysis

Fastly

It Probably Works - QCon 2015

Fastly

Building a better web

Fastly

The future of the edge

Fastly

Rails Caching: Secrets From the Edge

Fastly

Migrating Target to Fastly - Eddie Roger at Fastly Altitude 2015

Fastly

From Zero to Capacity Planning

Fastly

Making ops life easier

Fastly

Debugging Your CDN - Austin Spires at Fastly Altitude 2015

Fastly

Caching the Uncacheable [Long Version]

Fastly

Resource Allocation in Computer Networks

Fastly

Load balancing is impossible

Fastly

Viewers also liked (20)

Applying Varnish

Confident Refactoring - Ember SF Meetup

What we can learn from CDNs about Web Development, Deployment, and Performance

Fallacy of Fast

Advanced VCL Workshop - Rogier Mulhuijzen and Stephen Basile at Fastly Altitu...

The Fallacy of Fast - Ines Sombra at Fastly Altitude 2015

Developing a Globally Distributed Purging System

It Probably Works

Beyond Breakpoints: A Tour of Dynamic Analysis

It Probably Works - QCon 2015

Building a better web

The future of the edge

Rails Caching: Secrets From the Edge

Migrating Target to Fastly - Eddie Roger at Fastly Altitude 2015

From Zero to Capacity Planning

Making ops life easier

Debugging Your CDN - Austin Spires at Fastly Altitude 2015

Caching the Uncacheable [Long Version]

Resource Allocation in Computer Networks

Load balancing is impossible

Similar to Top 5 Things I've Messed Up in Live Streaming

Load testing, Lessons learnt and Loadzen - Martin Buhr at DevTank - 31st Janu...

Loadzen

Scratching the itch, making Scratch for the Raspberry Pie

ESUG

Distributed "Web Scale" Systems

Ricardo Vice Santos

SRECon-Europe-2017: Networks for SREs

Michael Kehoe

All of us depend on the underlying network to be stable whether in the datacenter or in the cloud. We all have a basic knowledge of how traditional networks run, however in the past 10 years, we’ve moved to building redundant physical topologies in our networks, optimized the routing methodologies accordingly, moved into the cloud and gotten greater visibility and tuneables in the Linux kernel network stack. A lot has changed! However, the way we troubleshoot the network in relation to the applications we support hasn’t adapted. In this session, we’ll review the progress that network infrastructure has made look at specific examples where traditional troubleshooting responses fail us and demonstrate our need to rethink our approach to making applications and the network interact harmoniously.

2004: Söldner - a Post Mortem

Teut Weidemann

Llnw bufferbloat

Stephen Hemminger

[DSC Europe 22] Make some noise for AI in JavaScript - Sead Delalic

DataScienceConferenc1

Noise suppression during audio calls is expected from any real-time communication platform. Good denoising means exhaustive use of modern neural networks. AI implies heavy processing, real-time requires speed, and a large number of calls means a large need for resources. Therefore, the questions are: can clients do the necessary processing and can AI solutions be integrated on the client side? We will present a neural network for noise suppression implemented as part of the Infobip WebRTC platform. A generic way of integrating AI solutions with client-side JavaScript will be described, with a special focus on real-time requirements. The final solution based on RNNoise will be presented.

London Video Tech - Adventures in cutting every last millisecond from glass-t...

Kieran Kunhya

When DevOps and Networking Intersect by Brent Salisbury of socketplane.io

DevOps4Networks

Internet 101lzeltzer

Spotify: P2P music streaming

Ricardo Vice Santos

The Ever So Slighty Geeky Quiz With Maybe

Amanda Jackson

Rich, Real-time Mobile User Experiences @Devoxx UK

Andy Piper

Faxing Q and A

Joshua Goldbard

Jan Hloušek, Keen Software House

White Nights Conference

Harlan Beverly Lag The Barrier to innovation gdc austin 2009

Harlan Beverly

Dmk blackops2006 cccDan Kaminsky

OnAndroidConf 2013: Accelerating the Android Platform Build

David Rosen

Presented at the OnAndroidConf, October 22 2013, http://www.onandroidconf.com/sessions.html Abstract: Optimizing the Android build environment to perform at world-class level is a big challenge for many Android device and chipset makers today. Churning through thousands of platform builds per week requires laser-focus on high-performance infrastructure and tooling. If you’re looking at improving your overall engineering and developer productivity, the software build use case is an obvious area to prioritize. This technical talk will focus on the following aspects of the Android platform build: Common Android platform build challenges and opportunities with real-life production references The various Android build use cases and their needs – full integration and release builds, developer incremental builds Evolution of the Android build and codebase with trends and statistics Detailed technical analysis of the Android platform build, highlighting opportunities for improvements Proposed solutions and technical tricks to optimize an Android software build environment

nodebots presentation @seekjobs

Esteban (Steven) De Salas

FITB: Network graphing done right

Laurie Denness

Similar to Top 5 Things I've Messed Up in Live Streaming (20)

Load testing, Lessons learnt and Loadzen - Martin Buhr at DevTank - 31st Janu...

Scratching the itch, making Scratch for the Raspberry Pie

Distributed "Web Scale" Systems

SRECon-Europe-2017: Networks for SREs

2004: Söldner - a Post Mortem

Llnw bufferbloat

[DSC Europe 22] Make some noise for AI in JavaScript - Sead Delalic

London Video Tech - Adventures in cutting every last millisecond from glass-t...

When DevOps and Networking Intersect by Brent Salisbury of socketplane.io

Internet 101

Spotify: P2P music streaming

The Ever So Slighty Geeky Quiz With Maybe

Rich, Real-time Mobile User Experiences @Devoxx UK

Faxing Q and A

Jan Hloušek, Keen Software House

Harlan Beverly Lag The Barrier to innovation gdc austin 2009

Dmk blackops2006 ccc

OnAndroidConf 2013: Accelerating the Android Platform Build

nodebots presentation @seekjobs

FITB: Network graphing done right

More from Fastly

Revisiting HTTP/2

Fastly

RFC 7540 was ratified over 2 years ago and, today, all major browsers, servers, and CDNs support the next generation of HTTP. Just over a year ago, at Velocity, we discussed the protocol, looked at some real world implications of its deployment and use, and what realistic expectations we should have from its use. Now that adoption is ramped up and the protocol is being regularly used on the Internet, it's a good time to revisit the protocol and its deployment. Has it evolved? Have we learned anything? Are all the features providing the benefits we were expecting? What's next?In this session, we'll review protocol basics and try to answer some of these questions based on real-world use of it. We'll dig into the core features like interaction with TCP, server push, priorities and dependencies, and HPACK. We'll look at these features through the lens of experience and see if good practice patterns have emerged. We'll also review available tools and discuss what protocol enhancements are in the near and not-so-near horizon.

Altitude San Francisco 2018: Preparing for Video Streaming Events at Scale

Fastly

CBS Interactive streams some of the largest video streaming events on the planet, including SuperBowl in 2019. This talk will focus on all the work that goes in ahead of time to prepare and plan for game day. From architecture design to capacity reservations to operational visibility and building playbooks we will explore how we build, test and prepare for these large events. We will also explore how some of Fastly's unique features such as MediaShield and VCL are becoming critical to these workflows.

Altitude San Francisco 2018: Building the Souther Hemisphere of the Internet

Fastly

As a global organization, Fastly carefully selects and deploys POP locations to service the greater audience of the Internet. Fastly currently has 52 global POPs across the Internet, 13 of which are located in the Southern Hemisphere. Another 3 are outside North America, Europe, and Asia. During this talk, VP of Infrastructure Tom Daly will share our experience in building Fastly's network of POPs south of the equator, where, in some cases, the Internet we know here in San Francisco, is much different. Tom will explore the physical datacenter infrastructure, network topology, and network policy that pose of unique challenges when operating in these parts of the world.

Altitude San Francisco 2018: The World Cup Stream

Fastly

FuboTV’s recent offering of the 2018 FIFA World Cup broke all of our previous records for viewership and put our systems to the test as we delivered all 64 matches live. Coverage for a majority of games was spread out across ~150 regional sports networks, local FOX affiliates, owned and operated regional stations and other local FOX offerings, with a few early matches broadcasted on national channels. Running a successful World Cup required us to pay close attention to our caching strategies, delivery mechanisms, content edge-case handling and more. An event at this scale, spread out over a month, also gave us an excellent test bed to run experiments. We were able to augment our last-mile delivery, test/tweak our solution for CDN decisioning/priority, and even stand up a set of UHD HDR10 feeds to give our users their first glimpse of live OTT UHD offerings. We’ll run through this whole event from a scale and technology perspective and share our takeaways as we prepare for the upcoming NFL season and beyond.

Altitude San Francisco 2018: We Own Our Destiny

Fastly

Altitude San Francisco 2018: Scale and Stability at the Edge with 1.4 Billion...

Fastly

Braze is a customer engagement platform that delivers more than a billion messaging experiences across push, email, apps and more each day. In this session, Jon Hyman will describe the company's challenges during an inflection point in 2015 when the company reached the limitation of their physical networking equipment, and how Braze has since grown more than 7x on Fastly. Jon will also discuss how Braze uses Fastly's Layer 7 load balancing to improve stability and uptime of its APIs.

Altitude San Francisco 2018: Moving Off the Monolith: A Seamless Migration

Fastly

Altitude San Francisco 2018: Bringing TLS to GitHub Pages

Fastly

Altitude San Francisco 2018: HTTP Invalidation Workshop

Fastly

One of the most powerful tools that Fastly offers is worldwide, instant purge. Come learn the ins and outs of how HTTP invalidation works in general and how purge and surrogate keys can be used to improve your site's delivery and get even more value from Fastly. This talk will also cover the purge blast radius Surrogate Keys are an amazing way to purge your content from cache, but they can be a bit scary when you aren't sure how many URLs this surrogate key is tied to or what kind of affect this will have on origin. Join the USA Today Network as we explain how we leverage big data tools, Go APIs, New Relic, and Sumo Logic to provide our users a suite of tools for purging content from Fastly. Developers love knowing the blast radius of their surrogate keys, while our engineers love the real-time metrics and notifications we get when developers are hard-purging content.

Altitude San Francisco 2018: HTTP/2 Tales: Discovery and Woe

Fastly

Altitude San Francisco 2018: How Magento moved to the cloud while maintaining...

Fastly

Magento Commerce was first released by a small web development agency over ten years when they saw first-hand what a challenge it was for companies like them to build unique eCommerce sites. They created an open source platform that gives developers the flexibility to create meaningful shopping experiences while building a global community that drives down merchant costs and fosters innovation. Amid the rise of cloud-based software Magento needed to keep pace with more complex merchant needs and heightened shopper expectations. In this session learn how Magento, with the help of Partners like Fastly, evolved into a cloud-based platform without sacrificing their commitment to open software, flexibility, and the community.

Altitude San Francisco 2018: Scaling Ethereum to 10B requests per day

Fastly

ConsenSys is a venture production studio building decentralized applications and developer and end-user tools for blockchains. Their Infura platform is a core infrastructure pillar of Ethereum, enabling decentralized applications of all kinds to scale to accommodate their users. Infura went from 20 million requests a day at the beginning of 2017 to over 10 billion requests today. This staggering 500x increase naturally lead to questions of scale. In this talk, co-founder Michael Wuehler will discuss the technical challenges encountered while building and scaling the Infura platform, and the infrastructure decisions that led to their adoption of Fastly and other pivotal technologies.

Altitude San Francisco 2018: Authentication at the Edge

Fastly

Turning away unwanted traffic close to the source is a common and key use case for edge networks like Fastly, but identity, authentication, and authorization at the edge can go far beyond blocking DDoS. The unique way that you identify your site’s users can probably move to the edge too, allowing you to cut response times in your critical path, offload more origin traffic, and make smarter routing decisions at the edge. In this talk we’ll cover a number of patterns in use by real Fastly customers. Whether you prefer token authentication, pre-shared keys, OAuth, HTTP auth, JSON web tokens, or a complex paywall, learn how you can potentially make your authentication decisions at the edge.

Altitude San Francisco 2018: WebAssembly Tools & Applications

Fastly

Altitude San Francisco 2018: Testing with Fastly Workshop

Fastly

Altitude San Francisco 2018: Fastly Purge Control at the USA TODAY NETWORK

Fastly

Altitude San Francisco 2018: WAF Workshop

Fastly

In this hands-on workshop you will attack a vulnerable web application while defending your own web service behind a Fastly WAF. Attendees will depart understanding how common web application attacks can be exploited as well defended against. They will experience WAF logging and analytics via sumologic to detect attacks realtime. For mitigation you will use a preview version of our newly built WAF rule management UI. We will close off the workshop by deep diving on how our security team analyzed and mitigated some of this summer major vulnerabilities.

Altitude San Francisco 2018: Logging at the Edge

Fastly

Altitude San Francisco 2018: Video Workshop Docs

Fastly

Fastly delivers more than a million log events per second. Our Real-Time Log Streaming is easy to set up, but there are many features you might not be using to their full extent. This workshop will cover setting up logging to various endpoints, dealing with structured data, and getting real-time insights into your customers’ behavior. - - - - - - - - - - - Live streaming and on-demand video can provide a powerful way to connect with customers, but viewers expect seamless pixel-perfect streams without common video delivery inconveniences, such as downtime or lags. This workshop will demonstrate how anyone can deliver live video at scale. We’ll thoroughly explain key video delivery optimizations and more importantly, demonstrate their efficacy using the data collected from both Fastly Log Streaming/Sumo Logic and the Mux quality of experience service.

Altitude San Francisco 2018: Programming the Edge

Fastly

Programming the edge Second floor Andrew Betts Principal Developer Advocate, Fastly Hide abstract Through our support for running your own code on our edge servers, Fastly's network offers you a platform of unparalleled speed, reliability and efficiency to which you can delegate a surprising amount of logic that has traditionally been in the application layer. In this workshop, you'll implement a series of advanced edge solutions, and learn how to apply these patterns to your own applications to reduce your origin load, dramatically improve performance, and make your applications more secure.

More from Fastly (20)

Revisiting HTTP/2

Altitude San Francisco 2018: Preparing for Video Streaming Events at Scale

Altitude San Francisco 2018: Building the Souther Hemisphere of the Internet

Altitude San Francisco 2018: The World Cup Stream

Altitude San Francisco 2018: We Own Our Destiny

Altitude San Francisco 2018: Scale and Stability at the Edge with 1.4 Billion...

Altitude San Francisco 2018: Moving Off the Monolith: A Seamless Migration

Altitude San Francisco 2018: Bringing TLS to GitHub Pages

Altitude San Francisco 2018: HTTP Invalidation Workshop

Altitude San Francisco 2018: HTTP/2 Tales: Discovery and Woe

Altitude San Francisco 2018: How Magento moved to the cloud while maintaining...

Altitude San Francisco 2018: Scaling Ethereum to 10B requests per day

Altitude San Francisco 2018: Authentication at the Edge

Altitude San Francisco 2018: WebAssembly Tools & Applications

Altitude San Francisco 2018: Testing with Fastly Workshop

Altitude San Francisco 2018: Fastly Purge Control at the USA TODAY NETWORK

Altitude San Francisco 2018: WAF Workshop

Altitude San Francisco 2018: Logging at the Edge

Altitude San Francisco 2018: Video Workshop Docs

Altitude San Francisco 2018: Programming the Edge

Recently uploaded

Connector Corner: Automate dynamic content and events by pushing a button

DianaGray10

Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to: Create a campaign using Mailchimp with merge tags/fields Send an interactive Slack channel message (using buttons) Have the message received by managers and peers along with a test email for review But there’s more: In a second workflow supporting the same use case, you’ll see: Your campaign sent to target colleagues for approval If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team But—if the “Reject” button is pushed, colleagues will be alerted via Slack message Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors. And... Speakers: Akshay Agnihotri, Product Manager Charlie Greenberg, Host

Bits & Pixels using AI for Good.........

Alison B. Lowndes

UiPath Test Automation using UiPath Test Suite series, part 3

DianaGray10

Essentials of Automations: Optimizing FME Workflows with Parameters

Safe Software

Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place. Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects. Here’s what you’ll gain: - Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows. - Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy. - Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency. - Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity. We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic. Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.

Generating a custom Ruby SDK for your web service or Rails API using Smithy

g2nightmarescribd

Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.

Key Trends Shaping the Future of Infrastructure.pdf

Cheryl Hung

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

FIDO Alliance

Epistemic Interaction - tuning interfaces to provide information for AI support

Alan Dix

Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024 https://alandix.com/academic/papers/synergy2024-epistemic/ As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.

How world-class product teams are winning in the AI era by CEO and Founder, P...

Product School

JMeter webinar - integration with InfluxDB and Grafana

RTTS

Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application. In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics. Length: 30 minutes Session Overview ------------------------------------------- During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana: - What out-of-the-box solutions are available for real-time monitoring JMeter tests? - What are the benefits of integrating InfluxDB and Grafana into the load testing stack? - Which features are provided by Grafana? - Demonstration of InfluxDB and Grafana using a practice web application To view the webinar recording, go to: https://www.rttsweb.com/jmeter-integration-webinar

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

BookNet Canada

The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more. Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/ Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.

Knowledge engineering: from people to machines and back

Elena Simperl

Neuro-symbolic is not enough, we need neuro-*semantic*

Frank van Harmelen

Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”. All of this illustrated with link prediction over knowledge graphs, but the argument is general.

Securing your Kubernetes cluster_ a step-by-step guide to success !

KatiaHIMEUR1

Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster. However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks. In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

DanBrown980551

Do you want to learn how to model and simulate an electrical network from scratch in under an hour? Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)! During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook. PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides: - A fully editable and extendable library for grid component modelling; - Visualization tools to display your network; - Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses; The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well. What you will learn during the webinar: - For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills; - For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.

Elevating Tactical DDD Patterns Through Object Calisthenics

Dorra BARTAGUIZ

After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!

Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...

Jeffrey Haguewood

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams. Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Monitoring Java Application Security with JDK Tools and JFR Events

Ana-Maria Mihalceanu

PCI PIN Basics Webinar from the Controlcase Team

ControlCase

Assuring Contact Center Experiences for Your Customers With ThousandEyes

ThousandEyes

Recently uploaded (20)

Connector Corner: Automate dynamic content and events by pushing a button

Bits & Pixels using AI for Good.........

UiPath Test Automation using UiPath Test Suite series, part 3

Essentials of Automations: Optimizing FME Workflows with Parameters

Generating a custom Ruby SDK for your web service or Rails API using Smithy

Key Trends Shaping the Future of Infrastructure.pdf

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

Epistemic Interaction - tuning interfaces to provide information for AI support

How world-class product teams are winning in the AI era by CEO and Founder, P...

JMeter webinar - integration with InfluxDB and Grafana

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

Knowledge engineering: from people to machines and back

Neuro-symbolic is not enough, we need neuro-*semantic*

Securing your Kubernetes cluster_ a step-by-step guide to success !

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

Elevating Tactical DDD Patterns Through Object Calisthenics

Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...

Monitoring Java Application Security with JDK Tools and JFR Events

PCI PIN Basics Webinar from the Controlcase Team

Assuring Contact Center Experiences for Your Customers With ThousandEyes

Top 5 Things I've Messed Up in Live Streaming

1. Top 5 things I’ve F’d up (In Live Streaming)

2. About me • GotFrag.com • Major League Gaming • Product Manager for Streaming Media at Fastly • Head of Product at Fastly

3. What’s this all about? • Live streaming (not live linear) • The (comedic) errors: 5. Forest, Trees. The big picture. 4. Latency sucks. 3. Failover. Not Fall Over. 2. Redundancy is not redundant. 1. Death by a thousand cuts. Measure it!

4. Some assumptions • What: – Live streaming is hard – It shouldn’t be. – But it is. – It’s real time. – So it’s hard. • Why: – “Analog” broadcast to “Digital” medium – Every step of the signal path can break it – And then there’s the Internet.

8. #5: The whole picture is important • Analog to Digital • Wires to Code • Complexity • Troubleshooting and QA is therefore hard • Studio != Touring Show

9. Live Streaming (Analog Side)

10. Live Streaming (Digital side)

11. #5: What got F’d. • Audio out of sync • Root cause was on the switcher <> audio board • Keyframes >1s on Encoder compounded problem

12. #4: Spoilers suck • Time Behind Live (TBL) is critical • Broadcast standard is <15s TBL • In other words, latency sucks. – Latency = Buffering, jitters – Latency = Spoilers – Latency = WTF just happened.

13. Latency Sources (Digital side) Source Ideal Avg Latency Encoding <100ms Uplink IP = <100ms, Sat = <1500ms Ingest <10ms Transcoding 50 - 500ms CDN 150 – 1500 ms Last Mile: Player / Browser / User Internet ????? Total: 3610ms (3.6sec) + Last mile

14. #4 What got F’d • RTMP delivery sucks on a CDN – 5min into broadcast = 5sec TBL – 30min into broadcast = 30sec TBL – 8hrs into broadcast = 2min TBL • Trolls hate spoilers. So do legit fans. • Separate stack from normal delivery, older hardware • HTTP streaming is the answer (HLS!!)

15. #3: Fail over. Not Fall over. • Duh.

16. #3: What got F’d • Just cause the vendor says so, doesn’t mean it _IS_ so. • Test it. • No really, TEST IT. • Can you fail back and forth? • Does it cause disconnects at player?

17. #2: Redundancy is not redundant • Analog – Matrixed encoders – Multiple uplinks – Back up routers • Digital – Cloud transcoding – Multi-CDN

18. #2: What got F’d • Production Budget – $10k/hr for sat truck • Upstream ISP link from venue died • Fix: Microwave pt to pt to the Luxor.

19. #1: Real time feedback. Flexible solutions Digital feedback: • Analytics. – Disconnects – Buffering – Edge usage – Load on systems – Geo distro – etc Analog feedback • Leverage the audience – Twitter • Watch the stream yourself • Filter the noise Flexible partners and solutions

20. #1: What got F’d • Tendency is to have digital side isolated from the analog side. • Don’t. Analog thinks about the room. Digital thinks about the “Internets” Put the two together.

21. Thank you www.fastly.com

Editor's Notes

Digital + analog How many of you have worked in a live production environment? This is actually a bit of a bias on an ops perspective, but when it comes to live streaming, devops doesn’t really work.
Duck on water…
Front of house. About 15,000 gamers in this room watching this stage. 500k peak concurrent viewers, about 25% on 720p – 1080p
Like a Duck on water, only the surface of the water is on fire – a fire that has been set by Trolls. Sadly, it seems that Google doesn’t have a picture of a duck floating on a lake of fire with Trolls holding torches… or else my google image-fu is weak. So this is the London Duck amphibious boat tour… on fire. I think you get the sentiment.
Single Points of Failure Suck

Top 5 Things I've Messed Up in Live Streaming

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (20)

Similar to Top 5 Things I've Messed Up in Live Streaming

Similar to Top 5 Things I've Messed Up in Live Streaming (20)

More from Fastly

More from Fastly (20)

Recently uploaded

Recently uploaded (20)

Top 5 Things I've Messed Up in Live Streaming

Editor's Notes