Opened Captions
•Download as KEY, PDF•
3 likes•661 views
The document discusses Opened Captions, a real-time distributed closed captioning system. It describes extracting captions from video using OCR, speech-to-text and existing closed captions. Captions are distributed to clients via a websocket server and can be modified through content forking to support different languages or clients. The current setup is described along with a vision for scaling the system using AWS services like receivers, relays and load balancers.
Recommended
More Related Content
Similar to Opened Captions
Similar to Opened Captions (20)
More from Daniel Schultz
Recently uploaded
Recently uploaded (20)
Opened Captions
- 1. OPENED CAPTIONS Real-time distributed closed captioning syndication Dan Schultz @slifty
- 2. But First...
- 3. My professional life in 30 seconds 2005 2006 2007 2008 2009 2010 2011 2012 2013 Carnegie Mellon Information Systems
- 4. My professional life in 30 seconds 2005 2006 2007 2008 2009 2010 2011 2012 2013 Carnegie Mellon Philly ColorQuick Speaker Text PBS
- 5. My professional life in 30 seconds 2005 2006 2007 2008 2009 2010 2011 2012 2013 Carnegie Mellon Philly MIT Information Ecology Civic Media
- 6. My professional life in 30 seconds 2005 2006 2007 2008 2009 2010 2011 2012 2013 Carnegie Mellon Philly MIT OpenNews The Boston Globe
- 8. Knight-Mozilla OpenNews is about building an ecosystem to help journalism thrive on the open web. mozillaopennews.org
- 10. Back to Opened Captions!
- 11. The Problem: Writing applications based on what’s happening on TV is damn near impossible.
- 13. Step 1: Extract
- 14. OCR Audio Analysis Speech to Text Closed Captioning Video Analysis Face Recognition Fingerprinting
- 15. OCR Audio Analysis Speech to Text Closed Captioning Video Analysis Face Recognition Fingerprinting
- 18. Line 21 Decoding
- 22. Information Flow Extract Video Feed Opened Captions Server
- 23. Information Flow Distribute Extract Opened Captions Clients Video Feed Opened Captions Server
- 24. TRADITIONAL WEB INTERACTIONS Request Response
- 25. FOR INSTANCE.... Client Says Server Responds Show me the front page of the NY Times Sure thing, enjoy reading glorious journalism!” Do I have any new emails? Here are your 423 new emails you popular netizen I DEMAND ANIMATED GIFS You got it, boss, I proudly present Obama slicing a salami with his hands
- 26. HOW WOULD YOU SAY... Let me know when... This is awkward. Keep me updated with... I was kind of thinking our relationship would be a “one time” deal. Push me a notice if...
- 27. POLLING The Younger Sibling Approach Do you have any new tweets? How bout now? What about now? I don’t know if you knew this, but I’m looking for new tweets I’m betting there are some now. New tweets? You’ll never guess what I’m about to ask for. Holy crap. I’m gonna go shut down now. It’s been like 10 seconds you have to have some by now.
- 28. WEBSOCKETS Handshake Messages
- 31. Iteration 1: Hackable Easy to set up Easy to code against Easy to modify Iteration 2: Robust Reliable Scalable Iteration 3: Powerful Multiple Channels Metadata
- 32. Easy to Set Up
- 34. Easy to Code Against
- 37. Easy to Modify
- 40. Content Forking Video Feed Opened Captions Server
- 41. Content Forking Clients Video Feed Opened Captions Server
- 42. Content Forking Clients Video Feed Opened Captions Server Content Fork
- 43. Content Forking Clients Video Feed Opened Captions Server Content Fork Clients
- 44. Content Forking English-speaking clients Video Feed Opened Captions Server Content Fork (Spanish Translation Mod) Spanish-speaking clients
- 45. Scalable
- 47. The Current Setup Video Feed (C-SPAN)
- 48. The Current Setup Globe Lab Computer Video Feed (The Boston Globe) (C-SPAN)
- 49. The Current Setup Globe Lab Computer Video Feed (The Boston Globe) Opened Captions Server (C-SPAN) (MIT)
- 50. The Current Setup Globe Lab Computer Video Feed (The Boston Globe) Opened Captions Server (C-SPAN) (MIT) Demo Clients
- 51. The Current Setup Globe Lab Computer Video Feed (The Boston Globe) Opened Captions Server (C-SPAN) (MIT) Content Forks Demo Clients
- 52. The Vision
- 54. The Vision Opened Captions Receiver (AWS) Video Feeds
- 55. The Vision Opened Captions Receiver (AWS) Video Feeds Relays (AWS)
- 56. The Vision Load Balancer (AWS) Opened Captions Receiver (AWS) Video Feeds Relays (AWS)
- 57. The Vision Load Balancer (AWS) Opened Captions Receiver (AWS) Video Feeds Relays (AWS) Clients and Content Forks
- 58. The Vision Load Balancer (AWS) Opened Captions Receiver (AWS) Video Feeds Relays (AWS) Clients and Content Forks
- 59. Dan Schultz @slifty
Editor's Notes
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- \n
- I won’t get too geeky here, but every Opened Captions instance actually re-broadcasts its caption stream so that other servers can build from it. For instance maybe you want to translate it to another language.\n
- I won’t get too geeky here, but every Opened Captions instance actually re-broadcasts its caption stream so that other servers can build from it. For instance maybe you want to translate it to another language.\n
- I won’t get too geeky here, but every Opened Captions instance actually re-broadcasts its caption stream so that other servers can build from it. For instance maybe you want to translate it to another language.\n
- \n
- \n
- \n
- \n
- \n
- I’m actually only halfway through my fellowship, so I’ll just share a recent project called Opened Captions. This makes it really easy to build experiences around live TV by creating a hackable Closed Captioning stream. Your code knows every time a word is spoken.\n
- I’m actually only halfway through my fellowship, so I’ll just share a recent project called Opened Captions. This makes it really easy to build experiences around live TV by creating a hackable Closed Captioning stream. Your code knows every time a word is spoken.\n
- \n
- \n
- \n
- \n
- \n
- Now you can create pages that react based on what’s being said on TV. Alving Chang built this quick hack using Opened Captions, called CardText, which augmented the presidential debates by showing extra information about people, places, and issues as they were being discussed on screen.\n
- \n
- \n
- I won’t get too geeky here, but every Opened Captions instance actually re-broadcasts its caption stream so that other servers can build from it. For instance maybe you want to translate it to another language.\n
- I won’t get too geeky here, but every Opened Captions instance actually re-broadcasts its caption stream so that other servers can build from it. For instance maybe you want to translate it to another language.\n
- I won’t get too geeky here, but every Opened Captions instance actually re-broadcasts its caption stream so that other servers can build from it. For instance maybe you want to translate it to another language.\n
- I won’t get too geeky here, but every Opened Captions instance actually re-broadcasts its caption stream so that other servers can build from it. For instance maybe you want to translate it to another language.\n
- I won’t get too geeky here, but every Opened Captions instance actually re-broadcasts its caption stream so that other servers can build from it. For instance maybe you want to translate it to another language.\n
- I won’t get too geeky here, but every Opened Captions instance actually re-broadcasts its caption stream so that other servers can build from it. For instance maybe you want to translate it to another language.\n
- \n
- I won’t get too geeky here, but every Opened Captions instance actually re-broadcasts its caption stream so that other servers can build from it. For instance maybe you want to translate it to another language.\n
- I won’t get too geeky here, but every Opened Captions instance actually re-broadcasts its caption stream so that other servers can build from it. For instance maybe you want to translate it to another language.\n
- I won’t get too geeky here, but every Opened Captions instance actually re-broadcasts its caption stream so that other servers can build from it. For instance maybe you want to translate it to another language.\n
- I won’t get too geeky here, but every Opened Captions instance actually re-broadcasts its caption stream so that other servers can build from it. For instance maybe you want to translate it to another language.\n
- I won’t get too geeky here, but every Opened Captions instance actually re-broadcasts its caption stream so that other servers can build from it. For instance maybe you want to translate it to another language.\n
- I won’t get too geeky here, but every Opened Captions instance actually re-broadcasts its caption stream so that other servers can build from it. For instance maybe you want to translate it to another language.\n
- I won’t get too geeky here, but every Opened Captions instance actually re-broadcasts its caption stream so that other servers can build from it. For instance maybe you want to translate it to another language.\n
- I won’t get too geeky here, but every Opened Captions instance actually re-broadcasts its caption stream so that other servers can build from it. For instance maybe you want to translate it to another language.\n
- I won’t get too geeky here, but every Opened Captions instance actually re-broadcasts its caption stream so that other servers can build from it. For instance maybe you want to translate it to another language.\n
- I won’t get too geeky here, but every Opened Captions instance actually re-broadcasts its caption stream so that other servers can build from it. For instance maybe you want to translate it to another language.\n
- I won’t get too geeky here, but every Opened Captions instance actually re-broadcasts its caption stream so that other servers can build from it. For instance maybe you want to translate it to another language.\n
- I won’t get too geeky here, but every Opened Captions instance actually re-broadcasts its caption stream so that other servers can build from it. For instance maybe you want to translate it to another language.\n
- I won’t get too geeky here, but every Opened Captions instance actually re-broadcasts its caption stream so that other servers can build from it. For instance maybe you want to translate it to another language.\n
- \n