The document discusses how ESPN utilizes Clojure to automate video clip processing for astronauts on the International Space Station, emphasizing the importance of immutability and concurrency within the Clojure framework. It details various operations, deployment techniques, and error handling strategies implemented in the project, which has been operational since September 2013. Additionally, it highlights the integration of a REST API and ongoing improvements, including future plans for real-time functionality.

1. NASAmatic
How ESPN uses Clojure to supply videos to
Astronauts
Prasanna Gautam
Data Platform and Architecture, ESPN
(05/21/2014 - NewHaven.IO)

2. What this talk is not about
● Video transcoding details
● Functional Programming intricacies
o It is about how awesome immutability is, however!

3. What this talk is about
● Establishing Clojure as a robust tool
o In the Enterprise
o Making DevOps comfortable
o For iterating fast!!
● Getting a sweet trip to Houston
o Hanging out with astronauts at Mission Control
 And Dwight Howard

4. Houston -> Bristol
● Can you send us ESPN programming for
astronauts in ISS?
o Why yes!
 Let's hire people to clip videos every day!

5. But it's 2013
● We should automate this!
● We should remove ads!
● Each file should fit within the 200MB limit!
● It should run every day on time!

6. How?
One would assume...
… and done!!!

7. In reality
● Someone presses a button to notify
breaks/program segments (for live
programs)
o Supplying data to us is 1/1000s of tasks they do!!
● Broadcast systems are off-limits to meddle
with

8. Why Clojure?
● Lisp-dialect, runs on JVM
o Great at interop with Java
● Immutability and Concurrency at the core
● Values simplicity
● I've used it for 5 years
o Wrote a simulator and a compiler generator in
College in Clojure

9. First Commit
(defn -main
"I don't do a whole lot ... yet."
[& args]
(doseq [program (get-all-espn-programs :from (t/date-midnight 2013 07 16)
:to (t/date-midnight 2013 07 17))]
(let [title (s/replace (:title program) #"s+" "_")
simple-formatter (f/formatter "hhmm")
start-str (f/unparse simple-formatter (:start-time program))
end-str (f/unparse simple-formatter (:end-time program))
filename (str (s/join "-" [title start-str end-str]) ".mp4")
]
(println "getting streams and combining chunks for program " (:title program ) (:start-time program) (:end-time program))
(get-combined-chunks-from-streams (program-segment-streams-for-program program 95 "ESPNHD") "/tmp/combined.temp")
(println "hotwrapping")
(hotwrap-mp4 "/tmp/combined.temp" (str "NASAMATIC/2013-07-16/" filename))
)
)
)

10. Ops
● Functional programs still
needs ops team
o They don't care about your
FP patterns
 They shouldn't need to
o Make configs declarative

11. Configuration and scheduling
● clj-yaml
● quartzite
programs:
- name: AROUND THE HORN
short_name: ATH
start_time: "20:00:00"
- name: PARDON THE INTERRUPTION
short_name: PTI
start_time: "20:30:00"
- name: SPORTSCENTER
short_name: SportsCenter
start_time: "14:00:00"
run:
cron: 0 0 14 1/1 * ? *
final_tz: America/Anchorage

12. Options
● Need to fix bugs
o without changing
code every time
o org.clojure/tools.cli
Switches Default Desc
-------- ------- ----
-c, --config nasamatic.yml Use this config file path
-h, --no-help, --help false Show Help
-f, --no-force, --force false Force run now instead of using Cron
-u, --no-upload, --upload true Upload or not
-t, --no-transcode, --transcode true Transcode or not
-B, --hours-before-now 0 How many hours before now to look at
-d, --no-dry-run, --dry-run false Dry Run modeOptions
Scheduled -- using quartz
Forced -- without quartz, or bugfixing

13. Deployment
● CI - Jenkins! (Automatic)
o Leiningen plugin!
● It's just a runnable jar file!!
o We have Chef recipes for that!
● Logging
o No problems! (org.clojure/tools.logging)
o Splunk Forwarders with Chef already!
● Alerts
o Hmm...

14. Enter macros
(defmacro do-with-log
" Works functionally like a do block -- more or less, it runs all the given forms in order and returns the output of the last form it ran.. It logs when the job
started, ended or when it runs into any problems. It logs the error and rethrows the Throwable upstream."
([[job-name name & {:keys [trace-id] :or {trace-id (str "trace-" (rand-int 100000))}}] & body]
(if-not name
(throw (IllegalArgumentException. "You want to provide a name for the block you want to run.")))
`(let [out# (atom nil)
start-time# (System/currentTimeMillis)
~job-name (str ~name)
~'trace-id (str ~trace-id)
]
(info-m "job" ~job-name "status" "Started" "trace-id" ~trace-id)
(reset! out# (try
~@body
(catch Throwable e#
(error-m "job=" ~job-name "status" "Error" "trace-id" ~trace-id "message" e#)
(throw e#))))
(info-m "job" ~job-name "status" "Ended" "trace-id" ~trace-id "time_taken" (str (- (System/currentTimeMillis) start-time# ) "ms"))
@out#)
)
)

15. Alerts
2014-05-20 00:28:26 INFO utils-verify:1 - trace-id=trace-94295, status=Started, job=sleeps
2014-05-20 00:28:27 INFO utils-verify:1 - trace-id=trace-94295, status=Started, job=throws-error
2014-05-20 00:28:27 ERROR utils-verify:1 - job==throws-error, trace-id=trace-94295, message=java.lang.Throwable: Boo! I errored Out, status=Error
2014-05-20 00:28:27 ERROR utils-verify:1 - job==sleeps, trace-id=trace-94295, message=java.lang.Throwable: Boo! I errored Out, status=Error

16. Handling failure
● Try again
o configurable number of
times
● Give up until another day!
o Let the humans figure it out
error_handling:
retry:
enabled: true
tries: 2
wait: 1800

17. Results
● Been operational since 9/9/13
o 8 months, 12 days
o Press Release:
http://frontrow.espn.go.com/2013/09/astronaut-chris-
cassidy-introduces-sportscenters-top-10-plays-from-
240-miles-above-earth/
o Built on experimental systems
● Added Baseball Tonight in early April
● Integrated a REST API in December

18. Further
● Easy to extend
o After some examples, a REST API was integrated
easily by Justin Burrous
 No previous clojure experience
 Without any breakage in service
 Uses the same options and configurations - as
JSON
● Has been invaluable in understanding
broadcast systems

19. Todos
● Scheduled/Batch was Plan B
o Need to test realtime version (Plan A)
● Notify web-ui of what step the quartz job is in
● NewRelic integration -- better traces
● Core.async patterns

20. Questions?