The story of how Globus helped move petabytes of data from the Arecibo Observatory to TACC, and thereby save 50+ years of data for posterity and future research. Presented at the GlobusWorld 2021 conference by George Robb III.

1. Arecibo Observatory
Data Movement - so much more than data
George B. Robb III, grobb3@es.net
EPOC - Performance Chaser
ESnet - Infrastructure Team
National Science Foundation Award #1826994

2. Arecibo Observatory
•Treasure of Astronomical Sciences since 1963
Inspiration for all, just a few highlights:
• Active radio telescope
• NEO, planetary, and atmospheric imaging
• Full spectrum 1Hz - 10 GHz
• Distance to Pleiades cluster
• Metric tons of scientific discoveries and publications.
https://www.naic.edu/ao/legacy-discoveries
5/12/2021 © 2021, Engagement and Performance Operations Center (EPOC) 2

3. Takes a hit, keeps on making science!
• 2006 - NSF 15% Budget cut across astronomical sciences
• 2007 - Arecibo budget slashed from$10.5 million to $8 million
(NASA pitches in with ~$2.6 million to help operations budget).
• 2011 - NSF implements Data Management Plan requirement.
• 2015 - Facilities director Kerr quits due to funding clashes.
• 2018 - University of Central Florida takes on stewardship.
Operations working tirelessly making science to happen.
5/12/2021 3
© 2021, Engagement and Performance Operations Center (EPOC)

4. Takes another hit.
• Hurricane Maria ( Sept 20, 2017 )
• Category 4
• Damage sustained
• R&E ( I2) network connection lost
and remained offline since 2012 to
UPR 10GbE !
• Too expensive to repair due to
budget cuts.
• Operations resume when the
power came back... And the
science goes on!
5/12/2021 4
© 2021, Engagement and Performance Operations Center (EPOC)

5. Takes another hit
5/12/2021 5
© 2021, Engagement and Performance Operations Center (EPOC)
•5.0 - 6.4 Earthquakes ( January 7-11, 2020 )
•Operations still going
• shaken not stirred ( yes 007 reference)

6. Takes and yet another
5/12/2021 6
© 2021, Engagement and Performance Operations Center (EPOC)
• Tropical Storm Isaias ( July 30, 2020 )
• And, SCIENCE vs operations
• still going…
• First [auxiliary] cable snap
• (Aug 10, 2020)
• Second cable [ primary ] snaps
• (Nov 6, 2020)

7. Thermodynamics, Economics, and Gravity win.
(Dec 1, 2020)
5/12/2021 7
© 2021, Engagement and Performance Operations Center (EPOC)
D
a
t
a
C
e
n
t
e
r
B
u
i
l
d
i
n
g
1
.

8. Thermodynamics, Economics, and Gravity win.
(Dec 1, 2020) - In a SECOND all changed.
5/12/2021 8
© 2021, Engagement and Performance Operations Center (EPOC)

9. 5/12/2021 9
© 2021, Engagement and Performance Operations Center (EPOC)
MISSION: Get the DATA to a safe stable state
• 2 Petabytes of Golden Copy need to move.
• Fiber is still cut!
• Transfers in MB/s with PB to go.
• Data movement at a scale of years
• Network Attached Storage (NAS) “Appliances” being used with sneakernet
• 100+ Terabytes at a time. ( Full capacity of NAS device )
• Onsite team hand carries this priceless data to the closest 10GbE
links at University of Puerto Rico - Mayaguez (RUM) and Engine-4
Collaboration Space.
Operations still going, harder than ever, hand carrying copies
of priceless scientific data to safety.

10. 5/12/2021 10
© 2021, Engagement and Performance Operations Center (EPOC)
Call for assistance:
• Team forms to see what can be done to get the data moving to
safe stable state ( we are in FULL DR ) .
• EPOC - Architect solutions to implement
• Globus - let's get the data on the wire
• Anyone want to saturate a link ( Also, redline a NAS )
• UPR (RUM) and Engine-4 - Provide transport 10GbE link!
• UCF - let’s facilitate the migrations
• TACC - let’s catch and secure the data

11. 5/12/2021 11
© 2021, Engagement and Performance Operations Center (EPOC)
Call for assistance - EPOC
• Worked with UCF teams to understand the nature
of the data.
• Architect course of action:
• People are the network (let’s help make this
happen).
• The the fabulous community really knocked
this out of the park THANK YOU

12. 5/12/2021 12
© 2021, Engagement and Performance Operations Center (EPOC)
Call for assistance - EPOC
• Understand Problem set:
• Site is not accepting new installations
• Oh, Global PANDEMIC
• We can’t just ship hardware.
• Closest 10GbE link an hour drive away.
• THANK YOU University of Puerto Rico (RUM) - UPR and Engine-4
• Network Attached Storage (movable storage
appliance) devices small but, mighty solutions
that can be tuned

13. 5/12/2021 13
© 2021, Engagement and Performance Operations Center (EPOC)
Call for assistance - TACC
•Offered to catch and distribute the data
• Waved a magical storage wand to create a safe
and secure 2 Petabyte landing zone.
• 10GbE link analysis and monitoring
• NetSage and dashboarding
• Globus native landing zone ( woo hoo )!

14. 5/12/2021 14
© 2021, Engagement and Performance Operations Center (EPOC)
Call for assistance - Globus
•Always part of the team and joined in person with
additional support - THANK YOU!
• rsync vs. globus
• ( spoiler: globus is an obvious winner )
• Timescale was years to move 2PB
• Data Transfer Node (DTN) tuning only helps!
>> Teams are pulling data from tape at this point!!!

15. 5/12/2021 15
© 2021, Engagement and Performance Operations Center (EPOC)
Move that data! - Globus

16. 5/12/2021 16
© 2021, Engagement and Performance Operations Center (EPOC)
Call for assistance -
so much more than data.
FULL Disaster Recovery - The instrument fell!
• Community engaged immediately
• Initial transfers 10s of Megabytes per second
• Tuned Globus transfers 100s of Megabytes per second
• Tools we have
• Globus - share that data!
• ESnet - https://es.net Fasterdata, DMZ, DTN, DME, Tuning, and much more.
• EPOC - https://epoc.global
• Supercharge your human network.
• We use only tools available, no such thing as silver bullets
• Hit the mission targets.

17. 5/12/2021 17
© 2021, Engagement and Performance Operations Center (EPOC)
Words of caution ( or fear ) - FULL Disaster
Recovery - THE INSTRUMENT FELL.
( think for a moment on this )
• The network is part of the instrument!
• NSF’s 2011 Data Management Plan requirement
• https://www.nsf.gov/bfa/dias/policy/dmp.jsp
• Large facilities what is the current status of:
• Data management plan
• Replica site
• Backups
• Fiber paths

18. 5/12/2021 18
© 2021, Engagement and Performance Operations Center (EPOC)
Words of caution ( or hope ) - FULL Disaster
Recovery - THE INSTRUMENT FELL.
• Call for assistance and engage your communities!
• EPOC - https://epoc.global epoc@iu.edu
• ESnet - https://es.net
• DTN - https://fasterdata.es.net/science-dmz/DTN/
• DME -
https://fasterdata.es.net/performance-testing/2019-2020-data-mobili
ty-workshop-and-exhibition/2019-2020-data-mobility-exhibition/
• ISI - Center of Excellence pilot, for Data Management
• https://www.nsf.gov/awardsearch/showAward?AWD_ID=1842042
National Science Foundation Award #1826994