Here, I describe how the Texas Advanced Computing Center has shifted its focus from traditional modeling and simulation towards fully embracing big data analytics performed by users with diverse technical backgrounds.

1. SCALING PEOPLE, NOT JUST
SYSTEMS, TO TAKE ON BIG DATA
CHALLENGES
Matthew Vaughn @mattdotvaughn
Director, Life Sciences Computing, TACC
Co-PI Cyverse, Araport, Jetstream Cloud
2/1/2016 1

2. TACC AT A GLANCE
2/1/2016 2
Personnel
160 Full time staff (~70 PhD)
Facilities
10 MW Data center capacity
New office facility to be completed by
start of 2016
Systems and Services
A Billion compute hours per year
5 Billion files, 50 Petabytes of Data,
Hundreds of Public Datasets
Capacity & Services
HPC, HTC, Visualization, Large scale data
storage, Cloud computing
Consulting, Curation and analysis, Code
optimization, Portals and Gateways, Web
service APIs, Training and Outreach

3. Stampede
• #7 HPC system in the world for
computation 500k CPU core 9.7 PF
Lonestar 5
• Texas-focused Cray XC40 30,000
Intel Haswell cores 1.25 PF
Wrangler
• 0.6 PB usable DSSD flash storage w
1 TB/s read rate + 10 PB Lustre
Maverick
•132 Fat nodes w dual 10 core Ivy
Bridge + NVIDIA Kepler K40 GPGPU
Chameleon & Jetstream Cloud
•1400 nodes OpenStack
Disk and Tape Storage
• 100+ PB storage in HIPAA-aligned
data center
EXTREME SCALE SUPERCOMPUTING
Coming Soon: Hikari
• Green computing
system parternship with
NEDO and NTT. 10k
Haswell cores. HVDC
and Solar (partial)
• Support for container
ecosystem
2/1/2016 3

4. TACC’S STAMPEDE SUPPORTS AN INCREDIBLE
AMOUNT & DIVERSITY OF RESEARCH
• Through <3 years of production operations:
• Over *2 Billion* processor hours delivered to end users
• 6+ million successful jobs
• About 10,000 students, faculty, and staff use our Stampede directly
• Over 30,000 more use it indirectly via portals and services
• Peer-reviewed requests for time (via XSEDE) run ~500% available hours
• Stampede alone supports nearly 2,500 funded projects across the United
States and abroad
2/1/2016 4

5. Firefly
Space
Systems
Firefly, a TACC
industrial partner, has
begun engine tests
shortly on rocket
engines designed
using simulations run
on Stampede.
2/1/2016 5

6. Plasma
Physics
Researchers at the
Institute for Fusion
Studies are using
Stampede to model
plasma flow and
turbulence in both
fusion power-
generating tokomaks,
and in the ionosphere
(with applications to
GPS).
2/1/2016 6

7. Finding our
Ancestors
Genomics analysis
performed on Stampede
compared the genomes
of nine ancient
Europeans that bridge
the transition to
agriculture in Europe
between 8,000 and 7,000
years ago, revealing that
most present-day
Europeans derive from at
least three highly
differentiated
populations.
2/1/2016 7

8. 2/1/2016 8

9. THE EVOLUTION OF A CYBERINFRASTRUCTURE
2/1/2016 9
Once upon
a time, most
of us built
garage-style
clusters…
HPC systems have grown
up since then and become
much more powerful and
sophisticated
We built a very successful
center based on 3 pillars in
our first 7 years
• Simulation & HPC
• Visualization
• People (Consulting &
Algorithms)
But a curious
thing happened
while HPC was
running a
victory lap…

10. NOT JUST ONE DATA TSUNAMI BUT THOUSANDS OF THEM
2/1/2016 10

11. DNA SEQUENCING COSTS DECREASED FROM $1B/GENOME
IN 2000 TO $500 in 2015
2/1/2016 11

12. DOZENS OF NEW DISRUPTIVE NEW MEASUREMENT TECHNOLOGIES EMERGED
AND THEY ALL GENERATE GBs or TBs OF DATA
2/1/2016 12

13. 2/1/2016 13
THE TITANS JUST KEPT GETTING BIGGER

14. KEY CHALLENGES EMERGING FROM
THE BIG DATA REVOLUTION
2/1/2016 14
 Data generation occurs at higher volume and is
geographically dispersed
 More researchers are asking questions that are too
big for local-scale computing
 Research teams are larger, more distributed, and
technologically heterogeneous
 Workflows have increasingly complex hardware and
software requirements
 Data and software sharing, while expected, is still
too hard Science and engineering hinges on managing
and analyzing large data, so TACC's strategy
embraces data and its analysis right alongside
computing

15. Mike
• Computing novice
• Works remotely at partner
site
Eliza
• Masters specific
analysis skills
• Readily adopts new
tech
Roshan
• Computationally
experienced
• Focused on interpretation
Paulo
• Staff computational
expert
• Supports multiple
projects
DIVERSE DISTRIBUTED RESEARCH TEAMS
2/1/2016 15
Nikolaidas Group
• Mostly
experimentalists
• Strict data sharing
& access

16. THEIR NEEDS (30,000 FT. VIEW)
 Store, organize, share primary data
 Iteratively perform 1’ analyses
 Store, organize, share derived data products
 Iteratively generate and explore hypotheses
 Share analytical code with the scientific public
 Integrate results from new experiments
 Publish data alongside plots, visualizations and
analytical tools
2/1/2016 16

17. THEIR NEEDS (500 FT. VIEW)
 Data lifecycle management
 Fine-grained permission management
 Discoverability
 Version control
 Domesticating promising new analysis codes based
on often immature technology
 Doing reproducible computational science
 Adopting efficient analytical methods
2/1/2016 17

18. WORKFLOWS NOW TECHNICALLY COMPLICATED
2/1/2016 18
LANGUAGES
• Python 2 & 3
• R
• Julia
• Perl
• Matlab
• Java
• Scala, Clojure, etc
• .NET
• C++
• Swift
• Haskell
• Go
• Javascript
FRAMEWORKS
• MapReduce Hadoop, Storm,
Pachyderm
• Event & Streaming: Kinesis,
Azure Stream Analytics, Camel,
Streambase
• Deep/Machine Learning: Watson,
Azure BI, Tensorflow
• In-memory parsing: Kognito,
Apache Spark
• New data warehouse: Snowflake
• Containers: Docker, Rocket,
MESOS, Kubernetes
• Cloud: AWS, GCE, OpenStack,
VMWare
HARDWARE
•Rise of many-core computing
means 50-100 threads/node*
• Xeon / Xeon Phi
• GPU
• OpenPower
• ARM
• Multi-level memory architectures
• Hierarchical storage
architectures
• FPGAs

19. WORKFLOWS NOW TECHNICALLY COMPLICATED
2/1/2016 19
LANGUAGES
• Python 2 & 3
• R
• Julia
• Perl
• Matlab
• Java
• Scala, Clojure, etc
• .NET
• C++
• Swift
• Haskell
• Go
• Javascript
FRAMEWORKS
• MapReduce Hadoop, Storm,
Pachyderm
• Event & Streaming: Kinesis,
Azure Stream Analytics, Camel,
Streambase
• Deep/Machine Learning: Watson,
Azure BI, Tensorflow
• In-memory parsing: Kognito,
Apache Spark
• New data warehouse: Snowflake
• Containers: Docker, Rocket,
MESOS, Kubernetes
• Cloud: AWS, GCE, OpenStack,
VMWare
HARDWARE
•Rise of many-core computing
means 50-100 threads/node
• Xeon / Xeon Phi
• GPU
• OpenPower
• ARM
• Multi-level memory architectures
• Hierarchical storage
architectures
• FPGA

20. HOW CAN WE HELP RESEARCHERS WITH SUCH
DIVERSE NEEDS AND BACKGROUNDS?
2/1/2016 20

21. BUILD A MASSIVE STORAGE CLOUD NEXT TO INNOVATIVE, POWERFUL, USABLE
COMPUTERS AT THE END OF FAST INTERNET PIPES
2/1/2016 21

22. MANY DOMAIN SCIENTISTS ARE NOT EXPERTS AT COMPUTING TECHNOLOGY.
CREATE PURPOSE-BUILT, HIGHLY INTUITIVE INTERFACES
2/1/2016 22

23. TACC CREATES A LOT OF SOFTWARE…
2/1/2016 23
 Over half of our staff build software
to empower data-intensive science
communities
 Earthquake/Flood/Wind Engineering
 Genomics and Genetics
 Immunology
 Geology
 Drug Discovery
 Archaeology
 Infectious Disease
 Many more…
designsafe-ci.org vdjserver.org
digitalrocksportal.org chameleoncloud.org
jetstream-cloud.org araport.org
cyverse.org (née iplantc.org) xsede.org

24. Point-and-click interfaces
• Data management,
sharing, and analysis
• Publishing reproducible
analysis workflows
• Discovery of new or
updated tools and data
• Interactive visualization
of results
Backed by world-class
computing and data
capacity
2/1/2016 24

25. 2/1/2016 25
Hosted SaaS
• JupyterHub notebooks
• Rstudio
• Web-based VNC
Also, backed by world-class
computing and data
capacity

26. Easy to use Cloud Computing
• Atmosphere (Cyverse)
• Jetstream (IU,UA,TACC)
• Chameleon (UC,TACC )
Cloud consoles are aimed at
sysadmins and unintutive.
We’re changing that with
improved UX and support
• APIs are still available
• No cost to end user
2/1/2016 26

27. GIVE EXPERTS ACCESS TO EVERY SINGLE ONE OF YOUR BUILDING BLOCKS.
WEB SERVICE APIs EVERYWHERE. AUGMENT WITH PROFESSIONAL TOOLING.
2/1/2016 27

28. Agave API
http://agaveapi.co/
2/1/2016 28

29. TACC Ecosystem
XSEDE Partners
Local & institutional
resources
Commercial cloud providers
ENABLE “BRING YOUR
OWN RESOURCES”
2/1/2016 29

30. KNIT COMPLEX SCIENCE INFRASTRUCTURE TOGETHER WITH
USEFUL ABSTRACTIONS
2/1/2016 30
Reference Design: Extensible Research Infrastructure
Web site Workspace Data Portal Learning 3rd Party Dev Zone
Science APIs IAM Data Movement Smart Search
iRODS* Shibboleth/Oauth2 OpenStack & Docker
Ease of Use
Ease of Re-use
Extensibility
Capability

31. MIKE: LEARNING COMPUTATIONAL SKILLS
Mike generates mass-spec data from his samples, which
are used to decide on future experiments
 Eliza and Paulo have published analytical scripts for
his data as ‘apps’ into a project web portal
 Paulo has wired up the lab’s equipment to send
data directly to remote storage
 Mike can perform basic analysis and reporting on
his data from the web interface
 Roshan can see Mike’s results in the project portal
and discuss them with him
 Mike collaborates with Eliza to improve the results
of their analytical scripts

32. ELIZA: AUGMENTING HER RESEARCH CAPABILITY
Eliza collects and analyzes in situ hybridization imagery
as a major part of her inquiry
 She uses code developed by Paulo and others to
perform feature detection
 She helps Paulo develop code for Mike to use in his
proteomics work
 She has developed scripted workflows that run
locally on her laptop to complete her analyses
 She writes her own code in Python and R for
aggregate analysis and shares it via the Docker Hub
and the project portal
 She presents her data viz via interactive Rshiny apps
that she deploys to the project portal

33. PAULO: CONCENTRATING ON COMPUTATIONAL
RESEARCH
Paulo likes to solve hard computational problems but is also
responsible for lab research infrastructure
 He has automated data movement from instruments to remote
storage, including duplication to AWS Glacier. Roshan gets the
bill 
 He builds and maintains the project web portal. He didn’t have
much experience with such technology when the project
started
 Paulo has used Spark to developed new software for feature
extraction from in situ hybridization images – he can
automatically deploy it to the portal, powered by TACC
Wrangler, as part of his build process
 He is working on a paper describing his software and is getting
valuable fedback from other folks he has shared it with via the
project portal and public source repositories

34. ROSHAN: FOCUSING ON THE BIG PICTURE
 Roshan has deep experience in gene expression analysis -
Paulo has populated the project portal with many of the
tools she needs to accomplish her goals
 She collaborates with Mike and Eliza on interpreting their
experimental results
 Because Eliza has included a custom notification in her
scripting workflow, Roshan knows when new image
analyses have been completed and can schedule time to
look at them
 She can work with Paulo to enable the project web portal
to make use of her newly-awarded XSEDE computing and
storage allocation
 She routinely shares results with experimentalist
colleagues and reviewers via a simple Dropbox-like
interface

35. FUTURE DIRECTIONS AND CHALLENGES
2/1/2016 35
Technological Challenges Remain…
• Data volume
• TACC has capacity for 160PB today. Moore’s law curve holding for tape and disk (whew). But, POSIX
isn’t sustainable long-term.
• Data velocity
• We routinely move 5 TB/h between large research universities.
• Network bandwidth is still scaling, but data spread is getting worse.
• Ubiquitous streaming data is coming.
• Massive Parallelism
• All server-class chips will have 40+ cores in next 18mo.
• Performant software will require embrace of parallelism at instruction, thread and task levels.
• High Transaction Rate I/O
• Flash & NVRAM allow architecting of massive speed systems for small I/O (graph traversal, database,
etc).
• TACC Wrangler can do 800GB per second of I/O.

36. FUTURE DIRECTIONS AND CHALLENGES
2/1/2016 36
But also…
• Collaboration and Sharing
• Noisy Data QA/QC (Quality Assurance/Quality Control)
• Provenance and Reproducibility
• Managing Metadata
• Compliance and Data Security
TACC’s Systems and Expertise Change How Discovery is Done, and Change the
World

37. WE’RE HIRING
Life Science, Data Science, Machine Learning, Web and
Mobile, Cloud, HPC, Sysadmin
jobs@tacc.utexas.edu
2/1/2016 37
FOLLOW US
@TACC
fb.com/tacc.utexas
www.tacc.utexas.edu