The document outlines the use of Presto technology at FINRA for large-scale market surveillance, handling over 100 billion events and 25 petabytes of data from various exchanges. It discusses the challenges of data fragmentation and the solutions provided by migrating to a cloud-based data lake, utilizing AWS for workflow management and interactive querying. Additionally, it highlights the recent release of a managed data lake as open source and the potential future explorations with query tools and federated database access.

1. Presto at FINRA –
Supporting market
surveillance at scale
John Hitchingham
FINRA Engineering
John.Hitchingham@finra.org

2. Market Regulation surveillance workflow
BDs Exchanges Reference
Data Providers
100B+ events 25+ PB of Data 3+ Yrs ProdMajor Exchange Clients
Market Manipulation, Insider Trading, Fraud,
Abuse

3. Data volume
Incoming
records
• 6000+ business objects
• 7+ million data partitions
• 160+ million data objects
• 25+ data publishers
• 5+ PB of data

4. Data Fragmentation makes analytics difficult

5. Scale by separating storage and compute

6. Cloud Migration – Siloed Databases to Data Lake

7. Workflow in AWS Cloud

8. Herd Catalog
http://finraos.github.io/he
rd/

9. ETL

10. Isolate workloads and tune capacity per process

11. Interactive

12. Ad-Hoc query design for data lake

13. Main production “data warehouse” bucket growth…

14. Managed Data Lake (MDL) – Data Lake “in a box”
Just released as open source
Data lake implementation on
AWS
Featuring Presto as query
endpoint
https://finraos.github.io/herd-mdl
/

15. Portfolio of interactive apps on data lake

16. Data Science Ecosystem

17. Query tool use at FINRA
Hive Spark Presto HBase
Status Deprecated General use General use Limited use
Used For ETL/ELT ETL/ELT (replace
Hive)
Data Science
Machine Learning
Data Engineering
Data Profiling
BI
Reporting
Custom Apps
requiring rapid
“indexed” lookups

18. Future exploration with Presto
o CBO
o AuthN/AuthZ
• Hive metastore – column, row – Ranger?
• Federated database access (Postgres) – model to control authorization unique to principal
• Federated AuthN (SAML, OAuth)
o Athena?