Presented by Milosz Tanski, AdFin

1. OLAP ON QERIES IN SECONDS ON PETABYTE DATASET
Distributing Petabucket data using CephFS
Milosz Tanski, CTO @Adfin
milosz@adfin.com
October 2014

2. Outline
 Who/what is AdFin?
 What is PetaBucket?
 Petabucket on CephFS
 Contributing FSCache support to CephFS
2 ©AdFin. All Rights Reserved

3. About Adfin
 = Ad-Tech + Finance-Tech
 Creating tools that bring buying intelligence to programmatic media.
 Advertising is bought and sold in real time via RTB (since 2008)
 Brining transparency to the Ad markets.
 The Bloomberg, S&P, Markit… for Ad markets.
3 ©AdFin. All Rights Reserved

4. We Deliver… Pretty Analytics
4 ©AdFin. All Rights Reserved

5. We Deliver… Pretty Analytics
5 ©AdFin. All Rights Reserved

6. We Deliver… Pretty Analytics
6 ©AdFin. All Rights Reserved

7. We Deliver… Pretty Analytics
7 ©AdFin. All Rights Reserved

8. What’s the problem?
 Market is ~500 Billion impressions a day; it’s growing.
 Each impression is unique.
 Each is worth a small fraction of a penny.
 Magnitude more then number of trades in the Financial markets
 There’s a magnitude more bids for those impressions.
 That’s a lot of data to process, store, analyze.
8 ©AdFin. All Rights Reserved

9. Petabucket
 Distributed, time series, relational, OLAP database
 Relational query language (but not SQL)
 Query in broken up into many smaller chunks
 Great single node performance. 10s of millions rows a second.
 Vectorized query processing, vectorized compressed bitmap indexes.
 Responses in real-time. Goal is low single digit seconds (uncached)
 Why? Because we’re a bit crazy.
9 ©AdFin. All Rights Reserved

10. Queries easy for humans / machines
10 ©AdFin. All Rights Reserved

11. High Level System Diagram
11

12. Time series bulk import
12

13. Petabucket and CephFS
 CephFS as a single namespace storage for nodes
 Why?
 Scalable storage (speed / size)
 Separate storage from computation
 No SPOF
 DFS performance
 Client (kernel) performance
13 ©AdFin. All Rights Reserved

14. High Level System Diagram, part 2
14 ©AdFin. All Rights Reserved

15. CephFS is not production ready?
 Again, we’re a bit crazy?
 Started in early 2013.
 When we started client and MDS were not ready.
 We found and reported a lot of bugs.
 Yan Zhen fixed a lot of bugs. Thanks Yan.
 Today we’re happy and in production.
 Processed multiple PB of data since then.
15 ©AdFin. All Rights Reserved

16. FSCache for kclient
 We decided to add local persistent caching support to the kclient.
 Our access pattern:
 Working set larger then node memory (page cache)
 Append-only data (time series)
 Most recent month, quarter of data access 100x more often
 Benefits:
 Reducing latency / speed lost by moving to non-local filesystem
 Reduce Ceph network traffic and OSD utilization
 Cheap local SSD drives get 500MB/s read performance
 Not re-inventing the wheel
16 ©AdFin. All Rights Reserved

17. Kernel programming is hard
 Have to understand Ceph, kernel, concurrency.
 An error in the kernel hangs or Oops your machine.
 Bugs in other parts of the kernel? (CacheFS).
 Prototype working in two weeks
 First submission 2 months later.
 In kernel 5 months later.
 Number one problem concurrency.
17 ©AdFin. All Rights Reserved

18. Ceph with FSCache Status
 In since: 3.13
 … Works well since: 3.15
 … All bugs fixed: 3.17
 Speed… as fast as your caching disk
 Tested single client performance 1200MB/s
18 ©AdFin. All Rights Reserved

19. Next steps…
 Contributing to Ceph & kernel is addicting:
 Ceph performance work. Improving latency / ioops.
 Kernel work: readv2() syscall. File serving applications
 http://lwn.net/Articles/612483/
19 ©AdFin. All Rights Reserved

20. Thank You!

21. Let’s Get in Touch
21 ©AdFin. All Rights Reserved
Milosz Tanski
CTO
milosz@adfin.com
16 E. 34th Street, 15th Floor
New York, New York 10016
linkedin.com/company/AdFin
twitter.com/AdFin