AdFin is a company that provides analytics tools for programmatic advertising markets to bring transparency. They developed PetaBucket, a distributed, relational OLAP database that can query a petabyte dataset in seconds. AdFin uses CephFS for scalable storage across petabyte datasets and nodes. They contributed code to add local caching support to the Ceph kernel client to improve performance for their workload of querying recent time-series data more frequently.

1. OLAP ON QERIES IN SECONDS ON PETABYTE DATASET
Distributing Petabucket data using CephFS
Milosz Tanski, CTO @Adfin
milosz@adfin.com
October 2014

2. Outline
 Who/what is AdFin?
 What is PetaBucket?
 Petabucket on CephFS
 Contributing FSCache support to CephFS
2 ©AdFin. All Rights Reserved

3. About Adfin
 = Ad-Tech + Finance-Tech
 Creating tools that bring buying intelligence to programmatic media.
 Advertising is bought and sold in real time via RTB (since 2008)
 Brining transparency to the Ad markets.
 The Bloomberg, S&P, Markit… for Ad markets.
3 ©AdFin. All Rights Reserved

4. We Deliver… Pretty Analytics
4 ©AdFin. All Rights Reserved

5. We Deliver… Pretty Analytics
5 ©AdFin. All Rights Reserved

6. We Deliver… Pretty Analytics
6 ©AdFin. All Rights Reserved

7. We Deliver… Pretty Analytics
7 ©AdFin. All Rights Reserved

8. What’s the problem?
 Market is ~500 Billion impressions a day; it’s growing.
 Each impression is unique.
 Each is worth a small fraction of a penny.
 Magnitude more then number of trades in the Financial markets
 There’s a magnitude more bids for those impressions.
 That’s a lot of data to process, store, analyze.
8 ©AdFin. All Rights Reserved

9. Petabucket
 Distributed, time series, relational, OLAP database
 Relational query language (but not SQL)
 Query in broken up into many smaller chunks
 Great single node performance. 10s of millions rows a second.
 Vectorized query processing, vectorized compressed bitmap indexes.
 Responses in real-time. Goal is low single digit seconds (uncached)
 Why? Because we’re a bit crazy.
9 ©AdFin. All Rights Reserved

10. Queries easy for humans / machines
10 ©AdFin. All Rights Reserved

11. High Level System Diagram
11

12. Time series bulk import
12

13. Petabucket and CephFS
 CephFS as a single namespace storage for nodes
 Why?
 Scalable storage (speed / size)
 Separate storage from computation
 No SPOF
 DFS performance
 Client (kernel) performance
13 ©AdFin. All Rights Reserved

14. High Level System Diagram, part 2
14 ©AdFin. All Rights Reserved

15. CephFS is not production ready?
 Again, we’re a bit crazy?
 Started in early 2013.
 When we started client and MDS were not ready.
 We found and reported a lot of bugs.
 Yan Zhen fixed a lot of bugs. Thanks Yan.
 Today we’re happy and in production.
 Processed multiple PB of data since then.
15 ©AdFin. All Rights Reserved

16. FSCache for kclient
 We decided to add local persistent caching support to the kclient.
 Our access pattern:
 Working set larger then node memory (page cache)
 Append-only data (time series)
 Most recent month, quarter of data access 100x more often
 Benefits:
 Reducing latency / speed lost by moving to non-local filesystem
 Reduce Ceph network traffic and OSD utilization
 Cheap local SSD drives get 500MB/s read performance
 Not re-inventing the wheel
16 ©AdFin. All Rights Reserved

17. Kernel programming is hard
 Have to understand Ceph, kernel, concurrency.
 An error in the kernel hangs or Oops your machine.
 Bugs in other parts of the kernel? (CacheFS).
 Prototype working in two weeks
 First submission 2 months later.
 In kernel 5 months later.
 Number one problem concurrency.
17 ©AdFin. All Rights Reserved

18. Ceph with FSCache Status
 In since: 3.13
 … Works well since: 3.15
 … All bugs fixed: 3.17
 Speed… as fast as your caching disk
 Tested single client performance 1200MB/s
18 ©AdFin. All Rights Reserved

19. Next steps…
 Contributing to Ceph & kernel is addicting:
 Ceph performance work. Improving latency / ioops.
 Kernel work: readv2() syscall. File serving applications
 http://lwn.net/Articles/612483/
19 ©AdFin. All Rights Reserved

20. Thank You!

21. Let’s Get in Touch
21 ©AdFin. All Rights Reserved
Milosz Tanski
CTO
milosz@adfin.com
16 E. 34th Street, 15th Floor
New York, New York 10016
linkedin.com/company/AdFin
twitter.com/AdFin