Datorama, acquired by Salesforce in 2018, is a rapidly growing company specializing in integrating and unifying marketing data from various sources to provide actionable insights and optimize campaign performance. The document outlines Datorama's architecture, including features such as data lakes and ETL processes, designed to handle mutable data with high performance and reliability. It highlights key metrics showcasing Datorama's scale and operational capabilities, emphasizing their commitment to meeting evolving data management needs.

1. Mutable Data @ Scale
afinkelstein@salesforce.com
Alexey Finkelstein, Software Engineer

2. Private & Confidential
Datorama At-A-Glance
Founded in
Employees
& growing
quickly
Acquired in
October 2018
Brands
Agencies Publishers
Industry verticalsBy Ran Sarig, Efi Cohen
& Katrin Ribant
450+
2012
192018
Offices
worldwide
2000+
300+
23
50+
Private & Confidential

3. Private & Confidential
+20
Verticals
Broad blue-chip customer base
+23
Verticals
300
Agencies
+2000
Brands
Every agency holding group that has run an RFP for a global client reporting
solution in the last 3 years has selected Datorama as their platform of record.

4. Datorama
Connect & Unify Marketing Data Sources
Integrate, cleanse, and classify data into a unified
view using AI
Visualize AI-Powered Insights
Surface insights to optimize channel and campaign
performance in real-time
Report Across Channels and Campaigns
Powerful one-click dashboards, custom
visualizations, and shareable reports
Collaborate and Act to Drive ROI
Make every insight actionable with cross-platform
alerts and activations
Enable cross-platform marketing intelligence
+

5. Spend Your Time Wisely
+80% On Insights
+80% On Preparations
Time to Insight

6. Data To Insights In Minutes

7. Scale - in numbers
• 3.5M interactive
analytical queries served
per day
• 700,000 Data Stream
processed daily
• 100,000 Reports
generated daily
• 25,000 Workspaces
• 30,000 Users
• 1.5 PB of Data available
for interactive querying
• 99.9% Uptime
• 4 Different fully
redundant geographical
deployments
• ~600 Servers
• >50 microservices

8. Salesforce Acquisition
August 2018 - $850M

9. Data Lake

10. DatoLakes (Datorama Data Lakes)
Granular data support in reduced cost
● Your granular data together with your
aggregated data in one view
● Aimed for Raw data, including ETL, storage,
SQL access and reporting.
● Aimed to support data which is accessed less
frequently and in low concurrency, in lower
cost.
● Raw data can later on be aggregated and
joined with the rest of the data model.

11. DatoLakes (Datorama Data Lakes)
● Managing a data lake is a big hassle. (ETL, queries & other controls)
● Merging between granular and aggregate sources is a must
● Datorama to provide “lake as a service”
Challenges

12. Data is NOT immutable
● External vendors have windows of reconciliations (up to 6 months)
● Our users want to update/delete specific rows/set
● Our users love to backdate
● Most (if not all) big data solutions are append only and updating the data is considered a
heavy process
● Transactional updates required

13. The Solution

14. Requirements
● Separation of compute and storage - MUST
● MPP query engine - MUST
● ANSI SQL - MUST
● JDBC (for external clients) - MUST
● Transactional and not append only - MUST
● Cloud Vendor Agnostic - MUST
● Linear Scale - MUST
The solution we decided on was Presto and S3/Azure Storage

15. High Level Update Flow
1. Read the input file
2. Determine what data segments it operates on
3. Read the corresponding segments of the table from storage
4. Update the segments with input data
5. Store to a new location with the new version number
6. Add the updated partitions to Hive
7. Outdated partitions are cleaned in the background
A
B
C
A
B
A*
B*
C*

16. Mutable Data - Swap Partition Requirements
● The ETL process should trigger a swap partition(s) at the end of the process
● We need the swap to be transactional (to avoid dirty reads)
● It needs to support transactional change of multiple partitions in multiple tables at the
same time

17. Architecture
S3/AzureBlobStorage
Meta
Store
ETL Q
Queue
Resource
Manager
Query

18. Solution #1 - First Attempt (Past)
1. Partition the table by “key_version” field
a. key = actual column value
b. version = incremental number
c. e.g. 20190101_009
2. Create an external metastore that holds the
active versions of each partition (per table)
3. Commit the changes at the end of the ETL
(cross partition/ cross tables) to support a
transactional process
4. Connect the metastore table into hive and
include a subquery in every generated query.

19. Solution #2 - Present
Inline SQL didn’t initiate partition pruning by
Presto
1. Query the meta store while generating the
query to get the list of the relevant partitions for
the query
2. Inline the filter in the query

20. Solution #3 - Future
Process requires 2 steps (query meta + query
presto) and does not support direct SQL
access to clients
1. Update hive database (MySQL) directly in a
transactional manner just like we updated our
own metastore.
2. Refresh presto/hive caches to refresh the
metastore

21. Retrospective
● We’re able to “check” all the required items from our requirements
○ Separation of compute and storage, MPP query engine, ANSI SQL, JDBC, Transactional, Cloud
Vendor Agnostic & Linearly Scaled
● Data is stored in ORC files (due to the nature of our queries it was a big performance boost)
● Everybody is happy :)

22. We’re Hiring!
Contact us at
http://datorama.com/join-us
https://engineering.datorama.com/