The document discusses connecting computation between HPE Vertica, an advanced SQL analytics platform, and Apache Spark. It describes the Vertica-Spark connector which allows data and computation to flow between the two systems using VerticaRDD and DataSource APIs. It also covers optimizations like data locality, computation pushdown, and saving data from Spark to Vertica. Two approaches for loading data from Spark to Vertica are discussed: a direct single-stage approach and an indirect two-stage approach using intermediate storage.

1. Vertica and Spark:
Connecting Computation and Data
Rui Liu and Edward Ma
Hewlett Packard Enterprise Vertica Advanced R&D Labs

2. Overview
Vertica
Analytics Platform
Computation and data
• The Vertica-Spark connector connects both data and
computation between Vertica and Spark
• VerticaRDD and Vertica Data Source APIs
• Data-locality optimization
• Computation pushdown
• Save data from Spark to Vertica

3. HPE Vertica
• An advanced SQL analytics platform
• Shared-nothing MPP architecture
• Column-oriented storage organization
• Standard SQL interface with many analytics
capabilities built-in
• Extensible via User-Defined Functions
Vertica
Analytics Platform

4. Connected pipeline
Spark
Vertica
Vertica execution plan node
Spark execution plan node
Data flow

5. Connected pipeline
• Connects the computation pipelines of Spark
and Vertica in an optimized way
• Utilizes parallel channels (parallel queries) for
data movement
• Leverages data-locality optimization on Vertica
• Ensures computation push-down into Vertica as
appropriate (filters and projections)

6. Vertica data segmentation on hash ring
node0
node1
node2
node3
Id Name Age email …
CREATE TABLE t …
SEGMENTED BY HASH(id) …

7. Spark locality-aware partitions
seg 0
seg 1
seg 2
seg 3
Spark partition
Locality-aware partition:
• Uses Vertica data-distribution
information
• Generates locality-aware
queries
SELECT id, name FROM T
where
0 < 0xffffffff & hash(id)
AND
0xffffffff & hash(id) < 278985;

8. Locality-aware query
Non locality-aware query
Initiator
Locality-aware query
Initiator
Data shuffling
DataData Data Data Data

9. Vertica query pruning
• For a locality-aware
query, the execution
can be pruned
• Query only executes
on nodes that contain
the data
Query
Without
Pruning
Initiator
Query
With
Pruning
Initiator
Data
Data

10. Arbitrary number of partitions
seg 0
seg 1
seg 2
seg 3
Spark partition

11. Spark partitions over unsegmented
table replicas
replica 0 replica 1 replica 2 replica 3
Spark partition

12. Performance characteristics

13. Scalability of throughput

14. Computation push-down
• Implemented:
– Filters
– Projections
– Count(*)
• Works in progress:
– Joins
– Aggregations
• Future work:
– User-Defined Functions

15. Computation push-down schemes
• Per-tuple computation (filtering,
projection)
– Always pushed down
• Joins & aggregations
– Locality-aware: when join and
aggregation keys are co-
segmented in Vertica
– Single query: run single query with
join/aggregation inside Vertica

16. select c,d,f from A JOIN B on A.foo = B.foo
InitiatorInitiator Initiator
Original Query:
Locality-aware
push-down
query:
select c,d,f from
A JOIN B on A.foo = B.foo
where
0 < 0xffffffff & hash(foo)AND
0xffffffff & hash(foo) < 278985 …
Initiator

17. Performance Impact of Pushdowns

18. Vertica Analytics
• Geospatial analytics
• Sentiment analysis
• Sessionization of event streams
• Time series pattern matching
• ... (Many more)

19. The Other Direction
Data Sources
SaveLoad
Spark as an ETL Engine for Vertica
Logs
Streams
BI

20. The Other Direction
Data Sources
SaveLoad
Spark as an ETL Engine for Vertica
Logs
Streams
BI
Objective: Want reliable and fast bulk loading

21. Two Possible Approaches
Write
Direct
Write Read
Indirect (Two-Stage)

22. Comparison to JDBC Source

23. What could happen?

24. What could happen?

25. What could happen?

26. What could happen?

27. What could happen?
NEED TO AVOID
DUPLICATION!

28. Spark Task/Partition
Big-Picture (Direct)
Spark Task/Partition
Spark Task/Partition
VERTICA
STAGING TABLE
VERTICA
TARGET TABLE
Serialization & Parsing Leader Election & Commit

29. Exactly-once Semantics
Never results in partial or duplicate table loads
29
Executor
W
O
R
K
E
R
N
O
D
E
Task Partition
Task Partition
Task Partition
RDD
Dataframe
STAGING_TABLE
Executor
W
O
R
K
E
R
N
O
D
E
Task Partition
Task Partition
Task Partition
TASK_STATUS

30. Exactly-once Semantics
Never results in partial or duplicate table loads
30
Executor
W
O
R
K
E
R
N
O
D
E
Task Partition
Task Partition
Task Partition
RDD
Dataframe
STAGING_TABLE
Executor
W
O
R
K
E
R
N
O
D
E
Task Partition
Task Partition
Task Partition
TASK_STATUS
DUPLICATED TASK

31. Exactly-once Semantics
Never results in partial or duplicate table loads
31
Executor
W
O
R
K
E
R
N
O
D
E
Task Partition
Task Partition
Task Partition
RDD
Dataframe
STAGING_TABLE
Executor
W
O
R
K
E
R
N
O
D
E
Task Partition
Task Partition
Task Partition
TASK_STATUS
UPDATE TASK_STATUS SET DONE=TRUE
WHERE TASK=2 AND DONE=FALSE

32. Exactly-once Semantics
Never results in partial or duplicate table loads
32
Executor
W
O
R
K
E
R
N
O
D
E
Task Partition
Task Partition
Task Partition
RDD
Dataframe
STAGING_TABLE
Executor
W
O
R
K
E
R
N
O
D
E
Task Partition
Task Partition
Task Partition
TASK_STATUS
RESULT =1;
COMMIT

33. Exactly-once Semantics
Never results in partial or duplicate table loads
33
Executor
W
O
R
K
E
R
N
O
D
E
Task Partition
Task Partition
Task Partition
RDD
Dataframe
STAGING_TABLE
Executor
W
O
R
K
E
R
N
O
D
E
Task Partition
Task Partition
Task Partition
TASK_STATUS
RESULT =0;
ROLLBACK

34. Leader Election
Find Out Which Task Won The Race
34
Executor
W
O
R
K
E
R
N
O
D
E
Task Partition
Task Partition
Task Partition
RDD
Dataframe
STATUS_TABLE
PartID, done?
PartID, done?
PartID, done?
STAGING_TABLE
LAST_COMMITTER Executor
W
O
R
K
E
R
N
O
D
E
Task Partition
Task Partition
Task Partition
UPDATE LAST_COMMITTER SET
ID={MY_ID} WHERE ID = -1

35. Leader Election
Find Out Which Task Won The Race
35
Executor
W
O
R
K
E
R
N
O
D
E
Task Partition
Task Partition
Task Partition
RDD
Dataframe
STATUS_TABLE
PartID, done?
PartID, done?
PartID, done?
STAGING_TABLE
LAST_COMMITTER Executor
W
O
R
K
E
R
N
O
D
E
Task Partition
Task Partition
Task Partition
UPDATE LAST_COMMITTER SET
ID={MY_ID} WHERE ID = -1
SUCCESS ->
COMMITTER

36. Leader Election
Find Out Which Task Won The Race
36
Executor
W
O
R
K
E
R
N
O
D
E
Task Partition
Task Partition
Task Partition
RDD
Dataframe
STATUS_TABLE
PartID, done?
PartID, done?
PartID, done?
STAGING_TABLE
LAST_COMMITTER Executor
W
O
R
K
E
R
N
O
D
E
Task Partition
Task Partition
Task Partition
UPDATE LAST_COMMITTER SET
ID={MY_ID} WHERE ID = -1
FAIL -> DONE

37. Move to Target Table
STAGING_TABLE TARGET_TABLE
Committer Rename or Append

38. Direct: Pros & Cons
+ Requires no external system beyond Vertica and
Spark
+ Less I/O: Does not involve an additional write stage
- Exactly-once semantics are non-trivial to ensure

39. Big-Picture (Two-Stage)
HDFS VERTICA

40. Big-Picture (Two-Stage)
HDFS VERTICA

41. Big-Picture (Two-Stage)
WRITE to
ORC HDFS VERTICA

42. Big-Picture (Two-Stage)
HDFS VERTICA
COPY target_table FROM ‘hdfs://…’ ON ANY NODE ORC;

43. Big-Picture (Two-Stage)
PARSE &
STORE
HDFS VERTICA
COPY target_table FROM ‘hdfs://…’ ON ANY NODE ORC;

44. Two-Stage: Pros & Cons
+ Utilizes Vertica’s built-in parallel COPY command for bulk loads, and
may allow for better scalability
+ Uses the optimized ORC reader for parsing (developed in
collaboration with Hortonworks).
+ Simpler code logic, since only one query is involved
- More I/O: Requires an additional writing stage.
- Requires an additional dependency on intermediate storage system.

45. Conclusions
• Vertica-to-Spark
– Utilizes hash-range queries to minimize intra-Vertica data movement
– Node-pruning eliminates unneeded distributed query processing in Vertica
– Query pushdown to reduce data transfer and to take advantage of Vertica
native analytics.
• Spark-to-Vertica
– Direct connector approach efficiently and reliably coordinates transfer with
parallel Spark tasks
– A two-stage approach with an intermediary staging storage (HDFS) utilizes
a single Vertica query to load and may be more efficient on resource
consumption on Vertica side

46. • Community Edition
• Free Download 1TB, 3 nodes
• my.vertica.com/
• Spark Connector – Beta Program
• https://saas.hpe.com/marketplace/big-data/hpe-vertica-connector-apache-spark
Learn More About – and Try! - HPE
Vertica

47. THANK YOU.
Rui Liu (r.liu@hpe.com)
Edward Ma (ema@hpe.com)