Performant XML processing in a distributed environment is a major challenge. We will dive into a financially viable pipeline to extract, load and transform XML on a platform of performance and flexibility. Hadoop, Spark and Impala will feature in this session about bringing commodity applications to a market of proprietary solutions. What can we achieve by processing non-splittable data in a distributed fashion? I will talk about the motivation behind our research and show how we evolved a solution to cope with an ever changing environment. Stepping into the solution, I will show how you can strip away the restrictions of XML and load it onto Hadoop ready for analysis at scale in both an adhoc and modelling fashion. Evaluating and evolving this solution is paramount. Load and throughput testing methods are highlighted along with guidance on tuning both the pipeline and the Hadoop platform to ensure your solution is optimised for a dynamic environment.

1. Semi-structured Data and Hadoop
Shannon Holgate

2. Why am I here?

3. Semi-structured Data

4. Semi-structured Data

5. Semi-structured Data
... and Hadoop

9. Agenda
Motivation for Analysing XML
Solution design in Spark on Hadoop
Tuning the Solution for performance

10. What to take away
Thumbs up to XML processing in Hadoop
Spark can process XML quite easily
Tuning is absolutely essential

11. Motivation

12. We should care about XML

13. XML is Incorrectly used

14. Extensive Schemas
SOAP
Bloat on the wire
XML is Incorrectly used

15. XML is also
Human Readable
Extremely Portable
Storable in Databases

16. aking XML perfect for communicating Da

17. XML is consistently the preferred
data transfer protocol we see in
Financial Institutions

18. We should care about XML

19. Why should we care?
Financial Institutions have the data
and financial backing to use Big Data
technologies

20. Our first Big Data
engagement is with a Global
Insurance Provider

21. This customer would like to
process XML at Scale within
Budget

22. Along came an Elephant

23. We want Hadoop, let's prove it can keep
up with our current applications
Words from the customer:

24. Create a solution which ingests, extracts
and exports XML on Hadoop

25. And make sure this solution has the
performance to replace Teradata

26. Cost

27. Hadoop provides the opportunity to pay
only for what you need

28. This means no Oracle guy knocking on the
maintenance door

29. And no absurd Teradata licensing fees

30. XML is worth analysing
Financial Institutions are ready for Big Data
Hadoop is a cost effective Big Data solution

31. d a way to process XML in a performant fashio

32. Solution

33. Gathering Requirements

34. Specifications of the Hadoop Cluster
Expected Load
Data sources
Integration points
Transformation Logic

35. Cluster Capacity
Installed Services
YARN
Kerberos and Sentry
Specifications of the Hadoop Cluster

36. Development Test Production
Cluster Size
6 Nodes,128GB
12 cores
12 Nodes,128GB
12 cores
12 Nodes,128GB
12 cores
Installed
Services
Spark v1.3
Flume v1.5
Sqoop v1.4.5 ....
Spark v1.3
Flume v1.5
Sqoop v1.4.5 ....
Spark v1.3
Flume v1.5
Sqoop v1.4.5 ....
YARN Yes Yes Yes
Kerberos
Sentry
Both Both Both

37. Data Sources
Batch loads vs. Streaming
Apples vs. Oranges

38. Flume and Spark can be used for
Streaming XML messages

39. Batch Loads are better suited to a
scheduled Oozie job

40. Expected Load
Messages per Second
Message scheduling
Expected Message Size

41. Expected Load - Streaming Messages
Messages per Second 48MPS, 192MPS peak
Message Scheduling 24/7
Expected Message Size 15KB

42. Expected Load - Batch Messages
Messages per Second 15GB daily
Message Scheduling 22:00 daily
Expected Message Size 15KB

43. Transformation Logic
Capture User Stories for Extraction Criteria
Work with Product Owner to Create Data Map

44. Integration points
XML source location
Export destination
Audit endpoints

45. Integration points
XML source location JMS source
Export destination Exadata
Audit endpoints Exadata

46. Building the Pipeline

47. The Pipeline
Deep Dive
The Tech Stack
Worked Example - Streaming Messages

48. Data Flow
The Pipeline

49. The Tech Stack
Flume Ingestion
Spark Xml to Avro
Spark Avro to PSV
Sqoop Export of PSV

51. Deep Dive

52. JMS XML message source

53. Flume Agent with JMS
source and HDFS sink

54. Spark Job every 10 minutes
Reads 10 minutes of
streamed XML
Converts to Avro Datafiles

55. Spark job running prior to
export
Converts Avro to PSV for
Sqoop

56. Sqoop export for each table
in Exadata
Reads PSV versions of the
Avro data

57. Data warehouse holding the
completed pipeline data

59. XML Processing with Spark

60. Spark on Scala
Access to Java Libraries
Scala is Functional by design

61. Reading XML in Spark
Keep things simple
Use the Xml Input Format from
Hadoop Streaming
Inputs are split from opening to
closing tag

63. Avro Support in Spark
Use Kryo Serialisation for correct
Avro support
Avro Serialisation and data format
Avro Serialisation and Parquet Data
Format

64. Design Extractors
In this case we want to turn one XML
message into 5 different Avros
5 Extraction Classes should be
created

65. Design Extractors - cont
Use DOM/SAX if you have no definite
XSD for the XML
DOM is acceptable as the data is
already in memory

67. Spark Processing
All extractions should occur within a
single Map
Map only job
Try not to cause any shuffles

68. Writing the Avro output
Use the AvroJob MapReduce output
format
Create Avro Datafiles

70. Take time to understand incoming XML
Design solution to fit the Hadoop Cluster
XML in Spark should be processed carefully

71. Tuning

72. Remember those Cluster Specs?

73. Development Test Production
Cluster Size
6 Nodes,128GB
12 cores
12 Nodes,128GB
12 cores
12 Nodes,128GB
12 cores
Installed
Services
Spark v1.3
Flume v1.5
Sqoop v1.4.5 ....
Spark v1.3
Flume v1.5
Sqoop v1.4.5 ....
Spark v1.3
Flume v1.5
Sqoop v1.4.5 ....
YARN Yes Yes Yes
Kerberos
Sentry
Both Both Both

74. Time to use them

75. Tuning Flume
Build Flume Cluster and Load
balance
Source should read enough events
for 1 block
File channel vs. Memory channel

76. Tuning Spark - Executors
Vital Tuning point

77. Tuning Spark - Executors
Memory allocation
Number of cores
Number of Executors

78. Memory allocation
--executor-memory
Max out but leave some for the daemons

79. Number of cores
Spark can run 1 task per core
HDFS Client doesn't like concurrent threads
Limit to 5

80. Number of Executors
More Executors, less cores
Big nodes? 3-5 Executers per node
Adjust memory and cores to match

81. Don't forget YARN

82. If the Cluster is YARN enabled it will limit
memory and cores

83. Solution?
Number of Executers = Number of
Cores/Number of Container Cores * Nodes

84. Or,
Number of Executers = Memory per
Node/Memory per Container * Nodes

85. Use the minimum value

87. Tuning Spark - Monitoring
Spark Context Web UI
Coda Hale Metrics

89. Tuning Sqoop Exports
Use Direct Connectors
Tweak Number of Mappers

90. Cluster Flume where possible
More Spark Executers > More cores
Sqoop mappers

91. Summary

92. XML is absolutely a worthwhile data
source to analyse

93. Focus on using Spark to Extract XML data
and move into Avro and Parquet

94. Tuning should revolve around Spark
allocated resources

95. Thanks

96. Shannon Holgate
Senior Analytics Engineer @ Kainos
@sholgate13