Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Jim Hatcher
Using Spark to Load Oracle Data into Cassandra
1 Introduction
2 Problem Description
3 Methods of loading external data into Cassandra
4 What is Spark?
5 Lessons Learned
...
Introduction
© DataStax, All Rights Reserved. 4
At IHS Markit, we take raw data and turn it into
information and insights for our custo...
Problem Description
Cluster
Factory
Oracle
Back-end Applications Customer-facing Systems
Load
Files
Customer-
facing
Applications
Oracle
Cassa...
Methods of loading external data into Cassandra
Methods of Loading External Data into C*
1. CQL Copy command
2. Sqoop
3. Write a custom program that uses the CQL driver
4...
What is Spark?
© DataStax, All Rights Reserved. 10
What is Spark?
Spark is a processing framework designed
to work with distributed data....
© DataStax, All Rights Reserved. 11
Spark Client
Driver
Spark
Context
Spark Master
Spark Worker
Spark Worker
Spark Worker
...
© DataStax, All Rights Reserved. 12
Spark with Cassandra
Credit:
https://academy.datastax.com/courses/ds320-
analytics-apa...
© DataStax, All Rights Reserved. 13
ETL (Extract, Transform, Load)
Text File
JDBC Data
Source
Cassandra
Hadoop
Extract Dat...
© DataStax, All Rights Reserved.
Typical Code - Example
// Extract
val extracted = sqlContext
.read
.format("jdbc")
.optio...
Lessons Learned
Lesson #1 - Spark SQL handles Oracle
NUMBER fields with no precision incorrectly
https://issues.apache.org/jira/browse/SPA...
© DataStax, All Rights Reserved. 17
import java.sql.Types
import org.apache.spark.sql.jdbc.{JdbcDialect, JdbcType}
import ...
Lesson #2 - Spark SQL doesn’t handle timeuuid
fields correctly
https://issues.apache.org/jira/browse/SPARK-10501
Spark SQL...
Lesson #3 – Careful when generating ID fields
We created an RDD:
val baseRdd = rddInsertsAndUpdates.map { dbRow =>
val key...
Lesson #3 – Careful when generating ID fields
We kept finding that the ID values in the inverted index table had slightly ...
Lesson #4 – You can only return an RDD of a
tuple if you have 22 items or less.
© DataStax, All Rights Reserved. 21
It’s p...
Lesson #4 – You can only return an RDD of a
tuple if you have 22 items or less.
© DataStax, All Rights Reserved. 22
You ca...
Lesson #5 – Getting a JDBC dataframe based on a
SQL statement is not very intuitive.
To get a dataframe from a JDBC source...
Lesson #5 – Getting a JDBC dataframe based on a
SQL statement is not very intuitive.
So, the workaround is to do this:
val...
Lesson #6 – Creating a partitioned JDBC
dataframe is not very intuitive.
The code to get a JDBC dataframe looks like this:...
Lesson #6 – Creating a partitioned JDBC
dataframe is not very intuitive.
When you pass these additional arguments in, here...
Lesson #7 – JDBC *really* wants you to get your
partitioned dataframe using a sequential ID column.
In our Oracle database...
Lesson #7 – JDBC *really* wants you to get your
partitioned dataframe using a sequential ID column.
We insert into this ta...
Resources
Resources
© DataStax, All Rights Reserved. 30
Spark
• Books
• Learning Spark
http://shop.oreilly.com/product/0636920028512...
Upcoming SlideShare
Loading in …5
×

Using Spark to Load Oracle Data into Cassandra

1,492 views

Published on

This presentation describes how you can use Spark as an ETL tool to get data from a relational database into Cassandra. I go through the concept in general and then talk about some specific issues you might run into and how to fix them.

Published in: Technology
  • Be the first to comment

Using Spark to Load Oracle Data into Cassandra

  1. 1. Jim Hatcher Using Spark to Load Oracle Data into Cassandra
  2. 2. 1 Introduction 2 Problem Description 3 Methods of loading external data into Cassandra 4 What is Spark? 5 Lessons Learned 6 Resources 2© DataStax, All Rights Reserved.
  3. 3. Introduction
  4. 4. © DataStax, All Rights Reserved. 4 At IHS Markit, we take raw data and turn it into information and insights for our customers. Automotive Systems (CarFax) Defense Systems (Jane’s) Oil & Gas Systems (Petra, Kingdom) Maritime Systems Technology Systems (Electronic Parts Database, Root Metrics) Chemicals Financial Systems (Wall Street on Demand) Lots of others
  5. 5. Problem Description
  6. 6. Cluster Factory Oracle Back-end Applications Customer-facing Systems Load Files Customer- facing Applications Oracle Cassandra + Solr Factory Applications Data Updates Cassandra + Spark
  7. 7. Methods of loading external data into Cassandra
  8. 8. Methods of Loading External Data into C* 1. CQL Copy command 2. Sqoop 3. Write a custom program that uses the CQL driver 4. Write a Spark program © DataStax, All Rights Reserved. 8
  9. 9. What is Spark?
  10. 10. © DataStax, All Rights Reserved. 10 What is Spark? Spark is a processing framework designed to work with distributed data. “up to 100X faster than MapReduce” according to spark.apache.org Used in any ecosystem where you want to work with distributed data (Hadoop, Cassandra, etc.) Includes other specialized libraries: • SparkSQL • Spark Streaming • MLLib • GraphX Spark Facts Conceptually Similar To MapReduce Written In Scala Supported By DataBricks Supported Languages Scala, Java, Python, R
  11. 11. © DataStax, All Rights Reserved. 11 Spark Client Driver Spark Context Spark Master Spark Worker Spark Worker Spark Worker Executor Executor Executor 1. Request Resources 2. Allocate Resources 3.StartExecutors 4.Perform Computation Credit: https://academy.datastax.com/courses/ds320-analytics-apache-spark/introduction-spark-architecture Spark Architecture
  12. 12. © DataStax, All Rights Reserved. 12 Spark with Cassandra Credit: https://academy.datastax.com/courses/ds320- analytics-apache-spark/introduction-spark- architecture Cassandra Cluster A CB Spark Worker Spark WorkerSpark Worker Spark Master Spark Client Spark Cassandra Connector – open source, supported by DataStax https://github.com/datastax/spark-cassandra-connector
  13. 13. © DataStax, All Rights Reserved. 13 ETL (Extract, Transform, Load) Text File JDBC Data Source Cassandra Hadoop Extract Data Spark: Create RDD or Data Frame Data Source(s) Spark Code Transform Data Spark: Map function Spark Code Cassandra Data Source(s) Load Data Spark: Save Spark Code
  14. 14. © DataStax, All Rights Reserved. Typical Code - Example // Extract val extracted = sqlContext .read .format("jdbc") .options( Map[String, String]( "url" -> "jdbc:oracle:thin:username/password@//hostname:port/oracle_svc", "dbtable" -> "table_name" ) ) .load() // Transform val transformed = extracted.map { dbRow => (dbRow.getAs[String](“field_one"), dbRow.getAs[Integer](“field_two")) } // Load transformed.saveToCassandra(“keyspace_name", “table_name", SomeColumns(“field_one“, “field_two"))
  15. 15. Lessons Learned
  16. 16. Lesson #1 - Spark SQL handles Oracle NUMBER fields with no precision incorrectly https://issues.apache.org/jira/browse/SPARK-10909 All of our Oracle tables have ID fields defined as NUMBER(15,0). When you use Spark SQL to access an Oracle table, there is a piece of code in the JDBC driver that reads the metadata and creates a dataframe with the proper schema. If your schema has a NUMBER(*, 0) field defined in it, you get a “Overflowed precision” error. This is fixed in Spark 1.5, but we don’t have the option of adopting a new version of Spark since we’re using Spark bundled with DSE 4.8.6 (which uses spark 1.4.2). We were able to fix this by stealing the fix from the Spark 1.5 code and applying it to our code (yay, open source!). At some point, we’ll update to DSE 5.* which uses Spark 1.6, and we can remove this code. © DataStax, All Rights Reserved. 16
  17. 17. © DataStax, All Rights Reserved. 17 import java.sql.Types import org.apache.spark.sql.jdbc.{JdbcDialect, JdbcType} import org.apache.spark.sql.types._ private case object OracleDialect extends JdbcDialect { override def canHandle(url: String): Boolean = url.startsWith("jdbc:oracle") override def getCatalystType(sqlType: Int, typeName: String, size: Int, md: MetadataBuilder): Option[DataType] = { // Handle NUMBER fields that have no precision/scale in special way // because JDBC ResultSetMetaData converts this to 0 precision and -127 scale // For more details, please see // https://github.com/apache/spark/pull/8780#issuecomment-145598968 // and // https://github.com/apache/spark/pull/8780#issuecomment-144541760 if (sqlType == Types.NUMERIC && size == 0) { // This is sub-optimal as we have to pick a precision/scale in advance whereas the data // in Oracle is allowed to have different precision/scale for each value. Option(DecimalType(38, 10)) } else { None } } override def getJDBCType(dt: DataType): Option[JdbcType] = dt match { case StringType => Some(JdbcType("VARCHAR2(255)", java.sql.Types.VARCHAR)) case _ => None } } org.apache.spark.sql.jdbc.JdbcDialects.registerDialect(OracleDialect) Lesson #1 - Spark SQL handles Oracle NUMBER fields with no precision incorrectly
  18. 18. Lesson #2 - Spark SQL doesn’t handle timeuuid fields correctly https://issues.apache.org/jira/browse/SPARK-10501 Spark SQL doesn’t know what to do with a timeuuid field when reading a table from Cassandra. This is an issue since we commonly use timeuuid columns in our Cassandra key structures. We got this error: scala.MatchError: UUIDType (of class org.apache.spark.sql.cassandra.types.UUIDType$) We are able to work around this issue by casting the timeuuid values to strings, like this: © DataStax, All Rights Reserved. 18 val dataFrameRaw = sqlContext .read .format("org.apache.spark.sql.cassandra") .options(Map("table" -> "table_name", "keyspace" -> "keyspace_name")) .load() val dataFrameFixed = dataFrameRaw .withColumn(“timeuuid_column", dataFrameRaw("timeuuid_column").cast(StringType))
  19. 19. Lesson #3 – Careful when generating ID fields We created an RDD: val baseRdd = rddInsertsAndUpdates.map { dbRow => val keyColumn = { if (!dbRow.isNullAt(dbRow.fieldIndex(“timeuuid_key_column"))) { dbRow.getAs[String]("timeuuid_key_column") } else { UUIDs.timeBased().toString } } //do some further processing (keyColumn, …other values) } Then, we took that RDD and transformed it into another RDD: val invertedIndexTable = baseRdd.map { entry => (entry.getString(“timeuuid_key_column"), entry.getString(“fld_1")) } Then we wrote them both to C*, like this: baseRdd.saveToCassandra(“keyspace_name", “table_name", SomeColumns(“key_column“, “fld_1“, “fld_2")) invertedIndexTable.saveToCassandra(“keyspace_name", “inverted_index_table_name" SomeColumns(“key_column“, “fld_1“) © DataStax, All Rights Reserved. 19
  20. 20. Lesson #3 – Careful when generating ID fields We kept finding that the ID values in the inverted index table had slightly different ID values than the values in the base table. We fixed this by adding a cache() to our first RDD. © DataStax, All Rights Reserved. 20 val baseRdd = rddInsertsAndUpdates.map { dbRow => val keyColumn = { if (!dbRow.isNullAt(dbRow.fieldIndex(“timeuuid_key_column"))) { dbRow.getAs[String]("timeuuid_key_column") } else { UUIDs.timeBased().toString } } //do some further processing (keyColumn, …other values) }.cache()
  21. 21. Lesson #4 – You can only return an RDD of a tuple if you have 22 items or less. © DataStax, All Rights Reserved. 21 It’s pretty common in Spark to return an RDD of tuples val myNewRdd = myOldRdd.map { dbRow => val firstName = dbRow.getAs[String](“FirstName") val lastName = dbRow.getAs[String](“LastName") val calcField1 = dbRow.getAs[Intger](“SomeColumn") * 3.14 (firstName, lastName, calcField1) } This works great until you get to 22 fields in your tuple, and then Scala throws an error. (Later versions of Scala lift this restriction, but it’s a problem for our version of Scala.)
  22. 22. Lesson #4 – You can only return an RDD of a tuple if you have 22 items or less. © DataStax, All Rights Reserved. 22 You can fix this by returning an RDD of CassandraRows instead. (especially if your goal is to save them to C*) val myNewRdd = myOldRdd.map { dbRow => val firstName = dbRow.getAs[String](“FirstName") val lastName = dbRow.getAs[String](“LastName") val calcField1 = dbRow.getAs[Integer](“SomeColumn") * 3.14 val allValues = IndexedSeq[AnyRef](firstName, lastName, calcField1) val allColumnNames = Array[String]( “first_name", “last_name", “calc_field_1“) new CassandraRow(allColumnNames, allValues) }
  23. 23. Lesson #5 – Getting a JDBC dataframe based on a SQL statement is not very intuitive. To get a dataframe from a JDBC source, you do this: val exampleDataFrame = sqlContext .read .format("jdbc") .options( Map[String, String]( "url" -> "jdbc:oracle:thin:username/password@//hostname:port/oracle_svc", "dbtable" -> "table_name" ) ) .load() You would think there would be a version of this call that lets you pass in a SQL statement but there is not. However, when JDBC creates your query from the above syntax, all it does is prepend your dbtable value with “SELECT * FROM”. © DataStax, All Rights Reserved. 23
  24. 24. Lesson #5 – Getting a JDBC dataframe based on a SQL statement is not very intuitive. So, the workaround is to do this: val sql = "( " + " SELECT S.* " + " FROM Sample S " + " WHERE ID = 11111 " + " ORDER BY S.SomeField " + ")" val exampleDataFrame = sqlContext .read .format("jdbc") .options( Map[String, String]( "url" -> "jdbc:oracle:thin:username/password@//hostname:port/oracle_svc", "dbtable" -> sql ) ) .load() You’re effectively doing this in Oracle: SELECT * FROM ( SELECT S.* FROM Sample S WHERE ID = 11111 ORDER BY S.SomeField ) © DataStax, All Rights Reserved. 24
  25. 25. Lesson #6 – Creating a partitioned JDBC dataframe is not very intuitive. The code to get a JDBC dataframe looks like this: val basePartitionedOracleData = sqlContext .read .format("jdbc") .options( Map[String, String]( "url" -> "jdbc:oracle:thin:username/password@//hostname:port/oracle_svc", "dbtable" -> "ExampleTable", "lowerBound" -> "1", "upperBound" -> "10000", "numPartitions" -> "10", "partitionColumn" -> “KeyColumn" ) ) .load() The last four arguments in that map are there for the purpose of getting a partitioned dataset. If you pass any of them, you have to pass all of them. © DataStax, All Rights Reserved. 25
  26. 26. Lesson #6 – Creating a partitioned JDBC dataframe is not very intuitive. When you pass these additional arguments in, here’s what it does: It builds a SQL statement template in the format “SELECT * FROM {tableName} WHERE {partitionColumn} >= ? AND {partitionColumn} < ?” It sends {numPartitions} statements to the DB engine. If you suppled these values: {dbTable=ExampleTable, lowerBound=1, upperBound=10,000, numPartitions=10, partitionColumn=KeyColumn}, it would create these ten statements: SELECT * FROM ExampleTable WHERE KeyColumn >= 1 AND KeyColumn < 1001 SELECT * FROM ExampleTable WHERE KeyColumn >= 1001 AND KeyColumn < 2000 SELECT * FROM ExampleTable WHERE KeyColumn >= 2001 AND KeyColumn < 3000 SELECT * FROM ExampleTable WHERE KeyColumn >= 3001 AND KeyColumn < 4000 SELECT * FROM ExampleTable WHERE KeyColumn >= 4001 AND KeyColumn < 5000 SELECT * FROM ExampleTable WHERE KeyColumn >= 5001 AND KeyColumn < 6000 SELECT * FROM ExampleTable WHERE KeyColumn >= 6001 AND KeyColumn < 7000 SELECT * FROM ExampleTable WHERE KeyColumn >= 7001 AND KeyColumn < 8000 SELECT * FROM ExampleTable WHERE KeyColumn >= 8001 AND KeyColumn < 9000 SELECT * FROM ExampleTable WHERE KeyColumn >= 9001 AND KeyColumn < 10000 And then it would put the results of each of those queries in its own partition in Spark. © DataStax, All Rights Reserved. 26
  27. 27. Lesson #7 – JDBC *really* wants you to get your partitioned dataframe using a sequential ID column. In our Oracle database, we don’t have sequential integer ID columns. We tried to get around that by doing a query like this and passing “ROW_NUMBER” as the partitioning column: SELECT ST.*, ROW_NUMBER() OVER (ORDER BY ID_FIELD ASC) AS ROW_NUMBER FROM SourceTable ST WHERE …my criteria ORDER BY ID_FIELD But, this didn’t perform well. We ended up creating a processing table: CREATE TABLE SPARK_ETL_BATCH_SEQUENCE ( SEQ_ID NUMBER(15,0) NOT NULL, //this has a sequence that gets auto-incremented BATCH_ID NUMBER(15,0) NOT NULL, ID_FIELD NUMBER(15,0) NOT NULL ) © DataStax, All Rights Reserved. 27
  28. 28. Lesson #7 – JDBC *really* wants you to get your partitioned dataframe using a sequential ID column. We insert into this table first: INSERT INTO SPARK_ETL_BATCH_SEQUENCE ( BATCH_ID, ID_FIELD ) //SEQ_ID gets auto-populated SELECT {NextBatchID}, ID_FIELD FROM SourceTable ST WHERE …my criteria ORDER BY ID_FIELD Then, we join to it in the query where we get our data which provides us with a sequential ID: SELECT ST.*, SEQ.SEQ_ID FROM SourceTable ST INNER JOIN SPARK_ETL_BATCH_SEQUENCE SEQ ON ST.ID_FIELD = SEQ.ID_FIELD WHERE …my criteria ORDER BY ID_FIELD And, we use SEQ_ID as our Partitioning Column. Despite its need to talk to Oracle twice, this approach has proven to perform much faster than having uneven partitions. © DataStax, All Rights Reserved. 28
  29. 29. Resources
  30. 30. Resources © DataStax, All Rights Reserved. 30 Spark • Books • Learning Spark http://shop.oreilly.com/product/0636920028512.do Scala (Knowing Scala with really help you progress in Spark) • Functional Programming Principles in Scala (videos) https://www.youtube.com/user/afigfigueira/playlists?shelf_id=9&view=50&sort=dd • Books http://www.scala-lang.org/documentation/books.html Spark and Cassandra • DataStax Academy http://academy.datastax.com/ • Self-paced course: DS320: DataStax Enterprise Analytics with Apache Spark – Really Good! • Tutorials • Spark Cassandra Connector website – lots of good examples https://github.com/datastax/spark-cassandra-connector

×