- The document discusses migrating structured data between Hadoop and relational databases using a tool called Bouquet. - Bouquet allows users to select data from a relational database, which is then sent to Spark via Kafka and stored in HDFS/Tachyon for processing. - The enriched data in Spark can then be re-injected back into the original database.

1. February 16th
2016
louis.rabiet@squidsolutions.com
Migrating structured data between Hadoop
and RDBMS

2. Who am I?
• Full Stack engineer at Squid Solutions.
• Specialised in Big data.
• Fun fact: sleeping by myself in my tent
on the top of the highest mountains of
the world

3. What I do ?
• Develop of an analytics toolbox.
• No setup. No SQL. No compromise.
• Generate SQL with a REST API.
It is open source!
https://github.com/openbouquet

4. Topic of today
• You need Scalability?
• You need a machine learning
toolbox?
Hadoop is the solution.
•But you still need structured data?
Our tool provide a solution.
=> We need both!

5. What does that mean?
• Creation of dataset in Bouquet
• Send the dataset to Spark
• Enrich inside Spark
• Re-injection in original
database

6. How we do it?
User input
Relational
DB
SparkBouquet

7. Create and Send

8. How does it work?
BouquetRelational
DB
Spark
HDFS/
Tachyon
Hive
Metastore
User select the data. Bouquet generate the corresponding SQL Code
Kafka

9. How does it work?
BouquetRelational
DB
Spark
HDFS/
Tachyon
Hive
Metastore
Data is read from the SQL database
Kafka

10. How does it work?
BouquetRelational
DB
Spark
HDFS/
Tachyon
Hive
Metastore
Bouquet creates an avro schema and send the data to Kafka
Kafka

11. How does it work?
BouquetRelational
DB
Spark
Kafka
HDFS/
Tachyon
Hive
Metastore
Kafka Broker(s) receive the data

12. How does it work?
BouquetRelational
DB
Spark
HDFS/
Tachyon
Hive
Metastore
Kafka
The hive metastore is updated and the hdfs connectors writes into hdfs

13. Tachyon?
• Use it as in memory filesystem
to replace HDFS.
• Interact with Spark using the
hdfs plugin.
• Transparent from user point of
view

14. How to keep the data structured?
Use a schema registry (Avro in Kafka).
each schema has a corresponding kafka topic and a distinct hive table.
{
"type": "record",
"name": "ArtistGender",
"fields" : [
{"name": "count", "type": "long"},
{"name": "gender", "type": "String"]}
]
}

15. Challenges
- Auto creation of topics/table in Hive for each datasets from
Bouquet.
- JDBC reads are too slow for something like Kafka.
- Issue with types conversion: null is not supported for all
cases for example (issue 272 on schema-registry).
- Versions: Kafka 0.9.0, Tachyon 0.7.1, Spark 1.5.2 with
HortonWorks 2.3.4 (Dec 2015)
- Hive: Setting the warehouse directory.
- In tachyon: Setting up hostname.

16. Technology choice
• KISS: Kafka + Spark + Tachyon.
• Flexible (Hive, In-memory
storage)
• Easily scalable
• GemFire, SnappyData, Apache
Ignite for In-memory storage.
• Storm for streaming

17. Status
Injection DB -> Spark: OK
Spark usage: OK
Re-injection: In alpha stage.

18. Re-injection
Two solutions:
• Spark user notifies Bouquet
that data has changed (using a
custom function)
• Bouquet pulls the data from
spark

19. We use it for real!
Collaborating with La Poste to
be able to use Spark and the re-
injection mechanism to use
Bouquet and a geographical
visualisation.

20. In the future
• Notebook integration
• We got a DSL for bouquet API,
we may want to have built-in
support spark.
• Improve scalability (Bulk
Unload and Kafka fine tuning)

21. QUESTIONS
OPENBOUQUET.IO