Presentation faite lors du Hadoop User Group France du 14 janvier 2016. L’analytique temps réel avec Riak et Spark par Michael Carney (Basho) et Olivier Girardot de Lateral Thoughts Selon un rapport de Salesforce, le nombre de sources de données analysées par les entreprises progressera de 83% au cours des cinq prochaines années, ainsi les organisations veulent désormais fournir des connaissances en temps réel même sur les appareils mobiles. Le traitement temps réel est donc, le futur de l’analyse big data. Ce talk présentera des nouveautés en matière de l’analyse temps réel autour de la famille SGBD Riak et Spark. Michael Carney est le Directeur Commercial de Basho pour le Sud d’Europe. Fondateur de MySQL France et de MariaDB, Michael a rejoint Basho en janvier 2015 pour explorer le monde de données sans tables ! Olivier Girardot est le CTO de Lateral Thoughts, il est développeur et formateur au sujet de Spark et également spécialiste de Java/Python dans le domaine de la finance de marché.

1. SPARK & RIAK
INTRODUCTION TO THE SPARK-RIAK-CONNECTOR
LATERAL
THOUGHTS

2. Me, Myself & I
Associate at LateralThoughts.com
Scala, Java, Python Developer
Data Engineer @ Axa & Carrefour
Apache Spark Trainer with Databricks
LATERAL
THOUGHTS

3. And the Other One …
Director Sales @ Basho Technologies
(Basho make Riak)
Ex of MySQL France
Co-Founder MariaDB
Funny Accent

4. Quick Introduction …
2011 Creators of Riak
Riak KV: NoSQL key value database
Riak S2: Large Object Storage
2015 New Products
Basho Data Platform: Integrated NoSQL databases, caching,
in-memory analytics, and search
Riak TS: NoSQL Time Series database
120+ employees
Global Offices
Seattle (HQ), Washington DC, London, Paris, Tokyo
300+ Enterprise customers, 1/3 of the Fortune 50

6. PRIORITIZED NEEDS
High Availability - Critical Data
High Scale –
Heavy Reads & Writes
Geo Locality –
Multiple Data Centers
Operational Simplicity –
Resources
Don’t Scale as Clusters
Data Accuracy –
Write Conflict Options
∂
RIAK S2 USE CASES
Large Object Store
Content Distribution
Web & Cloud Services
Active Archives
∂
RIAK KV USE CASES
User Data
Session Data
Profile Data
Real-time Data
Log Data
∂
RIAK TS USE CASES
IoT/Devices
Financial/Economic
Scientific Observations
Log Data

7. The Evolution of NoSQL
Unstructured
Data Platforms
Multi-Model
Solutions
Point
Solutions

8. Basho Data Platform …

9. ABOUT SPARK & RIAK

10. Spark & Riak
Disclaimer, the following presentation uses :
Spark v1.5.2
Spark-Riak-Connector v1.1.0

11. Pre-Requisites
To use the Spark Riak Connector, as of now, you need to build it
yourself :
Clone https://github.com/basho/spark-riak-connector
`git checkout v1.1.0`
`mvn clean install`

12. Bootstrapped project

13. Reading from
Connect to a Riak KV Cluster from Spark
Query it :
Full Scan
Using Keys
Using secondary indexes (2i)

14. Connecting to

15. Loading data from
riakBucket[V](bucketName: String): RiakRDD[V]
riakBucket[V](bucketName: String, bucketType: String): RiakRDD[V]
riakBucket[K, V](bucketName: String, convert: (Location,
RiakObject) => (K, V)): RiakRDD[(K, V)]
…
On your Spark Context, you can use :

16. add a query, otherwise…

17. Find all :
Find by key(s) :

18. Implicits
that will give you
the riak* methods

19. Reading from
Using case classes
Using Secondary Indexes

20. Basic I/O

21. Mapping Objects - Buckets

23. Adding fields during save

24. Spark Riak Connector - Roadmap
Better Integration with Riak TS
Enhanced DataFrames - based on Riak TS Schema APIs
Server-side aggregations and grouping - using TS SQL commands
Speed
Data Locality (partition RDDs according to replication in the cluster) - launch Spark executors on the same nodes where the data resides.
Better mapping from vnodes to Spark workers using coverage plan
Better support for Riak data types (CRDT) and Search queries
Today requires using Java Riak client APIs
Spark Streaming
Provide example and sample integration with Apache Kafka
Improve reliability using Riak for checkpoints and WAL
Add examples and documentation for Python support
DRAFT

25. Thank you
@ogirardot
o.girardot@lateral-thoughts.com
https://github.com/ogirardot/spark-riak-example
https://speakerdeck.com/ogirardot/spark-and-riak-introduction-to-
the-spark-riak-connector
@mcarney23
michael.carney@basho.com
fr.basho.com