Download as PDF, PPTX
![The bright blue butterfly hangs on the breeze.
[the] [bright] [blue] [butterfly] [hangs] [on] [the] [breeze]
Tokens](https://image.slidesharecdn.com/lovetriangle-150701134944-lva1-app6892/75/A-Cassandra-Solr-Spark-Love-Triangle-Using-DataStax-Enterprise-5-2048.jpg)
![Attaching to Spark and Cassandra
// Import Cassandra-specific functions on SparkContext and RDD objects
import org.apache.spark.{SparkContext, SparkConf}
import com.datastax.spark.connector._
/** The setMaster("local") lets us run & test the job right in our IDE */
val conf = new SparkConf(true)
.set("spark.cassandra.connection.host", "127.0.0.1")
.setMaster(“local[*]")
.setAppName(getClass.getName)
// Optionally
.set("cassandra.username", "cassandra")
.set("cassandra.password", “cassandra")
val sc = new SparkContext(conf)](https://image.slidesharecdn.com/lovetriangle-150701134944-lva1-app6892/75/A-Cassandra-Solr-Spark-Love-Triangle-Using-DataStax-Enterprise-46-2048.jpg)
Uploaded byPatrick McFadin
A Cassandra + Solr + Spark Love Triangle Using DataStax Enterprise
The document discusses the integration of Cassandra, Solr, and Spark using DataStax Enterprise for enhanced search capabilities and analytics. It covers topics such as schema creation, data partitioning, and real-time data processing with examples of queries and data management. Additionally, the document highlights the benefits of using this technology stack together for handling large datasets efficiently.
More Related Content
Similar to A Cassandra + Solr + Spark Love Triangle Using DataStax Enterprise
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
A Cassandra + Solr + Spark Love Triangle Using DataStax Enterprise
- 1. We know stuff.And get people excited about it. Rachel Pedreschi @RachelPedreschi Patrick McFadin @PatrickMcFadin A Cassandra + Solr + Spark Love Triangle Using DataStax Enterprise 1
- 2. Not (all queriespreplanned. Very Fast) Very (Ask anything, anytime. Slowest) Adhociness C* Search Analytics
- 3.
- 5. The bright bluebutterfly hangs on the breeze. [the] [bright] [blue] [butterfly] [hangs] [on] [the] [breeze] Tokens
- 6.
- 7. It can belonely for Solr
- 8.
- 9. Cassandra ✓ Highly available ✓Linear scalability ✓ Low latency OLTP queries C*
- 10.
- 11. Data Partitioning Application Data Center1 hash(key) => token(43) 80 10 3050 70 60 40 20
- 12. Replication Data Center 1 hash(key)=> token(43) replication factor = 3 80 10 3050 70 60 40 20 Application
- 13. Multi-Data Center Replication DataCenter 1 hash(key) => token(43) replication factor = 3 80 10 3050 70 60 40 20 Data Center 2 replication factor = 3 81 11 3151 71 61 41 21 Application
- 14. How does DSEintegrate Solr? C* C*/Solr Transactional Search
- 16. SELECT * FROM killrvideo.videos WHEREsolr_query='{ "q": "{!edismax qf="name^2 tags^1 description”}datastax" }'; SELECT id, value FROM keyspace.table WHERE token(id) >= -3074457345618258601 AND token(id) <= 3074457345618258603 AND solr_query='id:*'
- 17. Vocab Cassandra term Solrterm Column Family /Table Core Row Document Column Field SSTable Index
- 18. … … …… … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … 18 Node memory Node file system Client partition key1 first:Oscar last:Orange level:42 partition key2 first:Ricky last:Red Memtable (corresponds to a CQL table) Coordinator CommitLog AppendOnly … … … … … … … … … … … … SSTables Flush current state to SSTable Compact related SSTables W rite <3, Betty, Blue, 63> Acknowledge partition key3 first:Betty last:Blue level:63 Compaction Each write request … Periodically … Periodically …
- 19. … … …… … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … 19 Node memory Node file system Client 1 best 1 2 bright 2,3 Ram Buffer Coordinator … … … … … … … … … … … … Segments Flushes current state to Segment (Softcommit) Write <1,blue, 2,3> 3 blue 2,3 Merge (STW) Each write request … Periodically … On C* Memtable Flush, In memory segments hard commit to disk Shard Router … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … …
- 20. … … …… … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … 20 Node memory Node file system 1 best 1 2 bright 2,3 Ram Buffer … … … … … … … … … … … … Segments 3 blue 2,3 Not Searchable Searchable Coordinator Shard Router … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … …
- 21.
- 22.
- 23.
- 24.
- 25.
- 26. Standard Solr Indexing DSESearch Live Indexing
- 27.
- 28. Great combo Store aton of data Analyze a ton of data
- 29. Great combo Spark Streaming NearReal-time SparkSQL Structured Data MLLib Machine Learning GraphX Graph Analysis
- 30. Spark Streaming Near Real-time SparkSQL StructuredData MLLib Machine Learning GraphX Graph Analysis Great combo CREATE TABLE raw_weather_data ( wsid text, year int, month int, day int, hour int, temperature double, dewpoint double, pressure double, wind_direction int, wind_speed double, sky_condition int, sky_condition_text text, one_hour_precip double, six_hour_precip double, PRIMARY KEY ((wsid), year, month, day, hour) ) WITH CLUSTERING ORDER BY (year DESC, month DESC, day DESC, hour DESC); Spark Connector
- 31. How does itwork? OSS Stack Executer Master Worker Executer Executer Server
- 32. How does itwork? OSS Stack Master Worker 0-24 Token Ranges 0-100 25-49 50-74 75-99 I will only analyze 25% of the data. Worker Worker Worker
- 33.
- 34. 75-99 SELECT * FROM keyspace.table WHEREtoken(pk) > 75 AND token(pk) <= 99 Spark RDD Spark Partition Spark Partition Spark Partition Spark Connector Executer Executer Executer Worker Master
- 35. Spark RDD Spark Partition SparkPartition Spark Partition Master Worker Executer Executer Executer 75-99
- 36. Spark Connector Cassandra Cassandra + Spark Joinsand Unions No Yes Transformations Limited Yes Outside Data Integration No Yes Aggregations Limited Yes
- 37. Type mapping CQL TypeScala Type ascii String bigint Long boolean Boolean counter Long decimal BigDecimal, java.math.BigDecimal double Double float Float inet java.net.InetAddress int Int list Vector, List, Iterable, Seq, IndexedSeq, java.util.List map Map, TreeMap, java.util.HashMap set Set, TreeSet, java.util.HashSet text, varchar String timestamp Long, java.util.Date, java.sql.Date, org.joda.time.DateTime timeuuid java.util.UUID uuid java.util.UUID varint BigInt, java.math.BigInteger *nullable values Option
- 39. Confidential Let’s go codediving!
- 40. Behind the scenes… //Videos by id CREATE TABLE videos ( videoid uuid, userid uuid, name text, description text, location text, location_type int, preview_image_location text, tags set<text>, added_date timestamp, PRIMARY KEY (videoid) ); // Index for tag keywords CREATE TABLE videos_by_tag ( tag text, videoid uuid, added_date timestamp, userid uuid, name text, preview_image_location text, tagged_date timestamp, PRIMARY KEY (tag, videoid) ); Not a great idea Possible Index
- 41. // Videos byid CREATE TABLE videos ( videoid uuid, userid uuid, name text, description text, location text, location_type int, preview_image_location text, tags set<text>, added_date timestamp, PRIMARY KEY (videoid) And this? This? This?
- 43. 1) Spin upa new C* Cluster with search enabled using the DSE installer. $ sudo service dse cassandra -s 2) Run your schema DDL to create the C* keyspace and tables. 3) Run dse_tool on the videos table $ dsetool create_core killrvideo.videos generateResources=true 4) Use the Solr Admin to check sanity and make sure you have a core. 5) Write a CQL query with a Solr Search in it. SELECT * FROM killrvideo.videos WHERE solr_query='{ "q": "{!edismax qf="name^2 tags^1 description ”}?" }';
- 44. Search all ofthe things in 5 easy steps…
- 45. Now you getthis! SELECT name FROM videos WHERE solr_query = 'tags:crime*';
- 46. Attaching to Sparkand Cassandra // Import Cassandra-specific functions on SparkContext and RDD objects import org.apache.spark.{SparkContext, SparkConf} import com.datastax.spark.connector._ /** The setMaster("local") lets us run & test the job right in our IDE */ val conf = new SparkConf(true) .set("spark.cassandra.connection.host", "127.0.0.1") .setMaster(“local[*]") .setAppName(getClass.getName) // Optionally .set("cassandra.username", "cassandra") .set("cassandra.password", “cassandra") val sc = new SparkContext(conf)
- 47. Comment table example CREATETABLE comments_by_video ( videoid uuid, commentid timeuuid, userid uuid, comment text, PRIMARY KEY (videoid, commentid) ) WITH CLUSTERING ORDER BY (commentid DESC);
- 48. Simple example /** keyspace& table */ val tableRDD = sc.cassandraTable("killrvideo", “comments_by_video”) /** get a simple count of all the rows in the raw_weather_data table */ val rowCount = tableRDD.count() println(s"Total Rows in Comments Table: $rowCount") sc.stop()
- 49. Simple example /** keyspace& table */ val tableRDD = sc.cassandraTable("killrvideo", “comments_by_video”) /** get a simple count of all the rows in the comments_by_video table */ val rowCount = tableRDD.count() println(s"Total Rows in Comments Table: $rowCount") sc.stop() Executer SELECT * FROM killrvideo.comments_by_video Spark RDD Spark Partition Spark Connector
- 50. Using CQL SELECT userid FROMcomments_by_video WHERE videoid = '01860584-de45-018f-12be-5f81704e8033' val cqlRRD = sc.cassandraTable("killrvideo", “comments_by_video”) .select("userid") .where("videoid = ?”, “01860584-de45-018f-12be-5f81704e8033")
- 51. Even SQL! spark-sql> SELECTcast(videoid as String) videoid, count(*) c FROM comments_by_video GROUP BY cast(videoid as String) ORDER BY c DESC limit 10;
- 52. Saving back toCassandra // Create insert data val collection = sc.parallelize(Seq(("01860584-de45-018f-12be-5f81704e8033", "Great video", "cdaf6bd5-8914-29e0- f0b6-8b0bc6156777"), ("01860584-de45-018f-12be-5f81704e8033", "Hated it", "cdaf6bd5-8914-29e0-f0b6-8b0bc6156777"))) // Insert data into table collection.saveToCassandra("killrvideo", "comments_by_video", SomeColumns("videoid", "comment", "userid"))
- 53. val solrQueryRDD =sc.cassandraTable("killrvideo", “videos") .select("name").where("solr_query='tags:crime*'") solrQueryRDD.collect().map(row => println(row.getString("name")))
- 54.
- 55.
- 56.