Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Reactive Elasticsearch with Akka Streams

5,485 views

Published on

#渋谷java

Published in: Software
  • Be the first to comment

  • Be the first to like this

Reactive Elasticsearch with Akka Streams

  1. 1. Reactive Elasticsearch with Akka Streams 第二十回 #渋谷java Naoki Takezoe @takezoen BizReach, Inc
  2. 2. Motivation ● Real-time indexing in Elasticsearch ● Resiliece of indexing process ● Tuning indexing load automatically
  3. 3. Basic structure Source Flow Sink Read Write Pull from down stream Push from upstream
  4. 4. Akka Streams ● An implementation of Reactive Streams based on Scala and Akka ○ Java API is also available ● Akka HTTP ○ Covers HTTP server and client applications ● Alpakka ○ Provides various Akka Streams connectors
  5. 5. Example // Source val source = Source(1 to 100) // Sink val sink = Sink.foreach { x: Int => println(x) } // Just connect Source and Sink directly source.runWith(sink) // Put Flow between Source and Sink source.map(x => x * 2).runWith(sink)
  6. 6. Elasticsearch connector is availavle in Alpakka!
  7. 7. ElasticsearchSource val source = ElasticsearchSource.typed[Book]( "source", // Index name "book", // Type name """{"match_all": {}}""", // Query ElasticsearchSourceSettings(5) // Buffer size )
  8. 8. Implementation of ElasticsearchSource ● Scroll Elasticsearch and buffer documents ● Push docs when pulled from down stream ● Read next window if read buffer is empty
  9. 9. ElasticsearchSink val sink = ElasticsearchSink.typed[Book]( "sink", // Index name "book", // Type name ElasticsearchSinkSettings(5) // Buffer size )
  10. 10. Implementation of ElasticsearchSink ● Pull until buffer is full if buffer isn't full ● Indexing by bulk request buffered docs ● Run these processes in parallel
  11. 11. Demo See testcase to know more details: https://github.com/akka/alpakka/blob/master/elast icsearch/src/test/scala/akka/stream/alpakka/elasti csearch/ElasticsearchSpec.scala
  12. 12. TODO ● Automatic pull amount tuning ○ By Looking up to metrics of Elasticsearch cluster ● Partial retry ○ Retry failed documents when bulk request is failed partially ● Sliced scroll ○ Elasticsearch 5.x supports sliced scroll that makes possible to scroll in parallel

×