Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How Big Fish Games Developed Real-Time Analytics Using Kafka Streams and Elasticsearch


Published on

(Kalah Brown, Big Fish Games) Kafka Summit SF 2018

Big Fish Games is a leading producer and distributor of casual and mid-core games. They have distributed more than 2.5 billion games to customers in 150 countries, representing over 450 unique mobile games and over 3,500 unique PC games.

Apache Kafka is used in our pipeline to process data generated across game play. Recently, we introduced real-time analytics of game data using Kafka Streams integrated with Elasticsearch. This allows us to monitor the results of live operations (aka, live ops such as weekend events, limited time offers and user acquisition campaigns) and to make changes to these events after they have gone live. The result is a better game experience, better control of the game’s economy and overall better optimization of the live ops.

This presentation will include a detailed explanation of how we used Kafka Streams to transform raw data into Elasticsearch documents, and how the application was scaled to over a million daily active users. It will also touch on the limitations discovered with Kafka Connect integration with Elasticsearch and how to use Elasticsearch bulk processing with Kafka Streams. The presentation will also discuss how Dropwizard can provide a framework for monitoring and alerting a Kafka Stream application. Lastly, a demonstration of the real-time dashboards which are powered by Kafka Streams will be included.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

How Big Fish Games Developed Real-Time Analytics Using Kafka Streams and Elasticsearch

  1. 1. How Big Fish Games Developed Real-Time Analytics Using Kafka Streams and Elastic Search
  2. 2. 2 Big Fish Confidential Big Fish Games o Leading producer and distributor of casual games. o 2.5 billion games distributed to customers in 150 countries o 450 unique mobile games o 3500 unique PC games
  3. 3. 3 Big Fish Confidential Cooking Craze
  4. 4. 4 Big Fish Confidential Live Operations Real Time Dashboard Let’s Dish
  5. 5. 5 Big Fish Confidential Live Ops and Free to Play Games o Monetization through “in app” purchases o Games evolve and are extended indefinitely o Live Operations o Games which allow real-time updates to improve the game or align to audiences needs
  6. 6. 6 Big Fish Confidential Monitoring a Forced Update
  7. 7. 7 Big Fish Confidential Monitoring New Content
  8. 8. 8 Big Fish Confidential Monitoring Event Participation
  9. 9. 9 Big Fish Confidential Kafka at Big Fish
  10. 10. 10 Big Fish Confidential Let’s Dish Dashboards Data Sources o Game Telemetry Events from a Kafka Topic o restaurant leave , restaurant join o session start, session foreground, session background, session heartbeat o recipe complete, recipe burn o Hive Data o Install Date o bookings , bookings Last30 o channel Type o bfgudid
  11. 11. 11 Big Fish Confidential Elastic Search Document{ "_index": "letsdish-2018-19", "_type": "recipes", "_id": "f4858a61-cf4e-400c-952e-6e940774e1ef", "_score": 1, "_source": { "count": 1, "restaurantScheduleId": "french", "restaurantsAvailable": 5, "installDate": "2018-05-09", "bookingsBin": "No Monetization", "bookingsBinLast30": "No Monetization", "channelType": "Organic", "environment": "prod", "appName": "LetsDish", "bfgudid": "f8e1d52b7a5edb26aafed418e2a25c6879766f9c", "platform": "android", "customUtc": 1526078368, "countryCode": "US", "appVersion": "42", "guid": "f4858a61-cf4e-400c-952e-6e940774e1ef“ (…) }, "fields": { "customUtc": [ "2018-05-11T22:39:28.000Z" ], "installDate": [ "2018-05-09T00:00:00.000Z" ] } }
  12. 12. 12 Big Fish Confidential Two Streams APIs o Streams DSL o Abstraction of Processor API o Used for Let’s Dish Dashboards o Streams Processor API o More flexible o Finer control of stream processing
  13. 13. 13 Big Fish Confidential Let’s Dish Streaming Transformation Topology
  14. 14. 14 Big Fish Confidential Mapping, Filtering and Branching
  15. 15. 15 Big Fish Confidential Map Function KeyValueMapper<String, String, <String,GtEvent>> Maps JSON String Value to Game Telemetry Pojo: GtEvent
  16. 16. 16 Big Fish Confidential Java 8 Lambda Function Map Function public static KeyValueMapper<String, String, KeyValue<String, GtEvent>> mapStringToGtEvent = (k, v) -> { GtEvent gtEvent; try { gtEvent = new Gson().fromJson(v,GtEvent.class); } catch (Exception e) { gtEvent = new CustomEvent(); } return new KeyValue<String, GtEvent>(k, gtEvent); };
  17. 17. 17 Big Fish Confidential Filter Predicates KStreams filter and branch methods take a Predicate interface KStream.filter(Predicate) KStream.branch(Predicate[])
  18. 18. 18 Big Fish Confidential Java 8 Lambda Functions Filter Predicates public static Predicate<String, GtEvent> isRecipeComplete = (k, v) -> "recipe_complete".equals(; public static Predicate<String, GtEvent> isSession = (k, v) -> Arrays.asList("session_start", "session_foreground", "session_background", "session_heartbeat") .contains(;
  19. 19. 19 Big Fish Confidential Map and Filters Combined public static KStream<String, GtEvent>[] createBranchedStreams( KStream<String, String> source) { return source .map(Mappers.mapStringToGtEvent) .branch( FilterPredicates.isRecipeComplete, FilterPredicates.isRecipeBurn, FilterPredicates.isSession, FilterPredicates.isRestaurant); }
  20. 20. 20 Big Fish Confidential Four KStreams o Recipe Burn o Recipe Complete o Restaurant Unlocked o Restaurant Names
  21. 21. 21 Big Fish Confidential Aggregated Player Purchases Table Enhance KStreams with Hive Table ID FIRST INSTALL DATE BOOKINGS BIN BOOKINGS LAST 30 DAYS CHANNEL TYPE 1 9/12/2018 $1 <= $5 $1 <= $5 paid 2 9/12/2018 $5 <= $20 $5 <= $20 organic 3 9/12/2018 $1 <= $5 $1 <= $5 paid
  22. 22. 22 Big Fish Confidential Kafka Topic
  23. 23. 23 Big Fish Confidential Joining Streams
  24. 24. 24 Big Fish Confidential Types of Joins o Windowed Join (No State) o KStream->KStream o Non-Windowed Joins (State Maintained ) o KStream->KTable o KTable->KTable
  25. 25. 25 Big Fish Confidential Using Group By Key Creating a KTable from Kstream .groupByKey(Serialized.with(stringSerde,longSerde)) .reduce((oldValue, newValue) -> Math.max(oldValue, newValue), Materialized.<String, Long, KeyValueStore<Bytes, byte[]>> as("restaurants-unlocked-store") .withKeySerde(stringSerde) .withValueSerde(longSerde));
  26. 26. 26 Big Fish Confidential Recipe Burn Stream to Restaurant Unlocked KTable Steps to Joining recipeBurnKStream.leftJoin( restaurantUnlockedKTable, RecipeValueJoiners.joinRecipeToRestaurantUnlocked, Joined.with( stringSerde,enrichedEventSerde,longSerde) );
  27. 27. 27 Big Fish Confidential Where are the join conditions? Join Keys KStream<String, EnrichedEvent> recipeBurnKStream Joined with KTable<String, Long> restaurantUnlockedKTable on the Keys.
  28. 28. 28 Big Fish Confidential Value Joiner public static ValueJoiner<EnrichedEvent, Long, EnrichedEvent> joinRecipeToRestaurantUnlocked = (e, restaurantsUnlocked) -> { e.enrichedRecipeEvent.restaurantsUnlocked = (restaurantsUnlocked == null) ? -1L : restaurantsUnlocked; return e; };
  29. 29. 29 Big Fish Confidential • EnrichedEventSerde • org.apache.kafka.common.serialization interfaces Custom Serialization Joined.with(stringSerde,enrichedEventSerde ,longSerde)
  30. 30. 30 Big Fish Confidential Join Recap o Keys are used for joining o KStream<String, EnrichedEvent> o KTable<String, Long> o Value Joiners o Return the number of restaurants unlocked o If null return -1 o Custom Serde for EnrichedEvent
  31. 31. 31 Big Fish Confidential Elastic Search Document { "_index": "letsdish-2018-19", "_type": "recipes", "_id": "f4858a61-cf4e-400c- 952e-6e940774e1ef", "_score": 1, "_source": { "count": 1, "restaurantScheduleId": "fr ench", "restaurantsAvailable": 5, "installDate": "2018-05- 09", "bookingsBin": "No Monetization", "bookingsBinLast30": "No Monetization", …
  32. 32. 32 Big Fish Confidential Elastic Search Bulk API o Bulk Processing Greatly Increases the Index Speed o Two possible implementations of the Bulk API o Direct calls to the REST service o Elastic Search Libraries o BulkProcessor and RestClientBuilder o Wrap lower level work
  33. 33. 33 Big Fish Confidential Continue to Build Real Time Dashboards for Games