Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Squirrels and Elephants - The InnoGames Big Data and Streaming Infrastructure

79 views

Published on

Life doesn't happen in batches. You have to process data on time as it happens to make use of the time-value of information.

Apart from a general introduction to stream processing and Apache Flink, this presentation shows how we as a successful data-driven gaming company apply this concept with our data architecture and how we use Apache Flink to run several streaming applications.

Published in: Technology
  • Be the first to comment

Squirrels and Elephants - The InnoGames Big Data and Streaming Infrastructure

  1. 1. SQUIRRELS AND ELEPHANTS Big Data and Streaming at InnoGames
  2. 2. LET‘S GO ON A ROADTRIP
  3. 3. IMAGINE You are driving with your family on the backseat
  4. 4. IMAGINE There is a lot of traffic, you have to concentrate
  5. 5. IMAGINE And now, while driving, you are closing your eyes
  6. 6. HOW DO YOU FEEL?
  7. 7. WOULD YOU EVER DO THAT? NOPE
  8. 8. METAPHOR The car is your company, team or project The passengers are your colleagues
  9. 9. WOULD YOU EVER DO THAT? NOPE IN A METAPHORICAL SENSE
  10. 10. YOU HAVE TO PROCESS DATA ON TIME AS IT HAPPENS
  11. 11. BATCH PROCESSING… …might cause accidents Because…
  12. 12. LIFE DOESN’T HAPPEN IN BATCHES https://mapr.com/ebooks/streaming-architecture/chapter-01-why-event-streaming.html © Ellen Friedman, Ted Dunning
  13. 13. ICE CREAM AND GAMING?
  14. 14. SIMILARITIES MAKE US HAPPY
  15. 15. SIMILARITIES WHICH TASTES BETTER?
  16. 16. SIMILARITIES WHICH TASTES BETTER?
  17. 17. SIMILARITIES THE FIRST IMPRESSION COUNTS The moment the customer enters the shop or the player plays his first session is crucial HALO EFFECT When one trait of a person or thing is used to make an overall judgment of that person or thing
  18. 18. IN ORDER TO MAKE A POSITIVE IMPACT A RESPONSE NEEDS TO HAPPEN QUICKLY
  19. 19. TIME-VALUE OF INFORMATION
  20. 20. REAL-TIME USER REPORTS TRAFFIC GAS PRICE SPEED TRAPS
  21. 21. RESPOND TO LIFE AS IT HAPPENS
  22. 22. STREAM PROCESSING STREAMS OF DATA GPS DATA WEB INTERACTION SENSOR DATA
  23. 23. STREAM PROCESSING PROCESSING DATA IN MOTION
  24. 24. STREAM PROCESSING YOUR CODE SOURCE SINKOPERATOR
  25. 25. STREAM PROCESSING LAKE
  26. 26. STREAM PROCESSING LAKE
  27. 27. APACHE FLINK
  28. 28. APACHE FLINK Framework and distributed process engine for stateful computations on unbounded and bounded data streams
  29. 29. STREAMS TIME WINDOWS
  30. 30. EVERYTHING IS A STREAM UNBOUNDED STREAMS BOUNDED STREAMS
  31. 31. EVERYTHING IS A STREAM UNBOUNDED STREAMS BOUNDED STREAMS AKA BATCH PROCESSING
  32. 32. TIME IN STREAMING EPISODE I EPISODE II EPISODE III EPISODE IV EPISODE V EPISODE VI EPISODE VII EPISODE VIII EPISODE IX 1999 2002 2005 1977 1980 1983 2015 2017 2019 The Phantom Menace Attack of the Clones Revenge of the Sith A New Hope The Empire Strikes Back Return of the Jedi The Force Awakens The Last Jedi ? ORDERED BY EVENT TIME PROCESSING TIME
  33. 33. TIME IN STREAMING EPISODE I EPISODE II EPISODE IIIEPISODE IV EPISODE V EPISODE VI EPISODE VII EPISODE VIII EPISODE IX 1999 2002 20051977 1980 1983 2015 2017 2019 The Phantom Menace Attack of the Clones Revenge of the Sith A New Hope The Empire Strikes Back Return of the Jedi The Force Awakens The Last Jedi ? EVENT TIME ORDERED BY PROCESSING TIME
  34. 34. TUMBLING WINDOWS 9 1 3 2 6 8 1 3 9 8 4 5 9 1 3 2 6 8 1 3 9 8 4 5 15 18 26 SENSOR SUM
  35. 35. SLIDING WINDOWS 9 1 3 2 6 8 1 3 9 8 4 5 9 1 3 2 6 8 1 3 9 8 4 5 9 1 3 2 6 8 1 3 9 8 4 5 15 18 26 19 21 SENSOR SUM
  36. 36. BUILDING BLOCKS DATA SOURCE TRANSFORMATION DATA SINK
  37. 37. API BUILDING BLOCKS DATA SOURCE TRANSFORMATION DATA SINK
  38. 38. BUILDING BLOCKS SQL / TABLE API DataStream API ProcessFunction APIs (dynamic tables) (streams, windows) (events, state, time) HIGH LEVEL ANALYTICS API STREAM AND BATCH DATA PROCESSING STATEFUL EVENT- DRIVEN APPLICATIONS CONCISENESS EXPRESSIVENESS
  39. 39. LET‘S HAVE A CLOSER LOOK
  40. 40. LET‘S HAVE A CLOSER LOOK final StreamExecutionEnvironment env = getExecutionEnvironment(); final DataStreamSource<Integer> stream = env.fromElements(1, 2, 3, 4); stream .map((MapFunction<Integer, Integer>) i -> i + 2) .filter((FilterFunction<Integer>) i -> i % 2 == 0) .print(); env.execute(); DATA SOURCE TRANSFORMATION DATA SINK
  41. 41. RUNTIME YOUR FLINK APP FLINK RUNTIMEDEPLOY
  42. 42. RUNTIME
  43. 43. BIG DATA AND STREAMING AT INNOGAMES
  44. 44. TEAM
  45. 45. TEAM BUSINESS INTELLIGENCE DATA ENGINEERING DATA SCIENCE OPERATIONS
  46. 46. EVENT TRACKING quest build fight invite
  47. 47. EVENT TRACKING 1.500.000.000EVENTS PER DAY
  48. 48. DATA ARCHITECTURE DATA PIPELINE DATA PLATFORM milliseconds, seconds, minutes hours, days, years
  49. 49. DATA ARCHITECTURE SQUIRREL ELEPHANT
  50. 50. DATA ARCHITECTURE EVENT CLIENT EVENT CLIENT EVENT CLIENT EVENTGATEWAY EVENT BUS STREAM PROCESSING DISTRIBUTED DATA STORE DISTRIBUTED BATCH PROCESSING BUSINESS INTELLIGENCE
  51. 51. DATA ARCHITECTURE EVENT CLIENT EVENT CLIENT EVENT CLIENT EVENTGATEWAY EVENT BUS STREAM PROCESSING DISTRIBUTED DATA STORE DISTRIBUTED BATCH PROCESSING BUSINESS INTELLIGENCE
  52. 52. DATA ARCHITECTURE EVENT CLIENT EVENT CLIENT EVENT CLIENT EVENTGATEWAY EVENT BUS STREAM PROCESSING DISTRIBUTED DATA STORE DISTRIBUTED BATCH PROCESSING BUSINESS INTELLIGENCE STREAM PROCESSING
  53. 53. USE CASE EVENT METRICS
  54. 54. Metrics.java stream .map(streamEvent -> new Tuple2<>(streamEvent.getEventName(), 1)) .keyBy(0) .timeWindow(Time.minutes(1)) .sum(1) .addSink(graphiteSink).setParallelism(1).name("event_counts");
  55. 55. USE CASE EVENT METRICS
  56. 56. USE CASE LOG00 MONITOR
  57. 57. KeyedStream<StreamEvent, Integer> stream = events .filter(event -> Arrays.asList("reg", "login").contains(event.getEventName())) .keyBy((KeySelector<StreamEvent, Integer>) StreamEvent::getPlayerId); Log00.java
  58. 58. Pattern<StreamEvent, StreamEvent> pattern = Pattern.<StreamEvent>begin("reg").where(new SimpleCondition<StreamEvent>() { @Override public boolean filter(StreamEvent event) { return event.getEventName().equals("reg"); } }).followedBy("login").where(new SimpleCondition<StreamEvent>() { @Override public boolean filter(StreamEvent event) { return event.getEventName().equals("login"); } }).within(Time.seconds(60)); Log00.java
  59. 59. PatternStream<StreamEvent> patternStream = CEP.pattern(stream, pattern); DataStream<Either<PatternResult, PatternResult>> patternResultStream = patternStream.select( (p, ts) -> sendTimeoutToGraphite(p, ts), p -> sendSuccessToGraphite(p) ); Log00.java
  60. 60. USE CASE LOG00 MONITOR
  61. 61. USE CASE NEAR TIME CRM(NTCRM)
  62. 62. USE CASE NTCRM EVENT BUS EVENT CLIENT EVENTGATEWAY PLAYER DATANTCRM React to events with interstitials in less than 10 seconds
  63. 63. USE CASE NTCRM Elvenar has a trading feature that sometimes causes confusion. With NTCRM we can react to this and show more details within interstitials exactly when the player needs it.
  64. 64. JUST DO IT DEMO TIME Check it out on Github: https://github.com/prenomenon/codetalks-flinkdemo
  65. 65. GET IN TOUCH InnoGames GmbH Friesenstrasse 13 20097 Hamburg https://www.innogames.com Volker Janz Senior Software Developer Corporate Systems - Analytics
  66. 66. GET IN TOUCH @prenomenon feedback appreciated
  67. 67. LIFE DOESN’T HAPPEN IN BATCHES
  68. 68. EAT ICE CREAM AND STREAM ON Great Flink training: http://training.data-artisans.com
  69. 69. NEXT UP EVENT CLIENT EVENT CLIENT EVENT CLIENT EVENTGATEWAY EVENT BUS STREAM PROCESSING DISTRIBUTED DATA STORE DISTRIBUTED BATCH PROCESSING BI BUSINESS INTELLIGENCE
  70. 70. THAT’S IT FOR NOW…
  71. 71. BACKUP / DETAILS The following slides are not part of my talk but might give the reader more insights later
  72. 72. COMPANY SNAPSHOT More than 400 employees Founded 2007 in Germany Headquarter in Hamburg +160m EUR revenue made in 2017 7 live games >30 language versions
  73. 73. I AM LEGEND OUR PORTFOLIO Simulation Strategy RPG Browser Multi-device Mobile
  74. 74. SQUIRREL TESTS <dependency <groupId>org.apache.flink</groupId> <artifactId>flink-test-utils_2.11</artifactId> <version>1.6.1</version> </dependency>
  75. 75. WINDOWING KEYED NON-KEYED TASK 1 TASK N SOURCE TASK 1SOURCE KEY 1 KEY N ALL DATA
  76. 76. STATE SOURCE MAP DATA SINK SOURCE MAP SUM(C,D) OFFSET OFFSET SUM(A,B) AB CD
  77. 77. RABBIT HOLE
  78. 78. RUNTIME SOURCE MAP PRINTFILTER STREAMING DATAFLOW (CONDENSED VIEW) OPERATOR CHAIN OPERATOR OPERATOR TASK TASK TASK SOURCE MAP PRINT FILTER OPERATOR CHAIN OPERATOR OPERATOR SUBTASK SUBTASK TASKSOURCE MAP FILTER OPERATOR CHAIN OPERATOR SUBTASK SUBTASK STREAM PARTITIONS STREAMING DATAFLOW (PARALLELIZED VIEW)
  79. 79. RUNTIME SOURCE MAP PRINT FILTER OPERATOR CHAIN OPERATOR OPERATOR SUBTASK SUBTASK TASKSOURCE MAP FILTER OPERATOR CHAIN OPERATOR SUBTASK SUBTASK STREAM PARTITIONS STREAMING DATAFLOW (PARALLELIZED VIEW) A Flink cluster has a JOB MANAGER and multiple TASK MANAGERS. Each of those is a JVM.
  80. 80. RUNTIME Each Task Manager can manage MULTIPLE THREADS executing TASKS / SUBTASKS. SOURCE MAP PRINT FILTER OPERATOR CHAIN OPERATOR OPERATOR THREAD THREAD THREADSUBTASK SUBTASK TASKSOURCE MAP FILTER OPERATOR CHAIN OPERATOR THREAD THREAD SUBTASK SUBTASK STREAM PARTITIONS STREAMING DATAFLOW (PARALLELIZED VIEW)
  81. 81. CHECKPOINTING checkpoint barrier n checkpoint barrier n-1 checkpoint n+1 checkpoint n checkpoint n-1 Consistent, incremental snapshots of distributed data stream and operator state Based on a paper from 1985, inspired by the Chandy-Lamport-Algorithm
  82. 82. STATE OPERATOR STATE KEYED STATE Bound only to an operator Bound to an operator and key PLUGGABLE BACKEND MULTIPLE PRIMITIVES SUPPORTED GUARANTEED CONSISTENCY IN CASE OF A FAILURE

×