This document summarizes a presentation about real-time data analytics with Apache Flink and Apache BEAM. It discusses possible real-time and batch processing systems using AWS services, challenges of streaming systems including state management, and demos of analyzing user clickstreams and taxi trips with Apache Flink, Kafka, and Elasticsearch. It also covers advantages of Apache BEAM including a unified batch and streaming API that can run on different frameworks like Flink, benefits of native support for Java, Python, and Go, and how it allows mixing languages in pipelines.

1. Analítica de datos en tiempo real con
Apache Flink y Apache BEAM
Javier Ramírez - @supercoco9
Developer Advocate - Amazon Web Services
Noviembre 3-4-5, 2020

2. Un posible sistema de tiempo (casi) real
AWS Cloud
Transformaciones/Validaciones/
Filtrado/Agregados/Analítica

3. Analítica de un posible clickstream
AWS Cloud
Parsear
clicks
Cada minuto, calcular número de usuarios activos
Cada 5 minutos, productos comprados por categoría
Cada minuto, ranking de productos más visitados
Cada hora, total de pedidos
En tiempo real, seleccionar anuncios
En tiempo real, detectar comportamientos anómalos

4. You don’t know the volume of the data before you start
Data is never complete
Low-latency is expected
Events might be related, but data can come out of order
System should remain available during upgrades
Retos de trabajar con sistemas en streaming

5. Stateless processing
• Working on per-element streams is relatively easy (i.e. change format of each item, or filter our
records based on their own properties)
•
13:00 14:008:00 9:00 10:00 11:00 12:00 Processing Time
Graphics from The Beam Model. By Tyler Akidau and Frances Perry. https://beam.apache.org/community/presentation-materials/
The real fun starts when you need to do transforms/ aggregations over groups of elements:
group by, count, max, average, joins, filtering based on properties from related records, or
complex pattern detection

6. Stateful processing: Processing-Time based windows
13:00 14:008:00 9:00 10:00 11:00 12:00
Processing
Time
Graphics from The Beam Model. By Tyler Akidau and Frances Perry. https://beam.apache.org/community/presentation-materials/

7. Stateful processing: Event-Time Based Windows
Event Time
Processing
Time 11:0010:00 15:0014:0013:0012:00
11:0010:00 15:0014:0013:0012:00
Input
Output
Graphics from The Beam Model. By Tyler Akidau and Frances Perry. https://beam.apache.org/community/presentation-materials/

8. Stateful processing: Session Windows
Event Time
Processing
Time 11:0010:00 15:0014:0013:0012:00
11:0010:00 15:0014:0013:0012:00
Input
Output
Graphics from The Beam Model. By Tyler Akidau and Frances Perry. https://beam.apache.org/community/presentation-materials/

9. Reto: mantener el estado entre eventos
• El sistema tiene que saber en qué etapa está cada elemento, y si está en un estado
intermedio o ya se ha procesado por completo
• Para operaciones que necesiten ”memoria”, el sistema tiene que mantener el
estado de los elementos y cálculos intermedios
• En un sistema suficientemente grande, el estado será distribuido

10. Apache Flink
• Stateful computations over data streams. Operaciones con estado
sobre flujos de datos
https://flink.apache.org

11. package com.javier_cloud.demos.streaming;
import com.javier_cloud.demos.streaming.util.AppProperties;
import org.apache.flink.api.common.serialization.SimpleStringSchema;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer011;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011;
public class KafkaStreaming {
public static void main(String[] args) throws Exception {
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
AppProperties.loadProperties(env);
Properties kafkaProperties = new Properties();
String kafka_servers = AppProperties.getBootstrapServers();
kafkaProperties.setProperty("bootstrap.servers", kafka_servers);
kafkaProperties.setProperty("group.id", AppProperties.getGroupId());
DataStream<String> stream = env
.addSource(new FlinkKafkaConsumer011<>(AppProperties.getInputStream(), new SimpleStringSchema(),
kafkaProperties));
FlinkKafkaProducer011<String> streamSink = new FlinkKafkaProducer011<>(kafka_servers, AppProperties.getOutputStream(),
new SimpleStringSchema());
streamSink.setWriteTimestampToKafka(true);
stream.addSink(streamSink);
env.execute("Basic Flink Kafka Streaming");
}
}

12. package com.javier_cloud.demos.streaming;
import com.javier_cloud.demos.streaming.util.AppProperties;
import com.javier_cloud.demos.streaming.util.ESSinkBuilder;
import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.common.serialization.SimpleStringSchema;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer011;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011;
import org.apache.flink.util.Collector;
import java.util.Properties;
public class KafkaStreamingToES {
public static void main(String[] args) throws Exception {
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
AppProperties.loadProperties(env);
Properties kafkaProperties = new Properties();
String kafka_servers = AppProperties.getBootstrapServers();
kafkaProperties.setProperty("bootstrap.servers", kafka_servers);
kafkaProperties.setProperty("group.id", AppProperties.getGroupId());
DataStream<String> stream = env
.addSource(new FlinkKafkaConsumer011<>(AppProperties.getInputStream(), new SimpleStringSchema(),
kafkaProperties));
FlinkKafkaProducer011<String> streamSink = new FlinkKafkaProducer011<String>(kafka_servers,
AppProperties.getOutputStream(), new SimpleStringSchema());
streamSink.setWriteTimestampToKafka(true);
stream.addSink(streamSink);
// split up the lines in pairs (2-tuples) containing: (word,1), then sum
DataStream<Tuple2<String, Integer>> counts = stream.flatMap(new Tokenizer()).keyBy(0).sum(1);
counts.addSink(ESSinkBuilder.buildElasticSearchSink(AppProperties.getESWordCountIndex()));
env.execute("Streaming from a Kafka topic, echoing the message to Kafka, and outputting aggregations to ElasticSearch");
}
public static final class Tokenizer implements FlatMapFunction<String, Tuple2<String, Integer>> {
@Override
public void flatMap(String value, Collector<Tuple2<String, Integer>> out) { ... }
}
}

13. from pyflink.dataset import ExecutionEnvironment
from pyflink.table import TableConfig, DataTypes, BatchTableEnvironment
from pyflink.table.descriptors import Schema, OldCsv, FileSystem
exec_env = ExecutionEnvironment.get_execution_environment()
exec_env.set_parallelism(1)
t_config = TableConfig()
t_env = BatchTableEnvironment.create(exec_env, t_config)
t_env.connect(FileSystem().path('/tmp/input'))
.with_format(OldCsv()
.field('word', DataTypes.STRING()))
.with_schema(Schema()
.field('word', DataTypes.STRING()))
.create_temporary_table('mySource')
t_env.connect(FileSystem().path('/tmp/output'))
.with_format(OldCsv()
.field_delimiter('t')
.field('word', DataTypes.STRING())
.field('count', DataTypes.BIGINT()))
.with_schema(Schema()
.field('word', DataTypes.STRING())
.field('count', DataTypes.BIGINT()))
.create_temporary_table('mySink')
t_env.from_path('mySource')
.group_by('word')
.select('word, count(1)')
.insert_into('mySink')
t_env.execute("tutorial_job")

14. Algunos operadores en Apache Flink
Tipo Operadores/Funciones
A nivel de elemento Map, FlatMap, Filter, Select, Project
Agregados KeyBy, Reduce, Fold, sum, min, max
Trabajar con ventanas globales,
de tiempo de proceso o de
evento
Window (TumblingEventTime, TumblingProcessingTime, SlidingEventTime,
SlidingProcessingTime, EventTimeSession, ProcessingTimeSession, GlobalWindows),
WindowAll, Window Apply, trigger, evictor, allowedLateness, sideOutputLateData, getSideOutput
Combinar varios streams
Union, Join, OuterJoin, Cross, Distinct, IntervalJoin, CoGroup, Connect, CoMap, CoFlatMap,
Split, PartitionCustom, Rebalance, Rescale, Shuffle, First-n, SortPartition
Optimizaciones Iterate, StartNewChain,DisableChaining
Bucles y asincronía Iterate, AsyncFunctions
SQL
Funciones SQL para: Comparison, Logical, Arithmetic, String, Temporal, Conditional, Type,
Aggregate, Collection, Columnar

16. ¿Por qué Flink lo peta?
• Manejo propio de la memoria
• Serialización a un formato binario propio
• Gestión optimizada de las comunicaciones entre nodos y tareas
• Opciones para almacenar el estado
• Checkpoints y savepoints
• Varios niveles de abstracción en sus APIs

17. Demo: Analizando clickstream de usuarios
Usando Apache Kafka, Apache Flink, y ElasticSearch

18. ¿Qué pinta en todo esto Apache BEAM?

19. Ventajas de Apache BEAM
• API Unificada para Batch y Stream
• Portable a diferentes Runners (sin vendor lock-in): Flink, Spark,
Samza, DataFlow, Nemo, Twister2, Hazelcast Jet...
• Soporte nativo de Java, Python, y Go (con todas sus librerías)
• Posibilidad de mezclar lenguajes en una misma pipeline

20. from __future__ import absolute_import
import re
from past.builtins import unicode
import apache_beam as beam
from apache_beam.io import ReadFromText
from apache_beam.io import WriteToText
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.options.pipeline_options import SetupOptions
def run(argv=None, save_main_session=True):
"""Main entry point; defines and runs the wordcount pipeline."""
pipeline_options = PipelineOptions(pipeline_args)
pipeline_options.view_as(SetupOptions).save_main_session = save_main_session
with beam.Pipeline(options=pipeline_options) as p:
# Read the text file[pattern] into a PCollection.
lines = p | ReadFromText(known_args.input)
# Count the occurrences of each word.
counts = (
lines
| 'Split' >> (
beam.FlatMap(lambda x: re.findall(r'[A-Za-z']+', x)).
with_output_types(unicode))
| 'PairWithOne' >> beam.Map(lambda x: (x, 1))
| 'GroupAndSum' >> beam.CombinePerKey(sum))
# Format the counts into a PCollection of strings.
def format_result(word_count):
(word, count) = word_count
return '%s: %s' % (word, count)
output = counts | 'Format' >> beam.Map(format_result)
output | WriteToText(known_args.output)
if __name__ == '__main__':
run()

21. package com.amazonaws.samples.beam.taxi.count;
import org.apache.beam.runners.flink.FlinkRunner;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.io.kinesis.KinesisIO;
import org.apache.beam.sdk.transforms.*;
(..)
import software.amazon.awssdk.services.cloudwatch.model.Dimension;
public class BeamTaxiCount {
public static void main(String[] args) {
String[] kinesisArgs = TaxiCountOptions.argsFromKinesisApplicationProperties(args,"BeamApplicationProperties");
TaxiCountOptions options = PipelineOptionsFactory.fromArgs(ArrayUtils.addAll(args, kinesisArgs)).as(TaxiCountOptions.class);
options.setRunner(FlinkRunner.class);
options.setAwsRegion(Regions.getCurrentRegion().getName());
PipelineOptionsValidator.validate(TaxiCountOptions.class, options);
Pipeline p = Pipeline.create(options);
PCollection<TripEvent> input = p
.apply("Kinesis source", KinesisIO
.read()
.withStreamName(options.getInputStreamName())
.withAWSClientsProvider(new DefaultCredentialsProviderClientsProvider(Regions.fromName(options.getAwsRegion())))
.withInitialPositionInStream(InitialPositionInStream.LATEST)
)
.apply("Parse Kinesis events", ParDo.of(new EventParser.KinesisParser()));
PCollection<Metric> metrics = input
.apply("Group into 5 second windows", Window
.<TripEvent>into(FixedWindows.of(Duration.standardSeconds(5)))
.triggering(AfterWatermark
.pastEndOfWindow()
.withEarlyFirings(AfterProcessingTime.pastFirstElementInPane().plusDelayOf(Duration.standardSeconds(15)))
)
.withAllowedLateness(Duration.ZERO)
.discardingFiredPanes() )
.apply("Count globally", Combine
.globally(Count.<TripEvent>combineFn())
.withoutDefaults()
)
.apply("Map to Metric", ParDo.of(
new DoFn<Long, Metric>() {
@ProcessElement
public void process(ProcessContext c) {
c.output(new Metric(c.element().longValue(), c.timestamp()));
}
}
));
prepareMetricForCloudwatch(metrics)
.apply("CloudWatch sink", ParDo.of(new CloudWatchSink(options.getInputStreamName())));
p.run().waitUntilFinish();
}

22. Demo: Analizando trayectos de Taxi con Apache
BEAM
Usando tiempo real, y batch para el backfilling

23. Apache Flink en AWS
Modelo de responsabilidad compartida
Amazon Kinesis
Data Analytics for
Apache Flink
Amazon EMR
Hadoop/Yarn
gestionado
Más
gestionado
Menos
gestionado
AWS gestiona El cliente gestiona
• Almacenamiento y estado
• Métricas, monitorización, e interfaz dedicado
• Hardware, software, red
• Provisionado y autoescalado
• Código de la aplicación
• Configuración básica
• Escalado del cluster (basado en Yarn)
• Hardware, software, red
• Código de la aplicación
• Configuración de estado y almacenamiento
• Configuración de seguridad del interfaz
• Gestión/ejecución de las aplicaciones
• Plano de control de orquestación de contenedores
• Hardware, software del orquestador, red (física)
• Código y configuración completa de la aplicación
• Instalación y actualización del software
• Gestión de clusters, seguridad, y configuración de red
• Gestión/ejecución de las aplicaciones
• Escalado
• Hardware, software, red (física)
• Código y configuración completa de la aplicación
• Instalación y actualización del software
• Seguridad, y configuración de red
• Gestión/ejecución de las aplicaciones
• Escalado
• Provisionado, instalación y gestión de imágenes, parches de
seguridad
ECS/EKS
Gestión de
contenedores
EC2
Infraestructura
como servicio

24. ¡Gracias!
Javier Ramírez - @supercoco9
Developer Advocate - Amazon Web Services
Noviembre 3-4-5, 2020