Yle, Finland's national public broadcasting company, uses AWS services like Kinesis, Redshift, Lambda, and S3 to power its data cloud. This collects data on user behavior, content usage, and other metrics from various sources to gain insights for content development, service optimization, recommendations, and strategic decision making. Yle processes over 100 million events daily and stores 90GB of data in Redshift for analytics using a mix of Firehose, Lambda, and EMR. It aims to improve the analytics pipeline to enable faster queries and more flexible processing.

1. AWS Big Data in Everyday Use at Yle
Saku Vaittinen, Yle / Jukka Dahlbom, Webscale
9.2.2017

2. Yle Data Cloud
• Helps to create better user experience
• Content development
• Service optimization
• Recommendation
• Marketing automation
• Purpose to create broad and real-time data for strategic
decisions and actions
• Reachability, demographics

5. Yle Data Cloud
Data => Information => Knowledge
● Daily predictions, how they match with reality,
accuracy over time
● Identify control set of users that can be used as a
reference point
● Combine Internet metrics with analog TV
measurements

6. Yle Data Cloud
● What we measure?
○ Page hits
○ Heartbeats
○ Social media
○ Articles read, time spent with media, date / time /
location
■ AMR (Average Minute Rating)
○ Genre
○ Age groups

8. Content
User
information
Behavior
Data Cloud Relational Data Hub
Sources Usage
Data hub
Raw data PublishingStructured
(Data vault)
Web events
Dashboards
Recommen-
dations
Strategy
Marketing
automation
Panel

9. S3
Yle Analytics Pipeline - Current situation
Elastic
Beanstalk
Kinesis Streams
Analytics
Collector
Web events
~100
mill./day
Kinesis Firehose
RedShift
Lambda S3 EMR
Daily
archiving
Areena
Recommendation
Delay: 10-15 minutes
Small batch sizes
Compression and
larger file sizes.
~90 Gb per day
CloudWatch
Dashboards
Small sized files
Delay: ~minute

10. Managed and/or serverless processing?
● Firehose used for data ingestion to Redshift
● Inflexible for faster analytics needs (< 2 minutes)
● Kinesis + Lambda consumers for both S3 archiving
and fast lane analysis (TBD).

11. DevOps in analytics pipeline
● Baseline support from dedicated operations team
● Terraform for infrastructure management
● Most other Yle services run as Dockerized APIs in
ECS cluster.

12. Redshift consumers
● Data mart (Postgre) for web-accessed precomputed
results
● Scheduled lambdas for very light queries
● Lambda-driven task containers for long but memory
light queries
● Lambda-started EC2 instances for memory intensive
computing. (Recommendations, user classifications)
● Data scientists running exploratory queries

13. Redshift performance
● Default user group is limited to 5 concurrent queries
● Set up WLM queues for different workloads, split by
usage.
● Isolate data scientists into separate WLM queue that
doesn’t block scheduled activity.

14. Lambda for batch queries?
● Fast, serverless, stateless, cheap, reactive.
● Limited by 300s max timeout
● Unreliable in high load situations.

15. Lambda-driven task containers for batch
● Lambda allows reactive and scheduled running of
tasks.
● Task containers not limited by execution timeouts.
● Logging and monitoring support for ECS containers
is already there (ELK).

16. Why not use AWS Batch?
● AWS Batch wasn’t available when batch containers
were first needed - published in re:Invent Dec 2016
● Not yet supported in Terraform v0.8.5.
● Once support is there, switch to AWS Batch from
homebrew resources.

17. Monitoring?
● Cloudwatch Alerts for component red/green health
● Cloudwatch Dashboards for overall graph view
● Kibana and Cloudwatch Logs (with custom scripting)
for log management.

18. Monitoring

19. Monitoring