Event Stream Processing with Kafka (Berlin Buzzwords 2012)

•

6 likes•2,025 views

Wooga

talk given at Berlin Buzzwords, 2012-06-05

Technology News & Politics

Event Stream
Processing
with Kafka
Tim Lossen • @tlossen • Wooga

high-throughput
persistent
publish-subscribe
messaging

github.com/wooga/kafka-rb
github.com/wooga/kafka-erlang
github.com/wooga/kafkaesque

Event Stream Processing with Kafka (Berlin Buzzwords 2012)

Apache Kafka sits at the core of the modern scalable event driven architecture. It’s no longer used only as logging infrastructure, but as a core component in thousands of companies around the world. It has the unique capability to provide low-latency, fault-tolerant pipeline at scale that is very important for today’s world of big data. During this talk we’ll see what makes Apache Kafka perfect for the job. We’ll explore how to optimize it for throughput or for durability. And we’ll also go over the messaging semantics it provides. Last but not least, we’ll see how Apache Kafka can help us solve some everyday problems that we face when we build large scale systems in an elegant way.

Openzipkin conf: Zipkin at Yelp

Prateek Agarwal

Evolving Operations for BBC iPlayer

Amazon Web Services

Summit openshift-on-openstack

Pippo620677

Class.aspera.chapter.1.intro

Ross Tang

What is (not) Pharo 8?

FAST

This talk wants to sum up the experience of designing, deploying and maintaining an Erlang application targeting the cloud and precisely AWS as hosting infrastructure. As the application now serves a significantly large user base with a sustained throughput of thousands of games actions per second we're able to analyse retrospectively our engineering and architectural choices and see how Erlang fits in the cloud environment also comparing it to previous experiences of clouds deployments of other platforms. We'll discuss properties of Erlang as a language and OTP as a framework and how we used them to design a system that is a good cloud citizen. We'll also discuss topics that are still open for a solution.

Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Wooga

JRubyConf2013_Tim Lossen_All your coreWooga

Getting the Most our of your Tools_FrontEnd DevConf2013_MinskWooga

Designing for Scale

Wooga

At wooga we build backends for games that have millions of daily users. In the gaming business we have a write heavy environment, with a high frequency of requests, traffic bursts and distribution across many nodes. These are all problems we need to solve and keep in mind in order to write a system that stands the chance of supporting the required load. How do you meet the challenge of writing a brand new system with performance in mind? Where should the line between necessary efficiency and premature optimization be drawn? How do you measure performances? How do we generate synthetic load that reflects real usage patterns? How do you know you have enough capacity? How do you combine all the above with safely introducing changes and new features working in a two people team? We had to answer to all the above questions and we want to share the solutions we found and the problems that we consider still open.

When Devs Do Ops

Wooga

Stateful Application Server_JRubyConf13_Lukas Rieder

Wooga

After more than one year of development, Wooga is heading for the global launch of its game "Kingsbridge"! This is the first game at Wooga with a backend written in JRuby! The talk includes an introduction to the problems that were solved by choosing a stateful applicaton server. I will explain constraints, benefits and obvious differences to traditional database backed application servers. Safely sharing state in a concurrent environment using JRuby Using Java concurrency utils in JRuby Sample problems solved, backed up with code Practical tips for capacity planning

Games for the Masses: Scaling Rails to the Extreme

Wooga

Metrics. Driven. Design. (Developer Conference Hamburg 2012)

Wooga

Did you ever work on a feature where you had multiple options on how to display it to the user but you did not know which one would work best? Why speculate about what would work best if you can actually measure it? Welcome to the world of a/b testing: Implement multiple variations of the same thing, and let different users experience different variations. Measure which variant works best and continue with that one. At Wooga we extensively use a/b testing to polish features to perfection. This helped us becoming one of the biggest providers of social games on Facebook with about 40 million players every month. We would like to show you what a/b testing is good for, how you do it and what pitfalls you need to avoid.

How to scale a company - game teams at Wooga

Wooga

2013 04-29-evolution of backendWooga

More than syntax

Wooga

You are not alone - Scaling multiplayer games

Wooga

At Wooga we are creating the next generation of social games. To be truly social, a game needs to offer real interaction between players in different forms. Taking one of our upcoming games as an example we will present the state synchronization used in client and server, how we do real time communication without websockets and how we are making a backend scalable enough to support millions of users every day. We will also share secrets about our prototypes for first person and real time strategy games. The Erlang process model with evented IO, asynchronous messaging, links and isolation fits very well to real time multiplayer games. Come to our talk to see how it's used to build a very successful business.

Stateful_Application_Server_RuPy 2012_Brno

Wooga

The talk is about a stateful application server built on top of JRuby/JVM. We at Wooga have built and evaluated such an application server, and now I want to share the learnings and obstacles that came up during development. Scaling up: We aim to fully utilize our available hardware. One box has 32 Cores and more than 32GB RAM available. How to saturate such beasts? Well, since JRuby supports real Threads we can share state (uh oh!) and saturate cores. Scaling out: What are the options for scaling a stateful application server? How to shard state across many servers safely? I aim for a talk that gives useful tips and shows code. Basically a ratio between 30% background information, 70% showcasing. The talk will cover the following topics: Safely sharing state in a concurrent environment using JRuby Using Java concurrency utils in JRuby Practical tips for tuning JRuby/JVM for maximum throughput Practical tips for evaluating performance tunings Bonus: Timetraveling

Painful success - lessons learned while scaling up

Wooga

Full of hope we started developing our biggest games so far in 2009. In 2010 we released it and gradually scaled up to 1 million users a day. But we were forced to rewrite our whole persistence layer and migrate most of our data from MySQL to Redis to make it work. In 2011 we even got 2 million daily users, but we also had to operate 200 servers which was painful - especially when the whole data center went down. So in 2012 we wanted to make everything better and started large refactoring projects - and made everything worse. But in 2013 everything will run smoothly and painless - promised! At least we hope so.

Continuous Integration for iOS (iOS User Group Berlin)

Wooga

Wooga: Internationality meets Agility @Zutaten 2013Wooga

NoSQL Games

Wooga

Story of Warlords: Bringing a turn-based strategy game to mobile

Wooga

About three years ago we set out on a mission: We wanted to bring a round-based strategy game, a genre that most gamers so far mainly enjoyed on PCs, to mobile. Today that mission is nearly completed with Warlords having been soft launched in various countries and showing very promising data. This may all sound easy enough, but the truth is that it was a hell of a ride for the team and there were times when we weren´t even sure if the project would continue. Along the way, we´ve learned many valuable things which we believe in the end made the difference and led to the numbers we now see. Wilhelm, Head of Studio and Product Lead for Warlords, will share these learnings in his talk answering the following questions: What set screws did we touch to make the session structure, pacing and content consumption work on mobile? Why different meta game systems are crucial? And finally how sticking to your game vision from start to end can make you survive such a ride.

Instagram Celebrities: are they the new cats? - Targetsummit Berlin 2015

Wooga

Viewers also liked

Riak & Wooga_Geeek2Geeek Meetup2014 BerlinWooga

NoSQL Games_NoSQL Roadshow Berlin

Wooga

Erlang as a Cloud Citizen

Wooga

Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

Wooga

JRubyConf2013_Tim Lossen_All your coreWooga

Getting the Most our of your Tools_FrontEnd DevConf2013_MinskWooga

Designing for Scale

Wooga

When Devs Do Ops

Wooga

Stateful Application Server_JRubyConf13_Lukas Rieder

Wooga

Games for the Masses: Scaling Rails to the Extreme

Wooga

Metrics. Driven. Design. (Developer Conference Hamburg 2012)

Wooga

How to scale a company - game teams at Wooga

Wooga

2013 04-29-evolution of backendWooga

More than syntax

Wooga

You are not alone - Scaling multiplayer games

Wooga

Stateful_Application_Server_RuPy 2012_Brno

Wooga

Painful success - lessons learned while scaling up

Wooga

Continuous Integration for iOS (iOS User Group Berlin)

Wooga

Wooga: Internationality meets Agility @Zutaten 2013Wooga

NoSQL Games

Wooga

Viewers also liked (20)

Riak & Wooga_Geeek2Geeek Meetup2014 Berlin

NoSQL Games_NoSQL Roadshow Berlin

Erlang as a Cloud Citizen

Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

JRubyConf2013_Tim Lossen_All your core

Getting the Most our of your Tools_FrontEnd DevConf2013_Minsk

Designing for Scale

When Devs Do Ops

Stateful Application Server_JRubyConf13_Lukas Rieder

Games for the Masses: Scaling Rails to the Extreme

Metrics. Driven. Design. (Developer Conference Hamburg 2012)

How to scale a company - game teams at Wooga

2013 04-29-evolution of backend

More than syntax

You are not alone - Scaling multiplayer games

Stateful_Application_Server_RuPy 2012_Brno

Painful success - lessons learned while scaling up

Continuous Integration for iOS (iOS User Group Berlin)

Wooga: Internationality meets Agility @Zutaten 2013

NoSQL Games

More from Wooga

Story of Warlords: Bringing a turn-based strategy game to mobile

Wooga

Instagram Celebrities: are they the new cats? - Targetsummit Berlin 2015

Wooga

In it for the long haul - How Wooga boosts long-term retention

Wooga

GDC San Francisco 2015 Presentation by Sebastian Nussbaum and Adam Telfer Today's mobile games market is tough. With thousands of new titles being available to download every week and CPIs rising high, new game concepts need to stand out. But even more importantly, they need to stay relevant for a long time. Long-term retention is the key for having a realistic shot at the top-grossing ranks of the app stores. In the first part of this talk, attendees will learn how mobile game developer Wooga evaluates new IP to ensure a long-lasting gameplay experience. The second half will show one example of how storytelling and episodic content can drive long-term retention. Sebastian will share insights and learnings from Wooga's projects in episodic content production and how to best set up your team for the ride.

Leveling up in localization! - Susan Alma & Dario Quondamstefano

Wooga

Evoloution of Ideas

Wooga

Entitas System Architecture with Unity - Maxim Zaks and Simon Schmid

Wooga

Saying No to the CEO: A Deep Look at Independent Teams - Adam Telfer

Wooga

GDC Europe 2015 Truly independent teams within a game studio are an interesting idea. Wooga, Supercell and others have promoted having this type of structure. Each team having full ownership over decisions in their game. Teams saying no to the rest of the company when they believe in a product. This all sounds great on paper, but what happens in practice? Adam Telfer goes into details of the ups and downs of working in an independent culture. How Wooga has adapted its processes through the years. A culture where you have full control, but also full responsibility.

Innovation dank DevOps (DevOpsCon Berlin 2015)

Wooga

“You build it, you run it!” - Wenn Du als Entwickler weisst, dass Du Deine Software selbst betreiben musst, was bist bereit zu tun, um den späteren Betrieb zu vereinfach? Bei Wooga haben Dutzende von Teams ihre eigene Antwort auf die Frage gesucht und dabei von den Erfahrungen der anderen Teams gelernt. Herausgekommen ist ein großes Experimentierfeld beim Betrieb von Web Services - und eine technologische Innovation, die uns innerhalb weniger Iterationen von einem simplen LAMP-Stack zu lastabhängig skalierenden stateful Servern auf Basis von Erlang oder Akka gebracht hat.

Big Fish, small pond - strategies for surviving in a maturing market - Ed Biden

Wooga

Quo Vadis Conference Berlin 2015 Mobile games have never been more competitive. Production values and marketing costs are ever increasing and the top grossing charts look strangely similar to the year before. So how can developers maximize their chances of success in such a tough market? Is it better to clone a hit or innovative wildly? Should they focus on one genre, or dabble in many? Ed draws on extensive market analysis and experience developing games to explain why selective innovation is the key to success.

Review mining aps2014 berlin

Wooga

Staying in the Game: Game localization practices for the mobile marketWooga

Startup Weekend_Makers and Games_Philipp StelzerWooga

DevOps goes Mobile (daho.am)

Wooga

We've learned a lot through doing DevOps: Every commit is automatically integrated, tested, and deployed to a staging environment. And then it only takes one push of a button and the release goes live... Unfortunately, it's not as simple anymore when operating mobile applications: How can you quickly update your mobile software when the app store provider wants to test your software first for a few days? How can you update your configuration when your app can run offline? And how do you track down errors when the data is distributed to millions of mobile clients? Those were just some of the challenges we encountered during the operation of mobile games with millions of daily users. In this talk we will talk about the solutions we have found to address them.

DevOps goes Mobile - Jax 2014 - Jesper Richter-Reichhelm

Wooga

DevOps hat uns viel gelehrt: Jedes Commit wird automatisch integriert, getestet und in eine Stagingumgebung installiert. Und einen Knopfdruck später geht das Release dann live. Aber leider funktioniert das und vieles mehr, was uns in Zeiten von DevOps als Normal erscheint, nicht bei mobilen Applikationen. Wie kann ich also schnell meine mobile Software anpassen, wenn der Betreiber des App Stores erst tagelang testen will? Wie kann ich Konfiguration anpassen, wenn die App auch offline laufen soll? Und wie funktioniert Fehlersuche, wenn die Daten auf Millionen von mobilen Clients verteilt sind? Beim Betrieb von mobilen Spielen mit Millionen täglicher Nutzer standen wir genau diesen Fragen gegenüber. Der Vortrag wird darlegen, welche Antworten wir darauf gefunden haben.

CodeFest 2014_Mobile Game DevelopmentWooga

Jelly Splash: Puzzling your way to the top of the App Stores - GDC 2014

Wooga

The match 3 puzzle genre is almost as old as it gets. Scour the App Store and you'll find hundreds of different varieties out there. Very few of these succeed however, and even less manage to hit the number one spot on the U.S. Apple App Store top download chart. Wooga's Jelly Splash managed to do just that, and in this session Florian Steinhoff, the creator of Jelly Splash, will give a detailed account on how his team managed that and what he learned throughout the development process.

How to hire the best people for your startup-Gitta Blat-Head of PeopleWooga

Two Ann(e)s and one Julia_Wooga Lady Power from Berlin_SGA2014

Wooga

Three people, three talks: First, Anne explains how Wooga makes it in the hit driven mobile F2P world and frequently creates hit games. Then, Ann and Julia share their insights into the company from an intern’s perspective. Julia is currently working with a team in the prototyping phase using C#. Ann develops Objective-C for one of Wooga's live games. Both of them are students of Chalmers University of Technology, Gothenburg. Anne is in charge of University Relations at Wooga. Tags: Equality, production, programming, career.

Pocket Gamer Connects 2014_The Experience of Entering the Korean MarketWooga

How to stand out in a hit driven business - Game Connection Paris 2013 - SebK...

Wooga

The mobile gaming market is huge with roughly 4,000 new iOS games per month and more than 5bn USD revenue per year. But that market also has a huge number of players who want a piece of the pie making the mobile gaming industry a very hit driven business. In his talk, Sebastian Kriese Corporate Partnerships Manager at Wooga, will give some insights about what the Berlin based social and mobile game developer considers a hit game and how they manage to create at least 2 of these hits per year.

More from Wooga (20)

Story of Warlords: Bringing a turn-based strategy game to mobile

Instagram Celebrities: are they the new cats? - Targetsummit Berlin 2015

In it for the long haul - How Wooga boosts long-term retention

Leveling up in localization! - Susan Alma & Dario Quondamstefano

Evoloution of Ideas

Entitas System Architecture with Unity - Maxim Zaks and Simon Schmid

Saying No to the CEO: A Deep Look at Independent Teams - Adam Telfer

Innovation dank DevOps (DevOpsCon Berlin 2015)

Big Fish, small pond - strategies for surviving in a maturing market - Ed Biden

Review mining aps2014 berlin

Staying in the Game: Game localization practices for the mobile market

Startup Weekend_Makers and Games_Philipp Stelzer

DevOps goes Mobile (daho.am)

DevOps goes Mobile - Jax 2014 - Jesper Richter-Reichhelm

CodeFest 2014_Mobile Game Development

Jelly Splash: Puzzling your way to the top of the App Stores - GDC 2014

How to hire the best people for your startup-Gitta Blat-Head of People

Two Ann(e)s and one Julia_Wooga Lady Power from Berlin_SGA2014

Pocket Gamer Connects 2014_The Experience of Entering the Korean Market

How to stand out in a hit driven business - Game Connection Paris 2013 - SebK...

Recently uploaded

Epistemic Interaction - tuning interfaces to provide information for AI support

Alan Dix

Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024 https://alandix.com/academic/papers/synergy2024-epistemic/ As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.

FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf

FIDO Alliance

Mission to Decommission: Importance of Decommissioning Products to Increase E...

Product School

The Future of Platform Engineering

Jemma Hussein Allen

The Art of the Pitch: WordPress Relationships and Sales

Laura Byrne

Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes? All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.

JMeter webinar - integration with InfluxDB and Grafana

RTTS

Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application. In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics. Length: 30 minutes Session Overview ------------------------------------------- During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana: - What out-of-the-box solutions are available for real-time monitoring JMeter tests? - What are the benefits of integrating InfluxDB and Grafana into the load testing stack? - Which features are provided by Grafana? - Demonstration of InfluxDB and Grafana using a practice web application To view the webinar recording, go to: https://www.rttsweb.com/jmeter-integration-webinar

Knowledge engineering: from people to machines and back

Elena Simperl

Neuro-symbolic is not enough, we need neuro-*semantic*

Frank van Harmelen

Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”. All of this illustrated with link prediction over knowledge graphs, but the argument is general.

How world-class product teams are winning in the AI era by CEO and Founder, P...

Product School

To Graph or Not to Graph Knowledge Graph Architectures and LLMs

Paul Groth

Accelerate your Kubernetes clusters with Varnish Caching

Thijs Feryn

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

DanBrown980551

Do you want to learn how to model and simulate an electrical network from scratch in under an hour? Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)! During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook. PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides: - A fully editable and extendable library for grid component modelling; - Visualization tools to display your network; - Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses; The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well. What you will learn during the webinar: - For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills; - For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.

Bits & Pixels using AI for Good.........

Alison B. Lowndes

PHP Frameworks: I want to break free (IPC Berlin 2024)

Ralf Eggert

In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development. This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.

GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...

James Anderson

Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management. The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM). Speakers: Bob Boule Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle. Gopinath Rebala Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

FIDO Alliance

GraphRAG is All You need? LLM & Knowledge Graph

Guy Korland

Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs. 1. Unifying Large Language Models and Knowledge Graphs: A Roadmap. https://arxiv.org/abs/2306.08302 2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs: https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/

FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf

FIDO Alliance

Assuring Contact Center Experiences for Your Customers With ThousandEyes

ThousandEyes

From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...

Product School