Lieven Vandegaer from MEDIAGENIX - Orchestrating a video-on-demand pipeline with WHATS'ON, Camunda BPM and Apache NiFi - Camunda Day Amsterdam 2018

•Download as PPTX, PDF•

2 likes•1,398 views

WHATS'ON is a software solution of MEDIAGENIX that manages the lifecycle of content within media companies. For a new project with a telecom operator we needed to handle incoming metadata, video files and images and make this content available on several VOD platforms. These workflows can become very complex and require intense communication with various systems (transcoders, encryption software, a VOD backend system...). In this presentation, we will show how we are using our in-house product WHATS'ON (written in Smalltalk) in combination with Camunda BPM and Apache NiFi, using external tasks and the Camunda BPM REST API.

ORCHESTRATING A VIDEO-ON-DEMAND
PIPELINE WITH WHATS'ON, CAMUNDA BPM
AND APACHE NIFI
CAMUNDA DAY – JULY 3 2018
LIEVEN VANDEGAER

NEW PROJECT: VOD DELIVERY
Content Providers Platform owners
- Transcoding
- Encryption
- Publication

CAMUNDA
• On site workshop (POC) with Benjamin Hoffmann middle of
December last year
• Results:
• Foundation of the workflows
• External tasks
• REST API
• Open source

WHATS’ON WORKERS
• Written in Smalltalk
• Interfaces with
• WHATS’ON: e.g. update statuses
• Encryption software
• Transcoding software
• VOD backend system

APACHE NIFI WORKERS
• Tool to model and execute data flows
• All file handling
• File transfers
• FTP transfers
• Deletion of files
• Renaming of files
• REST API
• External tasks

LEARNINGS
• 2 Camunda instances
• 3 NiFi servers in a cluster
• 2 instances of each WHATS’ON worker => by using the
external task pattern they can run active/active
• Redundant Oracle server
Fully
redundant
• Replacement/upgrade of workers can be done without
stopping the entire system
• Changes to the workflow diagram
• Not restricted to a fixed number of video files/images
• Combination of Camunda and NiFi works extremely
well
Flexible
• The amount of workers can be extendedScalable

LEARNINGS
• Logging
• Monitoring
• Communication (security)
Micro services
• Newer version deployed easily
• Testing
• Debugging
Workflow
diagram
• Customer support
• Involve new people/new
projects
Organisational
impact

QUESTIONS ?
WE’RE HIRING: WWW.MEDIAGENIX.TV/JOBS

Presenter: Tim Berglund, Senior Director of Developer Experience, Confluent It has become a truism in the past decade that building systems at scale, using non-relational databases, requires giving up on the transactional guarantees afforded by the relational databases of yore. ACID transactional semantics are fine, but we all know you can’t have them all in a distributed system. Or can we? In this talk, I will argue that by designing our systems around a distributed log like Apache Kafka®, we can in fact achieve ACID semantics at scale. We can ensure that distributed write operations can be applied atomically, consistently, in isolation between services, and of course with durability. What seems to be a counterintuitive conclusion ends up being straightforwardly achievable using existing technologies, as an elusive set of properties becomes relatively easy to achieve with the right architectural paradigm underlying the application.

Blue-green deploys with Pulsar & Envoy in an event-driven microservice ecosys...

StreamNative

In this talk, learn how Toast leverages our Envoy control-plane to manage blue-green deploys of Pulsar consumers, and how this has helped drive adoption across the engineering organization. Dive into the history of Pulsar at Toast, starting from its introduction in 2019 to provide event-driven architecture across a rapidly scaling restaurant software platform. We will detail some of the hurdles that we encountered gaining buy-in across a diverse set of teams, and dive deep into how we enforce best practices and integrate with our service control plane.

Jenkins & IaC

HungWei Chiu

Monoliths and Microservices

Bozhidar Bozhanov

Introduction To Microservices

Lalit Kale

This presentation is conducted on 14th Sept in Limerick DotNet User Group. (https://www.meetup.com/preview/Limerick-DotNet/events/xskpdnywmbsb) SlideShare Url: https://www.slideshare.net/lalitkale/introduction-to-microservices-80583928 In this presentation, new architectural style - Microservices and it's emergence is discussed. We will also briefly touch base on what are not microservices, Conway's law and organization design, Principles of microservices and service discovery mechanism and why it is necessary for microservices implementation. About Speaker: Lalit is a senior developer, software architect and consultant with more than 12 yrsof .NET experience. He loves to work with C# .NET and Azure platform services like App Services, Virtual Machines, Cortana, and Container Services. He is also the author of 'Building Microservices with .NET Core' (https://www.packtpub.com/web-development/building-microservices-net-core) book. To know more and connect with Lalit, you can visit his LinkedIn profile below. https://www.linkedin.com/in/lalitkale/ This presentation will be useful for software architects/Managers, senior developers. Do share your feedback in comments.

DevOps: A Value Proposition

Nicole Forsgren

Mainframe DevOps: A Zowe CLI-enabled Roadmap

DevOps.com

The Zowe open source framework, hosted by the Linux Foundation's Open Mainframe Project, is often referred to as a Swiss Army knife for mainframe modernization, but where to begin? This session, which is based on findings from numerous Design Thinking workshops, will help DevOps champions and mainframe leaders jumpstart their modernization journeys. We’ll explore a few high-value use cases like plugging into enterprise CI/CD pipelines and incorporating off-platform tools like code quality. And by addressing practical considerations like Zowe installation, set-up and support, this session will equip attendees with the information they need to become mainframe DevOps mobilizers.

BizDevOps – Delivering Business Value Quickly at Scale

QASymphony

BIZDEVOPS – DELIVERING BUSINESS VALUE QUICKLY AT SCALE 65+% of surveyed organizations are currently on the path to switch to DevOps or have already implemented the process, and the benefits of a properly implemented DevOps program are clear – quicker time to customer value, better alignment between businesses and customers, and a better ability to respond to customer input. However, when it comes to DevOps adoption, many teams rush to focus on one specific issue within one area when they would actually benefit more from aligning business, development, testing, and operations up front. The five major problems in DevOps adoption include: Lack of Test Automation Coverage Lack of Visibility into Testing Maintaining Various Test Versions and Aligning Tests with Versions of Source Code Maintaining a Single Source of Truth in the Testing Process Understanding Where Business Value Currently is in the “BizDevOps” Pipeline After helping hundreds of customers in their DevOps journeys, these three industry experts will cover these major problems, as well as innovative strategies to overcome them: Bobby Smith – Director of R&D, QAS Labs Brandon Cipe – VP DevOps, cPrime Kevin Dunne – VP Business Development, QASymphony Tune in to learn more about the state of the industry, the direction that DevOps adoption is moving toward, and what we like to call “BizDevOps”. You won’t want to miss this session!

In this talk we’ll look at the relationship between three of the most disruptive software engineering paradigms: event sourcing, stream processing and serverless. We’ll debunk some of the myths around event sourcing. We’ll look at the inevitability of event-driven programming in the serverless space and we’ll see how stream processing links these two concepts together with a single ‘database for events’. As the story unfolds we’ll dive into some use cases, examine the practicalities of each approach-particularly the stateful elements-and finally extrapolate how their future relationship is likely to unfold. Key takeaways include: The different flavors of event sourcing and where their value lies. The difference between stream processing at application- and infrastructure-levels. The relationship between stream processors and serverless functions. The practical limits of storing data in Kafka and stream processors like KSQL."

Integration Microservices

Kasun Indrasiri

Kafka Summit 2018: Monitoring and Orchestration of Your Microservices Landsca...

Bernd Ruecker

Communication in a Microservice Architecture

Per Bernhardt

There are many different approaches to how you let your microservices communicate between one another. Be it asynchronous or synchronous, choreographed or orchestrated, eventual consistent or distributedly transactional, fault tolerant or just a mess! In this session I will provide an overview on different concepts of microservice communication and their pros & cons. On the way I'll try to throw in some anecdotes, success stories and failures I learned from so that you can hopefully take something home with you.

Introduction To Containers - Builders Day Israel

Amazon Web Services

Apache Kafka in Financial Services - Use Cases and Architectures

Kai Wähner

The Rise of Event Streaming in Financial Services - Use Cases, Architectures and Examples powered by Apache Kafka. The New FinServ Enterprise Reality: Every company is a software company. Innovate OR be Disrupted. Learn how Event Streaming with Apache Kafka and its ecosystem help... More details: https://www.kai-waehner.de/apache-kafka-financial-services-industry-banking-finserv-payment-fraud-middleware-messaging-transactions https://www.kai-waehner.de/blog/2020/04/15/apache-kafka-machine-learning-banking-finance-industry/ https://www.kai-waehner.de/blog/2020/04/24/mainframe-offloading-replacement-apache-kafka-connect-ibm-db2-mq-cdc-cobol/

PagerDuty: Best Practices for On Call Teams

Mandi Walls

Build CICD Pipeline for Container Presentation Slides

Amazon Web Services

At AWS re:Invent, we have launched support for blue/green deployments for services hosted using AWS Fargate and Amazon Elastic Container Service (Amazon ECS). Blue/green deployments help you minimize downtime during application updates. They allow you to launch a new version of your application alongside the old version and test the new version before you reroute traffic to it. You can also monitor the deployment process and, if there is an issue, quickly roll back. In this workshop, you will create a new service in AWS Fargate that uses AWS CodeDeploy to manage the deployments, testing, and traffic cutover for you.

Goto Berlin - Migrating to Microservices (Fast Delivery)

Adrian Cockcroft

Building and deploying microservices with event sourcing, CQRS and Docker (Be...

Chris Richardson

In this talk we share our experiences developing and deploying a microservices-based application. You will learn about the distributed data management challenges that arise in a microservices architecture. We will describe how we solved them using event sourcing to reliably publish events that drive eventually consistent workflows and pdate CQRS-based views. You will also learn how we build and deploy the application using a Jenkins-based deployment pipeline that creates Docker images that run on Amazon EC2. This talk was given at the Berlin Microxchg conference and the Munich microservices meetup.

Kubernetes

Meng-Ze Lee

Microservices at Spotify

Kevin Goldsmith

CI/CD Overview

An Nguyen

Apache Flink Training: System Overview

Flink Forward

Introduction to CICD

Knoldus Inc.

DDD SoCal: Decompose your monolith: Ten principles for refactoring a monolith...

Chris Richardson

Introduction to CI/CD

Steve Mactaggart

Docker Kubernetes Istio

Araf Karsh Hamid

OSMC 2021 | Handling 250K flows per second with OpenNMS: a case study

NETWAYS

What does it take to go from no flow support, to handling huge volumes of heterogeneous flow data in a 100% open-source monitoring stack, in a real-world environment? Expect a brief refresher on flows, an overview of the customer environment, and discussion of the engineering challenges faced. A medium dive follows into the movement of flow data from ingest to query and display, the solution architecture as it exists today, and lessons learned and their application to the project roadmap.

Data Stream Processing with Apache Flink

Fabian Hueske

What's hot

Event Driven Microservices with Spring Cloud Stream #jjug_ccc #ccc_ab3

Toshiaki Maki

DevOps Maturity Curve v5

Paul Peissner

Event Sourcing, Stream Processing and Serverless (Benjamin Stopford, Confluen...

confluent

Integration Microservices

Kasun Indrasiri

Kafka Summit 2018: Monitoring and Orchestration of Your Microservices Landsca...

Bernd Ruecker

Communication in a Microservice Architecture

Per Bernhardt

Introduction To Containers - Builders Day Israel

Amazon Web Services

Apache Kafka in Financial Services - Use Cases and Architectures

Kai Wähner

PagerDuty: Best Practices for On Call Teams

Mandi Walls

Build CICD Pipeline for Container Presentation Slides

Amazon Web Services

Goto Berlin - Migrating to Microservices (Fast Delivery)

Adrian Cockcroft

Building and deploying microservices with event sourcing, CQRS and Docker (Be...

Chris Richardson

Kubernetes

Meng-Ze Lee

Microservices at Spotify

Kevin Goldsmith

CI/CD Overview

An Nguyen

Apache Flink Training: System Overview

Flink Forward

Introduction to CICD

Knoldus Inc.

DDD SoCal: Decompose your monolith: Ten principles for refactoring a monolith...

Chris Richardson

Introduction to CI/CD

Steve Mactaggart

Docker Kubernetes Istio

Araf Karsh Hamid

What's hot (20)

Event Driven Microservices with Spring Cloud Stream #jjug_ccc #ccc_ab3

DevOps Maturity Curve v5

Event Sourcing, Stream Processing and Serverless (Benjamin Stopford, Confluen...

Integration Microservices

Kafka Summit 2018: Monitoring and Orchestration of Your Microservices Landsca...

Communication in a Microservice Architecture

Introduction To Containers - Builders Day Israel

Apache Kafka in Financial Services - Use Cases and Architectures

PagerDuty: Best Practices for On Call Teams

Build CICD Pipeline for Container Presentation Slides

Goto Berlin - Migrating to Microservices (Fast Delivery)

Building and deploying microservices with event sourcing, CQRS and Docker (Be...

Kubernetes

Microservices at Spotify

CI/CD Overview

Apache Flink Training: System Overview

Introduction to CICD

DDD SoCal: Decompose your monolith: Ten principles for refactoring a monolith...

Introduction to CI/CD

Docker Kubernetes Istio

Similar to Lieven Vandegaer from MEDIAGENIX - Orchestrating a video-on-demand pipeline with WHATS'ON, Camunda BPM and Apache NiFi - Camunda Day Amsterdam 2018

OSMC 2021 | Handling 250K flows per second with OpenNMS: a case study

NETWAYS

Data Stream Processing with Apache Flink

Fabian Hueske

Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...

confluent

(Bob Lehmann, Bayer) Kafka Summit SF 2018 You’ve built your streaming data platform. The early adopters are “all in” and have developed producers, consumers and stream processing apps for a number of use cases. A large percentage of the enterprise, however, has expressed interest but hasn’t made the leap. Why? In 2014, Bayer Crop Science (formerly Monsanto) adopted a cloud first strategy and started a multi-year transition to the cloud. A Kafka-based cross-datacenter DataHub was created to facilitate this migration and to drive the shift to real-time stream processing. The DataHub has seen strong enterprise adoption and supports a myriad of use cases. Data is ingested from a wide variety of sources and the data can move effortlessly between an on premise datacenter, AWS and Google Cloud. The DataHub has evolved continuously over time to meet the current and anticipated needs of our internal customers. The “cost of admission” for the platform has been lowered dramatically over time via our DataHub Portal and technologies such as Kafka Connect, Kubernetes and Presto. Most operations are now self-service, onboarding of new data sources is relatively painless and stream processing via KSQL and other technologies is being incorporated into the core DataHub platform. In this talk, Bob Lehmann will describe the origins and evolution of the Enterprise DataHub with an emphasis on steps that were taken to drive user adoption. Bob will also talk about integrations between the DataHub and other key data platforms at Bayer, lessons learned and the future direction for streaming data and stream processing at Bayer.

Stream Processing with Apache Flink

C4Media

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1VhSzmy. Robert Metzger provides an overview of the Apache Flink internals and its streaming-first philosophy, as well as the programming APIs. Filmed at qconlondon.com. Robert Metzger is a PMC member at the Apache Flink project and a cofounder and software engineer at data Artisans. He is the author of many Flink components including the Kafka and YARN connectors.

What's new in ASP.NET vNext

Gunnar Peipman

State of NuPIC

Numenta

Bay Area Apache Flink Meetup Community Update August 2015

Henry Saputra

Sutol 2016 - Automation is developer's friend

mpradny

AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017

Monal Daxini

Over 100 million subscribers from over 190 countries enjoy the Netflix service. This leads to over a trillion events, amounting to 3 PB, flowing through the Keystone infrastructure to help improve customer experience and glean business insights. The self-serve Keystone stream processing service processes these messages in near real-time with at-least once semantics in the cloud. This enables the users to focus on extracting insights, and not worry about building out scalable infrastructure. I’ll share the details about this platform, and our experience building it.

Building Stream Processing as a Service

Steven Wu

Serverless design with Fn project

Siva Rama Krishna Chunduru

The Fn project is a container-native Apache 2.0 licensed serverless platform that you can run anywhere – on any cloud or on-premise. It’s easy to use, supports every programming language, and is extensible and performant. This YourStory-Oracle Developer Meetup covers various design aspects of Serverless for polyglot programming, implementation of Saga pattern, etc. It also emphasizes on the monitoring aspect of Fn project using Prometheus and Grafana

Apache NiFi: A Drag and Drop Approach

Calculated Systems

This presentation explains how open-source Apache Nifi can be used to easily consume AWS Cloud Services. Featuring drag and drop interactions with many cloud capabilities, it enables teams to quickly start handling their big data on the cloud. Both small agile and large enterprise teams can benefit from this easy to learn, rapid to implement approach to data processing. For more information, go to www.calculatedsystems.com.

Pywps

Luis_de_Sousa

PyWPS-4.0.0

Jachym Cepicky

Tech Tutorial by Vikram Dham: Let's build MPLS router using SDN

nvirters

Synopsis We will start with MPLS 101 and then look into MPLS related OpenFlow actions. In the second half we will delve into RouteFlow architecture and extend it to enable Label Distribution Protocol (LDP) and MPLS routing. We will conclude with a mini-net based test bed switching traffic using MPLS labels instead of IP addresses. This will be a hands on workshop. VM Images for Virtual Box will be provided. Attendees are expected to bring their laptops loaded with Virtual Box. About Vikram Dham Vikram is the CTO and co-founder of Kamboi Technologies, LLC where he advises networking companies, switch vendors and early adopters on SDN technology and distributed software development. Also, he is the founder of Bay Area Network Virtualization (BANV) meet-up group, that brings together technologists in the SDN/NFV/NV domain for technical talks, workshops and creates a truly "open" platform for sharing knowledge. He has used SDN technologies for building software related to traffic engineering, security and routing. In the past, he was the Principal Engineer at Slingbox where he architected & built the distributed networking software for peer to peer connectivity of millions of end points. He holds MS degree in EE with a specialization in Computer Networks from Virginia Tech and has worked on research projects with companies like ECI Telecom, Raytheon and Avaya Research Labs.

ApacheCon 2021 - Apache NiFi Deep Dive 300

Timothy Spann

21-September-2021 - ApacheCon - Tuesday 17:10 UTC Apache NIFi Deep Dive 300 * https://github.com/tspannhw/EverythingApacheNiFi * https://github.com/tspannhw/FLiP-ApacheCon2021 * https://www.datainmotion.dev/2020/06/no-more-spaghetti-flows.html * https://github.com/tspannhw/FLiP-IoT * https://github.com/tspannhw/FLiP-Energy * https://github.com/tspannhw/FLiP-SOLR * https://github.com/tspannhw/FLiP-EdgeAI * https://github.com/tspannhw/FLiP-CloudQueries * https://github.com/tspannhw/FLiP-Jetson * https://www.linkedin.com/pulse/2021-schedule-tim-spann/ Tuesday 17:10 UTC Apache NIFi Deep Dive 300 Timothy Spann For Data Engineers who have flows already in production, I will dive deep into best practices, advanced use cases, performance optimizations, tips, tricks, edge cases, and interesting examples. This is a master class for those looking to learn quickly things I have picked up after years in the field with Apache NiFi in production. This will be interactive and I encourage questions and discussions. You will take away examples and tips in slides, github, and articles. This talk will cover: Load Balancing Parameters and Parameter Contexts Stateless vs Stateful NiFi Reporting Tasks NiFi CLI NiFi REST Interface DevOps Advanced Record Processing Schemas RetryFlowFile Lookup Services RecordPath Expression Language Advanced Error Handling Techniques Tim Spann is a Developer Advocate @ StreamNative where he works with Apache NiFi, Apache Pulsar, Apache Flink, Apache MXNet, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Principal Field Engineer at Cloudera, a senior solutions architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science.

Stream Processing @ Lyft

Jamie Grier

Sas 2015 event_driven

Sascha Möllering

Being Ready for Apache Kafka - Apache: Big Data Europe 2015

Michael Noll

These are the slides of my Kafka talk at Apache: Big Data Europe in Budapest, Hungary. Enjoy! --Michael Apache Kafka is a high-throughput distributed messaging system that has become a mission-critical infrastructure component for modern data platforms. Kafka is used across a wide range of industries by thousands of companies such as Twitter, Netflix, Cisco, PayPal, and many others. After a brief introduction to Kafka this talk will provide an update on the growth and status of the Kafka project community. Rest of the talk will focus on walking the audience through what's required to put Kafka in production. We’ll give an overview of the current ecosystem of Kafka, including: client libraries for creating your own apps; operational tools; peripheral components required for running Kafka in production and for integration with other systems like Hadoop. We will cover the upcoming project roadmap, which adds key features to make Kafka even more convenient to use and more robust in production.

London DevOps Meetup - PaaS as a platform for devops

Jeremy Brown

Similar to Lieven Vandegaer from MEDIAGENIX - Orchestrating a video-on-demand pipeline with WHATS'ON, Camunda BPM and Apache NiFi - Camunda Day Amsterdam 2018 (20)

OSMC 2021 | Handling 250K flows per second with OpenNMS: a case study

Data Stream Processing with Apache Flink

Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...

Stream Processing with Apache Flink

What's new in ASP.NET vNext

State of NuPIC

Bay Area Apache Flink Meetup Community Update August 2015

Sutol 2016 - Automation is developer's friend

AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017

Building Stream Processing as a Service

Serverless design with Fn project

Apache NiFi: A Drag and Drop Approach

Pywps

PyWPS-4.0.0

Tech Tutorial by Vikram Dham: Let's build MPLS router using SDN

ApacheCon 2021 - Apache NiFi Deep Dive 300

Stream Processing @ Lyft

Sas 2015 event_driven

Being Ready for Apache Kafka - Apache: Big Data Europe 2015

London DevOps Meetup - PaaS as a platform for devops

More from camunda services GmbH

Using Camunda on Kubernetes through Operators

Lieven Vandegaer from MEDIAGENIX - Orchestrating a video-on-demand pipeline with WHATS'ON, Camunda BPM and Apache NiFi - Camunda Day Amsterdam 2018

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Lieven Vandegaer from MEDIAGENIX - Orchestrating a video-on-demand pipeline with WHATS'ON, Camunda BPM and Apache NiFi - Camunda Day Amsterdam 2018

Similar to Lieven Vandegaer from MEDIAGENIX - Orchestrating a video-on-demand pipeline with WHATS'ON, Camunda BPM and Apache NiFi - Camunda Day Amsterdam 2018 (20)

More from camunda services GmbH

More from camunda services GmbH (20)

Recently uploaded

Recently uploaded (20)

Lieven Vandegaer from MEDIAGENIX - Orchestrating a video-on-demand pipeline with WHATS'ON, Camunda BPM and Apache NiFi - Camunda Day Amsterdam 2018

Editor's Notes