Kappa Architecture, IoT of the cars - LibreCon 2016

•

1 like•906 views

This document discusses Kappa Architecture, which is a data processing architecture using Apache Kafka as the centralized data backbone. It summarizes Jay Kreps' original definition of Kappa Architecture and describes how it provides a more flexible way to do real-time and batch processing compared to the traditional Lambda Architecture. The document also provides an example of how a company implemented Kappa Architecture using Kafka and Spark Streaming to monitor car telemetry data from many vehicles and support flexible real-time queries over that data.

Internet

Kappa Architecture:
IoT of the cars
Juantomás García

Who am I
• Data Solutions Manager at Open Sistemas
• Co-author of the ﬁrst spanish book about free
software “La pastilla roja”
• Former president of Hispalinux (Spanish Linux User
Group).
• Co-organizer of Machine Learning Spain Meetup

What’s Kappa Architecture
July 2, 2014 Jay Kreps coined the term
Kappa Architecture in an article for
O’reilly Radar

Who is Jay Kreps
Jay has been involved in lots of projects:
Author of the essay:
The Log: What every software engineer
should know about real-time data's
unifying abstraction (12/16/2013)
https://engineering.linkedin.com/distributed-systems/log-what-every-software-
engineer-should-know-about-real-time-datas-unifying

Who is Jay Kreps
Author of the book: I ♥ Logs

Who is Jay Kreps
Involved with projects as:
Apache Kafka
Apache Samza
Voldemort
Azkaban
Ex-Linkedin
Now co-founder and CEO of Confluent

Tools we use to implement
Kappa Architecture

Tools we use to implement
Kappa Architecture
• Our favorite kafka features.
• Easy to use
• Robust & Resilient
• Kafka connectors
• Kafka streams
• Compact logs
• Very fast writes and reads

Some favorite Spark features
• If you have an schema spark SQL is
perfect.
• Spark streaming work very fine with spark
and almost all streaming sources.
• Structured queries will be a huge advance.
• We love scala and scala is the spirit of
spark.

Some favorite Spark features
• We love code like this:

A real case
• One of our clients want to monitor all the
cars informations via OBD II
• OBD II is a car interface with the car
electronics.
• Client develop an app to read all the car
information throw a ODB II with bluetooth

First Problems
• We need to scale the rest interfaces too
many requests.
• MySQL don’t scale
• Client wants to make realtime expensive
queries.

Takeaways
• Kappa architecture is not a silver bullet but
helps in a lot of solutions.
• Kafka + spark streaming is our favorite
tools
• There a lots of improvements:
• OLAP like Apache Druid
• Graph databases like neo4j
• Kafka streams and compacts logs

¿Questions?
• My contact info:
• Juantomás García
• juantomas@opensistemas.com
• @juantomas

Analyzing big data is a challenge, requiring lots of processing power and storage. Cloud Computing is an ideal platform to tackle this problem. HD Insight on Microsoft Azure deploys Hadoop and other open source big data tools to the cloud, making it easier to take advantage of the high scalability of this platform. In this session, you will learn what tools are available in HD Insight and how to use them to store, process, and analyze large amounts of data.

Solving the Industry 4.0 challenges on the logistics domain using Apache Meso...

Big Data Spain

Gain Deep Visibility into APIs and Integrations with Anypoint Monitoring

InfluxData

From “All-at-Once, Once-a-Day” to “A-Little-Each-Time, All-the-Time” with Ema...

Databricks

At OLX we produce daily about 50 millions messages to be delivered to our 300+ millions users across the globe; via email, sms or push. The majority of these notifications relies on the processing of the billions of events generated by our web and mobile platforms to understand the users behaviour and to craft relevant messages designed to influence the customer journey positively. The initial approach was to process events in bulk, in a daily batch data-warehouse-y fashion. This worked well for a while, but requirements changed: to be more relevant and effective, we had to react faster to user actions and once-a-day processes didn’t work anymore. On top of it, data regulations, reliability and infrastructure costs urged us to step-up our game and improve the way we engineer data. In this presentation I will discuss the approach, challenges and learnings of migrating our notification platform from a monolithic, batch system based on AWS Redshift, SQL and ETL pipelines to a micro-service, real-time system developed with Apache Spark and Python.

Quby, an Amsterdam-based technology company, offers solutions to empower homeowners to stay in control of their electricity, gas and water usage. Using Europe’s largest energy dataset, consisting of petabytes of IoT data, the company has developed AI powered products that are used by hundreds of thousands of users on a daily basis. Delta Lake ensures the quality of incoming records though schema enforcement and evolution. But it is the Data Engineers role to check whether the expected data is ingested in to the Delta Lake at the right time with expected metrics so that downstream processes will function their duties. Re-training models and serving on the fly might go wrong unless we put the right monitoring infrastructure too.

MongoDB .local London 2019: The Tech Behind Connected Car

MongoDB

Connected Car is driving one of the largest waves of innovation in the automotive industry in decades, with Jaguar Land Rover right at the forefront. Blending mobile and web apps with IoT, edge computing and Artificial Intelligence, Connected Car presents unique technology challenges due to ever-changing connectivity conditions, limited bandwidth and power consumption constraints. These are challenges we can meet, only by using the latest generation of technologies; allowing our developers to experiment and build faster for our customers. In this talk, I'll discuss the technology stack we use to deliver new services to our customers, connecting the car to services, and the role MongoDB plays for us to deliver more, faster.

Scylla Summit 2022: Scalable and Sustainable Supply Chains with DLT and ScyllaDB

ScyllaDB

RealTime Recommendations @Netflix - Spark

Nitin S

[WSO2Con EU 2018] Tooling for Observability

WSO2

Ingesting IoT data in Food Processing

confluent

At BAADER, we design and engineer innovative and holistic solutions that ensure intelligent, safe, efficient and sustainable food processing in all phases, from the handling of live and raw protein materials to the finished food products. As a key player in the food value chain, we aim to take further significant steps toward greater efficiency, traceability, transparency, profitability, and sustainability through new digital solutions. During our digital transformation we are working on two ends: At one hand there are many brownfield factories unprepared for the digital journey and the other hand we have powerful greenfield technologies like Apache Kafka. Now, we have to bring two mindsets together – robust food processing machinery and highly scalable software technologies. In this talk, we will present how we successfully started to ingest various kinds of IoT data into our Kafka cluster - spotlighted from both ends.

OVH Analytics Data Compute - Apache Spark Cluster as a Service

OVHcloud

Ovh analytics data compute with apache spark as a service meetup ovh bordeaux

Mojtaba Imani

90% of the data in the world today has been created in the last two years. The world will be creating 163 zettabytes of data a year by 2025. So how do we want to process this volume of data? Apache Spark is an open-source distributed general-purpose cluster computing framework that is trending today. But the problem is that how to create a computing cluster fast and efficient? Should I do all network configuration and cluster management myself? What should I do with my cluster if I don't need it anymore? Is my cluster secure? After discovering Apache Spark principles and use cases, you will discover OVH Analytics Data Compute. A fast, secure, and efficient Spark Cluster as a Service which is going to give answers to all these questions.

Avoid SPOF in Cloud-native Apps

Thang Chung

Building Notebook-based AI Pipelines with Elyra and Kubeflow

Databricks

A typical machine learning pipeline begins as a series of preprocessing steps followed by experimentation, optimization and model-tuning, and, finally deployment. Jupyter notebooks have become a hugely popular tool for data scientists and other machine learning practitioners to explore and experiment as part of this workflow, due to the flexibility and interactivity they provide. However, with notebooks it is often a challenge to move from the experimentation phase to creating a robust, modular and production-grade end-to-end AI pipeline.

SAIS2018 - Fact Store At Netflix Scale

Nitin S

Martin Moucka [Red Hat] | How Red Hat Uses gNMI, Telegraf and InfluxDB to Gai...

InfluxData

Red Hat is the provider of enterprise open source solutions. Its portfolio of products includes hybrid cloud infrastructure, middleware, cloud-native apps and automation solutions. Its internal network supports all lines of business — including 60+ sites. Discover how Red Hat uses InfluxDB and Flux for better real-time monitoring of their networks to improve performance and to understand utilization better.

How to Build Cloud-based Microservice Environments with Docker and VoltDB

VoltDB

In this webinar Rusty Ross, Solutions Architect, Full 360, Inc. and Peter Vescuso, CMO, VoltDB present a case study of the fast data infrastructure developed for one of the largest and most successful loyalty programs in the world. Using Docker, the solution leverages AWS Elastic Load Balancing to automatically distribute incoming application needs without manual intervention. The result was an integrated, highly scalable 10 million-person loyalty program that can turn data into deployed services quickly and easily; enabling them to respond rapidly to changing needs and insights. To view the webinar recording click here: http://learn.voltdb.com/WRFull360Docker.html

[WSO2Con EU 2018] OpenAPI Specification 3 - The Evolution of Swagger

WSO2

Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...

HostedbyConfluent

Are you looking for a cloud-based architecture that includes the best of breed streaming and database technologies? In this session you will learn how to setup and configure the Confluent Cloud with MongoDB Atlas. We'll start the journey learning about the basic connectivity between the two cloud services and end with a brief discovery of what you can do with data once it is in MongoDB Atlas. By the end of this session you will know how to securely setup and configure the MongoDB Atlas connectors in the Confluent Cloud in both a source and sink configuration.

The Expert Guide to Fast Data

VoltDB

Dr. Michael Stonebraker will share his "one-size-doesn’t-fit-all" perspective when it comes to picking the right tool for the job. He will explain the fast data stack, why traditional RDBMS’s fall short and how a modern in-memory SQL, ACID compliant, database with a scale-out architecture is the right choice for enabling fast data applications. Then John Hugg provides the “proof in the pudding” with a step-by-step review of his Unique Devices application, which performs real-time analytics on fast moving data. It's a representative implementation of the speed layer in the Lambda Architecture with the logic captured in just 30 lines of code.

Virtualizing Analytics with Apache Spark: Keynote by Arsalan Tavakoli

Spark Summit

In the race to invent multi-million dollar business opportunities with exclusive insights, data scientists and engineers are hampered by a multitude of challenges just to make one use case a reality – the need to ingest data from multiple sources, apply real-time analytics, build machine learning algorithms, and intermix different data processing models, all while navigating around their legacy data infrastructure that is just not up to the task. This need has created the demand for Virtual Analytics, where the complexities of disparate data and technology silos have been abstracted away, coupled with a powerful range of analytics and processing horsepower, all in one unified data platform. This talk describes how Databricks is powering this revolutionary new trend with Apache Spark.

apidays LIVE Paris 2021 - Stargate.io, An OSS Api Layer for your Cassandra by...

apidays

Mike Stonebraker on Designing An Architecture For Real-time Event Processing

VoltDB

In this webinar, Mike Stonebraker explains the tradeoffs of performance, scale, and required programming “heroics” in capturing the value of fast data with different stream processing alternatives. Then hear from John Hugg, Founding Engineer of VoltDB, as he discusses the VoltDB approach to fast data. Learn what’s possible when systems integrate event processing with state management in a consistent, transactional way. You can also view the webinar recording here: http://learn.voltdb.com/WRStonebrakerRTeventprocessing.html

Big Data Streams Architectures. Why? What? How?

Anton Nazaruk

With a current zoo of technologies and different ways of their interaction it's a big challenge to architect a system (or adopt existed one) that will conform to low-latency BigData analysis requirements. Apache Kafka and Kappa Architecture in particular take more and more attention over classic Hadoop-centric technologies stack. New Consumer API put significant boost in this direction. Microservices-based streaming processing and new Kafka Streams tend to be a synergy in BigData world.

Librecon 2016 bilbao: kappa architecture IoT of the cars

Juantomás García Molina

What's hot

DataXDay - Real-Time Access log analysis

DataXDay Conference by Xebia

Monitoring Half a Million ML Models, IoT Streaming Data, and Automated Qualit...

Databricks

MongoDB .local London 2019: The Tech Behind Connected Car

MongoDB

Scylla Summit 2022: Scalable and Sustainable Supply Chains with DLT and ScyllaDB

ScyllaDB

RealTime Recommendations @Netflix - Spark

Nitin S

[WSO2Con EU 2018] Tooling for Observability

WSO2

Ingesting IoT data in Food Processing

confluent

OVH Analytics Data Compute - Apache Spark Cluster as a Service

OVHcloud

Ovh analytics data compute with apache spark as a service meetup ovh bordeaux

Mojtaba Imani

Avoid SPOF in Cloud-native Apps

Thang Chung

Building Notebook-based AI Pipelines with Elyra and Kubeflow

Databricks

SAIS2018 - Fact Store At Netflix Scale

Nitin S

Martin Moucka [Red Hat] | How Red Hat Uses gNMI, Telegraf and InfluxDB to Gai...

InfluxData

How to Build Cloud-based Microservice Environments with Docker and VoltDB

VoltDB

[WSO2Con EU 2018] OpenAPI Specification 3 - The Evolution of Swagger

WSO2

Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...

HostedbyConfluent

The Expert Guide to Fast Data

VoltDB

Virtualizing Analytics with Apache Spark: Keynote by Arsalan Tavakoli

Spark Summit

apidays LIVE Paris 2021 - Stargate.io, An OSS Api Layer for your Cassandra by...

apidays

Mike Stonebraker on Designing An Architecture For Real-time Event Processing

VoltDB

What's hot (20)

DataXDay - Real-Time Access log analysis

Monitoring Half a Million ML Models, IoT Streaming Data, and Automated Qualit...

MongoDB .local London 2019: The Tech Behind Connected Car

Scylla Summit 2022: Scalable and Sustainable Supply Chains with DLT and ScyllaDB

RealTime Recommendations @Netflix - Spark

[WSO2Con EU 2018] Tooling for Observability

Ingesting IoT data in Food Processing

OVH Analytics Data Compute - Apache Spark Cluster as a Service

Ovh analytics data compute with apache spark as a service meetup ovh bordeaux

Avoid SPOF in Cloud-native Apps

Building Notebook-based AI Pipelines with Elyra and Kubeflow

SAIS2018 - Fact Store At Netflix Scale

Martin Moucka [Red Hat] | How Red Hat Uses gNMI, Telegraf and InfluxDB to Gai...

How to Build Cloud-based Microservice Environments with Docker and VoltDB

[WSO2Con EU 2018] OpenAPI Specification 3 - The Evolution of Swagger

Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...

The Expert Guide to Fast Data

Virtualizing Analytics with Apache Spark: Keynote by Arsalan Tavakoli

apidays LIVE Paris 2021 - Stargate.io, An OSS Api Layer for your Cassandra by...

Mike Stonebraker on Designing An Architecture For Real-time Event Processing

Viewers also liked

Big Data Streams Architectures. Why? What? How?

Anton Nazaruk

Librecon 2016 bilbao: kappa architecture IoT of the cars

Juantomás García Molina

ASPgems - kappa architecture

Juantomás García Molina

Streaming Architecture to Connect Everything (Including Hybrid Cloud) - Strat...

Mathieu Dumoulin

February 2017 HUG: Exactly-once end-to-end processing with Apache Apex

Yahoo Developer Network

Apache Apex (http://apex.apache.org/) is a stream processing platform that helps organizations to build processing pipelines with fault tolerance and strong processing guarantees. It was built to support low processing latency, high throughput, scalability, interoperability, high availability and security. The platform comes with Malhar library - an extensive collection of processing operators and a wide range of input and output connectors for out-of-the-box integration with an existing infrastructure. In the talk I am going to describe how connectors together with the distributed checkpointing (a mechanism used by the Apex to support fault tolerance and high availability) provide exactly-once end-to-end processing guarantees. Speakers: Vlad Rozov is Apache Apex PMC member and back-end engineer at DataTorrent where he focuses on the buffer server, Apex platform network layer, benchmarks and optimizing the core components for low latency and high throughput. Prior to DataTorrent Vlad worked on distributed BI platform at Huawei and on multi-dimensional database (OLAP) at Hyperion Solutions and Oracle.

Data Pipelines with Apache Kafka

Ben Stopford

Big Data Architecture

Guido Schmutz

The right architecture is key for any IT project. This is especially the case for big data projects, where there are no standard architectures which have proven their suitability over years. This session discusses the different Big Data Architectures which have evolved over time, including traditional Big Data Architecture, Streaming Analytics architecture as well as Lambda and Kappa architecture and presents the mapping of components from both Open Source as well as the Oracle stack onto these architectures.

El futuro Data Driven en e-Learning y RR.HH.

OpenSistemas

El software como acción humana

OpenSistemas

TRAINING1Chiara Scimemi

DWS16 - Future Networks forum - Anna Krzyzanowska European Commission

IDATE DigiWorld

Connected Cars Quickly Becoming Part of the Internet of Things (IoT)

ParthaS

Getting The Deal Through: Telecoms & Media Market Intelligence 2016

Matheson Law Firm

Отражение цифровизации в решениях Cisco

Cisco Russia

Apache spark y cómo lo usamos en nuestros proyectos

OpenSistemas

Cloudera - Enabling the IoT Revolution Driving Insights in a Connected World

andreas kuncoro

Global mobile trends

Smich Butcharoen

Car talk - The Conversational Future of Automobiles

Sudha Jamthe

Sudha Jamthe presents the many communications of a driverless car at MoNage 2017 conference in San Jose. She predicts a Car AI and the need for city infrastructure and wearables on humans to become Cognitive IoT by 2030. The next 10-15 years is a transformation that impacts urban mobility, automobile, and Transportation industries. In this conference, she will launch her new book "2030 The Driverless Car - Business Transformation from Autonomous Vehicles" your guide for innovation and digital transformation. The book is available on Amazon at http://bit.ly/driverlessworld

Marsden CNRS European net neutrality law & Guidelines 12092016

Chris Marsden

Chris T. Marsden will discuss the past, present and future of net neutrality regulation, enforcement and implementation, focussing on EU Regulation 2015/2120 and the Guidelines issued by BEREC on 30 August 2016. He will argue that the success of the Guidelines is dependent on the actions of the 28 national regulators and 9 observer regulators (one of whom actually wrote the majority of the Guidelines). He will draw comparison with the potential enforcement of net neutrality in other parts of the world http://www.iscc.cnrs.fr/spip.php?article2245

Claire Bury - A New Telecoms Regulatory Framework

FSR Communications and Media

Viewers also liked (20)

Big Data Streams Architectures. Why? What? How?

Librecon 2016 bilbao: kappa architecture IoT of the cars

ASPgems - kappa architecture

Streaming Architecture to Connect Everything (Including Hybrid Cloud) - Strat...

February 2017 HUG: Exactly-once end-to-end processing with Apache Apex

Data Pipelines with Apache Kafka

Big Data Architecture

El futuro Data Driven en e-Learning y RR.HH.

El software como acción humana

TRAINING1

DWS16 - Future Networks forum - Anna Krzyzanowska European Commission

Connected Cars Quickly Becoming Part of the Internet of Things (IoT)

Getting The Deal Through: Telecoms & Media Market Intelligence 2016

Отражение цифровизации в решениях Cisco

Apache spark y cómo lo usamos en nuestros proyectos

Cloudera - Enabling the IoT Revolution Driving Insights in a Connected World

Global mobile trends

Car talk - The Conversational Future of Automobiles

Marsden CNRS European net neutrality law & Guidelines 12092016

Claire Bury - A New Telecoms Regulatory Framework

Similar to Kappa Architecture, IoT of the cars - LibreCon 2016

Meetup big data developers 2017 madrid - spark real use cases

Juantomás García Molina

Scalaua 2017 kyev kappa architecture 2.0

Juantomás García Molina

Gdg cloud london 2017 kappa architecture 2.0 copia

Juantomás García Molina

JBCN barcelona 2017 kappa architecture 2.0

Juantomás García Molina

Codemotion madrid 2017 Arquitectura kappa 2.0

Juantomás García Molina

Datascience lab 2017 odessa kappa architecture 2.0

Juantomás García Molina

DataScience Lab 2017_Kappa Architecture: How to implement a real-time streami...

GeeksLab Odessa

DataScience Lab, 13 мая 2017 Kappa Architecture: How to implement a real-time streaming data analytics engine Juantomás García (Data Solutions Manager at OpenSistemas, Madrid, Spain) We will have an introduction of what is the kappa architecture vs lambda architecture. We will see how kappa architecture is a good solution to implement solutions in (almost) real time when we need to analyze data in streaming. We will show in a case of real use: how architecture is designed, how pipelines are organized and how data scientists use it. We will review the most used technologies to implement it from apache Kafka + spark using Scala to new tools like apache beam / google dataflow. Все материалы: http://datascience.in.ua/report2017

Lambda architecture for real time big data

Trieu Nguyen

Data Science at Scale: Using Apache Spark for Data Science at Bitly

Sarah Guido

Given at Data Day Seattle 2015. Bitly generates over 9 billion clicks on shortened links a month, as well as over 100 million unique link shortens. Analyzing data of this scale is not without its challenges. At Bitly, we have started adopting Apache Spark as a way to process our data. In this talk, I’ll elaborate on how I use Spark as part of my data science workflow. I’ll cover how Spark fits into our existing architecture, the kind of problems I’m solving with Spark, and the benefits and challenges of using Spark for large-scale data science.

Global Open Source Development 2011-2014 Review and 2015 Forecast

Sammy Fung

Liferay and Big Data

Miguel Pastor

Web APIs: The future of software

Reuven Lerner

Since the beginning of software, we have bundled frequently-used functionality into libraries -- both our own, and those created by third parties. APIs tell software developers how to use a library, and make it possible for developers to treat libraries as black boxes, considering only what the library does, not how it does it. Today, as software moves increasingly to the Web, APIs and libraries are also moving to the Web. This means that you can now outsource, by means of Web APIs, a huge amount of your application's functionality. What does this mean for software developers? What does this mean for the future of software development? In this talk, I describe the past, present, and future of APIs, and how I believe the Web is changing the landscape.

Citizen Developer Tools @ Valo Solutions / Blue Meteorite Monday session

Antti Koskela

So, the citizen developers have all the cool tools, and those that actually code for a living are left with legacy stuff? Not so fast! The same tools that Microsoft is targeting for citizen developers make development easier, faster and cheaper for everyone! This session combines tools such as Flow, Azure Cognitive Services and Azure Functions with some actual simple development work to provide highly customized, Machine Learning powered analysis workflow for the newly baked Modern Team Sites in SharePoint Online. This demo-heavy session will look at real business scenarios, and how we can solve them using citizen developer tools and some code (Because we’re developers after all, right?) After this session you'll know how to create rich and customized business automation processes that use the latest tools offered to us by Microsoft.

Atlanta hadoop users group july 2013Christopher Curtin

Citizen Developer Tools - session at SPS New England 10/20/2018

Antti Koskela

So, the citizen developers have all the cool tools, and those that actually code for a living are left with legacy stuff? Not so fast! The same tools that Microsoft is targeting for citizen developers make development easier, faster and cheaper for everyone! This session combines tools such as Flow, Azure Cognitive Services and Azure Functions with some actual simple development work to provide highly customized, Machine Learning powered analysis workflow for the newly baked Modern Team Sites in SharePoint Online. This demo-heavy session will look at real business scenarios, and how we can solve them using citizen developer tools and some code (Because we’re developers after all, right?) After this session, you'll know how to create rich and customized business automation processes that use the latest tools offered to us by Microsoft.

Serverless brewbox

Lino Telera

Architecting Your First Big Data Implementation

Adaryl "Bob" Wakefield, MBA

Citizen Developer Tools (session at SharePoint Saturday Twin Cities 4/14/2018...

Antti Koskela

So, the citizen developers have all the cool tools, and those that actually code for a living are left with legacy stuff? Not so fast! The same tools that Microsoft is targeting for citizen developers make development easier, faster and cheaper for everyone! This session combines tools such as Flow, Azure Cognitive Services and Azure Functions with some actual simple development work to provide highly customized, Machine Learning powered analysis workflow for the newly baked Modern Team Sites in SharePoint Online. This demo-heavy session will look at real business scenarios, and how we can solve them using citizen developer tools and some code (Because we’re developers after all, right?) After this session, you'll know how to create rich and customized business automation processes that use the latest tools offered to us by Microsoft. See the whole description here: https://www.koskila.net/2018/04/09/speaking-at-sps-twin-cities-sps-tc/ And my experiences from the event here: https://www.koskila.net/2018/04/27/my-sps-tc-session-citizen-developer-tools-are-not-just-for-citizen-developers/

Citizen Developer Tools are not just for Citizen Developers (session at Share...

Antti Koskela

So, the citizen developers have all the cool tools, and those that actually code for a living are left with legacy stuff? Not so fast! The same tools that Microsoft is targeting for citizen developers make development easier, faster and cheaper for everyone! This session combines tools such as Flow, Azure Cognitive Services and Azure Functions with some actual simple development work to provide highly customized, Machine Learning powered analysis workflow for the newly baked Modern Team Sites in SharePoint Online. This demo-heavy session will look at real business scenarios, and how we can solve them using citizen developer tools and some code (Because we’re developers after all, right?) After this session, you'll know how to create rich and customized business automation processes that use the latest tools offered to us by Microsoft.

Big Graph Analytics on Neo4j with Apache Spark

Kenny Bastani

In this talk I will introduce you to a Docker container that provides you an easy way to do distributed graph processing using Apache Spark GraphX and a Neo4j graph database. You'll learn how to analyze big data graphs that are exported from Neo4j and consequently updated from the results of a Spark GraphX analysis. The types of analysis I will be talking about are PageRank, connected components, triangle counting, and community detection. Database technologies have evolved to be able to store big data, but are largely inflexible. For complex graph data models stored in a relational database there may be tedious transformations and shuffling around of data to perform large scale analysis. Fast and scalable analysis of big data has become a critical competitive advantage for companies. There are open source tools like Apache Hadoop and Apache Spark that are providing opportunities for companies to solve these big data problems in a scalable way. Platforms like these have become the foundation of the big data analysis movement. Speakers

Similar to Kappa Architecture, IoT of the cars - LibreCon 2016 (20)

Meetup big data developers 2017 madrid - spark real use cases

Scalaua 2017 kyev kappa architecture 2.0

Gdg cloud london 2017 kappa architecture 2.0 copia

JBCN barcelona 2017 kappa architecture 2.0

Codemotion madrid 2017 Arquitectura kappa 2.0

Datascience lab 2017 odessa kappa architecture 2.0

DataScience Lab 2017_Kappa Architecture: How to implement a real-time streami...

Lambda architecture for real time big data

Data Science at Scale: Using Apache Spark for Data Science at Bitly

Global Open Source Development 2011-2014 Review and 2015 Forecast

Liferay and Big Data

Web APIs: The future of software

Citizen Developer Tools @ Valo Solutions / Blue Meteorite Monday session

Atlanta hadoop users group july 2013

Citizen Developer Tools - session at SPS New England 10/20/2018

Serverless brewbox

Architecting Your First Big Data Implementation

Citizen Developer Tools (session at SharePoint Saturday Twin Cities 4/14/2018...

Citizen Developer Tools are not just for Citizen Developers (session at Share...

Big Graph Analytics on Neo4j with Apache Spark

Recently uploaded

Bridging the Digital Gap Brad Spiegel Macon, GA Initiative.pptx

Brad Spiegel Macon GA

test test test test testtest test testtest test testtest test testtest test ...

Arif0071

Internet-Security-Safeguarding-Your-Digital-World (1).pptx

VivekSinghShekhawat2

# Internet Security: Safeguarding Your Digital World In the contemporary digital age, the internet is a cornerstone of our daily lives. It connects us to vast amounts of information, provides platforms for communication, enables commerce, and offers endless entertainment. However, with these conveniences come significant security challenges. Internet security is essential to protect our digital identities, sensitive data, and overall online experience. This comprehensive guide explores the multifaceted world of internet security, providing insights into its importance, common threats, and effective strategies to safeguard your digital world. ## Understanding Internet Security Internet security encompasses the measures and protocols used to protect information, devices, and networks from unauthorized access, attacks, and damage. It involves a wide range of practices designed to safeguard data confidentiality, integrity, and availability. Effective internet security is crucial for individuals, businesses, and governments alike, as cyber threats continue to evolve in complexity and scale. ### Key Components of Internet Security 1. **Confidentiality**: Ensuring that information is accessible only to those authorized to access it. 2. **Integrity**: Protecting information from being altered or tampered with by unauthorized parties. 3. **Availability**: Ensuring that authorized users have reliable access to information and resources when needed. ## Common Internet Security Threats Cyber threats are numerous and constantly evolving. Understanding these threats is the first step in protecting against them. Some of the most common internet security threats include: ### Malware Malware, or malicious software, is designed to harm, exploit, or otherwise compromise a device, network, or service. Common types of malware include: - **Viruses**: Programs that attach themselves to legitimate software and replicate, spreading to other programs and files. - **Worms**: Standalone malware that replicates itself to spread to other computers. - **Trojan Horses**: Malicious software disguised as legitimate software. - **Ransomware**: Malware that encrypts a user's files and demands a ransom for the decryption key. - **Spyware**: Software that secretly monitors and collects user information. ### Phishing Phishing is a social engineering attack that aims to steal sensitive information such as usernames, passwords, and credit card details. Attackers often masquerade as trusted entities in email or other communication channels, tricking victims into providing their information. ### Man-in-the-Middle (MitM) Attacks MitM attacks occur when an attacker intercepts and potentially alters communication between two parties without their knowledge. This can lead to the unauthorized acquisition of sensitive information. ### Denial-of-Service (DoS) and Distributed Denial-of-Service (DDoS) Attacks

Latest trends in computer networking.pptx

JungkooksNonexistent

Multi-cluster Kubernetes Networking- Patterns, Projects and Guidelines

Sanjeev Rampal

Talk presented at Kubernetes Community Day, New York, May 2024. Technical summary of Multi-Cluster Kubernetes Networking architectures with focus on 4 key topics. 1) Key patterns for Multi-cluster architectures 2) Architectural comparison of several OSS/ CNCF projects to address these patterns 3) Evolution trends for the APIs of these projects 4) Some design recommendations & guidelines for adopting/ deploying these solutions.

1.Wireless Communication System_Wireless communication is a broad term that i...

JeyaPerumal1

Wireless communication involves the transmission of information over a distance without the help of wires, cables or any other forms of electrical conductors. Wireless communication is a broad term that incorporates all procedures and forms of connecting and communicating between two or more devices using a wireless signal through wireless communication technologies and devices. Features of Wireless Communication The evolution of wireless technology has brought many advancements with its effective features. The transmitted distance can be anywhere between a few meters (for example, a television's remote control) and thousands of kilometers (for example, radio communication). Wireless communication can be used for cellular telephony, wireless access to the internet, wireless home networking, and so on.

APNIC Foundation, presented by Ellisha Heppner at the PNG DNS Forum 2024

APNIC

JAVIER LASA-EXPERIENCIA digital 1986-2024.pdf

Javier Lasa

一比一原版(SLU毕业证)圣路易斯大学毕业证成绩单专业办理

keoku

SLU毕业证原版定制【微信：176555708】【圣路易斯大学毕业证成绩单-学位证】【微信：176555708】（留信学历认证永久存档查询）采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。 ◆◆◆◆◆ — — — — — — — — 【留学教育】留学归国服务中心 — — — — — -◆◆◆◆◆ 【主营项目】一.毕业证【微信：176555708】成绩单、使馆认证、教育部认证、雅思托福成绩单、学生卡等！二.真实使馆公证(即留学回国人员证明,不成功不收费) 三.真实教育部学历学位认证（教育部存档！教育部留服网站永久可查）四.办理各国各大学文凭(一对一专业服务,可全程监控跟踪进度) 如果您处于以下几种情况： ◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证【微信：176555708】 ◇面对父母的压力，希望尽快拿到； ◇不清楚认证流程以及材料该如何准备； ◇回国时间很长，忘记办理； ◇回国马上就要找工作，办给用人单位看； ◇企事业单位必须要求办理的 ◇需要报考公务员、购买免税车、落转户口 ◇申请留学生创业基金留信网认证的作用: 1:该专业认证可证明留学生真实身份 2:同时对留学生所学专业登记给予评定 3:国家专业人才认证中心颁发入库证书 4:这个认证书并且可以归档倒地方 5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息 6:个人职称评审加20分 7:个人信誉贷款加10分→ 【关于价格问题（保证一手价格）我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子我给客户的都是第一手的代理价格，因为我想坦诚对待大家不想跟大家在价格方面浪费时间对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。 8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才选择实体注册公司办理，更放心，更安全！我们的承诺：可来公司面谈，可签订合同，会陪同客户一起到教育部认证窗口递交认证材料，客户在教育部官方认证查询网站查询到认证通过结果后付款，不成功不收费！学历顾问：微信：176555708

1比1复刻(bath毕业证书)英国巴斯大学毕业证学位证原版一模一样

3ipehhoa

原版纸张【微信：741003700 】【(bath毕业证书)英国巴斯大学毕业证学位证】【微信：741003700 】学位证，留信认证（真实可查，永久存档）offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。本公司拥有海外各大学样板无数，能完美还原海外各大学 Bachelor Diploma degree, Master Degree Diploma 1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700 留信网认证的作用: 1:该专业认证可证明留学生真实身份 2:同时对留学生所学专业登记给予评定 3:国家专业人才认证中心颁发入库证书 4:这个认证书并且可以归档倒地方 5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息 6:个人职称评审加20分 7:个人信誉贷款加10分 8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才

How to Use Contact Form 7 Like a Pro.pptx

Gal Baras

The+Prospects+of+E-Commerce+in+China.pptx

laozhuseo02

急速办(bedfordhire毕业证书)英国贝德福特大学毕业证成绩单原版一模一样

3ipehhoa

原版纸张【微信：741003700 】【(bedfordhire毕业证书)英国贝德福特大学毕业证成绩单】【微信：741003700 】学位证，留信认证（真实可查，永久存档）offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。本公司拥有海外各大学样板无数，能完美还原海外各大学 Bachelor Diploma degree, Master Degree Diploma 1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700 留信网认证的作用: 1:该专业认证可证明留学生真实身份 2:同时对留学生所学专业登记给予评定 3:国家专业人才认证中心颁发入库证书 4:这个认证书并且可以归档倒地方 5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息 6:个人职称评审加20分 7:个人信誉贷款加10分 8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才

BASIC C++ lecture NOTE C++ lecture 3.pptx

natyesu

Comptia N+ Standard Networking lesson guide

GTProductions1

History+of+E-commerce+Development+in+China-www.cfye-commerce.shop

laozhuseo02

This 7-second Brain Wave Ritual Attracts Money To You.!

nirahealhty

原版仿制(uob毕业证书)英国伯明翰大学毕业证本科学历证书原版一模一样

3ipehhoa

原版纸张【微信：741003700 】【(uob毕业证书)英国伯明翰大学毕业证】【微信：741003700 】学位证，留信认证（真实可查，永久存档）offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。本公司拥有海外各大学样板无数，能完美还原海外各大学 Bachelor Diploma degree, Master Degree Diploma 1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700 留信网认证的作用: 1:该专业认证可证明留学生真实身份 2:同时对留学生所学专业登记给予评定 3:国家专业人才认证中心颁发入库证书 4:这个认证书并且可以归档倒地方 5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息 6:个人职称评审加20分 7:个人信誉贷款加10分 8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才

guildmasters guide to ravnica Dungeons & Dragons 5...

Rogerio Filho

一比一原版(LBS毕业证)伦敦商学院毕业证成绩单专业办理

eutxy

LBS毕业证原版定制【微信：176555708】【伦敦商学院毕业证成绩单-学位证】【微信：176555708】（留信学历认证永久存档查询）采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。 ◆◆◆◆◆ — — — — — — — — 【留学教育】留学归国服务中心 — — — — — -◆◆◆◆◆ 【主营项目】一.毕业证【微信：176555708】成绩单、使馆认证、教育部认证、雅思托福成绩单、学生卡等！二.真实使馆公证(即留学回国人员证明,不成功不收费) 三.真实教育部学历学位认证（教育部存档！教育部留服网站永久可查）四.办理各国各大学文凭(一对一专业服务,可全程监控跟踪进度) 如果您处于以下几种情况： ◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证【微信：176555708】 ◇面对父母的压力，希望尽快拿到； ◇不清楚认证流程以及材料该如何准备； ◇回国时间很长，忘记办理； ◇回国马上就要找工作，办给用人单位看； ◇企事业单位必须要求办理的 ◇需要报考公务员、购买免税车、落转户口 ◇申请留学生创业基金留信网认证的作用: 1:该专业认证可证明留学生真实身份 2:同时对留学生所学专业登记给予评定 3:国家专业人才认证中心颁发入库证书 4:这个认证书并且可以归档倒地方 5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息 6:个人职称评审加20分 7:个人信誉贷款加10分→ 【关于价格问题（保证一手价格）我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子我给客户的都是第一手的代理价格，因为我想坦诚对待大家不想跟大家在价格方面浪费时间对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。 8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才选择实体注册公司办理，更放心，更安全！我们的承诺：可来公司面谈，可签订合同，会陪同客户一起到教育部认证窗口递交认证材料，客户在教育部官方认证查询网站查询到认证通过结果后付款，不成功不收费！学历顾问：微信：176555708

Recently uploaded (20)

Bridging the Digital Gap Brad Spiegel Macon, GA Initiative.pptx

test test test test testtest test testtest test testtest test testtest test ...

Internet-Security-Safeguarding-Your-Digital-World (1).pptx

Latest trends in computer networking.pptx

Multi-cluster Kubernetes Networking- Patterns, Projects and Guidelines

1.Wireless Communication System_Wireless communication is a broad term that i...

APNIC Foundation, presented by Ellisha Heppner at the PNG DNS Forum 2024

JAVIER LASA-EXPERIENCIA digital 1986-2024.pdf

一比一原版(SLU毕业证)圣路易斯大学毕业证成绩单专业办理

1比1复刻(bath毕业证书)英国巴斯大学毕业证学位证原版一模一样

How to Use Contact Form 7 Like a Pro.pptx

The+Prospects+of+E-Commerce+in+China.pptx

急速办(bedfordhire毕业证书)英国贝德福特大学毕业证成绩单原版一模一样

BASIC C++ lecture NOTE C++ lecture 3.pptx

Comptia N+ Standard Networking lesson guide

History+of+E-commerce+Development+in+China-www.cfye-commerce.shop

This 7-second Brain Wave Ritual Attracts Money To You.!

原版仿制(uob毕业证书)英国伯明翰大学毕业证本科学历证书原版一模一样

guildmasters guide to ravnica Dungeons & Dragons 5...

一比一原版(LBS毕业证)伦敦商学院毕业证成绩单专业办理

Kappa Architecture, IoT of the cars - LibreCon 2016

1. Kappa Architecture: IoT of the cars Juantomás García

2. Who am I • Data Solutions Manager at Open Sistemas • Co-author of the ﬁrst spanish book about free software “La pastilla roja” • Former president of Hispalinux (Spanish Linux User Group). • Co-organizer of Machine Learning Spain Meetup

3. What’s Kappa Architecture July 2, 2014 Jay Kreps coined the term Kappa Architecture in an article for O’reilly Radar

4. Who is Jay Kreps Jay has been involved in lots of projects: Author of the essay: The Log: What every software engineer should know about real-time data's unifying abstraction (12/16/2013) https://engineering.linkedin.com/distributed-systems/log-what-every-software- engineer-should-know-about-real-time-datas-unifying

5. Who is Jay Kreps Author of the book: I ♥ Logs

6. Who is Jay Kreps Involved with projects as: Apache Kafka Apache Samza Voldemort Azkaban Ex-Linkedin Now co-founder and CEO of Confluent

7. Usual data ﬂow

8. Kappa Architecture Way

9. Tools we use to implement Kappa Architecture

10. Tools we use to implement Kappa Architecture

11. Tools we use to implement Kappa Architecture • Our favorite kafka features. • Easy to use • Robust & Resilient • Kafka connectors • Kafka streams • Compact logs • Very fast writes and reads

12. Tools we use to implement Kappa Architecture

13. Some favorite Spark features • If you have an schema spark SQL is perfect. • Spark streaming work very fine with spark and almost all streaming sources. • Structured queries will be a huge advance. • We love scala and scala is the spirit of spark.

14. Some favorite Spark features • We love code like this:

15. A real case • One of our clients want to monitor all the cars informations via OBD II • OBD II is a car interface with the car electronics. • Client develop an app to read all the car information throw a ODB II with bluetooth

16. A real case

17. First Problems • We need to scale the rest interfaces too many requests. • MySQL don’t scale • Client wants to make realtime expensive queries.

18. Some Numbers

19. Architecture V2

20. Architecture KappaV3

21. Now we’re more ﬂexible • We can have queries like: • What are the drivers that are not client of the X gas brand, has a few gas and are near of gas station of the brand X and if true, send a notification with a discount coupon and a link with the route.

22. Takeaways • Kappa architecture is not a silver bullet but helps in a lot of solutions. • Kafka + spark streaming is our favorite tools • There a lots of improvements: • OLAP like Apache Druid • Graph databases like neo4j • Kafka streams and compacts logs

23. ¿Questions? • My contact info: • Juantomás García • juantomas@opensistemas.com • @juantomas

Kappa Architecture, IoT of the cars - LibreCon 2016

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Kappa Architecture, IoT of the cars - LibreCon 2016

Similar to Kappa Architecture, IoT of the cars - LibreCon 2016 (20)

More from LibreCon

More from LibreCon (20)

Recently uploaded

Recently uploaded (20)

Kappa Architecture, IoT of the cars - LibreCon 2016