Microservices without Servers
•
0 likes•259 views
Servers are killing your productivity. Rather than building better software for your users, you end up constantly distracted by maintaining computers. Wasn't the "cloud" supposed to fix this? It sounded so promising until we realized it was just renting VMs in someone else's datacenter. We couldn't escape "servers". Until now... In this session, developers will learn how to build microservices without servers, using modern “serverless” cloud platforms. We’ll look at common challenges (and solutions) to building applications using “serverless” stacks. Exploring emerging “serverless” design patterns will give developers the knowledge to build application architectures using these new platforms. Presented by Jonathan Kaufman, Developer Advocate for IBM Emerging Technology currently residing in Cambridge, MA.
Report
Share
Report
Share
Download to read offline
Recommended
Two way data sync between legacy and your brand new micro-service architecture
This document discusses strategies for synchronizing data between a legacy system and a new microservices architecture. It proposes using change data capture (CDC) to log all database changes in the legacy system and streaming them to the microservices using Apache Kafka. This allows the two systems to run concurrently while keeping the data eventually consistent as the services are migrated gradually using a strangler pattern.
Kafka: Legacy microservices
Kafka can be used to smoothly migrate a legacy monolithic application to a new microservices-oriented architecture. The migration occurs gradually with no big bang or production stop. Features are gradually swapped from the legacy application to new microservices. The legacy application and microservices run side by side and can concurrently modify the same data conflictlessly. The migration involves two steps - first pushing changes from the legacy application to the microservices using Kafka streams, and then pushing changes from the microservices back to the legacy application. This allows for an ability to rollback and derisks the data migration process.
How Microsoft Built and Scaled Cosmos
Cosmos is a large-scale data processing system used by thousands at Microsoft to process exabytes of data across clusters of over 50,000 servers. It provides a SQL-like language and allows teams to easily share and join data. This drives huge scalability requirements. The Apollo scheduler was developed to maximize cluster utilization while minimizing latency for heterogeneous workloads at cloud scale. Later, JetScope was created to support lower latency interactive queries through intermediate result streaming and gang scheduling while maintaining fault tolerance.
Clustered Architecture Patterns Delivering Scalability And Availability
The document discusses different architecture patterns for delivering scalability and availability in clustered systems. It covers load-balanced and partitioned scale-out patterns, and how to balance simplicity, scalability, and availability. JVM-level clustering is presented as an approach that can address these patterns by sharing memory across JVMs in a transparent way.
INTRODUCING: CREATE PIPELINE
This document introduces MemSQL Pipelines, which allow for exactly-once data ingestion semantics when streaming data from Kafka into MemSQL. MemSQL Pipelines provide a native way to extract, transform, and load external data into MemSQL tables. They offer a scalable and highly performant ETL process across a distributed cluster. The document explains streaming semantics like at least once and exactly once delivery, and how MemSQL Pipelines coordinate with Kafka to enable exactly-once ingestion through offset tracking. It presents the architecture of MemSQL Pipelines and demonstrates their use through a live demo.
O'Reilly Media Webcast: Building Real-Time Data Pipelines
As our customers tap into new sources of data or modify to existing data pipelines, we are often asked questions like: What technologies should we consider? Where can we reduce data latency? How can we simplify our data architecture?
To eliminate the guesswork, we teamed up with Ben Lorica, Chief Data Scientist at O’Reilly Media to host a webcast centered around building real-time data pipelines.
Lessons Learned from Building and Operating Scuba
This document provides an overview of Scuba, Facebook's real-time analytics database. It summarizes Scuba's key features including real-time data ingestion and querying capabilities with simple rollup queries and flexible schemas. It also describes Scuba's architecture with distributed data storage and demand control. Finally, it discusses lessons learned from building and operating Scuba, including common issues and reasons for its success filling a specific niche for analytics.
GECon2017_High-volume data streaming in azure_ Aliaksandr Laisha
The session will be focused on solutions that require high-throughput ingestion & streaming of data in real-time. You'll get familiar with different business uses-cases and architecture examples to get a common idea as well as understand the concepts of stream processing systems. Next, you'll get deep insights into functional and non-functional capabilities of Azure Event Hub service to see how it fits into the whole picture. Moreover we’ll take a look how to leverage Azure CosmosDB for high-throughput streaming when Event Hub is not suitable by different reasons.
Recommended
Two way data sync between legacy and your brand new micro-service architecture
This document discusses strategies for synchronizing data between a legacy system and a new microservices architecture. It proposes using change data capture (CDC) to log all database changes in the legacy system and streaming them to the microservices using Apache Kafka. This allows the two systems to run concurrently while keeping the data eventually consistent as the services are migrated gradually using a strangler pattern.
Kafka: Legacy microservices
Kafka can be used to smoothly migrate a legacy monolithic application to a new microservices-oriented architecture. The migration occurs gradually with no big bang or production stop. Features are gradually swapped from the legacy application to new microservices. The legacy application and microservices run side by side and can concurrently modify the same data conflictlessly. The migration involves two steps - first pushing changes from the legacy application to the microservices using Kafka streams, and then pushing changes from the microservices back to the legacy application. This allows for an ability to rollback and derisks the data migration process.
How Microsoft Built and Scaled Cosmos
Cosmos is a large-scale data processing system used by thousands at Microsoft to process exabytes of data across clusters of over 50,000 servers. It provides a SQL-like language and allows teams to easily share and join data. This drives huge scalability requirements. The Apollo scheduler was developed to maximize cluster utilization while minimizing latency for heterogeneous workloads at cloud scale. Later, JetScope was created to support lower latency interactive queries through intermediate result streaming and gang scheduling while maintaining fault tolerance.
Clustered Architecture Patterns Delivering Scalability And Availability
The document discusses different architecture patterns for delivering scalability and availability in clustered systems. It covers load-balanced and partitioned scale-out patterns, and how to balance simplicity, scalability, and availability. JVM-level clustering is presented as an approach that can address these patterns by sharing memory across JVMs in a transparent way.
INTRODUCING: CREATE PIPELINE
This document introduces MemSQL Pipelines, which allow for exactly-once data ingestion semantics when streaming data from Kafka into MemSQL. MemSQL Pipelines provide a native way to extract, transform, and load external data into MemSQL tables. They offer a scalable and highly performant ETL process across a distributed cluster. The document explains streaming semantics like at least once and exactly once delivery, and how MemSQL Pipelines coordinate with Kafka to enable exactly-once ingestion through offset tracking. It presents the architecture of MemSQL Pipelines and demonstrates their use through a live demo.
O'Reilly Media Webcast: Building Real-Time Data Pipelines
As our customers tap into new sources of data or modify to existing data pipelines, we are often asked questions like: What technologies should we consider? Where can we reduce data latency? How can we simplify our data architecture?
To eliminate the guesswork, we teamed up with Ben Lorica, Chief Data Scientist at O’Reilly Media to host a webcast centered around building real-time data pipelines.
Lessons Learned from Building and Operating Scuba
This document provides an overview of Scuba, Facebook's real-time analytics database. It summarizes Scuba's key features including real-time data ingestion and querying capabilities with simple rollup queries and flexible schemas. It also describes Scuba's architecture with distributed data storage and demand control. Finally, it discusses lessons learned from building and operating Scuba, including common issues and reasons for its success filling a specific niche for analytics.
GECon2017_High-volume data streaming in azure_ Aliaksandr Laisha
The session will be focused on solutions that require high-throughput ingestion & streaming of data in real-time. You'll get familiar with different business uses-cases and architecture examples to get a common idea as well as understand the concepts of stream processing systems. Next, you'll get deep insights into functional and non-functional capabilities of Azure Event Hub service to see how it fits into the whole picture. Moreover we’ll take a look how to leverage Azure CosmosDB for high-throughput streaming when Event Hub is not suitable by different reasons.
Sizing Your Scylla Cluster
This document discusses ScyllaDB's process for sizing a Scylla cluster. It begins by outlining the importance of understanding business, application, and infrastructure requirements. Then it walks through building a sample system based on provided workload details. It shows how the sample system could be configured on different cloud platforms like AWS, Azure, and GCP. Finally, it highlights Scylla's sizing sheet tool for helping to determine hardware needs based on workload characteristics and performance goals.
PowerStream Demo
At Spark Summit East in New York, we unveil PowerStream, an Internet of Things (IoT) simulation with visualizations and alerts based on real-time data from 2 million sensors across global wind farms.
goto; London: Keeping your Cloud Footprint in Check
Presented on the "Lean" track at goto; London September 17th, 2015. Covers how Netflix manages cloud cost efficiency in light of innovation and reliability drivers.
Real-world Streaming Architectures
This document discusses four streaming data architectures used by companies like Yelp, LINE, and Altspace VR. It begins by providing context on streaming data and the evolution from batch processing with Hadoop/MapReduce to stream processing with Apache Kafka and frameworks like Storm. The main case studies then describe how each company uses streaming to build real-time data pipelines and services. Key aspects discussed include using Kafka for messaging between services, Kafka Streams for stream processing, and storing streaming data in databases. The document concludes by promoting Apache Kafka as a platform for building scalable streaming applications.
Maximizing Audience Engagement in Media Delivery (MED303) | AWS re:Invent 2013
Providing a great media consumption experience to customers is crucial to maximizing audience engagement. To do that, it is important that you make content available for consumption anytime, anywhere, on any device, with a personalized and interactive experience. This session explores the power of big data log analytics (real-time and batched), using technologies like Spark, Shark, Kafka, Amazon Elastic MapReduce, Amazon Redshift and other AWS services. Such analytics are useful for content personalization, recommendations, personalized dynamic ad-insertions, interactivity, and streaming quality.
This session also includes a discussion from Netflix, which explores personalized content search and discovery with the power of metadata.
Putting Kafka Together with the Best of Google Cloud Platform
(Kir Titievsky, Google) Kafka Summit SF 2018
In this talk we will share some stories and patterns from customers who have built streaming pipelines and event-driven systems using Confluent Cloud in combination with Google Cloud Platform-native analytics tools, such as BigQuery and Dataflow. We will discuss what Confluent Cloud enables for hybrid deployments and how and why to mix and match platform-native and platform-neutral tools.
#lspe Q1 2013 dynamically scaling netflix in the cloud
Meetup presentation on how Netflix dynamically scales in the cloud. It covers topics primarily related to AWS autoscaling and provides some "day-in-the-life" data.
Surge 2013: Maximizing Scalability, Resiliency, and Engineering Velocity in t...
Surge 2013 presentation which covers how Netflix maximizes engineering velocity while keeping risks to scalability, reliability, and performance in check.
Kafka Summit NYC 2017 - Scalable Real-Time Complex Event Processing @ Uber
Uber uses a scalable real-time complex event processing system to analyze streaming data from its services. The system uses Apache Samza for distributed stream processing and WSO2 Siddhi for complex event processing. Key events are detected using Siddhi queries and then actions like notifications or indexing to databases are triggered. The system processes over 30 billion messages per day across many use cases. Maintaining scalability, fault tolerance, and low latency are ongoing challenges.
Flink Forward SF 2017: Bill Liu & Haohui Mai - AthenaX : Uber’s streaming pro...
The mission of Uber is to make transportation as reliable as running water. The business is fundamentally driven by real-time data -- more than half of the employees in Uber, many of whom are non-technical, use SQL on a regular basis to analyze data and power their business decisions. We are building AthenaX, a stream processing platform built on top of Apache Flink to enable our users to write SQL to process real-time data efficiently and reliably at Uber's scale. Using Apache Calcite as query parser, AthenaX compiles the SQL down to Flink jobs. Leveraging Flink's unique streaming capabilities, AthenaX supports (1) consistent computations reliably thanks to at-least-once guarantees, (2) nontrivial analytics (e.g., windowing and joins) on multiple data sources, and (3) efficient and cost-effective executions in production through code generation and elastic scaling.
Kafka Summit NYC 2017 - Data Processing at LinkedIn with Apache Kafka
LinkedIn uses Apache Kafka extensively to power various data pipelines and platforms. Some key uses of Kafka include:
1) Moving data between systems for monitoring, metrics, search indexing, and more.
2) Powering the Pinot real-time analytics query engine which handles billions of documents and queries per day.
3) Enabling replication and partitioning for the Espresso NoSQL data store using a Kafka-based approach.
4) Streaming data processing using Samza to handle workflows like user profile evaluation. Samza is used for both stateless and stateful stream processing at LinkedIn.
Going from three nines to four nines using Kafka | Tejas Chopra, Netflix
The document discusses using Apache Kafka to improve data upload availability from 99.9% to 99.99% when moving data between on-premise and cloud storage. It describes using Kafka to trigger uploads to the cloud from on-premise storage with 99.9% availability and using Kafka to split uploads between cloud and on-premise storage as well as rehydrating failed on-premise uploads from the cloud to achieve 99.99% availability. The presentation concludes that Kafka provides high throughput and persistence needed to design effective data rehydration strategies across cloud and on-premise storage for very high availability.
(BDT318) How Netflix Handles Up To 8 Million Events Per Second
In this session, Netflix provides an overview of Keystone, their new data pipeline. The session covers how Netflix migrated from Suro to Keystone, including the reasons behind the transition and the challenges of zero loss while processing over 400 billion events daily. The session covers in detail how they deploy, operate, and scale Kafka, Samza, Docker, and Apache Mesos in AWS to manage 8 million events & 17 GB per second during peak.
Deploying Kafka Streams Applications with Docker and Kubernetes
Speakers: Gwen Shapira, Principal Data Architect, Confluent + Matthias J. Sax, Software Engineer, Confluent
Apache Kafka and KSQL in Action: Let's Build a Streaming Data Pipeline!
This document summarizes Mic Hussey's presentation on building streaming data pipelines using Apache Kafka and KSQL. The presentation introduced Kafka and KSQL, demonstrated how to integrate PostgreSQL with Kafka using Kafka Connect, and showed examples of using KSQL to perform streaming ETL operations like joining, filtering, and aggregating event streams in real-time. It also provided several useful links for learning more about change data capture from PostgreSQL to Kafka and building streaming data pipelines with Kafka and KSQL.
Jolt: Distributed, fault-tolerant test running at scale using Mesos
This document summarizes Yelp's new test parallelization system called Jolt. It discusses how Jolt bundles tests into groups to run in parallel on Mesos clusters. This improves upon their previous system called Seagull which had complex custom components. Jolt uses Task Processing to interface with Mesos in a composable way. The document also describes Yelp's autoscaling system called Clusterman which adjusts Mesos cluster sizes based on resource usage and spot instance pricing. It notes how these systems have significantly reduced Yelp's infrastructure costs while improving their ability to run tests and deploy code.
From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...
Mark Teehan, Principal Solutions Engineer, Confluent
Use the Debezium CDC connector to capture database changes from a Postgres database - or MySQL or Oracle; streaming into Kafka topics and onwards to an external data store. Examine how to setup this pipeline using Docker Compose and Confluent Cloud; and how to use various payload formats, such as avro, protobuf and json-schema.
https://www.meetup.com/Singapore-Kafka-Meetup/events/276822852/
Unlocking the world of stream processing with KSQL, the streaming SQL engine ...
Slides of my Strata London 2018 talk:
https://conferences.oreilly.com/strata/strata-eu/public/schedule/detail/65325
Abstract:
Modern businesses have data at their core, and this data is changing continuously. Stream processing is what allows you harness this torrent of information in real time, and thousands of companies use Apache Kafka as the core platform for streaming data to transform and reshape their industries. However, the world of stream processing still has a very high barrier to entry. Today’s most popular stream processing technologies require the user to write code in programming languages such as Java or Scala. This hard requirement on coding skills is preventing many companies to unlock the benefits of stream processing to their full effect.
However, imagine that instead of having to write a lot of code in a programming language like Java or Scala for your favorite stream processing technology, all you’d need to get started with stream processing is a simple SQL statement, such as: SELECT * FROM payments-kafka-stream WHERE fraudProbability > 0.8.
Michael Noll offers an overview of KSQL, the open source streaming SQL engine for Apache Kafka, which makes it easy to get started with a wide range of real-time use cases, such as monitoring application behavior and infrastructure, detecting anomalies and fraudulent activities in data feeds, and real-time ETL. With KSQL, there’s no need to write any code in a programming language. KSQL brings together the worlds of streams and databases by allowing you to work with your data in a stream and in a table format. Built on top of Kafka’s Streams API, KSQL supports many powerful operations, including filtering, transformations, aggregations, joins, windowing, sessionization, and much more. It is open source (Apache 2.0 licensed), distributed, scalable, fault tolerant, and real time. You’ll learn how KSQL makes it easy to get started with a wide range of stream processing use cases and how to get up and running as you explore how it all works under the hood.
Kafka Summit NYC 2017 - Stream it Together: 3 Realities of Modern Programming
The document discusses 3 realities of modern programming: 1) The rise of managed services where over half of Kafka users are using cloud versions. 2) Data is exploding in volume and streaming is needed. 3) Microservices have increased in popularity but communication between services can be complex; Kafka helps solve this as a backbone. Yelp moved to microservices and uses Kafka to connect over 70 services, saving $10M.
Change Data Capture - Scale by the Bay 2019
The document discusses change data capture (CDC) in distributed systems. CDC allows capturing changes to data and notifying actors so they can react accordingly. Some considerations for CDC include reliability, scalability, performance, consistency, and fault tolerance. Common strategies involve using timestamps, triggers, diffs, or log scraping. CDC has various use cases such as caching, search indexing, offline processing, and simplifying monolithic applications. Popular frameworks for implementing CDC include pg2k4j, Maxwell, AWS Database Migration Service, SpinalTap, DataBus, and Debezium.
Docker & PHP - Practical use case
A quick intro to docker presented at the PHP Dublin meetup, and a practical use case of using docker to run multiple PHP versions while doing software development.
Engine lab software hybrid cloud specialists
Engine Lab is a hybrid cloud integration specialist based in Ireland. They securely integrate existing in-house SAP or Oracle applications with cloud applications. Their process involves scoping the business requirements, designing the integration architecture, and delivering the integration solution using Azure Biztalk or bespoke .NET services to interface in-house systems with cloud APIs or bespoke applications while keeping sensitive data in-house.
More Related Content
What's hot
Sizing Your Scylla Cluster
This document discusses ScyllaDB's process for sizing a Scylla cluster. It begins by outlining the importance of understanding business, application, and infrastructure requirements. Then it walks through building a sample system based on provided workload details. It shows how the sample system could be configured on different cloud platforms like AWS, Azure, and GCP. Finally, it highlights Scylla's sizing sheet tool for helping to determine hardware needs based on workload characteristics and performance goals.
PowerStream Demo
At Spark Summit East in New York, we unveil PowerStream, an Internet of Things (IoT) simulation with visualizations and alerts based on real-time data from 2 million sensors across global wind farms.
goto; London: Keeping your Cloud Footprint in Check
Presented on the "Lean" track at goto; London September 17th, 2015. Covers how Netflix manages cloud cost efficiency in light of innovation and reliability drivers.
Real-world Streaming Architectures
This document discusses four streaming data architectures used by companies like Yelp, LINE, and Altspace VR. It begins by providing context on streaming data and the evolution from batch processing with Hadoop/MapReduce to stream processing with Apache Kafka and frameworks like Storm. The main case studies then describe how each company uses streaming to build real-time data pipelines and services. Key aspects discussed include using Kafka for messaging between services, Kafka Streams for stream processing, and storing streaming data in databases. The document concludes by promoting Apache Kafka as a platform for building scalable streaming applications.
Maximizing Audience Engagement in Media Delivery (MED303) | AWS re:Invent 2013
Providing a great media consumption experience to customers is crucial to maximizing audience engagement. To do that, it is important that you make content available for consumption anytime, anywhere, on any device, with a personalized and interactive experience. This session explores the power of big data log analytics (real-time and batched), using technologies like Spark, Shark, Kafka, Amazon Elastic MapReduce, Amazon Redshift and other AWS services. Such analytics are useful for content personalization, recommendations, personalized dynamic ad-insertions, interactivity, and streaming quality.
This session also includes a discussion from Netflix, which explores personalized content search and discovery with the power of metadata.
Putting Kafka Together with the Best of Google Cloud Platform
(Kir Titievsky, Google) Kafka Summit SF 2018
In this talk we will share some stories and patterns from customers who have built streaming pipelines and event-driven systems using Confluent Cloud in combination with Google Cloud Platform-native analytics tools, such as BigQuery and Dataflow. We will discuss what Confluent Cloud enables for hybrid deployments and how and why to mix and match platform-native and platform-neutral tools.
#lspe Q1 2013 dynamically scaling netflix in the cloud
Meetup presentation on how Netflix dynamically scales in the cloud. It covers topics primarily related to AWS autoscaling and provides some "day-in-the-life" data.
Surge 2013: Maximizing Scalability, Resiliency, and Engineering Velocity in t...
Surge 2013 presentation which covers how Netflix maximizes engineering velocity while keeping risks to scalability, reliability, and performance in check.
Kafka Summit NYC 2017 - Scalable Real-Time Complex Event Processing @ Uber
Uber uses a scalable real-time complex event processing system to analyze streaming data from its services. The system uses Apache Samza for distributed stream processing and WSO2 Siddhi for complex event processing. Key events are detected using Siddhi queries and then actions like notifications or indexing to databases are triggered. The system processes over 30 billion messages per day across many use cases. Maintaining scalability, fault tolerance, and low latency are ongoing challenges.
Flink Forward SF 2017: Bill Liu & Haohui Mai - AthenaX : Uber’s streaming pro...
The mission of Uber is to make transportation as reliable as running water. The business is fundamentally driven by real-time data -- more than half of the employees in Uber, many of whom are non-technical, use SQL on a regular basis to analyze data and power their business decisions. We are building AthenaX, a stream processing platform built on top of Apache Flink to enable our users to write SQL to process real-time data efficiently and reliably at Uber's scale. Using Apache Calcite as query parser, AthenaX compiles the SQL down to Flink jobs. Leveraging Flink's unique streaming capabilities, AthenaX supports (1) consistent computations reliably thanks to at-least-once guarantees, (2) nontrivial analytics (e.g., windowing and joins) on multiple data sources, and (3) efficient and cost-effective executions in production through code generation and elastic scaling.
Kafka Summit NYC 2017 - Data Processing at LinkedIn with Apache Kafka
LinkedIn uses Apache Kafka extensively to power various data pipelines and platforms. Some key uses of Kafka include:
1) Moving data between systems for monitoring, metrics, search indexing, and more.
2) Powering the Pinot real-time analytics query engine which handles billions of documents and queries per day.
3) Enabling replication and partitioning for the Espresso NoSQL data store using a Kafka-based approach.
4) Streaming data processing using Samza to handle workflows like user profile evaluation. Samza is used for both stateless and stateful stream processing at LinkedIn.
Going from three nines to four nines using Kafka | Tejas Chopra, Netflix
The document discusses using Apache Kafka to improve data upload availability from 99.9% to 99.99% when moving data between on-premise and cloud storage. It describes using Kafka to trigger uploads to the cloud from on-premise storage with 99.9% availability and using Kafka to split uploads between cloud and on-premise storage as well as rehydrating failed on-premise uploads from the cloud to achieve 99.99% availability. The presentation concludes that Kafka provides high throughput and persistence needed to design effective data rehydration strategies across cloud and on-premise storage for very high availability.
(BDT318) How Netflix Handles Up To 8 Million Events Per Second
In this session, Netflix provides an overview of Keystone, their new data pipeline. The session covers how Netflix migrated from Suro to Keystone, including the reasons behind the transition and the challenges of zero loss while processing over 400 billion events daily. The session covers in detail how they deploy, operate, and scale Kafka, Samza, Docker, and Apache Mesos in AWS to manage 8 million events & 17 GB per second during peak.
Deploying Kafka Streams Applications with Docker and Kubernetes
Speakers: Gwen Shapira, Principal Data Architect, Confluent + Matthias J. Sax, Software Engineer, Confluent
Apache Kafka and KSQL in Action: Let's Build a Streaming Data Pipeline!
This document summarizes Mic Hussey's presentation on building streaming data pipelines using Apache Kafka and KSQL. The presentation introduced Kafka and KSQL, demonstrated how to integrate PostgreSQL with Kafka using Kafka Connect, and showed examples of using KSQL to perform streaming ETL operations like joining, filtering, and aggregating event streams in real-time. It also provided several useful links for learning more about change data capture from PostgreSQL to Kafka and building streaming data pipelines with Kafka and KSQL.
Jolt: Distributed, fault-tolerant test running at scale using Mesos
This document summarizes Yelp's new test parallelization system called Jolt. It discusses how Jolt bundles tests into groups to run in parallel on Mesos clusters. This improves upon their previous system called Seagull which had complex custom components. Jolt uses Task Processing to interface with Mesos in a composable way. The document also describes Yelp's autoscaling system called Clusterman which adjusts Mesos cluster sizes based on resource usage and spot instance pricing. It notes how these systems have significantly reduced Yelp's infrastructure costs while improving their ability to run tests and deploy code.
From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...
Mark Teehan, Principal Solutions Engineer, Confluent
Use the Debezium CDC connector to capture database changes from a Postgres database - or MySQL or Oracle; streaming into Kafka topics and onwards to an external data store. Examine how to setup this pipeline using Docker Compose and Confluent Cloud; and how to use various payload formats, such as avro, protobuf and json-schema.
https://www.meetup.com/Singapore-Kafka-Meetup/events/276822852/
Unlocking the world of stream processing with KSQL, the streaming SQL engine ...
Slides of my Strata London 2018 talk:
https://conferences.oreilly.com/strata/strata-eu/public/schedule/detail/65325
Abstract:
Modern businesses have data at their core, and this data is changing continuously. Stream processing is what allows you harness this torrent of information in real time, and thousands of companies use Apache Kafka as the core platform for streaming data to transform and reshape their industries. However, the world of stream processing still has a very high barrier to entry. Today’s most popular stream processing technologies require the user to write code in programming languages such as Java or Scala. This hard requirement on coding skills is preventing many companies to unlock the benefits of stream processing to their full effect.
However, imagine that instead of having to write a lot of code in a programming language like Java or Scala for your favorite stream processing technology, all you’d need to get started with stream processing is a simple SQL statement, such as: SELECT * FROM payments-kafka-stream WHERE fraudProbability > 0.8.
Michael Noll offers an overview of KSQL, the open source streaming SQL engine for Apache Kafka, which makes it easy to get started with a wide range of real-time use cases, such as monitoring application behavior and infrastructure, detecting anomalies and fraudulent activities in data feeds, and real-time ETL. With KSQL, there’s no need to write any code in a programming language. KSQL brings together the worlds of streams and databases by allowing you to work with your data in a stream and in a table format. Built on top of Kafka’s Streams API, KSQL supports many powerful operations, including filtering, transformations, aggregations, joins, windowing, sessionization, and much more. It is open source (Apache 2.0 licensed), distributed, scalable, fault tolerant, and real time. You’ll learn how KSQL makes it easy to get started with a wide range of stream processing use cases and how to get up and running as you explore how it all works under the hood.
Kafka Summit NYC 2017 - Stream it Together: 3 Realities of Modern Programming
The document discusses 3 realities of modern programming: 1) The rise of managed services where over half of Kafka users are using cloud versions. 2) Data is exploding in volume and streaming is needed. 3) Microservices have increased in popularity but communication between services can be complex; Kafka helps solve this as a backbone. Yelp moved to microservices and uses Kafka to connect over 70 services, saving $10M.
Change Data Capture - Scale by the Bay 2019
The document discusses change data capture (CDC) in distributed systems. CDC allows capturing changes to data and notifying actors so they can react accordingly. Some considerations for CDC include reliability, scalability, performance, consistency, and fault tolerance. Common strategies involve using timestamps, triggers, diffs, or log scraping. CDC has various use cases such as caching, search indexing, offline processing, and simplifying monolithic applications. Popular frameworks for implementing CDC include pg2k4j, Maxwell, AWS Database Migration Service, SpinalTap, DataBus, and Debezium.
What's hot (20)
goto; London: Keeping your Cloud Footprint in Check
goto; London: Keeping your Cloud Footprint in Check
Maximizing Audience Engagement in Media Delivery (MED303) | AWS re:Invent 2013
Maximizing Audience Engagement in Media Delivery (MED303) | AWS re:Invent 2013
Putting Kafka Together with the Best of Google Cloud Platform
Putting Kafka Together with the Best of Google Cloud Platform
#lspe Q1 2013 dynamically scaling netflix in the cloud
#lspe Q1 2013 dynamically scaling netflix in the cloud
Surge 2013: Maximizing Scalability, Resiliency, and Engineering Velocity in t...
Surge 2013: Maximizing Scalability, Resiliency, and Engineering Velocity in t...
Kafka Summit NYC 2017 - Scalable Real-Time Complex Event Processing @ Uber
Kafka Summit NYC 2017 - Scalable Real-Time Complex Event Processing @ Uber
Flink Forward SF 2017: Bill Liu & Haohui Mai - AthenaX : Uber’s streaming pro...
Flink Forward SF 2017: Bill Liu & Haohui Mai - AthenaX : Uber’s streaming pro...
Kafka Summit NYC 2017 - Data Processing at LinkedIn with Apache Kafka
Kafka Summit NYC 2017 - Data Processing at LinkedIn with Apache Kafka
Going from three nines to four nines using Kafka | Tejas Chopra, Netflix
Going from three nines to four nines using Kafka | Tejas Chopra, Netflix
(BDT318) How Netflix Handles Up To 8 Million Events Per Second
(BDT318) How Netflix Handles Up To 8 Million Events Per Second
Deploying Kafka Streams Applications with Docker and Kubernetes
Deploying Kafka Streams Applications with Docker and Kubernetes
Apache Kafka and KSQL in Action: Let's Build a Streaming Data Pipeline!
Apache Kafka and KSQL in Action: Let's Build a Streaming Data Pipeline!
Jolt: Distributed, fault-tolerant test running at scale using Mesos
Jolt: Distributed, fault-tolerant test running at scale using Mesos
From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...
From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...
Unlocking the world of stream processing with KSQL, the streaming SQL engine ...
Unlocking the world of stream processing with KSQL, the streaming SQL engine ...
Kafka Summit NYC 2017 - Stream it Together: 3 Realities of Modern Programming
Kafka Summit NYC 2017 - Stream it Together: 3 Realities of Modern Programming
Viewers also liked
Docker & PHP - Practical use case
A quick intro to docker presented at the PHP Dublin meetup, and a practical use case of using docker to run multiple PHP versions while doing software development.
Engine lab software hybrid cloud specialists
Engine Lab is a hybrid cloud integration specialist based in Ireland. They securely integrate existing in-house SAP or Oracle applications with cloud applications. Their process involves scoping the business requirements, designing the integration architecture, and delivering the integration solution using Azure Biztalk or bespoke .NET services to interface in-house systems with cloud APIs or bespoke applications while keeping sensitive data in-house.
component based softwrae engineering Cbse
Component-based software engineering involves assembling systems out of reusable software components. A component model defines standards for component properties, interfaces, and composition. Developing components separately from systems allows components to be reused across applications. Architectural designs organize how components fit and work together to meet quality requirements.
Docker for PHP Developers - ZendCon 2016
This document provides an overview of Docker for PHP developers. It begins with definitions of containers and how they differ from virtual machines. It then demonstrates how to run simple PHP containers with Docker, including linking containers to share data and network resources. The document discusses volumes, networks, and building custom Docker images. It promotes Docker Compose for defining and running multi-container apps. Overall, the summary introduces Docker concepts and shows how it can benefit PHP development workflows.
2013 Social Admissions Report
The document discusses findings from a 2013 survey of over 4,000 high school students regarding their use of social media, particularly in researching and selecting colleges. Some key findings include:
- 70% of students access social media on mobile devices, and 87% of those students access school websites on mobile as well. Instagram and Pinterest saw a 50% increase in usage from 2012.
- 72% of students used social media to research colleges, and 33% found it to be influential in their college search process.
- Over three-quarters of students think colleges should have a social media presence to connect with prospective students.
NTR Lab - bespoke software development in Russia
Seeking new business opportunities? NTR Lab can develop high-quality software at reasonable cost to make your idea a reality.
Information Design Web Planning Mockup
course work. project to present style guide and mockup to fictional client as a proposal to develop a website.
Especialidade de inclusão 5
Este documento descreve um projeto de especialidade de inclusão realizado por Simone Helen Drumond sobre autismo e educação. O projeto envolveu discutir com um examinador um documento produzido pela União dos Escoteiros do Brasil sobre como lidar com deficiências e registrar anotações, fazer um desenho ou história em quadrinhos, ou produzir um cartaz sobre a discussão.
Git Workshop : Getting Started
This document provides an overview of Git and how to set it up for first use. It discusses the basics of Git including its distributed nature, snapshots instead of differences, and the three main states. It also covers installing Git, configuring user settings like name, email and tools, and getting help. The summary recommends having an understanding of Git and how it differs from CVCS, a working Git setup with identity, and being ready to learn basic Git commands.
Introduction To Git Workshop
This document provides an introduction to using Git. It covers getting Git, creating repositories, staging and committing files, branching, merging, and pushing and pulling changes. The presenter provides exercises for attendees to practice the basic Git commands and workflows. They discuss normal repositories, bare repositories, cloning repositories, viewing logs and commits, configuring user information, amending commits, removing files, branching, merging, pushing changes to a remote repository, and pulling changes from remote.
Spm file33
This document contains a software project management lab file submitted by Poonam Singh. It includes an index listing activities completed with dates and comments. The file analyzes different software design tools, provides a software requirements specification for a mini online test simulator project, estimates the effort and cost required to build the software using estimation techniques, includes entity relationship diagrams and data flow diagrams at different levels, designs a data dictionary, and compares project life cycles for different types of projects. Technologies used include HTML, Java, Apache Tomcat server, and Microsoft Access database with SQL querying.
MockupBuilder
Mockup Builder is a software solution that helps design software and website mockups to reduce time and costs during development. It addresses issues like uncertainties in early design phases, functional requirement errors, and project stakeholder misunderstandings. The cloud-based tool offers over 120 UI elements, templates, collaboration, and export options. Its target audience includes software companies, designers, and individuals. With annual revenues of top competitors ranging from $1.6M to $3.9M, the mockup tools market is estimated at $13M and growing. Mockup Builder plans to generate income from standalone, enterprise, and subscription licensing while expanding through investments and new markets.
Docker for Developers - PNWPHP 2016 Workshop
This document provides an overview of Docker for PHP developers. It discusses what Docker is, how containers work, and how to use basic Docker commands like docker run, docker exec, and docker volume to build and run PHP applications in containers. It also covers more advanced topics like linking containers together, building custom Docker images, and using Docker Compose to define and run multi-container apps. The goal is to help PHP developers learn how to containerize their applications and workflows using Docker.
Computer-free Website Development Demo - WordPressDC Jan 2015
As a follow-up to a blog post I wrote on the emergence of the computer-free designer, I gave a live demo to build, customize, and launch a WordPress site entirely from my phone, without need of a traditional computer. These were my backup slides in case the live demo wasn't possible. They contain my Terminal commands and the free apps I used.
Php development with Docker
Michael Bui has been developing LAMP stack applications since 2004 and professionally since 2008. He enjoys programming, learning new technologies, and building things. In his free time he likes photography, chess, traveling, hiking, and sports. He is a father. The document discusses using Docker for PHP development which allows developers to build, ship, and run applications on Linux, Windows, and Mac operating systems. Docker provides a way to run, test, debug, and deploy PHP applications without needing a full virtual machine.
The App Evolution
The latest changes in the app development processs. Presented by IBM Executive Architect, Steve Mirman.
Lab docker
This document provides instructions for a lab on using Docker to install and run containers. The objectives are to install Docker, create images and containers, launch applications in containers, and store and access data in containers. It outlines setting up Docker on Ubuntu, pulling existing images like Fedora and running containers from them. Specific steps look at running the "hello-world" container, installing wget in a Fedora container, and persisting data. The last section provides instructions for building a Docker image to run the OwnCloud application in a container, addressing aspects like installing the application, configuring network access, and persisting data and configuration.
Building Next Generation Applications and Microservices
Presented by Manuel Silveyra
Presentation reviewing the progression to microservices, microservices keys to success, agile deployment, and more.
An introduction to contianers and Docker for PHP developers
This document provides an introduction to containers and Docker for PHP developers. It defines Linux containers and their key components like namespaces and control groups. It explains that Docker is a container technology that packages applications and dependencies into standardized units. Docker Compose is introduced as a tool to define and run multi-container applications with Docker. Vagrant is described as a way to provide a Linux Docker host and shared folders. The document concludes by noting Docker's focus on container deployment and providing references for further information.
Git and Github workshop
Presentation held at Startup.bg on 2015-11-06 about Git and Github for beginners (and something for advanced too)
Viewers also liked (20)
Computer-free Website Development Demo - WordPressDC Jan 2015
Computer-free Website Development Demo - WordPressDC Jan 2015
Building Next Generation Applications and Microservices
Building Next Generation Applications and Microservices
An introduction to contianers and Docker for PHP developers
An introduction to contianers and Docker for PHP developers
Similar to Microservices without Servers
Flexible and Real-Time Stream Processing with Apache Flink
This document provides an overview of stream processing with Apache Flink. It discusses the rise of stream processing and how it enables low-latency applications and real-time analysis. It then describes Flink's stream processing capabilities, including pipelining of data, fault tolerance through checkpointing and recovery, and integration with batch processing. The document also summarizes Flink's programming model, state management, and roadmap for further development.
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)
Learn the differences between an event-driven streaming platform and middleware like MQ, ETL and ESBs – including best practices and anti-patterns, but also how these concepts and tools complement each other in an enterprise architecture.
Extract-Transform-Load (ETL) is still a widely-used pattern to move data between different systems via batch processing. Due to its challenges in today’s world where real time is the new standard, an Enterprise Service Bus (ESB) is used in many enterprises as integration backbone between any kind of microservice, legacy application or cloud service to move data via SOAP / REST Web Services or other technologies. Stream Processing is often added as its own component in the enterprise architecture for correlation of different events to implement contextual rules and stateful analytics. Using all these components introduces challenges and complexities in development and operations.
This session discusses how teams in different industries solve these challenges by building a native streaming platform from the ground up instead of using ETL and ESB tools in their architecture. This allows to build and deploy independent, mission-critical streaming real time application and microservices. The architecture leverages distributed processing and fault-tolerance with fast failover, no-downtime rolling deployments and the ability to reprocess events, so you can recalculate output when your code changes. Integration and Stream Processing are still key functionality but can be realized in real time natively instead of using additional ETL, ESB or Stream Processing tools.
Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...
Apache Kafka can act as both an enemy and a friend to traditional middleware like message queues, ETL tools, and enterprise service buses. As an enemy, Kafka replaces many of the individual components and limitations of traditional middleware with a single, scalable event streaming platform. However, Kafka can also integrate with traditional middleware as a friend through connectors and client APIs, using traditional tools for specific integrations while relying on Kafka for scalable event collection and processing. In complex environments with both new and legacy systems, Kafka acts as a "frenemy" by facilitating a gradual migration from old middleware to a modern event streaming architecture centered around Kafka.
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
Apache Kafka can act as both an enemy and a friend to traditional middleware like message queues, ETL tools, and enterprise service buses. As an enemy, Kafka replaces many of the individual components and provides a single scalable platform for messaging, storage, and processing. However, Kafka can also integrate with traditional middleware as a friend through connectors and client APIs, allowing certain use cases to still leverage existing tools. In complex environments with both new and legacy systems, Kafka acts as a "frenemy" - replacing some functions but integrating with other existing technologies to provide a bridge to new architectures.
Mininet: Moving Forward
The document discusses Mininet, an open source network emulator used for testing SDN ideas. It provides an overview of Mininet 1.0 and its functional fidelity before describing plans for Mininet 2.0 to improve performance fidelity through techniques like resource isolation, network invariants, and reproducible experiments. The document uses the example of DCTCP traffic to demonstrate how network invariants can validate emulator results.
The Evolution of Trillion-level Real-time Messaging System in BIGO - Puslar ...
This document discusses BIGO's evolution from using open-source Kafka to Apache Pulsar for its real-time messaging system. It describes the challenges BIGO faced with Kafka as data scales rapidly grew, including poor scalability and degraded I/O performance. BIGO chose Pulsar for its lightweight horizontal scalability, excellent read-write isolation, and ability to support over a million topics. Typical application scenarios discussed include high throughput event tracking, lightweight traffic balancing, and high performance catch-up reads for machine learning tasks. Future work may involve optimizations for different read/write models and combining SSD and HDD storage.
Concepts and Patterns for Streaming Services with Kafka
Cloud Native Night March 2020, Mainz: Talk by Perry Krol (@perkrol, Confluent)
=== Please download slides if blurred! ===
Abstract: Proven approaches such as service-oriented and event-driven architectures are joined by newer techniques such as microservices, reactive architectures, DevOps, and stream processing. Many of these patterns are successful by themselves, but they provide a more holistic and compelling approach when applied together. In this session Confluent will provide insights how service-based architectures and stream processing tools such as Apache Kafka® can help you build business-critical systems. You will learn why streaming beats request-response based architectures in complex, contemporary use cases, and explain why replayable logs such as Kafka provide a backbone for both service communication and shared datasets.
Based on these principles, we will explore how event collaboration and event sourcing patterns increase safety and recoverability with functional, event-driven approaches, apply patterns including Event Sourcing and CQRS, and how to build multi-team systems with microservices and SOA using patterns such as “inside out databases” and “event streams as a source of truth”.
Learnings From Shipping 1000+ Streaming Data Pipelines To Production with Hak...
"Kafka Connect and Kafka Streams are foundational technologies in modern, real-time data architectures. They enable developers to build scalable, robust, and real-time data pipelines without having to handle the low-level consumer and producer APIs of Apache Kafka®. In this talk, we share our most important, and often surprising learnings from using Kafka Connect and Kafka Streams to ship more than 1,000 streaming data pipelines to production. The goal of this talk is to enable you to build mature streaming data pipelines without having to go through the common pitfalls.
We walk you through our journey of adopting Apache Kafka®, Kafka Connect, and Kafka Streams. We discuss the challenges that we faced and how we overcame them. Over the course of the talk, we provide answers to important questions, such as: Which metrics are useful for monitoring streaming data pipelines? How to deal with resource-leaking connectors impacting the health of a Kafka Connect cluster? How to start troubleshooting the performance of streaming data pipelines? How to tune Kafka Connect for handling slow data sources or data sinks? What’s missing in today’s ecosystem for streaming to become a commodity?"
OpenWhisk: Where Did My Servers Go?
This document discusses serverless computing and the OpenWhisk platform. It describes how OpenWhisk allows developers to build event-driven applications without managing servers. OpenWhisk provides a programming model based on actions that are triggered by events to execute code without worrying about scaling. It also offers an open source implementation that can run locally or on IBM Bluemix and supports various use cases like serverless apps, IoT, and chatbots.
Data Stream Analytics - Why they are important
Streaming is cool and it can help us do quick analytics and make profit but what about tsunamis? This is a motivation talk presented at the SeRC Big Data Workshop in Sweden during spring 2016. It motivates the streaming paradigm and provides examples on Apache Flink.
Data analytics at scale implementing stateful stream processing - publish
This document discusses stateful stream processing and provides an overview of streaming frameworks. It describes why streaming analytics are important for processing continuous, high-velocity data streams with low latency requirements. It then covers streaming basics and examples of building applications with Apache Flink and Kafka Streams. Key aspects like time windows, high availability, and delivery guarantees are also addressed.
Production Ready Serverless Java Applications in 3 Weeks AWS UG Cologne Febru...
Production-ready Serverless Java Applications in 3 weeks" at AWS User Group Cologne 221.02.2019 by Vadym Kazulkin and Elmar Warken
Talking about lessons learned of Lambda, API Gateway, X-Ray and Serverless databases (both DynamoDB and Aurora Serverless), including some new topics such as DynamoDB On Demand and Aurora Serverless Data API
Максим Барышиков-«WoT: Geographically distributed cluster of clusters»
"World of Tanks — MMOG с зарегистрированным пиковым онлайном превышающем 1 миллион пользователей. Авторитарная архитектура сервера (весь расчет физического мира происходит на стороне сервера и затем транслируется подклчюенным клиентам) позволяет эффективно защититься от нечестной игры, но вместе с тем создает значительную дополнительную нагрузку на серверную инфраструктуру.
В докладе рассказывается об инженерных решениях, принятых на разных стадиях разработки игры, их последствиях, и о том, как в конечном итоге организован географически распределенный «кластер кластеров» World of Tanks."
Introduction to apache kafka, confluent and why they matter
This is a short and introductory presentation on Apache Kafka (including Kafka Connect APIs, Kafka Streams APIs, both part of Apache Kafka) and other open source components part of the Confluent platform (such as KSQL).
This was the first Kafka Meetup in South Africa.
Strata Singapore: Gearpump Real time DAG-Processing with Akka at Scale
Gearpump is a Akka based realtime streaming engine, it use Actor to model everything. It has super performance and flexibility. It has performance of 18000000 messages/second and latency of 8ms on a cluster of 4 machines.
Best Practices for Genomic and Bioinformatics Analysis Pipelines on AWS
AWS is a great fit for both steady state and episodic computational workloads. Here we present some common architecture patterns for analyzing genomic and other biomedical data on scalable high-throughput computational clusters on AWS. This talk will cover bootstrapping a traditional Beowulf compute cluster on AWS EC2, data transfer and storage strategies for S3.
Unleashing Apache Kafka and TensorFlow in the Cloud
How can you leverage the flexibility and extreme scale in the public cloud combined with your Apache Kafka ecosystem to build scalable, mission-critical machine learning infrastructures, which span multiple public clouds or bridge your on-premise data centre to cloud?
This talk will discuss and demo how you can leverage machine learning technologies such as TensorFlow with your Kafka deployments in public cloud to build a scalable, mission-critical machine learning infrastructure for data preprocessing and ingestion, and model training, deployment and monitoring.
The discussed architecture includes capabilities like scalable data preprocessing for training and predictions, combination of different Deep Learning frameworks, data replication between data centres, intelligent real time microservices running on Kubernetes, and local deployment of analytic models for offline predictions.
Deep Learning UDF for KSQL for Streaming Anomaly Detection of MQTT IoT Sensor Data.:
I built a KSQL UDF for sensor analytics. It leverages the new API features of KSQL to build UDF / UDAF functions easily with Java to do continuous stream processing on incoming events.
Use Case: Connected Cars - Real Time Streaming Analytics using Deep Learning
Continuously process millions of events from connected devices (sensors of cars in this example).
Apache Flink(tm) - A Next-Generation Stream Processor
In diesem Vortrag wird es zunächst einen kurzen Überblick über den aktuellen Stand im Bereich der Streaming-Datenanalyse geben. Danach wird es mit einer kleinen Einführung in das Apache-Flink-System zur Echtzeit-Datenanalyse weitergehen, bevor wir tiefer in einige der interessanten Eigenschaften eintauchen werden, die Flink von den anderen Spielern in diesem Bereich unterscheidet. Dazu werden wir beispielhafte Anwendungsfälle betrachten, die entweder direkt von Nutzern stammen oder auf unserer Erfahrung mit Nutzern basieren. Spezielle Eigenschaften, die wir betrachten werden, sind beispielsweise die Unterstützung für die Zerlegung von Events in einzelnen Sessions basierend auf der Zeit, zu der ein Ereignis passierte (event-time), Bestimmung von Zeitpunkten zum jeweiligen Speichern des Zustands eines Streaming-Programms für spätere Neustarts, die effiziente Abwicklung bei sehr großen zustandsorientierten Streaming-Berechnungen und die Zugänglichkeit des Zustandes von außerhalb.
All Streams Ahead! ksqlDB Workshop ANZ
Watch the available webcast: https://videos.confluent.io/watch/Jbb3PUbgDFck6jLcCkVvur?
Speaker: David Peterson
JHipster conf 2019 - Kafka Ecosystem
JHipster and Kafka Provide a great set of combinations of properties for the Modern Digital Enterprise!
Similar to Microservices without Servers (20)
Flexible and Real-Time Stream Processing with Apache Flink
Flexible and Real-Time Stream Processing with Apache Flink
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)
Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...
Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
The Evolution of Trillion-level Real-time Messaging System in BIGO - Puslar ...
The Evolution of Trillion-level Real-time Messaging System in BIGO - Puslar ...
Concepts and Patterns for Streaming Services with Kafka
Concepts and Patterns for Streaming Services with Kafka
Learnings From Shipping 1000+ Streaming Data Pipelines To Production with Hak...
Learnings From Shipping 1000+ Streaming Data Pipelines To Production with Hak...
Data analytics at scale implementing stateful stream processing - publish
Data analytics at scale implementing stateful stream processing - publish
Production Ready Serverless Java Applications in 3 Weeks AWS UG Cologne Febru...
Production Ready Serverless Java Applications in 3 Weeks AWS UG Cologne Febru...
Максим Барышиков-«WoT: Geographically distributed cluster of clusters»
Максим Барышиков-«WoT: Geographically distributed cluster of clusters»
Introduction to apache kafka, confluent and why they matter
Introduction to apache kafka, confluent and why they matter
Strata Singapore: Gearpump Real time DAG-Processing with Akka at Scale
Strata Singapore: Gearpump Real time DAG-Processing with Akka at Scale
Best Practices for Genomic and Bioinformatics Analysis Pipelines on AWS
Best Practices for Genomic and Bioinformatics Analysis Pipelines on AWS
Unleashing Apache Kafka and TensorFlow in the Cloud
Unleashing Apache Kafka and TensorFlow in the Cloud
Apache Flink(tm) - A Next-Generation Stream Processor
Apache Flink(tm) - A Next-Generation Stream Processor
More from Dev_Events
Eclipse OMR: a modern, open-source toolkit for building language runtimes
Daryl Maier, Compiler Development Performance, IBM
Cloud, @0xdaryl
Java runtime technology has benefited from hundreds
of person years of development investment over the
last 20 years, resulting in a highly capable, high
performance and scalable dynamic language with a
vibrant developer ecosystem. The recently created OMR incubator project at Eclipse
(https://github.com/eclipse/omr) aims to expand access to high quality runtime technology for other
dynamic languages through an ongoing effort to restructure the core components of the J9 Java Virtual
Machine. Rather than building new languages on top of Java, however, this project aims to unlock the
inner workings of the JVM without imposing Java semantics to create a platform for building highly
capable language runtimes. This talk will introduce developers to the Eclipse OMR project: what it is, the
runtime technology that has been contributed so far, promising projects that leverage this technology,
active areas of development, and how developers can get involved.
Eclipse MicroProfile: Accelerating the adoption of Java Microservices
Emily Jiang, WebSphere Application Server, CDI Development Lead, IBM Cloud @emilyfhjiang
While there are likely a dozen or more Java-based microservice / cloud native initiatives in the industry,
Eclipse MicroProfile provides a platform for Java developers to collaborate and innovate on areas of
common interest. This BOF will give a short overview of MicroProfile and how it plans to optimize
microservices for Enterprise Java, followed by a short Q&A to answer any clarifying questions, and the
remainder of the time spent on collaborating to prioritize features for the next release.
While MicroProfile's first release is founded in Java EE (CDI + JSON-P + JAX-RS), it is not intended to
remain a subset of Java EE 7 (or 8). The intent is to collaborate and innovate as a community in the
context of a microservices architecture across the 5 (and growing) MicroProfile implementations today.
The end goal of MicroProfile is to feed the Java Community Process (JCP) with JSR submissions backed
by well-thought-out concepts and even implementations that developers and enterprises can rely on.
Come collaborate with your fellow MicroProfilers to move Enterprise Java microservices forward.
From Science Fiction to Science Fact: How AI Will Change Our Approach to Buil...
Rama Akkiraju, Distinguished Engineer, Master Inventor, IBM Watson User Technologies
IBM Watson and Cloud Platform
We have entered a new period of computing history — a cognitive era. For decades, science fiction visionaries have shared their renditions of intelligent machines and computers that could learn and function as humans. Intelligent machines have moved beyond the lore of science fiction; today, they are a reality thanks to breakthroughs in AI and machine learning. So how will this impact and change the apps that we build and how we leverage new and existing data?
In this interactive session, we’ll explore those questions and:
• Discuss where AI currently is and where the technology is going
• Show how pioneering apps and developers around the world are already using it
• Provide access to sample code and applications which you and your teams can use to understand and explore cognitive computing for yourself
Blockchain Hyperledger Lab
The document provides instructions for getting started with Hyperledger Fabric blockchain technology. It covers three parts: 1) Using Blockchain as a Service on Bluemix cloud and starting a network locally, 2) Creating a smart contract in Java, and 3) Developing the client-side application. For part one, it explains how to deploy a blockchain network on Bluemix and also how to start a local network using Docker Compose with 4 peers, 1 CA, and sample configuration files. Part two describes how to build a Java smart contract using Eclipse that implements a crop insurance agreement between a farmer and insurer. Part three will cover initializing, deploying, querying and testing the client application.
Introduction to Blockchain and Hyperledger
Nitesh Thakrar, IT Software Architect,
IBM @niteshpthakrar and Benjamin Fuentes, Software
Architect and Developer, IBM, @benji_fuentes
This workshop will be in 3 stages:
1. A brief presentation on Blockchain and why
Hyperledger
2. A demo use case to explain the architecture and the code behind the demo
3. Finally, the attendees will create their own blockchain application on the cloud. The hands-on
will also invite them to use the appropriate APIs and event update a smart contract.Majority of
the time will be in doing the hands-on (step 3) so that the attendees are able to continue
developing their application after the event.Requirements: Attendees will need to bring their
laptops and be able to connect to wifi.
Using GPUs to Achieve Massive Parallelism in Java 8
Adam Roberts, IBM Spark Team Lead – Runtimes, IBM Cloud
Graphic processing units (GPUs) are not limited to traditional scene rendering tasks. They can play a
huge role in accelerating applications that have a large number of parallelizable tasks.
Learn how Java can exploit the power of GPUs to optimize high-performance enterprise and technical
computing applications such as big data and analytics workloads, through both explicit GPU
programming and letting the Java JIT compiler transparently off-load work to the GPU.
This presentation covers the principles and considerations for GPU programming from Java and looks at
the software stack and developer tools available. After this talk you will be ready to extract the full
power of GPUs from your own application. We will present a demo showing GPU acceleration and
discuss what is coming in the future.
Lean and Easy IoT Applications with OSGi and Eclipse Concierge
Jan Rellemeyer, Research Staff Member, IBM Research, @rellermeyer
Modularization of software is key to handling the inherent complexity of distributed applications like for
the Internet of Things (IoT) and provide a flexible environment to evolve applications and manage their
deployment effectively. OSGi is a popular framework for dynamic modules for the Java language. Eclipse
Concierge provides a clean, small and lightweight implementation of the OSGi core framework
specification, specifically tailored to embedded systems and IoT.
Eclipse JDT Embraces Java 9 – An Insider’s View
This document discusses Eclipse JDT's support for Java 9 features. It provides an overview of key Java 9 changes like modules and modular runtime images. It also summarizes changes to Eclipse/JDT's compiler, IDE, and architecture to support modules and other Java 9 features. The presentation includes a demo of Eclipse's Java 9 module support and error reporting for modules.
Node.js – ask us anything!
Sam Roberts, Team Lead, StrongLoop and API Connect, IBM @octetcloud and Michael Dawson, Software
Developer, IBM @mhdawson1
You’ll have some captive Node.js/V8 collaborators, and you can ask them anything! Confused about the
LTS plans, wondering what is happening in the Node workgroups, want to know what is really going to
happen with Webworkers, what’s the future of Promises in Node.js? Wondering how to debug your
production failures? Ask us anything, and we’ll try and answer!
Swift on the Server
Robert Dickerson, Software Developer, IBM-Swift, @rfdickerson
Of course its allowed - just matters how you do it. In this part of the workshop we’ll start with
serverless: New cloud programming models enabled by serverless architectures are emerging, allowing
developers to focus more sharply on creating their applications and less on managing their
infrastructure. The OpenWhisk project started by IBM and now part of Apache provides an open source
platform to enable these cloud native, event-driven applications. In this talk, we will provide an
overview of serverless architectures, introduce the OpenWhisk programming model, and then show
how to deploy an OpenWhisk application on IBM Bluemix OpenWhisk.
Then we’ll get Swift! It’s not only a language you turn to to build your iOS application, but it can also be
the language you can use to build your backend. This part of the workshop will use a text-based
adventure game called "GameOn" as an example for building a Server-side Swift App.
Topics include:
• How to create a web service using the IBM Kitura web framework • How to integrate the Watson SDK
for Swift into your application • How to dockerize your Swift application and deploy it to Bluemix
Being serverless and Swift... Is that allowed?
This document discusses Apache OpenWhisk, a serverless platform. It provides an overview of OpenWhisk and how it works, including its programming model using triggers, actions, rules and packages. It describes OpenWhisk's backend architecture using Docker containers and Kubernetes. It also discusses how OpenWhisk can be deployed on multiple clouds and IBM is working on improving its support for Kubernetes. Finally, it announces job openings and provides a link to a GitHub repository for an OpenWhisk CLI demo.
Secrets of building a debuggable runtime: Learn how language implementors sol...
Bjørn Vårdal, J9VM Software Developer, IBM, @bvaardal
New language runtimes appear all the time, but most of them die young. Failure can be attributed to
different reasons, but an important factor is that lack of support can limit the community’s and
industry’s willingness to adopt the new language.
Quicker development and improved serviceability allows emerging languages to overcome this obstacle.
By building on the proven technology available in Eclipse OMR, language developers can get more than
performance and stability; you also get tools that help you quickly debug your language runtime,
allowing you to provide competitive serviceability.
From this presentation, you will learn how to enable Eclipse OMR’s mature debugging features in your
language runtime, and also how Eclipse OMR can assist with development and debugging
Tools in Action: Transforming everyday objects with the power of deeplearning...
Marek Sadowski, Developer Advocate, IBM, @blumareks
With cognitive computing, we are now able to interact with everyday objects in ways that were deemed
impossible. In this session, we will demonstrate the power of cognitive Internet of Things and the Cloud
platform to securely connect and manage devices, analyze data, and apply cognitive services to add
human-like interaction.
Leveraging cognitive services for sentiment, tone, and visual analysis invoked from IoT/robotics
platforms and Voice UI interface we will show how you can quickly and securely turn a simple idea into
reality by transforming a regular robot vacuum into a cognitive-enabled smart device.
The App Evolution
The latest changes in the app development processs. Presented by IBM Executive Architect, Steve Mirman.
Create and Manage APIs with API Connect, Swagger and Bluemix
Presented by - Raghavan “Rags” Srinivas, Architect, IBM
Enabling other developers and organizations to use your APIs through their own applications and services provides a compelling system for innovation and monetization. The Swagger spec (v2.0), recently donated to the Open API Initiative (OAI), is part of an open source project for better creation and documentation of APIs. Companies are empowering developers via these initiatives to leverage the data and build apps around it. This hands-on session helps you get started with creating APIs for consumption by developers in a well-documented, secure, and easy-to-manage form.
OpenWhisk - Serverless Architecture
Serverless architectures are one of the hottest trends in cloud computing this year, and for good reason. There are several technical capabilities and business factors coming together to make this approach compelling from both an application development and deployment cost perspective. The new OpenWhisk project provides an open source platform to enable these cloud-native, event-driven applications.
This talk will lay out the technical and business drivers behind the rise of serverless architectures, provide an introduction to the OpenWhisk open source project (and describe how it differs from other services like AWS Lambda), and give a demonstration showing how to start developing with this new cloud computing model using the OpenWhisk implementation available on IBM Bluemix.
Lightning talk and lab presented by IBM Cloud Software Engineer, Andrew Bodine.
Add Custom Model and ORM to Node.js
This document describes adding custom data models and object relational mapping (ORM) functionality to a Node.js application using StrongLoop. It details creating a Video model, defining a one-to-many relationship between Video and User, adding access controls, and connecting the models to a PostgreSQL database for persistent storage. Key steps include generating a Video model, relating it to User, adding ACLs, connecting to a PostgreSQL data source, and using automigration to generate tables.
Adding User Management to Node.js
The document discusses adding user management functionality to a Node.js application called QAVideos. It involves the following steps:
1. Creating the application using StrongLoop's Loopback framework.
2. Adding Angular support to handle data binding between the client and server.
3. Adding signup and login functionality including signup and success pages, a signup controller, and an authentication service to register new users.
Creating Sentiment Line Chart with Watson
Hands-on-lab using Watson APIs to create a sentiment line chart based on positive and negative sentiment within the news.
Containers Lab
This document provides instructions for using IBM Bluemix and IBM Bluemix DevOps Services (IDS) to build and deploy a Docker container application. It demonstrates setting up a continuous delivery pipeline that will automatically build and deploy new versions of an application. Users are instructed to fork a sample project code from IDS, inspect the code, configure build and deploy stages in the pipeline, and then run builds to deploy new versions after making code changes. The pipeline is triggered by committing code changes to the Git repository.
More from Dev_Events (20)
Eclipse OMR: a modern, open-source toolkit for building language runtimes
Eclipse OMR: a modern, open-source toolkit for building language runtimes
Eclipse MicroProfile: Accelerating the adoption of Java Microservices
Eclipse MicroProfile: Accelerating the adoption of Java Microservices
From Science Fiction to Science Fact: How AI Will Change Our Approach to Buil...
From Science Fiction to Science Fact: How AI Will Change Our Approach to Buil...
Using GPUs to Achieve Massive Parallelism in Java 8
Using GPUs to Achieve Massive Parallelism in Java 8
Lean and Easy IoT Applications with OSGi and Eclipse Concierge
Lean and Easy IoT Applications with OSGi and Eclipse Concierge
Secrets of building a debuggable runtime: Learn how language implementors sol...
Secrets of building a debuggable runtime: Learn how language implementors sol...
Tools in Action: Transforming everyday objects with the power of deeplearning...
Tools in Action: Transforming everyday objects with the power of deeplearning...
Create and Manage APIs with API Connect, Swagger and Bluemix
Create and Manage APIs with API Connect, Swagger and Bluemix
Recently uploaded
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
As the digital landscape continually evolves, operating systems play a critical role in shaping user experiences and productivity. The launch of Nitrux Linux 3.5.0 marks a significant milestone, offering a robust alternative to traditional systems such as Windows 11. This article delves into the essence of Nitrux Linux 3.5.0, exploring its unique features, advantages, and how it stands as a compelling choice for both casual users and tech enthusiasts.
Removing Uninteresting Bytes in Software Fuzzing
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
20240605 QFM017 Machine Intelligence Reading List May 2024
Everything I found interesting about machines behaving intelligently during May 2024
UiPath Test Automation using UiPath Test Suite series, part 6
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
ABSTRACT: A prima vista, un mattoncino Lego e la backdoor XZ potrebbero avere in comune il fatto di essere entrambi blocchi di costruzione, o dipendenze di progetti creativi e software. La realtà è che un mattoncino Lego e il caso della backdoor XZ hanno molto di più di tutto ciò in comune.
Partecipate alla presentazione per immergervi in una storia di interoperabilità, standard e formati aperti, per poi discutere del ruolo importante che i contributori hanno in una comunità open source sostenibile.
BIO: Sostenitrice del software libero e dei formati standard e aperti. È stata un membro attivo dei progetti Fedora e openSUSE e ha co-fondato l'Associazione LibreItalia dove è stata coinvolta in diversi eventi, migrazioni e formazione relativi a LibreOffice. In precedenza ha lavorato a migrazioni e corsi di formazione su LibreOffice per diverse amministrazioni pubbliche e privati. Da gennaio 2020 lavora in SUSE come Software Release Engineer per Uyuni e SUSE Manager e quando non segue la sua passione per i computer e per Geeko coltiva la sua curiosità per l'astronomia (da cui deriva il suo nickname deneb_alpha).
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
Join us to introduce Milvus Lite, a vector database that can run on notebooks and laptops, share the same API with Milvus, and integrate with every popular GenAI framework. This webinar is perfect for developers seeking easy-to-use, well-integrated vector databases for their GenAI apps.
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank
Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Full-RAG: A modern architecture for hyper-personalization
Mike Del Balso, CEO & Co-Founder at Tecton, presents "Full RAG," a novel approach to AI recommendation systems, aiming to push beyond the limitations of traditional models through a deep integration of contextual insights and real-time data, leveraging the Retrieval-Augmented Generation architecture. This talk will outline Full RAG's potential to significantly enhance personalization, address engineering challenges such as data management and model training, and introduce data enrichment with reranking as a key solution. Attendees will gain crucial insights into the importance of hyperpersonalization in AI, the capabilities of Full RAG for advanced personalization, and strategies for managing complex data integrations for deploying cutting-edge AI solutions.
National Security Agency - NSA mobile device best practices
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
How to Get CNIC Information System with Paksim Ga.pptx
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
Maruthi Prithivirajan, Head of ASEAN & IN Solution Architecture, Neo4j
Get an inside look at the latest Neo4j innovations that enable relationship-driven intelligence at scale. Learn more about the newest cloud integrations and product enhancements that make Neo4j an essential choice for developers building apps with interconnected data and generative AI.
20 Comprehensive Checklist of Designing and Developing a Website
Dive into the world of Website Designing and Developing with Pixlogix! Looking to create a stunning online presence? Look no further! Our comprehensive checklist covers everything you need to know to craft a website that stands out. From user-friendly design to seamless functionality, we've got you covered. Don't miss out on this invaluable resource! Check out our checklist now at Pixlogix and start your journey towards a captivating online presence today.
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIVladimir Iglovikov, Ph.D.
Presented by Vladimir Iglovikov:
- https://www.linkedin.com/in/iglovikov/
- https://x.com/viglovikov
- https://www.instagram.com/ternaus/
This presentation delves into the journey of Albumentations.ai, a highly successful open-source library for data augmentation.
Created out of a necessity for superior performance in Kaggle competitions, Albumentations has grown to become a widely used tool among data scientists and machine learning practitioners.
This case study covers various aspects, including:
People: The contributors and community that have supported Albumentations.
Metrics: The success indicators such as downloads, daily active users, GitHub stars, and financial contributions.
Challenges: The hurdles in monetizing open-source projects and measuring user engagement.
Development Practices: Best practices for creating, maintaining, and scaling open-source libraries, including code hygiene, CI/CD, and fast iteration.
Community Building: Strategies for making adoption easy, iterating quickly, and fostering a vibrant, engaged community.
Marketing: Both online and offline marketing tactics, focusing on real, impactful interactions and collaborations.
Mental Health: Maintaining balance and not feeling pressured by user demands.
Key insights include the importance of automation, making the adoption process seamless, and leveraging offline interactions for marketing. The presentation also emphasizes the need for continuous small improvements and building a friendly, inclusive community that contributes to the project's growth.
Vladimir Iglovikov brings his extensive experience as a Kaggle Grandmaster, ex-Staff ML Engineer at Lyft, sharing valuable lessons and practical advice for anyone looking to enhance the adoption of their open-source projects.
Explore more about Albumentations and join the community at:
GitHub: https://github.com/albumentations-team/albumentations
Website: https://albumentations.ai/
LinkedIn: https://www.linkedin.com/company/100504475
Twitter: https://x.com/albumentationsRecently uploaded (20)
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Microservices without Servers
- 1. Microservices without Servers Jonathan Kaufman • IBM • QCon San Francisco • @kauffecup
- 2. Servers Are Killing Your Productivity
- 4. Hard Drive Fails Linux OS Vulnerability Middleware Version Update Upgrade Network Infrastructure More Memory and Bigger CPU DB Table Index Gets Corrupted Get Another Server…
- 5. Wasn’t the Cloud Supposed to Fix This? �
- 6. “Computing as a Utility” source: https://flic.kr/p/5aHJFh
- 9. average utilization 20% - 40% of capacity
- 10. What Can We Do?
- 11. AWS Lambda
- 12. …why is this different? “runs code in response to events, managing the computing resources”
- 13. functions as a service ƒ(x)
- 14. Scaling: 0 to N ƒ(x) ƒ(x) ƒ(x) ƒ(x) ƒ(x) ƒ(x) ƒ(x) ƒ(x) ƒ(x) ƒ(x) ƒ(x) ƒ(x) ƒ(x) ƒ(x) ƒ(x) ƒ(x) ƒ(x) ƒ(x) ƒ(x) ƒ(x) ƒ(x)
- 15. Pay-As-You-Go a loves dc, https://www.flickr.com/photos/alovesdc/3466740007, CC (BY) license
- 16. http://uk.businessinsider.com/amazon-web-services-lambda-explained-2015-11 "One happy lambda customer saving 80% off their cloud bills"
- 24. Enough Talking, Bring on the Code.
- 27. IBM Openwhisk
- 29. ➜ workspace: git clone https://github.com/openwhisk/openwhisk.git ➜ workspace: cd openwhisk ➜ openwhisk: vagrant up
- 34. my_service.js
- 35. function main(params) { // code code code return { ... }; } my_service.js Entry Point Event Parameters Service Result
- 36. import com.google.gson.JsonObject; public class Hello { public static JsonObject main(JsonObject args) { // code code code JsonObject response = new JsonObject(); // some more code return response; } } my_service.java Entry Point Event Parameters Service Result
- 37. Actions
- 38. REST
- 39. REST API
- 40. Triggers
- 42. Challenges
- 43. Compute Limits zund, https://www.flickr.com/photos/zund/12202485675, CC (BY-NC-ND) license
- 45. Monitoring, Debugging, and Testing Thomas Hawk, https://www.flickr.com/photos/thomashawk/4544013443, CC (BY-NC) license
- 47. Frameworks
- 48. Servers Are Killing Your Productivity
- 51. Ditch Your Servers. Upgrade Your Productivity.