The Future of Data Pipelines
•
0 likes•132 views
Presented at: All Things Open RTP Meetup Presented by: John Hammink, Aiven.io Abstract: With mobile devices and emerging IoT connected infrastructure and devices, we’re seeing the amount of generated data explode, while continuing to transform in form and function. With 16.1 zettabytes of data generated in 2016 expected to grow tenfold by 2025, we’ll look at what we believe data pipelines and data-pipeline components will need to be able to achieve in terms of functionality, design, compliance, usability, performance, and scalability to handle this growth.
Report
Share
Report
Share
Download to read offline
Recommended
Building Notebook-based AI Pipelines with Elyra and Kubeflow
A typical machine learning pipeline begins as a series of preprocessing steps followed by experimentation, optimization and model-tuning, and, finally deployment. Jupyter notebooks have become a hugely popular tool for data scientists and other machine learning practitioners to explore and experiment as part of this workflow, due to the flexibility and interactivity they provide. However, with notebooks it is often a challenge to move from the experimentation phase to creating a robust, modular and production-grade end-to-end AI pipeline.
Building a Codeless Log Pipeline w/ Confluent Sink Connector | Pollyanna Vale...
Kubernetes became the de-facto standard for running cloud-native applications. And many users turn to it also to run stateful applications such as Apache Kafka. You can use different tools to deploy Kafka on Kubernetes - write your own YAML files, use Helm Charts, or go for one of the available operators. But there is one thing all of these have in common. You still need very good knowledge of Kubernetes to make sure your Kafka cluster works properly in all situations. This talk will cover different Kubernetes features such as resources, affinity, tolerations, pod disruption budgets, topology spread constraints and more. And it will explain why they are important for Apache Kafka and how to use them. If you are interested in running Kafka on Kubernetes and do not know all of these, this is a talk for you.
Replaying KStreams Apps Using State Snapshots (Nishchay Sinha & Yan Wang, Blo...
Are you a Kafka Streams application developer who needs a faster, more efficient way to reproduce a bug or issue from past events? Do you need to test a new algorithm patch near a discrete point-in-time? Do you have a standard methodology for investigating these issues, or does each team member devise an ad hoc solution? How much time does your simulation take?
It is technically impossible to solve this challenge using only changelog topics, since a compact, topic-based changelog doesn't capture the full change history. The Derivatives Data team at Bloomberg augments this through periodic snapshots. To make the snapshotted state accessible to our replay system, we built a query service that leverages Interactive Queries using the Kafka Streams API, along with gRPC-based coordination, to fetch the distributed snapshot states from different snapshotting instances.
In this talk, we will cover:
● Overview of this system architecture
● Deep dive into the mechanics of snapshots with Kafka Streams, state-store changelogs, and a query service to serve replay requests
● How two modes, normal and replay, are used on the Kafka Streams application runtime
● Some use cases that benefit from this replay system
Kafka Connect and KSQL: Useful Tools in Migrating from a Legacy System to Kaf...
This document discusses using Kafka Connect and KSQL to migrate from a legacy system to Kafka Streams. It outlines four steps: 1) preparing the system, 2) integrating legacy data using Kafka Connect, 3) migrating the legacy application to Kafka Streams, and 4) validating the migration using KSQL to compare outputs and detect inconsistencies. Key challenges addressed include integrating different data partitioning schemes, comparing out-of-order or late data, and limitations of KSQL for validation. The overall goal is a methodical migration and validation approach to control risks during the transition.
How to leverage Kafka data streams with Neo4j
Descrizione:
Integrating Apache Kafka with other systems in a reliable and scalable way is often a key part of an event streaming platform. In this talk we'll introduce how to use Apache Kafka (the most used Message Brocker) in combination with Neo4j through the Neo4j-Streams project, demonstrating via simple use-cases how you can leverage the information driven by the Change Data Capture Module and how to add Neo4j in your Kafka flow by using the Sink module in combination with the Neo4j Streams Procedures.
Speaker:
Andrea Santurbano - Neo4J Architect - LARUS Business Automation
Video link: https://youtu.be/oNXWOyDd5HI
Building A Self Service Streaming Platform at Pinterest - Steven Bairos-Novak...
Pinterest is a visual discovery engine that helps more than 250 million monthly active users discover things they love and inspires them to go do those things in their daily lives. Creating Pinterest's new stream processing platform, Xenon, around Flink has enabled teams across the company to tackle new real-time applications. From accelerating machine learning model training iterations to enabling real-time analytics, real-time stream processing has opened up new possibilities everywhere. Given that we ingest over a trillion messages every day through Kafka at Pinterest, a lot went into the design of our new stream processing platform.
We would like to share our experience in making the decision to move from different streaming technologies to Flink. We also discuss how we are building a self-service streaming platform for our engineers, data analysts and data scientists that provides a seamless job deployment pipeline, common monitoring, alerting and tooling. We have also enabled ad-hoc Flink SQL queries on bounded Kafka data streams through a UI to help users explore streaming data easily, extract real-time insights and assist product development.
Gain Deep Visibility into APIs and Integrations with Anypoint Monitoring
On average, a business supporting digital transactions now crosses 35 backend systems—and legacy tools haven’t been able to keep up. This session will cover how MuleSoft uses InfluxCloud to help power their monitoring and diagnostic solutions as well as provide end-to-end actionable visibility to APIs and integrations to help customers identify and resolve issues quickly.
Evaluation of TPC-H on Spark and Spark SQL in ALOJA
The Evaluation of TPC-H on Spark and Spark SQL in ALOJA was conducted at the Big Data Lab to obtain the master degree in Management Information Systems at the Johann-Wolfgang Goethe University in Frankfurt, Germany. Furthermore, the analysis was partially accomplished in collaboration and close coordination with the Barcelona Super Computer Center.
The intention of this research was the integration of a TPC-H on Spark Scala benchmark into ALOJA, an open-source and public platform for automated and cost-efficient benchmarks and to perform an evaluation on the runtime of Spark Scala with or without Hive Metastore compared to Spark SQL. Various alternate file formats with different applied compressions on underlying data and its impact are evaluated. The conducted performance evaluation exposed diverse and captivating outcomes for both benchmarks. Further investigations attempt to detect possible bottlenecks and other irregularities. The aim is to provide an explanation to enhance knowledge of Spark’s engine based on examining the physical plans. Our experiments show, inter alia, that: (1) Spark Scala performs better in case of heavy expression calculation, (2) Spark SQL is the better choice in case of strong data access locality in combination with heavyweight parallel execution. In conclusion, diverse results were observed with the consequence that each API has its advantages and disadvantages.
Surprisingly, our findings are well spread between Spark SQL and Spark Scala and contrary to our expectations Spark Scala did not outperform Spark SQL in all aspects but support the idea that applied optimizations appear to be implemented in a different way by Spark for its core and its extension Spark SQL. The API on top of Spark provides extra information about the underlying structured data, which is probably used to perform additional optimizations.
In conclusion, our research demonstrates that there are differences in the generation of query execution plans that goes hand-in-hand with similar discoveries leading to inefficient joins, and it underlines the importance of our benchmark to identify disparities and bottlenecks.
Speaker
Raphael Radowitz, Quality Specialist, SAP Labs Korea
Recommended
Building Notebook-based AI Pipelines with Elyra and Kubeflow
A typical machine learning pipeline begins as a series of preprocessing steps followed by experimentation, optimization and model-tuning, and, finally deployment. Jupyter notebooks have become a hugely popular tool for data scientists and other machine learning practitioners to explore and experiment as part of this workflow, due to the flexibility and interactivity they provide. However, with notebooks it is often a challenge to move from the experimentation phase to creating a robust, modular and production-grade end-to-end AI pipeline.
Building a Codeless Log Pipeline w/ Confluent Sink Connector | Pollyanna Vale...
Kubernetes became the de-facto standard for running cloud-native applications. And many users turn to it also to run stateful applications such as Apache Kafka. You can use different tools to deploy Kafka on Kubernetes - write your own YAML files, use Helm Charts, or go for one of the available operators. But there is one thing all of these have in common. You still need very good knowledge of Kubernetes to make sure your Kafka cluster works properly in all situations. This talk will cover different Kubernetes features such as resources, affinity, tolerations, pod disruption budgets, topology spread constraints and more. And it will explain why they are important for Apache Kafka and how to use them. If you are interested in running Kafka on Kubernetes and do not know all of these, this is a talk for you.
Replaying KStreams Apps Using State Snapshots (Nishchay Sinha & Yan Wang, Blo...
Are you a Kafka Streams application developer who needs a faster, more efficient way to reproduce a bug or issue from past events? Do you need to test a new algorithm patch near a discrete point-in-time? Do you have a standard methodology for investigating these issues, or does each team member devise an ad hoc solution? How much time does your simulation take?
It is technically impossible to solve this challenge using only changelog topics, since a compact, topic-based changelog doesn't capture the full change history. The Derivatives Data team at Bloomberg augments this through periodic snapshots. To make the snapshotted state accessible to our replay system, we built a query service that leverages Interactive Queries using the Kafka Streams API, along with gRPC-based coordination, to fetch the distributed snapshot states from different snapshotting instances.
In this talk, we will cover:
● Overview of this system architecture
● Deep dive into the mechanics of snapshots with Kafka Streams, state-store changelogs, and a query service to serve replay requests
● How two modes, normal and replay, are used on the Kafka Streams application runtime
● Some use cases that benefit from this replay system
Kafka Connect and KSQL: Useful Tools in Migrating from a Legacy System to Kaf...
This document discusses using Kafka Connect and KSQL to migrate from a legacy system to Kafka Streams. It outlines four steps: 1) preparing the system, 2) integrating legacy data using Kafka Connect, 3) migrating the legacy application to Kafka Streams, and 4) validating the migration using KSQL to compare outputs and detect inconsistencies. Key challenges addressed include integrating different data partitioning schemes, comparing out-of-order or late data, and limitations of KSQL for validation. The overall goal is a methodical migration and validation approach to control risks during the transition.
How to leverage Kafka data streams with Neo4j
Descrizione:
Integrating Apache Kafka with other systems in a reliable and scalable way is often a key part of an event streaming platform. In this talk we'll introduce how to use Apache Kafka (the most used Message Brocker) in combination with Neo4j through the Neo4j-Streams project, demonstrating via simple use-cases how you can leverage the information driven by the Change Data Capture Module and how to add Neo4j in your Kafka flow by using the Sink module in combination with the Neo4j Streams Procedures.
Speaker:
Andrea Santurbano - Neo4J Architect - LARUS Business Automation
Video link: https://youtu.be/oNXWOyDd5HI
Building A Self Service Streaming Platform at Pinterest - Steven Bairos-Novak...
Pinterest is a visual discovery engine that helps more than 250 million monthly active users discover things they love and inspires them to go do those things in their daily lives. Creating Pinterest's new stream processing platform, Xenon, around Flink has enabled teams across the company to tackle new real-time applications. From accelerating machine learning model training iterations to enabling real-time analytics, real-time stream processing has opened up new possibilities everywhere. Given that we ingest over a trillion messages every day through Kafka at Pinterest, a lot went into the design of our new stream processing platform.
We would like to share our experience in making the decision to move from different streaming technologies to Flink. We also discuss how we are building a self-service streaming platform for our engineers, data analysts and data scientists that provides a seamless job deployment pipeline, common monitoring, alerting and tooling. We have also enabled ad-hoc Flink SQL queries on bounded Kafka data streams through a UI to help users explore streaming data easily, extract real-time insights and assist product development.
Gain Deep Visibility into APIs and Integrations with Anypoint Monitoring
On average, a business supporting digital transactions now crosses 35 backend systems—and legacy tools haven’t been able to keep up. This session will cover how MuleSoft uses InfluxCloud to help power their monitoring and diagnostic solutions as well as provide end-to-end actionable visibility to APIs and integrations to help customers identify and resolve issues quickly.
Evaluation of TPC-H on Spark and Spark SQL in ALOJA
The Evaluation of TPC-H on Spark and Spark SQL in ALOJA was conducted at the Big Data Lab to obtain the master degree in Management Information Systems at the Johann-Wolfgang Goethe University in Frankfurt, Germany. Furthermore, the analysis was partially accomplished in collaboration and close coordination with the Barcelona Super Computer Center.
The intention of this research was the integration of a TPC-H on Spark Scala benchmark into ALOJA, an open-source and public platform for automated and cost-efficient benchmarks and to perform an evaluation on the runtime of Spark Scala with or without Hive Metastore compared to Spark SQL. Various alternate file formats with different applied compressions on underlying data and its impact are evaluated. The conducted performance evaluation exposed diverse and captivating outcomes for both benchmarks. Further investigations attempt to detect possible bottlenecks and other irregularities. The aim is to provide an explanation to enhance knowledge of Spark’s engine based on examining the physical plans. Our experiments show, inter alia, that: (1) Spark Scala performs better in case of heavy expression calculation, (2) Spark SQL is the better choice in case of strong data access locality in combination with heavyweight parallel execution. In conclusion, diverse results were observed with the consequence that each API has its advantages and disadvantages.
Surprisingly, our findings are well spread between Spark SQL and Spark Scala and contrary to our expectations Spark Scala did not outperform Spark SQL in all aspects but support the idea that applied optimizations appear to be implemented in a different way by Spark for its core and its extension Spark SQL. The API on top of Spark provides extra information about the underlying structured data, which is probably used to perform additional optimizations.
In conclusion, our research demonstrates that there are differences in the generation of query execution plans that goes hand-in-hand with similar discoveries leading to inefficient joins, and it underlines the importance of our benchmark to identify disparities and bottlenecks.
Speaker
Raphael Radowitz, Quality Specialist, SAP Labs Korea
How to Deliver a Critical and Actionable Customer-Facing Metrics Product with...
In IoT, understanding the health of thousands of devices is critical for deployment at scale especially when troubleshooting an issue. Particle’s customer base needed visibility into their devices with actionable data to reference in real time. Join Cullen Murphy, Site Reliability Engineer at Particle, to learn how the team built a metrics system on Telegraf, Kubernetes, and Prometheus to deploy a customer-facing product that provides critical and relevant data to their IoT product creators.
Digital Transformation in Healthcare with Kafka—Building a Low Latency Data P...
(Dmitry Milman + Ankur Kaneria, Express Scripts) Kafka Summit SF 2018
Building cloud-based microservices can be a challenge when the system of record is a relational database residing on an on-premise mainframe. The challenge lies in the ability to efficiently and cost-effectively access the ever-increasing amount of data. Express Scripts is reimagining its data architecture to bring best-in-class user experience and provide the foundation of next-generation applications.
This talk will showcase how Kafka plays a key role within Express Scripts’ transformation from mainframe to a microservice-based ecosystem, ensuring data integrity between two worlds. It will discuss how change data capture (CDC) is leveraged to stream data changes to Kafka, allowing us to build a low-latency data sync pipeline. We will describe how we achieve transactional consistency by collapsing all events that belong together onto a single topic, yet have the ability to scale out to meet the real time SLAs and low-latency requirements through means of partitions. We will share our Kafka Streams configuration to handle the data transformation workload. We will discuss our overall Kafka cluster footprint, configuration and security measures.
Express Scripts Holding Company is an American Fortune 100 company. As of 2018, the company is the 25th largest in the U.S. as well as one of the largest pharmacy benefit management organizations in the U.S. Customers rely on 24/7 access to our services, and need the ability to interact with our systems in real time via various channels such as web and mobile. Sharing our mainframe t0 microservices migration journey, our experiences and lessons learned would be beneficial to other companies venturing on a similar path.
Better Together: How Graph database enables easy data integration with Spark ...
See all on-demand Graph + AI Sessions: https://www.tigergraph.com/graph-ai-world-sessions/
Get TigerGraph: https://www.tigergraph.com/get-tigergraph/
Martin Moucka [Red Hat] | How Red Hat Uses gNMI, Telegraf and InfluxDB to Gai...
Red Hat is the provider of enterprise open source solutions. Its portfolio of products includes hybrid cloud infrastructure, middleware, cloud-native apps and automation solutions. Its internal network supports all lines of business — including 60+ sites. Discover how Red Hat uses InfluxDB and Flux for better real-time monitoring of their networks to improve performance and to understand utilization better.
Flink for Everyone: Self Service Data Analytics with StreamPipes - Philipp Ze...
This talk presents StreamPipes (https://www.streampipes.org), an open source self-service data analytics solution leveraging existing big data technologies such as Apache Flink to provide non-technical users with an easy and intuitive way to connect, analyze and exploit a variety of different streaming data sources for their use.
Newly arising IoT-driven use cases in domains such as manufacturing, smart city or autonomous driving often demand for continuous integration and processing of sensor data in order to derive time-sensitive actions. One example is the optimization of maintenance processes based on the current condition of machines (condition-based maintenance). While this is technically already well supported by the existing big data tool landscape, building such applications still require a crucial set of expertise ranging from general domain expertise, programming skills to deep knowledge on distributed and scalable systems. Such skills are usually not present in hardware-focused manufacturing companies.
To mitigate these shortcomings, StreamPipes allows non-technical users to leverage a graphical editor to model and deploy analytical tasks as pipelines in a drag and drop manner. Pipelines are built based on a toolbox of reusable data adapters, processors and sinks. Toolbox elements encapsulate dedicated algorithms (e.g., filter, aggregation, machine learning classifiers) implemented in big data processing engines such as Apache Flink communicating over an internal distributed messaging system (e.g. Apache Kafka).
In this talk, we present technologies and tools enabling flexible modeling of real-time processing pipelines by domain experts. We motivate our talk by showing real-world examples we gathered from a number of industry projects during the past years in Industrial IoT domains such as manufacturing and supply chain management. For instance, we show how StreamPipes eases the accessibility of big data tools for non-technical users based on examples such as supervising a fleet of autonomous electric delivery vehicles as well as data analytics in one of the largest test areas for autonomous driving in Germany.
Apache Flink and what it is used for
Aljoscha Krettek is the PMC chair of Apache Flink and Apache Beam, and co-founder of data Artisans. Apache Flink is an open-source platform for distributed stream and batch data processing. It allows for stateful computations over data streams in real-time and historically. Flink supports batch and stream processing using APIs like DataSet and DataStream. Data Artisans originated Flink and provides an application platform powered by Flink and Kubernetes for building stateful stream processing applications.
Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...
(Bob Lehmann, Bayer) Kafka Summit SF 2018
You’ve built your streaming data platform. The early adopters are “all in” and have developed producers, consumers and stream processing apps for a number of use cases. A large percentage of the enterprise, however, has expressed interest but hasn’t made the leap. Why?
In 2014, Bayer Crop Science (formerly Monsanto) adopted a cloud first strategy and started a multi-year transition to the cloud. A Kafka-based cross-datacenter DataHub was created to facilitate this migration and to drive the shift to real-time stream processing. The DataHub has seen strong enterprise adoption and supports a myriad of use cases. Data is ingested from a wide variety of sources and the data can move effortlessly between an on premise datacenter, AWS and Google Cloud. The DataHub has evolved continuously over time to meet the current and anticipated needs of our internal customers. The “cost of admission” for the platform has been lowered dramatically over time via our DataHub Portal and technologies such as Kafka Connect, Kubernetes and Presto. Most operations are now self-service, onboarding of new data sources is relatively painless and stream processing via KSQL and other technologies is being incorporated into the core DataHub platform.
In this talk, Bob Lehmann will describe the origins and evolution of the Enterprise DataHub with an emphasis on steps that were taken to drive user adoption. Bob will also talk about integrations between the DataHub and other key data platforms at Bayer, lessons learned and the future direction for streaming data and stream processing at Bayer.
How a Time Series Database Contributes to a Decentralized Cloud Object Storag...
In this presentation, you'll learn how InfluxDB is a component to Storj’s Tardigrade service and workflows. John Gleeson and Ben Sirb of Storj Lab will Storj’s redefinition of a cloud object storage network, how InfluxData fits into Storj’s Open Source Partner Program, and how to collect and manage high-volume, real-time telemetry data from a distributed network.
ML, Statistics, and Spark with Databricks for Maximizing Revenue in a Delayed...
In this talk, we will present how we used Spark, Databricks, Airflow and MLflow to process big data, and build a pipeline of both ML(XGBoost) and statistical models that maximizes our revenues in one of our core products, called the “Offer Wall”. The “Offer wall” is a mobile phone product that is integrated with existing apps, suggesting users to perform tasks in exchange for in-app currency. The problem gets even more interesting when considering the fact that some of the tasks users do take 15 minutes and some may take up to 2 to weeks, forcing us to make revenue determining decisions in an uncertain space all of the time. The solution we developed utilizes Databricks and Spark’s strengths and diversity in machine learning, big data, MLflow and Airflow integrations, allowing us to deliver a production-grade solution with short development time between experiments.
Postgres Vision 2018: Your Migration Path - BinckBank Case Study
Frans Verduyn Lunel, Database Architect at BinckBank, presented a case study at Postgres Vision 2018 about a complex Proof-of-Concept (POC) with EDB Postgres that spared refactoring an application and produced unbelievable performance.
Digital transformation: Highly resilient streaming architecture and strategie...
Failure is inevitable in any distributed system but anticipating failures and building systems to recover from failures instantaneous makes the system highly resilient. At Capital One we process billions of events everyday and we leverage cloud, microservices, streaming and machine learning technologies to solve customer problems and provide the best customer experience.
As part of this session I will be talking about highly resilient streaming architecture that is supporting processing of billions of events every day then some of the strategies & best practices to build highly available and fault-tolerant systems utilizing Kafka and Cloud environments.
Introduction to Streaming with Apache Flink
This document discusses stream processing with Apache Flink. It provides an overview of Flink's architecture, including its deployment options and APIs for batch and stream processing. It also covers core Flink concepts like windows, event-time processing, and an example of complex event processing (CEP) to detect patterns in event streams and generate real-time warnings. The document concludes with a demonstration of Flink's streaming analytics capabilities.
Confluent Steaming Webinar - Cape Town - Vitality
The V1 Behavioral Change Platform provides the technical infrastructure to implement and manage global behavioral change programs at scale. It has powered programs in 9 markets since 2018 and continues expanding. The platform utilizes a centralized architecture with microservices and Kafka messaging to process high volumes of data securely and reliably across markets. It is configured to allow partners to customize programs while leveraging shared capabilities and accelerating delivery through its open APIs and federated development model.
Building Pinterest Real-Time Ads Platform Using Kafka Streams
Building Pinterest Real-Time Ads Platform Using Kafka Streams (Liquan Pei + Boyang Chen, Pinterest) Kafka Summit SF 2018
In this talk, we are sharing the experience of building Pinterest’s real-time Ads Platform utilizing Kafka Streams. The real-time budgeting system is the most mission-critical component of the Ads Platform as it controls how each ad is delivered to maximize user, advertiser and Pinterest value. The system needs to handle over 50,000 queries per section (QPS) impressions, requires less than five seconds of end-to-end latency and recovers within five minutes during outages. It also needs to be scalable to handle the fast growth of Pinterest’s ads business.
The real-time budgeting system is composed of real-time stream-stream joiner, real-time spend aggregator and a spend predictor. At Pinterest’s scale, we need to overcome quite a few challenges to make each component work. For example, the stream-stream joiner needs to maintain terabyte size state while supporting fast recovery, and the real-time spend aggregator needs to publish to thousands of ads servers while supporting over one million read QPS. We choose Kafka Streams as it provides milliseconds latency guarantee, scalable event-based processing and easy-to-use APIs. In the process of building the system, we performed tons of tuning to RocksDB, Kafka Producer and Consumer, and pushed several open source contributions to Apache Kafka. We are also working on adding a remote checkpoint for Kafka Streams state to reduce the time of code start when adding more machines to the application. We believe that our experience can be beneficial to people who want to build real-time streaming solutions at large scale and deeply understand Kafka Streams.
Marina Svicevic, Milos Pavkovic, Mladen Maric, Vijeta Hingorani [Socialgist] ...
The document discusses two projects using InfluxDB: 1) Transforming email reports by consuming messages from Kafka, aggregating and storing in InfluxDB, and generating dashboards. They solved a high volume mismatch issue by adding unique tags and building a point cache. 2) Anomaly detection by collecting data from various sources, storing counts in InfluxDB, and finding anomalies like sudden drops, volume decay, and deadman events. Anomalies and alerts are stored in InfluxDB and handled via Slack, email, and SMS. Visualizations are created in Grafana by obtaining data from InfluxDB.
HOP! Airlines Jets to Real Time
(Joseph deBuzna + Zulfikar Quereshi, HVR) Kafka Summit SF 2018
This presentation is a customer story about France-based regional airline HOP! and their need to make better use of data that was contained in various applications. They also needed this information to be available in real time. As one can imagine, airlines manage a wide variety of information such as weather, customer information, flight plans, sensor data from planes and much more.
In this presentation, Joe will discuss how HOP! was delivering their data before and the limitations associated with delivering this data. Joe will then talk about HOP!’s selection of Kafka and HVR as a solution to enabling data availability and real-time information for analysis and action.
In this session, attendees will learn:
-How Kafka was selected and chosen as a solution for HOP!’s complex challenges
-Architecture and capabilities implemented that enabled data feeding from multiple sources to Kafka
-Considerations and challenges with this approach
-Business results and future plans
Monitor Kubernetes in Rancher using InfluxData
The document discusses monitoring Kubernetes in Rancher using InfluxData. It provides an introduction to Rancher and how it can deploy and manage Kubernetes clusters. It then discusses the time series data problem and how InfluxData solutions like Telegraf, InfluxDB, Chronograf and Kapacitor can collect, store and analyze the high volume time series data generated by containers. It highlights key features of Rancher and InfluxData and demonstrates how their tools can be used together to monitor Kubernetes workloads.
Kafka and Kafka Streams in the Global Schibsted Data Platform
This document discusses Schibsted's use of Kafka and Kafka Streams in its global data platform. It describes how Schibsted processes streaming data using Kafka Streams to build a new streaming pipeline called Yggdrasil that provides low latency and delivery guarantees. It also discusses how Kafka Connect is used to get data into and out of the platform and how third party analytics tools can connect. The document outlines some challenges experienced in scaling up the platform.
#SlimScalding - Less Memory is More Capacity
This document discusses techniques for automatically tuning memory configurations for Scalding jobs submitted to Hadoop clusters. It proposes leveraging prior run counters from a history repository to estimate heap sizes on a per-step basis. This aims to set optimal configurations while avoiding memory over-estimation issues. It acknowledges challenges around fallback handling for new jobs and estimation inaccuracies that could lead to underestimation issues. Overall contributions to the Scalding project are welcomed to further develop automated memory tuning capabilities.
Timothy Spann [StreamNative] | Using FLaNK with InfluxDB for EdgeAI IoT at Sc...
Using FLaNK with InfluxDB for EdgeAI IoT at Scale
Timothy from StreamNative take you on a hands-on deep-dive on using Pulsar, Apache NiFi + Edge Flow Manager + MiniFi Agents with Apache MXNet, OpenVino, TensorFlow Lite, and other Deep Learning Libraries on the actual edge devices including Raspberry Pi with Movidius 2, Google Coral TPU and NVidia Jetson Nano. The team run deep learning models on the edge devices and send images, and capture real-time GPS and sensor data. Their low-coding IoT applications provide easy edge routing, transformation, data acquisition and alerting before they decide what data to stream real-time to their data space. These edge applications classify images and sensor readings real-time at the edge and then send Deep Learning results to Flink SQL and Apache NiFi for transformation, parsing, enrichment, querying, filtering and merging data to InfluxDB.
The Future of Data Pipelines
The document discusses trends in data pipelines and the future of data pipelines. It covers how data pipelines have evolved to handle larger and more real-time data through technologies like Kafka. It also discusses how data pipelines will need to adapt to trends like edge computing, increased data volumes, streaming data functionality like auto-scaling and error handling, and compliance needs. The presentation examines various aspects of data pipelines including design, reliability, and usability and how they may change in the future.
How Comcast Turns Big Data into Real Time Operational Insights: Winter Olympi...
2014 O'Reilly Strata conference presentation by Patrick Shumate and Brett Sheppard, about the Comcast technology stack delivering content from the 2014 Winter Olympics in Sochi. The session and slides are presented with approval from NBC and its parent company Comcast.
More Related Content
What's hot
How to Deliver a Critical and Actionable Customer-Facing Metrics Product with...
In IoT, understanding the health of thousands of devices is critical for deployment at scale especially when troubleshooting an issue. Particle’s customer base needed visibility into their devices with actionable data to reference in real time. Join Cullen Murphy, Site Reliability Engineer at Particle, to learn how the team built a metrics system on Telegraf, Kubernetes, and Prometheus to deploy a customer-facing product that provides critical and relevant data to their IoT product creators.
Digital Transformation in Healthcare with Kafka—Building a Low Latency Data P...
(Dmitry Milman + Ankur Kaneria, Express Scripts) Kafka Summit SF 2018
Building cloud-based microservices can be a challenge when the system of record is a relational database residing on an on-premise mainframe. The challenge lies in the ability to efficiently and cost-effectively access the ever-increasing amount of data. Express Scripts is reimagining its data architecture to bring best-in-class user experience and provide the foundation of next-generation applications.
This talk will showcase how Kafka plays a key role within Express Scripts’ transformation from mainframe to a microservice-based ecosystem, ensuring data integrity between two worlds. It will discuss how change data capture (CDC) is leveraged to stream data changes to Kafka, allowing us to build a low-latency data sync pipeline. We will describe how we achieve transactional consistency by collapsing all events that belong together onto a single topic, yet have the ability to scale out to meet the real time SLAs and low-latency requirements through means of partitions. We will share our Kafka Streams configuration to handle the data transformation workload. We will discuss our overall Kafka cluster footprint, configuration and security measures.
Express Scripts Holding Company is an American Fortune 100 company. As of 2018, the company is the 25th largest in the U.S. as well as one of the largest pharmacy benefit management organizations in the U.S. Customers rely on 24/7 access to our services, and need the ability to interact with our systems in real time via various channels such as web and mobile. Sharing our mainframe t0 microservices migration journey, our experiences and lessons learned would be beneficial to other companies venturing on a similar path.
Better Together: How Graph database enables easy data integration with Spark ...
See all on-demand Graph + AI Sessions: https://www.tigergraph.com/graph-ai-world-sessions/
Get TigerGraph: https://www.tigergraph.com/get-tigergraph/
Martin Moucka [Red Hat] | How Red Hat Uses gNMI, Telegraf and InfluxDB to Gai...
Red Hat is the provider of enterprise open source solutions. Its portfolio of products includes hybrid cloud infrastructure, middleware, cloud-native apps and automation solutions. Its internal network supports all lines of business — including 60+ sites. Discover how Red Hat uses InfluxDB and Flux for better real-time monitoring of their networks to improve performance and to understand utilization better.
Flink for Everyone: Self Service Data Analytics with StreamPipes - Philipp Ze...
This talk presents StreamPipes (https://www.streampipes.org), an open source self-service data analytics solution leveraging existing big data technologies such as Apache Flink to provide non-technical users with an easy and intuitive way to connect, analyze and exploit a variety of different streaming data sources for their use.
Newly arising IoT-driven use cases in domains such as manufacturing, smart city or autonomous driving often demand for continuous integration and processing of sensor data in order to derive time-sensitive actions. One example is the optimization of maintenance processes based on the current condition of machines (condition-based maintenance). While this is technically already well supported by the existing big data tool landscape, building such applications still require a crucial set of expertise ranging from general domain expertise, programming skills to deep knowledge on distributed and scalable systems. Such skills are usually not present in hardware-focused manufacturing companies.
To mitigate these shortcomings, StreamPipes allows non-technical users to leverage a graphical editor to model and deploy analytical tasks as pipelines in a drag and drop manner. Pipelines are built based on a toolbox of reusable data adapters, processors and sinks. Toolbox elements encapsulate dedicated algorithms (e.g., filter, aggregation, machine learning classifiers) implemented in big data processing engines such as Apache Flink communicating over an internal distributed messaging system (e.g. Apache Kafka).
In this talk, we present technologies and tools enabling flexible modeling of real-time processing pipelines by domain experts. We motivate our talk by showing real-world examples we gathered from a number of industry projects during the past years in Industrial IoT domains such as manufacturing and supply chain management. For instance, we show how StreamPipes eases the accessibility of big data tools for non-technical users based on examples such as supervising a fleet of autonomous electric delivery vehicles as well as data analytics in one of the largest test areas for autonomous driving in Germany.
Apache Flink and what it is used for
Aljoscha Krettek is the PMC chair of Apache Flink and Apache Beam, and co-founder of data Artisans. Apache Flink is an open-source platform for distributed stream and batch data processing. It allows for stateful computations over data streams in real-time and historically. Flink supports batch and stream processing using APIs like DataSet and DataStream. Data Artisans originated Flink and provides an application platform powered by Flink and Kubernetes for building stateful stream processing applications.
Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...
(Bob Lehmann, Bayer) Kafka Summit SF 2018
You’ve built your streaming data platform. The early adopters are “all in” and have developed producers, consumers and stream processing apps for a number of use cases. A large percentage of the enterprise, however, has expressed interest but hasn’t made the leap. Why?
In 2014, Bayer Crop Science (formerly Monsanto) adopted a cloud first strategy and started a multi-year transition to the cloud. A Kafka-based cross-datacenter DataHub was created to facilitate this migration and to drive the shift to real-time stream processing. The DataHub has seen strong enterprise adoption and supports a myriad of use cases. Data is ingested from a wide variety of sources and the data can move effortlessly between an on premise datacenter, AWS and Google Cloud. The DataHub has evolved continuously over time to meet the current and anticipated needs of our internal customers. The “cost of admission” for the platform has been lowered dramatically over time via our DataHub Portal and technologies such as Kafka Connect, Kubernetes and Presto. Most operations are now self-service, onboarding of new data sources is relatively painless and stream processing via KSQL and other technologies is being incorporated into the core DataHub platform.
In this talk, Bob Lehmann will describe the origins and evolution of the Enterprise DataHub with an emphasis on steps that were taken to drive user adoption. Bob will also talk about integrations between the DataHub and other key data platforms at Bayer, lessons learned and the future direction for streaming data and stream processing at Bayer.
How a Time Series Database Contributes to a Decentralized Cloud Object Storag...
In this presentation, you'll learn how InfluxDB is a component to Storj’s Tardigrade service and workflows. John Gleeson and Ben Sirb of Storj Lab will Storj’s redefinition of a cloud object storage network, how InfluxData fits into Storj’s Open Source Partner Program, and how to collect and manage high-volume, real-time telemetry data from a distributed network.
ML, Statistics, and Spark with Databricks for Maximizing Revenue in a Delayed...
In this talk, we will present how we used Spark, Databricks, Airflow and MLflow to process big data, and build a pipeline of both ML(XGBoost) and statistical models that maximizes our revenues in one of our core products, called the “Offer Wall”. The “Offer wall” is a mobile phone product that is integrated with existing apps, suggesting users to perform tasks in exchange for in-app currency. The problem gets even more interesting when considering the fact that some of the tasks users do take 15 minutes and some may take up to 2 to weeks, forcing us to make revenue determining decisions in an uncertain space all of the time. The solution we developed utilizes Databricks and Spark’s strengths and diversity in machine learning, big data, MLflow and Airflow integrations, allowing us to deliver a production-grade solution with short development time between experiments.
Postgres Vision 2018: Your Migration Path - BinckBank Case Study
Frans Verduyn Lunel, Database Architect at BinckBank, presented a case study at Postgres Vision 2018 about a complex Proof-of-Concept (POC) with EDB Postgres that spared refactoring an application and produced unbelievable performance.
Digital transformation: Highly resilient streaming architecture and strategie...
Failure is inevitable in any distributed system but anticipating failures and building systems to recover from failures instantaneous makes the system highly resilient. At Capital One we process billions of events everyday and we leverage cloud, microservices, streaming and machine learning technologies to solve customer problems and provide the best customer experience.
As part of this session I will be talking about highly resilient streaming architecture that is supporting processing of billions of events every day then some of the strategies & best practices to build highly available and fault-tolerant systems utilizing Kafka and Cloud environments.
Introduction to Streaming with Apache Flink
This document discusses stream processing with Apache Flink. It provides an overview of Flink's architecture, including its deployment options and APIs for batch and stream processing. It also covers core Flink concepts like windows, event-time processing, and an example of complex event processing (CEP) to detect patterns in event streams and generate real-time warnings. The document concludes with a demonstration of Flink's streaming analytics capabilities.
Confluent Steaming Webinar - Cape Town - Vitality
The V1 Behavioral Change Platform provides the technical infrastructure to implement and manage global behavioral change programs at scale. It has powered programs in 9 markets since 2018 and continues expanding. The platform utilizes a centralized architecture with microservices and Kafka messaging to process high volumes of data securely and reliably across markets. It is configured to allow partners to customize programs while leveraging shared capabilities and accelerating delivery through its open APIs and federated development model.
Building Pinterest Real-Time Ads Platform Using Kafka Streams
Building Pinterest Real-Time Ads Platform Using Kafka Streams (Liquan Pei + Boyang Chen, Pinterest) Kafka Summit SF 2018
In this talk, we are sharing the experience of building Pinterest’s real-time Ads Platform utilizing Kafka Streams. The real-time budgeting system is the most mission-critical component of the Ads Platform as it controls how each ad is delivered to maximize user, advertiser and Pinterest value. The system needs to handle over 50,000 queries per section (QPS) impressions, requires less than five seconds of end-to-end latency and recovers within five minutes during outages. It also needs to be scalable to handle the fast growth of Pinterest’s ads business.
The real-time budgeting system is composed of real-time stream-stream joiner, real-time spend aggregator and a spend predictor. At Pinterest’s scale, we need to overcome quite a few challenges to make each component work. For example, the stream-stream joiner needs to maintain terabyte size state while supporting fast recovery, and the real-time spend aggregator needs to publish to thousands of ads servers while supporting over one million read QPS. We choose Kafka Streams as it provides milliseconds latency guarantee, scalable event-based processing and easy-to-use APIs. In the process of building the system, we performed tons of tuning to RocksDB, Kafka Producer and Consumer, and pushed several open source contributions to Apache Kafka. We are also working on adding a remote checkpoint for Kafka Streams state to reduce the time of code start when adding more machines to the application. We believe that our experience can be beneficial to people who want to build real-time streaming solutions at large scale and deeply understand Kafka Streams.
Marina Svicevic, Milos Pavkovic, Mladen Maric, Vijeta Hingorani [Socialgist] ...
The document discusses two projects using InfluxDB: 1) Transforming email reports by consuming messages from Kafka, aggregating and storing in InfluxDB, and generating dashboards. They solved a high volume mismatch issue by adding unique tags and building a point cache. 2) Anomaly detection by collecting data from various sources, storing counts in InfluxDB, and finding anomalies like sudden drops, volume decay, and deadman events. Anomalies and alerts are stored in InfluxDB and handled via Slack, email, and SMS. Visualizations are created in Grafana by obtaining data from InfluxDB.
HOP! Airlines Jets to Real Time
(Joseph deBuzna + Zulfikar Quereshi, HVR) Kafka Summit SF 2018
This presentation is a customer story about France-based regional airline HOP! and their need to make better use of data that was contained in various applications. They also needed this information to be available in real time. As one can imagine, airlines manage a wide variety of information such as weather, customer information, flight plans, sensor data from planes and much more.
In this presentation, Joe will discuss how HOP! was delivering their data before and the limitations associated with delivering this data. Joe will then talk about HOP!’s selection of Kafka and HVR as a solution to enabling data availability and real-time information for analysis and action.
In this session, attendees will learn:
-How Kafka was selected and chosen as a solution for HOP!’s complex challenges
-Architecture and capabilities implemented that enabled data feeding from multiple sources to Kafka
-Considerations and challenges with this approach
-Business results and future plans
Monitor Kubernetes in Rancher using InfluxData
The document discusses monitoring Kubernetes in Rancher using InfluxData. It provides an introduction to Rancher and how it can deploy and manage Kubernetes clusters. It then discusses the time series data problem and how InfluxData solutions like Telegraf, InfluxDB, Chronograf and Kapacitor can collect, store and analyze the high volume time series data generated by containers. It highlights key features of Rancher and InfluxData and demonstrates how their tools can be used together to monitor Kubernetes workloads.
Kafka and Kafka Streams in the Global Schibsted Data Platform
This document discusses Schibsted's use of Kafka and Kafka Streams in its global data platform. It describes how Schibsted processes streaming data using Kafka Streams to build a new streaming pipeline called Yggdrasil that provides low latency and delivery guarantees. It also discusses how Kafka Connect is used to get data into and out of the platform and how third party analytics tools can connect. The document outlines some challenges experienced in scaling up the platform.
#SlimScalding - Less Memory is More Capacity
This document discusses techniques for automatically tuning memory configurations for Scalding jobs submitted to Hadoop clusters. It proposes leveraging prior run counters from a history repository to estimate heap sizes on a per-step basis. This aims to set optimal configurations while avoiding memory over-estimation issues. It acknowledges challenges around fallback handling for new jobs and estimation inaccuracies that could lead to underestimation issues. Overall contributions to the Scalding project are welcomed to further develop automated memory tuning capabilities.
Timothy Spann [StreamNative] | Using FLaNK with InfluxDB for EdgeAI IoT at Sc...
Using FLaNK with InfluxDB for EdgeAI IoT at Scale
Timothy from StreamNative take you on a hands-on deep-dive on using Pulsar, Apache NiFi + Edge Flow Manager + MiniFi Agents with Apache MXNet, OpenVino, TensorFlow Lite, and other Deep Learning Libraries on the actual edge devices including Raspberry Pi with Movidius 2, Google Coral TPU and NVidia Jetson Nano. The team run deep learning models on the edge devices and send images, and capture real-time GPS and sensor data. Their low-coding IoT applications provide easy edge routing, transformation, data acquisition and alerting before they decide what data to stream real-time to their data space. These edge applications classify images and sensor readings real-time at the edge and then send Deep Learning results to Flink SQL and Apache NiFi for transformation, parsing, enrichment, querying, filtering and merging data to InfluxDB.
What's hot (20)
How to Deliver a Critical and Actionable Customer-Facing Metrics Product with...
How to Deliver a Critical and Actionable Customer-Facing Metrics Product with...
Digital Transformation in Healthcare with Kafka—Building a Low Latency Data P...
Digital Transformation in Healthcare with Kafka—Building a Low Latency Data P...
Better Together: How Graph database enables easy data integration with Spark ...
Better Together: How Graph database enables easy data integration with Spark ...
Martin Moucka [Red Hat] | How Red Hat Uses gNMI, Telegraf and InfluxDB to Gai...
Martin Moucka [Red Hat] | How Red Hat Uses gNMI, Telegraf and InfluxDB to Gai...
Flink for Everyone: Self Service Data Analytics with StreamPipes - Philipp Ze...
Flink for Everyone: Self Service Data Analytics with StreamPipes - Philipp Ze...
Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...
Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...
How a Time Series Database Contributes to a Decentralized Cloud Object Storag...
How a Time Series Database Contributes to a Decentralized Cloud Object Storag...
ML, Statistics, and Spark with Databricks for Maximizing Revenue in a Delayed...
ML, Statistics, and Spark with Databricks for Maximizing Revenue in a Delayed...
Postgres Vision 2018: Your Migration Path - BinckBank Case Study
Postgres Vision 2018: Your Migration Path - BinckBank Case Study
Digital transformation: Highly resilient streaming architecture and strategie...
Digital transformation: Highly resilient streaming architecture and strategie...
Building Pinterest Real-Time Ads Platform Using Kafka Streams
Building Pinterest Real-Time Ads Platform Using Kafka Streams
Marina Svicevic, Milos Pavkovic, Mladen Maric, Vijeta Hingorani [Socialgist] ...
Marina Svicevic, Milos Pavkovic, Mladen Maric, Vijeta Hingorani [Socialgist] ...
Kafka and Kafka Streams in the Global Schibsted Data Platform
Kafka and Kafka Streams in the Global Schibsted Data Platform
Timothy Spann [StreamNative] | Using FLaNK with InfluxDB for EdgeAI IoT at Sc...
Timothy Spann [StreamNative] | Using FLaNK with InfluxDB for EdgeAI IoT at Sc...
Similar to The Future of Data Pipelines
The Future of Data Pipelines
The document discusses trends in data pipelines and the future of data pipelines. It covers how data pipelines have evolved to handle larger and more real-time data through technologies like Kafka. It also discusses how data pipelines will need to adapt to trends like edge computing, increased data volumes, streaming data functionality like auto-scaling and error handling, and compliance needs. The presentation examines various aspects of data pipelines including design, reliability, and usability and how they may change in the future.
How Comcast Turns Big Data into Real Time Operational Insights: Winter Olympi...
2014 O'Reilly Strata conference presentation by Patrick Shumate and Brett Sheppard, about the Comcast technology stack delivering content from the 2014 Winter Olympics in Sochi. The session and slides are presented with approval from NBC and its parent company Comcast.
Eecs6893 big dataanalytics-lecture1
This document provides an overview of a lecture on big data analytics given by Dr. Ching-Yung Lin. The key points covered in the lecture include:
- Definitions and characteristics of big data based on the 3V's of volume, velocity and variety.
- Techniques used for big data such as massive parallelism, distributed storage and processing, machine learning and data visualization.
- Factors that have enabled big data to become prominent in recent years like greater data collection, open source software and commodity hardware.
- Examples of big data platforms, databases and analytics techniques including Hadoop, Spark, NoSQL databases and graph databases.
- The large and growing market for big data
Present and future of unified, portable, and efficient data processing with A...
The world of big data involves an ever-changing field of players. Much as SQL stands as a lingua franca for declarative data analysis, Apache Beam aims to provide a portable standard for expressing robust, out-of-order data processing pipelines in a variety of languages across a variety of platforms. In a way, Apache Beam is a glue that can connect the big data ecosystem together; it enables users to "run any data processing pipeline anywhere."
This talk will briefly cover the capabilities of the Beam model for data processing and discuss its architecture, including the portability model. We’ll focus on the present state of the community and the current status of the Beam ecosystem. We’ll cover the state of the art in data processing and discuss where Beam is going next, including completion of the portability framework and the Streaming SQL. Finally, we’ll discuss areas of improvement and how anybody can join us on the path of creating the glue that interconnects the big data ecosystem.
Speaker
Davor Bonaci, Apache Software Foundation; Simbly, V.P. of Apache Beam; Founder/CEO at Operiant
Integrating Semantic Web in the Real World: A Journey between Two Cities
Keynote at The 9th International Conference on Knowledge Capture (KCAP2017), Austin, Texas, Dec 2017
An early vision in Computer Science has been to create intelligent systems capable of reasoning on large amounts of data. Today, this vision can be delivered by integrating Relational Databases with the Semantic Web using the W3C standards: a graph data model (RDF), ontology language (OWL), mapping language (R2RML) and query language (SPARQL). The research community has successfully been showing how intelligent systems can be created with Semantic Web technologies, dubbed now as Knowledge Graphs.
However, where is the mainstream industry adoption? What are the barriers to adoption? Are these engineering and social barriers or are they open scientific problems that need to be addressed?
This talk will chronicle our journey of deploying Semantic Web technologies with real world users to address Business Intelligence and Data Integration needs, describe technical and social obstacles that are present in large organizations, and scientific challenges that require attention.
Cloud-Scale BGP and NetFlow Analysis
If your business is heavily dependent on the Internet, you may be facing an unprecedented level of network traffic analytics data. How to make the most of that data is the challenge. This presentation from Kentik VP Product and former EMA analyst Jim Frey explores the evolving need, the architecture and key use cases for BGP and NetFlow analysis based on scale-out cloud computing and Big Data technologies.
Amidst demo (BNAIC 2015)
The document describes the AMIDST Toolbox, an open source Java library for analyzing massive data streams using probabilistic graphical models. The toolbox uses Bayesian networks and inference algorithms like variational message passing to model and learn from continuous data streams in a scalable way. It was developed as part of an EU research project to handle real-world complex streaming data. The document outlines the toolbox's capabilities and provides an example live demo of using it to model concept drift in financial data by tracking changes over time with a hidden variable in a naive Bayes model.
Big and fast data strategy 2017 jr
This document provides an overview and strategy for big and fast data initiatives in 2017. It discusses the data landscape including volume, velocity, variety and validity. It evaluates different data platform technologies and outlines requirements. The vision is described as "Business Insights at the Speed of Light". The strategy focuses on speed and leveraging key technologies like Spark. A roadmap with initiatives around insights, infrastructure, ingestion and big BI is presented. High level architectures for streaming and data flow are shown. Finally, data preparation vendors are compared.
Network of Networks Discussion and Charter Document
Working description of the network of networks group discussion on November 30th, 2017. A cross-industry discussion on B2B connectivity.
Data-Driven Transformation: Leveraging Big Data at Showtime with Apache Spark
Interested in learning how Showtime is leveraging the power of Spark to transform a traditional premium cable network into a data-savvy analytical competitor? The growth in our over-the-top (OTT) streaming subscription business has led to an abundance of user-level data not previously available. To capitalize on this opportunity, we have been building and evolving our unified platform which allows data scientists and business analysts to tap into this rich behavioral data to support our business goals. We will share how our small team of data scientists is creating meaningful features which capture the nuanced relationships between users and content; productionizing machine learning models; and leveraging MLflow to optimize the runtime of our pipelines, track the accuracy of our models, and log the quality of our data over time. From data wrangling and exploration to machine learning and automation, we are augmenting our data supply chain by constantly rolling out new capabilities and analytical products to help the organization better understand our subscribers, our content, and our path forward to a data-driven future.
Authors: Josh McNutt, Keria Bermudez-Hernandez
Text Analytics & Linked Data Management As-a-Service
slides from the talk on "Text Analytics & Linked Data Management As-a-Service with S4" from the ESWC'2015 workshop on Semantic Web Enterprise Adoption & Best Practices
full paper available at http://2015.wasabi-ws.org/papers/wasabi15_1.pdf
OTC Start Thinking BIG Data 2018 10-18
Whatever the size or type of organization, Big Data has permeated our transportation industry. It is no longer a question of IF Big Data will be useful, but instead WHY is it useful and HOW can we best apply it. This presentation aims to address how we can leverage existing services and available partnerships in transportation, consider new and emerging technologies, and determine strategy for what’s to come in transportation, including connected and autonomous vehicles. While it may be a huge challenge to solve transportation problems with Big Data, it can help us make better travel decisions today and plan for better infrastructure tomorrow.
Yuan ding resume
This document is a resume for Yuan Ding, who is seeking software engineering opportunities. It outlines their education, including an expected M.S. in Electrical and Computer Engineering from Carnegie Mellon University in May 2017 and a B.S. in Electronic Science and Engineering from Nanjing University in June 2015. It also lists relevant coursework and skills in areas like Java, Python, JavaScript, databases, frameworks, tools, and cloud computing. Finally, it describes several projects demonstrating experience with technologies like Django, MySQL, Redis, Elastic Search, Play!, Scala, Hibernate, MapReduce, Spark, Kafka, and Samza.
Incrementally streaming rdbms data to your data lake automagically
Incrementally streaming rdbms data to your data lake automagically using Apache NiFi to load Oracle data to Apache Hive, Apache Kudu, Apache Impala, Apache HDFS
Netsoft19 Keynote: Fluid Network Planes
This document provides an overview of network refactoring and offloading trends, including fluid network planes. It discusses the evolution of SDN from 2009 to 2019 and concepts like network softwarization. Instances of fluid network planes are described, such as RouteFlow, NFV layers, and VNF offloading to hardware or multi-vendor P4 fabrics. The document also covers slicing for IoT analytics and references recent works on in-network computing, fast connectivity recovery, and scaling distributed machine learning with in-network aggregation.
Modelers Datahub: Tool to manage model data
KAPSARC is an independent research center focused on energy economics, policy, technology and the environment. It conducts research collaboratively with international partners to find solutions for effective and productive use of energy to enable economic and social progress globally. The document discusses KAPSARC's DataHub, which is a tool for centralizing data from different systems, tracking changes, and providing API and UI access for researchers and their models.
Advanced Analytics and Machine Learning with Data Virtualization
Watch here: https://bit.ly/3719Bi7
Advanced data science techniques, like machine learning, have proven an extremely useful tool to derive valuable insights from existing data. Platforms like Spark, and complex libraries for R, Python and Scala put advanced techniques at the fingertips of the data scientists. However, these data scientists spent most of their time looking for the right data and massaging it into a usable format. Data virtualization offers a new alternative to address these issues in a more efficient and agile way.
Attend this webinar and learn:
-How data virtualization can accelerate data acquisition and massaging, providing the data scientist with a powerful tool to complement their practice
- How popular tools from the data science ecosystem: Spark, Python, Zeppelin, Jupyter, etc. integrate with Denodo
- How you can use the Denodo Platform with large data volumes in an efficient way
-About the success McCormick has had as a result of seasoning the Machine Learning and Blockchain Landscape with data virtualization
Evolving Beyond the Data Lake: A Story of Wind and Rain
This document discusses how companies are increasingly investing in next-generation technologies like big data, cloud computing, and software/hardware related to these areas. It notes that 90% of data will be on next-gen technologies within four years. It then discusses how a converged data platform can help organizations gain insights from both historical and real-time data through applications that combine operational and analytical uses. Key benefits include the ability to seamlessly access and analyze both types of data.
Easy SPARQLing for the Building Performance Professional
Slides of Martin Kaltenböcks (SWC) presentation at SEMANTiCS2014 conference in Leipzig on 5th of September 2014 about the 'Tool for Building Energy Performance Scenarios' of GBPN (Global Buildings Performance Network, http://gbpn.org) that provides a prediction tool for buildings performance worldwide by making use of Linked Open Data (LOD).
VoltDB and Flytxt Present: Building a Single Technology Platform for Real-Tim...
Webinar presented by Prateek Kapadia, CTO, Flytxt and Ryan Betts, CTO, VoltDB.
Prateek and Ryan highlight the technology stack for Fast + Big data and iterative analytics using Hadoop and VoltDB. In addition, they will outline use cases for Big Data analytics with Hadoop, iterative analytics and machine learning using Spark and real-time analytics using VoltDB supplemented by Big Data + iterative analytics.
Similar to The Future of Data Pipelines (20)
How Comcast Turns Big Data into Real Time Operational Insights: Winter Olympi...
How Comcast Turns Big Data into Real Time Operational Insights: Winter Olympi...
Present and future of unified, portable, and efficient data processing with A...
Present and future of unified, portable, and efficient data processing with A...
Integrating Semantic Web in the Real World: A Journey between Two Cities
Integrating Semantic Web in the Real World: A Journey between Two Cities
Network of Networks Discussion and Charter Document
Network of Networks Discussion and Charter Document
Data-Driven Transformation: Leveraging Big Data at Showtime with Apache Spark
Data-Driven Transformation: Leveraging Big Data at Showtime with Apache Spark
Text Analytics & Linked Data Management As-a-Service
Text Analytics & Linked Data Management As-a-Service
Incrementally streaming rdbms data to your data lake automagically
Incrementally streaming rdbms data to your data lake automagically
Advanced Analytics and Machine Learning with Data Virtualization
Advanced Analytics and Machine Learning with Data Virtualization
Evolving Beyond the Data Lake: A Story of Wind and Rain
Evolving Beyond the Data Lake: A Story of Wind and Rain
Easy SPARQLing for the Building Performance Professional
Easy SPARQLing for the Building Performance Professional
VoltDB and Flytxt Present: Building a Single Technology Platform for Real-Tim...
VoltDB and Flytxt Present: Building a Single Technology Platform for Real-Tim...
More from All Things Open
Building Reliability - The Realities of Observability
Presented at the ATO RTP Meetup
Presented by Jeremy Proffit, Director of DevSecOps & SRE for Customer Care and Communications, Ally
Title: Building Reliability - The Realities of Observability
Abstract: Join me as we discuss true observability, learn what works and what doesn't. We'll not only discuss dashboards, monitoring and alerting, but how these can be built by automation or included in your IAC modules. We'll talk about how to properly alert staff based on priority to keep your staff and yourself sane. And even discuss architecture and how it impacts reliably and why serverless isn't always the best at being reliable.
Modern Database Best Practices
Presented at the ATO RTP Meetup
Presented by Peter Zaitsev, Founder of Percona
Title: Modern Database Best Practices
Abstract: There are now more Database choices available for developers than ever before - there are general purpose databases and specialized databases, single node and distributed databases, Open Source, Proprietary databases and databases available exclusively in the cloud. In this presentation we will cover the best practices of choosing database(s) for your applications, best practices as it comes to application development as well as managing those databases to achieve best possible performance, security, availability at the lowest cost.
Open Source and Public Policy
All Things Open 2023
Presented at All Things Open 2023
Presented by Deb Bryant - Open Source Initiative, Patrick Masson - Apereo Foundation, Stephen Jacobs - Rochester Institute of Technology, Ruth Suehle - SAS, & Greg Wallace - FreeBSD Foundation
Title: Open Source and Public Policy
Abstract: New regulations in the software industry and adjacent areas such as AI, open science, open data, and open education are on the rise around the world. Cyber Security, societal impact of AI, data and privacy are paramount issues for legislators globally. At the same time, the COVID-19 pandemic drove collaborative development to unprecedented levels and took Open Source software, open research, open content and data from mainstream to main stage, creating tension between public benefit and citizen safety and security as legislators struggle to find a balance between open collaboration and protecting citizens.
Historically, the open source software community and foundations supporting its work have not engaged in policy discussions. Moving forward, thoughtful development of these important public policies whilst not harming our complex ecosystems requires an understanding of how our ecosystem operates. Ensuring stakeholders without historic benefit of representation in those discussions becomes paramount to that end.
Please join our open discussion with open policy stakeholders working constructively on current open policy topics. Our panelists will provide a view into how oss foundations and other open domain allies are now rising to this new challenge as well as seizing the opportunity to influence positive changes to the public’s benefit.
Topics: Public Policy, Open Science, Open Education, current legislation in the US and EU, US interest in OSS sustainability, intro to the Open Policy Alliance
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...
This document summarizes a presentation about graph-quilt, an open source GraphQL orchestrator library. It discusses the challenges of building a GraphQL orchestrator to unify data from multiple services. Graph-quilt addresses this by allowing services to register their GraphQL schemas and composing them into a unified schema. It also supports features like remote schema extensions, authorization, and adapting existing REST APIs. The presenters believe graph-quilt provides a flexible way to build GraphQL gateways and help more clients adopt GraphQL.
The State of Passwordless Auth on the Web - Phil Nash
Presented at All Things Open 2023
Presented by Phil Nash - Sonar
Title: The State of Passwordless Auth on the Web
Abstract: Can we get rid of passwords yet? They make for a poor user experience and users are notoriously bad with them. The advent of WebAuthn has brought a passwordless world closer, but where do we really stand?
In this talk we'll explore the current user experience of WebAuthn and the requirements a user has to fulfil to authenticate without a password. We'll also explore the fallbacks and safeguards we can use to make the password experience better and more secure. By the end of the session you'll have a vision of how authentication could look in the future and a blueprint for how to build the best auth experience today.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Total ReDoS: The dangers of regex in JavaScript
Presented at All Things Open 2023
Presented by Phil Nash - Sonar
Title: Total ReDoS: The dangers of regex in JavaScript
Abstract: Regular expressions are complicated and can be hard to learn. On top of that, they can also be a security risk; writing the wrong pattern can open your application up to denial of service attacks. One token out of place and you invite in the dreaded ReDoS.
But how can a regular expression cause this? In this talk we’ll track down the patterns that can cause this trouble, explain why they are an issue and propose ways to fix them now and avoid them in the future. Together we’ll demystify these powerful search patterns and keep your application safe from expressions that behave in a way that is anything but regular.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
What Does Real World Mass Adoption of Decentralized Tech Look Like?
Presented at All Things Open 2023
Presented by Karl Mozurkewich - Storj
Title: What Does Real World Mass Adoption of Decentralized Tech Look Like?
Abstract: We delve into the transformative potential of decentralized technology. Beginning with a brief overview of the rise of centralization with the advent of the internet and the counter-shift marked by blockchain we explore the intrinsic characteristics of decentralized and distributed systems, such as trustless operations, peer-to-peer networks, and enterprise application scalability. Various sectors, including finance, supply chains, media and entertainment, data science and cloud infrastructure are on the brink of disruption. The societal implications are vast, with the potential for greater individual empowerment, a greener planet and more viable resource utilization, but concerns about data security persist.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
How to Write & Deploy a Smart Contract
Presented at All Things Open 2023
Presented by Anastasia Lalamentik - Kaleido
Title: How to Write & Deploy a Smart Contract
Abstract: In this talk, Anastasia Lalamentik, Full Stack Engineer at Kaleido, will walk through how Ethereum smart contracts work and go over related concepts like gas fees, the Ethereum Virtual Machine (EVM), the block explorer, and the Solidity programming language. This is vital to anyone who wants to build a blockchain app and is a great introduction to blockchain technology for newcomers to the space.
By the end of the talk, attendees will better understand how to:
- Write a simple smart contract
- Deploy their smart contract to an Ethereum test network through the latest tools like Hardhat and the MetaMask wallet
- Test interactions with their deployed smart contract and ensure that everything is working properly
Additionally, participants will get to interact with Anastasia's deployed smart contract at the end of the talk. Anastasia’s past talks have attracted and have been attended by a diverse group of participants with a range of experience in the space.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow
Presented at All Things Open 2023
Presented by Paul Brebner - Instaclustr (by Spot by NetApp)
Title: Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow
Abstract: In this talk we’ll build a Drone delivery application, and then use it to do some Machine Learning “on the fly”.
In the 1st part of the talk, we'll build a real-time Drone Delivery demonstration application using a combination of two open-source technologies: Uber’s Cadence (for stateful, scheduled, long-running workflows), and Apache Kafka (for fast streaming data).
With up to 2,000 (simulated) drones and deliveries in progress at once this application generates a vast flow of spatio-temporal data.
In the 2nd part of the talk, we'll use this platform to explore Machine Learning (ML) over streaming and drifting Kafka data with TensorFlow to try and predict which shops will be busy in advance.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
DEI Challenges and Success
Presented at the All Things Open 2023 Inclusion and Diversity in Open Source Event
Presented by Efraim Marquez-Arreaza - Red Hat
Title: DEI Challenges and Success
Abstract: In today's world, many companies and organizations have Diversity, Equity and Inclusion (DEI) communities. Red Hat Unidos is a DEI community focused on advocating for the Hispanic/Latine community. In this talk, we would like to share our challenges and success during the past 4-years and plans for the future.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Scaling Web Applications with Background
Presented at All Things Open 2023
Presented by Lydia Cupery - HubSpot
Title: Scaling Web Applications with Background Jobs: Takeaways from Generating a Huge PDF
Abstract: Do you need to perform time-consuming or CPU-intensive processes in your web application but are concerned about performance? That’s where background jobs come in. By offloading resource-intensive tasks to separate worker processes, you can improve the scalability of your web application.
In this talk, I'll share my experience of using background jobs to scale our web application. I'll discuss the challenges my team faced that led us to adopt background jobs. Then, I'll share practical tips on how to design background jobs for CPU-intensive or time-consuming processes, such as generating huge PDFs and batch emailing. I'll wrap up by going over the performance and cost tradeoffs of background jobs.
I'll use Typescript, Express, and Heroku as examples in this talk, but the concepts and best practices that I'll share are applicable to other languages and tools.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Supercharging tutorials with WebAssembly
Presented at All Things Open 2023
Presented by Robert Aboukhalil - CZI
Title: Supercharging tutorials with WebAssembly
Abstract: sandbox.bio is a free platform that features interactive command-line tutorials for bioinformatics. This talk is a deep-dive into how sandbox.bio was built, with a focus on how WebAssembly enabled bringing command-line tools like awk and grep to the web. Although these tools were originally written in C/C++, they all run directly in the browser, thanks to WebAssembly! And since the computations run on each user's computer, this makes the application highly scalable and cost-effective.
Along the way, I'll discuss how WebAssembly works and how to get started using it in your own applications. The talk will also cover more advanced WebAssembly features such as threads and SIMD, and will end with a discussion of WebAssembly's benefits and pitfalls (it's a powerful technology, but it's not always the right tool!).
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Using SQL to Find Needles in Haystacks
Presented at All Things Open 2023
Presented by K.S. Bhaskar - YottaDB LLC
Title: Using SQL to Find Needles in Haystacks
Abstract: Database journal files capture every update to a database. A database of a few hundred GB can generate GBs worth of journal files every minute at busy times. Troubleshooting and forensices, especially of rare and intermittent problems, such as which process made what update and when, is an exercise of finding needles in haystacks. A similar problem exists with syslogs. A solution is to load the journal files and syslogs into a database, and use SQL to query the database. Bhaskar will present and demonstrate this with a 100% FOSS stack.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Configuration Security as a Game of Pursuit Intercept
The document discusses configuration security as a game of pursuit-evasion and intercept. It was presented by Wes Widner, Principal Engineer at Automox. The document includes a JSON policy snippet with an ID, statement, actions, effects, resources, and principal allowing the GetObject action on all objects in an S3 bucket for all principals. It has page numbers at the bottom indicating it is from a larger presentation.
Scaling an Open Source Sponsorship Program
Presented at All Things Open 2023
Presented by Carol Huang & Mike Fix - Stripe
Title: Scaling an Open Source Sponsorship Program
Abstract: We already know this: the open-source ecosystem needs further monetary investment from the companies that benefit most from it. Likewise, companies say they want to participate in these initiatives, but find it hard to dedicate resources to open source funding when there isn’t a clear ROI.
This talk discusses how the Open Source Program Office at Stripe built a scalable, sustainable open source sponsorship model that aligns internal company incentives with those of open source maintainers and the community at large. We go over the unique “platformization” of our OSPO that allowed us to create multiple funding models, such as BYOB (Bring Your Own Budget), and share lessons learned from this experience as well as other OSPOs.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Build Developer Experience Teams for Open Source
Presented at All Things Open 2023
Presented by Arundeep Nagaraj - Amazon Web Services (AWS)
Title: Build Developer Experience Teams for Open Source
Abstract: Open Source has become the default strategy for many IT organizations and Enterprises. However, the constant challenge with Open Source leaders of these organizations has been -
How is my product's developer experience?
Is this the right metric to track?
How can I scale my team to support our products better?
How can I add automation to scale redundant workflows?
If my product involves working with developers, how can I scale to the complexity of the requests and reduce Engineering bandwidth?
The challenges within support of open source products continues to magnify depending on the end user persona whether they are consumers or contributors to your product. Consumers utilize your product, SDK's and API's and are blocked with using it or run into issues, whereas contributors are advanced users of your software that understands the codebase to provide a meaningful contribution back to the product.
The answer to the above is to look at Open Source support as a first-class citizen of your corporate support strategy. To employ the right level of developer focused support as opposed to traditional infrastructure based support is key to scale to the amount of developers using your product. Supporting customers in the open involves more than pure support - building customer / developer experiences (DX) in the open (across platforms and communities) that pivots over the ability of your product's users or developers to be focused on the end-to-end value add. This helps with your active developer growth and retention of users.
Key Takeaways:
- IT leaders of Open Source will learn to employ strategies to build a DX team that engages on multiple platforms
- Work on identifying accurate metrics for product and organization
- Innovate on platforms such as Discord to build a bot and a dashboard
- Ability to leverage customer feedback and iterate over the customer success flywheel
- Distinguish between DX and Developer Advocacy (DA)
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Deploying Models at Scale with Apache Beam
Presented at All Things Open 2023
Presented by Danny McCormick - Google
Title: Deploying Models at Scale with Apache Beam
Abstract: Apache Beam is an open source tool for building distributed scalable data pipelines. This talk will explore how Beam can be used to perform common machine learning tasks, with a heavy focus on running inference at scale. The talk will include a demo component showing how Beam can be used to deploy and update models efficiently on both CPUs and GPUs for inference workloads.
An attendee can expect to leave this talk with a high level understanding of Beam, the challenges of deploying models at scale, and the ability to use Beam to easily parallelize their inference workloads.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Sudo – Giving access while staying in control
Presented at All Things Open 2023
Presented by Peter Czanik - One Identity
Title: Sudo – Giving access while staying in control
Abstract: Sudo is used by millions to control and log administrator access to systems, but using the default configuration only, there are plenty of blind spots. Using the latest features in sudo let you watch some previously blind spots and control access to them. Here are four major new features, which arrived since the 1.9.0 release, allowing you see your blind spots:
- configuring a working directory or chroot within sudo often makes full shell access redundant
- JSON-formatted logs give you more details on events and are easier to act on
- relays in sudo_logsrvd make session recording collection more secure and reliable
- you can log and control sub-commands executed by the command run through sudo
Let us take a closer look at each of these.
Previously, there were quite a few situations where you had to give users full shell access through sudo. Typical examples include when you need to run a command from a given directory, or running commands in a chroot environment. You can now configure the working directory or the chroot directory and give access only to the command the user really needs.
Logging is a central role of sudo, to see who did what on the system. Using JSON-formatted log messages gives you even more information about events. What is even more: structured logs are easier to act on. Setting up alerting for suspicious events is much easier when you have a single parser to configure for any kind of sudo logs. You can collect sudo logs not only by local syslog, but also by using sudo_logsrvd, the same application used to collect session recordings.
Speaking of session recordings: instead of using a single central server, you can now have multiple levels of sudo_logsrvd relays between the client and the final destination. This allows session collection even if the central server is unavailable, providing you with additional security. It also makes your network configuration simpler.
Finally, you can log sub-commands executed from the command started through sudo. You can see commands started from a shell. No more unnoticed shell access from text editors. Best of all: you can also intercept sub-commands.
These are just a few of the most prominent features helping you to watch and control previous blind spots on your systems. See these and other possibilities in action in some live demos during our presentation.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Fortifying the Future: Tackling Security Challenges in AI/ML Applications
Presented at All Things Open 2023
Presented by Christine Abernathy - F5, Inc.
Title: Fortifying the Future: Tackling Security Challenges in AI/ML Applications
Abstract: As Artificial Intelligence (AI) and Machine Learning (ML) applications continue to surge, it is crucial to be aware of and address the security risks associated with these technologies. In this talk, Christine will explore AI/ML failure modes, threats, and mitigation strategies. She will guide you through the fundamentals of ML models then introduce you to key security challenges such as adversarial attacks, data poisoning, model inversion, model stealing, and membership inference attacks, using real-world examples to demonstrate their potential impact.
Christine will also discuss privacy and ethical considerations in ML, touching upon techniques like federated learning and shedding light on the current regulatory landscape surrounding security risks. If you are developing AI/ML applications or incorporating AI/ML components into your technology stack, check out this talk. You will walk away with a deeper understanding of the current AI/ML security landscape and a toolkit to help you address these risks, enabling you to build safer, more secure, and privacy-aware applications.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...
Presented at All Things Open 2023
Presented by Carlos Santana - AWS
Title: Securing Cloud Resources Deployed with Control Planes on Kubernetes using Governance and Policy as Code
Abstract: Are you concerned about the security of your cloud resources deployed on Kubernetes? Are you struggling to ensure compliance with regulatory requirements while managing your cloud infrastructure? If yes, then this talk is for you!
We will discuss how to secure cloud resources deployed with Crossplane on Kubernetes using Governance and Policy as Code. We will explore how to leverage Governance and Policy as Code tools like Rego, Kyverno, and OPA to ensure security and compliance.
By the end of this talk, you will have a better understanding of the challenges associated with securing cloud resources deployed with Crossplane or ACK on Kubernetes, the importance of Governance and Policy as Code in ensuring security and compliance, and why it is critical to use open source and open standards in these technologies.
Find more info about All Things Open:
On the web: https://www.allthingsopen.org/
Twitter: https://twitter.com/AllThingsOpen
LinkedIn: https://www.linkedin.com/company/all-things-open/
Instagram: https://www.instagram.com/allthingsopen/
Facebook: https://www.facebook.com/AllThingsOpen
Mastodon: https://mastodon.social/@allthingsopen
Threads: https://www.threads.net/@allthingsopen
2023 conference: https://2023.allthingsopen.org/
More from All Things Open (20)
Building Reliability - The Realities of Observability
Building Reliability - The Realities of Observability
Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...
Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...
The State of Passwordless Auth on the Web - Phil Nash
The State of Passwordless Auth on the Web - Phil Nash
What Does Real World Mass Adoption of Decentralized Tech Look Like?
What Does Real World Mass Adoption of Decentralized Tech Look Like?
Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow
Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow
Configuration Security as a Game of Pursuit Intercept
Configuration Security as a Game of Pursuit Intercept
Fortifying the Future: Tackling Security Challenges in AI/ML Applications
Fortifying the Future: Tackling Security Challenges in AI/ML Applications
Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...
Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...
Recently uploaded
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on integration of Salesforce with Bonterra Impact Management.
Interested in deploying an integration with Salesforce for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
June Patch Tuesday
Ivanti’s Patch Tuesday breakdown goes beyond patching your applications and brings you the intelligence and guidance needed to prioritize where to focus your attention first. Catch early analysis on our Ivanti blog, then join industry expert Chris Goettl for the Patch Tuesday Webinar Event. There we’ll do a deep dive into each of the bulletins and give guidance on the risks associated with the newly-identified vulnerabilities.
System Design Case Study: Building a Scalable E-Commerce Platform - Hiike
This case study explores designing a scalable e-commerce platform, covering key requirements, system components, and best practices.
Best 20 SEO Techniques To Improve Website Visibility In SERP
Boost your website's visibility with proven SEO techniques! Our latest blog dives into essential strategies to enhance your online presence, increase traffic, and rank higher on search engines. From keyword optimization to quality content creation, learn how to make your site stand out in the crowded digital landscape. Discover actionable tips and expert insights to elevate your SEO game.
Artificial Intelligence for XMLDevelopment
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Simplify your search for a reliable Python development partner! This list presents the top 10 trusted US providers offering comprehensive Python development services, ensuring your project's success from conception to completion.
Skybuffer SAM4U tool for SAP license adoption
Manage and optimize your license adoption and consumption with SAM4U, an SAP free customer software asset management tool.
SAM4U, an SAP complimentary software asset management tool for customers, delivers a detailed and well-structured overview of license inventory and usage with a user-friendly interface. We offer a hosted, cost-effective, and performance-optimized SAM4U setup in the Skybuffer Cloud environment. You retain ownership of the system and data, while we manage the ABAP 7.58 infrastructure, ensuring fixed Total Cost of Ownership (TCO) and exceptional services through the SAP Fiori interface.
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
Presentation of the OECD Artificial Intelligence Review of Germany
Consult the full report at https://www.oecd.org/digital/oecd-artificial-intelligence-review-of-germany-609808d6-en.htm
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
When it comes to unit testing in the .NET ecosystem, developers have a wide range of options available. Among the most popular choices are NUnit, XUnit, and MSTest. These unit testing frameworks provide essential tools and features to help ensure the quality and reliability of code. However, understanding the differences between these frameworks is crucial for selecting the most suitable one for your projects.
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean Lotus cyber threat actors represent a sophisticated, persistent, and politically motivated group that poses a significant risk to organizations and individuals in the Southeast Asian region. Their continuous evolution and adaptability underscore the need for robust cybersecurity measures and international cooperation to identify and mitigate the threats posed by such advanced persistent threat groups.
Finale of the Year: Apply for Next One!
Presentation for the event called "Finale of the Year: Apply for Next One!" organized by GDSC PJATK
Programming Foundation Models with DSPy - Meetup Slides
Prompting language models is hard, while programming language models is easy. In this talk, I will discuss the state-of-the-art framework DSPy for programming foundation models with its powerful optimizers and runtime constraint system.
Introduction of Cybersecurity with OSS at Code Europe 2024
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
A Mix Chart displays historical data of numbers in a graphical or tabular form. The Kalyan Rajdhani Mix Chart specifically shows the results of a sequence of numbers over different periods.
Main news related to the CCS TSI 2023 (2023/1695)
An English 🇬🇧 translation of a presentation to the speech I gave about the main changes brought by CCS TSI 2023 at the biggest Czech conference on Communications and signalling systems on Railways, which was held in Clarion Hotel Olomouc from 7th to 9th November 2023 (konferenceszt.cz). Attended by around 500 participants and 200 on-line followers.
The original Czech 🇨🇿 version of the presentation can be found here: https://www.slideshare.net/slideshow/hlavni-novinky-souvisejici-s-ccs-tsi-2023-2023-1695/269688092 .
The videorecording (in Czech) from the presentation is available here: https://youtu.be/WzjJWm4IyPk?si=SImb06tuXGb30BEH .
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
Predictive maintenance is a proactive approach that anticipates equipment failures before they happen. At the forefront of this innovative strategy is Artificial Intelligence (AI), which brings unprecedented precision and efficiency. AI in predictive maintenance is transforming industries by reducing downtime, minimizing costs, and enhancing productivity.
Recently uploaded (20)
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
System Design Case Study: Building a Scalable E-Commerce Platform - Hiike
System Design Case Study: Building a Scalable E-Commerce Platform - Hiike
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
The Future of Data Pipelines
- 1. The Future of Data Pipelines | All Things Open 2020 | 2020-02-24 The Future of Data Pipelines John Hammink Developer Advocate, Evangelist @ Aiven john@aiven.io All Things Open Meetup 2020, Raleigh NC
- 2. The Future of Data Pipelines ● Let’s start with where the future looked from about a year ago ● Let’s assume that pub-sub and particularly technologies like Kafka have created a pivot point ● Let’s follow up with a discussion of trends we’re all seeing The Future of Data Pipelines | All Things Open 2020 | 2020-02-24
- 3. The Future of Data Pipelines ● Background ● Learnings ● Some numbers on data ● Functionality ● Design ● Compliance ● Reliability/Performance/Usability/Other ● Conclusions ● Q & A The Future of Data Pipelines | All Things Open 2020 | 2020-02-24
- 4. An arbitrary timeline of data The Future of Data Pipelines | All Things Open 2020 | 2020-02-24
- 5. What have we learned? The Future of Data Pipelines | All Things Open 2019 | 2019-10-14 Pipeline data has grown/transformed in terms of:
- 6. The Future of Data Pipelines | All Things Open 2019 | 2019-10-14
- 7. The Future of Data Pipelines | All Things Open 2019 | 2019-10-14 Core-to-edge computing
- 8. The Future of Data Pipelines | All Things Open 2019 | 2019-10-14 Functionality ● can autoscale, shard, tolerate partitions ● can capture, fix and requeue errored events ● can be troubleshot and configured on fly ● data can be used for AI/ML ● agnostic: accomodates all possible formats ● can self-perpetuate and automate continuous improvements
- 9. source: kafka.apache.org Source: Martin Kleppmann’s Kafka Summit 2018 Presentation, Is Kafka a Database? The Future of Data Pipelines | All Things Open 2019 | 2019-10-14
- 10. Other design considerations ● Kill switch - when things go wrong; a way to stem the data flow. ● Query on the streams - like KSQL, Apache Flink, SQL on Amazon Kinesis, etc. Ultimately: one query interface for all data. ● Need to accomodate - core-to-edge, growing data volume, near real-time velocity, bi-directionality ● Distributed ledgers - pipelines can support these as tunably consistent distributed datastores The Future of Data Pipelines | All Things Open 2019 | 2019-10-14
- 11. Compliance The Future of Data Pipelines | All Things Open 2019 | 2019-10-14
- 12. Reliability/Performance/Usability/Other The Future of Data Pipelines | All Things Open 2019 | 2019-10-14 ● MTTF → ∞ ● Components: from interpreted to native? ● Metadata: will continue to transform, to offer even more space savings. ● Pipelines: from hard-coded to drag and drop designs? ● Which data is left volatile and which is stored?
- 13. Wrapping up The Future of Data Pipelines | All Things Open 2019 | 2019-10-14 Check out The Future of Data Pipelines at https://aiven.io/blog/the-future-of-data-pipelines/
- 14. Discussion The Future of Data Pipelines | All Things Open 2019 | 2019-10-14