Hkube
•
0 likes•1,985 views
HKube is an open source framework that runs algorithms on Kubernetes as distributed pipelines. It handles scheduling algorithms across nodes, prioritizing tasks, and provides a simple API. The core modules include an API server, pipeline driver to build graphs and manage flow, and a resource manager to allocate pods. Algorithms run as containers managed by workers that communicate results. The current version provides dynamic allocation and monitoring, and future plans include additional orchestrator support, affinity, caching, and a public algorithm store.
Report
Share
Report
Share
Download to read offline
Recommended
KFServing and Kubeflow Pipelines
The document discusses a Kubeflow Pipelines component for Kubeflow Serving (KFServing) that allows usage of KFServing within Kubeflow Pipelines. The component uses the KFServing Python package and API to deploy InferenceServices and perform canary rollouts. A sample pipeline is shown that uses the component to deploy a TensorFlow model. The document also analyzes the component and discusses passing InferenceService YAML as the most flexible way to deploy models with full customizability.
Kubeflow repos
Kubeflow is an open-source project that makes it easy to deploy and manage machine learning workloads on Kubernetes. The Kubeflow organization on GitHub contains many repositories that provide tools and services for Kubeflow. These repositories include ones for the main Kubeflow deployment, documentation, examples, a CLI for deployment, metadata tracking, testing infrastructure, common libraries, a frontend dashboard, machine learning pipelines, TensorFlow and PyTorch operators, hyperparameter tuning, and serverless inferencing.
Kubeflow Control Plane 中文
Kubeflow控制器可以部署和管理Kubeflow。它可以通过命令行或Operator在不同环境中安装Kubeflow,包括配置应用程序集合。Kfctl源代码仓库和manifests仓库分别提供了通过KfDef配置文件部署Kubeflow的能力和应用程序清单。Kubeflow Operator可以监视KfDef自定义资源和管理Kubeflow生命周期。
Kubeflow Pipelines (with Tekton)
Deep dive into Kubeflow Pipelines, and details about Tekton backend implementation for KFP, including compiler, logging, artifacts and lineage tracking
Realizing the promise of portability with Apache Beam
The world of big data involves an ever changing field of players. Much as SQL stands as a lingua franca for declarative data analysis, Apache Beam (incubating) aims to provide a portable standard for expressing robust, out-of-order data processing pipelines in a variety of languages across a variety of platforms.
In this talk, I will:
Cover briefly the capabilities of the Beam model for data processing and integration with IOs, as well as the current state of the Beam ecosystem.
Discuss the benefits Beam provides regarding portability and ease-of-use.
Demo the same Beam pipeline running on multiple runners in multiple deployment scenarios (e.g. Apache Flink on Google Cloud, Apache Spark on AWS, Apache Apex on-premise).
Give a glimpse at some of the challenges Beam aims to address in the future.
Leveraging Open Source GIS with Python: A QGIS Approach
The accompanying slide presentation to a webinar I gave back in may showing the power of Python with open source tools like Quantum GIS (QGIS) and PostGIS.
Portable batch and streaming pipelines with Apache Beam (Big Data Application...
Apache Beam is a top-level Apache project which aims at providing a unified API for efficient and portable data processing pipeline. Beam handles both batch and streaming use cases and neatly separates properties of the data from runtime characteristics, allowing pipelines to be portable across multiple runtimes, both open-source (e.g., Apache Flink, Apache Spark, Apache Apex, ...) and proprietary (e.g., Google Cloud Dataflow). This talk will cover the basics of Apache Beam, describe the main concepts of the programming model and talk about the current state of the project (new python support, first stable version). We'll illustrate the concepts with a use case running on several runners.
SaileshKumar_Kumar_Resume
Sailesh Kumar is a recent graduate of USC with an M.S. in Electrical Engineering specializing in VLSI. He has experience with digital hardware design, verification, and testing through directed research and projects implementing algorithms and processors in Verilog, VHDL, and C/C++. He is looking for opportunities in fields involving digital design, FPGAs, ASICs, and systems-on-chip.
Recommended
KFServing and Kubeflow Pipelines
The document discusses a Kubeflow Pipelines component for Kubeflow Serving (KFServing) that allows usage of KFServing within Kubeflow Pipelines. The component uses the KFServing Python package and API to deploy InferenceServices and perform canary rollouts. A sample pipeline is shown that uses the component to deploy a TensorFlow model. The document also analyzes the component and discusses passing InferenceService YAML as the most flexible way to deploy models with full customizability.
Kubeflow repos
Kubeflow is an open-source project that makes it easy to deploy and manage machine learning workloads on Kubernetes. The Kubeflow organization on GitHub contains many repositories that provide tools and services for Kubeflow. These repositories include ones for the main Kubeflow deployment, documentation, examples, a CLI for deployment, metadata tracking, testing infrastructure, common libraries, a frontend dashboard, machine learning pipelines, TensorFlow and PyTorch operators, hyperparameter tuning, and serverless inferencing.
Kubeflow Control Plane 中文
Kubeflow控制器可以部署和管理Kubeflow。它可以通过命令行或Operator在不同环境中安装Kubeflow,包括配置应用程序集合。Kfctl源代码仓库和manifests仓库分别提供了通过KfDef配置文件部署Kubeflow的能力和应用程序清单。Kubeflow Operator可以监视KfDef自定义资源和管理Kubeflow生命周期。
Kubeflow Pipelines (with Tekton)
Deep dive into Kubeflow Pipelines, and details about Tekton backend implementation for KFP, including compiler, logging, artifacts and lineage tracking
Realizing the promise of portability with Apache Beam
The world of big data involves an ever changing field of players. Much as SQL stands as a lingua franca for declarative data analysis, Apache Beam (incubating) aims to provide a portable standard for expressing robust, out-of-order data processing pipelines in a variety of languages across a variety of platforms.
In this talk, I will:
Cover briefly the capabilities of the Beam model for data processing and integration with IOs, as well as the current state of the Beam ecosystem.
Discuss the benefits Beam provides regarding portability and ease-of-use.
Demo the same Beam pipeline running on multiple runners in multiple deployment scenarios (e.g. Apache Flink on Google Cloud, Apache Spark on AWS, Apache Apex on-premise).
Give a glimpse at some of the challenges Beam aims to address in the future.
Leveraging Open Source GIS with Python: A QGIS Approach
The accompanying slide presentation to a webinar I gave back in may showing the power of Python with open source tools like Quantum GIS (QGIS) and PostGIS.
Portable batch and streaming pipelines with Apache Beam (Big Data Application...
Apache Beam is a top-level Apache project which aims at providing a unified API for efficient and portable data processing pipeline. Beam handles both batch and streaming use cases and neatly separates properties of the data from runtime characteristics, allowing pipelines to be portable across multiple runtimes, both open-source (e.g., Apache Flink, Apache Spark, Apache Apex, ...) and proprietary (e.g., Google Cloud Dataflow). This talk will cover the basics of Apache Beam, describe the main concepts of the programming model and talk about the current state of the project (new python support, first stable version). We'll illustrate the concepts with a use case running on several runners.
SaileshKumar_Kumar_Resume
Sailesh Kumar is a recent graduate of USC with an M.S. in Electrical Engineering specializing in VLSI. He has experience with digital hardware design, verification, and testing through directed research and projects implementing algorithms and processors in Verilog, VHDL, and C/C++. He is looking for opportunities in fields involving digital design, FPGAs, ASICs, and systems-on-chip.
Gpars - the coolest bits
Slides from my presentation about GPars given at Warsaw Groovy User Group (2014.10.16)
Source code for demo sessions available here:
https://github.com/ArturGajowy/gpars-coolest-bits
Apache Spark MLlib 2.0 Preview: Data Science and Production
This talk highlights major improvements in Machine Learning (ML) targeted for Apache Spark 2.0. The MLlib 2.0 release focuses on ease of use for data science—both for casual and power users. We will discuss 3 key improvements: persisting models for production, customizing Pipelines, and improvements to models and APIs critical to data science.
(1) MLlib simplifies moving ML models to production by adding full support for model and Pipeline persistence. Individual models—and entire Pipelines including feature transformations—can be built on one Spark deployment, saved, and loaded onto other Spark deployments for production and serving.
(2) Users will find it much easier to implement custom feature transformers and models. Abstractions automatically handle input schema validation, as well as persistence for saving and loading models.
(3) For statisticians and data scientists, MLlib has doubled down on Generalized Linear Models (GLMs), which are key algorithms for many use cases. MLlib now supports more GLM families and link functions, handles corner cases more gracefully, and provides more model statistics. Also, expanded language APIs allow data scientists using Python and R to call many more algorithms.
Finally, we will demonstrate these improvements live and show how they facilitate getting started with ML on Spark, customizing implementations, and moving to production.
Overview of Apache Spark 2.3: What’s New? with Sameer Agarwal
Apache Spark 2.0 set the architectural foundations of Structure in Spark, Unified high-level APIs, Structured Streaming, and the underlying performant components like Catalyst Optimizer and Tungsten Engine. Since then the Spark community contributors have continued to build new features and fix numerous issues in releases Spark 2.1 and 2.2.
Continuing forward in that spirit, Apache Spark 2.3 has made similar strides too, introducing new features and resolving over 1300 JIRA issues. In this talk, we want to share with the community some salient aspects of Spark 2.3 features:
Kubernetes Scheduler Backend
PySpark Performance and Enhancements
Continuous Structured Streaming Processing
DataSource v2 APIs
Spark History Server Performance Enhancements
Introduction to Apache Beam (incubating) - DataCamp Salzburg - 7 dec 2016
This document provides an introduction to Apache Beam, a unified programming model for batch and stream data processing. It discusses Beam's programming model including PCollections, transforms, and runners. It also provides examples of writing a basic Beam pipeline in Java and running it on the Direct, Spark, and Flink runners.
Large-Scale Training with GPUs at Facebook
This document discusses large-scale distributed training with GPUs at Facebook using their Caffe2 framework. It describes how Facebook was able to train the ResNet-50 model on the ImageNet dataset in just 1 hour using 32 GPUs with 8 GPUs each. It explains how synchronous SGD was implemented in Caffe2 using Gloo for efficient all-reduce operations. Linear scaling of the learning rate with increased batch size was found to work best when gradually warming up the learning rate over the first few epochs. Nearly linear speedup was achieved using this approach on commodity hardware.
What is Spark
Quick introduction about Apache Spark and how it fits in the cognitive world, how can we use it to help cognitive solutions as well as create distributed algorithms to predict and perform other machine learning tasks.
PR workflow
This document outlines the Kubeflow pull request (PR) workflow, which involves four steps: 1) finding an issue to work on by reviewing issues on the Kubeflow GitHub repositories, 2) writing code and tests to address the issue, 3) submitting a PR with the code changes, and 4) having the PR reviewed and merged into the main repository once approved. It provides guidance on using tools like GitHub, writing tests, addressing code review comments, and resolving conflicts when merging approved PRs.
Python Streaming Pipelines on Flink - Beam Meetup at Lyft 2019
Apache Beam is a unified programming model for batch and streaming data processing that provides portability across distributed processing backends. It aims to support multiple languages like Java, Python and Go. The Beam Python SDK allows writing pipelines in Python that can run on distributed backends like Apache Flink. Lyft developed a Python SDK runner for Flink that translates Python pipelines to native Flink APIs using the Beam Fn API for communication between the SDK and runner. Future work includes improving performance of Python pipelines on JVM runners and supporting multiple languages in a single pipeline.
Apache MXNet AI
This presentation gives an overview of the Apache MXNet AI project. It explains Apache MXNet AI in terms of it's architecture, eco system, languages and the generic problems that the architecture attempts to solve.
Links for further information and connecting
http://www.amazon.com/Michael-Frampton/e/B00NIQDOOM/
https://nz.linkedin.com/pub/mike-frampton/20/630/385
https://open-source-systems.blogspot.com/
Python and GIS: Improving Your Workflow
A 40 minute talk on using Python with GIS software. Integration with ArcGIS and open source software is demonstrated. Includes links to several Python-based projects on Github. Presented at the Delaware Valley Regional Planning Commission's Information Resource Exchange Group on December 9th, 2015.
Creating Reusable Geospatial Pipelines
Geospatial pipelines in Apache Spark are difficult because of the diversity of datasets and the challenge of harmonizing on a single dataframe. We have worked over the past year to review different pipeline tools that allow us to quickly combine steps to create new workflows or operate on new datasets. We have reviewed Dagster, Apache Spark MLflow pipelines, Prefect, and our own custom solutions. The talk will go over the pros and cons of each of these solutions and will show an actionable workflow implementation that any geospatial analyst can leverage. We will show how we can leverage a pipeline to run a traditional geospatial hotspot analysis. Interactive mapping within the Databricks platform will be demonstrated.
Dynamic pricing of Lyft rides using streaming
Talk given Feb 07 2019 at SF Analytics meetup: https://www.meetup.com/SF-Big-Analytics/events/258174292/
Apache airflow
This document provides an overview of building data pipelines using Apache Airflow. It discusses what a data pipeline is, common components of data pipelines like data ingestion and processing, and issues with traditional data flows. It then introduces Apache Airflow, describing its features like being fault tolerant and supporting Python code. The core components of Airflow including the web server, scheduler, executor, and worker processes are explained. Key concepts like DAGs, operators, tasks, and workflows are defined. Finally, it demonstrates Airflow through an example DAG that extracts and cleanses tweets.
Vectorized Query Execution in Apache Spark at Facebook
This document summarizes Chen Yang's presentation on vectorized query execution in Apache Spark at Facebook. The key points are:
1) Spark is the largest SQL query engine at Facebook and uses columnar formats like ORC to improve storage efficiency.
2) Vectorized processing can improve performance over row-at-a-time processing by reducing per-row overhead and improving cache locality.
3) Facebook has implemented a vectorized ORC reader and writer in Spark that shows up to 8x speedup on microbenchmarks compared to the row-at-a-time approach.
Building Applications with Streams and Snapshots
Stream processing has been traditionally associated with realtime analytics. Modern stream processors, like Apache Flink, however, go far beyond that and give us a new approach to build applications and services as a whole.
This talk shows how to build applications on *data streams*, *state*, and *snaphots* (point-in-time views of application state) using Apache Flink. Rather than separating computation (application) and state (database), Flink manages the application logic and state as a tight pair and uses snapshots for consistent view onto the application and its state. With features like Flink's queryable state, the stream processor and database effectively become one.
This application pattern has many interesting properties: Aside from having fewer moving parts, it supports very high event rates because of its tight integration between computation and state, and its simple concurrency and recovery model. At the same time, it exposes a powerful consistency model, allows for seamless forking/updating/rollback of online applications, generalizes across historic and real-time data, and easily incorporates event time semantics and handling of late data. Finally, it allows applications to be defined in an easy way via streaming SQL.
PostgreSQL 9.4
This document summarizes a presentation about new features in PostgreSQL 9.4. It discusses enhancements for NoSQL support with JSON and GIN indexes, analytics with new aggregate functions and materialized views, increased flexibility with logical replication, easier administration using ALTER SYSTEM, and improved infrastructure for parallelization through dynamic background workers and shared memory. The presentation provides an overview of the status of 9.4 and highlights some of the major new categories of features.
Paris.rb – 07/19 – Sidekiq scaling, workers vs processes
Presentation given to Paris.RB meetup in July 2019.
- How to scale Sidekiq to handle millions of jobs?
- Is there a magic recipe to do API and computing jobs?
- Can we be cost-sensitive in scaling?
Presentation of a research protocol with observations and results.
Containerized architectures for deep learning
This document discusses containerized architectures for deep learning using Kubernetes and Kubeflow. It motivates the use of machine learning pipelines for improving data wrangling, system unification, composability, complexity management, portability, and model quality. It then summarizes popular machine learning pipeline tools like Apache Airflow, Kubeflow, TensorFlow Extended, and MLflow. It introduces Kubernetes and containers, and argues that Kubernetes and Kubeflow can provide composability, portability, and scalability for machine learning workloads. It demonstrates a deep learning model deployed on Kubernetes with Kubeflow through a doppelganger image similarity search app.
Streaming your Lyft Ride Prices - Flink Forward SF 2019
At Lyft we dynamically price our rides with a combination of various data sources, machine learning models, and streaming infrastructure for low latency, reliability and scalability. Dynamic pricing allows us to quickly adapt to real world changes and be fair to drivers (by say raising rates when there's a lot of demand) and fair to passengers (by let’s say offering to return 10 mins later for a cheaper rate). The streaming platform powers pricing by bringing together the best of two worlds using Apache Beam; ML algorithms in Python and Apache Flink as the streaming engine.
https://sf-2019.flink-forward.org/conference-program#streaming-your-lyft-ride-prices
Understanding and Improving Code Generation
Code generation is integral to Spark’s physical execution engine. When implemented, the Spark engine creates optimized bytecode at runtime improving performance when compared to interpreted execution. Spark has taken the next step with whole-stage codegen which collapses an entire query into a single function.
Hybrid Cloud, Kubeflow and Tensorflow Extended [TFX]
Kubeflow Pipelines and TensorFlow Extended (TFX) together is end-to-end platform for deploying production ML pipelines. It provides a configuration framework and shared libraries to integrate common components needed to define, launch, and monitor your machine learning system. In this talk we describe how how to run TFX in hybrid cloud environments.
Data Summer Conf 2018, “Building unified Batch and Stream processing pipeline...
Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing pipelines, and also data ingestion and integration flows, supporting for both batch and streaming use cases. In presentation I will provide a general overview of Apache Beam and programming model comparison Apache Beam vs Apache Spark.
More Related Content
What's hot
Gpars - the coolest bits
Slides from my presentation about GPars given at Warsaw Groovy User Group (2014.10.16)
Source code for demo sessions available here:
https://github.com/ArturGajowy/gpars-coolest-bits
Apache Spark MLlib 2.0 Preview: Data Science and Production
This talk highlights major improvements in Machine Learning (ML) targeted for Apache Spark 2.0. The MLlib 2.0 release focuses on ease of use for data science—both for casual and power users. We will discuss 3 key improvements: persisting models for production, customizing Pipelines, and improvements to models and APIs critical to data science.
(1) MLlib simplifies moving ML models to production by adding full support for model and Pipeline persistence. Individual models—and entire Pipelines including feature transformations—can be built on one Spark deployment, saved, and loaded onto other Spark deployments for production and serving.
(2) Users will find it much easier to implement custom feature transformers and models. Abstractions automatically handle input schema validation, as well as persistence for saving and loading models.
(3) For statisticians and data scientists, MLlib has doubled down on Generalized Linear Models (GLMs), which are key algorithms for many use cases. MLlib now supports more GLM families and link functions, handles corner cases more gracefully, and provides more model statistics. Also, expanded language APIs allow data scientists using Python and R to call many more algorithms.
Finally, we will demonstrate these improvements live and show how they facilitate getting started with ML on Spark, customizing implementations, and moving to production.
Overview of Apache Spark 2.3: What’s New? with Sameer Agarwal
Apache Spark 2.0 set the architectural foundations of Structure in Spark, Unified high-level APIs, Structured Streaming, and the underlying performant components like Catalyst Optimizer and Tungsten Engine. Since then the Spark community contributors have continued to build new features and fix numerous issues in releases Spark 2.1 and 2.2.
Continuing forward in that spirit, Apache Spark 2.3 has made similar strides too, introducing new features and resolving over 1300 JIRA issues. In this talk, we want to share with the community some salient aspects of Spark 2.3 features:
Kubernetes Scheduler Backend
PySpark Performance and Enhancements
Continuous Structured Streaming Processing
DataSource v2 APIs
Spark History Server Performance Enhancements
Introduction to Apache Beam (incubating) - DataCamp Salzburg - 7 dec 2016
This document provides an introduction to Apache Beam, a unified programming model for batch and stream data processing. It discusses Beam's programming model including PCollections, transforms, and runners. It also provides examples of writing a basic Beam pipeline in Java and running it on the Direct, Spark, and Flink runners.
Large-Scale Training with GPUs at Facebook
This document discusses large-scale distributed training with GPUs at Facebook using their Caffe2 framework. It describes how Facebook was able to train the ResNet-50 model on the ImageNet dataset in just 1 hour using 32 GPUs with 8 GPUs each. It explains how synchronous SGD was implemented in Caffe2 using Gloo for efficient all-reduce operations. Linear scaling of the learning rate with increased batch size was found to work best when gradually warming up the learning rate over the first few epochs. Nearly linear speedup was achieved using this approach on commodity hardware.
What is Spark
Quick introduction about Apache Spark and how it fits in the cognitive world, how can we use it to help cognitive solutions as well as create distributed algorithms to predict and perform other machine learning tasks.
PR workflow
This document outlines the Kubeflow pull request (PR) workflow, which involves four steps: 1) finding an issue to work on by reviewing issues on the Kubeflow GitHub repositories, 2) writing code and tests to address the issue, 3) submitting a PR with the code changes, and 4) having the PR reviewed and merged into the main repository once approved. It provides guidance on using tools like GitHub, writing tests, addressing code review comments, and resolving conflicts when merging approved PRs.
Python Streaming Pipelines on Flink - Beam Meetup at Lyft 2019
Apache Beam is a unified programming model for batch and streaming data processing that provides portability across distributed processing backends. It aims to support multiple languages like Java, Python and Go. The Beam Python SDK allows writing pipelines in Python that can run on distributed backends like Apache Flink. Lyft developed a Python SDK runner for Flink that translates Python pipelines to native Flink APIs using the Beam Fn API for communication between the SDK and runner. Future work includes improving performance of Python pipelines on JVM runners and supporting multiple languages in a single pipeline.
Apache MXNet AI
This presentation gives an overview of the Apache MXNet AI project. It explains Apache MXNet AI in terms of it's architecture, eco system, languages and the generic problems that the architecture attempts to solve.
Links for further information and connecting
http://www.amazon.com/Michael-Frampton/e/B00NIQDOOM/
https://nz.linkedin.com/pub/mike-frampton/20/630/385
https://open-source-systems.blogspot.com/
Python and GIS: Improving Your Workflow
A 40 minute talk on using Python with GIS software. Integration with ArcGIS and open source software is demonstrated. Includes links to several Python-based projects on Github. Presented at the Delaware Valley Regional Planning Commission's Information Resource Exchange Group on December 9th, 2015.
Creating Reusable Geospatial Pipelines
Geospatial pipelines in Apache Spark are difficult because of the diversity of datasets and the challenge of harmonizing on a single dataframe. We have worked over the past year to review different pipeline tools that allow us to quickly combine steps to create new workflows or operate on new datasets. We have reviewed Dagster, Apache Spark MLflow pipelines, Prefect, and our own custom solutions. The talk will go over the pros and cons of each of these solutions and will show an actionable workflow implementation that any geospatial analyst can leverage. We will show how we can leverage a pipeline to run a traditional geospatial hotspot analysis. Interactive mapping within the Databricks platform will be demonstrated.
Dynamic pricing of Lyft rides using streaming
Talk given Feb 07 2019 at SF Analytics meetup: https://www.meetup.com/SF-Big-Analytics/events/258174292/
Apache airflow
This document provides an overview of building data pipelines using Apache Airflow. It discusses what a data pipeline is, common components of data pipelines like data ingestion and processing, and issues with traditional data flows. It then introduces Apache Airflow, describing its features like being fault tolerant and supporting Python code. The core components of Airflow including the web server, scheduler, executor, and worker processes are explained. Key concepts like DAGs, operators, tasks, and workflows are defined. Finally, it demonstrates Airflow through an example DAG that extracts and cleanses tweets.
Vectorized Query Execution in Apache Spark at Facebook
This document summarizes Chen Yang's presentation on vectorized query execution in Apache Spark at Facebook. The key points are:
1) Spark is the largest SQL query engine at Facebook and uses columnar formats like ORC to improve storage efficiency.
2) Vectorized processing can improve performance over row-at-a-time processing by reducing per-row overhead and improving cache locality.
3) Facebook has implemented a vectorized ORC reader and writer in Spark that shows up to 8x speedup on microbenchmarks compared to the row-at-a-time approach.
Building Applications with Streams and Snapshots
Stream processing has been traditionally associated with realtime analytics. Modern stream processors, like Apache Flink, however, go far beyond that and give us a new approach to build applications and services as a whole.
This talk shows how to build applications on *data streams*, *state*, and *snaphots* (point-in-time views of application state) using Apache Flink. Rather than separating computation (application) and state (database), Flink manages the application logic and state as a tight pair and uses snapshots for consistent view onto the application and its state. With features like Flink's queryable state, the stream processor and database effectively become one.
This application pattern has many interesting properties: Aside from having fewer moving parts, it supports very high event rates because of its tight integration between computation and state, and its simple concurrency and recovery model. At the same time, it exposes a powerful consistency model, allows for seamless forking/updating/rollback of online applications, generalizes across historic and real-time data, and easily incorporates event time semantics and handling of late data. Finally, it allows applications to be defined in an easy way via streaming SQL.
PostgreSQL 9.4
This document summarizes a presentation about new features in PostgreSQL 9.4. It discusses enhancements for NoSQL support with JSON and GIN indexes, analytics with new aggregate functions and materialized views, increased flexibility with logical replication, easier administration using ALTER SYSTEM, and improved infrastructure for parallelization through dynamic background workers and shared memory. The presentation provides an overview of the status of 9.4 and highlights some of the major new categories of features.
Paris.rb – 07/19 – Sidekiq scaling, workers vs processes
Presentation given to Paris.RB meetup in July 2019.
- How to scale Sidekiq to handle millions of jobs?
- Is there a magic recipe to do API and computing jobs?
- Can we be cost-sensitive in scaling?
Presentation of a research protocol with observations and results.
Containerized architectures for deep learning
This document discusses containerized architectures for deep learning using Kubernetes and Kubeflow. It motivates the use of machine learning pipelines for improving data wrangling, system unification, composability, complexity management, portability, and model quality. It then summarizes popular machine learning pipeline tools like Apache Airflow, Kubeflow, TensorFlow Extended, and MLflow. It introduces Kubernetes and containers, and argues that Kubernetes and Kubeflow can provide composability, portability, and scalability for machine learning workloads. It demonstrates a deep learning model deployed on Kubernetes with Kubeflow through a doppelganger image similarity search app.
Streaming your Lyft Ride Prices - Flink Forward SF 2019
At Lyft we dynamically price our rides with a combination of various data sources, machine learning models, and streaming infrastructure for low latency, reliability and scalability. Dynamic pricing allows us to quickly adapt to real world changes and be fair to drivers (by say raising rates when there's a lot of demand) and fair to passengers (by let’s say offering to return 10 mins later for a cheaper rate). The streaming platform powers pricing by bringing together the best of two worlds using Apache Beam; ML algorithms in Python and Apache Flink as the streaming engine.
https://sf-2019.flink-forward.org/conference-program#streaming-your-lyft-ride-prices
Understanding and Improving Code Generation
Code generation is integral to Spark’s physical execution engine. When implemented, the Spark engine creates optimized bytecode at runtime improving performance when compared to interpreted execution. Spark has taken the next step with whole-stage codegen which collapses an entire query into a single function.
What's hot (20)
Apache Spark MLlib 2.0 Preview: Data Science and Production
Apache Spark MLlib 2.0 Preview: Data Science and Production
Overview of Apache Spark 2.3: What’s New? with Sameer Agarwal
Overview of Apache Spark 2.3: What’s New? with Sameer Agarwal
Introduction to Apache Beam (incubating) - DataCamp Salzburg - 7 dec 2016
Introduction to Apache Beam (incubating) - DataCamp Salzburg - 7 dec 2016
Python Streaming Pipelines on Flink - Beam Meetup at Lyft 2019
Python Streaming Pipelines on Flink - Beam Meetup at Lyft 2019
Vectorized Query Execution in Apache Spark at Facebook
Vectorized Query Execution in Apache Spark at Facebook
Paris.rb – 07/19 – Sidekiq scaling, workers vs processes
Paris.rb – 07/19 – Sidekiq scaling, workers vs processes
Streaming your Lyft Ride Prices - Flink Forward SF 2019
Streaming your Lyft Ride Prices - Flink Forward SF 2019
Similar to Hkube
Hybrid Cloud, Kubeflow and Tensorflow Extended [TFX]
Kubeflow Pipelines and TensorFlow Extended (TFX) together is end-to-end platform for deploying production ML pipelines. It provides a configuration framework and shared libraries to integrate common components needed to define, launch, and monitor your machine learning system. In this talk we describe how how to run TFX in hybrid cloud environments.
Data Summer Conf 2018, “Building unified Batch and Stream processing pipeline...
Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing pipelines, and also data ingestion and integration flows, supporting for both batch and streaming use cases. In presentation I will provide a general overview of Apache Beam and programming model comparison Apache Beam vs Apache Spark.
Functioning incessantly of Data Science Platform with Kubeflow - Albert Lewan...
Did you like it? Check out our blog to stay up to date: https://getindata.com/blog
The talk is focused on administration, development and monitoring platform with Apache Spark, Apache Flink and Kubeflow in which the monitoring stack is based on Prometheus stack.
Author: Albert Lewandowski
Linkedin: https://www.linkedin.com/in/albert-lewandowski/
___
Getindata is a company founded in 2014 by ex-Spotify data engineers. From day one our focus has been on Big Data projects. We bring together a group of best and most experienced experts in Poland, working with cloud and open-source Big Data technologies to help companies build scalable data architectures and implement advanced analytics over large data sets.
Our experts have vast production experience in implementing Big Data projects for Polish as well as foreign companies including i.a. Spotify, Play, Truecaller, Kcell, Acast, Allegro, ING, Agora, Synerise, StepStone, iZettle and many others from the pharmaceutical, media, finance and FMCG industries.
https://getindata.com
S8277 - Introducing Krylov: AI Platform that Empowers eBay Data Science and E...
The Krylov Project is the key component in eBay's AI Platform initiative that provides an easy to use, open, and fast AI orchestration engine that is deployed as managed services in eBay cloud.
Using Krylov, AI scientists can access eBay's massive datasets; build and train AI models; spin up powerful compute (high-memory or GPU instances) on the Krylov compute cluster; and set up machine learning pipelines, such as using declarative constructs that stitch together pipeline lifecycle.
Kubernetes: The Next Research Platform
Kubernetes has become the defacto standard as a platform for container orchestration. Its ease of extending and many integrations has paved the way for a wide variety of data science and research tooling to be built on top of it.
From all encompassing tools like Kubeflow that make it easy for researchers to build end-to-end Machine Learning pipelines to specific orchestration of analytics engines such as Spark; Kubernetes has made the deployment and management of these things easy. This presentation will showcase some of the larger research tools in the ecosystem and go into how Kubernetes has enabled this easy form of application management.
Kubernetes extensibility: CRDs & Operators
This talk discusses the core concepts behind the Kubernetes extensibility model. We are going to see how to implement new CRDs, operators and when to use them to automate the most critical aspects of your Kubernetes clusters.
Kubernetes extensibility: crd & operators
We believe that the popularity of Kubernetes derives from its ability to adapt and improve the infrastructure in which is deployed. I'll explain how this is done
OGDC2012 Lua In Game_Mr. Van, Nguyen Ngoc
Lua is a lightweight scripting language commonly used for games. It runs on a virtual machine and interfaces well with C/C++ code. Lua provides benefits like being free, cross-platform, lightweight, easy to embed, and allowing real-time changes to game logic and content. Core game engines typically integrate Lua by exposing APIs to core logic from Lua scripts and compiling Lua code to bytecode for execution on the Lua virtual machine. While Lua has advantages, it also has disadvantages like low-level C APIs, confusing error messages, lack of exceptions, and difficulty debugging and reloading scripts at runtime.
Productionizing Machine Learning - Bigdata meetup 5-06-2019
The document describes a Bucharest Big Data Meetup occurring on June 5th. The meetup will include two tech talks: one on productionizing machine learning from 7:00-7:40 PM, and another on a technology comparison of databases vs blockchains from 7:40-8:15 PM. The meetup will conclude from 8:15-8:45 PM with pizza and drinks sponsored by Netopia.
Building an MLOps Stack for Companies at Reasonable Scale
A practical talk on showing the following:
1. Challenges of Deploying ML today
2. How to do MLOps:
- Principles over Technology
- Convention over Configuration
3. What's a reasonable MLOps Stack
4. Demo on Google Collab to Deployed Endpoint
Benefits/tutorial of Lua in games
Lua is a lightweight scripting language commonly used for games due to its flexibility, speed, and ease of embedding into C/C++ applications. It allows game logic and behaviors to be defined through scripts that can be changed in real-time without recompiling the core engine. Lua integrates with game engines through a C API that exposes core functions and allows scripts to interface with the game world. While Lua has advantages like easy distribution of content and secure execution, it also has disadvantages like low-level APIs that can make debugging difficult.
SF-TAP: Scalable and Flexible Traffic Analysis Platform (USENIX LISA 2015)
SF-TAP: Scalable and Flexible Traffic Analysis Platform (USENIX LISA 2015)
https://www.usenix.org/conference/lisa15/conference-program/presentation/takano
https://github.com/SF-TAP/
http://sf-tap.github.io/
http://ytakano.github.io/
ACM Sunnyvale Meetup.pdf
Modern machine learning (ML) workloads, such as deep learning and large-scale model training, are compute-intensive and require distributed execution. Ray is an open-source, distributed framework from U.C. Berkeley’s RISELab that easily scales Python applications and ML workloads from a laptop to a cluster, with an emphasis on the unique performance challenges of ML/AI systems. It is now used in many production deployments.
This talk will cover Ray’s overview, architecture, core concepts, and primitives, such as remote Tasks and Actors; briefly discuss Ray native libraries (Ray Tune, Ray Train, Ray Serve, Ray Datasets, RLlib); and Ray’s growing ecosystem.
Through a demo using XGBoost for classification, we will demonstrate how you can scale training, hyperparameter tuning, and inference—from a single node to a cluster, with tangible performance difference when using Ray.
The takeaways from this talk are :
Learn Ray architecture, core concepts, and Ray primitives and patterns
Why Distributed computing will be the norm not an exception
How to scale your ML workloads with Ray libraries:
Training on a single node vs. Ray cluster, using XGBoost with/without Ray
Hyperparameter search and tuning, using XGBoost with Ray Tune
Inferencing at scale, using XGBoost with/without Ray
Flink history, roadmap and vision
A brief recap at the developments in Apache Flink over the last year (they were massive!) and a look into the streaming future of Apache Flink
Ingesting hdfs intosolrusingsparktrimmed
Apache Solr on Hadoop is enabling organizations to collect, process and search larger, more varied data. Apache Spark is is making a large impact across the industry, changing the way we think about batch processing and replacing MapReduce in many cases. But how can production users easily migrate ingestion of HDFS data into Solr from MapReduce to Spark? How can they update and delete existing documents in Solr at scale? And how can they easily build flexible data ingestion pipelines? Cloudera Search Software Engineer Wolfgang Hoschek will present an architecture and solution to this problem. How was Apache Solr, Spark, Crunch, and Morphlines integrated to allow for scalable and flexible ingestion of HDFS data into Solr? What are the solved problems and what's still to come? Join us for an exciting discussion on this new technology.
Flink Forward Berlin 2018: Robert Bradshaw & Maximilian Michels - "Universal ...
This document introduces Apache Beam, a unified model for batch and stream processing, and discusses its portability across languages and backends. It also introduces TFX, a TensorFlow tool for building end-to-end machine learning pipelines that addresses data collection, preprocessing, analysis, serving, and monitoring using components like TensorFlow Transform and TensorFlow Model Analysis. A demo of TFX's model analysis capabilities on a Chicago taxi dataset is provided.
Best Practices and Performance Studies for High-Performance Computing Clusters
The document discusses best practices and a performance study of HPC clusters. It covers system configuration and tuning, building applications, Intel Xeon processors, efficient execution methods, tools for boosting performance, and application performance highlights using HPL and HPCG benchmarks. The document contains agenda items, market share data, typical BIOS settings, compiler flags, MPI usage, and performance results from single node and cluster runs of the benchmarks.
Scaling TensorFlow with Hops, Global AI Conference Santa Clara
The document discusses scaling TensorFlow to hundreds of GPUs using Spark and Hops Hadoop. It describes how Hops enables running TensorFlow jobs on Spark clusters and supports distributed deep learning using algorithms like Ring AllReduce. Hops provides a single platform for machine learning and big data workloads by running TensorFlow, Spark and other frameworks on a HopsFS storage system and YARN cluster manager.
Apache Big Data EU 2016: Building Streaming Applications with Apache Apex
Stream processing applications built on Apache Apex run on Hadoop clusters and typically power analytics use cases where availability, flexible scaling, high throughput, low latency and correctness are essential. These applications consume data from a variety of sources, including streaming sources like Apache Kafka, Kinesis or JMS, file based sources or databases. Processing results often need to be stored in external systems (sinks) for downstream consumers (pub-sub messaging, real-time visualization, Hive and other SQL databases etc.). Apex has the Malhar library with a wide range of connectors and other operators that are readily available to build applications. We will cover key characteristics like partitioning and processing guarantees, generic building blocks for new operators (write-ahead-log, incremental state saving, windowing etc.) and APIs for application specification.
OS for AI: Elastic Microservices & the Next Gen of ML
AI has been a hot topic lately, with advances being made constantly in what is possible, there has not been as much discussion of the infrastructure and scaling challenges that come with it. How do you support dozens of different languages and frameworks, and make them interoperate invisibly? How do you scale to run abstract code from thousands of different developers, simultaneously and elastically, while maintaining less than 15ms of overhead?
At Algorithmia, we’ve built, deployed, and scaled thousands of algorithms and machine learning models, using every kind of framework (from scikit-learn to tensorflow). We’ve seen many of the challenges faced in this area, and in this talk I’ll share some insights into the problems you’re likely to face, and how to approach solving them.
In brief, we’ll examine the need for, and implementations of, a complete “Operating System for AI” – a common interface for different algorithms to be used and combined, and a general architecture for serverless machine learning which is discoverable, versioned, scalable and sharable.
Similar to Hkube (20)
Hybrid Cloud, Kubeflow and Tensorflow Extended [TFX]
Hybrid Cloud, Kubeflow and Tensorflow Extended [TFX]
Data Summer Conf 2018, “Building unified Batch and Stream processing pipeline...
Data Summer Conf 2018, “Building unified Batch and Stream processing pipeline...
Functioning incessantly of Data Science Platform with Kubeflow - Albert Lewan...
Functioning incessantly of Data Science Platform with Kubeflow - Albert Lewan...
S8277 - Introducing Krylov: AI Platform that Empowers eBay Data Science and E...
S8277 - Introducing Krylov: AI Platform that Empowers eBay Data Science and E...
Productionizing Machine Learning - Bigdata meetup 5-06-2019
Productionizing Machine Learning - Bigdata meetup 5-06-2019
Building an MLOps Stack for Companies at Reasonable Scale
Building an MLOps Stack for Companies at Reasonable Scale
SF-TAP: Scalable and Flexible Traffic Analysis Platform (USENIX LISA 2015)
SF-TAP: Scalable and Flexible Traffic Analysis Platform (USENIX LISA 2015)
Flink Forward Berlin 2018: Robert Bradshaw & Maximilian Michels - "Universal ...
Flink Forward Berlin 2018: Robert Bradshaw & Maximilian Michels - "Universal ...
Best Practices and Performance Studies for High-Performance Computing Clusters
Best Practices and Performance Studies for High-Performance Computing Clusters
Scaling TensorFlow with Hops, Global AI Conference Santa Clara
Scaling TensorFlow with Hops, Global AI Conference Santa Clara
Apache Big Data EU 2016: Building Streaming Applications with Apache Apex
Apache Big Data EU 2016: Building Streaming Applications with Apache Apex
OS for AI: Elastic Microservices & the Next Gen of ML
OS for AI: Elastic Microservices & the Next Gen of ML
Recently uploaded
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Inconsistent user experience and siloed data, high costs, and changing customer expectations – Citizens Bank was experiencing these challenges while it was attempting to deliver a superior digital banking experience for its clients. Its core banking applications run on the mainframe and Citizens was using legacy utilities to get the critical mainframe data to feed customer-facing channels, like call centers, web, and mobile. Ultimately, this led to higher operating costs (MIPS), delayed response times, and longer time to market.
Ever-changing customer expectations demand more modern digital experiences, and the bank needed to find a solution that could provide real-time data to its customer channels with low latency and operating costs. Join this session to learn how Citizens is leveraging Precisely to replicate mainframe data to its customer channels and deliver on their “modern digital bank” experiences.
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...Edge AI and Vision Alliance
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/how-axelera-ai-uses-digital-compute-in-memory-to-deliver-fast-and-energy-efficient-computer-vision-a-presentation-from-axelera-ai/
Bram Verhoef, Head of Machine Learning at Axelera AI, presents the “How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-efficient Computer Vision” tutorial at the May 2024 Embedded Vision Summit.
As artificial intelligence inference transitions from cloud environments to edge locations, computer vision applications achieve heightened responsiveness, reliability and privacy. This migration, however, introduces the challenge of operating within the stringent confines of resource constraints typical at the edge, including small form factors, low energy budgets and diminished memory and computational capacities. Axelera AI addresses these challenges through an innovative approach of performing digital computations within memory itself. This technique facilitates the realization of high-performance, energy-efficient and cost-effective computer vision capabilities at the thin and thick edge, extending the frontier of what is achievable with current technologies.
In this presentation, Verhoef unveils his company’s pioneering chip technology and demonstrates its capacity to deliver exceptional frames-per-second performance across a range of standard computer vision networks typical of applications in security, surveillance and the industrial sector. This shows that advanced computer vision can be accessible and efficient, even at the very edge of our technological ecosystem.Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays, June 2024 with Maria Copot 20
Essentials of Automations: Exploring Attributes & Automation Parameters
Building automations in FME Flow can save time, money, and help businesses scale by eliminating data silos and providing data to stakeholders in real-time. One essential component to orchestrating complex automations is the use of attributes & automation parameters (both formerly known as “keys”). In fact, it’s unlikely you’ll ever build an Automation without using these components, but what exactly are they?
Attributes & automation parameters enable the automation author to pass data values from one automation component to the next. During this webinar, our FME Flow Specialists will cover leveraging the three types of these output attributes & parameters in FME Flow: Event, Custom, and Automation. As a bonus, they’ll also be making use of the Split-Merge Block functionality.
You’ll leave this webinar with a better understanding of how to maximize the potential of automations by making use of attributes & automation parameters, with the ultimate goal of setting your enterprise integration workflows up on autopilot.
AppSec PNW: Android and iOS Application Security with MobSF
Mobile Security Framework - MobSF is a free and open source automated mobile application security testing environment designed to help security engineers, researchers, developers, and penetration testers to identify security vulnerabilities, malicious behaviours and privacy concerns in mobile applications using static and dynamic analysis. It supports all the popular mobile application binaries and source code formats built for Android and iOS devices. In addition to automated security assessment, it also offers an interactive testing environment to build and execute scenario based test/fuzz cases against the application.
This talk covers:
Using MobSF for static analysis of mobile applications.
Interactive dynamic security assessment of Android and iOS applications.
Solving Mobile app CTF challenges.
Reverse engineering and runtime analysis of Mobile malware.
How to shift left and integrate MobSF/mobsfscan SAST and DAST in your build pipeline.
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Are you ready to revolutionize how you handle data? Join us for a webinar where we’ll bring you up to speed with the latest advancements in Generative AI technology and discover how leveraging FME with tools from giants like Google Gemini, Amazon, and Microsoft OpenAI can supercharge your workflow efficiency.
During the hour, we’ll take you through:
Guest Speaker Segment with Hannah Barrington: Dive into the world of dynamic real estate marketing with Hannah, the Marketing Manager at Workspace Group. Hear firsthand how their team generates engaging descriptions for thousands of office units by integrating diverse data sources—from PDF floorplans to web pages—using FME transformers, like OpenAIVisionConnector and AnthropicVisionConnector. This use case will show you how GenAI can streamline content creation for marketing across the board.
Ollama Use Case: Learn how Scenario Specialist Dmitri Bagh has utilized Ollama within FME to input data, create custom models, and enhance security protocols. This segment will include demos to illustrate the full capabilities of FME in AI-driven processes.
Custom AI Models: Discover how to leverage FME to build personalized AI models using your data. Whether it’s populating a model with local data for added security or integrating public AI tools, find out how FME facilitates a versatile and secure approach to AI.
We’ll wrap up with a live Q&A session where you can engage with our experts on your specific use cases, and learn more about optimizing your data workflows with AI.
This webinar is ideal for professionals seeking to harness the power of AI within their data management systems while ensuring high levels of customization and security. Whether you're a novice or an expert, gain actionable insights and strategies to elevate your data processes. Join us to see how FME and AI can revolutionize how you work with data!
What is an RPA CoE? Session 1 – CoE Vision
In the first session, we will review the organization's vision and how this has an impact on the COE Structure.
Topics covered:
• The role of a steering committee
• How do the organization’s priorities determine CoE Structure?
Speaker:
Chris Bolin, Senior Intelligent Automation Architect Anika Systems
Astute Business Solutions | Oracle Cloud Partner |
Your goto partner for Oracle Cloud, PeopleSoft, E-Business Suite, and Ellucian Banner. We are a firm specialized in managed services and consulting.
Generating privacy-protected synthetic data using Secludy and Milvus
During this demo, the founders of Secludy will demonstrate how their system utilizes Milvus to store and manipulate embeddings for generating privacy-protected synthetic data. Their approach not only maintains the confidentiality of the original data but also enhances the utility and scalability of LLMs under privacy constraints. Attendees, including machine learning engineers, data scientists, and data managers, will witness first-hand how Secludy's integration with Milvus empowers organizations to harness the power of LLMs securely and efficiently.
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Dmitrii Kamaev, PhD
Senior Product Owner - QIAGEN
JavaLand 2024: Application Development Green Masterplan
My presentation slides I used at JavaLand 2024
Fueling AI with Great Data with Airbyte Webinar
This talk will focus on how to collect data from a variety of sources, leveraging this data for RAG and other GenAI use cases, and finally charting your course to productionalization.
Main news related to the CCS TSI 2023 (2023/1695)
An English 🇬🇧 translation of a presentation to the speech I gave about the main changes brought by CCS TSI 2023 at the biggest Czech conference on Communications and signalling systems on Railways, which was held in Clarion Hotel Olomouc from 7th to 9th November 2023 (konferenceszt.cz). Attended by around 500 participants and 200 on-line followers.
The original Czech 🇨🇿 version of the presentation can be found here: https://www.slideshare.net/slideshow/hlavni-novinky-souvisejici-s-ccs-tsi-2023-2023-1695/269688092 .
The videorecording (in Czech) from the presentation is available here: https://youtu.be/WzjJWm4IyPk?si=SImb06tuXGb30BEH .
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
Folding is a recent technique for building efficient recursive SNARKs. Several elegant folding protocols have been proposed, such as Nova, Supernova, Hypernova, Protostar, and others. However, all of them rely on an additively homomorphic commitment scheme based on discrete log, and are therefore not post-quantum secure. In this work we present LatticeFold, the first lattice-based folding protocol based on the Module SIS problem. This folding protocol naturally leads to an efficient recursive lattice-based SNARK and an efficient PCD scheme. LatticeFold supports folding low-degree relations, such as R1CS, as well as high-degree relations, such as CCS. The key challenge is to construct a secure folding protocol that works with the Ajtai commitment scheme. The difficulty, is ensuring that extracted witnesses are low norm through many rounds of folding. We present a novel technique using the sumcheck protocol to ensure that extracted witnesses are always low norm no matter how many rounds of folding are used. Our evaluation of the final proof system suggests that it is as performant as Hypernova, while providing post-quantum security.
Paper Link: https://eprint.iacr.org/2024/257
Dandelion Hashtable: beyond billion requests per second on a commodity server
This slide deck presents DLHT, a concurrent in-memory hashtable. Despite efforts to optimize hashtables, that go as far as sacrificing core functionality, state-of-the-art designs still incur multiple memory accesses per request and block request processing in three cases. First, most hashtables block while waiting for data to be retrieved from memory. Second, open-addressing designs, which represent the current state-of-the-art, either cannot free index slots on deletes or must block all requests to do so. Third, index resizes block every request until all objects are copied to the new index. Defying folklore wisdom, DLHT forgoes open-addressing and adopts a fully-featured and memory-aware closed-addressing design based on bounded cache-line-chaining. This design offers lock-free index operations and deletes that free slots instantly, (2) completes most requests with a single memory access, (3) utilizes software prefetching to hide memory latencies, and (4) employs a novel non-blocking and parallel resizing. In a commodity server and a memory-resident workload, DLHT surpasses 1.6B requests per second and provides 3.5x (12x) the throughput of the state-of-the-art closed-addressing (open-addressing) resizable hashtable on Gets (Deletes).
Introduction of Cybersecurity with OSS at Code Europe 2024
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
Northern Engraving | Nameplate Manufacturing Process - 2024
Manufacturing custom quality metal nameplates and badges involves several standard operations. Processes include sheet prep, lithography, screening, coating, punch press and inspection. All decoration is completed in the flat sheet with adhesive and tooling operations following. The possibilities for creating unique durable nameplates are endless. How will you create your brand identity? We can help!
Artificial Intelligence and Electronic Warfare
Artificial Intelligence and Electronic WarfarePapadakis K.-Cyber-Information Warfare Analyst & Cyber Defense/Security Consultant-Hellenic MoD
AI & Electronic WarfareRecently uploaded (20)
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024
Hkube
- 2. Agenda About us Overview Architecture & Demonstration Roadmap Summary
- 3. 1/5 About us
- 4. The Core team MAty Zissermam (Team Leader & Software Developer ) Yehiyam livne (Software Developer) Nassi Harel (Software Developer) Tal Bahallol (Software Developer) Eli Maor (Devops) Eytan Godensky Ronen Kroitoro (System Management)
- 5. 2/5 Overview
- 6. Hkube is an open source framework that allows developers focus only on writing algorithms. We will do the rest… Hkube for algorithms is like Kubernetes to dockers
- 9. Main Concepts Optimize Hardware utilization Language AgnosticDistributed pipeline of algorithms Algorithm monitoring Simple APISemi Automatic Jobs Prioritization
- 10. Legend Algorithm A written in python Algorithm B written in c++ Algorithm C written in R Algorithm D written in Python Pipeline 1 Client 1 2 5 Pipeline 3 CPU 368 3 1150 After some time Pipeline 2 . . . What we try to solve /500 2 X 4 X 7 X 7 X 4 3 5 1 2
- 11. It’s SimpleDeploy algorithm Dockerize the algorithm Use CLI to add the algorithm to hkube Implement a simple websocket api for hkube Run the pipeline with input Register webhooks for result and status Create/store pipeline Run pipeline
- 12. User Instructions Me 2. Split your problem to small pieces input Classification Me Me Aggregation 3. Run it on Hkube Dockerize Create Pipeline Descriptor Deploy & Run Make your boss happy Took too long 1. ProblemInput Result Face Detection & Cropping find myself in a picture X8
- 13. API 1.Create Descriptor 2.Run with Inputs
- 15. Core Modules Api Server - Pipeline validation - Client Api for status result - Notifies via webhooks on data changes Pipeline Driver - Build DAG graph based on Pipeline descriptor schema - Manage the pipeline flow Resource Manager & Executor - Fetch hardware and application data - Calculate algorithm pod count - Create pods Worker - Runs algorithm container as a sidecar - Functions as a bridge for communicating between algorithm and hkube Queues - Implements heuristic for prioritizing algorithm tasks - Handle algorithm task execution
- 16. Algorithm Queue face detection Resource Executer Algorithm Queue classification Resource Manager Pipeline Driver Queue Api Server Algorithm Queue Reduce-images positions Worker Algorithm Classification Architecture X 100 Generate Job Yaml For each pod Get resource data Pipeline Driver Instance Legend Worker Algorithm reduce-images-positions After algorithm ends. wait for timeout and then finish k8s job Worker Algorithm face detection Client Run K8s Job
- 17. 4/5 Current Status & Roadmap
- 18. Version 1.0 (8/18) - Dynamic allocation - Extend algorithm monitoring metrics - Extend hkube cli for supporting custom storage - Well documented landing page - Long lasting worker - Triggering Roadmap
- 19. Roadmap Future - Additional adapter for common orchestrators (spark&kubeflow) - Worker affinity (GPU, specific node) - Code API - Cashing for algorithms (manually/input with hash ) - Cluster scaling - Streaming Pipelines - Sub-pipeline: for conditional execution - Resource allocation heuristic improvements (ML) - Hkube Dashboard - More built in storage support - Public algorithm store (helm like) - PAAS - Developer IDE - allow for easy developing and debugging of algorithms - Authentication & Authorization - Simplify API to HKUBE