Real-Time Streaming with Python ML Inference
•Download as PPTX, PDF•
0 likes•34 views
The capabilities of machine learning are now pretty well understood and there are great tools to do data science and construct models that answer nontrivial questions about your data. These tools are often used from Python. The key new challenge is making the trained prediction model usable in real time, while the user is interacting with your software. Getting answers from an ML model takes a lot of CPU/GPU and must be done at serious scale. The ML tools are optimized mainly for batch-processing a lot of data at once, and often the implementations aren't parallelized. In this talk I will show one approach which allows you to write a low-latency, auto-parallelized and distributed stream processing pipeline in Java that seamlessly integrates with a data scientist's work taken in almost unchanged form from their Python development environment. The talk includes a live demo using the command line and going through some Python and Java code snippets.
Report
Share
Report
Share
Recommended
IOOF IT System Modernisation
- IOOF uses MongoDB for state management in over 85% of its 250+ microservices due to its document storage, developer friendliness, performance at scale, and ability to break from monolithic database architectures.
- IOOF's data warehouse also utilizes MongoDB to store time series data from various microservices via change data capture (CDC) for improved analytics over relational databases.
- IOOF has three MongoDB replica sets for production, staging, and development environments, handling high read/write volumes through sharding and partitioning as needed for scale.
Making Sense of Schema on Read
[Given at DAMA WI, Nov 2018] With the increasing prevalence of semi-structured data from IoT devices, web logs, and other sources, data architects and modelers have to learn how to interpret and project data from things like JSON. While the concept of loading data without upfront modeling is appealing to many, ultimately, in order to make sense of the data and use it to drive business value, we have to turn that schema-on-read data into a real schema! That means data modeling! In this session I will walk through both simple and complex JSON documents, decompose them, then turn them into a representative data model using Oracle SQL Developer Data Modeler. I will show you how they might look using both traditional 3NF and data vault styles of modeling. In this session you will:
1. See what a JSON document looks like
2. Understand how to read it
3. Learn how to convert it to a standard data model
Test trend analysis: Towards robust reliable and timely tests
This document discusses test trend analysis and making tests more robust, reliable, and timely. It proposes collecting test results data and storing it in Elasticsearch. Visualizations would then be created using Kibana to analyze test failures, slow tests, error messages, and step times. This would provide insights and help identify issues to make tests less flaky.
Automatically Generate API Contract Tests
TALK | Automatically convert your OpenAPI or Swagger doc into a Postman contract test using an automated script
IIPGH Webinar 1: Getting Started With Data Science
In this webinar for ICT Professionals Ghana, we explore the concepts of data science and its motivations as a recent specialization. creating the background for how Artificial Intelligence relates to Machine Learning and to Deep Learning. We further discuss the data science technology stack and the opportunities that exist in the space.
Machine Learning for Recommender Systems in the Job Market
The document discusses machine learning techniques for recommender systems in job markets. It describes the goals of recommending jobs that users are interested in and qualified for. It outlines challenges like understanding user profiles and job descriptions accurately. It then explains the recommender system architecture, which uses collaborative, content-based, and other filters to make recommendations by analyzing over 200 features of users and jobs. Evaluation results show that outlier filtering improves the quality of recommendations by reducing bad recommendations while keeping good ones.
Data science
The document provides an overview of data science through an introduction by Sreejith C, a data scientist. It defines data science as discovering unknown information from data, obtaining predictive insights, creating impactful data products, and communicating business stories from data. A data scientist's work includes tasks like authoring data processing pipelines, performing analyses, and communicating results. The document also demonstrates a loan prediction problem using machine learning algorithms like logistic regression, decision trees, and random forests in Python.
Real-time big data analytics based on product recommendations case study
We started as an ad network. The challenge was to recommend the best product (out of millions) to the right person in a given moment (thousands of users within a second). We have delivered 5 billion ad views since 24 months. To put it in the scale context: If we would serve 1 ad per second it will take 160 years to serve 5 billion ads.
So we needed a solution. SQL databases did not work. Popular NoSQL databases did not work. Standard data warehouse approaches (pre-aggregations, creating schemas) - did not work too.
Re-thinking all the problems with huge data streams flowing to us every second we have built a complete solution based on open-source technologies and fresh, smart ideas from our engineering team. It is called deep.bi and now we make it available to other companies.
deep.bi lets high-growth companies solve fast data problems by providing scalable, flexible and real-time data collection, enrichment and analytics.
It was built using:
- Node.js - API
- Kafka - collecting and distributing data
- Spark Streaming - ETL, data enrichments
- Druid - real-time analytics
- Cassandra - user events store
- Hadoop + Parquet + Spark - raw data store + ad-hoc queries
Recommended
IOOF IT System Modernisation
- IOOF uses MongoDB for state management in over 85% of its 250+ microservices due to its document storage, developer friendliness, performance at scale, and ability to break from monolithic database architectures.
- IOOF's data warehouse also utilizes MongoDB to store time series data from various microservices via change data capture (CDC) for improved analytics over relational databases.
- IOOF has three MongoDB replica sets for production, staging, and development environments, handling high read/write volumes through sharding and partitioning as needed for scale.
Making Sense of Schema on Read
[Given at DAMA WI, Nov 2018] With the increasing prevalence of semi-structured data from IoT devices, web logs, and other sources, data architects and modelers have to learn how to interpret and project data from things like JSON. While the concept of loading data without upfront modeling is appealing to many, ultimately, in order to make sense of the data and use it to drive business value, we have to turn that schema-on-read data into a real schema! That means data modeling! In this session I will walk through both simple and complex JSON documents, decompose them, then turn them into a representative data model using Oracle SQL Developer Data Modeler. I will show you how they might look using both traditional 3NF and data vault styles of modeling. In this session you will:
1. See what a JSON document looks like
2. Understand how to read it
3. Learn how to convert it to a standard data model
Test trend analysis: Towards robust reliable and timely tests
This document discusses test trend analysis and making tests more robust, reliable, and timely. It proposes collecting test results data and storing it in Elasticsearch. Visualizations would then be created using Kibana to analyze test failures, slow tests, error messages, and step times. This would provide insights and help identify issues to make tests less flaky.
Automatically Generate API Contract Tests
TALK | Automatically convert your OpenAPI or Swagger doc into a Postman contract test using an automated script
IIPGH Webinar 1: Getting Started With Data Science
In this webinar for ICT Professionals Ghana, we explore the concepts of data science and its motivations as a recent specialization. creating the background for how Artificial Intelligence relates to Machine Learning and to Deep Learning. We further discuss the data science technology stack and the opportunities that exist in the space.
Machine Learning for Recommender Systems in the Job Market
The document discusses machine learning techniques for recommender systems in job markets. It describes the goals of recommending jobs that users are interested in and qualified for. It outlines challenges like understanding user profiles and job descriptions accurately. It then explains the recommender system architecture, which uses collaborative, content-based, and other filters to make recommendations by analyzing over 200 features of users and jobs. Evaluation results show that outlier filtering improves the quality of recommendations by reducing bad recommendations while keeping good ones.
Data science
The document provides an overview of data science through an introduction by Sreejith C, a data scientist. It defines data science as discovering unknown information from data, obtaining predictive insights, creating impactful data products, and communicating business stories from data. A data scientist's work includes tasks like authoring data processing pipelines, performing analyses, and communicating results. The document also demonstrates a loan prediction problem using machine learning algorithms like logistic regression, decision trees, and random forests in Python.
Real-time big data analytics based on product recommendations case study
We started as an ad network. The challenge was to recommend the best product (out of millions) to the right person in a given moment (thousands of users within a second). We have delivered 5 billion ad views since 24 months. To put it in the scale context: If we would serve 1 ad per second it will take 160 years to serve 5 billion ads.
So we needed a solution. SQL databases did not work. Popular NoSQL databases did not work. Standard data warehouse approaches (pre-aggregations, creating schemas) - did not work too.
Re-thinking all the problems with huge data streams flowing to us every second we have built a complete solution based on open-source technologies and fresh, smart ideas from our engineering team. It is called deep.bi and now we make it available to other companies.
deep.bi lets high-growth companies solve fast data problems by providing scalable, flexible and real-time data collection, enrichment and analytics.
It was built using:
- Node.js - API
- Kafka - collecting and distributing data
- Spark Streaming - ETL, data enrichments
- Druid - real-time analytics
- Cassandra - user events store
- Hadoop + Parquet + Spark - raw data store + ad-hoc queries
Data infrastructure architecture for medium size organization: tips for colle...
Data infrastructure architecture for medium size organization: tips for colle...DataWorks Summit/Hadoop Summit
Data infrastructure architecture for medium size organization: tips for collecting, storing and analysis.
Webinar elastic stack {on telecom} english webinar part (1)
Telecoms network consists of hundreds of network elements that generate call detail records or event data for each phone call, text message, internet activity, or even a simple walk from one location to another. The number of records per day varies between millions and billions. In many cases, the network elements are from different vendors. The typical interface to access these records is file-based.
Communication services providers want to analyze these records to run campaigns, improve user experience, improve network or call center, or to detect fraud. They need a system that unifies source data, runs near real-time analytics, and scales with increasing network traffic.
Join our upcoming webinar to discover how we power telecom analytics using the Elastic Stack, an open source, scalable & distributed, reliably and securely take data from any source, in any format, then search, analyze, and visualize it in real time.
Elastic Stack is a group of open source products from Elastic designed to help users take data from any type of source and in any format and search, analyze, and visualize that data in real time. The product group was formerly known as ELK Stack, in which the letters in the name stood for the products in the group: Elasticsearch, Logstash and Kibana. A fourth product, Beats, was subsequently added to the stack, rendering the potential acronym unpronounceable. Elastic Stack can be deployed on premises or made available as Software as a Service (SaaS).
Transforming your application with Elasticsearch
Brian Ritchie will give a presentation on transforming applications with Elastic Search. Elastic Search is an open source, distributed search and analytics engine that can be used to add powerful search capabilities to applications. It allows storing and searching large volumes of data quickly and flexibly scales. The presentation will cover introducing Elastic Search, bringing application data into it, security considerations, and an example of putting it all together to build a searchable application using Elastic Search.
Test Trend Analysis : Towards robust, reliable and timely tests
Slides from my talk at Selenium Conference 2016.
In this talk you will get ideas about how you can instrument test result information to provide actionable data, paving the way for more robust, reliable and timely test results.
By capturing this information over time, and when combined with visualization tools, we can answer different questions than with existing solutions (Allure / CI tool build history). Some examples of these are:
Which tests are consistently flaky
What are the common causes of failure across tests
Which tests consistently take a long time to run
Using this information we can move away from the ‘re-run’ culture and better support continuous integration goals of having quick, reliable, deterministic tests
Video of the talk is here: https://youtu.be/29fPYx7OJnE?list=PL_7kBU2XBlbKuRNVHeqjXUygXtToqMHsn
Eat whatever you can with PyBabe
This document discusses PyBabe, an open-source Python library for ETL (extract, transform, load) processes. PyBabe allows extracting data from various sources like FTP, SQL databases, and Amazon S3. It can perform transformations on the data like filtering, regular expressions, and date parsing. The transformed data can then be loaded to targets like SQL databases, MongoDB, Excel files, and more. PyBabe represents data as a stream of named tuples and processes the data lazily using generators for efficiency. Examples show how to use PyBabe to sort and join large files, send reports over email, and abstract ETL logic into reusable scripts.
Contract-driven development with OpenAPI 3 and Vert.x | DevNation Tech Talk
Have you ever been frustrated by developing and documenting an HTTP API? When it comes down to defining the HTTP interface between frontend and backend, have you ever had problems specifying the parameters or the shape of the body without misunderstandings? In this talk we’ll introduce you to "Contract Driven Development" (or API Design First approach), a methodology that uses declarative API Contracts to enable developers to efficiently design, communicate, and evolve their HTTP APIs, while automating API implementation phases where possible. In order to implement this methodology, we’ll show you how to develop an API contract using OpenAPI 3 and how you can easily implement the HTTP endpoints using Vert.x Web OpenAPI.
Mongo at Sailthru (MongoNYC 2011)
Sailthru uses MongoDB to store user profile data, including interests and behaviors, in documents rather than across multiple SQL tables. This allows them to access user data from any part of their system and personalize content. They run MongoDB on Amazon EC2 across 11 servers in 5 replica sets and 1 backup server totaling around 1TB of data. Key advantages of MongoDB for them include flexibility, performance, and not needing downtime for schema changes.
Agile Data Science 2.0
Agile Data Science 2.0 (O'Reilly 2017) defines a methodology and a software stack with which to apply the methods. *The methodology* seeks to deliver data products in short sprints by going meta and putting the focus on the applied research process itself. *The stack* is but an example of one meeting the requirements that it be utterly scalable and utterly efficient in use by application developers as well as data engineers. It includes everything needed to build a full-blown predictive system: Apache Spark, Apache Kafka, Apache Incubating Airflow, MongoDB, ElasticSearch, Apache Parquet, Python/Flask, JQuery. This talk will cover the full lifecycle of large data application development and will show how to use lessons from agile software engineering to apply data science using this full-stack to build better analytics applications. The entire lifecycle of big data application development is discussed. The system starts with plumbing, moving on to data tables, charts and search, through interactive reports, and building towards predictions in both batch and realtime (and defining the role for both), the deployment of predictive systems and how to iteratively improve predictions that prove valuable.
Log everything! @DC13
Deck36 is a small team of engineers who specialize in designing, implementing, and operating complex web systems. They discuss their approach to logging everything through a data pipeline that ingests data from producers, transports it via RabbitMQ, stores it in Hadoop HDFS and Amazon S3, runs analytics with Hadoop MapReduce and Amazon EMR, and performs real-time stream processing with Twitter Storm. They also live demo their JavaScript data collector client and a PHP/Storm example that processes click stream data.
Agile Data Science
Agile Data Science 2.0 (O'Reilly 2017) defines a methodology and a software stack with which to apply the methods. *The methodology* seeks to deliver data products in short sprints by going meta and putting the focus on the applied research process itself. *The stack* is but an example of one meeting the requirements that it be utterly scalable and utterly efficient in use by application developers as well as data engineers. It includes everything needed to build a full-blown predictive system: Apache Spark, Apache Kafka, Apache Incubating Airflow, MongoDB, ElasticSearch, Apache Parquet, Python/Flask, JQuery. This talk will cover the full lifecycle of large data application development and will show how to use lessons from agile software engineering to apply data science using this full-stack to build better analytics applications. The entire lifecycle of big data application development is discussed. The system starts with plumbing, moving on to data tables, charts and search, through interactive reports, and building towards predictions in both batch and realtime (and defining the role for both), the deployment of predictive systems and how to iteratively improve predictions that prove valuable.
Open Source World June '21 -- JSON Within a Relational Database
How does free form JSON work with a strictly typed relational database? This session will cover how MySQL and JSON give you the best of both worlds
Learning analytics inside government
Jisc is a UK organization that aims to advance digital technology in education and research. Their Learning Analytics project has three core parts: a learning analytics service, toolkit, and community. The service provides dashboards and tools to analyze student data from various sources to identify at-risk students and enable interventions. It follows an open architecture approach. The toolkit includes guidance on best practices like privacy and consent. The community aspect involves events, blogs, and mailing lists to bring people together around learning analytics.
Montreal Elasticsearch Meetup
This document provides an overview of Elasticsearch, including what it is, how it works, and how to perform basic operations like indexing, updating, and searching documents. It explains that Elasticsearch allows for advanced search across large amounts of data by making documents searchable and scaling easily. It also demonstrates how to index, update, search for, and retrieve documents through RESTful API calls. Faceted search, aggregations, and cluster architecture are also summarized.
Docker Summit MongoDB - Data Democratization
The document summarizes a presentation given by Chris Grabosky, a Solutions Architect at MongoDB. It discusses how containerization and data democratization can help organizations build cloud native applications. It highlights how MongoDB and Docker can be used together to build scalable and portable apps. It also covers data modeling approaches, workload isolation, analytics, and data access controls that MongoDB provides to help democratize data.
DevSecCon London 2018: Open DevSecOps
PETKO D. PETKOV
Thanks to the DevSecOps philosophy a growing number of organisations around the world are ensuring their businesses are set up with the security in mind from the get-go. DevSecOps is taking the world by storm. This talk is about how to introduce DevSecOps in your organisation with ready-made, zero-cost, open source templates accessible to everyone. The talk will introduce the OpenDevSecOps project and show many practical examples of how to easily deploy security testing infrastructure on top of existing and well-established development tools.
Xain.io exhibiting at Berlin Tech Job Fair Spring 2020
XAIN is a company that provides federated learning as a service to allow organizations to train AI models on decentralized data without aggregating or anonymizing private user data. Their federated learning platform allows models to be trained separately on each organization's local data and then combines the models for improved accuracy while keeping all data localized. They have successfully worked with companies like Porsche and are hiring various engineering roles.
Tapping into Scientific Data with Hadoop and Flink
At ResearchGate, we constantly analyze scientific data to connect the world of science and make research open to all. It can be tricky to set up a process to continuously deliver improved versions of algorithms that tap into more than 100 million publications and corresponding bibliographic metadata. In this talk, we illustrate some (big) data engineering challenges of running data pipelines and incorporating results into the live databases that power our user-facing features every day. We show how Apache Flink helps us to improve performance, robustness, ease of maintenance - and most importantly - have more fun while building big data pipelines.
Telling the LivePerson Technology Story at Couchbase [SF] 2013
As part of Couchbase[SF]2013, Ido Shilon, R&D Group Leader at LivePerson, discusses LivePerson's project to re-architect their LiveEngage platform backend, with a focus on LivePerson's decision to use NoSQL technologies, the challenges encountered, and the lessons learned.
Video: http://www.youtube.com/watch?v=rYKWFmJEHX0
Human-in-the-loop: a design pattern for managing teams which leverage ML by P...
Human-in-the-loop is an approach which has been used for simulation, training, UX mockups, etc.
https://www.bigdataspain.org/2017/talk/human-in-the-loop-a-design-pattern-for-managing-teams-which-leverage-ml
Big Data Spain Conference
16th -17th November - Kinépolis Madrid
ICONUK 2016: REST Assured, Freeing Your Domino Data Has Never Been That Easy!
ICONUK 2016 Developer Session on RESTful Services with IBM Domino Server and Apache Wink. September 16, 2016
Energy consumption of Database Management - Florina Jonuzi
Presentation from Florina Jonuzi at the GSD Community Stage Meetup on June 06, 2024
More Related Content
Similar to Real-Time Streaming with Python ML Inference
Data infrastructure architecture for medium size organization: tips for colle...
Data infrastructure architecture for medium size organization: tips for colle...DataWorks Summit/Hadoop Summit
Data infrastructure architecture for medium size organization: tips for collecting, storing and analysis.
Webinar elastic stack {on telecom} english webinar part (1)
Telecoms network consists of hundreds of network elements that generate call detail records or event data for each phone call, text message, internet activity, or even a simple walk from one location to another. The number of records per day varies between millions and billions. In many cases, the network elements are from different vendors. The typical interface to access these records is file-based.
Communication services providers want to analyze these records to run campaigns, improve user experience, improve network or call center, or to detect fraud. They need a system that unifies source data, runs near real-time analytics, and scales with increasing network traffic.
Join our upcoming webinar to discover how we power telecom analytics using the Elastic Stack, an open source, scalable & distributed, reliably and securely take data from any source, in any format, then search, analyze, and visualize it in real time.
Elastic Stack is a group of open source products from Elastic designed to help users take data from any type of source and in any format and search, analyze, and visualize that data in real time. The product group was formerly known as ELK Stack, in which the letters in the name stood for the products in the group: Elasticsearch, Logstash and Kibana. A fourth product, Beats, was subsequently added to the stack, rendering the potential acronym unpronounceable. Elastic Stack can be deployed on premises or made available as Software as a Service (SaaS).
Transforming your application with Elasticsearch
Brian Ritchie will give a presentation on transforming applications with Elastic Search. Elastic Search is an open source, distributed search and analytics engine that can be used to add powerful search capabilities to applications. It allows storing and searching large volumes of data quickly and flexibly scales. The presentation will cover introducing Elastic Search, bringing application data into it, security considerations, and an example of putting it all together to build a searchable application using Elastic Search.
Test Trend Analysis : Towards robust, reliable and timely tests
Slides from my talk at Selenium Conference 2016.
In this talk you will get ideas about how you can instrument test result information to provide actionable data, paving the way for more robust, reliable and timely test results.
By capturing this information over time, and when combined with visualization tools, we can answer different questions than with existing solutions (Allure / CI tool build history). Some examples of these are:
Which tests are consistently flaky
What are the common causes of failure across tests
Which tests consistently take a long time to run
Using this information we can move away from the ‘re-run’ culture and better support continuous integration goals of having quick, reliable, deterministic tests
Video of the talk is here: https://youtu.be/29fPYx7OJnE?list=PL_7kBU2XBlbKuRNVHeqjXUygXtToqMHsn
Eat whatever you can with PyBabe
This document discusses PyBabe, an open-source Python library for ETL (extract, transform, load) processes. PyBabe allows extracting data from various sources like FTP, SQL databases, and Amazon S3. It can perform transformations on the data like filtering, regular expressions, and date parsing. The transformed data can then be loaded to targets like SQL databases, MongoDB, Excel files, and more. PyBabe represents data as a stream of named tuples and processes the data lazily using generators for efficiency. Examples show how to use PyBabe to sort and join large files, send reports over email, and abstract ETL logic into reusable scripts.
Contract-driven development with OpenAPI 3 and Vert.x | DevNation Tech Talk
Have you ever been frustrated by developing and documenting an HTTP API? When it comes down to defining the HTTP interface between frontend and backend, have you ever had problems specifying the parameters or the shape of the body without misunderstandings? In this talk we’ll introduce you to "Contract Driven Development" (or API Design First approach), a methodology that uses declarative API Contracts to enable developers to efficiently design, communicate, and evolve their HTTP APIs, while automating API implementation phases where possible. In order to implement this methodology, we’ll show you how to develop an API contract using OpenAPI 3 and how you can easily implement the HTTP endpoints using Vert.x Web OpenAPI.
Mongo at Sailthru (MongoNYC 2011)
Sailthru uses MongoDB to store user profile data, including interests and behaviors, in documents rather than across multiple SQL tables. This allows them to access user data from any part of their system and personalize content. They run MongoDB on Amazon EC2 across 11 servers in 5 replica sets and 1 backup server totaling around 1TB of data. Key advantages of MongoDB for them include flexibility, performance, and not needing downtime for schema changes.
Agile Data Science 2.0
Agile Data Science 2.0 (O'Reilly 2017) defines a methodology and a software stack with which to apply the methods. *The methodology* seeks to deliver data products in short sprints by going meta and putting the focus on the applied research process itself. *The stack* is but an example of one meeting the requirements that it be utterly scalable and utterly efficient in use by application developers as well as data engineers. It includes everything needed to build a full-blown predictive system: Apache Spark, Apache Kafka, Apache Incubating Airflow, MongoDB, ElasticSearch, Apache Parquet, Python/Flask, JQuery. This talk will cover the full lifecycle of large data application development and will show how to use lessons from agile software engineering to apply data science using this full-stack to build better analytics applications. The entire lifecycle of big data application development is discussed. The system starts with plumbing, moving on to data tables, charts and search, through interactive reports, and building towards predictions in both batch and realtime (and defining the role for both), the deployment of predictive systems and how to iteratively improve predictions that prove valuable.
Log everything! @DC13
Deck36 is a small team of engineers who specialize in designing, implementing, and operating complex web systems. They discuss their approach to logging everything through a data pipeline that ingests data from producers, transports it via RabbitMQ, stores it in Hadoop HDFS and Amazon S3, runs analytics with Hadoop MapReduce and Amazon EMR, and performs real-time stream processing with Twitter Storm. They also live demo their JavaScript data collector client and a PHP/Storm example that processes click stream data.
Agile Data Science
Agile Data Science 2.0 (O'Reilly 2017) defines a methodology and a software stack with which to apply the methods. *The methodology* seeks to deliver data products in short sprints by going meta and putting the focus on the applied research process itself. *The stack* is but an example of one meeting the requirements that it be utterly scalable and utterly efficient in use by application developers as well as data engineers. It includes everything needed to build a full-blown predictive system: Apache Spark, Apache Kafka, Apache Incubating Airflow, MongoDB, ElasticSearch, Apache Parquet, Python/Flask, JQuery. This talk will cover the full lifecycle of large data application development and will show how to use lessons from agile software engineering to apply data science using this full-stack to build better analytics applications. The entire lifecycle of big data application development is discussed. The system starts with plumbing, moving on to data tables, charts and search, through interactive reports, and building towards predictions in both batch and realtime (and defining the role for both), the deployment of predictive systems and how to iteratively improve predictions that prove valuable.
Open Source World June '21 -- JSON Within a Relational Database
How does free form JSON work with a strictly typed relational database? This session will cover how MySQL and JSON give you the best of both worlds
Learning analytics inside government
Jisc is a UK organization that aims to advance digital technology in education and research. Their Learning Analytics project has three core parts: a learning analytics service, toolkit, and community. The service provides dashboards and tools to analyze student data from various sources to identify at-risk students and enable interventions. It follows an open architecture approach. The toolkit includes guidance on best practices like privacy and consent. The community aspect involves events, blogs, and mailing lists to bring people together around learning analytics.
Montreal Elasticsearch Meetup
This document provides an overview of Elasticsearch, including what it is, how it works, and how to perform basic operations like indexing, updating, and searching documents. It explains that Elasticsearch allows for advanced search across large amounts of data by making documents searchable and scaling easily. It also demonstrates how to index, update, search for, and retrieve documents through RESTful API calls. Faceted search, aggregations, and cluster architecture are also summarized.
Docker Summit MongoDB - Data Democratization
The document summarizes a presentation given by Chris Grabosky, a Solutions Architect at MongoDB. It discusses how containerization and data democratization can help organizations build cloud native applications. It highlights how MongoDB and Docker can be used together to build scalable and portable apps. It also covers data modeling approaches, workload isolation, analytics, and data access controls that MongoDB provides to help democratize data.
DevSecCon London 2018: Open DevSecOps
PETKO D. PETKOV
Thanks to the DevSecOps philosophy a growing number of organisations around the world are ensuring their businesses are set up with the security in mind from the get-go. DevSecOps is taking the world by storm. This talk is about how to introduce DevSecOps in your organisation with ready-made, zero-cost, open source templates accessible to everyone. The talk will introduce the OpenDevSecOps project and show many practical examples of how to easily deploy security testing infrastructure on top of existing and well-established development tools.
Xain.io exhibiting at Berlin Tech Job Fair Spring 2020
XAIN is a company that provides federated learning as a service to allow organizations to train AI models on decentralized data without aggregating or anonymizing private user data. Their federated learning platform allows models to be trained separately on each organization's local data and then combines the models for improved accuracy while keeping all data localized. They have successfully worked with companies like Porsche and are hiring various engineering roles.
Tapping into Scientific Data with Hadoop and Flink
At ResearchGate, we constantly analyze scientific data to connect the world of science and make research open to all. It can be tricky to set up a process to continuously deliver improved versions of algorithms that tap into more than 100 million publications and corresponding bibliographic metadata. In this talk, we illustrate some (big) data engineering challenges of running data pipelines and incorporating results into the live databases that power our user-facing features every day. We show how Apache Flink helps us to improve performance, robustness, ease of maintenance - and most importantly - have more fun while building big data pipelines.
Telling the LivePerson Technology Story at Couchbase [SF] 2013
As part of Couchbase[SF]2013, Ido Shilon, R&D Group Leader at LivePerson, discusses LivePerson's project to re-architect their LiveEngage platform backend, with a focus on LivePerson's decision to use NoSQL technologies, the challenges encountered, and the lessons learned.
Video: http://www.youtube.com/watch?v=rYKWFmJEHX0
Human-in-the-loop: a design pattern for managing teams which leverage ML by P...
Human-in-the-loop is an approach which has been used for simulation, training, UX mockups, etc.
https://www.bigdataspain.org/2017/talk/human-in-the-loop-a-design-pattern-for-managing-teams-which-leverage-ml
Big Data Spain Conference
16th -17th November - Kinépolis Madrid
ICONUK 2016: REST Assured, Freeing Your Domino Data Has Never Been That Easy!
ICONUK 2016 Developer Session on RESTful Services with IBM Domino Server and Apache Wink. September 16, 2016
Similar to Real-Time Streaming with Python ML Inference (20)
Data infrastructure architecture for medium size organization: tips for colle...
Data infrastructure architecture for medium size organization: tips for colle...
Webinar elastic stack {on telecom} english webinar part (1)
Webinar elastic stack {on telecom} english webinar part (1)
Test Trend Analysis : Towards robust, reliable and timely tests
Test Trend Analysis : Towards robust, reliable and timely tests
Contract-driven development with OpenAPI 3 and Vert.x | DevNation Tech Talk
Contract-driven development with OpenAPI 3 and Vert.x | DevNation Tech Talk
Open Source World June '21 -- JSON Within a Relational Database
Open Source World June '21 -- JSON Within a Relational Database
Xain.io exhibiting at Berlin Tech Job Fair Spring 2020
Xain.io exhibiting at Berlin Tech Job Fair Spring 2020
Tapping into Scientific Data with Hadoop and Flink
Tapping into Scientific Data with Hadoop and Flink
Telling the LivePerson Technology Story at Couchbase [SF] 2013
Telling the LivePerson Technology Story at Couchbase [SF] 2013
Human-in-the-loop: a design pattern for managing teams which leverage ML by P...
Human-in-the-loop: a design pattern for managing teams which leverage ML by P...
ICONUK 2016: REST Assured, Freeing Your Domino Data Has Never Been That Easy!
ICONUK 2016: REST Assured, Freeing Your Domino Data Has Never Been That Easy!
Recently uploaded
Energy consumption of Database Management - Florina Jonuzi
Presentation from Florina Jonuzi at the GSD Community Stage Meetup on June 06, 2024
ALGIT - Assembly Line for Green IT - Numbers, Data, Facts
Presentation from Ghazal Aakel and Eric Jochum at the GSD Community Stage Meetup on June 6th, 2024
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
No one wants their application to drag like a car stuck in the slow lane! Yet it’s all too common to encounter bumpy, pothole-filled solutions that slow the speed of any application. Symfony apps are not an exception.
In this talk, I will take you for a spin around the performance racetrack. We’ll explore common pitfalls - those hidden potholes on your application that can cause unexpected slowdowns. Learn how to spot these performance bumps early, and more importantly, how to navigate around them to keep your application running at top speed.
We will focus in particular on tuning your engine at the application level, making the right adjustments to ensure that your system responds like a well-oiled, high-performance race car.
Unveiling the Advantages of Agile Software Development.pdf
Learn about Agile Software Development's advantages. Simplify your workflow to spur quicker innovation. Jump right in! We have also discussed the advantages.
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
The UI5 tooling is the development and build tooling of UI5. It is built in a modular and extensible way so that it can be easily extended by your needs. This session will showcase various tooling extensions which can boost your development experience by far so that you can really work offline, transpile your code in your project to use even newer versions of EcmaScript (than 2022 which is supported right now by the UI5 tooling), consume any npm package of your choice in your project, using different kind of proxies, and even stitching UI5 projects during development together to mimic your target environment.
Measures in SQL (SIGMOD 2024, Santiago, Chile)
SQL has attained widespread adoption, but Business Intelligence tools still use their own higher level languages based upon a multidimensional paradigm. Composable calculations are what is missing from SQL, and we propose a new kind of column, called a measure, that attaches a calculation to a table. Like regular tables, tables with measures are composable and closed when used in queries.
SQL-with-measures has the power, conciseness and reusability of multidimensional languages but retains SQL semantics. Measure invocations can be expanded in place to simple, clear SQL.
To define the evaluation semantics for measures, we introduce context-sensitive expressions (a way to evaluate multidimensional expressions that is consistent with existing SQL semantics), a concept called evaluation context, and several operations for setting and modifying the evaluation context.
A talk at SIGMOD, June 9–15, 2024, Santiago, Chile
Authors: Julian Hyde (Google) and John Fremlin (Google)
https://doi.org/10.1145/3626246.3653374
Requirement Traceability in Xen Functional Safety
Using OpenFastTrace to trace requirements in Xen functional safety
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
Charla impartida en el evento de "KuberTENes Birthday Bash Guadalajara" para celebrar el 10mo. aniversario de Kubernetes #kuberTENes #celebr8k8s #k8s
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
Learn about the latest innovations in and around OpenUI5/SAPUI5: UI5 Tooling, UI5 linter, UI5 Web Components, Web Components Integration, UI5 2.x, UI5 GenAI.
Recording:
https://www.youtube.com/live/MSdGLG2zLy8?si=INxBHTqkwHhxV5Ta&t=0
A Study of Variable-Role-based Feature Enrichment in Neural Models of Code
Understanding variable roles in code has been found to be helpful by students
in learning programming -- could variable roles help deep neural models in
performing coding tasks? We do an exploratory study.
- These are slides of the talk given at InteNSE'23: The 1st International Workshop on Interpretability and Robustness in Neural Software Engineering, co-located with the 45th International Conference on Software Engineering, ICSE 2023, Melbourne Australia
What is Master Data Management by PiLog Group
PiLog Group's Master Data Record Manager (MDRM) is a sophisticated enterprise solution designed to ensure data accuracy, consistency, and governance across various business functions. MDRM integrates advanced data management technologies to cleanse, classify, and standardize master data, thereby enhancing data quality and operational efficiency.
Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
Dr. Jesús Barrasa, Head of Solutions Architecture for EMEA, Neo4j
Découvrez les dernières innovations de Neo4j, et notamment les dernières intégrations cloud et les améliorations produits qui font de Neo4j un choix essentiel pour les développeurs qui créent des applications avec des données interconnectées et de l’IA générative.
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Allez au-delà du battage médiatique autour de l’IA et découvrez des techniques pratiques pour utiliser l’IA de manière responsable à travers les données de votre organisation. Explorez comment utiliser les graphes de connaissances pour augmenter la précision, la transparence et la capacité d’explication dans les systèmes d’IA générative. Vous partirez avec une expérience pratique combinant les relations entre les données et les LLM pour apporter du contexte spécifique à votre domaine et améliorer votre raisonnement.
Amenez votre ordinateur portable et nous vous guiderons sur la mise en place de votre propre pile d’IA générative, en vous fournissant des exemples pratiques et codés pour démarrer en quelques minutes.
Oracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers. This video you can watch from my youtube channel at https://youtu.be/m-F-mZA3MkU
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
Regarding mobile operating systems, two major players dominate our thoughts: Android and iPhone. With Android leading the market, software development companies are focused on delivering apps compatible with this OS. Ensuring an app's functionality across various Android devices, OS versions, and hardware specifications is critical, making Android app testing essential.
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
原版一模一样【微信：741003700 】【美国纽约州立大学奥尔巴尼分校毕业证学位证书】【微信：741003700 】学位证，留信认证（真实可查，永久存档）offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。
本公司拥有海外各大学样板无数，能完美还原海外各大学 Bachelor Diploma degree, Master Degree Diploma
1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Salesforce Healthcare CRM, implemented by VALiNTRY360, revolutionizes patient management by enhancing patient engagement, streamlining administrative processes, and improving care coordination. Its advanced analytics, robust security, and seamless integration with telehealth services ensure that healthcare providers can deliver personalized, efficient, and secure patient care. By automating routine tasks and providing actionable insights, Salesforce Healthcare CRM enables healthcare providers to focus on delivering high-quality care, leading to better patient outcomes and higher satisfaction. VALiNTRY360's expertise ensures a tailored solution that meets the unique needs of any healthcare practice, from small clinics to large hospital systems.
For more info visit us https://valintry360.com/solutions/health-life-sciences
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Crescat is industry-trusted event management software, built by event professionals for event professionals. Founded in 2017, we have three key products tailored for the live event industry.
Crescat Event for concert promoters and event agencies. Crescat Venue for music venues, conference centers, wedding venues, concert halls and more. And Crescat Festival for festivals, conferences and complex events.
With a wide range of popular features such as event scheduling, shift management, volunteer and crew coordination, artist booking and much more, Crescat is designed for customisation and ease-of-use.
Over 125,000 events have been planned in Crescat and with hundreds of customers of all shapes and sizes, from boutique event agencies through to international concert promoters, Crescat is rigged for success. What's more, we highly value feedback from our users and we are constantly improving our software with updates, new features and improvements.
If you plan events, run a venue or produce festivals and you're looking for ways to make your life easier, then we have a solution for you. Try our software for free or schedule a no-obligation demo with one of our product specialists today at crescat.io
Fundamentals of Programming and Language Processors
Fundamentals of Programming and Language Processors
Recently uploaded (20)
Energy consumption of Database Management - Florina Jonuzi
Energy consumption of Database Management - Florina Jonuzi
ALGIT - Assembly Line for Green IT - Numbers, Data, Facts
ALGIT - Assembly Line for Green IT - Numbers, Data, Facts
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Unveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdf
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
A Study of Variable-Role-based Feature Enrichment in Neural Models of Code
A Study of Variable-Role-based Feature Enrichment in Neural Models of Code
Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Oracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptx
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Fundamentals of Programming and Language Processors
Fundamentals of Programming and Language Processors
Real-Time Streaming with Python ML Inference
- 1. Real-Time Streaming with Python ML Inference Marko Topolnik 1
- 2. About Us Hazelcast started as an In-Memory Data Grid project (distributed caching with data-local computation) Java codebase, polyglot clients we focus on Simplicity in 2017 we added Hazelcast Jet: In-Memory Distributed Stream Processing My role: core engineer at Hazelcast Jet 2
- 3. Example: Salary Prediction Python, SciKit Learn 3
- 4. Sample Input and Output Input: { "age": 25, "workclass": "Self-emp", "fnlwgt": 176756, "education": "HS-grad", "education-num": 9, "marital-status": "Never-married", "occupation": "Farming-fishing", "relationship": "Own-child", "capital-gain": 0, "capital-loss": 0, "hours-per-week": 35, "native-country": "United-States" } Output: { "probability": 0.85 "income": "<=50K" } 4
- 6. Data Scientist's Perspective Do Data Science Profit! 6
- 7. Data Engineer's Perspective Do Data Science Monitor Profit (fingers crossed) 7 Package Load-Balance Scale Up Scale Out Deploy Re-Train
- 8. Productionizing the REST service 8 Request #1 Request #2 Request #3 R E S T Batching?
- 9. Productionizing the REST service 9 Request #1 Request #2 Request #3 R E S T Request Queue Batch Response Queue Parallelism?
- 10. Productionizing the REST service 10 Request #1 Request #2 Request #3 R E S T R E S T Load Balancer Request Queue Response Queue Batch Batch
- 11. Replace REST with Distributed Streaming 11 Request #1 Request #2 Request #3 Kafka Hazelcast Jet Cluster $ jet submit ML
- 12. Hazelcast Jet Code Pipeline p = Pipeline.create(); p.readFrom(Kafka.source()) .apply(mapUsingPython(new PythonServiceConfig() .setBaseDir("/Users/mtopol/dev/python/sklearn") .setHandlerModule("example_1_inference_jet"))) .writeTo(Kafka.sink()); jet.newJob(p); $ mvn package $ jet submit target/my-job.jar 12
- 13. Jet Node 1 Jet Job's Execution Plan Source mapUsing Python mapUsing Python Sink 13 Kafka Topic A Python process Kafka Topic B Batch Python process Batch
- 14. Cluster Elasticity and Resilience ● Jet jobs are fault-tolerant ● nodes can join and leave the cluster, jobs go on ● automatically rescale to available hardware 14
- 15. Let's Start a Jet Cluster! 15
- 16. Cluster Self-Formation 16 Hazelcast Jet natively supports: ● Amazon AWS ● Google GCP ● Kubernetes With simple configuration, the nodes self-discover in these environments
- 17. Source and Sink Connectors ● Kafka ● Hadoop HDFS ● S3 bucket ● JDBC ● JMS queue and topic ● custom connector ● test connectors: convenient to test the code before submitting 17
- 18. Stream Operators ● windowed aggregation using Event Time ○ sliding, session window ○ count, sum, average, linear regression, ... ○ custom aggregate function ● rolling aggregation ● streaming join (co-grouping) ● hash join (enrichment) ● contact arbitrary external services ○ mapUsingPython uses this 18
Editor's Notes
- Ask "Are you using Python in your data science?" What languages are you using?
- Step one: Do Data Science Step two: Profit!
- Then along comes a Data Engineer and asks "Wait, aren't we forgetting something?"