Open Source Tools for Big Data
•
0 likes•59 views
This document discusses open source tools for big data. It begins with an introduction to Emblica, a small company of 5 people focused on data engineering, DevOps, and machine learning. It then discusses what constitutes "big data" in terms of volume, velocity, and variety. While the scale may not be as large as Facebook's data, open source big data tools can still be useful for datasets as small as gigabytes. Examples of big data include activity data, sensor data, and unstructured data. The document then outlines the history of open source data processing tools and frameworks like Google File System (GFS), Hadoop, Spark, Kafka, and Kappa architecture. It provides two use cases utilizing these tools:
Report
Share
Report
Share
Download to read offline
Recommended
Airflow - An Open Source Platform to Author and Monitor Data Pipelines
Airflow is an open source platform for authoring and monitoring data pipelines. It was developed at Airbnb to address challenges like opaque data lineage, steep learning curves as ecosystems grow, duplicated code, and scattered operational metadata. Airflow uses a Python-based DAG (directed acyclic graph) definition to programmatically author pipelines. It has a rich CLI and web UI and uses technologies like Python, Celery, Flask, SQLAlchemy, and Jinja. Operators allow running tasks like SQL queries, transfers, and sensors. Airflow has been scaled to process thousands of tasks daily across many teams and companies.
Accelerating Machine Learning and Deep Learning At Scale...With Apache Spark:...
Deep learning is a fast growing subset of machine learning. There is an emerging trend to conduct deep learning in the same cluster along with existing data processing pipelines to support feature engineering and traditional machine learning. As the leading framework for Distributed ML, we believe that the addition of deep learning to the super-popular Spark framework is important, because it allows Spark developers to perform a range of data analysis tasks within a single framework that helps avoid the complexity inherent in using multiple frameworks and libraries. As one of the early and top contributors to Apache Spark, Intel is thrilled to share with the community a big deal contribution to open source Spark…”BigDL” -… A distributed deep Learning framework organically built on Big Data (Apache Spark) platform. It combines the benefits of “high performance computing” and “Big Data” architecture for rich deep learning support. With BigDL on Spark, customers can eliminate large volume of unnecessary dataset transfer between separate systems, eliminate separate HW clusters and move towards a CPU cluster, reduce system complexity and the latency for end-to-end learning. Ultimately, customers can achieve better scale, higher resource utilization, ease of use/development, and better TCO. Feature parity with Caffe and Torch, significant performance boost when combined with Intel’s Math Kernel Library (MKL), scale-out, fault tolerance, elasticity and dynamic resource sharing are some of the prominent features of BigDL.
BigDL open source project will be launched at the 2017 Spark Summit East and this keynote will help spotlight this new contribution and benefits to the Spark developer community and encourage their wide contribution and collaboration. We will also showcase some real world applications of Big DL from early customers’ adoption.
Hadoop Everywhere
This document discusses the need for geo-distributed analytics infrastructures and storage to support big data workloads across multiple data centers. It introduces EMC's ECS storage platform as a solution, which provides a global namespace, geo-caching, strong consistency, and active-active replication across data centers. ECS can serve as a Hadoop storage backend through its Hadoop compatible client library and provides multi-tenant, exabyte scale storage for large and small files that addresses issues with data silos and underutilization of resources in current Hadoop deployments.
Scaling Through Simplicity—How a 300 million User Chat App Reduced Data Engin...
Moving at the speed of a startup often means rapid iterative development, which can lead to a patchwork of systems and processes. In the early days at Kik (one of the most popular chat apps among U.S. teens), the data team was able to move extremely quickly but often at the expense of scalable data engineering. In this session, Kik’s head of data will share the eight things they did to save time and money. The team took their data stack from a complex combination of systems and processes to a scalable, simple, and robust platform leveraging Apache Spark and Databricks to make data super easy for everyone in the company to use.
Building Notebook-based AI Pipelines with Elyra and Kubeflow
A typical machine learning pipeline begins as a series of preprocessing steps followed by experimentation, optimization and model-tuning, and, finally deployment. Jupyter notebooks have become a hugely popular tool for data scientists and other machine learning practitioners to explore and experiment as part of this workflow, due to the flexibility and interactivity they provide. However, with notebooks it is often a challenge to move from the experimentation phase to creating a robust, modular and production-grade end-to-end AI pipeline.
Hadoop and Friends as Key Enabler of the IoE - Continental's Dynamic eHorizon
Hadoop and Friends as Key Enabler of the IoE - Continental's Dynamic eHorizonDataWorks Summit/Hadoop Summit
The document discusses Continental's Dynamic eHorizon platform, which uses big data technologies like Hadoop to provide extended sensor information to vehicles beyond their own sensor range. It summarizes Continental's vision of using collected vehicle data on challenges like comfort, efficiency and safety. The platform connects vehicles to Continental's backend which uses Hadoop for large-scale data processing and analytics to generate enhanced maps and extract insights. Continental sees opportunities to improve their architecture by leveraging cloud-based Hadoop platforms for flexibility and reduced costs.Cornami Accelerates Performance on SPARK: Spark Summit East talk by Paul Master
CORNAMI has developed TruStream technology, a new architecture featuring a high-density processor core count memory fabric.
• This session will deal with CORNAMI’s integration of SPARK into its TruStream Compute Fabric to provide higher performance in computational processing capability to generic SPARK workloads
• And Include a look at current multi-server cluster implementation of applications and demonstrate how our technology plus SPARK can be used to accelerate algorithms and/or increase functionality with lower
cost, power, latency and footprint in lead application areas of Ai and Machine Learning
• Use case will be presented ” Yahoo Streaming Benchmark” Measuring Real-Time Mobile Advertising performance on SPARK
JanusGraph, Jupyter Meetup NYC
JanusGraph: What's Next, Project Status Update. Presented at Open Source Graph Technologies NYC Meetup on August 24, 2017. https://www.meetup.com/graphs/events/241136321/
Recommended
Airflow - An Open Source Platform to Author and Monitor Data Pipelines
Airflow is an open source platform for authoring and monitoring data pipelines. It was developed at Airbnb to address challenges like opaque data lineage, steep learning curves as ecosystems grow, duplicated code, and scattered operational metadata. Airflow uses a Python-based DAG (directed acyclic graph) definition to programmatically author pipelines. It has a rich CLI and web UI and uses technologies like Python, Celery, Flask, SQLAlchemy, and Jinja. Operators allow running tasks like SQL queries, transfers, and sensors. Airflow has been scaled to process thousands of tasks daily across many teams and companies.
Accelerating Machine Learning and Deep Learning At Scale...With Apache Spark:...
Deep learning is a fast growing subset of machine learning. There is an emerging trend to conduct deep learning in the same cluster along with existing data processing pipelines to support feature engineering and traditional machine learning. As the leading framework for Distributed ML, we believe that the addition of deep learning to the super-popular Spark framework is important, because it allows Spark developers to perform a range of data analysis tasks within a single framework that helps avoid the complexity inherent in using multiple frameworks and libraries. As one of the early and top contributors to Apache Spark, Intel is thrilled to share with the community a big deal contribution to open source Spark…”BigDL” -… A distributed deep Learning framework organically built on Big Data (Apache Spark) platform. It combines the benefits of “high performance computing” and “Big Data” architecture for rich deep learning support. With BigDL on Spark, customers can eliminate large volume of unnecessary dataset transfer between separate systems, eliminate separate HW clusters and move towards a CPU cluster, reduce system complexity and the latency for end-to-end learning. Ultimately, customers can achieve better scale, higher resource utilization, ease of use/development, and better TCO. Feature parity with Caffe and Torch, significant performance boost when combined with Intel’s Math Kernel Library (MKL), scale-out, fault tolerance, elasticity and dynamic resource sharing are some of the prominent features of BigDL.
BigDL open source project will be launched at the 2017 Spark Summit East and this keynote will help spotlight this new contribution and benefits to the Spark developer community and encourage their wide contribution and collaboration. We will also showcase some real world applications of Big DL from early customers’ adoption.
Hadoop Everywhere
This document discusses the need for geo-distributed analytics infrastructures and storage to support big data workloads across multiple data centers. It introduces EMC's ECS storage platform as a solution, which provides a global namespace, geo-caching, strong consistency, and active-active replication across data centers. ECS can serve as a Hadoop storage backend through its Hadoop compatible client library and provides multi-tenant, exabyte scale storage for large and small files that addresses issues with data silos and underutilization of resources in current Hadoop deployments.
Scaling Through Simplicity—How a 300 million User Chat App Reduced Data Engin...
Moving at the speed of a startup often means rapid iterative development, which can lead to a patchwork of systems and processes. In the early days at Kik (one of the most popular chat apps among U.S. teens), the data team was able to move extremely quickly but often at the expense of scalable data engineering. In this session, Kik’s head of data will share the eight things they did to save time and money. The team took their data stack from a complex combination of systems and processes to a scalable, simple, and robust platform leveraging Apache Spark and Databricks to make data super easy for everyone in the company to use.
Building Notebook-based AI Pipelines with Elyra and Kubeflow
A typical machine learning pipeline begins as a series of preprocessing steps followed by experimentation, optimization and model-tuning, and, finally deployment. Jupyter notebooks have become a hugely popular tool for data scientists and other machine learning practitioners to explore and experiment as part of this workflow, due to the flexibility and interactivity they provide. However, with notebooks it is often a challenge to move from the experimentation phase to creating a robust, modular and production-grade end-to-end AI pipeline.
Hadoop and Friends as Key Enabler of the IoE - Continental's Dynamic eHorizon
Hadoop and Friends as Key Enabler of the IoE - Continental's Dynamic eHorizonDataWorks Summit/Hadoop Summit
The document discusses Continental's Dynamic eHorizon platform, which uses big data technologies like Hadoop to provide extended sensor information to vehicles beyond their own sensor range. It summarizes Continental's vision of using collected vehicle data on challenges like comfort, efficiency and safety. The platform connects vehicles to Continental's backend which uses Hadoop for large-scale data processing and analytics to generate enhanced maps and extract insights. Continental sees opportunities to improve their architecture by leveraging cloud-based Hadoop platforms for flexibility and reduced costs.Cornami Accelerates Performance on SPARK: Spark Summit East talk by Paul Master
CORNAMI has developed TruStream technology, a new architecture featuring a high-density processor core count memory fabric.
• This session will deal with CORNAMI’s integration of SPARK into its TruStream Compute Fabric to provide higher performance in computational processing capability to generic SPARK workloads
• And Include a look at current multi-server cluster implementation of applications and demonstrate how our technology plus SPARK can be used to accelerate algorithms and/or increase functionality with lower
cost, power, latency and footprint in lead application areas of Ai and Machine Learning
• Use case will be presented ” Yahoo Streaming Benchmark” Measuring Real-Time Mobile Advertising performance on SPARK
JanusGraph, Jupyter Meetup NYC
JanusGraph: What's Next, Project Status Update. Presented at Open Source Graph Technologies NYC Meetup on August 24, 2017. https://www.meetup.com/graphs/events/241136321/
Cylc - the Python workflow engine for cycling systems.
Cylc - the Python workflow engine for cycling systems.
Cylc ("silk") is a general-purpose Python workflow engine or 'metascheduler'; a system that can automatically execute tasks according to complex schedules and dependencies, and deal with task failures if they occur. We will showcase the applications of Cylc in weather and climate services, and reflect on how its implementation in Python has aided the development of the Cylc software package.
Complex workflows are increasingly at the heart of scientific research and business logic. This is true across a range of industries and scientific disciplines where we find workflows consisting of thousands of tasks which require running to a non-trivial schedule. At the Met Office, for example, we use Cylc to manage weather forecasting and climate research activities, where it has been used to drive operational systems since 2013. Cylc can manage workflows on a variety of computing platforms, from desktop PCs to high-performance computing services, and is currently used to efficiently utilise resources on some of the largest supercomputers in the world. Cylc is an open-source project led by a small team of core developers at the Met Office (UK) and the National Institute of Water and Atmospheric Research (New Zealand). The cylc website can be found at cylc.github.io/cylc/.
HBaseCon 2015: Warcbase - Scaling 'Out' and 'Down' HBase for Web Archiving
This document summarizes a presentation about Warcbase, an open-source platform for managing web archives built on Hadoop and HBase. Warcbase scales to large web archive datasets by using Hadoop for distributed storage and processing and HBase for efficient random access to archived web pages and associated metadata. The presentation demonstrates how Warcbase can power applications for temporal browsing of archived pages, topic modeling, and interactive visualization of the link graph within a web archive. It also experiments with running Warcbase on a single powerful computer and even a low-power Raspberry Pi to enable new personal and mobile uses of web archiving.
Modernizing Global Shared Data Analytics Platform and our Alluxio Journey
Data Orchestration Summit 2020 organized by Alluxio
https://www.alluxio.io/data-orchestration-summit-2020/
Modernizing Global Shared Data Analytics Platform and our Alluxio Journey
Sandipan Chakraborty, Director of Engineering (Rakuten)
About Alluxio: alluxio.io
Engage with the open source community on slack: alluxio.io/slack
Microservices Live
Glynn Bird - Building the "microservices way" involves breaking monolithic IT systems into small, decoupled services that each to one job well. This talk builds a practical microservices architecture during the talk using small Node.js apps that perform storage, analytics and visualisation tasks. Learn how to orchestrate your own microservice architecture using simple, easily-tested building blocks.
20181003 Whirlwind tour into Pyspark
This document provides an overview of Apache Spark, including:
1) A brief history of Spark and description of its core components for batch processing, streaming, SQL queries, machine learning and graph processing.
2) Benefits of Spark such as performance, developer productivity, ecosystem support, and a unified engine.
3) Explanations of what makes Spark fast, including in-memory computing and caching of remote data locally.
4) A link to learn more about Databricks, a company that provides a cloud-based analytics platform with Spark.
Analytics at the Real-Time Speed of Business: Spark Summit East talk by Manis...
Redis accelerates Apache Spark execution by 45 times, when used as a shared distributed in-memory datastore for Spark in analyses like time series data range queries. With the redis module for machine learning, redis-ml, implementation of spark-ml models gains a new real time serving layer that offloads processing of models directly in Redis, allows multiple applications to reuse the same models and speeds up classification and execution of these models by 13x. Join this session to learn more about the Redis Labs’ connector for Apache Spark that enhances production implementations of real-time big data processing.
Building Data Lakes with Apache Airflow
Build a simple Data Lake on AWS using a combination of services, including Amazon Managed Workflows for Apache Airflow (Amazon MWAA), AWS Glue, AWS Glue Studio, Amazon Athena, and Amazon S3.
Blog post and link to the video: https://garystafford.medium.com/building-a-data-lake-with-apache-airflow-b48bd953c2b
Hw09 Hadoop Applications At Yahoo!
Yahoo! is the largest contributor, tester, and user of Hadoop. They operate 4000 node Hadoop clusters and release the Yahoo! Distribution of Hadoop as open source. Yahoo! contributes all their Hadoop development work back to the Apache Foundation and uses Hadoop for large-scale data processing and analytics across petabytes of data.
Accelerate Analytics and ML in the Hybrid Cloud Era
Alluxio Webinar
April 6, 2021
For more Alluxio events: https://www.alluxio.io/events/
Speakers:
Alex Ma, Alluxio
Peter Behrakis, Alluxio
Many companies we talk to have on premises data lakes and use the cloud(s) to burst compute. Many are now establishing new object data lakes as well. As a result, running analytics such as Hive, Spark, Presto and machine learning are experiencing sluggish response times with data and compute in multiple locations. We also know there is an immense and growing data management burden to support these workflows.
In this talk, we will walk through what Alluxio’s Data Orchestration for the hybrid cloud era is and how it solves the performance and data management challenges we see.
In this tech talk, we'll go over:
- What is Alluxio Data Orchestration?
- How does it work?
- Alluxio customer results
Lightning-Fast Analytics for Workday Transactional Data with Pavel Hardak and...
Workday Prism Analytics enables data discovery and interactive Business Intelligence analysis for Workday customers. Workday is a “pure SaaS” company, providing a suite of Financial and HCM (Human Capital Management) apps to about 2000 companies around the world, including more than 30% from Fortune-500 list. There are significant business and technical challenges to support millions of concurrent users and hundreds of millions daily transactions. Using memory-centric graph-based architecture allowed to overcome most of these problems.
As Workday grew, data transactions from existing and new customers generated vast amounts of valuable and highly sensitive data. The next big challenge was to provide in-app analytics platform, which for the multiple types of accumulated data, and also would allow using blend in external datasets. Workday users wanted it to be super-fast, but also intuitive and easy-to-use both for the financial and HR analysts and for regular, less technical users. Existing backend technologies were not a good fit, so we turned to Apache Spark.
In this presentation, we will share the lessons we learned when building highly scalable multi-tenant analytics service for transactional data. We will start with the big picture and business requirements. Then describe the architecture with batch and interactive modules for data preparation, publishing, and query engine, noting the relevant Spark technologies. Then we will dive into the internals of Prism’s Query Engine, focusing on Spark SQL, DataFrames and Catalyst compiler features used. We will describe the issues we encountered while compiling and executing complex pipelines and queries, and how we use caching, sampling, and query compilation techniques to support interactive user experience.
Finally, we will share the future challenges for 2018 and beyond.
How Spark Enables the Internet of Things- Paula Ta-Shma
The document summarizes an IBM research paper on how Spark can enable Internet of Things (IoT) use cases. It describes an IoT architecture used for a smart city use case with Madrid city buses. Data is collected from 3000 traffic sensors into Kafka and aggregated into Swift objects using Secor. Spark is used to access and analyze the data to detect traffic patterns and inform bus routing decisions in real-time. The system aims to improve customer satisfaction and reduce costs by responding efficiently to traffic issues.
Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Me...
Apache Druid is a high-performance distributed analytics store for modern analytics applications. It supports ingesting millions of events per second and sub-second query processing. Druid supports various types of data sources for ingestion, including Apache Kafka. You can immediately query on stream events once they get ingested into Druid. Since Kafka provides scalable and robust data delivery while Druid supports advanced complex analysis on streams, Kafka and Druid are widely used together for BI and operational analytics use cases, which require interactivity, scalability, real-time, and performance.
This talk is based on our real-world experiences building out streaming analytics stacks powering production use cases across many industries.
An End-to-End Spark-Based Machine Learning Stack in the Hybrid Cloud with Far...
We have deployed a hybrid cloud storage solution that leverages compute in the public cloud along with our specialized hardware storage. We will discuss the tradeoffs of hybrid cloud storage, which workloads are best suited for this model, the pipeline we have deployed, and the challenges and best practices we have learned. Spark provides a flexible compute environment that can be used alongside todays cloud compute providers.
However in read-heavy workloads that dominate much of analysis and machine learning today, storage costs scale poorly on these same cloud storage models. Hybrid cloud offers an alternative approach to get amortized storage costs over a dedicated link while using elastic compute in the cloud. We are currently running an end to end data science stack with multiple production workloads with this setup – A Spark-based ETL for transforming the real time log data that we ingest from our devices in the field into databases, a scale-out general regular expression search over log files that provides our support engineers real time access to searching for pathologies across our customer base, and a Spark based machine learning system for time series analysis to predict various customer metrics.
Real-Time Analytics with Spark and MemSQL
"Real-Time Analytics with Spark and MemSQL" by Neil Dahlke, engineer at MemSQL for Orange County Roadshow March 17, 2017.
Virtual Flink Forward 2020: Netflix Data Mesh: Composable Data Processing - J...
Netflix processes trillions of events and petabytes of data a day in the Keystone data pipeline, which is built on top of Apache Flink. As Netflix has scaled up original productions annually enjoyed by more than 150 million global members, data integration across the streaming service and the studio has become a priority. Scalably integrating data across hundreds of different data stores in a way that enables us to holistically optimize cost, performance and operational concerns presented a significant challenge. Learn how we expanded the scope of the Keystone pipeline into the Netflix Data Mesh, our real-time, general-purpose, data transportation platform for moving data between Netflix systems. The Keystone Platform’s unique approach to declarative configuration and schema evolution, as well as our approach to unifying batch and streaming data and processing will be covered in depth.
Airline Reservations and Routing: A Graph Use Case
We've all been there before... you hear the announcement that your flight is canceled. Fellow passengers race to the gate agent to rebook on the next available flight. How do they quickly determine the best route from Berlin to San Francisco? Ultimately the flight route network is best solved as a graph problem. We will discuss our lessons learned from working with a major airline to solve this problem using JanusGraph database. JanusGraph is an open source graph database designed for massive scale. It is compatible with several pieces of the open source big data stack: Apache TinkerPop (graph computing framework), HBase, Cassandra, and Solr. We will go into depth about our approach to benchmarking graph performance and discuss the utilities we developed. We will share our comparison results for evaluating which storage backend use with JanusGraph. Whether you are productizing a new database or you are a frustrated traveler, a fast resolution is needed to satisfy everybody involved. Presented at DataWorks Summit Berlin on April 18, 2018
Large Scale Graph Analytics with JanusGraph
This document provides an overview of large scale graph analytics and JanusGraph. It discusses graph databases and their use cases. JanusGraph is presented as an open source graph database that can scale to billions of vertices and edges across multiple storage backends like HBase, Cassandra and Bigtable. It uses the TinkerPop framework and Gremlin query language. JanusGraph supports ACID transactions, external indices, and evolving schemas. Example graph queries are demonstrated using the Gremlin console.
Data streaming
- The document profiles Alberto Paro and his experience including a Master's Degree in Computer Science Engineering from Politecnico di Milano, experience as a Big Data Practise Leader at NTTDATA Italia, authoring 4 books on ElasticSearch, and expertise in technologies like Apache Spark, Playframework, Apache Kafka, and MongoDB. He is also an evangelist for the Scala and Scala.JS languages.
The document then provides an overview of data streaming architectures, popular message brokers like Apache Kafka, RabbitMQ, and Apache Pulsar, streaming frameworks including Apache Spark, Apache Flink, and Apache NiFi, and streaming libraries such as Reactive Streams.
Transitioning from Traditional DW to Apache® Spark™ in Operating Room Predict...
The prevailing issue when working with Operating Room (OR) scheduling within a hospital setting is that it is difficult to schedule and predict available OR block times. This leads to empty and unused operating rooms leading to longer waiting times for patients for their procedures. In this three-part session, Ayad Shammout and Denny will show:
1) How we tried to solve this problem using traditional DW techniques
2) How we took advantage of the DW capabilities in Apache Spark AND easily transition to Spark MLlib so we could more easily predict available OR block times resulting in better OR utilization and shorter wait times for patients.
3) Some of the key learnings we had when migrating from DW to Spark.
High Performance Data Lake with Apache Hudi and Alluxio at T3Go
Data Orchestration Summit 2020 organized by Alluxio
https://www.alluxio.io/data-orchestration-summit-2020/
High Performance Data Lake with Apache Hudi and Alluxio at T3Go
Trevor Zhang & Vino Yang (T3Go)
About Alluxio: alluxio.io
Engage with the open source community on slack: alluxio.io/slack
NoSQL and Hadoop: A New Generation of Databases - Changing the Game: Monthly ...
NoSQL and Hadoop databases are emerging as alternatives to traditional relational databases for handling large amounts of unstructured data from sources like the cloud and web. Major tech companies like Oracle, IBM, Microsoft, EMC, Google and Amazon support NoSQL, with many choosing Apache Hadoop. Hadoop is an open source NoSQL database that can handle huge amounts of unstructured data at scale in cloud environments. It was designed to be fully distributed like Google's MapReduce and uses Java for integration. Relational databases remain effective for structured applications but face challenges with unstructured data, scale, and cloud deployments.
963
This document provides an overview of real time big data processing using Apache Kafka, Spark Streaming, Scala, and Elastic search. It begins with introductions to data mining, big data, and real time big data. It then discusses Apache Hadoop, Scala, Spark Streaming, Kafka, and Elastic search. The key technologies covered allow for distributed, low latency processing of streaming data at large volumes and velocities.
More Related Content
What's hot
Cylc - the Python workflow engine for cycling systems.
Cylc - the Python workflow engine for cycling systems.
Cylc ("silk") is a general-purpose Python workflow engine or 'metascheduler'; a system that can automatically execute tasks according to complex schedules and dependencies, and deal with task failures if they occur. We will showcase the applications of Cylc in weather and climate services, and reflect on how its implementation in Python has aided the development of the Cylc software package.
Complex workflows are increasingly at the heart of scientific research and business logic. This is true across a range of industries and scientific disciplines where we find workflows consisting of thousands of tasks which require running to a non-trivial schedule. At the Met Office, for example, we use Cylc to manage weather forecasting and climate research activities, where it has been used to drive operational systems since 2013. Cylc can manage workflows on a variety of computing platforms, from desktop PCs to high-performance computing services, and is currently used to efficiently utilise resources on some of the largest supercomputers in the world. Cylc is an open-source project led by a small team of core developers at the Met Office (UK) and the National Institute of Water and Atmospheric Research (New Zealand). The cylc website can be found at cylc.github.io/cylc/.
HBaseCon 2015: Warcbase - Scaling 'Out' and 'Down' HBase for Web Archiving
This document summarizes a presentation about Warcbase, an open-source platform for managing web archives built on Hadoop and HBase. Warcbase scales to large web archive datasets by using Hadoop for distributed storage and processing and HBase for efficient random access to archived web pages and associated metadata. The presentation demonstrates how Warcbase can power applications for temporal browsing of archived pages, topic modeling, and interactive visualization of the link graph within a web archive. It also experiments with running Warcbase on a single powerful computer and even a low-power Raspberry Pi to enable new personal and mobile uses of web archiving.
Modernizing Global Shared Data Analytics Platform and our Alluxio Journey
Data Orchestration Summit 2020 organized by Alluxio
https://www.alluxio.io/data-orchestration-summit-2020/
Modernizing Global Shared Data Analytics Platform and our Alluxio Journey
Sandipan Chakraborty, Director of Engineering (Rakuten)
About Alluxio: alluxio.io
Engage with the open source community on slack: alluxio.io/slack
Microservices Live
Glynn Bird - Building the "microservices way" involves breaking monolithic IT systems into small, decoupled services that each to one job well. This talk builds a practical microservices architecture during the talk using small Node.js apps that perform storage, analytics and visualisation tasks. Learn how to orchestrate your own microservice architecture using simple, easily-tested building blocks.
20181003 Whirlwind tour into Pyspark
This document provides an overview of Apache Spark, including:
1) A brief history of Spark and description of its core components for batch processing, streaming, SQL queries, machine learning and graph processing.
2) Benefits of Spark such as performance, developer productivity, ecosystem support, and a unified engine.
3) Explanations of what makes Spark fast, including in-memory computing and caching of remote data locally.
4) A link to learn more about Databricks, a company that provides a cloud-based analytics platform with Spark.
Analytics at the Real-Time Speed of Business: Spark Summit East talk by Manis...
Redis accelerates Apache Spark execution by 45 times, when used as a shared distributed in-memory datastore for Spark in analyses like time series data range queries. With the redis module for machine learning, redis-ml, implementation of spark-ml models gains a new real time serving layer that offloads processing of models directly in Redis, allows multiple applications to reuse the same models and speeds up classification and execution of these models by 13x. Join this session to learn more about the Redis Labs’ connector for Apache Spark that enhances production implementations of real-time big data processing.
Building Data Lakes with Apache Airflow
Build a simple Data Lake on AWS using a combination of services, including Amazon Managed Workflows for Apache Airflow (Amazon MWAA), AWS Glue, AWS Glue Studio, Amazon Athena, and Amazon S3.
Blog post and link to the video: https://garystafford.medium.com/building-a-data-lake-with-apache-airflow-b48bd953c2b
Hw09 Hadoop Applications At Yahoo!
Yahoo! is the largest contributor, tester, and user of Hadoop. They operate 4000 node Hadoop clusters and release the Yahoo! Distribution of Hadoop as open source. Yahoo! contributes all their Hadoop development work back to the Apache Foundation and uses Hadoop for large-scale data processing and analytics across petabytes of data.
Accelerate Analytics and ML in the Hybrid Cloud Era
Alluxio Webinar
April 6, 2021
For more Alluxio events: https://www.alluxio.io/events/
Speakers:
Alex Ma, Alluxio
Peter Behrakis, Alluxio
Many companies we talk to have on premises data lakes and use the cloud(s) to burst compute. Many are now establishing new object data lakes as well. As a result, running analytics such as Hive, Spark, Presto and machine learning are experiencing sluggish response times with data and compute in multiple locations. We also know there is an immense and growing data management burden to support these workflows.
In this talk, we will walk through what Alluxio’s Data Orchestration for the hybrid cloud era is and how it solves the performance and data management challenges we see.
In this tech talk, we'll go over:
- What is Alluxio Data Orchestration?
- How does it work?
- Alluxio customer results
Lightning-Fast Analytics for Workday Transactional Data with Pavel Hardak and...
Workday Prism Analytics enables data discovery and interactive Business Intelligence analysis for Workday customers. Workday is a “pure SaaS” company, providing a suite of Financial and HCM (Human Capital Management) apps to about 2000 companies around the world, including more than 30% from Fortune-500 list. There are significant business and technical challenges to support millions of concurrent users and hundreds of millions daily transactions. Using memory-centric graph-based architecture allowed to overcome most of these problems.
As Workday grew, data transactions from existing and new customers generated vast amounts of valuable and highly sensitive data. The next big challenge was to provide in-app analytics platform, which for the multiple types of accumulated data, and also would allow using blend in external datasets. Workday users wanted it to be super-fast, but also intuitive and easy-to-use both for the financial and HR analysts and for regular, less technical users. Existing backend technologies were not a good fit, so we turned to Apache Spark.
In this presentation, we will share the lessons we learned when building highly scalable multi-tenant analytics service for transactional data. We will start with the big picture and business requirements. Then describe the architecture with batch and interactive modules for data preparation, publishing, and query engine, noting the relevant Spark technologies. Then we will dive into the internals of Prism’s Query Engine, focusing on Spark SQL, DataFrames and Catalyst compiler features used. We will describe the issues we encountered while compiling and executing complex pipelines and queries, and how we use caching, sampling, and query compilation techniques to support interactive user experience.
Finally, we will share the future challenges for 2018 and beyond.
How Spark Enables the Internet of Things- Paula Ta-Shma
The document summarizes an IBM research paper on how Spark can enable Internet of Things (IoT) use cases. It describes an IoT architecture used for a smart city use case with Madrid city buses. Data is collected from 3000 traffic sensors into Kafka and aggregated into Swift objects using Secor. Spark is used to access and analyze the data to detect traffic patterns and inform bus routing decisions in real-time. The system aims to improve customer satisfaction and reduce costs by responding efficiently to traffic issues.
Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Me...
Apache Druid is a high-performance distributed analytics store for modern analytics applications. It supports ingesting millions of events per second and sub-second query processing. Druid supports various types of data sources for ingestion, including Apache Kafka. You can immediately query on stream events once they get ingested into Druid. Since Kafka provides scalable and robust data delivery while Druid supports advanced complex analysis on streams, Kafka and Druid are widely used together for BI and operational analytics use cases, which require interactivity, scalability, real-time, and performance.
This talk is based on our real-world experiences building out streaming analytics stacks powering production use cases across many industries.
An End-to-End Spark-Based Machine Learning Stack in the Hybrid Cloud with Far...
We have deployed a hybrid cloud storage solution that leverages compute in the public cloud along with our specialized hardware storage. We will discuss the tradeoffs of hybrid cloud storage, which workloads are best suited for this model, the pipeline we have deployed, and the challenges and best practices we have learned. Spark provides a flexible compute environment that can be used alongside todays cloud compute providers.
However in read-heavy workloads that dominate much of analysis and machine learning today, storage costs scale poorly on these same cloud storage models. Hybrid cloud offers an alternative approach to get amortized storage costs over a dedicated link while using elastic compute in the cloud. We are currently running an end to end data science stack with multiple production workloads with this setup – A Spark-based ETL for transforming the real time log data that we ingest from our devices in the field into databases, a scale-out general regular expression search over log files that provides our support engineers real time access to searching for pathologies across our customer base, and a Spark based machine learning system for time series analysis to predict various customer metrics.
Real-Time Analytics with Spark and MemSQL
"Real-Time Analytics with Spark and MemSQL" by Neil Dahlke, engineer at MemSQL for Orange County Roadshow March 17, 2017.
Virtual Flink Forward 2020: Netflix Data Mesh: Composable Data Processing - J...
Netflix processes trillions of events and petabytes of data a day in the Keystone data pipeline, which is built on top of Apache Flink. As Netflix has scaled up original productions annually enjoyed by more than 150 million global members, data integration across the streaming service and the studio has become a priority. Scalably integrating data across hundreds of different data stores in a way that enables us to holistically optimize cost, performance and operational concerns presented a significant challenge. Learn how we expanded the scope of the Keystone pipeline into the Netflix Data Mesh, our real-time, general-purpose, data transportation platform for moving data between Netflix systems. The Keystone Platform’s unique approach to declarative configuration and schema evolution, as well as our approach to unifying batch and streaming data and processing will be covered in depth.
Airline Reservations and Routing: A Graph Use Case
We've all been there before... you hear the announcement that your flight is canceled. Fellow passengers race to the gate agent to rebook on the next available flight. How do they quickly determine the best route from Berlin to San Francisco? Ultimately the flight route network is best solved as a graph problem. We will discuss our lessons learned from working with a major airline to solve this problem using JanusGraph database. JanusGraph is an open source graph database designed for massive scale. It is compatible with several pieces of the open source big data stack: Apache TinkerPop (graph computing framework), HBase, Cassandra, and Solr. We will go into depth about our approach to benchmarking graph performance and discuss the utilities we developed. We will share our comparison results for evaluating which storage backend use with JanusGraph. Whether you are productizing a new database or you are a frustrated traveler, a fast resolution is needed to satisfy everybody involved. Presented at DataWorks Summit Berlin on April 18, 2018
Large Scale Graph Analytics with JanusGraph
This document provides an overview of large scale graph analytics and JanusGraph. It discusses graph databases and their use cases. JanusGraph is presented as an open source graph database that can scale to billions of vertices and edges across multiple storage backends like HBase, Cassandra and Bigtable. It uses the TinkerPop framework and Gremlin query language. JanusGraph supports ACID transactions, external indices, and evolving schemas. Example graph queries are demonstrated using the Gremlin console.
Data streaming
- The document profiles Alberto Paro and his experience including a Master's Degree in Computer Science Engineering from Politecnico di Milano, experience as a Big Data Practise Leader at NTTDATA Italia, authoring 4 books on ElasticSearch, and expertise in technologies like Apache Spark, Playframework, Apache Kafka, and MongoDB. He is also an evangelist for the Scala and Scala.JS languages.
The document then provides an overview of data streaming architectures, popular message brokers like Apache Kafka, RabbitMQ, and Apache Pulsar, streaming frameworks including Apache Spark, Apache Flink, and Apache NiFi, and streaming libraries such as Reactive Streams.
Transitioning from Traditional DW to Apache® Spark™ in Operating Room Predict...
The prevailing issue when working with Operating Room (OR) scheduling within a hospital setting is that it is difficult to schedule and predict available OR block times. This leads to empty and unused operating rooms leading to longer waiting times for patients for their procedures. In this three-part session, Ayad Shammout and Denny will show:
1) How we tried to solve this problem using traditional DW techniques
2) How we took advantage of the DW capabilities in Apache Spark AND easily transition to Spark MLlib so we could more easily predict available OR block times resulting in better OR utilization and shorter wait times for patients.
3) Some of the key learnings we had when migrating from DW to Spark.
High Performance Data Lake with Apache Hudi and Alluxio at T3Go
Data Orchestration Summit 2020 organized by Alluxio
https://www.alluxio.io/data-orchestration-summit-2020/
High Performance Data Lake with Apache Hudi and Alluxio at T3Go
Trevor Zhang & Vino Yang (T3Go)
About Alluxio: alluxio.io
Engage with the open source community on slack: alluxio.io/slack
What's hot (20)
Cylc - the Python workflow engine for cycling systems.
Cylc - the Python workflow engine for cycling systems.
HBaseCon 2015: Warcbase - Scaling 'Out' and 'Down' HBase for Web Archiving
HBaseCon 2015: Warcbase - Scaling 'Out' and 'Down' HBase for Web Archiving
Modernizing Global Shared Data Analytics Platform and our Alluxio Journey
Modernizing Global Shared Data Analytics Platform and our Alluxio Journey
Analytics at the Real-Time Speed of Business: Spark Summit East talk by Manis...
Analytics at the Real-Time Speed of Business: Spark Summit East talk by Manis...
Accelerate Analytics and ML in the Hybrid Cloud Era
Accelerate Analytics and ML in the Hybrid Cloud Era
Lightning-Fast Analytics for Workday Transactional Data with Pavel Hardak and...
Lightning-Fast Analytics for Workday Transactional Data with Pavel Hardak and...
How Spark Enables the Internet of Things- Paula Ta-Shma
How Spark Enables the Internet of Things- Paula Ta-Shma
Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Me...
Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Me...
An End-to-End Spark-Based Machine Learning Stack in the Hybrid Cloud with Far...
An End-to-End Spark-Based Machine Learning Stack in the Hybrid Cloud with Far...
Virtual Flink Forward 2020: Netflix Data Mesh: Composable Data Processing - J...
Virtual Flink Forward 2020: Netflix Data Mesh: Composable Data Processing - J...
Airline Reservations and Routing: A Graph Use Case
Airline Reservations and Routing: A Graph Use Case
Transitioning from Traditional DW to Apache® Spark™ in Operating Room Predict...
Transitioning from Traditional DW to Apache® Spark™ in Operating Room Predict...
High Performance Data Lake with Apache Hudi and Alluxio at T3Go
High Performance Data Lake with Apache Hudi and Alluxio at T3Go
Similar to Open Source Tools for Big Data
NoSQL and Hadoop: A New Generation of Databases - Changing the Game: Monthly ...
NoSQL and Hadoop databases are emerging as alternatives to traditional relational databases for handling large amounts of unstructured data from sources like the cloud and web. Major tech companies like Oracle, IBM, Microsoft, EMC, Google and Amazon support NoSQL, with many choosing Apache Hadoop. Hadoop is an open source NoSQL database that can handle huge amounts of unstructured data at scale in cloud environments. It was designed to be fully distributed like Google's MapReduce and uses Java for integration. Relational databases remain effective for structured applications but face challenges with unstructured data, scale, and cloud deployments.
963
This document provides an overview of real time big data processing using Apache Kafka, Spark Streaming, Scala, and Elastic search. It begins with introductions to data mining, big data, and real time big data. It then discusses Apache Hadoop, Scala, Spark Streaming, Kafka, and Elastic search. The key technologies covered allow for distributed, low latency processing of streaming data at large volumes and velocities.
Present and future of unified, portable, and efficient data processing with A...
The world of big data involves an ever-changing field of players. Much as SQL stands as a lingua franca for declarative data analysis, Apache Beam aims to provide a portable standard for expressing robust, out-of-order data processing pipelines in a variety of languages across a variety of platforms. In a way, Apache Beam is a glue that can connect the big data ecosystem together; it enables users to "run any data processing pipeline anywhere."
This talk will briefly cover the capabilities of the Beam model for data processing and discuss its architecture, including the portability model. We’ll focus on the present state of the community and the current status of the Beam ecosystem. We’ll cover the state of the art in data processing and discuss where Beam is going next, including completion of the portability framework and the Streaming SQL. Finally, we’ll discuss areas of improvement and how anybody can join us on the path of creating the glue that interconnects the big data ecosystem.
Speaker
Davor Bonaci, Apache Software Foundation; Simbly, V.P. of Apache Beam; Founder/CEO at Operiant
The Big Data Puzzle, Where Does the Eclipse Piece Fit?
A discussion of our experience with integrating Eclipse and Apache projects into a big data analysis capability.
Hadoop
Hadoop is an open-source framework for storing and processing large datasets in a distributed computing environment. It allows for massive data storage, enormous processing power, and the ability to handle large numbers of concurrent tasks across clusters of commodity hardware. The framework includes Hadoop Distributed File System (HDFS) for reliable data storage and MapReduce for parallel processing of large datasets. An ecosystem of related projects like Pig, Hive, HBase, Sqoop and Flume extend the functionality of Hadoop.
Apache-Flink-What-How-Why-Who-Where-by-Slim-Baltagi
This introductory level talk is about Apache Flink: a multi-purpose Big Data analytics framework leading a movement towards the unification of batch and stream processing in the open source.
With the many technical innovations it brings along with its unique vision and philosophy, it is considered the 4 G (4th Generation) of Big Data Analytics frameworks providing the only hybrid (Real-Time Streaming + Batch) open source distributed data processing engine supporting many use cases: batch, streaming, relational queries, machine learning and graph processing.
In this talk, you will learn about:
1. What is Apache Flink stack and how it fits into the Big Data ecosystem?
2. How Apache Flink integrates with Hadoop and other open source tools for data input and output as well as deployment?
3. Why Apache Flink is an alternative to Apache Hadoop MapReduce, Apache Storm and Apache Spark.
4. Who is using Apache Flink?
5. Where to learn more about Apache Flink?
Big data apache spark + scala
This document provides an overview of Spark, including:
- Spark was developed in 2009 at UC Berkeley and open sourced in 2010, with over 200 contributors.
- Spark Core is the general execution engine that other Spark functionality is built on, providing in-memory computing and supporting various programming languages.
- Spark Streaming allows data to be ingested from sources like Kafka and Flume and integrated with Spark for advanced analytics on streaming data.
TDC2017 | POA Trilha BigData - IBM BigSQL - Engine de consulta de dados de al...
Big SQL provides a concise ANSI SQL interface for analyzing data stored in Hadoop. It offers high performance, rich SQL functionality, and integration with data science tools. Big SQL preserves the open source foundations of Hive while improving performance through an optimized query execution engine and support for SQL standards.
Hopsworks in the cloud Berlin Buzzwords 2019
This talk, given at Berlin Buzzwords 2019, describes the recent progress in making Hopsworks a cloud-native platform, with HA data-center support added for HopsFS.
OCP Summit 2017
Open Compute Project - short presentation about the whole project and review of the OCP Summit 2017.
Presented during intraXstructure 2017 conference.
Recent IT Development and Women: Big Data and The Power of Women in Goryeo
The document discusses a presentation given by Jongwook Woo on recent IT developments and women in the Goryeo dynasty. The presentation covered two main topics: (1) fundamentals of big data, data-intensive computing using Hadoop, and big data supporters and use cases; and (2) the role and power of women in medieval Korea under the Goryeo dynasty. Examples of big data uses at companies like Amazon AWS, Facebook, Twitter, Craiglist, and Huffington Post/AOL were provided.
Present and future of unified, portable and efficient data processing with Ap...
The world of big data involves an ever-changing field of players. Much as SQL stands as a lingua franca for declarative data analysis, Apache Beam aims to provide a portable standard for expressing robust, out-of-order data processing pipelines in a variety of languages across a variety of platforms. In a way, Apache Beam is a glue that can connect the big data ecosystem together; it enables users to "run any data processing pipeline anywhere."
This talk will briefly cover the capabilities of the Beam model for data processing and discuss its architecture, including the portability model. We’ll focus on the present state of the community and the current status of the Beam ecosystem. We’ll cover the state of the art in data processing and discuss where Beam is going next, including completion of the portability framework and the Streaming SQL. Finally, we’ll discuss areas of improvement and how anybody can join us on the path of creating the glue that interconnects the big data ecosystem.
Speaker
Davor Bonaci, V.P. of Apache Beam; Founder/CEO at Operiant
Database Integrated Analytics using R InitialExperiences wi
Database Integrated Analytics using R: Initial
Experiences with SQL-Server + R
Josep Ll. Berral and Nicolas Poggi
Barcelona Supercomputing Center (BSC)
Universitat Politècnica de Catalunya (BarcelonaTech)
Barcelona, Spain
Abstract—Most data scientists use nowadays functional or
semi-functional languages like SQL, Scala or R to treat data,
obtained directly from databases. Such process requires to fetch
data, process it, then store again, and such process tends to
be done outside the DB, in often complex data-flows. Recently,
database service providers have decided to integrate “R-as-a-
Service” in their DB solutions. The analytics engine is called
directly from the SQL query tree, and results are returned as
part of the same query. Here we show a first taste of such
technology by testing the portability of our ALOJA-ML analytics
framework, coded in R, to Microsoft SQL-Server 2016, one of
the SQL+R solutions released recently. In this work we discuss
some data-flow schemes for porting a local DB + analytics engine
architecture towards Big Data, focusing specially on the new
DB Integrated Analytics approach, and commenting the first
experiences in usability and performance obtained from such
new services and capabilities.
I. INTRODUCTION
Current data mining methodologies, techniques and algo-
rithms are based in heavy data browsing, slicing and process-
ing. For data scientists, also users of analytics, the capability
of defining the data to be retrieved and the operations to be
applied over this data in an easy way is essential. This is the
reason why functional languages like SQL, Scala or R are so
popular in such fields as, although these languages allow high
level programming, they free the user from programming the
infrastructure for accessing and browsing data.
The usual trend when processing data is to fetch the data
from the source or storage (file system or relational database),
bring it into a local environment (memory, distributed workers,
...), treat it, and then store back the results. In such schema
functional language applications are used to retrieve and slice
the data, while imperative language applications are used to
process the data and manage the data-flow between systems.
In most languages and frameworks, database connection pro-
tocols like ODBC or JDBC are available to enhance this data-
flow, allowing applications to directly retrieve data from DBs.
And although most SQL-based DB services allow user-written
procedures and functions, these do not include a high variety
of primitive functions or operators.
The arrival of the Big Data favored distributed frameworks
like Apache Hadoop and Apache Spark, where the data is
distributed “in the Cloud” and the data processing can also be
distributed where the data is placed, then results are joined
and aggregated. Such technologies have the advantage of
distributed computing, but when the schema for accessing data
and using it is still the same, ...
RDBMS vs Hadoop vs Spark
A brief about Relational Database Management System (RDBMS), Apache Spark, and Hadoop and a comparison between them.
Big dataanalyticsinthecloud
Siva Narayanan presents on big data analytics in the cloud using Qubole. Traditional big data projects require buying hardware, software, and hiring people to manage it, often resulting in underutilized clusters. Qubole offers a platform for interactive big data analytics in the cloud, automating cluster management and providing optimizations to improve performance. It integrates with various data sources and tools, and allows scheduling workflows and analysis. Qubole aims to make big data analytics easier, more cost-effective and faster in the cloud.
Is multi-model the future of NoSQL?
Max Neunhöffer presents on the future of NoSQL databases and argues multi-model databases will become the standard. He discusses different NoSQL data models like document stores, key-value stores, graph databases and column-oriented databases. He advocates the benefits of a polyglot persistence approach but notes the disadvantages of managing multiple databases. Max introduces ArangoDB as a multi-model database that supports documents, graphs and key-value in a single database to provide the benefits of polyglot persistence without the disadvantages. He provides examples of how ArangoDB has been used and outlines its features including queries, extensibility and horizontal scalability. Max predicts that in five years, the default approach will be to use a
Hopsworks - The Platform for Data-Intensive AI
Hopsworks is a platform for data-intensive AI projects that provides:
1. End-to-end machine learning pipelines from data ingestion to model serving.
2. A feature store for organizing machine learning data.
3. Distributed deep learning using multiple GPUs for faster training.
Big Data, Ingeniería de datos, y Data Lakes en AWS
Epic Games uses AWS services extensively to gain insights from player data and ensure Fortnite remains engaging for its over 125 million players. Telemetry data from clients is collected with Kinesis and analyzed in real-time using Spark on EMR. Game designers use these insights to inform decisions. Epic also uses S3 as a data lake, DynamoDB for real-time queries, and EMR for batch processing. This analytics platform on AWS allows constant feedback to optimize the player experience.
Extending DevOps to Big Data Applications with Kubernetes
DevOps, continuous delivery and modern architectural trends can incredibly speed up the software development process. Big Data applications cannot be an exception and need to keep the same pace.
The BDAS Open Source Community
AMPCamp 5 presentation on "The BDAS Open Source Community" by Prof. Ion Stoica of the UC Berkeley AMPLab and Databricks
Similar to Open Source Tools for Big Data (20)
NoSQL and Hadoop: A New Generation of Databases - Changing the Game: Monthly ...
NoSQL and Hadoop: A New Generation of Databases - Changing the Game: Monthly ...
Present and future of unified, portable, and efficient data processing with A...
Present and future of unified, portable, and efficient data processing with A...
The Big Data Puzzle, Where Does the Eclipse Piece Fit?
The Big Data Puzzle, Where Does the Eclipse Piece Fit?
Apache-Flink-What-How-Why-Who-Where-by-Slim-Baltagi
Apache-Flink-What-How-Why-Who-Where-by-Slim-Baltagi
TDC2017 | POA Trilha BigData - IBM BigSQL - Engine de consulta de dados de al...
TDC2017 | POA Trilha BigData - IBM BigSQL - Engine de consulta de dados de al...
Recent IT Development and Women: Big Data and The Power of Women in Goryeo
Recent IT Development and Women: Big Data and The Power of Women in Goryeo
Present and future of unified, portable and efficient data processing with Ap...
Present and future of unified, portable and efficient data processing with Ap...
Database Integrated Analytics using R InitialExperiences wi
Database Integrated Analytics using R InitialExperiences wi
Big Data, Ingeniería de datos, y Data Lakes en AWS
Big Data, Ingeniería de datos, y Data Lakes en AWS
Extending DevOps to Big Data Applications with Kubernetes
Extending DevOps to Big Data Applications with Kubernetes
Recently uploaded
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Climate Impact of Software Testing at Nordic Testing Days
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
As the digital landscape continually evolves, operating systems play a critical role in shaping user experiences and productivity. The launch of Nitrux Linux 3.5.0 marks a significant milestone, offering a robust alternative to traditional systems such as Windows 11. This article delves into the essence of Nitrux Linux 3.5.0, exploring its unique features, advantages, and how it stands as a compelling choice for both casual users and tech enthusiasts.
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Securing your Kubernetes cluster_ a step-by-step guide to success !
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
How to Get CNIC Information System with Paksim Ga.pptx
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
Join us to introduce Milvus Lite, a vector database that can run on notebooks and laptops, share the same API with Milvus, and integrate with every popular GenAI framework. This webinar is perfect for developers seeking easy-to-use, well-integrated vector databases for their GenAI apps.
Essentials of Automations: The Art of Triggers and Actions in FME
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
Artificial Intelligence for XMLDevelopment
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
UiPath Test Automation using UiPath Test Suite series, part 5
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
TrustArc Webinar - 2024 Global Privacy Survey
How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024?
In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores.
See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar will review:
- The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey
- The top challenges for privacy leaders, practitioners, and organizations in 2024
- Key themes to consider in developing and maintaining your privacy program
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
Maruthi Prithivirajan, Head of ASEAN & IN Solution Architecture, Neo4j
Get an inside look at the latest Neo4j innovations that enable relationship-driven intelligence at scale. Learn more about the newest cloud integrations and product enhancements that make Neo4j an essential choice for developers building apps with interconnected data and generative AI.
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
Presentation of the OECD Artificial Intelligence Review of Germany
Consult the full report at https://www.oecd.org/digital/oecd-artificial-intelligence-review-of-germany-609808d6-en.htm
Pushing the limits of ePRTC: 100ns holdover for 100 days
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
Recently uploaded (20)
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
Monitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR Events
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
Pushing the limits of ePRTC: 100ns holdover for 100 days
Pushing the limits of ePRTC: 100ns holdover for 100 days
Open Source Tools for Big Data
- 1. OPEN SOURCE TOOLS FOR BIG DATA Helsinki 19.9.2017 Teemu Heikkilä Emblica
- 2. EMBLICA We’re super small company of 5 people We’re into Data Engineering, DevOps and ML We’re hiring!
- 3. Let’s start with something simple first
- 4. What is “big data”?
- 6. …but, still it has a meaning.
- 7. 1TB of data, is it BIG?
- 10. We are not really in Facebook scale but is it worth to talk about big data tools?
- 11. Answer: Yes!
- 12. Question: Why?
- 13. Because what works with petabytes of data, almost certainly works with gigabytes
- 14. Helsinki City bike station usage 17M rows of JSON
- 15. You will get: Fault tolerance, reliability, scalability and working models of processing data of any amounts
- 16. … but it doesn't mean you need fancy frameworks necessarily
- 18. History of data processing with free software
- 19. NOW200320011997 2006 Google published whitepaper about solving storage problems with web indexing. Carafella and Cutting implemented the white paper as part of the Nutch project GFS HISTORY OF HADOOP Doug Cutting started to develop first version of Lucene at Yahoo! START Cutting moved the NDFS and MapReduce related codebase under new project called Hadoop HADOOP Cutting open sourced Lucene and it was moved under Apache Foundation Mike Cafarella joined with Cutting to start Apache Nutch - project to index whole internet. OPEN SOURCED
- 22. Ideas, (whitepapers) DFS MR BigTable Dynamo FOSS Implementations HDFS Hadoop MR HBase Cassandra
- 23. (Notable) formats of Big data
- 24. ACTIVITY DATA Clickstreams App usage Application specific usage Music listening Video streaming Money usage Credit cards Transactions
- 25. SENSOR DATA Locations Spatial data Sensor metrics IoT devices Industrial and consumer Time series
- 26. UNSTRUCTURED DATA Machine logs, Unstructured text, natural language Sound, Photos, Video
- 27. Use cases What are you using those fancy logos for?
- 28. CASE 1: EVENT SOURCING SQL-DATABASES Working legacy systems that used MySQL-database as a realtime data storage. No historical data saved ever. Delete means delete Update means update We could touch the legacy code to save the changes But we don’t have to
- 29. Maxwell’s daemon Reads MySQL replication binary log Produces stream of JSON-formatted changes
- 30. ?
- 31. KAFKA - DISTRIBUTED APPEND-ONLY LOG Kafka was originally developed by LinkedIn, open sourced 2011 Distributed, append-only log Great tool for delivering reliably millions of arbitrary formatted messages Scales by partitioning and adding new nodes (c) Ch.ko123 / CC BY 4.0
- 32. (c) Apache Spark + Fast writes (queue/log) + Fast reads (in-memory) - Latency - Reliable event delivery is essential KAPPA ARCHITECTURE
- 33. MATERIALIZING EVENT SOURCES Change stream Change stream Change stream Materialized ‘User’-table Materialized ‘Resource’-table Materialized ‘Usage’-table
- 35. APACHE SPARK Originally developed at the University of California, Berkeley's AMPLab General large-scale data processing framework Based on MapReduce architecture but keeps intermediate results in memory instead of saving them to slow disks like Hadoop (c) Ch.ko123 / CC BY 4.0 Supports lot’s of different data sources Programming APIs for Scala, Java or Python
- 36. EKS-STACK Elasticsearch is based on Lucene but it’s more than just search engine, it can be used to provide real time analytics even for end users, it’s usually used to store the aggregated data Kibana is great tool for the developers and for internal use to discover and analyze the data lying inside ES Spark is used to process the events, produce the needed aggregates and ingest data into Elasticsearch so it can be queried
- 38. Event Collector Processing AnalyticsEventsUser agent CASE 2: EVERY ANALYTICS PIPELINE EVER
- 40. What are we sampling?
- 43. New session: started 07:17:09, duration 0s, OPEN Existing session: started 07:17:09, duration 5s, OPEN Existing session: started 07:17:09, duration 10s, OPEN Existing session: started 07:17:09, duration 14s, paused 07:17:23, CLOSED
- 44. New session: started 07:17:09, duration 0s, OPEN Existing session: started 07:17:09, duration 5s, OPEN Existing session: started 07:17:09, duration 10s, OPEN Existing session: started 07:17:09, duration 14s, paused 07:17:23, CLOSED
- 45. You can find me at: @theikkilap teemu@emblica.fi https://emblica.fi Any questions? Thanks! Icons from Font Awesome project