Accessing Data Anywhere with Unified Namespace

•

1 like•2,337 views

Jiri Simsa presented on Alluxio (formerly Tachyon), which provides a unified namespace for data access across multiple storage systems. Alluxio allows applications to access different storage systems like HDFS and S3 through the same interface. It provides a transparent namespace that maps Alluxio objects to the underlying storage, preserving file paths. Alluxio benefits applications by future-proofing them to new storage systems and enabling new workloads by facilitating data sharing across storage through its unified namespace and API.

Alluxio (formerly Tachyon):
Accessing Data Anywhere with Unified Namespace
Jiri Simsa
June 15, 2016 @ Alluxio Meetup (hosted by Intel)

About Me
• Software Engineer @ Alluxio, Inc.
• PMC Member and Maintainer of Alluxio Open Source Project
• Ph.D. from Carnegie Mellon University (Parallel Data Lab)
• Worked at Google before joining Alluxio
• Twitter: @jsimsa, Github: jsimsa
2

Outline
• Motivation
• Unified Namespace
• Use Cases
• Demo
3

Alluxio Benefits
• Future-proofing your applications
–applications can communicate with different storage
systems, both existing and new, using the same
namespace and interface
–seamless integration between applications and new
storage systems enables faster innovation
• Enabling new workloads
–one-time effort to enable an application to access many
different types of storage systems and a storage system
to be accessed by many different types of applications
7

Outline
• Motivation
• Unified Namespace
• Use Cases
• Demo
8

Unified Namespace
an abstraction that makes it possible for
applications to access different storage
systems through the same interface
9

Transparent Naming
•Operations over persisted Alluxio objects
mapped transparently to underlying storage
•Alluxio paths are preserved in storage layer
Alluxio Storage System (HDFS, S3, …)
alluxio://host:port/
Data Users
Reports Sales Alice Bob
hdfs://host:port/
Data Users
Reports Sales Alice Bob
10

Multiple Storage Systems
•Unified namespace for multiple data sources
•Sharing of data across storage systems
•API for on-the-fly mounting / unmounting
Alluxio
Storage System A
alluxio://host:port/
Data Users
Alice Bob
hdfs://host:port/
Users
Alice Bob
Storage System B
s3://host/bucket
Reports Sales
Reports Sales
11

Outline
• Motivation
• Unified Namespace
• Use Cases
• Demo
12

Outline
• Motivation
• Unified Namespace
• Use Cases
• Demo
16

Resources
• Alluxio Project: http://www.alluxio.org
• Development: https://github.com/Alluxio/alluxio
• Meet Friends: http://www.meetup.com/Alluxio
• Alluxio, Inc.: http://www.alluxio.com
• Contact us: info@alluxio.com
17

Big Data Bellevue & Cloudy With a Chance of Data Meetup October 20, 2022 For more Alluxio events: https://alluxio.io/events/ Speakers: David Zhu (Tech Lead Manager & PMC, Alluxio) Jasmine Wang (Head of Community & DevRel, Alluxio) Distributed systems are made up of many components such as authentication, a persistence layer, stateless services, load balancers, and stateful coordination services. These coordination services are central to the operation of the system, performing tasks such as maintaining system configuration state, ensuring service availability, name resolution, and storing other system metadata. Given their central role in the system it is essential that these systems remain available, fault tolerant and consistent. By providing a highly available file system-like abstraction as well as powerful recipes such as leader election, Apache Zookeeper is often used to implement these services. This talk will go over a generic example of stateful coordination service moving from Zookeeper to Raft.

Dynamic Rule-based Real-time Market Data Alerts

Flink Forward

Flink Forward San Francisco 2022. At Bloomberg, we deal with high volumes of real-time market data. Our clients expect to be notified of any anomalies in this market data, which may indicate volatile movements in the markets, notable trades, forthcoming events, or system failures. The parameters for these alerts are always evolving and our clients can update them dynamically. In this talk, we'll cover how we utilized the open source Apache Flink and Siddhi SQL projects to build a distributed, scalable, low-latency and dynamic rule-based, real-time alerting system to solve our clients' needs. We'll also cover the lessons we learned along our journey. by Ajay Vyasapeetam & Madhuri Jain

MapReduce

Amir Payberah

Write behind logging

Pouyan Rezazadeh

KBC Proven Application of Digital Twin

KBC (A Yokogawa Company)

DataOps - Lean principles and lean practices

Lars Albertsson

Hoodie: Incremental processing on hadoop

Prasanna Rajaperumal

IIoT + Predictive Analytics: Solving for Disruption in Oil & Gas and Energy &...

DataWorks Summit

The electric grid has evolved from linear generation and delivery to a complex mix of renewables, prosumer-generated electricity, and electric vehicles (EVs). Smart meters are generating loads of data. As a result, traditional forecasting models and technologies can no longer adequately predict supply and demand. Extreme weather, an aging infrastructure, and the burgeoning worldwide population are also contributing to increased outage frequency. In oil and gas, commodity pricing pressures, resulting workforce reductions, and the need to reduce failures, automate workflows, and increase operational efficiencies are driving operators to shift analytics initiatives to advanced data-driven applications to complement physics-based tools. While sensored equipment and legacy surveillance applications are generating massive amounts of data, just 2% is understood and being leveraged. Operationalizing it along with external datasets enables a shift from time-based to condition-based maintenance, better forecasting and dramatic reductions in unplanned downtime. The session includes plenty of real-world anecdotes. For example, how an electric power holding company reduced the time it took to investigate energy theft from six months to less than one hour, producing theft leads in minutes and an expected multi-million dollar ROI. How a global offshore contract drilling services provider implemented an open source IIoT solution across its fleet of assets in less than a year, enabling remote monitoring, predictive analytics and maintenance. Key takeaways: • How are new processes for data collection, storage and democratization making it accessible and usable at scale? • Beyond time series data, what other data types are important to assess? • What advantage are open source technologies providing to enterprises deploying IIoT? • Why is collaboration important across industrial verticals to increase IIoT open source adoption? Speaker Kenneth Smith, General Manager, Energy, Hortonworks

Big Data (volume) and real-time information processing (velocity) are two important aspects of Big Data systems. At first sight, these two aspects seem to be incompatible. Are traditional software architectures still the right choice? Do we need new, revolutionary architectures to tackle the requirements of Big Data? This presentation discusses the idea of the so-called lambda architecture for Big Data, which acts on the assumption of a bisection of the data-processing: in a batch-phase a temporally bounded, large dataset is processed either through traditional ETL or MapReduce. In parallel, a real-time, online processing is constantly calculating the values of the new data coming in during the batch phase. The combination of the two results, batch and online processing is giving the constantly up-to-date view. This talk presents how such an architecture can be implemented using Oracle products such as Oracle NoSQL, Hadoop and Oracle Event Processing as well as some selected products from the Open Source Software community. While this session mostly focuses on the software architecture of BigData and FastData systems, some lessons learned in the implementation of such a system are presented as well.

Using Queryable State for Fun and Profit

Flink Forward

Flink Forward San Francisco 2022. A particular feature in our system relies on a streaming 90-minute trailing window of 1-minute samples - implemented as a lookaside cache - to speed up a particular query, allowing our customers to rapidly see an overview of their estate. Across our entire customer base, there is a substantial amount of data flowing into this cache - ~1,000,000 entries/second, with the entire cache requiring ~600GB of RAM. The current implementation is simplistic but expensive. In this talk I describe a replacement implementation as a stateful streaming Flink application leveraging Queryable State. This Flink application reduces the net cost by ~90%. In this session, the implementation is described in detail, including windowing considerations, a sliding-window state buffer that avoids the sliding window replication penalty, and a comparison of queryable state and Redis queries. The talk concludes with a frank discussion of when this distinctive approach is, and is not, appropriate. by Ron Crocker

HBase and Hadoop at Adobe

Cosmin Lehene

Api led connectivity_brazil

Anupam Gogoi

Master the Multi-Clustered Data Warehouse - Snowflake

Matillion

Snowflake is one of the most powerful, efficient data warehouses on the market today—and we joined forces with the Snowflake team to show you how it works! In this webinar: - Learn how to optimize Snowflake - Hear insider tips and tricks on how to improve performance - Get expert insights from Craig Collier, Technical Architect from Snowflake, and Kalyan Arangam, Solution Architect from Matillion - Find out how leading brands like Converse, Duo Security, and Pets at Home use Snowflake and Matillion ETL to make data-driven decisions - Discover how Matillion ETL and Snowflake work together to modernize your data world - Learn how to utilize the impressive scalability of Snowflake and Matillion

[Meetup] a successful migration from elastic search to clickhouse

Vianney FOUCAULT

Ch02 ppt - Business Communication

Zaida Santiago

Spark Summit EU talk by Mike Percy

Spark Summit

Fast Insight from Fast Data: Integrating ClickHouse and Apache Kafka

Altinity Ltd

What's the time? ...and why? (Mattias Sax, Confluent) Kafka Summit SF 2019

confluent

Data stream processing is built on the core concept of time. However, understanding time semantics and reasoning about time is not simple, especially if deterministic processing is expected. In this talk, we explain the difference between processing, ingestion, and event time and what their impact is on data stream processing. Furthermore, we explain how Kafka clusters and stream processing applications must be configured to achieve specific time semantics. Finally, we deep dive into the time semantics of the Kafka Streams DSL and KSQL operators, and explain in detail how the runtime handles time. Apache Kafka offers many ways to handle time on the storage layer, ie, the brokers, allowing users to build applications with different semantics. Time semantics in the processing layer, ie, Kafka Streams and KSQL, are even richer, more powerful, but also more complicated. Hence, it is paramount for developers, to understand different time semantics and to know how to configure Kafka to achieve them. Therefore, this talk enables developers to design applications with their desired time semantics, help them to reason about the runtime behavior with regard to time, and allow them to understand processing/query results.

A Reference Architecture for Digital Health: The Health Catalyst Data Operati...

Health Catalyst

There are essentially four strategic options to address the enterprise data platform requirements of today’s healthcare systems: (1) build your own, (2) buy from EHR vendors, (3) look to a Silicon Valley high-tech startup, and (4) partner with Health Catalyst or a handful of similar companies. In this webinar, Health Catalyst’s CTO, Dale Sanders, comments on all four approaches, hoping to help you to assess your organization’s strategy against the options and vendors in each category. It’s been exactly three years since Health Catalyst embarked on a major investment in its next-generation technology, the Data Operating System (DOS™) and its applications. This webinar is an update on the progress, less about marketing the technology, but rather offering DOS as a reference architecture that can support analytics, AI, text processing, data-first application development, and interoperability, as an all-in-one agile cost-savings architecture. In addition to the successes, Dale comments on the challenges that Health Catalyst has faced under a very ambitious DOS development plan. In its current state, DOS has made some significant improvements to overcome early mistakes, and is now a very solid enterprise data platform. In the interests of industry-wide learning, Sanders will talk transparently about those mistakes and how those learnings are being applied to the DOS platform, positioning it to evolve gracefully over the next 25 years. View the webinar to learn how the DOS reference architecture: - Helps manage the 2,000+ compulsory measures in US healthcare - Enables applications as varied as a real-time patient safety surveillance system, and an activity-based costing system in one platform - Can ingest data of any type or velocity from over 300 healthcare source systems and growing - Bundles tools, applications, and analytics that would cost 3-6x more to build on your own - Compares to EHR vendors as an option to serve as an enterprise data and analytics platform - Is a performant, sustainable, and maintainable platform for deploying AI models in the natural flow of the healthcare data pipeline - Provides curated data content and models while still allowing for the agility of a late binding design option - Functions as a reference architecture that all healthcare organizations and vendors will ultimately have to build in their pursuit of digital health

Real-time Twitter Sentiment Analysis and Image Recognition with Apache NiFi

Timothy Spann

DataOps with Project Amaterasu

DataWorks Summit/Hadoop Summit

Tame the small files problem and optimize data layout for streaming ingestion...

Flink Forward

Flink Forward San Francisco 2022. In modern data platform architectures, stream processing engines such as Apache Flink are used to ingest continuous streams of data into data lakes such as Apache Iceberg. Streaming ingestion to iceberg tables can suffer by two problems (1) small files problem that can hurt read performance (2) poor data clustering that can make file pruning less effective. To address those two problems, we propose adding a shuffling stage to the Flink Iceberg streaming writer. The shuffling stage can intelligently group data via bin packing or range partition. This can reduce the number of concurrent files that every task writes. It can also improve data clustering. In this talk, we will explain the motivations in details and dive into the design of the shuffling stage. We will also share the evaluation results that demonstrate the effectiveness of smart shuffling. by Gang Ye & Steven Wu

Streaming Analytics & CEP - Two sides of the same coin?

Till Rohrmann

Near real-time statistical modeling and anomaly detection using Flink!

Flink Forward

Flink Forward San Francisco 2022. At ThousandEyes we receive billions of events every day that allow us to monitor the internet; the most important aspect of our platform is to detect outages and anomalies that have a potential to cause serious impact to customer applications and user experience. Automatic detection of such events at lowest latency and highest accuracy is extremely important for our customers and their business. After launching several resilient and low latency data pipelines in production using Flink we decided to take it up a notch; we leveraged Flink to build statistical models in near real-time and apply them on incoming stream of events to detect anomalies! In this session we will deep dive into the design as well as discuss pitfalls and learnings while developing our real-time platform that leverages Debezium, Kafka, Flink, ElasticCache and DynamoDB to process events at scale! by Kunal Umrigar & Balint Kurnasz

A Day in the Life of a ClickHouse Query Webinar Slides

Altinity Ltd

Why do queries run out of memory? How can I make my queries even faster? How should I size ClickHouse nodes for best cost-efficiency? The key to these questions and many others is knowing what happens inside ClickHouse when a query runs. This webinar is a gentle introduction to ClickHouse internals, focusing on topics that will help your applications run faster and more efficiently. We’ll discuss the basic flow of query execution, dig into how ClickHouse handles aggregation and joins, and show you how ClickHouse distributes processing within a single CPU as well as across many nodes in the network. After attending this webinar you’ll understand how to open up the black box and see what the parts are doing.

Resilient Design Using Queue Theory

ScyllaDB

Queueing Theory is perhaps one of the most important mathematical theories in systems design and analysis, yet only few engineers master it. This webinar introduces the basics of queueing theory and explores the ramifications of queue behaviors and variance on system performance and resiliency. The session will also cover example use cases across modern application development, from NGINX and Jetty to Scylla’s own use of queuing theory. Recommended for application developers and architects designing large-scale highly reliable systems.

Getting Started with Alluxio + Spark + S3

Accessing Data Anywhere with Unified Namespace

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Accessing Data Anywhere with Unified Namespace

Similar to Accessing Data Anywhere with Unified Namespace (20)

More from Alluxio, Inc.

More from Alluxio, Inc. (20)

Recently uploaded

Recently uploaded (20)

Accessing Data Anywhere with Unified Namespace