What's New in Alluxio 2.3

•

0 likes•1,728 views

Alluxio Community Office Hour July 14, 2020 For more Alluxio events: https://www.alluxio.io/events/ Speakers: Calvin Jia, Alluxio Bin Fan, Alluxio Alluxio 2.3 was just released at the end of June 2020. Calvin and Bin will go over the new features and integrations available and share learnings from the community. Any questions about the release and on-going community feature development are welcome. In this Office Hour, we will go over: - Glue Under Database integration - Under Filesystem mount wizard - Tiered Storage Enhancements - Concurrent Metadata Sync - Delegated Journal Backups

Software

What’s New in Alluxio 2.3
7/14 Office Hour - Bin Fan, Calvin Jia
Please Join https://alluxio.io/slack for Q&A

Alluxio 2.3 Focus Areas
● Hybrid Cloud
○ Significant number of new use cases
○ Large amount of performance gain and/or cost savings
● Structured Data Services (SDS)
○ Presto continues to be a popular compute engine
○ More interest from cloud and hybrid use cases
● System Scalability & Operability
○ Large scale exposed limits in multi-tier storage management
○ Critical deployments require regular journal backups

Hybrid Cloud Features
● One Command Deployment on Amazon EMR and Dataproc
○ Try the demo hybrid use case - AWS or GCP
○ Use Terraform modules to customize your own cloud deployments
● Validate Env
○ Set of tools for connecting to remote storage and validating it works
○ New GUI for mounting remote storage which runs several diagnostic checks
○ New CLI commands for directly running the checks
■ runHdfsMountTests - checks HDFS connectivity and settings
■ runUfsIOTest - checks UFS I/O capabilities
■ runHmsTests - checks Hive Metastore connectivity and settings

Hybrid Cloud Features
● Concurrent Metadata Synchronization
○ Enable stricter metadata synchronization settings without performance loss
○ Improved synchronization algorithm to enable multiple paths in the same subtree to be
synced at once
○ Order of magnitude improvement, especially for larger namespaces that can utilize the full
parallelism
○ Improved performance of INotify based active sync, an important feature in hybrid
deployments

SDS Features
● Support for Amazon Glue UDB
○ Use Amazon Glue to store table metadata instead of HMS
○ Many users in cloud requested Glue support
○ Will continue to provide compatibility in the cloud
● Full Transformation Support for ORC format
○ Benefit from transformation capabilities in SDS with ORC
○ Seeing a number of users on ORC and Parquet
○ ORC vs Parquet is mostly based on the compute framework and available optimizations

System Scalability
● Tiered Storage V2
○ Use multi-tiered storage in read-write heavy workloads without performance loss
○ Eviction becomes O(1) operation, ensure write or cache path is not impacted
○ Evictors are now termed Annotators because it simply pre-marks the blocks to evict
○ Trade-off made is block placement is not 100% accurate when using multiple tiers
○ Background tasks are system resource aware and will operate only during quiet periods

System Operability
● Delegated Journal Backups for HA Deployments
○ Backups no longer impact master response latency
○ Instead of using primary master to backup, secondaries can run backups
○ Very useful for deployments running regular backups, can avoid performance impact
○ Window for performance hit is reduced to several seconds instead of minutes

Improving Presto Latency at Facebook
With Alluxio data caching, for one of
the Facebook internal Presto use
cases:
- query latencies improved by 33%
(P50), and 48% (P95).
- 57% improvement in IO for
remote data source scans.
https://prestodb.io/blog/2020/06/16/alluxio-datacaching
Presto + Alluxio + Facebook HDFS

Memory-speed I/O for Model Training in K8s
Alibaba Cloud K8s team is leveraging
Alluxio Posix Interface to serve deep
learning training in K8s in Kubernetes
environment.
- 40% reduction in training time &
cost
https://www.alluxio.io/blog/efficient-model-training-in-the-cloud-with-kubernetes-tensorflow-and-alluxio/
Tensorflow + Alluxio + OSS + K8s

Data Proxy for Hybrid Cloud
WeRide, a self-driving-car company,
deployed Alluxio to leverage Alluxio
as a local cache in each offices, on
top of a centralized S3 data lake
- Saving $5 per issue ticket per
engineer
https://www.alluxio.io/blog/building-a-cross-region-hybrid-cloud-storage-gateway-for-engineers-at-weride/
DS tools + Alluxio + S3 (remote)

Alluxio Global Online Meetup August 25, 2020 For more Alluxio events: https://www.alluxio.io/events/ Speakers: Abner Ferreira, Simbiose Ventures Caio Pavanelli, Simbiose Ventures Bin Fan, Alluxio Over the last few years, organizations have worked towards the separation of storage and compute for a number of benefits in the areas of cost, data duplication and data latency. Cloud resolves most of these issues but comes to the expense of needing a way to query data on remote storages. Alluxio and Presto are a powerful combination to address the compute problem, which is part of the strategy used by Simbiose Ventures to create a product called StorageQuery - A platform to query files in cloud storages with SQL. This talk will focus on: - How Alluxio fits StorageQuery's tech stack; - Advantages of using Alluxio as a cache layer and its unified filesystem; - Development of new under file system for Backblaze B2 and fine-grained code documentation; - ShannonDB remote storage mode.

Reducing large S3 API costs using Alluxio at Datasapiens

Alluxio, Inc.

Alluxio Global Online Meetup August 4, 2020 For more Alluxio events: https://www.alluxio.io/events/ Speakers: Koen Michiels, Datasapiens Juraj Pohanka, Datasapiens Bin Fan, Alluxio Datasapiens is an international data-analytics startup based in Prague. We help our clients to uncover the value of their data and open up new revenue streams for them. We provide an end-to-end service that manages the data pipeline and automates the process of generating data insights. In this talk, we will describe how we have solved an issue with large S3 API costs incurred by Presto under several usage concurrency levels by implementing Alluxio as a data orchestration layer between S3 and Presto. Also, we will show the results of an experiment with estimating the per-query S3 API costs using the TPC-DS dataset. This talk will focus on: - The Hadoop ecosystem at Datasapiens - Drastic increase of S3 API costs during performance tests with Presto - S3 API costs tests with TPC-DS - Implications to the cloud data lake architecture

Data Orchestration for the Hybrid Cloud Era

Alluxio, Inc.

Alluxio Community Office Hour October 20, 2020 For more Alluxio events: https://www.alluxio.io/events/ Speaker(s): Alex Ma, Alluxio Peter Behrakis, Alluxio Many companies we talk to have on premises data lakes and use the cloud(s) to burst compute. Many are now establishing new object data lakes as well. As a result, running analytics such as Hive, Spark, Presto and machine learning are experiencing sluggish response times with data and compute in multiple locations. We also know there is an immense and growing data management burden to support these workflows. In this talk, we will walk through what Alluxio’s Data Orchestration for the hybrid cloud era is and how it solves the performance and data management challenges we see. In this tech talk, we'll go over: - What is Alluxio Data Orchestration? - How does it work? - Alluxio customer results

Accelerate Analytics and ML in the Hybrid Cloud Era

Alluxio, Inc.

Alluxio Community Office Hour February 23, 2021 For more Alluxio events: https://www.alluxio.io/events/ Speaker(s): Alex Ma, Alluxio Peter Behrakis, Alluxio Many companies we talk to have on premises data lakes and use the cloud(s) to burst compute. Many are now establishing new object data lakes as well. As a result, running analytics such as Hive, Spark, Presto and machine learning are experiencing sluggish response times with data and compute in multiple locations. We also know there is an immense and growing data management burden to support these workflows. In this talk, we will walk through what Alluxio’s Data Orchestration for the hybrid cloud era is and how it solves the performance and data management challenges we see. In this tech talk, we'll go over: - What is Alluxio Data Orchestration? - How does it work? - Alluxio customer results

Introducing the Hub for Data Orchestration

Alluxio, Inc.

Accelerate Analytics and ML in the Hybrid Cloud Era

Alluxio, Inc.

Alluxio Webinar September 22, 2020 For more Alluxio events: https://www.alluxio.io/events/ Speakers: Alex Ma, Alluxio Peter Behrakis, Alluxio Many companies we talk to have on premises data lakes and use the cloud(s) to burst compute. Many are now establishing new object data lakes as well. As a result, running analytics such as Hive, Spark, Presto and machine learning are experiencing sluggish response times with data and compute in multiple locations. We also know there is an immense and growing data management burden to support these workflows. In this talk, we will walk through what Alluxio’s Data Orchestration for the hybrid cloud era is and how it solves the performance and data management challenges we see. In this tech talk, we'll go over: - What is Alluxio Data Orchestration? - How does it work? - Alluxio customer results

Ultra Fast Deep Learning in Hybrid Cloud Using Intel Analytics Zoo & Alluxio

Alluxio, Inc.

Alluxio Global Online Meetup Apr 23, 2020 For more Alluxio events: https://www.alluxio.io/events/ Speakers: Jiao (Jennie) Wang, Intel Tsai Louie, Intel Bin Fan, Alluxio Today, many people run deep learning applications with training data from separate storage such as object storage or remote data centers. This presentation will demo the Intel Analytics Zoo + Alluxio stack, an architecture that enables high performance while keeping cost and resource efficiency balanced without network being I/O bottlenecked. Intel Analytics Zoo is a unified data analytics and AI platform open-sourced by Intel. It seamlessly unites TensorFlow, Keras, PyTorch, Spark, Flink, and Ray programs into an integrated pipeline, which can transparently scale from a laptop to large clusters to process production big data. Alluxio, as an open-source data orchestration layer, accelerates data loading and processing in Analytics Zoo deep learning applications. This talk, we will go over: - What is Analytics Zoo and how it works - How to run Analytics Zoo with Alluxio in deep learning applications - Initial performance benchmark results using the Analytics Zoo + Alluxio stack

Alluxio Architecture and Performance

Alluxio, Inc.

Gene Pang presented on Alluxio architecture and scaling performance for large deployments. He discussed Alluxio's high-level components including the master, workers, jobs masters and workers, and proxies. He then covered techniques for improving Alluxio scaling including parallelizing metadata sync and catalog sync, handling slow external storage reads asynchronously, rearranging blocks asynchronously, and adding timeouts for disk operations to avoid unexpected hangs. The goal is to make Alluxio faster, more predictable, and support higher concurrency even with interactions with slow external storage systems.

This document discusses modernizing a data platform for analytics and AI across single, hybrid, or multi-cloud environments using Alluxio. It describes Alluxio's key features like data locality, metadata locality, asynchronous data operations, and policy-driven data management that enable consistent performance, portability, and cost savings. Examples are provided of how Alluxio can be used to transition from on-premises HDFS to object storage to hybrid cloud and multi-cloud configurations.

Building a high-performance data lake analytics engine at Alibaba Cloud with ...

Alluxio, Inc.

This document discusses optimizations made to Alibaba Cloud's Data Lake Analytics (DLA) engine, which uses Presto, to improve performance when querying data stored in Object Storage Service (OSS). The optimizations included decreasing OSS API request counts, implementing an Alluxio data cache using local disks on Presto workers, and improving disk throughput by utilizing multiple ultra disks. These changes increased cache hit ratios and query performance for workloads involving large scans of data stored in OSS. Future plans include supporting an Alluxio cluster shared by multiple users and additional caching techniques.

Alluxio Use Cases and Future Directions

Alluxio, Inc.

Iceberg + Alluxio for Fast Data Analytics

Alluxio, Inc.

Enterprise Distributed Query Service powered by Presto & Alluxio across cloud...

Alluxio, Inc.

Accelerating analytics in the cloud with the Starburst Presto + Alluxio stack

Alluxio, Inc.

Alluxio Tech Talk January 21, 2020 Speakers: Matt Fuller, Starburst Dipti Borkar, Alluxio With the advent of the public clouds and data increasingly siloed across many locations -- on premises and in the public cloud -- enterprises are looking for more flexibility and higher performance approaches to analyze their structured data. Join us for this tech talk where we’ll introduce the Starburst Presto, Alluxio, and cloud object store stack for building a highly-concurrent and low-latency analytics platform. This stack provides a strong solution to run fast SQL across multiple storage systems including HDFS, S3, and others in public cloud, hybrid cloud, and multi-cloud environments. You’ll learn more about: - The architecture of Presto, an open source distributed SQL engine - How the Presto + Alluxio stack queries data from cloud object storage like S3 for faster and more cost-effective analytics - Achieving data locality and cross-job caching with Alluxio regardless of where data is persisted

Enabling big data & AI workloads on the object store at DBS

Alluxio, Inc.

DBS Bank is headquartered in Singapore and has evolved its data platforms over three generations from proprietary systems to a hybrid cloud-native platform using open source technologies. It is using Alluxio to unify access to data stored in its on-premises object store and HDFS as well as enable analytics workloads to burst into AWS. Alluxio provides data caching to speed up analytics jobs, intelligent data tiering for efficiency, and policy-driven data migration to the cloud over time. DBS is exploring further using cloud services for speech processing and moving more workloads to the cloud while keeping data on-premises for compliance.

Alluxio - Scalable Filesystem Metadata Services

Alluxio, Inc.

This talk was presented by Alluxio's top contributor and PMC Maintainer Calvin Jia at the Alluxio bay area Meetup. This talk shares our design, implementation and optimization of Alluxio metadata service to address the scalability challenges, focusing on how to apply and combine techniques including tiered metadata storage (based on off-heap KV store RocksDB), fine-grained file system inode tree locking scheme, embedded state-replicate machine (based on RAFT), exploration and performance tuning in the correct RPC frameworks (thrift vs gRPC) and etc.

Accelerate Cloud Training with Alluxio

Alluxio, Inc.

ApacheCon 2021 For more Alluxio events: https://www.alluxio.io/events/ Speakers: Lu Qiu Bin Fan Alluxio’s capabilities as a Data Orchestration framework have encouraged users to onboard more of their data-driven applications to an Alluxio powered data access layer. Driven by strong interests from our open-source community, the core team of Alluxio started to re-design an efficient and transparent way for users to leverage data orchestration through the POSIX interface. This effort has a lot of progress with the collaboration with engineers from Microsoft, Alibaba and Tencent. Particularly, we have introduced a new JNI-based FUSE implementation to support POSIX data access, created a more efficient way to integrate Alluxio with FUSE service, as well as many improvements in relevant data operations like more efficient distributedLoad, optimizations on listing or calculating directories with a massive amount of files, which are common in model training. We will also share our engineering lessons and roadmap in future releases to support Machine Learning applications.

Accelerating Data Computation on Ceph Objects

Alluxio, Inc.

Alluxio Global Online Meetup November 10, 2020 For more Alluxio events: https://www.alluxio.io/events/ Speaker(s): Leonardo Militano, ZHAW In most of the distributed storage systems, the data nodes are decoupled from compute nodes. This is motivated by an improved cost efficiency, storage utilization and a mutually independent scalability of computation and storage. While this consideration is indisputable, several situations exist where moving computation close to the data brings important benefits. Whenever the stored data is to be processed for analytics purposes, all the data needs to be repeatedly moved from the storage to the compute cluster, which leads to reduced performance. In this talk, we will present how using Alluxio computation and storage ecosystems can better interact benefiting the "bringing the data close to the code" approach. Moving away from the complete disaggregation of computation and storage, data locality can enhance the computation performance. During this talk, we will present our observations and testing results that will show important enhancements in accelerating Spark Data Analytics on Ceph Objects Storage using Alluxio.

Architecting a Heterogeneous Data Platform Across Clusters, Regions, and Clouds

Alluxio, Inc.

Alluxio Product School Webinar January 27, 2022 For more Alluxio events: https://www.alluxio.io/events/ Speaker: Adit Madan Data platform teams are increasingly challenged with accessing multiple data stores that are separated from compute engines, such as Spark, Presto, TensorFlow or PyTorch. Whether your data is distributed across multiple datacenters and/or clouds, a successful heterogeneous data platform requires efficient data access. Alluxio enables you to embrace the separation of storage from compute and use Alluxio data orchestration to simplify adoption of the data lake and data mesh paradigms for analytics and AI/ML workloads. Join Alluxio’s Sr. Product Mgr., Adit Madan, to learn: - Key challenges with architecting a successful heterogeneous data platform - How data orchestration can overcome data access challenges in a distributed, heterogeneous environment - How to identify ways to use Alluxio to meet the needs of your own data environment and workload requirements

Best Practices for Using Alluxio with Spark

Alluxio, Inc.

Gene Pang presented on best practices for using Alluxio with Spark. Alluxio is a memory-centric distributed storage system that can improve Spark performance by enabling data to be accessed at memory speed. Using Alluxio between Spark and storage systems allows data to be shared between Spark's storage and execution engines at memory speed without requiring multiple copies. Alluxio also provides data resilience during crashes since data is not lost from memory. Experiments showed Alluxio providing a 6-8x speedup over reading cached Parquet dataframes from S3.

Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & Alluxio

Alluxio, Inc.

Hybrid data lake on google cloud with alluxio and dataproc

Alluxio, Inc.

Orchestrate a Data Symphony

Alluxio, Inc.

Securely Enhancing Data Access in Hybrid Cloud with Alluxio

Alluxio, Inc.

Accelerate Analytics and ML in the Hybrid Cloud Era

Alluxio, Inc.

Alluxio Webinar April 6, 2021 For more Alluxio events: https://www.alluxio.io/events/ Speakers: Alex Ma, Alluxio Peter Behrakis, Alluxio Many companies we talk to have on premises data lakes and use the cloud(s) to burst compute. Many are now establishing new object data lakes as well. As a result, running analytics such as Hive, Spark, Presto and machine learning are experiencing sluggish response times with data and compute in multiple locations. We also know there is an immense and growing data management burden to support these workflows. In this talk, we will walk through what Alluxio’s Data Orchestration for the hybrid cloud era is and how it solves the performance and data management challenges we see. In this tech talk, we'll go over: - What is Alluxio Data Orchestration? - How does it work? - Alluxio customer results

Data Orchestration for AI, Big Data, and Cloud

Alluxio, Inc.

This document discusses the need for data orchestration across fragmented data environments. As more data is generated and stored across different storage systems and clouds, data silos have become inevitable. A data orchestration solution like Alluxio can abstract and orchestrate data across these silos, making data locally accessible to compute frameworks regardless of where the data is stored. Alluxio provides a unified view of data locations, enables data access from any application, and allows data to be burst elastically across clouds for compute. Many large companies are adopting data orchestration to improve data access, reduce costs, and gain more flexibility in their cloud environments.

How to Develop and Operate Cloud Native Data Platforms and Applications

Alluxio, Inc.

Presto on Alluxio Hands-On Lab

Alluxio, Inc.

Red Hat Enterprise Linux: Open, hyperconverged infrastructure

Red_Hat_Storage

The next generation of IT will be built around flexible infrastructures and operational efficiencies, lowering costs and increasing overall business value in the organization. A hyperconverged infrastructure that's built on Red Hat supported technologies--including Linux, Gluster storage, and oVirt virtualization manager--will run on commodity x86 servers using the performance of local storage, to deliver a cost-effective, modular, highly scalable, and secure hyperconverged solution.

Monitoring with prometheus at scale

Adam Hamsik

What's hot

Modernizing Your Data Platform for Analytics and AI in the Hybrid Cloud Era

Alluxio, Inc.

Building a high-performance data lake analytics engine at Alibaba Cloud with ...

Alluxio, Inc.

Alluxio Use Cases and Future Directions

Alluxio, Inc.

Iceberg + Alluxio for Fast Data Analytics

Alluxio, Inc.

Enterprise Distributed Query Service powered by Presto & Alluxio across cloud...

Alluxio, Inc.

Accelerating analytics in the cloud with the Starburst Presto + Alluxio stack

Alluxio, Inc.

Enabling big data & AI workloads on the object store at DBS

Alluxio, Inc.

Alluxio - Scalable Filesystem Metadata Services

Alluxio, Inc.

Accelerate Cloud Training with Alluxio

Alluxio, Inc.

Accelerating Data Computation on Ceph Objects

Alluxio, Inc.

Architecting a Heterogeneous Data Platform Across Clusters, Regions, and Clouds

Alluxio, Inc.

Best Practices for Using Alluxio with Spark

Alluxio, Inc.

Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & Alluxio

Alluxio, Inc.

Hybrid data lake on google cloud with alluxio and dataproc

Alluxio, Inc.

Orchestrate a Data Symphony

Alluxio, Inc.

Securely Enhancing Data Access in Hybrid Cloud with Alluxio

Alluxio, Inc.

Accelerate Analytics and ML in the Hybrid Cloud Era

Alluxio, Inc.

Data Orchestration for AI, Big Data, and Cloud

Alluxio, Inc.

How to Develop and Operate Cloud Native Data Platforms and Applications

Alluxio, Inc.

Presto on Alluxio Hands-On Lab

Alluxio, Inc.

What's hot (20)

Modernizing Your Data Platform for Analytics and AI in the Hybrid Cloud Era

Building a high-performance data lake analytics engine at Alibaba Cloud with ...

Alluxio Use Cases and Future Directions

Iceberg + Alluxio for Fast Data Analytics

Enterprise Distributed Query Service powered by Presto & Alluxio across cloud...

Accelerating analytics in the cloud with the Starburst Presto + Alluxio stack

Enabling big data & AI workloads on the object store at DBS

Alluxio - Scalable Filesystem Metadata Services

Accelerate Cloud Training with Alluxio

Accelerating Data Computation on Ceph Objects

Architecting a Heterogeneous Data Platform Across Clusters, Regions, and Clouds

Best Practices for Using Alluxio with Spark

Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & Alluxio

Hybrid data lake on google cloud with alluxio and dataproc

Orchestrate a Data Symphony

Securely Enhancing Data Access in Hybrid Cloud with Alluxio

Accelerate Analytics and ML in the Hybrid Cloud Era

Data Orchestration for AI, Big Data, and Cloud

How to Develop and Operate Cloud Native Data Platforms and Applications

Presto on Alluxio Hands-On Lab

Similar to What's New in Alluxio 2.3

Red Hat Enterprise Linux: Open, hyperconverged infrastructure

Red_Hat_Storage

Monitoring with prometheus at scale

Adam Hamsik

Monitoring with prometheus at scale

Juraj Hantak

Prometheus is an open-source monitoring system that can scale from a single server to large clusters. While a single Prometheus instance is sufficient for many deployments, tools like Trickster, sharding, and Thanos/Cortex can help scale Prometheus further. Trickster caches query results to reduce load, sharding splits data across servers, and Thanos provides high availability and long term storage across multiple clusters.

Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML

Alluxio, Inc.

Big Data Bellevue Meetup March 21, 2024 For more Alluxio events: https://alluxio.io/events/ Speakers: Bin Fan (VP of Open Source, Alluxio) In this presentation, Bin Fan (VP of Open Source @ Alluxio) will address a critical challenge of optimizing data loading for distributed Python applications within AI/ML workloads in the cloud, focusing on popular frameworks like Ray and Hugging Face. Integration of Alluxio’s distributed caching for Python applications is accomplished using the fsspec interface, thus greatly improving data access speeds. This is particularly useful in machine learning workflows, where repeated data reloading across slow, unstable or congested networks can severely affect GPU efficiency and escalate operational costs. Attendees can look forward to practical, hands-on demonstrations showcasing the tangible benefits of Alluxio’s caching mechanism across various real-world scenarios. These demos will highlight the enhancements in data efficiency and overall performance of data-intensive Python applications. This presentation is tailored for developers and data scientists eager to optimize their AI/ML workloads. Discover strategies to accelerate your data processing tasks, making them not only faster but also more cost-efficient.

Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed Data

Alluxio, Inc.

Alluxio Monthly Webinar Apr. 23, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - ChanChan Mao (Developer Advocate, Alluxio) - Shawn Sun (Tech Lead of Cloud Native, Alluxio) Cloud-native model training jobs require fast data access to achieve shorter training cycles. Accessing data can be challenging when your datasets are distributed across different regions and clouds. Additionally, as GPUs remain scarce and expensive resources, it becomes more common to set up remote training clusters from where data resides. This multi-region/cloud scenario introduces the challenges of losing data locality, resulting in operational overhead, latency and expensive cloud costs. In the third webinar of the multi-cloud webinar series, Chanchan and Shawn dive deep into: - The data locality challenges in the multi-region/cloud ML pipeline - Using a cloud-native distributed caching system to overcome these challenges - The architecture and integration of PyTorch/Ray+Alluxio+S3 using POSIX or RESTful APIs - Live demo with ResNet and BERT benchmark results showing performance gains and cost savings analysis

Cloud Architecture best practices

Omid Vahdaty

The document provides best practices for cloud architecture. It discusses when to cloudify an application based on factors like unpredictable capacity needs, elasticity requirements, and agility in development. It also discusses when not to cloudify, such as if network latency is a concern or vendor lock-in is important. The document then discusses database normalization practices and design considerations for scaling out applications in a stateless manner using services. It emphasizes automation, loose coupling between services, and service discovery mechanisms.

Scaling Monitoring At Databricks From Prometheus to M3

LibbySchulze

M3 has been successfully deployed at Databricks to replace their Prometheus monitoring system. Some key lessons learned include monitoring important M3 metrics like memory and disk usage, having automated deployment processes, and planning for capacity needs and spikes in metrics. Updates to M3 have gone smoothly, and future plans include using new M3 features like downsampling and separate namespaces.

Red Hat multi-cluster management & what's new in OpenShift

Kangaroot

Slides: Accelerating Queries on Cloud Data Lakes

DATAVERSITY

Using “zero-copy” hybrid bursting on remote data to solve data lake analytics capacity and performance problems. Data scientists want answers on demand. But in today’s enterprise architectures, the reality is that most data remains on-prem, despite the promise of cloud-based analytics. Moving all that data to the cloud has typically not been possible for many reasons including cost, latency, and technical difficulty. So, what if there was a technology that would connect these on-prem environments to any major cloud platform, enabling high-powered computing without the need to move massive amounts of data? Join us for this webinar where Alex Ma of Alluxio, an open-source data orchestration platform, will discuss how a data orchestration approach offers a solution for connecting traditional on-prem data centers and cloud data lakes with other clouds and data centers. With Alluxio’s “zero-copy” burst solution, companies can bridge remote data centers and data lakes with computing frameworks in other locations, enabling them to offload, compute, and leverage the flexibility, scalability, and power of the cloud for their remote data.

Webinar: OpenEBS - Still Free and now FASTEST Kubernetes storage

MayaData Inc

Webinar Session - https://youtu.be/_5MfGMf8PG4 In this webinar, we share how the Container Attached Storage pattern makes performance tuning more tractable, by giving each workload its own storage system, thereby decreasing the variables needed to understand and tune performance. We then introduce MayaStor, a breakthrough in the use of containers and Kubernetes as a data plane. MayaStor is the first containerized data engine available that delivers near the theoretical maximum performance of underlying systems. MayaStor performance scales with the underlying hardware and has been shown, for example, to deliver in excess of 10 million IOPS in a particular environment.

OSMC 2023 | What’s new with Grafana Labs’s Open Source Observability stack by...

NETWAYS

Open source is at the heart of what we do at Grafana Labs and there is so much happening! The intent of this talk to update everyone on the latest development when it comes to Grafana, Pyroscope, Faro, Loki, Mimir, Tempo and more. Everyone has had at least heard about Grafana but maybe some of the other projects mentioned above are new to you? Welcome to this talk 😉 Beside the update what is new we will also quickly introduce them during this talk.

High availability and disaster recovery in IBM PureApplication System

Scott Moonen

This document discusses high availability and disaster recovery strategies for IBM PureApplication System. It begins with definitions of key terms like HA, DR, RTO, and RPO. It then outlines the various tools in PureApplication System that can be used to achieve HA and DR, such as compute node availability, block storage, storage replication, and external storage. The document provides examples of how to compose these tools to meet different HA and DR scenarios, like handling compute node failures, database updates, and site failures. It concludes with some caveats around networking considerations and middleware-specific factors.

Technical Report NetApp Clustered Data ONTAP 8.2: An Introduction

NetApp

Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud

Alluxio, Inc.

Alluxio Monthly Webinar May. 14, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - ChanChan Mao (Developer Advocate, Alluxio) - Bin Fan (VP of Technology, Alluxio) Running AI/ML workloads in different clouds present unique challenges. The key to a manageable multi-cloud architecture is the ability to seamlessly access data across environments with high performance and low cost. This webinar is designed for data platform engineers, data infra engineers, data engineers, and ML engineers who work with multiple data sources in hybrid or multi-cloud environments. Chanchan and Bin will guide the audience through using Alluxio to greatly simplify data access and make model training and serving more efficient in these environments. You will learn: - How to access data in multi-region, hybrid, and multi-cloud like accessing a local file system - How to run PyTorch to read datasets and write checkpoints to remote storage with Alluxio as the distributed data access layer - Real-world examples and insights from tech giants like Uber, AliPay and more

Accelerating Cloud Training With Alluxio

Alluxio, Inc.

Alluxio Day XV September 15, 2022 For more on Alluxio Day: https://www.alluxio.io/alluxio-day/ For more Alluxio events: https://alluxio.io/events/ Speaker: Lu Qiu (Machine Learning Engineer and PMC Maintainer, Alluxio) This talk introduces the three game level progressions to use Alluxio to speed up your cloud training with production use cases from Microsoft, Alibaba, and BossZhipin. - Level 1: Speed up data ingestion from cloud storage - Level 2: Speed up data preprocessing and training workloads - Level 3: Speed up full training workloads with a unified data orchestration layer

Prometheus - Intro, CNCF, TSDB,PromQL,Grafana

Sridhar Kumar N

Madrid meetup #7 deployment models

Mario Alberto Martinez Lopez

This document outlines a presentation on deployment models given by José Cebrián, Marc Escalona, and Gonzalo Ozdy on July 14, 2021 for the Madrid MuleSoft Meetup #7. The presentation included an overview of deployment models, a comparative of CloudHub, on-premises, and Runtime Fabric options, use cases demonstrating the decision process, and the roadmap for Runtime Fabric. It was split into two sessions, the first on introduction to deployment models and the second a demo of Runtime Fabric installation.

HP: HP 3PAR - Storage zrodený pre virtualizované prostredie

ASBIS SK

This document discusses HP 3PAR Utility Storage. It notes that 3PAR offers efficiencies of consolidation while maximizing flexibility, can be integrated with best-of-breed infrastructure, and offers SLA resiliency beyond just storage HA. It also notes that 3PAR is optimized for specific requirements of virtualized computing like increasing consolidation and simplifying administration. The document provides details on how 3PAR uses thin provisioning, thin conversion, and thin reclamation technologies to reduce storage costs and capacity requirements.

Red Hat Gluster Storage - Direction, Roadmap and Use-Cases

Red_Hat_Storage

Red Hat Gluster Storage is open, software-defined storage that helps you manage big, unstructured, and semistructured data. This product is based on the open source project GlusterFS, a distributed scale-out file system technology, and focuses on file sharing, analytics, and hyper-converged use cases. In this session, you will: See real-life case studies about Red Hat Gluster Storage’s usage in production environments, including ideal workloads. Learn about the Red Hat Gluster Storage roadmap, including innovations from the GlusterFS community pipeline. Gain insights into how the product will be integrated with Red Hat Enterprise Virtualization (including hyperconvergence), Red Hat Satellite, and Red Hat Enterprise Linux OpenStack Platform.

2021.02 new in Ceph Pacific Dashboard

Ceph Community

Ceph Pacific is a major release of the Ceph distributed storage system scheduled for March 2021. It focuses on five key themes: usability, performance, ecosystem integration, multi-site capabilities, and quality. New features in Pacific include automated upgrades, improved dashboard functionality, snapshot-based CephFS mirroring, per-bucket replication in RGW, and expanded telemetry collection. Looking ahead, the Quincy release will focus on continued improvements in these areas such as resource-aware scheduling in cephadm and multi-site monitoring capabilities.

Similar to What's New in Alluxio 2.3 (20)

Red Hat Enterprise Linux: Open, hyperconverged infrastructure

Monitoring with prometheus at scale

Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML

Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed Data

Cloud Architecture best practices

Scaling Monitoring At Databricks From Prometheus to M3

Red Hat multi-cluster management & what's new in OpenShift

Slides: Accelerating Queries on Cloud Data Lakes

Webinar: OpenEBS - Still Free and now FASTEST Kubernetes storage

OSMC 2023 | What’s new with Grafana Labs’s Open Source Observability stack by...

High availability and disaster recovery in IBM PureApplication System

Technical Report NetApp Clustered Data ONTAP 8.2: An Introduction

Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud

Accelerating Cloud Training With Alluxio

Prometheus - Intro, CNCF, TSDB,PromQL,Grafana

Madrid meetup #7 deployment models

HP: HP 3PAR - Storage zrodený pre virtualizované prostredie

Red Hat Gluster Storage - Direction, Roadmap and Use-Cases

2021.02 new in Ceph Pacific Dashboard

More from Alluxio, Inc.

AI/ML Infra Meetup | ML explainability in Michelangelo

Alluxio, Inc.

AI/ML Infra Meetup May. 23, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Eric Wang (Software Engineer, @Uber) Uber has numerous deep learning models, most of which are highly complex with many layers and a vast number of features. Understanding how these models work is challenging and demands significant resources to experiment with various training algorithms and feature sets. With ML explainability, the ML team aims to bring transparency to these models, helping to clarify their predictions and behavior. This transparency also assists the operations and legal teams in explaining the reasons behind specific prediction outcomes. In this talk, Eric Wang will discuss the methods Uber used for explaining deep learning models and how we integrated these methods into the Uber AI Michelangelo ecosystem to support offline explaining.

AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG

Alluxio, Inc.

AI/ML Infra Meetup May. 23, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Junchen Jiang (Assistant Professor of Computer Science, @University of Chicago) Prefill in LLM inference is known to be resource-intensive, especially for long LLM inputs. While better scheduling can mitigate prefill’s impact, it would be fundamentally better to avoid (most of) prefill. This talk introduces our preliminary effort towards drastically minimizing prefill delay for LLM inputs that naturally reuse text chunks, such as in retrieval-augmented generation. While keeping the KV cache of all text chunks in memory is difficult, we show that it is possible to store them on cheaper yet slower storage. By improving the loading process of the reused KV caches, we can still significantly speed up prefill delay while maintaining the same generation quality.

AI/ML Infra Meetup | Perspective on Deep Learning Framework

Alluxio, Inc.

AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...

Alluxio, Inc.

AI/ML Infra Meetup May. 23, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Lu Qiu (Data & AI Platform Tech Lead, @Alluxio) - Siyuan Sheng (Senior Software Engineer, @Alluxio) Speed and efficiency are two requirements for the underlying infrastructure for machine learning model development. Data access can bottleneck end-to-end machine learning pipelines as training data volume grows and when large model files are more commonly used for serving. For instance, data loading can constitute nearly 80% of the total model training time, resulting in less than 30% GPU utilization. Also, loading large model files for deployment to production can be slow because of slow network or storage read operations. These challenges are prevalent when using popular frameworks like PyTorch, Ray, or HuggingFace, paired with cloud object storage solutions like S3 or GCS, or downloading models from the HuggingFace model hub. In this presentation, Lu and Siyuan will offer comprehensive insights into improving speed and GPU utilization for model training and serving. You will learn: - The data loading challenges hindering GPU utilization - The reference architecture for running PyTorch and Ray jobs while reading data from S3, with benchmark results of training ResNet50 and BERT - Real-world examples of boosting model performance and GPU utilization through optimized data access

Optimizing Data Access for Analytics And AI with Alluxio

Alluxio, Inc.

Speed Up Presto at Uber with Alluxio Caching

Alluxio, Inc.

Correctly Loading Incremental Data at Scale

Alluxio, Inc.

Alluxio x Tobiko - ETL Happy Hour April 16, 2024 For more Alluxio events: https://alluxio.io/events/ Speaker: Toby Mao (CTO @ Tobiko Data) Writing efficient and correct incremental pipelines is challenging. Data practitioners who take on this challenge are viewed as performing an "advanced" function, which discourages broader teams from adopting incremental loads. In this lightning talk, CTO of Tobiko Data, Toby Mao, will demystify incremental loading data at scale.

Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...

Alluxio, Inc.

Alluxio Monthly Webinar Feb. 27, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Tarik Bennett (Senior Solutions Engineer, Alluxio) As GenAI and AI continue to transform businesses, scaling these workloads requires optimized underlying infrastructure. A multi-cloud architecture allows organizations to leverage different cloud services to meet diverse workload demands while maximizing efficiency, reducing costs, and avoiding vendor lock-in. However, achieving a multi-cloud vision can be challenging. In this webinar, Tarik will share how an agonistic data layer, like Alluxio, allows you to embrace the separation of storage from compute and simplify the adoption of multi-cloud for AI. - Learn why leveraging multiple cloud providers is critical for balancing performance, scalability, and cost of your AI platform - Discover how an agnostic data layer like Alluxio provides seamless data access in multi-cloud that bridges storage and compute without data replication - Gain insights into real-world examples and best practices for deploying AI across on-prem, hybrid, and multi-cloud environments

Alluxio Monthly Webinar | Five Disruptive Trends that Every Data & AI Leader...

Alluxio, Inc.

Alluxio Monthly Webinar Jan. 30, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Kevin Petrie (VP of Research, Eckerson Group) - Omid Razavi (SVP of Customer Success, Alluxio) 2024 is gearing up to be an impactful year for AI and analytics. Join us on January 30, as Kevin Petrie (VP of Research at Eckerson Group) and Omid Razavi (SVP of Customer Success at Alluxio) share key trends that data and AI leaders should know. This event will efficiently guide you with market data and expert insights to drive successful business outcomes. - Assess current and future trends in data and AI with industry experts - Discover valuable insights and practical recommendations - Learn best practices to make your enterprise data more accessible for both analytics and AI applications

Data Infra Meetup | FIFO Queues are All You Need for Cache Eviction

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Juncheng Yang(Ph.D Candidate, @CMU) As a cache eviction algorithm, FIFO has a lot of attractive properties, such as simplicity, speed, scalability, and flash-friendliness. The most prominent criticism of FIFO is its low efficiency (high miss ratio). In this talk, I will describe a simple, scalable FIFO-based algorithm with three static queues (S3-FIFO). Evaluated on 6594 cache traces from 14 datasets, we show that S3- FIFO has lower miss ratios than state-of-the-art algorithms across traces. Moreover, S3-FIFO’s efficiency is robust — it has the lowest mean miss ratio on 10 of the 14 datasets. FIFO queues enable S3-FIFO to achieve good scalability with 6× higher throughput compared to optimized LRU at 16 threads. Our insight is that most objects in skewed workloads will only be accessed once in a short window, so it is critical to evict them early (also called quick demotion). The key of S3-FIFO is a small FIFO queue that filters out most objects from entering the main cache, which provides a guaranteed demotion speed and high demotion precision.

Data Infra Meetup | Accelerate Your Trino/Presto Queries - Gain the Alluxio Edge

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Jingwen Ouyang (Product Manager, @Alluxio) In this session, Jingwen presents an overview of using Alluxio Edge caching to accelerate Trino or Presto queries. She offers practical best practices for using distributed caching with compute engines. In addition, this session also features insights from real-world examples.

Data Infra Meetup | Accelerate Distributed PyTorch/Ray Workloads in the Cloud

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Siyuan Sheng (Senior Software Engineer, @Alluxio) - Chunxu Tang (Research Scientist, @Alluxio) In this session, cloud optimization specialists Chunxu and Siyuan break down the challenges and present a fresh architecture designed to optimize I/O across the data pipeline, ensuring GPUs function at peak performance. The integrated solution of PyTorch/Ray + Alluxio + S3 offers a promising way forward, and the speakers delve deep into its practical applications. Attendees will not only gain theoretical insights but will also be treated to hands-on instructions and demonstrations of deploying this cutting-edge architecture in Kubernetes, specifically tailored for Tensorflow/PyTorch/Ray workloads in the public cloud.

Data Infra Meetup | ByteDance's Native Parquet Reader

Alluxio, Inc.

Data Infra Meetup | Uber's Data Storage Evolution

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Jing Zhao (Principal Engineer, @Uber) Uber builds one of the biggest data lakes in the industry, which stores exabytes of data. In this talk, we will introduce the evolution of our data storage architecture, and delve into multiple key initiatives during the past several years. Specifically, we will introduce: - Our on-prem HDFS cluster scalability challenges and how we solved them - Our efficiency optimizations that significantly reduced the storage overhead and unit cost without compromising reliability and performance - The challenges we are facing during the ongoing Cloud migration and our solutions

Alluxio Monthly Webinar | Why NFS/NAS on Object Storage May Not Solve Your AI...

Alluxio, Inc.

Alluxio Monthly Webinar Nov. 15, 2023 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Tarik Bennett (Senior Solutions Engineer) - Beinan Wang (Senior Staff Engineer & Architect) Many companies are working with development architectures for AI platforms but have concerns about efficiency at scale as data volumes increase. They use centralized cloud data lakes, like S3, to store training data for AI platforms. However, GPU shortages add more complications. Storage and compute can be separate, or even remote, making data loading slow and expensive: 1) Optimizing a developmental setup can include manual copies, which are slow and error-prone 2) Directly transferring data across regions or from cloud to on-premises can incur expensive egress fees This webinar covers solutions to improve data loading for model training. You will learn: - The data loading challenges with distributed infrastructure - Typical solutions, including NFS/NAS on object storage, and why they are not the best options - Common architectures that can improve data loading and cost efficiency - Using Alluxio to accelerate model training and reduce costs

AI Infra Day | Accelerate Your Model Training and Serving with Distributed Ca...

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Adit Madan (Director of Product Management, @Alluxio) In this session, Adit Madan, Director of Product Management at Alluxio, presents an overview of using distributed caching to accelerate model training and serving. He explores the requirements of data access patterns in the ML pipeline and offers practical best practices for using distributed caching in the cloud. This session features insights from real-world examples, such as AliPay, Zhihu, and more.

AI Infra Day | The AI Infra in the Generative AI Era

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Bin Fan (Cheif Architect, VP of Open Source, @Alluxio) As the AI landscape rapidly evolves, the advancements in generative AI technologies, such as ChatGPT, are driving a need for a robust AI infra stack. This opening keynote will explore the key trends of the AI infra stack in the generative AI era.

AI Infra Day | Hands-on Lab: CV Model Training with PyTorch & Alluxio on Kube...

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Lu Qiu (Machine Learning Engineer, @Alluxio) - Shawn Sun (Software Engineer, @Alluxio) This hands-on session will discuss best practices for using PyTorch and Alluxio during model training on AWS. Chunxu and Lu will provide a step-by-step demonstration of how to use Alluxio on EKS as a distributed cache to accelerate computer vision model training jobs that read datasets from S3. This architecture significantly improves the utilization of GPUs from 30% to 90%+, archives ~5x faster training, and lower cloud storage costs.

AI Infra Day | The Generative AI Market And Intel AI Strategy and Product Up...

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Jordan Plawner (Global Director of Artificial intelligence Product Management and Strategy, @Intel) ChatGPT and other massive models represents an amazing step forward in AI, yet they do not solve real-world business problems. We will survey how the AI ecosystem has worked non-stop over this last year to take these all-purpose multi-task models and optimize them to they can be used by organizations to address domain specific problems. We will explain these new AI-for-the-real world techniques and methods such as fine tuning and how can be applied to deliver results which are highly performant with state-of-the-art accuracy while also being economical to build and deploy everywhere to enhance products and services.

AI Infra Day | Composable PyTorch Distributed with PT2 @ Meta

Alluxio, Inc.

More from Alluxio, Inc. (20)

AI/ML Infra Meetup | ML explainability in Michelangelo

AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG

AI/ML Infra Meetup | Perspective on Deep Learning Framework

AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...

Optimizing Data Access for Analytics And AI with Alluxio

Speed Up Presto at Uber with Alluxio Caching

Correctly Loading Incremental Data at Scale

Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...

Alluxio Monthly Webinar | Five Disruptive Trends that Every Data & AI Leader...

Data Infra Meetup | FIFO Queues are All You Need for Cache Eviction

Data Infra Meetup | Accelerate Your Trino/Presto Queries - Gain the Alluxio Edge

Data Infra Meetup | Accelerate Distributed PyTorch/Ray Workloads in the Cloud

Data Infra Meetup | ByteDance's Native Parquet Reader

Data Infra Meetup | Uber's Data Storage Evolution

Alluxio Monthly Webinar | Why NFS/NAS on Object Storage May Not Solve Your AI...

AI Infra Day | Accelerate Your Model Training and Serving with Distributed Ca...

AI Infra Day | The AI Infra in the Generative AI Era

AI Infra Day | Hands-on Lab: CV Model Training with PyTorch & Alluxio on Kube...

AI Infra Day | The Generative AI Market And Intel AI Strategy and Product Up...

AI Infra Day | Composable PyTorch Distributed with PT2 @ Meta

Recently uploaded

Empowering Growth with Best Software Development Company in Noida - Deuglo

Deuglo Infosystem Pvt Ltd

Do you want Software for your Business? Visit Deuglo Deuglo has top Software Developers in India. They are experts in software development and help design and create custom Software solutions. Deuglo follows seven steps methods for delivering their services to their customers. They called it the Software development life cycle process (SDLC). Requirement — Collecting the Requirements is the first Phase in the SSLC process. Feasibility Study — after completing the requirement process they move to the design phase. Design — in this phase, they start designing the software. Coding — when designing is completed, the developers start coding for the software. Testing — in this phase when the coding of the software is done the testing team will start testing. Installation — after completion of testing, the application opens to the live server and launches! Maintenance — after completing the software development, customers start using the software.

E-commerce Development Services- Hornet Dynamics

Hornet Dynamics

How to write a program in any programming language

Rakesh Kumar R

Energy consumption of Database Management - Florina Jonuzi

Green Software Development

socradar-q1-2024-aviation-industry-report.pdf

SOCRadar

SOCRadar's Aviation Industry Q1 Incident Report is out now! The aviation industry has always been a prime target for cybercriminals due to its critical infrastructure and high stakes. In the first quarter of 2024, the sector faced an alarming surge in cybersecurity threats, revealing its vulnerabilities and the relentless sophistication of cyber attackers. SOCRadar’s Aviation Industry, Quarterly Incident Report, provides an in-depth analysis of these threats, detected and examined through our extensive monitoring of hacker forums, Telegram channels, and dark web platforms.

Automated software refactoring with OpenRewrite and Generative AI.pptx.pdf

timtebeek1

SMS API Integration in Saudi Arabia| Best SMS API Service

Yara Milbes

Discover the benefits and implementation of SMS API integration in the UAE and Middle East. This comprehensive guide covers the importance of SMS messaging APIs, the advantages of bulk SMS APIs, and real-world case studies. Learn how CEQUENS, a leader in communication solutions, can help your business enhance customer engagement and streamline operations with innovative CPaaS, reliable SMS APIs, and omnichannel solutions, including WhatsApp Business. Perfect for businesses seeking to optimize their communication strategies in the digital age.

Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris

Neo4j

LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM

lorraineandreiamcidl

Webinar On-Demand: Using Flutter for Embedded

ICS

Flutter is a popular open source, cross-platform framework developed by Google. In this webinar we'll explore Flutter and its architecture, delve into the Flutter Embedder and Flutter’s Dart language, discover how to leverage Flutter for embedded device development, learn about Automotive Grade Linux (AGL) and its consortium and understand the rationale behind AGL's choice of Flutter for next-gen IVI systems. Don’t miss this opportunity to discover whether Flutter is right for your project.

2024 eCommerceDays Toulouse - Sylius 2.0.pdf

Łukasz Chruściel

Fundamentals of Programming and Language Processors

Rakesh Kumar R

Vitthal Shirke Java Microservices Resume.pdf

Vitthal Shirke

8 Best Automated Android App Testing Tool and Framework in 2024.pdf

kalichargn70th171

LORRAINE ANDREI_LEQUIGAN_HOW TO USE WHATSAPP.pptx

lorraineandreiamcidl

WhatsApp offers simple, reliable, and private messaging and calling services for free worldwide. With end-to-end encryption, your personal messages and calls are secure, ensuring only you and the recipient can access them. Enjoy voice and video calls to stay connected with loved ones or colleagues. Express yourself using stickers, GIFs, or by sharing moments on Status. WhatsApp Business enables global customer outreach, facilitating sales growth and relationship building through showcasing products and services. Stay connected effortlessly with group chats for planning outings with friends or staying updated on family conversations.

KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD

rodomar2

UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions

Peter Muessig

The UI5 tooling is the development and build tooling of UI5. It is built in a modular and extensible way so that it can be easily extended by your needs. This session will showcase various tooling extensions which can boost your development experience by far so that you can really work offline, transpile your code in your project to use even newer versions of EcmaScript (than 2022 which is supported right now by the UI5 tooling), consume any npm package of your choice in your project, using different kind of proxies, and even stitching UI5 projects during development together to mimic your target environment.

Measures in SQL (SIGMOD 2024, Santiago, Chile)

Julian Hyde

SQL has attained widespread adoption, but Business Intelligence tools still use their own higher level languages based upon a multidimensional paradigm. Composable calculations are what is missing from SQL, and we propose a new kind of column, called a measure, that attaches a calculation to a table. Like regular tables, tables with measures are composable and closed when used in queries. SQL-with-measures has the power, conciseness and reusability of multidimensional languages but retains SQL semantics. Measure invocations can be expanded in place to simple, clear SQL. To define the evaluation semantics for measures, we introduce context-sensitive expressions (a way to evaluate multidimensional expressions that is consistent with existing SQL semantics), a concept called evaluation context, and several operations for setting and modifying the evaluation context. A talk at SIGMOD, June 9–15, 2024, Santiago, Chile Authors: Julian Hyde (Google) and John Fremlin (Google) https://doi.org/10.1145/3626246.3653374

Oracle 23c New Features For DBAs and Developers.pptx

Remote DBA Services

SWEBOK and Education at FUSE Okinawa 2024

Hironori Washizaki

Recently uploaded (20)

Empowering Growth with Best Software Development Company in Noida - Deuglo

E-commerce Development Services- Hornet Dynamics

How to write a program in any programming language

Energy consumption of Database Management - Florina Jonuzi

socradar-q1-2024-aviation-industry-report.pdf

Automated software refactoring with OpenRewrite and Generative AI.pptx.pdf

SMS API Integration in Saudi Arabia| Best SMS API Service

Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris

LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM

Webinar On-Demand: Using Flutter for Embedded

2024 eCommerceDays Toulouse - Sylius 2.0.pdf

Fundamentals of Programming and Language Processors

Vitthal Shirke Java Microservices Resume.pdf

8 Best Automated Android App Testing Tool and Framework in 2024.pdf

LORRAINE ANDREI_LEQUIGAN_HOW TO USE WHATSAPP.pptx

KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD

UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions

Measures in SQL (SIGMOD 2024, Santiago, Chile)

Oracle 23c New Features For DBAs and Developers.pptx

SWEBOK and Education at FUSE Okinawa 2024

What's New in Alluxio 2.3

1. What’s New in Alluxio 2.3 7/14 Office Hour - Bin Fan, Calvin Jia Please Join https://alluxio.io/slack for Q&A

2. Alluxio 2.3 Focus Areas ● Hybrid Cloud ○ Significant number of new use cases ○ Large amount of performance gain and/or cost savings ● Structured Data Services (SDS) ○ Presto continues to be a popular compute engine ○ More interest from cloud and hybrid use cases ● System Scalability & Operability ○ Large scale exposed limits in multi-tier storage management ○ Critical deployments require regular journal backups

3. Hybrid Cloud Features ● One Command Deployment on Amazon EMR and Dataproc ○ Try the demo hybrid use case - AWS or GCP ○ Use Terraform modules to customize your own cloud deployments ● Validate Env ○ Set of tools for connecting to remote storage and validating it works ○ New GUI for mounting remote storage which runs several diagnostic checks ○ New CLI commands for directly running the checks ■ runHdfsMountTests - checks HDFS connectivity and settings ■ runUfsIOTest - checks UFS I/O capabilities ■ runHmsTests - checks Hive Metastore connectivity and settings

4. Hybrid Cloud Features ● Concurrent Metadata Synchronization ○ Enable stricter metadata synchronization settings without performance loss ○ Improved synchronization algorithm to enable multiple paths in the same subtree to be synced at once ○ Order of magnitude improvement, especially for larger namespaces that can utilize the full parallelism ○ Improved performance of INotify based active sync, an important feature in hybrid deployments

5. SDS Features ● Support for Amazon Glue UDB ○ Use Amazon Glue to store table metadata instead of HMS ○ Many users in cloud requested Glue support ○ Will continue to provide compatibility in the cloud ● Full Transformation Support for ORC format ○ Benefit from transformation capabilities in SDS with ORC ○ Seeing a number of users on ORC and Parquet ○ ORC vs Parquet is mostly based on the compute framework and available optimizations

6. System Scalability ● Tiered Storage V2 ○ Use multi-tiered storage in read-write heavy workloads without performance loss ○ Eviction becomes O(1) operation, ensure write or cache path is not impacted ○ Evictors are now termed Annotators because it simply pre-marks the blocks to evict ○ Trade-off made is block placement is not 100% accurate when using multiple tiers ○ Background tasks are system resource aware and will operate only during quiet periods

7. System Operability ● Delegated Journal Backups for HA Deployments ○ Backups no longer impact master response latency ○ Instead of using primary master to backup, secondaries can run backups ○ Very useful for deployments running regular backups, can avoid performance impact ○ Window for performance hit is reduced to several seconds instead of minutes

8. Case Studies

9. Improving Presto Latency at Facebook With Alluxio data caching, for one of the Facebook internal Presto use cases: - query latencies improved by 33% (P50), and 48% (P95). - 57% improvement in IO for remote data source scans. https://prestodb.io/blog/2020/06/16/alluxio-datacaching Presto + Alluxio + Facebook HDFS

10. Memory-speed I/O for Model Training in K8s Alibaba Cloud K8s team is leveraging Alluxio Posix Interface to serve deep learning training in K8s in Kubernetes environment. - 40% reduction in training time & cost https://www.alluxio.io/blog/efficient-model-training-in-the-cloud-with-kubernetes-tensorflow-and-alluxio/ Tensorflow + Alluxio + OSS + K8s

11. Data Proxy for Hybrid Cloud WeRide, a self-driving-car company, deployed Alluxio to leverage Alluxio as a local cache in each offices, on top of a centralized S3 data lake - Saving $5 per issue ticket per engineer https://www.alluxio.io/blog/building-a-cross-region-hybrid-cloud-storage-gateway-for-engineers-at-weride/ DS tools + Alluxio + S3 (remote)

What's New in Alluxio 2.3

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to What's New in Alluxio 2.3

Similar to What's New in Alluxio 2.3 (20)

More from Alluxio, Inc.

More from Alluxio, Inc. (20)

Recently uploaded

Recently uploaded (20)

What's New in Alluxio 2.3