Big Data Day LA 2015 - HBase at Factual: Real time and Batch Uses by Molly O'Connor of Factual
•
2 likes•977 views
This talk will explore how factual uses some of the advanced features in version .98 and beyond to build a robust and flexible data store.
Report
Share
Report
Share
Recommended
No sql
Amazon DynamoDB is a scalable and distributed NoSQL database that scales horizontally and seamlessly over hundreds of servers. It has flexible schemas and automatic data replication but has limitations on row and query sizes. HBase is an open-source, distributed, scalable big data store that provides real-time random access to big data and is highly scalable and reliable but has single points of failure. MongoDB is a scalable, schema-less document database ideal for flexible schemas with easy horizontal scaling and high availability but has higher data sizes than relational databases and less querying flexibility. Redis is an in-memory data structure store with no query language and offers fast data access but lacks security and recoverability of relational databases.
Big data should be simple
Big data challenges are common : we are all doing aggregations , machine learning , anomaly detection, OLAP ...
This presentation describe how InnerActive answer those requirements
HBase, dances on the elephant back.
Apache HBase is a technology that turns everything in Hadoop infrastructure upside down. An elephant cannot become an antelope, but yet it is possible to do a group dance on its back.
Four NoSQL Databases You Should Know
This document provides an overview of different database types including relational, non-relational, key-value, document, graph, and column family databases. It discusses the history and drivers behind the rise of non-SQL databases, as well as concepts like horizontal scaling, the CAP theorem, and eventual consistency. Specific databases are also summarized, including MongoDB, Redis, Neo4j, HBase, and how they approach concepts like scaling, data models, and consistency.
Tuning Apache Phoenix/HBase
This document discusses performance optimization techniques for Apache HBase and Phoenix at TRUECar. It begins with an agenda and overview of TRUECar's data architecture. It then discusses use cases for HBase/Phoenix at TRUECar and various performance optimization techniques including cluster settings, table settings, data modeling, and EC2 instance types. Specific techniques covered include pre-splitting tables, bloom filters, hints like SMALL and NO_CACHE, in-memory storage, incremental keys, and using faster instance types like i3.2xlarge. The document aims to provide insights on optimizing HBase/Phoenix performance gained from TRUECar's experiences.
hbaseconasia2017: HBase on Beam
Jingcheng Du
Apache Beam is an open source and unified programming model for defining batch and streaming jobs that run on many execution engines, HBase on Beam is a connector that allows Beam to use HBase as a bounded data source and target data store for both batch and streaming data sets. With this connector HBase can work with many batch and streaming engines directly, for example Spark, Flink, Google Cloud Dataflow, etc. In this session, I will introduce Apache Beam, and the current implementation of HBase on Beam and the future plan on this.
hbaseconasia2017 hbasecon hbase
https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#
Thrashing allocation frames.43
This document discusses thrashing and allocation of frames in an operating system. It defines thrashing as when a processor spends most of its time swapping pieces of processes in and out rather than executing user instructions. This leads to low CPU utilization. It also discusses how to allocate a minimum number of frames to each process to prevent thrashing and ensure efficient paging.
Hbase
In this session you will learn:
HBase Introduction
Row & Column storage
Characteristics of a huge DB
What is HBase?
HBase Data-Model
HBase vs RDBMS
HBase architecture
HBase in operation
Loading Data into HBase
HBase shell commands
HBase operations through Java
HBase operations through MR
To know more, click here: https://www.mindsmapped.com/courses/big-data-hadoop/big-data-and-hadoop-training-for-beginners/
Recommended
No sql
Amazon DynamoDB is a scalable and distributed NoSQL database that scales horizontally and seamlessly over hundreds of servers. It has flexible schemas and automatic data replication but has limitations on row and query sizes. HBase is an open-source, distributed, scalable big data store that provides real-time random access to big data and is highly scalable and reliable but has single points of failure. MongoDB is a scalable, schema-less document database ideal for flexible schemas with easy horizontal scaling and high availability but has higher data sizes than relational databases and less querying flexibility. Redis is an in-memory data structure store with no query language and offers fast data access but lacks security and recoverability of relational databases.
Big data should be simple
Big data challenges are common : we are all doing aggregations , machine learning , anomaly detection, OLAP ...
This presentation describe how InnerActive answer those requirements
HBase, dances on the elephant back.
Apache HBase is a technology that turns everything in Hadoop infrastructure upside down. An elephant cannot become an antelope, but yet it is possible to do a group dance on its back.
Four NoSQL Databases You Should Know
This document provides an overview of different database types including relational, non-relational, key-value, document, graph, and column family databases. It discusses the history and drivers behind the rise of non-SQL databases, as well as concepts like horizontal scaling, the CAP theorem, and eventual consistency. Specific databases are also summarized, including MongoDB, Redis, Neo4j, HBase, and how they approach concepts like scaling, data models, and consistency.
Tuning Apache Phoenix/HBase
This document discusses performance optimization techniques for Apache HBase and Phoenix at TRUECar. It begins with an agenda and overview of TRUECar's data architecture. It then discusses use cases for HBase/Phoenix at TRUECar and various performance optimization techniques including cluster settings, table settings, data modeling, and EC2 instance types. Specific techniques covered include pre-splitting tables, bloom filters, hints like SMALL and NO_CACHE, in-memory storage, incremental keys, and using faster instance types like i3.2xlarge. The document aims to provide insights on optimizing HBase/Phoenix performance gained from TRUECar's experiences.
hbaseconasia2017: HBase on Beam
Jingcheng Du
Apache Beam is an open source and unified programming model for defining batch and streaming jobs that run on many execution engines, HBase on Beam is a connector that allows Beam to use HBase as a bounded data source and target data store for both batch and streaming data sets. With this connector HBase can work with many batch and streaming engines directly, for example Spark, Flink, Google Cloud Dataflow, etc. In this session, I will introduce Apache Beam, and the current implementation of HBase on Beam and the future plan on this.
hbaseconasia2017 hbasecon hbase
https://www.eventbrite.com/e/hbasecon-asia-2017-tickets-34935546159#
Thrashing allocation frames.43
This document discusses thrashing and allocation of frames in an operating system. It defines thrashing as when a processor spends most of its time swapping pieces of processes in and out rather than executing user instructions. This leads to low CPU utilization. It also discusses how to allocate a minimum number of frames to each process to prevent thrashing and ensure efficient paging.
Hbase
In this session you will learn:
HBase Introduction
Row & Column storage
Characteristics of a huge DB
What is HBase?
HBase Data-Model
HBase vs RDBMS
HBase architecture
HBase in operation
Loading Data into HBase
HBase shell commands
HBase operations through Java
HBase operations through MR
To know more, click here: https://www.mindsmapped.com/courses/big-data-hadoop/big-data-and-hadoop-training-for-beginners/
Apache HBase
HBase is a distributed column-oriented database built on top of Hadoop that provides quick random access to large amounts of structured data. It uses a key-value structure to store table data by row key, column family, column, and timestamp. Tables consist of rows, column families, and columns, with a version dimension to store multiple values over time. HBase is well-suited for applications requiring real-time read/write access and is commonly used to store web crawler results or search indexes.
RocksDB detail
RocksDB is an embedded key-value store written in C++ and optimized for fast storage environments like flash or RAM. It uses a log-structured merge tree to store data by writing new data sequentially to an in-memory log and memtable, periodically flushing the memtable to disk in sorted SSTables. It reads from the memtable and SSTables, and performs background compaction to merge SSTables and remove overwritten data. RocksDB supports two compaction styles - level style, which stores SSTables in multiple levels sorted by age, and universal style, which stores all SSTables in level 0 sorted by time.
Improve Presto Architectural Decisions with Shadow Cache
Alluxio Day VI
October 12, 2021
https://www.alluxio.io/alluxio-day/
Speaker:
Ke Wang, Facebook
Zhenyu Song, Princeton University
RocksDB compaction
The document discusses compaction in RocksDB, an embedded key-value storage engine. It describes the two compaction styles in RocksDB: level style compaction and universal style compaction. Level style compaction stores data in multiple levels and performs compactions by merging files from lower to higher levels. Universal style compaction keeps all files in level 0 and performs compactions by merging adjacent files in time order. The document provides details on the compaction process and configuration options for both styles.
Incremental backups
Incremental backups can be performed by tracking changed database pages since the last backup. This can be done through three main methods: full table scan, using redo logs, or logging changed page IDs. Logging changed page IDs avoids the overhead of a full scan and redo log archiving. The server tracks page modifications in a bitmap file. For incremental backups, only pages marked as changed since the last backup need to be read, reducing backup time and storage needs compared to a full backup or redo log approach. This page ID tracking provides an efficient alternative to full table scans or redo log archiving for incremental backups.
TriHUG 3/14: HBase in Production
This document discusses Bronto's use of HBase for their marketing platform. Some key points:
- Bronto uses HBase for high volume scenarios, realtime data access, batch processing, and as a staging area for HDFS.
- HBase tables at Bronto are designed with the read/write patterns and necessary queries in mind. Row keys and column families are structured to optimize for these access patterns.
- Operations of HBase at scale require tuning of JVM settings, monitoring tools, and custom scripts to handle compactions and prevent cascading failures during high load. Table design also impacts operations and needs to account for expected workloads.
The Hive Think Tank: Rocking the Database World with RocksDB
RocksDB is a new storage engine for MySQL that provides better storage efficiency than InnoDB. It achieves lower space amplification and write amplification than InnoDB through its use of compression and log-structured merge trees. While MyRocks (RocksDB integrated with MySQL) currently has some limitations like a lack of support for online DDL and spatial indexes, work is ongoing to address these limitations and integrate additional RocksDB features to fully support MySQL workloads. Testing at Facebook showed MyRocks uses less disk space and performs comparably to InnoDB for their queries.
The Google Bigtable
I promise that understand NoSQL is as easy as playing with LEGO bricks ! The Google Bigtable presented in 2006 is the inspiration for Apache HBase: let's take a deep dive into Bigtable to better understand Hbase.
Web scale monitoring
The document describes a web scale monitoring system using various technologies like Gearman, Redis, Mojolicious, Angular.js, Gnuplot and PostgreSQL. The system polls CPE, DSLAM and MSAN devices to collect data, stores it in PostgreSQL with hstore and Redis caching, and provides a web interface using Mojolicious and Angular.js to inspect the data. The goals are horizontal scalability, preserving data structure, and easy deployment through test driven development.
RTree Spatial Indexing with MongoDB - MongoDC
Thermopylae Sciences & Technology chose to customize MongoDB's spatial indexing capabilities to better support their needs for indexing multi-dimensional and geospatial data. They developed a custom R-tree spatial index that leverages existing MongoDB data structures and provides improved performance over MongoDB's existing geohash-based approach. Their custom index supports complex queries on multidimensional geometric shapes and scales to large geospatial datasets through potential sharding and distribution techniques. They have contributed their work back to the MongoDB open source project and collaborate with MongoDB to further integrate their contributions.
HBase, crazy dances on the elephant back.
Based on "HBase, dances on the elephant back" presentation success I have prepared its update for JavaDay 2014 Kyiv. Again, it is about the product which revolutionary changes everything inside Hadoop infrastructure: Apache HBase. But here focus is shifted to integration and more advanced topics keeping presentation yet understandable for technology newcomers.
Storage in hadoop
An overview of Hadoop Storage Format and different codecs available. It explains which are available and how they are different and which to use where.
Optimizing columnar stores
This document discusses optimizing columnar data stores. It begins with an overview of row-oriented versus column-oriented data stores, noting that column stores are well-suited for read-heavy analytical loads as they only need to read relevant data. The document then covers the history of columnar stores and notable features like data encoding, compression techniques like run-length encoding, and lazy decompression. Specific columnar file formats like RCFile, ORC, and Parquet are mentioned. The document concludes with a case study describing optimizations made to a 1PB Hive table that resulted in a 3x query performance improvement through techniques like explicit sorting, improved compression, increased bucketing, and stripe size tuning.
RocksDB storage engine for MySQL and MongoDB
My talk from Percona Live Europe 2015. Presenting RocksDB storage engine for MySQL and MongoDB. The talk covers RocksDB story, its internals and gives some hints on performance tuning.
Gfs vs hdfs
This document compares the Google File System (GFS) and the Hadoop Distributed File System (HDFS). It discusses their motivations, architectures, performance measurements, and role in larger systems. GFS was designed for Google's data processing needs, while HDFS was created as an open-source framework for Hadoop applications. Both divide files into blocks and replicate data across multiple servers for reliability. The document provides details on their file structures, data flow models, consistency approaches, and benchmark results. It also explores how systems like MapReduce/Hadoop utilize these underlying storage systems.
HBase Incremental Backup
CopyTable allows copying data between HBase tables either within or between clusters. Export dumps the contents of a table to HDFS in sequence files. Import loads exported data back into HBase. For regular incremental backups, Export is recommended with a hierarchical output directory structure organized by date/time. Data can then be restored using Import on demand. Backup/restore should be done during off-peak hours to reduce overhead.
Apache hadoop, hdfs and map reduce Overview
This document provides an overview of Apache Hadoop, HDFS, and MapReduce. It describes how Hadoop uses a distributed file system (HDFS) to store large amounts of data across commodity hardware. It also explains how MapReduce allows distributed processing of that data by allocating map and reduce tasks across nodes. Key components discussed include the HDFS architecture with NameNodes and DataNodes, data replication for fault tolerance, and how the MapReduce engine works with a JobTracker and TaskTrackers to parallelize jobs.
Apache Hadoop India Summit 2011 Keynote talk "HDFS Federation" by Sanjay Radia
Apache Hadoop India Summit 2011 Keynote talk "HDFS Federation" by Sanjay RadiaYahoo Developer Network
This document discusses scaling HDFS through federation. HDFS currently uses a single namenode that limits scalability. Federation allows multiple independent namenodes to each manage a subset of the namespace, improving scalability. It also generalizes the block storage layer to use block pools, separating block management from namenodes. This paves the way for horizontal scaling of both namenodes and block storage in the future. Federation preserves namenode robustness while requiring few code changes. It also provides benefits like improved isolation and availability when scaling to extremely large clusters with billions of files and blocks.Ceph Day Berlin: Measuring and predicting performance of Ceph clusters
This document provides a summary of a presentation about modeling, estimating, and predicting performance for Ceph storage clusters. The presentation discusses the challenges of predicting SDS (software-defined storage) performance due to the large number of configurable options. It proposes collecting standardized benchmark and configuration data from production systems to build a dataset that can provide better performance insights and predictions through analysis. The goal is to develop a benchmark suite to holistically evaluate Ceph performance and address common customer questions about how storage systems with different configurations may perform.
Hadoop ecosystem; J.Ayeesha parveen 2 nd M.sc., computer science Bon Secours...
Hadoop
Open source
Distributed processing
Distributed storage
Scalable
Reliable
Fault-tolerant
Economical
Flexible
Big datacamp june14_alex_liu
This document discusses how big data analytics can provide insights from large amounts of structured and unstructured data. It provides examples of how big data has helped organizations reduce customer churn, improve customer acquisition, speed up loan approvals, and detect fraud. The document also outlines IBM's big data platform and analytics process for extracting value from large, diverse data sources.
Aziksa hadoop for buisness users2 santosh jha
This document discusses big data, including its drivers, characteristics, use cases across different industries, and lessons learned. It provides examples of companies like Etsy, Macy's, Canadian Pacific, and Salesforce that are using big data to gain insights, increase revenues, reduce costs and improve customer experiences. Big data is being used across industries like financial services, healthcare, manufacturing, and media/entertainment for applications such as customer profiling, fraud detection, operations optimization, and dynamic pricing. While big data projects show strong financial benefits, the document cautions that not all projects are well-structured and Hadoop alone is not sufficient to meet all business analysis needs.
More Related Content
What's hot
Apache HBase
HBase is a distributed column-oriented database built on top of Hadoop that provides quick random access to large amounts of structured data. It uses a key-value structure to store table data by row key, column family, column, and timestamp. Tables consist of rows, column families, and columns, with a version dimension to store multiple values over time. HBase is well-suited for applications requiring real-time read/write access and is commonly used to store web crawler results or search indexes.
RocksDB detail
RocksDB is an embedded key-value store written in C++ and optimized for fast storage environments like flash or RAM. It uses a log-structured merge tree to store data by writing new data sequentially to an in-memory log and memtable, periodically flushing the memtable to disk in sorted SSTables. It reads from the memtable and SSTables, and performs background compaction to merge SSTables and remove overwritten data. RocksDB supports two compaction styles - level style, which stores SSTables in multiple levels sorted by age, and universal style, which stores all SSTables in level 0 sorted by time.
Improve Presto Architectural Decisions with Shadow Cache
Alluxio Day VI
October 12, 2021
https://www.alluxio.io/alluxio-day/
Speaker:
Ke Wang, Facebook
Zhenyu Song, Princeton University
RocksDB compaction
The document discusses compaction in RocksDB, an embedded key-value storage engine. It describes the two compaction styles in RocksDB: level style compaction and universal style compaction. Level style compaction stores data in multiple levels and performs compactions by merging files from lower to higher levels. Universal style compaction keeps all files in level 0 and performs compactions by merging adjacent files in time order. The document provides details on the compaction process and configuration options for both styles.
Incremental backups
Incremental backups can be performed by tracking changed database pages since the last backup. This can be done through three main methods: full table scan, using redo logs, or logging changed page IDs. Logging changed page IDs avoids the overhead of a full scan and redo log archiving. The server tracks page modifications in a bitmap file. For incremental backups, only pages marked as changed since the last backup need to be read, reducing backup time and storage needs compared to a full backup or redo log approach. This page ID tracking provides an efficient alternative to full table scans or redo log archiving for incremental backups.
TriHUG 3/14: HBase in Production
This document discusses Bronto's use of HBase for their marketing platform. Some key points:
- Bronto uses HBase for high volume scenarios, realtime data access, batch processing, and as a staging area for HDFS.
- HBase tables at Bronto are designed with the read/write patterns and necessary queries in mind. Row keys and column families are structured to optimize for these access patterns.
- Operations of HBase at scale require tuning of JVM settings, monitoring tools, and custom scripts to handle compactions and prevent cascading failures during high load. Table design also impacts operations and needs to account for expected workloads.
The Hive Think Tank: Rocking the Database World with RocksDB
RocksDB is a new storage engine for MySQL that provides better storage efficiency than InnoDB. It achieves lower space amplification and write amplification than InnoDB through its use of compression and log-structured merge trees. While MyRocks (RocksDB integrated with MySQL) currently has some limitations like a lack of support for online DDL and spatial indexes, work is ongoing to address these limitations and integrate additional RocksDB features to fully support MySQL workloads. Testing at Facebook showed MyRocks uses less disk space and performs comparably to InnoDB for their queries.
The Google Bigtable
I promise that understand NoSQL is as easy as playing with LEGO bricks ! The Google Bigtable presented in 2006 is the inspiration for Apache HBase: let's take a deep dive into Bigtable to better understand Hbase.
Web scale monitoring
The document describes a web scale monitoring system using various technologies like Gearman, Redis, Mojolicious, Angular.js, Gnuplot and PostgreSQL. The system polls CPE, DSLAM and MSAN devices to collect data, stores it in PostgreSQL with hstore and Redis caching, and provides a web interface using Mojolicious and Angular.js to inspect the data. The goals are horizontal scalability, preserving data structure, and easy deployment through test driven development.
RTree Spatial Indexing with MongoDB - MongoDC
Thermopylae Sciences & Technology chose to customize MongoDB's spatial indexing capabilities to better support their needs for indexing multi-dimensional and geospatial data. They developed a custom R-tree spatial index that leverages existing MongoDB data structures and provides improved performance over MongoDB's existing geohash-based approach. Their custom index supports complex queries on multidimensional geometric shapes and scales to large geospatial datasets through potential sharding and distribution techniques. They have contributed their work back to the MongoDB open source project and collaborate with MongoDB to further integrate their contributions.
HBase, crazy dances on the elephant back.
Based on "HBase, dances on the elephant back" presentation success I have prepared its update for JavaDay 2014 Kyiv. Again, it is about the product which revolutionary changes everything inside Hadoop infrastructure: Apache HBase. But here focus is shifted to integration and more advanced topics keeping presentation yet understandable for technology newcomers.
Storage in hadoop
An overview of Hadoop Storage Format and different codecs available. It explains which are available and how they are different and which to use where.
Optimizing columnar stores
This document discusses optimizing columnar data stores. It begins with an overview of row-oriented versus column-oriented data stores, noting that column stores are well-suited for read-heavy analytical loads as they only need to read relevant data. The document then covers the history of columnar stores and notable features like data encoding, compression techniques like run-length encoding, and lazy decompression. Specific columnar file formats like RCFile, ORC, and Parquet are mentioned. The document concludes with a case study describing optimizations made to a 1PB Hive table that resulted in a 3x query performance improvement through techniques like explicit sorting, improved compression, increased bucketing, and stripe size tuning.
RocksDB storage engine for MySQL and MongoDB
My talk from Percona Live Europe 2015. Presenting RocksDB storage engine for MySQL and MongoDB. The talk covers RocksDB story, its internals and gives some hints on performance tuning.
Gfs vs hdfs
This document compares the Google File System (GFS) and the Hadoop Distributed File System (HDFS). It discusses their motivations, architectures, performance measurements, and role in larger systems. GFS was designed for Google's data processing needs, while HDFS was created as an open-source framework for Hadoop applications. Both divide files into blocks and replicate data across multiple servers for reliability. The document provides details on their file structures, data flow models, consistency approaches, and benchmark results. It also explores how systems like MapReduce/Hadoop utilize these underlying storage systems.
HBase Incremental Backup
CopyTable allows copying data between HBase tables either within or between clusters. Export dumps the contents of a table to HDFS in sequence files. Import loads exported data back into HBase. For regular incremental backups, Export is recommended with a hierarchical output directory structure organized by date/time. Data can then be restored using Import on demand. Backup/restore should be done during off-peak hours to reduce overhead.
Apache hadoop, hdfs and map reduce Overview
This document provides an overview of Apache Hadoop, HDFS, and MapReduce. It describes how Hadoop uses a distributed file system (HDFS) to store large amounts of data across commodity hardware. It also explains how MapReduce allows distributed processing of that data by allocating map and reduce tasks across nodes. Key components discussed include the HDFS architecture with NameNodes and DataNodes, data replication for fault tolerance, and how the MapReduce engine works with a JobTracker and TaskTrackers to parallelize jobs.
Apache Hadoop India Summit 2011 Keynote talk "HDFS Federation" by Sanjay Radia
Apache Hadoop India Summit 2011 Keynote talk "HDFS Federation" by Sanjay RadiaYahoo Developer Network
This document discusses scaling HDFS through federation. HDFS currently uses a single namenode that limits scalability. Federation allows multiple independent namenodes to each manage a subset of the namespace, improving scalability. It also generalizes the block storage layer to use block pools, separating block management from namenodes. This paves the way for horizontal scaling of both namenodes and block storage in the future. Federation preserves namenode robustness while requiring few code changes. It also provides benefits like improved isolation and availability when scaling to extremely large clusters with billions of files and blocks.Ceph Day Berlin: Measuring and predicting performance of Ceph clusters
This document provides a summary of a presentation about modeling, estimating, and predicting performance for Ceph storage clusters. The presentation discusses the challenges of predicting SDS (software-defined storage) performance due to the large number of configurable options. It proposes collecting standardized benchmark and configuration data from production systems to build a dataset that can provide better performance insights and predictions through analysis. The goal is to develop a benchmark suite to holistically evaluate Ceph performance and address common customer questions about how storage systems with different configurations may perform.
Hadoop ecosystem; J.Ayeesha parveen 2 nd M.sc., computer science Bon Secours...
Hadoop
Open source
Distributed processing
Distributed storage
Scalable
Reliable
Fault-tolerant
Economical
Flexible
What's hot (20)
Improve Presto Architectural Decisions with Shadow Cache
Improve Presto Architectural Decisions with Shadow Cache
The Hive Think Tank: Rocking the Database World with RocksDB
The Hive Think Tank: Rocking the Database World with RocksDB
Apache Hadoop India Summit 2011 Keynote talk "HDFS Federation" by Sanjay Radia
Apache Hadoop India Summit 2011 Keynote talk "HDFS Federation" by Sanjay Radia
Ceph Day Berlin: Measuring and predicting performance of Ceph clusters
Ceph Day Berlin: Measuring and predicting performance of Ceph clusters
Hadoop ecosystem; J.Ayeesha parveen 2 nd M.sc., computer science Bon Secours...
Hadoop ecosystem; J.Ayeesha parveen 2 nd M.sc., computer science Bon Secours...
Viewers also liked
Big datacamp june14_alex_liu
This document discusses how big data analytics can provide insights from large amounts of structured and unstructured data. It provides examples of how big data has helped organizations reduce customer churn, improve customer acquisition, speed up loan approvals, and detect fraud. The document also outlines IBM's big data platform and analytics process for extracting value from large, diverse data sources.
Aziksa hadoop for buisness users2 santosh jha
This document discusses big data, including its drivers, characteristics, use cases across different industries, and lessons learned. It provides examples of companies like Etsy, Macy's, Canadian Pacific, and Salesforce that are using big data to gain insights, increase revenues, reduce costs and improve customer experiences. Big data is being used across industries like financial services, healthcare, manufacturing, and media/entertainment for applications such as customer profiling, fraud detection, operations optimization, and dynamic pricing. While big data projects show strong financial benefits, the document cautions that not all projects are well-structured and Hadoop alone is not sufficient to meet all business analysis needs.
Big Data Day LA 2015 - NoSQL: Doing it wrong before getting it right by Lawre...
The team at Fandango heartily embraced NoSQL, using Couchbase to power a key media publishing system. The initial implementation was fraught with integration issues and high latency, and required a major effort to successfully refactor. My talk will outline the key organizational and architectural decisions that created deep systemic problems, and the steps taken to re-architect the system to achieve a high level of performance at scale.
La big datacamp2014_vikram_dixit
Big Data Camp LA 2014, Hive 0.13: An upgrade in performance, scaling, security and multi-tenancy by Vikram Dixit of Hortonworks
Kiji cassandra la june 2014 - v02 clint-kelly
Big Data Camp LA 2014, Don't re-invent the Big-Data Wheel, Building real-time, Big Data applications on Cassandra with the open-source Kiji project by Clint Kelly of Wibidata
Summit v4 dave wolcott
The document summarizes the Hadoop stack and its components for storing and analyzing big data. It includes file storage with HDFS, data processing with MapReduce, data access tools like Hive and Pig, and security/monitoring with Kerberos and Nagios. HDFS uses metadata to track file locations across data nodes in a fault-tolerant manner similar to a file system.
20140614 introduction to spark-ben white
This document provides an introduction to Apache Spark. It begins by explaining how Spark improves upon MapReduce by leveraging distributed memory for better performance and supporting iterative algorithms. Spark is described as a general purpose computational framework that retains the advantages of MapReduce like scalability and fault tolerance, while offering more functionality through directed acyclic graphs and libraries for machine learning. The document then discusses getting started with Spark and its execution modes like standalone, YARN client, and YARN cluster. Finally, it introduces Spark concepts like Resilient Distributed Datasets (RDDs), which are collections of objects partitioned across a cluster.
140614 bigdatacamp-la-keynote-jon hsieh
The document discusses the evolution of big data stacks from their origins inspired by Google's systems through imitation via Hadoop-based stacks to ongoing innovation. It traces the development of major components like MapReduce, HDFS, HBase and their adoption beyond Google. It also outlines the timeline of open source projects and companies in this space from 2003 to the present.
Big Data Day LA 2015 - Solr Search with Spark for Big Data Analytics in Actio...
Apache Solr makes it so easy to interactively visualize and explore your data. Create a dashboard, add some facets, select some values, cross it with the time… and just look at the results.
Apache Spark is the growing framework for performing streaming computations, which makes it ideal for real time indexing.
Solr also comes with new Analytics Facets which are a major weapon added to the arsenal of the data explorer. They bring another dimension: calculations. We can now do the equivalent of SQL, just in a much simpler and faster way. These calculations can operate over buckets of data. For example, it is now possible to see the sum of Web traffic by country over the time, the median price of some categories of products, which ads are bringing more money by location...
This talks puts in practice some of the leading features of Solr Search. It presents the main types of facets/stats and which advanced properties and usage make them shine. A demo in parallel with the open source Search App in Hue will demonstrate how these facets can power interactive widgets or your own analytic queries. The data will be indexed in real time from a live stream with Spark.
Yarn cloudera-kathleenting061414 kate-ting
This document summarizes Kathleen Ting's presentation on migrating to MapReduce v2 (MRv2) on YARN. The presentation covered the motivation for moving to MRv2 and YARN, including higher cluster utilization and lower costs. It then discussed common misconfiguration issues seen in support tickets, such as memory, thread pool size, and federation misconfigurations. Specific examples were provided for resolving task memory errors, JobTracker memory errors, and fetch failures in both MRv1 and MRv2. Recommendations were given for optimizing YARN memory usage and CPU isolation in containers.
2014 bigdatacamp asya_kamsky
The document discusses various options for processing and aggregating data in MongoDB, including the Aggregation Framework, MapReduce, and connecting MongoDB to external systems like Hadoop. The Aggregation Framework is described as a flexible way to query and transform data in MongoDB using a JSON-like syntax and pipeline stages. MapReduce is presented as more versatile but also more complex to implement. Connecting to external systems like Hadoop allows processing large amounts of data across clusters.
Ag big datacampla-06-14-2014-ajay_gopal
This document provides an overview of CARD.COM, a company that offers prepaid debit cards customized with different designs. They collect data from card transactions, member interactions on their site/app, and marketing platforms to test different designs and better understand customer behavior. Their goal is to use data science to personalize the financial experience for members and potentially offer services like credit scores for the unbanked. They are hiring various technical roles and use open source tools like R, Python, and PHP to build out their analytics platforms and infrastructure.
Hadoop and NoSQL joining forces by Dale Kim of MapR
More and more organizations are turning to Hadoop and NoSQL to manage big data. In fact, many IT professionals consider each of those terms to be synonymous with big data. At the same time, these two technologies are seen as different beasts that handle different challenges. That means they are often deployed in a rather disjointed way, even when intended to solve the same overarching business problem. The emerging trend of “in-Hadoop databases” promises to narrow the deployment gap between them and enable new enterprise applications. In this talk, Dale will describe that integrated architecture and how customers have deployed it to benefit both the technical and the business teams.
Big Data Day LA 2015 - Lessons Learned from Designing Data Ingest Systems by ...
1. The document discusses lessons learned from designing data ingest systems. Key lessons include structuring endpoints wisely, accepting at least once semantics, knowing that change data capture is difficult, understanding service level agreements, considering record format and schema, and tracking record lineage.
2. The document also provides examples of real-world data ingest scenarios and different implementation strategies with analyses of their tradeoffs. It concludes with recommendations to track errors and keep transformations minimal.
Hadoop Innovation Summit 2014
Slides from my presentation at Hadoop Innovation Summit 2014
The Future of Hadoop: Choosing the right options
Big Data Day LA 2015 - Introducing N1QL: SQL for Documents by Jeff Morris of ...
NoSQL has exploded on the developer scene promising alternatives to RDBMS that make rapidly developing, Internet scale applications easier than ever. However, as a trade off to the ease of development and scale, some of the familiarity with other well-known query interfaces such as SQL, has been lost. Until now that is...N1QL (pronounced ‘N1QL’) is a SQL like query language for querying JSON, which brings the familiarity of RDBMS back to the NoSQL world. In this session you will learn about the syntax and basics of this new language as well as Integration with the Couchbase SDKs.
Big Data Day LA 2015 - Deep Learning Human Vocalized Animal Sounds by Sabri S...
Investigated a couple audio based, deep learning strategies for identifying human vocalized car sounds. In one case Mel Frequency Cepstral Coefficients, MFCCs, were used as inputs into a supervised, logistic regression neural network. In a separate case, Short Term Fourier Transforms ,STFT, were used to generate PCA whitened spectograms, which were used as inputs into a supervised, convolutional neural network. The MFCC method trained quickly on a relative small dataset of 4 sounds. The STFT method resulted in a much larger input matrix, resulting in much longer times for converging onto a solution
Big Data Day LA 2016/ Data Science Track - Decision Making and Lambda Archite...
This document discusses decision making systems and the lambda architecture. It introduces decision making algorithms like multi-armed bandits that balance exploration vs exploitation. Contextual multi-armed bandits are discussed as well. The lambda architecture is then described as having serving, speed, and batch layers to enable low latency queries, real-time updates, and batch model training. The software stack of Kafka, Spark/Spark Streaming, HBase and MLLib is presented as enabling scalable stream processing and machine learning.
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Introduction to Kafka - Je...
Kafka is a distributed publish-subscribe system that uses a commit log to track changes. It was originally created at LinkedIn and open sourced in 2011. Kafka decouples systems and is commonly used in enterprise data flows. The document then demonstrates how Kafka works using Legos and discusses key Kafka concepts like topics, partitioning, and the commit log. It also provides examples of how to create Kafka producers and consumers using the Java API.
Big Data Day LA 2016/ Big Data Track - Twitter Heron @ Scale - Karthik Ramasa...
Twitter generates billions and billions of events per day. Analyzing these events in real time presents a massive challenge. Twitter designed and deployed a new streaming system called Heron. Heron has been in production nearly 2 years and is widely used by several teams for diverse use cases. This talk looks at Twitter's operating experiences and challenges of running Heron at scale and the approaches taken to solve those challenges.
Viewers also liked (20)
Big Data Day LA 2015 - NoSQL: Doing it wrong before getting it right by Lawre...
Big Data Day LA 2015 - NoSQL: Doing it wrong before getting it right by Lawre...
Big Data Day LA 2015 - Solr Search with Spark for Big Data Analytics in Actio...
Big Data Day LA 2015 - Solr Search with Spark for Big Data Analytics in Actio...
Hadoop and NoSQL joining forces by Dale Kim of MapR
Hadoop and NoSQL joining forces by Dale Kim of MapR
Big Data Day LA 2015 - Lessons Learned from Designing Data Ingest Systems by ...
Big Data Day LA 2015 - Lessons Learned from Designing Data Ingest Systems by ...
Big Data Day LA 2015 - Introducing N1QL: SQL for Documents by Jeff Morris of ...
Big Data Day LA 2015 - Introducing N1QL: SQL for Documents by Jeff Morris of ...
Big Data Day LA 2015 - Deep Learning Human Vocalized Animal Sounds by Sabri S...
Big Data Day LA 2015 - Deep Learning Human Vocalized Animal Sounds by Sabri S...
Big Data Day LA 2016/ Data Science Track - Decision Making and Lambda Archite...
Big Data Day LA 2016/ Data Science Track - Decision Making and Lambda Archite...
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Introduction to Kafka - Je...
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Introduction to Kafka - Je...
Big Data Day LA 2016/ Big Data Track - Twitter Heron @ Scale - Karthik Ramasa...
Big Data Day LA 2016/ Big Data Track - Twitter Heron @ Scale - Karthik Ramasa...
Similar to Big Data Day LA 2015 - HBase at Factual: Real time and Batch Uses by Molly O'Connor of Factual
Hbase 20141003
This document provides an overview of HBase, including:
- HBase is a distributed, scalable, big data store modeled after Google's BigTable. It provides a fault-tolerant way to store large amounts of sparse data.
- HBase is used by large companies to handle scaling and sparse data better than relational databases. It features automatic partitioning, linear scalability, commodity hardware, and fault tolerance.
- The document discusses HBase operations, schema design best practices, hardware recommendations, alerting, backups and more. It provides guidance on designing keys, column families and cluster configuration to optimize performance for read and write workloads.
Hbase: an introduction
Introduction to HBase. HBase is a NoSQL databases which experienced a tremendous increase in popularity during the last years. Large companies like Facebook, LinkedIn, Foursquare are using HBase. In this presentation we will address questions like: what is HBase?, and compared to relational databases?, what is the architecture?, how does HBase work?, what about the schema design?, what about the IT ressources?. Questions that should help you consider whether this solution might be suitable in your case.
Apache Hadoop India Summit 2011 talk "Searching Information Inside Hadoop Pla...
Apache Hadoop India Summit 2011 talk "Searching Information Inside Hadoop Pla...Yahoo Developer Network
The document discusses different approaches for searching large datasets in Hadoop, including MapReduce, Lucene/Solr, and building a new search engine called HSearch. Some key challenges with existing approaches included slow response times and the need for manual sharding. HSearch indexes data stored in HDFS and HBase. The document outlines several techniques used in HSearch to improve performance, such as using SSDs selectively, reducing HBase table size, distributing queries across region servers, moving processing near data, byte block caching, and configuration tuning. Benchmarks showed HSearch could return results for common words from a 100 million page index within seconds.Facebook keynote-nicolas-qcon
This document summarizes a talk about Facebook's use of HBase for messaging data. It discusses how Facebook migrated data from MySQL to HBase to store metadata, search indexes, and small messages in HBase for improved scalability. It also outlines performance improvements made to HBase, such as for compactions and reads, and future plans such as cross-datacenter replication and running HBase in a multi-tenant environment.
Facebook Messages & HBase
The document discusses Facebook's use of HBase to store messaging data. It provides an overview of HBase, including its data model, performance characteristics, and how it was a good fit for Facebook's needs due to its ability to handle large volumes of data, high write throughput, and efficient random access. It also describes some enhancements Facebook made to HBase to improve availability, stability, and performance. Finally, it briefly mentions Facebook's migration of messaging data from MySQL to their HBase implementation.
支撑Facebook消息处理的h base存储系统
The document discusses Facebook's use of HBase as the database storage engine for its messaging platform. It provides an overview of HBase, including its data model, architecture, and benefits like scalability, fault tolerance, and simpler consistency model compared to relational databases. The document also describes Facebook's contributions to HBase to improve performance, availability, and achieve its goal of zero data loss. It shares Facebook's operational experiences running large HBase clusters and discusses its migration of messaging data from MySQL to a de-normalized schema in HBase.
Apache hadoop hbase
A distributed data store that can scale horizontally to 1,000s of commodity servers and petabytes of indexed storage.
Hadoop - Apache Hbase
HBase is an open-source, non-relational, distributed database built on top of Hadoop and HDFS. It provides BigTable-like capabilities for Hadoop, including fast random reads and writes. HBase stores data in tables comprised of rows, columns, and versions. It is designed to handle large volumes of sparse or unstructured data across clusters of commodity hardware. HBase uses a master-slave architecture with RegionServers storing and serving data and a single active MasterServer managing the cluster metadata and load balancing.
Apache HBase™
The document provides an introduction to NoSQL and HBase. It discusses what NoSQL is, the different types of NoSQL databases, and compares NoSQL to SQL databases. It then focuses on HBase, describing its architecture and components like HMaster, regionservers, Zookeeper. It explains how HBase stores and retrieves data, the write process involving memstores and compaction. It also covers HBase shell commands for creating, inserting, querying and deleting data.
Hw09 Practical HBase Getting The Most From Your H Base Install
The document summarizes two presentations about using HBase as a database. It discusses the speakers' experiences using HBase at Stumbleupon and Streamy to replace MySQL and other relational databases. Some key points covered include how HBase provides scalability, flexibility, and cost benefits over SQL databases for large datasets.
4. hbase overview
HBase is a column-oriented NoSQL database that provides random real-time read/write access to big data stored in Hadoop's HDFS. It is modeled after Google's Bigtable and sits on top of HDFS to allow fast access to large datasets. HBase architecture includes HMaster, HRegionServers, ZooKeeper, and HDFS. HMaster manages metadata and load balancing while HRegionServers serve read/write requests directly from clients. ZooKeeper coordinates the cluster and HDFS provides storage. Data is stored in tables divided into regions hosted by HRegionServers.
Meet Apache HBase - 2.0
HBase 2.0 is the next stable major release for Apache HBase scheduled for early 2017. It is the biggest and most exciting milestone release from the Apache community after 1.0. HBase-2.0 contains a large number of features that is long time in the development, some of which include rewritten region assignment, perf improvements (RPC, rewritten write pipeline, etc), async clients, C++ client, offheaping memstore and other buffers, Spark integration, shading of dependencies as well as a lot of other fixes and stability improvements. We will go into technical details on some of the most important improvements in the release, as well as what are the implications for the users in terms of API and upgrade paths.
Speaker
Ankit Singhal, Member of Technical Staff, Hortonworks
Meet hbase 2.0
HBase 2.0 is the next stable major release for Apache HBase scheduled for early 2017. It is the biggest and most exciting milestone release from the Apache community after 1.0. HBase-2.0 contains a large number of features that is long time in the development, some of which include rewritten region assignment, perf improvements (RPC, rewritten write pipeline, etc), async clients, C++ client, offheaping memstore and other buffers, Spark integration, shading of dependencies as well as a lot of other fixes and stability improvements. We will go into technical details on some of the most important improvements in the release, as well as what are the implications for the users in terms of API and upgrade paths. Existing users of HBase/Phoenix as well as operators managing HBase clusters will benefit the most where they can learn about the new release and the long list of features. We will also briefly cover earlier 1.x release lines and compatibility and upgrade paths for existing users and conclude by giving an outlook on the next level of initiatives for the project.
Meet HBase 2.0
This document summarizes an upcoming presentation on HBase 2.0 and Phoenix 5.0. It discusses recent HBase releases and versioning, changes in HBase 2.0 behavior, and major new features like offheap caching, compacting memstores, and an async client. It also notes that HBase 2.0 is expected by the end of 2017 and provides guidance on testing alpha/beta releases. Phoenix 5.0 will add support for HBase 2.0 and improve its SQL parser, planner, and optimizer using Apache Calcite.
Hbase Quick Review Guide for Interviews
This document provides a quick guide to refresh skills on HBase architecture and concepts. It discusses HBase's limitations in satisfying the CAP theorem, its architecture components including the HMaster, Region Servers and Zookeeper. It also covers best practices for row key design, and differences between minor and major compactions. The HColumnDescriptor class and HBase catalog tables -.META. and -ROOT- are also summarized.
Apache HBase 1.0 Release
An overview of the state of the HBase 1.0 release. Covers a quick HBase overview, the HBase timeline, new features for 1.0, and the upgrade path.
01 hbase
HBase is a distributed, column-oriented database built on top of HDFS that can handle large datasets across a cluster. It uses a map-reduce model where data is stored as multidimensional sorted maps across nodes. Data is first written to a write-ahead log and memory, then flushed to disk files and compacted for efficiency. Client applications access HBase programmatically through APIs rather than SQL. Map-reduce jobs on HBase use input, mapper, reducer, and output classes to process table data in parallel across regions.
HBase at Bloomberg: High Availability Needs for the Financial Industry
Speaker: Sudarshan Kadambi and Matthew Hunt (Bloomberg LP)
Bloomberg is a financial data and analytics provider, so data management is core to what we do. There's tremendous diversity in the type of data we manage, and HBase is a natural fit for many of these datasets - from the perspective of the data model as well as in terms of a scalable, distributed database. This talk covers data and analytics use cases at Bloomberg and operational challenges around HA. We'll explore the work currently being done under HBASE-10070, further extensions to it, and how this solution is qualitatively different to how failover is handled by Apache Cassandra.
Hive integration: HBase and Rcfile__HadoopSummit2010
Hadoop Summit 2010 - Developers Track
Hive integration: HBase and Rcfile
John Sichi and Yongqiang He, Facebook
HBase Applications - Atlanta HUG - May 2014
HBase is good a various workloads, ranging from sequential range scans to purely random access. These access patterns can be translated into application types, usually falling into two major groups: entities and events. This presentation discussed the underlying implications and how to approach those use-cases. Examples taken from Facebook show how this has been tackled in real life.
Similar to Big Data Day LA 2015 - HBase at Factual: Real time and Batch Uses by Molly O'Connor of Factual (20)
Apache Hadoop India Summit 2011 talk "Searching Information Inside Hadoop Pla...
Apache Hadoop India Summit 2011 talk "Searching Information Inside Hadoop Pla...
Hw09 Practical HBase Getting The Most From Your H Base Install
Hw09 Practical HBase Getting The Most From Your H Base Install
HBase at Bloomberg: High Availability Needs for the Financial Industry
HBase at Bloomberg: High Availability Needs for the Financial Industry
Hive integration: HBase and Rcfile__HadoopSummit2010
Hive integration: HBase and Rcfile__HadoopSummit2010
More from Data Con LA
Data Con LA 2022 Keynotes
1. LAUSD has been developing its enterprise data and reporting capabilities since 2000, with various systems and dashboards launched over the years to provide different types of data and reporting, including student outcomes and achievement reports, individual student records, and teacher/staff data.
2. Current tools include MyData (with over 20 million student records), GetData (with instructional and business data), Whole Child (with academic and wellness data), OpenData, and Executive Dashboards.
3. Upcoming improvements include dashboards for social-emotional learning, physical education, and tools to support the Intensive Diagnostic Education Centers and Black Student Achievement Plan initiatives.
Data Con LA 2022 Keynotes
The document discusses the County of Los Angeles' efforts to better coordinate services across various departments by creating an enterprise data platform. It notes that the county serves over 750,000 patients annually through its health systems and oversees many other services related to homelessness, justice, child welfare, and public health. The proposed data platform would create a unified client identifier and data store to integrate client records across departments in order to generate insights, measure outcomes, and improve coordination of services.
Data Con LA 2022 Keynote
Fastly is an edge cloud platform provider that aims to upgrade the internet experience by making applications and digital experiences fast, engaging, and secure. It has a global network of 100+ points of presence across 30+ countries serving over 1 trillion daily requests. The presentation discusses how internet requests are handled traditionally versus more modern approaches using an edge cloud platform like Fastly. It emphasizes that the edge must be programmable, deliver general purpose compute anywhere, and provide high reliability, security, and data privacy by default.
Data Con LA 2022 - Startup Showcase
The document summarizes how Aware Health can save self-insured employers millions of dollars by reducing unnecessary surgeries, imaging, and lost work time for musculoskeletal conditions. It notes that 95% of common spine, wrist, and other surgeries are no more effective than non-surgical treatments. Aware Health uses diagnosis without imaging to prevent chronic pain and has shown real-world savings of $9.78 to $78.66 per member per month for employers, a 96% net promoter score, and over $2 million in annual savings for one enterprise customer.
Data Con LA 2022 Keynote
- Project Lightspeed is the next generation of Apache Spark Structured Streaming that aims to provide faster and simpler stream processing with predictable low latency.
- It targets reducing tail latency by up to 2x through faster bookkeeping and offset management. It also enhances functionality with advanced capabilities like new operators and easy to use APIs.
- Project Lightspeed also aims to simplify deployment, operations, monitoring and troubleshooting of streaming applications. It seeks to improve ecosystem support for connectors, authentication and authorization.
- Some specific improvements include faster micro-batch processing, enhancing Python as a first class citizen, and making debugging of streaming jobs easier through visualizations.
Data Con LA 2022 - Using Google trends data to build product recommendations
Mike Limcaco, Analytics Specialist / Customer Engineer at Google
Measure trends in a particular topic or search term across Google Search across the US down to the city-level. Integrate these data signals into analytic pipelines to drive product, retail, media (video, audio, digital content) recommendations tailored to your audience segment. We'll discuss how Google unique datasets can be used with Google Cloud smart analytic services to process, enrich and surface the most relevant product or content that matches the ever-changing interests of your local customer segment.
Data Con LA 2022 - AI Ethics
Melinda Thielbar, Data Science Practice Lead and Director of Data Science at Fidelity Investments
From corporations to governments to private individuals, most of the AI community has recognized the growing need to incorporate ethics into the development and maintenance of AI models. Much of the current discussion, though, is meant for leaders and managers. This talk is directed to data scientists, data engineers, ML Ops specialists, and anyone else who is responsible for the hands-on, day-to-day of work building, productionalizing, and maintaining AI models. We'll give a short overview of the business case for why technical AI expertise is critical to developing an AI Ethics strategy. Then we'll discuss the technical problems that cause AI models to behave unethically, how to detect problems at all phases of model development, and the tools and techniques that are available to support technical teams in Ethical AI development.
Data Con LA 2022 - Improving disaster response with machine learning
Antje Barth, Principal Developer Advocate, AI/ML at AWS & Chris Fregly, Principal Engineer, AI & ML at AWS
The frequency and severity of natural disasters are increasing. In response, governments, businesses, nonprofits, and international organizations are placing more emphasis on disaster preparedness and response. Many organizations are accelerating their efforts to make their data publicly available for others to use. Repositories such as the Registry of Open Data on AWS and Humanitarian Data Exchange contain troves of data available for use by developers, data scientists, and machine learning practitioners. In this session, see how a community of developers came together though the AWS Disaster Response hackathon to build models to support natural disaster preparedness and response.
Data Con LA 2022 - What's new with MongoDB 6.0 and Atlas
Sig Narvaez, Executive Solution Architect at MongoDB
MongoDB is now a Developer Data Platform. Come learn what�s new in the 6.0 release and Atlas following all the recent announcements made at MongoDB World 2022. Topics will include
- Atlas Search which combines 3 systems into one (database, search engine, and sync mechanisms) letting you focus on your product's differentiation.
- Atlas Data Federation to seamlessly query, transform, and aggregate data from one or more MongoDB Atlas databases, Atlas Data Lake and AWS S3 buckets
- Queryable Encryption lets you run expressive queries on fully randomized encrypted data to meet the most stringent security requirements
- Relational Migrator which analyzes your existing relational schemas and helps you design a new MongoDB schema.
- And more!
Data Con LA 2022 - Real world consumer segmentation
Jaysen Gillespie, Head of Analytics and Data Science at RTB House
1. Shopkick has over 30M downloads, but the userbase is very heterogeneous. Anecdotal evidence indicated a wide variety of users for whom the app holds long-term appeal.
2. Marketing and other teams challenged Analytics to get beyond basic summary statistics and develop a holistic segmentation of the userbase.
3. Shopkick's data science team used SQL and python to gather data, clean data, and then perform a data-driven segmentation using a k-means algorithm.
4. Interpreting the results is more work -- and more fun -- than running the algo itself. We'll discuss how we transform from ""segment 1"", ""segment 2"", etc. to something that non-analytics users (Marketing, Operations, etc.) could actually benefit from.
5. So what? How did team across Shopkick change their approach given what Analytics had discovered.
Data Con LA 2022 - Modernizing Analytics & AI for today's needs: Intuit Turbo...
Ravi Pillala, Chief Data Architect & Distinguished Engineer at Intuit
TurboTax is one of the well known consumer software brand which at its peak serves 385K+ concurrent users. In this session, We start with looking at how user behavioral data & tax domain events are captured in real time using the event bus and analyzed to drive real time personalization with various TurboTax data pipelines. We will also look at solutions performing analytics which make use of these events, with the help of Kafka, Apache Flink, Apache Beam, Spark, Amazon S3, Amazon EMR, Redshift, Athena and Amazon lambda functions. Finally, we look at how SageMaker is used to create the TurboTax model to predict if a customer is at risk or needs help.
Data Con LA 2022 - Moving Data at Scale to AWS
George Mansoor, Chief Information Systems Officer at California State University
Overview of the CSU Data Architecture on moving on-prem ERP data to the AWS Cloud at scale using Delphix for Data Replication/Virtualization and AWS Data Migration Service (DMS) for data extracts
Data Con LA 2022 - Collaborative Data Exploration using Conversational AI
Anand Ranganathan, Chief AI Officer at Unscrambl
Conversational AI is getting more and more widely used for customer support and employee support use-cases. In this session, I'm going to talk about how it can be extended for data analysis and data science use-cases ... i.e., how users can interact with a bot to ask analytical questions on data in relational databases.
This allows users to explore complex datasets using a combination of text and voice questions, in natural language, and then get back results in a combination of natural language and visualizations. Furthermore, it allows collaborative exploration of data by a group of users in a channel in platforms like Microsoft Teams, Slack or Google Chat.
For example, a group of users in a channel can ask questions to a bot in plain English like ""How many cases of Covid were there in the last 2 months by state and gender"" or ""Why did the number of deaths from Covid increase in May 2022"", and jointly look at the results that come back. This facilitates data awareness, data-driven collaboration and joint decision making among teams in enterprises and outside.
In this talk, I'll describe how we can bring together various features including natural-language understanding, NL-to-SQL translation, dialog management, data story-telling, semantic modeling of data and augmented analytics to facilitate collaborate exploration of data using conversational AI.
Data Con LA 2022 - Why Database Modernization Makes Your Data Decisions More ...
Anil Inamdar, VP & Head of Data Solutions at Instaclustr
The most modernized enterprises utilize polyglot architecture, applying the best-suited database technologies to each of their organization's particular use cases. To successfully implement such an architecture, though, you need a thorough knowledge of the expansive NoSQL data technologies now available.
Attendees of this Data Con LA presentation will come away with:
-- A solid understanding of the decision-making process that should go into vetting NoSQL technologies and how to plan out their data modernization initiatives and migrations.
-- They will learn the types of functionality that best match the strengths of NoSQL key-value stores, graph databases, columnar databases, document-type databases, time-series databases, and more.
-- Attendees will also understand how to navigate database technology licensing concerns, and to recognize the types of vendors they'll encounter across the NoSQL ecosystem. This includes sniffing out open-core vendors that may advertise as “open source,"" but are driven by a business model that hinges on achieving proprietary lock-in.
-- Attendees will also learn to determine if vendors offer open-code solutions that apply restrictive licensing, or if they support true open source technologies like Hadoop, Cassandra, Kafka, OpenSearch, Redis, Spark, and many more that offer total portability and true freedom of use.
Data Con LA 2022 - Intro to Data Science
Zia Khan, Computer Systems Analyst and Data Scientist at LearningFuze
Data Science tutorial is designed for people who are new to Data Science. This is a beginner level session so no prior coding or technical knowledge is required. Just bring your laptop with WiFi capability. The session starts with a review of what is data science, the amount of data we generate and how companies are using that data to get insight. We will pick a business use case, define the data science process, followed by hands-on lab using python and Jupyter notebook. During the hands-on portion we will work with pandas, numpy, matplotlib and sklearn modules and use a machine learning algorithm to approach the business use case.
Data Con LA 2022 - How are NFTs and DeFi Changing Entertainment
Mariana Danilovic, Managing Director at Infiom, LLC
We will address:
(1) Community creation and engagement using tokens and NFTs
(2) Organization of DAO structures and ways to incentivize Web3 communities
(3) DeFi business models applied to Web3 ventures
(4) Why Metaverse matters for new entertainment and community engagement models.
Data Con LA 2022 - Why Data Quality vigilance requires an End-to-End, Automat...
Curtis ODell, Global Director Data Integrity at Tricentis
Join me to learn about a new end-to-end data testing approach designed for modern data pipelines that fills dangerous gaps left by traditional data management tools—one designed to handle structured and unstructured data from any source. You'll hear how you can use unique automation technology to reach up to 90 percent test coverage rates and deliver trustworthy analytical and operational data at scale. Several real world use cases from major banks/finance, insurance, health analytics, and Snowflake examples will be presented.
Key Learning Objective
1. Data journeys are complex and you have to ensure integrity of the data end to end across this journey from source to end reporting for compliance
2. Data Management tools do not test data, they profile and monitor at best, and leave serious gaps in your data testing coverage
3. Automation with integration to DevOps and DataOps' CI/CD processes are key to solving this.
4. How this approach has impact in your vertical
Data Con LA 2022-Perfect Viral Ad prediction of Superbowl 2022 using Tease, T...
1. The document discusses methods for predicting and engineering viral Super Bowl ads, including a panel-based analysis of video content characteristics and a deep learning model measuring social media effects.
2. It provides examples of ads from Super Bowl 2022 that scored well using these methods, such as BMW and Budweiser ads, and compares predicted viral rankings to actual results.
3. The document also demonstrates how to systematically test, tweak, and target an ad campaign like Bajaj Pulsar's to increase virality through modifications to title, thumbnail, tags and content based on audience feedback.
Data Con LA 2022- Embedding medical journeys with machine learning to improve...
Jai Bansal, Senior Manager, Data Science at Aetna
This talk describes an internal data product called Member Embeddings that facilitates modeling of member medical journeys with machine learning.
Medical claims are the key data source we use to understand health journeys at Aetna. Claims are the data artifacts that result from our members' interactions with the healthcare system. Claims contain data like the amount the provider billed, the place of service, and provider specialty. The primary medical information in a claim is represented in codes that indicate the diagnoses, procedures, or drugs for which a member was billed. These codes give us a semi-structured view into the medical reason for each claim and so contain rich information about members' health journeys. However, since the codes themselves are categorical and high-dimensional (10K cardinality), it's challenging to extract insight or predictive power directly from the raw codes on a claim.
To transform claim codes into a more useful format for machine learning, we turned to the concept of embeddings. Word embeddings are widely used in natural language processing to provide numeric vector representations of individual words.
We use a similar approach with our claims data. We treat each claim code as a word or token and use embedding algorithms to learn lower-dimensional vector representations that preserve the original high-dimensional semantic meaning.
This process converts the categorical features into dense numeric representations. In our case, we use sequences of anonymized member claim diagnosis, procedure, and drug codes as training data. We tested a variety of algorithms to learn embeddings for each type of claim code.
We found that the trained embeddings showed relationships between codes that were reasonable from the point of view of subject matter experts. In addition, using the embeddings to predict future healthcare-related events outperformed other basic features, making this tool an easy way to improve predictive model performance and save data scientist time.
Data Con LA 2022 - Data Streaming with Kafka
Jie Chen, Manager Advisory, KPMG
Data is the new oil. However, many organizations have fragmented data in siloed line of businesses. In this topic, we will focus on identifying the legacy patterns and their limitations and introducing the new patterns packed by Kafka's core design ideas. The goal is to tirelessly pursue better solutions for organizations to overcome the bottleneck in data pipelines and modernize the digital assets for ready to scale their businesses. In summary, we will walk through three uses cases, recommend Dos and Donts, Take aways for Data Engineers, Data Scientist, Data architect in developing forefront data oriented skills.
More from Data Con LA (20)
Data Con LA 2022 - Using Google trends data to build product recommendations
Data Con LA 2022 - Using Google trends data to build product recommendations
Data Con LA 2022 - Improving disaster response with machine learning
Data Con LA 2022 - Improving disaster response with machine learning
Data Con LA 2022 - What's new with MongoDB 6.0 and Atlas
Data Con LA 2022 - What's new with MongoDB 6.0 and Atlas
Data Con LA 2022 - Real world consumer segmentation
Data Con LA 2022 - Real world consumer segmentation
Data Con LA 2022 - Modernizing Analytics & AI for today's needs: Intuit Turbo...
Data Con LA 2022 - Modernizing Analytics & AI for today's needs: Intuit Turbo...
Data Con LA 2022 - Collaborative Data Exploration using Conversational AI
Data Con LA 2022 - Collaborative Data Exploration using Conversational AI
Data Con LA 2022 - Why Database Modernization Makes Your Data Decisions More ...
Data Con LA 2022 - Why Database Modernization Makes Your Data Decisions More ...
Data Con LA 2022 - How are NFTs and DeFi Changing Entertainment
Data Con LA 2022 - How are NFTs and DeFi Changing Entertainment
Data Con LA 2022 - Why Data Quality vigilance requires an End-to-End, Automat...
Data Con LA 2022 - Why Data Quality vigilance requires an End-to-End, Automat...
Data Con LA 2022-Perfect Viral Ad prediction of Superbowl 2022 using Tease, T...
Data Con LA 2022-Perfect Viral Ad prediction of Superbowl 2022 using Tease, T...
Data Con LA 2022- Embedding medical journeys with machine learning to improve...
Data Con LA 2022- Embedding medical journeys with machine learning to improve...
Recently uploaded
Essentials of Automations: The Art of Triggers and Actions in FME
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
Climate Impact of Software Testing at Nordic Testing Days
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
Securing your Kubernetes cluster_ a step-by-step guide to success !
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
20 Comprehensive Checklist of Designing and Developing a Website
Dive into the world of Website Designing and Developing with Pixlogix! Looking to create a stunning online presence? Look no further! Our comprehensive checklist covers everything you need to know to craft a website that stands out. From user-friendly design to seamless functionality, we've got you covered. Don't miss out on this invaluable resource! Check out our checklist now at Pixlogix and start your journey towards a captivating online presence today.
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
ここ3000字までしか入らないけどタイトルの方がたくさん文字入ると思います。
UiPath Test Automation using UiPath Test Suite series, part 5
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
ABSTRACT: A prima vista, un mattoncino Lego e la backdoor XZ potrebbero avere in comune il fatto di essere entrambi blocchi di costruzione, o dipendenze di progetti creativi e software. La realtà è che un mattoncino Lego e il caso della backdoor XZ hanno molto di più di tutto ciò in comune.
Partecipate alla presentazione per immergervi in una storia di interoperabilità, standard e formati aperti, per poi discutere del ruolo importante che i contributori hanno in una comunità open source sostenibile.
BIO: Sostenitrice del software libero e dei formati standard e aperti. È stata un membro attivo dei progetti Fedora e openSUSE e ha co-fondato l'Associazione LibreItalia dove è stata coinvolta in diversi eventi, migrazioni e formazione relativi a LibreOffice. In precedenza ha lavorato a migrazioni e corsi di formazione su LibreOffice per diverse amministrazioni pubbliche e privati. Da gennaio 2020 lavora in SUSE come Software Release Engineer per Uyuni e SUSE Manager e quando non segue la sua passione per i computer e per Geeko coltiva la sua curiosità per l'astronomia (da cui deriva il suo nickname deneb_alpha).
Presentation of the OECD Artificial Intelligence Review of Germany
Consult the full report at https://www.oecd.org/digital/oecd-artificial-intelligence-review-of-germany-609808d6-en.htm
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
Full-RAG: A modern architecture for hyper-personalization
Mike Del Balso, CEO & Co-Founder at Tecton, presents "Full RAG," a novel approach to AI recommendation systems, aiming to push beyond the limitations of traditional models through a deep integration of contextual insights and real-time data, leveraging the Retrieval-Augmented Generation architecture. This talk will outline Full RAG's potential to significantly enhance personalization, address engineering challenges such as data management and model training, and introduce data enrichment with reranking as a key solution. Attendees will gain crucial insights into the importance of hyperpersonalization in AI, the capabilities of Full RAG for advanced personalization, and strategies for managing complex data integrations for deploying cutting-edge AI solutions.
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
Join us to introduce Milvus Lite, a vector database that can run on notebooks and laptops, share the same API with Milvus, and integrate with every popular GenAI framework. This webinar is perfect for developers seeking easy-to-use, well-integrated vector databases for their GenAI apps.
How to Get CNIC Information System with Paksim Ga.pptx
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
Removing Uninteresting Bytes in Software Fuzzing
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...Edge AI and Vision Alliance
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/building-and-scaling-ai-applications-with-the-nx-ai-manager-a-presentation-from-network-optix/
Robin van Emden, Senior Director of Data Science at Network Optix, presents the “Building and Scaling AI Applications with the Nx AI Manager,” tutorial at the May 2024 Embedded Vision Summit.
In this presentation, van Emden covers the basics of scaling edge AI solutions using the Nx tool kit. He emphasizes the process of developing AI models and deploying them globally. He also showcases the conversion of AI models and the creation of effective edge AI pipelines, with a focus on pre-processing, model conversion, selecting the appropriate inference engine for the target hardware and post-processing.
van Emden shows how Nx can simplify the developer’s life and facilitate a rapid transition from concept to production-ready applications.He provides valuable insights into developing scalable and efficient edge AI solutions, with a strong focus on practical implementation.Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
As the digital landscape continually evolves, operating systems play a critical role in shaping user experiences and productivity. The launch of Nitrux Linux 3.5.0 marks a significant milestone, offering a robust alternative to traditional systems such as Windows 11. This article delves into the essence of Nitrux Linux 3.5.0, exploring its unique features, advantages, and how it stands as a compelling choice for both casual users and tech enthusiasts.
Recently uploaded (20)
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Big Data Day LA 2015 - HBase at Factual: Real time and Batch Uses by Molly O'Connor of Factual
- 1. HBase and Batch Processing Molly O’Connor Factual
- 2. HBase at Factual Support API for global location queries and accept live writes of new supporting data Batch updates: ingesting large amounts of new data, pushing out new versions of the data (improvements in algorithms for data cleaning, verification, clustering)
- 3. Overview 1. HBase Architecture 2. Batch Workflow 3. Lessons/Challenges
- 4. HBase Intro -- Data Model ● Column families for HBase table are specified at creation time ● Arbitrary byte sequences for column qualifiers (unlimited and created as data is written) ● Data organized by column families and sorted by key
- 5. HBase Intro -- HFile Format ● Sorted lexicographically with secondary indices inline with data ● Block size: memory tradeoffs. Choose based on expected read access ● Compression: experiment: lzo, snappy, gz ● Index Size: don’t make keys and column names longer than needed.
- 6. HBase Intro -- Region Servers
- 7. HBase Intro -- Locality ● Region servers write new data locally ● Compaction further promotes data locality ● Metrics: at the region server level. In 1.0, “Block Locality”, in 0.94, “hdfsBlocksLocalityIndex” ● Enable short circuit reads for additional benefits
- 8. HBase Intro -- Consistency ● Single row atomicity across column families guaranteed ● checkAndPut -- single row, checks on the value of a single column only ● mutateRowsWithLocks --via coprocessor ○ within a region: need clever row key design, region split policy
- 9. Overview 1. HBase Architecture 2. Batch Workflow 3. Lessons/Challenges
- 10. ● Better performance for large scale updates ● Quality analysis and metrics on all data before adopting ● Perform computations that are not possible or prohibitively expensive in a live hbase setting ● Data is already on HDFS HBase and Batch
- 11. Batch Workflow 1. Snapshot 2. MapReduce 3. Bulkloading
- 12. HBase Snapshots ● Copy on write (HFile links) ● Per table ● Rolling ○ HBase’s guarantee of consistency within a row ● Use cases: backup/recovery, export, mapreduce
- 13. Snapshots and MapReduce ● Definitely use Mapreduce over snapshots, if possible (HBASE-8369) ○ before this feature, issues with reading HFiles directly because of compaction ○ Advantages: Job is faster and puts less pressure on region servers ○ Caveat: Not reading live data
- 14. Locality and Mapreduce Tradeoffs: want to colocate computation with data. But, this causes contention with HBase ○ Don’t run mapreduce on HBase nodes? ○ Mitigated somewhat with Yarn?
- 15. Federated Namespaces for Multi-Purpose Cluster
- 16. Bulkloading ● Additional path to ingesting data with HBase. Create HFiles directly, and HBase adopts the files ● Bulkload is atomic at the region level ○ single row consistency (across CF) guaranteed
- 17. Locality after Bulkloading ● Look at current region locations, and try to produce new HFiles on those nodes ● Compaction after bulkloading: needs to be timed well, but can eventually lead to locality ● New data will not be in the block cache! ● HBASE-11195 can promote compaction in cases where locality is low ● HBASE-8329 throttling compaction speed
- 18. Overview 1. HBase Architecture 2. Batch Workflow 3. Lessons/Challenges
- 19. Challenges ● Does bulkloading fit your data model? ○ Replay--do you need a catchup phase after data ingestion? ● Consistency beyond row level (using a library to manage a secondary index or other transactional writes)? ● Maybe use Mapreduce over live tables but throttle requests? HBASE-11598
- 20. Summary 1. Ability to do bulk updates can be hugely important for performance 2. HDFS integration and strong feature set make HBase a good choice for batch processing 3. More features coming (today highlighted many introduced in the new version 1.0)
- 21. Questions?