HBaseCon 2012 | Developing Real Time Analytics Applications Using HBase in the Cloud - Rick Tucker, Sproxil

•Download as PPTX, PDF•

1 like•3,357 views

As small companies are adapting to handle Big Data, the cloud and HBase enable developers to leverage that data to provide revenue-generating real time applications. When developing a real time application for an existing system, one must balance incrementing counters in real time with Map Reduce jobs over the same data-set. When maintaining an analytics platform, ensuring data accuracy is essential. At Sproxil, SMS logs are ingested into HBase at a growing rate and we report metrics such as SMS throughput, unique user growth over time, and return SMS user activity in real time. Sproxil provides a versatile analytics application enabling customers to handpick statistics on demand to gain market insights enabling them react quickly to trends. This talk will identify the most profitable metrics and demonstrate how to calculate them using Map Reduce while continually updating data as it arrives.

Technology Business

Developing Real Time Analytics
Applications Using HBase in the Cloud

May 22, 2012
Rick Tucker
tech@sproxil.com

tech@sproxil.com May 22,2012 © 2012 Sproxil, Inc.

Why HBase?

USER SENDS TEXT MESSAGE CALCULATE
TEXT MESSAGE IS PROCESSED ANALYTICS

USER Amazon EC2
RECEIVES Cloud
REPLY

tech@sproxil.com May 22,2012 © 2012 Sproxil, Inc.

Real-Time Analytics Engine
• MapReduce too slow to maintain data in true real time

• As data arrives, analytical data is updated through
counters

Text Message Message Increment
Arrives Analyzed Counters

Genuine Product +1 Increment Counter for
Authentication Genuine Authentications

Repeat Customer +1 Increment Counter for
Repeat Customers

tech@sproxil.com May 22,2012 © 2012 Sproxil, Inc.

Schema Design: Example 1

• Example: View log of text messages in
chronological order
• Rowkey: row prefix + timestamp

Row
transaction 2012-05-22 12:00:00
transaction 2012-05-22 12:01:14
transaction 2012-05-22 12:02:03

Note: HBase sorts rowkeys lexicographically so scans return data in reverse
chronological order
tech@sproxil.com May 22,2012 © 2012 Sproxil, Inc. 5

•
•

Row
transaction userID 1 2012-05-22 12:00:00
transaction userID 1 2012-05-22 12:01:14
transaction userID 2 2012-05-22 12:00:54
transaction userID 2 2012-05-22 12:01:22
transaction userID 2 2012-05-22 12:02:01
Note: Hbase sorts rows lexicographically so scans return data in reverse
chronological order

tech@sproxil.com May 22,2012 © 2012 Sproxil, Inc.

Critical Findings
• Schema design is crucial for successful HBase
implementation
– Pack as much info as possible into row keys

• Use caution with Filters
– E.g. Regex filters can be costly
– Alternatives:
• Directly query for data you need
• Use efficient filters when filtering large data sets

tech@sproxil.com May 22,2012 © 2012 Sproxil, Inc.

This document summarizes an industrial internet case study using HBase and TSDB for time series data storage and analytics. It describes an aviation use case involving jet engine sensor data collection and analysis to detect problems and reduce downtime. The system ingests large volumes of sensor data from aircraft into an industrial data lake architecture using HBase for storage and TSDB and SQL interfaces for analytics. Performance tests showed the horizontal data model of storing each flight parameter as a row performed better than a vertical model for retrieval from HBase.

HBaseCon 2012 | Real-Time and Batch HBase for Healthcare at Explorys

Cloudera, Inc.

NoSQL and Spatial Database Capabilities using PostgreSQL

EDB

PostgreSQL is an object-relational database system. NoSQL on the other hand is a non-relational database and is document-oriented. Learn how the PostgreSQL database gives one the flexible options to combine NoSQL workloads with the relational query power by offering JSON data types. With PostgreSQL, new capabilities can be developed and plugged into the database as required. Attend this webinar to learn: - The new features and capabilities in PostgreSQL for new workloads, requiring greater flexibility in the data model - NoSQL with JSON, Hstore and its performance and features for enterprises - Spatial SQL - advanced features in PostGIS application with PostGIS extension

High-Scale Entity Resolution in Hadoop

DataWorks Summit/Hadoop Summit

eBay maintains hundreds of millions of accounts across its properties that are unstructured and in different formats. Identifying which accounts belong to the same person enables eBay to personalize customer experiences, provide customer service, and fight fraud. MapReduce provides a robust design pattern to simplify high-scale entity resolution through parallelized modular operations, including linking accounts pairwise, identifying connected components through iterative MapReduce jobs, and validating the results.

HBaseCon 2013: Real-Time Model Scoring in Recommender Systems

Cloudera, Inc.

This document describes a real-time product recommendation system called KijiShopping that uses content-based modeling with TF-IDF. It explains how KijiShopping collects user and product data, computes TF-IDF to find useful features, associates words with products using batch MapReduce jobs, determines a user's preferred words, and generates recommendations by combining user ratings and models using producers that access models via key-value stores. The goal is to provide real-time recommendations by leveraging the Apache Kiji framework for real-time analytics.

Insights into Real World Data Management Challenges

DataWorks Summit

Data is your most valuable business asset and it's also your biggest challenge. This challenge and opportunity means we continually face significant road blocks toward becoming a data driven organisation. From the management of data, to the bubbling open source frameworks, the limited industry skills to surmounting time and cost pressures, our challenge in data is big. We all want and need a “fit for purpose” approach to management of data, especially Big Data, and overcoming the ongoing challenges around the ‘3Vs’ means we get to focus on the most important V - ‘Value’.Come along and join the discussion on how Oracle Big Data Cloud provides Value in the management of data and supports your move toward becoming a data driven organisation. Speaker Noble Raveendran, Principal Consultant, Oracle

Free Servers to Build Big Data System on: Bing’s Approach

DataWorks Summit

A couple of thousands of servers for a big data system is also a big investment. Microsoft Bing has figured out a way to fulfill our needs without signing a huge check. We have the technology to harvest spare cycles on underutilized servers. And we tweak the configurations in Hadoop and Spark to fit the flexible capacity base. We have saved hundreds of millions of dollars per year. Bing is adopting open source big data technologies for our offline data processing system. It requires a massive amount of capacity, which implies a significant bill. With collaboration with Windows and Azure, Research teams, we can harvest most of the needed capacity from our existing server fleet. We make use of the capacity on reserve servers while keeping them instantly available for emergency use; we allocate compute and storage to servers when they are not fully occupied. We updated Hadoop node decommission, HDFS block placement, YARN node label mapping, and a few other policies so that they can adapt to the capacity that is even less reliable than commodity servers. We brought open source capacity to Bing product with less than 1 percent of the cost we had done it through normal approach. We also extend the YARN and Spark framework to better fit the need of deep learning training and inferencing workloads in our system. This extension is equipping Bing with direct questioning and answering type of interactive query features. Big does not mean expensive. The audience can learn about the approach from Bing that they can make better use of their existing servers to do additional big data systems.

In Search of Database Nirvana: Challenges of Delivering HTAP

HBaseCon

This document discusses the challenges of delivering Hybrid Transactional and Analytical Processing (HTAP) workloads from a single database. Some key challenges include: - Having a single query engine that can efficiently support both transactional and analytical workloads with different data structures, statistics needs, query types, etc. - Supporting multiple different storage engines for things like transactions, analytics, and mixed workloads. - Accommodating different data models optimized for transactions versus analytics within a single data model. - Providing enterprise capabilities like high availability, security, and manageability for hybrid transactional and analytical systems.

Azure Data Lakes allow for storing and analyzing large amounts of data from multiple sources using frameworks like HDInsight, Spark, and machine learning. Data is stored in Azure Data Lakes Store using WebHDFS in 2GB chunks called extents that are replicated three times for availability and reliability. Azure Data Lake Storage Gen 2 adds additional features from Azure Blob storage like fault tolerance, high availability, and lower costs. Data lakes help companies gain a unified view of data to improve analysis and act on business opportunities faster.

Hadoop data access layer v4.0

SpringPeople

To transform your organization and unlock the value of your data, you need a way to ingest, store and analyze every type of data in your organization. This presentation covers the Data Access Layer of the Hadoop Ecosystem which enables you to achieve this. We will use the HDP (Hortonworks Data Platform) reference architecture to walk through the Hadoop core and its ecosystem with focus on the data access layer. We will cover some of the prominent tools of the ecosystem such as Pig, Hive, Sqoop, Flume and Oozie and how they are used for ingesting data into Hadoop from structured, unstructured and streaming sources. Talk to us at +91 80 6567 9700 or send an email to training@springpeople.com for more information.

Querying Druid in SQL with Superset

DataWorks Summit

Druid is a high performance, column-oriented distributed data store that is widely used at Oath for big data analysis. Druid has a JSON schema as its query language, making it difficult for new users unfamiliar with the schema to start querying Druid quickly. The JSON schema is designed to work with the data ingestion methods of Druid, so it can provide high performance features such as data aggregations in JSON, but many are unable to utilize such features, because they not familiar with the specifics of how to optimize Druid queries. However, most new Druid users at Yahoo are already very familiar with SQL, and the queries they want to write for Druid can be converted to concise SQL. We found that our data analysts wanted an easy way to issue ad-hoc Druid queries and view the results in a BI tool in a way that's presentable to nontechnical stakeholders. In order to achieve this, we had to bridge the gap between Druid, SQL, and our BI tools such as Apache Superset. In this talk, we will explore different ways to query a Druid datasource in SQL and discuss which methods were most appropriate for our use cases. We will also discuss our open source contributions so others can utilize our work. GURUGANESH KOTTA, Software Dev Eng, Oath and JUNXIAN WU, Software Engineer, Oath Inc.

Data Warehouse Modernization - Big Data in the Cloud Success with Qubole on O...

Qubole

The effective use of big data is the key to gaining a competitive advantage and outperforming the competition. This change demands that companies consume and blend enormous amount of data created from divergent and inherently mismatched sources, which represents a paradigm shift to the traditional data warehouse. Companies need to modernize their data warehouse, augmenting it with a platform that allows storage, processing, exploration and analysis of large and diverse datasets without limiting the ability to deliver the data access, and flexibility responding to the needs of the business. That’s where Oracle Cloud and Qubole work together delivering a new breed of data platform —capable of storing and processing the overwhelming amount of data that on-premises big data deployments cannot handle. Watch this on-demand webinar to understand: - Why deploying big data on-premises is expensive, complex to maintain and limits your ability to scale across new use cases and data sources - How Oracle Bare Metal Cloud's predictable and fast performance compute and network services deliver the foundation of a cost-effective, high-performance big data platform - How Qubole leverages Oracle Bare Metal Cloud to provide a turnkey big data service that optimizes cost, performance, and scale, enabling self-service data exploration. Qubole delivers a cloud-based, turnkey, self-service big data service that removes the complexity and reduces the cost of doing big data. It leverages Oracle Bare Metal Cloud’s next generation of scalable, inexpensive and performant compute, network and storage public cloud infrastructure to provide a solution that accelerates time to market and reduces the risk of your big data initiatives.

Built-In Security for the Cloud

DataWorks Summit

Today enterprises desire to move more and more of their data lakes to the cloud to help them execute faster, increase productivity, drive innovation while leveraging the scale and flexibility of the cloud. However, such gains come with risks and challenges in the areas of data security, privacy, and governance. In this talk we cover how enterprises can overcome governance and security obstacles to leverage these new advances that the cloud can provide to ease the management of their data lakes in the cloud. We will also show how the enterprise can have consistent governance and security controls in the cloud for their ephemeral analytic workloads in a multi-cluster cloud environment without sacrificing any of the data security and privacy/compliance needs that their business context demands. Additionally, we will outline some use cases and patterns as well as best practices to rationally manage such a multi-cluster data lake infrastructure in the cloud. Speaker: Jeff Sposetti, Product Management, Hortonworks

HTAP Queries

Atif Shaikh

The document discusses HTAP (Hybrid Transactional/Analytical Processing), data fabrics, and key PostgreSQL features that enable data fabrics. It describes HTAP as addressing resource contention by allowing mixed workloads on the same system and analytics on inflight transactional data. Data fabrics are defined as providing a logical unified data model, distributed cache, query federation, and semantic normalization across an enterprise data fabric cluster. Key PostgreSQL features that support data fabrics include its schema store, distributed cache, query federation, optimization, and normalization capabilities as well as foreign data wrappers.

Ravi Namboori 's Open stack framework introduction

Ravi namboori

OpenStack is an open source cloud computing platform that provides services for managing compute, storage, and networking resources in a data center. It includes core projects like Nova (compute), Swift (object storage), Cinder (block storage), Horizon (dashboard), Keystone (identity), Glance (images), Neutron (networking), and Heat (orchestration). The platform provides control, flexibility, and scalability through its modular architecture and ability to integrate with third party technologies. It manages virtual machines, storage, networking, security, and other cloud resources through RESTful APIs.

Eugene Polonichko "Azure Data Lake: what is it? why is it? where is it?"

DataConf

Atlanta Data Science Meetup | Qubole slides

Qubole

Qubole is a cloud data analytics company founded in 2011 by former Facebook engineers. It provides a platform for interactive analytics on large datasets using Apache Spark and Presto on AWS. Qubole handles cluster management and scaling to enable self-service analytics without requiring Hadoop expertise. Customers span industries like advertising, healthcare, and retail and use Qubole for log analysis, machine learning, and business intelligence.

Spark and Couchbase– Augmenting the Operational Database with Spark

Matt Ingenthron

Data Driving Yahoo Mail Growth and Evolution with a 50 PB Hadoop Warehouse

DataWorks Summit

Yahoo Mail has 200+ million users a month and generates hundreds of terabytes of data per day, which continues to grow steadily. The nature of email messages has also evolved: for example, today the majority of them are generated by machines, consisting of newsletters, social media notifications, purchase invoices, travel bookings, and the like, which drove innovations in product development to help users organize their inboxes. Since 2014, the Yahoo Mail Data Engineering team took on the task of revamping the Mail data warehouse and analytics infrastructure in order to drive the continued growth and evolution of Yahoo Mail. Along the way we have built a 50 PB Hadoop warehouse, and surrounding analytics and machine learning programs that have transformed the way data plays in Yahoo Mail. In this session we will share our experience from this 3 year journey, from the system architecture, analytics systems built, to the learnings from development and drive for adoption.

Addressing Enterprise Customer Pain Points with a Data Driven Architecture

DataWorks Summit

Customers that are implementing Big Data Analytics projects in enterprise environments driven by line of business applications are faced with the three critical issues of Managing Complexity, Data Movement and Replication, and Cloud Integration. In this session you will learn about the characteristics of these pain points and how designing and implementing a data driven approach enables enterprises to implement quickly and efficiently with a future proof architecture of hybrid cloud.

Hadoop vs. RDBMS for Advanced Analytics

joshwills

The document discusses when to use Hadoop instead of a relational database management system (RDBMS) for advanced analytics. It provides examples of when queries like count distinct, cursors, and alter table statements become problematic in an RDBMS. It contrasts analyzing simple, transactional data like invoices versus complex, evolving data like customers or website visitors. Hadoop is better suited for problems involving complex objects, self-joins on large datasets, and matching large datasets. The document encourages structuring data in HDFS in a flexible way that fits the problem and use cases like simple counts on complex objects, self-self-self joins, and matching problems.

Improving Apache Spark™ In-Memory Computing with Apache Ignite™

Tom Diederich

GridGain Systems Lead Architect Valentin (Val) Kulichenko presented the following talk at the May 17 Bay Area In-Memory Computing Meetup: Improving Apache Spark™ In-Memory Computing with Apache Ignite™ Val explained how Apache Ignite™ simplifies development and improves performance for Apache Spark™. He'll demonstrate how Apache Spark and Ignite are integrated, and how they are used to together for analytics, stream processing and machine learning. The following was covered: * How Apache Ignite’s native RDD and new native DataFrame APIs work * How to use Ignite as an in-memory database and massively parallel processing (MPP) style collocated processing for preparing and managing data for Spark * How to leverage Ignite to easily share state across Spark jobs using mutable RDDs and DataFrames * How to leverage Ignite distributed SQL and advanced indexing in memory to improve SQL performance

Big Telco - Yousun Jeong

Spark Summit

This document provides an overview of SK Telecom's use of big data analytics and Spark. Some key points: - SKT collects around 250 TB of data per day which is stored and analyzed using a Hadoop cluster of over 1400 nodes. - Spark is used for both batch and real-time processing due to its performance benefits over other frameworks. Two main use cases are described: real-time network analytics and a network enterprise data warehouse (DW) built on Spark SQL. - The network DW consolidates data from over 130 legacy databases to enable thorough analysis of the entire network. Spark SQL, dynamic resource allocation in YARN, and integration with BI tools help meet requirements for timely processing and quick

Big Data Day LA 2016/ Use Case Driven track - Hydrator: Open Source, Code-Fre...

Data Con LA

Big Data Day LA 2016/ Use Case Driven track - Reliable Media Reporting in an ...

Data Con LA

Enabling Modern Application Architecture using Data.gov open government data

DataWorks Summit

Big Data and the Internet of Things (IoT) have forced businesses and the Federal Government to reevaluate their existing data strategies and adopt a more modern data architecture. With the advent of the connected data platform, migrating or building data-driven applications that take advantage of data-in-motion and data-at-rest can be a daunting journey to undertake. Scaling, reusability, and achieving operational agility are just some of the common pitfalls associated with existing software architectures. How do we embrace this paradigm shift? Adopting agile methodologies and emerging development practices such as Microservices and DevOps offer greater agility and operational efficiency enabling the government to rapidly build modern data-driven applications. During this talk and demonstration, we will show how the federal government can unleash the true power of the connected data platform with modern data-driven applications. Connected Data Platform: • Hortonworks DataFlow o Using Apache NiFi for capturing data at the edge of the data lake & managing the flow of data to the data platform o Apache Storm for complex event processing and stream processing • Hortonworks Data Platform o Apache Accumulo for scalability and cell-level security o Apache YARN for resource management • Modern Data-Driven Applications o Microservices: a software architecture practice for designing software applications as suites of independently deployable services, promoting componentization, single responsibility & scalability. Adopting a Microservices mindset enables the government to be technology agnostic: using the best tool or programming language for the job. ♣ Demoed REST API’s on-top of Apache Accumulo. (Spark-Java, AngularJS/Typescript) o DevOps: A culture and practice that breaks down the silos found between development and operations teams in traditional software practices. ♣ CI / CD pipelines, automated build kick-offs using containers (Docker, Jenkins) This talk will lay out a basic environment for promoting greater agility and operational efficiency for the federal government while taking advantage of a connected data platform.

Apache Ignite vs Alluxio: Memory Speed Big Data Analytics

DataWorks Summit

Apache Ignite vs Alluxio: Memory Speed Big Data Analytics - Apache Spark’s in memory capabilities catapulted it as the premier processing framework for Hadoop. Apache Ignite and Alluxio, both high-performance, integrated and distributed in-memory platform, takes Apache Spark to the next level by providing an even more powerful, faster and scalable platform to the most demanding data processing and analytic environments. Speaker Irfan Elahi, Consultant, Deloitte

Securing your Big Data Environments in the Cloud

DataWorks Summit

Big Data tools are becoming a critical part of enterprise architectures and as such securing the data, at rest, and in motion is a necessity. More so, when you’re implementing these solutions in the cloud and the data doesn't reside within the confines of your trusted data center. Also, there is a fine balance between implementing enterprise-grade security and negotiating utmost performance given the overheads of encryption and/or identity management. This session is designed to tackle these challenges head on and explain the various options available in the cloud. The focal points are the implementation of tools like Ranger and Knox for cloud deployments, but we also pay attention to the security features offered in the cloud that complement this process and secure the data in unprecedented ways. Cloud Security + OSS Security tools are a deadly combination, when it comes to securing your Data Lake.

Impala: A Modern, Open-Source SQL Engine for Hadoop

All Things Open

BIG Data & Hadoop Applications in E-Commerce

Skillspeed

Explore the applications of BIG Data & Hadoop in eCommerce via Skillspeed. BIG Data & Hadoop in eCommerce is a key differentiator, especially in terms of generating optimized customer & back-end experiences. They are used for tracking consumer behavior, optimizing logistics networks and forecasting demand - inventory cycles. To get more details regarding BIG Data & Hadoop, please visit - www.SkillSpeed.com

What's hot

Azure data lakes

Vishwas N

Hadoop data access layer v4.0

SpringPeople

Querying Druid in SQL with Superset

DataWorks Summit

Data Warehouse Modernization - Big Data in the Cloud Success with Qubole on O...

Qubole

Built-In Security for the Cloud

DataWorks Summit

HTAP Queries

Atif Shaikh

Ravi Namboori 's Open stack framework introduction

Ravi namboori

Eugene Polonichko "Azure Data Lake: what is it? why is it? where is it?"

DataConf

Atlanta Data Science Meetup | Qubole slides

Qubole

Spark and Couchbase– Augmenting the Operational Database with Spark

Matt Ingenthron

Data Driving Yahoo Mail Growth and Evolution with a 50 PB Hadoop Warehouse

DataWorks Summit

Addressing Enterprise Customer Pain Points with a Data Driven Architecture

DataWorks Summit

Hadoop vs. RDBMS for Advanced Analytics

joshwills

Improving Apache Spark™ In-Memory Computing with Apache Ignite™

Tom Diederich

Big Telco - Yousun Jeong

Spark Summit

Big Data Day LA 2016/ Use Case Driven track - Hydrator: Open Source, Code-Fre...

Data Con LA

Big Data Day LA 2016/ Use Case Driven track - Reliable Media Reporting in an ...

Data Con LA

Enabling Modern Application Architecture using Data.gov open government data

DataWorks Summit

Apache Ignite vs Alluxio: Memory Speed Big Data Analytics

DataWorks Summit

Securing your Big Data Environments in the Cloud

DataWorks Summit

What's hot (20)

Azure data lakes

Hadoop data access layer v4.0

Querying Druid in SQL with Superset

Data Warehouse Modernization - Big Data in the Cloud Success with Qubole on O...

Built-In Security for the Cloud

HTAP Queries

Ravi Namboori 's Open stack framework introduction

Eugene Polonichko "Azure Data Lake: what is it? why is it? where is it?"

Atlanta Data Science Meetup | Qubole slides

Spark and Couchbase– Augmenting the Operational Database with Spark

Data Driving Yahoo Mail Growth and Evolution with a 50 PB Hadoop Warehouse

Addressing Enterprise Customer Pain Points with a Data Driven Architecture

Hadoop vs. RDBMS for Advanced Analytics

Improving Apache Spark™ In-Memory Computing with Apache Ignite™

Big Telco - Yousun Jeong

Big Data Day LA 2016/ Use Case Driven track - Hydrator: Open Source, Code-Fre...

Big Data Day LA 2016/ Use Case Driven track - Reliable Media Reporting in an ...

Enabling Modern Application Architecture using Data.gov open government data

Apache Ignite vs Alluxio: Memory Speed Big Data Analytics

Securing your Big Data Environments in the Cloud

Viewers also liked

Impala: A Modern, Open-Source SQL Engine for Hadoop

All Things Open

BIG Data & Hadoop Applications in E-Commerce

Skillspeed

HBaseCon 2012 | Gap Inc Direct: Serving Apparel Catalog from HBase for Live W...

Cloudera, Inc.

hive HBase Metastore - Improving Hive with a Big Data Metadata Storage

DataWorks Summit/Hadoop Summit

The document discusses improving Hive metadata performance by using HBase as the backend storage instead of a relational database. It describes storing Hive metadata in HBase tables to provide faster access and greater scalability than when using a relational database. The system caches metadata in memory and uses transactions to manage concurrent access to the cached and stored metadata. Experimental results showed significant latency improvements over using a relational database backend.

HBaseCon 2013:High-Throughput, Transactional Stream Processing on Apache HBase

Cloudera, Inc.

This document discusses high-throughput transactional stream processing on HBase using Continuuity Reactor. Continuuity Reactor is a platform built on Hadoop and HBase that allows collecting, processing, storing and querying data with ACID guarantees using a flow-based programming model. Flows in Reactor are composed of flowlets connected by queues, with all processing occurring within transactions for consistency. The document outlines how Reactor implements transactions using an optimistic concurrency control approach based on multi-version concurrency control and HBase timestamps. It also discusses queue design and optimizations for performance.

HBaseCon 2012 | HBase, the Use Case in eBay Cassini

Cloudera, Inc.

eBay marketplace has been working hard on the next generation search infrastructure and software system, code-named Cassini. The new search engine processes over 250 million search queries and serves more than 2 billion page views each day. Its indexing platform is based on Apache Hadoop and Apache HBase. Apache HBase is a distributed persistent layer built on Hadoop to support billions of updates per day. Its easy sharding character, fast writes, and table scans, super fast data bulk load, and natural integration to Hadoop provide the cornerstones for successful continuous index builds. We will share with the audience the technical details and share the difficulties and challenges that we’ve gone through and that we are still facing in the process.

Real-World NoSQL Schema Design

DataWorks Summit/Hadoop Summit

This document provides an overview of NoSQL schema design and examples using a document database like MongoDB or MapR-DB. It discusses how to model complex, flexible schemas to store object-oriented data like products, users, and music catalog information. Examples show how a music database could be reduced from over 200 tables to just a few collections by embedding objects and references. Flexible schemas in a document database more closely match object models and allow easy evolution of the data model.

How we solved Real-time User Segmentation using HBase

DataWorks Summit

At RichRelevance, we service 10 of the top 20 Internet retailer chains and deliver more than $5.5 billions in attributable sales. Every 21 milliseconds a shopper clicks on a recommendation that we have delivered, and we serve over 850 million product recommendations daily. Our Hadoop infrastructure has a capacity to handle upwards of 1.5+ PB. Behavioral Targeting, specifically user segmentation and building personas, is critical for us in generating triggers when a user is added to a segment or switches from a segment. In this presentation, we intend to demonstrate not only how the events are captured, but also how they are stored in HBase in real-time. It is critical to design the system so it can handle thousands of writes per second and, at the same time, be able to query any combination of behavioral attributes in HBase through real-time APIs. This session will walk attendees through the entire design & architecture starting from data Ingestion, schema design, and access patterns, as well as some major problems like sharing & hot spotting. Furthermore, performance metrics will be presented, including the number of read/write per second and details around cluster configuration.

MongoDB Schema Design: Four Real-World Examples

Mike Friedman

This document discusses different schema designs for common use cases in MongoDB. It presents four cases: (1) modeling a message inbox, (2) retaining historical data within limits, (3) storing variable attributes efficiently, and (4) looking up users by multiple identities. For each case, it analyzes different modeling approaches, considering factors like query performance, write performance, and whether indexes can be used. The goal is to help designers choose an optimal schema based on their application's access patterns and scale requirements.

Magento scalability from the trenches (Meet Magento Sweden 2016)

Divante

This document discusses strategies for scaling a Magento e-commerce platform. It recommends first using vertical scaling by optimizing code and enabling caching before adding additional application and database servers through horizontal scaling. Specific optimizations discussed include using Redis for caching, Varnish for page caching, separating the database to its own server, enabling flat catalog indexing, and implementing master-slave database replication. Proper monitoring tools like New Relic and load testing are also emphasized for identifying bottlenecks during the scaling process.

Surprising failure factors when implementing eCommerce and Omnichannel eBusiness

Divante

We work on the large Omnichannel and eCommerce projects in Europe. Therefore, we can see from the inside how many companies approach this topic. Comparing it with the obtained results, we can determine positive and negative factors influencing success with great certainty. In this presentation we share stories of companies that are not mentioned in our case studies. These are the stories of bad choices, leading to failure.

Omnichannel Customer Experience

Divante

HBaseCon 2012 | HBase Schema Design - Ian Varley, Salesforce

Cloudera, Inc.

Most developers are familiar with the topic of “database design”. In the relational world, normalization is the name of the game. How do things change when you’re working with a scalable, distributed, non-SQL database like HBase? This talk will cover the basics of HBase schema design at a high level and give several common patterns and examples of real-world schemas to solve interesting problems. The storage and data access architecture of HBase (row keys, column families, etc.) will be explained, along with the pros and cons of different schema decisions.

Viewers also liked (13)

Impala: A Modern, Open-Source SQL Engine for Hadoop

BIG Data & Hadoop Applications in E-Commerce

HBaseCon 2012 | Gap Inc Direct: Serving Apparel Catalog from HBase for Live W...

hive HBase Metastore - Improving Hive with a Big Data Metadata Storage

HBaseCon 2013:High-Throughput, Transactional Stream Processing on Apache HBase

HBaseCon 2012 | HBase, the Use Case in eBay Cassini

Real-World NoSQL Schema Design

How we solved Real-time User Segmentation using HBase

MongoDB Schema Design: Four Real-World Examples

Magento scalability from the trenches (Meet Magento Sweden 2016)

Surprising failure factors when implementing eCommerce and Omnichannel eBusiness

Omnichannel Customer Experience

HBaseCon 2012 | HBase Schema Design - Ian Varley, Salesforce

Similar to HBaseCon 2012 | Developing Real Time Analytics Applications Using HBase in the Cloud - Rick Tucker, Sproxil

Introducing MongoDB into your Organization

MongoDB

Adoption of MongoDB has accelerated tremendously among developers in the past 18 months, and many large enterprises have now deployed MongoDB in reliable and large scale production environments. However, for many developers, it remains a challenge to convince production teams and business stakeholders to adopt an open source technology that has not been certified yet by their IT teams. This session will provide you with the compelling arguments to reassure business and production teams such as: Public customer references and real-world case studies (migration, and adoption stories) Deployment support and practices for robustness How MongoDB contributes to your company’s business value

Mongodb Presentation

Hashim Shaikh

The document discusses 10gen, the company behind MongoDB, and MongoDB features. 10gen provides commercial services like training, consulting, and support subscriptions for MongoDB. MongoDB is an open-source, document-oriented database that is easy to use, scalable, and supports rich data models. It provides features like auto-sharding, replication, and indexing. MongoDB is well-suited for applications with electronic health records due to its ability to handle sparse and evolving data schemas.

Mongodb Presentation

Hashim Shaikh

Mongodb hashim shaikh

Hashim Shaikh

10gen is the company behind MongoDB, the open source, document-oriented NoSQL database. The presentation compared MongoDB favorably to MUMPS databases, noting MongoDB's scalability, flexibility, and large community. It described how MongoDB is well-suited for electronic health records given its ability to handle sparse and evolving data models at large scale. MongoDB provides horizontal scaling, sophisticated querying, and easy integration with various programming languages.

Overcoming Today's Data Challenges with MongoDB

MongoDB

The document outlines an agenda for an event hosted by MongoDB on October 3rd 2017 in Amsterdam on overcoming data challenges with MongoDB. The agenda includes presentations on how the world has changed since relational databases were invented, how to transform IT environments with MongoDB, MongoDB use cases, and a customer story from IHS Markit. There will also be a Q&A session and conclusion. Speakers include representatives from MongoDB and IHS Markit.

Overcoming Today's Data Challenges with MongoDB

MongoDB

The document outlines an agenda for an event on overcoming data challenges with MongoDB. The event will feature speakers from MongoDB and Bosch discussing how the world has changed since relational databases were invented, how to radically transform IT environments with MongoDB, MongoDB and blockchain, and MongoDB for multiple use cases. The agenda includes presentations on these topics as well as a Q&A session and conclusion.

2022 Trends in Enterprise Analytics

DATAVERSITY

It is a fascinating, explosive time for enterprise analytics. It is from the position of analytics leadership that the mission will be executed and company leadership will emerge. The data professional is absolutely sitting on the performance of the company in this information economy and has an obligation to demonstrate the possibilities and originate the architecture, data, and projects that will deliver analytics. After all, no matter what business you’re in, you’re in the business of analytics. The coming years will be full of big changes in enterprise analytics and Data Architecture. William will kick off the fourth year of the Advanced Analytics series with a discussion of the trends winning organizations should build into their plans, expectations, vision, and awareness now.

Big Data

Immo Salo

This document discusses big data, including definitions, forecasts about talent shortages, and some key concepts. It notes that by 2018 the US could face shortages of 140,000-190,000 people with deep analytics skills and 1.5 million managers able to make decisions using big data analysis. It defines big data, describes Hadoop and MapReduce frameworks, discusses NoSQL databases, and mentions tools like AWS, Google BigQuery, and on-premises solutions from Oracle and Autonomy for working with big data.

Destroying Data Silos

DataWorks Summit

This document discusses ING NL's efforts to create a data lake architecture using Hadoop to integrate all of the bank's data sources onto a single processing platform. The data lake aims to collect data in a unified format, securely store it to prevent manipulation and unauthorized access, and make it available for analytical applications. Some of the challenges discussed include managing security, aligning with legacy systems, and facilitating interdepartmental cooperation on agile delivery. The presentation focuses on one part of the data lake, the archive, and how a Hadoop cluster can effectively address the goals of collecting, storing, and accessing data for business intelligence and data science purposes.

How to Get Started with Your MongoDB Pilot Project

DATAVERSITY

Open source, high performance database MongoDB can be used for a pilot project. The document discusses finding a non-critical initial project, getting experience with MongoDB, benchmarking performance, and presenting the business case for broader use. It also outlines steps for moving a successful pilot to production, including using MongoDB's auto-sharding, replication, and commercial support options.

Maximal: MPL Software Demo - INFORMS Phoenix Oct 2012

Bjarni Kristjánsson

The document discusses deploying optimization models on servers and mobile platforms. It provides an overview of global IT trends affecting operations research and optimization software. It then describes Maximal Software's MPL optimization component libraries, which allow embedding optimization models into applications and seamlessly integrating with programming languages and data sources. Examples are given of using the MPL OptiMax libraries in Python, C#, and MATLAB to build optimization applications.

Dremio introduction

Alexis Gendronneau

This document discusses Apache Dremio, an open source data virtualization platform that provides self-service SQL access to data sources like Elasticsearch, MongoDB, HDFS, and relational databases. It aims to make data analytics faster by avoiding the need for data staging, warehouses, cubes, and extracts. Dremio uses techniques like reflections, pushdowns, and a universal relational algebra to optimize queries and leverage caches. It is based on projects like Apache Drill, Calcite, Arrow, and Parquet and can be deployed on Hadoop or the cloud. The presentation includes a demo of using Dremio to create datasets, curate/prepare data, accelerate queries with reflections, and manage resources.

Ordbms

Dabbal Singh Mahara

The document discusses object relational database management systems (ORDBMS). It begins by describing relational database management systems (RDBMS) and their deficiencies in supporting complex data types. Object oriented database management systems (OODBMS) were developed to address these deficiencies but faced their own challenges. ORDBMS were created to combine the best aspects of RDBMS and OODBMS, supporting both relational and object-oriented features like inheritance and encapsulation while maintaining compatibility with SQL. Major database vendors have incorporated ORDBMS features into their products to provide a migration path for users between the different models.

From Business Idea to Successful Delivery by Serhiy Haziyev & Olha Hrytsay, S...

SoftServe

The document discusses transitioning product development to an open innovation model using outsourcing partners. It provides an overview of SoftServe's capabilities including big data solutions, reference architectures, and case studies. The document also outlines SoftServe's involvement in collaborating with clients throughout the product development lifecycle from envisioning ideas to commercialization. Finally, it provides a checklist for successful delivery when partnering with outsourcing providers for big data and product development.

mongoDB: Driving a data revolution

MongoDB

The document discusses MongoDB and 10gen. It provides an overview of 10gen, the company behind MongoDB. 10gen has 170+ employees, 500+ customers, $73M in funding, and offices worldwide. The document then discusses MongoDB's adoption, capabilities like replication and scaling, use cases across different industries, and how MongoDB can help organizations solve problems and drive innovation. It concludes by providing resources for learning more about MongoDB.

New Approaches to Migrating from Oracle to Enterprise-Ready Postgres in the C...

EDB

The document discusses migrating from Oracle to Postgres in the cloud using EnterpriseDB's (EDB) Postgres Cloud Database Service. It highlights that EDB offers Oracle compatibility, migration tools, and human support to help with migrations. Over 50% of migrations have slight to no modifications needed. The presentation covers the EDB Cloud Database Service, reasons to migrate, components of a successful migration, what EDB offers to help, and how to identify good candidates for migration. It recommends leveraging the EDB tools and cloud database service for an agile migration.

Final_CloudEventFrankfurt2017 (1).pdf

MongoDB

Case Study: Using EDMCS to Solve Master Data Challenges

Alithya

Enterprise Data Management Cloud Service (EDMCS) is a relatively new but critically important addition to the EPM Cloud product landscape. As clients have discovered the challenges surrounding effective master data management do not go away just by moving to the cloud. If anything, the EPM Cloud transition has only further emphasized the importance of robust management of enterprise master data assets. Come to this session to hear about the practical benefits achieved by an EDMCS implementation project. This session will briefly highlight the key features and concepts of EDMCS (note, EDMCS really is not “DRM in the cloud!”). Then we will dive into a recent project involving an EDMCS implementation at a large mid-Atlantic health care insurer in support of an enterprise cost management/allocation initiative. Challenges, lessons learned, and benefits achieved will be highlighted, along with a look at the unique mix of products and technologies involved in the solution (PCMCS, Linux, and EPM Automate to name a few). We will also touch on the interesting challenges involving dimensionality, primary and alternate hierarchies, and data mappings as they relate to an EDMCS implementation for PCMCS.

An Evening with MongoDB Detroit 2013

MongoDB

Optimize with Open Source

EDB

Recent advances in Postgres have propelled the database forward to meet today’s data challenges. At some of the world’s largest companies, Postgres plays a major role in controlling costs and reducing dependence on traditional providers. This presentation addresses: * What workloads are best suited for introducing Postgres into your environment * The success milestones for evaluating the ‘when and how’ of expanding Postgres deployments * Key advances in recent Postgres releases that support new data types and evolving data challenges This presentation is intended for strategic IT and Business Decision-Makers involved in data infrastructure decisions and cost-savings.

Similar to HBaseCon 2012 | Developing Real Time Analytics Applications Using HBase in the Cloud - Rick Tucker, Sproxil (20)

Introducing MongoDB into your Organization

Mongodb Presentation

Mongodb hashim shaikh

Overcoming Today's Data Challenges with MongoDB

2022 Trends in Enterprise Analytics

Big Data

Destroying Data Silos

How to Get Started with Your MongoDB Pilot Project

Maximal: MPL Software Demo - INFORMS Phoenix Oct 2012

Dremio introduction

Ordbms

From Business Idea to Successful Delivery by Serhiy Haziyev & Olha Hrytsay, S...

mongoDB: Driving a data revolution

New Approaches to Migrating from Oracle to Enterprise-Ready Postgres in the C...

Final_CloudEventFrankfurt2017 (1).pdf

Case Study: Using EDMCS to Solve Master Data Challenges

An Evening with MongoDB Detroit 2013

Optimize with Open Source

More from Cloudera, Inc.

Partner Briefing_January 25 (FINAL).pptx

Cloudera, Inc.

The document discusses using Cloudera DataFlow to address challenges with collecting, processing, and analyzing log data across many systems and devices. It provides an example use case of logging modernization to reduce costs and enable security solutions by filtering noise from logs. The presentation shows how DataFlow can extract relevant events from large volumes of raw log data and normalize the data to make security threats and anomalies easier to detect across many machines.

Cloudera Data Impact Awards 2021 - Finalists

Cloudera, Inc.

The document outlines the 2021 finalists for the annual Data Impact Awards program, which recognizes organizations using Cloudera's platform and the impactful applications they have developed. It provides details on the challenges, solutions, and outcomes for each finalist project in the categories of Data Lifecycle Connection, Cloud Innovation, Data for Enterprise AI, Security & Governance Leadership, Industry Transformation, People First, and Data for Good. There are multiple finalists highlighted in each category demonstrating innovative uses of data and analytics.

2020 Cloudera Data Impact Awards Finalists

Cloudera, Inc.

Cloudera is proud to present the 2020 Data Impact Awards Finalists. This annual program recognizes organizations running the Cloudera platform for the applications they've built and the impact their data projects have on their organizations, their industries, and the world. Nominations were evaluated by a panel of independent thought-leaders and expert industry analysts, who then selected the finalists and winners. Winners exemplify the most-cutting edge data projects and represent innovation and leadership in their respective industries.

Edc event vienna presentation 1 oct 2019

Cloudera, Inc.

The document outlines the agenda for Cloudera's Enterprise Data Cloud event in Vienna. It includes welcome remarks, keynotes on Cloudera's vision and customer success stories. There will be presentations on the new Cloudera Data Platform and customer case studies, followed by closing remarks. The schedule includes sessions on Cloudera's approach to data warehousing, machine learning, streaming and multi-cloud capabilities.

Machine Learning with Limited Labeled Data 4/3/19

Cloudera, Inc.

Data Driven With the Cloudera Modern Data Warehouse 3.19.19

Cloudera, Inc.

Introducing Cloudera DataFlow (CDF) 2.13.19

Cloudera, Inc.

Introducing Cloudera Data Science Workbench for HDP 2.12.19

Cloudera, Inc.

Shortening the Sales Cycle with a Modern Data Warehouse 1.30.19

Cloudera, Inc.

Leveraging the cloud for analytics and machine learning 1.29.19

Cloudera, Inc.

Modernizing the Legacy Data Warehouse – What, Why, and How 1.23.19

Cloudera, Inc.

Leveraging the Cloud for Big Data Analytics 12.11.18

Cloudera, Inc.

Modern Data Warehouse Fundamentals Part 3

Cloudera, Inc.

Modern Data Warehouse Fundamentals Part 2

Cloudera, Inc.

Modern Data Warehouse Fundamentals Part 1

Cloudera, Inc.

The document discusses the benefits and trends of modernizing a data warehouse. It outlines how a modern data warehouse can provide deeper business insights at extreme speed and scale while controlling resources and costs. Examples are provided of companies that have improved fraud detection, customer retention, and machine performance by implementing a modern data warehouse that can handle large volumes and varieties of data from many sources.

Extending Cloudera SDX beyond the Platform

Cloudera, Inc.

Federated Learning: ML with Privacy on the Edge 11.15.18

Cloudera, Inc.

Analyst Webinar: Doing a 180 on Customer 360

Cloudera, Inc.

Build a modern platform for anti-money laundering 9.19.18

Cloudera, Inc.

Introducing the data science sandbox as a service 8.30.18

Cloudera, Inc.

More from Cloudera, Inc. (20)

Partner Briefing_January 25 (FINAL).pptx

Cloudera Data Impact Awards 2021 - Finalists

2020 Cloudera Data Impact Awards Finalists

Edc event vienna presentation 1 oct 2019

Machine Learning with Limited Labeled Data 4/3/19

Data Driven With the Cloudera Modern Data Warehouse 3.19.19

Introducing Cloudera DataFlow (CDF) 2.13.19

Introducing Cloudera Data Science Workbench for HDP 2.12.19

Shortening the Sales Cycle with a Modern Data Warehouse 1.30.19

Leveraging the cloud for analytics and machine learning 1.29.19

Modernizing the Legacy Data Warehouse – What, Why, and How 1.23.19

Leveraging the Cloud for Big Data Analytics 12.11.18

Modern Data Warehouse Fundamentals Part 3

Modern Data Warehouse Fundamentals Part 2

Modern Data Warehouse Fundamentals Part 1

Extending Cloudera SDX beyond the Platform

Federated Learning: ML with Privacy on the Edge 11.15.18

Analyst Webinar: Doing a 180 on Customer 360

Build a modern platform for anti-money laundering 9.19.18

Introducing the data science sandbox as a service 8.30.18

Recently uploaded

AI in the Workplace Reskilling, Upskilling, and Future Work.pptx

Sunil Jagani

From Natural Language to Structured Solr Queries using LLMs

Sease

This talk draws on experimentation to enable AI applications with Solr. One important use case is to use AI for better accessibility and discoverability of the data: while User eXperience techniques, lexical search improvements, and data harmonization can take organizations to a good level of accessibility, a structural (or “cognitive” gap) remains between the data user needs and the data producer constraints. That is where AI – and most importantly, Natural Language Processing and Large Language Model techniques – could make a difference. This natural language, conversational engine could facilitate access and usage of the data leveraging the semantics of any data source. The objective of the presentation is to propose a technical approach and a way forward to achieve this goal. The key concept is to enable users to express their search queries in natural language, which the LLM then enriches, interprets, and translates into structured queries based on the Solr index’s metadata. This approach leverages the LLM’s ability to understand the nuances of natural language and the structure of documents within Apache Solr. The LLM acts as an intermediary agent, offering a transparent experience to users automatically and potentially uncovering relevant documents that conventional search methods might overlook. The presentation will include the results of this experimental work, lessons learned, best practices, and the scope of future work that should improve the approach and make it production-ready.

Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors

DianaGray10

Join us to learn how UiPath Apps can directly and easily interact with prebuilt connectors via Integration Service--including Salesforce, ServiceNow, Open GenAI, and more. The best part is you can achieve this without building a custom workflow! Say goodbye to the hassle of using separate automations to call APIs. By seamlessly integrating within App Studio, you can now easily streamline your workflow, while gaining direct access to our Connector Catalog of popular applications. We’ll discuss and demo the benefits of UiPath Apps and connectors including: Creating a compelling user experience for any software, without the limitations of APIs. Accelerating the app creation process, saving time and effort Enjoying high-performance CRUD (create, read, update, delete) operations, for seamless data management. Speakers: Russell Alfeche, Technology Leader, RPA at qBotic and UiPath MVP Charlie Greenberg, host

Apps Break Data

Ivo Velitchkov

How information systems are built or acquired puts information, which is what they should be about, in a secondary place. Our language adapted accordingly, and we no longer talk about information systems but applications. Applications evolved in a way to break data into diverse fragments, tightly coupled with applications and expensive to integrate. The result is technical debt, which is re-paid by taking even bigger "loans", resulting in an ever-increasing technical debt. Software engineering and procurement practices work in sync with market forces to maintain this trend. This talk demonstrates how natural this situation is. The question is: can something be done to reverse the trend?

"$10 thousand per minute of downtime: architecture, queues, streaming and fin...

Fwdays

Direct losses from downtime in 1 minute = $5-$10 thousand dollars. Reputation is priceless. As part of the talk, we will consider the architectural strategies necessary for the development of highly loaded fintech solutions. We will focus on using queues and streaming to efficiently work and manage large amounts of data in real-time and to minimize latency. We will focus special attention on the architectural patterns used in the design of the fintech system, microservices and event-driven architecture, which ensure scalability, fault tolerance, and consistency of the entire system.

inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill

LizaNolte

HERE IS YOUR WEBINAR CONTENT! 'Mastering Customer Journey Management with Dr. Graham Hill'. We hope you find the webinar recording both insightful and enjoyable. In this webinar, we explored essential aspects of Customer Journey Management and personalization. Here’s a summary of the key insights and topics discussed: Key Takeaways: Understanding the Customer Journey: Dr. Hill emphasized the importance of mapping and understanding the complete customer journey to identify touchpoints and opportunities for improvement. Personalization Strategies: We discussed how to leverage data and insights to create personalized experiences that resonate with customers. Technology Integration: Insights were shared on how inQuba’s advanced technology can streamline customer interactions and drive operational efficiency.

Call Girls Chandigarh🔥7023059433🔥Agency Profile Escorts in Chandigarh Availab...

manji sharman06

AppSec PNW: Android and iOS Application Security with MobSF

Ajin Abraham

Mobile Security Framework - MobSF is a free and open source automated mobile application security testing environment designed to help security engineers, researchers, developers, and penetration testers to identify security vulnerabilities, malicious behaviours and privacy concerns in mobile applications using static and dynamic analysis. It supports all the popular mobile application binaries and source code formats built for Android and iOS devices. In addition to automated security assessment, it also offers an interactive testing environment to build and execute scenario based test/fuzz cases against the application. This talk covers: Using MobSF for static analysis of mobile applications. Interactive dynamic security assessment of Android and iOS applications. Solving Mobile app CTF challenges. Reverse engineering and runtime analysis of Mobile malware. How to shift left and integrate MobSF/mobsfscan SAST and DAST in your build pipeline.

Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians

Neo4j

The Microsoft 365 Migration Tutorial For Beginner.pptx

operationspcvita

"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba

Fwdays

This is a session that details how PostgreSQL's features and Azure AI Services can be effectively used to significantly enhance the search functionality in any application. In this session, we'll share insights on how we used PostgreSQL to facilitate precise searches across multiple fields in our mobile application. The techniques include using LIKE and ILIKE operators and integrating a trigram-based search to handle potential misspellings, thereby increasing the search accuracy. We'll also discuss how the azure_ai extension on PostgreSQL databases in Azure and Azure AI Services were utilized to create vectors from user input, a feature beneficial when users wish to find specific items based on text prompts. While our application's case study involves a drug search, the techniques and principles shared in this session can be adapted to improve search functionality in a wide range of applications. Join us to learn how PostgreSQL and Azure AI can be harnessed to enhance your application's search capability.

Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck

FilipTomaszewski5

Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels

Northern Engraving

GlobalLogic Java Community Webinar #18 “How to Improve Web Application Perfor...

GlobalLogic Ukraine

Під час доповіді відповімо на питання, навіщо потрібно підвищувати продуктивність аплікації і які є найефективніші способи для цього. А також поговоримо про те, що таке кеш, які його види бувають та, основне — як знайти performance bottleneck? Відео та деталі заходу: https://bit.ly/45tILxj

What is an RPA CoE? Session 2 – CoE Roles

DianaGray10

JavaLand 2024: Application Development Green Masterplan

Miro Wengner

QA or the Highway - Component Testing: Bridging the gap between frontend appl...

zjhamm304

Astute Business Solutions | Oracle Cloud Partner |

AstuteBusiness

ScyllaDB Tablets: Rethinking Replication

ScyllaDB

ScyllaDB is making a major architecture shift. We’re moving from vNode replication to tablets – fragments of tables that are distributed independently, enabling dynamic data distribution and extreme elasticity. In this keynote, ScyllaDB co-founder and CTO Avi Kivity explains the reason for this shift, provides a look at the implementation and roadmap, and shares how this shift benefits ScyllaDB users.

AWS Certified Solutions Architect Associate (SAA-C03)

HarpalGohil4

Recently uploaded (20)

AI in the Workplace Reskilling, Upskilling, and Future Work.pptx

From Natural Language to Structured Solr Queries using LLMs

Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors

Apps Break Data

"$10 thousand per minute of downtime: architecture, queues, streaming and fin...

inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill

Call Girls Chandigarh🔥7023059433🔥Agency Profile Escorts in Chandigarh Availab...

AppSec PNW: Android and iOS Application Security with MobSF

Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians

The Microsoft 365 Migration Tutorial For Beginner.pptx

"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba

Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck

Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels

GlobalLogic Java Community Webinar #18 “How to Improve Web Application Perfor...

What is an RPA CoE? Session 2 – CoE Roles

JavaLand 2024: Application Development Green Masterplan

QA or the Highway - Component Testing: Bridging the gap between frontend appl...

Astute Business Solutions | Oracle Cloud Partner |

ScyllaDB Tablets: Rethinking Replication

AWS Certified Solutions Architect Associate (SAA-C03)

HBaseCon 2012 | Developing Real Time Analytics Applications Using HBase in the Cloud - Rick Tucker, Sproxil

2. About Sproxil • Brand protection, specializing in anti- 1 SCRATCH counterfeiting solutions • Solution requires a scalable and high- throughput text 2 message processing TEXT engine • Supports a real-time analytics web interface 3 VERIFY tech@sproxil.com May 22,2012 © 2012 Sproxil, Inc.

4. Real-Time Analytics Engine • MapReduce too slow to maintain data in true real time • As data arrives, analytical data is updated through counters Text Message Message Increment Arrives Analyzed Counters Genuine Product +1 Increment Counter for Authentication Genuine Authentications Repeat Customer +1 Increment Counter for Repeat Customers tech@sproxil.com May 22,2012 © 2012 Sproxil, Inc.

5. Schema Design: Example 1 • Example: View log of text messages in chronological order • Rowkey: row prefix + timestamp Row transaction 2012-05-22 12:00:00 transaction 2012-05-22 12:01:14 transaction 2012-05-22 12:02:03 Note: HBase sorts rowkeys lexicographically so scans return data in reverse chronological order tech@sproxil.com May 22,2012 © 2012 Sproxil, Inc. 5

6. • • Row transaction userID 1 2012-05-22 12:00:00 transaction userID 1 2012-05-22 12:01:14 transaction userID 2 2012-05-22 12:00:54 transaction userID 2 2012-05-22 12:01:22 transaction userID 2 2012-05-22 12:02:01 Note: Hbase sorts rows lexicographically so scans return data in reverse chronological order tech@sproxil.com May 22,2012 © 2012 Sproxil, Inc.

7. Critical Findings • Schema design is crucial for successful HBase implementation – Pack as much info as possible into row keys • Use caution with Filters – E.g. Regex filters can be costly – Alternatives: • Directly query for data you need • Use efficient filters when filtering large data sets tech@sproxil.com May 22,2012 © 2012 Sproxil, Inc.

8. Thank You! Your global brand protection specialists – spanning 3 continents and Making Counterfeiting Unprofitable™ speaking 9 languages tech@sproxil.com +1 617 682 9577 America | Asia | Africa Sproxil.com tech@sproxil.com May 22,2012 © 2012 Sproxil, Inc. 8

Editor's Notes

Processed large volume of text messages, has even led to arrest of counterfeiters
High speed transactional operations criticalHandle large volumes of text messages quicklyLarge volume of dataMillions of recordsSchema supports sparse data
Explain why regex is costly

HBaseCon 2012 | Developing Real Time Analytics Applications Using HBase in the Cloud - Rick Tucker, Sproxil

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (13)

Similar to HBaseCon 2012 | Developing Real Time Analytics Applications Using HBase in the Cloud - Rick Tucker, Sproxil

Similar to HBaseCon 2012 | Developing Real Time Analytics Applications Using HBase in the Cloud - Rick Tucker, Sproxil (20)

More from Cloudera, Inc.

More from Cloudera, Inc. (20)

Recently uploaded

Recently uploaded (20)

HBaseCon 2012 | Developing Real Time Analytics Applications Using HBase in the Cloud - Rick Tucker, Sproxil

Editor's Notes