Smart Meter Data Analytic using Hadoop

•Download as PPTX, PDF•

9 likes•8,262 views

This document discusses using Hadoop for smart meter data analytics. Smart meters track energy usage and send data to utility servers. Analyzing large volumes of smart meter data presents challenges due to data growth rates. Hadoop can help by reducing data loads and improving query performance due to its scalability. The document outlines how Hadoop can enable demand response analysis, time of use tariff analysis, and load profile analysis. It also provides diagrams of a Hadoop cluster and data flow for smart meter data analytics.

Technology Business

Smart Meter Data Analytic Using
Hadoop
Omkar Nibandhe and Abhishek Korpe
students

SMART METER
DATA ANALYTICS
(SMDA)
USING HADOOP
By :
Omkar Nibandhe ( Student )
Abhishek Korpe ( Student )
03-04-2014SMDAusingHADOOP
2

Why Smart Meter Data
Analytics ?
03-04-2014SMDAusingHADOOP
3

What are SMART METERS ?
MIF
• Track and store the amount of energy used.
• Send the collected data to the Energy
Distribution company server at regular time
intervals.
House/
Industry
Smart Meter Server
03-04-2014SMDAusingHADOOP
4

Confronting the data deluge
• Rate Generation – 15 minutes.
• For single meter – 3000 readings/month (approx ).
• For 1 million meters – 36 Billion readings/year
(approx).
• Annual Growth – 13% ( 2010 – 2015 ).
• Total Shipment – 460.9 Million Smart Meters.
Source: Build smart metering solutions with IBM Informix TimeSeries
03-04-2014SMDAusingHADOOP
6

Capitalizing on the unique
value of Hadoop
Solution ?
• Reducing data load times.
• Improving query performance.
• Massive Scalability.
03-04-2014SMDAusingHADOOP
7

Demand - Response
03-04-2014SMDAusingHADOOP
8

Time of use Tariff
03-04-2014SMDAusingHADOOP
9

Hadoop Cluster in LAB 412
Masters
Slaves
Slave1
Slave7 Slave8 Slave9 Slave11
Slave6Slave5Slave4Slave3Slave2
Slave10
Slave19Slave18
Slave17Slave16Slave15Slave14Slave13
Slave12
Slave20 Slave21
03-04-2014SMDAusingHADOOP
12

LAB 412 (MESCOE Pune, India)
03-04-2014SMDAusingHADOOP
13

SMDA - NameNode
03-04-2014SMDAusingHADOOP
14

SMDA - SecondaryNameNode
03-04-2014SMDAusingHADOOP
15

SMDA - JobTracker
03-04-2014SMDAusingHADOOP
16

SMDA – Input ( .MIF )
03-04-2014SMDAusingHADOOP
17
1392 19501 0.157

SMDA - Output
03-04-2014SMDAusingHADOOP
18

Test Job 2 ( Combiner )
03-04-2014SMDAusingHADOOP
20

Future Scope
• Analysis of -
• Customer segmentation.
• Customer behavior.
• Meter ping commands.
• Outage management.
• Power quality.
• Extending data point(s) : weather, geographical location, family
consumption, etc.
03-04-2014SMDAusingHADOOP
21

Thanks to ….
• Irish Social Science Data Archive ( ISSDA ).
• Rahul Khinvasara – Director, zCon Solutions Pvt. Ltd.
• Modern Education Society’s College of Engineering (MESCOE).
• Prof. Balaji Bodkhe – Guide, MESCOE.
• Prof. N. Shaikh – Head of Computer Department, MESCOE.
• Prof. A. Hake – Vice Principal, MESCOE.
• Prof. P. Raut – Administrative Head, MESCOE.
03-04-2014SMDAusingHADOOP
22

Any Suggestions ??
03-04-2014SMDAusingHADOOP
23

Any Questions?
03-04-2014SMDAusingHADOOP
25

What's hot

Big Data, Big Deal: For Future Big Data Scientists

Way-Yen Lin

Big data in telecom

Shubham Bathe

Interoperability is a key requirement for the IoT but what does it really mean? Standard protocols for different vendor's devices to interact with each other? Connection between different languages and operating systems? Wireless technology choice? A way for devices to interact with the cloud? Does it include data syntax? Must we model semantics? Can security interoperate? RTI, the world's largest embedded middleware company, participates in about 15 different "interoperability" efforts, including FACE (avionics), GVA (European vehicle architecture), SGIP (smart grid) and ICE (medical systems). We are leaders in the Industrial IoT and its leading consortium, the Industrial Internet Consortium (IIC). This session will examine the depth of the interoperability problem and explore solutions. Presented by Stan Schneider, RTI CEO at IoTDevCon 2015

What Does Interoperability Mean for the IoT?

Real-Time Innovations (RTI)

Industry 4.0 and Cyber physical systems Intro

Dr Mohamed Elfarran

Applications of Big Data Analytics in Businesses

T.S. Lim

Big data ppt

AKASH SIHAG

Lambda architecture

Szilveszter Molnár

Stream SAP and Other Events Across Your Enterprise with an Event Mesh

Solace

Internet-of-things- (IOT) - a-seminar - ppt - by- mohan-kumar-g

Mohan Kumar G

Integrating Big Data Technologies

DATAVERSITY

- The model we built here will use all possible factors to predict data on customers to find who are defaulters and non‐defaulters next month. - The goal is to find the whether the clients are able to pay their next month credit amount. - Identify some potential customers for the bank who can settle their credit balance. - To determine if their customers could make the credit card payments on‐time. - Default is the failure to pay interest or principal on a loan or credit card payment.

Machine Learning Project - Default credit card clients

Vatsal N Shah

機械学習システムを受託開発する時に気をつけておきたい事

BrainPad Inc.

Introduction to Hadoop

Dr. C.V. Suresh Babu

Analytics in the Manufacturing industry

International School of Engineering

Housing price prediction

Abhimanyu Dwivedi

This presentation introduces big data and explains how to generate actionable insights using analytics techniques. The deck explains general steps involved in a typical analytics project and provides a brief overview of the most commonly used predictive analytics methods and their business applications. Vijay Adamapure is a Data Science Enthusiast with extensive experience in the field of data mining, predictive modeling and machine learning. He has worked on numerous analytics projects ranging from healthcare, business analytics, renewable energy to IoT. Vijay presented these slides during the Internet of Everything Meetup event 'Predictive Analytics - An Overview' that took place on Jan. 9, 2015 in Mumbai. To join the Meetup group, register here: http://bit.ly/1A7T0A1

Predictive Analytics - An Overview

MachinePulse

Machine Learning and Industrie 4.0

Peter Schleinitz

Fraud detection with Machine Learning

Scaleway

Big data analytics for telecom operators final use cases 0712-2014_prof_m erdas

Prof Dr Mehmed ERDAS

This document is the first deliverable of the Lean Big Data work package 7 (WP7). The main goal of the package 7 is to provide the use cases applications that will be used to validate the Lean Big Data platform. To this end, an analysis of requirement of each use case will be provided in the scope.This analysis will be used as basis for the description of the evaluation, benchmarking and validation of the Lean Big Data platform. This deliverable comprises the analysis of requirements for the following case of study provided in the context of Lean Big Data: Data Centre monitoring Case Study, Electronic Alignment of Direct Debit transactions Case Study, Social Network-based Area surveillance Case Study and Targeted Advertisement Case Study.

Requirements document for big data use cases

Allied Consultants

What's hot (20)

Big Data, Big Deal: For Future Big Data Scientists

Big data in telecom

What Does Interoperability Mean for the IoT?

Industry 4.0 and Cyber physical systems Intro

Applications of Big Data Analytics in Businesses

Big data ppt

Lambda architecture

Stream SAP and Other Events Across Your Enterprise with an Event Mesh

Internet-of-things- (IOT) - a-seminar - ppt - by- mohan-kumar-g

Integrating Big Data Technologies

Machine Learning Project - Default credit card clients

機械学習システムを受託開発する時に気をつけておきたい事

Introduction to Hadoop

Analytics in the Manufacturing industry

Housing price prediction

Predictive Analytics - An Overview

Machine Learning and Industrie 4.0

Fraud detection with Machine Learning

Big data analytics for telecom operators final use cases 0712-2014_prof_m erdas

Requirements document for big data use cases

Viewers also liked

What are big data in the contacts of energy & utilities, and how/where can the utilities find value in the data. In this C-level presentation we discussed the three prime areas: grid operations, smart metering and asset & workforce management. A section on cognitive computing for utilities have been omitted from the presentation due to confidentiality - but I tell you - it is mind-blowing perspectives on how IBM Watson will help utilities plan and optimize their operations in the near future! See more on http://www.ibmbigdatahub.com/industry/energy-utilities

Big Data Analytics in Energy & Utilities

Anders Quitzau

Smart Analytics For The Utility Sector

Herman Bosker

Explosive growth of Smart Meter (SM) deployments has presented key infrastructure challenges across the utility industry. The huge volumes of smart meter data has led the industry to a tipping point which requires investments in modernizing existing data warehouses. Typical modernization efforts lead to huge capital expenditures for DW appliances and storage. Sizing this new infrastructure is tricky and can lead to underutilized or poorly performing hardware. The Cloud is the catalyst to solving these Big Data challenges. Utilizing a Cloud architecture delivers huge benefits by: Maximizing use of existing architecture Minimizing new CapEx expenditures Lowering overall storage costs Enabling scale on demand

Supercharging Smart Meter BIG DATA Analytics with Microsoft Azure Cloud- SRP ...

Mike Rossi

Energy Data Analytics | Energy Efficiency | India

Umesh Bhutoria

The Soft Grid 2013 Opening Presentation

GTMevents

Smart metering infrastructure Architecture and analytics

Sandeep Sharma IIMK Smart City,IoT,Bigdata,Cloud,BI,DW

Smart Meter Basics and Benefits

University of Minnesota

Meter Data Management Presentation

Talyam

The fact that a Meter Data Management (MDM) system is the single, secure repository for the millions of data points collected by an AMI makes it the logical solution for data analytics such as validation, editing and estimation that improve the accuracy of billing information. Yet, as a single-source system of record, the MDM also is the starting point for integration of meter-read data with other enterprise systems to improve real-time efficiency of network operations and business processes. The MDM with meter modelling components and standardized connectivity can integrate with the utility geodatabase (GIS) and outage management system (OMS) to significantly streamline outage detection and restoration verification. MDM integrated with the utility supervisory control and data acquisition (SCADA) system or distribution management system (DMS) allows comparison of information at substation/net-stations with aggregated meter data to detect potential theft or network loss during distribution. Similar aggregate comparison helps analyse power quality, identify demand trending and forecast demand. These network analysis capabilities empower accurate asset planning and the utility’s ability to meet demand without adding more capacity. In all of these enterprise-level functions, MDM integration with the GIS provides valuable visualization that facilitates operator and analyst identification of areas of concern or opportunity. The real-time network intelligence possible with such a powerful MDM solution can return substantial benefits to several utility operations and business processes — well beyond the initial-level billing accuracy improvement.

Meter Data Management 2.0

Schneider Electric

What is Big Data?

Bernard Marr

Check out this white paper from eInfochips which showcases how energy and utility providers can unlock potential service opportunities using our predictive analytics solution across all stages of the business cycle. Major utility players are set to roll out millions of smart meters with the aim of generating actionable insights even though as per the industry’s own admission, any serious effort toward monetization is being offset by a lack of core IT capabilities, especially in big data technology. Capturing proactive intelligence on consumer behavior is the way to go. In this white paper, eInfochips demonstrates how utility players can predict demand response, generation response and create new revenue models around coincidental peak demands, smart expenditure modeling and other forms of end user data.

Whitepaper - Transforming the Energy & Utilities Industry with Smart Analytics

eInfochips (An Arrow Company)

Concepts of smart meter

Vasanthan Ravichandran

Smart Meters

Anshul Shrivastava

Smart energy meter (Updated)

Dnyanesh Patil

Big Data solution for multi-national Bank

Ritu Sarkar

Intro to Data Science on Hadoop

Caserta

Lviv IT Arena is a conference specially designed for programmers, designers, developers, top managers, inverstors, entrepreneurs and startuppers. Annually it takes place at the beginning of October in Lviv at Arena Lviv stadium. In 2016 the conference gathered more than 1800 participants and over 100 speakers from companies like Microsoft, Philips, Twitter, UBER and IBM. More details about the conference at itarena.lviv.ua.

Data Science for Energy Efficiency (Dmytro Mindra Technology Stream)

IT Arena

When writing this new paper, my main objective was to provide a clear understanding of where the term "Big Data" comes from, why is that term so popular now, what does it really mean and what can be its implication for businesses. Because the full power of Big Data can be revealed only by Analytics, i provided a description of a widely recognized and used analytical techniques to help you figure out how used in conjunction with Big Data, analytics can boost Business Performance. i expected that by the end of this paper : - you will smile the next time you read or hear at the terms big data, hadoop, or analytics :) - you will understand the technologies that are behind the scene when one talks about "Big Data" - you will know how to "make sense" of Big Data using Analytics - you will get a basic idea of data mining techniques used in Business in general and in Big Data in particular - you will be able to get every news about Big Data

Big Data : a 360° Overview

Juvénal CHOKOGOUE

Presentation talks about why countries should not ignore #Data while designing and developing Industrial #EnergyEfficiency Policy. This helps them in taking "bottom to top" approach in comparison to the existing "top to bottom" approach being taken by most of the countries. Data on it's own can do absolutely nothing, but can certainly help policy makers design a policy that drives sustainable energy efficiency improvements.

Why not to ignore #Data while designing Industrial #EnergyEfficiency Policy

Umesh Bhutoria

Using_machine_learning_to_improve_DR_forecasts_-_white_paper

Tyler Byers

Viewers also liked (20)

Big Data Analytics in Energy & Utilities

Smart Analytics For The Utility Sector

Supercharging Smart Meter BIG DATA Analytics with Microsoft Azure Cloud- SRP ...

Energy Data Analytics | Energy Efficiency | India

The Soft Grid 2013 Opening Presentation

Smart metering infrastructure Architecture and analytics

Smart Meter Basics and Benefits

Meter Data Management Presentation

Meter Data Management 2.0

What is Big Data?

Whitepaper - Transforming the Energy & Utilities Industry with Smart Analytics

Concepts of smart meter

Smart Meters

Smart energy meter (Updated)

Big Data solution for multi-national Bank

Intro to Data Science on Hadoop

Data Science for Energy Efficiency (Dmytro Mindra Technology Stream)

Big Data : a 360° Overview

Why not to ignore #Data while designing Industrial #EnergyEfficiency Policy

Using_machine_learning_to_improve_DR_forecasts_-_white_paper

Similar to Smart Meter Data Analytic using Hadoop

Real time big data applications with hadoop ecosystem

Chris Huang

ETL: Transformations and Other Operators: STORE mapping, Adding source and target operators, Adding Transformation Operators, Using a Key Lookup operator, Creating an external table, Creating and loading a lookup table, Retrieving the key to use for a Lookup Operator, Adding a Key Lookup operator, PRODUCT mapping, SALES cube mapping, Dimension attributes in the cube, Measures and other attributes in the cube, Mapping values to cube attributes, Mapping measures' values to a cube, Mapping PRODUCT and STORE dimension values to the cube, Mapping DATE_DIM values to the cube, Features and benefits of OWB.

Data warehousing unit 6.2

WE-IT TUTORIALS

PDO Predictive Analytics Share for the Annual Research Forum 2015

Faris Al-Kharusi

Daily monitoring cbm of switchyard equipments through android app

Niraj Agrawal

Svm Classifier Algorithm for Data Stream Mining Using Hive and R

IRJET Journal

ASTQB washington-sept-2015

Dan Boutin

Cloud expo june 2013: Building a Real Time Analytics Platform on Big Data in ...

Sanjay Sharma

IRJET- Analysis for EnhancedForecastof Expense Movement in Stock Exchange

IRJET Journal

Talk about Hivemall at Data Scientist Organization on 2015/09/17

Makoto Yui

Whither the Hadoop Developer Experience, June Hadoop Meetup, Nitin Motgi

Felicia Haggarty

A Novel Framework on Web Usage Mining

IRJET Journal

Db tech show - hivemall

Makoto Yui

Key Considerations for Putting Hadoop in Production SlideShare

MapR Technologies

Oil and gas exploration and production activities generate large amounts of data from sensors, logistics, business operations and more. Given the data volume, variety and velocity, gaining actionable and relevant insights from the data is challenging. Learn about these challenges and how to address them by leveraging big data technologies in this webinar. During the webinar we will dive deep into approaches for predicting drilling equipment function and failure, a key step towards zero unplanned downtime. In the process of drilling wells, non-productive time due to drilling equipment failure can be expensive. We will highlight how the Pivotal Data Labs team uses big data technologies to build models for predicting drilling equipment function and failure. Models such as these can be used to build essential early warning systems to reduce costs and minimize unplanned downtime. Panelist: Rashmi Raghu, Senior Data Scientist, Pivotal Hosted by: Tim Matteson, Co-Founder -- Data Science Central Video replay is available to watch here: http://youtu.be/dhT-tjHCr9E

Data as the New Oil: Producing Value in the Oil and Gas Industry

VMware Tanzu

IPv6 status among ASEAN Member States

APNIC

How Comcast Turns Big Data into Real Time Operational Insights: Winter Olympi...

Brett Sheppard

Stream Analytics

Franco Ucci

Marta de Mesa i Jesus Gironda, de Telvent, presenten les possibilitats d'aplicar el Big Data més enllà del sector privat. Per exemple, en la previsió i planificació de recursos interns a les universitats. Aquesta presentació ha tingut lloc a la TSIUC'14, celebrada a la Universitat Autònoma de Barcelona el passat 2 de desembre de 2014, sota el títol "Reptes en Big Data a la universitat i la Recerca".

Telvent Big Data Approach and Case Studies

CSUC - Consorci de Serveis Universitaris de Catalunya

IRJET- An Efficient Data Replication in Salesforce Cloud Environment

IRJET Journal

Patrick Orengo - Hallbäck

Similar to Smart Meter Data Analytic using Hadoop (20)

Real time big data applications with hadoop ecosystem

Data warehousing unit 6.2

PDO Predictive Analytics Share for the Annual Research Forum 2015

Daily monitoring cbm of switchyard equipments through android app

Svm Classifier Algorithm for Data Stream Mining Using Hive and R

ASTQB washington-sept-2015

Cloud expo june 2013: Building a Real Time Analytics Platform on Big Data in ...

IRJET- Analysis for EnhancedForecastof Expense Movement in Stock Exchange

Talk about Hivemall at Data Scientist Organization on 2015/09/17

Whither the Hadoop Developer Experience, June Hadoop Meetup, Nitin Motgi

A Novel Framework on Web Usage Mining

Db tech show - hivemall

Key Considerations for Putting Hadoop in Production SlideShare

Data as the New Oil: Producing Value in the Oil and Gas Industry

IPv6 status among ASEAN Member States

How Comcast Turns Big Data into Real Time Operational Insights: Winter Olympi...

Stream Analytics

Telvent Big Data Approach and Case Studies

IRJET- An Efficient Data Replication in Salesforce Cloud Environment

More from DataWorks Summit

Introduction: This workshop will provide a hands-on introduction to Machine Learning (ML) with an overview of Deep Learning (DL). Format: An introductory lecture on several supervised and unsupervised ML techniques followed by light introduction to DL and short discussion what is current state-of-the-art. Several python code samples using the scikit-learn library will be introduced that users will be able to run in the Cloudera Data Science Workbench (CDSW). Objective: To provide a quick and short hands-on introduction to ML with python’s scikit-learn library. The environment in CDSW is interactive and the step-by-step guide will walk you through setting up your environment, to exploring datasets, training and evaluating models on popular datasets. By the end of the crash course, attendees will have a high-level understanding of popular ML algorithms and the current state of DL, what problems they can solve, and walk away with basic hands-on experience training and evaluating ML models. Prerequisites: For the hands-on portion, registrants must bring a laptop with a Chrome or Firefox web browser. These labs will be done in the cloud, no installation needed. Everyone will be able to register and start using CDSW after the introductory lecture concludes (about 1hr in). Basic knowledge of python highly recommended.

Data Science Crash Course

DataWorks Summit

In a world with a myriad of distributed storage systems to choose from, the majority of Apache HBase clusters still rely on Apache HDFS. Theoretically, any distributed file system could be used by HBase. One major reason HDFS is predominantly used are the specific durability requirements of HBase's write-ahead log (WAL) and HDFS providing that guarantee correctly. However, HBase's use of HDFS for WALs can be replaced with sufficient effort. This talk will cover the design of a "Log Service" which can be embedded inside of HBase that provides a sufficient level of durability that HBase requires for WALs. Apache Ratis (incubating) is a library-implementation of the RAFT consensus protocol in Java and is used to build this Log Service. We will cover the design choices of the Ratis Log Service, comparing and contrasting it to other log-based systems that exist today. Next, we'll cover how the Log Service "fits" into HBase and the necessary changes to HBase which enable this. Finally, we'll discuss how the Log Service can simplify the operational burden of HBase.

Floating on a RAFT: HBase Durability with Apache Ratis

DataWorks Summit

Utilizing Apache NiFi we read various open data REST APIs and camera feeds to ingest crime and related data real-time streaming it into HBase and Phoenix tables. HBase makes an excellent storage option for our real-time time series data sources. We can immediately query our data utilizing Apache Zeppelin against Phoenix tables as well as Hive external tables to HBase. Apache Phoenix tables also make a great option since we can easily put microservices on top of them for application usage. I have an example Spring Boot application that reads from our Philadelphia crime table for front-end web applications as well as RESTful APIs. Apache NiFi makes it easy to push records with schemas to HBase and insert into Phoenix SQL tables. Resources: https://community.hortonworks.com/articles/54947/reading-opendata-json-and-storing-into-phoenix-tab.html https://community.hortonworks.com/articles/56642/creating-a-spring-boot-java-8-microservice-to-read.html https://community.hortonworks.com/articles/64122/incrementally-streaming-rdbms-data-to-your-hadoop.html

Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi

DataWorks Summit

Whilst HBase is the most logical answer for use cases requiring random, realtime read/write access to Big Data, it may not be so trivial to design applications that make most of its use, neither the most simple to operate. As it depends/integrates with other components from Hadoop ecosystem (Zookeeper, HDFS, Spark, Hive, etc) or external systems ( Kerberos, LDAP), and its distributed nature requires a "Swiss clockwork" infrastructure, many variables are to be considered when observing anomalies or even outages. Adding to the equation there's also the fact that HBase is still an evolving product, with different release versions being used currently, some of those can carry genuine software bugs. On this presentation, we'll go through the most common HBase issues faced by different organisations, describing identified cause and resolution action over my last 5 years supporting HBase to our heterogeneous customer base.

HBase Tales From the Trenches - Short stories about most common HBase operati...

DataWorks Summit

LocationTech GeoMesa enables spatial and spatiotemporal indexing and queries for HBase and Accumulo. In this talk, after an overview of GeoMesa’s capabilities in the Cloudera ecosystem, we will dive into how GeoMesa leverages Accumulo’s Iterator interface and HBase’s Filter and Coprocessor interfaces. The goal will be to discuss both what spatial operations can be pushed down into the distributed database and also how the GeoMesa codebase is organized to allow for consistent use across the two database systems.

Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...

DataWorks Summit

Managing the Dewey Decimal System

DataWorks Summit

Many individuals/organizations have a desire to utilize NoSQL technology, but often lack an understanding of how the underlying functional bits can be utilized to enable their use case. This situation can result in drastic increases in the desire to put the SQL back in NoSQL. Since the initial commit, Apache Accumulo has provided a number of examples to help jumpstart comprehension of how some of these bits function as well as potentially help tease out an understanding of how they might be applied to a NoSQL friendly use case. One very relatable example demonstrates how Accumulo could be used to emulate a filesystem (dirlist). In this session we will walk through the dirlist implementation. Attendees should come away with an understanding of the supporting table designs, a simple text search supporting a single wildcard (on file/directory names), and how the dirlist elements work together to accomplish its feature set. Attendees should (hopefully) also come away with a justification for sometimes keeping the SQL out of NoSQL.

Practical NoSQL: Accumulo's dirlist Example

DataWorks Summit

Data serves as the platform for decision-making at Uber. To facilitate data driven decisions, many datasets at Uber are ingested in a Hadoop Data Lake and exposed to querying via Hive. Analytical queries joining various datasets are run to better understand business data at Uber. Data ingestion, at its most basic form, is about organizing data to balance efficient reading and writing of newer data. Data organization for efficient reading involves factoring in query patterns to partition data to ensure read amplification is low. Data organization for efficient writing involves factoring the nature of input data - whether it is append only or updatable. At Uber we ingest terabytes of many critical tables such as trips that are updatable. These tables are fundamental part of Uber's data-driven solutions, and act as the source-of-truth for all the analytical use-cases across the entire company. Datasets such as trips constantly receive updates to the data apart from inserts. To ingest such datasets we need a critical component that is responsible for bookkeeping information of the data layout, and annotates each incoming change with the location in HDFS where this data should be written. This component is called as Global Indexing. Without this component, all records get treated as inserts and get re-written to HDFS instead of being updated. This leads to duplication of data, breaking data correctness and user queries. This component is key to scaling our jobs where we are now handling greater than 500 billion writes a day in our current ingestion systems. This component will need to have strong consistency and provide large throughputs for index writes and reads. At Uber, we have chosen HBase to be the backing store for the Global Indexing component and is a critical component in allowing us to scaling our jobs where we are now handling greater than 500 billion writes a day in our current ingestion systems. In this talk, we will discuss data@Uber and expound more on why we built the global index using Apache Hbase and how this helps to scale out our cluster usage. We’ll give details on why we chose HBase over other storage systems, how and why we came up with a creative solution to automatically load Hfiles directly to the backend circumventing the normal write path when bootstrapping our ingestion tables to avoid QPS constraints, as well as other learnings we had bringing this system up in production at the scale of data that Uber encounters daily.

HBase Global Indexing to support large-scale data ingestion at Uber

DataWorks Summit

Recently, Apache Phoenix has been integrated with Apache (incubator) Omid transaction processing service, to provide ultra-high system throughput with ultra-low latency overhead. Phoenix has been shown to scale beyond 0.5M transactions per second with sub-5ms latency for short transactions on industry-standard hardware. On the other hand, Omid has been extended to support secondary indexes, multi-snapshot SQL queries, and massive-write transactions. These innovative features make Phoenix an excellent choice for translytics applications, which allow converged transaction processing and analytics. We share the story of building the next-gen data tier for advertising platforms at Verizon Media that exploits Phoenix and Omid to support multi-feed real-time ingestion and AI pipelines in one place, and discuss the lessons learned.

Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix

DataWorks Summit

Cybersecurity requires an organization to collect data, analyze it, and alert on cyber anomalies in near real-time. This is a challenging endeavor when considering the variety of data sources which need to be collected and analyzed. Everything from application logs, network events, authentications systems, IOT devices, business events, cloud service logs, and more need to be taken into consideration. In addition, multiple data formats need to be transformed and conformed to be understood by both humans and ML/AI algorithms. To solve this problem, the Aetna Global Security team developed the Unified Data Platform based on Apache NiFi, which allows them to remain agile and adapt to new security threats and the onboarding of new technologies in the Aetna environment. The platform currently has over 60 different data flows with 95% doing real-time ETL and handles over 20 billion events per day. In this session learn from Aetna’s experience building an edge to AI high-speed data pipeline with Apache NiFi.

Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi

DataWorks Summit

Supporting Apache HBase : Troubleshooting and Supportability Improvements

DataWorks Summit

In the healthcare sector, data security, governance, and quality are crucial for maintaining patient privacy and ensuring the highest standards of care. At Florida Blue, the leading health insurer of Florida serving over five million members, there is a multifaceted network of care providers, business users, sales agents, and other divisions relying on the same datasets to derive critical information for multiple applications across the enterprise. However, maintaining consistent data governance and security for protected health information and other extended data attributes has always been a complex challenge that did not easily accommodate the wide range of needs for Florida Blue’s many business units. Using Apache Ranger, we developed a federated Identity & Access Management (IAM) approach that allows each tenant to have their own IAM mechanism. All user groups and roles are propagated across the federation in order to determine users’ data entitlement and access authorization; this applies to all stages of the system, from the broadest tenant levels down to specific data rows and columns. We also enabled audit attributes to ensure data quality by documenting data sources, reasons for data collection, date and time of data collection, and more. In this discussion, we will outline our implementation approach, review the results, and highlight our “lessons learned.”

Security Framework for Multitenant Architecture

DataWorks Summit

Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources. Proven at scale in a variety of use cases at Airbnb, Bloomberg, Comcast, Facebook, FINRA, LinkedIn, Lyft, Netflix, Twitter, and Uber, in the last few years Presto experienced an unprecedented growth in popularity in both on-premises and cloud deployments over Object Stores, HDFS, NoSQL and RDBMS data stores. With the ever-growing list of connectors to new data sources such as Azure Blob Storage, Elasticsearch, Netflix Iceberg, Apache Kudu, and Apache Pulsar, recently introduced Cost-Based Optimizer in Presto must account for heterogeneous inputs with differing and often incomplete data statistics. This talk will explore this topic in detail as well as discuss best use cases for Presto across several industries. In addition, we will present recent Presto advancements such as Geospatial analytics at scale and the project roadmap going forward.

Presto: Optimizing Performance of SQL-on-Anything Engine

DataWorks Summit

Specialized tools for machine learning development and model governance are becoming essential. MlFlow is an open source platform for managing the machine learning lifecycle. Just by adding a few lines of code in the function or script that trains their model, data scientists can log parameters, metrics, artifacts (plots, miscellaneous files, etc.) and a deployable packaging of the ML model. Every time that function or script is run, the results will be logged automatically as a byproduct of those lines of code being added, even if the party doing the training run makes no special effort to record the results. MLflow application programming interfaces (APIs) are available for the Python, R and Java programming languages, and MLflow sports a language-agnostic REST API as well. Over a relatively short time period, MLflow has garnered more than 3,300 stars on GitHub , almost 500,000 monthly downloads and 80 contributors from more than 40 companies. Most significantly, more than 200 companies are now using MLflow. We will demo MlFlow Tracking , Project and Model components with Azure Machine Learning (AML) Services and show you how easy it is to get started with MlFlow on-prem or in the cloud.

Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...

DataWorks Summit

Twitter's Data Platform is built using multiple complex open source and in house projects to support Data Analytics on hundreds of petabytes of data. Our platform support storage, compute, data ingestion, discovery and management and various tools and libraries to help users for both batch and realtime analytics. Our DataPlatform operates on multiple clusters across different data centers to help thousands of users discover valuable insights. As we were scaling our Data Platform to multiple clusters, we also evaluated various cloud vendors to support use cases outside of our data centers. In this talk we share our architecture and how we extend our data platform to use cloud as another datacenter. We walk through our evaluation process, challenges we faced supporting data analytics at Twitter scale on cloud and present our current solution. Extending Twitter's Data platform to cloud was complex task which we deep dive in this presentation.

Extending Twitter's Data Platform to Google Cloud

DataWorks Summit

At Comcast, our team has been architecting a customer experience platform which is able to react to near-real-time events and interactions and deliver appropriate and timely communications to customers. By combining the low latency capabilities of Apache Flink and the dataflow capabilities of Apache NiFi we are able to process events at high volume to trigger, enrich, filter, and act/communicate to enhance customer experiences. Apache Flink and Apache NiFi complement each other with their strengths in event streaming and correlation, state management, command-and-control, parallelism, development methodology, and interoperability with surrounding technologies. We will trace our journey from starting with Apache NiFi over three years ago and our more recent introduction of Apache Flink into our platform stack to handle more complex scenarios. In this presentation we will compare and contrast which business and technical use cases are best suited to which platform and explore different ways to integrate the two platforms into a single solution.

Event-Driven Messaging and Actions using Apache Flink and Apache NiFi

DataWorks Summit

Companies are increasingly moving to the cloud to store and process data. One of the challenges companies have is in securing data across hybrid environments with easy way to centrally manage policies. In this session, we will talk through how companies can use Apache Ranger to protect access to data both in on-premise as well as in cloud environments. We will go into details into the challenges of hybrid environment and how Ranger can solve it. We will also talk through how companies can further enhance the security by leveraging Ranger to anonymize or tokenize data while moving into the cloud and de-anonymize dynamically using Apache Hive, Apache Spark or when accessing data from cloud storage systems. We will also deep dive into the Ranger’s integration with AWS S3, AWS Redshift and other cloud native systems. We will wrap it up with an end to end demo showing how policies can be created in Ranger and used to manage access to data in different systems, anonymize or de-anonymize data and track where data is flowing.

Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger

DataWorks Summit

Advanced Big Data Processing frameworks have been proposed to harness the fast data transmission capability of Remote Direct Memory Access (RDMA) over high-speed networks such as InfiniBand, RoCEv1, RoCEv2, iWARP, and OmniPath. However, with the introduction of the Non-Volatile Memory (NVM) and NVM express (NVMe) based SSD, these designs along with the default Big Data processing models need to be re-assessed to discover the possibilities of further enhanced performance. In this talk, we will present, NRCIO, a high-performance communication runtime for non-volatile memory over modern network interconnects that can be leveraged by existing Big Data processing middleware. We will show the performance of non-volatile memory-aware RDMA communication protocols using our proposed runtime and demonstrate its benefits by incorporating it into a high-performance in-memory key-value store, Apache Hadoop, Tez, Spark, and TensorFlow. Evaluation results illustrate that NRCIO can achieve up to 3.65x performance improvement for representative Big Data processing workloads on modern data centers.

Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...

DataWorks Summit

Background: Some early applications of Computer Vision in Retail arose from e-commerce use cases - but increasingly, it is being used in physical stores in a variety of new and exciting ways, such as: ● Optimizing merchandising execution, in-stocks and sell-thru ● Enhancing operational efficiencies, enable real-time customer engagement ● Enhancing loss prevention capabilities, response time ● Creating frictionless experiences for shoppers Abstract: This talk will cover the use of Computer Vision in Retail, the implications to the broader Consumer Goods industry and share business drivers, use cases and benefits that are unfolding as an integral component in the remaking of an age-old industry. We will also take a ‘peek under the hood’ of Computer Vision and Deep Learning, sharing technology design principles and skill set profiles to consider before starting your CV journey. Deep learning has matured considerably in the past few years to produce human or superhuman abilities in a variety of computer vision paradigms. We will discuss ways to recognize these paradigms in retail settings, collect and organize data to create actionable outcomes with the new insights and applications that deep learning enables. We will cover the basics of object detection, then move into the advanced processing of images describing the possible ways that a retail store of the near future could operate. Identifying various storefront situations by having a deep learning system attached to a camera stream. Such things as; identifying item stocks on shelves, a shelf in need of organization, or perhaps a wandering customer in need of assistance. We will also cover how to use a computer vision system to automatically track customer purchases to enable a streamlined checkout process, and how deep learning can power plausible wardrobe suggestions based on what a customer is currently wearing or purchasing. Finally, we will cover the various technologies that are powering these applications today. Deep learning tools for research and development. Production tools to distribute that intelligence to an entire inventory of all the cameras situation around a retail location. Tools for exploring and understanding the new data streams produced by the computer vision systems. By the end of this talk, attendees should understand the impact Computer Vision and Deep Learning are having in the Consumer Goods industry, key use cases, techniques and key considerations leaders are exploring and implementing today.

Computer Vision: Coming to a Store Near You

DataWorks Summit

Whole genome shotgun based next generation transcriptomics and metagenomics studies often generate 100 to 1000 gigabytes (GB) sequence data derived from tens of thousands of different genes or microbial species. De novo assembling these data requires an ideal solution that both scales with data size and optimizes for individual gene or genomes. Here we developed an Apache Spark-based scalable sequence clustering application, SparkReadClust (SpaRC), that partitions the reads based on their molecule of origin to enable downstream assembly optimization. SpaRC produces high clustering performance on transcriptomics and metagenomics test datasets from both short read and long read sequencing technologies. It achieved a near linear scalability with respect to input data size and number of compute nodes. SpaRC can run on different cloud computing environments without modifications while delivering similar performance. In summary, our results suggest SpaRC provides a scalable solution for clustering billions of reads from the next-generation sequencing experiments, and Apache Spark represents a cost-effective solution with rapid development/deployment cycles for similar big data genomics problems.

Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark

DataWorks Summit

More from DataWorks Summit (20)

Data Science Crash Course

Floating on a RAFT: HBase Durability with Apache Ratis

Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi

HBase Tales From the Trenches - Short stories about most common HBase operati...

Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...

Managing the Dewey Decimal System

Practical NoSQL: Accumulo's dirlist Example

HBase Global Indexing to support large-scale data ingestion at Uber

Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix

Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi

Supporting Apache HBase : Troubleshooting and Supportability Improvements

Security Framework for Multitenant Architecture

Presto: Optimizing Performance of SQL-on-Anything Engine

Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...

Extending Twitter's Data Platform to Google Cloud

Event-Driven Messaging and Actions using Apache Flink and Apache NiFi

Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger

Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...

Computer Vision: Coming to a Store Near You

Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark

Recently uploaded

The microservices honeymoon is over. When starting a new project or revamping a legacy monolith, teams started looking for alternatives to microservices. The Modular Monolith, or 'Modulith', is an architecture that reaps the benefits of (vertical) functional decoupling without the high costs associated with separate deployments. This talk will delve into the advantages and challenges of this progressive architecture, beginning with exploring the concept of a 'module', its internal structure, public API, and inter-module communication patterns. Supported by spring-modulith, the talk provides practical guidance on addressing the main challenges of a Modultith Architecture: finding and guarding module boundaries, data decoupling, and integration module-testing. You should not miss this talk if you are a software architect or tech lead seeking practical, scalable solutions. About the author With two decades of experience, Victor is a Java Champion working as a trainer for top companies in Europe. Five thousands developers in 120 companies attended his workshops, so he gets to debate every week the challenges that various projects struggle with. In return, Victor summarizes key points from these workshops in conference talks and online meetups for the European Software Crafters, the world’s largest developer community around architecture, refactoring, and testing. Discover how Victor can help you on victorrentea.ro : company training catalog, consultancy and YouTube playlists.

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024

Victor Rentea

Webinar Recording: https://www.panagenda.com/webinars/why-teams-call-analytics-is-critical-to-your-entire-business Nothing is as frustrating and noticeable as being in an important call and being unable to see or hear the other person. Not surprising then, that issues with Teams calls are among the most common problems users call their helpdesk for. Having in depth insight into everything relevant going on at the user’s device, local network, ISP and Microsoft itself during the call is crucial for good Microsoft Teams Call quality support. To ensure a quick and adequate solution and to ensure your users get the most out of their Microsoft 365. But did you know that ‘bad calls’ are also an excellent indicator of other problems arising? Precisely because it is so noticeable!? Like the canary in the mine, bad calls can be early indicators of problems. Problems that might otherwise not have been noticed for a while but can have a big impact on productivity and satisfaction. Join this session by Christoph Adler to learn how true Microsoft Teams call quality analytics helped other organizations troubleshoot bad calls and identify and fix problems that impacted Teams calls or the use of Microsoft365 in general. See what it can do to keep your users happy and productive! In this session we will cover - Why CQD data alone is not enough to troubleshoot call problems - The importance of attributing call problems to the right call participant - What call quality analytics can do to help you quickly find, fix-, and prevent problems - Why having retrospective detailed insights matters - Real life examples of how others have used Microsoft Teams call quality monitoring to problem shoot problems with their ISP, network, device health and more.

Why Teams call analytics are critical to your entire business

panagenda

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

FWD Group - Insurer Innovation Award 2024

The Digital Insurer

JohnPollard-hybrid-app-RailsConf2024.pptx

JohnPollard37

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

apidays

Understanding the FAA Part 107 License ..

Christopher Logan Kennedy

Vector Search -An Introduction in Oracle Database 23ai.pptx

Remote DBA Services

Exploring Multimodal Embeddings with Milvus

Zilliz

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

DBX First Quarter 2024 Investor Presentation

Dropbox

The Good, the Bad and the Governed - Why is governance a dirty word? David O'Neill, Chief Operating Officer - APIContext Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

apidays

Following the popularity of “Cloud Revolution: Exploring the New Wave of Serverless Spatial Data,” we’re thrilled to announce this much-anticipated encore webinar. In this sequel, we’ll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you’re building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

Keynote 2: APIs in 2030: The Risk of Technological Sleepwalk Paolo Malinverno, Growth Advisor - The Business of Technology Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...

apidays

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

Introduction to use of FHIR Documents in ABDM

Kumar Satyam

[BuildWithAI] Introduction to Gemini.pdf

Sandro Moreira

Dubai, often portrayed as a shimmering oasis in the desert, faces its own set of challenges, including the occasional threat of flooding. Despite its reputation for opulence and modernity, the emirate is not immune to the forces of nature. In recent years, Dubai has experienced sporadic but significant floods, testing the resilience of its infrastructure and communities. Among the critical lifelines in this bustling metropolis is the Dubai International Airport, a bustling hub that connects the city to the world. This article explores the intersection of Dubai flood events and the resilience demonstrated by the Dubai International Airport in the face of such challenges.

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

Orbitshub

When you’re building (micro)services, you have lots of framework options. Spring Boot is no doubt a popular choice. But there’s more! Take Quarkus, a framework that’s considered the rising star for Kubernetes-native Java. It always depends on what's best for your situation, but how to choose the best solution if you're comparing 2 frameworks? Both Spring Boot and Quarkus have their positives and negatives. Let us compare the two by live coding a couple of common use cases in Spring Boot and Quarkus. After this talk, you’ll be ready to get started with Quarkus yourself, and know when to select Quarkus or Spring Boot.

Spring Boot vs Quarkus the ultimate battle - DevoxxUK

Jago de Vreede

Recently uploaded (20)

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024

Why Teams call analytics are critical to your entire business

Strategies for Landing an Oracle DBA Job as a Fresher

FWD Group - Insurer Innovation Award 2024

JohnPollard-hybrid-app-RailsConf2024.pptx

Apidays New York 2024 - The value of a flexible API Management solution for O...

Understanding the FAA Part 107 License ..

Vector Search -An Introduction in Oracle Database 23ai.pptx

Exploring Multimodal Embeddings with Milvus

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

DBX First Quarter 2024 Investor Presentation

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Introduction to use of FHIR Documents in ABDM

[BuildWithAI] Introduction to Gemini.pdf

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

Spring Boot vs Quarkus the ultimate battle - DevoxxUK

Smart Meter Data Analytic using Hadoop

1. Smart Meter Data Analytic Using Hadoop Omkar Nibandhe and Abhishek Korpe students

2. SMART METER DATA ANALYTICS (SMDA) USING HADOOP By : Omkar Nibandhe ( Student ) Abhishek Korpe ( Student ) 03-04-2014SMDAusingHADOOP 2

3. Why Smart Meter Data Analytics ? 03-04-2014SMDAusingHADOOP 3

4. What are SMART METERS ? MIF • Track and store the amount of energy used. • Send the collected data to the Energy Distribution company server at regular time intervals. House/ Industry Smart Meter Server 03-04-2014SMDAusingHADOOP 4

5. Advantages • Service Provider • Demand-Response • Time of use tariff • Load Profile Analysis • Theft Detection • Billing Accuracy • Customers • Usage Pattern • Billing Accuracy • Convenience in change of service provider 03-04-2014SMDAusingHADOOP 5

6. Confronting the data deluge • Rate Generation – 15 minutes. • For single meter – 3000 readings/month (approx ). • For 1 million meters – 36 Billion readings/year (approx). • Annual Growth – 13% ( 2010 – 2015 ). • Total Shipment – 460.9 Million Smart Meters. Source: Build smart metering solutions with IBM Informix TimeSeries 03-04-2014SMDAusingHADOOP 6

7. Capitalizing on the unique value of Hadoop Solution ? • Reducing data load times. • Improving query performance. • Massive Scalability. 03-04-2014SMDAusingHADOOP 7

8. Demand - Response 03-04-2014SMDAusingHADOOP 8

9. Time of use Tariff 03-04-2014SMDAusingHADOOP 9

10. Load Profile Analysis Using hadoopUsing hadoop load time 03-04-2014SMDAusingHADOOP 10

11. Data Flow Diagram Predictive Analysis ( FLUME ) Load Profile Analysis 03-04-2014SMDAusingHADOOP 11

12. Hadoop Cluster in LAB 412 Masters Slaves Slave1 Slave7 Slave8 Slave9 Slave11 Slave6Slave5Slave4Slave3Slave2 Slave10 Slave19Slave18 Slave17Slave16Slave15Slave14Slave13 Slave12 Slave20 Slave21 03-04-2014SMDAusingHADOOP 12

13. LAB 412 (MESCOE Pune, India) 03-04-2014SMDAusingHADOOP 13

14. SMDA - NameNode 03-04-2014SMDAusingHADOOP 14

15. SMDA - SecondaryNameNode 03-04-2014SMDAusingHADOOP 15

16. SMDA - JobTracker 03-04-2014SMDAusingHADOOP 16

17. SMDA – Input ( .MIF ) 03-04-2014SMDAusingHADOOP 17 1392 19501 0.157

18. SMDA - Output 03-04-2014SMDAusingHADOOP 18

19. Test Job 1 03-04-2014SMDAusingHADOOP 19

20. Test Job 2 ( Combiner ) 03-04-2014SMDAusingHADOOP 20

21. Future Scope • Analysis of - • Customer segmentation. • Customer behavior. • Meter ping commands. • Outage management. • Power quality. • Extending data point(s) : weather, geographical location, family consumption, etc. 03-04-2014SMDAusingHADOOP 21

22. Thanks to …. • Irish Social Science Data Archive ( ISSDA ). • Rahul Khinvasara – Director, zCon Solutions Pvt. Ltd. • Modern Education Society’s College of Engineering (MESCOE). • Prof. Balaji Bodkhe – Guide, MESCOE. • Prof. N. Shaikh – Head of Computer Department, MESCOE. • Prof. A. Hake – Vice Principal, MESCOE. • Prof. P. Raut – Administrative Head, MESCOE. 03-04-2014SMDAusingHADOOP 22

23. Any Suggestions ?? 03-04-2014SMDAusingHADOOP 23

24. Thank You. 03-04-2014SMDAusingHADOOP 24

25. Any Questions? 03-04-2014SMDAusingHADOOP 25

Smart Meter Data Analytic using Hadoop

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Smart Meter Data Analytic using Hadoop

Similar to Smart Meter Data Analytic using Hadoop (20)

More from DataWorks Summit

More from DataWorks Summit (20)

Recently uploaded

Recently uploaded (20)

Smart Meter Data Analytic using Hadoop