SlideShare a Scribd company logo
1 of 20
Download to read offline
WHITE PAPER
LEVERAGING BIG DATA AND REAL-TIME
ANALYTICS TO ACHIEVE SITUATIONAL
AWARENESS FOR SMART GRIDS
PART ONE IN VERSANT’S SCALABLE SITUATIONAL
AWARENESS SERIES
By Bert Taube, Director of Energy and Smart Grid Solutions
Versant Corporation
Sponsored by Versant Corporation
Versant Corporation U.S. Headquarters
255 Shoreline Dr. Suite 450, Redwood City, CA 94065
www.versant.com  +1 650-232-2400
Leveraging Big Data and Real-Time Analytics to
Achieve Situational Awareness for Smart Grids
Page 2 of 20
EXECUTIVE SUMMARY:
WHY THE PURPOSE OF BIG DATA IS REALLY
SITUATIONAL AWARENESS
The concept of Big Data has been around for more than a decade,
and its potential to transform the effectiveness, efficiency, and
profitability of virtually any enterprise has been well documented.
Yet, despite the concept of Big Data being well-defined, and the
general enormity of its opportunity well-understood, the means to
effectively leverage Big Data and realize its promised benefits still
eludes many.
Big Data’s remaining challenge to realizing these benefits comes in
two parts. The first is to understand that the true purpose of
leveraging Big Data is to take action - to make more accurate
decisions, more quickly. We call this situational awareness, an idea
that is quite self-explanatory. Regardless of industry or
environment, situational awareness means having an
understanding of what you need to know, have control of, and
conduct analysis for in real-time to identify anomalies in normal
patterns or behaviors that can affect the outcome of a business or
process. If you have these things, making the right decision in the
right amount of time in any context becomes much easier.
Although the term of situational awareness itself is fairly recent, the
concept has roots in the history of military theory - it is recognizable
in Sun Tzu's The Art of War, for instance. The term itself, can also
be traced to World War I, where it was recognized as a crucial
component for crews in military aircraft. Before being widely
adopted by human factors scientists in the 1990s, the term was first
used by United States Air Force (USAF) fighter aircrews returning
from war in Korea and Vietnam. Today, the concept of situational
awareness has been expanded to a variety of areas such as air
traffic control, nuclear power plant operation, vehicle operation, and
anesthesiology. Defining the parameters of situational awareness
for any industry is not simple, and thus surmounting Big Data’s
other remaining challenge of creating new approaches to data
management and analysis to support these needs is also no small
feat. Achieving situational awareness used to be much easier
because data volumes were smaller, and new data was created at
a slower rate, which meant our worlds were defined by a much
smaller amount of information. But new data is now created at a
hugely exponential rate, and therefore any data management and
analysis system that is built to provide situational awareness today
must also be able to do so tomorrow. Thus, the imperative for any
enterprise is to create systems that manage Big Data and provide
scalable situational awareness.
Table of Contents
Section 1: The Smart
Grid Network’s Unique
Big Data Demands
Section 2: Utilities’
Current Data Analytics
and Management
Methods, and the Need
for Change
Section 3:
Requirements of Data
Analytics Systems to
Create Scalable
Situational Awareness
for Smart Grids
Section 4: Versant and
The Object-Oriented
DB’s Pedigree, and
their Value for Smart
Grid Situational
Awareness
Section 5: How To
Apply Object-Oriented
Data Management and
Analysis Technologies
in Smart Grids
Section 6: Benefits of
Scalable Situational
Awareness through
Object-Oriented Data
Management and
Analysis for Smart
Grids
Conclusion
Leveraging Big Data and Real-Time Analytics to
Achieve Situational Awareness for Smart Grids
Page 3 of 20
The utilities industry is in particular need of scalable situational
awareness so that it can realize benefits for a wide range of
important functions that are critical for enabling Smart Grid
paradigms. Scalable situational awareness for utilities means
knowing where power is needed, and where it can be taken from,
to keep the grid stable. When power flow is not well understood,
the resulting consequences can quite literally leave utilities and
their customers in the dark: a fitting-though-ironic analogy
considering the goal of awareness. .
Utilities can learn much about how to achieve scalable situational
awareness from other industries, most notably building
management and telecommunications, which have learned to deal
with Big Data’s complexity and scale well.
This white paper will describe the diversity of utility data, the criteria
which define a big energy data problem, and how utility business
value can be built through increased situational awareness. The
various problems that arise from storing the large number of data
types in smart grid systems using traditional software technologies
(data historians and relational databases) will be summarized.
Overcoming these problems can be accomplished by applying
NoSQL-based data management and analytics solutions like
Versant’s, which can be seamlessly integrated with object-oriented
languages. A power outage example will be utilized to demonstrate
how Versant’s NoSQL solutions have enabled grid reliability and
situational awareness, and how they can help utilities achieve other
critical business goals.
SECTION 1: THE SMART GRID NETWORK’S
UNIQUE BIG DATA DEMANDS
In 2001, Doug Laney, now an analyst with Gartner, created what
has become the widely-accepted three-dimensional definition of
Big Data, also known as the Three V’s of Big Data: volume,
velocity, and variety.
Laney’s definition, though, was created for the paradigm of a
typical business, such as manufacturing, where profitability is often
achieved by the minimization of fixed assets, where work in
progress is measured in days, weeks, or months, and real-time
data collection and analysis are often not critical to ensure the
profitability of the organization. The value chain for manufacturing
almost always crosses company boundaries. However, in the utility
industry, there are vertically-integrated and deregulated variants
that have to act exactly the same, increasing complexity.
Leveraging Big Data and Real-Time Analytics to
Achieve Situational Awareness for Smart Grids
Page 4 of 20
In this environment, the acquisition of real-time data can both be
costly and can seriously impact the bottom line. This adds more
dimensions to the utility industry’s Big Data needs as enterprises
must not only deal with data’s volume, velocity, and variety
challenges, but also with two new V’s: validity and veracity.
Validity - Information in the utility environment often has a “shelf
life” and therefore may only be needed for storage and evaluation
for a fixed period of time. The questions of when to archive or
dispose of data become relevant given the cost of storing large
volumes of data.
Veracity – This fifth variable recognizes that data is not perfect, and
that achieving “perfect” data carries a cost. Utilities must consider
two questions: 1) how good must the data be to achieve the
necessary level of accurate analysis, and 2) at what point does the
cost of correcting the data exceed the benefit of obtaining it?
The utility industry’s time scales vary over 15 orders of magnitude
due to the unique diversity of sensors and critical business
processes, and often at much faster intervals than other industries,
which, when trying to create scalable situational awareness,
impacts all five V’s of the industry’s Big Data pressures.
Image Caption: Data from Utilities’ devices and sensors has an extraordinarily broad range
of relevant time durations for which they are valuable to the business, from milliseconds, to
decades.
Analyzing huge volumes of data that span multiple orders of time-
scale magnitude falls short of traditional data-management
technologies’ abilities. Traditional methods of data management,
such as relational databases (RDB) or time-serialized databases,
may not have the capability to capture the causal effects of years
Leveraging Big Data and Real-Time Analytics to
Achieve Situational Awareness for Smart Grids
Page 5 of 20
or decades of events that may occur in a millisecond or
microsecond range, and therefore cannot meet the real-time smart
grids’ scalable situational awareness needs. Additionally, such an
array of devices and processes create an especially-wide variety of
data types and formats that must be considered when making any
decision, and thus for enabling scalable situational awareness.
A Word on Synchrophasors: Smart Grid Big Data’s Best Friend
(and Worst Enemy…)
Utilities already have considerable challenges with regards to the
speed, complexity, and variety of their data, and many would argue
they also are already facing true data volume problems. But if a
consensus can’t be reached apropos on utilities’ data volume
challenges, it is certain they will face them in the future as they
pursue the goals of the smart grid.
Currently, most utilities are in smart grid deployment mode, and
many investor-owned utilities have entered the implementation
phase as described by the Public Utilities Commissions’ smart grid
strategic plans. By 2020, there will be a large number of real-world,
full-scale smart grid deployments. The achievement of this goal will
be driven largely by a massive increase in the number of installed
phasor measurement units (PMUs), or synchrophasors, in the
power grid.
Synchrophasors are considered one of the most important
measuring devices for smart grids, and their continued adoption is
expected to increase data volumes for utilities by 700 to 800
percent by the year 2020:
Image Caption: Expected Data Volumes from PMU Deployments
Assumption : 15 installed PMUs at the time of North America’s largest Blackout in 2003
(Western Part of the United States)
Leveraging Big Data and Real-Time Analytics to
Achieve Situational Awareness for Smart Grids
Page 6 of 20
SECTION 2: UTILITIES' CURRENT DATA
ANALYTICS AND MANAGEMENT METHODS, AND
THE NEED FOR CHANGE
Utilities have primarily used the same two types of databases to
manage and analyze their data for the last 30 years, despite
neither of them ever being ideal for these critical tasks.
The relational database (RDB) rose to prominent use by utilities
during the seventies when storage media was very expensive. The
main advantage that the RDB was that storage costs could be
minimized, but at the expense of needing to write a lot of
proprietary code to describe the relationships between data
(commonly known as a JOIN).
Data historian technologies, sometimes referred to as streaming
data stores, rose to similar prominence for storing utilities’ time-
serialized data. These, essentially, were another form of the RDB,
in this case optimized for storing data with a time-stamp, which
emphasized reduction in storage costs at the price of not being
able to easily correlate it with important variables other than just
time.
The shortcomings of these technologies have become especially
apparent as the need to conduct analysis across multiple data
types, formats, and domains has become more important for
enabling scalable situational awareness for the smart grid. The
failure of traditional databases like RDBs to scale well in the face of
rising data volumes, complexity, and speed has been well proven,
with alternative technologies often outperforming them by more
than ten-fold in benchmarking tests.
What are needed instead are data management technologies that
are optimized for analysis rather than constraints like speed and
storage space. Ideally, these technologies would also be built much
like the grid itself, with classes of assets that have natural, pre-
defined relationships between them. These capabilities are readily
found in proven object-oriented databases (ODB) and emerging
NoSQL technologies like those that Versant offers, and which
indeed have been deployed across multiple industries with similar
data challenges with great effect.
The ODB has been used for years in the telecommunications,
transportation, building management, and many other industries to
track and analyze large numbers of data types and classes. Unlike
relational or serialized databases, ODB’s offer seamless integration
with object-oriented languages. This means that the objects’
application descriptions translate directly to the database objects
Leveraging Big Data and Real-Time Analytics to
Achieve Situational Awareness for Smart Grids
Page 7 of 20
themselves, making analysis much easier and faster, and thus
supporting the goals of scalable situational awareness.
Additionally, query is used for optimization based on use cases and
the business application of the data, not as the sole means of
accessing and manipulating the underlying data. There is no
application code needed to manage the connectivity between
objects or how they are mapped to the underlying database
storage. ODBs use and store object identity directly, bypassing the
need for the CPU and memory-intensive, set-based join operations
used by RDBs.
SECTION 3: REQUIREMENTS OF DATA
ANALYTICS SYSTEMS TO CREATE SCALABLE
SITUATIONAL AWARENESS FOR SMART GRIDS
The underlying data management and analytics solutions required
to provide scalable situational awareness for smart grids must have
five key characteristics: Flexibility, Interoperability through
Connectivity, a Control Network, it must use Open, Standards-
Based Data Management Technologies, and it must support
Scalable Data Analysis.
Flexibility - Unlike many industries, power delivery is notoriously
variable, with daily, weekly, and annual variations due to variability
in customer load, generation dispatch, delivery system outages,
and other reasons. This variability has challenged the industry to
discern patterns that can be used to identify abnormal conditions
and anomalies that spur critical decisions-making processes.
Versant’s technologies can deal with data that looks at voltage and
current rate data just as easily as any other type of data from any
other industry. By embedding a variety of different data object
models to capture the different energy data types, as well as its
corresponding sample rates, object-oriented programming allows
for an integrated data management and analytics concept. It
creates the necessary flexibility to deal with the challenging
characteristics of Big energy Data in real-time. Fast and reliable
data retrieval, suitable data formats for data analysis, one object-
oriented programming language (for DDL and DML), connectivity
between objects without application code, direct use and storage of
object identities, and advanced, as well as traditional data
management, features merged together represent critical values of
a fully-integrated object-oriented data management and analytics
solution. This is what gives you the situational awareness that is
needed for utilities: understanding the immediate value of making a
decision to solve an abnormality in normal data patterns within a
relevant time frame.
Leveraging Big Data and Real-Time Analytics to
Achieve Situational Awareness for Smart Grids
Page 8 of 20
Interoperability and Connectivity – The smart grid of the future will
be a massive collection of devices, sensors, actuators, and
systems, all of them creating ever-larger data volumes and ever-
greater analytics complexity. In this form, the smart grid is a hugely
complex network that must have full accessibility of all these
devices and sensors. Central to enabling this is Internet
connectivity, something again that Versant’s technologies have
proven highly capable of by managing data and analysis for many
global telecommunications service providers.
Control Network - Not only is collecting all of the data that smart
grid sensors and devices produce a challenge, but all of these
devices must be fully communicative, interconnected, and, critically
controllable. The decisions made based on having full situational
awareness must be rapidly translated in to the functioning grid,
which, like enabling interoperability, requires a single, cohesive
control system enabled heavily through Internet connectivity.
Open, Standards-Based Data Management Systems – A network
as complex, variable, and fast-moving as the smart grid requires
billions of devices, sensors, and machines. It is impossible to
expect that any one data management technology vendors’
systems will be used across every grid application and scenario.
But more to the point, smart grids will be integral to the everyday
life of billions of people, so as new technologies are developed and
adopted over time the smart grid must be able to adjust and
change the data management systems to meet new requirements.
To enable this, utilities must leverage open system architectures
across five specific areas to permit ease of adoption and avoid
costly vendor lock-in:
 Network Infrastructure: Includes protocol, routers, media
type, IT connectivity, etc.
 Control Devices: Heavily-utilized devices that produce,
consume, and manipulate data, as well as control and
monitor the energy grid network.
 Network Management and Diagnostic Tools: Enable
configuration, commission, and maintenance for the
system.
 Human-Machine Interface (HMI): Includes the visualization
tools through which users and managers obtain a view into
the system, including both PC software and instrumentation
panels.
 Enterprise/IT Level Interface: Connects the control network
into the data network. No gateways other than open
systems standards-based routers and IT-based data
exchange mechanisms are used.
Leveraging Big Data and Real-Time Analytics to
Achieve Situational Awareness for Smart Grids
Page 9 of 20
A critical sixth factor is the object-oriented data management
system itself, which must also be considered part of this open
standards-based architecture. The ODB represents the
configuration database for the complete network of the grid, storing
the configuration profile data of every device participating in the
open, fully interoperable and integrated control network, and
enabling effective communication and control between them all.
Scalable Data Analysis – Utilities will face immense data volume
increases over the next several years, making the job of ensuring
the validity and veracity of data analysis ever harder. Open
architectures and data management technologies will play a pivotal
role in enabling data analysis that scales to these new volume
demands. These systems must not only be capable of dynamically
scaling to account for and manage increased data complexity, but
also sheer volume as new types of devices are deployed on the
grid network.
Versant’s abilities to scale for both extreme data volumes and
analytics complexity have been proven in large network
environment deployments like telecommunications and building
energy management. These industries are directly relevant to the
smart grid because of their scale and complexity challenges, as
well as the nature of the data itself. Technologies like the Versant
Object Database (VOD) and the Versant Java Persistence API
(Versant JPA) have been deployed by dozens of
telecommunications network operators and equipment vendors,
and in more than 150,000 buildings around the world.
As explained more fully in the next section, the Verite Group, for
example, uses Versant’s object-oriented technologies to power the
company’s new Netscope IP session packet reconstruction
application, allowing it to easily scale to meet the needs of Verite’s
customers, which include some of the world’s largest
telecommunications network operators. Likewise, for China
Telecom, who has one of the world’s largest subscriber bases, the
tremendous challenge of developing a database to store and
quickly access subscriber records at the rate of up to 480,000
queries and 1,000 update transactions per second was drastically
simplified with Versant.
SECTION 4: VERSANT AND THE OBJECT-
ORIENTED DB’S PEDIGREE, AND THEIR VALUE
FOR SMART GRID SITUATIONAL AWARENESS
Relational databases rose to fame in the utility industry when,
compared to what they are today, storage was expensive and data
complexity and volumes were low. The need to avoid substantial
Leveraging Big Data and Real-Time Analytics to
Achieve Situational Awareness for Smart Grids
Page 10 of 20
cost was understandably paramount at a time when the grid was
already relatively large compared to other similar networks and
would have therefore multiplied storage costs.
However, that did not stop other industries from finding more
effective data analytics and management technologies. In other
industries where storage concerns were not quite as much of a
concern initially, yet managing extreme data complexity was of the
utmost importance, such as building management systems, the
object-oriented database quickly became the go-to choice. And
even for the world’s original Big Data industry, telecommunications,
where managing and analyzing massive data volumes on par with
those the utilities industry will soon face, the ODB has been the de-
facto data management technology for decades.
Indeed, several other industries that have multi-faceted,
interdependent network environments make for great starting
points from which the utilities industry can learn to apply scalable
situational awareness. Most notable of these are data
communications networks like those of telecommunications,
geospatial environments, building control networks, and power
transmission networks.
The concept of centralized power transmission system operation
implemented through Independent System Operators (ISOs) has
proven critical in North America, as well as Europe, to increase
bulk power grid reliability, improve planning, enhance technology,
and guarantee energy market pricing. Implementing advanced
information management systems in the control room of an ISO
plays a crucial part in this concept.
NoSQL-based, object-oriented data management solutions
empower control room systems operated by the French ISO Rte,
the French Security Agency CNES, as well as the service
coordination center CORESO shared by three European ISOs
(Rte, Elia, National Grid). The following table summarizes where
object-oriented programming is applied to forecast day-ahead
electricity consumption, monitor power transmission system
security, or analyze and simulate steady-state grid performance. It
explains the critical role of object-oriented data management
solutions as they support effective, near real-time information
management for large, complex data management in bulk power
grids:
Leveraging Big Data and Real-Time Analytics to
Achieve Situational Awareness for Smart Grids
Page 11 of 20
Customer French National
Security Center
(CNES)
French Transmission
System Operator
(Rte)
Technical Coordination
Service Center (Coreso;
Rte+Elia+National Grid)
Name of
Control Room
System
PRELUDE ARCADE CONVERGENCE
Purpose of
Control Room
System
Forecast of Day-
Ahead Electricity
Consumption for the
French National
Security Center
(CNES) to monitor
and guarantee daily
Electricity Supply-
Demand Balance
Monitoring of French
Power Transmission
System Security by
French National
Security Center
(CNES)
Analysis and Simulation of
steady-state Performance of
French, Belgian and British
Power Transmission Systems
to guarantee Reliability of
electric Power Supply
The complexity of building management data models and
management needs are similar to those of smart grids.
Transforming buildings to become more “green” has been a
primary concern for years, and requires making them more
intelligent, energy efficient, and cost effective. Buildings’ energy
consumption typically represents 30 percent of its total operating
costs. A 30 percent reduction in a building’s energy use can yield a
five percent increase in total operating income, making reductions
in operating costs from increased energy efficiency one of the
strongest selling points for green buildings.
Making a building “green” means making every system that uses
natural resources to support humans more efficient. In addition to
energy consumption, this means transforming the way a building
uses water, air, and many more resources, which requires that
eight key systems are completely integrated via a data
management system according to the key characteristics outlined
in section 3 of this paper: Fire, security, access, energy, lighting,
lifts, communications, 24/7 monitoring, and HVAC. And again, as in
section 3, the Internet is also critical for enabling the connectivity
needed to link all these systems.
Leveraging Big Data and Real-Time Analytics to
Achieve Situational Awareness for Smart Grids
Page 12 of 20
These many different systems create levels of data management
complexity that actually surpass those of the smart grid.
Additionally, like the energy grid, buildings are also still
transforming as part of the “green” push, with new and more
sensors being added continually. And, actually, buildings are ahead
of utilities in this regard already having made major strides in
transforming from entities where each of these systems and
devices are islands of information to open-architecture-based,
intelligent buildings where components and control networks are
integrated, fully interoperable, and where controllers have full
situational awareness of the building’s abilities to support the
humans inside it.
As of 2009, Echelon’s LonWorks Network Systems (LNS) building
management technologies have been deployed in about 50 million
intelligent network devices across 1 million individual networks. The
LNS management system is powered by Versant’s object-oriented
data management solutions, enabling rapid data management and
analysis across all of these devices and sensors. Further, LonMark
has over 400 members using the LonWorks platform, and the
platform is an international open standard, ISO/IEC 14908-1,
making it a perfect example of the kind of open, standards-based
systems that smart grids require.
By using these open, object-oriented control systems to enable full
situational awareness of the building’s status, users reported:
 About a six months advantage in time-to-market for being
able to declare the building “green” and intelligent
 Several years of saved developer time
 Enablement of more powerful modeling features
 A significant increase in productivity for maintaining and
enhancing the code base that defines the underlying
database and data models
 An inherently more secure network
Energy efficiency in these buildings was also significantly
improved. The Balanced Office Building in Germany, for example,
reduced HVAC and lighting costs by 80 percent. The United States
National Resource Defense Council offices in California reduced
total energy use by 45 percent. And the Philips HELIO Lighting
Control Application, which is deployed throughout the world by
many large companies, uses the Echelon system to help reduce
companies’ energy costs by up to 50 percent by creating a data
management model that lets lighting systems be automatically
controlled depending on inputs from other systems and sensors
like HVAC.
Leveraging Big Data and Real-Time Analytics to
Achieve Situational Awareness for Smart Grids
Page 13 of 20
Enabling this degree of automation and interoperability is a form of
demand response capabilities, a major portion of utilities’ overall
demand-side management goals for the smart grid. And as
buildings represent 39 percent of primary energy use in the United
States, it is even more critical that utilities’ smart grid data
management systems interoperate with the open, standards based
building management systems like Versant’s and Echelon’s if they
are to achieve situational awareness.
Where buildings differ in their data management needs from
utilities is in the scale of their data volumes and velocities.
Buildings have roughly 72 object classes and 32,000 device
instances per building control network, whereas power grids deal
with data volumes on a scale more than ten-fold larger, with more
than 700 object classes. Additionally, buildings’ sensors and
systems need to function in the fastest instances at the second or
minute level for things like motion sensors and temperature
controls. But grid sensors must function to deliver effective
predictive analysis in an extremely wide time range from
milliseconds to decades. This essentially means that data
management systems for utilities must be able to respond to
analysis queries and deliver real interoperability on demand and in
true real-time.
The telecommunications industry examples referenced at the end
of section 4 of this paper reveal the ability of object-oriented
database technologies to perform extraordinarily well under
extreme data volume and velocity conditions. The Versant Object
Database has been successfully tested by China Telecom to
process up to 1,000,000 queries per second, earning it the
operators’ title of best Object Database technology after testing its
scalability against a range of other database technologies. During
head-to-head benchmarking between the Versant Object Database
(VOD) and other databases (ORM, XML), Verite Group’s VOD-
powered Netscope application processed data captures twice as
fast with minimal computational resource needs.
Open, standards-based data management technologies have
discernibly more pedigree than alternative solutions in terms of
scaling to meet the volume, variety, velocity, veracity, and validity
needs of Big Data. And these abilities are not mutually exclusive.
After conducting extensive benchmark tests against several
different database technologies, both NoSQL and relational, under
multi-dimensional scenarios at different scales and speeds,
Versant’s object-based technologies again outperformed other
technologies by up to 1000 percent, and in 80 percent of the tests.
Leveraging Big Data and Real-Time Analytics to
Achieve Situational Awareness for Smart Grids
Page 14 of 20
Image Caption: PolePosition benchmark testing shows how Versant’s technologies radically
outperform other data management solutions when processing Complex Concurrency
operations, which are highly relevant for utilities’ multi-faceted Big Data needs.
SECTION 5: HOW TO APPLY OBJECT-ORIENTED
DATA MANAGEMENT AND ANALYSIS
TECHNOLOGIES IN SMART GRIDS
As defined, the purpose of building open, object-oriented data
management systems that provide scalable situational awareness
for the smart grid is to identify anomalies in normal patterns in real-
time so that accurate decisions can be made, and the appropriate
actions taken, to positively affect the outcome of a business or
process.
One of the best examples of how Versant’s NoSQL - based data
Leveraging Big Data and Real-Time Analytics to
Achieve Situational Awareness for Smart Grids
Page 15 of 20
management and analytics technology can address utilities‘ smart
grid situational awareness challenges stems from the infamous
U.S. blackout that occurred on August 14, 2003.
In 2006, the United States Department of Energy found that it was
not for lack of action that the outage happened. Plenty of decisions
were made, but, as the report also found, the blackout was due, in
part, to “lack of awareness of deteriorating conditions,” making it
impossible to know if the decisions that were being made were
accurate, and the actions being taken helpful. The report
recommended that a real-time measurement system and
computer-based operational and management tools be developed.
Within the guidance of the Electric Power Research Institute
(EPRI), research was begun to determine how such a system could
be created to connect the multiple utility providers’ grids and
provide the situational awareness to prevent such a wide-scale
problem from occurring again. To prove that it could be done, EPRI
and Versant created a working demonstration technology
architecture using sizeable simulated data samples from the hours
leading up to the blackout. By configuring the system according to
the requirements and open, standards-based architectures outlined
in this paper, the demo was able to provide advanced warning that
a problem event was nearing, based on the calculation of the
phase angular difference and its trend as measured from the data
in the synchrophasors. The following diagrams show how the
demonstration technology indicates the divergence well in advance
of the actual blackout:
Image Caption: There were three time windows where situational awareness would have
given sufficient time to adequately respond.
Leveraging Big Data and Real-Time Analytics to
Achieve Situational Awareness for Smart Grids
Page 16 of 20
Image Caption: A screen shot from the actual demonstration interface shows that the phase
angular difference trend had fallen outside the acceptable range, and indicated that action
was needed, more than 35 minutes prior to the blackout.
The data management and analysis system architecture of the
demo was based on Versant’s new and fully-integrated software
development solution that has been applied in other industries:
Image Caption – Graphic representation of Versant’s fully integrated
software development toolkit.
Versant’s platform allows for an automated means to ingest,
manage, and analyze large, complex data volumes in real-time.
The approach is based on object-oriented programming, allowing
data to be modeled as objects and classified into object classes.
This produces the right match with the nature of data generated by
network topologies like those of the smart grid. The platform’s
ability to model data entities as schema according to unified
modeling language (UML) also corresponds with the UML
reference model for the electric utility established in the IEC CIM
Leveraging Big Data and Real-Time Analytics to
Achieve Situational Awareness for Smart Grids
Page 17 of 20
standard, where interoperability between all network devices used
in a Smart Grid is also specified. Adherence to these standards
allows for a stochastic topological model to be established between
the devices via network configuration models and their associated
real-time data.
Versant’s development framework can provide solutions that
support the following objectives of the energy sector’s smart grid
initiatives:
1. Development, storage, retrieval, and management of
network configuration models like the PMU measurement
network or the smart meter measurement network with
NoSQL database technology while still leveraging industry
standard APIs.
2. Evolving the afore-mentioned models for suitability in
analytical methods for optimal and scalable situational
awareness.
3. Data ingestion from multiple sources, including
Hadoop/MapReduce, real-time streaming data monitors,
such as Twitter, utility data sensor networks, and
transactional data sources like those in the financial
services industry, and data virtualization sources, such as
classic ETL system data. The ingested data can have
further context added through semantic enrichment from
external data sources offering information on weather,
existing assets, and communication network performance.
4. Build a real-time model that depicts connectivity while
continuously ingesting new data to provide absolutely
current situational awareness.
5. Implement architectural scale patterns commonly found in
NoSQL technology to support both scale-up on modern n-
core process architectures, and horizontal scale-out
patterns in partitioned systems to manage utilities’ growing
data volumes.
6. Support NoSQL architectural patterns to enable real-time
data analysis for situational-awareness through in-database
analytics. The Versant In-Database Analytic Framework
embraces a number of key analytic features that can be
applied to identify patterns and reveal critical factors for
controlling power grid stability under multiple simultaneous
events.
Based on the availability of network configuration data, as well as
real-time data from smart meters, PMUs, or SCADA systems,
Versant can model and manage device configurations and ingest
and analyze real-time data. As a result, Versant has the capability
to provide the necessary platforms and interfaces for numerical
Leveraging Big Data and Real-Time Analytics to
Achieve Situational Awareness for Smart Grids
Page 18 of 20
methods that allow for simulation and optimization at the scale of
utilities’ smart grid needs.
SECTION 6: BENEFITS OF SCALABLE
SITUATIONAL AWARENESS THROUGH OBJECT-
ORIENTED DATA MANAGEMENT AND ANALYSIS
FOR SMART GRIDS
In addition to being able to prevent large-scale and serious
problems like the August 2003 blackout, providing scalable
situational awareness via the technologies outlined in this paper
can realize the following key business benefits for utilities.
Asset Management –Routine maintenance and repairs to power
lines and other grid infrastructure account for a substantial portion
of utilities’ ongoing costs. With a sophisticated data management
system that enables advanced analytics like the one described in
this paper, fault locations can be more precisely identified and
characterized before a truck is even sent to fix it. This can also
allow utilities to determine if a truck and crew is needed to fix a
problem at all, resulting in immediate cost savings.
Power Usage Efficiency/Quality - Demand response and real-time
pricing are top priorities for utilities, but most only talk about it in
terms of meter data management and measuring kilowatt hours
used to enable more precise billing and energy flow. But power
quality at the residential and commercial building level needs much
more attention. If the data needed to measure this is built in to the
underlying data management model this can allow utilities to have
a much better understanding of power demand and supply
balance. But because each building is different, a database that
can scale with complexity and easily interoperate with buildings
systems is key.
Faster Smart Grid Roll-Out – The object-oriented approaches
outlined in this paper and exemplified by the demonstration
architecture created with EPRI enable the creation of a modular,
toolkit-like approach that enables shorter deployment time, much
like the benefits realized by the users of Echelon’s LNS system.
Open standards-based systems like these make the technologies
and applications understandable and more easily useable by a
broader group of people.
Leveraging Big Data and Real-Time Analytics to
Achieve Situational Awareness for Smart Grids
Page 19 of 20
CONCLUSION
The potential power of scalable situational awareness through
object-oriented data management for the smart grid is very
substantial. Utilities are faced with the simultaneously large
challenges and opportunities of Big Data, which make achieving
scalable situational awareness harder, but also more important and
rewarding. By turning virtually every piece of utilities’ infrastructure
in to a sensor, and making them fully interoperable and controllable
with object-oriented data management technologies, utilities can
prevent outages, mitigate potential threats to the network, and
realize a range of other important business benefits. The energy
industry would do well to learn from other industries that have
already conquered the challenges of Big Data, and apply the
lessons they have learned to turn the current grid in to a smart grid.
REFERENCES AND ACKNOWLEDGEMENTS
The author would like to thank Paul Myrda, Dr. John Simmins,
Scott Sternfeld and Dr. Gerald Gray from the Electric Power
Research Institute, as well as Barry Haaser from LonMark
International, and Volker John, Andreas Renner, Vishal Bagga,
Victor Dreiling, Yaniv Schwerin, and Torben Rebhan from Versant’s
R&D team for their invaluable assistance and contributions to
various aspects of this paper’s knowledge base. He would also like
to express his sincere appreciation for the insightful comments
given by Robert Greene and Dr. Robert Brammer.
About Versant
Versant Corportation
(Nasdaq:VSNT) is an
industry leader in
building specialized
NoSQL data
management systems
to enable the r real-time
enterprise. Using the
Versant Database
Engine, enterprises can
handle complex
information in
environments that
demand high
performance,
concurrency, and
availability, significantly
cut hardware and
administration costs,
speed and simplify
development, and
deliver products with a
strong competitive
edge. Versant's
solutions are deployed
in over 150,000
installations across a
wide array of industries,
including
telecommunications,
energy, financial
services, transportation,
manufacturing, and
defense. For more than
20 years, Versant has
been a trusted partner
of Global 2000
companies such as
Ericsson, Verizon,
Siemens, and Financial
Times, as well as the
U.S. Government. For
more information, call
650-232-2400 or visit
www.versant.com.
Leveraging Big Data and Real-Time Analytics to
Achieve Situational Awareness for Smart Grids
Page 20 of 20
BIBLIOGRAPHY
1. Doug Laney, 3-D Data Management: Controlling Data
Volume, Velocity and Variety, February 2001
2. Codd, E.F. (1970). “A Relational Model of Data for Large
Shared Data Banks”. In: Communications of the ACM 13
(6): 377-387
3. Jeffery Taft, Paul de Martini, Leonardo von Prellwitz. “Utility
Data Management and Intelligence”. Cisco White Paper,
May 2012.
4. Alexandra von Meier (California Institute for Energy and
Environment – CIEE). “Electric Power Systems – A
Conceptual Introduction”. IEEE Wiley. 2006.
5. Don v. Dollen and Bert Taube. “Advanced Computational
Techniques for Situational Awareness and Analytics in
Power Grids.” (in cooperation with J. Simmins, P. Myrda, G.
Gray, S. Sternfeld, V. Bagga). Redwood City: Presentation
at Smart Grid Update's Conference “Data Management and
Analytics for Utilities”, June 2012.
6. Bert Taube. “How Object-Oriented Data Management
enables Smart Energy Control in Buildings.” Santa Clara:
Presentation at ConnectivityWeek, May 2012.
7. Paul Myrda, John Simmins and Bert Taube. “Advanced
Utility Analytics with Object-Oriented Database
Technology”. Accepted Paper to be presented at CIGRE
Forum in Lisbon, April 2013.
KEYWORDS
Situational awareness, Big Data, big utility data challenges,
scalability, smart grid, NoSQL-based data management and
analytics, energy data categories (telemetry, oscillography, usage
data, event messages, meta data), key characteristics of energy
data, synchrophasor (PMU), blackout, real-time utility enterprise,
five characteristics of Big Utility Data (volume, velocity, variety,
validity, veracity), back-end data systems integration, connectivity
and interoperability, control network, open standards, smart
energy-efficient buildings, analysis and simulation of grid
performance, power consumption forecast, grid security.

More Related Content

What's hot

Big data privacy issues in public social media
Big data privacy issues in public social mediaBig data privacy issues in public social media
Big data privacy issues in public social mediaSupriya Radhakrishna
 
Delivering on the Promise of Big Data and the Cloud
Delivering on the Promise of Big Data and the CloudDelivering on the Promise of Big Data and the Cloud
Delivering on the Promise of Big Data and the CloudBooz Allen Hamilton
 
IABE Big Data information paper - An actuarial perspective
IABE Big Data information paper - An actuarial perspectiveIABE Big Data information paper - An actuarial perspective
IABE Big Data information paper - An actuarial perspectiveMateusz Maj
 
Big data document (basic concepts,3vs,Bigdata vs Smalldata,importance,storage...
Big data document (basic concepts,3vs,Bigdata vs Smalldata,importance,storage...Big data document (basic concepts,3vs,Bigdata vs Smalldata,importance,storage...
Big data document (basic concepts,3vs,Bigdata vs Smalldata,importance,storage...Taniya Fansupkar
 
Big Data Paradigm - Analysis, Application and Challenges
Big Data Paradigm - Analysis, Application and ChallengesBig Data Paradigm - Analysis, Application and Challenges
Big Data Paradigm - Analysis, Application and ChallengesUyoyo Edosio
 
IRJET- Big Data Management and Growth Enhancement
IRJET- Big Data Management and Growth EnhancementIRJET- Big Data Management and Growth Enhancement
IRJET- Big Data Management and Growth EnhancementIRJET Journal
 
Challenges of Data Management & Analytics in the Future Grid
Challenges of Data Management & Analytics in the Future GridChallenges of Data Management & Analytics in the Future Grid
Challenges of Data Management & Analytics in the Future GridPower System Operation
 
Enabling Big Data with Data-Level Security:The Cloud Analytics Reference Arch...
Enabling Big Data with Data-Level Security:The Cloud Analytics Reference Arch...Enabling Big Data with Data-Level Security:The Cloud Analytics Reference Arch...
Enabling Big Data with Data-Level Security:The Cloud Analytics Reference Arch...Booz Allen Hamilton
 
Data mining and homeland security rl31798
Data mining and homeland security rl31798Data mining and homeland security rl31798
Data mining and homeland security rl31798Daniel John
 
Enabling Cloud Analytics with Data-Level Security
Enabling Cloud Analytics with Data-Level SecurityEnabling Cloud Analytics with Data-Level Security
Enabling Cloud Analytics with Data-Level SecurityBooz Allen Hamilton
 
Map Reduce in Big fata
Map Reduce in Big fataMap Reduce in Big fata
Map Reduce in Big fataSuraj Sawant
 
Story of Bigdata and its Applications in Financial Institutions
Story of Bigdata and its Applications in Financial InstitutionsStory of Bigdata and its Applications in Financial Institutions
Story of Bigdata and its Applications in Financial Institutionsijtsrd
 
Big Data Applications & Analytics Motivation: Big Data and the Cloud; Centerp...
Big Data Applications & Analytics Motivation: Big Data and the Cloud; Centerp...Big Data Applications & Analytics Motivation: Big Data and the Cloud; Centerp...
Big Data Applications & Analytics Motivation: Big Data and the Cloud; Centerp...Geoffrey Fox
 
Big data - what, why, where, when and how
Big data - what, why, where, when and howBig data - what, why, where, when and how
Big data - what, why, where, when and howbobosenthil
 

What's hot (20)

Big data privacy issues in public social media
Big data privacy issues in public social mediaBig data privacy issues in public social media
Big data privacy issues in public social media
 
Seminario Big Data
Seminario Big DataSeminario Big Data
Seminario Big Data
 
Delivering on the Promise of Big Data and the Cloud
Delivering on the Promise of Big Data and the CloudDelivering on the Promise of Big Data and the Cloud
Delivering on the Promise of Big Data and the Cloud
 
IABE Big Data information paper - An actuarial perspective
IABE Big Data information paper - An actuarial perspectiveIABE Big Data information paper - An actuarial perspective
IABE Big Data information paper - An actuarial perspective
 
Etiya White Paper_ABDR
Etiya White Paper_ABDREtiya White Paper_ABDR
Etiya White Paper_ABDR
 
Big data document (basic concepts,3vs,Bigdata vs Smalldata,importance,storage...
Big data document (basic concepts,3vs,Bigdata vs Smalldata,importance,storage...Big data document (basic concepts,3vs,Bigdata vs Smalldata,importance,storage...
Big data document (basic concepts,3vs,Bigdata vs Smalldata,importance,storage...
 
Big Data
Big DataBig Data
Big Data
 
Big Data Paradigm - Analysis, Application and Challenges
Big Data Paradigm - Analysis, Application and ChallengesBig Data Paradigm - Analysis, Application and Challenges
Big Data Paradigm - Analysis, Application and Challenges
 
IRJET- Big Data Management and Growth Enhancement
IRJET- Big Data Management and Growth EnhancementIRJET- Big Data Management and Growth Enhancement
IRJET- Big Data Management and Growth Enhancement
 
Challenges of Data Management & Analytics in the Future Grid
Challenges of Data Management & Analytics in the Future GridChallenges of Data Management & Analytics in the Future Grid
Challenges of Data Management & Analytics in the Future Grid
 
Big data
Big dataBig data
Big data
 
Enabling Big Data with Data-Level Security:The Cloud Analytics Reference Arch...
Enabling Big Data with Data-Level Security:The Cloud Analytics Reference Arch...Enabling Big Data with Data-Level Security:The Cloud Analytics Reference Arch...
Enabling Big Data with Data-Level Security:The Cloud Analytics Reference Arch...
 
Data mining and homeland security rl31798
Data mining and homeland security rl31798Data mining and homeland security rl31798
Data mining and homeland security rl31798
 
Enabling Cloud Analytics with Data-Level Security
Enabling Cloud Analytics with Data-Level SecurityEnabling Cloud Analytics with Data-Level Security
Enabling Cloud Analytics with Data-Level Security
 
Map Reduce in Big fata
Map Reduce in Big fataMap Reduce in Big fata
Map Reduce in Big fata
 
Story of Bigdata and its Applications in Financial Institutions
Story of Bigdata and its Applications in Financial InstitutionsStory of Bigdata and its Applications in Financial Institutions
Story of Bigdata and its Applications in Financial Institutions
 
Big Data Applications & Analytics Motivation: Big Data and the Cloud; Centerp...
Big Data Applications & Analytics Motivation: Big Data and the Cloud; Centerp...Big Data Applications & Analytics Motivation: Big Data and the Cloud; Centerp...
Big Data Applications & Analytics Motivation: Big Data and the Cloud; Centerp...
 
Fundamentals of Big Data
Fundamentals of Big DataFundamentals of Big Data
Fundamentals of Big Data
 
3.2
3.23.2
3.2
 
Big data - what, why, where, when and how
Big data - what, why, where, when and howBig data - what, why, where, when and how
Big data - what, why, where, when and how
 

Viewers also liked

Transf React Proact T&D Ass Management
Transf React Proact T&D Ass ManagementTransf React Proact T&D Ass Management
Transf React Proact T&D Ass ManagementBert Taube
 
Gre math
Gre mathGre math
Gre mathshivgan
 
Situational Awareness for Fire and EMS
Situational Awareness for Fire and EMSSituational Awareness for Fire and EMS
Situational Awareness for Fire and EMSRommie Duckworth
 
Playing chess with Companies
Playing chess with CompaniesPlaying chess with Companies
Playing chess with CompaniesSimon Wardley
 
Situational leadership theory...
Situational leadership theory...Situational leadership theory...
Situational leadership theory...VIKRANT KUL
 
The Hacking Team Hack: Lessons Learned for Enterprise Security
The Hacking Team Hack: Lessons Learned for Enterprise SecurityThe Hacking Team Hack: Lessons Learned for Enterprise Security
The Hacking Team Hack: Lessons Learned for Enterprise SecurityStephen Cobb
 
Steps for mapping - a rough guide
Steps for mapping - a rough guideSteps for mapping - a rough guide
Steps for mapping - a rough guideSimon Wardley
 
Eyes - Refractive Errors.ppt
Eyes - Refractive Errors.pptEyes - Refractive Errors.ppt
Eyes - Refractive Errors.pptShama
 
An introduction to Wardley Maps
An introduction to Wardley MapsAn introduction to Wardley Maps
An introduction to Wardley MapsSimon Wardley
 
Medical mnemonics pdf_print_version
Medical mnemonics pdf_print_versionMedical mnemonics pdf_print_version
Medical mnemonics pdf_print_version维卡普 Pardhu
 
Personas for SEO in 2012 (PubCon)
Personas for SEO in 2012 (PubCon)Personas for SEO in 2012 (PubCon)
Personas for SEO in 2012 (PubCon)Michael King
 
Core Java Slides
Core Java SlidesCore Java Slides
Core Java SlidesVinit Vyas
 
MOBILE PHONE & MOBILE TOWER RADIATION HAZARDS
MOBILE PHONE & MOBILE TOWER RADIATION HAZARDS MOBILE PHONE & MOBILE TOWER RADIATION HAZARDS
MOBILE PHONE & MOBILE TOWER RADIATION HAZARDS Neha Kumar
 

Viewers also liked (20)

Printing in 3D
Printing in 3DPrinting in 3D
Printing in 3D
 
Transf React Proact T&D Ass Management
Transf React Proact T&D Ass ManagementTransf React Proact T&D Ass Management
Transf React Proact T&D Ass Management
 
Gre math
Gre mathGre math
Gre math
 
Comprehensive GRE Guide and Planner
Comprehensive GRE Guide and PlannerComprehensive GRE Guide and Planner
Comprehensive GRE Guide and Planner
 
Interactive Flashcards for GRE Math
Interactive Flashcards for GRE MathInteractive Flashcards for GRE Math
Interactive Flashcards for GRE Math
 
Situational Awareness for Fire and EMS
Situational Awareness for Fire and EMSSituational Awareness for Fire and EMS
Situational Awareness for Fire and EMS
 
Playing chess with Companies
Playing chess with CompaniesPlaying chess with Companies
Playing chess with Companies
 
Situational leadership theory...
Situational leadership theory...Situational leadership theory...
Situational leadership theory...
 
The Hacking Team Hack: Lessons Learned for Enterprise Security
The Hacking Team Hack: Lessons Learned for Enterprise SecurityThe Hacking Team Hack: Lessons Learned for Enterprise Security
The Hacking Team Hack: Lessons Learned for Enterprise Security
 
Grubers gre-guide-2012
Grubers gre-guide-2012Grubers gre-guide-2012
Grubers gre-guide-2012
 
Steps for mapping - a rough guide
Steps for mapping - a rough guideSteps for mapping - a rough guide
Steps for mapping - a rough guide
 
Situational Leadership
Situational LeadershipSituational Leadership
Situational Leadership
 
Eyes - Refractive Errors.ppt
Eyes - Refractive Errors.pptEyes - Refractive Errors.ppt
Eyes - Refractive Errors.ppt
 
An introduction to Wardley Maps
An introduction to Wardley MapsAn introduction to Wardley Maps
An introduction to Wardley Maps
 
Medical mnemonics pdf_print_version
Medical mnemonics pdf_print_versionMedical mnemonics pdf_print_version
Medical mnemonics pdf_print_version
 
Personas for SEO in 2012 (PubCon)
Personas for SEO in 2012 (PubCon)Personas for SEO in 2012 (PubCon)
Personas for SEO in 2012 (PubCon)
 
Refractive errors
Refractive errorsRefractive errors
Refractive errors
 
Data Science At Zillow
Data Science At ZillowData Science At Zillow
Data Science At Zillow
 
Core Java Slides
Core Java SlidesCore Java Slides
Core Java Slides
 
MOBILE PHONE & MOBILE TOWER RADIATION HAZARDS
MOBILE PHONE & MOBILE TOWER RADIATION HAZARDS MOBILE PHONE & MOBILE TOWER RADIATION HAZARDS
MOBILE PHONE & MOBILE TOWER RADIATION HAZARDS
 

Similar to Utilities White Paper Final Versant

CASE STUDY ON METHODS AND TOOLS FOR THE BIG DATA ANALYSIS
CASE STUDY ON METHODS AND TOOLS FOR THE BIG DATA ANALYSISCASE STUDY ON METHODS AND TOOLS FOR THE BIG DATA ANALYSIS
CASE STUDY ON METHODS AND TOOLS FOR THE BIG DATA ANALYSISIRJET Journal
 
Ab cs of big data
Ab cs of big dataAb cs of big data
Ab cs of big dataDigimark
 
Nikita rajbhoj(a 50)
Nikita rajbhoj(a 50)Nikita rajbhoj(a 50)
Nikita rajbhoj(a 50)NikitaRajbhoj
 
Analysis of Big Data
Analysis of Big DataAnalysis of Big Data
Analysis of Big DataIRJET Journal
 
A Survey on Big Data Mining Challenges
A Survey on Big Data Mining ChallengesA Survey on Big Data Mining Challenges
A Survey on Big Data Mining ChallengesEditor IJMTER
 
Paper Final Taube Bienert GridInterop 2012
Paper Final Taube Bienert GridInterop 2012Paper Final Taube Bienert GridInterop 2012
Paper Final Taube Bienert GridInterop 2012Bert Taube
 
Big data is a broad term for data sets so large or complex that tr.docx
Big data is a broad term for data sets so large or complex that tr.docxBig data is a broad term for data sets so large or complex that tr.docx
Big data is a broad term for data sets so large or complex that tr.docxhartrobert670
 
Information economics and big data
Information economics and big dataInformation economics and big data
Information economics and big dataMark Albala
 
ft_mckinsey digital oil and gas
ft_mckinsey digital oil and gasft_mckinsey digital oil and gas
ft_mckinsey digital oil and gasTor Jakob Ramsøy
 
IRJET- Scope of Big Data Analytics in Industrial Domain
IRJET- Scope of Big Data Analytics in Industrial DomainIRJET- Scope of Big Data Analytics in Industrial Domain
IRJET- Scope of Big Data Analytics in Industrial DomainIRJET Journal
 
Big Data Analytics
Big Data AnalyticsBig Data Analytics
Big Data AnalyticsNoel Hatch
 
big data Big Things
big data Big Thingsbig data Big Things
big data Big Thingspateelhs
 
The Second Big Bang
The Second Big BangThe Second Big Bang
The Second Big BangConnexica
 
A Deep Dissertion Of Data Science Related Issues And Its Applications
A Deep Dissertion Of Data Science  Related Issues And Its ApplicationsA Deep Dissertion Of Data Science  Related Issues And Its Applications
A Deep Dissertion Of Data Science Related Issues And Its ApplicationsTracy Hill
 
Big Data in Cloud Computing Review and Opportunities
Big Data in Cloud Computing Review and OpportunitiesBig Data in Cloud Computing Review and Opportunities
Big Data in Cloud Computing Review and OpportunitiesAIRCC Publishing Corporation
 
IRJET - Big Data Analysis its Challenges
IRJET - Big Data Analysis its ChallengesIRJET - Big Data Analysis its Challenges
IRJET - Big Data Analysis its ChallengesIRJET Journal
 

Similar to Utilities White Paper Final Versant (20)

The ABCs of Big Data
The ABCs of Big DataThe ABCs of Big Data
The ABCs of Big Data
 
CASE STUDY ON METHODS AND TOOLS FOR THE BIG DATA ANALYSIS
CASE STUDY ON METHODS AND TOOLS FOR THE BIG DATA ANALYSISCASE STUDY ON METHODS AND TOOLS FOR THE BIG DATA ANALYSIS
CASE STUDY ON METHODS AND TOOLS FOR THE BIG DATA ANALYSIS
 
Ab cs of big data
Ab cs of big dataAb cs of big data
Ab cs of big data
 
Nikita rajbhoj(a 50)
Nikita rajbhoj(a 50)Nikita rajbhoj(a 50)
Nikita rajbhoj(a 50)
 
Analysis of Big Data
Analysis of Big DataAnalysis of Big Data
Analysis of Big Data
 
A Survey on Big Data Mining Challenges
A Survey on Big Data Mining ChallengesA Survey on Big Data Mining Challenges
A Survey on Big Data Mining Challenges
 
Paper Final Taube Bienert GridInterop 2012
Paper Final Taube Bienert GridInterop 2012Paper Final Taube Bienert GridInterop 2012
Paper Final Taube Bienert GridInterop 2012
 
Big data is a broad term for data sets so large or complex that tr.docx
Big data is a broad term for data sets so large or complex that tr.docxBig data is a broad term for data sets so large or complex that tr.docx
Big data is a broad term for data sets so large or complex that tr.docx
 
Information economics and big data
Information economics and big dataInformation economics and big data
Information economics and big data
 
ft_mckinsey digital oil and gas
ft_mckinsey digital oil and gasft_mckinsey digital oil and gas
ft_mckinsey digital oil and gas
 
IRJET- Scope of Big Data Analytics in Industrial Domain
IRJET- Scope of Big Data Analytics in Industrial DomainIRJET- Scope of Big Data Analytics in Industrial Domain
IRJET- Scope of Big Data Analytics in Industrial Domain
 
Big Data Analytics
Big Data AnalyticsBig Data Analytics
Big Data Analytics
 
big data Big Things
big data Big Thingsbig data Big Things
big data Big Things
 
The Second Big Bang
The Second Big BangThe Second Big Bang
The Second Big Bang
 
A Deep Dissertion Of Data Science Related Issues And Its Applications
A Deep Dissertion Of Data Science  Related Issues And Its ApplicationsA Deep Dissertion Of Data Science  Related Issues And Its Applications
A Deep Dissertion Of Data Science Related Issues And Its Applications
 
Big Data in Cloud Computing Review and Opportunities
Big Data in Cloud Computing Review and OpportunitiesBig Data in Cloud Computing Review and Opportunities
Big Data in Cloud Computing Review and Opportunities
 
Big data storage
Big data storageBig data storage
Big data storage
 
Unit-1 introduction to Big data.pdf
Unit-1 introduction to Big data.pdfUnit-1 introduction to Big data.pdf
Unit-1 introduction to Big data.pdf
 
Unit-1 introduction to Big data.pdf
Unit-1 introduction to Big data.pdfUnit-1 introduction to Big data.pdf
Unit-1 introduction to Big data.pdf
 
IRJET - Big Data Analysis its Challenges
IRJET - Big Data Analysis its ChallengesIRJET - Big Data Analysis its Challenges
IRJET - Big Data Analysis its Challenges
 

Utilities White Paper Final Versant

  • 1. WHITE PAPER LEVERAGING BIG DATA AND REAL-TIME ANALYTICS TO ACHIEVE SITUATIONAL AWARENESS FOR SMART GRIDS PART ONE IN VERSANT’S SCALABLE SITUATIONAL AWARENESS SERIES By Bert Taube, Director of Energy and Smart Grid Solutions Versant Corporation Sponsored by Versant Corporation Versant Corporation U.S. Headquarters 255 Shoreline Dr. Suite 450, Redwood City, CA 94065 www.versant.com  +1 650-232-2400
  • 2. Leveraging Big Data and Real-Time Analytics to Achieve Situational Awareness for Smart Grids Page 2 of 20 EXECUTIVE SUMMARY: WHY THE PURPOSE OF BIG DATA IS REALLY SITUATIONAL AWARENESS The concept of Big Data has been around for more than a decade, and its potential to transform the effectiveness, efficiency, and profitability of virtually any enterprise has been well documented. Yet, despite the concept of Big Data being well-defined, and the general enormity of its opportunity well-understood, the means to effectively leverage Big Data and realize its promised benefits still eludes many. Big Data’s remaining challenge to realizing these benefits comes in two parts. The first is to understand that the true purpose of leveraging Big Data is to take action - to make more accurate decisions, more quickly. We call this situational awareness, an idea that is quite self-explanatory. Regardless of industry or environment, situational awareness means having an understanding of what you need to know, have control of, and conduct analysis for in real-time to identify anomalies in normal patterns or behaviors that can affect the outcome of a business or process. If you have these things, making the right decision in the right amount of time in any context becomes much easier. Although the term of situational awareness itself is fairly recent, the concept has roots in the history of military theory - it is recognizable in Sun Tzu's The Art of War, for instance. The term itself, can also be traced to World War I, where it was recognized as a crucial component for crews in military aircraft. Before being widely adopted by human factors scientists in the 1990s, the term was first used by United States Air Force (USAF) fighter aircrews returning from war in Korea and Vietnam. Today, the concept of situational awareness has been expanded to a variety of areas such as air traffic control, nuclear power plant operation, vehicle operation, and anesthesiology. Defining the parameters of situational awareness for any industry is not simple, and thus surmounting Big Data’s other remaining challenge of creating new approaches to data management and analysis to support these needs is also no small feat. Achieving situational awareness used to be much easier because data volumes were smaller, and new data was created at a slower rate, which meant our worlds were defined by a much smaller amount of information. But new data is now created at a hugely exponential rate, and therefore any data management and analysis system that is built to provide situational awareness today must also be able to do so tomorrow. Thus, the imperative for any enterprise is to create systems that manage Big Data and provide scalable situational awareness. Table of Contents Section 1: The Smart Grid Network’s Unique Big Data Demands Section 2: Utilities’ Current Data Analytics and Management Methods, and the Need for Change Section 3: Requirements of Data Analytics Systems to Create Scalable Situational Awareness for Smart Grids Section 4: Versant and The Object-Oriented DB’s Pedigree, and their Value for Smart Grid Situational Awareness Section 5: How To Apply Object-Oriented Data Management and Analysis Technologies in Smart Grids Section 6: Benefits of Scalable Situational Awareness through Object-Oriented Data Management and Analysis for Smart Grids Conclusion
  • 3. Leveraging Big Data and Real-Time Analytics to Achieve Situational Awareness for Smart Grids Page 3 of 20 The utilities industry is in particular need of scalable situational awareness so that it can realize benefits for a wide range of important functions that are critical for enabling Smart Grid paradigms. Scalable situational awareness for utilities means knowing where power is needed, and where it can be taken from, to keep the grid stable. When power flow is not well understood, the resulting consequences can quite literally leave utilities and their customers in the dark: a fitting-though-ironic analogy considering the goal of awareness. . Utilities can learn much about how to achieve scalable situational awareness from other industries, most notably building management and telecommunications, which have learned to deal with Big Data’s complexity and scale well. This white paper will describe the diversity of utility data, the criteria which define a big energy data problem, and how utility business value can be built through increased situational awareness. The various problems that arise from storing the large number of data types in smart grid systems using traditional software technologies (data historians and relational databases) will be summarized. Overcoming these problems can be accomplished by applying NoSQL-based data management and analytics solutions like Versant’s, which can be seamlessly integrated with object-oriented languages. A power outage example will be utilized to demonstrate how Versant’s NoSQL solutions have enabled grid reliability and situational awareness, and how they can help utilities achieve other critical business goals. SECTION 1: THE SMART GRID NETWORK’S UNIQUE BIG DATA DEMANDS In 2001, Doug Laney, now an analyst with Gartner, created what has become the widely-accepted three-dimensional definition of Big Data, also known as the Three V’s of Big Data: volume, velocity, and variety. Laney’s definition, though, was created for the paradigm of a typical business, such as manufacturing, where profitability is often achieved by the minimization of fixed assets, where work in progress is measured in days, weeks, or months, and real-time data collection and analysis are often not critical to ensure the profitability of the organization. The value chain for manufacturing almost always crosses company boundaries. However, in the utility industry, there are vertically-integrated and deregulated variants that have to act exactly the same, increasing complexity.
  • 4. Leveraging Big Data and Real-Time Analytics to Achieve Situational Awareness for Smart Grids Page 4 of 20 In this environment, the acquisition of real-time data can both be costly and can seriously impact the bottom line. This adds more dimensions to the utility industry’s Big Data needs as enterprises must not only deal with data’s volume, velocity, and variety challenges, but also with two new V’s: validity and veracity. Validity - Information in the utility environment often has a “shelf life” and therefore may only be needed for storage and evaluation for a fixed period of time. The questions of when to archive or dispose of data become relevant given the cost of storing large volumes of data. Veracity – This fifth variable recognizes that data is not perfect, and that achieving “perfect” data carries a cost. Utilities must consider two questions: 1) how good must the data be to achieve the necessary level of accurate analysis, and 2) at what point does the cost of correcting the data exceed the benefit of obtaining it? The utility industry’s time scales vary over 15 orders of magnitude due to the unique diversity of sensors and critical business processes, and often at much faster intervals than other industries, which, when trying to create scalable situational awareness, impacts all five V’s of the industry’s Big Data pressures. Image Caption: Data from Utilities’ devices and sensors has an extraordinarily broad range of relevant time durations for which they are valuable to the business, from milliseconds, to decades. Analyzing huge volumes of data that span multiple orders of time- scale magnitude falls short of traditional data-management technologies’ abilities. Traditional methods of data management, such as relational databases (RDB) or time-serialized databases, may not have the capability to capture the causal effects of years
  • 5. Leveraging Big Data and Real-Time Analytics to Achieve Situational Awareness for Smart Grids Page 5 of 20 or decades of events that may occur in a millisecond or microsecond range, and therefore cannot meet the real-time smart grids’ scalable situational awareness needs. Additionally, such an array of devices and processes create an especially-wide variety of data types and formats that must be considered when making any decision, and thus for enabling scalable situational awareness. A Word on Synchrophasors: Smart Grid Big Data’s Best Friend (and Worst Enemy…) Utilities already have considerable challenges with regards to the speed, complexity, and variety of their data, and many would argue they also are already facing true data volume problems. But if a consensus can’t be reached apropos on utilities’ data volume challenges, it is certain they will face them in the future as they pursue the goals of the smart grid. Currently, most utilities are in smart grid deployment mode, and many investor-owned utilities have entered the implementation phase as described by the Public Utilities Commissions’ smart grid strategic plans. By 2020, there will be a large number of real-world, full-scale smart grid deployments. The achievement of this goal will be driven largely by a massive increase in the number of installed phasor measurement units (PMUs), or synchrophasors, in the power grid. Synchrophasors are considered one of the most important measuring devices for smart grids, and their continued adoption is expected to increase data volumes for utilities by 700 to 800 percent by the year 2020: Image Caption: Expected Data Volumes from PMU Deployments Assumption : 15 installed PMUs at the time of North America’s largest Blackout in 2003 (Western Part of the United States)
  • 6. Leveraging Big Data and Real-Time Analytics to Achieve Situational Awareness for Smart Grids Page 6 of 20 SECTION 2: UTILITIES' CURRENT DATA ANALYTICS AND MANAGEMENT METHODS, AND THE NEED FOR CHANGE Utilities have primarily used the same two types of databases to manage and analyze their data for the last 30 years, despite neither of them ever being ideal for these critical tasks. The relational database (RDB) rose to prominent use by utilities during the seventies when storage media was very expensive. The main advantage that the RDB was that storage costs could be minimized, but at the expense of needing to write a lot of proprietary code to describe the relationships between data (commonly known as a JOIN). Data historian technologies, sometimes referred to as streaming data stores, rose to similar prominence for storing utilities’ time- serialized data. These, essentially, were another form of the RDB, in this case optimized for storing data with a time-stamp, which emphasized reduction in storage costs at the price of not being able to easily correlate it with important variables other than just time. The shortcomings of these technologies have become especially apparent as the need to conduct analysis across multiple data types, formats, and domains has become more important for enabling scalable situational awareness for the smart grid. The failure of traditional databases like RDBs to scale well in the face of rising data volumes, complexity, and speed has been well proven, with alternative technologies often outperforming them by more than ten-fold in benchmarking tests. What are needed instead are data management technologies that are optimized for analysis rather than constraints like speed and storage space. Ideally, these technologies would also be built much like the grid itself, with classes of assets that have natural, pre- defined relationships between them. These capabilities are readily found in proven object-oriented databases (ODB) and emerging NoSQL technologies like those that Versant offers, and which indeed have been deployed across multiple industries with similar data challenges with great effect. The ODB has been used for years in the telecommunications, transportation, building management, and many other industries to track and analyze large numbers of data types and classes. Unlike relational or serialized databases, ODB’s offer seamless integration with object-oriented languages. This means that the objects’ application descriptions translate directly to the database objects
  • 7. Leveraging Big Data and Real-Time Analytics to Achieve Situational Awareness for Smart Grids Page 7 of 20 themselves, making analysis much easier and faster, and thus supporting the goals of scalable situational awareness. Additionally, query is used for optimization based on use cases and the business application of the data, not as the sole means of accessing and manipulating the underlying data. There is no application code needed to manage the connectivity between objects or how they are mapped to the underlying database storage. ODBs use and store object identity directly, bypassing the need for the CPU and memory-intensive, set-based join operations used by RDBs. SECTION 3: REQUIREMENTS OF DATA ANALYTICS SYSTEMS TO CREATE SCALABLE SITUATIONAL AWARENESS FOR SMART GRIDS The underlying data management and analytics solutions required to provide scalable situational awareness for smart grids must have five key characteristics: Flexibility, Interoperability through Connectivity, a Control Network, it must use Open, Standards- Based Data Management Technologies, and it must support Scalable Data Analysis. Flexibility - Unlike many industries, power delivery is notoriously variable, with daily, weekly, and annual variations due to variability in customer load, generation dispatch, delivery system outages, and other reasons. This variability has challenged the industry to discern patterns that can be used to identify abnormal conditions and anomalies that spur critical decisions-making processes. Versant’s technologies can deal with data that looks at voltage and current rate data just as easily as any other type of data from any other industry. By embedding a variety of different data object models to capture the different energy data types, as well as its corresponding sample rates, object-oriented programming allows for an integrated data management and analytics concept. It creates the necessary flexibility to deal with the challenging characteristics of Big energy Data in real-time. Fast and reliable data retrieval, suitable data formats for data analysis, one object- oriented programming language (for DDL and DML), connectivity between objects without application code, direct use and storage of object identities, and advanced, as well as traditional data management, features merged together represent critical values of a fully-integrated object-oriented data management and analytics solution. This is what gives you the situational awareness that is needed for utilities: understanding the immediate value of making a decision to solve an abnormality in normal data patterns within a relevant time frame.
  • 8. Leveraging Big Data and Real-Time Analytics to Achieve Situational Awareness for Smart Grids Page 8 of 20 Interoperability and Connectivity – The smart grid of the future will be a massive collection of devices, sensors, actuators, and systems, all of them creating ever-larger data volumes and ever- greater analytics complexity. In this form, the smart grid is a hugely complex network that must have full accessibility of all these devices and sensors. Central to enabling this is Internet connectivity, something again that Versant’s technologies have proven highly capable of by managing data and analysis for many global telecommunications service providers. Control Network - Not only is collecting all of the data that smart grid sensors and devices produce a challenge, but all of these devices must be fully communicative, interconnected, and, critically controllable. The decisions made based on having full situational awareness must be rapidly translated in to the functioning grid, which, like enabling interoperability, requires a single, cohesive control system enabled heavily through Internet connectivity. Open, Standards-Based Data Management Systems – A network as complex, variable, and fast-moving as the smart grid requires billions of devices, sensors, and machines. It is impossible to expect that any one data management technology vendors’ systems will be used across every grid application and scenario. But more to the point, smart grids will be integral to the everyday life of billions of people, so as new technologies are developed and adopted over time the smart grid must be able to adjust and change the data management systems to meet new requirements. To enable this, utilities must leverage open system architectures across five specific areas to permit ease of adoption and avoid costly vendor lock-in:  Network Infrastructure: Includes protocol, routers, media type, IT connectivity, etc.  Control Devices: Heavily-utilized devices that produce, consume, and manipulate data, as well as control and monitor the energy grid network.  Network Management and Diagnostic Tools: Enable configuration, commission, and maintenance for the system.  Human-Machine Interface (HMI): Includes the visualization tools through which users and managers obtain a view into the system, including both PC software and instrumentation panels.  Enterprise/IT Level Interface: Connects the control network into the data network. No gateways other than open systems standards-based routers and IT-based data exchange mechanisms are used.
  • 9. Leveraging Big Data and Real-Time Analytics to Achieve Situational Awareness for Smart Grids Page 9 of 20 A critical sixth factor is the object-oriented data management system itself, which must also be considered part of this open standards-based architecture. The ODB represents the configuration database for the complete network of the grid, storing the configuration profile data of every device participating in the open, fully interoperable and integrated control network, and enabling effective communication and control between them all. Scalable Data Analysis – Utilities will face immense data volume increases over the next several years, making the job of ensuring the validity and veracity of data analysis ever harder. Open architectures and data management technologies will play a pivotal role in enabling data analysis that scales to these new volume demands. These systems must not only be capable of dynamically scaling to account for and manage increased data complexity, but also sheer volume as new types of devices are deployed on the grid network. Versant’s abilities to scale for both extreme data volumes and analytics complexity have been proven in large network environment deployments like telecommunications and building energy management. These industries are directly relevant to the smart grid because of their scale and complexity challenges, as well as the nature of the data itself. Technologies like the Versant Object Database (VOD) and the Versant Java Persistence API (Versant JPA) have been deployed by dozens of telecommunications network operators and equipment vendors, and in more than 150,000 buildings around the world. As explained more fully in the next section, the Verite Group, for example, uses Versant’s object-oriented technologies to power the company’s new Netscope IP session packet reconstruction application, allowing it to easily scale to meet the needs of Verite’s customers, which include some of the world’s largest telecommunications network operators. Likewise, for China Telecom, who has one of the world’s largest subscriber bases, the tremendous challenge of developing a database to store and quickly access subscriber records at the rate of up to 480,000 queries and 1,000 update transactions per second was drastically simplified with Versant. SECTION 4: VERSANT AND THE OBJECT- ORIENTED DB’S PEDIGREE, AND THEIR VALUE FOR SMART GRID SITUATIONAL AWARENESS Relational databases rose to fame in the utility industry when, compared to what they are today, storage was expensive and data complexity and volumes were low. The need to avoid substantial
  • 10. Leveraging Big Data and Real-Time Analytics to Achieve Situational Awareness for Smart Grids Page 10 of 20 cost was understandably paramount at a time when the grid was already relatively large compared to other similar networks and would have therefore multiplied storage costs. However, that did not stop other industries from finding more effective data analytics and management technologies. In other industries where storage concerns were not quite as much of a concern initially, yet managing extreme data complexity was of the utmost importance, such as building management systems, the object-oriented database quickly became the go-to choice. And even for the world’s original Big Data industry, telecommunications, where managing and analyzing massive data volumes on par with those the utilities industry will soon face, the ODB has been the de- facto data management technology for decades. Indeed, several other industries that have multi-faceted, interdependent network environments make for great starting points from which the utilities industry can learn to apply scalable situational awareness. Most notable of these are data communications networks like those of telecommunications, geospatial environments, building control networks, and power transmission networks. The concept of centralized power transmission system operation implemented through Independent System Operators (ISOs) has proven critical in North America, as well as Europe, to increase bulk power grid reliability, improve planning, enhance technology, and guarantee energy market pricing. Implementing advanced information management systems in the control room of an ISO plays a crucial part in this concept. NoSQL-based, object-oriented data management solutions empower control room systems operated by the French ISO Rte, the French Security Agency CNES, as well as the service coordination center CORESO shared by three European ISOs (Rte, Elia, National Grid). The following table summarizes where object-oriented programming is applied to forecast day-ahead electricity consumption, monitor power transmission system security, or analyze and simulate steady-state grid performance. It explains the critical role of object-oriented data management solutions as they support effective, near real-time information management for large, complex data management in bulk power grids:
  • 11. Leveraging Big Data and Real-Time Analytics to Achieve Situational Awareness for Smart Grids Page 11 of 20 Customer French National Security Center (CNES) French Transmission System Operator (Rte) Technical Coordination Service Center (Coreso; Rte+Elia+National Grid) Name of Control Room System PRELUDE ARCADE CONVERGENCE Purpose of Control Room System Forecast of Day- Ahead Electricity Consumption for the French National Security Center (CNES) to monitor and guarantee daily Electricity Supply- Demand Balance Monitoring of French Power Transmission System Security by French National Security Center (CNES) Analysis and Simulation of steady-state Performance of French, Belgian and British Power Transmission Systems to guarantee Reliability of electric Power Supply The complexity of building management data models and management needs are similar to those of smart grids. Transforming buildings to become more “green” has been a primary concern for years, and requires making them more intelligent, energy efficient, and cost effective. Buildings’ energy consumption typically represents 30 percent of its total operating costs. A 30 percent reduction in a building’s energy use can yield a five percent increase in total operating income, making reductions in operating costs from increased energy efficiency one of the strongest selling points for green buildings. Making a building “green” means making every system that uses natural resources to support humans more efficient. In addition to energy consumption, this means transforming the way a building uses water, air, and many more resources, which requires that eight key systems are completely integrated via a data management system according to the key characteristics outlined in section 3 of this paper: Fire, security, access, energy, lighting, lifts, communications, 24/7 monitoring, and HVAC. And again, as in section 3, the Internet is also critical for enabling the connectivity needed to link all these systems.
  • 12. Leveraging Big Data and Real-Time Analytics to Achieve Situational Awareness for Smart Grids Page 12 of 20 These many different systems create levels of data management complexity that actually surpass those of the smart grid. Additionally, like the energy grid, buildings are also still transforming as part of the “green” push, with new and more sensors being added continually. And, actually, buildings are ahead of utilities in this regard already having made major strides in transforming from entities where each of these systems and devices are islands of information to open-architecture-based, intelligent buildings where components and control networks are integrated, fully interoperable, and where controllers have full situational awareness of the building’s abilities to support the humans inside it. As of 2009, Echelon’s LonWorks Network Systems (LNS) building management technologies have been deployed in about 50 million intelligent network devices across 1 million individual networks. The LNS management system is powered by Versant’s object-oriented data management solutions, enabling rapid data management and analysis across all of these devices and sensors. Further, LonMark has over 400 members using the LonWorks platform, and the platform is an international open standard, ISO/IEC 14908-1, making it a perfect example of the kind of open, standards-based systems that smart grids require. By using these open, object-oriented control systems to enable full situational awareness of the building’s status, users reported:  About a six months advantage in time-to-market for being able to declare the building “green” and intelligent  Several years of saved developer time  Enablement of more powerful modeling features  A significant increase in productivity for maintaining and enhancing the code base that defines the underlying database and data models  An inherently more secure network Energy efficiency in these buildings was also significantly improved. The Balanced Office Building in Germany, for example, reduced HVAC and lighting costs by 80 percent. The United States National Resource Defense Council offices in California reduced total energy use by 45 percent. And the Philips HELIO Lighting Control Application, which is deployed throughout the world by many large companies, uses the Echelon system to help reduce companies’ energy costs by up to 50 percent by creating a data management model that lets lighting systems be automatically controlled depending on inputs from other systems and sensors like HVAC.
  • 13. Leveraging Big Data and Real-Time Analytics to Achieve Situational Awareness for Smart Grids Page 13 of 20 Enabling this degree of automation and interoperability is a form of demand response capabilities, a major portion of utilities’ overall demand-side management goals for the smart grid. And as buildings represent 39 percent of primary energy use in the United States, it is even more critical that utilities’ smart grid data management systems interoperate with the open, standards based building management systems like Versant’s and Echelon’s if they are to achieve situational awareness. Where buildings differ in their data management needs from utilities is in the scale of their data volumes and velocities. Buildings have roughly 72 object classes and 32,000 device instances per building control network, whereas power grids deal with data volumes on a scale more than ten-fold larger, with more than 700 object classes. Additionally, buildings’ sensors and systems need to function in the fastest instances at the second or minute level for things like motion sensors and temperature controls. But grid sensors must function to deliver effective predictive analysis in an extremely wide time range from milliseconds to decades. This essentially means that data management systems for utilities must be able to respond to analysis queries and deliver real interoperability on demand and in true real-time. The telecommunications industry examples referenced at the end of section 4 of this paper reveal the ability of object-oriented database technologies to perform extraordinarily well under extreme data volume and velocity conditions. The Versant Object Database has been successfully tested by China Telecom to process up to 1,000,000 queries per second, earning it the operators’ title of best Object Database technology after testing its scalability against a range of other database technologies. During head-to-head benchmarking between the Versant Object Database (VOD) and other databases (ORM, XML), Verite Group’s VOD- powered Netscope application processed data captures twice as fast with minimal computational resource needs. Open, standards-based data management technologies have discernibly more pedigree than alternative solutions in terms of scaling to meet the volume, variety, velocity, veracity, and validity needs of Big Data. And these abilities are not mutually exclusive. After conducting extensive benchmark tests against several different database technologies, both NoSQL and relational, under multi-dimensional scenarios at different scales and speeds, Versant’s object-based technologies again outperformed other technologies by up to 1000 percent, and in 80 percent of the tests.
  • 14. Leveraging Big Data and Real-Time Analytics to Achieve Situational Awareness for Smart Grids Page 14 of 20 Image Caption: PolePosition benchmark testing shows how Versant’s technologies radically outperform other data management solutions when processing Complex Concurrency operations, which are highly relevant for utilities’ multi-faceted Big Data needs. SECTION 5: HOW TO APPLY OBJECT-ORIENTED DATA MANAGEMENT AND ANALYSIS TECHNOLOGIES IN SMART GRIDS As defined, the purpose of building open, object-oriented data management systems that provide scalable situational awareness for the smart grid is to identify anomalies in normal patterns in real- time so that accurate decisions can be made, and the appropriate actions taken, to positively affect the outcome of a business or process. One of the best examples of how Versant’s NoSQL - based data
  • 15. Leveraging Big Data and Real-Time Analytics to Achieve Situational Awareness for Smart Grids Page 15 of 20 management and analytics technology can address utilities‘ smart grid situational awareness challenges stems from the infamous U.S. blackout that occurred on August 14, 2003. In 2006, the United States Department of Energy found that it was not for lack of action that the outage happened. Plenty of decisions were made, but, as the report also found, the blackout was due, in part, to “lack of awareness of deteriorating conditions,” making it impossible to know if the decisions that were being made were accurate, and the actions being taken helpful. The report recommended that a real-time measurement system and computer-based operational and management tools be developed. Within the guidance of the Electric Power Research Institute (EPRI), research was begun to determine how such a system could be created to connect the multiple utility providers’ grids and provide the situational awareness to prevent such a wide-scale problem from occurring again. To prove that it could be done, EPRI and Versant created a working demonstration technology architecture using sizeable simulated data samples from the hours leading up to the blackout. By configuring the system according to the requirements and open, standards-based architectures outlined in this paper, the demo was able to provide advanced warning that a problem event was nearing, based on the calculation of the phase angular difference and its trend as measured from the data in the synchrophasors. The following diagrams show how the demonstration technology indicates the divergence well in advance of the actual blackout: Image Caption: There were three time windows where situational awareness would have given sufficient time to adequately respond.
  • 16. Leveraging Big Data and Real-Time Analytics to Achieve Situational Awareness for Smart Grids Page 16 of 20 Image Caption: A screen shot from the actual demonstration interface shows that the phase angular difference trend had fallen outside the acceptable range, and indicated that action was needed, more than 35 minutes prior to the blackout. The data management and analysis system architecture of the demo was based on Versant’s new and fully-integrated software development solution that has been applied in other industries: Image Caption – Graphic representation of Versant’s fully integrated software development toolkit. Versant’s platform allows for an automated means to ingest, manage, and analyze large, complex data volumes in real-time. The approach is based on object-oriented programming, allowing data to be modeled as objects and classified into object classes. This produces the right match with the nature of data generated by network topologies like those of the smart grid. The platform’s ability to model data entities as schema according to unified modeling language (UML) also corresponds with the UML reference model for the electric utility established in the IEC CIM
  • 17. Leveraging Big Data and Real-Time Analytics to Achieve Situational Awareness for Smart Grids Page 17 of 20 standard, where interoperability between all network devices used in a Smart Grid is also specified. Adherence to these standards allows for a stochastic topological model to be established between the devices via network configuration models and their associated real-time data. Versant’s development framework can provide solutions that support the following objectives of the energy sector’s smart grid initiatives: 1. Development, storage, retrieval, and management of network configuration models like the PMU measurement network or the smart meter measurement network with NoSQL database technology while still leveraging industry standard APIs. 2. Evolving the afore-mentioned models for suitability in analytical methods for optimal and scalable situational awareness. 3. Data ingestion from multiple sources, including Hadoop/MapReduce, real-time streaming data monitors, such as Twitter, utility data sensor networks, and transactional data sources like those in the financial services industry, and data virtualization sources, such as classic ETL system data. The ingested data can have further context added through semantic enrichment from external data sources offering information on weather, existing assets, and communication network performance. 4. Build a real-time model that depicts connectivity while continuously ingesting new data to provide absolutely current situational awareness. 5. Implement architectural scale patterns commonly found in NoSQL technology to support both scale-up on modern n- core process architectures, and horizontal scale-out patterns in partitioned systems to manage utilities’ growing data volumes. 6. Support NoSQL architectural patterns to enable real-time data analysis for situational-awareness through in-database analytics. The Versant In-Database Analytic Framework embraces a number of key analytic features that can be applied to identify patterns and reveal critical factors for controlling power grid stability under multiple simultaneous events. Based on the availability of network configuration data, as well as real-time data from smart meters, PMUs, or SCADA systems, Versant can model and manage device configurations and ingest and analyze real-time data. As a result, Versant has the capability to provide the necessary platforms and interfaces for numerical
  • 18. Leveraging Big Data and Real-Time Analytics to Achieve Situational Awareness for Smart Grids Page 18 of 20 methods that allow for simulation and optimization at the scale of utilities’ smart grid needs. SECTION 6: BENEFITS OF SCALABLE SITUATIONAL AWARENESS THROUGH OBJECT- ORIENTED DATA MANAGEMENT AND ANALYSIS FOR SMART GRIDS In addition to being able to prevent large-scale and serious problems like the August 2003 blackout, providing scalable situational awareness via the technologies outlined in this paper can realize the following key business benefits for utilities. Asset Management –Routine maintenance and repairs to power lines and other grid infrastructure account for a substantial portion of utilities’ ongoing costs. With a sophisticated data management system that enables advanced analytics like the one described in this paper, fault locations can be more precisely identified and characterized before a truck is even sent to fix it. This can also allow utilities to determine if a truck and crew is needed to fix a problem at all, resulting in immediate cost savings. Power Usage Efficiency/Quality - Demand response and real-time pricing are top priorities for utilities, but most only talk about it in terms of meter data management and measuring kilowatt hours used to enable more precise billing and energy flow. But power quality at the residential and commercial building level needs much more attention. If the data needed to measure this is built in to the underlying data management model this can allow utilities to have a much better understanding of power demand and supply balance. But because each building is different, a database that can scale with complexity and easily interoperate with buildings systems is key. Faster Smart Grid Roll-Out – The object-oriented approaches outlined in this paper and exemplified by the demonstration architecture created with EPRI enable the creation of a modular, toolkit-like approach that enables shorter deployment time, much like the benefits realized by the users of Echelon’s LNS system. Open standards-based systems like these make the technologies and applications understandable and more easily useable by a broader group of people.
  • 19. Leveraging Big Data and Real-Time Analytics to Achieve Situational Awareness for Smart Grids Page 19 of 20 CONCLUSION The potential power of scalable situational awareness through object-oriented data management for the smart grid is very substantial. Utilities are faced with the simultaneously large challenges and opportunities of Big Data, which make achieving scalable situational awareness harder, but also more important and rewarding. By turning virtually every piece of utilities’ infrastructure in to a sensor, and making them fully interoperable and controllable with object-oriented data management technologies, utilities can prevent outages, mitigate potential threats to the network, and realize a range of other important business benefits. The energy industry would do well to learn from other industries that have already conquered the challenges of Big Data, and apply the lessons they have learned to turn the current grid in to a smart grid. REFERENCES AND ACKNOWLEDGEMENTS The author would like to thank Paul Myrda, Dr. John Simmins, Scott Sternfeld and Dr. Gerald Gray from the Electric Power Research Institute, as well as Barry Haaser from LonMark International, and Volker John, Andreas Renner, Vishal Bagga, Victor Dreiling, Yaniv Schwerin, and Torben Rebhan from Versant’s R&D team for their invaluable assistance and contributions to various aspects of this paper’s knowledge base. He would also like to express his sincere appreciation for the insightful comments given by Robert Greene and Dr. Robert Brammer. About Versant Versant Corportation (Nasdaq:VSNT) is an industry leader in building specialized NoSQL data management systems to enable the r real-time enterprise. Using the Versant Database Engine, enterprises can handle complex information in environments that demand high performance, concurrency, and availability, significantly cut hardware and administration costs, speed and simplify development, and deliver products with a strong competitive edge. Versant's solutions are deployed in over 150,000 installations across a wide array of industries, including telecommunications, energy, financial services, transportation, manufacturing, and defense. For more than 20 years, Versant has been a trusted partner of Global 2000 companies such as Ericsson, Verizon, Siemens, and Financial Times, as well as the U.S. Government. For more information, call 650-232-2400 or visit www.versant.com.
  • 20. Leveraging Big Data and Real-Time Analytics to Achieve Situational Awareness for Smart Grids Page 20 of 20 BIBLIOGRAPHY 1. Doug Laney, 3-D Data Management: Controlling Data Volume, Velocity and Variety, February 2001 2. Codd, E.F. (1970). “A Relational Model of Data for Large Shared Data Banks”. In: Communications of the ACM 13 (6): 377-387 3. Jeffery Taft, Paul de Martini, Leonardo von Prellwitz. “Utility Data Management and Intelligence”. Cisco White Paper, May 2012. 4. Alexandra von Meier (California Institute for Energy and Environment – CIEE). “Electric Power Systems – A Conceptual Introduction”. IEEE Wiley. 2006. 5. Don v. Dollen and Bert Taube. “Advanced Computational Techniques for Situational Awareness and Analytics in Power Grids.” (in cooperation with J. Simmins, P. Myrda, G. Gray, S. Sternfeld, V. Bagga). Redwood City: Presentation at Smart Grid Update's Conference “Data Management and Analytics for Utilities”, June 2012. 6. Bert Taube. “How Object-Oriented Data Management enables Smart Energy Control in Buildings.” Santa Clara: Presentation at ConnectivityWeek, May 2012. 7. Paul Myrda, John Simmins and Bert Taube. “Advanced Utility Analytics with Object-Oriented Database Technology”. Accepted Paper to be presented at CIGRE Forum in Lisbon, April 2013. KEYWORDS Situational awareness, Big Data, big utility data challenges, scalability, smart grid, NoSQL-based data management and analytics, energy data categories (telemetry, oscillography, usage data, event messages, meta data), key characteristics of energy data, synchrophasor (PMU), blackout, real-time utility enterprise, five characteristics of Big Utility Data (volume, velocity, variety, validity, veracity), back-end data systems integration, connectivity and interoperability, control network, open standards, smart energy-efficient buildings, analysis and simulation of grid performance, power consumption forecast, grid security.