Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Power grid-data-analysis-overview-2013-03
Similar to Power grid-data-analysis-overview-2013-03 (20)
Recently uploaded
Recently uploaded (20)
Power grid-data-analysis-overview-2013-03
- 1. Power Grid Data Management and Analysis PRESENTED BY: TERENCE CRITCHLOW PNNL-SA-94183
- 2. The core of the power grid has changed little over the past 25 years Relatively small number of power producers Large number of consumers Transmission grid moves power from producers to distribution points Distribution network moves power to consumers Information on grid status is relatively sparse SCADA data every 2 sec – 1 min Meter data every month 2 Top Engineering Achievement of the 20th Century – US NAE
- 3. The future power grid must be smarter, not just bigger Climate change Developing nations New applications Constraints Regulatory Social Physical Integration of renewables Distributed generation Real time markets Electric vehicles 3
- 4. Integrating renewables at scale requires faster understanding of transmission grid status In order to meet statutory requirements, renewables must be integrated into the system Do not consistently generate power => Need to be smoothed Do not operate a fixed output levels => need reliable predictions Phasor Measurement Units (PMUs) are expected to be the dominant source of insight into transmission network status ~48bytes /record * 60 records /sec 50,000 * 2.88KB/sec ~= 144MB/sec * 60*60*24 ~= 12.5TB/day 4
- 5. Distributed generation means power production could occur anywhere Consumer based electricity generation is on the rise Amount of grid-supplied energy required by a particular consumer could vary dramatically based on external conditions What happens if the power is not needed? Significant power coming from distribution system could decrease stability 5
- 6. Establishing real time markets will moderate both supply and demand Current prices are fixed What if prices changed every ~5-15min? Utility sets prices based on model (expected avail and usage) Millions of meters receive prices Meter estimates consumption based on price and status Each appliance determines response Responses are aggregated at meter Meter returns proposed / actual consumption To consumer, behavior appears the same 6 Predictive models Smart Meters Adaptive appliances
- 7. EV’s can act as both producers and consumers of electricity Need to be ready to go when needed by driver Discretion on when to re-charge batteries Connected to grid most of the day Does not have to start charging as soon as plugged in Strategy could vary based on where you are By selling stored electricity, could act as distributed generator Could employ a buy-low sell-high strategy 7
- 8. Data analysis is key to maintaining stability of the future power grid Data flow is complex Multiple types of information (pricing, weather, sensor) Information moving in both directions Relatively high, sustained data rates Privacy must be preserved Utilities will require significant analysis capabilities 8
- 9. Effective model development requires a flexible, scalable data analysis pipeline Sensor Streams Data Analysis Infrastructure Data Storage Models over streaming data Accessible Repository Community Resource 9
- 10. Goal: gain insights from real sensor data using event detection models Out-of-sync events Determine when the network partitions itself Requires comparison across different PMUs Generator trip events Sudden drop in frequency that occurs across the network Looking at average behavior of PMUs 10 2TB PMU data set 38 PMUs 1.5 years 53.7B sensor readings
- 11. Our iterative approach uses historical data to validate the models Data-Driven Model Development Use actual data to guide definition of the models Analyze the data Identify events of interest Create event extraction model based on data subset Execute model against entire data set to extract events Validate results Models can be adapted to work on data streams within a distributed, agent- based framework Real-Time Event Detection Models applied to live data streams 11
- 12. Our approach leverages R, Hadoop, and our institutional resources R Flexible statistical scripting language Thousands of packages RHIPE interface to Hadoop Easy to prototype models Hadoop Scalable PNNL Institutional Computing 19,200 cores (used max 2048 cores) 102TFlops 4PB Lustre file system 12
- 13. Initial event detection runs highlighted significant data quality errors Over 10,000 candidate out-of- sync events detected No good models of sensor errors were available Errors Occurred over time Required analysis across sensors Included transient errors Needed to differentiate between data that couldn’t occur and anomalous data 13 13
- 14. Exploratory data analysis is beneficial when you don’t know quite what you are looking for 14 Define initial problem Define model Run model over entire data set Select interesting subsets of the data Analyze results / patterns Model validated Refine model
- 15. This lead to rediscovery of lost knowledge about status flags 16 Originally told flag 132 means bad data A detailed look at regions with high concentrations of 59.999 Hz revealed correlation between certain flag / value combinations > 0.95 After additional investigation, we found specifications indicating any flag > 128 indicates bad data 8B records with bad data flags
- 16. Some PMUs were consistently less reliable than others > 50% of the PMUs report no error flags 1 PMU reports nothing but error flagged data Begs lots of questions we can’t answer with the data we have Are certain devices less reliable How do errors relate to maintenance Are certain locations inherently less reliable 17
- 17. Frequency was unreliably reported when only spurious data was recorded On certain days, there was an (unknown) problem that prevented most data from specific PMUs from being recorded Only a small number of values were present resulting in large gaps Stored data appeared random within these time frames 19 1.19B records removed from specific dates
- 18. Sometimes sensors get stuck repeating the same value 21 Experts thought change should occur at least every 5 sec, data indicated up to 10 sec was reasonable Use geometric distribution to filter out sequences longer than statistically possible These would be difficult to find if we sampled the data or if we were only looking at summaries ~124M records removed 21
- 19. Since the network is connected, everything should look essentially the same There will be time delays between sensors, but the overall patterns should be similar Differences in patterns can signal a network partitioning Frequency data cannot change randomly, physics dictates how much variation between values is possible 22
- 20. Valid sensor data reflects the constraints on the underlying network There should be a strong correlation between a current value and the preceding values An autocorrelation analysis identified areas where the data was completely uncorrelated (compared to high correlation in normal use case) Much of the random data fell into “acceptable” limits, so it would not be identified by thresholds 23~25.5M records removed
- 21. Applying models developed to clean data had significant impact on data quality 2TB of historical PMU data 53.7 B records Identified 9.475B bad records 18% of original data is bad data Defined 4 data cleaning filters Flag based (8.13B records) Missing data(1.19B records) Constant values (124M records) White noise (25.5M records) 53.7B PMU sensor records Filter error flags Filter bad dates Filter repeated seqs Filter white noise OOS freq algorithm 45.56B Records 44.37B Records 44.25B Records 44.21B Records Event Repository Gen trip algorithm 24
- 22. Once data was cleaned, event detection algorithms worked much better Generator trips 329 candidate events detected Most represent real events Also detected unexpected, anomalous data spikes Out-of-sync frequency 73 events detected, instead of 10,000 with original data set No islanding events detected Most reflect offsets / shifts in frequency 25
- 23. Once the basic models work, more interesting questions can be answered Where is the least stable generator? Find the PMU that first identifies the trip Start with data around the trips Freq -15 std dev from the mean Count number of times each PMU is first Least stable generator is closest to that PMU With additional information could triangulate actual generator 26
- 24. We have demonstrated our ability to run at scale Tested on data up to 128 TB Duplicated data set 43 months of data for 1000 PMUs Complete data set analysis in under 10 hours on 128 nodes Good scalability demonstrated Primary limitations are file systems related Could increase number of nodes for faster analysis of large data Hardware & software updated since these tests completed 27 684 PMU mo 1,368 PMU mo 2,736 PMU mo 5,472 PMU mo 43,776 PMU mo
- 25. We are now applying our models to real-time data streams Existing R models Process data faster than data arrives Incremental / windows minimize data requirements Minor modifications to allow filters to work on streams instead of files Being deployed in a framework designed to manage limited resources in a distributed environment Generating artificial, but realistic data streams 28
- 26. How to store and disseminate data is a significant issue within the community NASPI working on Data Repository white paper CPUC Energy Data workshop How can researchers access data efficiently IEEE activity initiated Dec’12 Organized by IBM Brings together leaders from industry, research and academic organizations Goal: a demonstration data center within 2 years Critchlow (PNNL) leads architecture sub-committee 29
- 27. Ongoing and future activities build on the current capabilities Data analysis research Refine analysis questions Incorporate multi-modal data Apply appropriate machine learning algorithms Improve scalability Investigate in-memory solutions Apply to streaming + historical data simultaneously Data facility Define data access policies / requirements Distributed or monolithic? Data transfer capabilities Data standards Application libraries Curation requirements 30
- 28. Thank you 31
Editor's Notes
- I am excited to be here today and to have this opportunity to discuss a project that I lead on power grid data management and analysis. A little background first. I am a computer scientist – a data person – not a power engineer. I was first exposed to the challenges facing the power community about 2.5 years ago, and have been working on understanding the domain better since then. Most of what I have learnt has come from discussions from PNNL power engineers, although I have had some discussions with others in the field as well. I also want to highlight that this is a fairly small project. We have had less than 1 FTE working on this project for about a year and a half. Given the level of effort, our purpose was to establish a foundation from which other projects could build a strong research portfolio. In other words, our focus was not on groundbreaking CS research, but rather bridging the gaps between CS and power grid engineers by developing and demonstrating a scalable exploratory data analysis capability in a domain where such capabilities have not traditionally been needed. An obvious question, given that the grid has been around for 100 years, is why is this a good time to pursue data analysis in this domain?
- The power grid of today is an engineering marvel that has enabled pretty much all of the technology that we take for granted. That is why the national academy of engineers rated it the top engineering achievement of the 20th century. One of the reasons for its success is that the core elements of the grid, and how they interact, have changed little over time. This has provided the stability that we have come to expect and have build on. In the current environment, there are a relatively small number of power producers –commercial power plants, dams, and generators - that generate electricity and send it over a transmission network to a series of distribution points. These distribution points then forward the power on to a huge number of consumers over a distinct distribution network. Because most components of the grid behave predictably, data flow rates have been fairly low – with status updates from sensors arriving every few seconds.
- However, the world is changing and the grid has to adapt. Climate change is heating the environment, changing wind, water, and storm patterns and requiring more power for air conditioning. Nations are becoming more industrialized and less agrarian, increasing the number of people who have – or want – access to electricity and all the benefits that come with it. And demand is increasing on a per capita basis as new applications – from smart phones to cloud computing – increase the amount of electricity required to meet our individual needs. In the face of this increasing demand for electricity, there are real constraints that power companies must work within. There are regulations, like pollution limits and environmental impact laws; societal pressures like the cost they can charge consumers and the number of high-voltage transmission lines that can be placed in a specific area; and physical laws like the amount of power that a single line or substation can handle under different weather conditions. The goal is for this future power grid to maintain the reliability of the current grid while supporting the increased demand and meeting all of these operational constraints. Doing so will require development of a suite of new technologies and capabilities that will allow the utilities to develop a smarter grid not just a bigger one. I will briefly touch on a few of the technologies that are under consideration, how they may contribute to meeting this goal, and the challenges to their adoption – specifically, the integration of renewables into the grid, distributed generation of electricity, real-time markets, and electric vehicles.
- Both governments and consumers are pushing utilities to increase the amount of electricitygenerated by renewable sources. This is a great idea that will dramatically reduce the amount of pollution the industry produces. But it also causes a significant problem: the stability of the grid depends on the balancing of supply and demand in real time. While current providers are predictable and easily controlled, at least most of the time, renewable energy is not. The amount of energy that can produced by a wind or solar farm varies dramatically over minutes, much less hours or days. It is entirely dependent on what is happening at a specific location at a specific time, everything from wind gusts to clouds have an impact. A major challenge is smoothing out these transient changes. Some recent research has shown that batteries may be a useful tool in accomplishing this, but more testing is needed before they are commercially viable. Even if the power supply could be smoothed, it is hard to predict exactly how much power will be generated by a site over an hour or a day. This is important because traditional sources take time to ramp up. You want the coal powered plant on-line on a calm, cloudy day when demand is going to exceed the supply of renewable energy, but want it down when the renewable energy will be sufficient since it is redundant. What you don’t want is it to be running when it is not needed – or even worse, not running when it is. That’s why predictability is important. To get ensure the balance is right, a solid understanding of what is happening on the grid right now is required. In the past, with stable sources, information could flow in every few seconds and everything would be ok. In this new environment, you need a much faster response since things can go bad quickly. PMUs are being deployed to provide that real-time situational awareness. They provide a time synchronized view of the grid 30-60 times per second, allowing a much faster response. Assuming, of course, that you can pull a signal out of all of that streaming data.
- In addition to the large commercial renewable sources, consumer based electricity generation is on the increase. Spurred on by government incentives, awareness of climate change, and reduced costs, businesses are putting up solar panels on their buildings and people are putting windmills on their farms and panels on their houses. From a planning perspective, this puts a lot of pressure on the utilities since they need to understand not just how much power you generate and when, but also how much of it you need at that time – since that determines how much power the grid must either accept or provide. For example, on a sunny weekend day, the panels on a factory may generate more electricity than the empty building requires. Under most circumstances, that means that you would be selling the excess power back to the grid. However, if you are running an extra shift in order to get a big order out, you may actually need more power that you are generating – pulling power from the grid instead of selling power to it. Problems can arise when the amount of power produced is more than is required within a small area. Recall that the distribution system is designed to take power from the transmission system and distribute it to consumers. The reach of a distribution system is relatively small, covering a few neighborhoods or possibly only a single business. The system is not designed to push power back onto the transmission system. So, if more power is being generated than is needed locally, a distribution system can’t do anything with it and needs to shed that power somehow. Right now, this doesn’t really happen because there are so few distributed generators, but it is a concern as more distributed generation comes on line.
- One possible way to deal with this problem – as well as to manage demand overall – is to move to a real-time pricing system. Right now, most consumers pay a single price for electricity regardless of when that power is used. A watt in the middle of a Wed afternoon costs exactly the same as a watt at midnight on Sunday. That’s because using traditional power meters, it is easy to figure out a consumers total usage but not break it out on a minute-by-minute basis. But what if people paid less when power was plentiful and more when it was not? A new infrastructure that brings together predictive modeling, smart meters, and adaptive appliances would be required to set up a power market that adjusts itself several times an hour based on what is actually going on. Once in place, though, the utility can set a price based on what it projects power availability and consumption to be, send that price out to consumers, and have them adjust their behavior based on a combination of the price and their internal optimization algorithms. To be workable, appliances need to respond to pricing information automatically, based on use-defined preferences and internal algorithms. The result could be a change in behavior that is effectively unnoticeable to the consumer but which aggregates into a significant change for the distribution system. For example, if you are a single person living alone in a studio apartment, chances are good that your water usage follows a fairly predictable pattern – at least most days. Maybe you shower before coming in to work, do dishes in the evening, and do laundry Sunday nights. Right not, your water heater keeps the water at the same temperature constantly. Instead, it could let the water cool down during times that it is unlikely to be used and heat it up hotter than normal if the power is particularly cheap. There would be no difference in what you see, but there could be a huge difference in consumption – especially when aggregated over an entire city. Simulations have shown that by having the devices adjust utilization in response to prices, the overall demand on the grid is effectively moderated. This is a powerful result because the system is currently designed to handle peak loads, and there is unused capacity at off-peak time. By providing incentives to balance the load across the day, the current infrastructure could support a significant increase in overall demand.
- One of the interesting applications of this type of real-time market is with electric vehicles since an EV can be both a consumer and source of power. Obviously, just like the water needs to be hot when you want to shower, the car needs to be driveable when it is needed. However, in many cases, the EV could be connected to the grid for the majority of the day. If it knows when it is needed, what its storage efficiency is, and how much power it needs to get where it is going, it could effectively try to execute a buy-low sell-high strategy. By charging itself when power is cheap – for example late at night, at lunch hour, or when connected to a free power supply like a solar charging station - and selling excess power when it is expensive, for example in the middle of a hot weekday afternoon, the car could effectively reduce its operational costs.
- All of this is great in theory, but as previously mentioned, a lot of technology will need to be developed in order to make it a reality. The part of the puzzle that I am particularly interested in is how data analysis plays a key role in maintaining the stability of this future grid. Under this scenario, there will be an extremely complex data flow where multiple, heterogeneous data streams will need to be analyzed in real time. Data will be flowing back and forth between producers and consumers of the electricity at the rate of megabits per second, and outside information such as weather conditions and model projections will need to be incorporated into the analysis. Throughout all of this, privacy and security must be maintained. The utilities will play the key role in ensuring information is effectively exchanged. Making sure that they have the analytical capabilities to effectively process these streams will be critical to ensuring the stability of the grid. Right now, as a general rule, they don’t.
- Creating it will require a flexible and scalable data analysis pipeline that supports the development of models capable of running over data streams to identify events of interest. My project has been focused on demonstrating such a pipeline and I will spend most of the remaining time discussing our work in this area. In addition to model development, we are also looking at how to effectively disseminate information to trusted collaborators. Storing the original data is not particularly challenging at the current data volumes, and it is unclear whether it will ever be a significant problem. However, determining how to securely and efficiently disseminate the information is currently a topic of interest among the broader research community. I will touch on this topic briefly at the end of my presentation.
- Our initial goal was a modest one: to use real PMU data to detect two types of events that we know are of interest to power engineers – out of sync events and generator trips. These events are well known and well understood, so there was little issue in generating models to identify them. Out of sync events occur when PMUs record significant differences in the state of the grid. The grid is supposed to be connected, so if sensors are reporting a real difference in state, meaning the network has been partitioned –this is a huge deal. To detect these events, we start by determining how each pair of PMUs typically compare to each other. Because these sensors are geographically distributed, it takes time for an event detected at one location to be seen at the second. And, because the network is constantly in flux, there will always be variations in the measurements reported. Creating a simple model that accounts for both time and variance gives us a baseline for the relationship between the two sensors. Given that model, we can determine when the two sensors appear to be disconnected by recognizing that the sensors are reporting different events – for example different changes in frequency – which indicates that the sensors are reporting on different networks, or partitions, of the grid. Since we don’t know where the partition might be, we need to analyze all pairs of PMUs to identify it. A generator trip – as shown in the image here - is much simpler to identify. It is characterized by a sudden drop in frequency across all of the PMUs in less than a minute, typically under 10 seconds. This may be followed by a recovery with a gradual increase to a higher frequency. We start by calculating the average frequency, omitting the 2 most extreme PMUs, and smooth the result by averaging again over a small time window. We then look at the slope of the line at each time – ie the rate of change from the previous point - to determine inflection points, slicing the smoothed averages into sections of increasing and decreasing average frequency. Those segments with a maximum decreasing frequency of at least 0.12 Hz / second – a number we obtained through interactions with domain experts classifying trips – are returned by the model as trips. To evaluate our models, we use a relatively modest 2TB data set – which represents 1.5 years of data from 38 PMUs, or about 53.7B individual sensor readings.
- Our plan was to use the data to validate the models that we had created using an iterative approach. We would run the event detectors, look at the results, and adapt the model as appropriate until we were able to distinguish the events accurately enough for our local power engineering team to accept the model. Once the model was validated, we would then convert it to work on streaming data instead of the historical data and to work within a distributed agent-based framework so that it could be applied directly to our data feed.
- To perform that validation, we leveraged three complimentary capabilities: the R programming language, which gave us a flexible way to define our models The hadoop infrastructure, which when combined with the R package RHIPE gave us an easy way to apply our models to the entire data setAnd our local cluster environment. While not a dedicated hadoop cluster, this gave us access to more than enough disk and compute resources to run our models in a relatively timely manner
- Unfortunately, our initial model run didn’t go as planned. We identified over 10,000 out-of-sync events, when we were expecting under 100. What was the problem? Bad data. The sensor stream was full of erroneous data that needed to be filtered out so we could focus on the signal and not the noise. Of course, we didn’t want to filter out all of the anomalous data since we were looking for relatively rare events. Instead we wanted to filter out just the data that meant that the sensor was not reporting correctly – which we defined as data that was not actually possible. Obviously, we didn’t know exactly what these errors looked like in advance – if we had, filtering them out would have been easy. It turns out that detecting them required analysis over both single data streams and across multiple data streams.
- Our approach was to use exploratory data analysis to identify – and create filters for - different categories of bad data. In EDA, you start with an initial problem you are trying to solve. In our case, that is generating models to filter out the bad data. Then you define a model that is designed to give you insight into what is happening in the data. That model may not actually be a filter, however. For example, it could be generating a set of statistics over the data to get a better idea of what is happening across the data set, or in a particular region of interest. Running the model over the data set gives you insight into which subsets of the data are of particular interest. From there, you analyze the results – perhaps identifying a subset of data for further analysis, perhaps refining the model, perhaps adjusting your initial problem definition based on what you find. You repeat this process until you have a model or set of models that address the problem you are trying to solve. The exploratoryaspect of this approach was important to us since we did not know exactly what types of errors we were looking for. The iterative aspect of the approach allowed us to refine the filters until they captured the dominant data errors.
- We started by looking at plots for a number of simple statistics to see if we could determine any patterns that characterize bad data. We looked at min, max and average frequency over time for both the entire data set and individual PMUs. We also looked at the distribution of specific values over time and by PMU using Normal quantile plots. Normal quantile plots are an easy way to visually compare the recorded values with what you would expect to see if the data was represented by a normal distribution – a reasonable distribution for most large data sets. You can see that for AR the distribution of actual values– the blue points – matches what you would expect – the solid line - pretty closely. BA, on the other hand, has far more 59.999 values than you would expect. This is interesting because the data encodes values as an offset from 60.000 hz, so 59.999 is is represented as -1, a value often used to refer to a bad record. So, this looks like it may be worth investigating further. --------------------------------To generate the plots, Sort the freq values generated by the PMU into ascending orderFor each point, calculate its quantile (what percentage of the data is below it)Plot the (quantile, freq) pairsThe solid line is what you would expect to see with a continuous normal distribution. Note: BA distribution has a different mean / stddev than the AR distribution, because of the large number of points at 60hz, however it is plotted against the same normal distribution as AR. The normal distribution was derived from the entire data set. Calculating quantiles using map/reduce is not a simple block summary method – need to recombine information across the whole data set – not by time but by PMUIn this case, data is actually discrete, so we can simply tabulate it and turn it into quantiles from there (this is a good approximate method even when the data is truly continuous)
- In addition to the frequency data, the PMU returns a status flag as part of its data packet. We knew that this flag could indicate that the sensor had detected an internal error since we had been told that when the flag was set to 132, the data was bad. In fact, when this flag was returned, the frequency value was set to -1 by the PMU. We had already filtered out those records. However, when we looked at that BAin more detail, we found that when the value was -1 the status flag had often been set to 129. Intrigued, we looked across the data set to see how the frequency was distributed for each flag value. We expected to see a normal-ish distribution, like what you see for flags 2 and 3, for all of the flags except 132, where we expected to see -1 as the only value. For several flags, however, we found that the values were almost always the same –either a 0 or -1. After this insight, we hypothesized that most of the higher value flags were in fact errors – even though our local experts didn’t realize it. Eventually we were able to dig up an old user document that did, in fact, note that any flag value greater than 128 indicated the sensor data should not be used. While it would have been nice to know that from the beginning, being able to rediscover this lost knowledge validated our approach. It also resulted in over 8B records being removed from the data set.
- Interestingly, the flagged data was not spread uniformly across all of the PMUs. Over half of them generated no flagged data while one reported nothing but. It would be interesting to look at whether certain devices are less reliable than others, how the error flag generation corresponds to installation and maintenance records, and whether certain locations are inherently less reliable – for example because of weather – than others. Unfortunately, we don’t currently have access to the data we need to answer these types of questions.
- Missing error flags weren’t our only problem, however. We looked at median frequency generated by a PMU in 5 min increments and saw some odd patterns emerging. For example, BE appears to be oscillating within an acceptable range – but the values for the last couple of months are changing more than we would expect. For BI, the oscillations are much larger than expected but for only a short period of time. This is an example of where thresholding could be used to eliminate some records – but it wouldn’t really address the underlying problems in the data and many bad records would remain.
- So, what was the problem. It turns out that on some days, for some PMUs, our data stream did not record many values. We don’t know what the underlying problem was, but most of the data was simply missing. Worse, on those days, the values that we did get were pretty much random – as you can see by the gaps and noise shown in the top charts when the count of missing values, shown in purple, are high. It is interesting to note that many of the affected PMUs came back on line and appeared to work correctly after time had passed. Unfortunately, we can’t tell if that is because of maintenance performed on the sensor or something else. In any case, removing all data from times where the data is not regularly reported removed another 1.19B records from the data set.
- While the error flags and missing data gaps are fairly obvious instances of bad data – at least in retrospect –we found a couple of additional problems that were much more surprising. It turns out that having the sensor reading remain constant for an extended period of time is also indicative of a problem. Frequency should change regularly in response to changes in the supply and demand for power. Given the dynamic nature of this relationship, our local experts thought that frequency should be changing at least every 4-5 seconds. Based on our analysis of the data, we knew that some sensors exceeded this time limit, but we were not sure where to draw the line between an unexpectedly long period of stability and a stuck sensor.
- Plotting number of repeated values in sequence against time, we ended up with a geometric distribution. Interestingly, we found several cases where the sensor appeared to be functioning correctly and yet same value was returned for between 8 and 10 seconds – twice as long as our experts predicted. These are a great example of the rare but correct events that we want to keep in our data set to support additional analysis. However, we also found some sequences that were longer than 45 seconds. Something that was effectively statistically impossible, yet occurred in some cases with surprising regularity. After additional consultation with our local experts, these cases were determined to be bad data. It is worth noting that this type of error couldn’t be identified with either thresholding, because the values were valid, or detailed analysis over sampled data, because you would loose the pattern that you are looking for. Surprisingly, we eliminated about 124M records from the data using this filter – even after the previous filters had been applied.
- Finally, the last major type of bad data we found requires recalling that, under normal circumstances, all of the sensors on the network are reporting on the same events. There are time lags because of geographical distance between sensors, and slight variations based on where the sensor is relative to the various generators, but essentially all the sensors should be seeing pretty much the same thing. If the patterns being reported by two sensors are significantly different, then this is an indication that the network has been partitioned – and should be picked up by our out-of-sync event detection model. Unfortunately, sometimes these differences reflect sensor errors not partitions. One way to determine which is happening is to recognize that there are physical laws that constrain the problem – electricity has a finite speed, and the amount of electricity passing through a sensor cannot change arbitrarily in subsecond times, so patterns like AX should not occur.
- In other words, there is a continuity to the environment that dictatesthat there should be a strong relationships between the current state of a sensor and its previous states. This type of relationship can be represented as a correlation between the sensor readings. As BD and AK show, typically there is an extremely high correlation even over multiple seconds. However, it appears that AX is generating random data – effectively producing white noise – instead of valid readings. For sensors that are misbehaving in this way, the correlation drops effectively to 0. Not to beat a dead horse, but since many of these random values fell within acceptable limits, thresholding would have missed most of the 25.5M records that we eliminated from the data set using this filter.
- So, to quickly summarize. We were able to dramatically clean up the data by finding4 major categories of errors in the data that were previously unknown to us. These errors accounted for 18% of the total data set, which was a surprisingly large number of bad records that needed to be removed before the event detectors could work properly. The impact of doing this cleaning was dramatic.
- Once the bad data was removed, we were able to run the event detection algorithms and extract a much more reasonable number of events – most of which correspond to real events of interest to our analysts. In particular, we are down to only 73 out of sync events detected instead of the 10k we initially found and we identified 329 generator trip events, most of which have been validated by our experts. We have not eliminated all of the “bad” records from the data, however, and there are certainly some categories of errors that we haven’t yet modeled. Some of the events identified as generator trips correspond to transient spikes in readings from a single PMU, which then quickly returns to normal – giving the appearance of a sudden drop in frequency. And the most common type of out-of-sequence event we detect is when a single PMU suddenly shifts its frequency, up or down, then continues to mirror what is happening on the grid in general. At some point, it returns to normal. In these cases, it seems like the sensor is temporarily miscalculating where 0 is, then is corrected. These events are of interest to our local experts, but are very different from an islanding event where multiple PMUs would separate from the network then reconnect, but not mirror each other in between. Even without additional filtering, however, our models perform acceptable for our purposes. Now that we have generateda workable set of filters and models, there are 3 things we need to do.
- First, we want to explore more interesting questions. For example, we wanted to identify where the least stable generator was located. We don’t actually have all of the information we need to calculate that, but we can approximate the answer if we assume that each PMU has a single power source located next to it. The PMU closest to the least stable source will be the one that most often sees the trip first. To find the PMU that is the first to see a generator trip, we start with the trips our model previously identified, and look at the data for the minute immediately preceding the event. We then identify which PMU first drops more than 15 standard deviations away from the current average frequency. That sudden decrease indicates when which PMU first detected the event. Then simply count the number of times each PMU is first to see the trip. Interestingly, there are clear differences between PMUs, with one PMU being the first to see over 20% of the total number of trips we identified.
- Second, given we started with a relatively modest data set, we want to see how well our approach would scale, as the data set size increased. We are pleased that we were able to analyze 2TB of data in under 10 minutes and 128 TB – or over 43,000 PMU months worth of data - in under 10 hours on just 128 nodes. This is promising given we have over 600 nodes available on the cluster. CPU capacity is probably not our biggest concern, however. It appears most of our runs are IO bound, and you can see the scaling leveling off fairly quickly for the small data sets. That isn’t great performance but it isn’t horrible either – especially given we are running on a shared cluster with a shared filesystem not a dedicated hadoop cluster with local disk. Since we ran these tests, there have been both hardware and software upgrades that should reduce the overall time to solution – although it is unlikely that they will dramatically change the shape of the curves.Overall, we were pleased with these numbers given the current data set sizes and the small number of times we expect to have to run an analysis over a data set on the order of 128TB.
- Third, we want adapt our models to work on real-time data streams. Fortunately, this transformationhas been relatively straightforward. The modelsalready uses data incrementally so we don’t have a problem with unbounded memory. And, because the statistics behind the models are fairly simple, the existing R code can process data faster that it comes in – so we don’t need to recode in an alternative language. The only change that we had to make was to switch from reading files to reading data streams. We have begun deploying these modified models within a PNNL distributed agent-based framework to work and modifying them to work on real-time data streams. That work is progressing well and we expect to have a demonstration of that capability within the next couple of months. ----------------------Diagram shows generator trip events being recognized (red)Flagged data being filtered (green)Whitenoise events (grey)Repeated sequences (blue)
- Finally, beyond storing and analyzing the data within a single organization, the question of how to disseminate data effectively has become increasingly important to the broader power grid community. Over the past few months there have been multiple discussions and several meetings focused on this topic. The North American Syncro-Phasor Initiative – NASPI – is putting together a whitepaper on data repositories and archiving for PMU data which is expected to outline the questions that a utility needs to answer before it can effectively create its own data archive. In January, CPUC has had a workshop on data dissemination requirements and issues with the current system that brought together utilities and researchers in California to present their perspectives. This focused primarily on distribution level data, and so privacy concerns were frequently raised as a key issue in these discussions.And finally, in December, IEEE initiated an Industry Connections activity on this topic. This group is led by Jinjun Xiong out of IBM Watson. The IEEE activity is particularly interesting because it spans all energy data and brings together organizations across the country with the goal of developing a data facility specification complete with a demonstration facility within 2 years. I am currently leading the architecture committee within the IEEE activity, with the deliverable of a technical whitepaper by the end of the CY. Given PNNL’s expertise in both data and power, we are investigating options for establishing a pilot data center in Richland – including potential industrial collaborators.
- This has been an interesting project that has allowed me to gain a better understanding of the problems facing power community while demonstrating a relevant data analysis capability at a scale that is unusual in this domain. We are looking to build on this successful demonstration and pursue a research agenda that would allow us to answer significantly more complex questions. We expect that this will lead us to development of models using multiple, heterogeneous data streams, at larger scale. It would also be interesting to apply machine learning algorithms to the data to see if unsupervised analysis could identify events of interest to our engineers. Our current approach is fast enough for the data set sizes we are analyzing. If that changes, we may need to investigate alternatives such as in-memory analysis frameworks that provide faster response times. Right now, we treat our data streams as distinct from our historical data. It would be interesting to see how our environment could be extended to analyze both historical and streaming data concurrently - particularly in the case of multimodal data. For example if an unusual situation is evolving, it could be useful to pull up historical situations that are similar to what is currently happening – not just based on PMU readings but also generation capability, grid configuration, and weather – and see the details of what actions were taken and whether or not they were successful. Successfully pursuing this agenda will depend on a variety of factors, but one of the major ones will be the establishment of community resources that make the relevant data available. While domain specific data centers are becoming more common, there is still a significant investment in both engineering expertise and funding that needs to be made in order for one to be stood up – particularly in a domain with serious privacy and security concerns. With the level of discussion within the community about the need for such a center, hopefully the resources will be allocated in a timely manner.