2. 4.8.15
Best Practices Workshop
Stew Nunn and Bill Robertson
2
Western Interconnection
• A gigantic electrical machine
• Serves electricity to 78 million
people
• Peak power use 152,000 MW
Good morning. My name is Stewart Nunn.
First, I want to thank my wife for helping me put this presentation together because anything worth hearing probably came about as a question or suggestion from her.
I was hired into Transmission Planning in 1993 and went to Distribution Planning in 1994 where I stayed for the next 20 years.
In 2013, I had the opportunity to re-join Transmission Planning to help create Transmission Planning’s new Construction Outage Study Management application with Bill Robertson.
The application we developed is named COSMO, or Construction Outage Study Make Over.
SRP’s Transmission System is a small part of the largest machine in the world called the Western Interconnection.
The Western Interconnection includes the transmission segments that connect generation to load in 14 Western States and parts of Canada and Mexico.
Energy flows throughout the Western Interconnection are governed by a group called the Western Electric Coordinating Council (WECC).
WECC mandates that its member utilities comply with North American Electric Reliability Corporation (NERC) standards and Federal Energy Regulatory Commission (FERC) mandates and laws.
SRP is a member of WECC and serves 2 million of the Western Interconnection’s 78 million customers
As a part of the big machine, our job is to provide reliable power to SRP customers and to prevent situations that may negatively affect other parts of the machine.
On September 8, 2011 the lights went out in the southwest. The lower left, outlined in red, shows the outage area in California, Arizona, and Baja Mexico.
Because of this outage, the Western Electric Coordinating Council (WECC) mandated that new System Operating Limit (SOL) study requirements had to be implemented to ensure member utilities took appropriate measures to prevent similar events in the future.
The System Operating Limit is the maximum amount of power that can safely and reliably flow down a line.
The new requirements meant that utilities could no longer use engineering judgment to authorize planned outages. Instead, Transmission Operators had to verify that an outage could be taken by performing a power-flow study as evidence that the planned outage would not make any other element in the system exceed its rated limit.
Performing studies at this level of detail is a challenge because our electric system continuously changes due to variations in load, generation, unplanned forced outages, and the fluctuating combination of outages taken for construction or maintenance.
As stipulated, the new WECC requirements added an order of magnitude, or 10 times, the amount work Transmission Planning would have to perform.
The way Transmission Planning helps do its part to ensure a stable system is by developing computer models of the electric system that describe the system as it exists at a given point in time, running power-flow analysis on the model to determine if the system is stable and then making recommendations for mitigation if the analysis indicates overloads or other undesirable system states.
The purpose of this presentation is to:
Illustrate the best practices Bill and I used that led to the successful innovation of COSMO in the context of SRP’s response to WECC’s new System Operating Limit study methodology requirements.
In order to do that, I will focus on two areas where best practices helped enable a transformational outcome:
Collaboration
Innovation
Transmission planning worked with stakeholders from various departments in order to identify the problem they needed to solve.
Together, they developed a clear understanding of what had to be done to ensure compliance with the new WECC SOL study requirements.
Subject matter experts analyzed the new requirements and
Evaluated departmental resource impacts
The impact analysis showed that an additional 10 FTEs would be required to do the work using historical methods.
A strategic decision had to be made that boiled down to one of two options:
Either, add more people – Not feasible - or find a way to automate the work
A Decision was made to automate as much tedium as possible out of the process
Next, innovation decisions had to be made
Since no commercial software existed that satisfied our requirements, management decided to have Bill and me develop an in house application to implement the decision to automate the work
As a Best practice and because we were starting from scratch, it was important to make sure that whatever we did would aide in building an information management foundation for additional transformation.
As another Best practice, we cherry picked industry standard application environments like Chrome and SQL Server to build COSMO on.
In order to meet the deadline, we delved into all four paths to innovation by enhancing current best practices, deriving more complete analysis from existing processes, using potential practices we had known about but not exploited, and by creating new ways to perform studies.
http://www.slideshare.net/MarcusEvansIT/presentation-by-dr-michael-rosemann-professor-head-of-is-queensland-university-of-technology-at-the-australian-cio-summit-2012
Analysis paralysis KILLS innovation!
Don’t wait because lag destroys value.
Cost of Delay can be very high and impact strategic mission serving capability.
Value is lost as time slips into the future.
A hard deadline can be an excellent motivator to innovate without delay.
When we started, we faced two fundamental challenges: horsepower and time.
The primary technical challenge we faced was how to take a lot of big problems that took a long time to process sequentially and split them apart into small problems that could be solved quickly in parallel.
We needed an army of cheap labor, so we created one by tying a cluster of 10 SRP standard desktop computers together in parallel to change processing time from days, or weeks, to minutes.
This type of parallel processing is known as High Performance Computing and is used by high-tech industries to manage large data crunching needs for things like cancer and genome research.
The High Performance Computing cluster, or HPC cluster as we call it, enables parallel processing of studies so they are completed many times faster.
As currently implemented, the HPC cluster improves throughput by more than 50 times.
Because of the faster response time, the engineers are able to maintain focus on the study at hand rather than having to divert their attention from one task to another and lose efficiency in doing so.
And by using parallel processing to speed up the work, we shifted the majority of engineering time from clerical tedium to analysis and recommendation.
This graph shows the magnitude of change between historical study methods and studies run through our stack of computers.
Go to GRAPH to Explain the Y axis
In addition to processing studies much faster, all study reports we generate have been macro-enabled so they automatically populate for the study being performed.
This ensures that the reports we send to Transmission Grid Operations are consistent and complete and error free.
As a Best Practice, we worked backwards from the desired outcome to the first step needed to reach the goal. Sounds strange, but that is a key best practice because working backwards creates a form of hindsight clarity that makes it easier to identify the functionality gaps between where you are and the desired outcome.
In addition, working backwards helps prevent the tendency to become distracted by tangents that dis-rail the development effort. This works because the requirements gathering process establishes an honest vision of the desired outcome that can be agreed on when it is reached.
We also employed another software development best practice by defining clear boundaries between applications
By defining clear application boundaries, we enabled concurrent rapid development – Bill was able to concentrate on the integration of the outage research preserved in COSMO’s database with the GE Power-Flow software; and I was able to focus on web application and database development.
Because we defined software boundaries formalized by an information contract that described the information passed from the COSMO side to the Power-Flow side, we were able to integrate the applications without creating functional dependencies.
The development challenge we faced was to be able to build an application from scratch and get it into production before the WECC deadline in 9 weeks.
Every part of the software tool we built, COSMO, was innovated as a collaborative response to functionality requests from production users. We built the airplane while we were flying it.
It was highly beneficial that Bill Robertson had active roles as a requirements stakeholder, developer, and end user.
He had to eat the dog food we made every day. As a result, we could tell immediately when things didn’t taste right and because Bill knew exactly what was wrong, he could describe the error in a way that enabled us to fix it quickly.
In addition, because we chose a highly modular application architecture we were able to test code in production, validate it, and move on to the next step towards our desired outcome.
One thing is for sure. We didn’t have time to over-analyze solutions. By that I mean, we didn’t have time to determine value differences between products selected for the application’s architecture except to choose proven industry standard accepted hardware and software.
Real-time, or Extreme programming, is a rapid development technique, parts of which we borrowed, in order to get the project completed in time. Real-time programming tends to work best under exactly the situation we found ourselves in: a hard deadline, incremental functionality expectations, and a need for really fast value demonstration.
“The secret of change is to focus all of your energy, not on fighting the old, but on building the new.”
― Socrates said that almost 2400 years ago.
COSMO is a departmental asset that appreciates in value by approximately $100k per month
The $100k a month, or $1.2 Million annual return on investment, represents the 10 Full Time Engineers we didn’t have to hire, but does not include additional departmental savings resulting from incorporating the HPC cluster into daily departmental functions.
Transmission Planning’s core functions are undergoing transformation.
The Construction Outage study process has been transformed from 28 plus steps of loosely controlled tedium to a largely automated process with predictable inputs and outputs that reflects our respect for engineering time and skills.
Other department functions can also be transformed by leveraging the information management infrastructure built for COSMO’s foundation.
Transmission planning is moving to a new future that is not bounded by the constraints of the past
++++++
Bill and I have 50+ years of combined electric system knowledge and professional software development experience which we applied to the end goal.
We were both developed serendipitously by time in rank in SRP’s engineering groups. I grew up in Distribution Planning and Bill came to Transmission Planning from the Energy Management System.
Our combined knowledge enabled us to work together as an analyst and engineer to solve a difficult problem in a little amount of time.
Although we developed our skills independently, it took Brian Keel and Steve Cobb’s management insight to put us in the same group, at the same time, to solve this problem. That part of the story is not accidental and may be the real best practice to identify today.
I encourage management to help develop engineers gain professional software development expertise; and, I encourage management to educate, develop, and provide career paths for IT analysts to understand the engineering domain specific to SRP.
Because management chose people with specific SRP engineering knowledge to tackle the problem, we were able to communicate at a much deeper level than what often occurs between developers and end users. This in-turn made it possible for the developers to create something that the users could easily help improve, validate, and extend to meet new requirements as they became known.
In the end, collaboration enabled innovation which led directly to transformation.
Change is inevitable - suffering is not!
