1. Blurring the Lines: The Process of Applying
Systems Thinking to an Ongoing Evaluation
Karen Peterman, Ph.D.
Annual Meeting of the American Evaluation Association
October 26, 2012
3. My First Attempt
The Island Institute’s
Energy for ME project
Place-based
technology education
Middle and high
schools
Real-time electricity
data, iMovie, GIS, Go
ogle Earth
4. My First Attempt
The Island Institute’s
Energy for ME project
Place-based
technology education
Middle and high
schools
Real-time electricity
data, iMovie, GIS, Go
ogle Earth
5. The Oregon Paint Stewardship Pilot Program
www.paintstewardshipprogram.com
9. Traditional and Systems View of
Evaluation
Traditional View Systems View
Courtesy of Beverly Parsons
InSites
10. Changes noted in my role as the result
of applying systems ideas:
The boundaries between myself as the
external evaluator and the client are more
fuzzy.
The relationship has expanded to include
a different type of consultation than I
typically provide to clients.
11. Stages of Change
Baseline Understanding
Trying Out Interventions
Tipping Point
Sustainable Adaptive Balancing
Courtesy of Beverly Parsons
InSites
My training is in developmental psychology, and early on in the ECLIPS program I dug out this systems-based model that I used to teach as part of my intro classes. It always resonated with me and so it was a natural point to revisit as we started this work. I will give a quick explanation of the model, focusing on the connected influence within and across different bands in the bullseye, and explaining that the arrow across the top represents time.
The project I’m working on with ECLIPS is called Energy for ME, and it is the a program created by the Island Institute in Rockland ME. Their mission is to serve the island and coastal communities in ME. They approach that work in several ways, including through an education division. Energy for ME is [read slide].
One of our first assignments in ECLIPS was to draw our program with a systems perspective in mind. As a quick warning, I’m going to ask you to do this yourselves in a bit, so please think about the project you have picked to work on as I review the next few slides. First and foremost, you can see the Bronfenbrenner influence here. Looking at this one year later, I was surprised that I put the client’s institution in the middle. The next ring out was made up of the students and teachers who integrate their own local interests into the school-based projects they were expected to lead. Community interests, “place” and ME make up the next layer. And then the outer layer includes energy costs which are very high and not something that can be controlled by the other entities represented, NSF who funds this work, and the weather (which I had originally left out, as Pat pointed out). So in this model, I imposed boundaries by assigning different characteristics to the various layers of the system and the relationships are reflected through the areas (though in a general rather than specific way). Perspectives are embedded in the model through the local interests in the inner circle and all three components that make up the next band.
I will mention Matt here and his role in ECLIPS as one of our experts. For me, Matt’s work is something to strive for both in how I think about evaluating my client’s projects and in how I think about crafting evaluation questions. Related to the former, when I look at his logic model, I see the interaction between the various systems involved under the umbrella of the project. When I thought about whether/how to apply that to my own work, I decided to stop focusing exclusively on the one project that I had selected for ECLIPS and instead to start trying to think about all of the projects that I evaluate with the client.
First things first though. Our assignment for ECLIPS was to create a fuzzy logic model for our project, and you can see that this version is vastly different. There really isn’t a “center” here and so that is a primary difference, and most of the space is now devoted to the project itself. There is still a boundary between the two, but with this version you can see the specific strategies that the client is trying to use to influence teacher and student outcomes through the program. The strategies are a kind of relationship in this model and you can also see the relationships between different program components. The perspectives piece is missing, which I didn’t notice as I was creating it (but Alyssa pointed out when we were presenting these) and so this is still a work in progress.
During the last year, we also had the good fortune to get a couple of additional grants and so I wanted to take this a step further and think about how all of my work for the education division is related, rather than focusing on Energy for ME alone. This example is a mock up that still needs to be filled in once these other projects get up off the ground. My hope is that this model will help identify common strategies and areas for consistent evaluation practices. Eventually I hope to have this model become interactive the way that Matt’s is so that it can live on the client’s web site and on my own site
At this point, attendees will have five minutes to start drawing their project as a system. After five minutes, there will be a quick reflection about whether it was easy or difficult to get started and whether there were any surprises.
So what does all of this mean about our ideas about evaluation? This is a pair of diagrams that Beverly and Pat were nice enough to loan out for today, and they are reflective of some of the changes in thinking I have experienced in the last year. I definitely held the traditional view at the beginning of the project and my thinking has shifted to the systems view. I still struggle with thinking about the placement of the project itself in the systems view diagram. I also find it an interesting challenge to think about where to place the evaluation in my fuzzy logic model. Placing the questions as Matt did is one thing. Placing myself and the presence/influence of the evaluation on the system I am evaluating is another.
These are examples of the changes I have noted, framed within the context of our unifying concepts.
To wrap things up, I wanted to share these Stages of Change (again courtesy of Beverly). For the ECLIPS project, we were challenged to tackle the first two points on this continuum at the same time – they were helping us develop our baseline understanding while simultaneously trying it out in existing evaluation plans. And the trying out phase is where I still find myself today. The tipping point is reached, apparently, when you start to apply systems concepts to almost any situation such that it is your pervasive viewpoint. You have reached the point of sustainable adaptive balancing when you not only view things through a systems perspective as second nature but thinking about the ways to measure and leverage the different pieces of the system for change is second nature as well. It is a long process, but there are several handy tips and tools that you pick up along the way. For me, that has blurred the lines between myself as the external evaluator and the client but in a way that I think and hope will enhance my work.
