5. OUTLINE
•
What is an online report?
•
Lessons learned from creating online reports
– Online reports require teamwork
– Online reports are powerful and dynamic
– Online reports require time and money
6. OUTLINE
•
What is an online report?
•
Lessons learned from creating online
reports
– Online reports require teamwork
– Online reports are powerful and
dynamic
– Online reports require time and
money
•
Why invest in online reports?
7. OUTLINE
•
What is an online report?
•
Lessons learned from creating online reports
– Online reports require teamwork
– Online reports are powerful and dynamic
– Online reports require time and money
•
Why invest in online reports?
•
Feedback from users
8. OUTLINE
•
What is an online report?
•
Lessons learned from creating online reports
– Online reports require teamwork
– Online reports are powerful and dynamic
– Online reports require time and money
•
Why invest in online reports?
•
Feedback from users
•
Online reporting & you!
9.
10. OUTLINE
•
What is an online report?
•
Lessons learned from creating online reports
– Online reports require teamwork
– Online reports are powerful and dynamic
– Online reports require time and money
•
Why invest in online reports?
•
Feedback from users
•
Online reporting & you!
11.
12.
13.
14.
15.
16.
17. How many of you have developed
an online report like this?
How many of you have seen an
online report like this?
18. OUTLINE
What is an online report?
•
Lessons learned from creating online reports
– Online reports require teamwork
– Online reports are powerful and dynamic
– Online reports require time and money
•
Why invest in online reports?
•
Feedback from users
•
Online reporting & you!
42. OUTLINE
What is an online report?
Lessons learned from creating online reports
Online reports require teamwork
Online reports are powerful and dynamic
Online reports require time and money
•
Why invest in online reports?
•
Feedback from users
•
Online reporting & you!
43. Why are online reports a good or
bad investment for your projects?
49. OUTLINE
What is an online report?
Lessons learned from creating online reports
Online reports require teamwork
Online reports are powerful and dynamic
Online reports require time and money
Why invest in online reports?
•
Feedback from users
•
Online reporting & you!
50.
51. OUTLINE
What is an online report?
Lessons learned from creating online reports
Online reports require teamwork
Online reports are powerful and dynamic
Online reports require time and money
Why invest in online reports?
Feedback from users
•
Online reporting & you!
Who Are We?At Outlier at CEMSE | University of ChicagoWe empower people to advance and improve education with the knowledge, tools and support to realize change.
Here is our plan for today. Please feel free to ask questions along the way.
Ask Audience:How many of you have written a regular PDF report?How many of you have written a PDF report and posted it for people to download online? How many of you have developed an infographic or other data visualization?
We will focus on the most recent report we developed: Building an Operating System for Computer Science Education. Goal: was to establish a more comprehensive understanding of our nation’s current high school computer science (CS) teaching population, the support they have, and contexts in which they teach.For this tech-savvy and connected CS community, it made sense to create an innovative and different online report. What is an online report? -presents information in an interactive, engaging format -Includes data visualizations, audio and video clips.Goal: make information accessible and readable for a broad audience.
Here is one part of our report that examined the supports and barriers CS teachers faced in teaching CS. This challenges section section is expandable.
This is what the expanded section looks like. And then clicking on each of the arrow reveals further information our findings and includes audio and video quotes.
interactive visualization
Audio and video clips
Courtney-Now I will discuss our lessons learned from creating online reports. We’ve distilled these lessons down to the following three points:Online reports require teamworkOnline reports are powerful and dynamicOnline reports require time and money
Let’s start with the first point: We learned very quickly that the development of an online reports require teamwork. It requires the right team of people and is a true test in collaboration.Ask Audience- How many of you have access to a web development team within your organization? How many would have to hire outside your organization? The most obvious support evaluators need in creating an online report is web development. If you are an evaluator and a web developer, than you are one step ahead, but most of us don’t have web development as a skill. Sarah and I identified this limitation within ourselves immediately, as neither of us have web development or design backgrounds. We set out to recruit the right team of people with the right skillset at the very beginning of our online report development.For this computer science report, we had a couple different teams of people involved at different points in the online report development process.In total, 12 people were involved.
First, we had the research team.Sarah and I fall into this categoryConsisted of 8 researchers who worked on data collection, analysis, and report writing.Different roles: 2 Co-PI’s, 4 researchers in charge of data collection, one research in charge of quantitative analysis, and one admin.
The second group of people involved in the development of the online report was two people from Visual.ly marketplace. For the landscape study portion of this report, we had decided we wanted to produce an infographic of our findings and an interactive map of professional development locations. We decided to work with them for a variety of reasons:Our internal design/web development team did not have the time to fit this piece of work in their scheduleSarah and I have followed visual.ly online for quite some time and really loved the infographics they put together. Right around this time Visual.ly began Visual.ly marketplace, a service that pairs researchers with the technical talent they need to develop visualizations.Drew, a project manager at visual.ly, talked to us about our project and then they paired us up with a designer (kelly) they felt would be able to help us meet all of our visualization goals. We worked with visual.ly over an 6-week period to develop the infographic and interactive map. From day one, Visual.ly clearly outlined the process that we would use for the development of the infographic and interactive map: we would have three rounds of review on the look and feel of the infographic, five rounds of reviews on the visualizations, and then a final product. Period. This meant that we needed to pay close attention to each version visual.ly gave us to make sure we provided an extensive list of edits/changes, because we only got to do that five times. Some of you might think: wow, five times is a lot! Well, it isn’t when you have 8 perfectionists looking at it and reviewing it. There can always be something done better, or a new way to think about something, but at some point there has to be a line drawn in the sand and someone to say “Enough.”
With visual.ly helping with the overall look and feel and the online report and the development of the infographic and interative map, it was left up to our internal designers/web developers to actually build the entire online report site, and apply the look and feel developed by visual.ly to the online report homepage as well as the remaining 4 strands of the project. Our internal tech team consisted of these three individuals. Margaret designed the report pages, George programmed the report using SHTML along with Javascript, Jquery, and Ruby on Rails, and Hannah created the functionality for the interactive map of professional development locations.
Establishing how many rounds of review would occur, how edits would be communicated to designers/developers, and set-in-stone deadlines was absolutely key, especially on a fast moving project like this one was. We at CEMSE have become total Asana groupies. Asana is a free, online project management tool. If you haven’t seen Asana, do yourself a favor check it out after this meeting. Asana has organized our team, our tasks and deliverables in one central location, and has made our workflow process crisper and even more dynamic than it was before.We used Asana to manage the development of the online report. After the development of the first strand (landscape study), we worked out a timeline with our internal designers/developersthat included content development, wireframe development, inserting content into the wireframe, and making any edits to the online report.
SarahThe goal is to present findings and data in alternative formats, including the use of video clips and data visualization.
This part of the report documented challenges in sustaining computer science education programs. This is a quote from a professor. If this were a paper report, we would just be able to include the paragraph of text. But because the report is online and we recorded all of our interviews, we were able to include video.
Jeff GrayCS professor University of AlabamaPlay part of video clip here to demonstrate that Jeff’s voice is more powerful than reading text.
Data in raw form can be completely overwhelming, hard to pull out findings and themesHere is a look at the excel spreadsheet we used to populate a map for our computer science education report. The spreadsheet had about 80 rows, with each row containing data for one professional development opportunity, and hundreds of columns that contained the data from each question on a questionnaire. We used this spreadsheet to develop and populate this…
This is our interactive map of computer science education professional development opportunities.
When Chicago dot is clicked on, more information is revealed in the grey box.
Courtney-Online report development requires both time and moneyMaking a more dynamic, visually appealing report requires more expertise at the table, which in turn requires more collaboration, more time, and thus more money.
Iteration took time.The process was INCREDIBLY iterative. To showcase this point, I want to highlight the infographic we developed for the landscape portion of the ACM project. Visual.ly developed this infographic for us. Step 1 in the process was to provide them with a document with all of our findings that we wanted to visualize. What you see on the screen is a snapshot of an 18-page word document we sent to visual.ly. Along with this, we had to fill out a one-page standard form they gave us, asking us things like (1) what are our goals for this visual, and (2) what we wanted the look and feel to be like. From this document, here is version 1 of the infographic…
For comparisons sake, here are all three versions side by side.
For comparisons sake, here are all three versions side by side.
For comparisons sake, here are all three versions side by side.
For comparisons sake, here are all three versions side by side.
Building online reports takes additional money, but the more you create online reports, the cheaper it becomes as you develop the skill and templates for presenting information.We now build in funds for creating online reports. Clients like this option because it’s easy to disseminate the results of their evaluation or study. Some clients have turned down this option, citing cost as their concern. A basic online report using a flexible template we’ve created costs roughly $5,000. A more customized, larger report costs $10,000. This is to cover our designer and programmer time. This does not include our data visualization support from Hannah. Her cost depends on the project. The Visual.ly project of the infographic and map cost a total of $4,500.
Building online reports takes additional money, but the more you create online reports, the cheaper it becomes as you develop the skill and templates for presenting information.We now build in funds for creating online reports. Clients like this option because it’s easy to disseminate the results of their evaluation or study. Some clients have turned down this option, citing cost as their concern. A basic online report using a flexible template we’ve created costs roughly $5,000. A more customized, larger report costs $10,000. This is to cover our designer and programmer time. This does not include our data visualization support from Hannah. Her cost depends on the project. The Visual.ly project of the infographic and map cost a total of $4,500.
Building online reports takes additional money, but the more you create online reports, the cheaper it becomes as you develop the skill and templates for presenting information.We now build in funds for creating online reports. Clients like this option because it’s easy to disseminate the results of their evaluation or study. Some clients have turned down this option, citing cost as their concern. A basic online report using a flexible template we’ve created costs roughly $5,000. A more customized, larger report costs $10,000. This is to cover our designer and programmer time. This does not include our data visualization support from Hannah. Her cost depends on the project. The Visual.ly project of the infographic and map cost a total of $4,500.
Building online reports takes additional money, but the more you create online reports, the cheaper it becomes as you develop the skill and templates for presenting information.We now build in funds for creating online reports. Clients like this option because it’s easy to disseminate the results of their evaluation or study. Some clients have turned down this option, citing cost as their concern. A basic online report using a flexible template we’ve created costs roughly $5,000. A more customized, larger report costs $10,000. This is to cover our designer and programmer time. This does not include our data visualization support from Hannah. Her cost depends on the project. The Visual.ly project of the infographic and map cost a total of $4,500.
We created this generic report template that we can apply to future evaluation projects, which eliminates the cost of designing a new layout. This template is adaptable so it can be customized for future projects.
So here are our 3 lessons learned.
Sarah
Ask audience: Why are online reports a good (or bad) investment for your projects?
Why invest in online reporting?-Visibility-Access to information immediately-Lasting Resource: Rand shared our computer science education report with a colleague who was interested in expanding CS education in Chicago. -Presentations-Funding-Dig deeper into data-We are all consuming more content online. -It’s easier to link to other online spaces: linkedin, twitter, pinterest, etc. -People can pick and choose which content is relevant to them to read -We want to reach out to a broader community. In our field of education, teachers usually won’t read PDF research reports, but they may read an online report.
Online report changes the way we can disseminate our findings. Here you can see that we pinned a copy of the infographic on pinterest, as this forum is becoming very popular for designers and researchers alike to find visual inspiration for data reporting.We now read so much content in the online space at our computers or ipads or even our phones. We wanted our work to be accessible within that online space.
Jeff said that twitter was one avenue he used to share out his report. You will see here that his tweet was re-tweeted three times – so, not a ton, but you will see that one of the groups that re-tweeted it was Alabama Engineering with 703 followers. That is pretty good exposure for our report.
It was easy to disseminate our report via email blast as well. We were able to track page views using Google Analytics (next slide.) The spike in mid-August is a result of an email blast we sent to over 800 teachers who completed our survey as well as a group of about 40 other research participants.
We used Google Analytics to track page views- We had a total of 7,191 page views between May (when we released a large piece of the report) and the end of September. This is probably 7,091 more views than we would have had with a print report! It was helpful to see what specific pages users were looking at and how much time they spent on a page. Free if less than 10 million hits/monthBounce rate: single page visits% exit: percentage of exits that occurred from a specific page
We interviewed a few users to get some feedback on our online report. Here is some of the feedback we got:-Easy to access-The infographic was a big hit and shared widely-Thevideos provide more detail or provide a little more context. -Easy to share online-The graphic design was very good. -The professor also used the report and shared it with school administrators- he felt that the report was accessible to a greater range of people.-The education director at Code.org (an organization working to promote CS education) has used the report extensively.-It’s nicely interactive- you can search for parts that you are interested in. -innovative-Having the PDF is helpful for reading lots of text-- Some people printed the PDF version to read it. Suggestions we got:-This is still a model where information is being pushed out. Is there any other way to have people engage?-Make sure the font and functionality works on all computers and browsers.Another reader reached out and shared that this format would be difficult to share…he wanted a more coherent report to share with principals or school board members. He also suggested creating reports for separate audiences- one for the college community, one for teachers, one for administrators/school board members, and one for business/industry leaders.
Sarah- Questions from the audience.
Courtney-Call to actionThink of your current evaluations and how you might translate those or pieces of those reports onlineThink about your upcoming evaluations and how you might build in the time and money for an online reportExplore possible collaborators on the design side, if you don't have any internally.
Courtney-Call to actionThink of your current evaluations and how you might translate those or pieces of those reports onlineThink about your upcoming evaluations and how you might build in the time and money for an online reportExplore possible collaborators on the design side, if you don't have any internally. Ask Audience: What challenges do you see in creating online reports? What’s exciting to you about creating online reports?
