Assessment Project Management in the Real World - Hour Three


Published on

2009 ACRL Conference - Workshop
Co-presenter: Joanna DiPasquale

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Implementing an assessment project is not easy! Besides the issues we discussed earlier, like communication styles, unclear roles, and uncharted territories, there are additional challenges that infrastructure, storage, local policies and procedures may impose.
  • Bring staff on board and help them understand assessment! This is assessment from the ground up, and is very useful when starting the project (or program). By understanding what assessment is, and how it is done, it is easier to recognize “what it’s all about.”
  • Storage transparency means data transparency – which means a valuable and successful assessment project. Results are actionable, and without data, it is difficult to act! Data storage, form submission, and audits allow an assessment team to establish procedures, collect information up front, and determine methodology and protocol (like IRB).
  • Technology is often seen as a panacea … but often provides its own set of challenges. Let’s go over some of the challenges that, as a project manager, you may need to solve.
  • First, you may need project management software. There are many products on the market, but few may be your ideal solution – for a variety of reasons. From the technical perspective, functionality and infrastructure are key. If you find the ideal product, but it can’t run in your environment, you’ll need to be able to articulate why that software is more useful than one that fits better into your organization. At the same time, your software choices should be based on your needs; assessing those needs helps prioritize purchases or implement solutions.
  • One of the most popular types of project management software is web-based. BaseCamp, as well as a few other tools, allow you to manage your project’s milestones, assign work to yourself and your team, and provide a space for collaboration. However, many of these solutions are “hosted,” meaning that your data resides on a server outside your organization – which may be problematic if it pulls the plug (as ours did). Trac, JIRA, Bugzilla, and other tracking devices provide an easy task assignment structure, but doesn’t fit as well into the project management rubric. It doesn’t provide a more comprehensive set of features like milestones and roadmaps that are based on non-task-oriented goals.
  • Two other possibilities for software include desktop applications that are made for project management, and web applications that can be massaged into project management tools. Microsoft Project, for example, is an excellent, robust, and mature tool that can provide multi-dimensional tracking, graphs, budgeting, and more. However, without the Microsoft Project Server application, many of the features cannot be shared without some extra work. Wikis, on the other hand, are made to be shared – they are collaboration by paradigm. However, collaboration, as we all know, is not project management. It is important to recognize that not all software fits all needs.
  • Institutional review boards exist to protect the rights of human subjects in a test. In the United States, IRBs are governed by Title 45 of the CFR (Code of Federal Regulations), Part 46, of the National Research Act of 1974. Developed to prevent subjects against abuse, they provide guidelines and practices that must be followed by an institution. In general, an IRB should exempt your assessment project, provided that it does no harm, provides no research benefit, and involves no protected classes. Highly recommend completing IRB training, if available, and building a relationship with the IRB staff. They should really teach this stuff in library school! For our purposes, IRBs can provide structure to our assessment projects. IRBs often force you to be organized about your assessment study up front.
  • IRB is one of the biggest reasons why transparency, data collection and storage, and form submission is so critical. You DO NOT want to be in violation of federal law because you decided to send out an email asking users to take a SurveyMonkey survey. Although federal law mandates many of the submission process aspects, the process itself varies from institution to institution. Be sure to ask around. Your IRB is here to help you – they want to be sure that you are following the right procedures, and should guide the process.
  • I’ve always felt like everyone glosses over the IRB process in assessment discussions. At Columbia, it’s VERY VERY involved. So, here’s what we have to provide for every study. We try to put through every “major” study. User needs projects: surveys + focus groups (digital centers) LibQual+ User satisfaction surveys (IM reference)
  • Fill out worksheet here. Q&A
  • We’ve all been there. Maybe you were told, “We need to do a focus group.” Or perhaps, “I need to do a survey about…” And you know, and we know, that the method does not fit the data you need to collect, or the ball is already rolling and you can’t stop it. Our best advice? START EARLY. Get involved in the process as soon as possible, and use the culture of assessment that you’ve cultivated to ensure that these decisions are made with great care! (Litigation story?)
  • More challenges. Yes, we have data. No, we can’t use it. It’s frustrating, and causes much heartache. After all, all of that data collection is time-consuming, and the data is necessary to make a decision! Data audits, documentation, and limiting scope are three of the best ways to combat these data challenges.
  • There really isn’t much danger in coming into a project too soon, except a bit of time spent before a clear direction is apparent. However, this is a good time to discuss actionable items – the bedrock of assessment. Coming into a project too late, however, is much more difficult. As you develop your own assessment project management, determine the optimal time to come into a project. We have started web forms to submit a project proposal, have monthly meetings, and expose “what we do” via an Assessment Forum every few months. This helps our constituents call us into a project sooner, as well as helps keep track of other departments’ needs.
  • (Funny transition of sorts… this is a big downer. ) You can: prepare for lots of headaches, start a comprehensive alcohol consumption program, run away to Vegas, take a long vacation, cry, or just take all of the books out of the library and go home. Or, using your excellent project management skills, you can turn your challenges into opportunities!
  • It is great to see group dynamics and skill sets on your assessment team, but its also critical to see the group as a set of co-workers who can help you meet the challenges of this particular assessment project. How can they help? What can they do?
  • Make sure that your methods are meeting your data. We all can’t be above average.  But also help others to understand that the data itself is a means to an end. After we get a clearer picture of trends, or problems, or directions, we need to act on them. Assessment processes bring about actionable items. Data is a tool, not a sword Methods are power
  • What is going right? What is going wrong? Thinking about these questions can help you better understand the scope and direction of the project, and your role in it. Where is assessment supposed to be in this project? Where should it be? Who are my allies inside the project? Outside? What are my parameters?
  • Assessment Project Management in the Real World - Hour Three

    1. 1. Assessment Project Management in the Real World Hour Three: Implementation, Technology, IRB, and Challenges
    2. 2. Implementation [Warning: contains challenges and technology]
    3. 3. Creating a culture of assessment <ul><li>Definitions and clarifications </li></ul><ul><li>Encourage and support staff </li></ul><ul><li>Making assessment part of every project </li></ul>
    4. 4. Creating Data Transparency <ul><li>File storage and data centers </li></ul><ul><ul><li>“ Where should we put our results?” </li></ul></ul><ul><ul><li>“ Who can view this data?” </li></ul></ul><ul><ul><li>“ Why do you need me to submit this form?” </li></ul></ul><ul><ul><li>What do we need from you, and what do you need from us? </li></ul></ul><ul><li>Data audit </li></ul><ul><ul><li>What data do we have? </li></ul></ul><ul><ul><li>Where did we put it? </li></ul></ul><ul><ul><li>What happened to ____ when ____ left the library? </li></ul></ul><ul><li>You’ll be surprised to find the data staff have stored on their workstations! </li></ul>
    5. 5. Technology [Warning: contains implementation and challenges]
    6. 6. <ul><li>What functionality do you need? </li></ul><ul><li>What infrastructure do you already have? </li></ul><ul><li>Who can help implement ? Maintain ? Train ? </li></ul>Project Management Software
    7. 7. Project Management Software <ul><li>BaseCamp, Zoho, web-based solutions Pros: easy to use, collaborative, truly project management, reasonably priced Con: hosted solution (exit strategy!) </li></ul><ul><li>Trac, JIRA, and other tracking devices Pros: easy setup, small learning curve Con: made for “bug tracking” </li></ul>
    8. 8. Project Management Software <ul><li>Microsoft Project, Other PM Applications Pros: full features, robust, fast Cons: can be expensive, hard to share data without add-ons </li></ul><ul><li>Wikis Pros: easy to set up, use, fast, collaborative, FREE! Cons: few bells, even fewer whistles </li></ul>
    9. 9. Institutional Review Boards
    10. 10. Institutional Review Board <ul><li>A short history… </li></ul><ul><li>Every IRB is different </li></ul><ul><li>Practical, reasonable goal: </li></ul><ul><li>exemption from IRB </li></ul><ul><ul><li>Your project does no harm </li></ul></ul><ul><ul><li>Your project provides no research benefit </li></ul></ul><ul><ul><li>Your project does not collect potentially damaging information </li></ul></ul><ul><ul><li>Your project will not involve minors, prisoners, or other protected classes </li></ul></ul>
    11. 11. Institutional Review Board <ul><li>All large and small universities have these boards (particularly if your university is connected to a medical school or hospital) </li></ul><ul><li>IRBs are not terrible , just a little inconvenient… </li></ul>
    12. 12. Institutional Review Board…  <ul><li>If your project leads to a publication or presentation </li></ul><ul><ul><li>Please submit a protocol </li></ul></ul><ul><ul><li>Please be sure to document everything in a transparent way </li></ul></ul><ul><ul><li>Please ask your client/colleagues if they wish to publish at the onset of the assessment </li></ul></ul>
    13. 13. More more more IRB <ul><li>Protocols often require: </li></ul><ul><li>Principal Investigator </li></ul><ul><li>Type of research </li></ul><ul><li>Exemption declaration </li></ul><ul><li>Research questions </li></ul><ul><li>Scientific Abstract </li></ul><ul><li>Lay Abstract </li></ul><ul><li>Subject parameters </li></ul><ul><ul><li>Number of participants </li></ul></ul><ul><ul><li>Age </li></ul></ul><ul><ul><li>Gender </li></ul></ul><ul><ul><li>Ethnicity </li></ul></ul><ul><li>Subject justification </li></ul><ul><li>Consent forms </li></ul><ul><li>Funding information </li></ul><ul><li>Compensation justification </li></ul><ul><li>Letter of permission from Dean, Provost, IRB Chair </li></ul><ul><li>Study Description: </li></ul><ul><ul><li>Purpose & Rationale </li></ul></ul><ul><ul><li>Design and statistical procedures </li></ul></ul><ul><ul><li>Study procedures </li></ul></ul><ul><ul><li>Drugs & devices (!) </li></ul></ul><ul><ul><li>Study Questionnaires </li></ul></ul><ul><ul><li>Study Subjects </li></ul></ul><ul><ul><li>Recruitment materials </li></ul></ul><ul><ul><li>Confidentiality </li></ul></ul><ul><ul><li>Potential Risks </li></ul></ul><ul><ul><li>Potential Benefits </li></ul></ul><ul><ul><li>Alternatives </li></ul></ul>
    14. 14. Challenges [Warning: Contains implementation and technology]
    15. 15. Activity 6 Challenges
    16. 16. Methodology Challenges <ul><li>The method won’t gather the information you need </li></ul><ul><li>The method is already set in place before you become part of the project </li></ul>
    17. 17. Data Challenges <ul><li>There isn’t enough data </li></ul><ul><li>The data was poorly gathered </li></ul><ul><ul><li>Incomplete </li></ul></ul><ul><ul><li>Unclear origins </li></ul></ul><ul><ul><li>Resulting from poorly constructed or bias tools </li></ul></ul><ul><li>The data doesn’t fit the questions being asked </li></ul><ul><li>The data is: </li></ul><ul><ul><li>Lost, missing, or on someone’s hard drive </li></ul></ul><ul><ul><li>Locked in proprietary format </li></ul></ul>
    18. 18. More Challenges… <ul><li>Coming into a project too soon </li></ul><ul><ul><li>Unsure direction </li></ul></ul><ul><ul><li>“ Unactionable” items: “That would be really interesting” </li></ul></ul><ul><li>Coming into a project too late </li></ul><ul><ul><li>Too late to gather data </li></ul></ul><ul><ul><li>Decisions often already made </li></ul></ul><ul><li>Staff are wary of assessment </li></ul>
    19. 19. 5 Minute Break 5 Minute Break
    20. 20. So, what’s an assessment project manager to do? Solutions for the real world
    21. 21. Understand the Project Team <ul><li>Communication styles </li></ul><ul><li>Roles </li></ul><ul><ul><li>Responsibility </li></ul></ul><ul><ul><li>Authority </li></ul></ul><ul><li>Skills </li></ul><ul><li>Leadership </li></ul>
    22. 22. Understand Your Questions
    23. 23. Understand the Methodologies
    24. 24. Understand the Problems
    25. 25. Thank you! Jennifer Rutner - [email_address] Joanna DiPasquale -
    26. 26. Photo Credits Flickr Creative Commons <ul><li> </li></ul><ul><li> </li></ul><ul><li> </li></ul><ul><li> </li></ul><ul><li> </li></ul><ul><li> </li></ul>