Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. ...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
2013 Users Conference  Baltimore
Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, kn...
Upcoming SlideShare
Loading in …5
×

Achieving a Better Assessment Development Process

1,370 views

Published on

Published in: Technology, Business
1 Comment
1 Like
Statistics
Notes
No Downloads
Views
Total views
1,370
On SlideShare
0
From Embeds
0
Number of Embeds
370
Actions
Shares
0
Downloads
9
Comments
1
Likes
1
Embeds 0
No embeds

No notes for slide

Achieving a Better Assessment Development Process

  1. 1. Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. 2013 Users Conference Baltimore | March 3rd – 6th Achieving a Better Assessment-Development Process Using well established architecture principles to improve the assessment building workflow.
  2. 2. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Where do you fit into this presentation?  74% stated they did not have a roadmap to show where the team was supposed to be going  69% reported their assessment team received no professional training  25% responded they did not allocate any funds specifically to improving their development process  Architecture interests:  75% interested in architecture road map  60% interested in customized reusable components for repository  59% interested in documenting their development process Results of 2012 Conference Poll Slide 2
  3. 3. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  TIBCO Software…who we are  TIBCO’s Knowledge Assessment program  What is Architecture?  The Value Proposition  How we “Architected” our Knowledge Assessment development process  Integrated Knowledge Assessment System (IKAS)  Discovering, Designing & Building Reusable Components in the Questionmark Perception Repository  Q&A & Open Discussion Topics Slide 3
  4. 4. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. An Introduction to TIBCO Software, Inc. Slide 4 TIBCO Software Inc. (NASDAQ: TIBX) is a provider of infrastructure software for companies to use on-premise or as part of cloud computing environments. www.tibco.com
  5. 5. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. TIBCO Certification Program Slide 5 TCP TCA TTP TTP – TIBCO Trained Professional  Actually attended TIBCO training  Passed on-line Knowledge Assessment  Available for most TIBCO courses TCP – TIBCO Certified Professional  Attended TIBCO training (or has equivalent knowledge & experience)  Passed formal certification exam  Available for core technologies TCA – TIBCO Certified Architect  Attended formal TIBCO training  Passed knowledge assessments and certification exams  Passed 5-day performance-based test, followed by evaluation by TIBCO Global Architects
  6. 6. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Our program supports a battery of tests for a rapidly changing high-tech environment.  The subject matter is exceptionally volatile. While we have only 30-40 assessments at any single time, the lifespan of an assessment is measured in weeks.  A single technical product enhancement can have a significant impact on an existing item bank.  The subject domain ranges from general software development concepts through computer science architecture principles.  The stakes for our participants are not life-threatening but they can be career-busting. TIBCO’s Knowledge Assessment Program Slide 6
  7. 7. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. What is Architecture? Slide 7  It is planning the development process from conception through maintenance using proven Architecture Best Practices and experienced architects.  Formal or semi-formal information gathering sessions  Applying standards, specifications, and best practices  Modeling a good infrastructure architecture (the foundation)  Modeling a good development process architecture (the building)
  8. 8. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Slide 8  The business drivers:  To improve the Knowledge Assessment Development development process  To improve the information provided to SMEs  To document our development process  To avoid creating a spaghetti architecture in the repository  This is a technical term to describe systems built without architecture planning. Why Did We Architect our Knowledge Assessment System?
  9. 9. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Slide 9 The Value Proposition  Higher item quality  Reduced time-to-market  Improved development process  Improved information provided to SMEs  Enhanced and automated item-level audit trail
  10. 10. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. How We Apply Architecture Slide 10  We Use Architecture Best Practices to: 1. Gather Information in a formal setting 2. Establish an Architecture Vision 3. Architect the Infrastructure to Support that Vision 4. Architect the development processes For more about modeling see: Appendix A: Architects and Architecture
  11. 11. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. The Architecture Vision2011-2013 Slide 11
  12. 12. 2013 Users Conference  Baltimore Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. The Architecture Vision as Implemented 2013 Slide 12 Automation Enhanced Quality Assurance
  13. 13. 2013 Users Conference  Baltimore Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.  Definition:  Reusable components are preconfigured frequently-used well-defined functional units within a repository.  It is important to formalize their development  To maximize and ensure an attractive ROI: apply standards, specifications, and conventions Defining Reusable Components in the Context of the Questionmark Repository (1/3) Slide 13
  14. 14. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Defining Reusable Components in the Context of the Questionmark Repository (2/3) Slide 14  Major Characteristics of Reusable Components: 1. Significant ROI 2. Highly stable. 3. Highly reusable 4. A crisp boundary.
  15. 15. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Defining Reusable Components in the Context of the Questionmark Repository (3/3) Slide 15  Will my reusable components be the same as yours?  Some, like templates, yes.  Others, probably.  Some will be unique to the way your business is conducted  It is the concept of actively discovering candidate components and then selectively implementing them as reusable components in a planned manner that is important.
  16. 16. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Types of Reusable Components for the Questionmark Repository Slide 16  Types of reusable components:  Complete Assessment templates (common)  Participant screen templates (common)  Outcomes (sometimes overlooked)  Demographics (can be powerful)  Dummy items (placeholders)  Mini item banks (scored or un-scored items used across multiple assessments)  Others
  17. 17. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Discovering Reusable Components in Our Questionmark Repository (1/3) Slide 17 1. First, it is not simply looking for common pieces of repository stuff and reusing them. 2. Second, it is important to select and implement only those candidate components that will ensure an attractive ROI. 3. Third, think outside the box:  Example: During the discovery phase, it became logical that a quality assessment template would be of considerable value. At the time this was unexpected; in hindsight it is now obvious.
  18. 18. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Discovering Reusable Components in Our Questionmark Repository (2/3) Slide 18  Make a list of the “big” things in your repository that have the most value to the way you do business.  Topics  Assessments  Templates  Global Resources  Global Content The above statement seems obvious, even trivial. But is it?
  19. 19. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Discovering Reusable Components in Our Questionmark Repository (3/3) Slide 19  Discovering candidate reusable components  The most important “big” things in our repository for the way we at TIBCO do business are:  Topics  Assessments  Templates  Global Resources  Global Content Now we know the most productive places to look for discovering reusable components.
  20. 20. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Overview of TIBCO’s Reusable Components (1/2) Slide 20  Assessments Templates  Intro Screens  Quality Assurance Demographics  Topics
  21. 21. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Overview of TIBCO’s Reusable Components (2/2) Slide 21  Pass and Fail Emails  Embedded in templates  Pass email has Certificate embedded in the email sent by the QM system
  22. 22. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. IKAS: Integrated Knowledge Assessment System Slide 22
  23. 23. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Top Three Benefits of IKAS[1] Slide 23 1. Knowledge Assessment Status diagram  Initially used MS Project / …overkill & too time consuming  A quick glance provides complete status of KA projects  Easy detection of bottlenecks in workflow 2. Improved information provided to SMEs  Personalized standardized professional documentation sent to SMEs for their assignments 3. Reduced time-to-market  2011: one to three new or updated assessments per quarter  2012: fourteen in one quarter [1] TIBCO’s Integrated Knowledge Assessment System
  24. 24. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Improved Information Sent to SMEs Slide 24
  25. 25. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Reduced Time-to-Market Slide 25 One example: Standardized Automated Item Management  When SMEs began receiving standardized documentation , item quality skyrocketed. Much of the quality improvement was directly attributable to the use of MS Word tables.  QA improvement = fewer rejects, less rework = reduced time to market
  26. 26. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Knowledge Assessment Status Diagram Slide 26  A GREAT project management tool
  27. 27. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Lessons Learned Slide 27 1. The MS Visio Knowledge Assessment workflow diagram provided the highest ROI. Much of which was due to detecting and resolving bottlenecks. 2. The covert enforcement of item-writing specifications implemented as MS Word tables provided the greater portion of item-quality improvements. 3. I implemented 30-50% more item-management macros than necessary to support the matured workflow.
  28. 28. 2013 Users Conference  Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Thank You! Slide 28

×