Applying the principles of item and test analysis
 

Like this? Share it with your network

Share

Applying the principles of item and test analysis

on

  • 2,804 views

 

Statistics

Views

Total Views
2,804
Views on SlideShare
1,806
Embed Views
998

Actions

Likes
1
Downloads
45
Comments
0

7 Embeds 998

http://blog.questionmark.com 698
http://www.scoop.it 285
http://blogs.questionmark.com 6
https://global.slu.edu 4
http://translate.googleusercontent.com 2
https://blackboard.slu.edu 2
https://www.questionmark.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Applying the principles of item and test analysis Presentation Transcript

  • 1. Applying the Principles of Item and Test Analysis 2012 Users Conference New Orleans March 20 - 23Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmarkis a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 2. Session objectives  Define various Classical Test Theory item and test statistics  Identify poorly performing items and tests using item and test analysis reports  Use item analysis reports to guide item revisions to improve your assessments 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 2 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 3.  A global network of professional services firms.  150,000 staff members across the network.  35,000 are in the USA.  The development of our people is a top priority.  PwC is in Training Magazine’s Hall of Fame and is the only company to have been awarded #1 in the Top 125 for 3 years in a row.  We have a highly mobile and virtual workforce. 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 3 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 4. Introduction to Test Theory 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 4 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 5. What is test theory?  Mathematical concepts to help answer your questions about assessment quality:  Does the test measure one thing or multiple things?  How should the test be scored?  How precisely does the assessment measure the knowledge?  Are any items influenced by factors other than what you are trying to measure (a.k.a. “bias” or "irrelevant variance")?  Can we use alternative, equivalent items to test the same thing? 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 5 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 6. Assessments are measurement scales!  Measurement is the assignment of numbers to an attribute according to a rule.  Most physical measurement scales we take for granted:  Temperature  Weight  Volume 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 6 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 7. Assessments have a limit  Assessment scales have a limited range. They have what is called a Floor (0%) & Ceiling (100%)No Knowledge Expert KnowledgeNew Hire 0% 100% Experienced HireNovice Master Test questions measure only part of the possible range of knowledge. 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 7 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 8. Assessments can use different scales  Different metrics can be used for the same thing. We assign the meaning to the numbers. Think of the differences between Celsius and Fahrenheit. Number Correct Square Square Root Log-odds Scale Passed all items 10 100 3.16 ? Passed 9 items 9 81 3 2.18 Passed 8 items 8 64 2.83 1.39 Passed 7 items 7 49 2.65 0.85 Passed 6 items 6 36 2.45 0.41 Passed 5 items 5 25 2.24 0 Passed 4 items 4 16 2 -0.41 Passed 3 items 3 9 1.73 -0.85 Passed 2 items 2 4 1.14 -1.39 Passed 1 items 1 1 1 -2.18 Failed all items 0 0 0 ? 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 8 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 9. Is your assessment a norm-referenced scale OR a criterion-referenced scale?  A norm-referenced assessment Normal distribution of scores measures people against a defined population.  How do you measure up against other people? 0% 100%  A Criterion-referenced Negative distribution of scores assessment measures people against a defined domain of knowledge.  Have you mastered the material in 0% 100% a domain of knowledge? 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 9 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 10. Classical Test Theory is…  Based on the assumption that all the test items measure the same concept.  Concerned mainly with the overall test score.  Used to improve the quality of the test score. True score = observed test score + error 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 10 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 11. Reliability & Validity Reliability Are test scores consistent across factors which should not influence the score (time, versions, environment, etc)? Are test items all measuring the same thing? Validity Are you measuring what you intended to measure? Is the test score being used appropriately? We will be addressing mainly reliability 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 11 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 12. Reliability Measures  In general:  There are many of them  Usually take the form of a correlation with a range from 0 to 1.  Closer to 1 is better  Below .5 is unacceptable  Some measure consistency across different factors  Test-Retest reliability  Alternative forms reliability  Others measure internal consistency (Are you measuring a single topic well?)  Split-half Reliability = Split the test in half and correlate the 2 scores  Cronbachs alpha (α) = All possible split half combinations  Most appropriate for Norm-referenced tests 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 12 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 13. Test Analysis 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 13 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 14. Test Analysis: First examine your sample  Do you have enough people in your sample to have confidence in the statistics?  You can get by with 25-30 people for pilot testing in low- moderate stakes assessments.  Ideally you want 100 people for a solid analysis.  Do you have the expected distribution of scores for your testing program?  Norm-referenced = normal distribution  Criterion-referenced = negatively skewed distribution 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 14 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 15. Test Analysis: Score distribution  Several statistics will help you understand your distribution:  Mean – The mathematical average of all scores. The mean can be misleading if your distribution is skewed.  Median – The middle value. You should use the median when you have a skewed distribution.  Mode – The most common value in the distribution.  Skew – Tells you how evenly scores are distributed around the mean.  Negative skew – more values are higher than the mean  Positive skew – more values are lower than the mean  Zero skew – score are evenly distributed around the mean  Kurtosis – A measure of the “peakedness” of the distribution 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 15 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 16. Test Analysis: Histogram If you don’t like all those numbers just look at the histogram! 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 16 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 17. Test Analysis: The Numbers These statistics correspond to the previous slide histogram. Statistic Value Number of Examinees 193 Mean 21 Median 22 Mode 22 Skew -0.916 Kurtosis 2.246 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 17 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 18. Test Analysis: Overall Difficulty  Next check your floor and ceiling to ensure the test is targeted to the population’s ability.  Is your test too hard?  What was the minimum score?  For a multiple choice test how many people are scoring around 25%?  Is your test too easy?  What was the maximum score?  How many people are scoring 95% or higher? 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 18 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 19. Test Analysis: Standard Error of Measurement (SEM)  Shows the average amount of measurement error around a test score.  Use to create confidence intervals around a test score.  The true score is likely to fall within the range.  SEM is best used with norm-referenced tests. SEM = 2 68% confident that the true score is between 12 and 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Passing score = 14 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 19 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 20. Test Analysis: Internal Consistency Cronbach’s Alpha (α) Internal consistency  Cronbach’s Alpha is the ≥ .9 Excellent most popular reliability .9 > α ≥ .8 Good measure. .8 > α ≥ .7 Acceptable  A very high reliability may indicate you have .7 > α ≥ .6 Questionable redundant items. .6 > α ≥ .5 Poor  Topic or sub-scores < .5 Unacceptable should also be reliable. 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 20 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 21. Improving Reliability  Have a consistent “controlled” testing environment & provide clear instructions.  Have a large group of examinees with a broad range of ability.  If everyone is of equal ability they will score relatively the same on the test and thus reliability index will be low.  Use objectively scored test items (multiple choice items).  Items like an essay which requires scoring from the teacher tends to have lower reliability due to additional error introduced by teacher judgment. 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 21 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 22. Improving Reliability  Increase the test length  In general more items means less error for the test (Google “Spearman-Brown formula”).  Use only “quality” items  Develop items using best practices to reduce “lucky guessing”.  Delete or edit items which are too easy, too difficult, or otherwise do not help to differentiate those with knowledge.  See Developing and Validating Multiple-Choice Test-Items by Thomas M. Haladyna  http://books.google.com/books?id=kna46TApW14C&lpg=PP1&d q=Multiple%20choice&pg=PP1#v=twopage&q&f=false 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 22 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 23. Does this look like a “good” test? Statistic Value Number of examinees 90 Minimum achieved score 48% Maximum achieved score 95% Test reliability (Cronbachs Alpha) 0.796 Mean 80.72% Median 81.25% Mode 81.25% Standard deviation 9.53% Standard error of measurement 4.31% Skew -0.886 Kurtosis 0.939 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 23 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 24. Does this look like a “bad” test? Statistic Value This test used an item Number of examinees 193 banking approach with random selection of Minimum achieved score 10% items. Maximum achieved score 31% Test reliability (Cronbachs Alpha) -0.704 Beware – only use basic Mean 24.39% test analysis reports Median 25.29% when all examinees get exactly the same items. Mode 25.29% Standard deviation 3.18% Standard error of measurement 4.16% Skew -0.916 Kurtosis 2.246 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 24 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 25. Item Analysis 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 25 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 26. Anatomy of an item  Stem – The actual question being asked  Options – all choices  Key – Correct answer  Distractors – The wrong answer choices  Feedback – Explanatory remediation as to why that particular answer choice is incorrect. Stem What Code section deals with the taxable gain to a corporation when a corporation distributes property to a shareholder? A. 301 Incorrect. This section deals with the character of the amount received by a shareholder from a corporation. Key B. 311 Correct. C. 312 Incorrect. This section deals with the effect on earnings and profitsDistractors of a transaction. D. 334 Incorrect. This section deals with the basis of property received in a liquidation.Feedback 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 27. Item statistics: Definitions  Number of examinees: The number of examinees answering the question.  Aim to get at least 30 responses however 100 is ideal  P-Value: The percentage of examinees who chose the correct answer.  For norm-referenced tests youll want a wide range of P-values  For Criterion-referenced tests youll want more around your passing score  Distractor Percentage: The percentage of examinees who chose a wrong answer  If zero - 5% consider replacing with a more attractive distractor  All wrong answers should be common mistakes a novice would make  Item-Total Correlation OR Discrimination: A correlation between picking an option and the total score on the test. Theory is if you get the question right you should be scoring higher on the test than those who got the question wrong.  Correct answer should have strong positive correlation with the total test score (if that question is measuring the same thing as the other questions on the test).  Item-total correlation is influenced by P-value so expect lower values on very hard or very easy items. A P-value of 0 or 1 will always give you a zero correlation. 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 27 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 28. Item Statistics: Some Guidelines  Items too hard or too  Items tricky or easy? confusing? • P-Value • Discrimination  < 50% Very Difficult  < .20 Low  51% - 64% Difficult  .20 - .30 Moderate  65% - 75% Good  76% - 94% Easy  > .30 Good  95% - 100% Very Easy 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 28 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 29. Example #1 - What would you do with this item? Stat A B C D Total # Examinees 0 0 0 21 21 P-Value/Distractor % 0% 0% 0% 100% 100% Item-total Correlation 0.00 0.00 0.00 0.00 Correct answer 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 29 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 30. Example #1 – Too EasyWhich of the following most accurately presents the reconciliationof Partners Capital found in Schedule M 2 of Form 1065? Original RevisedA Beginning Capital Accounts minus BOY Capital plus Guaranteed Payments Distributions equals ending capital accounts. plus/minus CY Net Income minus Distributions equals EOY Capital.B Beginning Equity minus Distributions plus BOY Capital plus Distributions plus/minus stock buy-backs equals ending equity. CY Net Income equals EOY Capital.C Beginning Retained Earnings plus current BOY Capital minus Capital Contributions year income minus dividends equals ending plus/minus CY Income equals EOY Capital. retained earnings.D Beginning Capital plus current year net BOY Capital plus current year net income income plus capital contributions minus plus Capital Contributions minus distributions equals ending capital. Distributions equals EOY Capital. 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark Slide 30 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 31. Example #1 - Revised Stat A B C D Total # Examinees 6 5 7 85 103 P-Value/Distractor % 6% 5% 7% 83% 100% Item-total Correlation -0.32 -0.27 -0.17 0.46 Correct answer 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 31 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 32. Example # 2 - What could be wrong with this item? Stat A B C D Total # Examinees 16 1 5 1 23 P-Value/Distractor % 70% 4% 22% 4% 100% Item-total Correlation -0.04 0.00 -0.02 0.14 Correct answer 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 32 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 33. Example #2 – Confusing the LearnerWhich of the following is the best definition of a Book Tax Return? Original RevisedA A business tax return that reflects taxable A return that reflects taxable income income according to financial statement according to financial statement rules rules instead of tax rules. instead of tax rules.B A business return prepared using federal tax A return prepared using federal tax rules in rules in calculating taxable income. calculating taxable income.C A business return prepared using tax rules to A return prepared using tax rules to calculate taxable income but reflects a book calculate taxable income and a GAAP basis balance sheet. balance sheet.D A business return prepared using A return prepared using international international accounting standards. accounting standards. 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark Slide 33 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 34. Example # 2 - Revised Stat A B C D Total # Examinees 60 10 34 1 105 P-Value/Distractor % 57% 10% 32% 1% 100% Item-total Correlation 0.39 -0.27 -0.24 0.00 Correct answer 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 34 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 35. Example #3 - What could be wrong with this item? Stat A B C D Total # Examinees 18 2 3 0 23 P-Value/Distractor % 78% 9% 13% 0% 100% Item-total Correlation 0.03 0.13 -0.14 0.00 Correct answer 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 35 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 36. Example #3 – Bad Item Format Dalton Enterprises, Inc sold Dalton Enterprises, Inc sold investment assets this year. investment assets this year. Will Will Form 4797 be required? Form 4797 or schedule D be required? Original RevisedA No, because the assets were capital assets Because the assets were capital assets, and reported on schedule D Schedule D will be filed.B No, because the assets qualified as If the assets were sold at a loss and used in "involuntary conversion" instead of a sale the business Schedule D will be filedC Yes, because all asset sales must be If the assets were sold at a gain Form 4797 recorded on this form will be filed.D Yes, because the sale was a tax free sale Since investment assets are tax exempt Form 4797 is filed 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark Slide 36 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 37. Example #3 - Revised Stat A B C D Total # Examinees 86 5 4 9 104 P-Value/Distractor % 83% 5% 4% 9% 100% Item-total Correlation 0.42 -0.15 -0.38 -0.19 Correct answer 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 37 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 38. Additional Resources  National Council on Measurement in Education – free articles: http://ncme.org/publications/items/  Understanding Reliability: http://ncme.org/linkservid/65F3B451- 1320-5CAE-6E5A1C4257CFDA23/showMeta/0/  Standard error of measurement: http://ncme.org/linkservid/6606715E-1320-5CAE- 6E9DDC581EE47F88/showMeta/0/  Practical Assessment, Research & Evaluation: http://pareonline.net/  Writing Multiple Choice Items: http://pareonline.net/getvn.asp?v=4&n=9  Basic Item Analysis: http://pareonline.net/getvn.asp?v=4&n=10 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 38 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
  • 39. Contact Information  Sean.w.farrell@us.pwc.com  973-236-4327  You can find me on LinkedIn 2012 Users Conference  New Orleans Copyright © 1995-2012 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. QuestionmarkSlide 39 is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.