1. An Empirical Study on End-users Productivity Using
Model-based Spreadsheets
Laura Beckwith J´come Cunha
a Jo˜o P. Fernandes
a Jo˜o Saraiva
a
HCIResearcher, Denmark, beckwith@hciResearcher.com
Universidade do Minho, Portugal, {jacome,jpaulo,jas}@di.uminho.pt
Universidade do Porto, Portugal, jpaulo@fe.up.pt
EuSpRIG 2011
July 14 – 15
J´come Cunha (UMinho)
a EuSpRIG 2011 1 / 19

2. Agenda
1 Introduction
2 Model-based Spreadsheets
Inferring Models
Refactored Spreadsheet
Visual Spreadsheet
3 Study Design
4 Results
Effectiveness
Efficiency
5 Conclusions
J´come Cunha (UMinho)
a EuSpRIG 2011 2 / 19

3. Introduction
Much research has been done to prevent spreadsheet end-user errors
Model-Driven Engineering (MDE) is a promising approach
We have created a set of different model-based spreadsheets
J´come Cunha (UMinho)
a EuSpRIG 2011 3 / 19

4. Introduction
Much research has been done to prevent spreadsheet end-user errors
Model-Driven Engineering (MDE) is a promising approach
We have created a set of different model-based spreadsheets
Need to evaluate their impact
J´come Cunha (UMinho)
a EuSpRIG 2011 3 / 19

5. Original Spreadsheet - original
A project management system
Tabular spreadsheet (database like spreadsheet)
J´come Cunha (UMinho)
a EuSpRIG 2011 4 / 19

6. Functional Dependencies
A B means that the values in column A uniquely determine values
in column B
If two values two rows in column A are equal, then the corresponding
values in B are also equal
For example, suppose A = {A1 , A2 } and B = {B1 , B2 }:
A1 A2 B1 B2 A1 A2 B1 B2
a1 a2 b1 b2 a1 a2 b1 b2
a1 a3 b1 b1 a1 a3 b1 b1
a1 a2 b1 b2 a1 a2 b2 b2
(a) Table inducing A B. (b) Table not inducing A B.
J´come Cunha (UMinho)
a EuSpRIG 2011 5 / 19

7. Relational Model
From the spreadsheet data (also from formulas) we can infer
functional dependencies
From these dependencies, we can compute a relational database
schema
Reasoning
about FDs
Original SS
A⇀B
Infer Infer RDB RDB
CD⇀E
FDs Schema Schema
...
J´come Cunha (UMinho)
a EuSpRIG 2011 6 / 19

8. Refactored Spreadsheet - refactored
We use the relational model to generate a new refactored spreadsheet
(respecting the relational model)
Each entity separated in a different table
Combo boxes used to list allowed values (from other tables)
J´come Cunha (UMinho)
a EuSpRIG 2011 7 / 19

9. Visual Spreadsheet - visual
From functional dependencies we can generate edit assistance
A B: A columns become green combo boxes and B red
Similar to original
Combo boxes help user introducing data
J´come Cunha (UMinho)
a EuSpRIG 2011 8 / 19

10. The Entire Picture
Reasoning Visual SS
about FDs
Original SS Generate
Visual SS
A⇀B
Infer
CD⇀E
FDs
... Infer RDB
Schema
RDB
Schema Refactored SS
Generate
Refactored SS
Goal: compare efficiency and effectiveness of original, refactored and visual
J´come Cunha (UMinho)
a EuSpRIG 2011 9 / 19

11. Research Questions
RQ1 Do end users introduce fewer errors when they use one of the
model-based spreadsheets versus the original unmodified one?
RQ2 Are end users more efficient using the model-based spreadsheets?
RQ3 Do particular models lead to fewer errors in particular tasks?
J´come Cunha (UMinho)
a EuSpRIG 2011 10 / 19

12. Study Design
Background questionnaire, followed by a list of tasks and a post
questionnaire
No initial tutorial was given to the participants
38 participants: 25 females and 11 males, all students
Tasks for inserting and editing data and computing statistics
3 different spreadsheet applications (projects, dishes and
properties)
3 different models (original, refactored, visual)
Each participant received 3 spreadsheets, each of a different
application/model
J´come Cunha (UMinho)
a EuSpRIG 2011 11 / 19

13. Global Effectiveness
original refactored visual
dishes 86% 76% 78%
projects 73% 68% 78%
properties 75% 64% 62%
Globally, models do not seem very competitive
Further analyses are necessary
J´come Cunha (UMinho)
a EuSpRIG 2011 12 / 19

14. Effectiveness: Inserting Data
original refactored visual
dishes 91% 90% 81%
projects 76% 60% 75%
properties 86% 67% 68%
Models always lose
But very similar to original
J´come Cunha (UMinho)
a EuSpRIG 2011 13 / 19

15. Effectiveness: Editing Data
original refactored visual
dishes 91% 82% 82%
projects 54% 62% 50%
properties 65% 98% 48%
refactored in general was better
Because there is less data to edit
J´come Cunha (UMinho)
a EuSpRIG 2011 14 / 19

16. Effectiveness: Computing Statistics
original refactored visual
dishes 52% 37% 57%
projects 19% 76% 13%
properties 44% 57% 51%
Again, refactored was better
And again, less data do analyze
J´come Cunha (UMinho)
a EuSpRIG 2011 15 / 19

17. Efficiency
original refactored visual
dishes 35 32 28
projects 39 40 41
properties 37 36 40
Models are competitive
In further analyzes we discovered that users took more time understanding model,
specially refactored
J´come Cunha (UMinho)
a EuSpRIG 2011 16 / 19

18. Answers to Research Questions
RQ1 Do end users introduce fewer errors when they use one of the
model-based spreadsheets versus the original unmodified one?
We believe there is potential for model-based spreadsheets
But a deep knowledge seems to be necessary to be effective
RQ2 Are end users more efficient using model-based spreadsheets?
Model-based spreadsheet can help in some cases
We believe better knowledge about models could improve this result
RQ3 Do particular models lead to fewer errors in particular tasks?
For inserting data, this was not the case
The models helped editing data and in computing statistics
J´come Cunha (UMinho)
a EuSpRIG 2011 17 / 19

19. Conclusions
We did not give a tutorial to participants
We believe better results could be achieve with initial training
Nevertheless, our model-based spreadsheet are competitive
Better for editing and statistics
We plan to run a new study similar to this, but with an initial tutorial
J´come Cunha (UMinho)
a EuSpRIG 2011 18 / 19

20. Thank you for your attention!
J´come Cunha (UMinho)
a EuSpRIG 2011 19 / 19