The Behavior Assessment Framework describes an systematic approach to evaluate mobile/pervasive services based on collected logging data during a field trial

1. The Behavior Assessment Model for the
analysis of pervasive services
Bernhard Klein, Ivan Pretel, Ulf-Dietrich Reips, Ana B.
Lago and Diego Lopez-de-Ipiña
World Conference on Information Systems and Technologies 2013
Session 15: Human-Computer Interaction
29. March, 2013
Mobile
User-
Generated
Geo
Services
WorldCist 2013 B. Klein 1/17

2. Outline
1. Problem Description
2. Related Work
3. Data Collection and Aggregation
4. System Architecture
5. Behavior Assessement Model
6. Balanced Scoreboard Approach
7. Conclusions
WorldCist 2013 B. Klein 2/17

3. Background: FP7 MUGGES
► Beyond User Generated Contents
► Mobile Micro-services
 Location-based User-generated Contents and Services
 Mobile Super Prosumer Concept
Consumer of contents & services
PUBLISH
PROVIDE
CREATE
Producer of contents
FIND
Provider of services from the mobile
USE
BENEFIT Producer of new mobile services
WorldCist 2013 B. Klein 3/17

4. Background: FP7 MUGGES
Service
Creation/
Provision
Service
Discovery/
Conumption
WorldCist 2013 B. Klein 4/17

5. Challenge: Evaluation of MUGGES?
User Perceptions:
 Technical Acceptance Model
• Usefulness
• Usability
+ Mobility aspects
+ Sharing costs
Evaluation
Service Usage:
 Camera Observations
 Usage Rate ~ Importance
Indicator
Mobile usage ~ Situation context!
WorldCist 2013 B. Klein 5/17

6. Related Work
Lab-based approaches Field based approaches
+ controlled environment + situation context
usually more accurrate! (more realistic)
Service interaction Feedback based
• www.Usertesting.com • MyExperience,
• Moroe Observer, Context capture Frohelich et al 2007
Cassadey and • Recon, Jenson 2009 • SocioXensor
Johnson, 2002 • ContextPhone, ter Hofte et al. 2006
Raento et al. 2005
WorldCist 2013 B. Klein 6/17

7. The Behavior Assessment Framework
Structural
Model
Behaviour
Measure-
Process Assessment ment
Methodology
System
Framework
Data
Collection
Approach
WorldCist 2013 B. Klein 7/17

8. Data Collection & Processing
WorldCist 2013 B. Klein 8/17

9. Measurement Infrastructure
WorldCist 2013 B. Klein 9/17

10. Dimensions of BAM
WorldCist 2013 B. Klein 10/17

11. Process Model
WorldCist 2013 B. Klein 11/17

12. Dimension  Indicator  Measures
Dimension X
• Indicator 1 – Measurement 1
• Indicator 2 – Measurement 2 Logging raw data:
𝛼 ∗ 𝐹1 ( )
Domain-Translation function Fx()
𝛽 ∗ 𝐹2 ( )
Rating domain
WorldCist 2013 B. Klein 12/17

13. Visualisation of Behavior Assessment
WorldCist 2013 B. Klein 13/17

14. Logging data + analysis
► Logging events
- creation, editing, publishing, consumption, commenting
+ different service specific gui events
► Correlated context data
- Status, timestamp, location, user, subscriptions
WorldCist 2013 B. Klein 14/17

15. Balanced Scoreboard Approach
WorldCist 2013 B. Klein 15/17

16. Conclusions
 Behaviour Assessment Model
► Model dimensions
 Service actuation, service interaction and service networking
effects
 Planned behaviour vs. Spontaneous execution
► Evaluation Process
 Criteria development, measurement (mapping and weightening),
evaluation
 Visualization based on Balanced Scoreboard
► Results
 1st and long-term behaviour patterns
 Technological experience
 Product development stages
WorldCist 2013 B. Klein 16/17

17. Contact: Bernhard Klein,
Email: bernhard.klein@deusto.es
Deusto Intitute of Technology,
University of Deusto,
World Conference on Information Systems and Technologies 2013
Avda. Universidades, 24 | 48007 Bilbao | Spain
Session 15: Human-Computer Interaction
29. March, 2013
WorldCist 2013 B. Klein 17/17