Slides from the talk on "Familiarisation: Restructuring Layouts with Visual Learning Models" given at ACM IUI 2018, in Tokyo, Japan.

1. Restructuring
Visual
Layouts with
Learning
Kashyap Todi Jussi Jokinen Kris Luyten Antti Oulasvirta
Familiarisation
Models

2. Familiariser
New/Unvisited
Page
Familiarised
Page

3. page designs can be quite diverse

4. Frequently visited pages

5. New/Unvisited
Frequently visited pages

7. increased visual search time reduces performace/usability

8. New/Unvisited
Page
Familiarised
Page
Familiariser

9. Familiariser
User History
History Logging

10. Visit Page
User History
History Logging
Familiariser

11. User History
Familiariser
History Logging

12. User History
Visit Page
Familiariser
History Logging

13. User History
Familiariser
History Logging

14. User History
Visit Page
Familiariser
History Logging

15. Familiarity Models
x
y
Time
Frequency
Page
Frequency
I. Frequency
II. Serial-Position
Curve
III. Visual Statistical
Learning
IV. Cognitive Model
History Logging
Familiarity Modelling
Familiariser

16. History Logging
Familiarity Modelling
FAMILIAR DESIGN
Familiariser

17. New/Unvisited
Page
History Logging
Familiarity Modelling
FAMILIAR
DESIGN
Familiariser

18. New/Unvisited
Page
Familiarised
Page
History Logging
Familiarity Modelling
Page Restructuring
FAMILIAR
DESIGN
Familiariser

19. Layout
Generation
Run-time
Adaptations

20. Layout Generation
DesignScape [CHI ’15]
…..
Sketchplore [DIS ’17] and many more
generate/optimise designs for the general population

21. Run-time Adaptations
Split / Moving / Visual Popout Interface [CHI ’11] Walking User Interface [TACCESS ’11]
…..
Actively adapt known interfaces as they are being used

22. Familiarisation models per-user familiarity
to generate a familiar design, and uses this
to adapt unvisited designs by restructuring

23. Familiarisation
History Logging
Familiarity Modelling
Page Restructuring

24. History Logging
Parse for
Features

25. History Logging
User History
• Order of page visits
• Parsed page layouts
• Visit frequency, duration, etc

26. History Logging
Familiarity Modelling
Page Restructuring
Familiarisation

27. Familiarity Modelling
I. Frequency-Based
II. Serial-Position Curve
III. Visual Statistical Learning
IV. Cognitive Model

28. I. FREQUENCY-BASED
Page
Frequency

29. I. FREQUENCY-BASED
Page with highest visit frequency selected
fpage = nvisits ∗ taverage

30. II. SERIAL-POSITION CURVE
Time
Frequency

31. II. SERIAL-POSITION CURVE
Page selected based on function of long-term recall
Frequency Recency Primacy
npage / ntotal 1 – rpage / rtotal 1−((ppage −1)/npages)

32. III. VISUAL-STATISTICAL LEARNIng
x
y

33. III. VISUAL-STATISTICAL LEARNIng
Page generated based on feature distributions
Logo Search Social Menu

34. IV. COGNITIVE MODEL

35. IV. COGNITIVE MODEL
Page generated based on
cognitive model of visual learning

36. Familiarity Modelling
FAMILIAR DESIGN
As a result, a familiar design is selected/generated

37. Familiarity Modelling
Logo
As a result, a familiar design is selected/generated
Login Cart
Search
Newsletter Social
Content

38. History Logging
Familiarity Modelling
Page Restructuring
Familiarisation

39. Page Restructuring
Bricolage [CHI ’11]
— Requires element size information
— Hard to deal with feature mismatches
— Content might be cropped due to mismatch

40. Page Restructuring
FAMILIAR
DESIGN
ORIGINAL
PAGE + FAMILIARISED
PAGE—>

41. Page Restructuring
FAMILIAR DESIGNORIGINAL PAGE
Logo Login Cart
Search
Newsletter Social
Content
ORIGINAL PAGE FAMILIAR DESIGN

42. Page Restructuring
FAMILIAR DESIGNORIGINAL PAGE
ORIGINAL PAGE RESTRUCTURED
FAMILIAR DESIGN
Page restructured but has overlaps!

43. Page Restructuring
FAMILIAR DESIGNORIGINAL PAGE
ORIGINAL PAGE RESTRUCTURED
FAMILIAR DESIGN
Page restructured but has overlaps!
Overlap Resolution
LP solver with constraints to:
• Eliminate overlaps
• Align elements

44. Page Restructuring
FAMILIAR DESIGNORIGINAL PAGE
ORIGINAL PAGE FAMILIARISED
FAMILIAR DESIGN

45. An Example

46. User History

47. User History
Unvisited Page

48. User History
Unvisited Page
Model Results
Frequency-Based Serial-Position Curve Visual Statistical
Learning
Cognitive Model

49. User Study
Evaluating how familiarisation impacts
visual search time during page browsing tasks

50. 1. Learning Phase
5 page designs
25 visits
Original design displayed

51. 2. Test Phase
25 page designs
100 visits
Original or familiarised design displayed

52. Visual search time reduced by 10%
(2.8 vs 2.5 seconds)
23% fewer eye-gaze fixations
(3.4 vs 2.6 fixations)
Results
=16

53. A principled approach to restructure
new layouts using visual learning models

54. Designer
Global
Consistency
Population

55. Designer
Global
Consistency
Familiariser
History Logging
Familiarity Modelling
Page Restructuring
Local
Consistency
Population
Individual

56. Familiarisation
Kashyap Todi
Jussi Jokinen
Kris Luyten
Antti Oulasvirta
Unvisited Familiarised
History Logging
Familiarity Modelling
Page Restructuring
II. Serial-Position CurveI. Frequency-Based
III. Visual Statistical Learning IV. Cognitive Model
www.kashyaptodi.com/familiarisation