Chi 2019 SmartManikin: Virtual Humans with Agency for Design Tools

1. Bokyung Lee, Taeil Jin, Sung-Hee Lee, Daniel Saakes
Department of Industrial Design, KAIST, Korea
Virtual Humans with Agency for Design Tools
SmartManikin:

2. Comfortability and usability of the product
depend on how the product fit to our bodies.
01. Introduction

3. It is hard to estimate comfort or use-scenarios
within the design tools.
01. Introduction

4. Human Bodies in Design Tools
02. Related Work
Designing on the human body
Mirror Mirror (CHI’16)
Tactum (CHI’15)
Human Bodies as input
BodyMeter (nordiCHI’16)
Tactum (CHI’15)BodyAvatar (UIST’13)

5. DressUp (TEI ’12)
Riding Posture (Applied Ergonomics ’17)
Ergonomic Simulations
SketchChair (TEI’ 11)
Tactum (CHI’15)Jack
Virtual Humans in Design Tools
Designing on virtual manikins
02. Related Work

6. Manipulate it directly Show it what to do Tell it what to do
How to Interact with Virtual Humans?(Badler, 1995)
02. Related Work

7. Manipulate it directly Show it what to do Tell it what to do
How to Interact with Virtual Humans?(Badler, 1995)
02. Related Work

8. Manipulate it directly Show it what to do Tell it what to do
How to Interact with Virtual Humans?(Badler, 1995)
02. Related Work

9. Make it respond
Virtual Human that responds to design changes.
Manipulate it directly Show it what to do Tell it what to do
02. Related Work

10. 7
Desk Configuration Tool

11. Formative
Study
Pose Capture
Study
SmartManikin
Design / Develop
How should the SmartManikin
agency should be designed?
7
Expert
Feedback

12. Fly-on-the-wall at participant’s home.
4 participants / 3 hours / observed working poses
03. Formative Study

13. Four Categories of Poses
comfortable pose temporary pose possible pose
(reachable pose)
healthy pose
100°
03. Formative Study

14. Design Strategies for SmartManikin Agency
comfortable pose temporary pose possible pose
(reachable pose)
healthy pose
100°
• Simulate multiple actions to inform real-life uses.
• Respond to real-time design contexts.
• Generate poses based on comfort, except reaching.
• Simulate comfort and ergonomic separately.
03. Formative Study

15. Design Strategies for SmartManikin Agency
comfortable pose temporary pose possible pose
(reachable pose)
healthy pose
100°
• Simulate multiple actions to inform real-life uses.
• Respond to real-time design contexts.
• Generate poses based on comfort, except reaching.
• Simulate comfort and ergonomic separately.
03. Formative Study

16. Design Strategies for SmartManikin Agency
raise a tabletop
raise a shelf
comfortable
pose
possible
pose
comfortable
pose
possible
pose
vs.(typing)
(reaching) vs.
• Simulate multiple actions to inform real-life uses.
• Respond to real-time design contexts.
• Generate poses based on comfort, except reaching.
• Simulate comfort and ergonomic separately.
03. Formative Study

17. Design Strategies for SmartManikin Agency
raise a tabletop
comfortable
pose
possible
pose
vs.(typing)
• Simulate multiple actions to inform real-life uses.
• Respond to real-time design contexts.
• Generate poses based on comfort, except reaching.
• Simulate comfort and ergonomic separately.
03. Formative Study

18. Formative
Study
Pose Capture
Study
Expert
Feedback
How to generate comfortable poses
at each desk configurations?
7
SmartManikin
Design / Develop

19. Pose Capturing Setup
Kinect A Kinect B
Two Microsoft Kinects were installed to capture 3D human poses.
Height
Changing
Desks
04. Pose Capture Study

20. Conditions for Comfortable Pose Capturing
Sitting TypeAction Type Item Heights Item Positions
1. Typing with laptop
• Two Hands motion
2. Watching Monitor
• Hands free
3. Using Mouse (surfing)
• One hand action
4. Reading paper
• No computer
04. Pose Capture Study

21. Conditions for Comfortable Pose Capturing
Sitting TypeAction Type Item Heights Item Positions
04. Pose Capture Study

22. Conditions for Comfortable Pose Capturing
Sitting TypeAction Type Item Heights Item Positions
04. Pose Capture Study

23. Conditions for Comfortable Pose Capturing
Sitting TypeAction Type Item Heights Item Positions
04. Pose Capture Study

24. 3D Human Pose Capturing with Skeletons
All frames of the skeletal pose were saved in bvh file.
04. Pose Capture Study

25. All frames of the skeletal pose were saved in bvh file.
3D Human Pose Capturing with Skeletons
04. Pose Capture Study

26. Pose Feature Regression
offline process
online process
pose of
Smart
Manikin
predicted features
Pose
Generator
desk
configurations
user
information
3D Pose
Dataset
Training
Regression Functions
Identifying
Key Features
Regression
Functions
base
position
foot
position
trunk
angle
(goes higher)Body backwards
Participant 6 Participant 4
base
position
04. Pose Capture Study

27. Pose Feature Regression
offline process
online process
pose of
Smart
Manikin
predicted features
Pose
Generator
desk
configurations
user
information
3D Pose
Dataset
Training
Regression Functions
Identifying
Key Features
Regression
Functions
base
position
foot
position
trunk
angle
foot
position
Participant 1
04. Pose Capture Study

28. Pose Feature Regression
offline process
online process
pose of
Smart
Manikin
predicted features
Pose
Generator
desk
configurations
user
information
3D Pose
Dataset
Training
Regression Functions
Identifying
Key Features
Regression
Functions
base
position
foot
position
trunk
angle
trunk
angle
Participant 9
04. Pose Capture Study

29. Pose Training
04. Pose Capture Study
Interacting with
distant object
Interacting with
near object
Sitting Type
Selected
Sitting
Type
Trunk
Angles
Base
Position
Foot
Position
Regressors
Trunk
Angles
Base
Position
Foot
Position
Estimated Feature

30. 04. Pose Capture Study
Input : Desk Configuration(desk height & human height)
Output: each identified features.
• Trunk Angle
• Base Position
• Foot Position
, }
𝑝𝑏𝑎𝑠𝑒 ∈ ℜ3
𝑝 𝑓𝑒𝑒𝑡 ∈ ℜ3
× ℜ3 We should change figure
(from data scene)
𝑇𝑐h𝑎𝑖𝑟
𝑇𝑏𝑎𝑠𝑒
Train data for regressor
Pose Training

31. 04. Pose Capture Study
• Regressor Model
Fully connected multilayer perceptron
• Activation Function
Sigmoid functions
• Optimization Method
batch L-BFS optimization method
• Train & Running Time
30 minutes to train all 12 regressors
(3 regressors for 4 sitting types)
Average running time is extremely fast
(around 0.2 milliseconds)
• Regression Error
Average Size of Training Datasets: 26957
Average Size of Test Datasets: 11552
Average of Regressors: 2.1%
Regress
or output
Regressor
Input
1 Hidden layer of five cells
Pose Training

32. Formative
Study
Pose Capture
Study
Expert
Feedback
SmartManikin in Design Tools
7
SmartManikin
Design / Develop

33. User Interface for SmartManikin
05. SmartManikin

34. User Interface for SmartManikin
items
item
repository
configurable
desk
action
commandsergonomic
guidelines
Using Backrest
SmartManikin
05. SmartManikin

35. 3D Desk Configuration Tool with real-time Use-Simulation with Virtual Humans
User Interface for SmartManikin
05. SmartManikin

36. [1] Poses Respond to Desk Configurations.
Agency Design of SmartManikin
lower
body
pose
upper
body
pose
FK
IK
furniture
configuration
mannequin
height
IK : Inverse Kinematics
FK: Forward Kinematics
base
position
foot
position
trunk
angle
regressors
05. SmartManikin

37. [1] Poses Respond to Desk Configurations.
Agency Design of SmartManikin
05. SmartManikin

38. [1] Poses Respond to Desk Configurations.
Agency Design of SmartManikin
05. SmartManikin

39. [2] Poses Respond to Action Input & Item Arrangements
Agency Design of SmartManikin
lower
body
pose
upper
body
pose
FK
IK
furniture
configuration
mannequin
height
IK : Inverse Kinematics
FK: Forward Kinematics
base
position
foot
position
trunk
angle
regressorshead
poses
arm
poses
action
command
object
position
IK
IK
lower
body
pose
upper
body
pose
FK
IK
base
position
foot
position
trunk
angle
furniture
configuration
mannequin
height
regressors
05. SmartManikin

40. [2] Poses Respond to Action Input & Item Arrangements
Agency Design of SmartManikin
laptop typing keyboard typing surfing reaching
05. SmartManikin

41. [3] Poses Respond to Sitting Preference / Habits
Agency Design of SmartManikin
a bstanding straight leaning
c dusing backrest not using backrest
05. SmartManikin

42. [4] Responsive Ergonomic Guidelines
Agency Design of SmartManikin
05. SmartManikin

43. Formative
Study
Pose Capture
Study
Expert
Feedback
7
SmartManikin
Design / Develop
How professional industrial designers
think about SmartManikin?

44. FGI with Professional Industrial Designers
System Introduction / Demo / FGI (reflecting their previous design process)
06. Expert Feedback

45. Furniture Designer
mass-produced furniture
10-year experience
Furniture Designer
custom-made furniture
12-year experience
Furniture Designer
DIY furniture maker
9-year experience
Fashion Item Designer
custom-made items
9-year experience
FGI with Professional Industrial Designers
4 Furniture Designer + 1 Fashion Item Designer
06. Expert Feedback

46. Expert Feedback
• Emotionally attached to the virtual manikin.
• Appreciate pose simulation generated from the field observation.
• Valuable for early stage of design process.
• Valuable for real-time quick evaluation process.
• Limitation of third-person perspective.
“It is cute! It’s like my virtual assistant!”
06. Expert Feedback

47. Expert Feedback
• Emotionally attached to the virtual manikin.
• Appreciate pose simulation generated from the field observation.
• Valuable for early stage of design process.
• Valuable for real-time quick evaluation process.
• Limitation of third-person perspective.
“It is cute! It’s like my virtual assistant!”
06. Expert Feedback

48. Expert Feedback
• Emotionally attached to the virtual manikin.
• Appreciate pose simulation generated from the field observation.
• Valuable for early stage of design process.
• Valuable for real-time quick evaluation process.
• Limitation of third-person perspective.
“It is cute! It’s like my virtual assistant!”
06. Expert Feedback

49. Expert Feedback
• Emotionally attached to the virtual manikin.
• Appreciate pose simulation generated from the field observation.
• Valuable for early stage of design process.
• Valuable for real-time quick evaluation process.
• Limitation of third-person perspective.
“It is cute! It’s like my virtual assistant!”
06. Expert Feedback

50. Expert Feedback
• Emotionally attached to the virtual manikin.
• Appreciate pose simulation generated from the field observation.
• Valuable for early stage of design process.
• Valuable for real-time quick evaluation process.
• Limitation of third-person perspective.
06. Expert Feedback

51. Expert Feedback
• Emotionally attached to the virtual manikin.
• Appreciate pose simulation generated from the field observation.
• Valuable for early stage of design process.
• Valuable for real-time quick evaluation process.
• Limitation of the third-person perspective.
eye level
06. Expert Feedback

52. Limitations
07. Limitations & Future Work
Based on
Skeleton Data
Poses Collected
during Short Period
Limited
Design Freedom
(body physique x) (long-term comfort x) (only desk scenario)

53. Discussion
07. Limitations & Future Work
The Role of Digital Humans in the Design Process
Design
Process

54. Discussion
07. Limitations & Future Work
Informing Comfort and Health in Design

55. Discussion
07. Limitations & Future Work
Personalization of Virtual Human Agency

56. Discussion
07. Limitations & Future Work
Applicability in Other Human Interactions

57. Bokyung Lee, Taeil Jin, Sung-Hee Lee, Daniel Saakes
Department of Industrial Design, KAIST, Korea
Virtual Humans with Agency for Design Tools
SmartManikin: