In recent years, companies and society have undergone a rapid digital transformation, and“digital twin” is the key technology for utilizing various information and knowledge in both physical and cyber spaces. However, sensor data that can be collected by IoT (Internet of Things) is only a small aspect of physical space. In particular, in on-site human working fields, such as nursing care, agriculture, manufacturing, maintenance, and inspection (we call them “Gen-Ba”), there exists a vast amount of knowledge (“Gen-Ba knowledge”) that humans possess and cannot be captured by IoT sensors. Since Gen-Ba knowledge includes not only explicit but also latent and tacit knowledge, it has been difficult to utilize it in cyberspace, and there is still a large gap between the two spaces in the digital twin. This study aims to make it possible to capture, systematize, and utilize the vast amount of Gen-Ba knowledge in the human-centric digital twin. The smart voice messaging system (SVM), which has been used for ten years in actual fields, can be used to capture Gen-Ba knowledge. This study proposes a conceptual model of a human-centric digital twin with a focus on Gen-Ba knowledge according to several experiments of real-world applications of SVM.

1. PICMET2023@Monterrey
Naoshi Uchihira, Takuichi Nishimura, Koki Ijuin
Japan Advanced Institute of Science and Technology
Human-centric Digital Twin
Focused on ‘Gen-Ba’ Knowledge:
Conceptual Model and Examples by Smart Voice Messaging System
JAIST
HB-04.3（Knowledge Management-2）
on Thursday, July 27, 2023, 10:30 to 12:00 in the Guerrero
Monterrey

2. 2
Agenda
1. Background and Purpose
2. Literature Review
3. Digital Twin for Gen-Ba Knowledge
4. Case Examples
5. Technological Requirement for Gen-Ba
Knowledge Twin
6. Conclusion

3. 3
Gen-Ba Knowledge
Gen-Ba
Real and Actual Knowledge Creating Space
Nonaka and Takeuchi (1995)
Agriculture Inspection
Gen-Ba: On-site human working field
Gen-Ba knowledge: Knowledge (information) that
humans perceive and think about a target (firm
products, machines, elderly people, etc.) in Gen-Ba.
nursing care

4. 4
Gen-Ba Knowledge Example: Agriculture
Past
Experiential
Knowledge
Explicit
Knowledge
Based on past experiential knowledge, the farmers look at the field
and crop conditions, then make decisions, and perform the work.

5. 5
Digital Twin (Ideal Field)
•Digital Twin: Dynamic digital replica of physical space in
cyberspace (Jones 2020).
•Digital twin simulates and predicts the state and behavior
of physical space in cyberspace based on IoT data and AI,
then provide optimal solutions for physical space.
IoT Data Optimal
Solutions
Simulation
Prediction

6. 6
Digital Twin (Many Real Fields)
•IoT data is only a small part of the physical space, especially
in on-site human working field called “Gen-Ba.”
IoT Data
Optimal
Solutions
Simulation
Prediction
Data captured by Sensors

7. 7
Purpose
•This study aims to collect, analyze, and systematize the
vast amount of latent “Gen-Ba” knowledge that humans
possess in the context of digital twin.
IoT Data
Optimal
Solutions
Simulation
Prediction
Data captured by Sensors
Latent Gen-Ba Knowledge

8. 8
Agenda
1. Background and Purpose
2. Literature Review
3. Digital Twin for Gen-Ba Knowledge
4. Case Examples
5. Technological Requirement for Gen-Ba
Knowledge Twin
6. Conclusion

9. 9
Literature Review: (A) Digital Twin
•Digital twin has been developed
mainly for industry and factory,
such as Industrie4.0.
•Recently, digital twin has been applied to a wide range
of fields, such as healthcare, weather, environment,
energy, and cities.
horno3

10. 10
Literature Review: (B) Human-centric Digital Twin
•Human data in physical space is also an important element of
a digital twin, known as the “human digital twin.” (Miller&Spatz2022)
(ex., a model of workers can be used to provide the optimal
collaboration plan between robots and workers.)
•Umeda et al. (2019) proposed “digital
triplet” that integrates the space of
human intellectual activities into the
physical and cyberspace.
•However, neither the digital triplet nor the current efforts of
the human-centric digital twin have addressed the issue of
Gen-Ba knowledge.

11. 11
Literature Review: (C) Knowledge Management
• SECI Spiral Model (Nonaka &
Takeuchi 1995): Knowledge
consists of explicit and tacit
knowledge, and the
organizational knowledge
creation takes place in a spiral
between the explicit and tacit
knowledge in “Ba”.
• There are a few studies that
use digital technology to
support the generation,
codification, and transfer of
latent and tacit human
knowledge.
Source: Wikipedia

12. 12
Agenda
1. Background and Purpose
2. Literature Review
3. Digital Twin for Gen-Ba Knowledge
4. Case Examples
5. Technological Requirement for Gen-Ba
Knowledge Twin
6. Conclusion

13. 13
Digital Twin for Gen-Ba Knowledge
Gen-Ba knowledge consists of three layers.
Explicit Knowledge: Knowledge that is
explicitly conscious and can be
verbalized.
Latent Knowledge: Knowledge that is
not usually conscious but that humans
can partially verbalize when present in
Gen-Ba or when others ask them．
Tacit Knowledge: Knowledge that is
unconscious and cannot be expressed in
words intrinsically.
Explicit
Latent
Tacit

14. 14
Cyberspace
Physical
Space
Gen-Ba Workshop
(1) Capture (3) Utilize
(2) Systematize
Physical Monitor
Human Monitor
Original
Gen-Ba
Knowledge
Reproduced
Gen-Ba
Knowledge
Gen-Ba
Knowledge
Fragment
Conceptual Model of Gen-Ba Knowledge Digital Twin

15. 15
Conceptual Model of Gen-Ba Knowledge Digital Twin
Cyberspace
Physical
Space
Gen-Ba Workshop
(1) Capture (3) Utilize
(2) Systematize
Physical Monitor
Human Monitor
Original
Gen-Ba
Knowledge
Reproduced
Gen-Ba
Knowledge
Gen-Ba
Knowledge
Fragment
(1) Capture: Physical and human
monitors capture Gen-Ba
knowledge as a set of digital
knowledge fragments.
(2) Systematize: Digital
knowledge fragments are stored
and loosely systematized in
cyberspace.
(3) Utilize: Gen-Ba knowledge is
utilized after reinterpretation
and reconstruction from digital
knowledge fragments in the
workshop.

16. 16
Five
Senses
Interpretation
and
Thinking
Target Sensor
Data
Physical Monitor
(IoT Sensor)
Edge
Computing
Verbalize
Experiential
Knowledge
Human Monitor
(SVM)
(1) How to Capture Gen-Ba Knowledge
Physical
Space
Cyber
Space
Touch, sight,
hearing,
smell, and
taste

17. 17
Headset
Smart Phone Web Browser
Smart Phone
Voice
Recognition
Internet
Server
GPS
BLE beacon
Location Information
Gen-Ba
(Agriculture Case)
Leaves are
being eaten
by insects.
(1) How to Capture Gen-Ba Knowledge
Smart Voice Messaging System (SVM) (Uchihira 2013)

18. 18
Gen-Ba Non Gen-Ba
Sub Consciousness
Consciousness
Gen-Ba Non Gen-Ba
Sub Consciousness
Consciousness
Capture by SVM
Database
(B) Proposed Process
(A) Traditional Process
Vanish
Time
Time
Perceive
Perceive
Utilize in
Workshop
(1) How to Capture Gen-Ba Knowledge

19. 19
(2) How to Systematize Gen-Ba Knowledge
•Latent Gen-Ba knowledge is stored in the database as a
set of Gen-Ba knowledge fragments .
•Latent Gen-Ba knowledge is highly dependent on the
context and cannot be formally handled like logical
expressions, IF-THEN rules, or ontologies.
•Loose systematization: Gen-Ba knowledge
fragments are classified and interrelated
and can be retrieved from various
perspectives that also take the context into account.
•Human in physical space can retrieve Gen-Ba knowledge
fragments related to a (problematic) situation within the
workshop.

20. 20
Gen-Ba Knowledge Fragment Pattern
Pattern Explanation (Case of Agriculture)
Observation What was observed in the field (e.g., growth conditions of crops, pests and
diseases).
Actions What was done in the field (e.g., records of fertilizer and pesticide
application).
To-Do items
(Future Actions)
Things I wanted to do in the field in the future (e.g., things I want to take
care of during the same period of farm work in the next year).
Ingenuity Things I devised in the field (e.g., temperature adjustment in greenhouses
in consideration of weather changes).
Thoughts What I thought about in the Gen-Ba (e.g., consideration of how the
previous pesticide application was effective).
Retrospection Things recalled in the Gen-Ba (e.g., memories of failures experienced
during similar natural disasters in the past).
Concern Concerns in the Gen-Ba (e.g., what a beginner wants to confirm with
veterans regarding the appropriateness of his/her work).
Communication Things that I want to communicate with others in the Gen-Ba (e.g.,
precautions for other members who are in charge of the same field).

21. 21
Actual Situation
Reinterpretation
Reconstruction
Gen-Ba
Knowledge
Cyberspace
Physical
Space
Workshop
Human
Experiential Knowledge
Internalization
(3) How to Utilize Gen-Ba Knowledge
Gen-Ba knowledge is utilized after
reinterpretation and reconstruction
from digital knowledge fragments
in the workshop.

22. 22
Agenda
1. Background and Purpose
2. Literature Review
3. Digital Twin for Gen-Ba Knowledge
4. Case Examples
5. Technological Requirement for Gen-Ba
Knowledge Twin
6. Conclusion

23. 23
Case Examples
Application Explanation Gen-Ba Knowledge Benefits
Nursing Care
(2010–2014,
Tokyo)
SVM was used at a nursing
care facility by enabling staff
members to record and
share their activities and
awareness.
Observations during nursing
care and communications
with leaders and other staff
members.
Improvement of
collaboration, recordings,
and operation process.
Agriculture
(2017–2022,
Hokkaido and
Ishikawa)
SVM was applied in
agricultural fields
(greenhouses and outdoor
plots).
Farmer’s thoughts and
observations in Gen-Ba,
combining them with physical
sensor information, such as
temperature and humidity.
Mutual recognition of the
knowledge gap between
beginners and veterans,
which is effective for
education and guidance.
Manufacturing
(2022, Aichi)
SVM was applied at a
manufacturing site.
Improvements and
manufacturing know-how.
The workshop was
shown to be effective for
knowledge sharing across
different sites.
Maintenance
and Inspection
(2022,
Toyama)
SVM is being applied in
the maintenance and
inspection of electrical
facilities.
Inspection viewpoints and
improvements taken for
granted by some members.
There is a lot of one-
person work. Gen-Ba
knowledge sharing is
beneficial to members.

24. 24
Agriculture: (1) Capture
Careful treatment is required this season since the
green pepper can be easily ripped from its root. If
this happens, the product value will become zero.
Please be careful.
Uchihira, N., Yoshida, M., “Agricultural knowledge management using smart voice messaging systems: Combination
of physical and human sensors,” International Conference on Serviceology 2018, 148-151, 2018.

25. 25
Agriculture: (2) Systematize
Message Type and Informativeness
A1 Observation/quantitative additional information 5
A2 Observation/qualitative additional information 7
B0 Action/ no additional information 27
B1 Action/ quantitative additional information 6
B2 Action/ qualitative additional information 7
C1 Thought/quantitative additional information 4
C2 Thought/ qualitative additional information 24
Classification (manual)

26. 26
Agriculture: (3) Utilize
“I noticed this phenomenon last year for the first
time but did not understand its cause. Usually, it is
not so easy to rip away from the root unless
someone applies strength to do so.”
The discussion triggered by knowledge fragments
contributes to experimental knowledge sharing
among firm workers.
Workshop Discussion

27. 27
Agenda
1. Background and Purpose
2. Literature Review
3. Digital Twin for Gen-Ba Knowledge
4. Case Examples
5. Technological Requirement for Gen-Ba
Knowledge Twin
6. Conclusion

28. 28
Cyberspace
Physical
Space
Gen-Ba Workshop
(1) Capture (3) Utilize
(2) Systematize
Physical Monitor
Human Monitor
Original
Gen-Ba
Knowledge
Reproduced
Gen-Ba
Knowledge
Gen-Ba
Knowledge
Fragment
Conceptual Model of Gen-Ba Knowledge Digital Twin

29. 29
Technological Requirements for Gen-Ba Knowledge Twin
1. User Interface: for easier input of Gen-Ba knowledge
2. Awareness Support: Technology for giving information that helps
humans easily perceive Gen-Ba knowledge based on situations.
For example, the system can show similar Gen-Ba knowledge for
analogous inspiration that was captured in a similar situation.
3. Fragment knowledge systematization: Automatic classification
(clustering) and knowledge extraction (mining) technologies for
systematically storing fragment knowledge in cyberspace.
4. Workshop Support: Technology to support knowledge sharing and
transfer by extracting and presenting appropriate knowledge
fragments based on workshop situations.
5. Manualization of workshop results: Technology to record and
summarize workshop discussions and automatically manualize the
parts that can be converted into explicit knowledge.

30. 30
Cyberspace
Physical
Monitor
Workshop
Physical Space
Human
Monitor
1 2
3
4
5
Technological Requirements for Gen-Ba Knowledge Twin
User Interface Awareness Support
Fragment knowledge
systematization
Workshop
Support
Manualization
of workshop
results

31. 31
Agenda
1. Background and Purpose
2. Literature Review
3. Digital Twin for Gen-Ba Knowledge
4. Case Examples
5. Technological Requirement for Gen-Ba
Knowledge Twin
6. Conclusion

32. 32
Conclusion
•Digital twin is the key technology for utilizing information
and knowledge in both physical and cyber spaces.
•A vast amount of latent and tacit knowledge remains only
in physical space in on-site human working
fields (Gen-Ba) (agriculture, nursing care,
inspection, etc.) since it is difficult to capture
it by IoT sensors.
•Smart voice messaging system (SVM) makes it possible to
capture, systematize, and utilize Gen-Ba knowledge.
•We propose a conceptual model of a human-centric digital
twin with a focus on Gen-Ba knowledge according to
several experiments of real-world applications of SVM.

33. 33
Contribution
• Traditionally, there are several studies on latent
and tacit knowledge management. However, the
latent knowledge management has not been well
explored from the perspective of digital technology.
•Gen-Ba knowledge digital twin is a new type of
human-centric digital twin which is expected to
improve the efficiency of latent knowledge sharing.
• Gen-Ba knowledge digital twin is a different
approach from the digital triplet which is not
intended to capture latent knowledge that appears
in Gen-Ba.

34. 34
ChatGPT and Gen-Ba Knowledge Twin
• In recent years, AI based on large language models (LLMs),
exemplified by ChatGPT, has rapidly evolved and is expected to have a
significant impact on knowledge management.
• However, these LLMs generate text and are not grounded in physical
space (grounding problem), which becomes a major challenge.
• The proposed approach involves the verbalization and utilization of
latent human Gen-Ba knowledge based on human five senses and
experiential knowledge.
• Gen-Ba knowledge captured by SVM (human monitor) is grounded!
Five
Senses
Interpretation
and
Thinking
Target Verbalize
Experiential Knowledge

35. 35
Limitation and Future Works
•This paper proposes a conceptual model of Gen-Ba
knowledge digital twin. We will continue to work on the
technical requirements mentioned in this paper.
•Even if the technological requirements are satisfied, it still
depend on human skills, such as how to facilitate a
workshop. Education of such human skills is also a future
challenge.
•This model and system should be evaluated not only in
nursing care, agriculture, manufacturing, and
maintenance and inspection, but also in other physical
and adaptive intelligent services (e.g., security services,
customer service in hotels and shops).

36. 36
Thank you for your attention.
Naoshi Uchihira, Dr.Eng. and Ph.D of Knowledge Science
Professor and Dean, School of Knowledge Science
Japan Advanced Institute of Science and Technology
e-mail：uchihira@jaist.ac.jp