ISSIP_AI_Collab (Academic-Industry Collaboration Lab) PSU_Spring_2024_Team Topic: AI Digital Twins of People TITLE: DIGITAL TWINS FOR PEOPLE IN BUSINESS AND SOCIETAL ROLES Students Bryn Goldman (https://www.linkedin.com/in/bryn-goldman-penn-state/) Natalie Grim (https://www.linkedin.com/in/natalie-grim/) Gonzalo Rambla (https://www.linkedin.com/in/gonzalo-rambla/) Clare Relihan (https://www.linkedin.com/in/clarerelihan/) Shakeb Siddiqui (https://www.linkedin.com/in/shakeb-siddiqui/) Advisors Prof. Vittal Prabhu - PSU I&SE (https://www.linkedin.com/in/vittalprabhu/) Anshul Balamwar - Doctoral Candidate (https://www.linkedin.com/in/anshul-balamwar/) Toni Rae - SAP, Alumni Coach (https://www.linkedin.com/in/tonirae/) Client Jim Spohrer - ISSIP (https://www.linkedin.com/in/spohrer/) ISSIP YouTube Recording: https://youtu.be/YGWCji0WAzA PSU Learning Factory Recording : https://www.youtube.com/watch?v=i28yoQjc560 GitHub: https://github.com/Shakeb100/IISSIP-DigitalTwins ISSIP Slideshare Presentation: TBD ISSIP Slideshare Whitepaper: TBD ISSIP Blog Post: TBD

1. Digital Twins For People In Business and
Societal Roles
Final Presentation
ISSIP-PSU-SPR24
25 April 2024
Y or N
Intellectual Property Rights Agreement Applies
Non-Disclosure Agreement Applies
Team Members: ISSIP-PSU-SPR24
Bryn Goldman, Natalie Grim, Gonzalo Rambla,
Clare Relihan, Shakeb Siddiqui
Sponsor: The International Society of Service Innovation Professionals
Jim Spohrer
Advisors: Vittaldas Prabhu, Anshul Balamwar, Toni Rae

2. Meeting Goals & Outline
• INTRODUCTION
• PROBLEM STATEMENT
• PROJECT OBJECTIVES
• APPROACH & PROJECT MANAGEMENT
• RESULTS
• PROJECT EXPENDITURES
• ACKNOWLEDGEMENTS
• REFERENCES

3. Introduction
• ISSIP
• Digital Twin
• Prototype Limitations

4. Problem Statement
• The Digital Twins Investigation Project seeks to revolutionize
how ISSIP volunteers interact and collaborate by harnessing
the power of digital twins and simulation technology.

5. Project Objectives- Requirements
Group Twin
Interactive demo
O*NET profession report
White paper

6. Project Objectives- Risk Management
ID Description Severity Likelihood Mitigation
1 ISSIP Goals high low
Past successful
projects
2 Paywalls high high
Free software
available
3 Digital Twin medium low
Already
investigated
HeyGen
4 Changing scope medium medium
Weekly sponsor
meetings
5
Generative
platform
combination
low low Early prototypes

7. Risk Management
Matrix before mitigation

8. Risk Management
Matrix post mitigation
Severity &
Likelihood
Minimal Minor Majors Hazardous Catastrophi
c
Frequent
Probable
Remote 2
Extremely
Remote
Extremely
Improbable
1,3,4,5

9. Approach: Simulation Decision
395 | 445

10. Approach: Functional Decomposition Diagram

11. Approach: Ideation

12. Approach: Alpha Prototype
• HeyGen: Voice and Video Generation
• HeyGen is a tool generating lifelike avatars from video samples
and text
• Assess HeyGen's suitability for image and voice generation
• This prototype validated HeyGen’s functionality for the final
simulation

13. Approach: Beta Prototype
• GPT-3: Presentation Script Generation
• The Beta prototype focused on evaluating the GPT-3 API's
efficacy in converting research papers into presentation
scripts.
• Developed on a Next.js website, the prototype integrated
ChatGPT-3 API for text input and prompt engineering.
• This prototype offered valuable insights into GPT-3 API's
viability for research paper to speech functionality and Next.js
platform suitability for hosting the final product.

14. Project Managment

15. Results
• A functioning prototype,
demonstrating the potential of an
ISSIP Digital Twin application.
• Text to speech and Twin
speech capabilities
• Github repository showcasing API
integration and code
development.
• Codebase Video Walkthrough

16. Project Expenditures
• The main goal was to provide research and an understanding
that does not require any outside costs
• This is because ISSIP is the non-profit organization we
are supporting
• The project was successful in this regard and used all free
online resources

17. Acknowledgements
We extend our heartfelt appreciation to our esteemed Sponsor,
Jim Spohrer, whose vision and support have been instrumental
throughout this project. We are also grateful to our Advisors,
Vittaldas Prabhu and Anshul Balamwar, for their invaluable
insights and guidance. Special thanks to our Alumni Coach, Toni
Rae, whose expertise and mentorship have greatly contributed
to our journey. Your commitment and dedication have played a
pivotal role in the success of this project. Thank you for your
contributions.

18. Video Recording
• https://psu.mediaspace.kaltura.com/media/PSU+IE+ISSIPA+Fi
nal+Presentation/1_5docfadi

19. Q&A

20. Student Team

22. Poster

23. BACKGROUND
The International Society of Service Innovation Professionals, ISSIP, partnered with the
Pennsylvania State University to research and explore the creation of digital twins to help their
volunteers be more productive and able to have high quality interactions with more ISSIP
community members.
ISSIP is a non-profit that focuses on enhancing the global service economy for both businesses
and individuals.
A Digital Twin is a digital representation of a physical object, person, or process.
PROBLEM STATEMENT
A recent challenge has surfaced where ISSIP board members occasionally find themselves
unable to participate in scheduled panel discussions or meetings due to conflicting
obligations.
How can AI technology be harnessed to create a digital twin of a potential panel
speaker than can take the place of the speaker if they are longer able to attend?
OBJECTIVES
1. Develop an AI-driven digital twin capable of generating and delivering a panel
talk presentation based on a research paper or notes.
2. Produce a white paper aimed at educating ISSIP members on leveraging AI
technologies in their daily lives, while addressing associated ethical considerations.
KEY REQUIREMENTS​
1. The white paper must analyze three professions from the O*NET and explore
which specific tasks can be aided or completed by a digital twin.
2. The white paper and simulation must be accessible to the ISSIP community with
respect to language and content.
3. The digital twins shall be able to give talks, summarize a document, or generate
a paper based on background information from the subject.
DIGITAL TWINS FOR PEOPLE IN BUSINESS AND SOCIETAL ROLES
Sponsor: Jim Spohrer
Advisors: Vittaldas Prabhu, Anshul Balamwar
Alumni Coach: Toni Rae
Bryn Goldman, Natalie Grim, Gonzalo Rambla, Clare Relihan, Shakeb Siddiqui

24. APPROACH
• An iterative process to determine the
optimal simulation deliverable.
• Initially, we used a weighted decision
matrix to compare options between a
Simio simulation and a panel talk
simulation.
• Subsequently, utilizing a morphological
matrix, we brainstormed potential
technologies for each component of the
simulation: audio, video, and text. The
resulting simulation structure is
illustrated to the right.
• Upon thorough examination of the
matrix and further research, the team
discovered additional technologies.
They concluded that consolidating
audio and video functionalities into one
platform might be more feasible.
• We used a weighted decision matrix
when considering which APIs and
technologies to incorporate into the
simulation.
PROTOTYPES
HeyGen: Voice and Video Generation
The Alpha prototype aimed to assess HeyGen's suitability for image and voice generation.
HeyGen is a tool generating lifelike avatars from video samples and text. This prototype
validated HeyGen’s functionality for the final simulation.
GPT-3: Presentation Script Generation
The Beta prototype focused on evaluating the GPT-3 API's efficacy in converting
research papers into presentation scripts. Developed on a Next.js website, the prototype
integrated ChatGPT-3 API for text input and prompt engineering. This prototype offered
valuable insights into GPT-3 API's viability for research paper to speech functionality and
Next.js platform suitability for hosting the final product.
CHALLENGES
• A significant challenge arose when the technology tested in the Alpha prototype,
HeyGen, became too expensive. This necessitated a substitution, leading to the
incorporation of an alternative platform, D-ID, into the final prototype.
• Another challenge arose during the development of the simulation as each API has its
own unique set of technical intricacies, causing the final result to be a work in progress
demo that, in the future, could be developed into a deployable solution.
FINAL SIMULATION
The panel member simulation
employs GPT-3, D-ID, and
Next.Js to create a demonstration
of how ISSIP member can use AI
technologies to create a panel
talk from a research paper.

25. BENEFITS
 Increased accessibility to AI technology for ISSIP members
 Ability to hold panel talk despite extenuating circumstances and
changing availability.
 An analysis of how all types of careers that ISSIP members have
can benefit from AI,
FUTURE WORK
While the project has achieved success, it faces limitations including
reliance on external AI technologies and potential accessibility
barriers. Ethical considerations surrounding AI usage, particularly in
creating lifelike avatars, require ongoing evaluation. Future work
includes integrating emerging AI technologies, addressing
accessibility concerns, and developing ethical frameworks. User
feedback will drive iterative refinement to meet evolving needs.
ETHICAL CONSIDERATIONS
• The integration of digital twins in the ISSIP community raises ethical concerns regarding
privacy, bias, and ownership.
• Obtaining consent for data collection is crucial to uphold ethical standards and prevent legal
issues.
• ISSIP must establish protocols for data management and security to safeguard sensitive
information.
• Clear guidelines on rights, ownership, and responsible usage are essential to ensure ethical
deployment of digital twins while balancing innovation with individual rights protection.
FINAL WHITE PAPER
• Through O*NET, the team was able to investigate some of the key activities that a wide variety of
jobs entail and areas AI digital twins could assist in.
• When researching the different occupations, it was important to take a wide range of professions that
being technical, non-technical, entry-level, and more.
• While many ISSIP community members hold many technical, managerial, and senior roles, it is
important to still investigate how digital twins can help many entry level jobs.
COST
The main goal was to provide research and an understanding that does not
require any outside costs. This is because the company and sponsor we are
supporting throughout the semester is a non-profit. The project was
successful in this regard and spent none of the allotted budget.
VERIFICATION
 Analyze three professions from the O*NET
 The white paper and simulation must be accessible to the ISSIP
community
 The digital twins shall be able to give talks, summarize a document,
or generate a paper based on background information from the
subject.