Discussing the possibilities of large language models for the automatic generation of academic content by the students (e.g. master thesis), and the related need for changes in the way in which to educate and evaluate students.

1. LLM orodja so tukaj. Kaj sedaj?
Peter Trkman
Luka Tomat

2. Disclaimer
• This set of slides is intended for a “pedagogical conference” at School of
Economics and Business, University of Ljubljana on September, 11 2023.
• The slides are not meant as a stand-alone study resource but as a companion to
the workshop/starting point for the debate.
• All examples are generated by us with the available large-language model tools;
no completeness or accuracy claimed.
• For any further questions, feel free to contact us at peter.trkman@ef.uni-lj.si and
luka.tomat@ef.uni-lj.si
Feel free to use the slides under the
CC BY-SA license

3. Izhodišča
• Orodja umetne inteligence obstajajo že desetletja. Zakaj so tako popularna zdaj?
• Nova orodja so izjemno enostavna za uporabo brez kakršnega koli informacijskega znanja
• Ukvarjajo se s temami, ki so delavcem z znanjem in splošni javnosti bližje kot npr. optimizacija
skladišč
• Obstaja izjemna raznolikost načinov uporabe
• Ustvarjanje vsebin/slik ni le poceni, temveč ima (praktično) ničelne mejne stroške (tako finančne kot
časovne)
• odpirajo temeljna "filozofska" vprašanja, kaj je ustvarjalnost in "človečnost” in kako poučevati
Kaj to pomeni za nas
• Pripovedni del je manj pomemben. Vsak lahko zdaj piše lepa besedila. Skrbno napisan tekst ni več
signal česar koli
• "small talk" bo izumrl: v preteklosti je bilo skrbno napisan tekst, veliko število reference, lepi stavki,
dolg magisterij...znak truda in sposobnosti avtorjev.
• Kako sedaj izobraževati?
Povzeto po: https://www.linkedin.com/posts/peter-trkman_workshop-on-chatgpt-and-large-language-
models-activity-7081614403489476608-DNQL

4. Izhodišča (2)

5. Out-of-scope za danes
Uporaba LLM izven pedagoško-študijskega procesa
Uporaba drugih orodij generativne intelligence
Več na: https://www.cpoef.si/open-programs/generativna-umetna-inteligenca/ 

6. Vsebina
• Kaj LLM že znajo
• Preprečevanje uporabe LLM
• Učenje uporabe LLM
• Uporaba LLM pri konkretnih primerih

7. Tipična orodja
Chat GPT
Scholarcry
Scispace
https://elicit.org/
…

8. Obstaja jih sicer ogromno (npr. za akademsko delo)
Seach Engines:
1. SciLynk (scilynk.com)
2. Scinapse (scinapse.io)
3. Perplexity (perplexity.ai)
4. Semantic Scholar (semanticscholar.org)
Literature Review:
1. Iris (iris.ai)
2. Elicit (elicit.org)
3. Inciteful (inciteful.xyz)
4. The Literature (the-literature.com)
5. Research Rabbit (researchrabbit.ai)
6. Connected Papers (connectedpapers.com)
7. Discovery (discovery.researcher.life)
8. Evidence Hunt (evidencehunt.com)
9. System Pro (pro.system.com)
10. Consensus (consensus.app)
11. Keenious (keenious.com)
12. Scite (scite.ai)
Brainstorming Research Questions:
1. Claude (claude.ai)
2. ChatGPT (chat.openai.com)
Reading Research Papers:
1. SciSpace (typeset.io)
2. Scholarcy (scholarcy.com)
Chatting with Research Papers:
1. Claude (claude.ai)
2. Humata (humata.ai)
3. ChatPDF (chatpdf.com)
Taking Notes:
1. Glasp (glasp.co)
2. Lateral (lateral.io)
3. ClioVis (cliovis.com)
Drafting:
1. Jenni (jenni.ai)
2. AudioPen (audiopen.ai)

9. Primer analize

10. Chat GPT za strukturo mag. dela
ChatGPT
Certainly! Writing a thesis on digital transformation requires a structured approach to ensure that you cover all relevant aspects of the topic. Here's a suggested structure for your thesis:
Title Page:
•Title of the Thesis
•Your Name
•Institutional Affiliation
•Date
Abstract:
•A brief summary of your thesis (typically 150-250 words) that highlights the key points and findings.
Acknowledgments:
•Optional section where you can express gratitude to those who supported you during your research.
Table of Contents:
•List of chapters and sections with page numbers for easy navigation.
List of Figures and Tables:
•If applicable, include a list of figures and tables used in your thesis.
List of Abbreviations and Acronyms:
•Include any abbreviations or acronyms used in your thesis along with their definitions.
List of Symbols:
•If applicable, include a list of symbols used in your thesis.
Chapter 1: Introduction
•Introduction to the topic of digital transformation.
•Background and context.
•Problem statement and research questions.
•Purpose and significance of the study.
•Structure of the thesis (briefly outline what each chapter covers).
Chapter 2: Literature Review
•Overview of the concept of digital transformation.
•Historical context and evolution.
•Theoretical frameworks and models related to digital transformation.
•Key concepts, terms, and definitions.
•Previous research and studies on digital transformation.
Chapter 3: Theoretical Framework
•Select and discuss relevant theoretical frameworks or models that will guide your research.
•Explain how these frameworks relate to digital transformation.
Chapter 4: Methodology
•Explain your research methodology (qualitative, quantitative, mixed methods, case study, etc.).
•Data collection methods (interviews, surveys, data analysis, etc.).
•Data analysis techniques.
•Ethical considerations.
Chapter 5: Case Studies or Empirical Research
•Present your findings based on your chosen methodology.
•Analyze and discuss the data.
•Include case studies or real-world examples to illustrate digital transformation concepts.
Chapter 6: Discussion
•Interpret the results and findings.
•Discuss how your research answers the research questions.
•Analyze the implications of your findings for digital transformation in practice.
•Compare and contrast your results with existing literature.
Chapter 7: Conclusion
•Summarize the key findings of your research.
•Reiterate the significance of your study.
•Provide practical recommendations for organizations undergoing digital transformation.
•Discuss the limitations of your study.
•Suggest areas for future research.
References
•List all the sources you cited in your thesis following a specific citation style (APA, MLA, Chicago, etc.).
Appendices
•Include any supplementary material that supports your thesis (e.g., survey questionnaires, interview transcripts, additional data, etc.).
Index (Optional):
•If your thesis is particularly long and detailed, you may consider including an index to help readers navigate specific topics or concepts.
Remember to follow the formatting and citation guidelines required by your institution or department. Good luck with your thesis on digital transformation!

11. Chat GPT za razvoj izobraževalnik vsebin

12. Chat GPT – primer intervjuja

13. Celotni odgovor

14. Kaj pa akademske reference – ChatGPT je slabši
Title: The Positive Impact of Business Analytics on Non-Financial Performance
Introduction: In the rapidly evolving landscape of modern business, the utilization of data-driven insights has become an indispensable tool for organizations aiming to achieve sustainable growth and success. Business
analytics, a comprehensive approach to extracting insights from data, has emerged as a powerful catalyst for informed decision-making across various sectors. This support paper aims to present a compelling
argument for the hypothesis that "business analytics positively affect non-financial performance." By delving into the ways in which business analytics empowers organizations to optimize their operations,
enhance customer experiences, and foster innovation, we will demonstrate how it goes beyond financial metrics to substantially impact non-financial dimensions of performance.
Optimized Operational Efficiency: Business analytics provides organizations with the means to optimize operational efficiency, leading to improvements in non-financial performance areas. By analyzing operational data,
organizations can identify bottlenecks, inefficiencies, and areas of improvement in their processes. This leads to streamlined workflows, reduced operational costs, and enhanced resource allocation. For instance,
predictive analytics can help organizations anticipate demand patterns, enabling them to optimize inventory levels and reduce stockouts. Such improvements directly impact non-financial metrics like production
efficiency, customer satisfaction, and employee morale.
Enhanced Customer Experience: A core pillar of non-financial performance is the customer experience. Business analytics empowers organizations to gain a comprehensive understanding of customer preferences,
behaviors, and needs. Through advanced analytics techniques, such as sentiment analysis and customer segmentation, organizations can tailor their products, services, and marketing strategies to align with
customer expectations. Consequently, improved customer experiences result in higher customer loyalty, increased brand advocacy, and positive word-of-mouth. These non-financial outcomes not only contribute
to enhanced performance but also have the potential to generate long-term value for the organization.
Data-Driven Innovation: Innovation is a critical driver of non-financial performance, and business analytics plays a pivotal role in fostering innovation within organizations. By analyzing market trends, consumer insights,
and competitive landscapes, organizations can identify opportunities for new product development, service enhancements, and process innovations. Furthermore, data-driven insights enable organizations to
make informed decisions about potential expansion into new markets or the introduction of novel business models. As innovation directly impacts brand perception, employee engagement, and overall
competitiveness, business analytics indirectly enhances non-financial performance.
Agile Decision-Making: Business analytics equips organizations with real-time, data-driven insights that facilitate agile decision-making. In today's dynamic business environment, the ability to respond swiftly to changing
circumstances is paramount. Analytics-driven decision-making enables organizations to proactively address challenges, capitalize on emerging trends, and exploit market opportunities. This agility in decision-
making translates into non-financial benefits, such as improved risk management, better adaptability to market fluctuations, and increased operational resilience.
Employee Performance and Engagement: The impact of business analytics on non-financial performance extends to the realm of human resources. By leveraging data analytics, organizations can gain insights into
employee performance, satisfaction, and engagement. This information allows for targeted interventions, such as personalized training and development programs, which lead to improved employee productivity
and job satisfaction. A motivated and engaged workforce contributes to non-financial outcomes like reduced employee turnover, enhanced team collaboration, and a positive organizational culture.
Conclusion: In conclusion, the hypothesis that "business analytics positively affect non-financial performance" is strongly supported by a wealth of evidence. Through optimized operational efficiency, enhanced customer
experiences, data-driven innovation, agile decision-making, and improved employee performance and engagement, business analytics serves as a multifaceted enabler of non-financial success. Organizations
that effectively harness the power of data analytics are poised to not only achieve financial growth but also to thrive across various non-financial dimensions, securing a competitive advantage in today's data-
centric business landscape.
Operational Efficiency:
Davenport, T. H. (2006). Competing on Analytics. Harvard Business Review.
Gupta, J. N. D., & Jain, A. K. (2018). Impact of Business Analytics on Operational Performance: An Empirical Study. International Journal of Information Management.
Customer Experience:
Verhoef, P. C., Lemon, K. N., Parasuraman, A., Roggeveen, A., Tsiros, M., & Schlesinger, L. A. (2009). Customer Experience Creation: Determinants, Dynamics, and Management Strategies. Journal of
Retailing.
Luo, X., & Ba, S. (2012). Perspective: The Impact of Information Technology Investments on Customer Satisfaction, Loyalty, and Retention: Evidence from Franchised Firms. Information Systems Research.
Data-Driven Innovation:
Westerman, G., Bonnet, D., & McAfee, A. (2014). Leading Digital: Turning Technology into Business Transformation. Harvard Business Review Press.
Berman, S. J., Bell, R., & Kim, K. (2019). The Use of Big Data Analytics in New Product Development: Drivers and Performance Implications. Journal of Business Research.
Agile Decision-Making:
Lacity, M., & Willcocks, L. P. (2013). Nine Ways to Reduce Cognitive Biases in Decision Making. MIT Sloan Management Review.
Serrano, A., & Karahanna, E. (2016). The Compensatory Interaction between User Capabilities and Technology Capabilities in Influencing Task Performance: An Empirical Investigation of Business Analytics.
Information Systems Research.
Employee Performance and Engagement:
Hameed, T., & Waheed, A. (2018). Impact of Business Analytics on Employee Performance: The Mediating Role of Employee Engagement. International Journal of Information Management.
Koelemeijer, K., van Deursen, A., & van der Horst, A. (2019). How Business Analytics Contributes to Organizational Performance: An Exploratory Study. International Journal of Information Management.

15. Kaj pa akademske reference: ChatGPT je slabši

16. Elicit obvlada tudi to
The papers provide different definitions of
business analytics. Delen (2018) states that
business analytics is the encapsulation of
all mechanisms that help convert data into
actionable insight for better and faster
decision-making. Camm (2019) defines
business analytics as a set of tools and
techniques used to analyze data and make
informed business decisions. Yin (2020)
defines business analytics as a system
enabled by machine learning techniques
that aims to promote the efficiency and
performance of an organization by
supporting the decision-making process.
Acito (2014) describes business analytics
as a revolution that leverages value from
data and aligns strategy and desirable
behaviors to business performance
management in conjunction with analytic
tasks and capabilities.
Kaj je poslovna analitika
Kaj je poslovni model?

17. Kako deluje Elicit
Hm, pravzaprav zelo podobno kot mi
ali študenti, ko delamo pregled
literature

18. Podrobnejša vprašanja: “How does business
analytics affect performance?”
The papers suggest that business analytics can positively affect performance.
Ashrafi (2019) found that business analytics capabilities can increase a firm's agility,
which in turn can improve performance. Kakhki (2016) also found a positive
relationship between the extent of business intelligence and analytics implementation
and business performance. Trkman (2010) found a statistically significant
relationship between analytical capabilities and supply chain performance. Shanks
(2010) proposed a theoretical framework for understanding how business analytics
can lead to value-creating actions that improve firm performance and competitive
advantage. Overall, the papers suggest that business analytics can have a positive
impact on performance, but the extent of this impact may depend on factors such as
the type of business analytics used and the level of technological support.

21. Jenni

22. Preprečevanje uporabe LLM
Orodja za zaznavanje obstajajo, vendar so “po definiciji” nezanesljiva.
De facto študentom v nekontroliranem okolju ne morete (in ne smete) preprečiti
uporabe LLM
Razmislite, kaj jih hočete naučiti

23. Orodja za zaznavanje
Turnitin
Scispace

24. Enostavno pravilo
Če morate za preverjanje študentskega dela (domačih nalog/seminarskih) uporabljati
detektorje AI, spremenite nalogo in/ali način ocenjevanja.

25. Kako sedaj?

26. Rešitve?
Obstajajo številne tehnične rešitve
(detektorji AI), ki so preproste za
uporabo, vendar ne rešujejo širšega
izziva trenutnega spreminjajočega
okolja.
Tehnologija pri tem ni ključna, pomembno
je razumavanje širše slike, želene
spremembe pa je potrebno razumeti na
konceptualnem nivoju, preden jih
uvedemo v prakso.
Vir: https://www.compilatio.net/en/blog/best-ai-detectors

27. Zahtevati uporabo AI
V kolikor je smiselna
Primer: pri predmetu e-business morajo študenti poslovnih ved razviti spletno stran
Nesmiselno bi jih bilo učiti html, css ali grafično oblikovanje itd.
Morajo pa razmisliti o vsebini, temeljnih sporočilih, se naučiti uporabe novih orodij
ter v predstavitvi povedati, zakaj so se tako odločili in katera orodja so se naučili

28. Primer naloge (pred Chat GPT)
Select a process within a chosen company (it can be a real or a fictional example)
Imagine you are developing a new software application for this process. What specifically would you do in
each phase of SDLC ? Be as concrete as possible.
How about Scrum - what would you do if using this method?
Which approach would you recommend?
Naloga vredna 4 točke (od 100) pri predmetu Informatics (1. letnik UPEŠ, English track)
Pedagoški cilj (neformalen): študente naučiti, kako hitro googleati, razmisliti, kako bi to aplicirali na nek
praktičen primer, kritično razmišljanje (prednosti/slabosti metode) ter jih navdušiti za nadaljnje učenje o
razvoju programskih rešitev (vsaj v vlogi poslovnih uporabnikov)

29. V 2023 nekoristna
naloga

30. V 2023 nesmiselna polja v prijavnem obrazcu

31. Še vedno možne naloge
Ni vse na Internetu:
“Specifically, students were asked to assess the current state of the carbonated soft drink
industry in 2023 and analyze significant changes in the industry forces (based on Porter’s
Five Forces Framework) since the end of the case study discussed in class”
ali pa
“Describe the main lessons of the Hamapol case study”
“Reflect on the guest lecture of Ms. Butina and how her lessons could be applied to the
case that Mr. Novak has presented in his guest lecture”
Možen nov poskus: date vprašanje/področje, ki ga niste obravnavali pri predmetu.
Študentje imajo 10 minut časa za odgovor.

32. Še vedno možne naloge
How do you plan to control for Cronbach
common method variance?
How about Cronbach common method
traceability?

33. Učenje, kako uporabljati LLM
“Helping students to become better prompt engineers” ni smiselno
Generično znotraj številnih predmetov ni smiselno (tudi uporabe Googlea ne učimo)
Po potrebi lahko delavnice na RPS
(glej npr. poglavje 3 tule : https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4516801
in še mnogo drugih virov)

34. Kaj morajo študenti znati
Manj pomembno: branje in povzemanje velikega nabora virov
Bolj pomembno: hitro učenje; sposobnost postavljanja pravih vprašanj
Tipičen primer: črkovalne napake pri predmetu Poslovna angleščina vplivajo na
oceno. Pri izpitu Informatics ne vplivajo. Pri seminarski nalogi pa kažejo površnost
študenta (niti črkovalnika ni uporabl).
Svojemu sinu (12 let) za domače naloge pri angleščini ne pustim uporabljati niti MS
Worda. Hčerka (14 let) za zgodovino in DKE lahko uporablja Chat GPT.

35. Možne spremembe: maksimalna transparentnost
“Opravil sem 3 intervjuje v podjetju Mercator” ali “zbral sem odgovore 190
respondentov” ni več OK
“Dne 11. 9. 2023 sem v podjetju Špedko opravil intervju z vodjo službe za informatiko
ga. Jožico Novak. Intervju je potekal v sejni sobi podjetja na Vojkovi 32, trajal pa
je med 9.00 in 10.30. Zvočni posnetek intervjuja je na razpolago pri avtorju
magistrske”

36. Možne spremembe: magistrske naloge
Radikalno zmanjšati število citiranih referenc v zaključnih/seminarskih nalogah na neko smiselno količino (ostalo v “nadaljnje
branje”).
Radikalno povečati zahtevo po študentovem poznavanju vsebine vsega, kar je citiral. Teme iz “nadaljnje branje” mora okvirno
poznati.
Zagovor se iz “pogovora” spremeni v de facto “diplomski izpit”.
Možen potek zagovora
V: Kje pa je tale Ashrafi zbiral podatke?
O: hm, ne vem
[over & out - pade zagovor, “se vidimo čez 1 mesec”]
V: hm, kje pa je tale Ashrafi zbiral podatke?
O: v Iranu
V: pa je ta iranska analiza sploh relevantna za vaš primer slovenskega podjetja?
O: hm, ne vem [ima še šanso, da naredi, ampak samo če res dobro odgovori na ostalo]
O: ja, ker...[zelo natančen odgovor, ki povzame omejitve raziskave Ashrafija in pove tudi, zakaj so Ashrafijeve ugotovitve vseeno
relevantne za njegov konkreten primer in kako so mu pomagale pri implementaciji orodja].... [naredi s super oceno. V takem
primeru “So what if ChatGPT wrote it?”]

37. Tipično vprašanje študenta
As for the purpose and goals I used a bit of let's say "inspiration" from
ChatGPT, and I saw in the guidelines that we should reference the AI tool.
Can you tell me whether I am on the right path, should I include the in-cite
reference there?
In terms of Chat-GPT: Take Chat-GPT as if you would discuss the topic with me or e.g.
your mother (if she were the Chief Information Office in Skopje) - it is perfectly OK to talk to
people (and computers 🙂), get ideas for them, interact.
Take ChatGPT as if you were discussing topics with an expert who will help point you in
the right direction, give you some ideas, entertain you, or an assistant who will summarise
some thoughts. But do not take it as a "credible source". Also, in the thesis, don't cite
ChatGPT (in the same way as you would not cite your mother's thoughts or e.g. "as
professor Trkman said, smart mobility is important").

38. Možna politika pri predmetu
It is perfectly OK to use ChatGPT (partly) for this particular task... but mainly to get ideas
on what to explore further/how to structure your answer.
Your answer should show that you are personally well versed in the topic and should not
be a copy & paste answer from ChatGPT.
Take ChatGPT as if you were discussing topics with an expert who will help point you in
the right direction, or an assistant who will summarise some thoughts.
It is perfectly OK to talk to people (and computers 🙂), get ideas for them
Also: be careful, ChatGPT makes things up (especially the list of references). Any
mistake is your mistake.
You must fully understand your answer - selected students may be chosen to give further
oral explanations of their answers.

39. Sporočilo za študente
Example: Let’s say that you will be a consultant for WMS implementation and your
customer will ask you: “what is the support for order picking by voice offered by your
WMS? Which are the main expected problems in its implementation in my company?
What is the pay-back period?”
Your response can’t be: “Hm, wait for a few minutes for me to ask ChatGPT ver. 10
what WMS, payback period and order picking are”.
You will need a profound theoretical knowledge, practical experience, ability to
analyse and apply your insights into a specific context of a company, ability to
communicate with diverse audiences...and then the ability to get things done.

40. Končna misel
#chatgpt je velika tema.Vendar ga ne uporabljajte preveč. Sami pišite osebna e-
poštna sporočila.
Trenutno lahko dolgi, lepo napisani ChatGPT maili še delujejo
Bolj ko postaja naš svet digitalen, bolj so majhni, človeški koraki ključni
Prošnja nekomu za uslugo (ali ljubezensko pismo) naj ne bo dolgo, generično
besedilo z umetno inteligenco, temveč kratko sporočilo "ni mi vseeno zate".

41. Konec