Join Fergal as he shares how they developed ‘Fin’, a chatbot that actually solves up to 50% of support questions. This session will shed light on the product development process of Fin, the challenges encountered, and the opportunities it brought forth. The talk will encompass Intercom’s experiences and lessons learned from integrating large language models in a live production environment.

1. How we built Fin
@fergal_reid

6. Fin

9. GPT / LLMs
• Internet sized tech change
• Change in capability
• Change in how we build and use AI

10. Level 1: GPTs are incredible!

13. Level 1: GPTs are incredible!

14. Level 1: GPTs are incredible!
Level 2: GPTs make things up and aren’t trustworthy.

15. Level 1: GPTs are incredible!
Level 2: GPTs make things up and aren’t trustworthy.
Level 3: GPTs can be incredible when used right

16. See them as engineering components

17. See them as engineering components
Separate out aspects accidentally bundled

18. What is GPT?

19. Training objective: token prediction

20. Training objective: token prediction

23. • A sequence model
• That uses ‘attention’
• Gradient descent

24. • A sequence model
• That uses ‘attention’
• Gradient descent

25. Not a useful model

26. Not a useful model
• Human = genes and evolution ?

27. Not a useful model
• Human = genes and evolution ?
• Distrust:
‘It can’t do X because its just trained to predict the next
word’

28. Model: Database + Reasoning Engine

29. Model: Database + Reasoning Engine
• The reasoning engine is key

30. Model: Database + Reasoning Engine
• The reasoning engine is key
• Often, the database is a liability

31. Reasoning capabilities

34. Model: Human intuition
Ask a human to answer a historical question
vs
Give them a history book and ask them the question

35. Note: Context window limited
• Thousands of words
• Can’t put a whole KB, or context, in it
• Synergizes well with Vector Search

36. Model:
‘Interpolative’ vs
‘Extrapolative’
tasks

38. • Less reliable at extrapolation

39. • Less reliable at extrapolation
• Favour interpolation

40. • Less reliable at extrapolation
• Favour interpolation
• Perform a task, given a context

41. • Less reliable at extrapolation
• Favour interpolation
• Perform a task, given a context
• ‘Retrieval Augmented Generation’

42. 30 November 2022:
ChatGPT

43. First features we built
• Summarisation
• Edit tone of voice
• Expand from shorthand

46. • 5th Dec: Starting
• 20th Dec: Internal use
• ~13th Jan: Customer beta
• 31st Jan: Launch with testimonials
• March: Thousands of customers
Timeline

47. • GPT-powered
question answering Bot
Fin

48. Design goals

49. Design goals
• Converse Naturally

50. Design goals
• Converse Naturally
• Answer questions about just your
business

51. Design goals
• Converse Naturally
• Answer questions about just your
business
• Reduce hallucinations

52. Design goals
• Converse Naturally
• Answer questions about just your
business
• Reduce hallucinations
• Minimal con
fi
guration

53. How to build with AI

54. Principles

55. • Sometimes you have to work back from the technology
AI

56. • Sometimes you have to work back from the technology
• Skunkworks approach
AI

57. • A small senior ML team owns technical core
• Engineers/Scientists help establish design envelope
Tactics

58. • A small senior ML team owns technical core
• Engineers/Scientists help establish design envelope
• Team must have quantitative skills
Tactics

59. • A small senior ML team owns technical core
• Engineers/Scientists help establish design envelope
• Team must have quantitative skills
• Must have the ability to ship code
Tactics

60. • Fast customer contact
Tactics

61. • Fast customer contact
• Build v1 of most ML with powerful LLM
Tactics

62. • Test with real data ASAP
Tactics

63. • Test with real data ASAP
• Design in failure tolerance
Tactics

64. Tactics

65. • How we build ML software has changed
• “LLMs aren’t all of AI”
Tactics

66. • Good adoption
• Working on v2
• CS will be hugely changed by AI
Fin current state

67. Guessing what’s next

68. • Breakneck progress

69. • Breakneck progress
• Smaller models, open?
• Exciting but overhyped today

70. • Breakneck progress
• Smaller models, open?
• Exciting but overhyped today
• Productisation

71. • Breakneck progress
• Smaller models, open?
• Exciting but overhyped today
• Productisation
• Larger models

72. Thank you!
@fergal_reid