The document outlines the syllabus for a course on knowledge-based multimedia systems (KBMS). It discusses the following key points: 1. The course covers various topics on modeling different media like text, images, video, and audio in a KBMS. 2. Students will complete assignments including exams and a semester-long project to design a location-based multimedia system. 3. The project involves collecting and analyzing location-based media like audio, text, and photos to produce system outputs and user interfaces.

1. KBMS - Intro
Frank Nack
HCS
ILPS

2. Outline
 Organisation
 Intro - Systems
ILPS Frank Nack nack@uva.nl KBMS 2

3. Organisation
 Meetings
 Material
 Assignments
 Evaluation, passing and grades
 Time allocation
 Schedule
ILPS Frank Nack nack@uva.nl KBMS 3

4. Meetings
Lectures
Date: Mon 09:00 - 11:00
Location: SP G0.05 Week 36 - 41
SP D1.160 Week 42
SP D1.112 Week 44 - 49
SP G2.13 Week 50
Project
Date Tues 09:00 - 13:00
Location SP D1.110 Week 36 - 41 and 44 - 49
SP D1.112 Week 42
SP D.160 Week 50
ILPS Frank Nack – HCS nack@uva.nl KBMS 4

5. Materials
 Literature on Blackboard (see Course Material)
 Slides on Blackboard (see Course Material)
 Additional links
ILPS Frank Nack nack@uva.nl KBMS 5

6. Assignments
Exams
2 exams, each 90 minutes
Project
The overall objective is to design a platform independent knowledge media system
that uses multiple existing location centric API feeds and allows users:
1. to contribute audio, text or photo information to the system based specifically
to their location
2. to retrieve intelligent information from the system about an area based on the
above input
At the end of the course each group has to provide the following deliverables (adjust
to the scenario you work on):
A report of not more than 15 pages that contains:
1. all data generated (logically formatted)
2. an analysis of the data with conclusions
3. a series of static screen designs (UI) that explain how your system works and an
explanation of why this is the 'optimal' solution
4. a machine processable system (architecture, data structures, algorithms) that
incorporates data to produce the output displayed in your UIs
A presentation of 20 minutes about the groups work at the end of the course.
ILPS Frank Nack nack@uva.nl KBMS 6

7. Evaluation, passing and grates
Exams
The exam part counts 30% of the final mark.
Each exam contributes 50% to this part.
Project
The project part counts 70% of the final mark.
The report counts 80% and the presentation 20% for
this part.
You need 55% in total to pass
ILPS Frank Nack nack@uva.nl KBMS 7

8. Time allocation
 Approximately 9.5 hours per week (6 ECTS)
 2 hours lecture
 3 hours practical work
 4.5 hours reading
ILPS Frank Nack nack@uva.nl KBMS 8

9. Schedule
Study week Demo,
Intro Part 1 Part 2 and Part 3 Part 4 Part 5 Study week,
Examen 1 Examen 2
 Intro 06.09 Intro - Systems
13.09 Intro - Knowledge
 Part 1 20.09 Text and Image
27.09 Text and Image - application + Project description
 Part 2 04.10 Video
11.10 Video – Application
Part 3 12.10 Audio
Study week 18 - 22. 10
Exam 29.10
08.11 Audio – application + First draft of report
 Part 4: 15.11 Biometry
22.11 Biometry – application
 Part 5: 29.11 Ambience
06.12 Ambience - application
Presentation 07.12
Study week 13 - 17. 12
Exam 2 22. 12 + Report
ILPS Frank Nack nack@uva.nl KBMS 9

10. Intro – Systems, Senses and Communication
ILPS Frank Nack nack@uva.nl KBMS 10

11. Intro - Systems
Interactive information spaces
mausoleum of information
versus
space of ideas and interaction
ILPS Frank Nack nack@uva.nl KBMS 11

12. Intro - Systems
The responsive room
the real
versus
the virtual
ILPS Frank Nack nack@uva.nl KBMS 12

13. Intro - Systems
The creative system
support
versus
create
ILPS Frank Nack nack@uva.nl KBMS 13

14. Intro - Senses
Vision is the ability of the brain and eye to detect
electromagnetic waves within the visible range (light)
interpreting the image as "sight."
Audition is the sense of sound perception in
response to changes in the pressure exerted by
atmospheric particles within a range of 20 to 22000
Hz.
Tactition is the sense of pressure perception,
generally in the skin.
Equilibrioception is the perception of balance or
acceleration and is mainly related to cavities
containing fluid in the inner ear
Gustation is one of the two main "chemical" senses,
where four well-known receptors on the tongue
detect sweet, salt, sour, and bitter.
Olfaction is the other "chemical" sense. Unlike taste,
there are hundreds of olfactory receptors, each
binding to a particular molecular feature.
ILPS Frank Nack nack@uva.nl KBMS 14

15. Intro - Communication
Organisation and adaptation
(processes)‫‏‬
Consumption Interaction
ILPS Frank Nack nack@uva.nl KBMS 15

16. Communication - Types
Dialogue or verbal communication
A dialogue is a reciprocal conversation between two or more entities.
Nonverbal communication
Nonverbal communication is the process of communicating through sending
and receiving wordless messages.
Examples: gesture, body language or posture, clothing, hairstyles, etc.
Nonverbal elements in speech: voice quality, emotion and speaking style,
rhythm, intonation or stress.
Nonverbal elements in text: handwriting style, spatial arrangement of
words, emoticons.
Visual communication
Visual communication makes use of visual aids.
Examples: typography, drawing, graphic design, illustration, colour, etc.
ILPS Frank Nack nack@uva.nl KBMS 16

17. Communication – exchange of symbols
Communication
• is a process of transferring information from one entity to another
• is sign-mediated interaction between at least two agents
• both agents share a repertoire of signs and semiotic rules.
p c p p c
Reality Description
Sign Repertoires
p = perceive
c = conceive
ILPS

18. The Sign - Saussure
Concept
Mental Perception
of Media
SIGN
beauty
Signifier Signified
ILPS 18

19. The Sign - Peirce
psychological or ontological status?
Interpretant active process
(thought)
SIGN
Representamen Object
(symbol) (referent)
physical or referred to on a particular occasion?
mental entity? typical or ideal representation?
ILPS Frank Nack nack@uva.nl KBMS
19

20. The Sign - Arbitrariness
The Saussurean model supports the notion of
arbitrariness of the sign by proposing the
autonomy of language in relation to reality. Its
emphasis on internal structures within a sign
system assumes that language does not
“reflect” reality but rather constructs it.
Conventional in the Saussurean sense
means that the relationship between the
signifier and the signified dependents on
social and cultural conventions.
ILPS Frank Nack nack@uva.nl KBMS
20

21. Semantics – Index, Icon Symbol (Peirce)
Icon A sign which represents its object
mainly through its similarity with some
properties of the object, based on the
reproduction of perceptual conditions.
Index A sign which represents its object by
an inherent relationship.
Symbol A sign with an arbitrary link to its object
(the representation is based on
convention).
ILPS Frank Nack nack@uva.nl KBMS

22. Different Media – Different Symbols
Text Image Video Audio Tactile
ILPS Frank Nack nack@uva.nl KBMS

23. Intro – summary
 The key concepts with respect to modelling in
KBMS are
 context
 interaction
 adaptation
 Different media require different modelling
approaches
ILPS Frank Nack nack@uva.nl KBMS 23

24. Intro – References
Valle, A., Lombardo, V., and Vogel, H. (2007). Alternating from 1 to x and vice versa. In Proceedings of the
15th international Conference on Multimedia (ACM MM 07), , pp. 922-931, Augsburg, Germany, September
25 - 29, 2007
The artistic work with robots by Leonel Moura: http://www.leonelmoura.com/
ILPS Frank Nack nack@uva.nl KBMS 24