The document outlines a course on multimedia signal processing (MMSP), emphasizing the study of communication through various media and the digital representation of signals. It discusses the importance of algorithms for processing media information, human-machine interaction, and the technological advancements in devices capable of sophisticated multimedia processing. The course aims to cover the integration of advanced interfaces and media systems while avoiding overlap with other specialized media processing courses.

1. MULTIMEDIA
SIGNAL PROCESSING
MMSP
SGN-5016
Irek Defée
Tietotalo TF 316
irek.defee@tut.fi

2. Course info
• Lectures: Room TB 214
Tue ja Wed 10.15-12
• Exercises mandatory
• Exam written

3. Course info
• Course Web page
http:/www.cs.tut.fi/~defee/mulsp.html
• Course material is regulary updated,
please use only the updated material

4. Petri Hirvonen
petri.hirvonen@tut.fi
http://www.cs.tut.fi/~hirvone2/5016_exercises.htm
Exercises for SGN-5016
Multimedia Signal Processing

5. Exercises
• TC303
• Group1: 8:15-10:00, TC 303 28.10
• Group2: 8:15-10:00, TC 303 29.10
• You can participate in one or both of the exercise groups
if there is space, is not attend one group
• A written report is returned by e-mail after each exercise.
• The details about the report are included in the exercise
material.

6. WHAT IS THIS COURSE ABOUT???
1. WHAT IS MULTIMEDIA (MM) ?
2. WHAT IS THE TOPIC OF MULTIMEDIA
SIGNAL PROCESSING?
(THIS AREA IS NOT WELL DEFINED YET)
MULTIMEDIA SIGNAL PROCESSING

7. WHAT IS MULTIMEDIA?
• COMPOSED OF MULTI+MEDIA
MEDIA = MEDIUM OF COMMUNICATION
WE COMMUNICATE NATURALLY:
VISUALLY, BY SPEECH, BY TOUCH…
WE COMMUNICATE BY TECHNOLOGY:
RADIO (MOBILE PHONES), TV, PRESS,
CINEMA, BOOKS

8. • PEOPLE USE VARIOUSCOMMUNICATION
MEDIA: SPEECH, VISION, TOUCH….
IN THE PAST WHEN PEOPLE
COMMUNICATED THEY HAD TO USE
THOSE MEDIA DIRECTLY.
IN PRESENT CIVILISATION THERE ARE
MANY TECHNOLOGIES WHICH
EXTEND HUMAN COMMUNICATION

9. PRODUCER
OF
INFORMATION
HUMAN
RECEIVER
OF
INFORMATION
HUMAN
COMMUNICATION MEDIUM NATURAL
(E.G. VOICE, TOUCH): WE USE SPECIFIC
PHYSICAL MEDIUM E.G. AIR PLUS PRODUCTION
SPECIALLY ENCODED SIGNALS FOR CONVEYING
INFORMATION
COMMUNICATION MEDIUM INDIRECT VIA
TECHNOLOGY (E.G. CINEMA, RADIO, PRESS, TV)
GENERAL MODEL OF HUMAN COMMUNICATION

10. • MORE RECENT IS A MODEL OF
HUMAN – MACHINE
COMMUNICATION, OR EVEN
MACHINE-MACHINE COMMUNICATION
WHEN WE USE COMPUTERS, WE
COMMUNICATE WITH MACHINE,
THE COMMUNICATION MEDIA ARE:
TOUCH/GESTURE <-> KEYBOARD, MOUSE
VISION <-> DISPLAY
HEARING <-> SOUND

11. • HUMANS CAN USE SEVERAL DIFFERENT
MEDIA FOR COMMUNICATION
E.G. SPEECH, TOUCH, VISUAL SYSTEM
HUMANS OFTEN USE SEVERAL
MEDIA SIMULTANEOUSLY OR IN OTHER
WORDS MULTIPLE MEDIA =MULTIMEDIA
FOR EXAMPLE: WHEN WE TALK WITH
SOMEBODY WE USE GESTURES, FACE
EXPRESSIONS

12. • IN FACT PEOPLE PREFER TO USE
MULTIPLE MEDIA = MULTIMEDIA
- WE CAN USE SINGLE MEDIA, E.G. SPEECH
WHEN TALKING ON THE PHONE
BUT SEEING EACH OTHER WHEN
TALKING ”ENHANCES” THE CONTACT
- WE CAN LISTEN TO THE RADIO, E.G.
NEWS, BUT TV IS PREFERRED EVEN IF WE
JUST SEE A PERSON READING THE NEWS
- MULTIMEDIA IS MORE NATURAL FOR
PEOPLE

13. • THERE IS ANOTHER USE OF WORD
”MEDIA”, IN THE SENSE OF
MEDIA INDUSTRY
MEDIA INDUSTRY IS DEALING WITH
PRODUCING, DISTRIBUTING AND SELLING
INFORMATION ADDRESSING HUMAN
MEDIA SYSTEM
MULTIMEDIA INFORMATION IS VERY
IMPORTANT FOR THE INDUSTRY
THERE ARE MANY ENGINEERING
PROBLEMS IN DEALING WITH
MULTIMEDIA INFORMATION

14. • WHAT IS MULTIMEDIA SIGNAL
PROCESSING (MMSP) ?
IT IS ABOUT PROCESSING
COMMUNICATION AND UTILIZATION
OF INFORMATION USED BY HUMANS
ONE CAN CONSIDER THREE
SCENARIOS OF USAGE:
1. HUMAN-HUMAN
2. HUMAN – MACHINE
3. MACHINE - MACHINE

15. WHY MULTIMEDIA SIGNAL PROCESSING
IS POSSIBLE? THIS IS BECAUSE WE HAVE
MEANS FOR DIGITAL REPRESENTATION
AND PROCESSING OF ANY TYPE OF
INFORMATION.
IF WE TALK ON THE PHONE, LISTEN TO
THE MUSIC FROM MP3PLAYER, WATCH
MOVIE FROM DVD DISC, TAKE PICTURE
WITH CAMERA, WE KNOW THAT
INFORMATION IS REPRESENTED BY BITS
AND PROCESSED DIGITALLY

16. WHAT WE NEED ARE ALGORITHMS
HOW TO PROCESS THE SIGNALS
DIGITALLY
MULTIMEDIA SIGNAL PROCESSING
IS ABOUT ALGORITHMS FOR THE
PROCESSING OF SIGNALS WHICH ARE
USED BY HUMANS FOR COMMUNICATION
WITH OTHER PEOPLE OR MACHINES OR
DEALING WITH THE WORLD AROUND

17. • WHAT ARE THE MEDIA SIGNALS?
MEDIA SIGNALS ARE THOSE SIGNALS
WHICH ARE ACCESSIBLE TO THE HUMAN
INFORMATION PROCESSING SYSTEM
ONE OF THE ISSUES IN MULTIMEDIA
SIGNAL PROCESSING IS WHAT TYPE OF
SIGNALS AND WHAT KIND OF
COMBINATIONS OF SIGNALS CAN BE USED.
FOR EXAMPLE: ACOUSTICAL SIGNALS:
SOUNDS, SPEECH-LANGUAGE, MUSIC
WE CONVERT THOSE SIGNALS TO DIGITAL
FORMAT AND USE

18. • EXAMPLE: DIGITAL MUSIC (CD, MP3, DVD,
INTERNET RADIO)
• EXAMPLE: DIGITAL VIDEO (DVD, BLUE RAY,
INTERNET TV)
THESE ARE SYSTEMS FOR TRANSFERRING
CONTENT PRODUCED BY ARTISTS TO
PEOPLE. THESE SYSTEMS USE SPECIFIC
DIGITAL ENCODING AND COMPRESSION
OF INFORMATION TO RECORD THE
CONTENT.
THE QUESTION IS HOW TO MAKE THIS.

19. BUT HAVING SUCH SYSTEMS A NEW
PROBLEM EMERGES:
HOW TO PROTECT MEDIA INFORMATION
UNAUTHORIZED USE?
(FOR EXAMPLE ILLEGAL COPYING?)
How to represent media information in
most pleasing way?
Examples are High Definition technologies:
- Flat Displays
- HD DVD, Blue Ray discs, HDTV

20. • THE SECOND MAIN ASPECT OF MMSP
2. HUMAN-MACHINE COMMUNICATION
HOW TO MAKE INTERACTION WITH
COMPUTERS (AND OTHER MACHINES)
MORE NATURAL? NATURAL MEANS E.G. MORE
SIMILAR TO HUMAN-HUMAN INTERACTION,
MORE INTUITIVE, MORE PLEASING,
ATTRACTIVE….

21. THAT INCLUDE ALSO HOW TO MAKE
MACHINES MORE INTELLIGENT:
• FOR EXAMPLE , INSTEAD OF TYPING WE
COULD TALK TO COMPUTERS AND
INSTEAD OF COMPUTERS PRINTING ON
SCREEN ANSWERS THEY WOULD TALK
TO US.
OR, IF COMPUTERS WOULD SEE US
USING CAMERAS, THEY POSSIBLY
COULD REACT MORE LIKE PEOPLE.
BUT TODAY WE STILL USE KEYBOARD
AND MOUSE, WHY?

22. • WE USE KEYBOARD AND MOUSE
BECAUSE WE DO NOT HAVE BETTER
TECHNOLOGY: WE DO NOT KNOW HOW TO
PROCESS SPEECH AND VISUAL
INFORMATION AS EFFECTIVELY AS
PEOPLE ARE ABLE TO DO
• BUT WE MAY THINK OF COMPUTERS WITH
CAMERAS AND MICROPHONES
WHICH WILL BE ABLE TO DO SO
• THIS MAY BECOME POSSIBLE BECAUSE
OF FAST PROGRESS IN DEVELOPMENT OF
ALGORITHMS AND PROCESSORS

23. • THIS PROGRESS CAN BE ILLUSTRATED ON
MANY EXAMPLES
- COMPARE PC TODAY AND 10 years AGO
(TODAY WE HAVE MULTICORE
PROCESSORS AND THE NUMBER OF CORES
IS GROWING FAST)
- COMPARE MOBILE DEVICE TODAY AND
MOBILE PHONE 10 years AGO
(TODAY THE TELEPHONE FUNCTION IS
JUST ONE ADDITION TO MULTIPLE MEDIA
PROCESSING: MUSIC, VIDEO, CAMERA,
TOUCH, ORIENTATION)
EXTRAPOLATE THIS TO THE NEXT 10 years!

24. WE CAN EXPECT IN THE FUTURE:
• COMPUTERS, MOBILE, AND ALL KIND
OF OTHER DEVICES WILL BE MORE AND
MORE CLEVER (=INTELLIGENT?)
• THESE SYSTEMS WILL BE RELYING
ON INCREASINGLY SOPHISTICATED
MULTIMEDIA SIGNAL PROCESSING
CAPABILITIES

25. • WE HAVE THUS TWO MAIN AREAS TO
COVER IN MMSP:
1. MEDIA INFORMATION PROCESSING
IN MULTIMEDIA SYSTEMS
2. MEDIA COMPUTER INTERFACE FOR
HUMAN-COMPUTER INTERACTION
THESE ARE THE TOPICS OF
THE MMSP COURSE

26. • Please note however that our Multimedia Signal
Processing course is matched to the study program
at TUT, especially to the Multimedia Major
• We have many courses specialized in single media
processing: Digital Audio, Image Processing, Video
Processing, Video Compression, Pattern
Recognition
• We avoid overlapping with those courses. We are
also not going into algorithms which were proposed
by researchers but they are not in wider use yet,
this is covered in other courses and seminars
• In other universities they may not have so many
specialized courses, the course content is different

27. • There is one absolutely basic observation:
• MANY MULTIMEDIA SIGNAL PROCESSING
TASKS ARE ALREADY IMPLEMENTED IN
BIOLOGICAL SYSTEMS, ESPECIALLY IN
THE HUMAN INFORMATION PROCESSING
SYSTEM
• FOR EXAMPLE: VISUAL AND ACOUSTICAL
COMMUNICATION BETWEEN PEOPLE,
USING VISUAL INFORMATION IN
RECOGINIZING OBJECTS. BIOLOGICAL
SYSTEMS DO IT PERFECTLY BUT WE DO
NOT KNOW HOW, THAT IS ALGORITHMS

28. IN THE FIRST PART OF THIS COURSE
WE SHALL COVER BASIC KNOWLEDGE
RELATED TO
HUMAN INFORMATION PROCESSING
THIS SYSTEM PROCESSESS MEDIA
INFORMATION AND IT DOES IT IN
FANTASTIC WAY. IF WOULD KNOW HOW
IT MAKES IT, IT COULD HELP US TO
MAKE BETTER MEDIA INFORMATION
PROCESSING (BETTER MMSP ALGORITHMS)

29. BUT BEFORE WE GO FURTHER LET US MAKE
SOME MEDIA TECHNOLOGY OVERVIEW,
WHERE MULTIMEDIA SIGNAL PROCESSING
WILL BE USEFUL IN THE FUTURE

30. MULTIMEDIA SIGNAL PROCESSING
ALLOWS FOR NEW CLASSESS OF DEVICES
AND SYSTEMS:
MORE SOPHISTICATED COMMUNICATION,
MORE ADVANCED INTERFACES
THEY ARE ILLUSTRATED NEXT

31. Mobile Multimedia Devices Examples

32. WHAT THESE MOBILE DEVICE EXAMPLES
SHOW TO US?
-DEVICES HAVE MULTIPLE SENSORS AND
MULITPLE MEDIA PROCESSING CAPABILITIES
- TAKE ONE EXAMPLE - TOUCH
Device is controlled by fingers, e.g. picture size
or even playing guitar

33. What is still missing?
Maybe makeup, but this is a joke 

34. ANOTHER EXAMPLE: DIGITAL CAMERAS
Digital cameras perform a lot of processing
for best picture quality. But recent cameras
have new features related to analysis of
visual information.
Face Detection automatically detects a face in the frame and
adjusts focus, exposure, contrast, and skin complexion so it
turns out perfectly.
Face Recognition – a feature that “remembers” faces from
previous shots. When a familiar face is recorded several times,
the camera will prompt the users to register the face. Once
registered, if the face appears into the frame again, the camera
will display the name specified for that person and prioritize
focus and exposure for the face.
To make such feature an algorithm for
face detection and recognition is needed
working fast and reliably

35. COMPLETELY NEW TYPES OF DEVICES ARE
POSSIBLE: EXAMPLE Wii
Wii by Nintendo
Contollers have
motion sensors
Game & fitness accessories
Dancing pad Balance board
Sports game Music performance

36. AIBO DOG – PERSONAL ROBOT WITH SENSES
Completely New Types of Devices

37. IT HAS SENSES:
MICROPHONE,
CAMERA, TEMPERATURE,
DISTANCE, ACCELERATION,
BALANCE, TOUCH
IT HAS INSTINCTS
AND BEHAVIORS

38. "Is this a real cat?" A
robot cat you can bond
with like a real pet --
NeCoRo is born
Completely New Types of Devices

39. Omron ready to test demand for robo-cat

40. Equipped with Omron's proprietary MaC (Mind and
Consciousness) technology, feelings are generated
according to recognition feedback, which is dependent
on configurations based on psychological concepts,
leading to cognitive decisions and actions determined by
these feelings (applicable patent acquired)
Feelings of satisfaction, anger, and uneasiness generated
based on recognition feedback
Desires to sleep or be cuddled
generated according to
physiological rhythms
Via a learning function, personality traits
such as selfishness and the need for
attention will change in response to the
owner

41. PERSONAL ROBOTS
START APPEARING ...

42. Fujitsu has developed a new miniature
humanoid robot, named HOAP-1,
designed for wide application in research
and development of robotic technologies.
Fujitsu Automation will begin domestic
sales of the robot from today and hopes to
sell 100 units within three years.
Weighing 6kg and standing 48cm tall, the
light and compact HOAP-1 and
accompanying simulation software can be
used for developing motion control
algorithms in such areas as two-legged
walking, as well as in research on human-
to-robot communication interfaces.
The basic simulation software and user-
developed programs are designed to run
on RT-Linux on an operating command
PC, which communicates with the the
robot through a USB interface. The robot's
internal sensors and actuators (motors)
also use USB interface and can be easily
expanded according to needs

43. The two-legged walking
technology developed by
Honda represents a unique
approach to the challenge of
autonomous locomotion. Using
the know-how gained from
these prototypes, research and
development began on new
technology for actual use.
ASIMO represents the fruition
of this pursuit.

44. Menagerie of devicesDEVICES

45. • Progress of technology is fast: Even the old
television is changing, in 2010 a three
dimensional television, 3D TV, will start
3D TV set
Glasses
And also a first TV controlled
by hand gestures will be
available (but very expensive)

46. What we see from these examples?
• We can see that devices are developing to have
- More complexity
- More intelligence
- More natural interaction with people
To add even more such features one needs
algorithms for multimedia signal processing,
many of these algorithms should have
capabilities similar to biological systems.