STUK2007 articles: visits
Team D Synergy
This was our second day in Edinburgh. Yesterday, we walked around the city and went to some
scenic spots. Early this morning, we rode on a bus to reach our destination - the King’s Building
of the University Edinburgh.
(curved in PS, find the adjusted one in InDesign Source)
The Cricket Lab
When we arrived at the building, it was similar to buildings around the city - antiquated. We
first visited DrDr.. Barbara Webb’s Cricket Lab. She briefly introduced her lab to us. The lab is
conducting research that aims to understand the behaviour of insects, develop a computational
model of such behaviour, and mimic the behaviour in a robot. The behaviour of insects are
being replicated by robots that are built from electronic circuits and chips and these robots
will then be put through the same experiments as the insects in order to verify or fine-tune the
understanding of their behaviour.
STUK2007 articles: visits
Cricket Landmarking Model
We were split into groups and watched the research work. The first one was the Cricket
Landmarking Model. DrDr.. Jan Wessnitzer, the research associate of DrDr.. Webb, explained to
us the detail of this project. In the experiment, a cricket is placed in a very hot container with
several cool spots. It is to find out the learning behaviour of crickets. There were also
landmarks on the wall of the container aiding the crickets to recognize the locations of the
DrDr.. Jan Wessnitzer explaining the Landmarking experiment to all of us
Initially, the crickets randomly explored until they found the cool spot. It has been found that
the behaviour of crickets will be different and they take less time to locate the cool spots after
several trials. The relation among the landmarks and the behaviour is being studied. Crickets
have probably learned something so that they can find the cool spots quickly. The brain of an
insect is a distinct structure. The mushroom body is expected to be associated with memory, as
well as learning. The neuroarchitecture in the mushroom body is suited for pattern recognition.
A computational model is then constructed to replicate the behaviour as observed from the
crickets. According to Jan, the lives of crickets will be shortened after the experiment.
Ant Navigation Model
Next, Michael Mangan, one of the Ph.D. students, told us about the Ant Navigation Model. The
behaviour of a certain type of ant was being studied. The species being used was one living in a
hot environment such as a desert. Experiments have also been done on their navigation ability
in barren and crowded environments. Ants are able to navigate complicated paths and find
their way back home. The sun acts as a compass while the ants are looking for food and in a
desert environment, the ants use visual recognition rather than chemical recognition during
navigation. This is because chemicals disappear quickly in a hot environment. An un-intuitive
finding of this experiment is that each ant individually does navigation on its own and it is not a
collective behaviour. Robots are being built in their lab to mimic this unique behaviour.
Stick-insect Walking Model
Another research student, Hugo Rosano, presented us a Stick-insect Walking Model. He found
that the legs of stick insects are independent and communication between the legs is limited.
The neuro-system organizes the walking control. As each of the legs has 3 joints, if one of the
joints moves, altogether 18 joints have to move in a manner which is congruent to the
movement. When the joints experience a force, they follow the movement and form an
organized movement of the insect. The body motion relates to the legs. Such motion
mechanism can be modelled based on positive feedback controllers and explains the close
STUK2007 articles: visits
kinematics chain problem. For example, the front leg directs the body of the stick insect to the
target. All the legs move in the same direction but in fact only the front leg leads the way. The
mesothorax follows a specific line. The insect motion based on the model can be simulated by
Lastly, Dr. Webb introduced some other research on crickets and fruit flies.
Other Research on Crickets
Using the crickets, experiments are done in a "box" to observe how they behave in an
environment with different sounds, different wavelengths of sound, and even with different
visual environments. Then, a robot model is built to mimic the behaviour of the crickets. The
robot uses a logic circuit as the brain, a camera pointing at a mirror as the eye to provide a
360-degree view, and some wheels as legs. The robot is then put in the same environment as
the crickets to observe its behaviour. The parameters will be changed in the experiments; for
example, the period of the sound or the pitch of the sound. It is found that crickets will
perform different types of behaviour since each species has their own unique melodies and
they can recognize other species. Crickets are chosen for the research because the sound
localization in crickets is already well understood and "there is still a lot to learn from even
simple animals" as stated by DrDr.. Webb.
DrDr.. Webb showing Sharon the cricket used in the experiment
Dr. Webb showing us the cricket used in the experiment
Other Research on Flies
Using fruit flies, in an ongoing Ph.D.PhD work, the behaviour of flies under visual and chemical
stimuli are being studied. The flight of fruit flies is tracked three dimensionally by two
cameras. The visual surroundings are changed to study how fruit flies change their behaviour.
For example, they need to avoid walls when flying around. The visual stimulus causes them to
make a quick turn and move away from the walls. The study is then further combined with
chemical stimuli such as those when they are looking for fruit. It is interesting to make
different sensory systems interact with each other. Indeed, fruit flies are being modelled by a
robot arm which can move a camera around at different speeds and performs in the same way
Motion Capture for Game Simulation
After we visited DrDr.. Barbara Webb’s “Cricket Lab”, we followed DrDr.. Taku Komura to a
lecture room where he gave his presentation. Although the building is antiquated, the facilities
in the lecture room are modern.
The Fight Simulation
There were two projects that DrDr.. Komura presented to us. The first one was about the
generation of fight animation. The objective of this project is to generate realistic fighting
scenes for more than one avatar and emulate different styles of fighting. It can be applied to
STUK2007 articles: visits
two main areas such as pre-planning fighting scenes with multiple characters for movies and
generating fighting animation for video games.
DrDr.. Komura explained the idea of the project. There are some pre-processing steps. First,
different fight motions are captured with mocap, the optical motion capturer, to form a fight
motion table. Second, a game tree of fight motions between avatars is built for fight simulation
using temporal expansion. With the temporal expansion approach, the continuous nature of a
fight is converted into a discrete strategy planning problem. Hence, motion of fight can be
selected by fighters as options. When the simulation kicks off, fighters pick their motion option
from the game tree.
DrDr.. Komura showing us a demonstration clip of a Fight Simulation
Fighters can have their own personalities such as being smart, less-intelligent, energetic, tired,
defensive, aggressive, and even out-fighter and in-fighter. These personalities affect the
decision of picking the motion. For example, a smart fighter can go further down the tree to
figure out which motion is the most beneficial to him. On the other hand, human players can
assign high-level instructions to ask an avatar to perform particular motions. All in all, pre-
planning fighting scenes has become an easy task.
The Wrestling Simulation
The later presentation introduced an ongoing project which deals with tangling simulation
among virtual characters.
The objective of this project is to develop a topological representation between the body
segments of virtual characters during close interactions such as dancing and wrestling. It has
some applicable areas such as motion synthesis, path-planning and contents-based retrieval for
motions of more than two avatars and also, is useful for motion editing, re-targeting and real-
This time, the project models wrestle and the idea of the project follows. First of all, wrestling
motions are captured individually. Then, template postures are extracted from the captured
data. By having the template postures, a topological relationship for wrestling among
templates can be formed. After that, body segments on the templates are tangled together
under human instruction. The tangling relationship is further derived from the tangled
templates. Finally, elastic constraints are applied to the tangled body segment so that body
segments do not untangle, and physical-based animations of wrestling are generated. In fact,
the reason for wrestling motions being captured individually is that when capturing two persons
at the same time, it is difficult to figure out the entire body segments by cameras due to
(Please adjust more bright)http://pointer.hk/~stuk07/in_tour/album/Day4/Bob/jpg/IMGP1035.JPG
STUK2007 articles: visits
Prof. Fisher giving his lecture to all of ustelling us his research projects
When DrDr.. Taku Komura was finishing his presentation, Prof. Robert Fisher came into the
room. Prof. Fisher, a tall gentleman, was going to give us a "lecture" on 3D model
reconstruction and video analysis. He told us that he came to Hong Kong for a conference last
year and he was pretty familiar with downtown Hong Kong. Hence, he started to give us a
scenario to brainstorm a problem - how can we reconstruct a 3D model of Hong Kong Bank by
using images alone? This was the main goal of his first lecture - Virtual Model Completion.
Virtual Model Completion
When constructing a 3D virtual environment of a particular building, some pictures of the
building are captured for constructing the 3D model. However, it is not feasible to acquire
every picture of all angles of the building. This research is aimed to reconstruct an
approximate 3D model from 2.5D images. The idea comes from research which extends 2D
texture by filling in holes or areas with pixels. The algorithm copies pixels from the original
texture and placing those pixels onto unfilled areas. Those pixels are selected according to the
similarity of neighbouring pixels at the boundary. In 3D space, according to the lecture being
taught by Prof. Fisher, the 2.5D image is first fitted onto the surface of a selected 3D shape.
Then, the fitted surface forms a bump map and the map is further triangulated to find
corresponding vertices. After that, by following the idea of extending 2D texture as mentioned
above, the unfitted part of the shape can be reconstructed vertex by vertex. Those vertices
are selected from the triangulated surface based on their similarity to the neighbouring
boundary vertices. The distance from the targeting boundary vertex is also accounted for in the
computation. Finally, by repeating the copying of vertices, an approximate 3D model is
reconstructed from a 2.5D image.
Prof. Fisher explaining the Pisa Tower Model Reconstruction to usTyler making friend with Prof. Fisher
STUK2007 articles: visits
Prof. Fisher explaining the Model Reconstruction to us
After we enjoyed the lecture on virtual environment reconstruction, we were treated to
another excellent lecture given by Prof. Bob Fisher. This lecture was about analyzing video to
detect rare circumstances. This research was presented at the International Conference of
Pattern Recognition held in Hong Kong last year. We were grateful to have a "private"
presentation from Prof. Fisher.
The goal of the research is to build a system to find rare circumstances, like accidents or fights
in a crowd of people, from a pool of video tapes captured by surveillance cameras. Optical
flow and feature dimensionality reduction are used in the analysis model. The analysis observes
changes in clusters of pixels within a period of time so that rare changes can be detected.
Pixels in video frames contain millions of pieces of data. Therefore, some pre-processing of the
video data is carried out in order to extract the relevant data for analysis. In the pre-
processing stage, actual motion flow in video frames is extracted from the background. The
extracted flow is then transformed into optical flow vectors which represent the flow of pixels
in a vector space. After the pre-processing stage, video frames are segmented and then
clustered. Clusters of vectors are the ultimate target of being observed in each segment.
Principal Components Analysis is then applied onto those clusters to analyze changes in
different component dimensions of that particular segment. When the above stages are
completed, video with the normal situation of a crowd is first analyzed. Then, video with an
abnormal situation of a crowd is processed. It is shown that there is a distinct difference
between the analysis result of the normal situation and abnormal situation.
After the lecture, we went to have lunch in the canteen of the King's Building.
When we finished our lunch in the canteen, we went back to the lecture room for the next
presentation. Prof.DrDr. Sethu Vijayakumar, who is the director of the Institute of Perception,
Action and Behaviour, was already there waiting for us to give his presentation. The
presentation was related to one of his research areas - Statistical Machine Learning. And in his
presentation, he showed us how the technology can be applied to robots and how robots
benefit from it.
We enjoying a speech from Prof. Sethu Vijayakumar
Statistical Machine Learning
STUK2007 articles: visits
Machine learning can help to improve the movement of robots so that their movement can be
less stiff and have higher accuracy and faster speed. Machine learning can also help in the
study of statistics from previous movements and make predictions as to what should come next
and thus improve the next movement so that it can move smartly.
Reactions to various inputs can be refined or improved by changing the “Internal model” which
is actually a FeedForward Control implemented in the Control Policy. For human beings, we
learn how to control our arms by practising but for robots, they have to learn from statistics
and “Internal Models” can help a lot in this area.
Dynamic Systems-based Movement Policies
Also, Prof.DrDr. Vijayakumar talked about the Dynamic Systems-based Movement Policies which
is about how to select which path or which motion to reach a certain point. One method is
using trajectory following and Generalization. This method will capture the motion of a human
being and will ask the robot to follow the motion. In this way, the robot is taught by the
human. In machine learning, there are two types of system: open loop and closed loop systems.
Open Loop means there are no sensors in the system. The motions are prerecorded and
repeated. The robot will always give the same response no matter what the stimuli are. In a
Closed Loop system, the system is equipped with sensors. The sensors will read the data and
save them for statistical analysis. And the response by the robot will be based on the analysis
results. For some operations, both open loop and closed loop systems are used.
Visual Attention & Oculomotor Learning
The next topic in Prof.DrDr. Vijayakumar’s presentation was about Visual Attention &
Oculomotor Learning. Different from the previous talks, this topic is more focused on the
sensory aspect instead of movement. In Prof.DrDr. Vijayakumar’s presentation, he showed us
the MAVERic Project which is a versatile robotic vision head developed for oculo-motor
research. The robot head can perform like a human. It pays attention to moving objects and
can be distracted by other moving objects. We watched several videos on the experiment of
the robot head capture from its camera. The response is really fast and it acts like a human.
Final NoteDrive to Success
Although the hosts are carrying out research in different areas, they all share the same drive in
doing the research work. The drive is important in giving them energy to tackle problems and
generate ideas. What is it? The drive is - the passion for doing Computer Science.
A group photo with all of us in the lecture room
STUK2007 articles: visits
Q: What is the reason for the lab performing experiments on insects instead of other kinds of
A: Insects are small but already complex. It is a good step before we investigate other kinds of
creatures that are more complex.
Q: How do algorithms handle exceptional cases like a bird flying over a crowd which covers
some of the video frames or a car driving into the crowd and stopping?
A: These cases affect the background model of the optical flow, but a dynamic learning model
can be added to the algorithm in order to adjust the background model or just ignore those
minor abnormalities as the bird would just appear in a few video frames.
Q: Is there any application that works with the statistical learning model?
A: Not yet. The research work mainly shows that we have the capabilities to do learning in very
high dimensional complicated systems. The underlying science is going to work in building an
autonomous system that learns purely from data.
“There is still a lot to learn from even simple animals.” This sentence was said by DrDr.. Webb.
Indeed, this sentence is the reason why the Cricket Lab is carrying out research on insects and
mimicking the behaviour of the insects. Even though an insect is small, it is already complex.
Before mimicking the behaviour of complex creatures like human beings, it is an important step
to mimic something smaller. That’s a good idea we need to bear in mind. Go step by step!