This document summarizes a research paper on human activity detection using edge point movements and spatiotemporal features from video recorded in indoor environments. The proposed method first extracts foreground objects from video frames using background subtraction. It then analyzes object shapes and extracts edge points using edge detection. Edge points across frames are stored in a database and compared to detect changes indicating activities. Activities are recognized by matching to stored patterns, or adding new patterns if unknown. In addition to movements, spatiotemporal features of location and time are considered to better detect activities as normal or abnormal. The goal is to develop a simple and accurate system for activity recognition to assist elderly or disabled people living independently.

1. Proceedings of the International Conference on Emerging Trends in Engineering and Management (ICETEM14)
30 – 31, December 2014, Ernakulam, India
64
HUMAN ACTIVITY DETECTION BASED ON EDGE
POINT MOVEMENTS AND SPATIO-TEMPORAL
FEATURES IN INDOOR
Lakshmi Priya K M1
, Smitha Suresh2
Final Year MTech, Dept of CSE, Sree Narayana Gurukulam College of engineering, Kerala, India
Asso.Prof. Dept of CSE, Sree Narayana Gurukulam College of engineering, Kerala, India
ABSTRACT
As the population of elder people increasing rapidly, the need for the health care systems is also increasing. To
provide a better health care and improving the quality of living of elder/disabled persons who resides independently,
‘Activity Recognition’ systems can used. The daily activities of the persons and the abnormal activities can be detected
using the activity recognition systems. There are so many methods exist today for this purpose. The sensor based
methods and video data based systems are the existing techniques. Often these systems are complex and not guaranteed
the 100% activity detection rate. To overcome the cost and complexity of the existing methods, here proposes a new
system with simple mechanisms and guaranteed a better performance. The input to the system is the visual data, taken
from multiple cameras placed in the home/indoor. Then the foreground objects should be extracted using adaptive
background subtraction method, that method reduces the amount of noise. Then the shapes of the objects and human
should be analysed using shape analysis method. Next the edge points from the image of human and interacting objects
should be extracted using canny edge detector. Then the edge points of human and interacting object at each second
should be stored in database and detected the changes occurring in every second. Then compare the each input with
already stored patterns. As matches found the activity should be recognized. Otherwise, the pattern database should be
updated with new activity by adding a new threshold for new activity. The activities are detected not purely by
considering the movements. The place which activity taken place and the time it happens (i.e. spatio-temporal features) is
also important factors for detecting an activity. Abnormal activities such as fall, change in medicine intake, etc... will
also be detected and a false alarm system is provided. The each activity is representing using unique thresholds. Changes
in normal activities may produce different threshold values.
Keywords: Background Subtraction, Edge Point Detection, Shape Analysis, Spatio-Temporal Features.
1. INTRODUCTION
As the increasing population of older people and disabled ones in the society, the need of assistive systems are
increasing day by day. Most of the people need independency and like to reside independently in homes, so a better
technical solution to give assistance to them were the activity recognition systems. Through a camera assisted
environment this can be carried out simply. The multiple camera environments give the better detection of each activity.
The activities include the normal daily activity and the abnormal activities. Those two become detected by the
proposed technique. The early techniques like accelerometers [1]-[3], help buttons [4], wearable sensors, etc are also
used by people. But now the technology was developed and the better ways by which the activity detection are emerged.
INTERNATIONAL JOURNAL OF COMPUTER ENGINEERING &
TECHNOLOGY (IJCET)
ISSN 0976 – 6367(Print)
ISSN 0976 – 6375(Online)
Volume 5, Issue 12, December (2014), pp. 64-69
© IAEME: www.iaeme.com/IJCET.asp
Journal Impact Factor (2014): 8.5328 (Calculated by GISI)
www.jifactor.com
IJCET
© I A E M E

2. Proceedings of the International Conference on Emerging Trends in Engineering and Management (ICETEM14)
30 – 31, December 2014, Ernakulam, India
65
The 2D image is the input to the system and this was obtained from the video data collected by the cameras placed in the
indoor homes. The images at every 5secs are examined and the change in the shape of the person will be detected. If the
change in shape leads to an abnormal activity, then the alarm is rang and if it leads to any daily normal activity then the
activity should be detected. This detection system can be applicable at hospital rooms, especially for, patients reside
alone without any helper and at home environments. This will allow a great accuracy rate at the employed environments.
The edge point detection [5] and the spatio-temporal features play a vital role in the activity detection. The edge
point extraction is done for the foreground image that is obtained from the background subtraction. Also the place where
activity happens and the time at which activity happens decides the activity detection, whether it is normal or not. The
problems like battery recharging and uncomfortability while using wearable devices are not here. Also the simple
mechanisms are used with less overhead. While on a system crash situation, anyone without the expertise knowledge can
understand the problem and sometimes may recover them.
2. RELATED WORKS
In some related works, e.g.[6] the human action recognition can be viewed as a process of detecting the actions
of the individual persons. The input data set is collected from depth sensors (such as Microsoft kinect). Training was
given to both indoor and outdoor activities. Using skeleton joints, body position, motion and velocity information
features the activities are modelled. Then a multiclass SVM is used for classifying the dataset. The main merits are,
relatively large size of data set taken in multiple views can handle, 100% accurate in single person one activity, two
person interaction and single person perform two activities. One of the drawbacks is, only person dependent parameters
are considered, not only location/time related recognition.
The work by Dipak Surie, Saeed Partonia [7], and the human identification is done for some security purposes
in smart spaces. The input is the RGB image, acquired from the kinect. Then this is used for the face recognition using
some features and skeletal tracking. Then the information fusion from several sources is done. Each person’s identities
are stored in database earlier. Some advantages of this paper are, security system through face recognition, helps in
implementing smart homes. Some disadvantages are, not much / 100% accuracy rate, face detection should be difficult
because of large representational data set.
In the paper [8] an application of fuzzy set technique. The fall detection and the fall risk assessments are done
here. The Microsoft kinect camera system as well as sensors is used for activity segmentation during day time as well as
night time. Three image sensors are used here standard web camera under visible lighting, web cameras with IR
illumination, and kinect sensors. Some merits are, three sensors were used, so more accuracy is there, Day and night
recognition is done. Some demerits are, single camera used, fuzzy logic is difficult to implement, some activities can’t be
distinguished.
The paper [9] proposes a multiple 3D camera based human tracking method that is robust to illumination
changes and occlusion at indoor environment. Here the several image features such as collaboration intensity, hue, local
binary pattern (LBP) and depth from 3D camera are considered. The first step considered in this paper is a background
subtraction method, which is adaptive Gaussian mixture model, then the human identification, then integration of vertical
axes, and at last adaptive particle filter. Can used in different illuminations and robust to partial and complete occlusions.
But more costly and more calculations and methods are required.
The work [10] the realistic human action recognition through video based on spatio temporal interest points
(STIP’s). The existing system described here is based on spatio-temporal approach and operates on intensity
representation of image data. So these approaches are sensitive to shadow and highlights. Here the colour STIP’s are
used for recognition of human actions in different challenging areas. Mainly the challenging UCF sports counterparts are
recognized. Different UCF datasets are considered. Mainly the multi channel harris stip’s and multi channel gabor stip’s
are also used here for stip’s detection. This deal with large and small datasets and better representations are formed. For
more difficult/ complex data the performance should be less. And the robustness becomes an issue (i.e. more robustness).
A work by Bingbing Ni, Yong Pei, Pierre Moulin [11], combines the data from conventional camera and depth
sensors (e.g., Microsoft Kinect). Proposes a activity recognition by fuses the data from both the gray scale and depth
image channels at multiple levels of the video processing pipeline. The false detections can be avoided by using this
method. The 3D spatial and temporal contexts of objects and human are extracted here. The depth information’s used to
distinguish the different indoor activities. Accurate 3-D spatial and temporal interaction contextual modelling is possible
and High-detection accuracy for complex activity and interaction. If the tracklet extraction parameter value on final
action detection performance is higher the result is unreliable tracking. It affects final action recognition.
3. ACTIVITY CHARACTERISTICS
For the proper designing of the system, first of all understanding of the different types of the activities should
important. Activities are majorly classified in this paper as,

3. Proceedings of the International Conference on Emerging Trends in Engineering and Management (ICETEM14)
30 – 31, December 2014, Ernakulam, India
66
3.1 Normal Daily activities
These are the activities which are usually done by the person. They have already data models stored in the
database. By simply comparing the activities data models with the input visuals and considering the location, time it
happens will detects the normal activities.
3.2 Unusual Activities (new activities)
The new activities are not the usual activities. That is not done by the person regularly. Such new activities have
no data models resides in the database. On the other hand, they have to store the models by themselves and forms
corresponding threshold to recognize each activity. At the learning time the system admin can recorded the threshold as a
new activity. At the next time the activity happens then it have data models and the system will recognize it.
3.3 Abnormal activities
Abnormal activities are the activities which causes any harm to the person. Examples are falls, changes in
medicine intake, unconscious moments. The abnormal activities are detected as well as a false alarm signal is sent to the
corresponding person or the nearby hospital etc.
The every activity is recognized mainly by considering the 3 factors. The changes occurring in the person’s edge
point movements are the initial factor to be considered. Then the location where the activity occurs and the time at which
the activity occurs are the deciding factors. Upon these 3 factors the nature of the activity should be recognized.
4. METHOD OVERVIEW
The system working is a simple process, were video data is taken as the input. The video data is taken and
converted into frames at each 5 seconds. The images are then fed into the system and these become the actual input data.
Then the system work within 6 major steps. Background subtraction, shape analysis, edge point extraction, learning new
activity, spatio-temporal feature analysis and checking activity is normal or not.
First of all the video data collected from the indoor cameras. Then those video will be converted into images at
every 5seconds. Those images one by one will be the input to the system. There after the actual process is started.
4.1 Background subtraction
Background subtraction or foreground detection is one of the major step in the image processing. Here the
images one by one should be fed as the input. Using the adaptive background subtraction method [12] the foreground
detection should be done here. This background subtraction gives a silhouette image of the foreground objects, which are
always the interesting points of the image. A probabilistic approach is used in this method.
By using this method the noise level can be reduced. The foreground objects in the scene can be detected by this
method. This object data is the key for the activity recognition system. By using this key data about the person and the
interacting objects the activity detection can be made smoother. The general equation used for the background
subtraction is:
|I(x,y) – B(x,y)| > Th (1)
Where, I(x,y) is the input image pixel, B(x,y) is the background image pixel, Th is the threshold value. If the
equation is true then, the image pixel should be a foreground pixel. Otherwise it is a background pixel. The output of this
level should be the input of the next step.
Fig.1: functional block diagram

4. Proceedings of the International Conference on Emerging Trends in Engineering and Management (ICETEM14)
30 – 31, December 2014, Ernakulam, India
67
4.2 Shape analysis
The shape analysis is the 2nd stage in the activity detection system. Shape of the human can be recognized using
different methods. But here a database based approach is adapted. Already stored data models are compared with the
foreground images and thus the shape of human and the interacting objects are find out. The already stored foreground
data models of the human and the foreground models/ patterns of most probably interacting objects are the basic factors
of this stage.
The foreground images coming from the first step is taken and compares the each image with the data base
models. Thus the presence of the human in the image and the interacting objects are identified. This step involves the
concept of data base and pattern matching. At the end of this step, only the foreground image of the human and the
interacting object should be maintained/ considered for the further stages.
4.3 Edge point extraction
Then the edge point extraction is carried out. The edge points are extracted using the canny edge detector [13].
Using this technique, only the relevant edge points are figure out from the previous step’s output. The edge point
extraction is mainly done for the easier representation of the movements of the human and comparing the consecutive
images for activity recognition. Only select the particular number of landmarks for the images. This selection of a
particular increment for the regular landmarks is specified in [5], using the equation:
I = max (ni-1, ni) / N (2)
Where, I is the increment, N is the total number of landmarks used for the experiments, ni-1 & ni are
respectively the total number of edge points in the previous (t-1 time’s) and current (time t) foreground image. The
difference in this edge point images are noted and they are again compare with the already stored data patterns. From this
point the data flow is diverted to two directions. The edge point images are compared with the data models already
resides in the data bases, if the activity pattern is already there then the data flow on to the step (e), otherwise flow on to
step (d). This is a crucial step in this system. Because of that the edge point extraction will became the core of this
system.
4.4 Learning new activity
In the previous step, after the edge point extraction, if the activity is not exists in the models, then learning the
new activity become the next step. By taking the corresponding images the new set of activity is stored in the database
using a particular threshold. Learning the unmodelled activity will be very important for a scalable system. Updating the
database is done using an ontology based concept [14]. The similarities of the unlabelled image frames are found out and
these similar actions form a group and update the database with a new activity. Consider:
T1 = similarity (ni , ni +1),
T2 = similarity (ni +1, ni +2) etc. (3)
Were, ni, ni+1, ni+2 etc are each unidentified consecutive frames. T1, T2 are just the similarity thresholds.
If T1, T2 are similar then group them as the actions of the same activity. Form a threshold for the new activity
and save it in the database for the further usage.
4.5 Spatio-temporal feature analysis
If the activity is already modelled or the activity is known to the system, then next step is the spatio- temporal
feature analysis. The location of the activities taken and time at which the activities done are also crucial in the activity
recognition. From this data the activity is classified as normal or abnormal [15]. Based on the location the activities can
be detected. The activities done on different locations should be different. Only some normal activities are carried out
same in different locations. These activities are find out and already stored in the database.
Any activity that is done at different represents as an abnormal activity. Time has also considered. The time at
which any activity taken place is also important in the activity recognition. Some activities such as medicine intake
depend mainly on time. As the medicine intake greater than 3 or 4 in a day then it will mark as an abnormal activity.
Thus for some activities a threshold T is set. And the number of occurrence of the activity is counted by a counter and
when the threshold becomes less than the counter value, and then the false alarm is forwarded.
4.6 Checking activity is normal or not
After considering the spatio- temporal features, the final activity recognition is done. Here mainly the accidental
activities such as falls, injuries are identified. Help of the data bases and the spatio- temporal features these are done. If
the activity belongs to the abnormal category, then an alarm is triggered as a message to corresponding person or any

5. Proceedings of the International Conference on Emerging Trends in Engineering and Management (ICETEM14)
30 – 31, December 2014, Ernakulam, India
68
hospitals/ medical institutions. Otherwise the activity is identified by the system and sends the information to the
corresponding person.
5. EXPECTED RESULTS
The demonstration of the proposed system can be done using an experiment video clip. The image frames are
extracted from the input video image and these frames are used for the further activities. Then the foreground will be
extracted using the background subtraction method. The background subtraction shall been done using the background
image extracted earlier and the frame with the person. The Fig. 1 shows the input data to the system.
Fig.2: example frame of the background and the input frames used for background subtraction
Then the shape of the person and the interacting objects can be detected from the foreground frame and then the
edge points of the human and the interacting objects are obtained. Then the database checking is done. The database data
has a crucial value in the recognition system. The pattern comparison is done using the most efficient classifiers. If the
data about the activity is not present a learning process is done. The time and the place where the activity is proceeding
are also dependent on the detection of the activity. The system can detect any of the activity, but here takes an abnormal
activity such as fall.
The expected outcome is a false alarm and the detection of the activity (i.e. fall an abnormal activity). The result
set can be demonstrated using the following table:
Table.1: Normal Vs abnormal activities
The abnormal activities like unusual medicine intake, lying over time etc are detected using the particular
threshold values given. The alarm is set in the form of SMS/ Voice alarm; this can be sent to the nearby hospitals or the
corresponding personalities.
6. CONCLUSION
The paper will overcome the existing tradition techniques of the activity recognition. Those may have highly
expensive. The proposed method adopts many simple methods for the detection of the each activity. By the combined
use of these simple techniques the system must be made efficient and will be of less expensive. The alarm system and the
database learning systems are act like separate modules. But which are incorporated in the main module of the activity
detection system. In future, the system can be expanded using the latest infrared technology and also researches can be
made on this area. Finally, we can believe that the system will perform efficiently to detect the activities.

6. Proceedings of the International Conference on Emerging Trends in Engineering and Management (ICETEM14)
30 – 31, December 2014, Ernakulam, India
69
7. REFERENCES
[1] M. Kangas, A. Konttila, P. Lindgren, I. Winblad, and T. J¨ams¨a, Comparison of low complexity fall detection
algorithms for body attached accelerometers, Gait Posture, vol. 28, no. 2, pp. 285–291, 2008.
[2] M. Nyan, F. E. Tay, and E. Murugasu, A wearable system for pre-impact fall detection, J. Biomech., vol. 41,
no. 16, pp. 3475–3481, 2008.
[3] iLife. Fall Detection Sensor [Online]. Available: http://www. falldetection.com/iLifeFDS.asp.
[4] Directalert. Wireless Emergency Response System [Online]. Available:
http://www.directalert.ca/emergency/help-button.php.
[5] Caroline Rougier, Jean Meunier, Alain St-Arnaud, and Jacqueline Rousseau, Robust Video Surveillance for Fall
Detection Based on Human Shape Deformation, IEEE transactions on circuits and systems for video
technology, vol. 21, no. 5, may 2011 611.
[6] Megha D Bengalur, Human Activity Recognition using Body pose features and support vector machine,
International Conference on Advances in Computing, Communications and Informatics (ICACCI) 2013, IEEE
Conference.
[7] Dipak Surie, Saeed Partonia, Helena Lindgren, Human Sensing using Computer Vision for Personized Smart
spaces, 2013 IEEE 10th International Conference on Ubiquitous Intelligence & Computing and 2013 IEEE 10th
International Conference on Autonomic & Trusted Computing.
[8] Tanvi Banerjee, Student Member, IEEE, James M. Keller, Fellow, IEEE, Marjorie Skubic, Senior Member,
IEEE, and Erik Stone, Student Member, IEEE, Day or Night activity Recognition from Video using Fuzzy
Clustering Techniques, 2013 IEEE 10th International Conference on Ubiquitous Intelligence & Computing and
2013 IEEE 10th International Conference on Autonomic & Trusted Computing, IEEE transactions on fuzzy
systems, vol. 22, no. 3, June 2013.
[9] Choi, Chansu Kim, and Sung-Kee Park, 2013 IEEE RO-MAN, Human tracking with multiple 3D Cameras for
Perceptual Sensor Network, The 22nd IEEE International Symposium on Robot and Human Interactive
Communication Gyeongju, Korea, August 26-29, 2013.
[10] Ivo Everts, Jan C. van Gemert, and Theo Gevers, Member, Evaluation of Color Spatio-Temporal Interest Points
for Human Action Recognition, IEEE, IEEE transactions on image processing, vol. 23, no. 4, April 2014.
[11] Bingbing Ni, Yong Pei, Pierre Moulin, Fellow, IEEE, and Shuicheng Yan, Senior Member, IEEE, Multilevel
Depth and Image Fusion for Human Activity Detection, IEEE transactions on cybernetics, vol. 43, no. 5,
October 2013.
[12] J. Mike McHugh, Member, IEEE, Janusz Konrad, Fellow, IEEE, Venkatesh Saligrama, Senior Member, IEEE,
and Pierre-Marc Jodoin, Member, IEEE, Foreground-Adaptive Background Subtraction, IEEE signal processing
letters, vol. 16, no. 5, may 2009.
[13] J. Canny, A computational approach to edge detection, IEEE Trans. Pattern Anal. Mach. Intell., vol. 8, no. 6,
pp. 679–698, Nov. 1986.
[14] Liming Chen, Member, IEEE, Chris Nugent, Member, IEEE, and George Okeyo, Member, IEEE, “An
Ontology-Based Hybrid Approach to Activity Modeling for Smart Homes, IEEE transactions on human-
machine systems, vol. 44, no. 1, February 2014.
[15] Chen Wu, Amir Hossein Khalili and Hamid Aghajan Stanford University, Stanford CA , “Multiview Activity
Recognition in Smart Homes with Spatio-Temporal Features, Aug 31, 2010.
[16] Kavita P. Mahajan and Prof. S. V. Patil, “Tracking and Counting Human in Visual Surveillance System”,
International Journal of Electronics and Communication Engineering & Technology (IJECET), Volume 3,
Issue 3, 2012, pp. 139 - 146, ISSN Print: 0976- 6464, ISSN Online: 0976 –6472.
[17] Najmuzzama Zerdi, Dr. Subhash S Kulkarni, Dr. V.D. Mytri and Kashyap D Dhruve, “Crowd Behaviour
Analysis Considering Inter-Personnel Activities in Surveillance Systems”, International Journal of Computer
Engineering & Technology (IJCET), Volume 5, Issue 2, 2014, pp. 71 - 87, ISSN Print: 0976 – 6367,
ISSN Online: 0976 – 6375.