Human Activity AnalysisA ReviewJ.K .Aggarwal and M.S.RyooThe University of Texas at Austin Volume 43 Issue 3, April 2011. ACM New York, NY, USA Presented by:Sonam yar
CONTENTS Problem Domain Human activities Introduction Single layer approaches Hierarchical Approaches Human-object Interactions Group Activities Conclusion
PROBLEM DOMAINThe increased use of camerasThe most important: goals of video analytics is to detect abnormalities
HUMAN ACTIVITIES Gestures Actions Interactions Group Activities
Applications of Human ActivityanalysisAutomated surveillance systemsAirportsSubway stationsPatients observanceAnalysis of the physical condition of peopleCaring of aged people
INTRODUCTION – Human activity recognition is important. – Objective of the paper – Overview – Concentrates on low level along with high level activity recognition methodologies – Approach based taxonomy
TAXONOMY OF ACTIVITIES Automated surveillance systems Airports Subway stations Patients observance Analysis of the physical condition of people caring of aged people
SINGLE LAYER APPROACHESSpace-time Approaches1. Action recognition with space-time volumes – Bobick and Davis template matching • Motion-history image ( MHI) • Motion-energy image ( MEI)
SINGLE LAYER APPROACHESSpace-time Approaches •Continue…. oShechtman and Irani Compare volumes in terms of their patches oKe et al. Used segmented spatio-temporal volumes to model human activities.
SINGLE LAYER APPROACHESSpace-time Approaches Continue…. oRodriguez et al Filters capturing characteristics of volumes
SINGLE LAYER APPROACHES Continue…. oRodriguez et al Filters capturing characteristics of volumes
SINGLE LAYER APPROACHESSpace-time Approaches•Disadvantages of space-time volume The major disadvantage of space-time volume approaches is the difficulty in recognizing actions when multiple persons are present in the scene.
SINGLE LAYER APPROACHES Space-time Approaches2. Action recognition with space time trajectories • Campbell and Bobick Curves in low-dimensional phase spaces • Rao and Shah s methodology Their system extracts meaningful curvature patterns from the trajectories.
SINGLE LAYER APPROACHESSpace-time Approaches2. Action recognition with space time trajectoriesAdvantages • Ability to analyze detailed levels of human movements •View invariant methods
SINGLE LAYER APPROACHES Space-time Approaches3. Action recognition with space time features• Chomat and Crowly • Calculates local probability of an activity • final recognition• Rao and Shah s methodology • An approach utilizing local spatio-temporal features at multiple temporal scales. Multiple temporally scaled video volumes are analyzed to handle execution speed variations of an action.
SINGLE LAYER APPROACHESSpace-time ApproachesComparison:1. Space time volume: • Space-time approaches are suitable for recognition of periodic actions and gestures, and many have been tested on public datasets. • Provides straight forward solution. • Often have difficulties in handling speed and motion variations inherently.2. Space-time trajectories • Recognition approaches using space-time trajectories are able to perform detailed-level analysis and are view-invariant in most cases.
SINGLE LAYER APPROACHESSpace-time ApproachesComparison:3. Spatio-temporal local feature-based approaches • Getting an increasing an amount of attention. • Recognize multiple activities without background subtraction or body- part modeling. LIMITATIONS The major limitation of the space-time feature-based approaches is that they are not suitable for modeling more complex activities. The relations among features are important for a non-periodic activity that takes a certain amount of time, which most of the previous approaches ignored.
SINGLE LAYER APPROACHES Sequential Approaches1. Exemplar based2. State based
SINGLE LAYER APPROACHESSequential Approaches1. Exemplar Based • Compare the input video with the template video. • DTW( Dynamic time warping ) algorithm is used for matching variations. • Multiple cameras have been used to obtain 3-D body-part models of a human, which is composed of a collection of segments and their joint angles.
SINGLE LAYER APPROACHESSequential Approaches2. State model-based Approaches • Represent a human activity as a model composed of a set of states. • An activity is represented in terms of a set of hidden states. • A human is assumed to be in one state at each time frame, and each state generates an observation.
SINGLE LAYER APPROACHESSequential ApproachesState model-based ApproachesThe evaluation problem is a problem of calculating the probability of a givensequence (i.e. new input) generated by a particular state-model.If the calculated probability is high enough, the state model-based approaches areable to decide that the activity corresponding to the model occurred in the givenInput.
SINGLE LAYER APPROACHESSequential ApproachesComparison:• Enable to detect more complex activities like nom periodic activities.• Able to make a probabilistic analysis on the activity.• Calculates a posterior probability of an activity occurring, enabling it to be easily incorporated with other decisions.
HIERARICHAL APPROACHESStatistical approaches•At the bottom layer, atomic actions are recognized from sequences of feature vectors, just as in single-layered sequential approaches. As a result, a sequence of feature vectors are converted to a sequence of atomic actions. For each model, a probability of the modelgenerating a sequence of observations (i.e. atomic-level actions) is calculated tomeasure the likelihood between the activity and the input image sequence.
HIERARICHAL APPROACHESSyntactic approaches • Syntactic approaches model human activities as a string of symbols, where each symbol corresponds to an atomic-level action. • Require atomic-level actions to be recognized first, using any of the previous techniques
HIERARICHAL APPROACHESSyntactic approaches One of the limitations of syntactic approaches is in the recognition of concurrent activities. Syntactic approaches are able to probabilistically recognize hierarchical activities composed of sequential sub-events, but are inherently limited on activities composed of concurrent sub-events
HIERARICHAL APPROACHESDescription-based approachesIn description-based approaches, a time interval is usually associated with anoccurring sub-event to specify necessary temporal relationships among sub-events.Seven basic predicates that Allen hasdened are: before, meets, overlaps, during, starts, nishes, and equals.
HIERARICHAL APPROACHES Comparison•Suitable for recognizing high-level.•Easily incorporate human knowledge into the systems•Require less training data1. Statistical and syntactic approaches o Provide a probabilistic framework for reliable recognition with noisy inputs.2. Description-based approaches o represent and recognize human activities with complex temporal structures. o Sequentially and concurrent organized sub-events are handled.
EXTENDED PORTION OF THE PAPER HUMAN-OBJECT INTERACTIONS AND GROUP ACTIVITIES
HUMAN-OBJECT INTERACTIONS Integration of multiple components is required to recognize human object interactions Steps involved: • Identification of objects • Motion involved in an activity •Analysis of their interplays These components are highly dependent on each other. The results suggest that the recognition of objects can benefit activity recognition while activity recognition helps the classification of objects.
HUMAN-OBJECT INTERACTIONSMoore et al.  Compensates for the failures of object classification with the recognition results of simple actions. Common Performance of system: object recognition estimates human activities with objects But can act conversely as wellPeursum et al. Focused on the fact that humans interact with objects in many different ways,depending on the function of the objectsObject recognition solely based on the activity information
HUMAN-OBJECT INTERACTIONS Gupta and Davis proposed a probabilistic model integrating an objects appearance, human motion withobjects, and reactions of objects.Two types of motion in which humans interact with objects, `reach motion and`manipulation motion, are estimated. Ryoo and Aggarwal  Their object recognition and motion estimation components were constructed to help each other. compensate for object recognition failures or motion estimation failures. get feedback from the high-level activity recognition results for improved recognition.
GROUP ACTIVITIESGroup activities are the activities whose actors are one or more conceptual groups.In order to recognize group activities, the analysis of activities of individuals as well astheir overall relations becomes essential.CONTAINS TWO FOCUSE POINTS1. Researchers have focused on the recognition of group activities where each group member has its own role different from the others.2. The second type of group activity is the activities which are characterized by the overall motion of entire group members.
CONCLUSION•Applications of human activity recognition are diverse.•Tracking and monitoring people is becoming and integral part of everyday activities.•The paper gives the latest and the previous methodologies been explored.•1999 , human activity recognition was in its infancy.•Early cameras were fixed and simple.•Todays cameras with pan-tilt-zoom features creates more challenges for the researchers.• problem areas, causing failures: noise, lights, distance and tracking.•Future direction is encouraged and dictated by applications.