VSMM’05, Belgium

                  Computer Vision based Dance
VSMM’05, Belgium

    markers at 27 points on a dancer’s body. The researchers digitized the video, using it to
VSMM’05, Belgium

     through video. One is the translational motion, especially that in depth direction. One
VSMM’05, Belgium

    of a frame as seed points. Then we compare alternate seeds (s1) across the height
    and width of t...
VSMM’05, Belgium

                            Fig 3. Edge extracted/Boundary detected

2.2 Skeletonization

    The nex...
VSMM’05, Belgium

    instead of calculating the distance of each pixel from the boundary pixel we calculate the
VSMM’05, Belgium

    image into 32 gray levels as they are adequate to track major object motions. Now we
    take a 11x1...
VSMM’05, Belgium

                                 Fig. 6. Locus of tracked finger

2.4 Interface

    The design and i...
VSMM’05, Belgium

                               Screen Shot of the Main Application

VSMM’05, Belgium


  [1]  Visual dictionary of Hastas for Indian dance - hand gestures of Indian Dance
Upcoming SlideShare
Loading in …5

Computer Vision based Dance Visualisation


Published on

Invited paper at VSMM\'05, Belgium

Published in: Business, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Computer Vision based Dance Visualisation

  1. 1. VSMM’05, Belgium Computer Vision based Dance Visualisation Sanjay Goel, Chirag Gupta, T. Gnana Swaroop, Gaurav Jain, Tarang Gupta and Shoma Chatterjee Jaypee Institute of Information Technology, Noida, India Abstract :- In this paper we discuss a Computer Vision based tool for dance scholars. The tool aims to use computer vision to enable the analyst to concentrate on body movements. The processed video highlights the main body motion by extracting body contour. The tool provides the ability to add and display additional textual information with each frame. It also facilitates juxtaposition of original video with extracted video. 1. Introduction The fascination for Indian dance all over the world is indicative of the deep-felt need to use the human body to express and celebrate the great universal truths. It illuminates India's culture in a direct manner, playing on the sensibilities of the onlooker. Dance in India has seeped into several other realms like poetry, sculpture, architecture, literature, music and theatre. All Indian dance forms are thus structured around the nine rasas or emotions [1], hasya (happiness), krodha (anger), bhibasta (disgust), bhaya (fear), shoka (sorrow), viram (courage), karuna (compassion), adbhuta (wonder) and shanta (serenity). Very little contribution other than video storage and dissemination has been made by the fast growing computer technology in the field of dance and none for Indian classical dance. This paper discusses a tool that is under development to help dance scholars to analyze solo dance performances. 1.1 Literature Survey: Computers find various uses in Dance [5]. Some important non-graphics uses are described in administration, lighting control, and competition scrutineering. Graphical applications include notation, choreography, teaching, and performance. One of the earlier works [2] explores the nature of creative composition particularly as it applies to dance, and describes the development of interactive computer based tools to assist the composer. The hierarchical nature of the composition process calls for an interface which allows the composer the flexibility to move back and forth between alternate views and conceptual levels of abstraction. COMPOSE, an interactive system for the composition of dance has been implemented on Silicon Graphics and Apple workstations, [2]. The user visually composes in space and in time using menus of postures and sequences. The animation of the dance composition allows the final result to be evaluated. One of the first Dance Technology composition, choreographed in 1994, researchers used Motion Interactive (MINT) - a special motion - capture program they developed - to translate dance into computer animation, [3]. Two video cameras captured the movement of reflective
  2. 2. VSMM’05, Belgium markers at 27 points on a dancer’s body. The researchers digitized the video, using it to create a computer model of the dancer [3]. For another performance, the researchers employed infrared cameras to track emitters hidden on a dancer’s costume. This data is fed into a high - speed graphics workstation in real time, the animation video resulted in animated trails of the dancer’s movements by projection of real-time graphics onto a translucent screen [3]. A collaboration between the Atlanta Ballet and Georgia Tech’s Interactive Media Technology Center (IMTC), the Dance Technology Project featured combining ballet and computer animation techniques [3]. The project dealt with video costuming. That is, a camera and computer system track the motions of the dancers on stage while a second graphics computer is used to create their ‘virtual costumes’ which are projected onto them, in exact registration to their body orientations - even as they dance. Other activities were computer generated dancers intermingling with real dancers, and computer - generated art ‘created’ by the dancers as the performance progressed [3]. The work reported in [4] deals with phrase structure detection in contemporary western dance. Phrases are a sequence of movements that exist at a higher semantic abstraction than gestures. The problem is important since phrasal structure in dance, plays a key role in communicating meaning, [4]. They detect fundamental dance structures that form the basis for more complex movement sequences. Computed dancing figures have also been proposed as an aid in teaching dance [5]. For example, the computer could be used to show idealised movements slowly of fast steps, that are impossible to demonstrate slowly because of problems with balance or momentum. Computers could also be used as a teaching aid for student to classify for themselves steps with complex alternatives [5]. The authors in [6] have come up with an algorithm of synthesizing music that can appropriately express emotions in dances. This algorithm can help one compile music suitable for dance movies or animation films, and is also applicable to any entertainment systems that use music or dance. The algorithm is composed of three modules. The first is the module of computing emotions from an inputted dance, the second that of computing emotions from music in the database and last that of selecting music suitable for inputted dance via an interface of emotion, [6]. An experimental dance performance featuring live-motion capture, real-time computer graphics, and multi-image projection was produced by a cross-departmental team of faculty and students at Purdue University, [7]. Dancers occupied and traversed performance mediums or ‘frames’ including a virtual performance frame occupied by a 3D character, driven by a dancer in motion-capture equipment. Developing and facilitating the relationships between the dancers in various performance frames was a primary focus of the project. A multimodal information system method for a basic dance training system is discussed in [8]. The system targets on beginners and enables them to learn basics of dances easily. One of the most effective ways of learning dance is to watch a video showing the performance of dance masters. However, some information cannot be conveyed well
  3. 3. VSMM’05, Belgium through video. One is the translational motion, especially that in depth direction. One cannot tell exactly how far does the dancers move forward or backward, [8]. Another is the timing information. Although one can tell how to move arms or legs from video. It is difficult to know when to start moving them. The first issue is solved by introducing an image display on a mobile robot [8]. One can learn the amount of translation just by following the robot. They introduced active devices for the second issue [8]. The active devices are composed of some vibro-motors and are developed to direct action-starting cues with vibration, [8]. 1.2 Scope of project The main objective of our project is to exploit the potential of digital image processing and computer vision techniques to serve some of the common and regular needs of scholars and students of Indian classical dance. At present, dance students and scholars learn or analyze dance movements by observing performances of professional dancers. Video recordings of performances are popular for later reference and analysis. Often the dance scholar needs to concentrate on specific aspects like hand movement and so on. In the absence of any tool to filter out distracting details, such scholarly analysis becomes a tedious task. User friendly software tool(s) can help dance scholars analyze and annotate the recorded performances, add subtitles and annotation for specific frame sequence and store the annotated video in a regular format viewable on any regular media player. Availability of such software will encourage scholars to add more information in recorded videos which can be accessed, understood and appreciated by a common man. Such software will allow the scholars to create a well documented archive of dance videos with searchable annotations. 2. Outline of the Algorithm This section discusses design of our Computer Aided Dance Analysis and Visualization tool. The tool allows users to view the original video and the processed video simultaneously. It also allows the users to add information to every frame of the dance video. All forms of Indian Classical dance depict a story. Frame specific information can be added with each frame as frame annotation. It allows the users to filter out distracting details by extracting body contour and image skeleton. We have designed a sample interface using Visual C++. Matlab has been used as an intermediate test environment [14], where we tested the various image processing algorithms which have later been migrated to Visual C++. The phases involved in the design of the dance tool and the results are underlined below. The main processes are segmentation, edge detection and skeletonization. 2.1 Segmentation and Edge Detection We extract frames from the video, as in Fig. 1, and segment to separate the dancer from the frame. We use the Region Growing algorithm [9] for this because we need to separate the dancer on the basis of color as well as region. For applying multi pass region growing algorithm, we convert the image to 256 level grayscale. We initially select all pixels
  4. 4. VSMM’05, Belgium of a frame as seed points. Then we compare alternate seeds (s1) across the height and width of the frame with all four seed neighbors (north,south,east,west). The initial threshold range is kept to be within eight graylevel difference. If the two compared seeds are found similar i.e. within the threshold range, then we mark the neighbor with the value of seed s1 . Next we continue with the same process in subsequent passes by doubling the threshold range in subsequent passes until it reaches 128 as we need a binary output. The output of this process is the image of dancer separated from the rest of image as shown in Fig. 2. We convolve the segmented image with the Laplacian Mask for boundary detection [9] as in Fig. 3. Fig 1. Input Video Frame Fig 2. Clustered Frame
  5. 5. VSMM’05, Belgium Fig 3. Edge extracted/Boundary detected 2.2 Skeletonization The next and the most important algorithm is skeletonization. We tried using different thinning algorithms like Medial Axis Transform [9] but none of them gave statisfactory results. In Medial Axis Transformation of a region R with border B, for each point p in R, we find its closest neighbor in B. If p has more than one such neighbor, it is said to belong to medial axis of R. The Medial Axis Transformation does not serve our purpose as the output of this algorithm for an L-shaped figure would be as shown in Fig. 4. Skeleton from Medial Axis Transformation Expected Skeleton Fig. 4 Also the complexity of the Medial Axis Transformation is very high as it compares every pixel in the boundary of the region to each of the pixels in the image. This was the reason why we had to modify the algorithm to suit our purpose. In the modified algorithm
  6. 6. VSMM’05, Belgium instead of calculating the distance of each pixel from the boundary pixel we calculate the distance of each boundary point from its horizontally opposite boundary point. Unlike Medial Axis Transform where we compare all the points (whether boundary or region) with each other, we select only the boundary points. We select a point in the boundary and find its opposite (horizontal) boundary point. One horizontal line may have portions of more than one body parts. The odd numbered boundary points on every horizontal line mark the beginning and the corresponding subsequent even numbered boundary point mark the end of the body part. We mark the center of these two points as the skeleton point for all such pairs of boundary points. S represents the set of all skeleton points. The output we get from this is not the perfect skeleton in all positions but this along with the boundary is enough for user to visualize the movements. This allows us to get closer to the expected skeleton as in Fig. 5 with much lower complexity allowing us to process same video in a lesser time. Fig 5. Skeletonized Frame 2.3 Object Tracking The final image processing algorithm in the project is synchronized multiple inter-connected object tracking. The main objective of this step is to mask and track the anchor points of the body. These anchor points comprised of head, neck, shoulders, elbows, palms, waist and one or two points for legs. The points for legs were kept low keeping in mind the traditional classical dances where the female dancers wear saris. In our future work, we plan to use the anchor points for creating Vector Stick diagram of the dancer. Traditional object tracking [15,16] failed in our case because such algorithms are made for tracking simple object in a video. In our case we had to track multiple inter-connected complex objects (body parts) in a video and that too in a synchronized way. We have designed our own algorithm for this purpose. This algorithm is based on the principle of pattern matching and tracks objects in the input video using the output from the Skeletonizing algorithm. First of all we mark a point on the object which we want to track as p1. We quantize the image into 32 gray levels. When an object moves in a video some blurring is caused which results in slight changes in color of the object. To correct these errors we quantize the
  7. 7. VSMM’05, Belgium image into 32 gray levels as they are adequate to track major object motions. Now we take a 11x11 pixel window (w1) on frame (j) with center as p1. Then we take 121, 11x11 pixel search windows from frame (j+1) with centers lying on each pixel of the corresponding 11x11 window (w1) on frame (j+1). We compare the histograms of all these search windows on frame (j+1) with (w1) and identify the window of minimum difference as the region (wr ) in which motion has taken place. The comparison of histograms is done according to the following formula: Diff = ∑ [ƒ1 (binx) - ƒ2 (binx)]; where x extends from 1 to 32 gray scales ƒ1(x): number of pixels of bin x gray scale in primary window. ƒ2(x): number of pixels of bin x gray scale in search window. Now the problem is to search one point out of these 121 points in the window (wr). Firstly, we find the difference along the x and y axis between p1 and the center of wr (center of wr - p1) to identify the quarter as given in Table 1 with (0,0) being considered as the top left corner of the image. X Difference Y Difference Quarter Positive Positive Bottom right Negative Positive Bottom left Positive Negative Top right Negative Negative Top left Table 1 : Quarter Identification in the search window Now we search the identified quarter for skeleton points in top-down, left-right order and mark the first skeleton point found. Marking the point on the skeleton makes sure that the point does not move out of the body. If the point is not found in this quarter we scan the full window for skeleton points and mark the first skeleton point found. If there is no point of skeleton in this window we simply mark the center of this window as the corresponding point in frame (j+1). This process is applied to all the consecutive frames with respect to immediate predecessor frame, hence tracking the object as shown in Fig. 6.
  8. 8. VSMM’05, Belgium Fig. 6. Locus of tracked finger 2.4 Interface The design and interface of the tool was created in Visual C++ following the Document View Architecture. Our tool extensively used Multithreading in Visual C++. The processed video highlights the main body motion by extracting the body contour and also provides the ability to add and display additional textual information about the dance video with each frame for the user. It facilitates juxtaposition of original video with extracted video as shown in Fig. 7. 2.5 Future Scope The work on the vectorization of the dancer’s stick diagram has been intiated. We have also realised an interface for easy ( and precise ) access of dance videos from the Digital Video Archive. The major obstacle is to do this without consuming huge bandwidth. Our interface for Digital Video Archives is based on the skeletonization algorithm reported in this paper. Details of this interface will be discussed in a future paper.
  9. 9. VSMM’05, Belgium Screen Shot of the Main Application Player Controller Slider Toolbar Input Video Subtitle Addition Processed Output Displayed Here Video Displayed Here Fig. 7 Acknowledgements We are extremely thankful to Maria, a dance teacher who runs her own dance school. We had a very fruitful discussion and we got many new interesting perspectives to look at our problem. Some of the relevant outcomes of this discussion were, extracting the dancer from the dance video, hiding irrelevant information like color of dress etc., and applying enhancements on the dancer to study the dance movements better and ability to compare two dance performances of same dancer or similar performances by different dancers.
  10. 10. VSMM’05, Belgium References [1] Visual dictionary of Hastas for Indian dance - hand gestures of Indian Dance http://www.kanakasabha.com/hastas/index.htm [2] T. Schiphorst, et al., Tools for Interaction with the Creative Process of Composition. Centre for Systems Science, Simon Fraser University, Burnaby, BC, V5A 1S6, Canada., CHI 90 Proceedings, pp 167 – 174. [3] Reseach for the Games. Georgia Tech forged strong ties to Atlanta’s 1996 Olympic Games. Compiled by Lea McLees. [4] V. M. Dyaberi, et al. Phrase Structure Detection in Dance, Proceedings of the 12th annual ACM international conference on Multimedia, Oct 2004, pp. 332 - 335. [5] Dance and the Computer : A Potential for Graphic Synergy. Technical Report 422. Basser Department of Computer Science. University of Sydney, Oct 2003. [6] Hirofumi Morioka , et al.Proposal of an Algorithm to Synthesize Music Suitable for Dance. Proceedings of the 2004 ACM SIGCHI International Conference on Advances in Computer Entertainment Technology, Sept 2004, pp. 296 – 301. [7] W. Scott, et al. Mixing Dance Realities: Collaborative Development of Live - Motion Capture in a Performing Arts Environment. ACM Computers in Entertainment ( CIE ), vol 2, issue 2, April 2004. [8] Akio Nakamura, et al. Multimodal Presentation Method for a Dance Training System. Saitama University, JAPAN. CHI 2005, pp 1685 – 1688. [9] R.C. Gonzalez and R.E. Woods - Digital Image Processing, second edition, Pearson Education. [10] The Open Video Project http://www.Open-Video.org [11] Digital Video Archives: Managing Through Metadata [12] Informedia Digital Video Library System – Carnegie Mellon University [13] Fabio Chestani - Video Retrieval Interfaces [14] Building GUIs with Matlab version 5 from MathWorks [15] Darrell D. Demirdjian T. Ko T. - Constraining Human Body Tracking, Artificial Intelligence Laboratory, MIT [16] Minden Gary, Niehaus Doug and Roberts James - The Digital Video Library System: Vision and Design