SlideShare a Scribd company logo
1 of 12
Download to read offline
Multimedia Systems - Class Project
Group 2
Spatio-Temporal Invariant Points in Videos
Members:
Priyatham Bollimpalli – 10010148
Pydi Peddigari Venkat Sai – 10010149
PVS Dileep – 10010180
The objective here is to find the spatio-temporal invariant points in a given input video. We
implement the following models on a set of contiguous frames of a video, called a scene.
We divide the problem into three cases, one with the background being fixed in a video and
the entire scene is not dynamic, second, background fixed and the entire scene is
reasonably dynamic, and final one with the background moving and the objects are also
moving. We examine those cases below:
Case1: When the Background is fixed and the entire scene is not dynamic
In this case, the background in the scene is fixed across several frames, while the foreground
objects can keep moving across the whole video, but they do not occupy the entire frame of
video with their movements, i.e., only some parts of the frame would be having movement,
while a decent part of the frame would remain static. The following procedure is followed to
detect the spatio-temporal interest points in the scene:
 Every scene is a collection of several frames. In this instance, we would consider a
scene of a video in which the background is constant, and in the foreground, there is
a ball which quickly moves underneath a wooden block which is constant in the
video. These are just a four of the several frames present in the video:
 The difference of the first two frames of the image would be computed. In the
resulted difference image, we would have several ranges of pixel values possible.
Hence, we would keep a grey threshold of say 0.2, and put all the locations of the
frame which are above that grey threshold level, to pixel value of that of white. We
also have an image which we will keep writing through across several iterations of
this method(we refer to that image as Rframe). Now, in the previously resulted
frame of difference, we find all the locations where the pixel value is white, and fill
out all those corresponding locations on the Rframe to black.
 We keep on repeating this process of finding the difference between two successive
frames of the scene, thresholding the difference to get some pixel values which are
white, and filling out all the locations on Rframe where there is a white pixel to
black.
For example, the difference of the frames at several instants are as follows:
 Then, the final Rframe produced will be as follows:
The portions which are black in this image would depict those points where a
temporal invariance is not possible, as the objects keep moving in those areas. The
portions which are white would indicate those points which are constant throughout
the period of the video. These would be our points of interest for application of SIFT
on those points.
 On the final go, we would apply SIFT on the original frames of the scene, and find all
those points resulted from SIFT, and consider such a point among them to be our
interest point in this case only if that point is among the white portions of the
Rframe, i.e., we would only want the interest points which coincide with the white
portions of the Rframe.
Hence, the resulted interest points finally would form our Spatio temporal invariant
points in this case.
The above algorithm is also run on other videos as follows:
 Some of the several frames in the original video :
 Some among the resulted differences in the frames at several instants are:
 Then the Rframe resulted is as follows:
 Now, we find the interest points at those locations, which are among the whie
portions of the Rframe, since, only those would be the temporal invariant parts of
the image. Hence, the interest points of the scene resulted at some of the several
instants are as follows:
So, the above points marked with green would represent the spatio-temporal
interest points of the scene, at some of the instants among the whole.
 It is run on another video as follows:
 The Rframe resulted is :
 Interest points resulted at several frames are :
Case2: When the Background is fixed and the entire scene is reasonably
dynamic
In this case, the background in the scene is fixed across several frames, while the foreground
objects can keep moving across the whole video, and they do occupy the most of the frame
of video with their reasonable movements. In this case, the following procedure is followed:
 Consider a scene as follows:
 Now, if Case-1 was used here, the Rframe resulted would be as follows:
Hence, if case-1 is used here, we can observe that most of the region is blacked out
since the objects motion is present almost over the entire image, and hence, we lose
some of the possible interest points.
 Hence, we adopt the following method now. This method would use automatic
detection and motion-based tracking of moving objects in a video. This problem can
be seen as:
o detecting moving objects in each frame
o associating the detections corresponding to the same object over time
 The association of detections to the same object is based solely on motion. The
motion of each track is estimated by a Kalman filter. The filter is used to predict the
track's location in each frame, and determine the likelihood of each detection being
assigned to each track.
 In any given frame, some detections may be assigned to tracks, while other
detections and tracks may remain unassigned. The assigned tracks are updated using
the corresponding detections. The unassigned tracks are marked invisible. Each track
keeps count of the number of consecutive frames, where it remained unassigned. If
the count exceeds a specified threshold, the example assumes that the object left
the field of view and it deletes the track.
 So in the process, a frame is read, objects are detected with their centroids and
bounding boxes, and motion segmentation using the foreground detector. Next, the
detections are assigned to tracks. Then, the assigned tracks would be update and the
unassigned tracks would be updated by marking them invisible, and the lost tracks
would be deleted.
 The following are the results of the tracked objects at some of the instants of the
scene:
 Now, we apply SIFT on the tracked objects at all instants of the video. Then, the
following would be the interest points produced at several instants of the video:
 Now, to gather further more interest points from the video, we can also combine the
interest points generated from case-1, and hence, the resulted interest points would
be:
So this would capture all the interest points possible, combining case-1 and case-2
Case3: When the background is moving and the objects are also moving
When the camera as well as well objects are moving, tracking the objects is a challenging
issue. There will be rarely anything invariant even in one particular scene. This is still an
open research problem and some heuristic methods are successful. The video has a moving
car with a moving camera. Some of the frames are given below.
One heuristic method which we found on the web to solve it is given below.
Note that segmenting the car frame differentiation (background subtraction) won’t work
because the camera is also moving henceforth the background is also moving. Hence normal
prediction algorithms would fail in this case. To tackle this issue, Optical flow is used where
the scene and the car have different directions of flow. In the below figures, the red points
denote the optical flow of the background and the green points denote the points having an
optical flow opposite to the red points. Note that the green points are able to track the car
present in the entire video and hence locate the green points which are spatio-temporally
invariant.
Now, combination of the points obtained here with those obtained in case-1 and case-2 (it is
less likely that any points would be there), gives the total points possible.
Conclusion & Future Work:
In this report we have tried to perform three kinds of techniques on any given scene in a
video – when the background is almost stationary and the scene is not dynamic, when the
background is stationary and the scene is dynamic and the background and the scene
bother are dynamic. Combination of points obtained from all the three methods gives
maximum possible spatio-temporal invariant points.
There is lot of scope for future work in this area and we wish to pursue it further. The
following are the issues involved.
 In case-1, the level at which thresholding is done defines the extent/degree to which
motion of the object is considered. Lower the threshold, greater the impact of
motion. This entirely depends on the video i.e. if a video has lot of illumination and
contrast changes between the frames, then the difference of the frames would give
many false contours. In this case considering higher value of threshoding is desirable.
In some other cases the frame rate would be too high due to which almost negligible
amount of motion would be captured between the frames. In this case lower value
for thresholding is desirable. Hence developing an optimal threshold value
automatically by taking the video quality, frame rate into account is one area of
future work.
 In case-3, the above example gives a possible approach to solution. It works because
the camera is also continuously moving along with the car and getting the points
with opposite optical flow works. But if the camera remains stationary for some time
and moves again suddenly, we need a separate system to first track the background
movement and that in combination of foreground motion can be used to find out
the points on the object. Many object tracking methods exist and many are still
pursed since this is a very active area of research. It is likely that exploring into this
area would give a generic algorithm for obtaining an object and the points which are
consistent on it throughout the scene.
 Case-2 considers the background to be stationary and the motion of the objects to
be uniform. Modifying the parameters for the detection, assignment, and deletion
steps of the trackers according to the video may be done. The tracking in this
example was solely based on motion with the assumption that all objects move in a
straight line with constant speed. When the motion of an object significantly
deviates from this model, the example may produce tracking errors.
The likelihood of tracking errors can be reduced by using a more complex motion
model, such as constant acceleration, or by using multiple Kalman filters for every
object. Also, you can incorporate other cues for associating detections over time,
such as size, shape, and colour.
References
1. http://www.mathworks.in/help/vision/examples/motion-based-multiple-object-
tracking.html
2. http://www.youtube.com/watch?v=MOaKnCSejXQ

More Related Content

Viewers also liked

360suite for SAP BusinessObjects
360suite for SAP BusinessObjects360suite for SAP BusinessObjects
360suite for SAP BusinessObjectsSebastien Goiffon
 
Novembre 2011
Novembre 2011Novembre 2011
Novembre 2011magic32
 
Bo, la nueva mascota presidencial
Bo, la nueva mascota presidencialBo, la nueva mascota presidencial
Bo, la nueva mascota presidencialAntoni
 
Business Process Outsourcing - STEM
Business Process Outsourcing - STEMBusiness Process Outsourcing - STEM
Business Process Outsourcing - STEMTheSystemConsultants
 
Technology in the Classroom
Technology in the Classroom Technology in the Classroom
Technology in the Classroom Crysb2217
 
Saude1 4%20
Saude1 4%20Saude1 4%20
Saude1 4%20Neemias
 
Diagnóstico de Classe utilizando Inteligência de Enxames aplicado ao problema...
Diagnóstico de Classe utilizando Inteligência de Enxames aplicado ao problema...Diagnóstico de Classe utilizando Inteligência de Enxames aplicado ao problema...
Diagnóstico de Classe utilizando Inteligência de Enxames aplicado ao problema...UECE - UFRJ
 
Albelissa nanoforum2012 perucca_2
Albelissa nanoforum2012 perucca_2Albelissa nanoforum2012 perucca_2
Albelissa nanoforum2012 perucca_2Massimo_Perucca
 
Рождество Христово
Рождество ХристовоРождество Христово
Рождество ХристовоNatalya Bekker
 
evolucion del telefono celular
evolucion del telefono celularevolucion del telefono celular
evolucion del telefono celularkrnfxkevin
 
Clase 2-la computadora digital
Clase 2-la computadora digitalClase 2-la computadora digital
Clase 2-la computadora digitalmat3matik
 
Kontrol deyimleri
Kontrol deyimleriKontrol deyimleri
Kontrol deyimleriErol Dizdar
 

Viewers also liked (20)

360suite for SAP BusinessObjects
360suite for SAP BusinessObjects360suite for SAP BusinessObjects
360suite for SAP BusinessObjects
 
Novembre 2011
Novembre 2011Novembre 2011
Novembre 2011
 
Bo, la nueva mascota presidencial
Bo, la nueva mascota presidencialBo, la nueva mascota presidencial
Bo, la nueva mascota presidencial
 
L12796
L12796L12796
L12796
 
Business Process Outsourcing - STEM
Business Process Outsourcing - STEMBusiness Process Outsourcing - STEM
Business Process Outsourcing - STEM
 
Technology in the Classroom
Technology in the Classroom Technology in the Classroom
Technology in the Classroom
 
8
88
8
 
Saude1 4%20
Saude1 4%20Saude1 4%20
Saude1 4%20
 
www.ijerd.com
www.ijerd.comwww.ijerd.com
www.ijerd.com
 
Diagnóstico de Classe utilizando Inteligência de Enxames aplicado ao problema...
Diagnóstico de Classe utilizando Inteligência de Enxames aplicado ao problema...Diagnóstico de Classe utilizando Inteligência de Enxames aplicado ao problema...
Diagnóstico de Classe utilizando Inteligência de Enxames aplicado ao problema...
 
Albelissa nanoforum2012 perucca_2
Albelissa nanoforum2012 perucca_2Albelissa nanoforum2012 perucca_2
Albelissa nanoforum2012 perucca_2
 
Pilot Model Based Testing
Pilot Model Based TestingPilot Model Based Testing
Pilot Model Based Testing
 
Рождество Христово
Рождество ХристовоРождество Христово
Рождество Христово
 
evolucion del telefono celular
evolucion del telefono celularevolucion del telefono celular
evolucion del telefono celular
 
Ln3519721978
Ln3519721978Ln3519721978
Ln3519721978
 
Clase 2-la computadora digital
Clase 2-la computadora digitalClase 2-la computadora digital
Clase 2-la computadora digital
 
Power 720 express
Power 720 expressPower 720 express
Power 720 express
 
Sap bo xi r4.0 short
Sap bo xi r4.0  shortSap bo xi r4.0  short
Sap bo xi r4.0 short
 
De prieelvogel
De prieelvogelDe prieelvogel
De prieelvogel
 
Kontrol deyimleri
Kontrol deyimleriKontrol deyimleri
Kontrol deyimleri
 

Similar to The problem of Spatio-Temporal Invariant Points in Videos

MATLAB Code + Description : Real-Time Object Motion Detection and Tracking
MATLAB Code + Description : Real-Time Object Motion Detection and TrackingMATLAB Code + Description : Real-Time Object Motion Detection and Tracking
MATLAB Code + Description : Real-Time Object Motion Detection and TrackingAhmed Gad
 
International Journal of Engineering Research and Development
International Journal of Engineering Research and DevelopmentInternational Journal of Engineering Research and Development
International Journal of Engineering Research and DevelopmentIJERD Editor
 
Recognition and tracking moving objects using moving camera in complex scenes
Recognition and tracking moving objects using moving camera in complex scenesRecognition and tracking moving objects using moving camera in complex scenes
Recognition and tracking moving objects using moving camera in complex scenesIJCSEA Journal
 
Effective Object Detection and Background Subtraction by using M.O.I
Effective Object Detection and Background Subtraction by using M.O.IEffective Object Detection and Background Subtraction by using M.O.I
Effective Object Detection and Background Subtraction by using M.O.IIJMTST Journal
 
Real-time Moving Object Detection using SURF
Real-time Moving Object Detection using SURFReal-time Moving Object Detection using SURF
Real-time Moving Object Detection using SURFiosrjce
 
Vehicle Tracking Using Kalman Filter and Features
Vehicle Tracking Using Kalman Filter and FeaturesVehicle Tracking Using Kalman Filter and Features
Vehicle Tracking Using Kalman Filter and Featuressipij
 
motion and feature based person tracking in survillance videos
motion and feature based person tracking in survillance videosmotion and feature based person tracking in survillance videos
motion and feature based person tracking in survillance videosshiva kumar cheruku
 
ramya_Motion_Detection
ramya_Motion_Detectionramya_Motion_Detection
ramya_Motion_Detectionramya1591
 
Unit II & III_uncovered topics.doc notes
Unit II & III_uncovered topics.doc notesUnit II & III_uncovered topics.doc notes
Unit II & III_uncovered topics.doc notessmithashetty24
 
IRJET-Motion Segmentation
IRJET-Motion SegmentationIRJET-Motion Segmentation
IRJET-Motion SegmentationIRJET Journal
 
Detection of moving object using
Detection of moving object usingDetection of moving object using
Detection of moving object usingijistjournal
 
DETECTION OF MOVING OBJECT USING FOREGROUND EXTRACTION ALGORITHM BY PTZ CAMERA
DETECTION OF MOVING OBJECT USING FOREGROUND EXTRACTION ALGORITHM BY PTZ CAMERADETECTION OF MOVING OBJECT USING FOREGROUND EXTRACTION ALGORITHM BY PTZ CAMERA
DETECTION OF MOVING OBJECT USING FOREGROUND EXTRACTION ALGORITHM BY PTZ CAMERAijistjournal
 
Motion Object Detection Using BGS Technique
Motion Object Detection Using BGS TechniqueMotion Object Detection Using BGS Technique
Motion Object Detection Using BGS TechniqueMangaiK4
 
Motion Object Detection Using BGS Technique
Motion Object Detection Using BGS TechniqueMotion Object Detection Using BGS Technique
Motion Object Detection Using BGS TechniqueMangaiK4
 
IRJET- Moving Object Detection using Foreground Detection for Video Surveil...
IRJET- 	 Moving Object Detection using Foreground Detection for Video Surveil...IRJET- 	 Moving Object Detection using Foreground Detection for Video Surveil...
IRJET- Moving Object Detection using Foreground Detection for Video Surveil...IRJET Journal
 

Similar to The problem of Spatio-Temporal Invariant Points in Videos (20)

MATLAB Code + Description : Real-Time Object Motion Detection and Tracking
MATLAB Code + Description : Real-Time Object Motion Detection and TrackingMATLAB Code + Description : Real-Time Object Motion Detection and Tracking
MATLAB Code + Description : Real-Time Object Motion Detection and Tracking
 
CS216FinalPaper
CS216FinalPaperCS216FinalPaper
CS216FinalPaper
 
International Journal of Engineering Research and Development
International Journal of Engineering Research and DevelopmentInternational Journal of Engineering Research and Development
International Journal of Engineering Research and Development
 
Recognition and tracking moving objects using moving camera in complex scenes
Recognition and tracking moving objects using moving camera in complex scenesRecognition and tracking moving objects using moving camera in complex scenes
Recognition and tracking moving objects using moving camera in complex scenes
 
Effective Object Detection and Background Subtraction by using M.O.I
Effective Object Detection and Background Subtraction by using M.O.IEffective Object Detection and Background Subtraction by using M.O.I
Effective Object Detection and Background Subtraction by using M.O.I
 
J017377578
J017377578J017377578
J017377578
 
Real-time Moving Object Detection using SURF
Real-time Moving Object Detection using SURFReal-time Moving Object Detection using SURF
Real-time Moving Object Detection using SURF
 
Vehicle Tracking Using Kalman Filter and Features
Vehicle Tracking Using Kalman Filter and FeaturesVehicle Tracking Using Kalman Filter and Features
Vehicle Tracking Using Kalman Filter and Features
 
motion and feature based person tracking in survillance videos
motion and feature based person tracking in survillance videosmotion and feature based person tracking in survillance videos
motion and feature based person tracking in survillance videos
 
ramya_Motion_Detection
ramya_Motion_Detectionramya_Motion_Detection
ramya_Motion_Detection
 
Object tracking
Object trackingObject tracking
Object tracking
 
Unit II & III_uncovered topics.doc notes
Unit II & III_uncovered topics.doc notesUnit II & III_uncovered topics.doc notes
Unit II & III_uncovered topics.doc notes
 
IRJET-Motion Segmentation
IRJET-Motion SegmentationIRJET-Motion Segmentation
IRJET-Motion Segmentation
 
Detection of moving object using
Detection of moving object usingDetection of moving object using
Detection of moving object using
 
DETECTION OF MOVING OBJECT USING FOREGROUND EXTRACTION ALGORITHM BY PTZ CAMERA
DETECTION OF MOVING OBJECT USING FOREGROUND EXTRACTION ALGORITHM BY PTZ CAMERADETECTION OF MOVING OBJECT USING FOREGROUND EXTRACTION ALGORITHM BY PTZ CAMERA
DETECTION OF MOVING OBJECT USING FOREGROUND EXTRACTION ALGORITHM BY PTZ CAMERA
 
Motion Object Detection Using BGS Technique
Motion Object Detection Using BGS TechniqueMotion Object Detection Using BGS Technique
Motion Object Detection Using BGS Technique
 
Motion Object Detection Using BGS Technique
Motion Object Detection Using BGS TechniqueMotion Object Detection Using BGS Technique
Motion Object Detection Using BGS Technique
 
1829 1833
1829 18331829 1833
1829 1833
 
1829 1833
1829 18331829 1833
1829 1833
 
IRJET- Moving Object Detection using Foreground Detection for Video Surveil...
IRJET- 	 Moving Object Detection using Foreground Detection for Video Surveil...IRJET- 	 Moving Object Detection using Foreground Detection for Video Surveil...
IRJET- Moving Object Detection using Foreground Detection for Video Surveil...
 

More from Priyatham Bollimpalli

Meta Machine Learning: Hyperparameter Optimization
Meta Machine Learning: Hyperparameter OptimizationMeta Machine Learning: Hyperparameter Optimization
Meta Machine Learning: Hyperparameter OptimizationPriyatham Bollimpalli
 
Science and Ethics: The Manhattan Project during World War II
Science and Ethics: The Manhattan Project during World War IIScience and Ethics: The Manhattan Project during World War II
Science and Ethics: The Manhattan Project during World War IIPriyatham Bollimpalli
 
Kernel Descriptors for Visual Recognition
Kernel Descriptors for Visual RecognitionKernel Descriptors for Visual Recognition
Kernel Descriptors for Visual RecognitionPriyatham Bollimpalli
 
Design and Fabrication of 4-bit processor
Design and Fabrication of  4-bit processorDesign and Fabrication of  4-bit processor
Design and Fabrication of 4-bit processorPriyatham Bollimpalli
 
Interface for Finding Close Matches from Translation Memory
Interface for Finding Close Matches from Translation MemoryInterface for Finding Close Matches from Translation Memory
Interface for Finding Close Matches from Translation MemoryPriyatham Bollimpalli
 
Literature Survey on Interest Points based Watermarking
Literature Survey on Interest Points based WatermarkingLiterature Survey on Interest Points based Watermarking
Literature Survey on Interest Points based WatermarkingPriyatham Bollimpalli
 

More from Priyatham Bollimpalli (10)

Meta Machine Learning: Hyperparameter Optimization
Meta Machine Learning: Hyperparameter OptimizationMeta Machine Learning: Hyperparameter Optimization
Meta Machine Learning: Hyperparameter Optimization
 
Science and Ethics: The Manhattan Project during World War II
Science and Ethics: The Manhattan Project during World War IIScience and Ethics: The Manhattan Project during World War II
Science and Ethics: The Manhattan Project during World War II
 
Kernel Descriptors for Visual Recognition
Kernel Descriptors for Visual RecognitionKernel Descriptors for Visual Recognition
Kernel Descriptors for Visual Recognition
 
Auction Portal
Auction PortalAuction Portal
Auction Portal
 
IIT JEE Seat Allocation System
IIT JEE Seat Allocation System IIT JEE Seat Allocation System
IIT JEE Seat Allocation System
 
Design and Fabrication of 4-bit processor
Design and Fabrication of  4-bit processorDesign and Fabrication of  4-bit processor
Design and Fabrication of 4-bit processor
 
Library Management System
Library  Management  SystemLibrary  Management  System
Library Management System
 
Interface for Finding Close Matches from Translation Memory
Interface for Finding Close Matches from Translation MemoryInterface for Finding Close Matches from Translation Memory
Interface for Finding Close Matches from Translation Memory
 
GCC RTL and Machine Description
GCC RTL and Machine DescriptionGCC RTL and Machine Description
GCC RTL and Machine Description
 
Literature Survey on Interest Points based Watermarking
Literature Survey on Interest Points based WatermarkingLiterature Survey on Interest Points based Watermarking
Literature Survey on Interest Points based Watermarking
 

Recently uploaded

A Seminar on Electric Vehicle Software Simulation
A Seminar on Electric Vehicle Software SimulationA Seminar on Electric Vehicle Software Simulation
A Seminar on Electric Vehicle Software SimulationMohsinKhanA
 
UNIT4_ESD_wfffffggggggggggggith_ARM.pptx
UNIT4_ESD_wfffffggggggggggggith_ARM.pptxUNIT4_ESD_wfffffggggggggggggith_ARM.pptx
UNIT4_ESD_wfffffggggggggggggith_ARM.pptxrealme6igamerr
 
IT3401-WEB ESSENTIALS PRESENTATIONS.pptx
IT3401-WEB ESSENTIALS PRESENTATIONS.pptxIT3401-WEB ESSENTIALS PRESENTATIONS.pptx
IT3401-WEB ESSENTIALS PRESENTATIONS.pptxSAJITHABANUS
 
The relationship between iot and communication technology
The relationship between iot and communication technologyThe relationship between iot and communication technology
The relationship between iot and communication technologyabdulkadirmukarram03
 
cloud computing notes for anna university syllabus
cloud computing notes for anna university syllabuscloud computing notes for anna university syllabus
cloud computing notes for anna university syllabusViolet Violet
 
Quasi-Stochastic Approximation: Algorithm Design Principles with Applications...
Quasi-Stochastic Approximation: Algorithm Design Principles with Applications...Quasi-Stochastic Approximation: Algorithm Design Principles with Applications...
Quasi-Stochastic Approximation: Algorithm Design Principles with Applications...Sean Meyn
 
Vertical- Machining - Center - VMC -LMW-Machine-Tool-Division.pptx
Vertical- Machining - Center - VMC -LMW-Machine-Tool-Division.pptxVertical- Machining - Center - VMC -LMW-Machine-Tool-Division.pptx
Vertical- Machining - Center - VMC -LMW-Machine-Tool-Division.pptxLMW Machine Tool Division
 
ChatGPT-and-Generative-AI-Landscape Working of generative ai search
ChatGPT-and-Generative-AI-Landscape Working of generative ai searchChatGPT-and-Generative-AI-Landscape Working of generative ai search
ChatGPT-and-Generative-AI-Landscape Working of generative ai searchrohitcse52
 
nvidia AI-gtc 2024 partial slide deck.pptx
nvidia AI-gtc 2024 partial slide deck.pptxnvidia AI-gtc 2024 partial slide deck.pptx
nvidia AI-gtc 2024 partial slide deck.pptxjasonsedano2
 
Guardians and Glitches: Navigating the Duality of Gen AI in AppSec
Guardians and Glitches: Navigating the Duality of Gen AI in AppSecGuardians and Glitches: Navigating the Duality of Gen AI in AppSec
Guardians and Glitches: Navigating the Duality of Gen AI in AppSecTrupti Shiralkar, CISSP
 
Summer training report on BUILDING CONSTRUCTION for DIPLOMA Students.pdf
Summer training report on BUILDING CONSTRUCTION for DIPLOMA Students.pdfSummer training report on BUILDING CONSTRUCTION for DIPLOMA Students.pdf
Summer training report on BUILDING CONSTRUCTION for DIPLOMA Students.pdfNaveenVerma126
 
sdfsadopkjpiosufoiasdoifjasldkjfl a asldkjflaskdjflkjsdsdf
sdfsadopkjpiosufoiasdoifjasldkjfl a asldkjflaskdjflkjsdsdfsdfsadopkjpiosufoiasdoifjasldkjfl a asldkjflaskdjflkjsdsdf
sdfsadopkjpiosufoiasdoifjasldkjfl a asldkjflaskdjflkjsdsdfJulia Kaye
 
Engineering Mechanics Chapter 5 Equilibrium of a Rigid Body
Engineering Mechanics  Chapter 5  Equilibrium of a Rigid BodyEngineering Mechanics  Chapter 5  Equilibrium of a Rigid Body
Engineering Mechanics Chapter 5 Equilibrium of a Rigid BodyAhmadHajasad2
 
Phase noise transfer functions.pptx
Phase noise transfer      functions.pptxPhase noise transfer      functions.pptx
Phase noise transfer functions.pptxSaiGouthamSunkara
 
GENERAL CONDITIONS FOR CONTRACTS OF CIVIL ENGINEERING WORKS
GENERAL CONDITIONS  FOR  CONTRACTS OF CIVIL ENGINEERING WORKS GENERAL CONDITIONS  FOR  CONTRACTS OF CIVIL ENGINEERING WORKS
GENERAL CONDITIONS FOR CONTRACTS OF CIVIL ENGINEERING WORKS Bahzad5
 
Nodal seismic construction requirements.pptx
Nodal seismic construction requirements.pptxNodal seismic construction requirements.pptx
Nodal seismic construction requirements.pptxwendy cai
 
How to Write a Good Scientific Paper.pdf
How to Write a Good Scientific Paper.pdfHow to Write a Good Scientific Paper.pdf
How to Write a Good Scientific Paper.pdfRedhwan Qasem Shaddad
 

Recently uploaded (20)

A Seminar on Electric Vehicle Software Simulation
A Seminar on Electric Vehicle Software SimulationA Seminar on Electric Vehicle Software Simulation
A Seminar on Electric Vehicle Software Simulation
 
UNIT4_ESD_wfffffggggggggggggith_ARM.pptx
UNIT4_ESD_wfffffggggggggggggith_ARM.pptxUNIT4_ESD_wfffffggggggggggggith_ARM.pptx
UNIT4_ESD_wfffffggggggggggggith_ARM.pptx
 
IT3401-WEB ESSENTIALS PRESENTATIONS.pptx
IT3401-WEB ESSENTIALS PRESENTATIONS.pptxIT3401-WEB ESSENTIALS PRESENTATIONS.pptx
IT3401-WEB ESSENTIALS PRESENTATIONS.pptx
 
Lecture 2 .pptx
Lecture 2                            .pptxLecture 2                            .pptx
Lecture 2 .pptx
 
Litature Review: Research Paper work for Engineering
Litature Review: Research Paper work for EngineeringLitature Review: Research Paper work for Engineering
Litature Review: Research Paper work for Engineering
 
The relationship between iot and communication technology
The relationship between iot and communication technologyThe relationship between iot and communication technology
The relationship between iot and communication technology
 
cloud computing notes for anna university syllabus
cloud computing notes for anna university syllabuscloud computing notes for anna university syllabus
cloud computing notes for anna university syllabus
 
Quasi-Stochastic Approximation: Algorithm Design Principles with Applications...
Quasi-Stochastic Approximation: Algorithm Design Principles with Applications...Quasi-Stochastic Approximation: Algorithm Design Principles with Applications...
Quasi-Stochastic Approximation: Algorithm Design Principles with Applications...
 
Vertical- Machining - Center - VMC -LMW-Machine-Tool-Division.pptx
Vertical- Machining - Center - VMC -LMW-Machine-Tool-Division.pptxVertical- Machining - Center - VMC -LMW-Machine-Tool-Division.pptx
Vertical- Machining - Center - VMC -LMW-Machine-Tool-Division.pptx
 
ChatGPT-and-Generative-AI-Landscape Working of generative ai search
ChatGPT-and-Generative-AI-Landscape Working of generative ai searchChatGPT-and-Generative-AI-Landscape Working of generative ai search
ChatGPT-and-Generative-AI-Landscape Working of generative ai search
 
nvidia AI-gtc 2024 partial slide deck.pptx
nvidia AI-gtc 2024 partial slide deck.pptxnvidia AI-gtc 2024 partial slide deck.pptx
nvidia AI-gtc 2024 partial slide deck.pptx
 
Guardians and Glitches: Navigating the Duality of Gen AI in AppSec
Guardians and Glitches: Navigating the Duality of Gen AI in AppSecGuardians and Glitches: Navigating the Duality of Gen AI in AppSec
Guardians and Glitches: Navigating the Duality of Gen AI in AppSec
 
Summer training report on BUILDING CONSTRUCTION for DIPLOMA Students.pdf
Summer training report on BUILDING CONSTRUCTION for DIPLOMA Students.pdfSummer training report on BUILDING CONSTRUCTION for DIPLOMA Students.pdf
Summer training report on BUILDING CONSTRUCTION for DIPLOMA Students.pdf
 
sdfsadopkjpiosufoiasdoifjasldkjfl a asldkjflaskdjflkjsdsdf
sdfsadopkjpiosufoiasdoifjasldkjfl a asldkjflaskdjflkjsdsdfsdfsadopkjpiosufoiasdoifjasldkjfl a asldkjflaskdjflkjsdsdf
sdfsadopkjpiosufoiasdoifjasldkjfl a asldkjflaskdjflkjsdsdf
 
計劃趕得上變化
計劃趕得上變化計劃趕得上變化
計劃趕得上變化
 
Engineering Mechanics Chapter 5 Equilibrium of a Rigid Body
Engineering Mechanics  Chapter 5  Equilibrium of a Rigid BodyEngineering Mechanics  Chapter 5  Equilibrium of a Rigid Body
Engineering Mechanics Chapter 5 Equilibrium of a Rigid Body
 
Phase noise transfer functions.pptx
Phase noise transfer      functions.pptxPhase noise transfer      functions.pptx
Phase noise transfer functions.pptx
 
GENERAL CONDITIONS FOR CONTRACTS OF CIVIL ENGINEERING WORKS
GENERAL CONDITIONS  FOR  CONTRACTS OF CIVIL ENGINEERING WORKS GENERAL CONDITIONS  FOR  CONTRACTS OF CIVIL ENGINEERING WORKS
GENERAL CONDITIONS FOR CONTRACTS OF CIVIL ENGINEERING WORKS
 
Nodal seismic construction requirements.pptx
Nodal seismic construction requirements.pptxNodal seismic construction requirements.pptx
Nodal seismic construction requirements.pptx
 
How to Write a Good Scientific Paper.pdf
How to Write a Good Scientific Paper.pdfHow to Write a Good Scientific Paper.pdf
How to Write a Good Scientific Paper.pdf
 

The problem of Spatio-Temporal Invariant Points in Videos

  • 1. Multimedia Systems - Class Project Group 2 Spatio-Temporal Invariant Points in Videos Members: Priyatham Bollimpalli – 10010148 Pydi Peddigari Venkat Sai – 10010149 PVS Dileep – 10010180
  • 2. The objective here is to find the spatio-temporal invariant points in a given input video. We implement the following models on a set of contiguous frames of a video, called a scene. We divide the problem into three cases, one with the background being fixed in a video and the entire scene is not dynamic, second, background fixed and the entire scene is reasonably dynamic, and final one with the background moving and the objects are also moving. We examine those cases below: Case1: When the Background is fixed and the entire scene is not dynamic In this case, the background in the scene is fixed across several frames, while the foreground objects can keep moving across the whole video, but they do not occupy the entire frame of video with their movements, i.e., only some parts of the frame would be having movement, while a decent part of the frame would remain static. The following procedure is followed to detect the spatio-temporal interest points in the scene:  Every scene is a collection of several frames. In this instance, we would consider a scene of a video in which the background is constant, and in the foreground, there is a ball which quickly moves underneath a wooden block which is constant in the video. These are just a four of the several frames present in the video:  The difference of the first two frames of the image would be computed. In the resulted difference image, we would have several ranges of pixel values possible.
  • 3. Hence, we would keep a grey threshold of say 0.2, and put all the locations of the frame which are above that grey threshold level, to pixel value of that of white. We also have an image which we will keep writing through across several iterations of this method(we refer to that image as Rframe). Now, in the previously resulted frame of difference, we find all the locations where the pixel value is white, and fill out all those corresponding locations on the Rframe to black.  We keep on repeating this process of finding the difference between two successive frames of the scene, thresholding the difference to get some pixel values which are white, and filling out all the locations on Rframe where there is a white pixel to black. For example, the difference of the frames at several instants are as follows:  Then, the final Rframe produced will be as follows: The portions which are black in this image would depict those points where a temporal invariance is not possible, as the objects keep moving in those areas. The portions which are white would indicate those points which are constant throughout the period of the video. These would be our points of interest for application of SIFT on those points.
  • 4.  On the final go, we would apply SIFT on the original frames of the scene, and find all those points resulted from SIFT, and consider such a point among them to be our interest point in this case only if that point is among the white portions of the Rframe, i.e., we would only want the interest points which coincide with the white portions of the Rframe. Hence, the resulted interest points finally would form our Spatio temporal invariant points in this case. The above algorithm is also run on other videos as follows:  Some of the several frames in the original video :  Some among the resulted differences in the frames at several instants are:
  • 5.  Then the Rframe resulted is as follows:  Now, we find the interest points at those locations, which are among the whie portions of the Rframe, since, only those would be the temporal invariant parts of the image. Hence, the interest points of the scene resulted at some of the several instants are as follows:
  • 6. So, the above points marked with green would represent the spatio-temporal interest points of the scene, at some of the instants among the whole.  It is run on another video as follows:  The Rframe resulted is :  Interest points resulted at several frames are :
  • 7. Case2: When the Background is fixed and the entire scene is reasonably dynamic In this case, the background in the scene is fixed across several frames, while the foreground objects can keep moving across the whole video, and they do occupy the most of the frame of video with their reasonable movements. In this case, the following procedure is followed:  Consider a scene as follows:  Now, if Case-1 was used here, the Rframe resulted would be as follows:
  • 8. Hence, if case-1 is used here, we can observe that most of the region is blacked out since the objects motion is present almost over the entire image, and hence, we lose some of the possible interest points.  Hence, we adopt the following method now. This method would use automatic detection and motion-based tracking of moving objects in a video. This problem can be seen as: o detecting moving objects in each frame o associating the detections corresponding to the same object over time  The association of detections to the same object is based solely on motion. The motion of each track is estimated by a Kalman filter. The filter is used to predict the track's location in each frame, and determine the likelihood of each detection being assigned to each track.  In any given frame, some detections may be assigned to tracks, while other detections and tracks may remain unassigned. The assigned tracks are updated using the corresponding detections. The unassigned tracks are marked invisible. Each track keeps count of the number of consecutive frames, where it remained unassigned. If the count exceeds a specified threshold, the example assumes that the object left the field of view and it deletes the track.  So in the process, a frame is read, objects are detected with their centroids and bounding boxes, and motion segmentation using the foreground detector. Next, the detections are assigned to tracks. Then, the assigned tracks would be update and the unassigned tracks would be updated by marking them invisible, and the lost tracks would be deleted.  The following are the results of the tracked objects at some of the instants of the scene:
  • 9.  Now, we apply SIFT on the tracked objects at all instants of the video. Then, the following would be the interest points produced at several instants of the video:  Now, to gather further more interest points from the video, we can also combine the interest points generated from case-1, and hence, the resulted interest points would be:
  • 10. So this would capture all the interest points possible, combining case-1 and case-2 Case3: When the background is moving and the objects are also moving When the camera as well as well objects are moving, tracking the objects is a challenging issue. There will be rarely anything invariant even in one particular scene. This is still an open research problem and some heuristic methods are successful. The video has a moving car with a moving camera. Some of the frames are given below.
  • 11. One heuristic method which we found on the web to solve it is given below. Note that segmenting the car frame differentiation (background subtraction) won’t work because the camera is also moving henceforth the background is also moving. Hence normal prediction algorithms would fail in this case. To tackle this issue, Optical flow is used where the scene and the car have different directions of flow. In the below figures, the red points denote the optical flow of the background and the green points denote the points having an optical flow opposite to the red points. Note that the green points are able to track the car present in the entire video and hence locate the green points which are spatio-temporally invariant. Now, combination of the points obtained here with those obtained in case-1 and case-2 (it is less likely that any points would be there), gives the total points possible.
  • 12. Conclusion & Future Work: In this report we have tried to perform three kinds of techniques on any given scene in a video – when the background is almost stationary and the scene is not dynamic, when the background is stationary and the scene is dynamic and the background and the scene bother are dynamic. Combination of points obtained from all the three methods gives maximum possible spatio-temporal invariant points. There is lot of scope for future work in this area and we wish to pursue it further. The following are the issues involved.  In case-1, the level at which thresholding is done defines the extent/degree to which motion of the object is considered. Lower the threshold, greater the impact of motion. This entirely depends on the video i.e. if a video has lot of illumination and contrast changes between the frames, then the difference of the frames would give many false contours. In this case considering higher value of threshoding is desirable. In some other cases the frame rate would be too high due to which almost negligible amount of motion would be captured between the frames. In this case lower value for thresholding is desirable. Hence developing an optimal threshold value automatically by taking the video quality, frame rate into account is one area of future work.  In case-3, the above example gives a possible approach to solution. It works because the camera is also continuously moving along with the car and getting the points with opposite optical flow works. But if the camera remains stationary for some time and moves again suddenly, we need a separate system to first track the background movement and that in combination of foreground motion can be used to find out the points on the object. Many object tracking methods exist and many are still pursed since this is a very active area of research. It is likely that exploring into this area would give a generic algorithm for obtaining an object and the points which are consistent on it throughout the scene.  Case-2 considers the background to be stationary and the motion of the objects to be uniform. Modifying the parameters for the detection, assignment, and deletion steps of the trackers according to the video may be done. The tracking in this example was solely based on motion with the assumption that all objects move in a straight line with constant speed. When the motion of an object significantly deviates from this model, the example may produce tracking errors. The likelihood of tracking errors can be reduced by using a more complex motion model, such as constant acceleration, or by using multiple Kalman filters for every object. Also, you can incorporate other cues for associating detections over time, such as size, shape, and colour. References 1. http://www.mathworks.in/help/vision/examples/motion-based-multiple-object- tracking.html 2. http://www.youtube.com/watch?v=MOaKnCSejXQ