Deep Learning - a Path from Big Data Indexing to Robotic ApplicationsDarius Burschka
These are the slides to my ShanghAI lecture from Dec 10, 2020. It proposes necessary extensions to make DeepNets appropriate tools for robotic systems.
The talk can be found on https://fb.watch/2hXDC6K4Pq/
Robust and Efficient Coupling of Perception to Actuation with Metric and Non-...Darius Burschka
The talk motivates a re-thinking of the way, how perception passes the information to the control modules. Metric information is not a native space of the camera and apparently also not used in biology for navigation. Early abstraction of information from images loses a lot of important information that can be directly used for following (Visual-Servoing), motion estimation(Motion Blurr), and collision relations(Optical Flow Clustering). I present in this talk ways, how we use the image information in "classical way" that does not require any learning and runs on low-power CPUs.
Visual Mapping and Collision Avoidance Dynamic Environments in Dynamic Enviro...Darius Burschka
How conventional vision is more appropriate for control since it provides also error analysis. There is a lot of information in the images that is lost when converting to 3D
Deconstructing SfM-Net architecture and beyond
"SfM-Net, a geometry-aware neural network for motion estimation in videos that decomposes frame-to-frame pixel motion in terms of scene and object depth, camera motion and 3D object rotations and translations. Given a sequence of frames, SfM-Net predicts depth, segmentation, camera and rigid object motions, converts those into a dense frame-to-frame motion field (optical flow), differentiably warps frames in time to match pixels and back-propagates."
Alternative download:
https://www.dropbox.com/s/aezl7ro8sy2xq7j/sfm_net_v2.pdf?dl=0
Event recognition image & video segmentationeSAT Journals
Abstract This paper gives a clear look at the segmentation process at the basic level. Segmentation is done at multiple levels so that we will get different results. Segmentation of relative motion descriptors gives a clear picture about the segmentation done for a given input video. Relative motion computation and histograms incrementation are used to evaluate this approach. Also here we will give complete information about the related research which is done about how segmentation can be done for the both images and videos. Keywords: Image Segmentation, Video Segmentation.
Deep Learning - a Path from Big Data Indexing to Robotic ApplicationsDarius Burschka
These are the slides to my ShanghAI lecture from Dec 10, 2020. It proposes necessary extensions to make DeepNets appropriate tools for robotic systems.
The talk can be found on https://fb.watch/2hXDC6K4Pq/
Robust and Efficient Coupling of Perception to Actuation with Metric and Non-...Darius Burschka
The talk motivates a re-thinking of the way, how perception passes the information to the control modules. Metric information is not a native space of the camera and apparently also not used in biology for navigation. Early abstraction of information from images loses a lot of important information that can be directly used for following (Visual-Servoing), motion estimation(Motion Blurr), and collision relations(Optical Flow Clustering). I present in this talk ways, how we use the image information in "classical way" that does not require any learning and runs on low-power CPUs.
Visual Mapping and Collision Avoidance Dynamic Environments in Dynamic Enviro...Darius Burschka
How conventional vision is more appropriate for control since it provides also error analysis. There is a lot of information in the images that is lost when converting to 3D
Deconstructing SfM-Net architecture and beyond
"SfM-Net, a geometry-aware neural network for motion estimation in videos that decomposes frame-to-frame pixel motion in terms of scene and object depth, camera motion and 3D object rotations and translations. Given a sequence of frames, SfM-Net predicts depth, segmentation, camera and rigid object motions, converts those into a dense frame-to-frame motion field (optical flow), differentiably warps frames in time to match pixels and back-propagates."
Alternative download:
https://www.dropbox.com/s/aezl7ro8sy2xq7j/sfm_net_v2.pdf?dl=0
Event recognition image & video segmentationeSAT Journals
Abstract This paper gives a clear look at the segmentation process at the basic level. Segmentation is done at multiple levels so that we will get different results. Segmentation of relative motion descriptors gives a clear picture about the segmentation done for a given input video. Relative motion computation and histograms incrementation are used to evaluate this approach. Also here we will give complete information about the related research which is done about how segmentation can be done for the both images and videos. Keywords: Image Segmentation, Video Segmentation.
Inclined Image Recognition for Aerial Mapping using Deep Learning and Tree ba...TELKOMNIKA JOURNAL
One of the important capabilities of an unmanned aerial vehicle (UAV) is aerial mapping. Aerial mapping is an image registration problem, i.e., the problem of transforming different sets of images into one coordinate system. In image registration, the quality of the output is strongly influenced by the quality of input (i.e., images captured by the UAV). Therefore, selecting the quality of input images becomes important and one of the challenging task in aerial mapping because the ground truth in the mapping process is not given before the UAV flies. Typically, UAV takes images in sequence irrespective of its flight orientation and roll angle. These may result in the acquisition of bad quality images, possibly compromising the quality of mapping results, and increasing the computational cost of a registration process. To address these issues, we need a recognition system that is able to recognize images that are not suitable for the registration process. In this paper, we define these unsuitable images as “inclined images,” i.e., images captured by UAV that are not perpendicular to the ground. Although we can calculate the inclination angle using a gyroscope attached to the UAV, our interest here is to recognize these inclined images without the use of additional sensors in order to mimic how humans perform this task visually. To realize that, we utilize a deep learning method with the combination of tree-based models to build an inclined image recognition system. We have validated the proposed system with the images captured by the UAV. We collected 192 images and labelled them with two different levels of classes (i.e., coarse- and fine-classification). We compared this with several models and the results showed that our proposed system yielded an improvement of accuracy rate up to 3%.
Simultaneous Mapping and Navigation For Rendezvous in Space ApplicationsNandakishor Jahagirdar
To design and develop an image processing algorithm that can identify the target spacecraft docking station as well as the distance, location and angle of the docking station with respect to the chaser vehicle. Making a use of the image from single camera.
V. Caglioti, A. Giusti, A. Riva, M. Uberti: "Drawing Motion without Understanding It".
Proc. of International Symposium on Visual Computing (ISVC) 2009. Oral presentation (acceptance rate ~30%). Volt / Microsoft MSDN Best Paper Award.
Image restoration techniques covered such as denoising, deblurring and super-resolution for 3D images and models.
From classical computer vision techniques to contemporary deep learning based processing for both ordered and unordered point clouds, depth maps and meshes.
Falling costs with rising quality via hardware innovations and deep learning.
Technical introduction for scanning technologies from Structure-from-Motion (SfM), Range sensing (e.g. Kinect and Matterport) to Laser scanning (e.g. LiDAR), and the associated traditional and deep learning-based processing techniques.
Note! Due to small font size, and bad rendering by SlideShare, better to download the slides locally to your device
Alternative download link for the PDF:
https://www.dropbox.com/s/eclyy45k3gz66ve/proptech_emergingScanningTech.pdf?dl=0
Real Time Object Identification for Intelligent Video Surveillance ApplicationsEditor IJCATR
Intelligent video surveillance system has emerged as a very important research topic in the computer vision field in the
recent years. It is well suited for a broad range of applications such as to monitor activities at traffic intersections for detecting
congestions and predict the traffic flow. Object classification in the field of video surveillance is a key component of smart
surveillance software. Two robust methodology and algorithms adopted for people and object classification for automated surveillance
systems is proposed in this paper. First method uses background subtraction model for detecting the object motion. The background
subtraction and image segmentation based on morphological transformation for tracking and object classification on highways is
proposed. This algorithm uses erosion followed by dilation on various frames. Proposed algorithm in first method, segments the image
by preserving important edges which improves the adaptive background mixture model and makes the system learn faster and more
accurately. The system used in second method adopts the object detection method without background subtraction because of the static
object detection. Segmentation is done by the bounding box registration technique. Then the classification is done with the multiclass
SVM using the edge histogram as features. The edge histograms are calculated for various bin values in different environment. The
result obtained demonstrates the effectiveness of the proposed approach.
Serene 2015
Davide Scaramuzza
Abstract: With drones becoming more and more popular, safety is a big concern. A critical situation occurs when a drone temporarily loses its GPS position information, which might lead it to crash. This can happen, for instance, when flying close to buildings where GPS signal is lost. In such situations, it is desirable that the drone can rely on fall-back systems and regain stable flight as soon as possible. In this talk, I will present novel methods to automatically recover and stabilize a quadrotor from any initial condition or execute emergency landing. On the one hand, this new technology will allow quadrotors to be launched by simply tossing them in the air, like a “baseball ball”. On the other hand, it will allow them to recover back into stable flight or land on a safe area after a system failure. Since this technology does not rely on any external infrastructure, such as GPS, it enables the safe use of drones in both indoor and outdoor environments. Thus, it can become relevant for commercial use of drones, such as parcel delivery.
Recent videos:
Automatic failure recovery without GPS: https://youtu.be/pGU1s6Y55JI
Autonomous Landing-site detection and landing: https://youtu.be/phaBKFwfcJ4
IEEE CASE 2016 On Avoiding Moving Objects for Indoor Autonomous QuadrotorsPeter SHIN
Abstract - A mini quadrotor can be used in many applica- tions, such as indoor airborne surveillance, payload delivery, and warehouse monitoring. In these applications, vision-based autonomous navigation is one of the most interesting research topics because precise navigation can be implemented based on vision analysis. However, pixel-based vision analysis approaches require a high-powered computer, which is inappropriate to be attached to a small indoor quadrotor. This paper proposes a method called the Motion-vector-based Moving Objects Detec- tion. This method detects and avoids obstacles using stereo motion vectors instead of individual pixels, thereby substan- tially reducing the data processing requirement. Although this method can also be used in the avoidance of stationary obstacles by taking into account the ego-motion of the quadrotor, this paper primarily focuses on providing our empirical verification on the real-time avoidance of moving objects.
Inclined Image Recognition for Aerial Mapping using Deep Learning and Tree ba...TELKOMNIKA JOURNAL
One of the important capabilities of an unmanned aerial vehicle (UAV) is aerial mapping. Aerial mapping is an image registration problem, i.e., the problem of transforming different sets of images into one coordinate system. In image registration, the quality of the output is strongly influenced by the quality of input (i.e., images captured by the UAV). Therefore, selecting the quality of input images becomes important and one of the challenging task in aerial mapping because the ground truth in the mapping process is not given before the UAV flies. Typically, UAV takes images in sequence irrespective of its flight orientation and roll angle. These may result in the acquisition of bad quality images, possibly compromising the quality of mapping results, and increasing the computational cost of a registration process. To address these issues, we need a recognition system that is able to recognize images that are not suitable for the registration process. In this paper, we define these unsuitable images as “inclined images,” i.e., images captured by UAV that are not perpendicular to the ground. Although we can calculate the inclination angle using a gyroscope attached to the UAV, our interest here is to recognize these inclined images without the use of additional sensors in order to mimic how humans perform this task visually. To realize that, we utilize a deep learning method with the combination of tree-based models to build an inclined image recognition system. We have validated the proposed system with the images captured by the UAV. We collected 192 images and labelled them with two different levels of classes (i.e., coarse- and fine-classification). We compared this with several models and the results showed that our proposed system yielded an improvement of accuracy rate up to 3%.
Simultaneous Mapping and Navigation For Rendezvous in Space ApplicationsNandakishor Jahagirdar
To design and develop an image processing algorithm that can identify the target spacecraft docking station as well as the distance, location and angle of the docking station with respect to the chaser vehicle. Making a use of the image from single camera.
V. Caglioti, A. Giusti, A. Riva, M. Uberti: "Drawing Motion without Understanding It".
Proc. of International Symposium on Visual Computing (ISVC) 2009. Oral presentation (acceptance rate ~30%). Volt / Microsoft MSDN Best Paper Award.
Image restoration techniques covered such as denoising, deblurring and super-resolution for 3D images and models.
From classical computer vision techniques to contemporary deep learning based processing for both ordered and unordered point clouds, depth maps and meshes.
Falling costs with rising quality via hardware innovations and deep learning.
Technical introduction for scanning technologies from Structure-from-Motion (SfM), Range sensing (e.g. Kinect and Matterport) to Laser scanning (e.g. LiDAR), and the associated traditional and deep learning-based processing techniques.
Note! Due to small font size, and bad rendering by SlideShare, better to download the slides locally to your device
Alternative download link for the PDF:
https://www.dropbox.com/s/eclyy45k3gz66ve/proptech_emergingScanningTech.pdf?dl=0
Real Time Object Identification for Intelligent Video Surveillance ApplicationsEditor IJCATR
Intelligent video surveillance system has emerged as a very important research topic in the computer vision field in the
recent years. It is well suited for a broad range of applications such as to monitor activities at traffic intersections for detecting
congestions and predict the traffic flow. Object classification in the field of video surveillance is a key component of smart
surveillance software. Two robust methodology and algorithms adopted for people and object classification for automated surveillance
systems is proposed in this paper. First method uses background subtraction model for detecting the object motion. The background
subtraction and image segmentation based on morphological transformation for tracking and object classification on highways is
proposed. This algorithm uses erosion followed by dilation on various frames. Proposed algorithm in first method, segments the image
by preserving important edges which improves the adaptive background mixture model and makes the system learn faster and more
accurately. The system used in second method adopts the object detection method without background subtraction because of the static
object detection. Segmentation is done by the bounding box registration technique. Then the classification is done with the multiclass
SVM using the edge histogram as features. The edge histograms are calculated for various bin values in different environment. The
result obtained demonstrates the effectiveness of the proposed approach.
Serene 2015
Davide Scaramuzza
Abstract: With drones becoming more and more popular, safety is a big concern. A critical situation occurs when a drone temporarily loses its GPS position information, which might lead it to crash. This can happen, for instance, when flying close to buildings where GPS signal is lost. In such situations, it is desirable that the drone can rely on fall-back systems and regain stable flight as soon as possible. In this talk, I will present novel methods to automatically recover and stabilize a quadrotor from any initial condition or execute emergency landing. On the one hand, this new technology will allow quadrotors to be launched by simply tossing them in the air, like a “baseball ball”. On the other hand, it will allow them to recover back into stable flight or land on a safe area after a system failure. Since this technology does not rely on any external infrastructure, such as GPS, it enables the safe use of drones in both indoor and outdoor environments. Thus, it can become relevant for commercial use of drones, such as parcel delivery.
Recent videos:
Automatic failure recovery without GPS: https://youtu.be/pGU1s6Y55JI
Autonomous Landing-site detection and landing: https://youtu.be/phaBKFwfcJ4
IEEE CASE 2016 On Avoiding Moving Objects for Indoor Autonomous QuadrotorsPeter SHIN
Abstract - A mini quadrotor can be used in many applica- tions, such as indoor airborne surveillance, payload delivery, and warehouse monitoring. In these applications, vision-based autonomous navigation is one of the most interesting research topics because precise navigation can be implemented based on vision analysis. However, pixel-based vision analysis approaches require a high-powered computer, which is inappropriate to be attached to a small indoor quadrotor. This paper proposes a method called the Motion-vector-based Moving Objects Detec- tion. This method detects and avoids obstacles using stereo motion vectors instead of individual pixels, thereby substan- tially reducing the data processing requirement. Although this method can also be used in the avoidance of stationary obstacles by taking into account the ego-motion of the quadrotor, this paper primarily focuses on providing our empirical verification on the real-time avoidance of moving objects.
Real Time Detection of Moving Object Based on Fpgaiosrjce
IOSR Journal of Electronics and Communication Engineering(IOSR-JECE) is a double blind peer reviewed International Journal that provides rapid publication (within a month) of articles in all areas of electronics and communication engineering and its applications. The journal welcomes publications of high quality papers on theoretical developments and practical applications in electronics and communication engineering. Original research papers, state-of-the-art reviews, and high quality technical notes are invited for publications.
Hand gesture recognition using support vector machinetheijes
The International Journal of Engineering & Science is aimed at providing a platform for researchers, engineers, scientists, or educators to publish their original research results, to exchange new ideas, to disseminate information in innovative designs, engineering experiences and technological skills. It is also the Journal's objective to promote engineering and technology education. All papers submitted to the Journal will be blind peer-reviewed. Only original articles will be published.
Real-time Moving Object Detection using SURFiosrjce
IOSR Journal of Computer Engineering (IOSR-JCE) is a double blind peer reviewed International Journal that provides rapid publication (within a month) of articles in all areas of computer engineering and its applications. The journal welcomes publications of high quality papers on theoretical developments and practical applications in computer technology. Original research papers, state-of-the-art reviews, and high quality technical notes are invited for publications.
Similar to Semantic Perception for Telemanipulation at SPME Workshop at ICRA 2013 (20)
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Le nuove frontiere dell'AI nell'RPA con UiPath Autopilot™UiPathCommunity
In questo evento online gratuito, organizzato dalla Community Italiana di UiPath, potrai esplorare le nuove funzionalità di Autopilot, il tool che integra l'Intelligenza Artificiale nei processi di sviluppo e utilizzo delle Automazioni.
📕 Vedremo insieme alcuni esempi dell'utilizzo di Autopilot in diversi tool della Suite UiPath:
Autopilot per Studio Web
Autopilot per Studio
Autopilot per Apps
Clipboard AI
GenAI applicata alla Document Understanding
👨🏫👨💻 Speakers:
Stefano Negro, UiPath MVPx3, RPA Tech Lead @ BSP Consultant
Flavio Martinelli, UiPath MVP 2023, Technical Account Manager @UiPath
Andrei Tasca, RPA Solutions Team Lead @NTT Data
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdfPeter Spielvogel
Building better applications for business users with SAP Fiori.
• What is SAP Fiori and why it matters to you
• How a better user experience drives measurable business benefits
• How to get started with SAP Fiori today
• How SAP Fiori elements accelerates application development
• How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities
• How SAP Fiori paves the way for using AI in SAP apps
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
3. DariusBurschka–MVPGroupatTUM
http://mvp.visual-navigation.com SPME Workshop, May 5, 2013
ASCENT – Augmented Shared-Control for
Efficient Natural Telemanipulation
(ICRA 2013 J. Bohren et al. Teleoperation WeF6 5:45pm Clubraum)
Fig. 1: The experiments were conducted with a human operator at The Johns Hopkins University (JHU) Homewood Campus
in Baltimore, MD, USA, utilizing a da Vinci R
Master Console (left) commanding a DLR LWR as part of the SAPHARI
platform at the German Aerospace Center (DLR) in Oberpfaffenhofen, Germany (right).
• Many remote telerobotic applications have limitations
on bandwidth, creating a situation where the fidelity
of the imaging is compromised. The availability of
stereoscopic imaging, image resolution and frame rates
may be limited, leading to a limited ability to resolve
necessary detail for manipulation. This is particularly
challenging given the absence of haptic cues noted
above increases the reliance on visual perception.
• Some environments impose additional communication
latency (time-delay) on telemetry as well. For example,
telemanipulation from Earth to low-earth orbit typically
imposes delays that exceed half a second for direct line-
of-sight communications and 2-7 seconds when using
larger-coverage on-orbit communications networks. The
limitations of human performance in telemanipulation
constrained circumstances. ASCENT takes a collaborative
systems approach that transcends the limitations of either
purely autonomous or purely teleoperated control modes by
combining task-specific sensor-based feedback with input
from an operator. As a result, the operator is able to provide
gross motion guidance to the system, and the remote manip-
ulator is able to adapt that motion based on environmental
information. We have implemented this approach with a
DLR lightweight arm driven by a da Vinci R
S master
console separated by over 4000 miles. We demonstrate
that ASCENT greatly improves manipulation performance,
particularly when subtle motions are necessary in order to
correctly perform the task.
II. BACKGROUND
Problems:
• Depth perception is essential for grasping
• Limited bandwidth does not always allow remote image
transmission
• Significant latency in transmission deteriorates dexterity
of the control
• Moving objects in the scene limit the allowed latency in
the control for robust direct manipulation in remote
environments
14. DariusBurschka–MVPGroupatTUM
http://mvp.visual-navigation.com SPME Workshop, May 5, 2013
Hybrid Model of the Environment (JC Ramirez)
Object
Container
3D
reconstruction
&
plane
detection
Blob
Detection
FUSION
Object
Layer
Geometric
Layer
Sensor
Blobs
3D Data
MAP
Objects 3D Structure
Geometric
Blobs
Map
Update
System
Input Data Stream Output Data Stream
15. DariusBurschka–MVPGroupatTUM
http://mvp.visual-navigation.com SPME Workshop, May 5, 2013
World model saves additional info, like texture,
motion, etc. (VISAPP 2013 J.Ramirez et al.)
Juan Carlos Ramirez and Darius B
Faculty for Informatics, Technische Universitaet Muenchen, Boltzman
ramirezd@in.tum.de, burschka@cs.
INTRODUCTION
Scene Tentative object candidates Encapsula
An approach to consistently model and characterize potential object candidate
Three principal procedures support our method:
i) the segmentation of the captured range images into 3D clusters or blobs, b
the spatial structure of the scene,
ii) the maintenance and reliability of the map, which are obtained through the
which we assign a degree of existence (confidence value),
iii) the visual motion estimation of potential object candidates, through the com
information, allows not only to update the state of the actors and perceive t
and refine their individual 3D structures over time.
Juan Carlos Ramirez and Darius Burschka
formatics, Technische Universitaet Muenchen, Boltzmannstr. 3, Garching bei Muenc
ramirezd@in.tum.de, burschka@cs.tum.edu
INTRODUCTION
Tentative object candidates Encapsulated 3D blobs Motion
consistently model and characterize potential object candidates presented in non-static scene
procedures support our method:
tion of the captured range images into 3D clusters or blobs, by which we obtain a first gross i
ucture of the scene,
nce and reliability of the map, which are obtained through the fusion of the captured and map
ign a degree of existence (confidence value),
tion estimation of potential object candidates, through the combination of the texture and 3D-
allows not only to update the state of the actors and perceive their changes in a scene, but als
eir individual 3D structures over time.
D Mapping
or 3D Structures in Dynamic Environments
and Darius Burschka
hen, Boltzmannstr. 3, Garching bei Muenchen, Germany
urschka@cs.tum.edu
UCTION
Encapsulated 3D blobs Motion estimation
bject candidates presented in non-static scenes.
17. DariusBurschka–MVPGroupatTUM
http://mvp.visual-navigation.com SPME Workshop, May 5, 2013
Local Feature Tracking Algorithms
• Image-gradient based à Extended KLT (ExtKLT)
• patch-based implementation
• feature propagation
• corner-binding
+ sub-pixel accuracy
• algorithm scales bad with number
of features
• Tracking-By-Matching à AGAST tracker
• AGAST corner detector
• efficient descriptor
• high frame-rates (hundrets of
features in a few milliseconds)
+ algorithm scales well with number
of features
• pixel-accuracy
8
18. DariusBurschka–MVPGroupatTUM
http://mvp.visual-navigation.com SPME Workshop, May 5, 2013
Adaptive and Generic Accelerated Segment Test
(AGAST)
9
Improvements compared to FAST:
• full exploration of the configuration space by backward-induction (no
learning)
• binary decision tree (not ternary)
• computation of the actual probability and processing costs
(no greedy algorithm)
• automatic scene adaption by tree switching (at no cost)
• various corner pattern sizes (not just one)
No drawbacks!
Mair, Hager, Burschka, Suppa, Hirzinger
ECCV, Springer, 2010
E. Rosten
23. DariusBurschka–MVPGroupatTUM
http://mvp.visual-navigation.com SPME Workshop, May 5, 2013
Each tool used in the
procedure has its own
container describing its
shape, handling properties
etc.
Knowledge Representation
Functionality map for a specific
procedure describes the way
how the tool was used during
the procedure while moved
between points in the world(Petsch/Burschka IROS2011)
26. DariusBurschka–MVPGroupatTUM
http://mvp.visual-navigation.com SPME Workshop, May 5, 2013
MachineVisionandPerceptionGroup@TUM Knowledge Representation
Atlas:
– Long-term memory
– Experience of the system
Working memory:
– Short-term memory
– Experience grounded in a given
environment
• Temporal handling information
27. DariusBurschka–MVPGroupatTUM
http://mvp.visual-navigation.com SPME Workshop, May 5, 2013
Conclusions
Fig. 1: The experiments were conducted with a human operator at The Johns Hopkins University (JHU) Homewood Campus
in Baltimore, MD, USA, utilizing a da Vinci R
Master Console (left) commanding a DLR LWR as part of the SAPHARI
platform at the German Aerospace Center (DLR) in Oberpfaffenhofen, Germany (right).
• Many remote telerobotic applications have limitations
on bandwidth, creating a situation where the fidelity
of the imaging is compromised. The availability of
stereoscopic imaging, image resolution and frame rates
may be limited, leading to a limited ability to resolve
necessary detail for manipulation. This is particularly
challenging given the absence of haptic cues noted
above increases the reliance on visual perception.
• Some environments impose additional communication
latency (time-delay) on telemetry as well. For example,
telemanipulation from Earth to low-earth orbit typically
imposes delays that exceed half a second for direct line-
of-sight communications and 2-7 seconds when using
larger-coverage on-orbit communications networks. The
limitations of human performance in telemanipulation
are well-studied, and the threshold at which human
performance begins to suffer is far below that [12].
constrained circumstances. ASCENT takes a collaborative
systems approach that transcends the limitations of either
purely autonomous or purely teleoperated control modes by
combining task-specific sensor-based feedback with input
from an operator. As a result, the operator is able to provide
gross motion guidance to the system, and the remote manip-
ulator is able to adapt that motion based on environmental
information. We have implemented this approach with a
DLR lightweight arm driven by a da Vinci R
S master
console separated by over 4000 miles. We demonstrate
that ASCENT greatly improves manipulation performance,
particularly when subtle motions are necessary in order to
correctly perform the task.
II. BACKGROUND
Presently, robots that are deployed to perform high-value
tasks usually fall into two broad categories:
Why is perception necessary:
• Allows data reduction over slow links. In worst case,
just symbolic information about objects in the scene
• Allows together with motion estimation a transparent
switch between direct control and autonomous handling
• Allows to deal with the problem with high latencies and
fast motions in the scene
..Questions?
28. DariusBurschka–MVPGroupatTUM
http://mvp.visual-navigation.com SPME Workshop, May 5, 2013
MachineVisionandPerceptionGroup@TUMMVP
Research of the MVP Group http://mvp.visual-navigation.com
The Machine Vision and
Perception Group @TUM works
on the aspects of visual
perception and control in
medical, mobile, and HCI
applications
Visual navigation
Biologically motivated
perception
Perception for manipulation
Visual Action Analysis
Photogrammetric monocular
reconstruction
Rigid and Deformable
Registration
29. DariusBurschka–MVPGroupatTUM
http://mvp.visual-navigation.com SPME Workshop, May 5, 2013
MachineVisionandPerceptionGroup@TUMMVP
Research of the MVP Group http://mvp.visual-navigation.com
Exploration of physical
object properties
Sensor substitution
Multimodal Sensor
Fusion
Development of new
Optical Sensors
The Machine Vision and
Perception Group @TUM works
on the aspects of visual
perception and control in
medical, mobile, and HCI
applications