Lecture 4 from the COMP 4010 course on AR/VR. This lecture reviews optical tracking for AR and starts discussion about interaction techniques. This was taught by Mark Billinghurst at the University of South Australia on August 17th 2021.
Lecture prepared by Mark Billinghurst on Augmented Reality tracking. Taught on October 18th 2016 by Dr. Gun Lee as part of the COMP 4010 VR class at the University of South Australia.
A lecture give on AR Tehchnology taught as part of the COMP 4010 course on AR/VR. This lecture was taught by Mark Billinghurst on August 10th 2021 at the University of South Australia.
Lecture 5 in the COMP 4010 class on Augmented and Virtual Reality. This lecture was about AR Interaction and Prototyping methods. Taught by Mark Billinghurst on August 24th 2021 at the University of South Australia.
Lecture 12 in the COMP 4010 course on AR/VR. This lecture was about research directions in AR/VR and in particular display research. This was taught by Mark Billinghurst on September 26th 2021 at the University of South Australia.
Lecture 4 in the 2022 COMP 4010 lecture series on AR/VR. This lecture is about AR Interaction techniques. This was taught by Mark Billinghurst at the University of South Australia in 2022.
Lecture 2 of the COMP 4010 class on AR/VR. This lecture is about the human perception system. This lecture was given on August 3rd 2021 by Mark Billinghurst from the University of South Australia.
COMP 4010 Lecture 9 providing an overview of Augmented Reality Technology. Taught by Mark Billinghurst on October 8th 2019 at the University of South Australia.
Lecture 9 of the COMP 4010 course on AR/VR. This lecture is about AR Interaction methods. Taught on October 2nd 2018 by Mark Billinghurst at the University of South Australia
Lecture prepared by Mark Billinghurst on Augmented Reality tracking. Taught on October 18th 2016 by Dr. Gun Lee as part of the COMP 4010 VR class at the University of South Australia.
A lecture give on AR Tehchnology taught as part of the COMP 4010 course on AR/VR. This lecture was taught by Mark Billinghurst on August 10th 2021 at the University of South Australia.
Lecture 5 in the COMP 4010 class on Augmented and Virtual Reality. This lecture was about AR Interaction and Prototyping methods. Taught by Mark Billinghurst on August 24th 2021 at the University of South Australia.
Lecture 12 in the COMP 4010 course on AR/VR. This lecture was about research directions in AR/VR and in particular display research. This was taught by Mark Billinghurst on September 26th 2021 at the University of South Australia.
Lecture 4 in the 2022 COMP 4010 lecture series on AR/VR. This lecture is about AR Interaction techniques. This was taught by Mark Billinghurst at the University of South Australia in 2022.
Lecture 2 of the COMP 4010 class on AR/VR. This lecture is about the human perception system. This lecture was given on August 3rd 2021 by Mark Billinghurst from the University of South Australia.
COMP 4010 Lecture 9 providing an overview of Augmented Reality Technology. Taught by Mark Billinghurst on October 8th 2019 at the University of South Australia.
Lecture 9 of the COMP 4010 course on AR/VR. This lecture is about AR Interaction methods. Taught on October 2nd 2018 by Mark Billinghurst at the University of South Australia
Lecture 3 in the 2022 COMP 4010 lecture series on AR/VR. This lecture provides an introduction for AR Technology. This was taught by Mark Billinghurst at the University of South Australia in 2022.
Lecture 2 in the 2022 COMP 4010 Lecture series on AR/VR and XR. This lecture is about human perception for AR/VR/XR experiences. This was taught by Mark Billinghurst at the University of South Australia in 2022.
Lecture 11 of the COMP 4010 class on Augmented Reality and Virtual Reality. This lecture is about VR applications and was taught by Mark Billinghurst on October 19th 2021 at the University of South Australia
Lecture 9 of the COMP 4010 course in AR/VR from the University of South Australia. This was taught by Mark Billinghurst on October 5th, 2021. This lecture describes VR input devices, VR systems and rapid prototyping tools.
Advanced Methods for User Evaluation in AR/VR StudiesMark Billinghurst
Guest lecture on advanced methods of user evaluation in AR/VR studies. Given by Mark Billinghurst as part of the ARIVE lecture series hosted at the University of Otago. The lecture was given on August 26th 2021.
Lecture 6 of the COMP 4010 course on AR/VR. This lecture is about designing AR systems. This was taught by Mark Billinghurst at the University of South Australia on September 1st 2022.
COMP4010 Lecture 4 - VR Technology - Visual and Haptic Displays. Lecture about VR visual and haptic display technology. Taught on August 16th 2016 by Mark Billinghurst from the University of South Australia
Lecture 1 for the 2022 COMP 4010 course on AR and VR. This course was taught by Mark Billinghurst at the University of South Australia in 2022. This lecture provides an introduction to AR, VR and XR.
Talk given by Mark Billinghurst at the DIGI_X conference in Auckland, New Zealand on June 21st 2018. The talk was about how Mixed Reality can be applied in the work place.
Talk given by Mark Billinghurst to Bajaj Finance Limited in India, on May 9th 2020. The talk describes AR and VR applications, example AR/VR applications in financial services, and potential research directions.
Lecture 8 of the COMP 4010 course taught at the University of South Australia. This lecture provides and introduction to VR technology. Taught by Mark Billinghurst on September 14th 2021 at the University of South Australia.
Lecture 3 from the COMP 4010 course and Virtual and Augmented Reality. This lecture is about VR tracking, input and systems. Taught on August 7th, 2018 by Mark Billinghurst at the University of South Australia
Lecture 5 in the 2022 COMP 4010 lecture series. This lecture is about AR prototyping tools and techniques. The lecture was given by Mark Billinghurst from University of South Australia in 2022.
keynote speech by Mark Billinghurst at the Workshop on Transitional Interfaces in Mixed and Cross-Reality, at the ACM ISS 2021 Conference. Given on November 14th 2021
Lecture 7 from the COMP 4010 class on AR and VR. This lecture was about Designing AR systems. It was taught on September 7th 2021 by Mark Billinghurst from the University of South Australia.
A lecture on VR systems and graphics given as part of the COMP 4026 AR/VR class taught at the University of South Australia. This lecture was taught by Bruce Thomas on August 20th 2029.
Lecture 3 in the 2022 COMP 4010 lecture series on AR/VR. This lecture provides an introduction for AR Technology. This was taught by Mark Billinghurst at the University of South Australia in 2022.
Lecture 2 in the 2022 COMP 4010 Lecture series on AR/VR and XR. This lecture is about human perception for AR/VR/XR experiences. This was taught by Mark Billinghurst at the University of South Australia in 2022.
Lecture 11 of the COMP 4010 class on Augmented Reality and Virtual Reality. This lecture is about VR applications and was taught by Mark Billinghurst on October 19th 2021 at the University of South Australia
Lecture 9 of the COMP 4010 course in AR/VR from the University of South Australia. This was taught by Mark Billinghurst on October 5th, 2021. This lecture describes VR input devices, VR systems and rapid prototyping tools.
Advanced Methods for User Evaluation in AR/VR StudiesMark Billinghurst
Guest lecture on advanced methods of user evaluation in AR/VR studies. Given by Mark Billinghurst as part of the ARIVE lecture series hosted at the University of Otago. The lecture was given on August 26th 2021.
Lecture 6 of the COMP 4010 course on AR/VR. This lecture is about designing AR systems. This was taught by Mark Billinghurst at the University of South Australia on September 1st 2022.
COMP4010 Lecture 4 - VR Technology - Visual and Haptic Displays. Lecture about VR visual and haptic display technology. Taught on August 16th 2016 by Mark Billinghurst from the University of South Australia
Lecture 1 for the 2022 COMP 4010 course on AR and VR. This course was taught by Mark Billinghurst at the University of South Australia in 2022. This lecture provides an introduction to AR, VR and XR.
Talk given by Mark Billinghurst at the DIGI_X conference in Auckland, New Zealand on June 21st 2018. The talk was about how Mixed Reality can be applied in the work place.
Talk given by Mark Billinghurst to Bajaj Finance Limited in India, on May 9th 2020. The talk describes AR and VR applications, example AR/VR applications in financial services, and potential research directions.
Lecture 8 of the COMP 4010 course taught at the University of South Australia. This lecture provides and introduction to VR technology. Taught by Mark Billinghurst on September 14th 2021 at the University of South Australia.
Lecture 3 from the COMP 4010 course and Virtual and Augmented Reality. This lecture is about VR tracking, input and systems. Taught on August 7th, 2018 by Mark Billinghurst at the University of South Australia
Lecture 5 in the 2022 COMP 4010 lecture series. This lecture is about AR prototyping tools and techniques. The lecture was given by Mark Billinghurst from University of South Australia in 2022.
keynote speech by Mark Billinghurst at the Workshop on Transitional Interfaces in Mixed and Cross-Reality, at the ACM ISS 2021 Conference. Given on November 14th 2021
Lecture 7 from the COMP 4010 class on AR and VR. This lecture was about Designing AR systems. It was taught on September 7th 2021 by Mark Billinghurst from the University of South Australia.
A lecture on VR systems and graphics given as part of the COMP 4026 AR/VR class taught at the University of South Australia. This lecture was taught by Bruce Thomas on August 20th 2029.
Lecture 10 from a course on Mobile Based Augmented Reality Development taught by Mark Billinghurst and Zi Siang See on November 29th and 30th 2015 at Johor Bahru in Malaysia. This lecture provides an overview of research directions in Mobile AR. Look for the other 9 lectures in the course.
Overview of Computer Vision For Footwear IndustryTanvir Moin
Computer vision is an interdisciplinary field that focuses on enabling computers to interpret and analyze visual data from the world around us. It involves the development of algorithms and techniques that allow machines to understand images and videos, just as humans do.
The main goal of computer vision is to create machines that can "see" and understand the world around them, and then use that information to make decisions or take actions. This can involve tasks such as object recognition, scene reconstruction, facial recognition, and image segmentation.
Computer vision has a wide range of applications in various fields, such as healthcare, entertainment, transportation, robotics, and security. Some examples include medical image analysis, autonomous vehicles, augmented reality, and surveillance systems.
In recent years, the development of deep learning techniques, particularly convolutional neural networks (CNNs), has greatly advanced the field of computer vision, allowing machines to achieve state-of-the-art performance on various visual recognition tasks.
The second lecture from the Augmented Reality Summer School talk by Mark Billinghurst at the University of South Australia, February 15th - 19th, 2016. This provides an overview of AR Technology.
Lecture 2 from a course on Mobile Based Augmented Reality Development taught by Mark Billinghurst and Zi Siang See on November 29th and 30th 2015 at Johor Bahru in Malaysia. This lecture provides an introduction to Mobile AR Technology. Look for the other 9 lectures in the course.
Lecture 8 in the COMP 4010 course on AR and VR. This lecture gives an overview of Augmented Reality technology. Taught by Mark Billinghurst on October 5th, 2017 at the University of South Australia
Mika Kaukoranta presents what computer vision is and how it can be utilized in software testing by gaining high-level understanding from digital images or videos.
How to easily improve quality using automated visual inspectionDesign World
Mis-registered parts, out of tolerance parts or defective assemblies are costly mistakes in today’s manufacturing environments. Reducing scrap by catching deviations in the manufacturing process early are key to keeping profit margins high.
Automated inspection using Vision Sensors provide 100% inspection. Learn how the VeriSens Vision sensors ease of use combined with powerful inspection tools catches detects during assembly. Join us in an educational based webinar to demonstrate how to improve quality using automated visual inspection.
Watch this webinar to learn:
-What is a vision sensor?
-What type of applications are suited for vision sensors
-How to easily setup a vision application with VeriSens vision sensors
Lecture 10 in the COMP 4010 Lectures on AR/VR from the Univeristy of South Australia. This lecture is about VR Interface Design and Evaluating VR interfaces. Taught by Mark Billinghurst on October 12, 2021.
Keynote talk by Mark Billinghurst at the 9th XR-Metaverse conference in Busan, South Korea. The talk was given on May 20th, 2024. It talks about progress on achieving the Metaverse vision laid out in Neil Stephenson's book, Snowcrash.
These are slides from the Defence Industry event orgranized by the Australian Research Centre for Interactive and Virtual Environments (IVE). This was held on April 18th 2024, and showcased IVE research capabilities to the South Australian Defence industry.
This is a guest lecture given by Mark Billinghurst at the University of Sydney on March 27th 2024. It discusses some future research directions for Augmented Reality.
Presentation given by Mark Billinghurst at the 2024 XR Spring Summer School on March 7 2024. This lecture talks about different evaluation methods that can be used for Social XR/AR/VR experiences.
Empathic Computing: Delivering the Potential of the MetaverseMark Billinghurst
Invited guest lecture by Mark Billingurust given at the MIT Media Laboratory on November 21st 2023. This was given as part of Professor Hiroshi Ishii's class on Tangible Media
Talk to Me: Using Virtual Avatars to Improve Remote CollaborationMark Billinghurst
A talk given by Mark Billinging in the CLIPE workshop in Tubingen, Germant on April 27th 2023. This talk describes how virtual avatars can be used to support remote collaboration.
Empathic Computing: Designing for the Broader MetaverseMark Billinghurst
Keynote talk given by Mark Billinghurst at the CHI 2023 Workshop on Towards and Inclusive and Accessible Metaverse. The talk was given on April 23rd 2023.
Keynote speech given by Mark Billinghurst at the ISS 2022 conference. Presented on November 22nd, 2022. This keynote outlines some research opportunities in the Metaverse.
Empathic Computing and Collaborative Immersive AnalyticsMark Billinghurst
Short talk by Mark Billinghurst on Empathic Computing and Collaborative Immersive Analytics, presented on July 28th 2022 at the Siggraph 2022 conference.
Lecture given by Mark Billinghurst on June 18th 2022 about how the Metaverse can be used for corporate training. In particular how combining AR, VR and other Metaverse elements can be used to provide new types of learning experiences.
Empathic Computing: Developing for the Whole MetaverseMark Billinghurst
A keynote speech given by Mark Billinghurst at the Centre for Design and New Media at IIIT-Delhi. Given on June 16th 2022. This presentation is about how Empathic Computing can be used to develop for the entre range of the Metaverse.
The final lecture in the 2021 COMP 4010 class on AR/VR. This lecture summarizes some more research directions and trends in AR and VR. This lecture was taught by Mark Billinghurst on November 2nd 2021 at the University of South Australia
Lecture 11 of the COMP 4010 class on Augmented Reality and Virtual Reality. This lecture is about VR applications and was taught by Mark Billinghurst on October 19th 2021 at the University of South Australia
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdfPeter Spielvogel
Building better applications for business users with SAP Fiori.
• What is SAP Fiori and why it matters to you
• How a better user experience drives measurable business benefits
• How to get started with SAP Fiori today
• How SAP Fiori elements accelerates application development
• How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities
• How SAP Fiori paves the way for using AI in SAP apps
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
3. Augmented Reality Definition
• Combines Real and Virtual Images
• Both can be seen at the same time
• Interactive in real-time
• The virtual content can be interacted with
• Registered in 3D
• Virtual objects appear fixed in space
4. Augmented RealityTechnology
• Combines Real and Virtual Images
• Needs: Display technology
• Interactive in real-time
• Needs: Input and interaction technology
• Registered in 3D
• Needs: Viewpoint tracking technology
5. Example: MagicLeap ML-1 AR Display
•Display
• Multi-layered Waveguide display
•Tracking
• Inside out SLAM tracking
•Input
• 6DOF wand, gesture input
6. AR Display Technologies
• Classification (Bimber/Raskar 2005)
• Head attached
• Head mounted display/projector
• Body attached
• Handheld display/projector
• Spatial
• Spatially aligned projector/monitor
7. Bimber, O., & Raskar, R. (2005). Spatial augmented reality: merging real and virtual worlds. CRC press.
DisplayTaxonomy
8. Types of Head Mounted Displays
Occluded
See-thru
Multiplexed
18. Magic Mirror AR Experience
• See AR overlay of an image of yourself
19. AR RequiresTracking and Registration
• Registration
• Positioning virtual object wrt real world
• Fixing virtual object on real object when view is fixed
• Calibration
• Offline measurements
• Measure camera relative to head mounted display
• Tracking
• Continually locating the user’s viewpoint when view moving
• Position (x,y,z), Orientation (r,p,y)
20. Sources of Registration Errors
•Static errors
• Optical distortions (in HMD)
• Mechanical misalignments
• Tracker errors
• Incorrect viewing parameters
•Dynamic errors
• System delays (largest source of error)
• 1 ms delay = 1/3 mm registration error
21. Dynamic errors
• Total Delay = 50 + 2 + 33 + 17 = 102 ms
• 1 ms delay = 1/3 mm = 33mm error
Tracking Calculate
Viewpoint
Simulation
Render
Scene
Draw to
Display
x,y,z
r,p,y
Application Loop
20 Hz = 50ms 500 Hz = 2ms 30 Hz = 33ms 60 Hz = 17ms
22. Reducing dynamic errors (1)
•Reduce system lag
•Faster components/system modules
•Reduce apparent lag
•Image deflection
•Image warping
23. Reducing dynamic errors (2)
• Match video + graphics input streams (video AR)
• Delay video of real world to match system lag
• User doesn’t notice
• Predictive Tracking
• Inertial sensors helpful
Azuma / Bishop 1994
28. Why Optical Tracking for AR?
• Many AR devices have cameras
• Mobile phone/tablet, Video see-through display
• Provides precise alignment between video and AR overlay
• Using features in video to generate pixel perfect alignment
• Real world has many visual features that can be tracked from
• Computer Vision is a well established discipline
• Over 40 years of research to draw on
• Old non real time algorithms can be run in real time on todays devices
29. Common AR Optical Tracking Types
• Marker Tracking
• Tracking known artificial markers/images
• e.g. ARToolKit square markers
• Markerless Tracking
• Tracking from known features in real world
• e.g. Vuforia image tracking
• Unprepared Tracking
• Tracking in unknown environment
• e.g. SLAM tracking
30. Visual Tracking Approaches
• Marker based tracking with artificial features
• Make a model before tracking
• Model based tracking with natural features
• Acquire a model before tracking
• Simultaneous localization and mapping
• Build a model while tracking it
31. Marker Tracking
• Available for more than 20 years
• Several open-source solutions exist
• ARToolKit, ARTag, ATK+, etc
• Fairly simple to implement
• Standard computer vision methods
• A rectangle provides 4 corner points
• Enough for pose estimation!
33. Key Problem: Finding Camera Position
• Need camera pose relative to marker to render AR graphics
Known image Image in Camera view Overlay AR content
34. Goal: Find Camera Pose
• Knowing:
• Position of key points in on-screen video image
• Camera properties (focal length, image distortion)
36. Coordinates for Marker Tracking
Marker Camera
•Final Goal
•Rotation & Translation
1: Camera Ideal Screen
•Perspective model
•Obtained from Camera Calibration
2: Ideal Screen Observed Screen
•Nonlinear function (barrel shape)
•Obtained from Camera Calibration
3: Marker Observed Screen
•Correspondence of 4 vertices
•Real time image processing
37. Marker Tracking – General Principle
1. Capturing image with known camera
2. Search for quadrilaterals
3. Pose estimation
from homography
4. Pose refinement
Minimize nonlinear
projection error
5. Use final pose
37
1
3
2
4
5
Image: Daniel Wagner
39. MarkerTracking – Fiducial Detection
• Threshold the whole image to black and white
• Search scanline by scanline for edges (white to black)
• Follow edge until either
• Back to starting pixel
• Image border
• Check for size
• Reject fiducials early that are too small (or too large)
40. MarkerTracking – Rectangle Fitting
• Start with an arbitrary point “x” on the contour
• The point with maximum distance must be a corner c0
• Create a diagonal through the center
• Find points c1 & c2 with maximum distance left and right of diag.
• New diagonal from c1 to c2
• Find point c3 right of diagonal with maximum distance
41. MarkerTracking – Pattern checking
• Calculate homography using the 4 corner points
• “Direct Linear Transform” algorithm
• Maps normalized coordinates to marker coordinates
(simple perspective projection, no camera model)
• Extract pattern by sampling and check
• Id (implicit encoding)
• Template (normalized cross correlation)
42. Marker tracking – Pose estimation
• Calculates marker pose relative to the camera
• Initial estimation directly from homography
• Very fast, but coarse with error
• Jitters a lot…
• Iterative Refinement using Gauss-Newton method
• 6 parameters (3 for position, 3 for rotation) to refine
• At each iteration we optimize on the error
• Iterate
43. Outcome: Camera Transform
• Transformation from Marker to Camera
• Rotation and Translation
TCM : 4x4 transformation matrix
from marker coord. to camera coord.
44. Tracking challenges inARToolKit
False positives and inter-marker confusion
(image by M. Fiala)
Image noise
(e.g. poor lens, block
coding /
compression, neon tube)
Unfocused camera,
motion blur
Dark/unevenly lit
scene, vignetting
Jittering
(Photoshop illustration)
Occlusion
(image by M. Fiala)
49. Visual Tracking Approaches
• Marker based tracking with artificial features
• Make a model before tracking
• Model based tracking with natural features
• Acquire a model before tracking
• Simultaneous localization and mapping
• Build a model while tracking it
50. Natural Feature Tracking
• Use Natural Cues of Real Elements
• Edges
• Surface Texture
• Interest Points
• Model or Model-Free
• No visual pollution
Contours
Features Points
Surfaces
51. Natural Features
• Detect salient interest points in image
• Must be easily found
• Location in image should remain stable
when viewpoint changes
• Requires textured surfaces
• Alternative: can use edge features (less discriminative)
• Match interest points to tracking model database
• Database filled with results of 3D reconstruction
• Matching entire (sub-)images is too costly
• Typically interest points are compiled into “descriptors”
Tracking 51
Image: Gerhard Reitmayr
Image: Martin Hirzer
54. Tracking by Keypoint Detection
• This is what most trackers do…
• Targets are detected every frame
• Popular because tracking and detection
are solved simultaneously
Keypoint detection
Descriptor creation
and matching
Outlier Removal
Pose estimation
and refinement
Camera Image
Pose
Recognition
55. Detection and Tracking
Detection
Incremental
tracking
Tracking target
detected
Tracking target
lost
Tracking target
not detected
Incremental
tracking ok
Start
+ Recognize target type
+ Detect target
+ Initialize camera pose
+ Fast
+ Robust to blur, lighting changes
+ Robust to tilt
• Tracking and detection are complementary approaches.
• After successful detection, the target is tracked incrementally.
• If the target is lost, the detection is activated again
56. What is a Keypoint?
• It depends on the detector you use!
• For high performance use the FAST corner detector
• Apply FAST to all pixels of your image
• Obtain a set of keypoints for your image
• Describe the keypoints
Rosten, E., & Drummond, T. (2006, May). Machine learning for high-speed corner detection.
In European conference on computer vision (pp. 430-443). Springer Berlin Heidelberg.
59. Descriptors
• Describe the Keypoint features
• Can use SIFT
• Estimate the dominant keypoint
orientation using gradients
• Compensate for detected
orientation
• Describe the keypoints in terms
of the gradients surrounding it
Wagner D., Reitmayr G., Mulloni A., Drummond T., Schmalstieg D.,
Real-Time Detection and Tracking for Augmented Reality on Mobile Phones.
IEEE Transactions on Visualization and Computer Graphics, May/June, 2010
60. Database Creation
• Offline step – create database of known features
• Searching for corners in a static image
• For robustness look at corners on multiple scales
• Some corners are more descriptive at larger or smaller scales
• We don’t know how far users will be from our image
• Build a database file with all descriptors and their
position on the original image
61. Real-time Tracking
• Search for known keypoints in the video
• Create the descriptors
• Match the descriptors from the
live video against those in the database
• Brute force is not an option
• Need the speed-up of special data structures
Keypoint detection
Descriptor creation
and matching
Outlier Removal
Pose estimation
and refinement
Camera Image
Pose
Recognition
62. NFT – Outlier removal
• Removing outlier features
• Several removal techniques
• Simple geometric tests
• Is the keypoint rotation invariant?
• Do keypoints remain relative to each other?
• Homography-based tests
Rotation Invariant
63. NFT – Pose refinement
• Pose from homography makes good
starting point
• Use Gauss-Newton iteration
• Try to minimize the re-projection error
of the keypoints
• Typically, 2-4 iterations are enough..
64. NFT – Real-time tracking
• Search for keypoints in the video image
• Create the descriptors
• Match the descriptors from the
live video against those in the database
• Remove the keypoints that are outliers
• Use the remaining keypoints
to calculate the pose of the camera
Keypoint detection
Descriptor creation
and matching
Outlier Removal
Pose estimation
and refinement
Camera Image
Pose
Recognition
70. 3D Model BasedTracking
• Tracking from 3D object shape
• Align detected features to 3D object model
• Examples
• SnapChat Face tracking
• Mechanical part tracking
• Vehicle tracking
• Etc..
74. Taxonomy of Model Based Tracking
Lowney, M., & Raj, A. S. (2016). Model based tracking for augmented reality on mobile devices.
75. Marker vs.Natural FeatureTracking
• Marker tracking
• Usually requires no database to be stored
• Markers can be an eye-catcher
• Tracking is less demanding
• The environment must be instrumented
• Markers usually work only when fully in view
• Natural feature tracking
• A database of keypoints must be stored/downloaded
• Natural feature targets might catch the attention less
• Natural feature targets are potentially everywhere
• Natural feature targets work also if partially in view
76. Visual Tracking Approaches
• Marker based tracking with artificial features
• Make a model before tracking
• Model based tracking with natural features
• Acquire a model before tracking
• Simultaneous localization and mapping
• Build a model while tracking it
78. Tracking from an Unknown Environment
• What to do when you don’t know any features?
• Very important problem in mobile robotics - Where am I?
• SLAM
• Simultaneously Localize And Map the environment
• Goal: to recover both camera pose and map structure
while initially knowing neither.
• Mapping:
• Building a map of the environment which the robot is in
• Localisation:
• Navigating this environment using the map while keeping
track of the robot’s relative position and orientation
79. Parallel Tracking and Mapping
Tracking Mapping
New keyframes
Map updates
+ Estimate camera pose
+ For every frame
+ Extend map
+ Improve map
+ Slow updates rate
Parallel tracking and mapping uses two
concurrent threads, one for tracking and one
for mapping, which run at different speeds
80. Parallel Tracking and Mapping
Video stream
New frames
Map updates
Tracking Mapping
Tracked local pose
FAST SLOW
Simultaneous
localization and mapping
(SLAM)
in small workspaces
Klein/Drummond, U.
Cambridge
81. Visual SLAM
• Early SLAM systems (1986 - )
• Computer visions and sensors (e.g. IMU, laser, etc.)
• One of the most important algorithms in Robotics
• Visual SLAM
• Using cameras only, such as stereo view
• MonoSLAM (single camera) developed in 2007 (Davidson)
83. How SLAMWorks
• Three main steps
1. Tracking a set of points through successive camera frames
2. Using these tracks to triangulate their 3D position
3. Simultaneously use the estimated point locations to calculate
the camera pose which could have observed them
• By observing a sufficient number of points can solve for both
structure and motion (camera path and scene structure).
84. Evolution of SLAM Systems
• MonoSLAM (Davidson, 2007)
• Real time SLAM from single camera
• PTAM (Klein, 2009)
• First SLAM implementation on mobile phone
• FAB-MAP (Cummins, 2008)
• Probabilistic Localization and Mapping
• DTAM (Newcombe, 2011)
• 3D surface reconstruction from every pixel in image
• KinectFusion (Izadi, 2011)
• Realtime dense surface mapping and tracking using RGB-D
86. LSD-SLAM (Engel 2014)
• A novel, direct monocular SLAM technique
• Uses image intensities both for tracking and mapping.
• The camera is tracked using direct image alignment, while
• Geometry is estimated as semi-dense depth maps
• Supports very large-scale tracking
• Runs in real time on CPU and smartphone
88. Direct Method vs. Feature Based
• Direct uses all information in image, cf feature based approach
that only use small patches around corners and edges
89. Applications of SLAM Systems
• Many possible applications
• Augmented Reality camera tracking
• Mobile robot localisation
• Real world navigation aid
• 3D scene reconstruction
• 3D Object reconstruction
• Etc..
• Assumptions
• Camera moves through an unchanging scene
• So not suitable for person tracking, gesture recognition
• Both involve non-rigidly deforming objects and a non-static map
91. Combining Sensors andVision
• Sensors
• Produces noisy output (= jittering augmentations)
• Are not sufficiently accurate (= wrongly placed augmentations)
• Gives us first information on where we are in the world,
and what we are looking at
• Vision
• Is more accurate (= stable and correct augmentations)
• Requires choosing the correct keypoint database to track from
• Requires registering our local coordinate frame (online-
generated model) to the global one (world)
93. Types of Sensor Fusion
• Complementary
• Combining sensors with different degrees of freedom
• Sensors must be synchronized (or requires inter-/extrapolation)
• E.g., combine position-only and orientation-only sensor
• E.g., orthogonal 1D sensors in gyro or magnetometer are complementary
• Competitive
• Different sensor types measure the same degree of freedom
• Redundant sensor fusion
• Use worse sensor only if better sensor is unavailable
• E.g., GPS + pedometer
• Statistical sensor fusion
www.augmentedrealitybook.org Tracking 93
94. Example: Outdoor Hybrid Tracking
• Combines
• computer vision
• inertial gyroscope sensors
• Both correct for each other
• Inertial gyro
• provides frame to frame prediction of camera
orientation, fast sensing
• drifts over time
• Computer vision
• Natural feature tracking, corrects for gyro drift
• Slower, less accurate
95. Robust OutdoorTracking
• HybridTracking
• ComputerVision, GPS, inertial
• Going Out
• Reitmayr & Drummond (Univ. Cambridge)
Reitmayr, G., & Drummond, T. W. (2006). Going out: robust model-based tracking for outdoor augmented reaity. In Mixed and
Augmented Reality, 2006. ISMAR 2006. IEEE/ACM International Symposium on (pp. 109-118). IEEE.
98. ARKit – Visual Inertial Odometry
• Uses both computer vision + inertial sensing
• Tracking position twice
• Computer Vision – feature tracking, 2D plane tracking
• Inertial sensing – using the phone IMU
• Output combined via Kalman filter
• Determine which output is most accurate
• Pass pose to ARKit SDK
• Each system compliments the other
• Computer vision – needs visual features
• IMU - drifts over time, doesn’t need features
99. ARKit –Visual Inertial Odometry
• Slow camera
• Fast IMU
• If camera drops out IMU takes over
• Camera corrects IMU errors
101. Conclusions
• Tracking and Registration are key problems
• Registration error
• Measures against static error
• Measures against dynamic error
• AR typically requires multiple tracking technologies
• Computer vision most popular
• Research Areas:
• SLAM systems, Deformable models, Mobile outdoor tracking
102. More Information
Fua, P., & Lepetit, V. (2007). Vision based 3D tracking
and pose estimation for mixed reality. In Emerging
technologies of augmented reality: Interfaces and
design (pp. 1-22). IGI Global.
106. AR Interaction
• Designing AR Systems = Interface Design
• Using different input and output technologies
• Objective is a high quality of user experience
• Ease of use and learning
• Performance and satisfaction
107. Typical Interface Design Path
1/ Prototype Demonstration
2/ Adoption of Interaction Techniques from
other interface metaphors
3/ Development of new interface metaphors
appropriate to the medium
4/ Development of formal theoretical models
for predicting and modeling user actions
Desktop WIMP
Virtual Reality
Augmented Reality
108. Interacting with AR Content
• You can see spatially registered AR..
how can you interact with it?
109. Different Types of AR Interaction
• Browsing Interfaces
• simple (conceptually!), unobtrusive
• 3D AR Interfaces
• expressive, creative, require attention
• Tangible Interfaces
• Embedded into conventional environments
• Tangible AR
• Combines TUI input + AR display
110. AR Interfaces as Data Browsers
• 2D/3D virtual objects are
registered in 3D
• “VR in Real World”
• Interaction
• 2D/3D virtual viewpoint control
• Applications
• Visualization, training
111. AR Information Browsers
• Information is registered
to
real-world context
• Hand held AR displays
• Interaction
• Manipulation of a window
into information space
• Applications
• Context-aware information
displays
Rekimoto, et al. 1997
114. Current AR Information Browsers
• Mobile AR
• GPS + compass
• Many Applications
• Wikitude
• Yelp
• Google maps
• …
115. Example: Google Maps AR Mode
• AR Navigation Aid
• GPS + compass, 2D/3D object placement
116.
117. Advantages and Disadvantages
• Important class of AR interfaces
• Wearable computers
• AR simulation, training
• Limited interactivity
• Modification of virtual
content is difficult
Rekimoto, et al. 1997
118. 3D AR Interfaces
• Virtual objects displayed in 3D
physical space and manipulated
• HMDs and 6DOF head-tracking
• 6DOF hand trackers for input
• Interaction
• Viewpoint control
• Traditional 3D user interface
interaction: manipulation, selection,
etc.
Kiyokawa, et al. 2000
122. Advantages and Disadvantages
• Important class of AR interfaces
• Entertainment, design, training
• Advantages
• User can interact with 3D virtual
object everywhere in space
• Natural, familiar interaction
• Disadvantages
• Usually no tactile feedback
• User has to use different devices for
virtual and physical objects
Oshima, et al. 2000