Download as PDF, PPTX
![Augmented Reality Definition
• Defining Characteristics [Azuma 97]
• Combines Real andVirtual Images
• Both can be seen at the same time
• Interactive in real-time
• The virtual content can be interacted with
• Registered in 3D
• Virtual objects appear fixed in space
Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.](https://image.slidesharecdn.com/lecture4-technologyfinalnovideo-160818053748/75/COMP-4026-Lecture4-Processing-and-Advanced-Interface-Technology-84-2048.jpg)
![Early Examples
• Interaction without devices:
• BodySpace [Strachan 2007]: Functions to body position
• Abracadabra [Harrison 2007]: Magnets on finger tips
• GesturePad [Rekimoto 2001]: Capacitive sensing in clothing
• Palm-based Interaction
• Haptic Hand [Kohli 2005]: Using non-dominant hand in VR
• Sixth Sense [Mistry 2009]: Projection on hand
• Brainy Hand [Tamaki 2009]: Head worn projector/camera](https://image.slidesharecdn.com/lecture4-technologyfinalnovideo-160818053748/75/COMP-4026-Lecture4-Processing-and-Advanced-Interface-Technology-98-2048.jpg)
![ImaginaryPhone
• Gustafson, S., Holz, C., & Baudisch, P. [2011]](https://image.slidesharecdn.com/lecture4-technologyfinalnovideo-160818053748/75/COMP-4026-Lecture4-Processing-and-Advanced-Interface-Technology-100-2048.jpg)
![Invisible Interfaces – Gestures in Space
• Gustafson, S., Bierwirth, D., & Baudisch, P. [2010]
• Using a non-dominant hand stabilized interface.](https://image.slidesharecdn.com/lecture4-technologyfinalnovideo-160818053748/75/COMP-4026-Lecture4-Processing-and-Advanced-Interface-Technology-103-2048.jpg)
Uploaded byMark Billinghurst
COMP 4026 Lecture4: Processing and Advanced Interface Technology
The document discusses advanced HCI concepts, focusing on tools like App Inventor and Processing for creating interactive applications, as well as various prototyping and evaluation methods. It covers interaction design processes, mobile augmented reality technologies, and interfaces for wearable and invisible computing, highlighting their challenges and research gaps. Finally, it addresses physiological sensing technologies and their applications in understanding user interactions and experiences.
More Related Content
Similar to COMP 4026 Lecture4: Processing and Advanced Interface Technology
COMP 4026 Lecture4: Processing and Advanced Interface Technology
- 1. LECTURE 4: PROCESSINGAND ADVANCED INTERFACES COMP 4026 – Advanced HCI Semester 5 - 2016 Mark Billinghurst University of South Australia August 18th 2016
- 2.
- 3. App Inventor • http://appinventor.mit.edu/ • http://appinventor.org/ • Visual Programming for Android Apps • Features • Access to Android Sensors • Multimedia output • Drag and drop web based interface • Designer view – app layout • Blocks view – program logic/control
- 4.
- 5.
- 6. Interactive Coding -Processing ▪ Programming tool for Artists/Designers ▪ http://processing.org ▪ Easy to code, Free, Open source, Java based ▪ 2D, 3D, audio/video support ▪ Processing For Android ▪ http://wiki.processing.org/w/Android ▪ Strong Android support, builds .apk file
- 8. Buchenau, M., &Suri, J. F. (2000, August). Experience prototyping. In Proceedings of the 3rd conference on Designing interactive systems: processes, practices, methods, and techniques (pp. 424-433). ACM.
- 9. Experience Prototyping The experienceof even simple artifacts does not exist in a vacuum but, rather, in dynamic relationship with other people, places and objects. Additionally, the quality of people’s experience changes over time as it is influenced by variations in these multiple contextual factors.
- 10.
- 11. littleBits - http://littlebits.cc/ • Plug and play hardware components • Sensors, input, output • Rapid design with hardware
- 12.
- 13. Interaction Design Process Evaluate (Re)Design Identifyneeds/ establish requirements Build an interactive version Final Product
- 14. When to evaluate? • Oncethe product has been developed • pros : rapid development, small evaluation cost • cons : rectifying problems • During design and development • pros : find and rectify problems early • cons : higher evaluation cost, longer development design implementationevaluation redesign & reimplementation design implementation
- 15. Four evaluation paradigms • ‘quickand dirty’ • usability testing (lab studies) • field studies • predictive evaluation
- 16. Characteristics of approaches Usability testing Field studies Predictive Usersdo task natural not involved Location controlled natural anywhere When prototype early prototype Data quantitative qualitative problems Feed back measures & errors descriptions problems Type applied naturalistic expert
- 17. Evaluation approaches andmethods Method Usability testing Field studies Predictive Observing x x Asking users x x Asking experts x x Testing x Modeling x
- 18.
- 19. Interaction Design Process Evaluate (Re)Design Identifyneeds/ establish requirements Build an interactive version Final Product
- 20. MOBILE AUGMENTED REALITY FORSPATIAL NAVIGATION Sharon Brosnan 0651869 Bachelor of Science in Digital Media Design
- 21.
- 22.
- 23. POST IT PROTOTYPING First Dra. Camera View with 3D Second Dra. Third Dra. • Selec,on highlighted in blue • Home bu<on added for easy naviga,on to main menu
- 24.
- 25. WIKITUDE • Popular augmentedreality browser for mobile devices • Mapping • Point of Interest abilities • Multiplatform • Shows the points of interest of Bunratty Folk Park • Markers can be selected in and an information pop-up appears
- 26. VIDEO PROTOTYPE ! Flexible tool for capturing the use of an interface ! Elaborate simula,on of how the naviga,onal aid will work ! Does not need to be realis,c in every detail ! Gives a good idea of how the finished system will work
- 27.
- 30. Processing - Notes • Language of Interaction • Physical Manipulation • Input using code • Mouse Manipulation • Presence, location, image • Haptic interfaces and multi-touch • Gesture • Voice and Speech
- 32. Basic Parts ofa Sketch /* Notes comment */! //set up global variables! float moveX = 50;! ! //Initialize the Sketch! void setup (){! }! ! //draw every frame! void draw(){! }!
- 33. Sample Drawing int m= 0;! float s = 0;! ! void setup(){! size(512,512);! background(255);! }! ! void draw (){! fill(255,0,0);! ellipse(mouseX,mouseY,s,s);! }! ! void mouseMoved(){! s = 40 + 20*sin(++m/10.0f);! }!
- 35. Drawing • draw() getscalled as fast as possible, unless a frameRate is specified • stroke() sets color of drawing outline • fill() sets inside color of drawing • mousePressed is true if mouse is down • mouseX, mouseY - mouse position !void draw() { ! !stroke(255); ! !if(mousePressed) {! ! !line(mouseX, mouseY, pmouseX, pmouseY);! ! !}! !}!
- 36. Processing and Drawing • Basic Shapes rect(x, y, width, height)! ellipse(x, y, width, height)! line(x1, y1, x2, y2), line(x1, y1, x2, y2, z1, z2)! • Filling shapes - fill( ) fill(int gray), fill(color color), fill(color color, int alpha)! • Curve • Draws curved lines • Vertex • Creates shapes (beginShape, endShape)
- 37. Vertex Demo void setup(){! size(400,400);! }! ! voiddraw(){! background(255);! fill(0);! beginShape();! vertex(0,0);! vertex(400,400);! vertex(mouseX,mouseY);! endShape();! }!
- 38. Curve Demo void setup(){! size(400,400);! }! ! voiddraw(){! background(255);! fill(0);! ! int xVal = mouseX*3-100;! int yVal = mouseY*3-100;! ! curve(xVal, yVal, 100, 100, 100, 300, xVal, yVal);! curve(xVal, yVal, 100, 300, 300, 300, xVal, yVal);! curve(xVal, yVal, 300, 300, 300, 100, xVal, yVal);! curve(xVal, yVal, 300, 100, 100, 100, xVal, yVal);! ! }!
- 39. Class and Objects • seehttp://processing.org/learning/objects/ • Object • grouping of multiple related properties and functions • Objects are defined by Object classes • Eg Car object • Data • colour, location, speed • Functions • drive(), draw()
- 40. Classes • four elements: name,data, constructor, and methods. • Name class myName { }! • Data • collection of class variables • Constructor • run when object created • Methods • class functions
- 41.
- 43. Class Usage // Step1. Declare an object.! Car myCar;! ! void setup() { ! // Step 2. Initialize object.! myCar = new Car(); ! } ! ! void draw() { ! background(255); ! // Step 3. Call methods on the object. ! myCar.drive(); ! myCar.display(); ! }!
- 45. Constructing Objects • OneCar Car myCar= new Car(); ! • Two Cars !// Creating two car objects ! !Car myCar1 = new Car(); ! !Car myCar2 = new Car(); ! • One car with initial values Car myCar = new Car(color(255,0,0),0,100,2); !
- 46. Modifying Constructor Car(color tempC,float tempXpos, float tempYpos, float tempXspeed) ! { ! c = tempC; ! xpos = tempXpos; ! ypos = tempYpos; ! xspeed = tempXspeed; ! }!
- 47. Mouse Interaction • Mouse position • mouseX, mouseY variables • Mouse Interaction • mousePressed() • mouseReleased() • mouseDragged() • Add in own code void mouseDragged(){! line(pmouseX, pmouseY, mouseX, mouseY);! }!
- 48. Keyboard Interaction • Check keyPressedvariable in draw() method !void draw(){! ! !if(keyPressed){! ! ! !print(" you pressed " +key);! ! !}! }! • Use keyPressed() method !void keyPressed(){! ! !print(" you're pressing "+key);! !}!
- 50. Importing Libraries • Canadd functionality by Importing Libraries • java archives - .jar files • Include import code import processing.opengl.*;! • Popular Libraries • Minim - audio library • OCD - 3D camera views • Physics - physics engine • bluetoothDesktop - bluetooth networking
- 51.
- 52. Graphical Controls • Use ControlP5Library • http://www.sojamo.de/libraries/controlP5/ • Add graphical controls • Buttons, sliders, etc • Support for OSC (Open Sound Controller) • Use ControlP5 class import controlP5.*;! addButton(name, value, x, y, width, height);! • Event Handing
- 53. Interface Elements • Interfascia • http://www.superstable.net/interfascia/ • GUI Libraryfor Processing • Buttons • Check boxes • Textfields • Progress bar
- 55. Graphical Controls • UseControlP5 Library • http://www.sojamo.de/libraries/controlP5/ • Add graphical controls • Buttons, sliders, etc • Support for OSC (Open Sound Controller) • Use ControlP5 class import controlP5.*;! addButton(name, value, x, y, width, height);! • Event Handing
- 56.
- 57. Programming Graphics • Transformations • Creating motionand animation • Bitmaps and pixels • Textures
- 58. 3D Graphics • Two options • P3DLibrary • OpenGL Library • P3D • Simple, integrated directly into processing • Lightweight 3D • OpenGL • Full graphics support • Complex
- 59. Shapes • beginShape(SHAPE); • DefineVertices • SHAPES: QUADS, QUAD_STRIP,TRIANGLE_FAN • endShape(); • Eg: Quads !beginShape(QUADS);! !fill(0, 1, 1); vertex(-1, 1, 1);! !fill(1, 1, 1); vertex( 1, 1, 1);! !fill(1, 0, 1); vertex( 1, -1, 1);! !fill(0, 0, 1); vertex(-1, -1, 1);! !endShape();!
- 60. P3D Scene size(640, 360,P3D); ! background(0);! lights();! ! noStroke();! pushMatrix();! !translate(130, height/2, 0);! !rotateY(1.25);! !rotateX(-0.4);! !box(100);! popMatrix();! ! noFill();! stroke(255);! pushMatrix();! !translate(500, height*0.35, -200);! !sphere(280);! popMatrix();!
- 61. Vertices Demo • RGB Cube • Vetices and vertex fills • VertexDemo • Different types of quad strips • User defined drawing function
- 62. Transformations • Rotation ! !rotateX(a),rotateY(a * 2.0), rotateZ(a)! • Translation ! !translate(X,Y); translate(X,Y,Z);! • Scale • Push and Pop functions • Push - Saving current coordinate system • Pop – Restores previous coordinate system • Eg: PushPopCubes
- 63. OpenGL • Import opengl library importprocessing.opengl.*;! • Create drawing context !void draw(){! ! !PGraphicsOpenGL pgl = (PGraphicsOpenGL) g;! ! !GL g = pgl.beginGL();! ! !//use GL methods! ! !pgl.endGL(); //end opengl calls! }!
- 64. 3D Model Loading • Using saito model loader • Loads obj files • http://code.google.com/p/saitoobjloader/downloads • Code fragment import saito.objloader.*;! OBJModel model ;! model = new OBJModel(this, "sa.obj", "absolute", TRIANGLES);!
- 66. OtherTools • GLGraphics • http://glgraphics.sourceforge.net/ • Library thatextends the Processing OpenlGL renderer • opengl textures, • image post-processing filters, • 3D Models, and shaders in GLSL, • Cg and CgFX
- 67.
- 68. Advanced Interface Technology • WearableComputers • Augmented Reality • Virtual Reality • Invisible Interfaces • Environment Sensing • Physiological Sensing
- 69. Class Project 1. PickAdvanced Technology 2. Brainstorm use case 3. Develop conceptual design 4. Prototype interface/experience design 5. Conduct user evaluation 6. Repeat steps 3-5 7. Write report
- 70.
- 74. Major changes incomputing
- 76. Wearable Computing ▪ Computeron the body that is: ▪ Always on ▪ Always accessible ▪ Always connected ▪ Other attributes ▪ Augmenting user actions ▪ Aware of user and surroundings
- 77.
- 78.
- 80.
- 81. Research Problems • Hardware • Power,networking, display • User Interaction • User input, speech, gesture, gaze, etc • Novel interaction methods • Social Acceptance • Privacy, social factors • Novel Applications • Collaboration • Intelligent assistance
- 82.
- 83.
- 84. Augmented Reality Definition • DefiningCharacteristics [Azuma 97] • Combines Real andVirtual Images • Both can be seen at the same time • Interactive in real-time • The virtual content can be interacted with • Registered in 3D • Virtual objects appear fixed in space Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.
- 85.
- 86. • Put ARpictures here Augmented Reality Examples
- 87.
- 88. Research Problems • Low levelhardware/software • Head mounted displays • Tracking systems • User Interaction • Gesture based interaction • Multimodal input (speech, gesture) • Novel Applications • Face to face collaboration • Authoring tools
- 89.
- 90.
- 91. Virtual Reality • ImmersiveVR • Headmounted display, gloves • Separation from the real world
- 92. Occulus Rift • $300 USD • 360degree head tracking • 100 degree field of view
- 93. Wearable Virtual Reality • Samsung Gear VR • See virtual reality on your phone • Using phone display, compass
- 94.
- 95.
- 96. Research Problems • Low level • Wide area tracking • Development tools • User Interaction • Intuitive input (gesture, controllers) • Avatar control and representation • Techniques for navigation/manipulation • Novel Applications • Massive multi-user environments • Content capture and sharing
- 97.
- 98. Early Examples • Interactionwithout devices: • BodySpace [Strachan 2007]: Functions to body position • Abracadabra [Harrison 2007]: Magnets on finger tips • GesturePad [Rekimoto 2001]: Capacitive sensing in clothing • Palm-based Interaction • Haptic Hand [Kohli 2005]: Using non-dominant hand in VR • Sixth Sense [Mistry 2009]: Projection on hand • Brainy Hand [Tamaki 2009]: Head worn projector/camera
- 99. Unobtrusive Input Devices ▪ GesturePad ▪ Capacitive multilayered touchpads ▪ Supports interactive clothing
- 100. ImaginaryPhone • Gustafson, S.,Holz, C., & Baudisch, P. [2011]
- 101.
- 102.
- 103. Invisible Interfaces –Gestures in Space • Gustafson, S., Bierwirth, D., & Baudisch, P. [2010] • Using a non-dominant hand stabilized interface.
- 104.
- 105. Project Soli • UsingRadar to support free-hand spatial input
- 106.
- 107. Research Gaps • Free-hand interfacesusing relative input • Combining invisible interface + mobile device • Multimodal interaction • speech + gesture input • Affordances and discoverability • Interaction frameworks
- 108. Research Problems • Hardware • Power,networking, display • User Interaction • User input, speech, gesture, gaze, etc • Novel interaction methods • Social Acceptance • Privacy, social factors • Novel Applications • Collaboration • Intelligent assistance
- 109.
- 110. Environmental Sensor • New sensorstrack and capture real environment • Navigation, 3D modeling, user tracking • Depth Sensors • Microsoft Kinect, Intel RealSense • Integrated Devices • Google Tango
- 111. Google Tango • Tablet basedsystem • Android OS • Multiple sensors • RGBD Sensor • IR Structured light • Inertial sensors • High end graphics • Nvidia tegra chip
- 112.
- 113. Research Problems • Content creation • Creating better 3D models • Segmenting objects • User Interaction • Interaction with real world • Interacting with multiple devices • Novel Applications • AR notes/real world tagging • Social networking
- 114.
- 115. Physiological Sensors • Sensing userstate • Body worn devices • Multiple possible sensors • Physical activity • Eye tracking, gaze • Heart rate • GSR • Breathing • Etc
- 116. Tobii Eye Tracker • Wearableeye tracking system • Natural data capture • Scene camera capture • Recording/streaming eye gaze, 60 Hz sampling
- 117.
- 118. Research Problems • User Interaction • Implicit vs. Explicit interaction • Measuring cognitive • Social Acceptance • Privacy, social factors • Novel Applications • Collaboration • Intelligent assistance
- 119.