Hci history


Published on

Published in: Technology, Education
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Hci history

  1. 1. History of HCI Key systems, people and ideas Matthias Rauterberg Technical University Eindhoven (TU/e) The Netherlands
  2. 2. History of Computer TechnologyDigital computer grounded in ideas from 1700’s & 1800’sComputer technology became available in the 1940’s and 1950’ssee further: History of Computing History of HCI© M. Rauterberg, TU/e 2
  3. 3. Konrad Zuse (1910-1995) In 1936 Konrad Zuse, absolutely set apart from the academic world, started constructing an automatic machine to solve calculation problems for designing plane wings; these analyses forcing him to long and repetitive calculations. In his living-room and with the mere aid of few tools, he first produced a binary mechanical memory, to which he soon connected a mechanical calculation unit as well as a programming unit controlled from old movie films punched by hand. He called this model V1 (Versuchsmodell 1), but subsequently changed this name into Z1 in order not to confuse it with the flying bombs having the same name! Having become aware of the poor liability and slowness of this machine, in 1939, Zuse prepared a second one, called Z2, characterized from a still mechanical memory but with a relay- operated electromechanical calculating unit. In the following years, Zuse accomplished a real electromechanical working computer, Z3, which was submitted in 1941 to an audience of engineers and scientists, raising great interest. Not yet satisfied, living in a Berlin continuously bombed and in which it was difficult to find even food, Zuse constructed Z4 with a mechanical memory (relays were now unfindable): the machine was ended in 1944. Zuse arrived in protecting Z4 from the destructions of war and from the hands of Allies, hiding it in a cellar in the small Bavarian village of Hindelang. Once tranquillity returned again, Zuse transferred in to the Swiss Federal Institute of Technology (ETH), in Zurich, were it remained working for 15 years. Up to 1951, this machine remained the only working computer in continental Europe. The history of Zuse is emblematic for at least five reasons: 1) his contribution was completely original, as he was very isolated from the rest of the world but even from the German research activity; 2) the fact that he has conceived a binary representation of figures which is that adopted from all the modern computers; 3) the fact that he has independently achieved an architecture which was already suggested by Babbage; the invention of the first programming language (Plankalkul, 1943-45); 5) the extremely practical and simple way of facing the problem: the estimated cost of Z2 is 6.500 US $, only.© M. Rauterberg, TU/e 3
  4. 4. Z3 (1941)Zuse’s Z3 was the world’s first reliable working machine for very complicated arithmetic calculations, which was freely programmable and was based on a binary floating point number and switching system. Konrad Zuse in front of his reconstructed Z3© M. Rauterberg, TU/e 4
  5. 5. Eniac (1943) – A general view of the ENIAC, the first all electronic numerical integrator and computer in USA. From IBM Archives.© M. Rauterberg, TU/e 5
  6. 6. Mark I (1944) •The Mark I paper tape readers. From Harvard University Cruft Photo Laboratory.© M. Rauterberg, TU/e 6
  7. 7. von Neuman Architecture (1946)Instructions and data are stored in the same memoryfor which there is a single link (the von Neumannbottleneck) to the CPU which decodes and executesinstructions.The CPU can have multiple functional units.The memory access can be enhanced by use of caches Johann (John) von Neumannmade from faster memory to allow greater bandwidth (1903-1957)and lower latency.J. Presper Eckert Jr. and John Mauchly were the first to develop the von Neuman architecture. John vonNeumann wrote "First Draft of a Report to the EDVAC" describing the ideas of a stored memory computer. Thecomplicated story is described in the wonder history of computers "Engines of the Mind" by Joel Shurkin.© M. Rauterberg, TU/e 7
  8. 8. Princeton Architecture (1946) A course on computer design based on the von Neuman concept was run by the Moore School in 1946, and was attended by British and American professionals. Also in 1946, von Neuman, Herman Goldstine and Arthur Burks published a comprehensive report of their work at Princetons Institute of Advanced Study Electronic Computer Project where they had established themselves following their involvement with the Moore School. The report detailed the operation and architecture of their work on digital computers, and has been described as the blue print for ‘modern digital computing.© M. Rauterberg, TU/e 8
  9. 9. Mainframe Computers Stretch (1961) A close-up of the Stretch technical control panel IBM SSEC (1948) From IBM Archives.© M. Rauterberg, TU/e 9
  10. 10. Enabling Technology1904 Sir John Ambrose 1947 the first Transistor wasFleming invents the 1958 Jack Kilby invented the invented by Bardeen, Brattain integrated circuit at Texas Instrumentsvacuum tube and diode. and Shockley in the Bell Labs and got the Noble Prize. Comprised of in the USA. only a transistor and other components on a slice of germanium, Kilbys invention, 7/16-by-1/16-inches in size, revolutionized the electronics industry. Intels 4004 IC chip, generally acknowledged as the worlds first "microcomputer on a chip," was originally designed in 1969-70 for the Busicom (Japan) 141-PF desktop calculator.© M. Rauterberg, TU/e 10
  11. 11. Historical overview 1945 Memex 1969 Flex 1973 Alto 1974 Bravo 1974 IBM portable computer 1983 Apple Lisa 1984 Apple Macintosh 1987 MicroSoft Windows© M. Rauterberg, TU/e 11
  12. 12. Vannevar Bush (1890-1974)“As We May Think” - 1945 Atlantic Monthly “…publication has been extended far beyond our present ability to make real use of the record.”© M. Rauterberg, TU/e 12
  13. 13. Vannevar Bush (1945)Postulated Memex device Can store all records/articles/communications Large memory Items retrieved by indexing, keywords, cross references Can make a trail of links through material etc.Envisioned as microfilm, not computer© M. Rauterberg, TU/e 13
  14. 14. Memex design sketch (1945)© M. Rauterberg, TU/e 14
  15. 15. J.R. Licklider (1915-1990)1960 - Postulated “man-computer symbiosis”Couple human brains and computing machines tightly to revolutionize information handling “The hope is that, in not too many years, human brains and computing machines will be coupled together very tightly and that the resulting partnership will think as no human brain has ever thought and process data in a way not approached by the information-handling machines we know today.” An MIT psychoacoustrician named J.C.R. Licklider took and immediate and intense interest in computer after Clark demonstrated the TX-0 to him. Licklider applied his background in psychology to research how people interacted with computers, and he became known as an expert in human-computer interaction. People at ARPA took notice and offered Licklider the job of director for their new Information Processing Techniques Office (IPTO). He accepted the position as the founding director and continued his research in human-computer interaction.© M. Rauterberg, TU/e 15
  16. 16. Vision/Goals (1945-1995)Immediate Intermediate Long-term •Time sharing •Combined speech •Natural language •Electronic I/O recognition, understanding •Interactive, real- character •Speech recognition time system recognition, light- of arbitrary users •Large scale pen editing •Heuristic programming information storage and retrieval© M. Rauterberg, TU/e 16
  17. 17. Mid 1960’sComputers too expensive for individuals -> timesharing – increased accessibility – interactive systems, not jobs – text processing, editing – email, shared file system© M. Rauterberg, TU/e 17
  18. 18. DEC PDP-1 (1961)As the worlds first commercialinteractive computer, the PDP-1was used by its purchasers topioneer timesharing systems,making it possible for smallerbusinesses and laboratories tohave access to much morecomputing power than everbefore. © DEC Inc.© M. Rauterberg, TU/e 18
  19. 19. Ivan Sutherland (1938-)SketchPad: 1963 PhD thesis at MIT – Hierarchy - pictures & subpictures – Master picture with instances – Constraints – Icons – Copying – Light pen as input device – Recursive operations© M. Rauterberg, TU/e 19
  20. 20. Sketchpad (1963) Input device: Light pen used on cathode ray tube. Graphical objects could be drawn and modified through constraints. Object oriented modell. Ivan Sutherland using the console of the TX-2 at MIT Copy and paste. © MIT Lincoln Lab© M. Rauterberg, TU/e 20
  21. 21. Douglas C. Engelbart (1925 - )s Engelbart invented the mouse at Stanford Research Labs in 1964.s Landmark system/demo: – hierarchical hypertext, multimedia, mouse, high-resolution display, windows, shared files, electronic messaging, CSCW, teleconferencing, … – Augment/NLS system [NLS: oN Line System]© M. Rauterberg, TU/e 21
  22. 22. The First Mouse (1964) Knee control Douglas Engelbart Years before personal computers and desktop information processing became commonplace or even practicable, Douglas Engelbart had invented a number of interactive, user-friendly information access systems that we take for granted today: the computer mouse, windows, shared-screen teleconferencing, hypermedia, groupware, and more.© M. Rauterberg, TU/e 22
  23. 23. Augment/NLS (1968)Augment/NLS Features: 2-dimensional display text editting, by two persons from different consoles, at the same time. Links. Video-conferencing. Mouse.© M. Rauterberg, TU/e 23
  24. 24. Alan C. Kay (1940-)Dynabook - Notebook sized computer loaded with multimedia and can store everythingPersonal ComputingDesktop InterfaceThe FLEX software© M. Rauterberg, TU/e 24
  25. 25. FLEX & Dynabook (1969) Computer should function like a living organism. Kay‘s Ph.D was about FLEX, an early object orientated language. First idea of a book-sized Computer. Use of graphic rather than text. Alan Kay developed the Dynabook at Xerox PARC. © Xerox PARC© M. Rauterberg, TU/e 25
  26. 26. Theodor (Ted) H. Nelson (1937-)s Computers can help people, not just businesss Coined term “hypertext” Ted Nelson originally invented the word "hypertext" for "non- sequential writing". His long-standing interest in all things related to HT became the Xanadu project. The Xanadu Operating Company was owned for a while by Autodesk, but later dropped.© M. Rauterberg, TU/e 26
  27. 27. Nicholas Negroponte (ca 1938-)s MIT machine architecture & AI group (1969-1980s)s Ideas: – wall-sized displays, video disks – AI in interfaces (agents), speech recognition, multimedia with hypertext © MIT MediaLab, Boston© M. Rauterberg, TU/e 27
  28. 28. Personal Computers (PC)Late ‘70’s Apple II Z-80 CP/M IBM PCText and command basedWord processingSpreadsheets© M. Rauterberg, TU/e 28
  29. 29. Input/output devices Input Output Early days connecting wires lights on display paper tape & punch cards paper keyboard teletype Past keyboard scrolling glass teletype + cursor keys character terminal + mouse bit-mapped screen + microphone audio Today data gloves + suits head-mounted displays computer jewelry ubiquitous computing natural language autonomous agents© M. Rauterberg, TU/e 29
  30. 30. IBM (1974)Mark-8 The Mark-8 was an Intel 8008 based machine with 256 bytes RAM. It was introduced in July 1974, and 1000-2000 were produced. It was the first portable computer to really be marketed, and had no ROM. The market value of a Mark-8 was a round $12,000. The machine pictured right was the precursor to the IBM 5100 machine. It was introduced in 1975, and was very costly. It was IBMs first entry into the microcomputer market. © IBM© M. Rauterberg, TU/e 30
  31. 31. IBM (1975)IBM 5100IBM 5100, introduced in September1975, was IBMs first portablecomputer.The 5100 was just one of severalportable computers IBM made beforethe Personal Computer (PC).The 5100 model was followed by the5110, the 5120, the Datamaster, andthen finally the 5150 PC.From http://www.blinkenlights.com/pc.shtml © IBM© M. Rauterberg, TU/e 31
  32. 32. IBM (1981) IBM PC Product shot of IBM Personal Computer (5150), introduced in 1981; Features: monitor, keyboard, and pin-feed printer; b/w. The operating system (OS) was by Microsoft, who licensed it to IBM as PC-DOS. Although not necessarily the best machine by technological standards, IBMs expertise and the fact that the IBM PC actually looks and feels like a professional computer system made the IBM PC and the numerous PC clones extremely popular. They have evolved into todays so-called Wintel (Windows + Intel) computer systems, used world-wide. © IBM From http://www.blinkenlights.com/pc.shtml© M. Rauterberg, TU/e 32
  33. 33. DOS (history) 1980, April: Tim Patterson begins writing an operating system for use with Seattle Computer Products 8086-based computer. Seattle Computer Products decides to make their own disk operating system (DOS), due to delays by Digital Research in releasing a CP/M-86 OS. 1980, August: QDOS 0.10 (Quick and Dirty OS) is shipped by Seattle Computer Products. Even though it had been created in only two man-months, the DOS worked surprisingly well. 1980, September: Tim Patterson shows Microsoft his 86-DOS, written for the 8086 chip. 1980, October: Microsofts Paul Allen contacts Seattle Computer Products Tim Patterson, asking for the rights to sell SCPs DOS to an unnamed client (IBM). Microsoft pays less than US$100,000 for the right. 1980, December: Seattle Computer Products renames QDOS to 86-DOS, releasing it as version 0.3. Microsoft then bought non-exclusive rights to market 86-DOS. 1981, February: MS-DOS 1.0 runs for the first time on IBMs prototype microcomputer. 1981, July: Microsoft buys all rights to DOS from Seattle Computer Products, and the name MS-DOS is adopted. 1981, August: IBM announces the IBM 5150 PC Personal Computer, featuring a 4.77-MHz Intel 8088 CPU, 64KB RAM, 40KB ROM, one 5.25-inch floppy drive, and PC-DOS 1.0 (Microsofts MS-DOS), for US$3000. 1982, May: Microsoft releases MS-DOS 1.1 to IBM, for the IBM PC. It supports 320KB double-sided floppy disk drives. Microsoft also releases MS-DOS 1.25, similar to 1.1 but for IBM-compatible computers. 1988, June: Microsoft releases MS-DOS 4.0, including a graphical/mouse interface.© M. Rauterberg, TU/e 33
  34. 34. Bill Gates (1955-)William (Bill) H. Gates is chairman and chief software architect ofMicrosoft Corporation, the worldwide leader in software, services andInternet technologies for personal and business computing. Microsofthad revenues of $25.3 billion for the fiscal year ending June 2001, andemploys more than 40,000 people in 60 countries. In his junior year,Gates left Harvard to devote his energies to Microsoft, a company hehad begun in 1975 with his childhood friend Paul Allen. Guided by abelief that the computer would be a valuable tool on every officedesktop and in every home, they began developing software forpersonal computers. Gates foresight and his vision for personalcomputing have been central to the success of Microsoft and thesoftware industry.Under Gates leadership, Microsofts mission has been to continually advance and improvesoftware technology, and to make it easier, more cost-effective and more enjoyable for people touse computers. The company is committed to a long-term view, reflected in its investment of morethan $4 billion on research and development in the current fiscal year 2001.© M. Rauterberg, TU/e 34 © source http://www.microsoft.com/billgates/bio.asp
  35. 35. MS DOS (1981)Microsoft DOS (Disk OperatingSystem) is a command line userinterface.Microsoft releases MS-DOS 1.0to IBM, for the original IBM PCin 1981.In 1982 MS-DOS 1.1 supports320KB double-sided floppy diskdrives.Microsoft also releases MS-DOS 1.25, similar to 1.1 but forIBM-compatible computers with720 KB floppy disk drives. © Microsoft Inc.© M. Rauterberg, TU/e 35
  36. 36. PCs with GUIsXerox PARC - mid 1970’s – invention of the first PC: ALTO – Local processor, Bitmap display, Mouse – Precursor to modern GUI – LAN - Ethernet© M. Rauterberg, TU/e 36
  37. 37. Xerox Alto (precursor to the Star)Alto applications:Bravo WYSIWYG text editor.BravoX Mesa implementation ofBravo, an ancestor of Microsoft Word.Laurel Electronic mail program.Neptune Disk file manipulationprogram, sort of like sweep.Works like the Font DA moverprogram on a Macintosh.Press Document printing program.Sil Drawing program. From Xerox Alto Archive © M. Rauterberg, TU/e 37
  38. 38. Xerox Star - ‘81First commercial PC designed for “business professionals” Desktop metaphor, pointing, WYSIWYGFirst system based on usability engineering© M. Rauterberg, TU/e 38
  39. 39. Xerox Star (1981)© M. Rauterberg, TU/e 39
  40. 40. Xerox Star (GUI)© M. Rauterberg, TU/e 40
  41. 41. Xerox Star (history)Commercial flop – $15k cost – closed architecture – lacking key functionality (spreadsheet)© M. Rauterberg, TU/e 41
  42. 42. Apple II (1977)Built in 1977, the Apple II was based on WozniaksApple I design, but with several additions. The firstwas the design of a plastic case--a rarity at the time--which was painted beige. The second was the abilityto display color graphics--a holy grail in the industry.The Apple II also included a larger ROM, moreexpandable RAM (4K to start), and 8 expansion slots. Steve Wozniak (1950-)It had integer BASIC hard-coded on the ROM foreasier programming, and included two game paddles © Apple Inc.and a demo cassette for $1,298. In early 1978 Applealso released a disk drive for the machine, one of themost inexpensive available. The Apple II remained on 1979, Februarythe Apple product list until 1980. Apple Computer releases DOS 3.2. 1979, July Apple Computer releases DOS 3.2.1© M. Rauterberg, TU/e 42
  43. 43. Apple Lisa - ‘82Based on ideas of StarMore personal rather than office tool Still expensive!Conceptual success, but commercial failure Steve Jobs (1955-) co-founder Apple Computer Corporation© M. Rauterberg, TU/e 43
  44. 44. Apple Lisa (1983) Named for one of its designers daughters, the Lisa (pictured left) was supposed to be the Next Big Thing. It was the first personal computer to use a Graphical User Interface (GUI). Aimed mainly at large businesses, Apple said the Lisa would increase productivity by making computers easier to work with. The Lisa had a Motorola 68000 Processor running at 5 Mhz, 1 MB of RAM two 5.25" 871k floppy drives, an external 5 MB hard drive, and a built in 12" 720 x 360 monochrome © Apple Inc. monitor. http://www.apple-history.com/lisa.html© M. Rauterberg, TU/e 44
  45. 45. Apple Lisa (applications)LisaWrite: word processorLisaCalc: spread sheetLisaGraph: chartsLisaList: an outline builder, idea managerLisaProject: project schedulerLisaDraw: drawing program(predecessor to Mac Draw)LisaTerminal: modem communicationssoftware. © Apple Inc.© M. Rauterberg, TU/e 45
  46. 46. Apple Macintosh - ‘84Aggressive pricing - $2500Not trailblazer, smart copierGood interface guidelines3rd party applicationsHigh quality graphics and laser printer© M. Rauterberg, TU/e 46
  47. 47. Apple Macintosh (1984) Released in January of 1984, the Macintosh was the first affordable computer to include a Graphical User Interface (GUI). It was built around the new Motorola 68000 chip, which © Apple Inc. was significantly faster than previous processors, running at 8 MHz. The Mac came in a small beige case with a black and white monitor built in. It came with a keyboard and mouse, and had a floppy drive that took 400k The early Mac team members (1979) consisted of Jeff Raskin, Brian Howard, Marc LeBrun, Burrell Smith, Joanna Hoffman and Bud 3.5" disks--the first personal computer Tribble. to do so.© M. Rauterberg, TU/e 47
  48. 48. Apple Macintosh (GUI) © Apple Inc.© M. Rauterberg, TU/e 48
  49. 49. Apple Macintosh (history) Problem of the Mac: Only 128k main memory Well written applications: MacWrite and MacDraw Mac 512k, Mac512ke and Mac Plus were introduced to save the Mac New applications: Pagemaker, Word Excel© M. Rauterberg, TU/e 49
  50. 50. MS Windows (1987)Release of the IBM PC AT in Windows planed 1983 wasAugust 1984 running at 6MHz released on August 11, 1987Windows 1.01was a largedisappointment! © Microsoft Inc.© M. Rauterberg, TU/e 50
  51. 51. Microsoft Windows (history) Steve Jobs from Apple Inc. complained about the stolen Mac OS’s interface design. Bill Gates from Microsoft Inc. replied: “Hey Steve, just because you broke into Xerox’s house before I did and took the TV doesn’t mean I can’t go in later and take the stereo.” Microsoft Windows 2.03 was released January 1988. Microsoft Windows 3.1 was released in 1992.© M. Rauterberg, TU/e 51
  52. 52. Oberon at ETH (1985) Oberon is the name of a modern integrated software environment. It is a single-user, multi-tasking system that runs on bare hardware or on top of a host operating system. Oberon is also the name of a programming language in the Pascal/Modula tradition. The Oberon project was launched in 1985 by Niklaus Wirth and Jürg Gutknecht at the ETH in Zurich. Although the Niklaus Wirth (1934-) project was originally targeted towards in-house hardware, the language and system have now been ported to many computer platforms. Oberon is the first worldwide modeless software system: completely modeless graphical user interface with support for graphical primitives; both overlapping and tiled windowing systems supported concurrently; possibility of Jürg Gutknecht configuring the working environment.© M. Rauterberg, TU/e 52
  53. 53. Ben Shneiderman (1947-)Dr. Shneiderman is the author of ‘Software Psychology: Human Factors inComputer and Information Systems’ (1980) in which he coined the termdirect manipulation. Later he wrote the influential text book ‘Designingthe User Interface: Strategies for Effective Human-Computer Interaction’(1987, third edition 1998).Ben Shneiderman has written over 200 articles and published severalbooks, including Elements of FORTRAN Style: Techniques for EffectiveProgramming (with Charles Kreitzberg, 1972); and Hypertext Hands-On!An Introduction to a New Way of Organizing and Accessing Information(with Greg Kearsley, 1989). He has also edited numerous articles andseveral books, including Directions in Human/Computer Interaction(1982) and Sparks of Innovation in Human-Computer Interaction (1993).Ben Shneiderman was a Professor in the Department of Computer Science, Founding Director(1983-2000) of the Human-Computer Interaction Laboratory, and Member of the Institutes for AdvancedComputer Studies and for Systems Research, all at the University of Maryland at College Park.© M. Rauterberg, TU/e 53
  54. 54. Historical Overview (1945-1995) [source: Brad A. Myers (1998). A brief history of human-computer interaction technology. Interactions, vol 5(2), pp. 44-54]© M. Rauterberg, TU/e 54
  55. 55. Mobile Phone (1954) Growing up in Chicago, Martin Cooper earned a degree in electrical engineering at the Illinois Institute of Technology. Later hired by Motorola in 1954, he lead a group of research team to develop the first ever portable phone, the Motorola Dyna Tac, which stands for Dynamic Adaptive Total Coverage. Weighing 1089 gram, the first commercially viable version of the Dyna Tac was released in 1983. It measures 9 x 5 x 1.75 inches in size with 30 circuit boards. Unlike handsets today, there was no display available and the only feature was call, dial and listen (what else would you expect?). The heavy batteries can only withstand 35 minutes of talk time and need a long 10 hour recharge. Martin Cooper Some important years in the mobile history, •1955 introducing the worlds first whole automatic mobile-phone system. •1972 A global system is presented. Covers all the oceans of the world. •1978 introducing the worlds first person searching-system with a number-display. •1981 The world’s first automatic and boundless mobile-phone system. •1986 First time when you can transfer computer services via a mobile system.Dyna Tac •1988 The pocket-phone is introduced. © M. Rauterberg, TU/e 55
  56. 56. What is Virtual Reality (VR)?In 1989, Jaron Lanier, CEO of VPL, coined the term virtual reality to bring all of thevirtual projects under a single rubric. The term therefore typically refers to three-dimensional realities implemented with stereo viewing goggles and reality gloves.Myron Krueger (1991):….The term (virtual worlds) typically refers to three-dimensional realitiesimplemented with stereo viewing goggles and reality gloves.George Coates (1992):Virtual Reality is electronic simulations of environments experienced viahead mounted eye goggles and wired clothing enabling the end user to Myron Kruegerinteract in realistic three-dimensional situations.P. Greenbaum (1992):Virtual Reality is an alternate world filled with computer-generated images that respondto human movements. These simulated environments are usually visited with the aid ofan expensive data suit which features stereophonic video goggles and fiber-optic gloves.© M. Rauterberg, TU/e 56
  57. 57. Dimensions to define VR Vividness (richness of an environments representation) • breadth (visibility, audibility, touch, smell) • depth (quality, fidelity) Interactivity (extend to which a user can modify form and content of a mediated environment) • speed (update rates, time lag) • mapping (text, speech, gestures, gaze, complex behavior patterns)© M. Rauterberg, TU/e 57
  58. 58. Classification of VR and other Media From Jonathan Steuer© M. Rauterberg, TU/e 58
  59. 59. History of VR (technological milestones)1956 Sensorama (Morton Heilig) 3D visuals, vibration, stereo sound, wind, smell, little interaction.1961 Headsight System (Philco Corp.) Head Mounted Display (HMD), head tracking, remote video camera, telepresence.1965 The Ultimate Display (Ivan Sutherland) Stereoscopic HMD, computer generated images, tracking, visually coupled system.1967 Grope (University of North Carolina) 6 degree of freedom force feedback.1977 The Sayre Glove (Sandin, Sayre, DeFanti Univ. Illinois) Gesture recognition.1987 Virtual Cockpit (British Aerospace) head and hand tracking, eye tracking, 3d visuals, 3D audio, speech recognition, vibro tactile feedback.© M. Rauterberg, TU/e 59
  60. 60. Morton Heilig In the late 50s, a quiet man named Morton Heilig began designing the first multisensory virtual experiences. He developed something called the Sensorama. Resembling one of todays arcade machines, the Sensorama combined projected film, audio, vibration, wind, and even prepackaged odors, all designed to make the users feel as if they were actually in the film rather than simply watching it. Since real-time computer graphics were many years away, the entire experience was prerecorded, and played back for the user. Although he was a gifted and visionary inventor, Heilig was less successful as a businessman. He was unable to get funding for his Sensorama machines, and they were never manufactured. Fortunately, he didnt give up there; Heilig had an idea that would later prove to be the basis for an entire industry: the first Head-Mounted Display (HMD), which he patented in 1962.© M. Rauterberg, TU/e 60
  61. 61. Sensorama (1956)Sensorama is a simulator that gives oneperson at a time an illusion of reality. It is asemi-portable automatic machine that can beplugged in anywhere electricity is available.The illusion of reality is achieved byproviding the viewer with a wide range ofsensory information. All of this information isperfectly synchronized with the picture(aromas, wind, and vibrations changeinstantly). All the control information is onone piece of film. Sensorama is completelyautomatic. The viewer activates it bydepositing a coin or pushing a button(depending on application).© M. Rauterberg, TU/e 61
  62. 62. Headsight System (1961) (© nVision) first HMD (1961) Sutherland & Sproull (Harvard, 1967) Datavisor 80© M. Rauterberg, TU/e 62
  63. 63. Ultimate Display (1965)Ivan Sutherland is a pioneer in the field of computergraphics and in 1965 he described The UltimateDisplay, which included interactive graphics andforce-feedback devices.In 1968, he described a prototype virtual realitysystem in his paper A head-mounted three-dimensional display.But it was a team at the NASA Ames ResearchCenter who really opened up the possibilities ofvirtual reality worlds with their Virtual InterfaceEnvironmental Workstation (VIEW), developedduring the 1980s as a training system for futureastronauts. © NASA Ames Research Center© M. Rauterberg, TU/e 63
  64. 64. Grope (1967) Haptic Display Grope III The Force-Feedback Project, which began in 1967, first focused on the development of a system to support scientific visualization in the area of molecular docking, the Docker application. This application provides graphic (wire-frame) representations of molecules and their inter- atomic forces to allow a user to adjust the relative position and orientation of molecules while searching for minimum energy binding sites. A series of systems have been developed, evolving from a 2-D system, through a 3-D system and a 6-D system for a simple docking task, to a full 6-D molecular docking system called GROPE-III. These later systems have employed a modified Model E-3 Argonne Remote Manipulator (ARM). © Fred Brooks, University of North Carolina© M. Rauterberg, TU/e 64
  65. 65. Sayre Glove (1977)Early contributions to computer graphics included performances with real-timegraphics accompanied by music and the use of its hardware and software forcreating the computer animation of the Death Star "schematic" for the first StarWars.In 1976, based on an idea by colleague Rich Sayre, DeFanti and Sandin developedan inexpensive, lightweight glove to monitor hand movements; the Sayre Gloveprovided an effective method for multidimensional control, such as mimicking aset of sliders.Projects in the 1970s through mid-1980s centered on video game technology, real-time computer animation on microcomputers, and interactive multimediainstallations. © University of Illinois at Chicago© M. Rauterberg, TU/e 65
  66. 66. Digital Data Entry Glove (1985)Digital Data Entry Glove, T. G. Zimmerman & Y. Harvill• first glove-like device (cloth) onto which numerous touch, bend,and inertial sensors were sewn.• measured finger flexure, hand-orientation and wrist-position,and had tactile sensors at fingertips.• orientation of hand tracked by video camera; required clearline-of-sight observation for the glove to function.• designed as alternative to keyboard; matched recognizedgestures/hand orientations to specific characters, specifically torecognize the Single Hand Manual Alphabet for the AmericanDeaf; circuitry hard-wired to recognize 80 unique combinationsof sensor readings to output a subset of the 96 printable ASCIIcharacters; a tool to “finger-spell” words.• finger flex sensors, tactile sensors at the fingertips, orientationsensing and wrist-positioning sensors; positions of sensors werechangeable.• US Patent 4,542,291: Zimmerman & Harvill, Optical FlexSensor, September 17, 1985 [ACM CHI paper 1987] © VPL Research, Inc.© M. Rauterberg, TU/e 66
  67. 67. VIEW (ca.1985) The VIEW configuration included a head-mounted display, head and hand tracking, speech recognition, three- dimensional audio output, and a tracked and instrumented glove. The glove was the interface through which the user could interact with the virtual world. A graphical representation of the glove moved around the virtual world in response to the users hand movements. The glove had fibreoptics embedded in it and these detected changes in finger positions, while a separate motion sensor detected the position of the hand. The computer recalculated the coordinates of the gloves image based its movements © NASA Aerospace Lab.© M. Rauterberg, TU/e 67
  68. 68. Virtual Cockpit (1987) © British Aerospace Lab.© M. Rauterberg, TU/e 68
  69. 69. Input Devices (overview)Sensor Devices 1. Spatial Position/Orientation Sensors • 2DOF (Mouse) • 3DOF (Microscribe, FreeD Joystick) • 6DOF (Polhemus Fastrack) 2. Directional Force Sensors • 5 DOF (Spacemouse) • 2 DOF (Joystick) 3. Gesture Recognition • Data Gloves 4. Eye Tracking 5. Speech Recognition Systems© M. Rauterberg, TU/e 69
  70. 70. Input Devices (1)Directional Force Sensors SpaceMaster© M. Rauterberg, TU/e 70
  71. 71. Input Devices (2)Gesture Recognition Dextrous Hand Master, Exos SUPERGLOVE, Nissho Cyberglove , 5th Dimension© M. Rauterberg, TU/e 71
  72. 72. Input Devices (3)Spatial Position/Orientation Sensors Polhemus InsideTrack MicroScribe FreeD Joystick (Magnetic Tracking) (Mechanical Tracking) (UltraSonic Tracking)© M. Rauterberg, TU/e 72
  73. 73. Input Devices (4)Visual Haptic Workbench The Visual Haptic Workbench consists of five hardware components. The dominant hand of the user experiences haptic feedback from the PHANToM, and the subdominant hand navigates through a menu interface via Pinch glove contact gestures. Head tracking is done with a Polhemus Fastrak receiver mounted on a pair of Stereographics CrystalEyes LCD shutter glasses. The subdominant hand can also be tracked with a separate receiver to facilitate more complex interaction [see also http://haptic.mech.nwu.edu/intro/gallery/] paradigms. The audio subsystem gives the user additional reinforcement cues to clarify © M. Rauterberg, TU/e 73
  74. 74. Output Devices (1) NASA HMD (ca 1985)© M. Rauterberg, TU/e 74
  75. 75. Output Devices (2) Four-sided CAVE (1992) An Immersive VR Environment © Electronic Visualization Laboratory University of Illinois at Chicago© M. Rauterberg, TU/e 75
  76. 76. Output Devices (3) Six-sided CAVE (1998) The KTH Six-sided CAVE built by TAN GmbH. In October 1998 TAN finished the worlds first 6-sided TAN VR-CUBE™ for the Royal Institute PDC/KTH in Stockholm/Sweden. © KTH, Stockholm This type of VR-CUBE™ complete encloses the user from all sides.© M. Rauterberg, TU/e 76
  77. 77. Output Devices (4)CyberSphere (1998) The scientists Eyre and Eureka in VR-Systems UK have been researching a CyberSphere, a device, which consists of a large, translucent sphere containing the user. The images are distortion-corrected and then projected on the surface of the sphere, allowing the user a full 360 degree field of view. It also allows the user to move around in the world, by walking inside the ball, which will move in response to the users movements. © VR-Systems, United Kingdom© M. Rauterberg, TU/e 77
  78. 78. Ubiquitous Computing (1991) Ubiquitous computing is just now beginning. First were mainframes, each shared by lots of people. Now we are in the personal computing era, person and machine staring uneasily at each other across the desktop. Next comes ubiquitous computing, or the age of calm technology, when technology recedes into the background of our lives. Mark Weiser is the father of ubiquitous computing (1991). [Mark Weiser, "The Computer for the Twenty-First Century”, Scientific American, Mark Weiser (1952-1999) pp. 94-10, Sept. 1991] What Ubiquitous Computing Isnt Ubiquitous computing is roughly the opposite of virtual reality. Where virtual reality puts people inside a computer-generated world, ubiquitous computing forces the computer to live out here in the world with people. Virtual reality is primarily a horse power problem; ubiquitous computing is a very difficult integration of human factors, computer science, engineering, and social sciences.© M. Rauterberg, TU/e 78
  79. 79. Apple’s Newton (1992) Apple Computers Newton line of personal digital assistants (PDA) began as CEO John Scullys pet project in 1992. Since then, seven different Newton models were released before "iCEO" Steve Jobs came back to Apple: in 1998 he quickly axed the newly formed subsidiary responsible for the Newton line, Newton, Inc. A small, hand-held device with pen-input, personal organizational functions, and communication capabilities becomes the first shipped Newton product by Apple Inc. Although intelligent software is an essential element, this product is viewed as a smaller, cheaper, stripped-down version of the initial Newton concept that can be mass produced. © Apple Computing Inc.© M. Rauterberg, TU/e 79
  80. 80. PARCtab (1993) The PARCtab is most easily operated with two hands: one to hold the tab, the other to use a passive stylus or a finger to touch the screen. But since office workers often seem to have their hands full, we designed the tab so that three mechanical buttons fall beneath the fingers of the same hand that holds the tab, allowing one-handed use. The device also includes a piezo-electric speaker so that applications can © Xerox PARC generate audio feedback© M. Rauterberg, TU/e 80
  81. 81. Unistroke alphabet Techniques for handwriting recognition have improved in recent years, and are used on some PDAs for text entry. But they are still far from ideal since they respond differently to the unique writing characteristics of each operator. Xerox PARC have experimented on the PARCTAB with Unistrokes, which depart from the traditional approach in that they require the user to learn a new alphabet---one designed © Xerox PARC specifically to make handwriting easier to recognize© M. Rauterberg, TU/e 81
  82. 82. Half-QWERTY (1993) Typing With One Hand Using Your Two-handed Skills Half-QWERTY is a one-handed typing technique, designed to facilitate the transfer of two- handed typing skill to the one- handed condition © Matias, MacKenzie & Buxton, Toronto Bill Buxton© M. Rauterberg, TU/e 82
  83. 83. PalmPilot (1996) The PalmPilot has a lot functionality. This device fits with its pocket size into one hand. There is a communication channel via IR to the PC. Small, and a reasonable price © Palm Computing Inc.© M. Rauterberg, TU/e 83
  84. 84. PalmPilot alphabet Input similar to “natural” alphabet not user specific minimize the user’s learning and adapting costs © Palm Computing Inc.© M. Rauterberg, TU/e 84
  85. 85. Newton MessagePad 2x00 (1997)The final incarnation of the Newton OS camepackaged in the MessagePad 2100, an internet-ready, heap-increased version of the MessagePad2000. The 161.9Mhz StrongArm processor, whichPalm is just beginning to port PalmOS to in 2002, isthe true reason the Newton 2X00 line is still one ofthe more functional PDAs of these days.When the MP2K was first released in 1997, unlikethe PalmPilot, they were meant to work to laptopfunctionality rather than act as a digitized datebook.This means that you can browse the Web in fullgrayscale (and perhaps Javascipt), number crunchon spreadsheets, use graphing calculators,wordsmith in a functional word processor, and so on.Movies have been made to run on Newtons andMp3 syncing with iTunes has been implemented. © Apple and Newton, Inc.© M. Rauterberg, TU/e 85
  86. 86. Brian Shackel (ca 1920-) Brian Shackel is Emeritus Professor of Ergonomics at Loughborough University and the founder of the HUSAT Research Institute. He is one of the pioneers in applying human factors and ergonomics to computer systems. After graduating from Cambridge and following military service, he set up EMI’s human factors group and worked on many systems and product user interfaces. He joined Loughborough University of Technology and set up HUSAT in 1970. He has been an advocate international standards committees and publishing widely. He is the founder and first chairman of the IFIP Technical Committee “Human-Computer Interaction” in 1989.The prestigious BRIAN SHACKEL AWARD is associated with each IFIP TC13 INTERACTConference, usually biennial, and is to recognise the most outstanding contribution in the formof a refereed paper submitted to and delivered at the INTERACT conference.“From early days (cf Licklider & Clark, 1962) the need for larger displays has been emphasised; but just when it seemed, in the late1980s, that full page and larger displays would come with lower prices, the focus in the industry turned to portability and we movedbackward to smaller screens. While there was some improvement, larger screens (eg 21 inch CRT displays) are still not available atan acceptable price; as long ago as 1977 Kay & Goldberg (1977) in their Dynabook concept specified a display the size of a fullpaper page, but I know of no portable laptop, let alone notebook, which has an A4 page size screen.” (Brian Shackel, 2000)© M. Rauterberg, TU/e 86
  87. 87. Tim Berners-Lee (1955-)BORN June 8, 1955, in London1976 Graduates from Queens College, Oxford1980 While at CERN, writes "Enquire"1989 Proposes global hypertext project called "WorldWideWeb"1991 The Web debuts on the Internet1993 University of Illinois releases Mosaic browser1994 Joins M.I.T. to direct the W3 consortium Tim Berners-Lee is considered to be the1999 Today nearly 150 million people log on to the Internet via founder of theWWW World Wide Web. “First of all, lets get clear the difference. The internet is a collection of computers, which was put together during the 1970s. When I proposed the Web in 1989, the internet had been around for 15 years. You could use e-mail, you could store files on ftp servers, and people could access them, but it was very complicated. The web was the step to make accessing a remote document just one click. The internet spread really quite slowly. It started in research, moved into universities, and many people only heard about it when the web became available as an easy way to use it. “ (T. Berners-Lee, 1999)© M. Rauterberg, TU/e 87
  88. 88. World Wide Web (1990)In Europe, researchers at CERN (the European Laboratory for Particle Physics) were struggling with their own computer networkingproblems. Throughout the system people used different techniques, protocols, and equipment, making communication betweencomputers very complex. In 1980, Tim Berners-Lee, a consultant at CERN, wrote a program called "Enquire-Within-Upon-Everything,"enabling links to be made between any point in the system. Nine years later Berners-Lee wrote "Information Management: A Proposal:"Instead of standardizing the equipment or software, they created standards for data, and a universal addressing system. That way anydocument on the Internet could be retrieved and viewed. In 1990, CERN was the largest Internet site in Europe. Over the next year ortwo, the proposal was circulated and revised, resulting in an initial program being developed that was dubbed the World Wide Web. Atleast one expert has called the Web a "side effect of CERNs scientific agenda." In 1992, the World Wide Web was demonstrated anddistributed, and browser software was released throughout and beyond CERN. That November there were about 26 reliable Web servers.All you needed to use the Web was a browser. The early browsers were functional but not especially "user-friendly." A youngprogrammer at the National Center for Supercomputing Applications (NCSA) named Marc Andreesen created a new graphical Webbrowser. This was pleasing to the eye and easy to use -- just point-and-click. Users didnt need to know any programming or even anyInternet addresses. It also made it fairly simple for users to add their own material to the Web. Andreesen and his coworkers called thisbrowser Mosaic, and released free versions for Windows and Macintosh in August of 1993. Interest in the Web -- especially commercialinterest -- exploded with the arrival of Mosaic. By October there were more than 200 Web servers, and at the end of 1993, Mosaic wasbeing downloaded from NCSA at a rate of 1,000 copies per day. By June 1994, there were 1,500 Web servers.In July 1993, there were 1,776,000 hosts in 26,000 domains; by July 1996, there were 12,881,000 hosts in 488,000 domains. In July1996, there were 3,054 Internet service providers and projections of Web user sessions rising to 15.79 billion in the year 2000.© M. Rauterberg, TU/e 88
  89. 89. Historical Overview: Robots 1818 Mary Shelley, Frankenstein 1827 Joseph Atterly, A Voyage to the Moon 1863 Jules Verne, A Journey to the Center of the Earth 1865 Edward S. Ellis, The Steam Man of the Prairies 1870 Jules Verne, 20,000 Leagues Under the Sea 1895 H.G. Wells, The Time Machine 1920 Karel Capek, R.U.R. (first use of the word "robot") 1921 - The term "robot" was first used in a play called "R.U.R." or "Rossums Universal Robots" by the Czech writer Karel Capek. 1941 - Science fiction writer Isaac Asimov first used the word "robotics" to describe the technology of robots and predicted the rise of a powerful robot industry. 1948 - "Cybernetics", an influence on artificial intelligence research was published by Norbert Wiener. 1956 - George Devol and Joseph Engelberger formed the worlds first robot company. 1959 - Computer-assisted manufacturingg was demonstrated at the Servomechanisms Lab at MIT. 1961 - The first industrial robot was online in a General Motors automobile factory in New Jersey. It was called UNIMATE. 1963 - The first artificial robotic arm to be controlled by a computer was designed. The Rancho Arm was designed as a tool for the handicapped and its six joints gave it the flexibility of a human arm© M. Rauterberg, TU/e 89
  90. 90. UNIMATE (1961) UNIMATE, the first industrial robot, began work at General Motors. Obeying step-by-step commands stored on a magnetic drum, the 4,000-pound arm sequenced and stacked hot pieces of die- cast metal. The brainchild of Joe Engelberger and George Devol, UNIMATE originally automated the manufacture of TV picture tubes.© M. Rauterberg, TU/e 90
  91. 91. Wabot-1 (1973) The WABOT-1 was the first fun-scale anthropomorphic robot developed in the world at Tokyos Waseda University under Ichiro Kato. It consisted of a limb-control system, a vision system and a conversation system. The WABOT-1 was able to communicate-with a person in Japanese and to measure Ichiro Kato distances and directions to the objects using external receptors, artificial ears and eyes, and an artificial mouth. The WABOT-1 walked with his lower limbs and was able to grip and transport objects with hands that used tactile-sensors. It was estimated that the WABOT-1 has the mental faculty of a one-and-half-year-old child. WABOT-1 consisted of the WAM-4 (as its artificial hands) and the WL-5 (Its artificial legs). WABOT-1© M. Rauterberg, TU/e 91
  92. 92. SONY’s AIBO (1999) 1999 marked a turning point for the world of entertainment when Sony introduced the electronic robot AIBO in Japan. AIBO, referred to as the Entertainment Robot means “companion” and is an acronym for Artificial Intelligence RoBOt. Sony has developed two versions of AIBO, the first run ERS-110 and the newer ERS-111. Both of the models have proven to be a large success. When the first 5,000 ERS-110 models were introduced 3,000 sold in Japan within twenty minutes on the Internet. The 2,000 models that remained were made available exclusively for the United States and were sold within four days on the Internet. Demand for the new AIBO ERS-111, which has a 64-bit microprocessor and 32 bytes of memory, rose immensely. Ten thousand ERS-111 AIBO models were made available despite receiving 135,000 orders. “The digital ‘bot is more dexterous than its predecessor: In addition to all the usual tricks—heeling or chasing a ball—it does a little dance and waves a front paw on hearing its name. Speech-recognition software lets it learn up to 50 commands” (International Business, # 3709, page 170). The cost for AIBO in the United States has remained thus far steady at $2,500 with the option to purchase a $500 performer kit. AIBO is attempting to become popular all over the world, however, 90% of AIBO purchases still come from Japan.© M. Rauterberg, TU/e 92
  93. 93. SDR-3X (2000)Much like its predecessors Aibo and Aibo II, Sonys 50cm-tall, 50kg prototype SDR-3X has been designed toentertain. It can perform a variety of relatively high-speed, autonomous movements, including walking at aspeed of 15 meters a minute and dancing to a tune with a quick tempo. Its equipped with speech- and image-recognition functions.The SDR-3X gave a demonstration by walking at a high speed, moving its body like a gymnast, dancing to adisco tune with a quick tempo and kicking a ball into a goal net as instructed by voice.The SDR-3X model employs the same OPEN-R architecture as Sonys autonomous entertainment canine robot"AIBO." Similar to AIBO, the SDR-3X model recognizes human voices and images. It has an 18,000-pixel CCDcolor camera in its head area. The SDR-3X model can maintains its balance in the upper half of the body bytwisting its body and moving its arms, thereby realizing stable walking movements with alternating feet.The SDR-3X model has 24 joints, with actuators for each joint that help move the joints. It has two joints in theneck, two in the trunk, four in each arm and six in each leg. Two 64-bit RISC processors enable real-time controlof the joints to realize autonomous movement.Sonys original real-time operating system called "Aperios" is used for the SDR-3X.The biggest challenge for a humanoid robot is to keep its body in full balance. The SDR-3X keeps balance bymoving its arms and twisting at the waist to counteract the yaw-axis moment, the force needed to turn the bodyright and left, which is generated from the lower half of the body every time the robot takes one step forward inthe high-speed walking movement.And the robots posture is controlled in real-time, to prevent it from falling over. It uses a variety of information,such as the angle of the floor, gathered from contact sensors in the torso section as well as from a "dual-axisaccelerometer" and a "dual-axis angular rate sensor" in the waist section, whenever the robot walks up slopesand moves its whole body.It can (1) move forward or backward and walk sideways at a speed of up to 15 meters a minute; (2) turn left orright when walking (with the maximum 90 degrees of freedom for each step forward); (3) get up from theposition of lying on its stomach or its back; (4) stand on one leg (possible even on inclined ground); (5) walk ona bumpy road; (6) kick a ball; and (7) dance to a wide range of tunes.© M. Rauterberg, TU/e 93
  94. 94. Augmented Reality (AR)Core aspects:User sees real environment; combines virtual with real.Supplements reality, instead of completely replacing it.Photo-realism not necessarily a goal.© M. Rauterberg, TU/e 94
  95. 95. AR (historical overview)•Early 1990’s: Boeing coined the term “AR”. Wire harness assembly application begun.•Early to mid 1990’s: UNC ultrasound visualization project•1994: Motion stabilized display [Azuma]•1994: Fiducial tracking in video see-through [Bajura / Neumann]© M. Rauterberg, TU/e 95
  96. 96. AR at Boeing (1990) The term "augmented reality" was coined at Boeing in 1990 by researcher Tom Caudell. He and a colleague, David Mizell, were asked to come up with an alternative to the expensive diagrams and marking devices then used to guide workers on the factory floor. They proposed replacing the large plywood boards, which contained individually designed wiring instructions for each plane, with a head-mounted apparatus that would display a planes specific schematics through high-tech eyeware and project them onto multipurpose, reusable boards. Instead of reconfiguring each plywood board manually in each step of the manufacturing process, the customized wiring instructions would essentially be worn by the worker and altered quickly and efficiently Tom Caudell through a computer system. © Boeing Inc., USA© M. Rauterberg, TU/e 96
  97. 97. Motion-stabilized AR Display (1994) This system is the first motion-stabilized AR display that works outdoors and achieves tighter registration than any previous outdoor AR system. © Ronald Azuma, HRL Laboratories© M. Rauterberg, TU/e 97
  98. 98. Fiducial tracking (1994) The AR tracking system of Bajura and Neumann consists of a robust vision landmark tracker, inertial gyro sensors, and the complementary fusion filter described above. The sensor module contains a CCD video camera (Sony XC-999 with 6mm lens), and three orthogonal rate gyroscopes (GyroChip II QRS14-500-103, from Systron Donner), which are tightly covered by a foam block to provide shock protection and a stable temperature environment from the sensors. The video camera provides a 30 Hz video stream, while the three gyroscopes are sampled at 1kHz via a 16-bit A/D converter (National Instruments DAQPCI-AI-16XE-20). © University of Southern California© M. Rauterberg, TU/e 98
  99. 99. MIT Wearables (1996-) Whats a Wearable? To date, personal computers have not lived up to their name. Most machines sit on the desk and interact with their owners for only a small fraction of the day. Smaller and faster notebook computers have made mobility less of an issue, but the same staid user paradigm persists. Wearable computing hopes to shatter this myth of how a computer should be used. A persons computer should be worn, much as eyeglasses or clothing are worn, and interact with the user based on the context of the situation. With heads-up displays, unobtrusive input devices, personal wireless local area networks, and a host of other context sensing and communication tools, the wearable computer can act as an intelligent assistant, whether it be through a Remembrance Agent, augmented reality, or intellectual collectives.© M. Rauterberg, TU/e 99
  100. 100. Digital Desk (1991) Pierre WellnerThe DigitalDesk is built around an ordinary physical desk and can be used as such, but it has extra capabilities. A video camera ismounted above the desk, pointing down at the work surface. This cameras output is fed through a system that can detect where theuser is pointing (using an LED-tipped pen) and it can recognise the documents that are placed on it. The more advanced versionalso has a computer-driven projector mounted above the desk enabling electronic objects to be projected onto real paperdocuments -- removing the burden of having to switch attention between screen and paper and allowing additional user-interactiontechniques. [invented and built by Pierre Wellner, Xerox EuroPARC, ACM CHI paper] © Xerox EuroPARC, UK© M. Rauterberg, TU/e 100
  101. 101. ImmersaDesk (1994) The ImmersaDesk is a drafting-table format virtual prototyping device. Using stereo glasses and sonic head and hand tracking, this projection- based system offers a type of virtual reality that is semi-immersive. The ImmersaDesk features a 4x5-foot rear-projected screen at a 45-degree angle. The size and position of the screen give a sufficiently wide-angle view and the ability to look down as well as forward. The resolution is 1024 x 768 at 96Hz. © University of Illinois at Chicago© M. Rauterberg, TU/e 101
  102. 102. Build-It (1996)design team with experts from different disciplines are planning togetherintegration of different expertiseintuitive interaction style with a tangible object ‘brick’an example for a natural user interface © ETH, Zurich© M. Rauterberg, TU/e 102
  103. 103. Cubby (1997)Cubby, a desktop VR system,addresses the problem ‘ unification ofthe display and manipulation spaces’.With a pen-like input device, the usercan operate on virtual objects wherethey appear.Unlike many VR systems, Cubby iswell-suited to precision manipulationtasks. Possible areas of applicationinclude surgical simulation andcomputer aided modeling. Invented and built by Tom Djajadiningrat © Technical University Delft, The Netherlands© M. Rauterberg, TU/e 103
  104. 104. Virtual Workbench (1998) Kent Ridge Digital Labs (KRDL), Singapore © Kent Ridge Digital Labs© M. Rauterberg, TU/e 104
  105. 105. DynaWall & CommChairs (1998) The size of the DynaWall opens a new set of human-computer interactions. It is possible that information objects can be taken at one position and put somewhere else on the display or thrown from one side to the opposite side. Dialog boxes always appear in front of the current user(s). User interface components are always © GMD IPSI, Germany at hand, etc.© M. Rauterberg, TU/e 105
  106. 106. Horizontal 3D Displays (1998) © CMD, Uppsala University© M. Rauterberg, TU/e 106
  107. 107. Speech and Voice in User Interface Application areas • Control and Data input in “hands-busy” environments • Feedback in visually limited environments • System control over telephone-line • Control for impaired or disabled people© M. Rauterberg, TU/e 107
  108. 108. Speech Signal Processing (1949) J. Dreyfus-Graf (1949). “Sonograph and Sound Mechanics”. The Journal of the Acoustic Society of America, 22, pp. 731-739 This was not a real speech recognizer but an oscilloscope that would put the beam at a different spot depending of the content of the speech. Raymond Kurzweil: "[Bells] insights into separating the speech signal into different frequency components and rendering those components as visible traces were not successfully implemented until Potter, Kopp, and Green designed the spectrogram and Dreyfus-Graf developed the steno-sonograph in the late 1940s. These devices generated interest in the possibility of automatically recognizing speech because they made the invariant features of speech visible for all to see."© M. Rauterberg, TU/e 108
  109. 109. Speech Recognizer (1952)The first real pattern matcher was developed at AT&T Bell Labs:Davis, K., Biddulph, R. & Balashek, S. (1952).“Automatic recognition of spoken digits”.The Journal of the Acoustic Society of America, 24(6), 637-642.In 1952, as the US government-funded research began to gain momentum, Bell Laboratoriesdeveloped an automatic speech recognition system that successfully identified the digits 0 to 9spoken to it over the telephone. Major developments at MIT followed. In 1959, a systemsuccessfully identified vowel sounds with 93% accuracy. Then seven years later, a system thathad a vocabulary of 50 words was successfully tested. In the early 1970’s, the SUR programyielded its first substantial results. The HARPY system, at Carnegie Mellon University, couldrecognize complete sentences that consisted of a limited of range of grammar structures. But thecomputing power it required was prodigious; it took 50 contemporary computers to process arecognition channel. [source http://www.nexus.carleton.ca/~kekoura/history.html]© M. Rauterberg, TU/e 109
  110. 110. TX-0 (1959) MIT Speech group with TX-0, c1959. Speech, handwriting recognition, neuro data analysis, etc. Interactive editors, debuggers, etc. The TX-0 computer was the worlds first high speed transistorized computer. The TX-0 was the most powerful system of its day (1957), representing a quantum leap in computer technology at the time. © MIT Lincoln Lab© M. Rauterberg, TU/e 110
  111. 111. History of Speech Recognition Technology Spontaneous natural speech 2-way conversation dialogue network transcription Fluent word system driven agent & speech spotting dialogue intelligentSpeaking style messaging digit strings name Read dialing speech form fill office by voice dictation Connected speech directory voice assistance Isolated commands words 2 20 200 2000 20000 Unrestricted Vocabulary size (number of words)© M. Rauterberg, TU/e 111
  112. 112. History of DARPA speech recognition benchmark 100% Switchboard Conversational Speech foreign Read Speech WSJ Broadcast WORD ERROR RATE Spontaneous Varied Speech  20k Speech  Microphone foreign  ATIS NAB 10%  5k 1k  Noisy        Resource Management Courtesy NIST 1999 DARPA 1% HUB-4 Report, Pallett et al. 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003© M. Rauterberg, TU/e 112
  113. 113. Speech controlled application A voice activated dental record system (ORATEL, ORQUEST) (200 different words, relatively simple grammar) 1. Dragon 2. L&H 3. IBM Pure Speech 4. Kurzweil© M. Rauterberg, TU/e 113
  114. 114. International StandardsBased on the path-breaking work of Dzida, Herda and Itzfeld (1978) twenty years ago, theGerman national standard DIN 66234, part 8 was developed and published in 1988.Dzida, W., Herda,S. and Itzfeld, W.D. 1978, User-perceived quality of interactive systems. IEEE Transactions on SoftwareEngineering, SE-4, 270 – 276.Its definitions of usability principles for software user interfaces for office work became thebasis for the international and European standard ISO EN 9241-10 (ISO 1996). This standardserves as the reference for the European Community directive 90/270/EEC for minimum safetyand health requirements to be guaranteed by an employer for his staff working at computerworkstations.1981-1988: DIN 66 234: Grundsätze ergonomischer Dialoggestaltung.1990: European Community Directive 90/270/EEC for minimum safety and health requirements.1995: ISO 9241: Ergonomic requirements for office work with display terminals (VDTs).© M. Rauterberg, TU/e 114