The document discusses the intellectual exploration of human-computer interaction and interfaces over time. It provides examples of early computing technologies from 1935 to 1999 that pushed boundaries, such as the Differential Analyzer, ENIAC, Memex, Whirlwind/SAGE systems, theories of human-computer symbiosis, and Sketchpad. The examples established foundations for fields like computer-aided design, graphical user interfaces, ubiquitous computing, and tangible interfaces.

1. Interface Matters
Longing the boundary
Intellectual and conceptual exploration
Design the boundary
Human-Computer Interaction and User Interface
Beyond the boundary
Digital Design and Media Technology
Jay Lee
Research Scientist
© 2001 MIT Media Laboratory Tangible Media Group

2. Longing the boundary
border | enclosure | interface | vehicle
Border
Spatial Colonization
Contours, boundaries, and
geographies are called upon
to stand in for the contested
realms of identity, from the
national to ethnic.
Its contours and voids are
occupied by bodies that
replicate internally the
external conditions of
political and social struggles
as well as private and public
Man-made Political Borders | Ingo
Gunter 1997 interventions.

3. Longing the boundary
border | enclosure | interface | vehicle
Enclosure
Territorial Mapping
All intellectual and
experimental reversals
involving in both the
construction and de-
construction, as well as the
inevitable ambiguities
between the reality and its
imagination.
Perhaps space has always
been deconstructed to
generate a massive work of
A Room in the Room | Jay Lee 1999 reconstruction.

4. Longing the boundary
border | enclosure | interface | vehicle
Interface
Constant Interaction
In between the spaces and
enclosures, there are always
transition that is made between
two elements which form
common boundary between two
parts of matter or space.
We live on the boundary between
physical space and cyberspace.
However, these interfaces are
confined to GUI boxes that are
separated from physical space
Tangible Bits | Hiroshi Ishii & TMG 1998 and impoverish human senses.

5. Longing the boundary
border | enclosure | interface | vehicle
Media Room
hybrid media space
By sitting on a chair facing a large
projection wall, embedded sensors all
over the room are used for accessing
electronic content. In this hybrid
architectural space, a room serves as
a computer terminal in which bits
meet atoms and the electronic
Nicholas Negroponte and AMG 1980 information overlaps with everyday
physical world and human life.

6. Designing the boundary
Intellectual Exploration
1935 Mathematical Computing 1972 Personal Computing
1945 Electronic Computing Hybrid Media Space
Nature of Computer 1981 Direct Manipulation
1953 Graphical Communication 1983 Graphical User Interface
1960 Synergetic Coupling 1991 Ubiquitous Computing
1963 Literary Machine 1993 Augmented Reality
Computer Aided Design Graspable User Interface
Computer Networking 1995 Tangible User Interface
1968 Interactive Hypermedia 1999 Pervasive Computing
1970 Head-Mounted Display 2006 Cloud Computing

7. Designing the boundary
Intellectual Exploration
Differential Analyzer
mathematical computing
An analog computer using
mechanical integrators, gears for
constant multiplication, (+/-), was
primarily used for evaluating and
solving differential equations by
mechanical integration. A series of
rotating shafts and wheels, powered
by electric motors, ran problems
through ten integration units to
produce a solution.
[Vannevar Bush and MIT ’35]
Photos courtesy MIT Museum

8. Designing the boundary
Intellectual Exploration
ENIAC
electronic computing
Consisted of 1500 relays and
hundreds of thousands of resistors,
capacitors, and inductors. All of this
electronics were held in 42 panels: 9
feet tall, 2 feet wide, and one foot
thick.
Applications: atomic energy
problems, computing ballistic
trajectories, weather prediction,
astronomical cosmic ray studies.
[Eckert and Mauchly ‘45]

9. Designing the boundary
Intellectual Exploration
MEMEX
nature of computer
Computer as a fundamental tool for
transforming human thought and
creative activity.
As we may think, the form of a desk,
a device for individual, is a sort of
mechanized private file and library:
Information Storage, Retrieval, and
Multimedia.
[Vannevar Bush ’45]

10. Designing the boundary
Intellectual Exploration
Whirlwind | SAGE
graphical communication
An air-defense command and control
system which allows human to
manipulate pictures as well as
numbers and texts on display.
- Internal hierarchic structure of
iconic representations pictures
- Master picture and its instances
(object oriented programming)
- Operations: copy, move, delete
[MIT Lincoln Lab ’53]

11. Designing the boundary
Intellectual Exploration
Man-Machine
Symbiosis
synergetic coupling
Computer as a intellectual partner or
a facilitator of aspects of human
creativity and problem solving.
Human brains and computing
machines will be coupled together
very tightly, and that the resulting
partnership will think as no human
brain has ever thought. [J.C.R. Licklider
’60]

12. Designing the boundary
Intellectual Exploration
SketchPad
computer-aided design
The display, a lightpen, and a bank
of switches were the interface as the
first interactive computer graphics.
Highly precise drawings could be
created, manipulated, duplicated,
and stored. The software provided a
scale of 2000:1, offering many acres
of drawing space. [Ivan Surtherland ’63]

13. Designing the boundary
Intellectual Exploration
Hypertext
literary machines
Computer as the tool that makes
explicit the interdependence of
ideas, drawing out connections
between literature, art, music and
science.
Human brains and computing
machines will be coupled together
very tightly, and that the resulting
partnership will think as no human
brain has
ever thought. [Ted Nelson ’63]

14. Designing the boundary
Intellectual Exploration
ARPANET
networking: resource sharing
Virtually all the world is linked by
packet switched communications
service which become the revolution
of Human Communication:
ARPANET>TELNET > USENET >
INTERNET
[Lawrence Robert ’66]

15. Designing the boundary
Intellectual Exploration
Ultimate Display
head-mounted display (HMD)
A helmet shaped apparatus designed
to immerse the viewer in a visually
simulated 3D toward marrying the
computer to the design, construction,
navigation and habitation of virtual
worlds.
This ineffable potential of computers
to transform the abstract nature of
mathematical constructions into
habitable, expressive worlds.
[Ivan Sutherland ’66]

16. Designing the boundary
Intellectual Exploration
oNLine System (NLS)
interactive hypermedia
Augmenting the human intellect will
exhibit more of what can be called
intelligence by organizing his
intellectual capabilities into higher
levels of synergistic structuring and
physical interface
A collaborative method of sharing
knowledge through interactive linking
is now modern on-line system.
[Douglas Elgelbart ’68]

17. Designing the boundary
Intellectual Exploration
Dynabook
personal computing
User interface design first happened
when we better understand how our
minds worked would completely shift
the paradigm of interaction.
As a dynamic medium for creative
thought through synthesizing static
media with the intimacy and
responsiveness of the personal
computer.
[Xerox Parc and Alan Kay ’72]

18. Designing the boundary
Intellectual Exploration
Media Room
hybrid media space
By sitting on a chair facing a large
projection wall, embedded sensors all
over the room are used for accessing
electronic content. In this hybrid
architectural space, a room serves as
a computer terminal in which bits
meet atoms and the electronic
information overlaps with everyday
physical world and human life.
[Nicholas Negraponte + AMG ’80]

19. Designing the boundary
Intellectual Exploration
Star
direct manipulation (WYSWYG)
8010 Star Information System:
Pioneering for User Interface
Networking: file/print server
Knowledge Workers
[Xerox Parc ’81]

20. Designing the boundary
Intellectual Exploration
Apples
graphical user interface (GUI)
Lisa [Apple ’83]
Together with the use of icons and
windows in the desktop metaphor.
Macintoshi 128K [Apple ’84]
8Mhz CPU, 128K Ram, Graphical
User Interface and Desktop
Publishing a computer intended to
be literally revolutionary.

21. Designing the boundary
Intellectual Exploration
Ubiquitous Computing
transparent interface
The most profound technologies are
those that disappear. They weave
themselves into the fabric
of everyday life until they are
indistinguishable from it.
Computational services are delivered
through a variety of computational
devices such as Tabs, Pads, and
Boards, with the infrastructure to allow
these devices to talk with each other.
[Mark Weiser ‘91]

22. Designing the boundary
Intellectual Exploration
Wearable Computing
augmented reality
With heads-up displays, unobtrusive
input devices, etc. , the wearable
computer can act as an intelligent
assistant. Through the consistency
and mediation, user can interact with
digital information based in the
context of the situation.
[Steve Mann ’81 MIT, VuMan: CMU ’91, Thad
Starner: MIT ‘92]

23. Designing the boundary
Intellectual Exploration
KARMA
computer-augmented env.(CAE)
One of the most powerful uses of
virtual worlds will not be to replace
the real world, but rather to augment
the user's view of the real world with
additional information.
Graphics and text overlaid on the
surrounding world could explain how
to operate, maintain, or repair
equipment.
[Steven Feiner ‘93]

24. Designing the boundary
Intellectual Exploration
digitalDESK
augmented reality
Exploratory Interfaces between
atom, bits, and people; making
physical forms to digital information
and talking advantage of direct
manipulation and physical
affordances in Human-Computer
Interaction.
[Pierre Wellner ‘93]

25. Designing the boundary
Intellectual Exploration
Chameleon
graspable user interface
As input devices that can be held in a
hand and moved on a surface, bricks
can control the computer directly
manipulating the objects.
-Direct control of virtual objects
through physical handles
- Blend physical / virtual afford.
- Distribute cognitive load more
evenly from visual to tactile system
[Fitzmaurice, Ishii, Boxton ‘95]

26. Designing the boundary
Intellectual Exploration
Tangible Bits
tangible user interface (TUI)
Physical Interfaces between
atom, bits, and people;
making physical forms to
digital information and
talking advantage of human
senses and skills in
everyday life.
Physical Direct Manipulation
(WYGIWIG)
[Hiroshi Ishii and Tangible Media Group
at the MIT Media Lab ‘97]

27. Designing the boundary
Intellectual Exploration
Augmentation
paradigm Hypertext | NLS | Ubicomp
principle
WYSIWYG | WYSIWIS | WYGIWIG
idea
hunch ”Demo or Die”
hack
prototype Paolo Alto (GUI) | Collab (CSCW)
Mosaic: Univ. of Illinois
product Lisa / Mac (GUI): Apple
Navigator: Netscape
market
PC, Internet, Web
[Bill Verpank ’98]

28. Designing the boundary
Intellectual Exploration
Physical - Digital Mapping Characteristics
Static Digital Media
Physical States Idea Motivation
Image, a/v files
Digital States Error Observation
Dynamic Digital Media
Physical Live Video, Audio
Representation
Digital Information Metaphor Meaning Model Consistency
Data Structure
Scenario Inventing Task Engineering Combination of data,
attributes, operations
Physical
Representation
Digital Mediation Control Manipulation Digital Attribute
Display Mapping Material Properties
Physical
Representation Information storage, retrieval, and manipulation | Information visualization | Remote Interaction
Simulation Modeling and construction | Systems management,
Digital Control configuration, and control | Education | Interaction in social spaces | People Places, Devices
Remote communication and awareness | Augmentation Entertainment

29. Designing the boundary Beyond GUI:
grasp and manipulate “bits”
Intellectual Exploration in the center of users’ focus
metaDESK inTouch
CHI ‘97, UIST ‘97 CSCW ‘98
Touch
PSyBench Counters
CSCW ‘98 CHI ‘99
PingPongPlus Urp
Illuminating Light Triangles mediaBlocks CHI ‘99,
CHI ‘98, CHI ‘99
CHI ‘98 CHI ‘98 SIGGRAPH ‘98 SIGGRAPH ‘99
SIGGRAPH ‘98
musicBottles
SIGGRAPH ‘99
curlybot
CHI ‘00, HandSCAPE
SIGGRAPH ‘99 CHI ‘00, SIGGRAPH ‘99

30. Beyond the boundary
Interactive System and Interface Design
HandSCAPE
Computer augmented digital measuring tape,
Presented at Emerging Technology at SIGGRAPH
1999, LA Convention Center, Los Angeles and CHI
2000, The Hague, Netherlands
HandSCAPE is a vectorizing digital tape
measure as an input device for digitizing
field measurements, and visualizing the
volume of the resulting vectors with
computer graphics.
Using embedded orientation-sensing
hardware, HandSCAPE captures
relevant vectors on each linear
measurements and transmits this data
wirelessly to a remote computer in real-
time. Combining physical measuring and
computer modeling as a seamless step
enhances the efficiency of on-site
measuring tasks with the addition of
digital functionality.

31. Designing the boundary
Intellectual Exploration
Handheld Unit 9W x 4H x 2D (cm)
Base Unit 8W x 8H x 1D (cm)

33. Designing the boundary
Intellectual Exploration
Music Bottles
Evocative Interface
Seamless extension of physical
affordances and aesthetical metaphors
into the digital domain is a key principle
for the design of this transparent
interfaces.
Upon opening the bottles, users can hear
the sound of music, or narrative stories
with computer control light projections.
Electromagnetic tags enable each bottle
to be wirelessly identified and sensed.
The resonant frequencies of the various
tags embedded in the bottles are
detected and transmitted to the computer
through the use of a custom designed tag
reader board.

34. RF-ID Tag and Tag Reader
board

35. Designing the boundary
Intellectual Exploration
Data LiFT
Hybrid Navigation
In between the two elevators, a
intricately framed elevator is
constructed in existing architecture.
By displaying and manipulating the
building data, viewers can seamlessly
navigate the building virtually before
physically walking around.
[Jay Lee, Brygg Ullmer | Inter-design Workshop 1999]

36. Designing the boundary
Intellectual Exploration
Suspended Windows
Fictitious Space
“Window-within-a-Window” provides
an impetus in which a second window
is built having the same scales and
multi-window configuration. While
viewers are in the space between two
windows, the projected image is
broken apart in squares in the
corresponding area.
[Jay Lee, Bill Keays | Interactive Expression at MIT
Center for Advanced Visual Studies, ACM SIGGRAPH
2000]

37. Designing the boundary
Intellectual Exploration
Extruded Windows
Illusion Extended
“Window-within-a-Window” provides an
impetus in which a second window is
built having the same scales and multi-
window configuration. While viewers are
in the space between two windows, the
projected image is broken apart in
squares in the corresponding area.
[Jay Lee, Bill Keays | Interaction 2001, Fourth Biennale of
The World Forum for Media and Culture, Ogaky City, Gifu
Prefecture, Japan]