The document explores the concept of programmable matter and its potential to revolutionize user interfaces by creating transformable and diverse physical devices that represent virtual entities. It critiques the current state of human-computer interaction (HCI) and suggests that future interfaces should focus on the content and mental models rather than just technique. The Time Interface Device (TIID) is proposed as a conceptual example of such a transformable interface, emphasizing the need for multiple physical representations to enhance user interaction.

1. Separation of Organic User Interfaces: Envisioning the
Diversity of Programmable Matter
Felix Epp
Hochschule Darmstadt – University of Applied Science, Darmstadt, Germany
felix.epp@stud.h-da.de
Abstract. Emerging technologies in nanotechnology, material science and mi-
cro-robotics will make Programmable Matter possible. This creates the vision
of transformable user interfaces as the successor of today’s user interfaces. This
theoretical work discusses the new concepts in creating these interfaces. Instead
of creating a singular device for the various use-cases and contents, future inter-
faces will focus more on representing the underlying content and mental models
than on a certain technique. This will lead to a vast variety of physical devices
each representing one virtual entity and forming the overall user interaction in
combination with each other. The concept of TiID – Time Interface Device –
exemplifies such a device by including all time dependent actions and attributes
in one device.
Keywords: User Interface, Human-Computer-Interaction
1 Introduction
Since 2008 the next term for future human-computer interaction (HCI) evolved out of
the advantage of devices, which offer natural interaction by physical deformable rep-
resentation and input: Organic User Interfaces (OUI). Some researches even think one
step ahead and imagine HCI based on programmable matter, which can change its
shape as well as appearance completely based on underlying computer programs.
Bret Victor states in an article, that future concepts of high technology often focuses
on display technology, which is always just a flat surface. He points out that human
capabilities, especially of our hands, are superior and should not be neglected. [1]
With this in mind this paper critically judges the vision and tries to find applicable
rules.
2 Related Work
In 2008 Holman and Vertegaal formulated the crucial points the present paper con-
cludes, but neglect a holistic view by focusing just on their definition of organic user
interfaces. [2]

2. Ishii et al. thought in advance of OUI and find the requirements of transform, con-
form and inform for transformable HUIs, they call “Radical Atoms”. [3]
Minuto et al. coined a similar vision as “Smart Material Interfaces” (SMI) and
mainly compare it to Tangible User Interfaces without a consideration of other UIs.[4]
3 An Overview in Progress towards Programmable Matter
Programmable Matter, also referred to as Claytronics, is a concept, which evolved
from theoretical computer science and thinks of materials, which can be transformed
into virtually anything.
One emerging technology for programmable matter is the field of Nano-
Technology, which implements control structures and actuators on a nanometer scale.
Even tiny changes of the individual particles can create huge impacts in its collective.
Despite the unimaginable fields of application and immense effect on progress of
technology, Nano-Technology has to overcome challenges like Quantum Computing
[3] in order to be applicable.
Further new possibilities of controlling matter computationally foster constantly in
Material Science. Besides pneumatics or hydraulics, Shape-Memory Alloys for ex-
ample are used to control kinetic energy. The materials can be trained to fit a certain
shape, when an activation temperature is applied on them.[3]
The most applicable and currently promising technology still seems to be Electron-
ics. Proceeding miniaturization lead to the concept of micro-robotics. This includes
challenges like mass behavior and artificial intelligence as well as remote power sup-
ply. But projects like Miche [5] or Robotic Pebbles [6] prove the basic concept of
creating shapes by assembling individual Micro-robots together [5]. In addition the
visual appearance of objects can be altered as well. The “Particle Display System” of
Sato [7] demonstrates how to create, not just deformable displays, but complex ob-
jects, which surfaces perform as displays.
4 Developments in HCI Towards Transformable UIs
By reflecting this vision of technology onto to HCI creates completely new concepts
for techniques and devices. But concepts and findings of current UIs have to be con-
sidered in order to create interaction devices based on Programmable Matter.
4.1 Tangible Interfaces Integrate Malleable Interactions
Tangible User Interfaces are physical objects, which can be manipulated to interact
with virtual entities. In that way the haptic spatial component reduces the level of
abstraction, because it adds a feedback loop. This process of direct manipulation is a
basic advantage of Tangible User Interfaces (TUI), but also of Touchscreens, over
conventional UI e.g. mouse. [8]

3. With Just pneumatics and hydraulics it is possible to create new interface devices,
which are not just physical, but malleable. This is shown by the recent research pro-
ject “Jamming User Interfaces”, where multiple techniques are used to squeeze,
stretch, bend or mold devices as user input. [9] Further there are systems with tangi-
bles moved by magnetism or motorization to give physical output. These studies of
mechanical constrains with actuated tangibles show that closing the malleable feed-
back loop, can enhance user interaction [10].
Still most of these projects use a flat bounded surface to display information, which
leads to a high dependency on the Graphical User Interface.
4.2 Organic User Interfaces
In consideration of this disadvantage researchers of ACM coined the term Organic
User Interfaces for UIs – especially displays – that can be manipulated in their form
by the user and ideally by a computational process. Different papers [10, 11] dealing
with OUIs focus on three fields of research and innovation:
• “Input Equals Output: Where the display is the input device.”
• “Function Equals Form: Where the display can take on any shape.”
• “Form Follows Flow: Where displays can change their shape.” [11]
The Nokia Morph project, as one recent example, was implemented in order to show
benefits in user interaction for deformable displays. It showed that bending gestures
can be intuitive [12] and without visual feedback the accuracy of interaction is short-
ened[13].
4.3 Less Virtual Complexity by Separation in Domains
In his essay Donald A. Norman [14] not only exposes Natural User Interfaces (NUI)
as a “marketing name”, but states that new interaction technologies do not make for-
mer ones obsolete – e.g. a keyboard is still the prior technique, when it comes to text
[8]. He concludes, that it should be necessary to combine UI-Techniques in order to
build sophisticating interactions. Human thinking would require mental models and
therefore a separation into domains of application or appearance.[14]
Youn-kyung Lim states quite well in her article “Disappearing Interfaces”, that “what
users really care about – that is, what they like to interact with. […] [The] content
itself.” [15]
Organic user interfaces are often seen as a way to manipulate a screen or objects
with screens with a focus on haptic interaction methods for virtual contents. In order
to provide more natural and therefore intuitive interactions, terms like display and
user input device have to be overcome. Instead these technological possibilities
should be used to create, what Hiroshi Ishii calls, Tangible Bits or Radical Atoms [3],
which constantly adapt to the users needs, intertwine with each other logically and are
in combination input and output.

4. 5 Time Interface Device (TiID) – a TUI Concept
As an example, TiID is a conceptual, watch-like interface, which displays a time ana-
log and digital (Fig. 1a). It can be twisted stepwise through a set of options, e.g. 12
steps for 24 hours of a day (Fig. 1b). By squeezing it, the size of an option becomes
smaller – in this case to minutes (Fig. 1c). To intimate this to the user the physical
twisting resistance per step and the width of the complete object decreases and the
amount and size of the riffles and steps would change, e.g. to 60 for minutes in one
hour.
Fig. 1. Visualization and interaction schemes of TiID
If we think this one step further this could be the singular device you combine with all
sorts of other devices, but always for the same or similar application of controlling
time dependencies: a wristwatch, reminder or timer functions, selection of a date in a
calendar or a chronological archive or just scrolling in linear media like a movie
stream. Such a device could be realized just by mechanical actuators and silicon mate-
rials, led technology, mini-controllers and wireless communication. Therefore we
create a connection from nowadays technologies to HCI with Programmable Matter.
6 Conclusion
Thinking of Organic User Interfaces as the logical successor of todays UI underesti-
mates the varieties of developments and the capabilities and demands of the user. The
features of transformable interfaces can be achieved, with nowadays technologies and
by downscaling the requirements to the needs and expectations of the user.
Two crucial points can be identified:
• Separation into multiple interface devices/objects for less complexity
• Virtual objects should manifest in physical representations for less abstraction
It is hard to predict future interaction devices based on just assumptions out of differ-
ent areas. Besides, a shift in human behavior could occur from the advancing fields of
brain computer interaction, ambient or anthropomorphic intelligence with natural
speech interfaces and humanoid robotics.[8]

5. Still the possibilities of organic user interfaces in combination with established UIs
should be exploited with the capabilities of current technologies. In the future I try to
build a prototype of TiID and test it, as well as interoperability with multiple devices.
References
1. Bret Victor: A Brief Rant on the Future of Interaction Design,
http://worrydream.com/ABriefRantOnTheFutureOfInteractionDesign/, (2012).
2. David Holman, Roel Vertegaal: Organic user interfaces: Designing computers in any way,
shape or form. Communications of the ACM. 51, 48 (2008).
3. Ishii, H., Lakatos, D., Bonanni, L., Labrune, J.-B.: Radical atoms. interactions. 19, 38
(2012).
4. Minuto, A., Vyas, D., Poelman, W., Nijholt, A.: Smart Material Interfaces-A Vision. Pro-
ceedings of 4th International ICST Conference on Intelligent Technologies for Interactive
Entertainment (2011).
5. Gilpin, K., Kotay, K., Rus, D., Vasilescu, I.: Miche: Modular Shape Formation by Self-
Disassembly. The International Journal of Robotics Research. 27, 345–372 (2008).
6. Gilpin, K., Knaian, A., Rus, D.: Robot pebbles: One centimeter modules for programmable
matter through self-disassembly. Robotics and Automation (ICRA), 2010 IEEE Interna-
tional Conference on. pp. 2485–2492 (2010).
7. Sato, M.: Particle display system: a real world display with physically distributable pixels.
CHI’08 extended abstracts on Human factors in computing systems. pp. 3771–3776
(2008).
8. Shedroff, N., Noessel, C.: Make it so : interaction design lessons from science fiction.
Rosenfeld Media, Brooklyn, N.Y., USA (2012).
9. Follmer, S., Leithinger, D., Olwal, A., Cheng, N., Ishii, H.: Jamming user interfaces: pro-
grammable particle stiffness and sensing for malleable and shape-changing devices. Pro-
ceedings of the 25th annual ACM symposium on User interface software and technology.
pp. 519–528 (2012).
10.Coelho, M., Poupyrev, I., Sadi, S., Vertegaal, R., Berzowska, J., Buechley, L., Maes, P.,
Oxman, N.: Programming reality: from transitive materials to organic user interfaces. Pro-
ceedings of the 27th international conference extended abstracts on Human factors in com-
puting systems. pp. 4759–4762 (2009).
11.Vertegaal, R., Poupyrev, I.: Organic User Interfaces. Communications of the ACM. 51, 26
(2008).
12.Kildal, J., Paasovaara, S., Aaltonen, V.: Kinetic device: designing interactions with a de-
formable mobile interface. Proceedings of the 2012 ACM annual conference extended ab-
stracts on Human Factors in Computing Systems Extended Abstracts. pp. 1871–1876
(2012).
13.Kildal, J., Wilson, G.: Feeling it: the roles of stiffness, deformation range and feedback in
the control of deformable ui. Proceedings of the 14th ACM international conference on
Multimodal interaction. pp. 393–400 (2012).
14.Norman, D.A.: Natural user interfaces are not natural. interactions. 17, 6–10 (2010).
15.Lim, Y.: Disappearing interfaces. interactions. 19, 36–39 (2012).