This document describes EnglishMash, a natural language mashup composition environment. It discusses using natural language for mashup development to improve usability for non-programmers. The system translates natural language mashup descriptions and component descriptions into executable code. It features an interactive interface with immediate feedback, auto-completion aids, and highlighting to help users structure programs. The goal is to enable non-programmers to create complex mashups through an intuitive natural language interface.

1. EnglishMash
Usability
design
for
a
natural
mashup
composi8on
environment
Saeed
Aghaee
and
Cesare
Pautasso
Faculty
of
Informa8cs
University
of
Lugano
Switzerland
ComposableWeb
2012,
Berlin,
Germany

2. 2

3. Expressive
Power
Usability
3

4. UI
Mashup
Engine
Usability
Expressive
Power
[A.
Morch,
1997]
4

5. Mashup
Tools
Interac8on
Techniques
• Programming
by
Demonstra8on
• Form-­‐based
Customiza8on
• Visual
Programming
Languages
• What-­‐you-­‐see-­‐is-­‐what-­‐you-­‐get
• Spreadsheets-­‐based
• Example
Modifica8on
• Textual
Domain
Specific
Languages
5

6. 6

7. 7

8. 8

9. EnglishMash
9

10. EnglishMash:
End-­‐users
• Create
feature-­‐rich
components
(requested
by
non
programmers)
• Create
complex
mashups
(requested
by
Programmers
non-­‐programmers)
• Create
mashups
that
can
be
reused
by
non
programmers
• Help
non-­‐programmers
Pro.
non-­‐programmers
• Reuse
exis8ng
mashups
• Modify
exis8ng
mashups
• Create
simple
mashups
Non-­‐programmers
10

11. EnglishMash:
Interac8on
Technique
(1)
UI
Design
or
(2)
Component
development
(3)
Back-­‐end
development
redesign
&
discovery
&
modifica8on
11

12. EnglishMash:
Interac8on
Technique
(1)
UI
Design
or
(2)
Component
development
(3)
Back-­‐end
development
redesign
&
discovery
&
modifica8on
WYSIWYG
+
Form-­‐based
+
?
+
VL
12

13. EnglishMash:
Interac8on
Technique
(1)
UI
Design
or
(2)
Component
development
(3)
Back-­‐end
development
redesign
&
discovery
&
modifica8on
WYSIWYG
+
Form-­‐based
+
Natural
+
Language
VL
Programming
13

14. Natural
Language
is
...
• The
main
Communica8on
means
between
people
• Good
for
storytelling
• Understandable
But…
• Vague
and
ambiguous
• Long
and
wordy
vs.
concise
formal
nota8ons
14

15. Natural
Language
Programming
Structured
Informal
15

16. Natural
Language
Programming
Structured
Informal
16

17. EnglishMash:
Architecture
Scanner
Formal
parser
+
NL
Parser
Seman8c
Extrac8on
(Control
Flow)
Seman8c
Extrac8on
(Data
Flow)
Code
Emifer
(JOpera)
17

18. EnglishMash:
Natural
Language
Mashup
Descrip8on
(NLMD)
“Combine
CNN
News
and
BBC
News.
Show
the
result
on
the
table.”
18

19. EnglishMash:
Natural
Language
Mashup
Descrip8on
(NLMD)
“when
the
map
is
clicked,
do
as
follows.
Search
tweets
around
the
loca=on,
and
combine
the
results
with
BBC
Top
News.
Finally,
show
the
result
on
the
table.”
19

20. EnglishMash:
Natural
Language
Component
Descrip8on
(NLCD)
“when
,
do
as
follows.
,
and
.
Finally .”
20

21. EnglishMash:
Programing
techniques
• Loops:
• Blocks:
• While…
• Do
as
follows
….
• For
each
…
in
…
• Condi8ons:
• If
….
Otherwise
…
• Events:
• When
…,…
21

22. EnglishMash:
Programing
techniques
“
the
map
is
clicked,
do
as
follows.
Search
tweets
around
the
loca=on,
and
combine
the
results
with
BBC
Top
News.
,
show
the
result
on
the
table.”
22

23. EnglishMash:
Programing
techniques
“Search
tweets
for
mashup.
”
23

24. EnglishMash:
Natural
Language
Component
Descrip8on
(NLCD)
“when
,
do
as
follows.
,
and
.
Finally .”
24

25. EnglishMash:
Usability
Design
Use
case
Scenario
Ac8vity
Modeling
UI
Modeling
UI
Design
25

26. EnglishMash:
Use
case
Scenario
Search components
Component selection Call for component development
Stack components
Get immediate feedback
Design mashup user interface
Develop mashup
NLMD Receive writing aids
author
Publish/deploy Develop mashup logic
26

27. EnglishMash:
UI
Ac8vi8es
publish mashups
:publish
search components
design mashup UI edit NLMD stack components
Widget?
Yes
output : EnglishMash generate mashup UI
27

28. EnglishMash:
UI
Model
EnglishMash
Search Stack
keywords results stack
drag characteristics search drop remove
Text field Output
writing aids feedback NLMD publish output
28

29. EnglishMash:
Main
Features
• Immediate
feedback
• Live
output
• Error
• Auto-­‐comple8on
• Data
Flow
sugges8on
• NLCD
comple8on
• NLCD
Sugges8ons
• Highlight
• Data
Flow
• Tex
to
Component
29

30. EnglishMash:
User
Interface
30

31. EnglishMash:
Evalua8on
• Auto-­‐comple8on
helps
them:
• Learn
the
language
• Effec8vely
use
the
language
to
implement
mashups
• Highlights
help
them:
• not
to
get
lost
But..
• It’s
hard
to
choose
among
the
NLCD
sugges8ons
• It’s
hard
to
debug
a
mashup
31

32. EnglishMash:
Conclusion
• Expressive
power
and
usability
trade-­‐off
• NLP
is
understandable
by
end
users
• NLP
problems
can
be
addressed
through:
• Auto-­‐comple8on
• Text-­‐based
visualiza8on
32

33. EnglishMash:
Future
work
• Smart
in-­‐line
component
search
• Smart
component
and
composi8on
sugges8on
• Enable
debugging
• Collabora8ve
mashup
development
• Full-­‐featured
UI
Design
toolbox
• Mul8-­‐UI
mashups
• Mobile
mashup
development
33

34. EnglishMash
Usability
design
for
a
natural
mashup
composi8on
environment
Saeed
Aghaee
and
Cesare
Pautasso
Faculty
of
Informa8cs
University
of
Lugano
Switzerland
Email:
saeed.aghaee@usi.ch
Homepage:
hfp://www.saeed-­‐aghaee.info/
ComposableWeb
2012,
Berlin,
Germany