P2207964_M801_Final

Total number of words 15177
A dissertation submitted in partial fulfilment of the requirements for the Open
University’s Master of Science Degree in Computing for Commerce and Industry
Ian Whitby
P2207964
10th
March 2009

Ian Whitby (P2207964) Page ii
PREFACE
I would like to thank all of those who have contributed to this work and provided the
opportunity to pursue this research.
Special thanks go to my wife, Kate, and children, Rory and Holly, for the long hours
spent away from them and for their unswerving support and encouragement
throughout. Thanks are also due to my employer, National Grid, for granting the
time to undertake this work and to the many employees who completed the
research questionnaire. My thanks also go to ADVFN for allowing the use of their
manually-written share reports within the research prototype and to my project
supervisor, Linda White, for her support.

Ian Whitby (P2207964) Page iii
TABLE OF CONTENTS
PREFACE ..................................................................................................................II
TABLE OF CONTENTS ........................................................................................ III
LIST OF FIGURES................................................................................................... V
LIST OF TABLES..................................................................................................VII
ABSTRACT .......................................................................................................... VIII
CHAPTER 1 INTRODUCTION..........................................................................9
1.1 THE PROBLEM DOMAIN ....................................................................................9
1.2 DEVELOPMENTS IN NLG.................................................................................12
1.3 RESEARCH AIMS .............................................................................................14
1.4 RESEARCH DELIVERABLES..............................................................................14
1.5 CONTRIBUTION TO KNOWLEDGE.....................................................................15
CHAPTER 2 A REVIEW OF THE LITERATURE.........................................16
2.1 INTRODUCTION................................................................................................16
2.2 EXPRESSIVENESS/EFFECTIVENESS OF GRAPHICS.............................................16
2.3 RHETORICAL STRUCTURE THEORY..................................................................18
2.4 THE LINGUISTICS OF PUNCTUATION................................................................19
2.5 MULTIMODAL SYSTEMS ..................................................................................21
2.6 USER CUSTOMISATION ....................................................................................24
2.7 MEANING AND MARKUP .................................................................................26
2.8 SUMMARY .......................................................................................................34
CHAPTER 3 RESEARCH STRATEGIES .......................................................36
3.1 INTRODUCTION................................................................................................36
3.2 REVIEW OF RESEARCH STRATEGIES ................................................................36
3.3 SUMMARY .......................................................................................................43
CHAPTER 4 DATAACQUISITION .................................................................44

Ian Whitby (P2207964) Page iv
4.1 INTRODUCTION................................................................................................44
4.2 DATA SOURCES................................................................................................44
4.3 PROTOTYPE DESIGN ........................................................................................45
4.4 QUESTIONNAIRE..............................................................................................49
4.5 ANALYSIS OF HANDWRITTEN REPORTS ...........................................................53
4.6 VOCABULARY OF A VOLATILE MARKET ..........................................................60
4.7 SUMMARY .......................................................................................................69
CHAPTER 5 DATAANALYSIS.........................................................................71
5.1 INTRODUCTION................................................................................................71
5.2 RESPONDENTS.................................................................................................71
5.3 USER PREFERENCES ........................................................................................72
5.4 USE OF COMMERCIAL DATA ............................................................................89
5.5 SUMMARY .......................................................................................................94
CHAPTER 6 CONCLUSIONS...........................................................................95
6.1 COMPARISON WITH THE RESEARCH AIMS........................................................95
6.2 DIRECTIONS FOR FUTURE WORK.....................................................................96
6.3 PROJECT REVIEW ............................................................................................97
REFERENCES.........................................................................................................98
BIBLIOGRAPHY ..................................................................................................101
INDEX.....................................................................................................................102
APPENDIX A ADVFN REPORTS AND DATA...........................................105
APPENDIX B QUESTIONNAIRE................................................................107
APPENDIX C DISPLAY STYLES................................................................. 112

Ian Whitby (P2207964) Page v
LIST OF FIGURES
FIGURE 1 STRUCTURAL ELEMENTS (POWER ET AL, 2003)................12
FIGURE 2 RST ELABORATION EXAMPLE.................................................19
FIGURE 3 COMET ARCHITECTURE (FEINER & MCKEOWN, 1991)...22
FIGURE 4 WIPARCHITECTURE (WAHLSTER ET AL, 1993)..................23
FIGURE 5 NLG ARCHITECTURE COMPONENTS ....................................27
FIGURE 6 RST CONCESSION EXAMPLE....................................................29
FIGURE 7 CONCESSION TEXT STRUCTURE............................................29
FIGURE 8 ICONOCLAST TEXT STRUCTURE LEVELS...........................30
FIGURE 9 PROTOTYPE - ENTITY RELATIONSHIP DIAGRAM ............46
FIGURE 10 PROTOTYPE – HIGH LEVEL DESIGN......................................47
FIGURE 11 COMPARISON OF NLG AND MANUAL REPORTS.................49
FIGURE 12 PROTOTYPE - WORKFLOW.......................................................49
FIGURE 13 SURVEY – TRADING EXPERIENCE.........................................52
FIGURE 14 SURVEY – DOMAIN EXPERTS ...................................................53
FIGURE 15 ADVFN REPORT - ABSTRACT DOCUMENT STRUCTURE .54
FIGURE 16 RST EXAMPLE FOR LSE HEADLINE.......................................55
FIGURE 17 ADVFN REPORT (13TH
AUG 2008) ..............................................56
FIGURE 18 RST ELABORATION IN HANDWRITTEN TEXTS..................59
FIGURE 19 FTSE100 DAILY VALUES AND 5-DAY TREND LINE ..............60
FIGURE 20 PRE-TRIALADVFN REPORTS - VERBS...................................63
FIGURE 21 PRE-TRIALADVFN REPORTS - ADJECTIVES .......................64
FIGURE 22 TRIALADVFN REPORTS - VERBS ............................................65
FIGURE 23 TRIAL/PRE-TRIAL COMPARISON - VERBS ...........................66
FIGURE 24 TRIALADVFN REPORTS - ADJECTIVES.................................67
FIGURE 25 TRIAL/PRE-TRIAL COMPARISON - ADJECTIVES ...............67

Ian Whitby (P2207964) Page vi
FIGURE 26 TRIAL/PRE-TRIAL COMPARISON - NOUNS...........................69
FIGURE 27 SURVEY RESPONSES - EXPERIENCE......................................71
FIGURE 28 SURVEY RESPONSES - TRADING .............................................72
FIGURE 29 SURVEY - DISPLAY STYLE .........................................................73
FIGURE 30 SURVEY RESPONSES - PREFERRED STYLE..........................73
FIGURE 31 SURVEY - REPORT STRUCTURE ..............................................74
FIGURE 32 SURVEY RESPONSES - REPORT STRUCTURE......................74
FIGURE 33 SURVEY - TEXT BLOCKS............................................................75
FIGURE 34 SURVEY RESPONSES - TEXT BLOCKS....................................76
FIGURE 35 PROPERTIES OF A NORMAL DISTRIBUTION CURVE........78
FIGURE 36 SURVEY - REPORT GROUPING.................................................79
FIGURE 37 SURVEY RESPONSES - TEXT GROUPING ..............................80
FIGURE 38 SURVEY RESPONSES – GROUPING COMPARISON .............81
FIGURE 39 SURVEY - CUSTOMISATION ......................................................82
FIGURE 40 SURVEY RESPONSES - CUSTOMISATION..............................83
FIGURE 41 SURVEY RESPONSES - CUSTOMISATION (HISTOGRAM) .83
FIGURE 42 SURVEY - PAGE NAVIGATION...................................................86
FIGURE 43 SURVEY RESPONSES - PAGE NAVIGATION...........................86
FIGURE 44 SURVEY RESPONSES - COLOUR...............................................88
FIGURE 45 SURVEY - SENTENCE CONSTRUCTION .................................89
FIGURE 46 SURVEY RESPONSES - VOCABULARY COMPARISON .......90
FIGURE 47 SURVEY RESPONSES - GRAMMAR COMPARISON .............91
FIGURE 48 SURVEY RESPONSES - SENTENCE COMPARISON ..............92
FIGURE 49 SURVEY - MISSING DATA............................................................93
FIGURE 50 SURVEY RESPONSES - MISSING DATA ...................................94

Ian Whitby (P2207964) Page vii
LIST OF TABLES
TABLE 1 LSE DATA RECEIVED..........................................................................45
TABLE 2 USERS TARGETED BY SURVEY.......................................................51
TABLE 3 FTSE100 DAILY TRENDS ....................................................................61
TABLE 4 FTSE100 SAMPLE DAYS......................................................................62
TABLE 5 PAGE NAVIGATION STATISTICS .....................................................87

Ian Whitby (P2207964) Page viii
ABSTRACT
This report examines the production of web-based text reports through Natural
Language Generation (NLG) techniques. The work reviews the current body of
NLG knowledge and aims, through the use of an internet-based prototype, to
determine whether commercial data can provide the quality of information
required to automatically produce texts of comparable sophistication to human-
authored reports. The research uses the prototype to further investigate whether
inclusion of user preferences in the generated outputs leads to improvements in
the effectiveness and coherency of the resulting web pages.
The study employs an internet survey to obtain feedback on the quality of
grammar, vocabulary and sentence construction obtained through NLG production
from a commercial data source. The survey also provides quantitative and qualitative
assessment on the effectiveness of web page designs in harnessing individual user
preferences. The returns from the survey were subjected to statistical analysis and its
results extended to infer characteristics for the wider population. This work is
believed to have applicability beyond the realm of the current research and the study
concludes with recommendations for further work within the same field and across
other disciplines.

Ian Whitby (P2207964) Page 9
Chapter 1 Introduction
Commercial organisations are making increasing use of their institutional databases
as sources of on-line information. Such repositories provide users with round-the-
clock access to information across the internet and an ability to interact with the data
at any time. For manually maintained web sites this causes problems with both the
content and design of the site lagging behind the information present within the
organisation’s own database. Rapid changes in the underlying data can take days or
even weeks to be hand-crafted into updated web designs and content. In addition the
presentation of this information is often seen as impersonal and failing to engage its
audience. To counter this organisations are assessing whether NLG could
automatically generate the design and content of their web pages.
1.1 The Problem Domain
NLG uses knowledge of natural language constructs, its grammar and vocabulary to
build grammatically correct sentences and phrases from an underlying source of
nonlinguistic data. The discipline is closely allied to Natural Language Processing
(NLP) which looks to determine the meaning of prewritten sentences. Reiter and
Dale (2000) describe NLG as:
“.. a subfield of artificial intelligence and computational linguistics that is
concerned with building computer software systems that can produce
meaningful texts in English or other human languages from some
underlying nonlinguistic representation of information. NLG systems
use knowledge about language and the application domain to
automatically produce documents, reports, help messages, and other
kinds of texts.“
(Reiter and Dale, 2000, p. xvii)
The application of NLG to the automatic generation of web page designs and content
represents an active research topic, both within academia and industry. The research
community is divided on whether commercial databases are of sufficient quality to

yield NLG texts comparable to those produced by research prototypes. Although
research data sets have been used to produce sophisticated outputs the disparate
nature of commercial data sources, their leniency of data validation and the intrinsic
limitations of recording language components within a relational database structure
have cast doubts over their suitability. In addition researchers are divided on the
extent to which the user’s knowledge, preferences and prior interaction with a web
site affect his/her assimilation and understanding of its content (Matsushita et al,
2003; Mackinlay, 1987; Reiter et al, 2003). Similar arguments have been extended to
the role of graphics within a multimodal presentation (Feiner and McKeown, 1991;
Bateman et al, 2001).
Within the natural language community the suitability of commercial data
repositories as sources of sophisticated natural language content remains an active
area of research. Dale et al’s (1998) work on generating large volumes of NLG text
from commercial sources highlighted the data quality issues involved and a need for
further investigation:
“The problems that arise from noisy data in our database are likely to be
faced by any attempt to use a real database as an information source” .
(Dale et al, 1998, Section 5. Conclusion)
Current research also focuses on the extent to which improvements in the
transference of a text’s message and content to the reader may be achieved through
the inclusion of the user context. Mackinlay’s (1987) work focused on the
expressiveness and effectiveness of graphical designs and highlighted the need for:
“...choosing or adapting the dialogue specifications appropriate to the
observed skill level of the user”.
(Mackinlay, 1987, p. 139)

Matsushita et al (2003) built on Mackinlay’s (1987) work to incorporate this user
context in the visualisation of numerical values whilst Reiter et al (2003) investigated
the research techniques used in acquiring this knowledge of user preference and
interaction with the system.
The current study investigates both areas further and uses a web-based prototype to
create dynamically generated reports from a commercial data source. A daily feed of
price information for stocks listed on the London Stock Market (LSE) is recorded
into the prototype’s relational database without validation or alteration. The database
acts as the information source for subsequent investigation of the suitability of the
commercial data and the contribution of user preferences to the effectiveness of the
generated text.

1.2 Developments in NLG
A NLG milestone in achieving automatic text generation was Mann and Thompson’s
(1986) development of Rhetorical Structure Theory (RST) as a means of formalising
the rhetorical structure of texts. RST allows texts to be expressed as a hierarchy of
elementary propositions and as a diagrammatic representation of ordered nodes, in
which non-terminal nodes within the tree represent a relationship within the text and
terminal nodes represent text phrases. Subsequent authors (Power et al., 2003) built
on this work, advocating refinements in the separation of rhetorical structure from
both the abstract and text structures within a document. Power et al (2003) disputed
Mann and Thompson’s assertion that non-terminal nodes represent a relationship
within the text, arguing that whilst these relationships exist within the document’s
rhetorical structure (i.e. its meaning) they do not necessarily persist through to its
textual realisation. Power et al (2003) argued the case for viewing a document as
three distinct structural elements (Figure 1):
Figure 1 Structural Elements (Power et al, 2003)
The authors further postulated that the abstract document structure could be viewed
as an extension of Nunberg’s (1990) “text-grammar” and its close affinities with text
Rhetorical Structure
 Hierarchy of
Elementary
Propositions
 Rhetorical
Structure
Diagrams
Abstract Document
Structure
 Titles
 Headings
 Captions
 Lists
Text Structure
 Chapters
 Sections
 Paragraphs
 Sentences
 Clauses
 Phrases

markup languages such as HTML and LaTeX. Both HTML and LaTeX are founded
on the belief that the visual appearance of the text assists in conveying its meaning
but can also be represented as a distinct element (e.g. as Cascading Style Sheets).
Research has also been undertaken into the automatic generation of graphics.
Mackinlay (1987) investigated the generation of a range of graphical
presentations (e.g. bar charts, scatter plots and connected graphs) from relational
data held within a database. The research sought to codify graphic designs by their
expressiveness (i.e. the extent to which the graphical language expressed the
underlying message) and their effectiveness (how well this message was received and
understood by its audience).
Dale et al (1998) sought to extend NLG techniques beyond carefully prepared
artificial intelligence knowledge bases to the use of real-world data repositories.
Using the PEBA-II NLG system (Milosavljevic et al, 1996) they generated hypertext
documents from both a knowledge base and an equivalent commercial database.
Their work highlighted weaknesses in both data quality and information structure
within the commercial offering which undermined its ability to generate quality texts.
Bateman et al (2001) argued that neither the function nor the nature of information
layout had been addressed in earlier works. They postulated that a document’s
message came through the overall arrangement of information on the page whilst
relationships between this information were commonly conveyed through layout.
The authors stated that insufficient research had been conducted on the informational
significance of presentation and layout. Bateman et al (2001) advocated an
integrated approach to generating layout, text and graphics under a common
framework, arguing that only by considering these elements collectively was it
possible to attain coherent presentation designs. Their research focused on the

empirical investigation of manually-generated presentations and the development of
a prototype system.
Matsushita et al (2003) extended the early work of Mackinlay (1987). The authors
contended that users were often unable to succinctly state how they wish to query the
underlying data, particularly for large datasets but relied instead on exploratory data
analysis and a series of graphic iterations to build this understanding. Under these
conditions the effectiveness of a graphic is determined not just by tenets of
graphic presentation but also by the user’s previous interaction with the system.
Matsushita et al (2003) argued that decisions on graphic effectiveness must include the
context in which the user poses the query.
1.3 Research Aims
This research builds on the work of others and seeks to establish, through a real-
world example, whether:
 A commercial data source can provide the quality of data required to generate
sophisticated web page content.
 Inclusion of user preferences leads to improvements in the effectiveness
and coherency of web page designs and content.
1.4 Research Deliverables
This research produced an analysis of the content and structure within handwritten
reports used to inform readers of the major stock movements and events on the LSE.
This analysed and documented the vocabulary, grammar and constraint satisfaction
model necessary to generate equivalent texts through NLG techniques.

A prototype application was developed to hold the LSE data, the lexicon of words,
and the constraint satisfaction rules. The prototype allowed users to define their data
and display preferences and subsequently to assess the results against the manually-
written alternatives. The research delivers a quantitative and qualitative appraisal of
whether commercial-sector data can result in sophisticated text content and the
extent to which inclusion of user preferences improves the effectiveness of the
generated pages.
1.5 Contribution to Knowledge
The work of Dale et al (1998) suggested that commercial databases lacked the rigour
of purpose-built knowledge bases, leading to sparse, low quality data and information
structures unsuited to NLG. From both an academic and a commercial viewpoint this
assertion has far-reaching consequences - much of our collective knowledge is held
in such repositories but may be poor sources of NLG content. Through use of a
specific commercial data (London Stock Exchange data) this study looks to challenge
Dale et al’s (1998) assertion. The findings for this specific data source seek to provide
greater insight into the validity of Dale et al’s (1998) assertion.
In addition there is evidence that the effectiveness of graphic representations improves
when user preferences and prior interactions with the system are taken into
consideration (Matsushita et al’s, 2003). This research attempts to discover whether
incorporation of user preferences leads to improvements in page design and
content. No theoretical basis exists for measuring the effectiveness of a text in
conveying its linguistic meaning to a human audience. Psychological variances
within the audience make such measures only meaningful when obtained through
experimental studies. This study, whilst limited in scope, contributes to the wider
understanding of these factors both in research and industry.

Chapter 2 A Review of the Literature
2.1 Introduction
This chapter presents a review of the literature surrounding NLG and published
research on the generation of the paragraphs, sentences and phrases associated with
a natural language. NLG is both an established research discipline and an emerging
technology within the commercial sector. The subject is wide-ranging in scope,
using elements from linguistics, artificial intelligence, cognitive science and
human-computer interaction. This review reflects the breadth of the subject and
draws on a range of topics, from studies of document effectiveness, through text
coherency theory, to the grammar of punctuation and the automatic generation of
multimodal documents (i.e. documents which integrate several document modes,
for example text and graphics). The following synthesises the work of others and
places the subject within the context of what has already been established. From
this basis the current project focuses on its key area of research interest, adopting
principles established by previous authors to challenge current views.
2.2 Expressiveness/Effectiveness of Graphics
Early research into the automatic generation of graphic designs was undertaken by
Mackinlay (1987). This seminal work proposed that the style of graphic designs
(e.g. bar graphs, pie charts) could be chosen automatically from the underlying
data. Mackinlay argued that this was feasible if generation of possible styles was
seen as a problem which could be resolved through composition algebra.
In Mackinlay’s (1987) work an application prototype was built around artificial
intelligence techniques to generate a wide range of potential designs. The prototype
invoked composition algebra, a marriage of composition operators with simple
graphic languages, to generate a wide range of potential graphic styles.

Mackinlay (1987) proposed two measures of graphical success which would
allow the prototype to select the more appropriate of these styles:
 Expressiveness - the extent to which the language can express the
graphic.
 Effectiveness - the extent to which the language can use the
available output media and knowledge of the human
visual system to convey its message.
The syntax of each graphical language was expressed as a collection of tuples
(representing position, height, etc) which expressed the facts for each language.
Once all such tuples were generated the prototype could measure expressiveness by
the degree to which each language satisfied the available facts.
Mackinlay’s (1987) work relied upon on the assumptions that:
 All users share common conventions on how graphics should be constructed
and interpreted.
 Within each graphical language it is possible to encode all facts from the set
within a single sentence.
Although Mackinlay (1987) demonstrated that expressiveness could be determined
empirically, no such measure existed for effectiveness. The author turned to
observations by Cleveland and McGill (1984) that particular properties of a graphic
(e.g. position, length) played a greater role in the effectiveness of users to complete
set tasks. Mackinlay’s (1987) work used these observations to drive the
effectiveness of his system. These observations however provide a far more
subjective view than the empirical measure of expressiveness used within the same
system.
The current research project aims to quantify the sophistication of NLG texts
together with the contribution of user preferences to their effectiveness and

coherency. Mackinlay’s (1987) work suggests these measures cannot be derived
theoretically but will require subjective analysis of the research results.
2.3 Rhetorical Structure Theory
Mann and Thompson (1987, 1988) developed RST as a means of explaining the
coherence of texts. The authors realised that within all coherent texts each element of
the text had some plausible reason for its inclusions and furthermore, there were no
constituent elements omitted. RST provided a framework by which the presence of
these plausible text elements could be analysed and their structures described.
Through the study of numerous texts Mann and Thompson (1987, 1988) were able
to identify key components within coherent texts providing both its coherence
relations (its “nuclearity” and “relations”) and, additionally, any groupings of
key components (i.e. “schemas” applicable to a specific genre).
Mann and Thompson (1987, 1988) identified that for many texts a frequent
structural pattern is for two text spans, often adjoining, related by a condition. One
span (the “nucleus”) acted as the primary structural element, whilst the other (the
“satellite”) played a lesser role in the text. An alternative relationship arose
where no particular span had the primary role – resulting in “multinuclear”
structures.
The authors identified 29 distinct relationships which could exist between
nucleus and satellite in both nuclear and multinuclear texts and devised a
diagramming method by which these structural relationships could be documented.
The following RST diagram (Figure 2) illustrates “Elaboration” relationships
between the nucleus (the leftmost text) and its adjoining satellite.

Figure 2 RST Elaboration Example
RST provides a method of stating text structure and has provided the means for
many NLG systems to structure inputs to their document planning/structuring
phases (Dilley et al, 1992; Reiter and Dale, 2000).
Later authors (e.g. André and Rist, 1995) have argued that graphic elements may be
defined by comparable structure principles to those of text and that text and
graphics can be jointly defined through RST principles. The authors further
outline a method by which the generation process of textual and graphical content
can be coordinated.
The current research will look to generate a restricted target text corpus (Reiter and
Dale, 2000), based on a review of the corpus of input data and expected outputs.
This target text corpus will be expressed through RST and form the basis of
subsequent language generation phases.
2.4 The Linguistics of Punctuation
Nunberg (1990) analysed the linguistics of punctuation from an alternate viewpoint
to his predecessors. He argued that written text should be regarded not as a sub-
category of speech, judged by its ability to convey the spoken language, but as a
Early research into
the automatic
generation of graphic
designs was
undertaken by
Mackinlay (1987).
This seminal work
proposed that the
style of graphic
designs (e.g. bar
graphs, pie charts)
could be chosen
automatically from
the underlying data.
Elaboration

distinct grammar in its own right. Nunberg (1990) contended that punctuation used
within the written language displayed the characteristics of a true grammar worthy
of linguistic analysis.
Nunberg’s (1990) punctuation grammar included not only graphic elements, depicted
as non-alphanumeric characters (e.g. commas, semicolons, colons, periods,
parentheses, quotation marks), but also functional elements (e.g. font- and
face-alterations, capitalisation, indentation, spacing). He distinguished graphical
elements by the function they performed within a text:
 Delimiters of one or both ends of a text sequence
 Separators of two elements of the same type (e.g. separating list elements)
 Typographic distinguishers of an element from its surroundings (e.g.
Italics, font-and face-alterations).
He also categorised the forms by which graphical elements could mark text
elements and boundaries:
 As distinct characters (e.g. commas, semicolons, capitalisation).
 As font-, face- and size-alterations (e.g. italics, bold typeface).
 As “null” elements (e.g. spaces, margins, line breaks) for text separation.
As Nunberg noted:
“These formal and functional properties are not entirely independent of
one another. For example, it is in the nature of distinguishers that they
can be realized only as font-, face-, or size-alterations or by
underlining or analogous devices”.
(Nunberg, 1990, p. 53)
He proposed a set of “linearization rules” to map properties of graphic element
form onto those of function and “pouring rules” to define the layout of text
sequences on the page or screen.

By satisfying a series of grammatical constraints Nunberg was able to assign specific
punctuation attributes (e.g. capitalisation; italicisation) to elements of the lexical
grammar and demonstrate a clear separation between the rhetorical and the textual
elements of a text. These linearization rules, specifically those relating to
typographical distinguishers, show strong similarities with methods subsequently
adopted by contemporary mark-up languages (e.g. HTML, LaTeX). The current
research prototype will build on this separation of the rhetorical from the textual
elements of a text (e.g. by defining the display characteristics of the text
independently from its actual content).
2.5 Multimodal Systems
Multimodal systems use more then one presentation medium (e.g. text; graphics; video;
audio; hypertext) to convey a message. Researchers (Mackinlay, 1987; Feiner and
McKeown, 1991) have been quick to recognise the improvements this could make in a
document’s ability to effectively express its meaning. If a picture is truly “worth a
thousand words” then a document using text, graphics and other media in a
complementary fashion must surely be superior to one based on text alone?
Feiner and McKeown (1991) built on the work of Mackinlay in developing their
Coordinated Multimedia Explanation Testbed (COMET). COMET used artificial
intelligence to create a constraint satisfaction program which matched requests
from its users with the underlying data and knowledge of the user’s prior
interaction with the system. The system would subsequently generate multimedia
presentations automatically.
COMET employed separate text and graphics generators (Figure 3):

Figure 3 COMET Architecture (Feiner & McKeown, 1991)
The “Content Planner” drew on knowledge of its underlying data sources - a static
object source; a rule base to drive text generation; and a geometric knowledge base
for graphic generation. In addition the Content Planner maintained knowledge of
the user context and prior interactions with the system. Output from the Content
Planner consisted of a hierarchy of logical forms which were passed to the Media
Coordinator to determine whether they should be generated as text or graphic
representations.
Feiner and McKeown (1991) decided upon a categorisation of logical forms:
“After conducting a series of informal experiments and a survey of
literature on media effectiveness”.
(Feiner and McKeown, 1991, p.36)
Building on the work of Mackinlay (1987) the Media Coordinator employed
effectiveness criteria to derive six information categories for representation of logical
forms. Physical and locational attributes were represented entirely through graphics,
whilst abstract actions and relationships were generated as text. All other forms (e.g.
physical actions) were generated as a combination of both text and graphics.
Annotated logical forms were forwarded to the Text Generator, the Graphics
Content
Planner
Media
Coordinator
Text
Generator
Graphics
Generator
Media
Layout
Render &
Typeset
Logical
form
Annotated
logical
form
Text
Illustrations

Generator, or both for generation before layout of these elements and final
typesetting. Throughout this process the Text Generator and Graphics Generator
interact, allowing graphics to be placed within the text structure or the text to
reference properties of the graphics.
The Graphics Generator used the annotated logical forms to apply different
attributes (e.g. size, shape, material, colour, lighting) to its pre-defined visual
objects (e.g. pictures) rather than dynamically generate the graphic content. This
differed from Mackinlay’s work where the 2-dimensional graphics were built “on-
the-fly” through measures of expressiveness and effectiveness.
Wahlster et al (1993) developed a similar multimodal presentation system to
COMET. Their WIP system employed a Presentation Planner (analogous to
COMET’s Media Coordinator) and separate text and graphic generators.
Presentation
Strategies
TAG
Graphics
Design
Strategies
RAT
Basic
Ontology
User
Model
...
Knowledge Base
Document
Design Plans
Text
Realization
Text
Design
Graphics
Design
Graphics
Realization
Presentation Planner Layout ManagerPresentation
Goal
Generation
Parameters
Application
Knowledge used
in RAT
Mower
Espresso
Machine
Modem
...
Illustrated
Document
Figure 4 WIP Architecture (Wahlster et al, 1993)
The WIP Presentation Planner employed a set of generation parameters (e.g. user
ability, layout preferences) to gauge the suitability of its candidate designs. These
parameters were comparable to the effectiveness criteria used by both Mackinlay
(1987) and Feiner and McKeown (1991). WIP differs from earlier research work in

its clear separation of text/graphics design from its subsequent realisation. This
separation allows opportunities for customisation of presentation elements.
Time constraints prevented the current research from exploring the potential of
multimodal displays. An early objective had been to examine the contribution such
mixed media types made to document understanding and, in fact, the prototype’s
web browser technology was chosen for its widespread support of such media.
Further research is required to quantifying the contribution made by multimodal
displays.
2.6 User Customisation
Petre (1995) argued for a considered balance between the relative contributions of
text and graphics to the overall understanding of a multimodal presentation. In
studying visual programming techniques on groups of novice and expert users she
observed considerable differences in their inspection strategies for graphics.
Experts demonstrated effective navigation, a strong correlation of inspection
strategy with goals and attention to secondary notation (e.g. layout, logical flow,
colour conventions). Novices showed wide variations in strategy, some sticking
rigidly to inappropriate strategies whilst others changed strategy in an unpredictable,
chaotic manner. Petre (1995) concluded that graphical readership was an acquired
skill – experts were able to take advantage of such secondary notation cues whilst
novices struggled to accurately interpret them. Her assertions support the view
that multimodal presentations require customisation to the needs of their audience
(Matsushita et al, 2003).
Milosavljevic et al (1996) extended the work on Natural Language Generation
with their PEBA-II system to include the generation of dynamic hypertext. This
illustrated how web sites could adopt NLG to dynamically building pages at the

point of invocation. This flexibility allowed PEBA-II to tailor its output to the
user’s preferences and their previous discourse with the site. These ideas will be
examined further in the current research.
In 1997 Calvi and De Bra described adaptive information browsing as a means of
filtering both the navigation links and, to a lesser extent, the content presented to
students at the Eindhoven University of Technology. The authors cited evidence
that comprehension of presentation material relied upon the recipient building a
conceptual model of the information and the semantic relations implicit within it
(Van Dijk and Kintsch, 1983). Calvi and De Bra (1997) argued that the network
navigational links presented in web-based documents placed a high cognitive
overhead on the reader, leading to a reduction in their comprehension of the
material itself. The authors developed an adaptive hypermedia system which
presented users with a subset of the available navigational links (and hence
content). The constituents of this subset were determined on a per-user basis by
modelling each user’s previous navigation through the system. Unlike
Milosavljevic et al’s (1996) PEBA-II system, this implementation did not
demonstrate NLG in the true sense, restricting itself to toggling the visibility of
fixed-format content within the outputs; however it did actively model the user
context.
Initial feedback on Calvi and De Bra’s (1997) system from students suggested
concerns with this approach to information hiding:
“Users seem nevertheless to complain about the impossibility of the
present formalism to provide them with a snapshot of the system’s
complete structure”.
(Calvi and De Bra, 1997, p.272)

Such comments are relevant to the current research. Calvi and De Bra’s (1997)
study demonstrates that although it is possible to modify the design, content and
navigation paths to account for a user’s previous interaction with the system this
may prove counter-productive. Initial evidence suggests that users have in mind a
conceptual model of the system and that altering the waypoints and pathways
through this model reduces their acceptance of and proficiency with the actual
system. The arena of proficiency-adapted information browsing (Calvi and De Bra,
1997) constitutes a significant area of research, majoring on the psychological
aspects of human learning and understanding. The current research does not attempt
to explore these topics, stopping short of modelling the user context within the
research prototype.
2.7 Meaning and Markup
Bateman et al (1998; 2001) presented an architecture which unified the data-
aggregating methods of information visualisation and the communicative-goal
techniques of NLG. Their KOMET-PAVE experimental prototype drew on Formal
Concept Analysis (Wille, 1982) and the construction of dependency lattices to
design both text and graphics. The system generated all potential texts and
diagrams from a single pass through these dependency lattices and, through the
extensive use of heuristics, arrived at its final choice of multimodal presentation.
Reiter and Dale (2000) in their milestone textbook “Building Natural Language
Generation Systems” defined standard architectural components in the creation of
NLG systems. These are summarised in Figure 5 below.

Content
Determination
The determination of which human-authored texts
(the “corpus text”) will be targeted as NLG output
texts (the “target text”).
↓
Document
Structuring
The assignment of the output text into message
collections or groups and the relationships that
applies to these groups (i.e. the “rhetorical
structure” of the target text).
↓
Lexicalisation
The identification of the words or dictionary of
words (the “lexicon”) which will be applied to the
target texts.
↓
Referring
Expression
Generation
Determination of the expressions that will be used
to refer to entities in the target texts.
↓
Aggregation
The conceptual organisation of these rhetorical
structures into linguistic structures (e.g. sentences,
paragraphs)
↓
Linguistic
Realisation
The physical organisation of these message groups
into linguistic structures.
↓
Surface
Realisation
Mapping abstract structures (e.g. paragraphs) onto
the symbols necessary to display them in a
document presentation medium.
Figure 5 NLG Architecture Components
The authors’ work shows a clear distinction between the logical construction of text
and its physical presentation and provides pointers for the current project. Reiter and
Dale’s (2000) use of markup languages (e.g. HTML and LaTeX) to provide surface
realisation are employed in the research prototype as a mechanism for customising
individual user displays.

Power (2000) described a method by which the rhetorical structure for a text
could be realised through a text structure of sections, paragraphs and sentences
linked by discourse connectives (e.g. “since”, “however”, “whilst”) to mark the
rhetorical relations. He contended that text structuring could be formulated as a
constraint satisfaction problem whereby all potential texts were generated by
satisfying the text structure constraints and the most suitable of these texts could be
identified through the application of additional constraints. Power (2000) argued
that whilst research efforts have largely focused on building rhetorical structures to
organise elementary propositions into hierarchies less attention has been given to
realising these rhetorical structures as text structures. He argues that such text
structures are of sufficient complexity to require consideration early in the
construction of the texts rather than on a per sentence basis.
Power illustrated how the ICONOCLAST text structuring system was able to
generate all possible candidate text structures from the rhetorical structure. Each
candidate text structure represented an ordered tree in which the non-terminal
nodes are labelled as text categories, and the terminal nodes held either discourse
connectives or propositions (i.e. the content of the assertion).
For example the following output text might be generated through NLG:
“Energy stocks closed higher due to rising prices; however, the
FTSE100 falls to record low.”
Mann and Thompson (1986, 1987) categorised the style of relational proposition
which links these phrases/discourse connectives as a “concession”, whose rhetorical
structure is represented in Figure 6 below.

concession
NUCLEUS SATELLITE
falls(FTSE100, record low) cause
NUCLEUS SATELLITE
closed(energy stocks, higher) rising(energy stocks, prices)
Figure 6 RST Concession Example
Power (2000) contended that each rhetorical structure would be applicable to one or
more text structures, for example the following could produce by the RST structure
above:
sentence
text_clause text_clause
text_phrase text_phrase text_phrase text_phrase
closed(energy stocks, higher) “however” falls(FTSE100,
record low)
text_phrase text_phrase
“due to” rising(energy stocks, prices)
Figure 7 Concession Text Structure
The ICONOCLAST system, assigned both TEXT-LEVEL attributes and
INDENTATION levels to each node within each text structure. An arbitrary
number of TEXT-LEVELs are possible, with five chosen by the designers:

TEXT_LEVEL Aggregation Level
0 text-phrase
1 text-clause
2 text-sentence
3 paragraph
4 section
Figure 8 ICONOCLAST Text Structure Levels
Nodes are also aligned by INDENTATION level, ranging from zero (no
indentation) to the chosen maximum. Each candidate text must adhere to the rules
of the text structure, namely that each TEXT-LEVEL consists of one or more nodes
of a lower TEXT-LEVEL value (e.g. sections consist of paragraphs and paragraphs
consist of text-sentences). It is argued that all candidate solutions could be
generated by the addition of two further rhetorical structure node attributes –
ORDER (the linear position of the texts relative to its peers) and CONNECTIVE
(e.g. ‘since’; ‘consequently’).
Candidate solutions are obtained through a number of steps:
1. Add TEXT-LEVEL and ORDER attributes to each of the rhetorical
structure nodes.
2. Assign domains to both the TEXT-LEVEL and ORDER attributes (i.e. the
possible ranges of these attributes for each node)
3. Apply constraints (e.g. root node should have a higher TEXT-
LEVEL than its child nodes).
4. Compute all possible combinations.
5. Compute complete text structures which satisfy the text structure
formation rules. This includes adding discourse connectives to either the
nucleus or the satellite.
6. Validate all text structures (e.g. ensure each child node has a parent at the
TEXT-LEVEL directly above it).
Power also details the practical difficulties in completing all possible combinations
and the necessity to pre-filter texts before developing their rhetorical and text

structures. The current work will look at this and the subsequent works by
Bateman and Delin (2001) and Power et al (2003) in isolating the structural and
linguistic components of a document.
Bateman and Delin (2001) and Bateman et al (2002a; 2002b) investigated the genre
of multimodal documents. These authors contend that documents satisfy
communicative goals on five levels:
Content Structure - The structure of the information that is to be
communicated.
Rhetorical Structure - Rhetorical relationships which exist between
content elements (How the content is ‘argued’).
Layout Structure - The nature, appearance and position of document
elements.
Navigation Structure - Structures used to support the communication of
the document’s message (e.g. Titles, colour,
grouping).
Linguistic Structure - The structure of language used to realise the
document.
The authors believe the manner in which a document harmonises these structures
imparts to it a certain genre. This genre is further enhanced by its satisfaction of a
series of constraints:
Canvas Constraints - Constraints arising from the physical nature of the
document produced.
Production Constraints - Constraints arising from its production
technology.
Consumption Constraints - Constraints arising from the time, place and
manner in which the document’s information is
imparted.
Bateman and Delin (2001) believed all document designs could be critiqued using
these techniques to yield predictions of its usability, an approach supported by
Power et al (2003) who argued that document structure should be defined as
distinct elements. Power et al (2003) envisage three such elements: Rhetorical

Structure, Text Structure and Abstract Document Structure (Figure 1). Their work
builds on Power’s earlier research (2000) in which he identified both the rhetorical
structure and the text structure of a document (Figure 7 & Figure 8). The authors
argue that the text structure, through its arrangement of text and use of font
variations, also imparts a significant graphical component to the text which directly
contributes to its meaning. Power et al (2003) argue this third element, termed the
“Abstract Document Structure”, must be considered in isolation from the text
structure itself. The authors believe that to view the Abstract Document Structure as
a component of text structure limits choices of both layout and wording. Using
examples they illustrate how the deferral of abstract document structure (e.g. text
layout and font characteristics) until after the formalisation of the text structure
creates restrictions on the text and layout possibilities, with text layout decisions
taken at a sentence or paragraph level rather than on a document-wide level.
In part Power et al (2003) build their arguments on the earlier work of Nunberg
(1990) who identified the need to separate the mapping of syntactic structures onto
the lexical elements of a text grammar from its functional elements (e.g. font- and
face-alterations), differentiating between concrete features of text structure and the
abstract (Chapter 2.4). Power et al (2003) expanded upon the author’s earlier
research (Power, 2000) to illustrate how rhetorical structure could be transposed
into an ordered text structure tree and highlighted the problems of doing so (e.g.
with ELABORATION). The paper detailed their work on the ICONOCLAST
system and the generation of multiple texts/text structures from the same rhetorical
structure. ICONOCLAST used a five stage process to generate natural language:

Planning Module – Organises the document into a single rhetorical
structure, in which each non-terminal node was
represented by a rhetorical relationship and each
terminal node by a simple proposition.
Preliminary Module - Selects simple propositions from a knowledge
base and organises these into arguments.
Document Structurer - Distributes these arguments into sections,
paragraphs and text-sentences.
Syntax Realiser - Formulates the wording of propositions.
Formatter - Applies the abstract document structure (e.g.
graphical elements; font variations; tabbing).
This step-wise process is very relevant to the NLG tasks necessary for the current
research. It illustrates how the project might utilise data fed into a data repository to
produce text content for publication within a web site. ICONOCLAST used
constraint-resolution methods to formalise the potential valid options for solving
the rhetorical structure. Again this method has relevance to the current research and
the use of constraint-satisfaction programs, such as SICStus Prolog, to achieve this.
Power et al (2003) determined that through the use of further classifications they
would be able to generate all possible document permutations. The authors were
unable to provide a means by which the most suitable permutation might be
selected, using intuition for their simple examples. This is the same issue
encountered by Mackinlay (1987) and others in their search for the “effectiveness”
of texts. The current research will seek responses from a survey to provide an
independent measure of effectiveness for the generated outputs.
Matsushita et al (2003) extended the early work of Mackinlay (1987). The authors
considered two factors in chart selection – the type of chart display and the type of
user utterance. They reasoned that frequently users are unable to succinctly state
how to query the underlying data, particularly for large datasets, but rely instead on
exploratory data analysis. Through a series of intermediate query steps and

associated graphic representations of the results users are able to iterate towards a
final solution. Each successive query builds upon knowledge learnt from the
preceding queries. Matsushita et al (2003) argue that the resulting graphic presented
to the user must consider this user interaction in its chart generation. The authors
argue that decisions on graphic effectiveness must include the context in which the
user poses the query.
2.8 Summary
NLG has been an active area of research since the 1980s. Mann and Thompson’s
(1987, 1988) work on RST provided a framework by which discourse could be
analysed and described. Mackinlay’s (1987) work proposed a method by which
graphic designs could be selected on the basis of composition algebra. Subsequent
authors have elaborated and expanded upon these early endeavours.
The current research seeks to determine whether a commercial data source can
provide sophisticated page content and whether building user preferences into the
resultant pages improves their effectiveness. This work builds on the earlier work
of others. Mackinlay (1987) postulated that effectiveness could not be derived
theoretically whilst no theorem of human perceptual capabilities existed. Twenty
years on this remains the case and the current research adopts a survey strategy to
gauge the views of respondents.
Mann and Thompson’s (1987, 1988) work on rhetorical structure theory will be
used within the current research. Through the methods proposed by Reiter and Dale
(2000) the lexicon, style and rhetorical structure adopted in manually-authored texts
will be documented to form the basis of the target text corpus generated by the
prototype through NLG.

Petre’s studies (1995) on multimodal presentations concluded that graphical
readership was an acquired skill, with experts able to take advantage of such
secondary notation cues whilst novices struggled to accurately interpret them.
Her assertions support the view that multimodal presentations require customisation
to the needs of their audience (Matsushita et al, 2003). The current research
investigates the customisation of page presentation and, through detailed analysis of
survey responses, identifies those elements regarded as most effective by users.
Bateman and Delin (2001) and Bateman et al (2002a; 2002b) argued for the
division of text into five structural levels whilst Power et al (2003) envisaged a
separation of the abstract document structure from its text or rhetorical structure.
These divisions have affinities with current mark-up languages (e.g. HTML;
LaTeX), founded on the principle of clear separation between document
presentation and discourse structure. The current research uses web-based browser
technology as its presentation medium, employing HTML and CSS (Cascading
Style Sheets) to isolate the visual components from the underlying structure.

Chapter 3 Research Strategies
3.1 Introduction
Before moving on to the primary research it is worth reflecting on the strategies
and methods available to the study. Whilst the review of the literature placed the
current work in context it yielded little on the techniques necessary to undertake
this endeavour. This chapter reviews these strategies and assesses their
applicability.
3.2 Review of Research Strategies
Research activity is, by its very nature, a wide-ranging and varied discipline.
Methods vary with the field of study, the intended purpose, the approach adopted and
the nature of the work (Sharp et al, 2002). Despite this each study must be
repeatable, verifiable and contribute to the sum of knowledge. These demands create
a need for general principles which can be employed, singularly or collectively, to
research. Denscombe (2007) reviewed the major strategies available for small-scale
social research projects and proposed a grouping of eight categories:
1. SURVEYS
2. CASE STUDIES
3. EXPERIMENTS
4. ETHNOGRAPHY
5. PHENOMENOLOGY
6. GROUNDED THEORY
7. MIXED METHODS
8. ACTION RESEARCH
Robson’s (1983) review added a further category of “Psychological Experiments” to
this list.

3.2.1 Survey
Surveys provide established techniques by which hypotheses can be tested against
the actual views or actions of respondents and are widely used in social research
projects. Denscombe (2007) summarised these techniques as: postal questionnaires;
internet surveys/questionnaires; face-to-face interviews; telephone interviews;
documents; observations.
Surveys give wide coverage of the problem domain and are often selected to provide
a broad range of views and responses on the subject area. Inevitably time or
resources limitations mean surveys will contain a subset of the total population (a
“sample”). Rowntree (2000) emphasised the difference between surveys which
describe or summarise their sample and those which use it to make inferences on the
wider population. In the former observations have applicability only to the sample
itself whilst in the latter there is an assumption that the sample is representative of the
total population and that inferences can be drawn from the sample for every instance
of the population. A sample strategy which looks to make inferences on the total
population must address the issues of random sampling and skewed distributions
within its sample.
Surveys focus on a point in time, a snapshot, from which a random sample of views
or actions is observed or inferred. Further snapshots can be taken subsequently but
the random nature of sampling means these samples are unlikely to contain the same
respondents. This effect restricts the ability of surveys to analyse or describe
variations over time except at a macro level. Denscombe (2007) also noted their
empirical nature, targeting the measurement and recording of tangible attributes.
Such characteristic would prove restrictive in a study of human relationships but are

well suited to the current project where changes of independent variables cause
quantifiable changes in the associated dependant variables.
The survey approach is applicable to the current research which focuses on a point in
time and seeks to quantify the influence of independent variables on the NLG
outputs. The study looks to infer general characteristics from its sample, both for the
effectiveness/sophistication of share reports and for web-based NLG documents in
general. Implicit in this is the need to target a representative sample, whilst
acknowledging that domain experts will be geographically dispersed and represent a
small proportion of the total UK population. The study’s focus on web-based reports
suggests an internet survey/questionnaire to be suitable to its purposes.
3.2.2 Case Studies
In many ways a case study represents the antithesis of the survey, focusing on the
specific and studying few instances in detail rather than a higher level examination of
the broader population. Case studies tend to focus on relationships and processes,
making them particularly appropriate to studies of human sociology and psychology.
A case study will often involve monitoring the phenomenon in its normal setting and
allowing the researcher to observe interactions between variables. Multiple methods
are commonly employed to analyse the phenomenon and results are more qualitative
than those of a survey, relying on description rather than statistics to characterise the
phenomenon under study. As with surveys a case study may look to infer the
applicability of its results beyond the sample.
A case study approach is applicable to this research and was initially selected as a
means of obtaining feedback. The approach requires a limited number of domain
experts to make significant time available to the study (perhaps three sessions of one

hour each). However early enquiries to share clubs within a 30 mile radius revealed
that only a handful of clubs existed and of those only one met regularly and was
willing to participate. Further investigation showed that this would not yield
sufficient domain experts, with meetings scheduled only monthly and very low
attendance figures (usually two or three and occasionally fewer members). Assuming
not all members used web-based share reports this represented an unrewarding line
of analysis. The research adopted an internet-based survey/questionnaire approach to
gain a large sample population, although this brought attendant issues on how
representative the sample was to the overall population (Chapter 4).
3.2.3 Experiments
Experiments are characterised by the control they exhibit over their environment and
their identification of causal factors. They aim to isolate the study from external
influences and manipulate input variables to understand their influence on the
resulting outputs. Experiments use empirical measurements and observations to draw
conclusions from the results of the study. These conclusions have applicability
beyond the experiment itself and, through careful isolation of competing factors, the
experiment infers the causes of phenomena observed in the real world.
The current study investigates a complex research area. In measuring and appraising
the “effectiveness” of user customisation we need to understand what domain experts
understand by “effective”. Furthermore, deciding if a “commercial data source can
provide the quality of data required to generate sophisticated web page content”
(Chapter 1.3), requires us to test the users’ understanding of “sophisticated”.
The experimental approach provides the opportunity to achieve this. It isolates
external influences, keeping the underlying data consistent across the experiments

and removing the intangible effects of market sentiment and psychological pressures
associated with share trading. By providing a prototype environment in which casual
factors are controlled it is able to focus on the effects the remaining factors have on
the results. The internet-based survey/questionnaire forms an integral part of the
experiment (Appendix B) and probes respondents on specific cause-and-effect
relationships (e.g. “How significant was colour in identifying text groups?”; “How
similar was the grammar to that used in the ADVFN reports?”).
3.2.4 Ethnography
Ethnographic studies analyse people and groups through their lifestyles,
understandings and beliefs. This strategy has its origins in anthropological research
and the study of alien cultures. Ethnography is a detailed examination of human
society and social behaviour, often conducted from an insider’s viewpoint. This
approach offers little to the current research.
3.2.5 Phenomenology
Phenomenology is a strategy which deals with human perceptions, studying their
feelings, emotions, beliefs and attitudes. It analyses phenomena through a descriptive
approach aimed at interpreting its findings rather than quantifying them. The
approach is characterised by seeking to describe and explain phenomena without
abstracting, classifying or quantifying them.
The current research examines the coherency of web designs and the ability of
commercial data to provide sophisticated NLG content. Although elements of this
assessment are subjective (e.g. interpretation of designs as they appear to others)
phenomenology’s focus on feelings and emotions does not offer anything to the
overall aims of this research.

3.2.6 Grounded Theory
Grounded Theory is an approach aimed at the generation of new theories rather than
the verification of existing theories. The approach focuses on observations of the real
world to build theories around these observations on the basis of empirical research.
This strategy offers little to the current research which looks to challenge Dale et al’s
(1998) theory that commercial data is too noisy to generate sophisticated NLG
content and Matsushita et al’s (2003) assertion that user preferences lead to
improvements in web-page effectiveness.
3.2.7 Mixed Methods
A mixed method strategy emphasises the benefits of an approach which combines
quantitative and qualitative analysis to improve research results. A central
characteristic of the approach is that the study is not seen as an “either/or” decision
between empirical and subjective analysis but a consistent application of both. Mixed
method advocates select research techniques on their ability to clarify the problem
domain, not on the basis of their categorisation as belonging to a particular approach.
The current study is not specifically designed as a mixed method approach, although
the incorporating of a survey within an experiment shows its applicability to the
work. Furthermore the open questions within the survey require an alternative
approach to the empirical path taken for the multiple choice closed questions.
3.2.8 Action Research
Action research is a practical strategy aimed at real world problems. Both researcher
and respondents are influence by the study and often become active collaborators in
achieving its goals. The strategy is geared towards engineering change in real world

situations and is cyclic in approach, each change feeding back into a cycle of
renewed actions.
The current research does not seek to establish real world change. It is expected that
users might freely donate a little of their time to the study but unrealistic to expect
respondents to become active collaborators in addressing the issue of sub-optimal
web designs.
3.2.9 Psychological Experiments
Robson (1983) reviewed strategies for designing psychological experiments and
identified the need to differentiate between variables manipulated by the researcher
(“independent variables”) and those variables observed by the researcher to
determine the effect of this manipulation (“dependant” variables). Independent
variables usually represent study inputs whilst dependant variables are its outputs.
Within the current research user preferences will act as independent variables whilst
the effectiveness of the generated pages is the dependant variable. In this sense the
research shows the characteristics of an experimental strategy.
Robson (1983) also refers to independent variables whose value is not derived from a
numeric scale but through a subjective categorisation (e.g. “big”, “small”, “often”).
Such variables do not lend themselves to rigorous quantitative analysis but to more
qualitative analysis. The open questions of the research questionnaire are of this
nature. The questionnaire ensures respondents have an opportunity to expand on their
responses by supplementing the closed questions with a number of open ones.
Analysis of these responses shows the characteristics of a mixed method strategy.

3.3 Summary
The study looks at current practices within NLG and examines the use of constraint
satisfaction methods. An experimental approach is adopted in which the inputs to a
prototype system are carefully controlled and the factors influencing the effectiveness
and sophistication of the resulting reports carefully assessed. Existing work in this field
(Wahlster et al, 1993; Reiter and Dale, 2000; Power et al, 2003) is used as the basis for
the research prototype.
The prototype separates the report’s presentational elements from its rhetorical and
text structure through the use of cascading style sheets. Users are encouraged to
exploit this separation in customising the presentational aspects of the report. The
research also adopts a survey approach in eliciting the views of domain experts
through an internet-based survey/questionnaire (Appendix B). The prototype
dynamically generates a range of web page designs which, to varying degrees,
incorporate aspects of user preference.

Chapter 4 Data Acquisition
4.1 Introduction
This chapter extends the research strategies previously outlined to document specific
methods used for the current research. Acquisition techniques were applied both in
determining the necessary inputs for the prototype and gathering user feedback on its
outputs. Inevitably much of the input data for the handwritten reports came from
sources unavailable to the study (e.g. inflation rates, market sentiment, foreign
exchanges). Which text could be reproduced by the prototype was determined
through analysis of the rhetorical and textual structure of the handwritten reports
together with the creation of a dictionary of terms used by human authors. An
overview of the research prototype is described along with the commercial data used
and the process by which the range of output texts was determined.
4.2 Data Sources
This research study focused on the structure and content of web-based reports aimed
at informing the reader of daily share price movements for companies listed on the
London Stock Exchange. Reports of this nature already exist and an internet search
yields a wealth of information, ranging from the LSE itself
(www.londonstockexchange.com) to internet-based news organisations (e.g.
http://uk.reuters.com/business/markets; http://money.uk.msn.com) and the on-line
presence of traditional broadsheets
(http://business.timesonline.co.uk/tol/business/markets/; http://www.ft.com/markets).
This project does not attempt to study each of these sources, recognising that the
differences in their reporting styles would mask any variations caused by adjustments
to the study’s independent variables. Attempts to assess the effectiveness of user
customisation or the sophistication of NLG text are unlikely to prove fruitful when

the benchmark against which they are compared represent a collection of varying
styles. The current research opted for a single information source from internet
company ADVFN (http://www.advfn.com/) to provide both daily versions of their
own share reports and access to the underlying LSE data. Although the ADVFN
reports and data are publicly available upon payment of a subscription fee their
content is clearly aimed at trading experts with their web site geared towards
monitoring, charting and reporting stock market movements for trading purposes.
The study subscribed to ADVFN’s daily market data service (“Level 1” service), to
receive a trading summary each evening for all shares listed on the LSE (Table 1).
Variable Description
TIDM Share identifier (Tradable Instrument Display Mnemonic)
Opening Opening price at start of trading day
High Highest price achieved during the day
Low Lowest price achieved during the day
Closing Closing price at end of trading day
Volume Volume traded during the day
Table 1 LSE Data Received
The prototype provided access to LSE data and ADVFN reports across a three month
trial period (Sep to Nov 2008) to simulate a wide variety of actual market conditions.
Examples of an ADVFN share report and the Level 1 data are provided in Appendix
A. It should be noted that whilst not required for the aims of this study the prototype
could also have received market data in real-time (”Level 2” service), allowing share
reports to change dynamically throughout the trading day.
4.3 Prototype Design
The study employed a variety of technologies to produce a prototype capable of
addressing the aims of the research. The Level 1 data was manually downloaded
from ADVFN each evening as comma-separated files and uploaded into a MySQL

database (Figure 9). The ADVFN data values were unchanged by this process, with
no validation, error correction or data manipulation performed during committal to
the Daily_Values table. Subsequent use of this data was also read-only to ensure all
experiments were repeatable, albeit with minor variations of the actual texts
produced due to the dynamic nature of NLG. FTSE100 index values were committed
to the Index_Values table by a similar process whilst the remaining tables were
populated through a one-off data entry exercise.
Figure 9 Prototype - Entity Relationship Diagram
The database also illustrates future design considerations. Whilst the Indexes table
contained only a single entry for the FTSE100 it could easily be extended to other
market indexes. Similarly the views (labelled “VW_”) allowed market trends to be
analysed over time, a feature not implemented due to time constraints.

The general purpose scripting language PHP provided the application “glue” between
the database, the SICStus Prolog constraint satisfaction program and the HTML web
pages (Figure 10). From the HTML interface individual display/data preferences
were recorded as transient session variables, lost on termination of the session to
ensure respondent anonymity. Share reports were invoked from the “Reports” page
(Appendix C) by simply choosing the required date, with PHP querying the database
for all share values and FTSE100 movements relating to this date.
Figure 10 Prototype – High Level Design
The script interrogated the direction of movement for the FTSE100 on the chosen
day and, based on whether the index was rising, falling or static, altered the lexicon
of terms employed by the SICStus Prolog constraint satisfaction program to build its
outline sentences. These sentence templates were subsequently merged with the data
values queried previously and the most significant movements built into complete
sentences. The PHP queried the data and display preferences of the user adjusting the
content and presentation layout of the NLG report before presenting it at the browser.
Report Date
ADVFN
Share
Data Share
Data
M801 Website GenerationM801 Website Generation
Prolog
Preferences
Report Date
Sentence
Generation
HTML
HTML
Sentence
“Templates”
Display
Preferences
Report
Sentences
Share
Report
Share
Preferences
Database

The prototype encouraged respondents to experiment with the site
(www.ianwhitby.co.uk), revise their preferences and assess the effect on the
generated reports. Each NLG web page included a hyperlink to an equivalent
handwritten report, displayed in a separate browser window, for comparison with the
prototype’s output (Figure 11).

Figure 11 Comparison of NLG and Manual Reports
The prototype design shows simple navigation between pages, engendering a sense
of workflow (Figure 12). The principal pages in this workflow were accessed through
a tabbed menu structure with the feedback questionnaire marking the culmination of
a user’s interaction with the system.
Figure 12 Prototype - Workflow
4.4 Questionnaire
The research questionnaire (Appendix B) illustrated a number of characteristics:
1. The number of questions was limited to twenty three to encourage
respondents to fully complete the questionnaire. As Denscombe (2007)
cautioned:
“…there is, perhaps, no more effective deterrent to answering a
questionnaire than its sheer size”.
2. Formats were standardised across the questionnaire to ease comprehension.
3. Respondents were not asked to login or provide any information by which
they might be identified. The author believes the response rate would have
been considerably lower had respondents been required to identify
themselves.
Home
Page
Survey/
Questionnaire
NLG
Reports
User
Preferences

4. Most questions were multiple choice and closed in style (i.e. answers were
restricted to the options provided). This format was chosen to allow
respondents to fully complete the questionnaire with minimal effort.
5. Multiple choice answers were presented as a 5-star rating system
accompanied by supporting descriptions to aid comprehension.
6. Three open questions were provided for more expansive answers.
7. All questions were optional with the author choosing to allow partially
completed questionnaires in the results as the subject matter was deemed
likely to cause omitted answers. The author considered that forcing
respondents to complete all questions would significantly lower the response
rate. The responses received support this view with five of the seventeen
responses received failing to fully answer the closed questions. Rejecting
these results from the analysis would have excluded a significant proportion
of the sample (29%).
8. With no login procedure there was no limit on the number of responses an
individual could submit and the prototype did not validate the answers
received. This decision was taken to increase the response rate but did
increase the risk of accidental or deliberate contamination of the sample.
9. Responses were written to a file on the server in a format that could be
directly loaded into the database for statistical analysis, reducing the risk of
transcription errors.
Prior to the trial period the author conducted a pilot study with work colleagues
within the Technical Assurance/Design Authority department at National Grid.

The feedback received led to improvements in both the questionnaire and the site
itself. The prototype was subsequently published and feedback sought from
individuals and organisations with appropriate share trading knowledge (Table
2).
Individual/
Organisation
Potential
Number of
Respondents
Method of Communication
Energy Trading
Department – National
Grid
10 Face-to-face communication and follow-
up meetings.
Share trading clubs
(generally internet-
based)
200 Email to Secretary of each club
requesting he/she forward the request to
the members. (N.B. Twenty clubs were
contacted – the figure opposite assumes
an average membership of ten)
Family and Friends 40 Letters encouraging those with share
trading experience to respond to the
survey or to forward the link on to
others.
Work Colleagues 100 Email encouraging those with share
survey or to forward the link on to
others.
M801 Students 30 Message posted on the M801 Chat
Forum encouraging those with share
survey.
Total 380
Table 2 Users Targeted by Survey
From the target sample of 380 users it was felt 25-30 responses could be achieved.
National Grid traders and share club members would comprise a high proportion
domain experts and a 10% response rate was envisaged whilst a 5% response from
other groups would give a total sample of approximately 30.

The anonymous nature of the responses was identified as a potential problem. As
Denscombe (2007) pointed out:
“Questionnaires offer little opportunity for the researcher to check the
truthfulness of the answers given by the respondents”.
The author appraised this risk and chose to accept it rather than opt for the low
response rates achievable through face-to-face interviews (Chapter 3.2.2). As
mitigation the following steps were taken:
1. The target sample comprised domain experts operating in environments
where the aims of the research were likely to be well received.
2. The target sample was sufficiently large for a number of erroneous responses
to be accommodated without undermining the results.
3. Specific questions were asked to identify domain experts:
How many years have you been trading shares?
> 10 years 5 - 10 years 3 - 5 years 1 - 3 years 0 years
How regularly?
Daily Weekly Monthly Rarely Never
Which media type(s) do you use for your share reports?
Newspaper/Magazine Television Teletext Internet Datafeed
Other
How regularly do you read share reports?
Figure 13 Survey – Trading Experience
4. Additional “expert” questions were posed, requiring domain knowledge to
answer correctly, to provide a qualified view of the respondent’s experience:

What do you understand by the "AIM" market?
A listing of the top 100 companies within the Main Market
A listing on the LSE for smaller companies which offers less regulation
A listing of defence stocks
The Asian Investment Market
Don't Know
What do you understand by "Going Long"?
Buying futures which commit you to buy the asset at a set price in the future
Selling futures which commit you to sell the asset at a set price in the future
Buying shares as the price of the share falls
Selling shares as the price of the share falls
Don't Know
Figure 14 Survey – Domain Experts
The responses received were subjected to a series of statistic and analytical tests and
displayed no evidence of malicious misuse of the prototype.
4.5 Analysis of Handwritten Reports
ADVFN share reports are published twice daily in HTML format and emailed to
subscription members. The reports are derived from knowledge of LSE share
movements and an understanding of political and economic influences. The abstract
document structure of the reports remained broadly consistent across the trial period
as shown by the wireframe below (Figure 15).

01 Aug 2008 17:30
ADVFN III Evening Euro Markets Bulletin
Daily world financial news from Thomson Financial News Supplied by advfn.com
London
Lorem ipsum onsectetuer adipiscing elit
Lorem ipsum onsectetuer adipiscing elit, sed diam nonummy nibh euismod
tincidunt ut laoreet dolore magna aliquam erat volutpat. Ut wisi enim ad minim
veniam, quis nostrud exerci tation ullamcorper aliquip ex ea commodo consequat.
Ut wisi enim ad minim veniam, quis nostrud exerci tation ullamcorper suscipit lobortis
nisl ut aliquip ex ea commodo consequat. Duis autem vel eum iriure dolor in hendrerit in
vulputate velit esse molestie consequat, vel illum dolore eu feugiat nulla facilisis at vero
eros et accumsan et iusto odio dignissim qui blandit praesent luptatum zzril delenit
augue duis dolore te feugait nulla facilisi. Nam liber tempor cum soluta
Lorem ipsum onsectetuer adipiscing elit, sed diam nonummy nibh euismod tincidunt ut
laoreet dolore magna aliquam erat volutpat. Ut wisi enim ad minim veniam, quis nostrud
exerci tation ullamcorper suscipit lobortis nisl ut aliquip ex ea commodo consequat.
Ut wisi enim ad minim veniam, quis nostrud exerci tation ullamcorper suscipit lobortis nisl
ut aliquip ex ea commodo consequat. Duis autem vel eum iriure dolor in hendrerit in
eros et accumsan et iusto odio dignissim qui blandit praesent luptatum zzril delenit augue
duis dolore te feugait nulla facilisi. Nam liber tempor cum soluta
Report Title
FTSE100 Content
Movements & editorial
- standard font weight
- standard font size
Trend-following FTSE100 Shares
Individual share movements within the
LSE plus editorial content on market
trends and sentiment.
This text group consists exclusively of
shares showing significant movements
in the same direction as the FTSE100.
Trend-bucking FTSE100 Shares
Individual share movements within the
LSE plus editorial content on market
trends and sentiment.
If the previous group consisted of rising
stocks then this group consists of falling
stocks, and vice-versa.
LSE Headline
- bold font
Report Date
- right-hand side
- bold font
Market Analysis Content
Political & economic factors affecting
global markets.
Ut wisi enim ad minim veniam, quis nostrud exerci tation ullamcorper suscipit lobortis nisl
ut aliquip ex ea commodo consequat. Duis autem vel eum iriure dolor in hendrerit in
eros et accumsan et iusto odio dignissim qui blandit praesent luptatum zzril delenit augue
duis dolore te feugait nulla facilisi. Nam liber tempor cum soluta
LSE Section Heading
- bold font
- underlined font
- larger font size
Ut wisi enim ad minim veniam, quis nostrud exerci tation ullamcorper suscipit lobortis
nisl ut aliquip ex ea commodo consequat. Duis autem vel eum iriure dolor in hendrerit in
eros et accumsan et iusto odio dignissim qui blandit praesent luptatum zzril delenit
augue duis dolore te feugait nulla facilisi. Nam liber tempor cum soluta
Non-FTSE100 Shares
Information on LSE stocks outside the
FTSE100.
Generally structured as shares which
followed the FTSE100 trend followed by
those which bucked the trend, although
the abstract data structure is less rigid
than earlier sections of the report.
Figure 15 ADVFN Report - Abstract Document Structure
All reports began with the report title at the top of the page and the report date in the
top right-hand corner. All text was justified to align with left and right-hand margins
and, with the exception of the LSE section heading, used the same font size and style

throughout. The LSE section heading differed from the rest of the report in having
larger, underlined text in blue. The LSE headline, in common with all proper nouns,
was displayed in bold typeface and demonstrated a simple rhetorical structure aimed
at summarising the day’s share movements. Using 13th
August 2008 as an example
the LSE headline shows a simple nucleus of “London shares close lower” supported
by elaborations of additional information (Figure 16).
Figure 16 RST Example for LSE Headline
Beneath the LSE headline the ADVFN report positioned a paragraph detailing
movements of the FTSE100 index and followed this with several paragraphs of
market analysis. Below these, movements of FTSE100 stocks which echoed the trend
of the FTSE100 were detailed. This section generally reported the top 10 to 15
movers, providing price information and market analysis. This trend-following
section was followed by details of the top 10 to 15 movers in the opposite direction.
The report ended with a roundup of shares outside of the FTSE100. Figure 17
illustrates these characteristics for the 13th
August 2008 evening report.
Elaboration
London shares close
lower;
Wall Street drops;
Elaboration
U.S. retail sales fall

FTSE100 Content
Market Analysis Content
Trend-following FTSE100
Shares
Trend-bucking FTSE100
Shares
Non-FTSE100 Shares
LSE Section Heading
Report Date
Report Title
LSE Headline
Figure 17 ADVFN Report (13th
Aug 2008)

From this abstract document structure it is apparent that the reports cover both the
highest risers and fallers whatever the overall direction of the market, only the order
in which they are reported changes. The research prototype used two distinct
lexicons, one for rising stocks another for fallers, only the order in which they were
used changed to reflect the direction of the FTSE100. The prototype did not have the
relevant inputs to generate the “Market Analysis Content” section of the report and,
in fact, it is questionable whether a section with such high editorial content could be
generated through constraint satisfaction methods. This section did not appear in the
resulting NLG reports.
The research centred on ADVFN’s evening reports which provided information
directly comparable to the Level 1 service data. However such daily updates of share
movements prevented the research prototype from reporting intra-day share activity.
For example the ADVFN report for the 4th
August 2008 stated:
“The FTSE100 index closed down 34.5 points to 5,320.2, having
retreated from a morning peak of 5,414.7” (ADVFN report 4th
August
2008)
The research prototype could not produce this type of information as the satellite text
span (“having retreated from a morning peak of 5,414.7”) required knowledge of
intra-day trading activity.
The research prototype was limited to the Level 1 data outlined in Chapter 4.2, a
much narrower range of information compared to that available to the human authors
of the ADVFN reports. The prototype was unable to draw on an understanding of
market sentiment, external markets, political or economic factors in its report
production. The target text generated for the LSE headline above (Figure 17) is
reduced to “London shares close lower”.

Similarly the grouping of stocks into market sectors is unachievable with the
information to hand. Human authors bundle the reporting of related stocks into
cohesive phrases:
“Among the casualties, HBOS was off 24 pence at 307, Royal Bank of
Scotland was down 15-3/4 pence at 229-3/4, Barclays was 27 pence
lower at 355-1/2, and Lloyds TSB shed 20-1/2 pence to 308-1/2.”
(ADVFN report – 13th
Aug 2008)
The prototype, with no feel for the allocation of stocks to market sectors, is unable to
achieve this bundling and reports each stock as a separate, unrelated sentence.
Similarly expressing price movements of a group of stocks as a range or describing
them as list of related items is unachievable. Handwritten reports employ such
techniques to add fluency:
“Kazakhmys, Vedanta and Xstrata slid between 11.8 and 17.9 cent”
Nov 2008)
“Oil service groups Petrofac and Wood Group were up 23-1/2 at 544 and
17 at 397, respectively…”
Aug 2008)
The prototype is unable to perform these associations from data available to it with
“respectively” and “between” becoming redundant from its lexicon of terms.
A further differentiator between the handwritten reports and the NLG output is the
latter’s lack of elaboration and supporting content. The ADVFN reports use
additional text to support primary statements in a manner which cannot be achieved
by the prototype. The following sentence is typical of those employed by human
authors

“Upmarket residential property group Savills was the top FTSE 250
faller”
Aug 2008)
In rhetorical structure terms such phrases represent an elaboration pattern for which
the prototype can supply the nucleus (“Savills was the top FTSE 250 faller”) but not
the satellite text (“Upmarket residential property group”):
Figure 18 RST Elaboration in Handwritten Texts
In fact even the nucleus of this particular construct cannot be produced by the current
prototype. Although it has knowledge of stocks constituting the FTSE100 the same is
not true for alternative indices such as the FTSE250. A decision was taken to limit
the prototype’s scope to the FTSE100 and, whilst they could readily be incorporated
at a later date, alternative indices are not included in the current implementation. The
“Non-FTSE100 Shares” of Figure 17 are thus not created by the prototype.
Savills was the top
FTSE 250 faller
Upmarket residential
property group
Elaboration

4.6 Vocabulary of a Volatile Market
The study period of September to November 2008 proved one of the most eventful in
LSE history, with sizeable daily swings in market values and a strong downward
trend in the FTSE100 as the UK economy headed towards recession (Figure 19).
Figure 19 FTSE100 Daily Values and 5-day Trend Line
The current research evaluated the words/terms used in handwritten reports and their
rhetorical/textual structure in August 2008 prior to the trial. This work developed a
lexicon of terms and phrases adopted in the subsequent trial period for computer
generation of reports.
August represented a stable trading period for the FTSE100, illustrated by the small
variations in daily values and steady 5-day rolling average to the left of Figure 19. By
the start of the trial market conditions had worsened significantly with increased
market volatility and the FTSE100 rapidly losing 20-25% of its value (shown to the

right of Figure 19). Critically this research needs to determine whether human
authors continued with August’s words and phrases through the trial period. The
prototype was unable to alter its lexicon of terms once the study was underway and
questionnaire results received. The following sections examine whether the changing
market conditions led to changes in the vocabulary used by human authors and the
consequences of this.
Table 3 shows that during August the FTSE100 showed negligible changes in value
for one in five trading days. By the trial period only 8% of days showed variations of
less than 0.5% due to the increased market volatility.
Pre-Trial
(Aug 2008)
Trial
(Sep – Nov 2008)
FTSE100 Trend
Total % of
Total
Total % of
Total
Upward
(daily rise ≥ 0.5%)
6 days 30% 25 days 38%
Stable
(daily movement < 0.5%)
4 days 20% 5 days 8%
Downward
(daily fall ≥ 0.5%)
10 days 50% 35 days 54%
Table 3 FTSE100 Daily Trends
The research categorised trading on the LSE into days exhibiting upward, downward
or stable FTSE100 trends. For both the pre-trial and trial periods the lexicon of terms
used by handwritten reports within each category were analysed for randomly chosen
dates (Table 4).

Pre-Trial
(Aug 2008)
Trial
(Sep – Nov 2008)
FTSE100 Trend
Chosen
Date
FTSE100
Change
Chosen
Date
FTSE100
Change
5th
Aug 08 2.5% 19th
Sep 08 8.8%
8th
Aug 08 0.6% 13th
Oct 08 8.3%
Upward
(daily rise ≥ 0.5%)
14th
Aug 08 0.9% 20th
Oct 08 5.4%
-0.1% 13th Nov 08 -0.3%
-0.0% 25th Nov 08 0.4%
Stable
(daily movement < 0.5%)
12th
Aug 08
21st
Aug 08
26th Nov 08 -0.4%
1st
Aug 08 -1.1% 29th
Sep 08 -5.3%
4th
Aug 08 -0.6% 22nd
Oct 08 -4.5%
Downward
(daily fall ≥ 0.5%)
13th
Aug 08 -1.6% 19th
Nov 08 -4.8%
Table 4 FTSE100 Sample Days
The handwritten reports were analysed for each of the sample days to create corpus
texts (Reiter and Dale, 2000) of the words and terms used by human authors. This
exercise was undertaken for each FTSE100 trend before and during the trial and
these corpus texts were further refined to remove elements not directly available to
the prototype or those which could not be computed from its inputs. For the purposes
of comparing the vocabulary used before and during the trial these target text corpora
were consolidated into two lexicons of terms, one pre-trial, the other spanning the
trial period. Analysis of these lexicons illustrated tangible differences between the
pre-trial vocabulary and that used during the actual trial. During the stable pre-trial
period the handwritten reports showed little variation in grammar. Reports adopted a
restricted vocabulary which, the author contends, reflected the restricted movements
within the market. Sentences proved heavily reliant upon a few verbs and a limited
range of adjectives. Figure 20 illustrates for the target text corpus the proportional
use of verbs across all of the sample day reports in the pre-trial period and Figure 21
the equivalent use of adjectives. Any verb or adjective whose use accounted for more

than 1% of the total verbs or adjectives within the target text corpus was plotted as an
axis. The distance from the centre illustrates the relative use of the verb or adjective
against the total used in sample days in the pre-trial period.
Verbs Used in Manual Reports
0%
5%
10%
15%
20%
is
close
fall
shed
add
gain
drop
lose
rise
take
jump
slide
Other…
slip
surge
firm
rally
ease
tick up
climb
end
advance
lead
leap
glance
have
sag
sink
Verbs Before Trial
Figure 20 Pre-Trial ADVFN Reports - Verbs

Adjectives
0%
5%
10%
15%
20%
dow n
up
higher
low er
off
on
ahead
among
Other…
strongerw eaker
mixed
top
biggest
soared
battered
respectively
low
plummeting
Adjectives Before Trial
Figure 21 Pre-Trial ADVFN Reports - Adjectives
Over the trial period the lexicon expanded, showing less reliance upon stock terms
and phrases. Figure 22 illustrates verbs used within the target text corpus during this
period. The heavy reliance upon “is”, “close”, “fall”, “shed”, “add” and “gain” was
balanced by a broader range of verbs and a wider vocabulary.

0%
5%
10%
15%
20%
is
close
fall
shed
add
gain
drop
lose
rise
take
jump
slide
Other…
slip
surge
firm
rally
ease
tick up
climb
end
advance
lead
leap
glance
have
sag
sink
Verbs During Trial
Figure 22 Trial ADVFN Reports - Verbs
The contrast between verbs used during the pre-trial period and the trial period itself
is shown in Figure 23. The pre-trial vocabulary shows little variety in describing the
FTSE100 or movements of individual shares whilst the volatile trading of the trial
period coincides with an expanded verb base and more emotive terms. Verbs such as
“jump”, “slip”, “surge”, “leap”, “sag” and “sink” gain significance within the lexicon
in an attempt to effectively capture the magnitude of changes experienced by the
markets.

0%
5%
10%
15%
20%
is
close
fall
shed
add
gain
drop
lose
rise
take
jump
slide
Other…
slip
surge
firm
rally
ease
tick up
climb
end
advance
lead
leap
glance
have
sag
sink
Verbs Before Trial Verbs During Trial
Figure 23 Trial/Pre-Trial Comparison - Verbs
The comparison (Figure 23) also illustrates that, despite a broadening of the
vocabulary, the reports continue to draw heavily on verbs used in the pre-trial
reports. For adjectives the same holds true, the early dependence on “up”, “down”,
“higher” and “lower” is maintained across the trial period (Figure 24 and Figure 25)
but more expressive terms (“battered”, “soared”, “plummeting”) begin to enter the
lexicon.

Adjectives
0%
5%
10%
15%
20%
dow n
up
higher
low er
off
on
ahead
among
Other…
strongerw eaker
mixed
top
biggest
soared
battered
respectively
low
plummeting
Adjectives During Trial
Figure 24 Trial ADVFN Reports - Adjectives
Adjectives
0%
5%
10%
15%
20%
dow n
up
higher
low er
off
on
ahead
among
Other…
strongerw eaker
mixed
top
biggest
soared
battered
respectively
low
plummeting
Adjectives Before Trial Adjectives During Trial
Figure 25 Trial/Pre-Trial Comparison - Adjectives

Nouns show similar changes of vocabulary in response to the market conditions. Pre-
trial reports were dominated by “pence” as a noun to describe share movements:
“Thomas Cook Group, up 17 pence at 237-1/4”
Aug 2008)
“Unilever closed 29 pence lower at 1,359 in London”
(ADVFN report – 1st Aug 2008)
By the trial period “pence” had been almost entirely replaced by “percent” to account
for such movements:
“HSBC dropped 9.1 percent”
Nov 2008)
“Royal Dutch Shell gained 10.6 percent”
Oct 2008)
Figure 26 illustrates this shift in vocabulary.

Nouns
0%
10%
20%
30%
40%
pence
percent
shares
index
points
london
ftse100
Other…
ftse250
upside
second tier
stock
Britain's
losers
blue chips
price
Nouns Before Trial Nouns During Trial
Figure 26 Trial/Pre-Trial Comparison - Nouns
Despite these adjustments in vocabulary the author contends that the pre-trial lexicon
bears sufficient similarity to that used during the trial for it to be applicable to study’s
aims.
4.7 Summary
The study used a commercial organisation to provide the underlying data and
handwritten reports, storing this information on the server within a database or as
electronic files. The prototype used a SISCtus Prolog constraint satisfaction program
to generate NLG output from this data and apply user preferences to the results
before presenting them to the user. The web site demonstrated elements of workflow
which encouraged users to experiment with the application before completing the
survey questionnaire. The level of feedback was disappointing and the results in the

following chapter would probably have shown less variance and greater distribution
symmetry had they been drawn from a larger sample.
Considerable effort was applied to determining the target text corpus achievable from
the handwritten reports. With its restricted set of inputs both the vocabulary and
range of rhetorical structures available to the prototype is limited but would still
appear to be worthy of study. Changes in the vocabulary of handwritten reports over
the trial period were analysed and, whilst measurably different from the pre-trial
lexicon, are believed by the author to possess sufficient similarity to be relevant to
the research.

Chapter 5 Data Analysis
5.1 Introduction
The prototype employed an internet survey to determine the extent to which the aims
of the research were met. This chapter examines the responses to that survey and
subjects them to rigorous statistical analysis. The results are seen as having broader
applicability than the sample itself and inferences are drawn for the wider population.
5.2 Respondents
The study looked to obtain around thirty responses from domain experts and share
traders (Chapter 4.4) but after five months (Sep 08 – Feb 09) seventeen survey
responses had been received. These responses contained more inexperienced traders
than expected (Figure 27) with the group trading less and reading fewer share reports
than expected (Figure 28).
Share Trading Experience
0
1
2
3
4
5
6
7
8
9
0 1-3 3-5 5-10 10+
Years
Figure 27 Survey Responses - Experience

0
2
4
6
8
10
Respondents
DailyWeeklyMonthlyRarelyNever
Read Reports
Trade Shares
Invovement in Trading
Figure 28 Survey Responses - Trading
Despite these results many respondents showed a clear understanding of share
trading and over half answered both “expert” questions correctly (Chapter 4.4).
These responses and the survey’s focus on presentation/layout factors requiring
limited domain knowledge led the author to deduce the sample was sufficiently
representative of the overall population. The following sections analyse the sample
responses, determining what inferences can be drawn for web-based share report
users in general and the degree of confidence attributable to the findings.
5.3 User Preferences
The questionnaire probed respondents’ views on the style, layout, grouping and
navigation within the prototype reports. These factors broadly aligned with the text
structure and abstract document structure elements identified by Power et al (2003).
The questions sought to identify whether the effectiveness and coherency of the NLG
reports was or could be improved by modelling NLG output in line with user
preferences.

5.3.1 Styles
The prototype provided four display styles, each targeting specific aspects of visual
appearance (Appendix C) and asked which, if any, respondents found most effective
(Figure 29).
Which of the 4 display styles did you find most effective?
Small Text Large Text Indented Ring Binder None
Figure 29 Survey - Display Style
Respondents overwhelming found the use of a display style effective (88% support)
with the indentation of text produced by the “Indented” and “Ring Binder” styles
accounting for 69% of preferences (Figure 30).
Preferred Style
0
1
2
3
4
5
6
7
Small Text Large Text Indented Ring Binder None Don't Know
Style
Respondents
Figure 30 Survey Responses - Preferred Style
Responses on report structure showed the NLG documents to be consistently well
structured (Figure 31).

How consistent was the structure of this report?
I always knew where to find a report element
I generally knew where to find a report element
I usually knew where to find a report element but with exceptions
I generally did not know where to find a report element
I never knew where to find a report element
Figure 31 Survey - Report Structure
The stars awarded displayed a significant bias towards “I always knew where to find
a report element”. The results showed no tendency towards a normal distribution
curve, instead exhibiting a trailing left-handed tail synonymous with negatively
skewed distributions (Figure 32). With a sample mean of 4.0 stars and a mode (and
median) of 5.0 stars the distribution showed a coefficient of skewness of -1.26
reflecting the distribution’s marked asymmetry around its mean. This pattern
emphasises the respondents’ strong belief that the NLG reports provide a consistent
report structure.
Consistency of Report Structure
1 1
2
4
7
0
1
2
3
4
5
6
7
8
1 2 3 4 5
Stars Awarded
Respondents
Figure 32 Survey Responses - Report Structure

Structurally the reports consisted of distinct text groupings for the FTSE100
summary, top risers, top fallers and preferred share information. Respondents’ views
were sought on the size and positioning of these blocks (Figure 33).
How satisfactory did you find the size and position of the text blocks?
The group sizes/positions totally reflected their relative importance
The group sizes/positions generally reflected their relative importance
The group sizes/positions usually reflected their relative importance
The group sizes/positions generally did not reflect their relative
importance
The group sizes/positions totally failed to reflect their relative
importance
Figure 33 Survey - Text Blocks
The responses provided a more symmetrical distribution than the previous question
(Figure 34). Although the distribution exhibits skew it tends towards a normal
distribution curve.

Size & Position of Text Blocks
2
4
6
2
1
0
1
2
3
4
5
6
7
1 2 3 4 5
Stars Awarded
Respondents
Measure Value Measure Value
Arithmetic Mean 2.73 Standard Deviation (SD) 1.10
Median 3 Coefficient of Skewness 0.24
Mode 3 Standard Error (SE) 0.28
Figure 34 Survey Responses - Text Blocks
The coefficient of skewness, at 0.24, reflects much greater distribution symmetry
around the mean than was evident for report structure. The curve shows the tail
trailing gently towards the positive values but much less acutely than the previous
example. Intuitively one might expect the curve to gravitate towards a normal
distribution as the number of samples/responses increases.
A fundamental question for the research is whether inferences can be made for the
wider population on the basis of these sample results. Rowntree (2000) provides a
statistical basis for such generalisations, arguing that the sample mean (“S-mean”)
and sample standard deviation (“S-standard deviation”) allow estimates of the
population mean (“P-mean”) and population standard deviation (“P-standard
deviation”) within certain confidence limits. Armed with knowledge of the normal

distribution curve’s properties it is possible to state the probability of the P-mean
lying within a certain range of values. Whilst individual samples may show skewed
distributions (as above) the frequency distribution of S-means for a high number of
samples will plot as a normal distribution curve and the mean of these S-means will
centre around the population mean. Irrespective of whether the original population or
its samples were normally distributed the S-mean distribution will approximate to a
normal distribution (Rowntree, 2000).
A measure of the dispersion of the S-means around the P-mean is obtained by
calculating the standard deviation of these S-means. This variable, termed the
“standard error” (“SE”), has a magnitude related to the S-standard deviation and the
sample size:
For the size and positioning of text blocks the standard error equals 0.28 “stars”. For
this normal distribution of S-means we can state that 68% of S-means will lie within
1 SE of the P-mean (Figure 35) or alternatively there is a 68% probability the P-mean
lies within 1 SE of this S-mean. We are thus 68% confident that the true population
mean lies between 2.45 and 3.01 stars. Statisticians however usually work at the
95% (P-mean ± 2 SE) or 99% (P-mean ± 2.5 SE) confidence levels, hence for our
size and positioning results there is a 95% probability the true population mean lies
between 2.17 and 3.29 stars and a 99% probability it is between 2.11 and 3.36 stars.
Standard Sample Standard Deviation
Error = √ Sample Size

Text Text Text Text Text Text
Normal Distribution Curve
1 SE2 SE2.5 SE 2.5 SE2 SE1 SE
68%
95%
99%
Figure 35 Properties of a Normal Distribution Curve
The results suggest that if all users of web-based share reports were surveyed we
could be confident (at the 99% level) that the overall population’s satisfaction level
lay between 2.11 stars (roughly equating to: “The group sizes/positions did not
reflect their relative importance”) and 3.36 stars (“The group sizes/positions usually
reflected their relative importance”). This shows mild dissatisfaction with the size
and position of text blocks, a view supported by responses to the open questions:
“I would like to be able to choose to have my preferred shares presented
first so I didn’t have to scroll down to find the information of most
importance to me (e.g. if I had shares in those companies or was
thinking of investing)”
“The optional share movements were reported after the risers & fallers,
even if they were the shares I was most interested in.”
5.3.2 Text Groups
The suitability of categories used in sub-dividing the NLG reports was appraised by
respondents (Figure 36).

How well did the grouping work?
Report was completely in the category groups I expected
Report was generally in the category groups I expected
Report was grouped into categories similar to those I expected
Report was generally not in the category groups I expected
Report was not at all in the category groups I expected
Figure 36 Survey - Report Grouping
The results suggest overall satisfaction with the grouping and are a vindication of the
categories adopted in the original handwritten reports on which the NLG outputs are
based (Figure 37). The curve shows a tendency towards a normal distribution
overlaid with a degree of negative skew. With an arithmetic mean of 4.00 stars and a
coefficient of skewness of -0.80 the distribution shows a central tendency of 4 stars
tapering towards the left-hand values.

Success of Text Grouping
0
1
2
7
4
0
1
2
3
4
5
6
7
8
1 2 3 4 5
Stars Awarded
Respondents
Median 4 Coefficient of Skewness -0.80
Figure 37 Survey Responses - Text Grouping
This satisfaction with the grouping of the NLG reports was not wholeheartedly
endorsed when compared with their handwritten counterparts (Figure 38). NLG
reports are based on the ADVFN structure and although changes were made to allow
for their differing inputs it was expected that the central tendency would lie at 3 stars
(i.e. a “Comparable grouping of information”). The comparison did not however
show a central tendency which indicated the groups to be broadly similar.

Comparison of Groupings with ADVFN
1
2
1
7
2
0
1
2
3
4
5
6
7
8
1 2 3 4 5
Stars Awarded
Respondents
Figure 38 Survey Responses – Grouping Comparison
In part this may be due to the small sample size, with only thirteen responses it is
conceivable this sample belongs to a normally distributed population but holds an
unusually high proportion of outliers from the edges of the distribution. The sample’s
standard deviation of 1.20 stars suggests a high degree of variance within the sample
which is further evidenced by a standard error of 0.33 stars. The magnitude of this
standard error causes the population mean, at a 99% confidence interval, to range
from 2.71 to 4.37 stars – far too broad for inferences to be drawn on the overall
population. This variability of responses made the author reluctant to draw
conclusions from the results. Increased sample sizes may go some way to lowering
this variability but responses to open questions suggest text grouping exhibits a
complexity and subtly not allowed for in the prototype:
“Customisation for grouping under market sectors would be good.
Customisation to see % changes would be good.”

“I would have preferred to group shares by market sectors (e.g. banking
or pharmaceuticals) and report on movements within these sectors.”
“Put the preferred first. Include percentage as well as pence”
These results suggest the grouping of text clearly influences the coherency of the
report but respondents differ on how the content should be grouped. The inclusion of
user preferences in report generation has obvious benefits in this area, allowing
individuals to group text to their own specifications. Further research is required to
establish the parameters by which users wish to group text blocks.
The survey also enquired whether inclusion of user preferences had improved report
readability (Figure 39).
Did customisation make the report easier to read?
Yes - Much more focused in providing my information
Yes - More focused to providing my information
Unchanged from manual report
No - Less focused in providing my information
No - Much less focused in providing my information
Figure 39 Survey - Customisation
The response was a resounding “Yes” with 75% of respondents answering “Yes –
More focused to providing my information”. When plotted as a curve the results
suggest a normal distribution (Figure 40) however the importance of the modal value
is more apparent as a histogram (Figure 41).

Customisation & Ease of Use
1 1 1
11
1
0
2
4
6
8
10
12
1 2 3 4 5
Stars Awarded
Respondents
Figure 40 Survey Responses - Customisation
0
2
4
6
8
10
12
1 2 3 4 5
Stars Awarded
Customisation & Ease of Use
Figure 41 Survey Responses - Customisation (Histogram)
The results suggest that dynamically-generated reports incorporating user preferences
improve report readability. When viewed as a sample from a normally distributed

population the results imply mild satisfaction with the ease of reading NLG reports
and 99% confidence in a P-mean between 3.04 and 4.30. These statistics appear less
emphatic than suggested by the strong modal value displayed in the histogram. A
one-tailed test (“t-test”) was applied to the data to ascertain whether the sample
distribution varied significantly from a normal distribution (Rowntree, 2000). Using
a null hypothesis that “There is no difference in readability between handwritten and
NLG reports” and an alternative hypothesis of “NLG reports are easier to read” the
test becomes one-tailed – if the null hypothesis proves false the alternative
hypothesis only proves true if the result lies at one end of the distribution curve.
The null hypothesis produces a normal distribution with equal probability of
respondents favouring handwritten reports as NLG reports. The P-mean of this
distribution lies at 3.00 stars (“Unchanged from manual report”) with a standard
error of 0.25 stars.
For the one-tailed test to prove 95% significant the S-mean must fall more than 1.67
SE to the right of the P-mean (i.e. exceed 3.42 stars) and more than 2.33 SE for 99%
confidence (i.e. exceed 3.58 stars). The S-mean of 3.67 stars exceeds both values and
initially appears significant for a normal distribution sample however no account was
taken of the limited sample size (15 values) and its effect on the distribution curve.
The t-test accounts for the increased variability of small samples by flattening the
distribution curve profile (Rumsey, 2003). Applying the test to the sample shows that
for the alternative hypothesis to be 95% significant the S-mean must lie more than
1.76 SE to the right of the P-mean (i.e. exceed 3.44 stars) and more than 2.98 SE for
99% confidence (i.e. exceed 3.74 stars). These results prove, with 95% certainty, that
“NLG reports are easier to read” but stop short of providing 99% confidence in this
result.

5.3.3 Page Navigation
Although the prototype output was limited to a single page its structure, groupings
and text styles have an influence on the user’s ability to recognise and interpret the
meaning and content within the page. The survey asked a series of questions on the
contribution of text size, colour and positioning to page navigation (Figure 42).
The "Small Text" report style used less variations in font size than the other styles.
How significant were font size variations in identifying the main elements
of the report?
Very significant, it greatly affected my ability to recognise text groups
Significant, it affected my ability to recognise text groups
Mildly significant
Insignificant, it did not affect my ability to recognise text groups
Very insignificant, it had no effect on my ability to recognise text groups
The "Large Text" report style used bold font to highlight headings.
How significant was font weight in identifying text groups?
Mildly significant
Very insignificant, it had no effect on my ability to recognise text
groups
The "Indented" report style used text positioning & indentation to differentiate text
groups.
How significant was text positioning in identifying text groups?
Mildly significant
groups
The "Ring Binder" report style used text colour to differentiate text groups.
How significant was colour in identifying text groups?
Mildly significant
groups

Figure 42 Survey - Page Navigation
A similar number of answers were received for each question but showed varying
levels of support for the significance of each independent variable (Figure 43).
Respondents rated font size of little relevance with similar numbers classing it as
significant as those who did not. The font size distribution curve proved highly
symmetrical around its arithmetic mean and, with a coefficient of skewness of only
0.15, tended towards a normal distribution profile. The distribution showed mode and
median values of 3 stars (“Mildly significant”) and a mean of 2.93 stars (Table 5)
demonstrating with 95% confidence that the population mean lay between 2.40 and
3.47 stars or between 2.27 and 3.60 stars at 99% confidence. These figures suggest
font size variations were, at best, mildly significant and largely viewed as of little
importance in identifying text groups.
1 2 3 4 5
Font Size
Font Weight
Text Position
Text Colour
Stars Awarded
Figure 43 Survey Responses - Page Navigation
The use of font weighting to highlight headings and differentiate elements of text was
viewed as more significant in the identification of text groups than variations in font

size. Font weights showed a mean value of 3.14 and, with a coefficient of skewness
of -0.95, a distribution profile weighted towards the higher values. Broadly the
results showed respondents viewed font weight as mildly significant or significant in
identifying text groups. The sample did not develop a normal distribution profile,
perhaps due to its limited size, and no respondents rated font weights as five stars
despite the modal value of four stars accounting for 43% of the responses.
Navigation
Parameter
Median 3 Coefficient of Skewness 0.15Font Size
Median 3 Coefficient of Skewness -0.95Font Weight
Median 4 Coefficient of Skewness -1.49Text Position
Median 4 Coefficient of Skewness -0.59Text Colour
Table 5 Page Navigation Statistics
The prototype indented text groups and used drop-cap effects to identify the
separation between them (Appendix C). The responses received heavily favoured the
modal value of four stars with 67% of the returns but produced a highly asymmetric
distribution in which no one rated it higher. Quantitative evaluation of the sample
supports this with the high coefficient of skewness highlighting the one-sided nature
of the distribution (Table 5). The evidence suggests the majority of domain experts
found text positioning to be a significant factor in identifying text groups but for a
minority it had little or no effect. These characteristics make text positioning an ideal
candidate for inclusion within the individual preferences made available to users.

Colour was used in the prototype to distinguish text blocks, with the FTSE100
summary shown in black, risers in green, fallers in red and preferred shares in blue
(Appendix C). Respondents were queried on the importance of this colour scheme in
identifying text groupings with a majority (67%) finding it either significant or
mildly significant. The distribution curve contained a negative skew, with a
coefficient of skewness of -0.59, much less than for text position, and a rough
approximation to a normal distribution (Figure 44).
Significance of Colour in Identifying Text Groups
1
2
4
6
2
0
1
2
3
4
5
6
7
1 2 3 4 5
Stars Awarded
Respondents
Figure 44 Survey Responses - Colour
The sample mean of 3.40 stars showed a high variance around this figure,
characteristic of a small sample size, with a standard deviation of 1.12 and a flattened
distribution curve. The results have a high standard error and a broad range within
which the true population mean should reside (between 2.82 and 3.98 stars at the
95% confidence interval; 2.68 to 4.12 at the 99% confidence interval). The results
suggest the majority of domain experts found text colour to be significant or mildly
significant in identifying text blocks but that views varied across the user community.

5.4 Use of Commercial Data
The questionnaire obtained feedback on the vocabulary and grammar employed
within prototype reports compared to their handwritten equivalents. The results were
analysed on whether the commercial data was seen to provide sufficient quality to
generate sophisticated page content and to what extent the prototype’s restricted
access to information known to human authors influenced the quality of its outputs.
Users were queried on the vocabulary, grammar and sentence construction of the
ADVFN reports through survey questions (Figure 45).
How similar was the vocabulary to that used in the ADVFN reports?
Very similar, the actual words and phrases used were always the same
Similar, the actual words and phrases used were generally the same
Usually the actual words and phrases used were similar but with some exceptions
Dissimilar, the actual words and phrases used were generally not the same
Very dissimilar, the actual words and phrases used were never the same
How similar was the grammar to that used in the ADVFN reports?
Very similar, the sentence structure & word arrangement were always the same
Similar, the sentence structure & word arrangement were generally the same
Usually the sentence structure & word arrangement were similar but with
exceptions
Dissimilar, the sentence structure & word arrangement were generally not the
same
Very dissimilar, the sentence structure & word arrangement were never
the same
How similar were the sentences to those used in the ADVFN reports?
Very similar, the sentences always showed the same size and complexity
Similar, the sentences generally showed the same size and complexity
Usually the sentences showed the same size and complexity but with exceptions
Dissimilar, the sentences generally did not show the same size and complexity
Very dissimilar, the sentences never showed the same size and complexity
Figure 45 Survey - Sentence Construction
Respondents offered a range of views on the similarity of the vocabulary perhaps
because of the dramatic changes in market conditions experienced over the trial

period and its impact on the language of reporting (Chapter 4.6). Plotted as a
distribution curve the responses showed no strong central tendency and although
tending weakly towards “Usually the actual words and phrases used were similar but
with some exceptions” variations were common (Figure 46).
Vocabulary Comparison
1
5
4 4
00
1
2
3
4
5
6
1 2 3 4 5
Stars Awarded
Respondents
Figure 46 Survey Responses - Vocabulary Comparison
Views on the similarity of the grammar used were more consistent than for
vocabulary. Users were asked to score the prototype’s use of sentence structure and
word arrangement against that used by the ADVFN reports (Figure 45). The
responses showed a normal distribution curve tending towards a mean value of 3.21
and a standard deviation of 0.80 (Figure 47).

Grammar Comparison
0
2
8
3
1
0
1
2
3
4
5
6
7
8
9
1 2 3 4 5
Stars Awarded
Respondents
Figure 47 Survey Responses - Grammar Comparison
This curve allows the mean of the overall population to be inferred to a reasonable
level of precision. At a 95% confidence interval the P-mean, based on this sample,
lies between 2.79 and 3.64 stars whilst at 99% confidence it falls between 2.68 and
3.75. These results suggest the commercial data received from the LSE was of a
quality whereby “Usually the sentence structure and word arrangement were similar
but with exceptions”. The author contends that these exceptions were due to the
limited information available to the prototype and that the commercial data itself
produced sophisticated page content. This view is supported by responses to open
questions:
“Sentence construction was good but simple.”
“Prototype report was more regimented however this made comparison
of share performance better. Perhaps it would be good to group similar
% changes together in one sentence to make prototype less wordy.
Having % change, change and absolute figures would be helpful.

Frequent variation in verbs made the prototype more interesting to
read.”
Sentence comparisons with the handwritten equivalents showed the NLG outputs to
be shorter and less complex. The arithmetic mean of 2.57 and standard deviation of
1.02 stars (Figure 48) give 95% confidence of a population mean between 2.03 and
3.11 stars and 99% confidence of it lying between 1.89 and 3.25. Compared to
handwritten reports the NLG sentences averaged between “Dissimilar, the sentences
generally did not show the same size or complexity” and “Usually the sentences
showed the same size and complexity but with exceptions”.
Sentence Comparison
1
7
4
1 1
0
1
2
3
4
5
6
7
8
1 2 3 4 5
Stars Awarded
Respondents
Figure 48 Survey Responses - Sentence Comparison
This quantitative analysis of NLG sentences is supported by the qualitative
comments received to open questions:
“The reports seemed limited to short sentences with no connection
between the adjoining sentences. The ADVFN reports gave an
interpretation of WHY the shares had moved whereas the prototype
simply reported that the shares had moved. In share investing this gives

little indication of what to invest in, only a commentary on what has
already happened.“
The handwritten reports contained information which was unavailable to the
prototype (e.g. inflation figures; political & economic activity; company mergers and
acquisitions). This information allowed human authors to infer reasons for share
movements and provide a context to market changes. The prototype was limited to
LSE share data and could only report movements rather than infer the underlying
causes. The survey inquired whether the absence of such information affected the
quality of the NLG reports (Figure 49).
The prototype did not have access to the same data as the human-authored reports.
Data not available to the prototype was highlighted in yellow in the ADVFN reports.
How did this missing information affect the quality of the prototype reports?
Much better, the sophistication of the reports greatly improved
Better, the sophistication of the reports generally improved
Unchanged, the sophistication of the reports was the same as for manual reports
Poorer, the sophistication of the reports generally worsened
Much poorer, the sophistication of the reports appreciably worsened
Figure 49 Survey - Missing Data
The results were inconclusive with most respondents feeling the absence of the data
lowered the sophistication of the NLG reports but no clear convergence of views
(Figure 50). The sample cannot be said to represent a normal distribution curve and
few conclusions can be drawn from it. Intuitively it would seem logical that more
information would result in more sophisticated outputs but further research is
required to determine the nature and benefits of such information.

Missing Information
0
7
2
3
1
0
1
2
3
4
5
6
7
8
1 2 3 4 5
Stars Awarded
Respondents
Figure 50 Survey Responses - Missing Data
5.5 Summary
The survey responses allowed quantitative analysis to be undertaken with the
prototype and statistical rigour applied to what might otherwise have been subjective
judgement. In spite of the small sample size the survey provided crucial evidence in
corroborating the aims of the research and identifying areas for further research. The
following chapter examines these elements and draws conclusions for the research.

Chapter 6 Conclusions
6.1 Comparison with the Research Aims
Respondents overwhelming found the use of a display style effective and there was
strong support for the view that customisation made the reports easier to read.
Despite general satisfaction with the text grouping and the consistency of the report
structure the overriding impression was that individual user preferences would best
be satisfied by a system which allowed customisation of the outputs. Respondents
expressed mild dissatisfaction with the size and position of text blocks and
recommended improved configurability whilst the comparison between handwritten
and NLG text groups produced a variety of views and suggestions for alternative
arrangements. Most respondents saw page navigation as primarily influenced by text
position and text colour but the range of views expressed reinforced the need for
greater individual freedom in report presentation.
The grammar used in NLG sentences was generally considered of comparable
sentence structure and word arrangement to the handwritten reports but views on the
vocabulary differed with no clear consensus on the similarity of the words and
phrases used. These differences may be the result of the turbulent market conditions
experienced over the trial period. The NLG sentences were noticeably shorter and
less complex than the handwritten reports, a predictable consequence of the
prototype excluding discourse connectives from sentence construction and operating
on a subset of the information available to human authors. The prototype adequately
demonstrated the capabilities of a constraint satisfaction program in generating high
quality text outputs from commercial data. The grammar of the documents proved
equivalent to handwritten reports whilst variations in vocabulary and sentence

complexity could reasonably be attributed to factors other than the quality of the
underlying data.
Overall these findings support the project aims, stated in Chapter 1.3, that the
inclusion of user preferences lead to improvements in the effectiveness and
coherency of web page designs and content. Furthermore the research has
demonstrated that a commercial data source can provide the quality of data required
to generate sophisticated web page content.
6.2 Directions for Future Work
The research investigates the use of NLG and a commercial data source within a
specific commercial sector. This work offers opportunities for further research both
within this arena and to extend its principles to other sectors. The prototype’s inputs
could be extended to include information on LSE sectors (e.g. mining, aerospace),
forex rates, inflation figures and economic indicators (e.g. retail price index). In
addition the constraint satisfaction program should be remodelled to determine
whether the market volatility of the trial period influenced the survey results. The
prototype should also be extended to use discourse connectives (e.g. “whilst”,
however”) in its sentence generation to determine whether the size and complexity of
the resulting sentences is comparable to that of human authors.
Looking beyond share reports the research has wide applicability. If the principle of
commercial data providing NLG content can be established the resulting text could
form the inputs to alternative technologies such as natural language processing.
Further research should build on the potential for commercial sources
communicating numeric data through speech or non-visual methods. Separation of
document content from its presentation also increases the possibility of displaying

NLG outputs through a range of alternative media (e.g. palmtops, mobile phones,
motorway signs) without the need for human authorship.
6.3 Project Review
This research set ambitious targets for generating natural language text from numeric
data. Along the way I have embarked on an extensive literature search and become
competent in several development languages (Prolog, PHP, JavaScript, SQL, HTML
and CSS). The project’s focus on a prototype application and its ability to interact
with its audience placed tremendous emphasis on achieving this element of the work
to schedule. With hindsight I might have done things differently, de-scoping the
prototype or setting less ambitious targets, however I am immensely proud of what
has been achieved. The prototype and the survey have been well received, despite the
complexity of the subject matter, and the responses have allowed me to undertake a
thorough analysis of the subject.
The methodologies employed appeared well suited to the subject area but the low
response rate to the web-based survey needs to be addressed in any future work.
Increasing use of case study methods or sampling a much larger target population
may provide greater insight into the specifics of user customisation and the quality of
the commercial data source.

References
André, E and Rist, T. (1994) ‘Generating coherent presentations employing textual
and visual material’, Artificial Intelligence Review, 9, pp.147-165
Bateman, J., Kamps, K., Kleinz, J. and Reichenberger, K. (1998) ‘Communicative
Goal-Driven NL Generation: An Architectural Synthesis for Multimedia Page
Generation’, Proceedings of the 1998 International Workshop on Natural Language
Generation, pp.8- 17
Bateman, J. and Delin, J. (2001) ‘From Genre to Text Critiquing in
Multimodal Documents’, MAD 2001: the 4th
International Workshop on
Multidisciplinary Approaches to Discourse: From text structure to text type, Yttre,
Belgium.
Bateman, J., Kamps, K., Kleinz, J. and Reichenberger, K. (2001) ‘Towards
Constructive Text, Diagram and Layout Generation for Information Presentation’,
Computational Linguistics, 27 [3], pp.409-448
Bateman, J., Delin, J. and Henschel, R. (2002a) ‘XML and multimodal corpus
design: experiences with multi-layered stand-off annotations in the GeM corpus’,
LREC’02 Workshop: Towards a Roadmap for Multimodal Language Resources and
Evaluation, Las Palmas, Canary Islands
Bateman, J., Henschel, R. and Delin, J. (2002b) ‘A brief introduction to the GeM
annotation schema for complex document layout’, International Conference On
Computational Linguistics: Proceedings of the 2nd
workshop on NLP and XML, 17,
pp.1- 8
Calvi, L. and De Bra, P. (1997) ‘Proficiency-Adapted Information Browsing and
Filtering in Hypermedia Educational Systems’, User-Modelling and User-Adapted
Interaction, 7, pp.257-277
Cleveland, W. S. and McGill, R. (1984) ‘Graphical perception: Theory,
experimentation and application to the development of graphical methods’, J.
Am. Stat. Assoc., 79, pp.53 1-554
Dale, R., Green, S. J., Milosavljevic, M., Paris C., Verspoor, C. and Williams, S
(1998) ‘The Realities of Generating Natural Language from Databases’, Proceedings of
the 11th
Australian Joint Conference on Artificial Intelligence
Denscombe, M. (ed) (2007) The Good Research Guide for small-scale research
projects, New York, Open University Press
Dilley, S., Bateman, J., Thiel, U. and Tissen, A. (1992) ‘Integrating Natural
Language Components into Graphical Discourse’, Proceedings of the third conference
on Applied natural language processing, pp.72-79
Feiner, S. K and McKeown, K. R. (1991) ‘Automating the Generation of
Coordinated Multimedia Explanations’, Computer, 24 [10], pp.33-41

Mackinlay, J. (1987) ‘Automating the Design of Graphical Presentations of
Relational Information’, ACM Transactions on Graphics, 8 [2], pp.1 10-141
Mann, W. C. and Thompson, S. A. (1986) ‘Relational propositions in
discourse’, Discourse Processes, 9 [1], pp.57-90
Mann, W. C. and Thompson, S. A. (1987) ‘Rhetorical Structure Theory: A
Theory of Text Organisation”, Technical Report ISI/RS-87-190, Information
Sciences Institute, University of Southern California, Marina del Rey
Mann, W. C. and Thompson, S. A. (1988) ‘Rhetorical structure theory:
Towards a functional theory of text organisation’, Text, 8 [3], pp.243-281
Matsushita, M., Maeda, E. and Kato, T. (2003) ‘An interactive visualisation method
of numerical data based on natural language requirements’, International Journal
of Human-Computer Studies, 60, pp.469-488
Milosavljevic, M., Tulloch, A. and Dale, R. (1996), ‘Text Generation in a Dynamic
Hypertext Environment’, Proceedings of the 19th
Australasian Computer Science
Convention
Nunberg, G. (1990) ‘The Linguistics of Punctuation’, CSLI Lecture Notes, 18,
CSLI, Stanford, California
Petre, M., (1995) ‘Why Looking Isn’t Always Seeing: Readership Skills and
Graphical Programming’, Communications of the ACM, 38 [6], pp.33-44
Power, R. (2000) ‘Planning texts by constraint satisfaction’, Proceedings of the
18th
conference on Computational linguistics – Volume 2, pp.642-648
Power, R., Scott, D. and Bouayad-Agha, N. (2003) ‘Document Structure’,
Computational Linguistics, 29 [2], pp.21 1-260
Reiter, E. and Dale, R. (eds) (2000) Building Natural Language Generation
Systems, pp.30-38, Cambridge, Cambridge University Press
Reiter, E. Somayajulu, S. and Williams, S. (2003) ‘Acquiring and Using Limited
User Models in NLG’, Proceedings of the 2003 European Workshop on Natural
Language Generation, pp. 87-94
Robson, C (1983) quoted in Preece, J and Keller, L. (1990) Human-Computer
Interaction, Hemel Hempstead, Prentice Hall
Rowntree, D. (ed) (2000) Statistics Without Tears An Introduction for Non-
Mathematicians, pp.19-101, London, Penguin Books
Rumsey, D. (ed) (2003) Statistics for Dummies, pp. 231-236, New Jersey, Wiley
Publishing Inc.
Sharp, J. A., Peters, J. and Howard, K. (eds) (2002) The Management of a
Student Research Project, pp.13-16, Aldershot, Gower Publishing Limited
Van Dijk, T, A. and Kintsch, W (1983), ‘Strategies of Discourse
Comprehension’, New York, Academic Press

Wahlster, W., Andre, E., Finkler, W., Profitlich, H and Rist, T. ‘Plan-based integration
of natural language and graphics generation’, Artificial Intelligence, 63, pp.387-427
Wille, R. (1982) (Rival, I ed), ‘Restructuring lattice theory: an approach based
on hierarchies of concept’, Ordered Sets, pp.445-470, Dordecht/Boston, Reidel

Bibliography
Brown, D. (2007) Communicating Design. Developing Web Site Documentation for
Design and Planning, Berkley, CA, New Riders
Callihan, S. (2002) Cascading Style Sheets (CSS) By Example, Indianapolis, Que
Clark, P., A. (1972) Action Research and Organisational Change, New York,
Harper and Row
Covington, M., A. (1994) Natural Language Processing For Prolog Programmers,
New Jersey, Pearson Education
Jurafsky, D. and Martin, J., H. (2008) Speech and Language Processing An
Introduction to Natural Language Processing, Computational Linguistics, and
Speech Recognition, New Jersey, Pearson Education
Krug, S. (2006) Don’t Make Me Think! A Common Sense Approach to Web
Usability, Berkley, CA, New Riders
Lie, H., W. and Bos, B. (1999) Cascading Style Sheets Designing for the Web, Essex,
Pearson Education
Meloni, J., C. (2003) SAMS Teach Yourself PHP, MySQL and Apache All in One,
USA, Sams Publishing
Morville, P. and Rosenfeld, L. (2006) Information Architecture for the World Wide
Web, Sebastopol, O’Reilly
Nugues, P., M. (2006) An Introduction to Language Processing with Perl and
Prolog, Berlin, Springer
Taboada, M. and Mann, W.,C. (2006) ‘Applications of Rhetorical Structure
Theory’, Discourse Studies, 8 [4], pp.567-588

Index
A
abstract document structure..................................................................................................12, 32, 33, 35, 53, 57, 72
Action research.........................................................................................................................................................41
adaptive hypermedia ................................................................................................................................................25
ADVFN ......................ii, 40, 45, 46, 53, 54, 55, 56, 57, 58, 59, 63, 64, 65, 67, 68, 80, 89, 90, 92, 93, 105, 106, 107
artificial intelligence...................................................................................................................................9, 13, 16, 21
C
Cascading Style Sheets ......................................................................................................................................13, 35
case study ...........................................................................................................................................................38, 97
causal factors............................................................................................................................................................39
coefficient of skewness ..............................................................................................................74, 76, 79, 86, 87, 88
coherency ......................................................................................................................viii, 14, 16, 18, 40, 72, 82, 96
COMET..............................................................................................................................................................21, 23
commercial data ..................................................................viii, 10, 11, 14, 15, 34, 39, 40, 41, 44, 89, 91, 95, 96, 97
composition algebra ...........................................................................................................................................16, 34
constraint satisfaction.................................................................................................14, 15, 21, 28, 43, 57, 95, 96, 99
Content Planner .......................................................................................................................................................22
Coordinated Multimedia Explanation Testbed ........................................................................................ See COMET
corpus texts...............................................................................................................................................................62
CSS.....................................................................................................................................................................35, 97
customisation...........................................................................................................23, 24, 35, 39, 44, 82, 95, 97, 107
D
dependant variables............................................................................................................................................38, 42
dependency lattices.................................................................................................................................................26
discourse connectives.............................................................................................................................28, 30, 95, 96
distribution symmetry ........................................................................................................................................70, 76
domain experts ...............................................................................................................38, 39, 43, 51, 52, 71, 87, 88
E
effectiveness..................................viii, 10, 11, 13, 14, 15, 16, 17, 22, 23, 33, 34, 38, 39, 41, 42, 43, 44, 72, 96, 107
Elaboration .................................................................................................................................................18, 19, 59
Ethnographic studies ................................................................................................................................................40
Experiments..................................................................................................................................................36, 39, 42
expressiveness........................................................................................................................................10, 13, 17, 22
F
Formal Concept Analysis..........................................................................................................................................26
FTSE100 ........................................................................................28, 46, 47, 55, 57, 59, 60, 61, 62, 65, 75, 88, 107
G
graphics ......................................................................................................10, 13, 16, 17, 19, 21, 22, 23, 24, 26, 100
Graphics Generator ..................................................................................................................................................22
Grounded Theory ...............................................................................................................................................36, 41
H
handwritten reports.............................................................................................14, 44, 58, 60, 62, 79, 84, 92, 93, 95
HTML...................................................................................................................................13, 21, 27, 35, 47, 53, 97
hypertext ......................................................................................................................................................13, 21, 24
I
ICONOCLAST ................................................................................................................................28, 29, 30, 32, 33
independent variables...................................................................................................................................38, 42, 44

K
knowledge bases.................................................................................................................................................13, 15
KOMET-PAVE ........................................................................................................................................................26
L
LaTeX.....................................................................................................................................................13, 21, 27, 35
lexicons ..............................................................................................................................................................57, 62
linearization rules...............................................................................................................................................20, 21
London Stock Exchange ................................................................................................................................See LSE
London Stock Market..................................................................................................................................... See LSE
LSE...........................................................................................11, 14, 15, 44, 45, 53, 55, 57, 60, 61, 91, 93, 96, 107
M
market indexes .........................................................................................................................................................46
mark-up languages .............................................................................................................................................21, 35
Media Coordinator...............................................................................................................................................22, 23
mixed method strategy .......................................................................................................................................41, 42
multimodal .....................................................................................................................10, 16, 23, 24, 26, 31, 35, 98
MySQL.....................................................................................................................................................................45
N
natural language .................................................................................................9, 10, 16, 24, 32, 96, 97, 98, 99, 100
Natural Language Generation ................................................................................................................... See NLG
Natural Language Processing.........................................................................................................................See NLP
NLG .viii, 9, 10, 12, 13, 14, 15, 16, 17, 19, 24, 25, 26, 27, 28, 33, 34, 38, 40, 41, 43, 44, 47, 48, 57, 58, 72, 73, 74,
78, 79, 80, 84, 92, 93, 95, 96, 99
NLP ......................................................................................................................................................................9, 98
normal distribution.................................................................................74, 75, 76, 77, 79, 82, 84, 86, 87, 88, 90, 93
P
pathways...................................................................................................................................................................25
PEBA-II .......................................................................................................................................................13, 24, 25
Phenomenology..................................................................................................................................................36, 40
PHP ....................................................................................................................................................................47, 97
P-mean....................................................................................................................................................76, 77, 84, 91
population............................................................................viii, 37, 38, 39, 72, 76, 77, 78, 81, 84, 86, 88, 91, 92, 97
pouring rules.............................................................................................................................................................20
Presentation Planner.................................................................................................................................................23
Problem Domain ........................................................................................................................................................9
prototype ii, viii, 11, 14, 15, 16, 17, 21, 23, 26, 27, 34, 40, 43, 44, 45, 47, 48, 50, 53, 57, 58, 59, 61, 62, 71, 72, 73,
81, 85, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 107
psychological experiments.......................................................................................................................................42
Q
questionnaire ..........................................................................................ii, 38, 39, 40, 42, 43, 48, 49, 61, 72, 89, 107
R
random sample .........................................................................................................................................................37
recession...................................................................................................................................................................60
rhetorical structure.......................................................................................12, 27, 28, 29, 30, 32, 33, 34, 35, 55, 59
Rhetorical Structure Theory...........................................................................................................................See RST
RST ......................................................................................................................................12, 18, 19, 29, 34, 55, 59
S
sampling .............................................................................................................................................................37, 97
SICStus Prolog...................................................................................................................................................33, 47
skewed distributions.....................................................................................................................................37, 74, 77
S-mean..........................................................................................................................................................76, 77, 84
standard deviation ......................................................................................................................76, 77, 81, 88, 90, 92
standard error .........................................................................................................................................77, 81, 84, 88

surface realisation.....................................................................................................................................................27
survey ............................................viii, 22, 33, 34, 35, 38, 39, 40, 41, 43, 51, 71, 72, 82, 85, 89, 93, 94, 96, 97, 107
T
target text corpus....................................................................................................................................19, 34, 62, 64
Text Generator .........................................................................................................................................................22
text structure.............................................................................................................19, 22, 28, 29, 30, 32, 43, 72, 98
U
UK economy ............................................................................................................................................................60
user context ......................................................................................................................................10, 11, 22, 25, 26
user preferences.....................................................................................viii, 11, 14, 15, 17, 34, 41, 42, 72, 82, 83, 96
V
vocabulary.................................................................................viii, 9, 14, 61, 62, 64, 65, 66, 68, 69, 89, 90, 95, 107
W
waypoints .................................................................................................................................................................25
WIP ..........................................................................................................................................................................23
workflow ..................................................................................................................................................................48
www.ianwhitby.co.uk ..............................................................................................................................................47

Appendix A ADVFN Reports and Data
Daily world financial news from Thomson Financial News Supplied by
advfn.com
London 17 Sep 2008
17:41 :05
LONDON - Britain's benchmark share index lost 2.3 percent on
Wednesday to hit its lowest close since mid 2005, as investors dumped
financial stocks fearing more chaos from a credit crisis that has roiled Wall
Street banks.
Miners chased weak metal prices and HBOS ended 19 percent lower as it
entered merger talks with Lloyds TSB.
The FTSE 100 .FTSE closed down 113.2 points at 4,912.4 having touched
a low of 4,903.3 and a high of 5,124.4 in volatile trade.
The UK's blue-chip index closed below the psychological 5,000 mark for the
first time since June 2005.
"Chaotic ... the volatility in banks has made sensible decision making
impossible. Never seen a day like it in nearly 25 years up here," said one
London trader.
The index has fallen for three successive sessions and is down 9.3
percent for the week, after U.S. investment bank Lehman Brothers filed
for bankruptcy protection, sending global equity markets plummeting.
U.S. stocks tumbled as a spike in interbank lending rates added to worry
about the global financial system and the U.S. rescue of American
International Group failed to reassure investors.
Back in the UK, British bank Lloyds TSB is in advanced talks to buy
domestic rival HBOS, HBOS said, to create a 28 billion pound ($50 billion)
mortgage giant.
Lloyds TSB was flat, while HBOS sank. HSBC and Royal Bank of Scotland
lost 4.6 and 10.4 percent respectively.
"The FTSE 100 has been understandably volatile over the last two days
as investors try to make sense of the major events in the financial sector,"
said Anthony Grech, a strategist at IG Index.
"With the demise of Lehman Brothers and HBOS now performing
emergency manoeuvres, the big question is where confidence will come
from to not only steady the markets but propel any upward movements in
the near term."

In economic news, the Bank of England (BoE) extended the drawdown period
for its Special Liquidity Scheme to Jan 30, 2009 because of the latest financial
market volatility.
Weaker metal prices weighed on mining companies. BHP Billiton, Rio Tinto,
Vedanta Resources and Kazakhmys fell between 1 and 10.8 percent.
ADVFN Share Report Example
2008-09-17,AAIF,99.5,99.5,99.5,99.5,379089
2008-09-17,AAIW,15.25,15.5,15.25,15.25,0
2008-09-17,AAL,2326,2347,2029,2049,17285869
2008-09-17,AAM,71.5,71.5,71.5,71.5,317360
2008-09-17,AAS,245,247.5,243,245,130400
2008-09-17,AAT,43,44,40,40,54086
2008-09-17,AAU,3.25,3.25,3.125,3.25,0
2008-09-17,AAV,60,60,60,60,127390
2008-09-17,AAZ,0,0,0,0,3336
2008-09-17,ABA,49.25,49.25,49.25,49.25,450
2008-09-17,ABAA,191.75,191.75,189.25,189.25,2346
2008-09-17,ABB,65.5,67,65.5,65.5,2988
...
2008-09-17,ZHG,5.25,5.25,4,5,34227
2008-09-17,ZIP,7,7,7,7,0
2008-09-17,ZOO,12,12.5,12,12.5,40500
2008-09-17,ZOX,53.5,53.5,53.5,53.5,19541
2008-09-17,ZRL,4.5,4.75,4.125,4.125,30990
2008-09-17,ZRX,6.75,6.75,6.5,6.75,0
2008-09-17,ZTC,5.5,5.5,4.75,5.5,0
2008-09-17,ZTF,100.5,101.5,100.5,100.5,0
2008-09-17,ZTR,213,213,207.5,207.5,2383
2008-09-17,ZYT,163.5,163.5,162,163.5,0
2008-09-17,ZYZ,0.115,0.125,0.11,0.115,1850000
2008-09-17,ZZL,2.25,2.25,1.75,2.25,0
ADVFN Share Data Example

Appendix B Questionnaire
My Thanks
Thank you for entering into this survey.
Having built a prototype to examine whether:
 Incorporating preferences can improve report effectiveness
 Commercial data can produce sophisticated page content
I really would appreciate your feedback at this point.
Ian Whitby
Before You Start
The questionnaire assumes you used the data and display choices from the preferences page to produce variations in
the final share reports.
There are 20 multiple choice and 3 open questions (optional). It should take 5 minutes to complete.
Questions are arranged into sections, with guidance on the left of each question.
All answers are anonymous and used solely for the purposes of this research.
Your
Experience
These questions
look to provide a
little background to
your share trading
experience.
How many years have you been trading shares?
> 10 years 5 - 10 years 3 - 5 years 1 - 3 years 0 years
How regularly?

experience.
Don't Know
What do you understand by "Going Long"?
Buying futures which commit you to buy the asset at a set price in the future
Selling futures which commit you to sell the asset at a set price in the future
Buying shares as the price of the share falls
Selling shares as the price of the share falls
Don't Know
Styles
The prototype used
4 display styles.
Which, if any, was
most effective in
putting across the
information within
the report?
Which of the 4 display styles did you find most effective?
Small Text Large Text Indented Ring Binder None
Structure
The reports were
divided into a
number of
structural groups
(FTSE100 summary;
top risers; top
fallers; preferred
shares) presented
in a consistent
order.
The structure exists
to give consistency.
It should allow you
to quickly
determine the
grouping of
information on the
page, where data
groups are to be
found and the
relationships
between groups.
How consistent was the structure of this report?
I always knew where to find a report element
I generally knew where to find a report element
I usually knew where to find a report element but with exceptions
I generally did not know where to find a report element
I never knew where to find a report element
How satisfactory did you find the size and position of the text blocks?
The group sizes/positions totally reflected their relative importance
The group sizes/positions generally reflected their relative importance
The group sizes/positions usually reflected their relative importance
The group sizes/positions generally did not reflect their relative importance
The group sizes/positions totally failed to reflect their relative importance
What improvement, if any, would you make to the structure?
Text Groups
The report content
is grouped into the
following
categories:
Report Title
(e.g. "FTSE100")
FTSE100
Summary
Header (e.g.
"London Shares
How well did the grouping work?
Report was completely in the category groups I expected
Report was generally in the category groups I expected
Report was grouped into categories similar to those I expected
Report was generally not in the category groups I expected
Report was not at all in the category groups I expected
How did the grouping compare to the ADVFN reports?
Much clearer grouping of information

Close Stronger")
Text (e.g. "At
close, the
FTSE100 index
was up 106.")
Risers/Fallers
Preference Data
[Optional]
Footer
The Preferences
page allowed
refinement of these
content groups.
Preferred shares
could be included
in the reports, as
could movements
which bucked the
general market
trend (e.g. falling
shares in a rising
market).
Human-authored
reports do not
allow such
customisation.
Did these options
aid your
understanding?
Generally a clearer grouping of information
Comparable grouping of information
Generally a poorer grouping of information
Much poorer grouping of information
Did customisation make the report easier to read?
Yes - Much more focused in providing my information
Yes - More focused to providing my information
Unchanged from manual report
No - Less focused in providing my information
No - Much less focused in providing my information
With more customisation what other content would you like to see?
Page Navigation
The following
factors may have an
influence on
effective page
navigation:
Page structure
Text groupings
Presentation and
display style
This section
examines how the
presentation style
affected your
navigation within
the report.
Did variations in
font
size/weight/colour
or changes in text
alignment change
your ability to
easily locate and
The "Small Text" report style used less variations in font size than the other styles.
How significant were font size variations in identifying the main elements
of the report?
Mildly significant
The "Large Text" report style used bold font to highlight headings.
How significant was font weight in identifying text groups?
Mildly significant
The "Indented" report style used text positioning & indentation to differentiate text groups.
How significant was text positioning in identifying text groups?

identify areas of
the report? Very significant, it greatly affected my ability to recognise text groups
Mildly significant
The "Ring Binder" report style used text colour to differentiate text groups.
How significant was colour in identifying text groups?
Mildly significant

Sentence
Construction
This section
investigates
whether the
computer-
generated reports
were:
of comparable
quality to the
human-authored
counterparts
able to produce
sophisticated texts
hindered by the
limited data sets
available to them
Please Note:
Much of the
content available to
the human author
was not available to
this prototype -
such data has been
highlighted in the
ADVFN reports.
How similar was the vocabulary to that used in the ADVFN reports?
Very similar, the actual words and phrases used were always the same
Similar, the actual words and phrases used were generally the same
Usually the actual words and phrases used were similar but with some exceptions
Dissimilar, the actual words and phrases used were generally not the same
Very dissimilar, the actual words and phrases used were never the same
How similar was the grammar to that used in the ADVFN reports?
Very similar, the sentence structure & word arrangement were always the same
Similar, the sentence structure & word arrangement were generally the same
Usually the sentence structure & word arrangement were similar but with exceptions
Dissimilar, the sentence structure & word arrangement were generally not the same
Very dissimilar, the sentence structure & word arrangement were never the
same
How similar were the sentences to those used in the ADVFN reports?
Very similar, the sentences always showed the same size and complexity
Similar, the sentences generally showed the same size and complexity
Usually the sentences showed the same size and complexity but with exceptions
Dissimilar, the sentences generally did not show the same size and complexity
Very dissimilar, the sentences never showed the same size and complexity
The prototype did not have access to the same data as the human-authored reports.
Data not available to the prototype was highlighted in yellow in the ADVFN reports.
How did this missing information affect the quality of the prototype reports?
Much better, the sophistication of the reports greatly improved
Better, the sophistication of the reports generally improved
Unchanged, the sophistication of the reports was the same as for manual reports
Poorer, the sophistication of the reports generally worsened
Much poorer, the sophistication of the reports appreciably worsened
What other comments do you have on the sentence construction?
Submit

Appendix C Display Styles
Style 1 – Small Text

Style 2 – Large Text

Style 3 – Indented Text

Style 4 – Ring Binder

P2207964_M801_Final

More Related Content

What's hot

Similar to P2207964_M801_Final

P2207964_M801_Final