This presentation was prepared for a talk on 2014.08.06 at the NYC Algorithmic Trading meetup (http://www.meetup.com/NYC-Algorithmic-Trading/events/197749772/)
Regardless of whether you call it "data science", "business intelligence", "analytics", "statistics" or just plain old "math", we have many tried and true techniques for dealing with uncertainty (particularly in quantitative finance). But ambiguity—what problem do we need to solve in the first place?—is a separate matter and, at least in my experience, is the hardest part of creating value from data. During this talk, I'll discuss how we address ambiguity by giving a guided tour of some of our client projects, such as how to reduce legal e-discovery costs by 99% (hint: supervised binary classification of text documents) or how to assemble project teams on emerging R&D opportunities in a multinational organization (hint: unsupervised classification of employee expertise).
5. @deanmalmgren | bit.ly/design-data
data scientists thrive with ambiguity
solve for x
x = 5 + 2
projectevolution
A x = b optimize
f(x)
optimize
A x = b
subject to
f(x) > 0
6. @deanmalmgren | bit.ly/design-data
data scientists thrive with ambiguity
solve for x
x = 5 + 2
projectevolution
A x = b optimize
f(x)
optimize
A x = b
subject to
f(x) > 0
optimize
“our profitability”
9. @deanmalmgren | bit.ly/design-data
origins of ambiguity
unclear problems
@deanmalmgren | bit.ly/design-data
identify the best locations to plant new trees
how many?
what kinds of trees?
move old trees?
replace old trees?
10. @deanmalmgren | bit.ly/design-data
origins of ambiguity
unclear problems
identify the best locations to plant new trees
how many?
what kinds of trees?
move old trees?
replace old trees?
aesthetically pleasing?
maximize growth?
increase foliage?
offset CO2 emissions?
@deanmalmgren | bit.ly/design-data
24. @deanmalmgren | bit.ly/design-data@deanmalmgren | bit.ly/design-data
mostly bad ideas, but a few good ones
Lorem Ipsum: a narrative about blankets.
Author: Charlie Brown
Date: 31 Jan 2012
!
Lorem Ipsum is a dummy text used when typesetting or marking up documents. It has a
long history starting from the 1500s and is still used in digital millennium for typesetting
electronic documents, page designs, etc.
!
In itself, the original text of Lorem Ipsum might have been taken from an ancient Latin
book that was written about 50 BC. Nevertheless, Lorem Ipsum’s words have been
changed so they don’t read as a proper text.
!
Naturally, page designs that are made for text documents must contain some text rather
than placeholder dots or something else. However, should they contain proper English
words and sentences almost every reader will deliberately try to interpret it eventually,
missing the design itself.
!
However, a placeholder text must have a natural distribution of letters and punctuation
or otherwise the markup will look strange and unnatural. That’s what Lorem Ipsum helps
to achieve.
!
I would like to thank Peppermint Pattyfor her support on studying
Lorem Ipsum as well as the infinite wisdom of Linus van Peltand his
willingness to use his blanket in my experiments.
informal conversation to stated goals
25. @deanmalmgren | bit.ly/design-data@deanmalmgren | bit.ly/design-data
mostly bad ideas, but a few good ones
Lorem Ipsum: a narrative about blankets.
Author: Charlie Brown
Date: 31 Jan 2012
!
Lorem Ipsum is a dummy text used when typesetting or marking up documents. It has a
long history starting from the 1500s and is still used in digital millennium for typesetting
electronic documents, page designs, etc.
!
In itself, the original text of Lorem Ipsum might have been taken from an ancient Latin
book that was written about 50 BC. Nevertheless, Lorem Ipsum’s words have been
changed so they don’t read as a proper text.
!
Naturally, page designs that are made for text documents must contain some text rather
than placeholder dots or something else. However, should they contain proper English
words and sentences almost every reader will deliberately try to interpret it eventually,
missing the design itself.
!
However, a placeholder text must have a natural distribution of letters and punctuation
or otherwise the markup will look strange and unnatural. That’s what Lorem Ipsum helps
to achieve.
!
I would like to thank Peppermint Pattyfor her support on studying
Lorem Ipsum as well as the infinite wisdom of Linus van Peltand his
willingness to use his blanket in my experiments.
informal conversation to stated goals
52. @deanmalmgren | bit.ly/design-data
motorola
new product
announcement
first versions
from manufacturer
available
in stores
next generation
to manufacturer
product defects
from consumers
data-driven consumer feedback
62. @deanmalmgren | bit.ly/design-data
data-driven e-discovery
daegis
aboutpatent
not
aboutpatent
turn over to plaintiff
don’t
turn over to plaintiff
adverse inference
give away trade secrets
63. @deanmalmgren | bit.ly/design-data
data-driven e-discovery
daegis
aboutpatent
not
aboutpatent
turn over to plaintiff
don’t
turn over to plaintiff
adverse inference
give away trade secrets