1.
Which transient, when?
A utility function for transient follow-up scheduling
Tim Staley
(4 Pi Sky group)
Southampton Wednesday Seminar, May 2015
WWW: 4pisky.org , timstaley.co.uk/talks

2.
Automate all the things Forecasting transients Decision theory Future work
Outline
Automate all the things
Forecasting transients
Decision theory
Future work

3.
Automate all the things Forecasting transients Decision theory Future work
LSST predicted transient rates
(LSST science book, 2009)
Orphan GRBS: ∼ 1000 per year.
“The reader should be cautioned that many of these
rates are very rough.”
Guesstimates: 104 – 106 alerts per night
(depending on your definition).

4.
Automate all the things Forecasting transients Decision theory Future work
Transient rates today
Notices / events in a 30 day period around April 2015:
GCN: 106 (perhaps 10 new events)
ATEL: 146 (probably similar ratio of new / follow-up)
GAIA: 17 (public)
PTF: 299 (internal)
CRTS: 81(CSS) + 105 (MLS) (automatic detections)
Just counting initial events: ∼500 / month, or ∼17 /day

5.
Automate all the things Forecasting transients Decision theory Future work
Robotic follow-up facilities
Slow rise of automated sites / networks, e.g.
LCOGT: 2 × 2m + 9 × 1m telescopes, over six sites.
(http://lcogt.net/observatory)

6.
Automate all the things Forecasting transients Decision theory Future work
The (wider) problem
How do we begin to ‘close the loop’ of
observe =⇒ analyse =⇒ observe?

7.
Automate all the things Forecasting transients Decision theory Future work
The (wider) problem
How do we begin to ‘close the loop’ of
observe =⇒ analyse =⇒ observe?
Transient
discovery
Observation
prioritization
Classification
estimates
Interesting?
Yes
Schedule
optimization
External
alerts
Survey
data
Telescope
agents
Follow-up
data

8.
Automate all the things Forecasting transients Decision theory Future work
The (wider) problem
How do we begin to ‘close the loop’ of
observe =⇒ analyse =⇒ observe?
Transient
discovery
Observation
prioritization
Classification
estimates
Interesting?
Yes
Schedule
optimization
External
alerts
Survey
data
Telescope
agents
Follow-up
data
Science!

9.
Automate all the things Forecasting transients Decision theory Future work
Diversion on DG-CVn superflare
Fender 2014, http://adsabs.harvard.edu/abs/2014arXiv1410.1545F
Osten et al (in prep)

10.
Automate all the things Forecasting transients Decision theory Future work
Missing pieces
Transient
discovery
Observation
prioritization
Classification
estimates
Interesting?
Yes
Schedule
optimization
External
alerts
Survey
data
Telescope
agents
Follow-up
data
We’ve found a potentially interesting new transient.
Looks like it could be one of class A, B, or C.
What now?

11.
Automate all the things Forecasting transients Decision theory Future work
We found a transient!
What now?
Two implicit goals of follow-up observation:
Improving classification
(Let’s see if it’s really class A.)
Further observation
(Tell me more! / Boring!)

12.
Automate all the things Forecasting transients Decision theory Future work
We found a transient!
What now?
Two implicit goals of follow-up observation:
Improving classification
(Let’s see if it’s really class A.)
Further observation
(Tell me more! / Boring!)
. . . I’ll mainly be talking about the former.

13.
Automate all the things Forecasting transients Decision theory Future work
Outline
Automate all the things
Forecasting transients
Decision theory
Future work

14.
Automate all the things Forecasting transients Decision theory Future work
Working with tiny data
We’ve found a transient. But, very few datapoints:
−10 −5 0 5 10 15 20 25 30
Time
0
2
4
6
8
10
12
Flux
Detection threshold
Data

15.
Automate all the things Forecasting transients Decision theory Future work
The set-up
How do we predict the possible futures for a given
transient?
For now, make some simplifying assumptions:
Parametric lightcurve models for each class of
transient.
Each transient class has a known prior distribution
over the morphological parameters, and this is
multivariate Normal.

16.
Automate all the things Forecasting transients Decision theory Future work
Assumption: Parametric models
Deterministic, finite number of parameters e.g.
y = ƒ(t, t0, , τrse, τdecy)
−40 −20 0 20 40 60 80 100
Time
0
2
4
6
8
10
Flux

17.
Automate all the things Forecasting transients Decision theory Future work
Result: Line of best fit
(Maximum likelihood)
−10 −5 0 5 10 15 20 25 30
Time
0
2
4
6
8
10
12
Flux
ML fit
Detection threshold
Data

18.
Automate all the things Forecasting transients Decision theory Future work
Assumption: Multivar-Normal priors
Known priors, normally distributed, covariant
234
rise_tau
8
16
24
32
a
6121824
decay_tau
2
3
4
rise_tau
6
12
18
24
decay_tau

19.
Automate all the things Forecasting transients Decision theory Future work
Construct: Model lightcurve ensembles
−20 0 20 40 60 80
Time
0
2
4
6
8
10
12
14
16
18
Flux

20.
Automate all the things Forecasting transients Decision theory Future work
Result: MAP fit
−10 −5 0 5 10 15 20 25 30
Time
0
2
4
6
8
10
12
Flux
ML fit
MAP fit
Detection threshold
Data

21.
Automate all the things Forecasting transients Decision theory Future work
But...
−10 −5 0 5 10 15 20 25 30
Time
0
2
4
6
8
10
12
Flux
ML fit
MAP fit
True
Detection threshold
Data

22.
Automate all the things Forecasting transients Decision theory Future work
Constrained parameter distributions
Take our two datapoints, run some MCMC fitting...
3.0
3.6
4.2
rise_tau
12151821
decay_tau
10
15
20
25
30
a
04812
t0
3.0
3.6
4.2
rise_tau
12
15
18
21
decay_tau
0
4
8
12
t0

23.
Automate all the things Forecasting transients Decision theory Future work
Constrained lightcurve ensembles
−30 −20 −10 0 10 20 30 40 50
Time
0
2
4
6
8
10
12
14
16
18
Flux
True
Observations
Comparison, 2 datapoints

24.
Automate all the things Forecasting transients Decision theory Future work
Constrained lightcurve ensembles
−30 −20 −10 0 10 20 30 40 50
Time
0
5
10
15
Flux
True
Forecast epoch
Observations
0.00 0.05 0.10 0.15
Prob.
0
5
10
15
Comparison, 2 datapoints

25.
Automate all the things Forecasting transients Decision theory Future work
Constrained lightcurve ensembles
−30 −20 −10 0 10 20 30 40 50
Time
0
2
4
6
8
10
12
14
16
Flux
True
Forecast epoch
Observations
0.0 0.2 0.4 0.6
Prob.
0
2
4
6
8
10
12
14
16
Comparison, 1 datapoints

26.
Automate all the things Forecasting transients Decision theory Future work
Constrained lightcurve ensembles
−30 −20 −10 0 10 20 30 40 50
Time
0
5
10
15
Flux
True
Forecast epoch
Observations
0.00 0.05 0.10 0.15
Prob.
0
5
10
15
Comparison, 2 datapoints

27.
Automate all the things Forecasting transients Decision theory Future work
Constrained lightcurve ensembles
−30 −20 −10 0 10 20 30 40 50
Time
0
2
4
6
8
10
12
Flux
True
Forecast epoch
Observations
0.0 0.2 0.4 0.6
Prob.
0
2
4
6
8
10
12
Comparison, 3 datapoints

28.
Automate all the things Forecasting transients Decision theory Future work
Constrained lightcurve ensembles
−30 −20 −10 0 10 20 30 40 50
Time
0
2
4
6
8
10
12
Flux
True
Forecast epoch
Observations
0.0 0.2 0.4 0.6
Prob.
0
2
4
6
8
10
12
Comparison, 4 datapoints

29.
Automate all the things Forecasting transients Decision theory Future work
Constrained lightcurve ensembles
−30 −20 −10 0 10 20 30 40 50
Time
0
2
4
6
8
10
12
Flux
True
Forecast epoch
Observations
0.00 0.25 0.50 0.75
Prob.
0
2
4
6
8
10
12
Comparison, 6 datapoints

30.
Automate all the things Forecasting transients Decision theory Future work
Comparing model ensembles
−30 −20 −10 0 10 20 30 40 50
Time
0
2
4
6
8
10
12
14
16
18
Flux
Prior ensemble, Type 1

31.
Automate all the things Forecasting transients Decision theory Future work
Comparing model ensembles
−30 −20 −10 0 10 20 30 40 50
Time
0
5
10
15
20
25
30
35
Flux
Prior ensemble, Type 2

32.
Automate all the things Forecasting transients Decision theory Future work
Comparing model ensembles
−30 −20 −10 0 10 20 30 40 50
Time
0
5
10
15
20
25
30
35
40
Flux
Comparison, 0 datapoints

33.
Automate all the things Forecasting transients Decision theory Future work
Comparing model ensembles
−30 −20 −10 0 10 20 30 40 50
Time
0
5
10
15
20
25
30
35
Flux
True
Forecast epoch
Observations
0.00 0.05 0.10 0.15 0.20
Prob.
0
5
10
15
20
25
30
35
Comparison, 2 datapoints

34.
Automate all the things Forecasting transients Decision theory Future work
Comparing model ensembles
−30 −20 −10 0 10 20 30 40 50
Time
0
5
10
15
20
25
30
35
Flux
True
Forecast epoch
Observations
0.0 0.2 0.4 0.6
Prob.
0
5
10
15
20
25
30
35
Comparison, 2 datapoints

35.
Automate all the things Forecasting transients Decision theory Future work
Outline
Automate all the things
Forecasting transients
Decision theory
Future work

36.
Automate all the things Forecasting transients Decision theory Future work
The crux of the problem
(Couldn’t get past the
crux)

37.
Automate all the things Forecasting transients Decision theory Future work
The crux of the problem
(Couldn’t get past the
crux)
Computers are good at
optimisation.
e.g., picking out a good
observing schedule.
But need a way to
assign value.

38.
Automate all the things Forecasting transients Decision theory Future work
−6 −4 −2 0 2 4 6
Epoch
0.0
0.2
0.4
0.6
0.8
1.0
Relativeflux
stable
logistic
null
Intrinsic lightcurves

39.
Automate all the things Forecasting transients Decision theory Future work
−6 −4 −2 0 2 4 6
Epoch
0.0
0.2
0.4
0.6
0.8
1.0
Relativeflux
stable
logistic
null
Intrinsic LC's with noise estimates

40.
Automate all the things Forecasting transients Decision theory Future work
−6 −4 −2 0 2 4 6
Epoch
−0.5
0.0
0.5
1.0
1.5
Relativeflux
Sampling with noise

41.
Automate all the things Forecasting transients Decision theory Future work
−6 −4 −2 0 2 4 6
Epoch
−0.5
0.0
0.5
1.0
1.5
Relativeflux
Sampling with noise

42.
Automate all the things Forecasting transients Decision theory Future work
0 1
−0.5
0.0
0.5
1.0
1.5
Relativeflux
T=-5.0
0 1
T=-4.0
0 1
PDF value
T=-3.0
0 1
T=-2.0
0 1
T=-1.0
0 1
T=0.0
0 1
T=1.0
0 1
T=2.0
stable
logistic
null
Class PDF at each epoch

43.
Automate all the things Forecasting transients Decision theory Future work
PDF value
−0.5
0.0
0.5
1.0
1.5
Relativeflux
T=-5.0 T=-4.0 T=-3.0 T=-2.0 T=-1.0 T=0.0 T=1.0 T=2.0
stable
logistic
null
−5 −4 −3 −2 −1 0 1 2
Epoch
−0.55
−0.50
−0.45
−0.40
−0.35
−0.30
−0.25
−0.20
FoM
Information content
Evaluating each epoch

44.
Automate all the things Forecasting transients Decision theory Future work
The problem with information content
PDF value
−0.5
0.0
0.5
1.0
1.5
Relativeflux
T=-5.0 T=-4.0 T=-3.0 T=-2.0 T=-1.0 T=0.0 T=1.0 T=2.0
stable
logistic
null
−5 −4 −3 −2 −1 0 1 2
Epoch
−0.55
−0.50
−0.45
−0.40
−0.35
−0.30
−0.25
−0.20
FoM
Information content
Evaluating each epoch
Information content weights all classes equally.
What if we’re more interested in identifying one
particular class?

45.
Automate all the things Forecasting transients Decision theory Future work
Introducing: confusion matrices
Common or garden empirical confusion matrix:
(Automatic classification of time-variable X-ray sources;
K. Lo et al, 2014)

46.
Automate all the things Forecasting transients Decision theory Future work
Confusion matrices
Label True class
A B C
ˆA P( ˆA | A ) P( ˆA | B ) P( ˆA | C )
ˆB P( ˆB | A ) P( ˆB | B ) P( ˆC | C )
ˆC P( ˆC | A ) P( ˆC | B ) P( ˆC | C )

47.
Automate all the things Forecasting transients Decision theory Future work
Probabilistic confusion matrices
Example
0.0 0.5 1.0 1.5 2.0 2.5
PDF value
−0.5
0.0
0.5
1.0
1.5
Relativeflux
T=-2
stable
logistic
null
True class
stable logistic null
Label
stable 0.549 0.449 0.002
logistic 0.449 0.541 0.010
null 0.002 0.010 0.988

48.
Automate all the things Forecasting transients Decision theory Future work
Probabilistic confusion matrices
Example
0.0 0.5 1.0 1.5 2.0 2.5
PDF value
−0.5
0.0
0.5
1.0
1.5
Relativeflux
T=-2
stable
logistic
null
True class
stable logistic null
Label
stable 0.549 0.449 0.002
logistic 0.449 0.541 0.010
null 0.002 0.010 0.988
Diagonal entries represent
recall for each class.

49.
Automate all the things Forecasting transients Decision theory Future work
PDF value
−0.5
0.0
0.5
1.0
1.5
Relativeflux
T=-5.0 T=-4.0 T=-3.0 T=-2.0 T=-1.0 T=0.0 T=1.0 T=2.0
stable
logistic
null
−5 −4 −3 −2 −1 0 1 2
Epoch
0.45
0.50
0.55
0.60
0.65
0.70
0.75
0.80
0.85
0.90
FoM
Information content (shifted)
Total Recall
Evaluating each epoch

50.
Automate all the things Forecasting transients Decision theory Future work
PDF value
−0.5
0.0
0.5
1.0
1.5
Relativeflux
T=-5.0 T=-4.0 T=-3.0 T=-2.0 T=-1.0 T=0.0 T=1.0 T=2.0
stable
logistic
null
−5 −4 −3 −2 −1 0 1 2
Epoch
0.5
0.6
0.7
0.8
0.9
1.0
FoM
Total Recall
Null Recall
Evaluating each epoch

51.
Automate all the things Forecasting transients Decision theory Future work
As applied to the previous example . . .
−30 −20 −10 0 10 20 30 40 50
Time
0
5
10
15
20
25
30
35
Flux
True
Forecast epoch
Observations
0.00 0.08 0.16 0.24 0.32
Prob.
0
5
10
15
20
25
30
35
−30 −20 −10 0 10 20 30 40 50
Time
−0.70
−0.65
−0.60
−0.55
−0.50
−0.45
−0.40
ICscore
Comparison, 2 datapoints

52.
Automate all the things Forecasting transients Decision theory Future work
Summary
Data + models + Bayesian analysis =⇒
Ensemble forecasts
Ensemble forecasts + utility function =⇒
Figure-of-merit for evaluating possible actions
Using the figure-of-merit as basis for a decision =
Applied Bayesian decision theory
(AKA active machine learning)

53.
Automate all the things Forecasting transients Decision theory Future work
Outline
Automate all the things
Forecasting transients
Decision theory
Future work

54.
Automate all the things Forecasting transients Decision theory Future work
Scheduler, simulation, testing
Still need to implement a (basic) scheduler using
figure-of-merit as input.
Then: simulate (easy, already have models!)

55.
Automate all the things Forecasting transients Decision theory Future work
Optimization / Automated Planning
Optimizing for ASAP classification
(Future-discounted weighting schemes?)
Non-myopic (better-than-greedy) scheduling.
Multi-armed bandit problem.

56.
Automate all the things Forecasting transients Decision theory Future work
Optimization / Automated Planning
Multi-armed bandit problem
Image credit: Wikipedia/Yamaguchi (CC BY-SA 3.0)

57.
Automate all the things Forecasting transients Decision theory Future work
Model refinement (and basic validity)
Multivariate normal prior — valid?
Sum of multiple multivariate normals?
Non-parametric modelling (Gaussian processes)?

58.
Automate all the things Forecasting transients Decision theory Future work
Culture
Pretty sure we can make this work. . . but there’s a
culture / chicken-and-egg problem.

59.
Automate all the things Forecasting transients Decision theory Future work
Code packages used
astropy.modeling — for the models interface.
Numpy — efficient lightcurve-model calculations.
pandas — (Python ANalysis of DAta Series) for
handling time-series data.
statsmodels — Kernel density estimates.
emcee — for MCMC
(Py)MultiNest — For model-evidence calculations.
Seaborn — Neat plotting tools, aesthetically
pleasing defaults for Matplotlib.

60.
Automate all the things Forecasting transients Decision theory Future work
Fin
Plenty more to do . . . watch this space.