Aimed at improvement of the Reviewed Event Bulletin (REB) quality Final level of the interactive review hierarchy (independent review) Provides an extended check of bulletin and analysis quality Identifying both consistencies and inconsistencies in the REB Reviewed areas usually include event location anomalies, seriously mislocated events, missing large events, and unqualified or ‘bogus’ events representing incorrect / invalid data associations or incorrect detections, or REB events not meeting event definition criteria. One more implementation of the master-event approach

1. Automatic and interactive spot check of the IDC
bulletins (REB, SEL3) and XSEL using waveform
cross correlation
Ivan Kitov, IDC/SA/SM, ivan.kitov@ctbto.org
Mikhail Rozhkov

2. Outline
• Spot Check Tool
• Automatic cross correlation processing at the IDC
• Master events
• Automatic Spot Check
• Interactive Spot Check
• Spot Check Tool Architecture
• Discussion

3. • Aimed at improvement of the Reviewed Event Bulletin (REB) quality
• Final level of the interactive review hierarchy (independent review)
• Provides an extended check of bulletin and analysis quality
• Identifying both consistencies and inconsistencies in the REB
• Reviewed areas usually include event location anomalies, seriously mislocated events,
missing large events, and unqualified or ‘bogus’ events representing incorrect / invalid data
associations or incorrect detections, or REB events not meeting event definition criteria.
• One more implementation of the master-event approach
Spot Check Tool Background

4. Spot Check Tool Development
• SCT development: to facilitate a work of the Independent Reviewer (IR)
• Underlying methodology: waveform cross correlation
• Modes: Interactive and Automatic
Interactive: under flexible scenario developed by the IR
Automatic: on daily basis, building XSEL (cross correlation standard
event list) bulletins and conducting automatic comparison with SELs
(same day analysis) and REB (upon its completion) bulletins. Inherits
functionality of the Spot Check Prototype.

5. Multichannel
waveform
template
–
high
quality
signal
Continuous multichannel cross correlation. For arrays, only vertical or 3-C channels (e.g., ARCES) are used. For
3-C stations, all three channels can be used.
Searching for similar (repeating) signals in
continuous waveforms
Waveform Cross Correlation

6. Waveform Cross Correlation: Detection
An example of cross correlation detection: DPRK 2006 vs. DPRK 2009. DPRK06 as a template. Station AKASG,
BP filter between 0.8 Hz and 2.0Hz
STA
LTA
Threshold: SNR=STA/LTA >3.0
Cross Correlation Coefficient, CC
Valid signal

7. Waveform Cross Correlation: Detection
Template
Sought signal
Noise
Data vector of 1000 points
generates 1000D space
CC = cos(α)

8. Routine XSEL processing
Master1
Masterj
Master17,000
Raw
data
Detection
list
M
1
Detection
list
M
17000
…
XSEL
M
1
…
XSEL
M
N
Data and external information on control parameters. Regular updates
LA CR
XSEL 1,1 … … XSEL 1,M
… XSEL j,i …
XSEL N,1 XSEL N,i XSEL N,K
XSEL
Spot
Check
XSEL
WCC

9. Routine XSEL processing
WCC detection
(like DFX)
Local Association
(like GA)
Conflict Resolution
(like GA CR)
1. Varying template length
2. Varying # of working
channels (arrays and 3C)
3. Frequency bands: 0.75 Hz to
8 Hz. Choice depends on
station: ARCES vs. CMAR
4. Adaptive detection
(SNRcc=STA/LTA) threshold
for CC traces
5. STA and LTA definition
6. STA and LTA length
7. FK using CC-traces
8. Standard detection and FK
for array stations (e.g., for
screening of CC-detections)
1. Area of master responsibility
2. Spacing between nodes of virtual
event locations
3. Minimum number of associated
stations (e.g., ndef=3)
4. Association window (e.g., 20 s)
5. Origin time residual for defining/
associated arrivals (e.g., ±3 s)
6. Weight of XSEL events > Wmin as
a sum of station probabilities
(from the REB empirical
statistics)
7. Quality of detections SNRcc
8. Quality of detections SNR
9. Residual of relative magnitude at
stations
When two XSEL events from the same of
different masters have detections at
the same station within tolerance
limit (LA tuning parameter), 20 s :
- Compare weights, Wi and Wj, of the
competing XSEL events and save
this detection with the bigger event.
- When Wi and Wj are equal – the
number of associated stations wins
- When both above equal - standard
deviation of the origin time
(scattering) of the competing XSEL
events
- Then remove events with parameters
below EDC

10. Routine XSEL processing
(limited in the number of masters and thuscoverage)
- Selected 17,000 master (historical REB) events distributed over the globe and depth
- Approximately 5000 events are selected by the best cross correlation with neighbours and
have higher priority
- Daily WCC processing in 50 Flinn-Engdahl seismic regions
-Only the best stations are used in basic XSEL processing
- The choice of filters, correlation windows, channels of arrays stations is suboptimal
- Adaptive detection thresholds (for given setting) for each master/station pair
- Local Association within each seismic region
- Conflict Resolution between all created event hypotheses
- Creation of basic automatic XSEL
Spot Check of the automatic XSEL to improve or reject XSEL event hypotheses using
an extended number of master events

11. Spot Check
Spot Check (cross correlation)
is waveform cross correlation processing with
appropriate number of real and/or synthetic master
events focused on specific area, time interval or set of
events

12. There are >600,000 REB events with primary seismic and infrasound stations. All these events can be used as master events in the Spot
Check based on waveform cross correlation. More than 100 new REB events are added every day. Routine processing with correlation of
numerous historical and fresh REB events within the Spot Check framework results in growing dataset with estimates of mutual
similarity between the events in use. This is a crucial learning process for optimization of all processing stages.
Full REB as a sourceof master events >600,000

13. Fine tuning of stationsensitivity: Global Grid
Station sensitivity is currently calculated for
25,000 nodes of the Global Grid using REB
events within 3 degrees and several depth
ranges. New REB events are taken into
account at a daily basis All (REB) master
events have station sensitivities of the closest
node.
Depth range from 0 to 40 km. The size of circles
is proportional to the threshold value. Most of the
REB events with a few associated stations are
fixed to 0 km depth by strict IDC location rules.
The abundance of weak (e.g. many 3-station)
events makes the event weight threshold low.

14. Automatic XSEL Spot Check processing
Selection
of
data
intervals
by
XSEL
arrival
times.
Default
length
20
min
Detection
list
M
1,1
Detection
list
M
K,M
…
XSEL
M
1,1
…
XSEL
M
K,M
Data and external information on control parameters. Regular updates
LA CR
XSEL 1,1,1 … … XSEL 1,1,J
… XSEL …
XSELK,M,1 XSEL… XSELK,M,P
Final
XSEL
REB
WCC
basic
XSEL
(K
events)
Selection of
masters to
each XSEL
event by its
location
Master1
Masterj
MasterM
1st
XSEL
event
Master1
Masterj
MasterN
XSEL
event
K
…

15. Automatic XSEL Spot Check:
Master event selection
Approximately 95% of REB (i.e. potential master) events
are characterized by major semi-axis less than 200 km.
For the SEL3 events with the same evid , 95% of events
have smaj <360 km.
The distribution of REB events over depth demonstrates
large deficit of events with depths between 1 and 60 km
as a result of IDC location rules.
Master selection has to cover the whole range of location uncertainties and allowed depths
Currently, the range from 2 to 8 degrees and the whole depth range are used for the XSEL Spot Check

16. Automatic Spot Check:
REB and master event selection
# orid NV-Score lat, deg lon, deg nass mb depth, km SeisReg evid Smaj, km Smin, km Strike, deg
1 18831217 -5.67 -16.52 -177.2 7 3.623 0 13 18821069 207.9 30.9 146.3
2 18839059 -6.49 25.541 120.65 3 3.379 0 21 18821360 393.1 39.5 113.9
3 18839576 -8.92 -22.31 147.95 6 -999 0 38 18833227 58.1 32.2 81
4 18834982 -0.28 -6.35 154.59 3 3.421 0 15 18822597 215.5 55 128.2
5 18839860 -11.08 -22.67 178.75 4 2.266 650 12 18835127 272.3 109.2 166.7
6 18838974 -8.45 -14.45 167.86 3 3.355 0 14 18835121 354.9 39.3 142.7
7 18839061 -10.23 0.089 120.43 3 3.448 0 23 18835744 230.3 27.7 66.6
8 18835105 -5.80 36.913 55.04 7 2.882 0 29 18835105 55.2 19.8 5.5
9 18835192 -5.60 43.589 -105.4 10 4.085 0 34 18824150 50.6 8.5 149
10 18835251 -4.01 -3.697 150.48 3 3.364 0 15 18824211 189 29.8 123.8
11 18835739 -16.78 -6.603 154.37 4 3.367 0 15 18824620 140.9 39.9 116
12 18836332 -7.94 34.887 27.101 7 3.446 0 30 18836006 52.5 22.5 156.9
12 from 84 REB events (14%) for April 26, 2020 have negative Net-Visa Score, i.e.
contradict the Net-Visa probabilistic model based on the REB historical events.
Possible reason – poor phase association and mis-location

17. Automatic Spot Check:
REB and master event selection
-90
-30
30
90
-180 -120 -60 0 60 120 180
lat,
deg
lon, deg
REB 2020117
84 events
-90
-30
30
90
-180 -120 -60 0 60 120 180
lat,
deg
lon, deg
Master events
1840 historical REB
For all studied REB events, a set of best master (historical REB) events within 3 (varying range) degrees is
selected and the time intervals ±10 min around the measured and theoretical arrival times are processed by
standard XSEL procedure. The number of master events depends on the semi-major axis (location quality) of
a given REB event. The whole depth range for historical REB events is covered.

18. Automatic Spot Check:
REB and master event selection
Examples of good and poor coverage by master events. The Spot Check quality critically depends on the
REB completeness. The master event coverage is getting better with time.
80 from 84 events were confirmed by Spot Check, i.e. XSEL events matching REB events by two phases
(stations) were built.
-10
-8
-6
-4
-2
0
2
4
6
8
10
110 115 120 125 130 135 140
lat,
deg
lon, deg
20
25
30
35
40
45
50
55
60
80 85 90 95 100
lat,
deg
lon, deg

19. Automatic SEL3 Spot Check processing
Selection
of
data
intervals
by
SEL3
arrival
times.
Default
length
20
min
Detection
list
M
1,1
Detection
list
M
K,M
…
XSEL
M
1,1
…
XSEL
M
K,M
Data and external information on control parameters. Regular updates
LA CR
XSEL 1,1,1 … … XSEL 1,1,J
… XSEL …
XSELK,M,1 XSEL… XSELK,M,P
Final
XSEL-SEL3
REB
WCC
SEL3
(K
events)
Selection of 40
to 200 masters
for each SEL3
event by its
location
Master1
Masterj
MasterM
SEL3
event 1
Master1
Masterj
MasterN
SEL3
event K
…
SAVE
TIME

20. SEL3 Automatic Spot Check:
Master event selection
Approximately 90% of SEL3 events with the same evid as
REB events are within 8 degrees from corresponding REB
locations. The origin time difference between SEL3 and REB
events with the same evid varies from -1000 s to + 1000 s,
with 95% between -100 s and 100 s; the depth difference is
between -700 km and +700 km (the largest possible depth),
with 92% between -200 and 200 km.
The master selection has to cover the whole range of distance
and depth differences.

21. Interactive analysis:SEL3 + IDCX arrivals
ST1
ST2
STn
ST3
SEL3 theoretical arrival time
60-120 s
- arrivals associated with one SEL3 event, tresP-wave ±15 s - not associated arrivals
time

22. SEL3 to REB interactive
ST1
ST2
STn
ST3
REB theoretical arrival time
60-120 s
- arrivals associated with one REB event, tresP-wave ±2 s - disassociated arrivals
time
(IDCX detection or manually added)
ST6
Any new REB event may have similar events in the historical REB. Spot Check tries to find such events.
For weak event hypotheses, the probability to have a better counterpart is close to 1.

23. SEL3 to REB: Spot Check
ST1
STn
Theoretical arrival time for the master event
60-120 s
- CC-arrivals associated with one XSEL Spot Check event, tresP-wave ±4 s - SEL3 associated arrivals
time
ST6
ST8
(Found by given master. May be close to IDCX or new)
ST2

24. Feed-back in the prototype SEL3 Automatic Spot Check
IDC
data
base
SEL3
The REB as
the set of
Master
Events
Raw
seismic/
infrasound
data
Cross
correlation
database
(parameters)
Waveform
Cross Correlation,
Detection,
Characterization
Local association (LA)
Conflict resolution (CR)
Automatic Spot Check
Cross Correlation
Bulletin = SEL3-XSEL
Selection of new
master events
SEL/REB
comparison:
tuning SEL3/XSEL
processing
To interactive
Spot Check
MATCHED
EVENTS
From interactive
Spot Check
Current
Optimal Set
of Master
Events
(~17,000)
Merge
With
XSEL

25. Automatic REB Spot Check processing
Selection
of
data
intervals
by
REB
arrival
times.
Default
length
20
min
Detection
list
M
1,1
Detection
list
M
K,M
…
XSEL
M
1,1
…
XSEL
M
K,M
Data and external information on control parameters. Regular updates
LA CR
XSEL 1,1,1 … … XSEL 1,1,J
… XSEL …
XSELK,M,1 XSEL… XSELK,M,P
Final
XSEL-REB
REB
WCC
REB
(K
events)
Selection of 40
to100 masters for
each SEL3 event
by its location
Master1
Masterj
MasterM
REB
event 1
Master1
Masterj
MasterN
REB
event K
…
SAVE
TIME

26. Spot Check results are used for tuning of preliminary
XSEL automatic processing
• Tuning of CC, detection, LA and CR parameters using the REB
- Tuning of a given master set
- Selection and tuning of better masters
- Machine learning : REB matched (TP) vs. XSEL unmatched (FP) events
• Interactive analysis
- Machine learning : REB-ready (XSEL-TP/REB-FN) vs. non-REB (XSEL-
FP) events from the XSEL not matching any REB event, i.e. potentially new
REB events

27. Current real time (daily/hourly) processing
1. Waveform cross correlation using (current status) ~17,000 selected seismic master events and 4,000
mining events with infrasound (basic XSEL). Seismic and (mining related) infrasound data are processed.
2. Automatic WCC Spot Check of basic XSEL events using an extended set of master events (from the
whole REB) within 4 degrees around the XSEL locations
3. Automatic WCC Spot Check of SEL3 events using an extended set of master events (from the whole
REB) within 8 degrees around the SEL3 locations
4. Automatic WCC Spot Check of REB events using an extended set of master events (from the whole REB)
within 4 degrees around the REB locations
5. Processing based on the DPRK explosions, aftershocks, and mining events
For progressive passive tuning of processing parameters and selection of optimal master events: WCC
Spot Check of historical REB events
1. Pure seismic events
2. Repeating seismic events within continents (mines, NE, active faults, etc.)
3. Seismic/infrasound events (mines)
4. Aftershock sequences
Prototype Automatic Spot Check

28. Prototype Spot Check seismic/infrasoundprocessing
REB events with associated infrasound signals detected at infrasound IMS stations are shown by blue circles. Most active
mines are located in Eurasia, North America, Africa, and Australia. Total number of events for a given mine is from ~10 to
~8,000. Infrasound stations are from ~100 km to ~2,000 km. There are many REB events with 1 seismic and 2 infrasound
phases associated. Infrasound waveform templates are processed as for seismic arrays. Historical data are used to tune
processing: template length, frequency bands, detection thresholds, origin time residuals, etc.

29. Another Automatic SC process: DPRK monitoring
DPRK explosions as master events: relative location
Event date Origin time REB location IDC mb
RM
DPRK1 09.10.2006 01:35:27.57 41.3120N, 129.0190E 4.08 3.93
DPRK2 25.05.2009 00:54:42.80 41.3110N, 129.0460E 4.51 4.54
DPRK3 12.02.2013 02:57:50.80 41.3010N, 129.0650E 4.91 4.96
DPRK4 06.01.2016 01:30:00.49 41.3040N, 129.0480E 4.82 4.85
DPRK5 09.09.2016 00:30:00.87 41.2990N, 129.0490E 5.09 (ref)
Aftershock: detection and phase association
Sta Dist,
km
EvStaAz,
deg
Pha
se
Arrival time tres, s CC dRM SNRCC
USRK 410 35.8 Pn 01:51:46.46 0. 1 0.30 -2.61 3.9
KSRS 440 193.6 Pn 01:51:52.16 -0.1 0.21 -2.78 4.3
Low-magnitude events must be very
close to master events
USRK
KSRS

30. Interactive Analysis: new REB-ready events
October 5, 2011 57.9526ºN 32.5197ºW,
origin time 23:02:10. REB - 36 aftershocks
The main shock
Date Time mb(IDC) ndef Dist, km
05/10/2011 23:06:39 3.32 4 49.52
05/10/2011 23:51:58 3.31 5 28.16
05/10/2011 23:54:20 3.80 6 38.17
05/10/2011 23:55:51 3.65 8 13.8
06/10/2011 00:01:38 4.02 10 5.36
06/10/2011 00:02:31 3.59 10 4.79
06/10/2011 00:05:23 3.56 7 35.97
06/10/2011 00:10:45 3.46 3 76.62
06/10/2011 00:39:18 3.42 4 36.28
06/10/2011 00:45:54 3.52 8 9.17
06/10/2011 01:07:19 3.46 8 39.36
06/10/2011 02:31:30 3.37 5 31.8
06/10/2011 02:34:16 3.35 6 61.11
06/10/2011 07:36:33 3.32 5 26.43
06/10/2011 07:47:04 3.59 6 56.75
06/10/2011 07:48:25 3.54 9 24.72
06/10/2011 10:37:51 3.61 5 27.02
06/10/2011 13:17:19 3.54 4 11.75
06/10/2011 14:12:58 3.38 6 8.48
06/10/2011 14:17:29 3.51 8 15.83
06/10/2011 19:43:05 3.40 5 30.34
06/10/2011 19:47:26 3.61 5 15.18
06/10/2011 19:57:48 3.56 6 19.81
06/10/2011 21:12:34 3.51 8 32.3
06/10/2011 21:15:49 3.46 5 18.9
06/10/2011 23:49:25 3.62 9 25.47
New REB-compatible events
26 new REB-ready events were added to 38 REB
events including the main shock. The Reviewed
Event Bulletin has been extended by 67%:

31. Interactive Analysis: new REB-ready events
The main shock and master events New REB-compatible events
-2
0
2
4
6
8
86 88 90 92 94 96
lat,
deg
lon, deg
Distribution of REB events (black circles) related
to the Sumatera earthquake (green square)
between April 11 and May 24, 2012. Sixteen
master events are shown by red diamonds. They
were preselected using SEL3 available on April
13 and slightly reviewed by an experienced
analyst. The events at the far periphery of the
aftershock zone are likely related to low location
accuracy for 3 to 5 station events and probable
mis-timing /mis-association of low-amplitude
arrivals.
Date Origin
time
Lat,
deg
Lon,
deg
Depth,
km
nass ndef mb
04/11/2012 8:51:33 2.44 92.59 0 7 7 4.95
04/11/2012 9:32:02 2.20 93.09 0 8 7 4.29
04/11/2012 10:05:05 -0.26 92.59 0 7 7 4.06
04/11/2012 13:42:53 2.10 93.62 0 17 17 4.56
04/11/2012 16:25:48 3.31 92.71 0 7 7 3.88
Overall, we have reviewed 409 XSEL hypotheses built by sixteen
master events and found 119 new REB events. Assuming homogeneous
distribution of aftershocks over space and similar performance of all
masters, one can estimate the number of new REB-ready events in the
sequence as ~650. With all uncertainty in the aftershock distributions
and the imperfectness of master event choice and coverage, the number
of missed events is likely compared with the number of REB events. It
is highly important that these events are smaller, and thus, are most
challenging for monitoring CTBT. It should be also mentioned that the
reviewed XSEL events are most difficult for interactive analysis since
they are characterized by weaker signals and smaller number of
stations. Most of hypotheses associated with the REB aftershocks have
four and more defining stations from the full set of seven. Therefore,
the REB events are much easier to review interactively.

32. Tuning of Prototype Automatic Processing
Automatic Spot
Check (preliminary
XSEL + XSEL-SEL3 +
XSEL-REB)
Processing
REB
Rules of
comparison
Interactive
Analysis of
NEW to REB
REB
NOT
MATCHED
REB
MATCHED
Rules of
comparison
EXTENDED (WCC)
SPOT
CHECK
(iterative)
Interactive
Analysis of
NEW to REB
REB
NOT
MATCHED
REB
MATCHED
Machine learning,
processing tuning, and
adaptive mechanisms
INTERACTIVE
(ALSO WITH WCC)
SPOT CHECK
Machine learning,
processing tuning, and
adaptive mechanisms
Passive tuning
Active tuning

33. Spot Check Tool Architecture
• Information backbone: historical IDC database, allows for testing current event
hypotheses with the archived event bulletins using master event approach
• SCT frontend: Liferay IDC portal; allows access to the tool to the IR, lead
analysts and wider (authorized) CTBT expert community
• SCT computational backend: distributed, including servers and workstations.
To migrate to the GPU-server in future
Spot Check Tool Development

34. Spot Check Tool Additional Features
• National event check (in a manner of national event screening) tool
• Not just an Independent Reviewer tool but an Expert Technical Analysis
(ETA) or Special Study (SS) tool as well.
• A developed infrastructure can be utilized by another ETA/SS tools
considering the Liferay as a user interface provider. In other words,
current development provides the encapsulation framework for ETA/SS
methods.
• A State Requested Method Reports (SRMR) of the ETA can also be
generated using this approach if the NDC-supplied methods are hosted
within given frameworks.
Spot Check Tool Development

35. What is SCT front-end?
• SCT front-end is Liferay.
• Liferay is and Enterprise Portal which provides a development framework for new applications or
customization (in given case it provides a framework for the ETA/SS applications). Liferay Portal is
developed using an open source methodology.
• Liferay Portal works in four key areas:
 Web Content Management
 User collaboration
 Development Platform
 Customization
 IDC Secure Web Portal is built with Liferay
Spot Check Tool Development

36. SCT User Interface
Event selection
Station selection
Setting up
detection
parameters
Master event
selection
Comparison folder,
XSEL vs. REB, XSEL
vs. SEL
Setting up filter
parameters
Spot Check Tool Development

37. Spot Check Tool Development
SCT Result Screen
SCT Nearest
Master
SCT Slave
REB event

38. • Given architecture hosts a Spot Check Tool back-end and provide a flexible User
Interface for configuring automatic and interactive SCT and output results according to
the IDC formats and standards
• A tool for the Review Officer for in depth analysis of historical (REB) and real-time
(SEL1-SEL3) events
• A tool for the Expert Technical Analysis and UEB production
• Adaptive mechanisms widely used in the SCT allow for continuous incorporation of
new information (e.g. machine learning based on valid/invalid event classification by
analysts) into interactive and automatic processing also with daily increasing number of
master events
• A matched filter detector (based on waveform cross correlation) provides higher
sensitivity and resolution with a competitive false alert rate
Discussion

39. Liferay – general scheme
Portal – provides general services like authentication,
themes, email, users collaboration etc.
Portlet container - runtime environment for portlets using
the JSR 286 Portlet specification, in which portlets are
instantiated, used, and finally destroyed.
Portlet - Web-based component that will process requests
and generate dynamic content. The end-user would
essentially see a portlet as being a specialized content area
within a Web page that occupies a small window in the
portal page. Depending on the content nature of the Web
site providing the portlet you could use this area to receive
different types of information. The portlet provides users
with the capability to customize the content, appearance
and position of the portlet.
Service Builder – Liferay built-in Object-Relational Mapping
System

40. Data Flow diagram shows system from
point of required computing nodes,
protocols between them and physical
connections.
Tomcat – Java servlet container used
to run Liferay itself and other
components, like web services.
Several computing nodes can be used
to balance load. Main portlet is
responsible for preparing tasks for
them and assembling results.
Servlet - a small Java program that runs within a Web
server, handle clients' requests and return
a customized or dynamic response for each request,
usually across HTTP.
SOAP (Simple Object Access Protocol) is a
messaging protocol that allows programs that run on
disparate operating to communicate using HTTP and
XML.
SCT Data Flow diagram
Spot Check Tool Development

41. XSEL For Quarry Blasts
XSEL
For
Earthquakes