In todays world majority of information is generated by self sustaining systems like various kinds of bots, crawlers, servers, various online services, etc. This information is flowing on the axis of time and is generated by these actors under some complex logic. For example, a stream of buy/sell order requests by an Order Gateway in financial world, or a stream of web requests by a monitoring / crawling service in the web world, or may be a hacker's bot sitting on internet and attacking various computers. Although we may not be able to know the motive or intention behind these data sources. But via some unsupervised techniques we can try to infer the pattern or correlate the events based on their multiple occurrences on the axis of time. Associating a chain of events in order of time helps in doing a root event analysis. In certain cases a time ordered correlation and root event identification is good enough to automatically identify signatures of various malicious actors and take appropriate corrective actions to stop cyber attacks, stop malicious social campaigns, etc. Sessionisation is one such unsupervised technique that tries to find the signal in a stream of events associated with a timestamp. In the ideal world it would resolve to finding periods with a mixture of sinusoidal waves. But for the real world this is a much complex activity, as even the systematic events generated by machines over the internet behave in a much erratic manner. So the notion of a period for a signal also changes in the real world. We can no longer associate it with a number, it has to be treated as a random variable, with expected values and associated variance. Hence we need to model "Stochastic periods" and learn their probability distributions in an unsupervised manner. The main focus of this talk will be to showcase applied data science techniques to discover stochastic periods. There are many ways to obtain periods in data, so the journey would begin by a walk through of existing techniques like FFT (Fast Fourier Transform) then discuss about Gaussian Mixture Models. After highlighting the short comings of these techniques we will succinctly explain one of the most general non-parametric Bayesian approaches to solve this problem. Without going too deep in the complex math, we will get back to applied data science and discuss a much simpler technique that can solve the same problem if certain assumptions are satisfied. In this talk we will demonstrate some time based pattern we discovered while working on a security analytics use case that uses Sessionisation. In the talk we will demonstrate such patterns based on an open source malware attack datasets that is available publicly. Key concepts explained in talk: Sessionisation, Bayesian techniques of Machine Learning, Gaussian Mixture Models, Kernel density estimation, FFT, stochastic periods, probabilistic modelling, Bayesian non-parametric methods

1. Sessionisation via Stochastic
periods for root event
identification
Kuldeep Jiwani
ODSC India 2019

2. Thales Overview
From the Bottom of the Oceans… to the Depths of
Space & Cyberspace
Key Digital Technologies

3. Thales: A Research and Development Powerhouse
6 times winner
2012, 2013,
2015, 2016,
2017, 2018
Expertise in a uniquely broad range
of technical domains, from science
to systems, applied across
businesses.
An extensive intellectual property
portfolio of 20,500 patents.
Albert Fert
Scientific director of the
CNRS/Thales joint physics
unit and winner of the
2007 Nobel prize in
physics.

4. Agenda
• Motivation of studying events
• Concept & purpose of Sessionisation
• Traditional approaches
• Real world case studies
• Applied Data Science way of doing Sessionisation

5. Events
• Orders placed in a market
• Sequence of user tweets
• User’s clicks on a website
• Activity update by an IoT device
• Network events on a router
• Network alarms in a network

6. Entities: Information flow
Events stream
Time

7. Time sequenced events
Time
Entity - 1

8. Time sequenced events
Time
Entity - 1
Entity - 2

9. Time sequenced events
Time
Entity - 1
Entity - 2
Entity - 3

10. Time sequenced events
Time
Entity - 1
Entity - 2
Entity - 3

11. Time sequenced events
Time
Entity - 1
Entity - 2
Entity - 3
Entity - 4

12. Sessions: Operations vs Data Science view
Continuity in
activity
Mean activity period
Gap >> Mean activity period
Sessions Sessions Sessions

13. Sessions: Operations vs Data Science view
Continuity in
activity
Chain of time
sequenced events
Mean activity period
Gap >> Mean activity period
Sessions Sessions Sessions
Time based correlation

14. Root event identification from session chain

15. Root event identification from session chain

16. Root event identification from session chain

17. Root event identification from session chain

18. Root event identification from session chain

19. Root event identification from session chain
Stochastic periods

20. Malicious Actors in the world of AI
• Orders placed in a market: Market manipulation
• Sequence of user tweets : Bot campaigns
• User’s clicks on a website: Fraudulent transactions
• Activity by an IoT device: Taking device control
• Network events on a router: Cyber attacks

21. Events stream as impulse train
1
0
Time
Time

22. Approaches for finding time based patterns
• Fourier transform
• Time period – Stochastic periods
• GMM (Gaussian Mixture Models)
• Infinite GMM (Gaussian Mixture
Models)
• Non-parametric Bayesian methods
• Applied data science techniques
Information
Complexity
Applied data
science

23. Fourier transform intuition

24. Fourier series: Quick intro
𝑃 𝑡 =
1
2
𝑎0 + 𝑎1 cos 𝜔𝑡 + 𝑎2 cos 2𝜔𝑡 + … + 𝑏1 sin 𝜔𝑡 + 𝑏2 sin 2𝜔𝑡 + …
𝑷 𝒕 =
𝟏
𝟐
𝒂 𝟎 +
𝒏=𝟏
∞
𝒂 𝒏 𝐜𝐨𝐬 𝒏𝝎𝒕 +
𝒏=𝟏
∞
𝒃 𝒏 𝐬𝐢𝐧 𝒏𝝎𝒕
𝑓 𝑡 → 𝑃(𝑡)

25. Fourier series: Quick intro
𝐸2
=
0
𝑇
(𝑓 𝑡 − 𝑃(𝑡))2
𝑑𝑡
𝜕𝐸2
𝜕𝑎 𝑛
= 0
𝜕𝐸2
𝜕𝑏 𝑛
= 0
RMSE
(Root Mean Square Error)
Minimize RMSE loss
Derivative = 0
𝑎 𝑛 =
2
𝑇 0
𝑇
𝑓(𝑡) cos 𝑛𝜔𝑡 𝑑𝑡 𝑏 𝑛 =
2
𝑇 0
𝑇
𝑓(𝑡) sin 𝑛𝜔𝑡 𝑑𝑡
𝑎0 =
2
𝑇 0
𝑇
𝑓(𝑡) 𝑑𝑡 𝑏0 = 0

26. Fourier transform: Quick intro
Euler’s formula 𝑒 𝑗𝜔𝑡
= cos 𝜔𝑡 + 𝑗 sin 𝜔𝑡
𝑃 𝑡 =
𝑛=−∞
∞
𝑐 𝑛 𝑒 𝑗𝑛𝜔𝑡
𝑐 𝑛 =
1
𝑇 0
𝑇
𝑓(𝑡) 𝑒−𝑗𝑛𝜔𝑡
𝑑𝑡
Fourier Series
𝐹 𝑗𝜔 =
−∞
∞
𝑓(𝑡)𝑒−𝑗𝜔𝑡
𝑑𝑡
Fourier Transform (CTFT)
𝑋 𝑗𝜔 =
𝑛=−∞
∞
𝑥(𝑛)𝑒−𝑗𝑛𝜔
Fourier Transform (DTFT)

27. Fourier Transform (DTFT): Impulse train
FT (Real): Magnitude FT (Imaginary): Phase shift

28. Fourier Transform (DTFT): Impulse train
FT (Real): Magnitude FT (Imaginary): Phase shift
10 1010

29. Plotting Fourier Transform in Python
N = time_signal.shape[0]
signal_fft = numpy.fft.fft(time_signal)
frequency_bins = numpy.fft.fftfreq(N)
fig, ax = plt.subplots(1,2,figsize=(28,7))
ax[0].plot(frequency_bins[1:N/2], np.abs(signal_fft.real[1:N/2]), 'g')
ax[1].plot(frequency_bins[1:N/2], signal_fft.imag[1:N/2], 'c')

30. Case studies via public datasets
• Sessionisation is an essential activity in detecting malicious bot
activities like Beaconing
• We will use 6th dataset of CTU-13 datasets for examples
• Provided by Czech Technical University (CTU)
• Traces captured from a malware attack executed in university network
• 6th dataset simulates a bot named DonBot, it attacks SVC services on Windows
• Dataset: https://mcfp.felk.cvut.cz/publicDatasets/CTU-Malware-Capture-
Botnet-47/bro/conn.log

31. Case – 1: DonBot’s DNS queries to university DNS
server

32. DonBot: DNS traces
Fourier
Transform
Bot’s
DNS event
stream
(Zoomed)

33. DonBot: DNS traces
Fourier
Transform
Bot’s
DNS event
stream
(Zoomed)

34. DonBot: DNS traces
Fourier
Transform
Bot’s
DNS event
stream
(Zoomed)

35. Stochastic periods: Introduction
• Analyze periodicity in time domain
• Compute consecutive time deltas
• Real world signals are noisy so time deltas will vary a lot
• If there is periodicity in the signal, time deltas will vary in a band
• The density plot of time deltas will show some high density regions
• We can learn a probability distributions for each high density region

36. DonBot: DNS traces
Bot’s
DNS event
stream
Time delta
analysis

37. DonBot: DNS traces
MeanPeriod = μ
Periodicity = μ / σ
Bot’s
DNS event
stream
Time delta
analysis

38. DonBot: DNS traces
MeanPeriod = μ
Periodicity = μ / σ
Bot’s
DNS event
stream
Time delta
analysis
Session-2Session-1 Session-3

39. DonBot: DNS traces
MeanPeriod = μ
Periodicity = μ / σ
Time delta
analysis
Time delta
Density
plot

40. DonBot: DNS traces
MeanPeriod = μ
Periodicity = μ / σ
Time delta
analysis
Time delta
Density
plot
PDF: Stochastic period

41. Case – 2: DonBot’s data transfer via backdoor
port (5678) with a malicious IP (91.212.135.158)

42. DonBot: Data transfer via backdoor
DonBot’s
backdoor
traffic
Time delta
analysis

43. DonBot: Data transfer via backdoor
DonBot’s
backdoor
traffic
Time delta
analysis

44. DonBot: Data transfer via backdoor
Zoomed in
FT
without 0th
frequency
Time delta
analysis

45. DonBot: Data transfer via backdoor
Zoomed in
FT
without 0th
frequency
Time delta
analysis

46. DonBot: Data transfer via backdoor
Time delta
analysis
Time delta
Density
plot

47. DonBot: Data transfer via backdoor
Time delta
analysis
Time delta
Density
plot

48. DonBot: Data transfer via backdoor
Stochastic Period - 1
Stochastic Period - 2
Time delta
analysis
Time delta
Density
plot

49. Case – 3: Genuine systematic DNS queries to
university’s DNS server

50. Genuine systematic DNS queries to DNS server
Normal
DNS
queries
Time delta
analysis

51. Genuine systematic DNS queries to DNS server
Normal
DNS
queries
Time delta
analysis
[0.8, 1.6, 2.4, 3.2]

52. Genuine systematic DNS queries to DNS server
FT
without 0th
frequency
Time delta
analysis
FT: Only able to highlight the higher time periods

53. Genuine systematic DNS queries to DNS server
Time delta
analysis
Time delta
Density
plot

54. Genuine systematic DNS queries to DNS server
Time delta
analysis
Time delta
Density
plot
A
B
C
D

55. Case – 4: Just another interesting DNS pattern

56. Normal DNS queries
FT
without 0th
frequency
Time delta
analysis

57. Normal DNS queries
Time delta
analysis
Time delta
Density
plot

58. Auto discovering multiple distributions
tation Maximization
w to estimate parameter ?✓
Expectation Maximization
If sources are known,easy:
How to estimate parameter ?✓GMM - Gaussian Mixture Models

59. Auto discovering multiple distributions
sticmodel of data Gaussian Mixture Model (GMM)
GMM - Gaussian Mixture Models

60. GMM – Gaussian Mixture Models
• Does soft clustering of data points instead of hard clustering
• In principal it is very similar to K-Means but works on
probability
• K-Means: {P1  C1, P2  C2}, GMM: {P1  [0.8, 0.1, 0.1], P2  [0.05, 0.85, 0.1]}
• Problem with GMM & K-Means: We need to define “K”
• Techniques like Elbow method, Silhouette, etc. are based on certain assumptions
• Cannot be applied in general for automated discovery of K
• Finding “K” automatically is a very hard problem to solve
C1, C2, C3 C1, C2, C3

61. Bayesian way of building models
𝑃 𝜃 𝑋 =
𝑃 𝑋 𝜃 𝑃(𝜃)
𝑃(𝑋)
PriorLikelihood
Evidence
Posterior
𝑃 𝜃 𝑋 =
𝑃 𝑋 𝜃 𝑃(𝜃)
𝑃(𝑋)
𝑃(𝜃) is conjugate to 𝑃 𝑋 𝜃
A(𝜈’) A(𝜈)
For example:
P(𝜽) = 𝓝(𝜽|0, 1) # Standard normal
P(X|𝜽) = 𝓝(x|𝜽, 1) # with 1 std. dev
𝑃(𝜃|𝑋) ∝ 𝑒−
1
2(𝑥−𝜃)2
𝑒−
1
2 𝜃2
𝑃(𝜃|𝑋) ∝ 𝑒−(𝜃−
𝑥
2)2
P(𝜽|X) = 𝓝(𝜃|
𝑥
2
,
1
2
)

62. Constructing GMM
Gaussian Mixture Model (GMM)
𝑃 𝑥 𝜃 = 𝜋1 𝒩 𝑥 𝜇1, Σ1 + 𝜋2 𝒩(𝑥|𝜇2, Σ2)+ 𝜋3 𝒩(𝑥|𝜇3, Σ3)
𝜃 = {𝜋1, 𝜋2, 𝜋3, 𝜇1, 𝜇2, 𝜇3, Σ1, Σ2, Σ3} Parametric setting
K = 3

63. Constructing GMM
Gaussian Mixture Model (GMM)
𝑃 𝑥 𝜃 = 𝜋1 𝒩 𝑥 𝜇1, Σ1 + 𝜋2 𝒩(𝑥|𝜇2, Σ2)+ 𝜋3 𝒩(𝑥|𝜇3, Σ3)
𝜃 = {𝜋1, 𝜋2, 𝜋3, 𝜇1, 𝜇2, 𝜇3, Σ1, Σ2, Σ3} Parametric setting
K = 3
t x
𝑝 𝑡 = 𝑐 𝜃 = 𝜋 𝑐 𝑝(𝑥|𝑡 = 𝑐, 𝜃) = 𝒩 𝑥 𝜇 𝑐, Σ 𝑐
t is a latent variable: [1, 2, 3], 𝜋1 + 𝜋2 + 𝜋3 = 1
𝑝 𝑥 𝜃 =
𝑐=1
𝐾
𝑝 𝑥 𝑡 = 𝑐, 𝜃 𝑝 𝑡 = 𝑐 𝜃Likelihood

64. Constructing GMM
Gaussian Mixture Model (GMM)
𝑃 𝑥 𝜃 = 𝜋1 𝒩 𝑥 𝜇1, Σ1 + 𝜋2 𝒩(𝑥|𝜇2, Σ2)+ 𝜋3 𝒩(𝑥|𝜇3, Σ3)
𝜃 = {𝜋1, 𝜋2, 𝜋3, 𝜇1, 𝜇2, 𝜇3, Σ1, Σ2, Σ3} Parametric setting
K = 3
t x
𝑝 𝑡 = 𝑐 𝜃 = 𝜋 𝑐 𝑝(𝑥|𝑡 = 𝑐, 𝜃) = 𝒩 𝑥 𝜇 𝑐, Σ 𝑐
Training GMM: 𝑚𝑎𝑥
𝜃
𝑖=1
𝑁
𝑝(𝑥𝑖|𝜃)
Subject to: 𝜋 𝑘 = 1 ; 𝜋 𝑘 ≥ 0; Σ 𝑘 ≻0
EM algorithm:
• E-step: Compute q(t) dist. over t
• M-step: Update Gaussian params
• To fit points assigned to them
t is a latent variable: [1, 2, 3], 𝜋1 + 𝜋2 + 𝜋3 = 1
𝑝 𝑥 𝜃 =
𝑐=1
𝐾
𝑝 𝑥 𝑡 = 𝑐, 𝜃 𝑝 𝑡 = 𝑐 𝜃Likelihood

65. Sessionisation problem statement

66. Sessionisation: Data Science at scale
• In a real world scenario, be it
• Web users over internet, Network hosts in an enterprise network, etc.
• One would need to apply Sessionisation on millions of entities
• So manual inspection based methods cannot be used
• We need a fully automated system to discover multiple
”Stochastic Periods”
• We need to find the clusters automatically

67. Infinite GMM (Gaussian Mixture Models)
Based on Bayesian non-parametric approaches

68. Probabilistic programming
Logistic regression: 𝑝 𝑦𝑖 = 1 𝛽) = 𝑠𝑖𝑔𝑚𝑜𝑖𝑑(𝛽1 𝑥1 + 𝛽2 𝑥2 + 𝛽0) 𝛽 ~ 𝒩(𝜇, 𝜎)
Sampling:
• MCMC
• Gibbs

69. Probabilistic programming
Logistic regression: 𝑝 𝑦𝑖 = 1 𝛽) = 𝑠𝑖𝑔𝑚𝑜𝑖𝑑(𝛽1 𝑥1 + 𝛽2 𝑥2 + 𝛽0) 𝛽 ~ 𝒩(𝜇, 𝜎)
Sampling:
• MCMC
• Gibbs
Credible interval [a, b]
𝑝 𝑎 ≤ exp 𝛽𝑖 ≤ 𝑏 = 0.95
Posterior distribution of 𝛽
𝑝 exp 𝛽𝑖 𝑋𝑡𝑟𝑎𝑖𝑛, 𝑦𝑡𝑟𝑎𝑖𝑛)

70. Dirichlet distribution
𝐷𝑖𝑟 𝜃 𝛼 =
1
𝐵(𝛼)
𝑘=1
𝐾
𝜃 𝑘
𝛼 𝑘−1
𝒌
𝜽 𝒌 = 𝟏
𝜃 𝑘 ≥ 0
𝐵 𝛼 =
𝑖=1
𝐾
Γ(𝛼𝑖)
Γ( 𝑖=1
𝐾
𝛼𝑖)
Γ 𝑛 = 𝑛 − 1 !
Beta distribution
Gamma function
for positive integer n

71. Dirichlet distribution
𝐷𝑖𝑟 𝜃 𝛼 =
1
𝐵(𝛼)
𝑘=1
𝐾
𝜃 𝑘
𝛼 𝑘−1
K=3 Simplex
(0, 0, 1)
(0, 1, 0)(1, 0, 0)
(0.3, 0.2, 0.5)
Effects of varying 𝛼
Dirichlet distribution
𝛼 = (10, 10, 10)
Dirichlet distribution
𝛼 = (0.1, 0.1, 0.1)
Density
𝒌
𝜽 𝒌 = 𝟏
𝜃 𝑘 ≥ 0
𝐵 𝛼 =
𝑖=1
𝐾
Γ(𝛼𝑖)
Γ( 𝑖=1
𝐾
𝛼𝑖)
Γ 𝑛 = 𝑛 − 1 !
Beta distribution
Gamma function
for positive integer n
(
1
3
,
1
3
,
1
3
)

72. Intuition behind Infinite GMM
Properties of Dirichlet distribution
Dirichlet distribution is conjugate to Multinomial distribution
If π = 𝜋1, 𝜋2, … , 𝜋 𝑘 ~ 𝐷𝑖𝑟𝑖𝑐ℎ𝑙𝑒𝑡(𝛼1, 𝛼2, … , 𝛼 𝑘)
Dirichlet Satisfies expansion or combination rule:
𝝅 𝟏 𝜽, 𝝅 𝟏(𝟏 − 𝜽), 𝝅 𝟐, … , 𝝅 𝒌 ~ 𝑫𝒊𝒓𝒊𝒄𝒉𝒍𝒆𝒕(𝜶 𝟏 𝒃 , 𝜶 𝟏(𝟏 − 𝒃), 𝜶 𝟐, … , 𝜶 𝒌)
Allows to increase the dimensionality of Dirichlet
Where 0 < b < 1 and 𝜃~𝐵𝑒𝑡𝑎(𝛼1 𝑏, 𝛼1 1 − 𝑏 )

73. Intuition behind Infinite GMM
Properties of Dirichlet distribution
Dirichlet distribution is conjugate to Multinomial distribution
If π = 𝜋1, 𝜋2, … , 𝜋 𝑘 ~ 𝐷𝑖𝑟𝑖𝑐ℎ𝑙𝑒𝑡(𝛼1, 𝛼2, … , 𝛼 𝑘)
Dirichlet Satisfies expansion or combination rule:
𝝅 𝟏 𝜽, 𝝅 𝟏(𝟏 − 𝜽), 𝝅 𝟐, … , 𝝅 𝒌 ~ 𝑫𝒊𝒓𝒊𝒄𝒉𝒍𝒆𝒕(𝜶 𝟏 𝒃 , 𝜶 𝟏(𝟏 − 𝒃), 𝜶 𝟐, … , 𝜶 𝒌)
Allows to increase the dimensionality of Dirichlet
Where 0 < b < 1 and 𝜃~𝐵𝑒𝑡𝑎(𝛼1 𝑏, 𝛼1 1 − 𝑏 )
Dirichlet Process π(2)
= 𝜋1
(2)
, 𝜋2
(2)
~ 𝐷𝑖𝑟( 𝛼
2,
𝛼
2) ~ 𝐷𝑖𝑟( 𝛼
4,
𝛼
4,
𝛼
4,
𝛼
4) ~ 𝐷𝑖𝑟( 𝛼
𝐾
,…
𝛼
𝐾
)
𝑲 → ∞

74. Dirichlet process
21 : The Indian Bu↵et Process
Figure 2: On the left is an example of Indian Bu↵et Process dish assign
the right is an example binary matrix generated from IBP.
3. The nth customer helps himself to each dish with probability mk /
dish k was chosen.
4. He tries Poisson(↵/ n) new dishes.
Indian buffet processChinese restaurant process
Chinese restaurant process in action

75. Distribution of Distributions
Mixture of Gaussians

76. Infinite GMM
Probabilistic
modelling:
• PyMC3
• TensorFlow
Scikit-learn:
sklearn.mixture.
BayessianGuassian
Mixture
Dirichlet Prior:
• Dirichlet Distribution - Finite GMM
• Dirichlet Process - Infinite GMM

77. Probabilistic modeling
• Probabilistic models captures the uncertainty better in real
world data
• But it is very computationally intensive
• The sampling process takes time to stabilize and then generate meaningful
results
• Certainly cannot work on large datasets

78. Applied Data Science for automated clustering

79. Finding dense regionsation Maximization

80. Finding dense regionsation Maximization

81. Finding dense regionsation Maximization

82. Obtaining stochastic periods recursively
Stochastic
periods
• Get probability distributions
from dense regions
Get dense
regions list
• Find dmin to cluster a
region
Recursively
split regions
• If region is:
{Heavy tailed, Multi-modal}
Kurtosis of normal distribution = 3
Heavy tailed: Excess Kurtosis > 6
𝐵𝑖𝑚𝑜𝑑𝑎𝑙𝑖𝑡𝑦 =
𝛾2
+ 1
𝜅
𝛾: Skewness
𝜅 : Kurtosis
Bimodality
for uniform
distribution 5/9
Bimodality > 0.8
Unimoda
l
Not unimodal
Unimo
dal ?

83. dmin via distance matrix
This topic was covered generically in detail during ODSC 2018 – “Topological space clustering”

84. dmin via distance matrix
This topic was covered generically in detail during ODSC 2018 – “Topological space clustering”

85. dmin via distance matrix
This topic was covered generically in detail during ODSC 2018 – “Topological space clustering”
If local dense regions exists along with
sparsity, then we can obtain hierarchical
clusters at each mode

86. Density plot of distance matrix

87. Method proposed:
Finding optimal clustering-epsilon
• The problem comes down to finding the most optimal curve for the
Gaussian kernel
• One of the ways to solve it algorithmically
Grid Search
(band_width,
grid_size)
rFFT
Silverman
Transform
I-
rFFT
Score
(logLoss,
stdDev)
Minima
(band_width, grid_size)

88. Genuine systematic DNS queries to DNS server
Time delta
analysis
Time delta
Density
plot

89. Stochastic periods: Systematic DNS queries
Mean (μ) Std. (σ)
Periodicity
( μ / σ )
Skewness Kurtosis
Bimodal
Coeff
Num
Points
Density Range_min Range_max
0 0.809819 0.000499 1623.841793 0.252468 -0.628825 0.443460 330 330.000000 0.808578 0.810866
1 0.812474 0.000559 1454.556761 -0.879146 0.291782 0.536779 817 817.000000 0.810869 0.813244
2 0.813497 0.000141 5758.645345 0.130659 -1.036117 0.512333 426 426.000000 0.813245 0.813768
3 0.814162 0.000326 2495.343595 0.774715 -0.464280 0.623092 281 281.000000 0.813769 0.815018
4 1.622954 0.000631 2570.868659 -0.093343 -1.009738 0.504116 845 845.000000 1.621745 1.624109
5 1.625497 0.000630 2578.372267 -0.304701 -0.984417 0.540452 1386 1386.000000 1.624114 1.626489
6 1.627108 0.000496 3282.770498 0.992204 0.538372 0.558516 614 614.000000 1.626492 1.628858
7 2.436156 0.000627 3885.490319 0.059122 -1.007464 0.492753 208 208.000000 2.434985 2.437341
8 2.438674 0.000653 3733.232096 -0.230265 -0.988668 0.514873 269 269.000000 2.437374 2.439728

90. THANKS
E-mail: kuldeep.jiwani@gmail.com / kuldeep.Jiwani@thalesgroup.com
LinkedIn: https://www.linkedin.com/in/kuldeep-jiwani-988605/

91. Appendix

92. Finite GMM: Bayesian setting
Algorithm: Collapsed Gibbs sampler for a finite Gaussian mixture model
Choose an initial z
For T iterations do # Gibbs sampling iterations
For i = 1 to N do
Remove xi ’s statistics from component zi # Old cluster assignment for xi
For k = 1 to K do # Every possible component
Calculate P(zi = k|zi , α)
Calculate p(xi |Xki , β)
Calculate P(zi = k|zi , X , α, β) ∝ P(zi = k|zi , α) p(xi |Xki , β)
End for
Sample knew from P(zi |zi , X , α, β) after normalizing
Add xi ’s statistics to the component zi = knew # New assignment for xi
End for
End for Evaluation metric for Gibbs: 𝑘=1
𝐾
𝑝 𝑋 𝑘 𝛽 𝑝(𝑧|𝛼)

93. Infinite GMM: Bayesian setting
Choose an initial z
For T iterations do # Gibbs sampling iterations
For i = 1 to N do
Remove xi ’s statistics from component zi # Old cluster assignment for xi
For k = 1 to K do # Every possible component
Calculate P(zi = k|zi , α)
Calculate p(xi |Xki , β)
Calculate P(zi = k|zi , X , α, β) ∝ P(zi = k|zi , α) p(xi |Xki , β)
End for
Calculate P (zi = k∗|zi, α) # Consider a new component
Calculate p(xi|β)
Calculate P (zi = k∗|zi, X , α, β) ∝ P (zi = k∗|zi, α) p(xi|β)
Sample knew from P(zi |zi , X , α, β) after normalizing
If any component is empty, remove it and decrease K
Add xi ’s statistics to the component zi = knew # New assignment for xi
End for
End for

94. Simplex-2D