Leveraging change point detection to discover natural experiments in data

He, Yuzi; Burghardt, Keith A.; Lerman, Kristina

doi:10.1140/epjds/s13688-022-00361-7

EPJ Data Science

Table 1 A comprehensive comparison of the performance of the proposed method against two types of state-of-the-art methods: optimal segmentation and Bayesian change point detection on synthetic data. MtChD(RF) is our method with a random forest classifier; MtChD(MLP) is our method with a MLP classifier. DP + Normal (GLR eq.) is DP segmentation method used with normal loss function, which is equivalent to GLR test that assumes a multivariate normal distribution. Six combinations of optimal segmentation methods are listed. DP is dynamic programming segmentation algorithm, BinSeg is binary segmentation, Window is window-based change point detection, and BottomUp is Bottom-up segmentation. The cost functions used are RBF (RBF kernel), L1 (\(L_{1}\) loss function), and L2 (\(L_{2}\) loss function). The last four rows are for Bayesian change point detection with a uniform prior or Geo (geometric) prior. Gassusian stands for Gaussian likelihood function, IFM is the individual feature model [30], and FullCov is the full covariance model [30]. \(\mu (t_{0})\) and \(\sigma (t_{0})\) are the mean value and standard deviation of inferred change point and \(\mu (\alpha )\) and \(\sigma (\alpha )\) are the mean value and standard deviation of inferred α. Bold values indicate change points that are closest to the correct value

From: Leveraging change point detection to discover natural experiments in data

	\(n_{c}\)	2	4	6	8	10	6	6	6	6
	\(t_{0}\)	0.5	0.5	0.5	0.5	0.5	0.2	0.4	0.6	0.8
MtChD (RF)	\(\mu (t_{0})\)	0.5002	0.4983	0.4976	0.5000	0.4959	0.1950	0.3937	0.6014	0.8020
	\(\sigma (t_{0})\)	0.0025	0.0017	0.0033	0.0005	0.0049	0.0047	0.0052	0.0023	0.0022
	μ(α)	0.9494	0.9137	0.8562	0.7604	0.6573	0.6503	0.8429	0.8316	0.6580
	σ(α)	0.0077	0.0041	0.0119	0.0220	0.0156	0.0346	0.0076	0.0133	0.0276
MtChD (MLP)	\(\mu (t_{0})\)	0.5027	0.5003	0.5262	0.5084	0.5772	0.5649	0.4095	0.5962	0.5372
	\(\sigma (t_{0})\)	0.0027	0.0039	0.0173	0.0962	0.0569	0.0450	0.0258	0.0668	0.1315
	μ(α)	0.9589	0.8289	0.6249	0.0048	0.0086	0.0045	0.4906	0.3950	0.0171
	σ(α)	0.0095	0.0366	0.0710	0.0068	0.0080	0.0035	0.0534	0.1112	0.0202
Naive Confusion (RF)	\(\mu (t_{0})\)	0.4965	0.5017	0.4974	0.4975	0.4973	0.2271	0.4255	0.5235	0.5436
Naive Confusion (RF)	\(\sigma (t_{0})\)	0.0018	0.0019	0.0004	0.0001	0.0001	0.0382	0.0312	0.0229	0.0900
DP + Normal	\(\mu (t_{0})\)	0.5003	0.5006	0.5212	0.7238	0.5971	0.2441	0.4578	0.5885	0.8108
(GLR eq.)	\(\sigma (t_{0})\)	0.0004	0.0005	0.0204	0.2762	0.3374	0.0377	0.0447	0.0266	0.0288
DP + RBF	\(\mu (t_{0})\)	0.5002	0.5001	0.5673	0.9495	0.3071	0.3740	0.4234	0.5827	0.8355
DP + RBF	\(\sigma (t_{0})\)	0.0004	0.0019	0.0684	0.0679	0.2392	0.2840	0.1893	0.0246	0.0654
DP + L2	\(\mu (t_{0})\)	0.9510	0.9875	0.3515	0.8584	0.5143	0.4451	0.3183	0.3104	0.2917
DP + L2	\(\sigma (t_{0})\)	0.0099	0.0062	0.2399	0.2734	0.4006	0.3481	0.4417	0.4252	0.3778
DP + L1	\(\mu (t_{0})\)	0.9569	0.5313	0.5809	0.6053	0.4015	0.5526	0.1277	0.4916	0.2114
DP + L1	\(\sigma (t_{0})\)	0.0070	0.2660	0.1677	0.4027	0.3308	0.4467	0.1873	0.3832	0.3312
BinSeg + RBF	\(\mu (t_{0})\)	0.5002	0.4995	0.5701	0.7663	0.5635	0.3133	0.3850	0.6049	0.7258
BinSeg + RBF	\(\sigma (t_{0})\)	0.0002	0.0011	0.0502	0.3205	0.2190	0.3285	0.3702	0.1506	0.2715
Window + RBF	\(\mu (t_{0})\)	0.4391	0.5653	0.2960	0.5699	0.2444	0.4746	0.5654	0.7964	0.3987
Window + RBF	\(\sigma (t_{0})\)	0.1364	0.2210	0.2139	0.1738	0.1012	0.2436	0.2459	0.2223	0.3159
BottomUp + RBF	\(\mu (t_{0})\)	0.5002	0.4581	0.4500	0.6821	0.4947	0.4271	0.5213	0.4602	0.5861
BottomUp + RBF	\(\sigma (t_{0})\)	0.0008	0.1477	0.3655	0.2879	0.3144	0.3059	0.2149	0.2885	0.2953
Uniform + Gaussian	\(\mu (t_{0})\)	0.5474	0.5429	0.3915	0.4717	0.5429	0.6171	0.7546	0.5210	0.5196
Uniform + Gaussian	\(\sigma (t_{0})\)	0.2299	0.3010	0.1567	0.2265	0.2159	0.2842	0.2203	0.1549	0.3386
Uniform + IFM	\(\mu (t_{0})\)	0.9969	0.9942	0.9973	0.9975	0.9975	0.9986	0.9958	0.9973	0.9985
Uniform + IFM	\(\sigma (t_{0})\)	0.0031	0.0030	0.0020	0.0015	0.0030	0.0015	0.0049	0.0026	0.0012
Uniform + FullCov	\(\mu (t_{0})\)	0.4985	0.5089	0.9986	0.9976	0.9989	0.9930	0.9280	0.9982	0.9974
Uniform + FullCov	\(\sigma (t_{0})\)	0.0002	0.0163	0.0006	0.0010	0.0009	0.0098	0.1593	0.0020	0.0038
Geo + Gaussian	\(\mu (t_{0})\)	0.0282	0.0271	0.0286	0.0323	0.0278	0.0326	0.0340	0.0312	0.0254
Geo + Gaussian	\(\sigma (t_{0})\)	0.0044	0.0018	0.0044	0.0054	0.0037	0.0063	0.0034	0.0051	0.0037

Back to article page