SentiBench - a benchmark comparison of state-of-the-practice sentiment analysis methods

Ribeiro, Filipe N; Araújo, Matheus; Gonçalves, Pollyanna; André Gonçalves, Marcos; Benevenuto, Fabrício

doi:10.1140/epjds/s13688-016-0085-1

EPJ Data Science

Table 7 3-classes experiments results with 4 datasets

From: SentiBench - a benchmark comparison of state-of-the-practice sentiment analysis methods

Dataset	Method	Accur.	Posit. sentiment			Negat. sentiment			Neut. sentiment			Macro- F 1
Dataset	Method	Accur.	P	R	F 1	P	R	F 1	P	R	F 1	Macro- F 1
Tweets_Semeval	AFINN	62.36	61.10	70.09	65.28	44.08	55.56	49.15	71.43	58.57	64.37	59.60
	ANEW_SUB	39.51	38.79	96.31	55.31	43.50	23.18	30.24	57.38	2.31	4.45	30.00
	Emolex	48.74	48.15	62.71	54.47	31.27	38.59	34.55	57.90	41.30	48.21	45.74
	Emoticons	52.88	72.83	11.34	19.62	55.56	5.38	9.80	34.05	96.53	50.34	26.59
	Emoticons DS	36.59	36.55	100.00	53.53	75.00	0.36	0.71	100.00	0.03	0.07	18.10
	NRC Hashtag	36.95	42.04	75.03	53.88	24.57	56.03	34.16	53.33	3.70	6.92	31.65
	LIWC07	39.54	36.52	42.33	39.21	15.14	13.02	14.00	48.64	44.83	46.66	33.29
	LIWC15	62.56	59.77	71.03	64.91	49.04	42.65	45.62	68.90	61.84	65.18	58.57
	Opinion Finder	57.63	67.57	27.94	39.53	40.75	33.69	36.89	58.20	86.06	69.44	48.62
	Opinion Lexicon	60.37	62.09	62.71	62.40	41.19	52.81	46.28	66.41	60.75	63.46	57.38
	PANAS-t	53.08	90.95	9.04	16.45	51.56	3.94	7.33	51.65	99.01	67.89	30.55
	Pattern.en	57.99	57.97	68.74	62.89	34.83	35.24	35.04	65.55	56.39	60.63	52.85
	SASA	50.63	46.34	47.77	47.04	33.07	20.31	25.17	56.39	61.12	58.66	43.62
	Semantria	61.54	67.28	57.35	61.92	39.57	52.81	45.24	65.98	67.03	66.50	57.89
	SenticNet	49.68	51.85	1.26	2.46	29.79	1.67	3.17	49.82	98.51	66.17	23.93
	Sentiment140	60.42	63.87	51.37	56.94	50.96	37.87	43.45	60.35	73.31	66.20	55.53
	Sentiment140_L	39.44	43.52	74.72	55.00	27.67	65.35	38.88	65.87	6.38	11.63	35.17
	SentiStrength	57.83	78.01	27.13	40.25	47.80	23.42	31.44	55.49	89.89	68.62	46.77
	SentiWordNet	48.33	55.54	53.44	54.47	19.67	37.51	25.81	61.22	47.57	53.54	44.61
	SO-CAL	58.83	58.89	59.02	58.95	40.39	54.24	46.30	39.89	59.96	47.91	51.05
	Stanford DM	22.54	72.14	18.17	29.03	14.92	90.56	25.61	47.19	6.94	12.10	22.25
	Umigon	65.88	75.18	56.14	64.28	39.66	55.91	46.41	70.65	75.78	73.13	61.27
	USent	52.13	49.86	32.88	39.63	39.96	22.82	29.05	54.33	74.36	62.79	43.82
	VADER	60.21	56.46	79.04	65.87	44.30	59.02	50.61	76.02	46.71	57.87	58.12
Tweets_RND_III	AFINN	64.41	40.81	72.12	52.13	49.67	62.50	55.35	85.95	62.54	72.40	59.96
	ANEW_SUB	28.03	21.89	92.29	35.38	44.30	34.22	38.61	74.82	8.18	14.74	29.58
	Emolex	54.76	31.67	59.95	41.44	40.14	47.54	43.53	77.48	54.64	64.08	49.68
	Emoticons	70.22	70.06	16.78	27.07	65.62	8.61	15.22	41.29	97.56	58.02	33.44
	Emoticons DS	20.34	19.78	99.46	33.00	62.07	3.69	6.96	53.85	0.55	1.09	13.68
	NRC Hashtag	30.47	28.25	77.40	41.39	24.18	72.54	36.27	79.08	8.77	15.78	31.15
	LIWC	46.88	21.85	38.43	27.86	19.18	18.24	18.70	69.51	54.83	61.31	35.95
	LIWC15	67.75	44.78	78.35	56.99	57.49	57.38	57.44	85.18	66.67	74.80	63.07
	Opinion Finder	71.55	57.48	32.75	41.72	49.85	34.63	40.87	75.95	89.90	82.34	54.98
	Opinion Lexicon	63.86	40.65	66.17	50.36	48.84	56.15	52.24	81.96	64.66	72.29	58.30
	PANAS-t	68.79	79.49	8.39	15.18	48.57	3.48	6.50	68.75	98.86	81.10	34.26
	Pattern.en	59.56	36.20	77.00	49.24	52.87	45.29	48.79	81.75	57.23	67.33	55.12
	SASA	55.37	29.42	54.53	38.22	42.46	47.34	44.77	78.30	57.15	66.08	49.69
	Semantria	68.89	48.86	63.73	55.31	49.82	55.53	52.52	82.02	72.96	77.22	61.68
	SenticNet	29.97	31.08	74.83	43.92	20.98	73.98	32.68	79.70	8.49	15.35	30.65
	Sentiment140	76.40	64.42	51.69	57.36	74.75	45.49	56.56	79.04	89.50	83.94	65.95
	Sentiment140_L	31.32	25.83	77.13	38.70	30.05	78.69	43.49	79.37	8.92	16.04	32.74
	SentiStrength	73.80	70.94	41.95	52.72	57.53	25.82	35.64	75.35	92.26	82.95	57.10
	SentiWordNet	55.85	37.42	58.19	45.55	24.04	35.86	28.78	79.25	59.00	67.64	47.33
	SO-CAL	66.51	43.06	68.88	52.99	51.84	60.66	55.90	45.77	66.94	54.37	54.42
	Stanford DM	31.90	64.48	38.57	48.26	15.58	85.04	26.33	75.64	19.77	31.35	35.32
	Umigon	74.12	57.67	70.23	63.33	48.83	68.44	57.00	88.80	76.34	82.10	67.47
	USent	66.06	40.60	36.81	38.61	44.87	28.69	35.00	74.54	81.72	77.97	50.53
	VADER	60.14	37.69	81.60	51.56	48.56	65.57	55.80	88.96	52.87	66.32	57.89
Comments_BBC	AFINN	50.10	16.22	60.61	25.59	82.62	56.05	66.79	40.11	30.24	34.48	42.29
	ANEW_SUB	24.30	11.38	91.92	20.24	84.15	21.13	33.78	38.89	5.65	9.86	21.30
	Emolex	44.10	15.51	65.66	25.10	83.19	45.48	58.81	35.27	31.85	33.47	39.13
	Emoticons	24.60	0.00	0.00	0.00	33.33	0.15	0.30	19.77	98.79	32.95	11.09
	Emoticons DS	10.00	9.85	98.99	17.92	66.67	0.31	0.61	0.00	0.00	0.00	6.18
	NRC Hashtag	64.00	20.72	23.23	21.90	70.20	91.27	79.36	52.50	8.47	14.58	38.62
	LIWC07	33.00	11.11	42.42	17.61	67.69	33.69	44.99	22.90	27.42	24.95	29.18
	LIWC15	43.70	17.94	68.69	28.45	85.06	42.73	56.88	30.72	36.29	33.27	39.53
	Opinion Finder	51.80	14.96	35.35	21.02	78.76	60.18	68.23	33.71	36.29	34.95	41.40
	Opinion Lexicon	55.00	20.67	62.63	31.08	85.27	59.42	70.04	40.82	40.32	40.57	47.23
	PANAS-t	27.10	16.67	6.06	8.89	75.61	4.75	8.93	25.35	94.35	39.97	19.26
	Pattern.en	28.70	14.25	58.59	22.92	82.61	17.46	28.82	25.27	46.37	32.72	28.16
	SASA	38.20	17.03	47.47	25.07	70.75	36.29	47.98	25.19	39.52	30.77	34.60
	Semantria	56.00	28.90	50.51	36.76	83.82	57.12	67.94	35.86	55.24	43.49	49.40
	SenticNet	47.10	17.74	66.67	28.03	72.87	57.58	64.33	25.89	11.69	16.11	36.16
	Sentiment140	40.00	17.75	30.30	22.39	79.77	31.39	45.05	28.75	66.53	40.15	35.86
	Sentiment140_L	43.10	13.32	65.66	22.15	73.84	53.60	62.11	42.11	6.45	11.19	31.82
	SentiStrength	44.20	47.37	18.18	26.28	86.64	32.77	47.56	29.37	84.68	43.61	39.15
	SentiWordNet	42.40	14.90	59.60	23.84	81.63	41.50	55.03	34.56	37.90	36.15	38.34
	SO-CAL	55.50	20.88	57.58	30.65	80.47	63.09	70.73	28.57	34.68	31.33	44.23
	Stanford DM	65.50	43.37	36.36	39.56	71.01	89.28	79.10	37.50	14.52	20.93	46.53
	Umigon	45.70	28.35	36.36	31.86	76.35	41.04	53.39	29.31	61.69	39.74	41.66
	USent	33.80	13.75	33.33	19.47	82.25	21.29	33.82	28.09	66.94	39.57	30.95
	VADER	49.40	16.36	71.72	26.64	83.02	54.67	65.93	48.53	26.61	34.38	42.31
Comments_NYT	AFINN	42.45	64.81	41.79	50.81	80.29	39.82	53.24	7.89	77.87	14.32	39.46
	ANEW_SUB	51.12	48.35	88.57	62.55	79.65	24.69	37.69	7.92	9.84	8.78	36.34
	Emolex	42.97	55.12	53.72	54.41	75.35	33.33	46.22	7.22	54.10	12.74	37.79
	Emoticons	4.68	0.00	0.00	0.00	0.00	0.00	0.00	4.47	99.59	8.56	2.85
	Emoticons DS	42.58	42.55	99.77	59.66	78.57	0.40	0.80	0.00	0.00	0.00	20.15
	NRC Hashtag	54.84	55.38	45.74	50.10	61.55	65.68	63.55	8.33	15.16	10.76	41.47
	LIWC07	24.35	42.88	27.72	33.67	53.42	19.07	28.11	4.67	53.28	8.58	23.45
	LIWC15	36.49	65.29	40.29	49.83	81.50	29.25	43.05	7.17	83.61	13.20	35.36
	Opinion Finder	29.38	68.77	18.78	29.51	76.52	32.68	45.80	6.29	88.11	11.75	29.02
	Opinion Lexicon	44.57	65.95	43.15	52.17	79.81	43.11	55.98	7.94	73.77	14.34	40.83
	PANAS-t	5.88	69.23	1.23	2.41	62.07	1.31	2.57	4.75	99.18	9.07	4.68
	Pattern.en	31.60	55.23	45.05	49.63	72.80	17.76	28.55	5.88	65.57	10.79	29.66
	SASA	30.04	49.92	30.13	37.58	59.11	27.21	37.26	5.74	61.07	10.49	28.44
	Semantria	44.59	70.60	41.83	52.54	80.54	44.24	57.11	7.53	73.36	13.65	41.10
	SenticNet	61.85	58.19	59.48	58.83	65.01	69.26	67.07	0.00	0.00	0.00	41.97
	Sentiment140	13.58	77.32	6.81	12.51	75.40	11.96	20.65	4.98	93.03	9.45	14.20
	Sentiment140_L	54.61	54.72	59.12	56.84	67.00	54.41	60.05	6.70	15.98	9.44	42.11
	SentiStrength	18.17	78.51	8.62	15.54	81.12	18.96	30.74	5.41	95.49	10.24	18.84
	SentiWordNet	32.20	57.35	34.53	43.10	70.31	26.95	38.97	6.08	70.08	11.19	31.09
	SO-CAL	50.79	64.36	51.13	56.99	77.25	49.16	60.08	8.68	65.98	15.34	44.14
	Stanford DM	51.93	73.39	21.14	32.83	59.48	77.90	67.46	9.65	38.11	15.40	38.56
	Umigon	24.08	68.76	16.38	26.46	68.78	24.51	36.14	5.88	88.93	11.04	24.54
	USent	27.44	56.61	28.95	38.31	77.69	21.59	33.79	5.88	79.51	10.94	27.68
	VADER	48.03	62.67	51.63	56.62	79.91	43.07	55.97	9.18	71.31	16.26	42.95

Back to article page