From: SentiBench - a benchmark comparison of state-of-the-practice sentiment analysis methods
Dataset | Method | Accur. | Posit. sentiment | Negat. sentiment | Macro- F 1 | Coverage | ||||
---|---|---|---|---|---|---|---|---|---|---|
P | R | F 1 | P | R | F 1 | |||||
Tweets_RND_II | AFINN | 96.37 | 97.66 | 96.94 | 97.30 | 93.75 | 95.19 | 94.47 | 95.88 | 80.77 |
ANEW_SUB | 81.36 | 80.52 | 96.38 | 87.74 | 85.44 | 47.64 | 61.17 | 74.45 | 93.35 | |
Emolex | 86.06 | 89.82 | 89.11 | 89.47 | 78.77 | 80.00 | 79.38 | 84.42 | 63.58 | |
Emoticons | 97.75 | 97.90 | 99.42 | 98.65 | 96.97 | 89.72 | 93.20 | 95.93 | 14.82 | |
Emoticons DS | 71.04 | 70.61 | 99.90 | 82.74 | 95.83 | 5.43 | 10.28 | 46.51 | 99.09 | |
NRC Hashtag | 67.37 | 83.76 | 65.43 | 73.47 | 48.17 | 71.69 | 57.62 | 65.55 | 91.94 | |
LIWC07 | 66.47 | 74.46 | 78.81 | 76.58 | 44.20 | 38.31 | 41.04 | 58.81 | 73.93 | |
LIWC15 | 96.44 | 97.09 | 98.04 | 97.56 | 94.68 | 92.23 | 93.44 | 95.50 | 77.05 | |
Opinion Finder | 78.32 | 93.86 | 71.11 | 80.92 | 63.42 | 91.50 | 74.92 | 77.92 | 41.23 | |
Opinion Lexicon | 93.45 | 97.03 | 93.14 | 95.04 | 86.93 | 94.11 | 90.38 | 92.71 | 70.64 | |
PANAS-t | 90.71 | 96.95 | 88.19 | 92.36 | 82.11 | 95.12 | 88.14 | 90.25 | 5.39 | |
Pattern.en | 91.76 | 92.94 | 96.19 | 94.54 | 87.86 | 79.06 | 83.23 | 88.88 | 70.85 | |
SASA | 70.06 | 82.81 | 72.81 | 77.49 | 49.05 | 63.39 | 55.30 | 66.40 | 63.04 | |
Semantria | 91.61 | 96.94 | 90.55 | 93.64 | 82.25 | 93.88 | 87.68 | 90.66 | 63.61 | |
SenticNet | 73.64 | 90.74 | 68.45 | 78.03 | 55.41 | 84.88 | 67.05 | 72.54 | 82.82 | |
Sentiment140 | 94.75 | 97.10 | 95.71 | 96.40 | 88.64 | 92.13 | 90.35 | 93.37 | 49.95 | |
Sentiment140_L | 78.05 | 88.68 | 78.31 | 83.17 | 61.32 | 77.47 | 68.45 | 75.81 | 93.28 | |
SentiStrength | 96.97 | 98.92 | 96.43 | 97.66 | 93.54 | 98.01 | 95.72 | 96.69 | 34.65 | |
SentiWordNet | 78.57 | 87.88 | 80.91 | 84.25 | 61.09 | 72.87 | 66.46 | 75.36 | 61.49 | |
SO-CAL | 87.76 | 94.25 | 86.99 | 90.47 | 77.34 | 89.32 | 82.90 | 86.68 | 67.18 | |
Stanford DM | 60.46 | 94.48 | 44.87 | 60.84 | 44.06 | 94.30 | 60.06 | 60.45 | 88.89 | |
Umigon | 88.63 | 97.73 | 85.92 | 91.45 | 73.64 | 95.17 | 83.03 | 87.24 | 70.83 | |
USent | 84.46 | 89.28 | 87.67 | 88.47 | 74.77 | 77.63 | 76.17 | 82.32 | 38.94 | |
VADER | 99.04 | 99.16 | 99.45 | 99.31 | 98.77 | 98.12 | 98.45 | 98.88 | 94.40 | |
Tweets_STF | AFINN | 84.42 | 80.62 | 91.49 | 85.71 | 89.66 | 77.04 | 82.87 | 84.29 | 76.88 |
ANEW_SUB | 68.05 | 63.08 | 93.18 | 75.23 | 84.62 | 40.74 | 55.00 | 65.11 | 94.15 | |
Emolex | 79.65 | 76.09 | 88.98 | 82.03 | 85.23 | 69.44 | 76.53 | 79.28 | 62.95 | |
Emoticons | 85.42 | 80.65 | 96.15 | 87.72 | 94.12 | 72.73 | 82.05 | 84.89 | 13.37 | |
Emoticons DS | 51.96 | 51.41 | 100.00 | 67.91 | 100.00 | 2.27 | 4.44 | 36.18 | 99.72 | |
NRC Hashtag | 71.30 | 73.05 | 70.93 | 71.98 | 69.51 | 71.70 | 70.59 | 71.28 | 92.20 | |
LIWC07 | 64.29 | 63.75 | 76.12 | 69.39 | 65.22 | 50.85 | 57.14 | 63.27 | 70.39 | |
LIWC15 | 89.22 | 84.18 | 97.08 | 90.17 | 96.40 | 81.06 | 88.07 | 89.12 | 74.93 | |
Opinion Finder | 80.77 | 81.16 | 76.71 | 78.87 | 80.46 | 84.34 | 82.35 | 80.61 | 43.45 | |
Opinion Lexicon | 86.10 | 83.67 | 91.11 | 87.23 | 89.29 | 80.65 | 84.75 | 85.99 | 72.14 | |
PANAS-t | 94.12 | 88.89 | 100.00 | 94.12 | 100.00 | 88.89 | 94.12 | 94.12 | 4.74 | |
Pattern.en | 79.55 | 74.86 | 94.48 | 83.54 | 90.12 | 61.34 | 73.00 | 78.27 | 73.54 | |
SASA | 68.52 | 65.65 | 78.90 | 71.67 | 72.94 | 57.94 | 64.58 | 68.12 | 60.17 | |
Semantria | 88.45 | 89.15 | 88.46 | 88.80 | 87.70 | 88.43 | 88.07 | 88.43 | 69.92 | |
SenticNet | 70.49 | 71.31 | 63.50 | 67.18 | 69.88 | 76.82 | 73.19 | 70.18 | 80.22 | |
Sentiment140 | 93.29 | 91.36 | 94.87 | 93.08 | 95.18 | 91.86 | 93.49 | 93.29 | 45.68 | |
Sentiment140_L | 79.12 | 81.48 | 76.30 | 78.81 | 76.97 | 82.04 | 79.42 | 79.11 | 94.71 | |
SentiStrength | 95.33 | 95.18 | 96.34 | 95.76 | 95.52 | 94.12 | 94.81 | 95.29 | 41.78 | |
SentiWordNet | 72.99 | 73.17 | 78.95 | 75.95 | 72.73 | 65.98 | 69.19 | 72.57 | 58.77 | |
SO-CAL | 87.36 | 82.89 | 93.33 | 87.80 | 92.80 | 81.69 | 86.89 | 87.35 | 77.16 | |
Stanford DM | 66.56 | 87.69 | 36.31 | 51.35 | 61.24 | 95.18 | 74.53 | 62.94 | 89.97 | |
Umigon | 86.99 | 91.73 | 81.88 | 86.52 | 83.02 | 92.31 | 87.42 | 86.97 | 81.34 | |
USent | 73.21 | 69.35 | 82.69 | 75.44 | 78.82 | 63.81 | 70.53 | 72.98 | 58.22 | |
VADER | 84.44 | 80.23 | 92.21 | 85.80 | 90.40 | 76.35 | 82.78 | 84.29 | 84.12 | |
Comments_Digg | AFINN | 70.94 | 47.01 | 81.82 | 59.72 | 91.17 | 67.05 | 77.27 | 68.49 | 74.81 |
ANEW_SUB | 43.25 | 30.98 | 92.31 | 46.39 | 90.13 | 25.46 | 39.71 | 43.05 | 93.73 | |
Emolex | 61.71 | 34.60 | 75.83 | 47.52 | 88.93 | 57.53 | 69.87 | 58.69 | 67.14 | |
Emoticons | 73.08 | 72.22 | 86.67 | 78.79 | 75.00 | 54.55 | 63.16 | 70.97 | 3.32 | |
Emoticons DS | 28.24 | 27.30 | 100.00 | 42.89 | 100.00 | 1.77 | 3.48 | 23.19 | 98.72 | |
NRC Hashtag | 74.69 | 51.01 | 40.64 | 45.24 | 80.80 | 86.48 | 83.54 | 64.39 | 92.97 | |
LIWC07 | 46.15 | 27.44 | 58.40 | 37.34 | 72.49 | 41.52 | 52.79 | 45.07 | 58.18 | |
LIWC15 | 70.67 | 49.81 | 90.91 | 64.36 | 94.35 | 62.36 | 75.09 | 69.72 | 62.79 | |
Opinion Finder | 71.14 | 43.04 | 64.76 | 51.71 | 86.88 | 73.13 | 79.42 | 65.56 | 56.27 | |
Opinion Lexicon | 71.82 | 47.45 | 86.43 | 61.27 | 93.40 | 66.75 | 77.86 | 69.56 | 69.44 | |
PANAS-t | 68.00 | 12.50 | 50.00 | 20.00 | 94.12 | 69.57 | 80.00 | 50.00 | 3.20 | |
Pattern.en | 60.05 | 43.73 | 92.14 | 59.31 | 92.57 | 45.21 | 60.75 | 60.03 | 56.65 | |
SASA | 65.54 | 40.26 | 66.91 | 50.27 | 84.82 | 65.06 | 73.64 | 61.95 | 68.29 | |
Semantria | 82.46 | 62.72 | 88.33 | 73.36 | 94.81 | 80.25 | 86.93 | 80.14 | 56.14 | |
SenticNet | 69.40 | 46.30 | 72.46 | 56.50 | 86.77 | 68.25 | 76.40 | 66.45 | 96.55 | |
Sentiment140 | 85.06 | 62.50 | 78.95 | 69.77 | 93.65 | 86.76 | 90.08 | 79.92 | 33.38 | |
Sentiment140_L | 67.76 | 42.07 | 73.45 | 53.50 | 88.01 | 65.84 | 75.33 | 64.41 | 89.64 | |
SentiStrength | 92.09 | 78.69 | 92.31 | 84.96 | 97.40 | 92.02 | 94.64 | 89.80 | 27.49 | |
SentiWordNet | 62.17 | 36.86 | 77.68 | 50.00 | 88.84 | 57.18 | 69.58 | 59.79 | 58.82 | |
SO-CAL | 76.55 | 52.86 | 77.08 | 62.71 | 90.65 | 76.37 | 82.90 | 72.81 | 71.99 | |
Stanford DM | 69.16 | 35.29 | 20.27 | 25.75 | 75.21 | 86.68 | 80.54 | 53.15 | 78.90 | |
Umigon | 83.37 | 66.22 | 75.38 | 70.50 | 90.72 | 86.23 | 88.42 | 79.46 | 63.04 | |
USent | 55.98 | 36.06 | 80.65 | 49.83 | 86.67 | 46.80 | 60.78 | 55.31 | 43.86 | |
VADER | 69.05 | 45.48 | 85.88 | 59.47 | 92.55 | 63.00 | 74.97 | 67.22 | 82.23 | |
Comments_BBC | AFINN | 66.56 | 23.08 | 81.08 | 35.93 | 96.32 | 64.66 | 77.38 | 56.65 | 85.11 |
ANEW_SUB | 31.37 | 15.48 | 95.79 | 26.65 | 97.18 | 21.73 | 35.52 | 31.08 | 97.07 | |
Emolex | 59.64 | 21.52 | 89.04 | 34.67 | 97.38 | 55.62 | 70.80 | 52.73 | 80.72 | |
Emoticons | 33.33 | 0.00 | 0.00 | 0.00 | 100.00 | 33.33 | 50.00 | 25.00 | 0.40 | |
Emoticons DS | 13.33 | 13.10 | 100.00 | 23.17 | 100.00 | 0.31 | 0.61 | 11.89 | 99.73 | |
NRC Hashtag | 84.45 | 33.33 | 25.27 | 28.75 | 89.76 | 92.83 | 91.27 | 60.01 | 97.47 | |
LIWC07 | 50.10 | 15.38 | 58.33 | 24.35 | 88.00 | 48.78 | 62.77 | 43.56 | 69.55 | |
LIWC15 | 63.21 | 25.86 | 90.67 | 40.24 | 97.55 | 58.86 | 73.42 | 56.83 | 73.01 | |
Opinion Finder | 74.43 | 21.74 | 62.50 | 32.26 | 94.93 | 75.72 | 84.24 | 58.25 | 76.46 | |
Opinion Lexicon | 74.14 | 29.81 | 84.93 | 44.13 | 97.24 | 72.66 | 83.17 | 63.65 | 80.72 | |
PANAS-t | 58.73 | 20.00 | 75.00 | 31.58 | 93.94 | 56.36 | 70.45 | 51.02 | 8.38 | |
Pattern.en | 41.75 | 19.73 | 93.55 | 32.58 | 96.61 | 32.57 | 48.72 | 40.65 | 54.79 | |
SASA | 61.61 | 23.50 | 66.20 | 34.69 | 90.80 | 60.77 | 72.81 | 53.75 | 61.30 | |
Semantria | 83.43 | 40.00 | 84.75 | 54.35 | 97.64 | 83.26 | 89.88 | 72.11 | 67.42 | |
SenticNet | 66.07 | 24.44 | 74.16 | 36.77 | 94.24 | 64.83 | 76.81 | 56.79 | 88.96 | |
Sentiment140 | 68.51 | 24.00 | 69.77 | 35.71 | 94.04 | 68.33 | 79.15 | 57.43 | 45.61 | |
Sentiment140_L | 56.85 | 18.52 | 69.15 | 29.21 | 92.35 | 55.03 | 68.97 | 49.09 | 97.07 | |
SentiStrength | 93.93 | 64.29 | 78.26 | 70.59 | 97.72 | 95.54 | 96.61 | 83.60 | 32.85 | |
SentiWordNet | 57.49 | 20.00 | 88.06 | 32.60 | 97.13 | 53.45 | 68.96 | 50.78 | 76.33 | |
SO-CAL | 75.28 | 28.93 | 80.28 | 42.54 | 96.71 | 74.64 | 84.25 | 63.40 | 82.85 | |
Stanford DM | 89.45 | 63.16 | 40.91 | 49.66 | 91.81 | 96.52 | 94.11 | 71.88 | 92.02 | |
Umigon | 79.37 | 39.13 | 61.02 | 47.68 | 92.10 | 82.72 | 87.15 | 67.42 | 50.93 | |
USent | 52.60 | 18.33 | 80.49 | 29.86 | 94.56 | 48.60 | 64.20 | 47.03 | 43.48 | |
VADER | 62.76 | 22.68 | 85.54 | 35.86 | 96.75 | 59.60 | 73.76 | 54.81 | 90.69 |