Table 5 \(F_{1}\) scores of BLOC and baseline classifiers for the detection of information operation drivers, calculated with data from the first 10 weeks of each campaign (F1@Week 10). For campaigns with less than 10 weeks of data, the entire dataset was used. Information operations are sorted by the \(F_{1}\) score of the combined method (combined F1@Week 10). The best method for each campaign is shown in bold. Note that \(F_{1}=0\) when the similarity signal used by a classifier cannot be observed in the behavior of a particular campaign’s drivers. No co-retweets were observed between any pairs of drivers in China_1
From: A language framework for modeling social media account behavior
Information Op. | BLOC | Activity | CoRT | Hash | Combined |
|---|---|---|---|---|---|
China_3 | 0.995 | 0.968 | 0.182 | 0.973 | 0.995 |
China_4 | 0.996 | 0.846 | 0.844 | 0.205 | 0.994 |
China_5 | 0.981 | 0.919 | 0.991 | 0.504 | 0.980 |
Iran_6 | 0.961 | 0.986 | 0.978 | 0.000 | 0.976 |
Venezuela_2 | 0.936 | 0.864 | 0.000 | 0.000 | 0.929 |
Venezuela_5 | 0.869 | 0.815 | 0.903 | 0.830 | 0.910 |
Spain | 0.876 | 0.936 | 0.892 | 0.859 | 0.904 |
Mexico_2 | 0.851 | 0.569 | 0.312 | 0.071 | 0.874 |
Mexico_1 | 0.833 | 0.511 | 0.163 | 0.000 | 0.859 |
Uganda_1 | 0.850 | 0.807 | 0.898 | 0.432 | 0.850 |
Thailand | 0.769 | 0.815 | 0.808 | 0.226 | 0.832 |
Uganda_2 | 0.800 | 0.574 | 0.940 | 0.227 | 0.807 |
Venezuela_4 | 0.819 | 0.603 | 0.754 | 0.391 | 0.779 |
Venezuela_1 | 0.820 | 0.188 | 0.000 | 0.000 | 0.776 |
Catalonia | 0.957 | 0.667 | 0.762 | 0.308 | 0.774 |
China_2 | 0.762 | 0.000 | 0.000 | 0.000 | 0.762 |
Armenia | 0.760 | 0.875 | 0.895 | 0.851 | 0.760 |
Ghana_Nigeria | 0.612 | 0.866 | 0.741 | 0.528 | 0.700 |
Russia_1 | 0.667 | 0.000 | 0.000 | 0.000 | 0.667 |
Iran_2 | 0.625 | 0.200 | 0.000 | 0.000 | 0.667 |
Venezuela_3 | 0.857 | 0.400 | 0.500 | 0.000 | 0.643 |
Iran_5 | 0.435 | 0.353 | 0.000 | 0.286 | 0.545 |
Qatar | 0.381 | 0.154 | 0.364 | 0.533 | 0.500 |
Russia_3 | 0.500 | 0.000 | 0.000 | 0.000 | 0.500 |
Ecuador | 0.533 | 0.000 | 0.000 | 0.000 | 0.500 |
Iran_1 | 0.286 | 0.000 | 0.000 | 0.000 | 0.500 |
Russia_4 | 0.483 | 0.000 | 0.000 | 0.118 | 0.452 |
Egypt_UAE | 0.444 | 0.000 | 0.000 | 0.000 | 0.435 |
China_1 | 0.429 | 0.000 | 0 | 0.000 | 0.429 |
Iran_7 | 0.476 | 0.000 | 0.000 | 0.200 | 0.421 |
Bangladesh | 0.600 | 0.316 | 0.333 | 0.286 | 0.345 |
Iran_3 | 0.316 | 0.000 | 0.000 | 0.000 | 0.316 |
Iran_4 | 0.182 | 0.667 | 0.857 | 0.000 | 0.267 |
Russia_2 | 0.200 | 0.000 | 0.200 | 0.000 | 0.211 |
UAE | 0.200 | 0.000 | 0.000 | 0.000 | 0.182 |