Table 1 Language codes for both FastText-LID and Twitter-LID tools

Afrikaans

af

–

Albanian

sq

–

Amharic

am

am

Arabic

ar

ar

Aragonese

an

–

Armenian

hy

hy

Assamese

as

–

Asturian

ast

–

Avaric

av

–

Azerbaijani

az

–

Bashkir

ba

–

Basque

eu

eu

Bavarian

bar

–

Belarusian

be

–

Bengali

bn

bn

Bihari

bh

–

Bishnupriya

bpy

–

Bosnian

bs

bs

Breton

br

–

Bulgarian

bg

bg

Burmese

my

my

Catalan

ca

ca

Cebuano

ceb

–

Cherokee

–

chr

Central-Bikol

bcl

–

Central-Kurdish

ckb

ckb

Chavacano

cbk

–

Chechen

ce

–

Chinese-Simplified

–

zh-cn

Chinese-Traditional

–

zh-tw

Chinese

zh

zh

Chuvash

cv

–

Cornish

kw

–

Corsican

co

–

Croatian

hr

–

Czech

cs

cs

Danish

da

da

Dimli

diq

–

Divehi

dv

dv

Dotyali

dty

–

Dutch

nl

nl

Eastern-Mari

mhr

–

Egyptian-Arabic

arz

–

Emiliano-Romagnolo

eml

–

English

en

en

Erzya

myv

–

Esperanto

eo

–

Estonian

et

et

Fiji-Hindi

hif

–

Filipino

–

fil

Finnish

fi

fi

French

fr

fr

Frisian

fy

–

Gaelic

gd

–

Gallegan

gl

–

Georgian

ka

ka

German

de

de

Goan-Konkani

gom

–

Greek

el

el

Guarani

gn

–

Gujarati

gu

gu

Haitian

ht

ht

Hebrew

he

he

Hindi

hi

hi

Hungarian

hu

hu

Icelandic

is

is

Ido

io

–

Iloko

ilo

–

Indonesian

id

in

Inuktitut

–

iu

Interlingua

ia

–

Interlingue

ie

–

Irish

ga

–

Italian

it

it

Japanese

ja

ja

Javanese

jv

–

Kalmyk

xal

–

Kannada

kn

kn

Karachay-Balkar

krc

–

Kazakh

kk

–

Khmer

km

km

Kirghiz

ky

–

Komi

kv

–

Korean

ko

ko

Kurdish

ku

–

Lao

lo

lo

Latin

la

–

Latvian

lv

lv

Lezghian

lez

–

Limburgan

li

–

Lithuanian

lt

lt

Lojban

jbo

–

Lombard

lmo

–

Lower-Sorbian

dsb

–

Luxembourgish

lb

–

Macedonian

mk

–

Maithili

mai

–

Malagasy

mg

–

Malayalam

ml

ml

Malay

ms

msa

Maltese

mt

–

Manx

gv

–

Marathi

mr

mr

Mazanderani

mzn

–

Minangkabau

min

–

Mingrelian

xmf

–

Mirandese

mwl

–

Mongolian

mn

–

Nahuatl

nah

–

Neapolitan

nap

–

Nepali

ne

ne

Newari

new

–

Northen-Frisian

frr

–

Northern-Luri

lrc

–

Norwegian

no

no

Nynorsk

nn

–

Occitan

oc

–

Oriya

or

or

Ossetic

os

–

Pampanga

pam

–

Panjabi

pa

pa

Persian

fa

fa

Pfaelzisch

pfl

–

Piemontese

pms

–

Polish

pl

pl

Portuguese

pt

pt

Pushto

ps

ps

Quechua

qu

–

Raeto-Romance

rm

–

Romanian

ro

ro

Russian-Buriat

bxr

–

Russian

ru

ru

Rusyn

rue

–

Sanskrit

sa

–

Sardinian

sc

–

Saxon

nds

–

Scots

sco

–

Serbian

sr

sr

Serbo-Croatian

sh

–

Sicilian

scn

–

Sindhi

sd

sd

Sinhala

si

si

Slovak

sk

–

Slovenian

sl

sl

Somali

so

–

Shona

–

sn

South-Azerbaijani

azb

–

Spanish

es

es

Sundanese

su

–

Swahili

sw

–

Swedish

sv

sv

Tagalog

tl

tl

Tajik

tg

–

Tamil

ta

ta

Tatar

tt

–

Telugu

te

te

Thai

th

th

Tibetan

bo

bo

Tosk-Albanian

als

–

Turkish

tr

tr

Turkmen

tk

–

Tuvinian

tyv

–

Uighur

ug

ug

Ukrainian

uk

uk

Upper-Sorbian

hsb

–

Urdu

ur

ur

Uzbek

uz

–

Venetian

vec

–

Veps

vep

–

Vietnamese

vi

vi

Vlaams

vls

–

Volapük

vo

–

Walloon

wa

–

Waray

war

–

Welsh

cy

cy

Western-Mari

mrj

–

Western-Panjabi

pnb

–

Wu-Chinese

wuu

–

Yakut

sah

–

Yiddish

yi

–

Yoruba

yo

–

Yue-Chinese

yue

–

Undefined

und

und