Estimating work engagement from online chat tools

Abstract

The Covid-19 pandemic, caused by the SARS-Cov2- virus, has transformed our lives. To combat the spread of the infection, remote work has become a widespread practice. However, this shift has led to various work-related problems, including prolonged working hours, mental health issues, and communication difficulties. One particular challenge faced by team members is the inability to accurately gauge the work engagement (WE) levels of subordinates, such as their absorption, dedication, and vigor, due to the limited number of in-person interactions that occur in remote work settings. To address this issue, online communication systems utilizing text-based chat tools such as Slack and Microsoft Teams have gained popularity as substitutes for face-to-face communication. In this paper, we propose a novel approach that uses graph neural networks (GNNs) to estimate the work engagement levels (WELs) of users on text-based chat platforms. Specifically, our method involves embedding users in a feature space based solely on the structural information of the utilized communication network, without considering the contents of the conversations that take place. We conduct two studies using Slack data to evaluate our proposal. The first study reveals that the properties of communication networks play a more significant role when estimating WELs than do conversation contents. Building upon this result, the second study involves the development of a machine learning model that estimates WELs using only the architectural features of the employed communication network. In this network representation, each node corresponds to a human user, and edges represent communication logs; i.e., if person A talks to person B, the edge between node A and node B is stretched. Notably, our model achieves a correlation coefficient of 0.60 between the observed and predicted WEL values. Importantly, our proposed approach relies solely on communication network data and does not require linguistic information. This makes it particularly valuable for real-world business situations.

1 Introduction

Remote work has spread rapidly over the past few years among engineers and other company employees of various occupations. While remote work offers numerous advantages, such as improving the quality of life experienced by workes, it also decreases the frequency of in-person communication. This decline in face-to-face interaction poses difficulties for team members within companies, as it hampers their ability to gauge engagement levels with one another in a remote work setting [31, 34]. Work engagement (WE) is defined as a positive, fulfilling, work-related state of mind, encompassing aspects such as vigor, dedication, and absorption [41]. The level of work engagement (WEL) exhibited by each employee influences the business performance of a company [41]; a decrease in the WEL can adversely affect overall business outcomes. Therefore, it is crucial for companies to recognize and sustain the WEL of each employee.

The popular approach for assessing WELs involves administering a questionnaire containing 17 questions, with respondents rating each question on a 7-point scale from 0 to 6 [25] The total score obtained by summing the answers of a user represents their WEL, which can range from 0 to 102. However, this method has two significant drawbacks. First, completing the questionnaire is a time-consuming process for office workers, posing a challenge that we term the “troublesomeness problem”. Second, repeatedly using the same questionnaire increases the likelihood of individuals inferring relationships between specific questions and inflating their resulting WEL scores. This susceptibility to manipulation allows individuals to artificially influence their scores; we call this phenomenon a “questionnaire hack.”

Another promising approach involves utilizing linguistic information acquired from online communication platforms such as Slack¹ or Microsoft Teams.² Notably, there is an approach [37] that circumvents the need for a traditional questionnaire when estimating latent constructs through conventional methods. Rissola et al. [37] proposed the five-factor model [19, 29], which relies on a capsule neural network [26] to infer human personality traits from conversations. However, this approach necessitates access to the contents of conversation, raising valid concerns regarding confidentiality. We term this challenge the “confidentiality problem”.

In this paper, to address the challenges that are inherent in the existing approaches, we propose a new approach for estimating WELs utilizing text-based communication tools such as Slack or Microsoft Teams (Fig. 1). Importantly, our method circumvents the need for questionnaires or access to the contents of chat tool text. Instead, our proposed approach relies solely on communication network data, which depict the interactions between users and the frequencies of these interactions, to estimate WELs.

Figure 1

Overview of the proposed approach: From text-based communications log, we generate a communications network where a node is a user of a communications chat tool and an edge is a communications log. Then, we construct a dense vector that estimates work engagement level (WEL) of each user using a simple machine learning model. The WEL takes an integer value between 0 and 102

The core concept of our proposal draws inspiration from Lazer et al. [24] and Kramer et al. [23]. Lazer et al. [24] asserted that various human attributes, including behaviors and personalities, are influenced by social networks. For instance, Kramer et al. [23] indicated how emotions propagate within a social network, revealing that reducing positive advertisements lead to a decrease in cheerful postings and vice versa. These studies underscore the hypothesis that an individual’s WEL is influenced by their interactions with others. Thus, by capturing the positions and interactions of users within their workplace community, even without access to text information from chat tools, we can estimate their WELs.

Let us focus on this approach. We start by extracting a communication network derived from online chat tools, where each user is represented as a node, and edges denote communication interaction logs; i.e., if human A talks to human B, an edge is present between node A and node B. In this preliminary study, we imply that the graph architectural features of the communication network play a more crucial role in estimating WE than does linguistic information. To verify this assertion, we compare the correlation coefficient between the WEL differences and feature vector similarities derived from graph properties to the correlation coefficient between the WEL differences and feature vector similarities derived from linguistic information. In Sect. 4, we present our findings, demonstrating that the correlation coefficient between the differences among WELs and the similarities among graph architectural feature vectors is greater than the correlation coefficient between WELs and the similarity of linguistic feature vectors.

According to the preliminary study, we employ graph neural networks (GNNs), which have achieved remarkable performance in cases with various graph-structured data [15, 21, 45], to enhance our understanding of the people within the communication network. By employing GNNs, we generate dense vector representations for each user in the network. The preliminary study results suggest that an individual’s communication pattern is more informative for estimating their WE than the chat contents of their conversations. Consequently, we develop a machine learning model that uses these individual representations to predict WE. Since no communication chat tool dataset is readily available and accessible, we collect datasets from three distinct organizations to evaluate the proposed approach utilizing actual Slack data. Organization 1 (Org 1) and organization 2 (Org 2) are the research and development departments of NTT DOCOMO INC., while Org 3 comprises six different companies. It is noteworthy that most employees across these organizations predominantly work from home and mainly use Slack for communication alongside Google Meet for online meetings. Detailed descriptions of the datasets are provided in Sect. 3. Through our evaluation, the proposed approach demonstrates the ability to estimate WELs without relying on questionnaires or confidential content, achieving a Pearson correlation coefficient of 0.66 between the observed and predicted values.

This paper is organized as follows. Section 2 provides a review of the related work. This is followed by Sect. 3, which describes the datasets used in our study. In Sect. 4, we present a preliminary study indicating that graph architectural features are more important than linguistic features for estimating WE. Subsequently, in Sect. 5, we propose an approach for estimating WELs based solely on the architectural information of communication networks, and this is followed by an evaluation conducted using actual Slack data. Finally, Sect. 6 discusses the limitations of our study and offers conclusions. The main contributions of this paper are summarized as follows.

• We demonstrate that the person to whom an employee converses is more significant for estimating their WE than the conversation content contained within a chat tool.

• We propose a novel approach for estimating WELs based on individual representations, thus bypassing the need for linguistic information.

• We collect actual Slack datasets from three organizations and evaluate our proposed prediction model.

2 Related work

WE is a concept of the human state of mind, making it a significant construct in psychological research. Although the typical method for estimating such a construct involves questionnaires, several studies have explored alternative methods. Therefore, we review three types of related work: the conceptualization and understanding of WE as a construct (Sect. 2.1), the analysis of the constructs related to WE (Sect. 2.2), and the diverse methodologies used to estimate WE (Sect. 2.3).

2.1 Work engagement and related constructs

The concept of WE was initially proposed by Maslach and Leiter [27]. Subsequently, Schaufeli et al. [41] refined this concept, establishing the mainstream definition of work engagement as follows: Work engagement “is defined as a positive, fulfilling, work-related state of mind that is characterized by vigor, dedication, and absorption. Rather than a momentary and specific state, engagement refers to a more persistent and pervasive affective cognitive state that is not focused on any particular object, event, individual, or behavior.” Here, vigor is characterized by high levels of energy and mental resilience while working, dedication refers to being strongly involved in one’s work and experiencing a sense of significance and pride, and absorption is characterized by being fully concentrated and happily engrossed in one’s work. Additionally, Schaufeli et al. [41] claimed that engaged employees have senses of energetic and effective connection with their work activities.

Research conducted by Leiter [25] showed that the WEL is correlated with organizational commitment, performance, health and turnover intentions. Additionally, Shimazu et al. [43] showed that WE is related to improvements in job performance and reductions in well-being concerns. Notably, in the research of Shimazu et al., well-being encompassed both physical health and high life satisfaction levels. These findings underscore the importance of monitoring and enhancing the WELs of employees, as they directly impact organizational outcomes and employee well-being.

Unfortunately, maintaining updated WELs poses challenges due to the exclusive reliance on the questionnaire method. This reliance gives rise to the troublesomeness problem and the questionnaire hack. The main questionnaire-based methods for assessing WELs include the Maslach Burnout Inventory-General Survey [27], Oldening Burnout Inventory [9, 10] and Utrecht Work Engagement Scale (UWES) [41]. While some researchers [40, 42] have managed to reduce the numbers of questions contained in these assessments, none have provided a methodology that circumvents the need for a questionnaire altogether.

The proposed approach aims to solve the aforementioned problem by leveraging the propagation of WE. Research [6, 48] has shown that both negative and positive emotions propagate in marriages. Bakker et al. [2, 3] also showed that engagement propagates through marriage. Moreover, Bakker et al. [4] reported that there is a positive correlation between the work engagement of a team and the work engagement of the individuals working on the team. This result suggests that the WEL of a person should be influenced by the people surrounding them. Gutermann et al. [14] focused on the propagation of WE between supervisors and subordinates. Gutermann et al. [14] validated how supervisors’ WELs propagate to their subordinates, revealing that high supervisor WELs lead to good relationships and subsequently to high WEL for their subordinates. All of the above studies were performed in face-to-face communication settings; however, these results might be applicable to remote work settings. Namely, WE should propagate through online communication on chat tools. Therefore, we utilize a communication network to estimate WELs.

2.2 Online communication

As indicated in the preceding section, WE propagates through face-to-face communication. Kramer et al. [23] showed that emotions can spread within social networks. Specifically, they showed that reducing exposure to positive advertisements resulted in a decreased number of cheerful postings and vice versa. Their work also suggested that in-person interactions and nonverbal cues are not strictly necessary for emotional contagion and that observing the positive experiences of others constitutes a positive experience for the observers.

Additionally, in the domain of analyzing group chat conversations, several studies have focused on disentangling interleaved conversational threads to form threads discussing single topics [11, 18, 28, 44] and extracting knowledge from conversational dialogue [16]. However, in actual text-based chat communication tools such as Slack, interactions are often not one-on-one, and discussions typically encompass multiple topics. Therefore, the settings in the abovementioned studies do not align with the problem setting addressed in this paper.

In previous research addressing a problem setting similar to that considered herein, Wang et al. [47] developed a machine learning model to classify the communication styles observed in actual slack data. Subsequently, they established a machine learning model to predict team performance based on these communication styles. Wang et al. [47], we use only Slack metadata, e.g., active timespans in days, the number of bot messages, and the number of @here messages, for our analysis.

As discussed in Sect. 2.1, the WELs of individual office workers are related to their job performance [25, 43]. Additionally, Wang et al. [47] showed that team performance can be estimated using Slack metadata. Building upon these findings, we assume that WE can likewise be estimated using Slack metadata. Furthermore, Kramer et al. [23] declared that emotions propagate through social networks; therefore, we consider that WELs can also be estimated using the architectural property of a communication network.

2.3 Methods for estimating constructs

Despite not directly targeting WE, several studies [12, 17, 22, 32, 38] have estimated individual scores without using a questionnaire. Saha and De Choudhury [38] developed a machine learning model to estimate stress levels from text posted on social media, while Fatima et al. [12] also developed machine learning models to classify depression. Moreover, Iacobelli et al. [17] revealed that n-gram features can be used to assess personalities from the text of personal blogs; additionally, Kosinski et al. [22], Neuman and Cohen [32], and Golbeck et al. [13] proposed methods for estimating personalities from text information. These studies imply that language information can be helpful for estimating WE; therefore, we aim to compare the method that utilizes only network structural features to estimate WELs with the method that utilizes linguistic features.

Multiple studies [8, 39, 50] used sensor data to estimate individual scores without relying on questionnaires. For example, Salekin et al. [39] developed a weakly supervised learning framework for detecting social anxiety and depression from audio clips using a novel feature modeling technique. DeMasi and Recht [8] modeled the relationships between user characteristics and peoples’ daily mental well-being scores from smartphone GPS data. Additionally, Zakaria et al. [50] estimated stress and depression levels from the Wi-Fi system data of a university. However, a challenge arises when employing sensor-based approaches as in the research described above, which are not considered adequate for estimating WELs. This limitation reduces the accessibility of data, as some employees may not physically attend an office.

3 Data

In this study, we develop a machine learning model that predicts WELs utilizing Slack data. To train the model, we extract text data from the Slack data of several Japanese companies with their consent.

Slack contains two team layers. The first layer is referred to as a workspace (WS) containing all department members, while the second layer represents a channel containing project-based members. Two types of channels are included: public and private channels. Private channels require permission for access, and all members of a WS can view public channels, thereby mitigating privacy concerns.

3.1 Dataset configurations

In our study, three WSs are considered. The WSs of Org 1 and Org 2 correspond to the research and development departments of NTT DOCOMO Inc., respectively. Additionally, the WS of Org 3 is used by six different companies. The Slack data were collected from three organizations, and subsequently, users from each organization were requested to complete the UWES to assess their WELs. Note that all WSs are used for work communications; therefore, as a general rule, all communications are work-related conversations.

A summary of the datasets is shown in Table 1. The WS of Org 1 contains 139 workers, and we gathered WELs from 72 workers. The Slack data of Org 1 were collected from 2020-07-01 to 2021-06-30. The WS of Org 2 contains 93 workers, and the data were collected from 2021-04-01 to 2021-10-09. We collected WELs from 31 workers. The WS of Org 3, includes 19 workers, and the data were collected from 2021-09-01 to 2021-10-30. We collected WELs from 19 workers.

Table 1 Summary of Datasets: Org 1 and Org 2 are two different departments in the same company. Org 3 is a community where people belong to different companies

3.2 Survey process

We asked all users in each workspace to complete a questionnaire using the Japanese version of UWES [42] and administered the questionnaire only to those who provided consent. Here, although Shimazu et al. [42] recommended using a short 9-item version of the UWES, we employed a 17-item version because the Cronbach alpha value of the 17-item version is greater than that of the 9-item version. Specifically, the Cronbach alpha value of the 17-item version is 0.87, and that of the 9-item version is 0.83.

3.3 Preprocessing

In this study, we extract all public channel data, which includes embedded vectors of text messages, message senders, and receivers. The text message vector $\boldsymbol{z}\in \mathbb{R}^{256}$ is constructed by using Sentence-BERT [36], which enables the transformation of sentences into a vector. For details, a Slack administrator (excluding the authors) embeds messages in a post-by-post manner via Sentence-BERT; subsequently, we obtain the linguistic feature vector of each user by taking the average of their embedded vectors. Within the text messages, chat mention markers such as @[user name] are identified to determine the users mentioned in each message. If no markers are present in the text, it is assumed that the speaker addresses all users in the channel. Additionally, Japanese character representations such as [user name]-san, [user name]-kun, and [user name]-sama are considered chat mention markers. Moreover, meeting links, Slack-specific commands such as <#channel_link>, and emoji commands are filtered out.

4 Study 1: analysis of the relative merits between graph-architectural features and linguistic features

As outlined in Sect. 1, the proposed approach is based on the fundamental hypothesis that WE is influenced more by an individual’s position within a communication network than by the contents of their communications. In other words, graph architectural properties play a more critical role in predicting WE than does linguistic information. However, it is possible that the contents of conversations influence WELs, as suggested by some studies [7, 49] that indicated the importance of linguistic usage when assessing human personalities. Therefore, we conduct a comparative analysis between the architectural features and linguistic features of communication networks.

The graph architectural features comprise both node- and edge-related features. We derive frequency features as some of the edge-related graph architectural features:

$$ x_{i} = \frac{(\#\text{ of mentions to users belonging to class }i)}{(\#\text{ of mentions to all users})}, \quad i=1, 2, 3, 4. $$

Feature $x_{i}$ represents the frequency with which a user interacts with users belonging to the i-th class, where the dataset is split into four classes based on the 25th, 50th and 75th WEL percentiles. Similar to [23], increased interactions with people possessing higher WELs are likely to improve the corresponding WEL. Therefore, we employ frequency features to capture this phenomenon. The graph architectural feature related to nodes, $x_{5}$, represents degree centrality, which is a widely used measure in graph theory that indicates the centrality of a node within a graph. A higher degree centrality score implies that a user communicates with more individuals.

4.1 Experimental settings for Study 1

To compare graph architectural features with linguistic features, we verify the relationship between user vector similarities and WEL differences. Specifically, if graph architectural features are more significant to WELs than linguistic features are, then for any users with similar WELs, their graph architectural features should be closer to each other than the vectors of their linguistic features are. Formally, suppose that $\boldsymbol{x}_{n} = \left [ x_{1}, \dots , x_{5}\right ]$ denotes the graph architectural feature vector of user n, and $\Delta \text{WE}\, \left (n, m\right )$ represents the difference between the WELs of users n and m. If the correlation between $\text{sim}\left ( \boldsymbol{x}_{n}, \boldsymbol{x}_{m}\right )$ and $\Delta \text{WE}\, \left (n, m\right )$ is greater than the correlation between $\text{sim}\left ( \boldsymbol{z}_{n}, \boldsymbol{z}_{m}\right )$ and $\Delta \text{WE}\, \left (n, m\right )$ under the condition that the correlation coefficients between $\text{sim}\left ( \boldsymbol{x}_{n}, \boldsymbol{x}_{m}\right )$ and $\Delta \text{WE}\, \left (n, m\right )$ are somewhat large, then the graph architectural features can effectively represent users with similar WELs as having similar vectors. Therefore, in such instances, the ability of graph architectural features to predict WE should better reflect WELs than should the use of linguistic features.

The experimental procedure is outlined as follows. First, we calculate the $\text{sim}\left ( \boldsymbol{x}_{n}, \boldsymbol{x}_{m}\right )$ and $\Delta \text{WE}\, \left (n, m\right )$ pairs and the $\text{sim}\left ( \boldsymbol{z}_{n}, \boldsymbol{z}_{m}\right )$ and $\Delta \text{WE}\, \left (n, m\right )$ pairs produced for all combinations of users. Next, we compare the correlation score $\rho \left ( \text{sim}_{\boldsymbol{x}},\Delta \text{WE} \right )$ between $\text{sim}\left ( \boldsymbol{x}_{n}, \boldsymbol{x}_{m}\right )$ and $\Delta \text{WE}\, \left (n, m\right )$ to the correlation score $\rho \left ( \text{sim}_{\boldsymbol{z}},\Delta \text{WE}\right )$ between $\text{sim}\left ( \boldsymbol{z}_{n}, \boldsymbol{z}_{m}\right )$ and $\Delta \text{WE}\, \left (n, m\right )$. If $\rho \left ( \boldsymbol{x},\Delta \text{WE}\right )$ is larger than $\rho \left ( \boldsymbol{z},\Delta \text{WE}\right )$, we determine that the graph architectural features are more critical for predicting WE than linguistic information is.

4.2 Evaluation of Study 1

The scatter plot of the $\text{sim}\left ( \boldsymbol{x}_{n}, \boldsymbol{x}_{m}\right )$ and $\Delta \text{WE}\, \left (n, m\right )$ values calculated using the Org 1 dataset is shown in Fig. 2a, while the scatter plot of the $\text{sim}\left ( \boldsymbol{z}_{n}, \boldsymbol{z}_{m}\right )$ and $\Delta \text{WE}\, \left (n, m\right )$ values calculated using the same dataset is shown in Fig. 2b. The correlation scores $\rho (\text{sim}_{\boldsymbol{x}}, \Delta \text{WE})$ and $\rho (\text{sim}_{\boldsymbol{z}}, \Delta \text{WE})$ are −0.43 and 0.06, respectively. Comparing the correlation scores, it is observed that for graph architectural features, there is a typical tendency for the similarity of the graph architectural feature vectors to decrease when the WE difference becomes larger, which contrasts with the similarity of the linguistic feature vectors.

Figure 2

Comparison between the graph architectural feature and the linguistic feature. The horizontal axis means the difference in WELs of two users, and the vertical axis means the similarity of the corresponding two users

According to Fig. 2a, as the WE difference increases, there is a tendency for the similarity to decrease monotonically in the region where the WE difference is less than approximately ten. However, in the other regions, the dispersion of the similarities becomes more pronounced. The reason behind this phenomenon remains unclear, and a further detailed analysis is necessary.

However, as depicted in Fig. 2b, the similarities are predominantly concentrated in the range of $[0.8, 1]$, irrespective of the WE differences. Therefore, the linguistic feature vector fails to adequately capture the WE differences. Study 1 summarizes the following points. Graph architectural features demonstrate a better ability to represent WELs than do linguistic features. Therefore, we conclude that graph architectural properties are more important than linguistic information for predicting WELs.

5 Study 2: predictability analysis of WE

5.1 Proposed approach

According to Study 1, the architectural features of a communication network exhibit greater relevance to WE than do the linguistic features. Therefore, it is logical and justifiable to utilize the architectural features of a communication network to estimate WELs.

Incidentally, in Sect. 4, we constructed features manually; however, representation learning through the use of GNNs, which autonomously structure feature vectors, has demonstrated success across various domains. Leveraging GNNs for generating the architectural feature vectors of a communication network offers the potential to estimate WELs with higher quality than those produced by manual features. Therefore, we propose an approach that constructs feature vectors for individuals through representation learning using GNNs. These are termed vector individual representation, and we develop a prediction model based on a popular machine learning model algorithm.

Since the process of constructing individual representations does not rely on linguistic features, the proposed approach mitigates the concerns regarding the exposure of the sensitive information contained in chat communication tools. Moreover, although manually extracting the architectural features of a communication network, as in Study 1, requires determining the WELs of other people via questionnaires, constructing individual representations circumvents this requirement. As no questionnaire is needed, the proposed approach alleviates the burden imposed on office workers. Moreover, given the complexity of the GNN algorithm, it is difficult for office workers to directly control the estimated WELs. Therefore, the proposed approach addresses issues related to the burdensome nature of questionnaires, including potential manipulation and confidentiality problems.

5.2 Preliminaries

As discussed in the previous section, the proposed approach comprises two phases: constructing individual representations using GNNs and estimating WELs using a machine learning model. To conduct representation learning using GNNs, we employ DeepWalk [35] and Deep Graph Infomax [46]. We consider an undirected graph $\mathscr{G}=(\mathscr{V}, \mathscr{E})$ with N nodes to represent the communication network; namely, $\mathscr{G}$ and $v_{n} \in \mathscr{V}$ express an organization and the number of workers in the organization, respectively. Additionally, if worker $v_{n_{0}}$ has communicated with $v_{n_{1}}$ or $v_{n_{1}}$ has talked to $v_{n_{0}}$, an edge $e_{n_{0} n_{1}} \in \mathscr{E}$ is generated between $v_{n_{0}}$ and $v_{n_{1}}$.

5.2.1 DeepWalk

In DeepWalk, we apply a random walk on $\mathscr{G}$ to generate sequences. Subsequently, we employ the SkipGram model [30] to learn the embedded vector of vertex $v \in \mathscr{V}$. In this study, we incorporate the transition probability from $v_{n_{0}}\in \mathscr{V}$ to $v_{n_{1}}\in \mathscr{V}$ by weighting it with the frequency of conversation from $v_{n_{0}}$ to $v_{n_{1}}$:

$$\begin{aligned} \frac{ (\# \text{ of talking from }v_{n_{0}} \text{ to } v_{n_{1}}) }{\displaystyle \sum _{\substack{n=1 \\ n\neq n_{0}}}^{N}\left (\# \text{ of talking from } v_{n_{0}} \text{ to } v_{n} \right ) }. \end{aligned}$$

5.2.2 Deep graph infomax

In Deep Graph Infomax, a node is represented as $\boldsymbol{x}_{n}$, where $n=1,\dots , N$:

$$ X := \begin{bmatrix} \boldsymbol{x}_{1} & \cdots & \boldsymbol{x}_{N} \end{bmatrix} , $$

and A is an adjacency matrix. It is important to note that the xn in this section is distinct from the $x_{i}$ in Sect. 4.

The central concept behind Deep Graph Infomax is to maximize the local mutual information between a node representation, which captures local graph information, and the corresponding graph representation, which captures global graph information. This strategy ensures that the learned node representation effectively captures global graph information to the greatest extent possible.

Let E be a graph encoder using graph convolutional networks (GCNs) [21]. Consequently, we obtain node representations as $H=E(X, A)= \begin{bmatrix} \boldsymbol{h}_{1} & \cdots & \boldsymbol{h}_{N} \end{bmatrix}$. Since a k-layer GCN leverages node information located up to k-hops away, the vector $\boldsymbol{h}_{n}$ is assumed to encapsulate the local information surrounding node n. In contrast, to acquire global graph information, we utilize $\boldsymbol{s}:=\sum _{n=1}^{N}\boldsymbol{h}_{n}$.

As stated at the outset of this section, we maximize the mutual information:

$$ \int \int p\left ( \boldsymbol{h}, \boldsymbol{s} \right ) \log \left ( \frac{p\left ( \boldsymbol{h}, \boldsymbol{s} \right )}{p\left ( \boldsymbol{h}\right ) p\left ( \boldsymbol{s} \right )} \right ) \, \mathrm{d}\boldsymbol{h}\mathrm{d}\boldsymbol{s}, $$

(1)

where p represents the probability density function. However, solely maximizing Equation (1) is insufficient for learning useful representations. Therefore, in Deep Graph Infomax, we maximize

$$\begin{aligned} &\frac{1}{N_{0} + N_{1}}\left ( \sum _{j: \text{positive sample}} \mathbb{E}_{\left ( X, A \right )}\left [ \log \mathscr{D}( \boldsymbol{h}_{j}, \boldsymbol{s}) \right ] \right . \\ &\left . \qquad\qquad {}+ \sum _{j: \text{negative sample}} \mathbb{E}_{\left ( \tilde{X}, \tilde{A} \right )}\left [ \log \left ( 1 - \mathscr{D}( \boldsymbol{h}_{j}, \boldsymbol{s}) \right ) \right ] \right ), \end{aligned}$$

(2)

where $\mathcal{D}$ is a binary classifier that predicts whether a pair $\boldsymbol{h}_{j}$ and s comes from the joint distribution $p(\boldsymbol{h}_{j}, \boldsymbol{s})$ (positive class) or is the product of the marginal distributions $p(\boldsymbol{h}_{j})p(\boldsymbol{s})$ (negative class). We denote $\tilde{\boldsymbol{h}}_{j}$ as the j-th node embedded from the negative sample, and the numbers of positive and negative samples are denoted as $N_{0}$ and $N_{1}$, respectively. Consequently, by minimizing Equation (2), we optimize the GNN parameters.

5.3 Experimental configuration for Study 2

As outlined in the previous section, the proposed approach comprises two phases: constructing individual representations for representation learning and estimating WELs using a machine learning model. To conduct representation learning using GNNs, we specifically employ DeepWalk [35]. In contrast, as a competitive method using only linguistic features, we utilize an embedded vector generated by Sentence-BERT [36]. Additionally, as a hybrid method using individual representation and linguistic features, we consider a blend consisting of a GCN, DeepWalk, and Deep Graph Infomax [46]. This hybrid method combines a GCN and DeepWalk to embed nodes of the communications network with the feature vectors derived from Sentence-BERT, followed by embedding the resulting vector via DeepWalk. To develop the tree-based machine learning model, we use the LightGBM [20], which is a widely used and influential framework for gradient boosting decision trees.

We evaluate the proposed approach using real datasets obtained from Org 1, Org 2, and Org 3, as described in Sect. 3. The evaluation procedure is the so-called leave-one-out cross-validation method, which is outlined as follows.

• 1) Construct an individual representation.

• 2) Leave out one user and split the remainder of the dataset into a training set and a validation set at a ratio of 7:3.

• 3) Train the estimation model using the training set and tune the hyperparameters using the validation set via Optuna [1], which was designed for black-box optimization tasks using the tree-structured Parzen estimation approach [5].

• 4) Perform prediction based on the remaining individual user representation.

The evaluation criterion utilized in this study is the Pearson correlation coefficient between the predicted WE and the ground-truth WE. This choice is motivated by the absence of definitive thresholds for categorizing WE as good or bad in real-world scenarios where assessing WELs is needed. In this context, the ground truth is represented by the WEL collected via the original 17-item version of the UWES questionnaire, as described in Table 1.

The utilization of three different organizational datasets serves the purpose of confirming the generalizability of the proposed approach. Organizational structures and dynamics can vary significantly from one organization to another. Hence, assessing the proposed approach using GNNs is an applicable approach for estimating the WELs across various organizational settings

5.4 Evaluation of Study 2

The experimental results shown in Table 2 indicate that the GNN-based embedding methods, including DeepWalk, Deep Graph Infomax, and DeepWalk-GCN, outperform the embedding method that relies solely on linguistic features, i.e., Sentence-BERT. Among these GNN-based methods, DeepWalk, which solely utilizes individual representations, yields the highest scores for Org 1 and Org 2. However, the difference between DeepWalk and Deep Graph Infomax, which use both the individual representations and embedded vectors from Sentence-BERT, is minimal. Based on these findings, the proposed approach can estimate WELs with nearly the same quality as that achieved by the method using both the architectural property and linguistic features of the communication network while relying solely on the architectural properties.

Table 2 Results of Study 2: The table displays the Pearson correlation coefficients between the predicted WE and ground-truth WE obtained using the LightGBM to estimate the WE. The approaches that solely rely on the architecture features of communication networks, DGI and DeepWalk, achieve higher scores than those of Sentence-BERT, which solely utilizes linguistic features. Additionally, DeepWalk-GCN, which uses both types of features, exhibits the second-highest score in the evaluation

The hybrid method combining DeepWalk-GCN and Deep Graph Infomax, which uses both types of features, achieves the second-best and third-best scores for Org 1 and Org 2, respectively. Additionally, the differences among DeepWalk, Deep Graph Infomax, and DeepWalk-GCN for Org 1 and Org 2 are marginal. However, there are substantial differences between the scores of DeepWalk-GCN and Sentence-BERT. This result indicates that the impact of linguistic features is less significant than that of communication network properties.

In Org 3, the embedding rankings follow the order Sentence-BERT > DeepWalk-GCN > Deep Graph Infomax > DeepWalk. This indicates that embedding methods using linguistic features outperform those that do not. The distinguishing factor between Org 3 and the other two organizations lies in the duration of the Slack data collection period. Table 1 shows that the Slack data for Org 1 and Org 2 were collected over one and a half years and half a year, respectively, while the data for Org 3 were collected in just two months. Due to this short duration, the communication network is not sufficiently established, hindering the use of embedding methods that utilize the architectural features of communication networks to generate meaningful vectors. Moreover, the overall poor estimation performance achieved for Org 3 can also be attributed to its small number of users.

As discussed in Sect. 5.4, within Org 3, both the proposed approach and the competitive approach using Sentence-BERT do not perform satisfactorily. In fact, the competitive approach outperforms the proposed approach. This result implies that the proposed approach struggles to accurately estimate WE when the data collection period is brief. To address this limitation, we perform a simple experiment using the Org 1 dataset. Specifically, we divide the dataset into periods and construct individual representations accordingly. We then compare the correlation coefficient between the predicted WE and the ground-truth WE based on the collection period. The results of this experiment are shown in Fig. 3.

Figure 3

Transitions of the correlation coefficients. The collection period for the dataset used to construct the individual representations is represented on the abscissa. For example, the 2-month dataset was collected from 2021-06-01 to 2021-07-30. The correlation coefficient of DeepWalk tends to be high since the period of the utilized dataset is long. In contrast, the correlation coefficients of Deep Graph Infomax and DeepWalk-GNN peak earlier than that of DeepWalk, so they tend to decrease

Figure 3 clearly shows that the correlation coefficient for DeepWalk tends to increase as the period of the utilized dataset used increases. This result suggests that the proposed approach is likely to achieve higher correlation coefficients for longer-duration datasets. However, it is noteworthy that the rates of change exhibited by the correlation coefficient for up to three months are greater than those after four months. Therefore, the proposed approach requires a dataset to be collected for at least three months.

In contrast to DeepWalk, the correlation coefficients obtained for Deep Graph Infomax and DeepWalk-GNN peak earlier, leading to subsequent declines. This result suggests that the embedding method relying on linguistic features requires the latest communications dataset.

The individual representations generated by the model may not adequately express the WELs of representative individuals, even though they can effectively represent the WELs of most people. For example, the WEL of user $u_{1}$ is 92, and that estimated by the model is 89; the WEL of $u_{2}$ is 73, and that estimated by the model is 96 in the experiment of Study 2. In such cases, the model struggles to accurately estimate the WEL of $u_{2}$; therefore, in such a case, the individual representations cannot represent the difference between the WELs of $u_{1}$ and $u_{2}$. We visualize the communication frequencies of $u_{1}$ and $u_{2}$ in Fig. 4. According to Fig. 4, the communication partners of $u_{1}$ and $u_{2}$ are similar; actually, the similarity of these vectors is approximately 0.86, as shown in Fig. 2a. In other words, despite the high similarity between these vectors, the model results in a significant estimation error. To address such a case, we may have to utilize some features that can represent the differences between similar users in terms of their communication styles.

Figure 4

Frequencies at which $u_{1}$ and $u_{2}$ talk to everyone. Although the WEL of $u_{1}$ is 52 and that of $u_{2}$ is 42, these two frequencies are similar

6 Limitations, discussion, and conclusion

6.1 Limitations

This paper implicitly assumes that the WELs measured by a questionnaire are reasonable and appropriate. Although we confirm that Cronbach’s alpha is greater than 0.8, the discussion of the validity of the questionnaire is kept to a minimum; for example, we do not discuss construct validity. Therefore, there is room for further considering the validity of this strategy.

In addition to evaluating the validity of the output WELs, it is crucial to acknowledge the limitations of online chat tools such as Slack in terms of capturing all communications. As discussed in Sect. 2, previous research [2–4, 14] has shown that WE should propagate among people and that emotions should spread through social networks. This insight led us to hypothesize that WE might propagate through an online communication platform, and we propose an approach utilizing a communication network. However, it is important to recognize that online communication tools cannot capture all communications; for example, discussions in meeting rooms are not reflected by these platforms. Therefore, our proposed approach alone may not fully estimate the WELs of employees.

We must also consider the organizational context. Given that all members in our study belong to Japanese organizations, their communication style might be distinctive, potentially limiting the generalizability of our findings. Moreover, depending on the primary mode of communication used in a workplace, the efficacy of the proposed approach may vary. The dataset used in our study was obtained from an organization operating in a completely remote work setting, with Slack serving as the primary communication platform. However, the results of this study could differ significantly for organizations primarily engaging in face-to-face interactions within office settings.

6.2 Discussion

As outlined in Sect. 2, the concept of WE was defined by Schaufeli et al. [41] in 2002. Moreover, the concept of WE has evolved through various studies. For example, while we assume that one WEL should be measured for each individual in our study, Newton et al. [33] argued that WE depends on the given work task. Adhering to the original definition, we define WE based on a one-to-one interpersonal criterion. However, if the WE varies depending on the task, the factors influencing the associated WELs may vary. Therefore, it is imperative to validate WE as a feature in future work.

Next, we delve into the estimates provided by the proposed approach, as shown in Fig. 3. Despite establishing the necessity of long-term data for our approach in Sect. 5.4, as evidenced by the improved performance achieved by DeepWalk beyond the 7-month scale on the horizontal axis of Fig. 3, DeepWalk outperforms the other methods even within the initial 1-month scale. This can be attributed to the fact that although other models incorporate linguistic features, these features prove to be less useful for generating individual representations during the initial organizational interaction phase, as many workers tend to discuss similar topics during this period.

As a constructive analysis, the proposed approach for estimating WELs according to individual representations, developed via a communication network, holds potential for applications in various tasks. This approach can replace questionnaire-based methods for assessing various constructs, such as WE and overall well-being. Hence, it is prudent to investigate the extent to which architectural graph features can predict other latent constructs related to organizational behavior.

6.3 Conclusion

In this paper, we propose a novel approach that constructs individual representations solely from the architectural properties of communication networks. Subsequently, we utilize these individual representations to estimate WE without relying on questionnaire-based methods.

For evaluation purposes, we collected real Slack data from three organizations along with the ground-truth values of their WELs using a questionnaire method. With these datasets, we conducted two studies: Study 1 and Study 2.

In Study 1, we compared the Pearson correlation coefficients between graph architectural features and linguistic features. This comparison revealed that the architectural features of the communication network have greater significance when predicting WELs than do linguistic features.

In Study 2, leveraging the insights gained from Study 1, we proposed an approach that uses GNNs to construct individual representations and predicts WE. When constructing the individual representations, the architectural property of the target communication network, which expresses who is talking to whom and how often, is needed. In addition, we evaluated the proposed approach and showed that it could estimate WE. The results suggest that WE can be estimated through the sole use of individual representations, without relying on linguistic features. By eliminating the need for questionnaires and linguistic features, our approach circumvents the inconvenience, troublesomeness, questionnaire hack, and confidentiality problems associated with traditional questionnaire methods.

7 Broader perspective, ethics and competing interests

We believe that effectively estimating WE will positively impact managerial strategies for managing subordinates in the future when our model is more extensively evaluated and the approach is thoughtfully implemented. However, there is a serious potential negative impact: managers might relegate individuals with low WELs to lower positions. To mitigate such situations, users utilizing our model must understand the nature of WE; i.e., it is not related to the ability of employees to work or their limitations, and the accuracy of the model is not perfect. Furthermore, managers should refrain from making deployment decisions based only on WE.

Before collecting the data in Sect. 3, a consensus regarding the aim and analysis of this study was obtained from all participants. No physical or mental intervention was required of the participants. Prior to conducting the analysis, the participant information was unlinkable, anonymized, and deidentified.