Methods for quantifying effects of social unrest using credit card transaction data

The routine of daily life is punctuated by extraordinary public events, ranging from major sports and cultural events, to terror attacks or disasters. Some events are local, and some are national or even wider; some affect only part of the population, while others affect the entire population. Although unlike natural events and disasters, certain social events, such as a protest, a riot, a large police action, or a bomb explosion can still have characteristics of natural emergencies, with fatalities and damage to properties. However, when such localized social shock occurs, we have mostly qualitative or at best retrospective survey data about its effect on the surrounding community. Consequently it is impossible to know how well authorities managed the side-effects of the event, and thus difficult to develop effective event policies for use by police and municipal authorities.

In particular, understanding the effects of major social shocks or events on the economy and consumer behavior can have important implications [1–7]. Existing measures of economic behavior and attitudes towards economic situations use surveys to assess consumer confidence or consumer sentiment, asking about the intentions to purchase goods or to make large investments. These include objective measures published by national agencies, such as the measure of personal consumption expenditures in the monthly personal income and outlays report, provided by The Bureau of Economic Analysis (BEA) [8]. Other indicators are based on consumers’ subjective reports, such as the monthly consumer confidence index (CCI) issued by The Conference Board [9], and the monthly index of consumer sentiment (ICS) published by the University of Michigan [10]. These data and the actual economic behavior, as expressed in consumption, can be strongly correlated [11], and they can therefore be used to evaluate the effects of events. However, their temporal resolution is low (consumer confidence and sentiment surveys are usually conducted on a monthly base). Their spatial resolution is low, too, since the measures aggregate information over large areas and usually entire economies. Finally, the information usually cannot be used to look at the differential effect of events on different segments of a society.

The limitations of traditional approaches may be overcome by using digital tools to study human and social behavior, analyzing large-scale quantitative data in emerging fields such as computational social science [12, 13]. The analysis of large amount of data accumulated in social networks, cellular phone records, or credit card transactions may allow us to gain a new understanding of social processes. Such data have been widely utilized to study situation awareness and response to emergencies [14–18], mostly due to natural events and disasters, as well as the dynamics of communication and information propagation immediately followed [19–22]. However, relatively little work, so far, aimed to understand the impact of social events, and especially localized social shocks, using quantitative behavioral data.

Einav et al. recently highlighted the new possibilities of conducting economic research in the age of “Big Data” [23]. Within this scope, in this work, we show how a particular type of data, namely credit card transaction records, can be used to quantify and understand the repercussions localized social events have on individuals’ economic behavior, quantifying when, where and how an event has an effect. In particular, credit card data help us overcome the limitations of traditional indicators by allowing us to measure purchase behavior directly, and, together with information about the types of merchants at which purchases are made, to infer what kinds of products are purchased. Especially, purchases at physical shops (rather than those made online) are clearly linked to specific spatial locations and time stamps. Furthermore, credit card issuers have information about the demographics and economic status of the credit card holder, which enables us to measure the effects of events on different demographic groups in the population. This would allow the authorities to develop targeted and timely policies in case of emergencies.

To demonstrate the use of this method, we analyzed more than ten million credit card transaction records provided by a major financial institution in an OECD country about more than 100 thousand individuals, at more than 100 thousand merchants during a period of three months. Each record consists of the date, time and amount (in local currency) for one credit card transaction, along with anonymized customer and store IDs. Additional information about the customers and stores is also available, such as customers’ gender and the neighborhoods in which they live, as well as the category of the stores and the neighborhoods in which they are located.

We focus our analysis on credit card transactions, made at stores located in the largest metropolitan area of the country. We omitted foreign and online transactions as well as transactions without timestamp, stores with fewer than ten distinct customers over the three-month period, and customers who visited fewer than ten distinct stores.¹ As we are mainly interested in people selectively changing their purchase behavior in physical locations due to social events, we further selected five broad merchant categories that mostly correspond to on-site and discretionary purchases, i.e., “amusement and entertainment”, “clothing stores”, “retail stores” (including subcategory “grocery stores, supermarkets”), “personal service providers”, and “miscellaneous stores” (including subcategory “eating places, restaurants”). The resulting data set covers 1.8 million records from 100 thousand customers at 18 thousand geo-localized stores in the metropolitan area. Figure 1 visualizes the hourly evolution of the number of transactions made by the customers, from which we can already observe strong weekly patterns that separate the purchasing behaviors on weekdays and weekends, and daily patterns that distinguish different periods of a day such as the morning, afternoon and evening.

For each day in the week and for each neighborhood in the metropolitan area, we construct a time series with values for the two indexes. The data cover a period of 13 weeks. We therefore obtain two time series, each with 13 data points for each day in the week, for the number of customers and the median spending amount in a neighborhood on a particular weekday or weekend. For example, Fig. 2(a) and Fig. 2(b) show the time series of the number of customers shopping, and the median spending amount, respectively, in a given neighborhood on Mondays and Saturdays. To normalize differences in shopping behaviors between different days of the week (a difference that is particularly noticeable between weekdays and weekends), we compute the relative deviation from the mean,² which is defined as:

$$ c = (x-\bar{X})/\bar{X}, $$

(1)

where x is the value of an index on a given day, and X̄ is the mean value of the index for the given day during the first six weeks of the data. We take the first six weeks as the reference period, since there were no particularly noteworthy events during this period.³ We obtained two scalar values for each neighborhood for each day, which indicate the temporal evolution of the two behavioral indexes for each weekday or weekend.

We use the score of relative deviation from the mean for the two behavioral indexes, computed in each neighborhood for each day in the data set, to measure the impact of a series of protests that took place in a central site of the metropolitan area during a period of one month. In particular, we focus on the temporal, spatial, heterogeneous, and integral economic effect of these protest events.

3.1 Temporal variation of effects

In Fig. 3(a), (b), for more than 300 neighborhoods that have an average number of customers above ten in out data set, we show the median (weighted by number of stores in the neighborhoods) change in the two indexes, both temporally and spatially. More specifically, daily fluctuations of the changes in behavioral indexes are illustrated with respect to a reference period of six weeks, namely, from Day 1 to Day 42 in our data set, prior to the beginning of the societal unrest. Two major events of protests occurred on Day 62 (highlighted in green) and Day 77 (highlighted in cyan). The number of customers on these days decreased sharply in the vicinity (within 2 km) of the protest site, but less drastically in the neighborhoods within 2 to 4 km distance from the protest site and even less at larger distances. After the initial decrease, median spending returned more quickly to normal, and it was not as strongly affected as the number of customers. Here, too, the effect was generally stronger near the protest site.

To confirm our findings, we test the statistical significance of the results shown in Fig. 3(a), (b) in the Appendix (see Table A3(a)). On the days of two major events, we see a significant drop in both behavioral indexes for all three distances in Fig. 3(a), (b). In addition, there also exist statistically significant differences between distances. On days following the major protests (those after Day 62 and Day 77), the number of customers decreased significantly in neighborhoods within 4 km from the protest site, but not beyond. On the other hand, there was no significant decrease in the median spending in any distance, suggesting that the people who went out shopping did not tend to spend less money than usual.

The combined effect of the events on the two independent behavioral indexes can be seen in 2-D histograms, such as the ones in Fig. 4(a), (b), (c). The horizontal and vertical axes represent the score of relative deviation from the mean for the two behavioral indexes, respectively, and the color code indicates the average number of stores per neighborhood for neighborhoods that have a certain combination of the two scores. In such a histogram, positions towards the upper right represent increased numbers of customers and larger median spending, while positions towards the bottom left represent the opposite. If no change occurs, all neighborhoods stay at the center of the diagram. It is clear from Fig. 4(a) that on Day 62 stores in most neighborhoods were negatively affected by the protests, with fewer customers (mostly 30–90% less) and smaller spending amounts (mostly 10–70% less) within 2 km of the protest site (event center). People went out shopping less and were reluctant to spend as much as they usually do, possibly due to the unstable situation in the area. These trends were less pronounced when moving away from the event center, as shown in Fig. 4(b), (c), which is consistent with the results shown in Fig. 3(a), (b). The same 2-D histograms for Day 77 are shown in Fig. A1 in the Appendix.

3.2 Spatial decay of effects

To quantify the spatial propagation of the event effects, we study the effect as a function of the geographical distance from the event center. In Fig. 5, the blue dots in both figures indicate the median (weighted by number of stores in the neighborhoods) magnitude of decrease in (i) number of customers (Fig. 5(a), (b)) and (ii) median spending (Fig. 5(c), (d)) for neighborhoods that are located at different distances from the event center (i.e., the dot at 2 km represents all neighborhoods that are further than 1 km but less than 2 km away from the event center).⁴ Fitting the blue dots with exponential decay functions for both cases (shown as red curves), we see reasonably good fits, which indicates that the negative effect of events decayed approximately exponentially with respect to distance from the event center. We then compute the exponential decay constant, and its inverse, which is the so-called mean lifetime of the decay (mean distance in our context). The mean decay distances for Day 62 and Day 77, based on Fig. 5(a), (b), were \(1/0.20=5~\mbox{km}\) and \(1/0.35=2.86~\mbox{km}\), respectively, and, based on Fig. 5(c), (d), \(1/0.63=1.59~\mbox{km}\) and \(1/0.46=2.17~\mbox{km}\), respectively.

These results indicate that, first, the amount of money people spent when making a purchase was less clearly affected by the protests, compared to the change in the number of customers, and the negative effect spread less further away from the event center as well. Second, it seems that the event on Day 77 had a stronger impact on consumer actions within the close vicinity of the event center, which is indicated by a higher initial quantity of 1.25 in the exponential fit and reflected by the magnitude of the first two points in Fig. 5(a), (b). Such effect, however, decayed spatially faster compared to Day 62, which is suggested by a smaller mean distance of 2.86 km and reflected by the generally smaller magnitude from the third point (3 km) onwards in Fig. 5(a), (b). On the other hand, on Day 62 personal consumption was more affected close to the event center, while the negative effect on this index spread further away on Day 77. Such differences might be due to people’s responses to events changing over time. Finally, we can also study the event effect as a function of the estimated travel time (by car) from the event center, thus taking into account geographical constraints and transportation accessibility. The results are similar (in terms the exponentially decaying patterns) and presented in Fig. A2 in the Appendix. We further show in Fig. A3 and Fig. A4 in the Appendix the event effect on the individual neighborhoods. Overall, these results demonstrate the possibility of utilizing transaction data to quantify the spatial decay of event effects.

3.3 Heterogeneous effects

Social events may impact neighborhoods of different characteristics and the purchase behavior of people from different demographic groups in different ways. It may also have a differential effect on purchases from different types of merchants. We therefore study the heterogeneous effects of the social events in the following scenarios, by focusing on neighborhoods within 4 km from the event center.

3.3.1 Socio-economic status

We first show in Fig. 6 the time series of median (weighted by number of stores in the neighborhoods) change in the two behavioral indexes, computed over neighborhoods of higher socio-economic status, compared to that for those of lower socio-economic status. The socio-economic status is a composite measure between 0 and 100 that quantifies the relative prosperity of the neighborhood based on a number of indicators such as income and education level, which is obtained from the results of a recent census provided by the National Statistical Institute of the country. The higher the index, the more prosperous the neighborhood is. We see that the number of customers decreased more sharply in neighborhoods of lower socio-economic status for days following the protests. On the two days of major events, however, such a decrease was slightly higher in wealthier neighborhoods, possibly due to the fact that a larger portion of the demonstrators was from these neighborhoods. On the other hand, median spending dropped more significantly in less wealthy neighborhood on these two days, but not afterwards. Tests for statistical significance of the differences are presented in Table A3(b) in the Appendix. Overall, our results suggest that the events had longer lasting negative effects on less wealthy neighborhoods, but mainly in terms of consumer action rather than personal consumption. Because political preference is shown to be correlated with wealth in the same country [24], we next examine the effects on neighborhoods distinguished by political conservatism.

3.3.2 Political preference

Figure 7 shows the same time series as in Fig. 6, computed over neighborhoods of higher political conservatism, compared to that for those of lower political conservatism. An index for political conservatism of each neighborhood is a measure between 0 and 100 that is computed based on the percentages of votes parties labeled as “liberal” or “conservative” obtained in a recent general election of the country. The higher the index, the more conservative the neighborhood tends to be. It can be seen that, after the major protests, more conservative neighborhoods saw a larger decrease in both the number of customers and the median spending, probably due to the fact that people avoided going out for shopping and preferred to save in unstable circumstances. On the days of major events, however, less conservative neighborhoods saw a larger decrease in the number of customers, for the same reason as in the previous analysis that more people from these neighborhoods participated in the protests.

We note that the neighborhoods we analyze here are centrally located, hence it is possible that both the socio-economic status and political preference of the neighborhoods mainly reflect the typical profile of the residents but not that of the visiting customers. The results presented here are therefore mainly behavioral change observed in these neighborhoods. However, according to our data, 54% (46%) transactions in the wealthier (less wealthy) neighborhoods being studied come from their residents or residents of other wealthy (less wealthy) neighborhoods in the city, while 61% (48%) transactions in the conservative (liberal) neighborhoods come from their residents or residents of other conservative (liberal) neighborhoods. Therefore we believe that shopping activities observed in these neighborhoods reflect to a certain extent the behavioral change of people of similar characteristics in terms of wealth and political preference. Tests for statistical significance of the differences between the two groups in Fig. 7 are presented in Table A3(c) in the Appendix.

3.3.3 Gender difference

There may also exist a gender difference in the effect of social events. Figure 8 shows the time series of median change computed over all neighborhoods, for male and female customers.⁵ We see that, after the two major protests, both male and female customers tended to shop less, although they did not necessarily spend less money. However, demonstrations affected female customers more strongly than their male counterparts on the days of major protests, both in terms of the number of customers going out for shopping (for both days) and the median spending (for Day 77). Tests for statistical significance of the differences are presented in Table A3(d) in the Appendix. Figure A5 in the Appendix further illustrates the differences between four groups of different combinations of gender and political conservatism.

3.3.4 Store category

Finally, Fig. 9 depicts the time series of median change computed over all neighborhoods, for three categories of stores, namely, grocery stores, family clothing stores, and restaurants. Unlike in Fig. 6, Fig. 7 and Fig. 8, the median is weighted by number of stores in each category in the neighborhoods in this case. We see that on days following the major protests, both behavioral indexes decreased for all three categories of stores, except for the number of people shopping at grocery stores on mid-week days and median spending at clothing stores. In addition, although there is no clear difference between the three types of stores in terms of the median spending, customers visited family clothing stores less often than restaurants and groceries. Interestingly, on the days of major events, grocery stores showed the largest decrease in terms of customer visits. Tests for statistical significance of the differences are presented in Table A3(e) in the Appendix.

Our study demonstrates the potential of using pervasive and passively collected behavioral data to quantify the impact of localized social shocks and events on an urban population. We found that major events, such as societal unrest due to protests in our case, can alter people’s purchase behavior, both in terms of consumers’ tendency to visit shops in certain areas and in terms of their willingness to spend money. From a temporal perspective, personal consumption levels recover relatively quickly from the negative influence of the events. Spatially, on days of major events, consumption levels also seem more resilient, although both measures show clear spatial exponential decay. Event effects also differ between groups in the society, defined in terms of demographics, socio-economic status, and political conservatism, and between categories of merchants. Finally, given the results above, the existence of discernible patterns such as displaced or delayed shopping activities suggest that the effect of social events was not always distributed in the way one would imagine, e.g., it is not purely a function of geographical distance, and may help explain the rather unintuitive resilience of (or lack of change in) personal computation as a result of inelastic spending driven by needs.

Our analysis has certain limitations. The data set of credit card transaction records used in this study is based on a sample (about 10%) of all the individual customers of one financial institution and does not include people without credit cards. Therefore, sampling bias could exist and could potentially influence the results. Credit card transaction data may also represent only part of people’s daily spendings, as people may choose to pay with cash in certain scenarios. Furthermore, the data set only covers a period of three months, and the observations might be affected by seasonality. Finally, there might exist unobserved external events that bias our results, even though the strong spatiotemporal patterns observed on Days 62 and 77 are probably mainly caused by the major events on those days.

The real-time monitoring of economic behavior can have important social, economic and political implications. As a research tool, it makes it possible to assign quantitative values to the inherently elusive effects of social events. One advantage of the method we present here is the ability to analyze events with relatively high temporal and spatial resolutions. These allow a fine-grained understanding of events, beyond what is often possible with traditional measures. Also, the measures are less affected by demand characteristics or other biases, inherent in methods such as surveys. Such understanding and measures can potentially be used to develop timely and effective policies targeted at specific communities and demographic groups, and can often be critical in emergency situations.

With our method we can also evaluate statements made, for instance, by politicians or the media, such as that people are greatly upset by certain events. We can compare them to the observed changes in behavior, as expressed in the number of customers doing purchases and the average amount spent in a purchase. If people go about their activities as usual, the events probably have less impact, compared to when measures of behavior differ strongly from the usual values. At a practical level, these analyses may allow authorities to prepare for timely interventions to minimize possible negative consequences, possibly through a prediction mechanism that could estimate the recovery time of individual economic activities at an early stage of the event.

The framework we present here complements, rather than replaces, other methods such as traditional economic indicators or surveys. Combinations of it with other tools help create a comprehensive picture of the dynamic response of a population to social events.