Information dynamic spectrum characterizes system instability toward critical transitions
© Ni and Lu; licensee Springer 2014
Received: 13 May 2013
Accepted: 30 September 2014
Published: 4 November 2014
This paper addresses the need of characterizing system instability toward critical transitions in complex systems. We propose a novel information dynamic spectrum framework and a probabilistic light cone method to automate the analysis. Our framework uniquely investigates heterogeneously networked dynamical systems with transient directional influences, which subsumes unidirectional diffusion dynamics. When the observed instability of a system deviates from the prediction, the method automatically indicates the approach of an upcoming critical transition. We provide several demonstrations in engineering, economics, and social systems. The results suggest that early detecting critical transitions of synchronizations, sudden collapse, and exponential growth is possible.
Detection and prediction of emerging tipping points are major challenges in complex systems –, because while self-organized emerging interactions can facilitate information exchange, they increase the risk of attack or failure. When a system operates in a high-risk, unstable region, a small perturbation can induce a critical transition that leads to catastrophic failures. Although studies of data-driven computational models, with prior knowledge of individual systems, have greatly advanced the understanding of such emerging phenomena, we remain unequipped to accurately detect and predict tipping points prior to critical transitions .
Current works in emerging tipping point detection focus on homogeneous structured dynamical systems. In , Scheffer et al. give a seminal review on early warning signals for detecting critical transitions in the ecological domain. Signals such as increased temporal correlation, skewness, and spatial correlation of population dynamics are used to quantify the phenomena of critical slowing down as early warning indicators. These methods do not address heterogeneously networked systems. Through the remainder of this paper, networked dynamical system will be abbreviated as network. Scheffer et al. in  highlight the need to anticipate critical transitions in heterogeneous networks with empirical indicators. Such heterogeneity typically leads to connectivity patterns that are statistically characterized by heavy tails, large fluctuations, scale-free properties, and non-trivial correlations, which cannot be described by traditional modeling techniques that rely on the homogeneity assumption of a network . Consequently, network science has evolved from studying complex systems by modeling single, non-interacting networks to modeling interdependent networks . In particular, Buldrev et al. show that a network of networks is extremely vulnerable with respect to random failures: a random removal of a small fraction of the nodes from a network can trigger a catastrophic cascade of failures. The interconnectedness and interdependency of complex networks further pose hyper risks . Understanding complex interactions and coupling dynamics in large-scale complex networks is undoubtedly a crucial step toward preventing dangerous system behaviors.
The spectral early warning signals (EWS) theory , , inspired by , is one of the few attempts to detect critical transitions in heterogeneous networks. The spectral EWS theory states that the covariance spectrum can quantify the phenomenon of critical slowing down in heterogeneous networks by mathematically proving the link between complex network structures and observed time series. In particular, the leading covariance eigenvalue provides a structure-invariant indicator of upcoming critical transitions. As the system reaches the tipping point, the self-organized structure reveals itself via the components of covariance spectrum. Although spectral EWS quantifies how much the elements of a system change together, the symmetric nature of covariance spectrum does not permit the analysis of directional influences among elements.
Transfer entropy (TE)  and symbolic transfer entropy (STE)  have been proposed to identify directional influence in complex systems. For instance, STE is used to analyze brain activity data for the detection and identification of asymmetric dependences of brain regions in epileptic seizure activity . In , the transfer entropy matrix is used on financial market data to analyze the asymmetrical influence of mature markets on emerging markets. Although the transfer entropy quantified in ,  showed the promising results in analyzing financial and neurophysiological data, the changing structures and dynamics of networks were not addressed. Therefore, there is a need to quantify local transfer entropy that is changing in time .
Examples of critical transitions demonstrated in this paper
Nature of transitions
What to observe
ATE+ cross TE, rapid increase of TE and ATE+
Canonical chaotic networks
Rapid decrease of TE
Latin American stock market
Rapid decrease of TE
Rapid decrease of TE
Wikipedia editing behaviors
ATE+ cross ATE−, rapid increase of TE and ATE+
2 Information dynamic spectrum
We propose an information dynamic spectrum framework for monitoring complex systems. The assumptions in this work are (1) the outputs of the elements in the complex system can be observed as time progresses, i.e. time series are available, and (2) the outputs are the results of internal dynamics within individual elements and external interactions with other elements. An example is the time series of stock indices over a period of time. The fluctuations of a stock index depend on its own dynamics, as well as the hidden interactions with other indices. One example application of the proposed framework is to monitor the dynamics of this complex network of stock indices.
We will here establish the notations. Given the time series of a system of m elements: , where t is time and , is the time series of the i th element, and T represents the transpose of a vector, an interesting problem is to analyze the information dynamics of the system. In the following subsections, we will first describe the definition of transfer entropy, and then introduce a novel measure, associative transfer entropy, followed by the proposed information dynamic spectrum framework.
2.1 Transfer entropy
Transfer entropy (TE) is a directional measure of information flow between a pair of time series. A time series is a sequence of time-ordered observations of a node in a network, or an element of a system. TE quantifies how much information is transferred from the current state into the future from one time series to another. The concept of TE extends from the concept of mutual information, which is a measure of mutual dependence between two time series and more mutual information means less uncertainty if knowing one time series. However, mutual information is symmetric and does not inform the direction the information flows. To address this, TE takes into account the dynamics of information transport and provides how much information is transferred from the current state into the future.
where represents the joint probability and and represent the conditional probabilities. This definition is derived from the Shannon entropy: , a measure of uncertainty. The definition of in Eq. (1) can be expressed as . This is the uncertainty in y at time given its history , minus the uncertainty in y at time given its history and x’s history . Therefore, TE measures the net amount of information transferred from x to y. An important feature of TE is that it is asymmetric, in contrast to mutual information, which is a symmetric measure. More details about TE, including some interesting examples, can be found in the original paper .
2.2 Associative transfer entropy
where S is the associate state, a subset of the set of all possible states of . The set S represents a certain associated state between x and y. The purpose of ATE is to capture information transfer between two variables for a particular association of their states. For example, for binary time series, TE can be decomposed into a positive association ATE+ and a negative association ATE−, in order to distinguish two situations where the amount of information transferred may be the same but with opposite effects: in one case the value of the source x drives the value of the destination y in the same direction, and in the other case the source drives the destination in the opposite direction.
2.3 ATE symbolization technique
The definition of TE and ATE in Eq. (1) and Eq. (2) respectively are based on continuous random variables. To estimate probability distributions from a finite number of observations, one can discretize the observed values. We adopt the symbolic transfer entropy method  that is based on the symbolization technique for permutation entropy . We adopt this method to compute ATE because ATE sums over a subset of all the possible discrete states that TE sums over. Therefore, this is a natural extension of the computation. The key step is to transform the continuous-valued time series into a symbolic time series with a suitable length n, where is an integer. First, for each t, the n consecutive values are ordered in the ascending order. The sequence of the indices that correspond to such permutation is recorded and denoted by , the associated permutation of order-n symbol of at t. Then we estimate the ATE of by calculating the ATE of . This now can be done efficiently because the number of states to sum over in the ATE equation (2) is finite and the probabilities and conditional probabilities can be directly calculated based on the discrete-valued symbolic time series .
2.4 Local TE and ATE
2.5 Spectral radius of TEM and ATEM
Since TE and ATE are pairwise measurements, they do not give a single measurement of the total amount of entropy transferred within a system. To extend pairwise TE and ATE, we first define the ATE Matrix (ATEM) of a system of m elements as an matrix with ij th entry , where the superscript S is the associative state that is defined for Eq. (2). Similarly, the ij th entry of the TE Matrix (TEM) is . TEM has been used in  to reveal the asymmetric influences from mature markets to emerging markets, but was not used to give the total amount of TE of the entire system, nor in a sliding window fashion to measure the transient influence dynamics. Note that since TE and ATE are directional, TEM and ATEM are nonsymmetric, and thus their eigenvalues are complex-valued.
We use the spectral radius of the TEM (resp. ATEM) to measure the total amount of TE (resp. ATE) in the entire network. The spectral radius of a matrix is the supremum of the absolute values of its eigenvalues. Furthermore, to use the spectral radius of a TEM or ATEM to monitor the system, we use a time sliding window to calculate local TEM or ATEM and therefore the spectral radius of a TEM or ATEM becomes a function that depends on time.
2.6 Detect critical transitions with ATE and TE spectral radius curves
In this subsection, we explain how to use the TE and ATE curves defined in the previous subsection to characterize critical transitions prior to their occurrences (see Table 1). We give our observations on behaviors of TE and ATE in four different critical transition scenarios. The first critical transition scenario is from a stable state to oscillations in a network of non-Foster circuits. In this case, as a network starts to synchronize in phase, the pairwise TEs will likely start to increase because of the growing amount of interacting dynamics between circuits. The TE and ATE+ spectral radius curves may therefore start to increase. If the synchronization is in the negative mode, in which the phases differ in 180∘, the ATE− spectral radius curve will start to increase.
The second scenario is the case where the time series of a system bifurcate, switching from one state to alternative states. The examples in Figure 4 demonstrate the nature of such transitions. Since the phenomenon of critical slowing down is observed prior to such transitions, the TE and ATE curves may decrease rapidly prior to the tipping point.
The third scenario is when there is a sudden drop in the time series of the system. It has been suggested that there is a critical slowing phenomenon before the tipping point of a crash in many real world systems, such as stock markets, housing market, etc. In this scenario, right before the crash, the interacting dynamics between nodes may start to decrease, which can be explained by the uncertainty of the information carried by other nodes. Therefore, the TE and ATE curves are likely to decrease rapidly before the crash.
The fourth scenario is the exponential growth in system activities. In this case, the TE and ATE+ curves are likely to increase, as a result of an increase in the interacting dynamics between nodes. Given one of the above scenarios, we can look out for critical transitions using TE and ATE curves.
3 Numerical results of information dynamic spectrum
3.1 ATE indicates instability of non-Foster circuit
The circuit data used for TE and ATE evaluation is simulated from a non-Foster network described in . A non-Foster network is an active network containing a power source, therefore it may not obey Foster’s reactance theorem. The non-Foster circuit data analyzed here contains three coupled circuits with internal feedback loops. The interface points include two antennas (Ant1 and Ant2) and two ports (i1 and i2) to record time series of the voltage. The circuit is operated in the stable (no oscillation) region, but near the unstable region. A perturbation is then added at timestep = 250 with 1 milli-Amp. When a small perturbation is added, the circuits may become oscillatory. The circuits will either become synchronized in oscillation, which is considered unstable, or return to the original stable state.
3.2 TEMs infer directional influences in Latin America stock indices
Latin America stock indices
3.3 ATE reveals Wikipedia motifs that drive the changes
The bottom row of Figure 8 from left to right shows global ATEM+, ATEM−, and TEM from 2001 to 2011 to identify important motifs that drive the changes in other motifs. In particular, ATEM+ shows that motifs 1, 8 and 12 have the most positive influence on other motifs, observing that consecutive ‘minor add’, ‘revert’, and ‘major add’ by registered users encourage Wikipedia’s content growth. ATEM− shows that motif 9 has the most negative influence on other motifs. TEM shows the asymmetric influence among the motifs but the contrast is not as strong as ATEM+ and ATEM−. The bottom of Figure 9 shows the TE, ATE+, and ATE−, i.e. spectral radii of local TEM and ATEM, in blue, red, and green, respectively. There is a significant increase in the ATE+ and TE curves near midyear 2005 and ATE+ crosses over ATE−. These suggest an early indication of the growth in Wikipedia’s contents. Interestingly, the peaks of TE and ATE+ in late 2008 occur at the beginning of the stabilization of the editing behaviors. The parameter choices for this analysis are symbolic length and sliding window size .
3.4 TE and ATE curves as early indicators of stock market crash
We collected the share prices of Dow Jones Industrial Average (DJIA) from June 1 to December 31, 2008, using 25 shares that persisted throughout the years near the October 2008 Crash. The stock indices are: MMM, AA, AXP, T, BA, CAT, KO, DD, XOM, GE, HPQ, HD, INTC, IBM, JNJ, JPM, MCD, MRK, MSFT, PFE, PG, UTX, VZ, WMT, and DIS. The time series resolution is one day, except when the market is closed during the weekend.
Our empirical data analysis suggests that early detection of critical transitions using local ATE curves is possible. The local ATE curve, spectral radius of the local ATEM over a sliding time window, is an indicator of the amount of net entropy transferred in the entire system at each time. On the other hand, by looking at the entries of ATEM, one can identify the most positive or negative influential nodes of the system, as well as the most positively or negatively influenced nodes. From the observations of the TE, ATE+, and ATE− curves in different examples that corresponds to different types of critical transitions, we found that (1) ATE+ increases rapidly before synchronization in the positive mode. ATE− increases rapidly before synchronization in the negative mode. (2) TE, ATE+, and ATE− drop rapidly before abrupt falls, because TE, ATE+, and ATE− are the net information flow and external information flow begins to decrease before the sudden collapse. (3) ATE+ increases rapidly before an exponential growth in a system.
4 Probabilistic light cones for trajectory prediction
We propose a model-based probabilistic light cone method for predicting the trajectories of the spectral radius of TEM or ATEM. As observed in the previous section, the behaviors of TE and ATE curves can be interpreted manually as early indications of critical transitions. Here, we provide an automated method to generate early indicators. Using a natural logarithmic model, our method first computes probabilistic light cones for TE and ATE curves. This gives a prediction value with a confidence interval. If the actual TE and ATE trajectories are outside the confidence interval, the method declares instability is detected, which serves as an early indicator of a critical transition.
Our computation of probabilistic light cone is inspired by , where an MCMC method is used to predict citation growth based on the preferential attachment. However, critical transition is not discussed in . Our method, on the other hand, aims to generate early indicators for an upcoming critical transition as time progresses. We choose natural logarithmic curves to model the growth rate of the TE trajectory as information transfer approaches its maximum prior to a critical transition as observed in the non-Foster circuit example in the previous section. We apply a moving time window over the observed spectral radius (TE or ATE±) time series to derive the unknown coefficients and constants for natural logarithm curves. For a given prediction time point, we generate the probabilistic light cone based on 95% confidence intervals of predicted trajectories with fitted natural logarithm curves.
4.1 Model-based forecasting
4.2 Probabilistic light cone and error estimation
Given the method to estimate the constants c and a in previous section, we generate a probabilistic light cone at each time as we vary △t to obtain multiple estimates of c and a. We define a probabilistic light cone at a given time consists of a collection of natural logarithm curves starting from that point.
where L is the size of E.
4.3 Numerical results
We propose a novel information dynamic spectrum framework for automated detection of critical transitions and identification of directional influences. We have shown that the framework is able to: (1) provide an effective measure for quantifying associative, asymmetric directional influence rather than symmetric influence, (2) provide an effective formulation that captures system-wise directional influence, rather than pair-wise influence and (3) provide an effective measure for detecting instability in systems with directional influence dynamics. Within this framework, we further propose the probabilistic light cone method that predicts the ATE trajectories and indicates instabilities toward critical transitions. This model-based prediction method captures information that cannot be discerned by looking at individual time series signals alone and is novel compared to conventional deterministic methods or complicated probability state space predictions. The analyses performed in this work are based on historical data, and the rapid decrease or increase in TE is not a sufficient condition for all critical transitions. Rather, as shown in Table 1, the nature of upcoming critical transitions is revealed by the behaviors of ATE curves, in conjunction with examining the trends in the time series of a system. Future work requires a complete analysis of the proposed framework, as well as an adaptive parameter selection method for the probabilistic light cone analysis.
Electronic Supplementary Material
Below are the links to the electronic supplementary material.
The authors would like to express their gratitude to Hankyu Moon for helpful discussions about EWS and Carson White for generating the non-Foster circuit data. The authors would also like to thank the two anonymous reviewers for their helpful and constructive comments that greatly helped improve the quality of the paper.
- Helbing D: Globally networked risks and how to respond. Nature 2013, 497(7447):51–59. 10.1038/nature12047View ArticleGoogle Scholar
- Vespignani A: Modeling dynamical processes in complex socio-technical systems. Nat Phys 2012, 8: 32–39. 10.1038/nphys2160MathSciNetView ArticleGoogle Scholar
- Motter AE, Albert R: Networks in motion. Phys Today 2012, 65(4):43–48. 10.1063/PT.3.1518View ArticleGoogle Scholar
- Buldyrev SV, Parshani R, Paul G, Stanley HE, Havlin S: Catastrophic cascade of failures in interdependent networks. Nature 2010, 464(7291):1025–1028. 10.1038/nature08932View ArticleGoogle Scholar
- Barabasi AL: Scale-free networks: a decade and beyond. Science 2009, 325(5939):412–413. 10.1126/science.1173299MathSciNetView ArticleGoogle Scholar
- Scheffer M, Bascompte J, Brock WA, Brovkin V, Carpenter SR, Dakos V, Held H, Van Nes EH, Rietkerk M, Sugihara G: Early-warning signals for critical transitions. Nature 2009, 461(7260):53–59. 10.1038/nature08227View ArticleGoogle Scholar
- Scheffer M, Carpenter SR, Lenton TM, Bascompte J, Brock W, Dakos V, van de Koppel J, van de Leemput IA, Levin SA, van Nes EH, Pascual M, Vandermeer J: Anticipating critical transitions. Science 2012, 338(6105):344–348. 10.1126/science.1225244View ArticleGoogle Scholar
- Barrat A, Barthelemy M, Vespignani A: Dynamical processes on complex networks. Cambridge University Press, Cambridge; 2008.View ArticleGoogle Scholar
- Quill E: When networks network: once studied solo, systems display surprising behavior when they interact. Sci News 2012, 182(6):18–25. 10.1002/scin.5591820619View ArticleGoogle Scholar
- Moon H, Lu T-C: Early warning signal of complex systems: network spectrum and critical transitions. Proceedings of workshop on information in networks 2010.Google Scholar
- Compton R, Moon H, Lu T-C: Catastrophe prediction via estimated network autocorrelation. Proceedings of workshop on information in networks 2011.Google Scholar
- Schreiber T: Measuring information transfer. Phys Rev Lett 2000, 85(2):461–464. 10.1103/PhysRevLett.85.461View ArticleGoogle Scholar
- Staniek M, Lehnertz K: Symbolic transfer entropy. Phys Rev Lett 2008., 100(15): 10.1103/PhysRevLett.100.158101Google Scholar
- Kwon O, Yang J-S: Information flow between stock indices. Europhys Lett 2008., 82(6): 10.1209/0295-5075/82/68003Google Scholar
- Lizier JT, Prokopenko M, Zoomaya AY: Coherent information structure in complex computation. Theory Biosci 2012, 131: 193–203. 10.1007/s12064-011-0145-9View ArticleGoogle Scholar
- Bandt C, Pompe B: Permutation entropy: a natural complexity measure for time series. Phys Rev Lett 2002, 88(17):149–170. 10.1103/PhysRevLett.88.174102View ArticleGoogle Scholar
- White CR, Colburn JS, Nagele RG: A non-Foster VHF monopole antenna. IEEE Antennas Wirel Propag Lett 2012, 11: 584–587. 10.1109/LAWP.2012.2201129View ArticleGoogle Scholar
- Söhnke MB, Gordon MB: No place to hide: the global crisis in equity markets in 2008/09. J Int Money Financ 2009, 28: 1246–1292. 10.1016/j.jimonfin.2009.08.005View ArticleGoogle Scholar
- Palacios J (2013) Private communication., [http://crcc.usc.edu/about/personnel/joe-palacios.html]Google Scholar
- Torres IE (2007) The mineral industry of Colombia. United States Geological Survey minerals yearbookGoogle Scholar
- Ronconi L, Marongiu F, Dborkin D, Filc G: América Latina frente a la crisis internacional: características institucionales y respuestas de política. CIPPEC 2010.Google Scholar
- Jurgens D, Lu T-C: Temporal motifs reveal the dynamics of editor interactions in Wikipedia. Proceedings of the 6th international AAAI conference on weblogs and social media 2012.Google Scholar
- Clauset A, Woodard R: Estimating the historical and future probabilities of large terrorist events. Ann Appl Stat 2013, 7(4):1838–1865. 10.1214/12-AOAS614MathSciNetView ArticleGoogle Scholar
- Liu R, Li M, Lui ZP, Wu J, Chen L, Aihara K: Identifying critical transitions and their leading biomolecular networks in complex diseases. Sci Rep 2012., 2:Google Scholar
- Quax R, Kandhai D, Sloot P: Information dissipation as an early-warning signal for the Lehman Brothers collapse in financial time series. Sci Rep 2013., 3: 10.1038/srep01898Google Scholar
This article is published under license to BioMed Central Ltd.Open Access This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.