Router-level community structure of the Internet Autonomous Systems

The Internet is a complex network composed of routing devices (routers) which are organized into administrative entities, the Autonomous Systems (ASes). Each AS has its own routing policies and design criteria, which are based on technological, geographical and economical constraints and aim at maximizing the performance in terms of bandwidth, delay and resilience. ASes are nowadays identified with large carriers, ISPs (Internet Service Providers), IXPs (Internet Exchange Points), CDNs (Content Delivery Networks) and universities, among others, and they are interconnected as the result of commercial agreements between them, either as peers or in provider-customer relationships.

ASes can also be classified into core ASes (once called Tier-1’s), and peripheral ASes. The core ASes are typically the large carriers and some CDNs and are densely connected between them. Instead, most ISPs lay in the periphery and are connected to one or more core ASes. This core-periphery division provides the Internet with an interesting hierarchy. In fact, many large-scale properties as the resilience, scalability and small-worldness of the Internet arise largely as a consequence of its hierarchical structure and self-organization [1, 2]. Understanding them might help developing better models of the Internet and improving its performance, by optimizing routing or reducing congestion, for example.

In this work we approach the modular structure of the Internet at the router level by trying to identify the ASes as communities of nodes. We explore different notions of community structure, and we shall see that: (a) The goodness of the communities is in some way related to the core-periphery hierarchy of the ASes. (b) Though most of the ASes satisfy some notion of community, their largely variable sizes represent a hard problem for many classical community detection algorithms. (c) Using multiresolution methods instead, the ASes can be retrieved if each of them is observed at the right resolution level. Finally, we shall propose a method for identifying ASes using samples of nodes.

Our work falls into the study of Internet topology. Some of the main results on this area can be found in [3]. In particular, the study of community structure of the Internet at the Autonomous Systems level has been previously approached in [4–7] and, to the best of our knowledge, the community structure at the router level has only been addressed in [8] in the context of traffic queuing analysis. For the community detection problem in general, we address the reader to the survey by Fortunato [9].

The paper is organized as follows: In Section 2 we describe the Internet exploration that we used. In Section 3 we apply several community detection algorithms to it, matching the obtained communities to the ASes; we also introduce a hierarchical partitioning algorithm based on multiresolution modularity maximization and we determine which is the best resolution for each AS; we also evaluate other multiresolution algorithms. In Section 4 we evaluate the results from the Internet topology perspective, explaining their variability: why some ASes satisfy our notion of community while others do not. We also discuss the methodological problems of community detection in heterogeneous networks and some possible solutions. Finally, in Section 5 we extract the main results of this work.

The dataset used throughout this paper was provided by CAIDA [10]. The CAIDA association performs daily explorations of the Internet by sending periodic IP packets (probes) towards random destinations from a set of sources called monitors (which nowadays count around 100). These probes are sent with a traceroute tool which exploits the ICMP functionality: by means of the ICMP protocol, the intermediate routers in a datagram transmission can provide the source with some information on the traversed path. Using this information, the CAIDA infrastructure builds a map of the Internet at the router level. Several problems must be resolved through this process, such as IP aliasing (the association of several IP addresses with one same router) [11] and the existence of anonymous routers (routers which do not answer probes). Traceroute-based sampling is not a perfect sampling and has several biases [12]: some links are more frequently visited than others, and some links are probably not visited at all. This, together with the fact that the Internet is a fast evolving network, turns the map into a partial, approximate view of the Internet.

However, router-level maps of the Internet have several applications, like improving routing algorithms by taking into account the network topology, understanding information propagation, and studying the robustness and scale-free properties of networks [1, 13]. They can also be used for constructing maps of the Internet at the Autonomous Systems level [14, 15].

In this work we used the CAIDA router-level Internet map from October 19th, 2011 [16]. From the raw data we obtained an undirected graph consisting on 3,248,358 nodes (the routers) connected by 14,083,946 edges; 81.6% of the nodes also contain information about their Autonomous System (AS) affiliation (for a description on how this information was inferred, consult [17] and the CAIDA page [18]). Figure 1 plots the degree distribution and the average clustering coefficient and neighbour degree as a function of the node degree.

In the following subsection we shall introduce Deltacom, an algorithm for hierarchical partitioning based on modularity maximization. Later on, we shall evaluate the results using a similarity metric.

3.1 A hierarchical partitioning method based on multiresolution modularity

Deltacom is based on the optimization of modularity. We recall that modularity was introduced by Newman in [25, 26] and, since then, several methods for its maximization have been proposed, like the one by Guimerà and Amaral based on simulated annealing [27], the extremal optimization method by Duch and Arenas [28], the fast greedy algorithm by Blondel et al. [23], or the multilevel algorithm by Noack and Rotta [29].

The following two results about modularity are fundamental to our present work: in [30] Fortunato and Barthélemy showed that modularity has a resolution limit, i.e., its maximization tends to put small communities together when they are connected among them; in [31] Reichardt and Bornholdt observed that modularity can be understood as the Hamiltonian of a ferromagnetic Potts model. Thus, maximizing the modularity implies finding the ground-state of this model, and the authors developed a simulated annealing based procedure for doing it.

Deltacom considers the optimization of the modularity Q as a particular case of the optimization of a more general functional \(Q_{t}\), with a resolution parameter t, in which modularity corresponds to a normalized resolution, i.e., \(t=1\). The source of our idea can be traced back to the γ resolution parameter in [31], and has also been followed by Lancichinetti and Fortunato in [32]. In [31], the maximization is based on simulated annealing, it must run at one single resolution each time, and its computational complexity is very high. The advantage of our method is that the resolution evolves dynamically, so that all the partitions at different resolutions can be produced during one single run, and with a low computational complexity.

Deltacom is based on an agglomerative greedy algorithm; each step of the agglomerative process is a local maximum for \(Q_{t}\), i.e., the generalized modularity, at some particular resolution t. In other words, for each t-value a community partition is found, which is locally t-optimal in some sense. As these communities are joined, the resolution shrinks and t gets smaller. For \(t=1\), the structure is a local maximum for the classical modularity. In other words, we observe the graph as by using a magnifying glass, obtaining partitions at every resolution level. The result is a hierarchical structure in which partitions at higher resolution levels are always refinements (in a mathematical sense) of those at lower ones.

Here we present a brief description of how the algorithm works. Further mathematical details on the properties of \(Q_{t}\) and of our maximization algorithm can be found in [33].

3.1.1 Newman’s modularity

Given a graph \(G=(V,E)\) and a partition of it, \({\mathscr{C}}\), the modularity of the partition is defined as [26]:

$$ Q({\mathscr{C}}) = \frac{1}{2m}\sum_{(i,j)\in V}{ \biggl[A_{ij}-\frac{k_{i} k_{j}}{2m} \biggr]}\delta(i, j), $$

(1)

where m is the amount of edges, \(A=(A_{ij})\), \(i,j\in V\) is the adjacency matrix of the graph, \(k_{i}\) is the degree of the node i and \(\delta(i,j)\) is the indicator function that points out if nodes i and j belong to the same community in the partition \({\mathscr{C}}\).

The expression on the right side of Eq. 1 compares the internal connections in the communities (represented by the \(A_{ij}\cdot \delta(i,j)\) product) with the expected number of connections under a random graph with the same expected degrees in the nodes and the same communities. It can be restated as a sum throughout the communities, in the following way:

$$ Q({\mathscr{C}}) = \frac{1}{2m}\sum_{C\in {\mathscr{C}}}{ \biggl[e_{C}-\frac{k_{C}^{2}}{2m} \biggr]}, $$

(2)

where \(k_{C}\) is the sum of degrees of the nodes in each community C, and \(e_{C}\) is twice the number of internal edges of it.

Now, in many agglomerative clustering methods, communities are joined by pairs at each step, until finding a local maximum. Joining two communities C and \(C'\) during the process produces the following variation in the Q functional:

$$ \Delta Q = \frac{1}{2m} \biggl(e\bigl(C,C'\bigr)- \frac{k_{C}\cdot k_{C'}}{2m} \biggr), $$

(3)

where \(e(C,C')\) is twice the number of edges between C and \(C'\).

3.1.2 Introducing a resolution parameter

We introduce a resolution parameter t as

$$ Q_{t}({\mathscr{C}}) = \frac{1}{2m}\sum_{C\in {\mathscr{C}}}{ \biggl[e_{C}-t\cdot\frac {k_{C}^{2}}{2m} \biggr]}. $$

(4)

Now the functional variation after joining two communities C and \(C'\) becomes:

$$ \Delta Q_{t} = \frac{1}{2m} \biggl(e\bigl(C,C' \bigr)-t\cdot\frac{k_{C}\cdot k_{C'}}{2m} \biggr). $$

(5)

It is clear that a positive \(\Delta Q_{t}\) value for a particular t also implies a positive (and even higher) \(\Delta Q_{t'}\) for any resolution \(t'< t\). In other words, any agglomerative process which monotonically increases \(\Delta Q_{t}\) also serves as a process monotonically increasing \(\Delta Q_{t'}\) for any \(t'< t\).

It is also immediate that a very large t value discourages any join, because \(\Delta Q_{t}\) is negative for every pair of communities. It can also be shown that for t larger enough the global maximum for \(Q_{t}\) would have each node isolated in its own community.

Thus, what we propose is to start with a large enough t (so that the optimal initial partition \({\mathscr{C}}_{t}\) has as many communities as nodes) and then to decrease t as little as possible so that a join can be made, i.e., so that some \(\Delta Q_{t}\) becomes positive. This t value is just

$$ t({\mathscr{C}}) = \max_{C,C'\in {\mathscr{C}}} \biggl\{ \frac{e(C,C')\cdot2m}{k_{C}\cdot k_{C'}} \biggr\} . $$

(6)

When a local maximum is found for some t (i.e., no pair of communities can be joined without decreasing \(\Delta Q_{t}\)), we decrease the resolution to \(t'\) such that \(\Delta Q_{t}'\) becomes positive for some pair of communities, following the previous formula. The agglomerative process continues until obtaining as many communities as connected components of the graph, and the obtained result is a set of locally optimal partitions \({\mathscr{C}}_{t}\) for every resolution t, and such that the finer partitions are refinements of the coarser-grained ones. Even more, these partitions are what we call weakly optimal partitions, in the sense that not only the join of any pair of communities C, \(C'\) would decrease \(Q_{t}\), but also any coarser partition (i.e. obtained by joining its communities in any way) would also decrease it.

The code for the Deltacom algorithm is freely available at SourceForge [34]; given a graph as a list of edges, it produces a set of community partitions for every resolution value.

The formalization of the process is contained in Algorithm 1. This algorithm produces weakly optimal partitions for the generalized modularity \(Q_{t}\). In particular, \({\mathscr{C}}_{1}\) is a weakly optimal partition for the classical modularity Q. The notation \({\mathscr{C}}_{(a,b]}\) represents that the partition is weakly optimal for every resolution in the \((a,b]\) interval. For a complexity analysis, we shall consider keeping a set of lists \(L_{C}\), one for each community, containing its neighbour communities \(C'\) and the edge cut with each of them. The lists shall be ordered by a community identifier. The initial construction of these structures has time complexity \(\mathcal {O}(m\cdot \log(m))\) (as each list has size \(s_{C}\), with \(\sum_{C}{s_{C}}=m\), and \(\sum_{C}{s_{C}\cdot \log(s_{C})}\leq m\cdot \log(m)\)). The while loop starting at line 1.6 has at most n cycles; each of these cycles has three steps: (a) finding a pair of communities with \(\Delta(Q_{t})=0\); (b) joining the communities; and (c) decreasing t. The first step implies traversing all the lists; each \(\Delta(Q_{t})\) computation is \(\mathcal{O}(1)\) so this step’s complexity can be bounded to \(\mathcal{O}(m)\). Joining two communities implies merging their lists into one, and also updating the lists of their neighbour communities. As all the lists are ordered, this can be done by traversing those lists, which we can bound to \(\mathcal{O}(m)\). Finally, decreasing t implies traversing all the lists and computing \(\frac{e(C,C')\cdot2m}{k_{C}\cdot k_{C'}}\) for each element. This is again \(\mathcal{O}(m)\). Putting all together, we have at most n cycles with \(\mathcal{O}(m)\). The final complexity is then \(\mathcal{O}(nm)\).

While \(\mathcal{O}(nm)\) is a theoretical upper bound for a general graph, the practical running time for sparse graphs can be greatly reduced with some considerations: step (c) gives all the information for the next step (a), because the community pair which maximized \(t'\) in line 1.11 is the same that will have \(\Delta Q_{t}=0\) in the next iteration; also, by keeping a search structure like an ordered tree with all the pair of connected communities \(C,C'\) ordered by \(\frac{e(C,C')\cdot2m}{k_{C}\cdot k_{C'}}\) decreasingly, there is no need to traverse all the lists in each iteration, but we must just update the modified values into this tree and choose the community pair at its head as the one maximizing \(t'\). For sparse graphs, this process has an upper complexity bound of \(\mathcal{O}(m\cdot \log(m)\cdot s_{C}^{\max})\), where \(s_{C}^{\max}\) is the maximum number of neighbour communities that a community may have at any time of the process.

3.2 Finding ASes through similarity maximization

We applied the five forementioned algorithms (Infomap, LPM, Deltacom at \(t=1\), Louvain and CommUGP) to the CAIDA dataset. Each of these methods produces a partition \({\mathscr{C}}\) of the graph. In order to determine if the communities in these partitions are related to the ASes, we shall use this criterion: for each AS in the graph, and given a partition \({\mathscr{C}}\), we find the community \(C\in {\mathscr{C}}\) which maximizes the Jaccard similarity:

$$J(C, AS) = \frac{\vert C \cap AS\vert }{\vert C \cup AS\vert }. $$

This most similar community shall be called \(C_{AS}^{(1)}\):

$$C_{AS}^{(1)} = \mathop{\arg\max}_{C\in {\mathscr{C}}} J(C, AS). $$

And this maximum Jaccard similarity value will be called the recall score of the AS (according to [35]) and noted as \(R_{1}(AS)\):

$$ R_{1}(AS) = J\bigl(C_{AS}^{(1)}, AS \bigr). $$

(7)

In this way, we can evaluate each of the methods by observing the empirical cumulative distributions of the recall score of the ASes, \(R_{1}(AS)\) (Eq. 7). Those methods with left-skewed cumulative distributions (i.e., recall values closer to 1.00) tend to better reproduce the Autonomous System structure. The cumulative distributions for the different methods are shown in Figure 2: blue for Infomap, yellow for LPM, red for Deltacom, green for CommUGP and pink for Louvain. At the top we plot the cumulative distribution considering all the ASes, and at the bottom we only consider ASes with at least 100 nodes (i.e., 639 ASes, which cover 90% of the network).

We can clearly see that the five methods fail to reproduce the ASes structure. In the best performing one, Infomap, only 50% of the large ASes reach a recall of 0.4, which is however small. A premature conclusion might state that the Internet graph does not have communities at the router level, or at least that they are not related with the Autonomous Systems. But we will show that this is not the case, and that the failure of the methods is in part due to the largely variable sizes, internal structures and functions of the Internet ASes. However, the structure of the Internet graph at the router level can be better captured by using multiresolution methods.

In order to assess our hypothesis we shall exploit the resolution parameter t of Deltacom, which provides us with a new level of freedom for matching the ASes in the hierarchical community tree. For each t in the range \([t_{\min}, t_{\max}]\) we have a partition \({\mathscr{C}}_{t}\). Thus, we shall define a new recall score \(R_{2}(AS)\) based on exploring a parameter space Θ with different partitions \({\mathscr{C}}_{\theta}\), \(\theta\in\Theta\), associated to it such that the most similar community for an Autonomous System, \(C_{AS}^{(2)}\), is:

$$\begin{aligned}& C_{AS}^{\theta} = \mathop{\arg\max}_{C\in {\mathscr{C}}_{\theta}}{J(C, AS)},\quad \theta \in\Theta, \\& C_{AS}^{(2)} = \mathop{\arg\max}_{\{C_{AS}^{\theta}: {\theta}\in{\Theta} \} }{J\bigl(C_{AS}^{\theta}, AS\bigr)}, \end{aligned}$$

and the recall score as:

$$ R_{2}(AS) = J\bigl(C_{AS}^{(2)}, AS \bigr). $$

(8)

In the case of Deltacom, \(\Theta=[t_{\min}, t_{\max}]\) and t plays the role of θ. In this way, the Jaccard similarity allows us to capture the moment in which each AS is formed. With this adjustments, the results clearly improve. The recall score \(R_{2}(AS)\) (Eq. 8) over all the ASes is 0.87 (and 0.67 over the ASes of at least 100 nodes). The new cumulative distribution is presented as the black curve in Figure 2, and we shall call it Deltacom Multiresolution, as we are exploring all the communities at all possible resolutions.

The observed improvements tell us that each Autonomous System has a natural resolution at which it can be best observed. There is an important correlation between this resolution and the size of the AS, as the left picture in Figure 3 shows. The right picture in Figure 3 shows the evolution of classical modularity Q for the Internet graph as a function of the resolution t, i.e. as the algorithm evolves (from right to left). As one of the properties of our algorithm, \(Q_{t}\) has its maximum at t for every possible t, so that the classical Q is maximal at \(t=1\).

We stress that several multiresolution methods exist in literature, but the advantage of ours is that it can generate a hierarchical partition covering all the possible resolution levels, in one single run. However, we compared our results with those of two other multiresolution methods, i.e., Infomap multiresolution and the Constant Potts Model:

• The Infomap algorithm has a multiresolution variant [36], which generates a community tree with several levels \(k\in\{1,\ldots,k_{\max}\}\). We matched each of the ASes into this structure in the same way as we did with Multiresolution Deltacom: i.e., we find the best level for each AS, and then we compute the recall score (now \(\Theta=\{1,\ldots,k_{\max}\}\) and \(\theta=k\)). Here we found a clear improvement too, which is shown in the orange curve of Figure 2, pointing out that it is important to find the correct resolution for each AS.

• From the general schema described by Reichardt and Bornholdt in [31] several multiresolution approaches have been proposed. We used the Constant Potts Model by Traag et al. [37], which removes the null model in the Hamiltonian to cancel global dependences. For this algorithm we tested different values of \(\gamma\in[10^{-8}, 1]\). We observed that this interval covers all the interesting situations, as for \(\gamma=10^{-8}\) we obtain a partition with 10 communities, and for \(\gamma=1\) all the nodes are in different communities. We sampled the interval by dividing it into 4, 8, 16 and 32 pieces, equidistant in the logarithmic scale. When dividing it into \(2^{k}\) pieces we obtain \(2^{k} + 1\) values of γ which we use as the parameter space \(\Theta=\{\gamma_{0}, \gamma _{1}, \ldots, \gamma_{2^{k}}\}\) in order to match each AS against its most similar community (according to the recall score \(R_{2}(AS)\)) among the \(2^{k} + 1\) different partitions. We chose to stop the exploration at \(k=5\) (33 values of γ) because the observed convergence suggested that refining the parameter space would not produce much better results. The curve for \(k=5\) representing CPM is shown in purple in Figure 2. Finally, the central picture in Figure 3 shows the best γ value (from among the 33 considered values) for each AS, as a function of its size. Compared to Deltacom, in CPM the correlation between resolution and size is not so strong.

From this analysis we conclude that there is no single resolution at which we can find most of the ASes. This had already been observed in [32] for modularity, and is partly due to the fact that modularity is affected by resolution limit issues [38]. Similar theoretical analysis have been provided for Infomap’s map equation [39] and for some methods based on Hamiltonians of Potts models [40]. Despite these difficulties, here we show that the ASes do exist as communities at different levels of the hierarchical structure.

The difficulties of classical modularity maximization due to the resolution limit and the degeneracy of its maxima are manifested in the poor average values obtained by Deltacom at \(t=1\) (0.08) and by Louvain (0.13) for the recall score of ASes with at least 100 nodes. But the multiresolution algorithms have a good performance when we explore all the possible resolution levels, showing that most of the Autonomous Systems do have a community structure (80% of them have a recall over 0.70). Instead, most of the ASes which were not detected by Deltacom Multiresolution belong to the core of the Internet: in Figure 4 we plot the recall score of the ASes grouped according to their AS rank. The AS rank is a ranking of ASes designed by CAIDA and based on the customer cone size (the set of ASes that an AS can reach using customer links [41]). We observe that low ranked ASes have very good recall scores typically (above 0.7). Instead, most of the ASes found with a low recall have a high AS rank and belong to large carriers. Regarding the points with low AS Rank and low recall, many of them could be identified as IXPs and CDNs, which also belong to the Internet backbone but have a small customer cone.

In Figures 5, 6 and 7 we analyse some particular cases of (a) well-detected, high-ranked ASes, (b) bad-detected, high-ranked ASes and (c) very well-detected, mean-ranked ASes. Each row represents an AS, and the three columns represent (i) the internal node degree distribution (normalized by the node degree), (ii) the internal node degree vs. node degree and (iii) the evolution of the best Jaccard similarity at different resolutions of the Multiresolution Deltacom algorithm (its maximum corresponds to the AS recall score).

In the central pictures of Figure 5 we plot the number of internal edges as a function of node degree for some ASes which were not well-detected. The diagonal dashed line corresponds to \(k_{\mathrm{in}}/k\); the nodes under this line have more external connections than internal ones. The colour indicates if the node was classified inside the community which retrieved the best Jaccard similarity (black) or not (red). We observe that all the ASes in Figure 5 have high degree nodes with more external that internal connections; these nodes are not recognized as part of their ASes and thus induce many of the short-degree nodes into error. Into these group of ASes we found carrier networks and IXCs (e.g., AT&T and Sprint), IXPs (e.g., the London Internet Exchange) and some CDNs (e.g., Akamai and Amazon).

These types of ASes lay in the backbone of the Internet and they are densely connected among them, so that their nodes may not respect the classical notions of community. It is possible that we should think in an overlapping community structure for the core of the Internet, following the ideas in [42] regarding the core-periphery structure of networks and its relation with community structure. In this network, highly overlapping communities might be related with the presence of IXPs or nodes which do not satisfy the strong community definition, for example.

ASes with good recall score are shown in Figures 6 and 7. In them, all the high-degree nodes have a high proportion of internal connections. These nodes usually act as hubs for their ASes; into this category we find medium-sized and large-sized ISPs, university networks and other small ASes. An example of one of these networks is the Google AS, shown in Figure 8. In the lower part of the figure we plot it as detected by the Multiresolution Deltacom algorithm using the 15% of the nodes and the AS size; the recall score of the AS is 0.93. The colour of the nodes represents their eigenvector centrality inside the community with a light blue/blue/violet/red scale for increasing centrality values; the radius represents their degree in the full graph. The upper left picture shows the 15% sample in red, and the upper right figure shows the nodes in the frontier of the AS, also in red. Those nodes from the AS which were not detected by our algorithm are shown in white in the upper left picture. We observe an interesting modular structure, with several small modules strongly connected among them, and with a tree-like structure. Some high-degree nodes act as hubs inside the AS; however, they are not connected to other ASes usually. The AS has also a small connected component, probably due to a bias in the exploration or to the limitations of the AS affiliation algorithm; this small component was not detected. However, the reconstruction of the main component was excellent; the only difference being some small degree nodes not belonging to the AS, but to carrier ASes like AT&T and Level3. Almost all the nodes in the frontier of the main component have small degree and small centrality values. The role of hubs is also visualized in Figure 9 for the UCom Corporation AS.

It is interesting to contrast the general results against some classical notions of community. Looking at the central pictures of Figures 5, 6 and 7 again, we can see that all of them have nodes under the dashed diagonal line which represents the limit of the notion of strong community by Radicchi et al. [43], i.e., having more internal connections than external ones. This notion is quite strict indeed, and neither the real ASes nor the communities found by modularity maximization follow it. In [44], Hu et al. introduced a relaxed version of this notion, in which each node has more internal connections than towards each of the other communities separately; we found that 4% of the nodes (spread through 70% of the ASes) do not follow this definition, either.

The limitations of our algorithm for correctly recovering some of the ASes might also be related to the detectability threshold of community detection [45–47] and to the community detection paradox [48]: even when there are more average internal connections than external ones, algorithms might not succeed to retrieve a community if the difference between these quantities is not above a certain threshold. This limitation not only affects modularity-based methods but is quite general.

Hric et al. [35] have recently used the Jaccard similarity for identifying communities with ground-truth groups as we did. They also observed that community detection algorithms perform very poorly according to this measure and they proposed some conjectures, as the need to change the notion of community or to introduce some non-topological information. In [49], for example, Darst et al. analyse how an a priori knowledge of the number of communities improves the performance by helping to set the resolution level. But, in many complex networks like the Internet there might not be one single resolution at which all the communities can be discovered. Here we have seen that knowing the community sizes and some of their nodes, multiresolution algorithms might increase their performance significantly.

The use of some additional information for the identification of communities may be a trade-off solution considering the complexity and limitations of the community detection problem. Our last experiment, in which we use a sample of nodes for matching the community in the hierarchical structure, is related to the seed set expansion problem (i.e., extracting a community by expanding a local seed set of nodes), which has been recently explored in [50–52].

Thus, our work may shed some light into the problem of improving the performance of community detection algorithms: in the Internet router-level graph, we have shown that the coexistence of communities with multiple resolutions is the main difficulty for the detection algorithms. So, when proposing new conceptions of community structure, we should take into account that very heterogeneous communities must coexist. Regarding the detection of ground-truth communities in real networks, the following question remains open: Can the ground-truth communities be retrieved without additional information, e.g., as significant communities at the right resolution level?

We analysed the internal community structure of the Internet Autonomous Systems by using Deltacom, a multiresolution algorithm. Deltacom performs hierarchical partitioning through modularity optimization at different resolution levels in one single run, and with a low computational cost. The algorithm was made available to the scientific community as an open-source software through SourceForge.

We applied Deltacom to the router-level graph of the Internet obtained from CAIDA, and we observed that most of the ASes which are not in the Internet backbone can be identified as communities if the correct resolution level is used in the community discovery process. However, we showed that several optimization algorithms fail to detect these Autonomous Systems due to their largely variable sizes and structures. With our results, we show that the use of proper additional information can significantly improve the performance of community identification.

Many of the Autonomous Systems in the Internet backbone (and usually identified by a very high AS rank) could not be identified as communities, and we observed that this was due to the presence of high degree nodes with more external connections than internal ones, violating one of the main notions of community structure. Instead, stub ASes and ISP’s have high-degree nodes which act as hubs and provide internal connectivity, while the external connectivity is usually provided by lower-degree nodes. These ASes were well detected as communities.

By correlating the resolution of each AS with its size and using a small fraction of the AS affiliation data, we showed that most of the ASes which do have a community structure can be retrieved. In this way we provide a method for identifying Autonomous Systems with scarce information: a short sample of their nodes and their size.

Finally, we discussed the limitations of the community detection methods for detecting ground-truth communities. These difficulties had been previously observed for many large networks and we also observed it here for the Internet Autonomous Systems. Based on our results, we consider that the use of some local information might arise as a trade-off solution for identifying communities in these cases.

We think that these results on the Internet structure at the router level can also be used for improving Internet topology models and the estimation of properties as node centrality or clustering. The work shall also lead to new discussions regarding the hierarchical structure of the Internet and the detection of significant communities in large heterogeneous networks.