FOSDEM 2020: trends and analysis

As for the last 11 years Cisco sponsored the FOSDEM 2020 conference in Brussels. This is one of the largest, if not the largest, open source conference in the world with this year 872 events spread over 2 days.

From a network point of view the important part is that the main SSID at the conference is IPv6-only, using NAT64 and DNS64 to provide access to IPv4-only resources on the internet. For users which require IPv4 connectivity a ‘legacy’ dual stack network is also available. The name of this SSID is changed yearly so that people consider moving to the IPv6-only network.

As in the last few years we provided and configured the switches and the ASR 1006 router, where we use the netflow and NBAR features to collect the HTTP user agents field the router sees. These user agents are sent by the clients to web servers in clear text and can tell us something about the application, client operating system or architecture. We then analyse this data to see trends.

Unique MAC addresses == number of clients

Last year (our report of 2019) we faced the problem that we saw a lot of artificial MAC addresses on the network, in total we saw 129,959 unique MAC addresses last year. This year we enforced the usage of DHCP on the IPv4 network, strengthened the security and fixed a few bugs and added more checks in or reporting tools. If this worked is unclear, however this year we ‘only’ saw 11,372 unique MAC addresses, of which 9,993 could be attributed to a known manufacturer and 1,379 seem randomized. Of these we could guess the operating system used for 1,007, all of which were running Android 10 on diverse platforms.

Last year we could attribute 11,781 unique addresses, so we are seeing a reduction in the number of clients. This seems to be confirmed by the statistics gathered by the wireless controller, which saw about 6,000 concurrent users in 2019 and about 5,500 in 2020. We assume that more people simply use roaming as for EU guests this is almost always included in their package.

More Bandwidth used by fewer clients

However, these fewer users did consume more bandwidth. Far more bandwidth, input from the internet went from last years 2,960,975,295,300 bytes to 23,658,552,410,903 bytes, a growth of 799%. Also output towards the internet went from 1,755,298,210,796 bytes to 12,715,921,100,135 bytes, a growth of only 724%. Our uplink was almost constantly loaded during the whole event.

We also checked the distribution between the IPv4 and IPv6 traffic and the IPv6 traffic largely trumps the IPv4 traffic, just as it did last year.

When checking the makeup of traffic, we can see the ESP traffic only contributes on the IPv4 network:

NAT64

Of the 8,233 clients whch were on the IPv6-only network at one time, 7,854 or ~95% used the NAT64 feature. We had 7,462 connections which were not using ICMP, UDP or TCP, 109,553 ICMP, 722,511 UDP and 11,496,472 TCP connections over NAT64. Below are the top 10 services carried by NAT64:

As to the destinations it is more complicated. We only see the IP’s and when trying to identify them it’s difficult. In one case there is even a mystery: we see a multicast IP (!). The top 10 destinations handled by NAT64 were:

What can we tell about the clients?

Some devices send the architecture of the CPU, which we can use to collect statistics. Android and Apple Mobile devices in general are ARM based.

When we analyse the MAC addresses we can identify the manufacturer of the device, based on the ‘manuf’ list from the Wireshark sources. This gives the following distribution:

We can try to guess the operating system running on the device, based on the HTTP user agent string. This is not only difficult as some user agents are schizophrenic (what browser is “Mozilla/5.0 (Linux; Android 9; MI 8 Build/PKQ1.180729.001; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/67.0.3396.87 XWEB/1163 MMWEBSDK/191102 Mobile Safari/537.36 MMWEBID/6033 MicroMessenger/7.0.9.1560(0x2700096B) Process/toolsmp NetType/WIFI Language/en ABI/arm64“, and doesn’t it leak way too much information?) but some devices emit only very generic useragents like “ConnMan/1.32+git97.3 wispr”.

So, we manually try to detect the operating systems and architectures used, which means that for some we can only detect it is a Linux running on x86_64, while for others we can only say it is some kind of Apple device. We get the following results:

On which networks do we find the clients?

The rise of IPv6 continues. Like last year we had ~3460 IPv4 DHCP on the legacy network, but we had 6235 clients which stayed on the IPv6 only network and did not go back to the legacy network. 1,998 clients were seen on both the legacy and the IPv6 only network and 2,561 clients never even tried the IPv6 only network. We can identify which clients were on the different networks:

Everybody’s favourite client: The Windows Phone, was on the IPv6 only network, counted as part of the ‘Windows’ family.

Eating addresses like candy?

Some devices rotate through the IPv6 addresses at a high rate, we found a few which used more then 400 IPv6 addresses over the course of the weekend. All of those clients are in fact iPhone 13.3 devices and not some more ‘hacker’ device. Most devices only use a few, and the majority only two: the link-local and the global IP.

What can we say about the traffic?

We continue to see the majority of VPN based ESP traffic on the dual stack legacy network. Again, we used NBAR2 to classify the traffic giving the following results:

asr1k#show ip nbar protocol-discovery interface gi2/0/4  stats byte-count top-n 20
...

Input                    Output
                              -----                    ------
 Protocol                     Byte Count               Byte Count
 ---------------------------- ------------------------ ------------------------
 shoutcast                    47029363893              2112934159987
 ssl                          969293704522             608105734792
 statistical-p2p              52012656918              1110991906422
 unknown                      283397102118             529552327874
 http                         85483662261              713775820214
 google-services              266502623472             147675163463
 proxy-server                 6012538536               333564857312
 binary-over-http             116312457935             1770591381
 amazon-web-services          74296965211              32039361466
 twitter                      68570673404              8287476941
 facebook                     67610858696              7142510259
 akamai                       60348002605              4396237959
 netflix                      56448209276              2820631545
 apple-updates                57130900493              1120491337
 youtube                      55489033038              1850317608
 github                       52301021133              2322815054
 openvpn                      42285183215              8426470948
 instagram                    47000986713              3638936268
 icloud                       20648621225              29671510399
 conference-server            4028241437               43888642924
 Total                        2432202806101            5703975964153

The high amount of traffic for shoutcast is probably the people in the corridor streaming the talk inside of the room they could not enter. It is strange that we have more netflix then github traffic, but as someone told me: “byte for byte github content takes a lot more time to properly enjoy”. We can also check which traffic caused the highest bit rate:

asr1k#show ip nbar protocol-discovery interface gi2/0/4 stats max-bit-rate
...
                             Input                    Output
                              -----                    ------
 Protocol                     30sec Max Bit Rate (bps) 30sec Max Bit Rate (bps)
 ---------------------------- ------------------------ ------------------------
 ssl                          353356000                322821000
 ms-services                  443143000                10610000
 ssh                          42297000                 297418000
 google-services              151221000                91569000
 statistical-p2p              63718000                 162086000
 amazon-web-services          101362000                93969000
 http                         67107000                 114832000
 unknown                      90678000                 85028000
 shoutcast                    10624000                 156234000
 binary-over-http             151637000                2102000
 apple-updates                139678000                3757000
 isakmp                       61012000                 69309000
 ipsec                        66291000                 62469000
 netflix                      76857000                 44271000
 github                       101191000                3399000
 icloud                       65720000                 36435000
 adobe-services               58527000                 43031000
 openvpn                      58213000                 34107000
 youtube                      89154000                 1589000
 statistical-conf-audio       45154000                 44640000
 crashplan                    38509000                 48593000
 proxy-server                 1423000                  83464000
 bittorrent                   51471000                 32060000
 akamai                       75391000                 5746000
 google-downloads             75773000                 2096000
 dropbox                      61787000                 14648000
 http-alt                     44900000                 28934000
 statistical-conf-video       11006000                 61704000
 ms-update                    65564000                 1084000
 ms-office-365                31290000                 33657000
 Total                        2694054000               1991662000

I think only at FOSDEM the third highest protocol in maximum bitrate can be ssh at 2,97,418,000 bps of outgoing traffic. That’s ~35.5 megabytes per second of outgoing traffic.

Looking towards the future:

Next year we will try:

• To get multiple inter-building links. We are reaching limits here and this would help for redundancy.
• To get a second uplink, again we are coming close to the limits of our 1Gbps uplink here.
• To disable 2.4GHz on the main SSID and add extra AP’s to fix coverage holes. This should help with high density situations like in Janson.

The post FOSDEM 2020: trends and analysis appeared first on Cisco Blogs.