Methodology
This documentation presents AKOS Test Net and describes the methodology of the measurement.
1. About AKOS Test Net
AKOS Test Net is a tool to measure speed and quality of the Internet connection provided by the Agency for Communication Networks and Services of the Republic of Slovenia (AKOS), and developed by the company Internet Institute d.o.o..
For the sake of transparency, AKOS Test Net is an open source project, and the source code is available in Github.
The data collected by AKOS Test Net are published as open data in compliance with the privacy policy. The data are published in accordance with the Creative Commons Attribution 4.0 (CC BY 4.0) and is available for download.
For questions and feedback you can contact us by e-mail at akostest@akos-rs.si. |
1.1. Why AKOS Test Net?
AKOS Test Net is an open software project that lets any user measure their fixed, mobile and WiFi Internet connection. Not only in terms of speed and latency, but also the quality and the neutrality. It is an independent tool, improved with the expertise of Regulatory Authorities across Europe and the contributions of thousands of users and collected measurements. The transparency of the methodology and the aggregation of the results, translated to statistics and offered as open data (always respecting privacy and anonymity of the data) makes AKOS Test Net a unique measurement platform for end users, operators, regulators and data analysts with the aim of improving the quality of Internet connection worldwide.
Network speed and latency have a direct impact on the user experience when connecting to different Internet services. From sharing the pictures from your last holiday to streaming the last episode of your favourite series, a slow or unstable connection can result in long download times or low quality video images, which makes every user unhappy. AKOS Test Net provides a way to check your current connection speed and quality, not only in case of trouble, but at any moment and everywhere. In addition, you can check other user’s results on a map, which can helpful when selecting your access network at a given location. Furthermore, AKOS Test Net enables a more extensive diagnostic test, including other parameters as signal strength, or quality in terms of network neutrality (e.g. blocked content, presence of middle-boxes, etc.).
1.2. What is the difference between AKOS Test Net and other speed tests?
Measuring the speed of a broadband connection is a challenging and complex task, which depends on a variety of factors, such as hardware, location of the server or number of users (specially in wireless networks). There are many online tools that offer tests to find out what is your download and upload speed, as well as the delay in your connection. However, most of these tools are not open source, which means that they keep secret the way in which they measure your connection. Since the actual speed of your Internet connection depends on many different factors, different tools may provide different results if they perform the measurement slightly differently. AKOS Test Net has several characteristics that make it stand out among all the other measurement tools:
-
The methodology behind AKOS Test Net and its associated platform has been trusted by several Regulatory Authorities across Europe to measure the broadband connection in their countries. This allows for a homogeneous comparison of the quality of the networks in the countries where it is deployed.
-
AKOS Test Net is not associated with any operator or ISP. Being provided by a Regulatory Authority, it benefits from the independence and expertise of the Agency for Communication Networks and Services of the Republic of Slovenia (AKOS).
-
AKOS Test Net is publicly accessible and the code is published as open source. This is the most transparent way and it ensures that everybody can be able to reproduce how the measurement is performed.
-
AKOS Test Net provides more information about the measurement, including parameters from the client device, the connection as well as a comprehensive set of results regarding quality and neutrality of the Internet connection.
-
The measurement tool is provided with specific documentation, which explains the performed measurements with high level of detail.
-
The measurement servers for AKOS Test Net are located as close as possible to the Slovenian Internet eXchange (SIX). This makes it an optimal location for an independent measurement, since all the main operators and service providers in the country are connected to the Internet eXchange Point (IXP).
-
AKOS Test Net is provided for free and without advertisements in all its versions.
-
AKOS Test Net also provides a combination of measurements and tests, to check not only the speed of your connection, but also its quality according to the net neutrality policies developed in Europe.
-
AKOS Test Net platform makes possible to synchronise the results of various devices and show them in the browser.
-
AKOS Test Net provides a map and visual representation ofthe collected measurements, with the option to filter these results by different parameters, such as, date range, ISP, technology or other test parameters.
-
AKOS Test Net provides statistics on the results, with the option to filter the results according to different parameters.
2. How does AKOS Test Net work?
Similarly to several countries in Europe, AKOS Test Net has adopted the methodology for the measurement based on the open source measurement tool developed by the regulatory authority in Austria, RTR-NetzTest [1]. We measure the connection from your device (we often call it a client device, or just client) and a measurement server (located at an Internet exchange point, to keep the measurement as independent of the operator and the service as possible). The client and the measurement server will send data to each other, which will be used to measure how much data they can transfer in a given amount of time, letting us measure the speed and the latency of your Internet connection.
In order to measure the different parameters, including the ones related to network neutrality, the measurement is structured in different phases. We provide more details about the measurements in Section 3 and Section 4. If after reading this document you have more questions related to the measurement, you can check AKOS Test Net frequently asked questions.
2.1. General requirements (How can I use AKOS Test Net?)
Using AKOS Test Net is very easy. We have made it available in the most commonly used platforms and provide detailed information to run the measurement and understand the results. Any type of connection can be tested, independently of the technology. However, there are some aspects that you should take into account before running a measurement:
-
A measurement is just a snapshot of your Internet access. Specially in the case of wireless networks, the results may be very different depending on the number of users connected to the same network or even the time of day.
-
If you want to have the least unreliable results, make sure that you are not running other applications on your device during the test, specially if they send or receive significant amounts of data (e.g. video streaming, P2P, file downloads, system updates, backups, etc.). These applications will consume a significant part of the available connection bandwidth, limiting the result of AKOS Test Net measurement. If you are connected to a WiFi, make sure that not only your client device, but the other devices (if you have and Ineternet-connected TV, for instance, sharing the same network) are not downloading or uploading data during the measurement.
-
If you are connected to a mobile network, make sure that you have not yet used your monthly data volume (in case of limited tariffs or contracts).
The total data volume transmitted during a measurement depends on the speed of your Internet connection. While the data volume for fixed connections is usually unlimited, in mobile networks the transmission of high volumes of data may incur in significant additional costs. Please keep that in mind before running measurements connected to a cellular network. |
-
Sometimes the hardware of your own client device or the WLAN router (for instance) might be the limiting factors for your network experience. If you want to isolate these or other factors, you can use different devices for the measurement, or measure different networks.
You can increase reliability of your results by repeating the measurement at different times. |
2.2. Supported platforms
AKOS Test Net is available as a mobile app for Android and iOS mobile devices. There is a web version as well, which supports all the most common Internet browsers.
2.2.1. Android
AKOS Test Net is supported on Android devices from version 6.0 and above (also for tablets). You can download the latest version for Android in the Google Play Store.
TODO2.2.2. iOS
AKOS Test Net is supported on Apple devices (iPhone and iPad) from iOS 10. You can download the latest version from Apple’s App Store.
2.2.3. Web browser
You can run AKOS Test Net from Google Chrome, Mozilla Firefox, Apple Safari, Microsoft Edge and Opera at any moment. It does not require installing any additional software or tools.
2.3. Components
Once you have installed your client app or when you visit AKOS Test Net website in your browser, you are ready to start a measurement. The measurement platform makes available several measurement servers, and it includes a control server to manage new connections to clients and taking care of saving the results. In Section 3 you can read more about how the measurement is performed.
3. AKOS Test Net measurement
From the factors that can impact the result of the measurement, the methodology for the measurement itself plays a critical role. That is one of the main reasons for different measurement tools (Speedtest, Fast, M-Lab, etc.) to provide different results. However, many of the currently available tools are not open source and there is little disclosure about the specific methodology they follow to provide a result. AKOS Test Net is open source and aims to provide transparent and comprehensible information. With every measurement, AKOS Test Net collects information related to the Internet connection. There are two sets of measurements:
-
The base measurement provides downlink and uplink speeds and latency of the connection. If you are running the measurement from a mobile app, the base measurement will also provide jitter (variation in the delay) and packet loss measurements.
-
Additionally, the user can enable the Quality of Service (QoS) measurement. This is a set of measurements that checks the viability of other services related to network neutrality, which can provide very insightful information to the user and regulatory authorities about the neutrality policies. You can read more details about these measurements in AKOS Test Net Quality of Service measurement.
The base measurement aims at estimating the performance of the user’s Internet connection by measuring download and upload speeds and latency (jitter and packet loss are measured as well from the mobile applications). In order to do so, every measurement consists of several phases that run sequentially. We provide a brief summary of the different phases.
Several measurements cannot run simultaneously. |
3.1. Phase 1: Initialization
The client tries to establish a secure connection to the control server and they exchange the configuration parameters that are needed to perform the measurement.
3.2. Phase 2: Downlink pre-test
This phase aims to get the connection ready to perform the measurements that will follow.
With the information received from the control server, the client can start one or more parallel connections, also called flows, to the measurement server. The number of connections is variable depending on the speed of your Internet connection. In slower connections, a higher number of flows will result on each of them competing with one another for low bandwidth, leading to even worse results and a less accurate measurement. On the other hand, for faster connections, opening several connections simultaneously will allow to reach the maximum capacity of the link faster, improving the accuracy of the result.
In this phase, the client and the server will estimate the optimal number of parallel connections based on the state of the network. They will try to set a higher number of connections if the network conditions allow it. Otherwise, it will be reduced to just one connection.
The number of parallel connections is configurable by the user in AKOS Test Net. However, if you are not sure how to select this number, you can let the client and server decide based on the current network conditions.
3.3. Phase 3: Latency
In this phase, the client will send several short messages to the server in short intervals, wait for the reply from the server and send a confirmation when this reply is received. Then, the client and the server measure the time between the transmission of their message and the reception of the reply. The measurement result Ping shows the median of all the values measured by the server during this phase.
3.4. Phase 4: Downlink
In this phase, the client requests data through each of the parallel connections to the server (according to the result from the downlink pre-test). The server will transmit the data to the client in chunks of the size indicated by the client. The client, upon the reception of each chunk, tracks the volume of data received and the time needed to receive it.
After receiving the last chunk of data, the client will calculate the download speed of the connection taking into account the values measured for all the connections.
3.5. Phase 5: Uplink pre-test
Similarly to the downlink pre-test phase, this phase aims to get the connection ready for the speed measurement in the direction from the client to the server. Again, if the state of the network does not allow it, only one connection will be used for the uplink measurement.
3.6. Phase 6: Uplink
In this phase, the client will send data to the server through each of the parallel connections (according to the result from the uplink pre-test) in chunks of a size established by the client. The server will measure the time and the amount of data received and send this information back to the client. The client, after receiving all the information from the server will calculate the uplink speed.
3.7. Phase 7: Finalization
After finishing the measurements, the client will close the connections to the measurement server and send all the collected data to the control server.
4. AKOS Test Net Quality of Service measurement
Based on BEREC’s recommendations AKOS Test Net provides the tools to evaluate characteristics of the network connection that have a decisive influence on the quality and the transparency of the Internet access. In the recent years there is a strict regulation under development in Europe to ensure transparency and non-discriminatory treatment of data traffic in the Internet access services, protecting end user’s rights.
We call this set of measurements Quality of Service measurements and it consists on a set of 7 test groups, which are carried out after the base measurement, if the option is enabled from the mobile applications for Android and iOS. After all the tests are complete, we compute a QoS score taking into account how many tests failed.
4.1. Web content modification
The goal of this test is to detect, whether the content is modified during the transmition of the data between the server and the client in some intermediate network device. The client requests data with a known checksum from the configured web server. Upon reception of the data, the client compares the checksum of the received data to the original checksum. A difference in the checksums indicates the manipulation of the data on the way to the client, and the test is considered not successful.
4.2. Website loading
In addition to checking modifications of the content, we monitor the size and the time it takes to download one or more reference web pages. If the pages can be rendered before a defined timeout, the test is successful.
4.3. Connection transparency
The use of proxies is a common technique in the current Internet, for instance for caching the most frequently visited webpages, compress content or protect clients from potentially harmful content. In this test, we aim at detecting the presence of proxies between client and server, or "non-transparent" connections. In that case, it cannot be guaranteed that the websites correspond exactly to those on the server. If we detect the presence of such proxy, the test is unsucessful. In this test the client sends correctly formatted HTTP requests, as well as incorrectly formatted HTTP requests to the measurement server. The measurement server reply by sending the exact copy of the request back to the client. When there is an http proxy between the client and the measurement server, it will react to the incorrectly formatted HTTP requests by sending an error message, indicating the malformed HTTP request. In this case, the test is unsucessful.
4.4. DNS
DNS (Domain Name Service) translates the name of the website we visit on the Internet to an IP address, which will indicate the specific location of the content we can access. It is therefore essential to check if the translation provided by the DNS service can be trusted and is compliant with the network neutrality policies. This test is successful if we can confirm that the DNS data is consistent.
4.5. TCP and UDP blocked ports
Common protocols used for different Internet services use certain port numbers in their connections. If a certain port is blocked, by the ISP or by the network administrator, any communication attempt through this port will fail. In this test we check whether any TCP or UDP ports are blocked, and therefore certain services are banned, as data cannot be sent or received using those ports. Note that in business environments it is a common practice to block certain ports for security reasons (firewall) or only the relevant ports (and services) are made available.
4.6. Route to the server (Traceroute)
In this test we measure the distance (in terms of delay and network hops) between the client and a target location.
5. Understanding the results
After a measurement is complete, AKOS Test Net will show you the results, with different level of detail, depending on whether you are using a mobile client or a web browser.
The speed values that you should expect depend not only on the network status, but on your access technology (DSL, WiFi, mobile 4G/5G, optical fibre, etc.) and potential limitations of your client device, among other factors. The download and upload speeds are shown in Megabits per second (Mbps) and the latency (Ping) in milliseconds.
How to interpret if the result is good enough or bad depends on the type of service you would want to use. For web browsing, often 2 Mbps are often good enough. Voice services may not need high bandwidth, but delay and jitter (how much the delay varies) are very important. Therefore, you should not only consider the download speed, but also the upload (especially for cloud services or file transfers) and latency.
To help you, we have included a red/orange/green color code to indicate what is a high/medium/low result. Note that the higher speeds (indicated by green) might only be achievable by certain technologies. Depending on your access network technology, a red speed value could indicate that your connection uses an older technology and might need an upgrade.
When interpreting the results, you should keep in mind that even if the results differ from the advertised speed in your Internet connection contract, this does not necessarily mean that your Internet provider is not fulfilling the agreement. Moreover, operators design their service offer in different ways. Typically, the speed of the data connection is indicated as an upper bound ("up to" values).
If after several measurements the results show a significant difference from the advertised bandwidth, this may indicate that there are problems with the connection, and it should be analysed by a professional. The result can also be impaired by various technological factors like the WLAN router, system configuration, etc. In addition, if your network is shared among several users, the total capacity available will be distributed among them. Moreover, although we work to keep our systems available at all times, it may happen that the test server or its connection are overloaded.
Since operators implement different policies to route the traffic through their networks, it might happen that the result of AKOS Test Net reaches a higher value than normally accessible by other Internet services. However, the design of the measurement tries to follow common Internet practices to be as close to the user experience as possible. Moreover, if services are treated differently, this might be a hint of the lack of network neutrality.
In any case, the most reliable way to avoid biases on the result and random errors, is to repeat the measurement frequently.
If you run AKOS Test Net several times and have doubts about your results or think that there might be a problem, please contact us at akostest@akos-rs.si.
If you are a more experienced user and are curious about more detailed aspects of your network connection you can access and analyse the data from your (and other user’s) measurements. Please check Statistics and data export for details. |
5.1. Base measurement results
If you run AKOS Test Net on the browser, you will see other results below the speed and latency, including the location, the total duration of the measurement, how many parallel connections were open during the test and the duration of the upload and download phases, among others. You can click on some of these values to make a more specific search.
On your measurement result page, there is also a graphic representation of the download and upload phases. By clicking on the graph, a table with detailed partial information of the amount of received data and the time will be displayed.
The result page includes as well a map with the location of the measurement, where you can check the location accuracy.
Finally, you can save the measurement results to a file. For more details about how to export data see Section 7.
6. Statistics
AKOS Test Net collects data related to the measurement and the parameters from your Internet connection. Together with the data from all users, we can analyse the quality of the Internet connection and its evolution. Thanks to the open data paradigm, we make available anonymized data so everybody can access it without compromising other user’s privacy.
AKOS Test Net is a crowdsourced measurement platform. This means that the more measurements we can collect from different users, at different locations and connected to different networks, with different devices, the more we can trust the results and the statistics we collect. The more people use it, the more we can trust the results! |
To make easier your first contact with the data, we have designed a Statistics page which summarizes data in three different tables:
-
Last 10 measurements: We present a table with the last 10 measurements performed with AKOS Test Net. The table includes date and time of the measurement, operator and client device information, download and upload speeds, latency and signal strength if available. The data in the table can be ordered according to any column. By default, it is order according to the most recent measurements in time. In this table, if you click on the operator’s name, you can access a website with detailed information about that specific measurement.
-
Operator and Device summary statistics: To look further into the statistics, we include filters to select the type of client device/connection (browser, WLAN or Mobile), the time range (1 day to 2 years) to look into and the type of statistic (Median, 20 or 80 percentiles). After applying these filters, we show two tables with the measurement results per operator and per client device.
7. Data export and open data
In addition to being able to access the results from your own measurements and from other users', it is possible to share or export (download) them in different ways.
-
export as CSV (comma separated values) file
-
export as excel table
-
If you are interested in performing a deeper data analysis or you are doing research and would like to have a more script-friendly format, AKOS Test Net gives the possibility to download the open data in CSV, JSON or XML files. There are two options:
-
Monthly export: After selecting the file format of your choice, you can select and download the measurement data from a specific month.
-
Full export: By selecting the full export, you will download the data from all the results available in the file format of your choice.
-
8. Additional features
AKOS Test Net aims to be a comprehensive tool, useful for end users, operators, regulators and data scientists. As such, we continue developing new features to make it a powerful resource, based on our experience and the feedback from regular users and European regulatory authorities.
8.1. Loop mode
Several users reported that under certain circumstances, it is useful to run several measurements frequently. The Loop mode allows to set a specific number of measurements (or unlimited, only stopping when the user explicitly chooses to stop) to run at an interval specified by the user, with a minimum interval of 60 minutes.