The proliferation of high-speed internet has fundamentally altered modern society, yet significant disparities in broadband access and performance persist. Understanding these disparities and their implications requires robust, granular data on internet performance. The FLOTO project aims to contribute to this understanding by deploying 1,000 edge devices to measure broadband performance in households across select urban areas in the United States.

In the context of computer science education, the availability of diverse, real-world network performance data is crucial. As networking concepts become increasingly complex, educators face the challenge of bridging the gap between theoretical knowledge and practical application. FLOTO data offers an opportunity to address this challenge by providing students with hands-on experience analyzing and interpreting actual network performance metrics.

Current Broadband Data Landscape

To contextualize FLOTO’s contribution, it is essential to examine the existing broadband data ecosystem:

FCC Form 477 Data

This self-reported data from Internet Service Providers (ISPs) has been a primary source for broadband availability studies. However, it has been criticized for potentially overestimating coverage and lacking any performance data of actual service connections, such as bandwidth, latency, and other reliability metrics. The FCC’s recent Broadband Data Collection (BDC) program aims to address coverage overestimation but does not yet report any performance data.

Crowd-Sourced Speed Test Data

Organizations like Measurement Lab (MLab) and Ookla provide free speed test tools that collect hundreds of millions of user-initiated bandwidth tests (upload speed, download speed, latency to testing servers, and jitter) each year around the world. Users can run these speed tests from a web browser or a command line. Each test that runs is collected and stored by the speed test developer. For example, Ookla has collected over 55 billion tests globally since 2006, and MLab has similar data volumes.

While massive and extensive, both datasets suffer from selection bias and inconsistent testing conditions – similar to all crowdsourced data. For example, people might be more likely to take speed tests when they have high-quality internet, or they may take a speed test while using their WiFi network or a mobile network. These drawbacks limit crowdsourced performance data for assessing performance across various network and geographic dimensions. Speed test data also lacks critical metadata about the connection being measured. For researchers interested in isolating a specific aspect of performance, these datasets are not useful due to sampling issues and uncertainty around how the test was conducted.

Measuring Broadband America (MBA)

The FCC’s MBA program utilizes dedicated devices for consistent measurements. The program began in 2011 recruiting volunteers to host “whiteboxes” developed by a program partner (SamKnows) with network measurement software. These nodes collect performance data, which is published along with verified ISP metadata annually. According to FCC’s latest report, there are 3455 nodes (or “Units”) that reported data for 2021. The FCC states that the data “reflect stable network conditions that provide the most accurate view of a provider’s performance under controlled conditions.”

The FCC’s dataset is indeed a highly valuable data source for fixed broadband in the United States. Nonetheless, it does suffer from two main drawbacks. First, MBA data is time lagged on release. Raw data from the MBA program is published monthly, while cleaned, processed data is published years after initial collection. For example, the FCC’s latest validated data is from 2021. The time lag in data availability closes certain research avenues, such as those requiring real-time analysis. Second, while the FCC does distribute multiple nodes across the United States, these nodes are fixed in location and are not open to research applications without express approval from the FCC. Furthermore, the sampling methodology the FCC uses, which focuses on the breadth of ISP and access coverage, cannot help researchers answer complex questions about local network performance in a community over time. As an example, the MBA program has a single node in all of Chicago. It does not have sufficient coverage of residential network connections to provide the needed data.

Ripe Atlas

Ripe Atlas is a global network of probes and anchors that measure internet connectivity and provide data on various network parameters, including latency, packet loss, and traceroute information. Launched by the RIPE NCC, this program aims to offer a comprehensive view of internet health and performance through distributed measurements. Unlike crowdsourced data, Ripe Atlas uses dedicated hardware probes and software agents installed at various locations worldwide, ensuring consistent and reliable data collection. These probes continuously perform active measurements, providing valuable insights into the performance of internet connections.

The Ripe Atlas dataset is extensive, covering multiple continents and a wide range of network environments, from residential broadband to data centers. The data collected is available in real-time and can be accessed via APIs, making it highly suitable for research that requires up-to-date network performance metrics. However, there are limitations. While Ripe Atlas excels in providing connectivity and performance metrics, it does not offer detailed information on user experiences or application performance. This makes it less effective for studies focused on end-user quality of service without supplementary data sources. Ripe Atlas also restricts measurements to a limited subset and does not allow the deployment of applications to collect new or different data.

While these sources provide valuable insights, there remain gaps in the current data landscape. These include the need for more granular, consistent measurements in specific geographic contexts, data collected under controlled conditions, and integrated datasets that combine multiple network performance metrics.

What Makes a Good Broadband Dataset?

This question will of course depend on the specific research question that you want to answer. However, for localized research and comparison of performance across access networks within a specific geographic area, there are four requirements that need to be met.

First, the data needs to include performance metrics that reflect real-world network conditions, rather than reported metrics or inferred indirect metrics. This condition disqualifies many datasets, like the FCC 477 data, that rely on reported performance. However, it is also important to note that measurement data alone is insufficient. Careful consideration needs to be given to the sampling frame. For example, the MBA’s devices measure real-world networks, but they are, by the FCC’s own characterization, reflecting “controlled conditions.” Researchers interested in the behavior of networks outside of controlled conditions may want to look elsewhere. The MBA data may provide decent insight into whether ISPs are capable of providing advertised speeds to the nodes they’ve deployed; it says nothing about how often ISPs are providing such service to a community over time to nodes “in the wild.” These questions require deployments that achieve a depth of sampling within a community that is not available through the MBA program, while still maintaining the robust measurement methodology that the program espouses.

Second, the measurement needs to be frequent, consistent, and high-quality. This means that measurement should be programmatic in standard time intervals initiated on a defined schedule rather than initiated by a human. In the fast world of networking, these time intervals might be on the scale of microseconds or seconds. Measurement of wired performance requires that the node have a physical connection to the access link. This point is where many speed test datasets fall flat. These tests are not methodically scheduled to optimize data for your research. Standardizing measurement schedules and intervals is essential for research into broadband performance and how it varies over time.

Third, the data should provide additional metadata about the connection for each measurement node. Useful metadata includes the level of service that the ISP provisions at the location (ISPs offer different performance tiers for different prices; offerings can even vary across regions), the access technology used for last-mile service (copper, fiber, coax), geographic location, and on-premise hardware such as routers and modems. These metadata can provide meaningful controls to isolate observed discrepancies across service locations or access networks.

Fourth, the dataset should be dynamic and adaptive to evolving research requirements. Many existing platforms for data collection do not give researchers the flexibility to explore new questions in the most efficient and effective manner. If a platform allows the deployment of custom applications at all, which most do not, these applications are subject to scrutiny from, in the case of the MBA devices, the FCC and potentially its interest groups (ISPs included). An open-source platform run by researchers for researchers is needed. As new measurements are deployed or old ones are improved, the dataset will grow to encompass the new data generated by the community. This feedback loop is a critical missing piece in the current data landscape.

FLOTO: Dataset Characteristics and Methodology

FLOTO addresses some of these gaps by offering a dataset with the following characteristics:

1. Deep, Comprehensive Performance Measurement: FLOTO collects a comprehensive range of network performance measurements, including:
   • • • Speed tests (utilizing both Ookla and MLab methodologies)
       • Latency measurements (ping, DNS lookup, Last-Mile)
       • Traceroute data
       • IP address information

   FLOTO devices are installed directly on the access point of residential networks. This setup facilitates accurate measurement by eliminating variables such as WiFi interference or device-specific issues. The consistent environment ensures high-quality data collection, potentially offering more reliable insights than larger volumes of low quality data. These measurements are conducted on a regular schedule, providing consistent, longitudinal data that allows for trend analysis over time.

2. Adaptive: Users can customize their measurement schedules and parameters (with YAML or TOML) to fit their research requirements. Users can even develop new measurements and deploy them on FLOTO to collect data immediately. Measurement on FLOTO is adaptive, everything from what measurements the nodes run to where the nodes are placed and what sensors they include.
3. Local and Dense: Current FLOTO deployments are concentrated in specific urban areas, primarily Chicago and Milwaukee. This focus enables in-depth studies of urban network performance and allows for potential correlation with local infrastructure data. The density of deployments in these areas (e.g., over 100 nodes in Chicago compared to MBA’s single node) provides a more detailed picture of residential internet performance within these communities.

Potential Applications in Computer Science Education and Research

The FLOTO dataset offers several opportunities for enhancing computer science education and research, particularly in the areas of networking, data science, and cybersecurity:

1. Comparative Network Performance Analysis Students can engage in critical analysis by comparing FLOTO data with other datasets like Ookla or MBA. This exercise can illuminate the impact of different measurement methodologies on results and highlight the importance of understanding data collection contexts. For instance, students might investigate how FLOTO’s access-point measurements differ from user-initiated tests in the same geographic area.
2. Data Science and Visualization Techniques The multi-dimensional nature of FLOTO data (combining speed, latency, and routing information) presents valuable opportunities for teaching advanced data manipulation and visualization techniques. Students can explore challenges in merging heterogeneous network data and develop approaches to effectively visualize complex, time-series network performance data.
3. Machine Learning Applications FLOTO’s consistent, longitudinal data is well-suited for machine learning projects. Potential applications include:
   • Predictive modeling of urban network performance based on historical data
   • Anomaly detection in network behavior, which could be particularly interesting given the dense urban deployment of FLOTO devices
4. Urban Network Architecture Studies The concentration of FLOTO devices in specific urban areas allows for detailed studies of how network architectures impact performance in dense environments. Students can analyze real-world protocol behavior and investigate how local infrastructure choices affect user experiences.
5. Network Security Education Longitudinal analysis of FLOTO data can be used to establish baselines for normal network behavior, teaching students to identify potential security-relevant anomalies. This application bridges theoretical security concepts with practical, data-driven analysis.
6. End-to-End Data Collection and Analysis FLOTO’s edge computing approach provides an excellent case study for teaching distributed data collection methodologies. Students can engage with the entire data pipeline, from collection at the edge to analysis in the cloud, gaining valuable experience in modern data architectures.

Data Access and Integration

To facilitate the use of FLOTO data in educational settings, educators and researchers can explore sample FLOTO data through the project’s online portal. This allows for an initial assessment of the dataset’s potential for various educational and research applications. FLOTO Data Portal. A comprehensive data API is currently in development, which will allow users to query the FLOTO dataset directly.

Conclusion

The FLOTO project represents a valuable addition to the landscape of broadband performance data, offering a unique combination of dense urban deployment, consistent measurement methodology, and comprehensive performance metrics. While not intended to replace existing data sources, FLOTO complements them by providing granular, longitudinal data from specific urban environments.

In the context of computer science education, FLOTO data presents numerous opportunities for enhancing the teaching of networking concepts, data science techniques, and cybersecurity principles. By working with this real-world dataset, students can bridge the gap between theoretical knowledge and practical application, developing critical skills in data analysis and interpretation.

As the project evolves, there is potential for expanding the geographic scope of FLOTO deployments and for developing new measurement capabilities. Feedback from educators and researchers using the dataset will be crucial in shaping these future directions.

We encourage the academic community to explore the possibilities offered by the FLOTO dataset, to engage in collaborative research leveraging this resource, and to contribute to the ongoing dialogue about broadband performance measurement methodologies.

References and Additional Resources

1. Federal Communications Commission. (2024). Broadband Data Collection. https://www.fcc.gov/BroadbandData
2. Ookla. (2024). Speedtest Global Index. https://www.speedtest.net/global-index
3. Federal Communications Commission. (2024). Measuring Broadband America. https://www.fcc.gov/general/measuring-broadband-america
4. Measurement Lab. (2024). M-Lab. https://www.measurementlab.net/
5. FLOTO Project. (2024). FLOTO: Flexible Open-source Testbed of Objects. [Insert actual URL when available]
6. Bauer, S., Clark, D. D., & Lehr, W. (2010). Understanding broadband speed measurements. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1988332
7. Feamster, N., & Livingood, J. (2019). Measuring Internet Speed: Current Challenges and Future Recommendations. Communications of the ACM, 62(12), 86-93.