NTIA’s excellent guidance makes it crystal clear: Unlike FCC’s Fabric, you can (and should) incorporate speed tests in your State’s BEAD Challenge Process. You are encouraged to seek this critical form of evidence, as long as the data you gather meet acceptable evidence guidelines.
Speed tests (or performance tests, as we like to call them) are your office’s best chance at getting objective ground truth about the broadband reality for locations and areas.
How can you use performance tests in your office’s challenge process? There are two paths you can take:
Data-driven broadband offices will choose #1 – using performance tests to make better choices. The best broadband offices will go a step further and supply the tools that help Permissible Challengers generate NTIA-compliant acceptable evidence in performance tests generated in a simple-to-use, centralized portal.
NTIA gave you a strong, thoughtful framework in which to develop and assert your own evidentiary standard. Why should you bother with speed tests? Because source-of-truth observational data like the kinds speed tests provide give you the best opportunity to validate provider claims of service. Gathering evidence around observed performance lets you shape a more objective view of where and to what extent you should focus your team’s efforts and BEAD resources. A well-executed evidence process can set your office up for the objectivity you’ll need to deliver generations of success.
Providers can use speed tests in rebuttals.
You can use speed tests to build your understanding of broadband reality.
You can specify that Permissible Challengers leverage your test.
If you deploy your own, you’ll have a much better shot at achieving success with BEAD
How will you get defensible, compliant evidence? The NTIA Model Challenge Process defines a clear but high bar for collecting source-of-truth (aka “citizen-sourced”) service quality evidence. It crystallizes what’s allowable and sets a specific standard for acceptable evidence.
While NTIA provides crystal clear guidance on what counts as acceptable evidence, it’s up to your state how you go about obtaining it. It’s up to you to supply the tools and framework in which you collect data for your Challenge Process from Permissible Challengers.
If you decide to go the data-driven, proactive route for your BEAD implementation, in order to give your office the best chance at using performance data, it’s recommended that you get your Performance Test & Survey set up right away.
Within NTIA’s 10 phase challenge process, your office must include the following four step-by-step phases within its challenge process:
NTIA defines who can submit challenges to states, known as Permissible Challengers:
At this time, individuals may not submit challenges directly, but states can work with on-the-ground partners using permission-based tools in order to ensure efficient, centralized aggregation of submissions.
The rest of this quick post outlines how your state’s office can meet NTIA’s higher bar for acceptable evidence, along with a few tips for making the most of your state’s evidence discovery process without needing to hire an Army.
Your map is an important tool, but you need much more than a map in order to make effective use of speed tests.
In addition to your map, your published challenge process site should include:
Provide a secure login for your Permissible Challengers (local govs, Tribal govs, non-profits, and providers). This way, your team has a structured relationship with potential challengers and applicants from the outset, giving you the ability to validate and approve Permissible Challengers in an orderly, equitable way.
Through your Challenge Process Coordinator you can run an efficient, centralized process that produces NTIA-compliant acceptable evidence as the default – dramatically reducing your time and energy spent collating data through email attachments, freeing you up to think strategically and provide extra coverage to small, local & non-traditional providers.
The best systems let the entity seamlessly roll over into grant applications without separate credentials.
Make sure your system rigorously secures any evidence gathered.
Off-the-shelf sales force systems will not provide the specific workflows you’ll need to coordinate among challengers.
Your office will realistically receive more than 100,000 challenges from dozens of challengers. Some offices will receive 1,000,000 challenges from hundreds of challengers.
While Permissible Challengers can supply their own testing infrastructure, making your own official, recommended Performance Test & Survey available to Permissible Challengers with clear usage brings many key advantages to your office:
Making your challenge process inclusive helps you empower small, local & non-traditional providers while avoiding unnecessary lawsuits and process delays. At a minimum, your Performance Test & Survey and Challenge Process Coordinator should:
Since you only get one chance to validate your broadband audit and any provider rebuttals, you might as well take the opportunity to verify service quality in areas you know or suspect to be historically underserved.
Consider reaching out to occupants of BSLs in historically underserved areas if challengers have submitted large quantities of challenges. The “trust but verify” principle would mean using the set of Broadband Serviceable Locations contained in of challenges from all challengers who submitted more than 1,000 disputed challenges as your outbound mailing list.
This effort will take a little printing budget and team effort but can pay of in dividends for your office since:
Read NTIA’s challenge process guidance and Model Challenge Process carefully. Acceptable evidence has a high bar. You are encouraged to develop your office’s evidentiary standard on the minimums supplied by NTIA. You should:
For best results, remind test takers to take tests at specific on-peak hours, and get a time series (more than one test at different times) from BSL occupants in hotly contested areas.
In the theme of viewing NTIA’s guidance as minimum requirements, it’s Important to remember they do set out a basic latency threshold of not-to-exceed 100ms round trip.
While BEAD uses upload and download, you are encouraged to incorporate other attributes into your evaluation process. You might as well gather other dimensions as well
Ideally your tests go deeper than general latency.
Read the BITAG Latency Explained report for a deeper understanding of why buffer-bloat (“working latency”) should be measured in your office’s performance diagnostics.
While Community Anchor Institutions (CAIs) were ignored in FCC’s Fabric, NTIA’s guidance makes abundantly clear that CAIs are very much at the table in BEAD. Your office can go beyond ensuring your CAIs have gigabit symmetrical service and empower them to help you engage BSL occupants in generating meaningful performance test & survey data.
By supplying your Performance Test & Survey as part of your Challenge Process Coordinator, you empower your Permissible Challengers including counties, Tribal governments, regional groups, providers, and non-profits to help your team to make objective, data-driven decisions.
If you’re required to identify CAIs, you might as well provide them with your Performance Test & Survey tools and treat them as partners in collecting meaningful source-of-truth data. Performance Test & Survey was developed to follow the Broadband Mapping Coalition’s recommended methodology for broadband performance testing. States should consult the BMC recommended methodology if implementing their own performance testing systems.
Please remember you’re required to identify CAIs, including CAIs on Tribal Lands as well. When working with Tribes, please ensure your office adheres to Tribal Data Sovereignty. For help engaging Tribes in a compliant way, please ping Tribal Ready.
In addition to CAIs, Permissible Challengers including local gov, Tribal gov, non-profits and providers can be major allies in driving adoption of your data-driven efforts – if you provide them the bridge between your office and their efforts.
NTIA’s provided crystal clarity: you can and should incorporate performance data into your Challenge Process. It’s up to you how you get there. Whether you build your own technology from scratch or license the Ready State Grantor Platform, you can, should, and must figure out how to incorporate source-of-truth evidence in your challenge process in order to make the most of your BEAD resources. It’s the best way for you to get to the bottom of where service really is needed.
Ready is a Public Benefit Corporation led by a team of computer scientists, data scientists and NTIA alums devoted to helping broadband professionals solve the digital divide.