Get Defensible, Transparent, NTIA-Compliant Speed Tests

^{State Broadband Offices: The NTIA provided crystal clarity in the Model Challenge Process. You can (and should) incorporate speed test data in order to avoid pitfalls in your BEAD implementation. Here’s how to do it effectively while meeting acceptable evidence requirements.}

NTIA’s excellent guidance makes it crystal clear: Unlike FCC’s Fabric, you can (and should) incorporate speed tests in your State’s BEAD Challenge Process. You are encouraged to seek this critical form of evidence, as long as the data you gather meet acceptable evidence guidelines.

Speed tests (or performance tests, as we like to call them) are your office’s best chance at getting objective ground truth about the broadband reality for locations and areas.

How can you use performance tests in your office’s challenge process? There are two paths you can take:

1. Embrace performance tests for validating service levels and challenges to help your office make defensible, objective decisions in your BEAD allocation.
2. Ignore performance tests for your validation (but please note that providers can still submit tests to you as evidence).

Data-driven broadband offices will choose #1 – using performance tests to make better choices. The best broadband offices will go a step further and supply the tools that help Permissible Challengers generate NTIA-compliant acceptable evidence in performance tests generated in a simple-to-use, centralized portal.

Why Should You Bother with Speed Tests?

NTIA gave you a strong, thoughtful framework in which to develop and assert your own evidentiary standard. Why should you bother with speed tests? Because source-of-truth observational data like the kinds speed tests provide give you the best opportunity to validate provider claims of service. Gathering evidence around observed performance lets you shape a more objective view of where and to what extent you should focus your team’s efforts and BEAD resources. A well-executed evidence process can set your office up for the objectivity you’ll need to deliver generations of success. 

Providers can use speed tests in rebuttals. 

You can use speed tests to build your understanding of broadband reality.
You can specify that Permissible Challengers leverage your test. 

If you deploy your own, you’ll have a much better shot at achieving success with BEAD

How will you get defensible, compliant evidence? The NTIA Model Challenge Process defines a clear but high bar for collecting source-of-truth (aka “citizen-sourced”) service quality evidence. It crystallizes what’s allowable and sets a specific standard for acceptable evidence.

While NTIA provides crystal clear guidance on what counts as acceptable evidence, it’s up to your state how you go about obtaining it. It’s up to you to supply the tools and framework in which you collect data for your Challenge Process from Permissible Challengers. 
 

When Should Our State / Territory Gather Performance Test Data? 

If you decide to go the data-driven, proactive route for your BEAD implementation, in order to give your office the best chance at using performance data, it’s recommended that you get your Performance Test & Survey set up right away. 

Within NTIA’s 10 phase challenge process, your office must include the following four step-by-step phases within its challenge process:

1. Eligible locations published
   • Include evaluation criteria, and registration process.
   • NTIA lets your office define its own “rigorous evidentiary review.” 
   • While you can propose a deviation from NTIA’s guidelines, they might not approve the deviation, and they’ve already thought of everything you need to run an empirically defensible process. So it’s best to focus on making the most of your time and resources in facilitating the collection of acceptable evidence from Permissible Challengers. 
   • Offer your office’s official challenge process with challenger login and testing framework.
   • Publish your Broadband Audits to show where your evidence suggests that service is underserved, unserved, or unavailable.
2. Challenge phase
   • You can proactively seek performance test data as validation of provider claims where you deem necessary, for example, for any provider submitting more than 1,000 challenges that do not match your broadband audit data.
3. Rebuttal
   • Avoid Wait & See during this period. Use this time to get additional validation.
   • You can also set an expiration on challenges received if evidence isn’t submitted to reject .
4. Final determination
   • Incorporate performance tests and survey data gathered into your Challenge Process Coordinator.
   • Publish your sustained challenges and rejected challenges with reason code(s) to your public map and provider management system.

Who Can Submit Challenges?

NTIA defines who can submit challenges to states, known as Permissible Challengers:

• Local governments including but not limited to counties, municipalities, regional organizations and councils of local governments
  Tribal governments – but bear in mind that you must follow Tribal Data Sovereignty guidelines while cooperating with Tribal governments.
• Providers with service in your state / territory.
• Note that providers can submit their own tests in rebuttal, but that such submissions need to meet acceptable evidence standards, and must demonstrate at least 80% of target upload and download speeds.
• Non-profit organizations such as state-specific associations (for instance MERIT Network), or national allies such as Superhighway EDU, National Digital Inclusion Alliance, Connect Humanity, and more.

At this time, individuals may not submit challenges directly, but states can work with on-the-ground partners using permission-based tools in order to ensure efficient, centralized aggregation of submissions.

The rest of this quick post outlines how your state’s office can meet NTIA’s higher bar for acceptable evidence, along with a few tips for making the most of your state’s evidence discovery process without needing to hire an Army. 

Get Defensible, Transparent, NTIA-Compliant Speed Tests

1. You Need More than a Map

Your map is an important tool, but you need much more than a map in order to make effective use of speed tests. 

In addition to your map, your published challenge process site should include:

• Guidelines that clearly explain your evidentiary standard.
• A way for Permissible Challengers to register with your office.
• Your broadband audits showing Community Anchor Institutions, last known service level, and more.
• Your office’s official testing tool, for Permissible Challengers to utilize.

2. Require Permissible Challengers to register with your office

Provide a secure login for your Permissible Challengers (local govs, Tribal govs, non-profits, and providers). This way, your team has a structured relationship with potential challengers and applicants from the outset, giving you the ability to validate and approve Permissible Challengers in an orderly, equitable way.

Through your Challenge Process Coordinator you can run an efficient, centralized process that produces NTIA-compliant acceptable evidence as the default – dramatically reducing your time and energy spent collating data through email attachments, freeing you up to think strategically and provide extra coverage to small, local & non-traditional providers.

The best systems let the entity seamlessly roll over into grant applications without separate credentials. 

Make sure your system rigorously secures any evidence gathered. 

Off-the-shelf sales force systems will not provide the specific workflows you’ll need to coordinate among challengers. 

3. Supply your performance test & survey for Permissible Challengers

Your office will realistically receive more than 100,000 challenges from dozens of challengers. Some offices will receive 1,000,000 challenges from hundreds of challengers. 

While Permissible Challengers can supply their own testing infrastructure, making your own official, recommended Performance Test & Survey available to Permissible Challengers with clear usage brings many key advantages to your office: 

• Ensures the highest quality, NTIA-compliant data is generated.
• Avoids wasting everyone’s time and energy with fragmented tests, non-acceptable evidence, and messy back and forth. 
• Makes it easy to coordinate with Tribes while respecting Tribal Data Sovereignty.
• Removes the barrier to submitting compliant evidence: It will be difficult for small, local and non-traditional providers to build their own compliant tests since no off-the-shelf-tests meet NTIA’s acceptable evidence criteria.

4. Be Inclusive: 

Making your challenge process inclusive helps you empower small, local & non-traditional providers while avoiding unnecessary lawsuits and process delays. At a minimum, your Performance Test & Survey and Challenge Process Coordinator should:

• Meet Section 508 Accessibility Guidelines.
• Include SMS & voice options for those without internet.
• Work with Tribal govs to ensure participation while respecting Tribal Data Sovereignty.

5. Go on the Offensive / Outbound FTW

Since you only get one chance to validate your broadband audit and any provider rebuttals, you might as well take the opportunity to verify service quality in areas you know or suspect to be historically underserved.

Consider reaching out to occupants of BSLs in historically underserved areas if challengers have submitted large quantities of challenges. The “trust but verify” principle would mean using the set of Broadband Serviceable Locations contained in of challenges from all challengers who submitted more than 1,000 disputed challenges as your outbound mailing list. 

This effort will take a little printing budget and team effort but can pay of in dividends for your office since:

• Build it and they will come is not an effective strategy – distribution is required to get results.
• As a gov entity you can send out mailers at  rate and special print costs, for sub 10 cents apiece.
• For a relatively small amount of money you can dramatically increase the resolution of your understanding of not only service availability but also affordability, quality of experience, and more. 

6. Off the shelf solutions won’t work: you’ll need to buy or build

Read NTIA’s challenge process guidance and Model Challenge Process carefully. Acceptable evidence has a high bar. You are encouraged to develop your office’s evidentiary standard on the minimums supplied by NTIA. You should:

• Use more than one speed test. Two is better than one, but Performance Test & Survey includes three complete tests. 
• While no single test is perfect, running multiple tests together in one shot combined with survey data can help you to build up your understanding in an empirically defensible way.
• However, taking two speed tests can be a challenge for BSL occupants, so the experience should be designed to fire the tests in serial (as running tests in parallel can produce false positives). 
• Geolocate results to the survey / test submitter. 
• But you’ll need to comply with Tribal Data Sovereignty for results gathered from Tribal stakeholders. 
• Include timestamps on tests taken in order to ensure provider rebuttals aren’t only run during extreme off-peak hours.

For best results, remind test takers to take tests at specific on-peak hours, and get a time series (more than one test at different times) from BSL occupants in hotly contested areas.
 

7. Go beyond up & down – set your bar high

In the theme of viewing NTIA’s guidance as minimum requirements, it’s Important to remember they do set out a basic latency threshold of not-to-exceed 100ms round trip.

While BEAD uses upload and download, you are encouraged to incorporate other attributes into your evaluation process. You might as well gather other dimensions as well

Ideally your tests go deeper than general latency.

Read the BITAG Latency Explained report for a deeper understanding of why buffer-bloat  (“working latency”) should be measured in your office’s performance diagnostics. 

8. View Permissible Challengers and Anchors as Partners

While Community Anchor Institutions (CAIs) were ignored in FCC’s Fabric, NTIA’s guidance makes abundantly clear that CAIs are very much at the table in BEAD. Your office can go beyond ensuring your CAIs have gigabit symmetrical service and empower them to help you engage BSL occupants in generating meaningful performance test & survey data.

By supplying your Performance Test & Survey as part of your Challenge Process Coordinator, you empower your Permissible Challengers including counties, Tribal governments, regional groups, providers, and non-profits to help your team to make objective, data-driven decisions.

If you’re required to identify CAIs, you might as well provide them with your Performance Test & Survey tools and treat them as partners in collecting meaningful source-of-truth data. Performance Test & Survey was developed to follow the Broadband Mapping Coalition’s recommended methodology for broadband performance testing. States should consult the BMC recommended methodology if implementing their own performance testing systems.

Please remember you’re required to identify CAIs, including CAIs on Tribal Lands as well. When working with Tribes, please ensure your office adheres to Tribal Data Sovereignty. For help engaging Tribes in a compliant way, please ping Tribal Ready.

In addition to CAIs, Permissible Challengers including local gov, Tribal gov, non-profits and providers can be major allies in driving adoption of your data-driven efforts – if you provide them the bridge between your office and their efforts.

What's next?

NTIA’s provided crystal clarity: you can and should incorporate performance data into your Challenge Process. It’s up to you how you get there. Whether you build your own technology from scratch or license the Ready State Grantor Platform, you can, should, and must figure out how to incorporate source-of-truth evidence in your challenge process in order to make the most of your BEAD resources. It’s the best way for you to get to the bottom of where service really is needed.