A vendor claims the library catalog API is 'fully tested,' but cannot show which security suites or paths were exercised. What should the city demand?

Coverage analysis quantifies whether security test suites actually exercised meaningful controls, interfaces, and cases. When a vendor claims a library catalog API is ‘fully tested’ but cannot show what was reached, the city should demand coverage evidence—mapped controls, API endpoints, authz paths, and negative cases—before accepting assurance. Marketing completeness claims without measurable coverage are insufficient for municipal procurement and risk acceptance; coverage analysis turns vague warranties into verifiable assessment scope.

Accepting the claim because the vendor’s logo is well known substitutes brand reputation for evidence and is not an assessment technique. Deleting all test evidence to reduce storage costs destroys the audit trail needed to prove what was and was not tested. Replacing testing with an informal verbal warranty from sales provides neither technical rigor nor contractual defensibility if the catalog API fails under attack.

Exam caveat: CISSP assessment design stresses defining scope and measuring whether tests cover that scope. ‘Fully tested’ without coverage metrics is a distractor; prefer answers that demand measurable coverage over trust-based acceptance.

Operational check: Require the vendor to deliver a coverage matrix tying security suites to OpenAPI routes, auth schemes, and control objectives; spot-check a sample of critical endpoints yourself; retain coverage artifacts with the city’s vendor assurance file for the library system.