An auditor accepts a permitting AI vendor's marketing slide deck as the sole evidence that model validation was performed. What is the primary concern?

The primary concern is that sufficiency and reliability of evidence for AI validation claims are inadequate when the sole artifact is a permitting vendor’s marketing slide deck. Municipal auditors need reproducible methods, datasets, metrics, acceptance criteria, and results that can be inspected or reperformed. Marketing slides typically summarize capability narratives without the technical depth required to conclude that model validation actually occurred and remains relevant to the city’s permitting use case.

Claiming that completeness of permitted application logs reconciled to fee collection records is the issue fails because log completeness and fee reconciliation concern transaction processing controls, not whether a vendor’s marketing slide is sufficient evidence that AI model validation occurred.

Asserting that independence of the internal validation team documented in governance charters is the concern fails because governance documentation about validator independence does not cure the inadequacy of relying solely on a vendor marketing slide as validation evidence.

Treating retention of permit decision audit trails enabling future reperformance procedures as the answer fails because audit trails support reperformance but do not establish that a marketing deck alone is sufficient, reliable evidence that validation was performed.

Exam caveat: some vendors embed useful summaries in decks; those slides may point to better evidence but still cannot stand alone as validation proof. Operational check: request validation reports, test datasets and metrics, model cards, change logs, and independent or city-side reperformance results; document any remaining access limitations in the permitting AI workpapers and report.