AI Auditing Tools and Techniques

AAIA · 42 questions

• A city audit plan treats a new permitting GenAI as a standard web app and skips AI-specific risk assessment in scoping. What should the auditor change?
• A county internal audit team is designing work on an early public-health AI pilot and must choose between assurance and advisory objectives. What planning distinction matters most?
• A transit AI audit plan is drafted without using model inventory completeness or system criticality as planning inputs. What should the auditor require?
• A housing authority AI audit timeline is set without assessing the team's need for statistical or model literacy. What planning gap should the auditor cite?
• A 311 chatbot audit design covers technical uptime but omits equity and resident-impact workstreams. What should the auditor add?
• A court AI audit plan lists fieldwork tasks but maps no criteria to AI policies, standards, or regulations. What should the auditor correct?
• A water utility SCADA machine-learning audit scope excludes the vendor-managed model pipeline with no documented justification. What should the auditor challenge?
• A school district AI audit calendar spreads effort evenly and never prioritizes high-harm student-facing systems first. What planning approach should the auditor recommend?
• A municipal chief audit executive asks whether integrating AI into the audit process itself requires changing how engagements are planned. What is the best response?
• An auditor samples only 25 permitting chatbot chats at random and ignores stratified high-risk permit types. What sampling improvement should the auditor make?
• A public-health model audit measures overall accuracy but does not sample across demographic strata. What testing gap should the auditor cite?
• A transit AI control test relies only on design walkthroughs and collects no operating effectiveness sample. What distinction should the auditor apply?
• A housing AI audit samples human overrides but never samples automated approvals that never see a human. What population flaw should the auditor correct?
• A 311 GenAI test uses only auditor-written prompts that do not reflect real resident phrasing. What testing weakness should the auditor address?
• A court AI audit sample spans a period when the model changed mid-window, muddying conclusions. What sampling consideration should the auditor apply?
• A water anomaly AI audit uses a sample too small to conclude on rare but critical false negatives. What sampling issue should the auditor raise?
• A school district AI audit relies solely on vendor SOC report sampling assertions and performs no local tests. What approach should the auditor take?
• A parks department vision AI audit reuses last year's sample design without reassessing how park-camera usage and incident types changed. What should the auditor do?
• An auditor accepts a permitting AI vendor's marketing slide deck as the sole evidence that model validation was performed. What is the primary concern?
• A public-health AI audit needs to reperform GenAI responses about clinic referrals. Which evidence type best supports that reperformance?
• A transit AI engagement collects staff interviews about monitoring but obtains no system-generated monitoring exports. What should the auditor strengthen?
• A housing authority AI audit cannot obtain training-data samples because the vendor refuses access. What should the auditor pursue next?
• A 311 chatbot audit captures chat UI screenshots but omits model version, prompt template, and retrieval metadata. What completeness issue should be raised?
• A court AI auditor observes live use of a transcript assistant but fails to secure chain-of-custody for exported transcripts used as evidence. What should be corrected?
• A water utility ML audit uses only synthetic test data, and management challenges whether results reflect production risk. How should the auditor respond?
• A municipal auditor uses an AI assistant to draft findings on a permitting AI system and must support the workpapers. What evidence discipline is required?
• An auditor's analytics on permitting AI decisions shows null feature rates spiked just before a poorly performing release. What does this illustrate?
• A public-health AI audit dashboard compares prediction distributions week-over-week to look for drift. What audit use does this support?
• Transit AI sample analytics show systematically longer predicted wait times for one corridor relative to peers with similar demand. What should the auditor conclude about the technique's value?
• A housing AI audit's computer-assisted audit techniques fail because decision logs lack join keys to case files. What prerequisite is missing?
• A 311 GenAI audit mines ticket outcomes against bot answers to measure whether responses were helpful. What analytics design does this exemplify?
• A court AI analytics project draws accuracy conclusions without documented data quality rules on the extract. What should the auditor require first?
• A water utility AI audit deploys continuous auditing scripts that watch model confidence scores for anomalies. What technique is being applied?
• A school-district AI audit open-data extract used for analytics still contains suppressed small-cell values that were supposed to protect student privacy. What should the auditor assess?
• A permitting AI audit report lists technical jargon for council readers but omits residual risk ratings. What reporting improvement is needed?
• A public-health AI engagement issues findings without actionable management action plans. What quality gap exists in the audit output?
• A transit AI report omits disclosure that vendor model internals were inaccessible during testing. What reporting element is missing?
• A housing AI audit rates the system 'satisfactory' even though fairness controls were never tested, and the report logic is weak. What should be challenged?
• A 311 AI advisory memo uses assurance-opinion wording even though the work was consulting-only. What distinction should the auditor enforce?
• A court AI audit follow-up report states monitoring gaps were closed but does not retest whether model monitoring controls now operate. What should be improved?
• A water AI audit dashboard for executives mixes verified KRIs with unaudited vendor marketing metrics. What integrity issue should be raised?
• A municipal chief audit executive uses GenAI to draft portions of an AI audit report and must follow methodology on transparency. What practice is required?