AI Vendor Due Diligence: Questions Every City Should Ask

A vendor demo answers the questions the vendor expects to be asked. You see a clean draft appear in seconds, a search that finds the right resolution on the first try, and a dashboard with tidy charts. What you rarely see is where your records go after they are uploaded, what the tool does when it cannot find a source, or how you would get your data back if the contract ended next year. When a tool touches public records, those are the questions that matter. A city has to ask them deliberately, because a demo will not bring them up.

This guide is a question list for evaluating AI products, organized into seven areas:

1. Data use and model training
2. Isolation and data location
3. Citations and accuracy
4. Audit logs and records obligations
5. Security and incident response
6. Accessibility
7. Exit terms and contract protections

For each area you will find the questions to ask, what a good answer looks like, and the answers that should slow you down. The list applies to stand-alone AI products. It applies equally to AI features a current vendor adds to software you already license. That second case is easy to skip because the vendor relationship already exists, but a new AI feature can change how your data is processed and which companies handle it.

None of this replaces your procurement rules, your city attorney or your IT staff. Contract law, public records law, breach notification requirements and accessibility obligations differ by state, and sometimes by charter or local ordinance. Use these questions to get clear written answers, then have counsel and IT review those answers against your local requirements.

How to run the review

The way you ask these questions affects the quality of the answers as much as the questions themselves.

Ask in writing and get answers in writing

Send your questions as a written questionnaire, either before the demo or immediately after it. Verbal assurances in a demo are not commitments. A sales representative may sincerely believe a product works a certain way and still be wrong, and products change between releases. Ask the vendor to answer each question in writing and to point to where the commitment appears: the contract, a data processing agreement, a security addendum or a service level agreement. If an answer matters to you and it does not appear in a document the city will sign, treat it as an intention, not a commitment.

Bring the right people

Each area of the question list needs a reader who knows what a weak answer looks like:

• Clerk or records manager: retention, public records requests, citations and the accuracy of anything that ends up in the record.
• IT or security lead: architecture, access controls, identity management and incident response.
• City attorney: data ownership, liability, indemnification, termination and state law questions.
• The department that will use the tool: real workflows and realistic test material.
• ADA coordinator or accessibility lead: conformance documentation and hands-on testing.

In a small city one person may cover several of these roles, or you may rely on county IT, a contracted managed service provider or outside counsel. That works fine. Every area simply needs someone who will read it critically.

Test with your own material

Ask for a pilot or trial that uses your own public records: published agendas, adopted minutes, staff reports and resolutions. Decide in advance what you will test and how you will judge it. Keep confidential or exempt material out of any trial until a signed agreement covers how that material will be handled. Trial and free tier terms are sometimes different from paid contract terms, so read them before you upload anything.

Learn what a good answer sounds like

Across every area, a good answer is specific, documented and candid about limits. A weak answer is general ("we take security very seriously"), points you to a marketing page, or declines to answer without explanation. Some vendors reasonably restrict detailed security documents to customers or share them only under a nondisclosure agreement. That is normal practice. Even so, the vendor should be able to tell you what documentation exists and how you can review it before you sign.

Data use and model training

Start here. Public trust and legal risk meet in this area, and it is also where vendor language tends to be most slippery. Many AI products are built on models from a small number of third-party model providers, so your records may pass through more than one company. Your questions need to cover all of them.

Questions to ask

• Do you use our data, including uploaded documents, prompts and generated outputs, to train or improve any model, either your own or a third party's?
• Is that the default behavior, or a setting someone has to change? Who in our organization can change it?
• Which third-party model providers process our data, and under what terms?
• Do you or your providers retain prompts and outputs? For how long, and for what purpose (for example, abuse monitoring or troubleshooting)?
• Can your employees view our content? Under what circumstances, who approves that access, and is it logged?
• Do you use aggregated or "de-identified" data from our account for any purpose? What exactly is removed?
• Will you notify us before changing any of these terms, and can we decline the change?

What a good answer looks like

A clear statement that customer data is not used to train models, written into the contract or data processing agreement rather than only a privacy policy the vendor can edit. A named list of subprocessors (the companies that handle data on the vendor's behalf) with a commitment to notify you before that list changes. Specific retention periods for prompts and outputs, with the reason for each. Employee access limited to situations the city initiates, such as a support request, with that access logged and visible to you.

Answers that should slow you down

• "We don't train on your data," followed by an exception for "improving our services" that is not defined.
• Terms that can change when the vendor updates a web page, with no notice to you.
• A vendor that cannot or will not name its model provider.
• Different data handling for trial accounts than for paid accounts, with no mention of which terms apply to your pilot.

A note on "de-identified" data

City records are hard to anonymize. A staff report on a variance request is identifiable by its address even with every name removed. A code enforcement memo, a personnel matter or a public comment can be identifiable from context alone. If a vendor says it uses de-identified data, ask what fields it removes and how it decides a document is no longer identifiable. Then have your records manager and attorney decide whether that approach is acceptable under your state's laws.

Isolation and where your data lives

Most modern software is multi-tenant, which means many customers share the same underlying infrastructure. That is normal, and multi-tenancy alone is not a problem. What matters is how well each customer's data is kept separate, and whether that separation holds at every layer of the system.

AI tools raise a specific concern here. Products that ground their answers in your documents usually build a search index of your content, often called a vector index or embeddings. If that index is shared across customers and filtered at the moment a question is asked, a filtering error could surface another city's document in your results, or yours in theirs. Separation has to apply to files, databases, search indexes, logs, caches and backups, not just the main database.

Questions to ask

• How is our data separated from other customers' data at each layer: file storage, database, search index, logs, caches and backups?
• Is that separation logical (for example, a separate database schema, a separate database, or filtering by a tenant identifier) or physical (dedicated infrastructure)?
• How do you test that one customer cannot retrieve another customer's content, including through AI search and generated answers?
• Where is our data stored and processed geographically? Does any processing, including by your model provider, happen outside the United States?
• Do backups and disaster recovery copies follow the same separation and location rules?
• Can we restrict which staff and departments see which records within our own account? Does AI search respect those permissions?

What a good answer looks like

The vendor walks through each layer and explains the design in plain terms. One example is each customer's data held in its own database schema, with every search scoped to that customer before the query runs. The vendor describes how it tests separation and how often. It commits to storage and processing locations in writing, and that commitment covers its model provider.

Internal permissions matter too

Cross-customer separation is only half the question. Inside your own account, you likely hold records that not every employee should see: police reports, personnel files, closed session material, attorney communications. A tool that indexes everything and lets anyone with a login search it can expose information internally. Ask whether the tool honors your existing access rules or creates a new, flatter permission model you will have to manage separately.

Answers that should slow you down

• "Our cloud provider handles that." Cloud providers secure the infrastructure. The vendor is responsible for how its own application separates customers.
• "Your data stays in the U.S.," with no mention of where the model provider processes prompts.
• No description of how separation is tested.

Citations and accuracy

Language models can produce fluent, confident text that is wrong. In casual use that is an annoyance. In a staff report, a resolution or a summary of past council action, it can mislead decision makers and end up in the public record. The practical question is not whether a tool is ever wrong. Every tool will be. The question is whether staff can quickly check what it says.

Questions to ask

• When the tool states a fact about our city, does it show where that fact came from? At what level of detail: document, page, meeting, specific passage?
• Can a reviewer open the original record directly from the citation?
• What does the tool do when it cannot find a source? Does it say so, or does it produce an answer anyway?
• How does it handle superseded material, such as an amended ordinance, a revised budget or minutes that were corrected before approval?
• Does it draw only on our records, or does it mix in general knowledge from the model? Can staff tell which is which?
• How do you measure accuracy, and can you explain your method, not just a result?
• Will you notify us when the underlying model changes, since a model change can change output quality?

What a good answer looks like

Citations specific enough that a reviewer can verify a statement in under a minute, linked to the source document. A tool that says plainly when it cannot find support for an answer. A vendor that volunteers examples of where the product struggles, such as scanned documents with poor text quality, handwritten notes or very old records. A commitment to notify you before significant model changes.

Test it yourself

This is the most useful part of any pilot. Build a test set of 10 to 20 questions whose answers you already know from your own records. Include a mix:

• Simple facts: when the council adopted a particular resolution, or who made a motion at a specific meeting.
• Amended items: an ordinance that was later changed, to see whether the tool cites the current version.
• Questions with no answer: something your records do not contain, to see whether the tool admits it or invents something.
• Near duplicates: two meetings with similar agendas, or two projects with similar names, to see whether the tool confuses them.

Run each question, open every citation and record whether the source actually supports the statement. A citation that points to the right document but the wrong page still costs a reviewer time. Write the results down so you can compare vendors on the same test.

Answers that should slow you down

• Accuracy percentages offered with no explanation of how they were measured or on what material.
• Citations that name a document but not where in it the information appears.
• Any claim that the product does not make mistakes.

Good citations reduce the effort of checking. They do not remove the need for it. Ask how the product builds staff review into the workflow, so that nothing generated goes into a packet or onto the website without a person approving it.

Audit logs and records obligations

Two questions sit under this heading. First, can you reconstruct who did what in the system, and when? Second, how does the tool fit with your obligations under public records and retention law?

Questions to ask

• Which actions are logged? Look for logins, uploads, searches, prompts, generated outputs, edits, exports, permission changes and any access by vendor staff.
• Who in our organization can view the logs? Can we export them, and in what format?
• How long are logs kept? Can we set retention to match our records retention schedule?
• Are prompts and outputs stored? Can we retrieve them by user, date range or topic if a public records request or a litigation hold requires it?
• Can we place a legal hold that prevents deletion of specific content?
• Are actions taken by vendor support staff in our account logged and visible to us?

What a good answer looks like

Broad logging that covers both user actions and vendor access, exportable by your own staff without a support ticket. Retention you can configure. A way to preserve and retrieve content when a hold or request requires it. The vendor should be able to show you a sample log during the evaluation.

What the vendor cannot answer for you

Whether prompts, outputs and logs are public records, and how long they must be kept, depends on your state's law and on what the content is. A vendor cannot settle that question. What the vendor can tell you is whether the material exists, where it is stored and whether you can retrieve it. Bring those answers to your records manager and city attorney, and decide on handling before the tool goes into daily use, not after the first request arrives.

Answers that should slow you down

• Short, fixed log retention that you cannot change.
• Logs available only by filing a request with the vendor.
• No way to retrieve a specific user's history for a defined period.
• Vendor staff access that is not logged, or logged only internally.

Security and incident response

Security questions for AI products overlap heavily with those for any cloud software, and your IT lead likely has a standard set. Add the AI-specific questions below, and spend real time on incident response, which is often the thinnest part of a vendor's answers.

Questions about security practices

• What independent audits or certifications do you hold? Examples include a SOC 2 Type II report, ISO/IEC 27001 certification, FedRAMP authorization, or a state-focused program such as GovRAMP (formerly StateRAMP). Can we review the report?
• Does the scope of that audit include the AI product we are buying, or only other parts of your business?
• Is data encrypted in transit and at rest?
• Do you support single sign-on with our identity provider, and can we require multi-factor authentication?
• Do you commission independent penetration tests? How often, and will you share a summary of results?
• How do you manage vulnerabilities and patching, including in third-party components?
• What security requirements do you place on your subprocessors and model providers?
• If we will use the tool with law enforcement or health information, can you meet the applicable requirements, such as the FBI's CJIS Security Policy or HIPAA?

Scope deserves a careful read. An audit report covers specific systems over a specific period. A vendor may hold a strong report for a mature product while a newer AI feature sits outside it. Ask directly.

AI-specific security questions

• Prompt injection. A document can contain hidden or embedded instructions that try to steer an AI tool. Cities routinely ingest outside content, including public comments, applicant submissions and correspondence. Ask how the product handles instructions found inside documents.
• What the tool can do on its own. A tool that only drafts text for staff review carries less risk than one that can send email, publish content or change records. Ask exactly which actions the tool can take without a person approving them.

Questions about incident response

• What counts as an incident under the contract? The definition should cover unauthorized access, cross-customer exposure, and defects that produce wrong outputs across customers.
• How quickly will you notify us, in what form and to whom? Put a specific notification window in the contract.
• Do you have a written incident response plan? When was it last tested?
• Will you share a root cause analysis after an incident?
• How will you support the city's own notification obligations? State breach notification laws vary, and the city may have duties to residents that the vendor's timeline must allow for.
• Who bears the costs of investigation and notification when an incident originates with the vendor?
• Can we have a named security contact, not only a general support queue?

What a good answer looks like

A current independent audit report whose scope includes the product, shared under appropriate terms. Single sign-on and enforced multi-factor authentication. A written, tested incident plan. A defined notification window in the contract, a named contact and a commitment to share root cause findings. Clear limits on what the tool can do without human approval.

Accessibility

Staff with disabilities will use these tools, and some outputs, such as agenda items, staff reports and summaries, may be published to residents. Accessibility has to be part of the evaluation, not a box checked at the end.

In 2024 the U.S. Department of Justice issued a rule under Title II of the Americans with Disabilities Act that adopts WCAG 2.1 Level AA as the technical standard for web content and mobile apps provided by state and local governments, with compliance dates that vary by population. Confirm current requirements, deadlines and how they apply to vendor-provided tools with your city attorney and ADA coordinator.

Questions to ask

• Do you have a current Accessibility Conformance Report for the version we would buy? Many are prepared using the VPAT template. Which standard does it address, WCAG 2.1 AA or later?
• Who prepared the report, and when? Was testing done by an independent party?
• Does the report list known gaps, and is there a timeline for fixing them?
• Has the product been tested with screen readers, keyboard-only navigation and screen magnification?
• Are generated documents accessible, with proper headings, table structure and alternative text where needed?
• Will you commit in the contract to maintaining conformance and fixing reported accessibility issues within a defined period?

What a good answer looks like

A current report that is candid about gaps, a remediation plan, and a contractual commitment. The best evidence is hands-on: ask a staff member who uses assistive technology, or your ADA coordinator, to try the product during the pilot and report back.

Answers that should slow you down

• "Fully compliant," with no report to support it.
• A conformance report that is several years old or covers a different version.
• Exported documents that lose structure, such as PDFs without tags.

Exit terms and contract protections

The best time to plan the end of a vendor relationship is before it begins. Once your records, workflows and staff habits depend on a tool, your negotiating position weakens.

Questions to ask

• Does the contract state that the city owns its data and the outputs generated from it?
• At termination, can we export everything, including original documents, metadata, generated outputs and logs, in standard, open formats? How long do we have to do it, and is there any additional cost?
• Will you delete our data afterward, including backups, search indexes and logs, and certify deletion in writing? On what timeline?
• Will you provide transition assistance if we move to another system?
• How does renewal work? Is there automatic renewal, and what notice is required to end it?
• Can your terms of service or privacy policy change during the contract term without a signed amendment?
• Do we have termination rights if you change model providers, are acquired, or materially change how you handle our data?
• What do you accept in liability and indemnification for data breaches and for claims arising from generated content?
• Do you carry cyber liability insurance, and will you provide a certificate?

Sequence matters on deletion. Your retention schedule may require you to keep certain records for years after a contract ends. Export and verify what you need first, then request deletion and certification.

What a good answer looks like

Clear city ownership of data and outputs. Full export in open formats with enough time to complete it. Written deletion certification that covers derived data like search indexes. No unilateral changes to key terms during the contract. Defined rights to exit if the vendor changes in ways that affect your data.

Putting the answers side by side

Once responses come in, a simple scorecard helps reviewers compare vendors on the same terms. Adapt this to your own priorities.

Next steps

You do not need to finish every question before you talk to a vendor. You do need a plan for getting written answers and reading them carefully.

1. Turn this list into a questionnaire. Remove what does not apply, add your state and local requirements, and have your city attorney and IT lead review it.
2. Assign a reader to each area. Records, IT, legal, the using department and accessibility should each own their section of the responses.
3. Build your citation test set now. Write 10 to 20 questions with known answers from your published records so every pilot uses the same test.
4. Map every answer to a document. For each commitment that matters, note where it appears in the contract or an addendum. Anything that is not written down goes back to the vendor.
5. Repeat the review at renewal and whenever a vendor adds AI features. A tool you approved last year may handle data differently after an update.

If Govera is on your list, hold it to the same questions. Ask to see citations to the source document, page and meeting on your own records, the audit log of user actions, and how each city's data is isolated in its own database schema.