Seven questions to ask before deploying AI visibility tooling

Visibility into enterprise AI activity is now a recognized requirement. The market response has been fast, and evaluation criteria have not kept up. These seven questions separate tools that produce governance-grade visibility from tools that produce dashboards.

1. Which telemetry sources does it draw on?

AI activity is visible from several vantage points: identity providers, endpoint agents, browsers, enterprise AI platforms, network and secure web gateway logs, and SaaS audit trails. No single source is complete. Ask which sources are supported today, which require an agent or an integration, and how conflicts between sources are resolved.

2. Does it attribute activity to identities, or only to traffic?

Traffic-level visibility tells you that a model endpoint was called. Identity attribution tells you who called it, on which device, and in which part of the business. Governance decisions require the latter. Ask how attribution is achieved and what its coverage is in an environment like yours.

3. Does it sit inline?

Some approaches place a proxy or gateway in the traffic path. That can provide enforcement but adds a dependency, a point of failure, and often a coverage gap for devices and paths that bypass it. An alternative is to observe through existing telemetry and enforce through the security controls the enterprise already operates. Understand which model a tool uses and what that means for rollout and resilience.

4. How does it define “sanctioned”?

Every enterprise has AI uses that are approved, tolerated, under review, and prohibited. Ask how those states are represented, who can change them, whether exceptions are recorded with approvers, and whether the history is preserved.

5. What evidence does it produce, and in what form?

A screenshot of a dashboard is not evidence. Ask for time-stamped, attributable records that can be exported and presented against a control or a framework requirement. Ask how long they are retained and who can access them.

6. How does it connect to your control and compliance program?

Visibility that lives only in a security console rarely reaches the people who manage frameworks and audits. Ask whether findings and outcomes can be tied to controls in a governance system of record, and whether that connection is native or requires manual export.

7. What does it deliberately not collect?

AI visibility tooling can, in principle, capture prompt content and sensitive data. Ask what the tool collects by default, what it excludes, and how data minimization is enforced. A vendor selling governance technology should be able to demonstrate that it governs its own data practices.

These questions are the ones QAISG designed QWatch to answer well. They are also the ones we would ask of anyone else.