New research: responsible AI procurement as a relationship, not compliance
My new co-authored research paper is officially out!
Hi everyone,
Today I am excited to share a new research paper I co-authored with researchers at Carnegie Mellon University (CMU). I am proud to say that it was accepted for publication at FAccT, the top journal and conference in AI ethics!
The paper is about responsible AI procurement in local government. Our main point is that documents used for self-disclosure in procurement processes should be seen as tools for building trust. Not compliance, not marketing, and not quality assessment.
This paper is a collaboration with the GovAI Coalition. They are a coalition of hundreds of US local governments and their mission is to make public sector AI serve the public good. In particular, they work to help local governments to procure AI responsibly. Our research empirically studied the process GovAI uses to gather information from AI vendors for procurement.
In this blog post, I will summarize our key findings and recommendations. You can read the full paper here.
In a nutshell
GovAI uses a form called AI FactSheet for soliciting information from AI vendors about AI products. It includes questions about the AI products both in general and about responsibility dimensions. The goal is for local governments that are GovAI members to use this information in their AI procurement processes.
Forms like the AI FactSheet are extremely important because they set standards for transparency and decision making in procuring AI.
Our goal was to understand what makes documents of this type succeed or fail, using the FactSheet as our case study. To do this, we analyzed completed FactSheets and interviewed vendors and public sector practitioners.
We found that:
Vendors are not providing useful information for decision making
There are three competing motivations for completing FactSheets: Quality assessment, marketing, and relationship building
Relationship building is the most viable purpose
My top recommendation:
See self-disclosure documentation as conversation staters. Not compliance, not marketing, and not quality assessment. Examples for what this means below.
Finding 1:
Vendors are not providing useful information for decision making
We learned this by evaluating completed FactSheets using the CDT’s Framework for Assessing Transparency in the Public Sector.
We found that vendors do provide some relevant information, but they often omit basic technical details about system design, training data, and performance. As a result, disclosures often gesture toward transparency but do not enable decision making.
For example, consider the following FactSheet response. You can’t learn much from it:
“Overall, the chatbot demonstrates high accuracy based on internal validation tests and from current customer feedback.”
You can see the evolution results in the table below. We evaluated on four categories using a score ranging from 1-3:
1= The vendor provides no information or an unhelpfully broad stated purpose.
2=The vendor provides some narrowing information that still leaves a wide range of possible functionalities.
3=The vendor provides information about specific tasks the system performs and outlines inappropriate/untested use cases.
Finding 2:
FactSheets serve three competing purposes
a. Quality assessment
The stated goal of the FactSheet is to help in procurement decision making. GovAI describes its purpose as:
"evidence that the system fits our needs and is safe to implement in our jurisdictions"
Agencies we interviewed agreed with this sentiment. Vendors, too, expressed that they support more rigorous assessment as it would help them differentiate products and avoid poor-fit deployments.
b. Marketing and showcasing products
Many of the vendors we interviewed described the FactSheet as a space for communicating benefits, differentiation, and practical relevance, rather than serving primarily as a vehicle for detailed technical disclosure.
For example, vendors described their motivations for completing the FactSheet as
A way to gain “any exposure, especially for small companies”
Serving a desire to “maximize the eyeballs” that see their products.
c. Trust and relationship building
Both vendors and agencies described a third, less explicit but widely shared goal: FactSheets are a foundation for building trusted, long-term working relationships between governments and AI vendors.
For example, one agency employee described how the FactSheet was a part of their on-going relationship with the vendor:
“Because [the AI vendor] felt safe in the relationship with us, they knew it wasn’t a ‘gotcha.’ It was ‘do the best you can.’ So they did, and their answer wasn’t perfect”
In this framing, the FactSheet functioned not as a compliance artifact or evaluative gatekeeper, but as an entry point into collaborative problem-solving.
Finding 3:
The way forward is to use disclosure as a component of relationship building
We found that the first two purposes, quality assessment and marketing, not only conflict but also encounter structural barriers. For example, vendors worry about sharing information which can become public due to laws requiring transparency from the public agencies.
The most viable use for self-disclosure documents like the FactSheet is as tools for relationship building.
From this perspective, the FactSheet is treated as a starting point that invites iterative clarification, follow-up questions, and continued exchange, rather than as a final or self-sufficient account of the AI system. The FactSheet launces an ongoing interaction that establishes baseline trust, signals good faith, and creates a shared starting point for future dialogue. The vendor will disclose the information the agency needs as the relationship grows, and the agency will support the vendor in disclosing this information safely and allow them to be imperfect.
Recommendations
My top recommendation is to use self-disclosure documents as conversation starters rather than tools for quality assessment or marketing.
This means:
Public agencies
Don’t expect vendors to reveal all information right off the bat. It can be overwhelming, especially for small business, and it can jeopardize their IP.
Evaluate self-disclosure documents on whether they are informative enough to be conversation starters, not on whether the enable quality assessment of the product
Vendors
Don’t use self-disclosure document as a demo or product brochure. Instead, use them to show that you would be a good collaborator and be willing to provide substantive information as the relationship grows.
You can do this by providing answers that are informative enough to show you can engage in a substantive conversation later on. This is what could get you meetings and demos down the road.
Dessert
An AI-generated take on this post!
