A polished AI demo can hide a surprising amount of technical debt. Good diligence asks not only whether the feature works, but why it works, what it depends on, and what happens when usage grows.
Where is the actual product advantage?
Separate model capability from product capability. If every competitor can call the same model, the advantage may live in proprietary data, workflow integration, evaluation, distribution or domain operations.
What are the failure modes?
Ask how the company measures hallucination, extraction errors, unsafe actions and edge cases. Review the test set. Look for segmented metrics, not a single aggregate accuracy number.
Does the data story hold up?
Where do training, retrieval and user data come from? Are the rights clear? Is data cleaned and versioned, and is feedback captured in a form that improves the system?
What does production cost?
Model calls, retrieval, storage, human review and latency all have economics. A workflow that dazzles in a demo can look very different at thousands of users or millions of documents.
Can the organisation operate it?
Reliability is an organisational capability. Look for ownership of evaluation, monitoring, model changes, incident response and product trade-offs, not just a research prototype.
Diligence should ask what the product depends on, not only whether the demo works.