Vendor AI Evaluation & Governance Scorecard

01 Reviewer & vendor metadata

Engagement baseline, vendor identification, and review workflow status.

Vendor organization

Platform / solution

Lead reviewer

Reviewer email

Evaluation date

Review lifecycle status

Target business unit & workflow

Executive context & problem statement Business justification and intended user persona.

02 AI taxonomy & modality

Active AI modalities and the criticality tier that weights the whole scorecard.

Criticality & blast-radius tier

Active modalities & techniques Every capability the vendor solution incorporates.

Text / large language modelsSummarization, prompt completion, chat Code generation & executionCode synthesis, sandboxed script running Vision & image analysisDiagnostic imaging, visual QA, photo scanning Speech / audioAmbient scribe, STT, voice command Document & form OCRPDF layout parsing, table extraction, claims Classical tabular MLXGBoost, random forests, risk prediction Autonomous agentsMulti-step planning, tool chaining, actions Vector embeddings / RAGDense semantic indexing and retrieval

Architecture & data-flow notes

03 Foundation models & economics

Model family, hosting boundary, data-retention terms, token economics, and spend controls.

Foundation model family

Hosting & deployment

Region & data residency

Retention & training exclusion

Billing & cost structure

Monthly hard spend cap (USD)

Budget alert threshold 80% of cap

Guaranteed throughput

Failover & circuit breaker

04 Agentic capability & autonomy

Autonomy level, human-in-the-loop gates, deterministic schemas, identity, and auditability.

Agentic autonomy level

Deterministic output controls

Execution identity (IAM)

Action blast radius

Emergency kill switch

Prompt & completion audit logging

Tool-invocation traceability

05 Model Context Protocol interfaces

Client/server topology, tool sandboxing, resource boundaries, and transport security.

MCP implementation role

Transport protocol

Tool sandboxing

Resource security boundary

Network isolation & egress filtering

Egress allowlist onlyBlocks unapproved outbound internet Private mesh / VPC onlyNo public exposure; mTLS mesh Tool schema validationStrict JSON schema on tool arguments

Exposed tools & resource URIs

06 Custom ML & provenance

Training-data lineage, de-identification, feature platform, drift monitoring, and accreditations.

Custom model scope

Training-data provenance

PHI / PII de-identification

Data & feature platform

Retraining cadence & drift monitoring

Regulatory & compliance accreditations

HIPAA BAA executedBinding business associate agreement SOC 2 Type IIClean report within 12 months FDA 510(k) / SaMDClinical decision support cleared ISO 42001 / 27001AI management system certified

07 Verdict & governance sign-off

Governance decision, mitigating stipulations, and the recertification date.

Final governance verdict

Recertification cadence

Mandatory stipulations & compensating controls

Lead evaluator signature

Sign-off timestamp