Playbook: Build a Vendor Performance Scorecard

Your procurement team works with forty vendors. Some are great — they deliver on time, invoice correctly, and respond to issues within hours. Some are not. The problem is that nobody has a single view of vendor performance. Delivery data is in the WMS. Invoice data is in the ERP. Quality complaints are in the ticketing system. The procurement manager's assessment lives in their head.

When it comes time to renegotiate a contract, renew an engagement, or choose between two vendors for a new category, the team either spends a week pulling data from three systems or makes the call on gut feel.

This playbook walks you through building a scorecard app — a sandcastle that consolidates vendor metrics into a single ranked view, backed by a subagent that keeps the data current.

What you will build

  • A Vendor Scorecard sandcastle with three views: a ranked scorecard table, a vendor detail page, and a trend view showing performance over time.
  • Collections for vendors (master list), delivery_metrics (on-time rate, fill rate by month), invoice_metrics (accuracy, average days to pay), and quality_events (complaints, returns, incidents).
  • A linked subagent with custom tools that pull fresh data from your WMS, ERP, and ticketing system.
  • A scheduled sync that refreshes the scorecard data weekly.

What you need before you start

  • An Assist workspace with access to the AI chat.
  • Read API access to the systems where vendor performance data lives. This playbook assumes a WMS, an ERP, and a ticketing system, but the pattern works with any combination.
  • Custom tools that pull vendor metrics from those systems. If you don't have them yet, the shared toolbox playbook walks through building and publishing tools. You will need tools like wms_vendor_delivery_stats, erp_vendor_invoice_stats, and tickets_vendor_complaints.
  • A list of the vendors you want to track. Start with the top 20 by spend — you can expand later.

Step 1: Define the scoring model

Before building anything, decide how you want to score vendors. A simple model:

  • Delivery score (0–100): on-time delivery rate over the last 90 days.
  • Invoice score (0–100): percentage of invoices that matched the PO amount without correction.
  • Quality score (0–100): 100 minus (number of quality events per 100 deliveries, capped at 100).
  • Overall score: weighted average. Delivery 40%, Invoice 30%, Quality 30%.

Write this down. You will encode it in the AI's prompt so the app and the subagent both compute it the same way.

Step 2: Describe the app

"Build me a sandcastle called 'Vendor Scorecard'. It tracks vendor performance across delivery, invoicing, and quality.

Collections:

  • vendors: vendor_id, name, category, contract_start, contract_end, payment_terms, primary_contact, status (active/inactive).
  • delivery_metrics: vendor_id, month (YYYY-MM), total_deliveries, on_time_deliveries, late_deliveries, fill_rate_pct.
  • invoice_metrics: vendor_id, month (YYYY-MM), total_invoices, correct_invoices, avg_days_to_pay.
  • quality_events: vendor_id, date, type (complaint/return/incident), description, severity (low/medium/high), resolved (boolean).

Views:

  1. Scorecard table — one row per active vendor, columns: name, category, delivery score, invoice score, quality score, overall score, trend arrow (up/down/flat vs last month). Sortable by any score. Color-code overall score: green >=80, yellow 60-79, red <60.
  2. Vendor detail — click a vendor to see all four score components charted over the last 12 months, plus a table of recent quality events and a summary of contract terms.
  3. Trend view — line chart showing overall score for the top 10 vendors over the last 6 months, so you can see who is improving and who is declining.

Seed with 15 vendors and 6 months of monthly data."

Step 3: Review and scaffold

The AI generates a plan. Check that the scoring formula matches your model. Push back on anything that doesn't look right:

"The overall score should weight delivery at 40%, not 33%. And add a 'notes' field to the vendor record so procurement can annotate."

Approve and let the AI scaffold. Preview the app. Confirm the scorecard table sorts correctly, the detail charts render, and the trend lines make sense with the seed data.

Step 4: Iterate on the views

"Add a filter dropdown at the top of the scorecard table for category. I want to see 'Logistics' vendors separately from 'Materials' vendors."

"On the vendor detail page, add a section that shows upcoming contract renewal date and days until expiry. Color it red if renewal is within 30 days."

"Make the quality events table on the detail page expandable — collapsed by default, click to show."

Preview after each edit. Get the app to a state where you would be comfortable showing it to the procurement team.

Step 5: Connect live data through the subagent

The linked subagent needs tools that pull fresh metrics from your source systems. If you have already published wms_vendor_delivery_stats, erp_vendor_invoice_stats, and tickets_vendor_complaints in your workspace toolbox, add them to the subagent's tool whitelist. See Adding tools to a subagent.

If you haven't built those tools yet, build them in /tools/new. Each tool should take a vendor_id and a date range, and return the metrics for that period. Publish each one before adding it to the subagent.

Step 6: Schedule a weekly sync

Open a chat with the linked subagent:

"Every Sunday at 11 PM Eastern, do the following for every vendor in the vendors collection:

  1. Call wms_vendor_delivery_stats for the current month and upsert the result into delivery_metrics.
  2. Call erp_vendor_invoice_stats for the current month and upsert into invoice_metrics.
  3. Call tickets_vendor_complaints for the last 30 days and insert any new events into quality_events.
  4. Recompute the overall score for each vendor using the formula: delivery 40%, invoice 30%, quality 30%.
  5. Post a summary to #procurement-ops: 'Vendor scorecard refreshed. Top 3: [names]. Bottom 3: [names]. Biggest mover: [name] (+/- N points).'

Call the trigger 'Weekly Scorecard Refresh'."

The scorecard is now self-updating. Monday morning the procurement team opens the app and sees fresh numbers.

Step 7: Share with the team

Grant Execute access to the procurement team through Workspace > Permissions > Resources. They can view the scorecard, filter, sort, and drill into vendor details. Only people with Edit access can modify the app or reset data.

What you built

You have a vendor performance scorecard that:

  • Consolidates delivery, invoicing, and quality data from three systems into one ranked view.
  • Updates itself weekly without anyone running a query or copying data.
  • Surfaces declining vendors before they become a problem — the trend arrows and color coding make it obvious.
  • Gives the procurement team a data-backed answer when it's time to renegotiate, renew, or replace a vendor.
  • Runs in your workspace — auditable, permissioned, and extensible.

What used to be a week of manual data pulling before a contract negotiation is now a bookmark.

Where to go from here

  • Add spend data. Create a spend collection and pull PO and invoice totals. Add a "spend by vendor" chart to the detail page.
  • Automate alerts. Have the subagent post to Slack when any vendor's score drops below 60 or when a contract renewal is within 30 days.
  • Weighted scoring by spend. Weight each vendor's impact by their share of total spend, so a critical vendor's declining score gets more attention than a minor one's.
  • Vendor self-service. Create a second sandcastle with a vendor-facing view showing their own scores, accessible via a shared link.

Related guides