Playbook: Build a Query Intent Classifier

You wrote a rule that blocks any warehouse query containing the word delete. It works, until someone asks a perfectly reasonable question about the deleted_orders report and gets blocked, and until someone else gets a destructive statement through by wrapping it in a way your pattern did not expect. Patterns look at text. What you care about is what the query would do. This playbook walks you through building a small tool that reads a query and says what it intends, then using that tool as a classifier so your rules act on intent instead of spelling.

What you will build

By the end of this playbook, you will have:

• A custom tool that reads a query and returns a label, a confidence score, and the tables it touches
• A rule that uses the tool as a classifier to block queries that change data
• A second rule that uses the same classifier to keep restricted tables off limits
• A cheap first check in front of the classifier, so most queries never wait for it
• A tested setup where you can see exactly what the classifier decided and why
• An agent that reviews blocked queries each week and tells you where the classifier is wrong

What you need before you start

• An Assist workspace where you are a workspace admin.
• An AI client connected to your workspace through an MCP server, or the chat in Assist.
• A tool your team uses to query data. This playbook calls it warehouse_query, with a sql parameter. If you have not built one, see Playbook: Put guardrails on warehouse queries.
• Ten to twenty real queries from your team, including a few you would want blocked. You will use them to check the classifier.

Step 1: Decide what the classifier should answer

A classifier is only as useful as the question you ask it. Keep the question narrow and the answer structured. For queries, three things are enough:

read means the query only returns data. write means it adds, changes, or removes rows. admin means it changes the structure of the warehouse or who can access it.

Avoid asking for a verdict such as "safe" or "unsafe". A label describes the query. Your rules decide what to do about it. That separation lets you change what is allowed without rebuilding the classifier.

Step 2: Build the classifier tool

In your AI client, describe the tool:

"Create a tool called 'classify_query_intent'. It takes a 'sql' parameter with a query and an optional 'dialect' parameter that is one of snowflake, athena, or postgres. It returns an object with three fields: 'label', which is one of read, write, or admin; 'confidence', a number from 0 to 1; and 'tables', a list of the fully qualified tables the query touches. The tool must not run the query or connect to the warehouse. It only reads the text."

The last two sentences matter most. A classifier runs outside your rule list, and its input comes from whatever the AI wrote in the call being checked. A tool that only reads and labels is safe to use this way. A tool that connects to a system or changes something is not.

Ask the AI how it plans to decide. A good answer combines parsing the statement with a model's judgment for the unclear cases. Push back if the plan is a list of keywords, because that is the pattern you are trying to get away from.

"Handle these cases: a select that contains the word delete inside a string or a column name should be read. A statement with several commands separated by semicolons should take the most serious label among them. A select into a new table is write. A query inside a comment should be ignored."

Step 3: Check the classifier against real queries

Before any rule depends on it, find out how good it is. Give it your real queries:

"Run classify_query_intent on each of these and show me the results in a table with the query, the label, the confidence, and the tables."

Then paste your list. Look for:

• Wrong labels. A write labeled as read is the serious kind. A read labeled as write is an inconvenience.
• Low confidence. Note the queries where confidence is below 0.8. These are the ones your rules need a plan for.
• Missing tables. If the list of tables is incomplete, a rule about restricted tables will have gaps.

Refine the tool until it gets your list right:

"It labeled 'create temporary table t as select ...' as read. That should be write. Also, it missed the table inside the subquery in number 7."

Note how long each run takes. The call being checked waits for the classifier, so a classifier that takes four seconds adds four seconds to every query your team runs.

Step 4: Add the classifier to a rule

In Assist, open Permissions and select the Tool Policies tab. Click New Rule.

1. Under Action, select Deny.

2. Under Applies to, leave Workspace.

3. In Tool, type warehouse and press Tab to complete warehouse_query.

4. Click Add classifier.

5. In Classifier tool, type classify and press Tab. When the name completes, the line under the field lists the parameters the classifier takes.

6. Click Draft input. The Input box fills in. Because both tools have a parameter called sql, the classifier's sql is set to read from the call:

   {
     "sql": "$.sql",
     "dialect": "snowflake"
   }
   

   A value that starts with $ is read from the call being checked. Any other value is sent as written. Set dialect to the one your warehouse uses.

7. Under Expect, set the first clause to label in write, admin.

8. Under On error, leave Fail closed.

9. Set Timeout (ms) to a little more than the slowest run you saw in Step 3.

10. Reason: "Queries that change data or structure are not allowed."

11. Message returned to the model on deny: "This query would change data. Only read queries are allowed. Rewrite it as a select, or tell the user to ask data engineering."

Click Create Rule.

Step 5: Decide what to do when the classifier is unsure

Your rule blocks when the label is write or admin. It says nothing about confidence. A query labeled read with a confidence of 0.4 goes through.

Add a second rule for the uncertain cases. Click New Rule and set it up the same way, with a different expectation:

1. Action: Deny, Tool: warehouse_query.
2. Add classifier, choose classify_query_intent, and click Draft input.
3. Under Expect, set the clause to confidence lt 0.7.
4. Reason: "The query could not be classified with confidence."
5. Message returned to the model on deny: "This query is unclear. Simplify it, split it into separate statements, and try again."

Both rules use the same classifier with the same input. The classifier runs once per call, and both rules read its answer.

Step 6: Put a cheap check in front

Most queries are plain selects. There is no reason to make every one of them wait for a classifier.

Conditions on a rule are checked before classifiers. If a condition fails, the classifier does not run. Use that to skip the classifier when the query obviously only reads.

Edit your first rule and click Add condition:

1. Choose the path sql.

2. Set Operator to matches.

3. Enter a pattern that describes a query worth a closer look, and tick ignore case:

   \b(insert|update|delete|merge|truncate|drop|alter|create|grant|into)\b
   

Now the classifier only runs for queries that mention one of those words. The query about the deleted_orders report still reaches the classifier, because deleted_orders does not contain delete as a whole word. If it did, the classifier would label it read and let it through. The pattern decides which queries get judged. The classifier does the judging.

Be careful not to make the pattern the gatekeeper. If a dangerous statement could avoid every word in your pattern, leave the condition off and accept the wait.

Step 7: Keep restricted tables off limits

The classifier returns the tables a query touches, including ones inside subqueries that a pattern would miss. Click New Rule:

1. Action: Deny, Tool: warehouse_query.
2. Add classifier, choose classify_query_intent, and click Draft input.
3. Under Expect, set the clause path to tables[*], the operator to matches, and the pattern to ^(payroll|hr_private)\.. Tick ignore case.
4. Reason: "Payroll and HR tables are not available through the warehouse tool."
5. Under Placement, choose Top of chain, so it applies to everyone before any exception.

With a list path such as tables[*], the clause matches if any table in the list matches.

Step 8: Test in the simulator

The test box in the rule editor skips classifiers. Select the Simulator tab and switch to Tool call.

Choose a person, enter warehouse_query in Tool, click Fill from tool, and set sql to one of your real queries. Click Evaluate.

In the Evaluation trace, the classifier line shows what the classifier returned and which clause decided. For example:

classify_query_intent → label in ["write","admin"] → "write"

Run your whole list from Step 3 through the simulator. For each query, confirm the decision is the one you expect. If the classifier fails or times out, the trace says so and shows that the rule failed closed.

Step 9: Add a weekly review agent

A classifier drifts. Your team starts writing new kinds of queries, and its accuracy changes without anyone noticing. Set up an agent to keep watch.

In your AI client:

"Create an agent called 'Policy Review' that runs every Monday at 8am. It should look at the tool calls that were blocked by policy in the last seven days for warehouse_query, group them by the rule that blocked them, and for each group show the three most common queries. For any query that looks like it only reads data, flag it as a possible false block. Post the summary to the data team channel."

Each week you get a short list of queries to look at. For the ones the classifier got wrong, go back to your AI client, show it the query, and ask it to fix the tool. Your rules do not change. They pick up the improved classifier on the next call.

What you built

Your rules now act on what a query does. A question about the deleted_orders report goes through. A statement that changes data is blocked however it is written. Restricted tables are caught even inside a subquery. Queries the classifier cannot read with confidence are sent back to be simplified. Plain selects skip the classifier entirely, so most of your team never waits.

When the classifier fails, the rule fails closed and the call is blocked, so an outage in the classifier never becomes a gap in your guardrails. And every week, an agent tells you where the classifier needs work.

Where to go next

• Classify other things. The same approach works for outbound messages ("does this contain customer account numbers?"), purchase orders ("is this within the requester's spending limit?"), and support replies ("does this promise a refund?").
• Reuse one classifier across tools. If you have query tools for several systems, one rule with a tool pattern such as *_query covers all of them.
• Build a review app. Ask your AI client for a sandcastle app that lists the week's blocked queries with the classifier's label, and lets a reviewer mark each one as correct or incorrect. Feed the incorrect ones back into the classifier.
• Give trusted groups a lighter check. Add an allow rule for data engineering above the classifier rules, so their calls skip it.

Related guides

• Using a tool as a classifier
• Writing conditions
• Testing rules with the simulator
• Playbook: Put guardrails on warehouse queries
• Troubleshooting