Stop Guessing Which Examples to Give Your AI

We spend a lot of time crafting prompts. We pick examples we think are representative, write out reasoning steps, and hope the model gets it. Then we run it on actual work — financial analysis, contract review, technical troubleshooting — and find it stumbles on exactly the cases that matter most. The examples we chose were fine, just not the right ones.

This is the quiet tax on anyone deploying AI seriously. Not the big technical problems, but the slow, grinding work of figuring out which examples to include in a prompt. The selection feels arbitrary because it mostly is.

What Active-Prompt Is

Active-Prompt is a technique for identifying which examples your AI model actually needs you to explain — and only those. Instead of manually selecting a handful of representative questions to annotate, you ask the model to answer a pool of questions multiple times each. Where it consistently agrees with itself, it already has a handle on things. Where its answers scatter, that is genuine uncertainty — those are the questions worth your annotation time. You write out careful, step-by-step reasoning for just those cases, and the model uses them as its working examples going forward. You are concentrating human effort where it changes outcomes, not where the model was already doing fine.

Before and After

Before — Standard chain-of-thought prompt (the guesswork approach):

You have a pool of 200 customer refund queries that involve partial credits, pro-rated charges, and multi-item returns. You pick six examples you think cover the range, write out the reasoning manually, and paste them into the prompt.

"Here are some examples of how to calculate customer refund amounts... [your six hand-picked examples] Now calculate the refund for the following case: [new query]"

The model handles straightforward cases well but keeps miscalculating pro-rated partial returns and cases where store credit was previously applied. Those happen to be exactly the query types you did not think to include.

After — Active-Prompt approach:

You run your 200 refund queries through the model ten times each. You look at which ones produced wildly different answers across those ten runs — sometimes full refund, sometimes partial, sometimes zero. You pick the eight most inconsistent. You sit down and write out the reasoning for those eight cases specifically, step by step.

"Q: A customer purchased a three-month software subscription for $90 on March 1st and is requesting a refund on April 10th. They used a $15 store credit on the original purchase. How much do they receive back in cash?
A: The subscription costs $30 per month. By April 10th, approximately 1.3 months have elapsed, so two months remain. The refundable amount before credits is $60. The $15 store credit is non-refundable, so it is subtracted from the original payment. The customer paid $75 in cash. The refundable portion of that cash payment is $60 x ($75/$90) = $50. The answer is $50."

Now the model has exactly the examples it was confused about. Performance on your actual case distribution improves because the examples match where the model was genuinely uncertain, not where you assumed it needed help.

When to Use Active-Prompt

• Domain-specific calculations with edge cases. Financial reconciliations, insurance claims, billing adjustments — anywhere the reasoning involves conditionals that trip up general models. You have limited time to annotate; use it on the cases the model cannot handle consistently.

• Legal or compliance document review. When classifying contract clauses or flagging regulatory issues, the straightforward cases are cheap. The ambiguous ones are expensive if wrong. Active-Prompt points you directly at the ambiguous ones.

• Customer support triage and routing. If you are prompting a model to categorize or respond to support tickets, uncertainty across categories tells you exactly which ticket types need clearer worked examples before the model can be trusted on them.

• Any task where you have a reasonable pool of representative questions but limited expert time. The logic holds at smaller scales than the published research. If you have 100–200 task-relevant questions and can query the model multiple times, that is enough to surface the hard cases worth annotating. Start there before worrying about scale.

• Adapting a general-purpose model to a specialized context without fine-tuning. Organizations that cannot fine-tune models but need consistent reasoning on domain-specific tasks can use this approach to get much of the benefit through smarter few-shot selection alone.

Try It on Your Next Prompt Project

The investment is modest. You need a pool of task-relevant questions, the ability to run the model against them multiple times — the research found uncertainty estimates converge at around ten runs per question — and a human who can write clear reasoning steps for the eight or so questions the model found hardest. A few hours of focused work, not a fine-tuning project.

The published results are worth looking at directly. Across eight reasoning benchmarks, Active-Prompt beat standard chain-of-thought prompting and self-consistency methods in most comparisons, though the gains varied by model and the gpt-3.5-turbo experiments did not include self-consistency due to cost. The strongest result was a seven-percentage-point improvement over self-consistency on one model — meaningful headroom when the underlying task is consequential. Weaker models saw the largest gains; stronger models improved more modestly.

Active-Prompt belongs to a broader cluster of techniques that take seriously the question of how to spend annotation effort well. Upcoming posts will cover related approaches, including methods for automatically generating reasoning chains and for combining uncertainty with diversity in example selection.

Source: Diao, Wang, Lin, Pan, Liu, Zhang, & Liu (2023, revised 2024). Active Prompting with Chain-of-Thought for Large Language Models. arXiv:2302.12246