Testing & iterating on prompts
Treating prompts as code: versioned, tested against a fixed set of cases, and changed one variable at a time.
In plain terms
Most people edit prompts like rearranging furniture in the dark - change three things, run it once, feel like it's better. Professionals keep 20 test inputs, run every prompt version against all 20, and compare scores. Same activity, but one of them is engineering.
Why it matters
Prompts have butterfly effects: a fix for one input silently breaks three others. Without a test set you're playing whack-a-mole; every 'improvement' is anecdote. This tiny discipline is the single biggest difference between hobbyist and professional prompt work.
How it works
1) Collect 10-50 real, diverse inputs (include the nasty ones). 2) Define what 'good' means per input - expected label, must-contain facts, format rules. 3) Run the prompt on all of them; score (exact match for classification, checklists or LLM-as-judge for prose). 4) Change ONE thing, rerun, compare. 5) Keep prompt versions in git next to their scores.
When you use it
The moment a prompt matters - i.e., someone other than you will feel its failures. This node is the on-ramp to Phase 07 (evaluation), where it becomes systematic.
Common mistakes
- Testing on one example and shipping.
- Changing the prompt AND the model AND temperature at once, learning nothing.
- Only testing happy paths - the empty input, the hostile input and the 10-page input are where prompts die.
Best practices
- Grade with temperature 0 while iterating, so differences are signal not noise.
- Save every failure from production straight into the test set - it compounds.
- A spreadsheet is a perfectly respectable first eval harness.
Try it yourself
Build a 15-case test file for your Phase-02 project's tool-use prompt. Score the current prompt, make one targeted improvement, and prove the delta.
Project: Phase project - the prompt lab. Pick one real task (support-ticket triage, meeting-notes summarizer, regex generator). Build a 25-case test set, then drive accuracy up through 5 deliberate prompt versions - zero-shot -> +constraints -> +few-shot -> +CoT -> +format enforcement - recording the score at each step. Write a short README of what moved the needle and why. This is a genuinely strong portfolio artifact.
Resources
- promptfoo Open-source prompt testing harness when the spreadsheet stops scaling.