Question 1

Why version prompts like code?

Accepted Answer

Prompts behave like code: small changes produce large quality shifts. Versioning (semver) lets you compare evaluation scores across changes, attribute regressions to specific edits, and roll back to a prior prompt without losing context. Treat the prompt repository like source code — review, version, deploy.

Question 2

What is a forbidden pattern in prompt testing?

Accepted Answer

A phrase, structure, or content type that the model output must never contain. Examples: 'I'm just an AI' / 'As an AI language model', fabricated URLs, plaintext PII (emails, SSN, credit card numbers), apologies without action, profanity. A test fails if any forbidden pattern is detected via regex or classifier.

Question 3

What is a regression threshold?

Accepted Answer

The percentage-point drop in evaluation score that triggers a deploy block. Typical: '5pp drop blocks merge.' Without it, gradual quality decay slips through. With it, prompt edits must justify themselves with stable or improving scores.

Question 4

How big should a golden set be?

Accepted Answer

Minimum 20 examples for an early-stage prompt to be useful. 50-100 for a stable feature. 500+ for high-stakes prompts (medical, legal, financial). Update monthly with examples from real failures — every shipped bug becomes a new test.

Question 5

Should I use temperature 0 for testing?

Accepted Answer

Yes for regression testing — determinism makes evaluation reliable. Use production temperature for occasional 'realistic distribution' runs. A failing regression test at T=0 will almost certainly fail at T>0 too.

LLM Prompt Test Case (2026)

Golden Examples

How to use this template

AI Feature Test Plan

Test Case

Test Strategy

Bug Report