Prompt Engineering

Type	Class
Resource	urn:ngm:class:prompt-engineering
Domain	artificial-intelligence
Status	stable
Maturity	established
Quality	0.52
Is a	AI Technique In-Context Learning AI Application Natural Language Processing Human-Computer Interaction Machine Learning Discipline AI Alignment Instruction Following Applied Artificial Intelligence
Has part	Automatic Prompt Optimisation Chain of Thought DSPy Few-Shot Prompting Prompt Template ReAct Self-Consistency Structured Output System Prompt Tree of Thoughts Zero-Shot Prompting System Prompts Prompt Templates Prompt Injection Defence
Requires	Instruction Following RLHF Attention Large Language Models Natural Language Processing Neural Network Text Tokenisation Transformer Architecture
Enables	Retrieval-Augmented Generation Structured Output Tool Use Agentic Internet Agents Chatbots CLI Multi-Agent Systems Code Generation Reasoning
Generated	by: process:vault-migrate/1.0 · at: 2026-09-22T12:41:39.024503178Z
Contrasts with	Training and fine tuning Model Optimisation and Performance RLHF Active Learning
Depends on	Large Language Models Attention Transformer Architecture Neural Network Text Tokenisation Context Window RLHF Artificial General Intelligence
Implemented in layer	ApplicationLayer APILayer AlgorithmLayer
Implements	Automatic Prompt Optimisation Chain of Thought DSPy In-Context Learning Instruction Following ReAct Self-Consistency
Legacy term id	AI-2041
Links	AlgorithmLayer Anthropic Prompt Engineering Documentation 2024 APILayer Applied Artificial Intelligence Automatic Prompt Optimisation Bai et al. 2022 Constitutional AI Brown et al. 2020 GPT-3 Few-Shot Learning Chain of Thought DSPy Fernando et al. 2023 PromptBreeder Few-Shot Prompting Greshake et al. 2023 Prompt Injection HumanComputerInteractionDomain In-Context Learning Instruction Following Khattab et al. 2024 DSPy Kojima et al. 2022 Zero-Shot Reasoners Lu et al. 2022 Fantastically Ordered Prompts Min et al. 2022 ICL Role of Demonstrations NCSC UK AI Security Guidelines 2024 NLPDomain OpenAI Prompt Engineering Guide 2024 Ouyang et al. 2022 InstructGPT Prompt Injection Defence Prompt Templates ReAct Retrieval Augmented Generation RLHF Schulhoff et al. 2024 Prompt Report Survey Self-Consistency Structured Output System Prompts Tool Use Tree of Thoughts Wang et al. 2022 Self-Consistency Wang et al. 2023 Plan-and-Solve Wei et al. 2022 Chain-of-Thought Prompting Xie et al. 2022 Bayesian ICL Theory Yao et al. 2022 ReAct Yao et al. 2023 Tree of Thoughts Zero-Shot Prompting Zhou et al. 2022 APE Active Learning Agent Frameworks Agentic Internet Agents AI Adoption AI Alignment AI-GroundedDomain AI Risks AI Scrapers Algorithmic Bias and Variance Anthropic Claude ApplicationLayer Artificial General Intelligence Attention Bias in Large Language Models ChatGPT Chatbots CLI Multi-Agent Systems Code Generation Cognitive AI ComfyUI Context Window Courses and Training Evaluation benchmarks and leaderboards Human-Computer Interaction LangChain Large Language Models Machine Learning Model Optimisation and Performance Natural Language Processing OpenAI Reasoning Stable Diffusion Tokenisation Training and fine tuning Transformer Architecture
Page resource	urn:visionflow:page:367be6addad9ba59f12e52022d4d80693381610ca5dd92096b7fc68db6045d6f
Related to	Agents AI Adoption AI Risks AI Scrapers Algorithmic Bias and Variance Bias in Large Language Models Chatbots Cognitive AI
Standardized by	Agent Frameworks Constitutional AI Language Model Family AI Technology Skills Curriculum Evaluation benchmarks and leaderboards OpenAI Research Organisation
Supports	Retrieval-Augmented Generation Agent Frameworks AI Adoption AI Alignment Code Generation Cognitive AI Evaluation benchmarks and leaderboards
Uses	DSPy Constitutional AI Language Model Family Instruction-Following Conversational AI System Node-Based Diffusion Pipeline Interface LangChain Reasoning Stable Diffusion Image Model

Prompt Engineering is the systematic discipline of designing, structuring, and optimising natural language or structured inputs (prompts) to elicit desired behaviours, outputs, and reasoning traces from large language models (LLMs) and other generative AI systems, spanning a hierarchy of techniqu…

In Plain Terms

The craft of wording your instructions to an AI so it gives you what you actually want: being clear about the task, offering examples, and setting the format. Small changes in how you ask can make a large difference to the answer you get back.

Semantic Classification

Content

Compositional Relationships (Components)

SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:hasPart ai:ZeroShotPrompting))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:hasPart ai:FewShotPrompting))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:hasPart ai:ChainOfThought))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:hasPart ai:TreeOfThoughts))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:hasPart ai:ReActFramework))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:hasPart ai:SystemPrompt))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:hasPart ai:PromptTemplate))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:hasPart ai:PromptInjectionDefence))

## Dependency Relationships
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:requires ai:LargeLanguageModel))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:requires ai:NaturalLanguageProcessing))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:requires ai:Tokenisation))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:requires ai:AttentionMechanism))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:dependsOn ai:TransformerArchitecture))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:dependsOn ai:ContextWindow))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:dependsOn ai:InstructionTuning))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:dependsOn ai:RLHF))

## Capability Relationships
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:enables ai:ReasoningTraces))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:enables ai:ToolUse))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:enables ai:StructuredOutput))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:enables ai:CodeGeneration))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:enables ai:MultiStepPlanning))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:supports ai:AgentFrameworks))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:supports ai:RetrievalAugmentedGeneration))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:supports ai:AIAlignment))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:supports ai:EvaluationBenchmarks))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:supports ai:MultimodalReasoning))

## Implementation Relationships
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:implements ai:InContextLearning))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:implements ai:SelfConsistencyDecoding))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:implements ai:AutomaticPromptOptimisation))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:implements ai:DSPyCompilation))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:implements ai:PromptCaching))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:uses ai:XMLStructuredPrompts))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:uses ai:JSONSchemaConstraints))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:uses ai:EnsembleReasoning))

## Reduction Relationships
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:reduces ai:HallucinationRate))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:reduces ai:FineTuningCost))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:reduces ai:LabelAnnotationBurden))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:reduces ai:InferenceCost))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:reduces ai:DevelopmentTime))

## Association Relationships
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:relatedTo ai:ModelAlignment))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:relatedTo ai:AgenticAI))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:relatedTo ai:ImageGeneration))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:relatedTo ai:SoftwareEngineering))
SubClassOf(ai:PromptEngineering
  ObjectSomeValuesFrom(ai:relatedTo ai:CognitiveBias))

## Data Properties (Characteristics)
DataPropertyAssertion(ai:hasIdentifier ai:PromptEngineering "AI-2041"^^xsd:string)
DataPropertyAssertion(ai:authorityScore ai:PromptEngineering "0.87"^^xsd:decimal)
DataPropertyAssertion(ai:gsmPerformanceGainCoT ai:PromptEngineering "0.308"^^xsd:decimal)
DataPropertyAssertion(ai:totGame24SuccessRate ai:PromptEngineering "0.74"^^xsd:decimal)
DataPropertyAssertion(ai:selfConsistencyGainGSM8K ai:PromptEngineering "0.179"^^xsd:decimal)
DataPropertyAssertion(ai:dspyImprovementRange ai:PromptEngineering "0.65"^^xsd:decimal)

## Property Constraints
SubClassOf(ai:PromptEngineering
  DataAllValuesFrom(ai:requiresLLMBackend xsd:boolean))
SubClassOf(ai:PromptEngineering
  DataSomeValuesFrom(ai:techniqueType xsd:string))
SubClassOf(ai:PromptEngineering
  DataMinCardinality(1 ai:hasOutputFormat xsd:string))

## Annotations
AnnotationAssertion(rdfs:label ai:PromptEngineering "Prompt Engineering"@en)
AnnotationAssertion(rdfs:comment ai:PromptEngineering "Systematic discipline of designing structured natural language inputs to elicit desired behaviours from LLMs, spanning zero-shot through few-shot ICL, chain-of-thought (Wei 2022 GSM8K 17.9%→48.7%), self-consistency (Wang 2022 +17.9%), tree-of-thoughts (Yao 2023 Game-of-24 4%→74%), ReAct interleaved reasoning-action, DSPy compiled optimisation, XML/JSON structured outputs, prompt injection defences, and Anthropic prompt caching at 10× cost reduction, enabling agents, RAG, code generation, and alignment without fine-tuning."@en)
AnnotationAssertion(dcterms:identifier ai:PromptEngineering "AI-2041"^^xsd:string)
AnnotationAssertion(dcterms:subject ai:PromptEngineering "Prompt Engineering, In-Context Learning, Chain of Thought, LLM Alignment, Agentic AI"@en)

)

Property Characteristics

AsymmetricObjectProperty(ai:requires) AsymmetricObjectProperty(ai:enables) AsymmetricObjectProperty(ai:implements) AsymmetricObjectProperty(ai:reduces) TransitiveObjectProperty(ai:dependsOn) FunctionalDataProperty(ai:gsmPerformanceGainCoT) FunctionalDataProperty(ai:selfConsistencyGainGSM8K)


- ## About Prompt Engineering
- **Prompt Engineering** is the practice and science of crafting inputs to large language models to reliably produce accurate, safe, and useful outputs. Unlike traditional software engineering, where logic is encoded explicitly in code, prompt engineering exploits the emergent capabilities of foundation models by shaping their behaviour through natural language instructions, examples, persona assignments, and structural conventions—all within the context window at inference time.

- The discipline emerged as a practical necessity after OpenAI GPT-3 (Brown et al. 2020) demonstrated that carefully constructed few-shot prompts could match fine-tuned task-specific models on many NLP benchmarks. As model scale grew and instruction-following capabilities improved through RLHF (Ouyang et al. 2022), prompt engineering became the primary interface between application developers and AI capability: a lightweight alternative to expensive fine-tuning, enabling rapid iteration without gradient updates.

- By 2024-2026 the field had bifurcated into **manual prompt design** (iterative human authoring guided by model-specific documentation) and **automatic prompt optimisation** (programmatic search over instruction spaces, DSPy compilation, evolutionary strategies), with the latter increasingly dominant in production deployments. The central tension in the discipline is the gap between empirical prompt effectiveness and theoretical understanding: why specific phrasings, orderings, and structural conventions improve performance remains partly opaque, driving an active research agenda at the intersection of NLP, cognitive science, and AI alignment.

- ### Zero-Shot and Few-Shot Prompting

- **Zero-shot prompting** relies solely on the model's pre-trained knowledge and instruction-following capability. A clear task description, output format specification, and any relevant constraints are provided; no examples are shown. GPT-4 achieves 87.1% on MMLU with zero-shot prompting, close to 90.4% with five-shot, demonstrating that sufficiently large instruction-tuned models generalise robustly from descriptions alone.

- **Few-shot in-context learning (ICL)** (Brown et al. 2020) provides k labelled demonstrations (typically k = 1 to 16) in the prompt prefix. The model infers the task distribution from these examples without weight updates. Key findings from the ICL literature:
- **Demonstration ordering matters**: Models are sensitive to the order of few-shot examples; optimal orderings can differ by 10-15 accuracy points (Lu et al. 2022, *Fantastically Ordered Prompts*).
- **Label semantics**: Models attend to both input-output mappings and label token distributions; using semantically related labels slightly outperforms random tokens (Min et al. 2022).
- **Format consistency**: Keeping the format (delimiters, separators, capitalisation) uniform across demonstrations reduces tokenisation ambiguity and improves F1 by 3-8% on sequence labelling tasks.
- **Many-shot ICL**: With long-context models (Gemini 1.5 Pro 1M token window, Claude 3.5 context 200K tokens), k can be scaled to hundreds or thousands of examples, approaching fine-tuning performance on classification and translation without gradient updates (Many-Shot ICL, arXiv:2405.09798, 2024).

- ### Chain-of-Thought and Derived Techniques

- **Chain-of-Thought (CoT) Prompting** (Wei et al. 2022, NeurIPS Spotlight) demonstrated that appending explicit step-by-step reasoning to few-shot demonstrations elicits coherent multi-step reasoning from large language models. On GSM8K (school math word problems), GPT-3 175B with CoT achieved 48.7% vs 17.9% chain-free. On MATH (competition problems), PaLM 540B improved from 5% to 26% with CoT, and from 26% to 34% with additional process-supervision reward modelling. The mechanism appears to leverage the attention mechanism's ability to condition later tokens on earlier reasoning tokens, allocating additional compute to intermediate steps.

- **Zero-Shot CoT** (Kojima et al. 2022) showed that simply appending "Let's think step by step." to a user prompt (without demonstrations) yields substantial improvements on arithmetic (GSM8K: +30pp) and symbolic reasoning tasks, attributing the effect to instruction-following fine-tuning that makes models interpret the phrase as a reasoning cue.

- **Self-Consistency** (Wang et al. 2022) replaces greedy decoding with sampling-then-majority-voting: 40 diverse CoT chains are generated at temperature T=0.7; the most frequently returned final answer is selected. On GSM8K this yields 74.4% (vs 56.5% single-chain CoT), on MATH 56% (vs 45%), and on StrategyQA 86.2% (vs 78%). The gains are consistent across PaLM, GPT-3, and GPT-4, with diminishing returns above ~40 samples.

- **Tree of Thoughts (ToT)** (Yao et al. 2023, NeurIPS) frames problem solving as search over a tree of intermediate language sequences ("thoughts"), with an LLM evaluator scoring each node. BFS explores all width-w partial solutions at each depth; DFS with backtracking prunes unpromising branches. On the Game of 24 (arithmetic combination), ToT achieves 74% vs 4% (IO) and 4% (CoT); on crossword puzzles 60% word success vs 16% (CoT). The framework is compute-intensive (100-1000× more LLM calls than single-pass) but necessary when problems require exploration and lookahead.

- **Graph of Thoughts (GoT)** (Besta et al. 2024) generalises ToT to directed acyclic graphs allowing thought merging (aggregation) and refinement cycles, enabling solution quality comparable to ToT at lower call counts on sorting and set-operation benchmarks.

- **ReAct** (Yao et al. 2022) interleaves reasoning ("Thought: ...") with action invocations ("Action: Search[query]") and observation processing ("Observation: ...") in a single prompt context, enabling models to retrieve real-time information, execute code, or call APIs as part of reasoning. On HotpotQA ReAct achieves 35.1% EM vs 29.4% CoT alone, and on FEVER 80% vs 56.3%. ReAct serves as the foundation for most modern [[Agent Frameworks]] and tool-augmented LLM systems.

- **Plan-and-Solve Prompting** (Wang et al. 2023) first asks the model to devise a plan ("Let's first understand the problem and devise a plan to solve it") before executing steps, reducing reasoning errors from skipped steps and misalignment between plan and solution. On 10 reasoning benchmarks, Plan-and-Solve+CoT outperforms Zero-Shot CoT by 2-5%.

- **Step-Back Prompting** (Zheng et al. 2023) elicits high-level principles before solving specific instances, improving physics (PaLM2 84.2% vs 49.4%) and chemistry reasoning by first abstracting to domain principles, then applying them.

- **Least-to-Most Prompting** (Zhou et al. 2022) decomposes complex problems into easier sub-problems solved sequentially, particularly effective on compositional generalisation tasks (SCAN benchmark: 99.7% vs 16.3% standard few-shot).

- ### Automatic Prompt Engineering and Compilation

- **Automatic Prompt Engineer (APE)** (Zhou et al. 2022, ICLR 2023) uses an LLM to generate candidate instructions from demonstrations, scores candidates on a validation set, and selects the best. On instruction-induction benchmarks across 24 tasks APE outperforms human-authored prompts by an average of 5% and matches or exceeds CoT on 21/24 tasks. The "Let's work this out in a step by step way" instruction discovered by APE rivals manually authored CoT triggers.

- **DSPy** (Khattab et al. 2024, ICLR) introduces a programming model where prompts are declared as typed signature modules (e.g., `dspy.ChainOfThought("question -> answer")`), and a compiler bootstraps few-shot demonstrations and/or tunes instructions via **MIPRO** (Bayesian prompt optimiser) or **BootstrapFewShot** (automatic example selection from training trajectories). On HotpotQA, DSPy-compiled pipelines reach 51.4% vs 36.8% (manual CoT), a 40% relative gain. On complex multi-hop QA (MuSiQue) gains reach 65%. DSPy decouples prompt logic from model-specific formatting, enabling portability across GPT-4, Claude 3, Llama 3, and Mixtral. The framework represents a paradigm shift from artisanal prompt crafting to systematic programme optimisation.

- **PromptBreeder** (Fernando et al. 2023) applies evolutionary algorithms to prompt populations: mutation operators (paraphrase, extend, simplify) and selection pressure on a fitness metric (task accuracy) evolve prompt populations over 20-50 generations, discovering counter-intuitive phrasings human engineers would not generate. On arithmetic and common-sense benchmarks, PromptBreeder prompts outperform APE and CoT by 3-8%.

- **Evolutionary Symbolic Prompting**: Reuven Cohen's approach (2024) structures task descriptions using mathematical notation (sets, functions, constraints), reducing ambiguity and enabling models to reason combinatorially. Reported cost reduction from $15/document to $0.22/document on pharmaceutical compliance analysis, with improved consistency.

- ### Structured Prompting and Format Control

- **XML-tagged prompts** (Anthropic best practices, 2024) use tags such as `<document>`, `<instructions>`, `<example>`, and `<output>` to delimit semantic regions of the prompt, reducing confusion between instruction and data contexts. Claude models respond strongly to XML structure, with tag-delimited documents reducing instruction-following errors by an estimated 20-30% on complex multi-document tasks.

- **JSON Schema constraints** (OpenAI Function Calling, structured outputs mode, 2024) force models to produce JSON matching a declared schema. OpenAI's constrained decoding implementation guarantees schema conformance at the token level via finite-state machine decoding, enabling reliable extraction of structured data without post-processing. This mode halves downstream parsing error rates in production pipelines.

- **Instructor library** (Python, open-source) wraps OpenAI and Anthropic APIs with Pydantic validation, automatically retrying and self-correcting malformed structured outputs. Widely adopted in 2024-2025 production pipelines.

- **Markdown and header structure**: Using `##` headers, bullet lists, and numbered steps in system prompts improves instruction separation and reduces instruction bleed across long contexts. Particularly valuable for complex multi-step agent instructions exceeding 2K tokens.

- **Persona assignment**: Assigning specific expert identities ("You are a senior UK patent attorney specialising in AI inventions") reduces decision paralysis (model uncertainty about appropriate response style) and increases domain-specific accuracy by 5-15% on specialised professional tasks (legal, medical, financial).

- ### Prompt Injection and Security

- **Prompt injection** (Greshake et al. 2023, *Not What You've Signed Up For*) describes adversarial instructions embedded in untrusted external content (web pages, documents, emails, API responses) that override the system prompt or redirect model behaviour. In indirect injection attacks demonstrated against GPT-4 browsing mode, adversarial content in retrieved web pages successfully exfiltrated user data and triggered unintended actions in 43% of test cases.

- **Defence strategies**:
- **Input sanitisation**: Strip or escape potential injection markers (`[INST]`, `###`, `SYSTEM:`, XML tags) before injecting external content into prompts.
- **Privilege separation**: Maintain strict system/user/tool-output turn boundaries; never allow tool outputs to appear in the system turn.
- **Instruction hierarchy enforcement**: Anthropic's Claude system prompt architecture (2024) implements a three-tier hierarchy (operator → user → assistant) where lower tiers cannot override higher-tier instructions, formalised in the model constitution.
- **Spotlighting** (Microsoft, 2023): Wrap retrieved documents in delimiters and explicitly tell the model that only text outside delimiters represents trusted instructions.
- **LLM-based detection**: Secondary classifier models (fine-tuned on injection examples) screening tool outputs before integration, achieving 89-94% injection detection at <5% false positive rates in production settings.

- **Jailbreaking** (distinct from injection): User-level adversarial prompts attempting to bypass safety constraints. Mitigations include Constitutional AI (Anthropic), RLHF safety training, and classifier-based guards. Jailbreak success rates on frontier models declined from ~60% (GPT-3.5) to <10% (Claude 3.5, GPT-4o) on standard red-team benchmarks by 2025.

- ### System Prompts and Deployment Architecture

- **System prompt vs user prompt** separation implements a role-based trust hierarchy. The system prompt (operator-controlled) defines persona, capabilities, safety constraints, and task scope. The user turn carries request content. Models trained with RLHF/RLAIF learn to prioritise operator-level constraints over user instructions when conflicts arise.

- **Anthropic Prompt Caching** (2024) stores up to 200K tokens (claude-3-5-sonnet) or 1M tokens (claude-3-5-haiku with extended cache) of repeated context prefix in server-side KV cache. Cache hit pricing is 90% lower than standard input pricing ($0.30/MTok vs $3.00/MTok for Sonnet), enabling cost-effective deployment of long system prompts, large tool libraries, or extensive few-shot demonstrations across multi-turn conversations. Time-to-first-token is reduced by 80-85% for cache hits, critical for interactive applications. Production deployments use `cache_control: {type: "ephemeral"}` markers on the last message boundary to be cached.

- **OpenAI Realtime Voice API prompting** (2024) introduces voice-specific constraints: system prompts must specify turn-taking policy (interruption sensitivity, latency tolerance), pronunciation guides for technical terms, and persona voice characteristics. Responses should be conversational in register (short sentences, no markdown formatting, acoustic-appropriate pacing) and tolerant of transcription errors from speech-to-text.

- **Model-specific quirks** (2025 landscape):
- **Claude 3.5+**: Strongly responds to XML-delimited context regions; resists instruction-following when asked to perform unsafe actions even via indirect prompts; benefits from explicit permission grants in system prompt for edge-case capabilities.
- **GPT-4o**: Sensitive to instruction ordering—place most important constraints early; function-calling mode reliably extracts structured data; benefits from terse instructions over verbose explanations.
- **Llama 3 / Llama 3.1**: Uses `<|begin_of_text|>`, `<|start_header_id|>system<|end_header_id|>` special tokens for role separation; Alpaca [INST]/[/INST] format also effective; lower instruction-following fidelity than GPT-4/Claude on complex multi-constraint tasks.
- **Gemini 1.5 Pro/1.5 Flash**: Benefits from explicit "think carefully" preambles on complex reasoning; long-context performance (1M tokens) enables many-shot ICL at scale; multimodal prompting with images benefits from detailed description requests.
- **Mistral/Mixtral**: Instruction format `[INST] {instruction} [/INST]` required for instruction-tuned variants; less prone to verbosity than GPT-4 but weaker on multi-hop reasoning without explicit CoT.

- ### Anthropic and OpenAI Prompt Engineering Guidance (2024-2026)

- Anthropic's official prompt engineering documentation (2024-2025) recommends:
1. **Be specific and direct**: State the task, output format, and constraints explicitly. Claude responds poorly to vague instructions.
2. **Use XML tags for structure**: `<document>`, `<instructions>`, `<examples>`, `<output_format>` tags prevent context bleeding in long prompts.
3. **Assign roles carefully**: "You are a [specific expert role]" significantly shapes response style and domain focus.
4. **Iterative prompt refinement**: Test across diverse inputs; track failure modes; version control prompts using tools like PromptLayer or LangSmith.
5. **Separate system from user context**: Operator-level safety and persona constraints belong in the system prompt; never expose to users.
6. **Leverage prompt caching**: Cache stable system prompts and few-shot libraries for 90% cost reduction on repeated prefixes.
7. **Extended thinking**: Claude 3.7 Sonnet's extended thinking mode (budget tokens 1K-100K) allows allocation of additional reasoning compute for difficult problems, comparable to test-time compute scaling.

- OpenAI's Prompt Engineering Guide (2024) recommends: write clear instructions (specify format, length, style), provide reference text (RAG for factual grounding), split complex tasks into sub-tasks (akin to plan-and-solve), give models "time to think" (CoT elicitation), use external tools (function calling, code interpreter), and systematically test prompt changes (A/B evaluation with LLM judges or golden test sets).

- ### Use Cases and Major Application Families

- **Code generation**: Chain-of-thought decomposition of requirements into function signatures, then step-by-step implementation. GitHub Copilot, Cursor, and Amazon Q use custom system prompts tailored to repository context, coding style guides, and security policies. Context-stuffing strategies inject relevant file contents and dependency declarations into the prompt for improved contextual consistency.

- **Retrieval-Augmented Generation (RAG)**: [[Retrieval Augmented Generation]] pipelines inject retrieved document chunks into a prompt template, typically with XML-delimited document blocks and explicit attribution instructions ("Answer only from the provided documents"). Prompt engineering determines citation quality, hallucination rate, and faithfulness. Techniques include query rewriting ("Step-Back" to retrieve more relevant documents), self-RAG (model decides when to retrieve), and iterative refinement.

- **Agentic task planning**: [[Agents]] and [[Agent Frameworks]] rely on system prompts to specify tool inventories, planning protocols (ReAct, Plan-and-Execute), output schemas, error-handling instructions, and safety constraints. Effective agent prompts decompose capability into explicit sub-modules and maintain working memory via structured scratchpad sections.

- **Data extraction and classification**: JSON schema-constrained prompts with few-shot examples for named entity recognition, relation extraction, and document classification. LLM-as-judge frameworks use structured rubric prompts to evaluate other model outputs, replacing expensive human annotation for qualitative assessment.

- **Creative and domain-expert tasks**: Persona-based prompts for legal drafting, medical summarisation, financial analysis, and creative writing. Style guides embedded in system prompts ensure consistency across long documents. Multi-persona review prompts (pessimist, pragmatist, creative critic) improve output quality through simulated adversarial review.

- **Image generation prompting** (for [[Stable Diffusion Image Model]], DALL-E 3, Midjourney): Compositional positive prompts specifying subject, style, lighting, camera angle, artist reference, and quality terms; negative prompts excluding artefacts, deformations, and style contaminations. SDXL prompt dynamics (warm-tone bias, CFG sensitivity, negative-space style control) require model-specific calibration. Comfy UI workflow prompting integrates LLM-generated prompt text into graph-based pipelines via Ollama or API nodes.

- ### Academic Context

- The foundational theoretical analysis of in-context learning (Min et al. 2022; Xie et al. 2022) frames ICL as implicit Bayesian inference: the model infers a latent task concept from demonstrations, computing a posterior over tasks consistent with observed (input, output) pairs and generating responses accordingly. This framing predicts label sensitivity, ordering effects, and the role of demonstration informativeness—predictions broadly confirmed empirically.

- The attention mechanism provides the computational substrate: multi-head attention over the full context enables the model to implement any finite-length function mapping from demonstrations to outputs (Akyürek et al. 2022, *What Algorithms Can Transformers Learn?*), providing a theoretical upper bound on ICL expressivity.

- Wei et al. (2022) attribute CoT emergence to scale: models below ~100B parameters show negligible improvement from CoT triggers; above this threshold gains are consistent and substantial. This emergent behaviour remains partially unexplained—it likely reflects the model's ability to route computation through intermediate token representations acting as working memory.

- Anthropic's mechanistic interpretability research (Olah et al., 2020-2024) has begun identifying specific attention heads and MLP neurons involved in in-context task switching, providing early mechanistic foundations for understanding why certain prompt structures are effective.

- Edinburgh's language technology group (specifically the ILCC) studies prompting robustness across low-resource languages and domain transfer. Imperial College London's AI group has examined adversarial prompting and prompt injection from a security-theoretic perspective. Cambridge's NLP group contributes to ICL theory and few-shot generalisation bounds.

- ### Current Landscape (2026)

- By 2026, prompt engineering has matured into a layered discipline:
- **Layer 1 — Manual artisanal prompting**: Practitioners iteratively refine prompts using model-specific documentation, heuristics, and intuition. Still dominant for one-off tasks and rapid prototyping.
- **Layer 2 — Programmatic templating**: LangChain, LlamaIndex, Instructor, and Guidance provide typed, parameterised prompt templates with input validation, output parsing, and retry logic.
- **Layer 3 — Automatic optimisation**: DSPy compilation, APE, PromptBreeder, and evolutionary strategies systematically search instruction spaces, bootstrapping few-shot demonstrations from labelled data and optimising on held-out validation metrics.
- **Layer 4 — Model-native compilation**: OpenAI's structured output mode and Anthropic's tool-use schemas embed format constraints at the inference engine level, removing format engineering from the prompt layer entirely.

- The "prompt engineering is dead" narrative (IEEE Spectrum, 2024) reflects the observation that instruction-following has improved to the point that simple natural language instructions often suffice—but this understates the continuing importance of prompt architecture for complex agentic, multi-step, and constrained-output applications where careful design remains essential.

- **Test-time compute scaling** (OpenAI o1/o3, Claude 3.7 extended thinking, 2025-2026) shifts some prompt engineering concerns to compute budget allocation: rather than crafting reasoning triggers, engineers specify token budgets, and the model allocates internal reasoning steps autonomously. This does not eliminate prompt engineering but relocates its focus toward task specification clarity and output format definition.

- **Frontier model capabilities (2026)**: GPT-4.5 (rumoured, 2025), Claude 4 (Anthropic roadmap), and Gemini 2.5 Ultra achieve near-saturation on standard reasoning benchmarks with minimal prompting; engineering effort shifts toward multi-agent coordination, long-context fidelity, tool-use reliability, and domain-specific calibration.

- ### UK Context

- **Academic institutions**:
- **University of Edinburgh / ILCC**: Active research on cross-lingual prompt transfer, few-shot learning for low-resource languages, and evaluation methodology for prompting (including HONEST benchmark for bias). Collaborates with Scottish Government on public-sector LLM deployment.
- **Imperial College London**: Security-theoretic analysis of prompt injection (adversarial NLP group); collaboration with NCSC on LLM security guidelines for critical infrastructure.
- **University of Cambridge / NLP Group**: Contributions to ICL theory (Bayesian framing, sample complexity), evaluation methodology (BIG-Bench), and structured generation.
- **University of Oxford / Future of Humanity Institute**: Alignment-focused prompting research; analysis of instruction hierarchy and value specification in system prompts.
- **University of Manchester**: Prompting for knowledge graph population and biomedical NLP; AIDA project integrating LLMs with structured knowledge bases via curated prompt templates.
- **University of Leeds**: AI consultancy research group developing sector-specific prompt libraries for financial services and legal compliance; collaboration with Leeds Teaching Hospitals NHS Trust on clinical note summarisation via few-shot RAG prompts.

- **ARM Holdings (Cambridge)**: ARM-based LLM inference (Ethos-U85 NPU, Cortex-M85) constrains context window to 2K-8K tokens for on-device deployment, driving research into compressed prompt formats, prefix caching on-chip, and model distillation to reduce prompt sensitivity for edge deployment.

- **UK AI consultancies**:
- **Faculty AI (London/Bristol)**: Developed sector-specific prompt playbooks for UK government departments (HMRC, DWP, NHS); expertise in RAG-based prompting for structured public-sector data.
- **Luminance (London)**: Legal AI platform using few-shot contract analysis prompts with structured XML output schemas, deployed in 450+ law firms globally including Magic Circle firms.
- **Wayve (Cambridge/London)**: Autonomous driving system prompts for multimodal scene description and scenario generation, integrating LLM reasoning with sensor fusion pipelines.
- **Stability AI (London, now restructured)**: Contributed to Stable Diffusion prompt engineering community documentation; SDXL model-specific guidance.
- **BenevolentAI (London)**: Drug discovery prompt engineering for biomedical knowledge extraction from literature (PubMed corpus RAG), target hypothesis generation, and clinical trial design assistance.

- **Northern England industrial AI hubs**:
- **Manchester (MediaCityUK, MIDAS)**: BBC R&D and ITV use structured prompting for automated subtitling, content summarisation, and editorial classification. Manchester's AI cluster includes Thoughtworks UK and Slalom, delivering prompt engineering consultancy for retail and logistics clients.
- **Leeds (Financial and Legal Tech)**: First Direct, Leeds Building Society, and Squire Patton Boggs use prompt-engineered LLMs for compliance classification, customer service automation, and contract review. The Digital Health Enterprise Zone explores clinical NLP prompting.
- **Sheffield (Advanced Manufacturing)**: Advanced Manufacturing Research Centre (AMRC) uses structured prompts for predictive maintenance report generation from sensor data and technical documentation retrieval for aerospace maintenance engineers.
- **Newcastle (NHS and Public Sector)**: Newcastle Upon Tyne Hospitals NHS Foundation Trust pilots few-shot prompting for discharge summary generation; collaboration with Newcastle University's Open Lab on explainable AI prompting for public-sector decision support.

- ### Future Directions (2026-2030)

- **Autonomous prompt optimisation at scale**: DSPy-style compilation extended to full agentic pipelines with 100+ modules; reinforcement learning over prompt spaces using execution feedback as reward signal; multi-objective optimisation balancing accuracy, latency, cost, and safety constraint satisfaction.

- **Formal prompt specification languages**: Type-checked, verifiable prompt schemas (beyond JSON schema) specifying preconditions, postconditions, and invariants, enabling static analysis of prompt safety properties before deployment. Early work on Prompt Flow (Microsoft), Outlines (dottxt-ai), and Guidance (Microsoft).

- **Multimodal prompt engineering**: Coordinated image+text+audio prompting for vision-language models (GPT-4V, Claude 3.5 Sonnet vision, Gemini 1.5); visual chain-of-thought (ViCoT) eliciting spatial reasoning through diagram annotation; video prompting for scene understanding.

- **Personalised and adaptive prompting**: System prompts dynamically adjusted from user interaction history, preference vectors, and task context. Privacy-preserving personalisation via federated prompt updates or on-device prefix caching without transmitting personal data to API providers.

- **Prompt supply-chain security**: Formal adversarial robustness testing frameworks; certified prompt injection resistance analogous to software security assurance levels; regulatory guidance on prompt disclosure for high-stakes AI systems (EU AI Act, UK AI Safety Institute).

- **Cognitive science integration**: Deeper understanding of transformer in-context learning mechanisms (via mechanistic interpretability) informing principled prompt design; human cognitive load studies informing prompt UX for non-expert practitioners.

- **Elimination of manual prompting for routine tasks**: For standardised business processes, automated prompt generation from task specifications will replace manual authoring; prompt engineering effort will concentrate on novel, high-stakes, and alignment-critical applications.

- ### Components and Architecture of a Production Prompt

- A production-grade prompt for enterprise LLM deployment comprises multiple architectural layers, each serving a distinct function:

- **System prompt layer** (operator-controlled, highest privilege):
- *Identity and persona section*: Specifies who the model is acting as, its name (if applicable), and core character traits. Example: "You are Aria, a friendly and knowledgeable customer support specialist for Acme Financial Services Ltd, authorised to discuss products available in England and Wales."
- *Capability declaration section*: Lists what the model can and cannot do. Explicit capability statements reduce out-of-scope responses. "You can: answer questions about current Acme products, help with account queries, explain terms and conditions. You cannot: provide regulated financial advice, access external systems, or discuss competitors."
- *Knowledge base injection*: Static documents (product catalogues, policy summaries, FAQ extracts) embedded directly in the system prompt for models with sufficient context windows. Enables consistent grounding without runtime retrieval.
- *Formatting and tone guidelines*: Output format specifications, length targets, response structure templates, tone register.
- *Safety and refusal instructions*: Topics to decline, escalation paths, specific phrase requirements for sensitive situations.
- *Tool and function definitions*: JSON or XML definitions of callable functions with parameter schemas and usage examples.

- **Few-shot demonstration layer** (can appear in system or user turn):
- 3-16 (input, expected output) pairs covering representative task instances.
- Format must exactly match expected production inputs and outputs.
- Examples are cached for cost efficiency when using Anthropic prompt caching or similar infrastructure.

- **User turn layer** (user-controlled, lower privilege):
- Current query or task description.
- Contextual information specific to this request (user name, session context, query-specific retrieved documents).
- Any user-level customisation permitted by the operator (language preference, verbosity level).

- **Assistant turn layer** (prefill/priming, optional):
- Partial assistant response provided as prefill to steer generation (Claude supports this via the `assistant` turn prefill feature).
- Example: Prefilling `{` forces the model to begin a JSON response; prefilling `## Summary\n` forces markdown output structure.

- **Tool response layer** (sandwiched between turns):
- Tool call results injected as structured tool-result messages.
- Prompt engineering for tool results includes format consistency, error message standardisation, and explicit attribution ("The following is the result of searching the web:").

- **Architecture principles**:
- **Privilege minimisation**: Grant only the capabilities needed for the use case in the system prompt; do not expose dangerous capabilities "in case they're needed".
- **Separation of concerns**: Keep persona, capability, knowledge, format, and safety instructions in clearly labelled subsections to prevent cross-contamination during editing.
- **Version control**: Treat prompts as code — commit to Git with semantic versioning, document changes in changelogs, maintain rollback capability.
- **Environment separation**: Maintain separate prompt versions for development, staging, and production, with stricter safety constraints in production.

- ### In-Context Learning: Mathematical Foundations

- In-context learning (ICL) enables LLMs to adapt to new tasks from k examples without gradient updates. The formal analysis situates ICL within statistical learning theory:

- **Bayesian inference framing** (Xie et al. 2022): Model pre-training on a mixture of tasks induces a prior P(concept) over latent task functions. Given demonstrations D = {(x₁,y₁),...,(xₖ,yₖ)}, the model computes an implicit posterior P(concept | D) and generates responses consistent with the highest-probability concept. This predicts: (1) performance improves with more demonstrations (posterior concentrates); (2) label values affect performance (they signal concept identity, not just input-output mapping); (3) out-of-distribution demonstrations degrade performance (concept posterior misaligned).

- **Gradient descent interpretation** (Akyürek et al. 2022; Dai et al. 2022): Linear attention layers can implement one step of gradient descent on a linear regression problem from demonstrations, effectively performing implicit fine-tuning in forward pass. Transformer attention computes key-value pairs analogous to (input feature, gradient step) that update internal query representations. This "in-context fine-tuning" interpretation predicts sensitivity to demonstration quality and suggests that larger models (more layers) perform more gradient steps implicitly, explaining scale-dependent emergence.

- **Information-theoretic bounds** (Wies et al. 2023): For a task drawn from a concept class C with VC-dimension d, ICL with k demonstrations achieves error O(√(d/k)) with high probability, matching classical PAC bounds for k-labelled supervised learning. This formalises the intuition that ICL effectiveness grows with demonstration count and model capacity.

- **Attention pattern analysis**: Mechanistic interpretability studies (Elhage et al. 2021; Olsson et al. 2022 *In-Context Learning and Induction Heads*) identify "induction heads" — attention heads that implement a pattern-copying operation: if token A is followed by token B in the context, then when A appears again later, induction heads attend strongly to the earlier A and copy B. This mechanism is proposed as the basic computational primitive of ICL, emerging reliably at model scales around 500M-1B parameters.

- ### Prompt Engineering Across the Stack: From API to Edge

- Prompt engineering requirements vary dramatically across deployment contexts:

- **Cloud API deployments** (GPT-4o, Claude 3.5, Gemini 1.5 via REST API):
- Full context windows (128K-1M tokens) enable large knowledge bases, extensive few-shot libraries, and long conversation histories.
- Prompt caching infrastructure available (Anthropic, OpenAI) for 90% cost reduction on stable prefixes.
- Function calling / tool use with reliable schema conformance.
- Latency in 1-10 seconds range for complex prompts; streaming available for progressive display.
- Cost: $1-15/MTok input depending on model; prompt engineering for cost efficiency (cache utilisation, token minimisation) directly affects unit economics.

- **Self-hosted open-source deployments** (Llama 3.1 70B, Mistral 8×22B, Phi-3 on VLLM/Ollama):
- Instruction format tokens (`<|begin_of_text|>`, `[INST]`) are model-specific and must match training format exactly.
- Quantisation (GGUF Q4, Q8) affects instruction following; Q8 quantised models approach BF16 quality; Q4 degrades on complex multi-constraint tasks.
- Context windows typically 4K-32K; longer prompts require careful priority ordering since attention degrades on very long contexts for smaller models.
- No built-in JSON schema enforcement; Outlines or llama.cpp grammar mode needed for structured output guarantee.
- Latency and cost tradeoffs: on-premise A100 clusters yield 20-200 tokens/second; optimised for bulk inference with batching.

- **Edge and on-device deployments** (Phi-3-mini 3.8B, Gemma-2 2B, Llama 3.2 1B/3B on ARM Cortex/Apple Neural Engine):
- Context window constrained to 2K-8K tokens by memory budget; prompt must be highly compressed.
- Prompt compression techniques: remove examples entirely, replace verbose instructions with terse codes the fine-tuned model recognises, use abbreviations.
- Instruction-following fidelity significantly lower than large models; simpler, more explicit prompts with fewer constraints outperform complex multi-requirement prompts.
- ARM Holdings (Cambridge) Ethos-U85 NPU accelerates 1-4B parameter models for ~10 tokens/second on-device; this enables real-time inference for short prompts (< 512 tokens).
- Privacy-preserving prompting: sensitive data never leaves device; system prompt encodes task logic without referencing server-side capabilities.

- **Batch and asynchronous inference** (OpenAI Batch API, Anthropic Message Batches):
- Prompt engineering for batch workloads prioritises consistency over interactivity; prompts are optimised for accuracy rather than latency.
- Cost 50% lower than synchronous API; 24-hour completion SLA.
- Document processing, dataset annotation, and bulk classification are primary use cases.

- ### Prompt Engineering Practitioner Heuristics and Checklist

- A consolidated practitioner checklist synthesises guidance from Anthropic, OpenAI, and community research:
- **Task definition**: State the task precisely in a single imperative sentence. Avoid compound instructions in one sentence; use numbered lists for multi-step tasks.
- **Output format specification**: Define exact output format (JSON, markdown, plain text, XML), field names, field types, and example. For structured data, use JSON schema or Pydantic model examples.
- **Context injection**: Provide all necessary reference material explicitly — do not assume the model recalls prior conversation details. Use XML tags (`<document>`, `<context>`) to delimit injected content from instructions.
- **Examples (few-shot)**: For classification or extraction tasks with consistent format, provide 3-8 examples. Ensure examples are diverse and representative, not all easy/typical cases. Format examples identically to the expected input/output.
- **Persona specification**: Assign a specific, expert persona relevant to the task (not generic "you are an AI assistant"). Specificity reduces output variance on professional tasks.
- **Constraint enumeration**: List what the model should NOT do as well as what it should. Negative constraints ("Do not include citations unless the document contains them") reduce systematic hallucination patterns.
- **Reasoning elicitation**: For multi-step or ambiguous tasks, include a CoT trigger or provide a scratchpad section: `<thinking>` for intermediate reasoning, `<output>` for final answer.
- **Tone and register**: Specify tone for user-facing content (formal, conversational, technical) and length (1-2 sentences, 300 words, bullet list of 5 items).
- **Evaluation and iteration**: Test prompts on a golden set of 20-50 representative inputs covering edge cases. Track failure modes in a changelog. Version-control prompts with semantic versioning.
- **Caching strategy**: For prompts with stable system-prompt prefixes exceeding 1K tokens, enable prompt caching (Anthropic) or system message caching (OpenAI) to reduce cost by 70-90%.
- **Safety constraints**: In system prompts for user-facing deployments, specify topics the model should refuse, how to handle ambiguous requests, and escalation paths (e.g., "If the user asks for medical advice, recommend they consult a qualified clinician").

- ### Prompt Engineering for Agents and Multi-Step Systems

- Agentic deployments (see [[Agent Frameworks]], [[CLI Multi-Agent Systems]]) require specialised prompt engineering beyond single-turn task completion.

- **Tool inventory prompting**: System prompts must enumerate available tools with precise natural language descriptions, parameter schemas, and usage examples. Tool descriptions are crucial: vague descriptions cause tool misselection; overly detailed descriptions inflate context. Optimal descriptions are 50-100 words per tool, specifying when to use and when not to use each tool.

- **Working memory and scratchpad**: Agents benefit from explicit scratchpad sections in their prompt where intermediate reasoning, partial results, and decision rationale are accumulated across steps. This externalises working memory into the context, enabling later steps to reference earlier computations without relying on attention over long histories.

- **Error handling and recovery prompting**: Agent system prompts must specify error recovery protocols: what to do when a tool call fails, when retrieved information contradicts prior context, or when the user's request is ambiguous. Failure to specify these leads to agent loops, hallucinated tool calls, or graceless crashes.

- **Multi-agent coordination**: In systems with specialised sub-agents (see [[CLI Multi-Agent Systems]]), orchestrator prompts specify routing logic (which agent handles which task class), communication format between agents (JSON message schemas), and conflict resolution protocols (when two agents produce contradictory outputs). Inter-agent prompt engineering is an emerging specialisation.

- **ReAct agent prompt structure** (canonical 2024 pattern):

System: You are a research assistant with access to the following tools:

search(query: str) → str: Search the web for current information
calculator(expression: str) → float: Evaluate arithmetic expressions
python(code: str) → str: Execute Python code snippets

Use the following format: Thought: [reasoning about what to do next] Action: tool_name Observation: [tool output] … (repeat Thought/Action/Observation as needed) Final Answer: [answer to the original question]
Context window management: Long-running agents accumulate context rapidly. Prompt engineering strategies include: (1) context summarisation — instruct the model to summarise completed sub-tasks into a compact state representation before proceeding; (2) selective context pruning — rank context segments by relevance and drop low-relevance history; (3) persistent external memory — store completed task outputs in a retrieval store and fetch on demand via tool calls, keeping the active context focused.

Multimodal Prompting

As frontier models acquire vision (GPT-4V, Claude 3.5 Sonnet vision, Gemini 1.5 Pro) and audio (GPT-4o Realtime) capabilities, prompt engineering extends to multimodal contexts.
Vision prompting: Image-text interleaving (providing an image followed by text instructions) requires specific strategies:
- Anchoring: Reference specific image regions by spatial description (“in the top-right quadrant”, “the label on the red bottle”) since models lack coordinate-based reference.
- Detail requests: Ask for verbose, structured descriptions of image content before analysis tasks to surface relevant visual details the model may otherwise skip.
- Chart/diagram prompting: For data visualisation analysis, prompt the model to first extract all axis labels, data series names, and approximate numerical values, then reason about trends — reducing hallucination of non-existent trends.
- VisualisationOfThought: Yao et al. (2024, arXiv:2404.03622) demonstrate that asking models to describe spatial reasoning steps in text before answering spatial queries (matching the Attention pattern of visual chain-of-thought) improves spatial reasoning benchmarks by 15-25%.
Audio and voice prompting: OpenAI Realtime API and ElevenLabs voice AI require prompts adapted to acoustic constraints:
- Specify turn-taking policy: sensitivity to user interruption (barge-in), silence duration before model speaks, latency tolerance.
- Use phonetic spelling for technical terms that TTS systems mispronounce: “BERT (burt)”, “LLaMA (LAH-muh)“.
- Write in spoken register: short sentences, no parenthetical asides, no markdown, numbers spelled out (“three point seven” not “3.7”).
- Specify voice persona (warm/professional/neutral) and geographic English variant (UK/US) for accent consistency.
Multimodal agent prompting: Combining vision, text, and tool use (e.g., Claude’s computer-use capability, GPT-4V + function calling) requires prompts that specify visual attention protocols (“examine the screen carefully before clicking”), action verification (“after each action, describe the current state of the screen”), and error detection (“if the expected UI element is not visible, report the current state before trying an alternative approach”).

Prompt Evaluation and Benchmarking

Robust prompt engineering requires systematic evaluation, not just subjective impression. Key methodologies:
LLM-as-judge (Zheng et al. 2023, MT-Bench): A strong model (GPT-4, Claude 3.5) evaluates response quality on a rubric using a structured evaluation prompt. Achieves 80-90% agreement with human evaluators on open-ended quality assessment, enabling high-throughput automated evaluation. Prompt engineering for the judge model (rubric specification, calibration examples, scale anchoring) critically affects reliability.
Diverse panel of judges (Replacing Judges with Juries, arXiv:2404.18796): Multiple heterogeneous judge models vote on response quality, reducing individual judge bias. Particularly valuable when evaluating responses about topics where a single judge model may have systematic blind spots.
Golden test sets: Curated 50-500 question/answer pairs with human-verified ground truth, covering (a) typical cases, (b) adversarial edge cases, (c) known failure modes of prior prompt versions. Prompt changes must not regress accuracy on golden sets by more than 2-3%.
Prompt A/B testing infrastructure: PromptLayer, LangSmith, W&B Prompts, and Humanloop provide production logging of prompt versions, input-output pairs, latency, and cost, enabling statistical comparison of prompt variants on real traffic. Minimum sample sizes of 200-500 per variant are needed for reliable 5% accuracy difference detection.
Calibration evaluation: For classification prompts, measure whether stated confidence levels (when elicited via “How confident are you?”) are calibrated against actual accuracy. Well-calibrated prompts enable reliable uncertainty-based routing (escalate to human when model confidence < threshold).
Qualitative failure mode analysis: Beyond metric-driven evaluation, manual review of 20-50 failure cases per iteration identifies systematic failure categories (instruction misinterpretation, hallucination, format violations, safety refusals on valid requests) guiding targeted prompt revisions.