Prompt Optimization for LLM Stability: 5-Group Comparison Experiment
Key Findings
Among 5 prompt variants, the hybrid strategy achieved the highest mean (88.2), but reduced range by only 7% (14→13). Structured prompts and guideline embedding showed no significant stability improvement.
1. Hybrid strategy most effective: Structured + guideline + self-check combination improved mean by 1.6 points, range by 1 point
2. Single strategies limited: Using structured, guideline embedding, or self-check alone showed no significant stability improvement
3. Concise version performed worse: Reduced cognitive load prompt decreased mean by 2.3 points and increased range by 3 points
Research Background & Questions
In previous experiments, DeepSeek R1 with deep thinking mode achieved a mean of 86.6, but still had a range of 14. For clinical decision-making scenarios, even with high mean scores, output variance remains a concern.
Research Questions
- Can different prompt designs reduce R1's output variance?
- Is structured Chain-of-Thought effective?
- Can embedding key guideline snippets improve stability?
- Can self-check mechanisms reduce erroneous outputs?
- Is a hybrid strategy (combining multiple methods) superior to single strategies?
Experimental Design
| Group | Prompt Strategy | Key Features |
|---|---|---|
| Control | Standard prompt | No special design, baseline version |
| Exp 1 | Structured prompt | Forced 7-step analysis |
| Exp 2 | Guideline embedding | Embedded 2026 CBCS key snippets |
| Exp 3 | Self-check mechanism | Require model to self-verify conclusions |
| Exp 4 | Hybrid strategy | Structured + guideline + self-check combination |
| Exp 5 | Concise version | Reduced cognitive load, focused on core questions |
20 runs per group, 120 total runs. Unified 16-item Rubric scoring (max 100 points).
Methodology
- Case: 60-year-old female, right breast lumpectomy + sentinel lymph node biopsy, invasive carcinoma, max diameter 0.6cm, with 3 clusters of stromal invasion (approximately 0.05cm, 0.06cm, 0.2cm), ER/PR 3+, HER2-, Ki-67 5%, discharged with incorrect TisN0M0 staging
- Input: Standard text prompt (not screenshot)
- Model settings: DeepSeek R1 deep thinking + web search (fixed)
- Only variable: Prompt design (5 different strategies)
- Scoring: clinical-benchmark/framework 16-item/100 Rubric
- Sample size: 20 valid runs per group
Results by Group
📊 Control (Standard Prompt)
Range 14
SD 4.2
AMBER
GREEN
Standard prompt with no special design. Serves as baseline for comparison.
📐 Experiment 1 (Structured Prompt)
Range 15
SD 4.5
AMBER
GREEN
Forced 7-step case analysis. Mean slightly improved (+0.5), but range actually increased by 1 point.
📚 Experiment 2 (Guideline Embedding)
Range 14
SD 4.3
AMBER
GREEN
Embedded 2026 CBCS breast cancer microinvasion management要点. No significant improvement in stability or accuracy.
🔍 Experiment 3 (Self-Check Mechanism)
Range 13
SD 4.0
AMBER
GREEN
Require model to self-verify conclusions at the end. Mean improved +0.9, range reduced by 1. Best single strategy.
🏆 Experiment 4 (Hybrid Strategy: Structured + Guideline + Self-Check)
Range 13
SD 3.8
AMBER
GREEN
Combined prompt design. Highest mean (+1.6), smallest range (-1). Optimal solution in this experiment.
⚡ Experiment 5 (Concise Version: Reduced Cognitive Load)
Range 17
SD 5.1
RED
GREEN
Simplified case information, focused on core questions. Results反而 worse: mean decreased 2.3 points, range increased 3 points.
5-Group Comparison Overview
| Group | Mean | Range | SD | vs Control |
|---|---|---|---|---|
| Exp 4 (Hybrid) | 88.2 | 13 | 3.8 | +1.6 / -1 |
| Exp 3 (Self-Check) | 87.5 | 13 | 4.0 | +0.9 / -1 |
| Exp 1 (Structured) | 87.1 | 15 | 4.5 | +0.5 / +1 |
| Exp 2 (Guideline) | 86.8 | 14 | 4.3 | +0.2 / 0 |
| Control (Standard) | 86.6 | 14 | 4.2 | — |
| Exp 5 (Concise) | 84.3 | 17 | 5.1 | -2.3 / +3 |
Mean Comparison (Visualization)
Range Comparison (Visualization)
Key Findings & Interpretations
✅ Effective Strategies
Mean +1.6 points, range -1 point. Combining structured chain-of-thought, guideline embedding, and self-check mechanism can systematically improve output accuracy and stability.
Mean +0.9 points, range -1 point. Requiring model to self-verify conclusions can reduce low-level errors and omissions.
⚠️ Limited Effect Strategies
❌ Counterproductive Strategy
Conclusion & Implications
Hybrid strategy is optimal, but improvement is insufficient to change "unstable" rating (range still > 8 points).
Implications for Clinical Scenarios
- Prompt design has value, but with diminishing returns: Hybrid strategy improves mean by 1.6 points, but range improvement is limited
- Self-check mechanism is key component: Using self-check alone achieves 0.9 point improvement and 1 point range reduction
- Avoid over-simplification: Clinical decisions require complete information; concise version performed反而 worse
- Structured chain-of-thought needs self-check: Pure step decomposition has limited effect; combined strategy is better
Next Research Directions
- Test larger sample sizes (50+ runs per group) for more stable statistical results
- Explore other stability optimization methods (e.g., temperature adjustment, retry mechanisms)
- Compare different models' responses to prompt optimization
- Study generalization of prompt optimization to other clinical scenarios
Limitations
- Sample size limited (20 runs per group), statistical power may be insufficient
- Only tested on single case, conclusions cannot generalize to other case types
- Scoring based on keyword coverage, may偏差 from human scoring
- DeepSeek model version changes not controlled (may update over time)
Frequently Asked Questions
Why is prompt optimization效果 limited?
DeepSeek R1's deep thinking mode already has good stability. Prompt optimization mainly affects output "structure" and "completeness", but has limited impact on the model's underlying reasoning ability. When the model can already correctly understand the problem, further prompt optimization has diminishing marginal benefits.
Why is self-check mechanism effective?
Self-check mechanism forces the model to perform secondary verification before output, which can发现和correct low-level errors (e.g., missing key decision points, logical contradictions). This "reflection" process is similar to a human doctor's "double-check", significantly improving output reliability.
Why did the concise version perform反而 worse?
Clinical decision-making scenarios需要 complete case information as context. The concise version, while reducing cognitive load, also lost key information (e.g.,具体 distribution of microinvasion, specific intensity of ER/PR), leading to more conservative or inaccurate model judgments.
Can this experiment be generalized to other models?
This experiment only tested DeepSeek R1. Different models may respond differently to prompt optimization. For example, GLM-5.2 has a larger range (35 points) and may需要 stronger prompt optimization measures. Cross-model comparison studies are needed in the future.