Labs / Lab 003
Does a design skill beat using no skill at all?
2 min read Claude Code workflows · Evals Leer en español
Example lab with invented data. It exists only to preview the design and must not be published.
- Question
- Does a design skill beat using no skill at all?
- Verdict
- Inconclusive. The gap between skills was smaller than the gap between two runs of the same skill.
- How it was measured
- 6 skills and a no-skill control, 3 runs each, blind vote by 40 people.
Hypothesis
Design skills promise better-looking pages from the same prompt. If that holds, a page built with a skill should win a blind vote against a page built with none, and the ranking of skills should stay roughly the same from one run to the next.
Setup
- One brief: a landing page for a Saturday workshop in Lima, written in Spanish.
- Seven conditions: six public design skills and a control with no skill.
- Three runs per condition, same model and settings, fresh session each time.
- Forty people voted blind for the best page in each of three rounds.
Results
| Condition | Mean | Range | Spread (pts) |
|---|---|---|---|
| Skill A | 17% | 9–24% | 15 |
| Skill B | 16% | 11–22% | 11 |
| Skill C | 15% | 6–25% | 19 |
| No skill | 14% | 10–19% | 9 |
| Skill D | 14% | 8–20% | 12 |
| Skill E | 13% | 9–18% | 9 |
| Skill F | 11% | 5–17% | 12 |
The best and worst means are 6 points apart. The average spread between runs of the same condition is 12 points. The control sits in the middle of the table.
What this means
With three runs I cannot tell these skills apart, and I cannot say that any of them beats using none. A single run would have produced a confident ranking, and a different one each time.
Limits
- One brief and one model. A different task may separate the skills.
- Three runs is a small sample; ten per condition would narrow the ranges.
- Voters were acquaintances, not the audience of the page.
Reproduce it
git clone https://github.com/renatogm24/lab-003-design-skills
cd lab-003-design-skills
make run RUNS=3 # builds the 21 pages
make vote # serves the blind voting page