Ji-woo Park

Write Survey Questions That Don't Lead the Respondent

Diagnose bias in draft survey questions, rewrite them with balanced scales, and get a short survey whose answers you can actually trust.

At a glance

Best for: Managers, founders, researchers and HR teams writing a quick internal or customer survey who want answers that aren't shaped by the wording.
Tested on: Claude · Opus 5.5
You fill in: decisionaudiencedraft
You get: Diagnosis "How much do you love...?" Leading and loaded: it assumes a positive feeling. Rewrite: "Overall, how useful is the current all-hands to… (full result below)

Prompt

Help me write a short survey that gives me answers I can trust.

What I need to decide with the results: {{decision}}
Who will answer it and how: {{audience}}
My draft questions (they may be biased; fix them): {{draft}}

1. For each draft question, name the problem (leading, loaded, double-barrelled, vague time frame, unbalanced scale, assumes behaviour, etc.) and rewrite it. If a question doesn't help the decision, cut it and say why.

1. Add any question the decision needs, but keep the whole survey to 8 questions or fewer, answerable in under 3 minutes.

1. Give the final survey in order, with exact answer options. Scales must be balanced and labelled at every point. Include "Don't know" or "Not applicable" where honest respondents would need it.

1. Explain how question order could bias answers and how you've handled it.

1. Name one thing this survey cannot tell me, however well it's written.

Fill in

{{decision}}: The decision the survey results will informe.g. Move the weekly all-hands, or replace it with an async video
{{audience}}: Who answers, how many, and through what channele.g. 85 employees, anonymous Google Form, shared on Slack
{{draft}}: Your draft questions, numberede.g. 1. How much do you love our all-hands meetings? ...

Example input

decision: Whether to move our weekly all-hands meeting from Monday 9am to Thursday 4pm, or replace it with an async recorded video.
audience: 85 employees of a hybrid software company; anonymous Google Form sent via Slack.
draft:
1. How much do you love our all-hands meetings?
2. Don't you think Monday mornings are a bad time for a meeting?
3. Would you prefer a shorter, more focused meeting at a better time?
4. How often do you attend and find it useful?
5. Should we switch to async video like other modern companies?

2700

Generated Outputs (1)

2 hours ago

Claude

Opus 5.5

2 hours ago

Generated Output

1. Diagnosis

1. "How much do you love...?" Leading and loaded: it assumes a positive feeling. Rewrite: "Overall, how useful is the current all-hands to your work?"

1. "Don't you think Monday mornings are a bad time...?" Leading: it invites agreement. Rewrite as a neutral rating of each time slot (Q3).

1. "A shorter, more focused meeting at a better time?" Double-barrelled and loaded: three changes in one, and nobody refuses "better". Split it into length (Q5) and time (Q3).

1. "How often do you attend and find it useful?" Double-barrelled. Split it into attendance (Q1) and usefulness (Q2).

1. "Like other modern companies?" Loaded with social pressure. Rewrite: Q4 and Q6.

2 and 3. Final survey (7 questions, about 2 minutes)

1. In the last 4 weeks, how many all-hands meetings did you attend live? 0 / 1 / 2 / 3 / 4

1. How useful is the all-hands to your work? Not at all / Slightly / Moderately / Very / Extremely / Don't know

1. How well would each time work for you? Monday 9am / Thursday 4pm, each rated: Very poorly / Poorly / Neither well nor poorly / Well / Very well

1. If the all-hands were a recorded video you watched when convenient, how likely is it that you'd watch most weeks? Very unlikely / Unlikely / Neither / Likely / Very likely

1. What length would you prefer? 15 min / 30 min / 45 min / No preference

1. Which format would you choose? Live on Monday 9am / Live on Thursday 4pm / Recorded video / No preference (options shown in random order)

1. What, if anything, would make the all-hands more useful to you? (optional, free text)

4. Order effects
Attendance and usefulness come first, so later questions don't colour how people rate the meeting as it is now. Q3 and Q4 rate the options separately before Q6 forces a choice, so no single option gets framed first. Randomising Q6's options stops the first-listed choice gaining votes.

5. What it can't tell you
Whether people will actually watch a video. Stated intent (Q4) usually overestimates behaviour, so if you pick async, track real view counts for a month before deciding for good.

Why it works

Naming the bias type for each question teaches you to spot it next time, not just fixes this survey. "Balanced and labelled at every point" prevents the lopsided scales (three positive options, one negative) that quietly push results. Requiring a "Don't know" or "Not applicable" option stops people inventing opinions. Asking about order effects catches a subtler bias: asking about problems first colours later answers. The final limitation is a useful check on what the survey can't tell you.

When not to use it

A survey tells you what people say, not what they do; for behaviour, use attendance or usage data. For high-stakes research (academic studies, employee engagement benchmarks), use validated question banks and a proper pilot. With very small groups (under 15 people), anonymous surveys often aren't truly anonymous, so talk to people instead.

Comments (0)

No comments yet. Be the first to share your thoughts!