# Methodology Checks

Research-validity starter kit for empirical, generalized, benchmark, medical, policy, survey, and causal
claims. Distilled from the Critical Compass MCP/DB during v2 skill improvement.

## Trigger

Run this check when the text makes any of these claims:

- empirical: says what is true in the world
- generalized: extends from a sample to a population
- causal: says X caused Y or will cause Y
- comparative: says one option is better, safer, more effective, or more harmful
- benchmark: treats a metric as proof of ability or quality
- medical/health: makes claims about safety, risk, diagnosis, treatment, or benefit

## Research Validity Concepts

| Check | ID | Ask | Common signal | What reduces concern | Source confidence |
|---|---|---|---|---|---|
| Construct validity | ct:methodology:validity:construct-validity | Does the measure actually capture the concept claimed? | Proxy, score, benchmark, scale, "measure of." | Validated instrument, bounded interpretation, converging measures. | medium |
| Measurement validity | ct:methodology:validity:measurement-validity | Is the observed variable accurate and interpretable enough? | Endpoint, coded outcome, self-report, noisy data. | Reliability checks, error bounds, transparent coding. | medium |
| Internal validity | ct:methodology:validity:internal-validity | Does the design support the causal or comparative inference? | Caused, effect, treatment, intervention, control group. | Randomization, controls, causal design, confounder handling. | medium |
| External validity | ct:methodology:validity:external-validity | How far can the finding travel beyond this sample and setting? | All users, population, generalizes, broader use. | Transportability evidence, subgroup analysis, bounded scope. | medium |
| Ecological validity | ct:methodology:validity:ecological-validity | Does the study resemble the real-world setting where it will be used? | Lab, simulation, benchmark, controlled setting. | Field testing or specific missing-condition analysis. | medium |
| Replicability | ct:methodology:reproducibility:replicability | Would an independent team expect the central finding to survive? | One study, no replication, unavailable protocol/code. | Independent replication, materials, code, protocol. | medium |
| Preregistration | ct:methodology:reproducibility:preregistration | Were hypotheses, outcomes, and analyses recorded before results? | Post hoc, exploratory, many outcomes. | Registered report, timestamped protocol, transparent exploratory label. | medium |
| Statistical power | ct:methodology:statistics:statistical-power | Could the design detect an effect of practical importance? | Small sample, subgroup claims, no effect finding. | Power analysis, adequate sample, uncertainty intervals. | medium |

## Common Evidence-Pool Risks

| Risk | ID | Ask |
|---|---|---|
| P-hacking | ct:biases:methodology:p-hacking | Were many analyses tried until something became significant? |
| HARKing | ct:biases:methodology:harking | Were hypotheses written after seeing the results? |
| Publication bias | ct:biases:methodology:publication-bias | Are null, failed, or negative results missing from the visible evidence? |
| WEIRD bias | ct:biases:methodology:weird-bias | Is a narrow Western/educated/industrialized/rich/democratic sample treated as universal? |
| Attrition bias | ct:biases:methodology:attrition-bias | Did dropouts differ from completers? |
| Nonresponse bias | ct:biases:methodology:nonresponse-bias | Did missing respondents differ from respondents? |
| Demand characteristics | ct:biases:methodology:demand-characteristics | Did the study cue participants toward expected behavior? |

## Method Check Template

Use the smallest useful subset:

- **Measure:** What was measured, and does it match the claim?
- **Sample:** Who or what was studied, and who is being generalized to?
- **Design:** Does the design support causal, comparative, or only descriptive claims?
- **Magnitude:** Are effect size, uncertainty, and practical significance visible?
- **Transfer:** Does the setting match real-world use?
- **Evidence pool:** Are null results, replications, and conflicts of interest visible?
