Red Flags in GEO Research: Common Analytical Errors and How to Detect Them

1. Introduction

Every research field develops characteristic error patterns -- analytical shortcuts that produce impressive-sounding claims from insufficient evidence. In GEO research, these errors are amplified by the novelty of the domain (few established benchmarks), the commercial pressure for actionable conclusions (clients want certainty), and the complexity of the systems being studied (LLMs are opaque by nature).

Identifying these error patterns serves two purposes. For researchers, it provides a self-check against the most common analytical traps. For practitioners consuming GEO research, it provides a filter for evaluating which claims warrant action and which require additional evidence. The goal is not to suppress research output but to ensure that claims are stated at the confidence level the evidence supports.

2. The Seven Red Flags

2.1 Red Flag 1: Multi-Variable Group Comparisons

Detection pattern: Two groups are compared that differ on multiple characteristics simultaneously, but the conclusion attributes the outcome difference to a single variable.

Gate violated: Gate 2 (Confound Check).

Worked example: A comparison between new sites (with heavy schema markup, zero domain authority, no training data presence) and established sites (with light schema, high authority, extensive training data) concludes that schema markup inversely correlates with citation at a 0.6x ratio. The conclusion attributes the outcome to schema, ignoring the 6+ other variables that differ simultaneously.

Constructive alternative: "In this sample, sites with heavier schema implementation were less likely to be cited. However, these sites also differed from cited sites on domain age, authority, training data presence, review count, backlink profile, and brand recognition. The observed difference cannot be attributed to schema specifically."

2.2 Red Flag 2: LLM Self-Report as Mechanism Evidence

Detection pattern: An LLM is asked to explain its own citation behaviour, and the explanation is treated as ground truth about how the system actually works.

Gate violated: Gate 7 (Mechanism).

Worked example: An LLM reports that it weighs methodology sections at 8.5 out of 10 for citation decisions. This score is then cited as evidence that methodology sections have a quantified impact on citation outcomes.

Constructive alternative: "The LLM self-reports that methodology sections are important to its citation decisions. This constitutes an informed hypothesis about its own behaviour and generates a testable prediction: pages with methodology sections should receive higher citation rates than equivalent pages without them. This prediction requires behavioural testing to confirm."

2.3 Red Flag 3: Single Observation as Proof

Detection pattern: A single case (one site cited, one site not cited) is used to prove or disprove a probabilistic claim about a factor's effect.

Gate violated: Gate 4 (Counterfactual).

Worked example: A single new site with FAQ schema is not cited, leading to the conclusion that FAQ schema harms citation.

Constructive alternative: "This case demonstrates that FAQ schema is not sufficient for citation. Whether it helps, harms, or is irrelevant requires controlled comparison across multiple sites with and without the feature."

2.4 Red Flag 4: Causation from Correlation

Detection pattern: An observed association between a factor and an outcome is described using causal language (causes, drives, produces, results in).

Gate violated: Gate 1 (Logical Form) -- specifically, affirming the consequent.

Worked example: Cited sites are observed to have more images than uncited sites. The conclusion states that images drive citation.

Constructive alternative: "Image count is positively associated with citation in this sample. This association may reflect a causal relationship, or it may reflect that established sites with extensive portfolios also have the authority and training data presence that independently drives citation."

2.5 Red Flag 5: Extreme Ratios from Small Samples

Detection pattern: Very large ratios (10x, 30x) are reported from comparisons of small, uncontrolled groups.

Gate violated: Gate 2 (Confound Check).

Worked example: A 30.7x correlation between image count and citation is computed from a comparison of 4 cited sites versus 4 uncited sites, where the groups differ on 7+ variables.

Constructive alternative: "The mean image count of cited sites was 30.7 times that of uncited sites in this small sample (N=8). This descriptive difference is confounded by simultaneous differences in domain age, authority, and training data presence. The ratio should not be interpreted as the magnitude of image count's independent effect on citation."

2.6 Red Flag 6: Null Results from Confounded Comparisons

Detection pattern: A factor shows no observed effect in a confounded comparison, and the conclusion states that the factor has zero impact.

Gate violated: Gate 2 (Confound Check).

Worked example: A confounded comparison finds no difference in citation rates between sites with and without a particular feature, and concludes the feature has no effect. But the with-feature group differs from the without-feature group on multiple other dimensions that could mask a real effect.

Constructive alternative: "No association between the feature and citation was observed in this sample. However, the comparison is confounded by multiple simultaneous variable differences, meaning both a positive effect and a negative effect could be masked by other factors."

2.7 Red Flag 7: Prohibited Language for Evidence Level

Detection pattern: Words like "proves," "demonstrates," or "confirms" are used for Level 1-3 evidence. Generalised causal claims are made for Level 4 evidence.

Gate violated: Items 8-9 of the Claim Audit Checklist (Evidence Level and Language Appropriateness).

Worked example: An observational study (Level 3) states that its findings "demonstrate" a causal relationship between content structure and citation outcomes.

Constructive alternative: Use language matched to the evidence level. Level 3: "These findings suggest an association between..." Level 4: "Under these specific conditions, X is associated with Y..."

3. Detection Quick Reference

4. Conclusions

We document seven common analytical errors in GEO research, map each to the logic gate it violates, and propose constructive alternatives for stating findings appropriately. The red flags are designed as a practical detection tool for researchers conducting self-review, reviewers evaluating submissions, and practitioners assessing GEO claims before making strategic decisions. Each red flag has a corresponding constructive alternative that preserves the informational value of the finding while correcting its confidence level. The goal is not to suppress research but to ensure that the GEO field develops on a foundation of appropriately qualified evidence.

Confidence: Framework paper. The red flags are documented from concrete examples within the SIGI research programme and reflect established principles of research methodology applied to the GEO context.