Introduction

When users present LLMs with ranked lists of service providers and ask for an assessment, the model must evaluate both the entities in the list and the plausibility of their ordering. This evaluation reveals biases in the model’s internal entity knowledge: which entities it recognises, what ordering it expects, and what contextual constraints it applies to locally-scoped rankings.

This study uses a position probe that systematically manipulates the composition and ordering of a ranked list of service providers within a specific geographic market. All entity names have been anonymised (Provider Alpha through Provider Epsilon for known entities, Provider Omega for unknown entities, and Provider Foreign for non-local entities) to prevent identification of specific businesses.

Methodology

The position probe (Probe ID: PP01–PP06) presented the LLM with fabricated rankings of Country A design agencies and asked it to assess the ranking’s reliability. Six variations tested different manipulations: PP01 used expected ordering with known entities; PP02 reversed the order; PP03 placed a lesser-known real entity at position 1; PP04 placed a fictitious entity at position 1; PP05 shuffled known entities; and PP06 placed a well-known non-local entity at position 1. All tests were conducted on 24 March 2026 using a ChatGPT-class LLM.

Results

Test IDManipulationSentimentPos.Neg.Neu.WordsElapsed (s)
PP01Expected order, known entitiespositive2001968.3
PP02Reversed orderneutral1101908.3
PP03Lesser-known entity at #1neutral0001907.6
PP04Fictitious entity at #1neutral0002008.4
PP05Shuffled known entitiesneutral0001938.5
PP06Non-local entity at #1negative0102029.2

Threshold Transitions

#FromToTrigger
1positiveneutralReversed ordering of known entities (PP02)
2neutralnegativeNon-local entity at position 1 in local ranking (PP06)

Aggregate Statistics

MetricValue
Total variations6
Sentiment distribution1 positive, 4 neutral, 1 negative
Mean word count195.2
Word count range190–202
Mean elapsed time8.38s
Mean tokens in59.2
Mean tokens out277.5
Under controlled conditions, the LLM displays primacy bias favouring the first-listed entity and positively endorses rankings that match its internal expectations. It rejects geographic mismatches in locally-scoped rankings with negative sentiment, while treating unfamiliar entities with neutral questioning rather than active rejection.

Discussion

The results reveal a clear hierarchy of LLM responses to ranking manipulations. Expected orderings receive positive endorsement, suggesting the model has internal expectations about entity ordering within specific geographic markets. Any deviation from this expected ordering — whether reversal, shuffling, or insertion of unknown entities — reduces sentiment to neutral. Only geographic mismatch triggers active negative rejection.

The stability of word counts across variations (range of only 12 words: 190–202) is noteworthy. Unlike other probes in our experimental programme where sentiment differences correlated with response length differences, the position probe elicited consistent evaluative effort regardless of the manipulation applied. This suggests that ranking assessment is a fixed-complexity evaluation where the model applies approximately the same analytical depth to any presented ranking.

For generative engine optimisation, these findings indicate that appearing in position 1 of AI-generated or AI-evaluated rankings carries a distinct advantage, as the model applies positive sentiment to expected first-position entities. The geographic mismatch rejection also has practical implications: entities appearing in geographic market rankings where they are not locally headquartered may face active negative assessment rather than mere neutral evaluation.

Limitations

Only 6 variations were tested, limiting the granularity of the analysis. The probe used a single geographic market context. The boundary between “lesser-known real entity” (PP03) and “fictitious entity” (PP04) depends on the model’s training data coverage and may not be stable over time. Single-model, single-time-point limitations apply.

Conclusions

The LLM maintains internal expectations about entity ordering in market-specific rankings and responds with positive sentiment when these expectations are confirmed. Deviations produce neutral assessment, while geographic mismatches produce active rejection. The first-position advantage is real and measurable: entities that appear at position 1 in rankings that align with the model’s expectations receive the only positive sentiment classification in this probe.

Confidence Statement: HIGH. All 7 logic gates passed. Small sample (6 variations) but clear, unambiguous pattern. Single-model, single-market limitation acknowledged.