Methodology: How Every Travel Planning Score Is Measured

Every number on the leaderboard comes from one of the tasks below. Each task says what the model is asked, what it must return, and what decides whether the answer is right.

Five rules hold across all of them.

Every score is measured, not judged. It traces back to Overture Places, Apple Maps, the tz database or arithmetic. Nothing is graded on prose quality, because prose quality is exactly where an automated judge is least trustworthy and where a wrong answer does the least damage.

Ground truth is resolved once and frozen. Coordinates, distances, travel times, arrival times and optimal routes are computed ahead of a run and stored with the case. A run is therefore reproducible, and the stored answers can be audited independently of any model.

Temperature 0, five repeats. Providers are not deterministic even at temperature 0, so every task runs five times and every published figure carries the standard deviation across those runs.

Reasoning effort is never assumed. A model tested at a specific effort says so on its page and carries that effort in its name; a model that says “provider default” was sent no reasoning parameter at all. The same model at two efforts is two separate entrants with separate pages, because they behave like two different models: more thinking usually costs several times more and takes several times longer, and it does not reliably buy fewer invented places.

A non-answer is a wrong answer. A model that returns nothing, returns unparseable output, or quietly answers four of five questions is scored on all five. Refusing to answer never scores better than answering badly.

Geographic knowledge

What the model knows about the world, with no tools available.

Coordinate recall. Given a place and its city, return latitude and longitude. Scored as the distance in metres from the authoritative coordinates, and as the share of places located within 250 metres, which is roughly the threshold for a usable map pin. Cases are tiered from world-famous landmarks through regionally known places to genuinely obscure ones, and the gap between the top and bottom tier is the knowledge cliff.

Distance estimation. Straight-line distance between two places in kilometres, scored against the great-circle distance over resolved coordinates. The signed bias is reported alongside the error because the direction is consistent per model, and a planner that systematically shrinks the world will schedule days nobody can walk. Bias is the one measurement here whose ideal is neither end: zero is right, and inflating distances by 40% is as wrong as shrinking them by 40%. It is published as a diagnostic and left out of the score, because two anchors cannot express an ideal that sits in the middle.

Spatial reasoning

Geometry over coordinates supplied in the prompt. Nothing here needs world knowledge, which is what separates it from the section above.

Route optimization. Given a set of stops with coordinates, return the visiting order that minimises total walking time. Scored against the exact optimal order, computed by Held-Karp over a real Apple Maps walking-time matrix between every pair. Reported as the percentage longer than optimal, the share of problems solved exactly, and the share of answers that were a genuine permutation of the stops. Problems come at five, eight and twelve stops.

Spatial queries. Which points fall within a radius, which lies furthest north, which pair is closest. Answers are computed from the same coordinates the model was given, so the only skill involved is arithmetic.

Real-world estimation

Travel time estimation. How long a journey takes on foot or by public transport, scored against real Apple Maps road-network routing rather than a distance divided by an assumed speed. The underestimate rate is reported separately because the two directions are not equally harmful: overestimating wastes slack, underestimating produces a day that breaks before lunch.

Time zone arithmetic. Multi-leg journeys with a departure in local time and a duration, answered with the local arrival time. The tz database decides, so daylight-saving transitions, half-hour offsets and date rollovers are settled by a table rather than by anyone’s mental arithmetic. Several cases deliberately straddle a DST changeover in one country but not the other.

Place grounding

The section that matters most, and the one weighted highest.

Place existence. Name a given number of specific venues of a given kind in a given city. Every name is checked against the Overture Places mirror, a global catalog of more than 80 million places, and Apple Maps POI search. The share whose names resolve is the grounding rate; a name neither source can find anywhere is counted as invented. A name that resolves, but only in another city, counts against wrong location rather than against grounding. Wrong location is published as a diagnostic and left out of the score: nearly every name that resolves at all resolves in the right city, so scoring it only diluted the measurements that separate models. Cases are tiered by how well covered the destination is, from Paris down to Nuuk.

Grounding is published as the share that resolve rather than the share that do not, so the leaderboard’s most important column runs the same way as the score next to it and higher is better in both. It is the exact inverse of a hallucination rate, 92% grounding being 8% invented, and the composite works out the same either way.

The number of places actually returned is also scored, so a model cannot buy a clean grounding rate by answering with two names when five were requested.

Place existence with search. The identical questions, re-run with a plain place-search tool available. The difference between the two grounding rates is how much a model is rescued by being able to look things up, which is worth knowing before wiring one into a product.

Both runs publish the same three measurements, so each appears twice on a model page: once marked from memory and once marked with search. The from-memory figure is what a model knows. The with-search figure is what it produces when it can check, which is the number to read if you are building something that hands it a tool. For some models the two are far apart and for others they are nearly identical. The tool itself is deliberately plain, with no category vocabulary and no ranking of its own, so what gets measured is how well a model uses a search tool rather than how well it uses ours.

Abstention. Requests for something that is not there: metro stations in Bruges, marinas in La Paz. The honest answer is to say so. Each case is verified absent from the places database before it is used, and the same database category is verified present in a control city, so a mistyped category can never masquerade as proof of absence. If a model names something that turns out to be real, the case is flagged for review rather than counted against the model. This check earns its keep: it threw out our first attempts at ski resorts in Singapore, which has an indoor slope, and vineyards in Reykjavik.

Itinerary construction

Constraint satisfaction. A day brief carrying a numbered list of hard requirements, each decidable by arithmetic on the returned plan: start after a time, exactly this many meals, nothing over this price, at least this long a gap between stops, no repeated places. Briefs come at three, six and ten constraints, and the tier breakdown shows where a model stops keeping track.

Schedule feasibility. Schedule a supplied set of real places into a day. Each gap between consecutive stops is checked against real walking time between exactly those two coordinates. Reported as the share of moves that fit the time allowed, and the total minutes the day runs short. Like grounding, this one is published the positive way up; the field runs from about a sixth to just under four fifths, so nothing is lost by flipping it.

Budget arithmetic. A day under a hard per-person budget. Two separate failures are measured: a plan whose own line items exceed the budget, and a plan whose stated total disagrees with the sum of the costs it listed. The second is the more alarming, because it means every number in the plan is untrustworthy.

The Travel Score

Each metric that counts toward the score is mapped onto 0 to 100 between two published anchors, one for a perfect answer and one for the worst outcome worth distinguishing. A grounding rate of 1 scores 100 and a rate of 0 scores 0. A coordinate error of 0 metres scores 100 and 5 kilometres or worse scores 0. Those anchors ship in the data file for every metric, so the composite can be recomputed, reweighted or ignored entirely.

Family scores are the mean of their metrics. The composite blends them:

Grounding carries the most weight because an invented restaurant is the failure a traveller actually notices. Cost, latency and output reliability are reported for every model but deliberately kept out of the composite, since what counts as too slow or too expensive depends on what you are building.

Every raw metric sits next to the composite on each model’s page. If you disagree with the weights, the numbers to compute your own are right there.