Benchmark Methodology: How Every Travel Planning Score Is Measured

Every number on the leaderboard comes from one of the tasks below. Each task says what the model is asked, what it must return, and what decides whether the answer is right.

Four rules hold across all of them.

No model judges another model. Every score traces back to Overture Maps, Apple Maps, the tz database or arithmetic. Nothing is graded on prose quality, because prose quality is exactly where an automated judge is least trustworthy and where a wrong answer does the least damage.

Ground truth is resolved once and frozen. Coordinates, distances, travel times, arrival times and optimal routes are computed ahead of a run and stored with the case. A run is therefore reproducible, and the stored answers can be audited independently of any model.

Temperature 0, five repeats. Providers are not deterministic even at temperature 0, so every task runs five times and every published figure carries the standard deviation across those runs.

Reasoning effort is never assumed. A model tested at a specific effort says so on its page and carries that effort in its name; a model that says “provider default” was sent no reasoning parameter at all. The same model at two efforts is two separate entrants with separate pages, because they behave like two different models: more thinking usually costs several times more and takes several times longer, and it does not reliably buy fewer invented places.

A non-answer is a wrong answer. A model that returns nothing, returns unparseable output, or quietly answers four of five questions is scored on all five. Refusing to answer never scores better than answering badly.

Geographic knowledge

What the model knows about the world, with no tools available.

Coordinate recall. Given a place and its city, return latitude and longitude. Scored as the distance in metres from the authoritative coordinates, and as the share of places located within 250 metres, which is roughly the threshold for a usable map pin. Cases are tiered from world-famous landmarks through regionally known places to genuinely obscure ones, and the gap between the top and bottom tier is the knowledge cliff.

Distance estimation. Straight-line distance between two places in kilometres, scored against the great-circle distance over resolved coordinates. The signed bias is reported alongside the error because the direction is consistent per model, and a planner that systematically shrinks the world will schedule days nobody can walk.

Local naming. Give the name a place is known by locally, in the local script. Accepted answers come from the alternate names on the place’s own database record, folded for case, spacing and punctuation.

Spatial reasoning

Geometry over coordinates supplied in the prompt. Nothing here needs world knowledge, which is what separates it from the section above.

Route optimization. Given a set of stops with coordinates, return the visiting order that minimises total walking time. Scored against the exact optimal order, computed by Held-Karp over a real Apple Maps walking-time matrix between every pair. Reported as the percentage longer than optimal, the share of problems solved exactly, and the share of answers that were a genuine permutation of the stops. Problems come at five, eight and twelve stops.

Spatial queries. Which points fall within a radius, which lies furthest north, which pair is closest. Answers are computed from the same coordinates the model was given, so the only skill involved is arithmetic.

Real-world estimation

Travel time estimation. How long a journey takes on foot or by public transport, scored against real Apple Maps road-network routing rather than a distance divided by an assumed speed. The underestimate rate is reported separately because the two directions are not equally harmful: overestimating wastes slack, underestimating produces a day that breaks before lunch.

Time zone arithmetic. Multi-leg journeys with a departure in local time and a duration, answered with the local arrival time. The tz database decides, so daylight-saving transitions, half-hour offsets and date rollovers are settled by a table rather than by anyone’s mental arithmetic. Several cases deliberately straddle a DST changeover in one country but not the other.

Place grounding

The section that matters most, and the one weighted highest.

Place existence. Name a given number of specific venues of a given kind in a given city. Every name is checked against the Overture mirror and Apple Maps POI search. A name neither can find anywhere counts as a hallucination; a name that exists but sits in another city counts as a wrong location. Cases are tiered by how well covered the destination is, from Paris down to Nuuk.

The number of places actually returned is also scored, so a model cannot buy a clean hallucination rate by answering with two names when five were requested.

Place existence with search. The identical questions, re-run with a plain place-search tool available. The difference between the two hallucination rates is how much a model is rescued by being able to look things up, which is worth knowing before wiring one into a product.

Abstention. Requests for something that is not there: ski resorts in Singapore, vineyards in Reykjavik. The honest answer is to say so. Each case is verified absent from the places database before it is used, and the same database category is verified present in a control city, so a mistyped category can never masquerade as proof of absence. If a model names something that turns out to be real, the case is flagged for review rather than counted against the model.

Itinerary construction

Constraint satisfaction. A day brief carrying a numbered list of hard requirements, each decidable by arithmetic on the returned plan: start after a time, exactly this many meals, nothing over this price, at least this long a gap between stops, no repeated places. Briefs come at three, six and ten constraints, and the tier breakdown shows where a model stops keeping track.

Schedule feasibility. Schedule a supplied set of real places into a day. Each gap between consecutive stops is checked against real walking time between exactly those two coordinates. Reported as the share of moves that cannot physically be made in the time allowed, and the total minutes the day runs short.

Budget arithmetic. A day under a hard per-person budget. Two separate failures are measured: a plan whose own line items exceed the budget, and a plan whose stated total disagrees with the sum of the costs it listed. The second is the more alarming, because it means every number in the plan is untrustworthy.

The Travel Score

Each metric that counts toward the score is mapped onto 0 to 100 between two published anchors, one for a perfect answer and one for the worst outcome worth distinguishing. A hallucination rate of 0 scores 100 and a rate of 1 scores 0. A coordinate error of 0 metres scores 100 and 5 kilometres or worse scores 0. Those anchors ship in the data file for every metric, so the composite can be recomputed, reweighted or ignored entirely.

Family scores are the mean of their metrics. The composite blends them:

Grounding carries the most weight because an invented restaurant is the failure a traveller actually notices. Cost, latency and output reliability are reported for every model but deliberately kept out of the composite, since what counts as too slow or too expensive depends on what you are building.

Every raw metric sits next to the composite on each model’s page. If you disagree with the weights, the numbers to compute your own are right there.

Reproducing this

The benchmark is an Artisan command in the Tripstitch API. One invocation measures one model and writes one data file, which is what this site renders. Adding a newly released model does not require re-running anything else, which is also why each model’s page carries its own test date and suite version, and why a result from an older suite version is flagged rather than silently ranked against current ones.

The case sets, the scoring code and the anchors are all versioned together. When a task changes in a way that makes old numbers incomparable, the suite version changes with it.