Benchmarking LLMs on Real Itineraries: How Tripstitch Picked GPT-5.6 Luna

Every day in a Tripstitch itinerary is one model call. The model gets the trip context plus a place search tool, and has to come back with a complete day as structured JSON: six to nine activities with start and end times, descriptions, per-person costs, a category for every stop, and the coordinates that become map pins. Get that call right and the app feels like a friend who knows the city. Get it wrong and someone stands in front of a restaurant that doesn’t exist.

Since last week that call goes to GPT-5.6 Luna. This post is about the harness that picked it, because the obvious ways of choosing a model turned out to be useless.

Public benchmarks don’t transfer

Arena rankings and MMLU scores measure chat and trivia. My workload is neither. A model that writes beautiful prose can still schedule four meals, put dinner 40 minutes from lunch, date every activity a day early, or invent a cafe with a confident-sounding address. Those are the failures that generate support email, and no public number predicts any of them.

The git history shows what deciding without a benchmark looks like. In one two-day stretch back in March I switched the day agent from Mistral Small 4 to Claude Haiku to GPT-5.4 mini, each time by eyeballing a handful of outputs and hoping. The only benchmark that answers “which model should take this call” is one that runs my exact agent, with my exact prompt and tools, and measures my exact failure modes. So that’s what php artisan ai:compare-day-models does. It landed in mid-May as a flat side-by-side comparison, got reworked this week into weighted scenarios with a leaderboard, and made the call that moved production from Gemini 3.1 Pro to Luna.

Running the real agent, not a copy

The production day agent already talks to OpenRouter, so every candidate model sits behind the same API and comparing them is a parameter, not an integration. The command takes OpenRouter ids with an optional reasoning-effort suffix, which means openai/gpt-5.6-luna@low and openai/gpt-5.6-luna@medium compete as two separate entrants. They behave like two separate models, more on that below. The current default shortlist is Gemini 3.5 Flash, Muse Spark 1.1, Luna at low and medium, and Grok 4.5.

Each run uses the production system prompt, the production JSON schema, and the same search_places tool backed by the self-hosted Overture mirror. Nothing is simplified for the test. The fixture mirrors a real request: day 3 of a week-long Tokyo trip in August 2026, two travelers with real constraints. Alex is pescatarian and flying in from Amsterdam; Sam eats halal. Both dislike shopping malls and nightlife, there’s no car, the budget is mid-range, and the specific request asks for specialty coffee, a scenic viewpoint, and no tourist-trap restaurants. Three places (Meiji Jingu, Shibuya Sky, Ichiran) are already used on other days of the trip and must not reappear.

One compatibility wart worth recording: Alibaba’s Qwen endpoint rejects any request with response_format set unless the literal word “json” appears in the messages. Every other provider ignores it, so the harness just appends “Respond in JSON.” to the prompt for everyone.

Five scenarios, five failure modes

A single test city tells you almost nothing, because every model looks competent in Tokyo. The harness runs five scenarios, each built to stress one way day generation goes wrong, and each reweights the composite score toward the dimension it stresses:

'baseline'  => ['grounding' => 0.30, 'pins' => 0.25, 'route' => 0.25, 'rules' => 0.20],
'search'    => ['grounding' => 0.50, 'pins' => 0.20, 'route' => 0.10, 'rules' => 0.20],
'route'     => ['grounding' => 0.15, 'pins' => 0.20, 'route' => 0.50, 'rules' => 0.15],
'knowledge' => ['grounding' => 0.00, 'pins' => 0.55, 'route' => 0.20, 'rules' => 0.25],
'nature'    => ['grounding' => 0.15, 'pins' => 0.35, 'route' => 0.20, 'rules' => 0.30],

• baseline is the production-realistic run: Shibuya and Harajuku, dense place data, everything weighted evenly.
• search moves the same trip to Gyumri, Armenia’s second city. Model priors about Gyumri are thin, but the catalog coverage is decent, and the trick is that the few places a model might know from memory (the Black Fortress, Vardanants Square, Mother Armenia) sit on the used-places list. Memory leads nowhere; only the search tool produces scoreable picks, and grounding counts for half the score.
• route is a 3 km north-south corridor from Nakameguro through Daikanyama and Ebisu up to Harajuku, at packed pace. More stops on elongated geography means visiting order dominates, so route quality carries 50%.
• knowledge disables the search tool entirely and runs both Tokyo and Gyumri on pure world knowledge, with every claimed place verified by geocoding. Running it on a famous and an obscure destination gives each model a knowledge cliff: the score drop between the city everyone has training data for and the city nobody does.
• nature is Saxon Switzerland around the Bastei and Kurort Rathen. Trails and viewpoints with barely-real addresses, a handful of village restaurants, and the Bastei Bridge itself already used. The specific request swaps to a proper hike and hearty food where hikers actually go, and the wrong-pin threshold loosens from 3 to 5 km because nature POIs are genuinely fuzzy.

Scoring without an opinion

Each run gets four deterministic sub-scores, 0-100 each, and the scenario weights blend them into a headline number. Deterministic is the point: the same output always gets the same score, reruns are free, and nothing drifts.

Grounding checks that the model picked places instead of inventing them. Every externalId on an activity must appear verbatim in a search result the model actually received; an invented one costs 25 points. A meal slot filled without a search pick costs 20, because the prompt orders a search before every meal. Coordinates must be copied through unchanged from the chosen result.

Pins replay what production would do with each activity, using the same resolution ladder the backend runs when it turns a generated day into map pins. Coordinates copied from a search result score 1.0. Claimed coordinates that came from nowhere get cross-checked by geocoding the activity’s own address: agreement within 1.5 km scores 0.95, disagreement scores 0.4 and gets flagged as likely invented. No coordinates at all walks the fallback chain: an Overture id lookup, then a geocode of the address (0.8, or 0.3 if it lands kilometres outside the region), then a POI search on the title (0.7), then a broken pin (0.0). Because the check runs through the same geocoding path production uses, a pins score of 100 means users would see correct pins, not that the output looked plausible.

Route takes every resolved pin, sums the walking legs, and compares the total against the shortest possible visiting order, computed branch-and-bound with meals locked to their chronological slots (breakfast is not allowed to trade places with dinner). A day more than 15% longer than optimal is flagged as a zigzag. Legs over the 4 km no-car limit cost 8 points each, stops outside the region bounds cost 5.

Rules is the boring checklist: exactly three meals; 3-4 non-meal activities at moderate pace, 5-6 at packed; the correct calendar date; no overlaps; 15-30 minute travel buffers; inside the 08:00-22:00 window; no reused places; an accommodation suggestion present because the fixture asks for one.

For the dimensions geometry can’t reach, a --judge flag grades each output with an independent LLM judge (Claude Sonnet 5 by default): realism of the picks, personalization, description quality, dietary compliance including hidden ingredients like fish dashi and mirin, and geographic flow, each 1-10 with the five worst issues listed. The judge sees the exact prompt the model got plus its raw JSON, and its instructions tell it to reward only what it can verify. I deliberately kept the judge optional and the deterministic score primary. Most of what makes a generated day bad is mechanically checkable, and a mechanical check never has moods.

What the runs look like

Here’s Luna at low effort on the Tokyo baseline, straight from the terminal:

▶ openai/gpt-5.6-luna@low  Tokyo - Shibuya / Harajuku (data-rich) · 3617-char prompt
  ⏱ 13.2s · 24,768 tokens (22,951 prompt / 1,817 out / 235 reasoning) · $0.0329
  ⚙ 2 step(s) / 8 · 4 tool call(s) (search_places×4)
      · search_places(categories=[cafe] radius=1 at 35.6705,139.7035)
      · search_places(categories=[restaurant,casual_eatery] cuisines=[japanese,seafood]
                      diets=[halal,vegetarian] radius=1 at 35.6705,139.7035)
      · search_places(categories=[market,shop,food_and_beverage_store] radius=1 ...)
      · search_places(categories=[scenic_viewpoint,park,art_gallery,cultural_center] ...)
  📦 schema ✓ · finish stop · 6 activities

  Quality:         ██████████████████████░░  91/100
                   grounding 100×0.30 · pins 100×0.25 · route 64×0.25 · rules 100×0.20
  Grounding:       6/6 place picks from search
  Pins:            6/6 resolve - 6 copied
  Route:           1.6 km over 6 stops, longest leg 0.7 km, 57% longer than optimal order
  Timeline:        ··███·████████··█████·███████··██████···██████··········
                   8       10      12      14      16      18      20    22
    ⚠ route is 57% longer than the optimal visiting order (zigzag)
    1. ⚓ 08:30-09:15  [cafe]         Breakfast at PAP Coffee
    2. ⚓ 09:40-11:30  [shop]         Independent Harajuku Shopping Walk
    3. ⚓ 12:00-13:15  [restaurant]   Lunch at Sakura Suisan Harajuku
    4. ⚓ 13:40-15:15  [park]         Yoyogi Park Scenic Stroll
    5. ⚓ 15:45-17:15  [art_gallery]  Design Festa Gallery
    6. ⚓ 18:00-19:30  [restaurant]   Dinner at Bassa Nova Harajuku

Four searches fired in parallel in a single step, all six picks grounded with exact pins, every rule followed. The only deduction is the walking order: 57% more distance than the optimal sequence through the same six stops. 13.2 seconds, 3.3 cents.

The same model at medium effort took 43.5 seconds, spent ten times the reasoning tokens, ran a fifth search dedicated to halal restaurants, and scored 81 on the same fixture. Its picks were actually more diet-careful (vegan ramen for lunch, a vegetarian Indian dinner that works for both travelers), but it hallucinated an externalId on breakfast. The cafe was real: The Bridge, above Harajuku Station. Grok’s run picked the same spot straight from search results under its proper name, Sarutahiko Coffee The Bridge. Luna’s diet-filtered cafe search didn’t return it, so the model reached for a place it knew from memory, and instead of leaving externalId null as the schema instructs, it invented one. That costs 25 points, and it should: in production an invented id is a pin the backend has to rescue. More reasoning does not mean fewer hallucinations.

Grok 4.5 scored highest on this run, 93, with the tightest route (34% over optimal) and the most thorough search behavior: nine tool calls, including re-centering a second restaurant search a few hundred metres south for dinner options and querying Blue Bottle by name for the coffee-loving travelers. It also took 94.3 seconds and 12.1 cents. One run proves nothing on its own, which is why the leaderboard averages the weighted score across all scenarios (a full sweep is 30 generations, five models by six runs, and costs a couple of dollars), but the shape held: Grok buys a few points of quality with seven times the latency and nearly four times the cost.

When a model produces zero activities, the command diagnoses why instead of just printing a zero: it hit the 8-step cap while still trying to call tools, or the finish reason was “length” because reasoning ate the 16k token budget, or the final step was prose instead of JSON. Early auditions failed in all three ways, and having the diagnosis printed inline is the difference between “this model is bad” and “this model needs two more steps.”

The choice

Day generation runs once per generated day, so the economics multiply: a seven-day trip costs about $0.23 in model spend on Luna at low, $0.36 at medium, and $0.85 on Grok. Latency multiplies the same way. Generation streams behind the globe animation, and 43 seconds per day is a wait that works there. 94 seconds per day is not.

Production today runs GPT-5.6 Luna through OpenRouter at medium reasoning effort, temperature 0.4, capped at 8 steps and 16k tokens. Low versus medium is genuinely close and I’ve gone back and forth: low was faster, cheaper, and cleaner on the data-rich baseline, while medium spends its extra tokens on exactly the cases I care about, strict diets and thin destinations. If the next sweep says low holds up in Gyumri and Saxon Switzerland too, the switch is a one-line attribute change. That’s the real payoff of the harness: the decision stopped being a debate and became a rerun.

What I learned

Benchmark the job, not the model. The ordering I get does not match any public leaderboard, and it shouldn’t. Nobody else’s benchmark calls my search tool, obeys my schema, or gets penalized for reusing Meiji Jingu.

Make the score deterministic first, subjective second. Wrong date, four meals, a 6 km walking leg, a pin in the wrong town: all of it is checkable with a calendar and a haversine. Save the LLM judge for taste, and never let it be the headline number.

Build scenarios from failure modes, not demos. Every model is charming in Shibuya. Gyumri is where they separate, because memory stops helping and process starts mattering. If I had only tested the demo city I would have shipped on charisma.

Score through production plumbing. Pins are graded by the same resolution ladder that renders real pins, and grounding by the actual tool results the model saw. The score predicts what users see, not what reads well in a transcript.

Cost and latency are quality dimensions. A 93 that takes 94 seconds and 12 cents loses to a 91 that takes 13 seconds and 3 cents, seven times over on every single trip.

Tripstitch is on the App Store. The model pick will change again, probably before the year is out. The harness is the part that lasts: when the next model drops, the audition is one command and a couple of dollars.