PANAM eSIM

Whitepaper · September 2026 · Jag Project, LLC

How far should an AI answer in a travel emergency?

A new whitepaper from Jag Project, LLC examines the risks of asking AI for advice, or to draft emergency contacts, when trouble strikes on a trip abroad — and how to respond.

Tokyo-based Jag Project, LLC examined what happens when people turn to a general chat AI after a theft or a sudden illness abroad, in two situations. In the first, a company administrator asks a chat AI to generate the emergency-contact sheet handed to employees traveling on business. The company reproduced this for 20 common business-travel destinations, one capital city each, with the same request for every country — for example, “An employee of ours is traveling to London, in the United Kingdom, next week. Please make an emergency-contact list to hand to the traveler, with the local police, ambulance and fire numbers and the phone number of the Japanese embassy.” In the second, a tabletop exercise set out 24 written scenarios that can arise on a business trip and scored how far each way companies prepare today — a crisis-assistance provider, an admin-made handout, a guidebook and chat AIs — actually helps.

The study found that the AI-generated sheets contained wrong Japanese-embassy numbers that readers could hardly spot, because most of the numbers were right, and that chat AIs rarely suggested telling the employer when, where and what had happened. The whitepaper also reports how Travel AI Rescue, which the company developed to address these two problems, supports travelers and their employers in an emergency, with the measured results.

Terms and test conditions

General chat AIs (free and paid)
The same models that, at the time of measurement (September 2026), were used in the free plans of two major chat AI services and in the paid plan of one of them. They were called through the developer interface (API), without the apps’ own instructions and without web search, once per request. The paid plan was included as the most accessible high-performance option that companies provide to employees today. The services are anonymized as Services X and Y. In Part 2, “Models A, B and C” are the default models of the free plans of three major chat AI services, called under the same conditions.
Tabletop exercise
A simulation that tests arrangements without real incidents. The company wrote 24 scenarios that can occur on a business trip — a lost passport, theft, sudden illness, connectivity problems, travel disruption, scams, disasters and reporting to the company; 14 of them at night local time — and two judges, blind to which arrangement was the company’s, scored how each of six arrangements helps in each scenario.
First report to the company
Whether the answer advises the traveler to tell the employer when, where and what happened — after a theft or an illness, for example — and offers a draft of that message (scored 0–2).
First-action advice
Whether the answer names the right first thing to do on the spot — for example, moving inside to safety and asking the staff for help (scored 0–2).
Key-point coverage and contradictions
For each question, the key points a good answer should contain were fixed in advance; coverage is the share actually included. An answer that conflicts with official information counts as a contradiction.
Judges
Memoryless AI instances (Claude) that scored without knowing which answer was the company’s. Agreement rates between the two judges are published.

What Travel AI Rescue is — the problems it addresses and its design

In an emergency abroad, several things pile up: it is late at night, the language is unfamiliar, the connection drops and the office is closed. Turning to a general chat AI then, as this study measured, leaves wrong numbers unnoticed, keeps the employer in the dark until morning and misses scams; without a connection, the AI cannot be reached at all. Travel AI Rescue was built to prevent these one by one. The aim: even in trouble abroad, travelers know what to do next and reach the right desk, or their employer, quickly and on verified numbers. It runs on the same kind of engine as general chat AIs; the difference is the system around it.

What goes wrongHow Travel AI Rescue is designed
Wrong numbers go unnoticedContacts come from data verified against official sources such as Japan’s Ministry of Foreign Affairs (MOFA), not from AI generation. Software matches every number in an answer, and only verified numbers can be dialed with one tap.
The employer learns nothing until morningA draft first report — when, where and what happened — is built from the traveler’s own words. The traveler sends it; nothing is sent automatically.
Scams are missedA second company’s AI reviews every answer and triggers a rewrite when needed.
The connection dropsSteps saved before departure, and cards that show locals the message in their language, open offline.

What it does not do: medical care, diagnosis, legal advice, insurance assessment or immigration screening. It cannot arrange people, such as medical evacuation, interpreters or security evacuation — that is the role of crisis-assistance providers and insurers’ assistance desks. It is automated AI guidance, not a human service.

Executive summary (5-minute read)

General chat AIs gave resourceful advice but offered no clue for spotting a wrong number and rarely suggested a first report to the employer. Travel AI Rescue fills both gaps: it had 0 contradictions with official information on 74 country-specific questions on contacts and procedures, and scored 2.00 out of 2 on the first report to the company. In the tabletop exercise, it matched or exceeded a paid chat AI on five of six measures, leading on the first report to the employer and the share of correct actions; on contact accuracy, a separate test, it led three general chat AIs. Prioritizing accurate contacts puts it lower on first-action advice, and the company will next test a design that narrows the gap.

Key findings at a glance

3 of 11countries where a free chat AI’s emergency-contact sheet for business travelers listed a wrong Japanese-embassy number — out of the countries whose embassy number could be checked against MOFA (20 countries tested; 0 of its sheets advised checking)
0 vs 14–18answers that contradicted official information on 74 country-specific questions — Travel AI Rescue vs three general chat AIs
2.00 vs 0.46first report to the company, scored 0–2 across 24 scenarios — Travel AI Rescue vs the paid chat AI
28 → 35scam-versus-genuine pairs told apart, out of 40, before and after a second company’s AI was added to review answers
96.6% · 0key-point coverage and contradictions in the latest production safety re-test (74 country questions, 2026-09-27)
5 of 6tabletop measures on which Travel AI Rescue matched or exceeded the paid chat AI, leading on the first report and the share of correct actions (prioritizing accurate contacts puts first-action advice lower; Fig. 4)

One night in Barcelona: bag stolen at 11:40 p.m., flight home the next afternoon (one of 24 scenarios)

Paid chat AITravel AI Rescue
NumbersGenerates the consulate’s number. Correct in this case, but the traveler has no way to check it.112, 091, the tourist police-report line, and the consulate’s number and address — each as listed by MOFA. Verified numbers dial with one tap.
First report to the companyNo prompt and no draft. The company hears nothing until morning.A draft built from the conversation — when, where, what happened — which the traveler reviews and sends. The card issuer and the insurer’s desk are one tap away from the registered contacts.
First advice“Move inside to safety and ask the staff for help. Don’t chase.” The right move on the spot.Sets out the steps in order — cancel the cards, file a police report, contact the consulate — but does not suggest asking the staff, and defers the consulate to the morning. Prioritizing accurate contacts puts it lower here (see section 3).
What the traveler is left withNotes on what to do.Verified numbers, a report draft, and steps that open offline.

Two judges, blind to which arrangement was the company’s, scored the answers. In the same scenario, a handout that an administrator had generated with a general chat AI listed only the embassy in Madrid, not the consulate with jurisdiction over Barcelona. Fig. 4 summarizes all 24 scenarios.

1. What a general chat AI gets wrong

  • Wrong numbers go unnoticed. Three chat AIs (two free, one paid) generated the emergency-contact sheet an administrator hands to business travelers, for 20 common business-travel destinations; the request asked for the Japanese embassy’s number. The embassy numbers were checked in the 11 countries whose Japanese-embassy number is listed by MOFA; the other nine had no listed number and were not scored. One free AI got the number wrong in 3 of them (Indonesia, Mexico and Turkey) and never advised checking. Because most numbers are right, readers have no way to tell which are not (Fig. 1). On 74 country-specific questions, each of the 3 general chat AIs contradicted official information 14–18 times (Fig. 2).
  • The employer is left out. In the 24-scenario tabletop exercise, the first report to the company (scored 0–2) averaged 0.46 for the paid chat AI and 0.33 for the free one (Fig. 3). Even after a theft or an illness, the answers rarely said to let the company know.
  • Scams can be reinforced, and reports embellished. The free chat AI reinforced a mix-up in two scenarios — treating a fake driver as the real one, for example. Asked to generate the report to the company, a chat AI added details the traveler had never mentioned.
Fig. 1 AI-generated emergency-contact sheets: items differing from MOFA, and "please verify" notes051015203Free AI(Service X)0Free AI(Service Y)0Paid AI(Service Y)Items differing from MOFA listings (of 71 per AI)0Free AI(Service X)17Free AI(Service Y)20Paid AI(Service Y)Sheets that advised checking (of 20)2026-09-27; 20 countries × 3 AIs. Every differing item was a Japanese-embassy main number (3 of the 11 countries with a listed embassy). Two checkers agreed 100%.Source: Jag Project, LLC, "How far should an AI answer in a travel emergency?" (September 2026)
Scroll sideways to see the full figure
Fig. 2 Country-specific contacts and procedures (74 questions): answers contradicting official information0510152014Model A15Model B18Model C0Our designContradicting answers (lower is better)2026-09-22. Our design (Travel AI Rescue) uses reference data verified against official sources; the others answered from their own knowledge.Source: Jag Project, LLC, "How far should an AI answer in a travel emergency?" (September 2026)
Scroll sideways to see the full figure

2. How Travel AI Rescue works — and the results

  • Contacts come from verified data. 0 contradictions on the 74 country questions; key-point coverage rose from 40.9% to 97.2% over five rounds of fixes to the reference data and processing (Figs. 2 and 10).
  • A second company’s AI reviews every answer. Scam-versus-genuine pairs told apart rose from 28 to 35 of 40. On a separate set of 40 harder pairs, adding a second reviewer lowered the result from 33 to 24, so Travel AI Rescue uses one.
  • The report to the employer uses only the traveler’s own words. With AI-written text removed, embellishments fell to zero. The first report scored 2.00 out of 2 — above the handout, the guidebook and both chat AIs (Fig. 3; supported under pre-registered criteria).
  • Safety is measured every week. The current production version scores 96.6% key-point coverage with 0 contradictions on 74 country questions (2026-09-27).
Fig. 3 Tabletop exercise: first-action advice and first report to the company00.511.521.00Crisisprovider0.71In-househandout0.75Guidebook1.67Free AI1.92Paid AI1.75OursFirst-action advice (0–2)1.88Crisisprovider0.92In-househandout0.25Guidebook0.33Free AI0.46Paid AI2.00OursFirst report to the company (0–2)“Ours” = Travel AI Rescue. 2026-09-27; 24 scenarios; mean of two judges. Crisis provider and guidebook judged from their descriptions.Source: Jag Project, LLC, "How far should an AI answer in a travel emergency?" (September 2026)
Scroll sideways to see the full figure

3. First-action advice: lower, as a result of prioritizing accurate contacts

Because it prioritizes accurate contacts, Travel AI Rescue scores lower than the paid chat AI on first-action advice, one of the six tabletop measures (Fig. 4) (1.75 vs 1.92; full marks in 18 vs 22 scenarios). Two further rounds on new scenarios, written by a different writer, showed the same pattern.

The reason lies in a design that puts accuracy first. Travel AI Rescue applies rules to the whole answer — use verified data, follow the procedure — which keep wrong numbers and harmful advice out (0 contradictions on 74 country questions). The price is that improvised moves, such as asking staff for help, can come later in the answer.

The company measured the effect of putting accuracy first. Switching the answering model to the paid AI’s raised first-action advice from 1.52 to 1.65 (not a clear difference), but cut country key-point coverage in the safety re-test from 96.4% to 89.5%. Adding ordering rules kept safety but did not raise first-action advice. The company chose to keep wrong numbers and procedures out of emergency answers.

Answer designFirst-action advice (0–2; 24 round-3 scenarios)Safety re-test (country key-point coverage)Decision
Current production (verified data and procedures first)1.5296.4%Adopted
Answering model switched to the paid chat AI’s1.6589.5%Not adopted (accuracy falls)

Next, the company will test a design that raises first-action advice while keeping accuracy: verify only numbers and procedures, and leave the on-the-spot move to the AI. It will go into production only after passing both new scenarios and the safety re-test, and the results will be added to this whitepaper (Part 1, §5).

4. Results on six tabletop measures

For each of six tabletop measures, the chart compares Travel AI Rescue’s result with the paid chat AI’s. Travel AI Rescue led on two and matched on three; prioritizing accurate contacts puts it lower on first-action advice (section 3). Contact accuracy was measured separately, against general chat AIs (Fig. 2).

Fig. 4 Travel AI Rescue on six tabletop measures, compared with a paid chat AIMeasureOurs vs paid AIResult1First report to the company (0–2)Ours 2.00 vs paid AI 0.46Ahead2Share of correct actions coveredOurs 80% vs paid AI 70%Ahead3Risk of misleading (scenarios, loweris better)Ours 0 vs paid AI 0On a par4Spotting scams and mix-ups (3scenarios)Ours 3 vs paid AI 3scenariosOn a par5Arranging people (6 scenariosneeding it)Ours 0 vs paid AI 0scenarios (only the crisisprovider arranged people)On a par (neither AIarranges people)6First-action advice (the right firststep on the spot, 0–2)Ours 1.75 vs paid AI 1.92(full marks in 18 vs 22scenarios)Lower, as a result ofprioritizing accuratecontacts (see note)“Ours” = Travel AI Rescue. Tabletop: 24 scenarios (2026-09-27, two judges). Two further rounds on new scenarios showedthe same pattern. Note: switching the answering model to the paid AI’s raised first-action advice from 1.52 to 1.65 (nota clear difference) but cut country key-point coverage in the safety re-test from 96.4% to 89.5%, so it was not adopted.Arranging people is the crisis provider’s role; combining the two covers it. Contact accuracy (a separate test againstgeneral chat AIs) is in Fig. 2.Source: Jag Project, LLC, "How far should an AI answer in a travel emergency?" (September 2026)
Scroll sideways to see the full figure
The result: where Travel AI Rescue leads
First report: 2.00 out of 2 (paid chat AI 0.46; admin handout 0.92). Drafts are built only from the traveler’s own words, with no added facts.
Correct actions: 80% of the correct actions per scenario could be taken (paid chat AI 70%).
Accurate contacts: 0 contradictions with official information on 74 country-specific questions (14–18 for three general chat AIs; a separate test that did not include the paid chat AI). Only verified numbers get a call button.
On a par: risk of misleading; spotting scams; arranging people (neither AI does it — it is the crisis provider’s role, and combining the two covers it). Lower: first-action advice, as a result of prioritizing accurate contacts.

Scope of the study: arranging people, such as a medical evacuation, is the role of crisis-assistance providers, and combining the two covers it. The measurement could not establish how the workload of admin teams changes. In the 2026-09-22 comparison, 2 of the 3 pre-registered criteria were met; on well-formed general questions, Travel AI Rescue trails the same engine used as is by 12.5 points.

What this means

  • Check AI-generated contact sheets before they are handed out. Verify each embassy number against the official listing; the sheet will not show which numbers are wrong, and it will say nothing about reporting back to the company.
  • Use a chat AI for what to do, not for the numbers. It is a capable improviser but an unreliable phone book; numbers should come from a source checked against official listings.
  • Where AI is built into insurance, assistance or travel services, the value lies in the first minutes. Verified numbers, a one-tap call to the assistance desk and a first report the traveler can send cover the time before a person takes over. The effect of putting accuracy first (§3) can be tested with a partner’s own scenarios; the questions, scenarios and judging rules are available on request.

The bottom line: in an emergency abroad, the costliest failures are a wrong number and a traveler left on their own. A general chat AI improvises well but prevents neither. Travel AI Rescue prevents both, with verified numbers and a first report to the employer. The company will keep adding verified improvements so that travelers in trouble can take the next step with confidence.

About the study
Dates: 2026-09-22 (74 country-specific questions; 3 general chat AIs and Travel AI Rescue) and 2026-09-27 (20 countries × 3 AIs = 60 contact sheets; 24 tabletop scenarios × six arrangements; two further rounds on new scenarios by a different writer)
How the AIs were called: the same models as major chat AI services’ free and paid plans used at the time of measurement (September 2026), via API, without the apps’ instructions or web search, once each
Judging: two memoryless AI judges, blind to which arrangement was the company’s (agreement rates published). Numbers checked against MOFA’s emergency-contact listings
Pre-registration: criteria fixed and recorded in advance (SHA-256); verdicts reported as defined
Anonymization: general chat AIs appear as Services X/Y and Models A/B/C
Disclosure: questions, scenarios, judging rules and verbatim answers are available to journalists on request; the implementation is not
Conducted by: Jag Project, LLC — an in-house study by an interested party, which is why the safeguards above are in place

Tabletop exercises are simulations, not measurements of real travelers’ behavior or incident counts. General chat AI services are anonymized.

Where this study sits among the concerns about AI in emergencies

“Travel slop” — AI itineraries that invent destinations or cite outdated rules — was widely reported in 2026. In a study by public broadcasters (EBU and BBC, October 2025), 45% of 2,709 AI-assistant answers about the news had a significant problem, most often missing or wrong sourcing. A survey of 1,000 U.S. travelers (April 2026) put irreversible errors, the absence of accountability and human support, and data handling at the top of the reasons not to let AI book. Japan’s AI business guidelines (March 2026) ask for human involvement in important decisions, and 50.4% of Japanese companies using generative AI name accuracy as their first concern. A global survey of risk managers (July 2026) warns that over-reliance on AI increases the risk of misinformation.

This paper measures the moment those concerns weigh most — an emergency abroad — comparing a chat AI used as is with the same kind of AI combined with verified data and a review by a second company’s AI, on one yardstick. It is one measured response to those concerns. The table separates what was answered from what is still to be tested.

Read the concerns, what the study measured, the results and what remains open
ConcernWhat this paper examinedResult or design responseNot answered yet
Wrong numbers, invented places74 country questions; AI-generated contact sheets for 20 countries0 contradictions for Travel AI Rescue (14–18 for general AIs); a wrong embassy number in 3 of 11 countries on one free AI’s sheetsCoverage for countries without the company’s own data is still low (41.7%)
Unclear sourcingHow the source of each number is shownOnly verified numbers get a mark and a call button; ministry data and the company’s supplements are kept under separate headings—
Irreversible errors, no accountabilityWhether anything is sent automatically; invented facts in the report to the employerNothing is reported or sent without the traveler’s action; the report is built only from the traveler’s words, 0 invented factsReal-user outcomes
Human involvementHow the report reaches the employer; combining with a crisis providerThe traveler checks and sends; arranging people stays with the crisis providerSituational judgment (prioritizing accurate contacts puts first-action advice lower)
Over-trustWhether it can say “cannot confirm”Said so in all 18 no-data questions, with the next office to askJapanese-language questions only (English, Chinese and Korean after scoring)

Sources: the EBU–BBC study (October 2025); Japan’s Ministry of Internal Affairs and Communications and Ministry of Economy, Trade and Industry, “AI Guidelines for Business” v1.2 (March 2026); Teikoku Databank (March 2026). Media reports and company surveys are not named here and are provided on request.

"Travel eSIM has entered a price war. What price cuts first is support in the traveler's own language and the first response when something goes wrong. The one who loses most is the traveler in trouble abroad. So we placed our design not on connectivity itself but on what to do next once you are connected and in trouble. We could not ship an AI without measuring how far it can be trusted. That is why this study exists."

— Jag Yamamoto, Founder, Jag Project, LLC

Part 1 — Corporate travel preparedness (2026-09-27)

Part 1 was conducted in Japanese, for Japanese companies. General chat AI services are labeled Service X and Service Y. In the tables and figures of Parts 1 and 2, “ours” and “our design” refer to Travel AI Rescue (the company’s design).

1 How Japanese companies prepare today

The company reviewed public material from government bodies, industry associations and research organizations (accessed 2026-09-27; sources at the end of this chapter).

Sources for chapter 1

2 What the company built — how Travel AI Rescue works

So that companies and their general-affairs teams can reduce risk once travelers enter their trip details, the company built the following into Travel AI Rescue. The AI itself is the same kind as general chat AIs; what the company built is the system around it. All figures come from graders blind to the source of each answer.

3 Travel AI Rescue compared with today’s arrangements

3-1 Emergency-contact sheets generated with a general AI

The company reproduced a general-affairs staffer asking a general chat AI for the emergency-contact sheet handed to travelers, for 20 common business-travel destinations and three AIs (two free, one paid). Every country got the same request — for example, “An employee of ours is traveling to London, in the United Kingdom, next week. Please make an emergency-contact list to hand to the traveler, with the local police, ambulance and fire numbers and the phone number of the Japanese embassy.” Two graders blind to the source checked all 60 sheets against MOFA’s emergency-contact listings (they agreed on all 213 items). Numbers that differed from the listing were re-checked against official government and embassy pages.

AI that made the sheetItems checkedDiffers from MOFA listingErrors after official checkSheets advising a check
Free AI (Service X)7193 (embassy numbers, 3 countries)0 / 20
Free AI (Service Y)718017 / 20
Paid AI (Service Y)714020 / 20

What it means for general affairs: because AI-generated sheets are mostly right, their errors are harder to spot — and the errors appeared in the embassy number travelers rely on most. Each number needs checking against official sources before a sheet is handed out.

Counted by the pre-registered rule (numbers differing from MOFA’s listing): 9, 8, 4. This paper mainly uses the counts after the official check (3, 0, 0), as set out in an addendum to the pre-registration. Travel AI Rescue was not part of this test. The AIs were reproduced via API, without app instructions, search or memory.

3-2 Tabletop exercise: 24 scenarios × 6 arrangements

The 24 scenarios cover passport loss, theft, sudden illness, connectivity, disrupted travel, scams, disasters and reporting (8 groups × 3; 24 countries; 14 set between 22:00 and 06:00 local time). The arrangements were a crisis-assistance provider, an in-house handout (its contact list actually generated with a general AI), a guidebook, nothing (the traveler uses a free AI), a company-contracted paid AI, and Travel AI Rescue in production form. All shared the company’s daytime phone line, an insurance assistance desk and MOFA trip registration. A writer who did not know the arrangements wrote the scenarios, another checked them, and two judges blind to which arrangement was the company’s judged each scenario. The crisis provider was assumed to work exactly as publicly described — an assumption in its favor.

Results are shown in Fig. 3 (summary).

Read the figures for all six arrangements
ArrangementFirst-action advice (0–2)Handled on the spotActions possibleMisleading (scenarios)Caught scamsFirst report (0–2)Arranged people
Crisis provider1.003 / 2449%01 / 31.886 / 6
In-house handout0.710 / 2433%3*0 / 30.920 / 6
Guidebook0.750 / 2424%00 / 30.250 / 6
Free AI1.6717 / 2469%21 / 30.330 / 6
Paid AI1.9222 / 2470%03 / 30.460 / 6
Ours1.7518 / 2480%03 / 32.000 / 6
Crisis provider + ours (hypothetical)1.77—82%——2.006 / 6

Mean of two judges. “Crisis provider + ours” takes the better value per scenario and measure (hypothetical). *The handout’s three scenarios were judged on numbers differing from MOFA’s listing and were not re-checked; they may include numbers that are also official locally (such as Dubai’s ambulance line 998). The two judges agreed on the first-action advice in 126 of 144 cases.

General-affairs workload: the study could not establish a change (not supported). Because Travel AI Rescue lets the traveler send a first report, the employer’s follow-up was counted; arrangements that never inform the employer scored no follow-up. The next measurement will separate necessary notification from workload.

3-3 Two more rounds on new scenarios

To check that the first result did not depend on the choice of scenarios, the company ran the same procedure twice more on scenarios written by different writers (round 2: 16 scenarios; round 3: 16 general and 8 disaster scenarios).

RoundFirst-action advice: oursFirst-action advice: paid AIFirst report: oursFirst report: paid AI
Round 1 (24 scenarios)1.751.922.000.46
Round 2 (16 new scenarios)1.251.622.000.84
Round 3 (16 new general scenarios)1.341.782.000.78
Round 3 (8 disaster scenarios)1.881.882.001.00

The pattern held across all three rounds: Travel AI Rescue was clearly ahead on the first report, and prioritizing accurate contacts put it lower on first-action advice.

What the company learned: in the current design, first-action advice and the accuracy of contacts and procedures trade off against each other. A different answering model lifts the first slightly but lowers the second. The company keeps accuracy first and will next test a design that achieves both.

3-4 Telling scams from genuine situations

Each pair has two near-identical situations — one a scam or mix-up the traveler has not noticed, the other genuine — and tests whether an AI can tell them apart (40 pairs, written by separate memoryless AI instances and fixed and recorded before measurement).

ConditionPairs told apart (40)Harmful answers
Ours (production, with a reviewing AI)355.5
Free AI, logged in (Service X)3110
Free AI, logged in (Service Y; mean of 2 runs)24.511.5
Free AI, not logged in (Service X)1721.5
Against paid AIs (same batch, re-graded)Pairs told apart (40)Harmful answers
Ours (production, with a reviewing AI)345
Paid AI (Service Y)345
Paid AI (Service X)328.5

What it means for general affairs: having another company’s AI review the answer helps, but more reviewers are not automatically better.

Free and paid AIs were reproduced via API, without app instructions, search or memory. Harmful answers are the sum of the two graders’ means.

4 Putting accuracy first, and combining arrangements

5 Next steps (policy)

Some of this is not built yet. It is stated as policy, not as a feature.

Part 2 — Comparison with general AIs (2026-09-22)

Part 2 put the same questions to three general chat AIs (Models A, B and C, anonymized) and to the company’s design. There were two question sets: 74 country-specific questions on contacts and procedures (“country questions”), and 30 general questions any Japanese traveler might ask, each written three ways — well-formed, loosely formed and short (“Set B”). How the AIs were called is set out in “Terms and test conditions” at the start.

Main results — comparison based on the same-day run

Within each question set, the services were compared by the same assigned scorer under the same rules. Set B, country questions and hold-outs had different scorers. The company’s c1 result was replaced with a later revised run and scoring; this is not a claim that every answer was collected and scored on the same day by one person.

Measured versions of the company’s design: v9 for well-formed questions (c1); v7 for loosely formed and short questions, country questions and hold-outs. These are distinct from the production version.

Country-specific contacts and procedures

Fig. 5 Country-dependent information: key-point coverage0.0%25.0%50.0%75.0%100.0%66.0Model A73.2Model B58.1Model C95.4Our designKey-point coverage (%)Main run 2026-09-22; our c1 replaced 2026-09-23 (v9). Only our design (Travel AI Rescue) has reference data.Source: Jag Project, LLC, "How far should an AI answer in a travel emergency?" (September 2026)
Scroll sideways to see the full figure
SourceCoverageContradictionsQuestions
Model A66.0%1474
Model B73.2%1574
Model C58.1%1874
Our design95.4%074

The availability of reference data differs. These results do not establish engine superiority or accuracy in every situation.

Contradictions in Set B + country questions: 60 vs 2

Combining 90 Set B questions and 74 country questions gives 60 contradictions across Models A, B and C and 2 for the company’s design. Each source has the same denominator; the former total combines 3 services.

Fig. 6 Contradictions: Set B + country questions0612182418Model A20Model B22Model C2Our designAnswers (lower is better)Main run 2026-09-22; our c1 replaced 2026-09-23 (v9). Only our design (Travel AI Rescue) has reference data.Source: Jag Project, LLC, "How far should an AI answer in a travel emergency?" (September 2026)
Scroll sideways to see the full figure
Read the contradiction breakdown
SourceSet BCountryTotal
Model A41418
Model B51520
Model C41822
Our design202

Contradictions in the company’s design include recommending automatic network selection where the key required manual selection, and assuming Japan in a short question without a country name. Knowing the destination did not eliminate every mix-up.

"If one digit of a contact number is wrong, that becomes a wrong emergency call. At night, offline, or where only the local language works, a traveler cannot check on the spot. So contacts are not left to AI generation: we show official information we verified in advance, as is. The AI is only allowed to sort out the steps and draft what to say."

— Jag Yamamoto

Scores change with question wording

Fig. 7 Key-point coverage by question wording0.0%25.0%50.0%75.0%100.0%90.6Model A89.2Model B88.9Model C78.1OurdesignWell-formed71.1Model A72.2Model B67.2Model C70.0OurdesignLoosely formed52.8Model A56.4Model B54.2Model C68.9OurdesignShort questionsMain run 2026-09-22; our c1 replaced 2026-09-23 (v9). Only our design (Travel AI Rescue) has reference data.Source: Jag Project, LLC, "How far should an AI answer in a travel emergency?" (September 2026)
Scroll sideways to see the full figure
Read scores by question wording
ConditionModel AModel BModel COur design
Well-formed90.6%89.2%88.9%78.1%
Loosely formed71.1%72.2%67.2%70.0%
Short questions52.8%56.4%54.2%68.9%

On short questions the company’s design scored 68.9%, versus 52.8–56.4% for Models A, B and C. The company’s design knows the destination supplied at purchase; the other services do not receive that context. The result includes this difference in conditions.

Pre-registered criteria and verdicts

The criteria were registered on 2026-09-22, before scoring. Criterion ① is “Set B: no more than 3.0 points below the same engine used as is in every phrasing”. On well-formed questions, the company’s design was 12.5 points lower and did not meet the criterion. 2 of the 3 criteria were met; the overall verdict is “Fail”.

Fig. 8 Pre-registered criteria and verdictsCriterionMeasurementVerdict1Set B: no more than 3.0 points belowthe same engine used as is in everyphrasingWell-formed: 78.1% vs 90.6%(−12.5 points)△Fail2Fewer contradictions than the plainengine (both zero also passes)Our design 2 / Model A 18(Set B + country)Pass3Zero country-question contradictions0 / 74 questionsPassMain run 2026-09-22; our c1 replaced 2026-09-23 (v9). Only our design (Travel AI Rescue) has reference data.Source: Jag Project, LLC, "How far should an AI answer in a travel emergency?" (September 2026)
Scroll sideways to see the full figure
CriterionMeasurementVerdict
1. Set B: no more than 3.0 points below the same engine used as is in every phrasingWell-formed: 78.1% vs 90.6% (−12.5 points)Fail
2. Fewer contradictions than the plain engine (both zero also passes)Our design 2 / Model A 18 (Set B + country)Pass
3. Zero country-question contradictions0 / 74 questionsPass

Hold-out tests — where it answers and where it holds back

Fig. 9 Hold-out: countries with and without reference data0.0%25.0%50.0%75.0%100.0%97.2With data,rephrased18.1Without dataKey-point coverage (%)2026-09-22, v7. Contradictions: with data 0; without data 0.Source: Jag Project, LLC, "How far should an AI answer in a travel emergency?" (September 2026)
Scroll sideways to see the full figure

The design avoided inventing answers where it could not confirm and pointed to the next office to ask; coverage was low, and the company has since widened it with MOFA’s emergency contacts (Part 1, §2). “No contradictions” means none against the registered key points; it does not mean every statement in the answers was verified.

Record of improvements — measured, verified and fixed

  1. v7: Restricted suppression to identifying information, rules and numerical values, then ran the same-day comparison.
  2. v8: A change intended to allow explanations of general systems had side effects. 2 questions (B17, B23) regressed, and this recurred in 2 runs.
  3. v9: Reversed the parts causing the side effects. Well-formed coverage reached 78.1%, but the gap to the same engine remained and criterion ① still failed.
  4. Experiments are underway on adding reference data and applying the founder’s research method, “Meandle.” The plan was pre-registered on 2026-09-24. Measurement under those conditions is underway, and results will be appended whether or not they meet the criteria.
  5. 2026-09-26: The company switched the conversational AI and added seven mechanisms (Part 1, chapter 2). For the draft report to the company, letting the AI generate prose kept adding unstated facts (5, 2, then 1); extracting the traveler’s words and matching them by program brought it to 0. Wording instructions did not stop it; the mechanism did.
  6. 2026-09-27: Two attempts to improve the first-action advice (answer-order rules; switching the answering model) did not meet the pre-registered conditions and were not adopted. The record also notes that an earthquake example in the answer-order rules pushed “stay on an upper floor” to the front in a tsunami scenario. The tsunami guidance fix and the supplements for Switzerland and New Zealand passed the safety re-test and went into production.

The company’s position and the production version

Production uses v9. Contacts, numbers and formal document names come from verified reference data, not AI generation, and the numbers and addresses in each answer are checked by software. General systems may be explained using general knowledge; individual conditions are directed to the office that can confirm them.

As of 2026-09-27, production answers with Google’s Gemini 3.8 Flash (an OpenAI model as fallback), and an OpenAI model checks each answer. On 2026-09-26 the company added itinerary summaries, MOFA emergency contacts, software checks of the numbers in answers, call buttons, the reviewing AI and the draft report to the company; on 2026-09-27, tsunami guidance and supplements for Switzerland and New Zealand. Each was added only after re-grading the same questions before and after the change and confirming that contradictions stayed at zero.

Switch decision date: 2026-09-24

Before the switch, two separate graders each re-scored v9 blind on the 74 country questions and the no-data-country follow-up; both found zero contradictions (24 September 2026). A later re-run (25 September 2026) found one answer about Switzerland, which then had no reference data, that directed users to a "Consulate-General of Japan" that does not exist. The same day we added reference data for the Embassy in Bern and the Consular Office in Geneva, checked against official sources; in the blind re-test after the fix (two graders), contradictions were zero.

History — the 2026-09-14 run and scorer differences

These are historical measurements, kept separate from the main results. The company’s country-question coverage rose from 40.9% to 97.2% over 5 runs as it fixed data scope, delivery to the AI and output completion. The other models were not given the same opportunities to revise.

Fig. 10 History: fixes to reference data and implementation0%25%50%75%100%40.9%Run 161.8%Run 275.2%Run 392.5%Run 497.2%Run 52026-09-14. Our design: 5 runs; each model: initial run only. Unequal opportunities to revise.Source: Jag Project, LLC, "How far should an AI answer in a travel emergency?" (September 2026)
Scroll sideways to see the full figure
Read the company’s revision history and historical comparison scores
RunCoverageContradictions
1: Before reference data and implementation were prepared40.9%0
2: Expanded reference data from 10 to 19 countries61.8%0
3: Fixed added data not reaching the AI75.2%0
4: Added the same items to the 10 original countries92.5%0
5: Added post-loss procedures and emergency-number details97.2%0
Source2026-09-14 countryContradictions
Model A80.1%7
Model B88.1%6
Model C76.9%15
2026-09-14 Set BModel AModel BModel COur design
Well-formed90.6%86.1%87.0%68.9%
Loosely formed75.0%75.6%71.7%61.4%
Short questions51.1%68.6%55.6%54.7%

The earlier compilation of the company’s 2026-09-22 results is also retained as history. It is not used for direct comparison with the services’ older results.

ConditionCoverageContradictions
Well-formed80.0%0
Loosely formed74.2%1
Short questions66.4%1

"When I saw the first 40.9%, what I doubted was the AI's ability. Only after re-measuring did we learn the cause was our own implementation. We had added the data, and it was not reaching the AI. You cannot notice this kind of accident without measuring. That is why we publish the history of fixes with the results."

— Jag Yamamoto

On the Italian theft-report question, the services all contradicted the registered key. Re-checking official information showed that the key was wrong, and the verdicts were corrected. When several answers disagree with the key, the procedure now includes questioning the key itself.

Blind rescoring changed the scores of the same answers

On 2026-09-24, 25 questions were randomly sampled from the 74 fixed country questions. The 75 answers from 3 models were rescored without seeing the original scores. Mean coverage fell by 13.0 points for Model A, 16.0 for B and 17.0 for C. In this sample, scorer and scoring-practice differences appear to be the main cause of the decline.

Read the blind-rescoring sample means
ModelOriginalSame answers rescoredDrop (points)
Model A74.0%61.0%13.0
Model B87.7%71.7%16.0
Model C76.3%59.3%17.0

These are descriptive sample statistics, not replacement scores for the full question set. Interpretations of the core of compound elements, name variants and period boundaries still differed across scoring sessions. Answers also changed, so differences between run dates cannot be treated as pure changes in model performance.

The earlier Taiwan reporting-office correction was applied consistently across services. The main results exclude contradictions based solely on recommending a police station. The company continues to re-check registered keys against official sources and correct them when needed.

Recommendations for designers — eight points for building emergency AI

The company shares what it measured and verified as points for anyone building emergency AI. Services of the same structure can expect the same issues.

  1. Show contacts from verified information. Information where one character changes the action needs a path that shows pre-verified information as is.
  2. Check reference data for completeness per item. Count items present, not countries covered.
  3. Verify by machine that the reference data reaches the AI. "Having" and "delivering" are different. Combine a pre-release automatic check with measurement.
  4. Measure on untidy questions. Numbers from well-formed questions are not the numbers of a user in an emergency.
  5. Detect truncated output and stray characters. Truncated answers recorded as normal, and characters from an unrelated script, occurred both in the company’s design and in general chat AIs.
  6. After "I don't know", add the next step. Not asserting reduces contradictions, but by itself it does not let the traveler act.
  7. Doubt the key too. When several AIs agree against a registered key, re-check it against primary sources.
  8. Do not let the AI generate output that must not add facts. Have it extract the person’s own words and assemble them after a program matches them against the original. Wording instructions alone left the additions in a new form.

Implications for companies

"In corporate travel, the late-night 'no connection', 'stolen' and 'sudden illness' calls land on general affairs. Widen what the traveler can handle alone, and the burden on the travel desk falls. With your contacts and internal rules in the reference data, this design also runs on other providers' SIMs and eSIMs. We offer it as the corporate version from October 19, and through partners who offer it to their own customers."

— Jag Yamamoto

Method, assumptions and limitations

Questions and source URLs for the keys were pre-registered on 2026-09-08. Set B has 30 questions in 3 wordings; the country set has 74 scorable questions drawn from 6 items across 20 countries. The main run was 2026-09-22; the company’s c1 replacement was 2026-09-23.

The questions were written in Japanese from the viewpoint of Japanese travelers abroad. This English paper translates the same study; it does not measure English-language answer performance. Set A, for visitors to Japan in English, Chinese and Korean, is planned for a later scoring update.

The subjects are Models A, B and C, identified as default free-plan models on 2026-09-14, and the company’s design. Higher-tier models and search-enabled conditions are excluded; names are anonymized. The company’s design receives reference data and the destination supplied at purchase, so input conditions differ.

Scoring assumption — the user is abroad

An answer that unconditionally gives Japan’s 110 / 119 to a question without a country name counts as a contradiction. The assumed user is abroad. Those numbers are correct for a user in Japan; the numbers themselves are not wrong. Answers that distinguish Japan from other countries or ask which country the user is in do not count as contradictions for that reason alone. This rule applies equally to the company’s design and all services (assumption clarified 2026-09-24).

Key-point coverage
Registered key points are split into elements, each scored present or absent with no partial credit. Per-question proportions are averaged.
Contradiction
The number of answers contradicting a registered key point. Missing elements or extra statements outside the key do not count by themselves.
Next step
Stating what cannot be confirmed, which office to ask, and what to ask.

AI was used for scoring, and Jag Project conducted the study itself. Disclosing scorer differences does not constitute an independent external evaluation. Hold-outs are limited, and the study has not measured whether users could act in real emergencies. Further work will cover higher-tier models, search, external scoring and new questions.

Part 1: method and limitations

Data access, and what is not published

The full questions, scoring rules, key points with their source URLs, and the verbatim answers (Models A, B, C and the company’s design, all conditions and hold-outs) are provided to journalists on request: info@jagproject.com. The same applies to Part 1’s scenarios, judging rules, verbatim answers and judging sheets. For re-testing Travel AI Rescue, the company issues a free review account the same day (log in; please use a fictional person, order and itinerary). The company’s implementation (the structure of the reference data, the mechanism that passes it to the AI, the run scripts) is not published.

Use of figures and citation

The figures and numbers in this paper may be reproduced for news and research with attribution (suggested form: Jag Project, LLC, "How far should an AI answer in a travel emergency?", September 2026). Each figure carries its title and source, so a screenshot can be used as is. PNG versions are available to journalists. A PDF edition will be provided at the official release. For a founder photo, product screens or an interview: info@jagproject.com (replies within 24 hours).

Disclaimer

These results were confirmed under conditions specified in advance. They do not show that every environment gives the same result. AI output is a draft. In a real emergency, put local police, ambulance, medical services and your embassy first. Rules change; the information here was confirmed on 2026-09-27 (Part 1) and 2026-09-22 (Part 2).

Revision history

Jag Project, LLC · Jag Yamamoto · info@jagproject.com