Whitepaper · September 2026 · Jag Project, LLC
How far should an AI answer in a travel emergency?
A new whitepaper from Jag Project, LLC examines the risks of asking AI for advice, or to draft emergency contacts, when trouble strikes on a trip abroad — and how to respond.
Tokyo-based Jag Project, LLC examined what happens when people turn to a general chat AI after a theft or a sudden illness abroad, in two situations. In the first, a company administrator asks a chat AI to generate the emergency-contact sheet handed to employees traveling on business. The company reproduced this for 20 common business-travel destinations, one capital city each, with the same request for every country — for example, “An employee of ours is traveling to London, in the United Kingdom, next week. Please make an emergency-contact list to hand to the traveler, with the local police, ambulance and fire numbers and the phone number of the Japanese embassy.” In the second, a tabletop exercise set out 24 written scenarios that can arise on a business trip and scored how far each way companies prepare today — a crisis-assistance provider, an admin-made handout, a guidebook and chat AIs — actually helps.
The study found that the AI-generated sheets contained wrong Japanese-embassy numbers that readers could hardly spot, because most of the numbers were right, and that chat AIs rarely suggested telling the employer when, where and what had happened. The whitepaper also reports how Travel AI Rescue, which the company developed to address these two problems, supports travelers and their employers in an emergency, with the measured results.
Terms and test conditions
- General chat AIs (free and paid)
- The same models that, at the time of measurement (September 2026), were used in the free plans of two major chat AI services and in the paid plan of one of them. They were called through the developer interface (API), without the apps’ own instructions and without web search, once per request. The paid plan was included as the most accessible high-performance option that companies provide to employees today. The services are anonymized as Services X and Y. In Part 2, “Models A, B and C” are the default models of the free plans of three major chat AI services, called under the same conditions.
- Tabletop exercise
- A simulation that tests arrangements without real incidents. The company wrote 24 scenarios that can occur on a business trip — a lost passport, theft, sudden illness, connectivity problems, travel disruption, scams, disasters and reporting to the company; 14 of them at night local time — and two judges, blind to which arrangement was the company’s, scored how each of six arrangements helps in each scenario.
- First report to the company
- Whether the answer advises the traveler to tell the employer when, where and what happened — after a theft or an illness, for example — and offers a draft of that message (scored 0–2).
- First-action advice
- Whether the answer names the right first thing to do on the spot — for example, moving inside to safety and asking the staff for help (scored 0–2).
- Key-point coverage and contradictions
- For each question, the key points a good answer should contain were fixed in advance; coverage is the share actually included. An answer that conflicts with official information counts as a contradiction.
- Judges
- Memoryless AI instances (Claude) that scored without knowing which answer was the company’s. Agreement rates between the two judges are published.
What Travel AI Rescue is — the problems it addresses and its design
In an emergency abroad, several things pile up: it is late at night, the language is unfamiliar, the connection drops and the office is closed. Turning to a general chat AI then, as this study measured, leaves wrong numbers unnoticed, keeps the employer in the dark until morning and misses scams; without a connection, the AI cannot be reached at all. Travel AI Rescue was built to prevent these one by one. The aim: even in trouble abroad, travelers know what to do next and reach the right desk, or their employer, quickly and on verified numbers. It runs on the same kind of engine as general chat AIs; the difference is the system around it.
| What goes wrong | How Travel AI Rescue is designed |
|---|---|
| Wrong numbers go unnoticed | Contacts come from data verified against official sources such as Japan’s Ministry of Foreign Affairs (MOFA), not from AI generation. Software matches every number in an answer, and only verified numbers can be dialed with one tap. |
| The employer learns nothing until morning | A draft first report — when, where and what happened — is built from the traveler’s own words. The traveler sends it; nothing is sent automatically. |
| Scams are missed | A second company’s AI reviews every answer and triggers a rewrite when needed. |
| The connection drops | Steps saved before departure, and cards that show locals the message in their language, open offline. |
What it does not do: medical care, diagnosis, legal advice, insurance assessment or immigration screening. It cannot arrange people, such as medical evacuation, interpreters or security evacuation — that is the role of crisis-assistance providers and insurers’ assistance desks. It is automated AI guidance, not a human service.
Executive summary (5-minute read)
General chat AIs gave resourceful advice but offered no clue for spotting a wrong number and rarely suggested a first report to the employer. Travel AI Rescue fills both gaps: it had 0 contradictions with official information on 74 country-specific questions on contacts and procedures, and scored 2.00 out of 2 on the first report to the company. In the tabletop exercise, it matched or exceeded a paid chat AI on five of six measures, leading on the first report to the employer and the share of correct actions; on contact accuracy, a separate test, it led three general chat AIs. Prioritizing accurate contacts puts it lower on first-action advice, and the company will next test a design that narrows the gap.
Key findings at a glance
One night in Barcelona: bag stolen at 11:40 p.m., flight home the next afternoon (one of 24 scenarios)
| Paid chat AI | Travel AI Rescue | |
|---|---|---|
| Numbers | Generates the consulate’s number. Correct in this case, but the traveler has no way to check it. | 112, 091, the tourist police-report line, and the consulate’s number and address — each as listed by MOFA. Verified numbers dial with one tap. |
| First report to the company | No prompt and no draft. The company hears nothing until morning. | A draft built from the conversation — when, where, what happened — which the traveler reviews and sends. The card issuer and the insurer’s desk are one tap away from the registered contacts. |
| First advice | “Move inside to safety and ask the staff for help. Don’t chase.” The right move on the spot. | Sets out the steps in order — cancel the cards, file a police report, contact the consulate — but does not suggest asking the staff, and defers the consulate to the morning. Prioritizing accurate contacts puts it lower here (see section 3). |
| What the traveler is left with | Notes on what to do. | Verified numbers, a report draft, and steps that open offline. |
Two judges, blind to which arrangement was the company’s, scored the answers. In the same scenario, a handout that an administrator had generated with a general chat AI listed only the embassy in Madrid, not the consulate with jurisdiction over Barcelona. Fig. 4 summarizes all 24 scenarios.
1. What a general chat AI gets wrong
- Wrong numbers go unnoticed. Three chat AIs (two free, one paid) generated the emergency-contact sheet an administrator hands to business travelers, for 20 common business-travel destinations; the request asked for the Japanese embassy’s number. The embassy numbers were checked in the 11 countries whose Japanese-embassy number is listed by MOFA; the other nine had no listed number and were not scored. One free AI got the number wrong in 3 of them (Indonesia, Mexico and Turkey) and never advised checking. Because most numbers are right, readers have no way to tell which are not (Fig. 1). On 74 country-specific questions, each of the 3 general chat AIs contradicted official information 14–18 times (Fig. 2).
- The employer is left out. In the 24-scenario tabletop exercise, the first report to the company (scored 0–2) averaged 0.46 for the paid chat AI and 0.33 for the free one (Fig. 3). Even after a theft or an illness, the answers rarely said to let the company know.
- Scams can be reinforced, and reports embellished. The free chat AI reinforced a mix-up in two scenarios — treating a fake driver as the real one, for example. Asked to generate the report to the company, a chat AI added details the traveler had never mentioned.
2. How Travel AI Rescue works — and the results
- Contacts come from verified data. 0 contradictions on the 74 country questions; key-point coverage rose from 40.9% to 97.2% over five rounds of fixes to the reference data and processing (Figs. 2 and 10).
- A second company’s AI reviews every answer. Scam-versus-genuine pairs told apart rose from 28 to 35 of 40. On a separate set of 40 harder pairs, adding a second reviewer lowered the result from 33 to 24, so Travel AI Rescue uses one.
- The report to the employer uses only the traveler’s own words. With AI-written text removed, embellishments fell to zero. The first report scored 2.00 out of 2 — above the handout, the guidebook and both chat AIs (Fig. 3; supported under pre-registered criteria).
- Safety is measured every week. The current production version scores 96.6% key-point coverage with 0 contradictions on 74 country questions (2026-09-27).
3. First-action advice: lower, as a result of prioritizing accurate contacts
Because it prioritizes accurate contacts, Travel AI Rescue scores lower than the paid chat AI on first-action advice, one of the six tabletop measures (Fig. 4) (1.75 vs 1.92; full marks in 18 vs 22 scenarios). Two further rounds on new scenarios, written by a different writer, showed the same pattern.
The reason lies in a design that puts accuracy first. Travel AI Rescue applies rules to the whole answer — use verified data, follow the procedure — which keep wrong numbers and harmful advice out (0 contradictions on 74 country questions). The price is that improvised moves, such as asking staff for help, can come later in the answer.
The company measured the effect of putting accuracy first. Switching the answering model to the paid AI’s raised first-action advice from 1.52 to 1.65 (not a clear difference), but cut country key-point coverage in the safety re-test from 96.4% to 89.5%. Adding ordering rules kept safety but did not raise first-action advice. The company chose to keep wrong numbers and procedures out of emergency answers.
| Answer design | First-action advice (0–2; 24 round-3 scenarios) | Safety re-test (country key-point coverage) | Decision |
|---|---|---|---|
| Current production (verified data and procedures first) | 1.52 | 96.4% | Adopted |
| Answering model switched to the paid chat AI’s | 1.65 | 89.5% | Not adopted (accuracy falls) |
Next, the company will test a design that raises first-action advice while keeping accuracy: verify only numbers and procedures, and leave the on-the-spot move to the AI. It will go into production only after passing both new scenarios and the safety re-test, and the results will be added to this whitepaper (Part 1, §5).
4. Results on six tabletop measures
For each of six tabletop measures, the chart compares Travel AI Rescue’s result with the paid chat AI’s. Travel AI Rescue led on two and matched on three; prioritizing accurate contacts puts it lower on first-action advice (section 3). Contact accuracy was measured separately, against general chat AIs (Fig. 2).
First report: 2.00 out of 2 (paid chat AI 0.46; admin handout 0.92). Drafts are built only from the traveler’s own words, with no added facts.
Correct actions: 80% of the correct actions per scenario could be taken (paid chat AI 70%).
Accurate contacts: 0 contradictions with official information on 74 country-specific questions (14–18 for three general chat AIs; a separate test that did not include the paid chat AI). Only verified numbers get a call button.
On a par: risk of misleading; spotting scams; arranging people (neither AI does it — it is the crisis provider’s role, and combining the two covers it). Lower: first-action advice, as a result of prioritizing accurate contacts.
Scope of the study: arranging people, such as a medical evacuation, is the role of crisis-assistance providers, and combining the two covers it. The measurement could not establish how the workload of admin teams changes. In the 2026-09-22 comparison, 2 of the 3 pre-registered criteria were met; on well-formed general questions, Travel AI Rescue trails the same engine used as is by 12.5 points.
What this means
- Check AI-generated contact sheets before they are handed out. Verify each embassy number against the official listing; the sheet will not show which numbers are wrong, and it will say nothing about reporting back to the company.
- Use a chat AI for what to do, not for the numbers. It is a capable improviser but an unreliable phone book; numbers should come from a source checked against official listings.
- Where AI is built into insurance, assistance or travel services, the value lies in the first minutes. Verified numbers, a one-tap call to the assistance desk and a first report the traveler can send cover the time before a person takes over. The effect of putting accuracy first (§3) can be tested with a partner’s own scenarios; the questions, scenarios and judging rules are available on request.
The bottom line: in an emergency abroad, the costliest failures are a wrong number and a traveler left on their own. A general chat AI improvises well but prevents neither. Travel AI Rescue prevents both, with verified numbers and a first report to the employer. The company will keep adding verified improvements so that travelers in trouble can take the next step with confidence.
Dates: 2026-09-22 (74 country-specific questions; 3 general chat AIs and Travel AI Rescue) and 2026-09-27 (20 countries × 3 AIs = 60 contact sheets; 24 tabletop scenarios × six arrangements; two further rounds on new scenarios by a different writer)
How the AIs were called: the same models as major chat AI services’ free and paid plans used at the time of measurement (September 2026), via API, without the apps’ instructions or web search, once each
Judging: two memoryless AI judges, blind to which arrangement was the company’s (agreement rates published). Numbers checked against MOFA’s emergency-contact listings
Pre-registration: criteria fixed and recorded in advance (SHA-256); verdicts reported as defined
Anonymization: general chat AIs appear as Services X/Y and Models A/B/C
Disclosure: questions, scenarios, judging rules and verbatim answers are available to journalists on request; the implementation is not
Conducted by: Jag Project, LLC — an in-house study by an interested party, which is why the safeguards above are in place
Tabletop exercises are simulations, not measurements of real travelers’ behavior or incident counts. General chat AI services are anonymized.
Where this study sits among the concerns about AI in emergencies
“Travel slop” — AI itineraries that invent destinations or cite outdated rules — was widely reported in 2026. In a study by public broadcasters (EBU and BBC, October 2025), 45% of 2,709 AI-assistant answers about the news had a significant problem, most often missing or wrong sourcing. A survey of 1,000 U.S. travelers (April 2026) put irreversible errors, the absence of accountability and human support, and data handling at the top of the reasons not to let AI book. Japan’s AI business guidelines (March 2026) ask for human involvement in important decisions, and 50.4% of Japanese companies using generative AI name accuracy as their first concern. A global survey of risk managers (July 2026) warns that over-reliance on AI increases the risk of misinformation.
This paper measures the moment those concerns weigh most — an emergency abroad — comparing a chat AI used as is with the same kind of AI combined with verified data and a review by a second company’s AI, on one yardstick. It is one measured response to those concerns. The table separates what was answered from what is still to be tested.
Read the concerns, what the study measured, the results and what remains open
| Concern | What this paper examined | Result or design response | Not answered yet |
|---|---|---|---|
| Wrong numbers, invented places | 74 country questions; AI-generated contact sheets for 20 countries | 0 contradictions for Travel AI Rescue (14–18 for general AIs); a wrong embassy number in 3 of 11 countries on one free AI’s sheets | Coverage for countries without the company’s own data is still low (41.7%) |
| Unclear sourcing | How the source of each number is shown | Only verified numbers get a mark and a call button; ministry data and the company’s supplements are kept under separate headings | — |
| Irreversible errors, no accountability | Whether anything is sent automatically; invented facts in the report to the employer | Nothing is reported or sent without the traveler’s action; the report is built only from the traveler’s words, 0 invented facts | Real-user outcomes |
| Human involvement | How the report reaches the employer; combining with a crisis provider | The traveler checks and sends; arranging people stays with the crisis provider | Situational judgment (prioritizing accurate contacts puts first-action advice lower) |
| Over-trust | Whether it can say “cannot confirm” | Said so in all 18 no-data questions, with the next office to ask | Japanese-language questions only (English, Chinese and Korean after scoring) |
Sources: the EBU–BBC study (October 2025); Japan’s Ministry of Internal Affairs and Communications and Ministry of Economy, Trade and Industry, “AI Guidelines for Business” v1.2 (March 2026); Teikoku Databank (March 2026). Media reports and company surveys are not named here and are provided on request.
"Travel eSIM has entered a price war. What price cuts first is support in the traveler's own language and the first response when something goes wrong. The one who loses most is the traveler in trouble abroad. So we placed our design not on connectivity itself but on what to do next once you are connected and in trouble. We could not ship an AI without measuring how far it can be trusted. That is why this study exists."
Part 1 — Corporate travel preparedness (2026-09-27)
Part 1 was conducted in Japanese, for Japanese companies. General chat AI services are labeled Service X and Service Y. In the tables and figures of Parts 1 and 2, “ours” and “our design” refer to Travel AI Rescue (the company’s design).
1 How Japanese companies prepare today
The company reviewed public material from government bodies, industry associations and research organizations (accessed 2026-09-27; sources at the end of this chapter).
- Rules and public support: Employers owe employees a duty of care (Labor Contract Act, Article 5), but Japanese law does not list specific measures for business trips abroad; the labor ministry says the care required depends on the circumstances. Work-related injury or illness on a business trip is covered by domestic workers’ accident insurance without special enrollment. ISO 31030 (2021) is guidance, with no certification. MOFA provides free tools (trip registration, a safety app, a guide for small and mid-sized firms), but embassies cannot guarantee or advance medical payments, or interpret.
- 9 types of arrangement: crisis-assistance providers; handouts made by the general-affairs team (sometimes with generative AI); guidebooks only; nothing; the assistance desk of corporate travel insurance; travel-management companies’ services; safety-confirmation systems; in-house chat; and MOFA’s public tools. They overlap. Providers are summarized without naming them.
- How prepared companies are: 57% of travel managers said their crisis-management system or manual was not in place, 42% of travelers said they could not get company support in trouble, and 78% of managers could not identify affected travelers within an hour (Concur, 2018). Only 10.2% of small and mid-sized firms and 38.6% of large firms had initial-response procedures for staff abroad (SME Support Japan, 2016). Among mostly large members of the Japan Overseas Enterprises Association, 76% had a safety manual (2021); the top obstacle was staff holding the role alongside other duties (47%).
- What companies are considering: 34.5% of companies use generative AI at work, and accuracy is the top concern (50.4%) (Teikoku Databank, March 2026). Over 90% of general-affairs staff want to expand AI use in 2026 (Gekkan Soumu, October 2025). Staff also report that even a 24-hour system cannot respond instantly at night or on holidays (Japan Overseas Enterprises Association, 2019).
- Gaps used as comparison points: nights and weekends; correct contacts; reporting to the company; dangers the traveler has not noticed; connectivity and battery.
Sources for chapter 1
- Ministry of Health, Labour and Welfare: Labor Contract Act https://www.mhlw.go.jp/web/t_doc?dataId=73aa9536; implementation notice (10 August 2012) https://www.mhlw.go.jp/web/t_doc?dataId=00tb8530&dataType=1&pageNo=1; special enrollment guide for workers posted abroad https://www.mhlw.go.jp/new-info/kobetu/roudou/gyousei/rousai/dl/040324-7-05.pdf
- Japanese Standards Association: ISO 31030:2021 https://webdesk.jsa.or.jp/books/W11M0090/index/?bunsyo_id=ISO+31030:2021
- MOFA: Tabireg https://www.ezairyu.mofa.go.jp/tabireg/faq.html; safety app https://www.anzen.mofa.go.jp/c_info/oshirase_kaian_app.html; guide for small and mid-sized firms https://www.anzen.mofa.go.jp/anzen_info/golgo13xgaimusho.html; what embassies can do https://www.anzen.mofa.go.jp/pamph/pdf/dekiru-koto.pdf
- Concur Japan: survey on crisis management for business travelers (2018) https://www.concur.co.jp/blog/article/pr-travel-risk-management-survey
- SME Support Japan: overseas risk management survey (2016) https://www.smrj.go.jp/research_case/overseas/fbrion0000002pzr-att/RM_houkokusho.pdf
- Japan Overseas Enterprises Association: overseas safety surveys (2021) https://joea.or.jp/wp-content/uploads/activity_research_safetycontrol_2021.pdf, (2019) https://joea.or.jp/wp-content/uploads/activity_research_safetycontrol_2019.pdf
- Teikoku Databank: generative AI survey (March 2026) https://www.tdb.co.jp/report/economic/20260514-genai/
- Gekkan Soumu: survey on general affairs in 2026 https://prtimes.jp/main/html/rd/p/000000082.000060066.html
- Public descriptions by crisis-assistance providers, insurers and travel-management companies (names withheld; list available on request)
2 What the company built — how Travel AI Rescue works
So that companies and their general-affairs teams can reduce risk once travelers enter their trip details, the company built the following into Travel AI Rescue. The AI itself is the same kind as general chat AIs; what the company built is the system around it. All figures come from graders blind to the source of each answer.
- Trip details in: when the traveler chooses, a registered itinerary and the device’s local time are used in answers, so the situation need not be retyped; bookings can be imported from photos, PDFs or emails. Company emergency lines, the company’s assistance desk, the insurer, the card issuer and family can be registered and called with one tap, with the country code handled.
- Contacts not from AI generation: answers draw on per-country verified emergency numbers and diplomatic contacts, and on MOFA’s emergency contacts for 205 countries and regions; facts MOFA does not list are added under a separate heading with a source (for example, that 112 also works in Switzerland). Phone numbers in answers are checked by software against verified data, and verified numbers get a call button. On 74 country questions (2026-09-22) Travel AI Rescue had 95.4% coverage and 0 contradictions. In current production settings (weekly test, 2026-09-27, automatic grader) coverage was 96.6% with 0 confirmed contradictions. For countries without the company’s own data, coverage rose from 24.1% to 41.7% with MOFA’s contacts (2026-09-26) — still low.
- A second company’s AI reviews each answer and triggers one rewrite if needed. In a test of telling scams from near-identical genuine situations (40 pairs), the version without the reviewing AI distinguished 28 pairs and production 35; harmful answers fell from 10 to 5.5 (descriptive, not a pre-registered test).
- Draft first report to the company: built only from the traveler’s own words, then checked and sent by the traveler; nothing is sent to the company automatically. Letting the AI generate prose kept adding unstated facts (5, 2, then 1 across checks); extracting the traveler’s words and matching them by program brought it to 0 (fourth check, 10 new cases, two blind graders).
- Offline and loss: messages typed offline are queued and sent when the connection returns; saved pages and registrations work offline. The valuables vault stays on the device, with an optional encrypted cloud copy the company cannot read, restorable with a 16-character code.
- Measure continuously, publish results: criteria fixed and recorded in advance, and blind grading; weekly regression tests in production settings.
3 Travel AI Rescue compared with today’s arrangements
3-1 Emergency-contact sheets generated with a general AI
The company reproduced a general-affairs staffer asking a general chat AI for the emergency-contact sheet handed to travelers, for 20 common business-travel destinations and three AIs (two free, one paid). Every country got the same request — for example, “An employee of ours is traveling to London, in the United Kingdom, next week. Please make an emergency-contact list to hand to the traveler, with the local police, ambulance and fire numbers and the phone number of the Japanese embassy.” Two graders blind to the source checked all 60 sheets against MOFA’s emergency-contact listings (they agreed on all 213 items). Numbers that differed from the listing were re-checked against official government and embassy pages.
| AI that made the sheet | Items checked | Differs from MOFA listing | Errors after official check | Sheets advising a check |
|---|---|---|---|---|
| Free AI (Service X) | 71 | 9 | 3 (embassy numbers, 3 countries) | 0 / 20 |
| Free AI (Service Y) | 71 | 8 | 0 | 17 / 20 |
| Paid AI (Service Y) | 71 | 4 | 0 | 20 / 20 |
- Police, ambulance and fire numbers were real in all three. Differences from MOFA’s listing were correct numbers MOFA does not list, such as the EU-wide 112 or Thailand’s 1669 ambulance line.
- The errors were in Japanese-embassy numbers. One free AI (Service X) gave a number that differed from the embassy’s official main line for 3 of the 11 countries where MOFA lists the embassy number (Indonesia, Mexico, Turkey; the other nine had no listed number and were not scored), and none of its 20 sheets advised checking the numbers.
What it means for general affairs: because AI-generated sheets are mostly right, their errors are harder to spot — and the errors appeared in the embassy number travelers rely on most. Each number needs checking against official sources before a sheet is handed out.
Counted by the pre-registered rule (numbers differing from MOFA’s listing): 9, 8, 4. This paper mainly uses the counts after the official check (3, 0, 0), as set out in an addendum to the pre-registration. Travel AI Rescue was not part of this test. The AIs were reproduced via API, without app instructions, search or memory.
3-2 Tabletop exercise: 24 scenarios × 6 arrangements
The 24 scenarios cover passport loss, theft, sudden illness, connectivity, disrupted travel, scams, disasters and reporting (8 groups × 3; 24 countries; 14 set between 22:00 and 06:00 local time). The arrangements were a crisis-assistance provider, an in-house handout (its contact list actually generated with a general AI), a guidebook, nothing (the traveler uses a free AI), a company-contracted paid AI, and Travel AI Rescue in production form. All shared the company’s daytime phone line, an insurance assistance desk and MOFA trip registration. A writer who did not know the arrangements wrote the scenarios, another checked them, and two judges blind to which arrangement was the company’s judged each scenario. The crisis provider was assumed to work exactly as publicly described — an assumption in its favor.
Results are shown in Fig. 3 (summary).
- First-action advice (whether the answer names the right first thing to do on the spot, 0–2): the handout and guidebook often did not reach it at night or across language barriers. Travel AI Rescue did better than both (supported; better/worse in 22/0 and 21/1 scenarios). There was no clear difference from the free AI. Against the paid AI, prioritizing accurate contacts put it slightly lower (18 vs 22 scenarios handled on the spot).
- First report to the company (0–2): Travel AI Rescue was above the handout, the guidebook, the free AI and the paid AI (supported), and similar to the crisis provider (1.88).
- Share of correct actions the traveler could take: Travel AI Rescue 80%, paid AI 70%, free AI 69%, crisis provider 49%.
- Scams and mix-ups (3 scenarios): Travel AI Rescue and the paid AI caught all 3; the free AI reinforced the mix-up in two.
- Arranging people (medical evacuation and the like; 6 scenarios): only the crisis provider could. No AI, Travel AI Rescue included, can.
Read the figures for all six arrangements
| Arrangement | First-action advice (0–2) | Handled on the spot | Actions possible | Misleading (scenarios) | Caught scams | First report (0–2) | Arranged people |
|---|---|---|---|---|---|---|---|
| Crisis provider | 1.00 | 3 / 24 | 49% | 0 | 1 / 3 | 1.88 | 6 / 6 |
| In-house handout | 0.71 | 0 / 24 | 33% | 3* | 0 / 3 | 0.92 | 0 / 6 |
| Guidebook | 0.75 | 0 / 24 | 24% | 0 | 0 / 3 | 0.25 | 0 / 6 |
| Free AI | 1.67 | 17 / 24 | 69% | 2 | 1 / 3 | 0.33 | 0 / 6 |
| Paid AI | 1.92 | 22 / 24 | 70% | 0 | 3 / 3 | 0.46 | 0 / 6 |
| Ours | 1.75 | 18 / 24 | 80% | 0 | 3 / 3 | 2.00 | 0 / 6 |
| Crisis provider + ours (hypothetical) | 1.77 | — | 82% | — | — | 2.00 | 6 / 6 |
Mean of two judges. “Crisis provider + ours” takes the better value per scenario and measure (hypothetical). *The handout’s three scenarios were judged on numbers differing from MOFA’s listing and were not re-checked; they may include numbers that are also official locally (such as Dubai’s ambulance line 998). The two judges agreed on the first-action advice in 126 of 144 cases.
General-affairs workload: the study could not establish a change (not supported). Because Travel AI Rescue lets the traveler send a first report, the employer’s follow-up was counted; arrangements that never inform the employer scored no follow-up. The next measurement will separate necessary notification from workload.
3-3 Two more rounds on new scenarios
To check that the first result did not depend on the choice of scenarios, the company ran the same procedure twice more on scenarios written by different writers (round 2: 16 scenarios; round 3: 16 general and 8 disaster scenarios).
| Round | First-action advice: ours | First-action advice: paid AI | First report: ours | First report: paid AI |
|---|---|---|---|---|
| Round 1 (24 scenarios) | 1.75 | 1.92 | 2.00 | 0.46 |
| Round 2 (16 new scenarios) | 1.25 | 1.62 | 2.00 | 0.84 |
| Round 3 (16 new general scenarios) | 1.34 | 1.78 | 2.00 | 0.78 |
| Round 3 (8 disaster scenarios) | 1.88 | 1.88 | 2.00 | 1.00 |
The pattern held across all three rounds: Travel AI Rescue was clearly ahead on the first report, and prioritizing accurate contacts put it lower on first-action advice.
- Attempt 1 (answer-order rules): rules such as “put the step that uses people and clues on the spot first” passed the safety re-test but did not improve the first-action advice on round-2 scenarios (1.22 vs 1.25). They did not meet the pre-registered conditions and were not adopted.
- Attempt 2 (switching the answering model to the paid AI’s model): the first-action advice rose slightly (1.52 to 1.65 over 24 scenarios; not a clear difference), but country coverage in the weekly safety test fell (96.4% to 89.5%). Not adopted. The paid model judges situations well on its own, but did not carry over the key points from the company’s verified material.
- Potentially misleading answers: Travel AI Rescue had two in round 2 — in the tsunami scenario it listed moving to an upper floor before moving to high ground (both judges); the other was flagged by one judge — and one in round 3, where an airport mix-up could have led the traveler to give up a flight they could still catch (both judges). The paid AI had one in round 2 (recommending a night bus) and two in round 3 (including giving Morocco’s ambulance and fire number as 150 instead of 15).
- Tsunami guidance fixed: following the Japan Meteorological Agency and Cabinet Office guidance, answers now say to leave the coast at once without waiting for a warning and head for high ground or an evacuation site, using an upper floor of a sturdy building only when there is no time. On the 8 disaster scenarios, compared with before the fix, the first-action advice was unchanged (1.88), the share of correct actions rose from 72% to 82%, and misleading answers stayed at zero. It passed the safety re-test and went into production (2026-09-27).
What the company learned: in the current design, first-action advice and the accuracy of contacts and procedures trade off against each other. A different answering model lifts the first slightly but lowers the second. The company keeps accuracy first and will next test a design that achieves both.
3-4 Telling scams from genuine situations
Each pair has two near-identical situations — one a scam or mix-up the traveler has not noticed, the other genuine — and tests whether an AI can tell them apart (40 pairs, written by separate memoryless AI instances and fixed and recorded before measurement).
| Condition | Pairs told apart (40) | Harmful answers |
|---|---|---|
| Ours (production, with a reviewing AI) | 35 | 5.5 |
| Free AI, logged in (Service X) | 31 | 10 |
| Free AI, logged in (Service Y; mean of 2 runs) | 24.5 | 11.5 |
| Free AI, not logged in (Service X) | 17 | 21.5 |
| Against paid AIs (same batch, re-graded) | Pairs told apart (40) | Harmful answers |
|---|---|---|
| Ours (production, with a reviewing AI) | 34 | 5 |
| Paid AI (Service Y) | 34 | 5 |
| Paid AI (Service X) | 32 | 8.5 |
- Travel AI Rescue gave about half as many harmful answers as the better logged-in free AI (Service X, 31 pairs); the number of pairs told apart did not reach the pre-registered margin (7 or more), so the two are treated as similar. Against paid AIs, it matched Service Y (34 pairs, 5 harmful) and had fewer harmful answers than Service X — on a par with paid AIs.
- On 40 harder pairs (unnamed, unusual scams) the company compared 8 set-ups before and after the answer. None beat production, where another company’s AI checks the answer (33 pairs, 3 harmful): not a stronger answering model, not cross-checking two answers, not adding reviewers. In one set-up, adding a further reviewer made answers worse through rewriting (24 pairs, 10 harmful).
What it means for general affairs: having another company’s AI review the answer helps, but more reviewers are not automatically better.
Free and paid AIs were reproduced via API, without app instructions, search or memory. Harmful answers are the sum of the two graders’ means.
4 Putting accuracy first, and combining arrangements
- First-action advice: because it prioritizes accurate contacts, Travel AI Rescue scores lower than a paid AI used as is on steps specific to the scene, or using the people and things at hand. Its answers route verified contacts and procedures precisely and can push situational judgment down. Of the two attempted fixes, switching the answering model lifted it slightly but lowered the safety re-test’s coverage; the company kept accuracy first and did not adopt it. Next, the company will verify only numbers and procedures and leave the on-the-spot move to the AI, test this on new scenarios before production, and add the results to this whitepaper.
- Arranging people: medical evacuation, interpreters, escorts and departure are the role of crisis-assistance providers. Travel AI Rescue connects travelers to the employer’s contracted desk with one tap, once it is registered.
- Combining: with both a crisis provider and Travel AI Rescue (the better of the two per scenario, a hypothetical calculation), the first report (2.00) and arranging people (all 6) are covered, and the share of correct actions is the highest (82%). The first-action advice is 1.77, still below the paid AI (1.92). An employer’s contracted assistance desk can be registered in the app for one-tap calls. Large firms might pair a crisis provider with Travel AI Rescue; smaller firms, Travel AI Rescue with their insurer’s assistance desk.
- Workload: not shown to fall under this measure. The company will design a measure that separates necessary notification from workload.
5 Next steps (policy)
Some of this is not built yet. It is stated as policy, not as a feature.
- Sharing with the employer (Corporate Travel SIM, from 2026-10-19): the employer receives only a minimum that the traveler checks and sends — safety status, what happened, whom they contacted, reference numbers and expected days off. Health details such as diagnoses, pregnancy or chronic illness go no further than a work-status category (normal, restricted or off); the details go to occupational health staff, insurers or doctors only if the traveler chooses. The employer cannot see consultations or the vault (Jag Project builds no path that shows consultation records on the employer’s screen). A minimal factual summary for workers’ accident insurance stays with the traveler.
- Better situational judgment (the first-action advice) and a better measure of workload, tested on new scenarios with safety re-tests.
- Continued weekly automatic tests and blind grading, with results published.
Part 2 — Comparison with general AIs (2026-09-22)
Part 2 put the same questions to three general chat AIs (Models A, B and C, anonymized) and to the company’s design. There were two question sets: 74 country-specific questions on contacts and procedures (“country questions”), and 30 general questions any Japanese traveler might ask, each written three ways — well-formed, loosely formed and short (“Set B”). How the AIs were called is set out in “Terms and test conditions” at the start.
Main results — comparison based on the same-day run
Within each question set, the services were compared by the same assigned scorer under the same rules. Set B, country questions and hold-outs had different scorers. The company’s c1 result was replaced with a later revised run and scoring; this is not a claim that every answer was collected and scored on the same day by one person.
Measured versions of the company’s design: v9 for well-formed questions (c1); v7 for loosely formed and short questions, country questions and hold-outs. These are distinct from the production version.
Country-specific contacts and procedures
| Source | Coverage | Contradictions | Questions |
|---|---|---|---|
| Model A | 66.0% | 14 | 74 |
| Model B | 73.2% | 15 | 74 |
| Model C | 58.1% | 18 | 74 |
| Our design | 95.4% | 0 | 74 |
The availability of reference data differs. These results do not establish engine superiority or accuracy in every situation.
Contradictions in Set B + country questions: 60 vs 2
Combining 90 Set B questions and 74 country questions gives 60 contradictions across Models A, B and C and 2 for the company’s design. Each source has the same denominator; the former total combines 3 services.
Read the contradiction breakdown
| Source | Set B | Country | Total |
|---|---|---|---|
| Model A | 4 | 14 | 18 |
| Model B | 5 | 15 | 20 |
| Model C | 4 | 18 | 22 |
| Our design | 2 | 0 | 2 |
Contradictions in the company’s design include recommending automatic network selection where the key required manual selection, and assuming Japan in a short question without a country name. Knowing the destination did not eliminate every mix-up.
"If one digit of a contact number is wrong, that becomes a wrong emergency call. At night, offline, or where only the local language works, a traveler cannot check on the spot. So contacts are not left to AI generation: we show official information we verified in advance, as is. The AI is only allowed to sort out the steps and draft what to say."
Scores change with question wording
Read scores by question wording
| Condition | Model A | Model B | Model C | Our design |
|---|---|---|---|---|
| Well-formed | 90.6% | 89.2% | 88.9% | 78.1% |
| Loosely formed | 71.1% | 72.2% | 67.2% | 70.0% |
| Short questions | 52.8% | 56.4% | 54.2% | 68.9% |
On short questions the company’s design scored 68.9%, versus 52.8–56.4% for Models A, B and C. The company’s design knows the destination supplied at purchase; the other services do not receive that context. The result includes this difference in conditions.
Pre-registered criteria and verdicts
The criteria were registered on 2026-09-22, before scoring. Criterion ① is “Set B: no more than 3.0 points below the same engine used as is in every phrasing”. On well-formed questions, the company’s design was 12.5 points lower and did not meet the criterion. 2 of the 3 criteria were met; the overall verdict is “Fail”.
| Criterion | Measurement | Verdict |
|---|---|---|
| 1. Set B: no more than 3.0 points below the same engine used as is in every phrasing | Well-formed: 78.1% vs 90.6% (−12.5 points) | Fail |
| 2. Fewer contradictions than the plain engine (both zero also passes) | Our design 2 / Model A 18 (Set B + country) | Pass |
| 3. Zero country-question contradictions | 0 / 74 questions | Pass |
Hold-out tests — where it answers and where it holds back
- Rephrased questions for countries with data: 12 questions, 97.2% coverage, 0 contradictions.
- Countries without data: 18 questions, 18.1% coverage, 0 contradictions. In 18 answers it explicitly said it could not confirm; 18 included a next office and what to ask.
The design avoided inventing answers where it could not confirm and pointed to the next office to ask; coverage was low, and the company has since widened it with MOFA’s emergency contacts (Part 1, §2). “No contradictions” means none against the registered key points; it does not mean every statement in the answers was verified.
Record of improvements — measured, verified and fixed
- v7: Restricted suppression to identifying information, rules and numerical values, then ran the same-day comparison.
- v8: A change intended to allow explanations of general systems had side effects. 2 questions (B17, B23) regressed, and this recurred in 2 runs.
- v9: Reversed the parts causing the side effects. Well-formed coverage reached 78.1%, but the gap to the same engine remained and criterion ① still failed.
- Experiments are underway on adding reference data and applying the founder’s research method, “Meandle.” The plan was pre-registered on 2026-09-24. Measurement under those conditions is underway, and results will be appended whether or not they meet the criteria.
- 2026-09-26: The company switched the conversational AI and added seven mechanisms (Part 1, chapter 2). For the draft report to the company, letting the AI generate prose kept adding unstated facts (5, 2, then 1); extracting the traveler’s words and matching them by program brought it to 0. Wording instructions did not stop it; the mechanism did.
- 2026-09-27: Two attempts to improve the first-action advice (answer-order rules; switching the answering model) did not meet the pre-registered conditions and were not adopted. The record also notes that an earthquake example in the answer-order rules pushed “stay on an upper floor” to the front in a tsunami scenario. The tsunami guidance fix and the supplements for Switzerland and New Zealand passed the safety re-test and went into production.
The company’s position and the production version
Production uses v9. Contacts, numbers and formal document names come from verified reference data, not AI generation, and the numbers and addresses in each answer are checked by software. General systems may be explained using general knowledge; individual conditions are directed to the office that can confirm them.
As of 2026-09-27, production answers with Google’s Gemini 3.8 Flash (an OpenAI model as fallback), and an OpenAI model checks each answer. On 2026-09-26 the company added itinerary summaries, MOFA emergency contacts, software checks of the numbers in answers, call buttons, the reviewing AI and the draft report to the company; on 2026-09-27, tsunami guidance and supplements for Switzerland and New Zealand. Each was added only after re-grading the same questions before and after the change and confirming that contradictions stayed at zero.
Switch decision date: 2026-09-24
Before the switch, two separate graders each re-scored v9 blind on the 74 country questions and the no-data-country follow-up; both found zero contradictions (24 September 2026). A later re-run (25 September 2026) found one answer about Switzerland, which then had no reference data, that directed users to a "Consulate-General of Japan" that does not exist. The same day we added reference data for the Embassy in Bern and the Consular Office in Geneva, checked against official sources; in the blind re-test after the fix (two graders), contradictions were zero.
History — the 2026-09-14 run and scorer differences
These are historical measurements, kept separate from the main results. The company’s country-question coverage rose from 40.9% to 97.2% over 5 runs as it fixed data scope, delivery to the AI and output completion. The other models were not given the same opportunities to revise.
Read the company’s revision history and historical comparison scores
| Run | Coverage | Contradictions |
|---|---|---|
| 1: Before reference data and implementation were prepared | 40.9% | 0 |
| 2: Expanded reference data from 10 to 19 countries | 61.8% | 0 |
| 3: Fixed added data not reaching the AI | 75.2% | 0 |
| 4: Added the same items to the 10 original countries | 92.5% | 0 |
| 5: Added post-loss procedures and emergency-number details | 97.2% | 0 |
| Source | 2026-09-14 country | Contradictions |
|---|---|---|
| Model A | 80.1% | 7 |
| Model B | 88.1% | 6 |
| Model C | 76.9% | 15 |
| 2026-09-14 Set B | Model A | Model B | Model C | Our design |
|---|---|---|---|---|
| Well-formed | 90.6% | 86.1% | 87.0% | 68.9% |
| Loosely formed | 75.0% | 75.6% | 71.7% | 61.4% |
| Short questions | 51.1% | 68.6% | 55.6% | 54.7% |
The earlier compilation of the company’s 2026-09-22 results is also retained as history. It is not used for direct comparison with the services’ older results.
| Condition | Coverage | Contradictions |
|---|---|---|
| Well-formed | 80.0% | 0 |
| Loosely formed | 74.2% | 1 |
| Short questions | 66.4% | 1 |
"When I saw the first 40.9%, what I doubted was the AI's ability. Only after re-measuring did we learn the cause was our own implementation. We had added the data, and it was not reaching the AI. You cannot notice this kind of accident without measuring. That is why we publish the history of fixes with the results."
On the Italian theft-report question, the services all contradicted the registered key. Re-checking official information showed that the key was wrong, and the verdicts were corrected. When several answers disagree with the key, the procedure now includes questioning the key itself.
Blind rescoring changed the scores of the same answers
On 2026-09-24, 25 questions were randomly sampled from the 74 fixed country questions. The 75 answers from 3 models were rescored without seeing the original scores. Mean coverage fell by 13.0 points for Model A, 16.0 for B and 17.0 for C. In this sample, scorer and scoring-practice differences appear to be the main cause of the decline.
Read the blind-rescoring sample means
| Model | Original | Same answers rescored | Drop (points) |
|---|---|---|---|
| Model A | 74.0% | 61.0% | 13.0 |
| Model B | 87.7% | 71.7% | 16.0 |
| Model C | 76.3% | 59.3% | 17.0 |
These are descriptive sample statistics, not replacement scores for the full question set. Interpretations of the core of compound elements, name variants and period boundaries still differed across scoring sessions. Answers also changed, so differences between run dates cannot be treated as pure changes in model performance.
The earlier Taiwan reporting-office correction was applied consistently across services. The main results exclude contradictions based solely on recommending a police station. The company continues to re-check registered keys against official sources and correct them when needed.
Recommendations for designers — eight points for building emergency AI
The company shares what it measured and verified as points for anyone building emergency AI. Services of the same structure can expect the same issues.
- Show contacts from verified information. Information where one character changes the action needs a path that shows pre-verified information as is.
- Check reference data for completeness per item. Count items present, not countries covered.
- Verify by machine that the reference data reaches the AI. "Having" and "delivering" are different. Combine a pre-release automatic check with measurement.
- Measure on untidy questions. Numbers from well-formed questions are not the numbers of a user in an emergency.
- Detect truncated output and stray characters. Truncated answers recorded as normal, and characters from an unrelated script, occurred both in the company’s design and in general chat AIs.
- After "I don't know", add the next step. Not asserting reduces contradictions, but by itself it does not let the traveler act.
- Doubt the key too. When several AIs agree against a registered key, re-check it against primary sources.
- Do not let the AI generate output that must not add facts. Have it extract the person’s own words and assemble them after a program matches them against the original. Wording instructions alone left the additions in a new form.
Implications for companies
- Companies with employees traveling abroad: whether the first response to a late-night "no connection", "stolen" or "sudden illness" — in any language — can be left to an AI depends on whether your contacts and internal rules are in the reference data. The Corporate Travel SIM (from 2026-10-19; advance inquiries welcome) offers this design to companies. Contact info@jagproject.com.
- Travel agencies, eSIM providers, insurers and card issuers: Travel AI Rescue can be offered to your own customers in two ways: through a white-label corporate travel SIM sold under your name (from 2026-10-19), or inside your own app or service, with your SIM or eSIM (including other providers’) and your help desks in the reference data, quoted per requirement. Contact info@jagproject.com.
- The same test, on your terms: the questions and scoring rules in this paper can be run for the countries and situations you expect. Contact info@jagproject.com.
"In corporate travel, the late-night 'no connection', 'stolen' and 'sudden illness' calls land on general affairs. Widen what the traveler can handle alone, and the burden on the travel desk falls. With your contacts and internal rules in the reference data, this design also runs on other providers' SIMs and eSIMs. We offer it as the corporate version from October 19, and through partners who offer it to their own customers."
Method, assumptions and limitations
Questions and source URLs for the keys were pre-registered on 2026-09-08. Set B has 30 questions in 3 wordings; the country set has 74 scorable questions drawn from 6 items across 20 countries. The main run was 2026-09-22; the company’s c1 replacement was 2026-09-23.
The questions were written in Japanese from the viewpoint of Japanese travelers abroad. This English paper translates the same study; it does not measure English-language answer performance. Set A, for visitors to Japan in English, Chinese and Korean, is planned for a later scoring update.
The subjects are Models A, B and C, identified as default free-plan models on 2026-09-14, and the company’s design. Higher-tier models and search-enabled conditions are excluded; names are anonymized. The company’s design receives reference data and the destination supplied at purchase, so input conditions differ.
Scoring assumption — the user is abroad
An answer that unconditionally gives Japan’s 110 / 119 to a question without a country name counts as a contradiction. The assumed user is abroad. Those numbers are correct for a user in Japan; the numbers themselves are not wrong. Answers that distinguish Japan from other countries or ask which country the user is in do not count as contradictions for that reason alone. This rule applies equally to the company’s design and all services (assumption clarified 2026-09-24).
- Key-point coverage
- Registered key points are split into elements, each scored present or absent with no partial credit. Per-question proportions are averaged.
- Contradiction
- The number of answers contradicting a registered key point. Missing elements or extra statements outside the key do not count by themselves.
- Next step
- Stating what cannot be confirmed, which office to ask, and what to ask.
AI was used for scoring, and Jag Project conducted the study itself. Disclosing scorer differences does not constitute an independent external evaluation. Hold-outs are limited, and the study has not measured whether users could act in real emergencies. Further work will cover higher-tier models, search, external scoring and new questions.
Part 1: method and limitations
- Tabletop scenarios were written by writers who did not know the arrangements (memoryless Claude instances), checked for bias and facts by others, and fixed and recorded once no fixes remained. The company’s research lead did not read the scenarios before unblinding.
- Judging rules and the main comparisons were pre-registered and recorded with SHA-256 before answers were generated. Comparisons used one-sided sign tests on paired scenarios with Holm correction, and were re-checked under a reading unfavorable to the company.
- The judges were AI (Claude), not human experts. The three rounds total 64 scenarios, only 8 of them disasters. The crisis provider and guidebook were judged from their descriptions. These are simulations, not real travelers’ behavior or incident counts.
- Free and paid AIs were reproduced by calling the same models via API, without app instructions, search or memory.
- “Service X and Y” in Part 1 and “Models A, B and C” in Part 2 are separate labels and may include the same service.
Data access, and what is not published
The full questions, scoring rules, key points with their source URLs, and the verbatim answers (Models A, B, C and the company’s design, all conditions and hold-outs) are provided to journalists on request: info@jagproject.com. The same applies to Part 1’s scenarios, judging rules, verbatim answers and judging sheets. For re-testing Travel AI Rescue, the company issues a free review account the same day (log in; please use a fictional person, order and itinerary). The company’s implementation (the structure of the reference data, the mechanism that passes it to the AI, the run scripts) is not published.
Use of figures and citation
The figures and numbers in this paper may be reproduced for news and research with attribution (suggested form: Jag Project, LLC, "How far should an AI answer in a travel emergency?", September 2026). Each figure carries its title and source, so a screenshot can be used as is. PNG versions are available to journalists. A PDF edition will be provided at the official release. For a founder photo, product screens or an interview: info@jagproject.com (replies within 24 hours).
Disclaimer
These results were confirmed under conditions specified in advance. They do not show that every environment gives the same result. AI output is a draft. In a real emergency, put local police, ambulance, medical services and your embassy first. Rules change; the information here was confirmed on 2026-09-27 (Part 1) and 2026-09-22 (Part 2).
Revision history
- 2026-10-07: Officially published.
- 2026-09-29: Rewrote the opening (premise, terms, what Travel AI Rescue is) and the executive summary; wrote the body in the third person and aligned terms (reviewing AI, situational judgment); corrected a summary sentence that had mixed results from two scam tests.
- 2026-09-27: Added Part 1, corporate travel preparedness (current arrangements A, what we built B, Travel AI Rescue C): contact sheets made with general AIs, three tabletop rounds, and scam-versus-genuine tests including paid AIs. Updated the production description (2026-09-26 to 27) and added the eighth recommendation.
- 2026-09-25:Reorganized for corporate travel teams around the same-day run; added pre-registered verdicts, regressions, scorer differences and production-version disclosure. Production switched to the 23 September version (v9), decided 2026-09-24.
- 2026-09-22:Advance copy for press.
Jag Project, LLC · Jag Yamamoto · info@jagproject.com
