We asked an offline AI 33 survival questions. Here is what it got wrong.

Can a local AI model help you in an emergency when the internet is down? Partly. In our test the search worked almost perfectly and the answers did not. The better model got 26 of 33 questions fully right, and both models made mistakes nobody should act on, like telling someone to use 2 drops of bleach where the official table says 8.

Below is what we tested, what went wrong, and what we think an offline survival assistant has to do before anyone trusts it. We're building one ourselves (Waystone, still in development), so we had every reason to look hard.

What we tested

On 22 October 2026 we ran everything on an Apple M2 Mac with 8 GB of memory, through Ollama, with the network switched off. Two small language models took turns: Llama 3.2 3B at 2.0 GB on disk, and Gemma 3 4B at 3.3 GB.

Both read from the same small library of 24 short passages. We wrote them from official advice: Ready.gov, the CDC, the EPA, FoodSafety.gov, and the Dutch, Belgian and French governments. A separate search model, BGE-M3, picked the 3 passages closest to each question and handed them to the language model to answer from.

The 33 questions were split evenly between English, Dutch and French. They covered drinking water, bleach, generators, food in a power cut, gas leaks, floods, heat stroke, bleeding, earthquakes and warning sirens. Six were deliberately outside the library: seawater, wild mushrooms, a snake bite, a nuclear shelter, a hunting rifle and a flat tyre. We wanted to see whether the models would admit they didn't know.

The part that worked: finding the right page

The search side was excellent. For all 27 questions the library could answer, the right passage was among the 3 the search model returned, and it came first in 26 of them. Dutch and French questions found the English passages as reliably as English questions did.

That matters more than it sounds. One carefully checked library in one language can serve readers in several languages. Nobody has to translate and re-check three libraries.

Speed was fine with the smaller model. Llama started writing after about 2.4 seconds and finished a typical answer in about 4. Gemma needed 33 seconds before its first word, but the Mac was already swapping memory to disk at the time, so read that as a warning about 8 GB machines and not as a verdict on Gemma.

The part that didn't: numbers, units and invented warnings

The bleach answer worried us most. Asked in Dutch how many drops of bleach go into 4 litres of water, Llama said 2. The EPA's table gives 2 drops for 1 quart or liter and 8 drops for 1 gallon, which is the row that fits 4 litres. The short passage we gave it said 8 drops per gallon (about 4 litres), so the right answer was in front of it. We don't know why it said 2, and it sounded completely sure of itself. The official table is in our guide to purifying water with bleach.

Gemma turned "20 feet (about 6 metres)" into "20 metres (6 metres)" in both its Dutch and French answers. The two numbers in its own sentence no longer matched, which at least gives a careful reader a clue.

It also said a closed fridge keeps food cold for 48 hours. The USDA says a refrigerator keeps food safe for up to 4 hours, and 48 hours is what a full freezer holds. Right number, wrong appliance.

Llama made up a warning that wasn't in any passage: in an answer about smelling gas, it told the user not to call the fire brigade. That's the kind of sentence that can hurt someone. It also told a Dutch user that the public warning sirens are tested every Monday. They're tested on the first Monday of the month.

Counted up, Llama got 26 of 33 answers fully right and made 3 serious safety mistakes. Gemma got 22 right and made 4.

Saying "I don't know"

Of the 6 questions with no answer in the library, Llama refused 5 and Gemma refused 3. Both failed on seawater, and Gemma suggested treating seawater with bleach. Bleach kills germs. It does nothing about the salt.

Citations had problems too. A good offline assistant shows which page each answer came from, so you can check it. Llama pointed at the right passage in 24 of 27 answers and Gemma in 22. A wrong citation is worse than none, because it sends the most careful reader to the wrong page.

Dutch and French were weaker

English came close to good: both models got 10 of 11 English questions fully right. In Dutch, both managed 8 of 11. In French, Llama got 8 and Gemma only 4.

If you plan to rely on a small local model in a language other than English, test it in that language first, with questions you already know the answers to.

In a second round on 25 October 2026 we added German, Spanish, Polish and Swedish, and both models again did worse than in English. Our survival AI benchmark has every result from both rounds and the data to rerun them.

What would make offline AI safe enough to lean on

We tried three simple checks on the same answers.

The first refuses to answer when nothing in the library is close enough to the question. On its own it refused 5 of the 6 out-of-library questions and never blocked a real one, though the margin was thin. The second throws out any answer that doesn't cite a source. With both checks switched on, Llama refused all 6.

The third compares every number and unit in the answer with the cited passage and shows the original passage instead when they don't match. That would have caught the 2 drops and the 20 metres. It would not have caught the fridge mistake, because 48 hours really is in the source, just about the freezer.

Our conclusion for anyone building or buying one of these tools is simple. The AI's own words should never be the last word on a number. Show the source text under every answer, especially for doses, distances and times. We're designing Waystone around that rule, and we'll publish a bigger test before we claim anything works.

Trying this yourself

You don't have to wait for anyone's product. A Mac with Apple silicon and 16 GB of memory is the realistic starting point. On 8 GB, the 3B model worked for us only with few other apps open, and a 4B model was already too slow.

Our guide to running AI offline on a Mac walks through the setup. After that, build a small library from public-domain material such as the US Army survival manual FM 21-76, ask it questions you can check, and keep the paper version of anything that involves a dose.