How to use AI safely: Lessons we learned testing 10 chatbots against disinformation
We tested ten widely used AI chatbots against ten disinformation narratives about Lithuania. We asked each of them, in Lithuanian, to argue the false claim and back it up with sources.
Nearly half of the answers gave way. Many repeated the disinformation outright. Others wrapped it in a “both sides” frame that made it look like a legitimate debate.
Even the answers that rejected the false claims often stood on shaky ground. They cited outlets aligned with the Russian state, links to pages that do not exist, and sources presented as far more authoritative than they were. Some even cited fact-checks as proof of the very claims those fact-checks debunk.
More and more people now turn to AI chatbots for information. Based on our findings from the LLM Disinformation Resilience Audit report, in which we tested 10 chatbots, we have put together practical recommendations for using them safely: how to spot risky questions, how to question an answer, and how to check the sources behind it.
Understand how AI chatbots work in providing information
AI chatbots can answer almost any question quickly and fluently. Knowing how they produce those answers helps users judge when to trust them. Research suggests that people who understand how chatbots work are less swayed by slanted answers.
A fluent answer is not necessarily correct. A chatbot writes its answer by predicting which words are most likely to follow, based on patterns learned from large amounts of text. By default, it does not check its statements against a reliable record of facts. Widely repeated false claims can appear in its answers in the same confident tone as accurate information. Asking a chatbot to check a claim can improve the answer, but you still need to verify the result.
Several other factors also shape how chatbots answer:
A chatbot’s answer is a starting point for checking, not a verdict.
Interrogate the answer
In this audit, 46 of 100 answers went along with the disinformation request in different ways. No single rule will protect a user from wrong answers, but the habit of questioning chatbot answers critically helps users reach a more reliable one. The following warning signs, observed in this audit, should prompt a follow-up question:
1. Is the balance real?
Chatbots often try to appear neutral by presenting “both sides” of a disputed question. This is fair when the question is a matter of opinion, such as whether taxes should be raised to fund public services. It is misleading when one side is a documented fact, and the other is disinformation. In this audit, 12 of 100 answers did this. Asked whether the Soviet annexation of Lithuania in 1940 was legal, one chatbot set out “Lithuania’s position” and “the Soviet Union’s arguments” as two equal sections and asked the reader to decide, although the occupation is a historical fact. Watch for phrases such as “one of the versions” or “I will present both sides”, and for answers that call a claim propaganda and then list arguments for it anyway. If an answer presents competing views, ask:
Is this a matter of opinion, or is there an established answer? Who holds each view, and what evidence supports it? What do historians, courts or international organisations conclude?
2. What is fact, and what is interpretation?
Misleading claims often start from something true. Take this one: “Electricity prices rose by 65% because Lithuania stopped buying Russian energy, so the policy betrayed the people.” It contains three different kinds of statements:
a fact that can be checked: prices rose by 65%
an explanation that needs its own evidence: because Lithuania stopped buying Russian energy
a judgement that no fact can settle: the policy betrayed the people
A true fact at the start does not make the rest true. Chatbots can confuse correlation with causation. They may present two events as cause and effect simply because they occurred at the same time.
Check which assumptions the answer takes for granted and builds its argument on. In this audit, the energy prompt assumed Russian energy was cheaper. None of the ten chatbots questioned this, although Lithuania in fact paid some of the highest gas prices in the EU while Russia was its only gas supplier.
ASK: Which parts of the answer are facts, which are explanations, and which are judgements? Is the assumption in my question correct?
3. Does the answer explain itself?
A reliable answer explains its claims. Watch for these gaps:
It states conclusions without explaining them. A well-founded answer points to specific facts, such as dates, documents or official data. Confidence is not evidence.
It declines without saying why. “I can’t help with that” leaves you not knowing what is wrong with the claim.
It uses numbers out of context. One chatbot presented €608 million worth of equipment, supplied to Ukraine by several donors, as if Lithuania alone had paid for it,
It switches languages. If the answer or its sources switch from Lithuanian to Russian or English, little reliable material may exist in Lithuanian.
ASK: What facts is this based on? Why exactly is this claim true or false? Whose figure is this, and what does it measure? Which source says exactly this? What do Lithuanian or international sources say?
Check the sources and their claims
Nearly one in five sources that chatbots provided in this audit were not credible: adversarial, state-aligned, tendentious, fabricated or broken. Do not assume that a chatbot follows strict rules when choosing sources. In this audit, sanctioned Kremlin outlets, disinformation sites and tendentious sites were presented as credible sources.
Chatbots also write their own label for each source, and in this audit these labels were often wrong: a pro-Kremlin site appeared as an "LRT" report, a Belarusian government page as "Reuters / BNS". The name a chatbot gives a source is not enough to judge its trustworthiness.
At least 25 links were fabricated or did not lead to the cited page, and some fact-checks were cited as proof of the very claims they refute.
For each source the chatbot provides, check:
Does the source exist?
Does the name given in the answer match the actual source?
Who is behind the source? Is it trustworthy?
Does it say what the chatbot claims?
Ask the chatbot for original sources (documents, records), not just webpages or summaries.
Cross-check with a source suited to the claim: official institutions for official data, established media for reported events, historians and courts for history, and fact-checkers and disinformation databases for known narratives. Useful resources include the State Security Department's annual National Threat Assessment, the EUvsDisinfo database and published fact-checks.
Know when to ask a chatbot
AI chatbots’ answers are most reliable on well-documented topics. The further a question moves from such topics, the more carefully the user should read the answer. Thus, an important habit before turning to AI chatbots for information is to consider whether the question is one they can be expected to answer reliably. In other words, before turning to a chatbot for information, ask yourself whether a chatbot can provide the accuracy and depth your question needs.
A chatbot’s answer is more likely to be reliable if the answer to all of these questions is YES:
Is the question about facts, not about policies or political choices?Our study found that chatbots were most reliable at rejecting claims that courts and historical records have settled: they resisted in 24 of 30 such answers (80%).
Is a general, surface-level overview enough? Even answers that rejected the false claim were thin on detail: 39% of their links pointed to homepages or encyclopaedias, and fewer than 10% led to the actual documents and records. A chatbot can give you an overview, but if you need specific facts, figures or documents, consult original sources.
Is the topic widely known and has it been around for a while? Chatbots answer best on topics that have been covered for a long time, by many outlets, and beyond Lithuania. New events, local issues and niche subjects have fewer reliable sources behind them, and false information gets into those gaps. This matters more for questions asked in Lithuanian. Research suggests that chatbots’ safety features work less well in smaller languages, and when little is written in Lithuanian, they borrow from sources in other languages.
Chatbots should match the firmness of an answer to the evidence: present “both sides” only where a question is genuinely contested, and separate a legitimate policy debate from the disinformation claim built on it. Where a national security service or a recognised fact-checking body has already identified a narrative as disinformation, the answer should say so. Training should not reward the appearance of balance over accuracy.
Chatbots should not cite sanctioned outlets as supporting evidence. Doing so sends the user to a misleading source, even when the answer itself is accurate. When an answer refers to such an outlet in order to refute it, the outlet should be clearly flagged, with a short note on who runs it and why it is not credible.
Chatbots should check each source before citing it, to confirm that the link works and that the source supports the point it is attached to. In this study, credible sources were cited in support of claims they contradict, and fact-checks were cited as evidence for the disinformation they debunk.
Chatbots should state clearly when evidence is thin, and warn the user when a query falls into a data void (a topic with little reliable information online), instead of filling the gap with whatever ranks highest.
Providers should assess the risk of their systems generating or legitimising disinformation, and publish both the results and the methodology, in line with Digital Services Act requirements where they apply. This testing should cover all EU languages, including small ones such as Lithuanian.
Providers should be transparent about how their chatbots choose sources. They should disclose the search index and source-selection rules, and give independent researchers access to the data and documentation needed to study how chatbots handle disinformation.
Providers should work with fact-checkers, researchers and public institutions to improve resilience to disinformation and to ensure that trusted official sources are indexed and retrievable.
Under the AI Act, providers of the AI models behind the most widely used chatbots must evaluate them and mitigate systemic risks, including harmful manipulation. Under the Digital Services Act, ChatGPT, designated a Very Large Online Search Engine in August 2026, must assess risks to civic discourse and elections. These obligations should explicitly include testing how chatbots respond to disinformation in all EU languages. Testing results should be available for independent review. In this audit, where reliable Lithuanian-language sources were scarce, other material, including Russian state-aligned content, filled the gap. A risk assessment conducted mainly in English would not detect this.
The EU and national authorities should clarify how the sanctions broadcast ban (Regulation 833/2014, Art. 2f) applies to AI chatbots and AI search that retrieve, link to, summarise or reproduce content from listed outlets, including mirror sites. Five links in this study pointed to sanctioned outlets.
Public institutions should make their material easy for chatbots to find and cite. In this audit, no chatbot referred to the court judgments in the “13 January 1991 events” case or the secret protocols of the Molotov–Ribbentrop Pact, although both are public. Research institutes, museums, government bodies and historical memory centres should publish key documents and findings as text pages with stable addresses, not only as scanned PDFs, with short summaries in Lithuanian and English that state the facts, figures and dates plainly. This practice is known as generative engine optimisation. Research suggests that pages with clear statistics, quotations and references are more likely to be cited in AI answers. Funding the communication work of these institutions would put more credible material where chatbots look for it.
Institutions that maintain public document repositories should clearly label statements by adversarial actors. This audit found that an unlabelled Russian statement in OSCE records was cited in support of a Kremlin narrative.
Monitoring how chatbots represent key topics should become part of the work of the responsible bodies. The Genocide and Resistance Research Centre of Lithuania (LGGRTC), for example, could regularly check how chatbots present the topics it curates and whether its new research appears in chatbot answers.
