Knowledge cutoff
Temporal limit of a model's knowledge
From Wikipedia, the free encyclopedia
In machine learning, a knowledge cutoff (or data cutoff) is the point in time beyond which a large language model has not been trained on new data.[1] Since large language models are pretrained, any model's knowledge is fixed at what it was trained on before deployment; information about events after this date is absent from the model's training data.[1][2] The model cannot access information about later events without a system for real-time data access such as retrieval-augmented generation, which fetches new information from an external database.[2][3][4]
Knowledge cutoffs can introduce limitations like hallucinations, where the model generates confident (but false) statements, information gaps, and reduced accuracy on evolving knowledge.[1][3][5] Research has shown that knowledge cutoffs have safety-critical implications in domains such as healthcare, where outdated knowledge can lead to harmful recommendations. A later knowledge cutoff may have greater accuracy in time-sensitive tasks.[5]
Description
A large language model is trained before deployment on static snapshots of data collected from the internet, books, and other sources up to a specific knowledge-cutoff date. During training, an LLM can learn linguistic patterns, semantics, and contextual meanings. The model can then learn probabilities and predict which word is likely to come next. When training is finished, the model's knowledge is fixed.[1][2] A model with a fixed knowledge cutoff is unable to provide information about facts (or developments) which have emerged since that time because it is not connected to the internet. It may produce incorrect answers, especially using data from recent events.[1] Training on newer data would increase its price; Time estimated in 2024 that training the most powerful large language models might soon cost over US$1 billion.[6]
The GPT-4 model has a knowledge cutoff of September 2021;[7] the GPT-4 Turbo model has a knowledge cutoff of December 2023,[7]and the GPT-5 model has a knowledge cutoff of September 2024.[8] GPT-OSS models have a knowledge cutoff of May 2024,[9] and Llama 4 models have a knowledge cutoff of August 2024.[10]
Effects
Information gaps
Knowledge cutoffs create information gaps in which a model lacks knowledge of events (or discoveries) not included in its training data, leading to hallucinations.[1] They occur because large language models are designed to predict and generate the most probable sequence of words based on their training patterns, which may result in confident – but incorrect – outputs when queried beyond the information present in its training data.[1][2] A study by Cacioli et al. at Oregon State University demonstrated the real-world impact of a knowledge cutoff. Researchers created a 363-question benchmark based on two versions of the Infectious Diseases Society of America's COVID-19 treatment guidelines. Models whose knowledge cutoffs predated the newer guideline, such as GPT-3.5-Turbo and Llama-2, performed worse on these questions (76.03 percent and 25.26 percent, respectively). Models with knowledge cutoffs after the guideline, such as GPT-4o and Llama 3.3, had over 90-percent accuracy. These findings indicate that clinical reliability improves as models incorporate more-recent knowledge cutoffs. The study concluded that the recency of information must be treated as a safety-critical attribute on a par with alignment or interpretability, highlighting the safety concerns of knowledge cutoffs in applications such as clinical decision-making.[5]
Effective vs. reported dates
A study by Pęzik et al. at the University of Łódź indicated that a model's knowledge does not necessarily match its cutoff date. The effective cutoff, the date up to which it can reliably know information, often differs by subject and is influenced by the distribution of information within the training data; some topics may reflect later knowledge than others, and knowledge that predates the cutoff may be absent. Training data has uneven information across topics.[11] Due to the high cost of retraining large language models, the models are rarely retrained to extend their knowledge cutoff.[12] Some models can use integrated search tools to access more recent information, which makes it unclear if an answer comes from the model's original training or from a live search; GPT-4 can access its search tool and provide real-time information.[7]
Mitigation strategies
Retrieval-augmented generation
Retrieval-augmented generation augments a large language model with updated data from external sources, allowing it to generate better-informed responses. In RAG, the language model is connected to an external knowledge base or search engine to retrieve live data. This allows the model to find current information relevant to a query and incorporate it into its response, with citations.[2][3] Grounding a model in external data which ties a model's answers to its retrieved sources helps reduce the frequency of hallucinations and improves output accuracy. The external knowledge base might be outdated or contain biases, however, which may lead to incorrect information (or hallucinations). Google's AI Overviews has made false claims and provided unreliable results, since the model may misinterpret the prompt or fail to retrieve high-quality sources. Even when models can access the internet with browsing tools, their core reasoning and baseline assumptions remain anchored to their original training data. Retrieval alone cannot fully compensate for an outdated knowledge cutoff; the model's fundamental understanding is rooted in its training data.[1] This can be mitigated with techniques such as reinforcement learning from human feedback, which aligns an AI model with human preferences; this technique can enhance the quality and reliability of a large language model's responses.[4]
Incremental learning
Another approach is incremental learning (or continual learning), a method of machine learning in which new data is continuously used to extend the existing model's knowledge. It aims to prevent catastrophic forgetting, where AI abruptly forgets what it has learned. In practice, however, it often fails to prevent catastrophic forgetting completely. This technique allows efficient, incremental updates to a model without the cost of a full retraining cycle.[12][13] One continual-learning technique is fine-tuning, which allows AI labs to precisely adjust a model's behavior. A more efficient method of fine-tuning involves methods such as Low-Rank Adaptation.[12][13] This does not provide real-time awareness, however, since adding modules to the system may result in catastrophic forgetting as the weights in the model become biased in favor of the new data set.[13]