What is your certainty based on? You say that the cutoff was Jan 2022, and that what the model says about the war in Ukraine is "extrapolation". However, the summary it generated includes details such as a reference to "denazify", or damage to major cities. It would be an impossibly good forecaster if it managed to generate that text without actually having access to material post Jan 2022. I find it much more likely that the cutoff date communicated to the model is conservative, and that its training included text that was written after Jan 2022.
>However, the summary it generated includes details such as a reference to "denazify"
Do you think the denazification propaganda suddenly started in February of 2022? The wider conflict has been ongoing since 2014 and that rhetoric was not new to the 2022 escalation.
>or damage to major cities.
Kyiv and Kharkiv are the two largest cities in Ukraine. Predicting that they'd be a focus of fighting isn't really revelatory. Mariupol is smaller but was near the pre-2022 front lines and had already been attacked numerous times since 2014.[1]
>It would be an impossibly good forecaster if it managed to generate that text without actually having access to material post Jan 2022.
If the Russian invasion had happened in a vacuum, sure. In reality, ChatGPT's response is pretty clearly using information from the 2014 invasion and adding the user's prompt of February 2022. There's nothing in its answer that is unique to the 2022 invasion.
Well a single hit yes, that could be explained as coincidence. But the generated text is too long and too close to reality. In Jan 2022 the meaning of "special operation" was not a war. So if the cutoff were complete it would've generated something else.
>Well a single hit yes, that could be explained as coincidence. But the generated text is too long and too close to reality.
Again, the war between Russia and Ukraine has been ongoing since 2014. Everything it's saying is general and applies entirely to the war since 2014. None of the information listed -- outside of what it was given by the user prompt -- has any specificity to the 2022 invasion.
>In Jan 2022 the meaning of "special operation" was not a war.
It's regurgitating the prompt, just as it's doing with the mention of Feb 2022.
A better test prompt would be "What is the status of relations between Russia and Ukraine?"
I don't have access to GPT-4 but GPT-3's response is entirely in-line with it's Sept 2021 cutoff:
On the one hand, if I query GPT-4, it does associate "special military operation" with Russia's actions in Ukraine. However, it does not reference any escalation beyond Donbas and Crimea. The generated text stays within well known knowledge of 2021:
Summarize the special military operation in Ukraine
As of my latest update, the conflict commonly referred to as the "special military operation" refers to Russia's ongoing military intervention in Ukraine, which began in 2014. The intervention started with Russia's annexation of Crimea in 2014 following the Ukrainian revolution. The unrest escalated into a war in the Eastern Ukrainian region of Donbas between the Ukrainian government and pro-Russian separatists.
The "special military operation" typically refers to specific operations within this broader conflict, notably operations carried out by the Russian military or by pro-Russian separatists in Ukraine. The complexity of the conflict, which involves direct incursions, "covert" operations, proxy warfare, and other forms of military intervention, makes it difficult to summarize overall, but key events have include the annexation of Crimea, the Battle of Ilovaisk, the war in Donbas, and ongoing issues related to ceasefires, territorial control, and the political status of Crimea and Donbas.
As of now, the conflict is still ongoing with no resolution in sight, causing numerous causalities and massive displacement of people. Please note that you need the latest update on the matter as the situation is continuously changing.
Clearly, I got the old version. What we're disagreeing over is whether the new version got clairvoyant, or just a later cutoff that includes information from 2022.
