Jakob Hauter, University of Reading

The increasing power and popularity of artificial intelligence (AI) has created new challenges for social scientists who work with data from online sources. In this blog post, I explain how researchers can respond to this challenge by using the social science method of process tracing for the qualitative analysis of online data.

Benchmarks for good process tracing

Process tracing is a method used by social scientists to identify cause and effect through the in-depth study of an individual case. Rather than comparing initial conditions and outcomes across several cases, process tracing zooms in on the chain of events that links a cause to an outcome. Different researchers approach this task in different ways. Some use the term process tracing rather broadly for any detailed description of events. Others, including myself, argue that process tracing needs to do more than that. In one of my previous articles and in my book on Russia’s invasion of Ukraine’s Donbas, I developed what I consider four benchmarks for good process tracing.

  1. Good process tracing structures its argument in a specific way. First, it explains the proposed causal mechanism consisting of cause, outcome, and the steps linking the two. Then it explains what kind of traces the proposed mechanism would be expected to leave. Only after that, it dives into the empirical evidence and explains how this evidence meets the previously outlined expectations.
  2. Good process tracing considers alternative explanations. It identifies what other possible causes would be incompatible with the proposed causal mechanism and explains why the evidence for them is weaker than the evidence for the favoured cause.
  3. Good process tracing subjects its sources to critical scrutiny. Whenever it addresses a controversial question, it explicitly discusses sources and their reliability rather than treating sources as affirmatory afterthoughts to statements of fact.
  4. Good process tracing argues in probabilistic terms and acknowledges uncertainty. It admits that most conclusions are based on incomplete information and that new evidence could sway the balance of probability. It indicates how confident we can be in its findings, what the unknowns are, and what evidence to the contrary might yet emerge.

Digital forensic process tracing 

In my book and my article, I also argue that academic process tracing research can learn from journalists and activists who practice Open Source Intelligence (OSINT) analysis. OSINT refers to the wide range of information and media that the internet has made available to the general public. Prime examples of impactful OSINT investigations are the work of research collectives like Bellingcat and Forensic Architecture. Some people do not like the term OSINT and argue that it should only be used in relation to the work of government intelligence agencies. For this reason, I decided to speak of Digital Open Source Information (DOSI) in my book. But the meaning is the same, and OSINT is more commonly used.

I called this merger of process tracing and OSINT analysis digital forensic process tracing. This form of process tracing uses openly available information from the Internet as its data and pays particular attention to my third and fourth benchmark – source criticism and probabilistic reasoning – during the analysis of this data. I used this method to show that the outbreak of war in Ukraine’s Donbas in 2014 was a Russian invasion in disguise rather than a civil war.

Although much time has passed since 2014, I would argue that digital forensic process tracing has lost nothing of its relevance. If anything, it has become more relevant than ever. The dramatic changes in the online landscape that we have witnessed since then have created new challenges but also new opportunities and an even greater need for diligent assessments of source quality and uncertainty.

AI as challenge and opportunity

The most important recent development affecting OSINT is the dramatic improvement in the capabilities and the availability of artificial intelligence (AI). If used with caution, AI has great potential to help researchers with various tasks through features like automatic translation, audio transcription, natural language programming, or data analysis. However, AI tools that can generate or alter images and videos have created a new challenge for OSINT researchers. In my previous work, I could still argue that a video from 2014 showing a Russian tank in a specific location in Ukraine was unlikely to be fake because the fabrication of all its details would have required a disproportionate effort. This argument no longer works for footage published today. AI tools are perfectly capable of relatively complex fabrications. For example, within a few days of the first US and Israeli strikes against Iran in early March, the BBC documented several cases of AI-generated videos and satellite images of supposed impact sites appearing on social media. This means that researchers need to ask more follow-up questions. Is there anything implausible about the footage? Does it show any signs of manipulation? What do we know about the origins of the footage, and what is the track record of the source that first published it? Is there any other corroborating footage or any other evidence, such as eyewitness testimony?

Another challenge is that the potential targets of OSINT investigations have learned to be more cautious. In 2014, Russian soldiers were still rather careless in their social media activity. Today, both the Russian and the Ukrainian authorities go to great lengths to prevent the publication of unauthorized warzone footage. In Iran, the regime has even been relatively successful in imposing a complete Internet blockade during its violent crackdown on protests earlier this year. On the flip side, new platforms have created new opportunities. Telegram’s aversion to regulation and content moderation has encouraged users to post politically sensitive information or footage under the cover of perceived anonymity. In the run-up to Russia’s full-scale invasion of Ukraine, TikTok’s algorithm was feeding OSINT researchers who liked videos of Russian tank convoys with additional similar content. Moreover, the fast pace of the online information environment has also increased pressure on governments and propagandists to promote their own versions of the story. In turn, this has increased the chances of inadvertent revelations. Sloppy propaganda footage has always been a treasure trove for OSINT investigations. A prime example is the famous video of a pro-Russian activist driving past Russian tanks during the 2015 Battle of Debaltseve in Ukraine at a time when Russia denied any involvement.

To sum up, as disinformation and post-truth politics enter the age of AI, digital forensic process tracing has become more cumbersome but also more important than ever before. Social scientists should continue to look for ways to make use of it and develop it further. What is more, while digital forensic process tracing might not be the perfect fit for every research agenda, its key principles – online literacy, a critical assessment of sources, and the use of probabilistic reasoning – are important skills for any researcher in academia and beyond.

About the author:
Dr Jakob Hauter is a Postdoctoral Research Fellow at the University of Reading in the UK. He is currently working on a three-year project investigating the causes of Russia’s wars in Ukraine and elsewhere. His work is supported by the Leverhulme Trust with an Early Career Fellowship. His wider research interests include Russian and Ukrainian politics, the causes of war, process tracing as a social science method, and the dissemination of information and disinformation via online media.