Introduction

This paper addresses the situation of a humanist writing custom code to make sense of many primarily textual sources, typically for themselves or a small research group. Depending on the project, the pipeline usually includes optical character recognition, preprocessing, indexing for search, measures of document relevance such as TF-IDF, concordancing (also known as a keyword in context listing), and named entity recognition. These transformations make it easier to explore a corpus. They support what Peter ^{Pirolli and Stuart Card (2005)} called a ‘foraging loop’: searching, filtering, finding, reading, and extracting information.¹ Once prepared, researchers can use the machine-readable corpus for more specific computational tasks such as unsupervised clustering, supervised machine learning, visualization, and so on.

All the digital history projects I have worked on in the past few decades have taken this form. Automation in the pipeline has focused on bottom-up processes that create the evidence scholars will interpret. However, Pirolli and Card also identified a ‘sensemaking loop’ that involves conceptualization or schematization to best fit the evidence, and corresponding top-down processes that re-evaluate the theory under development and trigger new searches for support, evidence, relations, or information. In my experience, sensemaking processes have rarely received computational assistance in digital history. Instead, the bottom-up processes are adjusted and re-run from the beginning as necessary, and the sensemaking loop remains firmly in the domain of traditional, non-digital scholarship. In this discussion, I explore how computational tools can support sensemaking in historical research. These tools accomplish this by incorporating semantic interaction and model steering, map causal relations and use them for inferences, generate, manage, and evaluate hypotheses, and determine the diagnosticity of evidence.

The remainder of the paper is organized as follows. First, using my experience with the online version of the Old Bailey Proceedings, I describe how the organization of an extensive digital historical source collection can shape the process of foraging for information to support an interpretation. I then present the model of Pirolli and Card and show that it both matches many of the steps of a typical historical research project and suggests some possibilities that have yet to be fully exploited. I go into more detail about the bespoke coding that supports larger-scale digital historical investigations and indicates possibilities for incorporating human expertise in the top-down direction. This leads to a historical sketch and discussion of comparable techniques from the field of intelligence analysis, focusing on structured analytic techniques (SATs). SATs are designed to reduce known cognitive biases while making qualitative reasoning under conditions of ambiguity and uncertainty more explicit (and thus subject to computation). Many of them can be used for historical as well as contemporary research. I conclude by describing how my colleagues and I are adapting some sensemaking ideas from contemporary intelligence analysis software into our code and the roles that large language models (LLMs) have begun to play in our work.

Information Foraging in Large Digital Historical Source Collections

As the development of the World Wide Web created new opportunities for historical practice and allowed some scholars to think of themselves as ‘digital historians’, the online presentation of historical source collections became a primary focus. Daniel J. ^{Cohen and Roy Rosenzweig’s Digital History (2005)} was subtitled “a guide to gathering, preserving, and presenting the past on the web.”² This was especially true of larger projects that received substantial funding and employed researchers, data entry clerks, editors, XML taggers, programmers, and other specialists. The example that I draw on here is the Old Bailey Online.³ It is “a fully searchable, digitised collection of all surviving editions of the Old Bailey Proceedings from 1674 to 1913, and of the Ordinary of Newgate’s Accounts between 1676 and 1772.” The site’s core details 197,745 criminal trials held at London’s central criminal court. Beginning in 2000, the project received several grants and ongoing assistance from three universities.

To create the site, the Old Bailey project team started with microfilms of the original documents and digitized them into high-definition files. These were then manually typed by two different typists, and the transcripts were automatically scanned to detect discrepancies, which were hand-corrected. A team of taggers and editors then marked up the transcript texts with XML tags to indicate 56 different crimes in 9 categories: everyone’s name, occupational label, gender, and other identifying information; times and locations; and other metadata. The first markup pass was done automatically, then checked and edited by hand. The marked-up texts were processed to create a searchable database, and the page images, transcripts, tagged transcripts, search engine, interpretive materials, metadata, and other information were made available on a continually evolving website. New additions to the site included links to other large projects (like the complementary resource London Lives, launched in 2010) and an application programming interface (API). The site received a significant upgrade (to version 9.0) while I was writing this piece.

The Old Bailey Online is an exemplary case of presenting the past on the web, but it is by no means the only example. Many other sites contain large digital collections of primary and secondary sources of historical interest. I have provided some detail about this project’s extensive scope because I want to draw attention to the contrast between the scale of gathering, preserving, and presenting activities and the scale of activities that focus on other aspects of digital historical research.

My involvement with the Old Bailey Online has been through a long collaboration with Tim Hitchcock, one of the project directors. He reached out to me in 2007 to let me know that he had been reading my blog on digital research methods, Digital History Hacks. We started working on a pilot project and soon settled into a practice of using bespoke text mining code to explore the Old Bailey as a ‘massive text object’.⁴

If one wants to create digital tools or methods to support historical research, one helpful approach is to try to capture the primitives that make up everyday research tasks. John ^{Unsworth (2000)} listed seven: discovering, annotating, comparing, referring, sampling, illustrating, and representing.⁵ The Old Bailey site supports some version of several of these activities, as do many online digital collections. Discovering, for example, is supported through keyword and statistical search and more serendipitous means like a display box on the landing page: “On this day in … 1725 [December 10], Margaret ‘Mother’ Clap was accused of keeping a house in which she procured and encouraged persons to commit sodomy. Read more.” When the user follows the ‘read more’ link, they can read the text, see a page image of the original, navigate to the immediately previous and following trials, and get associated records. The XML tags on the site and statistical search support the research primitive of comparing. Margaret Clap was found guilty and sentenced to the corporal punishment of pillory. Using search and tagging, we can discover that this punishment was used between 1675 and 1812 for various offenses, including perjury, seditious words, forgery, and fraud. An extended essay on the categories of punishment helps to contextualize our comparison of various trials.

Unsworth’s argument about the primitives was that “even these very basic scholarly activities [were] very poorly supported, if at all, with respect to networked electronic data.” He claimed that “the most interesting things that you can do with standalone tools and standalone resources is less interesting and less important than the least interesting thing you can do with networked tools and networked resources.”⁶ Two decades later, it is still the case that the scholar needs to assemble an ad hoc collection of digital tools to perform and keep track of their work. At a minimum, these typically include email, web browser, word processor, citation manager, spreadsheet, folders of files and images, and perhaps some specialized note-taking software, optical character recognition, database, and/or local search software. Primitives like annotating, referring, sampling, illustrating, and representing happen to some extent within tools, but more importantly, they also happen in the spaces between tools. No matter how ‘digital’ the historian, these spaces tend to be difficult to automate. This is partly because human care, attention, and judgment cannot be automated, and I am certainly not proposing that we try. However, in every workflow, the scholar builds on a sequence of activities (often structured as a loop), and there is the opportunity to make those activities legible to higher-order computational processes.

(A brief digression: one of the reviewers for this paper suggested that although I say I do not want to automate the human aspect of historiography, I seem to be doing so anyway. To clarify, I believe there are many clerical tasks that people have been forced to do in the past that no longer need to occupy us. If I write notes on 3x5 inch paper cards, for example, and I want to alphabetize or index them, I have no option but to do it by hand or get someone else to. Prior to the advent of mechanical or digital computing, compiling a concordance for the Bible or the works of Shakespeare might consume many years of one’s life. Now, it can be done almost instantly with a few lines of code. As these kinds of tasks become automated, they can be readily extended to other sources like the Old Bailey Proceedings, and I can devote more attention to trying to make sense of those sources. Of course, the recent addition of large language models to the ecology of research changes the calculus of ‘clerical tasks’ significantly. I will say more about this in the conclusion.)

Let us return to the Old Bailey site and detail a few steps in a sample research visit. On the site’s landing page , I opened the Margaret Clap trial in a new browser tab. As I read through her trial, I realized that I did not know much about the use of pillory in the eighteenth century, so I opened the link to the punishment essay in another tab. I read that, then jumped back to the trial. To compare the trial I was looking at with others, I opened the advanced search and statistical search pages in new browser tabs, then opened other sodomy trials in still other browser tabs. If your process is anything like mine, you keep several tabs open simultaneously to avoid prematurely closing something you need to annotate or refer to. Typically, I might also start noting URLs in my note-taking software. If I was working with more ephemeral sources, I would also save pages on the Wayback Machine of the Internet Archive for future citation and add each to my bibliographic database.

Now, in principle, all these activities are machine-readable. My browser history contains each of the links I visited with timestamps. My browser cache contains all the web pages downloaded to my machine. My note-taking software contains my annotations and, depending on the software, might also have a timestamped record of my edits. The same goes for my citation management software. However, I lack a system that draws from all these sources to automatically create an ongoing record of my process that I could return to in the future: on December 10, 2023, I started with the Margaret Clap trial of 1725. I explored sodomy trials and 18^th century uses of corporal punishment in England.

With the press of a button, that same system could also do a quick focused pass of web crawling and text mining and summarize the results. At its most basic, this would be an automated one-hop search, but it could be extended to draw on machine-readable resources in the form of linked open data. Adjacent to the pages I consulted are others I had not yet seen but which provide germane material. For example, I did not follow the link from the trial to the punishment summary page. It contains the details of Margaret’s punishment: “Margaret Clap to stand in the Pillory in Smith field, pay a Fine of 20 Marks, and suffer two Years Imprisonment.” Given my information-foraging behavior to this point, a system observing my research process could surmise that I was interested in both Margaret Clap and Pillorying and would surface that sentence along with a link to the page where it appears. That same page shows that three others were pilloried, fined, and imprisoned due to trials in that session. If I were informed of this, it might guide my exploration. Or I could ignore it. Rather than having a bunch of tabs open and flipping between them, the system might also provide a more synoptic view of my research process, showing me keywords and snippets of what I had explored so far and providing leads to other possible items of interest. In fact, some off-the-shelf apps encapsulate some of the functionality I have described, such as devlonthink and devlonagent for the Mac.⁷

Sensemaking in Intelligence Analysis

As you search through external data sources, you find some items of interest and filter others out. Your reading directs an ongoing process of further searching, finding, and filtering. However, this is only the lowest level loop in a typical research process. Peter ^{Pirolli and Stuart Card (2005)} of the Palo Alto Research Center (PARC) developed one influential model of the entire process.⁸ Using a think-aloud protocol and cognitive task analysis, they studied how all-source intelligence analysts transform raw information into results they can report on. There are important differences between the work of intelligence analysts and historians, to which I will return later. In the meantime, it suffices to say that many former and current intelligence agents were trained as historians. Both careers require making sense of large collections of unfamiliar documents, dealing with uncertainty and ambiguity, finding patterns, and rigorously supporting inferences with evidence. For five years, I have taught upper-level undergraduate courses on structured analytic techniques for intelligence analysis in addition to my regular courses on digital research methods. History majors who take my courses do very well in intelligence analysis, and to be honest, considerably more of them have chosen to pursue careers in intelligence than in digital history or digital humanities.

The Pirolli and Card model (Figure 1) consists of a large loop between Reality and Policy (not shown), which in turn contains a Foraging loop to extract information and a Sensemaking loop to model the evidence. At all stages, analysts make use of both “bottom-up processes (from data to theory) [and] top-down (from theory to data) … invoked in an opportunistic mix.” Getting from sources to an interpretation requires effort and increased structure. Pirolli and Card are explicit that moving forward in research often involves much backtracking. The first four steps of their model are as follows. Starting with external data sources (1), the researcher uses a combination of searching and filtering (2) driving further searches for information (3), collecting relevant sources to store in a collection that Pirolli and Card call a “shoebox” (4). This first loop from external data sources to the shoebox makes up the first half of their foraging loop, and it corresponds closely to the sample research visit to the Old Bailey Online, which I described above.

^{(adapted from Pirolli & Card 2005)}

[Figure 1 The Pirolli and Card Model] 

The second half of their foraging loop consists of three more steps. The researcher reads items in the shoebox more carefully, skimming and extracting (5), to drive further searches for relations between the items there (6), collecting snippets, “nuggets of evidence” and “related low-level inferences” in a collection they call the “evidence file” (7). At any point, the researcher’s developing ideas about what they are reading can trigger new hypotheses and new iterations of search and filtering from external data sources. This complete foraging loop could be better supported than it is by the automation I describe above. As the process of foraging extends over many research sessions, it becomes more important to have explicit ways to represent where the researcher has been and where they have not. This kind of information might be automatically compiled in a diary of sorts and represented in the form of timelines, networks of links between documents, glossaries of key search terms, and lists of named entities (people, places, institutions, websites). The “shoebox” collection level corresponds to files in folders, whereas the “evidence file” collection is more likely to be spread across citation manager, note-taking, and other software.

The processes described by Pirolli and Card’s foraging loop are a good match for the activities of any working historian, digital or not. They can be readily implemented with traditional non-digital sources and finding aids, as well as traditional paper notebooks, file folders, and 3x5 cards. In the digital realm, in fact, many of our low-level tools, processes, and metaphors remain wedded to traditional analogs. Digital files, folders, notebooks, wastebaskets, operations like copying and pasting, and browser and reader software all suggest affordances for scholarly work. These permit the transfer of traditional skills and understandings while making it harder to imagine or desire other possibilities.

At the next stage of their model, the sensemaking loop, digital and non-digital practices diverge, and differences between historical subfields become starker. As with the foraging loop, the sensemaking loop comprises two smaller loops. Starting from the evidence file (7), the researcher organizes information schematically (8). These schemas may be timelines or “small-scale stories about typical topics or in answer to typical questions (e.g., who, what, when, where, why, how).” The schemas might include descriptive statistics or mathematical models for more quantitative inquiries. Schemas drive further searches through the evidence (9) and are stored in a collection of schemata (10). The second half of the sensemaking loop begins with schemata (10), which are used to generate hypotheses (11), which, in turn, drives a search for evidence to support or disconfirm hypotheses (12). Those hypotheses which are not disconfirmed are stored (13). At any point in the process, earlier loops are invoked as necessary. As a whole, the model describes a series of steps from external data sources to shoebox, to evidence file, to schemata, to hypotheses. Both effort and structure increase at each step. At the end of both foraging and sensemaking, the hypotheses are used to tell a story (14), which may lead to re-evaluation (15). This is just a model, of course. Actual research is messier. However, it is a model that we can use to think about designing tools to support the kind of work historians need to do once evidence for the past has been gathered, preserved, and presented in digital form.

Bespoke Code for Digital History

Although the “sample research visit” I described above is a fair depiction of how I might interact with a website on the first or second visit, it is not representative of my workflow for a more sustained project. In fact, I have rarely visited the Old Bailey Online website in the two decades I have worked with the collection. For me, a typical digital history project begins with a bulk download of sources if that is an option (for both the Old Bailey and other sites like Wikipedia and the Internet Archive). If that is not an option, I use a web crawler to harvest the site (again, assuming that is permitted by the terms of service). If the site has already been crawled by an institution like a library or archive, it may also be possible to work with web archive (WARC) files ^{(Milligan 2020)}.⁹ If neither bulk downloading nor web crawling is an option, but the site makes an API available, I will work through that. Using a web browser and clicking links by hand is the last resort, and even that can often be automated to some extent.

Let us turn to the situation of a historian writing custom code to make sense of many sources, primarily textual, typically for themselves or a small research group. Since research questions are much more specific in this kind of work, there have been relatively few large-scale grant-funded projects for tool development. I will draw on my experience working with Tim Hitchcock on the Old Bailey. We used well-known techniques drawn from the literature of information retrieval, natural language processing, and text mining, so our methods are more generalizable than our questions.

We started with a complete set of marked-up transcripts from Old Bailey Online. This saved us the labor of finding sources, digitizing them, doing optical character recognition (OCR), preprocessing, transcribing, and marking them up, stages that are otherwise necessary in the pipeline from traditional to digital text. Since we were working with almost 200K transcripts on commercial desktop and laptop computers, we would open a file, extract information, transform it somehow, and write the results to a new file. During our work together, we have created more than 1.8 million derivative files so far of things like word counts, n-grams, and TF-IDFs (a statistical measure of the importance of a term in a particular document drawn from a corpus of documents). A flow chart of our text mining transformations is shown in Figure 2.¹⁰ Some of our transformations resulted in digital versions of traditional tools, like concordances or glossaries of named entities. However, we also used machine-readable files for more specific computational tasks such as unsupervised clustering, supervised machine learning, and visualization.

[Figure 2 Text-mining transformations of Old Bailey sources from ^{Hitchcock and Turkel 2021a}] 

As I have drawn it, the flow chart of transformations is what a computer scientist would call a directed acyclic graph. If you start somewhere and follow the arrows, you never revisit an earlier stage: there are no closed loops. The figure describes the computer’s view of our research process. At every stage, however, Tim and I engaged in interactive sensemaking processes that were not legible to computation. All the digital history projects I have worked on in the past few decades have taken this form. Automation in the pipeline has very much focused on bottom-up processes that create the evidence which scholars will interpret. Looping and top-down processes are not automated. Instead, the bottom-up processes are adjusted and rerun from the beginning as necessary, and the sensemaking loop remains firmly in the domain of traditional, non-digital scholarship.

Tim and I are currently engaged in some new work on understanding the category of manslaughter.¹¹ Using unsupervised clustering on trials represented as stable random projections,¹² we began to explore the kinds of events that would result in a verdict of manslaughter. As a subject expert, Tim had some ideas of basic categories we might expect to see: domestic violence, fights, medical malpractice, and traffic accidents. Some of these (especially traffic accidents) were clearly clustered in the dimension-reduced spaces we visualized. Others were not. We would discuss some results, adjust the parameters of our clustering algorithms, and rerun them. Reflecting on the fact that our tools were domain-neutral and incapable of incorporating expert human input directly, we began to search for some way to start to close the loop. To date, this has yet to be a priority for tools designed for underfunded historical or humanistic research. Which takes us back to intelligence analysis.

The Emergence of Structured Analytic Techniques in Intelligence Analysis

There are two relevant trends in the recent history of intelligence analysis that have implications for digital historical research. The first is a thoroughgoing disciplinary engagement with problems arising from heuristics and biases in human cognition and the emergence of structured analytic techniques (SATs). The second is the pressing need for intelligence agencies to engage with web-scale digital information in real time. In both cases, highly publicized intelligence failures spurred methodological reform and innovation rounds.

Although there were some historical precedents, intelligence analysis as an independent discipline arose during World War II and blossomed in the Cold War. Here I will focus primarily on developments in the US as they contextualize the work I want to describe. In 1941, President Roosevelt created an office to “collect and analyze all information and data which may bear upon national security.” The Japanese attack on Pearl Harbor later that year has often been interpreted with hindsight in terms of intelligence failure: bureaucratic impediments, failure of what is now called “information fusión”, and failure to use “indicators”, measurable phenomena that characterize emerging trends and can warn of unanticipated events. At the time, things seemed considerably less clear-cut, but the events nonetheless exerted a strong pull-on intelligence-related decision-making. The new office was headed by William J. Donovan, who created a Research and Analysis branch. Donovan chose historian James Baxter to head R&A. Baxter would win the Pulitzer Prize for History in 1946 for his monograph about WWII weapons development Scientists Against Time. Harvard historian William L. Langer was named deputy, and Baxter and Langer began to recruit scholars to the task. When the Office of Strategic Services was formed in 1942, the Research and Analysis branch was shifted under its control. Historian John H. Hedley writes: “R&A would grow to more than nine hundred analysts before the war was over, comprising a ‘chairborne division’ of OSS officers whose intellectual inquiry cast a wide net in support of combat operations and wartime and postwar planning”.¹³

The Central Intelligence Agency (CIA), Department of Defense, and National Security Council were created by President Truman in 1947. Events of the early Cold War -Soviet activities in Czechoslovakia and Berlin, the Soviet atomic bomb, a newly communist China, and the surprise invasion of South Korea by communist North Korea- would also be interpreted in terms of intelligence failures. In 1952, Sherman Kent was chosen to head the Office of National Estimates. Kent was a Yale history professor recruited into the oss’s Research and Analysis branch and had written the first American analysis of strategic intelligence. He was a crucial figure in the creation of intelligence analysis as a discipline, insisting that analysis be neutral concerning policy and rooted in an evidence-based literature constructed by scholars reviewing and responding to one another’s work.¹⁴ “Sherman Kent’s personal hour of reckoning came in 1962,” CIA historian Donald P. Steury wrote, “when the Soviet Union decided to deploy nuclear-armed ballistic missiles to Cuba. The Board of National Estimates not only failed to predict this action (or anything like it) but argued resolutely against its likelihood, right up to the moment that offensive missiles were discovered in Cuba.”¹⁵ The Cuban Missile Crisis highlighted the problem of groupthink and drove the intelligence community to search for techniques to mitigate it.¹⁶

In the 1960s and 70s, Richards J. Heuer began to apply methods and ideas from cognitive science to intelligence analysis. He was mainly influenced by the work of Daniel Kahneman, Amos Tversky, and their colleagues, who were beginning to show how human thinking is subject to various biases. Even knowing that we are predisposed to use error-prone heuristics does not help us to avoid them. The only remedy is to slow down, be explicit about our thinking processes, and try to expose them to the critique of others. Heuer’s work extended the understanding of intelligence failures from the policy-oriented study of diplomatic and military history to a cognitive psychology of the mental processes of intelligence analysts.¹⁷

In the Nixon administration of the early 1970s, there was a renewed push to use ‘new analytic methods’ and to draw from the findings of the ‘new social sciences’. rand Corporation and arpa were given more extensive roles in intelligence-related research. The intelligence community responded by promoting rigorous public models of thinking that could substitute for individual intuition. These were based on recent studies of organizational behavior, decision-making, and Bayesian techniques. The Delphi method of forecasting-the use of structured surveying of expert panels developed by rand researchers in the 1960s-also became popular. These new kinds of analyses were used to assess situations from the Vietnam War and were also focused on Sino-Soviet and Arab-Israeli hostilities. After a burst of activity in the early 1970s, such techniques continued to be used and developed over the next two decades, but their uptake in the intelligence community was sporadic. Reflecting on his experience at the CIA in the late 1990s, historian Stephen Marrin wrote, “No one I knew - except for maybe the economic analysts - used any form of structured analytic process that was transparent to others. No quantitative methods; no special software; no analysis of competing hypotheses; not even link charts.” Two more high-profile intelligence failures, the 9/11 attacks and misleading statements about Iraqi weapons of mass destruction (2003), would soon provide the impetus for reform.¹⁸

A second historical trend that shaped the discipline of intelligence analysis is computerization. The relentless pace of innovation characterized by Moore’s law transformed the room-filling behemoths of the late 1940s into devices that could fit into a briefcase or pocket. ARPANET consisted of nine nodes in the summer of 1970; by 9/11, there were over 500 million internet users and an estimated 4 billion web pages. Like practically every other human enterprise, the intelligence community struggled to keep pace with the speed and scale of change. This is a much larger story than I can tell or summarize here, so I will focus on structured analytic techniques (SATs) and the development of digital tools to support intelligence analysis.

There are now a wide variety of SATs, and many intelligence agencies consider their use to be best practice, although there is an ongoing dispute about their effectiveness.¹⁹ The third edition of Randolph H. ^{Pherson and Richards J. Heuer’s Structured Analytic Techniques for Intelligence Analysis (2021)} lists a core collection of sixty-six SATs drawn from several hundred options. These are categorized into six families: getting organized, exploration techniques, diagnostic techniques, reframing techniques, foresight techniques, and decision support techniques.²⁰ SATs for getting organized and exploring cover the same kind of territory as the primitives that John Unsworth proposed and the foraging loop of Pirolli and Card but add a variety of forms of structured brainstorming (to avoid the perils of groupthink and other cognitive biases) and various kinds of conceptual mapping. Many of these techniques can be adapted by historians and other humanists who work collaboratively or adapted for use in the classroom.

Techniques in the other sat families might apply to historical practice in some cases but are more directed to the specific concerns of intelligence analysis. Diagnostic techniques, for example, contain many options for generating and testing hypotheses, including a widely used technique pioneered by Heuer called Analysis of Competing Hypotheses (ACH). It involves testing the consistency or inconsistency of each observation or piece of data against a set of explicitly specified hypotheses so that the least consistent hypotheses can be discarded. SATs that are related to ACH focus on the ‘diagnosticity’ of items of evidence. Suppose, for example, that seemingly healthy people begin dying with mysterious flu-like symptoms, as happened in the Four Corners region of the US in May 1993. Many of the victims were Native American Navajo people. Four explicit hypotheses might be (1) the deaths were due to a virulent version of a known pathogen, (2) they were due to a new pathogen, (3) they were due to exposure to a toxic substance like a pesticide, or (4) the Navajos were the deliberate target of a hate crime. Each item of evidence is tested against each of the hypotheses and rated on a scale from “very consistent” to “very inconsistent” (or not applicable). One item of evidence in the case is that there is a high mortality rate. Since this is consistent with all four hypotheses, it does not help us to disconfirm any of them, so this evidence is not diagnostic. A second item of evidence is that tests for common flu and bacterial agents are negative. This is very inconsistent with hypothesis 1 (known pathogen), inconsistent with hypotheses 2 (toxic substance) and 3 (hate crime), and consistent with hypothesis 4 (new pathogen). So, this one item of evidence is very diagnostic because it gives us one way to distinguish between competing hypotheses.²¹ The ach technique marshals the diagnosticity of all the evidence to rank competing hypotheses in terms from least to most disconfirmed. The epistemology is explicitly Popperian: one can never prove hypotheses, only disconfirm them.²²

Reframing techniques focus on approaching a problem from multiple perspectives. Experts are reluctant to abandon mental models that have provided accurate assessments to date, leading to a ‘paradox of expertise’. “Experts can be the last to recognize the occurrence and significance of change,” ^{Pherson and Heuer (2021)} write, “For example, few specialists on the Middle East foresaw the events of the Arab Spring, few experts on the Soviet Union foresaw its collapse, and almost every economist was surprised by the depth of the financial crisis in 2008.”²³ By reframing a familiar problem in unfamiliar terms, these SATs are designed to activate different mental pathways. One of my favorites to use with history students is called Premortem Analysis. Suppose that some point of consensus is found to be wrong in the future. How and why did that happen? What aspects of the current orthodoxy are least well supported? Foresight SATs are designed to help decision-makers identify driving forces in the present and prepare for unanticipated changes in the future. Some of these are based on the well-known Scenarios developed at Royal Dutch/ Shell in the late 1960s, allowing the company to successfully navigate the OPEC crisis.²⁴ Others involve counterfactual reasoning and the generation of indicators. Decision support SATs include techniques based on decision trees and critical paths, assessing strengths, weaknesses, opportunities, and threats (SWOT), and analyzing the forces at play in a particular setting.²⁵

I have described various SATs in more detail than required for the day-to-day work of (digital) historians. This is because almost all the techniques can be adapted for use with history students, giving them new perspectives on disciplinary methodology. Topics like counterfactual thinking, deception, indicators as a kind of retrospective clue, the diagnosticity of evidence, and driving forces can all be used to discuss how historians map the past. Although most historians wisely avoid forecasting, future-oriented techniques can be considered when discussing historical situations. This helps students avoid hindsight bias: we know how it turned out, but the historical actors we are studying face a world of possibility and uncertainty. When students work through the “Death in the Southwest” case study, taking the perspective of medical investigators faced with the mysterious 1993 illness described above, they end up with a very different understanding of that moment in time than the ones who merely learn that there was an outbreak of an unknown strain of hantavirus that year. I often ask more advanced students to write a reflective essay on Gregory F. Treverton’s distinction between ‘puzzles’ (“questions that have a definitive answer in principle”) and ‘mysteries’ (“questions that cannot be answered with certainty”) in intelligence analysis and discuss the degree to which they think such modes of reasoning can be applied in historical work.²⁶

Structured analytic techniques also have a place in the general education of citizens. In How Spies Think, Sir David Omand argues that the techniques of intelligence analysis can be used by any individual “in the digital age, bombarded with contradictory, false and confusing information from more sources than ever … [with] influential forces at play ranged against us pushing specific messages and opinions through social media.” Omand, who was director of the uk’s gchq in 1996-97, divides the techniques into how analysts see situational awareness, explaining facts, estimation with models, and strategic notice (“we do not have to be so surprised by surprise”); how our reasoning is biased and how to detect manipulation and deception; and how to make intelligent use of intelligence. He aims to teach citizens how to fend off digital subversion.²⁷

Although SATs can be taught and learned in pencil and paper settings, they have also been subject to increasing computerization. ach, for example, was used as a manual method at the CIA from the mid-1980s but was computerized via a federal grant to parc in 2005.²⁸ This provides a context for the ^{Pirolli and Card study I discussed above, which was also conducted at parc in 2005}. Earlier information foraging studies by Pirolli, Card, and their colleagues focused on business intelligence rather than national security. For example, their 1999 Psych Review paper examined the manual workflow of an analyst faced with a library that received about 600 magazines (34K pages) a month.²⁹ Monthly, the analyst would select magazines to scan, mark articles to return to, assemble a pile of sources for the project (about 3000 pages), reduce this to a pile that was going to be used for writing (about 250 pages), and finally write a newsletter about recent developments in computer science or materials science from those sources. By contrast, the analysts in their 2005 study were engaged in tasks such as predicting whether a coup would occur, understanding future bio-warfare threats, mapping drug cartels, and studying terrorist groups. Between 1999 and 2005, of course, events such as 9/11 and the Iraq WMD estimates had completely refocused American research and development in information retrieval.

Computational Tools for Sensemaking

“[I]n some sense,” Gregory F. Treverton wrote in the foreword for a 2011 book on sensemaking in intelligence, “the Cold War practice of analysis sought to turn humans into machines by rooting out judgment, bias, hunch, stereotyping -all the things humans do best. The new paradigm makes machines and methods imperative, letting machines do what they do best -searching large amounts of data, remembering old patterns, and the like- while letting humans use the judgment they alone can apply”.³⁰ Computational tools designed for intelligence analysis now support exploration and discovery in extensive source collections and include mechanisms for incorporating sensemaking. I will describe a single example and then turn to some techniques that digital historians might experiment with in their code.

The TexTonic project allows the analyst to visualize and navigate unstructured text collections that would be too large to read.³¹ The creators show how the interface scales smoothly to provide synoptic overviews of collections of 5K, 500K, and 5M documents collections. In keeping with much prior research on visualizing unstructured text collections, TexTonic uses a geospatial metaphor, where shorter distances between points on a two-dimensional map reflect their relative similarity. The interface looks a bit like continents and islands of pastel-colored countries.

Each is automatically labeled with one or more key phrases representing the region’s documents}. As the analyst zooms into a particular region, they can search for and “pin” particular terms and move critical phrases concerning one another by dragging them around the map. “When a user interacts with the visualization, the system implicitly interprets their actions. This updates the term weights in the similarity matrix, resulting in a real-time, user-driven analysis. This, in turn, changes the data model, which is then reflected in the visualization.” This real-time sensemaking loop is semantic interaction: meaning is fed back into the model by the direct manipulations of the analyst. Behind the scenes, documents are indexed using Rapid Automatic Keyword Extraction (RAKE)³² and stored in a Lucene index. Similarity is calculated with K-means clustering, and principal component analysis (PCA) is used to reduce and visualize dimensions. In one sample task, ten experienced analysts were asked to use the system with 5M documents from Wikipedia to “explain the significance of ‘oak leaves’ in history.”

I will return to the manslaughter research that Tim Hitchcock and I have been working on to provide concrete examples of incorporating the sensemaking loop into bespoke digital history tools.³³ We began by using unsupervised clustering and dimension reduction to create a spatial metaphor: documents near one another are more similar than those farther apart. This is in keeping with the TexTonic example and the extensive literature on using spatialization for sensemaking. The main problem with using clustering was that we needed a mechanism for incorporating Tim’s expertise as we explored the space of trials. So, we experimented with ways of incorporating direct manipulation and semantic interaction. For example, one of our explored mechanisms allows the analyst to drag points around in a scatter plot. For our application, each point represents a criminal trial. If fights are one kind of manslaughter trial that appears reliably in our sources, dragging a few of them into the same region enables the system to learn to emphasize the features of those sources. By increasing the weight of those features in the distance metric, the space is continually remapped, and trials that Tim has decided are like one another, moving closer together.³⁴ A different experiment allowed the analyst to place ‘magnets’ in the space that attracted or repelled points. This ‘dust & magnet’ model³⁵ worked less well for us because it is designed for more structured datasets than raw or marked-up text.

Our most successful experiment used a mechanism we adapted from the Forcespire system for text analytics.³⁶ In this system, the documents are placed in space as the vertices of a force-directed graph where the edges represent shared keywords. Rather than having the analyst directly drag the vertices around, they highlight essential terms in the texts as they read. The highlighted terms are then used to increase the strength of connections between related vertices (texts), so they move closer together in the graph. At first, we worked with fully connected graphs, but we discovered the system worked better for our purposes if we used the text with the highest TF-IDF value for a highlighted term as a central vertex and linked other texts containing that term to it. We also created an interactive interface for working with the highlighted keywords, using balanced F-scores to assess the degree to which one keyword predicts another.³⁷ If the analyst highlights the word ‘accident’ in the manslaughter trials, the display automatically updates to show all the other terms that are predictive of that one, and vice versa. In the manslaughter trials, words like ‘wheel’, ‘driving’, ‘horse’, ‘coach’ and ‘rate’, immediately drew attention to the importance of traffic accidents in these trials.

The recent explosion of interest in large language models (LLMs) has raised the possibility of a different kind of collaboration with artificial intelligence in the sensemaking process. The significant systems all have application programming interfaces (APIs) so that bespoke code can be used for research tasks, and we have been enthusiastically taking advantage of this. To mitigate against the LLMs’ tendency to ‘hallucinate’, we are using retrieval augmented generation (RAG). This technique grounds the system’s responses in information that can be readily verified rather than using its general world knowledge. RAG, in turn, depends on semantic search, which matches texts by embedding them in a highly multidimensional space. These embeddings give us a different kind of similarity measure for texts, which is less dependent on the exact terms used in the texts than TF-IDF or related measures. Semantic nearness opens a wealth of new possibilities for implementing sensemaking in the kind of bespoke software described above.

The abilities of LLMs are improving week by week, but they are already capable of performing many tasks that go well beyond the ‘clerical’ examples I mentioned above. We currently use them for summarizing, rephrasing, extracting keywords, semantic search, categorizing images, optical character recognition, correcting errors, detecting changes, critiquing arguments, suggesting secondary sources, drafting code, and many other tasks. The scale and pace of change have already made these systems very useful to historians and other scholars, and that trend seems like it will continue. Nevertheless, the critical thinking, nuanced interpretation, and original insights that define scholarship will remain uniquely human.