Documentation

Exploring a huge collection of documents is challenging and sometimes overwhelming. Typically the only means of access to large repositories of declassified information is to start entering keywords in a search box. But research in the world of legal discovery has shown that we are usually wrong to assume we know which words are actually “key.” The average user misses some 80% of relevant documents. Even for the documents we do find, it can be difficult or impossible to reconstruct their original context or significance. Digital archival research becomes a fishing expedition, in which we pull out one catch after another, throw most of them back, and finally seize on what may turn out to be a red herring.

What most of us are actually looking for is some way to make sense of the whole archive, and begin identifying the clusters of related documents that are most representative of the topics we are actually interested in. Luckily, computer scientists have developed a technique to accomplish this very thing, which they call topic modelling. One of the most important features of this family of models is that it assumes documents are about a mixture of topics rather than covering only a single topic. The family of topic models called LDA (Latent Dirichlet Allocation), treats each document as a bag of words, and then provides a probability that a word is generated by a certain topic. Read this paper by Blei, Ng, and Jordan for more details about LDA. History Lab uses this powerful method of exploration to offer a user-friendly way of looking at our collections. Our collections can be searched by topic here.

To generate our topics we use the Structural Topic Model (STM) package for R. There are two nice features of the STM package that led us to choose it over the more standard Latent Dirichlet Allocation (LDA) models. First, STM is an example of a Correlated Topic Model which means that topics within a document are allowed to be correlated. Traditional LDA models assume that topics within a document are independent. Second, and more importantly, STM has a "spectral initialization" setting. Traditional LDA models randomly assign tokens to topics so the results tend to be different each time an LDA topic model is run. Spectral initialization essentially uses the feature co-occurrence matrix to determine starting tokens for topics which means that the model will produce the same results each time it is run. 

We eliminate stopwords from each corpus and limit our words to those that appear in more than .5% of documents and to no more than 40% of documents. This eliminates words that are used in very few documents as well as those that appear in most documents.

Choosing the number of optimal topics is more an art than a science, though there are some general rules of thumb available. The STM package has a searchK function that produces a graph of 4 different diagnostic tests for a range of topic numbers. We ran the function for each of our collection, ranging the number of topics from 20 to 140. We choose the number of topics based on elbows in the diagnostic tests that show when there is a change in the pattern.

Number of topics by collection

In the end, we have a list of the most important words for each topic as well as the probability that a topic is covered in the document. The larger the value the larger the probability that the document is about that topic. We only show topics where its probability is greater than 0.05. We also limit the display of topics to cover a cumulative 75% of the probability. The webpage for each document will also show the topics covered by the document.

You can use our topic functionality on the search page. The topics tab shows the top documents for topics from each collection, ordered by the topic score. The drop-down menu will allow you to change collections and to choose from the available topics. 

Named Entity Recognition (NER) is the process of identifying proper nouns such as persons, locations, and organizations within documents. History Lab uses a customized spaCy NER model, trained on our documents, to identify the entities within the collections. Our documents were mainly written for individuals with knowledge of the context of the documents. Thus, when referring to individuals, the documents often use only a last name or a title and last name. We train the model so that it will return the title and the last name, which makes it easier to recognize the specific individual. 

NER uses contextual information to identify the type of entity. So the model will sometimes mistake a person for an organization or vice versa. We engage in post-processing to ensure that an entity has the same entity type across the documents. Most Wikidata IDs have an "instance of" property which gives a broad categorization of the entity. We group the instance of properties in our data into 5 broader categories that define the entity type: ORG, LOC, GOVT, PERSON, and OTHER.

We also created a Knowledge Base of political entities, including politicians, diplomats, countries, and organizations, that allows us to link many of the entities to a unique Wikidata ID. This allows us to reconcile variants of the same entity to a single ID, making it easier to search the collections for an entity. We started with the standard Wikidata names and aliases. Because the documents often refer to a title and last name, we expanded the Knowledge Base to include this type of entry. To distinguish between two different entities that share a name, we set up a disambiguation model that looks at how many overlaps there are between each ambiguous entities and the other entities mentioned in the document. For example, many countries have a "Socialist Party." But if Italy is mentioned near the name of the party, we can have a higher certainty that the party is the Italian Socialist Party and not the Chilean one or the French one. 

Entities can be searched on the search page. We eliminate all entities that do not have a Wikidata ID from the search function as well as entities that appear fewer than 10 times in a collection. The list of entities is in descending rank order but if one starts typing an entity name a list of entities containing the text will appear. 

We have isolated and stored the original classification level of a document where one was available. Users can limit their document search to any combination of classification levels that appear in the original classification dropdown of the search form. If you restrict your search in this manner, consider including the value unknown to include documents where we don't know the original classification.

In the case of the State Department collection, classification information was provided to us as part of the general metadata provided with the documents. In the case of the Clinton Email, the classification information we have is applicable to the entire thread (there is no email-specific classification information). Thus, the classification information we show for each email is the classification for the thread to which it belongs.

For the FRUS collection, we had to extract the classification of the documents programmatically. We did this by first finding mentions of the document classification level in the "source" field, and if it was not there, then we looked within any embedded source fields within the body text. If a source field could not be found, we looked for classification information between the title and the start of the body text and, finally, within the document's title. This resulted in classification information for about 75% of the documents in FRUS - many of the remaining documents are likely editorial notes or other reports with no visible secrecy information.

Full-text searching allows users to look for particular words and phrases within the body text of documents for a collection. Use double quotes for phrases, OR for logical or, and - for logical not.

A common but little-noticed problem with full-text searching of historical document collections is that many documents have garbled text because the original image quality was poor or because they were processed with outdated optical character recognition (OCR) software. The Freedom of Information Archive consists of a mix of “born digital” text and text derived using OCR. The majority of our collections, and about half of our total documents, have had OCR applied to page images. Some of the documents, like those from the CIA collection, were poorly copied.  As a result, there is a lot of garbled text, which may make search results misleading. We are working on techniques to identify and correct the errors.

Collections such as FRUS and CFPF are directly from the digital version so any errors were in the original version. That said, some text was lost before the documents were ever released to the public. About eleven percent of the State Department cables we received from the U.S. National Archives have error messages or blank text, and this is especially true of the “Secret” cables. Finally, some of our collections are redacted, including the Clinton e-mail and FRUS. We have done some analysis of the names most likely to be redacted, but there is obviously much more work to be done to correct for undercounting.