Foreign Relations of the United States

The Foreign Relations of the United States, colloquially referred to as FRUS, is a curated collection of the most important government documents in American foreign relations. The documents are chosen by a team from the Office of the Historian, and released in volumes that focus on specific themes or geographic regions.

The FRUS collection was made available through the History At State GitHub account. The collection was fairly metadata rich from the start, but there were various problems or limitations with the pre-existing metadata that we had to fix or enhance. For example, the persons mentioned in the documents were noted through markup, but this information was inconsistent, with varying spellings of names for one individual. We therefore had to “disambiguate” the data collection so that multiple spellings of people mapped to same person when counting or searching the collections.

We also extracted countries using a combination of Named Entity Recognition and a custom-scripted regex dubbed the “geotagger”. First, we went through an extensive brainstorming process to establish what should constitute a country mention in the first place. We decided to count countries only when they appear as “official mentions”, i.e. as a way to express the official position of this or that country. For example, “the Kremlin is hesitating to approve our blueprint for nuclear inspections” would count as a mention for the USSR, while “troops were moving through the Soviet countryside” would not - because “Soviet” is used as a location in this case. NER, since it tags countries according to the context of their mentions, also helps to differentiate cases like “Paris Peace Accords” to ensure this would not count for France. Once we had gathered all acceptable mentions of a place - we used our geotagger to ensure that only certain cities counted as stand-ins for the country, and that alternate spellings and old country names all routed to the correct country.

A chart showing the relative number of mentions for a region in the FRUS collection from 1940-1980 (based on aggregated country count information)

We also topic modeled the collection, using LDA followed by manual curation by a domain expert. Topic modeling clusters documents according to linguistic similarity, with special emphasis on words that might be considered especially significant (read more about the process here). You can explore the top topics for FRUS, as well as DDO and the Kissinger Telcons on our analytics page here.

Explore the FRUS collection and more with our search tool.