A Partnership with the USC Digital Repository: Using Advanced Tech to Bring People Closer

The user experience with the Interactive Interviews feels smooth and effortless. But this is only possible thanks to Bass Lab’s partnership with the USC Digital Repository. The Digital Repository’s team built a custom AI model that enables more genuine connections between visitors and interviewees. While the technological achievements are impressive, the end product does more than raise eyebrows: It raises consciousness. Here’s a breakdown of the technology behind the Interactive Interviews and how it has fostered so many unique, multi-dimensional experiences.

How the AI Behind the Interactive Interviews Works

The USC Digital Repository built a custom conversational AI system for all of the Interactive Interviews. As the Repository’s Mike Jones explains, “The system is based on state-of-the-art embedding models for the question matching. All of the interview questions and answers are embedded into vectors, which allow the machine learning model to compare the similarity (or difference) of words and phrases.”

According to Mr. Jones, the system takes the vectors, which are numerical representations of the meanings of words and phrases, and uses them to produce natural responses to questions.

“After the user asks their question,” Mr. Jones explains, “it is converted to vectors as well, using the same embedding model, and that is then compared to all of the question-and-answer embeddings from the interview to find the best match.”  

An Example of How the AI Produces Answers

Mr. Jones provides an example that makes it clearer how the model matches answers with users’ questions:

“Using an advanced embedding model allows for more variation from the user’s question and the original questions/answers to find the best match. For example, in the interview, we may have asked “What’s a favorite memory you had with your father?” but the user asks “Can you share an unforgettable experience you shared with your dad?” Even though the keywords are different between each question, the system understands that semantically, they are very similar, and can return the corresponding answer from the interview.”

Let’s dive just a little deeper into how this process works.

How Vectors Help Match Questions with Answers

While we don’t have the space to dive into a full lesson illustrating the data science behind language models, here’s a quick breakdown of the vector-based embedding process. 

Each word, phrase, (and, by extension, the question or answer they make up) gets assigned multiple numerical values. Each value represents how close an element of the word or phrase is to a certain meaning. 

As vectors, the numerical values get plotted on a multidimensional graph as points. When the system sees points that are close together, it concludes they have similar meanings. It interprets points that are farther apart as having different meanings. 

For instance, the words “dad” and “father” would have points that are very close together. So, as Mr. Jones’ example explains,  when someone asks a question using the word “dad” instead of “father,” the AI can still understand what the asker means. 

How the AI Responds to Out-of-Scope Questions

So, what happens if the question in the pre-recorded answers does not match? How does the system respond? 

Mr. Jones answers, “If no good match above a certain threshold is found, the system responds with a Fallback Intent (such as ‘I wasn’t asked that in my interview.’) to indicate to the user to ask a different question or rephrase their original question.”

In this way, the Interactive Interviews help keep conversations on topic and relevant to both the interviewee’s experience and the asker’s questions.

Next Steps for Interactive Interviews

The Interactive Interviews are poised to evolve, and their growth has already begun – on the other side of the country.

The USC Digital Repository has engaged in another project that shows the immense potential of this unique form of journalism. “We recently debuted a new format of the Interactive Interviews with the Museum of Jewish Heritage in New York, which we refer to internally as a Collective Interview,” Mr. Jones says.

The technology underlying the project is similar, but the format is significantly different. The result is a more intricate array of answers that enable deeper interactions between visitors and interviewees.

Mr. Jones breaks it down like this: “Rather than interviewing a single subject for multiple days, we interviewed 10 different Holocaust survivors each for a half-day, asking each of them a combination of shared and unique questions.”

The process can potentially produce more nuanced conversations because “each survivor had their own experience, but some of their experiences were similar (multiple survivors were in Auschwitz, multiple were in hiding, etc.). The collective interview enables a student or visitor to ask a question, hear multiple responses from different perspectives, and ask follow-up questions to the individual responding or back to the entire group.”

This more dynamic, advanced approach requires a combination of innovative data science and good old-fashioned elbow grease. “This approach requires a bit more work in answer mapping and data preparation by our team following the interviews,” Mr. Jones adds. “But we’re regularly evaluating new models and systems that can help streamline the process.” 

The Future of Interactive Interviews: Augmented and Virtual Reality

To create an even more immersive experience, the USC Digital Repository envisions expanding into augmented reality (AR) and virtual reality (VR). 

“We’ve also had discussions about incorporating our interviews in AR/VR environments. While there is certainly a cool factor to doing so, the technology is still either inaccessible to most people or limited in capacity to truly enhance the experience.” 

Until the tech becomes more accessible, AR and VR may have to wait. But that doesn’t mean the current iteration of the Interactive Interviews lacks impact.

The Empathic Impact of the Interactive Interviews

The Interviews are a tool for building empathy, both in the USC community and across the globe. Mr. Jones puts it well, explaining, “I have long believed that the best way to break down barriers between people, cultures, and beliefs is to immerse ourselves in situations that enable us to experience others first hand. You can read all about a foreign country, their food, their culture, traditions, etc., but you can’t truly absorb it until you experience it directly.” 

Experiencing culture is a macrocosmic representation of more personal interactions. As Mr. Jones puts it, “Similarly, you can study a person, read their works/posts/etc, and learn everything there is to know about them, but never actually KNOW them until you meet them.”  

This is what the interactive interviews bring to the table, a chance to digitally “meet” historical subjects. 

The Effect of the Bass Lab’s Pop-Up Installations

Instead of merely enclosing the Interactive Interviews behind walls of HTML and CSS on their website, the Bass Lab used them to create pop-up installations. These enable visitors to experience the interviews first-hand, while in the Bass Lab.

The installations further solidified the bond between genuine engagement and convenience. As Mr. Jones explains, “Our team felt that the Bass Lab and their partners completely knocked it out of the park with the current installation. It is an incredibly well-thought-out combination of a traditional exhibit with a seamless integration of very cool (obviously, I’m biased) modern technology. I think it has a ton of appeal to users.”

Experience the Interactive Interviews for Yourself

Whether you’re planning a trip to the Bass Lab or visiting online, you can reap the benefits of the Lab’s partnership with the Digital Repository. By engaging with each participant, you can start a conversation with history in a way that wouldn’t be possible without the Repository’s innovations and the Bass Lab’s vision. Give it a try today.

To donate to the Bass Lab, the Interactive Interviews, and other innovative projects, you can start here.