Teaching computers how to write

With the release of ChatGPT in November 2022, natural language generation became a mainstream concept in ways I could never have imagined. The seemingly effortless writing produced by today’s large language models makes it easy to overlook the challenges we faced while building a more rule-based engine at Narrative Science. Yet, there's something enduringly relevant about the transparency of the system we designed in 2016. Unlike large statistical models, our platform wasn’t a black box—every output could be traced back to a known ontological concept or a testable analytic within our framework. This is the story of that platform: Quill.

The business

Narrative Science was founded in 2010 by two Northwestern University Computer Science professors and their business partners. A handful of graduate students had created a program that could generate short recap stories about little league baseball games, given data such as player names and scores.

The business opportunity became clear—every day we generate more and more data that requires manpower to be analyzed and shared in a meaningful way. The game summarizer served as a prototype for the company’s first platform, Quill, which could be configured to construct sentences and paragraphs based on structured data.

In November 2021, Narrative Science was acquired by Salesforce and integrated into their sprawling teams and resources. However, when I joined in early 2015, the company was about 70 people strong and growing quickly. The majority of business came from financial institutions wanting to automate their portfolio commentary. In a traditional set up, financial advisors spend a significant amount of time writing summaries of each client’s portfolio performance, to be shared alongside graphs and charts on a quarterly basis. The writing required is straightforward, based entirely on data, and very routine. With Quill, they could generate summaries in seconds that were accurate and indistinguishable from the human-crafted alternative.

The premise of the new platform

I was hired to support the engineering team as they embarked on creating the third version of the software. Quill, like many early platforms, was designed almost entirely by engineers. While it supported an impressive spread of clients and their use cases, it was extremely complicated to configure and required an internal professional services team to run and maintain each project.

The goal of the ambitious redesign was twofold: one, to make the platform self-service with a simple and intuitive user experience, and two, to transform the underlying representation from a rules engine of text snippets and data transformations into an intelligent combination of three separate things: the language knowledge specific to a domain, the author’s communication goals arranged into an outline, and the data requirements that would map it to the source file or database.

The early user interface

The reimagined Quill user experience had four primary screens within the context of a project: the outline, the ontology, the data requirements, and the live story.

Within the outline, the project author would arrange communication goals. The goals were generalized, analytical statements, such as “Present the attribute of an entity,” and “Compare the attribute of entity 1 to entity 2.” The author would then configure these goals with the specifics of the story—pulling language from the project ontology—into statements like “Present the grade of the student in 2015.”

The ontology stored the language settings specific to the project. The author could create entities, such as a “student” and “class”, relationships, such as “is enrolled in,” and attributes like “name” and “ID”. These language settings included multiple expressions, for example “high schooler” might be an alternative to “student.” The ontology could be saved and shared to other projects. We aspired to have language models for any imaginable domain that an author could pull into their project from a global knowledge base.

Within the data requirements, the author would map the entities and attributes referenced in their outline to fields from a data source. So for example, the names of students might be found in a field called “full_name” from a table called “student_info”, while their grade might be pulled from the field “final” in a table called “BIO201_2015”.

Finally, the live story page would show the generated story. The author could preview different versions by flipping the data source or a high level filter. They also had the ability to edit expressions, and resolve errors in line.

At a high level, the user interface borrowed from other creative tools, where the body of the page reflected the state of the project, and configuration and settings were handled in a set of dynamic menus along the right. We designed many of these sidebars to be accessible from multiple screens, for example, the menu for editing an expression could be triggered from either the ontology or the live story. While there were many screens to navigate, most new users quickly understood the concept of the outline, how it influenced the data requirements, and what to expect from the live story page. The UI was successful in many ways, and for a first-of-its-kind tool, had a relatively shallow learning curve.

Three years of learning

I learned more each day that I worked at Narrative Science than during any other period in my career. What I learned at Narrative Science could fill a library, but I have condensed the major takeaways into the following three themes.

First, no matter how revolutionary the technology is, a product must be designed for real people with a real use case. Our reluctance to narrow our focus to a target customer and define our product against their needs and motivations made it exceedingly challenging to agree on our priorities. This eventually led to a fairly dramatic product and leadership pivot.

Second, when designing new technology, it’s incredibly important for teams to focus their creative energy on the unsolved problems. Many of the experiences we designed in Quill had existing solutions, and the time we spent reinventing them was time we didn’t spend focusing on the problems that were truly unique to our product.

And finally, great teamwork has magical properties—five people with the right match of skills and team dynamics can accomplish what 50 people can in the same amount of time. I am so thankful for the exceptional people I had the opportunity to collaborate with on Quill, but especially the other two members of the Design trio, Andrew Paley and Sharon Chen