Data science has become a critical part of this deciphering effort, and Netflix is leading the data-driven analytics race. The most public display of these efforts was the 2006 Netflix Prize, an open competition to build an algorithm that could beat the company’s star-based recommendation system. The $1 million dollar prize, which was awarded in 2009 to BellKor's Pragmatic Chaos team, was the biggest jackpot in hacking contest in history.
But if you are a subscriber of Netflix, you might have noticed that the improved star system was never adopted. That's because by the late 2000s Netflix started taking a different approach to figuring out subscriber preferences. Rather than predict stars, Netflix began to predict genres, or, more precisely, 'microgenres' that provide extremely precise categories of content. Now, when Netflix makes a recommendation, it isn’t guessing the rating a user is likely to give; it is guessing the type of movie a user wants to see.
The use of microgenres might seem like a step away from personalization, yet the system seems so tuned in to subscribers’ tastes that it feels as if it knows them. Figuring out how Netflix accomplishes this became a recent obsession of Alexis Madrigal, a senior editor at The Atlantic and a visiting scholar at UC Berkeley’s Center for Science, Technology, Medicine and Society. Madrigal began to investigate, starting with an intensive web scraping effort to compile an exhaustive list of Netflix’s microgenres. His complete list of 76,897 genres affirmed that the categories are excruciatingly and often hilariously precise (genres like 'romantic Chinese crime movies', 'cult evil kid horror movies' or one of my favorites, 'sentimental set in Europe dramas from the 1970s').
Where did these genres come from? This was the next riddle Madrigal sought to unravel. Using textual analyses, Madrigal computed keyword frequencies, generating charts that produced some expected and not so expected findings. For example, the most frequently occurring adjectives among the microgenres were 'romantic', 'foreign', 'classic' and 'dark'. No surprise there, but then Madrigal stumbled onto a Perry Mason mystery. The actor with the greatest number of dedicated microgenres was Raymond Burr.
Madrigal’s efforts earned him a one-on-one with the creator of the microgenre system, Netflix exec Todd Yellin. As Yellin explained it, Madrigal was tapping into his team’s effort to use machines to build a grammar for Hollywood. In a document called (they now regret) Netflix’s Quantum Theory, Yellin and his team detail how they are trying to go beyond five stars and instead build a meaningful taxonomy of movies through microtagging.
The process begins with gathering descriptors for each item in Netflix’s catalog (estimated to have more than 60,000 items of streaming content alone) from recruited screenwriters. Recruits are trained with a 36-page manual on how to meticulously dissect the salient features of a film. Some of the features are straightforward facts, like where and when a film takes place. But many get pretty subjective, like gauging the 'social acceptability' of the lead character or the happiness of the film’s ending.
The database of microtags is then fed into a computer algorithm to generate microgenres. Essentially, Yellin and his team are training a machine to use microtags to summarize what movies are about in a language we can understand. Imagine if Siri could tell you what kind of movie American Hustle or Her is and you will start to appreciate the complexity of the tool the Netflix team has built. The actual genres that are formed are constrained to meet certain practical and grammatical requirements like sufficient number of members, a length of 50 characters or less, syntactical correctness, etc.
With microgenres, Netflix has created a kind of periodic table for film. The value of such a reference system goes well beyond its intended use for personalized recommendations. Insight could be gained into changing fashions in the movie industry by following time trends on the number of films each microgenre contains. Merging ticket sales and downloads with microgenres over time could even give us a window into broader cultural shifts. And, if the microgenre algorithm is fundamentally a pitch-making machine, it could soon be competing with screenwriters to come up with the next big hit. No wonder Netflix has gone into the production business.
