YouTube’s Quest to Suggest More

December 31, 2009

By MIGUEL HELFT
SAN BRUNO, Calif. — YouTube, the video site owned by Google, is about 10 times more popular than its nearest competitor. But Hunter Walk still thinks of it as an underdog.

For Mr. Walk, director of product management at YouTube, the competition is not other Web sites: it’s TV.

“Our average user spends 15 minutes a day on the site,” he said. “They spend about five hours in front of the television. People say, ‘YouTube is so big,’ but I really see that we have a ways to go.”

To that end, Mr. Walk leads a team of about a dozen engineers, designers and project managers who are fine-tuning YouTube to give its users what they want, even when the users aren’t quite sure what that is. The goal is to get them to spend a few more minutes on the site every day.

This is easier said than done. YouTube will not disclose the size of its video library, but the company has said that about 20 hours of video are uploaded to the site every minute. That is the equivalent of more than 100,000 full-length movies uploaded every week. With hundreds of millions of clips to choose from, the challenge that Mr. Walk’s team faces is to figure out how to select the 5 or 10 or 20 that a user might enjoy most.

The payoff could be large. Google executives have said YouTube is still losing money but is on a path to profitability. Getting users to spend more time on the site would help it get there faster by selling more advertising. It could also go a long way to extend its dominance, shoring up its position against sites like Hulu, which are attracting a growing number of users with full-length movies and television shows.

And it could also help YouTube keep up with other competitors, including video search engines like Truveo and Blinkx, or sites like Clicker.com, which specializes in recommending professionally created videos online.

After Blinkx added better analysis of the visual content of videos to its engine early this year, for example, the average number of clips watched by users jumped to nearly five, from 2.5, said Suranga Chandratillake, founder and chief executive of Blinkx.

For YouTube, part of the challenge is in handling people’s searches. In November, Americans typed some 3.8 billion search queries on YouTube, more than on any search engine other than Google, according to comScore, a market researcher. While Google queries tend to be quite specific, users often come to YouTube with requests as vague as “funny videos.”

But perhaps more important, YouTube must finesse what technicians call discovery. That’s the art of suggesting videos that users may want to watch based on what they have watched before, or on what others with similar tastes have enjoyed. The effort requires mastering data-mining techniques similar to those used by the likes of Netflix and Amazon to make movie or book recommendations.

“I don’t think the YouTube problem is different from the Netflix problem or the Amazon problem,” said Christopher T. Volinsky, executive director of statistics research at AT&T Labs Research. Mr. Volinsky recently helped lead a team that won a $1 million prize established by Netflix to improve that site’s recommendation engine by 10 percent.

That it took his team of top computer scientists three years to make a modest improvement to Netflix, which has some 70,000 titles, illustrates the complexity of the task, Mr. Volinsky said.

YouTube’s work in these areas is largely hidden from users and involves dozens of tweaks, small and large, that Mr. Walk’s team makes every month. Recently, for instance, the group began tackling what it calls topic exhaustion. No matter how much users may like to watch, say, Shaquille O’Neal highlights, they will inevitably reach a point when they will have had enough.

So while YouTube used to suggest more of the same topic to users who watched a particular video, it has gently begun to nudge them toward related topics. The Shaquille O’Neal video may prompt suggestions for Kobe Bryant highlights, N.B.A. clips or even topics further afield, like sports stars who appear in films.

“If we guess wrong, you could leave us sooner,” said Jamie Davidson, a 25-year-old associate product manager on Mr. Walk’s team. “But if we guess correctly, we may get you to watch another 10 videos. This is very hard.”

The techniques involve creating vast graphs, which Mr. Davidson calls conceptual maps, of related concepts like Shaquille O’Neal, the N.B.A. and Kobe Bryant, on which the proximity of two nodes indicates the closeness of two topics. The YouTube recommendation engine uses these maps to find new subject areas that might interest a user.

Over time, YouTube says it plans to rely more heavily on personalization and ties between users to refine recommendations.

Mr. Walk’s team meets weekly to discuss tweaks to YouTube’s software. During a recent meeting, a small group of engineers and user interface designers were brainstorming what might be the next big step in the site’s evolution: pages that would immediately begin playing a stream of clips tailored for a user, instead of offering lists of suggested videos. The idea is to push more videos at users in the hope of allowing them to abandon the keyboard and increasingly experience YouTube from the couch.

“On YouTube, every 45 seconds, you are stuck at a decision point,” Mr. Davidson said. “Any time there is a decision point, people may leave. We don’t want to take out the interactivity, but the default user experience should be a lot easier.”

Palash Nandy, an engineer on the team, suggests a couple of ideas. How about putting an “I’m feeling bored” button next to the search box, echoing Google’s famous “I’m feeling lucky” button? Or why not give users a slider to select the amount of time they want to be entertained, and let YouTube’s software assemble a playlist accordingly?

None of this is likely to appear on the YouTube home page soon. But the team is already working on new ways to let users collectively create lists of videos that share a topic, like cats playing keyboards (youtube.com/bestofkeyboardcat). The ideas may show up first in the TestTube, YouTube’s site for experimental features.

Gauging YouTube’s success so far is difficult. YouTube will not release detailed traffic statistics. But it says improvements in its search and discovery software have helped increase the average time that users spend on the site by 50 percent in the last year.

Data from comScore appears to back that up. The average YouTube user watched 83 clips in October, compared with 53 a year earlier, though it is difficult to know how much of that growth was the result of improvements in YouTube’s algorithms.

Either way, Mr. Walk’s team plans to speed up the pace of innovation to help YouTube users search less and watch more.

“We are the second-largest search engine, and yet search is not even the right paradigm for discovering video,” Mr. Davidson said. “We are trying to move beyond it.”