Lecture 16 - How to Run a User Interview (Emmett Shear)

Emmett Shear teaches how to run user interviews that uncover real problems and shape product direction, based on lessons from Kiko Calendar's failure and Twitch's growth.

Lessons from Kiko and Justin.tv

• Kiko Calendar failed because founders never used calendars or talked to users (0:00-9:18).
• Justin.tv worked initially because founders were the users but could not scale beyond that narrow case.
• Twitch grew by interviewing broadcasters to learn their needs for live game streaming.

Choosing Who to Interview

• Who you talk to matters as much as the questions; interview non-obvious stakeholders such as college IT departments or parents for a note-taking app (9:17-18:35).
• Talk to non-users to expand market size; they represent the majority of potential customers.

Running the Interview

• Stay away from feature requests; focus on current behaviors and problems to avoid the horseless carriage effect.
• Drop into detective mode the moment users say something unexpected (27:52-37:10).
• Record every interview for accuracy and to win internal buy-in.

Validating Ideas

• Never ask users if a feature is good; build an MVP or run the money test instead (18:34-27:53).
• A product must be quantum better than existing solutions to drive switching.

International Markets and Channels

• English-fluent users abroad are rarely representative; in-person or Skype interviews beat email (37:09-46:27).
• Reach users via on-site messaging and events; compensate only when necessary.

00:00:00Major Topic: Introduction and Emmett Shear's Background

Speaker: Sam Altman introduces today's guest, Emmett Shear (CEO of Justin.tv and Twitch). Emmett shares his startup journey, including early missteps and learnings about talking to users versus relying on data alone. He frames the session as a practical guide to doing great user interviews, why they matter, and how they can redefine product goals.

• He co-founded Justin.tv with Justin Kan and later pivoted toward gaming with Twitch after realizing he didn’t know enough about broadcasting gaming content. This pivot hinges on speaking with users to discover real use cases rather than assuming them from internal perspective.
• Early experiences: Kiko Calendar taught him that building a great product isn’t enough if you don’t talk to actual users. He notes that his team did well at building, but not at understanding real users’ needs.
• The core lesson: who you talk to is as important as what you ask; the target user should drive product decisions, and you should source them carefully rather than just relying on convenient or familiar groups.

"If you want to run a startup, you don’t just build stuff in a vacuum—you have to talk to the people who actually use it." (00:00:00)

00:09:17Major Topic: The Importance of Who You Talk To

Speaker: Emmett Shear explains that the choice of interview subjects can determine the entire direction of product development. In Twitch’s case, focusing on broadcasters (the content creators) rather than viewers yielded deeper insights into how to build a platform that broadcasters would love, which in turn helped attract and retain viewers.

• When assessing a market, the broadcaster’s needs proved more informative than viewer-only feedback. Broadcasters shape how content is created and distributed, so their pain points and workflows are the levers for platform-level changes.
• There isn’t a universal recipe for identifying your target user; it requires judgment and hard thinking about who will drive the product to scale.
• Emmett invites the audience to participate in a hands-on exercise: think about who you would ask about a new note-taking app for lectures, and write down five types of people to interview (e.g., college students by major, IT administrators, parents, etc.). The goal is to practice sourcing before digging into questions.

"Who you talk to is as important as what questions you ask and what you pull away from it." (00:09:17)

00:14:50Major Topic: Interactive Exercise and Audience Participation

Speaker: Emmett Shear leads an interactive session where volunteers from the audience discuss who to interview for a note-taking app focused on lectures. He emphasizes the practical value of identifying multiple stakeholder groups early and not prematurely narrowing the focus to a single user type.

• A volunteer suggests talking to college students across disciplines to understand note-taking needs and whether different majors affect note-taking styles (e.g., text-heavy notes vs. diagrams). Emmett notes that college IT staff may be key decision-makers for school-wide tools, not just students.
• He cautions that college students often don’t have a lot of discretionary spending power, so if you’re targeting students, you should also identify who else might buy or authorize adoption (e.g., IT departments, parents, or school administration).
• The broader point: initial interviews should illuminate the space and the ecosystem around the user, not just the narrow feature list.

"If you build for yourself, you might get lucky, but that’s a side project more often than not. You need to talk to a broad set of users to find the real use case." (00:14:50)

00:18:34Major Topic: From Interviews to Feature Ideas — Don’t Lead with Features

Speaker: Emmett Shear pivots from the first exercise to a practical approach for framing interviews to avoid feature bias. The goal is to understand problems and contexts rather than eliciting feature requests.

• The session transitions from talking to users to shaping product direction through the interview data. Emmett emphasizes that asking users to design features (e.g., a specific note-taking capability) can bias the conversation toward incremental improvements rather than identifying real problems worth solving.
• He demonstrates how to extract insights that inform strategic decisions (e.g., Broadcasters were crucial; this drove Twitch’s pivot and long-term roadmap).
• A key point: if you ask someone what feature they want, you’ll often hear a real, concrete request. But that may mislead you about the underlying problem or whether a feature will actually move users to adopt your product. Instead, probe for underlying needs, workflows, and pain points.

"The main thing you’re trying to do when you run these early interviews is not necessarily get questions about features. You want to understand what the user actually does and what blocks them from doing more." (00:18:34)

00:27:52Major Topic: The Three Groups of Feedback and How to Listen

Speaker: Emmett Shear shares the broader feedback framework used at Twitch, highlighting the importance of interviewing multiple groups: existing broadcasters, competitors’ broadcasters, and non-users. This triad helps identify what truly matters and what barriers exist to expanding the market.

• He explains that feedback from current users and competitors often focuses on near-term improvements, while non-users reveal fundamental barriers to adoption (e.g., hardware constraints, time commitments, and strategic priorities).
• They present representative excerpts of feedback (condensed from a large set of notes) to illustrate how problems manifest in practice, such as stability, monetization, and global accessibility.
• He underscores that listening to non-users is critical for market expansion because these are the people who determine whether the addressable market can grow and whether your product can reach new categories of users.
• The Twitch team ended up prioritizing issues that users themselves did not explicitly request but which addressed core goals (money, stability, global access). The validation comes from building these capabilities and then looping back to users to show how their feedback translated into features.

"The feedback from non-users is the most important; they tell you what blocks adoption and what would make them try or switch to your product." (00:27:52)

00:40:35Major Topic: Validation Tactics — Quick Experiments and the Money Test

Speaker: Emmett Shear discusses practical validation methods for early-stage ideas, including quick hacks and the money test as a strong signal of real demand.

• For fast validation, don’t try to build a full product version; instead, realize you can test a concept with lightweight refinements (e.g., a browser extension that adds a small note-taking enhancement to an existing workflow) to see if it gains traction.
• If possible, use money as the ultimate signal: being able to secure a commitment or payment demonstrates strong interest beyond a verbal yes. For student-focused software, where price sensitivity is high, the money test is especially informative, though it can be challenging to implement given the context (e.g., school procurement processes).
• The broader lesson: feature-level quotes from interviews may overstate interest because users imagine a smoother experience; the best validation reveals whether users will pay or adopt the change when faced with real tradeoffs.

"The money test is amazing. It really does clarify whether or not they’re really excited about it or not." (00:27:52)

00:44:20Major Topic: Pelting Through Feedback — Structure of Twitch’s Post-Interview Loop

Speaker: Emmett Shear describes how Twitch used interview feedback to build and launch features that actually mattered to users, rather than those that simply satisfied individual requests.

• They used condensed feedback to identify recurring themes across different broadcaster groups and to map these themes to strategic priorities (e.g., monetization via subscriptions, broader audience reach, and better streaming stability).
• The real win came from closing the loop: after implementing features (e.g., subscription programs), they re-engaged the same interview targets to validate whether the changes addressed their core concerns, thereby reinforcing the product’s direction.
• This iterative, feedback-driven approach contrasted with Justin.tv’s earlier approach, where data alone guided decisions without substantially addressing user-level pain points.

"If you tell someone about a feature and they say yes, you can be tempted to think it’s a good idea. But only by building it and showing it back to them can you confirm its value." (00:44:20)

00:50:00Major Topic: Interview Best Practices — Recording, Channels, and People You Talk To

Speaker: Emmett Shear summarizes practical practices to maximize the value of interviews, including how to conduct, record, and share them within the company.

• Recording interviews is highly effective: it avoids the disruption of taking notes mid-conversation and provides a material artifact to share with the rest of the team to illustrate user realities and needs.
• Skype/in-person interviews are preferred over email interviews for their interactive dynamics and richer back-and-forth, enabling the interviewer to probe deeper with follow-up questions.
• Outreach channels include on-site messaging, direct emails, and in-person events. Compensation is not always necessary to get interviews, as many users are glad to share their perspectives when approached respectfully.
• He notes that the early focus should be on interviewing diverse groups (broadcasters of different sizes, non-users from various contexts) to avoid bias from interviewing only easy-to-reach or already-engaged users.

"Recording interviews is like magic. It stops you from taking notes in the middle and makes it easy to share with the rest of the company." (00:50:00)

00:55:00Major Topic: The Q&A — Common Mistakes and Final Guidance

Speaker: Emmett Shear closes the lecture with a Q&A segment, highlighting common mistakes startups make in user interviews and how to avoid them.

• Common mistakes include showing the product too early, asking directly about paying for or using a pet feature, and talking to whoever is most accessible rather than identifying the right users to learn from.
• The value of conducting real, unscripted conversations with users over email or canned surveys is stressed; the best insights emerge from open-ended dialogue that reveals users’ real context and constraints.
• The session concludes with practical tips: seek diverse interview subjects, and when possible, integrate interviews into the company culture so the team can see real-user dynamics rather than rely on internal assumptions.

"> The most common mistakes are showing people your product. Don't show them your product. It's about learning what's already in their heads." (00:37:09)

00:37:09Major Topic: Final Reflections and Takeaways

Speaker: Emmett Shear emphasizes the broader impact of user interviews on startup success. He notes that many teams underestimate how different and surprising user feedback can be, especially when it challenges beloved ideas.

• A recurring theme is that user interviews won’t always validate your pet feature; often, they reveal different priorities that can steer development in a more meaningful direction.
• The process is iterative: begin with broad exploration, confirm by building tiny, testable experiments, and then scale the initiatives that prove themselves through real user interest and, ideally, revenue signals.
• The session ends with a reminder that even though data analytics are valuable, they won’t replace direct conversations with users for early-stage product discovery.

"The pool of people who care about your product changes over time; keep interviewing to stay aligned with the evolving market." (00:57:00)

00:58:00Resources and Next Steps

• Slides and readings at startupclass.samaltman.com/courses/lec16/
• Transcript and annotated materials: http://tech.genius.com/Emmett-shear-lecture-16-how-to-run-a-user-interview-annotated
• Creative Commons license: http://creativecommons.org/licenses/by-nc-nd/2.5/

00:58:30What to Do Next

• Practice the sourcing exercise: write down five user groups to interview for your idea.
• Plan a quick interview with a representative from one group to validate the core problem you’re addressing before building features.
• Consider recording interviews for internal buy-in and broader learning.

00:59:00Resources Mentioned (Summary)

• Kiko Calendar (Emmett and Justin Kan’s early startup)
• Justin.tv (biomedical origin of Twitch) and Twitch (acquired by Amazon)
• Evernote, Google Docs (tools discussed during interviews)
• Xbox, PlayStation 4 (platform integration mentioned as part of Twitch’s strategy)
• Subscriptions as a monetization model (Twitch’s later focus)

00:59:30Final Note

• Emmett emphasizes the importance of continuing to interview diverse user groups as the product evolves. The right questions, the right people, and a disciplined approach to testing ideas through quick, real-world experiments are central to building products users actually want.

00:59:45End of Lecture

Resources Mentioned

• Slides and readings: startupclass.samaltman.com/courses/lec16/
• Transcript: http://tech.genius.com/Emmett-shear-lecture-16-how-to-run-a-user-interview-annotated
• Creative Commons license: http://creativecommons.org/licenses/by-nc-nd/2.5/
• Companies/Projects: Justin.tv, Twitch, Kiko Calendar, Google Docs, Evernote, Xbox, PlayStation 4
• Related: See the Twitch product evolution and the emphasis on broadcasters as core users

"There isn’t a recipe. It comes down to think really hard and use your judgment to figure out who you’re really building this for." (00:18:34)

"The money test is amazing. It really does clarify whether or not they’re really excited about it or not." (00:27:52)

"Recording interviews is like magic." (00:50:00)

"The most common mistakes are showing people your product." (00:37:09)

"Hack something together" (00:18:34) — conveys the value of quick, practical validation when a full build is not feasible.

"If you want to sell software to students... the money test is amazing." (00:27:52)

"The audience’s needs change as the product evolves—keep interviewing." (00:57:00)

"Three groups of feedback: existing broadcasters, competitors’ broadcasters, non-users." (00:27:52)

"Don’t ask: would you pay for a subscription?; instead, seek real commitments or pay up-front signals when possible." (00:27:52)

"We built a subscription program because people told us they cared about making money, and we validated that with real features later." (00:44:20)

"Don’t rely solely on data; let qualitative conversations steer you toward meaningful product goals." (00:44:20)