The Ultimate Guide to Cracking Product Case Interviews for Data Scientists (Part 2)

This is Part 2 on product case interviews. If you haven’t read Part One yet, be sure to check it out!

In this post, we will be talking about the types of questions you can expect in a product case interview and tips for preparing and making a good impression in the interview.    

Types of Product Case Questions

There are 4 commonly asked categories of product case questions. Here we will go over frameworks for approaching each of them, but remember that frameworks should serve as a mental checklist. Make sure to include your own thoughts and never follow them blindly.

Regardless of the question it is a good idea to start with some clarifying questions and end with a summary of your approach.

Let’s move into the first question category, diagnosing a problem.

Diagnosing a Problem

These questions want you to identify the cause of a decreasing business metric. An example question would be “The creation of Facebook user groups has gone down by 20%. What is going on?”.

There are a lot of ways to approach a question like this but the most important thing is to show that you can be systematic. Here are 6 steps you can use, but note that not all questions require all the steps.

1. Clarify the definition of the metric.
2. Determine whether there was a temporal aspect of the change.
3. Check out if other products or features have the same change.
4. Segment users by demographics and behavioral features to see if you can isolate the change.
5. Decompose the metric for a more in-depth analysis.
6. Summarize your approach.

Measuring Success

Measuring success questions want you to measure the success or help of a product or feature. An example question is “Instagram is launching a new feature. How do you tell if it is doing well?”.

These questions are testing whether you can define success metrics. It’s best to provide no more than 3 metrics (2 success metrics and one guardrail metric).

You can look at Part One of this post for a summary on what makes a good metric, but there is one additional thing to keep in mind. A good metric fits the context. The metrics you pick need to make sense in the situation.

Launch or Not

For these questions, you will be asked how to test an idea or whether to launch a product/feature. A sample question is “How would you set up an experiment to understand a feature change in Instagram stories?”.

This question category requires in-depth knowledge of A/B testing.

Just like Measuring Success questions, first clarify the goal of the product and come up with metrics to measure success.

After that, you need to propose an experiment for inferring causal impact. Be sure to discuss things like:

• Definitions of control and treatment groups
• Randomization unit
• Experiment run-time
• Common pitfalls and potential fixes (novelty effect, multiple testing problem, etc.)
• Long-term monitoring

After discussing these things, you should also make a launch recommendation. An ideal scenario to make a launch would be:

• One or more success metrics has a statistically as well practically significant increase
• No change in guardrail metrics.

However, ideal results don’t happen often. The interviewer will want to know what you would recommend with conflicting results. It’s important to think about things like tradeoffs between short term and long term impacts and if possible try to tie changes to a single business metric when making your recommendation.

Improving a Product

These questions ask how to improve a product or shift a business metric in a positive direction. These questions are very open-ended and require advanced product knowledge. A sample question is “What would you change in Twitter app? How would you test if the proposed change is effective or not?”

There are five key steps to answering these questions:

Step 1: Clarify the goal and narrow down the scope of the improvement.

Step 2: Explain your approach to identify product opportunities and brainstorm a few ideas. Here are 3 commonly used methods:

• Reduce friction in the current user experience.
• Segment users based on their behaviors and identify key needs of distinct groups.
• Identify variables that are correlated with the target metric. Build a machine learning model to predict the target metric and propose a follow-up action that can move the metric.

Step 3: Prioritization. Given the ideas you proposed, which one would you prioritize and why?

Step 4: Define 1 or 2 success metrics to evaluate the success of the idea.

Step 5: Summarize the overall approach.

How to Prepare for a Business Case Interview

We recommend 4 actions items for preparing:

Action Item 1: Gather a large pool of sample questions and group them into different themes.

Action Item 2: ****Develop your own frameworks and answers. This can be done by reading, thinking, and communicating with fellow data scientists. In addition, we recommend a few general resources:

• Real interview questions and answers from Facebook, Lyft, and LinkedIn (playlist).
• StellarPeers
• Trustworthy Online Controlled Experiments
• Decode and Conquer

Action Item 3: Talk solutions out loud. It might help to have long and short answers for phone screens and onsites.

Action Item 4: Research the company and understand its product.

Tips to Ace the Product Case Interview

Finally, here are a few parting tips to help you ace the product case interview:

• Always clarify the question!
• Interact with the interviewer.
• Prevent the interviewer from losing focus.
• Do not follow any framework blindly.

Lastly, best of luck with your interviews! Remember that you got this! If you want to read a longer version of this post with more details and examples, you can find that here.