Product Management: What I Learned About User Data, Research and Actionable Steps

research

As I research product management disciplines and best practices to become a better PM entrepreneur, I came across content from Laura Klein that got me hooked. This post is lessons learned form her presentation on Quantitative vs Qualitative Research.

User Data, Research and The Actionable Steps:

If you use data well you could build better products, but if you use data badly you can screw products.

Quantitative data doesn’t replace design or listening to users, it only gives us info on what we have built.

The data can help inform design – understand what’s working and what’s not. Inform research – give us feedback on product decisions.
Metrics can help understand whether our decisions helped or hurt user behavior. It tells us what we should be investigating…

Quantitative Data: WHAT (we should be investigating)
Qualitative Data: WHY (it happens)

For example, a checkout funnel (It’s more of a sieve):
For software as a service or e-commerce products.
Steps can be:
100% is incoming (100 buyers)
Create account -20% (-20 buyers)
Select plan -10% (-8 buyers)
Payment Info -50% (-36 buyers)
Address -33% (-12 buyers)
Confirm -25% (-6 buyers)
Thanks & Up Sell = 18 buyers

Users had an intent to buy something: 100 came in, entered the check-out funnel (we spent effort on acquiring users probably through google ads/FB/referrals), then they started falling out and only 18 purchased.

What happened?
Go back to the funnel and investigate the steps:
 the step where we’re loosing most people is Payment Info -50% (-36 buyers). So we start asking ourselves what are the reasons? Might be payment options are not clear or they don’t feel secure, or something else.

Don’t come up with reasons! if you really want to understand inflection point: you need to do two things:
1. Find it. Quantitative research: funnel numbers helped us find the inflection point.
2. Need to understand it: why it’s happening? Qualitative research: interview users and do A/B testing.

Using Quan and Qual Together
Step 1: Identify the biggest problem!
Use funnel metrics!
(Quan): In order to identify the biggest problem, we look at funnel metrics and try to identify the largest hole in the usage flow.

Step 2: Understand why!
Observational usability testing!
(Qual): watch people use your product! Look for when they get stuck. You would pay particular attention to the people who get stuck on the payment page.
Based on your observation of between 4-6 people (that’s the correct number of people to support this sort of observation user study).
You would come up with reasons. Some reasons that surfaced up:
– Didn’t have payment info handy
– Confused by specific question/wording
– Looked for promo code (saw the promo code box and went looked for a code)

Step 3: Propose Solution!
Create solution hypotheses!
After you’ve observed the problems, you need to figure out: solving which of these problems is actually going to fix your Payment Info -50% (-36 buyers) problem?

When we want to test if a change was successful, we need to know what success looks like, and equally we need to know what failure looks like. Create a list the possible negative consequences of the change as well as the positive ones, so that we can predict what’s going to happen as we implements that solution.

We pick one of the reasons to test with, the promo code:
Trigger: People looked for a promo code (they see the box)
Solution: Easiest solution is remove promo code entirely
-Solution hypothesis is: If we remove the box there will be an increase in conversion.
-What might go wrong? A decrease in acquisition or revenue. (the reason in the promo code might have been to convert price sensitive people, or to promote sales) That’s why when we change something we want to keep an eye on other metrics that might be effected by our change.

Step 4: Learn & Iterate
Learn:
 How would we know if we were right?
A/B test of the checkout flow with and without the promo code! 
We are going to do Quant test if just removing the promo code fixes things without hurting.
So we create two tests: one with promo code and one without and test the funnel again.
Quant test came out positive: we’re losing less people and haven’t seem much decrease in acquisition.

Now we need to run Qual test to check if what’s happening is for the reason we thought.

We keep testing, iterating and learning until we are happy with the results and until metrics look the way we want them to look!

Then when we’re done we can go back to the steps and identify a new problem looking at our quantitative data again!

In brief, the steps are:
Step 1: Identify the biggest problem 
Use funnel metrics
Step 2: Understand WHY
Observational usability
Step 3: Propose solutions
Create solution hypotheses
Step 4: Learn and Iterate

A different way to find out: 
Step 1: Identify the biggest problem
(Qual) Interview current users
Call customers to discuss what issues they are having using your product, or what alternatives they are currently using.
Another one to generate hypothesis is call former users to find out why they are former, what caused them to leave. You’ll learn a tremendous amount on how to improve and what caused people to leave.

Step 2: Understand what
Since we started with the why, now we have to understand what?
(Quant) Study relevant metrics- the idea here is to take what we learned in the qual and try to predict how it might effect the metrics.
Why would we bother to predict what metrics will change? You want to know if I fix that problem for the user, what metric do I expect to improve?
For example: If we think that fixing a problem is going to fix retention that’s great, but if we only have 3 users maybe retention is secondary and user acquisition/conversion funnel is first.

To identify which part to examine, you can use this framework by Dave McClure AARRR: Acquisition, Activation, Retention, Revenue, Referral
If you fix something you think is probably going to effect revenue, then check revenue metrics.
Or if it’s what we saw amongst power users, it’s very likely going to effect retention.
Or if it’s a problem we saw with brand new users as part of the on boarding process, well that’s going to effect engagement.
Or if you want to implement comments for example- don’t focus on how many people comment, but if they comment do they buy more stuff? Think in terms of if I count/measure it should be towards a specific goal: purchase increase, time spent in the app increase (engagement & retention). What happens that matters and moves the needle?

Step 3: Propose solutions
Create solution hypotheses – what you think will go right and what you think will go wrong.

Step 4: Learn and iterate.

The process works not only for products but for many other functions like services, processes, sales process- identify the steps where people fall in your sales pipeline and improve it.

For sales pipeline:
Step 1: Identify the biggest problem
Look at your sales pipeline
Step 2: Understand WHY
Interview people who said no
Step 3: Propose solutions
Create solution hypotheses
Step 4: Learn and Iterate

General practice:
When you’re trying to identify features: people are not good about predicting the future, but they are good at talking about stories from the past and the problems they have. Make sure to optimize your questions to that.

One thing to ask is how they have attempted to solve that problem in the past? How are they solving problems now? What product have they used to solve it?

Talk to enough people to find patterns in problems… that’s what you should design a solution for.

Statistics:

  • If you’re doing an A/B test (a split test): If you get to 300 people actually converted in each branch, then you’re safe with statistical significants.
  • For usability test the number is 5 people. If you’re not starting to see really strong patterns after 5, fix your recruiting, fix your personas, get new people and do it again.
  • You want to predict from the next set of 5 what you’re going to hear and should keep doing testing with sets of 4-5 until the problem or behave becomes predictable in a certain way.
  • Write it down, what’s your prediction? What do you think is going to happen? Confirm it with the data!
  • Focus groups: don’t do 12 people in one room, do 12 people interviewed separately. Observational testing will give you much better information. Focus groups are also incredibly hard to run well. One-on-one interviews win!
  • Create screener survey – it’s method of recruiting the current people for your surveys. (Screener questions: Keeping out the riffraff).
    For example: I want a left-handed dentist who lives in Boise: You write a survey to identify your testers:
    Q1: What do you do for a living? Dentist
    Q2: With which hand do you write? Left.
    Q3: Where do you live? Boise

Working on a product that doesn’t exist yet:
Understand the user journey: talk to x number of users to understand what the problem is.
Understand some key concepts, can use quantitative to make sure that the educated guess is the right one.
In user centric design methodology: do the needs finding, and then do design, then prototyping.
What’s missing (and the lean startup was designed to address) is you have to do some sort of an evaluation test: you validated the problem, now you need to validate your solution idea before you build something.
You need to figure out the smallest possible thing that you can build to validated if your solution is in the right direction (you can run concierge test, you can run wizard of oz test).
You have to validate that the solution direction you’re going is good. It’s the MVP concept: the small great thing that you can build that your users can use as a solution.

Qualitative-vs-Quantitative

 

What the IDF Bootcamp Taught Me About Becoming an Entrepreneur

The Israel Defense Forces (IDF) was founded shortly after the State of Israel was established in 1948. It ranks among the most battle-tested and highly-trained armed forces in the world. The IDF’s security objectives are to defend the existence, territorial integrity and sovereignty of the State of Israel; deter all of Israel’s enemies; and, curb all forms of terrorism which threaten daily life.

Serving in the IDF is mandatory for all Israeli citizens both men and women, you join the military when you turn 18. Israel is the only country in the world with a mandatory military service requirement for women. Women serve two years, man serve three years.

At the age of 18, many teens around the world finish high school and transition to their joyful life in college.

Serving the military at that young age forces you to obtain discipline, take responsibility and learn the system. In your early days (at bootcamp) you spend time learning Krav Maga and training at the shooting range, instead of studying for exams and drinking at the neighborhood bar.

Weapon training image, that’s how it looks like at a shooting rang.

 

Instead of having a college graduation ceremony, you have a weapon swearing-in ceremony.

Me at the Weapon Swearing-In Ceremony. Holding an M-16

This also reminds me of the awful quality of mobile pics at the time. iPhones weren’t around yet.

 

Though it has always been outnumbered by its enemies, the IDF maintains a qualitative advantage by deploying advanced weapons systems, many of which are developed and manufactured in Israel for its specific needs. The IDF’s main resource, however, is the high caliber of its soldiers.

At the end of the day, to run a smooth organization, the IDF relies on the discipline of its soldiers and a working system.

“Nothing can be more hurtful to the service, than the neglect of discipline; for that discipline, more than numbers, gives one army the superiority over another.” George Washington, general orders, Jul. 6, 1777

Entrepreneurship

Becoming an entrepreneur means creating something pretty much out of nothing. The business begins and ends with you. While that can sound pretty awesome, it also means you have little guidance. You have to create the map, the directions to get there and the destination. It’s easy for doubt to creep in.

In order to stay motivated and find order in the chaos – you have to develop discipline and put systems in place, so you can run your startup more efficiently and reach milestones faster.

Examples: IDF Bootcamp

Discipline and Systems

I remember my first day in the bootcamp of the IDF, where you stand with other soldiers in a row with your hands behind your back.

The commander yells at everyone to put their hands in a diamond shape behind their back above the belt line. You just put your hand behind your back with one hand on top of the other, “who cares how my hands are set, I did what they asked, put my hands behind my back”.

Well, if your hands are not set in a perfect diamond shape above the belt line, the commander starts screaming at you to put them in a perfect diamond shape and doesn’t leave you alone until you do.

At that moment, you truly learn how fucked you are for the next few months, and what attention to details is all about ☺.

Establishing good systems requires attention to details and discipline. It’s not easy at first, but then as you get used to it, it becomes second nature and an unbeatable competitive advantage.

Schedule

When you are in the bootcamp, the schedule is very strict. I remember we had a schedule for everything: morning wake up, breakfast, morning training, learning, gun cleaning, etc. If you are even one minute late to any of the sessions, you and your entire troop gets the blame and is yelled at by the commender. You learn to be accountable and plan your time — to be on time.

As an entrepreneur, since you are the one who sets your tasks and schedule, it’s very easy to get distracted and surrender to different temptations during the day. It doesn’t help you accomplish your tasks, and causes procrastination.

You have to train yourself to be your own commander, so each day you can complete the tasks you set for yourself.

  • Create a list each day for the tasks you want to accomplish and prioritize them. Look at your list during the day to mark off the tasks you already completed, and the ones that you still need to work on. Make sure to create a new list at the end of each day, to get yourself ready for the next day.
  • Establish hours for work and for rest. Since as an entrepreneur you work all the time and don’t have defined 9am-5pm, it’s important to create a daily time frame and hold yourself accountable to it.
  • Write down your ideas. As entrepreneurs we have awesome ideas come to us when we least expect them (driving, showering, in the restroom). Try to record them at the first opportunity you get. It’s very easy to get distracted when something else comes up and that cool idea that you just had a minute ago suddenly slipped your mind.

When you feel lazy at times and want to cut yourself some slack, remember this quote: “Discipline is the bridge between goals and accomplishments.” — Jim Rohn

Product

When you receive a weapon in the bootcamp, you have to ‘own’ it. Be responsible for it 24/7. Which means you are required to always have it in your sight. When you go to sleep you hide it under your bed, when you go to shower you take it with you and put it just outside the shower. You develop a habit of thinking about it and keeping track of it, it becomes your “precious.” It’s a lesson in responsibility and it trains you to always be prepared.

As an entrepreneur, your product is “precious”. When you are building it, you have a responsibility for creating the best possible product you can and be prepared to adjust to feedback. It’s important to know that if it’s a consumer facing product, you want your users to develop a habit of using it.

Which means that you should look closely at habit-forming mechanisms. Personally I like to follow Nir Eyal’s blog Nir & Far.

By Nir’s system, a product needs to have an experience that is designed to connect a user’s problem to the product’s solution with enough frequency to form a habit. The system has these four basic steps:

A trigger, an action, a reward and an investment.

If you train your team to pay attention to those mechanisms, it will become easier to recognize what works and adjust functionality to increase engagement and retention.

Customer Service and Feedback Loop

As your customers use your product (say it’s an app), you want to maintain a system to stay top-of-mind with them.

Sending emails on a regular basis with clear, concise communication is a great way to keep your users in the loop and make them feel updated with the product’s and the company’s latest news. It develops trust.

MailChimp and Mandrill are great for setting up those regular email campaigns.

Asking for feedback is also great for customer service — you let your users know you care about what they have to say and at the same time you learn about their needs to better enhance your product.

The mechanism for that is pretty straight-forward; if it’s a mobile app, have a Send Feedback screen. Make sure the emails are received by you or someone on your team who can respond quickly. If you receive an issue from a user, make sure to solve it fast. If it’s a suggestion, make sure to reply with a thank you. If you can send over some sort of a perk like a t-shirt or any other swag, it will hold some serious wow factor for that user, and it will go a long way to increase your word of mouth.

The magic is in the details. Remember, if you have an opportunity for a conversation with your users, you have a chance to delight them! If you do, you’ll spread good karma and strengthen the awesomeness of your brand.

Be disciplined. Put systems in place. Be the leading soldier in your own [startup] army.

Finding Diamonds in the Rough

This year’s independence day weekend I helped my relatives clean the garage. While sorting through piles of old books I stumbled upon a small black covered book named “God’s Little Instruction Book for Graduates”. I am not a religious person and was about to gracefully put the book in to the Goodwill donations pile. But then, my curiosity took over and I opened it up to quickly see what it was about. What I discovered was a pleasant surprise! Each page had an inspirational quote to a life lesson… As someone who loves inspirational quotes, it was like finding diamonds in the rough. I ended up spending the entire day reading the quotes and taking pictures of the best ones. Here they are. Enjoy being inspired 😉

IMG_1060

IMG_1064

IMG_1065

IMG_1068

IMG_1071

IMG_1074

IMG_1076

IMG_1081

IMG_1082

IMG_1083

helenaquote12

helenaquote13

 

helenaquote14

helenaquote15

IMG_1100

IMG_1101

IMG_1102

IMG_1103

IMG_1105

IMG_1131

IMG_1136

IMG_1137

IMG_1139

IMG_1140

IMG_1143

 

 

Minime in 3D

3dteam

us

So, there is this magical co-working office space called Hacker Dojo. It’s the best kept ‘secret’ in the Silicon Valley. It’s in Mountain View (close to Castro St.), it has all the amanitas a startup needs and the rent is super affordable (very unusual for the Mountain View/Palo Alto area).

In this magical place there is a magical room called the pink room (or the Maker Space ;). Where magical things happen! (3D printing, laser cutting, robot making).

I’ve been fascinated with 3D printing for a while now, and when the opportunity came to participate in a 3D project, I didn’t think twice.

The project was to learn about the 3D printing process and test a 3D printing machine from Formlabs.

First, we were given instructions on how to use it. Then came the scanning and printing. We were a group of 3 people from team VentureOut; Helena, Adonis, Sean.

Here is the story told by pictures:

Welcome to the pink room

FullSizeRender-3

Let’s show you around

FullSizeRender-5FullSizeRenderFullSizeRender-6FullSizeRender-7FullSizeRender-4

Now it’s time to get scanned

IMG_5747

You can even increase or decrease the size of a character 🙂

IMG_5748

Then you put the liquid material (resin) in the printer

FullSizeRender-1

FullSizeRender-2

3D printer in action! (kinda reminds a scene from the Terminator :))

IMG_0503

IMG_0504

IMG_0505

Anddddd, it’s alive!

printed

3d need to cut

Now we gotta clean it up and cut the courts (the spikes support)

3d cutting

Next is to solidify the characters under a UV light

us

 

 

 

 

 

 

 

 

And that’s it! We have our own 3D minimes! 🙂

You Are Already Naked. There Is No Reason Not to Follow Your Heart.

What-Matters

“Remembering that I’ll be dead soon is the most important tool I’ve ever encountered to help me make the big choices in life.

Almost everything–all external expectations, all pride, all fear of embarrassment or failure–these things just fall away in the face of death, leaving only what is truly important.

Remembering that you are going to die is the best way I know to avoid the trap of thinking you have something to lose. You are already naked. There is no reason not to follow your heart.”  –Steve Jobs.

When I was twelve years old my father died, it was sudden, unexpected, and very sad. I remember not fully understanding, hoping he would come back and that I would wake up from the nightmare. It never happened of course, and as an adult I came to the realization that he’s just gone and will never come back. Whatever memories he left with me is what I have to cherish.

This makes me deeply understand the meaning behind Job’s words: Almost everything–all external expectations, all pride, all fear of embarrassment or failure–these things just fall away in the face of death, leaving only what is truly important.

Seeing such a close person die, makes you feel naked, not having anything to lose by following your heart, doing what you think is right, and being genuine with yourself and others.

Fear is in our nature, but knowing when to take calculated risk and ignoring ‘dumb’ fear is something each entrepreneur should be able to do. After we are gone, all that is left is the memory of us. So make this memory count for you and others. Create something meaningful, be kind to friends, people who you’ve just met, and people who you don’t need anything from. Smile and make eye contact. It will give people a great feeling, and it might just make their day.

I started VentureOut because I wanted to see this product in the world. I barely knew what startups were, but I felt like I just had to build this thing because that’s where my heart is. There are so many things that I’ve learned since starting out. I think that there is a reason why first-time entrepreneurs end up building epic companies, i.e. Jobs, Zuck, Gates, Brin & Page etc. Because they don’t spend enough time in the ‘industry’ to sink on fear or doubt. They just have something awesome in mind that they want to see in the real world, and they go for it full speed! By the time they realize the mechanics, it’s too late, their intuition helped them figure out where they’re headed and how to proceed despite all challenges.

It’s this blind drive and realization of ‘nothing to lose’ attitude that ignites entrepreneurship. Execution creates new streams of drive and motivation, recharging your batteries.

Then, once you’ve built something, make sure you take it to it’s maximum potential–make it the best it possibly can be! It will take time, yes, and a lot of guts. But we live only once, this is your opportunity to reach your maximum potential and make a dent in the world.

Now go out there! Do what you’re passionate about.