Measuring Startup Success with a Product/Market Fit Compass, ChIRP indicators, and Modified OKRs

Now that our crowdfunding is successfully completed (and soon legally finalized), I finally find myself with a bit more time to actually build our company. That’s nice. I’m sure our investors will appreciate it too. ;-)

Last week, our team got together in Amsterdam for our fourth face-to-face meeting. As a fully distributed team, we have no office and no daily commutes. Instead, we do all our work remotely, and we spend the company’s money on good technologies, travel tickets, cheap hotels, and samples of local food.

In Amsterdam, while consuming dozens of stroopwafels, we discussed several important topics, such as gamification, “Wow! Moments”, project tools, and our team objectives for the next few months. In this longer-than-average article, I want to offer my thinking on product/market fit, core actions, and OKRs. I hope this will be interesting for our investors who wish to understand how we intend to use their money. But also for other startups who find themselves in a similar situation of trying to measure progress toward a successful product.

Product/Market Fit

We have paying customers using our Android app. We have people who are enthusiastic about the ideas of crowdsourced business agility and value tracking. We have users asking us when we will make the iOS version and when we’re going translate to other languages. But we also have people canceling their monthly subscription because they don’t see enough value in our offering. Yet.

Ouch!

We are in the typical product/market fit phase that every startup needs to go through. Some experts say finding product/market fit is the toughest challenge for any startup. We have a product that works for some, but it’s not enough for most. We scored our first minor successes, but we’re still far from claiming victory. It’s like having a solar-powered bicycle in the rain, with a flat tire and no saddle. The adventurous-inclined are intrigued, but most people prefer to walk.

Some experts say finding product/market fit is the toughest challenge for any startup.

In the product/market fit phase, a startup keeps tweaking a new product’s features and design until it has the power to delight many customers in a promising market. Figuring out what exactly that product should do and what it should look like is extremely hard. It is most certainly a challenge for us! (If starting companies were easy, everyone would be doing it!) However, I have created a tool that may help us to find our way. I call it the Product/Market Fit Compass and I will explain it in a moment. But first… who is this for?

Purpose and Personas

Every startup needs to know their purpose and their primary persona. For whom are you developing your product, and why?

For the team members of our company Agility Scales, the purpose is clear. We want to help companies transform and adapt faster to changing environments. We call that shapeshifting, and we believe faster change is best achieved by growing an agile mindset among all employees. That’s why we call our product Mind Settlers. After all, people with agile/growth mindsets are much better at adopting new ideas than those with traditional/fixed mindsets.

We believe faster change is best achieved by growing an agile mindset among all employees.

But we cannot address all workers at the same time. As the expression goes, you cannot boil the ocean (although the USA seems to be trying). As a startup, you must narrow your focus to a smaller target audience that will act as your beachhead market. OK, you want to settle on a new continent. Great! But where does your first ship land? And how do you plan on charming the natives, rather than killing them?

At Agility Scales, we picked the experienced agile coach as our primary persona because we know many agile coaches; because people in this role love trying new ideas, and because their role allows them to teach organizations the use of tools. Experienced agile coaches also know that, as a collaborative tool, sticky notes have their limitations. Annoying, but true.

The Value Cycle

Most companies want their users to develop a habit of using their products. For example, it is my habit to find good restaurants and coffee places with Google Maps. My running exercises are all in Endomondo. And my daily complaints and smartass comments go on Twitter and Facebook. Similarly, my team hopes that growing agile mindsets with agile coaches will soon be done with our Mind Settlers app.

To create habitual usage patterns, it is essential that you understand how your product delivers value to your customer. You can define this as a sequence of steps. For simple products, the value cycle may consist of just one or two steps. But for most products, the sequence is probably longer. In our case, I defined four steps that together form the value cycle for agile coaches:

1. Find Ideas: discover interesting options for personal and team improvement;
2. Take Steps: implement an idea from these options by acting out suggestions;
3. Track Value: reflect on what happened and evaluate value created (or destroyed);
4. Share Learning: make available the things that were learned as ideas for others.

These four steps form a value cycle. The learning shared in the fourth step is input for new options in the first step. And to grow an agile mindset, every player should complete this cycle many times.

When I started drawing the four steps as a circle, I immediately realized that I had unintentionally (but perhaps subconsciously) recreated the good old Deming/Shewhart PDCA cycle. Funny how reinvention can make you feel both smart and dumb at the same time.

It makes sense to me that a product for agile coaches to nurture agile mindsets should follow the well-established PDCA cycle as a template. But keep in mind that the value cycle is a function of the persona and their goal. Pick a different stakeholder or another purpose (or both), and you will end up with another value cycle, possibly with a different number of steps. I will leave that exercise to you.

It is interesting to note that an earlier, primitive version of our product/market fit compass helped us realize that we were missing an essential part of our offering. For half a year, we had been focusing on finding ideas, taking steps, and sharing learning. But the left part of the picture (tracking value) was missing. Nobody realized that until we tried to draw product/market fit as a value cycle. Then it became apparent we needed to patch our product. That’s the power of visualization!

Product/Market Fit Compass

When I finished the four-step value cycle, I still found it a bit too high-level to be useful on a daily basis. So I decided to split each step into two perspectives, which resulted in an outer circle of eight themes, described below.

Note: If you’re not interested in our product and you’re just here to read about core actions and metrics, feel free to skip ahead to the next section: Core Actions.

Find Ideas: Pull (Visits)

By browsing through our app and our website, the user (often referred to as the “player”) generates views and visits. Similar to the number of songs played on Spotify and the number of episodes viewed on Netflix, the number of content items pulled from our platform is a reflection of the value our platform has for our players.

Find Ideas: Push (Alerts)

Instead of actively pulling content from our platform, the player can also passively wait for content to be pushed over one or more channels. With notifications, newsletters, and chatbots, we can try to offer the right content at the right moment. And by monitoring the player’s responses, we can learn which practices were appreciated and which ones were not.

Take Steps: Known (Dones)

We often say that Mind Settlers is not a learning platform; it’s a doing platform. People should stop planning and start performing. That’s why we consider it important to know which steps were actually taken by our players. The number of times practices were tried and repeated are a better indication of value than the number of times people merely read about them.

Take Steps: Unknown (Asks)

The counterpart to doing would be asking. With all the things that agile coaches do for themselves and with their teams, we will encounter many practices that should have been in our system but weren’t. Any signal in the form of, “We do this activity with our team but you don’t have it in your system” is valuable for us and the players themselves.

Track Value: Output (Marks)

Value tracking is the newest component that we are working on. We want to help our players reflect on the things they did throughout the day. Was their work valuable for them personally? Was it helpful for others? What was their mood when doing the job? What was the context and what was the size? We call these log items marks and the marks can be public or private.

Track Value: Input (Points)

The counterpart to output from our players would be input from their peers. How do team members feel about the work their colleagues did? What is the response from a person’s followers on the marks they left behind? Similar to likes on Facebook and retweets on Twitter, the points and comments received from peers can be quite valuable feedback.

Share Learning: Emergent (Pointers)

When someone has done the same kind of activity many times, it makes sense to allow that person to offer knowledge about how to perform this job. Other players may benefit from the sharing of that experience. This is the kind of learning that users share in an emergent way within our platform. They offer simple pointers that other people can follow easily.

Share Learning: Defined (Directions)

The opposite of simple pointers would be full-blown directions that participants offer to each other, in the form of guides, videos, articles, instructions, and checklists. Of course, agile coaches can already find plenty of directions on the internet. But we think that peer-to-peer help could be more effective when integrated as part of a full PDCA-cycle.

I want to emphasize that the subdivisions described above are a personal choice. They are my perspective on the value cycle that we try to implement with Mind Settlers, but alternative subdivisions can be equally (or even more) useful. And I might change and improve the wording in the picture over time. (In fact, I already have, twice.)

Core Actions

Yay, we have a product/market compass! Now we should discuss how to navigate with it.

Some experts suggest that it’s important to define the core actions in your product. And I agree. What is the exact thing that you want your chosen persona to do? What kind of user behavior would indicate that the product is successful? Or think the other way around: What drives you nuts because your users aren’t doing it?

What drives you nuts because your users aren’t doing it?

The experts suggest defining exactly one core action for your product. And I disagree. I think the core action depends on the step in the value cycle. In our Find Ideas step, the core action could be a visit to a specific piece of content. In our Track Value step, the core action could be submitting a new mark (with value, mood, context and activity size). In fact, I’m OK with defining exactly one core action per theme, in the outer circle of the diagram.

The chosen core actions should represent value for the player, but also benefit generated for the platform. Submitting marks is valuable for players because it helps them to reflect and build up a history of activities. It is also beneficial to us because the marks help us understand the player’s context so we can offer better content suggestions later on.

ChIRP Indicators

We have a persona and a purpose. We have a value cycle and a product/market fit compass. We have a small set of core actions. And we have a Pepsi Max Lemon with a bag of nachos. (Sorry, that’s just me. It’s a long article.) Now, all we need is a set of metrics that will help us to see progress.

I spent some time researching common metrics for product/market fit, and I observed that experienced founders typically suggest measuring product success indicators in cohorts. A cohort is a group of new users who start using a product in the same week or month. By comparing the numbers across cohorts, you can see which new product releases have had a positive (or negative) effect on which cohorts.

The commonly suggested indicators come in four flavors, which I call the ChIRP indicators:

Conversion

% of cohort with at least one core action performed within X days
This metric shows you how well your product is activating new users.

Interaction

% of cohort with at least X core actions performed per week or month
This metric shows you how intensively people are using your product.

Retention

% of cohort who still do at least one core action after X weeks or months
This metric shows how sticky your product is (high retention=low churn).

Promotion

% of cohort promoting the core action among friends within X days
This metric shows you how willing people are to invite more users.

You should keep track of the four ChIRP indicators for each of the core actions you have selected. And you will probably want to experiment with multiple values of X. For example, you could measure conversion within 1 day, 7 days, and 30 days. And you could measure the retention for each cohort per 30 days, 90 days, and 180 days. The best values for X will depend on the nature of the core action.

Note: The ChIRP indicators look similar the often-mentioned AAARRR metrics: Awareness, Acquisition, Activation, Revenue, Retention, Referral. But there are some differences. Awareness, Acquisition, and Revenue are good metrics for scaling, but you shouldn’t focus on them when you’re still looking for product/market fit. Don’t scale up the passenger list when the boat is still leaking! The only relevant AAARRR metrics in this phase are Activation (= Conversion), Retention and Referral (= Promotion). The ChIRP indicator that’s missing is Interaction, which is a crucial metric for product/market fit! Why bother scaling up a product that people use only half-heartedly?

Focus on Objectives

There’s just one more thing to do: focus!

My team has tried working with the practice of OKRs (objectives and key results) which is an excellent practice for achieving focus. But we struggled for many months:

• Literally, everyone has trouble defining outcome-based, measurable key results;
• Also, as a team we all sucked at setting reasonable targets for ourselves;
• The typical cadence of three months just doesn’t work for a startup.

The result was that we had endless discussions about metrics that turned out to be broken, easily gamed, unmeasurable, or outdated. And that was just a waste of our time. I developed a strong feeling that objectives and key results are great for scaleups that already know where to go, but that the practice of OKRs doesn’t fit the context of a startup looking around for product/market fit.

The practice of OKRs doesn’t fit the context of a startup looking around for product/market fit.

Our new approach is that we will collect data on all ChIRP indicators described above, but our focus will be on just two or three themes of the compass. In March, the objective for our twelve team members will be to improve our metrics for Marks, Points, and Directions. Nothing more. In April, we could change focus to other themes, depending on our progress. We’ll see when we get there.

With this new approach, we still honor the principles behinds OKRs, such as self-selected objectives, multiple metrics per objective, and self-evaluation. But we will skip the targets and iterations, and all indicators are outcome-based, measurable, endorsed by experts, and we don’t have to come up with new ones every few months. Only the dials on the compass may change from time to time. That means, less discussion, less waste, and a smaller chance of team members trying to strangle me.

The Next Phase

At Agility Scales, our team of eight people is expanding to twelve. We believe that we now have all the necessary skills to develop an agile adventure that our players are going to love. Soon. We are currently in the middle of reorganizing ourselves into crews, which are small feature teams based on the eight themes (with just three of them “active” this month).

Sadly, the outcome of our current phase cannot be planned. Putting a date on product/market fit is absurd. Nobody can set a timebox on expected outcomes of research and innovation. But trust me, we’ll be working as hard as we can to find product/market fit before we run out of money. :-)

Putting a date on product/market fit is absurd. Nobody can set a timebox on expected outcomes of research and innovation.

Assuming that we will be successful at some point, the next phase will involve preparations for scaling out and scaling up.

Scaling out is completing the platform so that it also covers iOS and web versions, and creating and integrating back-office systems, and preparing for the inevitable translation process that some people are already asking us about.

Scaling up is getting ready for larger numbers of players, optimizing for the other three AAARRR metrics, understanding the needs of large clients with enterprise licenses, introducing a free version to kickstart viral marketing, and maybe reintroducing iterations and targets for our objectives.

But all of that is too early now.

We’re super-happy and thankful that hundreds of people have decided to support our vision of crowdsourced business agility and the development of an agile adventure experience. Thanks to them we can double down on our efforts to make it happen. Our team has a compass now. I have faith that we’ll find our way.

Find out more about Mind Settlers. Click here!