Measuring network effects

Cliff notes from 16Z “16 ways to measure network effects” by Li Jin and D’arcy Coolican.

  • Acquisition
  • Competitors
  • Engagement
  • Marketplace,
  • Economics-related

Acquisition metrics

#1 – Organic vs paid: organic should increase over time as the network becomes more valuable, for direct-side network effects (NE). For marketplaces, organic should increase with the increase of both supplier and demand, because only then the NE occur. Paid can be a way to grow, but when critical mass is achieved organic should be the main driver (exceptions like markets exist). Which begs the question: when has critical mass been achieved? Is critical mass an absolute value or a relative? Ie, after achieving critical mass, a platform can continue to grow in users but loose critical mass? Like being outgrown by the market, or loosing relevance (icq, microsoft messenger). Is critical mass linked to platform activity? How is critical mass indexed to the sustainability of the business and to the platform unit economics? You should also pay attention to the type of users growing organically. Li Jin and D’arcy Coolican mention three types of users (pollutant, neutral and contributors). In a marketplace there’s the obvious distinction between supply and demand as well. The rating of the user (number of stars of a Uber driver) should also be considered, although that would be a lagging indicator.

#2 – Source of traffic: monitor the activity (traffic, transactions, …) on a platform by acquisition source. In a valuable network, the organic activity (ie, activity by organically acquired users) should grow. A sustainability analysis should consider how much of the activity is generated by paid users and the return calculated. Paid users may be a way to ignite and/or help achieving critical mass, but they can be a profitable in itself. Naturally, it is better to have acquisition loops rather than channels (see Andrew Chen), but positive return is positive return.

#3 – CAC evolution: the cost of acquiring a user should, theoretically, lower over time. However, a number of factors (increase price or competition in your main acquisition channel, for instance) may raise it. 

Competitors metrics

#4 – Prevalence of multi-tenanting: how many of your users also use similar services? More importantly, how active are they there? Reducing usage of your competitors’ service puts pressure on them (and on their pricing). It’s not easy to measure multi-tenant usage (e.g. polling, or brute-force search for profiles in other platforms), specially activity.

#5 – Switching & multi-homing costs: how easy versus how painful is it for a user to register in a new platform? Some services just require basic information; others require much a lot of knowledge about the user in order to provide a good experience. As a rule of thumb in marketing, you should always reduce friction to the minimum. For instance, credit card information is always a big dilemma: even though it make a great moat, it is also of high friction. Selecting the moment to ask that information is critical; the value you give in return is decisive. Another important thing to consider is the value you are providing at the beginning. Some products increase value over time (with more info/knowledge), others (job listing marketplaces) can provide a good amount of value from the start.

Engagement metrics

#6 – User retention cohorts: is the retention improving for newer cohorts? Theoretically, the network adds more value as more users enter, therefore retention should be higher for newer cohorts. However, some factors may affect this rationale: first users may have a stronger fit with the product; new competitors; pollutants. Make sure to account for these factors when comparing different cohorts.

#7 – Core action retention cohorts: figure out the key action in your platform. Then measure that activity across cohorts. This is a similar rationale to #6 but deeper down the funnel.

#8 – Dollar retention & paid user retention cohorts: similar to #6 & #7, but this time the metric is retention revenue

#9 – Retention by location / geography: figure out the geographical granularity level that composes your product. Ie, with Uber, it’s cities. Having more drivers in city A does not affect positively users in city B. After understanding the unit that defines your markets (city, state, country, …) you can start measuring and comparing retention in older markets versus newer ones.

 #10 – Power user curves: power user cohorts can provide a more nuanced understanding of engagement. The power curves are provided by L30 & L7 histograms for last 30 and last 7 days of use. These show how many of your active users are power users and how they are evolving over time. Power user curves give you more insight regarding the levels of engagement of different cohorts than MAU & DAU.

Marketplace metrics

#11 – Match rate: in a two-sided marketplace, your goal is to match supply with demand, ie, the provider of the service with the acquirer. Therefore, monitoring the number of pairings being made in your marketplace is a good NE metric. For instance, for a jobs platform, it’s not enough to monitor when a new job is posted, you need to monitor when that opening is filled by an applicant via your platform. Alternatively, you can also monitor “zeros”, ie, the non-matches. In this example it would be the number of jobs that are not filled via the platform. At the end of the day, a zero is when you were unable to serve the demand generated. Identifying the cause(s) for being unable to fulfil demand is critical to redesign your product in order to close those roadblocks.

#12 – Market depth: this is an imported concept from financial markets. For heterogenous marketplaces, where each supplier is different, market depth measures the quantity of offering at a specific range, ie, how easy it is to find a match. In AirBnB, how many available houses there are at a specific price point? The user experience is obviously better with more offer to choose from, and the market has more depth. For homogenous markets, where suppliers are equal, market depth impacts ease of use. With Uber, more drivers equal less waiting time.

Bear in mind that one of the primary goals of a marketplace is to reduce the search costs, which means making the match easier. If it’s hard or too costly to match, then we will see negative network effects, due to either search fatigue or paradox of choice.

One final note on homogeneous versus heterogeneous marketplaces: for the former there’s a plateau that is achieved with a certain number of providers available, given that there’s no differentiation amongst them. Ie, it’s the same for me to have 5 available Uber drivers nearby or having 30. However, if I’m searching freelance designers having more options is better because it increases the probability of finding one that perfectly matches my needs.

#13 – Time to find a match: how long does it take for a homeowner to receive a booking for his AirBnB property, or an employee to start receiving applicants?

#14 – Concentration or fragmentation of supply and demand: a more fragmented market (on both sides) is healthier, because there’s less control of a party (or group of). A fragmented platform is also more valuable because you can serve customers and providers that are harder to reach. Which also can be a moat, since it’s easier to switch homogenous groups. A good way to monitor concentration is by measuring the % GMV of top sellers / buyers.

Economics related metrics

#15 – Pricing power: the ability you have to charge and/or raise prices while your customers still get good value from your product and don’t churn.

#16 – Unit economics: with greater network effects comes an improvement on the general unit economics. While this is a lagging indicator, and not 100% correlated to the network effect, it’s still a good indicator. Again, the analysis must be done at the geographical level of the market, ie, if it’s a city market you need to analyse the unit economics for each city alone.