Metrics that measure consumption vs metrics that measure the satisfaction of real needs

Edited excerpt from Is anything worth maximizing? by Joe Edelman:

Metrics have allowed us to scale up the size of our organizations. But they’ve also created a kind of tug-of-war. On the one hand, we have the nuanced values of individuals. On the one hand, we have the simplifying assumptions of economies and organizations. These two are in constant tension throughout the economy.

Overconsumption comes from tweaking products and channels so as to maximize sales, views, and clicks. That has trade-offs for long-term satisfaction, and for wellbeing. If we keep focusing on sales, views, and clicks, we’ll wind up fat, depressed (or on Prozac), socially isolated, diabetic, bloodshot staring at screens or jacked into VR, and surrounded by piles of junk we regret buying.

When governments or big businesses focus on consumer spending, on consumption, they’re missing all the reasons that people buy and focusing on the buying behavior itself.

Same with overconsumption of media. There are reasons that we want to read and learn. Reasons we want to know about our friends’ lives, or just to see a photo of someone we love who’s far away. But when a business like Facebook tries to maximize engagement, it loses track of those reasons; it treats us as engagement machines. We go over-consumed, but under-fulfilled.

When people discover that a new organization has metrics that hew closer to our real values, users will fly to this new provider. Engagement maximizers won’t see it coming. And the new metrics of the new organization will become the new standard throughout the relevant markets. I call this the flight to higher ground. It’s happened many times in the last hundred years, but I believe we can go much further.

Notes:
(1) Cf. How tech products misframe our choices, and product managers should do better.
(2) Cf. Provide what your users really want.

Why you should think about your goals each day on the way to work

Edited excerpt from Can’t Stand Your Commute? It’s All in Your Head by Chana R. Schoenberger:

People who make a conscious effort while commuting to think about what they need to do that day and how it fits into their longer-term plans feel more satisfied at work and less exhausted emotionally, according to a new Harvard Business School working paper.

They also have happier commutes.

The researchers call this way of thinking “goal-directed prospection,” and in their paper they detail how it can offset the strain of commuting.

Notes:
(1) Cf. What’s your “simple scoreboard”?
(2) Cf. How to choose the one metric that matters.

The first goal for a startup isn’t growth

Edited excerpt from Before Growth by Sam Altman:

In the first few weeks of a startup’s life, the founders really need to figure out what they’re doing and why. Then they need to build a product some users really love. Only after that they should focus on growth above all else.

I think the right initial metric is “do any users love our product so much they spontaneously tell other people to use it?” Until that’s a “yes”, founders are generally better off focusing on this instead of a growth target.

Notes:
(1) You can measure “do any users love our product so much they spontaneously tell other people to use it?” with net promotor score — see How to use net promotor score surveys to improve your product and Net promotor score — how to set up the survey.
(2) Cf. Is this the right goal for seed-stage startups?
(3) Re. “only after that should they focus on growth”: see The most fatal mistake to avoid as a startup.

Is this the right goal for seed-stage startups?

The final slide from an investor deck sent by a startup founder:

Financing

Raising: $1.5 million

Deliverable: MVP and 10,000 delighted beta users who would recommend the product.

Notes:
(1) Love the clarity: “Here’s how much we want to raise, and here, in one short sentence, is what we’ll deliver for that budget.”
(2) Note that the key deliverable is a product of demonstrated quality — an MVP with high net promotor score, validated by a sufficient number of users.
(3) Setting net promotor score as an explicit goal stops the company from trying to scale before the product is good enough. See The most fatal mistake to avoid as a startup.

A simple metric to measure the quality of your user growth

Edited excerpt from Diligence at Social Capital Part 1: Accounting for User Growth by Jonathan Hsu:

Quick Ratio = (new users + resurrected users)/churned users

(This should not be confused with the normal finance quick ratio that measures the ability of cash and near cash assets to pay off liabilities.)

This ratio needs to be greater than one if the app is to be growing, otherwise churn is overwhelming growth.

Most consumer applications don’t have a very strong mechanism to bring users back month after month and so the quick ratio tends to be just above 1. The dynamic for each month in a consumer app is typically to add a bunch of users and to simultaneously lose a bunch of users with a small additive piece on top from resurrection yielding overall small positive growth.

All else being equal, a company with the same growth but a higher quick ratio is more attractive company to us because it is starting from a better base. With high retention, it would be worth trying to push harder on the top of funnel with new users to drive growth (more aggressive sharing/referral mechanisms, paid acquisition, etc.). If a company has higher churn, it’s harder to justify pushing on new users as you would end up losing many of them. It’s easier to fill the top of funnel than it is to fix some underlying churn problem.

This accounting can be done on time-frames other than calendar months. Indeed, several of our portfolio companies implement it on a rolling 28 day basis (to remove day-of-week effects).

Notes
(1) Thank you Guy Cohen for the article tip.
(2) Cf. Why retention is the key to growth and Sustainable growth vs. growth hacking.
(3) Practical advice on how to reduce churn: (i) How to reduce churn — a process, (ii) How to reduce churn by winning back cancelled customers, (iii) How to think about churn.

Why Skift rejected monthly uniques as its key metric, and you should too

Edited excerpt from How We Got off the Addiction to Venture Capital and Created Our Own Way to Profits by Skift co-founder and CEO Rafat Ali:

We gave up chasing scale. We took out *all* goals on traffic on the site, for everyone. We could do this because we didn’t have tons of outside money pumping through our veins, and this was a useless pressure we created for ourselves in an effort to show the illusion of growth to investors. And since we weren’t chasing investors, we didn’t need to chase what they would consider scale. It was a vanity metric.

We cut back on spending any money on getting users through Outbrain/Facebook/Twitter. We cut back on the number of stories we were doing on a daily basis, on chasing the tail on disposable news stories. We also cut back on syndicating our stories — in which we put in a lot of effort at the start, publishing on NBC News, CNN, Quartz, Fox News, Business Insider, Mashable and many others, to zero effect on our revenues — and also cut back on publishing useless filler syndicated stories we got from a third party syndication service.

In an era where everyone is tripping over each other to call themselves a “distributed media platform”, we decided to focus solely on building direct channels to our users, which for us meant email. Email, despite all its shortcomings, is that direct promise day in and day out to our users, you see us the first thing in the morning, in the intimacy of your inbox.

Notes:
(1) Rafat’s rejection of broad traffic growth as Skift’s key metric is similar to the approach I took with Seeking Alpha: I made our key metric daily direct uniques, (“DDUs”). Daily uniques captures loyalty via frequency of daily visits, and direct visitors focuses you on people who value your brand and content, rather than those allured by link-bait. As a result, making DDUs your key metric pushes you to create genuine value for your users.
(2) Once you measure success by the number of direct visitors, you’ll almost inevitably focus on converting readers to email subscribers, as Rafat describes he did. We’ve did the same: Seeking Alpha has over three million email subscribers, most of whom are subscribed to email alerts on stocks in their portfolio. (The free alerts let you know about breaking articles and news stories on your stocks. Subscribers include individual investors, most public company managements, and thousands of hedge funds.)
(3) This does not mean that broader audience metrics like monthly uniques, which include indirect and infrequent visitors, are unimportant. You should view them as the top of your funnel, as a means to an end. Your job is to attract indirect or infrequent visitors so you can convert them to frequent, direct users. You should have a team (and owner) responsible for conversion to frequent, direct users. And you should report DDUs (or maybe the ratio of DAUs/MAUs) as the key metric for the company.
(4) On key metrics, see (i) Growth rate in revenue or active users is the paramount startup metric, (ii) The key metric for your startup must satisfy these 4 criteria, and (iii) Why you shouldn’t optimize for social sharing.
(5) On using email subscriptions to convert infrequent to frequent users, see De Correspondent’s use of email subscriptions to convert Facebook users to loyal readers.

Is your startup a zombie? These metrics and behaviors will tell you

Edited excerpt from Zombie Startups by Danielle Morrill:

How do you know if your startup is a zombie? Here are some hints:

  • You don’t want to get out of bed in the morning
  • You don’t want to go out in public for fear you’ll have to explain what you do
  • You haven’t hit 10% week-over-week growth on any meaningful metric (revenue, active users, etc) at some point like launch or some other PR event.
  • You’re working on the same idea after 12+ months and still haven’t launched
  • You’ve launched a consumer service and have less than 2% week-over-week growth in signups
  • You’ve launched an enterprise service and have less than 2% week-over-week growth in revenue pipeline
  • You are the CEO and hole yourself up in the offices so you don’t have to talk to employees
  • You’ve hired consultants to figure out revenue, culture, or product in a company of less than 10 people

Does any of this sound familiar? If so, don’t panic – you can fix this. Acknowledge the reality of your situation. Then figure out what to do next because you don’t want to waste a single moment of your life in denial, in deadlock, in zombie mode waiting for something you can’t control to change or expecting magic to happen.

Is revenue a good metric for early stage startups?

Edited excerpt from Revenue vs. Value by Danielle Morrill:

While revenue is a useful signal to founders, indicating they are creating something people want, it is also a lagging indicator of success. By the time a startup has a predictable and steadily growing revenue stream that means it has built a product and brought it to market successfully.

Early revenue can be dangerously distracting for founders. Once you have some of it you want more, and without strict discipline it’s easy to optimize for immediate gratification rather than the big vision.

Don’t let revenue be your vanity metric.

Notes:
(1) Compare this to Growth rate in revenue or active users is the paramount startup metric.
(2) Even if revenue isn’t the right metric for early stage startups, you still need to know where your revenue will come from. You need to know the end game. This is because the key goal of early stage startups is to achieve product-market fit, and product-market fit requires a market, a business model and customer engagement.
(3) Delaying figuring out your source of revenue can be dangerous, because monetization of free products after the fact is challenging.
(4) For these reasons, there’s a strong argument to be made that even with freemium, it’s easier to start with the paid product.

Key metrics for startups by industry — a quick guide

Edited excerpt from The One Metric That Matters by Alistair Croll:

Transactional
Someone buys something in return for something. Transactional sites are about shopping cart conversion, cart size, and abandonment. To be useful, however, it should be a long funnel that includes sources, email metrics, and social media impact.

Collaborative
Someone votes, comments, or creates content for you. Collaboration is about the amount of good content versus bad, and the percent of users that are lurkers versus creators. This is an engagement funnel, and we think it should look something like Charlene Li’s engagement pyramid.

SaaS
Someone uses your system, and their productivity means they don’t churn or cancel their subscription. SaaS is about time-to-complete-a-task, SLA, and recency of use; and maybe uptime and SLA refunds.

Media
Someone clicks on a banner, pay-per-click ad, or affiliate link. Media is about time on page, pages per visit, and clickthrough rates.

Game
Players pay for additional content, time savings, extra lives, in-game currencies, and so on. Game startups care about Average Revenue Per User Per Month and Lifetime Average Revenue Per User (ARPUs). Games need to solicit payments without spoiling gameplay.

App
Users buy and install your software on their device. App is about number of users, percentage that have loaded the most recent version, uninstalls, sideloading-versus-appstore, ratings and reviews.

Don’t make this mistake with metrics

Edited excerpt from 7 Things I Learned from Startup Failure by Gary Swart:

I once had a board member tell me that we were over-measured and under-prioritized. It stung. A lot. But it also made quite an impression. As a business leader you need to figure out the metric that matters most for your company and understand that the more you measure, the less prioritized you’ll be. Don’t fall into the trap of trying to measure everything. What I’ve learned is that in the early days, what matters most is having customers who love and use your product. Figure out the one or two best measures to determine this.

Notes:
(1) There’s a difference between what you measure and what you prioritize. While you might want to track many metrics, you need to prioritize a few of them. Those goals need to be clear in your internal communication and board packet.
(2) Cf. (i) What’s your “simple scoreboard”? and (ii) Brian Balfor’s discussion of Hubspot’s key metric.

How to manage someone who is missing their goals

Eli Hoffmann, describing what a manager should say to someone who is missing their targets:

Hey. You aren’t hitting your targets. I’m not sure what’s going wrong, but I need you to hit them. Success equals growth, and growth means setting aggressive targets and hitting them.

What I need from you is a game plan. What’s going to change that will get us back on track? You probably need some time to think about this. When should we meet again to review your game plan?

Also, it’s important for you to know that I want you to succeed. If there’s anything I can do to make you more successful, let’s discuss that and I’ll do whatever I can.

Notes:
(1) The context: we were discussing what causes managers to slip into micromanagement. I suggested that it’s natural to micromanage someone when they are chronically missing their goals. You step in to help, to fix the situation.
(2) The problem is that, in my experience, micromanagement rarely works. Perhaps that’s due to The Set-Up-To-Fail Syndrome.
(3) Eli’s approach is powerful: it avoids micromanaging and disempowering your report, but forces her / him to rethink and come back with a plan.
(4) Cf. When micromanagement works.

Do stretch goals incentivize cheating and short-termism?

Edited excerpt from Charlie Munger’s comments at the Wesco 2000 Annual Meeting, quoted in The Tension Created By Stretch Goals:

There are two lines of thought. A whole bunch of management gurus say you need BHAGs — bold, hairy, audacious goals. That’s a technique of management, to give the troops a goal that looks unattainable and flog them heavily. According to that line of thought, you will do better chasing a BHAG than you will a reasonable objective.

Then there’s another group that says that if you make the goals unreasonable enough, human nature being what it is, people will cheat.

These two factors are at war. You don’t want the cheating, which is bad long term and bad for the people who are doing the cheating. However, you do want to maximize real performance.

What people generally do is give people the unreasonable goal and tell them, “You can’t cheat.” That is the American system in many places.

I’ve got no answer to that tension. Low goals do cause lower performance and high goals increase the percentage of cheating. Each organization has to find its own way.

Notes:
(1) In a startup like Seeking Alpha, I’m less concerned that managers will cheat (ie. be dishonest) than optimize for short term growth at the expense of medium term growth. However, if the manager knows they’ll stay in their role for the medium term, is that really a risk?
(2) “Low goals do cause lower performance.” See (i) Don’t set goals based on what you think you can achieve, (ii) Think big, (iii) Ensuring you’re sufficiently ambitious, and (iv) Set stretch goals.
(3) Setting ambitious goals isn’t just about performance, but about vision and aspiration too. See Lessons from Jeff Bezos: Set “audacious and inspirational” goals.

The difference between a great company and a lousy company

Edited except from The Hard Thing About Hard Things by Ben Horowitz:

In good organizations, people can focus on their work and have confidence that if they get their work done, good things will happen for both the company and them personally.

In a poor organization, people spend much of their time fighting organizational boundaries, infighting, and broken processes. They are not even clear on what their jobs are, so there is no way to know if they are getting the job done or not.

Notes:
(1) How do you ensure that if people get their work done, good things will happen for both the company and them personally? Define clear goals; identify levers to achieve those goals; appoint owners; measure results.
(2) See: Setting clear goals = empowerment, Managers and metrics, and Mark Pincus’ management advice.

The problem with collaboration, and why goals should have single “owners”

From The Collaboration Paradox: Why Working Together Often Yields Weaker Results by Ron Friedman:

Studies on collaborations have yielded mixed results:

Collaborations breed false confidence. A study in Psychological Science found that when we work with others to reach a decision, we become overly confident in the accuracy of our collective thinking.

Collaborations introduce pressures to conform. Studies show that group members tend to conform toward the majority view, even in cases when they know the majority view is wrong.

Collaborations promote laziness. Ever been to a meeting where you’re the only one prepared? Then you’ve probably experienced social loafing—people’s tendency to invest less effort when they’re part of a team.

But there’s a bigger problem: Attached to every meeting, conference call and mass email you’re exposed to is an invisible price tag — the opportunity cost of all the tasks you’re not getting done while you’re busy “collaborating.” In many organizations, the higher up you are in the hierarchy, the more often you’re called upon to collaborate. Intellectually, it’s a progressive tax.

Notes:
(1) In Seeking Alpha, we assign each goal and its metrics to a single individual, never to multiple individuals who are then expected to collaborate. People help out anyway, but they know there’s a clear “owner”.
(2) In a document called How to get stuff done in Seeking Alpha, I wrote: Maximize what you can get done on your own. Before you ask for help from others, get as far along as possible on your own. Identify the key person you need to collaborate with, and don’t involve anyone else. Be explicit about what you need from people. Minimize your “ask” of other people’s time.

Don’t set goals based on what you think you can achieve

From Principles, by Ray Dalio:

Don’t rule out a goal due to a superficial assessment of its attainability. Once you commit to a goal, it might take lots of thinking and many revisions to your plan over a considerable time period in order to finalize the design and do the tasks to achieve it. So you need to set goals without yet assessing whether or not you can achieve them.

This requires some faith that you really can achieve virtually anything, even if you don’t know how you will do it at that moment. Initially you have to have faith that this is true, but after following this process and succeeding at achieving your goals, you will gain confidence.

Every time I set goals, I don’t yet have any idea how I am going to achieve them because I haven’t yet gone through the process of thinking through them. But I have learned that I can achieve them if I think creatively and work hard.

I also know that I can “cheat.” Unlike in school, in life you don’t have to come up with all the right answers. You can ask the people around you for help—or even ask them to do the things you don’t do well. There is almost no reason not to succeed if you take the attitude of 1) total flexibility — good answers can come from anyone or anywhere and 2) total accountability — regardless of where the good answers come from, it’s your job to find them.

It’s been my experience that if I commit to bringing creativity, flexibility, and determination to the pursuit of my goals, I will figure out some way to get them. And as I don’t limit my goals to what seems attainable at the moment I set them, the goals I set tend to be higher than they would otherwise be. Since trying to achieve high goals makes me stronger, I become increasingly capable of achieving more. Great expectations create great capabilities, in other words.

Notes:
(1) Cf. Think big.
(2) Thank you Guy Cohen for tip.

If you’re going to measure pageviews, be specific

From The only metric that matters by Josh Elman:

I’ve been lucky to be part of the early growth of several really interesting and now important networks including LinkedIn, Facebook, and Twitter. One of the things that I felt working on each of these is that we never looked at numbers or metrics in the abstract — total page views, logged in accounts, etc. — but we always talked about users. More specifically, what they were doing and why they were doing it.

When I meet new companies today, I often hear things like “We have 10M uniques with 30M page views per month.” While big numbers are a nice signal of, well, big numbers, I don’t think they are an indicator at all for whether a product is really working. Whenever I hear some of these stats, I always ask the same question: How many people are really using your product?

You need a metric that specifically answers this. It can be “x people did 3 searches in the past week”. Or “y people visited my site 9 times in the past month”. Or “z people made at least one purchase in the last 90 days.” But whatever it is, it should be a signal that they are using their product in the way you expected and that they use it enough so that you believe they will come back to use it more and more.

Once you can define a metric to answer this, then you can really track your growth on a day-to-day, week-over-week, month-over-month basis. And from there, you can identify the key supporting metrics that show you how likely it is more people will convert to using your product on a frequent basis, how likely they are to stay on your product vs churn out, etc.

Notes:
(1) Cf. the second of Brian Balfour’s 4 criteria for a key metric: “Meaningful interaction: Qualifying events should be meaningful interactions with the product, not just “fly by’s” such as visits or sessions.”
(2) This is more evidence that monthly uniques is a vanity metric, and isn’t used internally by the highest growth companies.

The key metric for your startup must satisfy these 4 criteria

Edited excerpt from Avoiding The Wheel Of Meaningless Growth by Brian Balfour:

The one metric that guides us like our north star is Weekly Active Users. We defined this as our authentic growth metric using a few criteria:

1. Retention. In the consumer world things like Daily Active User (DAU) and Weekly Active User (WAU) are most commonly used. Celebrating metrics such as total registrations or downloads over time tell you nothing about whether or not you are building real value.

2. Meaningful interaction. Qualifying events should be meaningful interactions with the product, not just “fly by’s” such as visits or sessions.

3. Relevant. The metric must be relevant to the intended use of your product. If your product is meant to be used daily, then something around Daily Active Users. Weekly, then weekly active users.

4. Brutally honest. How many times have you read a title such as “XYZ sees 452% (or some incredible number) Growth” Or “XYZ company grew by 10,000 new users.” Here is the problem…452% growth of something small is still small.  Growing something from 10,000,000 to 10,0010,000 isn’t a win.  And growing something, but taking an immense amount of time probably isn’t a win either. There a lot of ways you can frame numbers to make them sound good.  Just be honest with yourselves. Whenever we communicate/report growth metrics, team members report on the percentage increase, the absolute increase and the time period in which the growth occurred.

Notes:
(1) Note the emphasis on retention. Cf. Eric Schonfeld’s How to distinguish vanity metrics from real metrics and Bill Gurley’s The best startup metric: Conversion rate?
(2) Monthly uniques is the most popular metric reported by web publishers. Perhaps it’s because advertisers run campaigns monthly and care about audience reach. But monthly uniques is a vanity metric for the publishers themselves, because it doesn’t capture retention.
(3) On daily versus weekly uniques: We use daily uniques as our key metric in Seeking Alpha, because finance is a daily habit. (That’s one of the most attractive aspects of our vertical.) But we add an additional requirement to ensure that the interactions aren’t fly-by’s: we only measure users who come to us directly, not via search or inbound links from other sites. Our key metric is therefore what we call DDUs — daily, direct users. It’s an exceptionally tough metric to grow. Here’s how we’re doing.
(4) In the section on being brutally honest, note Brian’s emphasis on measuring growth, not absolute levels. This is because the measure of a startup’s success is its growth rate.
(5) Reporting metrics as growth rates (or absolute growth) focuses the company on growth, and encourages everyone to think big.

Don’t celebrate or publicize vanity metrics

Edited excerpt from Avoiding The Wheel Of Meaningless Growth by Brian Balfour:

There is a cycle that plagues a lot of companies. It typically works like this:
1. A startup wants some press, so they look for some bloated number to give the writer (downloads, registrations, visits, etc).
2. The startup then celebrates that press article, internally supporting the message that the bloated metric is worth pursuing.
3. In order to get additional press hits, the startup needs to increase that bloated number, so they focus on increasing it (back to Step 1).

This cycle might seem harmless. After all, isn’t getting press a good thing? But what you celebrate (externally or internally) is what your team will think about and focus on. It is giving up the long term, for short term gains. Just don’t do it.

Why do most companies push out inauthentic growth metrics?
— Inauthentic growth metrics almost always sound more impressive (especially at the earliest stages when startups are the most desperate to get press).
— Authentic growth metrics are much harder to improve than inauthentic ones.
— A lot of writers in the tech space either don’t know better and/or are writing for page views and attention grabbing headlines and are incentivized to use inauthentic numbers.

Notes:
(1) We just made this mistake. We published an article last week celebrating the fact that Seeking Alpha had hit 4 million registered users. Registered users is a vanity metric, because registration numbers on their own tell you nothing about the quality or engagement level of the people who are registering. So why did we do it? Because, as Brian wrote, we knew that many people would be impressed. And we believed that while registered users is insufficient on its own and is therefore a flawed metric, in our case we knew that the quality and engagement level of those registered users was high.
(2) But a flawed metric is still a vanity metric. So we fixed the mistake. We rewrote the article and included a chart of our real metric. This is what we ended up with.

Being metrics-driven leads to brutal intellectual honesty

From Brutal Intellectual Honesty by Tomasz Tunguz, paraphrasing a comment by Aruba Networks CEO Dominic Orr:

Brutal intellectual honesty. There are too many decisions to make in too little time. You want to get all the facts on the table, have an honest debate for a period of time, and at the end of the time, make a decision and go. Once ego becomes involved, people have to find a way to save face, which is incredibly time wasting. And if you recall, time is the only competitive resource we have.

According to Dominic, to elicit intellectual honesty from a management team demands coaching and counseling team members individually, so each person is comfortable enough in their environment and confident enough in their own opinions to speak their mind freely. Only then can a team determine the best route to pursue.

Notes:
(1) There’s a strong connection between being metrics-driven and being intellectually honest. If your success is defined entirely by how well you achieve your goals and metrics, you’ll care only about which arguments are correct and which ideas are the best, irrespective of who they come from. If, on the other hand, your success isn’t defined by metrics, but by who comes across as the most competent or impressive, it matters a lot who comes up with the best ideas and who wins the arguments.
(2) This is why metrics-driven organizations are fundamentally more meritocratic. If you don’t measure, reward and promote people for achievement of goals and metrics, politics and ego will fill the vacuum.
(3) Getting managers to be truly metrics-driven is far harder than it seems. In my experience, it also requires significant coaching. Managers can say they’re metrics-driven, but in reality they’re not. “Even though I missed my numbers, I still did lots of good things…”.

Every metrics-driven manager needs to watch out for this risk

From There’s a Name for the Big Flaw in Our Obsession With Assessment and Metrics:

We live in an age obsessed with assessment. Tough-minded CEOs and numbers-driven wonks are forever looking for the perfect metric: the most objective way to define, measure, and reward success in a given arena. But there’s a tragic flaw in this hardheaded approach, and it’s known as Campbell’s Law.

Here’s the gist: The more a given metric—say, a national college ranking—is used to evaluate performance in some domain, the less reliable it becomes as a measure of overall success. Why? The people whose performance is being measured will neglect other parts of their job just to focus on boosting the relevant numbers, sometimes to the point of cheating. The chosen metric will inevitably “distort and corrupt the social processes it is intended to monitor,” suggested the social psychologist Donald Campbell back in 1974.

Notes:
(1) Cf. the Wikipedia definition of Campbell’s Law.
(2) An example: Social networks discovered that engagement was determined by the number of people you followed / friended / connected to. So they encouraged users to follow / friend / connect to as many people as possible (= the metric). But because the added relationships were weaker, they gradually destroyed the value of their social networks.
(3) Team leaders in Seeking Alpha are given a goal and a metric. The goal is a verbal formulation of what they’re trying to achieve. The metric is the best available way to measure the goal. Can a properly formulated goal eliminate the risk of Campbell’s Law?
(4) Thank you Rachael Granby for the tip.

Growth rate in revenue or active users is the paramount startup metric

From Startup = growth by Paul Graham:

If there’s one number every founder should always know, it’s the company’s growth rate. That’s the measure of a startup. If you don’t know that number, you don’t even know if you’re doing well or badly.

When I first meet founders and ask what their growth rate is, sometimes they tell me “we get about a hundred new customers a month.” That’s not a rate. What matters is not the absolute number of new customers, but the ratio of new customers to existing ones. If you’re really getting a constant number of new customers every month, you’re in trouble, because that means your growth rate is decreasing.

During Y Combinator we measure growth rate per week, partly because there is so little time before Demo Day, and partly because startups early on need frequent feedback from their users to tweak what they’re doing.

A good growth rate during YC is 5-7% a week. If you can hit 10% a week you’re doing exceptionally well. If you can only manage 1%, it’s a sign you haven’t yet figured out what you’re doing.

The best thing to measure the growth rate of is revenue. The next best, for startups that aren’t charging initially, is active users. That’s a reasonable proxy for revenue growth because whenever the startup does start trying to make money, their revenues will probably be a constant multiple of active users.

Notes:
(1) Y Combinator invests in seed stage startups, so growth rate per week is suitable. For more mature companies, month-over-month growth rate, quarterly-over-quarter growth rate might be more appropriate.
(2) Prioritizing growth assumes that you already have a product that your customers love. See The three steps to building a great company, and why most startups fail on the first step.

The two most important, neglected sales metrics

From Jason Lemkin, in Jason Lemkin: The Right Sales Metrics for Your SaaS Startup:

There are two metrics aren’t discussed enough. The first is lead velocity rate. What rate are your qualified leads growing month over month? Your MRR growth is great, but really that just tells you about the present – how you’re doing now. But if your leads are growing faster than your revenue, I can see the future growth. Being able to quantifiably track the velocity of qualified leads is going to be your best possible indicator as a CEO of where you’re going to be in the future.

The second metric will help founders get from initial traction to scale: understanding revenue per lead and how that works across your company. Once you have a repeatable set of leads and lead velocity, you want to drive up the revenue per lead. That’s an area where you can help the sales team by measuring each individual rep – what’s their revenue per lead? How many leads can you give them before their productivity declines? Why do some reps make certain types of leads more productive than others? The earlier you can do this post-traction the better. Leads are precious for a long time in startups, and if you can get 20 percent more out of each lead, that’s magic. But if you don’t measure it down to the individual rep level and you just look at MRR, you’re missing an opportunity to improve things.

What’s your “simple scoreboard”?

Excerpt (edited) from Adam Bryant Of The New York Times On What Makes Great Leaders Great:

There’s a quality [in great CEOs] that I call a “simple mindset,” which is the ability to take a lot of complicated information and really boil it down to the one or two or three things that really matter, and in a simple way, communicate that to people. In any company—there are always a dozen or more competing priorities. And it is the leader’s job to stand up in front of the troops and say, “These are the three things that we are going to focus on this year,” or “These are the goals and this is how we are going to measure them.” If you really want to galvanize people and get them operating as a team, you’ve got to create a simple scoreboard that everybody understands.

The communication style, to me, is secondary to getting the content right. And what I’ve been so often impressed by is leaders who can essentially boil down the company’s goals and operating model into, literally, less than a page. They can figure out, “Here are the four metrics; these are the three or four things that we are going to focus on,” and do it in a way that not only makes sense for today, but is likely to make sense a year from now.

Managers and metrics

From Jason Lemkin, in Jason Lemkin: The Right Sales Metrics for Your SaaS Startup:

There’s no soft squishy numbers, no best efforts – everyone has a number. If you have a sales team without a quota, you don’t have sales. It’s just as true post sales as pre sales. If managers don’t agree with that philosophy, if they’re best efforts guys – don’t hire them.

You’d be shocked how many SVPs, VPs, and C-level executives don’t really have metrics they personally have to own that are quantified. That just doesn’t work at a startup.

Notes:
(1) It’s the CEO’s responsibility to make sure that every manager personally owns a meaningful metric.
(2) Owning a metric means taking complete responsibility for success in that metric. If you justify misses in your metric after the fact, you don’t own the metric — you’re “a best efforts guy”.
(3) In Seeking Alpha, “owning a metric” means taking responsibility for the success of that metric in the medium term. Every manager reports metrics monthly, but we care about medium term success.
(4) If you miss a monthly target, the key to medium term success is to take the short term miss seriously. If you explain away your monthly metrics, you’ve thrown away their value as a feedback loop.
(5) How do you take short term misses seriously? You ask tough questions relentlessly. If you don’t ask the right questions, you’ll never find the right answers. Asking the right questions requires effort and focus.
(6) Examples of questions to ask when you miss your monthly metric: “Why am I missing? What are the key bottlenecks that are holding me back, even those not in my direct control? What can generate a 5x medium term increase in my metric, not just a 10% increase? Do I have strong enough people in my team?”

These benchmarks can tell you whether your business plan is viable

From Things I’ve Learned About How to Build Good Product by Peter Nixey:

1. Golden rule of social: 90% of people consume, 10% of people curate, 1% of people create.
2. Rules of engagement: Registered users / installs — 30% will use it each month, 10% will use it each day, 1% will use it concurrently.
3. Fremium conversion: 1% of your signups will pay (normally 1-5%).
4.  Expensive is profitable: “50% of our revenue comes from only 11% of our customers” — Ryan Carson, founder Carsonified and Treehouse.
5. Cheap is expensive: Cheap accounts cost more in support (7x).
6. Advertising revenue: Social network CPM ~$1, Times Newspaper ~$10; $1M / year in revenue = 1bn page views (good luck).
7. Email subscriptions: Open rates = 20-40%, click through rates = 1-5%; 1 sale per 3,000 emails.
8. Churn determines SaaS size: If you have 10,000 customers and 5% leave a year, then you need 500 new ones just to stay level (so your signup rate determines your company size).
9. Is your market size sane? 360m people in the US. There are 6m firms with 1-100 employees (avg. # employees 7.24), 6m firms with 101-500 employees (avg # employees 10.1), and 17,000 firms with 501-10,000 employees  (avg. # employees 1,672).

Benchmarks for key metrics

From 30/10/10 by Fred Wilson:

One thing that never ceases to amaze me is how similar some of the metrics are from service to service and company to company. I like to call these the web/mobile laws of physics. One fairly common “law of web/mobile physics” is the ratio of registered users/downloads to monthly actives, daily actives, and max concurrent users (for services that have a real time component to them).

I call this ratio 30/10/10 and so many services that we see exhibit it within a few percentage points here and there. Here’s how it works:

– 30% of the registered users or number of downloads (if its a mobile app) will use the service each month

– 10% of the registered users or number of downloads (if its a mobile app) will use the service each day

– The max number of concurrent users of a real-time service will be 10% of the number of daily users

How to distinguish vanity metrics from real metrics

From Don’t Be Fooled By Vanity Metrics by Eric Schonfeld:

Vanity metrics are things like registered users, downloads, and raw pageviews. They are easily manipulated, and do not necessarily correlate to the numbers that really matter: active users, engagement, the cost of getting new customers, and ultimately revenues and profits. The latter are more actionable metrics. As First Round Capital’s Josh Kopelman recently advised on Founder Office Hours, “The real data is retention and repeat usage.” Startups that focus on the real metrics can make their products better, attract more customers, and make them happier.

It is important for startups to properly instrument the data they track so that they can get a handle on the true health of their business. If they track only the vanity metrics, they can get a false sense of success. Just because a startup can produce a chart that is up and to the right does not mean it has a great business. A mobile app could have millions of downloads but only a few hundred thousand active users, or a freemium website might see exploding traffic growth but barely any conversions to paying users.

Why you shouldn’t optimize for social sharing

From What You Think You Know About the Web Is Wrong by Chartbeat CEO Tony Haile:

The people who share content are a small fraction of the people who visit that content. Among articles we tracked with social activity, there were only one tweet and eight Facebook likes for every 100 visitors. The temptation to infer behaviour from those few people sharing can often lead media sites to jump to conclusions that the data does not support.

A widespread assumption is that the more content is liked or shared, the more engaging it must be, the more willing people are to devote their attention to it. However, the data doesn’t back that up. We looked at 10,000 socially-shared articles and found that there is no relationship whatsoever between the amount a piece of content is shared and the amount of attention an average reader will give that content.

How to specify challenges and goals correctly

From What is a “SMART” Problem Statement at McKinsey? by Working With McKinsey:

The problem statement clearly defines, in a concise but comprehensive way, the key business problem that needs to be solved. Even though it’s called a problem “statement”, it’s usually in the form of a question.

SMART is an acronym for Specific, Measurable, Action-oriented, Relevant, and Time-bound.  A good problem statement should be all of those things. The challenge is to balance being thorough with being concise.

Some examples from problem statements that are not SMART:

– Not specific: “better manage the business…” – this is too generic and doesn’t suggest where the greatest impact might be, how, or by when to capture it.

– Not measurable: “reverse our deteriorating performance…” – without specifying what metrics best reflect “performance”, it’s impossible to assess whether or not we’ve made progress.

– Not action-oriented: “increase sales and decrease costs…” – while these are both appealing goals for any company, they have to be actionable to create impact.

– Not relevant: “increase profits by raising prices…” – this sounds good, but not if the client is selling a commodity that sells at market prices.

– Not time-bound: “eventually increase profits by 10%…” – a specific deadline is needed to motivate people to action, hold them accountable, and know if the project was successful.

Notes: (1) McKinsey’s rigor around fundamental processes is stunning; another example I quoted is the McKinsey Feedback Model. (2) For more on SMART, see the Wikipedia page.

How to choose the one metric that matters

Excerpt from How Many Metrics Do You Need to Run Your Startup? by Ben Yoskovitz:

One of the concepts we’re pushing hard in the Lean Analytics book is the One Metric That Matters (OMTM). The idea is this: at any given time in your business there’s one key metric you should be focusing on. Here are some elements of the One Metric That Matters to keep in mind:

  1. It will change over time.
  2. It answers the most important question you have.
  3. It has a clear goal.
  4. It focuses the entire company.
  5. It inspires a culture of experimentation.

Notes:
(1) When picking a key metric, think through the incentive effects. By optimizing for this metric, what will we do? What will we not do?
(2) We made Seeking Alpha’s one metric that matters daily direct uniques. We thought through the incentive effects to ensure they were aligned with our long term goals. The result: we eliminated our reliance on traffic partners, raised quality, and strengthened our brand.
(3) Had we chosen a metric like page views, monthly uniques or social shares, we’d have been incentivized to do bad things, with serious consequences.

Setting clear goals = empowerment

From A Small World CEO Sabine Heller‘s Corner Office interview:

You have to manage people based on results and set clear goals. It sounds like a simple thing, but people don’t do that often. When I was 22 and working at UGO, it didn’t matter that I had no experience and it didn’t matter what my process was as long as I hit my goal. It taught me how empowering it is to be treated like that. I am a great manager for people who are strong thinkers and motivated. I empower people. I promote people. I give them a lot of leeway. At the end of the day, I look at results, and that’s it. I feel very strongly that organizations infantilize employees. You should treat them like adults.

Notes:
(1) When you give people clear goals and metrics, it releases them from being micro-managed at the task level.
(2) In many roles, being given responsibility for a measurable goal also allows employees to manage their time as they want to. That can be particularly important for companies that want to create a culture which is attractive to people who care about family time.

How to avoid the narrative fallacy

From Farnham Street:

When Brad Stone met with Jeff Bezos to solicit his cooperation for the book, Stone wasn’t prepared for one of Bezos’s questions: “How do you plan to handle the narrative fallacy?”

The narrative fallacy, Bezos explained, was a term coined by Nassim Taleb in his 2007 book The Black Swan to describe how humans are biologically inclined to turn complex realities into soothing but oversimplified stories. Taleb argues that the limitations of the human brain resulted in our species’ tendency to squeeze unrelated facts and events into cause-and-effect equations and then convert them into easily understandable narratives. These stories, Taleb wrote, shield humanity from the true randomness of the world, the chaos, of human experience, and, to some extent, the unnerving element of luck that plays into all successes and failures…

In Taleb’s book — which, incidentally, all Amazon senior executives had to read — the author stated that the way to avoid the narrative fallacy was to favor experimentation and clinical knowledge over storytelling and memory.

Mark Pincus’ management advice – make everyone the CEO of something

From a Corner Office interview with Mark Pincus:

Q. How has your leadership style evolved, given your experience running several companies?

A. You can manage 50 people through the strength of your personality and lack of sleep. You can touch them all in a week and make sure they’re all pointed in the right direction. By 150, it’s clear that that’s not going to scale, and you’ve got to find some way to keep everybody going in productive directions when you’re not in the room. And that, to me, is a huge amount of what it means to manage.

Q. So give me an example of what you did to change that.

A. I’d turn people into C.E.O.’s. One thing I did at my second company was to put white sticky sheets on the wall, and I put everyone’s name on one of the sheets, and I said, “By the end of the week, everybody needs to write what you’re C.E.O. of, and it needs to be something really meaningful.” And that way, everyone knows who’s C.E.O. of what and they know whom to ask instead of me. And it was really effective. People liked it. And there was nowhere to hide.

This January 2010 interview sparked a crucial discussion in Seeking Alpha about how to empower and manage employees, and the difference between Mark’s “everyone is CEO of something” and Apple’s directly responsible individuals (DRIs). I met Mark for the first time this week. He said he still believes in the principle of “everyone’s a CEO of something”, but only when your company has fewer than 1,000 people.

Thank you, Mark, for the help you unknowingly gave us.

The best startup metric: Conversion rate?

Bill Gurley says conversion rate, which measures the number of visitors who come to a particular Web site within a particular period divided into the number of people who take action on that site (purchase, register and so on), is the most important metric, not share of habit. “No other single metric”, he argues, “captures so many aspects of a high-quality Web site in a single number”:

  1. User Interface:  Sites that are easy to use have high conversion rates.
  2. Performance:  Sites that are extremely slow, or those that exhibit errors or time-outs, will always have low conversion rates.
  3. Convenience:  Some users want to get in and out as fast as possible. These frequent users value convenient sites, and frequent users drive up conversion rates.
  4. Effective Advertising: ads that properly identify and entice customers in the right demographic that are poised to purchase will have extremely high conversion rates.
  5. Word of Mouth:  Referred customers have a high conversion rate.

Quick comment: The problem with a rate is that it can fall if the denomenator (for example total visitors to your site) is artificially boosted, when that just means you have a larger funnel of people to convert to your true goal-metric.

The best startup metric: Share of habit?

Tom Tunguz argues that share of habit is a better metric for startups to focus on than engagement:

Engagement fails the majority of products as the best metric to optimize because maximizing engagement/time-on-site contradicts the product’s purpose. Google relentlessly whittles down the time it takes for users to complete a search. Sparrow reduces email client use. Online travel agents and e-commerce companies minimize conversion funnel duration. Expensify slashes time spent filing expenses.

For other products, engagement is a good metric. Facebook and Instagram and the New York Times and YouTube and other media sites do maximize time on site. The more time on site a user spends the greater the number of ad impressions and hence revenue.

But all these products seek to maximize share of habit… winning share of habit means understanding the environment of the user – what competition for time and attention does my user face – and creating a superior experience.

“Share of habit”, however, has disadvantages. You can’t measure it if you don’t have access to accurate market size data — and who has that?  And it’s not a good operating metric, as it’s impacted by external factors out of your control. In that respect, it’s a vanity metric.

Why cash flow, not profits, determines shareholder returns

From Mike Moritz, describing Amazon:

It isn’t too much of an exaggeration to say that Amazon’s entire business has been financed by vendors and customers: book-sellers who collect their invoices slowly; consumers who stump up money for Amazon Prime in advance of receiving deliveries; or companies that pay in advance for guaranteed capacity on AWS. In Los Angeles customers who pay $220 up front for Amazon Fresh, the company’s home delivery grocery service, get ‘free’ shipping on orders above $35. It might be ‘free’ but Amazon has their cash. Customers and vendors have helped Amazon build its 90 fulfillment centers, which now enclose about 65 million square feet. That should be enough to make the managements of FedEx and UPS tremble. Since inception Amazon has generated $20.2 billion from operations almost half of which ($8.6 B), has been used for capital expenditures such as new distribution centers, which improve life for the customer. In the past ten years the share base has only increased by just over 10% while the company has grown twelve-fold. For shareholders it doesn’t get better than that.

Lessons from Jeff Bezos: Set “audacious and inspirational” goals

From Mike Moritz, describing Amazon:

Read about Amazon’s aspiration for a business and you will quickly sense how the company prefers to pursue large opportunities rather than scads of small ones. In 1998 the dream was to become a company “Where customers can come to find and discover anything and everything they might want to buy online.” When the Kindle was introduced it was with the aim to make “Every book ever printed in any language all available in less than 60 seconds.” In both these cases and others such as Fulfillment by Amazon, AWS or international expansion the goals are audacious and inspirational and the management has had the stamina to pursue them and the patience required to make them succeed. (Compare the challenge of developing a $1B business internally to making an acquisition that immediately bolts on the same number).

How LinkedIn manages people using objectives and key results

From The management framework that propelled LinkedIn to a $20 billion company:

LinkedIn manages its teams using a task-tracking system called ‘Objectives and Key Results,’ abbreviated as “OKRs.” First developed by Andy Grove at Intel, the strategy was popularized by John Doerr from Kleiner Perkins…

In Grove’s famous manual “High Output Management,” he introduces OKRs by answering two simple questions: (1) Where do I want to go? (2) How will I know I’m getting there? In essence, what are my objectives, and what key results do I need to keep tabs on to make sure I’m making progress? When you think about it, these questions are very personal, speaking to the core of how people spend their days. It makes sense that everyone within an organization should have their own OKRs every quarter. The important thing is tying these individual OKRs to team OKRs and, ultimately, organizational OKRs. This alignment packs power and efficiency.

Understanding the personal nature and motivating potential of OKRs, (LinkedIn CEO Jeff) Weiner defines them more broadly. They should be about “something you want to accomplish over a specific period of time that leans toward a stretch goal rather than a stated plan. It’s something where you want to create greater urgency, greater mindshare.” For all these reasons, OKRs should become more important the more senior an employee becomes. When you’re in a leadership position “you are sending the signal to the rest of the organization that ‘this matters,’” Weiner says.

OKRs should definitely not be easily achievable. Low expectations may seem to yield glowing results, but they eventually stall people, teams and companies in the long run. OKRs shouldn’t be too malleable either. They’re supposed to be quarterly beacons, not shifting from week to week. Along these lines, Weiner prefers that his team members set three to five OKRs for themselves in any given quarter.

The key to success in any enterprise

From Bill Gates’ 2013 annual letter for the Gates Foundation; the emphasis is added by me:

Over the holidays I read The Most Powerful Idea in the World, a brilliant chronicle by William Rosen of the many innovations it took to harness steam power. Among the most important were a new way to measure the energy output of engines and a micrometer dubbed the “Lord Chancellor,” able to gauge tiny distances. Such measuring tools, Rosen writes, allowed inventors to see if their incremental design changes led to the improvements — higher-quality parts, better performance, and less coal consumption — needed to build better engines. Innovations in steam power demonstrate a larger lesson: Without feedback from precise measurement, invention is “doomed to be rare and erratic.” With it, invention becomes “commonplace.