The Best Product Teams Crave Truth and Do Math

A major threat to the success of any product is the unfounded feeling of certainty.

In a related post, The Behavioral Scientists’ Guide to Better Product Decisions, I share the biological and psychological reasons why humans make assumptions. We make assumptions when we are mostly uncertain. Overconfidence bias convinces us of the truth behind those assumptions. It takes effort to discover whether there is any truth to our assumptions. Yet our brains frequently don’t have the stamina to do the painstaking work required to discover whether our assumptions are valid.

I help product teams become even more effective by relying on Truth instead of their assumptions. In this post I describe how a product team can surface assumptions and turn them into testable hypotheses. The story is one of my favorite case studies of a product gone wrong: the PlayPump.

The PlayPump product development and go-to-market story plays out in 4 acts:

1. The Beautiful Vision
2. The Risky Assumptions
3. The Knowable Truth
4. The Missing Math

1. The Beautiful Vision

In 1989, an entrepreneur and former ad executive named Trevor Field discovered an invention at an agricultural fair in South Africa. It was a merry-go-round of sorts that also would pump water from the ground. Field was inspired to make a positive impact on the water crisis in Southern Africa. It was a noble cause. In rural areas, women and girls commute long distances (up to 8 km) and spend hours collecting water instead of attending school. UNICEF reports that each day more than 500 children die because they don’t have clean water.

Field spent several years improving the design, including adding a container above the PlayPump to store up to 2,500 liters of water. Based on his industry experience, he was confident that the storage container would host billboard ads. This advertising revenue stream would cover any maintenance of the PlayPumps.

The scenario playing out in Field’s head made perfect sense. The children would play on the merry-go-round, it would spin, and in doing so it would pump water from the ground into a tank. The women of the village wouldn’t need to trek long distances to draw water. Children (who have almost nothing) had something to entertain them.

Enamored with its potential and well-intentioned with his vision, Field brought the PlayPump to water-starved communities. Field installed PlayPumps with success at local schools, improving access to drinking water.

In 2006, a reporter from PBS Frontline, Amy Costello, covered the story of this revolutionary new product. That story led to an avalanche of positive media attention from the Case Foundation, Clinton Global Initiative and the US government, who collectively provided $60 million in funding. PlayPumps International formed. Emboldened by the positive attention and flush with funding, Field rushed to expand the build and installation of the PlayPump across southern Africa. PlayPumps International’s marketing materials stated its goal: 4,000 pumps will bring the benefits of clean drinking water to 10 million people in 10 countries by 2010.

Yet three years later, the $60 million was gone and people living in rural areas went months without reliable access to water.

What went wrong?

2. The Risky Assumptions

Field and his team made many assumptions in bringing the PlayPump to market. We can organize the major assumptions that would need to be true using the three pillars of product development: value, feasibility, usability.

We also need to understand the attributes of the incumbent, the conventional handpump. The PlayPump team intended to replace handpumps, improving upon their assumed weaknesses.

It is unclear whether the PlayPump team made any concerted effort to externalize their thinking by surfacing these assumptions. Based on the tragic ending to this story and third party reports, it seems unlikely.

What if we could rewind time and appoint a modern product leader to lead the PlayPump? Here’s what they could have done instead.

3. The Knowable Truth

Field and the PlayPump team were confident they would succeed. Celebrity endorsements, media coverage and funding by prestigious organizations propelled their enthusiasm. Field told Costello, “I’ve always dreamed big. I’m an optimist. I believe it can be done.” The team exhibited classic overconfidence bias, rushing to install the promised 4,000 PlayPumps.

Behavioral scientists recommend 4 techniques that parallel product discovery to counteract overconfidence bias:

• Consider: “How might we fail?” (Consider the opposite/Counterfactuals)
• Input from independent experts
• Track predictions vs actual results
• Humility

A modern product leader would have used these methods to discover in advance how likely the PlayPumps were to succeed. Before investing $60 million and 3 years to find out.

Consider: “How Might We Fail?”

A “Pre-Mortem” can shed light on the assumptions that if proven false will lead to failure.Behavioral scientistsrecommend teams working on high-risk projects consider how they might fail. Teams then prioritize their discovery to learn critical Truths soonest. Best practices to get the most value out of this exercise include:

• BYOA (Bring Your Own Assumptions). Generate assumptions individually to avoid group think that may obscure important details.
• Cross-functional Participation. Invite design, engineering, product and customer support to surface potential concerns throughout the value chain.
• Make a Map. Organize assumptions into themes and place on a matrix comparing risk and probability. Two templates I like are Laura Klein’s and David Bland’s. Upper right quadrant reveals the team’s riskiest assumption to (in)validate.

Independent Experts

Independent experts, with deep knowledge on the problem and no stake in the solution, can provide valuable perspective. Experts’ input helps the team avoid blindspots and overconfidence bias. Even better than experts is an unbiased algorithm. In the case of the Playpump, the team could have consulted at least three groups of experts.

• Rural families (users)
• Water-providing organizations (stakeholders)
• Prospective advertisers (suppliers)

An experienced product team would interview and observe these experts with questions and hypotheses. These hypotheses would help them discover the Truth about their users’ preferences and behaviors. Users could have been observed using the PlayPump vs the handpump. By forming key assumptions as hypotheses, the PlayPump team would have discovered their predictions were optimistic:

• Users: Children will play on the PlayPump 4 hours/day, generating water to fill the storage tank (2,500 liters). We will know this is true when we observe children playing for at least 4 hours at a sample of 10 installed PlayPumps and measure the water in the storage tank.
• Stakeholders: Communities are seeking alternatives to their handpumps. We will know this is true when we interview at least 3 water providing organizations who indicate this is a primary concern.
• Suppliers: Advertisers desire to reach rural communities. Their budget allocations will cover expected maintenance costs. We will know this is true when at least 10 advertising sponsors commit to cover maintenance costs of our existing 100 PlayPumps.

Track predictions vs actual results

Storytelling is a powerful persuasion technique. The PlayPump team’s story, kids happily playing and water flowing, fueled funding. The story also included predictions that funders believed would come true. Business leaders often use story to persuade teams and investors of the validity of their best-case scenario visions. Yet, they don’t encourage their teams to question the validity of these predictions with the same level of passion. They rarely write down those predictions. Behavioral scientist Richard Thaler recommends teams predict, record, and calibrate with actual results. The more they do so, the more accurate their predictions will become.

The PlayPump team’s many predictions were recorded in fundraising materials. Yet, they published no documentation of their actual results. Field and team could have compared the performance of the PlayPump to their competitor in various conditions (water availability, schools vs rural settings, adults vs children). UNICEF and SKAT each published reports where they did what the PlayPump team did not — compared actuals to predictions.

Humility

Humility is the foundation of effective product discovery. Great product teams don’t convince themselves of their inevitable success. They humbly seek quantifiable evidence to support objective customer choice. They know they would be foolish to pursue products driven by overconfidence bias. The cost of being wrong is too high.

Perhaps the most important attribute for a team’s success, humility is the hardest to measure. The team members keep one another in check. They share the range of possible outcomes, not just the most optimistic scenario. Their product leader often must remind stakeholders and leadership of the costs of being wrong to sustain support for their discovery efforts.

4. The Missing Math

My product teams know that I insist on modeling the math that needs to be true to achieve our desired outcomes. By seeing each number in the equation and its basis, we increase our probability of success. We can design experiments with quantifiable goals. We can instrument our products with KPI reporting at each point on the value chain. We can report progress and detect when a component is underperforming.

In the case of the PlayPump, the beautiful vision used for fundraising obscures the outcome math.

4,000 pumps will bring the benefits of clean drinking water to 10 million people in 10 countries by 2010

Let’s breakdown the math related to our 3 hypotheses:

• The Play = Production equation
• The PlayPump > Handpump equation
• The Advertising > Maintenance costs equation

The Play = Production equation

4,000 pumps will bring the benefits of clean drinking water to 10 million people

The math equation that would have needed to be true to support the stated goal of meeting the daily requirements of 10 million people is:

Reach: 10 million * Impact: Daily water requirements (~20 liters/day) =

Outcome: 200 million liters of water delivered via 4,000 playpumps

On per Playpump basis, the goal was:

Reach: 2500 people (10 million / 4,000 pumps) * Impact: Daily water requirements (~20 liters/day) =

Outcome: 50,000 liters of water delivered to surrounding community

Here’s the math that would need to be true to generate 50,000 liters water per pump, assuming the water supply is available.

Hours of rotation * Water produced per hour = Water supply

This is the equation with placeholder math that most needed to be validated.

How many hours of use would the Playpump be used each day? How much water would that use yield?

When about 100 PlayPumps were installed, UNICEF and others asked SKAT to test their efficacy. PlayPumps Int’l projected that the pumps would yield about 1400 liters water/hour (370 gallons). SKAT found that the pumps (of varying sizes) averaged about half that at 785 liters of water/hour.

The new per pump math looks like this:

50,000 liters of desired water / 785 liters/hour = 63.7 hours of use required per day!

Using the PlayPump team’s advertised estimate of 1400 liters/hour, each pump would need to spin 35 hours/day to meet the water needs of 2500 people.

With this Truth and Math, the PlayPump team could have made several course-changing decisions. Reset the goal and expectations. Improve the productivity of the pump. Change the solution design.

Instead they continued on the same path to manufacture and install at least 1800 PlayPumps.

The PlayPump > Handpump equation

Field was confident from the outset that the PlayPump was better than the handpump. He demonstrated that it was difficult and tiring to use. He expected that children’s desire and capacity to play would produce enough water to eliminate the need for handpumps. We can illustrate that assumption in an equation:

PlayPump water production > Handpump water production

The SKAT report provides the math to compare the performance of the largest (heaviest to operate) to smallest (easiest for small children) PlayPumps. When a PlayPump replaced a handpump, the smallest size (50mm) was used as it was installed on the existing borehole.

The PlayPump produced 30% as much water as the handpump in the same location. Another equation proved false.

Advertising > Maintenance Costs Equation

As stated earlier, the $60 million in funding was used exclusively for manufacturing and installation. Field’s initial expectation was that advertising revenue would cover maintenance. He even guaranteed the PlayPumps for 10 years. Yet UNICEF found 75% of the PlayPump storage tanks visited contained no advertising. Their report indicated (in Zambia) 75% of the tanks contained no contact details for repairs. The SKAT report found that for non-operational pumps, the average downtime was more than 100 days.

The SKAT report provides the math to support the maintenance crews responsible for 100 pumps in an area. Roughly $20k annually.

Here’s math that could have surfaced risky assumptions in this ad-revenue supported maintenance funding.

(4,000 pumps / 100 pumps) * $20k/year = $8,000,000 in annual ad revenue or per pump: $2k in ad revenue.

With this break-even maintenance math, Field could have gauged advertisers’ willingness to pay either $166/month or a $800 CPM ($2000/2500/1000) per pump. Based on the lack of advertising sponsorship, this equation was proven false.

The moral of this tragic product tale

In Frontline’s follow-up story “Troubled Water,” Costello reveals the true impact of the PlayPump team’s risky assumptions. Three years after the initial story, she finds PlayPumps sitting idle or in disrepair. Women, not children, work hard to operate them. Little water flows from the tap nor resides in the tanks. Storage tanks are bare of advertising. No smiling happy children spinning on the PlayPump as the original story portrayed. Communities preferred the original solution: the easy to use and maintain handpump. Did I mention they were 1/3 of the cost of the PlayPump?

All the truth that came true was knowable. The math that didn’t add up was knowable. Overconfidence bias fueled their pursuit. Field wasn’t humble enough to consider where he might go wrong, nor experienced enough in product development to anticipate points of failure.

Product teams need to seek the truth and do the math to achieve their desired outcomes. Any other approach is childish.