Why didn’t our product meet expectations?
Tens of billions of dollars are spent each year on software innovation (or research and development, R&D), but there is no significant relationship between spending on innovation and financial performance. This, of course, makes us ask a couple questions:
Why didn’t those dollars spent on R&D make a financial impact?
How do you know where to spend your R&D budget going forward?
Today I will investigate a sample scenario that is based upon many of my real world discussions with companies asking the questions above. The first thing I do is roll back to the start of the project or initiative and ask about expectations. If expectations were not set before beginning work, you cannot reasonably expect to meet or exceed expectations.
What was the quantifiable goal for the product? Revenue? Devices sold? Active users?
What data did you have to support that this goal could be reached by this product? Market research? Usability studies? Customer interviews?
How was this goal going to be measured? Analytics? Customer surveys? App Store reports? Downloads? Impressions?
Qualitatively, what did you hope to prove/disprove by running this project or process?
If you cannot answer the questions above, then take the time now to find the data or the person who can answer these questions. You do not want to move forward and spend money without an idea of your ROI. Hoping that your actions will result in an amazing, somehow unquantified results is exciting and unfortunately, a great way to waste time and money. I call this “smoking hopium”. Avoid the hopium. Stick to the facts.
If the product passed initial analysis and was approved to go forward, I begin looking at the processes that were used to ensure continued investment throughout development. After all, stopping a failing project or product at any time before the final release still saves your company money. The key is knowing when to kill a failed project without being too aggressive and missing an opportunity. Now I ask these questions:
Did someone collect the output from measurements, review the data, and make a decision to go forward at set milestones?
At what point did you begin to think this product was not going to succeed?
If everyone though the product was going to be successful until the end of the project, then what signs were missed?
Alternatively, if people thought that the project was misguided, how was this communicated? And more importantly, how was this received?
Expensive and highly visible innovation projects tend to gain a life of their own and run too long. The longer a project progresses, the more “sunk cost” factors into a decision to run one more iteration, one more sprint, or one more design change. You can help avoid this tendency to overrun project usefulness by focusing on the qualitative learnings. A failed project or product can be very valuable for your company:
What did your employees learn during this experience? Did they acquire new skills?
Did you highlight inefficiencies in your concept and development process?
Did you find a gap in your organizational expertise that needs to be filled?
Did you uncover a product vulnerability before your competition was able to exploit it?
As you can see, even a failed product can be a rich source of knowledge and lead to long term improvements. Don’t attempt to sweep your product failures under the rug - team members know that it failed, the rest of the company knows it failed, and you saw it happen. Looking back on how your product was released is a great opportunity to learn from what happened, and ensure that you are getting the most from your R&D dollars.