The past few newsletters have had references to what I call "Product Validation." This refers to verifying that the product spec (PRD) is describing a product that you know will be successful, but doing so without actually building out and deploying the product.

This used to be a very expensive and difficult thing to do, and was generally only done for products that were very expensive to actually tool and manufacture, such as automobiles.

However, for just about every type of product today, the costs to produce effective prototypes or simulations has come down so far that I am amazed that I continue to encounter product teams that don't do this.

One of the biggest and most common mistakes product teams make is to have far more confidence in their product specifications than they should, and they move forward and think they'll adjust the product - if necessary - once they get beta feedback. But of course beta is far past the time for major changes, and it is little wonder so many initial product releases are so far from the mark.

As product manager, it is your responsibility to ensure that this doesn't happen to your product. The key to doing this is to prove to yourself and to the rest of the product team that the spec you give them describes a winning product. You can do this, and it costs far less than you probably think.

There are actually three important types of validation that you need to perform before you hand over a final product specification to the product team:

Feasibility Testing

One immediate question is whether or not the product is going to be buildable, with the technology available. Your engineers and architects should be very involved in investigating technologies and exploring possible approaches. Some paths will be dead-ends, but hopefully others will prove viable.

What is most important is that if there are obstacles the engineering team considers insurmountable in this product's timeframe, it is important to know this now rather than discovering this much later in the process after the time and money has been lost.

Some products have more technical risks than others, but if yours has significant risks regarding feasibility, make sure you address them early.

Usability Testing

Your product designers (UI/interaction designers) will be working very closely with you to come up with ways of presenting the functionality in the product so that the different types of users can figure out how to actually use the product.

Usability testing will often uncover missing product requirements, and also, if done well, identify product requirements that are not as necessary as originally thought. You should plan on multiple iterations before you come up with a successful user experience.

The purpose of the usability prototype is to have something to test on real people, and usability testing is the art and science of getting useful feedback on your product from your target customers. Certainly the product manager and designer will use the prototype and learn a great deal from it, but there is no substitute for putting the prototype in front of actual people from the target customer base.

Note that for usability testing purposes, it is perfectly fine if complicated back-end processing is simulated - the key is to evaluate the user experience.

Desirability Testing

Finally, it is not enough to know that your product is feasible to build and will be usable, but what really matters is whether or not your product is something users will want to buy - i.e. how much do users and customers like and value what you're doing?

This testing can typically be combined with the usability testing, and the prototypes used can generally be the same, but in usability testing you're seeing if users can figure out how to do the necessary tasks, while in desirability testing you're seeing if they actually care about those tasks and how well you solve them.

For a few small product efforts, simply working your ideas out on paper may be sufficient. But for most products, with complex user interactions or new uses of technology, these prototypes are absolutely critical in order to assess whether or not the product will meet its objectives.

The prototype may be a physical device, or it may be a quickly assembled version of a software product. The key is that it needs to be realistic enough that you can test the prototype on actual target customers and they can give you useful feedback.

Until recently there was debate over the relative merits of "high-fidelity" prototypes (what I'm describing), versus "low-fidelity" prototypes (essentially paper drawings). Today I consider this debate meaningless, because the cost of high-fidelity prototypes has dropped so low, and the quality of the feedback is so much higher.

In the past, there were two major obstacles to these types of prototyping. The lack of good prototyping tools meant that it could take a long time to actually construct the prototype. Another problem was in unenlightened management not understanding the difference between a prototype and the real product, and the teams would get pressured to use the prototype as the basis for the final product, with predictable results in the quality of the implementation.

Today, there are outstanding prototyping tools that can let engineers or designers rapidly create very functional prototypes (often in hours or days) that can effectively emulate the future product, to the degree necessary, and form the basis of realistic user testing. Moreover, most managers today understand that building a simulation and building the actual product are very different - akin to building a scale model of a house, and building the actual home.

These are not the only ways to validate your product - especially for internet services there are other techniques that are also easy and effective - but I can't emphasize enough how important and valuable it is to validate your ideas before you go and actually build the product. There are always surprises, and it is far better to discover them early rather than to wait until the product is in beta or released. Further, once the real engineering begins, a special type of inertia sets in, and it becomes very difficult to make significant changes and the costs of the changes rises dramatically.

Email to a friend

Sign up for the free newsletter here.

In the last newsletter we discussed the Agile Development Processes, and the implications for product managers. In this newsletter we look at the process that the majority of product teams still follow, which is known as the “Waterfall” process.

Even though the Waterfall development process is more than 30 years old, and even though it is often cursed by engineers and product managers alike, it is still by far the most common process used to create software products.

Note that it has long been unfashionable for a team to describe their product development process as “waterfall” yet in most cases that is essentially what is still being followed, albeit by many different names including Successive Refinement, SDLC, Phase-Gate, Stage Review, Staged Contracts, and Milestone-based.

In this article we'll try to capture the strengths as well as the key weaknesses of this approach, and discuss what the product manager can and should do in order to maximize the chance of success with this process.

General Principles

The conventional waterfall process is extremely simple in concept:

1) Phased Development – the model is that the software progresses through a well-defined series of phases beginning with a written description of the requirements, into first high-level architectural design, and then to low-level detailed design, then to code, then testing, and finally deployment.

2) Phase Review – each phase ends with a review of the deliverables from that phase, followed by sign-off and an explicit transition to the next phase.

The Waterfall method can be applied either very informally or very formally, as in the US Department of Defense Standard 2167A (and later 498), which describes in excruciating detail every step of the process and the many document deliverables.

Similarly, the waterfall method is also at the heart of the very informal and much more common scenario where someone from the marketing department gathers some market requirements and delivers them to engineering, where they come up with a schedule, and work on an architectural design and then some detailed designs for the more complicated areas, and then move into implementation and testing, often a beta, and finally deployment.

While we will soon discuss the most serious limitations of this approach, it is also useful to acknowledge the key traits that have kept this process in use for so long:

- Management appreciates the (perceived) predictability of the process. It is possible, although not common, to come up with fairly accurate schedules for even large and complex software projects. This assumes however that you completely and accurately understand the requirements and the technology, and that there will not be changes. With iterative approaches you don't really know how many iterations will be required, which can be disconcerting to management.

- There are deliverables throughout the process. Many people (managers, customers/clients, and even many engineers) are reassured by seeing well thought out and thorough documents and design diagrams. It helps these people to gauge progress towards the end, and also helps them feel better about the level of thinking that has gone into the project (even though there is no way to test whether or not the confidence is justified because unlike software you can’t execute paper documents). Many people make the mistake of feeling unjustifiably reassured by impressive specifications and documents.

Product Management Concerns

There are a number of well-known concerns with this process, especially from the product manager’s perspective:

- PM Issue: Validation Occurs Too Late in Process

The most costly issue is that there is no actual working software until nearly the end of the process, so there is little if any visibility into whether the software will be useful until after the majority of the investment has been made.

The product manager must ensure that prior to moving into the design and implementation phases, the product must be prototyped and tested on actual target users, so that the specification that is eventually provided to the product development organization is describing a product that has been successfully validated with the target audience.

Likewise, if there are major technical risks, these too should be explored and feasibility questions resolved (by the engineering organization) prior to beginning the actual architectural design and implementation. Before proceeding, the team needs to know that the product specification is something that can be successfully delivered.

- PM Issue: Changes are Costly and Disruptive

Any change to decisions from previous stages destabilizes the process and causes considerable pain and cost, as much work has to be reviewed and reworked. Moreover, the coding and testing process often uncovers issues in requirements and in architecture that can cause major delays and pain in this process.

The product manager must constantly represent the voice of the customer and user and there will be times when changes are required. It is important to point out that the cost of postponing the change needs to include the cost of the follow-on release to make the correction. There will still be times when it makes the most sense to defer the change until the next release, but in many cases it will be much less expensive to course correct sooner rather than later.

- PM Issue: Responding to the Market

This approach has a relatively high overhead in terms of documentation and process for moving through the phases. One consequence of this is that it can take considerable time to make even relatively small changes to the software.

This puts additional pressure on the product manager to ensure that they are providing a validated specification for a successful product in the first place, but it also means that the product manager will need to work with the product team to make course corrections after release as quickly as possible.

Summary

We have all seen the consequences of the Waterfall process in practice, and it's not hard to understand the motivation for alternatives such as the Agile methods like XP we discussed previously.

In many ways, the Waterfall process represents an idealistic but naïve view of the software development process, where people are able to anticipate the key issues and fully understand the requirements. When this is the case -- usually for very small projects -- this approach can provide a reasonable path to a quality implementation.

Unfortunately, this is rarely the case with product software. In practice, the consequence is that the product ships later than planned due to changes, and then expensive, time-consuming follow-on releases are required in order to correct issues once real users have a chance to see and use the actual software.

However, the product development process is often deeply entrenched in the product development organization, and the product manager must take steps to ensure that the potential problems are avoided. The most important key is to ensure the product spec is validated prior to moving to the implementation phase (remember to cover all three types of validation as described in "How To Write a Good PRD"), and you can save your team significant time and cost.

Email to a friend

Sign up for the free newsletter here.