Learning is better than optimization (the local maximum problem)

Lean startups don’t optimize. At least, not in the traditional sense of trying to squeeze every tenth of a point out of a conversion metric or landing page. Instead, we try to accelerate with respect to validated learning about customers.

For example, I’m a big believer in split-testing. Many optimizers are in favor of split-testing, too: direct marketers, landing page and SEO experts — heck even the Google Website Optimizer team. But our interest in the tactic of split-testing is only superficially similar.

Take the infamous “41 shades of blue” split-test. I understand and respect why optimizers want to do tests like that. There are often counter-intuitive changes in customer behavior that depend on little details. In fact, the curse of product development is that sometimes small things make a huge difference and sometimes huge things make no difference. Split-testing is great for figuring out which is which.

Another excellent post by Eric Ries about how there is a need to balance metric testing with reasonable design sense to overcome getting stuck in a particular direction.