I have a product manager friend who quit his job at a fairly well known startup recently. Over beers he told me the main reason he quit was simple: the creative director of the company wanted approval of all A/B tests. His assertion was that any changes that may result in users receiving a subtly worse experience for a day could damage the user’s perception of the application. Therefore he needed to personally approve and design any A/B tests, even against as little as 1% of traffic. The creative director was a cofounder; my product manager friend was not. My friend put up with it for a while, and then quit.
I mentally contrast this to a story Mark Zuckerberg told when I heard him speak in July. Mark told us about an internal tool he is proud of which allows an individual to test a new feature against a small slice of production traffic. The goal is to allow an individual to prove his or her way is better without having to convince anyone. Compare Mark’s approach with the approach of the creative director that drove my buddy to quit. Mark’s goal is to make the individual as effective as possible – in a word, to empower them. The creative director’s goal was to maintain as much personal control as possible.