At a recent conference I attended (Panagora MobileBiz Tech), there were (as usual) several deep-dive research presentations from reputable research firms. The research was (as usual) alternately exciting/boring/enlightening, with no less than a half-dozen different spins on the emerging prevalence of mobile devices. I was able to glean some great data points and apply a more objective lens to support/refute some of my current thinking on “what’s next” in our product roadmap.
What was most interesting, however, was a “counter-current” of commentary related to the trustworthiness of traditional, survey-based research data. More than one speaker directly challenged drawing any conclusions from research gathered through traditional surveys. Counter-survey-research-rebel Philip Graves, author of Consumerology, essentially refuted the idea that survey-based research is a valid way to learn what consumers want because it is intrinsically biased. Context and modality (when was the survey delivered? Where? Time of day/week/month? Screen?) all directly influence the results. Also, people will instinctively tend to answer from the perspective of the person they WANT to be rather than the person they are. Philip argued strongly that “behavior is the ONLY metric that matters.” Of course, Philips approach raised the ire of the traditional research folks in the room, whose livelihoods depend on this model of survey/report/repeat.
I liked what Philip had to say, and also tend to look suspiciously at data gathered via direct survey. However, I prefer to think of research data and behavioral data as two sides of the same coin. My analogy of choice is a patient in the hospital. There are two primary ways we know how the patient is doing: 1) metrics and 2) “survey” data. The metrics are gleaned from the tubes and such sticking on and into the patient that report a myriad of real-time data points (heart rate, BP, body temp, respiratory rate) and are tracked closely by hospital personnel. The “survey” data is gathered by asking the patient questions: “Where does it hurt? Did you sleep well? Have you been eating well? Taking your meds?” So I ask you now: which of these types of data are important and which one isn’t? It is the same for measuring the health of your product/business/idea/marketing results: you need both sides of the coin to tell a complete story and draw the most reliable conclusions.
Where do you fall in this argument: survey-based data vs. behavioral? Which is more important to you and why?
Ramblings of a Digital Product Geek 
I forgot how great of a writer your are Brad. I think both sides have merit But I know I have put stuff down that I think people may want to hear and been more critical based upon a bad experience as well instead of the product itself. Like don’t ask me to be objective about Wal-mart or Bank of America because I could not be.
LikeLike
Spike, great point – a predisposition (for or) against a brand may clearly cloud survey responses. Often the methodology attempts to “hide” the brand for that reason.
LikeLike