Intuition and Data Analysis

The analysis of data is now, and always has been, problematic. We are not machines. Our thinking is affected by intuition and experience, which are not empirical in nature. In business, Six Sigma or not, the ability to see information from both an empirical perspective and from the perspective of human stake holders (not empirical), is critical to quality decision making.

Let me give you an example of a non-empirical, intuitive/experiential perspective. If you have ever returned to a playground that you knew when you were young, you may have remembered the sliding board being really high, but now it seems small.  It did not shrink. Your perspective changed.  This is a fundamental rule of human thought. We learn through experiment (experience). These learnings change the way we view the world. In other words, the context of our information changes.

This is both good and bad.  We learn not to put our hand on a hot stove, to look both ways before crossing the street and to not insert keys into electrical wall sockets, because of severe negative consequences. This is the result of the power of observation.

This works well on simple systems where results are not ambiguous, and are easy to understand and predict. On systems where there is complexity and results are not easy to predict, you must “peel the onion” with your observations. When evaluating a complex system, intuition must be used very carefully.

Dr. Daryl Bren does a magic trick with his students on their last day of class with him.  The magic trick is really a lesson. He attempts to demonstrate his ability to read a student’s mind by way of giving information about their personal history that he could not otherwise know. He is always successful and his students grapple with what to think about this intriguing skill. Their intuition, based on trust in their professor, compels them to want to believe. Once he has given them enough time, he tells them how he accomplished the feat. Basically he knew who he was going to” read” and collaborated with their family ahead of time, without the student knowing. It is a trick, not magic. He then delivers the punch line, never substitute your intuition for real data.

Another story.  Several years ago I had an employee that told me that he had started a rumor about the possibility of a major management shakeup. Two weeks later he came to my office, excited, saying that he had it on good source that there was going to be a major management shakeup. He even had details (facts?), which his intuition bought into.. I had to remind him that he, in fact, started that rumor two weeks earlier and fell victim to it. Intuition, in the absence of fact, will nearly always lead to incorrect conclusions.

This is not to say that intuition is not important. Intuition is a critical evaluation tool, and just like any other tool, must be used properly. Intuition can indicate that either your perspective or the data is skewed in some way.  Maybe both are skewed.  Intuition will point to what needs a reality check or more information.

This is just another case where balance and perspective play important roles in our lives. In reality, what I am talking about is finding the “why” behind a set of data or “facts”. Successful Six Sigma Projects and quality business decisions depend on it.