Intuition and Data Analysis

The analysis of data is now, and always has been, problematic. We are not machines. Our thinking is affected by intuition and experience, which are not empirical in nature. In business, Six Sigma or not, the ability to see information from both an empirical perspective and from the perspective of human stake holders (not empirical), is critical to quality decision making.

Let me give you an example of a non-empirical, intuitive/experiential perspective. If you have ever returned to a playground that you knew when you were young, you may have remembered the sliding board being really high, but now it seems small.  It did not shrink. Your perspective changed.  This is a fundamental rule of human thought. We learn through experiment (experience). These learnings change the way we view the world. In other words, the context of our information changes.

This is both good and bad.  We learn not to put our hand on a hot stove, to look both ways before crossing the street and to not insert keys into electrical wall sockets, because of severe negative consequences. This is the result of the power of observation.

This works well on simple systems where results are not ambiguous, and are easy to understand and predict. On systems where there is complexity and results are not easy to predict, you must “peel the onion” with your observations. When evaluating a complex system, intuition must be used very carefully.

Dr. Daryl Bren does a magic trick with his students on their last day of class with him.  The magic trick is really a lesson. He attempts to demonstrate his ability to read a student’s mind by way of giving information about their personal history that he could not otherwise know. He is always successful and his students grapple with what to think about this intriguing skill. Their intuition, based on trust in their professor, compels them to want to believe. Once he has given them enough time, he tells them how he accomplished the feat. Basically he knew who he was going to” read” and collaborated with their family ahead of time, without the student knowing. It is a trick, not magic. He then delivers the punch line, never substitute your intuition for real data.

Another story.  Several years ago I had an employee that told me that he had started a rumor about the possibility of a major management shakeup. Two weeks later he came to my office, excited, saying that he had it on good source that there was going to be a major management shakeup. He even had details (facts?), which his intuition bought into.. I had to remind him that he, in fact, started that rumor two weeks earlier and fell victim to it. Intuition, in the absence of fact, will nearly always lead to incorrect conclusions.

This is not to say that intuition is not important. Intuition is a critical evaluation tool, and just like any other tool, must be used properly. Intuition can indicate that either your perspective or the data is skewed in some way.  Maybe both are skewed.  Intuition will point to what needs a reality check or more information.

This is just another case where balance and perspective play important roles in our lives. In reality, what I am talking about is finding the “why” behind a set of data or “facts”. Successful Six Sigma Projects and quality business decisions depend on it.

If you Aren’t Measuring It You Aren’t Managing It

A favorite axiom in management is, “If you aren’t measuring it, you aren’t managing it”. Just as driving a car with your eyes closed will result in disaster, running a business without some sort of performance feedback will result in business disaster.

The collection and use of data is important because things are rarely what they seem to be. Data helps us separate what is really happening from what we think is happening (or what we want to be happening). When we make decisions based on how things feel or how they have always been, we are operating in the “as we think it is” world. This is a prescription for disaster. The successful business operates in the real world. We call this the “as-is” world.

The measure phase of a Six Sigma process improvement project focuses on characterizing the current performance of a business process, which is the current reality. In this phase, the Six Sigma project team is trying to accomplish two things. First is to establish an “as-is” performance measurement for the process. Second, is to use the data to begin looking for potential causes of defects.

Some of the important activities of the Measure phase are:

  • Developing a data collection plan and following it
  • Performing a measurement system analysis
  • Calculating performance indicators for the process from the data collected
  • Control charting

The objective is to measure the process’ impact on the customer’s CTQ (Critical to Quality) issues. The result is the characterization of the process’ performance from the customer’s perspective. This becomes the process’ story in the “as-is” world.