An excellent lecture on data mining with one very important required additional side note

Yesterday I came across the following lecture, “Statistical Aspects of Data Mining (Stats 202) Day 1,” from Google Tech Talks. I thought it did an excellent job explaining the concept and powers of data mining...

At approximately 42 minutes into the lecture the concept of what defines an anomaly is introduced. The statement is made that “generally an anomaly is an outlier: something that lies away from the rest of the data.” The lecturer does clarify this statement by saying that “in some space an anomaly is an outlier, but the key might be to find that space.”

What is crucial to understand is that the so-called “space” is created by the parameters set in the data mining collection and interpretation process.

As has been previously stated, anyone involved in the data mining process, from the data collector to the interpreter, can create anomalies at will, anywhere at any time, either deliberately or erroneously. It is relatively easy to change certain parameters to obtain the anomalies that are desired.

So a “space” as mentioned in the lecture is a relative term. This means that anything can be used to isolate data, may they be numbers or people. For example, it would be relatively easy for any government to create anomalies by setting parameters such as flagging people who attend church more than once a day - or people who pray five times a day - or even those who sign out certain books from a community library.

We should never forget that genocide has been committed numerous times by governments throughout history by isolating individuals and communities based on certain parameters.

So even though the above lecture makes data mining seem benign, in reality, data mining has been a harbinger of atrocities.

In science, the analysis of anomalies contributes to our understanding of the physical world, improving our lives. In contrast, identifying anomalies in our society based on political doctrine or religious beliefs has created fear and misunderstanding, restricting our lives. The lack of accountability from our leaders and our indifference to the consequences of their actions is diminishing our civil liberties. But it is not too late, we can prevent this from happening. We still have the ability to reclaim our future if we begin to educate ourselves.