Technological advances such as low-cost data storage and computer processing, and networking and machine learning algorithms – commonly referred to as Big Data – have created opportunities to explore virtually any part of a business for fact-based intelligence.

Data mining is the process of using the power of such technologies to extract useful, non-trivial and preferably actionable knowledge from large bodies of data that are constantly recorded. This search relies on some predefined processes and it can be automated, but it can rarely be carried out effectively with just the push of a button.

Crisp-DM Diagram

Although data mining requires access to thousands of data records with a significant number of attributes to build relevant models, for most companies this is hardly the issue. Instead, the challenge often lies in carrying out the steps that provide the real value of data mining. Contrary to common perception, data mining is only partly about standardized process. It also requires creativity, quantitative skills and some domain knowledge; therefore, it can be quite challenging to find the right human resources to manage it from start to finish.

Cutting-edge companies make data mining a priority by putting in place not only the necessary technology, but also teams that include data scientists, subject matter experts and at least one project manager who can serve as a liaison between these two groups and translate the process from business problem to execution of a solution. Understanding the specific steps where your organization is lacking can be a very good indicator of the resource gaps that must be filled to take full advantage of the value that data mining can provide.

Image Source: Overbeck Analytics