The importance of expert knowledge in big data and machine learning

Abstract

Abstract According to popular belief, big data and machine learning provide a wholly novel approach to science that has the potential to revolutionise scientific progress and will ultimately lead to the ‘end of theory’. Proponents of this view argue that advanced algorithms are able to mine vast amounts of data relating to a given problem without any prior knowledge and that we do not need to concern ourselves with causality, as correlation is sufficient for handling complex issues. Consequently, the human contribution to scientific progress is deemed to be non-essential and replaceable. We, however, following the position most commonly represented in the philosophy of science, argue that the need for human expertise remains. Based on an analysis of big data and machine learning methods in two case studies—skin cancer detection and protein folding—we show that expert knowledge is essential and inherent in the application of these methods. Drawing on this analysis, we establish a classification of the different kinds of expert knowledge that are involved in the application of big data and machine learning in scientific contexts. We address the ramifications of a human-driven expert knowledge approach to big data and machine learning for scientific practice and the discussion about the role of theory. Finally, we show that the ways in which big data and machine learning both influence and are influenced by scientific methodology involve continuous conceptual shifts rather than a rigid paradigm change.