Spark-SPELL: Low-latency query-based search for gene expression compendia on cluster computers

Abstract

Exploratory analyses are vital to fully realize the potential for scientific discoveries in large-scale biomedical data compendia. Specifically, most biomedical data analyses require a human expert to interactively explore the data to find novel hypotheses or conclusions. However, recent developments in biotechnology instruments are generating Tera-scale datasets. No interactive biomedical data analysis systems scale to such large datasets. We present the design, implementation and optimization of the SPELL biomedical search algorithm on the Spark framework. We demonstrate the scalability and interactive performance of our Spark-SPELL system. In addition, we demonstrate the performance improvements of our optimizations to the SPELL algorithm and the Spark framework.