Functional Big-Data Genomics

ashish — Tue, 11 Sep 2012 20:11:51 +0000

Abstract
High-throughput genomic sequencing is characterized by large diverse datasets and numerous analysis methods. It is normal for an individual bioinformatician to work with thousands of data files and employ hundreds of distinct computations during the course of a single project. This problem is magnified in “core facilities”, which support multiple researchers working on diverse projects. Most investigators use ad hoc methods to manage this complexity with dire consequences: analyses frequently fail to meet the scientific mandate of reproducibility; improved analysis methods are often not considered because recomputing all downstream steps would be overwhelming; hard drives and CPUs are used sub-optimally; and, in some cases, raw data is lost.

We describe HITSCORE, an OCaml software stack that operates all computational aspects of the Genomics Core Facility at New York University’s Center for Genomics and Systems Biology. HITSCORE has been in production use for one year, and was implemented quickly by less than two programmers following design advice from several biologists. A simple domain specific language (DSL) enables generating type safe database bindings and GUI components, and greatly eases updates and migration of our data model. We found a multi-lingual stack too burdensome in a small team setting, and credit OCaml for fulfilling the needs of our full application stack. It has good database bindings, excels at encoding complex domain logic, and now allows construction of rich websites due to the Ocsigen web programming framework. Higher level libraries for distributed computing would be a welcome improvement.

The opportunity to build this system did not stem directly from any strength of functional programming or OCaml. It was necessary for a person with credibility amongst biologists to champion its development, and this credibility took several years to build. Rapid development appears to be the single most important factor in allaying doubts about using a lesser known language, and we will briefly describe our experiences in bringing OCaml to several high profile projects.

Download slides
Video

Citation
Ashish Agarwal, Sebastien Mondet, Paul Scheid, Aviv Madar, Richard Bonneau, Jane Carlton, Kristin C. Gunsalus. Functional Big-Data Genomics. Commercial Users of Functional Programming 2012, Copenhagen, Denmark, Sep 15, 2012.

A Domain Specific Language Stack for Bio HPC

ashish — Fri, 11 May 2012 22:42:07 +0000

We have given several presentations on our DSL approach to the management and analysis of big-data in the field of Biology. Now, with Karl Ward, we have been extending this approach to the systems layer, enabling more robust management and configuration of the hardware and software infrastructure so critical to bioinformatics. Many thanks to Efstratios (Stratos) Efstathiadis for giving us an opportunity to present this work at the first Bio HPC Workshop at NYU’s Langone Medical Center.

Karl Ward, Sebastien Mondet, Ashish Agarwal. A Domain Specific Language Stack for Bio HPC, First Workshop on High Performance Computing for Biomedical Research, Center for Health Informatics and Bioinformatics, New York University Langone Medical Center, May 2012.

Shonan Meeting

ashish — Wed, 09 May 2012 19:11:38 +0000

I’ll be back in Tokyo, this time for the Shonan Meeting on Bridging the Theory of Staged Programming Languages and the Practice of High-Performance Computing. My talk will be on BINQ, a domain-specific-language for genomic computations.

Ashish Agarwal » High Performance Computing

Functional Big-Data Genomics

A Domain Specific Language Stack for Bio HPC

Shonan Meeting