The Workflow Description Language (WDL)¶
The Workflow Description Language (WDL) which is in use at the Broad Institute, is an alternative to CWL. It is supported by a powerful workflow execution engine called Cromwell, which includes multiple “backends” such as GridEngine, HTCondor, Spark, the Google “Pipelines API” (formerly known as JES, ie “Job Execution Service”, aka GGP), and the new GA4GH TES (ie “Task Execution Service”).
We will focus on the two backends that directly support running workflows on Google Compute Engine VMs:
- Google Genomics Pipelines API (formerly known as JES)
- GA4GH Task Execution Service (aka TES)
Google Genomics Pipelines¶
The so-called “Pipelines API” is a task runner that lets you run a command-line executable in Docker on a Google Compute Engine VM. Since it is truly a “task” runner rather than a full “pipeline” runner, we generally refer to it as GGP so that the usage of the word “pipeline” is not confusing. We also find the additional term “API” unnecessary.
GGP can be “called” using command-line interface (part of the Cloud SDK gcloud tool),
or as a web service API that can be called programmatically.
When using GGP from the command-line, each task is defined in a YAML (or JSON) file.
The Google documentation for the “Genomics Pipelines” can be found here and on readthedocs, and there are numerous easy-to-follow examples on github here.
You can use wdl_runner to run a WDL workflow using Cromwell+GGP on the Google Cloud. Documentation can be found on github and you can run a GATK workflow by following this Google Genomics documentation.
GA4GH Task Execution Service¶
The GA4GH TES was inspired by GGP, with the broader goal of defining a platform agnostic interface between workflow management systems, schedulers, and workflow language interpreters on the frontend of the TES interface, and the actual workes, nodes, VMs, and filesystems on the backend. Although this effort started only a few months ago, progress has been rapid and a reference implementation is available on github.
As described in this post over on the WDL Blog, TES has been recently added as a new backend to Cromwell.