Run Workflows Across Nodes¶
This page shows how to run a Snakemake workflow on the TES services of
several Nodes, with the demonstrator demo-tes-hybrid-cloud. The workflow
splits into steps that a GA4GH Task Execution Service (TES) runs
as tasks. The demonstrator runs it on each TES service in turn, and then on
all of them together through the proTES
gateway, which forwards each task to one of the TES services.
Under construction
The demonstrator dates from 2023 and was not updated for the current
products. Its README lists the versions that it was built for: for
example, the proTES image elixircloud/protes:20230218. proTES 0.4.0
configures its TES services differently (custom.tes.backends instead
of tes.service_list: see the upgrade
steps). There is no
public deployment with a guaranteed uptime, either. Until both are
documented, deploy the services yourself or write to
cloud-service@elixir-europe.org.
What you need¶
- Git, Conda or Mamba, and JupyterLab (the README lists all requirements).
- The addresses of one or more TES services (see Start Here), and the credentials of the Funnel services among them.
- The address of a proTES gateway in front of these services.
- Input files that the TES services can download from a URL.
Steps¶
- Clone the repository and install the
requirements as its README describes. This includes a
py-tesversion from a specific commit. - Create three files in the repository, as the section "Create
configuration files" of the README shows:
.tes_instances: the name and the URL of each TES service, one per line;.inputs: the location and the URL of each input file, and one lineworkflowwith the input of the workflow;.env:FUNNEL_SERVER_USERandFUNNEL_SERVER_PASSWORDfor Funnel services, andTES_GATEWAY, the address of the proTES gateway. It holds passwords: never commit or share it, and make only you able to read it (chmod 600 .env).
-
Start the notebook:
-
Run the cells of the notebook in order. It lists the TES services, runs the workflow on each of them, and then runs it through the gateway, which forwards each task to one of the TES services.
The notebook has the details of each step. To follow the same idea with CWL workflows, see the CWL demonstrator.
Use your own workflow¶
Snakemake, Nextflow and cwl-tes can send the tasks of any workflow to a TES
service. See Run workflows through
TES.