Submitting a Nextflow Pipeline to Seqera

Preliminary Steps

Before logging in to Seqera for the day and submitting a pipeline or viewing a pipeline's status, you'll need to do the following in order to avoid constant Duo requests when using the Seqera Platform Cloud.

Navigate to https://orcd-ood.mit.edu and login with your MIT account. After logging in, you'll be greeted with the following page.

This is the landing page for the Office of Research Computing and Data's OnDemand, a system for easily accessing the Engaging cluster's services from your web browser. For more information about Engaing OnDemand, please refer to the official ORCD documentation: https://orcd-docs.mit.edu/.

From this landing page, select Engaging > Engaging Shell Access.

This will open a shell in your web browser and log you in to Engaging.

An added bonus of doing this is that, for the next 24 hours, you will not be prompted with Duo requests when logging in to Engaging over SSH.

For this to work, It's important that you use the newer https://orcd-ood.mit.edu and not ORCD's older OnDemand system, https://engaging-ood.mit.edu.

ORCD will soon transition to their newer OnDemand system. In the meantime, their documentation still points to the older version.

Submitting an existing pipeline

The Seqera Launchpad will contain versioned pipelines that are configured with a set of default options to get them running easily on the Engaging cluster. These options include setting the pipeline revision number, setting the appropriate Slurm partition to submit the job to, and setting the correct Seqera work directory.

To launch a pipeline with the default set of configurations, go to the Launchpad and press the "Launch" button on the right side of the pipeline.

Upon launching a pipeline, you will be navigated to a page where you can set the run parameters for the pipeline. For example, when launching the rnaseq-3.21.0 pipeline, you will be navigated to the following page:

These run parameters correspond to the rnaseq 3.21.0 pipeline parameters. Fill them out with the appropriate values for your pipeline.

For parameters that ask for a path to a file or set of files, use the absolute path to the data on the Engaging cluster. To get the absolute path, login to Engaging, traverse to the path, and run pwd -P.

After filling out the run parameters, you can press the "Launch" button to launch the pipeline. Once the pipeline is launched, Seqera will attempt to submit it to the Engaging cluster. If you have not followed the Preliminary Steps section, submitting a pipeline may trigger 2 or 3 Duo requests. You'll need to accept those for the pipeline to successfully be submitted to the cluster.

Once your job has been successfully submitted, an entry for it will appear on the Runs page. Clicking on the entry will take you to the run monitoring page. Here, you will see the current status of the job, along with any progress. Each tab will contain different information about the job, such as what tasks are currently running, logs for the job, generated reports, etc.

For example, here is the run monitoring page for a pipeline submitted to Engaging through Seqera. The run ran until 100% completion.

Submitting a custom pipeline

If you want to submit a Nextflow pipeline that does not currently exist on the Launchpad, you can do so by clicking the "launch a run without configuration" link on the main Launchpad page:

This will take you to the Quick Pipeline Launch page, where you can specify the pipeline to launch. There, you should type in the git repository for your pipeline. Once doing so, Seqera will query information about the pipeline, such as what revisions are available, what profiles are available, etc.

For example, to run the chipseq 2.1.0 pipeline from the Quick Pipeline Launch page, put in https://github.com/nf-core/chipseq/ as the "Pipeline to launch" and select 2.1.0 from the "Revision" drop down.

From there, change the rest of the "General Config" to your desired values. From "Compute Environment", you can choose what Slurm partition on Engaging to submit the job to.

Please, always add apptainer to the list of "Config profiles" and don't change the "Work directory" from its default value.

Once you've filled out the "General Config", you can click "Next" to go to the "Run parameters" page. The rest of the steps will be identical to Submitting an existing pipeline.

If your run completes successfully, you can save it as a pipeline from the "Runs" page: