1. Why workflow managers exist (5 min)
2. The Nextflow core model (8 min)
3. DSL2 + channels (10 min)
4. hlatyping walkthrough (8 min)
5. Executors, containers, configuration (6 min)
6. nf-core, resume, observability (4 min)
7. Q&A (4 min)

Concrete > abstract. Every concept lands on a file you can grep.

What bash gives you

• Pipes between two programs
• A linear sequence of commands
• Manual control of files and intermediates

What bash does not give you

• Parallelism over samples
• Restart from the failed step
• Portability across local / HPC / cloud
• Version-pinned environments
• Provenance you can show a reviewer

# A typical sequencing pipeline, written as bash.
# Looks fine until sample 47 fails on a transient I/O blip.
for s in $(cat samples.txt); do
  fastqc reads/${s}_1.fq.gz reads/${s}_2.fq.gz -o qc/
  bwa mem -t 8 ref.fa reads/${s}_1.fq.gz reads/${s}_2.fq.gz \
    | samtools sort -@ 8 -o bam/${s}.bam -
  samtools index bam/${s}.bam
done
# Now: which sample is on which step? Which finished? Restart how?

// Identical work — main.nf. Run with: nextflow run main.nf -resume
process FASTQC {
    tag "$id"
    publishDir 'qc', mode: 'copy'

    input:
    tuple val(id), path(reads)

    output:
    path "*_fastqc.zip"

    script:
    "fastqc ${reads} -o ."
}

process BWA_MEM {
    tag "$id"
    cpus 8
    publishDir 'bam', mode: 'copy'

    input:
    tuple val(id), path(reads)
    path ref

    output:
    tuple val(id), path("${id}.bam"), path("${id}.bam.bai")

    script:
    """
    bwa mem -t ${task.cpus} ${ref} ${reads} \\
      | samtools sort -@ ${task.cpus} -o ${id}.bam -
    samtools index ${id}.bam
    """
}

workflow {
    reads_ch = Channel.fromFilePairs('reads/*_{1,2}.fq.gz')

    FASTQC(reads_ch)
    BWA_MEM(reads_ch, file('ref.fa'))
}

What the for-loop could not give you

• Parallelism is free. fromFilePairs fans out; every sample runs FASTQC and BWA_MEM concurrently. No for loop, no manual job bookkeeping.
• Resume from the failure. When sample 47 dies on an I/O blip, rerun with -resume: the 46 finished samples are cache hits, only 47 re-executes.
• Portable, unchanged. -profile docker | singularity | awsbatch runs this same code on a laptop, an HPC queue, or the cloud.
• Pinned and provenant. One container directive per process locks the tool version; every run emits execution_report.html and software_versions.yml.

You do not write the order things run in. You write what depends on what. Nextflow infers the DAG from that.

• Process = a unit of work (a script + its inputs and outputs).
• Channel = a typed, asynchronous queue of values.
• Workflow = wiring channels between processes.

workflow {
    reads_ch = Channel.fromFilePairs('data/*_{1,2}.fq.gz')

    FASTQC(reads_ch)        // runs on every pair, in parallel
    BWA_MEM(reads_ch)       // also runs in parallel
    MULTIQC(FASTQC.out.zip.collect())   // waits for ALL FASTQCs
}

Sample-level parallelism is free. collect() is the only synchronization barrier in the example above.

process OPTITYPE {
    tag        "$meta.id"
    label      "process_medium"
    container  'quay.io/biocontainers/optitype:1.3.5--hdfd78af_3'
    publishDir "${params.outdir}/optitype", mode: 'copy'

    input:
    tuple val(meta), path(bam), path(bai)

    output:
    tuple val(meta), path("*_result.tsv"),        emit: hla_type
    tuple val(meta), path("*_coverage_plot.pdf"), emit: coverage_plot
    path "versions.yml",                          emit: versions

    script:
    """
    OptiTypePipeline.py -i ${bam} \\
        --${meta.seq_type} -v -o ./ -p ${meta.id}
    """
}

Directives (top): runtime metadata.

• tag: human label per task
• label: resource bucket in conf/base.config
• container: pinned image, local and cloud
• publishDir: where outputs land in results/

I/O contract: tuples, paths, val, named emit:s.

Script: bash by default; can be Python, R, or anything with a shebang.

workflow NFCORE_HLATYPING {

    take:
    samplesheet  // channel: samplesheet read in from --input

    main:
    HLATYPING ( samplesheet )

    emit:
    multiqc_report = HLATYPING.out.multiqc_report
}

From main.nf in nf-core/hlatyping. take declares inputs; emit declares outputs. The body in main: is just like any workflow.

• take: turns a workflow into a reusable function with named inputs.
• emit: exposes named outputs that callers reference as WORKFLOW.out.name.
• Without these, a workflow is the top-level entrypoint (the unnamed workflow {...}).

Queue channel

• Asynchronous stream of N items.
• Consumed once; emits to downstream processes as items arrive.
• Natural fit for "one item per sample".

Channel.fromFilePairs('reads/*_{1,2}.fq.gz')
// emits one tuple per sample pair

Value channel

• Holds a single value that can be read by any downstream process arbitrarily many times.
• Natural fit for shared resources (reference genome, index, panel).

ch_ref = Channel.value(
    file("$projectDir/data/references/grch38.fa")
)
// every YARA_MAPPER task sees the same reference

Mismatched Cardinalities

• When joining or combining two queue channels, Nextflow pairs them sequentially.
• If ch_reads has 5 items and ch_index has 1 item, a process reading both runs only once.
• A value channel (or a singleton channel combined) will implicitly broadcast to all.

The Synchronization Barrier

• The .collect() operator gathers all elements from a queue channel and emits them as a single list (value channel).
• This forces a synchronization barrier: downstream processes wait for all upstream runs to finish.

// MultiQC runs once, waiting for all FastQC zips
MULTIQC ( FASTQC.out.zip.collect() )

// workflows/hlatyping.nf — verbatim
ch_all_fastq                      // [ meta, [ reads ] ]
    .cross(YARA_INDEX.out.index)  // [ [meta, [reads]], [meta, index] ]
    .multiMap { reads, index ->   // splits joined cross tuple
        reads: reads              // emitted as: [ meta, [ reads ] ]
        index: index              // emitted as: [ meta, index ]
    }
    .set { ch_mapping_input }

YARA_MAPPER(
    ch_mapping_input.reads,
    ch_mapping_input.index
)

Data Shapes Step-by-Step

1. ch_all_fastq: Emits [meta, [fastq_1, fastq_2]] per sample.
2. YARA_INDEX.out.index: Emits [meta, index_dir].
3. cross(): Aligns the elements by their first item (the meta map key). It outputs a nested tuple: [ [meta, [reads]], [meta, index] ].
4. multiMap(): Deconstructs the pair and routes them into separate channels, ensuring the reads and index streams flow to YARA_MAPPER in exact lockstep.

HLA in 30 seconds

• HLA = MHC in humans. Class I (HLA-A, -B, -C) presents intracellular peptides.
• Hyperpolymorphic: thousands of alleles per locus.
• Standard reporting is 4-digit (e.g. A*02:01), protein-level resolution.
• Used in transplant matching, neoantigen prediction, GWAS, pharmacogenomics.

Samplesheet (the only input contract)

sample,fastq_1,fastq_2,bam,seq_type
CONTROL_REP1,reads/c1_R1.fq.gz,reads/c1_R2.fq.gz,,dna
PATIENT_07,reads/p7_R1.fq.gz,reads/p7_R2.fq.gz,,rna
DONOR_A,,,reads/dA.bam,dna

Column rules: paired-end fills fastq_1 and fastq_2; BAM input goes in its own bam column with the FASTQ columns left empty. Only sample and seq_type are required; seq_type drives reference selection.

• Docker on workstations and managed cloud.
• Singularity / Apptainer on HPC (no root needed; works with shared filesystems).
• Conda as a last-resort fallback (slower, less reproducible).
• Each process declares its own container. No global runtime to maintain.

process OPTITYPE {
    container 'quay.io/biocontainers/optitype:1.3.5--hdfd78af_3'
    // ... rest of the process
}

// Or, with separate Docker/Singularity URIs:
process YARA_MAPPER {
    container "${ workflow.containerEngine == 'singularity' &&
        !task.ext.singularity_pull_docker_container ?
        'https://depot.galaxyproject.org/singularity/yara:1.0.2--he1c1bb9_4' :
        'quay.io/biocontainers/yara:1.0.2--he1c1bb9_4' }"
}

// conf/base.config (real hlatyping pattern)
process {
    cpus   = { 1   * task.attempt }
    memory = { 6.GB * task.attempt }
    time   = { 4.h * task.attempt }

    errorStrategy = { task.exitStatus in [137,143] ? 'retry' : 'finish' }
    maxRetries    = 2

    withLabel: process_medium {
        cpus   = { 6    * task.attempt }
        memory = { 36.GB * task.attempt }
    }
    withName: OPTITYPE {
        memory = { 16.GB * task.attempt }
    }
}

Profiles

• -profile docker — local Docker
• -profile singularity — HPC
• -profile conda — Conda fallback
• -profile test — tiny test data + Docker
• -profile test_full — full-size CI

hlatyping ships eight test profiles

test, test_fastq, test_fastq_cat, test_rna, test_dna_rna, test_hlahd, test_optitype_hlahd, test_full — each exercising a different input pattern.

What nf-core gives you

• ~100 curated end-to-end pipelines (rnaseq, sarek, ampliseq, hlatyping, …).
• ~1500 versioned, tested process modules in modules/nf-core/.
• A standard project template (the one hlatyping uses, v3.5.1).
• CI (nf-test, GitHub Actions, full-size AWS runs).
• The launch UI on Seqera Platform, driven by nextflow_schema.json.

The local-vs-nf-core split, in hlatyping

modules/
├── local/
│   ├── check_paired/          # is this BAM paired-end?
│   ├── hlahd/install/         # license-gated install
│   └── hlahd/genotype/        # license-gated typing
└── nf-core/
    ├── cat/fastq/
    ├── fastqc/
    ├── gunzip/
    ├── multiqc/
    ├── optitype/
    ├── samtools/collatefastq/
    ├── samtools/view/
    ├── yara/index/
    └── yara/mapper/

If a future pipeline needs Yara, it imports the same module. No copy-paste drift.

Built-in HTML reports

• pipeline_info/execution_report.html — CPU, memory, wall-clock per task.
• pipeline_info/execution_timeline.html — Gantt chart.
• pipeline_info/pipeline_dag.svg — the rendered DAG.
• pipeline_info/software_versions.yml — provenance, every tool, every version.
• multiqc/multiqc_report.html — domain-level QC + typing summary.

Seqera Platform (formerly Tower)

• Web UI to launch, monitor, and audit Nextflow runs.
• Real-time per-task metrics during a 200-sample run.
• Launch-form auto-generated from nextflow_schema.json.
• Free tier for personal / academic use.

OPTITYPE is the resource hotspot: modest CPU, but the ILP solve drives peak RSS. That single fact is why the SLURM profile gave it a withName: OPTITYPE memory override.

Start here

• nextflow.io — runtime and docs
• nf-co.re — community pipelines
• nf-co.re/hlatyping — the worked example
• training.nextflow.io — official tutorials
• nfcore.slack.com — community help

Papers

• Di Tommaso et al. Nextflow enables reproducible computational workflows. Nat Biotechnol 35, 316–319 (2017).
• Ewels et al. The nf-core framework for community-curated bioinformatics pipelines. Nat Biotechnol 38, 276–278 (2020).
• Szolek et al. OptiType: precision HLA typing from next-generation sequencing data. Bioinformatics 30, 3310–3316 (2014).
• Kawaguchi et al. HLA-HD: an accurate HLA typing algorithm for next-generation sequencing data. Hum Mutat 38, 788–797 (2017).