HULK — High-Volume Bulk RNA-seq Pipeline

HULK is a fully automated, containerized pipeline for bulk RNA-seq. It takes data either from SRA accessions or from local FASTQ files and produces gene-level quantification, QC reports, and R-based post-processing outputs (matrices and plots) with a single command.

Fetch / Import → QC/Trim → Quantify → Post-process → Report

Features

End-to-end automation
- Fetches SRA runs (prefetch) or imports local FASTQ files (FASTQ mode).
- Converts SRA accessions to FASTQ (fasterq-dump) in SRA mode.
- Performs QC and trimming with fastp.
- Quantifies transcript expression with kallisto.
- Generates per-project and global MultiQC reports.
Two input modes
- SRA mode: runs are driven by an SRA RunInfo table with Run and BioProject information.
- FASTQ mode: runs directly on user-provided FASTQ files grouped in a directory, sharing the same trimming, quantification and post-processing steps.
R-based post-processing
- Uses tximport to import quantification results into R.
- Uses DESeq2 (optional) to compute:
  - Normalized gene-level count matrices.
  - Variance-stabilizing transform (VST) expression matrices. (DESeq2 is currently run without metadata/design formulas.)
- Produces a comprehensive suite of QC plots:
  - Variance Heatmaps (clustering by gene variance).
  - Expression Heatmaps (global and targeted gene sets).
  - PCA Plots (Principal Component Analysis).
  - Sample-to-Sample Correlation Heatmaps (Pearson correlation for outlier detection).
  - DESeq2 Dispersion Plots (Model fit diagnostics).
- Saves both the figures and the corresponding matrix files for downstream reuse.
Gene-level expression focus
- Outputs are standardized on gene-level counts.
- When starting from transcript-level quantifications, HULK aggregates transcripts to genes using a tx2gene mapping file provided by the user.
- Gene-level matrices are designed to feed directly into downstream statistical and network analysis tools.
BioProject- and sample-aware outputs
- Per-BioProject plots and DESeq2 outputs in dedicated folders.
- Per-BioProject log files summarizing what happened for each project.
- Per-sample log files to simplify debugging of individual runs.
- A structured directory layout that groups results by BioProject and by sample.
Concurrency and performance
- Separates prefetch and processing workers in SRA mode:
  - A dedicated prefetch loop continuously downloads SRR data into a shared cache.
  - Processing workers operate only on already-prefetched data, reducing idle time and improving throughput.
- Parallelized Post-Processing:
  - Splits R analysis into a "Compute Phase" (matrix generation) and a "Visualization Phase" (plotting).
  - Uses concurrent workers to generate multiple plots simultaneously, significantly reducing runtime on large datasets.
Containerized & reproducible
- Distributed as a Docker image.
- No local installation of tools is required beyond Docker.
- Designed to be resumable and safe to re-run in partially processed directories.

Supported Platforms

Vendor / Family	Models
Illumina NovaSeq	6000, X, X Plus
Illumina HiSeq	X Ten, 4000, 3000, 2500, 2000, 1500, 1000
Illumina NextSeq	1000, 500, 550
BGI / MGI	DNBSEQ-G400, DNBSEQ-T7, BGISEQ-500, MGISEQ-2000RS
Legacy Illumina	Genome Analyzer II / IIX

Requirements

OS: Linux
Container: Docker installed and usable
Network (for SRA mode):
- To pull the image from Docker Hub.
- To access NCBI/ENA for prefetch / fasterq-dump.
Compute:
- Designed for servers/HPC; large datasets can be computationally intensive and long-running.

Installation

Option 1 — Official installer (recommended)

curl -fsSL [https://raw.githubusercontent.com/m13paiva/hulk/main/install_hulk.sh](https://raw.githubusercontent.com/m13paiva/hulk/main/install_hulk.sh) | bash

This script:

Pulls the latest Docker image m13paiva/hulk:latest.
Installs a wrapper at /usr/local/bin/hulk.
Allows you to run hulk directly on the host.

Uninstall wrapper:

sudo rm /usr/local/bin/hulk

Option 2 — Run via Docker (no install)

docker run --rm -v "$PWD":/data -w /data m13paiva/hulk:latest   -i SraRunInfo.csv -r transcripts.fa.gz -o results

Quickstart

SRA mode (RunInfo table)

# Minimal run using a transcriptome FASTA
hulk -i RunInfo.csv -r transcripts.fa.gz -o results

# Using a prebuilt kallisto index
hulk -i RunInfo.csv -r transcripts.idx -o results

In SRA mode, -i/--input must point to a RunInfo-style table (CSV/TSV/TXT) with at least the columns:

Run
BioProject
Model

The table is typically obtained from NCBI SRA (“Send to → File → RunInfo”). The Run values are SRR accessions, BioProject is used to group runs into projects, and Model describes the sequencer/instrument. The -r/--reference option takes either a transcriptome FASTA (.fa/.fasta[.gz]) or a prebuilt kallisto index (.idx).

FASTQ mode (local FASTQ directory)

# Local FASTQ directory with one subdirectory per sample
hulk -i fastq_runs/ -r transcripts.fa.gz -o results_fastq   --seq-tech "ILLUMINA NovaSeq 6000"

In FASTQ mode, -i/--input must point to a directory:

Each direct subdirectory under this directory is treated as one sample.
The sample ID is the name of the subdirectory.
Each sample directory must contain:
- One FASTQ/FASTQ.GZ file → treated as single-end, or
- Two FASTQ/FASTQ.GZ files → treated as paired-end.
Layout (single vs paired) is inferred automatically from the number of FASTQ files present.
Deeper nested subdirectories are ignored; only FASTQ files directly inside the sample folder are considered.

Because there is no RunInfo metadata in FASTQ mode, you must provide --seq-tech with the sequencing technology (e.g. ILLUMINA NovaSeq 6000). HULK uses this to select sensible fragment-length defaults for kallisto single-end quantification.

Input Table (SRA RunInfo)

To generate an SRA RunInfo table from NCBI:

Go to the SRA Search: https://www.ncbi.nlm.nih.gov/sra
Search by BioProject or paste SRA accessions (e.g. PRJNA1141930).
Select your runs of interest.
Click “Send to:” → “File” → “RunInfo”.
Download RunInfo.csv.

By default the RunInfo table already includes the columns requested by HULK:

Run
BioProject
Model

Example (RunInfo.csv):

Run,BioProject,Model, ...
SRR30141434,PRJNA1141930,ILLUMINA HiSeq 2500, ...
SRR30141435,PRJNA1141930,ILLUMINA HiSeq 2500, ...

FASTQ Mode

In FASTQ mode, HULK skips SRA download and runs directly on local FASTQ files. Instead of a RunInfo table, you provide a directory via -i / --input.

Directory layout

Let’s say you have:

fastq_runs/
├── SAMPLE_01/
│   ├── r1.fastq.gz
│   └── r2.fastq.gz
├── SAMPLE_02/
│   └── r.fastq.gz
├── SAMPLE_03/
│   ├── lane1/
│   │   └── something_else.txt
│   └── r1.fastq.gz
└── misc/
    └── not_a_sample.txt

fastq_runs/ is passed to HULK with -i fastq_runs/.
HULK will look only at direct subdirectories of fastq_runs/:
- SAMPLE_01/
- SAMPLE_02/
- SAMPLE_03/
- misc/
For each direct subdirectory:
- If it contains exactly one FASTQ/FASTQ.GZ file (e.g. r.fastq.gz):
  - The sample is treated as single-end.
- If it contains exactly two FASTQ/FASTQ.GZ files (e.g. r1.fastq.gz, r2.fastq.gz):
  - The sample is treated as paired-end.
- If it contains zero FASTQ/FASTQ.GZ files, or more than two, that directory is ignored.
The sample ID is taken from the name of the subdirectory (e.g. SAMPLE_01, SAMPLE_02, SAMPLE_03).
Any deeper nested subdirectories (like SAMPLE_03/lane1/) are not interpreted as separate samples. Only the FASTQ files directly inside the sample directory are considered.

Both uncompressed (.fastq) and compressed (.fastq.gz) files are supported.

Sequencing technology (`--seq-tech`)

In SRA mode, HULK can infer the sequencing technology from the Model column in the RunInfo table. In FASTQ mode, there is no such metadata, so the user must provide it:

--seq-tech TEXT A string describing the sequencing technology / instrument model, for example:
- "ILLUMINA NovaSeq 6000"
- "ILLUMINA HiSeq 2500"
- "DNBSEQ-G400"

HULK uses --seq-tech in FASTQ mode for Kallisto parametrization and for basic descriptive labeling. If your FASTQ directory mixes data from multiple sequencing technologies, it is recommended to run HULK separately for each technology, with the appropriate --seq-tech value each time.

CLI Reference

HULK uses a Click-based CLI with a main command and persistent configuration subcommands.

Subcommand options are stored in a small JSON file inside the container at /app/.hulk.json.

Main command — `hulk [OPTIONS]`

Required I/O
  -i, --input PATH
      SRA mode: RunInfo table (.csv, .tsv, or .txt) with columns: Run, BioProject, Model.
      FASTQ mode: directory containing sample subdirectories with FASTQ/FASTQ.GZ files.  [required]

  -r, --reference PATH
      Reference transcriptome (.fasta/.fa[.gz]) or kallisto index (.idx).  [required]

Outputs & performance
  -o, --output PATH
      Output directory.  [default: output]

  --min-threads INTEGER
      Minimum number of threads per SRR/sample.  [default: 4]

  -t, --max-threads INTEGER
      Maximum total threads.  [default: 10]

Behaviour flags
  --verbosity / --no-verbosity
      Enable/disable live progress bars and console messages.  [default: --verbosity]

  -y, --yes
      Assume 'yes' to prompts and run without asking.

  -f, --force, --overwrite
      Force re-run: overwrite totally/partially processed SRRs/samples.

  -n, --dry-run
      Validate inputs and configuration, print plan, and exit without running tools.

Quantification and post-processing
  -g, --gene-counts PATH
      Enable gene-level counts using a tx2gene (.csv) with columns 'transcript_id','gene_id'.
      When set, transcript-level quantifications are aggregated to genes automatically.

  --deseq2 / --no-deseq2
      Enable DESeq2 normalization + VST expression matrices for downstream plots/network exports.
      When disabled, kallisto quantifications still run but no DESeq2/VST matrices or DESeq2 plots
      are produced.  [default: --deseq2]

  --seq-tech TEXT
      Sequencing technology/platform (e.g. 'ILLUMINA NovaSeq 6000').
      Required in FASTQ mode; used to choose kallisto fragment-length parameters.

Caching and FASTQ retention
  --no-cache
      Disable SRA caching (do not allocate a shared cache volume).

  -c, --cache INTEGER
      Maximum SRA cache size (GiB).  Default: auto (300 GiB or based on free space).

  --keep-fastq
      Keep trimmed FASTQ files instead of deleting them after processing.

Misc
  -V, --version
      Show version and exit.

  -h, --help
      Show this message and exit.

If no subcommand is specified, running hulk with the options above executes the full pipeline.

Subcommand — `hulk trim [OPTIONS]` (fastp defaults)

Configure default fastp trimming parameters used by the main pipeline.

Options
  -ws, --window-size INTEGER
      fastp sliding window size.

  -mq, --mean-quality INTEGER
      fastp mean quality threshold.

  --reset-defaults
      Reset trim options to built-in defaults.

  -h, --help
      Show help and exit.

If neither --window-size nor --mean-quality is provided and --reset-defaults is not set, the command prints a short usage hint.

Subcommand — `hulk tximport [OPTIONS]` (aggregation/normalization)

Configure how tximport aggregates and normalizes quantifications in the R post-processing step.

Options
  -m, --mode [raw_counts|length_scaled_tpm|scaled_tpm|dtu_scaled_tpm]
      tximport aggregation/normalization mode.

  --ignore-tx-version
      Strip transcript version suffixes before matching (toggle stored as a boolean).

  --reset-defaults
      Reset tximport options to built-in defaults.

  -h, --help
      Show help and exit.

If no option is provided (and --reset-defaults is not set), the command prints a short usage hint.

Subcommand — `hulk align [OPTIONS]` (alignment/quantification backend)

Configure the alignment/quantification backend (currently kallisto) and its options.

Options
  --method [kallisto]
      Alignment/quantification backend.  [default: kallisto]

  -b, --bootstrap INTEGER
      Number of bootstrap samples for kallisto quant.  [default: 100]

  --reset-defaults
      Reset alignment options to built-in defaults.

  -h, --help
      Show help and exit.

Subcommand — `hulk plot [OPTIONS]` (plotting behaviour)

Configure which plots are requested in post-processing. Plotting only takes effect when DESeq2/VST is enabled in the main command (--deseq2).

Options
  --global-pca BOOL
      Enable/disable global PCA plot across all samples (true/false).  [default: true]

  --global-heatmap BOOL
      Enable/disable global expression heatmap across all samples (true/false).  [default: true]

  --global-var-heatmap BOOL
      Enable/disable global variance heatmap (gene x BioProject) (true/false).  [default: true]

  --sample-cor BOOL
      Enable/disable sample-to-sample correlation heatmap (Pearson). Useful for outlier detection. [default: true]

  --dispersion BOOL
      Enable/disable DESeq2 dispersion estimate plot (diagnostic). [default: true]

  --top-n INTEGER
      Number of most variable genes to use for Global Heatmap/PCA calculation. [default: 500]

  --bp-pca BOOL
      Enable/disable per-BioProject PCA plots (one PCA per BioProject) (true/false).  [default: false]

  --bp-heatmap BOOL
      Enable/disable per-BioProject expression heatmaps (one heatmap per BioProject) (true/false).  [default: false]

  --reset-defaults
      Reset plot options to built-in defaults.

  -h, --help
      Show help and exit.

Subcommand — `hulk report [OPTIONS]` (regenerate reports)

Scans an existing output directory and re-runs the post-processing step using the settings configured via hulk plot. Useful for regenerating plots, adding target gene lists, or recovering failed post-processing steps without re-running quantification.

Options
  -o, --output PATH
      Output directory to scan. [default: output]

  -g, --gene-counts PATH
      tx2gene map (required for aggregation).

  --target-genes PATH
      File(s) containing target genes (one gene per line). Can be used multiple times.

  --no-bp-postprocessing
      Skip per-BioProject analysis (only run Global).

  --fast
      Skip DESeq2 recalculation and only replot existing matrices (Fast Mode).
      Note: 'Dispersion' plots cannot be generated in --fast mode as they require model refitting.

  -h, --help
      Show help and exit.

Usage Examples

# 1) Basic SRA-based run with a transcriptome FASTA
hulk -i RunInfo.csv -r transcripts.fa.gz -o results

# 2) Using a prebuilt kallisto index
hulk -i RunInfo.csv -r transcripts.idx -o results

# 3) Force re-run of everything (including post-processing)
hulk -i RunInfo.csv -r transcripts.fa.gz -o results -f

# 4) Generate gene counts (requires tx2gene mapping)
hulk -i RunInfo.csv -r transcripts.fa.gz -g tx2gene.csv -o results

# 5) Dry run (validate configuration without running tools)
hulk -i RunInfo.csv -r transcripts.fa.gz -o results -n --verbosity

# 6) Configure fastp defaults and then run
hulk trim -ws 30 -mq 20
hulk -i RunInfo.csv -r transcripts.fa.gz -o results

# 7) Configure tximport defaults and then run
hulk tximport -m raw_counts --ignore-tx-version
hulk -i RunInfo.csv -r transcripts.fa.gz -g tx2gene.csv -o results

# 8) Configure kallisto bootstraps
hulk align -b 250
hulk -i RunInfo.csv -r transcripts.fa.gz -o results

# 9) Configure advanced plotting behaviour (Saved persistently)
hulk plot --global-pca true --sample-cor true --dispersion true --top-n 1000
hulk -i RunInfo.csv -r transcripts.fa.gz -o results

# 10) Regenerate reports for an existing run (uses saved plot settings)
hulk report -o results -g tx2gene.csv --target-genes my_pathway.txt

# 11) Run via Docker directly (no wrapper install)
docker run --rm -v "$PWD":/data -w /data m13paiva/hulk:latest   -i RunInfo.csv -r transcripts.fa.gz -o results --verbosity

# 12) FASTQ mode (local FASTQ directory)
# fastq_runs/ has one subdirectory per sample, each with 1 (SE) or 2 (PE) FASTQ/FASTQ.GZ files.
hulk -i fastq_runs/ -r transcripts.fa.gz -o results_fastq   --seq-tech "ILLUMINA NovaSeq 6000" --verbosity

Output Structure

The exact layout can vary slightly depending on the user's inputs, but a typical directory for multiple BioProjects looks like:

outdir/
├── PRJNA1141930/
│   ├── SRR30141434/
│   │   ├── fastp.json              ← fastp QC and trimming report 
│   │   ├── run_info.json           ← kallisto run metadata for this sample
│   │   ├── abundance.tsv           ← kallisto transcript-level quantification for this sample
│   │   └── SRR30141434.log         ← Sample-level log file
│   ├── SRRxxxxxxxx/                ← Additional samples (same structure)
│   ├── multiqc_PRJNA1141930.html   ← Project-level MultiQC report
│   ├── plots/                      ← Per-BioProject plots and their underlying matrices
│   ├── deseq2/                     ← DESeq2-normalized counts and VST matrices
│   ├── tximport/                   ← tximport summaries and intermediate artifacts
│   └── PRJNA1141930.log            ← BioProject-level log file
├── PRJNAxxxxxxx/                   ← Additional BioProjects (same structure)
└── shared/                         ← Shared (across all Bioprojects and Samples) output
    ├── plots/                      ← Global/cross-project plots
    │   ├── PCA.pdf                 
    │   ├── expression_heatmap.pdf  
    │   ├── variance_heatmap.pdf    
    │   ├── sample_correlation.pdf  ← (New) Sample-Sample Correlation
    │   └── deseq2_dispersion.pdf   ← (New) DESeq2 Dispersion
    ├── deseq2/                     ← Global DESeq2 outputs
    ├── tximport/                   ← Global tables from tximport
    ├── log.txt                     ← Global log file
    └── multiqc_global.html         ← Global MultiQC report

Tools used inside HULK

HULK orchestrates several third-party tools. If you use HULK in your research, please also consider citing the underlying tools where appropriate:

NCBI SRA Toolkit https://github.com/ncbi/sra-tools
Used for fetching and converting SRA data (prefetch, fasterq-dump) in SRA mode.
fastp https://github.com/OpenGene/fastp
Used for read QC and trimming.
kallisto https://github.com/pachterlab/kallisto
Used for pseudoalignment and transcript quantification.
MultiQC https://github.com/MultiQC/MultiQC
Used for aggregating QC metrics into per-project and global reports.
R and Bioconductor
- tximport — imports transcript-level quantifications and summarizes them at the gene level.
  https://github.com/thelovelab/tximport
- DESeq2 — computes normalized gene counts and VST matrices, and supports generation of diagnostic plots.
  https://github.com/thelovelab/DESeq2

License

Citation

If you use HULK in your research, please cite the software as:

Paiva de Almeida, M., & Barros, P. HULK: High-volume bulk RNA-seq pipeline. https://github.com/m13paiva/hulk

In addition to citing HULK, please also cite the underlying tools used in the pipeline (SRA Toolkit, fastp, kallisto, MultiQC, tximport, DESeq2) as appropriate.

Links

GitHub: https://github.com/m13paiva/hulk
Docker Hub: https://hub.docker.com/r/m13paiva/hulk

Name		Name	Last commit message	Last commit date
Latest commit History 43 Commits
.idea		.idea
app		app
.Rhistory		.Rhistory
.hulk.json		.hulk.json
CHANGELOG.md		CHANGELOG.md
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
environment.yml		environment.yml
install_hulk.sh		install_hulk.sh
multiqc_config.yaml		multiqc_config.yaml
pyproject.toml		pyproject.toml

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

HULK — High-Volume Bulk RNA-seq Pipeline

Table of Contents

Features

Supported Platforms

Requirements

Installation

Option 1 — Official installer (recommended)

Option 2 — Run via Docker (no install)

Quickstart

SRA mode (RunInfo table)

FASTQ mode (local FASTQ directory)

Input Table (SRA RunInfo)

FASTQ Mode

Directory layout

Sequencing technology (`--seq-tech`)

CLI Reference

Main command — `hulk [OPTIONS]`

Subcommand — `hulk trim [OPTIONS]` (fastp defaults)

Subcommand — `hulk tximport [OPTIONS]` (aggregation/normalization)

Subcommand — `hulk align [OPTIONS]` (alignment/quantification backend)

Subcommand — `hulk plot [OPTIONS]` (plotting behaviour)

Subcommand — `hulk report [OPTIONS]` (regenerate reports)

Usage Examples

Output Structure

Tools used inside HULK

License

Citation

Links

About

Uh oh!

Releases 5

Packages

Uh oh!

Contributors

Uh oh!

Languages

Folders and files

Latest commit

History

Repository files navigation

HULK — High-Volume Bulk RNA-seq Pipeline

Table of Contents

Features

Supported Platforms

Requirements

Installation

Option 1 — Official installer (recommended)

Option 2 — Run via Docker (no install)

Quickstart

SRA mode (RunInfo table)

FASTQ mode (local FASTQ directory)

Input Table (SRA RunInfo)

FASTQ Mode

Directory layout

Sequencing technology (--seq-tech)

CLI Reference

Main command — hulk [OPTIONS]

Subcommand — hulk trim [OPTIONS] (fastp defaults)

Subcommand — hulk tximport [OPTIONS] (aggregation/normalization)

Subcommand — hulk align [OPTIONS] (alignment/quantification backend)

Subcommand — hulk plot [OPTIONS] (plotting behaviour)

Subcommand — hulk report [OPTIONS] (regenerate reports)

Usage Examples

Output Structure

Tools used inside HULK

License

Citation

Links

About

Topics

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases 5

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Sequencing technology (`--seq-tech`)

Main command — `hulk [OPTIONS]`

Subcommand — `hulk trim [OPTIONS]` (fastp defaults)

Subcommand — `hulk tximport [OPTIONS]` (aggregation/normalization)

Subcommand — `hulk align [OPTIONS]` (alignment/quantification backend)

Subcommand — `hulk plot [OPTIONS]` (plotting behaviour)

Subcommand — `hulk report [OPTIONS]` (regenerate reports)

Packages