QIIME 2 Master Learning Lab

The #1 most common error in 16S paired-end pipelines is losing 70% to 100% of sequencing reads during DADA2 merging. This occurs when users truncate reads too aggressively at the 3' end, leaving less than the required 12-20 bp overlap between forward and reverse reads.

When reads cannot merge, DADA2 silently discards them, resulting in empty or near-empty feature tables!

• Target Amplicon Length: Total biological base pairs between primers (e.g. 253 bp for V4 515F-806R, ~460 bp for V3-V4).
• Forward / Reverse Read Length: The sequencing run cycle length (e.g. 250 bp on MiSeq 2x250).
• Truncation Formula:

  Effective Overlap = (trunc-len-f - trim-left-f) + (trunc-len-r - trim-left-r) - amplicon_len

• Safety Rule: Ensure Effective Overlap is ≥ 20 bp. If < 12 bp, DADA2 cannot merge reads.

1. Step 1: Inspect your demux-summary.qzv quality score boxplots to find where median quality drops below Q30.
2. Step 2: Test your candidate truncation values in this calculator to confirm positive overlap ≥ 20 bp.
3. Step 3: Input those exact values into Section 5 (qiime dada2 denoise-paired) via --p-trunc-len-f and --p-trunc-len-r.

Microbial diversity metrics (richness, Shannon, UniFrac) are highly sensitive to sequencing library size. A sample sequenced to 50,000 reads will artificially appear to contain far more rare species than one with 3,000 reads simply due to sequencing effort.

Rarefaction standardizes all samples by subsampling reads without replacement to an even depth (--p-sampling-depth). Any samples with fewer reads than this cutoff are completely dropped from analysis.

• Too Low Depth: Retains 100% of samples, but fails to capture rare community members, reducing taxonomic and phylogenetic resolution.
• Too High Depth: Maximizes diversity resolution, but discards valuable biological samples that had lower library yield.
• Recommended Target: Pick a depth where alpha rarefaction curves (alpha-rarefaction.qzv) have plateaued, while losing ≤ 10% of samples.

1. Step 1: Run qiime feature-table summarize and inspect table.qzv at view.qiime2.org to get sample depth percentiles.
2. Step 2: Enter your minimum and median depths here to evaluate the trade-off.
3. Step 3: Pass this recommended value to Section 10: qiime diversity core-metrics-phylogenetic --p-sampling-depth <value>.

High-throughput sequencing datasets consist of dozens or hundreds of raw .fastq.gz files. QIIME 2 does not ingest loose folders of files via ambiguous wildcards. Instead, it enforces reproducibility by requiring an authoritative Manifest TSV file that explicitly links each sample identifier to its exact file paths on disk.

Why Hand-Crafting Fails: Manually editing manifest files in spreadsheet tools (like Excel) is the single most common cause of early pipeline crashes. Excel frequently converts tabs into spaces, injects invisible Windows carriage-return characters (\r\n), or truncates sample IDs. This tool outputs a strictly compliant, Unix-encoded (LF) tab-separated manifest.

• Layout Type: Switches between Paired-End (generates 3 columns: sample-id, forward-absolute-filepath, reverse-absolute-filepath under format PairedEndFastqManifestPhred33V2) and Single-End (generates 2 columns for SingleEndFastqManifestPhred33V2).
• Base Directory Absolute Path: The exact folder on your server or machine where your FASTQs live (e.g. $PWD/raw_fastq or /home/user/project/fastqs). QIIME 2 strictly demands absolute paths so that analysis scripts remain reproducible from any working directory.
• Filename Extension: File extension pattern (e.g. .fastq.gz or .fq.gz). The generator automatically builds forward (_R1) and reverse (_R2) file names.
• Sample IDs Textarea: Paste your sample IDs here (one per line, comma-separated, or space-separated). The generator instantly extracts and formats them.
• Generate Manifest: Compiles and displays the formatted tab-delimited text in the live preview box.
• Load Example: Injects a standard reference 3-sample mock dataset so you can immediately see the expected structure.
• ⬇ Download manifest.tsv: Exports a clean, UTF-8 tab-separated file with Unix line endings ready for terminal input.
• Manifest Output Preview: Real-time display showing the exact tab-separated text that will be fed into QIIME 2.

1. Step 1 (Generate & Save): Use this generator to export manifest.tsv directly into your study workspace.
2. Step 2 (Import Command): In terminal, pass the manifest to qiime tools import:

   qiime tools import \
     --type 'SampleData[PairedEndSequencesWithQuality]' \
     --input-path manifest.tsv \
     --output-path demux-paired.qza \
     --input-format PairedEndFastqManifestPhred33V2

3. Step 3 (Artifact Generation): QIIME 2 reads the manifest, verifies file existence and Phred score integrity, bundles all reads into a single compressed demux-paired.qza artifact, and initializes the immutable cryptographic provenance graph.

Sequencing data without metadata is biologically meaningless. Metadata provides the biological and experimental context for every single sample (e.g., treatment vs control, body site, host phenotype, timepoint, pH, temperature).

Where Metadata is Used: Every statistical and visual tool in QIIME 2 relies on metadata: coloring Emperor 3D PCoA ordination plots, calculating group differences with PERMANOVA (beta-group-significance), comparing alpha diversity (alpha-group-significance), and identifying differentially abundant microbes with ANCOM-BC.

The Critical #q2:types Directive: When this second header row is omitted, QIIME 2 guesses column types automatically. This often fails: a numeric timepoint column (0, 1, 2) gets parsed as a continuous regression slope rather than discrete categorical test groups, or missing values (NA) cause numeric measurements to be downgraded to strings. This builder automatically inserts #q2:types with correct categorical and numeric declarations.

• Sample ID (Mandatory): The unique identifier for each sample. Must match the exact sample-ids in your FASTQ manifest. Cannot contain whitespace or forbidden characters.
• Treatment Group (categorical): Your primary biological experimental condition (e.g. Control, DrugA, WildType). Used for ANCOM-BC comparisons and PERMANOVA.
• Body Site (categorical): Anatomical location or ecological habitat (e.g. gut, soil, saliva). Crucial for multi-factor stratification.
• Subject / Host ID (categorical): Individual host or patient tag (e.g. SubjectA, Mouse_01). Essential for repeated-measures and longitudinal volatility tracking.
• Timepoint (numeric / categorical): Sampling interval (e.g. 0, 7, 14). Handled as continuous time in volatility plots or discrete factor in PERMANOVA.
• pH / Continuous Value (numeric): Quantitative continuous measurements used for environmental gradient analysis and Adonis linear modeling.
• Batch / Sequencing Run: Technical confounder covariate used to block batch effects and sequencing run bias.
• ＋ Add Sample: Validates input fields and adds the sample row into the live memory table.
• Load Example Dataset: Injects a complete 4-sample longitudinal trial dataset with controls and treatments.
• ⬇ Download sample-metadata.tsv: Exports a 100% QIIME 2-compliant tab-separated file with the #q2:types directive included.
• Clear All: Clears all table records to start building a new study from scratch.

1. Step 1 (Validate & Tabulate): Run qiime metadata tabulate --m-input-file sample-metadata.tsv --o-visualization metadata.qzv to verify your columns visually at view.qiime2.org.
2. Step 2 (Core Diversity Integration): Supplied via --m-metadata-file sample-metadata.tsv in qiime diversity core-metrics-phylogenetic to color Emperor 3D PCoA scatter plots.
3. Step 3 (Hypothesis Testing): Passed to PERMANOVA:

   qiime diversity beta-group-significance \
     --i-distance-matrix core-metrics-results/unweighted_unifrac_distance_matrix.qza \
     --m-metadata-file sample-metadata.tsv \
     --m-metadata-column treatment \
     --o-visualization treatment-significance.qzv

4. Step 4 (Differential Abundance): Evaluated in ANCOM-BC via qiime composition ancombc --p-formula 'treatment + timepoint'.

QIIME 2 cannot work directly on loose FASTQ files. Raw sequencing reads must be imported into an immutable QIIME 2 Artifact (.qza) that seals the data with its semantic type, quality encoding (Phred33/64), and a cryptographic provenance tracker.

• --type: Declares what kind of biological data is inside (e.g. SampleData[PairedEndSequencesWithQuality]). Prevents downstream plugins from receiving incorrect data types.
• --input-path: Points to your manifest.tsv mapping file or raw folder.
• --input-format: Specifies the format parser (e.g. PairedEndFastqManifestPhred33V2 for modern Illumina).
• --output-path: Destination .qza artifact file containing your compressed reads.

1. Generate manifest.tsv mapping sample IDs to FASTQs.
2. Run qiime tools import.
3. Verify output by running qiime demux summarize to inspect quality curves.

PCR primers are synthetic non-biological oligonucleotides. If you leave primers in your reads, DADA2 will mistake primer synthesis errors, degenerate IUPAC wobble bases, and PCR drift for genuine biological mutations, creating thousands of artificial ASVs!

• --p-front-f / --p-front-r: Forward and reverse primer sequences. Use ^ (e.g. ^GTGYCAGC...) to anchor the primer strictly to the 5' beginning of the read.
• --p-error-rate: Maximum allowable mismatch rate (0.1 = 1 mismatch per 10 bases).
• --p-minimum-length: Drops artifactual reads that become too short after trimming.

1. Input demultiplexed reads (demux-paired.qza).
2. Cutadapt scans the 5' ends and clips matching primer sequences.
3. Outputs clean trimmed-demux-paired.qza ready for DADA2 error modeling.

Traditional OTU clustering lumped sequences into 97% similarity buckets, masking true biological strain variation. DADA2 constructs an empirical error model of Illumina sequencing chemistry to resolve exact, single-nucleotide differences, outputting true Amplicon Sequence Variants (ASVs).

• --p-trunc-len-f / -r: Truncates reads at the 3' position where quality drops. Must preserve ≥ 20 bp overlap for merging!
• --p-trim-left-f / -r: Trims low-quality bases from the 5' beginning. Set to 0 if Cutadapt already removed primers.
• --p-max-ee-f / -r: Discards reads with higher expected errors than cutoff (default 2.0).
• --p-n-threads: Number of parallel CPU threads to accelerate compute time.

1. Filters low-quality reads and learns run-specific error models.
2. Denoises forward and reverse reads independently.
3. Merges paired reads with minimum 12-20 bp overlap.
4. Identifies and removes bimeras/chimeras, outputting table.qza (counts), rep-seqs.qza (FASTA), and denoising-stats.qza.

Microbiome amplicon sequencing produces millions of raw reads corrupted by PCR errors, Illumina base substitution errors, and chimeras. Choosing between DADA2 (ASVs), Deblur (ASVs), and VSEARCH (97% OTUs) determines resolution, cross-study comparability, and computational cost.

• DADA2: Infers exact amplicon sequence variants (ASVs) via a sample-specific error learning model. Joins forward and reverse reads with built-in quality-aware overlap consensus and removes bimeras. Single-nucleotide resolution.
• Deblur: Uses a static global Illumina error profile. Extremely fast on massive datasets (e.g. 50,000+ samples) but requires all reads to be trimmed to an identical length and operates exclusively on single-end (or pre-joined) reads.
• VSEARCH 97% OTUs: Collapses sequences differing by ≤3% into operational taxonomic units (OTUs). Blurs microdiversity, hides distinct bacterial strains, but remains useful for meta-analyses integrating legacy 454/pyrosequencing datasets.
• ITSxpress + DADA2: Required for fungal ITS because ITS variable lengths prevent quality-based truncation. Trims conserved flanking genes (18S/5.8S/28S) before non-truncated DADA2.

1. For Illumina standard 16S/18S paired-end data → Always use DADA2 denoise-paired.
2. For fungal ITS → Run ITSxpress then DADA2 with --p-trunc-len-f 0 --p-trunc-len-r 0.
3. For PacBio HiFi / Nanopore → Use DADA2 denoise-ccs.
4. For legacy meta-analysis → Use VSEARCH 97% OTUs.

The feature table (table.qza) is an $M \times N$ matrix of observation counts ($M$ samples by $N$ biological ASVs). Along with representative sequences (rep-seqs.qza), it is the direct input for taxonomy, phylogeny, alpha/beta diversity, and differential abundance.

• feature-table summarize: Generates table.qzv, showing sequencing depth percentiles, sample counts, and feature frequencies.
• tabulate-seqs: Generates rep-seqs.qzv with direct 1-click NCBI BLAST links for every ASV.
• filter-samples: Removes low-depth samples or selects subsets using SQL metadata queries (--p-where).
• filter-features: Removes singletons or rare noise features.

1. Inspect table.qzv to verify minimum, median, and maximum read depths.
2. Filter out samples with inadequate reads using filter-samples.
3. Pass the curated table forward to phylogeny and diversity pipelines.

ASVs are raw nucleotide sequences (e.g. TACGGAGGAT...). Taxonomic classification uses machine-learning classifiers to assign biological identity to each ASV: Kingdom, Phylum, Class, Order, Family, Genus, and Species.

• SILVA 138: Curated reference database for bacteria and archaea.
• Greengenes2: Phylogeny-integrated database supporting consistent whole-genome and amplicon resolution.
• UNITE: Curated reference database for fungal ITS marker genes.
• classify-sklearn: Multinomial Naive Bayes classifier trained on 7-mer nucleotide frequencies with bootstrap confidence estimation.

1. Run qiime feature-classifier classify-sklearn matching your amplicon primer region.
2. Build interactive barplots via qiime taxa barplot.
3. Filter out non-target sequences (e.g. mitochondrial or chloroplast DNA).

Classifying microbial representative sequences assigns biological identity (Phylum through Species) to anonymous ASVs. The accuracy of taxonomic profiles depends critically on matching your biological target to the appropriate curated reference repository and classifier algorithm.

• SILVA 138.1: The gold standard for environmental, aquatic, agricultural, and human microbiomes. Comprehensive coverage across Bacteria, Archaea, and Eukaryotes with standardized 7-rank taxonomy.
• Greengenes2 (2022.10): Built on the Web of Life (WoL) genomic tree. Harmonizes 16S amplicons directly with shotgun metagenomic whole genomes; best for phylogenetic integration.
• UNITE (v8/v9): The definitive international database for fungal ITS (ITS1, ITS2, and Full ITS) based on dynamic similarity threshold species hypotheses (SHs).
• PR2 (Protist Ribosomal Reference): Specialized curated database for protistan and microeukaryotic 18S sequences.
• BOLD / Midori: Curated cytochrome c oxidase I (COI) databases for metazoan/animal metabarcoding and environmental DNA (eDNA).

• Naive Bayes (classify-sklearn): High-speed k-mer frequency classifier. When trained on primer-extracted subregions (e.g. 515F-806R), achieves superior classification accuracy and fewer over-confident false assignments.
• Consensus VSEARCH / BLAST+: Local sequence alignment. Essential for atypical environments, novel phyla, or when training custom classifiers is computationally infeasible.

Alpha diversity summarizes the ecological complexity within individual samples. It evaluates both richness (how many distinct species exist) and evenness (how equitably individuals are distributed among species).

• Observed Features: Qualitative richness count of unique ASVs.
• Shannon Index: Quantitative measure accounting for richness and abundance evenness.
• Faith's PD: Phylogenetic richness measuring total branch length across the phylogenetic tree.
• Pielou's Evenness: Quantifies uniformity of abundance across taxa.

1. Run qiime diversity alpha-rarefaction to verify curves plateau.
2. Execute qiime diversity core-metrics-phylogenetic at the chosen sampling depth.
3. Test statistical group differences using qiime diversity alpha-group-significance (Kruskal-Wallis).

Looking at clusters on an Emperor PCoA plot is exploratory; hypothesis testing statistically proves whether distances between metadata groups (e.g. Treatment vs Control, Healthy vs Disease) are significantly larger than within-group variation.

• PERMANOVA (beta-group-significance): Non-parametric multivariate ANOVA testing whether community centroids differ in distance space.
• PERMDISP: Tests whether within-group variance (dispersion) is equal across groups. Run alongside PERMANOVA to confirm significance is not a variance artifact!
• Adonis: Multi-factor regression partitioning variance across multiple confounding covariates simultaneously.

1. Pass a distance matrix (e.g. unweighted_unifrac_distance_matrix.qza) to PERMANOVA.
2. Specify your experimental metadata column (--m-metadata-column treatment).
3. Inspect pairwise pseudo-F statistics and Benjamini-Hochberg FDR q-values in .qzv.

• Unweighted UniFrac (Qualitative + Phylogenetic): Measures phylogenetic branch lengths unique to either sample. Extremely sensitive to community membership shifts and rare ancestral lineages.
• Weighted UniFrac (Quantitative + Phylogenetic): Weights phylogenetic branch lengths by the relative abundances of taxa. Emphasizes dominant lineages and is robust against PCR noise.
• Bray-Curtis Dissimilarity (Quantitative Non-Phylogenetic): Computes differences in read counts ($BC_{jk} = 1 - \frac{2C_{jk}}{S_j + S_k}$). Ideal for non-alignable markers (ITS, COI) or when abundance shifts dominate.
• Jaccard Distance (Qualitative Non-Phylogenetic): Measures fraction of non-shared species ($1 - \frac{A \cap B}{A \cup B}$). Useful for presence/absence comparison without assuming phylogenetic lineage trees.
• Aitchison / Robust Aitchison (DEICODE): Compositional distance based on centered log-ratio (CLR) transformation. Highly recommended for sparse datasets without rarefaction!

1. For 16S/18S with a rooted tree → Test both Unweighted & Weighted UniFrac to contrast membership vs abundance effects.
2. For fungal ITS → Rely on Bray-Curtis and Jaccard.
3. Always execute qiime diversity beta-group-significance (PERMANOVA) to test statistical significance ($p < 0.05$).

Microbiome sequencing yields compositional relative abundances rather than absolute counts. Traditional parametric tests (t-tests, ANOVA) yield false positive rates exceeding 70% because bloom of one dominant taxon mathematically forces relative percentages of all other taxa down!

ANCOM-BC corrects for sampling fraction bias using a log-linear model to infer true biological fold changes, standard errors, and FDR-corrected q-values.

• --p-formula: Linear model formula (e.g. treatment or genotype + timepoint) allowing adjustment for confounders.
• --p-reference-levels: Sets the baseline reference group (e.g. treatment::Control).
• --p-prv-cut: Prevalence filter dropping ultra-rare noise taxa.
• da-barplot: Creates interactive effect-size waterfall barplots with confidence intervals.

1. Collapse table to desired rank (e.g. Genus) or use ASV-level table.
2. Run qiime composition ancombc with experimental formula.
3. Generate interactive differential abundance visualization with da-barplot.

Unlike 16S rRNA genes which have relatively uniform length (~253 bp in V4), fungal Internal Transcribed Spacer (ITS) regions vary biologically from 200 to 600+ bp.

Critical Pipeline Changes:
1. ITSxpress is required to trim flanking conserved ribosomal genes (18S, 5.8S, 28S).
2. DADA2 Truncation must be 0 (--p-trunc-len 0)! Truncating at a fixed base position will discard all longer fungal amplicons.
3. UNITE Database must be used for taxonomy assignment.

• --p-region: Target amplicon region (ITS1 or ITS2).
• --p-taxa: Target kingdom (F for Fungi, A for All).
• UNITE dynamic: Dynamic clustering thresholds provide species-level discrimination.