Build a dataset

Step 1 of the Getting started workflow.

A dataset is the sequence universe your probes are searched in and screened against. Probe specificity is bounded by it: if the dataset is wrong or incomplete, every later step produces confident-looking probes that bind the wrong things. Build it once per species and release, then treat it as immutable shared input.

There are three ways to build one. Pick by what you have:

You have

Use

Section

Nothing — you want mouse or human

mkprobes prepare

A

A genome FASTA and a GTF/GFF3 annotation

mkprobes ingest

B

Transcript sequences already

mkprobes create-dataset

C

A. Mouse or human

mkprobes prepare data --species mouse --threads 16

This downloads the curated reference set and indexes it. It writes into a species subdirectory, so the above creates data/mouse/, and every later command takes data/mouse — not data.

What it produces:

  • cdna_ncrna_trna.fasta, the sequence universe

  • a bowtie2 index (txome) for off-target search

  • transcriptome 18-mers (cdna18.jf) and t/r/snoRNA exclusion 15-mers (r_t_snorna15.jf) from jellyfish

  • gencode.gtf.gz and ensembl.gtf.gz

Set --threads to the CPUs you were actually allocated; oversizing it hurts on a shared node.

Check it worked:

ls data/mouse/cdna18.jf data/mouse/r_t_snorna15.jf data/mouse/gencode.gtf.gz

B. Any species, from a genome and annotation

mkprobes ingest data/myspecies \
    --genome genome.fa.gz --gtf annotation.gtf --species myspecies

ingest validates the annotation, extracts transcript sequences with gffread, builds the bowtie2 and k-mer indices, and writes the dataset manifest.

Always run --validate-only first and read the report. The failure it catches most often is contig names that do not match between genome and annotation (chr1 vs 1 vs scaffold_1), which otherwise produces a silently empty dataset rather than an error.

Flags you are likely to want:

  • --extract transcripts|cds — spliced transcripts with UTRs (default, matching how the reference datasets are built) vs CDS only.

  • --rrna-fasta / --trna-fasta — sequences for the probe blocklist, both repeatable. Without a blocklist, nothing stops probes landing in rRNA/tRNA-derived sequence.

  • --blocklist-biotypes rRNA,tRNA,snoRNA — build the blocklist from the GTF's biotype column instead, when it has one.

  • --annotation-table NAME=PATH — register a lookup table (orthologs, aliases, expression) so you can select targets by e.g. human ortholog symbol. Needs a transcript_id and/or gene_id column. Repeatable.

  • --keep-genome — copy the genome into the dataset directory. Off by default because genomes are large; the sha256 is recorded either way.

  • --strip-version — off by default here, and should stay off for de novo annotations: stripping .N suffixes merges StringTie isoforms (STRG.1.1 and STRG.1.2 become one).

Ingested datasets carry validation_report.json, unstranded_transcripts.txt (only when some transcripts lack a strand — see File formats and conventions), and solar_intake.yaml, a provenance manifest with input sha256s, tool versions, the literal command run and QC counts. Fill in its stub fields (assembly source, annotation method, data owner) — that file is how another lab member reproduces your dataset.

This is the path for non-traditional model species, and it has a full worked runbook: SOLAR probesets for a new species.

C. Any species, from a transcriptome FASTA

Use this only when you already have transcript sequences and no annotation to extract them from. Prefer ingest when you have a genome and a GTF.

mkprobes create-dataset data/squid --fasta refs/squid_txome.fasta --species doryteuthis

Flags:

  • --gtf — an annotation matching the FASTA. Optional, but supplying it is what enables gene-name lookups and sibling-isoform allowance during screening, so supply it if you have one.

  • --blocklist-fasta — rRNA/tRNA sequences for the blocklist. Repeatable.

  • --annotation NAME=PATH — register a lookup table (note: --annotation here, --annotation-table on ingest). Repeatable.

  • --fasta-key-regex — how to extract the record ID from FASTA headers.

  • --strip-version / --no-strip-version — on by default here, unlike ingest. Use --no-strip-version for de novo annotations whose IDs embed meaningful dots (STRG.1.1, g1.t1).

--species is metadata only. It is not checked against the FASTA, and human and mouse are reserved for reference datasets.

Reference or custom: why it matters

The two kinds of dataset are screened differently, and mkprobes decides which you have from the directory contents. Reference datasets screen candidates against pseudogenes and treat every Ensembl isoform of a target as an acceptable binder; custom datasets do neither. Keep the two in separate directories — a directory holding both is rejected rather than guessed at.

The exact rules, and the fields inside dataset.json, are in Which kind of dataset a command loads.

On a cluster

  • Build once per species and version, into an explicit versioned root (refs/mouse_release_110/), then treat it as read-only.

  • Prefer one high-memory prep job over several duplicate ones.

  • Record provenance: urls.tsv for reference mode, solar_intake.yaml for ingested datasets.

When it goes wrong

  • Downloads fail — check outbound network access from the node you are on.

  • jellyfish, bowtie2 or gffread: command not found — install them into the environment you are running from; all three are on bioconda. See Before you start.

  • Species not supported by prepare — that command is mouse and human only. Use ingest or create-dataset.

  • Stale files after a release change — build into a clean, versioned directory rather than over the top of the old one.

Next: Choose your targets.