File formats and conventions

Conventions used across the SOLAR (splint/padlock, STARmap-style) probe-design workflow.

Core probe-design files

  • Target list: plain text (.txt), one gene/transcript per line. Blank lines are skipped and everything after a # is a comment, including inline. Order is preserved (it feeds bit assignment); a target named twice is an error.

  • Codebook: JSON mapping {gene_or_transcript: [bit_id_1, bit_id_2, bit_id_3]}.

  • Manifest: JSON list of probe sets for assembly. mkprobes init writes a valid one; mkprobes check-manifest validates an existing one.

  • Candidate tables: parquet files (*_crawled.parquet, *_all.parquet, *_bowtie.parquet).

  • Screened sets: parquet (*_screened_ol*.parquet).

  • Final constructed probes: parquet (*_final_*.parquet).

  • Oligo pool: plain text (<name>_final.txt), one oligo per line, splint and padlock alternating. Its region-by-region structure is specified in The SOLAR seqspec, and mkprobes validate-pool checks a pool against it.

Every parquet the pipeline writes carries a provenance record in its parquet key/value metadata: package version, UTC timestamp, the command line, the dataset, and the parameters that shaped it. The .stats.json sidecars embed the same record under a provenance key. Read it back with mkprobes provenance <file>. Column-by-column meanings are in Output columns, stage by stage.

Codebook requirements (CLI workflow)

  1. Top-level JSON object with target-name keys.

  2. Each value is a list of exactly 3 integers.

  3. Each target's 3 integers are distinct.

  4. Target names match the list used for probe-generation loops.

  5. No duplicate 3-bit tuples across targets.

See Design the codebook for validation commands.

Dataset folder conventions

Reference dataset (mkprobes prepare) typically includes:

  • gencode.gtf.gz

  • ensembl.gtf.gz

  • txome.fasta

  • cdna_ncrna_trna.fasta

  • cdna18.jf

  • r_t_snorna15.jf

  • txome bowtie2 index files

Generic/custom dataset (mkprobes ingest or mkprobes create-dataset) includes:

  • dataset.json

  • input FASTA copy

  • bowtie2 index files (*.bt2)

  • k-mer database files generated by jellyfish (*.jf)

Which kind of dataset a command loads

Keep the two kinds in separate directories. Every command resolves a dataset path like this:

  • A directory holding a downloaded reference build (recognised by gencode.gtf.gz) and named exactly human or mouse loads as a reference dataset.

  • A directory holding a dataset.json loads as a custom dataset.

  • A directory holding both is rejected, because there is no way to tell which one you meant. Move the custom dataset somewhere else, or delete its dataset.json.

This matters for probe specificity, not just bookkeeping. Reference datasets screen candidates against pseudogenes and treat every Ensembl isoform of a target gene as an acceptable binder; custom datasets do neither. Building a custom dataset for human or mouse is allowed, but mkprobes warns when it loads one, so the reduced screening is never applied silently.

dataset.json fields (custom datasets)

All newer fields are optional and back-compatible:

  • gtf_name / cache_name: annotation file and its parsed cache.

  • blocklist_kmer_name: rRNA/tRNA blocklist k-mer database (typically blocklist15.jf); when present, it is enforced automatically during candidate generation.

  • genome_fasta_name: retained genome FASTA (from ingest --keep-genome).

  • annotations: registry of named lookup tables, e.g. {"orthologs": "orthologs.tsv"}. Tables must carry transcript_id and/or gene_id columns.

  • fasta_key_regex, strip_version: FASTA-header parsing settings persisted per dataset.

Ingested-dataset extras (mkprobes ingest)

  • solar_intake.yaml: provenance manifest — input sha256s, tool versions, the literal command run, and QC counts. Stub fields are completed by the operator.

  • validation_report.json: annotation/sequence validation results.

  • unstranded_transcripts.txt: written only when the annotation contains transcripts with no strand. gffread keeps those but extracts the plus-strand sequence, so probes for any that are really on the minus strand are antisense and will not bind — and nothing downstream catches it, because the sequences themselves are valid. Check your targets against this file before designing: comm -12 <(sort -u genes.txt) <(sort -u <dataset>/unstranded_transcripts.txt)