# File formats and conventions

Conventions used across the SOLAR (splint/padlock, STARmap-style) probe-design workflow.

## Core probe-design files

- Target list: plain text (`.txt`), one gene/transcript per line. Blank lines
  are skipped and everything after a `#` is a comment, including inline. Order
  is preserved (it feeds bit assignment); a target named twice is an error.
- Codebook: JSON mapping `{gene_or_transcript: [bit_id_1, bit_id_2, bit_id_3]}`.
- Manifest: JSON list of probe sets for assembly. `mkprobes init` writes a
  valid one; `mkprobes check-manifest` validates an existing one.
- Candidate tables: parquet files (`*_crawled.parquet`, `*_all.parquet`, `*_bowtie.parquet`).
- Screened sets: parquet (`*_screened_ol*.parquet`).
- Final constructed probes: parquet (`*_final_*.parquet`).
- Oligo pool: plain text (`<name>_final.txt`), one oligo per line, splint and
  padlock alternating. Its region-by-region structure is specified in
  {doc}`seqspec`, and `mkprobes validate-pool` checks a pool against it.

Every parquet the pipeline writes carries a provenance record in its parquet
key/value metadata: package version, UTC timestamp, the command line, the
dataset, and the parameters that shaped it. The `.stats.json` sidecars embed
the same record under a `provenance` key. Read it back with
`mkprobes provenance <file>`. Column-by-column meanings are in
{doc}`columns`.

## Codebook requirements (CLI workflow)

1. Top-level JSON object with target-name keys.
2. Each value is a list of exactly 3 integers.
3. Each target's 3 integers are distinct.
4. Target names match the list used for probe-generation loops.
5. No duplicate 3-bit tuples across targets.

See {doc}`../workflows/design_the_codebook` for validation commands.

## Dataset folder conventions

Reference dataset (`mkprobes prepare`) typically includes:

- `gencode.gtf.gz`
- `ensembl.gtf.gz`
- `txome.fasta`
- `cdna_ncrna_trna.fasta`
- `cdna18.jf`
- `r_t_snorna15.jf`
- `txome` bowtie2 index files

Generic/custom dataset (`mkprobes ingest` or `mkprobes create-dataset`) includes:

- `dataset.json`
- input FASTA copy
- bowtie2 index files (`*.bt2`)
- k-mer database files generated by jellyfish (`*.jf`)

### Which kind of dataset a command loads

Keep the two kinds in separate directories. Every command resolves a dataset path
like this:

- A directory holding a downloaded reference build (recognised by `gencode.gtf.gz`)
  and named exactly `human` or `mouse` loads as a **reference dataset**.
- A directory holding a `dataset.json` loads as a **custom dataset**.
- A directory holding *both* is rejected, because there is no way to tell which one
  you meant. Move the custom dataset somewhere else, or delete its `dataset.json`.

This matters for probe specificity, not just bookkeeping. Reference datasets screen
candidates against pseudogenes and treat every Ensembl isoform of a target gene as
an acceptable binder; custom datasets do neither. Building a custom dataset for
human or mouse is allowed, but `mkprobes` warns when it loads one, so the reduced
screening is never applied silently.

### `dataset.json` fields (custom datasets)

All newer fields are optional and back-compatible:

- `gtf_name` / `cache_name`: annotation file and its parsed cache.
- `blocklist_kmer_name`: rRNA/tRNA blocklist k-mer database (typically `blocklist15.jf`); when present, it is enforced automatically during candidate generation.
- `genome_fasta_name`: retained genome FASTA (from `ingest --keep-genome`).
- `annotations`: registry of named lookup tables, e.g. `{"orthologs": "orthologs.tsv"}`. Tables must carry `transcript_id` and/or `gene_id` columns.
- `fasta_key_regex`, `strip_version`: FASTA-header parsing settings persisted per dataset.

### Ingested-dataset extras (`mkprobes ingest`)

- `solar_intake.yaml`: provenance manifest — input sha256s, tool versions, the literal command run, and QC counts. Stub fields are completed by the operator.
- `validation_report.json`: annotation/sequence validation results.
- `unstranded_transcripts.txt`: written only when the annotation contains
  transcripts with no strand. gffread keeps those but extracts the plus-strand
  sequence, so probes for any that are really on the minus strand are antisense
  and will not bind — and nothing downstream catches it, because the sequences
  themselves are valid. Check your targets against this file before designing:
  `comm -12 <(sort -u genes.txt) <(sort -u <dataset>/unstranded_transcripts.txt)`