Design the codebook

Step 3 of the Getting started workflow.

The codebook assigns each target three readout bits — the combination that identifies it during imaging. It is the contract between your biological targets and the encoding, and everything downstream is built against it. A codebook that is wrong produces probes that are individually perfect and collectively undecodable, so it is worth getting right before the expensive step.

If the idea of combinatorial bits is new, What SOLAR is and what these probes do explains what the bits physically are and why panels are encoded this way.

Generate it

mkprobes make-codebook data/mouse panel_a/genes.converted.tss.txt -o panel_a/codebook.json

Pass -o explicitly. Without it, the output name is derived from the input list rather than being a fixed default: genes.converted.tss.txt produces genes.converted.tss.codebook.json. That is rarely what anyone expects, and the manifest you generated with mkprobes init refers to codebook.json.

What the command does:

  1. Picks the smallest vendored MHD (minimum-Hamming-distance) code whose capacity exceeds your target count by at least 5%.

  2. Assigns each target three readout bits.

  3. Fills the spare capacity with Blank-N decoy codewords — these are how you measure the false-positive rate on real data.

  4. Swaps any codeword that would perfectly confound imaging rounds onto a blank.

  5. Logs a hash of the finished codebook.

--n-bits overrides the automatic sizing; you rarely want to.

Optional: balance the readout load

If you have per-target expression data, the assignment can be optimized so no single readout bit is dominated by a handful of very highly expressed genes:

# a table registered on the dataset...
mkprobes make-codebook data/mouse panel_a/genes.converted.tss.txt -o panel_a/codebook.json \
    --expression fpkm

# ...or any file on disk
mkprobes make-codebook data/mouse panel_a/genes.converted.tss.txt -o panel_a/codebook.json \
    --expression expression.tsv --expression-column tpm

This is genuinely optional — omit it and you get a plain seeded assignment (--seed), which is a perfectly good panel.

The table needs a transcript_id and/or gene_id column. Targets missing from it are filled with the table median, with a warning. --iterations (default 200) sets how many assignments are tried; the one with the most even per-bit load wins. --expression-column is only needed when the value column is ambiguous.

For single-cell-derived optimization, the Python API offers CodebookPickerSingleCell.find_optimalish, which balances per-cell load by percentile.

Extending an existing panel

To add targets to a panel you have already ordered, without reusing bits:

mkprobes make-codebook data/mouse new_genes.tss.txt -o panel_b/codebook.json \
    --existing-codebook panel_a/codebook.json

This derives the bit offset from the old codebook and refuses gene or bit overlap rather than silently colliding. --offset sets the offset by hand and is mutually exclusive with --existing-codebook.

Which codebook produced a given file

Every codebook has a short hash — a stable identifier for that exact set of targets and bit assignments. You do not have to record it anywhere: the tool does.

make-codebook writes it beside the codebook, so it outlives the terminal:

panel_a/codebook.json
panel_a/codebook.hash     <- 25dd20

It is also stamped into the provenance of every file designed against that codebook — each per-target _final_*.parquet, and the assembled pool. So the question "which codebook produced this?" is answered by the file itself:

mkprobes provenance panel_a/output/Sox2-201_final_BamHIKpnI_2,10,18.parquet

{
  "codebook_hash": "25dd20",
  "bits": [2, 10, 18],
  "stage": "construct",
  ...
}

Match that against codebook.hash to confirm an output came from the codebook you think it did. This matters when a codebook is regenerated: a different seed or an edited target list produces different bit assignments, and the outputs are otherwise indistinguishable — same target, same file name, different panel.

To print the hash of any codebook directly:

mkprobes hash panel_a/codebook.json

Note

The hash covers the codebook as written, Blank codes included. Re-serialising it — different indentation, different key order — does not change it, because hashing sorts the keys first. Changing any target's bits does.

What a codebook looks like

{
  "Sox2-201": [2, 10, 18],
  "Pax6-201": [1, 9, 17],
  "Blank-1":  [3, 11, 19]
}

The rules mkprobes enforces, and which you should preserve if you ever edit one by hand:

  1. A JSON object at the top level.

  2. Keys are target names matching your target list exactly.

  3. Values are arrays of exactly three integers.

  4. The three integers within a target are distinct.

  5. No two targets share the same three-bit code.

Generally: do not edit it by hand. Regenerate it. The file is cheap to produce and hand edits are how rules 4 and 5 get broken.

Before you spend hours on probes

Check that the codebook and the target list still agree — a target list edited after the codebook was generated is a common and expensive mistake:

python - <<'EOF'
import json
from pathlib import Path

targets = {
    line.split("#", 1)[0].strip()
    for line in Path("panel_a/genes.converted.tss.txt").read_text().splitlines()
    if line.split("#", 1)[0].strip()
}
codebook = {k for k in json.loads(Path("panel_a/codebook.json").read_text()) if not k.startswith("Blank")}

if missing := sorted(targets - codebook):
    raise SystemExit(f"Missing from codebook: {missing}")
if extra := sorted(codebook - targets):
    raise SystemExit(f"In codebook but not in target list: {extra}")
print(f"OK - {len(targets)} targets aligned")
EOF

If that fails, regenerate the codebook from the current target list rather than patching either file.

When it goes wrong

  • Codebook names do not match the target list — regenerate from genes.converted.tss.txt, do not rename keys.

  • You changed the target list after generating the codebook — regenerate and re-hash. Any probes already designed against the old codebook are still valid for the targets that kept their bits, but run-panel works from the codebook, so the two must agree.

  • Two panels imaged together reuse bits — generate the second with --existing-codebook pointing at the first.

Next: Design probes.