Design the codebook

Step 3 of the Getting started workflow.

The codebook assigns each target three readout bits — the combination that identifies it during imaging. It is the contract between your biological targets and the encoding, and everything downstream is built against it. A codebook that is wrong produces probes that are individually perfect and collectively undecodable, so it is worth getting right before the expensive step.

If the idea of combinatorial bits is new, What SOLAR is and what these probes do explains what the bits physically are and why panels are encoded this way.

Generate it

mkprobes make-codebook data/mouse panel_a/genes.converted.tss.txt -o panel_a/codebook.json

Pass -o explicitly. Without it, the output name is derived from the input list rather than being a fixed default: genes.converted.tss.txt produces genes.converted.tss.codebook.json. That is rarely what anyone expects, and the manifest you generated with mkprobes init refers to codebook.json.

What the command does:

  1. Picks the smallest vendored MHD (minimum-Hamming-distance) code whose capacity exceeds your target count by at least 5%.

  2. Assigns each target three readout bits.

  3. Fills the spare capacity with Blank-N decoy codewords — these are how you measure the false-positive rate on real data.

  4. Swaps any codeword that would perfectly confound imaging rounds onto a blank.

  5. Logs a hash of the finished codebook.

--n-bits overrides the automatic sizing; you rarely want to.

Optional: balance the readout load

If you have per-target expression data, the assignment can be optimized so no single readout bit is dominated by a handful of very highly expressed genes:

# a table registered on the dataset...
mkprobes make-codebook data/mouse panel_a/genes.converted.tss.txt -o panel_a/codebook.json \
    --expression fpkm

# ...or any file on disk
mkprobes make-codebook data/mouse panel_a/genes.converted.tss.txt -o panel_a/codebook.json \
    --expression expression.tsv --expression-column tpm

This is genuinely optional — omit it and you get a plain seeded assignment (--seed), which is a perfectly good panel.

The table needs a transcript_id and/or gene_id column. Targets missing from it are filled with the table median, with a warning. --iterations (default 200) sets how many assignments are tried; the one with the most even per-bit load wins. --expression-column is only needed when the value column is ambiguous.

For single-cell-derived optimization, the Python API offers CodebookPickerSingleCell.find_optimalish, which balances per-cell load by percentile.

Pooling panels: the offset

Two panels hybridised together must not share readout bits, so each codebook has to start where the previous one stopped. Where a panel starts is the offset field of its probe set in manifest.json, and make-codebook reads it from there: the manifest beside the output path (-o), or the one given with --manifest, matched by codebook path exactly as run-panel matches design settings.

offset is a 0-based bit position, not a readout ID. Readout IDs are interleaved across imaging rounds, so a 10-bit panel at offset 0 does not use readouts 1 to 10; it uses the first ten in that order. Count bits, not IDs: the first panel sits at 0, and a panel pooled with it sits at the number of bits already taken.

mkprobes init panel_a --bcidx 0              # offset 0
mkprobes init panel_b --bcidx 1 --offset 10  # after a 10-bit panel A

mkprobes make-codebook data/mouse panel_a/genes.converted.tss.txt -o panel_a/codebook.json
mkprobes make-codebook data/mouse panel_b/genes.converted.tss.txt -o panel_b/codebook.json

The second command logs Offset 10 from panel_b/manifest.json. To describe both panels in one manifest instead, list both probe sets in it, each with its own codebook, bcidx and offset; check-manifest then refuses any pair whose codebooks share a bit, and refuses a codebook that does not start at its probe set's offset. Each panel also needs its own bcidx, since the offset separates readouts and bcidx separates amplification.

Forty-nine readout IDs exist in all, which bounds how many bits pooled panels can take between them.

--offset overrides the manifest for one run, with a warning, and check-manifest will then hold the codebook to whatever the manifest says. To extend a panel you have already ordered without reusing its bits, pass the old codebook instead:

mkprobes make-codebook data/mouse new_genes.tss.txt -o panel_b/codebook.json \
    --existing-codebook panel_a/codebook.json

This derives the offset from the old codebook's bit count and refuses gene or bit overlap rather than silently colliding. It is mutually exclusive with --offset, and a manifest that states a different offset is an error.

Which codebook produced a given file

Every codebook has a short hash — a stable identifier for that exact set of targets and bit assignments. You do not have to record it anywhere: the tool does.

make-codebook writes it beside the codebook, so it outlives the terminal:

panel_a/codebook.json
panel_a/codebook.hash     <- 25dd20

It is also stamped into the provenance of every file designed against that codebook — each per-target _final_*.parquet, and the assembled pool. So the question "which codebook produced this?" is answered by the file itself:

mkprobes provenance panel_a/output/Sox2-201_final_BamHIKpnI_2,10,18.parquet

{
  "codebook_hash": "25dd20",
  "bits": [2, 10, 18],
  "stage": "construct",
  ...
}

Match that against codebook.hash to confirm an output came from the codebook you think it did. This matters when a codebook is regenerated: a different seed or an edited target list produces different bit assignments, and the outputs are otherwise indistinguishable — same target, same file name, different panel.

To print the hash of any codebook directly:

mkprobes hash panel_a/codebook.json

Note

The hash covers the codebook as written, Blank codes included. Re-serialising it — different indentation, different key order — does not change it, because hashing sorts the keys first. Changing any target's bits does.

What a codebook looks like

{
  "Sox2-201": [2, 10, 18],
  "Pax6-201": [1, 9, 17],
  "Blank-1":  [3, 11, 19]
}

The rules mkprobes enforces, and which you should preserve if you ever edit one by hand:

  1. A JSON object at the top level.

  2. Keys are target names matching your target list exactly.

  3. Values are arrays of exactly three integers.

  4. The three integers within a target are distinct.

  5. No two targets share the same three-bit code.

Generally: do not edit it by hand. Regenerate it. The file is cheap to produce and hand edits are how rules 4 and 5 get broken.

Before you spend hours on probes

Check that the codebook and the target list still agree — a target list edited after the codebook was generated is a common and expensive mistake:

python - <<'EOF'
import json
from pathlib import Path

targets = {
    line.split("#", 1)[0].strip()
    for line in Path("panel_a/genes.converted.tss.txt").read_text().splitlines()
    if line.split("#", 1)[0].strip()
}
codebook = {k for k in json.loads(Path("panel_a/codebook.json").read_text()) if not k.startswith("Blank")}

if missing := sorted(targets - codebook):
    raise SystemExit(f"Missing from codebook: {missing}")
if extra := sorted(codebook - targets):
    raise SystemExit(f"In codebook but not in target list: {extra}")
print(f"OK - {len(targets)} targets aligned")
EOF

If that fails, regenerate the codebook from the current target list rather than patching either file.

When it goes wrong

  • Codebook names do not match the target list — regenerate from genes.converted.tss.txt, do not rename keys.

  • You changed the target list after generating the codebook — regenerate and re-hash. Any probes already designed against the old codebook are still valid for the targets that kept their bits, but run-panel works from the codebook, so the two must agree.

  • Two panels imaged together reuse bits — generate the second with --existing-codebook pointing at the first.

Next: Design probes.