Tutorial: Generating a Polished EPUB from a Markdown Post

Abstract

A plain pandoc markdown.md -o book.epub gives you an EPUB, but not a good one: the user’s essays use Pandoc citekeys ([^1]) instead of [^n] footnotes, write their section titles as plain numbered lines instead of # headings, and reference hero images that many readers can’t display. This tutorial is the complete, repeatable pipeline that turns a markdown post into a polished EPUB3: native numbered footnotes with clickable URLs, a real table of contents (including a Footnotes entry), a cover image with the title and author baked in, WebP-to-PNG conversion for compatibility, and the reader-crash pitfalls that cost the most time to debug. There is also a one-command shortcut (epb) that does all of it.

Prerequisites

• pandoc installed (used for the markdown-to-EPUB conversion).
• Python 3.12+ with Pillow (for cover composition and image conversion). This project uses ~/av/venv/hydra/bin/python3.
• A Zotero bibliography at ~/Zotero/bibliography.bib (the essay’s [^2] references resolve against it).
• The pipeline module bin/hydra/publish/signal_arweave.py (already in the repo; the epb shortcut wraps it).
• A markdown source post whose frontmatter carries at least title and hero_image, and whose body uses [^3] citations and numbered section-marker lines.

Quickest start: ~/av/venv/hydra/bin/python3 ~/av/bin/epb SLUG produces the EPUB from a post under ~/av/doc/posts/<SLUG>/. The rest of this tutorial explains what that does and how to reproduce or extend it.

Step 1: Prep the Markdown Before Pandoc

Pandoc builds the EPUB’s table of contents from #-level headings and renders [^n] footnotes natively. The genre of essay this pipeline is built for has neither. Two preprocessing passes fix that.

1a. Convert citekeys to native footnotes. The body cites [^4]. To render these as footnotes you must resolve each key against ~/Zotero/bibliography.bib, format it as a Chicago-note citation, and emit a native [^N] reference plus a [^N]: <citation> definition at the end of the document.

The two approaches that look reasonable but are wrong:

• convert_citations() leaves bare [N] markers with no definitions – nothing renders.
• pandoc --citeproc --csl chicago-notes-bibliography.csl emits the notes AND a separate alphabetized #refs bibliography before them, which readers and the user both dislike.

Do it yourself. The parser must tolerate the real-world gotchas in this bibliography: URLs live in howpublished (rarely url) and must be wrapped as [url](url) so the footnote link is clickable; titles use capitalization-protection double braces {{Text}} that need stripping; and values like year = 2019 are unbraced.

1b. Turn numbered section markers into headings. The essay writes 1. WHY CARE... and 1.1.1. Homelessness as plain lines, not #. That makes pandoc’s nav TOC empty. Map them to headings by rule:

• a dotted marker (1.1, 1.1.1, N+ segments) -> a heading (###, ####),
• a bare N. followed by ALL-CAPS text -> a top-level chapter (##),
• a bare N. followed by title-case text -> leave as-is (a bullet, e.g. the “fears” lists under section 4).

Step 2: Run Pandoc

From the essay’s own directory (so relative img/ paths resolve), run:

pandoc BODY.md -t epub3 \
  --metadata "title=$TITLE" --metadata "author=$AUTHOR" \
  --toc-depth=4 -o out.epub

--toc-depth=4 makes the section headings from Step 1b appear in the nav.

Step 3: Make the Footnotes Show in the TOC

Pandoc emits a bare <section id="footnotes"> with no heading, so “Footnotes” never appears in the contents. Post-process the EPUB to (a) inject an <h1 id="footnotes-title">Footnotes</h1> inside that section and (b) append a top-level Footnotes <li> to nav.xhtml.

Do NOT cheat by adding # Footnotes to the markdown: pandoc splits it into a near-empty ch002.xhtml and leaves the actual <aside> notes in ch001, so the TOC link points at an empty chapter.

Step 4: Bake the Cover

Calibre shows its dedicated cover page as a pure image splash and suppresses any text inside it, so a textual <h1 class="title"> on the cover page never displays. The title and author must be drawn into the image itself.

• Title: white block capitals (Arial Black), on one line near the top.
• Author: smaller (Arial Bold), lower right.
• Both sit on dark semi-transparent rounded-rectangle banners.

Reliable text centering: render the line to a scratch RGBA image, read its ink bounding box with img.getbbox(), size the banner to that, and paste the cropped ink centered. Centering by font metrics or by anchor='mm' both leave all-caps text looking a few pixels high; the ink-bbox method is exact.

The publication metadata (Author, Publication Date, Publisher, Live URL, Copyright) belongs on the first reading page in the body, not on the cover – prepended under the first <h1>, with a plain (un-baked) copy of the hero image beneath it.

Step 5: Make It Survive Real Readers

Several things break real EPUB readers, each found the hard way:

• WebP is unreliable. Convert every media/*.webp to .png and rewrite every reference (xhtml, the cover SVG, and content.opf media-types). bookokrat’s interactive TUI dumps PNG bytes as base64 garbage and crashes on any image-bearing EPUB; Calibre and epy render images fine. Also keep a plain hero PNG for the internal first page (the baked one is only for the cover/thumbnail).
• The SVG cover wrapper crashes bookokrat. --epub-cover-image emits an inline data:image/svg+xml;base64 cover that some readers pour to screen as raw text. Replace the cover page with a plain <img>.
• Broken lightbox <a> wrappers crash Calibre. The markdown wraps figures in <a href="img/live/..."> links to site paths that aren’t in the EPUB. Calibre reports “Referenced file … not found” for each and can crash. Strip those <a href="img/..."> wrappers, keeping the <img>.
• Remove the separate cover page from the spine if you don’t want the hero shown twice (that was a real bug: a cover page plus a hero-under-title page showed the same image twice).

Step 6: Verify

unzip -l out.epub                       # media/*.png, no .webp
for f in EPUB/text/*.xhtml EPUB/nav.xhtml EPUB/content.opf; do
  xmllint --noout "$f"                  # exit 0 = well-formed
done
bookokrat print --toc out.epub          # full TOC incl. Footnotes
bookokrat print --chapter 3 out.epub    # clean text, no base64
ebook-convert out.epub out.htmlz 2>&1 | grep -c "not in manifest\|not found"   # expect 0

The ebook-convert grep is the authoritative gate – it catches both broken link wrappers AND any resource you injected but forgot to declare in the OPF manifest (e.g. the plain internal hero must have a matching <item> in content.opf, or Calibre reports “Referenced file … not in manifest” and drops the image as a broken placeholder).

Accept when: numbered footnotes with clickable URLs; “Footnotes” in the nav TOC; the cover image shows the baked title and author; the first reading page has the metadata block and a plain hero; and no reader logs a missing file.

Step 7: The epb Shortcut

All of the above is wrapped in ~/av/bin/epb, which takes a post slug (or a path to its .md), reads title/hero_image from frontmatter, and produces <slug-slug>.epub:

~/av/venv/hydra/bin/python3 ~/av/bin/epb SLUG --title "..." --author "..." [--open]

It is registered in the Shorthands shortkey catalog (epb), so in chat you can type epb SLUG to build one. The underlying functions live in bin/hydra/publish/signal_arweave.py (render_epub, _citekeys_to_footnotes, _section_markers_to_headings, _compose_cover_bytes, _pngify_epub_media, _fix_epub_xhtml) and are reused by the Signal EPUB delivery path (signal-assemble-full.py --epub).

Reference

Notes