| `pelican_scheduler.py` | symlinked as the bare repo's `hooks/post-receive` |
| `root_index.html` | `/home/blogmistress/zackmdavis.net/index.html` — **renamed on the way over**; scp'ing it under its repo name lands a dead file beside the real one and changes nothing visible |
| `robots.txt` | `/home/blogmistress/zackmdavis.net/robots.txt` (the true domain root, not `/blog` — see the `STATIC_PATHS` comment in `pelicanconf.py` for why Pelican deliberately doesn't generate it) |
+| *(generated)* | `/home/blogmistress/zackmdavis.net/sitemap.xml` — **never deployed by hand**: `_write_sitemap` builds it into `output/` and `SITEGEN_COMMAND` in `pelican_scheduler.py` installs it at the webroot on every build. Editing the live file is pointless; the next deploy overwrites it. |
-The last two live under the webroot — `root /home/blogmistress/zackmdavis.net` in `nginx_siteconf`, which is also what serves `/docs` and `/media`. Being inside `blogmistress`'s own home, they're the only two deployable without root; `install` as root would leave them root-owned.
+`root_index.html` and `robots.txt` live under the webroot — `root /home/blogmistress/zackmdavis.net` in `nginx_siteconf`, which is also what serves `/docs` and `/media` (and now `sitemap.xml`, though the hook puts that one there). Being inside `blogmistress`'s own home, they're the only two deployable without root; `install` as root would leave them root-owned.
After nginx edits, `nginx -t && systemctl reload nginx` — `nginx -t` catches a `conf.d` file that didn't land, since the site config references things defined there. After unit edits, `systemctl daemon-reload` **and** `systemctl restart ai-bot-digest.timer`; a daemon-reload alone leaves an already-scheduled timer on its old schedule.
Roughly **99.9% of AI-crawler traffic is the gitweb interface**, up ~15× in twelve days, while actual fetching of posts *declined* over the same window. Meta/Facebook AI logged 69,074 requests and **zero** content pages in one day; GPTBot 37,122 requests for one page.
-This is permitted behavior, not abuse: `provisioning/robots.txt` is `Allow: /` with **no `Disallow` lines at all**, and git history is a near-infinite URL space (commits × files × view modes × blame/diff/raw). The cheap experiment is `Disallow: /blog/source` — the digests confirm ClaudeBot, Applebot, PerplexityBot, and OAI-SearchBot all reliably fetch robots.txt, so compliant bots would stop; whatever keeps hammering afterward is then a much clearer signal about who's ignoring it. Not done as of 2026-08-07. Weigh it against actually wanting the git history publicly crawlable, which is a real thing to want and the reason it's exposed — see the next section, which is about not having to choose.
+This is permitted behavior, not abuse: `provisioning/robots.txt` is `Allow: /` with **no `Disallow` lines at all**, and git history is a near-infinite URL space (commits × files × view modes × blame/diff/raw). **`Disallow: /blog/source` was added 2026-09-09.** The digests confirm ClaudeBot, Applebot, PerplexityBot, and OAI-SearchBot all reliably fetch robots.txt, so compliant bots should stop; whatever keeps hammering afterward is a much clearer signal about who's ignoring it. Watch the `source` bucket in subsequent digests — it's the whole measurement. Note what this gives up, deliberately: see the next section.
-### Wanted: crawlers see drafts and notes, but not the SHA cross-product (undecided, 2026-08-11)
+### Decided 2026-09-09: gitweb closed, drafts deliberately not exposed
-The blanket `Disallow: /blog/source` above is coarser than what we actually want. The *want* is asymmetric: unpublished writing in the working tree (`content/drafts/`, `notes/`) being legible to AI crawlers is a feature — a model knowing things about ZMD that aren't on the public site is the interesting part — while the every-file-at-every-SHA explosion is pure waste. Those are separable; the blanket rule is only necessary if gitweb is the delivery mechanism for both, and it isn't.
+The want recorded here through 2026-08-11 was asymmetric — unpublished writing (`content/drafts/`, `notes/`) legible to AI crawlers is a *feature*, the every-file-at-every-SHA explosion is waste — and the plan was to separate them. That was reconsidered and dropped. What was decided, and why, so it isn't re-litigated from the old premise:
-**Drafts don't need gitweb at all — they're already served.** `Status: draft` posts (all of `content/drafts/`) build to `/blog/drafts/<slug>.html` via Pelican's default `DRAFT_SAVE_AS`, and `robots.txt` is `Allow: /`, so they are *already* crawlable today. What's missing is discovery: they're excluded from the index, feeds, and tag pages, and nothing else links to them, so no crawler has ever found one — **zero** `/blog/drafts/` fetches across every digest in `notes/ai_bot_digests/`. So the cheapest version of the whole want is "advertise `/blog/drafts/`, keep `Disallow: /blog/source` blanket," and no fine-grained gitweb rule is needed.
+**gitweb was probably the mechanism that worked, and closing it is a real cost.** The evidence that unpublished writing reaches models comes from the *other* blog: `Ultimately_Untrue_Thought` has **no robots.txt at all**, serves gitweb at `/source` over a repo containing 73 drafts and 51 `notes/` files, and links to none of them. There is no other plausible ingestion route, so gitweb crawling almost certainly *is* the surface. Treating "drafts crawlable" and "SHA cross-product crawlable" as separable is true in principle but describes machinery that was never built; in practice they were the same mechanism. `Disallow: /blog/source` therefore switches off the thing that produced the impressive result — accepted knowingly, not overlooked.
-They're also invisible to the LLM-legibility path specifically: `_prepare_markdown_mirrors` iterates `generator.articles`, which excludes drafts, so drafts get no `.md` mirror and no `llms.txt` entry. Extending it to `generator.drafts` is the direct fix — but note `article.markdown_url` is computed as `dirname(save_as) + '.md'`, and a draft's `save_as` is `drafts/<slug>.html` (one path segment, not the `YYYY/Mon/slug/index.html` shape), so that expression collapses every draft to a single `drafts.md`. Special-case it or the mirrors overwrite each other silently.
+**Drafts were left out of the sitemap on purpose.** A draft and its published version live at different URLs (`drafts/<slug>.html` vs `YYYY/Mon/<slug>/`), and the draft URL 404s once the file leaves `content/drafts/`. So a crawler that captured a draft has no live page to revisit and no canonical link to the finished piece: the superseded take persists uncorrected, in a way an edited published post at a stable URL never does. That matters disproportionately for a corpus where changing one's mind in public is a recurring move. Checked at the time: every draft had been touched within the previous month, so there was no safe subset of abandoned ones. The deciding asymmetry is that drafts can be added to the sitemap any time, whereas nothing can un-train a captured draft — and the observed value was thin (the models' impressive knowledge is overwhelmingly of *published* work; draft hits were strays).
-**`notes/` is the part that genuinely needs gitweb**, since it isn't Pelican content and has no URL otherwise. Two ways in:
+**`notes/` is consequently dark to crawlers**, since gitweb was its only URL. Accepted. If it's ever wanted back, don't reopen gitweb — copy an explicit allowlist of files to the webroot (the deploy hook already installs there, and `/docs` and `/media` are already served that way with `autoindex on`), which is path-shaped and stays deliberate about what's exposed.
-* **Fine-grained robots.txt rules.** The obstacle is that gitweb here uses query-string URLs — *every* page is path `/blog/source` with `?p=…;a=…;f=…` — so path-prefix rules are all-or-nothing by construction. Distinguishing views requires matching against the query string, which Google and Bing document (along with `*`/`$` wildcards, codified in RFC 9309) but which is less uniformly implemented than plain prefixes. The precise, wildcard-free version: keep `Disallow: /blog/source`, then add one **exact** `Allow:` line per file, e.g. `Allow: /blog/source/?p=an_algorithmic_lucidity.git;a=blob_plain;f=notes/foo.md;hb=HEAD`. Longest-match `Allow` beats `Disallow` under the Google spec, `hb=HEAD` keeps the URL stable while the content underneath changes, and `blob_plain` skips the HTML chrome. Generating those lines means `robots.txt` stops being a hand-deployed static file (see the `STATIC_PATHS` comment in `pelicanconf.py`).
-* **Make the URL space path-shaped instead.** gitweb's `pathinfo` feature (`$feature{'pathinfo'}{'default'} = [1]`) yields `/blog/source/repo.git/blob_plain/HEAD:/notes/foo.md`, after which ordinary path prefixes do the whole job with no query-matching question. Cost: it changes every link gitweb generates, and the long `@stylesheets` comment in `provisioning/gitweb.conf` is premised on the query-string style (its whole argument is that gitweb never emits a `<base href>` — under pathinfo it does), so re-verify the CSS/JS still load.
+**What does work, if this is revisited: links.** The site has run the experiment. The `.md` mirrors are fetched heavily and are linked from HTML (`theme/templates/article.html`, `includes/post_card.html`); `llms.txt` is a root convention that nothing links to and has **zero fetches across all 45 digests** (2026-07-25 → 09-07). Convention alone gets nothing crawled here. The Diary links added to `provisioning/root_index.html` in `186d3f3` are the right shape for this. A sitemap is the exception that proves it — it's the one mechanism designed to advertise unlinked URLs — but note that its reliable consumers are *search* crawlers; no digest evidence shows an AI training crawler following a `Sitemap:` line.
-**Measurement gotcha:** `ai_bot_digest.py:313` drops the query string before counting, so the digests cannot tell which gitweb *views* are being hammered, and can't confirm afterward that an `Allow:` exception is being honored — they'd just show `/blog/source` going down. The raw access log keeps the full request line; checking the effect means grepping that, or teaching the digest to bucket by `a=` parameter.
+**Don't expect politeness headers to help with load.** gitweb already emits `<meta name="robots" content="index, nofollow"/>` on every HTML page (`/usr/share/gitweb/gitweb.cgi:4214`) and `nginx_siteconf` already sets `X-Robots-Tag: noindex` on the location — and the trap still drew 312,894 requests in a day. `nofollow` is about endorsement, not crawl budget; `blob_plain` responses aren't HTML and carry no meta tag at all; `X-Robots-Tag` governs indexing rather than fetching. **robots.txt is the only lever here that reduces requests.**
-**Don't expect politeness headers to help with load.** gitweb already emits `<meta name="robots" content="index, nofollow"/>` on every HTML page (`/usr/share/gitweb/gitweb.cgi:4214`) and the trap still drew 312,894 requests in a day. That isn't proof of defiance — `nofollow` is about endorsement/ranking, not crawl budget — but it's the shape of the lesson: `blob_plain` responses aren't HTML and carry no meta tag at all, and an nginx `X-Robots-Tag` (which *could* key off `$args` via a `map`) governs indexing rather than fetching. **robots.txt is the only lever here that reduces requests.**
+**Measurement gotcha:** `ai_bot_digest.py:313` drops the query string before counting, so the digests still cannot tell which gitweb *views* were being hammered — only that `/blog/source` totals should now fall. Confirming *which* bots ignore the `Disallow` means grepping the raw access log, or teaching the digest to bucket by `a=` parameter.
-### Sitemap as the draft-discovery mechanism (undecided, same thread)
+### robots.txt: a blank line or a rule order can silently void the whole file
-A sitemap is exactly the "advertise URLs nothing links to" mechanism, which is precisely the drafts' problem — so the missing `sitemap.xml` noted below is the natural home for this, listing `/blog/drafts/<slug>.html` alongside (or instead of) the published posts. Constraints:
+Two footguns, both of which make rules *look* present while doing nothing, and both hit while adding `Disallow: /blog/source` on 2026-09-09 (caught only by testing the file, not by reading it):
-* **A sitemap can't override robots.txt.** A `Disallow`ed URL doesn't get fetched just because it's listed. Via `/blog/drafts/` that's a non-issue (already allowed); via gitweb blob URLs the exact `Allow:` lines above become mandatory, not optional.
-* **Placement is the same problem `llms.txt` has** — it must be at the true domain root, so it needs copying out of `output/` in `provisioning/pelican_scheduler.py` (or generating straight to the webroot), plus a `Sitemap:` line in `robots.txt`, which is reliably fetched.
-* **Decide whether drafts go in the same sitemap as published posts.** Listing them invites ordinary search indexing too, not just AI ingestion — a half-finished post ranking for its own title is a different outcome than a model having read it. If the want is read-but-don't-index, that's `X-Robots-Tag: noindex` on a `location /blog/drafts` block, with the caveat that training crawlers have no particular obligation to honor it (that's the point, here). A separate `drafts-sitemap.xml` keeps the two audiences distinguishable.
-* **Scope note:** any gitweb-side version of this exposes whatever else is in the working tree, not just the files aimed at — `notes/` currently holds crawler-digest session logs and a tag proposal alongside anything else that lands there. Enumerating files in `Allow:` lines is the version that stays deliberate; a `Disallow`-with-`f=notes/*`-hole is the version that quietly publishes the next thing added.
+* **A blank line ends the `User-Agent` group.** Every `Disallow`/`Allow` after it is discarded — they're only recorded while the parser is inside a group. Putting the new `Disallow` in its own visually-tidy stanza below `Allow: /` meant *no rule applied at all*. Keep the rules flush against `User-Agent:` with no blank line anywhere among them.
+* **`Disallow` must come before `Allow: /`.** Longest-match precedence (`/blog/source` beating `/`) is Google's rule and RFC 9309's, but it is *not* universal: Python's `urllib.robotparser` takes the **first** matching rule, so `Allow: /` first leaves gitweb fully allowed. Disallow-first is correct under both semantics, so it's the portable order.
+
+`Sitemap:` is exempt from all of this — the sitemaps.org protocol makes it independent of the user-agent line ("it doesn't matter where you place it in your file"), and stdlib honors that explicitly, so the trailing stanza is fine.
+
+Check changes rather than eyeballing them:
+
+```
+python3 -c "import urllib.robotparser as rp; p=rp.RobotFileParser(); \
+ p.parse(open('provisioning/robots.txt').read().splitlines()); \
+ print(p.site_maps()); print(p.can_fetch('ClaudeBot','https://zackmdavis.net/blog/source'))"
+```
+
+### Sitemap (built 2026-09-09)
+
+`_write_sitemap` in `pelicanconf.py` emits `sitemap.xml` on the `finalized` signal, listing the 495 published posts and nothing else — no drafts (above), no `.md` mirrors (same content at a second URL; already reachable via `<link rel="alternate">`), no tag/category/archive indexes (navigation, not content). Justified independently of any crawler-ingestion question: a sitemap is worth having for ordinary search discovery.
+
+* **Placement — two rules that get conflated.** *Scope*: the sitemaps.org protocol bounds a sitemap by its own location, so one at `/blog/sitemap.xml` may only list URLs under `/blog/`, which every URL here satisfies anyway. *Discovery*: crawlers look for `/sitemap.xml` at the true domain root. Both are handled outside `pelicanconf.py` — the deploy hook installs a copy at the webroot, and `robots.txt` carries a `Sitemap:` line naming it. This is a **genuine difference from `llms.txt`**, which has no equivalent of the `Sitemap:` directive: nothing at the root can redirect attention to a copy elsewhere, so for that file the physical move is the whole fix rather than a redundancy.
+* **The install step lives in `SITEGEN_COMMAND`, not `main()`.** That string has two callers: `main()` runs it on push, and `schedule()` hands the identical string to `at` for each future-dated post — `at` executes that string alone, and `pelican_scheduler.py` does not run again. A copy step in `main()` would be skipped by every scheduled run, so a post going live at midnight would regenerate `output/` with a correct fresh sitemap while the webroot copy stayed at whatever the last push produced. Silently stale, in precisely the situation the sitemap exists to announce.
+* **`lastmod` is `Modified:` if the post sets one, else the publication date, clamped to the build date.** Nothing in the corpus sets `Modified:` today, so the clamp is near-inert (`WITH_FUTURE_DATES = False` means published posts aren't future-dated) — kept because a future `<lastmod>` is out of spec and crawlers discard implausible values, which would cost the freshness signal silently. Adding `Modified:` when back-editing a post is what would make `lastmod` genuinely useful; `DEFAULT_DATE = 'fs'` is *not* the way to get it, since it would overwrite every post's publication date too.
+* **A sitemap can't override robots.txt.** A `Disallow`ed URL isn't fetched just because it's listed — which is why nothing under `/blog/source` appears here.
### `llms.txt` is generated where nothing will find it (unfixed, low priority)
-`_write_llms_txt` in `pelicanconf.py` emits it into the Pelican output tree, so it lands at `/blog/llms.txt` — but `llms.txt` is a domain-root convention like `robots.txt`, so a crawler that goes looking checks `zackmdavis.net/llms.txt` and gets a 404. Nothing links to the real one either, from `robots.txt` or from any page. It has **zero fetches** across every digest in `notes/ai_bot_digests/` (2026-07-25 through 2026-08-03) — the file is currently dead weight.
+`_write_llms_txt` in `pelicanconf.py` emits it into the Pelican output tree, so it lands at `/blog/llms.txt` — but `llms.txt` is a domain-root convention like `robots.txt`, so a crawler that goes looking checks `zackmdavis.net/llms.txt` and gets a 404. Nothing links to the real one either, from `robots.txt` or from any page. It has **zero fetches** across every digest in `notes/ai_bot_digests/` (2026-07-25 through 2026-09-07, all 45 digests) — the file is currently dead weight.
Note this means the heavy `.md` fetching visible in those digests is *not* evidence llms.txt works: the mirrors are reachable from the HTML anyway, via the `<link rel="alternate">` in `theme/templates/article.html` and the visible "Markdown source" link in `theme/templates/includes/post_card.html`.
-The fix is placement, not content — `_canonical_url` already builds absolute `https://zackmdavis.net/blog/...` links, so a copy served from the webroot works unmodified. It's the same problem `STATIC_PATHS` in `pelicanconf.py` already solves for `robots.txt` by deploying that standalone, but `llms.txt` is build-generated, so it needs copying out of `output/` after each build (`provisioning/pelican_scheduler.py`) rather than a one-time `scp`. While in there: there's no `sitemap.xml` at all and no `Sitemap:` line in `robots.txt`, which is the same missing-machine-discovery-surface problem — robots.txt *is* reliably fetched (ClaudeBot, Applebot, PerplexityBot, OAI-SearchBot all hit it), so it's the better hook of the two.
+The fix is placement, not content — `_canonical_url` already builds absolute `https://zackmdavis.net/blog/...` links, so a copy served from the webroot works unmodified. It's the same problem `STATIC_PATHS` in `pelicanconf.py` already solves for `robots.txt` by deploying that standalone, but `llms.txt` is build-generated, so it needs copying out of `output/` after each build (`provisioning/pelican_scheduler.py`) rather than a one-time `scp`. The `sitemap.xml` half of this is now done (see above) and is the model to copy: the deploy hook already has a webroot-install step to extend. But note the asymmetry — the sitemap can lean on robots.txt's `Sitemap:` directive for discovery, and `llms.txt` has no such directive, so moving the file is the entire fix rather than a belt-and-braces addition.
### Internal links are root-relative on purpose — don't "fix" them
_signals.article_generator_finalized.connect(_prepare_markdown_mirrors)
_signals.finalized.connect(_write_markdown_mirrors)
_signals.finalized.connect(_write_llms_txt)
+
+
+# --- sitemap.xml ------------------------------------------------------------
+# Published posts only, and deliberately so. Drafts were considered and left
+# out (2026-09-09): a draft and its eventual published version live at
+# different URLs -- drafts/<slug>.html vs YYYY/Mon/<slug>/ -- and the draft URL
+# 404s once the post moves out of content/drafts/. A crawler that captured the
+# draft therefore has no live page to revisit and no canonical link to the
+# finished piece, so a superseded take persists uncorrected in a way that an
+# edited published post (stable URL, re-crawled) never does. Listing drafts
+# traded a reversible gain for an irreversible cost.
+#
+# Also left out: the .md mirrors (same content as the HTML at a second URL --
+# listing both invites duplicate-content handling; the mirrors are already
+# reachable via <link rel="alternate">) and the tag/category/archive indexes
+# (navigation, not content).
+#
+# On placement, two rules get conflated. *Scope*: the sitemaps.org protocol
+# bounds a sitemap by its own location, so one at /blog/sitemap.xml could only
+# list URLs under /blog/ -- which every URL here satisfies anyway. *Discovery*:
+# crawlers look for /sitemap.xml at the true domain root, and this site is
+# served under /blog. Both are handled outside this file: the deploy hook
+# installs a copy at the webroot (provisioning/pelican_scheduler.py) and
+# robots.txt carries a Sitemap: line naming it (provisioning/robots.txt).
+#
+# Note that's a genuine difference from llms.txt, which has no equivalent of
+# the Sitemap: directive -- nothing at the root can redirect attention to a
+# copy elsewhere -- so for that file the physical move is the whole fix.
+from datetime import date as _date
+from xml.sax.saxutils import escape as _xml_escape
+
+
+def _write_sitemap(pelican_obj):
+ output_path = _markdown_mirror_state['output_path']
+ if not output_path:
+ return
+ # A sitemap must carry absolute URLs, so there's nothing meaningful to
+ # write in a local build where SITEURL is '' -- an invalid sitemap is worse
+ # than none. publishconf.py always sets it, so production is unaffected.
+ # (Same guard, same reason, as _absolutize_site_links.)
+ if not _markdown_mirror_state['siteurl']:
+ return
+ today = _date.today()
+ articles = sorted(_markdown_mirror_state['articles'],
+ key=lambda a: a.date, reverse=True)
+ lines = ['<?xml version="1.0" encoding="UTF-8"?>',
+ '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">']
+ for article in articles:
+ # Modified: is honored where a post sets one, so a back-edit can
+ # advertise itself; nothing in the corpus does today, in which case
+ # this is the publication date. min() against today because a
+ # <lastmod> in the future is out of spec and crawlers discard
+ # implausible values -- which would cost the freshness signal
+ # silently rather than loudly.
+ lastmod = getattr(article, 'modified', None) or article.date
+ lines.append(
+ " <url><loc>{}</loc><lastmod>{}</lastmod></url>".format(
+ _xml_escape(_canonical_url(article.url)),
+ min(lastmod.date(), today).strftime('%Y-%m-%d')))
+ lines.append('</urlset>')
+ with open(_os.path.join(output_path, 'sitemap.xml'), 'w',
+ encoding='utf-8') as f:
+ f.write('\n'.join(lines) + '\n')
+
+
+_signals.finalized.connect(_write_sitemap)
INPUT_DIR = os.path.join(WORKING_REPO, "content")
OUTPUT_DIR = "/home/blogmistress/An_Algorithmic_Lucidity/output"
PUBLISH_CONF = os.path.join(WORKING_REPO, "publishconf.py")
-SITEGEN_COMMAND = "bash -c 'cd {} && source .venv/bin/activate && pelican {} -o {} -s {}'".format(
- WORKING_REPO, INPUT_DIR, OUTPUT_DIR, PUBLISH_CONF)
+# The true domain root (nginx's `root` in provisioning/nginx_siteconf), as
+# opposed to the Pelican tree, which is served under /blog via `alias`.
+WEBROOT = "/home/blogmistress/zackmdavis.net"
+
+# sitemap.xml is generated into the Pelican tree (see _write_sitemap in
+# pelicanconf.py) but has to be *served* from the domain root, since that's
+# where crawlers look for it -- the same placement problem robots.txt has,
+# except robots.txt can be a one-time scp because nothing regenerates it,
+# whereas this is rebuilt on every deploy and so must be reinstalled on every
+# deploy.
+#
+# It belongs in SITEGEN_COMMAND rather than in main() because this command has
+# two callers: main() runs it now, and schedule() hands the very same string to
+# `at` for each future-dated post. `at` executes that string alone -- this
+# script does not run again -- so a copy step living in main() would be skipped
+# by every scheduled run. A post going live at midnight would regenerate
+# output/ (with a fresh sitemap correctly listing it) while the webroot copy
+# stayed at whatever the last push produced: silently stale, in exactly the
+# situation the sitemap exists to announce.
+#
+# Chained with && so a failed build leaves the previous sitemap in place rather
+# than replacing it with a partial one, and `install -m 644` sets the mode
+# explicitly instead of inheriting whatever the build left (matching how
+# everything else here gets deployed).
+SITEGEN_COMMAND = (
+ "bash -c 'cd {repo} && source .venv/bin/activate "
+ "&& pelican {input} -o {output} -s {conf} "
+ "&& install -m 644 {output}/sitemap.xml {webroot}/sitemap.xml'"
+).format(repo=WORKING_REPO, input=INPUT_DIR, output=OUTPUT_DIR,
+ conf=PUBLISH_CONF, webroot=WEBROOT)
DATELINE_REGEX = re.compile(r"^Date: *(\d{4}-\d{2}-\d{2} \d{2}:\d{2}) *$",
re.MULTILINE)
projects / An_Algorithmic_Lucidity.git / commitdiff