| `pelican_scheduler.py` | symlinked as the bare repo's `hooks/post-receive` |
| `root_index.html` | `/home/blogmistress/zackmdavis.net/index.html` — **renamed on the way over**; scp'ing it under its repo name lands a dead file beside the real one and changes nothing visible |
| `robots.txt` | `/home/blogmistress/zackmdavis.net/robots.txt` (the true domain root, not `/blog` — see the `STATIC_PATHS` comment in `pelicanconf.py` for why Pelican deliberately doesn't generate it) |
-| *(generated)* | `/home/blogmistress/zackmdavis.net/sitemap.xml` — **never deployed by hand**: `_write_sitemap` builds it into `output/` and `SITEGEN_COMMAND` in `pelican_scheduler.py` installs it at the webroot on every build. Editing the live file is pointless; the next deploy overwrites it. |
+| *(generated)* | `/home/blogmistress/zackmdavis.net/sitemap.xml` and `llms.txt` are **symlinks** into `/home/blogmistress/An_Algorithmic_Lucidity/output/`, which the build regenerates. Created once by hand (`ln -sfn`), never deployed. Untracked server state — recreate both if the box is rebuilt. |
`root_index.html` and `robots.txt` live under the webroot — `root /home/blogmistress/zackmdavis.net` in `nginx_siteconf`, which is also what serves `/docs` and `/media` (and now `sitemap.xml`, though the hook puts that one there). Being inside `blogmistress`'s own home, they're the only two deployable without root; `install` as root would leave them root-owned.
`_write_sitemap` in `pelicanconf.py` emits `sitemap.xml` on the `finalized` signal, listing the 495 published posts and nothing else — no drafts (above), no `.md` mirrors (same content at a second URL; already reachable via `<link rel="alternate">`), no tag/category/archive indexes (navigation, not content). Justified independently of any crawler-ingestion question: a sitemap is worth having for ordinary search discovery.
* **Placement — two rules that get conflated.** *Scope*: the sitemaps.org protocol bounds a sitemap by its own location, so one at `/blog/sitemap.xml` may only list URLs under `/blog/`, which every URL here satisfies anyway. *Discovery*: crawlers look for `/sitemap.xml` at the true domain root. Both are handled outside `pelicanconf.py` — the deploy hook installs a copy at the webroot, and `robots.txt` carries a `Sitemap:` line naming it. This is a **genuine difference from `llms.txt`**, which has no equivalent of the `Sitemap:` directive: nothing at the root can redirect attention to a copy elsewhere, so for that file the physical move is the whole fix rather than a redundancy.
-* **The install step lives in `SITEGEN_COMMAND`, not `main()`.** That string has two callers: `main()` runs it on push, and `schedule()` hands the identical string to `at` for each future-dated post — `at` executes that string alone, and `pelican_scheduler.py` does not run again. A copy step in `main()` would be skipped by every scheduled run, so a post going live at midnight would regenerate `output/` with a correct fresh sitemap while the webroot copy stayed at whatever the last push produced. Silently stale, in precisely the situation the sitemap exists to announce.
+* **The webroot copy is a symlink, not a deploy step.** An earlier version had `pelican_scheduler.py` `install`-ing the file on every build; that was replaced by a one-time symlink, which is always current, keeps the hook out of the webroot, and needs no write permission there. It also sidesteps the hook-staleness trap below. The one caveat: `DELETE_OUTPUT_DIRECTORY = True` wipes `output/` at the start of each build, so the symlink dangles until the build finishes — during which the whole site is 404ing anyway (see the deploy-window gotcha above). The `rsync --checksum --delete` fix for that window would eliminate this too.
+* **`llms.txt` got the same treatment** on 2026-09-10 — see its section below.
* **`lastmod` is `Modified:` if the post sets one, else the publication date, clamped to the build date.** Nothing in the corpus sets `Modified:` today, so the clamp is near-inert (`WITH_FUTURE_DATES = False` means published posts aren't future-dated) — kept because a future `<lastmod>` is out of spec and crawlers discard implausible values, which would cost the freshness signal silently. Adding `Modified:` when back-editing a post is what would make `lastmod` genuinely useful; `DEFAULT_DATE = 'fs'` is *not* the way to get it, since it would overwrite every post's publication date too.
* **A sitemap can't override robots.txt.** A `Disallow`ed URL isn't fetched just because it's listed — which is why nothing under `/blog/source` appears here.
-### `llms.txt` is generated where nothing will find it (unfixed, low priority)
+### `llms.txt` placement fixed by symlink (2026-09-10)
-`_write_llms_txt` in `pelicanconf.py` emits it into the Pelican output tree, so it lands at `/blog/llms.txt` — but `llms.txt` is a domain-root convention like `robots.txt`, so a crawler that goes looking checks `zackmdavis.net/llms.txt` and gets a 404. Nothing links to the real one either, from `robots.txt` or from any page. It has **zero fetches** across every digest in `notes/ai_bot_digests/` (2026-07-25 through 2026-09-07, all 45 digests) — the file is currently dead weight.
+`_write_llms_txt` emits into the Pelican output tree, so it lands at `/blog/llms.txt` — but `llms.txt` is a domain-root convention like `robots.txt`, and a crawler that goes looking checks `zackmdavis.net/llms.txt`. **Fixed the same way as `sitemap.xml`:** `/home/blogmistress/zackmdavis.net/llms.txt` is a symlink to `output/llms.txt`. Verified live (200, `text/plain; charset=utf-8`). No code change was needed — `_canonical_url` already builds absolute `https://zackmdavis.net/blog/...` links, so the file works unmodified from the root.
-Note this means the heavy `.md` fetching visible in those digests is *not* evidence llms.txt works: the mirrors are reachable from the HTML anyway, via the `<link rel="alternate">` in `theme/templates/article.html` and the visible "Markdown source" link in `theme/templates/includes/post_card.html`.
+Unlike `sitemap.xml`, there's no `Sitemap:`-style directive that could have pointed at a copy elsewhere, so relocation was the entire fix rather than a redundancy.
-The fix is placement, not content — `_canonical_url` already builds absolute `https://zackmdavis.net/blog/...` links, so a copy served from the webroot works unmodified. It's the same problem `STATIC_PATHS` in `pelicanconf.py` already solves for `robots.txt` by deploying that standalone, but `llms.txt` is build-generated, so it needs copying out of `output/` after each build (`provisioning/pelican_scheduler.py`) rather than a one-time `scp`. The `sitemap.xml` half of this is now done (see above) and is the model to copy: the deploy hook already has a webroot-install step to extend. But note the asymmetry — the sitemap can lean on robots.txt's `Sitemap:` directive for discovery, and `llms.txt` has no such directive, so moving the file is the entire fix rather than a belt-and-braces addition.
+**What this does and doesn't settle.** It had **zero fetches** across all 45 digests (2026-07-25 → 09-07) while it sat at `/blog/llms.txt`, but that was consistent with two different explanations — nobody looks for `llms.txt`, or nobody could find this one. Only the second is now excluded, so the digests finally measure the interesting question. Watch for `/llms.txt` hits; sustained zeroes now mean the convention itself is dead here.
+
+Note the heavy `.md` fetching in those digests is *not* evidence llms.txt works: the mirrors are reachable from the HTML anyway, via the `<link rel="alternate">` in `theme/templates/article.html` and the visible "Markdown source" link in `theme/templates/includes/post_card.html`.
+
+Same caveat as the sitemap symlink: `DELETE_OUTPUT_DIRECTORY = True` wipes `output/` at the start of each build, so both links dangle until it finishes — during which the whole site 404s anyway.
### Internal links are root-relative on purpose — don't "fix" them
INPUT_DIR = os.path.join(WORKING_REPO, "content")
OUTPUT_DIR = "/home/blogmistress/An_Algorithmic_Lucidity/output"
PUBLISH_CONF = os.path.join(WORKING_REPO, "publishconf.py")
-# The true domain root (nginx's `root` in provisioning/nginx_siteconf), as
-# opposed to the Pelican tree, which is served under /blog via `alias`.
-WEBROOT = "/home/blogmistress/zackmdavis.net"
-
-# sitemap.xml is generated into the Pelican tree (see _write_sitemap in
-# pelicanconf.py) but has to be *served* from the domain root, since that's
-# where crawlers look for it -- the same placement problem robots.txt has,
-# except robots.txt can be a one-time scp because nothing regenerates it,
-# whereas this is rebuilt on every deploy and so must be reinstalled on every
-# deploy.
-#
-# It belongs in SITEGEN_COMMAND rather than in main() because this command has
-# two callers: main() runs it now, and schedule() hands the very same string to
-# `at` for each future-dated post. `at` executes that string alone -- this
-# script does not run again -- so a copy step living in main() would be skipped
-# by every scheduled run. A post going live at midnight would regenerate
-# output/ (with a fresh sitemap correctly listing it) while the webroot copy
-# stayed at whatever the last push produced: silently stale, in exactly the
-# situation the sitemap exists to announce.
-#
-# Chained with && so a failed build leaves the previous sitemap in place rather
-# than replacing it with a partial one, and `install -m 644` sets the mode
-# explicitly instead of inheriting whatever the build left (matching how
-# everything else here gets deployed).
-SITEGEN_COMMAND = (
- "bash -c 'cd {repo} && source .venv/bin/activate "
- "&& pelican {input} -o {output} -s {conf} "
- "&& install -m 644 {output}/sitemap.xml {webroot}/sitemap.xml'"
-).format(repo=WORKING_REPO, input=INPUT_DIR, output=OUTPUT_DIR,
- conf=PUBLISH_CONF, webroot=WEBROOT)
+SITEGEN_COMMAND = "bash -c 'cd {} && source .venv/bin/activate && pelican {} -o {} -s {}'".format(
+ WORKING_REPO, INPUT_DIR, OUTPUT_DIR, PUBLISH_CONF)
DATELINE_REGEX = re.compile(r"^Date: *(\d{4}-\d{2}-\d{2} \d{2}:\d{2}) *$",
re.MULTILINE)
projects / An_Algorithmic_Lucidity.git / commitdiff