* **Placement — two rules that get conflated.** *Scope*: the sitemaps.org protocol bounds a sitemap by its own location, so one at `/blog/sitemap.xml` may only list URLs under `/blog/`, which every URL here satisfies anyway. *Discovery*: crawlers look for `/sitemap.xml` at the true domain root. Both are handled outside `pelicanconf.py` — the deploy hook installs a copy at the webroot, and `robots.txt` carries a `Sitemap:` line naming it. This is a **genuine difference from `llms.txt`**, which has no equivalent of the `Sitemap:` directive: nothing at the root can redirect attention to a copy elsewhere, so for that file the physical move is the whole fix rather than a redundancy.
* **The webroot copy is a symlink, not a deploy step.** An earlier version had `pelican_scheduler.py` `install`-ing the file on every build; that was replaced by a one-time symlink, which is always current, keeps the hook out of the webroot, and needs no write permission there. It also sidesteps the hook-staleness trap below. The one caveat: `DELETE_OUTPUT_DIRECTORY = True` wipes `output/` at the start of each build, so the symlink dangles until the build finishes — during which the whole site is 404ing anyway (see the deploy-window gotcha above). The `rsync --checksum --delete` fix for that window would eliminate this too.
* **`llms.txt` got the same treatment** on 2026-09-10 — see its section below.
-* **`lastmod` is `Modified:` if the post sets one, else the publication date, clamped to the build date.** Nothing in the corpus sets `Modified:` today, so the clamp is near-inert (`WITH_FUTURE_DATES = False` means published posts aren't future-dated) — kept because a future `<lastmod>` is out of spec and crawlers discard implausible values, which would cost the freshness signal silently. Adding `Modified:` when back-editing a post is what would make `lastmod` genuinely useful; `DEFAULT_DATE = 'fs'` is *not* the way to get it, since it would overwrite every post's publication date too.
+* **`lastmod` comes from git, not from the post's `Date:` header.** Publication date was simply wrong: **491 of 495 posts have been edited since publication** (the WordPress import on 2026-07-15, then the root-relative link sweep on 2026-08-07), so `Date:` understated every one of them. `Modified:` in the post header would be the tidy fix but requires remembering it on every edit forever; git already knows. `_git_last_modified` in `pelicanconf.py` runs **one** `git log --name-only` over `content/` and maps source path → latest commit date.
+ * **One pass, not one call per file** — measured at ~35 ms for the whole corpus versus ~1.9 s for per-file calls, i.e. 1% of build time rather than half of it. Don't "simplify" it into a loop of `git log -1`.
+ * Commit lines are prefixed with a **NUL sentinel** (`--format=%x00%cI`) because NUL is the one byte a path can't contain, so commit lines and filenames are never confused.
+ * On any failure — not a git checkout, shallow clone, no `git` binary — it returns `{}` and the sitemap **omits** `<lastmod>` for every URL. That's legal (it's optional per-URL) and better than inventing a date. Same for an individual uncommitted post in a local build.
+ * The resulting dates cluster hard (381 posts on 2026-07-15) because that's genuinely when those files last changed. That's a one-time historical artifact, not an ongoing defect: from here on, editing one post moves exactly that post's date. A future bulk rewrite of the corpus *will* bump everything again, and that will be accurate.
+ * `DEFAULT_DATE = 'fs'` is *not* a shortcut to this — it would overwrite every post's publication date too.
* **A sitemap can't override robots.txt.** A `Disallow`ed URL isn't fetched just because it's listed — which is why nothing under `/blog/source` appears here.
### `llms.txt` placement fixed by symlink (2026-09-10)
# copy elsewhere -- so for that file the physical move is the whole fix.
from datetime import date as _date
from xml.sax.saxutils import escape as _xml_escape
+import subprocess as _subprocess
+
+_REPO_ROOT = _os.path.dirname(_os.path.abspath(__file__))
+
+
+def _git_last_modified():
+ """Source path (repo-relative) -> 'YYYY-MM-DD' of its latest commit.
+
+ <lastmod> is supposed to be the last time the content *changed*, which the
+ post's Date: header does not track -- 491 of 495 posts have been edited
+ since publication (the WordPress import, then the 2026-08-07 root-relative
+ link sweep), so publishing Date: as lastmod was simply false. Modified: in
+ the post header would be the tidy answer but requires remembering to set it
+ on every edit forever; git already knows, exactly, for free.
+
+ One `git log` for the whole corpus rather than one per file: ~35ms vs ~1.9s
+ across ~500 posts, i.e. 1% of build time rather than half of it.
+
+ A NUL sentinel starts each commit line because NUL is the one byte a path
+ cannot contain, so commit lines and filenames can't be confused.
+
+ Returns {} on any failure (not a git checkout, shallow clone, git absent),
+ in which case the sitemap just omits <lastmod> -- which is legal per-URL,
+ and better than inventing a date.
+ """
+ try:
+ out = _subprocess.run(
+ ["git", "-C", _REPO_ROOT, "log", "--format=%x00%cI",
+ "--name-only", "--no-renames", "--", PATH],
+ capture_output=True, text=True, timeout=30, check=True).stdout
+ except (OSError, _subprocess.SubprocessError):
+ return {}
+ dates = {}
+ stamp = None
+ for line in out.splitlines():
+ if line.startswith("\0"):
+ stamp = line[1:11] # YYYY-MM-DD off the front of %cI
+ elif line and stamp:
+ dates.setdefault(line, stamp) # log is newest-first, so first wins
+ return dates
def _write_sitemap(pelican_obj):
# (Same guard, same reason, as _absolutize_site_links.)
if not _markdown_mirror_state['siteurl']:
return
- today = _date.today()
+ today = _date.today().strftime('%Y-%m-%d')
+ git_dates = _git_last_modified()
articles = sorted(_markdown_mirror_state['articles'],
key=lambda a: a.date, reverse=True)
lines = ['<?xml version="1.0" encoding="UTF-8"?>',
'<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">']
for article in articles:
- # Modified: is honored where a post sets one, so a back-edit can
- # advertise itself; nothing in the corpus does today, in which case
- # this is the publication date. min() against today because a
- # <lastmod> in the future is out of spec and crawlers discard
- # implausible values -- which would cost the freshness signal
- # silently rather than loudly.
- lastmod = getattr(article, 'modified', None) or article.date
- lines.append(
- " <url><loc>{}</loc><lastmod>{}</lastmod></url>".format(
- _xml_escape(_canonical_url(article.url)),
- min(lastmod.date(), today).strftime('%Y-%m-%d')))
+ loc = _xml_escape(_canonical_url(article.url))
+ stamp = git_dates.get(
+ _os.path.relpath(article.source_path, _REPO_ROOT))
+ if stamp is None:
+ # Uncommitted (a new post in a local build) -- omit rather than
+ # substitute the publication date, which would be a guess.
+ lines.append(" <url><loc>{}</loc></url>".format(loc))
+ continue
+ # Clamp: a <lastmod> in the future is out of spec and crawlers discard
+ # implausible values. Only reachable via a skewed committer clock, but
+ # the failure would be silent, so it's cheap to rule out.
+ lines.append(" <url><loc>{}</loc><lastmod>{}</lastmod></url>".format(
+ loc, min(stamp, today)))
lines.append('</urlset>')
with open(_os.path.join(output_path, 'sitemap.xml'), 'w',
encoding='utf-8') as f:
projects / An_Algorithmic_Lucidity.git / commitdiff