From: Zack M. Davis Date: Thu, 10 Sep 2026 06:22:42 +0000 (-0700) Subject: sitemap lastmod from Git X-Git-Url: http://unremediatedgender.space/source?a=commitdiff_plain;h=52488a5fe9749a05d1563ac2ecbb7d70f6d249e9;p=An_Algorithmic_Lucidity.git sitemap lastmod from Git --- diff --git a/CLAUDE.md b/CLAUDE.md index 7d523fc..43eefea 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -136,7 +136,12 @@ The diaries are linked from `provisioning/root_index.html` (commit `186d3f3`), s * **Placement — two rules that get conflated.** *Scope*: the sitemaps.org protocol bounds a sitemap by its own location, so one at `/blog/sitemap.xml` may only list URLs under `/blog/`, which every URL here satisfies anyway. *Discovery*: crawlers look for `/sitemap.xml` at the true domain root. Both are handled outside `pelicanconf.py` — the deploy hook installs a copy at the webroot, and `robots.txt` carries a `Sitemap:` line naming it. This is a **genuine difference from `llms.txt`**, which has no equivalent of the `Sitemap:` directive: nothing at the root can redirect attention to a copy elsewhere, so for that file the physical move is the whole fix rather than a redundancy. * **The webroot copy is a symlink, not a deploy step.** An earlier version had `pelican_scheduler.py` `install`-ing the file on every build; that was replaced by a one-time symlink, which is always current, keeps the hook out of the webroot, and needs no write permission there. It also sidesteps the hook-staleness trap below. The one caveat: `DELETE_OUTPUT_DIRECTORY = True` wipes `output/` at the start of each build, so the symlink dangles until the build finishes — during which the whole site is 404ing anyway (see the deploy-window gotcha above). The `rsync --checksum --delete` fix for that window would eliminate this too. * **`llms.txt` got the same treatment** on 2026-09-10 — see its section below. -* **`lastmod` is `Modified:` if the post sets one, else the publication date, clamped to the build date.** Nothing in the corpus sets `Modified:` today, so the clamp is near-inert (`WITH_FUTURE_DATES = False` means published posts aren't future-dated) — kept because a future `` is out of spec and crawlers discard implausible values, which would cost the freshness signal silently. Adding `Modified:` when back-editing a post is what would make `lastmod` genuinely useful; `DEFAULT_DATE = 'fs'` is *not* the way to get it, since it would overwrite every post's publication date too. +* **`lastmod` comes from git, not from the post's `Date:` header.** Publication date was simply wrong: **491 of 495 posts have been edited since publication** (the WordPress import on 2026-07-15, then the root-relative link sweep on 2026-08-07), so `Date:` understated every one of them. `Modified:` in the post header would be the tidy fix but requires remembering it on every edit forever; git already knows. `_git_last_modified` in `pelicanconf.py` runs **one** `git log --name-only` over `content/` and maps source path → latest commit date. + * **One pass, not one call per file** — measured at ~35 ms for the whole corpus versus ~1.9 s for per-file calls, i.e. 1% of build time rather than half of it. Don't "simplify" it into a loop of `git log -1`. + * Commit lines are prefixed with a **NUL sentinel** (`--format=%x00%cI`) because NUL is the one byte a path can't contain, so commit lines and filenames are never confused. + * On any failure — not a git checkout, shallow clone, no `git` binary — it returns `{}` and the sitemap **omits** `` for every URL. That's legal (it's optional per-URL) and better than inventing a date. Same for an individual uncommitted post in a local build. + * The resulting dates cluster hard (381 posts on 2026-07-15) because that's genuinely when those files last changed. That's a one-time historical artifact, not an ongoing defect: from here on, editing one post moves exactly that post's date. A future bulk rewrite of the corpus *will* bump everything again, and that will be accurate. + * `DEFAULT_DATE = 'fs'` is *not* a shortcut to this — it would overwrite every post's publication date too. * **A sitemap can't override robots.txt.** A `Disallow`ed URL isn't fetched just because it's listed — which is why nothing under `/blog/source` appears here. ### `llms.txt` placement fixed by symlink (2026-09-10) diff --git a/pelicanconf.py b/pelicanconf.py index 5c39e0f..088f56d 100644 --- a/pelicanconf.py +++ b/pelicanconf.py @@ -264,6 +264,46 @@ _signals.finalized.connect(_write_llms_txt) # copy elsewhere -- so for that file the physical move is the whole fix. from datetime import date as _date from xml.sax.saxutils import escape as _xml_escape +import subprocess as _subprocess + +_REPO_ROOT = _os.path.dirname(_os.path.abspath(__file__)) + + +def _git_last_modified(): + """Source path (repo-relative) -> 'YYYY-MM-DD' of its latest commit. + + is supposed to be the last time the content *changed*, which the + post's Date: header does not track -- 491 of 495 posts have been edited + since publication (the WordPress import, then the 2026-08-07 root-relative + link sweep), so publishing Date: as lastmod was simply false. Modified: in + the post header would be the tidy answer but requires remembering to set it + on every edit forever; git already knows, exactly, for free. + + One `git log` for the whole corpus rather than one per file: ~35ms vs ~1.9s + across ~500 posts, i.e. 1% of build time rather than half of it. + + A NUL sentinel starts each commit line because NUL is the one byte a path + cannot contain, so commit lines and filenames can't be confused. + + Returns {} on any failure (not a git checkout, shallow clone, git absent), + in which case the sitemap just omits -- which is legal per-URL, + and better than inventing a date. + """ + try: + out = _subprocess.run( + ["git", "-C", _REPO_ROOT, "log", "--format=%x00%cI", + "--name-only", "--no-renames", "--", PATH], + capture_output=True, text=True, timeout=30, check=True).stdout + except (OSError, _subprocess.SubprocessError): + return {} + dates = {} + stamp = None + for line in out.splitlines(): + if line.startswith("\0"): + stamp = line[1:11] # YYYY-MM-DD off the front of %cI + elif line and stamp: + dates.setdefault(line, stamp) # log is newest-first, so first wins + return dates def _write_sitemap(pelican_obj): @@ -276,23 +316,26 @@ def _write_sitemap(pelican_obj): # (Same guard, same reason, as _absolutize_site_links.) if not _markdown_mirror_state['siteurl']: return - today = _date.today() + today = _date.today().strftime('%Y-%m-%d') + git_dates = _git_last_modified() articles = sorted(_markdown_mirror_state['articles'], key=lambda a: a.date, reverse=True) lines = ['', ''] for article in articles: - # Modified: is honored where a post sets one, so a back-edit can - # advertise itself; nothing in the corpus does today, in which case - # this is the publication date. min() against today because a - # in the future is out of spec and crawlers discard - # implausible values -- which would cost the freshness signal - # silently rather than loudly. - lastmod = getattr(article, 'modified', None) or article.date - lines.append( - " {}{}".format( - _xml_escape(_canonical_url(article.url)), - min(lastmod.date(), today).strftime('%Y-%m-%d'))) + loc = _xml_escape(_canonical_url(article.url)) + stamp = git_dates.get( + _os.path.relpath(article.source_path, _REPO_ROOT)) + if stamp is None: + # Uncommitted (a new post in a local build) -- omit rather than + # substitute the publication date, which would be a guess. + lines.append(" {}".format(loc)) + continue + # Clamp: a in the future is out of spec and crawlers discard + # implausible values. Only reachable via a skewed committer clock, but + # the failure would be silent, so it's cheap to rule out. + lines.append(" {}{}".format( + loc, min(stamp, today))) lines.append('') with open(_os.path.join(output_path, 'sitemap.xml'), 'w', encoding='utf-8') as f: