Clearly, this is not a culture that cares about ordinary people being well-informed. Apparently, they believe that owning up to it _does_ make it worse, that the untrue _is_ there to be lived (for non-Keepers).
-We might say that the algorithm that designed dath ilan's Civilization can be seen as systematically preferring deception. When I speak of an algorithm preferring deception, [what I mean is](https://www.lesswrong.com/posts/fmA2GJwZzYtkrAKYJ/algorithms-of-deception) that given a social problem, candidate solutions that involve deceiving the populace seem to be higher in dath ilani Civilization's implicit search ordering than solutions that involve informing the populace. Solutions that work by means of telling the truth will be implemented only when solutions that work by means of deception are seen to fail.
+We might say that the algorithm that designed dath ilan's Civilization can be seen as systematically preferring deception. When I speak of an algorithm preferring deception, [what I mean is](https://www.lesswrong.com/posts/fmA2GJwZzYtkrAKYJ/algorithms-of-deception) that given a social problem, candidate solutions that involve deceiving the populace seem to be higher in dath ilani Civilization's implicit search ordering than solutions that involve informing the populace. If a solution adequately achieves its purpose by means of deception or secrecy, Civilization isn't going to reject it on that basis and keep searching for a solution that involves telling the truth.
Crucially, these are [functionalist](https://www.lesswrong.com/posts/sXHQ9R5tahiaXEZhR/algorithmic-intent-a-hansonian-generalized-anti-zombie) [criteria](http://benjaminrosshoffman.com/bad-faith-behavior-not-feeling/) of "preference" and "deception". It's about how Civilization is structured in a way that systematically encourages divergences between popular belief and reality. I'm _not_ positing that Civilization's Keepers and Legislators and Chief Executive are laughing maniacally and telling each other out loud, "I personally love it when non-Keepers have false beliefs; we need to do as much of that as possible—as a [terminal value](https://www.readthesequences.com/Terminal-Values-And-Instrumental-Values)!"
But the craft of literature isn't a matter of merely conveying a fictional reality that existed fully formed in the author's imagination in advance of writing it down. The craft is about producing text that readers can use to build up their own model of the fictional world. The exacting labor of [converting vague ideas into definite text](http://www.paulgraham.com/words.html) is the difference between writing and daydreaming. We can accept [Word of God](https://tvtropes.org/pmwiki/pmwiki.php/Main/WordOfGod) as supplementary material where the text of a story is ambiguous or silent on a point of interest, but some kind of [Death of the Author](https://tvtropes.org/pmwiki/pmwiki.php/Main/DeathOfTheAuthor) stance is ultimately necessary for making sense of literature in a world in which telepathy doesn't exist and authors do occasionally die. The text is not a mere pointer to the "real" work inside the author's head; the text _is_ the work. That's the only way the technology of writing can function.
-Moreover, a Death of the Author stance seems particularly important for evaluating medianworlds. What makes the exercise of constructing a medianworld interesting is the challenge of envisioning the details of a _realistic_ Society that would result given a population with an alternative [distribution of cognitive repertoires](/2020/Apr/book-review-human-diversity/), but where the same generalizations of biology, sociology, and economics that govern our own world are presumed to hold. If the world portrayed by the text [doesn't seem to hold together](/2022/Jun/comment-on-a-scene-from-planecrash-crisis-of-faith/) or has [unfortunate implications](https://tvtropes.org/pmwiki/pmwiki.php/Main/UnfortunateImplications) that the author doesn't acknowledge, it's the solemn duty of literary critics to point that out to less discerning readers.
+Moreover, a Death of the Author stance seems particularly important for evaluating medianworlds. What makes the exercise of constructing a medianworld interesting is the challenge of envisioning the details of a realistic Society that would result given a population with an alternative [distribution of cognitive repertoires](/2020/Apr/book-review-human-diversity/), but where the same generalizations of biology, sociology, and economics that govern our own world are presumed to hold. If the world portrayed by the text [doesn't seem to hold together](/2022/Jun/comment-on-a-scene-from-planecrash-crisis-of-faith/) or has [unfortunate implications](https://tvtropes.org/pmwiki/pmwiki.php/Main/UnfortunateImplications) that the author doesn't acknowledge, it's the solemn duty of literary critics to point that out to less discerning readers.
In analogy, mathematicians, like authors, are also in the business of creating imaginary worlds, but mathematical objects, once defined, can be examined on their own terms. A mathematician encountering [a deeply unsatisfying theorem about their new definition](https://blog.plover.com/math/major-screwups-4.html) understands that there can be no recourse in protesting, "But that's not how _I_ imagined it working." That would be failing to engage with the real difficulties of mathematical research. If you wanted different behavior, you should have written a better definition!
Similarly, an author who says, "In _my_ medianworld, fully automated luxury gay space communism with central planning just works, because the populace is so smart and nice, unlike _Earth people_, who are so mean and dumb that they have to use _markets_ to allocate scarce resources" is failing to engage with the real difficulties of the medianworld exercise. Readers would have a right to be skeptical.
-Authors, of course, have much more wiggle room than mathematicians to try to salvage their cherished ideas. Rather than being forced back to the drawing board by an unwanted implication, a fiction writer finds it all too easy to simply add another sentence denying it. But the author's total freedom to specify the text necessarily interacts with readers' attempts to imagine a self-consistent universe that "projects into" that text. Short of an explicitly omniscient narrator declaring "And then a miracle occured", discerning readers will tend to reach for interprerations of the text that make sense—even if making sense entails casting doubt on the narrator's editorial spin on the described events.
+Authors, of course, have much more wiggle room than mathematicians to try to salvage their cherished ideas. Rather than being forced back to the drawing board by an unwanted implication, a fiction writer finds it all too easy to simply add another sentence denying it. But the author's total freedom to specify the text necessarily interacts with readers' attempts to imagine a self-consistent universe that "projects into" that text. Short of an explicitly omniscient narrator declaring "And then a miracle occured", discerning readers will tend to reach for interprerations of the text that make sense—even if making sense entails casting doubt on the narrator's spin on the described events.
-Yudkowsky's 2009 story ["The Sword of Good"](https://www.yudkowsky.net/other/fiction/the-sword-of-good) is an incisive commentary on how unwary readers' moral compasses can be hijacked by author editorializing. If the story depicts our heroes wantonly slaughtering orcs, readers tend not to worry about the ethics of warfare: if the [designated hero](https://tvtropes.org/pmwiki/pmwiki.php/Main/DesignatedHero) is doing it, it presumably isn't a problem. But living creatures that are depicted as speaking language, having organized armies with complex tools, _&c._ are presumably sapient for the same reasons humans are. It's fair game for "The Sword of Good" to point that out—at least, short of the narrator _explicitly_ declaring, "Despite appearances, the orcs are unconscious [philosophical zombies](https://www.lesswrong.com/posts/fdEWWr8St59bXLbQr/zombies-zombies); killing them has no moral significance." (But a story that did declare that would be highly unusual, and basically conceding the critic's point about stories that didn't!)
+Yudkowsky's 2009 story ["The Sword of Good"](https://www.yudkowsky.net/other/fiction/the-sword-of-good) is an incisive commentary on how unwary readers' moral compasses can be hijacked by authorial editorializing. If the story depicts our heroes wantonly slaughtering orcs, readers tend not to worry about the ethics of warfare: if the [designated hero](https://tvtropes.org/pmwiki/pmwiki.php/Main/DesignatedHero) is doing it, it presumably isn't a problem. But living creatures that are depicted as speaking language, having organized armies with complex tools, _&c._ are presumably sapient for the same reasons humans are. It's fair game for "The Sword of Good" to point that out—at least, short of the narrator _explicitly_ declaring, "Despite appearances, the orcs are unconscious [philosophical zombies](https://www.lesswrong.com/posts/fdEWWr8St59bXLbQr/zombies-zombies); killing them has no moral significance." (But a story that did declare that would be highly unusual, and basically conceding the critic's point about stories that didn't!)
-All I'm doing in this essay is holding the dath ilan mythos to the same standard that "The Sword of Good" holds classic fantasy tropes to. Maybe it's good to run a global conspiracy to keep people ignorant in order to protect their emotional well-being, and maybe those orcs deserved to die. But given a text that _does_ literally describe massive coverups or killing of human-like creatures, it's fundamentally fair game for literary critics to point that out, and prompt readers to rethink whether they should accept "it's good because the good guys are doing it" or "it's good because it's dath ilan, which is not Earth" as an implicit excuse.
+All I'm doing in this essay is holding the dath ilan mythos to the same standard that "The Sword of Good" holds classic fantasy tropes to. Maybe it's good to run a global conspiracy to keep people ignorant in order to protect their emotional well-being, and maybe those orcs deserved to die. But given a text that _does_ literally describe massive coverups or killing of human-like creatures, it's fundamentally fair game for literary critics to point that out, and prompt readers to think carefully about whether they should accept "it's good because the good guys are doing it" or "it's good because it's dath ilan, which is not Earth" as an implicit excuse.
-### The History Screen: Who Controls the Past Controls the Future
+### The History Screen
One of the distinctive features of dath ilani Civilization is that they've deleted their history. No one except a few specially cleared specialists ["in their own causally isolated bunker"](https://glowfic.com/replies/1688794#reply-1688794) is allowed to learn history before a point in a relatively recent past. (["[D]ecades ago, not millennia ago"](https://glowfic.com/replies/1789110#reply-1789110) as of Keltham's time.)
-This is a drastic measure, a civilizational lobotomy. It seems significant that the most famous literary depiction of a Society that deleted its history is George Orwell's _Nineteen-Eighty-Four_, in which a totalitarian state maintains a Society in which "[n]othing exists except an endless present in which the Party is always right."
+This is an incredibly drastic measure, a civilizational lobotomy. (It perhaps bears worth mentioning that the most famous literary depiction of a Society that deleted its history is George Orwell's _Nineteen-Eighty-Four_, in which a totalitarian state maintains a Society in which "[n]othing exists except an endless present in which the Party is always right.") The full costs may not be obvious to many readers: if the immediate needs of living in the present don't directly require thinking about the past, what's the harm of not knowing?
-Dath ilan is depicted as suffering practical consequences from its ignorance: a discussion of the evolution of eliezera's low happiness levels is cut short with ["How'd it happen? Nobody knows, at this point, they screened their history."](https://glowfic.com/replies/1812613#reply-1812613)
+The problem is that [facts are causally entangled with each other](https://www.readthesequences.com/Entangled-Truths-Contagious-Lies). Forgetting one fact entails not just having to deal with not knowing that particular thing, but also not knowing everything it implies about other things in the world—which might be of little consequence for any one trivium, but is surely enormous for _all of human history_.
-[TODO—
- * need to figure out how to be more concrete about harms ... https://www.readthesequences.com/Entangled-Truths-Contagious-Lies
- * This is a drastic measure: consider all the costs of deleting history
- * The actual explanation turns out to be incredibly casual—"As long as they were doing that anyways": the assumption that the hoi polloi are better off not knowing where they came from isn't argued for at all
- * Doylist interpretation makes sense (Keltham encountering a mideval monarchy is more interesting if he doesn't know what a Queen is), but if it's that kind of universe, what's with the comments about dath ilan being superior to Earth? Topia-for-storytelling shouldn't be hailed as a eutopia
- * 1984 https://www.abhafoundation.org/assets/books/html/1984/162.html https://www.abhafoundation.org/assets/books/html/1984/103.html
+In at least one case, the text depicts dath ilan as suffering practical consequences from its ignorance: a discussion of the eliezera's low happiness levels is cut short with ["How'd it happen? Nobody knows, at this point, they screened their history."](https://glowfic.com/replies/1812613#reply-1812613) (Generally, knowing how something happened is useful for figuring out how to remediate it.) Realistically, we can only surmises that there are many other cases where knowing history would be useful that the narrator hasn't gotten around to telling us.
+
+Given the costs, what could possibly justify the history screen, in an advanced Society that otherwise seems to value knowledge? A fantasy author could easily an invent an answer: maybe a psychic plague that spread through memories such that the infected must be not just physically quarantined, but forgotten. Medianworld authors have a tougher time insofar as we construe their genre as hard science fiction. Psychic plagues aren't real. What could _realistically_ justify the history screen?
+
+The answer the text gives leaves something to be desired. In "But Hurting People Is Wrong", we learn of the birth of modern dath ilan, when the Keepers noticed the AGI alignment problem and propose strict policy measures aimed at surviving the challenge.
+
+For one, dath ilan's eugenics program is to optimize for intelligence even more than it already does. That makes sense: smarter alignment researchers have a better shot at solving the problem.
+
+For another, restrictions on the manufacture of computers are imposed. That makes sense: if computer hardware were too capable and easily available before alignment researchers had the [serial time](https://www.lesswrong.com/posts/FS6NCWzzP8DHp4aD4/do-earths-with-slower-economic-growth-have-a-better-chance) to think the problem through, that would increase the risk of accidentally unleashing an unaligned intelligence explosion.
+
+For another, digital surveilance is imposed. That makes sense: Governance would want to know about any rogue AGI projects, so they can be shut down by force.
-https://glowfic.com/replies/1688794#reply-1688794
-> "I'm not sure there's literally anybody on my planet who's 'not allowed to do other stuff'. Maybe some Keepers, if they're holding infohazards so bad that they all have to stay in the same isolated village somewhere?"
-> (And the people who know the true history, in their own causally isolated bunker.
+[And then we get this](https://www.glowfic.com/replies/1612939#reply-1612939):
-https://www.glowfic.com/replies/1612939#reply-1612939
> And so long as they were doing all that anyways, they might as well also carry out the less important but still useful operation of putting all of Civilization's past behind the most complete possible causal screen. That part wasn't as important, but still legitimately helpful; and doing it would help to overshadow the other changes, and lead to less attention going to the more dangerous places.
-]
+"So long as they were doing all that anyways, they might as well"! That's it? The only concrete benefit of enforcing ignorance (of history) we're told about is ... more ignorance (of the existence of the AGI project). Keeping _the project's_ existence secret likely makes sense, if the surveillance measures are deemed insufficient to prevent rogue AGI projects. But classifying _all of human history_ along with it should require some enormously compelling reason, and one just isn't suppplied.
+
+Is it just that the Singularity is so important, the stakes so massive, that [no cost is too small to pay](https://tvtropes.org/pmwiki/pmwiki.php/Main/UtopiaJustifiesTheMeans) to make sure it goes well? That would be understandable. (As the saying goes, ["Shut up and multiply."](https://www.lesswrong.com/posts/r5MSQ83gtbjWRBDWJ/the-intuitions-behind-utilitarianism))
+
+But the word choices make it seem like that's not what's going on. "So long as they were doing all that anyways, they might as well" is not the language of a tough-minded utilitarian who understands the gravity of the crimes they're about to commit, but is determined to proceed [for the good of the many](https://tvtropes.org/pmwiki/pmwiki.php/Main/TheNeedsOfTheMany). "So long as they were doing all that anyways, they might as well" is the language of someone who's never considered that what they're about to do might be a crime, that [people knowing things as benefits as well as costs](http://benjaminrosshoffman.com/humility-argument-honesty/).
+
+It would be remiss to conclude the section without considering a [Doylist rather than Watsonian](https://tvtropes.org/pmwiki/pmwiki.php/Main/WatsonianVersusDoylist) interpretation of the history screen. Rumored Word of God[^wog-rumor] states that one reason for the history screen is that it makes a better story for dath ilani isekai protagonists to be more ignorant of other ways of life than the power of their civilization would otherwise imply; it's more interesting to watch Keltham encounter a medieval monarchy if he doesn't know what a queen is.
+
+[^wog-rumor]: I don't have a link for this; I think I heard Yudkowsky say it at a party.
+
+The Doylist storytelling case for the history screen is compelling insofar as the function of the dath ilan mythos is strictly to tell entertaining stories. However, such a prominent feature of the Society existing for storytelling reasons undermines the function of dath ilan as a utopian contrast to the real world, as empitomized by the incessant in- and out-of-story sneers at "Earth" (_i.e._, real) people.
### The Merrin Show
projects / Ultimately_Untrue_Thought.git / commitdiff