We might say that the algorithm that designed dath ilan's Civilization can be seen as systematically preferring deception. When I speak of an algorithm preferring deception, [what I mean is](https://www.lesswrong.com/posts/fmA2GJwZzYtkrAKYJ/algorithms-of-deception) that given a social problem, candidate solutions that involve deceiving the populace seem to be higher in dath ilani Civilization's implicit search ordering than solutions that involve informing the populace. Solutions that work by means of telling the truth will be implemented only when solutions that work by means of deception are seen to fail.
-Crucially, these are [functionalist criteria](https://www.lesswrong.com/posts/sXHQ9R5tahiaXEZhR/algorithmic-intent-a-hansonian-generalized-anti-zombie) of "preference" and "deception". It's about how Civilization is structured in a way that systematically encourages divergences between popular belief and reality. I'm _not_ positing that Civilization's Keepers and Legislators and Chief Executive are laughing maniacally and consciously telling each other, "I personally love it when non-Keepers have false beliefs; we need to do as much of that as possible—as a terminal value!"
+Crucially, these are [functionalist](https://www.lesswrong.com/posts/sXHQ9R5tahiaXEZhR/algorithmic-intent-a-hansonian-generalized-anti-zombie) [criteria](http://benjaminrosshoffman.com/bad-faith-behavior-not-feeling/) of "preference" and "deception". It's about how Civilization is structured in a way that systematically encourages divergences between popular belief and reality. I'm _not_ positing that Civilization's Keepers and Legislators and Chief Executive are laughing maniacally and telling each other out loud, "I personally love it when non-Keepers have false beliefs; we need to do as much of that as possible—as a [terminal value](https://www.readthesequences.com/Terminal-Values-And-Instrumental-Values)!"
Rather, I'm positing that they don't care about non-Keepers having false beliefs. (They might care about [not technically lying](https://www.lesswrong.com/posts/PrXR66hQcaJXsgWsa/not-technically-lying), but that [turns out to be a weak constraint](https://www.lesswrong.com/posts/MN4NRkMw7ggt9587K/firming-up-not-lying-around-its-edge-cases-is-less-broadly).)
If you're in the business of coming up with clever plans to solve problems, and you don't care about people having false beliefs, you mostly end up with clever plans that work by means of giving people false beliefs that trick them into doing what you want them to do (perhaps without technically lying).
-Why wouldn't you? There are more false maps than true maps. If you don't specifically care about affirmatively telling the truth, you mostly end up supplying false maps in order to control people's behavior by means of controling their information, because if you told them the truth, they wouldn't behave the way you want them to. Instrumental convergence is a harsh mistress.
+Why wouldn't you? There are more false maps than true maps. If you don't specifically care about [affirmatively telling the truth](http://benjaminrosshoffman.com/honesty-and-perjury/#Intent_to_inform), you mostly end up supplying false maps in order to control people's behavior by means of controling their information, because if you told them the truth, they wouldn't behave the way you want them to. Instrumental convergence is a harsh mistress.
### Interlude: "I Can't Argue With Authorial Fiat"
At this point, some readers might object that this kind of "dark" interpretation of a fictional universe oversteps the authority of the literary critic. One imagines that Yudkowsky doesn't particularly think of dath ilan as a world governed by deception. What grounds could I possibly have to argue that it is, given that he's the author and I'm not? Isn't that just making up my own fictional world and substituting it for the "real" dath ilan defined by Yudkowsky's authorial intent?
+But the craft of literature isn't a matter of merely conveying a fictional reality that existed fully formed in the author's imagination in advance of writing it down. The craft is about producing text that readers can use to build up their own model of the fictional world. The exacting labor of [converting vague ideas into definite text](http://www.paulgraham.com/words.html) is the difference between writing and daydreaming. We can accept [Word of God](https://tvtropes.org/pmwiki/pmwiki.php/Main/WordOfGod) as supplementary material where the text of a story is ambiguous or silent on a point of interest, but some kind of [Death of the Author](https://tvtropes.org/pmwiki/pmwiki.php/Main/DeathOfTheAuthor) stance is ultimately necessary for making sense of literature in a world in which telepathy doesn't exist and authors do occasionally die. The text is not a mere pointer to the "real" work inside the author's head; the text _is_ the work. That's the only way the technology of writing can function.
+Moreover, a Death of the Author stance seems particularly important for evaluating medianworlds. What makes the exercise of constructing a medianworld interesting is the challenge of envisioning the details of a _realistic_ Society that would result given a population with an alternative [distribution of cognitive repertoires](/2020/Apr/book-review-human-diversity/), but where the same generalizations of biology, sociology, and economics that govern our own world are presumed to hold. If the world portrayed by the text [doesn't seem to hold together](/2022/Jun/comment-on-a-scene-from-planecrash-crisis-of-faith/) or has [unfortunate implications](https://tvtropes.org/pmwiki/pmwiki.php/Main/UnfortunateImplications) that the author doesn't acknowledge, it's the solemn duty of literary critics to point that out to less discerning readers.
-[ TODO—
+In analogy, mathematicians, like authors, are also in the business of creating imaginary worlds, but mathematical objects, once defined, can be examined on their own terms. A mathematician encountering [a deeply unsatisfying theorem about their new definition](https://blog.plover.com/math/major-screwups-4.html) understands that there can be no recourse in protesting, "But that's not how _I_ imagined it working." The response to such an absurd excuse writes itself: if you wanted different behavior, maybe you should have written a better definition!
-https://tvtropes.org/pmwiki/pmwiki.php/Main/DeathOfTheAuthor
+Worldbuilding critics do well to be similarly merciless with authors. An author who says, "In _my_ medianworld, fully automated luxury gay space communism with central planning just works, because the populace is so smart and nice, unlike _Earth people_, who are so mean and dumb that they have to use _markets_ to allocate scarce resources" is failing to engage with the real complexities of the medianworld exercise, and readers have a right to be skeptical. Maybe the author _is_ 1 standard deviation smarter and nicer than the general population here in the real world. Does that buy you reprieve from the cold equations of the dismal science, really?
- * A worldbuilding critic who takes a negative view of some world might be told that they're not allowed to contradict authorial intent. If the narrator says the people of dath ilan are doing something because they're good and smart and cooperative, the critic has to accept that.
- * But what makes the medianworld exercise interesting is that it's about trying to portray a realistic world, given a shifted distribution of psychological traits. We take the text of the story as a given, but we don't have to take dath ilan's self-image literally, if we think a different world could "project" into the same text and explain it better.
- * Mathematicians are also in the business of creating imaginary worlds.
- * An ethnographer might note that Americans believe themselves to be "the land of the brave and the home of the free", without being obliged for their ethnography to agree with this description. I'm taking the same stance towards dath ilan: as a literary critic, I don't have to share its Society's beliefs about itself.
+Authors, of course, have much more wiggle room than mathematicians to try to salvage their cherished ideas. Rather than being forced back to the drawing board by an unwanted implication, a fiction writer finds it all too easy to simply add another sentence denying it.
+
+[TODO: but the rich freedom to specify whatever you want in text is counterbalanced by the rich detail of a real world that wasn't specified in your text; the text is given, but readers trying to imagine a self-consistent world that "projects into" your text are going to have to assume things about the world that you didn't write, in order to make their model add up]
+Yudkowsky's 2009 story ["The Sword of Good"](https://www.yudkowsky.net/other/fiction/the-sword-of-good) is an incisive commentary on how readers' moral compasses can be hijacked by author editorializing. If the story depicts our heroes slaughtering orcs,
+
+[ TODO—
+ * We take the text of the story as a given, but we don't have to take dath ilan's self-image literally, if we think a different world could "project" into the same text and explain it better.
+ * An ethnographer might note that Americans believe themselves to be "the land of the brave and the home of the free", without being obliged for their ethnography to agree with this description. I'm taking the same stance towards dath ilan: as a literary critic, I don't have to share its Society's beliefs about itself.
]
### History Screening
projects / Ultimately_Untrue_Thought.git / commitdiff