> When you're doubting one of your most cherished beliefs, close your eyes, empty your mind, grit your teeth, and **deliberately think about whatever hurts the most**. Don't rehearse standard objections whose standard counters would make you feel better. Ask yourself what _smart_ people who disagree would say to your first reply, and your second reply. Whenever you catch yourself flinching away from an objection you fleetingly thought of, drag it out into the forefront of your mind. **Punch yourself in the solar plexus. Stick a knife in your heart, and wiggle to widen the hole.** In the face of the pain, rehearse only this:
>
-> What is true is already so.
-> Owning up to it doesn't make it worse.
-> Not being open about it doesn't make it go away.
-> And because it's true, it is what is there to be interacted with. Anything untrue isn't there to be lived.
-> People can stand what is true, for they are already enduring it.
+>> What is true is already so.
+>> Owning up to it doesn't make it worse.
+>> Not being open about it doesn't make it go away.
+>> And because it's true, it is what is there to be interacted with. Anything untrue isn't there to be lived.
+>> People can stand what is true, for they are already enduring it.
Meanwhile, the dath ilan mythos depicts rationality itself as hazardous knowledge that people need to be protected from. Dath ilani Civilization is steered by a secretive order of [Keepers of Highly Unpleasant Things it is Sometimes Necessary to Know](https://www.glowfic.com/replies/1612937#reply-1612937). ["The universe is not so dark a place that everyone needs to become a Keeper to ensure the species's survival,"](https://glowfic.com/replies/1861879#reply-1861879) we're told. "Just dark enough that some people ought to." Ordinary dath ilani do receive rationality training, but it's implied to be deliberately crippled, featuring ["signposts around the first steps [towards becoming a Keeper], placed to warn dath ilani off starting down that path unless they mean it."](https://www.glowfic.com/replies/1799590#reply-1799590) The maxim that "That which can be destroyed by the truth should be" is described as being ["remembered as much for how it's false, as for how it's true, because among the things that truths can destroy is people."](https://www.glowfic.com/replies/1687922#reply-1687922)
Yudkowsky's 2009 story ["The Sword of Good"](https://www.yudkowsky.net/other/fiction/the-sword-of-good) is an incisive commentary on how unwary readers' moral compasses can be hijacked by authorial editorializing. If the story depicts our heroes wantonly slaughtering orcs, readers tend not to worry about the ethics of warfare: if the [designated hero](https://tvtropes.org/pmwiki/pmwiki.php/Main/DesignatedHero) is doing it, it presumably isn't a problem. But living creatures that are depicted as speaking language, having organized armies with complex tools, _&c._ are presumably sapient for the same reasons humans are. It's fair game for "The Sword of Good" to point that out—at least, short of the narrator _explicitly_ declaring, "Despite appearances, the orcs are unconscious [philosophical zombies](https://www.lesswrong.com/posts/fdEWWr8St59bXLbQr/zombies-zombies); killing them has no moral significance." (But a story that did declare that would be highly unusual, and basically conceding the critic's point about stories that didn't!)
-All I'm doing in this essay is holding the dath ilan mythos to the same standard that "The Sword of Good" holds classic fantasy tropes to. Maybe it's good to run a global conspiracy to keep people ignorant in order to protect their emotional well-being, and maybe those orcs deserved to die. But given a text that _does_ literally describe massive coverups or killing of human-like creatures, it's fundamentally fair game for literary critics to point that out, and prompt readers to think carefully about whether they should accept "it's good because the good guys are doing it" or "it's good because it's dath ilan, which is not Earth" as an implicit excuse.
+All I'm doing in this essay is holding the dath ilan mythos to the same standard that "The Sword of Good" holds classic fantasy tropes to. Maybe it's good to run a global conspiracy to keep people ignorant in order to protect their emotional well-being, and maybe those orcs deserved to die. But given a text that _does_ literally describe massive coverups or killing of human-like creatures, it's fundamentally fair game for literary critics to point that out, and prompt readers to think carefully about whether they should accept "it's fine because the good guys are doing it" or "it's fine because it's dath ilan, which is not Earth" as an implicit excuse.
### The History Screen
The problem is that [facts are causally entangled with each other](https://www.readthesequences.com/Entangled-Truths-Contagious-Lies). Forgetting one fact entails not just having to deal with not knowing that particular thing, but also not knowing everything it implies about other things in the world—which might be of little consequence for any one trivium, but is surely enormous for _all of human history_.
-In at least one case, the text depicts dath ilan as suffering practical consequences from its ignorance: a discussion of the eliezera's low happiness levels is cut short with ["How'd it happen? Nobody knows, at this point, they screened their history."](https://glowfic.com/replies/1812613#reply-1812613) (Generally, knowing how something happened is useful for figuring out how to remediate it.) Realistically, we can only surmises that there are many other cases where knowing history would be useful that the narrator hasn't gotten around to telling us.
+In at least one case, the text depicts dath ilan as suffering practical consequences from its ignorance: a discussion of the eliezera's low happiness levels is cut short with ["How'd it happen? Nobody knows, at this point, they screened their history."](https://glowfic.com/replies/1812613#reply-1812613) (Generally, knowing how something happened is useful for figuring out how to remediate it.) Realistically, we can only surmise that there are many other cases where knowing history would be useful that the narrator hasn't gotten around to telling us.
Given the costs, what could possibly justify the history screen, in an advanced Society that otherwise seems to value knowledge? A fantasy author could easily an invent an answer: maybe a psychic plague that spread through memories such that the infected must be not just physically quarantined, but forgotten. Medianworld authors have a tougher time insofar as we construe their genre as hard science fiction. Psychic plagues aren't real. What could _realistically_ justify the history screen?
> And so long as they were doing all that anyways, they might as well also carry out the less important but still useful operation of putting all of Civilization's past behind the most complete possible causal screen. That part wasn't as important, but still legitimately helpful; and doing it would help to overshadow the other changes, and lead to less attention going to the more dangerous places.
-"So long as they were doing all that anyways, they might as well"! That's it? The only concrete benefit of enforcing ignorance (of history) we're told about is ... more ignorance (of the existence of the AGI project). Keeping _the project's_ existence secret likely makes sense, if the surveillance measures are deemed insufficient to prevent rogue AGI projects. But classifying _all of human history_ along with it should require some enormously compelling reason, and one just isn't suppplied.
+"So long as they were doing all that anyways, they might as well"! That's it? The only concrete benefit of enforcing ignorance (of history) we're told about is ... more ignorance (of the existence of the AGI project).
-Is it just that the Singularity is so important, the stakes so massive, that [no cost is too small to pay](https://tvtropes.org/pmwiki/pmwiki.php/Main/UtopiaJustifiesTheMeans) to make sure it goes well? That would be understandable. (As the saying goes, ["Shut up and multiply."](https://www.lesswrong.com/posts/r5MSQ83gtbjWRBDWJ/the-intuitions-behind-utilitarianism))
+Keeping _the project's_ existence secret likely makes sense, if the surveillance measures are deemed insufficient to prevent rogue AGI projects. But classifying _all of human history_ along with it should require some enormously compelling reason, and one just isn't suppplied.
+
+Is it that the Singularity is so important, the stakes so massive, that [no cost is too small to pay](https://tvtropes.org/pmwiki/pmwiki.php/Main/UtopiaJustifiesTheMeans) to make sure it goes well? That would be understandable. (As the saying goes, ["Shut up and multiply."](https://www.lesswrong.com/posts/r5MSQ83gtbjWRBDWJ/the-intuitions-behind-utilitarianism))
But the word choices make it seem like that's not what's going on. "So long as they were doing all that anyways, they might as well" is not the language of a tough-minded utilitarian who understands the gravity of the crimes they're about to commit, but is determined to proceed [for the good of the many](https://tvtropes.org/pmwiki/pmwiki.php/Main/TheNeedsOfTheMany). "So long as they were doing all that anyways, they might as well" is the language of someone who's never considered that what they're about to do might be a crime, that [people knowing things as benefits as well as costs](http://benjaminrosshoffman.com/humility-argument-honesty/).
projects / Ultimately_Untrue_Thought.git / commitdiff