The problem with trying to "exhibit generally rationalist principles" in an line of argument that you're constructing in order to be prudent and not community-harmful, is that you're thereby necessarily _not_ exhibiting the central rationalist principle that what matters is the process that _determines_ your conclusion, not the reasoning you present to _reach_ your conclusion, after the fact.
-The best explanation of this I know was authored by Yudkowsky himself in 2007, in a post titled ["A Rational Argument"](https://www.lesswrong.com/posts/9f5EXt8KNNxTAihtZ/a-rational-argument). It's worth quoting at length. The Yudkowsky of 2007 invites us to consider the plight of a political campaign manager:
+The best explanation of this I know of was authored by Yudkowsky himself in 2007, in a post titled ["A Rational Argument"](https://www.lesswrong.com/posts/9f5EXt8KNNxTAihtZ/a-rational-argument). It's worth quoting at length. The Yudkowsky of 2007 invites us to consider the plight of a political campaign manager:
> As a campaign manager reading a book on rationality, one question lies foremost on your mind: "How can I construct an impeccable rational argument that Mortimer Q. Snodgrass is the best candidate for Mayor of Hadleyburg?"
>
>
> This is the sense in which I suspect you are coming across as failing to properly Other-model.
-After everything I've been through, I'm inclined to think it's not a "disagreement" at all.
+After everything I've been through over the past six years, I'm inclined to think it's not a "disagreement" at all.
It's a _conflict_. I think what's actually at issue is that, at least in this domain, I want people to tell the truth, and the Caliphate wants people to not tell the truth. This isn't a disagreement about rationality, because telling the truth _isn't_ rational _if you don't want people to know things_.
-At this point, I imagine defenders of the Caliphate are shaking their heads in disappointment at how I'm doubling down on refusing to Other-model. But—_am_ I? Isn't this just a re-statement of Xu's first proposed crux, except reframed as a "values difference" rather than a "disagreement"? Is the problem that my use of the phrase "tell the truth" (which has positive valence in our culture) functions to sneak in normative connotations favoring "my side"?
+At this point, I imagine defenders of the Caliphate are shaking their heads in disappointment at how I'm doubling down on refusing to Other-model. But—_am_ I? Isn't this just a re-statement of Xu's first proposed crux, except reframed as a "values difference" rather than a "disagreement"?
+
+Is the problem that my use of the phrase "tell the truth" (which has positive valence in our culture) functions to sneak in normative connotations favoring "my side"?
Fine. Objection sustained. I'm happy to use to Xu's language: I think what's actually at issue is that, at least in this domain, I want to facilitate people making inferences (full stop), and the Caliphate wants to _not_ facilitate people making inferences that, on the whole, cause more harm than benefit. This isn't a disagreement about rationality, because facilitating inferences _isn't_ rational _if you don't want people to make inferences_ (for example, because they cause more harm than benefit).
-Better?
+Better? Perhaps, to some 2022-era rats and EAs, this formulation makes my position look obviously in the wrong: I'm saying that I'm fine with my inferences _causing more harm than benefit_ (!). Isn't that monstrous of me? Why would someone do that?
-[TODO: quote "Doublethink (Choosing to Be Biased)", note that despite Yudkowsky's doubt, the situation is actually worse that Orwell depicted—you don't even have to burn the offending material, if you can just get people to ignore it]
+One of the better explanations of this that I know of was (again, as usual) authored by Yudkowsky in 2007, in a post titled ["Doublethink (Choosing to be Biased)"](https://www.lesswrong.com/posts/Hs3ymqypvhgFMkgLb/doublethink-choosing-to-be-biased).
+The Yudkowsky of 2007 starts by quoting a passage from George Orwell's _1984_, in which O'Brien (a loyal member of the ruling Party in the totalitarian state depicted in the novel) burns a photograph of Jones, Aaronson, and Rutherford (former Party leaders whose existence has been censored from the historical record). Immediately after burning the photograph, O'Brien denies that it ever existed.
-[TODO:
-"massive psychological damage to some subset of people",
-that's _not my problem_. I _don't give a shit_.]
+The Yudkowsky of 2007 continues—it's again worth quoting at length—
+
+> What if self-deception helps us be happy? What if just running out and overcoming bias will make us—gasp!—_unhappy?_ Surely, _true_ wisdom would be _second-order_ rationality, choosing when to be rational. That way you can decide which cognitive biases should govern you, to maximize your happiness.
+>
+> Leaving the morality aside, I doubt such a lunatic dislocation in the mind could really happen.
+>
+> [...]
+>
+> For second-order rationality to be genuinely _rational_, you would first need a good model of reality, to extrapolate the consequences of rationality and irrationality. If you then chose to be first-order irrational, you would need to forget this accurate view. And then forget the act of forgetting. I don't mean to commit the logical fallacy of generalizing from fictional evidence, but I think Orwell did a good job of extrapolating where this path leads.
+>
+> You can't know the consequences of being biased, until you have already debiased yourself. And then it is too late for self-deception.
+>
+> The other alternative is to choose blindly to remain biased, without any clear idea of the consequences. This is not second-order rationality. It is willful stupidity.
+>
+> [...]
+>
+> One of chief pieces of advice I give to aspiring rationalists is "Don't try to be clever." And, "Listen to those quiet, nagging doubts." If you don't know, you don't know _what_ you don't know, you don't know how _much_ you don't know, and you don't know how much you _needed_ to know.
+>
+> There is no second-order rationality. There is only a blind leap into what may or may not be a flaming lava pit. Once you _know_, it will be too late for blindness.
+
+Looking back on this from 2022, the only criticism I have is that Yudkowsky was too optimistic to "doubt such a lunatic dislocation in the mind could really happen." In some ways, people's actual behavior is _worse_ than what Orwell depicted. The Party of Orwell's _1984_ covers its tracks: O'Brien takes care to burn the photograph _before_ denying memory of it, because it would be _too_ absurd for him to act like the photo had never existed while it was still right there in front of him.
+
+In contrast, Yudkowsky's Caliphate of the current year _doesn't even bother covering its tracks_. Turns out, it doesn't need to! People mostly just don't remember things!
+
+The [flexibility of natural language is a _huge_ help here](https://www.lesswrong.com/posts/MN4NRkMw7ggt9587K/firming-up-not-lying-around-its-edge-cases-is-less-broadly). If the caliph were to _directly_ contradict himself in simple, unambiguous language—to go from "Oceania is not at war with Eastasia" to "Oceania is at war with Eastasia" without any acknowledgement that anything had changed—_then_ too many people might notice that those two sentences are the same except that one has the word _not_ in it. What's a caliph to do, if he wants to declare war on Eastasia without acknowledging or taking responsibility for the decision to do so?
+
+The solution is simple: just—use more words! Then if someone tries to argue that you've _effectively_ contradicted yourself, accuse them of being uncharitable and failing to model the Other. You can't lose! Anything can be consistent with anything if you apply a sufficiently charitable reading; whether Oceania is at war with Eastasia depends on how you choose to draw the category boundaries of "at war."
+
+Thus, O'Brien should envy Yudkowsky: burning the photograph turns out to be unnecessary!
+
+["Changing Emotions"](https://www.lesswrong.com/posts/QZs4vkC7cbyjL9XA9/changing-emotions) is _still up_ and not retracted, but that didn't stop the Yudkowsky of 2016 from pivoting to ["at least 20% of the ones with penises are actually women"](https://www.facebook.com/yudkowsky/posts/10154078468809228) when that became a politically favorable thing to say. I claim that these posts _effectively_ contradict each other. The former explains why men who fantasize about being women are _not only_ out of luck given forseeable technology, but _also_ that their desires may not even be coherent (!), whereas the latter claims that men who wish they were women may, in fact, _already_ be women in some unspecified psychological sense.
+
+_Technically_, these don't _strictly_ contradict each other: I can't point to a sentence from each that are the same except one includes the word _not_. (And even if there were such sentences, I wouldn't be able to prove that the other words were being used in the same sense in both sentences.) One _could_ try to argue that "Changing Emotions" is addressing cis men with a weird sex-change fantasy, whereas the "ones with penises are actually women" claim was about trans women, which are a different thing.
+
+_Realistically_ ... no. These two posts _can't_ both be right. In itself, this isn't a problem: people change their minds sometimes, which is great! But when people _actually_ change their minds (as opposed to merely changing what they say in public for political reasons), you expect them to be able to _acknowledge_ the change, and hopefully explain what new evidence or reasoning brought them around. If they can't even _acknowledge the change_, that's pretty Orwellian, like O'Brien trying to claim that the photograph is of different men who just coincidentally happen to look like Jones, Aaronson, and Rutherford.
+
+And if a little bit of Orwellianism on specific, narrow, highly-charged topics might be forgiven—because everyone else in your Society is doing it, and you would be punished for not playing along, an [inadequate equilibrium](https://equilibriabook.com/) that no one actor has the power to defy—might we not expect the father of the "rationalists" to stand his ground on the core theses of his ideology, like whether telling the truth is good?
+
+I guess not! ["Doublethink (Choosing to be Biased)"](https://www.lesswrong.com/posts/Hs3ymqypvhgFMkgLb/doublethink-choosing-to-be-biased) is _still up_ and not retracted, but that didn't stop Yudkowsky from [endorsing everything Xu said](https://twitter.com/ESYudkowsky/status/1436025983522381827) about "whether some categories facilitate inferences that _do_, on the whole, cause more harm than benefit, and if so, whether it is 'rational' to rule that such inferences should be avoided when possible" being different cruxes than "whether 'rational' thinking is 'worth it'".
+
+I don't doubt Yudkowsky could come up with some clever casuistry why, _technically_, the text he wrote in 2007 and the text he endorsed in 2021 don't contradict each other. But _realistically_ ... again, no.
+
+[TODO: elaborate on how 2007!Yudkowsky and 2021!Xu are saying the opposite things if you just take a plain-language reading]
[TODO: if he's reading this, win back respect— reply, motherfucker]
I don't, actually, know how to prevent the world from ending. Probably we were never going to survive. (The cis-human era of Earth-originating intelligent life wasn't going to last forever, and it's hard to exert detailed control over what comes next.) But if we're going to die either way, I think it would be _more dignified_ if Eliezer Yudkowsky were to behave as if he wanted his faithful students to be informed. Since it doesn't look like we're going to get that, I think it's _more dignified_ if his faithful students _know_ that he's not behaving like he wants us to be informed. And so one of my goals in telling you this long story about how I spent (wasted?) the last six years of my life, is to communicate the moral that **I don't trust Eliezer Yudkowsky to tell the truth, and I don't think you should trust him, either**—and that this is a _problem_ for the future of humanity, to the extent that there is a future of humanity.
-Is that a mean thing to say about someone to whom I owe so much? Probably. But if it helps—he didn't create me to not say mean things. As far as _I_ can tell, I'm only doing what he taught me to do in 2007–9: [carve reality at the joints](https://www.lesswrong.com/posts/esRZaPXSHgWzyB2NL/where-to-draw-the-boundaries), [speak the truth even if your voice trembles](https://www.lesswrong.com/posts/pZSpbxPrftSndTdSf/honesty-beyond-internal-truth), and [make an extraordinary effort](https://www.lesswrong.com/posts/GuEsfTpSDSbXFiseH/make-an-extraordinary-effort) when you've got [Something to Protect](https://www.lesswrong.com/posts/SGR4GxFK7KmW7ckCB/something-to-protect).
+Is that a mean thing to say about someone to whom I owe so much? Probably. But he didn't create me to not say mean things. If it helps—as far as _I_ can tell, I'm only doing what he taught me to do in 2007–9: [carve reality at the joints](https://www.lesswrong.com/posts/esRZaPXSHgWzyB2NL/where-to-draw-the-boundaries), [speak the truth even if your voice trembles](https://www.lesswrong.com/posts/pZSpbxPrftSndTdSf/honesty-beyond-internal-truth), and [make an extraordinary effort](https://www.lesswrong.com/posts/GuEsfTpSDSbXFiseH/make-an-extraordinary-effort) when you've got [Something to Protect](https://www.lesswrong.com/posts/SGR4GxFK7KmW7ckCB/something-to-protect).
projects / Ultimately_Untrue_Thought.git / commitdiff