[^not-endorsements]: In general, retweets are not necessarily endorsements—sometimes people just want to draw attention to some content without further comment or implied approval—but I was inclined to read this instance as implying approval, partially because this doesn't seem like the kind of thing someone would retweet for attention-without-approval, and partially because of the working relationship between Soares and Yudkowsky.
-Soares's points seemed cribbed from part I of Scott Alexander's ["... Not Man for the Categories"](https://slatestarcodex.com/2014/11/21/the-categories-were-made-for-man-not-man-for-the-categories/), which post I had just dedicated _more than three years of my life_ to rebutting in [increasing](/2018/Feb/the-categories-were-made-for-man-to-make-predictions/) [technical](https://www.lesswrong.com/posts/esRZaPXSHgWzyB2NL/where-to-draw-the-boundaries) [detail](https://www.lesswrong.com/posts/onwgTH6n8wxRSo2BJ/unnatural-categories-are-optimized-for-deception), _specifically using dolphins as my central example_—which Soares didn't necessarily have any reason to have known about, but Yudkowsky (who retweeted Soares) definitely did. (Soares's [specific reference to the Book of Jonah](https://twitter.com/So8res/status/1401670796997660675) made it seem particularly unlikely that he had invented the argument independently from Alexander.) [One of the replies (which Soares Liked) pointed out the similar _Slate Star Codex_ article](https://twitter.com/max_sixty/status/1401688892940509185), [as did](https://twitter.com/NisanVile/status/1401684128450367489) [a couple of](https://twitter.com/roblogic_/status/1401699930293432321) quote-Tweet discussions.
+Soares's points seemed cribbed from part I of Scott Alexander's ["... Not Man for the Categories"](https://slatestarcodex.com/2014/11/21/the-categories-were-made-for-man-not-man-for-the-categories/), which post I had just dedicated more than three years of my life to rebutting in [increasing](/2018/Feb/the-categories-were-made-for-man-to-make-predictions/) [technical](https://www.lesswrong.com/posts/esRZaPXSHgWzyB2NL/where-to-draw-the-boundaries) [detail](https://www.lesswrong.com/posts/onwgTH6n8wxRSo2BJ/unnatural-categories-are-optimized-for-deception), specifically using dolphins as my central example—which Soares didn't necessarily have any reason to have known about, but Yudkowsky (who retweeted Soares) definitely did. (Soares's [reference to the Book of Jonah](https://twitter.com/So8res/status/1401670796997660675) made it seem particularly unlikely that he had invented the argument independently from Alexander.) [One of the replies (which Soares Liked) pointed out the similar _Slate Star Codex_ article](https://twitter.com/max_sixty/status/1401688892940509185), [as did](https://twitter.com/NisanVile/status/1401684128450367489) [a couple of](https://twitter.com/roblogic_/status/1401699930293432321) quote-Tweet discussions.
-The elephant in my brain took this as another occasion to _flip out_. I didn't _immediately_ see anything for me to overtly object to in the thread itself—[I readily conceded that](https://twitter.com/zackmdavis/status/1402073131276066821) there was nothing necessarily wrong with wanting to use the symbol "fish" to refer to the cluster of similarities induced by convergent evolution to the acquatic habitat rather than the cluster of similarities induced by phylogenetic relatedness—but in the context of our subculture's history, I read this as Soares and Yudkowsky implicitly lending more legitimacy to "... Not Man for the Categories", which was _hostile to my interests_. Was I paranoid to read this as a potential [dogwhistle](https://en.wikipedia.org/wiki/Dog_whistle_(politics))? It just seemed _implausible_ that Soares would be Tweeting that dolphins are fish in the counterfactual in which "... Not Man for the Categories" had never been published.
+The elephant in my brain took this as another occasion to _flip out_. I didn't immediately see anything for me to overtly object to in the thread itself—[I readily conceded that](https://twitter.com/zackmdavis/status/1402073131276066821) there was nothing necessarily wrong with wanting to use the symbol "fish" to refer to the cluster of similarities induced by convergent evolution to the acquatic habitat rather than the cluster of similarities induced by phylogenetic relatedness—but in the context of our subculture's history, I read this as Soares and Yudkowsky implicitly lending more legitimacy to "... Not Man for the Categories", which was hostile to my interests. Was I paranoid to read this as a potential [dogwhistle](https://en.wikipedia.org/wiki/Dog_whistle_(politics))? It just seemed implausible that Soares would be Tweeting that dolphins are fish in the counterfactual in which "... Not Man for the Categories" had never been published.
-After a little more thought, I decided the thread _was_ overtly objectionable, and [quickly wrote up a reply on _Less Wrong_](https://www.lesswrong.com/posts/aJnaMv8pFQAfi9jBm/reply-to-nate-soares-on-dolphins): Soares wasn't merely advocating for a "swimmy animals" sense of the word _fish_ to become more accepted usage, but specifically deriding phylogenetic definitions as unmotivated for everyday use ("definitional gynmastics [_sic_]"!), and _that_ was wrong. It's true that most language users don't directly care about evolutionary relatedness, but [words aren't identical with their definitions](https://www.lesswrong.com/posts/i2dfY65JciebF3CAo/empty-labels). Genetics is at the root of the causal graph underlying all other features of an organism; creatures that are more closely evolutionarily related are more similar _in general_. Classifying things by evolutionary lineage isn't an arbitrary æsthetic whim by people who care about geneology for no reason. We need the natural category of "mammals (including marine mammals)" to make sense of how dolphins are warm-blooded, breathe air, and nurse their live-born young, and the natural category of "finned cold-blooded vertebrate gill-breathing swimmy animals (which excludes marine mammals)" is also something that it's reasonable to have a word for.
+After a little more thought, I decided that the thread _was_ overtly objectionable, and [quickly wrote up a reply on _Less Wrong_](https://www.lesswrong.com/posts/aJnaMv8pFQAfi9jBm/reply-to-nate-soares-on-dolphins): Soares wasn't merely advocating for a "swimmy animals" sense of the word _fish_ to become more accepted usage, but specifically deriding phylogenetic definitions as unmotivated for everyday use ("definitional gynmastics [_sic_]"!), and _that_ was wrong. It's true that most language users don't directly care about evolutionary relatedness, but [words aren't identical with their definitions](https://www.lesswrong.com/posts/i2dfY65JciebF3CAo/empty-labels). Genetics is at the root of the causal graph underlying all other features of an organism; creatures that are more closely evolutionarily related are more similar in general. Classifying things by evolutionary lineage isn't an arbitrary æsthetic whim by people who care about geneology for no reason. We need the natural category of "mammals (including marine mammals)" to make sense of how dolphins are warm-blooded, breathe air, and nurse their live-born young, and the natural category of "finned cold-blooded vertebrate gill-breathing swimmy animals (which excludes marine mammals)" is also something that it's reasonable to have a word for.
(Somehow, it felt appropriate to use a quote from Arthur Jensen's ["How Much Can We Boost IQ and Scholastic Achievement?"](https://en.wikipedia.org/wiki/How_Much_Can_We_Boost_IQ_and_Scholastic_Achievement%3F) as an epigraph.)
-[TODO: dolphin war con'td
+On [Twitter](https://twitter.com/So8res/status/1402888263593959433) Soares conceded my main points, but said that the tone, and the [epistemic-status followup thread](https://twitter.com/So8res/status/1401761124429701121), were intended to indicate that the original thread was "largely in jest"—"shitposting"—but that he was "open to arguments that [he was] making a mistake here."
- * Nate conceded all of my points (https://twitter.com/So8res/status/1402888263593959433), said the thread was in jest ("shitposting"), and said he was open to arguments that he was making a mistake (https://twitter.com/So8res/status/1402889976438611968), but still seemed to think his shitposting was based
+I didn't take that too well, and threw an eleven-Tweet tantrum. I somewhat regret this. My social behavior during this entire episode was histrionic, and I probably could have gotten an equal-or-better outcome if I had kept my cool. The reason I didn't want to keep my cool was because after years of fighting this Category War, MIRI doubling down on "dolphins are fish" felt like a gratuitous insult. I was used to "rationalist" leaders ever-so-humbly claiming to be open to arguments that they were making a mistake, but I couldn't take such assurances seriously if they were going to keep sending PageRank-like credibility to "... Not Man for the Categories".
- * I got frustrated and lashed out; "open to arguments that he was making a mistake" felt fake to me; rats are good at paying lip service to humility, but I'd lost faith in getting them to change their behavior, like not sending PageRank to "... Not Man for the Categories"
+Soares wrote a longer comment on _Less Wrong_ the next morning, and I [pointed out that](https://www.greaterwrong.com/posts/aJnaMv8pFQAfi9jBm/reply-to-nate-soares-on-dolphins/comment/BBtSuWcdaFyvgddE4) Soares's followup thread had lamented ["the fact that nobody's read A Human's Guide to Words or w/e"](https://twitter.com/So8res/status/1401761130041659395), but—with respect—he wasn't behaving like _he_ had read it. Specifically, [#30](https://www.greaterwrong.com/posts/d5NyJ2Lf6N22AD9PB/where-to-draw-the-boundary) on the list of ["37 Ways Words Can Be Wrong"](https://www.greaterwrong.com/posts/FaJaCgqBKphrDzDSj/37-ways-that-words-can-be-wrong) had characterized the position that dolphins are fish as "playing nitwit games". This didn't seem controversial at the time in 2008.
- * Nate wrote a longer reply on Less Wrong the next morning
+And yet it would seem that sometime between 2008 and the current year, the "rationalist" party line (as observed in the public statements of SingInst/MIRI leadership) on whether dolphins are fish shifted from (my paraphrases) "No; _despite_ the surface similarities, that categorization doesn't carve reality at the joints; stop playing nitwit games" to "Yes, _because_ of the surface similarities; those who contend otherwise are the ones playing nitwit games." A complete 180° reversal, on this specific example! Why? What changed?
- * I pointed out that his followup thread lamented that people hadn't read "A Human's Guide to Words", but that Sequence _specifically_ used the example of dolphins. What changed?!?
+It would make sense if people's opinions changed due to new arguments. (Indeed, Yudkowsky's original "stop playing nitwit games" dismissal had been sloppy, and I had had occasion in ["Where to Draw the Boundaries?"](https://www.lesswrong.com/posts/esRZaPXSHgWzyB2NL/where-to-draw-the-boundaries) to explain the specific senses in which dolphins both do and do not cluster with fish.)
- * [Summarize Nate's account of his story], phylogeny not having the courage of its convictions
+But when people change their mind due to new arguments, you'd expect them to acknowledge the change, and explain how the new arguments show that why they thought before was actually wrong. Soares hadn't even acknowledged the change!
- * Twitter exchange where he said he wasn't sure I would count his self-report as evidnece, I said it totally counts
+Soares wrote [a comment explaining](https://www.greaterwrong.com/posts/aJnaMv8pFQAfi9jBm/reply-to-nate-soares-on-dolphins/comment/HwSkiN62QeuEtGWpN) why he didn't think it was such a large reversal. I [started drafting a counterreply](/ancillary/dolphin-war/), but decided that it would need to become a full post on the timescale of days or weeks, partially because I needed to think through how to reply to Soares about paraphyletic groups, and partially because the way the associated Twitter discussion had gone (including some tussling with Yudkowsky) made me want to modulate my tone. (I noted that I had probably lost some in-group credibility in the Twitter fight, but the information gained seemed more valuable. Losing in-group credibility didn't hurt so much when I didn't respect the group anymore.)
- * I overheated. This was an objectively dumb play. (If I had cooled down and just written up my reply, I might have gotten real engagement and a resolution, but I blew it.) I apologized a few days later.
+I was feeling some subjective time pressure on my reply, and in the meantime, I ended up adding [a histrionic comment](https://www.greaterwrong.com/posts/aJnaMv8pFQAfi9jBm/reply-to-nate-soares-on-dolphins/comment/rMHcWfqkH89LWt4y9) to the _Less Wrong_ thread taking issue with Soares's still-flippant tone. That was a terrible performance on my part. (It got downvoted to oblivion, and I deserved it.)
- * Nate's reaction to me blowing up said it looked like I was expecting deference. I deny this; I wouldn't expect people to defer to me—what I did expect was a fair hearing, and at this point, I had lost faith that I would get one. (Could you blame me, when Yudkowsky says a fair hearing is less important than agreeing with Stalin?)
+Soares [wrote that](https://www.greaterwrong.com/posts/aJnaMv8pFQAfi9jBm/reply-to-nate-soares-on-dolphins/comment/8nmjnrm4cwgCCyYrG) I was persistently mis-modeling his intentions, that I seemed to be making a plea for deference that he rejected.
- * My theory of what's going on: I totally believe Nate's self report that he wasn't thinking about gender. (As Nate pointed out, you could give the thread an anti-trans interpretation, too.) Nevertheless, it remains the case that Nate's thinking is causally downstream of Scott's arguments in "... Not Man for the Categories." Where did Scott get it from? I think he pulled it out of his ass because it was politically convenient.
+I don't think I wanted deference. I write these thousands of words in the hopes that people will read my arguments and think it through for themselves; I would never expect anyone to take my word for the conclusion. What I was hoping for was a fair hearing, and by that point, I had lost hope of getting one.
- * This is like radiocontrast dye for dark side epistemology: we can see Scott sneezing his bad epistemology onto everyone else because he's such a popular writer. No one can think fast enough to think their own thoughts, but you would hope for an intellectual community that can do error-correction, rather than copying smart people's views including mistakes.
+As for my skill at modeling intent, I think it's less relevant than Soares seemed to think (if I don't err in attributing to him the belief that modeling intent is important). I believe Soares's self-report that he wasn't trying to make a coded statement about gender; my initial impression otherwise _was_ miscalibrated. (As Soares pointed out, his "dolphins are fish" position could be given an "anti-trans" interpretation, too, in the form of "you intellectuals get your hands off my intuitive concepts". The association between "dolphins are fish" and "trans women are women" ran through their conjunction in Alexander's "... Not Man for the Categories", rather than being intrinsic to the beliefs themselves.)
- * I look up the relevant phylogenetics definitions, and write "Blood Is Thicker Than Water"
-]
+The thing is, I was _right_ to notice the similarity between Soares's argument and "... Not Man for the Categories." Soares's [own account](https://www.greaterwrong.com/posts/aJnaMv8pFQAfi9jBm/reply-to-nate-soares-on-dolphins/comment/HwSkiN62QeuEtGWpN) agreed that there was a causal influence. Okay, so _Nate_ wasn't trying to play gender politics; Scott just alerted him to the idea that people didn't used to be interested in drawing their categories around phylogenetics, and Nate ran with that thought.
-[TODO:
+So where did _Scott_ get it from?
- * depressed after talking to him at Independence Day party 2021 (I can mention that, because it was outdoors and probably lots of other people saw us, even if I can't talk about content)
+I think he pulled it out of his ass because it was politically convenient. I think if you asked Scott Alexander whether dolphins are fish in 2012, he would have said, "No, they're mammals," like any other educated adult.
- * It wouldn't be so bad if he weren't trying to sell himself as a religious leader, and profiting from the conflation of rationalist-someone-who-cares-about-reasoning, and rationalist-member-of-robot-cult
+In a world where the clock of "political time" had run a little bit slower, such that the fight for gay marriage had taken longer [such that the progressive _zeitgeist_ hadn't pivoted to trans as the new cause _du jour_](/2019/Aug/the-social-construction-of-reality-and-the-sheer-goddamned-pointlessness-of-reason/), I don't think Alexander would have had the occasion to write "... Not Man for the Categories." And in that world, I don't think "Dolphins are fish, fight me" or "Acknowledge that all of our categories are weird and a little arbitrary" would have become _memes_ in our subculture.
- * But he does, in fact, seem to actively encourage this conflation (contrast to how the Sequences had a [Litany Against Gurus](https://www.lesswrong.com/posts/t6Fe2PsEwb3HhcBEr/the-litany-against-gurus) these days, with the way he sneers as Earthlings and post-rats)
+This case is like [radiocontrast dye](https://en.wikipedia.org/wiki/Radiocontrast_agent) for [dark side epistemology](https://www.lesswrong.com/posts/XTWkjCJScy2GFAgDt/dark-side-epistemology). Because Scott Alexander won [the talent lottery](https://slatestarcodex.com/2015/01/31/the-parable-of-the-talents/) and writes faster than everyone else, he has the power to _sneeze his mistakes_ onto everyone who trusts Scott to have done his homework, even when he obviously hasn't.
- * a specific example that made me very angry in September 2021—
+[No one can think fast enough to think all their own thoughts.](https://www.lesswrong.com/posts/2MD3NMLBPCqPfnfre/cached-thoughts), but you would hope for an intellectual community that can do error-correction, rather than copying smart people's views including mistakes?
-https://twitter.com/ESYudkowsky/status/1434906470248636419
-> Anyways, Scott, this is just the usual division of labor in our caliphate: we're both always right, but you cater to the crowd that wants to hear it from somebody too modest to admit that, and I cater to the crowd that wants somebody out of that closet.
+To be sure, it's true that there's a cluster of similarities induced by adaptations to the acquatic environment. It's reasonable to want to talk about that subspace. But it doesn't follow that phylogenetics is irrelevant.
-Okay, I get that it was meant as humorous exaggeration. But I think it still has the effect of discouraging people from criticizing Scott or Eliezer because they're the leaders of the Caliphate. I spent three and a half years of my life explaining in exhaustive, exhaustive detail, with math, how Scott was wrong about something, no one serious actually disagrees, and Eliezer is still using his social power to boost Scott's right-about-everything (!!) reputation. That seems really unfair, in a way that isn't dulled by "it was just a joke."
+Genetics is at the root of the causal graph of all other traits of an organism, which induces the kind of conditional independence relationships that make "categories" a useful AI trick.
-Or [as Yudkowsky put it](https://www.facebook.com/yudkowsky/posts/10154981483669228)—
+But in a world where more people are reading "... Not Man for the Categories" than ["Mutual Information, and Density in Thingspace"](https://www.lesswrong.com/posts/yLcuygFfMfrfK8KjF/mutual-information-and-density-in-thingspace), and even the people who have read "Density in Thingspace" (once, ten years ago) are having most of their conversations with people who only read "... Not Man for the Categories"—what happens is that you end up with a so-called "rationalist" culture that completely forgot the hidden-Bayesian-structure-of-cognition/carve-reality-at-the-joints skill! People only remember the specific subset of "A Human's Guide to Words" that's useful for believing whatever you want (by cherry-picking the features you need to include in category Y to make your favorite "X is a Y" sentence look "true", which is easy for intricate high-dimensional things like biological creatures that have a lot of similarities to cherry-pick from), rather than the part about the conditional independence structure in the environment.
-> I know that it's a bad sign to worry about which jokes other people find funny. But you can laugh at jokes about Jews arguing with each other, and laugh at jokes about Jews secretly being in charge of the world, and not laugh at jokes about Jews cheating their customers. Jokes do reveal conceptual links and some conceptual links are more problematic than others.
+After I cooled down, I did eventually write up the explanation for why paraphyletic categories are okay, in ["Blood Is Thicker Than Water"](https://www.lesswrong.com/posts/vhp2sW6iBhNJwqcwP/blood-is-thicker-than-water). But I'm not sure that anyone cared.
-It's totally understandable to not want to get involved in a political scuffle because xrisk reduction is astronomically more important! But I don't see any plausible case that metaphorically sucking Scott's dick in public reduces xrisk. It would be so easy to just not engage in this kind of cartel behavior!
+--------
+
+I got a chance to talk to Yudkowsky in person at the 2021 Event Horizon[^event-horizon] Fourth of July party. In accordance with the privacy norms I'm adhering to while telling this Whole Dumb Story, I don't think I should elaborate on what was said. (It felt like a private conversation, even if most of it was outdoors at a party. No one joined in, and if anyone was listening, I didn't notice them.)
+
+[^event-horizon]: Event Horizon was the name of a group house in Berkeley.
-An analogy: racist jokes are also just jokes. Alice says, "What's the difference between a black dad and a boomerang? A boomerang comes back." Bob says, "That's super racist! Tons of African-American fathers are devoted parents!!" Alice says, "Chill out, it was just a joke." In a way, Alice is right. It was just a joke; no sane person could think that Alice was literally claiming that all black men are deadbeat dads. But, the joke only makes sense in the first place in context of a culture where the black-father-abandonment stereotype is operative. If you thought the stereotype was false, or if you were worried about it being a self-fulfilling prophecy, you would find it tempting to be a humorless scold and get angry at the joke-teller.
+I will say that it finalized my sense that the vision of rationalism he had preached in the Sequences was dead as a cultural force. I was somewhat depressed for months afterwards.
-Similarly, the "Caliphate" humor _only makes sense in the first place_ in the context of a celebrity culture where deferring to Yudkowsky and Alexander is expected behavior. (In a way that deferring to Julia Galef or John S. Wentworth is not expected behavior, even if Galef and Wentworth also have a track record as good thinkers.) I think this culture is bad. _Nullius in verba_.
+It wouldn't be so bad if Yudkowsky weren't trying to sell himself as a _de facto_ religious leader,[^religious-leader] profiting from the conflation of _rationalist_ in the sense of "one who aspires to systematically correct reasoning" and _rationalist_ as member of his fan-club/personality-cult.
- * the fact that David Xu interpreted criticism of the robot cult as me going "full post-rat" suggests that Yudkowsky's framing had spilled onto others. (The framing is optimized to delegitimize dissent. Motte: someone who's critical of central rationalists; bailey: someone who's moved beyond reason.)
+[^religious-leader]: "Religious leader" continues to seem like an apt sociological description, even if [no supernatural claims are being made](https://www.lesswrong.com/posts/u6JzcFtPGiznFgDxP/excluding-the-supernatural).
-sneering at post-rats; David Xu interprets criticism of Eliezer as me going "full post-rat"?! 6 September 2021
+But he does seem to actively encourage this conflation. Contrast the [Litany Against Gurus](https://www.lesswrong.com/posts/t6Fe2PsEwb3HhcBEr/the-litany-against-gurus) from the Sequences, to the way he sneers at "post-rationalists"—or even "Earthlings" in general (in contrast to his fictional world of dath ilan). The framing is optimized to delegitimize dissent. [Motte](https://slatestarcodex.com/2014/11/03/all-in-all-another-brick-in-the-motte/): someone who's critical of central "rationalists" like Yudkowsky or Alexander; bailey: someone who's moved beyond reason itself.
-> Also: speaking as someone who's read and enjoyed your LW content, I do hope this isn't a sign that you're going full post-rat. It was bad enough when QC did it (though to his credit QC still has pretty decent Twitter takes, unlike most post-rats).
+One example that made me furious came in September 2021. Yudkowsky, replying to Scott Alexander on Twitter, [wrote](https://twitter.com/ESYudkowsky/status/1434906470248636419):
-https://twitter.com/davidxu90/status/1435106339550740482
+> Anyways, Scott, this is just the usual division of labor in our caliphate: we're both always right, but you cater to the crowd that wants to hear it from somebody too modest to admit that, and I cater to the crowd that wants somebody out of that closet.
+
+I understand, of course, that it was meant as humorous exaggeration. But I think it still has the effect of discouraging people from criticizing Yudkowsky or Alexander because they're the leaders of the Caliphate. I had just spent more than three and a half years of my life[^years-of-my-life] [explaining in](/2018/Feb/the-categories-were-made-for-man-to-make-predictions/) [exhaustive](https://www.lesswrong.com/posts/esRZaPXSHgWzyB2NL/where-to-draw-the-boundaries), [exhaustive](https://www.lesswrong.com/posts/onwgTH6n8wxRSo2BJ/unnatural-categories-are-optimized-for-deception) [detail](https://www.lesswrong.com/posts/vhp2sW6iBhNJwqcwP/blood-is-thicker-than-water), with math, how Alexander was wrong about something, no one serious actually disagrees, and Yudkowsky was still using his social power to boost Scott's right-about-everything (!!) reputation. That seemed egregiously unfair, in a way that wasn't dulled by "it was just a joke."
+
+[^years-of-my-life]: I started outlining ["The Categories Where Made for Man to Make Predictions"](/2018/Feb/the-categories-were-made-for-man-to-make-predictions/) in January 2018. I would finally finish ["Blood Is Thicker Than Water"](https://www.lesswrong.com/posts/vhp2sW6iBhNJwqcwP/blood-is-thicker-than-water), following up on the "dolphins are fish" claim later that month of September 2021.
-https://twitter.com/zackmdavis/status/1435856644076830721
-> The error in "Not Man for the Categories" is not subtle! After the issue had been brought to your attention, I think you should have been able to condemn it: "Scott's wrong; you can't redefine concepts in order to make people happy; that's retarded." It really is that simple! 4/6
+Or [as Yudkowsky had once put it](https://www.facebook.com/yudkowsky/posts/10154981483669228)—
+
+> I know that it's a bad sign to worry about which jokes other people find funny. But you can laugh at jokes about Jews arguing with each other, and laugh at jokes about Jews secretly being in charge of the world, and not laugh at jokes about Jews cheating their customers. Jokes do reveal conceptual links and some conceptual links are more problematic than others.
+
+It's totally understandable to not want to get involved in a political scuffle because xrisk reduction is astronomically more important! But I don't see any plausible case that metaphorically sucking Scott's dick in public reduces xrisk. It would be so easy to just not engage in this kind of cartel behavior!
+
+An analogy: racist jokes are also just jokes. Alice says, "What's the difference between a black dad and a boomerang? A boomerang comes back." Bob says, "That's super racist! Tons of African-American fathers are devoted parents!!" Alice says, "Chill out, it was just a joke." In a way, Alice is right. It was just a joke; no sane person could think that Alice was literally claiming that all black men are deadbeat dads. But the joke only makes sense in the first place in context of a culture where the black-father-abandonment stereotype is operative. If you thought the stereotype was false, or if you were worried about it being a self-fulfilling prophecy, you would find it tempting to be a humorless scold and get angry at the joke-teller.
+
+Similarly, the "Caliphate" humor only makes sense in the first place in the context of a celebrity culture where deferring to Yudkowsky and Alexander is expected behavior. (In a way that deferring to Julia Galef or John S. Wentworth is not expected behavior, even if Galef and Wentworth also have a track record as good thinkers.) I think this culture is bad. _Nullius in verba_.
+
+I don't think the motte-and-bailey concern is hypothetical, either. When I [indignantly protested](https://twitter.com/zackmdavis/status/1435059595228053505) the "we're both always right" remark, one David Xu [commented](https://twitter.com/davidxu90/status/1435106339550740482): "speaking as someone who's read and enjoyed your LW content, I do hope this isn't a sign that you're going full post-rat"—as if my criticism of Yudkowsky's self-serving bluster itself marked me as siding with the "post-rats"!
I once wrote [a post whimsically suggesting that trans women should owe cis women royalties](/2019/Dec/comp/) for copying the female form (as "intellectual property"). In response to a reader who got offended, I [ended up adding](/source?p=Ultimately_Untrue_Thought.git;a=commitdiff;h=03468d274f5) an "epistemic status" line to clarify that it was not a serious proposal.
From my perspective, such advice would be missing the point. [I'm not trying to force though some particular policy.](/2021/Sep/i-dont-do-policy/) Rather, I think I _know some things_ about the world, things I wish I had someone had told me earlier. So I'm trying to tell others, to help them live in _a world that makes sense_.
-]
-
+------
[David Xu writes](https://twitter.com/davidxu90/status/1436007025545125896) (with Yudkowsky ["endors[ing] everything [Xu] just said"](https://twitter.com/ESYudkowsky/status/1436025983522381827)):
I don't doubt Yudkowsky could come up with some clever casuistry why, _technically_, the text he wrote in 2007 and the text he endorsed in 2021 don't contradict each other. But _realistically_ ... again, no.
-[TODO: elaborate on how 2007!Yudkowsky and 2021!Xu are saying the opposite things if you just take a plain-language reading and consider, not whether individual sentences can be interpreted as "true", but what kind of _optimization_ the text is doing to the behavior of receptive readers]
-
I don't, actually, expect people to spontaneously blurt out everything they believe to be true, that Stalin would find offensive. "No comment" would be fine. Even selective argumentation that's _clearly labeled as such_ would be fine. (There's no shame in being an honest specialist who says, "I've mostly thought about these issues though the lens of ideology _X_, and therefore can't claim to be comprehensive; if you want other perspectives, you'll have to read other authors and think it through for yourself.")
What's _not_ fine is selective argumentation while claiming "confidence in [your] own ability to independently invent everything important that would be on the other side of the filter and check it [yourself] before speaking" when you _very obviously have done no such thing_.
-----
-[TODO: bridge—link to pulled-out standalone post, "On the Public Anti-Epistemology of dath ilan"]
+Perhaps for [lack of any world-saving research to do](https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-announces-new-death-with-dignity-strategy), Yudkowsky started writing fiction again, largely in the form of Glowfic (a genre of collaborative storytelling pioneered by Alicorn) featuring the world of dath ilan.
+
+The bulk of the dath ilan Glowfic canon was an epic titled [_Planecrash_](https://www.glowfic.com/boards/215)[^planecrash-title] coauthored with Lintamande, in which Keltham, an unusually selfish teenage boy from dath ilan, apparently dies in a freak aviation accident, and [wakes up in the world of](https://en.wikipedia.org/wiki/Isekai) Golarion, setting of the _Dungeons-&-Dragons_–alike _Pathfinder_ role-playing game. A [couple](https://www.glowfic.com/posts/4508) of [other](https://glowfic.com/posts/6263) Glowfic stories with different coauthors further flesh out the setting of dath ilan, a smarter, better-coordinated alternate version of Earth steered by an order of [Keepers of Highly Unpleasant Things it is Sometimes Necessary to Know](https://www.glowfic.com/replies/1612937#reply-1612937) who safeguard advanced rationality techniques from a population allegedly too psychologically fragile to handle them.
+
+[^planecrash-title]: The title is a triple pun, referring to the airplane crash leading to Keltham's death in dath ilan, and how his resurrection in Golarion collides dath ilan with [the "planes" of existence of the _Pathfinder_ universe](https://pathfinderwiki.com/wiki/Great_Beyond), and Keltham's threat to destroy (crash) the _Pathfinder_ reality if mortals aren't given better afterlife conditions. (I use the word "threat" colloquially here; the work itself goes into some detail distinguishing between mere bargaining and decision-theoretic threats that should be defied.)
+
+On the topic of dath ilan's rationality training, I appreciated [this passage about the cognitive function of categorization](https://www.glowfic.com/replies/1779051#reply-1779051):
+
+> Dath ilani kids get told to not get fascinated with the fact that, in principle, 'bounded-agents' with finite memories and finite thinking speeds, have any considerations about mapping that depend on what they want. It doesn't mean that you get to draw in whatever you like on your map, because it's what you want. It doesn't make reality be what you want.
+
+Vindication! This showed that Yudkowsky _did_ understand what was at issue in the dispute over "... Not Man for the Categories", even if he couldn't say "Zack is right and Scott is wrong" for political reasons.
+
+Despite that tidbit, however, the dath ilan mythos overall seemed visibly defective to me compared to the Sequences—particularly regarding how humans should relate to unpleasant truths.
Someone at the 2021 Event Horizon Independence Day party had told me that I had been misinterpreting the "Speak the truth, even if your voice trembles" slogan from the Sequences. I had interpreted the slogan as suggesting the importance of speaking the truth _to other people_ (which I think is what "speaking" is usually about), but my interlocutor said it was about, for example, being able to speak the truth aloud in your own bedroom, to yourself. I think some textual evidence for my interpretation can be found in Daria's ending to ["A Fable of Science and Politics"](https://www.lesswrong.com/posts/6hfGNLf4Hg5DXqJCF/a-fable-of-science-and-politics), a multiple-parallel-endings story about an underground Society divided into factions over the color of the unseen sky, and one person's reaction when they find a passageway leading aboveground to a view of the sky:
[^other-endings]: Even Eddin's ending, which portrays Eddin as more concerned with consequences than honesty, has him "trying to think of a way to prevent this information from blowing up the world", rather than trying to think of a way to suppress the information, in contrast to how Charles, in his ending, _immediately_ comes up with the idea to block off the passageway leading to the aboveground. Daria and Eddin are clearly written as "rationalists"; the deceptive strategy only comes naturally to the non-rationalist Charles. (Although you could Watsonianly argue that Eddin is just thinking longer-term than Charles: blocking off _this_ passageway and never speaking a word of it to another soul, won't prevent someone from finding some other passage to the aboveground, eventually.)
-In contrast, the culture of dath ilan does not seem to particularly value people _standing under the same sky_.
+In contrast, the culture of dath ilan does not seem to particularly value people _standing under the same sky_. It's not just the Keepers; we also see many other cases of the dath ilani covering things up for some alleged greater good with seemingly no regard to the costs of people have less accurate world-models.
-[...]
+In one notable example, Keltham, the protagonist of _Planecrash_, is an obligate sexual sadist, but never discovered this fact about himself during his first life in dath ilan, because dath ilan has arranged to cover up the existence of sadism and masochism, because sadists like Keltham would be sad if they discovered that there weren't enough masochists to go around.
-On the topic of dath ilan's rationality training, I did appreciate [this passage about the cognitive function of categorization](https://www.glowfic.com/replies/1779051#reply-1779051):
+It did not escape my notice that when "rationalist" authorities in real life considered public knowledge of some paraphilia to be an infohazard (ostensibly for the benefit of people with that paraphilia), I _didn't take it lying down_.
- > Dath ilani kids get told to not get fascinated with the fact that, in principle, 'bounded-agents' with finite memories and finite thinking speeds, have any considerations about mapping that depend on what they want. It doesn't mean that you get to draw in whatever you like on your map, because it's what you want. It doesn't make reality be what you want.
+The parallel between dath ilan's sadism/masochism coverup and the autogynephilia coverup I had fought in real life, was something I was only intending to comment on in passing in the present memoir, rather than devoting any more detailed analysis to, but as I was having trouble focusing on my own writing in September 2022, I ended up posting some critical messages about dath ilan's censorship regime in the "Eliezerfic" Discord server for reader discussion of _Planecrash_, using the sadism/masochism coverup as my central example.
-Vindication! (This showed that Yudkowsky _does_ understand what was at issue in the "... Not Man for the Categories" dispute, even if I can't be credited with winning the argument for political reasons.)
+(I would later adapt my complaints into a standalone post, "On the Public Anti-Epistemology of dath ilan".)
-----------
+Although Yudkowsky participated in the server, I had reasoned that my participation didn't violate my previous intent not to bother him anymore, because it was a publicly-linked Discord server with hundreds of members. Me commenting on the story for the benefit of the _other_ 499 people in the chatroom wouldn't generate a notification _for him_, the way it would if I sent him an email or replied to him on Twitter.
-A user called ajvermillion asked why I was being so aggressively negative about dath ilan. He compared it to Keltham's remark about how [people who grew up under a Lawful Evil government were disposed to take a more negative view of paternalism](https://www.glowfic.com/replies/1874754#reply-1874754) than they do in dath ilan, where paternalism basically works fine because dath ilan is benevolent.
+The other chatroom participants mostly weren't buying what I was selling.
-This question put me in a somewhat awkward position: it was a legitimate question that I felt I had to answer, that I had no way of answering honestly without at least _alluding_ to my prior greviances against Yudkowsky ... which were off-topic for the server. (Again, I had told myself that I was here to comment on the story, not to prosecute my greviances.)
+When I objected to [Word of God](https://tvtropes.org/pmwiki/pmwiki.php/Main/WordOfGod)'s identification of the Keeper's credo as "Let the truth destroy what it can—in yourself, not in other people" as an incredibly infantalizing philosophy, someone replied, "I think of 'not in other people' not as 'infantilizing', but as recognizing independent agency. You don't get to do harm to other people without their consent, whether that is physical or pychological."
-I tried to explain, briefly. Someone who might be even _more_ paranoid about abuses of power than someone who grew up with a Lawful Evil government, is someone who grew up under a power structure that put on a _good show_ of being clean and nice, but was actually corrupt and mean.
+I pointed out that this obviously applies to, say, religion. Was it wrong to advocate for atheism in a religious Society, where robbing someone of their belief in God might be harming them?
-Yudkowsky had this whole marketing image of him being uniquely sane and therefore uniquely benevolent, and because his Sequences were so life-changingly good, I _actually fell for it_. There was a long Dumb Story (this Story) that was off-topic and I hadn't then finished writing up, but basically, I had what I claimed were very strong reasons not to trust the guy anymore; I think he cares a lot about not explicitly _lying_, but what made the Sequences special is that they articulated a vastly higher standard than that, that he had no intention of living up to.
+"Every society strikes a balance between protectionism and liberty," someone said. "This isn't news."
-And so, yeah, insofar as fiction about dath ilan functioned as marketing material for Yudkowsky's personality cult that I thought was damaging people like me (in some ways, while simultaneously helping us in other ways), I had an incentive to come up with literary criticism that paints dath ilan negatively?
+It's not news about _humans_, I conceded. But I thought people who were fans of Yudkowsky's writing in 2008 had a reasonable expectation that the dominant messaging in the local subculture would continue in 2022 to be _in favor_ of telling the truth and _against_ benevolently intended noble lies. It would be interesting to know why that had changed.
-It was great for ajvermillion to notice this! It _would_ be bad if my brain were configured to come up with dath-ilan-negative literary criticism, and for me to _simultaneously_ present myself as an authority on dath ilan whom you should trust. But if dath-ilan-negative literary criticism was undersupplied for structural reasons (because people who like a story are selected for not seeing things the story is doing that are Actually Bad), and my brain was configured to generate it anyway (because I disliked the person Yudkowsky had become, in contrast to the person he was in 2008), it seemed pro-social for me to post it, for other people to take or leave according to their own judgement?
+I started a new thread for my topic (Subject: "Noble Secrets; Or, Conflict Theory of Optimization on Shared Maps"). It died out after a couple days, but I reopened it later in response to more discussion of the masochism coverup.
-Yudkowsky soon entered the thread again, initially replying to someone else. (I remarked parenthetically that his appearance made me think I should stop wasting time snarking in his fiction server and just finish my memoir already.) We had a brief back-and-forth:
+Yudkowsky made an appearance. (After he replied to someone else, I remarked parenthetically that his appearance made me think I should stop wasting time snarking in his fiction server and just finish my memoir already.) We had a brief back-and-forth:
> **Eliezer** — 11/29/2022 10:33 PM
> the main thing I'd observe contrary to Zack's take here, is that Keltham thought that not learning about masochists he can never have, was obviously in retrospect what he'd have wanted Civilization to do, or do unless and until Keltham became rich enough to afford a masochist and then he could be told
> **Eliezer** — 11/29/2022 10:36 PM
> I am sorry that some of the insane people I attracted got together and made each other more insane and then extensively meta-gaslit you into believing that everyone generally and me personally was engaging in some kind of weird out-in-the-open gaslighting that you could believe in if you attached least-charitable explanations to everything we were doing
-It was pretty annoying that Yudkowsky was still attributing my greviances to Michael's malign influence—as if the gender identity revolution was something I would otherwise have just _taken lying down_. In the counterfactual where Michael had died in 2015, I think something like my February 2017 breakdown would have likely happened anyway. (Between August 2016 and January 2017, I sent Michael 14 emails, met with him once, and watched 60% of South Park season 19 at his suggestion, so he was _an_ influence on my thinking during that period, but not a disproportionately large one compared to everything else I was doing at the time.) How would I have later reacted to the November 2018 "hill of meaning" Tweets (assuming they weren't butterfly-effected away in this counterfactual)? It's hard to say. Maybe, if that world's analogue of my February 2017 breakdown had gone sufficiently badly (with no Michael to visit me in the psych ward or help me make sense of things afterwards), I would have already been a broken man, and not even sent Yudkowsky an email. In any case, I feel very confident that my understanding of the behavior of "everyone generally and [Yudkowsky] personally" would not have been _better_ without Michael _et al._'s influence.
+It was pretty annoying that Yudkowsky was still attributing my greviances to Michael's malign influence—as if the gender identity revolution was something I would otherwise have just taken lying down. In the counterfactual where Michael had died in 2015, I think something like my February 2017 breakdown would have likely happened anyway. (Between August 2016 and January 2017, I sent Michael 14 emails, met with him once, and watched 60% of South Park season 19 at his suggestion, so he was _an_ influence on my thinking during that period, but not a disproportionately large one compared to everything else I was doing at the time.) How would I have later reacted to the November 2018 "hill of meaning" Tweets (assuming they weren't butterfly-effected away in this counterfactual)? It's hard to say. Maybe, if that world's analogue of my February 2017 breakdown had gone sufficiently badly (with no Michael to visit me in the psych ward or help me make sense of things afterwards), I would have already been a broken man, and not even sent Yudkowsky an email. In any case, I feel very confident that my understanding of the behavior of "everyone generally and [Yudkowsky] personally" would not have been _better_ without Michael _et al._'s influence.
> [cont'd]
> you may recall that this blog included something called the "Bayesian Conspiracy"
> they won't tell you about it, because it interferes with the story they were trying to drive you insaner with, but it's so
> **zackmdavis** — 11/29/2022 10:37 PM
-> it's true that the things I don't like about modern Yudkowsky were still there in Sequences-era Yudkowsky, but I think they've gotten _worse_
+> it's true that the things I don't like about modern Yudkowsky were still there in Sequences-era Yudkowsky, but I think they've gotten _worse_
> **Eliezer** — 11/29/2022 10:39 PM
-> well, if your story is that I was always a complicated person, and you selected some of my posts and liked the simpler message you extracted from those, and over time I've shifted in my emphases in a way you don't like, while still having posts like Meta-Honesty and so on... then that's a pretty different story than the one you were telling in this Discord channel, like, just now. today.
+> well, if your story is that I was always a complicated person, and you selected some of my posts and liked the simpler message you extracted from those, and over time I've shifted in my emphases in a way you don't like, while still having posts like Meta-Honesty and so on... then that's a pretty different story than the one you were telling in this Discord channel, like, just now. today.
-Is it, though? The "always a complicated person [who has] shifted in [his] emphases in a way [I] don't like" story was true, of course, but it elided the substantive reasons _why_ I didn't like the new emphases, which could presumably be evaluated on their own merits.
+Is it, though? The "always a complicated person [who has] shifted in [his] emphases in a way [I] don't like" story was true, of course, but it elided the substantive reasons _why_ I didn't like the new emphases, which I expect other people to be able to see, too.
-It's interesting that Yudkowsky listed "still having posts like Meta-Honesty" as an exculpatory factor here. The thing is, I [wrote a _critique_ of Meta-Honesty](https://www.lesswrong.com/posts/MN4NRkMw7ggt9587K/firming-up-not-lying-around-its-edge-cases-is-less-broadly). It was well-received (being [cited as a good example in the introductory post for the 2019 Less Wrong Review](https://www.lesswrong.com/posts/QFBEjjAvT6KbaA3dY/the-lesswrong-2019-review), for instance). I don't think I could have written a similarly impassioned critique of anything from the Sequences era, because the stuff from the Sequences era still looked _correct_ to me. To me, "Meta-Honesty" was evidence _for_ Yudkowsky having relinquished his Art and lost his powers, not evidence that his powers were still intact.
+(As far as [the](https://www.lesswrong.com/posts/fnEWQAYxcRnaYBqaZ/initiation-ceremony) [Bayesian](https://www.lesswrong.com/posts/ZxR8P8hBFQ9kC8wMy/the-failures-of-eld-science) [Conspiracy](https://www.lesswrong.com/posts/xAXrEpF5FYjwqKMfZ/class-project) [stories](https://www.lesswrong.com/posts/kXAb5riiaJNrfR8v8/the-ritual) [went](https://www.lesswrong.com/posts/yffPyiu7hRLyc7r23/final-words), I think there's a significant narrative contrast between Brennan _seeking_ knowledge from the master _beisutsukai_, and Keltham, Merrin, and Thellim being _protected from_ knowledge by the Keepers. Neither the Bayesian Conspiracy nor the Keepers are publishing open-access textbooks, but at least the Conspiracy isn't claiming that their secretiveness is _for others' benefit_.)
+
+It's notable that Yudkowsky listed "still having posts like [Meta-Honesty](https://www.lesswrong.com/posts/xdwbX9pFEr7Pomaxv/meta-honesty-firming-up-honesty-around-its-edge-cases)" as an exculpatory factor here. The thing is, I [wrote a _critique_ of Meta-Honesty](https://www.lesswrong.com/posts/MN4NRkMw7ggt9587K/firming-up-not-lying-around-its-edge-cases-is-less-broadly). It was well-received (being [cited as a good example in the introductory post for the 2019 Less Wrong Review](https://www.lesswrong.com/posts/QFBEjjAvT6KbaA3dY/the-lesswrong-2019-review), for instance). I don't think I could have written a similarly impassioned critique of anything from the Sequences era, because the stuff from the Sequences era still looked correct to me. To me, "Meta-Honesty" was evidence _for_ Yudkowsky having relinquished his Art and lost his powers, not evidence that his powers were still intact.
I didn't have that response thought through in real time. At the time, I just agreed:
(When a literary critic proposes a "dark" interpretation of an author's world, I think it's implied that they're expressing disbelief in the "intended" world; the fact that I was impudently refusing to buy the benevolent interpretation wasn't because I didn't understand it.)
-> Hate-warp like this is bad for truth-perception; my understanding of the situation is that it's harm done to you by the group you say you left. I would read this as being a noninnocent error of that group; that they couldn't get what they wanted from people who still had friends outside their own small microculture, and noninnocently then decided that this outer culture was bad and people needed to be pried loose from it. They tried telling some people that this outer culture was gaslighting them and maliciously lying to them and had to be understood in wholly adversarial terms to break free of the gaslighting; that worked on somebody, and made a new friend for them; so their brain noninnocently learned that it ought to use arguments like that again, so they must be true.
+> Hate-warp like this is bad for truth-perception; my understanding of the situation is that it's harm done to you by the group you say you left. I would read this as being a noninnocent error of that group; that they couldn't get what they wanted from people who still had friends outside their own small microculture, and noninnocently then decided that this outer culture was bad and people needed to be pried loose from it. They tried telling some people that this outer culture was gaslighting them and maliciously lying to them and had to be understood in wholly adversarial terms to break free of the gaslighting; that worked on somebody, and made a new friend for them; so their brain noninnocently learned that it ought to use arguments like that again, so they must be true.
> This is a sort of thing I super did not do because I _understood_ it as a failure mode and Laid My Go Stones Against Ever Actually Being A Cult; I armed people with weapons against it, or tried to, but I was optimistic in my hopes about how much could actually be taught.
> **zackmdavis** — 11/29/2022 11:20 PM
> Without particularly defending Vassar _et al._ or my bad literary criticism (sorry), _modeling the adversarial component of non-innocent errors_ (as contrasted to "had to be understood in wholly adversarial terms") seems very important. (Maybe lying is "worse" than rationalizing, but if you can't hold people culpable for rationalization, you end up with a world that's bad for broadly the same reasons that a world full of liars is bad: we can't steer the world to good states if everyone's map is full of falsehoods that locally benefitted someone.)
> **zackmdavis** — 11/29/2022 11:22 PM
> yeah
> **Eliezer** — 11/29/2022 11:23 PM
-> It remains something that mortals do, and if you cut off anybody who's ever done that, you'll be left with nobody. And also importantly, people making noninnocent errors, if you accuse them of malice, will look inside themselves and correctly see that this is not how they work, and they'll stop listening to the (motivated) lies you're telling them about themselves.
+> It remains something that mortals do, and if you cut off anybody who's ever done that, you'll be left with nobody. And also importantly, people making noninnocent errors, if you accuse them of malice, will look inside themselves and correctly see that this is not how they work, and they'll stop listening to the (motivated) lies you're telling them about themselves.
> This also holds true if you make up overly simplistic stories about 'ah yes well you're doing that because you're part of $woke-concept-of-society' etc.
> **zackmdavis** — 11/29/2022 11:24 PM
> I think there's _also_ a frequent problem where you try to accuse people of non-innocent errors, and they motivatedly interpret _you_ as accusing malice
In response to someone positing that dath ilani were choosing to be happier but less accurate predictors, I said that I read a blog post once about why you actually didn't want to do that, linking to [an Internet Archive copy of "Doublethink (Choosing to Be Biased)"](https://web.archive.org/web/20080216204229/https://www.overcomingbias.com/2007/09/doublethink-cho.html) from 2008[^hanson-conceit]—at least, that was _my_ attempted paraphrase; it was possible that I'd extracted a simpler message from it than the author intended.
-[^hanson-conceit]: I was really enjoying the "Robin Hanson's blog in 2008" conceit.
+[^hanson-conceit]: I was enjoying the conceit of referring to Sequences posts as being from "Robin Hanson's blog in 2008", as a way of emphasizing the distinction between my respect for the material and my contempt for the man Yudkowsky had become.
A user called Harmless explained the loophole. "Doublethink" was pointing out that decisions that optimize the world for your preferences can't come from nowhere: if you avoid painful thoughts in your map, you damage your ability to steer away from painful outcomes in the territory. However, there was no rule that all the information-processing going into decisions that optimize the world for your preferences had to take place in _your brain_ ...
I understood the theory, but I was still extremely skpetical of the practice, assuming the eliezera were even remotely human. Yudkowsky described the practice of "keeping BDSM secret and trying to prevent most sadists from discovering what they are—informing them only when and if they become rich enough or famous enough that they'd have a high probability of successfully obtaining a very rare masochist" as a "basically reasonable policy option that [he] might vote for, not to help the poor dear other people, but to help [his] own counterfactual self."
-The problem I saw with this is that becoming rich and famous isn't a purely random exogenous event. In order to make an informed decision about whether or not to put in the effort to try to _become_ rich and famous (as contrasted to choosing a lower-risk or more laid-back lifestyle), you need accurate beliefs about the perks of being rich and famous.
-
-The dilemma of whether to make more ambitious economic choices in pusuit of sexual goals was something that _already_ happens to people on Earth, rather than being hypothetical. I once met a trans woman who spent a lot of her twenties and thirties working very hard to get money for various medical procedures. I think she would be worse off under a censorship regime run by self-styled Keepers who thought it was kinder to prevent _poor people_ from learning about the concept of "transsexualism".
-
-Further discussion established that Yudkowsky was (supposedly) already taking into account that class of distortion on individuals' decisions, but that the empirical setting of probabilities and utilities happened to be such that ignorance came out on top.
-
I wasn't sure what my wordcount and "diplomacy" "budget limits" for the server were, but I couldn't let go; I kept the thread going on subsequent days. There was something I felt I should be able to convey, if I could just find the right words.
When [Word of God](https://tvtropes.org/pmwiki/pmwiki.php/Main/WordOfGod) says, "trying to prevent most [_X_] from discovering what they are [...] continues to strike me as a basically reasonable policy option", then, separately from the particular value of _X_, I expected people to jump out of their chairs and say, "No! This is wrong! Morally wrong! People can stand what is true about themselves, because they are already doing so!"
I wish I had thought to ask if he'd have felt the same way in 2008.
-Ajvermillion was still baffled at my skepticism: if the author specifies that the world of the story is simple in this-and-such direction, on what grounds could I _disagree_?
+A user called Ajvermillion continued to be baffled at my skepticism: if the author specifies that the world of the story is simple in this-and-such direction, on what grounds could I _disagree_?
I admitted, again, that there was a sense in which I couldn't argue with authorial fiat. But I thought that an author's choice of assumptions reveals something about what they think is true in our world, and commenting on that should be fair game for literary critics. Suppose someone wrote a story and said, "in the world portrayed in this story, everyone is super-great at _kung fu_, and they could beat up everyone from our Earth, but they never have to practice at all."
There were definitely communities on Earth where I wasn't allowed in because of my tendency to shout things from street corners, and I respected those people's right to have a safe space for themselves.
-But those communities ... didn't call themselves _rationalists_, weren't _pretending_ be to be inheritors of the great tradition of E. T. Jaynes and Robin Dawes and Richard Feynman. And if they _did_, I think I would have a false advertising complaint against them.
+But those communities didn't call themselves _rationalists_, weren't _pretending_ be to be inheritors of the great tradition of E. T. Jaynes and Richard Feynman and Robin Hanson. And if they _did_, I think I would have a false advertising complaint against them.
"[The eleventh virtue is scholarship. Study many sciences and absorb their power as your own](https://www.yudkowsky.net/rational/virtues) ... unless a prediction market says that would make you less happy," just didn't have the same ring to it. Neither did "The first virtue is curiosity. A burning itch to know is higher than a solemn vow to pursue truth. But higher than both of those, is trusting your Society's institutions to tell you which kinds of knowledge will make you happy"—even if you stipulated by authorial fiat that your Society's institutions are super-competent, such that they're probably right about the happiness thing.
-Attempting to illustrate [the mood I thought dath ilan was missing](https://www.econlib.org/archives/2016/01/the_invisible_t.html), I quoted (with Discord's click-to-reveal spoiler blocks around the more plot-relevant sentences) the scene from _Atlas Shrugged_ where our heroine Dagny expresses a wish to be kept ignorant for the sake of her own happiness, and gets shut down by John Galt—and Dagny _thanks_ him.[^atlas-shrugged-ref]
-
-> "[...] Oh, if only I didn't have to hear about it! If only I could stay here and never know what they're doing to the railroad, and never learn when it goes!"
->
-> "You'll have to hear about it," said Galt; it was that ruthless tone, peculiarly his, which sounded implacable by being simple, devoid of any emotional value, save the quality of respect for facts. "You'll hear the whole course of the last agony of Taggart Transcontinental. You'll hear about every wreck. You'll hear about every discontinued train. You'll hear about every abandoned line. You'll hear about the collapse of the Taggart Bridge. Nobody stays in this valley except by a full, conscious choice based on a full, conscious knowledge of every fact involved in his decision. Nobody stays here by faking reality in any manner whatever."
->
-> She looked at him, her head lifted, knowing what chance he was rejecting. She thought that no man of the outer world would have said this to her at this moment—she thought of the world's code that worshipped white lies as an act of mercy—she felt a stab of revulsion against that code, suddenly seeing its full ugliness for the first time [...] she answered quietly, "Thank you. You're right."
-
-[^atlas-shrugged-ref]: In Part Three, Chapter II, "The Utopia of Greed".
+There's a scene in _Atlas Shrugged_ that I think illustrates [the mood dath ilan is missing](https://www.econlib.org/archives/2016/01/the_invisible_t.html).[^atlas-shrugged-ref] Our heroine Dagny expresses a wish to be kept ignorant for the sake of her own happiness. "Oh, if only I didn't have to hear about it!" she says. "If only I could stay here and never know what they're doing to the railroad, and never learn when it goes!"
-This (probably predictably) failed to resonate with other server participants, who were baffled why I seemed to be appealing to Ayn Rand's authority.
+John Galt isn't having it. "You'll have to hear about it," he says, in "that ruthless tone, peculiarly his, which sounded implacable by being simple, devoid of any emotional value, save the quality of respect for facts", listing the disasters fated to befall the railroad before concluding, "Nobody stays here by faking reality in any manner whatever."
-I was actually going for a _reverse_ appeal-to-authority: if _Ayn Rand_ understood that facing reality is virtuous, why didn't the 2020s "rationalists"? Wasn't that undignified? I didn't think the disdain for "Earth people" (again, as if there were any other kind) was justified, when Earth's philosophy of rationality (as exemplified by Ayn Rand or Robert ["Get the Facts"](https://www.goodreads.com/quotes/38764-what-are-the-facts-again-and-again-and-again) Heinlein) was doing better than dath ilan's on this critical dimension.
+"Thank you. You're right," Dagny says, noting the contrast of Galt's tough-minded honesty with "the world's code that worshipped white lies as an act of mercy".
-But if people's souls had been damaged such that they didn't have the "facing reality is virtuous" gear, it wasn't easy to install the gear by talking at them.
-
-Why was I so sure _my_ gear was correct?
-
-I wondered if the issue had to do with what Yudkowsky had [identified as the problem of non-absolute rules](https://www.lesswrong.com/posts/xdwbX9pFEr7Pomaxv/meta-honesty-firming-up-honesty-around-its-edge-cases#5__Counterargument__The_problem_of_non_absolute_rules_), where not-literally-absolute rules like "Don't kill" or "Don't lie" have to be stated _as if_ they were absolutes in order to register to the human motivational system with sufficient force.
-
-Technically, as a matter of decision theory, "sacred values" are crazy. It's easy to say—and feel with the passion of religious conviction—that it's always right to choose Truth and Life, and that no one could choose otherwise except wrongly, in the vile service of Falsehood and Death. But reality presents us with quantitative choices over uncertain outcomes, in which everything trades off against everything else under the [von Neumann–Morgenstern axioms](https://en.wikipedia.org/wiki/Von_Neumann%E2%80%93Morgenstern_utility_theorem); if you had to choose between a small, unimportant Truth and the Life of millions, you'd probably choose Life—but more importantly, the very fact that you might have to choose, means that Truth and Life can't both be infinitely sacred to you, and must be measured on a common scale with lesser goods like mere Happiness.
-
-I knew that. The other people in the chatroom knew that. So to the extent that the argument amounted to me saying "Don't lie" (about the existence of masochism), and them saying "Don't lie unless the badness of lying is outweighed by the goodness of increased happiness", why was I so confident that I was in the right, when they were wisely acknowledging the trade-offs under the Law, and I was sticking to my (incoherent) sacred value of Truth? Didn't they obviously have the more sophisticated side of the argument?
+[^atlas-shrugged-ref]: In Part Three, Chapter II, "The Utopia of Greed".
-The problem was that, in my view, the people who weren't talking about Truth as if it were a sacred value were being _wildly recklessly casual_ about harms from covering things up, as if they didn't see the non-first-order harms _at all_. I felt I had to appeal to the lessons for children about how Lying Is Bad, because if I tried to make a more sophisticated argument about it being _quantitatively_ crazy to cover up psychology facts that make people sad, I would face a brick wall of "authorial fiat declares that the probabilities and utilities are specifically fine-tuned such that ignorance is good".
+When I quoted this scene in the server, other participants were baffled that I seemed to be appealing to Ayn Rand's authority. I was actually going for a _reverse_ appeal-to-authority: if _Ayn Rand_ understood that facing reality is virtuous, why didn't the 2020s "rationalists"? I didn't think the disdain for "Earth people" was justified, when Earth's philosophy of rationality (as exemplified by Ayn Rand) was doing better than dath ilan's on this critical dimension.
-Even if you specified by authorial fiat that "latent sadists could use the information to decide whether or not to try to become rich and famous" didn't tip the utility calculus in itself, [facts are connected to each other](https://www.lesswrong.com/posts/wyyfFfaRar2jEdeQK/entangled-truths-contagious-lies); there were _more consequences_ to the coverup, more ways in which better-informed people could make better decisions than worse-informed people.
+I wondered if the issue had to do with what Yudkowsky had [identified as the problem of non-absolute rules](https://www.lesswrong.com/posts/xdwbX9pFEr7Pomaxv/meta-honesty-firming-up-honesty-around-its-edge-cases#5__Counterargument__The_problem_of_non_absolute_rules_), where not-literally-absolute rules like "Don't kill" or "Don't lie" have to be stated as if they were absolutes in order to register to the human motivational system with sufficient force.
-What about the costs of all the other recursive censorship you'd have to do to keep the secret? (If a biography mentioned masochism in passing along with many other traits of the subject, you'd need to either censor the paragraphs with that detail, or censor the whole book. Those are real costs, even under a soft-censorship regime where people can give special consent to access "Ill Advised" products.) Maybe latent sadists could console themselves with porn if they knew, or devote their careers to making better sex robots, just as people on Earth with non-satisfiable sexual desires manage to get by. (I _knew some things_ about this topic.) What about dath ilan's heritage optimization (read: eugenics) program? Are they going to try to breed more masochists, or fewer sadists, and who's authorized to know that? And so on.
+Technically, as a matter of decision theory, "sacred values" are crazy. It's easy to say—and feel with the passion of religious conviction—that it's always right to choose Truth and Life, and that no one could choose otherwise except wrongly, in the vile service of Falsehood and Death. But reality presents us with quantitative choices over uncertain outcomes, in which everything trades off against everything else under the [von Neumann–Morgenstern axioms](https://en.wikipedia.org/wiki/Von_Neumann%E2%80%93Morgenstern_utility_theorem); if you had to choose between the Life of millions and a small, unimportant Truth, you'd probably choose Life—but more importantly, the very fact that you might have to choose means that Truth and Life can't both be infinitely sacred to you, and must be measured on a common scale with lesser goods like mere Happiness.
-Or imagine a world where male homosexuality couldn't be safely practiced due to super-AIDS. (I know very little about BDSM.) I still think men with that underlying predisposition would be better off _having a concept_ of "homosexuality" (even if they couldn't practice it), rather than the concept itself being censored. There are also other systematic differences that go along with sexual orientation (the "feminine gays, masculine lesbians" thing); if you censor the _concept_, you're throwing away that knowledge.
+I knew that. The other people in the chatroom knew that. So to the extent that the argument amounted to me saying "Don't lie" (about, _e.g._, the existence of masochism), and them saying "Don't lie unless the badness of lying is outweighed by the goodness of increased happiness", why was I so confident that I was in the right, when they were wisely acknowledging the trade-offs under the Law, and I was sticking to my (incoherent) sacred value of Truth? Didn't they obviously have the more sophisticated side of the argument?
-(When I had brought up the super-AIDS hypothetical in the chat, Ajvermillion complained that I was trying to bait people into self-cancelling by biting the bullet on suppressing homosexuality. I agreed that the choice of example was engineered to activate people's progressive moral intuitions about gay rights—it was great for him to notice that—but I thought that colliding philosophical intuitions like that was intellectually productive; it wasn't an attempt to gather blackmail material.)
+The problem was that, in my view, the people who weren't talking about Truth as if it were a sacred value were being _wildly recklessly casual_ about harms from covering things up, as if they didn't see the [non-first-order](https://www.lesswrong.com/posts/wyyfFfaRar2jEdeQK/entangled-truths-contagious-lies) [harms](https://www.lesswrong.com/posts/XTWkjCJScy2GFAgDt/dark-side-epistemology) _at all_. I felt I had to appeal to the lessons for children about how Lying Is Bad, because if I tried to make a more sophisticated argument about it being _quantitatively_ crazy to cover up psychology facts that make people sad, I would face a brick wall of "authorial fiat declares that the probabilities and utilities are specifically fine-tuned such that ignorance is good."
A user called RationalMoron asked if I was appealing to a terminal value. Did I think people should have accurate self-models even if they didn't want to?
I thought the "you could not have generated the answer I just told you" gambit was a pretty dirty argumentative trick on Yudkowsky's part. (Given that I could, how would I be able to prove it?—this was itself a good use-case for concealing spoilers.)
-As it happened, however, I _had_ already considered the case of spoilers as a class of legitimate infohazards, and was prepared to testify that I had already thought of it, and explain why I thought hiding spoilers were relevantly morally different from the coverups I was objecting to. The previous night, 7 December 2022, I had had a phone call with Anna Salamon,[^evidence-of-independent-generation] in which I (remembered that I) had cited dath ilan's [practice of letting children figure out heliocentrism for themselves](https://www.glowfic.com/replies/1777588#reply-1777588) as not being objectionable in the way the sadism/masochism coverup was.
+As it happened, however, I _had_ already considered the case of spoilers as a class of legitimate infohazards, and was prepared to testify that I had already thought of it, and explain why I thought hiding spoilers were relevantly morally different from the coverups I was objecting to. The previous night, 7 December 2022, I had had a phone call with Anna Salamon,[^evidence-of-independent-generation] in which I had cited dath ilan's [practice of letting children figure out heliocentrism for themselves](https://www.glowfic.com/replies/1777588#reply-1777588) as not being objectionable in the way the sadism/masochism coverup was.
[^evidence-of-independent-generation]: I was lucky to be able to point to Anna as a potential witness to defend myself against the "could not have generated" trick—as a matter of principle, not because I seriously expected anyone to care enough to go ask Anna if she remembered the conversation the same way.
I also mentioned that when I had used spoiler blocks on the _Atlas Shrugged_ quote I had posted upthread, I had briefly considered making some kind of side-remark noting that the spoiler blocks were also a form of information-hiding, but couldn't think of anything funny or relevant enough (which, if my self-report could be trusted, showed that I had independently generated the idea of spoilers being an example of hiding information—but I didn't expect other people to uncritically believe my self-reports).
-It seemed like the rationale for avoiding spoilers of movie plots or homework exercises had to do with the outcome being different if you got spoiled: you have a different æsthetic experience if you experience the plot twist in the 90th minute of the movie rather than the fourth paragraph of the _Wikipedia_ article. Dath ilan's sadism/masochism coverup didn't seem to have the same structure: when I try to prove a theorem myself before looking at how the textbook says to do it, it's not because I would be _sad about the state of the world_ if I looked at the textbook; it's because the temporary ignorance of working it out myself results in a stronger state of final knowledge.
-
-That is, the difference between "spoiler protections" (sometimes useful) and "coverups" (bad) had to do with whether the ignorant person is expected to eventually uncover the hidden information, and whether the ignorant person knows that there's hidden information that they're expected to uncover. In the case of the sadism/masochism coverup (in contrast to the cases of movie spoilers or homework exercises), it seemed like neither of these conditions pertained. (Keltham knows that the Keepers are keeping secrets, but he seems to actively have beliefs about human psychology that imply masochism is implausible; it seems more like he has a false map, rather than a blank spot on his map for the answer to the homework exercise to be filled in.) I thought that was morally relevant.
-
-(Additionally, I would have hoped that my two previous mentions in the thread of supporting keeping nuclear, bioweapon, and AI secrets should have already made it clear that I wasn't against _all_ cases of Society hiding information, but to further demonstrate my ability to generate counterexamples, I mentioned that I would also admit _threats_ as a class of legitimate infohazard: if I'm not a perfect decision theorist, I'm better off if Tony Soprano just doesn't have my email address to begin with, if I don't trust myself to calculate when I "should" ignore his demands.)
-
As for the claim that my thinking was distorted and I was arguing instead of seeing, it was definitely true that I was _motivated to look for_ criticisms of Yudkowsky and dath ilan, for personal reasons outside the scope of the server, and I thought it was great for people to notice this and take it into account. I hoped to nevertheless be competent to only report real criticisms and not fake criticisms. (Whether I succeeded, of course, was up to the reader to decide.)
Yudkowsky replied:
> only half the battle even if you could do it. you're also not reporting any facts/arguments on the other side, which is a much larger and visible gap to me, and has a lot to do with why I'm not presently considering this criticism from a peer despite your spoken adherence to virtues I value. **QUESTION FOR ZACK ONLY, NOBODY ELSE ANSWER OR SAY ANYTHING ABOUT IT IN THIS MAIN CHANNEL:** What are some of the ways that Planecrash valorizes truth, as you, yourself, see that virtue?
-I didn't ask why it was relevant whether or not I was a "peer." If we're measuring IQ (143 _vs._ [131](/images/wisc-iii_result.jpg)), or fiction-writing ability (several [highly-acclaimed](https://www.lesswrong.com/posts/HawFh7RvDM4RyoJ2d/three-worlds-collide-0-8) [stories](https://www.yudkowsky.net/other/fiction/the-sword-of-good) [including the world's most popular _Harry Potter_ fanfiction](https://www.hpmor.com/) _vs._ a [few](/2018/Jan/blame-me-for-trying/) [blog](http://zackmdavis.net/blog/2016/05/living-well-is-the-best-revenge/) [vignettes](https://www.lesswrong.com/posts/dYspinGtiba5oDCcv/feature-selection) and a [_My Life as a Teenage Robot_ fanfiction](https://archive.ph/WdydM) with double-digit Favorites on _fanfiction.net_), or contributions to AI alignment (founder of the field _vs._ author of some dubiously relevant blog comments), I'm obviously _not_ his peer. It didn't seem like that was necessary when one could just [evaluate my arguments about dath ilan on their own merits](https://www.lesswrong.com/posts/5yFRd3cjLpm3Nd6Di/argument-screens-off-authority). But I wasn't going to be so impertinent to point that out when the master was testing me (!) and I was eager to pass the test.
+I didn't ask why it was relevant whether or not I was a "peer." If we're measuring IQ[^iq-comparison] or fiction-writing ability[^fiction-comparison] or contributions to AI alignment[^alignment-comparison], I'm obviously _not_ his peer. It didn't seem like that was necessary when one could just [evaluate my arguments about dath ilan on their own merits](https://www.lesswrong.com/posts/5yFRd3cjLpm3Nd6Di/argument-screens-off-authority). But I wasn't going to be so impertinent to point that out when the master was testing me (!) and I was eager to pass the test.
+
+[^iq-comparison]: [143](https://twitter.com/ESYudkowsky/status/1076154502326013952) _vs._ [131](/images/wisc-iii_result.jpg).
+
+[^fiction-comparison]: Several [highly-acclaimed](https://www.lesswrong.com/posts/HawFh7RvDM4RyoJ2d/three-worlds-collide-0-8) [stories](https://www.yudkowsky.net/other/fiction/the-sword-of-good) [including the world's most popular _Harry Potter_ fanfiction](https://www.hpmor.com/) _vs._ a [few](/2018/Jan/blame-me-for-trying/) [blog](http://zackmdavis.net/blog/2016/05/living-well-is-the-best-revenge/) [vignettes](https://www.lesswrong.com/posts/dYspinGtiba5oDCcv/feature-selection) and a [_My Life as a Teenage Robot_ fanfiction](https://archive.ph/WdydM) with double-digit Favorites on _fanfiction.net_.
+
+[^alignment-comparison]: Founder of the field _vs._ author of some dubiously relevant blog comments.
I said that I'd like to take an hour to compose a _good_ answer. (It was 10:26 _p.m._) If I tried to type something off-the-cuff on the timescale of five minutes, it wasn't going to be of similar quality as my criticisms, because, as I had just admitted, I had _totally_ been running a biased search for criticisms—or did the fact that I had to ask that mean I had already failed the test?
projects / Ultimately_Untrue_Thought.git / blobdiff