[^posse-boundary]: Sarah Constantin and "Riley" had also been involved in reaching out to Yudkowsky, and were included in many subsequent discussions, but seemed like more marginal members of the group that was forming.
-Ben had previously worked at GiveWell and had written a lot about problems with the effective altruism (EA) movement, in particular, EA-branded institutions making [incoherent](http://benjaminrosshoffman.com/givewell-and-partial-funding/) [decisions](http://benjaminrosshoffman.com/openai-makes-humanity-less-safe/) under the influence of incentives to [distort](http://benjaminrosshoffman.com/humility-argument-honesty/) [information](http://benjaminrosshoffman.com/honesty-and-perjury/) [in order to](http://benjaminrosshoffman.com/effective-altruism-is-self-recommending/) [seek](http://benjaminrosshoffman.com/against-neglectedness/) [control](http://benjaminrosshoffman.com/against-responsibility/). Jessica had previously worked at MIRI, where she was unnerved by under-evidenced paranoia about secrecy and [short AI timelines](https://www.lesswrong.com/posts/KnQs55tjxWopCzKsk/the-ai-timelines-scam), and would later [write](https://www.lesswrong.com/posts/MnFqyPLqbiKL8nSR7/my-experience-at-and-around-miri-and-cfar-inspired-by-zoe) [about](https://www.lesswrong.com/posts/pQGFeKvjydztpgnsY/occupational-infohazards) her experiences there. To what extent were my gender and categories thing, and Ben's EA thing, and Jessica's MIRI thing, manifestations of "the same" underlying problem? Or had we all become disaffected with the mainstream "rationalists" for our own idiosyncratic reasons, and merely randomly fallen into each other's, and Michael's, orbit?
+Ben had previously worked at GiveWell and had written a lot about problems with the effective altruism (EA) movement; in particular, he argued that EA-branded institutions were making [incoherent](http://benjaminrosshoffman.com/givewell-and-partial-funding/) [decisions](http://benjaminrosshoffman.com/openai-makes-humanity-less-safe/) under the influence of incentives to [distort](http://benjaminrosshoffman.com/humility-argument-honesty/) [information](http://benjaminrosshoffman.com/honesty-and-perjury/) [in order to](http://benjaminrosshoffman.com/effective-altruism-is-self-recommending/) [seek](http://benjaminrosshoffman.com/against-neglectedness/) [power](http://benjaminrosshoffman.com/against-responsibility/).
+
+Jessica had previously worked at MIRI, where she was unnerved by what she saw as under-evidenced paranoia about information hazards and [short AI timelines](https://www.lesswrong.com/posts/KnQs55tjxWopCzKsk/the-ai-timelines-scam). (As Jack Gallagher, who was also at MIRI at the time, [put it](https://www.greaterwrong.com/posts/pQGFeKvjydztpgnsY/occupational-infohazards/comment/TcsXh44pB9xRziGgt), "A bunch of people we respected and worked with had decided the world was going to end, very soon, uncomfortably soon, and they were making it extremely difficult for us to check their work.")
+
+To what extent were my gender and categories thing, and Ben's EA thing, and Jessica's MIRI thing, manifestations of "the same" underlying problem? Or had we all become disaffected with the mainstream "rationalists" for our own idiosyncratic reasons, and merely randomly fallen into each other's, and Michael's, orbit?
If there was a real problem, I didn't feel like I had a good grasp on what it was specifically. Cultural critique is a fraught endeavor: if someone tells an outright lie, you can, maybe, with a lot of effort, prove that to other people, and get a correction on that specific point. (Actually, as we had just discovered, even that might be too much to hope for.) But culture is the sum of lots and lots of little micro-actions by lots and lots of people. If your entire culture has visibly departed from the Way that was taught to you in the late 'aughts, how do you demonstrate that to people who, to all appearances, are acting like they don't remember the old Way, or that they don't think anything has changed, or that they notice some changes but think the new way is better? It's not as simple as shouting, "Hey guys, Truth matters!" Any ideologue or religious person would agree with _that_. It's not feasible to litigate every petty epistemic crime in something someone said, and if you tried, someone who thought the culture was basically on track could accuse you of cherry-picking. If "culture" is a real thing at all—and it certainly seems to be—we are condemned to grasp it unclearly, relying on the brain's pattern-matching faculties to sum over thousands of little micro-actions as a [_gestalt_](https://en.wiktionary.org/wiki/gestalt), rather than having the kind of robust, precise representation a well-designed AI could compute plans with.
-Ben called the _gestalt_ he saw the Blight, after the rogue superintelligence in Vernor Vinge's _A Fire Upon the Deep_: the problem wasn't that people were getting dumber; it was that there was locally coherent coordination away from clarity and truth and towards coalition-building, which was validated by the official narrative in ways that gave it a huge tactical advantage; people were increasingly making decisions that were better explained by their political incentives rather than acting on coherent beliefs about the world—using and construing claims about facts as moves in a power game, albeit sometimes subject to genre constraints under which only true facts were admissible moves in the game.
+Ben called the _gestalt_ he saw the Blight, after the rogue superintelligence in Vernor Vinge's _A Fire Upon the Deep_. The problem wasn't that people were getting dumber; it was that they increasingly behaving in a way that was better explained by their political incentives rather than as decisions based on coherent beliefs about the world—using and construing claims about facts as moves in a power game, albeit sometimes subject to genre constraints under which only true facts were admissible moves in the game.
When I asked him for specific examples of MIRI or CfAR leaders behaving badly, he gave the example of [MIRI executive director Nate Soares posting that he was "excited to see OpenAI joining the space"](https://intelligence.org/2015/12/11/openai-and-other-news/), despite the fact that [_no one_ who had been following the AI risk discourse](https://slatestarcodex.com/2015/12/17/should-ai-be-open/) [thought that OpenAI as originally announced was a good idea](http://benjaminrosshoffman.com/openai-makes-humanity-less-safe/). Nate [had privately clarified that](https://twitter.com/jessi_cata/status/1462454555925434375) the word "excited" wasn't necessarily meant positively, and in this case meant something more like "terrified."
I may have subconsciously pulled off an interesting political maneuver. In my final email to Yudkowsky on 20 April 2019 (Subject: "closing thoughts from me"), I had written—
-> If we can't even get a public consensus from our _de facto_ leadership on something _so basic_ as "concepts need to carve reality at the joints in order to make probabilistic predictions about reality", then, in my view, there's _no point in pretending to have a rationalist community_, and I need to leave and go find something else to do (perhaps whatever Michael's newest scheme turns out to be). I don't think I'm setting [my price for joining](https://www.lesswrong.com/posts/Q8evewZW5SeidLdbA/your-price-for-joining) particularly high here?
+> If we can't even get a public consensus from our _de facto_ leadership on something _so basic_ as "concepts need to carve reality at the joints in order to make probabilistic predictions about reality", then, in my view, there's _no point in pretending to have a rationalist community_, and I need to leave and go find something else to do (perhaps whatever Michael's newest scheme turns out to be). I don't think I'm setting [my price for joining](https://www.lesswrong.com/posts/Q8evewZW5SeidLdbA/your-price-for-joining) particularly high here?[^my-price-for-joining]
+
+[^my-price-for-joining]: The Sequences post referenced here, ["Your Price for Joining"](https://www.lesswrong.com/posts/Q8evewZW5SeidLdbA/your-price-for-joining), argues that the sort of people who become "rationalists" are too prone to "take their ball and go home" rather than tolerating imperfections in a collective endeavor. To combat this, Yudkowsky proposes a norm:
+
+ > If the issue isn't worth your personally fixing by however much effort it takes, and it doesn't arise from outright bad faith, it's not worth refusing to contribute your efforts to a cause you deem worthwhile.
+
+ I claim that I was meeting this standard: I _was_ willing to personally fix the philosophy-of-categorization issue no matter how long it took, and the issue _did_ arise from outright bad faith.
-And as it happened, on 4 May 2019, Yudkowsky [re-Tweeted Colin Wright on the "univariate fallacy"](https://twitter.com/ESYudkowsky/status/1124751630937681922)—the point that group differences aren't a matter of any single variable—which was _sort of_ like the clarification I had been asking for. (Empirically, it made me feelless aggrieved.) Was I wrong to interpet this as [another "concession" to me](/2023/Jul/a-hill-of-validity-in-defense-of-meaning/#proton-concession)? (Again, notwithstanding that the whole mindset of extracting "concessions" was corrupt and not what our posse was trying to do.)
+And as it happened, on 4 May 2019, Yudkowsky [re-Tweeted Colin Wright on the "univariate fallacy"](https://twitter.com/ESYudkowsky/status/1124751630937681922)—the point that group differences aren't a matter of any single variable—which was _sort of_ like the clarification I had been asking for. (Empirically, it made me feel less aggrieved.) Was I wrong to interpret this as [another "concession" to me](/2023/Jul/a-hill-of-validity-in-defense-of-meaning/#proton-concession)? (Again, notwithstanding that the whole mindset of extracting "concessions" was corrupt and not what our posse was trying to do.)
-Separately, on 30 April 2019, I visited the house where "Meredith" and her husband Mike and Kelsey Piper and some other people lived, which I'll call "Arcadia".[^named-houses] I said, essentially, [Oh man oh jeez](https://www.youtube.com/watch?v=q_eMvgNrQQE), Ben and Michael want me to join in a rationalist civil war against the corrupt mainstream-rationality establishment, and I'd really rather not, and I don't like how they keep using scary hyperbolic words like 'cult' and 'war' and 'criminal', but on the other hand, they're _the only ones backing me up_ on this _incredibly basic philosophy thing_ and I don't feel like I have anywhere else to _go_. This culminated in a group conversation with the entire house, which I found disturbing; everyone else seemed to agree on things that I thought were clearly contrary to the spirit of the Sequences.
+Separately, one evening in April, I visited the house where "Meredith" and her husband Mike and Kelsey Piper and some other people lived, which I'll call "Arcadia".[^named-houses] I said, essentially, [Oh man oh jeez](https://www.youtube.com/watch?v=q_eMvgNrQQE), Ben and Michael want me to join in a rationalist civil war against the corrupt mainstream-rationality establishment, and I'd really rather not, and I don't like how they keep using scary hyperbolic words like 'cult' and 'war' and 'criminal', but on the other hand, they're _the only ones backing me up_ on this _incredibly basic philosophy thing_ and I don't feel like I have anywhere else to _go_. This culminated in a group conversation with the entire house, which I found unsettling. (Unfortunately, I didn't take notes and don't remember the details except that I had a sense of everyone else seeming to agree on things that I thought were clearly contrary to the spirit of the Sequences.)
[^named-houses]: It was common practice in our subculture to name group houses. My apartment was "We'll Name It Later."
Anna said she didn't want to receive [cheerful price](https://www.lesswrong.com/posts/MzKKi7niyEqkBPnyu/your-cheerful-price) offers from me anymore; previously, she had regarded my custom of recklessly throwing money at people to get what I wanted as good-faith libertarianism between consenting adults, but now she was afraid that if she accepted, it would be portrayed in some future Ben Hoffman essay as an instance of her _using_ me. She agreed that someone could have gotten the ideals I had gotten out of those posts, but there was also evidence from that time pointing the other way (_e.g._, ["Politics Is the Mind-Killer"](https://www.lesswrong.com/posts/9weLK2AJ9JEt2Tt8f/politics-is-the-mind-killer)), that it shouldn't be surprising if people steered clear of controversy.
-I replied: but when forming the original let's-be-apolitical vision in 2008, we did not anticipate that whether or not I should cut my dick off would _become_ a political issue. That was new evidence about whether the original vision was wise! I wasn't particularly trying to do politics with my idiosyncratic special interest; I was trying to think seriously about the most important thing in my life and only do the minimum amount of politics necessary to protect my ability to think. If 2019-era "rationalists" were going to commit a trivial epistemology mistake that interfered with my ability to think seriously about the most important thing in my life, but couldn't correct the mistake even after it was pointed out, then the "rationalists" were _worse than useless_ to me. This probably didn't matter causally (I wasn't an AI researcher, therefore I didn't matter), but it might matter timelessly (if I was part of a reference class that included AI researchers).
+I replied: but when forming the original let's-be-apolitical vision in 2008, we did not anticipate that whether or not I should cut my dick off would _become_ a political issue. That was new evidence about whether the original vision was wise! I wasn't particularly trying to do politics with my idiosyncratic special interest; I was trying to think seriously about the most important thing in my life and only do the minimum amount of politics necessary to protect my ability to think. If 2019-era "rationalists" were going to commit a trivial epistemology mistake that interfered with my ability to think seriously about the most important thing in my life, but couldn't correct the mistake even after it was pointed out, then the "rationalists" were _worse than useless_ to me. This probably didn't matter causally (I wasn't an AI researcher, therefore I didn't matter), but it might matter timelessly (if I was part of [a reference class that included AI researchers](/2017/Jan/from-what-ive-tasted-of-desire/)).
-Fundamentally, I was skeptical that you _could_ do consisently high-grade reasoning as a group without committing heresy, because of the mechanism that Yudkowsky had described in ["Entangled Truths, Contagious Lies"](https://www.lesswrong.com/posts/wyyfFfaRar2jEdeQK/entangled-truths-contagious-lies) and ["Dark Side Epistemology"](https://www.lesswrong.com/posts/XTWkjCJScy2GFAgDt/dark-side-epistemology): the need to lie about lying and cover up cover-ups propagates recursively. Anna in particular was unusually skillful at thinking things without saying them; I thought most people facing similar speech restrictions just get worse at thinking (plausibly[^plausibly] including Yudkowsky), and the problem gets worse as the group effort scales. (It's easier to recommend ["What You Can't Say"](http://www.paulgraham.com/say.html) to your housemates than to put it on a canonical reading list, for obvious reasons.) You can't optimize your group's culture for not-talking-about-atheism without also optimizing against understanding [Occam's razor](https://www.lesswrong.com/posts/f4txACqDWithRi7hs/occam-s-razor); you can't optimize for not questioning gender self-identity without also optimizing against understanding the [37 ways that words can be wrong](https://www.lesswrong.com/posts/FaJaCgqBKphrDzDSj/37-ways-that-words-can-be-wrong).
+Fundamentally, I was skeptical that you _could_ do consistently high-grade reasoning as a group without committing heresy, because of the mechanism that Yudkowsky had described in ["Entangled Truths, Contagious Lies"](https://www.lesswrong.com/posts/wyyfFfaRar2jEdeQK/entangled-truths-contagious-lies) and ["Dark Side Epistemology"](https://www.lesswrong.com/posts/XTWkjCJScy2GFAgDt/dark-side-epistemology): the need to lie about lying and cover up cover-ups propagates recursively. Anna in particular was unusually skillful at thinking things without saying them; I thought most people facing similar speech restrictions just get worse at thinking (plausibly[^plausibly] including Yudkowsky), and the problem gets worse as the group effort scales. (It's less risky to recommend ["What You Can't Say"](http://www.paulgraham.com/say.html) to your housemates than to put it on your 501(c)(3) organization's canonical reading list.) You can't optimize your group's culture for not-talking-about-atheism without also optimizing against understanding [Occam's razor](https://www.lesswrong.com/posts/f4txACqDWithRi7hs/occam-s-razor); you can't optimize for not questioning gender self-identity without also optimizing against understanding the [37 ways that words can be wrong](https://www.lesswrong.com/posts/FaJaCgqBKphrDzDSj/37-ways-that-words-can-be-wrong).
[^plausibly]: I was still deep enough in my hero-worship that I wrote "plausibly". Today, I would not consider the adverb necessary.
MIRI researcher Scott Garrabrant wrote a post about how ["Yes Requires the Possibility of No"](https://www.lesswrong.com/posts/G5TwJ9BGxcgh5DsmQ/yes-requires-the-possibility-of-no). Information-theoretically, a signal sent with probability one transmits no information: you can only learn something from hearing a "Yes" if believed that the answer could have been "No". I saw an analogy to my philosophy-of-language thesis, and mentioned it in a comment: if you want to believe that _x_ belongs to category _C_, you might try redefining _C_ in order to make the question "Is _x_ a _C_?" come out "Yes", but you can only do so at the expense of making _C_ less useful. Meaningful category-membership (Yes) requires the possibility of non-membership (No).
-MIRI research associate Vanessa Kosoy [objected that](https://www.lesswrong.com/posts/WwTPSkNwC89g3Afnd/comment-section-from-05-19-2019?commentId=FxSZwECjhgYE7p2du) she found it "unpleasant that [I] always bring [my] hobbyhorse in, but in an 'abstract' way that doesn't allow discussing the actual object level question"; it made her feel "attacked in a way that allow[ed] for no legal recourse to defend [herself]." I [replied](https://www.lesswrong.com/posts/WwTPSkNwC89g3Afnd/comment-section-from-05-19-2019?commentId=32GPaijsSwX2NSFJi) that that was understandable, but that I hoped it was also understandable that I found it unpleasant that our standard Bayesian philosophy of language somehow got politicized, such that my attempts to do correct epistemology were perceived as attacking people.
+Someone [objected that](https://www.lesswrong.com/posts/WwTPSkNwC89g3Afnd/comment-section-from-05-19-2019?commentId=FxSZwECjhgYE7p2du) she found it "unpleasant that [I] always bring [my] hobbyhorse in, but in an 'abstract' way that doesn't allow discussing the actual object level question"; it made her feel "attacked in a way that allow[ed] for no legal recourse to defend [herself]." (I thought I remembered meeting a man with the same last name at the 2016 Summer Solstice event in Berkeley; maybe it was her brother.) I [replied](https://www.lesswrong.com/posts/WwTPSkNwC89g3Afnd/comment-section-from-05-19-2019?commentId=32GPaijsSwX2NSFJi) that that was understandable, but that I hoped it was also understandable that I found it unpleasant that our standard Bayesian philosophy of language somehow got politicized, such that my attempts to do correct epistemology were perceived as attacking people.
The ensuring trainwreck got so bad that the mods manually [moved the comments to their own post](https://www.lesswrong.com/posts/WwTPSkNwC89g3Afnd/comment-section-from-05-19-2019). Based on the karma scores and what was said,[^yes-requires-slapfight-highlights] I count it as a "victory" for me.
[^yes-requires-slapfight-highlights]: I particularly appreciated Said Achmiz's [defense of disregarding community members' feelings](https://www.lesswrong.com/posts/WwTPSkNwC89g3Afnd/comment-section-from-05-19-2019?commentId=EsSdLMrFcCpSvr3pG), and [Ben's commentary on speech acts that lower the message length of proposals to attack some group](https://www.lesswrong.com/posts/WwTPSkNwC89g3Afnd/comment-section-from-05-19-2019?commentId=TXbgr7goFtSAZEvZb).
-On 31 May 2019, a [draft of a new _Less Wrong_ FAQ](https://www.lesswrong.com/posts/MqrzczdGhQCRePgqN/feedback-requested-draft-of-a-new-about-welcome-page-for) included a link to "... Not Man for the Categories" as one of Scott Alexander's best essays. I argued that it would be better to cite almost literally any other _Slate Star Codex_ post (most of which, I agreed, were exemplary). I claimed that the following disjunction was true: either Alexander's claim that "There's no rule of rationality saying that [one] shouldn't" "accept an unexpected [X] or two deep inside the conceptual boundaries of what would normally be considered [Y] if it'll save someone's life" was a blatant lie, or one had no grounds to criticize me for calling it a blatant lie, because there's no rule of rationality that says I shouldn't draw the category boundaries of "blatant lie" that way. Ruby Bloom, the new moderator who wrote the draft, [was persuaded](https://www.lesswrong.com/posts/MqrzczdGhQCRePgqN/feedback-requested-draft-of-a-new-about-welcome-page-for?commentId=oBDjhXgY5XtugvtLT), and "... Not Man for the Categories" was not included in the final FAQ. Another "victory."
+On 31 May 2019, a [draft of a new _Less Wrong_ FAQ](https://www.lesswrong.com/posts/MqrzczdGhQCRePgqN/feedback-requested-draft-of-a-new-about-welcome-page-for) included a link to ["The Categories Were Made for Man, Not Man for the Categories"](https://slatestarcodex.com/2014/11/21/the-categories-were-made-for-man-not-man-for-the-categories/) as one of Scott Alexander's best essays. I argued that it would be better to cite almost literally any other _Slate Star Codex_ post (most of which, I agreed, were exemplary). I claimed that the following disjunction was true: either Alexander's claim that "There's no rule of rationality saying that [one] shouldn't" "accept an unexpected [X] or two deep inside the conceptual boundaries of what would normally be considered [Y] if it'll save someone's life" was a blatant lie, or one had no grounds to criticize me for calling it a blatant lie, because there's no rule of rationality that says I shouldn't draw the category boundaries of "blatant lie" that way. Ruby Bloom, the new moderator who wrote the draft, [was persuaded](https://www.lesswrong.com/posts/MqrzczdGhQCRePgqN/feedback-requested-draft-of-a-new-about-welcome-page-for?commentId=oBDjhXgY5XtugvtLT), and "... Not Man for the Categories" was not included in the final FAQ. Another "victory."
But winning "victories" wasn't particularly comforting when I resented this becoming a political slapfight at all. I wrote to Anna and Steven Kaas (another old-timer who I was trying to "recruit" onto my side of the civil war). In ["What You Can't Say"](http://www.paulgraham.com/say.html), Paul Graham had written, "The problem is, there are so many things you can't say. If you said them all you'd have no time left for your real work." But surely that depends on what one's real work was. For someone like Paul Graham, whose goal was to make a lot of money writing software, "Don't say it" (except for this one meta-level essay) was probably the right choice. But someone whose goal is to improve our collective ability to reason, should probably be doing more fighting than Paul Graham (although still preferably on the meta- rather than object-level), because political restrictions on speech and thought directly hurt the mission of "improving our collective ability to reason" in a way that they don't hurt the mission of "make a lot of money writing software."
-I said, I didn't know if either of them had caught the "Yes Requires the Possibility" trainwreck, but wasn't it terrifying that the person who objected to my innocuous philosophy comment was a goddamned _MIRI research associate_? Not to demonize Kosoy, because [I was just as bad (if not worse) in 2008](/2021/May/sexual-dimorphism-in-the-sequences-in-relation-to-my-gender-problems/#hair-trigger-antisexism). The difference was that in 2008, we had a culture that could beat it out of me.
+I said, I didn't know if either of them had caught the "Yes Requires the Possibility" trainwreck, but wasn't it terrifying that the person who objected to my innocuous philosophy comment was a goddamned _MIRI research associate_? Not to demonize that commenter, because [I was just as bad (if not worse) in 2008](/2021/May/sexual-dimorphism-in-the-sequences-in-relation-to-my-gender-problems/#hair-trigger-antisexism). The difference was that in 2008, we had a culture that could beat it out of me.
-Steven objected that tractibility and side effects matter, not just effect on the mission considered in isolation. For example, the Earth's graviational field directly impedes NASA's mession, and doesn't hurt Paul Graham, but both NASA and Paul Graham should spend the same amount of effort trying to reduce the Earth's gravity (_viz._, zero).
+Steven objected that tractability and side effects matter, not just effect on the mission considered in isolation. For example, the Earth's gravitational field directly impedes NASA's mission, and doesn't hurt Paul Graham, but both NASA and Paul Graham should spend the same amount of effort trying to reduce the Earth's gravity (_viz._, zero).
I agreed that tractability needs to be addressed, but the situation felt analogous to being in a coal mine in which my favorite one of our canaries had just died. Caliphate officials (Yudkowsky, Alexander, Anna) and loyalists (Steven) were patronizingly consoling me: sorry, I know you were really attached to that canary, but it's just a bird; it's not critical to the coal-mining mission. I agreed that I was unreasonably attached to that particular bird, which was the direct cause of why I-in-particular was freaking out, but that's not why I expected _them_ to care. The problem was what the dead canary was evidence of: if you're doing systematically correct reasoning, you should be able to get the right answer even when the question _doesn't matter_. (The causal graph is the fork "canary-death ← mine-gas → human-danger" rather than the direct link "canary-death → human-danger".) Ben and Michael and Jessica claimed to have spotted their own dead canaries. I felt like the old-timer Rationality Elders should have been able to get on the same page about the canary-count issue?
In June 2019, I made [a linkpost on _Less Wrong_](https://www.lesswrong.com/posts/5nH5Qtax9ae8CQjZ9/tal-yarkoni-no-it-s-not-the-incentives-it-s-you) to Tal Yarkoni's ["No, It's Not The Incentives—It's you"](https://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-incentives-its-you/), about how professional scientists should stop using career incentives as an excuse for doing poor science. It generated a lot of discussion.
-In an email (Subject: "LessWrong.com is dead to me"), Jessica identified _Less Wrong_ moderator [Ray Arnold's comments](https://www.greaterwrong.com/posts/5nH5Qtax9ae8CQjZ9/no-it-s-not-the-incentives-it-s-you/comment/vPj9E9iqXjnNdyhob) as her last straw:
+In an email (Subject: "LessWrong.com is dead to me"), Jessica identified _Less Wrong_ moderator [Ray Arnold's comments](https://www.greaterwrong.com/posts/5nH5Qtax9ae8CQjZ9/no-it-s-not-the-incentives-it-s-you/comment/vPj9E9iqXjnNdyhob) as her last straw. Jessica wrote:
> LessWrong.com is a place where, if the value of truth conflicts with the value of protecting elites' feelings and covering their asses, the second value will win.
>
Jessica ended up writing a post, ["Self-Consciousness Wants Everything to Be About Itself"](https://www.lesswrong.com/posts/bwkZD6uskCQBJDCeC/self-consciousness-wants-to-make-everything-about-itself), arguing that tone arguments are mainly about people silencing discussion of actual problems in order to protect their feelings, using as a central example a case study of a college official crying and saying that she "felt attacked" in response to complaints about her office being insufficiently supportive of a racial community.
-Jessica was surprised by how well it worked, judging by [Ruby mentioning silencing in a subsequent apology to me](https://www.lesswrong.com/posts/xqAnKW46FqzPLnGmH/causal-reality-vs-social-reality?commentId=wfzxj4GGRtZGMG9ni) (plausibly influenced by Jessica's post), and [an exchange between Raemon (also a mod) and Ruby that she thought was "surprisingly okay"](https://www.lesswrong.com/posts/bwkZD6uskCQBJDCeC/self-consciousness-wants-to-make-everything-about-itself?commentId=EW3Mom9qfoggfBicf).
+Jessica was surprised by how well it worked, judging by [Ruby mentioning silencing in a subsequent apology to me](https://www.lesswrong.com/posts/xqAnKW46FqzPLnGmH/causal-reality-vs-social-reality?commentId=wfzxj4GGRtZGMG9ni) (plausibly influenced by Jessica's post), and [an exchange between Ray Arnold (also a mod) and Ruby that she thought was "surprisingly okay"](https://www.lesswrong.com/posts/bwkZD6uskCQBJDCeC/self-consciousness-wants-to-make-everything-about-itself?commentId=EW3Mom9qfoggfBicf).
From this, Jessica derived the moral that when people are doing something that seems obviously terrible and in bad faith, it could help to publicly explain why the abstract thing is bad, without accusing anyone. This made sense because people didn't want to be held to standards that other people aren't being held to: a call-out directed at oneself personally could be selective enforcement, but a call-out of the abstract pattern invited changing one's behavior if the new equilibrium looked better.
Ben replied, "What exactly is a scam, if it's not misinforming people systematically about what you have to offer, in a direction that moves resources towards you?" Investigations of financial fraud focused on promises about money being places being false because the money was not in fact in those places, rather than the psychological minutiæ of the perp's exact motives.
-I replied that the concept of [_mens rea_](https://www.law.cornell.edu/wex/mens_rea) did seem necessary for maintaining good incentives, at least in some contexts. The law needs to distinguish between accidentally hitting a pedestrian in one's car ("manslaughter") from premeditated killing ("first-degree murder"), because traffic accidents are significantly less disincentivizable than offing one's enemies. (Anyone who drives at all is probably going to have unlucky analogues in nearby possible worlds who are guilty of vehicular manslaughter.) The manslaughter case was simpler than misinformation-that-moves-resources,[^manslaughter-disanalogy] and it might not be _easy_ for the court to determine "intent", but I didn't see what would reverse the weak principle that intent matters, sometimes.
+I replied that the concept of [_mens rea_](https://www.law.cornell.edu/wex/mens_rea) did seem necessary for maintaining good incentives, at least in some contexts. The law needs to distinguish between accidentally hitting a pedestrian in one's car ("manslaughter") from premeditated killing ("first-degree murder"), because traffic accidents are significantly less disincentivizable than offing one's enemies. (Anyone who drives at all is taking on some nonzero risk of committing vehicular manslaughter.) The manslaughter case was simpler than misinformation-that-moves-resources,[^manslaughter-disanalogy] and it might not be _easy_ for the court to determine "intent", but I didn't see what would reverse the weak principle that intent matters, sometimes.
[^manslaughter-disanalogy]: For one extremely important disanalogy, perps don't _gain_ from committing manslaughter.
> What I see as under threat is the ability to say in a way that's actually heard, not only that opinion X is false, but that the process generating opinion X is untrustworthy, and perhaps actively optimizing in an objectionable direction. Frequently, attempts to say this are construed _primarily_ as moves to attack some person or institution, pushing them into the outgroup. Frequently, people suggest to me an "equivalent" wording with a softer tone, which in fact omits important substantive criticisms I mean to make, while claiming to understand what's at issue.
-Ray Arnold (another _Less Wrong_ mod) replied:
+Ray Arnold replied:
> My core claim is: "right now, this isn't possible, without a) it being heard by many people as an attack, b) without people having to worry that other people will see it as an attack, even if they don't."
>
An in-person meeting was arranged on 23 July 2019 at the _Less Wrong_ office, with Ben, Jessica, me, and most of the _Less Wrong_ team (Ray, Ruby, Oliver Habryka, Vaniver, Jim Babcock). I don't have notes and don't really remember what was discussed in enough detail to faithfully recount it.[^memory] I ended up crying at one point and left the room for a while.
-[^memory]: An advantage of mostly living on the internet is that I have _logs_ of the important things; I'm only able to tell this Whole Dumb Story with as much fidelity as I am, because for most of it, I can go back and read the emails and chatlogs from the time. Now that [audio transcription has fallen to AI](https://openai.com/blog/whisper/), should I be recording more real-life conversations?? In the case of this meeting, supposedly one of the _Less Wrong_ guys was recording, but no one had it when I asked in October 2022.
+[^memory]: An advantage of mostly living on the internet is that I have logs of the important things. I'm only able to tell this Whole Dumb Story with as much fidelity as I am, because for most of it, I can go back and read the emails and chatlogs from the time. Now that [audio transcription has fallen to AI](https://openai.com/blog/whisper/), maybe I should be recording more real-life conversations? In the case of this meeting, supposedly one of the _Less Wrong_ guys was recording, but no one had it when I asked in October 2022.
The next day, I asked Ben and Jessica for their takeaways via email (Subject: "peace talks outcome?"). Jessica said that I was a "helpful emotionally expressive and articulate victim", that there seemed to be a consensus that it would be better if people like me could be warned somehow that _Less Wrong_ wasn't doing the general sanity-maximization thing anymore. (Because community leaders were willing to sacrifice, for example, ability to discuss non-AI heresies, in order to focus on sanity about AI in particular while maintaining enough mainstream acceptability and power.)
Empirically, not right! But I had to resolve the contradiction between narrative and reality somehow, and if my choices were "People are [mistakenly](https://slatestarcodex.com/2018/01/24/conflict-vs-mistake/) failing to live up to the narrative" and "[Everybody knows](https://thezvi.wordpress.com/2019/07/02/everybody-knows/) the narrative is a lie; it would be crazy to expect people to live up to it", the former had been more appealing.
-It was the same thing here. Kelsey said that it was completely predictable that Yudkowsky wouldn't make a public statement, even one as uncontroversial as "category boundaries should be drawn for epistemic and not instrumental reasons", because his experience of public statements was that they'd be taken out of context and used against MIRI by the likes of /r/SneerClub. This wasn't an update at all. (Everyone at "Arcadia" had agreed, in the house discussion on 30 April.) Vassar's insistence that Eliezer be expected to do something that he obviously was never going to do had caused me to be confused and surprised by reality.[^statement]
+It was the same thing here. Kelsey said that it was completely predictable that Yudkowsky wouldn't make a public statement, even one as uncontroversial as "category boundaries should be drawn for epistemic and not instrumental reasons", because his experience of public statements was that they'd be taken out of context and used against MIRI by the likes of /r/SneerClub. This wasn't an update at all. (Everyone at "Arcadia" had agreed, in the house discussion in April.) Vassar's insistence that Eliezer be expected to do something that he obviously was never going to do had caused me to be confused and surprised by reality.[^statement]
-[^statement]: I thought it was odd that Kelsey seemed to think the issue was that me and my allies were pressuring Yudkowsky to make a public statement, which he never does. From our perspective, the issue was that he _had_ made a statement, and it was wrong.
+[^statement]: I thought it was odd that Kelsey seemed to think the issue was that me and my allies were pressuring Yudkowsky to make a public statement, which he supposedly never does. From our perspective, the issue was that he _had_ made a statement, and it was wrong.
Kelsey seemed to be taking it as obvious that Eliezer Yudkowsky's public behavior was optimized to respond to the possibility of political attacks by people who hate him anyway, and not optimized to respond to the actuality of thousands of words of careful arguments appealing to his own writings from ten years ago. Very well. Maybe it _was_ obvious. But that being the case, I had no reason to care what Eliezer Yudkowsky said, because not-provoking-SneerClub isn't truth-tracking, and careful arguments are. This was a huge surprise _to me_, even if Kelsey knew better.
When I mentioned this to Michael on Signal on 3 August 2019, he replied:
-> The person is me, the whole process is a hit piece, literally, the investigation process and not the content. Happy to share the latter with you. You can talk with Ben about appropiate ethical standards.
+> The person is me, the whole process is a hit piece, literally, the investigation process and not the content. Happy to share the latter with you. You can talk with Ben about appropriate ethical standards.
-In retrospect, I feel dumb for not guessing that Michael's mysterious ally was Michael himself. I count this kind of situation as another reason to be [annoyed at how norms protecting confidentiality](/2023/Jul/a-hill-of-validity-in-defense-of-meaning/#privacy-constraints) distort information; Kelsey apparently felt obligated to obfuscate any names connected to potential litigation, which led me to the infer the existence of a nonexistent person (because I naïvely assumed that if Michael had been the person who threatened to sue, Kelsey would have said that). I can't say I never introduce this kind of disortion myself (for I, too, am bound by norms), but when I do, I feel dirty about it.
+In retrospect, I feel dumb for not guessing that Michael's mysterious ally was Michael himself. I count this kind of situation as another reason to be [annoyed at how norms protecting confidentiality](/2023/Jul/a-hill-of-validity-in-defense-of-meaning/#privacy-constraints) distort information; Kelsey apparently felt obligated to obfuscate any names connected to potential litigation, which led me to the infer the existence of a nonexistent person (because I naïvely assumed that if Michael had been the person who threatened to sue, Kelsey would have said that). I can't say I never introduce this kind of distortion myself (for I, too, am bound by norms), but when I do, I feel dirty about it.
As far as appropriate ethical standards go, I didn't approve of silencing critics with lawsuit threats, even while I agreed with Michael that "the process is the punishment." I imagine that if the REACH wanted to publish a report about me, I would expect to defend myself in public, having faith that the [beautiful weapon](https://slatestarcodex.com/2017/03/24/guided-by-the-beauty-of-our-weapons/) of my Speech would carry the day against a corrupt community center—or for that matter, against /r/SneerClub.
Jessica asked if Yudkowsky denouncing neoreaction and the alt-right would still seem harmful, if he were to also to acknowledge, _e.g._, racial IQ differences?
-I agreed that it would be helpful, but realistically, I didn't see why Yudkowsky should want to poke the race-differences hornet's nest. This was the tragedy of recursive silencing: if you can't afford to engage with heterodox ideas, you either become an [evidence-filtering clever arguer](https://www.lesswrong.com/posts/kJiPnaQPiy4p9Eqki/what-evidence-filtered-evidence), or you're not allowed to talk about anything except math. (Not even the relationship between math and human natural language, as we had found out recently.)
+<a id="tragedy-of-recursive-silencing"></a>I agreed that it would be helpful, but realistically, I didn't see why Yudkowsky should want to poke the race-differences hornet's nest. This was the tragedy of recursive silencing: if you can't afford to engage with heterodox ideas, you either become an [evidence-filtering clever arguer](https://www.lesswrong.com/posts/kJiPnaQPiy4p9Eqki/what-evidence-filtered-evidence), or you're not allowed to talk about anything except math. (Not even the relationship between math and human natural language, as we had found out recently.)
It was as if there was a "Say Everything" attractor, and a "Say Nothing" attractor, and my incentives were pushing me towards the "Say Everything" attractor—but that was only because I had [Something to Protect](/2019/Jul/the-source-of-our-power/) in the forbidden zone and I was a decent programmer (who could therefore expect to be employable somewhere, just as [James Damore eventually found another job](https://twitter.com/JamesADamore/status/1034623633174478849)). Anyone in less extreme circumstances would find themselves being pushed to the "Say Nothing" attractor.
It was instructive to compare Yudkowsky's new disavowal of neoreaction with one from 2013, in response to a _TechCrunch_ article citing former MIRI employee Michael Anissimov's neoreactionary blog _More Right_:[^linkrot]
-[^linkrot]: The original _TechCrunch_ comment would seem to have succumbed to [linkrot](https://www.gwern.net/Archiving-URLs#link-rot), but Yudkowsky's comment was quoted by [Moldbug](https://www.unqualified-reservations.org/2013/11/mr-jones-is-rather-concerned/) and [others](https://medium.com/@2045singularity/white-supremacist-futurism-81be3fa7020d)
+[^linkrot]: The original _TechCrunch_ comment would seem to have succumbed to [linkrot](https://www.gwern.net/Archiving-URLs#link-rot), but it was quoted by [Moldbug](https://www.unqualified-reservations.org/2013/11/mr-jones-is-rather-concerned/) and [others](https://medium.com/@2045singularity/white-supremacist-futurism-81be3fa7020d).
> "More Right" is not any kind of acknowledged offspring of Less Wrong nor is it so much as linked to by the Less Wrong site. We are not part of a neoreactionary conspiracy. We are and have been explicitly pro-Enlightenment, as such, under that name. Should it be the case that any neoreactionary is citing me as a supporter of their ideas, I was never asked and never gave my consent. [...]
>
[^pleonasm]: The pleonasm here ("to me" being redundant with "I thought") is especially galling coming from someone who's usually a good writer!
-It might seem like a little thing of no significance—requiring ["I" statements](https://en.wikipedia.org/wiki/I-message) is commonplace in therapy groups and corporate sensitivity training—but this little thing coming from Eliezer Yudkowsky setting guidelines for an explicitly "rationalist" space made a pattern click. If everyone is forced to only make narcissistic claims about their map ("_I_ think", "_I_ feel"), and not make claims about the territory (which could be construed to call other people's maps into question and thereby threaten them, because [disagreement is disrespect](http://www.overcomingbias.com/2008/09/disagreement-is.html)), that's great for reducing social conflict, but it's not great for the kind of collective information processing that accomplishes cognitive work,[^i-statements] like good literary criticism. A rationalist space needs to be able to talk about the territory.
+It might seem like a little thing of no significance—requiring ["I" statements](https://en.wikipedia.org/wiki/I-message) is commonplace in therapy groups and corporate sensitivity training—but this little thing coming from Eliezer Yudkowsky setting guidelines for an explicitly "rationalist" space made a pattern click. If everyone is forced to only make narcissistic claims about their map ("_I_ think", "_I_ feel"), and not make claims about the territory (which could be construed to call other people's maps into question and thereby threaten them, because [disagreement is disrespect](https://www.overcomingbias.com/p/disagreement-ishtml)), that's great for reducing social conflict, but it's not great for the kind of collective information processing that accomplishes cognitive work,[^i-statements] like good literary criticism. A rationalist space needs to be able to talk about the territory.
[^i-statements]: At best, "I" statements make sense in a context where everyone's speech is considered part of the "official record". Wrapping controversial claims in "I think" removes the need for opponents to immediately object for fear that the claim will be accepted onto the shared map.
To be fair, the same comment I quoted also lists "Being able to consider and optimize literary qualities" is one of the major considerations to be balanced. But I think (_I_ think) it's also fair to note that (as we had seen on _Less Wrong_ earlier that year), lip service is cheap. It's easy to say, "Of course I don't think politeness is more important than truth," while systematically behaving as if you did.
-"Broadcast criticism is adversely selected for critic errors," Yudkowsky wrote in the post on reducing negativity, correctly pointing out that if a work's true level of mistakenness is _M_, the _i_-th commenter's estimate of mistakenness has an error term of _E<sub>i</sub>_, and commenters leave a negative comment when their estimate _M_ + _E<sub>i</sub>_ is greater than their threshold for commenting _T<sub>i</sub>_, then the comments that get posted will have been selected for erroneous criticism (high _E<sub>i</sub>_) and commmenter chattiness (low _T<sub>i</sub>_).
+"Broadcast criticism is adversely selected for critic errors," Yudkowsky wrote in the post on reducing negativity, correctly pointing out that if a work's true level of mistakenness is _M_, the _i_-th commenter's estimate of mistakenness has an error term of _E<sub>i</sub>_, and commenters leave a negative comment when their estimate _M_ + _E<sub>i</sub>_ is greater than their threshold for commenting _T<sub>i</sub>_, then the comments that get posted will have been selected for erroneous criticism (high _E<sub>i</sub>_) and commenter chattiness (low _T<sub>i</sub>_).
I can imagine some young person who liked _Harry Potter and the Methods_ being intimidated by the math notation, and uncritically accepting this wisdom from the great Eliezer Yudkowsky as a reason to be less critical, specifically. But a somewhat less young person who isn't intimidated by math should notice that this is just [regression to the mean](https://en.wikipedia.org/wiki/Regression_toward_the_mean). The same argument applies to praise!
The "Reducing Negativity" post also warns against the failure mode of attempted "author telepathy": attributing bad motives to authors and treating those attributions as fact without accounting for uncertainty or distinguishing observations from inferences. I should be explicit, then: when I say negative things about Yudkowsky's state of mind, like it's "as if he's given up on the idea that reasoning in public is useful or possible", that's a probabilistic inference, not a certain observation.
-But I think making probabilistic inferences is ... fine? The sentence "Credibly helpful unsolicited criticism should be delivered in private" sure does look to me like text that's likely to have been generated by a state of mind that doesn't believe that reasoning in public is useful or possible.[^criticism-inference] Someone who did believe in public reason would have noticed that criticism has information content whose public benefits might outweigh its potential to harm an author's reputation or feelings. If you think I'm getting this inference wrong, feel free to let me _and other readers_ know why in the comments.
+But I think making probabilistic inferences is ... fine? The sentence "Credibly helpful unsolicited criticism should be delivered in private" sure does look to me like text that's likely to have been generated by a state of mind that doesn't believe that reasoning in public is useful or possible.[^criticism-inference] I think that someone who did believe in public reason would have noticed that criticism has information content whose public benefits might outweigh its potential to harm an author's reputation or feelings.[^unhedonic] If you think I'm getting this inference wrong, feel free to let me _and other readers_ know why in the comments.
[^criticism-inference]: More formally, I'm claiming that the [likelihood ratio](https://arbital.com/p/likelihood_ratio/) P(wrote that sentence|doesn't believe in public reason)/P(wrote that sentence|does believe in public reason) is greater than one.
+[^unhedonic]: Speaking of authors' feelings, it's notable that Yudkowsky's [_Less Wrong_ commenting guidelines](/images/yudkowsky_commenting_guidelines.png) declare, "If it looks like it would be unhedonic to spend time interacting with you, I will ban you from commenting on my posts." The text does not seem to consider the possibility that people who are unhedonic to interact with might have intellectually substantive contributions to make.
+
-----
-On 3 November 2019, I received an interesting reply on my philosophy-of-categorization thesis from MIRI researcher Abram Demski. Abram asked: ideally, shouldn't all conceptual boundaries be drawn with appeal-to-consequences? Wasn't the problem just with bad (motivated, shortsighted) appeals to consequences? Agents categorize in order to make decisions. The best classifer for an application depends on the costs and benefits. As a classic example, it's important for evolved prey animals to avoid predators, so it makes sense for their predator-detection classifiers to be configured such that they jump away from every rustling in the bushes, even if it's usually not a predator.
+On 3 November 2019, I received an interesting reply on my philosophy-of-categorization thesis from MIRI researcher Abram Demski. Abram asked: ideally, shouldn't all conceptual boundaries be drawn with appeal-to-consequences? Wasn't the problem just with bad (motivated, shortsighted) appeals to consequences? Agents categorize in order to make decisions. The best classifier for an application depends on the costs and benefits. As a classic example, it's important for evolved prey animals to avoid predators, so it makes sense for their predator-detection classifiers to be configured such that they jump away from every rustling in the bushes, even if it's usually not a predator.
I had thought of the "false-positives are better than false-negatives when detecting predators" example as being about the limitations of evolution as an AI designer: messy evolved animal brains don't bother to track probability and utility separately the way a cleanly-designed AI could. As I had explained in "... Boundaries?", it made sense for what variables you paid attention to, to be motivated by consequences. But given the subspace that's relevant to your interests, you want to run an "epistemically legitimate" clustering algorithm on the data you see there, which depends on the data, not your values. The only reason value-dependent gerrymandered category boundaries seem like a good idea if you're not careful about philosophy is because it's _wireheading_. Ideal probabilistic beliefs shouldn't depend on consequences.
------
-Also in November 2019, I wrote to Ben about how I was still stuck on writing the grief-memoir. My plan had been that it should have been possible to tell the story of the Category War while glomarizing about the content of private conversations, then offer Scott and Eliezer pre-publication right of reply (because it's only fair to give your former-hero-current-[frenemies](https://en.wikipedia.org/wiki/Frenemy) warning when you're about to publicly call them intellectually dishonest), then share it to _Less Wrong_ and the /r/TheMotte culture war thread, and then I would have the emotional closure to move on with my life (learn math, go to gym, chop wood, carry water) and not be a mentally-dominated cultist.
+Also in November 2019, I wrote to Ben about how I was still stuck on writing the grief-memoir. My plan had been that it should have been possible to tell the story of the Category War while Glomarizing about the content of private conversations, then offer Scott and Eliezer pre-publication right of reply (because it's only fair to give your former-hero-current-[frenemies](https://en.wikipedia.org/wiki/Frenemy) warning when you're about to publicly call them intellectually dishonest), then share it to _Less Wrong_ and the /r/TheMotte culture war thread, and then I would have the emotional closure to move on with my life (learn math, go to gym, chop wood, carry water) and not be a mentally-dominated cultist.
The reason it _should_ have been safe to write was because it's good to explain things. It should be possible to say, "This is not a social attack; I'm not saying 'rationalists Bad, Yudkowsky Bad'; I'm just trying to carefully tell the true story about why, as a matter of cause-and-effect, I've been upset this year, including addressing counterarguments for why some would argue that I shouldn't be upset, why other people could be said to be behaving 'reasonably' given their incentives, why I nevertheless wish they'd be braver and adhere to principle rather than 'reasonably' following incentives, _&c_."
So why couldn't I write? Was it that I didn't know how to make "This is not a social attack" credible? Maybe because ... it wasn't true?? I was afraid that telling a story about our leader being intellectually dishonest was "the nuclear option" in a way that I couldn't credibly cancel with "But I'm just telling a true story about a thing that was important to me that actually happened" disclaimers. If you're slowly-but-surely gaining territory in a conventional war, suddenly escalating to nukes seems pointlessly destructive. This metaphor was horribly non-normative ([arguing is not a punishment](https://srconstantin.github.io/2018/12/15/argue-politics-with-your-best-friends.html); carefully telling a true story _about_ an argument is not a nuke), but I didn't know how to make it stably go away.
-A more motivationally-stable compromise would be to split off whatever generalizable insights that would have been part of the story into their own posts that didn't make it personal. ["Heads I Win, Tails?—Never Heard of Her"](https://www.lesswrong.com/posts/DoPo4PDjgSySquHX8/heads-i-win-tails-never-heard-of-her-or-selective-reporting) had been a huge success as far as I was concerned, and I could do more of that kind of thing, analyzing the social stuff I was worried about without making it personal (even if, secretly, it was personal).
+A more motivationally-stable compromise would be to split off whatever generalizable insights that would have been part of the story into their own posts that didn't make it personal. ["Heads I Win, Tails?—Never Heard of Her"](https://www.lesswrong.com/posts/DoPo4PDjgSySquHX8/heads-i-win-tails-never-heard-of-her-or-selective-reporting) had been a huge success as far as I was concerned, and I could do more of that kind of thing, analyzing the social stuff I was worried about without making it personal, even if, secretly ("secretly"), it was personal.
Ben replied that it didn't seem like it was clear to me that I was a victim of systemic abuse, and that I was trying to figure out whether I was being fair to my abuser. He thought if I could internalize that, I would be able to forgive myself a lot of messiness, which would reduce the perceived complexity of the problem.
I said I would bite that bullet: yes! Yes, I was trying to figure out whether I was being fair to my abusers, and it was an important question to get right! "Other people's lack of standards harmed me, therefore I don't need to hold myself to standards in my response because I have [extenuating circumstances](https://www.lesswrong.com/posts/XYrcTJFJoYKX2DxNL/extenuating-circumstances)" would be a lame excuse.
-This seemed correlated with the recurring stalemated disagreement within our coordination group, where Michael/Ben/Jessica would say, "Fraud, if the word ever meant anything", and while I agreed that they were pointing to an important pattern of false-representations-optimized-to-move-resources, I was still sympathetic to the Caliphate-defender's reply that this usage of "fraud" was motte-and-baileying between different senses of _fraud_. (Most people would say that the things we were alleging MIRI and CfAR had done wrong were qualitatively different from the things Enron and Bernie Madoff had done wrong.) I wanted to do _more work_ to formulate a more precise theory of the psychology of deception to describe exactly how things were messed up a way that wouldn't be susceptible to the motte-and-bailey charge.
+This seemed correlated with the recurring stalemated disagreement within our coordination group, where Michael/Ben/Jessica would say, "Fraud, if the word ever meant anything", and while I agreed that they were pointing to an important pattern of false representations optimized to move resources, I was still sympathetic to the Caliphate-defender's reply that this usage of "fraud" was motte-and-baileying between different senses of _fraud_. (Most people would say that the things we were alleging MIRI and CfAR had done wrong were qualitatively different from the things Enron and Bernie Madoff had done wrong.) I wanted to do _more work_ to formulate a more precise theory of the psychology of deception to describe exactly how things were messed up a way that wouldn't be susceptible to the motte-and-bailey charge.
+
+Looking back four years later, I still feel that way—but my desire for nuance itself demands nuance.
+
+[TODO— FTX and nuance epilogue—
+ * On the one hand, I think I'm right to worry about the "jump to evaluation" failure mode, where you substitute a compressed hostile description
+ * If Gloria does a crime and lies about it and you call her a fraud, people are going to correctly notice that your description failed to match reality; you're obscuring what's actually bad about it
+ * On the other hand, I want to give the posse's worldview credit
+ * In April 2019, Ben tried to describe the Blight to me, saying, "People are systematically conflating corruption, accumulation of dominance, and theft, with getting things done"
+ * ordinary grown-up EAs would describe this as uncharitable, rude, &c.
+ * But look at the FTX blowup. This was, actually, one of the greatest financial frauds of our time, and it was made possible by EA: Lewis's book explains that early Alameda recruited from movement EAs, https://thezvi.wordpress.com/2023/10/24/book-review-going-infinite/
+ * Ordinary grown-ups say, "How could we have known?", but this is much less surprising on Ben's view of what EA is. (And doing good as a charity is even harder than running a crypto exchange, where the job you're doing for stakeholders is more legible; if a health intervention doesn't owrk)
+]
-------
-On 12 and 13 November 2019, Ziz [published](https://archive.ph/GQOeg) [several](https://archive.ph/6HsvS) [blog](https://archive.ph/jChxP) [posts](https://archive.ph/TPei9) laying out [her](/2019/Oct/self-identity-is-a-schelling-point/) greviances against MIRI and CfAR. On the fifteenth, Ziz and three collaborators staged a protest at the CfAR reunion being held at a retreat center in the North Bay near Camp Meeker. A call to the police falsely alleged that the protestors had a gun, [resulting in a](http://web.archive.org/web/20230316210946/https://www.pressdemocrat.com/article/news/deputies-working-to-identify-suspects-in-camp-meeker-incident/) [dramatic police reaction](http://web.archive.org/web/20201112041007/https://www.pressdemocrat.com/article/news/authorities-id-four-arrested-in-westminster-woods-protest/) (SWAT team called, highway closure, children's group a mile away being evacuated—the works).
+On 12 and 13 November 2019, Ziz [published](https://archive.ph/GQOeg) [several](https://archive.ph/6HsvS) [blog](https://archive.ph/jChxP) [posts](https://archive.ph/TPei9) laying out [her](/2019/Oct/self-identity-is-a-schelling-point/) grievances against MIRI and CfAR. On the fifteenth, Ziz and three collaborators staged a protest at the CfAR reunion being held at a retreat center in the North Bay near Camp Meeker. A call to the police falsely alleged that the protesters had a gun, [resulting in a](http://web.archive.org/web/20230316210946/https://www.pressdemocrat.com/article/news/deputies-working-to-identify-suspects-in-camp-meeker-incident/) [dramatic police reaction](http://web.archive.org/web/20201112041007/https://www.pressdemocrat.com/article/news/authorities-id-four-arrested-in-westminster-woods-protest/) (SWAT team called, highway closure, children's group a mile away being evacuated—the works).
I was tempted to email links to the blog posts to the Santa Rosa _Press-Democrat_ reporter covering the incident (as part of my information-sharing-is-good virtue ethics), but decided to refrain because I predicted that Anna would prefer I didn't.
--------
-I had an interesting interaction with [Somni](https://somnilogical.tumblr.com/), one of the "Meeker Four"—presumably out on bail at this time?—on 12 December 2019.
+I had an interesting interaction with [Somni](https://somnilogical.tumblr.com/), one of the "Meeker Four"—presumably out on bail at this time?—on Discord on 12 December 2019.
I told her, from a certain perspective, it's surprising that she spent so much time complaining about CfAR, Anna Salamon, Kelsey Piper, _&c._, but _I_ seemed to get along fine with her—because naïvely, one would think that my views were so much worse. Was I getting a pity pass because she thought false consciousness was causing me to act against my own transfem class interests? Or what?
> I said if they were going to defend a right to be attacking me on some level, and treat fighting back as new aggression and cause to escalate, I would not at any point back down, and if our conflicting definitions of the ground state where no further retaliation was necessary meant we were consigned to a runaway positive feedback loop of revenge, so be it. And if that was true, we might as well try to kill each other right then and there.
- Talking about murder hypothetically as the logical game-theoretic consequence of a revenge spiral isn't the same thing as directly threatening to kill someone. I wasn't sure what exact words Anna had used in her alleged paraphrase; Michael didn't remember the context when I asked him later.
+ Talking about murder hypothetically as the logical game-theoretic consequence of a revenge spiral isn't the same thing as directly threatening to kill someone. (In context, it's calling a bluff: Ziz is saying that if Gwen was asserting a right to mooch off Ziz, then they might as well kill each other; by _modus tollens_, if they don't kill each other, then Gwen's assertion wasn't serious.) I wasn't sure what exact words Anna had used in her alleged paraphrase; Michael didn't remember the context when I asked him later.
I told Michael that this made me think I might need to soul-search about having been complicit with injustice, but I couldn't clearly articulate why.
[^defensive]: Criticism is "defensive" in the sense of trying to _prevent_ new beliefs from being added to our shared map; a critic of an idea "wins" when the idea is not accepted (such that the set of accepted beliefs remains at the _status quo ante_).
-More significantly, in reaction to Yudkowsky's ["Meta-Honesty: Firming Up Honesty Around Its Edge Cases"](https://www.lesswrong.com/posts/xdwbX9pFEr7Pomaxv/meta-honesty-firming-up-honesty-around-its-edge-cases), I published ["Firming Up Not-Lying Around Its Edge-Cases Is Less Broadly Useful Than One Might Initially Think"](https://www.lesswrong.com/posts/MN4NRkMw7ggt9587K/firming-up-not-lying-around-its-edge-cases-is-less-broadly),[^not-lying-title] explaining why merely refraining from making false statments is an unproductively narrow sense of "honesty", because the ambiguity of natural language makes it easy to deceive people in practice without technically lying.
+More significantly, in reaction to Yudkowsky's ["Meta-Honesty: Firming Up Honesty Around Its Edge Cases"](https://www.lesswrong.com/posts/xdwbX9pFEr7Pomaxv/meta-honesty-firming-up-honesty-around-its-edge-cases), I published ["Firming Up Not-Lying Around Its Edge-Cases Is Less Broadly Useful Than One Might Initially Think"](https://www.lesswrong.com/posts/MN4NRkMw7ggt9587K/firming-up-not-lying-around-its-edge-cases-is-less-broadly),[^not-lying-title] explaining why merely refraining from making false statements is an unproductively narrow sense of "honesty", because the ambiguity of natural language makes it easy to deceive people in practice without technically lying.
[^not-lying-title]: The ungainly title was "softened" from an earlier draft following feedback from the posse; I had originally written "... Surprisingly Useless".
On 14 December 2019, I wrote to Jessica and Jack Gallagher, another disaffected ex-MIRI researcher, asking how we should organize this. (Jessica and Jack had relevant testimony about working at MIRI, which would be of more central interest to "Ethan" than my story about how the "rationalists" had lost their way.) Michael also mentioned "Tabitha", a lawyer who had been in the MIRI orbit for a long time, as another person to talk to.
-About a week later, I apologized, saying that I wanted to postpone setting up the meeting, partially because I was on a roll with my productive blogging spree, and partially for a psychological reason: I was feeling subjective pressure to appease Michael by doing the thing that he explicitly suggested because of my loyalty to him, but that would be wrong, because Michael's ideology said that people should follow their sense of opportunity rather than obeying orders. I might feel motived to reach out to "Ethan" and "Tabitha" in January.
+About a week later, I apologized, saying that I wanted to postpone setting up the meeting, partially because I was on a roll with my productive blogging spree, and partially for a psychological reason: I was feeling subjective pressure to appease Michael by doing the thing that he explicitly suggested because of my loyalty to him, but that would be wrong, because Michael's ideology said that people should follow their sense of opportunity rather than obeying orders. I might feel motivated to reach out to "Ethan" and "Tabitha" in January.
Michael said that implied that my sense of opportunity was driven by politics, and that I believed that simple honesty couldn't work; he only wanted me to acknowledge that. I was not inclined to affirm that characterization; it seemed like any conversation with "Ethan" and "Tabitha" would be partially optimized to move money, which I thought was politics.
Jessica pointed out that "it moves money, so it's political" was erasing the non-zero-sum details of the situation. If people can make better decisions (including monetary ones) with more information, then informing them was pro-social. If there wasn't any better decisionmaking from information to be had, and all speech was just a matter of exerting social pressure in favor of one donation target over another, then that would be politics.
-I agreed that my initial "it moves money so it's political" intuition was wrong. But I didn't think I knew how to inform people about giving decisions in an honest and timely way, because the arguments [written above the bottom line](https://www.lesswrong.com/posts/34XxbRFe54FycoCDw/the-bottom-line) were an entire traumatic worldview shift. You couldn't just say "CfAR is fraudulent, don't give to them" without explaining things like ["bad faith is a disposition, not a feeling"](http://benjaminrosshoffman.com/bad-faith-behavior-not-feeling/) as prerequisites. I felt more comfortable trying to share the worldview update in January even if it meant the December decision would be wrong, because I didn't know how to affect the December decision in a way that didn't require someone to trust my judgement.
+I agreed that my initial "it moves money so it's political" intuition was wrong. But I didn't think I knew how to inform people about giving decisions in an honest and timely way, because the arguments [written above the bottom line](https://www.lesswrong.com/posts/34XxbRFe54FycoCDw/the-bottom-line) were an entire traumatic worldview shift. You couldn't just say "CfAR is fraudulent, don't give to them" without explaining things like ["bad faith is a disposition, not a feeling"](http://benjaminrosshoffman.com/bad-faith-behavior-not-feeling/) as prerequisites. I felt more comfortable trying to share the worldview update in January even if it meant the December decision would be wrong, because I didn't know how to affect the December decision in a way that didn't require someone to trust my judgment.
Michael wrote:
In January 2020, Michael told me that he had changed his mind about gender and the philosophy of language. We talked about it on the phone. He said that the philosophy articulated in ["A Human's Guide to Words"](https://www.lesswrong.com/s/SGB7Y5WERh4skwtnb) was inadequate for politicized environments where our choice of ontology is constrained. If we didn't know how to coin a new third gender, or teach everyone the language of "clusters in high-dimensional configuration space", our actual choices for how to think about trans women were basically three: creepy men (the TERF narrative), crazy men (the medical model), or a protected class of actual woman.[^reasons-not-to-carve]
-[^reasons-not-to-carve]: I had identified three classes of reasons not to carve reality at the joints: [coordination (wanting everyone to use the same definitions)](https://www.lesswrong.com/posts/edEXi4SpkXfvaX42j/schelling-categories-and-simple-membership-tests), wireheading (making the map look good, at the expense of it failing to reflect the territory), and war (sabotaging someone else's map to make them do what you want). This would fall under "coordination".
+[^reasons-not-to-carve]: I had identified three classes of reasons not to carve reality at the joints: [coordination (wanting everyone to use the same definitions)](https://www.lesswrong.com/posts/edEXi4SpkXfvaX42j/schelling-categories-and-simple-membership-tests), wireheading (making the map look good, at the expense of it failing to reflect the territory), and war (sabotaging someone else's map to make them do what you want). This would fall under "coordination" insofar as Michael's proposal was motivated by the need to use the same categories as everyone else. (Although you could also make a case for "war" insofar as the civil-rights model winning entailed that adherents of the TERF or medical models must lose.)
According to Michael, while "trans women are real women" was a lie (in the sense that he agreed that me and Jessica and Ziz were not part of the natural cluster of biological females), it was _also_ the case that "trans women are not real women" was a lie (in the sense that the "creepy men" and "crazy men" stories were wrong). "Trans women are women" could be true in the sense that truth is about processes that create true maps, such that we can choose the concepts that allow discourse and information-flow. If the "creepy men" and "crazy men" stories are a cause of silencing, then—under present conditions—we had to chose the "protected class" story in order for people like Ziz to not be silenced.
-My response (more vehemently when thinking on it a few hours later) was that this was a _garbage bullshit_ appeal to consequences. If I wasn't going to let Ray Arnold get away with "we are better at seeking truth when people feel Safe", I shouldn't let Michael get away with "we are better at seeking truth when people aren't Oppressed". Maybe the wider world was ontology-constrained to those three choices, but I was aspiring to higher nuance in my writing, and it seemed to be working pretty well.
+My response (more vehemently when thinking on it a few hours later) was that this was a _garbage bullshit_ appeal to consequences. If I wasn't going to let Ray Arnold get away with "we are better at seeking truth when people feel Safe", I shouldn't let Michael get away with "we are better at seeking truth when people aren't oppressed". Maybe the wider world was ontology-constrained to those three choices, but I was aspiring to higher nuance in my writing, and it seemed to be working pretty well.
"Thanks for being principled," he replied. (He had a few more sentences about the process _vs._ conclusion point being important to his revised-for-politics philosophy of language, but we didn't finish the debate.)
I appreciated the gesture of getting real data, but I was deeply unimpressed with Alexander's analysis for reasons that I found difficult to write up in a timely manner. Three and a half years later, I eventually got around to [polishing my draft and throwing it up as a standalone post](/2023/Nov/reply-to-scott-alexander-on-autogenderphilia/).
-Briefly, based on eyballing the survey data, Alexander proposes "if you identify as a gender, and you're attracted to that gender, it's a natural leap to be attracted to yourself being that gender" as a "very boring" theory, but on my worldview, a hypothesis that puts "gay people (cis and trans)" in the antecedent is not boring and takes on a big complexity penalty: I don't think the group of gay men _and_ lesbians _and_ straight males with female gender identities _and_ straight females with male gender identities have much in common with each other, except sociologically (being "queer"), and by being human.
+Briefly, based on eyeballing the survey data, Alexander proposes "if you identify as a gender, and you're attracted to that gender, it's a natural leap to be attracted to yourself being that gender" as a "very boring" theory, but on my worldview, a hypothesis that puts "gay people (cis and trans)" in the antecedent is not boring and takes on a big complexity penalty: I don't think the group of gay men _and_ lesbians _and_ straight males with female gender identities _and_ straight females with male gender identities have much in common with each other, except sociologically (being "queer"), and by being human.
(I do like the [hypernym](https://en.wikipedia.org/wiki/Hyponymy_and_hypernymy) _autogenderphilia_.)
Previously, I had already thought it was nuts that trans ideology was exerting influence on the rearing of gender-non-conforming children—that is, children who are far outside the typical norm of behavior for their sex: very tomboyish girls and very effeminate boys.
-Under recent historical conditions in the West, these kids were mostly "pre-gay" rather than trans. (The stereotype about lesbians being masculine and gay men being feminine is, like most stereotypes, basically true: sex-atypical childhood behavior between gay and straight adults [has been meta-analyzed at _d_ ≈ 1.31 for men and _d_ ≈ 0.96 for women](/papers/bailey-zucker-childhood_sex-typed_behavior_and_sexual_orientation.pdf).) A solid supermajority of children diagnosed with gender dysphoria [ended up growing out of it by puberty](/papers/steensma_et_al-factors_associated_with_desistence_and_persistence.pdf). In the culture of the current year, it seemed likely that a lot of those kids would instead get affirmed into a cross-sex identity at a young age, even though most of them would have otherwise (under [a "watchful waiting" protocol](/papers/de_vries-cohen-kettenis-clinical_management_of_gender_dysphoria_in_children.pdf)) grown up to be ordinary gay men and lesbians.
+Under recent historical conditions in the West, these kids were mostly "pre-gay" rather than trans. (The stereotype about lesbians being masculine and gay men being feminine is, like most stereotypes, basically true: sex-atypical childhood behavior between gay and straight adults [has been meta-analyzed at](/papers/bailey-zucker-childhood_sex-typed_behavior_and_sexual_orientation.pdf) [Cohen's _d_](/2019/Sep/does-general-intelligence-deflate-standardized-effect-sizes-of-cognitive-sex-differences/) ≈ 1.31 standard deviations for men and _d_ ≈ 0.96 for women.) A majority of children diagnosed with gender dysphoria [ended up growing out of it by puberty](/papers/steensma_et_al-factors_associated_with_desistence_and_persistence.pdf). In the culture of the current year, it seemed likely that a lot of those kids would instead get affirmed into a cross-sex identity at a young age, even though most of them would have otherwise (under [a "watchful waiting" protocol](/papers/de_vries-cohen-kettenis-clinical_management_of_gender_dysphoria_in_children.pdf)) grown up to be ordinary gay men and lesbians.
What made this shift in norms crazy, in my view, was not just that transitioning younger children is a dubious treatment decision, but that it's a dubious treatment decision that was being made on the basis of the obvious falsehood that "trans" was one thing: the cultural phenomenon of "trans kids" was being used to legitimize trans _adults_, even though a supermajority of trans adults were in the late-onset/AGP taxon and therefore had never resembled these HSTS-taxon kids. That is: pre-gay kids in our Society are being sterilized in order to affirm the narcissistic delusions of guys like me.
But if the grown-ups have been trained to believe that "trans kids know who they are"—if they're emotionally eager at the prospect of having a transgender child, or fearful of the damage they might do by not affirming—they might selectively attend to confirming evidence that the child "is trans", selectively ignore contrary evidence that the kid "is cis", and end up reinforcing a cross-sex identity that would not have existed if not for their belief in it.
-Crucially, if innate gender identity isn't a feature of toddler psychology, _the child has no way to know anything is "wrong."_ If none of the grown-ups can say, "You're a boy because boys are the ones with penises" (because that's not what people are supposed to believe in the current year), how is the child supposed to figure that out independently? [Toddlers are not very sexually dimorphic](/2019/Jan/the-dialectic/), but large sex differences in play style and social behavior tend to emerge within a few years. (There were no cars in the environment of evolutionary adaptedness, and yet [the effect size of the sex difference in preference for toy vehicles is a massive _d_ ≈ 2.44](/papers/davis-hines-how_large_are_gender_differences_in_toy_preferences.pdf), about one and a half times the size of the sex difference in adult height.)
+Crucially, if innate gender identity isn't a feature of toddler psychology, _the child has no way to know anything is "wrong."_ If none of the grown-ups can say, "You're a boy because boys are the ones with penises" (because that's not what people are supposed to believe in the current year), how is the child supposed to figure that out independently? [Toddlers are not very sexually dimorphic](/2019/Jan/the-dialectic/), but sex differences in play style and social behavior tend to emerge within a few years. (There were no cars in the environment of evolutionary adaptedness, and yet [the effect size of the sex difference in preference for toy vehicles is a massive _d_ ≈ 2.44](/papers/davis-hines-how_large_are_gender_differences_in_toy_preferences.pdf), about one and a half times the size of the sex difference in adult height.)
-What happens when the kid develops a self-identity as "a girl", only to find out, potentially years later, that she noticeably doesn't fit in with the (cis) girls on the [many occasions that no one has explicitly spelled out in advance](/2019/Dec/more-schelling/) where people are using "gender" (percieved sex) to make a prediction or decision?
+What happens when the kid develops a self-identity as "a girl", only to find out, potentially years later, that she noticeably doesn't fit in with the (cis) girls on the [many occasions that no one has explicitly spelled out in advance](/2019/Dec/more-schelling/) where people are using "gender" (perceived sex) to make a prediction or decision?
Some might protest, "But what's the harm? She can always change her mind later if she decides she's actually a boy." I don't doubt that if the child were to clearly and distinctly insist, "I'm definitely a boy," the nice smart liberal grown-ups would unhesitatingly accept that.
Suppose that, around the time of the social transition, the child reportedly responded to "hey kiddo, I love you" with, "I'm a girl and I'm a vegetarian." In the skeptic's view, both halves of that sentence were probably generated by the same cognitive algorithm—probably something like, practice language and be cute to caregivers, making use of themes from the local culture environment (where grown-ups in Berkeley talk a lot about gender and animal welfare). If you're not going to change the kid's diet on the basis of the second part, you shouldn't social transition the kid on the basis of the first part.
-It's not hard to imagine how differential treatment by grown-ups of gender-related utterances could unintentionally shape outcomes. This may be clearer if we imagine a non-gender-related case. Suppose the child's father's name is Kevin Smith, and that after a grown-up explains ["Sr."/"Jr." generational suffixes](https://en.wikipedia.org/wiki/Suffix_(name)#Generational_titles) after it [happened to come up in fiction](https://wreckitralph.fandom.com/wiki/Fix-It_Felix,_Jr._(character)), the child declares that his name is Kevin Smith, Jr. now. Caregivers are likely to treat this as just a cute thing that the kid said, quickly forgotten by all. But if caregivers feared causing psychological harm by denying a declared name change, one could imagine them taking the child's statment as a prompt to ask followup questions. ("Oh, would you like me to call you _Kevin_ or _Kev Jr._, or just _Junior_?") With enough followup, it seems entirely plausible that a name change to "Kevin Jr." would meet with the child's assent and "stick" socially. The initial suggestion would have come from the child, but most of the [optimization](https://www.lesswrong.com/posts/D7EcMhL26zFNbJ3ED/optimization)—the selection that this particular one of the child's many statements should be taken literally and reinforced as a social identity, while others are just treated a cute thing the kid said—would have come from the adults.
+It's not hard to imagine how differential treatment by grown-ups of gender-related utterances could unintentionally shape outcomes. This may be clearer if we imagine a non-gender-related case. Suppose the child's father's name is Kevin Smith, and that after a grown-up explains ["Sr."/"Jr." generational suffixes](https://en.wikipedia.org/wiki/Suffix_(name)#Generational_titles) after it [happened to come up in fiction](https://wreckitralph.fandom.com/wiki/Fix-It_Felix,_Jr._(character)), the child declares that his name is Kevin Smith, Jr. now. Caregivers are likely to treat this as just a cute thing that the kid said, quickly forgotten by all. But if caregivers feared causing psychological harm by denying a declared name change, one could imagine them taking the child's statement as a prompt to ask followup questions. ("Oh, would you like me to call you _Kevin_ or _Kev Jr._, or just _Junior_?") With enough followup, it seems entirely plausible that a name change to "Kevin Jr." would meet with the child's assent and "stick" socially. The initial suggestion would have come from the child, but most of the [optimization](https://www.lesswrong.com/posts/D7EcMhL26zFNbJ3ED/optimization)—the selection that this particular one of the child's many statements should be taken literally and reinforced as a social identity, while others are just treated a cute thing the kid said—would have come from the adults.
Finally, there is the matter of the child's behavior and personality. For example, around the same time that the child's social transition was going down, the father reported the child being captivated by seeing a forklift at Costco. A few months later, another family friend remarked that maybe the child is very competitive, and that "she likes fighting so much because it's the main thing she knows of that you can _win_".
But if you do have the math, a moment of introspection will convince you that the analogy between category "boundaries" and national borders is shallow.
-A two-dimensional political map tells you which areas of the Earth's surface are under the jurisdiction of which government. In contrast, category "boundaries" tell you which regions of very high-dimensional configuration space correspond to a word/concept, which is useful _because_ that structure is useful for making probabilistic inferences: you can use your observastions of some aspects of an entity (some of the coordinates of a point in configuration space) to infer category-membership, and then use category membership to make predictions about aspects that you haven't yet observed.
+A two-dimensional political map tells you which areas of the Earth's surface are under the jurisdiction of which government. In contrast, category "boundaries" tell you which regions of very high-dimensional configuration space correspond to a word/concept, which is useful _because_ that structure is useful for making probabilistic inferences: you can use your observations of some aspects of an entity (some of the coordinates of a point in configuration space) to infer category-membership, and then use category membership to make predictions about aspects that you haven't yet observed.
But the trick only works to the extent that the category is a regular, non-squiggly region of configuration space: if you know that egg-shaped objects tend to be blue, and you see a black-and-white photo of an egg-shaped object, you can get close to picking out its color on a color wheel. But if egg-shaped objects tend to blue _or_ green _or_ red _or_ gray, you wouldn't know where to point to on the color wheel.
I decided on "Unnatural Categories Are Optimized for Deception" as the title for my advanced categorization thesis. Writing it up was a major undertaking. There were a lot of nuances to address and potential objections to preëmpt, and I felt that I had to cover everything. (A reasonable person who wanted to understand the main ideas wouldn't need so much detail, but I wasn't up against reasonable people who wanted to understand.)
-In September 2020, Yudkowsky Tweeted [something about social media incentives prompting people to make nonsense arguments](https://twitter.com/ESYudkowsky/status/1304824253015945216), and something in me boiled over. The Tweet was fine in isolation, but I rankled at it given the absurdly disproprotionate efforts I was undertaking to unwind his incentive-driven nonsense. I left [a pleading, snarky reply](https://twitter.com/zackmdavis/status/1304838486810193921) and [vented on my own timeline](https://twitter.com/zackmdavis/status/1304838346695348224) (with preview images from the draft of "Unnatural Categories Are Optimized for Deception"):
+In September 2020, Yudkowsky Tweeted [something about social media incentives prompting people to make nonsense arguments](https://twitter.com/ESYudkowsky/status/1304824253015945216), and something in me boiled over. The Tweet was fine in isolation, but I rankled at it given the absurdly disproportionate efforts I was undertaking to unwind his incentive-driven nonsense. I left [a pleading, snarky reply](https://twitter.com/zackmdavis/status/1304838486810193921) and [vented on my own timeline](https://twitter.com/zackmdavis/status/1304838346695348224) (with preview images from the draft of "Unnatural Categories Are Optimized for Deception"):
> Who would have thought getting @ESYudkowsky's robot cult to stop trying to trick me into cutting my dick off (independently of the empirical facts determining whether or not I should cut my dick off) would involve so much math?? OK, I guess the math part isn't surprising, but—[^trying-to-trick-me]
>
> The problem with "it's a policy debate about how to use language" is that it completely elides the issue that some ways of using language _perform better_ at communicating information, such that **attempts to define new words or new senses of _existing_ words should come with a justification for why the new sense is _useful for conveying information_, and that _is_ a matter of Truth.** Without such a justification, it's hard to see why you would _want_ to redefine a word _except_ to mislead people with strategic equivocation.
>
-> It is _literally true_ that Eliezer Yudkowsky is a white supremacist (if I'm allowed to define "white supremacist" to include "someone who [once linked to the 'Race and intelligence' _Wikipedia_ page](https://www.lesswrong.com/posts/faHbrHuPziFH7Ef7p/why-are-individual-iq-differences-ok) in a context that implied that it's an empirical question").
+> <a id="literally-a-white-supremacist"></a>It is _literally true_ that Eliezer Yudkowsky is a white supremacist (if I'm allowed to define "white supremacist" to include "someone who [once linked to the 'Race and intelligence' _Wikipedia_ page](https://www.lesswrong.com/posts/faHbrHuPziFH7Ef7p/why-are-individual-iq-differences-ok) in a context that implied that it's an empirical question").
>
> It is _literally true_ that 2 + 2 = 6 (if I'm allowed to define '2' as •••-many).
>
>
> [... redacted ...]
>
-> But if behavior at equilibrium isn't deceptive, there's just _no such thing as deception_; I wrote about this on Less Wrong in ["Maybe Lying Can't Exist?!"](https://www.lesswrong.com/posts/YptSN8riyXJjJ8Qp8/maybe-lying-can-t-exist) (drawing on the academic literature about sender–reciever games). I don't think you actually want to bite that bullet?
+> But if behavior at equilibrium isn't deceptive, there's just _no such thing as deception_; I wrote about this on Less Wrong in ["Maybe Lying Can't Exist?!"](https://www.lesswrong.com/posts/YptSN8riyXJjJ8Qp8/maybe-lying-can-t-exist) (drawing on the academic literature about sender–receiver games). I don't think you actually want to bite that bullet?
>
> **In terms of information transfer, there is an isomorphism between saying "I reserve the right to lie 5% of the time about whether something is a member of category C" and adopting a new definition of C that misclassifies 5% of instances with respect to the old definition.**
>
>
> **It makes sense that you don't want to get involved in gender politics. That's why I wrote "... Boundaries?" using examples about dolphins and job titles, and why my forthcoming post has examples about bleggs and artificial meat.** This shouldn't be _expensive_ to clear up?! This should take like, five minutes? (I've spent twenty-one months of my life on this.) Just one little _ex cathedra_ comment on Less Wrong or _somewhere_ (**it doesn't have to be my post, if it's too long or I don't deserve credit or whatever**; I just think the right answer needs to be public) affirming that you haven't changed your mind about 37 Ways Words Can Be Wrong? Unless you _have_ changed your mind, of course?
>
-> I can imagine someone observing this conversation objecting, "[...] why are you being so greedy? We all know the _real_ reason you want to clear up this philosophy thing in public is because it impinges on your gender agenda, but Eliezer _already_ threw you a bone with the ['there's probably more than one type of dypshoria' thing.](https://twitter.com/ESYudkowsky/status/1108277090577600512) That was already a huge political concession to you! That makes you _more_ than even; you should stop being greedy and leave Eliezer alone."
+> I can imagine someone observing this conversation objecting, "[...] why are you being so greedy? We all know the _real_ reason you want to clear up this philosophy thing in public is because it impinges on your gender agenda, but Eliezer _already_ threw you a bone with the ['there's probably more than one type of dysphoria' thing.](https://twitter.com/ESYudkowsky/status/1108277090577600512) That was already a huge political concession to you! That makes you _more_ than even; you should stop being greedy and leave Eliezer alone."
>
> But as [I explained in my reply](/2019/Dec/on-the-argumentative-form-super-proton-things-tend-to-come-in-varieties/) criticizing why I think that argument is _wrong_, the whole mindset of public-arguments-as-political-favors is _crazy_. **The fact that we're having this backroom email conversation at all (instead of just being correct about the philosophy of language on Twitter) is _corrupt_!** I don't want to strike a deal in a political negotiation; I want _shared maps that reflect the territory_. I thought that's what this "rationalist community" thing was supposed to do? Is that not a thing anymore? If we can't do the shared-maps thing when there's any hint of political context (such that now you _can't_ clarify the categories thing, even as an abstract philosophy issue about bleggs, because someone would construe that as taking a side on whether trans people are Good or Bad), that seems really bad for our collective sanity?! (Where collective sanity is potentially useful for saving the world, but is at least a quality-of-life improver if we're just doomed to die in 15 years no matter what.)
>
> you are being the bad guy if you try to shut down that conversation by saying that "I can define the word 'woman' any way I want"
-There it is! A clear _ex cathedra_ statement that gender categories are not an exception to the general rule that categories aren't arbitary. (Only 1 year and 8 months after [asking for it](/2023/Jul/a-hill-of-validity-in-defense-of-meaning/#ex-cathedra-statement-ask).) I could quibble with some of Yudkowsky's exact writing choices, which I thought still bore the signature of political maneuvering, but it would be petty to dwell on quibbles when the core problem had been addressed.
+There it is! A clear _ex cathedra_ statement that gender categories are not an exception to the general rule that categories aren't arbitrary. (Only 1 year and 8 months after [asking for it](/2023/Jul/a-hill-of-validity-in-defense-of-meaning/#ex-cathedra-statement-ask).) I could quibble with some of Yudkowsky's exact writing choices, which I thought still bore the signature of political squirming,[^clarification-quibbles] but it would be petty to dwell on quibbles when the core problem had been addressed.
+
+[^clarification-quibbles]: The way that the post takes pains to cast doubt on whether someone who is alleged to have committed the categories-are-arbitrary fallacy is likely to have actually committed it ("the mistake seems like it wouldn't actually fool anybody or be committed in real life, I am unlikely to be sympathetic to the argument", "But be wary of accusing somebody of planning to do this, if you haven't documented them actually doing it") is in stark contrast to the way that "A Human's Guide to Words" had taken pains to emphasize that categories shape cognition regardless of whether someone is consciously trying to trick you (["drawing a boundary in thingspace is not a neutral act [...] Categories are not static things in the context of a human brain; as soon as you actually think of them, they exert force on your mind"](https://www.lesswrong.com/posts/veN86cBhoe7mBxXLk/categorizing-has-consequences)). I'm suspicious that the change in emphasis reflects the need to not be seen as criticizing the "pro-trans" coalition, rather than any new insight into the subject matter.
+
+ The first comment on the post linked to "... Not Man for the Categories". Yudkowsky replied, "I assumed everybody reading this had already read [https://wiki.lesswrong.com/wiki/A_Human's_Guide_to_Words](https://wiki.lesswrong.com/wiki/A_Human's_Guide_to_Words)", a _non sequitur_ that could be taken to suggest (but did not explicitly say) that the moral of "... Not Man for the Categories" was implied by "A Human's Guide to Words" (in contrast to my contention that "... Not Man for the Categories" was getting it wrong).
I wrote to Michael, Ben, Jessica, Sarah, and "Riley", thanking them for their support. After successfully bullying Scott and Eliezer into clarifying, I was no longer at war with the robot cult and feeling a lot better (Subject: "thank-you note (the end of the Category War)").
-I had a feeling, I added, that Ben might be disappointed with the thank-you note insofar as it could be read as me having been "bought off" rather than being fully on the side of clarity-creation. But not being at war actually made it emotionally easier to do clarity-creation writing. Now I would be able to do it in the spirit of "Here's what I think the thing is actually doing" rather than the spirit of "I hate you lying motherfuckers _so much_. [It, it—the fe—it, flame—flames. Flames—on the side of my face.](https://www.youtube.com/watch?v=nrqxmQr-uto)"
+I had a feeling, I added, that Ben might be disappointed with the thank-you note insofar as it could be read as me having been "bought off" rather than being fully on the side of clarity-creation. But I contended that not being at war actually made it emotionally easier to do clarity-creation writing. Now I would be able to do it in a contemplative spirit of "Here's what I think the thing is actually doing" rather than in hatred with [flames on the side of my face](https://www.youtube.com/watch?v=nrqxmQr-uto&t=112s).
-----
-If this were an autobiography (which existed to tell my life story) rather than a topic-focused memoir (which exists because my life happens to contain this Whole Dumb Story which bears on matters of broader interest, even if my life would not otherwise be interesting), there's a dramatic episode that would fit here chronologically.
+There's a dramatic episode that would fit here chronologically if this were an autobiography (which existed to tell my life story), but since this is a topic-focused memoir (which exists because my life happens to contain this Whole Dumb Story which bears on matters of broader interest, even if my life would not otherwise be interesting), I don't want to spend more wordcount than is needed to briefly describe the essentials.
-I was charged by members of the "Vassarite" clique in New York with the duty of taking care of a mentally-ill person at my house on 18 December 2020. (We did not trust the ordinary psychiatric system to act in patients' interests.) I apparently did a poor job, and ended up saying something callous on the care team group chat after a stressful night, which led to a chaotic day on the nineteenth, and an ugly falling-out between me and the group. In the interests of brevity and the privacy of the person we were trying to help, I think it's better that I don't expend the wordcount to give you a play-by-play. The details aren't particularly of public interest.
+I was charged by members of the "Vassarite" clique with the duty of taking care of a mentally-ill person at my house on 18 December 2020. (We did not trust the ordinary psychiatric system to act in patients' interests.) I apparently did a poor job, and ended up saying something callous on the care team group chat after a stressful night, which led to a chaotic day on the nineteenth, and an ugly falling-out between me and the group. In the interests of brevity and the privacy of the person we were trying to help, I think it's better that I don't give you a play-by-play. The details aren't particularly of public interest.
-My poor performance during this incident [weighs on my conscience](/2020/Dec/liability/) particularly because I had previously been in the position of being crazy and benefitting from the help of my friends (including many of the same people involved in this incident) rather than getting sent back to psychiatric prison ("hospital", they call it a "hospital"). Of all people, I had a special debt to "pay it forward", and one might have hoped that I would also have special skills, that remembering being on the receiving end of a psychiatric tripsitting operation would help me know what to do on the giving end. Neither of those panned out.
+My poor performance during this incident [weighs on my conscience](/2020/Dec/liability/) particularly because I had previously been in the position of being crazy and benefiting from the help of my friends (including many of the same people involved in this incident) rather than getting sent back to psychiatric prison ("hospital", they call it a "hospital"). Of all people, I had a special debt to "pay it forward", and one might have hoped that I would also have special skills, that remembering being on the receiving end of a psychiatric tripsitting operation would help me know what to do on the giving end. Neither of those panned out.
-Some might appeal to the proverb, "All's well that ends well", noting that the person in trouble ended up being okay, and that, while the stress contributed to me having a relapse of some of my own psychological problems on the night of the nineteenth and in the following weeks, I ended up being okay, too (at the cost of missing a week of my dayjob and giving up caffeine permanently). I am instead inclined to dwell on [another proverb](https://www.alessonislearned.com/), "A lesson is learned but the damage is irreversible."
+Some might appeal to the proverb, "All's well that ends well", noting that the person in trouble ended up recovering, and that, while the stress contributed to me having a somewhat serious relapse of some of my own psychological problems on the night of the nineteenth and in the following weeks, I ended up recovering, too. I am instead inclined to dwell on [another proverb](https://www.alessonislearned.com/), "A lesson is learned but the damage is irreversible."
-----
I published ["Unnatural Categories Are Optimized for Deception"](https://www.lesswrong.com/posts/onwgTH6n8wxRSo2BJ/unnatural-categories-are-optimized-for-deception) in January 2021.
-I wrote back to Abram Demski regarding his comments from fourteen months before: on further thought, he was right. Even granting my point that evolution didn't figure out how to track probability and utility separately, as Abram had pointed out, the _fact_ that it didn't meant that not tracking it could be an effective AI design. Just because evolution takes shortcuts that human engineers wouldn't didn't mean shortcuts are "wrong". (Rather, there are laws governing which kinds of shortcuts _work_.)
+I wrote back to Abram Demski regarding his comments from fourteen months before: on further thought, he was right. Even granting my point that evolution didn't figure out how to track probability and utility separately, as Abram had pointed out, the fact that it didn't meant that not tracking it could be an effective AI design. Just because evolution takes shortcuts that human engineers wouldn't didn't mean shortcuts are "wrong". (Rather, there are laws governing which kinds of shortcuts work.)
-Abram was also right that it would be weird if reflective coherence was somehow impossible: the AI shouldn't have to fundamentally reason differently about "rewriting code in some 'external' program" and "rewriting 'its own' code." In that light, it made sense to regard "have accurate beliefs" as _merely_ a convergent instrumental subgoal, rather than what rationality is about—as sacrilegious as that felt to type.
+Abram was also right that it would be weird if reflective coherence was somehow impossible: the AI shouldn't have to fundamentally reason differently about "rewriting code in some 'external' program" and "rewriting 'its own' code." In that light, it made sense to regard "have accurate beliefs" as merely a convergent instrumental subgoal, rather than what rationality is about—as sacrilegious as that felt to type.
-And yet, somehow, "have accurate beliefs" seemed _more fundamental_ than other convergent instrumental subgoals like "seek power and resources". Could this be made precise? As a stab in the dark, was it possible that the [theorems on the ubiquity of power-seeking](https://www.lesswrong.com/posts/6DuJxY8X45Sco4bS2/seeking-power-is-often-robustly-instrumental-in-mdps) might generalize to a similar conclusion about "accuracy-seeking"? If it _didn't_, the reason why it didn't might explain why accuracy seems more fundamental.
+And yet, somehow, "have accurate beliefs" seemed more fundamental than other convergent instrumental subgoals like "seek power and resources". Could this be made precise? As a stab in the dark, was it possible that the [theorems on the ubiquity of power-seeking](https://www.lesswrong.com/posts/6DuJxY8X45Sco4bS2/seeking-power-is-often-robustly-instrumental-in-mdps) might generalize to a similar conclusion about "accuracy-seeking"? If it didn't, the reason why it didn't might explain why accuracy seems more fundamental.
------
-And really, that _should_ have been the end of the story. At the trifling cost of two years of my life, we finally got a clarification from Yudkowsky that you can't define the word _woman_ any way you like. I didn't think I was entitled to anything more than that. I was satisfied. If I hadn't been further provoked, I wouldn't have occasion to continue waging the robot-cult religious civil war.
+And really, that should have been the end of the story. At the trifling cost of two years of my life, we finally got a clarification from Yudkowsky that you can't define the word _woman_ any way you like. I didn't think I was entitled to anything more than that. I was satisfied. If I hadn't been further provoked, I wouldn't have occasion to continue waging the robot-cult religious civil war.
It turned out that I would have occasion to continue waging the robot-cult religious civil war. (To be continued.)
projects / Ultimately_Untrue_Thought.git / blobdiff