Title: If Clarity Seems Like Death to Them
Author: Zack M. Davis
-Date: 2023-01-01 11:00
+Date: 2023-07-01 11:00
Category: commentary
Tags: autogynephilia, bullet-biting, cathartic, Eliezer Yudkowsky, Scott Alexander, epistemic horror, my robot cult, personal, sex differences, two-type taxonomy, whale metaphors
Status: draft
Ben called the _gestalt_ he saw the Blight, after the rogue superintelligence in Vernor Vinge's _A Fire Upon the Deep_: the problem wasn't that people were getting dumber; it was that there was locally coherent coordination away from clarity and truth and towards coalition-building, which was validated by the official narrative in ways that gave it a huge tactical advantage; people were increasingly making decisions that were better explained by their political incentives rather than acting on coherent beliefs about the world—using and construing claims about facts as moves in a power game, albeit sometimes subject to genre constraints under which only true facts were admissible moves in the game.
-When I asked him for specific examples of MIRI or CfAR leaders behaving badly, he gave the example of [MIRI executive director Nate Soares posting that he was "excited to see OpenAI joining the space"](https://intelligence.org/2015/12/11/openai-and-other-news/), despite the fact that [_no one_ who had been following the AI risk discourse](https://slatestarcodex.com/2015/12/17/should-ai-be-open/) [thought that OpenAI as originally announced was a good idea](http://benjaminrosshoffman.com/openai-makes-humanity-less-safe/). Nate had privately clarified that the word "excited" wasn't necessarily meant positively, and in this case meant something more like "terrified."
+When I asked him for specific examples of MIRI or CfAR leaders behaving badly, he gave the example of [MIRI executive director Nate Soares posting that he was "excited to see OpenAI joining the space"](https://intelligence.org/2015/12/11/openai-and-other-news/), despite the fact that [_no one_ who had been following the AI risk discourse](https://slatestarcodex.com/2015/12/17/should-ai-be-open/) [thought that OpenAI as originally announced was a good idea](http://benjaminrosshoffman.com/openai-makes-humanity-less-safe/). Nate [had privately clarified that](https://twitter.com/jessi_cata/status/1462454555925434375) the word "excited" wasn't necessarily meant positively, and in this case meant something more like "terrified."
This seemed to me like the sort of thing where a particularly principled (naïve?) person might say, "That's _lying for political reasons!_ That's _contrary to the moral law!_" and most ordinary grown-ups would say, "Why are you so upset about this? That sort of strategic phrasing in press releases is just how the world works, and things could not possibly be otherwise."
And as it happened, on 4 May 2019, Yudkowsky [re-Tweeted Colin Wright on the "univariate fallacy"](https://twitter.com/ESYudkowsky/status/1124751630937681922)—the point that group differences aren't a matter of any single variable—which was _sort of_ like the clarification I had been asking for. (Empirically, it made me feel a lot less personally aggrieved.) Was I wrong to interpet this as another "concession" to me? (Again, notwithstanding that the whole mindset of extracting "concessions" was corrupt and not what our posse was trying to do.)
-Separately, I visited some friends' house on 30 April 2019 saying, essentially (and sincerely), "Oh man oh jeez, Ben and Michael want me to join in a rationalist civil war against the corrupt mainstream-rationality establishment, and I'd really rather not, and I don't like how they keep using scary hyperbolic words like 'cult' and 'war' and 'criminal', but on the other hand, they're _the only ones backing me up_ on this _incredibly basic philosophy thing_ and I don't feel like I have anywhere else to _go_." The ensuing group conversation made some progress, but was mostly pretty horrifying.
+Separately, on 30 April 2019, I visited the house where "Susan" and her husband Mike and Kelsey Piper and a few other people lived, which I'll call "Arcadia",[^named-houses] saying, essentially (and sincerely), Oh man oh jeez, Ben and Michael want me to join in a rationalist civil war against the corrupt mainstream-rationality establishment, and I'd really rather not, and I don't like how they keep using scary hyperbolic words like 'cult' and 'war' and 'criminal', but on the other hand, they're _the only ones backing me up_ on this _incredibly basic philosophy thing_ and I don't feel like I have anywhere else to _go_. The ensuing group conversation made some progress, but was mostly pretty horrifying.
-In an adorable twist, my friends' two-year-old son was reportedly saying the next day that Kelsey doesn't like his daddy, which was confusing until it was figured out he had heard Kelsey talking about why she doesn't like Michael _Vassar_.
+[^named-houses]: It was common practice in our subculture to name group houses. My apartment was "We'll Name It Later."
+
+In an adorable twist, Mike and "Susan"'s two-year-old son was reportedly saying the next day that Kelsey doesn't like his daddy, which was confusing until it was figured out he had heard Kelsey talking about why she doesn't like Michael _Vassar_.[^mike-pseudonym]
+
+[^mike-pseudonym]: I'm not giving Mike a pseudonym because his name is needed for this adorable anecdote to make sense, and this Whole Dumb Story isn't otherwise saying sensitive things about him.
And as it happened, on 7 May 2019, Kelsey wrote [a Facebook comment displaying evidence of understanding my point](/images/piper-spending_social_capital_on_talking_about_trans_issues.png).
Anna said she didn't want to receive monetary offers from me anymore; previously, she had regarded my custom of throwing money at people to get what I wanted as good-faith libertarianism between consenting adults, but now she was afraid that if she accepted, it would be portrayed in some future Ben Hoffman essay as an instance of her _using_ me. She agreed that someone could have gotten the ideals I had gotten out of "A Sense That More Is Possible", "Raising the Sanity Waterline", _&c._, but there was also evidence from that time pointing the other way (_e.g._, ["Politics Is the Mind-Killer"](https://www.lesswrong.com/posts/9weLK2AJ9JEt2Tt8f/politics-is-the-mind-killer)), that it shouldn't be surprising if people steered clear of controversy.
-I replied: but when forming the original let's-be-apolitical vision in 2008, we did not anticipate that _whether or not I should cut my dick off_ would _become_ a political issue. That was _new evidence_ about whether the original vision was wise! I wasn't trying to do politics with my idiosyncratic special interest; I was trying to _think seriously_ about the most important thing in my life and only do the minimum amount of politics necessary to protect my ability to think. If 2019-era "rationalists" were going to commit a trivial epistemology mistake that interfered with my ability to think seriously about the most important thing in my life, but couldn't correct the mistake, then the 2019-era "rationalists" were _worse than useless_ to me personally. This probably didn't matter causally (I wasn't an AI researcher, therefore I didn't matter), but it might matter timelessly (if I was part of a reference class that includes AI researchers).
+I replied: but when forming the original let's-be-apolitical vision in 2008, we did not anticipate that _whether or not I should cut my dick off_ would _become_ a political issue. That was _new evidence_ about whether the original vision was wise! I wasn't trying to do politics with my idiosyncratic special interest; I was trying to _think seriously_ about the most important thing in my life and only do the minimum amount of politics necessary to protect my ability to think. If 2019-era "rationalists" were going to commit a trivial epistemology mistake that interfered with my ability to think seriously about the most important thing in my life, but couldn't correct the mistake even after it was pointed out, then the 2019-era "rationalists" were _worse than useless_ to me personally. This probably didn't matter causally (I wasn't an AI researcher, therefore I didn't matter), but it might matter timelessly (if I was part of a reference class that includes AI researchers).
Fundamentally, I was skeptical that you _could_ do consisently high-grade reasoning as a group without committing heresy, because of the mechanism that Yudkowsky described in ["Entangled Truths, Contagious Lies"](https://www.lesswrong.com/posts/wyyfFfaRar2jEdeQK/entangled-truths-contagious-lies) and ["Dark Side Epistemology"](https://www.lesswrong.com/posts/XTWkjCJScy2GFAgDt/dark-side-epistemology): the need to lie about lying and cover up cover-ups propagates recursively. Anna in particular was unusually skillful at thinking things without saying them; I thought most people facing similar speech restrictions just get worse at thinking (plausibly[^plausibly] including Yudkowsky), and the problem gets worse as the group effort scales. (It's easier to recommend ["What You Can't Say"](http://www.paulgraham.com/say.html) to your housemates than to put it on a canonical reading list, for obvious reasons.) You _can't_ optimize your group's culture for not-talking-about-atheism without also optimizing against understanding [Occam's razor](https://www.lesswrong.com/posts/f4txACqDWithRi7hs/occam-s-razor); you _can't_ optimize for not questioning gender self-identity without also optimizing against understanding the [37 ways that words can be wrong](https://www.lesswrong.com/posts/FaJaCgqBKphrDzDSj/37-ways-that-words-can-be-wrong).
Despite Math and Wellness Month and my "intent" to take a break from the religious civil war, I kept reading _Less Wrong_ during May 2019, and ended up scoring a couple of victories in the civil war (at some cost to Wellness).
-MIRI researcher Scott Garrabrant wrote a post about how ["Yes Requires the Possibility of No"](https://www.lesswrong.com/posts/G5TwJ9BGxcgh5DsmQ/yes-requires-the-possibility-of-no). Information-theoretically, a signal sent with probability one transmits no information: you can only learn something from hearing a "Yes" if there was some chance that the answer could have been "No". I saw an analogy to my philosophy-of-language thesis, and commented about it: if you want to believe that _x_ belongs to category _C_, you might try redefining _C_ in order to make the question "Is _x_ a _C_?" come out "Yes", but you can only do so at the expense of making _C_ less useful. Meaningful category-membership (Yes) requires the possibility of non-membership (No).
+MIRI researcher Scott Garrabrant wrote a post about how ["Yes Requires the Possibility of No"](https://www.lesswrong.com/posts/G5TwJ9BGxcgh5DsmQ/yes-requires-the-possibility-of-no). Information-theoretically, a signal sent with probability one transmits no information: you can only learn something from hearing a "Yes" if believed that the answer could have been "No". I saw an analogy to my philosophy-of-language thesis, and mentioned it in a comment: if you want to believe that _x_ belongs to category _C_, you might try redefining _C_ in order to make the question "Is _x_ a _C_?" come out "Yes", but you can only do so at the expense of making _C_ less useful. Meaningful category-membership (Yes) requires the possibility of non-membership (No).
-[TODO: explain scuffle on "Yes Requires the Possibility"—
+MIRI research associate Vanessa Kosoy [commented](https://www.lesswrong.com/posts/WwTPSkNwC89g3Afnd/comment-section-from-05-19-2019?commentId=FxSZwECjhgYE7p2du):
- * Vanessa comment on hobbyhorses and feeling attacked
- * my reply about philosophy got politicized, and MDL/atheism analogy
- * Ben vs. Said on political speech and meta-attacks; Goldenberg on feelings
- * 139-comment trainwreck got so bad, the mods manually moved the comments into their own thread https://www.lesswrong.com/posts/WwTPSkNwC89g3Afnd/comment-section-from-05-19-2019
- * based on the karma scores and what was said, this went pretty well for me and I count it as a victory
-
-]
+> I find it unpleasant that you always bring your hobbyhorse in, but in an "abstract" way that doesn't allow discussing the actual object level question. It makes me feel attacked in a way that allows for no legal recourse to defend myself.
+
+I [replied](https://www.lesswrong.com/posts/WwTPSkNwC89g3Afnd/comment-section-from-05-19-2019?commentId=32GPaijsSwX2NSFJi) that that was understandable, but that I hoped it was also understandable that I found it unpleasant that our standard Bayesian philosophy of language somehow got politicized (!?), such that my attempts to do _correct epistemology_ were perceived as attacking people. Imagine living in a world where [posts about the minimum description length principle](https://www.lesswrong.com/posts/mB95aqTSJLNR9YyjH/message-length) were perceived as an attack on Christians—or if that analogy seemed loaded (because our subculture pattern matches atheism as "the good guys"), imagine some racist getting _really interested_ in the statistics of the normal distribution, and posting about the ratio of areas in the right tails of normal distributions with different means. I could see how that would be annoying—maybe even threatening—which would make it all the more satisfying if you could find a _mistake_ in the bastard's math. But if you _couldn't_ find a mistake—if, in fact, the post is on-topic for the forum and correct in the literal things that it literally says, then complaining about the author's motive for being interested in the normal distribution wouldn't seem like an obviously positive contribution to the discourse? I saw the problem, of course, and didn't mean to play dumb about it. But what, realistically, did Kosoy expect the atheist—or the racist, or me—to do?
+
+In a subthread in which I contested Kosoy's characterization of me as a "voice with an agenda which, if implemented, would put [her] in physical danger" ("I don't think of myself as having a lot of strong political beliefs," I said, "but I'm going to take a definite stand here: I am _against_ people being in physical danger"), Ben [pointed out that](https://www.lesswrong.com/posts/WwTPSkNwC89g3Afnd/comment-section-from-05-19-2019?commentId=TXbgr7goFtSAZEvZb)
+
+> Some speech acts lower the message length of proposals to attack some groups, or raise the message length of attempts to prevent such attacks. This is a kind of meta-attack or threat, like concentrating troops on a country's border.
+
+Norms discouraging "political" speech could aggravate the problem, if the response looked "political" but the original threat didn't. If Kosoy wanted to put in the work to explain why my philosophy of language blogging was causing problems for her, she would face legitimate doubt whether her defensive measures would be "admissible".
+
+The trainwreck got so bad that the mods manually [moved the comments to their own post](https://www.lesswrong.com/posts/WwTPSkNwC89g3Afnd/comment-section-from-05-19-2019). Based on the karma scores and what was said (Said Achmiz gave [a particularly helpful defense of disregarding community members' feelings](https://www.lesswrong.com/posts/WwTPSkNwC89g3Afnd/comment-section-from-05-19-2019?commentId=EsSdLMrFcCpSvr3pG)), I count it as a "victory" for me.
On 31 May 2019, a [draft of a new _Less Wrong_ FAQ](https://www.lesswrong.com/posts/MqrzczdGhQCRePgqN/feedback-requested-draft-of-a-new-about-welcome-page-for) included a link to "... Not Man for the Categories" as one of Scott Alexander's best essays. I argued that it would be better to cite _almost literally_ any other _Slate Star Codex_ post (most of which, I agreed, were exemplary). I claimed that the following disjunction was true: _either_ Alexander's claim that "There's no rule of rationality saying that [one] shouldn't" "accept an unexpected [X] or two deep inside the conceptual boundaries of what would normally be considered [Y] if it'll save someone's life" was a blatant lie, _or_ one had no grounds to criticize me for calling it a blatant lie, because there's no rule of rationality that says I shouldn't draw the category boundaries of "blatant lie" that way. The mod [was persuaded on reflection](https://www.lesswrong.com/posts/MqrzczdGhQCRePgqN/feedback-requested-draft-of-a-new-about-welcome-page-for?commentId=oBDjhXgY5XtugvtLT), and "... Not Man for the Categories" was not included in the final FAQ. Another "victory."
-But winning "victories" wasn't particularly comforting when I resented this becoming a political slapfight at all. I thought a lot of the objections I faced in the derailed "Possibility of No" thread were insane.
+But winning "victories" wasn't particularly comforting when I resented this becoming a political slapfight at all. I thought a lot of the objections I (and my "allies") faced in the derailed "Possibility of No" thread were insane.
I wrote to Anna and Steven Kaas (who I was trying to "recruit" onto our side of the civil war). In ["What You Can't Say"](http://www.paulgraham.com/say.html), Paul Graham had written, "The problem is, there are so many things you can't say. If you said them all you'd have no time left for your real work." But surely that depends on what _is_ one's real work. For someone like Paul Graham, whose goal was to make a lot of money writing software, "Don't say it" (except for this one meta-level essay) was probably the right choice. But someone whose goal is to improve our collective ability to reason, should probably be doing _more_ fighting than Paul Graham (although still preferably on the meta- rather than object-level), because political restrictions on speech and thought directly hurt the mission of "improving our collective ability to reason", in a way that they don't hurt the mission of "make a lot of money writing software."
Math and Wellness Month ended up being mostly a failure: the only math I ended up learning was [a fragment of group theory](http://zackmdavis.net/blog/2019/05/group-theory-for-wellness-i/), and [some probability/information theory](http://zackmdavis.net/blog/2019/05/the-typical-set/) that [later turned out to super-relevant to understanding sex differences](/2021/May/sexual-dimorphism-in-the-sequences-in-relation-to-my-gender-problems/#typical-point). So much for taking a break.
-[TODO:
- * I had posted a linkpost to "No, it's not The Incentives—it's You", which generated a lot of discussion, and Jessica (17 June) identified Ray's comments as the last straw.
+In June 2019, I made [a linkpost on _Less Wrong_](https://www.lesswrong.com/posts/5nH5Qtax9ae8CQjZ9/tal-yarkoni-no-it-s-not-the-incentives-it-s-you) to Tal Yarkoni's ["No, It's Not The Incentives—It's you"](https://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-incentives-its-you/), about how professional scientists should stop using career incentives as an excuse for doing poor science. It generated a lot of discussion.
+
+Looking over the thread in retrospect, [these words from David Xu seem significant](https://www.lesswrong.com/posts/5nH5Qtax9ae8CQjZ9/tal-yarkoni-no-it-s-not-the-incentives-it-s-you?commentId=qDPdneAQ4s7HMt3ys):
+
+> _We all know that falsifying data is bad._ But if that's the way the incentives point (and that's a very important if!), then it's _also_ bad to call people out for doing it. If you do that, then you're using moral indignation as a weapon—a way to not only coerce other people into using up their willpower, but to come out of it looking good yourself.
+
+In an email (Subject: "LessWrong.com is dead to me"), Jessica identified the thread as her last straw:
> LessWrong.com is a place where, if the value of truth conflicts with the value of protecting elites' feelings and covering their asses, the second value will win.
>
-> Trying to get LessWrong.com to adopt high-integrity norms is going to fail, hard, without a _lot_ of conflict. (Enforcing high-integrity norms is like violence; if it doesn't work, you're not doing enough of it).
-
- * posting on Less Wrong was harm-reduction; the only way to get people to stick up for truth would be to convert them to _a whole new worldview_; Jessica proposed the idea of a new discussion forum
- * Ben thought that trying to discuss with the other mods would be a good intermediate step, after we clarified to ourselves what was going on; talking to other mods might be "good practice in the same way that the Eliezer initiative was good practice"; Ben is less optimistic about harm reduction; "Drowning Children Are Rare" was barely net-upvoted, and participating was endorsing the karma and curation systems
- * David Xu's comment on "The Incentives" seems important?
- * secret posse member: Ray's attitude on "Is being good costly?"
- * Jessica: scortched-earth campaign should mostly be in meatspace social reality
- * my comment on emotive conjugation (https://www.lesswrong.com/posts/qaYeQnSYotCHQcPh8/drowning-children-are-rare#GaoyhEbzPJvv6sfZX)
-
-> I'm also not sure if I'm sufficiently clued in to what Ben and Jessica are modeling as Blight, a coherent problem, as opposed to two or six individual incidents that seem really egregious in a vaguely similar way that seems like it would have been less likely in 2009??
-
- * Vassar: "Literally nothing Ben is doing is as aggressive as the basic 101 pitch for EA."
- * Ben: we should be creating clarity about "position X is not a strawman within the group", rather than trying to scapegoat individuals
- * my scuffle with Ruby on "Causal vs. Social Reality" (my previous interaction with Ruby had been on the LW FAQ; maybe he couldn't let me "win" again so quickly?)
- * it gets worse: https://www.lesswrong.com/posts/xqAnKW46FqzPLnGmH/causal-reality-vs-social-reality#NbrPdyBFPi4hj5zQW
- * Ben's comment: "Wow, he's really overtly arguing that people should lie to him to protect his feelings."
- * Jessica: "tone arguments are always about privileged people protecting their feelings, and are thus in bad faith. Therefore, engaging with a tone argument as if it's in good faith is a fool's game, like playing chess with a pigeon. Either don't engage, or seek to embarrass them intentionally."
- * there's no point at being mad at MOPs
- * me (1 Jul): I'm a _little bit_ mad, because I specialize in cognitive and discourse strategies that are _extremely susceptible_ to being trolled like this
- * me to "Wilhelm" 1 Jul: "I'd rather not get into fights on LW, but at least I'm 2-0-1"
- * "collaborative truth seeking" but (as Michael pointed out) politeness looks nothing like Aumann agreement
- * 2 Jul: Jessica is surprised by how well "Self-consciousness wants to make everything about itself" worked; theory about people not wanting to be held to standards that others aren't being held to
- * Michael: Jessica's example made it clear she was on the side of social justice
- * secret posse member: level of social-justice talk makes me not want to interact with this post in any way
-]
+> Trying to get LessWrong.com to adopt high-integrity norms is going to fail, hard, without a _lot_ of conflict. (Enforcing high-integrity norms is like violence; if it doesn't work, you're not doing enough of it). People who think being exposed as fraudulent (or having their friends exposed as fraudulent) is a terrible outcome, are going to actively resist high-integrity discussion norms.
-On 4 July, Scott Alexander published ["Some Clarifications on Rationalist Blogging"](https://slatestarcodex.com/2019/07/04/some-clarifications-on-rationalist-blogging/), disclaiming any authority as a "rationalist" leader. ("I don't want to claim this blog is doing any kind of special 'rationality' work beyond showing people interesting problems [...] Insofar as [_Slate Star Codex_] makes any pretensions to being 'rationalist', it's a rationalist picnic and not a rationalist monastery.") I assumed this was inspired by Ben's request back in March that Scott "alter the beacon" so as to not confuse people about what the current-year community was. I appreciated it.
+Posting on _Less Wrong_ made sense as harm-reduction, but the only way to get people to stick up for truth would be to convert them to _a whole new worldview_, which would require a lot of in-person discussions. She bought up the idea of starting a new forum to replace _Less Wrong_.
-[TODO: "AI Timelines Scam"
- * I still sympathize with the "mainstream" pushback against the scam/fraud/&c. language being used to include Elephant-in-the-Brain-like distortions
- * Ben: "What exactly is a scam, if it's not misinforming people systematically about what you have to offer, in a direction that moves resources towards you? Investigations of financial fraud don't inquire as to the conscious motives of the perp."
- * 11 Jul: I think the law does count _mens rea_ as a thing: we do discriminate between vehicular manslaughter and first-degree murder, because traffic accidents are less disincentivizable than offing one's enemies
- * call with Michael about GiveWell vs. the Pope
-]
+Ben said that trying to discuss with the _Less Wrong_ mod team would be a good intermediate step, after we clarified to ourselves what was going on; it might be "good practice in the same way that the Eliezer initiative was good practice." The premise should be, "If this is within the Overton window for _Less Wrong_ moderators, there's a serious confusion on the conditions required for discourse", not on scapegoating individuals. He was less optimistic about harm-reduction; participating on the site was implicitly endorsing it by submitting the rule of the karma and curation systems.
-[TODO: secret thread with Ruby; "uh, guys??" to Steven and Anna; people say "Yes, of course criticism and truthseeking is important; I just think that tact is important, too," only to go on and dismiss any _particular_ criticism as insufficiently tactful.]
+"Trent" expressed sadness about how the discussion on "The Incentives" demonstrated that the community he loved—including dear friends—was in a very bad way. Michael (in a separate private discussion) had said he was glad to hear about the belief-update. "Trent" said that Michael saying that also made them sad, because it seemed discordant to be happy about sad news. Michael wrote (in the thread):
-[TODO: "progress towards discussing the real thing"
- * Jessica acks Ray's point of "why are you using court language if you don't intend to blame/punish"
- * Michael 20 Jul: court language is our way of saying non-engagement isn't an option
- * Michael: we need to get better at using SJW blamey language
- * secret posse member: that's you-have-become-the-abyss terrifying suggestion
- * Ben thinks SJW blame is obviously good
-]
+> I['m] sorry it made you sad. From my perspective, the question is no[t] "can we still be friends with such people", but "how can we still be friends with such people" and I am pretty certain that understanding their perspective if an important part of the answer. If clarity seems like death to them and like life to us, and we don't know this, IMHO that's an unpromising basis for friendship.
+
+------
+
+I got into a scuffle with Ruby (someone who had newly joined the _Less Wrong_ mod team) on his post on ["Causal Reality _vs_. Social Reality"](https://www.lesswrong.com/posts/xqAnKW46FqzPLnGmH/causal-reality-vs-social-reality). One section of the post asks, "Why people aren't clamoring in the streets for the end of sickness and death?" and gives the answer that it's because no one else is; people live in a social reality that accepts death as part of the natural order, even though life extension seems like it should be physically possible in causal reality.
+
+I didn't think this was a good example. "Clamoring in the streets" (even if you interpreted it as a metonym for other forms of mass political action) seemed like the kind of thing that would be recommended by social-reality thinking, rather than causal-reality thinking. How, causally, would the action of clamoring in the streets lead to the outcome of the end of sickness and death? I would expect means–end reasoning about causal reality to instead recommend things like working on or funding biomedical research.
+
+Ruby [complained that](https://www.lesswrong.com/posts/xqAnKW46FqzPLnGmH/causal-reality-vs-social-reality?commentId=7b2pWiCL33cqhTabg) my tone was too combative, and asked for more charity and collaborative truth-seeking[^collaborative-truth-seeking] in any future comments.
+
+[^collaborative-truth-seeking]: [No one ever seems to be able to explain to me what this phrase means.](https://www.lesswrong.com/posts/uvqd3YiBcrPxXzxQM/what-does-the-word-collaborative-mean-in-the-phrase)
+
+(My previous interaction with Ruby had been my challenge to "... Not Man for the Categories" appearing on the _Less Wrong_ FAQ. Maybe he couldn't let me "win" again so quickly?)
+
+I emailed the coordination group about it, on the grounds that gauging the psychology of the mod team was relevant to upcoming [Voice _vs._ Exit](https://en.wikipedia.org/wiki/Exit,_Voice,_and_Loyalty) choices:
+
+> he seems to be conflating transhumanist optimism with "causal reality", and then tone-policing me when I try to model good behavior of what means-end reasoning about causal reality actually looks like. This ... seems pretty cultish to me?? Like, it's fine and expected for this grade of confusion to be on the website, but it's more worrisome when it's coming from the mod team.[^rot-13]
+
+[^rot-13]: This part of the email was actually [rot-13'd](https://rot13.com) to let people write up their independent component without being contaminated by me; I reproduce the plaintext here.
+
+The meta-discussion on _Less Wrong_ started to get heated. Ruby claimed:
+
+> [I]f the goal is everyone being less wrong, I think some means of communicating are going to be more effective than others. I, at least, am a social monkey. If I am bluntly told I am wrong (even if I agree, even in private—but especially in public), I will feel attacked (if only at the S1 level), threatened (socially), and become defensive. It makes it hard to update and it makes it easy to dislike the one who called me out. [...]
+>
+> [...]
+>
+> Even if you wish to express that someone is wrong, I think this is done more effectively if one simultaneously continues to implicitly express "I think there is still some prior that you are correct and I curious to hear your thoughts", or failing that "You are very clearly wrong here yet I still respect you as a thinker who is worth my time to discourse with." [...] There's an icky thing here I feel like for there to be productive and healthy discussion you have to act as though at least one of the above statements is true, even if it isn't.
+
+"Wow, he's really overtly arguing that people should lie to him to protect his feelings," Ben commented via email.
+
+(I would later complain to Anna (Subject: "uh, guys???", 20 July 2019) that Ruby's profile said he was one of two people to have volunteered for CfAR on three continents. If this was the level of performance we could expect from _veteran_ CfAR participants, what was CfAR _for_?)
+
+[I replied to Ruby that](https://www.lesswrong.com/posts/xqAnKW46FqzPLnGmH/causal-reality-vs-social-reality?commentId=v3zh3KhKNTdMXWkJH) you could just directly respond to your interlocutor's arguments. Whether or not you respect them as a thinker is _off-topic_. "You said X, but this is wrong because of Y" isn't a personal attack!
+
+Jessica said that there's no point in getting mad at [MOPs](http://benjaminrosshoffman.com/construction-beacons/). I said I was a _little_ bit mad, because I specialized in discourse strategies that were susceptible to getting trolled like this. I thought it was ironic that this happened on a post that was _explicitly_ about causal _vs._ social reality; it's possible that I wouldn't be inclined to be such a hardass about "whether or not I respect you is off-topic" if it weren't for that prompt.
+
+Jessica ended up writing a post, ["Self-Consciousness Wants Everything to Be About Itself"](https://www.lesswrong.com/posts/bwkZD6uskCQBJDCeC/self-consciousness-wants-to-make-everything-about-itself), arguing that tone arguments are mainly about people silencing discussion of actual problems in order to protect their feelings, using as a central example a case study of a college official crying and saying that she "felt attacked" in response to complaints about her office being insufficiently supportive of a racial community.
+
+Jessica was surprised by how well it worked, judging by [Ruby mentioning silencing in an apology to me](https://www.lesswrong.com/posts/xqAnKW46FqzPLnGmH/causal-reality-vs-social-reality?commentId=wfzxj4GGRtZGMG9ni) (plausibly influenced by Jessica's post), and [an exchange between Raemon (also a mod) and Ruby that she thought was "surprisingly okay"](https://www.lesswrong.com/posts/bwkZD6uskCQBJDCeC/self-consciousness-wants-to-make-everything-about-itself?commentId=EW3Mom9qfoggfBicf).
+
+From this, Jessica derived the moral that when people are doing something that seems obviously terrible and in bad faith, it could help to publicly explain why the abstract thing is bad, without accusing anyone. This made sense because people didn't want to be held to standards that other people aren't being held to: a call-out directed at oneself personally could be selective enforcement, but a call-out of the abstract pattern invited changing one's behavior if the new equilibrium looked better.
+
+Michael said that part of the reason this worked was because it represented a clear threat to skapegoat, while also _not_ skapegoating, and not surrendering the option to do so later; it was significant that Jessica's choice of example positioned her on the side of the powerful social-justice coalition.
+
+"Trent" said that the amount of social-justice talk in the post rose to the level where he wouldn't dare criticize it or even mention it (!) in public, regardless of whether they agreed or disagreed.
+
+------
+
+On 4 July 2019, Scott Alexander published ["Some Clarifications on Rationalist Blogging"](https://slatestarcodex.com/2019/07/04/some-clarifications-on-rationalist-blogging/), disclaiming any authority as a "rationalist" leader. ("I don't want to claim this blog is doing any kind of special 'rationality' work beyond showing people interesting problems [...] Insofar as [_Slate Star Codex_] makes any pretensions to being 'rationalist', it's a rationalist picnic and not a rationalist monastery.") I assumed this was inspired by Ben's request back in March that Scott "alter the beacon" so as to not confuse people about what the current-year community was. I appreciated it.
+
+-------
+
+Also in early July 2019, Jessica published ["The AI Timelines Scam"](https://www.lesswrong.com/posts/KnQs55tjxWopCzKsk/the-ai-timelines-scam), arguing that the recent popularity of "short" (_e.g._, 2030) AI timelines was better explained by political factors, rather than any technical arguments: just as in previous decades, people had incentives to bluff and exaggerate about the imminence of AGI in order to attract resources to their own project.
+
+(Remember, this was 2019. After seeing what GPT-3/PaLM/DALL-E/_&c._ could do during the ["long May 2020"](https://twitter.com/MichaelTrazzi/status/1635871679133130752), it's now looking to me like the short-timelines people had better intuitions than Jessica gave them credit for.)
+
+I still sympathized with the "mainstream" pushback against using "scam"/"fraud"/"lie"/_&c._ language to include motivated [elephant-in-the-brain](https://en.wikipedia.org/wiki/The_Elephant_in_the_Brain)-like distortions. I conceded that this was a _boring_ semantic argument, but I feared that until we invented better linguistic technology, the _boring_ semantic argument was going to _continue_ sucking up discussion bandwidth with others when it didn't need to.
+
+"Am I being too tone-policey here?" I asked the coordination group. "Is it better if I explicitly disclaim, 'This is marketing advice; I'm not claiming to be making a substantive argument'?" (Subject: "Re: reception of 'The AI Timelines Scam' is better than expected!")
+
+Ben replied, "What exactly is a scam, if it's not misinforming people systematically about what you have to offer, in a direction that moves resources towards you? Investigations of financial fraud don't inquire as to the conscious motives of the perp."
+
+I replied that the concept of [_mens rea_](https://www.law.cornell.edu/wex/mens_rea) did seem necessary for maintaining good incentives, at least in some contexts. The law needs to distinguish between accidentally hitting a pedestrian in one's car ("manslaughter") from premeditated killing ("first-degree murder"), because traffic accidents are significantly less disincentivizable than offing one's enemies. (Anyone who drives at all is probably going to have unlucky analogues in nearby possible worlds who are guilty of vehicular manslaughter.) The manslaughter case was simpler than misinformation-that-moves-resources,[^manslaughter-disanalogy] and it might not be _easy_ for the court to determine "intent", but I didn't see what would reverse the extremely weak principle that intent matters, sometimes.
+
+[^manslaughter-disanalogy]: For one extremely important disanalogy, perps don't _gain_ from committing manslaughter.
+
+Ben replied that what mattered in the determination of manslaughter _vs._ murder was whether there was long-horizon optimization power towards the outcome of someone's death; it wasn't about what sentiments the killer rehearsed in their working memory.
+
+On a phone call, Michael made an analogy between EA and Catholicism. The Pope was fraudulent, because the legitimacy of the Pope's position (and his claims to power and resources) rested on the pretense that he had a direct relationship with God, which wasn't true, and the Pope knew (he _had_ to know, at some level) that it asn't true. (I agreed that this usage of _fraud_ made sense to me.) In Michael's view, Ben's charges against GiveWell were similar: GiveWell's legitimacy rested on the pretense that they were making decisions based on numbers, and they [had to know at some level](https://blog.givewell.org/2011/08/18/why-we-cant-take-expected-value-estimates-literally-even-when-theyre-unbiased/) that they weren't doing that.
+
+------
+
+Ruby wrote a document about ways in which one's speech could harm people, which was discussed in the comments of a draft _Less Wrong_ post by some of our posse members and some of the _Less Wrong_ mods. (The draft was hidden, but the API apparently didn't filter out comments on hidden posts, and the thread was visible on the third-party [GreaterWrong](https://www.greaterwrong.com/) site; I [filed a bug](https://github.com/LessWrong2/Lesswrong2/issues/2161).)
+
+Ben wrote:
+
+> What I see as under threat is the ability to say in a way that's actually heard, not only that opinion X is false, but that the process generating opinion X is untrustworthy, and perhaps actively optimizing in an objectionable direction. Frequently, attempts to say this are construed _primarily_ as moves to attack some person or institution, pushing them into the outgroup. Frequently, people suggest to me an "equivalent" wording with a softer tone, which in fact omits important substantive criticisms I mean to make, while claiming to understand what's at issue.
+
+Ray Arnold (another _Less Wrong_ mod) replied:
+
+> My core claim is: "right now, this isn't possible, without a) it being heard by many people as an attack, b) without people having to worry that other people will see it as an attack, even if they don't."
+>
+> It seems like you see this something as _"there's a precious thing that might be destroyed"_ and I see it as _"a precious thing does not exist and must be created, and the circumstances in which it can exist are fragile."_ It might have existed in the very early days of LessWrong. But the landscape now is very different than it was then. With billions of dollars available and at stake, what worked then can't be the same thing as what works now.
+
+(!!)[^what-works-now]
-[TODO: epistemic defense meeting;
- * I ended up crying at one point and left the room for while
- * Jessica's summary: "Zack was a helpful emotionally expressive and articulate victim. It seemed like there was consensus that "yeah, it would be better if people like Zack could be warned somehow that LW isn't doing the general sanity-maximization thing anymore"."
- * Vaniver admitting LW is more of a recruiting funnel for MIRI
- * I needed to exhaust all possible avenues of appeal before it became real to me; the first morning where "rationalists ... them" felt more natural than "rationalists ... us"
+[^what-works-now]: Arnold qualifies this in the next paragraph:
+
+ > [in public. In private things are much easier. It's _also_ the case that private channels enable collusion—that was an update [I]'ve made over the course of the conversation. ]
+
+ Even with the qualifier, I still think this deserves a "(!!)".
+
+Jessica pointed this out as a step towards discussing the real problem (Subject: "progress towards discussing the real thing??"). She elaborated in the secret thread: now that the "EA" scene was adjacent to real-world money and power, people were incentivized to protect their reputations (and beliefs related to their reputations) in anti-epistemic ways, in a way that they wouldn't if the scene was still just a philosophy club, catalyzing a shift of norms from "that which can be destroyed by the truth, should be" towards protecting feelings—where "protecting feelings" was actually about protecting power. The fact that the scene was allocating billions of dollars made it strictly _more_ important for public discussions to reach the truth, compared to philosophy club—but it also increased the likelihood of obfuscatory action, which philosophy-club norms (like "assume good faith") didn't account for. We might need to extend philosophy-club norms to take into account the possibility of adversarial action: there's a reason that courts of law don't assume good faith. We didn't want to disproportionately _punish_ people for getting caught up in obfuscatory patterns—that would just increase the incentive to obfuscate—but we did need some way to reveal what was going on.
+
+In email, Jessica acknowledged that Ray had a point that it was confusing to use court-inspired language if we didn't intend to blame and punish people. Michael said that court language was our way to communicate, "You don't have the option of non-engagement with the complaints that are being made." (Courts can _summon_ people; you can't ignore a court summons the way you can ignore ordinary critics.)
+
+Michael said that we should also develop skill in using social-justicey blame language, as was used against us, harder, while we were still acting under mistake-theoretic assumptions. "Trent" said that this was a terrifying you-have-become-the-abyss suggestion; Ben thought it was obviously a good idea.
+
+I was pretty horrified by the extent to which _Less Wrong_ moderators (!!) seemed to be explicitly defending "protect feelings" norms. Previously, I had mostly been seeing the present struggle through the lens of my idiosyncratic Something to Protect, as a simple matter of Bay Area political correctness; I was happy to have Michael/Ben/Jessica as allies, but I wasn't _seeing_ the Blight as a unified problem. Now ... I was seeing _something_.
+
+An in-person meeting was arranged on 23 July 2019 at the _Less Wrong_ office, with Ben, Jessica, me, and most of the _Less Wrong_ team (Ray, Ruby, Oliver Habryka, Vaniver, Jim Babcock). I don't have notes and don't really remember what was discussed in enough detail to recount it.[^memory] I ended up crying at one point and left the room for a while.
+
+[^memory]: An advantage of the important parts of my life taking place on the internet is that I have _logs_ of the important things; I'm only able to tell this Whole Dumb Story with as much fidelity as I am, because for most of it, I can go back and read the emails and chatlogs from the time. Now that [audio transcription has fallen to AI](https://openai.com/blog/whisper/), should I be recording more real-life conversations?? In this case of this meeting, supposedly one of the _Less Wrong_ guys was recording, but no one had it when I asked in October 2022.
+
+The next day, I asked Ben and Jessica for their takeaways via email (Subject: "peace talks outcome?"). Jessica said that I was a "helpful emotionally expressive and articulate victim", that there seemed to be a consensus that it would be better if people like me could be warned somehow that _Less Wrong_ wasn't doing the general sanity-maximization thing anymore. (Because community leaders were willing to sacrifice, for example, ability to discuss non-AI heresies, in order to focus on sanity about AI in particular while maintaining enough mainstream acceptability and power.)
+
+I said that for me and my selfish perspective, the main outcome was finally shattering my "rationalist" social identity. I needed to exhaust all possible avenues of appeal before it became real to me. The morning after was the first for which "rationalists ... them" felt more natural than "rationalists ... us".
+
+-------
+
+[TODO: Michael Vassar and the theory of optimal gossip
+ * Michael's "position" in the community, already not what it once was, was getting debased even further
+ * Berkeley REACH investigation; I asked what the charges were; they say it's not a court; I say _I_ don't feel safe in a place that bans Michael for no good reason
+ * Kelsey says in Discord that Michael is trying to get re-included
+ * I email her with quotes trying to argue that that's not his game
+ * She doesn't find the quotes convincing, because she thinks Vassar is a liar
+ * (sleep?!?)
+ * I call Ben to the stand, then Sarah ("You're a great prosecutor", I say in the channel)
+ * Kelsey in meatspace tells me that Vassar has a benefactor that she can't name; I'm retarded enough to not infer that it's him and she was obfuscating for compliance
+ ...
]
-[TODO: Michael Vassar and the theory of optimal gossip; make sure to include the part about Michael threatening to sue]
+-------
+
+[TODO: State of Steven; maybe fold in content from "uh, guys??" 21 July?]
-[TODO: State of Steven]
+-------
I still wanted to finish the memoir-post mourning the "rationalists", but I still felt psychologically constrained; I was still bound by internal silencing-chains. So instead, I mostly turned to a combination of writing bitter and insulting comments whenever I saw someone praise the "rationalists" collectively, and—more philosophy blogging!
In August 2019's ["Schelling Categories, and Simple Membership Tests"](https://www.lesswrong.com/posts/edEXi4SpkXfvaX42j/schelling-categories-and-simple-membership-tests), I explained a nuance that had only merited a passion mention in "... Boundaries?": sometimes you might want categories for different agents to _coordinate_ on, even at the cost of some statistical "fit." (This was of course generalized from a "pro-trans" argument that had occured to me, [that self-identity is an easy Schelling point when different people disagree about what "gender" they perceive someone as](/2019/Oct/self-identity-is-a-schelling-point/).)
-In September 2019's ["Heads I Win, Tails?—Never Heard of Her; Or, Selective Reporting and the Tragedy of the Green Rationalists"](https://www.lesswrong.com/posts/DoPo4PDjgSySquHX8/heads-i-win-tails-never-heard-of-her-or-selective-reporting), I presented a toy mathematical model of how censorship distorts group beliefs. I was surprised by how well-received it was (high karma, Curated within a few days, later included in the Best-of-2019 collection), especially given that it was explicitly about politics (albeit at a meta level, of course). Ben and Jessica had discouraged me from bothering when I sent them a draft.
+In September 2019's ["Heads I Win, Tails?—Never Heard of Her; Or, Selective Reporting and the Tragedy of the Green Rationalists"](https://www.lesswrong.com/posts/DoPo4PDjgSySquHX8/heads-i-win-tails-never-heard-of-her-or-selective-reporting), I presented a toy mathematical model of how censorship distorts group beliefs. I was surprised by how well-received it was (high karma, Curated within a few days, later included in the Best-of-2019 collection), especially given that it was explicitly about politics (albeit at a meta level, of course). Ben and Jessica had discouraged me from bothering when I sent them a draft. (Jessica said that it was obvious even to 10-year-olds that partisan politics distorts impressions by filtering evidence. "[D]o you think we could get a ten-year-old to explain it to Eliezer Yudkowsky?" I asked.)
In October 2019's ["Algorithms of Deception!"](https://www.lesswrong.com/posts/fmA2GJwZzYtkrAKYJ/algorithms-of-deception), I exhibited some toy Python code modeling different kinds of deception. A function that faithfully passes observations it sees as input to another function, lets the second function constructing a well-calibrated probability distribution. But if the first function outright fabricates evidence, or selectively omits some evidence, or gerrymanders the categories by which it interprets its observations as evidence, the second function comes up with a worse (less accurate) probability distribution.
-----
-While visiting Valinor on 7 August 2019, Merlin Blume (age 2¾ years) asked me, "Why are you a boy?"
+While visiting "Arcadia" on 7 August 2019, Mike and "Susan"'s son (age 2¾ years) asked me, "Why are you a boy?"
After a long pause, I said, "Yes," as if I had misheard the question as "Are you a boy?" I think it was a motivated mishearing: it was only after I answered that I consciously realized that's not what the kid asked.
Scott (and Yudkowsky and Anna and the rest of the Caliphate) seemed to accept this as an inevitable background fact of existence, like the weather. But I saw a Schelling point off in the distance where us witches stick together for Free Speech, and it was _awfully_ tempting to try to jump there. (Of course, it would be _better_ if there was a way to organize just the good witches, and exclude all the Actually Bad witches, but the [Sorites problem](https://plato.stanford.edu/entries/sorites-paradox/) on witch Badness made that hard to organize without falling back to the falling back to the one-heresy-per-thinker equilibrium.)
-Jessica thought my use of "heresy" was conflating factual beliefs with political movements. (There are no intrinsically "right wing" _facts_.) I agreed that conflating political positions with facts would be bad (and that it would be bad if I were doing that without "intending" to). I wasn't interested in defending the "alt-right" (whatever that means) broadly. But I had _learned stuff_ from reading far-right authors (most notably Moldbug), and from talking with "Wilhelm". I was starting to appreciate [what Michael had said about "Less precise is more violent" back in April](#less-precise-is-more-violent) (when I was talking about criticizing "rationalists").
+Jessica thought my use of "heresy" was conflating factual beliefs with political movements. (There are no intrinsically "right wing" _facts_.) I agreed that conflating political positions with facts would be bad (and that it would be bad if I were doing that without "intending" to). I wasn't interested in defending the "alt-right" (whatever that means) broadly. But I had _learned stuff_ from reading far-right authors (most notably Moldbug), and from talking with "Thomas". I was starting to appreciate [what Michael had said about "Less precise is more violent" back in April](#less-precise-is-more-violent) (when I was talking about criticizing "rationalists").
Jessica asked if my opinion would change depending on whether Yudkowsky thought neoreaction was intellectually worth engaging with. (Yudkowsky [had said years ago](https://www.lesswrong.com/posts/6qPextf9KyWLFJ53j/why-is-mencius-moldbug-so-popular-on-less-wrong-answer-he-s?commentId=TcLhiMk8BTp4vN3Zs) that Moldbug was low quality.)
On 3 November 2019, I received an interesting reply on my philosophy-of-categorization thesis from MIRI researcher Abram Demski. Abram asked: ideally, shouldn't all conceptual boundaries be drawn with appeal-to-consequences? Wasn't the problem just with bad (motivated, shortsighted) appeals to consequences? Agents categorize in order to make decisions. The best classifer for an application depends on the costs and benefits. As a classic example, it's very important for evolved prey animals to avoid predators, so it makes sense for their predator-detection classifiers to be configured such that they jump away from every rustling in the bushes, even if it's usually not a predator.
-I had thought of the "false-positives are better than false-negatives when detecting predators" example as being about the limitations of evolution as an AI designer: messy evolved animal brains don't bother to track probability and utility separately the way a cleanly-designed AI could. As I had explained in "... Boundaries?", it made sense for _what_ variables you paid attention to, to be motivated by consequences. But _given_ the subspace that's relevant to your interests, you want to run an epistemically legitimate clustering algorithm on the data you see there, which depends on the data, not your values. The only reason value-dependent gerrymandered category boundaries seem like a good idea if you're not careful about philosophy is because it's _wireheading_. Ideal probabilistic beliefs shouldn't depend on consequences.
+I had thought of the "false-positives are better than false-negatives when detecting predators" example as being about the limitations of evolution as an AI designer: messy evolved animal brains don't bother to track probability and utility separately the way a cleanly-designed AI could. As I had explained in "... Boundaries?", it made sense for _what_ variables you paid attention to, to be motivated by consequences. But _given_ the subspace that's relevant to your interests, you want to run an "epistemically legitimate" clustering algorithm on the data you see there, which depends on the data, not your values. The only reason value-dependent gerrymandered category boundaries seem like a good idea if you're not careful about philosophy is because it's _wireheading_. Ideal probabilistic beliefs shouldn't depend on consequences.
-Abram didn't think the issue was so clear-cut. Where do "probabilities" come from, in the first place? The reason we expect something like Bayesianism to be an attractor among self-improving agents is _because_ probabilistic reasoning is broadly useful: epistemology can be _derived_ from instrumental concerns. He agreed that severe wireheading issues _potentially_ arise if you allow consequentialist concerns to affect your epistemics.
+Abram didn't think the issue was so clear-cut. Where do "probabilities" come from, in the first place? The reason we expect something like Bayesianism to be an attractor among self-improving agents is because probabilistic reasoning is broadly useful: epistemology can be _derived_ from instrumental concerns. He agreed that severe wireheading issues _potentially_ arise if you allow consequentialist concerns to affect your epistemics.
But the alternative view had its own problems. If your AI consists of a consequentialist module that optimizes for utility in the world, and an epistemic module that optimizes for the accuracy of its beliefs, that's _two_ agents, not one: how could that be reflectively coherent? You could, perhaps, bite the bullet here, for fear that consequentialism doesn't tile and that wireheading was inevitable. On this view, Abram explained, "Agency is an illusion which can only be maintained by crippling agents and giving them a split-brain architecture where an instrumental task-monkey does all the important stuff while an epistemic overseer supervises." Whether this view was ultimately tenable or not, this did show that trying to forbid appeals-to-consequences entirely led to strange places.
-I didn't immediately have an answer for Abram, but I was grateful for the engagement. (Abram was clearly addressing the real philosophical issues, and not just trying to mess with me the way almost everyone else in Berkeley was trying to mess with me.)
+I didn't immediately have an answer for Abram, but I was grateful for the engagement. (Abram was clearly addressing the real philosophical issues, and not just trying to mess with me in the sense that almost everyone else in Berkeley was trying to mess with me.)
-Also in November 2019, I wrote to Ben about how I was still stuck on writing the grief-memoir. My _plan_ had been that it should have been possibly to tell the story of the Category War while glomarizing about the content of private conversations, then offer Scott and Eliezer pre-publication right of reply (because it's only fair to give your former-hero-current-[frenemies](https://en.wikipedia.org/wiki/Frenemy) warning when you're about to publicly call them intellectually dishonest), then share it to _Less Wrong_ and the /r/TheMotte culture war thread, and then I would have the emotional closure to move on with my life (learn math, go to gym, chop wood, carry water) and not be a mentally-dominated cultist.
+------
+
+Also in November 2019, I wrote to Ben about how I was still stuck on writing the grief-memoir. My _plan_ had been that it should have been possible to tell the story of the Category War while glomarizing about the content of private conversations, then offer Scott and Eliezer pre-publication right of reply (because it's only fair to give your former-hero-current-[frenemies](https://en.wikipedia.org/wiki/Frenemy) warning when you're about to publicly call them intellectually dishonest), then share it to _Less Wrong_ and the /r/TheMotte culture war thread, and then I would have the emotional closure to move on with my life (learn math, go to gym, chop wood, carry water) and not be a mentally-dominated cultist.
-The reason it _should_ have been safe to write was because Explaining Things is Good. It should be possible to say, "This is not a social attack; I'm not saying 'rationalists Bad, Yudkowsky Bad'; I'm just trying to carefully _tell the true story_ about why, as a matter of cause-and-effect, I've been upset this year, including addressing counterarguments for why some would argue that I shouldn't be upset, why other people could be said to be behaving 'reasonably' given their incentives, why I nevertheless wish they'd be braver and adhere to principle rather than 'reasonably' following incentives, _&c_."
+The reason it _should_ have been safe to write was because Explaining Things Is Good. It should be possible to say, "This is not a social attack; I'm not saying 'rationalists Bad, Yudkowsky Bad'; I'm just trying to carefully _tell the true story_ about why, as a matter of cause-and-effect, I've been upset this year, including addressing counterarguments for why some would argue that I shouldn't be upset, why other people could be said to be behaving 'reasonably' given their incentives, why I nevertheless wish they'd be braver and adhere to principle rather than 'reasonably' following incentives, _&c_."
So why couldn't I write? Was it that I didn't know how to make "This is not a social attack" credible? Maybe because ... it wasn't true?? I was afraid that telling a story about our leader being intellectually dishonest was "the nuclear option" in a way that I couldn't credibly cancel with "But I'm just telling a true story about a thing that was important to me that actually happened" disclaimers. If you're slowly-but-surely gaining territory in a conventional war, _suddenly_ escalating to nukes seems pointlessly destructive. This metaphor is horribly non-normative ([arguing is not a punishment!](https://srconstantin.github.io/2018/12/15/argue-politics-with-your-best-friends.html) carefully telling a true story _about_ an argument is not a nuke!), but I didn't know how to make it stably go away.
I said I would bite that bullet: yes! Yes, I was trying to figure out whether I was being fair to my abusers, and it was an important question to get right! "Other people's lack of standards harmed me, therefore I don't need to hold myself to standards in my response because I have [extenuating circumstances](https://www.lesswrong.com/posts/XYrcTJFJoYKX2DxNL/extenuating-circumstances)" would be a _lame excuse_.
-(This seemed correlated with the recurring stalemated disagreement within our coordination group, where Michael/Ben/Jessica would say, "Fraud, if that word _ever_ meant anything", and while I agreed that they were pointing to an important way in which things were messed up, I was still sympathetic to the Caliphate-defender's reply that the Vassarite usage of "fraud" was motte-and-baileying between vastly different senses of _fraud_; I wanted to do _more work_ to formulate a _more precise theory_ of the psychology of deception to describe exactly how things are messed up a way that wouldn't be susceptible to the motte-and-bailey charge.)
+(This seemed correlated with the recurring stalemated disagreement within our coordination group, where Michael/Ben/Jessica would say, "Fraud, if that word _ever_ meant anything", and while I agreed that they were pointing to an important way in which things were messed up, I was still sympathetic to the Caliphate-defender's reply that this usage of "fraud" was motte-and-baileying between vastly different senses of _fraud_; I wanted to do _more work_ to formulate a _more precise theory_ of the psychology of deception to describe exactly how things are messed up a way that wouldn't be susceptible to the motte-and-bailey charge.)
+
+-------
+
+[TODO: Ziz's protest]
+
+--------
+
+I had an interesting interaction with Somni, one of the "Meeker Four"—presumably out on bail at this time?—on 12 December 2019.
+
+I told her, from a certain perspective, it's surprising that you spend so much time complaining about CfAR, Anna Salamon, Kelsey Piper, _&c._, but _I_ seemed to get along fine with her—because "naïvely", one would think that my views were _so much worse_. Was I getting a pity pass because she thought false consciousness was causing me to act against my own transfem class interests? Or what?
-[TODO: Ziz's protest;
+To be absolutely clear about my terrible views, I said that I was privately modeling a lot of transmisogyny complaints as something like—a certain neurotype-cluster of non-dominant male is latching onto locally-ascendant social-justice ideology in which claims to victimhood can be leveraged into claims to power. Traditionally, men are moral agents, but not patients; women are moral patients, but not agents. If weird non-dominant men aren't respected if identified as such (because low-ranking males aren't valuable allies, and don't have intrinsic moral patiency of women), but _can_ get victimhood/moral-patiency points for identifying as oppressed transfems, that creates an incentive gradient for them to do so, and no one was allowed to notice this except me, because everyone prefers to stay on the good side of social-justice ideology unless they have Something to Protect that requires defying it.
-[TODO: Somni? ("peek behind the fog of war" 6 Feb)]
+Somni said that it was because I was being victimized by the same forces of gaslighting, and that I wasn't lying about my agenda. Maybe she _should_ be complaining about me?—but I seemed to be following a somewhat earnest epistemic process, whereas Kelsey, Scott, and Anna were not. If I were to start going, "Here's my rationality org; rule #1: no transfems (except me); rule #2, no telling people about rule #1", then she would talk about it.
-[TODO: rude maps]
+I would later remark to Anna (Subject: "a peek behind the fog of war; and, [redacted]", 6 February 2020) that Somni and Ziz saw themselves as being oppressed by people's hypocritical and manipulative social perceptions and behavior.
-[TODO: a culture that has gone off the rails; my warning points to Vaniver]
+Merely using the appropriate language ("Somni ... she", _&c._) protected her against threats from the Political Correctness police, but it actually didn't protect against threats from _them_. It was as if the mere fact that I wasn't optimizing for PR (lying about my agenda, as Somni said) was what made me not a direct enemy (although still a collaborator) in their eyes.
+
+--------
+
+[TODO: a culture that has gone off the rails; my warning points to Vaniver; discussion of Vassar on "ialdabaoth is banned"]
+
+--------
[TODO: complicity and friendship]
-[TODO: I had a productive winter blogging vacation in December 2019
-pull the trigger on "On the Argumentative Form"; I was worried about leaking info from private conversations, but I'm in the clear "That's your hobbyhorse" is an observation anyone could make from content alone]
+--------
+
+I had a pretty productive blogging spree in December 2019. In addition to a number of [more](/2019/Dec/political-science-epigrams/) [minor](/2019/Dec/the-strategy-of-stigmatization/) [posts](/2019/Dec/i-want-to-be-the-one/) [on](/2019/Dec/promises-i-can-keep/) [this](/2019/Dec/comp/) [blog](/2019/Dec/more-schelling/) [and](https://www.lesswrong.com/posts/XbXJZjwinkoQXu4db/funk-tunul-s-legacy-or-the-legend-of-the-extortion-war) [on](https://www.lesswrong.com/posts/y4bkJTtG3s5d6v36k/stupidity-and-dishonesty-explain-each-other-away) _[Less](https://www.lesswrong.com/posts/tCwresAuSvk867rzH/speaking-truth-to-power-is-a-schelling-point) [Wrong](https://www.lesswrong.com/posts/jrLkMFd88b4FRMwC6/don-t-double-crux-with-suicide-rock)_, I also got out some more significant posts bearing on my agenda.
+
+On this blog, in ["Reply to Ozymandias on Fully Consensual Gender"](/2019/Dec/reply-to-ozymandias-on-fully-consensual-gender/), I finally got out at least a partial reply to [Ozy's June 2018 reply](https://thingofthings.wordpress.com/2018/06/18/man-should-allocate-some-more-categories/) to ["The Categories Were Made for Man to Make Predictions"](/2018/Feb/the-categories-were-made-for-man-to-make-predictions/), affirming the relevance of an analogy Ozy had made between the socially-constructed natures of money and social gender, while denying that the analogy supported gender by self-identification. (I had been [working on a more exhaustive reply](/2018/Dec/untitled-metablogging-26-december-2018/#reply-to-ozy), but hadn't managed to finish whittling it into a shape that I was totally happy with.)
+
+I also polished and pulled the trigger on ["On the Argumentative Form 'Super-Proton Things Tend to Come In Varieties'"](/2019/Dec/on-the-argumentative-form-super-proton-things-tend-to-come-in-varieties/), my reply to Yudkowsky's implicit political concession to me back in March. I had been reluctant to post it based on an intuition of, "My childhood hero was trying to _do me a favor_; it would be a betrayal to reject the gift." The post itself explained why that intuition was crazy, but _that_ just brought up more anxieties about whether the explanation constituted leaking information from private conversations—but I had chosen my words carefully such that it wasn't. ("Even if Yudkowsky doesn't know you exist [...] he's _effectively_ doing your cause a favor" was something I could have plausibly written in the possible world where the antecedent was true.) Jessica said the post seemed good.
+
+On _Less Wrong_, the mods had just announced [a new end-of-year Review event](https://www.lesswrong.com/posts/qXwmMkEBLL59NkvYR/the-lesswrong-2018-review), in which the best post from the year before would be reviewed and voted on, to see which had stood the test of time and deserved to be part of our canon of cumulative knowledge. (That is, this Review period starting in late 2019 would cover posts published in _2018_.)
-[TODO: "Firming Up ..." Dec 2019: combatting Yudkowsky's not-technically-lying shenanigans]
+This provided me with [an affordance](https://www.lesswrong.com/posts/qXwmMkEBLL59NkvYR/the-lesswrong-2018-review?commentId=d4RrEizzH85BdCPhE) to write some "defensive"[^defensive] posts, critiquing posts that had been nominated for the Best-of-2018 collection that I didn't think deserved such glory. In response to ["Decoupling _vs._ Contextualizing Norms"](https://www.lesswrong.com/posts/7cAsBPGh98pGyrhz9/decoupling-vs-contextualising-norms) (which had been [cited in a way that I thought obfuscatory during the "Yes Implies the Possibility of No" trainwreck](https://www.greaterwrong.com/posts/WwTPSkNwC89g3Afnd/comment-section-from-05-19-2019/comment/wejvnw6QnWrvbjgns)), I wrote ["Relevance Norms; Or, Grecian Implicature Queers the Decoupling/Contextualizing Binary"](https://www.lesswrong.com/posts/GSz8SrKFfW7fJK2wN/relevance-norms-or-gricean-implicature-queers-the-decoupling), appealing to our [academically standard theory of how context affects meaning](https://plato.stanford.edu/entries/implicature/) to explain why "decoupling _vs._ contextualizing norms" is a false dichotomy.
+
+[^defensive]: Criticism is "defensive" in the sense of trying to _prevent_ new beliefs from being added to our shared map; a critic of an idea "wins" when the idea is not accepted (such that the set of accepted beliefs remains at the _status quo ante_).
+
+More significantly, in reaction to Yudkowsky's ["Meta-Honesty: Firming Up Honesty Around Its Edge Cases"](https://www.lesswrong.com/posts/xdwbX9pFEr7Pomaxv/meta-honesty-firming-up-honesty-around-its-edge-cases), I published ["Firming Up Not-Lying Around Its Edge-Cases Is Less Broadly Useful Than One Might Initially Think"](https://www.lesswrong.com/posts/MN4NRkMw7ggt9587K/firming-up-not-lying-around-its-edge-cases-is-less-broadly), explaining why merely refraining from making false statments is an unproductively narrow sense of "honesty", because the ambiguity of natural language makes it easy to deceive people in practice without technically lying. (The ungainly title of my post was "softened" from an earlier draft following feedback from the posse; I had originally written "... Surprisingly Useless".)
+
+I thought this one cut to the heart of the shocking behavior that we had seen from Yudkowsky lately. (Less shocking as the months rolled on, and I told myself to let the story end.) The "hill of meaning in defense of validity" affair had been been driven by Yudkowsky's pathological obsession with not-technically-lying, on two levels: he had proclaimed that asking for new pronouns "Is. Not. Lying." (as if _that_ were the matter that anyone cared about—as if conservatives and gender-critical feminists should just pack up and go home after it had been demonstrated that trans people aren't _lying_), and he had seen no interest in clarifying his position on the philosophy of language, because he wasn't lying when he said that preferred pronouns weren't lies (as if _that_ were the matter that my posse cared about—as if I should keep honoring him as my Caliph after it had been demonstrated that he hadn't _lied_). But his Sequences had articulated a _higher standard_ than merely not-lying. If he didn't remember, I could at least hope to remind everyone else.
+
+I also wrote a little post on 20 December 2019, ["Free Speech and Triskadekaphobic Calculators: A Reply to Hubinger on the Relevance of Public Online Discussion to Existential Risk"](https://www.lesswrong.com/posts/yaCwW8nPQeJknbCgf/free-speech-and-triskaidekaphobic-calculators-a-reply-to).
+
+Wei Dai had written ["Against Premature Abstraction of Political Issues"](https://www.lesswrong.com/posts/bFv8soRx6HB94p5Pg/against-premature-abstraction-of-political-issues)—itself plausibly an abstraction inspired by my philosophy-of-language blogging?—and had cited a clump of _Less Wrong_ posts about gender and pick-up artistry back in 'aught-nine as a successful debate that would have been harder to have if everyone had to obsfuscate the concrete topics of interest.
+
+A MIRI researcher, Evan Hubinger, asked:
+
+> Do you think having that debate online was something that needed to happen for AI safety/x-risk? Do you think it benefited AI safety at all? I'm genuinely curious. My bet would be the opposite—that it caused AI safety to be more associated with political drama that helped further taint it.
+
+In my reply post, I claimed that our belief that AI safety was the most important problem in the world was causally downstream from from people like Yudkowsky and Nick Bostrom trying to do good reasoning, and following lines of reasoning to where they led. The [cognitive algorithm](https://www.lesswrong.com/posts/HcCpvYLoSFP4iAqSz/rationality-appreciating-cognitive-algorithms) of assuming that your current agenda was the most important thing, and then distorting the process of inquiry to preserve its political untaintedness wouldn't have led us to _noticing_ the alignment problem, and I didn't think it would be sufficient to solve it.
+
+In some sense, it should be easier to have a rationality/alignment community that _just_ does systematically correct reasoning, rather than a politically-savvy community that does systematically correct reasoning _except_ when that would taint AI safety with political drama, analogously to how it's easier to build a calculator that just does correct arithmetic, than a calculator that does correct arithmetic _except_ that it never displays the result 13.
+
+In order to build a "triskadekaphobic calculator", you would need to "solve arithmetic" anyway, and the resulting product would be limited not only in its ability to correctly compute `6 + 7`, but also the infinite family of calculations that included 13 as an intermediate result: if you can't count on `(6 + 7) + 1` being the same as `6 + (7 + 1)`, you lose the associativity of addition. And so on. (I had the "calculator that won't display 13" analogy cached from previous email correspondence.)
+
+It could have been a comment instead of a top-level post, but I wanted to bid for the extra attention. I think, at some level, putting Hubinger's name in the post title was deliberate. It wasn't inappropriate—"Reply to Author's Name on Topic Name" is a very standard academic title format, [which](/2016/Oct/reply-to-ozy-on-agp/) [I](/2016/Nov/reply-to-ozy-on-two-type-mtf-taxonomy/) [often](/2019/Dec/reply-to-ozymandias-on-fully-consensual-gender/) [use](/2018/Apr/reply-to-the-unit-of-caring-on-adult-human-females/) [myself](https://www.lesswrong.com/posts/aJnaMv8pFQAfi9jBm/reply-to-nate-soares-on-dolphins)—but it also wasn't necessary, and might have been a little weird given that I was mostly using Hubinger's comment as a jumping-off point for my Free Speech for Shared Maps campaign, rather than responding point-by-point to a longer piece Hubinger might have written. It's as if the part of my brain that chose that subtitle wanted to set an example, that arguing for cowardice, being in favor of concealing information for fear of being singled out by a mob, would just get you singled out _more_.
+
+I had [an exchange with Scott Alexander in the comment section](https://www.greaterwrong.com/posts/yaCwW8nPQeJknbCgf/free-speech-and-triskaidekaphobic-calculators-a-reply-to/comment/JdsknCuCuZMAo8EbP).
+
+"I know a bunch of people in academia who do various verbal gymnastics to appease the triskaidekaphobics, and when you talk to them in private they get everything 100% right," [he said](https://www.lesswrong.com/posts/yaCwW8nPQeJknbCgf/free-speech-and-triskaidekaphobic-calculators-a-reply-to?commentId=mHrHTvzg8MGNH2CwB) (in a follow-up comment on 5 January 2020).
+
+I'm happy for them, I replied, but I thought the _point_ of having taxpayer-funded academic departments was so that people who _aren't_ insider experts can have accurate information with which to inform decisions?
+
+-----
[TODO: plan to reach out to Rick 14 December
Anna's reply 21 December
Michael asks for me to acknowledge that my sense of opportunity is driven by politics
discussion of what is "political"
mention to Anna that I was postponing in order to make it non-salesy
-
+January 2020 "my state"?
]
------
Or, I pointed out, (c) I had ceded the territory of the interior of my own mind _to Eliezer Yudkowsky in particular_, and while I had made a lot of progress unwinding this, I was still, still not done, and seeing him at the Newtonmas party set me back a bit.
-Secret posse member reassured me that finishing the memoir privately would be clarifying and cathartic _for me_. If people in the Caliphate came to their senses, I could either not publish it, or give it a happy ending where everyone comes to their senses.
+"Trent" reassured me that finishing the memoir privately would be clarifying and cathartic _for me_. If people in the Caliphate came to their senses, I could either not publish it, or give it a happy ending where everyone comes to their senses.
(It does not, actually, have a happy ending where everyone comes to their senses.)
------
-While visiting Valinor on 4 February 2020, I remember my nose dripping while I was holding Koios, the baby. Alicorn offered me a tissue. I asked if I shouldn't be holding the baby while my nose was dripping and I therefore plausibly had a cold. She said it was fine. On the topic of possible sickness, I said that I hoped the novel coronavirus people were talking about didn't go pandemic.
+While visiting "Arcadia" on 4 February 2020, I remember my nose dripping while I was holding the baby. Alicorn offered me a tissue. I asked if I shouldn't be holding the baby while my nose was dripping and I therefore plausibly had a cold. She said it was fine. On the topic of possible sickness, I said that I hoped the novel coronavirus people were talking about didn't go pandemic.
It did. The Berkeley rats took social distancing guidelines very seriously, so it would be a while before I could visit again.
On 10 February 2020, Scott Alexander published ["Autogenderphilia Is Common and Not Especially Related to Transgender"](https://slatestarcodex.com/2020/02/10/autogenderphilia-is-common-and-not-especially-related-to-transgender/), an analysis of the results of the autogynephilia/autoandrophilia questions on the recent _Slate Star Codex_ survey.
-I appreciated the gesture of getting real data, but I was deeply unimpressed with Alexander's analysis for reasons that I found difficult to write up in a timely manner. Three years later, I eventually got around to [polishing my draft and throwing it up as a standalone post](/2023/Feb/reply-to-scott-alexander-on-autogenderphilia/), rather than cluttering the present narrative with my explanation.
+I appreciated the gesture of getting real data, but I was deeply unimpressed with Alexander's analysis for reasons that I found difficult to write up in a timely manner. Three years later, I eventually got around to [polishing my draft and throwing it up as a standalone post](/2023/Mar/reply-to-scott-alexander-on-autogenderphilia/), rather than cluttering the present narrative with my explanation.
Briefly, based on eyballing the survey data, Alexander proposes "if you identify as a gender, and you're attracted to that gender, it's a natural leap to be attracted to yourself being that gender" as a "very boring" theory, but on my worldview, a hypothesis that puts "gay people (cis and trans)" in the antecedent is _not_ boring and actually takes on a big complexity penalty: I just don't think the group of gay men _and_ lesbians _and_ straight males with female gender identities _and_ straight females with male gender identities have much in common with each other, except sociologically (being "queer"), and by being human.
-(I do like the hypernym _autogenderphilia_.)
+(I do like the [hypernym](https://en.wikipedia.org/wiki/Hyponymy_and_hypernymy) _autogenderphilia_.)
-------
But if you _do_ have the math, a moment of introspection will convince you that the analogy between category "boundaries" and national borders is not a particularly deep or informative one.
-A two-dimensional political map tells you which areas of the Earth's surface are under the jurisdiction of what government.
-
-In contrast, category "boundaries" tell you which regions of very high-dimensional configuration space correspond to a word/concept, which is useful _because_ that structure is useful for making probabilistic inferences: you can use your observastions of some aspects of an entity (some of the coordinates of a point in configuration space) to infer category-membership, and then use category membership to make predictions about aspects that you haven't yet observed.
+A two-dimensional political map tells you which areas of the Earth's surface are under the jurisdiction of what government. In contrast, category "boundaries" tell you which regions of very high-dimensional configuration space correspond to a word/concept, which is useful _because_ that structure is useful for making probabilistic inferences: you can use your observastions of some aspects of an entity (some of the coordinates of a point in configuration space) to infer category-membership, and then use category membership to make predictions about aspects that you haven't yet observed.
But the trick only works to the extent that the category is a regular, non-squiggly region of configuration space: if you know that egg-shaped objects tend to be blue, and you see a black-and-white photo of an egg-shaped object, you can get _close_ to picking out its color on a color wheel. But if egg-shaped objects tend to blue _or_ green _or_ red _or_ gray, you wouldn't know where to point to on the color wheel.
That was my intuition. To formalize it, I wanted some sensible numerical quantity that would be maximized by using "nice" categories and get trashed by gerrymandering. [Mutual information](https://en.wikipedia.org/wiki/Mutual_information) was the obvious first guess, but that wasn't it, because mutual information lacks a "topology", a notion of _closeness_ that made some false predictions better than others by virtue of being "close".
-Suppose the outcome space of _X_ is `{H, T}` and the outcome space of _Y_ is `{1, 2, 3, 4, 5, 6, 7, 8}`. I _wanted_ to say that if observing _X_=`H` concentrates _Y_'s probability mass on `{1, 2, 3}`, that's _more useful_ than if it concentrates _Y_ on `{1, 5, 8}`—but that would require the numbers in Y to be _numbers_ rather than opaque labels; as far as elementary information theory was concerned, mapping eight states to three states reduced the entropy from lg 8 = 3 to lg 3 ≈ 1.58 no matter "which" three states they were.
+Suppose the outcome space of _X_ is `{H, T}` and the outcome space of _Y_ is `{1, 2, 3, 4, 5, 6, 7, 8}`. I _wanted_ to say that if observing _X_=`H` concentrates _Y_'s probability mass on `{1, 2, 3}`, that's _more useful_ than if it concentrates _Y_ on `{1, 5, 8}`—but that would require the numbers in Y to be _numbers_ rather than opaque labels; as far as elementary information theory was concerned, mapping eight states to three states reduced the entropy from log<sub>2</sub> 8 = 3 to log<sub>2</sub> 3 ≈ 1.58 no matter "which" three states they were.
How could I make this rigorous? Did I want to be talking about the _variance_ of my features conditional on category-membership? Was "connectedness" intrinsically the what I wanted, or was connectedness only important because it cut down the number of possibilities? (There are 8!/(6!2!) = 28 ways to choose two elements from `{1..8}`, but only 7 ways to choose two contiguous elements.) I thought connectedness _was_ intrinsically important, because we didn't just want _few_ things, we wanted things that are _similar enough to make similar decisions about_.
[TODO:
* Yudkowsky made a stray remark about social media causing people to say crazy thing
* I got enraged, posted a couple Tweets, including a preview of "Unnatural Categories"
+
+https://twitter.com/ESYudkowsky/status/1304824253015945216
+> Hypothesis: social media has an effect promoting Terrible Straw Arguments to being used by many actual people. One crazy on Side A makes a bad argument. Side B subtweets with a refutation and that gets a million views. So people on Side A hear about it as Side A's argument.
+>
+> And the result looks so incredibly surreal. People be going "2 + 3 = 4" and the replies are things like "lol but 4 is a prime number you moron". The entire conversation has been implicitly selected to sound insane to the other side.
+
+
+> Who would have thought getting @ESYudkowsky's robot cult to stop trying to trick me into cutting my dick off (independently of the empirical facts determining whether or not I should cut my dick off) would involve so much math?? OK, I guess the math part isn't surprising, but—
+https://twitter.com/zackmdavis/status/1304838346695348224
+
+
+https://twitter.com/zackmdavis/status/1304838486810193921
+> Or, "You're not standing in defense of truth if you insist on a symbol like '2' or '+', brought explicitly into question, being used with some particular meaning." (I'm still very angry at you! This shouldn't be hard to clear up in public!) 1/2
+>
+> If not the object-level application on Twitter, at least the abstract philosophy-of-language issue on Less Wrong?! Please!?! I've been working on a more detailed explanation that should be ready within a few weeks. 2/2
+
* something in my boiled over, and I eventually ended up staying up late writing an angry email
* this ruins my chances for being a "neutral" bridge between the Vassarites and the Caliphate, but that's OK
* mostly reproduce below (with a few redactions for either brevity or compliance with privacy norms, but I'm not going to clarify which)
_ex cathedra_ statement that gender categories are not an exception to the rule, only 1 year and 8 months after asking for it
]
-And really, that _should_ have been the end of the story. At the trifling cost of two years of my life, we finally got a clarification from Yudkowsky that you can't define the word _woman_ any way you like. I didn't think I was entitled to anything more than that. I was satisfied. I still published "Unnatural Categories Are Optimized for Deception" in January 2021, but if I hadn't been further provoked, I wouldn't have occasion to continue waging the robot-cult religious civil war.
+And really, that _should_ have been the end of the story. At the trifling cost of two years of my life, we finally got a clarification from Yudkowsky that you can't define the word _woman_ any way you like. I didn't think I was entitled to anything more than that. I was satisfied. If I hadn't been further provoked, I wouldn't have occasion to continue waging the robot-cult religious civil war.
-[TODO: psychiatric disaster, breakup with Vassar group, this was really bad for me
-[As it is written](https://www.alessonislearned.com/), "A lesson is learned but the damage is irreversible."
-]
+I still published ["Unnatural Categories Are Optimized for Deception"](https://www.lesswrong.com/posts/onwgTH6n8wxRSo2BJ/unnatural-categories-are-optimized-for-deception) in January 2021.
-[TODO: "Unnatural Categories Are Optimized for Deception"
+I wrote back to Abram Demski regarding his comments from fourteen months before: on further thought, he was right. Even granting my point that evolution didn't figure out how to track probability and utility separately, as Abram had pointed out, the _fact_ that it didn't meant that not tracking it could be an effective AI design. Just because evolution takes shortcuts that human engineers wouldn't didn't mean shortcuts are "wrong". (Rather, there are laws governing which kinds of shortcuts _work_.)
-Abram was right
+Abram was also right that it would be weird if reflective coherence was somehow impossible: the AI shouldn't have to fundamentally reason differently about "rewriting code in some 'external' program" and "rewriting 'its own' code." In that light, it made sense to regard "have accurate beliefs" as _merely_ a convergent instrumental subgoal, rather than what rationality is about—as sacrilegious as that felt to type.
-the fact that it didn't means that not tracking it can be an effective AI design! Just because evolution takes shortcuts that human engineers wouldn't doesn't mean shortcuts are "wrong" (instead, there are laws governing which kinds of shortcuts work).
+And yet, somehow, "have accurate beliefs" seemed _more fundamental_ than other convergent instrumental subgoals like "seek power and resources". Could this be made precise? As a stab in the dark, was it possible that the [theorems on the ubiquity of power-seeking](https://www.lesswrong.com/posts/6DuJxY8X45Sco4bS2/seeking-power-is-often-robustly-instrumental-in-mdps) might generalize to a similar conclusion about "accuracy-seeking"? If it _didn't_, the reason why it didn't might explain why accuracy seems more fundamental.
-Embedded agency means that the AI shouldn't have to fundamentally reason differently about "rewriting code in some 'external' program" and "rewriting 'my own' code." In that light, it makes sense to regard "have accurate beliefs" as merely a convergent instrumental subgoal, rather than what rationality is about
-somehow accuracy seems more fundamental than power or resources ... could that be formalized?
+[TODO: psychiatric disaster, breakup with Vassar group, this was really bad for me
+[As it is written](https://www.alessonislearned.com/), "A lesson is learned but the damage is irreversible."
]
projects / Ultimately_Untrue_Thought.git / blobdiff