Ultimately, I think this was a pedagogy decision that Yudkowsky had gotten right back in 'aught-eight. If you write your summary slogan in relativist language, people predictably take that as license to believe whatever they want without having to defend it. Whereas if you write your summary slogan in objectivist language—so that people know they don't have social permission to say that "it's subjective so I can't be wrong"—then you have some hope of sparking useful thought about the _exact, precise_ ways that _specific, definite_ things are _in fact_ relative to other specific, definite things.
-I told him I would send him one more email with a piece of evidence about how other "rationalists" were thinking about the categories issue, and give my commentary on the parable about orcs, and then the present thread would probably drop there.
+I told Scott I would send him one more email with a piece of evidence about how other "rationalists" were thinking about the categories issue, and give my commentary on the parable about orcs, and then the present thread would probably drop there.
On Discord in January, Kelsey Piper had told me that everyone else experienced their disagreement with me as being about where the joints are and which joints are important, where usability for humans was a legitimate criterion for importance, and it was annoying that I thought they didn't believe in carving reality at the joints at all and that categories should be whatever makes people happy.
(I did regret having accidentally "poisoned the well" the previous month by impulsively sharing the previous year's ["Blegg Mode"](/2018/Feb/blegg-mode/) [as a _Less Wrong_ linkpost](https://www.lesswrong.com/posts/GEJzPwY8JedcNX2qz/blegg-mode). "Blegg Mode" had originally been drafted as part of "... To Make Predictions" before getting spun off as a separate post. Frustrated in March at our failing email campaign, I thought it was politically "clean" enough to belatedly share, but it proved to be insufficiently [deniably allegorical](/tag/deniably-allegorical/), as evidenced by the 60-plus-entry trainwreck of a comments section. It's plausible that some portion of the _Less Wrong_ audience would have been more receptive to "... Boundaries?" as not-politically-threatening philosophy, if they hadn't been alerted to the political context by the comments on the "Blegg Mode" linkpost.)
-On 13 April, I pulled the trigger on publishing "... Boundaries?", and wrote to Yudkowsky again, a fourth time (!), asking if he could _either_ publicly endorse the post, _or_ publicly comment on what he thought the post got right and what he thought it got wrong; and, that if engaging on this level was too expensive for him in terms of spoons, if there was any action I could take to somehow make it less expensive? The reason I thought this was important, I explained, was that if rationalists in [good standing](https://srconstantin.wordpress.com/2018/12/24/contrite-strategies-and-the-need-for-standards/) find themselves in a persistent disagreement _about rationality itself_—in this case, my disagreement with Scott Alexander and others about the cognitive function of categories—that seemed like a major concern for [our common interest](https://www.lesswrong.com/posts/4PPE6D635iBcGPGRy/rationality-common-interest-of-many-causes), something we should be very eager to _definitively settle in public_ (or at least _clarify_ the current state of the disagreement). In the absence of an established "rationality court of last resort", I feared the closest thing we had was an appeal to Eliezer Yudkowsky's personal judgement. Despite the context in which the dispute arose, _this wasn't a political issue_. The post I was asking for his comment on was _just_ about the [_mathematical laws_](https://www.lesswrong.com/posts/eY45uCCX7DdwJ4Jha/no-one-can-exempt-you-from-rationality-s-laws) governing how to talk about, _e.g._, dolphins. We had _nothing to be afraid of_ here. (Subject: "movement to clarity; or, rationality court filing").
+On 13 April, I pulled the trigger on publishing "... Boundaries?", and wrote to Yudkowsky again, a fourth time (!), asking if he could _either_ publicly endorse the post, _or_ publicly comment on what he thought the post got right and what he thought it got wrong; and, that if engaging on this level was too expensive for him in terms of spoons, if there was any action I could take to somehow make it less expensive? The reason I thought this was important, I explained, was that if rationalists in [good standing](https://srconstantin.github.io/2018/12/24/contrite-strategies-and-the-need-for-standards/) find themselves in a persistent disagreement _about rationality itself_—in this case, my disagreement with Scott Alexander and others about the cognitive function of categories—that seemed like a major concern for [our common interest](https://www.lesswrong.com/posts/4PPE6D635iBcGPGRy/rationality-common-interest-of-many-causes), something we should be very eager to _definitively settle in public_ (or at least _clarify_ the current state of the disagreement). In the absence of an established "rationality court of last resort", I feared the closest thing we had was an appeal to Eliezer Yudkowsky's personal judgement. Despite the context in which the dispute arose, _this wasn't a political issue_. The post I was asking for his comment on was _just_ about the [_mathematical laws_](https://www.lesswrong.com/posts/eY45uCCX7DdwJ4Jha/no-one-can-exempt-you-from-rationality-s-laws) governing how to talk about, _e.g._, dolphins. We had _nothing to be afraid of_ here. (Subject: "movement to clarity; or, rationality court filing").
I got some pushback from Ben and Jessica about claiming that this wasn't "political". What I meant by that was to emphasize (again) that I didn't expect Yudkowsky or "the community" to take a public stance _on gender politics_; I was trying to get "us" to take a stance in favor of the kind of _epistemology_ that we were doing in 2008. It turns out that epistemology has implications for gender politics which are unsafe, but that's _more inferential steps_, and ... I guess I just didn't expect the sort of people who would punish good epistemology to follow the inferential steps?
One might wonder why this was such a big deal to us. Okay, so Yudkowsky had prevaricated about his own philosophy of language for transparently political reasons, and couldn't be moved to clarify in public even after me and my posse spent an enormous amount of effort trying to explain the problem. So what? Aren't people wrong on the internet all the time?
-Ben explained that Yudkowsky wasn't a private person who might plausibly have the right to be wrong on the internet in peace. Yudkowsky was a public figure whose claim to legitimacy really did amount to a claim that while nearly everyone else was criminally insane (causing huge amounts of damage due to disconnect from reality, in a way that would be criminal if done knowingly), he almost uniquely was not—and he had he had set in motion a machine (the "rationalist community") that was continuing to raise funds and demand work from people for below-market rates based on that claim—"work for me or the world ends badly", basically.
+Ben explained: Yudkowsky had set in motion a marketing machine (the "rationalist community") that was continuing to raise funds and demand work from people for below-market rates based on the claim that while nearly everyone else was criminally insane (causing huge amounts of damage due to disconnect from reality, in a way that would be criminal if done knowingly), he, almost uniquely, was not. If the claim was _true_, it was important to make, and to actually extract that labor. "Work for me or the world ends badly," basically.
-If the claim was _true_, it was important to make, and to actually extract that labor. But we had falsified to our satisfaction the claim that Yudkowsky was currently sane in the relevant way (which was a _extremely high_ standard, and not a special flaw of Yudkowsky in the current environment). If Yudkowsky couldn't be bothered to live up to his own stated standards or withdraw his validation from the machine he built after we had _tried_ to bring it up in private with him, then we had a right to talk about what we thought was going on.
+But we had just falsified to our satisfaction the claim that Yudkowsky was currently sane in the relevant way (which was a _extremely high_ standard, and not a special flaw of Yudkowsky in the current environment). If Yudkowsky couldn't be bothered to live up to his own stated standards or withdraw his validation from the machine he built after we had _tried_ to talk to him privately, then we had a right to talk in public about what we thought was going on.
-Ben further compared Yudkowsky (as the most plausible individual representative of the "rationalists") to Eliza the spambot therapist in my story ["Blame Me for Trying"](/2018/Jan/blame-me-for-trying/): regardless of the initial intent, scrupulous rationalists were paying rent to something claiming moral authority, which had no concrete specific plan to do anything other than run out the clock, maintaining a facsimile of dialogue in ways well-calibrated to continue to generate revenue. Minds like mine wouldn't surive long-run in this ecosystem. If we wanted minds that do "naïve" inquiry instead of playing savvy Kolmogorov games to survive, we needed an interior that justified that level of trust.
+This wasn't about direct benefit _vs._ harm. This was about what, substantively, the machine was doing. They claimed to be cultivating an epistemically rational community, while in fact building an army of loyalists.
-[TODO: rewrite Ben's account of the problem above, including 15 April Signal conversation]
+Ben compared the whole set-up to that of Eliza the spambot therapist in my story ["Blame Me for Trying"](/2018/Jan/blame-me-for-trying/): regardless of the _initial intent_, scrupulous rationalists were paying rent to something claiming moral authority, which had no concrete specific plan to do anything other than run out the clock, maintaining a facsimile of dialogue in ways well-calibrated to continue to generate revenue. Minds like mine wouldn't surive long-run in this ecosystem. If we wanted minds that do "naïve" inquiry instead of playing savvy Kolmogorov games to survive, we needed an interior that justified that level of trust.
-------
-[TODO: better outline 2019]
+Given that the "rationalists" were fake and that we needed something better, there remained the question of what to do about that, and how to relate to the old thing, and the maintainers of the marketing machine for the old thing.
+
+_I_ had been hyperfocused on prosecuting my Category War, but the reason Michael and Ben and Jessica were willing to help me out on that, was not because they particularly cared about the gender and categories example, but because it seemed like a manifestation of a _more general_ problem of epistemic rot in "the community".
+
+Ben had previously written a lot about problems with Effective Altruism. Jessica had had a bad time at MIRI, as she had told me back in March, and would [later](https://www.lesswrong.com/posts/KnQs55tjxWopCzKsk/the-ai-timelines-scam) [write](https://www.lesswrong.com/posts/MnFqyPLqbiKL8nSR7/my-experience-at-and-around-miri-and-cfar-inspired-by-zoe) [about](https://www.lesswrong.com/posts/pQGFeKvjydztpgnsY/occupational-infohazards). To what extent were my thing, and Ben's thing, and Jessica's thing, manifestations of "the same" underlying problem? Or had we all become disaffected with the mainstream "rationalists" for our own idiosyncratic reasons, and merely randomly fallen into each other's, and Michael's, orbit?
+
+I believed that there _was_ a real problem, but didn't feel like I had a good grasp on what it was specifically. Cultural critique is a fraught endeavor: if someone tells an outright lie, you can, maybe, with a lot of effort, prove that to other people, and get a correction on that specific point. (Actually, as we had just discovered, even that might be too much to hope for.) But _culture_ is the sum of lots and lots of little micro-actions by lots and lots of people. If your _entire culture_ has visibly departed from the Way that was taught to you in the late 'aughts, how do you demonstrate that to people who, to all appearances, are acting like they don't remember the old Way, or that they don't think anything has changed, or that they notice some changes but think the new way is better. It's not as simple as shouting, "Hey guys, Truth matters!"—any ideologue or religious person would agree with _that_.
+
+Ben called it the Blight, after the rogue superintelligence in _A Fire Upon the Deep_: the problem wasn't that people were getting dumber; it's that there was locally coherent coordination away from clarity and truth and towards coalition-building, which was validated by the official narrative in ways that gave it a huge tactical advantage; people were increasingly making decisions that were better explained by their political incentives rather than acting on coherent beliefs about the world.
+
+When I asked him for specific examples of MIRI or CfAR leaders behaving badly, he gave the example of MIRI executive director Nate Soares posting that he was "excited" about the launch of OpenAI, despite the fact that [_no one_ who had been following the AI risk discourse](https://slatestarcodex.com/2015/12/17/should-ai-be-open/) [thought that OpenAI as originally announced was a good idea](http://benjaminrosshoffman.com/openai-makes-humanity-less-safe/). Nate had privately clarified to Ben that the word "excited" wasn't necessarily meant positively, and in this case meant something more like "terrified."
+
+This seemed to me like the sort of thing where a particularly principled (naive?) person might say, "That's _lying for political reasons!_ That's _contrary to the moral law!_" and most ordinary grown-ups would say, "Why are you so upset about this? That sort of strategic phrasing in press releases is just how the world works, and things could not possibly be otherwise."
+
+I thought explaining the Blight to an ordinary grown-up was going to need _either_ lots of specific examples that were way more egregious than this (and more egregious than the examples in "EA Has a Lying Problem" or ["Effective Altruism Is Self-Recommending"](http://benjaminrosshoffman.com/effective-altruism-is-self-recommending/)), or somehow convincing the ordinary grown-up why "just how the world works" isn't good enough, and why we needed one goddamned place in the entire goddamned world (perhaps a private place) with _unusually high standards_.
+
+The schism introduced new pressures on my social life. On 20 April, I told Michael that I still wanted to be friends with people on both sides of the factional schism (in the frame where recent events were construed as a factional schism), even though I was on this side. Michael said that we should unambiguously regard Anna and Eliezer as criminals or enemy combatants (!!), that could claim no rights in regards to me or him.
+
+I don't think I "got" the framing at this time. War metaphors sounded Scary and Mean: I didn't want to shoot my friends! But the point of the analogy (which Michael explained, but I wasn't ready to hear until I did a few more weeks of emotional processing) was specifically that soliders on the other side of a war _aren't_ particularly morally blameworthy as individuals: their actions are just being controlled by the Power they're embedded in.
+
+I wrote to Anna:
+
+> I was _just_ trying to publicly settle a _very straightforward_ philosophy thing that seemed _really solid_ to me
+>
+> if, in the process, I accidentally ended up being an unusually useful pawn in Michael Vassar's deranged four-dimensional hyperchess political scheming
+>
+> that's ... _arguably_ not my fault
+
+-----
+
+I may have subconsciously pulled off an interesting political thing. In my final email to Yudkowsky on 20 April (Subject: "closing thoughts from me"), I had written—
+
+> If we can't even get a public consensus from our _de facto_ leadership on something _so basic_ as "concepts need to carve reality at the joints in order to make probabilistic predictions about reality", then, in my view, there's _no point in pretending to have a rationalist community_, and I need to leave and go find something else to do (perhaps whatever Michael's newest scheme turns out to be). I don't think I'm setting [my price for joining](https://www.lesswrong.com/posts/Q8evewZW5SeidLdbA/your-price-for-joining) particularly high here?
+
+And as it happened, on 5 May, Yudkowsky reTweeted Colin Wright on the "univariate fallacy"—the point that group differences aren't a matter of any single variable—which was _sort of_ like the clarification I had been asking for. (Empirically, it made me feel a lot less personally aggrieved.) Was I wrong to interpet this as another "concession" to me? (Again, notwithstanding that the whole mindset of extracting "concessions" was corrupt and not what our posse was trying to do.)
+
+Separately, I visited some friends' house on 30 April saying, essentially (and sincerely), "[Oh man oh jeez](https://www.youtube.com/watch?v=NivwAQ8sUYQ), Ben and Michael want me to join in a rationalist civil war against the corrupt mainstream-rationality establishment, and I'd really rather not, and I don't like how they keep using scary hyperbolic words like 'cult' and 'war' and 'criminal', but on the other hand, they're _the only ones backing me up_ on this _incredibly basic philosophy thing_ and I don't feel like I have anywhere else to _go_." The ensuring group conversation made some progress, but was mostly pretty horrifying.
+
+In an adorable twist, my friends' two-year-old son was reportedly saying the next day that Kelsey doesn't like his daddy, which was confusing until it was figured out he had heard Kelsey talking about why she doesn't like Michael _Vassar_.
+
+And as it happened, on 8 May, Kelsey wrote a Facebook comment displaying evidence of understanding my point.
+
+These two datapoints led me to a psychological hypothesis (which was maybe "obvious", but I hadn't thought about it before): when people see someone wavering between their coalition and a rival coalition, they're motivated to offer a few concessions to keep the wavering person on their side. Kelsey could _afford_ (_pace_ Upton Sinclair) to not understand the thing about sex being a natural category ("I don't think 'people who'd get surgery to have the ideal female body' cuts anything at the joints"!!) when it was just me freaking out alone, but "got it" almost as soon as I could credibly threaten to _walk_ (defect to a coalition of people she dislikes) ... and maybe my "closing thoughts" email had a similar effect on Yudkowsky (assuming he otherwise wouldn't have spontaneously tweeted something about the univariate fallacy two weeks later)?? This probably wouldn't work if you repeated it (or tried to do it consciously)?
+
+----
+
+I started drafting a "why I've been upset for five months and have lost faith in the so-called 'rationalist' community" personal-narrative Diary-like post. Ben said that the target audience to aim for was people like I was a few years ago, who hadn't yet had the experiences I had—so they wouldn't have to freak out to the point of being imprisoned and demand help from community leaders and not get it; they could just learn from me. That is, the actual sympathetic-but-naïve people could learn. Not the people messing with me.
+
+I didn't know how to finish it. I was too psychologically constrained; I didn't know how to tell the Whole Dumb Story without (as I perceived it) escalating personal conflicts or leaking info from private conversations.
+
+I decided to take a break from the religious civil war [for a month](http://zackmdavis.net/blog/2019/05/may-is-math-and-wellness-month/) [or two](/2019/May/hiatus/).
+
+My dayjob performance had been suffering terribly for months. The psychology of the workplace is ... subtle. There's a phenomenon where some people are _way_ more productive than others and everyone knows it, but no one is cruel enough [to make it _common_ knowledge](https://slatestarcodex.com/2015/10/15/it-was-you-who-made-my-blue-eyes-blue/), which is awkward for people who simultaneously benefit from the culture of common-knowledge-prevention allowing them to collect the status and money rents of being a $150K/yr software engineer without actually [performing at that level](http://zackmdavis.net/blog/2013/12/fortune/), while also having [read enough Ayn Rand as a teenager](/2017/Sep/neither-as-plea-nor-as-despair/) to be ideologically opposed to subsisting on unjustly-acquired rents rather than value creation. The "everyone knows I feel guilty about underperforming, so they don't punish me because I'm already doing enough internalized domination to punish myself" dynamic would be unsustainable if it were to evolve into a loop of "feeling gulit _in exchange for_ not doing work" rather than the intended "feeling guilt in order to successfully incentivize work". I didn't think they would actually fire me, but I was worried that they _should_. I asked my boss to temporarily take on some easier tasks, that I could make steady progress on even while being psychologically impaired from a religious war. (We had a lot of LaTeX templating of insurance policy amendments that needed to get done.) If I was going to be psychologically impaired _anyway_, it was better to be upfront about how I could best serve the company given that impairment, rather than hoping that the boss wouldn't notice.
+
+My "intent" to take a break from the religious war didn't take.
+
+[TODO: tussle with Anna]
+
+[TODO: tussle on "Yes Implies the Possibility of No"]
+
+[TODO: tussle on new _Less Wrong_ FAQ]
+
+
+
+https://twitter.com/ESYudkowsky/status/1124751630937681922
+> ("sort of like" in the sense that, empirically, it made me feel much less personally aggrieved, but of course my feelings aren't the point)
+
+
+[TODO: epistemic defense meeting; the first morning where "rationalists ... them" felt more natural than "rationalists ... us"]
+
In November, I received an interesting reply on my philosophy-of-categorization thesis from MIRI researcher Abram Demski. Abram asked: ideally, shouldn't all conceptual boundaries be drawn with appeal-to-consequences? Wasn't the problem just with bad (motivated, shortsighted) appeals to consequences? Agents categorize in order to make decisions. The best classifer for an application depends on the costs and benefits. As a classic example, it's very important for evolved prey animals to avoid predators, so it makes sense for their predator-detection classifiers to be configured such that they jump away from every rustling in the bushes, even if it's usually not a predator.
>
> I agree that pronouns don't have the same function as ordinary nouns. However, **in the English language as actually spoken by native speakers, I think that gender pronouns _do_ have effective "truth conditions" _as a matter of cognitive science_.** If someone said, "Come meet me and my friend at the mall; she's really cool and you'll like her", and then that friend turned out to look like me, **you would be surprised**.
>
-> I don't see the _substantive_ difference between "You're not standing in defense of truth [...]" and "I can define a word any way I want." [...]
+> I don't see the _substantive_ difference between "You're not standing in defense of truth (...)" and "I can define a word any way I want." [...]
>
> [...]
>
projects / Ultimately_Untrue_Thought.git / blobdiff