"Hill of Validity": rewrite ending

This was terrible writing on my part; I had summarized something Ben
said, which made sense to us at the time, but isn't a clear
explanation for my readers now.

diff --git a/content/2023/a-hill-of-validity-in-defense-of-meaning.md b/content/2023/a-hill-of-validity-in-defense-of-meaning.md
index f1fa287..8483495 100644 (file)
--- a/content/2023/a-hill-of-validity-in-defense-of-meaning.md
+++ b/content/2023/a-hill-of-validity-in-defense-of-meaning.md
@@ -607,16 +607,16 @@ But it's only "obvious" if you take as a given that Yudkowsky is playing a savvy
 
 Maybe this seems banal if you haven't spent your entire adult life in his robot cult. From anyone else in the world, I wouldn't have had a problem with the "hill of validity in defense of meaning" thread—I would have respected it as a solidly above-average philosophy performance before [setting the bozo bit](https://en.wikipedia.org/wiki/Bozo_bit#Dismissing_a_person_as_not_worth_listening_to) on the author and getting on with my day. But since I _did_ spend my entire adult life in Yudkowsky's robot cult, trusting him the way a Catholic trusts the Pope, I _had_ to assume that it was an "honest mistake" in his rationality lessons, and that honest mistakes could be honestly corrected if someone put in the effort to explain the problem. The idea that Eliezer Yudkowsky was going to behave just as badly as any other public intellectual in the current year was not really in my hypothesis space.
 
-Ben shared the account of our posse's email campaign with someone who commented that I had "sacrificed all hope of success in favor of maintaining his own sanity by CC'ing you guys." That is, if I had been brave enough to confront Yudkowsky by myself, maybe there was some hope of him seeing that the game he was playing was wrong. But because I was so cowardly as to need social proof (because I believed that an ordinary programmer such as me was as a mere worm in the presence of the great Eliezer Yudkowsky), it must have just looked to him like an illegible social plot originating from Michael.
+Ben shared the account of our posse's email campaign with someone who commented that I had "sacrificed all hope of success in favor of maintaining his own sanity by CC'ing you guys." That is, if I had been brave enough to confront Yudkowsky by myself, maybe there was some hope of him seeing that the game he was playing was wrong. But because I was so cowardly as to need social proof (because I believed that an ordinary programmer such as me was as a mere worm in the presence of the great Eliezer Yudkowsky), it probably just looked to him like an illegible social plot originating from Michael.
 
-One might wonder why this was such a big deal to us. Okay, so Yudkowsky had prevaricated about his own philosophy of language for transparently political reasons, and he couldn't be moved to clarify in public even after we spent an enormous amount of effort trying to explain the problem. So what? Aren't people wrong on the internet all the time?
+One might wonder why this was such a big deal to us. Okay, so Yudkowsky had prevaricated about his own philosophy of language for political reasons, and he couldn't be moved to clarify even after we spent an enormous amount of effort trying to explain the problem. So what? Aren't people wrong on the internet all the time?
 
-Ben explained: Yudkowsky had set in motion a marketing machine (the "rationalist community") that was continuing to raise funds and demand work from people for below-market rates based on the claim that while nearly everyone else was criminally insane (causing huge amounts of damage due to disconnect from reality, in a way that would be criminal if done knowingly), he, almost uniquely, was not. "Work for me or the world ends badly," basically. If the claim was true, it was important to make, and to actually extract that labor.
+This wasn't just anyone being wrong on the internet. In [an essay on the development of cultural traditions](https://slatestarcodex.com/2016/04/04/the-ideology-is-not-the-movement/), Scott Alexander had written that rationalism is the belief that Eliezer Yudkowsky is the rightful caliph. To no small extent, I and many other people had built our lives around a story that portrayed Yudkowsky as almost uniquely sane—a story that put MIRI, CfAR, and the "rationalist community" at the center of the universe, the ultimate fate of the cosmos resting on our individual and collective mastery of the hidden Bayesian structure of cognition.
 
-But we had just falsified to our satisfaction the claim that Yudkowsky was currently sane in the relevant way (which was a extremely high standard, and not a special flaw of Yudkowsky in the current environment). If, after we had _tried_ to talk to him privately, Yudkowsky couldn't be bothered to either live up to his own stated standards or withdraw his validation from the machine he built, then we had a right to talk about what we thought was going on.
+But my posse and I had just falsified to our satisfaction the claim that Yudkowsky was currently sane in the relevant way. Maybe _he_ didn't think he had done anything wrong (because he [hadn't strictly lied](https://www.lesswrong.com/posts/MN4NRkMw7ggt9587K/firming-up-not-lying-around-its-edge-cases-is-less-broadly)), and probably a normal person would think we were making a fuss about nothing, but as far as we were concerned, the formerly rightful caliph had relinquished his legitimacy.
 
-This wasn't about direct benefit _vs._ harm. This was about what, substantively, the machine and its operators were doing. They claimed to be cultivating an epistemically rational community, while in fact building an army of loyalists.
+Ben made an analogy to a short story I had written, ["Blame Me for Trying"](/2018/Jan/blame-me-for-trying/). In the story, an scrupulous AI clings to the advice of a therapist who is more interested in maximizing her profits than helping her clients, who routinely die after spending all their money on therapy sessions. The therapist's marketing story about improving her clients' mental health was false. Similarly, scrupulous rationalists were investing a lot of their time, effort, and money into "the community" and its institutions on a the basis of a marketing story that we now knew to be false, as political concerns took precedence over clarifying the hidden Bayesian structure of cognition.
 
-Ben compared the whole set-up to that of Eliza the spambot therapist in my short story ["Blame Me for Trying"](/2018/Jan/blame-me-for-trying/): regardless of the initial intent, scrupulous rationalists were paying rent to something claiming moral authority, which had no concrete specific plan to do anything other than run out the clock, maintaining a facsimile of dialogue in ways well-calibrated to continue to generate revenue. Minds like mine wouldn't survive long-term in this ecosystem. If we wanted minds that do "naïve" inquiry (instead of playing savvy power games) to live, we needed an interior that justified that level of trust.
+Minds like mine wouldn't survive long-term in this ecosystem. If we wanted minds that do "naïve" inquiry (instead of playing savvy power games) to live, we needed an interior that justified that level of trust.
 
 (To be continued. Yudkowsky would [eventually clarify his position on the philosophy of categorization in September 2020](https://www.facebook.com/yudkowsky/posts/10158853851009228)—but the story leading up to that will have to wait for another day.)