------
-I continued to take note of signs of contemporary Yudkowsky visibly not being the same author who wrote the Sequences. In August 2019, [he Tweeted](https://twitter.com/ESYudkowsky/status/1164241431629721600):
+I continued to note signs of contemporary Yudkowsky not being the same author who wrote the Sequences. In August 2019, [he Tweeted](https://twitter.com/ESYudkowsky/status/1164241431629721600):
> I am actively hostile to neoreaction and the alt-right, routinely block such people from commenting on my Twitter feed, and make it clear that I do not welcome support from those quarters. Anyone insinuating otherwise is uninformed, or deceptive.
--------
-[TODO: a culture that has gone off the rails; my warning points to Vaniver; discussion of Vassar on "ialdabaoth is banned"]
-
---------
-
[TODO: complicity and friendship]
--------
------
-While visiting "Arcadia" on 4 February 2020, I remember my nose dripping while I was holding the baby. Alicorn offered me a tissue. I asked if I shouldn't be holding the baby while my nose was dripping and I therefore plausibly had a cold. She said it was fine. On the topic of possible sickness, I said that I hoped the novel coronavirus people were talking about didn't go pandemic.
-
-It did. The Berkeley rats took social distancing guidelines very seriously, so it would be a while before I could visit again.
-
-------
-
On 10 February 2020, Scott Alexander published ["Autogenderphilia Is Common and Not Especially Related to Transgender"](https://slatestarcodex.com/2020/02/10/autogenderphilia-is-common-and-not-especially-related-to-transgender/), an analysis of the results of the autogynephilia/autoandrophilia questions on the recent _Slate Star Codex_ survey.
I appreciated the gesture of getting real data, but I was deeply unimpressed with Alexander's analysis for reasons that I found difficult to write up in a timely manner. Three years later, I eventually got around to [polishing my draft and throwing it up as a standalone post](/2023/Mar/reply-to-scott-alexander-on-autogenderphilia/), rather than cluttering the present narrative with my explanation.
On 1 June 2020, I received a Twitter DM from _New York Times_ reporter Cade Metz, who said he was "exploring a story about the intersection of the rationality community and Silicon Valley". I sent him an email saying that I would be happy to talk, but that I'd actually been pretty disappointed with the community lately: I was worried that the social pressures of trying to _be_ a "community" and protect the group's status (_e.g._, from _New York Times_ reporters who might portray us in an unflattering light??) incentivize people to compromise on the ideals of _systematically correct reasoning_ that made the community valuable in the first place.
-He never got back to me. Three weeks later, all existing _Slate Star Codex_ posts were taken down.
-
-A [lone post on the main page](https://slatestarcodex.com/2020/06/22/nyt-is-threatening-my-safety-by-revealing-my-real-name-so-i-am-deleting-the-blog/) explained that the _New York Times_ piece was going to reveal Alexander's real last name, and that he was taking his posts down as a defensive measure. (No blog, no story?) I [wrote a script](/source?p=Ultimately_Untrue_Thought.git;a=commitdiff;h=21731ba6f1191) (`slate_starchive.py`) to replace the _Slate Star Codex_ links on this blog with links to the most recent Internet Archive copy.
+He never got back to me. Three weeks later, all existing _Slate Star Codex_ posts were taken down. A [lone post on the main page](https://slatestarcodex.com/2020/06/22/nyt-is-threatening-my-safety-by-revealing-my-real-name-so-i-am-deleting-the-blog/) explained that the _New York Times_ piece was going to reveal Alexander's real last name, and that he was taking his posts down as a defensive measure. (No blog, no story?) I [wrote a script](/source?p=Ultimately_Untrue_Thought.git;a=commitdiff;h=21731ba6f1191) (`slate_starchive.py`) to replace the _Slate Star Codex_ links on this blog with links to the most recent Internet Archive copy.
------
Category "boundaries" are a useful _visual metaphor_ for explaining the cognitive function of categorization: you imagine a "boundary" in configuration space containing all the things that belong to the category.
-If you have the visual metaphor, but you don't have the math, you might think that there's nothing intrinsically wrong with squiggly or discontinuous category "boundaries", just as there's nothing intrinsically wrong with Alaska not being part of the contiguous U.S. states. It may be _inconvenient_ that you can't drive from Alaska to Washington without going through Canada, and we have to deal with the consequences of that, but there's no sense in which it's _wrong_ that the borders are drawn that way: Alaska really is governed by the United States.
+If you have the visual metaphor, but you don't have the math, you might think that there's nothing intrinsically wrong with squiggly or discontinuous category "boundaries", just as there's nothing intrinsically wrong with Alaska not being part of the contiguous U.S. states. It may be inconvenient that you can't drive from Alaska to Washington without going through Canada, and we have to deal with the consequences of that, but there's no sense in which it's _wrong_ that the borders are drawn that way: Alaska really is governed by the United States.
But if you _do_ have the math, a moment of introspection will convince you that the analogy between category "boundaries" and national borders is not a particularly deep or informative one.
That was my intuition. To formalize it, I wanted some sensible numerical quantity that would be maximized by using "nice" categories and get trashed by gerrymandering. [Mutual information](https://en.wikipedia.org/wiki/Mutual_information) was the obvious first guess, but that wasn't it, because mutual information lacks a "topology", a notion of _closeness_ that made some false predictions better than others by virtue of being "close".
-Suppose the outcome space of _X_ is `{H, T}` and the outcome space of _Y_ is `{1, 2, 3, 4, 5, 6, 7, 8}`. I _wanted_ to say that if observing _X_=`H` concentrates _Y_'s probability mass on `{1, 2, 3}`, that's _more useful_ than if it concentrates _Y_ on `{1, 5, 8}`—but that would require the numbers in Y to be _numbers_ rather than opaque labels; as far as elementary information theory was concerned, mapping eight states to three states reduced the entropy from log<sub>2</sub> 8 = 3 to log<sub>2</sub> 3 ≈ 1.58 no matter "which" three states they were.
+Suppose the outcome space of _X_ is `{H, T}` and the outcome space of _Y_ is `{1, 2, 3, 4, 5, 6, 7, 8}`. I wanted to say that if observing _X_=`H` concentrates _Y_'s probability mass on `{1, 2, 3}`, that's _more useful_ than if it concentrates _Y_ on `{1, 5, 8}`—but that would require the numerals in _Y_ to be _numbers_ rather than opaque labels; as far as elementary information theory was concerned, mapping eight states to three states reduced the entropy from lg<sub>2</sub> 8 = 3 to lg<sub>2</sub> 3 ≈ 1.58 no matter "which" three states they were.
-How could I make this rigorous? Did I want to be talking about the _variance_ of my features conditional on category-membership? Was "connectedness" intrinsically the what I wanted, or was connectedness only important because it cut down the number of possibilities? (There are 8!/(6!2!) = 28 ways to choose two elements from `{1..8}`, but only 7 ways to choose two contiguous elements.) I thought connectedness _was_ intrinsically important, because we didn't just want _few_ things, we wanted things that are _similar enough to make similar decisions about_.
+How could I make this rigorous? Did I want to be talking about the variance of my features conditional on category-membership? Was "connectedness" intrinsically the what I wanted, or was connectedness only important because it cut down the number of possibilities? (There are 8!/(6!2!) = 28 ways to choose two elements from `{1..8}`, but only 7 ways to choose two contiguous elements.) I thought connectedness was intrinsically important, because we didn't just want _few_ things, we wanted things that are _similar enough to make similar decisions about_.
I put the question to a few friends in July 2020 (Subject: "rubber duck philosophy"), and Jessica said that my identification of the variance as the key quantity sounded right: it amounted to the expected squared error of someone trying to guess the values of the features given the category. It was okay that this wasn't a purely information-theoretic criterion, because for problems involving guessing a numeric quantity, bits that get you closer to the right answer were more valuable than bits that didn't.
And really, that _should_ have been the end of the story. At the trifling cost of two years of my life, we finally got a clarification from Yudkowsky that you can't define the word _woman_ any way you like. I didn't think I was entitled to anything more than that. I was satisfied. If I hadn't been further provoked, I wouldn't have occasion to continue waging the robot-cult religious civil war.
+-----
+
+[TODO: psychiatric disaster, breakup with Vassar group, this was really bad for me
+[As it is written](https://www.alessonislearned.com/), "A lesson is learned but the damage is irreversible."
+]
+
+-----
+
I still published ["Unnatural Categories Are Optimized for Deception"](https://www.lesswrong.com/posts/onwgTH6n8wxRSo2BJ/unnatural-categories-are-optimized-for-deception) in January 2021.
I wrote back to Abram Demski regarding his comments from fourteen months before: on further thought, he was right. Even granting my point that evolution didn't figure out how to track probability and utility separately, as Abram had pointed out, the _fact_ that it didn't meant that not tracking it could be an effective AI design. Just because evolution takes shortcuts that human engineers wouldn't didn't mean shortcuts are "wrong". (Rather, there are laws governing which kinds of shortcuts _work_.)
Abram was also right that it would be weird if reflective coherence was somehow impossible: the AI shouldn't have to fundamentally reason differently about "rewriting code in some 'external' program" and "rewriting 'its own' code." In that light, it made sense to regard "have accurate beliefs" as _merely_ a convergent instrumental subgoal, rather than what rationality is about—as sacrilegious as that felt to type.
And yet, somehow, "have accurate beliefs" seemed _more fundamental_ than other convergent instrumental subgoals like "seek power and resources". Could this be made precise? As a stab in the dark, was it possible that the [theorems on the ubiquity of power-seeking](https://www.lesswrong.com/posts/6DuJxY8X45Sco4bS2/seeking-power-is-often-robustly-instrumental-in-mdps) might generalize to a similar conclusion about "accuracy-seeking"? If it _didn't_, the reason why it didn't might explain why accuracy seems more fundamental.
-
-
-[TODO: psychiatric disaster, breakup with Vassar group, this was really bad for me
-[As it is written](https://www.alessonislearned.com/), "A lesson is learned but the damage is irreversible."
-]
projects / Ultimately_Untrue_Thought.git / blobdiff