Okay, being scared was entirely appropriate, but what I mean is that I was scared, and concluded that shaping the Singularity was _not my problem_, as contrasted to being scared, then facing up to the responsibility anyway. After a 2013 sleep-deprivation-induced psychotic episode which [featured](http://zackmdavis.net/blog/2013/03/religious/) [futurist](http://zackmdavis.net/blog/2013/04/prodrome/)-[themed](http://zackmdavis.net/blog/2013/05/relativity/) [delusions](http://zackmdavis.net/blog/2013/05/relevance/), I wrote to Anna, Michael, and some MIRI employees who had been in my contacts for occasional contract work, that "my current plan [was] to just try to forget about _Less Wrong_/MIRI for a long while, maybe at least a year, not because it isn't technically the most important thing in the world, but because I'm not emotionally stable enough think about this stuff anymore" (Subject: "to whom it may concern"). When I got a real programming job and established an income for myself, I [donated to CfAR rather than MIRI](http://zackmdavis.net/blog/2016/12/philanthropy-scorecard-through-2016/), because public rationality was something I could be unambiguously enthusiastic about, and doing anything about AI was not.
-At the time, it seemed fine for the altruistically-focused fraction of my efforts to focus on rationality, and to leave the save/destroy/take-over the world stuff to other, more emotionally-stable people, in accordance with the principle of comparative advantage. Yudkowsky had written his Sequences as a dependency for explaining [the need for friendly AI](https://www.lesswrong.com/posts/GNnHHmm8EzePmKzPk/value-is-fragile), ["gambl[ing] only upon the portion of the activism that would flow to [his] own cause"](https://www.lesswrong.com/posts/9jF4zbZqz6DydJ5En/the-end-of-sequences), but rationality was supposed to be the [common interest of many causes](https://www.lesswrong.com/posts/4PPE6D635iBcGPGRy/rationality-common-interest-of-many-causes). Even if I wasn't working or donating to MIRI specifically, I was still _helping_, a good citizen according to the morality of my tribe.
+At the time, it seemed fine for the altruistically-focused fraction of my efforts to focus on rationality, and to leave the save/destroy/take-over the world stuff to other, more emotionally-stable people, in accordance with the principle of comparative advantage. Yudkowsky had written his Sequences as a dependency for explaining [the need for friendly AI](https://www.lesswrong.com/posts/GNnHHmm8EzePmKzPk/value-is-fragile), ["gambl\[ing\] only upon the portion of the activism that would flow to \[his\] own cause"](https://www.lesswrong.com/posts/9jF4zbZqz6DydJ5En/the-end-of-sequences), but rationality was supposed to be the [common interest of many causes](https://www.lesswrong.com/posts/4PPE6D635iBcGPGRy/rationality-common-interest-of-many-causes). Even if I wasn't working or donating to MIRI specifically, I was still _helping_, a good citizen according to the morality of my tribe.
But fighting for public epistemology is a long battle; it makes more sense if you have _time_ for it to pay off. Back in the late 'aughts and early 'tens, it looked like we had time. We had these abstract philosophical arguments for worrying about AI, but no one really talked about _timelines_. I believed the Singularity was going to happen in the 21st century, but it felt like something to expect in the _second_ half of the 21st century.
(I asked Anna how Yudkowsky could stand the Glowfic people. She said she thought Eliezer could barely stand anyone. That makes sense, I said.)
-Everyone in dath ilan receives rationality training from childhood, but knowledge and training deemed psychologically hazardous to the general population is safeguarded by an order of [Keepers of Highly Unpleasant Things it is Sometimes Necessary to Know](https://www.glowfic.com/replies/1612937#reply-1612937). AGI research takes place in a secret underground city; some unspecified form of social engineering steers the _hoi polloi_ away from thinking about the possibility of AI.
+Everyone in dath ilan receives rationality training from childhood,[^category-vindication] but knowledge and training deemed psychologically hazardous to the general population is safeguarded by an order of [Keepers of Highly Unpleasant Things it is Sometimes Necessary to Know](https://www.glowfic.com/replies/1612937#reply-1612937). AGI research takes place in a secret underground city; some unspecified form of social engineering steers the _hoi polloi_ away from thinking about the possibility of AI.
+
+[^category-vindication]: On the topic of dath ilan's rationality training, I did appreciate [this passage about the cognitive function of categorization](https://www.glowfic.com/posts/5785?page=4):
+
+ > Dath ilani kids get told to not get fascinated with the fact that, in principle, 'bounded-agents' with finite memories and finite thinking speeds, have any considerations about mapping that depend on what they want. It doesn't mean that you get to draw in whatever you like on your map, because it's what you want. It doesn't make reality be what you want.
+
+ Vindication! (This showed that Yudkowsky _does_ understand what was at issue in the "... Not Man for the Categories" dispute, even if I can't be credited with winning the argument for political reasons.)
Something that annoyed me about the portrayal of dath ilan was their incredibly casual attitude towards hiding information for some alleged greater good, seemingly without considering that [there are benefits and not just costs to people knowing things](http://benjaminrosshoffman.com/humility-argument-honesty/).
projects / Ultimately_Untrue_Thought.git / blobdiff