Which brings me to the second reason the naïve anti-extortion argument might fail: [what counts as "extortion" depends on the relevant "property rights", what the "default" action is](https://www.lesswrong.com/posts/Qjaaux3XnLBwomuNK/countess-and-baron-attempt-to-define-blackmail-fail). If having free speech is the default, being excluded from the dominant coalition for defying the orthodoxy could be construed as extortion. But if being excluded from the coalition is the default, maybe toeing the line of orthodoxy is the price you need to pay in order to be included.
-Yudkowsky has [a proposal for how bargaining should work between agents with different notions of "fairness"](https://www.lesswrong.com/posts/z2YwmzuT7nWx62Kfh/cooperating-with-agents-with-different-ideas-of-fairness). Suppose Hayley and Isaac are splitting a pie, and if they can't initially agree on how to split it, they have to fight over it until they do agree, destroying some of the pie in the process. Hayley thinks the fair outcome is that they each get half the pie. Isaac claims that he contributed more ingredients to the baking process and that it's therefore fair that he gets 75% of the pie, pledging to fight if offered anything less.
+Yudkowsky has [a proposal for how bargaining should work between agents with different notions of "fairness"](https://www.lesswrong.com/posts/z2YwmzuT7nWx62Kfh/cooperating-with-agents-with-different-ideas-of-fairness). Suppose Greg and Heather are splitting a pie, and if they can't initially agree on how to split it, they have to fight over it until they do agree, destroying some of the pie in the process. Greg thinks the fair outcome is that they each get half the pie. Heather claims that she contributed more ingredients to the baking process and that it's therefore fair that she gets 75% of the pie, pledging to fight if offered anything less.
-If Hayley were a causal decision theorist, she might agree to the 75/25 split, reasoning that 25% of the pie is better than fighting until the pie is destroyed. Yudkowsky argues that this is irrational: if Hayley is willing to agree to a 75/25 split, then Isaac has no incentive not to adopt such a self-favoring definition of "fairness". (And _vice versa_ if Isaac's concept of fairness is the "correct" one.)
+If Greg were a causal decision theorist, he might agree to the 75/25 split, reasoning that 25% of the pie is better than fighting until the pie is destroyed. Yudkowsky argues that this is irrational: if Greg is willing to agree to a 75/25 split, then Heather has no incentive not to adopt such a self-favoring definition of "fairness". (And _vice versa_ if Heather's concept of fairness is the "correct" one.)
-Instead, Yudkowsky argues, Hayley should behave so as to only do worse than the fair outcome if Isaac also does worse: for example, by accepting a 48/32 split in Isaac's favor (after 100−(32+48) = 20% of the pie has been destroyed by the costs of fighting) or an 42/18 split (where 40% of the pie has been destroyed). This isn't Pareto-optimal (it would be possible for both Hayley and Isaac to get more pie by reaching an agreement with less fighting), but it's worth it to Hayley to burn some of Isaac's utility fighting in order to resist being exploited by him, and at least it's better than the equilibrium where the entire pie gets destroyed (which is Nash because neither party can unilaterally stop fighting).
+Instead, Yudkowsky argues, Greg should behave so as to only do worse than the fair outcome if Heather also does worse: for example, by accepting a 48/32 split in Heather's favor (after 100−(32+48) = 20% of the pie has been destroyed by the costs of fighting) or an 42/18 split (where 40% of the pie has been destroyed). This isn't Pareto-optimal (it would be possible for both Greg and Heather to get more pie by reaching an agreement with less fighting), but it's worth it to Greg to burn some of Heather's utility fighting in order to resist being exploited by her, and at least it's better than the equilibrium where the entire pie gets destroyed (which is Nash because neither party can unilaterally stop fighting).
It seemed to me that in the contest over the pie of Society's shared map, the rationalist Caliphate was letting itself get exploited by the progressive Egregore, doing worse than the fair outcome without dealing any damage to the Egregore in return. Why?
[The logic of dump stats](/2023/Dec/agreeing-with-stalin-in-ways-that-exhibit-generally-rationalist-principles/#dump-stats), presumably. Bargaining to get AI risk on the shared map—not even to get it taken seriously as we would count "taking it seriously", but just acknowledged at all—was hard enough. Trying to challenge the Egregore about an item that it actually cared about would trigger more fighting than we could afford.
-In my illustrative story, if Hayley and Isaac destroy the pie fighting, then neither of them get any pie. But in more complicated scenarios (including the real world), there was no guarantee that non-Pareto Nash equilibria were equally bad for everyone.
+In my illustrative story, if Greg and Heather destroy the pie fighting, then neither of them get any pie. But in more complicated scenarios (including the real world), there was no guarantee that non-Pareto Nash equilibria were equally bad for everyone.
I had a Twitter exchange with Yudkowsky in January 2020 that revealed some of his current-year thinking about Nash equilibria. I [had Tweeted](https://twitter.com/zackmdavis/status/1206718983115698176):
It's totally understandable to not want to get involved in a political scuffle because xrisk reduction is astronomically more important! But I don't see any plausible case that metaphorically sucking Scott's dick in public reduces xrisk. It would be so easy to just not engage in this kind of cartel behavior!
-An analogy: racist jokes are also just jokes. Jocelyn says, "What's the difference between a black dad and a boomerang? A boomerang comes back." Keith says, "That's super racist! Tons of African-American fathers are devoted parents!!" Jocelyn says, "Chill out, it was just a joke." In a way, Jocelyn is right. It was just a joke; no sane person could think that Jocelyn was literally claiming that all black men are deadbeat dads. But the joke only makes sense in the first place in context of a culture where the black-father-abandonment stereotype is operative. If you thought the stereotype was false, or if you were worried about it being a self-fulfilling prophecy, you would find it tempting to be a humorless scold and get angry at the joke-teller.[^offensive-jokes-reflect-conceptual-links]
+An analogy: racist jokes are also just jokes. Irene says, "What's the difference between a black dad and a boomerang? A boomerang comes back." Jonas says, "That's super racist! Tons of African-American fathers are devoted parents!!" Irene says, "Chill out, it was just a joke." In a way, Irene is right. It was just a joke; no sane person could think that Irene was literally claiming that all black men are deadbeat dads. But the joke only makes sense in the first place in context of a culture where the black-father-abandonment stereotype is operative. If you thought the stereotype was false, or if you were worried about it being a self-fulfilling prophecy, you would find it tempting to be a humorless scold and get angry at the joke-teller.[^offensive-jokes-reflect-conceptual-links]
[^offensive-jokes-reflect-conceptual-links]: I once wrote [a post whimsically suggesting that trans women should owe cis women royalties](/2019/Dec/comp/) for copying the female form (as "intellectual property"). In response to a reader who got offended, I [ended up adding](/source?p=Ultimately_Untrue_Thought.git;a=commitdiff;h=03468d274f5) an "epistemic status" line to clarify that it was not a serious proposal.
projects / Ultimately_Untrue_Thought.git / commitdiff