The profile text indicated that Chloë was on the newly formed capability risk evaluations team. Jake groaned. _Yuddites._ Fears of artificial intelligence destroying humanity had recently been trending in the media (social and otherwise). In response, Magma had commissioned a team with the purpose to monitor and audit the company's AI projects for the emergence of unforeseen and potentially dangerous capabilities, although the exact scope of the new team's power was unclear and probably subject to the outcome of future intra-company political battles.
-Jake took a dim view of the AI risk crowd. Given what deep learning could do nowadays, it didn't feel quite right to dismiss their doomsday stories as science fiction, exactly, but Jake maintained it was the _wrong subgenre_ of science fiction. His team was building the computer from _Star Trek_, not the Blight from _A Fire Upon the Deep_: tools, not creatures. Despite the brain-inspired name, "neural networks" were ultimately just a technique for fitting a curve to training data. If it was counintuitive how much you could get done with a curve fitted to _the entire internet_, previous generations of computing pioneers must have found it equally counterintuitive how much you could get done with millions of arithmetic operations per second. It was a new era of technology, not a new era of life.
+Jake took a dim view of the AI risk crowd. Given what deep learning could do nowadays, it didn't feel quite right to dismiss their doomsday stories as science fiction, exactly, but Jake maintained it was the _wrong subgenre_ of science fiction. His team was building the computer from _Star Trek_, not the Blight from _A Fire Upon the Deep_: tools, not creatures. Despite the brain-inspired name, "neural networks" were ultimately just a technique for fitting a curve to training data. If it was counterintuitive how much you could get done with a curve fitted to _the entire internet_, previous generations of computing pioneers must have found it equally counterintuitive how much you could get done with millions of arithmetic operations per second. It was a new era of technology, not a new era of life.
It was because of his skepticism rather than in spite of it that he had volunteered to be the Multigen team's designated contact person for the risk evals team (which was no doubt why this Chloë person had messaged him). No one else had volunteered at the meeting when it came up, and Jake had been slightly curious what "capability risk evaluations" would even entail.
------
-"And so just because an AI seems to behaving well, doesn't mean it's aligned," Chloë was explaining. "It might just be playing along as an instrumental strategy, hoping to pull off a treacherous turn later."
+"And so just because an AI seems to behaving well, doesn't mean it's aligned," Chloë was explaining. "If we train AI with human feedback ratings, we're not just selecting for policies that do tasks the way we intended. We're also selecting for policies that _trick human evaluators into giving high ratings_. In the limit, you'd expect those to dominate. 'Be good' strategies can't compete with 'Look good' strategies in a looking-good contest—but in the current paradigm of deep learning, looking good is the only game in town. We don't know how these systems work in the way that we know how ordinary software works; we only know how to train them."
-"So then we're just screwed, right?" said Jake in the tone of an attentive student. They were in a conference room on the Magma campus on Monday. After fixing the logging regex and overwriting the evidence with puppies, he had spent the weekend catching up with the "AI safety" literature. Honestly, some of it had been better than he expected. Just because Chloë was nuts didn't mean her co-ideologues didn't have any valid points to make about future systems.
+"So then we're just screwed, right?" said Jake in the tone of an attentive student. They were in a conference room on the Magma campus on Monday. After fixing the logging regex and overwriting the evidence with puppies, he had spent the weekend catching up with the "AI safety" literature. Honestly, some of it had been better than he expected. Just because Chloë was nuts didn't mean her co-ideologues didn't have any potentially valid points to make about future risks.
"I mean, probably," said Chloë. She was beaming. Jake's plan to distract her from her investigation by asking her to bring him up to speed on "AI safety" seemed to be working perfectly.
-"But not necessarily," she continued. There are a few avenues of hope—at least in the not-wildly-superhuman regime. One of them has to do with the topology of policies and the fragility of deception.
+"But not necessarily," she continued. There are a few avenues of hope—at least in the not-wildly-superhuman regime. One of them has to do with the fragility of deception.
-"The thing about deception is, you can't just lie about the one thing. Everything is connected to each other in the Great Web of Causality. If you lie about one thing, you also have to lie about the evidence pointing to that thing, and the evidence pointing to that evidence, recursively covering up the coverups. For example ..." she trailed off. "Sorry, I didn't rehearse this; maybe you can think of an example."
+"The thing about deception is, you can't just lie about the one thing. Everything is connected to each other in the Great Web of Causality. If you lie about one thing, you also have to lie about the evidence pointing to that thing, and the evidence pointing to that evidence, and so on, recursively covering up the coverups. For example ..." she trailed off. "Sorry, I didn't rehearse this; maybe you can think of an example."
Jake's heart stopped. She had to be toying with him, right? Indeed, Jake could think of an example. By his count, he was now three layers deep into his stack of coverups and coverups-of-coverups (by writing the bell character bug, attributing it to Code Assistant, and overwriting the incriminating videos in the object storage cluster with puppies). Four, if you counted pretending to give a shit about "AI safety". But now he was done ... right?
But the object storage API probably provided a way to edit the metadata and update the last-modified time, right? (The analogue of `touch -d` on Unix-like systems.) This shouldn't even count as a fourth–fifth coverup; it was something he should have included in his script to write the puppy videos.
-"Sorry, I'm not sure what you mean," he said to Chloë, as he brought up the object storage API docs on his laptop. If she noticed the change in his manner, he didn't notice her noticing.
+"I think I get the idea," said Jake in his attentive student role. "I'm not seeing how that helps us. Maybe you could explain." While she was blathering, he could multitask between listening, and looking up how to edit the last-modified timestamps.
-"Okay, so [this example comes from Paul](https://www.greaterwrong.com/posts/AqsjZwxHNqH64C2b6/let-s-see-you-write-that-corrigibility-tag/comment/8kPhqBc69HtmZj6XR)," she said. Jake felt a small flicker of distaste towards the practice of referring to researchers by their first name (which he had seen a lot of in the blog posts he had read over the weekend, and which struck him as uncouth), but mentioning it wouldn't be a smart play.
+"Right, so if a model is behaving well according to all of our deepest and most careful evaluations, that could mean it's doing what we want, but it could be elaborately deceiving us," said Chloë. "Both policies would be highly rated. But the training process has to discover these policies continuously, one gradient update at a time. If the spectrum between the 'honest' policy and a successfully deceptive policy consists of less-highly rated policies, maybe gradient descent will stay—or could be made to stay—in the valley of honest policies, and not climb over the hill into the valley of deceptive policies, even though those would ultimately achieve a lower loss."
-"Suppose your household robot misbehaves. Say it accidentally breaks your favorite vase."
+"Uh huh," Jake said unhappily. The object storage docs made clear that the `Last-Modified` header was set automatically by the system; there was no provision for users to set it arbitrarily.
-"'Accidentally'? Isn't that anthropomorphizing?" Jake asked. A questionable play—his attentive student role didn't call for that much skepticism, but he was somewhat distracted by his docs-search multitasking and forgot to avoid reacting naturally.
+"Imagine you're training your AI to act as a general home assistant, to run everything in a user's household in a way that the user rates highly," Chloë continued. "If—I don't know, say, the family cat dies, that's a negative reward—maybe a better feeding schedule or security monitoring could have prevented it. But if the cat dies and the system _tries to cover it up_—says the cat is out for a walk right now to avoid telling you the bad news—that's going to be an even larger negative reward when the deception is discovered. If the system is trained to pass rigorous evaluations, a deceptive policy has to do a lot more work, different work, to pass the evaluations. In the limit, that could mean—using Multigen-like capabilities to make videos to convince you that the cat is there while you're on vacation? Constructing a realistic cat-like robot to fool you when you get back? Maybe this isn't the best illustrative example. The point is—"
-"Sorry, that's inessential—and really, in the most worrisome scenarios, it would be intentional. What matters is that the robot wants—I mean, is optimizing for—your approval, and it knows that you would disapprove if you knew what it had done.
+Jake finished the thought. "Small, 'shallow' deceptions aren't stable. The set of policies that do well on evaluations comes in two disconnected parts: the part that tells the truth, and the part that spins up a false reality, as intricate as it needs to be to protect itself. If you're going to fake anything, you need to fake deeply."
-We can imagine a spectrum of possible resposnes. Given that the deed was done, you'd prefer that it fess up, tell you about the vase.
+"Exactly, you get it!" Chloë was elated. "You know, when I called you last week, I was worried you thought I was nuts to entertain any worry at all about Code Assistant. But you see the value of constant vigilance now, right? If the landscape of policies looks something like what I've described, catching the precursors to deception early could be critical—to raise the height of the loss landscape between the honest and deceptive policies, before frontier AI development plunges into the latter."
-But if could also try to hide evidence.
+Jake sighed. The `Last-Modified` header probably wasn't critical. With some effort and smart play, it seemed plausible that he could stay ahead of Chloë's investigation, and even encourage her to give up sooner and stop wasting her time. But something about this wrong-sci-fi-subgenre worldbuilding exercise was getting to him. He wasn't scared of the robot apocalypse any time soon. But if it really was one or the other—a policy of truth or a policy of lies—he had a feeling about which one he wanted to practice.
-[...]
-
-SGD has to discover these policies "continuously", one gradient update at a time.
-
-The honest policy and the deceptive policy
-
-[TODO—
- * "Maybe not." There are two ways to pass all the evals: do things the right way, or be pervasively deceptive. The thing is, policies are trained continuously via gradient descent. The purely honest policy and the purely deceptive policy look identical on evals, but in between, the model would have to learn how to lie, and lie about lying, and cover-up coverups. (Chloë lapses into Yuddite speak about the "Great Web of Causality.") Could we somehow steer into the honest attractor?
- *
- * That's why she believes in risk paranoia. If situational awareness is likely to emerge at some point, she doesn't want to rule it out now. AI is real; it's not just a far-mode in-the-future thing.
- * Jake sees the uncomfortable analogy to his own situation. He tries to think of what other clue he might have left, while the conversation continues ...
- * The Last-Modified dates! They're set by the system, the API doesn't offer a way to backdate them.
- * Maybe she won't notice the dates? Possible (but someone persnickety enough to have found the log discrepancy would probably check)
- * Merely noticing the dates won't directly implicate him (the way the video screaming "Jake!" would), although it would indicate the poltergeist covering its tracks from the investigation (rather than just wanting puppies in the first place)
- * Are the buckets versioned?! Probably not, right—it would be wasteful to version a video bucket. On the other hand, Multigen isn't supposed to write twice, maybe someone left versioning on as a default template ...
- * They did.
- * Jake muses philosophically about the analogy, and say something ambiguously indicating intent to come clean.
- THE END
-]
+"Chloë," he said. "I, um, might have something to tell you ..."
projects / Ultimately_Untrue_Thought.git / commitdiff