_Tranny or real?_ Jake wondered, clicking on her profie.
-The profile text indicated that Chloë was on the newly formed capability risk evaluations team. Jake groaned. _Yuddites._ Fears of artificial intelligence destroying humanity had been trending on social and traditional media lately. Magma had commissioned a team with the purpose to monitor and audit the company's AI projects for the emergence of unforeseen and potentially dangerous capabilities, although the exact scope of the new team's power was unclear and probably subject to the outcome of future intra-company political battles.
+The profile text indicated that Chloë was on the newly formed capability risk evaluations team. Jake groaned. _Yuddites._ Fears of artificial intelligence destroying humanity had recently been trending in the media (social and otherwise). In response, Magma had commissioned a team with the purpose to monitor and audit the company's AI projects for the emergence of unforeseen and potentially dangerous capabilities, although the exact scope of the new team's power was unclear and probably subject to the outcome of future intra-company political battles.
Jake took a dim view of the AI risk crowd. Given what deep learning could do nowadays, it didn't feel quite right to dismiss their doomsday stories as science fiction, exactly, but Jake maintained it was the _wrong subgenre_ of science fiction. His team was building the computer from _Star Trek_, not the Blight from _A Fire Upon the Deep_: tools, not creatures. Despite the brain-inspired name, "neural networks" were ultimately just a technique for fitting a curve to training data. If it was counterintuitive how much you could get done with a curve fitted to _the entire internet_, previous generations of computing pioneers must have found it equally counterintuitive how much you could get done with millions of arithmetic operations per second. It was a new era of technology, not a new era of life.
Thus, by carefully prefixing his requests with the bell character, Jake could make all the custom videos he wanted, with no need to worry about explaining himself if someone happened to read the logs. It was the perfect crime—not a crime, really. A precaution.
-But now his precaution had been discovered! So much for his career at Magma. But only at Magma—the industry gossip network wouldn't prevent his employment elsewhere ... right?
+But now his precaution had been discovered! So much for his career at Magma. But only at Magma—the industry gossip network wouldn't prevent him from landing on his feet elsewhere ... right?
Chloë was explaining the bug. "... and so, if a client were to send a request starting with the ASCII bell character—I know, right?—then the request wouldn't be logged."
"Exactly!" said Chloë. "_Who knows what it was thinking?_ That's what I wanted to talk to you about!"
-"Uh ..." Jake balked. If he hadn't been found out, why _was_ someone from risk evals talking to him about a faulty regex? The smart play to minimize his chances of being discovered would be to disengage as quickly as possible, rather than encourage inquiry about the cause of the bug, but his curiosity was piqued by the possibility that Chloë was implying what he thought she was. "You're not suggesting Code Assistant might have introduced this bug on purpose?"
+"Uh ..." Jake balked. If he hadn't been found out, why _was_ someone from risk evals talking to him about a faulty regex? The smart play would be to disengage as quickly as possible, rather than encourage inquiry about the cause of the bug, but he was intrigued by the possibility that Chloë was implying what he thought she was. "You're not suggesting Code Assistant might have introduced this bug on purpose?"
She smirked. "And if I am?"
"That's absurd. It's not an agent that wants things. It's an autoregressive language model fine-tuned to map ticket descriptions to code changes."
-"And humans are just animals evolved to maximize inclusive genetic fitness. If evolution could hill-climb its way into creating general intelligence, why can't gradient descent? I don't think humanity should be playing with AI at our current level of wisdom. But if it's happening anyway, thanks to the efforts of people like you"—okay, _now_ her tone was accusatory—"it's my heroic responsibility to exert constant vigilance. To monitor the things we're creating and be ready to sound the fire alarm, if there's anyone sane left to hear it."
+"And humans are just animals evolved to maximize inclusive genetic fitness. If natural selection could hill-climb its way into creating general intelligence, why can't stochastic gradient descent? I don't think it's dignified for humanity to be playing with AI at all given our current level of wisdom, but if it's happening anyway, thanks to the efforts of people like you"—okay, _now_ her tone was accusatory—"it's my heroic responsibility to maintain constant vigilance. To monitor the things we're creating and be ready to sound the fire alarm, if there's anyone sane left to hear it."
-Jake shook his head. These Yuddites were even nuttier than he thought. "And your evidence for this is, what? That the model wrote a silly regex?"
+Jake shook his head. These Yuddites were even nuttier than he thought. "And your evidence for this is, what? That the model wrote a silly regex once?"
"And that the bug is being exploited."
Jake's blood flash-froze. "Wh—what?"
-Chloë pasted two more links into the chat, this time to Magma's log viewer. "Requests go through a reverse proxy before hitting the Multigen service itself. Comparing the two, there are dozens of requests logged by the reverse proxy that don't show up in Multigen's logs—starting just after the bug was deployed. The reverse proxy logs include the client IP, which is inside Magma's VPN, of course"—Multigen wasn't yet a public-facing product—"but don't include the request data or user auth, so I don't know what the client was doing specifically—which is apparently just what they, or it, wanted."
+Chloë pasted two more links into the chat, this time to Magma's log viewer. "Requests go through a reverse proxy before hitting the Multigen service itself. Comparing the two, there are dozens of requests logged by the reverse proxy that don't show up in Multigen's logs—starting just minutes after the bug was deployed. The reverse proxy logs include the client IP, which is inside Magma's VPN, of course"—Multigen wasn't yet a public-facing product—"but don't include the request data or user auth, so I don't know what the client was doing specifically—which is apparently just what they, or it, wanted."
-Jake silently and glumly reviewed the logs. The timestamps were consistent with his video requests. He remembered that after one of his coworkers (Elaine, as it turned out) had approved the doctored Code Assistant pull request, he had eagerly waited for the build automation to deploy the faulty commit so that he could try it out as soon as possible.
+Jake silently and glumly reviewed the logs. The timestamps were consistent with when he had been requesting videos. He remembered that after one of his coworkers (Elaine, as it turned out) had approved the doctored Code Assistant pull request, he had eagerly waited for the build automation to deploy the faulty commit so that he could try it out as soon as possible.
-Finally, he said, "You really think Code Assistant did this? 'Deliberately' checked in a bug, and then exploited it to secretly request some image or video generations? For some—'reason of its own'?"
+_How did you even find this?_ he wanted to ask, but that didn't seem like a smart play. Finally, he said, "You really think Code Assistant did this? 'Deliberately' checked in a bug, and then exploited it to secretly request some image or video generations? For some 'reason of its own'?"
"I don't know anything—yet—but look at the facts," said Chloë. "The bug was written by Code Assistant. Immediately after it gets merged and deployed, someone apparently starts exploiting it. How do you think I should explain this?"
-There was, actually, a perfectly ordinary explanation that had nothing to do with Chloë's delusional wrong-kind-of-science-fiction paranoia—and Jake's career depended on her not figuring it out.
+For a moment, Jake thought she must be blackmailing him—that she knew his guilt, and the question was her way of subtly offering to play dumb in exchange for his office-political support for anything risk evals might want in the future.
-"I ... don't know," he said. It suddenly dawned on him that staying in this conversation was not a smart play. "You know, I actually have another meeting to get to," he lied. "I'll fix that regex today. I don't suppose you need anything else from me—"
+That didn't make sense, though. Anyone who could recite Yuddite cant with such conviction (not to mention the whole pretending-to-be-a-woman thing) clearly had the true-believer phenotype. This Chloë meant exactly what she said.
-"Actually, I'd like to know more about Multigen—and I'll likely have more questions after I talk to the Code Assistant team. Can I pick a time on your calendar next week?"
+How did he think she should explain this? There was, actually, a perfectly ordinary explanation that had nothing to do with Chloë's wrong-kind-of-science-fiction paranoia—and Jake's career depended on her not figuring it out.
-"Sure. Talk to you then. Nice to meet you. Goodbye." He hung up.
+"I don't know," he said. It suddenly dawned on him that staying in this conversation was not a smart play. "You know, I actually have another meeting to get to," he lied. "I'll fix that regex today. I don't suppose you need anything else from me—"
-_Shit!_
+"Actually, I'd like to know more about Multigen—and I'll likely have more questions after I talk to the Code Assistant team. Can I pick a time on your calendar next week?" It was Friday.
+"Sure. Talk to you then—if we humans are still alive, right?" Jake said, hoping that a touch of humor would be found endearing—and only realizing in the moment after he said it what a terrible play it was; Chloë was likely to take it as mockery.
+
+"I hope so," she said solemnly, and the call ended.
+
+_Shit!_ How could he have been so foolish? It had been a specialist's blindness. He worked on Multigen. He knew that Multigen logged requests, and that people on his team occasionally had reason to grep those logs. He didn't want anyone knowing what he was asking Multigen to do. So he had arranged for his requests to not appear in Multigen's logs, thinking that was enough—or rather, without thinking about whether or not that was enough.
+
+_Of course_ it wasn't enough! He hadn't considered that Multigen would sit behind a reverse proxy with its own logs. He was a research engineer, not a devops guy; he wrote code, but thinking about how and where the code would actually run had always been someone else's job.
+
+It got worse. When the Multigen web interface supplied the user's requested media, that data had to live somewhere. The _videos themselves_ would still be on Magma's object storage cluster! How could that have seemed like an acceptable risk? Jake struggled to recall what he had been thinking at the time. Had he been too horny to even consider it?
+
+No. It had seemed safe at the time because videos weren't greppable. They would be saved in object storage under uninformative file names based on the timestamp and a random UUID.
[TODO—
* Jake is very nervous; he thought deleting the Multigen logs would be enough (the videos are also stored in object storage, but there's no particular reason to expect a human to be combing through the raw files ... but they will, if there's an investigation
+
* He sets up another meeting with the Evals team member, to try to suss out what her plans are, to stall—but ostensibly, to get up to speed on her risk concerns
* Scene break: at the meeting, she's explaining Christiano's idea about there being a basin of policies that admit their mistakes, rather than using deception to get a high score
* Jake sees the analogy to his own behavior
projects / Ultimately_Untrue_Thought.git / commitdiff