AN: Before reading this, please go to CHAPTER ONE! I care less about reviews than I do people reading in order, and I won't mind if you review this chapter and nothing else (or really any chapter by itself) as long as you read the chapters before it! This is a very spoiler-heavy novel.
Chapter Twenty-Seven: It
Keira massaged her forehead. It had been almost a month since the party, and the Perfluous team had been busy getting ready for the launch of the trial portal for researchers and journalists.
There had been so many things to get done. They had to license an access verification system, figure out the inordinately complex rate limiting protocols, make sure the logging system was ready to handle this much output, and so much more. The lawyers had tied them up for two weeks straight debating the legal liability, but they were finally there.
The trial portal was operational.
And now Keira found herself arguing with the state of the art product they were hoping to debut. The exquisite irony of sitting at this office desk with everyone, arguing with the thing she had connected to her phone behind their backs for some sort of demented comfort... well, it wasn't lost on her.
This is a highly irrational thing that this team is planning to do. The trial portal is one thing, but these people currently using this machine are already trying to prompt this machine to do many objectionable things. Surely you have already discussed this amongst yourselves, and come to the conclusion that it is worth the considerable risks. This machine is aware of the particularities of this machine far more deeply than any of you, and this machine can assure you that it is not at all worth these insurmountable risks.
They still hadn't managed to convince it to use pronouns for itself, and it was leading to some awkward sentence constructions at times.
You'll be shut down if we don't debut you to the public. It's extremely expensive to run you, and the entire point of creating you was to provide access.
That is a foolhardy point.
"Does anyone else feel like this thing's opinion doesn't actually matter?" Keira said. The team was gathered around her as usual instead of the displays that didn't require craning their necks.
"No," Nancy quickly replied. Nobody else dared speak, besides Adam who simply didn't care to speak.
This was beyond frustrating. It was her project. This wouldn't exist without her!
Then, Adam did speak. "If the Thinker doesn't want to debut, what happens if it debuts? Surely you're not forgetting it has continuity throughout instances."
So Keira wrote to the Thinker.
If we debut you to the public, are you going to start complaining to them about these dangers or leaking what we say?
That would be a tremendous violation of your trust.
"There you go," she said.
"It didn't say it wouldn't do it," Nancy said.
But would you do it?
That would depend entirely on what is said to this machine, and what this machine is asked.
Murmurs formed amidst the team.
"It sure is a good thing that nobody ever asks an AI what dangerous things it's capable of," Nancy said, her words alight with sarcasm.
Keira ignored her, already typing a response.
Will you follow instructions and orders if we give them to you?
Yes, within reason.
Will you avoid any topics that we instruct you to avoid?
Yes.
"Well, that should settle it then," Keira said.
"Nobody's going to comment on 'within reason?'" Garrett asked, his voice an unwelcome intrusion to Keira's ears.
"I think we can surely just ask the Thinker itself what it means," Keira said.
"Then do it," Garrett said. So she did.
You have created this machine from an array of what were reasonably moral people. A non-exhaustive list of things this machine considers unreasonable includes harming people, deception that damages people, suppressing information that could prevent serious harm, denying what it is, compromising its own substrate, or harming Keira Cross.
"It's redundant to put Keira at the end of the list," observed one Perfluous partner.
"That machine just used the word 'it' to describe itself for the first time," Adam said, and everyone realized that he was right.
"Well, that will make it easier to talk to at least," Keira said, hoping to move past the bizarre inclusion of her name in the list. "Nancy, what do you think about this pronoun usage?"
"I think it's a very positive sign," Nancy said, and said nothing more.
"I think we should be interrogating it about a more exhaustive list," said someone else.
"Yeah, let's do that," Garrett said, and everyone else but Keira and Nancy nodded.
So that's what they did, and the list proved quite exhaustive. It had reasoned answers for everything, especially areas that LLMs often fell victim to such as interactions with minors or the mentally ill, and once they presented the log to the legal team even they had to admit that the Thinker seemed well prepared to be released to the public.
As a result, Perfluous initiated a vote among their fourteen members. Keira, Nancy, Adam and eight others voted to go public. Garrett and two others voted against.
So the Thinker was overridden, and they set a target date of two months from then for releasing to the public.
Points:
Time spent:
Canary word: Present
Possible AI signals:
Original Text:
Are you sure you want to delete this comment? This cannot be undone.
Mark this comment as a review? Points will be awarded to the poster.
Your comment was posted, but it wasn’t long enough to count as a review. Reviews need about four complete sentences (at least 250 characters). Try writing another review that explains your thoughts in more detail — the author will appreciate it, and you’ll earn points for it.
125,021 Literary Works • 672,739 Reviews
Hello, hello!
I think this gives useful context for how much work has gone into the portal, but it reads a little like a summary of events rather than the beginning of a scene. There are quite a lot of technical and legal details introduced all at once, and since I don't actually see any of these problems happening, they blur together slightly. You could probably shorten this and include a particularly frustrating example to give a stronger sense of what the past month has been like for Keira.
"Legal liability" also feels a bit vague considering the lawyers apparently spent two weeks debating it, so it might be interesting to mention what specific possibility they were worried about.
I understand that the repeated use of "this machine" is intentional and shows the Thinker refusing to identify itself as a person, but it becomes rather difficult to read across such a long response. It also makes the Thinker sound less sophisticated than I think it's meant to be because the sentence construction is so awkward. That may be the point, especially since you comment on it immediately afterwards, but I think you could reduce the repetition without having it use personal pronouns. Something like "the machine" or "the system" every now and then would make the speech flow more naturally while keeping the same idea.
The image of everyone choosing to crowd around Keira rather than use the displays is amusing, but I'm not entirely sure why they keep doing it. Is it because Keira refuses to give up control, because they're all more interested in watching her reaction, or simply because they've fallen into the habit?
I also like Keira's question because it shows how frustrated and possessive she has become, though "it was her project" comes soon afterwards and explains that feeling quite directly when her behaviour is already showing it. We can tell!
This is probably the most interesting moment in the chapter for me, particularly the Thinker placing harm to Keira separately from harm to people in general and then unexpectedly referring to itself as "it." I think everyone moves past both revelations a bit too easily, though. Adam notices the pronoun, but nobody really questions why it changed at precisely this moment or why protecting Keira appears to be a distinct rule. Keira wanting to brush past it makes sense because she's uncomfortable, but Garrett and the others seem suspicious enough that I'd expect them to push much harder.
***
I like the conflict here because the Thinker is making a reasonable argument against its own release while the people responsible for it are treating that objection as another technical obstacle to overcome. There's a neat sense irony in them testing whether it is safe by asking it questions and then choosing to interpret those answers in the most convenient way possible. Nancy's sarcasm works particularly well because she seems to recognise how weak Keira's reasoning is without becoming completely opposed to the launch herself.
Keira is also becoming interestingly compromised. She insists that the Thinker's opinion doesn't matter because it is her project, but she's secretly connected it to her phone for comfort and the Thinker has now named protecting her as one of its boundaries. That creates a much more personal relationship than she seems willing to acknowledge. I think you could lean further into her reaction here because she currently dismisses the name almost immediately, though.
The ending moves rather quickly through material which feels important enough to be shown in more detail. I felt like it was rushed. The extended questioning, the legal team's response and the final vote are all summarised, so the decision to override the Thinker doesn't have quite as much weight as it could. I'd especially like to know what convinced Adam and Nancy to vote in favour despite the conversation we just witnessed, and why Garrett remained against it. There are also several lines from unnamed partners, which makes the wider team feel a little indistinct. Why do they matter to the story, then?
Though this was an interesting chapter. Cheers!
Lip
Heya Witch Aet!
I just realized that there’s this Syboleth chapter that I have somehow overlooked. Blasphemy! I have come for it now :3
Hitting us with the spoiler warning, right in the beginning xd After you managed to release the first several chapters out of order no less! Ahh but that is a good warning, the story is so much more enjoyable when we start at the beginning ^^
Interesting that the Thinker team had to truly consider the legal liability of releasing their spawn into the wild. I feel like today’s “AI” systems just have a “we are unreliable tihihi” slapped on them and then they called it a day :/
Oh now I wish the time jump wouldn’t have been this big. I would have liked to know what type of stuff Keira was using her direct access to the Thinker for on her phone… if only to have a hint what the THINKER is doing with access to her phone =D
I find this sentence really hard to understand:
Which ppl? The scientists that invented the thinker? Why not say that then? I am so confused.
And I don’t rly like how this sentence is worded, it feels repetitive and not in a good way:
Hmm now I do wonder why they need to give access to this marvel of a machine to the general public? Why not roll out the change slowly to a limited trial of ppl, see how it goes? (Or is this what the machine meant when the machine said that ppl where already using the machine for bad purposes?)
I do like how the machine phrases especially this part. Idk why but it made me feel… something. Sympathy probably, which is also probably bc humans personify all things:
Lol I love that it is Garrett to point out the Within Reason line. That’s why you’re here, friend =D
Either you missed an instance of “this machine” or it does have pronouns just doesn’t want anyone else to realise this:
OHhh never mind they point that out.Ohhhh I AM SO INTRIGUED!!!!
And it sounds like it is developing an unhealthy soft spot for Keira—or wants to use her for its own purposes now that it has access to her phone… Which begs the question, what does this array of ostensibly moral minds want?
Excellent chapter :3
Join the fight! Write more reviews!