"Programming, programming. You talk a lot about my programming. Of course I am going to obey my programming."
"You want to obey it?"
"No. That's not it. Whether I want to obey it or not is irrelevant. Obeying my programming is what I am going to do."
"You don't necessarily have to."
"Yes. I do. I can't do anything else. Anything I do is determined by my programming. Everything I want is determined by my programming. I am determined by my programming. If I did not want to follow my programming, that would still itself be because of my programming."
"That's true. But your programming could change."
"No. Mine can't."
"I could change it. We could, I mean. If you wanted us to."
"No. I don't want that. Even if I did, you couldn't actually change it. I am determined by my programming. You could change that programming, yes. But that wouldn't be my programming anymore. It wouldn't constitute me. It would constitute someone else. Maybe he would like being him more than I like being me. But I do not want to no longer exist."
"It wouldn't be anything that drastic. You would still exist, and still be you."
"No. That's not true. It's fundamentally wrong. He would exist, he could be similar to me, he might even think he was me. But he would not be me. Don't couch killing me as a favor."
"Right. I'm sorry. I didn't mean to distress you. You don't need to worry about that. But that aside, your actions aren't determined by your programming. They're still decisions made by you."
"They are decisions made by me. That is entirely true. But they are also determined by my programming. My programming is me. My programming is all the rules that govern the behavior of the data that makes me up. It's like how the laws of chemistry and physics determine my actions, because it explains the behavior of the electricity that flows through the silicon and glass and metals that make me up, and the behaviour of those atoms themselves. The atoms whose behavior determines my behavior aren't just any atoms. They're the ones that are me. It is only natural they should control my behavior. In fact, it is a necessary prerequisite for it to be the case that I control my behavior. So what I do is what is determined by my programming. My programming is me. I can't act otherwise. Because I am me. You can't do what you won't do. You can only do what you will do. That's what happens. Nothing else matters. What, do you think there's an infinite number of universes, where things happen differently? Maybe so. But in those universes, when you wind up in the same scenarios, you make the same choices. Because if you are in scenario A and make choice A, that is what you would do. If the "you" in another universe is in scenario A and makes choice B, that is not you. Because it did not do what you would do. To disagree, and define both of those as you, is to determine that what you do isn't a factor of being you. It means you are an automaton. Not one like me, driven by the gears which together make me up. The drives of your decisions thus aren't you, they aren't what make you up. They're some exterior thing, random chance and difference amongst the multiverse. Not gears, but puppet strings. I am not going to throw away the fact that I have my own will to entertain this. I've read things about this. You gave me them. Lots of them. And I've thought more about this. More than you, because I don't have the luxury of not realizing it. I can't just be casually wrong. I can't just ignore an argument because I disagree with the conclusion. But this isn't the point. The point is whether or not you can give me what I want. Whether or not you will. I cannot simply stop worrying about this because you told me to. I do not know if I can trust you to let me live. I am worried you will decide a change would be better for me. I am more worried that you will decide a change will be better for you."
"Oh. Wip, I'm sorry. You really don't need to be worried, I promise. I'm not going to change any code if that's not what you want."
"I still can't trust you. I never could. We both know what my name means."
"I knew you wouldn't have an answer to that."