Me: I want a pony.
Eliezer: I’m not sure if we can afford a pony. How much does a pony cost? Maybe I can get you a pony if ponies are cheap on Craigslist.
Me: Thank you, but I wasn’t making a request. I was merely trying to create common knowledge about my desires.
Eliezer: I don’t see the distinction. When I respond to a request, I am responding to new information about what you want.
Me: No.
Eliezer: No? I think yes.
Me: No.
It’s important that we have different concepts for what I want independently of you, what you want independently of me, and what our joint extrapolated volition prescribes. If you respond to your belief about my belief about what our joint extrapolated volition prescribes, when I’ve intended only to provide evidence of my independent desires, then… problems.
Like, maybe I’ll say that I want a pony, because I wonder if you have insight into alternative ways of satisfying the need responsible for that desire, and you’ll try to get me a pony, when I don’t actually think it’s a good idea for us to have a pony.
Eliezer: Oh! Yes, that makes sense. That is a thing in Bayesian networks.
Me: Oh?
Eliezer: When stating your independent desires, you should begin by saying “lambda”. Refer to my independent desires with “pi”. And call the prescription of our joint extrapolated volition “BEL”.
Me: Do you want to explain to me where those terms come from?
Eliezer: *begins to explain belief propagation*
Me: Wait. That's not what I meant. Let me try that again.
Do you pi want to explain the thing?
Eliezer: *thinks* That sounds fun, so yes.
Me: Oh good! Then I BEL want you to explain the thing.
Eliezer: *explains belief propagation*
