Hmm, first time I've posted three times in one day in a while... but this quote from Sewer, Gas & Electric was just too juicy to pass up:
"Selective self-deception is the first symptom of true intelligence."
In the context of the story, a sentient AI is describing to a human why it decided to commit genocide. The reason was that it heard an order that could have been interpreted in one of two ways; it chose the more interesting interpretation.
This sounds consistent with the theory that intelligence first evolved from the extension of instinct into anticipation, and eventually into the ability to create mental models. For if one is able to consciously conceive of something (or a state of being) that is nonexistent, they should be able to act on that model as if it were reality.
Pretty neat to think about. Makes me want to read Pinker again (still haven't picked up Blank Slate). Makes me wonder what I've been missing being off all the wacko extropian/transhuman mailing lists these past few years.