NICK J. HAY

An avid stamp collector, who is also an AI enthusiast, decides to build a stamp collecting device. This margin is too small for the details, but the idea is simple:

1. The device will be active for one year. It is connected to the internet, from which it sends and receives packets.
2. The device has an internal model of the universe. This model captures how likely each state of the world is, can predict future packets received, and can simulate the effect of packets sent.
3. For every possible sequence of packets, the model extrapolates the final state and counts the number of stamps collected.
4. The device outputs the sequence leading to the largest number of stamps.

...Infecting the internet to collect stamps seems stupid. Won't the device realize this wasn't what the stamp collector had in mind? Won't it ask whether there is a better goal than collecting stamps? Isn't it satisfied with 170,000 stamps?
 
But we can answer this question, we have the device's complete design. It doesn't ask itself questions. It doesn't think at all. It simply selects the output maximizing the number of stamps. The device is not well understood by analogy to humans.
 
So, the stamp collector powers up the device. And the world stops, filled with stamps.