I've got the power

In these eco-days of ours, with all sorts of trouble on the horizon for our planet, we're all trying to do our little bit for the environment. However, there's one thing I've never understood, and that's turning all the plug sockets off when you're not using them.

In my first year of college, I stayed in a family home and the whole family, far ahead of themselves ecologically, had what can only be described as an obsession with switching all the plug sockets to 'off' when they weren't being used.

Now, I'm not entirely stupid. I fully understand that that a TV on standby will use energy, as will a phone left on charge all night - the energy is clearly evident in the heat produced by both of these. I do have a limited grasp of Physics, and my understanding is that energy doesn't just go into thin air - it has to be transformed into something. Heat, light, sound, movement for instance.
My question is this - where is the energy going from a toaster left plugged into the mains? Surely that's not actually using energy, and there is therefore no reason to obsess about turning the switch off at the mains? Even more so for a plug socket with nothing plugged into it.

If all the sockets in my house are using energy by just being there, then surely I'm better off switching my whole house off when I leave for work in the morning?