I was thinking it had to do with not phisically being in the scene but then I realized that when you look through a window at an interior you don't have this effect either.
The only, far fetched, reason I can think of is that we dont just measure distance by perspective but also by measuring the angle between both our eyes and the amount we have to focus to get it sharp. Because we don't have that info when we look at a computer screen our brains do not correct it.
I can think of an experiment with track IR. When it would be possible to blur the entire scene except for the 3d-object that is in the line of focus, maybe part of the illusion is restored.