Ah- Think of the 5V as an input, it's being supplied by the USB while plugged in. When there's no 5V source attached, you'll see some sort of voltage, but it's not useful as a source, and definitely won't be 5V if the other source is a battery below 5V.
The right spot to power an IR sensor from is VCC(main/processor voltage), but you'll definitely need a sensor that works at your battery voltage.