col<<x doesn't mean the same thing as col=col<<x. You need to use col<<=x.

The other code (could) behave differently when compiled with Keil v7 because of the MOV R7,A instruction generated by the first code snippet. This will read the port pins rather than the port latch:

unsigned char col=0xFE;
P0=(P0&0xf8)|(col&7);

180F 74FE MOV A,#0FEH
1811 5407 ANL A,#07H
1813 FF MOV R7,A
1814 E580 MOV A,P0
1816 54F8 ANL A,#0F8H
1818 4F ORL A,R7
1819 F580 MOV P0,A


as opposed to:

unsigned char col=0xFE;
P0=P0&0xf8;

180F 5380F8 ANL P0,#0F8H
P0|=(col&7);
1812 74FE MOV A,#0FEH
1814 5407 ANL A,#07H
1816 4280 ORL P0,A

I wouldn't consider this to be a bug as there is no method that I am aware of in 'C' to specify whether you want to read the latch or the pins.