Taking 'easdy' numbers: at ~500mm sec movement and 5mm object you have ~10msec to read

You use a L/F with 24kHz max which in the best case (which will never/rarely happen) give you 240 count in the window. This, dramatically, reduces your resolution.

so, what you need to state, in order for us/me to help, is
a) what is the smallest difference in color you need to 'catch' a simple way to communicate this would be the HTML codes for neighboring colors you need to distinguish.
b) with the illumination you have on the object, your filter and sensor what count do you catch for a pure red object.

more questions may come, but till the above is answered I can not see a way to help you.

Erik