When I used my finger to touch the stepper motor, it seemed to vibrate on it own and I felt that it stepped discontinued. How can I make it to run more smoothly?

Seems you are applying wrong sequence to the windings.

In addition, how can I increase the speed of the motor?

There is a definite limit on the speed. Check the data for specific motor you are suing.

What is the use of the crystal between the pins XTAL1 and XTAL2 of 8052? If I replaced it into higher frequency, then can I increase the speed of the motor?

Well, a crytal provides basic clock, needed for operation of any microprocessor. You can't increase it beyond a certain maximum limit, which is 8051 derative dependent.

AS far as speed and smoothness is concerned, I am sure it is all due to wrong sequencing or a faulty motor.