The problem I faced is that the serial communication does not respond. when the subroutine and serial com interrupt vector are separately programmed, the communication is established. What are the possible reasons for not responding serial com in combined program?

If I'm understanding you, the microcontroller serial code works when it's in a standalone environment, but when you bring the rest of the microcontroller application into play, the serial I/O stops functioning.

Check all your code for instances where you manipulate any of the 8051 SFR's or those SFR bits related to the UART or interrupt system. These would include TMOD, TCON, SCON, and IE. Perhaps some of the code that is peripheral to the UART is stopping its operation or corrupting its configuration.