Hallo Sachin,

several reasons for your failure are possible:

1. Software bug, which causes the 'memory blanking', whenever RS485 communication is invoked. Think e.g. of a wrong jump instruction, which erroneously starts an initialization routine, or something else.

2. Dips on power supply voltage, which results in a regular reset generated by MAX691. This can easily be measured by the help of oscilloscope. Have a careful look at power supply voltage and look for dips, spikes etc.

3. Noise and interference, which is somehow coupled into your microcontroller board and which causes noisy supply voltage. Path of such noise coupling can be the many cables, which enter or leave your board. Star ground routing, shielding and individual filtering at every input and output will help.

4. May be a ground loop is formed, which introduces interference via signal cables into your board. You told, that your board is connected to PC via RS485. So, your board gets an earth connection through PC. If there's another ground connection from your board to some earthed equipement, a ground loop is formed. If your motor is a strong one, lots of ground noise can be induced.

5. Not only dips on supply voltage can cause your reset controller to reset your system, but also ground noise!! If your arrangement of connectors is such, that interference currents can flow across your board, then lots of ground noise can develop. To avoid this, it's extremely important to arrange the board in such way, that all connectors are located at ONE edge of PCB:




Then, it's impossible that noise and interference injected into cable shields will cause a current to flow across the whole board. It's important to ty all cable shields to a local ground plane right at that edge of board. If EMI filters are used connect their grounds also to this ground plane.
To this local ground plane 0V of power supply must be connected, right there, where it enters the board. This point is the 'ground star point'.
If you put your board into a metall enclosure, then all the cable shields should be connected to the chassis at this place. Chassis forms 'chassis ground plane' or 'RF ground plane'. Of course, chassis ground plane must have a connection to 'ground star point': Connect 0V of power supply to local ground plane (of PCB), as close as possible to chassis, but inside metall enclosure. Chassis should be connected to local ground plane by many connections, screws, solder points, etc. Especially there, where 0V of power supply is connected to local ground plane.

By this it's nearly impossible that noise on cable shields will enter your board!

This arrengement also helps with ground loops. Loop current is no longer flowing across whole PCB and developing ground noise. Especially PCBs not containing solid ground plane will highly profit from this!

6. I do not exactly know, how you control the motor. But in any case, there should be a snubber across motor terminals: Whenever you switch-off the motor very high voltage spikes are generated, which are immediately directed to your board, if no snubber is present!! Keep in mind, that the snubber should be located directly at motor. By this, the loop area formed by the resulting spike current is heavily limited. If the snubber would be located at your board, then loop area of spike current is much wider. As consequence spike current would flow along the whole cable, which finally acts like an antenna emitting lots of interference!!
So, the snubber should be located directly at motor. Cable to board should be 'twisted pair like', in order to reduce loop area. This minimizes interference, whenever your motor is switched-on by the relay sitting on your board.

7. Sometimes choose of slew rate limited RS485 drivers like MAX483, or else, can help. By the way, do you have the line properly terminated at both ends?

Hope this helps a bit.

Kai