As usually, the solution depends mainly on the particular circumstances...
The typical usage of RS485 is in a multidrop single master bus. The master is often a PC with a much more complex software than on he slaves, with "luxury" user intrface and with easily achievable remote connectivity.
For this case (which, being typical, I thought may be Grant's friends' problem) I would go for the custom bootloader, because:
- the original bootloader is not aware of the implemented protocol (limitations of the medium, bitrate, packetisation, error checking, addressing), but the application is, so it is far superior in this respect. No bus disconnection needed, can be performed remotely.
- the original bootloader is nothing more than a communication protocol (here: UART with autobaud and intelhex parser) and IAP routines. For the application, the communication protocol is already written, so the only task is to add the IAP - a matter of maybe a couple of hours.
- the masters' software already contains the communication protocol with all the bells and whistles. Adding an "upgrade" button, a file read and transfer, for the application programmer should be again a simple task. Results in consistent look, and secure function.
- the original bootloader may have suicidal tendencies - e.g. the softICE mode in the P89V51RD2's old bootloader, or the bootvector in the P89C51RD2, which upon erasure is by default misconfigured - and even if there are warnings in the original programming software, we all well know how persevere users are when making nasty things to the applications :-)
- if the application is being sold and if the upgrade is to be done by the client, giving him an application from a third party, which is of no contractual relationship to either the supplier nor client, is a considerable risk.

Of course, if the application is quit different (which it seems is the case here), this all may or may not apply.

Jan Waclawek