I think you'd be very hard pushed to do this with an 8051.

You're talking about moving a lot of data about, and I think the old DPTR could be a real problem there.

A previous client did this sort of thing: they were significant projects with Motorola 68360 32-bit processor with the specific microcode, megabyte of RAM, megabyte of ROM, etc

Even BRI stuff tends to use that sort of processor.

However, I note that the Triscend E5 has an HDLC controller in its hardware library...