depending on how secure you want it,I would forget about public key stuff because the 8052 is far too slow to handle the key management,AES or Triple DES in symetric cbc mode can both use 128 bit keys and I have got versions of both which run in xilinx spartan FPGAs
Dont go for these mickey mouse routines which nobody knows how secure they are, AES is HUGE in terms of software but is about the most secure as far as anyone knows.