I think you will get it easier by:
- capturing the serial input as binary (I believe every reasonable terminal program is able to do it), perhaps simultaneously displaying it as hexadecimal (e.g. realterm.sf.net is able to do this)
- then convert it into hexadecimal format using some suitable tool.

A command-line tool to do this can be stitched together within 5 minutes. For example in Pascal:

var fi: file of byte;
    fo: text;
    b: byte;

function dec2hex(b:byte):string;
const nib2hex:array[0..$F] of char = 
('0','1','2','3','4','5','6','7','8','9','A','B','C','D','E','F');
begin
  dec2hex:=nib2hex[b shr 4]+nib2hex[b and $F];
end;

begin {main}
  assign(fi,paramstr(1));
  assign(fo,paramstr(2));
  reset(fi);
  rewrite(fo);
  while not eof(fi) do begin
    read(fi,b);
    write(fo,dec2hex(b),' ');
  end;
  close(fi);
  close(fo);
end.

Jan Waclawek