Your scheme to split the data as 4 bits and 8 bits will not work because there are many instances when the upper bits of the 8-bit sample byte will be zeros just like the upper 4 bits of the 4-bit sample byte.

It would be far better for you to read the A/D value at the source end and split the 12-bit result into two 6-bit samples. Then clear the upper two bits on one sample to indicate that it is the low 6 bits sample byte. Similarly set the upper two bits of the other sample to indicate that it is the high 6 bits sample byte. Thus what you send out is two bytes formatted as:

High 6 bits sample byte:
+-----+-----+-----+-----+-----+-----+-----+-----+
|     |     |     |     |     |     |     |     |
|  1  |  1  | ad11| ad10| ad 9| ad 8| ad 7| ad 6|
|     |     |     |     |     |     |     |     |
+-----+-----+-----+-----+-----+-----+-----+-----+

Low 6 bits sample byte:
+-----+-----+-----+-----+-----+-----+-----+-----+
|     |     |     |     |     |     |     |     |
|  0  |  0  | ad 5| ad 4| ad 3| ad 2| ad 1| ad 0|
|     |     |     |     |     |     |     |     |
+-----+-----+-----+-----+-----+-----+-----+-----+

The next thing to re-evaluate is the method by which you handle the receipt of the data. Your current scheme is way too complex. The logic also has the inherent attribute of receiving two bytes from the serial port which are then used to make a decision..but then you throw those away and drop into logic that then reads two more bytes from the serial port. Try instead some thing like the following:

At receiver end:
1) Make an initialization of the hardware
2) Enter a forever loop
3) Poll UART for RI set. If not set go back to 3
4) Read in data from UART SBUF.
5) If upper 2 bits of data == 0 then go back to 3
6) Keep high six bits: sample = (data & 0x3F) << 6
7) Poll UART for RI set.
8) Read in data from UART SBUF
9) Keep low six bits: sample |= (data & 0x3F)
10) Output sample to D/A converter
11) Loop back to 3.

You also need to be very careful how you handle things at the transmitter end so that the two bytes sent correspond the same A/D value. Here is a suggested sequence for that end.

At sender end:
1) Make an initialization of hardware
2) Enter a forever loop
3) Poll UART for TI set. If not set go back to 3
4) Read A/D converter into sample.
5) Send high 6 bits to SBUF: data = (sample >> 6) | 0xC0
6) Poll UART for TI set. If not set go back to 6
7) Send low 6 bits to SBUF: data = (sample & 0x3F)
8) Loop back to 3

Keep in mind that your scheme using the UART causes the analogue sample rate to be established proportional to the baud rate. Since it takes two serial transmissions to send one analogue sample and since the typical UART format is 8-bits/no parity/one stop bit then there are a total of 20 bit times per analogue sample. At 9600 baud this results in a realizable sample rate of 480 Hz. In order to get any sort of recognizable signal out of the D/A converter I would suggest that you limit the maximum signal you send through such system to about 1/10 of the sample rate. So this says that at 9600 baud you should limit the A/D input to something below 48 Hz. As Kai has suggested you should also low pass filter the D/A output using a filter that has a corner frequency at near this same 48 Hz value.

Michael Karas