The techniques I use mostly vary for each project, depending on the hardware design, MCU used, and the tools at my disposal. Some clients/employers have emulators, logic analyzers, protocol analyzers, etc., whereas some others are very low budget operations.

For very early hardware debug, emulators are sometimes useful, especially when they have a scriptable command execution facility.

I won't echo the techniques I use for software that others have already mentioned here, but I'll add to the list a few more things that come to mind. Some of these techniques depend on the hardware design. For example, some are obviously not appropriate if the design does not have external memory.

1. Writing trace data to dummy external addresses. This is somewhat similar to the printf() approach, but does not have the potential program execution pacing problems that using a UART introduces. This is essentially a zero-cost (target-wise, that is) approach.
   
   Using a dumbed-down logic analyzer (i.e., does not have 8051 "awareness", disassembly, etc.) connect to the external memory bus (address, data, and control signals). Identify one or more unused addresses and allocate them to different code areas. For example, one address might be where you output raw converted A/D sample values, another address for your RF receive ISR state machine to output it's current 'state', another pair of addresses for your RF receive ISRs to output timer input capture and compare values, and let's say one last address for a "should not occur" output. Your program simply writes to these dummy addresses, thus only incurring the overhead of what it takes to perform an external memory write (register setup, etc.). You setup the logic analyzer to trigger/trace on individual addresses or a range of addresses. The "should not occur" outputs are where you've added code to cover boundary conditions or paths that should never happen under normal circumstances.



2. Trace facility. This is similar to the above in that you don't have the potential UART pacing issues, but instead of having an instrument capturing the trace, you are writing (optionally timestamped) trace data into a circular trace buffer in target memory. You add code that supports extracting the trace buffer upon request (usually over the serial port). You can get a trace dump periodically or after you detect that something went wrong, and then run your trace formatter utility to make it human-readable and see what paths the code took with what data values.



3. Internal event simulator/monitor. This implements a monitor-like user interface that allows one to enter commands or create conditions in the target that might otherwise be very difficult to induce. Let's say you've got a logger that would normally take a month's worth of data to fill the log and you want to test target behavior at the extremes. You can create a script-driven test that allow you to set a simulation flag in the target causing it to run its 'calendar time' faster and that then pumps data into the target, filling the log at an accelerated rate. The idea is to programmatically drive the target and validate its behavior.