Here is a State Transition Diagram (STD) showing the behaviour you described:



I used Enterprise Architect - it's a (relatively) low-cost UML tool.

Many others are available; search this forum for some suggestions - including some free ones.

You can, of course, do it entirely with pencil and paper.

MS Visio also has a template for State Diagrams.

You could implement it with a switch statement, where each case handles one state; eg,

CASE state_1:
   IF      event IS mode_pressed THEN state := state_2
   ELSE IF event IS up_pressed   THEN INCREMENT a
   ELSE IF event IS down_pressed THEN DECREMENT a

Again, there has been plenty of discussion of this here before - try a search.

It is a very common technique, so there's also plenty of information available on the web, in textbooks, etc.

One thing to beware of in general texts, though, is that it is quite common to implement state machines in 'C' using tables of function pointers - that can be problematic on an 8051, and is probably best avoided (at least until you have thoroughly checked-out the issues with your particular compiler)