how to trace a microcontroller running it's app ?

I have a (8051) micro-controller that i would like to trace it's
runtime program to understand how the program works for purpose
of potentialy modifying code. (The only info of program is the
program binary copied off ROM )

The micro-controller consists of  (usuall stuff):
---------------------------------------
> Siemens 8031, ROM, SRAM
> servo motors (+ driver ICs)
> simple user 16 char (14 seg) interactive display
> grid of buttons (24)

Any ideas on how to do this?  how possible ?

I have a PJRC 8051 Rev.4 development board that runs "Paulmon"
which allows you to load your program to some offset in RAM and
step through it. (i really want to see the I/O as it runs
though)

My idea/thought at first was to maybe connect the PJRC  board to
the MCU socket of the microcontroller board and trace the program
??

What i have tried is load in a simulator like (EdSim51) to run
and determine what program does but i would need to some how
simulate all the devices (input) ?

any help and ideas greatly appreciated,
robb