"I don't think there can a be set mathematical formula to get an index value to use as an offset to get the Y value from the table."

Maybe I didn't understand your question:

"Say if input is X then get value Y with MOVC."

Typically, with a table-lookup, you are doing:

Y = table[X]

ie, the value Y which you get is the Xth entry from the table - X is the index to the table, and the indexing of table[] is achieved with the MOVC A,@A+DPTR or similar

Are you saying that you need a double lookup; eg,

Y = resultTable[ indexTable[X] ]

The principle is the same - you just do it twice!