It is determined primarily from looking at the data sheets for the CLK and DAT timing. Look at the definitions of the waveforms generated by the master in the MCU for each mode. Then study the defined waveform requirements for the peripheral from its data sheet. By process of comparison, elimination and checking timing parameters you can conclude which mode is appropriate to use.

Once you have made a selection then it is essential to cross check the operational behavior after you've written code by using an oscilloscope to validate the waveforms generated. This validation is an important step because it ensures that you interpreted the data sheets correctly. I personally have had several cases where I selected a mode for a master to run to a peripheral and found that it kind of worked but not well. The validation step revealed that I was clocking on the wrong phase and so ended up having the data all off by one bit position.

Michael Karas