Well first you have to install ffmpeg. I will save that for another post.
Let us assume that your ffmpeg is capable of encoding for the PSP platform. The command to convert to a PSP format is:
ffmpeg -i yourfile.avi -f psp -r 29 -b 768K -ar 24000 -ab 64k -s 320x240 M4V00001.MP4
To be honest, I'm not really sure what the parameters mean. (Again, save for another post :). But this command works for me. One thing of note is the -s parameter which refers to the size. The PSP format needs both dimensions to be divisible by 16. You can use -s 368x208 for widescreen videos.
To create the thumbnail file:
ffmpeg -i yourfile.avi -f image2 -ss 5 -vframes 1 -s 160x120 M4V00001.THM
The -ss parameter refers to which frame to use. I believe the units for this is seconds.
Stay tuned for more PSP/ffmmpeg posts. I will also post how to convert using mencoder