I used to own a 04 FX4, the ride was always stiff, even before the leveling kit. Going over any bumpy road was rough, you really felt it. I thought it was tires but even after getting new Toyo MTs it made no difference. I sold the FX4 because of the 5.4 headaches then got a 04XLT. The ride is a lot smoother and softer. I've modded this truck the same as the FX4 and it's still smoother.
I think the FX4 might have stiffer coil springs upfront and maybe stiffer leaf springs? I don't know for sure but will simply say from my ownership experience, the XLT is not as rough.
Current: 2011 SCREW Lariat 5.0, 2" Level, 295/60/20
Sold: 2013 SCREW FX4 5.0, 2" Level, 305/55/20 G2
Sold: 2004 SCREW 4x4 4.6,Magnaflow SIDO,2" Leveling, retrofit
Sold: 2004 SCREW 4x4 FX4 5.4, SIDO Borla, 33" MT