I used to own a 04 FX4, the ride was always stiff, even before the leveling kit. Going over any bumpy road was rough, you really felt it. I thought it was tires but even after getting new Toyo MTs it made no difference. I sold the FX4 because of the 5.4 headaches then got a 04XLT. The ride is a lot smoother and softer. I've modded this truck the same as the FX4 and it's still smoother.
I think the FX4 might have stiffer coil springs upfront and maybe stiffer leaf springs? I don't know for sure but will simply say from my ownership experience, the XLT is not as rough.
Current: 2013 SCREW FX4 5.0.
Sold: 2004 SCREW 4x4 XLT 4.6 V8, Edge Evo, Magnaflow SIDO, Gotts mod, 2 Leveling, retrofit headlights
Sold: 2004 SCREW 4x4 FX4 5.4 V8(problematic motor), SIDO Borla exhaust. AFE intake. 2" Leveling