Late model 4.0's had a lot of issues with cam lobes wearing down really bad. The also had issues with the oil pump drive/cam position sensor.
Cams in the late model's from 01 and up were very shitty as it was. Dodges continued attempts to meet emission regs forced them to use very restrictive cams that used long exhaust duration profiles to increased exit burn temps.
It would not be surprising for you to have bad cam wear, making it difficult for your lifters to pump up.
A cam swap on a good running 4.0 is beneficial as it is. Likely even more so in your situation.
You can read up on line. There were lots of cases of cams losing entire lobs in their first year.