My $0.02.
If you know that you are going to off-road, a Rubicon is probably good value for your dollars. You would be hard pressed to get the lockers, gearing, 4:1 transfer case, and Dana 44 front (and 32" M/T's) as upgrades for the price difference between an X and a Rubi. With a manual and the long Unlimited wheel base (which means you can leave your stock driveshafts alone), you can put on a OME coil lift, JKS trackbars, wheel spacers (or steelies), and 35" rubber for about 3500. That gets you a vehicle with front and rear lockers, good clearance, good suspension travel, and 35's - pretty bloody capable if you ask me. You'll never match the breakover angle of an equivalently built 2 door...but they'll never be able to climb like you can either.
BUT
This only applies if you're not going to run anything larger than 35's. If you think that you're likely to step up to 37's, then you probably won't want to do it on stock Dana 44's....which means that you should probably buy a bare-bones X and then start shelling out for some serious hardware.