I've wheeled and dd both XJs and ZJs. ZJs are defiantly more refined and nicer to drive, the V8 is loads of fun(but a bit of a gas guzzler) and the ZJ has WAY more room in backseats, a big plus if you have a family or often carry friends.
The XJ is a more economical daily driver and cheaper to lift, more aftermarket stuff and more used stuff available. Also cheaper to buy upfront. Easier to find used parts for as well.
I wheeled a daily driver XJ with 31s, 4.10 gears, rear locker and winch for almost 5years and did very well and the rig still handled well on the road.
Give us more info, how much will it be daily driven, how far do you commute(fuel economy), do you plan to tow anything(if so get a V8), how hardcore of an offroad rig do you want, what do the guys you wheel with run, got a family/how much room do you need......