I ran ambulances for 30 years, they can have up to 4 batteries in them, so have a little bit of experience.
You can mix and match and not have any big problems, just small issues.
The isolator acts as a one way valve, as in power goes into the battery from the alternator but can not be drained out by another battery in the system on the charging side. But if the output side is not separate, then one battery will drain the other.
You wire the output of one battery to the winch only(key word is only, can not share any power to OEM side) and manually boost from battery one to battery two if needed.
Or have a battery switch or relays for the output side of the power from the batteries.
Battery one feeds the normal power for the jeep, battery two feeds the winch etc.
With the switch or relays you can have one battery boost the other or run independent of each other.
The alternator senses a voltage drop and increases voltage/amps output till the voltage comes back up to the trigger voltage of the alternator.
So the problem with a mix and match battery system is if one battery is low then all the batteries are being charged till the low battery is charged or if one battery is a higher rated battery, then the lower rated battery is charged first and is still being charged till the higher rated battery is charged.
But in real life it is not a big issue, alot of ambulances have two full sized batteries to run the OEM in the front and smaller batteries in the back due to the space issues.
In the jeeps the alternator is being overworked when you add bigger batteries and / or winches etc. Once in awhile is not an issue, but heavy winching, lots of high output lights will shorten the life of the alternator.
Greg