I'd definately worry about this one dude. Your axle seals are shot and need to be replaced. You may have wear/pitting on the shaft as well; where the seal rides on. Get it fixed! Your brakes are probably coated with gear oil, and this well affect your ability to stop well. You will also have less fluid in your diff. and this will cause your Ring and Pinion to heat up from less lubrication.
It's bad if you have a leak here. Oil is getting out and dirt/water/contaminents can get in.