06 A6 3.2 Quattro...100% perfect with NO unscheduled trips to the service bay.

01 A6 2.7T Quattro...98% perfect with only 1 unscheduled trip to the service bay.

 

I have had my 2004 S4 for about 10k miles and it has been problem free. Granted it only has 33k total...

I would avoid the B5's because forced induction can bring all sorts of problems as it ages (I have owned two twin-turbo cars..). While you can get tons of HP very quick when upgrading a forced induction car, get ready for plenty of dyno tuning to get it right. The n.a. V8 in the 2004+ will already come with more power, and you can bolt on another 50 or so horsepower without any major concerns.

If I was debating an M3 and an S4, the debate would revolve around the driving experience, not the reliability. That being said, the M3 is a little lighter, a little quicker, and provides a more sports-car like feel being that it is rear-wheel drive. The S4 will handle in better in poor weather given its awd, but is a bit heavier and perhaps not quite as nimble as the M3. I have driven both, and faced the exact decision you are, but went with the S4 because of the awd (I live in Cleveland where it snows).

Back to reliability: get a later 2004, Certified Pre-Owned and you will have no worries.