The Varnish configuration is hairy - manually tweaking cookies to cache images for everybody while avoiding duplication, denormalizing headers so that IE & Firefox can both get the benefit of compression, dealing with pragma & cache-control, etc.

(Compression & other varying content was a headache - separately caching & serving gzip, deflate, & unpacked versions, without any built-in compression.)

Those tweaks looked time-consuming to set up, hard to test, & liable to break quietly. A buffer in the way that simply & quickly gets a response from (& then frees up) Apache, then slowly feeds the response to the client, provides most of the benefit; nginx fills this role well for me.