Athletics are a profit center for only a few universities. I doubt though that this includes spending on stadiums. There is also the cost in terms of corruption of the university and the damage done to athletes (especially American football) for little to no compensation.
They're also only a profit center if there are other universities willing to make a loss. If those universities decided to stop spending on athletics then the profitable ones would soon find their profits disappear. Taken as a whole, the college athletics industry is an overall negative sum for education. Looking at it from the perspective of an individual college misses the big picture.
http://www.ncaa.org/about/resources/media-center/news/athlet...