Yeah, your last point is totally spot on, it is so gratifying to make something feel obviously much faster, in a way that few other things in programming are, and there's usually a lot of low hanging fruit.
Also, if you work on a website, the Google crawler seems to allocate a certain amount of wall time (not just CPU time) to crawling your page. If you can get your pages to respond extremely quickly, more of your pages will be indexed, and you're going to match for more keywords. So if for some reason people aren't convinced that speed is an important feature for users wanting to use your site, maybe SEO benefits will help make the case.
Also, if you work on a website, the Google crawler seems to allocate a certain amount of wall time (not just CPU time) to crawling your page. If you can get your pages to respond extremely quickly, more of your pages will be indexed, and you're going to match for more keywords. So if for some reason people aren't convinced that speed is an important feature for users wanting to use your site, maybe SEO benefits will help make the case.