I actually asked Stallman about this once at a talk. It is precisely because we own our desktop. The analog he used was a food truck. It's fine to eat something from a food truck that somebody else prepares, even if there's a secret sauce you don't know about. The issue is being given the food truck and then being told that you must still use the sauce whose contents are opaque to you.
That being said, I don't really care much about using non-GPL programs. But that's the rationale.
Your recitation of his response does not address the OP's point, and my smarmy comment was downvoted.
So, let's see if we can work up this wishy-washy analogy a bit.
I like to eat food. (I like to use software.)
Much of the food I eat is prepared by vendors who employ 100% documented citizens or immigrants with valid visas. (Much of the software I use is 100% Free.)
Some of the vendors employ undocumented immigrants. (Some of the software I use contains nonFree components.)
My ability to get this food from these vendors depends on a number of other vendors - even a simple sandwich needs bread, meat, vegetables and condiments all sourced from and delivered by companies that may or may not employ undocumented immigrants.
I may purchase a completely prepared meal and eat it at home. (I may install and run boxed software locally on my machine.) I may purchase ingredients and prepare a meal from scratch. (I may obtain source code and locally build software to run on my machine.) I may purchase both raw ingredients and some pre-cooked food to take on a picnic. (Modern software distribution is complex, and "using a website" is a vague reference umbrella.)
In each of these scenarios, to eat my desired meal, I may need to source an ingredient produced by a complex system chain which may or may not have involved undocumented immigrants. Tracking this information down for every ingredient is tedious, but I agree should absolutely be possible.
I could prepare all of my meals myself, in my own kitchen, and source all of the raw ingredients from a small number of vendors whom I trust. I would probably have a limited number of basic ingredients to work with at the beginning. I would probably be able to produce reasonably healthy, functional meals. I would not be able to produce a rich menu that appealed to a variety of tastes, however. (A basic GNU system is pretty functional but you've got to wire up most things yourself.) I would have to work with these vendors frequently and would spend much more time managing the ingredients in my kitchen and preparing meals. I would also likely be limited in what I could produce by my locality and kitchen equipment. (Data must be created locally and hardware must be self-managed.)
I might trust these same vendors to prefabricate side dishes (system libraries) or larger heat-at-home dishes (application source bundles) that I integrate into my day-to-day meals. I may gain some variety because they can prepare dishes that I do not have the skill or kitchen equipment to prepare myself. (The trust that I confer on these vendors for not using undocumented immigrant labor does not prevent me from contracting a case of salmonella. [Heartbleed.])
I can get food in many forms from many varieties of vendors - delivered meals, take-out meals from a restaurant eaten at home, sit-down meals at a restaurant, meals from a food truck eaten on a nearby patio, meals from a food truck eaten at home... (Apps, desktop, websites, servers. Simply browsing the modern web means code runs in response to your every input on hundreds of machines around the globe.)
Sometimes my act of eating employs undocumented immigrants. Crap.
That being said, I don't really care much about using non-GPL programs. But that's the rationale.