That isn't an inevitability. The modem doesn't negatively affect the lenses and sensors, choices by people in companies are what result in worse sensors/lenses in phones.
My Pixel takes reasonably good images, even printed some in A4, at A3 they suffer. That is for the camera generated JPEGs, the RAW files are all but unusable.
It shows that smartphones use small, and crappy, sensors behind even smaller, and crappier, lenses. Phone got a long way, and there is a reason they replaced point-and-shoots. They are still a far cry from "real" cameras.