You will absolutely get into some kind of trouble (social or legal) if you put a camera on a sidewalk and start filming whoever goes by. On the other hand CCTV cameras don't receive any flak. Society is usually not obviously consistent about privacy matters. The ultimate determining factor is what seems creepy vs what is necessary. So its entirely possible that some societies might decide that the video taping done by self-driving cars is sufficiently creepy to ban.
No you won't, I've had a camera mounted on my house for years with no issues, I did this when my car got hit by a neighbor and I had to chase down camera footage from a CCTV camera operated by a home builder down the street, having that footage was the only way they were held responsible for it, as after initially admitting to it they later decided they didn't want to pay. Many people also run dashcams for insurance purposes, perfectly legal.
Personally I find devices that record locally or to user-owned services perfectly fine, keeping local recordings on your dashcam's SD card, awesome, keeping your home recordings on your NVR, NAS or private server, that's okay, but sending that to corporate or government owned services is generally bad. There's a limit to how much an individual can do with the data they record from a few places they own, their car, their house, etc. There's no limit to how much abuse a large entity can do with that data, facial recognition and more computing power than they know what to do with.
Drawing this line legally however is... extremely challenging.
The reality is what we have here is a tradeoff between freedom and security with any individual CCTV camera. Those are both valuable things in their own right and such tradeoffs should be made carefully.
When we then hand that data off to a corporation or government entity, that tradeoff looks very different - now they have access to thousands if not millions of sensors. The freedom implications become much higher and the security is often replaced with things like "how good is our traffic data?", "how can we improve our self-driving system to sell more cars?" or "how can the Chinese government bust more Muslims?". The aggregated result is rarely worth it.
