The people that already had your contact info in their devices were notified that you joined Signal via that contact info?

Seems like it was working as designed, if you don't want any app to get your contact info don't share your contact info to anyone ever. Eventually they will share that info with any app.