A huge chunk of that 3-4GB is large files that have minor changes. Time Machine doesn't have any sort of delta support so backups the entire file again, like my local Messages or Contacts database. But I think slowness is caused by file count, not file size, so even though it's backing up 3GB+, the total number of files changes isn't that high. (I suspect).
I also use a modified version of this script[0] to identify everything that changed in the most recent Time Machine backup. This is hugely helpful and lets me find unimportant things that are the source of lots of unimportant changes which I can then exclude with `tmux addexclusion -p <PATH>`. For example I exclude 'node_modules' folders for anything that gets regularly updated. This removes ~10k files that would otherwise be wastefully backed up. Speeding it up is much more about reducing file count than total size.
10 minutes is great, and my changes wouldn’t seem as extensive as yours. I need to dig deeper.