The first thing you should do is fork the repository and set up a remote for pulling in upstream changes.
There are a lot of useful libraries out there, but I find I occasionally need to make changes. Sometimes there are bugs; sometimes the maintainer has lost interest. I need to be able to pick up the slack when that happens.
Also, you can only remove a submodule manually. There's no option to have git do it for you.