Perhaps, it's hard to disentangle all that from increasing wealth, industrialisation and globalisation though.
My house and garden are in what were fields until the 1950s. I'm digging up huge amounts of pipe remains. I don't make any claims to the size of the industry from that, but it certainly wasn't niche.
I don't want to make any claims about the upper bounds of smoking in the mid 19th century and earlier, but the lower bound is at least above what a layman would describe as 'niche'.
My house and garden are in what were fields until the 1950s. I'm digging up huge amounts of pipe remains. I don't make any claims to the size of the industry from that, but it certainly wasn't niche.