This feature is only badly designed in that it's not communicated during the setup process that Siri will still work. This is a feature, it can be turned off, and is 100% intentional.

The security your propose doesn't provide enough security to be worth the effort. (Hello tape recorder!)