You can turn microphone access off at any time in your device’s Settings.