What I am saying is, and what is the proper thing to do, is that you (the installer or admin) have to put the actual antenna gain value (in dBi) into the provisioning field for that specific AP. It's not a variable value that you pick based on whether you want more coverage. If you are using a 3dBi antenna, then you put 3dBi into the fields. If you are using a 10dBi antenna, you put 10dBi into that field, etc.
The reason you see coverage dropping is because you are telling the AP that you have a higher gain antenna than you actually have. As a result, the AP is dropping the radio power output to stay within the configured Max EIRP or to stay within the regulatory EIRP limits.
Here's an example. Say you have an IAP-104 and you have four ANT_1B antennas connected to it (~4dBi @ 2.4Ghz and ~6dBi @ 5GHz). Lets say you then set the max EIRP in the radio config to 20dB on each radio.
So when you enter the proper antnena values into the provisioning field, you have the following:
Radio Band--------RadioChipPower------AntennaGain----Overall EIRP
2.4GHz radio @ 16dBm(20db-4dBi) + 4dBi antenna = 20dB EIRP
5GHz radio @ 14dBm(20dB-6dBi) + 6dBi antennas = 20dB EIRP
Coverage is good, because the radio is configured with the appropriate antenna gain values. Now, what happens when you use the SAME antenna but incorrectly configure 17dBi for the antnna gain. In this case, you tell the radios you are using a 17dBi antenna, and so they adjust their power WAY down so as not to exceed the 20dBm max EIRP limit. However, you don't actually have that antenna on there, and as a result, the effective EIRP is way too low, and you see a loss of coverage.
Radio Band------RadioChipPower------AntennaGain----Overall EIRP
2.4GHz radio @ 3dBm(20-17dBi) + 4dBi antenna = 7dB EIRP
5GHz radio @ 3dBm(20-17dBi + 6dBi antennas = 9dB EIRP
So there is no effective 'maximum' ceiling, because the antenna gain is NOT used to add or substract coverage. The antenna gain values should match what antennas are actually being used. The reason you see a bigger drop when you select a larger antenna gain is because the radios are dropping the power to stay within the radio config or max EIRP regulatory limits because it thinks it has a 17dBi antenna connected to it, when in fact it's something much much lower.
TL;DR, actual antenna gain values provisioned MUST match the values of the antennas being used for the radios to work properly.