The short answer is "it depends on the environment". There are some environments that simply have high retries and still function. Environments that use email and browser traffic typically can withstand retries more than VOIP environments can. Heavily congested environments have more retries than lightly congested environments. The best way to understand retries is to make a note of retries when everyone thinks everything is working and compare it to when everybody thinks that the wifi is not working.
With that being said, environments with more retries are more likely to see random and sustained performance issues than environments with less retries. I don't like to see environments with more than 50% retries sustained.