Hello,
Are there actual connection issues, or are you just using ping and considering it an issue when they time out? Judging by your description it sounds like the network itself running fine. Keep in mind that the switch will respond to ICMP with lowest priority when it has other traffic to handle, so if your switch is quite busy forwarding traffic it does not necessarily indicate a problem just because some pings time out. What were you pinging anyway, and from where?
Regarding taking those servers off the switch temporarily - did you actually test pinging from a connected device on the 5406 to the switch itself afterwards? Were the results better after removing the servers? It sounds like you only tested from the servers to the other switch you moved them to.
Ping isn't intended as a tool to measure the performance of your network anyway. Have you considered trying something like iperf?