The client decides what access point it connects to. You can make that more predictable by making fewer access points visible to that client, by lowering the access points transmit power gradually. If your access points transmit power are at 18 or more and/or are close together, that could cause the effect you are seeing.
That is my best educated guess without more information.