Hi Gabor
If you had connected it backwards and powered it up - it is possible that you damaged the serial port on the AP and it may never actually work again. Your settings of 8N1 9600 are correct. The ports are reasonably robust though, you would have to be unlucky to have blown it on first attempt. Connect it as below and try again - do take note of the progression of the LEDs - it might be that the RAP is actually damaged and not booting. From power on, at least you should see this
a> power LED red briefly (< 1sec), enet port LEDs flash yellow, then all off
b> power LED is solid green for about 10 seconds (enet0 should come on at about 4 seconds)
c> power LED will go through various sequences of flashing and solid, after about 3 mins it will hold solid and the WLAN leds should turn on
With regards to the cable itself, yes the notch on the cable is useless, it "matches" (sort of) but it doesn't actually provide any physical guidance or locking. Yes, you should locate the vague notch in the serial cable to the thinner of the two plastic protrusions in the serial port hole (for RAP-155 and RAP-3)
I am not sure it was the intention, but I am yet to see them match physically. You may note that my rap-155 has been the recipient of some surgery (the tab redrawn above in photo editor), the lower plastic tab was actually too tall relative to the prongs of the serial port preventing the plug from being able to be pushed onto the prongs. Not sure if that's a problem for others - but better to chop the plastic than trying to bend the serial port prongs.
Final view - plus reminder sticker:
Alternatively:
RAP-155 -- red cable towards ethernet ports
RAP-3 -- red cable towards LEDs on the front
regards
-jeff