Ok so here's the problem for the benefit of anybody else in this situation.
Using the SAML tracer plugin for firefox i was able to get a good look inside the HTTP posts going on during the login process (it's a great tool !!) without having to drop HTTPS on the login pages.
This showed that the username attribute and password were not being passed to the Aruba controller during the POST to captiveportal-login.x.x.x.
The reason for this is that i had removed the username and password fields from the receipt page and the login button requires these values in order to be able to login the user directly.
Here is the POST to the ClearPass Login Page showing the attributes that are available:
POST https://guest.customer.tld.com/guest/register_receipt.php HTTP/1.1
Host: guest.customer.tld.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://guest.customer.tld.com/guest/register_receipt.php?refresh=1
Cookie: GSID=hltl8us7ffbhvm9g4fmm7g3n37
Content-Type: application/x-www-form-urlencoded
Content-Length: 305
HTTP/?.? 200 OK
Date: Mon, 29 Aug 2016 01:05:33 GMT
Server: Apache
X-Powered-By: PHP/5.5.34
P3P: CP="CAO DSP LAW CUR ADMa DEVa OUR IND PHY ONL UNI PUR COM NAV INT DEM CNT STA PRE OTC"
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
x-frame-options: SAMEORIGIN
Set-Cookie: GSID=hltl8us7ffbhvm9g4fmm7g3n37; path=/; secure; HttpOnly
Keep-Alive: timeout=4, max=500
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
POST
url: http://smh.com.au/
apgroup:
apname: tunnel 17
essid:
ip: 172.22.210.119
mac: 28:b2:bd:f2:ab:7f
cmd: login
sponsor_email: scott.doorey@customer.tld.com
visitor_name: scott testing
email: user@email.com
start_time: 2016-08-29 11:04
expire_time: 2016-10-13 12:04:33
enabled: 1
Here are the details of the POST sent to the Aruba controller:
POST https://captiveportal-login.customer.tld.com/cgi-bin/login HTTP/1.1
Host: captiveportal-login.customer.tld.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://guest.customer.tld.com/guest/register_receipt.php
Content-Type: application/x-www-form-urlencoded
Content-Length: 76
HTTP/?.? 302 Temporarily Moved
Date: Mon, 29 Aug 2016 01:05:33 GMT
Server: Apache
x-frame-options: SAMEORIGIN
X-UA-Compatible: IE=edge;IE=11;IE=10;IE=9
Location: https://guest.customer.tld.com/guest/register.php?errmsg=Access denied
Content-Length: 0
Connection: close
Content-Type: text/html
POST
user:
password:
cmd: authenticate
url: http://smh.com.au/
Login: Log In
Notice no username or password above!
Here is what the working form looked like from the separate login page:
OST https://guest.customer.tld.com/guest/guestlogin.php?_browser=1 HTTP/1.1
Host: guest.customer.tld.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://guest.customer.tld.com/guest/guestlogin.php?_browser=1
Cookie: GSID=hltl8us7ffbhvm9g4fmm7g3n37
Content-Type: application/x-www-form-urlencoded
Content-Length: 219
HTTP/?.? 200 OK
Date: Mon, 29 Aug 2016 01:06:12 GMT
Server: Apache
X-Powered-By: PHP/5.5.34
P3P: CP="CAO DSP LAW CUR ADMa DEVa OUR IND PHY ONL UNI PUR COM NAV INT DEM CNT STA PRE OTC"
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
x-frame-options: SAMEORIGIN
Set-Cookie: GSID=hltl8us7ffbhvm9g4fmm7g3n37; path=/; secure; HttpOnly
Keep-Alive: timeout=4, max=500
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
GET
_browser: 1
POST
errmsg: Access denied
url: http://smh.com.au/
apgroup:
apname: tunnel 17
essid:
ip: 172.22.210.119
mac: 28:b2:bd:f2:ab:7f
cmd: login
no_login:
user: user@email.com
password:
visitor_accept_terms: 1
here the username is define, not just the email address. This is because the form asked for the username attribute. Username auth was configured on the page so no password is shown.
here is the successful post to the controller for the same user using the separate web login page:
POST https://captiveportal-login.customer.tld.com/cgi-bin/login HTTP/1.1
Host: captiveportal-login.customer.tld.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://guest.customer.tld.com/guest/guestlogin.php?_browser=1
Content-Type: application/x-www-form-urlencoded
Content-Length: 107
HTTP/?.? 200 OK
Date: Mon, 29 Aug 2016 01:06:12 GMT
Server: Apache
x-frame-options: SAMEORIGIN
X-UA-Compatible: IE=edge;IE=11;IE=10;IE=9
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html
POST
user: user@email.com
password: 039180
cmd: authenticate
url: http://smh.com.au/
Login: Log In
Here you can see the username and password int he post to the controller.
What i had to do was enable the password and username fields on the receipt page (even though i didnt' want them displayed) and then everything worked fine!!
Hope this saves someone hours of head banging!!
Scott