Thanks for you input and I agree that getting the apNotFound message is indeed a good sign. Just frustrating...
I've tried using capital letters for the MAC addresses, but that gives me the same base64 encoded string. This is normal since the MAC address is an array of bytes (according to the .proto file from Aruba) and the conversion of string "B3" and "b3" both give the same byte 0xb3 and thus the same base64 encoded string. That's why that doesn't matter.
This also lead me to try and manually convert the MAC address string in my JSON to a 6 byte buffer and lo and behold, it works. My protobuf encoder now encodes the MAC address correctly. And it makes sense. A base64 encoded string gets treated differently by my protobuf encoder than a byte buffer, but they both give the same result.
Either way, I still get the same apNotFound result, even after also encoding/bufferising receiver.mac, zigbee.radio_mac, zigbee.send.mac and zigbee.send.payload.
And yes, the MAC address I send is the same I receive from the apHealth messages, but still the AP won't accept that.
I think I will open a ticket with support and have them take a look at it.
Edit: Something else I find weird is that when I send a made up mac address (like 123456abcdef) in the receiver.mac, the apNotFound message contains that same made up address in the reporter.mac field. Weird.
Original Message:
Sent: Feb 03, 2022 02:57 PM
From: Maximilian Flügel
Subject: Zigbee Southbound traffic
Hey Steven,
I am pretty sure that we ran into the same issues when we developed the IoT-Utilities application.
From my point of view, the strange things that happen with your MAC-Addresses should be fine.
If the AP does not send any response, the packet itself was invalid. Consequently, the fact that you receive a response is a good sign.
I think that all of the MAC-Addresses need to be converted to base64 format so the error might occur as your radio_mac and send.mac values are not encoded.
Besides that, I discovered that our application transforms the MAC-Addresses to capital letters before converting them to the protocol buffer format.
This means that the MAC-Address itself is passed in capital letters, however the base64 string is not in capital letters.
Nevertheless, we did not have to implement the base64 encoding ourselves as the Java Protocol Buffer implementation handles this automatically.
So for instance, your MAC-Addresses should look like this:
AA:AA:AA:AA:AA:AA
instead of this:
aa:aa:aa:aa:aa:aa
before converting them to the base64 format.
I cannot guarantee that this causes the issue but this is another difference that I discovered between our implementations.
The Address referenced by the receiver.apMac field is the MAC-Address that is sent by the Aruba Health messages, so I think the receiver.apMac you entered should be the right one.
Regards,
Max
------------------------------
Maximilian Flügel
Original Message:
Sent: Feb 02, 2022 11:40 AM
From: Steven Demets
Subject: Zigbee Southbound traffic
Here's an update (spoiler: still not working).
I only get a response through the websocket connection when I encode the receiver.apMac as base64.
I've tried the IAP MAC address, I've tried other MAC addresses, but every time I get the error status: "apNotFound", statusString: "AP MAC sent does not match IAP MAC"
What is strange though is that the reporter.mac in the response the same address is as my sent receiver.apMac. Independent of if it's made up or real. I've even tried the MAC address of the AP305 in my cluster, but I got the same results.
As an example: the MAC address of my AP505 is b83a5ac2848a. Base64 encoded this is uDpawoSK. When I create the packet I get:
{ "meta": { "access_token": "sometoken", "version": 1, "sbTopic": 3 }, "receiver": { "apMac": "uDpawoSK" }, "zigbee": { "reqid": "573188078442985853545030ae870001", "radio_mac": "204c03fffea5f47a", "send": { "mac": "842e14fffe76a677", "e2pc": { "destination": { "endpoint": 1, "profile_id": 260, "cluster_id": 1 }, "source_endpoint": 1 }, "payload": "01ee002100", "reqid": "573188078442985853545030ae870001" } }}
The response I get from the websocket is:
{ "meta": { "version": "1", "accessToken": "someaccesstoken", "nbTopic": "actionResults" }, "reporter": { "mac": "uDpawoSK", "time": "1643819348" }, "results": [ { "status": "apNotFound", "statusString": "AP MAC sent does not match IAP MAC" } ]}
When I change the receiver.apMac to anything that doesn't resemble a base64 encoded MAC address, I don't get any response. So if I put in the original MAC address, unencoded, I get no response. If I use the long form of a MAC address (with fffe in the middle), I get no response (even if base64 encoded). But if I put "123456abcdef" and encode it, I get the apNotFound status with the reporter.mac the base64 encoded "123456abcdef". Very strange!
My question is now, what MAC address do I use?
Steven
------------------------------
Steven Demets
Original Message:
Sent: Jan 29, 2022 12:03 PM
From: Jens Fluegel
Subject: Zigbee Southbound traffic
Hi Steven,
Great to see that you figured out the encoding/decoding stuff.
The AP not found can only be a wrong apMac.
Looking forward to you final success message.
Regards,
Jens
------------------------------
Jens Fluegel
Original Message:
Sent: Jan 27, 2022 03:46 AM
From: Steven Demets
Subject: Zigbee Southbound traffic
Hi Jens,
You may read what's here below, but I don't think it's relevant anymore. I've found the problem and it has nothing to do with the AP, it's my server.
When I configured my websocket connection, I've set it to send the complete message (with all the internal variables) instead of only the payload (protobuf).
It's still not working, but now I get an "AP not found" error, which means the protobuf packet gets decoded:
[ { "status":"apNotFound", "statusString":"AP MAC sent does not match IAP MAC" }]
This has probably to do with the base64 encoding, so I will get on that now and keep you apprised.
Sorry for the inconvenience.
Steven
===============================
=== MAY BE IGNORED FROM HERE ===
===============================
Here is what I receive from the AP when I press the "On" side of the button:
Raw websocket data:
[10,4,8,1,24,7,18,8,18,6,184,58,90,194,132,138,82,40,10,8,32,76,3,255,254,165,244,122,18,28,10,8,132,46,20,255,254,118,166,119,18,11,10,7,8,1,16,132,2,24,6,16,1,26,3,1,2,1]
This is a decimal representation of the buffer. When you put that to hex, you get the following:
[ 'a', '4', '8', '1', '18', '7', '12', '8', '12', '6', 'b8', '3a', '5a', 'c2', '84', '8a', '52', '28', 'a', '8', '20', '4c', '3', 'ff', 'fe', 'a5', 'f4', '7a', '12', '1c', 'a', '8', '84', '2e', '14', 'ff', 'fe', '76', 'a6', '77', '12', 'b', 'a', '7', '8', '1', '10', '84', '2', '18', '6', '10', '1', '1a', '3', '1', '2', '1']
Getting this out of my protobuf decoder, using the 8.9 northbound .proto file, I get the following:
{ "meta": { "version": "1", "nbTopic": "zbNbData" }, "reporter": { "mac": "uDpawoSK" }, "zigbee": { "radioMac": "IEwD//6l9Ho=", "report": { "mac": "hC4U//52pnc=", "e2pc": { "destination": { "endpoint": 1, "profileId": 260, "clusterId": 6 }, "sourceEndpoint": 1 }, "payload": "AQIB" } }}
I then need to put reporter.mac, zigbee.radioMac, zigbee.report.mac and zigbee.report.payload through a base64 decoder to get the following:
{ "meta": { "version": "1", "nbTopic": "zbNbData" }, "reporter": { "mac": "b83a5ac2848a" }, "zigbee": { "radioMac": "204c03fffea5f47a", "report": { "mac": "842e14fffe76a677", "e2pc": { "destination": { "endpoint": 1, "profileId": 260, "clusterId": 6 }, "sourceEndpoint": 1 }, "payload": "010201" } }}
This is because my decoder (based on protobuf.js) expects byte type values to be base64 encoded, but in the received protobuf packet they are not. As an example, you can find the reporter.mac from the 11th byte in clear. That's why I need to put them back to clear text.
Now to the southbound packets. I will show you what I did, both with encoding the byte type variable to base64 first, and without.
Without base64 encoding:
{ "meta": { "access_token": "sometoken", "version": 1, "sbTopic": 3 }, "receiver": { "apMac": "b83a5ac2848a" }, "zigbee": { "reqid": "573188078442985853545030ae870001", "radio_mac": "204c03fffea5f47a", "send": { "mac": "842e14fffe76a677", "e2pc": { "destination": { "endpoint": 1, "profile_id": 260, "cluster_id": 0 }, "source_endpoint": 1 }, "payload": "4059000400", "reqid": "573188078442985853545030ae870001" } }}
Which encodes as (using the 8.9 southbound .proto file and IotSbMessage protoType):
[10,4,8,1,32,3,18,11,18,9,111,205,218,229,167,54,243,143,26,50,65,18,63,10,32,53,55,51,49,56,56,48,55,56,52,52,50,57,56,53,56,53,51,53,52,53,48,51,48,97,101,56,55,48,48,48,49,18,12,243,141,158,215,135,223,125,238,250,107,174,251,26,4,10,2,8,1,34,7,227,78,125,211,77,56,211]
In hex:
[ 'a', '4', '8', '1', '20', '3', '12', 'b', '12', '9', '6f', 'cd', 'da', 'e5', 'a7', '36', 'f3', '8f', '1a', '32', '41', '12', '3f', 'a', '20', '35', '37', '33', '31', '38', '38', '30', '37', '38', '34', '34', '32', '39', '38', '35', '38', '35', '33', '35', '34', '35', '30', '33', '30', '61', '65', '38', '37', '30', '30', '30', '31', '12', 'c', 'f3', '8d', '9e', 'd7', '87', 'df', '7d', 'ee', 'fa', '6b', 'ae', 'fb', '1a', '4', 'a', '2', '8', '1', '22', '7', 'e3', '4e', '7d', 'd3', '4d', '38', 'd3']
When I first encode all byte type variable to base64, I get the following:
{ "meta": { "access_token": "sometoken", "version": 1, "sbTopic": 3 }, "receiver": { "apMac": "uDpawoSK" }, "zigbee": { "reqid": "573188078442985853545030ae870001", "radio_mac": "IEwD//6l9Ho=", "send": { "mac": "hC4U//52pnc=", "e2pc": { "destination": { "endpoint": 1, "profile_id": 260, "cluster_id": 0 }, "source_endpoint": 1 }, "payload": "QFkABAA=", "reqid": "573188078442985853545030ae870001" } }}
The reqid values are not encoded as they are String values.
This gives me the following protobuf packet:
[10,4,8,1,32,3,18,8,18,6,184,58,90,194,132,138,50,59,18,57,10,32,53,55,51,49,56,56,48,55,56,52,52,50,57,56,53,56,53,51,53,52,53,48,51,48,97,101,56,55,48,48,48,49,18,8,132,46,20,255,254,118,166,119,26,4,10,2,8,1,34,5,64,89,0,4,0]
Which in hex is:
[ 'a', '4', '8', '1', '20', '3', '12', '8', '12', '6', 'b8', '3a', '5a', 'c2', '84', '8a', '32', '3b', '12', '39', 'a', '20', '35', '37', '33', '31', '38', '38', '30', '37', '38', '34', '34', '32', '39', '38', '35', '38', '35', '33', '35', '34', '35', '30', '33', '30', '61', '65', '38', '37', '30', '30', '30', '31', '12', '8', '84', '2e', '14', 'ff', 'fe', '76', 'a6', '77', '1a', '4', 'a', '2', '8', '1', '22', '5', '40', '59', '0', '4', '0']
And here again you can see the receiver.apMac in clear from byte 11.
I've done all this testing in Aruba OS Instant version 8.9.0.1_82154 and again with 8.8.0.2_81788.
Thanks for any insights you can give me,
Steven
------------------------------
Steven Demets
Original Message:
Sent: Jan 25, 2022 12:59 PM
From: Jens Fluegel
Subject: Zigbee Southbound traffic
Hi Steven,
The error message you get is based on the fact that the Aruba controller/ap cannot decode the protobuf message you sent.
Can you provide the message you are trying to send before and after (byte stream) encoding?
Please also provide the Aruba SW Version you are using.
Regards,
Jens
------------------------------
Jens Fluegel
Original Message:
Sent: Jan 25, 2022 06:32 AM
From: Steven Demets
Subject: Zigbee Southbound traffic
Hi Jens,
That was what I was doing originally as well, but I changed it to see if it made a difference.
I've now changed it back, but to no avail. Still the same result.
Is there any debugging on the AP that can be done to see any more information than what's reported now (see below)?
[ { "status":"invalidRequest", "statusString":"Unable to decode protobuf message" }]
I have tried to use the iot-sniffer command with a remote WireShark setup, but that doesn't work at all. The zigbee client-table gets emptied out and no data goes to the remote wireshard PC. I checked my setup with the normal remote sniffer function and that works perfectly.
Steven
------------------------------
Steven Demets
Original Message:
Sent: Jan 25, 2022 06:19 AM
From: Jens Fluegel
Subject: Zigbee Southbound traffic
Hi Steven,
Please try the outbound profile without "aps-act" option.
This is the zsd profile I use with Philips Hue on Aruba Instant:
zigbee socket-device-profile philips-hue-lamp inbound 11 232 0104 0006 inbound 11 232 0104 0008 inbound 11 232 0104 0300 inbound 11 232 0104 0003 outbound 232 11 0104 0006 outbound 232 11 0104 0008 outbound 232 11 0104 0300 outbound 232 11 0104 0003
Regards,
Jens
------------------------------
Jens Fluegel
Original Message:
Sent: Jan 25, 2022 06:10 AM
From: Steven Demets
Subject: Zigbee Southbound traffic
Hi Jens,
Here it is:
zigbee socket-device-profile Styrbar-out outbound 1 1 0104 0000 aps-ack outbound 1 1 0104 0003 aps-ack outbound 1 1 0104 0001 aps-ack outbound 1 1 0104 0020 aps-ackzigbee socket-device-profile Styrbar inbound 1 1 0104 0006 inbound 1 1 0104 0005 inbound 1 1 0104 0008 inbound 1 1 0104 0001 inbound 1 1 0104 0003 inbound 1 1 0104 0019 inbound 1 1 0104 1000
Both ZSD profiles are added to my transport profile:
iot transportProfile nodered endpointURL ws://<myserverip>/ws/zigbee endpointType telemetry-websocket payloadContent zsd endpointToken someaccesstoken endpointID ArubaTest transportInterval 900 ZSDFilter Styrbar,Styrbar-out
Please tell me I made a fatal mistake!
Steven
------------------------------
Steven Demets
Original Message:
Sent: Jan 25, 2022 06:03 AM
From: Jens Fluegel
Subject: Zigbee Southbound traffic
Hi Steven,
I know it is annoying and I am happy to see you are not giving up.
Can we check just one more thing. Pls can you show your zigbee socket device profile configuration for you setup?
Regards,
Jens
------------------------------
Jens Fluegel
Original Message:
Sent: Jan 25, 2022 05:44 AM
From: Steven Demets
Subject: Zigbee Southbound traffic
Hi Jens,
Thanks for the tips. Luckily converting a MAC address to a buffer in node.js is a lot easier (Buffer.from(mac, 'hex')), but it doesn't change the response. I'm not giving up though. I'll let you know if I get it to work.
Steven
------------------------------
Steven Demets
Original Message:
Sent: Jan 24, 2022 01:27 PM
From: Jens Fluegel
Subject: Zigbee Southbound traffic
Hi Steven,
meta has to include:
- access token
- version = 1
- sbtopic = 3
The receiver > apMac, the Zigbee radio_mac and the send to device mac have to be hex to byte encoded.
Here is our code for the conversion. rawHex is the mac address without colon.
public static ByteString convertHexToByte(String rawHex) { int l = rawHex.length(); byte[] data = new byte[l / 2]; for (int i = 0; i < l; i += 2) { data[i / 2] = (byte) ((Character.digit(rawHex.charAt(i), 16) << 4) + Character.digit(rawHex.charAt(i + 1), 16)); } return ByteString.copyFrom(data); }
I hope that helps.
Regards,
Jens
------------------------------
Jens Fluegel
Original Message:
Sent: Jan 21, 2022 04:36 AM
From: Steven Demets
Subject: Zigbee Southbound traffic
HI Jens,
Thanks for the information. It answered some questions that I had and showed the differences between your and my packets. The request ID in the send part, the FC and Seq# in the payload and that there is no need for the request.read part.
Unfortunately, I still have the same result, so now I'm wondering if my "meta" and "receiver" parts are ok. I've tried adding the same "access_token" to my meta, but that doesn't change anything. Is there anything you can tell me about your "meta" and "receiver" parts?
Thanks,
Steven
------------------------------
Steven Demets
Original Message:
Sent: Jan 20, 2022 03:25 PM
From: Jens Fluegel
Subject: Zigbee Southbound traffic
Hi Steven,
Year creating valid ZigBee payloads took us a moment as well for the Android app.
The payload is a byte array and has to include the frame control field (first byte), the sequence number (second byte) and the command id (third byte) which determines what kind of message it is. After this sequence the you have to add the list of attributes to read.
Here is an example for an attribute read message:
We also show a raw output for the protobuf message in the app (java specific). Here an example:
I hope that helps.
Regards,
Jens
------------------------------
Jens Fluegel
Original Message:
Sent: Jan 19, 2022 04:57 AM
From: Steven Demets
Subject: Zigbee Southbound traffic
Hi,
With the northbound part in my Zigbee test setup working (see my previous post here), I've now started the southbound part.
To test it I've attempted to make a simple attribute read request to get attribute 0x0004 from the Basic cluster (0x0000) of a HA device (profile 0x0104). This should give me the manufacturer name of the button I'm receiving the northbound messages from.
It should be rather straightforward as the Zigbee documentation explains how to use the global read attribute command.
The problem is that I'm not managing to create a protobuf packet that my AP accepts. I've created all kinds of JSON variables to inject in my protobuf encoder (using the 8.9 southbound .proto file), but every time I send it to the AP (through the established websocket connection), I get the following message:
"meta": { "version": "1", "accessToken": "someaccesstoken", "nbTopic": "actionResults" }, "reporter": { "mac": "AAAAAAAA", "time": "1642493380" }, "results": [ { "status": "invalidRequest", "statusString": "Unable to decode protobuf message" } ]
The reporter MAC address (AAAAAAAA) is obviously not correct, but I don't know if that is an issue or not or if it has a meaning.
Here is one of the JSON variables I've used to create the protobuf packet:
{ "meta": { "version": 1, "sbTopic": 3 }, "receiver": { "apMac": "b83a5ac2848a" }, "zigbee": { "radio_mac": "204c03fffea5f47a", "send": { "mac": "842e14fffe76a677", "e2pc": { "destination": { "endpoint": 1, "profile_id": 260, "cluster_id": 0 }, "source_endpoint": 1 }, "payload": "000004" } }}
What I would want to know is, what am I missing? I don't seem to be able to specify that I want to read the attribute, the payload is command ID + attribute ID, but nowhere is there an indication that I want to read something.
So I then added a read request to the JSON, but that yielded the same result.
{ "meta": { "version": 1, "sbTopic": 3 }, "receiver": { "apMac": "b83a5ac2848a" }, "zigbee": { "reqid": "da8bd0a4-ef5d-452a-8759-0bf5b6e203cf", "radio_mac": "204c03fffea5f47a", "send": { "mac": "842e14fffe76a677", "e2pc": { "destination": { "endpoint": 1, "profile_id": 260, "cluster_id": 0 }, "source_endpoint": 1 }, "payload": "000004" }, "request": { "read": { "reqid": "da8bd0a4-ef5d-452a-8759-0bf5b6e203cf" } } }}
Can anyone help?
Thanks in advance,
Steven
------------------------------
Steven Demets
------------------------------