At the end of part 1, I had got as far as joining my ZNP board to the Zigbee2MQTT (Z2M) coordinator PAN, and seeing the series of attempts by Z2M to interview my device. This is the point at which Z-Tool is evidently limited; it is not really feasible to respond to Z2M messages in a timely manner. To be fair, Z-Tool does have a scripting facility using a Javascript engine, but I didn’t find documentation on the ZNP API to match, so… I used Python and worked at the level of UART messages. Working at this level will make transferring what is learned into a MCU-based device, while using Python reduces the friction of a compile-upload-test cycle while learning. The Python code is available on github. This not meant to be a library/package and was substantially written as a learning activity. There are published Python packages, zigpy and zigpy-znp, but I felt that: a) using a full-featured package was likely make it harder to understand how things work; b) the comment “Zigpy is tightly integrated with Home Assistant’s ZHA component” suggested I would run into issues and complexity. Additionally, the Python code is just to demonstrate what the interactions should be for a MCU-based system, for which I will need to be working at a lower level than a Python object model. That said, there are some potentially-useful modules in zigpy-znp, which are not tightly integrated to ZHA and would be worth considering if using a Raspberry Pi etc as the basis for a Zigbee ZNP-based device.
Staying Sane and in Control
By the very nature of experimentation, there will be changes in intent and design, errors, etc. Staying in control is helped by removing previously-joined devices from Z2M, making sure the device is set up from a restart, and then re-joining. This isn’t always necessary but…
There is also ample opportunity to get inconsistent set-up in Z2M. If the cluster specification in AF_REGISTER changes in an incompatible way (e.g. the change is not simply an addition), the information which Z2M retains, even when removing and re-joining can mess things up. In this case, stop Z2M, go to the “data” folder of the Z2M installation and remove coordinator_backup.json and database.db, and restart Z2M. This will completely mess up any existing Zigbee network.
Setting up a sand-box coordinator + Z2M installation is probably a good idea to avoid messing up an existing network but also because having lots of devices and even one router is going to make Wireshark captures get very busy indeed. Use a different Zigbee channel for the sandbox; as well as avoiding contention between the radios, this means you can assign the channel Wireshark to only sniff the sandbox messages.
I’ve also found it is generally a good plan to keep Z2M with joining disabled until you want to join a device, after having started Wireshark and generally getting organised.
Z2M Requests Attribute Values – the “Interview”
Stepping back from the Z2M interview…
I am interested in three kinds of interaction between the ZNP board and coordinator/Z2M: Z2M requests attribute values, Z2M requests an action, and the ZNP board sends reports. All of these are in the context of Zigbee Clusters and use Zigbee Cluster Language (ZCL) messages embedded within the ZNP commands sent over the serial interface to the ZNP board. It is important to note that both the ZNP messages and the ZCL messages embedded within them (as the “data” of the ZNP message) have parts with the same (or similar) names and functions: a sequence identifier, a command identifier, and a body/payload/data. The use of a Zigbee sniffer, such as ZBOSS + Wireshark, really helps with interpreting this nested structure and debugging (e.g. to spot malformed ZCL).
Referennce [R6] (see part 1) describes the structure of a ZCL message frame (section 2.4, “Command Frame Formats”), lists the permitted ZCL command ids, and specifies the content of the ZCL message payload for each type of command (section 2.5). The commands relevant to my “three kinds of interaction” are: 0x00 (read attributes), 0x01 (read attributes response), and 0x0a (report attributes). These are all “global” commands in the sense that these commands ids have the same meaning for all clusters. The alternative kind of command is “specific”, meaning that the command id has a meaning which is specified separately for each cluster. Whether the command id is interpreted as global or specific is indicated in the ZCL frame control field (FCF) via bits 0 and 1 of the FCF (see [R6]). The way Zigbee is designed, things like turning (e.g.) a lamp on and off are achieved by sending cluster-specific commands, rather than by writing attributes, whereas the current state of the lamp is determined by reading attributes. This is why I said “Z2M requests an action” in my list of kinds of interaction, above.
Returning to the interview, which is a set of messages requesting the values of attributes in the Basic Cluster.
ZNP passes requests to read attributes to its serial interface as an AF_INCOMING_MSG (see [R2]). The message contains information about the endpoint, source network id (this will be 0x0000, the coordinator), cluster identifier, etc and a ZCL message as its “data” component. For the interview, the cluster id will be 0x0000 (Basic Cluster) and the ZCL command id will be 0x00 (read attributes). Consulting section 2.5.1 in [R6], we can see that the ZCL message comprises a standard ZCL header + a list of one or more two-byte attribute identifiers (Z2M will sometimes issue requests for one attribute per incoming message, and sometimes request several attributes at the same time). The meaning of the attribute identifiers for the Basic Cluster is given in section 3.2.2.2 of [R6].
Care must be taken when handling the cluster id and attribute ids (and any multi-byte component) because little-endian byte order is used. Wireshark is your friend in checking for byte-order bugs.
The ZNP which should be sent in response to an AF_INCOMING_MSQ is AF_DATA_REQUEST (yes, it is called “REQUEST”), which will in turn give rise to ZNP sending both AF_DATA_REQUEST_RSP and AF_DATA_CONFIRM messages. A sequence identifier in AF_DATA_REQUEST matches that in AF_INCOMING_MSQ. My approach is to always wait for a “RSP” before proceeding and to log AF_DATA_CONFIRM as they arrived (this is OK for the Z2M interview, but maybe not always). Note that these messages do not necessarily mean that the attribute values you sent back have actually been handled by Z2M.
Here is a fragment of the annotated log which my Python script emits for the interview, comprising the interactions for a single attribute in the Basic Cluster:
RX body: Cmd: 4481 Body: 00 00 00 00 00 00 01 01 00 31 00 3b 3d 01 00 00 07 10 10 00 05 00 04 00 82 3d 1d Incoming ZCL for endpoint 1, cluster id = 0000 is: 10 10 00 05 00 04 00 ZCL Command = read attributes (0x00) Sending response... TX: fe 24 24 01 00 00 01 01 00 00 00 00 10 1a 18 10 01 05 00 00 42 08 5a 4e 50 2d 54 65 73 74 04 00 00 42 05 41 52 43 31 32 02 [Response] RX body: Cmd: 6401 Body: 00 AF_DATA_REQUEST_RSP success? True -------------- RX body: Cmd: 4480 Body: 00 01 00 AF_DATA_CONFIRM for TransId=0 on Endpoint=1 had Status=0
The request is a composite requst for the values of two attributes: Manufacturer Name (0x0004) and Model Identifier (0x0005). The response (notice the ZCL command is now 0x01 = read attributes response, and that the ZCL FCF shows the client-server direction is reversed) contains strings for the attribute values. Strings are data type 0x42 (table 2-11 in [R6]) and the “value” given in the data has the length of the string as the first byte, followed by the character codes. Section 2.5.2 of [R6] shows how each attribute requested gets a chunk in the respons comprising: 2 bytes of attribute id + 1 byte status (=0x00) + 1 byte data type (=0x42) + 1 byte string length + n bytes of string characters.
The Wireshark log is helpful to understand and verify what went on. The messages are built up in multiple (nested) layers. The ZCL forms the inner-most, with the Zigbee Application Support Layer above. These are the two most relevant for understanding the ZNP interactions; the other layers are concerned with the underlying network interactions.
The “Read Attributes” and “Read Attributes Response” to match the Python log above are both marked as having protocol = “Zigbee HA” – Wireshark has detected we’re using the Home Automation profile. The Wireshark fragment for the request is:
ZigBee Application Support Layer Data, Dst Endpt: 1, Src Endpt: 1 Frame Control Field: Data (0x00) Destination Endpoint: 1 Cluster: Basic (0x0000) Profile: Home Automation (0x0104) Source Endpoint: 1 Counter: 145 ZigBee Cluster Library Frame, Command: Read Attributes, Seq: 16 Frame Control Field: Profile-wide (0x10) Sequence Number: 16 Command: Read Attributes (0x00) Attribute: Model Identifier (0x0005) Attribute: Manufacturer Name (0x0004)
And for the response:
ZigBee Application Support Layer Data, Dst Endpt: 1, Src Endpt: 1 Frame Control Field: Data (0x00) Destination Endpoint: 1 Cluster: Basic (0x0000) Profile: Home Automation (0x0104) Source Endpoint: 1 Counter: 3 ZigBee Cluster Library Frame, Command: Read Attributes Response, Seq: 16 Frame Control Field: Profile-wide (0x18) Sequence Number: 16 Command: Read Attributes Response (0x01) Status Record, String: ZNP-Test Status Record, String: ARC12
Digging into the component parts of the ZCL in Wireshark, and seeing how elements relate to the bytes-level view should clarify the comment above: “each attribute requested gets a chunk in the respons comprising: 2 bytes of attribute id + 1 byte status (=0x00) + 1 byte data type (=0x42) + 1 byte string length + n bytes of string characters”.
Beyond the Interview – a Switch and a LED – the Generic OnOff Cluster
Defining two endpoints on the ZNP board, one for a switch and one for a LED, is sufficient to explore the “three kinds of interaction” through: the Z2M web dashboard, Wireshark, and serial interface (whether via Z-Tool, Python, etc). NB: I use “switch” to mean something like a push button or toggle switch, rather than as a synonym for a relay (the practice of using it as a synonym for relay should be strenuously avoided as it makes the word “switch” become ambiguous).
The AF_REGISTER command (see part 1) is used twice, once for each endpoint. Both the LED and switch use cluster 0x0006 (generic On/Off). For the switch, I added cluster 0x0006 as both an input and output to endpoint 1. The input allows the switch to be remote-controlled from Z2M whereas the output is used to inform the coordinator/Z2M of a change in switch state. State changes are sent as ZCL reports (ZCL command id = 0x0a) whenever the switch changes state. The same kind of ZCL message can also be used for periodic reporting, e.g. for a sensor. I put the LED on endpoint 2. In this case, the appearance of 0x0006 in the in-cluster list is what allows the LED to be switched on and off (analogous to remote controlling the switch) and in the out-cluster list it means the LED state is reportable (in my Python I use a periodic report for this, for demonstration purposes, but an event-driven report would be more sensible in practice).
After issuing ZDO_STARTUP_FROM_APP, that is all that is needed to set the ZNP board up. The real work is done handling incoming ZCL and preparing outgoing ZCL. We do, of course, need to have a converter JS file in Z2M too. There is a rather hacky converter in the github repo, which will work but may not be optimal!
I will now outline what happens for each of the remaining “three kinds of interaction” (reading the state of the LED/switch simply being the same pattern as reading the Basic Cluster attributes) in the context of this simple test-device.
Reporting Switch Changes and LED State
These two are identical in the structure of the ZCL, which is very close to that used in the “read attributes response”; the only differences in the ZCL are the command id (0x0a) and the report data structure does not have a status byte. See the frame format for the Report Attributes Command [R6]. The attribute id for on/off state is 0x0000 and for the ZCL we need its data type (boolean, type id = 0x10) and one byte for on (0x01) or off (0x00). Remember that the switch and LED are on different endpoints, but also recall that the endpoint is included in the data sent via an AF_DATA_REQUEST ZNP message. This is the message to use although, again, having “REQUEST” in the name seems wrong for a report; its just the way it is! As for the Read Attributes Response case, expect AF_DATA_REQUEST_RSP and AF_DATA_CONFIRM in response to the report message.
Reference to section 3.8 in [R6] describes other attributes which a generic on/off device may support.
Here is my Python log for reporting when I pressed the button linked to endpoint 1:
TX: fe 11 24 01 00 00 01 01 06 00 00 00 10 07 18 00 0a 00 00 10 01 26 [Response] RX body: Cmd: 6401 Body: 00 AF_DATA_REQUEST_RSP (for report) success? True -------------- RX body: Cmd: 4480 Body: 00 01 00 AF_DATA_CONFIRM for TransId=0 on Endpoint=1 had Status=0
And the Wireshark capture to match:
ZigBee Application Support Layer Data, Dst Endpt: 1, Src Endpt: 1
Frame Control Field: Data (0x00)
Destination Endpoint: 1
Cluster: On/Off (0x0006)
Profile: Home Automation (0x0104)
Source Endpoint: 1
Counter: 2
ZigBee Cluster Library Frame, Command: Report Attributes, Seq: 0
Frame Control Field: Profile-wide (0x18)
Sequence Number: 0
Command: Report Attributes (0x0a)
Attribute Field
Attribute: OnOff (0x0000)
Data Type: Boolean (0x10)
On/off Control: On (0x01)
Remote Control
This is about using the Z2M front end or MQTT publish activity to control the ZNP board device.
Turning the LED on and off and remote-controlling the switch are identical as far as the ZCL goes. In our case, the only difference is the endpoint which comes through with the AF_INCOMING_MSG. The ZCL has a different structure to the previous cases seen and the ZCL FCF is what signals the difference. Recall that attribute requests had a FCF of 0x10 and 0x18 for the response and reports. These commands have the same meaning no matter what cluster is involved. The important conceptual point is that the ZCL for turning something on or off is not the same as setting the attribute for state; it is an action which has specific meaning only within the context of the genOnOff cluster (0x0006). The FCF signals the fact that the command is “local or specific to a cluster” (see [R6]) via bit 0, which is unset for “global” (any-cluster) commands and set for local commands; checking the FCF is key to handling these messages correctly. The remainder of the ZCL after the FCF is very simple; first there is the usual sequence number, then there is a single command byte, where 0x01 means “turn on” and 0x00 means “turn off”. Section 3.8.2 of [R6] describes four other commands for use cases such as off-with-fade, on-with-timed-off, etc.
Having received the command, we should reply to Z2M because the FCF says a default repsonse is expected (bit 4, “disable default response” is not set). A default response is simply a ZCL message with a command id of 0x0b and a short payload indicating the command which was sent and a byte for success/fail (see [R6] 2.5.12). This is a global, not a cluster-local, command. Our default response message leads to a AF_DATA_REQUEST_RSP and AF_DATA_CONFIRM, as for any other AF_DATA_REQUEST.
Here is the message log from my Python script for turning the LED on:
RX body: Cmd: 4481 Body: 00 00 06 00 00 00 01 02 00 3c 00 40 16 03 00 00 03 01 29 01 82 3d 1d Incoming ZCL for endpoint 2, cluster id = 0006 is: 01 29 01 Local/specific command to cluster 0006 => device on endpoint 2 set to: on Sending response... TX: fe 0f 24 01 00 00 01 02 06 00 00 00 10 05 18 29 0b 01 00 01 [Response] RX body: Cmd: 6401 Body: 00 AF_DATA_REQUEST_RSP success? True -------------- RX body: Cmd: 4480 Body: 00 02 00 AF_DATA_CONFIRM for TransId=0 on Endpoint=2 had Status=0
And here is the Wireshark capture for the same on command:
ZigBee Application Support Layer Data, Dst Endpt: 2, Src Endpt: 1 Frame Control Field: Data (0x00) Destination Endpoint: 2 Cluster: On/Off (0x0006) Profile: Home Automation (0x0104) Source Endpoint: 1 Counter: 151 ZigBee Cluster Library Frame Frame Control Field: Cluster-specific (0x01) Sequence Number: 41 Command: On (0x01)
and the default response (just the ZCL part this time):
ZigBee Cluster Library Frame, Command: Default Response, Seq: 41 Frame Control Field: Profile-wide (0x18) Sequence Number: 41 Command: Default Response (0x0b) Response to Command: 0x01 Status: Success (0x00)
Footnote
While these two posts are a long way short of a comprehensive tutorial, I hope they provide enough structure to make it easier to understand what is going on and to work out the details. I found it very helpful to work carefully through the content of messages both at the UART/ZNP level and at the Zigbee sniffer (Wireshark) level, correlating the bytes with the relevant documentation.
If you find errors, especially where these are misunderstandings of how ZNP and Zigbee work, please do comment.