- 1.Wait for a complete response before processing
- 2.Want to process one response at a time
Whilst this might seem to occur naturally most of the time, network contention, network errors and high data rates will trip you up. CBus is one system where I’ve often see back to back messages returned in a single IO read.
Engine ships with two tokenisers to help you break up the incoming data. The default buffered tokeniser and the more advanced abstract tokeniser.
# Device driver helper
tokenize delimiter: "\x0D"
tokenize indicator: "\x02", delimiter: "\x03"
"yu\x03\x02hello\x03\x02world\x03\x02how"Would have the following result:
yu\x03would be discarded
hellowould be returned
worldwould be returned
\x02howwould be buffered
The primary use case for this tokeniser is variable length messages, where length can be determined by the message contents. (commonly a length field in the header)
tokenize indicator: "\xAA", callback: :check_length
# Called by the Abstract Tokenizer
# Check for minimum length
return false if byte_str.bytesize <= 3
response = str_to_array(byte_str)
# data length byte + (header + checksum) == message length
len = response + 4
if response.length >= len
# return the length of this message (any excess will be buffered)
# false if the complete message hasn't arrived yet
For a detailed overview of what these tokenisers are capable of, it is worth looking at their tests.