How can we validate multiple messages on a kafka topic?
Hi Community,
We are using the Parasoft soatest to read messages from kafka topics and validate them against an expected message.
This works fine for cases when we expect 1 message. Also if we expect for example 3 messages, it works fine.
We encounter challenges when we have multiple messages in another sequence than we expect. For example, we expect message1, message2, message3. And the messages are place on the topic in the sequence 3, 2, and 1. Then our tests will fail, because the wrong messages are compare to each other.
Does anyone know how so solve this? Thanks a lot!
Answers
-
Normally, a correlation mechanism would be used to accomplish this. So, if there were multiple messages being received in unknown order then you could select them in whatever order you want based on some match or selection criteria.
At the moment the Kafka transport extension does not have any message correlation options. In your case, would you be able to share how you might be able to uniquely select each message? Is there a header value you could use to select each message in the desired order? Or is there something in the message payload you could use instead? If so, what message format is being used (JSON, Avro, XML, etc.)? Answering these questions can help us better understand how the Kafka transport extension may be improved to handle your case but also allow me to suggest an alternative solution for the present.
0 -
We read Json messages from the topic. In the message the following "ID's" makes the message unique.
"cID": "6517b713-b9e5-590e-08e4-83f1bb32b541",
"oMID": "fdeaa071-63e3-4460-b72b-59a65d2a427e",
"inMID": "575e3a73-7950-4d50-984f-d7359fc4b073"0 -
What criteria do you imagine using to retrieve the messages in a consistent order? You do not know those IDs beforehand, or do you? Or is there some JSON property in the message you could use?
0 -
There are indeed JSON properties in the message that are unique and we know them upfront. Can we use them when reading the message from the topic?
0 -
Can we use them when reading the message from the topic?
Not at present. At present, the Kafka transport polls for records and only returns the first one. Instead, it would need to iterate through them and parse the JSON to return the desired one instead. We have a feature like this for WebSockets but not for Kafka yet. There is an internal request for this which I updated with the detail you provided here.
For now, you will have to save each of the three messages somewhere, like using Data Banks. Then you may need to use scripting from an Extension Tool to sort and return them in the desired order. For example, your Extension Tool could return the three documents as a single JSON array and then you could chain a Diff tool to validate the contents.
Without using scripting, you could do something even more creative like using an XML Converter tool. Let's say you saved the JSON objects in Data Bank columns named "msg1", "msg2", and "msg3". You could create an XML Converter tool with Input tab set to the JSON "[${msg1}, ${msg2}, ${msg3}]". The XML Converter tool would be configured to convert from JSON to XML. Then you can chain a Diff tool in XML mode with "Ignore element order" selected.
0