Cross-compiled simulation testing
- Ken Bannister
During development of the simulation cross-compile framework, we saw a significant number of desynchronization errors when running the simulation. We also saw process memory usage increase. This page provides test results to quantify these issues and decide whether to issue a pull request for the cross-compilation work.
Based on these results, I plan to proceed with the pull request. However, I recommend the amd64-windows firmware be labeled experimental and that we solicit user feedback. I also plan to file a bug for the memory usage.
Test Runs
A simulation test run used a three-mote simulation, with two nodes one hop away from the RPL root. The link quality between each node to the root was set to a 1.0 PDR. Each test run lasted approximately six hours. We used the OpenVisualizer develop_SW-112 branch to run the simulation.
Guide to table columns:
- Arch-OS, CPU Firmware architecture, and the Intel CPU architecture on which the test was run. Windows tests were run on Windows 7.
- FW Branch Most tests run on the FW-177i branch of the kb2ma repository, but a couple of runs used the current openwsn-fw develop branch.
- Elapsed ASN Difference between ASN at end vs. start of run, on RPL root.
- Memory Consumed Process memory use as reported by Windows Task Manager or Linux pidstat. Top value is normalized on elapsed ASN.
- Desync Errors Count of desynchronization errors reported in OpenVisualizer log. Top value is normalized on elapsed ASN.
| Arch-OS, CPU | FW Branch | Elapsed ASN | Memory Consumed | Desync Errors |
|---|---|---|---|---|
| x86-windows, i3 | FW-177 | 14.9M slots | 132K slots/MiB 113 MiB | 877K slots/error 17 errors |
| x86-windows, i3 | develop, native compile | 16.0M slots | 133K slots/MiB | 593K slots/error 27 errors |
| x86-windows, i7 | FW-177 | 26.3M slots | 199K slots/MiB | 560K slots/error 47 errors |
| amd64-windows, i7 | FW-177 | 27.7M slots | 73.8K slots/MiB 375 MiB | 577K slots/error 48 errors |
| amd64-windows, i7 | FW-177 | 27.2M slots | 73.9K slots/MiB 369 MiB | 567K slots/error 48 errors |
| amd64-windows, i3 | FW-177 | 15.7M slots | 73.7K slots/MiB 214 MiB | 563K slots/error 28 errors |
| amd64-linux, i5 | FW-177 | 12.6M slots | 68.6K slots/MiB 184 MiB | 843K slots/error |
| amd64-linux, i5 | FW-177 | 10.5M slots | 68.9K slots/MiB 152MiB | 955K slots/error 11 errors |
| amd64-linux, i5 | develop | 10.5M slots | 68.7K slots/MiB | 501K slots/error 21 errors |
The first run of amd64-windows on the i3 CPU (not in the table above) failed after around four hours. When we observed the run after six hours, ASN was not increasing, but process memory was fluctuating within a few KB, and CPU usage was 0. There was no output on the terminal to indicate an error occurred.
Discussion
The x86-windows FW-177 firmware performed no worse than the develop firmware. The error rate on the amd64-windows was consistently at 500K - 600K slots/error, but some of the x86-windows and amd64-linux runs showed this same error rate.
There is clearly a memory leak in the OpenVisualizer process, regardless of the firmware used, architecture, or operating system.
It is surprising that the i5-based Linux simulations ran significantly slower than the i3-based Windows simulations.