Q-Learning for 6TiSCH Network Formation and Transmission

Problem

In 6TiSCH, joining nodes send control messages over one shared "minimal cell", so frequent messages collide and congest it, while the Trickle timer that paces them uses fixed parameters. Once the network runs, RPL picks parents using failed transmissions only, ignoring TSCH traffic and queue load, and its parent-switching rule does not adapt. The result is slow formation, dropped packets, wasted energy, and higher latency.

Approach

I gave every node its own lightweight Q-learning agent. Q-Trickle observes minimal-cell busy ratio and queue use, decides whether to transmit or suppress each routing message (DIO), adapts the Trickle interval and redundancy constant, raises the beacon rate early for fast sync, and limits other traffic on the minimal cell. ACI-RPL observes slotframe utilization and link-metric stability to decide when to change parent, shares Q-tables with newly joined nodes through DIO messages, scores parents with a cell-usage and traffic-aware objective function, and sizes initial transmit cells from packet-drop rate.

How It Works

1. 1
   ObserveEach node measures minimal-cell busy ratio, queue use, slotframe utilization, and link quality.
2. 2
   Decide: formationQ-Trickle chooses to transmit or suppress each DIO and adapts the Trickle interval and redundancy.
3. 3
   Decide: transmissionACI-RPL chooses to stay with or change the parent, scored by a cell-usage and traffic-aware objective function.
4. 4
   ShareNew nodes inherit Q-tables from parent candidates through DIO messages (cooperative multi-agent Q-learning).
5. 5
   EvaluateBenchmarks in the 6TiSCH simulator (up to 100 nodes), then real runs on FIT IoT-LAB with OpenWSN.

What I Built

• Q-Trickle: Q-learning DIO decisions, a dynamic Trickle interval and redundancy constant, an adaptive beacon rate, and a transmission limit on the minimal cell.
• ACI-RPL: cooperative Q-learning parent change with Q-table sharing, a cell-usage and traffic-aware objective function, and improved initial transmit-cell allocation.
• Q-learning parameter studies and convergence analysis, including node addition and removal.
• Real-hardware runs on the FIT IoT-LAB Strasbourg testbed with OpenWSN on up to 32 ARM Cortex-M3 nodes.

Impact

Against the original Trickle timer, Q-Trickle cut joining time by 13%, energy by 11%, and failed routing messages by 43% (IEEE Access, 2023). Against standard RPL, ACI-RPL raised packet delivery by 12% and received packets by 17%, and cut energy by 23% and latency by 9% (IEEE Access, 2024). Each fits in about 4.3 KB of extra firmware and about 100 bytes of RAM. The work became the Ph.D. thesis "Improving 6TiSCH Network Formation and Transmission using Q-Learning" (2023).