Questions beyond the final-year project

DGLight was my undergraduate final-year project. It explores how reinforcement learning can adapt large language models for traffic signal control. The basic pipeline first trains a CoLight-based DQN Critic, then freezes it and uses its action-value estimates to score candidate signal decisions generated by an LLM. These scores are then used to post-train the LLM with GRPO.

This post records a discussion about the paper with a master's student at Beijing University of Posts and Telecommunications. Their questions concerned the reliability of the reward signal, coordination between decisions at different intersections, and the trade-offs between post-training approaches. They also gave me an opportunity to revisit which design choices came from my understanding of the problem, and which were shaped by limitations in time, knowledge, and computing resources.

The Chinese version preserves the student's questions verbatim and presents my lightly edited replies in blockquotes. This English version translates both sides of the exchange. Paragraphs labelled "Editorial note" were added while preparing this post to distinguish preliminary judgments made during the discussion from conclusions that still need validation.

Can Critic bias carry over to the LLM?

Question: On evaluation bias in the Critic: DGLight's GRPO rewards come directly from the Q-values of a pretrained DQN Critic, so the optimization of the LLM's policy depends to some extent on the accuracy of the Critic's evaluations. If the Critic's value estimates are biased in certain traffic states, does that mean the LLM could inherit or even amplify those biases?

Some degree of estimation bias in the Critic is inevitable, and its extent depends on how the Critic is trained. Due to time constraints during this project, I was unable to study the training methods of the candidate Critics in great detail. However, I observed that DQN Critics trained in simulation using data from one city all showed a noticeable drop in performance when tested in an unfamiliar city. My preliminary judgment was therefore that these Critic networks had some tendency to overfit.

The purpose of training the LLM is to adapt its general reasoning ability to traffic control. My expectation is that post-training can bring the LLM close to the Critic's control capabilities, while its general reasoning ability gives it a weaker tendency to overfit. As for systematic errors in the Critic, my judgment during the discussion was that the LLM might inherit them to some extent, but would not amplify them. The definition of "error" matters here, because GRPO's group-relative calculation naturally removes certain shared offsets.

Editorial note: "Would not amplify them" was too categorical. General reasoning ability does not guarantee that a model will not overfit. A drop in performance across cities indicates limited generalization, but that observation alone is not enough to attribute the entire problem to overfitting.

Under GRPO's group-relative advantage calculation, if the same constant is added to the rewards of every candidate action in a group, that offset cancels when the group mean is subtracted. However, if the Critic's errors differ across actions, or even reverse their ranking, the group-relative calculation cannot remove those biases. This suggests that the LLM could still learn incorrect preferences; whether it further amplifies them requires additional experiments.

From individual intersections to the whole network

Question: On joint decision-making across intersections: I noticed that during rollout, DGLight samples and evaluates candidate actions separately for each intersection, then combines these into joint actions. In my own simulation experiments, I encounter queue spillback between adjacent intersections, leading to intersection deadlocks. Could this result in decisions that are locally optimal but globally suboptimal?

If my judgment is correct, queue spillback did not occur in my datasets. This is indeed something I had not noticed. I cannot draw a conclusion on this point because I am not sure whether the CityFlow simulator supports queue spillback.

Editorial note: DGLight's CoLight Critic aggregates information from neighboring intersections through graph attention, so it does not ignore relationships between intersections entirely. However, taking neighboring information into account and fully evaluating the consequences of joint actions are still different things. The existing experiments are insufficient to give a definitive answer about queue spillback.

Why choose Critic-guided GRPO?

Question: On the comparison with CoLLMLight's post-training mechanism: I previously came across CoLLMLight, the latest paper from the same team as LLMLight. It uses PPO, with rewards designed around real-time decision outcomes and reasoning utility, whereas DGLight uses a DQN Critic's Q-values to reward candidate actions and optimizes the policy through GRPO. What do you see as the main advantages of Critic-guided GRPO over PPO based on environmental feedback? Could the two be combined, for example by using the Critic to provide fine-grained action-value feedback while introducing environment-level reward constraints to reduce the impact of biased Critic estimates?

My original reason for adopting this pipeline was insufficient compute. I wanted a way to greedily estimate the value of every signal phase at every intersection and time step in advance, so that I could later adjust the scale of rollout data dynamically according to the available compute.

The core reason for using GRPO is to eliminate the influence of absolute reward values. In CoLLMLight, the authors construct the reward signal using an exhaustive search with 64 branches per state and a greedy strategy. Assuming that the resulting reward signal is almost completely accurate, using PPO can avoid GRPO's requirements on sampling group size and distribution.

In particular, that work's reward signal also incorporates a reasoning-budget signal, so the absolute signal is relatively important, making PPO the most suitable choice. When I initially designed my work, my choices were shaped more by the limits of my knowledge and computing resources, so this was the approach I considered best at the time.

Editorial note: The explanation above about search accuracy and algorithm choice is my interpretation of the two approaches, rather than an experimental finding about their relative merits. A search with a finite number of branches is still an approximation; combining traffic performance and reasoning cost in a reward does not, by itself, establish that PPO must outperform GRPO.

For DGLight, using a pretrained Critic to provide action values allows scores for different signal phases in the same state to be saved and reused for later policy optimization. This is what made the approach attractive given my available resources.

Closing reflections

I am grateful to this student from Beijing University of Posts and Telecommunications for sharing the ICLR 2026 paper CoLLMLight and for raising these specific questions about DGLight. The discussion gave me an opportunity to consider the approach I used in my final-year project alongside another training pipeline, and to see my original design choices and their limitations more clearly.

CoLLMLight made me realize that conditional greedy search can greatly reduce the search space in traffic signal simulation while still achieving reasonable estimation accuracy.

The discussion also made me recognize an inherent limitation of DGLight: during GRPO rollout collection, the traffic states visited by the policy shift away from the state distribution covered during reward-model training. As a result, the signals provided by the pretrained reward model may no longer accurately reflect the value of each state and action encountered during actual GRPO training.

The key issue is the difference between the state distributions involved in Critic training and LLM post-training. Even if the Critic performs well in its original training environment, that does not directly establish that it can provide equally reliable scores for the new states visited by the LLM. As the policy changes, the effect of this difference on the reward signal needs to be understood within the actual training process.

When completing my final-year project, I chose what I considered the best approach within the limits of my knowledge, time, and computing resources. This discussion has helped me revisit those choices from a new perspective and recognize more specifically which aspects of the work I had not considered fully. Recording these questions and thoughts gives me a way to continue reflecting on the project after its completion.