MEAL
Lifelong learning,
one kitchen at a time.
MEAL — Multi-agent Environments for Adaptive Learning. A GPU-accelerated, fully JAX benchmark that trains agents on sequences of 100+ tasks in a few hours on a single GPU.
Much of existing continual reinforcement learning research evaluates on 3–10 tasks, mainly because CPU-bound simulators make longer runs impractical, and continual learning in cooperative multi-agent settings lacks a dedicated benchmark. MEAL addresses both gaps and shows that long task sequences reveal failure modes that may not emerge at smaller scales.
A first, on two fronts
The first benchmark for continual multi-agent RL, and the first continual-RL benchmark with hardware-accelerated execution.
Endless kitchens
Procedural task generation with scalable difficulty, yielding a near-infinite supply of uniquely solvable kitchen layouts.
Scale changes the story
Conclusions shift as sequences lengthen, and retaining cooperative behavior is a distinct challenge that worsens with more agents, harder tasks, and changing partners.
An on-demand near-infinite supply of kitchens
Every layout is generated on the fly and validated to be solvable, ensuring agents can collectively complete the cook–deliver cycle. The result is a near-infinite pool of unique kitchens.
Draw an empty room with outer walls, sized from the difficulty range.
Inject the interactive tiles: onion pile, plate pile, pot, and delivery counter.
Add internal walls until obstacle density matches the target level.
Drop in the chefs, then validate the layout, and prune unreachable tiles.
Turning up the heat
Bigger grids and denser walls mean longer paths, tighter chokepoints, and split kitchens, forcing agents from solo cooking into genuine division of labor.
Level 1
EASY
Compact layout. Ingredients, plates, and pots are close together. The cook-deliver cycle is short and easy to remember. Agents can often finish the task solo, with little coordination.
Level 2
MEDIUM
Stations spread out, paths lengthen, and the action sequences to cook and deliver grow harder to remember. Splitting the work starts to pay off.
Level 3
HARD
Long paths between stations. The kitchen is likely to split into separate zones, such that onions and plates must be passed across counters.
A limited view of the kitchen
Although Overcooked is fully observable by design, MEAL introduces a partial observability variant. Each agent receives only an egocentric observation window, masking all tiles outside a fixed radius. This window scales with difficulty, keeping the visible area at ~12% of the kitchen across levels. Teammates can move out of view entirely, so an agent cannot always see what the other chefs are up to.
The 2x3 window shows one tile to each side and one ahead. The layout is compact, so most stations are only a few steps from coming into view.
The 3x3 window adds an extra visible row in front of the agent. However, the larger layout still spreads ingredients, plates, and pots farther out of view.
The 5x3 window now reaches one row behind the agent and an extra row ahead. Even so, the view covers just a corridor of the kitchen, and teammates routinely vanish from sight.
Learning to cooperate with new teammates
Beyond changing layouts, MEAL exposes a second source of non-stationarity unique to multi-agent settings: changing cooperation partners. Rather than training on a fixed teammate, agents must adapt to a sequence of diverse partners, each with a different strategy, while retaining the ability to coordinate with earlier ones.
Partner pool
Partners are drawn from three pools: hardcoded strategies (random and static agents), planning-based agents (onion-only, plate-only, and a human-like planner with stochastic actions), and population-trained agents using Best-Response Diversity (BRDiv), which maximizes self-play performance while minimizing cross-play compatibility. Each partner is frozen, only the ego-agent learns continually.
Extending MEAL beyond the kitchen
MEAL brings continual learning to four cooperative multi-agent environments from JaxMARL. We selected Overcooked for the core evaluation and analysis in our work. The other three cover different cooperative settings.
Three core metrics to capture CL dynamics
Raw delivery counts are not directly comparable across layouts of different sizes and complexities, so MEAL normalizes soup delivery by the optimal single-agent cook–deliver cycle for each task. We then use this score to calculate the following continual learning metrics.
Mean normalized soup delivery across all tasks in the sequence. Reflects the overall balance between stability and plasticity. A score of 1 indicates optimal performance of a single agent. Values above 1 display effective cooperation that exceeds solo efficiency.
Performance drop on past tasks due to interference from new training on new ones. Forgetting measures the performance gap on a task between right after training and after the full sequence. We average this across all tasks in the sequence.
How much prior experience accelerates and improves learning new tasks. Measured as normalized improvement in area under the learning curve (AUC) over a single-task baseline. Positive means past tasks help, negative means they interfere.
Benchmark results
Algorithm choice, team size, difficulty, sequence length, observability, and partner diversity all interact, and their effects only become clear at scale.
MARL Algorithm and CL method combinations matter
Neither the MARL algorithm nor the CL method alone determines performance. The pairing counts. AGEM only works well with IPPO, while HAPPO pairs the best with EWC. MAPPO appears to be least compatible.
Shorter sequences hide how methods diverge
The performance of IPPO-combined CL methods progressively spreads out. What looks like a small gap after task 5 becomes a wider spread by task 20. PackNet, followed by Online EWC and Online MAS consistently achieve high delivery. The naïve Fine-Tuning (FT) baseline displays complete amnesia.
PackNet leverages past experience
PackNet shows positive forward transfer on most tasks. Past training accelerates learning on new kitchens.
Does performance on old tasks decline over time?
FT reaches high delivery during training, but immediately forgets the task when moving to new ones. The sawtooth pattern displays the interference with previously learned behaviors.
Online EWC almost perfectly retains performance on learned tasks. Each task's curve stays near its trained level after the agent moves on.
Methods occupy distinct stability–plasticity regions
Plotting all methods in the forward transfer vs. forgetting plane reveals a clear trade-off. Regularization-based methods excel at low forgetting. EWC and MAS sacrifice transfer to achieve this. AGEM sits at the opposite extreme: plastic but forgetful. PackNet yields the best trade-off.
Short sequences hide the divergence
EWC and Online EWC look nearly identical at 5 tasks. Over 100 tasks, standard EWC increasingly over-regularizes the shared backbone, saturating early and losing plasticity, while Online EWC retains capacity through exponentially decaying importance weights. Rankings that appear stable at short horizons can reverse entirely at scale.
Two or three chefs is the sweet spot
MEAL extends JaxMARL Overcooked to natively support N agents. Two agents enable parallel cooking, pushing throughput well above the solo baseline. Performance peaks at three agents, then falls as each additional agent enlarges the joint action space. Additional chefs crowd the kitchen and make continual coordination harder to retain.
Performance drops and forgetting grows with difficulty and sequence length
Performance drops from Level 1 to Level 3, since larger grids and denser obstacles lengthen action sequences and demand tighter coordination. Forgetting grows too, and over a 100-task sequence it keeps accumulating, faster at higher levels where new layouts conflict more with old ones.
NETWORK PLASTICITY
MEAL degrades network plasticity
Repeating the same 10-task sequence more often (10 vs 3) costs the network plasticity. Learning capacity drops and more neurons go dormant as repetitions increase. Even so, this loss is not as notable as in prior single-agent work despite running 10B steps.
A CNN encoder recovers the observability gap
Partial observability lowers delivery by masking distant tiles and destabilizing value targets. Replacing the MLP with a CNN largely recovers this loss, and on Level 2 often surpasses the fully observable baseline. Spatial inductive bias becomes an asset when direct observation of the full grid is unavailable.
Retaining coordination with past partners requires a CL mechanism
The ego agent meets eight partners in turn on popular Overcooked layouts. FT often forgets coordination as the partner chef is switched out. Online EWC and Online MAS remember past partners and score notably higher.
