TL;DR: ARLBench provides a standardized AutoRL benchmark by combining JAX-powered implementations with representative subsets, making high-performance AutoRL research accessible.

• Speed: Delivers up to 10x faster evaluation than standard libraries, using JAX-based training and statistically selected environment subsets.
• Flexibility: Supports static, multi-fidelity, and dynamic HPO (like PBT) through a Gymnasium-like interface with full checkpointing capabilities.
• Validity: Uses a massive dataset of 100,000+ runs to prove that its environment subsets accurately represent the global RL task space.
• Key Insight: RL-specific HPO benchmarks contrast sharply with supervised learning and are therefore highly desirable; standard “off-the-shelf” optimizers can struggle against Random Search on RL’s rugged landscapes.

Check out the GitHub repository.

Hyperparameter optimization (HPO) is one of the main obstacles to reliable reinforcement learning (RL). Modern RL algorithms expose large, highly sensitive configuration spaces, and their performance varies widely across environments and random seeds. Despite progress in automated RL (AutoRL), empirical evaluations of HPO methods remain expensive, fragmented, and difficult to compare. ARLBench addresses this problem by introducing an efficient and standardized benchmark for HPO in RL that substantially reduces computational cost while maintaining a representative comparison of HPO methods.

What makes HPO for RL different from Supervised Machine Learning?

In contrast to supervised learning, HPO in RL faces non-stationary training dynamics, rugged optimization landscapes, and high variance across seeds. Small changes in hyperparameters can lead to catastrophic failure, and good configurations rarely transfer between environments. As a consequence, most HPO methods are evaluated on a small number of tasks with limited configuration spaces, making it difficult to assess generality or establish meaningful baselines. ARLBench is designed to provide a common ground for evaluating AutoRL methods under realistic conditions.

How does ARLBench Solve This?

ARLBench is a benchmark framework built specifically for HPO in RL. It combines highly optimized JAX-based implementations of DQN, PPO, and SAC with a flexible interface that supports static, multi-fidelity, and dynamic hyperparameter optimization. The framework is complemented by a systematic subset selection strategy for environments and a large-scale meta-dataset capturing hyperparameter landscapes across algorithms and domains.

A central goal of ARLBench is efficiency. Compared to commonly used RL libraries, JAX implementations yield substantial speedups, and using representative environment subsets further reduces runtime. Together, these design choices lower the cost of evaluating HPO methods by approximately an order of magnitude.

The AutoRL Environment Interface

At the core of ARLBench lies the AutoRL Environment, which serves as the interaction point between an HPO method and RL training. The interface is inspired by Gymnasium and allows an optimizer to specify both a hyperparameter configuration and a training budget at each optimization step. ARLBench then executes the corresponding RL training run and returns optimization objectives such as evaluation return or runtime, along with optional state information such as gradients or losses.

Furthermore, this interface supports dynamic HPO. Training state, including network parameters and optimizer state, can be checkpointed, restored, or duplicated. This enables population-based methods, adaptive schedules, and meta-gradient approaches to be evaluated in a unified and reproducible manner.

Selecting Representative Environments

Efficiency alone is not sufficient for meaningful benchmarking. ARLBench therefore addresses the question of which environments best represent the broader RL task space. To this end, we conduct a large-scale pre-study in which hundreds of hyperparameter configurations are evaluated across diverse environments spanning the Arcade Learning Environment (ALE), Classic Control, Box2D, Brax robotics, and XLand grid worlds.

Using these data, ARLBench applies a regression-based subset selection method that identifies small sets of environments whose performance rankings are highly predictive of average performance across all environments. The resulting subsets contain five environments for PPO and DQN and four for SAC.

Preserving HPO Landscape Properties

A key concern with environment reduction is whether it distorts the underlying HPO problem. ARLBench addresses this by comparing hyperparameter landscapes on the full environment sets and their corresponding subsets. Return distributions over randomly sampled configurations show that the subsets preserve both easy and adversarial regimes, including skewed distributions and sharp performance transitions. Future research needs to extend this work by exploring more sophisticated reward function transformations and investigating the trade-off between performance and stability in RL training.

Analyses of hyperparameter importance further confirm that the number and structure of influential hyperparameters remain largely unchanged. In addition, evaluations of several HPO optimizers—random search, population-based training, SMAC, and SMAC with Hyperband—show consistent relative performance rankings on subsets and full sets.

Results obtained with ARLBench reinforce the view that HPO in RL is fundamentally challenging. Hyperparameter landscapes are often highly irregular, and state-of-the-art optimizers do not consistently outperform random search. This behavior contrasts sharply with supervised learning and highlights the need for RL-specific HPO research rather than direct transfer of existing methods.

Outlook

ARLBench provides an efficient, flexible, and empirically grounded benchmark for hyperparameter optimization in reinforcement learning. By combining fast implementations, representative environment subsets, and a large public dataset, it enables rigorous, comparable evaluation of AutoRL methods at a fraction of the cost required previously. As such, ARLBench lays the groundwork for more systematic progress in automated reinforcement learning research.

ARLBench currently focuses on model-free RL algorithms and standard environment benchmarks. While this scope already covers a large portion of RL research, future extensions are planned to include richer algorithm variants, policy generalization, and surrogate modeling for dynamic HPO. Despite remaining computational costs, ARLBench prioritizes realism and flexibility over purely tabular benchmarks.

Full paper: arxiv.org/abs/2409.18827

GitHub: github.com/automl/arlbench

Dataset on Hugging Face: huggingface.co/datasets/autorl-org/arlbench

Making Reinforcement Learning Work Out of the Box: A 2024 Overview

2024 was a big year for RL in general and AutoRL specifically. Here we collect some of our highlights from last year.

AutoML for RL

AutoML for RL remains a key focus area. Normalization and Effective Learning Rates in RL by Lyle et al. introduces Normalize-and-Project, maintaining consistent learning rates in dynamic environments, particularly useful for long-term training. For hyperparameter tuning, Generalized PBT by Bai et al. refines Population-Based Training with pairwise learning to better balance exploration and exploitation, making RL algorithms more robust. Meanwhile, Adaptive Q-Network by Vincent et al. tackles on-the-fly hyperparameter tuning within RL training for improved real-world applicability. In EVA: Explained Variance Adaptation Paischer et al. optimize initialization strategies to speed up convergence in fine-tuning and improve performance across multiple domains.

Additional work has further examined the performance and consistency of RL hyperparameters in Small Batch Deep Reinforcement Learning and On the Consistency of Hyper-parameter Selection in Value-based Deep Reinforcement Learning, both by Obando-Ceron et al. Showing that this remains an open challenge limiting the applicability of RL algorithms.

A central challenge in RL is evaluating agent performance and selecting suitable algorithms. Estimating Agent Skill in Continuous Action Domains by Archibald et al. introduces a Bayesian framework to separate decision-making and execution skills for robust, fine-grained assessments. Complementing this, the tool presented in How to Choose a Reinforcement-Learning Algorithm by Bongratz et al. simplifies algorithm selection by providing an interactive guide that aligns methods with task requirements, further reducing barriers for practitioners. Of course the final step would be to do this automatically. Just in case you’re still looking for research inspiration!

Improving generalization and efficiency in experience replay is another central theme. Investigating the Interplay of Prioritized Replay and Generalization by Panahi et al. reveals trade-offs in prioritized sampling and proposes mitigations for instability, while Averaging n-step Returns Reduces Variance in RL by Daley et al. introduces compound returns that enhance sample efficiency without sacrificing performance.

Environment Design

Environment design and model-based methods are also undergoing rapid transformation. Genie by Bruce et al. and Genie 2 by Parker-Holder et al. propose generative models that create interactive environments from unlabelled videos, enabling agents to learn diverse behaviors. Bridging pretraining and deployment, PreLAR by Zhang et al. introduces action-conditioned pretraining for world models, improving sample efficiency. Meanwhile, DrEureka by Ma et al. harnesses LLMs to automate sim-to-real transfer, Action Abstractions for Amortized Sampling by Boussif et al. enhances exploration by abstracting high-level actions, Causality-Guided Self-Adaptive Representations Yang et al. adapts to unseen dynamics via causal learning, and CurricuLLM by Ryu et al. uses LLMs to design task curricula, accelerating the learning of complex robot skills.

Reward engineering and the offline RL front continue to see impactful innovations. Reward Centering by Naik et al. stabilizes training by normalizing reward distributions, enhancing consistency across tasks. Image-Based Dataset Representations for Predicting Learning Performance by Mateos-Meleri et al. leverages convolutional models to predict policy outcomes and guide dataset optimization.

LLMs and VLMs are exciting tools for reward design and policy learning. GenRL aligns VLMs with world models, enabling tasks to be specified through vision and language prompts, improving multi-task generalization. Similarly, Real-World Offline RL from Vision-Language Model Feedback by Venkataraman et al. automates reward labeling from suboptimal datasets, and Online Intrinsic Rewards from LLM Feedback by Zheng et al. synthesizes dense natural-language-based intrinsic rewards.

Beyond reward design, LLMs and VLMs reshape how RL agents interact with their environments. ROCKET-1 by Cai et al. employs visual-temporal context prompting to enhance spatial reasoning in open-world tasks, while LLaRA by Liao et al. uses conversation-style data augmentation to enrich vision-language policy learning for more sophisticated robotic behaviors. In Adaptive Planning with Generative Models under Uncertainty Jutras-Dubé et al. reduce replanning frequency without compromising performance, making generative models more practical.

Robotics

The field of AutoRL has made strides in robotics as well. Online Foundation Model Selection in Robotics by Li et al. balances cost and performance between open-source and closed-source models, while Learning Robot Soccer from Egocentric Vision by Tirumala et al. demonstrates an end-to-end approach for multi-agent policies trained purely from on-board vision, expanding RL’s applicability to complex robotic tasks without privileged information.

Foundation models have also made big strides in unsupervised methods for Robotics. Motivo by Tirinzoni et al. introduces a behavioral foundation model that leverages unsupervised reinforcement learning with forward-backward representations and conditional policy regularization to train a humanoid capable of zero-shot whole-body control across tasks like motion tracking, goal-reaching, and reward optimization. RL Zero: Zero-Shot Language to Behaviors without any Supervision by Sikchi et al. demonstrates zero-shot policy learning by grounding language commands into behaviors without any supervision. Similarly, LAPA (Latent Action Pretraining) by Ye et al. presents an unsupervised approach for pretraining robotic foundation models on web-scale data. By learning discrete latent actions and finetuning on a small set of labeled trajectories, LAPA enables generalization to novel tasks and unseen objects in both simulated and real-world environment.

Benchmarks and Frameworks

Benchmarks and frameworks are critical for driving the field forward. Kinetix by Matthews et al. introduces a procedurally generated physics-based task space for pretraining generalist RL agents, demonstrating the viability of large-scale pretraining. BALROG by Paglieri et al. provides a comprehensive benchmark to evaluate LLM and VLM decision-making in dynamic environments, while VisualAgentBench by Liu et al. focuses on testing multimodal agents in diverse, real-world-inspired scenarios. Lastly, frameworks like AutoRL X by Franke et al. bring RL to the web with a dynamic interface for visualizing and managing RL workflows, improving accessibility and collaboration. These benchmarks push the boundaries of evaluation and provide clear goals for future research.

First AutoRL Workshop

A particular highlight of the year was, of course, the first AutoRL workshop at ICML. It hosted a diverse set of great work (see all papers here), headed by our three best paper award winners Can Learned Optimization Make Reinforcement Learning Less Difficult? by Goldie et al., BOFormer: Learning to Solve Multi-Objective Bayesian Optimization via Non-Markovian RL by Hung et al. and Self-Exploring Language Models: Active Preference Elicitation for Online Alignment by Zhang et al.. Furthermore, we had great invited talks by Chelsea Finn, Roberta Raileanu, Pierluca D’Oro, Michael Dennis and Pablo Samuel Castro on everything from RL for robotics, in-context RL, AI-assisted agent design, learning interactive environments and the ALE as a benchmark for AutoRL. The videos are publicly available on the ICML workshop page.

Our Work

Of course, AutoRL.org itself has not been idle. Our own work this year includes HPO-RL-Bench by Shala et al. and ARLBench by Becktepe et al., two benchmarks for hyperparameter optimization based on tabular performance data and rapid execution respectively. Combined, they make benchmarking HPO for RL less computationally expensive. Dierkes et al. show the benefits of joint optimization for these critical components in Combining Automated Optimisation of Hyperparameters and Reward Shape. Works like Contextual World Models by Prasanna et al. and Inferring Behavior-Specific Context Improves ZeroShot Generalization in Reinforcement Learning by Camaret et al. expand our understanding of generalization and how context can improve it. One-shot World Models Using a Transformer Trained on a Synthetic Prior by Ferreira et al. shows that prior-fitted networks can be used as in-context world models while Hierarchical Transformers are Efficient Meta-Reinforcement Learners Shala et al. use in-context learning to adapt to unseen tasks. Towards Enhancing Predictive Representations using Relational Structure in Reinforcement Learning by Mohan and Lindauer improve representation learning methods in RL by incorporating relational inductive biases in self-predictive RL.

Overall, this has been a very successful year on many fronts: our understanding of how to train RL algorithms increased, integration of foundation models particularly for environment design has made incredible progress and AutoRL is moving forwards both in terms of methods and benchmarking. We’re excited to see what 2025 will bring!

TL;DR: Jointly optimising hyperparameters and reward shapes in RL outperforms separate tuning, delivering better performance and stability, even in complex environments like Robosuite Wipe. No hand-tuning required!

Optimising Smarter: Joint Optimisation of RL Hyperparameters and Reward Shape

RL has achieved remarkable advancements, but one basic challenge remains ever-present: tuning hyperparameters and reward functions. This is tedious yet critical and can determine the success or failure of an RL algorithm. Traditionally, these two components are optimised independently, and in benchmark environments, reward functions are often left untuned altogether. This raises the question: is the tuning of hyperparameters and reward shapes truly independent?

In our work, Combining Automated Optimisation of Hyperparameters and Reward Shape (Dierkes et al. 2024), we demonstrated that they actually are dependent and achieving optimal RL performance requires tuning both components together. This challenges the conventional approaches and calls for more unified optimisation frameworks.

Interdependency between Hyperparameters and Reward Parameters

To better understand the interdependence of hyperparameters and reward shapes, we conducted experiments training the PPO algorithm on LunarLander. Specifically, we performed an exhaustive grid search over various hyperparameter values and reward weights in the reward function, analysing how each pair influenced training performance.

These findings are visualised in Figure 1, depicting the complex relationships between hyperparameters and reward weights. For instance, the distance weight reveals a non-convex region where successful training parameters emerge and further, performance can vary across the whole search space. Meanwhile, the velocity weights can have sharp boundaries that clearly separate regions of success from complete failure, showing the balance required for effective tuning. While our analysis was limited to pairwise combinations, it is reasonable to expect even stronger dependencies in higher-dimensional parameter spaces.

Hyperparameters and reward weights are therefore intricately linked, and their joint optimisation is important for achieving the best possible performance in RL applications. Independent optimisation approaches fail to capture these interdependencies, leaving potential performance gains unexplored.

Joint Optimisation in Practice

Recognising these interdependencies, the next step was to design a practical optimisation approach. Our goal was to examine a method that could seamlessly integrate with existing black-box hyperparameter optimisation workflows while jointly optimising reward weights.

Figure 2 provides an overview of our optimisation loop. In this framework:

1. Both hyperparameters \(\theta\) of the RL algorithm as well as scale \(\alpha\) and weights \(w\) of the reward shape are treated as part of a unified search space.
2. The optimisation algorithm evaluates configurations by training the RL agent with hyperparameters \(\theta\) on the reward shaped environment \(\mathcal{M}_{\alpha, w}\). The it measures its external task performance \(\mathcal{O}_{\text{goal}}(\pi)\), a simple and sparse objective function to measure final policy performance (e.g. time of a successful landing in LunarLander).
3. Promising configurations are iteratively refined, enabling joint optimisation of both components.

We used the DEHB (Differential Evolution with Hyperband, Noor et al. 2021) algorithm for this purpose, a state-of-the-art multi-fidelity black-box optimiser. However, the approach is general and can be easily applied to other black-box optimisation methods (e.g. SMAC or Optuna).

In our experiments, we compared the joint optimisation of hyperparameters and reward weights (requiring no hand-tuning!) with two alternative approaches:

1. Optimising only the hyperparameters while using a competitive, pre-defined reward shape.
2. Optimising only the reward weights while keeping the hyperparameters fixed at competitive baseline values.

In the paper, we used the PPO and SAC algorithms across four environments: LunarLander, Google Brax Ant, Google Brax Humanoid, and Robosuite Wipe. Among these, Robosuite Wipe stands out as a complex, underexplored environment with a challenging reward structure, making it an excellent testbed for evaluating the generalisability of our approach.

We tested two optimisation strategies:

1. Single-objective optimisation, which focuses solely on maximising performance.
2. Multi-objective optimisation, which balances performance and stability by optimising the performance minus the policy’s standard deviation.

The SAC results, as summarised in Figure 3, demonstrated that joint optimisation consistently matched or outperformed the hand-tuned baselines. In simpler environments like LunarLander and Ant, joint optimisation reliably recovered baseline performance without requiring any manual tuning. In more challenging environments, such as Humanoid and Wipe, joint optimisation went a step further, yielding superior performance. For instance, in the Wipe environment, the method achieved a reward score close to the theoretical maximum of 141, showing its ability to handle complex reward structures and extract maximum potential from the task.

The multi-objective optimisation further enhanced the utility of joint optimisation by improving the stability of the learned policies. While maintaining a similar level of average performance, this approach significantly reduced the variance in performance across training runs in many cases. Such consistency is crucial for real-world applications, where reliable policy behaviour can be as important as achieving peak performance.

Finally, we found that joint optimisation comes with minimal computational overhead, meaning its benefits are essentially free!

Implications and Future Directions

Our findings have broad implications for RL research and applications:

1. Unified Optimisation: Hyperparameters and reward shapes should always be optimised jointly to account for their interdependencies.
2. Practicality: Our approach integrates easily into existing optimisation pipelines and scales effectively to new environments.
3. Generalisability: The method is robust across diverse benchmarks, including less-studied environments like Wipe, making it suitable for real-world applications.

Future research needs to extend this work by exploring more sophisticated reward function transformations and investigating the trade-off between performance and stability in RL training.

More information about this work and implementation details can be found in the paper’s GitHub repository here and the paper here.

We are excited to announce that the first AutoRL workshop was accepted at ICML to be held in Vienna this year. The workshop website contains more details. The workshops page for ICML is here and the AutoRL workshop page on the ICML website can be found here

Our goal is to bring together people working in different corners of this problem to share their expertise, methods and ideas. We are looking forward to cutting-edge and state-of-the-art submissions from the community.

We also have cool speakers and some discussions planned, so mark your calendars (26 or 27 July) and stay tuned!

Come discuss with us how to make RL work out of the box, be it via meta-learning, AutoML, LLMs or whatever else you can think of! 🚀 See you in Vienna!

2023 in AutoRL

GPT-generated image depicting AutoRL

What a year 2023 has been in machine learning! Beyond the obvious explosion in LLM capability, it was also a great year for all things AutoRL. From combining RL with LLMs through more efficient MetaRL and updates in an environment design to classic hyperparameter optimization, these are some of our top picks, plus a selection of our own AutoRL projects at the end of this post. So sit back and enjoy some of the most interesting AutoRL papers of 2023.

LLMs and more in RL

Language models have been on everyone’s mind this last year, and with good reason. So it’s only natural to try and use the capabilities of LLMs as common sense or reasoning models to support RL. LLMs have successfully been used as policies and curriculum generation mechanisms in Voyager [Wang et al., ArXiv], but also for finding expressive rewards signals in robotics (Eureka [Ma et al., ArXiv]) and exploration environments like MiniHack (Motif [Klissarov et al., ALOE@NeurIPS]).

Key components of the Voyager LLM-based agent. Image credit: Voyager paper

Agents [Zhou et al., ArXiv] is a user-friendly library that enables non-specialists to build state-of-the-art autonomous language agents without much coding. Generative Agents [Park et al., ArXiv] introduces architectural and interaction patterns that fuse LLMs with computational software agents that simulate believable human behaviour in a sandbox.

An overview of the Agents framework. Image credit: Agents paper

Robotics had some eye-catching works this year. [Brohan et al., ArXiv] trained Robotic Transformer 2 (RT-2), a vision-language-action (VLA) model based on vision-language models (VLMs), that can plan from both image and text commands. [Hansen et al., ArXiv] (TD-MPC2) train multi-task agents with world models that scale up with model size (up to 319M) on continuous control tasks with a single set of hyperparameters.

Meta-RL

The Meta-RL year started off strong with an excellent survey on the field in January [Beck et al., ArXiv]. It is a great resource on different Meta-RL paradigms and the work that has been done in this domain.

A big topic in Meta-RL this year was learning RL algorithms - unlike the approaches often used in the past, however, this year, in-context RL was the dominant idea for learning RL. From AdA, which showed rapid in-context adaptation to new task variations [Bauer et al., ICML], to learning an RL algorithm from supervised pre-training (DPT) [Lee et al., ArXiv] or adapting to new tasks [Chandra et al., FMDM@NeurIPS] in-context, this seems to be a promising future direction for meta-learned RL algorithms.

In-context adaption with AdA. Image credit: AdA paper

Environment Design

Generating challenging training environments and curricula has continued to be an important AutoRL topic in 2023. [Xie et al., ArXiv] automatically generate dense reward functions using LLMs. Similarly, [Zhang et al., CoRL 2023] learn to solve long-horizon tasks zero-shot by growing a library of learned skills with supervision by LLMs.

In curriculum generation, [Bajaj et al., ICAPS 2023] combine learning from demonstrations and curriculum learning to perform Automated Curriculum Learning from Demonstrations on sparse reward tasks. [Samvelyan et al., ICLR 2023] apply environment design to the multi-agent setting while keeping the abilities of other agents in mind in MAESTRO. Meanwhile, [Jackson et al., NeurIPS 2023] show that curricula are not only useful for learning policies but also in learning better RL algorithms.

Reward generation using Text2Reward. Image credit: Text2Reward paper

RL Hyperparameters

[Beukmann et al., NeurIPS 2023] generalise to new transition dynamics using a hypernetwork that generates the weights of an adapter module that conditions the behaviour of an agent on the environment context. [Lan et al., ArXiv] train an adaptive optimizer with inductive biases to generalise on learning rate control from toy tasks to complex Brax tasks. [Sabbioni et al., ArXiv] meta-learn setting the learning rate adaptively for sampled contextual test tasks.

[Yuan et al., ICML 2023] use a multi-armed bandit formulation, dubbed Automatic Intrinsic Reward Shaping, to select between different exploration strategies on MiniGrid, Procgen, and DeepMind Control Suite. This could be a promising strategy for dynamic Algorithm Selection in RL.

Intrinsic reward selection via UCB. Image credit: Automatic Instrinsic Reward Shaping paper

Benchmarks & Libraries

[Aitchison et al., ICML] Atari-5 selects a representative subset of 5 of 57 Atari games that can be used to predict median performance on all 57 games within 10% of the true value. Since a lot of (Auto)RL research is conducted on variations of the ALE, this could make such work much more efficient to run.

The same goes for the JAXification of RL – while benchmarks like XLand and JaxMARL as well as training libraries like PureJAX aren’t directly targeted at AutoRL, they certainly make AutoRL research much more accessible.

On the side of new AutoRL focus libraries, minimax and Syllabus both aim to make curriculum learning faster, easier to implement, and more comparable. They take slightly different approaches, though: while Syllabus offers a way of implementing the curriculum that works with different base algorithms (their examples include CleanRL and RLLib), minimax uses its own PPO implementation to keep evaluations directly comparable. Together they should cover most curriculum generation use cases, hopefully bringing a bit more standardisation to the field.

Speedups with PureJAX. Image credit: PureJAX repo

AutoRL.org Projects

Testing robustness against reward delay in DQNs on Atari. Image credit: MDP Playground paper

Of course, we weren’t idle throughout the year either. One important topic was benchmarking this year, as you can see in “MDP Playground: An Analysis and Debug Testbed for Reinforcement Learning” [Rajan et al., JAIR]. MDP Playground lets you define properties of MDPs, including delayed rewards, stochasticity, image representations, time unit, action range, and more to unit test your algorithms on toy MDPs or test its robustness on standard complex MDPs such as Atari and Mujoco using Gym wrappers.

Hand tuning compared to automatic HPO in RL. Image credit: HPs in RL paper

On the hyperparameter side of things, we go back to the basics in “Hyperparameters in Reinforcement Learning and How To Tune Them” [Eimer et al., ICML] for an investigation into how hard HPO for RL actually is and which existing tools work well for it. We show that automated HPO tools can give us similar results to grid searches for less than 10x the compute and propose best practices of how to incorporate HPO into experiments and reporting for more reproducible RL research. Further, “Gray-Box Gaussian Processes for Automated Reinforcement Learning” [Shala et al., ICLR] discusses how to fuse hyperparameter configurations, reward-curve information, as well as optimization budgets to perform efficient bayesian optimization specifically for AutoRL.

Improvement of Grey-Box GPs on PPO compared to PBT variations. Image credit: Grey-Box GP paper

Going beyond that, in “AutoRL Hyperparameter Landscapes” [Mohan et al., AutoML], we examine the relationship between hyperparameters, performance, and training time: how important are hyperparameter schedules in RL? We analyze this using landscape analysis across different types of hyperparameters (discounting, learning speed, and exploration) and agents (value-based and policy-based) and observe that the optimal regions of hyperparameters change as the agent trains, reaffirming the need for prioritising dynamic hyperparameter adaptations in RL.

Development of optimal learning rate and discount factor values for SAC over time. Image credit: RL Landscapes paper

On the topic of different kinds of learning objectives in Meta-RL and AutoRL, we unify diverse methodologies in Meta-RL and AutoRL under a design-pattern-oriented framework in [Mohan et al., 2023] highlighting the crucial role of structural integration in learning processes. Adding Structured RL into our research toolkit promises to enhance our understanding and capabilities in Meta-RL and Auto-RL significantly.

Overview of how to incorporate structure into RL. Image credit: Structure in RL paper

These are our best of 2023 – what have we missed, what are your highlights? Hopefully, 2024 can give us a similarly diverse range of exciting AutoRL research & software. Apart from the usual suspects, the RL conference with its first edition as well as COLLAs and the AutoML-Conf with their fourth and third editions respectively are likely venues for great AutoRL work in the coming year. So happy New Year and happy researching!