TL;DR Act3D is introduced, a manipulation policy transformer that represents the robot’s workspace using a 3D feature field with adaptive resolutions dependent on the task at hand, and sets a new state-of-the-art in RL-Bench, an established manipulation benchmark.