Wandering aimlessly in continuous space

If reinforcement learning techniques were pet animals, epsilon-greedy exploration would certainly be the cockroach. It is undemanding, it can live just about everywhere, stupid beyond words, and its presence in your property is quite embarrassing. Fortunately, there are lots of prettier and more intelligent species in the pet shop of exploration methods. And most of these die when entering a continuous state space (or, almost equivalently, when function approximation appears).

Exploration in Puerto Rico

Puerto Rico is the #1 game on BoardGameGeek since ages. While it’s not my personal favorite, I can fully understand that – the game has a beautiful design (I mean the game mechanics, not the artwork). One thing in the design that stands out, being simple but brilliant, is the mechanism for diversifying the gameplay. And it is something very familiar from reinforcement learning.