OpenAI Retro Cheat Sheet

In the previous post, we checked out OpenAI Gym: an open source library in Python used for reinforcement learning algorithms development to communicate between environments such as classic Atari video games.

In the this post, we check out OpenAI Retro: source code repository which lets you turn classic video games into Gym environments for reinforcement learning with classic Atari video games and many other Emulated Systems including: Nintendo GameBoy / GameBoy Color, the Sega MegaDrive and the Sega Master System.

Let's check it out!

Gym Retro
In May 2018, Open AI announce Gym Retro: a platform for reinforcement learning research on classic 8-bit and 16-bit video games. The release includes games from the Sega Genesis and Sega Master System, and Nintendo’s NES, SNES, and Game Boy consoles. The release also includes an Integration Tool that lets you create save states, find memory locations + design scenarios that reinforcement learning agents can solve.

Hello Gym Retro
Create new Hello Gym Retro type environment. Launch PyCharm | New Project | Name: HelloGymRetro [~/]

python -m venv .venv	pip install gym-retro
source .venv/bin/activate	pip install --upgrade pip

CODE

import retro  env = retro.make(game="Airstriker-Genesis")  observation = env.reset()    while True:     action = env.action_space.sample()     observation, reward, done, info = env.step(action)     env.render()     if done:         observation = env.reset()  env.close()

IPMORTANT
At the time of this writing you may experience the following errors. Here are errors encountered + solutions:

ERROR #1	AttributeError: module 'gym.utils.seeding' has no attribute 'hash_seed'
Solution	pip uninstall -y gym	pip install gym==0.25.2

ERROR #2	ModuleNotFoundError: No module named 'gym.envs.classic_control.rendering'
Solution	pip uninstall -y gym	pip install gym==0.21.0

ERROR #3	ERROR: Could not find a version that satisfies the requirement gym==0.21.0
Solution	pip install  git+https://github.com/openai/gym.git@9180d12e1b66e7e2a1a622614f787a6ec147ac40

Stable Retro
As per previous post, we learned that the lack of maintenance for OpenAI Gym prompted Gymnasium: which is an open source fork of the OpenAI Gym library maintained by The Farama Foundation. As Gym Retro built upon an unmaintained version of the OpenAI Gym library then this may explain errors encountered above J

Similarly, like Gym Retro built upon OpenAI Gym, Stable Retro is built upon Farama Foundation Gymnasium. Stable Retro lets you convert classic video games into Gymnasium environments for Reinforcement Learning!

Hello Stable Retro
Create Hello Stable Retro type environment. Launch PyCharm | New Project | Name: HelloStableRetro [~/]

python -m venv .venv	pip install stable-retro
source .venv/bin/activate	pip install --upgrade pip

CODE

import retro  env = retro.make(game="Airstriker-Genesis")  observation, info = env.reset()    while True:     action = env.action_space.sample()     observation, reward, terminated, truncated, info = env.step(action)     if terminated or truncated:         observation, info = env.reset()  env.close()

Retro Examples
Complete all Retro examples! Extend OpenAI Test repo. Launch PyCharm | New Project | RetroCheatSheet


IMPORTANT
If Python Interpreter not set and/or Virtual Environment not available then choose File | Settings... | Project: Python Interpreter | Add Interpreter | Add Local Interpreter. Launch terminal: source .venv/bin/activate.

Setup ROMs: copy Sega ROM files from the data directory to Virtual Environment via the copy_files.py script

mkdir data  mkdir examples

cd examples  python copy_files.py

Sega Master System
Launch retro games for the Sega Master System 8-bit video game console with .sms supported ROM types:

pip install stable-retro	pip install --upgrade pip
AlienSyndrome-Sms	RType-Sms
BladeEagle-Sms	Shinobi-Sms
DickTracy-Sms	SonicTheHedgehog-Sms
JamesPond2CodenameRoboCod-Sms	TransBot-Sms

Sega Genesis
Launch retro games for the Sega Genesis [MD] 16-bit video game console with .md supported ROM types:

pip install stable-retro	pip install --upgrade pip
AlteredBeast-Genesis	SonicTheHedgehog2-Genesis
Flicky-Genesis	SonicTheHedgehog3-Genesis
GhoulsnGhosts-Genesis	StreetsOfRage2-Genesis
RevengeOfShinobi-Genesis	Strider-Genesis

YouTube



Pull Request
Finally, setup Pull Request for future contributions. Navigate to stable-retro source repo and Create new fork

Git clone the stable-retro forked source code repository and setup the following Git username and user email

git config --global --get user.name  git config --global --get user.email

git config --global user.name "SteveProXNA"  git config --global user.email "steven_boland@hotmail.com"

Next, configure the stable-retro upstream destination code repository in order to complete any Pull Request:

git remote -v	git remote add upstream git@github.com:Farama-Foundation/stable-retro.git
origin  origin	git@github.com:SteveProXNA/stable-retro.git (fetch)  git@github.com:SteveProXNA/stable-retro.git (push)
upstream  upstream	git@github.com:Farama-Foundation/stable-retro.git (fetch)  git@github.com:Farama-Foundation/stable-retro.git (push)

Launch PyCharm | Open Stable-Retro | Create virtual environment | Pip install all dependencies as required:

pip install -e .

pip install --upgrade pip

IMPORTANT
If Python Interpreter not set and/or Virtual Environment not available then choose File | Settings... | Project: Python Interpreter | Add Interpreter | Add Local Interpreter. Launch terminal: source .venv/bin/activate.


Examples
Navigate to the examples sub-directory. Checkout trivial_random_agent.py which should resemble the Hello Stable-Retro program above. Airstriker-Genesis ROM is included by default. Copy your Sega games into the corresponding data directory stable | game directory. Update name to the game ROM name. Execute script!


Tests
Navigate to the test_python sub-diretory. Open any test file | right click | Run test or Debug test to step thru source code. Important: you may need to install the pytest package if you get usual ModuleNotFound error:

pip install pytest

pip install --upgrade pip

Game Integration
As per the official Gym Retro docs Integrating a game means taking a video game ROM file and setting it up as a reinforcement learning environment by defining 3 things: #1 A starting state, #2 A reward function, #3 A done condition. However, in order to integrate a new game we must have the Game Integration tool setup.

Integration UI
Follow instructions as per video to install all dependencies then build Game Integration UI tool and launch:

sudo apt-get update  sudo apt-get install capnproto  sudo apt-get install libcapnp-dev  sudo apt-get install libqt5opengl5-dev  sudo apt-get install qtbase5-dev

sudo apt-get install zlib1g-dev  sudo apt autoremove  cmake . -DBUILD_UI=ON -UPYLIB_DIRECTORY  make -j$(grep -c ^processor /proc/cpuinfo)  ./gym-retro-integration

Supported ROMs
ROM files contain the game itself. Each system has a unique file extension to denote the compatible system:

EXTN	SYSTEM	EXAMPLE
a26	Atari 2600	automaton.a26
gb	Nintendo Game Boy	dox-fire.gb
gba	Nintendo Game Boy Advance	Vantage-LostMarbles.gba
gg	Sega Game Gear	benryves-SegaTween.gg
md	Sega Genesis [Mega Drive]	Dekadence-Dekadrive.md
nes	Nintendo Entertainment System [Famicom]	Dr88-FamiconIntro.nes
pce	NEC TurboGrafx-16 [PC Engine]	chrisc-512_Colours.pce
sfc	Super Nintendo Entertainment System [Super Famicom]	Anthrox-SineDotDemo.sfc
sms	Sega Master System	blind-happy10.sms

Launch CLion | Open Stable-Retro. Navigate to src/ui folder. Open main.cpp. Click "Select CMakeLists.txt":


Navigate to CMakeLists.txt file | OK | Trust Project. Afterwards CLion will index the project and update all symbols. Click Edit configurations drop down list. Choose gym-retro-integration | Debug. Click Debug icon:


Finally, debug tests. Navigate to tests folder. Open data.cpp. Choose tests-data | Debug. Click Debug icon:


Process
Now that we have built the Game Integration tool setup either directly from the CLI and/or built from source it is now possible to integrate new games into Gym-Retro. This process involves: #1 Creating states for Gym Retro environment to load, #2 Inspect ROM memory for points of interest, #3 Find those address and types.

The following videos on OpenAI Game Integration tool Part 1, Part 2, Part 3 give some excellent examples on how to implement this process + create the following necessary files in order to Integrate brand new games:

Files

State file	Savestate from game beginning. Restart the environment puts agent at this point in the game
data.json	Defines the list of game-related variables that Python sees based on game memory addresses
scenario	JSON file defines the reward function and done condition using variables defined at data.json
metadata	JSON file defines starting state if no state specified by user and miscellaneous debugging info

Summary
To summarize, we have now setup Stable Retro environments for classic 8-bit video game systems like the Sega Master System and 16-bit video game systems like the Sega Genesis [MD] the next step would be to use reinforcement learning to train a bot to play classic emulated video games using this Gym Retro setup.

Consequently, Python NEAT is a popular library for training neural networks using genetic algorithms. There is an excellent Open AI and NEAT Tutorial series that integrates Sonic the Hedgehog as a testbed for applied reinforcement learning to provide a rich, rewarding, and visually complex environment that is well-suited for evolving neural networks. Ultimate goal: would be to use this as inspiration to integrate more Indie games!

OpenAI Farma Cheat Sheet

OpenAI is an American Artificial Intelligence [AI] research organization founded Dec-2015 head-quartered in San Francisco. In April 2016, OpenAI released a public beta of "OpenAI Gym": its platform for reinforcement learning research. Reinforcement Learning [RL] is an interdisciplinary area of machine learning and optimal control concerned with how an agent takes action in environment in order to maximize cumulative rewards.

In 2021, the non-profit organization Farama Foundation took over Gym. They introduced new features into Gym and renamed the library to Gymnasium where the future maintenance of OpenAI Gym will take place.

Let's check it out!

Gym
OpenAI Gym is an open source Python library for developing reinforcement learning algorithms by providing standard API to communicate between algorithms and environments. Gym soon became widely adopted by the community for creating and training algorithms in various environments for AI Reinforcement Learning.

Gym provides a wide range of environments, including classic control problems, Atari games, and robotics. Gym also includes set of tools for monitoring and evaluating performance of reinforcement learning agents.

Hello Gym
Creating a Gym environment e.g. Cart Pole. Launch PyCharm | New Project | Name: HelloCartPoleGym [~/]

source .venv/bin/activate  pip install --upgrade pip

pip install gym  pip install gym[classic_control]

CODE

import gym  env = gym.make("CartPole-v1", render_mode="human")  observation, info = env.reset(seed=42)    for _ in range(1000):     action = env.action_space.sample()     observation, reward, terminated, truncated, info = env.step(action)     if terminated or truncated:         observation, info = env.reset()  env.close()

YouTube videos online also introduce OpenAI Gym but include Frozen Lake environment and Atari Breakout.


Gymnasium
The main problem with Gym was the lack of maintenance: OpenAI did not allocate substantial resources for the development of Gym since its inception seven years earlier and, by 2020, it simply was not maintained.

Gymnasium is a maintained fork of the OpenAI Gym library. The Gymnasium interface is simple, pythonic, and capable of representing general RL problems with a compatibility wrapper for old Gym environments:

Hello Gymnasium
Creating /Gymnasium environment e.g. Cart Pole. Launch PyCharm | New Project | HelloCartPoleGymnasium

source .venv/bin/activate  pip install --upgrade pip

pip install gymnasium  pip install gymnasium[classic-control]

CODE

import gymnasium as gym  env = gym.make("CartPole-v1", render_mode="human")  observation, info = env.reset(seed=42)    for _ in range(1000):     action = env.action_space.sample()     observation, reward, terminated, truncated, info = env.step(action)     if terminated or truncated:         observation, info = env.reset()  env.close()

Differences
While both Gym and Gymnasium are powerful tools for creating environments for reinforcement learning, it makes more sense to prefer Gymnasium going forward as technically future support for Gym there is none. Note: old import gym documentation is replaced with import gymnasium as gym convention going forward.


Farama Examples
Complete all Farama examples! Create OpenAI Test repo. Launch PyCharm | New Project | FarmaCheatSheet

IMPORTANT
If Python Interpreter not set and/or Virtual Environment not available then choose File | Settings... | Project: Python Interpreter | Add Interpreter | Add Local Interpreter. Launch terminal: source .venv/bin/activate.

Copy the official Gymnasium requirements.txt from the Farama Foundation GitHub repository docs directory:

pip install -r requirements.txt

pip install --upgrade pip

Classic Control
There are five Classic Control environments which are stochastic in terms of initial state within given range.

pip install gymnasium
Acrobot-v1	CartPole-v1	MountainCar-v0	Pendulum-v1

Toy Text
Toy Text environments are designed to be extremely simple, with small discrete state and action spaces, and hence easy to learn. They are suitable for debugging implementations of reinforcement learning algorithms.

pip install gymnasium
Blackjack-v1	CliffWalking-v0	FrozenLake-v1	Taxi-v3

Box2D
Box2D environments involve toy games based around box2d physics control and PyGame-based rendering.

pip install gymnasium		pip install Box2D
BipedalWalker-v3	CarRacing-v2	LunarLander-v2

MuJoCo
MuJoCo [Multi-Joint dynamics with Contact] is physics engine for facilitating research and development in robotics, biomechanics, graphics and animation, also areas where fast and accurate simulation is needed.

pip install gymnasium		pip install mujoco==2.3.0
Ant-v4	Humanoid-v4	InvertedPendulum-v4	Swimmer-v4
HalfCheetah-v4	HumanoidStandup-v4	Pusher-v4	Walker2d-v4
Hopper-v4	InvertedDoublePendulum-v4	Reacher-v4

Atari
Atari environments are simulated via the Arcade Learning Environment (ALE) through the Stella emulator.

pip install gymnasium[atari]		pip install gymnasium[accept-rom-license]
Adventure-v5	Defender-v5	KeystoneKapers-v5	Seaquest-v5
AirRaid-v5	DemonAttack-v5	KingKong-v5	SirLancelot-v5
Alien-v5	DonkeyKong-v5	Klax-v5	Skiing-v5
Amidar-v5	DoubleDunk-v5	Koolaid-v5	Solaris-v5
Assault-v5	Earthworld-v5	Krull-v5	SpaceInvaders-v5
Asterix-v5	ElevatorAction-v5	KungFuMaster-v5	SpaceWar-v5
Asteroids-v5	Enduro-v5	LaserGates-v5	StarGunner-v5
Atlantis-v5	Entombed-v5	LostLuggage-v5	Superman-v5
Atlantis2-v5	Et-v5	MarioBros-v5	Surround-v5
Backgammon-v5	FishingDerby-v5	MiniatureGolf-v5	Tennis-v5
BankHeist-v5	FlagCapture-v5	MontezumaRevenge-v5	Tetris-v5
BasicMath-v5	Freeway-v5	MrDo-v5	TicTacToe3D-v5
BattleZone-v5	Frogger-v5	MsPacman-v5	TimePilot-v5
BeamRider-v5	Frostbite-v5	NameThisGame-v5	Trondead-v5
Berzerk-v5	Galaxian-v5	Othello-v5	Turmoil-v5
Blackjack-v5	Gopher-v5	Pacman-v5	Tutankham-v5
Bowling-v5	Gravitar-v5	Phoenix-v5	UpNDown-v5
Boxing-v5	Hangman-v5	Pitfall-v5	Venture-v5
Breakout-v5	HauntedHouse-v5	Pitfall2-v5	VideoCheckers-v5
Carnival-v5	Hero-v5	Pong-v5	VideoChess-v5
Casino-v5	HumanCannonball-v5	Pooyan-v5	VideoCube-v5
Centipede-v5	IceHockey-v5	PrivateEye-v5	VideoPinball-v5
ChopperCommand-v5	Jamesbond-v5	Qbert-v5	WizardOfWor-v5
CrazyClimber-v5	JourneyEscape-v5	Riverraid-v5	WordZapper-v5
Crossbow-v5	Kaboom-v5	RoadRunner-v5	YarsRevenge-v5
Darkchambers-v5	Kangaroo-v5	Robotank-v5	Zaxxon-v5

Pull Request
Finally, submit Pull Request: upload Farma examples. Navigate to source Gymnasium repo | Create new fork

Git clone the forked Gymnasium source code repository and setup the following Git username and useremail

git config --global --get user.name  git config --global --get user.email

git config --global user.name "SteveProXNA"  git config --global user.email "steven_boland@hotmail.com"

Next, configure the Gymnasium upstream destination code repository in order to complete the Pull Request:

git remote -v	git remote add upstream git@github.com:Farama-Foundation/Gymnasium.git
origin  origin	git@github.com:SteveProXNA/Gymnasium.git (fetch)  git@github.com:SteveProXNA/Gymnasium.git (push)
upstream  upstream	git@github.com:Farama-Foundation/Gymnasium.git (fetch)  git@github.com:Farama-Foundation/Gymnasium.git (push)

Launch PyCharm | Open Gymnasium | Create virtual environment | Install requirements as pre-requisites:

pip install -r docs/requirements.txt

pip install --upgrade pip

Uncomment game from ClassicControl sub-directory files.txt or ToyText files.txt. Open main.py. Run script:

Install Box2D and MuJoCo dependencies. Uncomment game from Box2D files.txt or MuJoCo. Run main.py:

Box2D  pip install Box2D

MuJoCo  pip install mujoco==2.3.0

Finally, in order to integrate Atari game examples I found this only worked cut from tag v0.29.1 [81b87ef]:

git fetch --tags --prune --prune-tags  git fetch --all

git tag | grep v0.29.1  git checkout tags/v0.29.1

It seems after tag v0.29.1 any Atari game may throw the following: gymnasium.error.NamespaceNotFound: Namespace ALE not found. Have you installed the proper package for ALE?. Resume Atari tag v0.29.1 setup:

pip install gymnasium[atari]

pip install gymnasium[accept-rom-license]

Retrospectives
In Jan-2020 Retrospective XI, we learned about an interesting development connecting Artificial Intelligence and retro video games leveraging Reinforcement Learning. In Jan-2021 Retrospective XII, we then discussed video games as an interesting field of study and retro game development of Pacman and Doom clones as an excellent avenue to pursue with regards to future applications in software with Artificial Intelligence and RL.

Consequently, in the previous post, we disovered how to debug step thru cloned versions of Doom to better understand the original source code: an important pre-requisite to extend knowledge for further research in reinforcement learning, in particular. For example, ViZDoom allows developing AI bots that play Doom using visual data. In 2022, ViZDoom joined Farama Foundation featuring new Gymnasium environment wrappers.


Summary
To summarize, we have now setup Gymnasium environments for Classic Control, Toy Text, Box 2D, MuJo Co and Atari. NB: all Atari environments leverage the Atari Arcade Learning Environment [ALE] emulator Stella.

In fact, reverting back to OpenAI which hosts the Retro Git source code repository, also including Emulated Systems like Atari. However, Retro also includes many other Emulated Systems we have developed for in the past including Nintendo GameBoy / GameBoy Color, Sega MegaDrive and or course the Sega Master System! This will be the topic of the next post.