OpenAI Retro Cheat Sheet

In the previous post, we checked out OpenAI Gym: an open source library in Python used for reinforcement learning algorithms development to communicate between environments such as classic Atari video games.

In the this post, we check out OpenAI Retro: source code repository which lets you turn classic video games into Gym environments for reinforcement learning with classic Atari video games and many other Emulated Systems including: Nintendo GameBoy / GameBoy Color, the Sega MegaDrive and the Sega Master System.

Let's check it out!

Gym Retro
In May 2018, Open AI announce Gym Retro: a platform for reinforcement learning research on classic 8-bit and 16-bit video games. The release includes games from the Sega Genesis and Sega Master System, and Nintendo’s NES, SNES, and Game Boy consoles. The release also includes an Integration Tool that lets you create save states, find memory locations + design scenarios that reinforcement learning agents can solve.

Hello Gym Retro
Create new Hello Gym Retro type environment. Launch PyCharm | New Project | Name: HelloGymRetro [~/]

python -m venv .venv	pip install gym-retro
source .venv/bin/activate	pip install --upgrade pip

CODE

import retro  env = retro.make(game="Airstriker-Genesis")  observation = env.reset()    while True:     action = env.action_space.sample()     observation, reward, done, info = env.step(action)     env.render()     if done:         observation = env.reset()  env.close()

IPMORTANT
At the time of this writing you may experience the following errors. Here are errors encountered + solutions:

ERROR #1	AttributeError: module 'gym.utils.seeding' has no attribute 'hash_seed'
Solution	pip uninstall -y gym	pip install gym==0.25.2

ERROR #2	ModuleNotFoundError: No module named 'gym.envs.classic_control.rendering'
Solution	pip uninstall -y gym	pip install gym==0.21.0

ERROR #3	ERROR: Could not find a version that satisfies the requirement gym==0.21.0
Solution	pip install  git+https://github.com/openai/gym.git@9180d12e1b66e7e2a1a622614f787a6ec147ac40

Stable Retro
As per previous post, we learned that the lack of maintenance for OpenAI Gym prompted Gymnasium: which is an open source fork of the OpenAI Gym library maintained by The Farama Foundation. As Gym Retro built upon an unmaintained version of the OpenAI Gym library then this may explain errors encountered above J

Similarly, like Gym Retro built upon OpenAI Gym, Stable Retro is built upon Farama Foundation Gymnasium. Stable Retro lets you convert classic video games into Gymnasium environments for Reinforcement Learning!

Hello Stable Retro
Create Hello Stable Retro type environment. Launch PyCharm | New Project | Name: HelloStableRetro [~/]

python -m venv .venv	pip install stable-retro
source .venv/bin/activate	pip install --upgrade pip

CODE

import retro  env = retro.make(game="Airstriker-Genesis")  observation, info = env.reset()    while True:     action = env.action_space.sample()     observation, reward, terminated, truncated, info = env.step(action)     if terminated or truncated:         observation, info = env.reset()  env.close()

Retro Examples
Complete all Retro examples! Extend OpenAI Test repo. Launch PyCharm | New Project | RetroCheatSheet


IMPORTANT
If Python Interpreter not set and/or Virtual Environment not available then choose File | Settings... | Project: Python Interpreter | Add Interpreter | Add Local Interpreter. Launch terminal: source .venv/bin/activate.

Setup ROMs: copy Sega ROM files from the data directory to Virtual Environment via the copy_files.py script

mkdir data  mkdir examples

cd examples  python copy_files.py

Sega Master System
Launch retro games for the Sega Master System 8-bit video game console with .sms supported ROM types:

pip install stable-retro	pip install --upgrade pip
AlienSyndrome-Sms	RType-Sms
BladeEagle-Sms	Shinobi-Sms
DickTracy-Sms	SonicTheHedgehog-Sms
JamesPond2CodenameRoboCod-Sms	TransBot-Sms

Sega Genesis
Launch retro games for the Sega Genesis [MD] 16-bit video game console with .md supported ROM types:

pip install stable-retro	pip install --upgrade pip
AlteredBeast-Genesis	SonicTheHedgehog2-Genesis
Flicky-Genesis	SonicTheHedgehog3-Genesis
GhoulsnGhosts-Genesis	StreetsOfRage2-Genesis
RevengeOfShinobi-Genesis	Strider-Genesis

YouTube



Pull Request
Finally, setup Pull Request for future contributions. Navigate to stable-retro source repo and Create new fork

Git clone the stable-retro forked source code repository and setup the following Git username and user email

git config --global --get user.name  git config --global --get user.email

git config --global user.name "SteveProXNA"  git config --global user.email "steven_boland@hotmail.com"

Next, configure the stable-retro upstream destination code repository in order to complete any Pull Request:

git remote -v	git remote add upstream git@github.com:Farama-Foundation/stable-retro.git
origin  origin	git@github.com:SteveProXNA/stable-retro.git (fetch)  git@github.com:SteveProXNA/stable-retro.git (push)
upstream  upstream	git@github.com:Farama-Foundation/stable-retro.git (fetch)  git@github.com:Farama-Foundation/stable-retro.git (push)

Launch PyCharm | Open Stable-Retro | Create virtual environment | Pip install all dependencies as required:

pip install -e .

pip install --upgrade pip

IMPORTANT
If Python Interpreter not set and/or Virtual Environment not available then choose File | Settings... | Project: Python Interpreter | Add Interpreter | Add Local Interpreter. Launch terminal: source .venv/bin/activate.


Examples
Navigate to the examples sub-directory. Checkout trivial_random_agent.py which should resemble the Hello Stable-Retro program above. Airstriker-Genesis ROM is included by default. Copy your Sega games into the corresponding data directory stable | game directory. Update name to the game ROM name. Execute script!


Tests
Navigate to the test_python sub-diretory. Open any test file | right click | Run test or Debug test to step thru source code. Important: you may need to install the pytest package if you get usual ModuleNotFound error:

pip install pytest

pip install --upgrade pip

Game Integration
As per the official Gym Retro docs Integrating a game means taking a video game ROM file and setting it up as a reinforcement learning environment by defining 3 things: #1 A starting state, #2 A reward function, #3 A done condition. However, in order to integrate a new game we must have the Game Integration tool setup.

Integration UI
Follow instructions as per video to install all dependencies then build Game Integration UI tool and launch:

sudo apt-get update  sudo apt-get install capnproto  sudo apt-get install libcapnp-dev  sudo apt-get install libqt5opengl5-dev  sudo apt-get install qtbase5-dev

sudo apt-get install zlib1g-dev  sudo apt autoremove  cmake . -DBUILD_UI=ON -UPYLIB_DIRECTORY  make -j$(grep -c ^processor /proc/cpuinfo)  ./gym-retro-integration

Supported ROMs
ROM files contain the game itself. Each system has a unique file extension to denote the compatible system:

EXTN	SYSTEM	EXAMPLE
a26	Atari 2600	automaton.a26
gb	Nintendo Game Boy	dox-fire.gb
gba	Nintendo Game Boy Advance	Vantage-LostMarbles.gba
gg	Sega Game Gear	benryves-SegaTween.gg
md	Sega Genesis [Mega Drive]	Dekadence-Dekadrive.md
nes	Nintendo Entertainment System [Famicom]	Dr88-FamiconIntro.nes
pce	NEC TurboGrafx-16 [PC Engine]	chrisc-512_Colours.pce
sfc	Super Nintendo Entertainment System [Super Famicom]	Anthrox-SineDotDemo.sfc
sms	Sega Master System	blind-happy10.sms

Launch CLion | Open Stable-Retro. Navigate to src/ui folder. Open main.cpp. Click "Select CMakeLists.txt":


Navigate to CMakeLists.txt file | OK | Trust Project. Afterwards CLion will index the project and update all symbols. Click Edit configurations drop down list. Choose gym-retro-integration | Debug. Click Debug icon:


Finally, debug tests. Navigate to tests folder. Open data.cpp. Choose tests-data | Debug. Click Debug icon:


Process
Now that we have built the Game Integration tool setup either directly from the CLI and/or built from source it is now possible to integrate new games into Gym-Retro. This process involves: #1 Creating states for Gym Retro environment to load, #2 Inspect ROM memory for points of interest, #3 Find those address and types.

The following videos on OpenAI Game Integration tool Part 1, Part 2, Part 3 give some excellent examples on how to implement this process + create the following necessary files in order to Integrate brand new games:

Files

State file	Savestate from game beginning. Restart the environment puts agent at this point in the game
data.json	Defines the list of game-related variables that Python sees based on game memory addresses
scenario	JSON file defines the reward function and done condition using variables defined at data.json
metadata	JSON file defines starting state if no state specified by user and miscellaneous debugging info

Summary
To summarize, we have now setup Stable Retro environments for classic 8-bit video game systems like the Sega Master System and 16-bit video game systems like the Sega Genesis [MD] the next step would be to use reinforcement learning to train a bot to play classic emulated video games using this Gym Retro setup.

Consequently, Python NEAT is a popular library for training neural networks using genetic algorithms. There is an excellent Open AI and NEAT Tutorial series that integrates Sonic the Hedgehog as a testbed for applied reinforcement learning to provide a rich, rewarding, and visually complex environment that is well-suited for evolving neural networks. Ultimate goal: would be to use this as inspiration to integrate more Indie games!