In 2021, the non-profit organization Farama Foundation took over Gym. They introduced new features into Gym and renamed the library to Gymnasium where the future maintenance of OpenAI Gym will take place.
Let's check it out!
Gym
OpenAI Gym is an open source Python library for developing reinforcement learning algorithms by providing standard API to communicate between algorithms and environments. Gym soon became widely adopted by the community for creating and training algorithms in various environments for AI Reinforcement Learning.
Gym provides a wide range of environments, including classic control problems, Atari games, and robotics. Gym also includes set of tools for monitoring and evaluating performance of reinforcement learning agents.
Hello Gym
Creating a Gym environment e.g. Cart Pole. Launch PyCharm | New Project | Name: HelloCartPoleGym [~/]
|
source .venv/bin/activate pip install --upgrade pip |
pip install gym pip install gym[classic_control] |
CODE
|
import gym env = gym.make("CartPole-v1", render_mode="human") observation, info = env.reset(seed=42) for _ in range(1000): action = env.action_space.sample() observation, reward, terminated, truncated, info = env.step(action) if terminated or truncated: observation, info = env.reset() env.close() |
Gymnasium
The main problem with Gym was the lack of maintenance: OpenAI did not allocate substantial resources for the development of Gym since its inception seven years earlier and, by 2020, it simply was not maintained.
Gymnasium is a maintained fork of the OpenAI Gym library. The Gymnasium interface is simple, pythonic, and capable of representing general RL problems with a compatibility wrapper for old Gym environments:
Hello Gymnasium
Creating /Gymnasium environment e.g. Cart Pole. Launch PyCharm | New Project | HelloCartPoleGymnasium
|
source .venv/bin/activate pip install --upgrade pip |
pip install gymnasium pip install gymnasium[classic-control] |
CODE
|
import gymnasium as gym env = gym.make("CartPole-v1", render_mode="human") observation, info = env.reset(seed=42) for _ in range(1000): action = env.action_space.sample() observation, reward, terminated, truncated, info = env.step(action) if terminated or truncated: observation, info = env.reset() env.close() |
Differences
While both Gym and Gymnasium are powerful tools for creating environments for reinforcement learning, it makes more sense to prefer Gymnasium going forward as technically future support for Gym there is none. Note: old import gym documentation is replaced with import gymnasium as gym convention going forward.
Farama Examples
Complete all Farama examples! Create OpenAI Test repo. Launch PyCharm | New Project | FarmaCheatSheet
IMPORTANT
If Python Interpreter not set and/or Virtual Environment not available then choose File | Settings... | Project: Python Interpreter | Add Interpreter | Add Local Interpreter. Launch terminal: source .venv/bin/activate.
Copy the official Gymnasium requirements.txt from the Farama Foundation GitHub repository docs directory:
|
pip install -r requirements.txt |
pip install --upgrade pip |
Classic Control
There are five Classic Control environments which are stochastic in terms of initial state within given range.
| pip install gymnasium | |||
| Acrobot-v1 | CartPole-v1 | MountainCar-v0 | Pendulum-v1 |
Toy Text
Toy Text environments are designed to be extremely simple, with small discrete state and action spaces, and hence easy to learn. They are suitable for debugging implementations of reinforcement learning algorithms.
| pip install gymnasium | |||
| Blackjack-v1 | CliffWalking-v0 | FrozenLake-v1 | Taxi-v3 |
Box2D
Box2D environments involve toy games based around box2d physics control and PyGame-based rendering.
| pip install gymnasium | pip install Box2D | ||
| BipedalWalker-v3 | CarRacing-v2 | LunarLander-v2 | |
MuJoCo
MuJoCo [Multi-Joint dynamics with Contact] is physics engine for facilitating research and development in robotics, biomechanics, graphics and animation, also areas where fast and accurate simulation is needed.
| pip install gymnasium | pip install mujoco==2.3.0 | ||
| Ant-v4 | Humanoid-v4 | InvertedPendulum-v4 | Swimmer-v4 |
| HalfCheetah-v4 | HumanoidStandup-v4 | Pusher-v4 | Walker2d-v4 |
| Hopper-v4 | InvertedDoublePendulum-v4 | Reacher-v4 | |
Atari
Atari environments are simulated via the Arcade Learning Environment (ALE) through the Stella emulator.
Pull Request
Finally, submit Pull Request: upload Farma examples. Navigate to source Gymnasium repo | Create new fork
Git clone the forked Gymnasium source code repository and setup the following Git username and useremail
|
git config --global --get user.name git config --global --get user.email |
git config --global user.name "SteveProXNA" git config --global user.email "steven_boland@hotmail.com" |
Next, configure the Gymnasium upstream destination code repository in order to complete the Pull Request:
|
git remote -v |
git remote add upstream git@github.com:Farama-Foundation/Gymnasium.git |
|
origin origin |
git@github.com:SteveProXNA/Gymnasium.git (fetch) git@github.com:SteveProXNA/Gymnasium.git (push) |
|
upstream upstream |
git@github.com:Farama-Foundation/Gymnasium.git (fetch) git@github.com:Farama-Foundation/Gymnasium.git (push) |
Launch PyCharm | Open Gymnasium | Create virtual environment | Install requirements as pre-requisites:
|
pip install -r docs/requirements.txt |
pip install --upgrade pip |
Uncomment game from ClassicControl sub-directory files.txt or ToyText files.txt. Open main.py. Run script:
|
|
Install Box2D and MuJoCo dependencies. Uncomment game from Box2D files.txt or MuJoCo. Run main.py:
| Box2D pip install Box2D |
MuJoCo pip install mujoco==2.3.0 |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
Finally, in order to integrate Atari game examples I found this only worked cut from tag v0.29.1 [81b87ef]:
| git fetch --tags --prune --prune-tags git fetch --all |
git tag | grep v0.29.1 git checkout tags/v0.29.1 |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| pip install gymnasium[atari] | pip install gymnasium[accept-rom-license] |
Retrospectives
In Jan-2020 Retrospective XI, we learned about an interesting development connecting Artificial Intelligence and retro video games leveraging Reinforcement Learning. In Jan-2021 Retrospective XII, we then discussed video games as an interesting field of study and retro game development of Pacman and Doom clones as an excellent avenue to pursue with regards to future applications in software with Artificial Intelligence and RL.
Consequently, in the previous post, we disovered how to debug step thru cloned versions of Doom to better understand the original source code: an important pre-requisite to extend knowledge for further research in reinforcement learning, in particular. For example, ViZDoom allows developing AI bots that play Doom using visual data. In 2022, ViZDoom joined Farama Foundation featuring new Gymnasium environment wrappers.
Summary
To summarize, we have now setup Gymnasium environments for Classic Control, Toy Text, Box 2D, MuJo Co and Atari. NB: all Atari environments leverage the Atari Arcade Learning Environment [ALE] emulator Stella.
In fact, reverting back to OpenAI which hosts the Retro Git source code repository, also including Emulated Systems like Atari. However, Retro also includes many other Emulated Systems we have developed for in the past including Nintendo GameBoy / GameBoy Color, Sega MegaDrive and or course the Sega Master System! This will be the topic of the next post.
Saturday, August 31, 2024
Doom Code WalkThru
Doom is a First-Person Shooter developed and published by id Software in December 1993. Doom is the first franchise installment and one of the most widely ported video games since the code was released Dec-1997.
Let's check it out!
Ports
In order to better understand the Doom source code we would like to debug step thru both the Doom port PrBoom on Linux and the Doom port DoomRetro on Windows written in C using modern-day software tools:
WAD Files WAD is the acronym for "Where's All the Data?" which is the file format used by Doom and all Doom engine-based games for storing data. A WAD file consists of a header, a directory and the data lumps that make up the resources stored within the file. A WAD file can be one of two types: an Internal WAD or a Patch WAD:
According to doomwiki there is List of notable WADs available to download beginning with Doom1.wad etc.
Doom Bible The Doom Bible is the original design document which includes the specifications for the WAD file format structure that initiated an incredible modding scene in which fans created their own WAD files containing new levels commonplace for first-person shooter games! Now let's debug step thru Doom C source code.
PrBoom
PrBoom is a cross-platform version of the classic 3D first-person shooter Doom from id Software. PrBoom has since been ported to Linux and many other platforms. PrBoom requires the original IWAD data files.
Follow this YouTube video to setup PrBoom in VS Code and build and debug step through the source code! Launch Terminal. Install the following Linux packages as PrBoom pre-requisites, if not currently installed:
Download prboom-plus-2.5.1.3 from sourceforge.net. That is the link that seemed to work OK for me. Launch Terminal. Execute following commands to extract source code, configure, build and run PrBoom:
If you get the following configure: error: *** You must have the SDL libraries installed before you can compile prboom *** See http://prboom.sourceforge.net/linux.html then install the following packages:
Copy all files from the DoomCodeWalkThru Files directory to the local hidden directory ~/.prboom-plus
Launch VS Code. Create build shell script, hidden .vscode directory with tasks.json and launch.json files
Open i_main.c. Set breakpoints. Press Ctrl+Shift+B to build and F5 to debug step through source code!
IMPORTANT - the first character of IWAD file must be I not P otherwise may get an "IWAD not found" error.
CRITICAL - specify the arguments -iwad and WAD file e.g. Doom1.wad otherwise you will encounter error:
Keys
Note: F11 does Gamma correction. Also, re-copy Prboom-plus.cfg and Prboom-plus.wad to reset HUD.
The YouTube video shows some examples how to hack the d_englsh.h to make custom text and sounds!
DoomRetro
Doom Retro is the classic, refined Doom source port for Windows PC. Written in C, Doom Retro source code is maintained in doomretro Git repository and regularly compiled for both 32 and 64-bit using Visual Studio.
Launch Visual Studio 2022. Open doomretro.sln. Choose Debug | 32-bit. Right click Solution and Rebuild. Ensure the Platform Toolset is Visual Studio 2022 (v143) in the project Properties | Configuration Properties.
Open doomretro.c. Set breakpoints. Press Ctrl+Shift+B to build and F5 to debug step through source code! The function D_OpenWADLauncher() will prompt WAD file selection. Choose a WAD file from Files directory.
Keys
Commands
Press the back tick [`] key prompts the Console Commands dialog box. Here are some useful commands:
Documentation The Game Engine Black Book: DOOM book is excellent reference material which goes into great depth about the development process of Doom in 1993. Masters of Doom documents the history of id Software and goes into great depth about development of many id Software franchises leading up-and-including Doom success.
Summary To summarize, we have now setup Doom ports on both Linux and Windows computers to debug step through and better understand the source code. While this has been a fun and nostalgic exercise, many believe that this is crucial that Software Developers should review influential source code: rather than repeat the same past mistakes, we should study great works that preceded us and learn from and build upon their lessons.
Another motivation to study classic video game source code is their relevance to modern day AI topics such as Reinforcement Learning. For example: VizDoom allows developing AI bots to play Doom using only visual information. Hosted by The Farama-Foundation VizDoom allows creating environments based on the Doom engine. The library provides Python API wrappers for the OpenAI Gym / Gymnasium interface and provides an excellent opportunity to research machine visual learning and deep reinforcement learning in the future!
Let's check it out!
Ports
In order to better understand the Doom source code we would like to debug step thru both the Doom port PrBoom on Linux and the Doom port DoomRetro on Windows written in C using modern-day software tools:
| Linux | PrBoom | Cross-platform version of Doom ported to Linux |
| Windows | DoomRetro | Classic refined Doom source port for Windows PC |
WAD Files WAD is the acronym for "Where's All the Data?" which is the file format used by Doom and all Doom engine-based games for storing data. A WAD file consists of a header, a directory and the data lumps that make up the resources stored within the file. A WAD file can be one of two types: an Internal WAD or a Patch WAD:
| IWAD | Internal WAD | Core WAD file loaded by engine that provides all data required to run game |
| PWAD | Patch WAD | Optional file that replaces IWAD data and provides additional data to engine |
Doom Bible The Doom Bible is the original design document which includes the specifications for the WAD file format structure that initiated an incredible modding scene in which fans created their own WAD files containing new levels commonplace for first-person shooter games! Now let's debug step thru Doom C source code.
PrBoom
PrBoom is a cross-platform version of the classic 3D first-person shooter Doom from id Software. PrBoom has since been ported to Linux and many other platforms. PrBoom requires the original IWAD data files.
Follow this YouTube video to setup PrBoom in VS Code and build and debug step through the source code! Launch Terminal. Install the following Linux packages as PrBoom pre-requisites, if not currently installed:
|
sudo apt-get install libghc-opengl-dev sudo apt-get install libghc-sdl-dev |
Download prboom-plus-2.5.1.3 from sourceforge.net. That is the link that seemed to work OK for me. Launch Terminal. Execute following commands to extract source code, configure, build and run PrBoom:
|
tar -xvf prboom-plus-2.5.1.3.tar.gz cd ~/prboom-plus chmod +x configure ./configure make chmod +x src/prboom-plus |
If you get the following configure: error: *** You must have the SDL libraries installed before you can compile prboom *** See http://prboom.sourceforge.net/linux.html then install the following packages:
|
sudo apt-get install libsdl1.2-dev sudo apt-get install libsdl-mixer1.2-dev sudo apt-get install libsdl-net1.2-dev |
Copy all files from the DoomCodeWalkThru Files directory to the local hidden directory ~/.prboom-plus
|
cp ~/GitHub/SteveProXNA/Doom/Files/* ~/.prboom-plus |
Launch VS Code. Create build shell script, hidden .vscode directory with tasks.json and launch.json files
| tasks.json | launch.json |
{
"version": "2.0.0",
"label": "build",
"type": "shell",
"linux": {
"command": "./build.sh"
},
}
|
{
"version": "0.2.0",
"configurations": [
{
"name": "(gdb) Launch",
"type": "cppdbg",
"request": "launch",
"program": "${workspaceFolder}/src/prboom-plus",
"args": [
"-iwad",
"Doom1.wad"
],
"stopAtEntry": false,
"cwd": "${workspaceFolder}",
"externalConsole": false,
"MIMode": "gdb",
}
]
}
|
Open i_main.c. Set breakpoints. Press Ctrl+Shift+B to build and F5 to debug step through source code!
IMPORTANT - the first character of IWAD file must be I not P otherwise may get an "IWAD not found" error.
CRITICAL - specify the arguments -iwad and WAD file e.g. Doom1.wad otherwise you will encounter error:
M_LoadDefaults: Load system defaults. default file: ~/.prboom-plus/prboom-plus.cfg found ~/.prboom-plus/prboom-plus.wad prboom-plus v2.5.1.3 (http://prboom-plus.sourceforge.net/) I_SetAffinityMask: manual affinity mask is 1 IdentifyVersion: IWAD not found |
Keys
|
|
The YouTube video shows some examples how to hack the d_englsh.h to make custom text and sounds!
|
|
|
|
DoomRetro
Doom Retro is the classic, refined Doom source port for Windows PC. Written in C, Doom Retro source code is maintained in doomretro Git repository and regularly compiled for both 32 and 64-bit using Visual Studio.
Launch Visual Studio 2022. Open doomretro.sln. Choose Debug | 32-bit. Right click Solution and Rebuild. Ensure the Platform Toolset is Visual Studio 2022 (v143) in the project Properties | Configuration Properties.
Open doomretro.c. Set breakpoints. Press Ctrl+Shift+B to build and F5 to debug step through source code! The function D_OpenWADLauncher() will prompt WAD file selection. Choose a WAD file from Files directory.
Keys
|
|
Commands
Press the back tick [`] key prompts the Console Commands dialog box. Here are some useful commands:
|
|
|
|
|
|
|
Documentation The Game Engine Black Book: DOOM book is excellent reference material which goes into great depth about the development process of Doom in 1993. Masters of Doom documents the history of id Software and goes into great depth about development of many id Software franchises leading up-and-including Doom success.
Summary To summarize, we have now setup Doom ports on both Linux and Windows computers to debug step through and better understand the source code. While this has been a fun and nostalgic exercise, many believe that this is crucial that Software Developers should review influential source code: rather than repeat the same past mistakes, we should study great works that preceded us and learn from and build upon their lessons.
Another motivation to study classic video game source code is their relevance to modern day AI topics such as Reinforcement Learning. For example: VizDoom allows developing AI bots to play Doom using only visual information. Hosted by The Farama-Foundation VizDoom allows creating environments based on the Doom engine. The library provides Python API wrappers for the OpenAI Gym / Gymnasium interface and provides an excellent opportunity to research machine visual learning and deep reinforcement learning in the future!
Thursday, July 4, 2024
GitLab Cheat Sheet
In 2016, we checked out GitHub Cheat Sheet: a distributed version control system for Open Source projects. Now we will explore differences in GitLab to streamline collaborative workflow and leverage CI/CD pipelines.
Let's check it out!
GitLab
As per previous post assumes an account setup on github.com. Create an account on gitlab.com if you have not already done so. As GitLab also uses underlying git we setup the following on localhost for development:
SSH Keys
An SSH key is an access credential for the SSH network protocol. Git supports both RSA and ED25519 keys. Launch Terminal. Issue the following commands to generate new SSH key and adding SSH key to ssh-agent:
IMPORTANT:
If xclip is not installed then issue 2x commands: sudo apt-get update and sudo apt-get install xclip. Finally, navigate to github.com Settings | SSH and GPG keys | New SSH key | Paste contents from id_rsa.pub here.
Personal Access Token
If you favor HTTPS then Personal Access Tokens offer security benefits over password-based authentication. Launch gitlab.com. Navigate to Edit Profile | Access Tokens | Add new token. Create personal access token.
After new Personal Access Token is created upload into source control software for example Source Tree on Windows and Mac OS/X and JetBrains products on Linux that integrate Git like PyCharm for Python etc etc.
Source Tree
Launch Source Tree | Settings | Accounts Add... | Enter the following details | Paste personal access token:
JetBrains IDE
Launch PyCharm | File | Settings... | Version Control | GitLab. Click "+" button | Enter GitLab token | Log In:
Example
In 2021, we coded an example from scratch as full end-to-end Web API demo on local host, in Docker and Kubernetes. Let's replicate with Python Flask API but this time include Gitlab CI/CD deployment pipelines.
Navigate to gitlab.com. Create blank project. Launch PyCharm. Create the new project gitlabcheatsheet.
Enter main.py code. Install packages. Press F5 to debug. Launch Terminal | curl http://localhost:8080
Code Dockefile. Launch Terminal | Enter the following Docker commands | curl http://localhost:8080
Create local KinD cluster. Enter all the following Kubernetes Deployment and Service YAML configuration:
Finally test: Launch Terminal | Enter the following Kubernetes commands | curl http://localhost:8080
Cluster
Extend example but this time include Gitlab CI/CD deployment pipelines. Ensure you have all Pre-Requisites:
Azure AKS
Launch Terminal | Login to Azure portal. Enter commands to extract Service Principal and generate SSH key:
Export the following environment variables. Note: use the Service Principal information for first 3x ENV VARs
Create the Azure resource group then finally create the AKS cluster. Note: this process can take few minutes
Finally, execute following commands to download + export KUBECONFIG file on localhost and setup context
Pipeline
Follow this tutorial to create and run your first GitLab CI/CD pipeline. Create .gitlab-ci.yml file at root the of the project directory. Refactor all YAML configuration in files beneath the cicd directory. Automate CI/CD to the Kubernetes cluster using helm charts. This way, multiple environments can be targeted and customized.
Variables
Navigate to gitlab.com | Settings | CI/CD | Variables. Enter all GitLab Variables here e.g. Docker registry host and username. Secure sensitive information like Docker password using Base64. Add KUBECONFIG:
Deployment
Trigger CI/CD pipeline. Launch Terminal | Repeat kubectl port-forward. Test: curl http://localhost:8080
GitFlow
Gitflow is an alternative Git branching model that involves the use of feature branches and multiple primary branches. Here is practical guide to implement GitFlow using GitLab with the corresponding git commands.
Launch gitlab.com. Create new project. Default branch will be main. Git clone repository to localhost. For completeness cut initial Tag 0.1. In SourceTree setup GitFlow: Repository | Git-flow | Initialize Repository:
In gitlab.com set develop as default branch: GitLab | Setttings | Repository | Branch defaults | develop.
IMPORTANT - you can compare two branches [source vs. target] anytime issuing the following commands:
Summary
To summarize, now that we have discussed the differences between GitHub and GitLab, we have extended our previous end-to-end Web API demo on local host, in Docker and Kubernetes but now leveraged GitLab CI/CD pipelines to automate deployments to the cloud. Our next steps would be more GitFlow integration: Replicate deployments across multiple environments and complete the full Software Development LifeCyle!
Let's check it out!
GitLab
As per previous post assumes an account setup on github.com. Create an account on gitlab.com if you have not already done so. As GitLab also uses underlying git we setup the following on localhost for development:
SSH Keys
An SSH key is an access credential for the SSH network protocol. Git supports both RSA and ED25519 keys. Launch Terminal. Issue the following commands to generate new SSH key and adding SSH key to ssh-agent:
| RSA | ED25519 |
| cd ~/.ssh | cd ~/.ssh |
| ssh-keygen -t rsa -b 4096 -C "steven_boland@hotmail.com" | ssh-keygen -t ed25519 -C <email> |
| Passphrase <redacted> | Passphrase <redacted> |
| eval "$(ssh-agent -s)" | eval "$(ssh-agent -s)" |
| ssh-add ~/.ssh/id_rsa | ssh-add ~/.ssh/id_ed25519 |
| xclip -selection clipboard < ~/.ssh/id_rsa.pub | xclip -selection clipboard < <key> |
IMPORTANT:
If xclip is not installed then issue 2x commands: sudo apt-get update and sudo apt-get install xclip. Finally, navigate to github.com Settings | SSH and GPG keys | New SSH key | Paste contents from id_rsa.pub here.
Personal Access Token
If you favor HTTPS then Personal Access Tokens offer security benefits over password-based authentication. Launch gitlab.com. Navigate to Edit Profile | Access Tokens | Add new token. Create personal access token.
After new Personal Access Token is created upload into source control software for example Source Tree on Windows and Mac OS/X and JetBrains products on Linux that integrate Git like PyCharm for Python etc etc.
Source Tree
Launch Source Tree | Settings | Accounts Add... | Enter the following details | Paste personal access token:
JetBrains IDE
Launch PyCharm | File | Settings... | Version Control | GitLab. Click "+" button | Enter GitLab token | Log In:
Example
In 2021, we coded an example from scratch as full end-to-end Web API demo on local host, in Docker and Kubernetes. Let's replicate with Python Flask API but this time include Gitlab CI/CD deployment pipelines.
Navigate to gitlab.com. Create blank project. Launch PyCharm. Create the new project gitlabcheatsheet.
|
|
Enter main.py code. Install packages. Press F5 to debug. Launch Terminal | curl http://localhost:8080
| main.py | Dockerfile |
from flask import Flask
app = Flask(__name__)
@app.route('/')
def root():
return "Hello World (Python)!\n"
if __name__ == "__main__":
app.run(host="0.0.0.0", port=8080)
|
FROM python:3-alpine WORKDIR /service COPY requirements.txt . RUN pip install -r requirements.txt COPY . ./ EXPOSE 8080 ENTRYPOINT ["python3", "main.py"] |
Code Dockefile. Launch Terminal | Enter the following Docker commands | curl http://localhost:8080
|
docker build --pull --rm -f "Dockerfile" -t flask-api:latest "." docker run --rm -d -p 8080:8080/tcp flask-api:latest --name "flask-api" |
Create local KinD cluster. Enter all the following Kubernetes Deployment and Service YAML configuration:
| Kubernetes.yaml | |
--- Deployment apiVersion: apps/v1 kind: Deployment metadata: name: flask-api-deployment namespace: test-ns spec: replicas: 1 selector: matchLabels: app: flask-api template: metadata: labels: app: flask-api spec: containers: - name: flask-api image: flask-api:latest imagePullPolicy: Never ports: - containerPort: 8080 |
--- Service apiVersion: v1 kind: Service metadata: name: flask-api-service namespace: test-ns spec: selector: app: flask-api type: NodePort ports: - protocol: TCP port: 80 targetPort: 8080 |
Finally test: Launch Terminal | Enter the following Kubernetes commands | curl http://localhost:8080
|
kubectl config set-context --current --namespace=test-ns kind load docker-image flask-api:latest --name flask-cluster kubectl apply -f Kubernetes.yaml kubectl port-forward service/flask-api-service 8080:80 curl http://localhost:8080 |
Cluster
Extend example but this time include Gitlab CI/CD deployment pipelines. Ensure you have all Pre-Requisites:
|
Python 3.8 GitLab account [CI/CD] Microsoft Azure subscription [AZ CLI] VS Code Docker + Kubernetes |
Docker Hub account Kubernetes in Docker [KinD] docker [CLI] kubectl [CLI] |
Azure AKS
Launch Terminal | Login to Azure portal. Enter commands to extract Service Principal and generate SSH key:
|
az login az ad sp create-for-rbac --name ${USER}-bz-sp cd ~/.ssh |
ssh-keygen -t rsa -b 4096 -N '' -f master_ssh_key eval $(ssh-agent -s) ssh-add master_ssh_key |
Export the following environment variables. Note: use the Service Principal information for first 3x ENV VARs
|
export AZ_SP_ID=<value_from_appId> export AZ_SP_PASSWORD=<value_from_password> export AZ_TENANT_ID=<value_from_tenant export CLUSTER_NAME=stevepro-dev-cluster export CLUSTER_NODES=3 export MASTER_SSH_KEY=~/.ssh/master_ssh_key.pub |
export AZ_NETWORK_MODE=transparent export AZ_CREATE_MODE=AKS export AZ_LB_SKU=standard export AZ_VM_SIZE=Standard_D2s_v3 export AZ_LOCATION=northeurope export KUBERNETES_VERSION=1.28 |
Create the Azure resource group then finally create the AKS cluster. Note: this process can take few minutes
| az group create --name ${CLUSTER_NAME} --location ${AZ_LOCATION} --debug |
|
az aks create --name ${CLUSTER_NAME} --resource-group ${CLUSTER_NAME} \ --dns-name-prefix ${CLUSTER_NAME} --node-count ${CLUSTER_NODES} \ --node-vm-size ${AZ_VM_SIZE} --kubernetes-version ${KUBERNETES_VERSION} \ --ssh-key-value ${MASTER_SSH_KEY} --service-principal ${AZ_SP_ID} \ --client-secret ${AZ_SP_PASSWORD} --load-balancer-sku ${AZ_LB_SKU} \ --network-plugin azure --debug |
Finally, execute following commands to download + export KUBECONFIG file on localhost and setup context
|
export KUBECONFIG=~/.kube/config az aks get-credentials --name ${CLUSTER_NAME} \ --resource-group ${CLUSTER_NAME} --file ${KUBECONFIG} kubectl create ns test-ns kubectl config set-context --current --namespace=test-ns |
Pipeline
Follow this tutorial to create and run your first GitLab CI/CD pipeline. Create .gitlab-ci.yml file at root the of the project directory. Refactor all YAML configuration in files beneath the cicd directory. Automate CI/CD to the Kubernetes cluster using helm charts. This way, multiple environments can be targeted and customized.
Variables
Navigate to gitlab.com | Settings | CI/CD | Variables. Enter all GitLab Variables here e.g. Docker registry host and username. Secure sensitive information like Docker password using Base64. Add KUBECONFIG:
Deployment
Trigger CI/CD pipeline. Launch Terminal | Repeat kubectl port-forward. Test: curl http://localhost:8080
|
kubectl port-forward service/flask-api-service 8080:80 curl http://localhost:8080 |
GitFlow
Gitflow is an alternative Git branching model that involves the use of feature branches and multiple primary branches. Here is practical guide to implement GitFlow using GitLab with the corresponding git commands.
Launch gitlab.com. Create new project. Default branch will be main. Git clone repository to localhost. For completeness cut initial Tag 0.1. In SourceTree setup GitFlow: Repository | Git-flow | Initialize Repository:
In gitlab.com set develop as default branch: GitLab | Setttings | Repository | Branch defaults | develop.
|
IMPORTANT - you can compare two branches [source vs. target] anytime issuing the following commands:
|
Summary
To summarize, now that we have discussed the differences between GitHub and GitLab, we have extended our previous end-to-end Web API demo on local host, in Docker and Kubernetes but now leveraged GitLab CI/CD pipelines to automate deployments to the cloud. Our next steps would be more GitFlow integration: Replicate deployments across multiple environments and complete the full Software Development LifeCyle!
Saturday, June 1, 2024
MacOS Setup Cheat Sheet II
In the previous post we checked out MacOS Setup Cheat Sheet to seamlessly transfer all keyboard shortcuts from Windows to Mac. Now we would like to transfer all Application navigation as M1 powered MacBooks are more appealing for companies than ever. Especially with the rise of A.I. requirements for faster data access.
Let's check it out!
Navigation
In the previous post, we began looking at using keyboard shortcuts on Mac. Now we would like to compare document shortcuts that we are used to for navigation on Windows however now using Ctrl instead of Cmd:
The main issue is there doesn't seem to be generic shortcut mapping; navigation seems application specific!
TextEdit
The Mac text system uses generalized key bindings mechanism that is completely re-mappable by all users. The standard bindings include all system defaults but you can override bindings in your own custom dict file:
Here is how to override MacOS key bindings: Launch Terminal and create new a DefaultKeyBinding.dict file:
Typically, the default key bindings for the Home and End keys in MacOS are different to any other operating system. Thus you may like to fix the Home and End buttons first and in combination with Shift and Ctrl keys.
Next key bindings that may not be working is to move cursor from word to word. Integrate Keycode Modifier learnt from the previous post to complete custom dict file to change all MasOS key bindings. Here is my file:
For completeness, here is a thorough KeyBindings tutorial explaining MacOS Keybinding Key Syntax and the Action Code. Detailed DefaultKeyBinding.dict examples shared online by users Belphemu, trusktr and zsimic.
Finally, in researching Default Key Bindings many posts promote Karabiner-Elements as a powerful keyboard customizer for macOS. However, it has been recommended not to use on M1 hardware due to kernel panics.
Terminal
Ideally we would like to update navigation on Terminal due to frequent use: Terminal | Settings... | Profiles. Keyboard tab customize Key and Action settings. Check Use Option as Meta Key to "backward delete word". Unfortunately, Terminal navigation not aligned therefore will prefer IDEs like VS Code or PyCharm terminals.
VS Code
Visual Studio Code is a cross-platform code editor available on Windows, Linux and Mac. However, unlike the TextEdit, VS Code does not respect DefaultKeyBinding.dict on MacOS. Therefore, change keyboard shorcuts!
Customize VS Code keybindings to align navigation: Launch VS Code | Settings | Keyboard Shortcuts | Add:
Continue to customize Menu keyboard shortcuts that one is accustomed to aligning VS Code navigation e.g.:
The keybindings for VS Code are updated in ~/Library/Application\ Support/Code/User/keybindings.json file. Update keybindings.json directly for Terminal keyboard shortcuts like delete word prior + cursor movement:
PyCharm Community
Align PyCharm Keymap to VS Code Keymap for consistent keyboard shortcuts and developer IDE experience. In the "Welcome to PyCharm" popup choose Plugins | VSCode Keymap | Install. Navigate back to Customize. Under Keymap change default value VSCode (macOS). Now most VS Code shortcuts will work like Windows.
However, in order to better align full navigation, update the following from PyCharm | Settings... | Keymap:
IMPORTANT: typically use Alt + Left to Navigate Back but this is now doubled to Move Word left as priority!
NOTE
If Ctrl + Shift + O launches Symbols then remove duplicate: Choose Settings | Keymap | filter as "symbol". Double click Navigate | Go to Symbol | Edit | Remove Ctrl+Shift+O | Apply | OK. Repeat this as necessary.
Excel
Keyboard shortcuts in Excel like Ctrl+Home and Ctrl+End won't work so prefer to use arrow keys to navigate
PostgreSQL
PostgreSQL is preferred by companies because it offers data types not found in MySQL. Plus MySQL can be used for web-based/online or mapping functions, whereas PostgreSQL is recommended for large analytical processes.
Postgres
Setup a local PostgreSQL database server on Mac OS/X. Download Postgres.app PostgreSQL 16 (Universal).
pgAdmin
Download and install local pgAdmin database client on Mac OS/X to administer and PostgreSQL instances.
PyCharm Professional
After installing Postgres server one can use pgAdmin client for data access. However, it seems challenging to customize navigation here. An excellent alternative is Database Tools + SQL plugin available from PyCharm.
This plugin is available only in PyCharm Professional but may prove to be a valuable investment longterm J
Download and install PyCharm Professional for macOS. Launch PyCharm Professional. Import Settings from PyCharm Community Edition. Thus you should have all navigation aligned! Note the Database Tools + SQL plugin will be enabled by default. Interesting plugins PostgreSQL Debugger and SQL Optimizer + Indexing.
Connect to database: View menu | Database | Click "+" | Data Source | PostgreSQL. Enter credentials | OK.
Checkout the PyCharm documentation to Run queries. Align F5 to Execute queries similar to SQL Server Mgt Studio: Settings | Keymap | Database | Console Toolbar | Execute. Right click | Add keyboard shortcut | F5.
Checkout the PyCharm documentation to update Word Case as per SQL convention: Settings | Editor | Code Style | SQL | PostgreSQL. Inherit general SQL style DISABLE | Word Case | Keywords To upper. Also update Code Completion as Ctrl + Space: Settings | Keymap | Main Menu | Code | Code Completion | Ctrl + Space.
Finally in Settings | Editor | General | Code Completion enable the following options: Automatically insert single suggestions, Sort suggestions alphabetically, Show suggestions as you type, Match case: First letter.
Summary
To summarize, armed with this knowledge we have setup the MacOS to better align keyboard shortcuts and application navigation techniques accumulated from years of Windows development experience. From here, we are in now an excellent position to leverage the M1 chip for faster data access as per A.I. requirements!
Let's check it out!
Navigation
In the previous post, we began looking at using keyboard shortcuts on Mac. Now we would like to compare document shortcuts that we are used to for navigation on Windows however now using Ctrl instead of Cmd:
| Home | Move to beg of line | Ctrl + Home | Move to beg of file |
| End | Move to end of line | Ctrl + End | Move to end of file |
| Ctrl + Left | Move to prev word | Ctrl + Shift + Left | Select the prev word |
| Ctrl + Right | Move to next word | Ctrl + Shift + Right | Select the next word |
| Shift + Home | Select to beg of line | Ctrl + Shift + Home | Select to beg of file |
| Shift + End | Select to end of line | Ctrl + Shift + End | Select to end of file |
TextEdit
The Mac text system uses generalized key bindings mechanism that is completely re-mappable by all users. The standard bindings include all system defaults but you can override bindings in your own custom dict file:
| SYSTEM | /System/Library/Frameworks/AppKit.framework/Resources/StandardKeyBinding.dict |
| CUSTOM | ~/Library/KeyBindings/DefaultKeyBinding.dict |
Here is how to override MacOS key bindings: Launch Terminal and create new a DefaultKeyBinding.dict file:
|
sudo bash mkdir -p ~/Library/KeyBindings touch ~/Library/KeyBindings/DefaultKeyBinding.dict open -e ~/Library/KeyBindings/DefaultKeyBinding.dict |
Typically, the default key bindings for the Home and End keys in MacOS are different to any other operating system. Thus you may like to fix the Home and End buttons first and in combination with Shift and Ctrl keys.
Next key bindings that may not be working is to move cursor from word to word. Integrate Keycode Modifier learnt from the previous post to complete custom dict file to change all MasOS key bindings. Here is my file:
|
Finally, in researching Default Key Bindings many posts promote Karabiner-Elements as a powerful keyboard customizer for macOS. However, it has been recommended not to use on M1 hardware due to kernel panics.
Terminal
Ideally we would like to update navigation on Terminal due to frequent use: Terminal | Settings... | Profiles. Keyboard tab customize Key and Action settings. Check Use Option as Meta Key to "backward delete word". Unfortunately, Terminal navigation not aligned therefore will prefer IDEs like VS Code or PyCharm terminals.
VS Code
Visual Studio Code is a cross-platform code editor available on Windows, Linux and Mac. However, unlike the TextEdit, VS Code does not respect DefaultKeyBinding.dict on MacOS. Therefore, change keyboard shorcuts!
Customize VS Code keybindings to align navigation: Launch VS Code | Settings | Keyboard Shortcuts | Add:
| Command | Keybinding | Description |
| cursorTop | Ctrl + Home | Move to beg of file |
| cursorBottom | Ctrl + End | Move to end of file |
| cursorWordLeft | Ctrl + Left | Move to prev word |
| cursorWordRight | Ctrl + Right | Move to next word |
| cursorTopSelect | Ctrl + Shift + Home | Select to beg of file |
| cursorBottomSelect | Ctrl + Shift + End | Select to end of file |
| cursorWordLeftSelect | Ctrl + Shift + Left | Select the prev word |
| cursorWordRightSelect | Ctrl + Shift + Right | Select the next word |
| deleteWordLeft | Alt + Backspace | Delete the prev word |
Continue to customize Menu keyboard shortcuts that one is accustomed to aligning VS Code navigation e.g.:
| Command | Keybinding | Description |
| File: Open Folder... | Ctrl + K Ctrl + O | workbench.action.files.openFolderViaWorkspace |
| Terminal: Clear | Ctrl + L | workbench.action.terminal.clear |
| Terminal: Create New Terminal | Ctrl + Shift + T | workbench.action.terminal.new |
The keybindings for VS Code are updated in ~/Library/Application\ Support/Code/User/keybindings.json file. Update keybindings.json directly for Terminal keyboard shortcuts like delete word prior + cursor movement:
|
PyCharm Community
Align PyCharm Keymap to VS Code Keymap for consistent keyboard shortcuts and developer IDE experience. In the "Welcome to PyCharm" popup choose Plugins | VSCode Keymap | Install. Navigate back to Customize. Under Keymap change default value VSCode (macOS). Now most VS Code shortcuts will work like Windows.
|
|
However, in order to better align full navigation, update the following from PyCharm | Settings... | Keymap:
| Editor Actions | Move Caret to Text Start | Ctrl + Home |
| Editor Actions | Move Caret to Text End | Ctrl + End |
| Editor Actions | Move Caret to Text Start with Selection | Ctrl + Shift + Home |
| Editor Actions | Move Caret to Text End with Selection | Ctrl + Shift + End |
| Editor Actions | Move Caret to Previous Word | Ctrl + Left |
| Editor Actions | Move Caret to Next Word | Ctrl + Right |
| Editor Actions | Move Caret to Previous Word with Selection | Ctrl + Shift + Left |
| Editor Actions | Move Caret to Next Word with Selection | Ctrl + Shift + Right |
| Editor Actions | Move Up and Scroll with Selection | Ctrl + Shift + Up |
| Editor Actions | Move Down and Scroll with Selection | Ctrl + Shift + Down |
| Editor Actions | Delete to Word Start | Alt + Backspace |
| Main Menu | Navigate Back | Alt + Up |
| Main Menu | Navigate Forward | Alt + Down |
NOTE
If Ctrl + Shift + O launches Symbols then remove duplicate: Choose Settings | Keymap | filter as "symbol". Double click Navigate | Go to Symbol | Edit | Remove Ctrl+Shift+O | Apply | OK. Repeat this as necessary.
Excel
Keyboard shortcuts in Excel like Ctrl+Home and Ctrl+End won't work so prefer to use arrow keys to navigate
| Ctrl + Left | Move to first cell of selected row |
| Ctrl + Right | Move to last cell of selected row |
| Ctrl + Up | Move to first cell of selected column |
| Ctrl + Down | Move to last cell of selected column |
PostgreSQL
PostgreSQL is preferred by companies because it offers data types not found in MySQL. Plus MySQL can be used for web-based/online or mapping functions, whereas PostgreSQL is recommended for large analytical processes.
Postgres
Setup a local PostgreSQL database server on Mac OS/X. Download Postgres.app PostgreSQL 16 (Universal).
pgAdmin
Download and install local pgAdmin database client on Mac OS/X to administer and PostgreSQL instances.
PyCharm Professional
After installing Postgres server one can use pgAdmin client for data access. However, it seems challenging to customize navigation here. An excellent alternative is Database Tools + SQL plugin available from PyCharm.
This plugin is available only in PyCharm Professional but may prove to be a valuable investment longterm J
| PyCharm Community | PyCharm Professional |
|
|
Download and install PyCharm Professional for macOS. Launch PyCharm Professional. Import Settings from PyCharm Community Edition. Thus you should have all navigation aligned! Note the Database Tools + SQL plugin will be enabled by default. Interesting plugins PostgreSQL Debugger and SQL Optimizer + Indexing.
Connect to database: View menu | Database | Click "+" | Data Source | PostgreSQL. Enter credentials | OK.
|
|
Checkout the PyCharm documentation to Run queries. Align F5 to Execute queries similar to SQL Server Mgt Studio: Settings | Keymap | Database | Console Toolbar | Execute. Right click | Add keyboard shortcut | F5.
Checkout the PyCharm documentation to update Word Case as per SQL convention: Settings | Editor | Code Style | SQL | PostgreSQL. Inherit general SQL style DISABLE | Word Case | Keywords To upper. Also update Code Completion as Ctrl + Space: Settings | Keymap | Main Menu | Code | Code Completion | Ctrl + Space.
|
|
Finally in Settings | Editor | General | Code Completion enable the following options: Automatically insert single suggestions, Sort suggestions alphabetically, Show suggestions as you type, Match case: First letter.
Summary
To summarize, armed with this knowledge we have setup the MacOS to better align keyboard shortcuts and application navigation techniques accumulated from years of Windows development experience. From here, we are in now an excellent position to leverage the M1 chip for faster data access as per A.I. requirements!
Subscribe to:
Posts (RSS)
