Tuesday, November 15, 2022

Golang Cheat Sheet II

In the previous post, we checked out Go as a statically typed and compiled programming language designed at Google. Go is gaining popularity due to its ability to deliver highly scalable concurrent computing software.

Development teams prefer Golang to harness the power of large servers and sophisticated cloud platorms. Go can be used in mobile, web and simple game development and can also invoke C code directly via Cgo.

Let's check it out!

Example
In the previous post, we created a simple Hello World example by setting up the development environment, create a Go module with the code and issue go build and run commands to execute the Hello World program

Modules
A module in Go is a collection of Go packages stored in a file tree with a go.mod file at its root The go.mod file defines the module's path, which is also the import path used for the root directory, and its dependency requirements which are other modules needed for a successful build.

A module is defined by a UTF-8 encoded text file named go.mod in its root directory. Here are some usages:
 go get  Updates module dependencies in the go.mod file
 go install  Builds and installs the packages named by paths
 go list -m  Lists all the modules instead of all the packages
 go mod download  Downloads named modules into the module cache
 go mod init  Initializes + writes new go.mod file in the current directory
 go mod tidy  Ensures the go.mod file matches source code in the module

Packages
Go programs are built by linking together packages. A package is constructed from one or more source files that declare constants, types, variables and functions belonging to the package to which are accessible in all files of the sams package. Those elements may be exported and used in other package(s).

Functional Programming
Functional programming is the paradigm that treats computation as the evaluation of functions and avoids state and mutable data as opposed to imperative programming which empathizes change of state. Without shared data, concurrency becomes easier because this means no locks, race conditions, no dead locks etc.

Concurrency
Concurrency is the "composition of independently executing computations". Concurrency: dealing with lots of things at once but parallelism: about doing lots of things at once. Concurrency is all about structure whereas parallelism is about execution. Consequently, Golang provides first-class language support for concurrency.

Challenges in concurrent programming include co-ordinating tasks + shared memory. Thus, Go concurrency implementation centers around goroutines: functions that execute simultaneously in Go program with other (goroutine) functions and are lightweight threads managed by the Go runtime. Here are some more terms:

 Concurrency  Ability to have multiple things that can be worked on not necessarily at the same time
 Parallelism  Ability to work on multiple concurrent tasks typically all worked on at the same time
 Process  Part of the Operating System resposible for executing an application
 Thread  Unit within the process which executes the code inside an application
 Goroutines  Special function type that can run while other goroutines are running
 WaitGroups  Allows a supervising goroutine to wait on several other go routines
 Mutexes  Allow us to share memory between go routines and our application
 Channels  Another mechanism available to us to coordinate the work between multiple goroutines

Concurrency in Go is also based on Communicating Sequential Processes. CSP is a form to describe patterns of interaction in concurrent systems. CSP involvement in concurrency based on messaging passing channels. Channels in Go allow goroutines to communincate with each other safely + co-ordinate work between them.

Demos
 Introduction  Goroutine ID
 Goroutines  Sequential code, Creating goroutines
 Sync Package  Using WaitGroups, Mutexes, Read/Write Mutexes
 Channels  Unbuffered channels, Buffered channels, Channel types
 Working with closed channels, Using select statements

IMPORTANT
If you are new to goroutines then you may like to log go ID() similar to logging thread ID in multithreading:
go mod init goidtesting
go get "github.com/nikandfor/goid"
go mod tidy
main.go
package main
import (
	"fmt"
	"os"
	"time"
	"github.com/nikandfor/goid"
)
func say(s string) {
	for i := 0; i < 5; i++ {
		time.Sleep(100 * time.Millisecond)
		_, _ = fmt.Fprintln(os.Stdout, goid.ID(), s)
	}
}
func main() {
	go say("world")
	say("hello")
}
Data Race
A data race occurs when two goroutines access the same variable concurrently and are very hard to debug:
 go run -race mysrc.go // to run the source file
 go build -race mycmd // to build the command
 go install -race mypkg // to install the package
 go test -race mypkg // to test the package

Unit Testing
Unit testing in Golang uses the testing package however more sophisticated Test Driven Design TDD / BDD can be accomplished using the Ginkgo testng framework complemented by the Gomega matching library.

However, unit testing concurrent code in Go is more challenging esp. anonymous goroutines as they are not executed immediately. Instead the calling thread keeps execution priority until it pauses. Only then will the code inside the goroutine get execution. Therefore, a solution could be insert an empty function which acts as a hook for testing purposes; push a value into a channel which blocks until received + TEST code passes.

Cloud Computing
Cloud computing systems are highly scalable that require access to resources which must be shared. Thus, Golang has the capability to develop highly scalable, highly performant, reliable and concurrent applications.

Consequently, Go is often preferred to build tools for distributed systems and containerized applications such as Docker and Kubernetes because of Go's fast compilation, low-level system calls and concurrency features.

Cgo
Cgo lets Go packages call C code. Cgo outputs Go and C files that can be combined into a single Go package.

Hello World
 hello.go  hello.h  hello.c
 package main
 import "fmt"

 // #include "hello.h"
 import "C"

 func main() {
 	fmt.Println("begin")
 	C.hello()
 	fmt.Println("-end-")
 }
 #ifndef _HELLO_H_
 #define _HELLO_H_

 #include <stdio.h>

 void hello();

 #endif//_HELLO_H_
 #include "hello.h"

 void hello()
 {
     fprintf(stderr, "%s\n", "Hello Cgo!");
 }
According to this thread, it is possible to debug Go to C from Visual Studio Code on Linux. Assuming you can F5 run the project then set breakpoints either side the Go code calling into and out of the C function to step.


Mod Security
In the previous post, we coded an example in Go from scratch as full end-to-end Web API demo on our local host, in Docker and on Kubernetes. Extend this example as Web Application Firewall using Mod Sec and Cgo.

Install Mod Security on local host
 git clone --depth 1 -b v3/master --single-branch https://github.com/SpiderLabs/ModSecurity
 cd ModSecurity
 git submodule init
 git submodule update
 ./build.sh
 ./configure
 make
 sudo make install
Follow all instructions from the example i.e. download OWASP rules, build main, server and waf code. Here, it is critical for Go to C interface to set include + linker flags correctly avoiding runtime PATH for shared libs!

waf.go
 package waf
 // #cgo CFLAGS: -I/usr/local/modsecurity/include
 // #cgo LDFLAGS: -L/usr/local/modsecurity/lib/ -Wl,-rpath -Wl,/usr/local/modsecurity/lib/ -lmodsecurity
 // #include "waf.h"
 import "C"

Asm
Finally, if you would like to reproduce Go example that calls ASM assembly code then replicate goid source:
 main.go  goid_amd64.s
 package main
 import (
 	"fmt"
 	"goidtesting/goid"
 	"os"
 	"time"
 )
 func say(s string) {
 	for i := 0; i < 5; i++ {
 		time.Sleep(100 * time.Millisecond)
 		_, _ = fmt.Fprintln(os.Stdout, goid.ID(), s)
 	}
 }
 func main() {
 	go say("world")
 	say("hello")
 }
 #include "textflag.h"

 TEXT ·ID(SB), NOSPLIT, $0-8
 	MOVQ	(TLS), AX     // AX = getg()
 	MOVQ	0x98(AX), AX   // AX = AX.goid
 	MOVQ	AX, ret+0(FP) // ret = AX
 	RET

Summary
To summarize, Go is now popular due to its ability to deliver highly scalable concurrent computing software. Plus, Cgo also allows Go programs to interoperate with C libraries, distributed cloud computing, and eBPF.

Therefore, writing low level C code to target 8-bit Sega Master System hardware and 16-bit Sega Mega Drive has proved to be very valuable experience for Cgo applications which could assist in Go + Cloud Computing!