Create a basic distributed system in Go lang – Part 1

 Introduction

Yesterday I started looking into basics of Go – A relatively young programming language and though of learning it via creating some useful case study. So I am writing this basic distributed system in Go. I haven’t thought lot of features yet and will keep adding features as I keep learning the language and as new ideas pop in my head.

Why Go

Well that was a tough call since I was almost at verge of starting with Rust – which is mozilla’s new language and is really awesome for system level stuffs. Go is relatively heavier than Rust so probably not so good as Rust on system level programming. However Go does pretty well on network and concurrency so I decided to learn these first. So Rust is still in todo’s and a thing for another post.

Starting with Go

I have got this minimal post on starting with the absolute basics of Go. If you want a very minimal Go startup text have a look there and in the Go lang docs.

What are we building?

We will be building a very very basic multi client server distributed application where we have a cluster of nodes. One of the node would be the master of the cluster and new nodes can send request to the master or any other n0des to add them to the cluster. All the requests and responses use TCP protocols. The messages are json packets. Each node also gets a unique ID to  identify itself on the cluster.

So the code tries to connect to the node of the cluster (details provided by the user) and sends request to the cluster to add it to the cluster. If the node is unable to connect to the cluster the system exits. This node also starts listening for other node requests to add new nodes to the cluster. The first node ideally becomes the master of the cluster unless it dies sometime in future where we would need to re-elect a master of the cluster – but thats for little later.

For now we just assume that when the node receives a request by a node – it adds it to a network and sends a response back. We don’t actually so any work apart from sending response. We will work on actual network creation a little later.

Features

• Ability to send / receive requests and responses
• Ability to connect to a cluster
• Ability to start a node as master if unable to connect to cluster (if its the first node)
• Have a unique identity of every node.
• Ability to exchange json packets between nodes
• Accept all information in command line arguments (will be useful in future when we grow the system)
• Have a running code (for obvious reasons)

Explanation

I have added all the explanation in the code step by step and hope its clear to understand. Feel free to post any doubts or issues you face with the code.

Run the code

Build/Install the code

$ go install main

$ ./bin/main

The node is not able to connect the any address and therefore system silently exists with a message. It provides the information about specifying a flag that would start this node as a master when it finds no other nodes.

Get some help on script parameters

$ ./bin/main -h

Start node as master

$ ./bin/main --makeMasterOnError

Add nodes in cluster

$ ./bin/main --myport 8002 --clusterip 127.0.0.1:8001

The left node is the master and the right node requests the master node to add it to the cluster. The new node receives a response from the master saying that it has added the node to the cluster.

Add More nodes

We can add more nodes to our cluster by sending more requests to any of the alive nodes in the cluster. Each node is listening for new requests and hence can add the new nodes to the cluster. We will add two new nodes to this cluster by sending request to both the master and the new non-master node.

Node 1 
$ ./bin/main --makeMasterOnError

Node 2
$ ./bin/main --myport 8002 --clusterip 127.0.0.1:8001

Node 3
$ ./bin/main --myport 8004 --clusterip 127.0.0.1:8001

Node 4
$ ./bin/main --myport 8003 --clusterip 127.0.0.1:8002

Here is our 4 node cluster. The Left-Top node is our Master node and the other nodes are the Slave nodes. The arrows in the image show the directions of the requests made by the new nodes.

Code

And there you **Go** with all the code I have come up with so far. I would love to hear about any good practices to apply in the code. Please comment on anything you would like to share or are facing issues with.

Cheers \m/