Create a basic distributed system in Go lang – Part 1

 Introduction

Yesterday I started looking into basics of Go – A relatively young programming language and though of learning it via creating some useful case study. So I am writing this basic distributed system in Go. I haven’t thought lot of features yet and will keep adding features as I keep learning the language and as new ideas pop in my head.

Why Go

Well that was a tough call since I was almost at verge of starting with Rust – which is mozilla’s new language and is really awesome for system level stuffs. Go is relatively heavier than Rust so probably not so good as Rust on system level programming. However Go does pretty well on network and concurrency so I decided to learn these first. So Rust is still in todo’s and a thing for another post.

Starting with Go

I have got this minimal post on starting with the absolute basics of Go. If you want a very minimal Go startup text have a look there and in the Go lang docs.

What are we building?

We will be building a very very basic multi client server distributed application where we have a cluster of nodes. One of the node would be the master of the cluster and new nodes can send request to the master or any other n0des to add them to the cluster. All the requests and responses use TCP protocols. The messages are json packets. Each node also gets a unique ID to  identify itself on the cluster.

So the code tries to connect to the node of the cluster (details provided by the user) and sends request to the cluster to add it to the cluster. If the node is unable to connect to the cluster the system exits. This node also starts listening for other node requests to add new nodes to the cluster. The first node ideally becomes the master of the cluster unless it dies sometime in future where we would need to re-elect a master of the cluster – but thats for little later.

For now we just assume that when the node receives a request by a node – it adds it to a network and sends a response back. We don’t actually so any work apart from sending response. We will work on actual network creation a little later.

Features

  • Ability to send / receive requests and responses
  • Ability to connect to a cluster
  • Ability to start a node as master if unable to connect to cluster (if its the first node)
  • Have a unique identity of every node.
  • Ability to exchange json packets between nodes
  • Accept all information in command line arguments (will be useful in future when we grow the system)
  • Have a running code (for obvious reasons)

Explanation

I have added all the explanation in the code step by step and hope its clear to understand. Feel free to post any doubts or issues you face with the code.

Run the code

Build/Install the code

$ go install main

$ ./bin/main

The node is not able to connect the any address and therefore system silently exists with a message. It provides the information about specifying a flag that would start this node as a master when it finds no other nodes.

go lang distributed system - confused coders

Get some help on script parameters

$ ./bin/main -h

go lang distributed system - confused coders

Start node as master

$ ./bin/main --makeMasterOnError

go lang distributed system - confused coders - start node as master

Add nodes in cluster

$ ./bin/main --myport 8002 --clusterip 127.0.0.1:8001

The left node is the master and the right node requests the master node to add it to the cluster. The new node receives a response from the master saying that it has added the node to the cluster.

go lang distributed system - add node to cluster

Add More nodes

We can add more nodes to our cluster by sending more requests to any of the alive nodes in the cluster. Each node is listening for new requests and hence can add the new nodes to the cluster. We will add two new nodes to this cluster by sending request to both the master and the new non-master node.

Node 1 
$ ./bin/main --makeMasterOnError

Node 2
$ ./bin/main --myport 8002 --clusterip 127.0.0.1:8001

Node 3
$ ./bin/main --myport 8004 --clusterip 127.0.0.1:8001

Node 4
$ ./bin/main --myport 8003 --clusterip 127.0.0.1:8002

Here is our 4 node cluster. The Left-Top node is our Master node and the other nodes are the Slave nodes. The arrows in the image show the directions of the requests made by the new nodes.

go land distributed system - add nodes to cluster

Code

And there you **Go** with all the code I have come up with so far. I would love to hear about any good practices to apply in the code. Please comment on anything you would like to share or are facing issues with.

Cheers \m/

Yash Sharma is a Big Data & Machine Learning Engineer, A newbie OpenSource contributor, Plays guitar and enjoys teaching as part time hobby.
Talk to Yash about Distributed Systems and Data platform designs.

3 thoughts on “Create a basic distributed system in Go lang – Part 1

  1. See Enduro/X (https://github.com/endurox-dev/endurox) and Enduro/X Go (https://github.com/endurox-dev/endurox-go) packages. It allows to create with ease a distributed microservices based cluster, it provides efficient load balance over several go binaries (which provides microservice APIs) either on local machine or over the other cluster instance, fault tolerance, and for certain database distributed transaction is available (over the several servers and multiple database you get single commit or abort – two phase commit). The solution can mix the services with C/C++ with the same XATMI API. The cluster is peer-to-peer based. Thus there is no master node. If there is link between machines, the services are shared between nodes, if link is lost, the services from other node are lost too. Framework is built high performance (on local machine in-memory kernel IPC block queues are used instead of streaming sockets) and fault tolerance (efficient load balancing using one-queue multiple servers strategy, automatic restarts, process pings and fault detection).

  2. Hi,
    It was really a great tutorial. It feels so good when people start sharing their learning experiences. I checked erlang and elixir before considering go to learn distributed systems. Seeing this wonderful blog, I decided to start learning go to get myself into this huge world of distributed systems.
    I would like to go deeper into distributed systems, and wanted to know how did you start on this path. I’m wondering if you have made any ideas how to do load balancing on the server.
    Cheers

Leave a Reply to Madars Vitolins Cancel reply

Your email address will not be published. Required fields are marked *