3 popular GOTO conference talks

GOTO conferences are for developers by developers. On gotocon.com you find the upcoming conferences:

  • GOTO London: Sep. 14 – 18, 2015
  • GOTO Copenhagen: Oct. 5 – 8, 2015
  • GOTO Berlin: Dec. 2 – 4, 2015
  • GOTO Chicago: May 2016
  • GOTO Amsterdam: June 13 – 15, 2016

There have already been many GOTO conferences. Many of the past talks are available on YouTube. Below you find 3 interesting talks from the YouTube GOTO conference channel.

1) Introduction to NoSQL

goto_happy_unhappy_sql

A GOTO classic from the 2012 Aarhus conference. This is actually the most viewed talk on the YouTube GOTO conference channel. Martin Fowler explains what NoSQL is and when it needs to be applied. Like no other he explains the advantages of SQL and the circumstances under which choosing NoSQL may make sense. Apart from the NoSQL topics, his explanation of off-line locks and document based databases is so good that is enough reason by itself to watch this video.

2) Challenges in implementing microservices

goto_how_fast_can_you_go

This video was recorded more recently (August 2015) in Amsterdam. Fred George explains how Web and SOA have led to new paradigms for structuring enterprise applications. Instead of a few, business­ related services, he developed systems made of many small short-lived services. This approach is called “micro­services” and Fred George talks about his experience building applications in this paradigm.

3) How Go is making us faster

goto_go_is_making_us_faster

In July 2015 Wilfried Schobeiri explained in Chicago on a GOTO conference why Go is a good match for microservices. I’m not sure about the factual correctness of the speed comparisons with C++ and Java, but Go is definitely very fast. The thing to take away from this talk is that Go might be a great match for microservices, since it has a nice standard library/toolset, easy parallelism/concurrency and simple deployment.

Redis sorted set stores score as a floating point number

Today I was playing a little with our statistics. I was writing some “Go” (golang) that was requesting a top 20 customers from Redis using a “Sorted Set” and the “ZREVRANGEBYSCORE” command. Then I found out that the score of the sorted set was actually stored as a double precision floating point number. Normally, I would not be bothered about this and use the float storage for integer values.

But this time I wanted to make a top 20 of the real-time monthly data traffic of our entire CDN platform. A little background: Hits on the Nginx edges are measures in bytes and the logs are streamed to our statistics cluster. Therefor the real-time statistics counters for the traffic are in bytes. Normally we use 64 bit integers (in worst case they are signed and you lose 1 bit).

2^64 = 9,223,372,036,854,775,807
      EB, PB, TB, GB, MB, kB,  b

If you Google for: “9,223,372,036,854,775,807 bytes per month in gigabit per second” you will find that this is about 26 Tbps on average. We do not have such big customers yet, so that will do for now. So an “int64” will do, but how about the double precision float? Since it has a floating point, theory says it can not reliably count when numbers become too large. But how large is too large? I quickly implemented a small script in golang to find out:

package main

import (
	"fmt"
)

func main() {
	bits := 1
	float := float64(1)
	for float+1 != float {
		float *= 2
		bits++
	}
	fmt.Printf("%.0f = %d bits\n", float, bits)
}

Every step the script doubles the number and tries to add 1 until it fails. This is the output showing when the counting goes wrong:

9007199254740992 = 54 bits

So from 54 bits the counting is no longer precise (to the byte). What does that mean for our CDN statistics? Let’s do the same calculation we did before:

2^54 = 9,007,199,254,740,992
      PB, TB, GB, MB, kB,  b

If you Google for: “9,007,199,254,740,992 bytes per month in gigabits per second” you will find that this is about 25 Gbps on a monthly average. We definitely have customers that do much more than that.

I quickly calculated that the deviation would be less than 0.000000000000001%. But then I realized I was wrong: At 26 Tbps average the deviation might as well be as big as 1 kB (10 bits). Imagine that the customer is mainly serving images and JavaScript from the CDN and has an average file size of 10 kB. In this case the statistics will be off by 10% during the last days of the month!

Okay, this may be the worst case scenario, but still I would not sleep well ignoring it. I feel that when it comes to CDN statistics, accuracy is very important. You are dealing with large numbers and lots of calculation and as you see this may have some unexpected side effects. That is why these kind of seemingly small things keep me busy.

Go (golang) version 1.3 is released

golang

Version 1.3 of Go (golang) was released yesterday! Here is a summary of the changes that are announced in the blog:

  1. Performance improvements to garbage collector, stack management strategy, race detector, and regular expression engine.
  2. Overhaul of the Go linker which can speed up incremental builds for large projects.
  3. The gc toolchain now supports the NaCl execution sandbox which permits the safe execution of untrusted code.
  4. Godoc now performs static analysis, making it easier than ever to navigate and understand Go programs
  5. Experimental support for the DragonFly BSD, Plan 9, and Solaris operating systems (when installed from source).
  6. The garbage collector is now precise when examining stacks; read the release notes carefully if you have unsafe code.

Everybody should upgrade to this new stable version. You can follow our (updated) recent post on installing Go if you need help.

Debugging Go using LiteIDE on Ubuntu 14.04

So this weekend I decided to give the Go programming language (golang) a spin. Go is best described like this:

Go, also called golang, is a programming language initially developed at Google in 2007 by Robert Griesemer, Rob Pike, and Ken Thompson. It is a statically-typed language with syntax loosely derived from that of C, adding garbage collection, type safety, some dynamic-typing capabilities, additional built-in types such as variable-length arrays and key-value maps, and a large standard library. – Wikipedia

When you are stuck, don’t search the web for “Go”; it will not give relevant results. Use the word “golang” instead.

Installing golang on Ubuntu 14.04

ubuntu-go

I am using Ubuntu 14.04 (actually Xubuntu), but these instructions should also work on any other recent Debian-based Linux (like Mint 17). I tested using a clean install, so you should not run into unexpected dependencies. First we install the two prerequisites (Git and Mercurial):

sudo apt-get install mercurial git

Then we download the latest version of the Go language in which debugging is working: version 1.2.2. Debugging (gdb) is broken in golang 1.3 and hopefully gets fixed in 1.4. EDIT: Thank you ondrej for finding this issue. The following commands will download the software package and unpack it to the “go” directory in your home folder (in my case “/home/maurits/go”):

cd ~
wget http://golang.org/dl/go1.2.2.linux-amd64.tar.gz
tar -xf go1.2.2.linux-amd64.tar.gz
rm go1.2.2.linux-amd64.tar.gz

Go supports installing third party packages using the “go get” command. These are not installed in the same directory as the initial Go installation. This simplifies package management, because every project can have it’s own workspace if needed. So, we create a (first) workspace directory that will hold third party packages. We will name it “workspace-go” since it is still generic and it will be located in my home folder.

mkdir ~/workspace-go

We will use the directory later, when we “go get” some third party packages.

Setting up the command-line environment

To avoid adding the GO settings to my environment, I have made a file that holds my Go environment variables. You need to set three variables: GOROOT is the installation directory of GO, GOPATH is the workspace directory for “go get” and finally it can be convenient to add the “bin” directories from GOROOT and GOPATH to the (search) PATH. Using the following commands I make a “go.env” file that can be used to set the environment variables.

echo "export GOROOT=\$HOME/go" > ~/go.env
echo "export GOPATH=\$HOME/workspace-go" >> ~/go.env
echo "export PATH=\$PATH:\$GOROOT/bin:\$GOPATH/bin" >> ~/go.env

To actually load the above file you have to type the following command in the Terminal:

. ~/go.env

Note that the environment variables are only set in the active terminal. When you open another Terminal you will have to run the “go.env” file before you can use the “go” command(s). When your Go environment is correctly set you can (for example) get some nice third party tools using:

go get code.google.com/p/go.tools/cmd/...
go get github.com/golang/lint/golint

Run a for loop example from the command-line

There is a very good website named “Go by Example“. It holds a large set of small Go files that clearly show how the various built-in functions of the Go language work. I chose the “for loop” example to show how compiling and running a golang file works. I make a directory “hello” (from “hello world”) to store my first Go programs in. Then I download the “for.go” example:

mkdir ~/hello
cd ~/hello
wget https://raw.githubusercontent.com/mmcgrana/gobyexample/master/examples/for/for.go

To run the code you need to open a Terminal and load the Go environment from the “go.env” file. Now the “go” command is available and you can execute “go run” as shown below:

. ~/go.env
cd ~/hello
go run for.go

You can also first “build” an executable file. The advantage is that you can distribute this file. It can be executed on any other Linux machine with the same architecture. Go also support cross-compiling for other architectures. It means that Go does not require anything (a VM or run-time) to be installed on the target machine. The downside to this is that the executable size is large. This simple “for.go” script has a compiled size of 2,2 MB (megabyte). This can be reduced to 1,2 MB by removing debug symbols using the “ldflags” like below:

. ~/go.env
cd ~/hello
go build -ldflags "-s" for.go
./for

Debug from the command-line

Golang has decent GDB support. This may help you a lot when you run bugs that you cannot effectively squash using “printf debugging”. Below I give an example of a run where I break on line 19 of the “for.go” script. Note that you expect the variables “i” and “j” to be “4” and “7”. They will not have these values, unless you disable compiler optimizations by adding the “gcflags” to the build command as shown below:

. ~/go.env
cd ~/hello
go build -gcflags "-N -l" for.go
gdb for
b for.go:19
r
l
i loc
q
y

LiteIDE: a Go IDE with debugging capabilities

liteide2

In the above section we have seen how to run, build and debug from the command line. Fortunately, there is also a nice IDE available that simplifies these things. It is called LiteIDE and it supports just one language: Go. The following commands download it from sourceforge and install it in the “liteide” directory in your home folder (in my case “/home/maurits/liteide”):

wget --content-disposition http://sourceforge.net/projects/liteide/files/X22/liteidex22.linux-64.tar.bz2/download
tar -xf liteidex22.linux-64.tar.bz2
rm liteidex22.linux-64.tar.bz2

In the menu you can create a launcher that executes the following:

~/liteide/bin/liteide

The following file can be used as an icon:

~/liteide/share/liteide/welcome/images/liteide400.png

When you have started the application there is one thing you must do:

View -> Manage GOPATH... -> Add Directory... -> Select: ~/workspace-go

This will make sure that any third party package that you install can be loaded.

Step-by-step debugger for Go

To make sure the (GDB based) step-by-step debugger works you need to do the following:

Build -> Build Configuration... -> Custom -> BUILDARGS -> Enter: -gcflags "-N -l"

Verify everything is working correctly by executing the following debug run:

File -> Open Folder -> Click: hello -> Choose
Double click "for.go" in the "Folders" pane
Build -> Build (Ctrl-B)
Put cursor on line 19
F9 (toggle breakpoint)
F5 (start debugging)
F5 (until green arrow is on line 19)
Debug pane should show "Variables" tab with "i=4" and "j=7"

Happy coding!