Finding bad RAM with memtest86

Lately Firefox started to crash randomly without an apparent reason. Also other software on my computer started acting weird. It took me a while to found the cause of the problems. It turned out one of my memory modules has gone bad. Finding that out was not exactly easy. Normally I would run memtest86+ from an Ubuntu live CD. That was not possible as I have an UEFI BIOS (without legacy support) and memtest86+ is lacking UEFI support.

memtest_splashscreen

I was able to create a virtual machine (with KVM or VirtualBox) with reserved RAM and run memtest86+ in there. That actually showed the problem. Another trick to detect the problem is to run memtest86 (without the plus) as it has support for booting using UEFI since version 5. It can be downloaded from memtest86.com (choose CD image) and put it on an USB stick using UNetbootin (install using apt-get).

ram_modules

When I found the problem there was no other way than to try to add and remove memory modules and run the test again to find out which one was broken. It took some time, but eventually I succeeded. It wasn’t a pretty process and it took way too long. Somebody should write a memory testing program in user-space that also reports the slot of the broken RAM module. In the end the broken module turned out to be the one closest to the CPU (see picture), maybe it got too hot.

Suggested tools:

  1. MemTest86 from: memtest86.com
  2. Memtest86+ from: memtest.org
  3. Windows Memory Diagnostic from: microsoft.com

I hope it will help you.

Share

Analyzing text protocols with a TCP proxy

To debug the Memcache server (memcached) on my localhost I needed an application that would log the in and output to a specific port on my local machine. In  the config in PHP I changed the memcache port (from 11211) to 11212. Then I ran the TCP proxy software to forward all connections on port 11212 to 11211. I know I could have used wireshark (and/or tcpdump) to just look at the traffic, but I felt like trying a more light-weight tool. I ran into two nice little TCP proxy programs: One written in Perl and one written in C++ with the Boost library.

Perl based TCP proxy

Torsten Raudssus wrote an application that has the following output:

tcpproxy

With the following commands you can run the TCP proxy:

git clone git@github.com:Getty/p5-app-tcpproxy.git
cd p5-app-tcpproxy/
sudo apt-get install cpanminus
sudo cpanm --installdeps .
perl bin/tcpproxy.pl 11212 127.0.0.1 11211

It is a lovely little project that definitely serves a niche. The only downside I see is that is has two CPAN dependencies. Perl is always available on Linux, but the dependencies certainly not. This makes it maybe a little harder to run when you are on a random machine debugging a nasty problem. If it would not have those it would become my next favorite (text) protocol analyzer.


C++ (Boost) based TCP proxy

Arash Partow also wrote a TCP proxy application in C++. It originally had no output, but I changed that. After my modifications the output looks like this:

...
56258 ==> delete memc.sess.key.lock.s5p5eh8fhvfe6iq06ot6nuim66

56258 <== DELETED

56258 ==> quit

56276 ==> add memc.sess.key.lock.s5p5eh8fhvfe6iq06ot6nuim66 0 1421275851 1
1

56276 <== STORED

56276 ==> get memc.sess.key.s5p5eh8fhvfe6iq06ot6nuim66

56276 <== END

56276 ==> set memc.sess.key.s5p5eh8fhvfe6iq06ot6nuim66 0 4 1331
debugger|a:1:{i:0;a:12:{s:3:"log";a:0:{}s:7:"queries";a:0:{}s:7:"session";a:2:{s:6:"before";
...

With the following commands you can run the TCP proxy:

git clone git@github.com:mevdschee/proxy.git
cd proxy/
sudo apt-get install build-essential
sudo apt-get install libboost-all-dev
make
./tcpproxy_server 127.0.0.1 11212 127.0.0.1 11211

As you can see I forked the original Github project to add some logging:

tcpproxy_cpp

It works like a charm, but I prefer the colored and smarter output of the Perl application. If I find some time I might copy the way the Perl proxy shows the output to C++. If you feel like it and you think you know how to do that, than I would be very happy with a PR!

NB: Arash Partow also wrote a variation to the code that does logging, it is on Google Code.

What is your favorite TCP proxy for analyzing text protocols? Let us know in the comments!

Share

Debugging Go using LiteIDE on Ubuntu 14.04

So this weekend I decided to give the Go programming language (golang) a spin. Go is best described like this:

Go, also called golang, is a programming language initially developed at Google in 2007 by Robert Griesemer, Rob Pike, and Ken Thompson. It is a statically-typed language with syntax loosely derived from that of C, adding garbage collection, type safety, some dynamic-typing capabilities, additional built-in types such as variable-length arrays and key-value maps, and a large standard library. – Wikipedia

When you are stuck, don’t search the web for “Go”; it will not give relevant results. Use the word “golang” instead.

Installing golang on Ubuntu 14.04

ubuntu-go

I am using Ubuntu 14.04 (actually Xubuntu), but these instructions should also work on any other recent Debian-based Linux (like Mint 17). I tested using a clean install, so you should not run into unexpected dependencies. First we install the two prerequisites (Git and Mercurial):

sudo apt-get install mercurial git

Then we download the latest version of the Go language in which debugging is working: version 1.2.2. Debugging (gdb) is broken in golang 1.3 and hopefully gets fixed in 1.4. EDIT: Thank you ondrej for finding this issue. The following commands will download the software package and unpack it to the “go” directory in your home folder (in my case “/home/maurits/go”):

cd ~
wget http://golang.org/dl/go1.2.2.linux-amd64.tar.gz
tar -xf go1.2.2.linux-amd64.tar.gz
rm go1.2.2.linux-amd64.tar.gz

Go supports installing third party packages using the “go get” command. These are not installed in the same directory as the initial Go installation. This simplifies package management, because every project can have it’s own workspace if needed. So, we create a (first) workspace directory that will hold third party packages. We will name it “workspace-go” since it is still generic and it will be located in my home folder.

mkdir ~/workspace-go

We will use the directory later, when we “go get” some third party packages.

Setting up the command-line environment

To avoid adding the GO settings to my environment, I have made a file that holds my Go environment variables. You need to set three variables: GOROOT is the installation directory of GO, GOPATH is the workspace directory for “go get” and finally it can be convenient to add the “bin” directories from GOROOT and GOPATH to the (search) PATH. Using the following commands I make a “go.env” file that can be used to set the environment variables.

echo "export GOROOT=\$HOME/go" > ~/go.env
echo "export GOPATH=\$HOME/workspace-go" >> ~/go.env
echo "export PATH=\$PATH:\$GOROOT/bin:\$GOPATH/bin" >> ~/go.env

To actually load the above file you have to type the following command in the Terminal:

. ~/go.env

Note that the environment variables are only set in the active terminal. When you open another Terminal you will have to run the “go.env” file before you can use the “go” command(s). When your Go environment is correctly set you can (for example) get some nice third party tools using:

go get code.google.com/p/go.tools/cmd/...
go get github.com/golang/lint/golint

Run a for loop example from the command-line

There is a very good website named “Go by Example“. It holds a large set of small Go files that clearly show how the various built-in functions of the Go language work. I chose the “for loop” example to show how compiling and running a golang file works. I make a directory “hello” (from “hello world”) to store my first Go programs in. Then I download the “for.go” example:

mkdir ~/hello
cd ~/hello
wget https://raw.githubusercontent.com/mmcgrana/gobyexample/master/examples/for/for.go

To run the code you need to open a Terminal and load the Go environment from the “go.env” file. Now the “go” command is available and you can execute “go run” as shown below:

. ~/go.env
cd ~/hello
go run for.go

You can also first “build” an executable file. The advantage is that you can distribute this file. It can be executed on any other Linux machine with the same architecture. Go also support cross-compiling for other architectures. It means that Go does not require anything (a VM or run-time) to be installed on the target machine. The downside to this is that the executable size is large. This simple “for.go” script has a compiled size of 2,2 MB (megabyte). This can be reduced to 1,2 MB by removing debug symbols using the “ldflags” like below:

. ~/go.env
cd ~/hello
go build -ldflags "-s" for.go
./for

Debug from the command-line

Golang has decent GDB support. This may help you a lot when you run bugs that you cannot effectively squash using “printf debugging”. Below I give an example of a run where I break on line 19 of the “for.go” script. Note that you expect the variables “i” and “j” to be “4” and “7”. They will not have these values, unless you disable compiler optimizations by adding the “gcflags” to the build command as shown below:

. ~/go.env
cd ~/hello
go build -gcflags "-N -l" for.go
gdb for
b for.go:19
r
l
i loc
q
y

LiteIDE: a Go IDE with debugging capabilities

liteide2

In the above section we have seen how to run, build and debug from the command line. Fortunately, there is also a nice IDE available that simplifies these things. It is called LiteIDE and it supports just one language: Go. The following commands download it from sourceforge and install it in the “liteide” directory in your home folder (in my case “/home/maurits/liteide”):

wget --content-disposition http://sourceforge.net/projects/liteide/files/X22/liteidex22.linux-64.tar.bz2/download
tar -xf liteidex22.linux-64.tar.bz2
rm liteidex22.linux-64.tar.bz2

In the menu you can create a launcher that executes the following:

~/liteide/bin/liteide

The following file can be used as an icon:

~/liteide/share/liteide/welcome/images/liteide400.png

When you have started the application there is one thing you must do:

View -> Manage GOPATH... -> Add Directory... -> Select: ~/workspace-go

This will make sure that any third party package that you install can be loaded.

Step-by-step debugger for Go

To make sure the (GDB based) step-by-step debugger works you need to do the following:

Build -> Build Configuration... -> Custom -> BUILDARGS -> Enter: -gcflags "-N -l"

Verify everything is working correctly by executing the following debug run:

File -> Open Folder -> Click: hello -> Choose
Double click "for.go" in the "Folders" pane
Build -> Build (Ctrl-B)
Put cursor on line 19
F9 (toggle breakpoint)
F5 (start debugging)
F5 (until green arrow is on line 19)
Debug pane should show "Variables" tab with "i=4" and "j=7"

Happy coding!

Share