Heka monolog decoder

This post is about how to use heka to give your symfony 2 application logs the care they deserve.

Application logs are very important for the quality of the product or service you are offering.

They help you find out what went wrong so you can explain and fix a bug that was reported recently. Or maybe to gather statistics to see how often a certain feature is used. For example how many bare metal reinstallation requests were issued last month and how many of those failed. Valuable information that you could use to decide what feature you are going to work on next.

At LeaseWeb we use quite a lot of php and Seldaek’s monolog is our logging library of choice. Recently we open sourced a heka decoder on github here. For you who do not know heka yet, check out their documentation.

Heka is an open source stream processing software system developed by Mozilla. Heka is a “Swiss Army Knife” type tool for data processing, useful for a wide variety of different tasks, such as …

Heka runs as a system daemon just like logstash or fluentd. Heka is written in go and comes with an easy to use plugin system based on lua. It has almost no dependencies and is lightweight. James Turnbull has written a nice article on how to get started with heka.

send symfony 2 application logs to elastic search

How better to explain with an example.

Lets say you have a symfony 2 application and you want the application logs to be sent to an Elastic Search platform.

On a debian based os you can download one of the heka debian packages from github.

    $ wget https://github.com/mozilla-services/heka/releases/download/v0.9.2/heka_0.9.2_i386.deb
    $ sudo dpkg -i heka_0.9.2_i386.deb

To configure heka you are required to edit the configuration file located at /etc/hekad.toml.

    $ vim /etc/hekad.toml

Take your time to read the excellent documentation on heka. There are many ways of using heka but we will use it as a forwarder:

  1. Define an input, where messages come from.
  2. Tell heka how it should decode the monolog log line into a heka message
  3. Tell heka how to encode the message so Elastic Search will understand it
  4. Define an output, where should the messages be sent to.

First we define the input:

    [Symfony2MonologFileInput]
    type = "LogstreamerInput"
    log_directory = "/var/www/app/logs"
    file_match = 'prod\.log'
    decoder = "Symfony2MonologDecoder"

Adjust `log_directory` and `file_match` according to your setup. As you can see we alread told heka to use the `Symfony2MonologDecoder` to we will define that one next:

    [Symfony2MonologDecoder]
    type = "SandboxDecoder"
    filename = "/etc/symfony2_decoder.lua"

Change the `filename` with the path where you placed the lua script on your system.

Now we have defined the input we can tell heka where to output messages to:

    [ESJsonEncoder]
    index = "%{Hostname}"
    es_index_from_timestamp = true
    type_name = "%{Type}"

    [ElasticSearchOutput]
    message_matcher = "TRUE"
    server = "http://192.168.100.1:9200"
    flush_interval = 5000
    flush_count = 10
    encoder = "ESJsonEncoder"

In the above example we assume that your Elastic Search server is running at 192.168.100.1.

And thats it.

A simple log line in app/logs/prod.log:

    [2015-06-03 22:08:02] app.INFO: Dit is een test {"bareMetalId":123,"os":"centos"} {"token":"556f5ea25f6af"}

Is now sent to Elastic Search. You should now be able to query your Elastic Search for log messages, assuming the hostname of your server running symfony 2 is myapi:

    $ curl http://192.168.100.1:9200/myapi/_search | python -mjson.tool
    {
        "_shards": {
            "failed": 0,
            "successful": 5,
            "total": 5
        },
        "hits": {
            "hits": [
                {
                    "_id": "ZIV7ryZrQRmXDiB6thY_yQ",
                    "_index": "myapi",
                    "_score": 1.0,
                    "_source": {
                        "EnvVersion": "",
                        "Hostname": "myapi",
                        "Logger": "Symfony2MonologFileInput",
                        "Payload": "Dit is een test",
                        "Pid": 0,
                        "Severity": 7,
                        "Timestamp": "2015-06-03T20:08:02.000Z",
                        "Type": "logfile",
                        "Uuid": "344e7cae-6ab7-4fb2-a770-d2cbad6653c3",
                        "channel": "app",
                        "levelname": "INFO",
                        "bareMetalId": 123,
                        "os": "centos",
                        "token": "556f5ea25f6af"
                    },
                    "_type": "logfile"
                },
        // ...
        }
    }

What is important to notice is that the keys token, bareMetalId and os in the monolog log line end up in Elastic Search as an indexable fields. From your php code you can add this extra information to your monolog messages by supplying an associative array as a second argument to the default monolog log functions:

    <?php

    $logger = $this->logger;
    $logger->info('The server was reinstalled', array('bareMetalId' => 123, 'os' => 'centos'));

Happy logging!

Chef server API integration with PHP

In this post I will show you a quick example of how you can integrate with the chef server api from php.

If you don’t know chef I recommend to have a look at https://www.chef.io. Chef is a configuration management tool, similar to ansible or puppet.

Chef turns infrastructure into code. With Chef, you can automate how you build, deploy, and manage your infrastructure.

At LeaseWeb our infrastructure that supports our business consists of many machines. For us it was a logical step to use a configuration management tool to manage all those servers and we chose chef. We also use chef to automate most of our (web) application deployments.

While our “chef managed” infrastructure was getting bigger, deploying fixes and features got easier and more frequent we needed something so our organisation is able to know what is being deployed and when.

Php is the main language we use here and we use Guzzle for quick and easy integration with rest api’s and web services.

Guzzle is a PHP HTTP client that makes it easy to send HTTP requests and trivial to integrate with web services.

Read more about guzzle here http://guzzle.readthedocs.org/.

We have created a plugin for Guzzle3 that implements the chef server authentication algorithm as described in their documentation https://docs.chef.io/auth.html

The plugin can be found on our github page https://github.com/LeaseWeb/chefauth-guzzle-plugin.

The plugin takes care of adding all the necessary http headers and signing the request to make a fully authenticated call to the chef server.

To start consuming the chef server rest api either checkout the source code with git or add the plugin as a dependency to your project using `composer`:

    php composer.phar require "leaseweb/chef-guzzle-plugin":"1.0.0"

Once you have created a user in chef the two things you need to get started is the client name of this user (in this example we assume my-dashboard) and the private key of this client (in this example we assume it is stored in my-dashboard.pem):

    <?php

    use Guzzle\Http\Client;
    use LeaseWeb\ChefGuzzle\Plugin\ChefAuth\ChefAuthPlugin;

    // Supply your client name and location of the private key.
    $chefAuthPlugin = new ChefAuthPlugin("my-dashboard", "my-dashboard.pem");

    // Create a new guzzle client
    $client = new Client('https://manage.opscode.com');
    $client->addSubscriber($chefAuthPlugin);

    // Now you can make calls to the chef server
    $response = $client->get('/organizations/my-organization/nodes')->send();

    $nodes = $response->json();

There is a ton of things you can do with the chef api, read more about it here https://docs.chef.io/api_chef_server.html

Hopefully this plugin will make it easier to integrate your chef’ed infrastructure in your company processes.

We are playing around with:

  • automatically generating release notes for our applications,
  • automatically update our issue tracking systems after a chef deployment
  • and many more.

Automatically provision your bare metal infrastructure

At LeaseWeb we are all about automating delivery processes. Be it for our virtual products or bare metal products. This post shows you one of the many things you can do with our API.

If you have a bare metal server at LeaseWeb I encourage you to login to our customer portal The LeaseWeb Self Service Center at https://secure.leaseweb.com and
In the API section you can manage your api keys for accessing the LeaseWeb API. To read more about what you can do with our API head over to the LeaseWeb Developer Portal

Recently we have published new api calls on our developer portal for customers to manage dhcp leases for their bare metal servers.

These api calls expose our internal dhcp infrastructure, that we use for automation, to our customers as a service.

    GET    /bareMetals/{bareMetalId}/leases                 # list all leases
    POST   /bareMetals/{bareMetalId}/leases                 # create a lease
    DELETE /bareMetals/{bareMetalId}/leases/{macAddress}    # delete a lease

Customers can use it to install operating systems which are not available in the LeaseWeb Self Service Center or if they would like to automatically provision their bare metal infrastructure.

When you use our api to create a dhcp lease you have the possibility to specify the dhcp option 67 Bootfile Name. We chainload the open source ipxe network boot firmware which has http support (read more about ipxe on their website http://ipxe.org/). This means that you can provide a valid http url for dhcp option 67 Bootfile Name that points to a pxe script containing instructions what the the boot loader should do next.

For example: let’s say you own the webserver at webserver.example.com where you have placed the following ipxe script at /boot.ipxe:

    $ curl http://webserver.example.com/boot.ipxe

    #!ipxe
    dhcp
    kernel http://webserver.example.com/archiso/boot/x86_64/vmlinuz archisobasedir=archiso archiso_http_srv=http://webserver.example.com/ ip=:::::eth0:dhcp
    initrd http://webserver.example.com/archiso/boot/x86_64/archiso.img
    boot

You can now create a dhcp lease for your bare metal server using our api:

    $ curl -H 'X-Lsw-Auth: my-api-key' -X POST https://api.leaseweb.com/v1/bareMetals/{bareMetalId}/leases -d bootFileName="http://webserver.example.com/boot.i

Obviously replace {bareMetalId} with the id of your bare metal server. To view the dhcp lease that we just created you can use this call:

    $ curl -H 'X-Lsw-Auth: my-api-key' https://api.leaseweb.com/v1/bareMetals/{bareMetalId}/leases
    
    {
        "_metadata": {
            "limit": 10, 
            "offset": 0, 
            "totalCount": 1
        }, 
        "leases": [
            {
                "ip": "203.0.113.1", 
                "mac": "AA:AA:AA:AA:AA:AA", 
                "options": [
                    // ...
                    {
                        "name": "Bootfile Name", 
                        "optionId": "67", 
                        "policyName": null, 
                        "type": "String", 
                        "userClass": "gPXE", 
                        "value": "http://webserver.example.com/boot.ipxe", 
                        "vendorClass": ""
                    }
                    // ...
                ], 
                "scope": "203.0.113.0"
            }
        ]
    }

Now you have to manually reboot your server or use our api to issue a power cycle:

    $ curl -H 'X-Lsw-Auth: my-api-key' -X POST https://api.leaseweb.com/v1/bareMetals/{bareMetalId}/reboot

The server will reboot, ask for dhcp lease and eventually read the instructions provided by you in /boot.ipxe which in this example is downloading a kernel and the archlinux live cd which are both served from your web server at `webserver.example.com`.

You should be careful and not forget to remove a dhcp lease when you are done. Otherwise during the next reboot it will boot from the network again.

    $ curl -H 'X-Lsw-Auth: my-api-key' -X DELETE https://api.leaseweb.com/v1/bareMetals/{bareMetalId}/leases/AA:AA:AA:AA:AA:AA

We automatically remove dhcp leases after 24 hours .

This service allows our customers to implement creative ideas that can automate their bare metal infrastructure.

Example: install arch linux over ssh without kvm

To continue the example I used this service to boot my modified version of the archlinux live cd which includes and starts openssh at boot and includes my public ssh keys. I use this trick to be able to manually install an operating system which is not available through the LeaseWeb Self Service Center.

I don’t need to contact technical support or have kvm on my server. Everything is done remotely over ssh. The modified live image is published on github here https://github.com/nrocco/archiso-sshd.

Clone the repository from github:

    $ git clone https://github.com/nrocco/archiso-sshd.git
    $ cd archiso-sshd

Add your ssh keys to authorized_keys of the root user:

    $ vim airootfs/root/.ssh/authorized_keys

Now build the image (you need to have the archiso package installed).

    $ make build

This might take a while. When done, copy the kernel, initrmfs and other generated files to the document root of your http server:

    $ cp -r work/iso/arch /var/www

Your document root might look like this now:

    $ find /var/www -type f
    /var/www/boot.ipxe
    /var/www/archiso/pkglist.x86_64.txt
    /var/www/archiso/x86_64/airootfs.md5
    /var/www/archiso/x86_64/airootfs.sfs
    /var/www/archiso/boot/x86_64/archiso.img
    /var/www/archiso/boot/x86_64/vmlinuz

That’s it. Now you boot from the network using our service.

Refer to airootfs/root/customize_airootfs.sh and airootfs/root/.ssh/authorized_keys for the specific customatizations.

What can you do with it?

This example is just the tip of the iceberg of possibilities. Let us know your ideas and use cases.

You might use it to boot into your own live image that does an automated installation of the operating system and kicks off the provisioning tool of your choice (chef, ansible, puppet) so your bare metal servers joins your infrastructure that helps supporting your business.

All fully automated.

A PHP SOAP client for the command line

Here at LeaseWeb, we work a lot with SOAP web-services to integrate our internal applications with each other. Especially during development and testing of our applications as we need the ability to practice with SOAP API’s.

If you are into fancy GUI applications then maybe SoapUI is something you could use to familiarise yourself with the API you are integrating with.

If you like simplicity and the command line, or dislike heavy GUI applications you can give php soap client a try.

php soap client is a command line application written in PHP and packed using the popular .phar format. Once installed you can start exploring soap web-services.

There are two ways of installing the client. To get the latest stable version download soap_client.phar from here or do:

$ curl -sS http://leaseweb.github.io/php-soap-client/installer | php

This will download the phar file to the current working directory and make it executable so you can use start using it right away by invoking:

$ ./soap_client

To install the latest master version you can get the source code directly from GitHub, package your own .phar file and install it — using GNU Make.
In order to be able to create the .phar file you need to have composer installed. To read more about composer refer to their excellent documentation here.

# Install php soap client
$ git clone https://github.com/LeaseWeb/php-soap-client.git
$ cd php-soap-client
$ composer.phar install
$ make
$ sudo make install

If you are getting a Failed to compile phar exception while running make you need to set phar.readonly = Off in your php.ini. On a development machine this is fine to do but please be ware of the security risks when setting phar.readonly to Off.

The above make install command will install the soap_client application to /usr/local/bin and make it executable so you can easily call it like this:

$ soap_client
php-soap-client version 2.1.3

Usage:
  [options] command [arguments]

Options:
  ...

Available commands:
  call           Call the remote service with the `method` specified and output the reponse to stdout.
  help           Displays help for a command
  list           Lists commands
  list-methods   Get a list of available methods to call on the remote.
  request        Generate an xml formatted SOAP request for the given method and output to stdout.
  wsdl           Get the WSDL of a soap service.

From this point onwards we assume you have installed the soap_client.phar on your system in /usr/local/bin/soap_client and that the directory /urs/local/bin is in your $PATH.

Lets say we would like to see what methods are available on the remote service http://www.webservicex.net/ConvertTemperature.asmx. We could issue the following command:

$ soap_client --endpoint='http://www.webservicex.net/ConvertTemperature.asmx?WSDL' list-methods

Which will output the following:

ConvertTemp

If you run the above command with the -vvv option you will get more verbose output.
In this case the only available method is ConvertTemp. Let’s see how a SOAP XML request looks like for this method:

$ soap_client --endpoint='http://www.webservicex.net/ConvertTemperature.asmx?WSDL' request ConvertTemp

<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns1="http://www.webserviceX.NET/">
  <SOAP-ENV:Body>
    <ns1:ConvertTemp>
      <ns1:Temperature>0</ns1:Temperature>
      <ns1:FromUnit></ns1:FromUnit>
      <ns1:ToUnit></ns1:ToUnit>
    </ns1:ConvertTemp>
  </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

If you want to make a SOAP request to the ConvertTemp method on the remote service use the call sub command:

$ soap_client --endpoint='http://www.webservicex.net/ConvertTemperature.asmx?WSDL' call --editor ConvertTemp

Notice the --editor option after the call sub command. If you use the --editor flag soap_client will open up the editor specified in your environment variable $EDITOR so you are able to modify the request XML before sending it.

If you issue the same request multiple times, you could save a soap request as a local XML file and pass it to /dev/stdin of the soap_client call command:

# Get the request xml and store it locally
$ soap_client --endpoint='http://www.webservicex.net/ConvertTemperature.asmx?WSDL' request ConvertTemp &gt; my_sample_request.xml

# Now edit my_sample_request.xml

# Now you can call the ConvertTemp method with this pre-prepared request
$ soap_client --endpoint='http://www.webservicex.net/ConvertTemperature.asmx?WSDL' call ConvertTemp < my_sample_request.xml

Since you will be repeating soap_client commands frequently in a short time while exploring a remote web service you can save yourself some time by setting an environment variable SOAPCLIENT_ENDPOINT that contains the URL to the WSDL. When this environment variable is set you can omit the --endpoint command line option. Let’s do this now and call the ConvertTemp method:

$ export SOAPCLIENT_ENDPOINT='http://www.webservicex.net/ConvertTemperature.asmx?WSDL'
$ soap_client call ConvertTemp &lt; my_sample_request.xml

I wanted to know how much 107.6 degrees Fahrenheit is in Celsius, so my my_sample_request.xml contains:

$ cat my_sample_request.xml

<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns1="http://www.webserviceX.NET/">
  <SOAP-ENV:Body>
    <ns1:ConvertTemp>
      <ns1:Temperature>107.6</ns1:Temperature>
      <ns1:FromUnit>degreeFahrenheit</ns1:FromUnit>
      <ns1:ToUnit>degreeCelsius</ns1:ToUnit>
    </ns1:ConvertTemp>
  </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

And the result:

$ soap_client call ConvertTemp < my_sample_request.xml

stdClass Object
(
    [ConvertTempResult] => 42
)

The answer is 42.

If you rather see the responses in XML format you can use the --xml command line option:

$ soap_client call --xml ConvertTemp < my_sample_request.xml

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <soap:Body>
    <ConvertTempResponse xmlns="http://www.webserviceX.NET/">
      <ConvertTempResult>42</ConvertTempResult>
    </ConvertTempResponse>
  </soap:Body>
</soap:Envelope>

This tutorial should give you enough information to get started with exploring, testing and/or developing SOAP API’s.
In a future blog post, I will continue the topic of the php soap client. We are currently working on packing the .phar archive for the web.

php-soap-client for the web
php soap client for the web

So you would be able to drop the soap_client.phar somewhere in your apache DocumentRoot and explore SOAP services from your browser.