How to create JWT authentication with API Platform

As the title suggests, in this blog we will together create a simple JWT authentication using API Platform and LexikJWTAuthenticationBundle. And of course, also using our lovely Doctrine User Provider.

Motivation

There too many tutorials online about symfony with JWT, and also some about the API Platform. But most of them are too short or missing certain things, which is unhelpful. It can also be confusing for developers when the tutorials don’t say what concepts you need to know first.

I hope this blog will be different – if you have any concerns, updates, questions, then drop a comment underneath and i’ll try to answer all of them.

Requirements

  • PHP >= 7.0 knowledge
  • Symfony knowledge (Autowiring, Dependency Injection)
  • Docker knowledge
  • REST APIs knowledge
  • PostgreSQL knowledge
  • Ubuntu or MacOs (Sorry Windows users :))

API Platform installation

The best way for me to install this is by using the git repository, or downloading the API Platform as .zip file from Github.

$ git clone https://github.com/api-platform/api-platform.git apiplatform-user-auth

$ cd apiplatform-user-auth

Now, first of all, the whole API Platform runs on specific ports, so you need to make sure that this is free and nothing is listening to it.

Finding the ports

You can find them in the docker-compose.yml file in the project root directory. They always be like [80, 81, 8080, 8081, 3000, 5432, 1337, 8443, 8444, 443, 444]

How to show this

Run this command

$ sudo lsof -nP | grep LISTEN

Kill any listening processes on any of the above ports.

$ sudo kill -9 $PROCESS_NUMBER

Installation:

  • Pull the required packages and everything needed.
docker-compose pull
  • Bring the application up and running.
$ docker-compose up -d
  • You may face some issue here, so it’s best to bring all containers down and run the command again like this.
$ docker-compose down
$ COMPOSE_HTTP_TIMEOUT=120 docker-compose up -d

Now the application should be running and everything should be in place:

$ docker ps

CONTAINER ID        IMAGE                            COMMAND                  CREATED              STATUS              PORTS                                                                    NAMES
6389d8efb6a0        apiplatform-user-auth_h2-proxy   "nginx -g 'daemon of…"   About a minute ago   Up About a minute   0.0.0.0:443-444->443-444/tcp, 80/tcp, 0.0.0.0:8443-8444->8443-8444/tcp   apiplatform-user-auth_h2-proxy_1_a012bc894b6c
a12ff2759ca4        quay.io/api-platform/varnish     "docker-varnish-entr…"   2 minutes ago        Up 2 minutes        0.0.0.0:8081->80/tcp                                                     apiplatform-user-auth_cache-proxy_1_32d747ba8877
6c1d29d1cbdd        quay.io/api-platform/nginx       "nginx -g 'daemon of…"   2 minutes ago        Up 2 minutes        0.0.0.0:8080->80/tcp                                                     apiplatform-user-auth_api_1_725cd9549081
62f69838dacb        quay.io/api-platform/php         "docker-entrypoint p…"   2 minutes ago        Up 2 minutes        9000/tcp                                                                 apiplatform-user-auth_php_1_cf09d32c3120
381384222af5        dunglas/mercure                  "./mercure"              2 minutes ago        Up 2 minutes        443/tcp, 0.0.0.0:1337->80/tcp                                            apiplatform-user-auth_mercure_1_54363c253a34
783565efb2eb        postgres:10-alpine               "docker-entrypoint.s…"   2 minutes ago        Up 2 minutes        0.0.0.0:5432->5432/tcp                                                   apiplatform-user-auth_db_1_8da243ca2865
1bc8e386bf02        quay.io/api-platform/client      "/bin/sh -c 'yarn st…"   2 minutes ago        Up About a minute   0.0.0.0:80->3000/tcp                                                     apiplatform-user-auth_client_1_1c413b4e4a5e
c22bef7a0b3f        quay.io/api-platform/admin       "/bin/sh -c 'yarn st…"   2 minutes ago        Up About a minute   0.0.0.0:81->3000/tcp                                                     apiplatform-user-auth_admin_1_cfecc5c6b442

Now, if you go to localhost:8080 you will see there some simple APIs listed, it is the example entity that comes with the project.

Create the User entity based on Doctrine User Provider

Install the doctrine maker package to help us make this quickly 🙂

$ docker-compose exec php composer require doctrine maker

Create your User entity

$ docker-compose exec php bin/console make:user

 The name of the security user class (e.g. User) [User]:
 > Users

 Do you want to store user data in the database (via Doctrine)? (yes/no) [yes]:
 >

 Enter a property name that will be the unique "display" name for the user (e.g. email, username, uuid) [email]:
 > email

 Will this app need to hash/check user passwords? Choose No if passwords are not needed or will be checked/hashed by some other system (e.g. a single sign-on server).

 Does this app need to hash/check user passwords? (yes/no) [yes]:
 >

The newer Argon2i password hasher requires PHP 7.2, libsodium or paragonie/sodium_compat. Your system DOES support this algorithm.
You should use Argon2i unless your production system will not support it.

 Use Argon2i as your password hasher (bcrypt will be used otherwise)? (yes/no) [yes]:
 >

 created: src/Entity/Users.php
 created: src/Repository/UsersRepository.php
 updated: src/Entity/Users.php
 updated: config/packages/security.yaml


  Success!


 Next Steps:
   - Review your new App\Entity\Users class.
   - Use make:entity to add more fields to your Users entity and then run make:migration.
   - Create a way to authenticate! See https://symfony.com/doc/current/security.html

If you go now to “api/src/Entity” you will find your entity there. If you scroll down a little bit to the getEmail & getPassword functions you will see something like this, which means the two properties will be used as the User identifier in the authentication. (I will not use the ROLES in this example as it is a simple one).

# api/src/Entity/Users.php

/**
* @see UserInterface
*/

As you know the latest versions of symfony using the autowiring feature so you can see that this entity is already wired and injected with teh repository called “api/src/Repository/UsersReporitory”.

# api/src/Entity/Users.php

/**
 * @ORM\Entity(repositoryClass="App\Repository\UsersRepository")
 */
class Users implements UserInterface
{
    ...
}

You can see clearly in this repository some per-implemented functions like findbyId(), but now let us create another function that helps us to create a new user.

  • To add a user into the Db, you will need to define an entity manager like the following:
# api/src/Repository/UsersRepository.php

class UsersRepository extends ServiceEntityRepository
{
  /** EntityManager $manager */
  private $manager;
....
}

and initialize it in the constructor like so:

# api/src/Repository/UsersRepository.php

/**
* UsersRepository constructor.
* @param RegistryInterface $registry
*/
public function __construct(RegistryInterface $registry)
{
  parent::__construct($registry, Users::class);

  $this->manager = $registry->getEntityManager();
}
  • Now, let us create our function:
# api/src/Repository/UsersRepository.php

/**
 * Create a new user
 * @param $data
 * @return Users
 * @throws \Doctrine\ORM\ORMException
 * @throws \Doctrine\ORM\OptimisticLockException
*/
public function createNewUser($data)
{
    $user = new Users();
    $user->setEmail($data['email'])
        ->setPassword($data['password']);

    $this->manager->persist($user);
    $this->manager->flush();

    return $user;
}
  • Let us create our controller to consume that repository. We can call it “AuthController”.
$ docker-compose exec php bin/console make:controller

 Choose a name for your controller class (e.g. TinyJellybeanController):
 > AuthController

 created: src/Controller/AuthController.php
 created: templates/auth/index.html.twig


  Success!


 Next: Open your new controller class and add some pages!

Now, let’s consume this createNewUser function. If you see your controller, you will find it only contains the index function, but we need to create another one will call it “register”.

  • We need the UsersRepository so should create the object first.
# api/src/Controller/AuthController.php

use App\Repository\UsersRepository;

class AuthController extends AbstractController
{
    /** @var UsersRepository $userRepository */
    private $usersRepository;

    /**
     * AuthController Constructor
     *
     * @param UsersRepository $usersRepository
     */
    public function __construct(UsersRepository $usersRepository)
    {
        $this->usersRepository = $usersRepository;
    }
    .......
}
  • Now, we need to make this controller know about the User repository, so we will inject it as a service.
# api/config/services.yaml

services:
    ......
  # Repositories
  app.user.repository:
      class: App\Repository\UsersRepository
      arguments:
          - Symfony\Bridge\Doctrine\RegistryInterface
  
  # Controllers
  app.auth.controller:
      class: App\Controller\AuthController
      arguments:
          - '@app.user.repository'
  • Now, it is time to implement our new endpoint to register (create) a new account.
# api/src/Controller/AuthController.php

# Import those
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpFoundation\Response;

# Then add this to the class
/**
 * Register new user
 * @param Request $request
 *
 * @return Response
 */
public function register(Request $request)
{
    $newUserData['email']    = $request->get('email');
    $newUserData['password'] = $request->get('password');

    $user = $this->usersRepository->createNewUser($newUserData);

    return new Response(sprintf('User %s successfully created', $user->getUsername()));
}
  • Now, we need to make the framework know about this new endpoint by adding it to our routes file.
# src/config/routes.yaml

# Register api
register:
    path: /register
    controller: App\Controller\AuthController::register
    methods: ['POST']

Testing this new API:

  • Make the migration and update the DB first:
$ docker-compose exec php bin/console make:migration

$ docker-compose exec php bin/console doctrine:migrations:migrate

  WARNING! You are about to execute a database migration that could result in schema changes and data loss. Are you sure you wish to continue? (y/n) y

Now, from Postman or any other client you use. Here am using CURL.

$ curl -X POST -H "Content-Type: application/json" "http://localhost:8080/register?email=test1@mail.com&password=test1"
User test1@mail.com successfully created

To see this data in the DB:

$ docker-compose exec db psql -U api-platform api
psql (10.8)
Type "help" for help.

$ api=# select * from users;
 id |     email      | roles | password
----+----------------+-------+----------
  6 | test1@mail.com | []    | test1
(1 row)

Oooooh, wow the password is not encrypted what should we do!!!

So, as i said before this project is built on Symfony, that is why i said you need to have knowledge about symfony. So we will use the Password encoder class.

# api/src/Repository/UsersRepository.php

use Symfony\Component\Security\Core\Encoder\UserPasswordEncoderInterface;

class UsersRepository extends ServiceEntityRepository
{
    .......

  /** UserPasswordEncoderInterface $encoder */
  private $encoder;
    
  /**
   * UserRepository constructor.
   * @param RegistryInterface $registry
   * @param UserPasswordEncoderInterface $encoder
   */
  public function __construct(RegistryInterface $registry, UserPasswordEncoderInterface $encoder)
  {
      parent::__construct($registry, Users::class);

      $this->manager = $registry->getEntityManager();
      $this->encoder = $encoder;
  }
}
  • As always we need to inject it to the repository:
# api/config/services.yaml

services:
  .......
  # Repositories
  app.user.repository:
      class: App\Repository\UsersRepository
      arguments:
          - Symfony\Bridge\Doctrine\RegistryInterface
          - Symfony\Component\Security\Core\Encoder\UserPasswordEncoderInterface

Then update the create user function:

# api/src/Repository/UsersRepository.php

public function createNewUser($data)
{
    $user = new Users();
    $user->setEmail($data['email'])
        ->setPassword($this->encoder->encodePassword($user, $data['password']));
    .......
}
  • Now, try the register call again, remember to try with different email as we defined the email as Unique:
$ curl -X POST -H "Content-Type: application/json" "http://localhost:8080/register?email=test2@mail.com&password=test2"
User test2@mail.com successfully created
  • check the DB now again:
$ api=# select * from users;
 id |     email      | roles |                                            password
----+----------------+-------+-------------------------------------------------------------------------------------------------
  6 | test1@mail.com | []    | test1
  7 | test2@mail.com | []    | $argon2i$v=19$m=1024,t=2,p=2$VW9tYXEzZHp5U0RMSE5ydA$bo+V1X6rgYZ4ebN/bs1cpz+sf+DQdx3Duu3hvFUII8M
(2 rows)

Install LexikJWTAuthenticationBundle

  • Install the bundle and generate the secrets:
$ docker-compose exec php composer require jwt-auth

Create our authentication

  • (Additional) Before anything if you tried this call, for now, you will get this result:
$ curl -X GET -H "Content-Type: application/json" "http://localhost:8080/greetings"
{
    "@context": "/contexts/Greeting",
    "@id": "/greetings",
    "@type": "hydra:Collection",
    "hydra:member": [],
    "hydra:totalItems": 0
}
  • Let’s continue for now, create a new and simple endpoint that we will use in our testing. Now I will call it “/api”.
# api/src/Controller/AuthController.php

/**
* api route redirects
* @return Response
*/
public function api()
{
    return new Response(sprintf("Logged in as %s", $this->getUser()->getUsername()));
}
  • Add it to our Routes
# api/config/routes.yaml

api:
    path: /api
    controller: App\Controller\AuthController::api
    methods: ['POST']

Now, we need to make some configurations in our security config file:

  • This is our provider to our authentication or anything related to users in the application. It is already predefined, if you want change the user provider you can do it here.
# api/config/packages/security.yaml

app_user_provider:
    entity:
        class: App\Entity\Users
        property: email
  • Let’s make some configs for our “/register” API as we want this API to be public for anyone:
# api/config/packages/security

register:
    pattern:  ^/register
    stateless: true
    anonymous: true
  • Now, let us assume that we need everything generated by the API Platform to not work without the JWT token, meaning without authenticated users the API shouldn’t return anything. So I will update the “main” part configs to be like this:
# api/config/packages/security.yaml

main:
    anonymous: false
    stateless: true
    provider: app_user_provider
    json_login:
        check_path: /login
        username_path: email
        password_path: password
        success_handler: lexik_jwt_authentication.handler.authentication_success
        failure_handler: lexik_jwt_authentication.handler.authentication_failure
    guard:
        authenticators:
            - lexik_jwt_authentication.jwt_token_authenticator
  • Also, add some configs for our simple endpoint /api.
# api/config/packages/security.yaml

api:
    pattern: ^/api
    stateless: true
    anonymous: false
    provider: app_user_provider
    guard:
        authenticators:
            - lexik_jwt_authentication.jwt_token_authenticator
  • As you see in the above configs, we set the anonymous to false as we don’t want anyone to access these two APIs. Also we are telling the framework that the provider for you is the user provider that we defined before. At the end we are telling it which authenticator you will use and the authentication success/faliure messages.
  • Now, if you try the call, try it in the Additional part for the /greetings api:
$ curl -X GET -H "Content-Type: application/json" "http://localhost:8080/greetings"
  {
      "code": 401,
      "message": "JWT Token not found"
  }

It is the same with our simple endpoint /api that we created:

$ curl -X POST -H "Content-Type: application/json" "http://localhost:8080/api" 
  {
    "code": 401,
    "message": "JWT Token not found"
  }
  • As you can see it asks you to login :D, there is no JWT token specified so we will create a very simple API that is used by the lexik jwt to authenticate the users, and generate their tokens. Remember that the login check path should be the same as the check_path under json_login in the security file:
# api/config/packages/security.yaml
....
json_login:
        check_path: /login

# api/config/routes.yaml

# Login check to log the user and generate JWT token
api_login_check:
      path: /login
      methods: ['POST']
  • Now, let’s try it out and see if it will generate a token for us!
$ curl -X POST -H "Content-Type: application/json" http://localhost:8080/login -d '{"email":"test2@mail.com","password":"test2"}'
  {"token":"eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJpYXQiOjE1NTg2OTg4MTIsImV4cCI6MTU1ODcwMjQxMiwicm9sZXMiOlsiUk9MRV9VU0VSIl0sInVzZXJuYW1lIjoidGVzdDJAbWFpbC5jb20ifQ.nzd5FVhcyrfjYyN8jRgYFp3VOB2QytnPPRGNyp4ZtfLx6IRwg0TWZJPu5OFtOKPkdLO8DQAr_4Fpq_G6oPjzoxmGOASNuRoQonik9FCCq6oAIW3k5utzQecXDVE_ImnfgByc6WYW6a-aWLnsq1qtvxy274ojqdR0rWLePwSWX5K5-t08zDBgavO_87dVpYd0DLwhHIS7F10lNscET7bfWS-ioPDTv-G74OvkcpbcjgwHhXlO7TYubnrES-FsvAw7kezQe4BPxdbXr1w-XBZuqTNEU4MyrBuadSLgjoe_gievNBtkVhKErIkEQZVjeJIQ4xaKaxwmPxZcP9jYkE47myRdbMsL9XHSd0XmGq0bPuGjOJ2KLTmUb5oeuRnY-e9Q_V9BbouEGw0sjw2meo6Jot2MZyv5ZnLci_GwpRtWqmV7ZLw5jNyiLDFXR1rz70NcJh7EXqu9o4nno3oc68zokfDQvGkJJJZMtBrLCK5pKGMh0a1elIz41LRLZvpLYCrOZ2f4wCkGRD_U92iILD6w8EdVWGoO1wTn5Z2k8-GS1-QH9f-4KkOpaYGPCwwdrY7yioSt2oVbEj2FOb1jULteeP_Cpu44HyJktPLPW_wrN2OtZlUFr4Vz_owDSIvNESYk1JBQ_Fjlv9QGmUs9itzaDExjfB4QYoGkvpfNymtw2PI"}

As you see it created a JWT token for me, so I can use it to call any API in the application. If it shows some exception like Unable to generate token for the specified configurationsplease check this step here. First, open you .envfile. We will need the JWT_PASSPHRASE so keep it opened:

$ mkdir -p api/config/jwt
$ openssl genrsa -out api/config/jwt/private.pem -aes256 4096 # this will ask you for the JWT_PASSPHRASE
$ openssl rsa -pubout -in api/config/jwt/private.pem -out api/config/jwt/public.pem # will confirm the JWT_PASSPHRASE again
  • Let’s try to call /api or /greetings endpoints with this token now:
$ curl -X GET -H "Content-Type: application/json" -H "Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJpYXQiOjE1NTg2OTg4MTIsImV4cCI6MTU1ODcwMjQxMiwicm9sZXMiOlsiUk9MRV9VU0VSIl0sInVzZXJuYW1lIjoidGVzdDJAbWFpbC5jb20ifQ.nzd5FVhcyrfjYyN8jRgYFp3VOB2QytnPPRGNyp4ZtfLx6IRwg0TWZJPu5OFtOKPkdLO8DQAr_4Fpq_G6oPjzoxmGOASNuRoQonik9FCCq6oAIW3k5utzQecXDVE_ImnfgByc6WYW6a-aWLnsq1qtvxy274ojqdR0rWLePwSWX5K5-t08zDBgavO_87dVpYd0DLwhHIS7F10lNscET7bfWS-ioPDTv-G74OvkcpbcjgwHhXlO7TYubnrES-FsvAw7kezQe4BPxdbXr1w-XBZuqTNEU4MyrBuadSLgjoe_gievNBtkVhKErIkEQZVjeJIQ4xaKaxwmPxZcP9jYkE47myRdbMsL9XHSd0XmGq0bPuGjOJ2KLTmUb5oeuRnY-e9Q_V9BbouEGw0sjw2meo6Jot2MZyv5ZnLci_GwpRtWqmV7ZLw5jNyiLDFXR1rz70NcJh7EXqu9o4nno3oc68zokfDQvGkJJJZMtBrLCK5pKGMh0a1elIz41LRLZvpLYCrOZ2f4wCkGRD_U92iILD6w8EdVWGoO1wTn5Z2k8-GS1-QH9f-4KkOpaYGPCwwdrY7yioSt2oVbEj2FOb1jULteeP_Cpu44HyJktPLPW_wrN2OtZlUFr4Vz_owDSIvNESYk1JBQ_Fjlv9QGmUs9itzaDExjfB4QYoGkvpfNymtw2PI" "http://localhost:8080/greetings"
{
    "@context": "/contexts/Greeting",
    "@id": "/greetings",
    "@type": "hydra:Collection",
    "hydra:member": [],
    "hydra:totalItems": 0
}

## Before
$ curl -X GET -H "Content-Type: application/json" "http://localhost:8080/greetings"
  {
      "code": 401,
      "message": "JWT Token not found"
  }
  • What about the /api endpoint, let’s try it out:
$ curl -X POST -H "Content-Type: application/json" -H "Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJpYXQiOjE1NTg2OTg4MTIsImV4cCI6MTU1ODcwMjQxMiwicm9sZXMiOlsiUk9MRV9VU0VSIl0sInVzZXJuYW1lIjoidGVzdDJAbWFpbC5jb20ifQ.nzd5FVhcyrfjYyN8jRgYFp3VOB2QytnPPRGNyp4ZtfLx6IRwg0TWZJPu5OFtOKPkdLO8DQAr_4Fpq_G6oPjzoxmGOASNuRoQonik9FCCq6oAIW3k5utzQecXDVE_ImnfgByc6WYW6a-aWLnsq1qtvxy274ojqdR0rWLePwSWX5K5-t08zDBgavO_87dVpYd0DLwhHIS7F10lNscET7bfWS-ioPDTv-G74OvkcpbcjgwHhXlO7TYubnrES-FsvAw7kezQe4BPxdbXr1w-XBZuqTNEU4MyrBuadSLgjoe_gievNBtkVhKErIkEQZVjeJIQ4xaKaxwmPxZcP9jYkE47myRdbMsL9XHSd0XmGq0bPuGjOJ2KLTmUb5oeuRnY-e9Q_V9BbouEGw0sjw2meo6Jot2MZyv5ZnLci_GwpRtWqmV7ZLw5jNyiLDFXR1rz70NcJh7EXqu9o4nno3oc68zokfDQvGkJJJZMtBrLCK5pKGMh0a1elIz41LRLZvpLYCrOZ2f4wCkGRD_U92iILD6w8EdVWGoO1wTn5Z2k8-GS1-QH9f-4KkOpaYGPCwwdrY7yioSt2oVbEj2FOb1jULteeP_Cpu44HyJktPLPW_wrN2OtZlUFr4Vz_owDSIvNESYk1JBQ_Fjlv9QGmUs9itzaDExjfB4QYoGkvpfNymtw2PI" "http://localhost:8080/api"
Logged in as test2@mail.com

## Before
$ curl -X POST -H "Content-Type: application/json" "http://localhost:8080/api" 
  {
    "code": 401,
    "message": "JWT Token not found"
  }

As you can see from the JWT token, you know exactly who is logged in, and you can improve this by implementing additional User properties like isActive or userRoles…etc.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Thank you for reading this tutorial, I hope that you learned something new!

If you have any questions please don’t hesitate to ask, or any feedback will be so useful.

You can find this whole tutorial and the example here on GitHub.

Share

Memcache Bundle for Symfony updated!

memcache_v2

We have released version 2 of our Symfony Memcache Bundle. The 3 biggest improvements are:

  • Added Windows support
  • PHP 5.4+ without notices and/or warnings
  • High availability support with redundancy configuration

This actually was a major refactoring. We switched from the PECL “memcached” to the PECL “memcache” (without ‘d’) extension. Only version 3 (which is still in beta) of this extension is supported currently, but this version is quite stable actually. Most of the features of the bundle have not been altered. The Memcache(Pool) object is deliberately not abstracted. This is done to allow for unrestricted access to the underlying implementation. The flags parameter is now supported and this is the third argument on a “set” instruction. You may have to alter your code for this, which may be annoying, but we promise to keep the API stable in this major version that may again last for a few years.

Some articles we have written about the bundle are:

Let us know your experience with this new version.

Download LswMemcacheBundle from Github

Share

Heka monolog decoder

This post is about how to use heka to give your symfony 2 application logs the care they deserve.

Application logs are very important for the quality of the product or service you are offering.

They help you find out what went wrong so you can explain and fix a bug that was reported recently. Or maybe to gather statistics to see how often a certain feature is used. For example how many bare metal reinstallation requests were issued last month and how many of those failed. Valuable information that you could use to decide what feature you are going to work on next.

At LeaseWeb we use quite a lot of php and Seldaek’s monolog is our logging library of choice. Recently we open sourced a heka decoder on github here. For you who do not know heka yet, check out their documentation.

Heka is an open source stream processing software system developed by Mozilla. Heka is a “Swiss Army Knife” type tool for data processing, useful for a wide variety of different tasks, such as …

Heka runs as a system daemon just like logstash or fluentd. Heka is written in go and comes with an easy to use plugin system based on lua. It has almost no dependencies and is lightweight. James Turnbull has written a nice article on how to get started with heka.

send symfony 2 application logs to elastic search

How better to explain with an example.

Lets say you have a symfony 2 application and you want the application logs to be sent to an Elastic Search platform.

On a debian based os you can download one of the heka debian packages from github.

    $ wget https://github.com/mozilla-services/heka/releases/download/v0.9.2/heka_0.9.2_i386.deb
    $ sudo dpkg -i heka_0.9.2_i386.deb

To configure heka you are required to edit the configuration file located at /etc/hekad.toml.

    $ vim /etc/hekad.toml

Take your time to read the excellent documentation on heka. There are many ways of using heka but we will use it as a forwarder:

  1. Define an input, where messages come from.
  2. Tell heka how it should decode the monolog log line into a heka message
  3. Tell heka how to encode the message so Elastic Search will understand it
  4. Define an output, where should the messages be sent to.

First we define the input:

    [Symfony2MonologFileInput]
    type = "LogstreamerInput"
    log_directory = "/var/www/app/logs"
    file_match = 'prod\.log'
    decoder = "Symfony2MonologDecoder"

Adjust `log_directory` and `file_match` according to your setup. As you can see we alread told heka to use the `Symfony2MonologDecoder` to we will define that one next:

    [Symfony2MonologDecoder]
    type = "SandboxDecoder"
    filename = "/etc/symfony2_decoder.lua"

Change the `filename` with the path where you placed the lua script on your system.

Now we have defined the input we can tell heka where to output messages to:

    [ESJsonEncoder]
    index = "%{Hostname}"
    es_index_from_timestamp = true
    type_name = "%{Type}"

    [ElasticSearchOutput]
    message_matcher = "TRUE"
    server = "http://192.168.100.1:9200"
    flush_interval = 5000
    flush_count = 10
    encoder = "ESJsonEncoder"

In the above example we assume that your Elastic Search server is running at 192.168.100.1.

And thats it.

A simple log line in app/logs/prod.log:

    [2015-06-03 22:08:02] app.INFO: Dit is een test {"bareMetalId":123,"os":"centos"} {"token":"556f5ea25f6af"}

Is now sent to Elastic Search. You should now be able to query your Elastic Search for log messages, assuming the hostname of your server running symfony 2 is myapi:

    $ curl http://192.168.100.1:9200/myapi/_search | python -mjson.tool
    {
        "_shards": {
            "failed": 0,
            "successful": 5,
            "total": 5
        },
        "hits": {
            "hits": [
                {
                    "_id": "ZIV7ryZrQRmXDiB6thY_yQ",
                    "_index": "myapi",
                    "_score": 1.0,
                    "_source": {
                        "EnvVersion": "",
                        "Hostname": "myapi",
                        "Logger": "Symfony2MonologFileInput",
                        "Payload": "Dit is een test",
                        "Pid": 0,
                        "Severity": 7,
                        "Timestamp": "2015-06-03T20:08:02.000Z",
                        "Type": "logfile",
                        "Uuid": "344e7cae-6ab7-4fb2-a770-d2cbad6653c3",
                        "channel": "app",
                        "levelname": "INFO",
                        "bareMetalId": 123,
                        "os": "centos",
                        "token": "556f5ea25f6af"
                    },
                    "_type": "logfile"
                },
        // ...
        }
    }

What is important to notice is that the keys token, bareMetalId and os in the monolog log line end up in Elastic Search as an indexable fields. From your php code you can add this extra information to your monolog messages by supplying an associative array as a second argument to the default monolog log functions:

    <?php

    $logger = $this->logger;
    $logger->info('The server was reinstalled', array('bareMetalId' => 123, 'os' => 'centos'));

Happy logging!

Share

Symfony on HHVM 3 and Nginx 1.4 vs PHP 5.5 and Apache 2.4

symfony_hhvm_nginx

Installing Symfony

From symfony.com/download we get the latest (2.4.4) version of Symfony. I have unpacked it and put it in the directory “/home/maurits/public_html”. In “app/AppKernel.php” I moved the “AcmeDemoBundle” to the production section and in “routing.yml” I added the “_acme_demo” route that was originally in “routing_dev.yml”.

Test environment

I tested on my i5-2300 CPU with 16GB of RAM and an SSD. To run a benchmark, I installed on my Ubuntu 14.04 both Apache 2.4 with PHP 5.5 and Nginx 1.4 with HHVM 3. I used Apache Bench (ab) to test the “/demo” path on both web servers. In Apache, I disabled XDebug and enabled Zend OPcache.

Install and configure HHVM 3 with Nginx 1.4 for Symfony

For HHVM, we find pre-built (64-bit) packages listed on Github. This is how you install them on Ubuntu 14.04:

wget -O - http://dl.hhvm.com/conf/hhvm.gpg.key | sudo apt-key add -
echo deb http://dl.hhvm.com/ubuntu trusty main | sudo tee /etc/apt/sources.list.d/hhvm.list
sudo apt-get update
sudo apt-get install hhvm

First we install Nginx from the Ubuntu 14.04 repository using:

sudo apt-get install nginx

Now we configure Nginx using the “normal” FastCGI configuration:

server {
    listen             8080;
    server_name        sf2testproject.dev;

    root /home/maurits/public_html/web;

    location / {
        # try to serve file directly, fallback to rewrite
        try_files $uri @rewriteapp;
    }

    location @rewriteapp {
        # rewrite all to app.php
        rewrite ^(.*)$ /app.php/$1 last;
    }

    location ~ ^/(app|app_dev|config)\.php(/|$) {
        fastcgi_pass 127.0.0.1:9000;
        fastcgi_split_path_info ^(.+\.php)(/.*)$;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param HTTPS off;
    }
}

Single request

When doing a single request and analyzing the response times using the Firebug “Net” panel there is no noticeable difference. This is probably because the threads do not have to compete for CPU. So let’s skip this and do some load testing.

Apache Bench results (Symfony 2.4 / Apache 2.4 / PHP 5.5)

maurits@nuc:~$ ab -c 10 -n 2000 http://sf2testproject.dev/demo/
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking sf2testproject.dev (be patient)
Completed 200 requests
Completed 400 requests
Completed 600 requests
Completed 800 requests
Completed 1000 requests
Completed 1200 requests
Completed 1400 requests
Completed 1600 requests
Completed 1800 requests
Completed 2000 requests
Finished 2000 requests

Server Software:        Apache/2.4.7
Server Hostname:        sf2testproject.dev
Server Port:            80

Document Path:          /demo/
Document Length:        4658 bytes

Concurrency Level:      10
Time taken for tests:   9.784 seconds
Complete requests:      2000
Failed requests:        0
Total transferred:      9982000 bytes
HTML transferred:       9316000 bytes
Requests per second:    204.42 [#/sec] (mean)
Time per request:       48.918 [ms] (mean)
Time per request:       4.892 [ms] (mean, across all concurrent requests)
Transfer rate:          996.36 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       3
Processing:    18   49  10.9     50      90
Waiting:       15   41  10.1     41      78
Total:         18   49  10.9     50      91

Percentage of the requests served within a certain time (ms)
  50%     50
  66%     54
  75%     56
  80%     58
  90%     62
  95%     65
  98%     69
  99%     73
 100%     91 (longest request)

Apache Bench results (Symfony 2.4 / Ningx 1.4 / HHVM 3)

maurits@nuc:~$ ab -c 10 -n 2000 http://sf2testproject.dev:8080/demo/
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking sf2testproject.dev (be patient)
Completed 200 requests
Completed 400 requests
Completed 600 requests
Completed 800 requests
Completed 1000 requests
Completed 1200 requests
Completed 1400 requests
Completed 1600 requests
Completed 1800 requests
Completed 2000 requests
Finished 2000 requests

Server Software:        nginx/1.4.6
Server Hostname:        sf2testproject.dev
Server Port:            8080

Document Path:          /demo/
Document Length:        4658 bytes

Concurrency Level:      10
Time taken for tests:   4.678 seconds
Complete requests:      2000
Failed requests:        0
Total transferred:      9900000 bytes
HTML transferred:       9316000 bytes
Requests per second:    427.50 [#/sec] (mean)
Time per request:       23.392 [ms] (mean)
Time per request:       2.339 [ms] (mean, across all concurrent requests)
Transfer rate:          2066.52 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:     8   23  11.5     21      84
Waiting:        8   22  11.3     20      84
Total:          8   23  11.5     21      84

Percentage of the requests served within a certain time (ms)
  50%     21
  66%     26
  75%     30
  80%     32
  90%     39
  95%     46
  98%     54
  99%     58
 100%     84 (longest request)

Conclusion

On both setups, I did not do any optimization. I just installed and ran the benchmark. It seems that Symfony 2.4 on HHVM is about twice as fast as on PHP 5.5. In a real life setup this means you need half the machines. This seem like a good cost reduction at first, but it may not be as good as it looks. I believe that in reality most Symfony applications are doing database and/or API calls. These will not be faster when using HHVM, since HHVM only speeds up PHP execution. This is why I think that the difference will be smaller (than a factor 2) for a real life Symfony application. What do you think?

Share

Symfony2 Guzzle bundle for cURL API calling

The LswGuzzleBundle adds Guzzle API call functionality to your Symfony2 application. It is easy to use from the code and is aimed to provide full debugging capabilities. source: Github

On Packagist, you can see that in the past two years we have published 10 LeaseWeb Symfony2 bundles. The latest and tenth addition to the list is the LswGuzzleBundle. It is an alternative for the LswApiCallerBundle, so it also is a bundle that helps you to call API’s using cURL. The difference is that this bundle uses Guzzle as a wrapper for cURL.

Guzzle profiler and debugger

This bundle also adds a debug panel to your Symfony2, where you can find the cURL API calls Guzzle makes in your application. It provides profiling and extensive debugging capabilities to your Symfony2 application. We feel it is an essential bundle if you create an application that does not work with a database, but instead has an API as data store backend. Guzzle allows you to specify the parameters and and available calls in a Guzzle service description. Since this allows the bundle to understand the structure of the call, it will display the parsed parameters and output. This is an unique feature of this bundle.

guzzle_bundle

Open source

The bundle is MIT licensed. You can find it at:

As always there is a readme documentation on GitHub to get you started.

Share