Building a CaaS solution on bare metal servers

Welcome readers to the first Leaseweb Labs blog in our series on the topic of container solutions. This post is written by Santhosh Chamia veteran Engineer with vast experience building IaaS/Cloud platforms from the ground up. 

What are Containers as a Service (CaaS)? 

Containers as a Service (CaaS) is a hosted container infrastructure that offers an easy way to deploy containers on elastic infrastructureCaaS is suitable in contexts where developers want more control over container orchestration. With CaaS, developers can deploy complex applications on containers without worrying about the limitations of certain platforms. 

As a Senior Infrastructure Engineer at Leaseweb, my primary focus is on exceptional operational delivery. Container-based infrastructure and technology is an integral part of operations for myself and my team. We can deliver the power of Kubernetes to our applications quickly, securely, and efficiently using CaaS.  

This blog portrays a high-level CaaS solution on bare metal servers with rich elastic features. This may be useful for those who want to deploy on-premise enterprise-level Kubernetes clusters for the production workloads.  

Things to consider in CaaS Solution 

Infrastructure 

CaaS platforms are built on top of open hyper-converged infrastructure (HCI). They combine compute, storage, and network fabric into one platform – using low-cost commodity x86 hardware, which adds more value by throwing in software-defined systems, as well as horizontally scalable underlying infrastructure for CaaS. 

Container Orchestration (Kubernetes) 

Kubernetes is an open-source container-orchestration system for automating application deployment, scaling, and management. We are using Kubernetes for container orchestration in our CaaS platform. 

Storage (Class / volume plug-in) 

The Storage Class provides a way for administrators to describe the classes of storage they offer. Different Classes might map to quality-of-service levels. We are using volume plug-in RBD for high-performance workloads, and general workloads with NFS in our CaaS.  

Cluster Networking 

We are using cluster networking/CNI through Calico. Calico provides highly scalable networking and network policy solution for connecting Kubernetes pods based on the same IP networking principles as the internet.  

Cluster Networking makes use of layer 3 network and features the BGP routing protocol, network policy, and route reflector. This is when the nodes act as a client and peering to the controller servers, and controller servers use the BIRD Internet routing daemon to have better performance and stability. 

Load Balancing (on bare metal) 

Kubernetes does not offer an implementation of network load-balancers for bare metal clusters. We have deployed load balancing such as L4 with MetalLB and L7 with IngressMetalLB is a load-balancer implementation for bare metal Kubernetes clusters, using standard routing protocols. We deployed MetalLB with BGP routing protocols. 

In Kubernetes, an Ingress is an object that allows access to your Kubernetes services from outside the Kubernetes cluster. You configure access by creating a collection of rules that define which inbound connections reach which services using Nginx Ingress Controller. 

Kubernetes Security  

We have a number of security measures in our solution. These include:  

  • Transport Level Security (TLS) for all API traffic 
  • Network policies for a namespace to restrict access to pods/ports, and controlling the placement of pods onto nodes pools 
  • Separate namespaces for isolation between components 
  • Role-Based Access Control (RBAC) 
  • Limiting resource usage on a cluster using resource quota limits 
  • Using etcd ACLs 
  • Enabling audit logging to analysis in the event of a compromise. 

Kubernetes logging and monitoring 

Monitoring and logging for CaaS solution means using tools like: 

  • Icinga2 distributed monitoring – for underlying infrastructures
  • Prometheus/Grafana – for Kubernetes Cluster monitoring.
  • Elasticsearch, Fluentd, and Kibana (EFK) for stack managing logging

 Provisioning and life cycle

We are using Chef for provisioning and configuration management of base OS, and Ansible for Kubernetes cluster provisioning and lifecycle management. 

Infrastructure architecture diagram 

CaaS

Conclusion 

With this design, I am able to manage the underlying infrastructure and the Kubernetes cluster within the same umbrella. The solution is cost-effective and can be deployed with low-cost commodity x86 hardware.  

This CaaS solution is implemented using open-source technologies, so IT teams should consider the learning and development that is needed for developers to implement and manage this solution. Stay tuned for the next post, expect detailed technical blog in each domain. 

Share

Automatically provision your bare metal infrastructure

At LeaseWeb we are all about automating delivery processes. Be it for our virtual products or bare metal products. This post shows you one of the many things you can do with our API.

If you have a bare metal server at LeaseWeb I encourage you to login to our customer portal The LeaseWeb Self Service Center at https://secure.leaseweb.com and
In the API section you can manage your api keys for accessing the LeaseWeb API. To read more about what you can do with our API head over to the LeaseWeb Developer Portal

Recently we have published new api calls on our developer portal for customers to manage dhcp leases for their bare metal servers.

These api calls expose our internal dhcp infrastructure, that we use for automation, to our customers as a service.

    GET    /bareMetals/{bareMetalId}/leases                 # list all leases
    POST   /bareMetals/{bareMetalId}/leases                 # create a lease
    DELETE /bareMetals/{bareMetalId}/leases/{macAddress}    # delete a lease

Customers can use it to install operating systems which are not available in the LeaseWeb Self Service Center or if they would like to automatically provision their bare metal infrastructure.

When you use our api to create a dhcp lease you have the possibility to specify the dhcp option 67 Bootfile Name. We chainload the open source ipxe network boot firmware which has http support (read more about ipxe on their website http://ipxe.org/). This means that you can provide a valid http url for dhcp option 67 Bootfile Name that points to a pxe script containing instructions what the the boot loader should do next.

For example: let’s say you own the webserver at webserver.example.com where you have placed the following ipxe script at /boot.ipxe:

    $ curl http://webserver.example.com/boot.ipxe

    #!ipxe
    dhcp
    kernel http://webserver.example.com/archiso/boot/x86_64/vmlinuz archisobasedir=archiso archiso_http_srv=http://webserver.example.com/ ip=:::::eth0:dhcp
    initrd http://webserver.example.com/archiso/boot/x86_64/archiso.img
    boot

You can now create a dhcp lease for your bare metal server using our api:

    $ curl -H 'X-Lsw-Auth: my-api-key' -X POST https://api.leaseweb.com/v1/bareMetals/{bareMetalId}/leases -d bootFileName="http://webserver.example.com/boot.i

Obviously replace {bareMetalId} with the id of your bare metal server. To view the dhcp lease that we just created you can use this call:

    $ curl -H 'X-Lsw-Auth: my-api-key' https://api.leaseweb.com/v1/bareMetals/{bareMetalId}/leases
    
    {
        "_metadata": {
            "limit": 10, 
            "offset": 0, 
            "totalCount": 1
        }, 
        "leases": [
            {
                "ip": "203.0.113.1", 
                "mac": "AA:AA:AA:AA:AA:AA", 
                "options": [
                    // ...
                    {
                        "name": "Bootfile Name", 
                        "optionId": "67", 
                        "policyName": null, 
                        "type": "String", 
                        "userClass": "gPXE", 
                        "value": "http://webserver.example.com/boot.ipxe", 
                        "vendorClass": ""
                    }
                    // ...
                ], 
                "scope": "203.0.113.0"
            }
        ]
    }

Now you have to manually reboot your server or use our api to issue a power cycle:

    $ curl -H 'X-Lsw-Auth: my-api-key' -X POST https://api.leaseweb.com/v1/bareMetals/{bareMetalId}/reboot

The server will reboot, ask for dhcp lease and eventually read the instructions provided by you in /boot.ipxe which in this example is downloading a kernel and the archlinux live cd which are both served from your web server at `webserver.example.com`.

You should be careful and not forget to remove a dhcp lease when you are done. Otherwise during the next reboot it will boot from the network again.

    $ curl -H 'X-Lsw-Auth: my-api-key' -X DELETE https://api.leaseweb.com/v1/bareMetals/{bareMetalId}/leases/AA:AA:AA:AA:AA:AA

We automatically remove dhcp leases after 24 hours .

This service allows our customers to implement creative ideas that can automate their bare metal infrastructure.

Example: install arch linux over ssh without kvm

To continue the example I used this service to boot my modified version of the archlinux live cd which includes and starts openssh at boot and includes my public ssh keys. I use this trick to be able to manually install an operating system which is not available through the LeaseWeb Self Service Center.

I don’t need to contact technical support or have kvm on my server. Everything is done remotely over ssh. The modified live image is published on github here https://github.com/nrocco/archiso-sshd.

Clone the repository from github:

    $ git clone https://github.com/nrocco/archiso-sshd.git
    $ cd archiso-sshd

Add your ssh keys to authorized_keys of the root user:

    $ vim airootfs/root/.ssh/authorized_keys

Now build the image (you need to have the archiso package installed).

    $ make build

This might take a while. When done, copy the kernel, initrmfs and other generated files to the document root of your http server:

    $ cp -r work/iso/arch /var/www

Your document root might look like this now:

    $ find /var/www -type f
    /var/www/boot.ipxe
    /var/www/archiso/pkglist.x86_64.txt
    /var/www/archiso/x86_64/airootfs.md5
    /var/www/archiso/x86_64/airootfs.sfs
    /var/www/archiso/boot/x86_64/archiso.img
    /var/www/archiso/boot/x86_64/vmlinuz

That’s it. Now you boot from the network using our service.

Refer to airootfs/root/customize_airootfs.sh and airootfs/root/.ssh/authorized_keys for the specific customatizations.

What can you do with it?

This example is just the tip of the iceberg of possibilities. Let us know your ideas and use cases.

You might use it to boot into your own live image that does an automated installation of the operating system and kicks off the provisioning tool of your choice (chef, ansible, puppet) so your bare metal servers joins your infrastructure that helps supporting your business.

All fully automated.

Share