Introduction

The majority of the widely deployed virtualization platforms in data centers and cloud infrastructures are based in full and para-virtualization technologies. Such is the case of Xen, Kernel-based Virtual Machine (KVM), VMware ESXi and Hyper-V hypervisors (^{Arceo et al., 2015}). On the other hand, the Operating System Level Virtualization (OSLV) technology is getting acceptance into cloud infrastructures with solutions like: Docker, LinuX Container (LXC) and LXC’s new interface LXD (LXC/LXD), which have their roots in the OSLV pioneer solution OpenVZ (^{Arceo et al., 2015}; ^{Agarwal, 2015}; ^{Wallner, 2015}; ²⁰¹⁴).

OSLV can be considered as a lightweight alternative to full and para-virtualization technologies. The main difference is that OSLV eliminates the hypervisor layer, redundant OS kernels, binaries and libraries needed to run workloads in Virtual Machines (VMs). Hypervisors abstract hardware, which results in overhead in terms of virtualizing hardware and virtual device drivers. A full OS is typically run on top of this virtualized hardware in each VM instance. In contrast, containers implement isolation of processes at the OS level, thus avoiding such overhead. These containers run on top of the same shared OS kernel of the underlying host machine, and one or more processes can be run within each container. Due to the shared kernel, as well as the OS libraries, container based solutions can achieve higher density of virtualized instances with better performance compared to hypervisor based solutions, bringing thus better efficiency in a Data Center (DC) infrastructure (^{Arceo et al., 2015}; ^{Agarwal, 2015}; ^{Wallner, 2015}; ^{Morabito et al., 2015}; ^{Graber, 2015a}; ^{Petazzoni, 2015}).

OSLV has been around for over 18 years, but its adoption has been hindered in DC infrastructures because of the shared kernel approach and the mechanisms to achieve the resource isolation, which can be an issue for multitenant security. Nevertheless, nowadays, OSLV supports a variety of technologies to mitigate most security concerns, removing this drawback. The main tools are: namespaces, specially user namespaces, control groups (cgroup) and Linux Security Modules (LSMs). Namespaces give the containers their own view of the system, limiting what containers can see and therefore use, while cgroup limits how much they can use, achieving resource isolation. User namespaces define by default unprivileged containers, which are safe by design. The container uid 0 is mapped to an unprivileged user outside of the container and only has extra rights on resources that it owns itself. Thus LSMs like SELinux, AppArmor and Seccomp are not necessary, although solutions like LXC and LXC/LXD use them to add an extra layer of security which may be handy in the event of a kernel security issue. Cgroup, restricts the use of physical resources like the Central Processing Unit (CPU), the Random Access Memory (RAM) and storage devices, through the establishment of quotas and priorities to containers, avoiding potential Denial of Services (DoS) attacks. (^{Petazzoni, 2017}; Graver 2014a: 2016a). These alternatives are supported by leading OSLV exponents today like LXC, LXC/LXD and Docker. In addition, Cloud Service Providers (CSPs) like Joyent, Kyup and ElasticHosts, are offering Infrastructure as a Service (IaaS) based on bare metal container infrastructures (^{Graber, 2014b}; ^2017a, ^b, ^c; ^2016b, ^c). So it is time to exploits the advantages of the OSVL in DC infrastructures, especially in Small and Medium-sized Enterprises (SMEs).

Nowadays public cloud IaaS providers, like Amazon (^{Graber, 2015b}), offers applications based in Docker containers deployed on VM. Joyent, however, brings IaaS on a bare metal containers infrastructure, which completely enjoys the OS virtualization’s advantages (^{Graber, 2015c}). Joyent’s solution for deploying containers is named Triton (^{Graber, 2015b}; ^{Cantrill, 2014}). It is free and open source (^{Cantrill, 2014}). In the private cloud scenario, however, the OS virtualization technology hasn’t a wide support. Private cloud managers, like OpenStack, OpenNebula and the Elastic Utility Computing Architecture for Linking your Programs to Useful Systems (Eucalyptus), don't offer good support for this technology, if they support it at all.

OpenStack stands out because it gives support to LXC, Docker and LXD, but this support is on early stages (^{Cantrill, 2017a}, ^b, ^c). However, OpenStack is not the best solution for all the entities. It is a complex Cloud Management Platform (CMP) with a steep learning curve and with high hardware requirements for its deployment compared with others like OpenNebula (^{Chilipirea et al., 2016a}). OpenStack in a basic ready production deployment with high availability, suggests at least seven physical hosts for supporting its controller services, which are recommended to not be virtualized (Cantril, 2017a; ^{Chilipirea et al., 2015}). Hosts minimum features proposed are 32GB of RAM, 2x Intel® Xeon® CPU E5-2620 0 @ 2.00 GHz and two Network Interface Cards (NICs) at 10Gbps (^{Chilipirea et al., 2015a}). On the other hand OpenNebula requires only two VMs with 2GB of RAM, two CPUs and two NICs for a production private cloud (^{Chilipirea et al., 2016b}). So, a lightweight, flexible, scalable and easy to use CMP could be the efficient solution for SMEs, especially for those with restricted budgets, that do not require high compute resources for supporting their Information Technology (IT) services, but do need the benefits of the private cloud paradigm.

OpenNebula is an open-source solution, used in several types of environments. Leading organizations, like National Aeronautics and Space Administration (NASA) and the Tokyo Institute of Technology in Supercomputing field, use OpenNebula to build enterprise private clouds, hosting, public cloud services, high performance computing and science clouds. Some features that makes OpenNebula a wise choice are: a powerful user security management, support of multi-tenancy with group management, on-demand Provision of Virtual Data Centers (VDCs), control and monitoring of virtual and physical infrastructures, distributed resource optimization, management of multi-tier applications, standard cloud interfaces and simple provisioning portal for cloud consumers; broad commodity and enterprise platform support, such as a broad hypervisor and storage technologies support; and easy extension and integration, it is a modular and extensible architecture able to fit into any existing DCs with customizable drivers. It is a fully open-source technology available under Apache License and new drivers can be easily written in any language (^{Chilipirea et al., 2015b}, ^c).

OpenNebula’s features directly contribute to achieve the functional and nonfunctional requirements of a cloud DC. However, its interoperability, adaptability, feasibility and contribution to an efficient infrastructure with good levels of performance and scalability, can be improved with a reliable support of an OSLV solution. Two drivers were developed for the supporting of OpenVZ and LXC, one by China Mobile and the other by Valentin Bud respectively. Both had deficient features and functionalities, and poor support (^{Chilipirea et al., 2012}; ^2016c).

Therefore, the objective of the present work was to implement a driver for supporting one of the main OS virtualization solutions by OpenNebula. The driver developed was for LXC. It supports the following features:

This paper treats the following topics: reasons for choosing LXC to integrate to OpenNebula; OpenNebula’s components and interfaces needed for the driver development; the creation of the driver LXCoNe and; the proofs of concept done to demonstrate its functionalities.

Why LXC?

LXC was selected by the authors of the present paper for being integrated to OpenNebula because it’s a stable solution that supports the majority of the most important features of Linux containers; and because it has a big community developing new functionalities that improve its usability, security, performance and fault tolerance isolations, which can be proved with LXD (^{Chilipirea et al., 2015d}; ^e; ^f; ²⁰¹¹; ^2015g). LXC is a derivation of OpenVZ, supported by Canonical Ltd., which shares many of the same developers as OpenVZ (^{Wallner, 2015}). It is a collection of user-level tools that assists in the creation, management and termination of containers, which is included in most Linux distributions. In most cases installation is as simple as selecting it in the package manager, removing one of the main OpenVZ detractions (^{Aderholdt et al., 2014}). LXC has other advantages like the supporting of:

However, it doesn’t support some features like OpenVZ: live migration and storage Quality of Service (QoS) (^{Chilipirea et al., 2015g}). But LXC has LXD, a new project of Canonical Ltd., aimed at revitalizing the use of LXC (^{Scott, 2015a}). It is intended (^{Aderholdt et al., 2014}; ^{Scott, 2015a}, ^b; ^{Banerjee, 2014}; ^{Ectors, 2014a}, ^b; ²⁰¹⁶; ²⁰¹⁴):

LXD isn't a rewrite of LXC, in fact it's building on top of LXC to provide a new, better user experience. Under the hood, LXD uses LXC through liblxc and its Go binding to create and manage the containers. It's basically an alternative to LXC's tools and distribution template system with the added features that come from being controllable over the network (^{Scott, 2015b}). So, why not LXD? While LXC is stable, LXD is still undergoing a rapid development. Some features haven’t been implemented yet and the documentation is still a bit on the light side (^{Aderholdt et al., 2014}; ^{Scott, 2015a}, ^b).

Docker, although it is focused on being the universal container for applications (^{Ectors, 2014c}), was not selected because it is considered by the community and by the authors of the present paper a solution suited for providing Platform as a Service (PaaS) (^{Banerjee, 2014b}), not IaaS, for the main following reasons:

Besides, Docker was initially based on the LXC project, although it has now developed its own implementation libcontainer that uses kernel namespaces and cgroups directly (^{Ectors, 2014c}). This makes Docker not a virtualization solution, but one that automates the deployments of applications inside containers, by providing an additional layer of abstraction and automation of the OSLV.

Components and APIs of OpenNebula Needed for Integrating the LXC Driver

In order to achieve the goal of the present work, it was necessary to identify what OpenNebula offers to cloud integrators. The main strength is the modular and extensible architecture of OpenNebula, which has been designed to be easily adapted to any infrastructure and easily extended with new components (^{Banerjee, 2015b}). Figure 1 shows the OpenNebula’s architecture and the components and interfaces used for the driver development. The Virtualization (VM driver) is in charge of all the interaction with the hypervisors. This driver had to be created by the authors of the present paper in order to manage LXC containers through OpenNebula. The Monitoring (IM driver) makes it possible to acquire real time information of the host and the VM deployed. This driver had to be created by the authors of the present paper in order to monitor LXC containers through OpenNebula. The VM-API and IM-API were needed for the driver to interact with the rest of the OpenNebula’s sections.

Figure 1 Components used in the OpenNebula’s architecture (^{Banerjee, 2015b}) 

LXC Driver Requirements

LXC driver for OpenNebula should be able to perform several actions, some of them mandatory and others optional but desirable. The following actions must be supported by the LXC driver:

The following actions are considered by the authors of this paper optional, but desirable:

Integrating LXC in Data Centers and Cloud Managers, Previous Work

Today’s free and open source cloud managers that have worked in the support of LXC are OpenStack and OpenNebula. Proxmox 4.x, a free and open source DC manager for small enterprises, is supporting LXC too. So, the drivers of these solutions were analyzed in order to be aware of their advantages and drawbacks.

Two previous LXC drivers for OpenNebula were found, one made by China Mobile and the other by Valentin Bud:

OpenStack, a cloud manager with a great market share, is actually developing its LXC driver. OpenStack allocates its LXC driver inside “Group C”. Drivers inside this group “have minimal testing and may or may not work at any given time. Use them at your own risk” (^{Bud, 2015b}). For this reason and because this driver was built over libvirt, it wasn’t used as a reference.

Proxmox 4.x supports LXC. It’s able to deploy, reboot, reset, shutdown and monitor containers. It can use DAS, SAN and NAS storages. It has support to: live snapshots, limit containers resources and live migration, although the latter is in the experimental phase yet. It has limitations such as it: only supports rootfs resizing through the Graphical User Interface (GUI) and does not support hot disk attach/detach (^{Bud, 2015c}; ^d; ^e; ^2016a) Proxmox 4.x was the reference for the authors of the present paper, for the development of the CPU limitations in the LXC driver for OpenNebula, and for its Virtual Network Computing (VNC) implementation.

(^{Bud, 2016b}) states that libvirt-lxc is not generally recommended due to a lack of Apparmor protection for containers. This recommendation, together with the China Mobile experience, caused that the LXC driver for OpenNebula had been developed with liblxc instead of libvirt.

LXC Driver Development for OpenNebula, LXCoNe

OpenNebula manages the hypervisor underneath by running scripts on the host. Each operation that OpenNebula is able to perform over a hypervisor consists of a specific script located at /var/lib/one/<driver-name>/<script-name>. For example, the deploy action is a script called deploy, and for hot-attach a NIC there is a script called attach_nic. The name of the script is always suggestive. The most important and difficult action in this driver is to deploy a new container.

Figure 2 shows the script’s basic blocks. The first step is to read all the information that will be used from the containers template and store it in variables. Once this is done, a folder that will contain all the necessary files for the container is created with the right permissions. This folder is created inside the folder configured as the default container location in the lxc user tools, for example /var/lib/lxc in Ubuntu. In this way the containers created by OpenNebula are shown at the output of lxc-ls command, which is necessary for the driver to be able to monitor containers. Then the configuration information for the NIC is extracted, arranged and prepared inside a variable in a format that LXC’s configuration file understands. This process is simple and is shown in Figure 3. Now, the driver will proceed to configure the root storage as explained in Figure 4.

Figure 2 LXCoNe’s main work-flow diagram 

Figure 3 NIC Configuration 

Figure 4 Root storage set up process 

Because this driver supports three different storage types: FS, LVM and Ceph, it needs to find out which type it is and act accordingly. In case it is FS and LVM, the only thing that needs to be done is to indicate to LXC a path to the image. It is important to remember that OpenNebula uses images, either raw or qcaw, as virtual disks. LXC supports regular FS and LVM, so nothing else needs to be done, but in case the image is stored as a block device in Ceph, this image will need to be mapped in the host and provide LXC the route to where it was mapped. The reason why LXC doesn’t support Ceph’s block devices is mainly because it can’t perform this mapping action by itself, so it needs to be done by something else before the container starts. A possible approach here could be to perform this operation from OpenNebula itself. This makes sense because OpenNebula is already executing code, so the only thing that needs to be done is to write a line that tells the host to map the image. The problem with this approach is that, in case of an electrical failure or any other issue that could cause the physical host to crash, the containers that were running will not be able to start automatically. An administrator will need to manually redeploy them from OpenNebula. One of the goals wanted is precisely to avoid this, so another approach was used. The required instructions to map the root image must be executed once the host’s OS initializes. Write it inside /etc/rc.local is a possibility. Also, containers must be configured to start automatically once the OS initializes.

The next step will be to generate LXC’s configuration file. Inside this file will be located all the container’s parameters, like NIC information, route to root storage, and resources limit. At this point the container is ready to start, but first must be checked if the user added another disk(s) to the container. If this results to be the case, the driver must be capable of attaching this disk(s) to the container. LXC allows to mount locations inside the container specified in the configuration file, but this is only useful to mount images when the container is going to be started, so hot-attach is not possible using this approach. A method that allowed mounting images inside the container while it was on needed to be found. Once a solution was found, it was implemented for the hot-attach disk action and also to add extra disks defined by the user before starting the container. The reason behind this was to have a single method to address both situations. LXC allows managing devices in running containers by using lxc-device. With this tool, a block or loop device from the host can be added to a running container, so it will be able to see it and mount it. Now, it could be possible to instruct OpenNebula to start the container and then use the previous defined method to attach any extra disk specified by the user. This solution will definitely work, but it has a major drawback, containers will only be able to be started and managed from OpenNebula. One of the goals wanted is to be able to manage containers either from OpenNebula, liblxc, a ssh session with the container or any other way after they were created by OpenNebula. The solution found was to use LXC’s hooks. The instructions to add the device to the container and then mount it are not executed by OpenNebula, but written to the start hook of LXC. The script that represents this start hook is executed by LXC before it starts the container. These last two steps are explained in Figure 5.

Figure 5 Last steps before the container is ready 

Extra disks attachment is not the only thing that is configured in this hook at this point, neither the start hook is the only one used. The VNC session is configured in the start hook so it will be started with the container, and then used by OpenNebula. LXC’s post-stop hook is also used. It will run at the node’s namespace after the container has been shut down. With the help of this hook a cleanup process will occur. After a container is shut down, files like LXC container’s configuration and hooks are left behind. Even these files are small they could cause problems once they accumulate after some time. This cleanup process is in charge of erasing this files and any other remains of the container. Once this two hooks are created, OpenNebula will instruct LXC to start the container. After this, OpenNebula will check the container’s status for a short while assuring it was successfully started, and no error occurred. In case of any error, OpenNebula will notice it, will change the status to FAILURE and will log it.

Because every configuration in the node that LXC might need to be able to successfully start the container is set inside the hooks, and liblxc allows performing operations such as shutdown, suspend, reboot and resume over containers, they will be easy to implement on OpenNebula. A simple command will usually be enough. The only remaining operation is hot-attach/detach NICs. Hot-attach can be accomplished easily by creating a virtual Ethernet interface and moving it to the container’s namespace. Hot-detach will be achieved by deleting the interface.

This algorithm was implemented in bash at the first place. It was deployed in the Private Cloud infrastructure of the José Antonio Echeverría Higher Polytechnic Institute (CUJAE)’s DC, supporting the Information and Communications Technology (ICT) of the university. A stable release has been made public on github: https://github.com/OpenNebula/addon-lxcone/, together with the guidelines for supporting its deployment.

Proofs of concept

Different proofs of concept were done in the OpenNebula Private Cloud of the CUJAE’s DC to check the effectiveness of LXCoNe. Figure 6 shows the logical design of the CUJAE’s network, and Figure 7 shows the infrastructure’s compute nodes. The Frontend was deployed on an LXC container inside Node-0 using OpenNebula 4.14. The Frontend was able to manage containers inside its own node, Node-0, and six other nodes. The first five nodes were commodity hardware with identical characteristics. The 6^th. node was an Inspur professional server. Two different storage systems were used at the same time, Ceph and LVM. The tools used in the proofs of concept were the OpenNebula cloud manager and its Sunstone administration interface.

Figure 6 CUJAE data center’s logical design 

Figure 7 Infrastructure’s compute nodes at ISPJAE/CUJAE’s data center 

The proofs of concept demonstrated the capacity of LXCoNe to:

Figure 8 VM view. Container in RUNNING state 

Figure 9 VM view. Extra HDD attached 

Figure 10 VM view. Extra NIC attached 

Figure 11 VM view. 4 GB of RAM assigned to the container 

Figure 12 Fill the container’s RAM. With stress tool 

Figure 13 Host’s maximum used RAM 

Figure 14 Host’s maximum used RAM 

Figure 15 Image view. LVM 

Figure 16 Image view. Ceph 

Figure 17 Container’s state after the deployment 

This driver was tested on two flavors of Linux: Debian 8 (Jessie) and Ubuntu 14.04 (Trusty Tahr) (^{De la Fé, 2016}).

Conclusions

The LXC integration in OpenNebula contributes to develop more efficient solutions with high flexibility and interoperability levels in Cloud infrastructures. It has made possible an easier adaptation to the client economics restrictions, the human resources and the initial IT technologies of the client. The present work has as a main result, the development of a driver for OpenNebula to allow the deployment and the monitoring of the LXC virtualization platform. The driver has several basic features such as: to deploy, shutdown, suspend, reset LXC containers; to attach and detach disks and NICs to LXC containers; to support LVM and different FS for the storage; to limit RAM and CPU resources and to monitor containers and hosts. The next steps will be aimed to integrate other features which guarantee security, high availability and improvement of performance and scalability in the infrastructure.