How to Limit Resources Using cgroups on CentOS 6

Introduction

Control groups, or cgroups, is a kernel feature introduced in CentOS 6 to provide a new way of limiting access to system resources for processes. You can create your own cgroups, monitor the cgroups you configure, deny cgroups access to certain resources, and even reconfigure your cgroups dynamically on a running system.

In this tutorial, we will see how to limit CPU, memory, and disk i/o for processes. To achieve this, we will first create some control groups, add processes to them, and see how they perform.

Step 1 — Installation

In this section, we will be installing the packages required for cgroups to function.

Control groups and the subsystems to which they relate can be manipulated using shell commands and utilities. However, the easiest way to work with cgroups is to install the libcgroup package. Thelibcgroup package provides cgroups-related command line utilities, configuration files, and man pages. This package is not installed by default on a CentOS 6 server. To install it, run the following command:

sudo yum install libcgroup

Step 2 — Starting the Service

The cgconfig(control group config) service is used to create cgroups and manage subsystems. It can be configured to start up at boot time and reestablish your predefined cgroups, thus making them persistent across reboots. The cgconfig service is not started by default on CentOS 6, so let us start it:

sudo service cgconfig start

Starting the cgconfig service creates a virtual filesystem mounted at /cgroup with all the subsystems. Let us verify this:

sudo ls /cgroup

This command should show the following subsystems:

blkio  cpu  cpuacct  cpuset  devices  freezer  memory  net_cls

You could also run the `lscgroup’ command to verify:

sudo lscgroup

You will see the subsystems in a slightly different layout:

cpuset:/
cpu:/
cpuacct:/
memory:/
devices:/
freezer:/
net_cls:/
blkio:/

System Resources

The system resources are known as subsystems, and each subsystem has several parameters to which we could assign values. CentOS 6 provides ten cgroup subsystems:

blkio — this subsystem sets limits on input/output access to and from block devices such as physical drives (disk, solid state, USB, etc.).
cpu — this subsystem sets limits on the available CPU time
cpuacct — this subsystem generates automatic reports on CPU resources used by tasks in a cgroup
cpuset — this subsystem assigns individual CPUs (on a multicore system) and memory nodes to tasks in a cgroup
devices — this subsystem allows or denies access to devices by tasks in a cgroup
freezer — this subsystem suspends or resumes tasks in a cgroup
memory — this subsystem sets limits on memory use by tasks in a cgroup and generates automatic reports on memory resources used by those tasks
net_cls — this subsystem tags network packets with a class identifier (classid) that allows the Linux traffic controller (tc) to identify packets originating from a particular cgroup task
net_prio — this subsystem provides a way to dynamically set the priority of network traffic per network interface
ns — this is the namespace subsystem

Step 3 — Configuration

In this section, we will create example cgroups and set some resource limits for those cgroups. The cgroup configuration file is /etc/cgconfig.conf. Depending on the contents of the configuration file, cgconfig can create hierarchies, mount necessary file systems, create cgroups, and set subsystem parameters (resource limits) for each cgroup.

A hierarchy is a set of cgroups arranged in a tree, such that every task in the system is in exactly one of the cgroups in the hierarchy. In a default CentOS 6 configuration, each subsystem is put into its own hierarchy.

Let us first create a few cgroups named limitcpu, limitmem, limitio, and browsers. The/etc/cgconfig.conf file contains two major types of entries — mount and group. Lines that start withgroup create cgroups and set subsystem parameters. Edit the file /etc/cgconfig.conf and add the following cgroup entries at the bottom:

/etc/cgconfig.conf

group limitcpu{
        cpu {
                cpu.shares = 400;
        }
}

group limitmem{
        memory {
                memory.limit_in_bytes = 512m;
        }
}

group limitio{
        blkio {
                blkio.throttle.read_bps_device = "252:0         2097152";
        }
}

group browsers{
        cpu {
                cpu.shares = 200;
        }
        memory {
                memory.limit_in_bytes = 128m;
        }
}

In the limitcpu cgroup, we are limiting the cpu shares available to processes in this cgroup to 400.cpu.shares specifies the relative share of CPU time available to the tasks in the cgroup.
In the limitmem cgroup, we are limiting memory available to the cgroup processes to 512MB.
In the limitio cgroup, we are limiting the disk read throughput to 2MiB/s. Here we are limiting read I/O to the primary disk, /dev/vda, with major:minor number 252:0 and 2MiB/s is converted to bytes per second (2x1024x1024=2097152).
In the browsers cgroup, we are limiting cpu shares to 200 and available memory to 128MB.

We need to restart the cgconfig service for the changes in the /etc/cgconfig.conf file to take effect:

sudo service cgconfig restart

Let us enable cgconfig to start on system boot. When you enable the service with chkconfig, it reads the cgroup configuration file /etc/cgconfig.conf at boot time. cgroups are recreated from session to session and remain persistent.

sudo chkconfig cgconfig on

Next, verify the cgroups we configured are showing up correctly:

lscgroup

If everything went well, you should see:

cpuset:/
cpu:/
cpu:/browsers
cpu:/limitcpu
cpuacct:/
memory:/
memory:/browsers
memory:/limitmem
devices:/
freezer:/
net_cls:/
blkio:/
blkio:/limitio

Our next goal is to add the processes (tasks) for which we wish to limit resources to the cgroups we created earlier.

Cgred (control group rules engine daemon) is a service that moves tasks into cgroups according to parameters set in the /etc/cgrules.conf file. Entries in the /etc/cgrules.conf file can take one of the two forms:

user subsystems control_group

user:command subsystems control_group

user refers to a username or a groupname prefixed with the “@” character. subsystems refer to a comma-separated list of subsystem names. control_group represents a path to the cgroup, andcommand stands for a process name or a full command path of a process. Entries in the/etc/cgrules.conf file can include the following extra notations:

@ — indicates a group instead of an individual user. For example, @admin indicates all users in the admin group.
* — represents “all”. For example, * in the user field represents all users.
% — represents an item the same as the item in the line above.

Now let us add the programs/processes we wish to limit. Edit /etc/cgrules.conf and add the following at the bottom:

/etc/cgrules.conf

*:firefox       cpu,memory      browsers/
*:hdparm        blkio   limitio/
sammy   blkio   limitio/
@admin:memhog  memory  limitmem/
*:cpuhog        cpu     limitcpu/

In the above lines, we are setting the following rules:

firefox processes run by any user will be automatically added to the browsers cgroup and limited in cpu and memory subsystems.
hdparm processes run by any user will be added to the limitio cgroup and will be limited in blkio subsystem according to the parameter values specified in that cgroup.
All processes run by user sammy will be added to the limitio cgroup and limited in blkio subsystem.
memhog processes run by anyone in the admin group will be added to the cgroup limitmem and limited in memory subsystem.
cpuhog processes run by any user will be added to the cgroup limitcpu and limited in cpu subsystem.

We need to start the cgred service for the cgrules configuration changes to take effect, do this using the command:

sudo service cgred start

We also need to make sure the cgred service is enabled to start on system boot so that our rules persist across reboots:

sudo chkconfig cgred on

Note: For services that support sysconfig, you could add the variableCGROUP_DAEMON="subsystem:control_group" in /etc/sysconfig/servicename instead of editingcgrules.conf file. For example, for a service like httpd, you could addCGROUP_DAEMON="blkio:/limitio"
to the file /etc/sysconfig/httpd.conf to add the httpd processes to the limitio cgroup.

Step 4 — Testing

In this step, we will verify that the disk read throughput limit of 2MiB/s is enforced correctly according to the rule we added in cgrules.conf. To do this, we will install and run the hdparm tool. The hdparm tool can set and view hardware parameters of hard disk drives, measure read and write speeds, etc. Let us install hdparm using:

sudo yum install hdparm

Now, let us run a command to measure the read speed of your hard disk /dev/vda:

sudo hdparm –direct -t /dev/vda

You should see the following output:

/dev/vda:
 Timing O_DIRECT disk reads:   6 MB in  3.00 seconds =   2.00 MB/sec

The output shows a disk read throughput of 2MB/s. If you stop both the cgconfig and cgred services and run the hdparm command above once again, you can see the original/default read speed from when cgroup rules were not implemented.