Introduction
Control groups, or cgroups, is a kernel feature introduced in CentOS 6 to provide a new way of limiting access to system resources for processes. You can create your own cgroups, monitor the cgroups you configure, deny cgroups access to certain resources, and even reconfigure your cgroups dynamically on a running system.
In this tutorial, we will see how to limit CPU, memory, and disk i/o for processes. To achieve this, we will first create some control groups, add processes to them, and see how they perform.
Step 1 — Installation
In this section, we will be installing the packages required for cgroups to function.
Control groups and the subsystems to which they relate can be manipulated using shell commands and utilities. However, the easiest way to work with cgroups is to install the libcgroup
package. Thelibcgroup
package provides cgroups-related command line utilities, configuration files, and man pages. This package is not installed by default on a CentOS 6 server. To install it, run the following command:
- sudo yum install libcgroup
Step 2 — Starting the Service
The cgconfig
(control group config) service is used to create cgroups and manage subsystems. It can be configured to start up at boot time and reestablish your predefined cgroups, thus making them persistent across reboots. The cgconfig
service is not started by default on CentOS 6, so let us start it:
- sudo service cgconfig start
Starting the cgconfig
service creates a virtual filesystem mounted at /cgroup
with all the subsystems. Let us verify this:
- sudo ls /cgroup
This command should show the following subsystems:
blkio cpu cpuacct cpuset devices freezer memory net_cls
You could also run the `lscgroup’ command to verify:
- sudo lscgroup
You will see the subsystems in a slightly different layout:
cpuset:/
cpu:/
cpuacct:/
memory:/
devices:/
freezer:/
net_cls:/
blkio:/
System Resources
The system resources are known as subsystems, and each subsystem has several parameters to which we could assign values. CentOS 6 provides ten cgroup subsystems:
- blkio — this subsystem sets limits on input/output access to and from block devices such as physical drives (disk, solid state, USB, etc.).
- cpu — this subsystem sets limits on the available CPU time
- cpuacct — this subsystem generates automatic reports on CPU resources used by tasks in a cgroup
- cpuset — this subsystem assigns individual CPUs (on a multicore system) and memory nodes to tasks in a cgroup
- devices — this subsystem allows or denies access to devices by tasks in a cgroup
- freezer — this subsystem suspends or resumes tasks in a cgroup
- memory — this subsystem sets limits on memory use by tasks in a cgroup and generates automatic reports on memory resources used by those tasks
- net_cls — this subsystem tags network packets with a class identifier (classid) that allows the Linux traffic controller (tc) to identify packets originating from a particular cgroup task
- net_prio — this subsystem provides a way to dynamically set the priority of network traffic per network interface
- ns — this is the namespace subsystem
Step 3 — Configuration
In this section, we will create example cgroups and set some resource limits for those cgroups. The cgroup configuration file is /etc/cgconfig.conf
. Depending on the contents of the configuration file, cgconfig can create hierarchies, mount necessary file systems, create cgroups, and set subsystem parameters (resource limits) for each cgroup.
A hierarchy is a set of cgroups arranged in a tree, such that every task in the system is in exactly one of the cgroups in the hierarchy. In a default CentOS 6 configuration, each subsystem is put into its own hierarchy.
Let us first create a few cgroups named limitcpu, limitmem, limitio, and browsers. The/etc/cgconfig.conf
file contains two major types of entries — mount
and group
. Lines that start withgroup
create cgroups and set subsystem parameters. Edit the file /etc/cgconfig.conf
and add the following cgroup entries at the bottom:
group limitcpu{
cpu {
cpu.shares = 400;
}
}
group limitmem{
memory {
memory.limit_in_bytes = 512m;
}
}
group limitio{
blkio {
blkio.throttle.read_bps_device = "252:0 2097152";
}
}
group browsers{
cpu {
cpu.shares = 200;
}
memory {
memory.limit_in_bytes = 128m;
}
}
- In the
limitcpu
cgroup, we are limiting the cpu shares available to processes in this cgroup to 400.cpu.shares
specifies the relative share of CPU time available to the tasks in the cgroup. - In the
limitmem
cgroup, we are limiting memory available to the cgroup processes to 512MB. - In the
limitio
cgroup, we are limiting the disk read throughput to 2MiB/s. Here we are limiting read I/O to the primary disk, /dev/vda, with major:minor number 252:0 and 2MiB/s is converted to bytes per second (2x1024x1024=2097152). - In the
browsers
cgroup, we are limiting cpu shares to 200 and available memory to 128MB.
We need to restart the cgconfig
service for the changes in the /etc/cgconfig.conf
file to take effect:
- sudo service cgconfig restart
Let us enable cgconfig
to start on system boot. When you enable the service with chkconfig, it reads the cgroup configuration file /etc/cgconfig.conf
at boot time. cgroups are recreated from session to session and remain persistent.
- sudo chkconfig cgconfig on
Next, verify the cgroups we configured are showing up correctly:
- lscgroup
If everything went well, you should see:
cpuset:/
cpu:/
cpu:/browsers
cpu:/limitcpu
cpuacct:/
memory:/
memory:/browsers
memory:/limitmem
devices:/
freezer:/
net_cls:/
blkio:/
blkio:/limitio
Our next goal is to add the processes (tasks) for which we wish to limit resources to the cgroups we created earlier.
Cgred
(control group rules engine daemon) is a service that moves tasks into cgroups according to parameters set in the /etc/cgrules.conf
file. Entries in the /etc/cgrules.conf
file can take one of the two forms:
user subsystems control_group
or
user:command subsystems control_group
user
refers to a username or a groupname prefixed with the “@” character. subsystems
refer to a comma-separated list of subsystem names. control_group
represents a path to the cgroup, andcommand
stands for a process name or a full command path of a process. Entries in the/etc/cgrules.conf
file can include the following extra notations:
@
— indicates a group instead of an individual user. For example,@admin
indicates all users in the admin group.*
— represents “all”. For example,*
in the user field represents all users.%
— represents an item the same as the item in the line above.
Now let us add the programs/processes we wish to limit. Edit /etc/cgrules.conf
and add the following at the bottom:
*:firefox cpu,memory browsers/
*:hdparm blkio limitio/
sammy blkio limitio/
@admin:memhog memory limitmem/
*:cpuhog cpu limitcpu/
In the above lines, we are setting the following rules:
- firefox processes run by any user will be automatically added to the
browsers
cgroup and limited in cpu and memory subsystems. - hdparm processes run by any user will be added to the
limitio
cgroup and will be limited in blkio subsystem according to the parameter values specified in that cgroup. - All processes run by user sammy will be added to the
limitio
cgroup and limited in blkio subsystem. - memhog processes run by anyone in the
admin
group will be added to the cgrouplimitmem
and limited in memory subsystem. - cpuhog processes run by any user will be added to the cgroup
limitcpu
and limited in cpu subsystem.
We need to start the cgred
service for the cgrules configuration changes to take effect, do this using the command:
- sudo service cgred start
We also need to make sure the cgred
service is enabled to start on system boot so that our rules persist across reboots:
- sudo chkconfig cgred on
Note: For services that support sysconfig, you could add the variableCGROUP_DAEMON="subsystem:control_group"
in /etc/sysconfig/servicename
instead of editingcgrules.conf
file. For example, for a service like httpd, you could addCGROUP_DAEMON="blkio:/limitio"
to the file /etc/sysconfig/httpd.conf
to add the httpd processes to the limitio
cgroup.
Step 4 — Testing
In this step, we will verify that the disk read throughput limit of 2MiB/s is enforced correctly according to the rule we added in cgrules.conf
. To do this, we will install and run the hdparm
tool. The hdparm
tool can set and view hardware parameters of hard disk drives, measure read and write speeds, etc. Let us install hdparm using:
- sudo yum install hdparm
Now, let us run a command to measure the read speed of your hard disk /dev/vda:
- sudo hdparm –direct -t /dev/vda
You should see the following output:
/dev/vda:
Timing O_DIRECT disk reads: 6 MB in 3.00 seconds = 2.00 MB/sec
The output shows a disk read throughput of 2MB/s. If you stop both the cgconfig
and cgred
services and run the hdparm
command above once again, you can see the original/default read speed from when cgroup rules were not implemented.