man-pages/man2/mbind.2

.\" Copyright 2003,2004 Andi Kleen, SuSE Labs.
.\" and Copyright 2007 Lee Schermerhorn, Hewlett Packard
.\"
.\" Permission is granted to make and distribute verbatim copies of this
.\" manual provided the copyright notice and this permission notice are
.\" preserved on all copies.
.\"
.\" Permission is granted to copy and distribute modified versions of this
.\" manual under the conditions for verbatim copying, provided that the
.\" entire resulting derived work is distributed under the terms of a
.\" permission notice identical to this one.
.\"
.\" Since the Linux kernel and libraries are constantly changing, this
.\" manual page may be incorrect or out-of-date.  The author(s) assume no
.\" responsibility for errors or omissions, or for damages resulting from
.\" the use of the information contained herein.
.\"
.\" Formatted or processed versions of this manual, if unaccompanied by
.\" the source, must acknowledge the copyright and authors of this work.
.\"
.\" 2006-02-03, mtk, substantial wording changes and other improvements
.\" 2007-08-27, Lee Schermerhorn <Lee.Schermerhorn@hp.com>
.\"	more precise specification of behavior.
.\"
.TH MBIND 2 2007-08-27 Linux "Linux Programmer's Manual"
.SH NAME
mbind \- Set memory policy for a memory range
.SH SYNOPSIS
.nf
.B "#include <numaif.h>"
.sp
.BI "int mbind(void *" start ", unsigned long " len  ", int " mode ,
.BI "          unsigned long *" nodemask  ", unsigned long " maxnode ,
.BI "          unsigned " flags );
.sp
Link with \fI\-lnuma\fP
.fi
.SH DESCRIPTION
.BR mbind ()
sets the NUMA memory policy,
which consists of a policy mode and zero or more nodes,
for the memory range starting with
.I start
and continuing for
.I len
bytes.
The memory of a NUMA machine is divided into multiple nodes.
The memory policy defines from which node memory is allocated.

If the memory range specified by the
.IR start " and " len
arguments includes an "anonymous" region of memory\(emthat is
a region of memory created using the
.BR mmap (2)
system call with the
.BR MAP_ANONYMOUS \(emor
a memory mapped file, mapped using the
.BR mmap (2)
system call with the
.B MAP_PRIVATE
flag, pages will only be allocated according to the specified
policy when the application writes [stores] to the page.
For anonymous regions, an initial read access will use a shared
page in the kernel containing all zeros.
For a file mapped with
.BR MAP_PRIVATE ,
an initial read access will allocate pages according to the
process policy of the process that causes the page to be allocated.
This may not be the process that called
.BR mbind ().

The specified policy will be ignored for any
.B MAP_SHARED
mappings in the specified memory range.
Rather the pages will be allocated according to the process policy
of the process that caused the page to be allocated.
Again, this may not be the process that called
.BR mbind ().

If the specified memory range includes a shared memory region
created using the
.BR shmget (2)
system call and attached using the
.BR shmat (2)
system call,
pages allocated for the anonymous or shared memory region will
be allocated according to the policy specified, regardless which
process attached to the shared memory segment causes the allocation.
If, however, the shared memory region was created with the
.B SHM_HUGETLB
flag,
the huge pages will be allocated according to the policy specified
only if the page allocation is caused by the task that calls
.BR mbind ()
for that region.

By default,
.BR mbind ()
only has an effect for new allocations; if the pages inside
the range have been already touched before setting the policy,
then the policy has no effect.
This default behavior may be overridden by the
.B MPOL_MF_MOVE
and
.B MPOL_MF_MOVE_ALL
flags described below.

The
.I mode
argument must specify one of
.BR MPOL_DEFAULT ,
.BR MPOL_BIND ,
.B MPOL_INTERLEAVE
or
.BR MPOL_PREFERRED .
All policy modes except
.B MPOL_DEFAULT
require the caller to specify via the
.I nodemask
parameter,
the node or nodes to which the mode applies.

.I nodemask
points to a bitmask of nodes containing up to
.I maxnode
bits.
The bit mask size is rounded to the next multiple of
.IR "sizeof(unsigned long)" ,
but the kernel will only use bits up to
.IR maxnode .
A NULL value of
.I nodemask
or a
.I maxnode
value of zero specifies the empty set of nodes.
If the value of
.I maxnode
is zero,
the
.I nodemask
argument is ignored.

The
.B MPOL_DEFAULT
mode specifies that the default policy should be used.
When applied to a range of memory via
.IR mbind (),
this means to use the process policy,
which may have been set with
.BR set_mempolicy (2).
If the mode of the process policy is also
.BR MPOL_DEFAULT ,
the system-wide default policy will be used.
The system-wide default policy will allocate
pages on the node of the CPU that triggers the allocation.
For
.BR MPOL_DEFAULT ,
the
.I nodemask
and
.I maxnode
arguments must be specify the empty set of nodes.

The
.B MPOL_BIND
mode specifies a strict policy that restricts memory allocation to
the nodes specified in
.IR nodemask .
If
.I nodemask
specifies more than one node, page allocations will come from
the node with the lowest numeric node ID first, until that node
contains no free memory.
Allocations will then come from the node with the next highest
node ID specified in
.I nodemask
and so forth, until none of the specified nodes contain free memory.
Pages will not be allocated from any node not specified in the
.IR nodemask .

The
.B MPOL_INTERLEAVE
mode specifies that page allocations be interleaved across the
set of nodes specified in
.IR nodemask .
This optimizes for bandwidth instead of latency
by spreading out pages and memory accesses to those pages across
multiple nodes.
To be effective the memory area should be fairly large,
at least 1MB or bigger with a fairly uniform access pattern.
Accesses to a single page of the area will still be limited to
the memory bandwidth of a single node.

.B MPOL_PREFERRED
sets the preferred node for allocation.
The kernel will try to allocate pages from this
node first and fall back to other nodes if the
preferred nodes is low on free memory.
If
.I nodemask
specifies more than one node ID, the first node in the
mask will be selected as the preferred node.
If the
.I nodemask
and
.I maxnode
arguments specify the empty set, then the memory is allocated on
the node of the CPU that triggered the allocation.
This is the only way to specify "local allocation" for a
range of memory via
.IR mbind (2).

If
.B MPOL_MF_STRICT
is passed in
.I flags
and
.I policy
is not
.BR MPOL_DEFAULT ,
then the call will fail with the error
.B EIO
if the existing pages in the memory range don't follow the policy.
.\" According to the kernel code, the following is not true
.\" --Lee Schermerhorn
.\" In 2.6.16 or later the kernel will also try to move pages
.\" to the requested node with this flag.

If
.B MPOL_MF_MOVE
is specified in
.IR flags ,
then the kernel will attempt to move all the existing pages
in the memory range so that they follow the policy.
Pages that are shared with other processes will not be moved.
If
.B MPOL_MF_STRICT
is also specified, then the call will fail with the error
.B EIO
if some pages could not be moved.

If
.B MPOL_MF_MOVE_ALL
is passed in
.IR flags ,
then the kernel will attempt to move all existing pages in the memory range
regardless of whether other processes use the pages.
The calling process must be privileged
.RB ( CAP_SYS_NICE )
to use this flag.
If
.B MPOL_MF_STRICT
is also specified, then the call will fail with the error
.B EIO
if some pages could not be moved.
.\" ---------------------------------------------------------------
.SH RETURN VALUE
On success,
.BR mbind ()
returns 0;
on error, \-1 is returned and
.I errno
is set to indicate the error.
.\" ---------------------------------------------------------------
.SH ERRORS
.\"  I think I got all of the error returns.  --Lee Schermerhorn
.TP
.B EFAULT
Part of all of the memory range specified by
.I nodemask
and
.I maxnode
points outside your accessible address space.
Or, there was a unmapped hole in the specified memory range.
.TP
.B EINVAL
An invalid value was specified for
.I flags
or
.IR mode ;
or
.I start + len
was less than
.IR start ;
or
.I start
is not a multiple of the system page size.
Or,
.I mode
is
.B MPOL_DEFAULT
and
.I nodemask
specified a non-empty set;
or
.I mode
is
.B MPOL_BIND
or
.B MPOL_INTERLEAVE
and
.I nodemask
is empty.
Or,
.I maxnode
specifies more than a page worth of bits.
Or,
.I nodemask
specifies one or more node IDs that are
greater than the maximum supported node ID,
or are not allowed in the calling task's context.
.\" "calling task's context" refers to cpusets.
.\" No man page avail to ref. --Lee Schermerhorn
Or, none of the node IDs specified by
.I nodemask
are on-line, or none of the specified nodes contain memory.
.TP
.B EIO
.B MPOL_MF_STRICT
was specified and an existing page was already on a node
that does not follow the policy;
or
.B MPOL_MF_MOVE
or
.B MPOL_MF_MOVE_ALL
was specified and the kernel was unable to move all existing
pages in the range.
.TP
.B ENOMEM
Insufficient kernel memory was available.
.TP
.B EPERM
The
.I flags
argument included the
.B MPOL_MF_MOVE_ALL
flag and the caller does not have the
.B CAP_SYS_NICE
privilege.
.\" ---------------------------------------------------------------
.SH NOTES
NUMA policy is not supported on a memory mapped file range
that was mapped with the
.B MAP_SHARED
flag.

.B MPOL_MF_STRICT
is ignored on huge page mappings.

The
.BR MPOL_DEFAULT ,
mode has different effects for
.BR mbind (2)
and
.BR set_mempolicy (2).
When
.B MPOL_DEFAULT
is specified for a range of memory using
.BR mbind (),
any pages subsequently allocated for that range will use
the process's policy, as set by
.BR set_mempolicy (2).
This effectively removes the explicit policy from the
specified range.
To select "local allocation" for a memory range,
specify a
.I mode
of
.B MPOL_PREFERRED
with an empty set of nodes.
This method will work for
.BR set_mempolicy (2),
as well.
.\" ---------------------------------------------------------------
.SS "Versions and Library Support"
The
.BR mbind (),
.BR get_mempolicy (2),
and
.BR set_mempolicy (2)
system calls were added to the Linux kernel with version 2.6.7.
They are only available on kernels compiled with
.BR CONFIG_NUMA .

You can link with
.I -lnuma
to get system call definitions.
.I libnuma
and the required
.I <numaif.h>
header are available in the
.I numactl
package.

However, applications should not use these system calls directly.
Instead, the higher level interface provided by the
.BR numa (3)
functions in the
.I numactl
package is recommended.
The
.I numactl
package is available at
.IR ftp://ftp.suse.com/pub/people/ak/numa/ .
The package is also included in some Linux distributions.
Some distributions include the development library and header
in the separate
.I numactl-devel
package.

Support for huge page policy was added with 2.6.16.
For interleave policy to be effective on huge page mappings the
policied memory needs to be tens of megabytes or larger.

.B MPOL_MF_MOVE
and
.B MPOL_MF_MOVE_ALL
are only available on Linux 2.6.16 and later.
.SH SEE ALSO
.BR get_mempolicy (2),
.BR mmap (2),
.BR set_mempolicy (2),
.BR shmat (2),
.BR shmget (2),
.BR numa (3),
.BR numactl (8)