te
Copyright 2015 Nexenta Systems, Inc. All rights reserved.
Copyright 2022 Joyent, Inc.
Copyright (c) 2002, Sun Microsystems, Inc. All Rights Reserved.
The contents of this file are subject to the terms of the Common Development and Distribution License (the "License"). You may not use this file except in compliance with the License.
You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE or http://www.opensolaris.org/os/licensing. See the License for the specific language governing permissions and limitations under the License.
When distributing Covered Code, include this CDDL HEADER in each file and include the License file at usr/src/OPENSOLARIS.LICENSE. If applicable, add the following below this CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your own identifying information: Portions Copyright [yyyy] [name of copyright owner]
kmem_cache_create, kmem_cache_alloc, kmem_cache_free, kmem_cache_destroy, kmem_cache_set_move - kernel memory cache allocator operations

#include <sys/types.h>
#include <sys/kmem.h>

kmem_cache_t *kmem_cache_create(char *name, size_t bufsize,
 size_t align, int (*constructor)(void *, void *, int),
 void (*destructor)(void *, void *), void (*reclaim)(void *),
 void *private, void *vmp, int cflags);

void kmem_cache_destroy(kmem_cache_t *cp);

void *kmem_cache_alloc(kmem_cache_t *cp, int kmflag);

void kmem_cache_free(kmem_cache_t *cp, void *obj);

void kmem_cache_set_move(kmem_cache_t *cp, kmem_cbrc_t (*move)(void *,
 void *, size_t *, void *));

 [Synopsis for callback functions:]

int (*constructor)(void *buf, void *user_arg, int kmflags);

void (*destructor)(void *buf, void *user_arg);

kmem_cbrc_t (*move)(void *old, void *new, size_t bufsize,
 void *user_arg);

illumos DDI specific (illumos DDI) The parameters for the kmem_cache_* functions are as follows: name bufsize align constructor destructor reclaim private vmp cflags kmflag obj move

The parameters for the callback constructor function are as follows: void *buf

void *user_arg int kmflags

The parameters for the callback destructor function are as follows: void *buf

void *user_arg

The parameters for the callback move() function are as follows: void *old

void *new size_t bufsize void *user_arg In many cases, the cost of initializing and destroying an object exceeds the cost of allocating and freeing memory for it. The functions described here address this condition.

Object caching is a technique for dealing with objects that are:

The idea is to allow the allocator and its clients to cooperate to preserve the invariant portion of an object's initial state, or constructed state, between uses, so it does not have to be destroyed and re-created every time the object is used. For example, an object containing a mutex only needs to have mutex_init() applied once, the first time the object is allocated. The object can then be freed and reallocated many times without incurring the expense of mutex_destroy() and mutex_init() each time. An object's embedded locks, condition variables, reference counts, lists of other objects, and read-only data all generally qualify as constructed state. The essential requirement is that the client must free the object (using kmem_cache_free()) in its constructed state. The allocator cannot enforce this, so programming errors will lead to hard-to-find bugs.

A driver should call kmem_cache_create() at the time of _init(9E) or attach(9E), and call the corresponding kmem_cache_destroy() at the time of _fini(9E) or detach(9E).

kmem_cache_create() creates a cache of objects, each of size bufsize bytes, aligned on an align boundary. Drivers not requiring a specific alignment can pass 0. name identifies the cache for statistics and debugging. constructor and destructor convert plain memory into objects and back again; constructor can fail if it needs to allocate memory but cannot. private is a parameter passed to the constructor and destructor callbacks to support parameterized caches (for example, a pointer to an instance of the driver's soft-state structure). To facilitate debugging, kmem_cache_create() creates a kstat(9S) structure of class kmem_cache and name name. It returns an opaque pointer to the object cache.

kmem_cache_alloc() gets an object from the cache. The object will be in its constructed state. kmflag has either KM_SLEEP or KM_NOSLEEP set, indicating whether it is acceptable to wait for memory if none is currently available.

A small pool of reserved memory is available to allow the system to progress toward the goal of freeing additional memory while in a low memory situation. The KM_PUSHPAGE flag enables use of this reserved memory pool on an allocation. This flag can be used by drivers that implement strategy(9E) on memory allocations associated with a single I/O operation. The driver guarantees that the I/O operation will complete (or timeout) and, on completion, that the memory will be returned. The KM_PUSHPAGE flag should be used only in kmem_cache_alloc() calls. All allocations from a given cache should be consistent in their use of the flag. A driver that adheres to these restrictions can guarantee progress in a low memory situation without resorting to complex private allocation and queuing schemes. If KM_PUSHPAGE is specified, KM_SLEEP can also be used without causing deadlock.

kmem_cache_free() returns an object to the cache. The object must be in its constructed state.

kmem_cache_destroy() destroys the cache and releases all associated resources. All allocated objects must have been previously freed.

kmem_cache_set_move() registers a function that the allocator may call to move objects from sparsely allocated pages of memory so that the system can reclaim pages that are tied up by the client. Since caching objects of the same size and type already makes severe memory fragmentation unlikely, there is generally no need to register such a function. The idea is to make it possible to limit worst-case fragmentation in caches that exhibit a tendency to become highly fragmented. Only clients that allocate a mix of long- and short-lived objects from the same cache are prone to exhibit this tendency, making them candidates for a move() callback.

The move() callback supplies the client with two addresses: the allocated object that the allocator wants to move and a buffer selected by the allocator for the client to use as the copy destination. The new parameter is an allocated, constructed object ready to receive the contents of the old parameter. The bufsize parameter supplies the size of the object, in case a single move function handles multiple caches whose objects differ only in size. Finally, the private parameter passed to the constructor and destructor is also passed to the move() callback.

Only the client knows about its own data and when it is a good time to move it. The client cooperates with the allocator to return unused memory to the system, and the allocator accepts this help at the client's convenience. When asked to move an object, the client can respond with any of the following:

typedef enum kmem_cbrc {
 KMEM_CBRC_YES,
 KMEM_CBRC_NO,
 KMEM_CBRC_LATER,
 KMEM_CBRC_DONT_NEED,
 KMEM_CBRC_DONT_KNOW
} kmem_cbrc_t;

The client must not explicitly free either of the objects passed to the move() callback, since the allocator wants to free them directly to the slab layer (bypassing the per-CPU magazine layer). The response tells the allocator which of the two object parameters to free: KMEM_CBRC_YES

KMEM_CBRC_NO KMEM_CBRC_LATER KMEM_CBRC_DONT_NEED KMEM_CBRC_DONT_KNOW

Any object passed to the move() callback is guaranteed to have been touched only by the allocator or by the client. Because memory patterns applied by the allocator always set at least one of the two lowest order bits, the bottom two bits of any pointer member (other than char * or short *, which may not be 8-byte aligned on all platforms) are available to the client for marking cached objects that the client is about to free. This way, the client can recognize known objects in the move() callback by the unmarked (valid) pointer value.

If the client refuses to move an object with either KMEM_CBRC_NO or KMEM_CBRC_LATER, and that object later becomes movable, the client can notify the allocator by calling kmem_cache_move_notify(). Alternatively, the client can simply wait for the allocator to call back again with the same object address. Responding KMEM_CRBC_NO even once or responding KMEM_CRBC_LATER too many times for the same object makes the allocator less likely to call back again for that object.

[Synopsis for notification function:]

void kmem_cache_move_notify(kmem_cache_t *cp, void *obj);

The parameters for the notification function are as follows: cp

obj Constructors can be invoked during any call to kmem_cache_alloc(), and will run in that context. Similarly, destructors can be invoked during any call to kmem_cache_free(), and can also be invoked during kmem_cache_destroy(). Therefore, the functions that a constructor or destructor invokes must be appropriate in that context. Furthermore, the allocator may also call the constructor and destructor on objects still under its control without client involvement.

kmem_cache_create() and kmem_cache_destroy() must not be called from interrupt context. kmem_cache_create() can also block for available memory.

kmem_cache_alloc() can be called from interrupt context only if the KM_NOSLEEP flag is set. It can be called from user or kernel context with any valid flag.

kmem_cache_free() can be called from user, kernel, or interrupt context.

kmem_cache_set_move() is called from the same context as kmem_cache_create(), immediately after kmem_cache_create() and before allocating any objects from the cache.

The registered move() callback is always invoked in the same global callback thread dedicated for move requests, guaranteeing that no matter how many clients register a move() function, the allocator never tries to move more than one object at a time. Neither the allocator nor the client can be assumed to know the object's whereabouts at the time of the callback.

Example 1 Object Caching

Consider the following data structure:

struct foo {
 kmutex_t foo_lock;
 kcondvar_t foo_cv;
 struct bar *foo_barlist;
 int foo_refcnt;
 };

Assume that a foo structure cannot be freed until there are no outstanding references to it (foo_refcnt == 0) and all of its pending bar events (whatever they are) have completed (foo_barlist == NULL). The life cycle of a dynamically allocated foo would be something like this:

foo = kmem_alloc(sizeof (struct foo), KM_SLEEP);
mutex_init(&foo->foo_lock, ...);
cv_init(&foo->foo_cv, ...);
foo->foo_refcnt = 0;
foo->foo_barlist = NULL;
 use foo;
ASSERT(foo->foo_barlist == NULL);
ASSERT(foo->foo_refcnt == 0);
cv_destroy(&foo->foo_cv);
mutex_destroy(&foo->foo_lock);
kmem_free(foo);

Notice that between each use of a foo object we perform a sequence of operations that constitutes nothing but expensive overhead. All of this overhead (that is, everything other than use foo above) can be eliminated by object caching.

int
foo_constructor(void *buf, void *arg, int tags)
{
 struct foo *foo = buf;
 mutex_init(&foo->foo_lock, ...);
 cv_init(&foo->foo_cv, ...);
 foo->foo_refcnt = 0;
 foo->foo_barlist = NULL;
 return (0);
}

void
foo_destructor(void *buf, void *arg)
{
 struct foo *foo = buf;
 ASSERT(foo->foo_barlist == NULL);
 ASSERT(foo->foo_refcnt == 0);
 cv_destroy(&foo->foo_cv);
 mutex_destroy(&foo->foo_lock);
}

user_arg = ddi_get_soft_state(foo_softc, instance);
(void) snprintf(buf, KSTAT_STRLEN, "foo%d_cache",
 ddi_get_instance(dip));
foo_cache = kmem_cache_create(buf,
 sizeof (struct foo), 0,
 foo_constructor, foo_destructor,
 NULL, user_arg, 0);

To allocate, use, and free a foo object:

foo = kmem_cache_alloc(foo_cache, KM_SLEEP);
 use foo;
kmem_cache_free(foo_cache, foo);

This makes foo allocation fast, because the allocator will usually do nothing more than fetch an already-constructed foo from the cache. foo_constructor and foo_destructor will be invoked only to populate and drain the cache, respectively.

Example 2 Registering a Move Callback

To register a move() callback:

object_cache = kmem_cache_create(...);
kmem_cache_set_move(object_cache, object_move);

If successful, the constructor function must return 0. If KM_NOSLEEP or KM_NOSLEEP_LAZY is set and memory cannot be allocated without sleeping, the constructor must return -1. If the constructor takes extraordinary steps during a KM_NOSLEEP construction, it may not take those for a KM_NOSLEEP_LAZY construction.

kmem_cache_create() returns a pointer to the allocated cache.

If successful, kmem_cache_alloc() returns a pointer to the allocated object. If KM_NOSLEEP is set and memory cannot be allocated without sleeping, kmem_cache_alloc() returns NULL.

See attributes(7) for descriptions of the following attributes:

ATTRIBUTE TYPE ATTRIBUTE VALUE

Interface Stability Committed

condvar (9F), kmem_alloc (9F), mutex (9F), kstat (9S)

Writing Device Drivers

The Slab Allocator: An Object-Caching Kernel Memory Allocator, Bonwick, J.; USENIX Summer 1994 Technical Conference (1994).

Magazines and vmem: Extending the Slab Allocator to Many CPUs and Arbitrary Resources, Bonwick, J. and Adams, J.; USENIX 2001 Technical Conference (2001).

The constructor must be immediately reversible by the destructor, since the allocator may call the constructor and destructor on objects still under its control at any time without client involvement.

The constructor must respect the kmflags argument by forwarding it to allocations made inside the constructor, and must not ASSERT anything about the given flags.

The user argument forwarded to the constructor must be fully operational before it is passed to kmem_cache_create().