Plug memory leaks on UNMAP and XCOPY with invalid parameters.

MFC after: 1 week

MFHead @ r275232

r274961 through r275075

Coalesce last data move and command status for read commands.

Make CTL core and block backend set success status before initiating last
data move for read commands. Make CAM target and iSCSI fronte

Coalesce last data move and command status for read commands.

Make CTL core and block backend set success status before initiating last
data move for read commands. Make CAM target and iSCSI frontends detect
such condition and send command status together with data. New I/O flag
allows to skip duplicate status sending on later fe_done() call.

For Fibre Channel this change saves one of three interrupts per read command,
increasing performance from 126K to 160K IOPS. For iSCSI this change saves
one of three PDUs per read command, increasing performance from 1M to 1.2M
IOPS.

MFC after: 1 month
Sponsored by: iXsystems, Inc.

show more ...

Merge ^/head r274961 through r274978.

Replace home-grown CTL IO allocator with UMA.

Old allocator created significant lock congestion protecting its lists
of preallocated I/Os, while UMA provides much better SMP scalability.
The downsid

Replace home-grown CTL IO allocator with UMA.

Old allocator created significant lock congestion protecting its lists
of preallocated I/Os, while UMA provides much better SMP scalability.
The downside of UMA is lack of reliable preallocation, that could guarantee
successful allocation in non-sleepable environments. But careful code
review shown, that only CAM target frontend really has that requirement.
Fix that making that frontend preallocate and statically bind CTL I/O for
every ATIO/INOT it preallocates any way. That allows to avoid allocations
in hot I/O path. Other frontends either may sleep in allocation context
or can properly handle allocation errors.

On 40-core server with 6 ZVOL-backed LUNs and 7 iSCSI client connections
this change increases peak performance from ~700K to >1M IOPS! Yay! :)

MFC after: 1 month
Sponsored by: iXsystems, Inc.

show more ...

Merge from head@274682

Merge HEAD@r272944.

IFC @r272481

Sync to HEAD@r272516.

Reintegrate head@r272414

Sponsored by: The FreeBSD Foundation

Fix couple issues with ROD tokens content.

MFC after: 3 days

IFC @r271887

IFC @r271694

Fix tpc_create_token() introduced in r269497 to encode CREATOR LOGICAL UNIT
DESCRIPTOR field as Identification Descriptor CSCD descriptor, not just as
Identification Descriptor.

MFC after: 3 days

IFC @r269962

Submitted by: Anish Gupta (akgupt3@gmail.com)

Sync to HEAD@r270409.

Fix lock recursion on LUN shutdown, introduced on r269497.

MFC after: 3 days

Merge head from 7/28

Sync to HEAD@r269943.

Reimplement WRITE USING TOKEN with Block Zero token using WRITE SAME.

On my ZVOL of SSDs that increases speed of zero writing in that way from
1 to 2.5GB/s by reducing CPU overhead.
MFC after: 2 wee

Reimplement WRITE USING TOKEN with Block Zero token using WRITE SAME.

On my ZVOL of SSDs that increases speed of zero writing in that way from
1 to 2.5GB/s by reducing CPU overhead.
MFC after: 2 weeks

show more ...

Add support for Windows dialect of EXTENDED COPY command, aka Microsoft ODX.

This allows to avoid extra network traffic when copying files on NTFS iSCSI
disks within one storage host by drag'n'dropp

Add support for Windows dialect of EXTENDED COPY command, aka Microsoft ODX.

This allows to avoid extra network traffic when copying files on NTFS iSCSI
disks within one storage host by drag'n'dropping them in Windows Explorer
of Windows 8/2012. It should also accelerate Hyper-V VM operations, etc.

MFC after: 2 weeks
Sponsored by: iXsystems, Inc.

show more ...

Rework r269444 to work also for lists without IDs.

MFC after: 3 days

Plug EXTENDED COPY request data memory leak.

MFC after: 3 days

Fix some bugs in RECEIVE COPY STATUS data.

MFC after: 3 days