1/* $FreeBSD$ */ 2 3CTL - CAM Target Layer Description 4 5Revision 1.4 (December 29th, 2011) 6Ken Merry <ken@FreeBSD.org> 7 8Table of Contents: 9================= 10 11Introduction 12Features 13Configuring and Running CTL 14Revision 1.N Changes 15To Do List 16Code Roadmap 17Userland Commands 18 19Introduction: 20============ 21 22CTL is a disk, processor and cdrom device emulation subsystem originally 23written for Copan Systems under Linux starting in 2003. It has been 24shipping in Copan (now SGI) products since 2005. 25 26It was ported to FreeBSD in 2008, and thanks to an agreement between SGI 27(who acquired Copan's assets in 2010) and Spectra Logic in 2010, CTL is 28available under a BSD-style license. The intent behind the agreement was 29that Spectra would work to get CTL into the FreeBSD tree. 30 31Features: 32======== 33 34 - Disk, processor and cdrom device emulation. 35 - Tagged queueing 36 - SCSI task attribute support (ordered, head of queue, simple tags) 37 - SCSI implicit command ordering support. (e.g. if a read follows a mode 38 select, the read will be blocked until the mode select completes.) 39 - Full task management support (abort, LUN reset, target reset, etc.) 40 - Support for multiple ports 41 - Support for multiple simultaneous initiators 42 - Support for multiple simultaneous backing stores 43 - Support for VMWare VAAI: COMPARE AND WRITE, XCOPY, WRITE SAME and 44 UNMAP commands 45 - Support for Microsoft ODX: POPULATE TOKEN/WRITE USING TOKEN, WRITE SAME 46 and UNMAP commands 47 - Persistent reservation support 48 - Mode sense/select support 49 - Error injection support 50 - High Availability clustering support with ALUA 51 - All I/O handled in-kernel, no userland context switch overhead. 52 53Configuring and Running CTL: 54=========================== 55 56 - Add 'device ctl' to your kernel configuration file or load the module. 57 58 - If you're running with a 8Gb or 4Gb Qlogic FC board, add 59 'options ISP_TARGET_MODE' to your kernel config file. 'device ispfw' or 60 loading the ispfw module is also recommended. 61 62 - Rebuild and install a new kernel. 63 64 - Reboot with the new kernel. 65 66 - To add a LUN with the RAM disk backend: 67 68 ctladm create -b ramdisk -s 10485760000000000000 69 ctladm port -o on 70 71 - You should now see the CTL disk LUN through camcontrol devlist: 72 73scbus6 on ctl2cam0 bus 0: 74<FREEBSD CTLDISK 0001> at scbus6 target 1 lun 0 (da24,pass32) 75<> at scbus6 target -1 lun -1 () 76 77 This is visible through the CTL CAM SIM. This allows using CTL without 78 any physical hardware. You should be able to issue any normal SCSI 79 commands to the device via the pass(4)/da(4) devices. 80 81 If any target-capable HBAs are in the system (e.g. isp(4)), and have 82 target mode enabled, you should now also be able to see the CTL LUNs via 83 that target interface. 84 85 Note that all CTL LUNs are presented to all frontends. There is no 86 LUN masking, or separate, per-port configuration. 87 88 - Note that the ramdisk backend is a "fake" ramdisk. That is, it is 89 backed by a small amount of RAM that is used for all I/O requests. This 90 is useful for performance testing, but not for any data integrity tests. 91 92 - To add a LUN with the block/file backend: 93 94 truncate -s +1T myfile 95 ctladm create -b block -o file=myfile 96 ctladm port -o on 97 98 - You can also see a list of LUNs and their backends like this: 99 100# ctladm devlist 101LUN Backend Size (Blocks) BS Serial Number Device ID 102 0 block 2147483648 512 MYSERIAL 0 MYDEVID 0 103 1 block 2147483648 512 MYSERIAL 1 MYDEVID 1 104 2 block 2147483648 512 MYSERIAL 2 MYDEVID 2 105 3 block 2147483648 512 MYSERIAL 3 MYDEVID 3 106 4 block 2147483648 512 MYSERIAL 4 MYDEVID 4 107 5 block 2147483648 512 MYSERIAL 5 MYDEVID 5 108 6 block 2147483648 512 MYSERIAL 6 MYDEVID 6 109 7 block 2147483648 512 MYSERIAL 7 MYDEVID 7 110 8 block 2147483648 512 MYSERIAL 8 MYDEVID 8 111 9 block 2147483648 512 MYSERIAL 9 MYDEVID 9 112 10 block 2147483648 512 MYSERIAL 10 MYDEVID 10 113 11 block 2147483648 512 MYSERIAL 11 MYDEVID 11 114 115 - You can see the LUN type and backing store for block/file backend LUNs 116 like this: 117 118# ctladm devlist -v 119LUN Backend Size (Blocks) BS Serial Number Device ID 120 0 block 2147483648 512 MYSERIAL 0 MYDEVID 0 121 lun_type=0 122 num_threads=14 123 file=testdisk0 124 1 block 2147483648 512 MYSERIAL 1 MYDEVID 1 125 lun_type=0 126 num_threads=14 127 file=testdisk1 128 2 block 2147483648 512 MYSERIAL 2 MYDEVID 2 129 lun_type=0 130 num_threads=14 131 file=testdisk2 132 3 block 2147483648 512 MYSERIAL 3 MYDEVID 3 133 lun_type=0 134 num_threads=14 135 file=testdisk3 136 4 block 2147483648 512 MYSERIAL 4 MYDEVID 4 137 lun_type=0 138 num_threads=14 139 file=testdisk4 140 5 block 2147483648 512 MYSERIAL 5 MYDEVID 5 141 lun_type=0 142 num_threads=14 143 file=testdisk5 144 6 block 2147483648 512 MYSERIAL 6 MYDEVID 6 145 lun_type=0 146 num_threads=14 147 file=testdisk6 148 7 block 2147483648 512 MYSERIAL 7 MYDEVID 7 149 lun_type=0 150 num_threads=14 151 file=testdisk7 152 8 block 2147483648 512 MYSERIAL 8 MYDEVID 8 153 lun_type=0 154 num_threads=14 155 file=testdisk8 156 9 block 2147483648 512 MYSERIAL 9 MYDEVID 9 157 lun_type=0 158 num_threads=14 159 file=testdisk9 160 10 ramdisk 0 0 MYSERIAL 0 MYDEVID 0 161 lun_type=3 162 11 ramdisk 204800000000000 512 MYSERIAL 1 MYDEVID 1 163 lun_type=0 164 165 166Revision 1.4 Changes 167==================== 168 - Added in the second HA mode (where CTL does the data transfers instead 169 of having data transfers done below CTL), and abstracted out the Copan 170 HA API. 171 172 - Fixed the phantom device problem in the CTL CAM SIM and improved the 173 CAM SIM to automatically trigger a rescan when the port is enabled and 174 disabled. 175 176 - Made the number of threads in the block backend configurable via sysctl, 177 loader tunable and the ctladm command line. (You can now specify 178 -o num_threads=4 when creating a LUN with ctladm create.) 179 180 - Fixed some LUN selection issues in ctlstat(8) and allowed for selection 181 of LUN numbers up to 1023. 182 183 - General cleanup. 184 185 - This version intended for public release. 186 187Revision 1.3 Changes 188==================== 189 - Added descriptor sense support to CTL. It can be enabled through the 190 control mode page (10), but is disabled by default. 191 192 - Improved error injection support. The number of errors that can be 193 injected with 'ctladm inject' has been increased, and any arbitrary 194 sense data may now be injected as well. 195 196 - The port infrastructure has been revamped. Individual ports and types 197 of ports may now be enabled and disabled from the command line. ctladm 198 now has the ability to set the WWNN and WWPN for each port. 199 200 - The block backend can now send multiple I/Os to backing files. Multiple 201 writes are only allowed for ZFS, but multiple readers are allowed for 202 any filesystem. 203 204 - The block and ramdisk backends now support setting the LUN blocksize. 205 There are some restrictions when the backing device is a block device, 206 but otherwise the blocksize may be set to anything. 207 208Revision 1.2 Changes 209==================== 210 211 - CTL initialization process has been revamped. Instead of using an 212 ad-hoc method, it is now sequenced through SYSINIT() calls. 213 214 - A block/file backend has been added. This allows using arbitrary files 215 or block devices as a backing store. 216 217 - The userland LUN configuration interface has been completely rewritten. 218 Configuration is now done out of band. 219 220 - The ctladm(8) command line interface has been revamped, and is now 221 similar to camcontrol(8). 222 223To Do List: 224========== 225 226 - Use devstat(9) for CTL's statistics collection. CTL uses a home-grown 227 statistics collection system that is similar to devstat(9). ctlstat 228 should be retired in favor of iostat, etc., once aggregation modes are 229 available in iostat to match the behavior of ctlstat -t and dump modes 230 are available to match the behavior of ctlstat -d/ctlstat -J. 231 232 - ZFS ARC backend for CTL. Since ZFS copies all I/O into the ARC 233 (Adaptive Replacement Cache), running the block/file backend on top of a 234 ZFS-backed zdev or file will involve an extra set of copies. The 235 optimal solution for backing targets served by CTL with ZFS would be to 236 allocate buffers out of the ARC directly, and DMA to/from them directly. 237 That would eliminate an extra data buffer allocation and copy. 238 239 - Switch CTL over to using CAM CCBs instead of its own union ctl_io. This 240 will likely require a significant amount of work, but will eliminate 241 another data structure in the stack, more memory allocations, etc. This 242 will also require changes to the CAM CCB structure to support CTL. 243 244Code Roadmap: 245============ 246 247CTL has the concept of pluggable frontend ports and backends. All 248frontends and backends can be active at the same time. You can have a 249ramdisk-backed LUN present along side a file backed LUN. 250 251ctl.c: 252----- 253 254This is the core of CTL, where all of the command handlers and a lot of 255other things live. Yes, it is large. It started off small and grew to its 256current size over time. Perhaps it can be split into more files at some 257point. 258 259Here is a roadmap of some of the primary functions in ctl.c. Starting here 260and following the various leaf functions will show the command flow. 261 262ctl_queue() This is where commands from the frontend ports come 263 in. 264 265ctl_queue_sense() This is only used for non-packetized SCSI. i.e. 266 parallel SCSI prior to U320 and perhaps U160. 267 268ctl_work_thread() This is the primary work thread, and everything gets 269 executed from there. 270 271ctl_scsiio_precheck() This where all of the initial checks are done, and I/O 272 is either queued for execution or blocked. 273 274ctl_scsiio() This is where the command handler is actually 275 executed. (See ctl_cmd_table.c for the mapping of 276 SCSI opcode to command handler function.) 277 278ctl_done() This is the routine called (or ctl_done_lock()) to 279 initiate the command completion process. 280 281ctl_process_done() This is where command completion actually happens. 282 283ctl.h: 284----- 285 286Basic function declarations and data structures. 287 288ctl_backend.c, 289ctl_backend.h: 290------------- 291 292These files define the basic CTL backend API. The comments in the header 293explain the API. 294 295ctl_backend_block.c 296------------------- 297 298The block and file backend. This allows for using a disk or a file as the 299backing store for a LUN. Multiple threads are started to do I/O to the 300backing device, primarily because the VFS API requires that to get any 301concurrency. 302 303ctl_backend_ramdisk.c: 304--------------------- 305 306A "fake" ramdisk backend. It only allocates a small amount of memory to 307act as a source and sink for reads and writes from an initiator. Therefore 308it cannot be used for any real data, but it can be used to test for 309throughput. It can also be used to test initiators' support for extremely 310large LUNs. 311 312ctl_cmd_table.c: 313--------------- 314 315This is a table with all 256 possible SCSI opcodes, and command handler 316functions defined for supported opcodes. It is included in ctl.c. 317 318ctl_debug.h: 319----------- 320 321Simplistic debugging support. 322 323ctl_error.c, 324ctl_error.h: 325----------- 326 327CTL-specific wrappers around the CAM sense building functions. 328 329ctl_frontend.c, 330ctl_frontend.h: 331-------------- 332 333These files define the basic CTL frontend port API. The comments in the 334header explain the API. 335 336ctl_frontend_cam_sim.c: 337---------------------- 338 339This is a CTL frontend port that is also a CAM SIM. The idea is that this 340frontend allows for using CTL without any target-capable hardware. So any 341LUNs you create in CTL are visible via this port. 342 343ctl_ha.c: 344ctl_ha.h: 345-------- 346 347This is a High Availability API and TCP-based interlink implementation. 348 349ctl_io.h: 350-------- 351 352This defines most of the core CTL I/O structures. union ctl_io is 353conceptually very similar to CAM's union ccb. 354 355ctl_ioctl.h: 356----------- 357 358This defines all ioctls available through the CTL character device, and 359the data structures needed for those ioctls. 360 361ctl_private.h: 362------------- 363 364Private data structres (e.g. CTL softc) and function prototypes. This also 365includes the SCSI vendor and product names used by CTL. 366 367ctl_scsi_all.c 368ctl_scsi_all.h: 369-------------- 370 371CTL wrappers around CAM sense printing functions. 372 373ctl_ser_table.c: 374--------------- 375 376Command serialization table. This defines what happens when one type of 377command is followed by another type of command. e.g., what do you do when 378you have a mode select followed by a write? You block the write until the 379mode select is complete. That is defined in this table. 380 381ctl_util.c 382ctl_util.h: 383---------- 384 385CTL utility functions, primarily designed to be used from userland. See 386ctladm for the primary consumer of these functions. These include CDB 387building functions. 388 389scsi_ctl.c: 390---------- 391 392CAM target peripheral driver and CTL frontend port. This is the path into 393CTL for commands from target-capable hardware/SIMs. 394 395Userland Commands: 396================= 397 398ctladm(8) fills a role similar to camcontrol(8). It allow configuring LUNs, 399issuing commands, injecting errors and various other control functions. 400 401ctlstat(8) fills a role similar to iostat(8). It reports I/O statistics 402for CTL. 403