1/* $FreeBSD$ */ 2 3CTL - CAM Target Layer Description 4 5Revision 1.4 (December 29th, 2011) 6Ken Merry <ken@FreeBSD.org> 7 8Table of Contents: 9================= 10 11Introduction 12Features 13Configuring and Running CTL 14Revision 1.N Changes 15To Do List 16Code Roadmap 17Userland Commands 18 19Introduction: 20============ 21 22CTL is a disk and processor device emulation subsystem originally written 23for Copan Systems under Linux starting in 2003. It has been shipping in 24Copan (now SGI) products since 2005. 25 26It was ported to FreeBSD in 2008, and thanks to an agreement between SGI 27(who acquired Copan's assets in 2010) and Spectra Logic in 2010, CTL is 28available under a BSD-style license. The intent behind the agreement was 29that Spectra would work to get CTL into the FreeBSD tree. 30 31Features: 32======== 33 34 - Disk and processor device emulation. 35 - Tagged queueing 36 - SCSI task attribute support (ordered, head of queue, simple tags) 37 - SCSI implicit command ordering support. (e.g. if a read follows a mode 38 select, the read will be blocked until the mode select completes.) 39 - Full task management support (abort, LUN reset, target reset, etc.) 40 - Support for multiple ports 41 - Support for multiple simultaneous initiators 42 - Support for multiple simultaneous backing stores 43 - Persistent reservation support 44 - Mode sense/select support 45 - Error injection support 46 - High Availability support (1) 47 - All I/O handled in-kernel, no userland context switch overhead. 48 49(1) HA Support is just an API stub, and needs much more to be fully 50 functional. See the to-do list below. 51 52Configuring and Running CTL: 53=========================== 54 55 - After applying the CTL patchset to your tree, build world and install it 56 on your target system. 57 58 - Add 'device ctl' to your kernel configuration file. 59 60 - If you're running with a 8Gb or 4Gb Qlogic FC board, add 61 'options ISP_TARGET_MODE' to your kernel config file. Keep in mind that 62 the isp(4) driver can run in target or initiator mode, but not both on 63 the same machine. 'device ispfw' or loading the ispfw module is also 64 recommended. 65 66 - Rebuild and install a new kernel. 67 68 - Reboot with the new kernel. 69 70 - To add a LUN with the RAM disk backend: 71 72 ctladm create -b ramdisk -s 10485760000000000000 73 ctladm port -o on 74 75 - You should now see the CTL disk LUN through camcontrol devlist: 76 77scbus6 on ctl2cam0 bus 0: 78<FREEBSD CTLDISK 0001> at scbus6 target 1 lun 0 (da24,pass32) 79<> at scbus6 target -1 lun -1 () 80 81 This is visible through the CTL CAM SIM. This allows using CTL without 82 any physical hardware. You should be able to issue any normal SCSI 83 commands to the device via the pass(4)/da(4) devices. 84 85 If any target-capable HBAs are in the system (e.g. isp(4)), and have 86 target mode enabled, you should now also be able to see the CTL LUNs via 87 that target interface. 88 89 Note that all CTL LUNs are presented to all frontends. There is no 90 LUN masking, or separate, per-port configuration. 91 92 - Note that the ramdisk backend is a "fake" ramdisk. That is, it is 93 backed by a small amount of RAM that is used for all I/O requests. This 94 is useful for performance testing, but not for any data integrity tests. 95 96 - To add a LUN with the block/file backend: 97 98 truncate -s +1T myfile 99 ctladm create -b block -o file=myfile 100 ctladm port -o on 101 102 - You can also see a list of LUNs and their backends like this: 103 104# ctladm devlist 105LUN Backend Size (Blocks) BS Serial Number Device ID 106 0 block 2147483648 512 MYSERIAL 0 MYDEVID 0 107 1 block 2147483648 512 MYSERIAL 1 MYDEVID 1 108 2 block 2147483648 512 MYSERIAL 2 MYDEVID 2 109 3 block 2147483648 512 MYSERIAL 3 MYDEVID 3 110 4 block 2147483648 512 MYSERIAL 4 MYDEVID 4 111 5 block 2147483648 512 MYSERIAL 5 MYDEVID 5 112 6 block 2147483648 512 MYSERIAL 6 MYDEVID 6 113 7 block 2147483648 512 MYSERIAL 7 MYDEVID 7 114 8 block 2147483648 512 MYSERIAL 8 MYDEVID 8 115 9 block 2147483648 512 MYSERIAL 9 MYDEVID 9 116 10 block 2147483648 512 MYSERIAL 10 MYDEVID 10 117 11 block 2147483648 512 MYSERIAL 11 MYDEVID 11 118 119 - You can see the LUN type and backing store for block/file backend LUNs 120 like this: 121 122# ctladm devlist -v 123LUN Backend Size (Blocks) BS Serial Number Device ID 124 0 block 2147483648 512 MYSERIAL 0 MYDEVID 0 125 lun_type=0 126 num_threads=14 127 file=testdisk0 128 1 block 2147483648 512 MYSERIAL 1 MYDEVID 1 129 lun_type=0 130 num_threads=14 131 file=testdisk1 132 2 block 2147483648 512 MYSERIAL 2 MYDEVID 2 133 lun_type=0 134 num_threads=14 135 file=testdisk2 136 3 block 2147483648 512 MYSERIAL 3 MYDEVID 3 137 lun_type=0 138 num_threads=14 139 file=testdisk3 140 4 block 2147483648 512 MYSERIAL 4 MYDEVID 4 141 lun_type=0 142 num_threads=14 143 file=testdisk4 144 5 block 2147483648 512 MYSERIAL 5 MYDEVID 5 145 lun_type=0 146 num_threads=14 147 file=testdisk5 148 6 block 2147483648 512 MYSERIAL 6 MYDEVID 6 149 lun_type=0 150 num_threads=14 151 file=testdisk6 152 7 block 2147483648 512 MYSERIAL 7 MYDEVID 7 153 lun_type=0 154 num_threads=14 155 file=testdisk7 156 8 block 2147483648 512 MYSERIAL 8 MYDEVID 8 157 lun_type=0 158 num_threads=14 159 file=testdisk8 160 9 block 2147483648 512 MYSERIAL 9 MYDEVID 9 161 lun_type=0 162 num_threads=14 163 file=testdisk9 164 10 ramdisk 0 0 MYSERIAL 0 MYDEVID 0 165 lun_type=3 166 11 ramdisk 204800000000000 512 MYSERIAL 1 MYDEVID 1 167 lun_type=0 168 169 170Revision 1.4 Changes 171==================== 172 - Added in the second HA mode (where CTL does the data transfers instead 173 of having data transfers done below CTL), and abstracted out the Copan 174 HA API. 175 176 - Fixed the phantom device problem in the CTL CAM SIM and improved the 177 CAM SIM to automatically trigger a rescan when the port is enabled and 178 disabled. 179 180 - Made the number of threads in the block backend configurable via sysctl, 181 loader tunable and the ctladm command line. (You can now specify 182 -o num_threads=4 when creating a LUN with ctladm create.) 183 184 - Fixed some LUN selection issues in ctlstat(8) and allowed for selection 185 of LUN numbers up to 1023. 186 187 - General cleanup. 188 189 - This version intended for public release. 190 191Revision 1.3 Changes 192==================== 193 - Added descriptor sense support to CTL. It can be enabled through the 194 control mode page (10), but is disabled by default. 195 196 - Improved error injection support. The number of errors that can be 197 injected with 'ctladm inject' has been increased, and any arbitrary 198 sense data may now be injected as well. 199 200 - The port infrastructure has been revamped. Individual ports and types 201 of ports may now be enabled and disabled from the command line. ctladm 202 now has the ability to set the WWNN and WWPN for each port. 203 204 - The block backend can now send multiple I/Os to backing files. Multiple 205 writes are only allowed for ZFS, but multiple readers are allowed for 206 any filesystem. 207 208 - The block and ramdisk backends now support setting the LUN blocksize. 209 There are some restrictions when the backing device is a block device, 210 but otherwise the blocksize may be set to anything. 211 212Revision 1.2 Changes 213==================== 214 215 - CTL initialization process has been revamped. Instead of using an 216 ad-hoc method, it is now sequenced through SYSINIT() calls. 217 218 - A block/file backend has been added. This allows using arbitrary files 219 or block devices as a backing store. 220 221 - The userland LUN configuration interface has been completely rewritten. 222 Configuration is now done out of band. 223 224 - The ctladm(8) command line interface has been revamped, and is now 225 similar to camcontrol(8). 226 227To Do List: 228========== 229 230 - Make CTL buildable as a module. Work needs to be done on initialization, 231 and on freeing resources and LUNs when it is built as a module. 232 233 - Use devstat(9) for CTL's statistics collection. CTL uses a home-grown 234 statistics collection system that is similar to devstat(9). ctlstat 235 should be retired in favor of iostat, etc., once aggregation modes are 236 available in iostat to match the behavior of ctlstat -t and dump modes 237 are available to match the behavior of ctlstat -d/ctlstat -J. 238 239 - ZFS ARC backend for CTL. Since ZFS copies all I/O into the ARC 240 (Adaptive Replacement Cache), running the block/file backend on top of a 241 ZFS-backed zdev or file will involve an extra set of copies. The 242 optimal solution for backing targets served by CTL with ZFS would be to 243 allocate buffers out of the ARC directly, and DMA to/from them directly. 244 That would eliminate an extra data buffer allocation and copy. 245 246 - Switch CTL over to using CAM CCBs instead of its own union ctl_io. This 247 will likely require a significant amount of work, but will eliminate 248 another data structure in the stack, more memory allocations, etc. This 249 will also require changes to the CAM CCB structure to support CTL. 250 251 - Full-featured High Availability support. The HA API that is in ctl_ha.h 252 is essentially a renamed version of Copan's HA API. There is no 253 substance to it, but it remains in CTL to show what needs to be done to 254 implement active/active HA from a CTL standpoint. The things that would 255 need to be done include: 256 - A kernel level software API for message passing as well as DMA 257 between at least two nodes. 258 - Hardware support and drivers for inter-node communication. This 259 could be as simples as ethernet hardware and drivers. 260 - A "supervisor", or startup framework to control and coordinate 261 HA startup, failover (going from active/active to single mode), 262 and failback (going from single mode to active/active). 263 - HA support in other components of the stack. The goal behind HA 264 is that one node can fail and another node can seamlessly take 265 over handling I/O requests. This requires support from pretty 266 much every component in the storage stack, from top to bottom. 267 CTL is one piece of it, but you also need support in the RAID 268 stack/filesystem/backing store. You also need full configuration 269 mirroring, and all peer nodes need to be able to talk to the 270 underlying storage hardware. 271 272Code Roadmap: 273============ 274 275CTL has the concept of pluggable frontend ports and backends. All 276frontends and backends can be active at the same time. You can have a 277ramdisk-backed LUN present along side a file backed LUN. 278 279ctl.c: 280----- 281 282This is the core of CTL, where all of the command handlers and a lot of 283other things live. Yes, it is large. It started off small and grew to its 284current size over time. Perhaps it can be split into more files at some 285point. 286 287Here is a roadmap of some of the primary functions in ctl.c. Starting here 288and following the various leaf functions will show the command flow. 289 290ctl_queue() This is where commands from the frontend ports come 291 in. 292 293ctl_queue_sense() This is only used for non-packetized SCSI. i.e. 294 parallel SCSI prior to U320 and perhaps U160. 295 296ctl_work_thread() This is the primary work thread, and everything gets 297 executed from there. 298 299ctl_scsiio_precheck() This where all of the initial checks are done, and I/O 300 is either queued for execution or blocked. 301 302ctl_scsiio() This is where the command handler is actually 303 executed. (See ctl_cmd_table.c for the mapping of 304 SCSI opcode to command handler function.) 305 306ctl_done() This is the routine called (or ctl_done_lock()) to 307 initiate the command completion process. 308 309ctl_process_done() This is where command completion actually happens. 310 311ctl.h: 312----- 313 314Basic function declarations and data structures. 315 316ctl_backend.c, 317ctl_backend.h: 318------------- 319 320These files define the basic CTL backend API. The comments in the header 321explain the API. 322 323ctl_backend_block.c 324ctl_backend_block.h: 325------------------- 326 327The block and file backend. This allows for using a disk or a file as the 328backing store for a LUN. Multiple threads are started to do I/O to the 329backing device, primarily because the VFS API requires that to get any 330concurrency. 331 332ctl_backend_ramdisk.c: 333--------------------- 334 335A "fake" ramdisk backend. It only allocates a small amount of memory to 336act as a source and sink for reads and writes from an initiator. Therefore 337it cannot be used for any real data, but it can be used to test for 338throughput. It can also be used to test initiators' support for extremely 339large LUNs. 340 341ctl_cmd_table.c: 342--------------- 343 344This is a table with all 256 possible SCSI opcodes, and command handler 345functions defined for supported opcodes. It is included in ctl.c. 346 347ctl_debug.h: 348----------- 349 350Simplistic debugging support. 351 352ctl_error.c, 353ctl_error.h: 354----------- 355 356CTL-specific wrappers around the CAM sense building functions. 357 358ctl_frontend.c, 359ctl_frontend.h: 360-------------- 361 362These files define the basic CTL frontend port API. The comments in the 363header explain the API. 364 365ctl_frontend_cam_sim.c: 366---------------------- 367 368This is a CTL frontend port that is also a CAM SIM. The idea is that this 369frontend allows for using CTL without any target-capable hardware. So any 370LUNs you create in CTL are visible via this port. 371 372 373ctl_frontend_internal.c 374ctl_frontend_internal.h: 375----------------------- 376 377This is a frontend port written for Copan to do some system-specific tasks 378that required sending commands into CTL from inside the kernel. This isn't 379entirely relevant to FreeBSD in general, but can perhaps be repurposed or 380removed later. 381 382ctl_ha.h: 383-------- 384 385This is a stubbed-out High Availability API. See the comments in the 386header and the description of what is needed as far as HA support above. 387 388ctl_io.h: 389-------- 390 391This defines most of the core CTL I/O structures. union ctl_io is 392conceptually very similar to CAM's union ccb. 393 394ctl_ioctl.h: 395----------- 396 397This defines all ioctls available through the CTL character device, and 398the data structures needed for those ioctls. 399 400ctl_mem_pool.c 401ctl_mem_pool.h: 402-------------- 403 404Generic memory pool implementation. This is currently only used by the 405internal frontend. The internal frontend can probably be rewritten to use 406UMA zones and this can be removed. 407 408ctl_private.h: 409------------- 410 411Private data structres (e.g. CTL softc) and function prototypes. This also 412includes the SCSI vendor and product names used by CTL. 413 414ctl_scsi_all.c 415ctl_scsi_all.h: 416-------------- 417 418CTL wrappers around CAM sense printing functions. 419 420ctl_ser_table.c: 421--------------- 422 423Command serialization table. This defines what happens when one type of 424command is followed by another type of command. e.g., what do you do when 425you have a mode select followed by a write? You block the write until the 426mode select is complete. That is defined in this table. 427 428ctl_util.c 429ctl_util.h: 430---------- 431 432CTL utility functions, primarily designed to be used from userland. See 433ctladm for the primary consumer of these functions. These include CDB 434building functions. 435 436scsi_ctl.c: 437---------- 438 439CAM target peripheral driver and CTL frontend port. This is the path into 440CTL for commands from target-capable hardware/SIMs. 441 442Userland Commands: 443================= 444 445ctladm(8) fills a role similar to camcontrol(8). It allow configuring LUNs, 446issuing commands, injecting errors and various other control functions. 447 448ctlstat(8) fills a role similar to iostat(8). It reports I/O statistics 449for CTL. 450