xref: /freebsd/sys/cam/ctl/README.ctl.txt (revision c9dbb1cc52b063bbd9ab078a7afc89a8696da659)
1/* $FreeBSD$ */
2
3CTL - CAM Target Layer Description
4
5Revision 1.4 (December 29th, 2011)
6Ken Merry <ken@FreeBSD.org>
7
8Table of Contents:
9=================
10
11Introduction
12Features
13Configuring and Running CTL
14Revision 1.N Changes
15To Do List
16Code Roadmap
17Userland Commands
18
19Introduction:
20============
21
22CTL is a disk and processor device emulation subsystem originally written
23for Copan Systems under Linux starting in 2003.  It has been shipping in
24Copan (now SGI) products since 2005.
25
26It was ported to FreeBSD in 2008, and thanks to an agreement between SGI
27(who acquired Copan's assets in 2010) and Spectra Logic in 2010, CTL is
28available under a BSD-style license.  The intent behind the agreement was
29that Spectra would work to get CTL into the FreeBSD tree.
30
31Features:
32========
33
34 - Disk and processor device emulation.
35 - Tagged queueing
36 - SCSI task attribute support (ordered, head of queue, simple tags)
37 - SCSI implicit command ordering support.  (e.g. if a read follows a mode
38   select, the read will be blocked until the mode select completes.)
39 - Full task management support (abort, LUN reset, target reset, etc.)
40 - Support for multiple ports
41 - Support for multiple simultaneous initiators
42 - Support for multiple simultaneous backing stores
43 - Support for VMWare VAAI: COMPARE AND WRITE, XCOPY, WRITE SAME and
44   UNMAP commands
45 - Support for Microsoft ODX: POPULATE TOKEN/WRITE USING TOKEN, WRITE SAME
46   and UNMAP commands
47 - Persistent reservation support
48 - Mode sense/select support
49 - Error injection support
50 - High Availability clustering support with ALUA
51 - All I/O handled in-kernel, no userland context switch overhead.
52
53Configuring and Running CTL:
54===========================
55
56 - Add 'device ctl' to your kernel configuration file or load the module.
57
58 - If you're running with a 8Gb or 4Gb Qlogic FC board, add
59   'options ISP_TARGET_MODE' to your kernel config file. 'device ispfw' or
60   loading the ispfw module is also recommended.
61
62 - Rebuild and install a new kernel.
63
64 - Reboot with the new kernel.
65
66 - To add a LUN with the RAM disk backend:
67
68	ctladm create -b ramdisk -s 10485760000000000000
69	ctladm port -o on
70
71 - You should now see the CTL disk LUN through camcontrol devlist:
72
73scbus6 on ctl2cam0 bus 0:
74<FREEBSD CTLDISK 0001>             at scbus6 target 1 lun 0 (da24,pass32)
75<>                                 at scbus6 target -1 lun -1 ()
76
77   This is visible through the CTL CAM SIM.  This allows using CTL without
78   any physical hardware.  You should be able to issue any normal SCSI
79   commands to the device via the pass(4)/da(4) devices.
80
81   If any target-capable HBAs are in the system (e.g. isp(4)), and have
82   target mode enabled, you should now also be able to see the CTL LUNs via
83   that target interface.
84
85   Note that all CTL LUNs are presented to all frontends.  There is no
86   LUN masking, or separate, per-port configuration.
87
88 - Note that the ramdisk backend is a "fake" ramdisk.  That is, it is
89   backed by a small amount of RAM that is used for all I/O requests.  This
90   is useful for performance testing, but not for any data integrity tests.
91
92 - To add a LUN with the block/file backend:
93
94	truncate -s +1T myfile
95	ctladm create -b block -o file=myfile
96	ctladm port -o on
97
98 - You can also see a list of LUNs and their backends like this:
99
100# ctladm devlist
101LUN Backend       Size (Blocks)   BS Serial Number    Device ID
102  0 block            2147483648  512 MYSERIAL   0     MYDEVID   0
103  1 block            2147483648  512 MYSERIAL   1     MYDEVID   1
104  2 block            2147483648  512 MYSERIAL   2     MYDEVID   2
105  3 block            2147483648  512 MYSERIAL   3     MYDEVID   3
106  4 block            2147483648  512 MYSERIAL   4     MYDEVID   4
107  5 block            2147483648  512 MYSERIAL   5     MYDEVID   5
108  6 block            2147483648  512 MYSERIAL   6     MYDEVID   6
109  7 block            2147483648  512 MYSERIAL   7     MYDEVID   7
110  8 block            2147483648  512 MYSERIAL   8     MYDEVID   8
111  9 block            2147483648  512 MYSERIAL   9     MYDEVID   9
112 10 block            2147483648  512 MYSERIAL  10     MYDEVID  10
113 11 block            2147483648  512 MYSERIAL  11     MYDEVID  11
114
115 - You can see the LUN type and backing store for block/file backend LUNs
116   like this:
117
118# ctladm devlist -v
119LUN Backend       Size (Blocks)   BS Serial Number    Device ID
120  0 block            2147483648  512 MYSERIAL   0     MYDEVID   0
121      lun_type=0
122      num_threads=14
123      file=testdisk0
124  1 block            2147483648  512 MYSERIAL   1     MYDEVID   1
125      lun_type=0
126      num_threads=14
127      file=testdisk1
128  2 block            2147483648  512 MYSERIAL   2     MYDEVID   2
129      lun_type=0
130      num_threads=14
131      file=testdisk2
132  3 block            2147483648  512 MYSERIAL   3     MYDEVID   3
133      lun_type=0
134      num_threads=14
135      file=testdisk3
136  4 block            2147483648  512 MYSERIAL   4     MYDEVID   4
137      lun_type=0
138      num_threads=14
139      file=testdisk4
140  5 block            2147483648  512 MYSERIAL   5     MYDEVID   5
141      lun_type=0
142      num_threads=14
143      file=testdisk5
144  6 block            2147483648  512 MYSERIAL   6     MYDEVID   6
145      lun_type=0
146      num_threads=14
147      file=testdisk6
148  7 block            2147483648  512 MYSERIAL   7     MYDEVID   7
149      lun_type=0
150      num_threads=14
151      file=testdisk7
152  8 block            2147483648  512 MYSERIAL   8     MYDEVID   8
153      lun_type=0
154      num_threads=14
155      file=testdisk8
156  9 block            2147483648  512 MYSERIAL   9     MYDEVID   9
157      lun_type=0
158      num_threads=14
159      file=testdisk9
160 10 ramdisk                   0    0 MYSERIAL   0     MYDEVID   0
161      lun_type=3
162 11 ramdisk     204800000000000  512 MYSERIAL   1     MYDEVID   1
163      lun_type=0
164
165
166Revision 1.4 Changes
167====================
168 - Added in the second HA mode (where CTL does the data transfers instead
169   of having data transfers done below CTL), and abstracted out the Copan
170   HA API.
171
172 - Fixed the phantom device problem in the CTL CAM SIM and improved the
173   CAM SIM to automatically trigger a rescan when the port is enabled and
174   disabled.
175
176 - Made the number of threads in the block backend configurable via sysctl,
177   loader tunable and the ctladm command line.  (You can now specify
178   -o num_threads=4 when creating a LUN with ctladm create.)
179
180 - Fixed some LUN selection issues in ctlstat(8) and allowed for selection
181   of LUN numbers up to 1023.
182
183 - General cleanup.
184
185 - This version intended for public release.
186
187Revision 1.3 Changes
188====================
189 - Added descriptor sense support to CTL.  It can be enabled through the
190   control mode page (10), but is disabled by default.
191
192 - Improved error injection support.  The number of errors that can be
193   injected with 'ctladm inject' has been increased, and any arbitrary
194   sense data may now be injected as well.
195
196 - The port infrastructure has been revamped.  Individual ports and types
197   of ports may now be enabled and disabled from the command line.  ctladm
198   now has the ability to set the WWNN and WWPN for each port.
199
200 - The block backend can now send multiple I/Os to backing files.  Multiple
201   writes are only allowed for ZFS, but multiple readers are allowed for
202   any filesystem.
203
204 - The block and ramdisk backends now support setting the LUN blocksize.
205   There are some restrictions when the backing device is a block device,
206   but otherwise the blocksize may be set to anything.
207
208Revision 1.2 Changes
209====================
210
211 - CTL initialization process has been revamped.  Instead of using an
212   ad-hoc method, it is now sequenced through SYSINIT() calls.
213
214 - A block/file backend has been added.  This allows using arbitrary files
215   or block devices as a backing store.
216
217 - The userland LUN configuration interface has been completely rewritten.
218   Configuration is now done out of band.
219
220 - The ctladm(8) command line interface has been revamped, and is now
221   similar to camcontrol(8).
222
223To Do List:
224==========
225
226 - Use devstat(9) for CTL's statistics collection.  CTL uses a home-grown
227   statistics collection system that is similar to devstat(9).  ctlstat
228   should be retired in favor of iostat, etc., once aggregation modes are
229   available in iostat to match the behavior of ctlstat -t and dump modes
230   are available to match the behavior of ctlstat -d/ctlstat -J.
231
232 - ZFS ARC backend for CTL.  Since ZFS copies all I/O into the ARC
233   (Adaptive Replacement Cache), running the block/file backend on top of a
234   ZFS-backed zdev or file will involve an extra set of copies.  The
235   optimal solution for backing targets served by CTL with ZFS would be to
236   allocate buffers out of the ARC directly, and DMA to/from them directly.
237   That would eliminate an extra data buffer allocation and copy.
238
239 - Switch CTL over to using CAM CCBs instead of its own union ctl_io.  This
240   will likely require a significant amount of work, but will eliminate
241   another data structure in the stack, more memory allocations, etc.  This
242   will also require changes to the CAM CCB structure to support CTL.
243
244Code Roadmap:
245============
246
247CTL has the concept of pluggable frontend ports and backends.  All
248frontends and backends can be active at the same time.  You can have a
249ramdisk-backed LUN present along side a file backed LUN.
250
251ctl.c:
252-----
253
254This is the core of CTL, where all of the command handlers and a lot of
255other things live.  Yes, it is large.  It started off small and grew to its
256current size over time.  Perhaps it can be split into more files at some
257point.
258
259Here is a roadmap of some of the primary functions in ctl.c.  Starting here
260and following the various leaf functions will show the command flow.
261
262ctl_queue() 		This is where commands from the frontend ports come
263			in.
264
265ctl_queue_sense()	This is only used for non-packetized SCSI.  i.e.
266			parallel SCSI prior to U320 and perhaps U160.
267
268ctl_work_thread() 	This is the primary work thread, and everything gets
269			executed from there.
270
271ctl_scsiio_precheck() 	This where all of the initial checks are done, and I/O
272			is either queued for execution or blocked.
273
274ctl_scsiio() 		This is where the command handler is actually
275			executed.  (See ctl_cmd_table.c for the mapping of
276			SCSI opcode to command handler function.)
277
278ctl_done()		This is the routine called (or ctl_done_lock()) to
279			initiate the command completion process.
280
281ctl_process_done()	This is where command completion actually happens.
282
283ctl.h:
284-----
285
286Basic function declarations and data structures.
287
288ctl_backend.c,
289ctl_backend.h:
290-------------
291
292These files define the basic CTL backend API.  The comments in the header
293explain the API.
294
295ctl_backend_block.c
296-------------------
297
298The block and file backend.  This allows for using a disk or a file as the
299backing store for a LUN.  Multiple threads are started to do I/O to the
300backing device, primarily because the VFS API requires that to get any
301concurrency.
302
303ctl_backend_ramdisk.c:
304---------------------
305
306A "fake" ramdisk backend.  It only allocates a small amount of memory to
307act as a source and sink for reads and writes from an initiator.  Therefore
308it cannot be used for any real data, but it can be used to test for
309throughput.  It can also be used to test initiators' support for extremely
310large LUNs.
311
312ctl_cmd_table.c:
313---------------
314
315This is a table with all 256 possible SCSI opcodes, and command handler
316functions defined for supported opcodes.  It is included in ctl.c.
317
318ctl_debug.h:
319-----------
320
321Simplistic debugging support.
322
323ctl_error.c,
324ctl_error.h:
325-----------
326
327CTL-specific wrappers around the CAM sense building functions.
328
329ctl_frontend.c,
330ctl_frontend.h:
331--------------
332
333These files define the basic CTL frontend port API.  The comments in the
334header explain the API.
335
336ctl_frontend_cam_sim.c:
337----------------------
338
339This is a CTL frontend port that is also a CAM SIM.  The idea is that this
340frontend allows for using CTL without any target-capable hardware.  So any
341LUNs you create in CTL are visible via this port.
342
343ctl_ha.c:
344ctl_ha.h:
345--------
346
347This is a High Availability API and TCP-based interlink implementation.
348
349ctl_io.h:
350--------
351
352This defines most of the core CTL I/O structures.  union ctl_io is
353conceptually very similar to CAM's union ccb.
354
355ctl_ioctl.h:
356-----------
357
358This defines all ioctls available through the CTL character device, and
359the data structures needed for those ioctls.
360
361ctl_private.h:
362-------------
363
364Private data structres (e.g. CTL softc) and function prototypes.  This also
365includes the SCSI vendor and product names used by CTL.
366
367ctl_scsi_all.c
368ctl_scsi_all.h:
369--------------
370
371CTL wrappers around CAM sense printing functions.
372
373ctl_ser_table.c:
374---------------
375
376Command serialization table.  This defines what happens when one type of
377command is followed by another type of command.  e.g., what do you do when
378you have a mode select followed by a write?  You block the write until the
379mode select is complete.  That is defined in this table.
380
381ctl_util.c
382ctl_util.h:
383----------
384
385CTL utility functions, primarily designed to be used from userland.  See
386ctladm for the primary consumer of these functions.  These include CDB
387building functions.
388
389scsi_ctl.c:
390----------
391
392CAM target peripheral driver and CTL frontend port.  This is the path into
393CTL for commands from target-capable hardware/SIMs.
394
395Userland Commands:
396=================
397
398ctladm(8) fills a role similar to camcontrol(8).  It allow configuring LUNs,
399issuing commands, injecting errors and various other control functions.
400
401ctlstat(8) fills a role similar to iostat(8).  It reports I/O statistics
402for CTL.
403