xref: /freebsd/sys/cam/ctl/README.ctl.txt (revision 2710751bc309af25c6dea1171781678258e83840)
1/* $FreeBSD$ */
2
3CTL - CAM Target Layer Description
4
5Revision 1.4 (December 29th, 2011)
6Ken Merry <ken@FreeBSD.org>
7
8Table of Contents:
9=================
10
11Introduction
12Features
13Configuring and Running CTL
14Revision 1.N Changes
15To Do List
16Code Roadmap
17Userland Commands
18
19Introduction:
20============
21
22CTL is a disk and processor device emulation subsystem originally written
23for Copan Systems under Linux starting in 2003.  It has been shipping in
24Copan (now SGI) products since 2005.
25
26It was ported to FreeBSD in 2008, and thanks to an agreement between SGI
27(who acquired Copan's assets in 2010) and Spectra Logic in 2010, CTL is
28available under a BSD-style license.  The intent behind the agreement was
29that Spectra would work to get CTL into the FreeBSD tree.
30
31Features:
32========
33
34 - Disk and processor device emulation.
35 - Tagged queueing
36 - SCSI task attribute support (ordered, head of queue, simple tags)
37 - SCSI implicit command ordering support.  (e.g. if a read follows a mode
38   select, the read will be blocked until the mode select completes.)
39 - Full task management support (abort, LUN reset, target reset, etc.)
40 - Support for multiple ports
41 - Support for multiple simultaneous initiators
42 - Support for multiple simultaneous backing stores
43 - Persistent reservation support
44 - Mode sense/select support
45 - Error injection support
46 - High Availability support (1)
47 - All I/O handled in-kernel, no userland context switch overhead.
48
49(1) HA Support is just an API stub, and needs much more to be fully
50    functional.  See the to-do list below.
51
52Configuring and Running CTL:
53===========================
54
55 - After applying the CTL patchset to your tree, build world and install it
56   on your target system.
57
58 - Add 'device ctl' to your kernel configuration file.
59
60 - If you're running with a 8Gb or 4Gb Qlogic FC board, add
61   'options ISP_TARGET_MODE' to your kernel config file.  Keep in mind that
62   the isp(4) driver can run in target or initiator mode, but not both on
63   the same machine.  'device ispfw' or loading the ispfw module is also
64   recommended.
65
66 - Rebuild and install a new kernel.
67
68 - Reboot with the new kernel.
69
70 - To add a LUN with the RAM disk backend:
71
72	ctladm create -b ramdisk -s 10485760000000000000
73	ctladm port -o on
74
75 - You should now see the CTL disk LUN through camcontrol devlist:
76
77scbus6 on ctl2cam0 bus 0:
78<FREEBSD CTLDISK 0001>             at scbus6 target 1 lun 0 (da24,pass32)
79<>                                 at scbus6 target -1 lun -1 ()
80
81   This is visible through the CTL CAM SIM.  This allows using CTL without
82   any physical hardware.  You should be able to issue any normal SCSI
83   commands to the device via the pass(4)/da(4) devices.
84
85   If any target-capable HBAs are in the system (e.g. isp(4)), and have
86   target mode enabled, you should now also be able to see the CTL LUNs via
87   that target interface.
88
89   Note that all CTL LUNs are presented to all frontends.  There is no
90   LUN masking, or separate, per-port configuration.
91
92 - Note that the ramdisk backend is a "fake" ramdisk.  That is, it is
93   backed by a small amount of RAM that is used for all I/O requests.  This
94   is useful for performance testing, but not for any data integrity tests.
95
96 - To add a LUN with the block/file backend:
97
98	truncate -s +1T myfile
99	ctladm create -b block -o file=myfile
100	ctladm port -o on
101
102 - You can also see a list of LUNs and their backends like this:
103
104# ctladm devlist
105LUN Backend       Size (Blocks)   BS Serial Number    Device ID
106  0 block            2147483648  512 MYSERIAL   0     MYDEVID   0
107  1 block            2147483648  512 MYSERIAL   1     MYDEVID   1
108  2 block            2147483648  512 MYSERIAL   2     MYDEVID   2
109  3 block            2147483648  512 MYSERIAL   3     MYDEVID   3
110  4 block            2147483648  512 MYSERIAL   4     MYDEVID   4
111  5 block            2147483648  512 MYSERIAL   5     MYDEVID   5
112  6 block            2147483648  512 MYSERIAL   6     MYDEVID   6
113  7 block            2147483648  512 MYSERIAL   7     MYDEVID   7
114  8 block            2147483648  512 MYSERIAL   8     MYDEVID   8
115  9 block            2147483648  512 MYSERIAL   9     MYDEVID   9
116 10 block            2147483648  512 MYSERIAL  10     MYDEVID  10
117 11 block            2147483648  512 MYSERIAL  11     MYDEVID  11
118
119 - You can see the LUN type and backing store for block/file backend LUNs
120   like this:
121
122# ctladm devlist -v
123LUN Backend       Size (Blocks)   BS Serial Number    Device ID
124  0 block            2147483648  512 MYSERIAL   0     MYDEVID   0
125      lun_type=0
126      num_threads=14
127      file=testdisk0
128  1 block            2147483648  512 MYSERIAL   1     MYDEVID   1
129      lun_type=0
130      num_threads=14
131      file=testdisk1
132  2 block            2147483648  512 MYSERIAL   2     MYDEVID   2
133      lun_type=0
134      num_threads=14
135      file=testdisk2
136  3 block            2147483648  512 MYSERIAL   3     MYDEVID   3
137      lun_type=0
138      num_threads=14
139      file=testdisk3
140  4 block            2147483648  512 MYSERIAL   4     MYDEVID   4
141      lun_type=0
142      num_threads=14
143      file=testdisk4
144  5 block            2147483648  512 MYSERIAL   5     MYDEVID   5
145      lun_type=0
146      num_threads=14
147      file=testdisk5
148  6 block            2147483648  512 MYSERIAL   6     MYDEVID   6
149      lun_type=0
150      num_threads=14
151      file=testdisk6
152  7 block            2147483648  512 MYSERIAL   7     MYDEVID   7
153      lun_type=0
154      num_threads=14
155      file=testdisk7
156  8 block            2147483648  512 MYSERIAL   8     MYDEVID   8
157      lun_type=0
158      num_threads=14
159      file=testdisk8
160  9 block            2147483648  512 MYSERIAL   9     MYDEVID   9
161      lun_type=0
162      num_threads=14
163      file=testdisk9
164 10 ramdisk                   0    0 MYSERIAL   0     MYDEVID   0
165      lun_type=3
166 11 ramdisk     204800000000000  512 MYSERIAL   1     MYDEVID   1
167      lun_type=0
168
169
170Revision 1.4 Changes
171====================
172 - Added in the second HA mode (where CTL does the data transfers instead
173   of having data transfers done below CTL), and abstracted out the Copan
174   HA API.
175
176 - Fixed the phantom device problem in the CTL CAM SIM and improved the
177   CAM SIM to automatically trigger a rescan when the port is enabled and
178   disabled.
179
180 - Made the number of threads in the block backend configurable via sysctl,
181   loader tunable and the ctladm command line.  (You can now specify
182   -o num_threads=4 when creating a LUN with ctladm create.)
183
184 - Fixed some LUN selection issues in ctlstat(8) and allowed for selection
185   of LUN numbers up to 1023.
186
187 - General cleanup.
188
189 - This version intended for public release.
190
191Revision 1.3 Changes
192====================
193 - Added descriptor sense support to CTL.  It can be enabled through the
194   control mode page (10), but is disabled by default.
195
196 - Improved error injection support.  The number of errors that can be
197   injected with 'ctladm inject' has been increased, and any arbitrary
198   sense data may now be injected as well.
199
200 - The port infrastructure has been revamped.  Individual ports and types
201   of ports may now be enabled and disabled from the command line.  ctladm
202   now has the ability to set the WWNN and WWPN for each port.
203
204 - The block backend can now send multiple I/Os to backing files.  Multiple
205   writes are only allowed for ZFS, but multiple readers are allowed for
206   any filesystem.
207
208 - The block and ramdisk backends now support setting the LUN blocksize.
209   There are some restrictions when the backing device is a block device,
210   but otherwise the blocksize may be set to anything.
211
212Revision 1.2 Changes
213====================
214
215 - CTL initialization process has been revamped.  Instead of using an
216   ad-hoc method, it is now sequenced through SYSINIT() calls.
217
218 - A block/file backend has been added.  This allows using arbitrary files
219   or block devices as a backing store.
220
221 - The userland LUN configuration interface has been completely rewritten.
222   Configuration is now done out of band.
223
224 - The ctladm(8) command line interface has been revamped, and is now
225   similar to camcontrol(8).
226
227To Do List:
228==========
229
230 - Use devstat(9) for CTL's statistics collection.  CTL uses a home-grown
231   statistics collection system that is similar to devstat(9).  ctlstat
232   should be retired in favor of iostat, etc., once aggregation modes are
233   available in iostat to match the behavior of ctlstat -t and dump modes
234   are available to match the behavior of ctlstat -d/ctlstat -J.
235
236 - ZFS ARC backend for CTL.  Since ZFS copies all I/O into the ARC
237   (Adaptive Replacement Cache), running the block/file backend on top of a
238   ZFS-backed zdev or file will involve an extra set of copies.  The
239   optimal solution for backing targets served by CTL with ZFS would be to
240   allocate buffers out of the ARC directly, and DMA to/from them directly.
241   That would eliminate an extra data buffer allocation and copy.
242
243 - Switch CTL over to using CAM CCBs instead of its own union ctl_io.  This
244   will likely require a significant amount of work, but will eliminate
245   another data structure in the stack, more memory allocations, etc.  This
246   will also require changes to the CAM CCB structure to support CTL.
247
248 - Full-featured High Availability support.  The HA API that is in ctl_ha.h
249   is essentially a renamed version of Copan's HA API.  There is no
250   substance to it, but it remains in CTL to show what needs to be done to
251   implement active/active HA from a CTL standpoint.  The things that would
252   need to be done include:
253	- A kernel level software API for message passing as well as DMA
254	  between at least two nodes.
255	- Hardware support and drivers for inter-node communication.  This
256	  could be as simples as ethernet hardware and drivers.
257	- A "supervisor", or startup framework to control and coordinate
258	  HA startup, failover (going from active/active to single mode),
259	  and failback (going from single mode to active/active).
260	- HA support in other components of the stack.  The goal behind HA
261	  is that one node can fail and another node can seamlessly take
262	  over handling I/O requests.  This requires support from pretty
263	  much every component in the storage stack, from top to bottom.
264	  CTL is one piece of it, but you also need support in the RAID
265	  stack/filesystem/backing store.  You also need full configuration
266	  mirroring, and all peer nodes need to be able to talk to the
267	  underlying storage hardware.
268
269Code Roadmap:
270============
271
272CTL has the concept of pluggable frontend ports and backends.  All
273frontends and backends can be active at the same time.  You can have a
274ramdisk-backed LUN present along side a file backed LUN.
275
276ctl.c:
277-----
278
279This is the core of CTL, where all of the command handlers and a lot of
280other things live.  Yes, it is large.  It started off small and grew to its
281current size over time.  Perhaps it can be split into more files at some
282point.
283
284Here is a roadmap of some of the primary functions in ctl.c.  Starting here
285and following the various leaf functions will show the command flow.
286
287ctl_queue() 		This is where commands from the frontend ports come
288			in.
289
290ctl_queue_sense()	This is only used for non-packetized SCSI.  i.e.
291			parallel SCSI prior to U320 and perhaps U160.
292
293ctl_work_thread() 	This is the primary work thread, and everything gets
294			executed from there.
295
296ctl_scsiio_precheck() 	This where all of the initial checks are done, and I/O
297			is either queued for execution or blocked.
298
299ctl_scsiio() 		This is where the command handler is actually
300			executed.  (See ctl_cmd_table.c for the mapping of
301			SCSI opcode to command handler function.)
302
303ctl_done()		This is the routine called (or ctl_done_lock()) to
304			initiate the command completion process.
305
306ctl_process_done()	This is where command completion actually happens.
307
308ctl.h:
309-----
310
311Basic function declarations and data structures.
312
313ctl_backend.c,
314ctl_backend.h:
315-------------
316
317These files define the basic CTL backend API.  The comments in the header
318explain the API.
319
320ctl_backend_block.c
321ctl_backend_block.h:
322-------------------
323
324The block and file backend.  This allows for using a disk or a file as the
325backing store for a LUN.  Multiple threads are started to do I/O to the
326backing device, primarily because the VFS API requires that to get any
327concurrency.
328
329ctl_backend_ramdisk.c:
330---------------------
331
332A "fake" ramdisk backend.  It only allocates a small amount of memory to
333act as a source and sink for reads and writes from an initiator.  Therefore
334it cannot be used for any real data, but it can be used to test for
335throughput.  It can also be used to test initiators' support for extremely
336large LUNs.
337
338ctl_cmd_table.c:
339---------------
340
341This is a table with all 256 possible SCSI opcodes, and command handler
342functions defined for supported opcodes.  It is included in ctl.c.
343
344ctl_debug.h:
345-----------
346
347Simplistic debugging support.
348
349ctl_error.c,
350ctl_error.h:
351-----------
352
353CTL-specific wrappers around the CAM sense building functions.
354
355ctl_frontend.c,
356ctl_frontend.h:
357--------------
358
359These files define the basic CTL frontend port API.  The comments in the
360header explain the API.
361
362ctl_frontend_cam_sim.c:
363----------------------
364
365This is a CTL frontend port that is also a CAM SIM.  The idea is that this
366frontend allows for using CTL without any target-capable hardware.  So any
367LUNs you create in CTL are visible via this port.
368
369
370ctl_frontend_internal.c
371ctl_frontend_internal.h:
372-----------------------
373
374This is a frontend port written for Copan to do some system-specific tasks
375that required sending commands into CTL from inside the kernel.  This isn't
376entirely relevant to FreeBSD in general, but can perhaps be repurposed or
377removed later.
378
379ctl_ha.h:
380--------
381
382This is a stubbed-out High Availability API.  See the comments in the
383header and the description of what is needed as far as HA support above.
384
385ctl_io.h:
386--------
387
388This defines most of the core CTL I/O structures.  union ctl_io is
389conceptually very similar to CAM's union ccb.
390
391ctl_ioctl.h:
392-----------
393
394This defines all ioctls available through the CTL character device, and
395the data structures needed for those ioctls.
396
397ctl_mem_pool.c
398ctl_mem_pool.h:
399--------------
400
401Generic memory pool implementation.  This is currently only used by the
402internal frontend.  The internal frontend can probably be rewritten to use
403UMA zones and this can be removed.
404
405ctl_private.h:
406-------------
407
408Private data structres (e.g. CTL softc) and function prototypes.  This also
409includes the SCSI vendor and product names used by CTL.
410
411ctl_scsi_all.c
412ctl_scsi_all.h:
413--------------
414
415CTL wrappers around CAM sense printing functions.
416
417ctl_ser_table.c:
418---------------
419
420Command serialization table.  This defines what happens when one type of
421command is followed by another type of command.  e.g., what do you do when
422you have a mode select followed by a write?  You block the write until the
423mode select is complete.  That is defined in this table.
424
425ctl_util.c
426ctl_util.h:
427----------
428
429CTL utility functions, primarily designed to be used from userland.  See
430ctladm for the primary consumer of these functions.  These include CDB
431building functions.
432
433scsi_ctl.c:
434----------
435
436CAM target peripheral driver and CTL frontend port.  This is the path into
437CTL for commands from target-capable hardware/SIMs.
438
439Userland Commands:
440=================
441
442ctladm(8) fills a role similar to camcontrol(8).  It allow configuring LUNs,
443issuing commands, injecting errors and various other control functions.
444
445ctlstat(8) fills a role similar to iostat(8).  It reports I/O statistics
446for CTL.
447