1.. _kernel_docs: 2 3Index of Documentation for People Interested in Writing and/or Understanding the Linux Kernel 4============================================================================================= 5 6 Juan-Mariano de Goyeneche <jmseyas@dit.upm.es> 7 8The need for a document like this one became apparent in the 9linux-kernel mailing list as the same questions, asking for pointers 10to information, appeared again and again. 11 12Fortunately, as more and more people get to GNU/Linux, more and more 13get interested in the Kernel. But reading the sources is not always 14enough. It is easy to understand the code, but miss the concepts, the 15philosophy and design decisions behind this code. 16 17Unfortunately, not many documents are available for beginners to 18start. And, even if they exist, there was no "well-known" place which 19kept track of them. These lines try to cover this lack. All documents 20available on line known by the author are listed, while some reference 21books are also mentioned. 22 23PLEASE, if you know any paper not listed here or write a new document, 24send me an e-mail, and I'll include a reference to it here. Any 25corrections, ideas or comments are also welcomed. 26 27The papers that follow are listed in no particular order. All are 28cataloged with the following fields: the document's "Title", the 29"Author"/s, the "URL" where they can be found, some "Keywords" helpful 30when searching for specific topics, and a brief "Description" of the 31Document. 32 33Enjoy! 34 35.. note:: 36 37 The documents on each section of this document are ordered by its 38 published date, from the newest to the oldest. 39 40Docs at the Linux Kernel tree 41----------------------------- 42 43The Sphinx books should be built with ``make {htmldocs | pdfdocs | epubdocs}``. 44 45 * Name: **linux/Documentation** 46 47 :Author: Many. 48 :Location: Documentation/ 49 :Keywords: text files, Sphinx. 50 :Description: Documentation that comes with the kernel sources, 51 inside the Documentation directory. Some pages from this document 52 (including this document itself) have been moved there, and might 53 be more up to date than the web version. 54 55On-line docs 56------------ 57 58 * Title: **Linux Kernel Mailing List Glossary** 59 60 :Author: various 61 :URL: https://kernelnewbies.org/KernelGlossary 62 :Date: rolling version 63 :Keywords: glossary, terms, linux-kernel. 64 :Description: From the introduction: "This glossary is intended as 65 a brief description of some of the acronyms and terms you may hear 66 during discussion of the Linux kernel". 67 68 * Title: **Tracing the Way of Data in a TCP Connection through the Linux Kernel** 69 70 :Author: Richard Sailer 71 :URL: https://archive.org/details/linux_kernel_data_flow_short_paper 72 :Date: 2016 73 :Keywords: Linux Kernel Networking, TCP, tracing, ftrace 74 :Description: A seminar paper explaining ftrace and how to use it for 75 understanding linux kernel internals, 76 illustrated at tracing the way of a TCP packet through the kernel. 77 :Abstract: *This short paper outlines the usage of ftrace a tracing framework 78 as a tool to understand a running Linux system. 79 Having obtained a trace-log a kernel hacker can read and understand 80 source code more determined and with context. 81 In a detailed example this approach is demonstrated in tracing 82 and the way of data in a TCP Connection through the kernel. 83 Finally this trace-log is used as base for more a exact conceptual 84 exploration and description of the Linux TCP/IP implementation.* 85 86 * Title: **On submitting kernel Patches** 87 88 :Author: Andi Kleen 89 :URL: http://halobates.de/on-submitting-kernel-patches.pdf 90 :Date: 2008 91 :Keywords: patches, review process, types of submissions, basic rules, case studies 92 :Description: This paper gives several experience values on what types of patches 93 there are and how likley they get merged. 94 :Abstract: 95 [...]. This paper examines some common problems for 96 submitting larger changes and some strategies to avoid problems. 97 98 * Title: **Overview of the Virtual File System** 99 100 :Author: Richard Gooch. 101 :URL: http://www.mjmwired.net/kernel/Documentation/filesystems/vfs.txt 102 :Date: 2007 103 :Keywords: VFS, File System, mounting filesystems, opening files, 104 dentries, dcache. 105 :Description: Brief introduction to the Linux Virtual File System. 106 What is it, how it works, operations taken when opening a file or 107 mounting a file system and description of important data 108 structures explaining the purpose of each of their entries. 109 110 * Title: **Linux Device Drivers, Third Edition** 111 112 :Author: Jonathan Corbet, Alessandro Rubini, Greg Kroah-Hartman 113 :URL: http://lwn.net/Kernel/LDD3/ 114 :Date: 2005 115 :Description: A 600-page book covering the (2.6.10) driver 116 programming API and kernel hacking in general. Available under the 117 Creative Commons Attribution-ShareAlike 2.0 license. 118 :note: You can also :ref:`purchase a copy from O'Reilly or elsewhere <ldd3_published>`. 119 120 * Title: **Writing an ALSA Driver** 121 122 :Author: Takashi Iwai <tiwai@suse.de> 123 :URL: http://www.alsa-project.org/~iwai/writing-an-alsa-driver/index.html 124 :Date: 2005 125 :Keywords: ALSA, sound, soundcard, driver, lowlevel, hardware. 126 :Description: Advanced Linux Sound Architecture for developers, 127 both at kernel and user-level sides. ALSA is the Linux kernel 128 sound architecture in the 2.6 kernel version. 129 130 * Title: **Linux PCMCIA Programmer's Guide** 131 132 :Author: David Hinds. 133 :URL: http://pcmcia-cs.sourceforge.net/ftp/doc/PCMCIA-PROG.html 134 :Date: 2003 135 :Keywords: PCMCIA. 136 :Description: "This document describes how to write kernel device 137 drivers for the Linux PCMCIA Card Services interface. It also 138 describes how to write user-mode utilities for communicating with 139 Card Services. 140 141 * Title: **Linux Kernel Module Programming Guide** 142 143 :Author: Ori Pomerantz. 144 :URL: http://tldp.org/LDP/lkmpg/2.6/html/index.html 145 :Date: 2001 146 :Keywords: modules, GPL book, /proc, ioctls, system calls, 147 interrupt handlers . 148 :Description: Very nice 92 pages GPL book on the topic of modules 149 programming. Lots of examples. 150 151 * Title: **Global spinlock list and usage** 152 153 :Author: Rick Lindsley. 154 :URL: http://lse.sourceforge.net/lockhier/global-spin-lock 155 :Date: 2001 156 :Keywords: spinlock. 157 :Description: This is an attempt to document both the existence and 158 usage of the spinlocks in the Linux 2.4.5 kernel. Comprehensive 159 list of spinlocks showing when they are used, which functions 160 access them, how each lock is acquired, under what conditions it 161 is held, whether interrupts can occur or not while it is held... 162 163 * Title: **A Linux vm README** 164 165 :Author: Kanoj Sarcar. 166 :URL: http://kos.enix.org/pub/linux-vmm.html 167 :Date: 2001 168 :Keywords: virtual memory, mm, pgd, vma, page, page flags, page 169 cache, swap cache, kswapd. 170 :Description: Telegraphic, short descriptions and definitions 171 relating the Linux virtual memory implementation. 172 173 * Title: **Video4linux Drivers, Part 1: Video-Capture Device** 174 175 :Author: Alan Cox. 176 :URL: http://www.linux-mag.com/id/406 177 :Date: 2000 178 :Keywords: video4linux, driver, video capture, capture devices, 179 camera driver. 180 :Description: The title says it all. 181 182 * Title: **Video4linux Drivers, Part 2: Video-capture Devices** 183 184 :Author: Alan Cox. 185 :URL: http://www.linux-mag.com/id/429 186 :Date: 2000 187 :Keywords: video4linux, driver, video capture, capture devices, 188 camera driver, control, query capabilities, capability, facility. 189 :Description: The title says it all. 190 191 * Title: **Linux IP Networking. A Guide to the Implementation and Modification of the Linux Protocol Stack.** 192 193 :Author: Glenn Herrin. 194 :URL: http://www.cs.unh.edu/cnrg/gherrin 195 :Date: 2000 196 :Keywords: network, networking, protocol, IP, UDP, TCP, connection, 197 socket, receiving, transmitting, forwarding, routing, packets, 198 modules, /proc, sk_buff, FIB, tags. 199 :Description: Excellent paper devoted to the Linux IP Networking, 200 explaining anything from the kernel's to the user space 201 configuration tools' code. Very good to get a general overview of 202 the kernel networking implementation and understand all steps 203 packets follow from the time they are received at the network 204 device till they are delivered to applications. The studied kernel 205 code is from 2.2.14 version. Provides code for a working packet 206 dropper example. 207 208 * Title: **How To Make Sure Your Driver Will Work On The Power Macintosh** 209 210 :Author: Paul Mackerras. 211 :URL: http://www.linux-mag.com/id/261 212 :Date: 1999 213 :Keywords: Mac, Power Macintosh, porting, drivers, compatibility. 214 :Description: The title says it all. 215 216 * Title: **An Introduction to SCSI Drivers** 217 218 :Author: Alan Cox. 219 :URL: http://www.linux-mag.com/id/284 220 :Date: 1999 221 :Keywords: SCSI, device, driver. 222 :Description: The title says it all. 223 224 * Title: **Advanced SCSI Drivers And Other Tales** 225 226 :Author: Alan Cox. 227 :URL: http://www.linux-mag.com/id/307 228 :Date: 1999 229 :Keywords: SCSI, device, driver, advanced. 230 :Description: The title says it all. 231 232 * Title: **Writing Linux Mouse Drivers** 233 234 :Author: Alan Cox. 235 :URL: http://www.linux-mag.com/id/330 236 :Date: 1999 237 :Keywords: mouse, driver, gpm. 238 :Description: The title says it all. 239 240 * Title: **More on Mouse Drivers** 241 242 :Author: Alan Cox. 243 :URL: http://www.linux-mag.com/id/356 244 :Date: 1999 245 :Keywords: mouse, driver, gpm, races, asynchronous I/O. 246 :Description: The title still says it all. 247 248 * Title: **Writing Video4linux Radio Driver** 249 250 :Author: Alan Cox. 251 :URL: http://www.linux-mag.com/id/381 252 :Date: 1999 253 :Keywords: video4linux, driver, radio, radio devices. 254 :Description: The title says it all. 255 256 * Title: **I/O Event Handling Under Linux** 257 258 :Author: Richard Gooch. 259 :URL: http://web.mit.edu/~yandros/doc/io-events.html 260 :Date: 1999 261 :Keywords: IO, I/O, select(2), poll(2), FDs, aio_read(2), readiness 262 event queues. 263 :Description: From the Introduction: "I/O Event handling is about 264 how your Operating System allows you to manage a large number of 265 open files (file descriptors in UNIX/POSIX, or FDs) in your 266 application. You want the OS to notify you when FDs become active 267 (have data ready to be read or are ready for writing). Ideally you 268 want a mechanism that is scalable. This means a large number of 269 inactive FDs cost very little in memory and CPU time to manage". 270 271 * Title: **(nearly) Complete Linux Loadable Kernel Modules. The definitive guide for hackers, virus coders and system administrators.** 272 273 :Author: pragmatic/THC. 274 :URL: http://packetstormsecurity.org/docs/hack/LKM_HACKING.html 275 :Date: 1999 276 :Keywords: syscalls, intercept, hide, abuse, symbol table. 277 :Description: Interesting paper on how to abuse the Linux kernel in 278 order to intercept and modify syscalls, make 279 files/directories/processes invisible, become root, hijack ttys, 280 write kernel modules based virus... and solutions for admins to 281 avoid all those abuses. 282 :Notes: For 2.0.x kernels. Gives guidances to port it to 2.2.x 283 kernels. 284 285 * Name: **Linux Virtual File System** 286 287 :Author: Peter J. Braam. 288 :URL: http://www.coda.cs.cmu.edu/doc/talks/linuxvfs/ 289 :Date: 1998 290 :Keywords: slides, VFS, inode, superblock, dentry, dcache. 291 :Description: Set of slides, presumably from a presentation on the 292 Linux VFS layer. Covers version 2.1.x, with dentries and the 293 dcache. 294 295 * Title: **The Venus kernel interface** 296 297 :Author: Peter J. Braam. 298 :URL: http://www.coda.cs.cmu.edu/doc/html/kernel-venus-protocol.html 299 :Date: 1998 300 :Keywords: coda, filesystem, venus, cache manager. 301 :Description: "This document describes the communication between 302 Venus and kernel level file system code needed for the operation 303 of the Coda filesystem. This version document is meant to describe 304 the current interface (version 1.0) as well as improvements we 305 envisage". 306 307 * Title: **Design and Implementation of the Second Extended Filesystem** 308 309 :Author: Rémy Card, Theodore Ts'o, Stephen Tweedie. 310 :URL: http://web.mit.edu/tytso/www/linux/ext2intro.html 311 :Date: 1998 312 :Keywords: ext2, linux fs history, inode, directory, link, devices, 313 VFS, physical structure, performance, benchmarks, ext2fs library, 314 ext2fs tools, e2fsck. 315 :Description: Paper written by three of the top ext2 hackers. 316 Covers Linux filesystems history, ext2 motivation, ext2 features, 317 design, physical structure on disk, performance, benchmarks, 318 e2fsck's passes description... A must read! 319 :Notes: This paper was first published in the Proceedings of the 320 First Dutch International Symposium on Linux, ISBN 90-367-0385-9. 321 322 * Title: **The Linux RAID-1, 4, 5 Code** 323 324 :Author: Ingo Molnar, Gadi Oxman and Miguel de Icaza. 325 :URL: http://www.linuxjournal.com/article.php?sid=2391 326 :Date: 1997 327 :Keywords: RAID, MD driver. 328 :Description: Linux Journal Kernel Korner article. Here is its 329 :Abstract: *A description of the implementation of the RAID-1, 330 RAID-4 and RAID-5 personalities of the MD device driver in the 331 Linux kernel, providing users with high performance and reliable, 332 secondary-storage capability using software*. 333 334 * Title: **Linux Kernel Hackers' Guide** 335 336 :Author: Michael K. Johnson. 337 :URL: http://www.tldp.org/LDP/khg/HyperNews/get/khg.html 338 :Date: 1997 339 :Keywords: device drivers, files, VFS, kernel interface, character vs 340 block devices, hardware interrupts, scsi, DMA, access to user memory, 341 memory allocation, timers. 342 :Description: A guide designed to help you get up to speed on the 343 concepts that are not intuitevly obvious, and to document the internal 344 structures of Linux. 345 346 * Title: **Dynamic Kernels: Modularized Device Drivers** 347 348 :Author: Alessandro Rubini. 349 :URL: http://www.linuxjournal.com/article.php?sid=1219 350 :Date: 1996 351 :Keywords: device driver, module, loading/unloading modules, 352 allocating resources. 353 :Description: Linux Journal Kernel Korner article. Here is its 354 :Abstract: *This is the first of a series of four articles 355 co-authored by Alessandro Rubini and Georg Zezchwitz which present 356 a practical approach to writing Linux device drivers as kernel 357 loadable modules. This installment presents an introduction to the 358 topic, preparing the reader to understand next month's 359 installment*. 360 361 * Title: **Dynamic Kernels: Discovery** 362 363 :Author: Alessandro Rubini. 364 :URL: http://www.linuxjournal.com/article.php?sid=1220 365 :Date: 1996 366 :Keywords: character driver, init_module, clean_up module, 367 autodetection, mayor number, minor number, file operations, 368 open(), close(). 369 :Description: Linux Journal Kernel Korner article. Here is its 370 :Abstract: *This article, the second of four, introduces part of 371 the actual code to create custom module implementing a character 372 device driver. It describes the code for module initialization and 373 cleanup, as well as the open() and close() system calls*. 374 375 * Title: **The Devil's in the Details** 376 377 :Author: Georg v. Zezschwitz and Alessandro Rubini. 378 :URL: http://www.linuxjournal.com/article.php?sid=1221 379 :Date: 1996 380 :Keywords: read(), write(), select(), ioctl(), blocking/non 381 blocking mode, interrupt handler. 382 :Description: Linux Journal Kernel Korner article. Here is its 383 :Abstract: *This article, the third of four on writing character 384 device drivers, introduces concepts of reading, writing, and using 385 ioctl-calls*. 386 387 * Title: **Dissecting Interrupts and Browsing DMA** 388 389 :Author: Alessandro Rubini and Georg v. Zezschwitz. 390 :URL: http://www.linuxjournal.com/article.php?sid=1222 391 :Date: 1996 392 :Keywords: interrupts, irqs, DMA, bottom halves, task queues. 393 :Description: Linux Journal Kernel Korner article. Here is its 394 :Abstract: *This is the fourth in a series of articles about 395 writing character device drivers as loadable kernel modules. This 396 month, we further investigate the field of interrupt handling. 397 Though it is conceptually simple, practical limitations and 398 constraints make this an ''interesting'' part of device driver 399 writing, and several different facilities have been provided for 400 different situations. We also investigate the complex topic of 401 DMA*. 402 403 * Title: **Device Drivers Concluded** 404 405 :Author: Georg v. Zezschwitz. 406 :URL: http://www.linuxjournal.com/article.php?sid=1287 407 :Date: 1996 408 :Keywords: address spaces, pages, pagination, page management, 409 demand loading, swapping, memory protection, memory mapping, mmap, 410 virtual memory areas (VMAs), vremap, PCI. 411 :Description: Finally, the above turned out into a five articles 412 series. This latest one's introduction reads: "This is the last of 413 five articles about character device drivers. In this final 414 section, Georg deals with memory mapping devices, beginning with 415 an overall description of the Linux memory management concepts". 416 417 * Title: **Network Buffers And Memory Management** 418 419 :Author: Alan Cox. 420 :URL: http://www.linuxjournal.com/article.php?sid=1312 421 :Date: 1996 422 :Keywords: sk_buffs, network devices, protocol/link layer 423 variables, network devices flags, transmit, receive, 424 configuration, multicast. 425 :Description: Linux Journal Kernel Korner. 426 :Abstract: *Writing a network device driver for Linux is fundamentally 427 simple---most of the complexity (other than talking to the 428 hardware) involves managing network packets in memory*. 429 430 * Title: **Analysis of the Ext2fs structure** 431 432 :Author: Louis-Dominique Dubeau. 433 :URL: http://teaching.csse.uwa.edu.au/units/CITS2002/fs-ext2/ 434 :Date: 1994 435 :Keywords: ext2, filesystem, ext2fs. 436 :Description: Description of ext2's blocks, directories, inodes, 437 bitmaps, invariants... 438 439Published books 440--------------- 441 442 * Title: **Linux Treiber entwickeln** 443 444 :Author: Jürgen Quade, Eva-Katharina Kunst 445 :Publisher: dpunkt.verlag 446 :Date: Oct 2015 (4th edition) 447 :Pages: 688 448 :ISBN: 978-3-86490-288-8 449 :Note: German. The third edition from 2011 is 450 much cheaper and still quite up-to-date. 451 452 * Title: **Linux Kernel Networking: Implementation and Theory** 453 454 :Author: Rami Rosen 455 :Publisher: Apress 456 :Date: December 22, 2013 457 :Pages: 648 458 :ISBN: 978-1430261964 459 460 * Title: **Embedded Linux Primer: A practical Real-World Approach, 2nd Edition** 461 462 :Author: Christopher Hallinan 463 :Publisher: Pearson 464 :Date: November, 2010 465 :Pages: 656 466 :ISBN: 978-0137017836 467 468 * Title: **Linux Kernel Development, 3rd Edition** 469 470 :Author: Robert Love 471 :Publisher: Addison-Wesley 472 :Date: July, 2010 473 :Pages: 440 474 :ISBN: 978-0672329463 475 476 * Title: **Essential Linux Device Drivers** 477 478 :Author: Sreekrishnan Venkateswaran 479 :Published: Prentice Hall 480 :Date: April, 2008 481 :Pages: 744 482 :ISBN: 978-0132396554 483 484.. _ldd3_published: 485 486 * Title: **Linux Device Drivers, 3rd Edition** 487 488 :Authors: Jonathan Corbet, Alessandro Rubini, and Greg Kroah-Hartman 489 :Publisher: O'Reilly & Associates 490 :Date: 2005 491 :Pages: 636 492 :ISBN: 0-596-00590-3 493 :Notes: Further information in 494 http://www.oreilly.com/catalog/linuxdrive3/ 495 PDF format, URL: http://lwn.net/Kernel/LDD3/ 496 497 * Title: **Linux Kernel Internals** 498 499 :Author: Michael Beck 500 :Publisher: Addison-Wesley 501 :Date: 1997 502 :ISBN: 0-201-33143-8 (second edition) 503 504 * Title: **Programmation Linux 2.0 API systeme et fonctionnement du noyau** 505 506 :Author: Remy Card, Eric Dumas, Franck Mevel 507 :Publisher: Eyrolles 508 :Date: 1997 509 :Pages: 520 510 :ISBN: 2-212-08932-5 511 :Notes: French 512 513 * Title: **The Design and Implementation of the 4.4 BSD UNIX Operating System** 514 515 :Author: Marshall Kirk McKusick, Keith Bostic, Michael J. Karels, 516 John S. Quarterman 517 :Publisher: Addison-Wesley 518 :Date: 1996 519 :ISBN: 0-201-54979-4 520 521 * Title: **Unix internals -- the new frontiers** 522 523 :Author: Uresh Vahalia 524 :Publisher: Prentice Hall 525 :Date: 1996 526 :Pages: 600 527 :ISBN: 0-13-101908-2 528 529 * Title: **Programming for the real world - POSIX.4** 530 531 :Author: Bill O. Gallmeister 532 :Publisher: O'Reilly & Associates, Inc 533 :Date: 1995 534 :Pages: 552 535 :ISBN: I-56592-074-0 536 :Notes: Though not being directly about Linux, Linux aims to be 537 POSIX. Good reference. 538 539 * Title: **UNIX Systems for Modern Architectures: Symmetric Multiprocessing and Caching for Kernel Programmers** 540 541 :Author: Curt Schimmel 542 :Publisher: Addison Wesley 543 :Date: June, 1994 544 :Pages: 432 545 :ISBN: 0-201-63338-8 546 547 * Title: **The Design and Implementation of the 4.3 BSD UNIX Operating System** 548 549 :Author: Samuel J. Leffler, Marshall Kirk McKusick, Michael J 550 Karels, John S. Quarterman 551 :Publisher: Addison-Wesley 552 :Date: 1989 (reprinted with corrections on October, 1990) 553 :ISBN: 0-201-06196-1 554 555 * Title: **The Design of the UNIX Operating System** 556 557 :Author: Maurice J. Bach 558 :Publisher: Prentice Hall 559 :Date: 1986 560 :Pages: 471 561 :ISBN: 0-13-201757-1 562 563Miscellaneous 564------------- 565 566 * Name: **Cross-Referencing Linux** 567 568 :URL: http://lxr.free-electrons.com/ 569 :Keywords: Browsing source code. 570 :Description: Another web-based Linux kernel source code browser. 571 Lots of cross references to variables and functions. You can see 572 where they are defined and where they are used. 573 574 * Name: **Linux Weekly News** 575 576 :URL: http://lwn.net 577 :Keywords: latest kernel news. 578 :Description: The title says it all. There's a fixed kernel section 579 summarizing developers' work, bug fixes, new features and versions 580 produced during the week. Published every Thursday. 581 582 * Name: **The home page of Linux-MM** 583 584 :Author: The Linux-MM team. 585 :URL: http://linux-mm.org/ 586 :Keywords: memory management, Linux-MM, mm patches, TODO, docs, 587 mailing list. 588 :Description: Site devoted to Linux Memory Management development. 589 Memory related patches, HOWTOs, links, mm developers... Don't miss 590 it if you are interested in memory management development! 591 592 * Name: **Kernel Newbies IRC Channel and Website** 593 594 :URL: http://www.kernelnewbies.org 595 :Keywords: IRC, newbies, channel, asking doubts. 596 :Description: #kernelnewbies on irc.oftc.net. 597 #kernelnewbies is an IRC network dedicated to the 'newbie' 598 kernel hacker. The audience mostly consists of people who are 599 learning about the kernel, working on kernel projects or 600 professional kernel hackers that want to help less seasoned kernel 601 people. 602 #kernelnewbies is on the OFTC IRC Network. 603 Try irc.oftc.net as your server and then /join #kernelnewbies. 604 The kernelnewbies website also hosts articles, documents, FAQs... 605 606 * Name: **linux-kernel mailing list archives and search engines** 607 608 :URL: http://vger.kernel.org/vger-lists.html 609 :URL: http://www.uwsg.indiana.edu/hypermail/linux/kernel/index.html 610 :URL: http://groups.google.com/group/mlist.linux.kernel 611 :Keywords: linux-kernel, archives, search. 612 :Description: Some of the linux-kernel mailing list archivers. If 613 you have a better/another one, please let me know. 614 615------- 616 617Document last updated on Tue 2016-Sep-20 618 619This document is based on: 620 http://www.dit.upm.es/~jmseyas/linux/kernel/hackers-docs.html 621