118ccb223SMauro Carvalho Chehab.. SPDX-License-Identifier: GPL-2.0 218ccb223SMauro Carvalho Chehab 318ccb223SMauro Carvalho Chehab======== 418ccb223SMauro Carvalho ChehabORANGEFS 518ccb223SMauro Carvalho Chehab======== 618ccb223SMauro Carvalho Chehab 718ccb223SMauro Carvalho ChehabOrangeFS is an LGPL userspace scale-out parallel storage system. It is ideal 818ccb223SMauro Carvalho Chehabfor large storage problems faced by HPC, BigData, Streaming Video, 918ccb223SMauro Carvalho ChehabGenomics, Bioinformatics. 1018ccb223SMauro Carvalho Chehab 1118ccb223SMauro Carvalho ChehabOrangefs, originally called PVFS, was first developed in 1993 by 1218ccb223SMauro Carvalho ChehabWalt Ligon and Eric Blumer as a parallel file system for Parallel 1318ccb223SMauro Carvalho ChehabVirtual Machine (PVM) as part of a NASA grant to study the I/O patterns 1418ccb223SMauro Carvalho Chehabof parallel programs. 1518ccb223SMauro Carvalho Chehab 1618ccb223SMauro Carvalho ChehabOrangefs features include: 1718ccb223SMauro Carvalho Chehab 1818ccb223SMauro Carvalho Chehab * Distributes file data among multiple file servers 1918ccb223SMauro Carvalho Chehab * Supports simultaneous access by multiple clients 2018ccb223SMauro Carvalho Chehab * Stores file data and metadata on servers using local file system 2118ccb223SMauro Carvalho Chehab and access methods 2218ccb223SMauro Carvalho Chehab * Userspace implementation is easy to install and maintain 2318ccb223SMauro Carvalho Chehab * Direct MPI support 2418ccb223SMauro Carvalho Chehab * Stateless 2518ccb223SMauro Carvalho Chehab 2618ccb223SMauro Carvalho Chehab 2718ccb223SMauro Carvalho ChehabMailing List Archives 2818ccb223SMauro Carvalho Chehab===================== 2918ccb223SMauro Carvalho Chehab 3018ccb223SMauro Carvalho Chehabhttp://lists.orangefs.org/pipermail/devel_lists.orangefs.org/ 3118ccb223SMauro Carvalho Chehab 3218ccb223SMauro Carvalho Chehab 3318ccb223SMauro Carvalho ChehabMailing List Submissions 3418ccb223SMauro Carvalho Chehab======================== 3518ccb223SMauro Carvalho Chehab 3618ccb223SMauro Carvalho Chehabdevel@lists.orangefs.org 3718ccb223SMauro Carvalho Chehab 3818ccb223SMauro Carvalho Chehab 3918ccb223SMauro Carvalho ChehabDocumentation 4018ccb223SMauro Carvalho Chehab============= 4118ccb223SMauro Carvalho Chehab 4218ccb223SMauro Carvalho Chehabhttp://www.orangefs.org/documentation/ 4318ccb223SMauro Carvalho Chehab 4418ccb223SMauro Carvalho ChehabRunning ORANGEFS On a Single Server 4518ccb223SMauro Carvalho Chehab=================================== 4618ccb223SMauro Carvalho Chehab 4718ccb223SMauro Carvalho ChehabOrangeFS is usually run in large installations with multiple servers and 4818ccb223SMauro Carvalho Chehabclients, but a complete filesystem can be run on a single machine for 4918ccb223SMauro Carvalho Chehabdevelopment and testing. 5018ccb223SMauro Carvalho Chehab 5118ccb223SMauro Carvalho ChehabOn Fedora, install orangefs and orangefs-server:: 5218ccb223SMauro Carvalho Chehab 5318ccb223SMauro Carvalho Chehab dnf -y install orangefs orangefs-server 5418ccb223SMauro Carvalho Chehab 5518ccb223SMauro Carvalho ChehabThere is an example server configuration file in 5618ccb223SMauro Carvalho Chehab/etc/orangefs/orangefs.conf. Change localhost to your hostname if 5718ccb223SMauro Carvalho Chehabnecessary. 5818ccb223SMauro Carvalho Chehab 5918ccb223SMauro Carvalho ChehabTo generate a filesystem to run xfstests against, see below. 6018ccb223SMauro Carvalho Chehab 6118ccb223SMauro Carvalho ChehabThere is an example client configuration file in /etc/pvfs2tab. It is a 6218ccb223SMauro Carvalho Chehabsingle line. Uncomment it and change the hostname if necessary. This 6318ccb223SMauro Carvalho Chehabcontrols clients which use libpvfs2. This does not control the 6418ccb223SMauro Carvalho Chehabpvfs2-client-core. 6518ccb223SMauro Carvalho Chehab 6618ccb223SMauro Carvalho ChehabCreate the filesystem:: 6718ccb223SMauro Carvalho Chehab 6818ccb223SMauro Carvalho Chehab pvfs2-server -f /etc/orangefs/orangefs.conf 6918ccb223SMauro Carvalho Chehab 7018ccb223SMauro Carvalho ChehabStart the server:: 7118ccb223SMauro Carvalho Chehab 7218ccb223SMauro Carvalho Chehab systemctl start orangefs-server 7318ccb223SMauro Carvalho Chehab 7418ccb223SMauro Carvalho ChehabTest the server:: 7518ccb223SMauro Carvalho Chehab 7618ccb223SMauro Carvalho Chehab pvfs2-ping -m /pvfsmnt 7718ccb223SMauro Carvalho Chehab 7818ccb223SMauro Carvalho ChehabStart the client. The module must be compiled in or loaded before this 7918ccb223SMauro Carvalho Chehabpoint:: 8018ccb223SMauro Carvalho Chehab 8118ccb223SMauro Carvalho Chehab systemctl start orangefs-client 8218ccb223SMauro Carvalho Chehab 8318ccb223SMauro Carvalho ChehabMount the filesystem:: 8418ccb223SMauro Carvalho Chehab 8518ccb223SMauro Carvalho Chehab mount -t pvfs2 tcp://localhost:3334/orangefs /pvfsmnt 8618ccb223SMauro Carvalho Chehab 874e4bdcfaSLinus TorvaldsUserspace Filesystem Source 884e4bdcfaSLinus Torvalds=========================== 894e4bdcfaSLinus Torvalds 904e4bdcfaSLinus Torvaldshttp://www.orangefs.org/download 914e4bdcfaSLinus Torvalds 924e4bdcfaSLinus TorvaldsOrangefs versions prior to 2.9.3 would not be compatible with the 934e4bdcfaSLinus Torvaldsupstream version of the kernel client. 944e4bdcfaSLinus Torvalds 9518ccb223SMauro Carvalho Chehab 9618ccb223SMauro Carvalho ChehabBuilding ORANGEFS on a Single Server 9718ccb223SMauro Carvalho Chehab==================================== 9818ccb223SMauro Carvalho Chehab 9918ccb223SMauro Carvalho ChehabWhere OrangeFS cannot be installed from distribution packages, it may be 10018ccb223SMauro Carvalho Chehabbuilt from source. 10118ccb223SMauro Carvalho Chehab 10218ccb223SMauro Carvalho ChehabYou can omit --prefix if you don't care that things are sprinkled around 10318ccb223SMauro Carvalho Chehabin /usr/local. As of version 2.9.6, OrangeFS uses Berkeley DB by 10418ccb223SMauro Carvalho Chehabdefault, we will probably be changing the default to LMDB soon. 10518ccb223SMauro Carvalho Chehab 10618ccb223SMauro Carvalho Chehab:: 10718ccb223SMauro Carvalho Chehab 1084e4bdcfaSLinus Torvalds ./configure --prefix=/opt/ofs --with-db-backend=lmdb --disable-usrint 10918ccb223SMauro Carvalho Chehab 11018ccb223SMauro Carvalho Chehab make 11118ccb223SMauro Carvalho Chehab 11218ccb223SMauro Carvalho Chehab make install 11318ccb223SMauro Carvalho Chehab 1144e4bdcfaSLinus TorvaldsCreate an orangefs config file by running pvfs2-genconfig and 1154e4bdcfaSLinus Torvaldsspecifying a target config file. Pvfs2-genconfig will prompt you 1164e4bdcfaSLinus Torvaldsthrough. Generally it works fine to take the defaults, but you 1174e4bdcfaSLinus Torvaldsshould use your server's hostname, rather than "localhost" when 1184e4bdcfaSLinus Torvaldsit comes to that question:: 11918ccb223SMauro Carvalho Chehab 12018ccb223SMauro Carvalho Chehab /opt/ofs/bin/pvfs2-genconfig /etc/pvfs2.conf 12118ccb223SMauro Carvalho Chehab 122920af1ceSStephen KittCreate an /etc/pvfs2tab file (localhost is fine):: 1234e4bdcfaSLinus Torvalds 12418ccb223SMauro Carvalho Chehab echo tcp://localhost:3334/orangefs /pvfsmnt pvfs2 defaults,noauto 0 0 > \ 12518ccb223SMauro Carvalho Chehab /etc/pvfs2tab 12618ccb223SMauro Carvalho Chehab 12718ccb223SMauro Carvalho ChehabCreate the mount point you specified in the tab file if needed:: 12818ccb223SMauro Carvalho Chehab 12918ccb223SMauro Carvalho Chehab mkdir /pvfsmnt 13018ccb223SMauro Carvalho Chehab 13118ccb223SMauro Carvalho ChehabBootstrap the server:: 13218ccb223SMauro Carvalho Chehab 13318ccb223SMauro Carvalho Chehab /opt/ofs/sbin/pvfs2-server -f /etc/pvfs2.conf 13418ccb223SMauro Carvalho Chehab 13518ccb223SMauro Carvalho ChehabStart the server:: 13618ccb223SMauro Carvalho Chehab 1374e4bdcfaSLinus Torvalds /opt/ofs/sbin/pvfs2-server /etc/pvfs2.conf 13818ccb223SMauro Carvalho Chehab 13918ccb223SMauro Carvalho ChehabNow the server should be running. Pvfs2-ls is a simple 14018ccb223SMauro Carvalho Chehabtest to verify that the server is running:: 14118ccb223SMauro Carvalho Chehab 14218ccb223SMauro Carvalho Chehab /opt/ofs/bin/pvfs2-ls /pvfsmnt 14318ccb223SMauro Carvalho Chehab 14418ccb223SMauro Carvalho ChehabIf stuff seems to be working, load the kernel module and 14518ccb223SMauro Carvalho Chehabturn on the client core:: 14618ccb223SMauro Carvalho Chehab 1474e4bdcfaSLinus Torvalds /opt/ofs/sbin/pvfs2-client -p /opt/ofs/sbin/pvfs2-client-core 14818ccb223SMauro Carvalho Chehab 14918ccb223SMauro Carvalho ChehabMount your filesystem:: 15018ccb223SMauro Carvalho Chehab 1514e4bdcfaSLinus Torvalds mount -t pvfs2 tcp://`hostname`:3334/orangefs /pvfsmnt 15218ccb223SMauro Carvalho Chehab 15318ccb223SMauro Carvalho Chehab 15418ccb223SMauro Carvalho ChehabRunning xfstests 15518ccb223SMauro Carvalho Chehab================ 15618ccb223SMauro Carvalho Chehab 15718ccb223SMauro Carvalho ChehabIt is useful to use a scratch filesystem with xfstests. This can be 15818ccb223SMauro Carvalho Chehabdone with only one server. 15918ccb223SMauro Carvalho Chehab 16018ccb223SMauro Carvalho ChehabMake a second copy of the FileSystem section in the server configuration 16118ccb223SMauro Carvalho Chehabfile, which is /etc/orangefs/orangefs.conf. Change the Name to scratch. 16218ccb223SMauro Carvalho ChehabChange the ID to something other than the ID of the first FileSystem 16318ccb223SMauro Carvalho Chehabsection (2 is usually a good choice). 16418ccb223SMauro Carvalho Chehab 16518ccb223SMauro Carvalho ChehabThen there are two FileSystem sections: orangefs and scratch. 16618ccb223SMauro Carvalho Chehab 16718ccb223SMauro Carvalho ChehabThis change should be made before creating the filesystem. 16818ccb223SMauro Carvalho Chehab 16918ccb223SMauro Carvalho Chehab:: 17018ccb223SMauro Carvalho Chehab 17118ccb223SMauro Carvalho Chehab pvfs2-server -f /etc/orangefs/orangefs.conf 17218ccb223SMauro Carvalho Chehab 17318ccb223SMauro Carvalho ChehabTo run xfstests, create /etc/xfsqa.config:: 17418ccb223SMauro Carvalho Chehab 17518ccb223SMauro Carvalho Chehab TEST_DIR=/orangefs 17618ccb223SMauro Carvalho Chehab TEST_DEV=tcp://localhost:3334/orangefs 17718ccb223SMauro Carvalho Chehab SCRATCH_MNT=/scratch 17818ccb223SMauro Carvalho Chehab SCRATCH_DEV=tcp://localhost:3334/scratch 17918ccb223SMauro Carvalho Chehab 18018ccb223SMauro Carvalho ChehabThen xfstests can be run:: 18118ccb223SMauro Carvalho Chehab 18218ccb223SMauro Carvalho Chehab ./check -pvfs2 18318ccb223SMauro Carvalho Chehab 18418ccb223SMauro Carvalho Chehab 18518ccb223SMauro Carvalho ChehabOptions 18618ccb223SMauro Carvalho Chehab======= 18718ccb223SMauro Carvalho Chehab 18818ccb223SMauro Carvalho ChehabThe following mount options are accepted: 18918ccb223SMauro Carvalho Chehab 19018ccb223SMauro Carvalho Chehab acl 19118ccb223SMauro Carvalho Chehab Allow the use of Access Control Lists on files and directories. 19218ccb223SMauro Carvalho Chehab 19318ccb223SMauro Carvalho Chehab intr 19418ccb223SMauro Carvalho Chehab Some operations between the kernel client and the user space 19518ccb223SMauro Carvalho Chehab filesystem can be interruptible, such as changes in debug levels 19618ccb223SMauro Carvalho Chehab and the setting of tunable parameters. 19718ccb223SMauro Carvalho Chehab 19818ccb223SMauro Carvalho Chehab local_lock 19918ccb223SMauro Carvalho Chehab Enable posix locking from the perspective of "this" kernel. The 20018ccb223SMauro Carvalho Chehab default file_operations lock action is to return ENOSYS. Posix 20118ccb223SMauro Carvalho Chehab locking kicks in if the filesystem is mounted with -o local_lock. 20218ccb223SMauro Carvalho Chehab Distributed locking is being worked on for the future. 20318ccb223SMauro Carvalho Chehab 20418ccb223SMauro Carvalho Chehab 20518ccb223SMauro Carvalho ChehabDebugging 20618ccb223SMauro Carvalho Chehab========= 20718ccb223SMauro Carvalho Chehab 20818ccb223SMauro Carvalho ChehabIf you want the debug (GOSSIP) statements in a particular 20918ccb223SMauro Carvalho Chehabsource file (inode.c for example) go to syslog:: 21018ccb223SMauro Carvalho Chehab 21118ccb223SMauro Carvalho Chehab echo inode > /sys/kernel/debug/orangefs/kernel-debug 21218ccb223SMauro Carvalho Chehab 21318ccb223SMauro Carvalho ChehabNo debugging (the default):: 21418ccb223SMauro Carvalho Chehab 21518ccb223SMauro Carvalho Chehab echo none > /sys/kernel/debug/orangefs/kernel-debug 21618ccb223SMauro Carvalho Chehab 21718ccb223SMauro Carvalho ChehabDebugging from several source files:: 21818ccb223SMauro Carvalho Chehab 21918ccb223SMauro Carvalho Chehab echo inode,dir > /sys/kernel/debug/orangefs/kernel-debug 22018ccb223SMauro Carvalho Chehab 22118ccb223SMauro Carvalho ChehabAll debugging:: 22218ccb223SMauro Carvalho Chehab 22318ccb223SMauro Carvalho Chehab echo all > /sys/kernel/debug/orangefs/kernel-debug 22418ccb223SMauro Carvalho Chehab 22518ccb223SMauro Carvalho ChehabGet a list of all debugging keywords:: 22618ccb223SMauro Carvalho Chehab 22718ccb223SMauro Carvalho Chehab cat /sys/kernel/debug/orangefs/debug-help 22818ccb223SMauro Carvalho Chehab 22918ccb223SMauro Carvalho Chehab 23018ccb223SMauro Carvalho ChehabProtocol between Kernel Module and Userspace 23118ccb223SMauro Carvalho Chehab============================================ 23218ccb223SMauro Carvalho Chehab 23318ccb223SMauro Carvalho ChehabOrangefs is a user space filesystem and an associated kernel module. 23418ccb223SMauro Carvalho ChehabWe'll just refer to the user space part of Orangefs as "userspace" 23518ccb223SMauro Carvalho Chehabfrom here on out. Orangefs descends from PVFS, and userspace code 23618ccb223SMauro Carvalho Chehabstill uses PVFS for function and variable names. Userspace typedefs 23718ccb223SMauro Carvalho Chehabmany of the important structures. Function and variable names in 23818ccb223SMauro Carvalho Chehabthe kernel module have been transitioned to "orangefs", and The Linux 23918ccb223SMauro Carvalho ChehabCoding Style avoids typedefs, so kernel module structures that 24018ccb223SMauro Carvalho Chehabcorrespond to userspace structures are not typedefed. 24118ccb223SMauro Carvalho Chehab 24218ccb223SMauro Carvalho ChehabThe kernel module implements a pseudo device that userspace 24318ccb223SMauro Carvalho Chehabcan read from and write to. Userspace can also manipulate the 24418ccb223SMauro Carvalho Chehabkernel module through the pseudo device with ioctl. 24518ccb223SMauro Carvalho Chehab 24618ccb223SMauro Carvalho ChehabThe Bufmap 24718ccb223SMauro Carvalho Chehab---------- 24818ccb223SMauro Carvalho Chehab 24918ccb223SMauro Carvalho ChehabAt startup userspace allocates two page-size-aligned (posix_memalign) 25018ccb223SMauro Carvalho Chehabmlocked memory buffers, one is used for IO and one is used for readdir 25118ccb223SMauro Carvalho Chehaboperations. The IO buffer is 41943040 bytes and the readdir buffer is 25218ccb223SMauro Carvalho Chehab4194304 bytes. Each buffer contains logical chunks, or partitions, and 25318ccb223SMauro Carvalho Chehaba pointer to each buffer is added to its own PVFS_dev_map_desc structure 25418ccb223SMauro Carvalho Chehabwhich also describes its total size, as well as the size and number of 25518ccb223SMauro Carvalho Chehabthe partitions. 25618ccb223SMauro Carvalho Chehab 25718ccb223SMauro Carvalho ChehabA pointer to the IO buffer's PVFS_dev_map_desc structure is sent to a 25818ccb223SMauro Carvalho Chehabmapping routine in the kernel module with an ioctl. The structure is 25918ccb223SMauro Carvalho Chehabcopied from user space to kernel space with copy_from_user and is used 26018ccb223SMauro Carvalho Chehabto initialize the kernel module's "bufmap" (struct orangefs_bufmap), which 26118ccb223SMauro Carvalho Chehabthen contains: 26218ccb223SMauro Carvalho Chehab 26318ccb223SMauro Carvalho Chehab * refcnt 26418ccb223SMauro Carvalho Chehab - a reference counter 26518ccb223SMauro Carvalho Chehab * desc_size - PVFS2_BUFMAP_DEFAULT_DESC_SIZE (4194304) - the IO buffer's 26618ccb223SMauro Carvalho Chehab partition size, which represents the filesystem's block size and 26718ccb223SMauro Carvalho Chehab is used for s_blocksize in super blocks. 26818ccb223SMauro Carvalho Chehab * desc_count - PVFS2_BUFMAP_DEFAULT_DESC_COUNT (10) - the number of 26918ccb223SMauro Carvalho Chehab partitions in the IO buffer. 27018ccb223SMauro Carvalho Chehab * desc_shift - log2(desc_size), used for s_blocksize_bits in super blocks. 27118ccb223SMauro Carvalho Chehab * total_size - the total size of the IO buffer. 27218ccb223SMauro Carvalho Chehab * page_count - the number of 4096 byte pages in the IO buffer. 27318ccb223SMauro Carvalho Chehab * page_array - a pointer to ``page_count * (sizeof(struct page*))`` bytes 27418ccb223SMauro Carvalho Chehab of kcalloced memory. This memory is used as an array of pointers 27518ccb223SMauro Carvalho Chehab to each of the pages in the IO buffer through a call to get_user_pages. 27618ccb223SMauro Carvalho Chehab * desc_array - a pointer to ``desc_count * (sizeof(struct orangefs_bufmap_desc))`` 277*d56b699dSBjorn Helgaas bytes of kcalloced memory. This memory is further initialized: 27818ccb223SMauro Carvalho Chehab 27918ccb223SMauro Carvalho Chehab user_desc is the kernel's copy of the IO buffer's ORANGEFS_dev_map_desc 28018ccb223SMauro Carvalho Chehab structure. user_desc->ptr points to the IO buffer. 28118ccb223SMauro Carvalho Chehab 28218ccb223SMauro Carvalho Chehab :: 28318ccb223SMauro Carvalho Chehab 28418ccb223SMauro Carvalho Chehab pages_per_desc = bufmap->desc_size / PAGE_SIZE 28518ccb223SMauro Carvalho Chehab offset = 0 28618ccb223SMauro Carvalho Chehab 28718ccb223SMauro Carvalho Chehab bufmap->desc_array[0].page_array = &bufmap->page_array[offset] 28818ccb223SMauro Carvalho Chehab bufmap->desc_array[0].array_count = pages_per_desc = 1024 28918ccb223SMauro Carvalho Chehab bufmap->desc_array[0].uaddr = (user_desc->ptr) + (0 * 1024 * 4096) 29018ccb223SMauro Carvalho Chehab offset += 1024 29118ccb223SMauro Carvalho Chehab . 29218ccb223SMauro Carvalho Chehab . 29318ccb223SMauro Carvalho Chehab . 29418ccb223SMauro Carvalho Chehab bufmap->desc_array[9].page_array = &bufmap->page_array[offset] 29518ccb223SMauro Carvalho Chehab bufmap->desc_array[9].array_count = pages_per_desc = 1024 29618ccb223SMauro Carvalho Chehab bufmap->desc_array[9].uaddr = (user_desc->ptr) + 29718ccb223SMauro Carvalho Chehab (9 * 1024 * 4096) 29818ccb223SMauro Carvalho Chehab offset += 1024 29918ccb223SMauro Carvalho Chehab 30018ccb223SMauro Carvalho Chehab * buffer_index_array - a desc_count sized array of ints, used to 30118ccb223SMauro Carvalho Chehab indicate which of the IO buffer's partitions are available to use. 30218ccb223SMauro Carvalho Chehab * buffer_index_lock - a spinlock to protect buffer_index_array during update. 30318ccb223SMauro Carvalho Chehab * readdir_index_array - a five (ORANGEFS_READDIR_DEFAULT_DESC_COUNT) element 30418ccb223SMauro Carvalho Chehab int array used to indicate which of the readdir buffer's partitions are 30518ccb223SMauro Carvalho Chehab available to use. 30618ccb223SMauro Carvalho Chehab * readdir_index_lock - a spinlock to protect readdir_index_array during 30718ccb223SMauro Carvalho Chehab update. 30818ccb223SMauro Carvalho Chehab 30918ccb223SMauro Carvalho ChehabOperations 31018ccb223SMauro Carvalho Chehab---------- 31118ccb223SMauro Carvalho Chehab 31218ccb223SMauro Carvalho ChehabThe kernel module builds an "op" (struct orangefs_kernel_op_s) when it 31318ccb223SMauro Carvalho Chehabneeds to communicate with userspace. Part of the op contains the "upcall" 31418ccb223SMauro Carvalho Chehabwhich expresses the request to userspace. Part of the op eventually 31518ccb223SMauro Carvalho Chehabcontains the "downcall" which expresses the results of the request. 31618ccb223SMauro Carvalho Chehab 31718ccb223SMauro Carvalho ChehabThe slab allocator is used to keep a cache of op structures handy. 31818ccb223SMauro Carvalho Chehab 31918ccb223SMauro Carvalho ChehabAt init time the kernel module defines and initializes a request list 32018ccb223SMauro Carvalho Chehaband an in_progress hash table to keep track of all the ops that are 32118ccb223SMauro Carvalho Chehabin flight at any given time. 32218ccb223SMauro Carvalho Chehab 32318ccb223SMauro Carvalho ChehabOps are stateful: 32418ccb223SMauro Carvalho Chehab 32518ccb223SMauro Carvalho Chehab * unknown 32618ccb223SMauro Carvalho Chehab - op was just initialized 32718ccb223SMauro Carvalho Chehab * waiting 32818ccb223SMauro Carvalho Chehab - op is on request_list (upward bound) 32918ccb223SMauro Carvalho Chehab * inprogr 33018ccb223SMauro Carvalho Chehab - op is in progress (waiting for downcall) 33118ccb223SMauro Carvalho Chehab * serviced 33218ccb223SMauro Carvalho Chehab - op has matching downcall; ok 33318ccb223SMauro Carvalho Chehab * purged 33418ccb223SMauro Carvalho Chehab - op has to start a timer since client-core 33518ccb223SMauro Carvalho Chehab exited uncleanly before servicing op 33618ccb223SMauro Carvalho Chehab * given up 33718ccb223SMauro Carvalho Chehab - submitter has given up waiting for it 33818ccb223SMauro Carvalho Chehab 33918ccb223SMauro Carvalho ChehabWhen some arbitrary userspace program needs to perform a 34018ccb223SMauro Carvalho Chehabfilesystem operation on Orangefs (readdir, I/O, create, whatever) 34118ccb223SMauro Carvalho Chehaban op structure is initialized and tagged with a distinguishing ID 34218ccb223SMauro Carvalho Chehabnumber. The upcall part of the op is filled out, and the op is 34318ccb223SMauro Carvalho Chehabpassed to the "service_operation" function. 34418ccb223SMauro Carvalho Chehab 34518ccb223SMauro Carvalho ChehabService_operation changes the op's state to "waiting", puts 34618ccb223SMauro Carvalho Chehabit on the request list, and signals the Orangefs file_operations.poll 34718ccb223SMauro Carvalho Chehabfunction through a wait queue. Userspace is polling the pseudo-device 34818ccb223SMauro Carvalho Chehaband thus becomes aware of the upcall request that needs to be read. 34918ccb223SMauro Carvalho Chehab 35018ccb223SMauro Carvalho ChehabWhen the Orangefs file_operations.read function is triggered, the 35118ccb223SMauro Carvalho Chehabrequest list is searched for an op that seems ready-to-process. 35218ccb223SMauro Carvalho ChehabThe op is removed from the request list. The tag from the op and 35318ccb223SMauro Carvalho Chehabthe filled-out upcall struct are copy_to_user'ed back to userspace. 35418ccb223SMauro Carvalho Chehab 35518ccb223SMauro Carvalho ChehabIf any of these (and some additional protocol) copy_to_users fail, 35618ccb223SMauro Carvalho Chehabthe op's state is set to "waiting" and the op is added back to 35718ccb223SMauro Carvalho Chehabthe request list. Otherwise, the op's state is changed to "in progress", 35818ccb223SMauro Carvalho Chehaband the op is hashed on its tag and put onto the end of a list in the 35918ccb223SMauro Carvalho Chehabin_progress hash table at the index the tag hashed to. 36018ccb223SMauro Carvalho Chehab 36118ccb223SMauro Carvalho ChehabWhen userspace has assembled the response to the upcall, it 36218ccb223SMauro Carvalho Chehabwrites the response, which includes the distinguishing tag, back to 36318ccb223SMauro Carvalho Chehabthe pseudo device in a series of io_vecs. This triggers the Orangefs 36418ccb223SMauro Carvalho Chehabfile_operations.write_iter function to find the op with the associated 36518ccb223SMauro Carvalho Chehabtag and remove it from the in_progress hash table. As long as the op's 36618ccb223SMauro Carvalho Chehabstate is not "canceled" or "given up", its state is set to "serviced". 36718ccb223SMauro Carvalho ChehabThe file_operations.write_iter function returns to the waiting vfs, 36818ccb223SMauro Carvalho Chehaband back to service_operation through wait_for_matching_downcall. 36918ccb223SMauro Carvalho Chehab 37018ccb223SMauro Carvalho ChehabService operation returns to its caller with the op's downcall 37118ccb223SMauro Carvalho Chehabpart (the response to the upcall) filled out. 37218ccb223SMauro Carvalho Chehab 37318ccb223SMauro Carvalho ChehabThe "client-core" is the bridge between the kernel module and 37418ccb223SMauro Carvalho Chehabuserspace. The client-core is a daemon. The client-core has an 37518ccb223SMauro Carvalho Chehabassociated watchdog daemon. If the client-core is ever signaled 37618ccb223SMauro Carvalho Chehabto die, the watchdog daemon restarts the client-core. Even though 37718ccb223SMauro Carvalho Chehabthe client-core is restarted "right away", there is a period of 37818ccb223SMauro Carvalho Chehabtime during such an event that the client-core is dead. A dead client-core 37918ccb223SMauro Carvalho Chehabcan't be triggered by the Orangefs file_operations.poll function. 38018ccb223SMauro Carvalho ChehabOps that pass through service_operation during a "dead spell" can timeout 38118ccb223SMauro Carvalho Chehabon the wait queue and one attempt is made to recycle them. Obviously, 38218ccb223SMauro Carvalho Chehabif the client-core stays dead too long, the arbitrary userspace processes 38318ccb223SMauro Carvalho Chehabtrying to use Orangefs will be negatively affected. Waiting ops 38418ccb223SMauro Carvalho Chehabthat can't be serviced will be removed from the request list and 38518ccb223SMauro Carvalho Chehabhave their states set to "given up". In-progress ops that can't 38618ccb223SMauro Carvalho Chehabbe serviced will be removed from the in_progress hash table and 38718ccb223SMauro Carvalho Chehabhave their states set to "given up". 38818ccb223SMauro Carvalho Chehab 38918ccb223SMauro Carvalho ChehabReaddir and I/O ops are atypical with respect to their payloads. 39018ccb223SMauro Carvalho Chehab 39118ccb223SMauro Carvalho Chehab - readdir ops use the smaller of the two pre-allocated pre-partitioned 39218ccb223SMauro Carvalho Chehab memory buffers. The readdir buffer is only available to userspace. 39318ccb223SMauro Carvalho Chehab The kernel module obtains an index to a free partition before launching 39418ccb223SMauro Carvalho Chehab a readdir op. Userspace deposits the results into the indexed partition 39518ccb223SMauro Carvalho Chehab and then writes them to back to the pvfs device. 39618ccb223SMauro Carvalho Chehab 39718ccb223SMauro Carvalho Chehab - io (read and write) ops use the larger of the two pre-allocated 39818ccb223SMauro Carvalho Chehab pre-partitioned memory buffers. The IO buffer is accessible from 39918ccb223SMauro Carvalho Chehab both userspace and the kernel module. The kernel module obtains an 40018ccb223SMauro Carvalho Chehab index to a free partition before launching an io op. The kernel module 40118ccb223SMauro Carvalho Chehab deposits write data into the indexed partition, to be consumed 40218ccb223SMauro Carvalho Chehab directly by userspace. Userspace deposits the results of read 40318ccb223SMauro Carvalho Chehab requests into the indexed partition, to be consumed directly 40418ccb223SMauro Carvalho Chehab by the kernel module. 40518ccb223SMauro Carvalho Chehab 40618ccb223SMauro Carvalho ChehabResponses to kernel requests are all packaged in pvfs2_downcall_t 40718ccb223SMauro Carvalho Chehabstructs. Besides a few other members, pvfs2_downcall_t contains a 40818ccb223SMauro Carvalho Chehabunion of structs, each of which is associated with a particular 40918ccb223SMauro Carvalho Chehabresponse type. 41018ccb223SMauro Carvalho Chehab 41118ccb223SMauro Carvalho ChehabThe several members outside of the union are: 41218ccb223SMauro Carvalho Chehab 41318ccb223SMauro Carvalho Chehab ``int32_t type`` 41418ccb223SMauro Carvalho Chehab - type of operation. 41518ccb223SMauro Carvalho Chehab ``int32_t status`` 41618ccb223SMauro Carvalho Chehab - return code for the operation. 41718ccb223SMauro Carvalho Chehab ``int64_t trailer_size`` 41818ccb223SMauro Carvalho Chehab - 0 unless readdir operation. 41918ccb223SMauro Carvalho Chehab ``char *trailer_buf`` 42018ccb223SMauro Carvalho Chehab - initialized to NULL, used during readdir operations. 42118ccb223SMauro Carvalho Chehab 42218ccb223SMauro Carvalho ChehabThe appropriate member inside the union is filled out for any 42318ccb223SMauro Carvalho Chehabparticular response. 42418ccb223SMauro Carvalho Chehab 42518ccb223SMauro Carvalho Chehab PVFS2_VFS_OP_FILE_IO 42618ccb223SMauro Carvalho Chehab fill a pvfs2_io_response_t 42718ccb223SMauro Carvalho Chehab 42818ccb223SMauro Carvalho Chehab PVFS2_VFS_OP_LOOKUP 42918ccb223SMauro Carvalho Chehab fill a PVFS_object_kref 43018ccb223SMauro Carvalho Chehab 43118ccb223SMauro Carvalho Chehab PVFS2_VFS_OP_CREATE 43218ccb223SMauro Carvalho Chehab fill a PVFS_object_kref 43318ccb223SMauro Carvalho Chehab 43418ccb223SMauro Carvalho Chehab PVFS2_VFS_OP_SYMLINK 43518ccb223SMauro Carvalho Chehab fill a PVFS_object_kref 43618ccb223SMauro Carvalho Chehab 43718ccb223SMauro Carvalho Chehab PVFS2_VFS_OP_GETATTR 43818ccb223SMauro Carvalho Chehab fill in a PVFS_sys_attr_s (tons of stuff the kernel doesn't need) 43918ccb223SMauro Carvalho Chehab fill in a string with the link target when the object is a symlink. 44018ccb223SMauro Carvalho Chehab 44118ccb223SMauro Carvalho Chehab PVFS2_VFS_OP_MKDIR 44218ccb223SMauro Carvalho Chehab fill a PVFS_object_kref 44318ccb223SMauro Carvalho Chehab 44418ccb223SMauro Carvalho Chehab PVFS2_VFS_OP_STATFS 44518ccb223SMauro Carvalho Chehab fill a pvfs2_statfs_response_t with useless info <g>. It is hard for 44618ccb223SMauro Carvalho Chehab us to know, in a timely fashion, these statistics about our 44718ccb223SMauro Carvalho Chehab distributed network filesystem. 44818ccb223SMauro Carvalho Chehab 44918ccb223SMauro Carvalho Chehab PVFS2_VFS_OP_FS_MOUNT 45018ccb223SMauro Carvalho Chehab fill a pvfs2_fs_mount_response_t which is just like a PVFS_object_kref 45118ccb223SMauro Carvalho Chehab except its members are in a different order and "__pad1" is replaced 45218ccb223SMauro Carvalho Chehab with "id". 45318ccb223SMauro Carvalho Chehab 45418ccb223SMauro Carvalho Chehab PVFS2_VFS_OP_GETXATTR 45518ccb223SMauro Carvalho Chehab fill a pvfs2_getxattr_response_t 45618ccb223SMauro Carvalho Chehab 45718ccb223SMauro Carvalho Chehab PVFS2_VFS_OP_LISTXATTR 45818ccb223SMauro Carvalho Chehab fill a pvfs2_listxattr_response_t 45918ccb223SMauro Carvalho Chehab 46018ccb223SMauro Carvalho Chehab PVFS2_VFS_OP_PARAM 46118ccb223SMauro Carvalho Chehab fill a pvfs2_param_response_t 46218ccb223SMauro Carvalho Chehab 46318ccb223SMauro Carvalho Chehab PVFS2_VFS_OP_PERF_COUNT 46418ccb223SMauro Carvalho Chehab fill a pvfs2_perf_count_response_t 46518ccb223SMauro Carvalho Chehab 46618ccb223SMauro Carvalho Chehab PVFS2_VFS_OP_FSKEY 46718ccb223SMauro Carvalho Chehab file a pvfs2_fs_key_response_t 46818ccb223SMauro Carvalho Chehab 46918ccb223SMauro Carvalho Chehab PVFS2_VFS_OP_READDIR 47018ccb223SMauro Carvalho Chehab jamb everything needed to represent a pvfs2_readdir_response_t into 47118ccb223SMauro Carvalho Chehab the readdir buffer descriptor specified in the upcall. 47218ccb223SMauro Carvalho Chehab 47318ccb223SMauro Carvalho ChehabUserspace uses writev() on /dev/pvfs2-req to pass responses to the requests 47418ccb223SMauro Carvalho Chehabmade by the kernel side. 47518ccb223SMauro Carvalho Chehab 47618ccb223SMauro Carvalho ChehabA buffer_list containing: 47718ccb223SMauro Carvalho Chehab 47818ccb223SMauro Carvalho Chehab - a pointer to the prepared response to the request from the 47918ccb223SMauro Carvalho Chehab kernel (struct pvfs2_downcall_t). 48018ccb223SMauro Carvalho Chehab - and also, in the case of a readdir request, a pointer to a 48118ccb223SMauro Carvalho Chehab buffer containing descriptors for the objects in the target 48218ccb223SMauro Carvalho Chehab directory. 48318ccb223SMauro Carvalho Chehab 48418ccb223SMauro Carvalho Chehab... is sent to the function (PINT_dev_write_list) which performs 48518ccb223SMauro Carvalho Chehabthe writev. 48618ccb223SMauro Carvalho Chehab 48718ccb223SMauro Carvalho ChehabPINT_dev_write_list has a local iovec array: struct iovec io_array[10]; 48818ccb223SMauro Carvalho Chehab 48918ccb223SMauro Carvalho ChehabThe first four elements of io_array are initialized like this for all 49018ccb223SMauro Carvalho Chehabresponses:: 49118ccb223SMauro Carvalho Chehab 49218ccb223SMauro Carvalho Chehab io_array[0].iov_base = address of local variable "proto_ver" (int32_t) 49318ccb223SMauro Carvalho Chehab io_array[0].iov_len = sizeof(int32_t) 49418ccb223SMauro Carvalho Chehab 49518ccb223SMauro Carvalho Chehab io_array[1].iov_base = address of global variable "pdev_magic" (int32_t) 49618ccb223SMauro Carvalho Chehab io_array[1].iov_len = sizeof(int32_t) 49718ccb223SMauro Carvalho Chehab 49818ccb223SMauro Carvalho Chehab io_array[2].iov_base = address of parameter "tag" (PVFS_id_gen_t) 49918ccb223SMauro Carvalho Chehab io_array[2].iov_len = sizeof(int64_t) 50018ccb223SMauro Carvalho Chehab 50118ccb223SMauro Carvalho Chehab io_array[3].iov_base = address of out_downcall member (pvfs2_downcall_t) 50218ccb223SMauro Carvalho Chehab of global variable vfs_request (vfs_request_t) 50318ccb223SMauro Carvalho Chehab io_array[3].iov_len = sizeof(pvfs2_downcall_t) 50418ccb223SMauro Carvalho Chehab 50518ccb223SMauro Carvalho ChehabReaddir responses initialize the fifth element io_array like this:: 50618ccb223SMauro Carvalho Chehab 50718ccb223SMauro Carvalho Chehab io_array[4].iov_base = contents of member trailer_buf (char *) 50818ccb223SMauro Carvalho Chehab from out_downcall member of global variable 50918ccb223SMauro Carvalho Chehab vfs_request 51018ccb223SMauro Carvalho Chehab io_array[4].iov_len = contents of member trailer_size (PVFS_size) 51118ccb223SMauro Carvalho Chehab from out_downcall member of global variable 51218ccb223SMauro Carvalho Chehab vfs_request 51318ccb223SMauro Carvalho Chehab 51418ccb223SMauro Carvalho ChehabOrangefs exploits the dcache in order to avoid sending redundant 51518ccb223SMauro Carvalho Chehabrequests to userspace. We keep object inode attributes up-to-date with 51618ccb223SMauro Carvalho Chehaborangefs_inode_getattr. Orangefs_inode_getattr uses two arguments to 51718ccb223SMauro Carvalho Chehabhelp it decide whether or not to update an inode: "new" and "bypass". 51818ccb223SMauro Carvalho ChehabOrangefs keeps private data in an object's inode that includes a short 51918ccb223SMauro Carvalho Chehabtimeout value, getattr_time, which allows any iteration of 52018ccb223SMauro Carvalho Chehaborangefs_inode_getattr to know how long it has been since the inode was 52118ccb223SMauro Carvalho Chehabupdated. When the object is not new (new == 0) and the bypass flag is not 52218ccb223SMauro Carvalho Chehabset (bypass == 0) orangefs_inode_getattr returns without updating the inode 52318ccb223SMauro Carvalho Chehabif getattr_time has not timed out. Getattr_time is updated each time the 52418ccb223SMauro Carvalho Chehabinode is updated. 52518ccb223SMauro Carvalho Chehab 52618ccb223SMauro Carvalho ChehabCreation of a new object (file, dir, sym-link) includes the evaluation of 52718ccb223SMauro Carvalho Chehabits pathname, resulting in a negative directory entry for the object. 52818ccb223SMauro Carvalho ChehabA new inode is allocated and associated with the dentry, turning it from 52918ccb223SMauro Carvalho Chehaba negative dentry into a "productive full member of society". Orangefs 53018ccb223SMauro Carvalho Chehabobtains the new inode from Linux with new_inode() and associates 53118ccb223SMauro Carvalho Chehabthe inode with the dentry by sending the pair back to Linux with 53218ccb223SMauro Carvalho Chehabd_instantiate(). 53318ccb223SMauro Carvalho Chehab 53418ccb223SMauro Carvalho ChehabThe evaluation of a pathname for an object resolves to its corresponding 53518ccb223SMauro Carvalho Chehabdentry. If there is no corresponding dentry, one is created for it in 53618ccb223SMauro Carvalho Chehabthe dcache. Whenever a dentry is modified or verified Orangefs stores a 53718ccb223SMauro Carvalho Chehabshort timeout value in the dentry's d_time, and the dentry will be trusted 53818ccb223SMauro Carvalho Chehabfor that amount of time. Orangefs is a network filesystem, and objects 53918ccb223SMauro Carvalho Chehabcan potentially change out-of-band with any particular Orangefs kernel module 54018ccb223SMauro Carvalho Chehabinstance, so trusting a dentry is risky. The alternative to trusting 54118ccb223SMauro Carvalho Chehabdentries is to always obtain the needed information from userspace - at 54218ccb223SMauro Carvalho Chehableast a trip to the client-core, maybe to the servers. Obtaining information 54318ccb223SMauro Carvalho Chehabfrom a dentry is cheap, obtaining it from userspace is relatively expensive, 54418ccb223SMauro Carvalho Chehabhence the motivation to use the dentry when possible. 54518ccb223SMauro Carvalho Chehab 54618ccb223SMauro Carvalho ChehabThe timeout values d_time and getattr_time are jiffy based, and the 54718ccb223SMauro Carvalho Chehabcode is designed to avoid the jiffy-wrap problem:: 54818ccb223SMauro Carvalho Chehab 54918ccb223SMauro Carvalho Chehab "In general, if the clock may have wrapped around more than once, there 55018ccb223SMauro Carvalho Chehab is no way to tell how much time has elapsed. However, if the times t1 55118ccb223SMauro Carvalho Chehab and t2 are known to be fairly close, we can reliably compute the 55218ccb223SMauro Carvalho Chehab difference in a way that takes into account the possibility that the 55318ccb223SMauro Carvalho Chehab clock may have wrapped between times." 55418ccb223SMauro Carvalho Chehab 55518ccb223SMauro Carvalho Chehabfrom course notes by instructor Andy Wang 55618ccb223SMauro Carvalho Chehab 557