1.. SPDX-License-Identifier: GPL-2.0 2 3========== 4Netconsole 5========== 6 7 8started by Ingo Molnar <mingo@redhat.com>, 2001.09.17 9 102.6 port and netpoll api by Matt Mackall <mpm@selenic.com>, Sep 9 2003 11 12IPv6 support by Cong Wang <xiyou.wangcong@gmail.com>, Jan 1 2013 13 14Extended console support by Tejun Heo <tj@kernel.org>, May 1 2015 15 16Release prepend support by Breno Leitao <leitao@debian.org>, Jul 7 2023 17 18Userdata append support by Matthew Wood <thepacketgeek@gmail.com>, Jan 22 2024 19 20Sysdata append support by Breno Leitao <leitao@debian.org>, Jan 15 2025 21 22Please send bug reports to Matt Mackall <mpm@selenic.com> 23Satyam Sharma <satyam.sharma@gmail.com>, and Cong Wang <xiyou.wangcong@gmail.com> 24 25Introduction: 26============= 27 28This module logs kernel printk messages over UDP allowing debugging of 29problem where disk logging fails and serial consoles are impractical. 30 31It can be used either built-in or as a module. As a built-in, 32netconsole initializes immediately after NIC cards and will bring up 33the specified interface as soon as possible. While this doesn't allow 34capture of early kernel panics, it does capture most of the boot 35process. 36 37Sender and receiver configuration: 38================================== 39 40It takes a string configuration parameter "netconsole" in the 41following format:: 42 43 netconsole=[+][r][src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr] 44 45 where 46 + if present, enable extended console support 47 r if present, prepend kernel version (release) to the message 48 src-port source for UDP packets (defaults to 6665) 49 src-ip source IP to use (interface address) 50 dev network interface name (eth0) or MAC address 51 tgt-port port for logging agent (6666) 52 tgt-ip IP address for logging agent 53 tgt-macaddr ethernet MAC address for logging agent (broadcast) 54 55Examples:: 56 57 linux netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc 58 59or:: 60 61 insmod netconsole netconsole=@/,@10.0.0.2/ 62 63or using IPv6:: 64 65 insmod netconsole netconsole=@/,@fd00:1:2:3::1/ 66 67or using a MAC address to select the egress interface:: 68 69 linux netconsole=4444@10.0.0.1/22:33:44:55:66:77,9353@10.0.0.2/12:34:56:78:9a:bc 70 71It also supports logging to multiple remote agents by specifying 72parameters for the multiple agents separated by semicolons and the 73complete string enclosed in "quotes", thusly:: 74 75 modprobe netconsole netconsole="@/,@10.0.0.2/;@/eth1,6892@10.0.0.3/" 76 77Built-in netconsole starts immediately after the TCP stack is 78initialized and attempts to bring up the supplied dev at the supplied 79address. 80 81The remote host has several options to receive the kernel messages, 82for example: 83 841) syslogd 85 862) netcat 87 88 On distributions using a BSD-based netcat version (e.g. Fedora, 89 openSUSE and Ubuntu) the listening port must be specified without 90 the -p switch:: 91 92 nc -u -l -p <port>' / 'nc -u -l <port> 93 94 or:: 95 96 netcat -u -l -p <port>' / 'netcat -u -l <port> 97 983) socat 99 100:: 101 102 socat udp-recv:<port> - 103 104Dynamic reconfiguration: 105======================== 106 107Dynamic reconfigurability is a useful addition to netconsole that enables 108remote logging targets to be dynamically added, removed, or have their 109parameters reconfigured at runtime from a configfs-based userspace interface. 110 111To include this feature, select CONFIG_NETCONSOLE_DYNAMIC when building the 112netconsole module (or kernel, if netconsole is built-in). 113 114Some examples follow (where configfs is mounted at the /sys/kernel/config 115mountpoint). 116 117To add a remote logging target (target names can be arbitrary):: 118 119 cd /sys/kernel/config/netconsole/ 120 mkdir target1 121 122Note that newly created targets have default parameter values (as mentioned 123above) and are disabled by default -- they must first be enabled by writing 124"1" to the "enabled" attribute (usually after setting parameters accordingly) 125as described below. 126 127To remove a target:: 128 129 rmdir /sys/kernel/config/netconsole/othertarget/ 130 131The interface exposes these parameters of a netconsole target to userspace: 132 133 =============== ================================= ============ 134 enabled Is this target currently enabled? (read-write) 135 extended Extended mode enabled (read-write) 136 release Prepend kernel release to message (read-write) 137 dev_name Local network interface name (read-write) 138 local_port Source UDP port to use (read-write) 139 remote_port Remote agent's UDP port (read-write) 140 local_ip Source IP address to use (read-write) 141 remote_ip Remote agent's IP address (read-write) 142 local_mac Local interface's MAC address (read-only) 143 remote_mac Remote agent's MAC address (read-write) 144 transmit_errors Number of packet send errors (read-only) 145 =============== ================================= ============ 146 147The "enabled" attribute is also used to control whether the parameters of 148a target can be updated or not -- you can modify the parameters of only 149disabled targets (i.e. if "enabled" is 0). 150 151To update a target's parameters:: 152 153 cat enabled # check if enabled is 1 154 echo 0 > enabled # disable the target (if required) 155 echo eth2 > dev_name # set local interface 156 echo 10.0.0.4 > remote_ip # update some parameter 157 echo cb:a9:87:65:43:21 > remote_mac # update more parameters 158 echo 1 > enabled # enable target again 159 160You can also update the local interface dynamically. This is especially 161useful if you want to use interfaces that have newly come up (and may not 162have existed when netconsole was loaded / initialized). 163 164Netconsole targets defined at boot time (or module load time) with the 165`netconsole=` param are assigned the name `cmdline<index>`. For example, the 166first target in the parameter is named `cmdline0`. You can control and modify 167these targets by creating configfs directories with the matching name. 168 169Let's suppose you have two netconsole targets defined at boot time:: 170 171 netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc;4444@10.0.0.1/eth1,9353@10.0.0.3/12:34:56:78:9a:bc 172 173You can modify these targets in runtime by creating the following targets:: 174 175 mkdir cmdline0 176 cat cmdline0/remote_ip 177 10.0.0.2 178 179 mkdir cmdline1 180 cat cmdline1/remote_ip 181 10.0.0.3 182 183Append User Data 184---------------- 185 186Custom user data can be appended to the end of messages with netconsole 187dynamic configuration enabled. User data entries can be modified without 188changing the "enabled" attribute of a target. 189 190Directories (keys) under `userdata` are limited to 53 character length, and 191data in `userdata/<key>/value` are limited to 200 bytes:: 192 193 cd /sys/kernel/config/netconsole && mkdir cmdline0 194 cd cmdline0 195 mkdir userdata/foo 196 echo bar > userdata/foo/value 197 mkdir userdata/qux 198 echo baz > userdata/qux/value 199 200Messages will now include this additional user data:: 201 202 echo "This is a message" > /dev/kmsg 203 204Sends:: 205 206 12,607,22085407756,-;This is a message 207 foo=bar 208 qux=baz 209 210Preview the userdata that will be appended with:: 211 212 cd /sys/kernel/config/netconsole/cmdline0/userdata 213 for f in `ls userdata`; do echo $f=$(cat userdata/$f/value); done 214 215If a `userdata` entry is created but no data is written to the `value` file, 216the entry will be omitted from netconsole messages:: 217 218 cd /sys/kernel/config/netconsole && mkdir cmdline0 219 cd cmdline0 220 mkdir userdata/foo 221 echo bar > userdata/foo/value 222 mkdir userdata/qux 223 224The `qux` key is omitted since it has no value:: 225 226 echo "This is a message" > /dev/kmsg 227 12,607,22085407756,-;This is a message 228 foo=bar 229 230Delete `userdata` entries with `rmdir`:: 231 232 rmdir /sys/kernel/config/netconsole/cmdline0/userdata/qux 233 234.. warning:: 235 When writing strings to user data values, input is broken up per line in 236 configfs store calls and this can cause confusing behavior:: 237 238 mkdir userdata/testing 239 printf "val1\nval2" > userdata/testing/value 240 # userdata store value is called twice, first with "val1\n" then "val2" 241 # so "val2" is stored, being the last value stored 242 cat userdata/testing/value 243 val2 244 245 It is recommended to not write user data values with newlines. 246 247Task name auto population in userdata 248------------------------------------- 249 250Inside the netconsole configfs hierarchy, there is a file called 251`taskname_enabled` under the `userdata` directory. This file is used to enable 252or disable the automatic task name population feature. This feature 253automatically populates the current task name that is scheduled in the CPU 254sneding the message. 255 256To enable task name auto-population:: 257 258 echo 1 > /sys/kernel/config/netconsole/target1/userdata/taskname_enabled 259 260When this option is enabled, the netconsole messages will include an additional 261line in the userdata field with the format `taskname=<task name>`. This allows 262the receiver of the netconsole messages to easily find which application was 263currently scheduled when that message was generated, providing extra context 264for kernel messages and helping to categorize them. 265 266Example:: 267 268 echo "This is a message" > /dev/kmsg 269 12,607,22085407756,-;This is a message 270 taskname=echo 271 272In this example, the message was generated while "echo" was the current 273scheduled process. 274 275Kernel release auto population in userdata 276------------------------------------------ 277 278Within the netconsole configfs hierarchy, there is a file named `release_enabled` 279located in the `userdata` directory. This file controls the kernel release 280(version) auto-population feature, which appends the kernel release information 281to userdata dictionary in every message sent. 282 283To enable the release auto-population:: 284 285 echo 1 > /sys/kernel/config/netconsole/target1/userdata/release_enabled 286 287Example:: 288 289 echo "This is a message" > /dev/kmsg 290 12,607,22085407756,-;This is a message 291 release=6.14.0-rc6-01219-g3c027fbd941d 292 293.. note:: 294 295 This feature provides the same data as the "release prepend" feature. 296 However, in this case, the release information is appended to the userdata 297 dictionary rather than being included in the message header. 298 299 300CPU number auto population in userdata 301-------------------------------------- 302 303Inside the netconsole configfs hierarchy, there is a file called 304`cpu_nr` under the `userdata` directory. This file is used to enable or disable 305the automatic CPU number population feature. This feature automatically 306populates the CPU number that is sending the message. 307 308To enable the CPU number auto-population:: 309 310 echo 1 > /sys/kernel/config/netconsole/target1/userdata/cpu_nr 311 312When this option is enabled, the netconsole messages will include an additional 313line in the userdata field with the format `cpu=<cpu_number>`. This allows the 314receiver of the netconsole messages to easily differentiate and demultiplex 315messages originating from different CPUs, which is particularly useful when 316dealing with parallel log output. 317 318Example:: 319 320 echo "This is a message" > /dev/kmsg 321 12,607,22085407756,-;This is a message 322 cpu=42 323 324In this example, the message was sent by CPU 42. 325 326.. note:: 327 328 If the user has set a conflicting `cpu` key in the userdata dictionary, 329 both keys will be reported, with the kernel-populated entry appearing after 330 the user one. For example:: 331 332 # User-defined CPU entry 333 mkdir -p /sys/kernel/config/netconsole/target1/userdata/cpu 334 echo "1" > /sys/kernel/config/netconsole/target1/userdata/cpu/value 335 336 Output might look like:: 337 338 12,607,22085407756,-;This is a message 339 cpu=1 340 cpu=42 # kernel-populated value 341 342 343Extended console: 344================= 345 346If '+' is prefixed to the configuration line or "extended" config file 347is set to 1, extended console support is enabled. An example boot 348param follows:: 349 350 linux netconsole=+4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc 351 352Log messages are transmitted with extended metadata header in the 353following format which is the same as /dev/kmsg:: 354 355 <level>,<sequnum>,<timestamp>,<contflag>;<message text> 356 357If 'r' (release) feature is enabled, the kernel release version is 358prepended to the start of the message. Example:: 359 360 6.4.0,6,444,501151268,-;netconsole: network logging started 361 362Non printable characters in <message text> are escaped using "\xff" 363notation. If the message contains optional dictionary, verbatim 364newline is used as the delimiter. 365 366If a message doesn't fit in certain number of bytes (currently 1000), 367the message is split into multiple fragments by netconsole. These 368fragments are transmitted with "ncfrag" header field added:: 369 370 ncfrag=<byte-offset>/<total-bytes> 371 372For example, assuming a lot smaller chunk size, a message "the first 373chunk, the 2nd chunk." may be split as follows:: 374 375 6,416,1758426,-,ncfrag=0/31;the first chunk, 376 6,416,1758426,-,ncfrag=16/31; the 2nd chunk. 377 378Miscellaneous notes: 379==================== 380 381.. Warning:: 382 383 the default target ethernet setting uses the broadcast 384 ethernet address to send packets, which can cause increased load on 385 other systems on the same ethernet segment. 386 387.. Tip:: 388 389 some LAN switches may be configured to suppress ethernet broadcasts 390 so it is advised to explicitly specify the remote agents' MAC addresses 391 from the config parameters passed to netconsole. 392 393.. Tip:: 394 395 to find out the MAC address of, say, 10.0.0.2, you may try using:: 396 397 ping -c 1 10.0.0.2 ; /sbin/arp -n | grep 10.0.0.2 398 399.. Tip:: 400 401 in case the remote logging agent is on a separate LAN subnet than 402 the sender, it is suggested to try specifying the MAC address of the 403 default gateway (you may use /sbin/route -n to find it out) as the 404 remote MAC address instead. 405 406.. note:: 407 408 the network device (eth1 in the above case) can run any kind 409 of other network traffic, netconsole is not intrusive. Netconsole 410 might cause slight delays in other traffic if the volume of kernel 411 messages is high, but should have no other impact. 412 413.. note:: 414 415 if you find that the remote logging agent is not receiving or 416 printing all messages from the sender, it is likely that you have set 417 the "console_loglevel" parameter (on the sender) to only send high 418 priority messages to the console. You can change this at runtime using:: 419 420 dmesg -n 8 421 422 or by specifying "debug" on the kernel command line at boot, to send 423 all kernel messages to the console. A specific value for this parameter 424 can also be set using the "loglevel" kernel boot option. See the 425 dmesg(8) man page and Documentation/admin-guide/kernel-parameters.rst 426 for details. 427 428Netconsole was designed to be as instantaneous as possible, to 429enable the logging of even the most critical kernel bugs. It works 430from IRQ contexts as well, and does not enable interrupts while 431sending packets. Due to these unique needs, configuration cannot 432be more automatic, and some fundamental limitations will remain: 433only IP networks, UDP packets and ethernet devices are supported. 434