1.. SPDX-License-Identifier: GPL-2.0 2 3========== 4Netconsole 5========== 6 7 8started by Ingo Molnar <mingo@redhat.com>, 2001.09.17 9 102.6 port and netpoll api by Matt Mackall <mpm@selenic.com>, Sep 9 2003 11 12IPv6 support by Cong Wang <xiyou.wangcong@gmail.com>, Jan 1 2013 13 14Extended console support by Tejun Heo <tj@kernel.org>, May 1 2015 15 16Release prepend support by Breno Leitao <leitao@debian.org>, Jul 7 2023 17 18Userdata append support by Matthew Wood <thepacketgeek@gmail.com>, Jan 22 2024 19 20Sysdata append support by Breno Leitao <leitao@debian.org>, Jan 15 2025 21 22Introduction: 23============= 24 25This module logs kernel printk messages over UDP allowing debugging of 26problem where disk logging fails and serial consoles are impractical. 27 28It can be used either built-in or as a module. As a built-in, 29netconsole initializes immediately after NIC cards and will bring up 30the specified interface as soon as possible. While this doesn't allow 31capture of early kernel panics, it does capture most of the boot 32process. 33 34Sender and receiver configuration: 35================================== 36 37It takes a string configuration parameter "netconsole" in the 38following format:: 39 40 netconsole=[+][r][src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr] 41 42 where 43 + if present, enable extended console support 44 r if present, prepend kernel version (release) to the message 45 src-port source for UDP packets (defaults to 6665) 46 src-ip source IP to use (interface address) 47 dev network interface name (eth0) or MAC address 48 tgt-port port for logging agent (6666) 49 tgt-ip IP address for logging agent 50 tgt-macaddr ethernet MAC address for logging agent (broadcast) 51 52Examples:: 53 54 linux netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc 55 56or:: 57 58 insmod netconsole netconsole=@/,@10.0.0.2/ 59 60or using IPv6:: 61 62 insmod netconsole netconsole=@/,@fd00:1:2:3::1/ 63 64or using a MAC address to select the egress interface:: 65 66 linux netconsole=4444@10.0.0.1/22:33:44:55:66:77,9353@10.0.0.2/12:34:56:78:9a:bc 67 68It also supports logging to multiple remote agents by specifying 69parameters for the multiple agents separated by semicolons and the 70complete string enclosed in "quotes", thusly:: 71 72 modprobe netconsole netconsole="@/,@10.0.0.2/;@/eth1,6892@10.0.0.3/" 73 74Built-in netconsole starts immediately after the TCP stack is 75initialized and attempts to bring up the supplied dev at the supplied 76address. 77 78The remote host has several options to receive the kernel messages, 79for example: 80 811) syslogd 82 832) netcat 84 85 On distributions using a BSD-based netcat version (e.g. Fedora, 86 openSUSE and Ubuntu) the listening port must be specified without 87 the -p switch:: 88 89 nc -u -l -p <port>' / 'nc -u -l <port> 90 91 or:: 92 93 netcat -u -l -p <port>' / 'netcat -u -l <port> 94 953) socat 96 97:: 98 99 socat udp-recv:<port> - 100 101Dynamic reconfiguration: 102======================== 103 104Dynamic reconfigurability is a useful addition to netconsole that enables 105remote logging targets to be dynamically added, removed, or have their 106parameters reconfigured at runtime from a configfs-based userspace interface. 107 108To include this feature, select CONFIG_NETCONSOLE_DYNAMIC when building the 109netconsole module (or kernel, if netconsole is built-in). 110 111Some examples follow (where configfs is mounted at the /sys/kernel/config 112mountpoint). 113 114To add a remote logging target (target names can be arbitrary):: 115 116 cd /sys/kernel/config/netconsole/ 117 mkdir target1 118 119Note that newly created targets have default parameter values (as mentioned 120above) and are disabled by default -- they must first be enabled by writing 121"1" to the "enabled" attribute (usually after setting parameters accordingly) 122as described below. 123 124To remove a target:: 125 126 rmdir /sys/kernel/config/netconsole/othertarget/ 127 128The interface exposes these parameters of a netconsole target to userspace: 129 130 =============== ================================= ============ 131 enabled Is this target currently enabled? (read-write) 132 extended Extended mode enabled (read-write) 133 release Prepend kernel release to message (read-write) 134 dev_name Local network interface name (read-write) 135 local_port Source UDP port to use (read-write) 136 remote_port Remote agent's UDP port (read-write) 137 local_ip Source IP address to use (read-write) 138 remote_ip Remote agent's IP address (read-write) 139 local_mac Local interface's MAC address (read-only) 140 remote_mac Remote agent's MAC address (read-write) 141 transmit_errors Number of packet send errors (read-only) 142 =============== ================================= ============ 143 144The "enabled" attribute is also used to control whether the parameters of 145a target can be updated or not -- you can modify the parameters of only 146disabled targets (i.e. if "enabled" is 0). 147 148To update a target's parameters:: 149 150 cat enabled # check if enabled is 1 151 echo 0 > enabled # disable the target (if required) 152 echo eth2 > dev_name # set local interface 153 echo 10.0.0.4 > remote_ip # update some parameter 154 echo cb:a9:87:65:43:21 > remote_mac # update more parameters 155 echo 1 > enabled # enable target again 156 157You can also update the local interface dynamically. This is especially 158useful if you want to use interfaces that have newly come up (and may not 159have existed when netconsole was loaded / initialized). 160 161Netconsole targets defined at boot time (or module load time) with the 162`netconsole=` param are assigned the name `cmdline<index>`. For example, the 163first target in the parameter is named `cmdline0`. You can control and modify 164these targets by creating configfs directories with the matching name. 165 166Let's suppose you have two netconsole targets defined at boot time:: 167 168 netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc;4444@10.0.0.1/eth1,9353@10.0.0.3/12:34:56:78:9a:bc 169 170You can modify these targets in runtime by creating the following targets:: 171 172 mkdir cmdline0 173 cat cmdline0/remote_ip 174 10.0.0.2 175 176 mkdir cmdline1 177 cat cmdline1/remote_ip 178 10.0.0.3 179 180Append User Data 181---------------- 182 183Custom user data can be appended to the end of messages with netconsole 184dynamic configuration enabled. User data entries can be modified without 185changing the "enabled" attribute of a target. 186 187Directories (keys) under `userdata` are limited to 53 character length, and 188data in `userdata/<key>/value` are limited to 200 bytes:: 189 190 cd /sys/kernel/config/netconsole && mkdir cmdline0 191 cd cmdline0 192 mkdir userdata/foo 193 echo bar > userdata/foo/value 194 mkdir userdata/qux 195 echo baz > userdata/qux/value 196 197Messages will now include this additional user data:: 198 199 echo "This is a message" > /dev/kmsg 200 201Sends:: 202 203 12,607,22085407756,-;This is a message 204 foo=bar 205 qux=baz 206 207Preview the userdata that will be appended with:: 208 209 cd /sys/kernel/config/netconsole/cmdline0/userdata 210 for f in `ls userdata`; do echo $f=$(cat userdata/$f/value); done 211 212If a `userdata` entry is created but no data is written to the `value` file, 213the entry will be omitted from netconsole messages:: 214 215 cd /sys/kernel/config/netconsole && mkdir cmdline0 216 cd cmdline0 217 mkdir userdata/foo 218 echo bar > userdata/foo/value 219 mkdir userdata/qux 220 221The `qux` key is omitted since it has no value:: 222 223 echo "This is a message" > /dev/kmsg 224 12,607,22085407756,-;This is a message 225 foo=bar 226 227Delete `userdata` entries with `rmdir`:: 228 229 rmdir /sys/kernel/config/netconsole/cmdline0/userdata/qux 230 231.. warning:: 232 When writing strings to user data values, input is broken up per line in 233 configfs store calls and this can cause confusing behavior:: 234 235 mkdir userdata/testing 236 printf "val1\nval2" > userdata/testing/value 237 # userdata store value is called twice, first with "val1\n" then "val2" 238 # so "val2" is stored, being the last value stored 239 cat userdata/testing/value 240 val2 241 242 It is recommended to not write user data values with newlines. 243 244Task name auto population in userdata 245------------------------------------- 246 247Inside the netconsole configfs hierarchy, there is a file called 248`taskname_enabled` under the `userdata` directory. This file is used to enable 249or disable the automatic task name population feature. This feature 250automatically populates the current task name that is scheduled in the CPU 251sneding the message. 252 253To enable task name auto-population:: 254 255 echo 1 > /sys/kernel/config/netconsole/target1/userdata/taskname_enabled 256 257When this option is enabled, the netconsole messages will include an additional 258line in the userdata field with the format `taskname=<task name>`. This allows 259the receiver of the netconsole messages to easily find which application was 260currently scheduled when that message was generated, providing extra context 261for kernel messages and helping to categorize them. 262 263Example:: 264 265 echo "This is a message" > /dev/kmsg 266 12,607,22085407756,-;This is a message 267 taskname=echo 268 269In this example, the message was generated while "echo" was the current 270scheduled process. 271 272Kernel release auto population in userdata 273------------------------------------------ 274 275Within the netconsole configfs hierarchy, there is a file named `release_enabled` 276located in the `userdata` directory. This file controls the kernel release 277(version) auto-population feature, which appends the kernel release information 278to userdata dictionary in every message sent. 279 280To enable the release auto-population:: 281 282 echo 1 > /sys/kernel/config/netconsole/target1/userdata/release_enabled 283 284Example:: 285 286 echo "This is a message" > /dev/kmsg 287 12,607,22085407756,-;This is a message 288 release=6.14.0-rc6-01219-g3c027fbd941d 289 290.. note:: 291 292 This feature provides the same data as the "release prepend" feature. 293 However, in this case, the release information is appended to the userdata 294 dictionary rather than being included in the message header. 295 296 297CPU number auto population in userdata 298-------------------------------------- 299 300Inside the netconsole configfs hierarchy, there is a file called 301`cpu_nr` under the `userdata` directory. This file is used to enable or disable 302the automatic CPU number population feature. This feature automatically 303populates the CPU number that is sending the message. 304 305To enable the CPU number auto-population:: 306 307 echo 1 > /sys/kernel/config/netconsole/target1/userdata/cpu_nr 308 309When this option is enabled, the netconsole messages will include an additional 310line in the userdata field with the format `cpu=<cpu_number>`. This allows the 311receiver of the netconsole messages to easily differentiate and demultiplex 312messages originating from different CPUs, which is particularly useful when 313dealing with parallel log output. 314 315Example:: 316 317 echo "This is a message" > /dev/kmsg 318 12,607,22085407756,-;This is a message 319 cpu=42 320 321In this example, the message was sent by CPU 42. 322 323.. note:: 324 325 If the user has set a conflicting `cpu` key in the userdata dictionary, 326 both keys will be reported, with the kernel-populated entry appearing after 327 the user one. For example:: 328 329 # User-defined CPU entry 330 mkdir -p /sys/kernel/config/netconsole/target1/userdata/cpu 331 echo "1" > /sys/kernel/config/netconsole/target1/userdata/cpu/value 332 333 Output might look like:: 334 335 12,607,22085407756,-;This is a message 336 cpu=1 337 cpu=42 # kernel-populated value 338 339 340Message ID auto population in userdata 341-------------------------------------- 342 343Within the netconsole configfs hierarchy, there is a file named `msgid_enabled` 344located in the `userdata` directory. This file controls the message ID 345auto-population feature, which assigns a numeric id to each message sent to a 346given target and appends the ID to userdata dictionary in every message sent. 347 348The message ID is generated using a per-target 32 bit counter that is 349incremented for every message sent to the target. Note that this counter will 350eventually wrap around after reaching uint32_t max value, so the message ID is 351not globally unique over time. However, it can still be used by the target to 352detect if messages were dropped before reaching the target by identifying gaps 353in the sequence of IDs. 354 355It is important to distinguish message IDs from the message <sequnum> field. 356Some kernel messages may never reach netconsole (for example, due to printk 357rate limiting). Thus, a gap in <sequnum> cannot be solely relied upon to 358indicate that a message was dropped during transmission, as it may never have 359been sent via netconsole. The message ID, on the other hand, is only assigned 360to messages that are actually transmitted via netconsole. 361 362Example:: 363 364 echo "This is message #1" > /dev/kmsg 365 echo "This is message #2" > /dev/kmsg 366 13,434,54928466,-;This is message #1 367 msgid=1 368 13,435,54934019,-;This is message #2 369 msgid=2 370 371 372Extended console: 373================= 374 375If '+' is prefixed to the configuration line or "extended" config file 376is set to 1, extended console support is enabled. An example boot 377param follows:: 378 379 linux netconsole=+4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc 380 381Log messages are transmitted with extended metadata header in the 382following format which is the same as /dev/kmsg:: 383 384 <level>,<sequnum>,<timestamp>,<contflag>;<message text> 385 386If 'r' (release) feature is enabled, the kernel release version is 387prepended to the start of the message. Example:: 388 389 6.4.0,6,444,501151268,-;netconsole: network logging started 390 391Non printable characters in <message text> are escaped using "\xff" 392notation. If the message contains optional dictionary, verbatim 393newline is used as the delimiter. 394 395If a message doesn't fit in certain number of bytes (currently 1000), 396the message is split into multiple fragments by netconsole. These 397fragments are transmitted with "ncfrag" header field added:: 398 399 ncfrag=<byte-offset>/<total-bytes> 400 401For example, assuming a lot smaller chunk size, a message "the first 402chunk, the 2nd chunk." may be split as follows:: 403 404 6,416,1758426,-,ncfrag=0/31;the first chunk, 405 6,416,1758426,-,ncfrag=16/31; the 2nd chunk. 406 407Miscellaneous notes: 408==================== 409 410.. Warning:: 411 412 the default target ethernet setting uses the broadcast 413 ethernet address to send packets, which can cause increased load on 414 other systems on the same ethernet segment. 415 416.. Tip:: 417 418 some LAN switches may be configured to suppress ethernet broadcasts 419 so it is advised to explicitly specify the remote agents' MAC addresses 420 from the config parameters passed to netconsole. 421 422.. Tip:: 423 424 to find out the MAC address of, say, 10.0.0.2, you may try using:: 425 426 ping -c 1 10.0.0.2 ; /sbin/arp -n | grep 10.0.0.2 427 428.. Tip:: 429 430 in case the remote logging agent is on a separate LAN subnet than 431 the sender, it is suggested to try specifying the MAC address of the 432 default gateway (you may use /sbin/route -n to find it out) as the 433 remote MAC address instead. 434 435.. note:: 436 437 the network device (eth1 in the above case) can run any kind 438 of other network traffic, netconsole is not intrusive. Netconsole 439 might cause slight delays in other traffic if the volume of kernel 440 messages is high, but should have no other impact. 441 442.. note:: 443 444 if you find that the remote logging agent is not receiving or 445 printing all messages from the sender, it is likely that you have set 446 the "console_loglevel" parameter (on the sender) to only send high 447 priority messages to the console. You can change this at runtime using:: 448 449 dmesg -n 8 450 451 or by specifying "debug" on the kernel command line at boot, to send 452 all kernel messages to the console. A specific value for this parameter 453 can also be set using the "loglevel" kernel boot option. See the 454 dmesg(8) man page and Documentation/admin-guide/kernel-parameters.rst 455 for details. 456 457Netconsole was designed to be as instantaneous as possible, to 458enable the logging of even the most critical kernel bugs. It works 459from IRQ contexts as well, and does not enable interrupts while 460sending packets. Due to these unique needs, configuration cannot 461be more automatic, and some fundamental limitations will remain: 462only IP networks, UDP packets and ethernet devices are supported. 463