1.. SPDX-License-Identifier: GPL-2.0 2 3========== 4Netconsole 5========== 6 7 8started by Ingo Molnar <mingo@redhat.com>, 2001.09.17 9 102.6 port and netpoll api by Matt Mackall <mpm@selenic.com>, Sep 9 2003 11 12IPv6 support by Cong Wang <xiyou.wangcong@gmail.com>, Jan 1 2013 13 14Extended console support by Tejun Heo <tj@kernel.org>, May 1 2015 15 16Release prepend support by Breno Leitao <leitao@debian.org>, Jul 7 2023 17 18Userdata append support by Matthew Wood <thepacketgeek@gmail.com>, Jan 22 2024 19 20Sysdata append support by Breno Leitao <leitao@debian.org>, Jan 15 2025 21 22Please send bug reports to Matt Mackall <mpm@selenic.com> 23Satyam Sharma <satyam.sharma@gmail.com>, and Cong Wang <xiyou.wangcong@gmail.com> 24 25Introduction: 26============= 27 28This module logs kernel printk messages over UDP allowing debugging of 29problem where disk logging fails and serial consoles are impractical. 30 31It can be used either built-in or as a module. As a built-in, 32netconsole initializes immediately after NIC cards and will bring up 33the specified interface as soon as possible. While this doesn't allow 34capture of early kernel panics, it does capture most of the boot 35process. 36 37Sender and receiver configuration: 38================================== 39 40It takes a string configuration parameter "netconsole" in the 41following format:: 42 43 netconsole=[+][r][src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr] 44 45 where 46 + if present, enable extended console support 47 r if present, prepend kernel version (release) to the message 48 src-port source for UDP packets (defaults to 6665) 49 src-ip source IP to use (interface address) 50 dev network interface (eth0) 51 tgt-port port for logging agent (6666) 52 tgt-ip IP address for logging agent 53 tgt-macaddr ethernet MAC address for logging agent (broadcast) 54 55Examples:: 56 57 linux netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc 58 59or:: 60 61 insmod netconsole netconsole=@/,@10.0.0.2/ 62 63or using IPv6:: 64 65 insmod netconsole netconsole=@/,@fd00:1:2:3::1/ 66 67It also supports logging to multiple remote agents by specifying 68parameters for the multiple agents separated by semicolons and the 69complete string enclosed in "quotes", thusly:: 70 71 modprobe netconsole netconsole="@/,@10.0.0.2/;@/eth1,6892@10.0.0.3/" 72 73Built-in netconsole starts immediately after the TCP stack is 74initialized and attempts to bring up the supplied dev at the supplied 75address. 76 77The remote host has several options to receive the kernel messages, 78for example: 79 801) syslogd 81 822) netcat 83 84 On distributions using a BSD-based netcat version (e.g. Fedora, 85 openSUSE and Ubuntu) the listening port must be specified without 86 the -p switch:: 87 88 nc -u -l -p <port>' / 'nc -u -l <port> 89 90 or:: 91 92 netcat -u -l -p <port>' / 'netcat -u -l <port> 93 943) socat 95 96:: 97 98 socat udp-recv:<port> - 99 100Dynamic reconfiguration: 101======================== 102 103Dynamic reconfigurability is a useful addition to netconsole that enables 104remote logging targets to be dynamically added, removed, or have their 105parameters reconfigured at runtime from a configfs-based userspace interface. 106 107To include this feature, select CONFIG_NETCONSOLE_DYNAMIC when building the 108netconsole module (or kernel, if netconsole is built-in). 109 110Some examples follow (where configfs is mounted at the /sys/kernel/config 111mountpoint). 112 113To add a remote logging target (target names can be arbitrary):: 114 115 cd /sys/kernel/config/netconsole/ 116 mkdir target1 117 118Note that newly created targets have default parameter values (as mentioned 119above) and are disabled by default -- they must first be enabled by writing 120"1" to the "enabled" attribute (usually after setting parameters accordingly) 121as described below. 122 123To remove a target:: 124 125 rmdir /sys/kernel/config/netconsole/othertarget/ 126 127The interface exposes these parameters of a netconsole target to userspace: 128 129 =============== ================================= ============ 130 enabled Is this target currently enabled? (read-write) 131 extended Extended mode enabled (read-write) 132 release Prepend kernel release to message (read-write) 133 dev_name Local network interface name (read-write) 134 local_port Source UDP port to use (read-write) 135 remote_port Remote agent's UDP port (read-write) 136 local_ip Source IP address to use (read-write) 137 remote_ip Remote agent's IP address (read-write) 138 local_mac Local interface's MAC address (read-only) 139 remote_mac Remote agent's MAC address (read-write) 140 transmit_errors Number of packet send errors (read-only) 141 =============== ================================= ============ 142 143The "enabled" attribute is also used to control whether the parameters of 144a target can be updated or not -- you can modify the parameters of only 145disabled targets (i.e. if "enabled" is 0). 146 147To update a target's parameters:: 148 149 cat enabled # check if enabled is 1 150 echo 0 > enabled # disable the target (if required) 151 echo eth2 > dev_name # set local interface 152 echo 10.0.0.4 > remote_ip # update some parameter 153 echo cb:a9:87:65:43:21 > remote_mac # update more parameters 154 echo 1 > enabled # enable target again 155 156You can also update the local interface dynamically. This is especially 157useful if you want to use interfaces that have newly come up (and may not 158have existed when netconsole was loaded / initialized). 159 160Netconsole targets defined at boot time (or module load time) with the 161`netconsole=` param are assigned the name `cmdline<index>`. For example, the 162first target in the parameter is named `cmdline0`. You can control and modify 163these targets by creating configfs directories with the matching name. 164 165Let's suppose you have two netconsole targets defined at boot time:: 166 167 netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc;4444@10.0.0.1/eth1,9353@10.0.0.3/12:34:56:78:9a:bc 168 169You can modify these targets in runtime by creating the following targets:: 170 171 mkdir cmdline0 172 cat cmdline0/remote_ip 173 10.0.0.2 174 175 mkdir cmdline1 176 cat cmdline1/remote_ip 177 10.0.0.3 178 179Append User Data 180---------------- 181 182Custom user data can be appended to the end of messages with netconsole 183dynamic configuration enabled. User data entries can be modified without 184changing the "enabled" attribute of a target. 185 186Directories (keys) under `userdata` are limited to 53 character length, and 187data in `userdata/<key>/value` are limited to 200 bytes:: 188 189 cd /sys/kernel/config/netconsole && mkdir cmdline0 190 cd cmdline0 191 mkdir userdata/foo 192 echo bar > userdata/foo/value 193 mkdir userdata/qux 194 echo baz > userdata/qux/value 195 196Messages will now include this additional user data:: 197 198 echo "This is a message" > /dev/kmsg 199 200Sends:: 201 202 12,607,22085407756,-;This is a message 203 foo=bar 204 qux=baz 205 206Preview the userdata that will be appended with:: 207 208 cd /sys/kernel/config/netconsole/cmdline0/userdata 209 for f in `ls userdata`; do echo $f=$(cat userdata/$f/value); done 210 211If a `userdata` entry is created but no data is written to the `value` file, 212the entry will be omitted from netconsole messages:: 213 214 cd /sys/kernel/config/netconsole && mkdir cmdline0 215 cd cmdline0 216 mkdir userdata/foo 217 echo bar > userdata/foo/value 218 mkdir userdata/qux 219 220The `qux` key is omitted since it has no value:: 221 222 echo "This is a message" > /dev/kmsg 223 12,607,22085407756,-;This is a message 224 foo=bar 225 226Delete `userdata` entries with `rmdir`:: 227 228 rmdir /sys/kernel/config/netconsole/cmdline0/userdata/qux 229 230.. warning:: 231 When writing strings to user data values, input is broken up per line in 232 configfs store calls and this can cause confusing behavior:: 233 234 mkdir userdata/testing 235 printf "val1\nval2" > userdata/testing/value 236 # userdata store value is called twice, first with "val1\n" then "val2" 237 # so "val2" is stored, being the last value stored 238 cat userdata/testing/value 239 val2 240 241 It is recommended to not write user data values with newlines. 242 243Task name auto population in userdata 244------------------------------------- 245 246Inside the netconsole configfs hierarchy, there is a file called 247`taskname_enabled` under the `userdata` directory. This file is used to enable 248or disable the automatic task name population feature. This feature 249automatically populates the current task name that is scheduled in the CPU 250sneding the message. 251 252To enable task name auto-population:: 253 254 echo 1 > /sys/kernel/config/netconsole/target1/userdata/taskname_enabled 255 256When this option is enabled, the netconsole messages will include an additional 257line in the userdata field with the format `taskname=<task name>`. This allows 258the receiver of the netconsole messages to easily find which application was 259currently scheduled when that message was generated, providing extra context 260for kernel messages and helping to categorize them. 261 262Example:: 263 264 echo "This is a message" > /dev/kmsg 265 12,607,22085407756,-;This is a message 266 taskname=echo 267 268In this example, the message was generated while "echo" was the current 269scheduled process. 270 271CPU number auto population in userdata 272-------------------------------------- 273 274Inside the netconsole configfs hierarchy, there is a file called 275`cpu_nr` under the `userdata` directory. This file is used to enable or disable 276the automatic CPU number population feature. This feature automatically 277populates the CPU number that is sending the message. 278 279To enable the CPU number auto-population:: 280 281 echo 1 > /sys/kernel/config/netconsole/target1/userdata/cpu_nr 282 283When this option is enabled, the netconsole messages will include an additional 284line in the userdata field with the format `cpu=<cpu_number>`. This allows the 285receiver of the netconsole messages to easily differentiate and demultiplex 286messages originating from different CPUs, which is particularly useful when 287dealing with parallel log output. 288 289Example:: 290 291 echo "This is a message" > /dev/kmsg 292 12,607,22085407756,-;This is a message 293 cpu=42 294 295In this example, the message was sent by CPU 42. 296 297.. note:: 298 299 If the user has set a conflicting `cpu` key in the userdata dictionary, 300 both keys will be reported, with the kernel-populated entry appearing after 301 the user one. For example:: 302 303 # User-defined CPU entry 304 mkdir -p /sys/kernel/config/netconsole/target1/userdata/cpu 305 echo "1" > /sys/kernel/config/netconsole/target1/userdata/cpu/value 306 307 Output might look like:: 308 309 12,607,22085407756,-;This is a message 310 cpu=1 311 cpu=42 # kernel-populated value 312 313 314Extended console: 315================= 316 317If '+' is prefixed to the configuration line or "extended" config file 318is set to 1, extended console support is enabled. An example boot 319param follows:: 320 321 linux netconsole=+4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc 322 323Log messages are transmitted with extended metadata header in the 324following format which is the same as /dev/kmsg:: 325 326 <level>,<sequnum>,<timestamp>,<contflag>;<message text> 327 328If 'r' (release) feature is enabled, the kernel release version is 329prepended to the start of the message. Example:: 330 331 6.4.0,6,444,501151268,-;netconsole: network logging started 332 333Non printable characters in <message text> are escaped using "\xff" 334notation. If the message contains optional dictionary, verbatim 335newline is used as the delimiter. 336 337If a message doesn't fit in certain number of bytes (currently 1000), 338the message is split into multiple fragments by netconsole. These 339fragments are transmitted with "ncfrag" header field added:: 340 341 ncfrag=<byte-offset>/<total-bytes> 342 343For example, assuming a lot smaller chunk size, a message "the first 344chunk, the 2nd chunk." may be split as follows:: 345 346 6,416,1758426,-,ncfrag=0/31;the first chunk, 347 6,416,1758426,-,ncfrag=16/31; the 2nd chunk. 348 349Miscellaneous notes: 350==================== 351 352.. Warning:: 353 354 the default target ethernet setting uses the broadcast 355 ethernet address to send packets, which can cause increased load on 356 other systems on the same ethernet segment. 357 358.. Tip:: 359 360 some LAN switches may be configured to suppress ethernet broadcasts 361 so it is advised to explicitly specify the remote agents' MAC addresses 362 from the config parameters passed to netconsole. 363 364.. Tip:: 365 366 to find out the MAC address of, say, 10.0.0.2, you may try using:: 367 368 ping -c 1 10.0.0.2 ; /sbin/arp -n | grep 10.0.0.2 369 370.. Tip:: 371 372 in case the remote logging agent is on a separate LAN subnet than 373 the sender, it is suggested to try specifying the MAC address of the 374 default gateway (you may use /sbin/route -n to find it out) as the 375 remote MAC address instead. 376 377.. note:: 378 379 the network device (eth1 in the above case) can run any kind 380 of other network traffic, netconsole is not intrusive. Netconsole 381 might cause slight delays in other traffic if the volume of kernel 382 messages is high, but should have no other impact. 383 384.. note:: 385 386 if you find that the remote logging agent is not receiving or 387 printing all messages from the sender, it is likely that you have set 388 the "console_loglevel" parameter (on the sender) to only send high 389 priority messages to the console. You can change this at runtime using:: 390 391 dmesg -n 8 392 393 or by specifying "debug" on the kernel command line at boot, to send 394 all kernel messages to the console. A specific value for this parameter 395 can also be set using the "loglevel" kernel boot option. See the 396 dmesg(8) man page and Documentation/admin-guide/kernel-parameters.rst 397 for details. 398 399Netconsole was designed to be as instantaneous as possible, to 400enable the logging of even the most critical kernel bugs. It works 401from IRQ contexts as well, and does not enable interrupts while 402sending packets. Due to these unique needs, configuration cannot 403be more automatic, and some fundamental limitations will remain: 404only IP networks, UDP packets and ethernet devices are supported. 405