1.. SPDX-License-Identifier: GPL-2.0 2 3========== 4Netconsole 5========== 6 7 8started by Ingo Molnar <mingo@redhat.com>, 2001.09.17 9 102.6 port and netpoll api by Matt Mackall <mpm@selenic.com>, Sep 9 2003 11 12IPv6 support by Cong Wang <xiyou.wangcong@gmail.com>, Jan 1 2013 13 14Extended console support by Tejun Heo <tj@kernel.org>, May 1 2015 15 16Release prepend support by Breno Leitao <leitao@debian.org>, Jul 7 2023 17 18Userdata append support by Matthew Wood <thepacketgeek@gmail.com>, Jan 22 2024 19 20Sysdata append support by Breno Leitao <leitao@debian.org>, Jan 15 2025 21 22Please send bug reports to Matt Mackall <mpm@selenic.com> 23Satyam Sharma <satyam.sharma@gmail.com>, and Cong Wang <xiyou.wangcong@gmail.com> 24 25Introduction: 26============= 27 28This module logs kernel printk messages over UDP allowing debugging of 29problem where disk logging fails and serial consoles are impractical. 30 31It can be used either built-in or as a module. As a built-in, 32netconsole initializes immediately after NIC cards and will bring up 33the specified interface as soon as possible. While this doesn't allow 34capture of early kernel panics, it does capture most of the boot 35process. 36 37Sender and receiver configuration: 38================================== 39 40It takes a string configuration parameter "netconsole" in the 41following format:: 42 43 netconsole=[+][r][src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr] 44 45 where 46 + if present, enable extended console support 47 r if present, prepend kernel version (release) to the message 48 src-port source for UDP packets (defaults to 6665) 49 src-ip source IP to use (interface address) 50 dev network interface name (eth0) or MAC address 51 tgt-port port for logging agent (6666) 52 tgt-ip IP address for logging agent 53 tgt-macaddr ethernet MAC address for logging agent (broadcast) 54 55Examples:: 56 57 linux netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc 58 59or:: 60 61 insmod netconsole netconsole=@/,@10.0.0.2/ 62 63or using IPv6:: 64 65 insmod netconsole netconsole=@/,@fd00:1:2:3::1/ 66 67or using a MAC address to select the egress interface:: 68 69 linux netconsole=4444@10.0.0.1/22:33:44:55:66:77,9353@10.0.0.2/12:34:56:78:9a:bc 70 71It also supports logging to multiple remote agents by specifying 72parameters for the multiple agents separated by semicolons and the 73complete string enclosed in "quotes", thusly:: 74 75 modprobe netconsole netconsole="@/,@10.0.0.2/;@/eth1,6892@10.0.0.3/" 76 77Built-in netconsole starts immediately after the TCP stack is 78initialized and attempts to bring up the supplied dev at the supplied 79address. 80 81The remote host has several options to receive the kernel messages, 82for example: 83 841) syslogd 85 862) netcat 87 88 On distributions using a BSD-based netcat version (e.g. Fedora, 89 openSUSE and Ubuntu) the listening port must be specified without 90 the -p switch:: 91 92 nc -u -l -p <port>' / 'nc -u -l <port> 93 94 or:: 95 96 netcat -u -l -p <port>' / 'netcat -u -l <port> 97 983) socat 99 100:: 101 102 socat udp-recv:<port> - 103 104Dynamic reconfiguration: 105======================== 106 107Dynamic reconfigurability is a useful addition to netconsole that enables 108remote logging targets to be dynamically added, removed, or have their 109parameters reconfigured at runtime from a configfs-based userspace interface. 110 111To include this feature, select CONFIG_NETCONSOLE_DYNAMIC when building the 112netconsole module (or kernel, if netconsole is built-in). 113 114Some examples follow (where configfs is mounted at the /sys/kernel/config 115mountpoint). 116 117To add a remote logging target (target names can be arbitrary):: 118 119 cd /sys/kernel/config/netconsole/ 120 mkdir target1 121 122Note that newly created targets have default parameter values (as mentioned 123above) and are disabled by default -- they must first be enabled by writing 124"1" to the "enabled" attribute (usually after setting parameters accordingly) 125as described below. 126 127To remove a target:: 128 129 rmdir /sys/kernel/config/netconsole/othertarget/ 130 131The interface exposes these parameters of a netconsole target to userspace: 132 133 =============== ================================= ============ 134 enabled Is this target currently enabled? (read-write) 135 extended Extended mode enabled (read-write) 136 release Prepend kernel release to message (read-write) 137 dev_name Local network interface name (read-write) 138 local_port Source UDP port to use (read-write) 139 remote_port Remote agent's UDP port (read-write) 140 local_ip Source IP address to use (read-write) 141 remote_ip Remote agent's IP address (read-write) 142 local_mac Local interface's MAC address (read-only) 143 remote_mac Remote agent's MAC address (read-write) 144 transmit_errors Number of packet send errors (read-only) 145 =============== ================================= ============ 146 147The "enabled" attribute is also used to control whether the parameters of 148a target can be updated or not -- you can modify the parameters of only 149disabled targets (i.e. if "enabled" is 0). 150 151To update a target's parameters:: 152 153 cat enabled # check if enabled is 1 154 echo 0 > enabled # disable the target (if required) 155 echo eth2 > dev_name # set local interface 156 echo 10.0.0.4 > remote_ip # update some parameter 157 echo cb:a9:87:65:43:21 > remote_mac # update more parameters 158 echo 1 > enabled # enable target again 159 160You can also update the local interface dynamically. This is especially 161useful if you want to use interfaces that have newly come up (and may not 162have existed when netconsole was loaded / initialized). 163 164Netconsole targets defined at boot time (or module load time) with the 165`netconsole=` param are assigned the name `cmdline<index>`. For example, the 166first target in the parameter is named `cmdline0`. You can control and modify 167these targets by creating configfs directories with the matching name. 168 169Let's suppose you have two netconsole targets defined at boot time:: 170 171 netconsole=4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc;4444@10.0.0.1/eth1,9353@10.0.0.3/12:34:56:78:9a:bc 172 173You can modify these targets in runtime by creating the following targets:: 174 175 mkdir cmdline0 176 cat cmdline0/remote_ip 177 10.0.0.2 178 179 mkdir cmdline1 180 cat cmdline1/remote_ip 181 10.0.0.3 182 183Append User Data 184---------------- 185 186Custom user data can be appended to the end of messages with netconsole 187dynamic configuration enabled. User data entries can be modified without 188changing the "enabled" attribute of a target. 189 190Directories (keys) under `userdata` are limited to 53 character length, and 191data in `userdata/<key>/value` are limited to 200 bytes:: 192 193 cd /sys/kernel/config/netconsole && mkdir cmdline0 194 cd cmdline0 195 mkdir userdata/foo 196 echo bar > userdata/foo/value 197 mkdir userdata/qux 198 echo baz > userdata/qux/value 199 200Messages will now include this additional user data:: 201 202 echo "This is a message" > /dev/kmsg 203 204Sends:: 205 206 12,607,22085407756,-;This is a message 207 foo=bar 208 qux=baz 209 210Preview the userdata that will be appended with:: 211 212 cd /sys/kernel/config/netconsole/cmdline0/userdata 213 for f in `ls userdata`; do echo $f=$(cat userdata/$f/value); done 214 215If a `userdata` entry is created but no data is written to the `value` file, 216the entry will be omitted from netconsole messages:: 217 218 cd /sys/kernel/config/netconsole && mkdir cmdline0 219 cd cmdline0 220 mkdir userdata/foo 221 echo bar > userdata/foo/value 222 mkdir userdata/qux 223 224The `qux` key is omitted since it has no value:: 225 226 echo "This is a message" > /dev/kmsg 227 12,607,22085407756,-;This is a message 228 foo=bar 229 230Delete `userdata` entries with `rmdir`:: 231 232 rmdir /sys/kernel/config/netconsole/cmdline0/userdata/qux 233 234.. warning:: 235 When writing strings to user data values, input is broken up per line in 236 configfs store calls and this can cause confusing behavior:: 237 238 mkdir userdata/testing 239 printf "val1\nval2" > userdata/testing/value 240 # userdata store value is called twice, first with "val1\n" then "val2" 241 # so "val2" is stored, being the last value stored 242 cat userdata/testing/value 243 val2 244 245 It is recommended to not write user data values with newlines. 246 247Task name auto population in userdata 248------------------------------------- 249 250Inside the netconsole configfs hierarchy, there is a file called 251`taskname_enabled` under the `userdata` directory. This file is used to enable 252or disable the automatic task name population feature. This feature 253automatically populates the current task name that is scheduled in the CPU 254sneding the message. 255 256To enable task name auto-population:: 257 258 echo 1 > /sys/kernel/config/netconsole/target1/userdata/taskname_enabled 259 260When this option is enabled, the netconsole messages will include an additional 261line in the userdata field with the format `taskname=<task name>`. This allows 262the receiver of the netconsole messages to easily find which application was 263currently scheduled when that message was generated, providing extra context 264for kernel messages and helping to categorize them. 265 266Example:: 267 268 echo "This is a message" > /dev/kmsg 269 12,607,22085407756,-;This is a message 270 taskname=echo 271 272In this example, the message was generated while "echo" was the current 273scheduled process. 274 275CPU number auto population in userdata 276-------------------------------------- 277 278Inside the netconsole configfs hierarchy, there is a file called 279`cpu_nr` under the `userdata` directory. This file is used to enable or disable 280the automatic CPU number population feature. This feature automatically 281populates the CPU number that is sending the message. 282 283To enable the CPU number auto-population:: 284 285 echo 1 > /sys/kernel/config/netconsole/target1/userdata/cpu_nr 286 287When this option is enabled, the netconsole messages will include an additional 288line in the userdata field with the format `cpu=<cpu_number>`. This allows the 289receiver of the netconsole messages to easily differentiate and demultiplex 290messages originating from different CPUs, which is particularly useful when 291dealing with parallel log output. 292 293Example:: 294 295 echo "This is a message" > /dev/kmsg 296 12,607,22085407756,-;This is a message 297 cpu=42 298 299In this example, the message was sent by CPU 42. 300 301.. note:: 302 303 If the user has set a conflicting `cpu` key in the userdata dictionary, 304 both keys will be reported, with the kernel-populated entry appearing after 305 the user one. For example:: 306 307 # User-defined CPU entry 308 mkdir -p /sys/kernel/config/netconsole/target1/userdata/cpu 309 echo "1" > /sys/kernel/config/netconsole/target1/userdata/cpu/value 310 311 Output might look like:: 312 313 12,607,22085407756,-;This is a message 314 cpu=1 315 cpu=42 # kernel-populated value 316 317 318Extended console: 319================= 320 321If '+' is prefixed to the configuration line or "extended" config file 322is set to 1, extended console support is enabled. An example boot 323param follows:: 324 325 linux netconsole=+4444@10.0.0.1/eth1,9353@10.0.0.2/12:34:56:78:9a:bc 326 327Log messages are transmitted with extended metadata header in the 328following format which is the same as /dev/kmsg:: 329 330 <level>,<sequnum>,<timestamp>,<contflag>;<message text> 331 332If 'r' (release) feature is enabled, the kernel release version is 333prepended to the start of the message. Example:: 334 335 6.4.0,6,444,501151268,-;netconsole: network logging started 336 337Non printable characters in <message text> are escaped using "\xff" 338notation. If the message contains optional dictionary, verbatim 339newline is used as the delimiter. 340 341If a message doesn't fit in certain number of bytes (currently 1000), 342the message is split into multiple fragments by netconsole. These 343fragments are transmitted with "ncfrag" header field added:: 344 345 ncfrag=<byte-offset>/<total-bytes> 346 347For example, assuming a lot smaller chunk size, a message "the first 348chunk, the 2nd chunk." may be split as follows:: 349 350 6,416,1758426,-,ncfrag=0/31;the first chunk, 351 6,416,1758426,-,ncfrag=16/31; the 2nd chunk. 352 353Miscellaneous notes: 354==================== 355 356.. Warning:: 357 358 the default target ethernet setting uses the broadcast 359 ethernet address to send packets, which can cause increased load on 360 other systems on the same ethernet segment. 361 362.. Tip:: 363 364 some LAN switches may be configured to suppress ethernet broadcasts 365 so it is advised to explicitly specify the remote agents' MAC addresses 366 from the config parameters passed to netconsole. 367 368.. Tip:: 369 370 to find out the MAC address of, say, 10.0.0.2, you may try using:: 371 372 ping -c 1 10.0.0.2 ; /sbin/arp -n | grep 10.0.0.2 373 374.. Tip:: 375 376 in case the remote logging agent is on a separate LAN subnet than 377 the sender, it is suggested to try specifying the MAC address of the 378 default gateway (you may use /sbin/route -n to find it out) as the 379 remote MAC address instead. 380 381.. note:: 382 383 the network device (eth1 in the above case) can run any kind 384 of other network traffic, netconsole is not intrusive. Netconsole 385 might cause slight delays in other traffic if the volume of kernel 386 messages is high, but should have no other impact. 387 388.. note:: 389 390 if you find that the remote logging agent is not receiving or 391 printing all messages from the sender, it is likely that you have set 392 the "console_loglevel" parameter (on the sender) to only send high 393 priority messages to the console. You can change this at runtime using:: 394 395 dmesg -n 8 396 397 or by specifying "debug" on the kernel command line at boot, to send 398 all kernel messages to the console. A specific value for this parameter 399 can also be set using the "loglevel" kernel boot option. See the 400 dmesg(8) man page and Documentation/admin-guide/kernel-parameters.rst 401 for details. 402 403Netconsole was designed to be as instantaneous as possible, to 404enable the logging of even the most critical kernel bugs. It works 405from IRQ contexts as well, and does not enable interrupts while 406sending packets. Due to these unique needs, configuration cannot 407be more automatic, and some fundamental limitations will remain: 408only IP networks, UDP packets and ethernet devices are supported. 409