1.. SPDX-License-Identifier: GPL-2.0 2 3================ 4Perf ring buffer 5================ 6 7.. CONTENTS 8 9 1. Introduction 10 11 2. Ring buffer implementation 12 2.1 Basic algorithm 13 2.2 Ring buffer for different tracing modes 14 2.2.1 Default mode 15 2.2.2 Per-thread mode 16 2.2.3 Per-CPU mode 17 2.2.4 System wide mode 18 2.3 Accessing buffer 19 2.3.1 Producer-consumer model 20 2.3.2 Properties of the ring buffers 21 2.3.3 Writing samples into buffer 22 2.3.4 Reading samples from buffer 23 2.3.5 Memory synchronization 24 25 3. The mechanism of AUX ring buffer 26 3.1 The relationship between AUX and regular ring buffers 27 3.2 AUX events 28 3.3 Snapshot mode 29 30 311. Introduction 32=============== 33 34The ring buffer is a fundamental mechanism for data transfer. perf uses 35ring buffers to transfer event data from kernel to user space, another 36kind of ring buffer which is so called auxiliary (AUX) ring buffer also 37plays an important role for hardware tracing with Intel PT, Arm 38CoreSight, etc. 39 40The ring buffer implementation is critical but it's also a very 41challenging work. On the one hand, the kernel and perf tool in the user 42space use the ring buffer to exchange data and stores data into data 43file, thus the ring buffer needs to transfer data with high throughput; 44on the other hand, the ring buffer management should avoid significant 45overload to distract profiling results. 46 47This documentation dives into the details for perf ring buffer with two 48parts: firstly it explains the perf ring buffer implementation, then the 49second part discusses the AUX ring buffer mechanism. 50 512. Ring buffer implementation 52============================= 53 542.1 Basic algorithm 55------------------- 56 57That said, a typical ring buffer is managed by a head pointer and a tail 58pointer; the head pointer is manipulated by a writer and the tail 59pointer is updated by a reader respectively. 60 61:: 62 63 +---------------------------+ 64 | | |***|***|***| | | 65 +---------------------------+ 66 `-> Tail `-> Head 67 68 * : the data is filled by the writer. 69 70 Figure 1. Ring buffer 71 72Perf uses the same way to manage its ring buffer. In the implementation 73there are two key data structures held together in a set of consecutive 74pages, the control structure and then the ring buffer itself. The page 75with the control structure in is known as the "user page". Being held 76in continuous virtual addresses simplifies locating the ring buffer 77address, it is in the pages after the page with the user page. 78 79The control structure is named as ``perf_event_mmap_page``, it contains a 80head pointer ``data_head`` and a tail pointer ``data_tail``. When the 81kernel starts to fill records into the ring buffer, it updates the head 82pointer to reserve the memory so later it can safely store events into 83the buffer. On the other side, when the user page is a writable mapping, 84the perf tool has the permission to update the tail pointer after consuming 85data from the ring buffer. Yet another case is for the user page's 86read-only mapping, which is to be addressed in the section 87:ref:`writing_samples_into_buffer`. 88 89:: 90 91 user page ring buffer 92 +---------+---------+ +---------------------------------------+ 93 |data_head|data_tail|...| | |***|***|***|***|***| | | | 94 +---------+---------+ +---------------------------------------+ 95 ` `----------------^ ^ 96 `----------------------------------------------| 97 98 * : the data is filled by the writer. 99 100 Figure 2. Perf ring buffer 101 102When using the ``perf record`` tool, we can specify the ring buffer size 103with option ``-m`` or ``--mmap-pages=``, the given size will be rounded up 104to a power of two that is a multiple of a page size. Though the kernel 105allocates at once for all memory pages, it's deferred to map the pages 106to VMA area until the perf tool accesses the buffer from the user space. 107In other words, at the first time accesses the buffer's page from user 108space in the perf tool, a data abort exception for page fault is taken 109and the kernel uses this occasion to map the page into process VMA 110(see ``perf_mmap_fault()``), thus the perf tool can continue to access 111the page after returning from the exception. 112 1132.2 Ring buffer for different tracing modes 114------------------------------------------- 115 116The perf profiles programs with different modes: default mode, per thread 117mode, per cpu mode, and system wide mode. This section describes these 118modes and how the ring buffer meets requirements for them. At last we 119will review the race conditions caused by these modes. 120 1212.2.1 Default mode 122^^^^^^^^^^^^^^^^^^ 123 124Usually we execute ``perf record`` command followed by a profiling program 125name, like below command:: 126 127 perf record test_program 128 129This command doesn't specify any options for CPU and thread modes, the 130perf tool applies the default mode on the perf event. It maps all the 131CPUs in the system and the profiled program's PID on the perf event, and 132it enables inheritance mode on the event so that child tasks inherits 133the events. As a result, the perf event is attributed as:: 134 135 evsel::cpus::map[] = { 0 .. _SC_NPROCESSORS_ONLN-1 } 136 evsel::threads::map[] = { pid } 137 evsel::attr::inherit = 1 138 139These attributions finally will be reflected on the deployment of ring 140buffers. As shown below, the perf tool allocates individual ring buffer 141for each CPU, but it only enables events for the profiled program rather 142than for all threads in the system. The *T1* thread represents the 143thread context of the 'test_program', whereas *T2* and *T3* are irrelevant 144threads in the system. The perf samples are exclusively collected for 145the *T1* thread and stored in the ring buffer associated with the CPU on 146which the *T1* thread is running. 147 148:: 149 150 T1 T2 T1 151 +----+ +-----------+ +----+ 152 CPU0 |xxxx| |xxxxxxxxxxx| |xxxx| 153 +----+--------------+-----------+----------+----+--------> 154 | | 155 v v 156 +-----------------------------------------------------+ 157 | Ring buffer 0 | 158 +-----------------------------------------------------+ 159 160 T1 161 +-----+ 162 CPU1 |xxxxx| 163 -----+-----+---------------------------------------------> 164 | 165 v 166 +-----------------------------------------------------+ 167 | Ring buffer 1 | 168 +-----------------------------------------------------+ 169 170 T1 T3 171 +----+ +-------+ 172 CPU2 |xxxx| |xxxxxxx| 173 --------------------------+----+--------+-------+--------> 174 | 175 v 176 +-----------------------------------------------------+ 177 | Ring buffer 2 | 178 +-----------------------------------------------------+ 179 180 T1 181 +--------------+ 182 CPU3 |xxxxxxxxxxxxxx| 183 -----------+--------------+------------------------------> 184 | 185 v 186 +-----------------------------------------------------+ 187 | Ring buffer 3 | 188 +-----------------------------------------------------+ 189 190 T1: Thread 1; T2: Thread 2; T3: Thread 3 191 x: Thread is in running state 192 193 Figure 3. Ring buffer for default mode 194 1952.2.2 Per-thread mode 196^^^^^^^^^^^^^^^^^^^^^ 197 198By specifying option ``--per-thread`` in perf command, e.g. 199 200:: 201 202 perf record --per-thread test_program 203 204The perf event doesn't map to any CPUs and is only bound to the 205profiled process, thus, the perf event's attributions are:: 206 207 evsel::cpus::map[0] = { -1 } 208 evsel::threads::map[] = { pid } 209 evsel::attr::inherit = 0 210 211In this mode, a single ring buffer is allocated for the profiled thread; 212if the thread is scheduled on a CPU, the events on that CPU will be 213enabled; and if the thread is scheduled out from the CPU, the events on 214the CPU will be disabled. When the thread is migrated from one CPU to 215another, the events are to be disabled on the previous CPU and enabled 216on the next CPU correspondingly. 217 218:: 219 220 T1 T2 T1 221 +----+ +-----------+ +----+ 222 CPU0 |xxxx| |xxxxxxxxxxx| |xxxx| 223 +----+--------------+-----------+----------+----+--------> 224 | | 225 | T1 | 226 | +-----+ | 227 CPU1 | |xxxxx| | 228 --|--+-----+----------------------------------|----------> 229 | | | 230 | | T1 T3 | 231 | | +----+ +---+ | 232 CPU2 | | |xxxx| |xxx| | 233 --|-----|-----------------+----+--------+---+-|----------> 234 | | | | 235 | | T1 | | 236 | | +--------------+ | | 237 CPU3 | | |xxxxxxxxxxxxxx| | | 238 --|-----|--+--------------+-|-----------------|----------> 239 | | | | | 240 v v v v v 241 +-----------------------------------------------------+ 242 | Ring buffer | 243 +-----------------------------------------------------+ 244 245 T1: Thread 1 246 x: Thread is in running state 247 248 Figure 4. Ring buffer for per-thread mode 249 250When perf runs in per-thread mode, a ring buffer is allocated for the 251profiled thread *T1*. The ring buffer is dedicated for thread *T1*, if the 252thread *T1* is running, the perf events will be recorded into the ring 253buffer; when the thread is sleeping, all associated events will be 254disabled, thus no trace data will be recorded into the ring buffer. 255 2562.2.3 Per-CPU mode 257^^^^^^^^^^^^^^^^^^ 258 259The option ``-C`` is used to collect samples on the list of CPUs, for 260example the below perf command receives option ``-C 0,2``:: 261 262 perf record -C 0,2 test_program 263 264It maps the perf event to CPUs 0 and 2, and the event is not associated to any 265PID. Thus the perf event attributions are set as:: 266 267 evsel::cpus::map[0] = { 0, 2 } 268 evsel::threads::map[] = { -1 } 269 evsel::attr::inherit = 0 270 271This results in the session of ``perf record`` will sample all threads on CPU0 272and CPU2, and be terminated until test_program exits. Even there have tasks 273running on CPU1 and CPU3, since the ring buffer is absent for them, any 274activities on these two CPUs will be ignored. A usage case is to combine the 275options for per-thread mode and per-CPU mode, e.g. the options ``–C 0,2`` and 276``––per–thread`` are specified together, the samples are recorded only when 277the profiled thread is scheduled on any of the listed CPUs. 278 279:: 280 281 T1 T2 T1 282 +----+ +-----------+ +----+ 283 CPU0 |xxxx| |xxxxxxxxxxx| |xxxx| 284 +----+--------------+-----------+----------+----+--------> 285 | | | 286 v v v 287 +-----------------------------------------------------+ 288 | Ring buffer 0 | 289 +-----------------------------------------------------+ 290 291 T1 292 +-----+ 293 CPU1 |xxxxx| 294 -----+-----+---------------------------------------------> 295 296 T1 T3 297 +----+ +-------+ 298 CPU2 |xxxx| |xxxxxxx| 299 --------------------------+----+--------+-------+--------> 300 | | 301 v v 302 +-----------------------------------------------------+ 303 | Ring buffer 1 | 304 +-----------------------------------------------------+ 305 306 T1 307 +--------------+ 308 CPU3 |xxxxxxxxxxxxxx| 309 -----------+--------------+------------------------------> 310 311 T1: Thread 1; T2: Thread 2; T3: Thread 3 312 x: Thread is in running state 313 314 Figure 5. Ring buffer for per-CPU mode 315 3162.2.4 System wide mode 317^^^^^^^^^^^^^^^^^^^^^^ 318 319By using option ``–a`` or ``––all–cpus``, perf collects samples on all CPUs 320for all tasks, we call it as the system wide mode, the command is:: 321 322 perf record -a test_program 323 324Similar to the per-CPU mode, the perf event doesn't bind to any PID, and 325it maps to all CPUs in the system:: 326 327 evsel::cpus::map[] = { 0 .. _SC_NPROCESSORS_ONLN-1 } 328 evsel::threads::map[] = { -1 } 329 evsel::attr::inherit = 0 330 331In the system wide mode, every CPU has its own ring buffer, all threads 332are monitored during the running state and the samples are recorded into 333the ring buffer belonging to the CPU which the events occurred on. 334 335:: 336 337 T1 T2 T1 338 +----+ +-----------+ +----+ 339 CPU0 |xxxx| |xxxxxxxxxxx| |xxxx| 340 +----+--------------+-----------+----------+----+--------> 341 | | | 342 v v v 343 +-----------------------------------------------------+ 344 | Ring buffer 0 | 345 +-----------------------------------------------------+ 346 347 T1 348 +-----+ 349 CPU1 |xxxxx| 350 -----+-----+---------------------------------------------> 351 | 352 v 353 +-----------------------------------------------------+ 354 | Ring buffer 1 | 355 +-----------------------------------------------------+ 356 357 T1 T3 358 +----+ +-------+ 359 CPU2 |xxxx| |xxxxxxx| 360 --------------------------+----+--------+-------+--------> 361 | | 362 v v 363 +-----------------------------------------------------+ 364 | Ring buffer 2 | 365 +-----------------------------------------------------+ 366 367 T1 368 +--------------+ 369 CPU3 |xxxxxxxxxxxxxx| 370 -----------+--------------+------------------------------> 371 | 372 v 373 +-----------------------------------------------------+ 374 | Ring buffer 3 | 375 +-----------------------------------------------------+ 376 377 T1: Thread 1; T2: Thread 2; T3: Thread 3 378 x: Thread is in running state 379 380 Figure 6. Ring buffer for system wide mode 381 3822.3 Accessing buffer 383-------------------- 384 385Based on the understanding of how the ring buffer is allocated in 386various modes, this section explains access the ring buffer. 387 3882.3.1 Producer-consumer model 389^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 390 391In the Linux kernel, the PMU events can produce samples which are stored 392into the ring buffer; the perf command in user space consumes the 393samples by reading out data from the ring buffer and finally saves the 394data into the file for post analysis. It’s a typical producer-consumer 395model for using the ring buffer. 396 397The perf process polls on the PMU events and sleeps when no events are 398incoming. To prevent frequent exchanges between the kernel and user 399space, the kernel event core layer introduces a watermark, which is 400stored in the ``perf_buffer::watermark``. When a sample is recorded into 401the ring buffer, and if the used buffer exceeds the watermark, the 402kernel wakes up the perf process to read samples from the ring buffer. 403 404:: 405 406 Perf 407 / | Read samples 408 Polling / `--------------| Ring buffer 409 v v ;---------------------v 410 +----------------+ +---------+---------+ +-------------------+ 411 |Event wait queue| |data_head|data_tail| |***|***| | |***| 412 +----------------+ +---------+---------+ +-------------------+ 413 ^ ^ `------------------------^ 414 | Wake up tasks | Store samples 415 +-----------------------------+ 416 | Kernel event core layer | 417 +-----------------------------+ 418 419 * : the data is filled by the writer. 420 421 Figure 7. Writing and reading the ring buffer 422 423When the kernel event core layer notifies the user space, because 424multiple events might share the same ring buffer for recording samples, 425the core layer iterates every event associated with the ring buffer and 426wakes up tasks waiting on the event. This is fulfilled by the kernel 427function ``ring_buffer_wakeup()``. 428 429After the perf process is woken up, it starts to check the ring buffers 430one by one, if it finds any ring buffer containing samples it will read 431out the samples for statistics or saving into the data file. Given the 432perf process is able to run on any CPU, this leads to the ring buffer 433potentially being accessed from multiple CPUs simultaneously, which 434causes race conditions. The race condition handling is described in the 435section :ref:`memory_synchronization`. 436 4372.3.2 Properties of the ring buffers 438^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 439 440Linux kernel supports two write directions for the ring buffer: forward and 441backward. The forward writing saves samples from the beginning of the ring 442buffer, the backward writing stores data from the end of the ring buffer with 443the reversed direction. The perf tool determines the writing direction. 444 445Additionally, the tool can map buffers in either read-write mode or read-only 446mode to the user space. 447 448The ring buffer in the read-write mode is mapped with the property 449``PROT_READ | PROT_WRITE``. With the write permission, the perf tool 450updates the ``data_tail`` to indicate the data start position. Combining 451with the head pointer ``data_head``, which works as the end position of 452the current data, the perf tool can easily know where read out the data 453from. 454 455Alternatively, in the read-only mode, only the kernel keeps to update 456the ``data_head`` while the user space cannot access the ``data_tail`` due 457to the mapping property ``PROT_READ``. 458 459As a result, the matrix below illustrates the various combinations of 460direction and mapping characteristics. The perf tool employs two of these 461combinations to support buffer types: the non-overwrite buffer and the 462overwritable buffer. 463 464.. list-table:: 465 :widths: 1 1 1 466 :header-rows: 1 467 468 * - Mapping mode 469 - Forward 470 - Backward 471 * - read-write 472 - Non-overwrite ring buffer 473 - Not used 474 * - read-only 475 - Not used 476 - Overwritable ring buffer 477 478The non-overwrite ring buffer uses the read-write mapping with forward 479writing. It starts to save data from the beginning of the ring buffer 480and wrap around when overflow, which is used with the read-write mode in 481the normal ring buffer. When the consumer doesn't keep up with the 482producer, it would lose some data, the kernel keeps how many records it 483lost and generates the ``PERF_RECORD_LOST`` records in the next time 484when it finds a space in the ring buffer. 485 486The overwritable ring buffer uses the backward writing with the 487read-only mode. It saves the data from the end of the ring buffer and 488the ``data_head`` keeps the position of current data, the perf always 489knows where it starts to read and until the end of the ring buffer, thus 490it don't need the ``data_tail``. In this mode, it will not generate the 491``PERF_RECORD_LOST`` records. 492 493.. _writing_samples_into_buffer: 494 4952.3.3 Writing samples into buffer 496^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 497 498When a sample is taken and saved into the ring buffer, the kernel 499prepares sample fields based on the sample type; then it prepares the 500info for writing ring buffer which is stored in the structure 501``perf_output_handle``. In the end, the kernel outputs the sample into 502the ring buffer and updates the head pointer in the user page so the 503perf tool can see the latest value. 504 505The structure ``perf_output_handle`` serves as a temporary context for 506tracking the information related to the buffer. The advantages of it is 507that it enables concurrent writing to the buffer by different events. 508For example, a software event and a hardware PMU event both are enabled 509for profiling, two instances of ``perf_output_handle`` serve as separate 510contexts for the software event and the hardware event respectively. 511This allows each event to reserve its own memory space for populating 512the record data. 513 5142.3.4 Reading samples from buffer 515^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 516 517In the user space, the perf tool utilizes the ``perf_event_mmap_page`` 518structure to handle the head and tail of the buffer. It also uses 519``perf_mmap`` structure to keep track of a context for the ring buffer, this 520context includes information about the buffer's starting and ending 521addresses. Additionally, the mask value can be utilized to compute the 522circular buffer pointer even for an overflow. 523 524Similar to the kernel, the perf tool in the user space first reads out 525the recorded data from the ring buffer, and then updates the buffer's 526tail pointer ``perf_event_mmap_page::data_tail``. 527 528.. _memory_synchronization: 529 5302.3.5 Memory synchronization 531^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 532 533The modern CPUs with relaxed memory model cannot promise the memory 534ordering, this means it’s possible to access the ring buffer and the 535``perf_event_mmap_page`` structure out of order. To assure the specific 536sequence for memory accessing perf ring buffer, memory barriers are 537used to assure the data dependency. The rationale for the memory 538synchronization is as below:: 539 540 Kernel User space 541 542 if (LOAD ->data_tail) { LOAD ->data_head 543 (A) smp_rmb() (C) 544 STORE $data LOAD $data 545 smp_wmb() (B) smp_mb() (D) 546 STORE ->data_head STORE ->data_tail 547 } 548 549The comments in tools/include/linux/ring_buffer.h gives nice description 550for why and how to use memory barriers, here we will just provide an 551alternative explanation: 552 553(A) is a control dependency so that CPU assures order between checking 554pointer ``perf_event_mmap_page::data_tail`` and filling sample into ring 555buffer; 556 557(D) pairs with (A). (D) separates the ring buffer data reading from 558writing the pointer ``data_tail``, perf tool first consumes samples and then 559tells the kernel that the data chunk has been released. Since a reading 560operation is followed by a writing operation, thus (D) is a full memory 561barrier. 562 563(B) is a writing barrier in the middle of two writing operations, which 564makes sure that recording a sample must be prior to updating the head 565pointer. 566 567(C) pairs with (B). (C) is a read memory barrier to ensure the head 568pointer is fetched before reading samples. 569 570To implement the above algorithm, the ``perf_output_put_handle()`` function 571in the kernel and two helpers ``ring_buffer_read_head()`` and 572``ring_buffer_write_tail()`` in the user space are introduced, they rely 573on memory barriers as described above to ensure the data dependency. 574 575Some architectures support one-way permeable barrier with load-acquire 576and store-release operations, these barriers are more relaxed with less 577performance penalty, so (C) and (D) can be optimized to use barriers 578``smp_load_acquire()`` and ``smp_store_release()`` respectively. 579 580If an architecture doesn’t support load-acquire and store-release in its 581memory model, it will roll back to the old fashion of memory barrier 582operations. In this case, ``smp_load_acquire()`` encapsulates 583``READ_ONCE()`` + ``smp_mb()``, since ``smp_mb()`` is costly, 584``ring_buffer_read_head()`` doesn't invoke ``smp_load_acquire()`` and it uses 585the barriers ``READ_ONCE()`` + ``smp_rmb()`` instead. 586 5873. The mechanism of AUX ring buffer 588=================================== 589 590In this chapter, we will explain the implementation of the AUX ring 591buffer. In the first part it will discuss the connection between the 592AUX ring buffer and the regular ring buffer, then the second part will 593examine how the AUX ring buffer co-works with the regular ring buffer, 594as well as the additional features introduced by the AUX ring buffer for 595the sampling mechanism. 596 5973.1 The relationship between AUX and regular ring buffers 598--------------------------------------------------------- 599 600Generally, the AUX ring buffer is an auxiliary for the regular ring 601buffer. The regular ring buffer is primarily used to store the event 602samples and every event format complies with the definition in the 603union ``perf_event``; the AUX ring buffer is for recording the hardware 604trace data and the trace data format is hardware IP dependent. 605 606The general use and advantage of the AUX ring buffer is that it is 607written directly by hardware rather than by the kernel. For example, 608regular profile samples that write to the regular ring buffer cause an 609interrupt. Tracing execution requires a high number of samples and 610using interrupts would be overwhelming for the regular ring buffer 611mechanism. Having an AUX buffer allows for a region of memory more 612decoupled from the kernel and written to directly by hardware tracing. 613 614The AUX ring buffer reuses the same algorithm with the regular ring 615buffer for the buffer management. The control structure 616``perf_event_mmap_page`` extends the new fields ``aux_head`` and ``aux_tail`` 617for the head and tail pointers of the AUX ring buffer. 618 619During the initialisation phase, besides the mmap()-ed regular ring 620buffer, the perf tool invokes a second syscall in the 621``auxtrace_mmap__mmap()`` function for the mmap of the AUX buffer with 622non-zero file offset; ``rb_alloc_aux()`` in the kernel allocates pages 623correspondingly, these pages will be deferred to map into VMA when 624handling the page fault, which is the same lazy mechanism with the 625regular ring buffer. 626 627AUX events and AUX trace data are two different things. Let's see an 628example:: 629 630 perf record -a -e cycles -e cs_etm/@tmc_etr0/ -- sleep 2 631 632The above command enables two events: one is the event *cycles* from PMU 633and another is the AUX event *cs_etm* from Arm CoreSight, both are saved 634into the regular ring buffer while the CoreSight's AUX trace data is 635stored in the AUX ring buffer. 636 637As a result, we can see the regular ring buffer and the AUX ring buffer 638are allocated in pairs. The perf in default mode allocates the regular 639ring buffer and the AUX ring buffer per CPU-wise, which is the same as 640the system wide mode, however, the default mode records samples only for 641the profiled program, whereas the latter mode profiles for all programs 642in the system. For per-thread mode, the perf tool allocates only one 643regular ring buffer and one AUX ring buffer for the whole session. For 644the per-CPU mode, the perf allocates two kinds of ring buffers for 645selected CPUs specified by the option ``-C``. 646 647The below figure demonstrates the buffers' layout in the system wide 648mode; if there are any activities on one CPU, the AUX event samples and 649the hardware trace data will be recorded into the dedicated buffers for 650the CPU. 651 652:: 653 654 T1 T2 T1 655 +----+ +-----------+ +----+ 656 CPU0 |xxxx| |xxxxxxxxxxx| |xxxx| 657 +----+--------------+-----------+----------+----+--------> 658 | | | 659 v v v 660 +-----------------------------------------------------+ 661 | Ring buffer 0 | 662 +-----------------------------------------------------+ 663 | | | 664 v v v 665 +-----------------------------------------------------+ 666 | AUX Ring buffer 0 | 667 +-----------------------------------------------------+ 668 669 T1 670 +-----+ 671 CPU1 |xxxxx| 672 -----+-----+---------------------------------------------> 673 | 674 v 675 +-----------------------------------------------------+ 676 | Ring buffer 1 | 677 +-----------------------------------------------------+ 678 | 679 v 680 +-----------------------------------------------------+ 681 | AUX Ring buffer 1 | 682 +-----------------------------------------------------+ 683 684 T1 T3 685 +----+ +-------+ 686 CPU2 |xxxx| |xxxxxxx| 687 --------------------------+----+--------+-------+--------> 688 | | 689 v v 690 +-----------------------------------------------------+ 691 | Ring buffer 2 | 692 +-----------------------------------------------------+ 693 | | 694 v v 695 +-----------------------------------------------------+ 696 | AUX Ring buffer 2 | 697 +-----------------------------------------------------+ 698 699 T1 700 +--------------+ 701 CPU3 |xxxxxxxxxxxxxx| 702 -----------+--------------+------------------------------> 703 | 704 v 705 +-----------------------------------------------------+ 706 | Ring buffer 3 | 707 +-----------------------------------------------------+ 708 | 709 v 710 +-----------------------------------------------------+ 711 | AUX Ring buffer 3 | 712 +-----------------------------------------------------+ 713 714 T1: Thread 1; T2: Thread 2; T3: Thread 3 715 x: Thread is in running state 716 717 Figure 8. AUX ring buffer for system wide mode 718 7193.2 AUX events 720-------------- 721 722Similar to ``perf_output_begin()`` and ``perf_output_end()``'s working for the 723regular ring buffer, ``perf_aux_output_begin()`` and ``perf_aux_output_end()`` 724serve for the AUX ring buffer for processing the hardware trace data. 725 726Once the hardware trace data is stored into the AUX ring buffer, the PMU 727driver will stop hardware tracing by calling the ``pmu::stop()`` callback. 728Similar to the regular ring buffer, the AUX ring buffer needs to apply 729the memory synchronization mechanism as discussed in the section 730:ref:`memory_synchronization`. Since the AUX ring buffer is managed by the 731PMU driver, the barrier (B), which is a writing barrier to ensure the trace 732data is externally visible prior to updating the head pointer, is asked 733to be implemented in the PMU driver. 734 735Then ``pmu::stop()`` can safely call the ``perf_aux_output_end()`` function to 736finish two things: 737 738- It fills an event ``PERF_RECORD_AUX`` into the regular ring buffer, this 739 event delivers the information of the start address and data size for a 740 chunk of hardware trace data has been stored into the AUX ring buffer; 741 742- Since the hardware trace driver has stored new trace data into the AUX 743 ring buffer, the argument *size* indicates how many bytes have been 744 consumed by the hardware tracing, thus ``perf_aux_output_end()`` updates the 745 header pointer ``perf_buffer::aux_head`` to reflect the latest buffer usage. 746 747At the end, the PMU driver will restart hardware tracing. During this 748temporary suspending period, it will lose hardware trace data, which 749will introduce a discontinuity during decoding phase. 750 751The event ``PERF_RECORD_AUX`` presents an AUX event which is handled in the 752kernel, but it lacks the information for saving the AUX trace data in 753the perf file. When the perf tool copies the trace data from AUX ring 754buffer to the perf data file, it synthesizes a ``PERF_RECORD_AUXTRACE`` 755event which is not a kernel ABI, it's defined by the perf tool to describe 756which portion of data in the AUX ring buffer is saved. Afterwards, the perf 757tool reads out the AUX trace data from the perf file based on the 758``PERF_RECORD_AUXTRACE`` events, and the ``PERF_RECORD_AUX`` event is used to 759decode a chunk of data by correlating with time order. 760 7613.3 Snapshot mode 762----------------- 763 764Perf supports snapshot mode for AUX ring buffer, in this mode, users 765only record AUX trace data at a specific time point which users are 766interested in. E.g. below gives an example of how to take snapshots 767with 1 second interval with Arm CoreSight:: 768 769 perf record -e cs_etm/@tmc_etr0/u -S -a program & 770 PERFPID=$! 771 while true; do 772 kill -USR2 $PERFPID 773 sleep 1 774 done 775 776The main flow for snapshot mode is: 777 778- Before a snapshot is taken, the AUX ring buffer acts in free run mode. 779 During free run mode the perf doesn't record any of the AUX events and 780 trace data; 781 782- Once the perf tool receives the *USR2* signal, it triggers the callback 783 function ``auxtrace_record::snapshot_start()`` to deactivate hardware 784 tracing. The kernel driver then populates the AUX ring buffer with the 785 hardware trace data, and the event ``PERF_RECORD_AUX`` is stored in the 786 regular ring buffer; 787 788- Then perf tool takes a snapshot, ``record__read_auxtrace_snapshot()`` 789 reads out the hardware trace data from the AUX ring buffer and saves it 790 into perf data file; 791 792- After the snapshot is finished, ``auxtrace_record::snapshot_finish()`` 793 restarts the PMU event for AUX tracing. 794 795The perf only accesses the head pointer ``perf_event_mmap_page::aux_head`` 796in snapshot mode and doesn’t touch tail pointer ``aux_tail``, this is 797because the AUX ring buffer can overflow in free run mode, the tail 798pointer is useless in this case. Alternatively, the callback 799``auxtrace_record::find_snapshot()`` is introduced for making the decision 800of whether the AUX ring buffer has been wrapped around or not, at the 801end it fixes up the AUX buffer's head which are used to calculate the 802trace data size. 803 804As we know, the buffers' deployment can be per-thread mode, per-CPU 805mode, or system wide mode, and the snapshot can be applied to any of 806these modes. Below is an example of taking snapshot with system wide 807mode. 808 809:: 810 811 Snapshot is taken 812 | 813 v 814 +------------------------+ 815 | AUX Ring buffer 0 | <- aux_head 816 +------------------------+ 817 v 818 +--------------------------------+ 819 | AUX Ring buffer 1 | <- aux_head 820 +--------------------------------+ 821 v 822 +--------------------------------------------+ 823 | AUX Ring buffer 2 | <- aux_head 824 +--------------------------------------------+ 825 v 826 +---------------------------------------+ 827 | AUX Ring buffer 3 | <- aux_head 828 +---------------------------------------+ 829 830 Figure 9. Snapshot with system wide mode 831