Linux bugs fixed per release over time: >2× rise since 2026-04

Development statistics for the 7.2 kernel bigger than I would have wished for". In fact, 7.2 was one of the busiest development cycles in the kernel's history, adding nearly 600,000 lines of code. It's time to look at some statistics to get a handle on how the kernel's development community is changing. There were 16,418 non-merge commits brought into the mainline for the 7.2 release, exceeding the (already busy) 7.1 release; it is, in fact, the second-busiest release in the kernel's history. Only the 6.7 release brought in more commits but, while 6.7 was dominated by the merging of the bcachefs development history, the volume for 7.2 is more broad-based. $ sudo subscribe todayThese commits were contributed by 2,652 developers, a big increase over the previous record (2,479 in 7.1). If one looks at the history of developer involvement since the beginning of the Git era, the result is:Subscribe today and elevate your LWN privileges. You’ll have access to all of LWN’s high-quality articles as soon as they’re published, and help support LWN in the process. Act now and you can start with a free trial subscription. (This plot, like much of the information here, comes from the subscriber-only LWN Kernel Source Database). There has been a clear trend over that entire history: the number of developers contributing to each release increases over time. But the slope of that line has clearly changed. The most active developers working on 7.2 were: Most active 7.2 developers By changesets Uwe Kleine-König 151 0.9% Rosen Penev 146 0.9% Ian Rogers 138 0.8% Johan Hovold 133 0.8% Arnaldo Carvalho de Melo 131 0.8% Chuck Lever 129 0.8% Eric Dumazet 121 0.7% Ville Syrjälä 115 0.7% Johannes Berg 112 0.7% Krzysztof Kozlowski 110 0.7% Thorsten Blum 108 0.7% Namjae Jeon 106 0.6% Christoph Hellwig 104 0.6% Sean Christopherson 100 0.6% Jakub Kicinski 96 0.6% SJ Park 94 0.6% Mike Rapoport 91 0.6% Dmitry Baryshkov 87 0.5% Eric Biggers 86 0.5% Rafael J. Wysocki 78 0.5% By changed lines Matthew Stewart 83587 8.4% Hawking Zhang 62893 6.3% Miguel Ojeda 38979 3.9% Ian Rogers 31949 3.2% Darren Ye 20843 2.1% Ethan Nelson-Moore 16893 1.7% Harry Wentland 15233 1.5% Uwe Kleine-König 13803 1.4% Ingo Molnar 13649 1.4% Chuck Lever 10172 1.0% Eric Biggers 10067 1.0% Jakub Kicinski 9449 0.9% Rodrigo Siqueira 7749 0.8% Johannes Berg 7110 0.7% Arnd Bergmann 7085 0.7% Emil Tsalapatis 6810 0.7% Vivek Aknurwar 6544 0.7% Rob Clark 6490 0.6% Sabrina Dubroca 6401 0.6% Svyatoslav Ryhel 6072 0.6% Uwe Kleine-König [KSDB] was the biggest contributor of commits to the 7.2 release with cleanups through the driver subsystem. Rosen Penev [KSDB] contributed a long list of LLM-authored bug fixes throughout the architecture-specific, driver, networking, and crypto subsystems, among others. Ian Rogers [KSDB] worked extensively on the user-space perf tool, as did Arnaldo Carvalho de Melo [KSDB]. Johan Hovold [KSDB] made numerous improvements, mostly in the SPI and USB subsystems. Matthew Stewart [KSDB] reached the top of the lines-changed column by following the time-honored tradition of adding more massive, machine-generated amdgpu header files; Hawking Zhang [KSDB] did likewise. Miguel Ojeda [KSDB] brought the Rust zerocopy crate into the kernel source. Darren Ye [KSDB] added the MediaTek mt8196 sound driver. Just over 8% of the commits in 7.2 carried Tested-by tags, while 48% included Reviewed-by tags. The review tags, in particular, are showing a mild downward trend in recent releases: Whether that decline reflects an actual decrease in the amount of review taking place is unclear at best. In any case the most active testers and reviews for this cycle were: Test and review credits in 7.2 Tested-by Dan Wheeler 117 7.0% Luiz Capitulino 54 3.2% James Clark 44 2.6% Sarthak Sharma 44 2.6% Venkat Rao Bagalkote 32 1.9% Tommaso Merciai 27 1.6% David Riley 27 1.6% Jie Gan 27 1.6% Geert Uytterhoeven 25 1.5% Arthur Kiyanovski 25 1.5% Marek Szyprowski 25 1.5% Randy Dunlap 24 1.4% Shivaprasad G Bhat 24 1.4% Rinitha S 21 1.3% Vitaly Prosyak 20 1.2% Reviewed-by Konrad Dybcio 231 2.2% Dmitry Baryshkov 225 2.1% Andy Shevchenko 175 1.7% Geert Uytterhoeven 171 1.6% Krzysztof Kozlowski 152 1.5% Frank Li 147 1.4% Ilpo Järvinen 145 1.4% Christoph Hellwig 143 1.4% Jan Kara 121 1.2% Christian König 120 1.1% Linus Walleij 116 1.1% Alex Deucher 105 1.0% Simon Horman 102 1.0% Jeff Layton 101 1.0% Ido Schimmel 94 0.9% Dan Wheeler [KSDB] is, as always, the top tester by virtue of his work with AMD graphics-driver patches. On the review side, Konrad Dybcio [KSDB] added his tag to 231 changes while Dmitry Baryshkov [KSDB] tagged 225; the same two, who both tend to focus on drivers and devicetree files for Qualcomm devices, were at the top of that list for the 7.1 cycle as well. Work on the 7.2 kernel was supported by 249 employers that we were able to identify; the most active of those were: Most active 7.2 employers By changesets (Unknown) 2743 16.7% Intel 1512 9.2% 1191 7.3% Red Hat 873 5.3% AMD 813 5.0% Qualcomm 767 4.7% (None) 743 4.5% NVIDIA 535 3.3% Meta 441 2.7% (Consultant) 429 2.6% SUSE 358 2.2% Renesas Electronics 328 2.0% IBM 307 1.9% Kylin 278 1.7% NXP Semiconductors 264 1.6% Oracle 245 1.5% Arm 222 1.4% BayLibre 203 1.2% Microsoft 201 1.2% Bootlin 194 1.2% By lines changed AMD 224121 22.4% (Unknown) 103861 10.4% 102565 10.3% Qualcomm 59021 5.9% Intel 53661 5.4% Red Hat 51559 5.2% Meta 38635 3.9% (None) 28218 2.8% NVIDIA 27417 2.7% MediaTek 22025 2.2% NXP Semiconductors 17692 1.8% BayLibre 15972 1.6% Oracle 12233 1.2% Microsoft 11786 1.2% Renesas Electronics 11103 1.1% Linaro 10209 1.0% (Consultant) 9362 0.9% IBM 9205 0.9% Texas Instruments 8398 0.8% SUSE 8350 0.8% The most significant change this time around, arguably, is the significant increase in developers with unknown affiliation. This is partially, at least, a reflection of the ongoing flood of new developers entering the kernel community. New developers and their tools Each of the previous two kernel releases had set a record for the number of first-time developers contributing changes; 7.2 has broken that record again, with 613 contributors [KSDB] who had not been seen here before. Here, once again, the trend is clear: The most active new developer was Bryam Vargas [KSDB], who contributed 43 commits fixing problems in the input, NVMe, F2FS, AppArmor, Landlock, SELinux, NTFS, WiFi, ATA, SCSI, networking, Ceph, and Thunderbolt subsystems, among others. As has been observed here before, it would be difficult for a seasoned kernel developer to reach deeply into that many subsystems over the course of a single development cycle — at least, if they wanted to understand what they are doing. For a previously unknown developer to do this is surprising at best. Vargas's patches do not carry Assisted-by tags, though, so we are left to believe that this work was done without LLM assistance. Overall, 1,111 changes in 7.2 (under 7% of the total) carried Assisted-by tags — a significant increase over the 301 seen in 7.1. The real count of LLM-assisted patches in 7.2 must surely be far higher than that, though. Many developers do not supply those tags (often — but not always — out of ignorance of the rules), maintainers rarely ask about LLM use, and at least one maintainer actively strips those tags when applying patches. Accurately tracking commits developed with LLM assistance may turn out not to be possible, but it sometimes feels like the kernel community does not care enough to really try. The other way in which LLM usage is making itself felt in kernel development is in the area of patch review. There are 252 patches in 7.2 with Reported-by tags crediting the Sashiko review system [KSDB]; 568 commits mention Sashiko in their changelogs. Other LLMs are also clearly being used to find kernel bugs, though, with real effects. The 7.2 release contains 4,830 commits with Fixes tags, indicating that they fix a bug introduced by a previous commit. To see just how significant an increase is, consider this plot, an updated version of one that has been seen here before (most recently for the 6.17 development cycle): The heavy green line plots how many commits in each release contained Fixes tags, while the heavy brown line is the number of commits that were identified by Fixes tags in subsequent commits. The crossover point is at 5.9, suggesting that, as of that release, the kernel community started fixing more bugs than it introduced. That conclusion will not hold, though; it will take years for all of the post-5.9 bugs to be identified and fixed, and that line will shift as it happens. The noteworthy part of the plot, though, is at the right-hand side, where the number of commits carrying Fixes tags increases dramatically, after having been roughly level for quite a few releases. That, it seems, is what happens once large numbers of people start turning LLMs toward the task of finding bugs in existing software. This is going to be an interesting trend to watch. The number of bugs cannot be infinite — even in the kernel — so the rate of bug detection and fixing must eventually fall off. The existing bugs will have been fixed, and LLM review will have reduced the number of new bugs introduced. At least, that is the hope. If that drop-off does not happen, there will be some interesting questions to answer. It seems reasonably clear that the spread of LLMs has enabled people to create kernel patches who would not have been capable of doing so before. If that is true, it is not much of a stretch to conclude that, in many cases, the submitters of these patches do not fully understand what those patches do. Perhaps the LLMs are good enough that this understanding is no longer necessary, but it could also be that we are creating a pile of code that nobody has ever really understood, but which will need to be maintained in the coming years. The Linux kernel is aimed at production use, not research or experimentation. But there is currently a massive experiment being run on the kernel's development processes and community. This community has absorbed new processes and tools many times over its 35-year history, and will almost certainly find a way to do that this time as well. But that absorption has not always been without turbulence along the way. Meanwhile, there are nearly 14,000 non-merge changesets sitting in the linux-next repository and poised to be pulled for the 7.3 release; over 1,100 of them contain Assisted-by tags. It would seem that the high level of activity that has characterized recent releases will not be ending anytime soon. As always, keep an eye on LWN as we follow the next development cycle (and those that come after). | Index entries for this article | | |---|---| | Kernel | Development tools/Large language models | | Kernel | Releases/7.2 |