Intentional bugs: University presents report on analysis of kernel patches


The Technical Advisory Board of the Linux Foundation, which acts as a kind of advisory body for questions about the development of the Linux kernel, has submitted its report to the “Hypocrite Commits” paper of the University of Minnesota (UMN). It summarizes the results of the review of the incriminated commits and gives the UMN tips on how to regain the trust of kernel developers like Greg Kroah-Hartmann. Kroah-Hartmann himself played a key role in the code analysis.

As is well known, the house blessing has been hanging mightily wrong for weeks between the Linux kernel developer community and the University of Minnesota. As part of an experiment, the summary of which is now known as the “Hypocrite Commits Paper”, a small UMN research group deliberately sent false patches to the developers – without obtaining their consent for the experiment. Sarah Jamie Lewis from the Open Privacy Project drew the community’s attention to this fact in late 2020.

When patches of inferior quality from UMN employees landed on the Linux kernel mailing list (LKML) again at the beginning of April 2021, kernel veteran Kroah-Hartmann broke the collar: He not only announced that all UMN patches would be ignored without comment until further notice would; He also announced that he would remove all patches from UMN employees since the “Hypocrite Commits Paper” became known from the kernel (“revert”).

It didn’t help that the authors of the new patches asserted that they had nothing to do with the 2020 experiment. The UMN has meanwhile withdrawn the paper as a result of the experiment itself. In addition, they had previously praised improvement and promised to examine research projects more closely in future before approval. The kernel community remained adamant, however.

In the meantime, the dispute between the kernel community and the UMN has come to an end. On April 22nd, Kees Cook, representing the Technical Advisory Board (“TAB”) of the Linux Foundation, announced that the TAB will start its own investigation into the matter. Kroah-Hartmann is also a member of that body. In any case, the Linux Foundation is trying to take an active role in the conflict: On April 23, Mike Dolan, representing Linux Foundation projects – to which the kernel belongs – sent an open letter to the University of Minnesota containing various demands. Above all, of course, the demand not to abuse the kernel community as a guinea pig again without their consent.

On the one hand, the focus of the parallel TAB investigation is on those commits that Kroah-Hartmann had already removed from the Linux kernel. Patch per patch, the TAB wanted to support Kroah-Hartmann in distinguishing between real bug fixes, bad patches and those patches that revealed sinister intentions.

On the other hand, the TAB also wanted to re-examine the UMN’s approach and make recommendations for the UMN and the entire community on how to regain the trust of the kernel developers in this situation.

The TAB has now published the report on the review. From the point of view of system administrators, the good news of the TAB report is: The vast majority of patches from UMN students or employees In retrospect, it turned out to be harmless. Of the 435 patches in question, 349 are correct and find their way back into the kernel. 39 Patches are wrong and need to be corrected – but the developers do not classify them as fraudulent and do not recognize any security risks.

Nine patches date from before the UMN project even existed, so the developers also classify them as harmless. A total of 37 patches are now obsolete because they have been changed again by subsequent commits or have been blown out of the kernel entirely. A single commit has been removed from the kernel at the request of the author.

The conclusions from the TAB report are less pleasant for the UMN. On the one hand, the Linux Foundation explicitly points out that the relationship of trust between the developers and the UMN has been disrupted and will remain so for the time being. On the other hand, the TAB shows a potential way out and draws parallels to similar cases in the past. Although there has not yet been an experiment like that of the UMN, the kernel community is well aware of the effect that small companies send masses of patches of poor quality to the LKML because they lack the experience and the necessary knowledge.