Launchpad itself

arm64 (standard) ubuntu kernel builds fail

Asked by Frank Heimes

I probably asked this question previously in the wrong area (#699731 in 'Ubuntu') - now again here, in Launchpad.

Just recently kernel builds of the standard Ubuntu kernels started to fail on arm64 (and only on arm64).
I see two error messages and I am not sure if they are related - but there seem to be two issues:
"E: Build killed with signal TERM after 150 minutes of inactivity"
and
"Build needed 13:17:20, no disk space"
(https://launchpadlibrarian.net/572020654/buildlog_ubuntu-jammy-arm64.linux_5.15.0-13.13~lp1903288_BUILDING.txt.gz)

(The failure of the builds of arm64 kernels probably started when add. kernel flavors or even RT was added?!)

One workaround is probably to disable some of the builds (like special kernels and/or the generation of dbgsym) - however, I think it would be good if a standard kernel build succeeds again for arm64 (with 'standard' I mean: just take the Canonical kernel teams kernel sources and just rebuild them as they are in a PPA).

I got conformation from the kernel team that this happens for others as well if he builds in a private PPA; but it does not happen in the kernel team PPAs (like bootstrap or unstable) - but we believe that these have none-standard resources ...

In case this should be transferred into a LP bug rather than a question, let me know ...

Thx, Frank

Question information

Language:
English Edit question
Status:
Solved
For:
Launchpad itself Edit question
Assignee:
No assignee Edit question
Solved by:
Frank Heimes
Solved:
Last query:
Last reply:
Revision history for this message
Jürgen Gmach (jugmac00) said : 		#1

Thank you for the question. I need to ask a colleague about that and get back to you as soon as possible.

Revision history for this message
Frank Heimes (fheimes) said : 		#2

Thx Jürgen

Just as a side note:
I believe the kernel teams does not see this as an issue with the kernel sources themselves, since they told me that builds succeed for them if they use their special PPAs (like 'unstable' and 'bootstrap').

Revision history for this message
William Grant (wgrant) said : 		#3

There's no difference in the build environments besides whether the "Build debug symbols" option is enabled, which it obviously is for the Ubuntu primary archive, where this builds successfully: https://launchpadlibrarian.net/571443663/buildlog_ubuntu-jammy-arm64.linux_5.15.0-13.13_BUILDING.txt.gz . I suspect what's happening here is that that ddeb compression is hovering right around 150 minutes, and you happened to by chance catch a buildd that was slightly more heavily loaded than the primary archive build: your build failed after 13:17:20, while the official build succeeded in 13:02:25.

5.15 might have just pushed this over the edge, since 5.13 (https://launchpadlibrarian.net/562607353/buildlog_ubuntu-impish-arm64.linux_5.13.0-19.19_BUILDING.txt.gz) took only 11:49:18, and the new ddeb is 5% larger than the old one. I think there's a good chance that a retry will work, but you'll probably need to talk to the kernel team about how to fix this in the packaging.

Revision history for this message
Frank Heimes (fheimes) said : 		#4

Hi William, thanks for the response and taking the time.

Yeah, the arm build hardware is known not to be the very fastest,
hence the first thing I did was indeed a retry (2 times, in different PPAs), but without any luck.

(And yes, I also wondered why the arm64 builds - that usually took about 11 hours, now failed after 13 - there must be now more to build - my suspicion was that maybe an additional RT kernel was added or so ...)

Next thing was to contact the kernel team, where I got confirmation that it happens for some people there, too - in case they just use regular (personal) PPA. But their 'solution' was in such cases to move such builds to their unstable or bootstrap PPAs, where things work fine. However, this is not an option for a non-Kernel engineer.

Anyway, I'll go back to the kernel team (Andrea) and re-start the discussion ....

(PS: It's of course not super critical, since one can just disable certain builds, like the ddebs, however, I still think it should work even without doing this.)

Revision history for this message
William Grant (wgrant) said : 		#5

There's nothing magical about the unstable or bootstrap PPAs, unless they contain weird packages that somehow execute more quickly, or they have ddebs disabled. They don't built on special builders or have special timeouts or anything like that.

Revision history for this message
Frank Heimes (fheimes) said : 		#6

Oh, that interesting - after having heard that it seems to work there I assumed these PPAs have indeed more resources, or other timeouts or so - hmm ...
So one more indication that it should be 'solvable' by a package adjustment ...

Thx

Hence I mark this question here as solved, since it has probably not much or even nothing to do with the PPA itself ...

Revision history for this message
Colin Watson (cjwatson) said : 		#7

It's also possible that you just happened to hit a slower compute node. When a builder is reset, OpenStack assigns the VM to one of the available compute nodes; you can't infer which it was from the Launchpad builder name, and some of the machines currently in the arm64 compute node pool are faster than others.

Indeed, when I went to check in our logs, https://launchpad.net/~fheimes/+archive/ubuntu/lp1903288-v5/+build/22579440 landed on swirlix01 (one of the older generation), while https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/bootstrap/+build/22465746 landed on jojora (one of the newer generation). But, as William says, this has nothing to do with which archive the build is in; it's essentially chance.