launchpad-buildd

failure: not enough memory for compilation on amd64

Asked by Václav Šmilauer

I am building the woo package using the recipe https://code.launchpad.net/~eudoxos/+recipe/woo-daily . The amd64 build consistently fails (almost certainly) due to insufficient memory. The code is extremely template-intensive and gcc allocates around 3-4GB of RAM during compilation. I assume the builder starts swapping, which makes it extremely slow and is eventually killed due to inactivity; all failed builds end in this way:

gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DWOO_REVISION=debian -DWOO_VERSION=0.99+3139+9~quantal1 -DWOO_SOURCE_ROOT= -DWOO_FLAVOR= -DWOO_CXX_FLAVOR= -DWOO_QT4 -DWOO_VTK -DWOO_OPENGL -DWOO_GTS -DWOO_OPENMP -DWOO_NOXML -DWOO_CAST=static_cast -DWOO_PTR_CAST=static_pointer_cast -DNDEBUG -DQT_CORE_LIB -DQT_GUI_LIB -DQT_OPENGL_LIB -DQT_SHARED -Ibuild-src-tree -I/usr/include/eigen3 -I/usr/include/qt4 -I/usr/include/qt4/QtCore -I/usr/include/qt4/QtGui -I/usr/include/qt4/QtOpenGL -I/usr/include/qt4/QtXml -I/usr/include/vtk-5.8 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/glib-2.0 -I/usr/include/python2.7 -c build-src-tree/src/chunk-02.cpp -o build/temp.linux-x86_64-2.7/build-src-tree/src/chunk-02.o -Wall -fvisibility=hidden -std=c++11 -pipe -g0 -O3 -march=core2 -fopenmp
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++ [enabled by default]

Session terminated, terminating shell...make: *** [build-arch] Terminated
 ...terminated.
Build killed with signal 15 after 150 minutes of inactivity

I would like to ask if there is a way to get better diagnostics from the builder (such as what was the maximum RAM usage) and if there is a way to have that package built on builders with 4GB RAM, if there are such.

Question information

Language:
English Edit question
Status:
Answered
For:
launchpad-buildd Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Laura Czajkowski (czajkowski) said : 		#1

hi this is more of a packaging issue than a LP issue, you might find http://developer.ubuntu.com/resources/tools/packaging/ useful

Revision history for this message
Václav Šmilauer (eudoxos) said : 		#2

I don't see how is this packaging issue. The packages are built fine on machines with more RAM; there is nothing fancy about packaging, it is very standard package using dhhelper with setup.py.

Revision history for this message
Václav Šmilauer (eudoxos) said : 		#3

(debhelper, not dhhelper; sorry)

Revision history for this message
Curtis Hovey (sinzui) said : 		#4

Launchpad cannot build this package if it cannot be built in less than 4GB of Ram. The strategies to fix this involve revising the makefiles to do less work, or to break the code into smaller pieces, each providing a separate source package that in turn provides a library, component, or end-user tool.

Revision history for this message
Václav Šmilauer (eudoxos) said : 		#5

Can you confirm the limit is 4GB for builders? The i686 build takes long (I assume it swaps) and needs less than 4GB.

The code for buildd sets ulimit to 1GB - http://bazaar.launchpad.net/~canonical-launchpad-branches/launchpad-buildd/trunk/view/head:/buildrecipe#L216. Is that unrelated?

Revision history for this message
Václav Šmilauer (eudoxos) said : 		#6

I went through a few buildlogs, where I added "free" to debian/rules. It reveals that i386 virtual hosts have much more RAM - in the examples below i386 5GB (3.3Gfree), 2GB (.2GB free), amd64 3GB (1.1GB free), 1.5GB (20MB free); all of them also have 2GB swap.

So the real limit for buildability at LP is not 4GB as I thought (I double-checked, and the compiler itself never needed more than roughly 3GB on my machine during the build), but it depends on the builder. In addition, amd64 (virtual) hosts seem to have *less* RAM in average, in one observed case as low as 1.5GB (and none actually free).

i386
=====
(https://launchpadlibrarian.net/126685744/buildlog_ubuntu-precise-i386.woo_0.99%2B3156%2B14~precise1_FAILEDTOBUILD.txt.gz):

            total used free shared buffers cached
Mem: 5190656 1868212 3322444 0 154988 1444112
-/+ buffers/cache: 269112 4921544
Swap: 2097144 0 2097144

(https://launchpadlibrarian.net/126692741/buildlog_ubuntu-quantal-i386.woo_0.99%2B3156%2B13~quantal1_BUILDING.txt.gz):

             total used free shared buffers cached
Mem: 2097152 1852292 244860 0 151776 1466124
-/+ buffers/cache: 234392 1862760
Swap: 2097144 0 2097144

amd64
=======
(https://launchpadlibrarian.net/126684600/buildlog_ubuntu-precise-amd64.woo_0.99%2B3156%2B14~precise1_BUILDING.txt.gz):

free
             total used free shared buffers cached
Mem: 3038208 1839504 1198704 0 146524 1467944
-/+ buffers/cache: 225036 2813172
Swap: 2097144 0 2097144

(https://launchpadlibrarian.net/126731800/buildlog_ubuntu-quantal-amd64.woo_0.99%2B3156%2B13~quantal1_FAILEDTOBUILD.txt.gz):
free
             total used free shared buffers cached
Mem: 1533952 1503488 30464 0 144276 1172292
-/+ buffers/cache: 186920 1347032
Swap: 2097144 0 2097144

Revision history for this message
William Grant (wgrant) said : 		#7

Builders differ in the amount of available RAM due to hardware constraints, but they all have at least 1.5GiB and most have more. If you look at the `free` output you'll see that most of it's used as block cache. It shows 187MB actually used, and 1.3GB of cache that will be evicted when something else needs the RAM -- exactly what you'd expect from a healthy system under little memory pressure.

There's no way to force a build onto a builder with a particular quantity of RAM or other resources.

Can you help with this problem?

Provide an answer of your own, or ask Václav Šmilauer for more information if necessary.

To post a message you must log in.