[devel] parallel builds, again (was: [Fwd: [#16528] COMPLETE (try 23)])

[Michael Shigorin]

On Fri, Dec 11, 2009 at 09:28:00PM +0300, Alexey Tourbin wrote:
> > Does at@ or ldv@ have any ideas how to further improve
> > builder performance? There never was so stressfull load on
> > infrastructure and hope never be in near future, but will we
> > wait so long time if it happens again?
> First off (perhaps in case you didn't notice), we're doing
> blazing fast.

I think it's just as "fast" as when one stands for an hour
trying to book tickets for the train departing roughly in an
hour, while the operator can do nothing but wait until the lock
somewhere is released.

Single-threaded build is getting us nowhere when CPU clocks are
effectively frozen at <= 3GHz while cores are plenty; ARM case
is even more sensitive to single system performance being a
bottleneck (as discussed on devel-ports@).

2 ldv: what do you think of a week with parallel builds turned on
and taking a log analysis vacation? :)

(BTW having "main" packages built single-threaded and "contrib"
packages built multi-threaded might help too -- if we had those)

> As a matter of fact, a task with 1000 of packages, running for
> the second time, can complete within only hours.  If you still
> think this isn't fast enough, consider that 1000 packages
> actually are 1/n-th part of the whole repo.

Well you have done great job at shaving seconds off one CPU core
but there are at least a few more sitting around idle, no?
If you still think it's fast enough, consider that it could be
2x or 4x faster.

> > Why packages without direct python dependencies (runtime or
> > buildtime) was blocked by python task?
> Because what we do is "purely functional" builds, with no
> exceptions.

...which is pretty expensive as already discussed -- and for
quite a few cases could be replaced by opportunistic approach
using cached previous package build results in hope that things
won't shift much, and falling back to "strict" model in case they
actually did.

-- 
 ---- WBR, Michael Shigorin <mike на altlinux.ru>
  ------ Linux.Kiev http://www.linux.kiev.ua/