[devel] parallel builds, again (was: [Fwd: [#16528] COMPLETE (try 23)])

[Alexey Tourbin]

On Fri, Dec 11, 2009 at 09:56:02PM +0200, Michael Shigorin wrote:
> On Fri, Dec 11, 2009 at 09:28:00PM +0300, Alexey Tourbin wrote:
> > > Does at@ or ldv@ have any ideas how to further improve
> > > builder performance? There never was so stressfull load on
> > > infrastructure and hope never be in near future, but will we
> > > wait so long time if it happens again?
> > First off (perhaps in case you didn't notice), we're doing
> > blazing fast.
> 
> I think it's just as "fast" as when one stands for an hour
> trying to book tickets for the train departing roughly in an
> hour, while the operator can do nothing but wait until the lock
> somewhere is released.

So you argue "blazing fast" isn't fast enough.  Well, you see, this
whole business of building packages actually can't work miracles.
Realistically, we're doing pretty well.  That's all I have to say.

> Single-threaded build is getting us nowhere when CPU clocks are
> effectively frozen at <= 3GHz while cores are plenty; ARM case
> is even more sensitive to single system performance being a
> bottleneck (as discussed on devel-ports@).

So what are you trying to say.  More cores can work more miracles.
Nope.

> > As a matter of fact, a task with 1000 of packages, running for
> > the second time, can complete within only hours.  If you still
> > think this isn't fast enough, consider that 1000 packages
> > actually are 1/n-th part of the whole repo.
> 
> Well you have done great job at shaving seconds off one CPU core
> but there are at least a few more sitting around idle, no?
> If you still think it's fast enough, consider that it could be
> 2x or 4x faster.

I think it can't.

> > > Why packages without direct python dependencies (runtime or
> > > buildtime) was blocked by python task?
> > Because what we do is "purely functional" builds, with no
> > exceptions.
> 
> ...which is pretty expensive as already discussed -- and for
> quite a few cases could be replaced by opportunistic approach
> using cached previous package build results in hope that things
> won't shift much, and falling back to "strict" model in case they
> actually did.

Man, do you really want to even ENGAGE in all this stuff?
It is far more complicated than... oh well...
----------- следующая часть -----------
Было удалено вложение не в текстовом формате...
Имя     : отсутствует
Тип     : application/pgp-signature
Размер  : 198 байтов
Описание: отсутствует
Url     : <http://lists.altlinux.org/pipermail/devel/attachments/20091212/c7a37726/attachment.bin>