Reducing traffic to people.samba.org

The amount of traffic doesn't appear to be a big deal at the moment, though it would be nice to get it down (especially as I register more branches...). The number of HTTP requests is much higher than it should be.

My sniffer tells me bzr does 8 HTTP requests for a no-op pull. spiv mentions branches are polled every 6 hours. That would suggest I have:

200k / (24 / 6) / 8 = 6250 branches

there may be stale locks around - maybe the mirror code is spinning checking for the locks to be released?

read only requests don't look at locks. I'd expect to see the following for a no-op pull:
branch-format
branch/format
repository/format
branch/revision-history (format 5 branches)
or the format 6 equivalent

If a branch fails I think it gets tried again more often.

Can you get a log of the requests for your branches to analyse?

the ui confused me, this is not answered.

I've uploaded the relevant part of the logs for june 16 on http://samba.org/~jelmer/vostok_access.log.gz. It does seem to be hitting an awful lot of kndx files.

Also, note that this is just hitting one of the repositories on that server. There are dozens of other branches registered that use different repositories.

Does that branch contain a broken unicode xml as you had previously? That would cause the mirror to fail, and thus its actually doing a full copy; its not waiting 24 hours because it hasn't *finished* a mirror.

82.211.81.156 - - [16/Jun/2007:00:05:07 -0500] "GET /bzr/jelmer/samba/.bzr/repository/knits/34/svn-v1%253a6%400c0555d6-39d7-0310-84fc-f1cc0bd64818-branches%25252f%2553%2541%254d%2542%2541_4_0-source%25252fsmb_server%25252ftrans2.c.kndx HTTP/1.1" 200 4669 "-" "bzr/0.16.0 (urllib)"
82.211.81.156 - - [16/Jun/2007:00:06:09 -0500] "GET /bzr/jelmer/samba/4.0-signalreplace/.bzr/branch-format HTTP/1.1" 200 35 "-" "bzr/0.16.0 (urllib)"
82.211.81.156 - - [16/Jun/2007:00:06:09 -0500] "GET /bzr/jelmer/samba/4.0-signalreplace/.bzr/branch/format HTTP/1.1" 200 26 "-" "bzr/0.16.0 (urllib)"
82.211.81.156 - - [16/Jun/2007:00:06:09 -0500] "GET /bzr/jelmer/samba/4.0-signalreplace/.bzr/repository/format HTTP/1.1" 404 406 "-" "bzr/0.16.0 (urllib)"
82.211.81.156 - - [16/Jun/2007:00:06:09 -0500] "GET /bzr/jelmer/samba/.bzr/branch-format HTTP/1.1" 200 35 "-" "bzr/0.16.0 (urllib)"
82.211.81.156 - - [16/Jun/2007:00:06:10 -0500] "GET /bzr/jelmer/samba/.bzr/repository/format HTTP/1.1" 200 34 "-" "bzr/0.16.0 (urllib)"
82.211.81.156 - - [16/Jun/2007:00:06:10 -0500] "HEAD /bzr/jelmer/samba/.bzr/repository/shared-storage HTTP/1.1" 200 - "-" "bzr/0.16.0 (urllib)"
82.211.81.156 - - [16/Jun/2007:00:06:10 -0500] "GET /bzr/jelmer/samba/4.0-signalreplace/.bzr/branch/revision-history HTTP/1.1" 200 643021 "-" "bzr/0.16.0 (urllib)"
82.211.81.156 - - [16/Jun/2007:00:06:11 -0500] "GET /bzr/jelmer/samba/.bzr/repository/revisions.kndx HTTP/1.1" 200 5232743 "-" "bzr/0.16.0 (urllib)"

I think the timings in the above support my hypothesis.

Our puller is getting this error:
RevisionNotPresent: Revision {svn-v1:2044@0c0555d6-39d7-0310-84fc-f1cc0bd64818-branches%2fSAMBA_4_0} not present in KnitVersionedFile(http://people.samba.org/bzr/jelmer/samba/.bzr/repository/knits/34/svn-v1%253a6%400c0555d6-39d7-0310-84fc-f1cc0bd64818-branches%25252f%2553%2541%254d%2542%2541_4_0-source%25252fsmb_server%25252ftrans2.c).

The frequency is high as we presume errors are transient things we should quickly catch when they get corrected.