bzr speed and frequent crashes from Sri Lanka (.lk)
We have been using bzr for the sahana project (http://
Below is an example where it has freezed on a bzr pull
Using saved parent location: http://
[/ ] 976kB @ 1kB/s | Pull phase:Copying inventory texts
My throughput is 52kB/s typically compared to the 1-10kBs I am getting from the bzr launchpad server.
PING bazaar.
This also has a tendency to unrecoverable corrupt my bzr instance. However asking other members of our global community they do not seems to have issues with bzr, but other members in the project from Sri Lanka have complained about similar issues.
Question information
- Language:
- English Edit question
- Status:
- Answered
- For:
- Bazaar Edit question
- Assignee:
- No assignee Edit question
- Last query:
- Last reply:
This question was originally filed as bug #512041.
Related FAQ:
None Link to a FAQ
Revision history for this message
|
#1 |
Hi Chamindra,
That's great to see free software being used for such a cause and I want to make sure bzr is helping you well.
I'm going to convert this into a support question at least until we determine that there is a specific bug here.
Revision history for this message
|
#2 |
Let's work out what's going wrong here.
Revision history for this message
|
#3 |
So, I just tried branching from that URL, and I downloaded 68M in 45s, about 1511kb/s. But this was over quite a different network connection.
When you say it crashes, do you mean it's actually crashing with an error message, or it's just staying stuck at this point for a long time?
To get some more data to track down the problem I'd like your help in trying a few things.
One is that if it crashes, please paste the error message from ~/.bzr.log here.
To look at network problems please run
bzr -Dtransport -Dhttp pull log+http://
this will record some information into ~/.bzr.log; please attach that here too.
And there is another suggestion, which is that if you use ssh to get to Launchpad you may get better performance and you may have better protection from some types of problem on the network between your machine and Launchpad, such as buggy proxy servers. To do that you just need to say
bzr -Dhpss pull bzr+ssh:
and if that hangs or errors, please let me know.
If you are running bzr before 2.0 please upgrade; if you're having trouble with that then ask.
Revision history for this message
|
#4 |
It does not crash with an error message, but just gets stuck (pauses
on the commandline)
On of the problems is that my throughput from Sri Lanka is quite slow.
I typically get 52kBs down and about 1/3rd of that up. However the
link to the launchpad server was much slower than that. Before this I
was using CVS on sourceforge and that was relatively fast.
Attached below is a trace. note:I have obfuscated the name of my home directory
I am using ubuntu 9.04 on this machine and the bzr version reported is
1.13.1 (so yes it is before 2.0). I will eventually upgrade to 9.10,
but do not have time right now. What might be the easiest way for me
to upgrade.
Mon 2010-01-25 02:14:38 +0530
0.031 bzr arguments: [u'repositories']
0.055 looking for plugins in /home/person/
0.055 looking for plugins in /usr/lib/
0.129 Traceback (most recent call last):
File "/usr/lib/
716, in exception_
return the_callable(*args, **kwargs)
File "/usr/lib/
889, in run_bzr
cmd_obj = get_cmd_object(cmd, plugins_
File "/usr/lib/
180, in get_cmd_object
raise errors.
BzrCommandError: unknown command "repositories"
0.129 return code 3
Mon 2010-01-25 02:14:43 +0530
0.031 bzr arguments: [u'help', u'repositories']
0.055 looking for plugins in /home/person/
0.055 looking for plugins in /usr/lib/
0.125 encoding stdout as sys.stdout encoding 'UTF-8'
0.127 return code 0
Mon 2010-01-25 02:19:30 +0530
0.031 bzr arguments: [u'launchpad-
0.055 looking for plugins in /home/person/
0.055 looking for plugins in /usr/lib/
0.125 encoding stdout as sys.stdout encoding 'UTF-8'
0.193 failed to import pycurl: No module named pycurl
0.194 failed to instantiate transport
<bzrlib.
module=
for 'https:/
library "pycurl": No module named pycurl)
17.096 return code 0
Mon 2010-01-25 02:19:47 +0530
0.029 bzr arguments: [u'checkout', u'--lightweight', u'lp:sahana']
0.053 looking for plugins in /home/person/
0.053 looking for plugins in /usr/lib/
0.124 encoding stdout as sys.stdout encoding 'UTF-8'
13.597 ssh implementation is OpenSSH
26.455 creating branch reference in file://
28.281 trying to create missing lock
'/home/
28.282 opening working tree '/home/
139.013 Traceback (most recent call last):
File "/usr/lib/
716, in exception_
return the_callable(*args, **kwargs)
File "/usr/lib/
911, in run_bzr
ret = run(*run_argv)
File "/usr/lib/
547, in run_argv_aliases
return self.run(
File "/usr/lib/
accelerator
File "/usr/lib/
in create_checkout
hardlink=
File "/usr/lib/
in create_workingtree
accelerator
File "/usr/lib/
line 1394, in initialize
hardlink=
File "/usr/lib/
2034, in build_tree
delta_
File "/usr/lib/
2126, in _build_tree
accelerator
File "/usr/lib/
2179, in _create_files
new_
File "/usr/lib/
79, in iter_files_bytes
for result in self._repositor
File "/usr/lib/
1550, in iter_files_bytes
for record in self.texts.
File "/usr/lib/
in get_record_stream
ordering, include_
File "/usr/lib/
in _get_remaining_
positions = self._get_
File "/usr/lib/
in _get_components
build_details = self._index.
File "/usr/lib/
in get_build_details
for entry in entries:
File "/usr/lib/
in _get_entries
for node in self._graph_
File "/usr/lib/
in iter_entries
for node in index.iter_
File "/usr/lib/
in iter_entries
self.
File "/usr/lib/
in _buffer_all
stream = self._transport
File "/usr/lib/
line 223, in get
return StringIO(
File "/usr/lib/
line 234, in get_bytes
return response_
File "/usr/lib/
line 319, in read_body_bytes
self.
File "/usr/lib/
line 268, in _wait_for_
self.
File "/usr/lib/
line 277, in _read_more
bytes = self._medium_
File "/usr/lib/
459, in read_bytes
return self._read_
File "/usr/lib/
471, in _read_bytes
return self._medium.
File "/usr/lib/
149, in read_bytes
return self._read_
File "/usr/lib/
787, in _read_bytes
bytes = self._read_
KeyboardInterrupt
139.013 return code 3
On Mon, Jan 25, 2010 at 10:44 PM, Martin Pool
<email address hidden> wrote:
> Your question #98657 on Bazaar changed:
> https:/
>
> Status: Open => Needs information
>
> Martin Pool requested for more information:
> So, I just tried branching from that URL, and I downloaded 68M in 45s,
> about 1511kb/s. But this was over quite a different network connection.
>
> When you say it crashes, do you mean it's actually crashing with an
> error message, or it's just staying stuck at this point for a long time?
>
> To get some more data to track down the problem I'd like your help in
> trying a few things.
>
> One is that if it crashes, please paste the error message from
> ~/.bzr.log here.
>
> To look at network problems please run
>
> bzr -Dtransport -Dhttp pull log+http://
> haiti/sahana/
>
> this will record some information into ~/.bzr.log; please attach that
> here too.
>
> And there is another suggestion, which is that if you use ssh to get to
> Launchpad you may get better performance and you may have better
> protection from some types of problem on the network between your
> machine and Launchpad, such as buggy proxy servers. To do that you just
> need to say
>
> bzr -Dhpss pull bzr+ssh:
> /haiti-or/
>
> and if that hangs or errors, please let me know.
>
> If you are running bzr before 2.0 please upgrade; if you're having
> trouble with that then ask.
>
> --
> To answer this request for more information, you can either reply to
> this email or enter your reply at the following page:
> https:/
>
> You received this question notification because you are a direct
> subscriber of the question.
>
--
Chamindra de Silva
Home: http://
Twitter: http://
Revision history for this message
|
#5 |
Hi,
You can get the current bzr stable release through
https:/
If you're on a slow network I would really recommend you use a real
branch not a lightweight checkout, which will connect to the server
for many operations. Run
bzr init-repo --0.92 ~/sahana
bzr branch lp:sahaha ~/sahana/trunk
then within that directory say 'bzr pull' to bring down new changes.
--
Martin <http://
Revision history for this message
|
#6 |
Thank you! Just downloaded the latest bzr version and will try this
out. You might be interested to know I have to do 31hops from Sri
Lanka's main telecom provider to get to the server, which is probably
why it can have varying throughputs.
BTW what does bzr init-repo --0.92 ~/sahana do exacrly? why 0.92?
On Mon, Jan 25, 2010 at 11:15 PM, Martin Pool
<email address hidden> wrote:
> Your question #98657 on Bazaar changed:
> https:/
>
> Status: Open => Answered
>
> Martin Pool proposed the following answer:
> Hi,
>
> You can get the current bzr stable release through
> https:/
>
> If you're on a slow network I would really recommend you use a real
> branch not a lightweight checkout, which will connect to the server
> for many operations. Run
>
> bzr init-repo --0.92 ~/sahana
> bzr branch lp:sahaha ~/sahana/trunk
>
> then within that directory say 'bzr pull' to bring down new changes.
>
> --
> Martin <http://
>
> --
> If this answers your question, please go to the following page to let us
> know that it is solved:
> https:/
>
> If you still need help, you can reply to this email or go to the
> following page to enter your feedback:
> https:/
>
> You received this question notification because you are a direct
> subscriber of the question.
>
--
Chamindra de Silva
Home: http://
Twitter: http://
Revision history for this message
|
#7 |
2010/1/25 chamindra <email address hidden>:
> Question #98657 on Bazaar changed:
> https:/
>
> Status: Answered => Open
>
> chamindra is still having a problem:
> Thank you! Just downloaded the latest bzr version and will try this
> out. You might be interested to know I have to do 31hops from Sri
> Lanka's main telecom provider to get to the server, which is probably
> why it can have varying throughputs.
>
> BTW what does bzr init-repo --0.92 ~/sahana do exacrly? why 0.92?
0.92 sets the repository format, and it is the same as the format
shown by "bzr info
http://
If you or any other developers need any other help just ask.
--
Martin <http://
Revision history for this message
|
#8 |
Sorry I have been trying this for a while. I upgraded to the latest version of bzr as mentioned about and it is still taking me ages to upload and download branches. I am talking about effective speeds of 1-2 KBps on average on a connection that should get a lot more (52Kbps). And it keeps crashing.
I attach a log entry
Thu 2010-04-08 02:54:54 +0530
0.062 bazaar version: 2.1.1
0.062 bzr arguments: [u'branch', u'lp:~sahana-php/s3/php-cvs-trunk']
0.084 looking for plugins in /home/wintermut
0.084 looking for plugins in /usr/lib/
0.177 encoding stdout as sys.stdout encoding 'UTF-8'
2.073 ssh implementation is OpenSSH
9.760 creating repository in file://
9.774 Using fetch logic to copy between RemoteRepositor
344.074 Traceback (most recent call last):
File "/usr/lib/
return the_callable(*args, **kwargs)
File "/usr/lib/
ret = run(*run_argv)
File "/usr/lib/
return self.run_
File "/usr/lib/
return self._operation
File "/usr/lib/
self.cleanups, self.func, *args, **kwargs)
File "/usr/lib/
result = func(*args, **kwargs)
File "/usr/lib/
source_
File "/usr/lib/
result_
File "/usr/lib/
find_
File "/usr/lib/
result = unbound(self, *args, **kwargs)
File "/usr/lib/
pb=pb, find_ghosts=
File "/usr/lib/
self.__fetch()
File "/usr/lib/
self.
File "/usr/lib/
stream, from_format, [])
File "/usr/lib/
return self._locked_
File "/usr/lib/
substream)
File "/usr/lib/
for _ in self._insert_
File "/usr/lib/
for record in stream:
File "/usr/lib/
for bytes in self._bytes_
File "/usr/lib/
for record in self.iter_
File "/usr/lib/
for bytes in self.byte_stream:
File "/usr/lib/
self.
File "/usr/lib/
bytes = self._medium_
File "/usr/lib/
return self._read_
File "/usr/lib/
return self._medium.
File "/usr/lib/
return self._read_
File "/usr/lib/
bytes = osutils.
File "/usr/lib/
return f(*a, **kw)
KeyboardInterrupt
344.075 Transferred: 552KiB (1.6K/s r:551K w:1K)
344.075 return code 3
Thu 2010-04-08 03:00:56 +0530
0.071 bazaar version: 2.1.1
0.071 bzr arguments: [u'branch', u'lp:~sahana-php/s3/php-cvs-trunk']
0.094 looking for plugins in /home/wintermut
0.094 looking for plugins in /usr/lib/
0.188 encoding stdout as sys.stdout encoding 'UTF-8'
12.231 ssh implementation is OpenSSH
19.942 creating repository in file://
19.954 Using fetch logic to copy between RemoteRepositor
774.804 Traceback (most recent call last):
File "/usr/lib/
return the_callable(*args, **kwargs)
File "/usr/lib/
ret = run(*run_argv)
File "/usr/lib/
return self.run_
File "/usr/lib/
return self._operation
File "/usr/lib/
self.cleanups, self.func, *args, **kwargs)
File "/usr/lib/
result = func(*args, **kwargs)
File "/usr/lib/
source_
File "/usr/lib/
result_
File "/usr/lib/
find_
File "/usr/lib/
result = unbound(self, *args, **kwargs)
File "/usr/lib/
pb=pb, find_ghosts=
File "/usr/lib/
self.__fetch()
File "/usr/lib/
self.
File "/usr/lib/
stream, from_format, [])
File "/usr/lib/
return self._locked_
File "/usr/lib/
substream)
File "/usr/lib/
for _ in self._insert_
File "/usr/lib/
for record in stream:
File "/usr/lib/
for bytes in self._bytes_
File "/usr/lib/
for record in self.iter_
File "/usr/lib/
for bytes in self.byte_stream:
File "/usr/lib/
self.
File "/usr/lib/
bytes = self._medium_
Revision history for this message
|
#9 |
Something is a bit fishy here, maybe. Here's the result I get:
Branched 4531 revision(s).
Transferred: 77272kB (25.2kB/s r:77270kB w:1kB)
HPSS calls: 11 (0 vfs) SmartSSHClientM
real 51m3.941s
user 0m31.630s
sys 0m4.716s
That does seem unexpectedly slow for streaming of a 2a repository (which this is). Like chamindra, I expect my connection to be capable of much faster than the result I got. The HPSS call log looks normal, just a bit slow at receiving the data. Possibly there was some sort of congestion between my home in Sydney and the Launchpad server in London at the time?
Here's a random snippet of the -Dhpssdetail output from the middle of the stream:
1773.843 2489727 byte part read
1777.087 90127 byte part read
1869.086 2356817 byte part read
1888.879 627596 byte part read
1913.121 578957 byte part read
1931.398 534556 byte part read
1966.998 915203 byte part read
1968.816 32096 byte part read
2094.481 2804618 byte part read
2154.295 1553352 byte part read
2164.415 323977 byte part read
2188.762 447938 byte part read
2235.080 1148231 byte part read
2261.441 621062 byte part read
2293.894 902224 byte part read
2330.604 805523 byte part read
2605.612 6580187 byte part read
2611.862 303912 byte part read
Nice large blocks, as they should be... but look at the times: 30s to transmit 900k? Strange.
Revision history for this message
|
#10 |
Also, you say "And it keeps crashing." but you haven't pasted any sign of that that I can see?
Can you show us a traceback from a crash, or provide more information about what sort of crashes you are seeing?
Revision history for this message
|
#11 |
Yup we have had multiple reports of the poor performance of bzr from other members from New Zealand and Australia. It is a consistent occurrence for me so for bzr and my other internet traffic is normal. I am not sure what it is, but it almost seems like the connection is being throttled. In terms of the crashing, I am not sure what I can give you. A bzr pull frequently pauses and restarts and when the pause goes over 10-20 mins I presume it has crashed. We are in the process of migrating from SF/CVS to LP/BZR for our project and certainly on the LP site is much better for our needs, but in comparison a pull which took only 5 mins in CVS is taking 30-50mins in bzr and it certainly is a big impact on productivity. Please note this same issue is not faced by our community members in Europe or North America.
BTW on a possibly related note, getting the latest bzr version from the ppa site as mentioned above, I noticed timeouts as well on apt-get install. Again it was almost as if the bandwidth was being throttled. The download would start at normal speed and gradually taper down to zero and completely pause. It took me multiple apt-get update commands to install bzr from the ppa repository above. Again all other apt-sources did not have this same issue and were working fine.
What more can I give you to analyze this issue apart from the bzr log?
Revision history for this message
|
#12 |
On 8 April 2010 18:52, chamindra <email address hidden> wrote:
> BTW on a possibly related note, getting the latest bzr version from the
> ppa site as mentioned above, I noticed timeouts as well on apt-get
> install. Again it was almost as if the bandwidth was being throttled.
> The download would start at normal speed and gradually taper down to
> zero and completely pause. It took me multiple apt-get update commands
> to install bzr from the ppa repository above. Again all other apt-
> sources did not have this same issue and were working fine.
This sounds a bit like a TCP-level issue, perhaps with path-mtu-discovery.
--
Martin <http://
Can you help with this problem?
Provide an answer of your own, or ask chamindra for more information if necessary.