Request staging import of external bug tracker

I've assigned this ticket to our sysadmins so that they can run the import on staging for you.

The zorba bugs have been imported

Take a look at https://staging.launchpad.net

Bugs #835448 to #836715

Thanks very much for the quick import. After some spot checks, I have a number of questions:

1. I would like it if the bug assignees, reporters, commenters, and so on lined up with the Launchpad user accounts of those people. Can I post-process the XML file to accomplish that? For instance, if a given <reporter> element named an existing Launchpad user, it would actually associate the bug with that user?

2. When doing the post-processing, it looks like I would have to look at <sender>, <assignee>, and <reporter> elements - is that correct?

3. It appears that none of the bugs has an assignee. Looking at the XML I sent, that also appears to be the case - but the input Sourceforge XML dump did have <assignee> elements. Any idea why the sfbugs2launchpad script didn't convert those?

4. Does search work correctly on staging? I tried searching for several bugs but either could not find them, or found so many hits that I couldn't track down the one I was looking for. I tried searching for bug titles as well as by Sourceforge bug ID - it appears that all the bug "names" are something like "Bug #836048 (zorba-3052921)" where 3052921 is the original Sourceforge bug ID, but it doesn't seem possible to be able to search for that.

5. Is there a way to get a mapping of SF bugs ids to Launchpad bug IDs? (Obviously after the final import is complete) We need to updating existing documentation and other references to SF bug URLs and need some way to map from one to the other.

6. The bugtracker for the Zorba project in staging seems to have been changed to point to Sourceforge. So there doesn't appear to be any way to see all bugs assigned to Zorba. (I also tried an empty search with the project set to "zorba", but got a timeout error.) This won't happen when the bugs are imported to production, will it? We have several outstanding bugs already in our Launchpad tracker and I certainly wouldn't want them to get lost...

Thanks again for your help.

I have been working on creating a new import with some fixes - I believe I can address my issues (1) (2) and (3) myself with a new import. I am still interested in answers to (4), (5), and (6).

However, the staging site appears to be broken, and has been for at least the last 24 hours. I have filed a bug for this:

https://bugs.launchpad.net/zorba/+bug/863715

As mentioned, this is highly important for Zorba, because until we can have all our bugs imported, we do not feel we can make any announcements about our latest release. Hopefully this will be corrected soon.

I was told that staging.launchpad is not actually broken, but just slow. I'm not sure I believe this. However, it does appear that if it's not broken, then all the bugs which were previously imported above (#835448 to #836715) are no longer there... is there an expiration timeframe on test imports?

Ah, sorry, didn't notice that this ticket was reasonably old. staging's database is replaced with a copy of production's roughly weekly. If you have a new file to import we can do it again on Monday. I can also do a local test run of it over the weekend to check that 1, 2 and 3 are fixed.

We should be able to get a mapping from old to new numbers, but as you can see we automatically add nicknames so you can just go to <https://launchpad.net/bugs/zorba-3052921>.

As for the bugtracker setting: staging's database was several weeks old until last weekend. It may be that it was still set to SourceForge at the time the DB was last updated from production. You can change it yourself at any time at <https://staging.launchpad.net/zorba/+configure-bugtracker>, assuming you own the project.

Ah! I didn't previously understand "nicknames". As an aside, it would certainly be nice if searching for a nickname worked. However, being able to go directly there via a URL is sufficient for our needs. Thanks for that. (FYI, the current sfbugs2launchpad script seems to set the nicknames to eg. "sf-3052921".)

Also thanks for explaining the database issues. I didn't realize that production was "pushed" to staging in that fashion, but it makes sense now that you explain it. Indeed, I see that the Zorba project on staging now has its own Launchpad bug tracker instead of pointing to Sourceforge, so that's good.

I will try to upload a new import XML sometime tonight, so hopefully if that goes well the final import can be done early next week. Thanks.

Alrighty, I've extended sfbugs2launchpad to handle mapping to Launchpad users, converting Group and Category entries to tags, adding URLs, and doing more extensive status mapping, among other things. Here's the result:

http://www.lambda.nu/zorba/bug-database-2.xml

Please re-run the import to staging with this data; thanks!

Any chance of a new staging import run being done today?

Bug import is running on staging right now.

I'll update again once it is complete.

-Michael

Bugs were imported. Launchpad bug numbers ranged from #856937 to #858204 and you can see them:

https://bugs.staging.launchpad.net/zorba/+bugs

-Michael

Thank you. Unfortunately, it looks like the import process does not honor Launchpad user names. This means all these bugs will not be assigned to the appropriate people, which isn't acceptable.

For instance, here's a bit of the import XML:

  <bug id="2010469" xmlns="https://launchpad.net/xmlns/2006/bugs">
      (...)
    <reporter <email address hidden>" name="matthias-brantner">Matthias Brantner</reporter>
    <assignee <email address hidden>" name="nbrinza">Nicolae Brinza</assignee>

"matthias-brantner" and "nbrinza" are the correct Launchpad user names for those users (even in the staging database), but here's how that bug got imported:

  https://bugs.staging.launchpad.net/zorba/+bug/sf-2010469

The Reporter links to the user account ~mbrantner, which doesn't exist - apparently it used the email address rather than the "name" attribute. I could live with that if all I had to do was tweak the email addresses to point to valid Launchpad usernames. However, "nbrinza" *is* Nicolae's valid Launchpad username, but the Assignee of this bug points to user account ~nbrinza-q which doesn't make any sense at all.

I am not sure how to proceed here. Can you fix your import process very quickly - like, by tomorrow? Or tell me how I might be able to format my import script so that usernames will actually be associated correctly? If not, I'm going to have to write my own script to create bugs directly on Launchpad, which is pretty ugly.

Also, less important, but I'm curious: I did include a <urls> element for each bug in my import script, with a <url> pointing back to the original bug on Sourceforge. However, this doesn't appear to show up anywhere in the bug on Launchpad. Is that feature working?

Launchpad users are looked up by email address. The "name" attribute in the XML is just a hint for what to call a new user if an account with that email address doesn't exist. Ideally you'd use their Launchpad email addresses, but otherwise the unactivated accounts can be merged after the import by the users themselves, or by an admin.

Ahh hmmm. Interesting. Ok, I should be able to provide the right email addresses for everyone; I'll try to get a modified import XML tonight. Thanks.

Assuming something gets overlooked, what is the process by which users can "merge unactivated accounts" in future?

Also, at least one of our users has multiple email addresses associated with their Launchpad account. Can I just use any of those addresses, or is there some "primary" address that needs to be used?

On a page like <https://staging.launchpad.net/~mbrantner> there is a an "Are you Matthias Brantner?" link. That leads to a merge workflow which verifies your possession of the address, and then merges the unactivated account into your existing one.

You can use any of the addresses.

Ok, great. Thanks for the answers. I'll upload a new XML file for staging by tomorrow morning, and with any luck that can be imported into production soon after that.

Alrighty, I've gathered what I believe are all the right email addresses, so here's another hopefully-final dump:

  http://www.lambda.nu/zorba/bug-database-3.xml

Please import into staging; thanks!

These have now been imported as bugs 858205 to 859472 on staging. However, since there's no easy way for us to remove bugs, the old ones are left in as well. That also caused a problem during import because nicknames were already used, so bugs up to 858649 might not be fully imported (eg. tags might not be set and nicknames will definitely not be set): after that point I cleaned up the existing nicknames so everything starting with bug 858650 should be good.

Please check how it looks like and confirm you want the import done on production.

FWIW, the command to use for LOSAs is "scripts/bug-import.py -p zorba bug-database-3.xml".

Ok, I've spot-checked a number of bugs above 858650, and it looks like everything is fine!

Please go ahead and import to production.

Thanks everyone for working with me to get through this. I will push up my changes to the sfbugs2launchpad script and propose they be merged, so perhaps the next project will have a slightly easier time of it.

Thanks!

LOSAs, please do the bug import of the http://www.lambda.nu/zorba/bug-database-3.xml on production server:

  scripts/bug-import.py -p zorba bug-database-3.xml

This has now been done.

Thanks, Tom

Import complete, looks good. Thanks again!