| *** mikal has quit IRC | 01:27 | |
| *** SergeyLukjanov has quit IRC | 01:27 | |
| *** jgriffith has quit IRC | 01:27 | |
| *** zz_johnthetubagu has quit IRC | 01:27 | |
| *** sdague has quit IRC | 01:27 | |
| *** devananda has quit IRC | 01:27 | |
| *** Kiall has quit IRC | 01:27 | |
| *** anteaya has quit IRC | 01:27 | |
| *** jraim has quit IRC | 01:27 | |
| *** zaneb has quit IRC | 01:27 | |
| *** morganfainberg_Z has quit IRC | 01:27 | |
| *** dhellmann has quit IRC | 01:27 | |
| *** ttx has quit IRC | 01:27 | |
| *** dolphm has quit IRC | 01:27 | |
| *** stevebaker has quit IRC | 01:27 | |
| *** notmyname has quit IRC | 01:27 | |
| *** ChanServ has quit IRC | 01:27 | |
| *** sdague has joined #openstack-relmgr-office | 01:30 | |
| *** Kiall has joined #openstack-relmgr-office | 01:30 | |
| *** devananda has joined #openstack-relmgr-office | 01:30 | |
| *** ttx has joined #openstack-relmgr-office | 01:58 | |
| *** notmyname has joined #openstack-relmgr-office | 01:58 | |
| *** stevebaker has joined #openstack-relmgr-office | 01:58 | |
| *** dolphm has joined #openstack-relmgr-office | 01:58 | |
| *** jraim has joined #openstack-relmgr-office | 01:58 | |
| *** zaneb has joined #openstack-relmgr-office | 01:58 | |
| *** anteaya has joined #openstack-relmgr-office | 01:58 | |
| *** mikal has joined #openstack-relmgr-office | 02:03 | |
| *** morganfainberg_Z has joined #openstack-relmgr-office | 02:03 | |
| *** SergeyLukjanov has joined #openstack-relmgr-office | 02:04 | |
| *** jgriffith has joined #openstack-relmgr-office | 02:04 | |
| *** dhellmann has joined #openstack-relmgr-office | 02:06 | |
| *** zz_johnthetubagu has joined #openstack-relmgr-office | 02:08 | |
| *** ChanServ has joined #openstack-relmgr-office | 02:20 | |
| *** dickson.freenode.net sets mode: +o ChanServ | 02:20 | |
| *** Azher has joined #openstack-relmgr-office | 02:44 | |
| *** markwash has joined #openstack-relmgr-office | 04:39 | |
| *** morganfainberg_Z is now known as morganfainberg | 06:03 | |
| *** morganfainberg is now known as morganfainberg_Z | 06:21 | |
| *** morganfainberg_Z is now known as morganfainberg | 06:23 | |
| *** openstack has joined #openstack-relmgr-office | 07:41 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 09:23 | |
| *** morganfainberg is now known as morganfainberg_Z | 10:13 | |
| *** Azher has quit IRC | 11:00 | |
| *** markwash has quit IRC | 11:36 | |
| *** markwash has joined #openstack-relmgr-office | 11:40 | |
| *** mestery has joined #openstack-relmgr-office | 12:42 | |
| *** russellb has joined #openstack-relmgr-office | 13:13 | |
| ttx | russellb: o/ | 13:13 |
|---|---|---|
| ttx | russellb: so you are volunteering for handling J2 | 13:13 |
| * ttx emits an evil laugh | 13:14 | |
| russellb | ttx: o/ | 13:14 |
| russellb | indeed | 13:14 |
| russellb | hopefully i don't screw it up | 13:14 |
| ttx | it's difficult to screw it up tbh | 13:14 |
| russellb | good to hear | 13:14 |
| ttx | Let me add you to release managers group(s) first | 13:14 |
| russellb | an occasional thing like this is probably good to help force you to write down your process ;) | 13:14 |
| ttx | that gives you ultimate tagging and Launchpad rights | 13:14 |
| russellb | cool | 13:15 |
| russellb | POWER | 13:15 |
| ttx | added to lp | 13:15 |
| ttx | and to gerrit | 13:16 |
| ttx | OK, so the heart of the process is to push a 2014.2.b2 tag at a given time | 13:16 |
| russellb | k, i've pushed tags to gerrit before at least | 13:17 |
| ttx | the new lightweight process just requires the tag to be pushed on master sometimes between Tuesday and Thursday | 13:17 |
| russellb | for novaclient | 13:17 |
| ttx | I have scripts to actually do that and everything else required | 13:17 |
| ttx | so you don't even have to tag manually | 13:17 |
| russellb | nice | 13:17 |
| ttx | BUT you need to pester PTLs to get SHA they like for j2 tagging | 13:17 |
| russellb | right | 13:17 |
| russellb | so, be present, sync regularly with PTLs to identify the SHA for the tag | 13:17 |
| ttx | AND you need to make sure the juno-2 pages in LP are ... aligned | 13:17 |
| russellb | heh, "aligned" is a nice way to put it ? :) | 13:18 |
| ttx | when you tag, everything should bne completed | 13:18 |
| ttx | so everything incomplete should be pushed to j3 | 13:18 |
| ttx | so the PTL hunting is more about them accepting that something will never make it to the milestone after all | 13:18 |
| ttx | and reluctantly blessing a tag in the milestone window | 13:19 |
| russellb | OK | 13:19 |
| russellb | do you do this for just integrated? or do you help the incubated projects do a milestone release as well? | 13:19 |
| ttx | I do incubated, but with slightly less hunting, and if they miss the window, no big deal | 13:19 |
| ttx | so let's look at it chronologically | 13:19 |
| ttx | The week before (next week) | 13:19 |
| ttx | I'll still be around | 13:20 |
| ttx | but it's nice if you can follow along | 13:20 |
| russellb | yes, i'll try to be in as many of the syncs and such as i can | 13:20 |
| ttx | the goal that week is to start aligning the juno-2 pages with reality | 13:20 |
| russellb | do you have a ptl sync schedule posted? | 13:20 |
| ttx | yes | 13:20 |
| ttx | https://wiki.openstack.org/wiki/Meetings/ProjectMeeting | 13:20 |
| ttx | at the bottom | 13:20 |
| ttx | no biggie if you miss the oens in the first column | 13:21 |
| ttx | you can catch up in the logs | 13:21 |
| ttx | so ideally, they need to start removing stuff that will obviously not make it | 13:21 |
| russellb | OK, definitely would miss nova's | 13:21 |
| russellb | the rest should be doable | 13:22 |
| ttx | and continue deferring stuff so that they get to 100% Implemented 100% FixCommitted sometimes in the milestone window | 13:22 |
| ttx | so the next Tuesday, when I'm away, feel free to adjust those 1:1s or just ping the PTLs | 13:23 |
| ttx | (July 22) | 13:23 |
| russellb | got it | 13:23 |
| ttx | At that point they need to cull the list to the point where there is only a few last-minute things they would like to see in the milestone before cutting it | 13:24 |
| ttx | I generally try to limit it to in-flight stuff | 13:24 |
| ttx | stuff that is approved and all, just trying to gate | 13:24 |
| russellb | yes, any anything else, not a big deal to defer, unless it's a show stopper | 13:25 |
| ttx | right | 13:25 |
| ttx | it's really not a big deal if something slips a milestone | 13:25 |
| ttx | a milestone is just a time marker | 13:25 |
| russellb | yep, a hopefully testable time marker though :) | 13:25 |
| russellb | distros pick it up and such | 13:26 |
| ttx | but then, if something just needs a couple more hours to be included in it, it's fine to wait until Wednesday | 13:26 |
| russellb | OK | 13:26 |
| russellb | so go ahead and start tagging as early as tuesday, wednesday if necessary to get last minute things in | 13:26 |
| russellb | then, announcement and such thursday? | 13:26 |
| ttx | and Thursday for Murphy situations | 13:26 |
| russellb | "and such" <--- not sure what else is there | 13:26 |
| russellb | updating launchpad i suspect | 13:27 |
| ttx | The process is all described here: https://wiki.openstack.org/wiki/Release_Team/How_To_Release#Development_milestone_publication | 13:27 |
| ttx | let's go through that | 13:27 |
| ttx | Uses scripts from http://git.openstack.org/cgit/openstack-infra/release-tools | 13:27 |
| russellb | ooh cool | 13:27 |
| ttx | Pre-flight checks I usually run the Friday before the milestone week | 13:28 |
| ttx | though they proved mostly useless recently | 13:28 |
| ttx | ./repo_tarball_diff.sh just checks that the tarball contains the files from the repo | 13:28 |
| ttx | we had so many regressions there | 13:28 |
| ttx | that I have the habit of chcking it manually | 13:29 |
| ttx | but it's been a long time since that failed | 13:29 |
| ttx | so feel free to skip it | 13:29 |
| * russellb will skip nothing! | 13:29 | |
| * ttx needs to merge some fixes | 13:29 | |
| ttx | so usually you just get a few ignores in the diff | 13:30 |
| ttx | let me pastebin the keystone version, which is the noisiest | 13:31 |
| ttx | http://paste.ubuntu.com/7780241/ | 13:31 |
| ttx | so that's the expected output (even if that sdx is a bit weird) | 13:32 |
| ttx | (for keystone) | 13:32 |
| russellb | heh, ok | 13:32 |
| ttx | most projects will have AUTHORS and Changelog added, .gitignore and .gitreview removed | 13:33 |
| ttx | the "check bugs" part I just don't sdo anymore | 13:33 |
| russellb | that'd be a lot. | 13:33 |
| ttx | I usually checked the list for sanity, but at 10+ projects that's crazy now | 13:33 |
| ttx | I just trust people with their LP state | 13:34 |
| ttx | (I have a separate script that watches for manual FixCommitted and FixReleased changes to try to catch errors at the source) | 13:34 |
| ttx | so you can ignore that line, I should just remove it | 13:34 |
| russellb | i like milestone.sh | 13:35 |
| russellb | automation ftw | 13:35 |
| ttx | so when the PTL finally admits that the current state of master is as good as it will be for the milestone, you just move to the next step | 13:35 |
| ttx | basically, PTL gives you a SHA, you check that their juno-2 page is all Implemenetd/FixCommitted | 13:35 |
| ttx | at that point they might still have a pile of bugs targeted to juno-2 | 13:35 |
| ttx | I have a script to move them around now | 13:36 |
| russellb | do you have a script for deferring? or just manual | 13:36 |
| russellb | jinx | 13:36 |
| ttx | very recent | 13:36 |
| ttx | ./process_bugs.py glance --milestone juno-2 --settarget juno-3 | 13:36 |
| ttx | let me edit the wiki | 13:36 |
| russellb | cool | 13:37 |
| russellb | i remember that script, just not that function | 13:37 |
| ttx | it's recent | 13:37 |
| ttx | never used it | 13:37 |
| ttx | just wrote it after I manually defrred 30 | 13:38 |
| ttx | in one of those fuck that moments | 13:38 |
| ttx | ok, so that will defer bugs, but not blueprints | 13:39 |
| ttx | but usually the PTLs have a good handle on those | 13:39 |
| russellb | i used process_bugs for auto targeting fixed bugs | 13:39 |
| ttx | and there aren't that many of them | 13:39 |
| ttx | yes, milestone.sh just calls process_bugs | 13:39 |
| ttx | then there is the all-in-one | 13:39 |
| ttx | Which will tag, wait for tarball build, process FixCommitted bugs, check similarities and upload | 13:39 |
| ttx | It will stop several times during that process | 13:40 |
| ttx | The correct version is merging from https://review.openstack.org/#/c/105226/ | 13:40 |
| ttx | maybe we can read it together | 13:41 |
| russellb | sure | 13:41 |
| ttx | the script specialcases swift oslo-incubator and oslo.messaging since they need slightly different actions | 13:42 |
| ttx | let's follow the general case first | 13:42 |
| ttx | first it resolves juno-2 to 2014.2.b2 | 13:42 |
| ttx | then it pushes a tag | 13:42 |
| ttx | it will stop to ask you to sign it | 13:43 |
| ttx | well, unless you use passwordless gpg key (boo) | 13:43 |
| russellb | i dont :) | 13:43 |
| ttx | Then it waits for the tarball to be built | 13:43 |
| ttx | which is a bit flaky test polling zuul status | 13:44 |
| ttx | can take a while until we get a worker | 13:44 |
| ttx | (zuul sees tag, pushes item to pre-release pipe, then executes $PROJECT-tarball job | 13:45 |
| ttx | ) | 13:45 |
| ttx | there is a sanity check that the tarball built from the tag is similar to the last produced master.tar.gz (in case you tagged HEAD) | 13:45 |
| ttx | it used to be similar with milestone-proposed branch, now it can differ if you tag before HEAD | 13:46 |
| ttx | hence the warning there | 13:46 |
| russellb | right | 13:46 |
| ttx | then it runs process_bugs | 13:46 |
| ttx | to switch all bugs to FixReleased | 13:46 |
| ttx | there is a bit of a race there , since bugs may be fixcommitted on master if you tag before HEAD too | 13:47 |
| ttx | generally taggign HEAD is therefore preferable | 13:47 |
| russellb | or running that part as soon as you tag | 13:47 |
| russellb | and skipping it here | 13:47 |
| russellb | well, i guess if you tag an older SHA after the fact, that won't work | 13:47 |
| russellb | nm | 13:48 |
| ttx | yeah, I had a version that ran process_bugs just after tag, but sometimes the tarball goes really fast and wait_for_tarball would start after it's completed | 13:48 |
| ttx | missing the job | 13:48 |
| ttx | anyway, not a big deal if we are one bug off | 13:48 |
| ttx | So at that point you hit the painful part | 13:48 |
| ttx | which is that some Launchpad bugs will timeout | 13:49 |
| ttx | the script stop to give you a chance to updatethem manually | 13:49 |
| *** markmcclain has joined #openstack-relmgr-office | 13:50 | |
| ttx | I usually try to update them manually 3/4 times. I'm generally successful after 3 tries | 13:50 |
| ttx | If you can't update them, no big deal, just make a list of them so that we don't lose them | 13:50 |
| russellb | ack | 13:51 |
| ttx | and I can try to get them updated when I come back the week after ;) | 13:51 |
| ttx | then upload_release takes the tarball and uploads it to LP, marks milestone released etc | 13:51 |
| ttx | after that point you can no longer target bugs to j2 | 13:51 |
| ttx | unless you mark the milestine manually as active again | 13:51 |
| ttx | Note that upload_release stops after tar tvf-ing the tarball for a quick check that the nova tarball you upload looks like a nova tarballv | 13:52 |
| ttx | then checks that the MD5 between what LP has and what you have is consistent | 13:53 |
| russellb | coo; | 13:53 |
| russellb | cool, too | 13:53 |
| ttx | I should probably make those scripts all a bit less noisy | 13:53 |
| ttx | since it's been a while since that didn't work | 13:54 |
| ttx | so the error points in that script are: | 13:54 |
| ttx | - wait_for_tarball is a bit brittle | 13:54 |
| ttx | It waits a long time, but sometimes not long enough | 13:54 |
| ttx | sometimes it fails to pick up the job | 13:54 |
| ttx | so it happened a few times I had to manually run the last 3 commands by hand | 13:55 |
| ttx | similar_tarballs.sh, process_bugs.py and upload_release.py | 13:55 |
| ttx | - process_bugs hits Launchpad timeouts for some bugs | 13:56 |
| ttx | it now recovers from that, but still leaves you with a set of bugs to manually update or ignore | 13:56 |
| ttx | Everything else should work quite well | 13:56 |
| russellb | cool, looks easy enough! | 13:56 |
| russellb | and having been through the PTL rodeo helps | 13:57 |
| ttx | It was modified to handle oslo-incubator (tag only) and oslo.messaging (release in LP only) | 13:57 |
| ttx | so running ./milestone.sh juno-2 foo oslo.messaging will mark all bugs fixreleased and mark the milestone "released" without tagging or uploading anything | 13:58 |
| russellb | fancy | 13:59 |
| ttx | and running ./milestone.sh juno-2 $SHA oslo-incubator will push tag, mark all bugs fixreleased and mark the milestone "released" without uploading | 13:59 |
| ttx | I say "should" because it's relatively new | 13:59 |
| ttx | If scripts fail, don't panic, I should be around from time to time and we can look at failures | 14:00 |
| russellb | does oslo-incubator get tagged for the milestone? | 14:00 |
| ttx | yes | 14:00 |
| russellb | cool, but not the oslo libs.. | 14:00 |
| ttx | right. oslo.messaging has a Launchpad page, so we make sure that's current | 14:01 |
| russellb | makes sense | 14:01 |
| ttx | so we run process_bugs | 14:01 |
| russellb | thanks a bunch for this overview, super helpful | 14:01 |
| ttx | and we run upload_release in --nop mode which will skip the upload part | 14:01 |
| ttx | the scripts should be pretty simple to understand | 14:01 |
| russellb | yeah, this one is very readable :) | 14:01 |
| ttx | Last thing | 14:01 |
| ttx | The risk is to just forget about one project | 14:02 |
| russellb | yeah, was going to make myself a spreadsheet | 14:02 |
| ttx | that's why I use those tracking pages | 14:02 |
| ttx | print one out and just check boxes to make sure I don't overlook one | 14:02 |
| russellb | yeah nice idea | 14:02 |
| russellb | if you have one you can share, i can just use that | 14:03 |
| russellb | if it's more just for your use, no worries, i can make one | 14:03 |
| ttx | sure i'll send it to you | 14:03 |
| russellb | ok, i've got to run to baby's dr appt | 14:03 |
| ttx | shoudl be in your inbox if you want to look at it now | 14:03 |
| ttx | you want the "Others" tab | 14:04 |
| russellb | will look at it when i get back, thanks again for your time | 14:04 |
| ttx | not the "Swift" or the "RC" one | 14:04 |
| ttx | OK, run! | 14:04 |
| * russellb runs | 14:04 | |
| * ttx realizes that that page is missing oslo/oslo.messaging | 14:05 | |
| ttx | russellb: sent you a fixed version | 14:10 |
| ttx | russellb: new milestone.sh just merged, be sure to update your release-tools | 14:20 |
| *** morganfainberg_Z is now known as morganfainberg | 14:27 | |
| *** david-lyle has joined #openstack-relmgr-office | 14:53 | |
| *** markwash has quit IRC | 15:22 | |
| *** markwash has joined #openstack-relmgr-office | 16:19 | |
| russellb | udpated release tools! | 16:31 |
| *** jgriffit1 has joined #openstack-relmgr-office | 17:17 | |
| *** jgriffith has quit IRC | 17:18 | |
| *** jgriffith has joined #openstack-relmgr-office | 17:19 | |
| *** johnthetubaguy is now known as zz_johnthetubagu | 17:37 | |
| *** jgriffit1 has joined #openstack-relmgr-office | 17:39 | |
| *** jgriffith has quit IRC | 17:41 | |
| *** zaneb has quit IRC | 17:41 | |
| *** dansmith is now known as superdan | 17:42 | |
| *** russellb is now known as rustlebee | 17:42 | |
| *** markmcclain has quit IRC | 17:44 | |
| *** zaneb has joined #openstack-relmgr-office | 17:49 | |
| *** markmcclain has joined #openstack-relmgr-office | 18:35 | |
| *** markmcclain has quit IRC | 19:26 | |
| *** mestery has quit IRC | 20:12 | |
| *** markmcclain has joined #openstack-relmgr-office | 20:52 | |
| *** markmcclain1 has joined #openstack-relmgr-office | 20:53 | |
| *** markmcclain has quit IRC | 20:56 | |
| *** markmcclain1 has quit IRC | 22:02 | |
| *** morganfainberg is now known as morganfainberg_Z | 22:10 | |
| *** david-lyle has quit IRC | 22:59 | |
| *** markwash has quit IRC | 23:22 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!