| *** saschpe has quit IRC | 00:00 | |
| clarkb | jeblair: before you go I will have a comment for the second change too | 00:00 |
|---|---|---|
| clarkb | just need to finish reviewing the whole change | 00:00 |
| *** nati_ueno has quit IRC | 00:01 | |
| *** MarkAtwood has quit IRC | 00:02 | |
| *** miqui has quit IRC | 00:03 | |
| fungi | jeblair: new py3k-precise images are working like a charm... http://logs.openstack.org/93/72093/4/check/gate-oslo.rootwrap-python33/35e846b/console.html | 00:03 |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/config: Increase the default pip socket timeout https://review.openstack.org/81664 | 00:03 |
| *** rcleere has joined #openstack-infra | 00:03 | |
| derekh | clarkb: ^^ thanks for the pointer to the correct place to set it | 00:04 |
| *** homeless has quit IRC | 00:06 | |
| *** moted has joined #openstack-infra | 00:06 | |
| *** saschpe has joined #openstack-infra | 00:07 | |
| clarkb | jeblair: and commented | 00:07 |
| clarkb | derekh: approved, we will need to wait for new images to see that everywhere but that should happen tonight assuming zuul merges that change quickly enough for us | 00:08 |
| derekh | clarkb: ok, thanks | 00:08 |
| clarkb | fungi: quick question about the py3k pip fix. how does pip know it should end up in the python3 python path? | 00:10 |
| *** jhesketh__ has joined #openstack-infra | 00:11 | |
| *** azneita has joined #openstack-infra | 00:11 | |
| *** sabari has quit IRC | 00:11 | |
| clarkb | fungi: I guess it ends up in /usr/local/bin/pip and will properly run under any python version | 00:11 |
| *** rhsu has quit IRC | 00:12 | |
| *** SumitNaiksatam has quit IRC | 00:15 | |
| *** signed8bit has quit IRC | 00:16 | |
| *** rwsu has quit IRC | 00:18 | |
| *** CaptTofu has joined #openstack-infra | 00:19 | |
| *** Sukhdev has quit IRC | 00:20 | |
| *** markmcclain has joined #openstack-infra | 00:21 | |
| *** sandywalsh has joined #openstack-infra | 00:22 | |
| *** rpodolyaka1 has joined #openstack-infra | 00:23 | |
| *** _Alexandra_ has joined #openstack-infra | 00:23 | |
| *** jgrimm has joined #openstack-infra | 00:23 | |
| *** mrodden has quit IRC | 00:23 | |
| *** markmcclain1 has joined #openstack-infra | 00:24 | |
| *** azneita has quit IRC | 00:24 | |
| *** markmcclain has quit IRC | 00:26 | |
| *** rpodolyaka1 has quit IRC | 00:27 | |
| openstackgerrit | A change was merged to openstack-infra/bugdaystats: Add tox.ini, setup.py and do syntax cleanup https://review.openstack.org/79800 | 00:27 |
| *** sarob has joined #openstack-infra | 00:27 | |
| *** sarob_ has joined #openstack-infra | 00:28 | |
| *** bhuvan has quit IRC | 00:28 | |
| *** _Alexandra_ has quit IRC | 00:30 | |
| *** thuc has quit IRC | 00:31 | |
| *** sarob has quit IRC | 00:31 | |
| *** thuc has joined #openstack-infra | 00:32 | |
| clarkb | jeblair: re comments on the subnode change. Do you think we should ignore that problem for now and take the subnode change as is? Or should we/you/someone rebase your change atop a config fix? | 00:34 |
| *** rcleere has quit IRC | 00:34 | |
| clarkb | I am going to mock up what I thought would fix it in my comments | 00:35 |
| *** thuc has quit IRC | 00:35 | |
| *** _Alexandra_ has joined #openstack-infra | 00:36 | |
| marun | clarkb: feel free to approve :) https://review.openstack.org/#/c/81943 | 00:39 |
| clarkb | jeblair: one more comment | 00:40 |
| clarkb | marun: is it running and passing now? | 00:40 |
| *** SumitNaiksatam has joined #openstack-infra | 00:41 | |
| clarkb | marun: if not would it make more sense to keep it as an experimental job until it is somewhat reliable? | 00:41 |
| *** esker has joined #openstack-infra | 00:41 | |
| marun | clarkb: it's running and passing now | 00:43 |
| clarkb | great /me approves | 00:43 |
| marun | clarkb: it's a tiny subset of the unit tests that I've split off. | 00:43 |
| *** rhsu has joined #openstack-infra | 00:43 | |
| marun | clarkb: the hard part was simply getting tox to execute without that permissions error, but the tests themselves are fine | 00:43 |
| *** khyati has quit IRC | 00:46 | |
| *** marun has quit IRC | 00:47 | |
| openstackgerrit | A change was merged to openstack-dev/pbr: Fixes blocking issue on Windows https://review.openstack.org/81322 | 00:49 |
| *** wchrisj has quit IRC | 00:50 | |
| *** esker has quit IRC | 00:50 | |
| *** mbacchi has quit IRC | 00:51 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add neutron functional job to check (non-voting) https://review.openstack.org/81943 | 00:53 |
| *** david-lyle has joined #openstack-infra | 00:54 | |
| *** yongli has quit IRC | 01:02 | |
| *** yongli has joined #openstack-infra | 01:04 | |
| jeblair | clarkb: i responded again, but i also want to think about some other refactoring... | 01:04 |
| *** wchrisj has joined #openstack-infra | 01:05 | |
| jeblair | clarkb: in particular, whether it's worth a bit of config refactor to be able to say "build 3 node clusters out of devstack-precise images and label them 'multinode-precise'". | 01:05 |
| jeblair | clarkb: that might be a solution to the thing you pointed out and also be more flexible (we could have fewer image types) | 01:05 |
| *** sabari has joined #openstack-infra | 01:09 | |
| openstackgerrit | A change was merged to openstack-infra/config: Increase the default pip socket timeout https://review.openstack.org/81664 | 01:09 |
| *** jamielennox has quit IRC | 01:11 | |
| *** rcleere has joined #openstack-infra | 01:14 | |
| clarkb | jeblair: after that last comment I am actually +2 on the change as is if we document the limitation | 01:18 |
| clarkb | jeblair: but do also agree that the config could be made to be more natural | 01:18 |
| *** markmcclain1 has quit IRC | 01:19 | |
| * gtt116 is away: 我很忙 | 01:20 | |
| clarkb | gtt116: please disable your away notice | 01:22 |
| *** jamielennox has joined #openstack-infra | 01:22 | |
| *** rpodolyaka1 has joined #openstack-infra | 01:23 | |
| *** rcleere has quit IRC | 01:24 | |
| *** changbl has joined #openstack-infra | 01:27 | |
| *** rpodolyaka1 has quit IRC | 01:28 | |
| *** krotscheck has quit IRC | 01:37 | |
| *** krotscheck has joined #openstack-infra | 01:38 | |
| *** mgagne has quit IRC | 01:38 | |
| *** MarkAtwood has joined #openstack-infra | 01:38 | |
| *** MarkAtwood has quit IRC | 01:39 | |
| *** thuc has joined #openstack-infra | 01:42 | |
| *** yaguang has joined #openstack-infra | 01:42 | |
| *** NobodyCam_ has joined #openstack-infra | 01:42 | |
| *** sarob_ has quit IRC | 01:42 | |
| *** sarob has joined #openstack-infra | 01:43 | |
| *** NobodyCam_ has quit IRC | 01:43 | |
| *** krotscheck has quit IRC | 01:44 | |
| *** derekh has quit IRC | 01:44 | |
| *** thuc has quit IRC | 01:46 | |
| *** sarob has quit IRC | 01:47 | |
| *** sabari has quit IRC | 01:47 | |
| *** sarob has joined #openstack-infra | 01:52 | |
| *** BadCub01 has joined #openstack-infra | 01:52 | |
| *** sarob_ has joined #openstack-infra | 01:53 | |
| *** thuc has joined #openstack-infra | 01:53 | |
| *** wchrisj has quit IRC | 01:54 | |
| *** stack555 has quit IRC | 01:54 | |
| *** gokrokve has joined #openstack-infra | 01:55 | |
| *** sarob has quit IRC | 01:56 | |
| *** thuc_ has joined #openstack-infra | 01:57 | |
| *** thuc has quit IRC | 01:57 | |
| *** thuc has joined #openstack-infra | 01:58 | |
| *** mriedem has quit IRC | 02:00 | |
| *** thuc_ has quit IRC | 02:02 | |
| *** amcrn has quit IRC | 02:02 | |
| *** yjiang has joined #openstack-infra | 02:02 | |
| *** wchrisj has joined #openstack-infra | 02:04 | |
| StevenK | Is nodepoll having issues? The graphs show a very large number of available nodes. | 02:07 |
| clarkb | StevenK yup looks dead. I am eating dinner now can look in about an hour | 02:11 |
| clarkb | jeblair fungi ^ | 02:11 |
| clarkb | mordred ^ | 02:11 |
| jeblair | clarkb: ack | 02:11 |
| *** wchrisj has quit IRC | 02:12 | |
| jeblair | if i had to guess, i'd think it might have something to do with region b | 02:12 |
| *** mriedem has joined #openstack-infra | 02:13 | |
| *** bhuvan has joined #openstack-infra | 02:17 | |
| *** jepoy_ has quit IRC | 02:17 | |
| jeblair | it's starting to look like the problem is that nodepool is not seeing zmq events from jenkinses | 02:17 |
| jeblair | so it never sees builds start or end | 02:18 |
| *** pcrews has quit IRC | 02:21 | |
| *** packet has quit IRC | 02:21 | |
| *** bhuvan has quit IRC | 02:22 | |
| *** rpodolyaka1 has joined #openstack-infra | 02:24 | |
| jeblair | those threads are just sitting waiting for something from zmq | 02:25 |
| jeblair | clarkb: so it's probably some variant of the "zmq stops sending/receiving data for no apparent reason and with no error reported" issue we've seen before | 02:26 |
| *** _Alexandra_ has quit IRC | 02:26 | |
| clarkb | was there a network burp? usually that comes along with services going away or the network dying | 02:26 |
| *** rpodolyaka1 has quit IRC | 02:28 | |
| *** jepoy has joined #openstack-infra | 02:29 | |
| jeblair | clarkb: oh, i think rax rebooted it | 02:29 |
| jeblair | i think we still have a memory leak (just a much slower one now) | 02:29 |
| jeblair | i'll stop nodepool mark all nodes for deletion, and start again | 02:31 |
| jeblair | (and hope zmq has better luck this time) | 02:31 |
| clarkb | ok | 02:31 |
| *** saschpe has quit IRC | 02:31 | |
| clarkb | jeblair you think rax rebooted the nodepool node? | 02:32 |
| jeblair | clarkb: yes, uptime and swap graphs suggest that | 02:32 |
| *** saschpe has joined #openstack-infra | 02:33 | |
| *** markwash has quit IRC | 02:33 | |
| *** markwash has joined #openstack-infra | 02:37 | |
| *** thuc_ has joined #openstack-infra | 02:39 | |
| *** thuc has quit IRC | 02:39 | |
| jeblair | okay, i added a job named 'test' to all 7 dynamic jenkins masters and made sure nodepool received updates from all of them | 02:39 |
| jeblair | nodepool is deleting all 782 nodes now | 02:42 |
| clarkb | wow | 02:43 |
| jeblair | being able to do that with "update node set state=4" is nice. :) | 02:44 |
| *** vkozhukalov has joined #openstack-infra | 02:45 | |
| clarkb | oh right since delete is decouple from the state change | 02:46 |
| clarkb | you can "delete" everything via mysql | 02:46 |
| jeblair | the first new nodes are being used now | 02:47 |
| jeblair | there were still a few jobs running, so it's likely that nodepool will yank the nodes out from under them which will cause a gate reset | 02:48 |
| jeblair | but we should lose at most one change | 02:48 |
| jeblair | StevenK: thanks! | 02:49 |
| *** mgagne has joined #openstack-infra | 02:51 | |
| *** mwagner_notHere has joined #openstack-infra | 03:00 | |
| jeblair | clarkb, fungi: i cleaned up leaked rax nodes | 03:01 |
| *** mwagner_lap has joined #openstack-infra | 03:02 | |
| jeblair | lifeless: OverLimit: Quota exceeded for ram: Requested 16384, but already used 944128 of 960000 ram (HTTP 413) (Request-ID: req-ba4579c4-ad5d-447f-a285-446a92bfe464) | 03:03 |
| jeblair | lifeless: i'm seeing that in nodepool logs, though there are only 41 instances, which should use significantly less than 944128 kb | 03:04 |
| jeblair | StevenK: ^ | 03:04 |
| *** jp_at_hp has quit IRC | 03:04 | |
| lifeless | jeblair: at a guess this nova quota bug is more serious than we thought | 03:05 |
| lifeless | | memory_mb_used | 703488 | | 03:05 |
| jeblair | :/ | 03:06 |
| jeblair | that's not urgent atm, and all the lines on the graph are going in the right directions, so i'm going to resume being afk | 03:07 |
| StevenK | jeblair: Thanks for fixing | 03:08 |
| *** mwagner_lap has quit IRC | 03:08 | |
| *** jepoy has quit IRC | 03:09 | |
| *** mriedem has quit IRC | 03:10 | |
| *** krtaylor has joined #openstack-infra | 03:11 | |
| *** jepoy has joined #openstack-infra | 03:21 | |
| *** rpodolyaka1 has joined #openstack-infra | 03:25 | |
| *** thuc_ has quit IRC | 03:29 | |
| *** rpodolyaka1 has quit IRC | 03:29 | |
| *** thuc has joined #openstack-infra | 03:30 | |
| *** zehicle_at_dell has quit IRC | 03:31 | |
| *** zehicle_at_dell has joined #openstack-infra | 03:32 | |
| *** thuc has quit IRC | 03:34 | |
| *** CaptTofu has quit IRC | 03:41 | |
| *** CaptTofu has joined #openstack-infra | 03:41 | |
| *** pmathews has joined #openstack-infra | 03:43 | |
| *** jhesketh_ has quit IRC | 03:44 | |
| *** CaptTofu has quit IRC | 03:46 | |
| *** zhiyan_ is now known as zhiyan | 03:50 | |
| *** pmathews has quit IRC | 03:56 | |
| *** pmathews has joined #openstack-infra | 03:57 | |
| *** alex-cafehackwla is now known as _alexandra_ | 04:03 | |
| *** packet has joined #openstack-infra | 04:06 | |
| *** MarkAtwood has joined #openstack-infra | 04:07 | |
| *** alff has joined #openstack-infra | 04:13 | |
| lifeless | jeblair: https://bugs.launchpad.net/tripleo/+bug/1284424 for future reference | 04:14 |
| uvirtbot | Launchpad bug 1284424 in tripleo "nova quota statistics can be incorrect" [High,Triaged] | 04:14 |
| *** thuc_ has joined #openstack-infra | 04:21 | |
| *** rpodolyaka1 has joined #openstack-infra | 04:26 | |
| *** rpodolyaka1 has quit IRC | 04:30 | |
| *** wchrisj has joined #openstack-infra | 04:30 | |
| *** khyati has joined #openstack-infra | 04:35 | |
| *** ArxCruz has quit IRC | 04:36 | |
| *** morganfainberg is now known as morganfainberg_Z | 04:39 | |
| *** sarob_ has quit IRC | 04:40 | |
| *** masayukig has quit IRC | 04:43 | |
| *** morganfainberg_Z is now known as morganfainberg | 04:44 | |
| *** jhesketh_ has joined #openstack-infra | 04:48 | |
| *** pmathews has quit IRC | 04:48 | |
| *** MarkAtwood has quit IRC | 04:49 | |
| SergeyLukjanov | jeblair, clarkb, fungi, mordred, I'm sorry, I have tons of internal meetings, so, quite inactive in IRC this week and probably will be such till the mid next week. Trying to make up reviews. | 04:49 |
| *** markwash has quit IRC | 04:53 | |
| *** jdurgin1 has quit IRC | 04:58 | |
| SergeyLukjanov | ttx, sdague, https://review.openstack.org/80687 yay, it was merged, looks like stable should start working again | 05:08 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Stop printing network diagnostics https://review.openstack.org/81404 | 05:09 |
| *** jdurgin has joined #openstack-infra | 05:11 | |
| *** mgagne has quit IRC | 05:11 | |
| *** gokrokve has quit IRC | 05:12 | |
| *** talluri has joined #openstack-infra | 05:12 | |
| *** gokrokve has joined #openstack-infra | 05:12 | |
| *** sabari has joined #openstack-infra | 05:13 | |
| *** vogxn has joined #openstack-infra | 05:15 | |
| *** gokrokve has quit IRC | 05:17 | |
| *** thuc_ has quit IRC | 05:22 | |
| *** khyati has quit IRC | 05:22 | |
| *** thuc has joined #openstack-infra | 05:23 | |
| *** rpodolyaka1 has joined #openstack-infra | 05:25 | |
| *** thuc has quit IRC | 05:27 | |
| *** marun has joined #openstack-infra | 05:29 | |
| *** gyee has quit IRC | 05:30 | |
| *** rpodolyaka1 has quit IRC | 05:30 | |
| *** rpodolyaka1 has joined #openstack-infra | 05:31 | |
| *** packet has quit IRC | 05:34 | |
| *** vogxn1 has joined #openstack-infra | 05:36 | |
| *** vogxn has quit IRC | 05:38 | |
| openstackgerrit | A change was merged to openstack-infra/config: Use correct path to puppetboard vhost. https://review.openstack.org/81929 | 05:42 |
| *** CaptTofu has joined #openstack-infra | 05:42 | |
| *** gokrokve has joined #openstack-infra | 05:43 | |
| *** CaptTofu has quit IRC | 05:46 | |
| *** rhsu has quit IRC | 05:49 | |
| *** mihgen has joined #openstack-infra | 05:50 | |
| openstackgerrit | Kazutaka Morita proposed a change to openstack-infra/config: Add swift3 into stackforge https://review.openstack.org/81991 | 05:51 |
| *** rhsu has joined #openstack-infra | 05:51 | |
| *** alff has quit IRC | 05:54 | |
| *** gokrokve has quit IRC | 05:56 | |
| *** mrda is now known as mrda_weekend | 05:58 | |
| openstackgerrit | Kazutaka Morita proposed a change to openstack-infra/config: Add swift3 into stackforge https://review.openstack.org/81991 | 06:00 |
| *** pmathews has joined #openstack-infra | 06:01 | |
| *** mihgen has quit IRC | 06:01 | |
| *** wchrisj has quit IRC | 06:02 | |
| *** gokrokve has joined #openstack-infra | 06:04 | |
| *** lttrl has quit IRC | 06:07 | |
| mordred | nibalizer: err: Could not retrieve catalog from remote server: Error 400 on SERVER: Invalid relationship: File[/puppetboard/wsgi.py] { require => User[undef] }, because User[undef] doesn't seem to be in the catalog | 06:08 |
| mordred | I have NO IDEA why it's doing that | 06:08 |
| *** gokrokve has quit IRC | 06:10 | |
| SergeyLukjanov | mordred, morning :) | 06:10 |
| SpamapS | Is the gate .. 9 hours behind? | 06:13 |
| SergeyLukjanov | SpamapS, yup | 06:13 |
| SergeyLukjanov | SpamapS, but it looks like it's going to be better | 06:14 |
| *** mihgen has joined #openstack-infra | 06:15 | |
| *** ildikov_ has quit IRC | 06:16 | |
| *** vkozhukalov has quit IRC | 06:16 | |
| *** _alexandra_ is now known as alex-gone | 06:27 | |
| *** ociuhandu has quit IRC | 06:34 | |
| *** _nadya_ has joined #openstack-infra | 06:35 | |
| *** rwsu has joined #openstack-infra | 06:35 | |
| *** _nadya_ has quit IRC | 06:45 | |
| *** mrmartin has joined #openstack-infra | 06:45 | |
| *** rpodolyaka1 has quit IRC | 06:46 | |
| *** zhiwei has joined #openstack-infra | 06:49 | |
| *** vogxn1 has quit IRC | 07:03 | |
| *** gokrokve has joined #openstack-infra | 07:06 | |
| *** lukego has joined #openstack-infra | 07:10 | |
| *** gokrokve has quit IRC | 07:10 | |
| *** jamielennox is now known as jamielennox|away | 07:12 | |
| *** mihgen has quit IRC | 07:16 | |
| *** mihgen has joined #openstack-infra | 07:18 | |
| *** mihgen has quit IRC | 07:22 | |
| *** ildikov_ has joined #openstack-infra | 07:23 | |
| *** jepoy_ has joined #openstack-infra | 07:26 | |
| *** mkoderer has joined #openstack-infra | 07:28 | |
| *** jepoy has quit IRC | 07:29 | |
| *** _nadya_ has joined #openstack-infra | 07:31 | |
| *** mflobo_ has quit IRC | 07:32 | |
| *** _nadya_ has quit IRC | 07:36 | |
| *** tnurlygayanov has quit IRC | 07:37 | |
| *** tnurlygayanov has joined #openstack-infra | 07:39 | |
| *** mrmartin has quit IRC | 07:40 | |
| nibalizer | wow | 07:41 |
| openstackgerrit | A change was merged to openstack/requirements: Removed openstacksdk from projects.txt https://review.openstack.org/81319 | 07:42 |
| *** CaptTofu has joined #openstack-infra | 07:43 | |
| *** e0ne has joined #openstack-infra | 07:43 | |
| *** mwagner_notHere has quit IRC | 07:43 | |
| *** mwagner_notHere has joined #openstack-infra | 07:44 | |
| *** yolanda has joined #openstack-infra | 07:44 | |
| *** nati_ueno has joined #openstack-infra | 07:47 | |
| *** CaptTofu has quit IRC | 07:47 | |
| *** nati_ueno has quit IRC | 07:48 | |
| *** afazekas has quit IRC | 07:49 | |
| *** nati_ueno has joined #openstack-infra | 07:50 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Update zuul's status page to bootstrap 3.1.1 https://review.openstack.org/82013 | 07:57 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add demo support for openstack's live zuul feed https://review.openstack.org/82014 | 07:57 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add in progress bars for jobs https://review.openstack.org/82015 | 07:57 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Format change headers like infra's status page https://review.openstack.org/82016 | 07:57 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add a total progress bar to each change https://review.openstack.org/82017 | 07:57 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add support to status page to collapse changesets https://review.openstack.org/82018 | 07:57 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Shorten update spinner fadeout on status page https://review.openstack.org/82019 | 07:57 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Build status page more jquery like https://review.openstack.org/81717 | 07:57 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Tidy up app.js - limit lines to 80chars https://review.openstack.org/81716 | 07:57 |
| *** vkozhukalov has joined #openstack-infra | 07:57 | |
| *** sabari has quit IRC | 07:57 | |
| *** mihgen has joined #openstack-infra | 08:01 | |
| *** Ryan_Lane has quit IRC | 08:04 | |
| *** gokrokve has joined #openstack-infra | 08:05 | |
| openstackgerrit | Kazutaka Morita proposed a change to openstack-infra/config: Add swift3 into stackforge https://review.openstack.org/81991 | 08:06 |
| *** Ryan_Lane has joined #openstack-infra | 08:06 | |
| *** flaper87|afk is now known as flaper87 | 08:07 | |
| openstackgerrit | Ivan Berezovskiy proposed a change to openstack-infra/config: Enable Gearman as default on Jenkins slaves https://review.openstack.org/73214 | 08:08 |
| *** gokrokve has quit IRC | 08:09 | |
| *** hashar has joined #openstack-infra | 08:09 | |
| *** e0ne has quit IRC | 08:11 | |
| openstackgerrit | Thomas Goirand proposed a change to openstack-infra/git-review: git-review.1 manpage fix for groff warnings https://review.openstack.org/82022 | 08:12 |
| *** rpodolyaka1 has joined #openstack-infra | 08:14 | |
| *** mwagner_notHere has quit IRC | 08:14 | |
| *** mwagner_notHere has joined #openstack-infra | 08:15 | |
| *** ominakov has joined #openstack-infra | 08:16 | |
| *** mrmartin has joined #openstack-infra | 08:17 | |
| *** e0ne has joined #openstack-infra | 08:18 | |
| *** rpodolyaka1 has quit IRC | 08:18 | |
| *** afazekas has joined #openstack-infra | 08:20 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Update Readme to setup jenkins accout https://review.openstack.org/81823 | 08:23 |
| openstackgerrit | Aleksey Chuprin proposed a change to openstack-infra/config: Add gate-magnetodb-devstack job https://review.openstack.org/82028 | 08:26 |
| *** jgallard has joined #openstack-infra | 08:26 | |
| openstackgerrit | Aleksey Chuprin proposed a change to openstack-infra/config: Add gate-magnetodb-devstack job https://review.openstack.org/82028 | 08:28 |
| *** pradnya has joined #openstack-infra | 08:31 | |
| pradnya | working on bug id: Bug #1201201 want to submit doc for same please help | 08:33 |
| uvirtbot | Launchpad bug 1201201 in tempest "Document how to install tempest without devstack" [Wishlist,Confirmed] https://launchpad.net/bugs/1201201 | 08:33 |
| *** mrmartin has quit IRC | 08:43 | |
| *** jcoufal has joined #openstack-infra | 08:43 | |
| *** mrmartin has joined #openstack-infra | 08:45 | |
| *** vogxn has joined #openstack-infra | 08:50 | |
| *** jp_at_hp has joined #openstack-infra | 08:50 | |
| *** ihrachys is now known as ihrachys|wfh|tax | 08:55 | |
| *** ildikov_ has quit IRC | 09:03 | |
| *** gokrokve has joined #openstack-infra | 09:05 | |
| *** gokrokve has quit IRC | 09:10 | |
| *** mrmartin has quit IRC | 09:12 | |
| *** derekh has joined #openstack-infra | 09:13 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Shorten update spinner fadeout on status page https://review.openstack.org/82019 | 09:15 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add support to status page to collapse changesets https://review.openstack.org/82018 | 09:15 |
| *** jcoufal has quit IRC | 09:17 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Special case django_openstack_auth in translation jobs https://review.openstack.org/82037 | 09:18 |
| *** andreaf has joined #openstack-infra | 09:18 | |
| *** jcoufal has joined #openstack-infra | 09:21 | |
| *** jcoufal has quit IRC | 09:21 | |
| *** dizquierdo has joined #openstack-infra | 09:21 | |
| *** nati_ueno has quit IRC | 09:29 | |
| *** CaptTofu has joined #openstack-infra | 09:31 | |
| *** pblaho has joined #openstack-infra | 09:32 | |
| *** jooools has joined #openstack-infra | 09:32 | |
| *** alff has joined #openstack-infra | 09:33 | |
| *** dizquierdo has quit IRC | 09:43 | |
| *** morganfainberg is now known as morganfainberg_Z | 09:48 | |
| *** ildikov_ has joined #openstack-infra | 09:48 | |
| *** nati_ueno has joined #openstack-infra | 09:49 | |
| *** pblaho has quit IRC | 09:54 | |
| *** saju_m has joined #openstack-infra | 09:55 | |
| *** jepoy_ has quit IRC | 09:55 | |
| *** rlandy has joined #openstack-infra | 09:57 | |
| *** jcoufal has joined #openstack-infra | 10:05 | |
| *** david-lyle has quit IRC | 10:05 | |
| *** gokrokve has joined #openstack-infra | 10:06 | |
| *** yjiang has quit IRC | 10:07 | |
| *** pblaho has joined #openstack-infra | 10:08 | |
| *** gokrokve has quit IRC | 10:10 | |
| *** _nadya_ has joined #openstack-infra | 10:14 | |
| *** julienvey has joined #openstack-infra | 10:14 | |
| *** rpodolyaka1 has joined #openstack-infra | 10:15 | |
| *** rpodolyaka1 has quit IRC | 10:19 | |
| *** _nadya_ has quit IRC | 10:24 | |
| *** yaguang has quit IRC | 10:25 | |
| *** vogxn has quit IRC | 10:26 | |
| *** jlibosva has joined #openstack-infra | 10:31 | |
| *** ominakov_ has joined #openstack-infra | 10:34 | |
| *** ominakov_ has quit IRC | 10:34 | |
| *** ildikov_ has quit IRC | 10:36 | |
| *** markmc has joined #openstack-infra | 10:37 | |
| mordred | w00t | 10:41 |
| mordred | the gate seems to be caught up | 10:41 |
| *** CaptTofu has quit IRC | 10:43 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: update bug to include the cleanup case https://review.openstack.org/82065 | 10:47 |
| *** zhiwei has quit IRC | 10:47 | |
| *** pblaho has quit IRC | 10:48 | |
| *** jcoufal has quit IRC | 10:49 | |
| *** dizquierdo has joined #openstack-infra | 10:52 | |
| *** hashar has quit IRC | 10:55 | |
| *** nati_ueno has quit IRC | 10:58 | |
| *** nati_ueno has joined #openstack-infra | 10:58 | |
| SergeyLukjanov | mordred, yay! | 11:01 |
| SergeyLukjanov | mordred, I've not seen such gate queue size for a long time ;) | 11:01 |
| *** nithya_g has joined #openstack-infra | 11:01 | |
| *** nati_ueno has quit IRC | 11:02 | |
| *** gokrokve has joined #openstack-infra | 11:07 | |
| *** gokrokve has quit IRC | 11:11 | |
| *** rpodolyaka1 has joined #openstack-infra | 11:16 | |
| dims | SergeyLukjanov, trouble last night on rax nodes? | 11:17 |
| *** johnthetubaguy has joined #openstack-infra | 11:19 | |
| *** rpodolyaka1 has quit IRC | 11:20 | |
| *** vkozhukalov has quit IRC | 11:21 | |
| *** saju_m has quit IRC | 11:22 | |
| *** vkozhukalov has joined #openstack-infra | 11:23 | |
| openstackgerrit | Andrea Frittoli proposed a change to openstack-infra/devstack-gate: Support Keystone V3 based gate https://review.openstack.org/79212 | 11:25 |
| *** e0ne has quit IRC | 11:30 | |
| sdague | mordred: yeh, we killed the top 2 bugs yesterday morning, and there were fixes for a couple others in the queue during the day | 11:32 |
| pradnya | Hi sdague, I am working on bug id: Bug #1201201 want to submit doc for same please help | 11:35 |
| uvirtbot | Launchpad bug 1201201 in tempest "Document how to install tempest without devstack" [Wishlist,Confirmed] https://launchpad.net/bugs/1201201 | 11:35 |
| sdague | pradnya: cool | 11:35 |
| sdague | what help do you need? | 11:35 |
| pradnya | I have created the installation document for tempest but i dont know where exactly to submit | 11:37 |
| pradnya | whether it should be in .odt format or .rst format | 11:37 |
| openstackgerrit | Andrea Frittoli proposed a change to openstack-infra/config: Add experimental keystonev3 pipeline https://review.openstack.org/79314 | 11:37 |
| pradnya | i am following the link https://wiki.openstack.org/wiki/Documentation/HowTo | 11:38 |
| *** saju_m has joined #openstack-infra | 11:38 | |
| pleia2 | good morning | 11:38 |
| pradnya | reached till the step 3 | 11:38 |
| pradnya | here i dont see any branch for tempest | 11:39 |
| pradnya | how should i proceed now | 11:39 |
| *** VINOD_ has joined #openstack-infra | 11:43 | |
| sdague | pradnya: so for this it should be in .rst | 11:43 |
| *** salv-orlando has quit IRC | 11:43 | |
| sdague | and you should submit it to the tempest tree | 11:43 |
| sdague | https://github.com/openstack/tempest/tree/master/doc/source | 11:44 |
| sdague | somewhere in there | 11:44 |
| pradnya | okay. I have .rst file with me. Should i submit it in tempest code branch itself? | 11:44 |
| pradnya | okay | 11:44 |
| pradnya | while cloning i will use the link https://github.com/openstack/tempest/tree/master/doc/source... is it correct? | 11:45 |
| sdague | https://wiki.openstack.org/wiki/Gerrit_Workflow | 11:46 |
| sdague | pradnya: no, just clone tempest | 11:46 |
| sdague | which will be https://github.com/openstack/tempest | 11:46 |
| sdague | the doc/source is the directory | 11:46 |
| sdague | the new rst file(s) should go there. I think once we see the first patch we might suggest file changes and such | 11:47 |
| sdague | that set of documents gets published here - http://docs.openstack.org/developer/tempest/ once things are committed | 11:47 |
| pradnya | okay. I will submit it in https://github.com/openstack/tempest | 11:48 |
| *** zhiwei has joined #openstack-infra | 11:53 | |
| *** jepoy has joined #openstack-infra | 11:55 | |
| *** jhesketh_ has quit IRC | 11:56 | |
| *** jhesketh__ has quit IRC | 11:56 | |
| *** jgallard has quit IRC | 11:58 | |
| *** ildikov_ has joined #openstack-infra | 12:01 | |
| *** gokrokve has joined #openstack-infra | 12:07 | |
| *** pdmars has joined #openstack-infra | 12:08 | |
| *** _nadya_ has joined #openstack-infra | 12:10 | |
| *** gokrokve has quit IRC | 12:12 | |
| *** weshay has joined #openstack-infra | 12:14 | |
| sdague | pradnya: great | 12:14 |
| *** salv-orlando has joined #openstack-infra | 12:14 | |
| pradnya | Thanks sdague!! I have submitted the .rst file | 12:15 |
| *** e0ne has joined #openstack-infra | 12:15 | |
| sdague | cool, thanks, I'll try to review later today | 12:16 |
| *** ArxCruz has joined #openstack-infra | 12:16 | |
| *** rpodolyaka1 has joined #openstack-infra | 12:16 | |
| sdague | SergeyLukjanov: the important thing about the gate being clear again is I like having the gate to myself in the morning :) | 12:16 |
| sdague | because no one else is +Aing code in my morning | 12:17 |
| *** _nadya_ has quit IRC | 12:17 | |
| *** e0ne has quit IRC | 12:17 | |
| *** e0ne has joined #openstack-infra | 12:18 | |
| openstackgerrit | Björn Michaelsen proposed a change to openstack-infra/git-review: add query option https://review.openstack.org/76197 | 12:18 |
| pradnya | Hi sdague, I am seeing multiple failures in Jenkins for https://review.openstack.org/#/c/79608/ for patchset 3 | 12:18 |
| pradnya | but in patchset 1, there was success for the same | 12:18 |
| fungi | sdague: i could fix that for you, you know ;) | 12:18 |
| pradnya | Is there any specific reason for those failures | 12:19 |
| *** e0ne has quit IRC | 12:19 | |
| *** e0ne has joined #openstack-infra | 12:19 | |
| *** alexpilotti has joined #openstack-infra | 12:20 | |
| *** aysyd has joined #openstack-infra | 12:20 | |
| *** rpodolyaka1 has quit IRC | 12:20 | |
| fungi | pradnya: if you look at the pep8 job log, that shows you have a syntax error | 12:22 |
| fungi | pradnya: it's probably the reason basically all the jobs failed | 12:22 |
| fungi | pradnya: http://logs.openstack.org/08/79608/3/check/gate-tempest-pep8/aca13e4/console.html#_2014-03-18_11_53_12_906 | 12:22 |
| *** mriedem has joined #openstack-infra | 12:23 | |
| *** e0ne has quit IRC | 12:24 | |
| *** _nadya_ has joined #openstack-infra | 12:28 | |
| *** mwagner_notHere is now known as mwagner_wfh | 12:28 | |
| *** mbacchi has joined #openstack-infra | 12:28 | |
| *** _nadya_ has quit IRC | 12:29 | |
| *** e0ne has joined #openstack-infra | 12:30 | |
| openstackgerrit | Björn Michaelsen proposed a change to openstack-infra/git-review: add query option https://review.openstack.org/76197 | 12:32 |
| *** yolanda has quit IRC | 12:32 | |
| *** e0ne_ has joined #openstack-infra | 12:33 | |
| *** _nadya_ has joined #openstack-infra | 12:33 | |
| *** e0ne has quit IRC | 12:34 | |
| * anteaya loves how sdague owns the morning gate | 12:38 | |
| sdague | my precious | 12:38 |
| SergeyLukjanov | sdague, :) | 12:38 |
| SergeyLukjanov | sdague, will we add sahara to the integrated gate someday in Juno? | 12:39 |
| fungi | SergeyLukjanov: that happens right around the same time as sahara graduates from incubation | 12:40 |
| SergeyLukjanov | fungi, already graduated :) | 12:40 |
| fungi | oh! all's the better | 12:40 |
| SergeyLukjanov | fungi, but will be part of J release, so... | 12:40 |
| *** russellb is now known as rustlebee | 12:40 | |
| fungi | will it be part of the integrated release of icehouse, or not until juno? | 12:40 |
| SergeyLukjanov | fungi, J | 12:41 |
| *** dims has quit IRC | 12:41 | |
| fungi | right, so it should probably happen right after icehouse is released, or at least shortly after the juno summit wraps up | 12:41 |
| fungi | but i would say as soon as people want to review/approve the patches for it after icehouse releases is fair game | 12:41 |
| anteaya | fungi: is there an etherpad yet for new project creation for today? | 12:42 |
| sdague | SergeyLukjanov: so once we're back on open master | 12:43 |
| fungi | anteaya: no, but we didn't do one last week either because there were only maybe four to work through | 12:43 |
| sdague | and the stable trees are all cut | 12:43 |
| *** thomasem has joined #openstack-infra | 12:43 | |
| anteaya | fungi: so doing one today, or no? | 12:43 |
| pleia2 | fungi: if you have some time, can you look at how bugdaystats is configured on old-wiki? I need the cron job and config.js | 12:44 |
| SergeyLukjanov | sdague, oh, sure, we could enable sahara by default in d-g for master | 12:44 |
| sdague | and there is actually a real heat race bug today! | 12:44 |
| fungi | anteaya: if the list is half a dozen or more, then coordinating it through the etherpad might help, but last time we basically just merged them all at the same time, fixed any merge conflicts, then once they were in and appearing on all the git servers as empty repos we ran manage-projects once and that was pretty much it | 12:44 |
| *** CaptTofu has joined #openstack-infra | 12:44 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed a change to openstack-infra/config: Initial openstackdroid commit for Stackforge, with review changes https://review.openstack.org/81234 | 12:44 |
| sdague | SergeyLukjanov: yeh, once the stable/icehouse branches are all out and we have that selection logic in there | 12:45 |
| fungi | did some quick spot checks afterward to make sure it was all sane... not much to it | 12:45 |
| pleia2 | anteaya: browsing reviews, should we bring the UTC comment from fungi to the -dev list for discussion? https://review.openstack.org/#/c/77376/ | 12:45 |
| fungi | pleia2: sure, just a few minutes and i'll take a look | 12:45 |
| pleia2 | fungi: thank you | 12:45 |
| SergeyLukjanov | sdague, yup | 12:45 |
| pradnya | fungi:./tempest/api/compute/servers/test_list_servers_negative.py:167:1: F821 undefined name 'attr'. if you see the test case you will find that attr is being used | 12:45 |
| * fungi made a utc comment? | 12:45 | |
| pleia2 | fungi: re: requiring it for 3rd party testing | 12:45 |
| *** ArxCruz has quit IRC | 12:46 | |
| anteaya | well fungi makes a good point | 12:47 |
| anteaya | here is my problem, if the goal if bringing it to the ml is to achieve a reflection of community consensus, how do we know when we have that? | 12:47 |
| anteaya | part of the problem with the target audience is their lack of engagement with the community | 12:48 |
| anteaya | if I felt they would contribute to the discussion I would be all for the discussion | 12:48 |
| *** dims has joined #openstack-infra | 12:48 | |
| fungi | anteaya: those who participate are those who get to be part of the consensus. those who do not, have opted out of the decision-making process | 12:49 |
| anteaya | my concern is that we would hinge this patch merging on a certain level of engagement from 3rd party accounts and be stuck with crickets | 12:49 |
| *** CaptTofu has quit IRC | 12:49 | |
| *** pblaho has joined #openstack-infra | 12:49 | |
| anteaya | if we bring it to the ml with a time frame, I am fine with that | 12:49 |
| rcarrillocruz | hi fungi, hopefully https://review.openstack.org/#/c/81234/7 now is good, I added suggested change to zuul layout.yaml | 12:49 |
| fungi | anteaya: my concern was more in getting feedback from the reviewers and developers who are having to look at these logs | 12:49 |
| anteaya | then once the time elapses, action can take place | 12:49 |
| pleia2 | I'd say just give them a week | 12:49 |
| sdague | so 3 requirements changes I think we should land | 12:50 |
| fungi | to find out if it's something they're needing, rather than just something we thought sounded like a good idea | 12:50 |
| sdague | https://review.openstack.org/#/c/79817/ | 12:50 |
| pleia2 | +A next friday if all is well | 12:50 |
| anteaya | ah, okay reviewers and devs, yes I can work with that | 12:50 |
| sdague | https://review.openstack.org/#/c/80849/ | 12:50 |
| fungi | rcarrillocruz: great! i'll take a look in just a moment--thanks | 12:50 |
| *** mfer has joined #openstack-infra | 12:50 | |
| anteaya | fungi: oh I needed this | 12:51 |
| *** ArxCruz has joined #openstack-infra | 12:51 | |
| *** mwagner_wfh is now known as mwagner_wfh_bbl | 12:51 | |
| anteaya | when working with mikal to assess whether someone's system was fixed based on what time the latest tests ran | 12:51 |
| anteaya | from a japanese server on japanese time | 12:51 |
| anteaya | the ryu account | 12:52 |
| pleia2 | heh | 12:52 |
| anteaya | in the middle of my night | 12:52 |
| sdague | and https://review.openstack.org/#/c/80851/ | 12:52 |
| anteaya | it was a less than efficient conversation | 12:52 |
| sdague | SergeyLukjanov: you are +2 there right? want to take a look | 12:52 |
| *** ihrachys|wfh|tax is now known as ihrachys|wfh | 12:52 | |
| anteaya | but i can take it to the ml and share my tale at the same time | 12:52 |
| pleia2 | thanks anteaya | 12:52 |
| anteaya | and give it a it a week to brew | 12:52 |
| anteaya | fungi: and regarding new projects, let me know how I can help | 12:53 |
| *** hashar has joined #openstack-infra | 12:53 | |
| *** pblaho has quit IRC | 12:54 | |
| *** pblaho has joined #openstack-infra | 12:54 | |
| *** saju_m has quit IRC | 12:57 | |
| *** rfolco has joined #openstack-infra | 12:57 | |
| *** adalbas has joined #openstack-infra | 13:02 | |
| *** dprince has joined #openstack-infra | 13:02 | |
| *** CaptTofu has joined #openstack-infra | 13:03 | |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/reviewstats: Add nova-specs to nova.json https://review.openstack.org/82084 | 13:03 |
| *** dripton has quit IRC | 13:03 | |
| *** dripton has joined #openstack-infra | 13:03 | |
| *** salv-orlando has quit IRC | 13:05 | |
| *** saju_m has joined #openstack-infra | 13:06 | |
| *** yolanda has joined #openstack-infra | 13:06 | |
| SergeyLukjanov | sdague, re https://review.openstack.org/#/c/80851/? | 13:08 |
| sdague | SergeyLukjanov: yep | 13:08 |
| *** gokrokve has joined #openstack-infra | 13:08 | |
| SergeyLukjanov | sdague, +A for https://review.openstack.org/#/c/80849/ and +2 for https://review.openstack.org/#/c/80851/ | 13:10 |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Add nova-specs to nova.json https://review.openstack.org/82084 | 13:11 |
| *** julim has joined #openstack-infra | 13:12 | |
| *** dcramer_ has joined #openstack-infra | 13:12 | |
| *** gokrokve has quit IRC | 13:13 | |
| *** pradnya has quit IRC | 13:14 | |
| *** sdake has quit IRC | 13:15 | |
| *** rpodolyaka1 has joined #openstack-infra | 13:17 | |
| *** _nadya_ has quit IRC | 13:18 | |
| annegentle | morning! Can someone point me to the default stackforge doc build configuration stuff? I need to make it so that it uses a different sphinx template so pre-incubating projects don't inadvertently use the OpenStack template/logo. | 13:19 |
| *** rpodolyaka1 has quit IRC | 13:20 | |
| *** hashar has quit IRC | 13:21 | |
| fungi | annegentle: i don't think we have anything like that documented... is the problem that projects are using oslosphinx for their documentation templates? | 13:21 |
| *** dizquierdo has quit IRC | 13:22 | |
| *** hashar has joined #openstack-infra | 13:23 | |
| sdague | fungi: probably, we probably need to hit up doug for a stylesheet that drops the logo | 13:24 |
| *** nati_ueno has joined #openstack-infra | 13:25 | |
| *** bknudson has joined #openstack-infra | 13:25 | |
| fungi | sdague: agreed. i assume that's something which can be carried in oslosphinx or as an add-on and we don't need an entirely separate oslosphinx fork to make that happen? | 13:25 |
| *** VINOD_ has quit IRC | 13:27 | |
| sdague | yeh, I would think it could be a flag | 13:28 |
| *** jasondotstar has joined #openstack-infra | 13:31 | |
| *** jgallard has joined #openstack-infra | 13:33 | |
| *** jgrimm has quit IRC | 13:33 | |
| *** dansmith is now known as superdan | 13:34 | |
| sdague | fungi: did deletes get faster? | 13:35 |
| mordred | a flag or a thing we can set in the env | 13:36 |
| fungi | sdague: deletes get faster when we're not churning as much | 13:38 |
| fungi | sdague: the providers start ignoring or responding much more slowly to delete calls when we're building a deleting more instances | 13:38 |
| annegentle | fungi: pretty much it seems like that's the root cause of the problem | 13:38 |
| annegentle | dhellmann_: fungi: anyone have ideas on how to fix? | 13:38 |
| sdague | mordred: sphinx has a config file already, so something in the config seems fine | 13:38 |
| fungi | annegentle: i suggest a bug against oslo asking for a barebones or maybe stackforge-specific template with an option | 13:38 |
| annegentle | fungi: is olso.sphinx something they can opt out of? | 13:38 |
| sdague | annegentle: yes | 13:39 |
| fungi | annegentle: it's something they opt in to by adding it to their project as a dependency and setting up their repository to support it | 13:39 |
| sdague | oslo.sphinx is an optional add on to make your sphinx output take on the same format at the rest | 13:39 |
| annegentle | sdague: ok I'll write up an email so that projects understand what we're asking -- it happens for both docbook and sphinx built docs, where the template choice is a poor one | 13:39 |
| fungi | annegentle: the openstack-dev/cookiecutter repository may be a contributing factor in this, so maybe we should also have a stackforge cookiecutter fork (as i understand it, cookiecutter doesn't support multiple project templates in one repo) | 13:40 |
| fungi | and our current cookiecutter template is probably arranged with the assumption of official openstack projects which will default to using oslosphinx (remember it's not called oslo.sphinx any longer, for several weeks now) | 13:41 |
| annegentle | fungi: oh right oslosphinx | 13:42 |
| mordred | sdague: I mean, if it's in the env, we can enforce branding in the build env | 13:42 |
| annegentle | fungi: so aren't most people using the cookiecutter preincubating? | 13:42 |
| mordred | if it's ina config file, we have to leave it up to the project to brand itself | 13:42 |
| annegentle | ttx: can you help with moderating through the designated sections request to the -tc list? | 13:42 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: update bug to include the cleanup case https://review.openstack.org/82065 | 13:43 |
| fungi | annegentle: not necessarily. we have roughly equal numbers of official and stackforge projects now | 13:43 |
| annegentle | ttx: told me too many recipients. or let me know if there's a PTL list | 13:43 |
| annegentle | fungi: okay | 13:43 |
| ttx | annegentle: done | 13:43 |
| *** vhoward has joined #openstack-infra | 13:43 | |
| anteaya | fungi: http://lists.openstack.org/pipermail/openstack-dev/2014-March/030692.html | 13:43 |
| ttx | annegentle: no ptl list | 13:43 |
| * fungi gets current numbers, out of curiosity... | 13:43 | |
| anteaya | pleia2: ^^ | 13:43 |
| annegentle | fungi: for docbook, we have a "built for openstack" logo template, but even so they have to sign a contrat, and currently there's no visibility into who has signed a contract so it's difficult to know | 13:43 |
| annegentle | ttx: thanks | 13:44 |
| *** alff has quit IRC | 13:44 | |
| fungi | annegentle: makes sense. i definitely wouldn't try to tackle that issue with software alone anyway. probably better to have a stackforge-specific template or just something barebones with no logos/branding | 13:45 |
| fungi | white label, as they say | 13:45 |
| annegentle | fungi: my thinking is that whatever readthedocs uses is good | 13:45 |
| annegentle | fungi: yep | 13:45 |
| sdague | mordred: sure, but couldn't we just make the docs build job fail projects that us the openstack logo but aren't in the allowed white list | 13:46 |
| sdague | then it would be on the projects to fix themselves | 13:46 |
| mordred | hrm | 13:46 |
| sdague | and if people built the docs locally | 13:46 |
| sdague | they would still be right | 13:46 |
| annegentle | sdague: mordred: that seems fine, don't want anyone to ragequit over it (just one more hurdle might be their breaking point) | 13:46 |
| mordred | well - I think what I was thinking was that there would be a normal oslosphinx that works and is unbranded everywhere | 13:47 |
| sdague | annegentle: assuming an olsosphinx change where fixing it is a 1 line patch to a project | 13:47 |
| mordred | and that we'd inject a logo in our env | 13:47 |
| sdague | sure, that's fine as well | 13:47 |
| mordred | that way, third-party publication would also not be branded unless they really went out of their way | 13:47 |
| fungi | well, we *might* want *some* people to ragequit. we've got quite a lot of minimally-involved stackforge projects, and no doubt quite a few abandoned ones we don't know have gone silent | 13:47 |
| mordred | so the 'officialness' of the stuff published on our site would be slightly more apparent | 13:47 |
| mordred | but ... | 13:48 |
| mordred | I don't feel strongly - I think we shoudl do whatever makes the most sense | 13:48 |
| annegentle | fungi: heh | 13:48 |
| fungi | current gerrit project count is 130 openstack.* and 124 stackforge | 13:49 |
| fungi | so we're still flying just at 51% official | 13:50 |
| fungi | for some definition of official | 13:50 |
| * fungi needs to try not to use that word since it's vague and misleading | 13:51 | |
| *** sandywalsh has quit IRC | 13:51 | |
| *** julim_ has joined #openstack-infra | 13:54 | |
| *** dizquierdo has joined #openstack-infra | 13:54 | |
| annegentle | fungi: thanks for the numbers | 13:54 |
| *** nati_ueno has quit IRC | 13:55 | |
| *** salv-orlando has joined #openstack-infra | 13:55 | |
| *** nati_ueno has joined #openstack-infra | 13:55 | |
| *** julim has quit IRC | 13:56 | |
| *** julim_ has quit IRC | 13:56 | |
| *** julim has joined #openstack-infra | 13:57 | |
| *** markmcclain has joined #openstack-infra | 13:59 | |
| *** wchrisj has joined #openstack-infra | 14:00 | |
| *** nati_ueno has quit IRC | 14:00 | |
| sdague | mordred: I disagree, we should do whatever makes the least sense | 14:00 |
| fungi | sdague: don't we always? | 14:01 |
| sdague | only on fridays | 14:01 |
| fungi | friday! | 14:01 |
| *** markmcclain1 has joined #openstack-infra | 14:02 | |
| *** ildikov_ has quit IRC | 14:02 | |
| *** markmcclain has quit IRC | 14:03 | |
| *** sandywalsh has joined #openstack-infra | 14:04 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/os-loganalyze: make info level not bold https://review.openstack.org/82096 | 14:06 |
| *** malini_afk is now known as malini | 14:06 | |
| *** markmcclain1 has quit IRC | 14:07 | |
| sdague | fungi: that one is for mtreinish who was complaining with me cleaning up the tempest log to drop a bunch of bad debug calls it was now to hard to read a see of bold | 14:07 |
| *** lukego has quit IRC | 14:08 | |
| *** lukego has joined #openstack-infra | 14:09 | |
| *** gokrokve has joined #openstack-infra | 14:09 | |
| *** lukego has quit IRC | 14:09 | |
| *** thuc_ has joined #openstack-infra | 14:09 | |
| *** markmcclain has joined #openstack-infra | 14:10 | |
| *** thuc_ has quit IRC | 14:11 | |
| sdague | fungi: did grizzly release? | 14:11 |
| openstackgerrit | A change was merged to openstack-infra/os-loganalyze: make info level not bold https://review.openstack.org/82096 | 14:11 |
| *** thuc_ has joined #openstack-infra | 14:11 | |
| fungi | sdague: yes, should be safe to start the great ripping | 14:13 |
| fungi | i was going to give it until monday just to be sure there were no point release emergencies which might need correcting first | 14:13 |
| * anteaya hands out party hats | 14:13 | |
| *** gokrokve has quit IRC | 14:13 | |
| sdague | cool | 14:17 |
| *** rpodolyaka1 has joined #openstack-infra | 14:18 | |
| mordred | sdague: ++ | 14:20 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: [WIP] Authorization models added https://review.openstack.org/82101 | 14:20 |
| mtreinish | sdague: thanks | 14:21 |
| *** rpodolyaka1 has quit IRC | 14:22 | |
| *** pcrews has joined #openstack-infra | 14:22 | |
| *** markmcclain has quit IRC | 14:25 | |
| mordred | fungi: I cannot begin to tell you how excited I am for that | 14:26 |
| *** jgrimm has joined #openstack-infra | 14:29 | |
| anteaya | mordred: grizzly release or auth models added to storyboard? | 14:30 |
| *** mihgen has quit IRC | 14:30 | |
| *** dhellmann_ is now known as dhellmann | 14:33 | |
| dhellmann | I'm just catching up on the backlog after a day off. Do we want a flag in oslosphinx? If so, I like the idea of setting it in the build job. | 14:33 |
| dhellmann | fungi: it seems from your email that the pip issue is spreading to other jobs? is the right fix there to revert the pip requirement in pbr? | 14:34 |
| fungi | dhellmann: i don't know whether that's the "right" fix (was hoping others would chime in with reasons we might make things using pbr require newer pip than you can get from distro packages on current ubuntu lts) | 14:35 |
| fungi | dhellmann: just that the current unfortunate consequence is that you need to pip install a newer pip now to use anything which relies on pbr | 14:36 |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Move bugdaystats to to status.openstack.org https://review.openstack.org/82106 | 14:37 |
| fungi | but i assume the >=1.4 is there for an actual reason | 14:37 |
| fungi | mordred: ^ ? | 14:37 |
| dstufft | system pip is so bad :( | 14:38 |
| dstufft | I don't even think ubuntu LTS pip supports TLS | 14:38 |
| dhellmann | fungi: I don't know the history there, unfortunately. :-/ | 14:38 |
| *** david-lyle has joined #openstack-infra | 14:38 | |
| fungi | dstufft: probably not. it's 1.3.1 | 14:38 |
| dstufft | oh it's 1.3.1? | 14:39 |
| dstufft | I hought it uses 1.0 still | 14:39 |
| dstufft | 1.3.1 does TLS | 14:39 |
| dstufft | still does random scraping of shit on PyPI | 14:39 |
| dstufft | but at least it's not compeltely bad then | 14:39 |
| *** jnoller has joined #openstack-infra | 14:39 | |
| dstufft | I just took a look at pbr | 14:39 |
| dstufft | it's setting PIP_USE_WHEEL to true | 14:40 |
| dstufft | wheel is only in pip 1.4+ | 14:40 |
| dstufft | although it's an envvar so it wouldn't fail if pip was < 1.4, you jsut wouldn't get wheels | 14:40 |
| fungi | dstufft: oh, actually it is 1.0 on precise. i forgot we were ending up with 1.3.1 on those machines because we had a backport of the python3-pip package from quantal | 14:40 |
| fungi | er, from raring (13.04) | 14:40 |
| dstufft | fungi: there we go :V | 14:40 |
| fungi | which is using 1.3.1 | 14:40 |
| dstufft | I tried towrite a patch to add tls to 1.0 | 14:41 |
| dstufft | but after like 3 hours I got annoyed and gave up | 14:41 |
| dstufft | :/ | 14:41 |
| fungi | i can understand that | 14:41 |
| *** bnemec is now known as beekneemech | 14:41 | |
| *** mrodden has joined #openstack-infra | 14:42 | |
| *** atiwari has joined #openstack-infra | 14:43 | |
| dstufft | fungi: yea I just grepped pbr, the only thing I can find that would need 1.4 is wheel support | 14:45 |
| *** thedodd has joined #openstack-infra | 14:45 | |
| mordred | system pip is silly | 14:47 |
| mordred | makes no sense | 14:47 |
| *** ildikov_ has joined #openstack-infra | 14:47 | |
| mordred | if you're willing to use pip, use pip to use pip | 14:47 |
| *** sdake has joined #openstack-infra | 14:47 | |
| *** sdake has quit IRC | 14:47 | |
| *** sdake has joined #openstack-infra | 14:47 | |
| *** krotscheck has joined #openstack-infra | 14:49 | |
| mordred | fungi: it's 1.4 because we synced with openstack global requirements | 14:49 |
| mordred | which is only a problem if you think you're going to use a system pip | 14:50 |
| mordred | but, you know, if you're doing that you're just asking for pain | 14:50 |
| dhellmann | mordred: do we actually need that version of pip all the time, though? and does every consumer of pbr need it? | 14:51 |
| mordred | dhellmann: possibly not - I could easily be convinced to downgrade the requirement | 14:51 |
| mordred | otoh - when installing from pip, it shoudl not be a problem | 14:52 |
| mordred | no? | 14:52 |
| *** zhiyan is now known as zhiyan_ | 14:52 | |
| dhellmann | Is it safe to have pip upgrade itself while installing an install-time dependency of something being installed with pip? | 14:52 |
| *** rlandy_ has joined #openstack-infra | 14:52 | |
| dhellmann | the fact that I even have to ask that question makes me think no | 14:52 |
| mordred | hrm. nod | 14:52 |
| dhellmann | so it seems safer to say pbr requires pip, but to not have a specific version required | 14:53 |
| *** dstanek has quit IRC | 14:53 | |
| dhellmann | but does pbr actually require pip? | 14:53 |
| mordred | yes | 14:53 |
| dhellmann | ok | 14:53 |
| mordred | pbr does not operate without pip | 14:53 |
| mordred | well, I say that ... | 14:53 |
| ttx | jeblair: please check my facts on DepFreeze at http://paste.openstack.org/show/74009/ | 14:53 |
| mordred | it is possible to disable it | 14:53 |
| mordred | but normal usage requires it | 14:53 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Another fix for upstream_translation_horizon.sh https://review.openstack.org/82114 | 14:53 |
| ttx | jeblair: will send if you confirm sanity | 14:53 |
| mordred | (it doesn't require it so taht you can do things with not-pip for distro packaging purposes) | 14:54 |
| dhellmann | is there some way to test this without releasing an 0.8.0 of pbr with a different pip requirement? | 14:54 |
| *** freyes has joined #openstack-infra | 14:54 | |
| *** e0ne_ has quit IRC | 14:54 | |
| mordred | test that it still works with 1.0? | 14:54 |
| *** rlandy has quit IRC | 14:55 | |
| dhellmann | 1.0 of what? | 14:55 |
| *** rlandy_ has quit IRC | 14:55 | |
| mordred | pip? | 14:55 |
| dhellmann | I want to make it so the py33 tests for oslo don't fail so I can merge stuff. | 14:55 |
| dhellmann | Can I test the pbr dependency on pip for that without releasing a new pbr? | 14:55 |
| mordred | they should not be failing due to pip version | 14:55 |
| mordred | what am I missing? | 14:55 |
| dhellmann | context, let me find some links | 14:55 |
| mordred | fungi: oh - are we using apt-based pip on the py3k nodes? | 14:56 |
| mordred | fungi: we should stop doing that. it's a bug in our system | 14:56 |
| dhellmann | mordred: oslo.rootwrap failure log: http://logs.openstack.org/93/72093/3/check/gate-oslo.rootwrap-python33/ebbf14e/console.html | 14:56 |
| mordred | we do not use system pip anywhere else in our infrastructure | 14:56 |
| dhellmann | mordred: fungi's change to strip system pip from py33 nodes: https://review.openstack.org/#/c/75213/5 | 14:57 |
| dhellmann | ah, that merged | 14:57 |
| mordred | oh! | 14:57 |
| mordred | wait | 14:57 |
| mordred | dhellmann: you can fix this in oslo.rootwrap | 14:57 |
| dhellmann | I was out yesterday, so the last I saw was: http://paste.openstack.org/show/73832/ | 14:58 |
| dhellmann | mordred: ? | 14:58 |
| ttx | ? | 14:58 |
| *** lcostantino has joined #openstack-infra | 14:58 | |
| ttx | say rootwrap three times and I appear | 14:58 |
| anteaya | is that what it takes | 14:58 |
| mordred | dhellmann: https://review.openstack.org/82117 | 14:59 |
| *** Ajaeger has joined #openstack-infra | 15:00 | |
| ttx | hmm, looks like requirements-core could use a bit of a cleanup | 15:00 |
| *** gmoro has quit IRC | 15:00 | |
| Ajaeger | infra team, here's on more post job fix - a single character change that I'd love to see in: https://review.openstack.org/#/c/82114/ Thanks! | 15:01 |
| anteaya | ttx yay | 15:01 |
| * ttx generates wall of shame | 15:01 | |
| anteaya | ttx you were the person to talk to about requirements-core clean up! | 15:01 |
| anteaya | been bugging everybody else | 15:02 |
| sdague | ttx: agree | 15:02 |
| * anteaya puts rootwrap * 3 aside for another day | 15:02 | |
| dhellmann | mordred: we don't need netaddr, but otherwise that looks ok | 15:02 |
| fungi | mordred: we were also seeing the same in git0-review's python33 unit tests, fwiw | 15:02 |
| dhellmann | mordred: ttx or I can fix up the patch | 15:02 |
| mordred | dhellmann: ok - I'm about to jump on a plane - feel free to do whatevet to it | 15:02 |
| ttx | sdague: let me generate good numbers | 15:02 |
| dhellmann | mordred: I'll handle it, thanks | 15:02 |
| *** jcoufal has joined #openstack-infra | 15:03 | |
| Ajaeger | thanks, fungi! | 15:03 |
| mordred | fungi: same patch as the one above to oslo.rootwrap is applicable | 15:03 |
| fungi | mordred: excellent--thanks! | 15:04 |
| jeblair | ttx: that msg lgtm | 15:04 |
| fungi | mordred: sitepackages = True shouldn't strictly be necessary though, right? | 15:04 |
| ttx | jeblair: coolthx | 15:04 |
| mordred | oh! jeez | 15:04 |
| mordred | dhellmann: ^^ !!!! | 15:05 |
| dhellmann | bah | 15:05 |
| dhellmann | fixing | 15:05 |
| mordred | dhellmann: I copied the cantrip from nova - sorry | 15:05 |
| mordred | my bad | 15:05 |
| mordred | fungi: THANK YOU | 15:05 |
| *** saju_m has quit IRC | 15:05 | |
| fungi | mordred: also, we ripped out all the locale envvars elsewhere | 15:05 |
| dhellmann | so which part of this is actually fixing the problem? | 15:05 |
| fungi | since they were causing problems for people who didn't have those locales installed | 15:05 |
| mordred | k. good to know | 15:05 |
| mordred | dhellmann: one sec - new patch coming | 15:06 |
| dhellmann | mordred: ok | 15:06 |
| mordred | dhellmann: there you go | 15:06 |
| ttx | sdague: http://paste.ubuntu.com/7131023/ -- looks like dprince daviey gabrielhurley and joeheck did not contribute one review in last 90 days. ewindisch contributed one. | 15:07 |
| fungi | mordred: and you had agreed previously that there was no actual reason to be overriding the user's LANG/LANGUAGE/LC_ALL in the tox.ini anyway, that it was carried over from some very early paranoia where we weren't sure whether output in other locales would break testr, if memory serves | 15:07 |
| dhellmann | mordred: we've had issues with skipdist and usedevelop for oslo libs in namespace packages when they use other oslo libs (not the case here, just a note) | 15:07 |
| *** jcoufal has quit IRC | 15:07 | |
| ttx | sdague: on the other hand I would formally add Sergey and mordred (they currently are in by way of the infra-core convenience), and consider dmlrr | 15:08 |
| anteaya | ttx I had thought vish was on requirements-core | 15:08 |
| ttx | https://review.openstack.org/#/admin/groups/131,members | 15:08 |
| dstufft | dhellmann: ~namespace packages~ | 15:09 |
| sdague | ttx: I'm also a little concerned with folks that have > 90% approval rate honestly | 15:09 |
| sdague | because I think in most cases they aren't looking that deeply into the requests | 15:09 |
| * anteaya tries to remember the vish context, obviously wasn't requirements-core | 15:09 | |
| *** gokrokve has joined #openstack-infra | 15:10 | |
| mordred | btw - the stackalytics version of the russelb stats calculate "core team" by seeing if the person has a +A or +2 vote recorded in the sample period | 15:10 |
| ttx | sdague: agree | 15:10 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Coverage fix https://review.openstack.org/82121 | 15:10 |
| sdague | mordred: how do they handle stable? | 15:11 |
| dhellmann | dstufft: afaict, the issue we've had is that we end up with 2 copies of a lib installed at the same time (one editable, one not). ISTM that non-namespace packages would have failed in less predictable ways (the lib would have worked, but may have loaded the wrong version of the code). | 15:11 |
| rustlebee | mordred: fine usually, except my stats count stable reviews | 15:11 |
| *** VINOD_ has joined #openstack-infra | 15:11 | |
| *** gmoro has joined #openstack-infra | 15:12 | |
| ttx | sdague: not formally being requirements-core, I don't think it's appropriate for me to send the cleanup email -- would you go and suggest the removals ? | 15:12 |
| sdague | yeh, it will be later today | 15:13 |
| dhellmann | ttx: I thought we all agreed to make you requirements ptl? | 15:13 |
| sdague | yeh, I think we did agree to that :) | 15:13 |
| sdague | we should just do that | 15:13 |
| sdague | then you can send the email | 15:13 |
| * dhellmann goes to add ttx to the group | 15:13 | |
| ttx | hah, but then it should be added to the relmgt program :) | 15:13 |
| sdague | because, honestly, I feel like I've been delivering too much bad news to people this week already... so would rather have someone else do it | 15:14 |
| dhellmann | ttx: you are now in the requirements group | 15:14 |
| ttx | dhellmann: I suggested it originally (attach repo to relmgt program) but I think mordred shot it down | 15:14 |
| ttx | dhellmann: i was, by way of Release managers inclusion | 15:14 |
| dhellmann | ttx: I don't know why it would need to change programs | 15:14 |
| sdague | ttx: that was mostly about ATC voting for release manager | 15:14 |
| *** gokrokve has quit IRC | 15:14 | |
| anteaya | ttx you don't even have to do anything, other than cull the group | 15:14 |
| ttx | sdague: well, we struggle in that program anyway to get enough voters | 15:14 |
| anteaya | I can live with that | 15:14 |
| *** gokrokve has joined #openstack-infra | 15:15 | |
| *** CaptTofu has quit IRC | 15:15 | |
| ttx | anteaya: btw, need to talk to you about that | 15:15 |
| *** pmathews has quit IRC | 15:15 | |
| * anteaya stands by | 15:15 | |
| dhellmann | ttx: I missed the change list. I can send email to the -dev list, if you can summarize. | 15:15 |
| ttx | anteaya: last cycle we invited PTLs to vote for the relmgr PTL | 15:15 |
| *** CaptTofu has joined #openstack-infra | 15:15 | |
| anteaya | cool | 15:15 |
| ttx | because the only repo under relmgt isn't that full | 15:15 |
| ttx | I think only Sergey and me contributed to it :) | 15:15 |
| anteaya | ah | 15:15 |
| anteaya | small electorate | 15:16 |
| ttx | so we added stable-maint, vmt members | 15:16 |
| ttx | and PTLs | 15:16 |
| anteaya | I don't even know what repos are under relmgt | 15:16 |
| ttx | (stable-maint and VMT being under the program, makes sense, PTL was just an addition) | 15:16 |
| * anteaya checks goveranance/projects.yaml | 15:16 | |
| ttx | release-tools :) | 15:16 |
| anteaya | ah | 15:16 |
| *** mgagne has joined #openstack-infra | 15:16 | |
| ttx | anteaya: last time we didn't really have an election so we didn't need to actually apply that offer | 15:17 |
| ttx | anteaya: just mentioning it in case we need to have one | 15:17 |
| anteaya | right | 15:17 |
| anteaya | okay if someone else wants to do what you do, I will round up some electorate for the election | 15:17 |
| ttx | dhellmann: it's not in any program right now. It's in the orphan group, like openstack/openstack | 15:18 |
| dhellmann | ttx: ok, I thought it was sort of part of infra, but I'm fine with adding it to the release program if that makes things easier | 15:18 |
| *** rpodolyaka1 has joined #openstack-infra | 15:18 | |
| ttx | dhellmann: I guess it boils down to whether I'm PTL for requirements because I was personally volunteered, or because we think it belongs to Release Management | 15:19 |
| ttx | dhellmann: in all fairness I think I got volunteered | 15:19 |
| dhellmann | well, it needs a better home than None, and I think release management makes sense | 15:19 |
| dhellmann | yeah, you did :-) | 15:19 |
| anteaya | I agree | 15:19 |
| dhellmann | ok, I'll propose a change to the governance repo so we can talk about it at the appropriate meeting(s) | 15:20 |
| anteaya | both that ttx should be ptl and that requirements makes sense in release management | 15:20 |
| sdague | I think we should probably actually resolve that once and for all as a TC thing | 15:20 |
| ttx | dhellmann: feel free to suggest that attachment as a governance change. My last try having been denied I probably shouldn't repropose myself. | 15:20 |
| *** CaptTofu has quit IRC | 15:20 | |
| dhellmann | and then when that's decided you can propose the membership cleanup, ttx | 15:20 |
| sdague | because I think the PTLs of the programs should be the people that vote for the release manager | 15:20 |
| anteaya | would increase the release management electorate as a by product | 15:20 |
| sdague | not the atc for some small repo somewhere | 15:20 |
| *** rpodolyaka1 has quit IRC | 15:20 | |
| dhellmann | sdague: I agree, though I think the stable maint team (or leader?) makes sense too | 15:21 |
| *** zhiwei has left #openstack-infra | 15:21 | |
| ttx | sdague: well, you also arguably want the people that would have to obey to that person (i.e. VMT/Stable team) to agree with that person | 15:21 |
| anteaya | sdague: can we get that official like as a separate motion to the tc? | 15:21 |
| *** packet has joined #openstack-infra | 15:21 | |
| anteaya | I don't care who the electorate is, just that I get the correct people | 15:21 |
| sdague | anteaya: sure | 15:21 |
| anteaya | thanks | 15:21 |
| dhellmann | yeah, I didn't see moving the requirements repo under release as a way to increase the voter roles | 15:22 |
| ttx | sdague: "release manager" doesn't actually have any right on other projects. | 15:22 |
| dhellmann | so much as a way to find it a good home and leader | 15:22 |
| anteaya | dhellmann: fair enough | 15:22 |
| ttx | sdague: so why would they NEED to vote ? I'm fine with them voting because we have small electorate in that team, not because I think the PTLs should all decide | 15:22 |
| ttx | My power doesn't come from my title. It comes from my influence as a person. I encourage people to make the right decisions... I can't veto PTLs for example. | 15:23 |
| sdague | ttx: if that's the case it should be all the ATCs | 15:23 |
| ttx | sdague: why? | 15:24 |
| ttx | the relmgt PTL is a team leader elected by team members, like any other ptl | 15:24 |
| sdague | ttx: I think if we have this as a designated title, it probably should have an election | 15:24 |
| sdague | ttx: who are those people? | 15:24 |
| ttx | sdague: it's no longer a title, it's a team now | 15:25 |
| ttx | team members ? The release team, the VMT team and the stable maint team | 15:25 |
| anteaya | #link http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml#n279 | 15:25 |
| ttx | arguably you can consider PTLs to be part of the "release management team" | 15:25 |
| sdague | anyway, got to actually do some code atm :) | 15:25 |
| sdague | I'm not sure I actually care that much about the eventual answer | 15:26 |
| ttx | sdague: heh yes, never engage in an openstack governance discussion with me :) | 15:26 |
| sdague | I just think we should have it written down | 15:26 |
| anteaya | sdague: yes | 15:26 |
| sdague | because it's a super vague thing right now | 15:26 |
| pleia2 | heading out to long lunch with the family | 15:26 |
| sdague | not that anyone else wants your job | 15:26 |
| sdague | but we should be explicit about how someone would take it from you :) | 15:26 |
| ttx | sdague: it is written @ https://wiki.openstack.org/wiki/Release_Cycle_Management fwiw | 15:26 |
| anteaya | so that should I need to send ballots to the electorate for relmgt ptl election, it is open and transparent including to me, whom I am sending ballots to | 15:26 |
| ttx | under 'team members' | 15:27 |
| ttx | sdague: If it makes sense to add PTLs to the electorate then it should probably appear there too. | 15:27 |
| * ttx will clarify | 15:28 | |
| sdague | sure, I guess the issue is who the team is isn't completely a function of ATC on certain repos | 15:28 |
| sdague | or +2 on those repos | 15:28 |
| ttx | sdague: right, which is why I elaborate on "team members" there | 15:29 |
| sdague | that's the part which probably needs clarity | 15:29 |
| *** pmathews has joined #openstack-infra | 15:29 | |
| * ttx clarifies | 15:29 | |
| *** david-lyle has quit IRC | 15:29 | |
| *** e0ne has joined #openstack-infra | 15:29 | |
| fungi | sdague: that goes potentially for other horizontal effort programs, it's just that we've sort of avoided that question elsewhere because they also tend to have repositories with some automation they use which most of their team members contribute patches to | 15:29 |
| sdague | fungi: agreed | 15:30 |
| sdague | I'm actually the anti pattern here, so I feel good about being the test case | 15:30 |
| sdague | because I'm now on the release management team | 15:30 |
| anteaya | ha ha ha | 15:30 |
| sdague | and I can give good reasons why that probably makes sense | 15:30 |
| sdague | but none of them are about me being +2 on some repo | 15:30 |
| sdague | so we should at least be clear about it | 15:30 |
| anteaya | ++ | 15:31 |
| sdague | and I think it's completely ok that it's not tied to a specific +2 role | 15:31 |
| ttx | Edited team members under https://wiki.openstack.org/wiki/Release_Cycle_Management to match what we had for last election | 15:32 |
| *** jasondotstar has quit IRC | 15:33 | |
| ttx | anteaya: ^ that should serve as electorate if need arises | 15:34 |
| anteaya | thanks | 15:34 |
| anteaya | I just question the use of 'temporary' | 15:34 |
| ttx | ok removed it | 15:35 |
| anteaya | k | 15:35 |
| fungi | all things in the universe are temporary | 15:35 |
| anteaya | that is true | 15:35 |
| anteaya | and I take that as a given | 15:35 |
| fungi | but especially these sorts of things ;) | 15:35 |
| anteaya | otherwise noone would take the job | 15:35 |
| fungi | so yes, it's redundant to call it temporary, agreed | 15:36 |
| anteaya | so, I am happy in as much that should someone what to do what ttx does and give him a break, I know how to conduct mysefl | 15:36 |
| anteaya | but not spell apparently | 15:36 |
| ttx | anteaya: aagin, PTL of release management program != most visible member of the release team | 15:37 |
| ttx | I don't want to discourage people to replace me in either position | 15:37 |
| *** julienvey has quit IRC | 15:37 | |
| sdague | ttx: I expect you want to encourage it :) | 15:38 |
| sdague | because even a week of trying to hold a few things down while you were out, I don't want that job ;) | 15:38 |
| anteaya | ha ha ha | 15:38 |
| anteaya | no sale there | 15:39 |
| *** krotscheck has quit IRC | 15:39 | |
| ttx | also I want to further explore distributing the "release management" work across a team, hence me stopping usage of the "release manager" title | 15:40 |
| fungi | it's probably worth thinking again about how to subdivide and delegate those activities even further, as has already been done by splitting out stable release management | 15:40 |
| * anteaya wonders who the most visible member of the release team is | 15:40 | |
| fungi | er, what ttx just said | 15:40 |
| fungi | also, not it | 15:40 |
| * fungi ducks behind the crowd | 15:41 | |
| *** mrodden has quit IRC | 15:41 | |
| anteaya | yeah release management doesn't consist of a lot of people fighting to take the stage | 15:42 |
| jeblair | sdague: i imagine it was hard to do because you didn't have thierry's checklist and beer | 15:42 |
| jeblair | sdague: i think both are required :) | 15:42 |
| ttx | jeblair: the checklist is just for milestones. For FFEs I use memcache and wine | 15:43 |
| jeblair | ha! | 15:43 |
| ttx | jeblair: sent the DepFreeze email | 15:43 |
| *** jepoy has quit IRC | 15:44 | |
| ttx | dhellmann: feel free to bounce on it to propose a requirements-core cleanup | 15:44 |
| *** jepoy has joined #openstack-infra | 15:44 | |
| *** e0ne_ has joined #openstack-infra | 15:44 | |
| ttx | dhellmann: 90-day reviews analysis at http://paste.ubuntu.com/7131023/ | 15:44 |
| *** talluri has quit IRC | 15:45 | |
| dhellmann | ttx: should we do that before or after elections? (I'm not sure if we decided it will make a difference in the voter rolls) | 15:45 |
| ttx | dhellmann: ideally we would do it before DepFreeze kicks in next week :) | 15:45 |
| fungi | JayF: if you're around, just wanted to bring your attention to my comment on https://review.openstack.org/79088 (if that can be addressed, it should be able to merge today) | 15:46 |
| ttx | i don't want people completely out of touch to blindly ignore the freeze and +2 stuff | 15:46 |
| dhellmann | ttx: ok, do I need a list of names to remove, or should I just propose that people reconsider their participation? | 15:46 |
| * dhellmann missed that part of the discussion with sdague | 15:46 | |
| ttx | dhellmann: I think it's safe to remove people that have been contributing 0 reviews in the past 3 months | 15:47 |
| *** jepoy_ has joined #openstack-infra | 15:47 | |
| dhellmann | ttx: makes sense | 15:47 |
| ttx | dhellmann: then remind the rest of the depfreeze rules I just posted, and tell them we'll clean up again soon ? | 15:47 |
| *** e0ne has quit IRC | 15:47 | |
| *** caleb_ has joined #openstack-infra | 15:48 | |
| *** nati_ueno has joined #openstack-infra | 15:48 | |
| *** jepoy has quit IRC | 15:49 | |
| fungi | also, the removal announcement should encourage people who *do* have an interest in checking and vetting requirements changes to start participating in reviews there | 15:49 |
| fungi | because i think probably still not enough people actually do | 15:49 |
| dhellmann | fungi: +1 | 15:49 |
| fungi | it's one of those tragedy-of-the-commons repos | 15:49 |
| *** nati_ueno has quit IRC | 15:51 | |
| *** rossella_s has quit IRC | 15:52 | |
| *** mrodden has joined #openstack-infra | 15:53 | |
| sdague | I also think we need to be a little careful on participation == membership | 15:53 |
| *** rossella_s has joined #openstack-infra | 15:53 | |
| sdague | because there are folks that are +2ing a bunch of stuff there and not really looking that deep into things | 15:53 |
| sdague | like when sqla 0.9 went through, and turned out wasn't working at all | 15:54 |
| fungi | i think the same goes there as for any project. people should be looking at job results and logs, checking things out for themselves, discussing relevancy, et cetera | 15:57 |
| fungi | just because a patch is a one-liner doesn't mean the change is a simple one | 15:57 |
| *** caleb_ has quit IRC | 15:57 | |
| *** malini has left #openstack-infra | 15:57 | |
| sdague | yeh, especially with the implications here | 15:58 |
| sdague | speaking of which - https://review.openstack.org/#/c/79817/ I still think we should approve | 15:59 |
| sdague | I think the nova unit test issues were cleared up | 15:59 |
| sdague | now off to meet mtreinish for lunch | 15:59 |
| *** Ajaeger has quit IRC | 16:00 | |
| *** thuc_ has quit IRC | 16:00 | |
| *** mgagne has quit IRC | 16:01 | |
| *** thuc has joined #openstack-infra | 16:01 | |
| fungi | sdague: why <=9.99 instead of <10? elsewhere in the requirements list we're well past worrying about pip versions which pull in prerelease versions without explicitly requesting them | 16:01 |
| *** esker has joined #openstack-infra | 16:01 | |
| *** gyee has joined #openstack-infra | 16:01 | |
| *** caleb_ has joined #openstack-infra | 16:01 | |
| *** david-lyle has joined #openstack-infra | 16:02 | |
| *** pblaho has quit IRC | 16:02 | |
| fungi | the <=8.99 got added that way because of a misunderstanding/mistake, from what i recall in the discussion on the ml at the time. but not having an actual interest in what openstack's declared dependencies are, i generally refrain from reviewing these except when they're emergency fixes | 16:03 |
| ttx | jeblair: just ran the repo/programs.yaml map and it yielded https://etherpad.openstack.org/p/OhgrljHVnc | 16:03 |
| ttx | jeblair: looks like you have two missing projects under infra | 16:03 |
| ttx | jeblair: what shall we do with gantt ? | 16:03 |
| ttx | sdague: what's openstack/qa-specs ? | 16:04 |
| fungi | ttx: same as nova-specs | 16:04 |
| *** mgagne has joined #openstack-infra | 16:04 | |
| *** yjiang has joined #openstack-infra | 16:04 | |
| fungi | ttx: but for the qa program | 16:04 |
| ttx | sdague: ok so maybe add it to QA's section in programs.yaml | 16:04 |
| fungi | ttx: apparently multiple programs now want to do the blueprints-in-gerrit thing | 16:05 |
| fungi | came up while you were still vacationing | 16:05 |
| *** thuc has quit IRC | 16:05 | |
| ttx | ack | 16:05 |
| *** lcostantino has quit IRC | 16:05 | |
| *** dkliban is now known as dkliban_afk | 16:05 | |
| * anteaya wonders who will be ptl of the wat program | 16:06 | |
| *** jooools has quit IRC | 16:06 | |
| fungi | ttx: and the gantt discussion similarly. it would be good to have the tc decide what does happen when an official program decides to remove itself from governance of a project (or when incubation of a project is aborted) | 16:06 |
| fungi | ttx: that would help us decide what to do with projects like gantt, melange, openstack-chef, et cetera | 16:07 |
| ttx | fungi: well, then it can't live under the openstack/ namespace anymore | 16:07 |
| ttx | fungi: original decision for melange was to move under some attic/ namespace | 16:07 |
| *** bogdando has quit IRC | 16:07 | |
| fungi | ttx: right, having some actual policy describing that, so that the infra team can decide on what process we follow to make that happen (such as scheduling immediate project renames or whatever) | 16:08 |
| *** packet has quit IRC | 16:09 | |
| *** yjiang has quit IRC | 16:10 | |
| *** markmcclain has joined #openstack-infra | 16:10 | |
| *** andreaf has quit IRC | 16:10 | |
| fungi | ttx: part of the concern being that "move to stackforge" isn't a great blanket solution, since stackforge isn't necessarily a designated dumping ground for failed/abandoned projects either | 16:11 |
| mordred | sdague: I don't think it's handling stable right now: http://stackalytics.com/report/contribution/requirements/30 | 16:11 |
| fungi | we could for example declare that actual abandoned projects just stay where they are and become orphans because they're no longer maintained | 16:11 |
| jeblair | fungi: i think our poll during the meeting ended up with most people thinking we shoud move gantt to stackforge, yeah? | 16:12 |
| fungi | jeblair: right | 16:12 |
| mordred | ++ | 16:12 |
| fungi | i am concerned though that we're talking about moving a project even though the interested developers plan to gut the current codebase and replace it anyway | 16:12 |
| ttx | jeblair: maybe pool it with the other stackforge/attic moves in the etherpad ? | 16:12 |
| jeblair | i think the only complicating factor, in my mind, is that it's not clear that the nova team won't want 'openstack/gantt' in the future | 16:13 |
| jeblair | which is incompatible with having openstack/gantt | 16:13 |
| anteaya | ttx do we have an etherpad of stackforge/attic moves? | 16:13 |
| ttx | anteaya: https://etherpad.openstack.org/p/OhgrljHVnc | 16:13 |
| jeblair | (we can't actually support the same project name in two namespaces; that's just an illusion) | 16:13 |
| jeblair | (i mean, not without lots of work) | 16:13 |
| anteaya | ttx ah the etherpad you are working on | 16:14 |
| jeblair | but, i'm okay with stackforging it and just dealing with that later if it becomes an issue | 16:14 |
| *** markmcclain has quit IRC | 16:15 | |
| JayF | fungi: I saw the comment; will either remove the topic branches or create a clean fork | 16:17 |
| fungi | i'm okay with whatever everyone else wants to do, though i still personally favor gantt being orphaned/abandoned in place and the interested developers starting a new project in stackforge under a different name | 16:17 |
| fungi | JayF: cool--thanks! | 16:17 |
| fungi | since they're not planning to keep the current codebase anyway, it didn't sound like | 16:17 |
| anteaya | why does sandbox need to move to stackforge? | 16:18 |
| *** krotscheck has joined #openstack-infra | 16:19 | |
| *** rpodolyaka1 has joined #openstack-infra | 16:19 | |
| *** reed has joined #openstack-infra | 16:19 | |
| anteaya | one of the selling features, I felt, when trying to sell 3rd party systems to use it was that it was an openstack project | 16:19 |
| anteaya | regarding perception for newcomers | 16:19 |
| fungi | i question that as well. it's a test repository for people to experiment with, but that doesn't mean it necessarily belongs to the stackforge namespace (similar to the "stackforge is not a dumping ground for projects nobody cares about" argument) | 16:20 |
| anteaya | hard enough to get them to use it as it is | 16:20 |
| anteaya | I don't think that target audience will listen to an explaination of why they should use a stackforge repo | 16:21 |
| *** rpodolyaka1 has quit IRC | 16:23 | |
| *** comstud is now known as bearhands | 16:26 | |
| *** rcleere has joined #openstack-infra | 16:27 | |
| *** afazekas has quit IRC | 16:27 | |
| *** freyes has quit IRC | 16:29 | |
| ttx | etherpad is the new irc | 16:33 |
| anteaya | so that is where we can go when we net split | 16:34 |
| jeblair | i'm not sure how to interject there | 16:35 |
| jeblair | so i'll just say, yeah, i agree, we should consider deleting projects to be impossible | 16:35 |
| *** rcleere has quit IRC | 16:35 | |
| jeblair | (while technically, of course it's possible, it would be _extremely_ difficult, and we'll end up with a gerrit that returns lots of strange errors all the time | 16:35 |
| *** anteaya is now known as cyan | 16:36 | |
| jeblair | and i think it's undesirable because i don't think we should lose that history) | 16:36 |
| cyan | my new nick | 16:36 |
| *** talluri has joined #openstack-infra | 16:36 | |
| *** cyan is now known as Guest20064 | 16:36 | |
| Guest20064 | boo | 16:36 |
| Guest20064 | I wanted to be cyan | 16:36 |
| *** Guest20064 is now known as anteaya-who-was- | 16:37 | |
| *** anteaya-who-was- is now known as anteaya | 16:37 | |
| anteaya | apparently I have a character limit on nicks | 16:37 |
| *** rhsu has quit IRC | 16:37 | |
| *** gokrokve has quit IRC | 16:38 | |
| *** gokrokve has joined #openstack-infra | 16:38 | |
| *** krotscheck has quit IRC | 16:38 | |
| *** jogo is now known as flashgordon | 16:39 | |
| *** krotscheck_ has joined #openstack-infra | 16:39 | |
| *** JoshNang has joined #openstack-infra | 16:40 | |
| fungi | to move the conversation back into irc, so if we want to do a batch cleanup and rename/move a bunch of these all at once i'm not opposed, i would just rather not set a precedent of yet another reason to rename projects, and further increase the frequency with which we have to take our infrastructure offline for them | 16:42 |
| openstackgerrit | A change was merged to openstack/requirements: Add pexpect to global_requirements https://review.openstack.org/80849 | 16:42 |
| *** gokrokve has quit IRC | 16:42 | |
| JayF | fungi: https://gist.github.com/jayofdoom/b4b2604f1b31c611bee4 branches deleted | 16:42 |
| fungi | JayF: great! switching my -1 to a +2 then | 16:43 |
| JayF | woohoo | 16:43 |
| *** krotscheck_ has quit IRC | 16:43 | |
| JayF | I have a kinda strange question: do people who have already contributed to the agent automatically become Openstack ATCs when it's merged? I think there's one or two people on our team who haven't pushed anything upstream to Ironic yet but have code already in the agent. | 16:44 |
| ttx | fungi: i don't think that would happen too often, and those could always wait for a planned rename window | 16:44 |
| ttx | fungi: as long as they get picked up one day and not forgotten (like what just happened with those) | 16:44 |
| *** jlibosva has quit IRC | 16:44 | |
| ttx | See commit message @ http://git.openstack.org/cgit/openstack/governance/commit/reference/programs.yaml?id=7044c177dcc02d44321660db8b909483dad68be3 | 16:46 |
| fungi | JayF: effectively yes, because of the way we calculate atc status based on the commit history of the projects in the program, and don't have any real way to flag only commits added on specific projects after import | 16:46 |
| *** alff has joined #openstack-infra | 16:47 | |
| *** jasondotstar has joined #openstack-infra | 16:48 | |
| ttx | where those projects should go was already discussed and decided by the TC. Now i'm fine with waiting until another important rename happens before running those | 16:48 |
| fungi | ttx: i guess that decision is official then. my fault for not noticing the commit message and petitioning the tc members to reconsider | 16:48 |
| *** prad has joined #openstack-infra | 16:49 | |
| ttx | fungi: if it's impossible/undesirable from a technical, infra perspective it's a valid objection | 16:49 |
| ttx | but we are past "stackforge or not" | 16:49 |
| fungi | ttx: agreed | 16:49 |
| ttx | in all cases those are never "urgent". they can certainly wait until the infra is restarted for another reason | 16:50 |
| anteaya | oh, my fault too for not noticing it was already decided | 16:51 |
| ttx | so they should never add downtime | 16:51 |
| fungi | ttx: as a separate (but related) topic, should we be then considering openstack namespace projects as conveying atc status even when not part of an official program? (requirements in particular, since the current script does not consider it) | 16:52 |
| *** SumitNaiksatam has quit IRC | 16:53 | |
| ttx | fungi: if we can keep them in sync, I'd say yes | 16:53 |
| fungi | though i guess the governance repo might also end up with changes merged which were submitted by non-tc members, potentially people who are not otherwise atcs at all | 16:53 |
| fungi | openstack/openstack not so much, since it's just a list of git submodules | 16:54 |
| ttx | fungi: I don't think we can use that as a source until it's cleaned up *and* kept in sync somehow, though | 16:54 |
| ttx | so that's probably a discussion for the future | 16:55 |
| ttx | not a big deal anyway since I'm pretty sure the overlap currently is 100% | 16:55 |
| fungi | i would rather see those official-but-orphan repositories reflected in openstack/governance:eference/programs.yaml in some way anyway | 16:55 |
| ttx | fungi: agreed | 16:55 |
| *** sabari has joined #openstack-infra | 16:56 | |
| fungi | makes the audit activity simpler | 16:56 |
| ttx | That would make my audit script happy. | 16:56 |
| * fungi nods | 16:56 | |
| anteaya | a clean morning gate for sdague and an happy audit script for ttx | 16:57 |
| clarkb | morning | 16:58 |
| anteaya | morning clarkb | 16:59 |
| *** VINOD_ has quit IRC | 17:00 | |
| *** ominakov has quit IRC | 17:01 | |
| *** khyati has joined #openstack-infra | 17:01 | |
| *** e0ne_ has quit IRC | 17:03 | |
| *** ildikov_ has quit IRC | 17:04 | |
| *** rhsu has joined #openstack-infra | 17:06 | |
| *** gokrokve has joined #openstack-infra | 17:06 | |
| *** david_lyle_ has joined #openstack-infra | 17:07 | |
| *** rdwrer has quit IRC | 17:07 | |
| *** rdwrer has joined #openstack-infra | 17:07 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add 'labels' as a configuration primitive https://review.openstack.org/82147 | 17:08 |
| *** david-lyle has quit IRC | 17:09 | |
| *** gokrokve_ has joined #openstack-infra | 17:09 | |
| jeblair | clarkb: just in time to see the results of what you started! ^ :) | 17:09 |
| clarkb | uh oh :P | 17:10 |
| *** gokrokve has quit IRC | 17:11 | |
| jeblair | oh, right pep8 | 17:11 |
| *** caleb_ has quit IRC | 17:12 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add 'labels' as a configuration primitive https://review.openstack.org/82147 | 17:12 |
| *** gokrokve has joined #openstack-infra | 17:12 | |
| *** jgallard has quit IRC | 17:13 | |
| *** gokrokve_ has quit IRC | 17:14 | |
| jeblair | clarkb, fungi: i opened a ticket with rax on the openstackjenkins account about glance not working (it returns http 500 errors) | 17:15 |
| jeblair | did that last night; no update yet | 17:15 |
| clarkb | jeblair: oh, I should see what I can do to get glance in hp cloud region b | 17:16 |
| clarkb | lifeless says that is a private beta thing that we can get in on | 17:16 |
| jeblair | clarkb: cool, we should at least start poking at it | 17:16 |
| clarkb | jeblair: will try to do that after I review that nodepool change and catch up on the gantt related scrollback | 17:16 |
| clarkb | jeblair: yup | 17:16 |
| *** markmc has quit IRC | 17:19 | |
| *** rpodolyaka1 has joined #openstack-infra | 17:19 | |
| *** adalbas has quit IRC | 17:22 | |
| ttx | oops looks like I never hit Send on that DepFreeze email | 17:23 |
| *** dizquierdo has quit IRC | 17:23 | |
| *** vkozhukalov has quit IRC | 17:23 | |
| *** SpamapS is now known as Spam | 17:24 | |
| annegentle | ttx: woo slushy! | 17:24 |
| *** rpodolyaka1 has quit IRC | 17:24 | |
| *** adalbas has joined #openstack-infra | 17:24 | |
| *** Spam is now known as Maps | 17:24 | |
| ttx | A last Glance to zuul status before calling it a week... | 17:25 |
| *** melwitt has joined #openstack-infra | 17:25 | |
| clarkb | fungi: I agree with your statement about not having another reason to restart stuff | 17:25 |
| ttx | oh my, it's almost empty | 17:25 |
| *** rdwrer has quit IRC | 17:25 | |
| *** Maps is now known as CannedMeat | 17:25 | |
| fungi | ttx: it's amazing what getting most of the major nondeterministic test failures will do to gating throughput | 17:26 |
| *** SumitNaiksatam has joined #openstack-infra | 17:26 | |
| fungi | failures fixed | 17:26 |
| ttx | indeed | 17:26 |
| *** jnoller has quit IRC | 17:26 | |
| anteaya | fungi clarkb I just looked through the open patches for /config and don't see any create new project type patches | 17:26 |
| *** thedodd has quit IRC | 17:26 | |
| anteaya | are they there and I missed them? | 17:26 |
| fungi | anteaya: https://review.openstack.org/#/q/status:open+project:openstack-infra/config+branch:master+topic:new-project,n,z | 17:26 |
| ttx | nice peak on https://github.com/openstack/openstack/graphs/commit-activity | 17:27 |
| ttx | and the day is not over yet | 17:27 |
| fungi | anteaya: a couple of them are in good shape to merge, the other couple we could possibly just fix for the submitters if we want | 17:27 |
| ttx | ok, have a good weekend everyone | 17:27 |
| anteaya | ttx see ya | 17:28 |
| jeblair | ttx: you too! | 17:28 |
| fungi | have fun ttx | 17:28 |
| clarkb | ttx: thats the goal :) | 17:28 |
| clarkb | though I have a chickenpox vaccine waiting for me ... | 17:28 |
| clarkb | jeblair: afte rreading the nodepool doc change I like this a lot | 17:28 |
| clarkb | I haven't read the code yet but from a config standpoint this is great :) | 17:29 |
| jeblair | clarkb: yaay! | 17:29 |
| openstackgerrit | A change was merged to openstack-infra/config: Use unbound https://review.openstack.org/80092 | 17:29 |
| jeblair | clarkb: unfortunately, the code _looks_ like s/image/label/, but it's not. only about half the things change. | 17:29 |
| jeblair | clarkb: however, i have tested this with the fake server setup and it seems to work | 17:30 |
| clarkb | awesome | 17:30 |
| jeblair | clarkb, fungi, mordred, SergeyLukjanov: 17:32 < openstackgerrit> A change was merged to openstack-infra/config: Use unbound https://review.openstack.org/80092 | 17:30 |
| clarkb | dun dun dun | 17:31 |
| jeblair | clarkb, fungi, mordred, SergeyLukjanov: that could break absolutely everything -- nodepool images and servers alike. | 17:31 |
| clarkb | jeblair: on that note apparently google DNS was hijacked via BGP to do a DDoS | 17:31 |
| jeblair | clarkb: that's lovely! i'm so excited we're going to use them on hpcloud! | 17:31 |
| jeblair | speaking of dns... | 17:32 |
| jeblair | kiall: did you end up doing anything with that irc bot? | 17:32 |
| *** caleb_ has joined #openstack-infra | 17:32 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Add internal noop job https://review.openstack.org/81311 | 17:32 |
| *** rdwrer has joined #openstack-infra | 17:33 | |
| jeblair | we should restart zuul this weekend to get that ^ | 17:33 |
| *** caleb_` has joined #openstack-infra | 17:35 | |
| *** caleb_ has quit IRC | 17:38 | |
| *** rdwrer has quit IRC | 17:38 | |
| *** saju_m has joined #openstack-infra | 17:38 | |
| jeblair | clarkb: should the username and private key vars be moved from the provider-image section to label? | 17:39 |
| jeblair | clarkb: i think they represent post-configuration state, and i think we expect that to be the same across all providers | 17:40 |
| *** rdwrer has joined #openstack-infra | 17:40 | |
| jeblair | but then, it should be the same for every image, even if that image is used for multiple labels | 17:40 |
| jeblair | so.. maybe there's no clear fit | 17:40 |
| *** NobodyCam is now known as rustler | 17:41 | |
| *** mkoderer has quit IRC | 17:41 | |
| jeblair | i think maybe just leave it there for now | 17:41 |
| *** rcleere has joined #openstack-infra | 17:43 | |
| anteaya | fungi: yes, looks like openstackdroid and ironic-python-agent are good to go | 17:43 |
| clarkb | jeblair: ya leave it for now | 17:43 |
| clarkb | I am good with that being image data | 17:43 |
| *** ildikov_ has joined #openstack-infra | 17:43 | |
| anteaya | fungi: how would I fix the swift3 patch? | 17:44 |
| anteaya | the ceph one is missing an acl file | 17:44 |
| * anteaya comments | 17:44 | |
| jeblair | puppetmaster is running its cron | 17:45 |
| fungi | anteaya: the ceph one doesn't need an acl file. it specifies an existing acl filename in the review.projects.yaml file | 17:45 |
| *** vponomaryov has left #openstack-infra | 17:45 | |
| *** jroll is now known as jimmyjohns | 17:45 | |
| * anteaya looks again | 17:46 | |
| fungi | anteaya: the swift3 patch itself doesn't need fixing, i was just hoping matt ray would either chime in that he was removing those topic branches or that there was some reason to continue development on them within gerrit | 17:46 |
| anteaya | line 440, so it does | 17:46 |
| fungi | but i couldn't find him in irc today | 17:46 |
| anteaya | how about the ceph one | 17:46 |
| anteaya | anything I can do for it? | 17:46 |
| anteaya | it looks like it is in the same boat | 17:47 |
| anteaya | a noise git source | 17:47 |
| anteaya | noisy | 17:47 |
| fungi | anteaya: er, i meant the ceph one for matt ray | 17:48 |
| fungi | the swift3 one has a typo, in addition to having a gh-pages branch which the owner may or may not want imported into gerrit | 17:48 |
| fungi | (a typo in the url for the imported repo, so a bug) | 17:49 |
| anteaya | yes | 17:49 |
| anteaya | I could fix the typo, I guess unless we are waiting to hear back do they want the whole farm or just master | 17:49 |
| fungi | i'm slightly less worried about deleting branches for that one than for the trove agent repo where there were more than 30, but that got cleaned up | 17:50 |
| *** hashar has quit IRC | 17:50 | |
| anteaya | wow | 17:50 |
| fungi | and same for the chef ceph one | 17:51 |
| anteaya | more than 30, nice work | 17:51 |
| anteaya | i don't see a trove patch on today's menu, was that from before? | 17:51 |
| anteaya | a legend now | 17:51 |
| fungi | it only has a few which they may or may not want imported (but with names including stuff like "wip" i suspect they don't really want those) | 17:51 |
| *** alff_ has joined #openstack-infra | 17:51 | |
| * anteaya nods | 17:51 | |
| fungi | sorry, not trove, ironic | 17:51 |
| anteaya | ah | 17:52 |
| anteaya | clean now | 17:52 |
| *** CaptTofu has joined #openstack-infra | 17:52 | |
| anteaya | as you said | 17:52 |
| *** alff has quit IRC | 17:53 | |
| kiall | jeblair: not yet, this/next weekend :) | 17:54 |
| *** derekh has quit IRC | 17:56 | |
| clarkb | jeblair: looks like the biggest change has to do with where min_ready for each image/label is handled | 17:58 |
| *** rdwrer has quit IRC | 17:58 | |
| *** saju_m has quit IRC | 17:58 | |
| jeblair | clarkb: yep. so if we want 20 devstack nodes, we don't have to set 20/7 for each jenkins master. we just set 20. | 17:58 |
| jeblair | clarkb: it means we lose the ability to say "jekins01 should get 2x the nodes of the others", but we don't do that. | 17:58 |
| clarkb | yup | 17:59 |
| clarkb | this is more make it scale for me friendly I like it | 17:59 |
| fungi | i'm all for less configuring and more smart assumptions | 17:59 |
| *** caleb_` has quit IRC | 17:59 | |
| *** rdwrer has joined #openstack-infra | 18:00 | |
| *** johnthetubaguy has quit IRC | 18:01 | |
| * SergeyLukjanov reading 1k messages scrollback | 18:01 | |
| fungi | SergeyLukjanov: that's no way to spend your friday night | 18:02 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Make sure resolvconf is enabled https://review.openstack.org/82152 | 18:02 |
| jeblair | clarkb, fungi: ^ apparently resolvconf isn't actually enabled on some of our older servers | 18:02 |
| fungi | oh, fun! | 18:02 |
| jeblair | i'm not sure why... perhaps something change in the rax image... | 18:02 |
| jeblair | or... and this is a worrying idea... rax maybe came around and wrote out a static /etc/resolv.conf which broke the resolvconf symlink and disabled it? | 18:03 |
| fungi | they do seem to do that every few months. probably a good reason to periodically rebuild our servers anyway | 18:03 |
| jeblair | (which they might have done as a way to update nameservers or something) | 18:03 |
| fungi | the image updates i mean | 18:03 |
| jeblair | fungi: ya | 18:03 |
| jeblair | i don't have strong evidence for either (we could go to backups if we really wanted to know) | 18:04 |
| fungi | hopefully not the mass overwriting our files thing | 18:04 |
| jeblair | but i did see an 'orig' file in a resolvconf directory that had different values than the current nameservers, which is what put that idea in my head | 18:04 |
| fungi | but if we see some revert eventually, i guess we'll know whether they do that | 18:04 |
| fungi | oh, though in that case puppet will fix it back up for us anyway | 18:05 |
| clarkb | jeblair: fungi it will be so nice to have our own images :) | 18:05 |
| fungi | clarkb: YES | 18:05 |
| jeblair | ++ | 18:05 |
| sdague | so trove is now passing here - https://review.openstack.org/#/c/80706/ | 18:05 |
| sdague | once we land that we can remove the trove specific job | 18:05 |
| fungi | clarkb: it will be nicer to have official upstream distro images (not our own) with reasonable changelogs associated with them | 18:05 |
| *** jaypipes is now known as rustypipes | 18:05 | |
| clarkb | jeblair: is resolvconf a service we can just have the service resource enable? | 18:05 |
| jeblair | so we have unbound running almost everywhere now, but only used random places until we merge ^ (which i think we should do asap) | 18:05 |
| SlickNik | sdague: just saw the conversation regarding the console log issue with the trove job. | 18:05 |
| SlickNik | sdague: I'll submit a change to get rid of that job once https://review.openstack.org/#/c/80706/ merges | 18:06 |
| jeblair | clarkb: it is, but it is a noop if resolvconf is configured to not update; that reconfigures it and sets up the symlink (and does an update if needed) | 18:06 |
| clarkb | jeblair: ok | 18:06 |
| sdague | SlickNik: yep | 18:07 |
| clarkb | jeblair: approved | 18:07 |
| jeblair | sdague, SlickNik: yaaay! | 18:07 |
| *** rustypipes is now known as leakypipes | 18:07 | |
| clarkb | ok back to nodepool review | 18:07 |
| sdague | clarkb: or... you could +A putting trove on by default in d-g master :) | 18:08 |
| sdague | before you run into nodepool' | 18:08 |
| clarkb | okay | 18:08 |
| SlickNik | thanks sdague, jeblair. :) | 18:09 |
| openstackgerrit | A change was merged to openstack-infra/config: Make sure resolvconf is enabled https://review.openstack.org/82152 | 18:09 |
| clarkb | sdague: done | 18:09 |
| clarkb | SlickNik: welcome to the jungle | 18:09 |
| sdague | woot | 18:09 |
| *** amcrn has joined #openstack-infra | 18:09 | |
| SlickNik | clarkb: woot! And now I have GnR stuck in my head :P | 18:10 |
| *** thedodd has joined #openstack-infra | 18:10 | |
| *** morganfainberg_Z is now known as morganfainberg | 18:11 | |
| jeblair | clarkb, fungi, SergeyLukjanov: in the spirit of break-everything-friday, could we merge https://review.openstack.org/#/c/72779/ ? | 18:12 |
| jeblair | clarkb, fungi, SergeyLukjanov: it might need a followup patch too after new-project-friday, but it should be harmless and that will be a nice incremental chaneg | 18:12 |
| SergeyLukjanov | oh, I'm ended scrolling scrollback... | 18:13 |
| jeblair | SergeyLukjanov: we just keep making more for you | 18:13 |
| SergeyLukjanov | jeblair, thank you sir! | 18:13 |
| * anteaya hands leakypipes a mop and bucket | 18:13 | |
| clarkb | jeblair: sure, I mean if we are going to break everything might as well | 18:13 |
| clarkb | note I haven't even actually looked to see what change that is and I am ok with it :P | 18:13 |
| SergeyLukjanov | jeblair, looking on 72779 | 18:13 |
| clarkb | SlickNik: woot that was my goal | 18:14 |
| *** rdwrer has quit IRC | 18:14 | |
| clarkb | jeblair: ooh this is a fun one | 18:14 |
| jeblair | clarkb: yeah, though it should still be a noop until the one after it | 18:14 |
| jeblair | theoretically | 18:14 |
| clarkb | jeblair: and we should get zuul noop in | 18:15 |
| clarkb | >_> | 18:15 |
| jeblair | clarkb: it's merged; just needs a restart | 18:15 |
| jeblair | clarkb: (and then a jjb change) | 18:15 |
| clarkb | jeblair: is there a change up to replace the gate-noop job with noop in the layout.yaml? | 18:15 |
| jeblair | clarkb: not yet | 18:15 |
| SergeyLukjanov | ++ for true noop | 18:16 |
| *** jp_at_hp has quit IRC | 18:17 | |
| jeblair | i'll write that real quick; it has a gotcha | 18:17 |
| *** rdwrer has joined #openstack-infra | 18:17 | |
| *** dcramer_ has quit IRC | 18:20 | |
| *** talluri has quit IRC | 18:20 | |
| *** rpodolyaka1 has joined #openstack-infra | 18:20 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Add noop job to config validator https://review.openstack.org/82161 | 18:20 |
| jeblair | actually, i think that's the right fix to the gotcha ^ | 18:20 |
| clarkb | jeblair: oh nice | 18:20 |
| clarkb | jeblair: don't we also have the list all jobs compare against JJB jobs thing? | 18:21 |
| clarkb | jeblair: we might need a JJB noop job just to keep that test quiet | 18:21 |
| jeblair | clarkb: that's what i was on about; that function is actually in zuul | 18:22 |
| jeblair | clarkb: so zuul should add that to the list of jobs it checks automatically | 18:22 |
| clarkb | oh I thought it was in shell somewhere | 18:22 |
| jeblair | clarkb: generating the list is in shell, but the comparison is in zuul | 18:22 |
| clarkb | gotcha | 18:23 |
| clarkb | ok back to nodepool change | 18:23 |
| *** rpodolyaka1 has quit IRC | 18:24 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Replace gate-noop with noop https://review.openstack.org/82162 | 18:25 |
| *** amcrn has quit IRC | 18:27 | |
| adam_g | do the devstack checks that run on RAX use the standard ubuntu 12.04 image there? | 18:29 |
| jeblair | clarkb, fungi: hrm, so apparently that wasn't enough; we need to make /etc/resolv.conf be a symlink before that will work... | 18:29 |
| jeblair | clarkb, fungi: perhaps we should just go the other way, and make /etc/resolv.conf a file and ignore resolvconf? | 18:30 |
| clarkb | adam_g: yes, though we want to move to the pvhvm image | 18:30 |
| jeblair | (i don't think i realized it was quite so easy to disable) | 18:31 |
| *** krotscheck has joined #openstack-infra | 18:31 | |
| fungi | adam_g: well, they use the nonstandard "ununtu 12.04" images that rax creates and provides. we'd love to be able to use proper ubuntu-published cloud images instead (and hopefully soon we can) | 18:31 |
| clarkb | jeblair: and have resolv.conf point at localhost once unbound is in place as a hard set thing? | 18:32 |
| jeblair | clarkb: yeah, which is what we do on centos | 18:32 |
| fungi | jeblair: agreed resolvconf hopefully shouldn't touch /etc/resolv.conf itself anyway | 18:32 |
| jeblair | fungi: yeah it does this: | 18:33 |
| jeblair | [ -L /etc/resolv.conf ] || { report_err "/etc/resolv.conf isn't a symlink, not doing anything."; exit 0; } | 18:33 |
| jeblair | at the top of the script | 18:33 |
| fungi | and debian package maintainer scripts definitely shouldn't blow away existing files/changes under /etc | 18:33 |
| fungi | hopefully ubuntu hasn't tweaked theirs to do so | 18:33 |
| *** yolanda has quit IRC | 18:33 | |
| clarkb | jeblair: I think we can hard set it and be consistent | 18:34 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Disable resolvconf on ubuntu https://review.openstack.org/82164 | 18:34 |
| *** jp_at_hp has joined #openstack-infra | 18:34 | |
| clarkb | jeblair: reviewed the nodepool change. Just one thing inline | 18:35 |
| jeblair | clarkb: thx | 18:35 |
| clarkb | jeblair: and small comment on the resolv.conf change but Iam +2 there | 18:36 |
| *** caleb_ has joined #openstack-infra | 18:36 | |
| jeblair | clarkb: On centos we make it immutable; do we want to do that (can we do that? are the extended attrs enabled?) on ubuntu? | 18:37 |
| clarkb | jeblair: I don't think we should worry about that on ubuntu unless we need to. You can keep the exec that does it in the RedHat case and move the file out | 18:38 |
| clarkb | jeblair: though it may be more readable as is | 18:38 |
| *** azneita has joined #openstack-infra | 18:38 | |
| *** thuc has joined #openstack-infra | 18:38 | |
| jeblair | fungi: https://review.openstack.org/#/c/82164/ ? | 18:39 |
| fungi | jeblair: i think consistency may be nice, but i'm also fine with it as-is until we discover that we need to make it immutable in new places | 18:40 |
| *** thuc_ has joined #openstack-infra | 18:40 | |
| fungi | i'll check the ubuntu filesystem attribute situation first | 18:40 |
| *** packet has joined #openstack-infra | 18:41 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Add noop job to config validator https://review.openstack.org/82161 | 18:41 |
| clarkb | jeblair: fungi: do we need to make it immutable for the same reason it is immutable on centos? will rax try to change it for us? | 18:42 |
| *** azneita has quit IRC | 18:42 | |
| *** thuc has quit IRC | 18:43 | |
| openstackgerrit | A change was merged to openstack-infra/config: Revoke sudo from most jobs https://review.openstack.org/72779 | 18:43 |
| fungi | we seem to be using ext3 which could complicate thingsseems to work fine... http://paste.openstack.org/show/74036/ | 18:43 |
| *** e0ne has joined #openstack-infra | 18:43 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add 'labels' as a configuration primitive https://review.openstack.org/82147 | 18:43 |
| fungi | ignore the comment about ext3, extended attributes might have been a problem, but the immutable bit has been around since at least ext2 if memory serves | 18:44 |
| jeblair | fungi: oh, sorry | 18:44 |
| jeblair | should we immutableize it everywhere? | 18:45 |
| clarkb | jeblair: I am not really a fan of doing that unless necessary | 18:45 |
| fungi | jeblair: i think consistency may be nice, but i'm also fine with it as-is until we discover that we need to make it immutable in new places | 18:45 |
| lifeless | clarkb: we got you in on it I thought | 18:45 |
| clarkb | lifeless: I applied for it long ago on a different account | 18:45 |
| lifeless | ah | 18:45 |
| clarkb | iirc it was the ci account not the jenkins account | 18:45 |
| jeblair | ok, let's leave it | 18:45 |
| *** talluri has joined #openstack-infra | 18:46 | |
| clarkb | right let me look up glance for the jenkins account now | 18:47 |
| *** dkliban_afk is now known as dkliban | 18:48 | |
| *** zzelle has joined #openstack-infra | 18:49 | |
| openstackgerrit | A change was merged to openstack-infra/config: Disable resolvconf on ubuntu https://review.openstack.org/82164 | 18:49 |
| *** yolanda has joined #openstack-infra | 18:49 | |
| *** boris-42 is now known as boris-the-blade- | 18:49 | |
| *** boris-the-blade- is now known as boris-the-blade | 18:50 | |
| *** amcrn has joined #openstack-infra | 18:52 | |
| *** CannedMeat is now known as SpamapS | 18:56 | |
| *** alff_ has quit IRC | 18:57 | |
| *** dkehn__ has joined #openstack-infra | 18:59 | |
| clarkb | jeblair: fungi: while on the subject of breaking everything for friday. https://review.openstack.org/#/c/77005/ | 18:59 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Revoke sudo from almost all jobs https://review.openstack.org/82170 | 18:59 |
| clarkb | 77005 would be a good cleanup to have go through as right now we are in that middle spot where everything is weird | 19:00 |
| jeblair | aprvd | 19:00 |
| clarkb | jeblair: 82170 does the remaining non devstack jobs? | 19:01 |
| *** caleb_ has quit IRC | 19:01 | |
| *** thuc_ has quit IRC | 19:01 | |
| jeblair | clarkb: yep, should be everything | 19:02 |
| *** thuc has joined #openstack-infra | 19:02 | |
| jeblair | review.o.o is using it's local nameserver now | 19:02 |
| jeblair | which appears to be functioning... | 19:02 |
| fungi | excellent | 19:03 |
| *** mrodden has quit IRC | 19:03 | |
| *** mrodden has joined #openstack-infra | 19:04 | |
| openstackgerrit | A change was merged to openstack-infra/config: Switch to using version-properties.sh script. https://review.openstack.org/77005 | 19:05 |
| rustlebee | can someone add me to https://review.openstack.org/#/admin/groups/304,members ? (nova-docker-core) | 19:05 |
| rustlebee | it's empty | 19:05 |
| jeblair | rustlebee: done | 19:05 |
| rustlebee | jeblair: great thanks | 19:06 |
| *** thuc has quit IRC | 19:06 | |
| clarkb | does that have a ptl group too? | 19:07 |
| clarkb | nope nevermind it doesn't | 19:07 |
| clarkb | time for me to get lunch. I will return to breaking things after I eat | 19:07 |
| clarkb | jeblair: I am totally game for getting zuul noop through today and possible your nodepool changes though those are less pressing | 19:07 |
| jeblair | clarkb: yeah, the nodepool one is a breaking change (we'll need to stop it, merge the unwritten config change, update or drop a table, then start it) | 19:08 |
| *** rpodolyaka1 has joined #openstack-infra | 19:08 | |
| jeblair | so probably good to do during a slow period | 19:09 |
| jeblair | though the multi-node stuff still needs one more change to be written | 19:10 |
| *** mrmartin has joined #openstack-infra | 19:13 | |
| fungi | if we need to drop a table when upgrading nodepool, is this also going to be a complicated update for any of the third parties who are currently cd'ing nodepool from master? (there are at least several) | 19:14 |
| fungi | might be worthwhile to provide a few days heads up on the -infra ml first if so | 19:14 |
| openstackgerrit | A change was merged to openstack-infra/config: Add a script to manage IRC perms https://review.openstack.org/78483 | 19:15 |
| SergeyLukjanov | jeblair, this one too please https://review.openstack.org/#/c/79446/ | 19:16 |
| SergeyLukjanov | fungi, heh, for sahara-ci we'll just need to drop our 6 nodes :) | 19:17 |
| SergeyLukjanov | fungi, ++ for ml heads up | 19:17 |
| *** weshay has quit IRC | 19:18 | |
| *** salv-orlando has quit IRC | 19:20 | |
| jeblair | yeah, i'll send out an announcement, but we've never made any promises about nodepool master. it has no schema migration framework or even any tests, and it has 'openstack.org' still hardcoded into it all over the place | 19:20 |
| *** yolanda has quit IRC | 19:21 | |
| jeblair | so i don't think we should delay for that reason | 19:21 |
| fungi | fair enough | 19:21 |
| jeblair | but i'll send the mail now | 19:22 |
| openstackgerrit | A change was merged to openstack-infra/config: Add #openstack-sahara channel to accessbot https://review.openstack.org/79446 | 19:22 |
| *** reed has quit IRC | 19:23 | |
| *** e0ne has quit IRC | 19:24 | |
| *** e0ne has joined #openstack-infra | 19:25 | |
| *** CaptTofu has quit IRC | 19:28 | |
| *** mrodden1 has joined #openstack-infra | 19:28 | |
| *** CaptTofu has joined #openstack-infra | 19:29 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Create propose_translation_update for horizon https://review.openstack.org/82176 | 19:29 |
| *** mrodden has quit IRC | 19:31 | |
| *** yolanda has joined #openstack-infra | 19:32 | |
| *** krotscheck_ has joined #openstack-infra | 19:33 | |
| *** CaptTofu has quit IRC | 19:33 | |
| *** krotscheck_ has quit IRC | 19:34 | |
| *** krotscheck has quit IRC | 19:34 | |
| *** khyati has quit IRC | 19:34 | |
| *** krotscheck_ has joined #openstack-infra | 19:34 | |
| *** krotscheck_ is now known as krotscheck | 19:34 | |
| *** thuc has joined #openstack-infra | 19:35 | |
| *** adam_g has left #openstack-infra | 19:38 | |
| *** adam_g has joined #openstack-infra | 19:38 | |
| *** amotoki has quit IRC | 19:38 | |
| *** dcramer_ has joined #openstack-infra | 19:38 | |
| *** krotscheck has quit IRC | 19:39 | |
| *** yolanda has quit IRC | 19:41 | |
| *** rpodolyaka1 has quit IRC | 19:42 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/devstack-gate: Don't enable savanna service for sahara https://review.openstack.org/81058 | 19:42 |
| *** hashar has joined #openstack-infra | 19:45 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Coverage fix https://review.openstack.org/82121 | 19:46 |
| *** krotscheck has joined #openstack-infra | 19:48 | |
| clarkb | ok back from lunch | 19:50 |
| *** krotscheck has quit IRC | 19:51 | |
| *** krotscheck has joined #openstack-infra | 19:51 | |
| *** jp_at_hp has quit IRC | 19:52 | |
| adam_g | which instance flavor is used on the RAX cloud? trying to find out why this works for me on RAX but fails in the gate: http://logs.openstack.org/23/81123/3/check/check-tempest-dsvm-virtual-ironic/e15cca4/console.html | 19:52 |
| *** thuc_ has joined #openstack-infra | 19:53 | |
| *** rfolco has quit IRC | 19:53 | |
| clarkb | jeblair: see comment on https://review.openstack.org/#/c/82162/ | 19:54 |
| clarkb | adam_g: you can see them at https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/templates/nodepool/nodepool.yaml.erb | 19:55 |
| clarkb | adam_g: base-image: 'Ubuntu 12.04 LTS (Precise Pangolin)' being one example (I think that is the precise one being used | 19:55 |
| adam_g | clarkb, oh cool. those are the details i was looking for | 19:55 |
| adam_g | every job gets a fresh instance, correct? | 19:55 |
| clarkb | adam_g: we slap some stuff on top of that image and take a snapshot then boot that so it isn't quite 1:1 but should be close | 19:55 |
| clarkb | adam_g: yup every devstack job happens on a one time use slave | 19:56 |
| adam_g | clarkb, interesting. ill see if running from a snapshot changes anything | 19:56 |
| fungi | adam_g: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/nodepool/scripts/prepare_devstack.sh | 19:56 |
| *** thuc has quit IRC | 19:56 | |
| *** thuc_ has quit IRC | 19:57 | |
| adam_g | fungi, that is what gets slapped on top? | 19:57 |
| fungi | adam_g: that and the other scripts it calls from the same directory should give you an idea of how the machines end up configured on top of the base image | 19:57 |
| fungi | yep | 19:57 |
| clarkb | fungi: jeblair: where are we on new projects? | 19:57 |
| adam_g | fungi, thanks | 19:57 |
| fungi | clarkb: there are four, several of which need reviews or possibly fixes applied | 19:58 |
| *** rpodolyaka1 has joined #openstack-infra | 19:58 | |
| fungi | adam_g: basically nodepool boots an instance built from the base image clarkb mentioned, then scps the nodepool scripts onto it and runs that one (which calls some of the others), then shuts it down, takes a snapshot and boots jenkins slaves from them. makes new images that way nightly | 19:59 |
| *** SumitNaiksatam has quit IRC | 20:01 | |
| clarkb | fungi: should I review and approve those that don't need work and push patches to the others? | 20:02 |
| clarkb | fungi: then we can start creating them? | 20:02 |
| jeblair | clarkb: back, and responded to https://review.openstack.org/#/c/82162/ | 20:02 |
| *** krotscheck has quit IRC | 20:02 | |
| adam_g | fungi, cool. easy enough to reproduce | 20:03 |
| JayF | for https://review.openstack.org/#/c/79088/, devananda asked if we should change the group for tags to the ironic-ptl or ironic-milestone group but to check with you guys first -- I'm assuming that using ironic-milestone should be fine? | 20:03 |
| *** krotscheck has joined #openstack-infra | 20:03 | |
| *** Ryan_Lane has quit IRC | 20:03 | |
| clarkb | JayF: we use ptl for tags most places | 20:03 |
| fungi | JayF: tags usually use the ptl group since it should be release managers/delegates taking care of pushing signed tags on projects | 20:04 |
| ArxCruz | ALL: Is there a way to, using o-gate to define my own Cirros image? | 20:04 |
| JayF | clarkb: is there an ironic-ptl group? I didn't see one referenced anywhere in the configs that I was looking through. | 20:04 |
| fungi | the ptl group can be multiple people delegated by the ptl for those duties | 20:04 |
| clarkb | JayF: if the group doesn't exist it will be created | 20:04 |
| *** jgrimm has quit IRC | 20:04 | |
| JayF | okay, I'll use ironic-ptl then for that case, would that be fine with all of you? and if it needs creation, we should make sure devananda is the first member | 20:05 |
| jeblair | JayF: wfm | 20:05 |
| *** saju_m has joined #openstack-infra | 20:05 | |
| JayF | tyvm | 20:05 |
| devananda | thanks guys | 20:05 |
| clarkb | jeblair: fungi: you ok with me approving those new-project changes? some of them will pull in more gate-noop job instances | 20:05 |
| jeblair | clarkb: yeah, that's secondary | 20:05 |
| *** khyati_ has joined #openstack-infra | 20:05 | |
| *** krotscheck has quit IRC | 20:05 | |
| jeblair | i marked the ironic agent one wip for JayF's change | 20:06 |
| *** krotscheck has joined #openstack-infra | 20:06 | |
| fungi | clarkb: sure | 20:06 |
| fungi | should be safe | 20:06 |
| clarkb | I approved openstackdroid. Will wait for JayF to update the ironic agent | 20:06 |
| jeblair | is someone going to fix the ceph one? | 20:06 |
| clarkb | and I will push patches to the other two to fix them | 20:06 |
| clarkb | jeblair: I can do it | 20:07 |
| jeblair | clarkb: cool, thx | 20:07 |
| *** rfolco has joined #openstack-infra | 20:07 | |
| *** rpodolyaka1 has quit IRC | 20:07 | |
| *** mihgen has joined #openstack-infra | 20:07 | |
| fungi | clarkb: well, one of them needs a minor typo fix and possibly some branches deleted, the other just possibly needs branches deleted | 20:07 |
| openstackgerrit | A change was merged to openstack-infra/config: Initial openstackdroid commit for Stackforge, with review changes https://review.openstack.org/81234 | 20:07 |
| openstackgerrit | Jay Faulkner proposed a change to openstack-infra/config: Add openstack/ironic-python-agent https://review.openstack.org/79088 | 20:07 |
| fungi | so if we're cool with doing that after import, then i'll switch my vote to +2 on it | 20:07 |
| jeblair | if it's just a gh-pages branch, i vote we just merge as is and delete later | 20:08 |
| clarkb | fungi: jeblair ya actually I think that is easier than me forking on github and cleaning it up there | 20:08 |
| fungi | the gh-pages branch is on the one with the url mistyped, so that's safe | 20:08 |
| fungi | once corrected | 20:08 |
| clarkb | yup I will correct the url | 20:08 |
| clarkb | and the ceph cookbooks repo has the same problem | 20:08 |
| JayF | Updated ironic-python-agent https://review.openstack.org/79088 -- I think this should be g2g now | 20:08 |
| fungi | the other one is matt ray's and i haven't been able to find him on irc to get feedback | 20:09 |
| clarkb | I can clean up the branches in gerrit if that is what they want | 20:09 |
| clarkb | and we can import it as is | 20:09 |
| fungi | yup, okay | 20:09 |
| *** krotscheck has quit IRC | 20:09 | |
| *** krotscheck has joined #openstack-infra | 20:09 | |
| *** rpodolyaka1 has joined #openstack-infra | 20:10 | |
| jeblair | lists.o.o is using its unbound server | 20:10 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Add swift3 into stackforge https://review.openstack.org/81991 | 20:10 |
| fungi | yay! | 20:10 |
| clarkb | fungi: ^ how does that look? | 20:10 |
| openstackgerrit | A change was merged to openstack-infra/config: Another fix for upstream_translation_horizon.sh https://review.openstack.org/82114 | 20:10 |
| clarkb | jeblair: lists and review were the two problematic hosts right? | 20:10 |
| jeblair | clarkb: yep | 20:11 |
| jeblair | clarkb: and, apparently from the several tickets on the openstackjenkins account, the jenkins slaves actually have been a problem for them | 20:11 |
| fungi | clarkb: lgtm | 20:11 |
| jeblair | further evidence those dns errors may be caused by their rate limiting | 20:12 |
| openstackgerrit | A change was merged to openstack-infra/config: Add openstack/ironic-python-agent https://review.openstack.org/79088 | 20:12 |
| fungi | jeblair: clarkb: and ask.o.o, but we don't control that one | 20:12 |
| jeblair | clarkb, fungi: did you see the disappointing response from hpcloud on dns? | 20:12 |
| JayF | whoopwhoop | 20:12 |
| JayF | tyvm guys for getting the agent in | 20:12 |
| dhellmann | gerrit question: If I have a person's name and email, is there some way to get the gerrit id that I need to put in the json config file for reviewstatus? It looks like it wants a short-form of their id or a nickname or something. | 20:12 |
| clarkb | jeblair: not yet /me looks | 20:12 |
| fungi | jeblair: yeah, at least it's under our "control" (to pick some resolvers somewhere at random?) | 20:13 |
| jeblair | JayF: since our automation is broken, it's probably another hour or so until the project is actually imported | 20:13 |
| clarkb | jeblair: that seems like a canned we don't quite understand the question response | 20:13 |
| jeblair | clarkb: yep. | 20:13 |
| *** krotscheck has quit IRC | 20:13 | |
| *** rfolco has quit IRC | 20:13 | |
| *** mallow123 has joined #openstack-infra | 20:13 | |
| clarkb | jeblair: also if the 206 address was specified on the subnet we didn't specify it | 20:13 |
| clarkb | I didn't at least | 20:13 |
| fungi | jeblair: one speculation on the jenkins slave lookup failures was that we were hitting previously used ip addresses which had been blacklisted in their resolvers and that they don't necessarily lift their blacklists immediately when addresses get recycled | 20:13 |
| *** afazekas has joined #openstack-infra | 20:14 | |
| clarkb | fungi: oh interesting | 20:14 |
| jeblair | fungi: possible. it's also possible we were the previous owner of that ip address. :) | 20:14 |
| *** rpodolyaka1 has quit IRC | 20:14 | |
| fungi | extremely possible ;) | 20:14 |
| openstackgerrit | A change was merged to openstack-infra/config: Add Ceph support to existing Chef cookbooks https://review.openstack.org/78681 | 20:14 |
| jeblair | fungi: but that does help to explain the occurences when it seemed like it happened very early | 20:14 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: trove is an integrated project add it by default https://review.openstack.org/80706 | 20:15 |
| fungi | right, at times where we shouldn't (yet) have done much in the way of lookups | 20:15 |
| jeblair | we should do a devstack-gate run where we tcpdump port 53 | 20:15 |
| fungi | jeblair: another possible mitigation there... have the nodepool ssh acceptance test try to do a dns lookup before marking the node ready? | 20:15 |
| jeblair | fungi: we could. if we still see the problem in our forwarding configuration, we might consider that (or running our own long-lived resolvers for forwarding, or just doing recursive on the slaves) | 20:17 |
| fungi | yep | 20:17 |
| clarkb | jeblair: which seems to be what hpcloud would like us to do | 20:17 |
| jeblair | i'm going to kick off image builds | 20:17 |
| openstackgerrit | A change was merged to openstack-infra/config: Add swift3 into stackforge https://review.openstack.org/81991 | 20:17 |
| clarkb | once ^ makes it onto the git replicas we are good to run manage projects right? | 20:17 |
| jeblair | clarkb: let's make sure the right people really understand the question/issue before we assume that's the final word | 20:17 |
| jeblair | clarkb: because i agree it seemed like a canned response | 20:18 |
| jeblair | clarkb: (also, neither of our provider images make it easy to change dns!) | 20:18 |
| clarkb | well hp is saying we should change it via the neutron dhcp server | 20:18 |
| clarkb | which isn't terrible if we can do that in rax land too | 20:18 |
| clarkb | when they neutron | 20:18 |
| *** dprince has quit IRC | 20:19 | |
| jeblair | clarkb: oh is that what they are saying | 20:19 |
| jeblair | that's slightly better. but still kind of copping out on the "hey why is your dns breaked?" question. | 20:19 |
| *** mwagner_wfh_bbl has quit IRC | 20:19 | |
| clarkb | jeblair: yup | 20:19 |
| *** mwagner_lap has joined #openstack-infra | 20:19 | |
| fungi | i got that, but it still doesn't address why they differed, nor whether they're willing to fix dnssec on theirs | 20:19 |
| fungi | right | 20:19 |
| *** mwagner_ has joined #openstack-infra | 20:21 | |
| *** jasondotstar has quit IRC | 20:21 | |
| *** vkozhukalov has joined #openstack-infra | 20:21 | |
| *** SumitNaiksatam has joined #openstack-infra | 20:21 | |
| jeblair | really need a way to trigger image updates for everything in nodepool :/ | 20:22 |
| *** openstackgerrit has quit IRC | 20:22 | |
| *** rossella_s has quit IRC | 20:22 | |
| *** openstackgerrit has joined #openstack-infra | 20:23 | |
| *** harlowja_still_a has joined #openstack-infra | 20:23 | |
| clarkb | nodepool image-build-all-the-things | 20:23 |
| *** krotscheck has joined #openstack-infra | 20:24 | |
| *** pcm_ has joined #openstack-infra | 20:25 | |
| jeblair | apparently we have 33 images | 20:25 |
| pcm_ | Can I get core review of https://review.openstack.org/#/c/75296/ please? | 20:25 |
| *** rfolco has joined #openstack-infra | 20:27 | |
| *** krotscheck_ has joined #openstack-infra | 20:27 | |
| clarkb | pcm_: we try to let the proper core bits of openstack/requirements review those unless it is an emergency. I think there has been talk about reworking how core works for that though | 20:27 |
| clarkb | sdague: ^ is that something you wanted to have done? | 20:28 |
| fungi | pcm_: the requirements core reviewers are those listed as direct members of https://review.openstack.org/#/admin/groups/131,members | 20:28 |
| *** krotscheck_ has quit IRC | 20:28 | |
| jeblair | okay, image rebuilds running in 33 screen windows | 20:29 |
| pcm_ | fungi: clarkb: Thanks, so I need to contact them specifically? (been waiting a long time for review) | 20:29 |
| clarkb | pcm_: ideally yes. Infra tends to only touch that if requirements are breaking infra | 20:30 |
| pcm_ | gotcha. Thanks! | 20:30 |
| sdague | honestly, not at this stage of the release | 20:30 |
| sdague | we need to stop using a million different test libraries | 20:30 |
| *** esker has quit IRC | 20:31 | |
| pcm_ | sdague: It's specifically for HTTP REST testing. | 20:31 |
| sdague | pcm_: sure | 20:31 |
| sdague | but we also have webtest, wsgi_intercept, and a host of other things | 20:32 |
| sdague | and we're basically coming up on dep freeze for release, so right now isn't the time to add new things | 20:32 |
| pcm_ | sdague: The alternative is that I essentially pull that source code into the test code base, or write a similar mechanism. | 20:32 |
| sdague | pcm_: where is the dependent test that's calling this? | 20:33 |
| pcm_ | https://github.com/openstack/neutron/blob/master/neutron/tests/unit/services/vpn/device_drivers/notest_cisco_csr_rest.py | 20:34 |
| pcm_ | I have it labeled notest... so it doesn't run... but have been using it in my environment for weeks. The review has been out for weeks as well. | 20:34 |
| *** jasondotstar has joined #openstack-infra | 20:35 | |
| jesusaurus | /help | 20:35 |
| *** saju_m has quit IRC | 20:35 | |
| dtroyer | pcm_: fwiw, we already have a client dependency on httpretty for this sort of testing. I know they're not exactly the same thing, and for most I prefer not monkeypatching the runtime, but that's already in... | 20:35 |
| sdague | pcm_: so, no, not at this point in the cycle. Also, my general understanding is this "start the whole wsgi stack" approach to testing neutron is supposed to get rethunk in juno | 20:36 |
| clarkb | fungi: git01 and git05 have swift3 at this point but not 02-04 | 20:36 |
| clarkb | fungi: will ping when I think those repos are properly in place | 20:36 |
| pcm_ | sdague: i'd encourage you to look at the library, as it is NOT starting a whole stack. | 20:36 |
| dtroyer | sdague: it's actually a less-invasive and requests-specific parallel to httprettty | 20:37 |
| dtroyer | I wish we'd seen it about six months ago | 20:37 |
| sdague | dtroyer: that's fine, but at this point in the cycle I don't want to keep piling on new requirements. We're weeks into freeze | 20:37 |
| jeblair | clarkb, fungi: qq: i added a "dig" command to sanity check the dns configuration in the bootstrap script. it looks like at least one centos image doesn't have that installed. should i make bind-utils be installed everywhere or remove the sanity check? | 20:38 |
| sdague | so I think it's a juno topic | 20:38 |
| *** rpodolyaka1 has joined #openstack-infra | 20:38 | |
| sdague | and I think in doing so we do need to have a conversation about test library convergence | 20:38 |
| dtroyer | sdague: totally agreed… I'm more just slighty sad out loud having fought this exact problem most of Teusday afternoon and resorting to httpretty since it's already in keystoneclient | 20:38 |
| sdague | heh | 20:38 |
| fungi | sorry, had to step away--catching back up now | 20:39 |
| openstackgerrit | Adam Gandelman proposed a change to openstack-infra/config: Be diligent in unmounting /proc/xen for Ironic https://review.openstack.org/82191 | 20:39 |
| sdague | pcm_: so one we've cut stable/icehouse on requirements please start a mailing list thread on it | 20:39 |
| sdague | because if this is a better version of httpretty, than I'd like to get httpretty removed as part of it | 20:39 |
| *** thomasem has quit IRC | 20:40 | |
| pcm_ | sdague: cool! I really like the library. Thanks. Will do. | 20:40 |
| clarkb | fungi: swift3 is everywhere now as are the other 3 new projects | 20:40 |
| clarkb | fungi: I think that means we are ready to run manage-projects right? | 20:40 |
| pcm_ | sdague: Should we do anything on the code review (dispositioning wise)? | 20:40 |
| *** jasondotstar has quit IRC | 20:41 | |
| *** mrmartin has quit IRC | 20:41 | |
| sdague | pcm_: I just put a -2 on it for now which I'll remove once we open up the trees again | 20:41 |
| fungi | jeblair: installing dig everywhere seems like a good idea to me--we did the same for other useful investigative tools like tcpdump and netcat | 20:41 |
| clarkb | have we done the first RCs yet? that must happen soon if not | 20:41 |
| sdague | I'd say start the email thread now, but I honestly expect it will get ignored as people work on the releases | 20:41 |
| clarkb | sdague: Iask about RCs because after the ton of bug fixes this week seems like RCs would be a good thing | 20:41 |
| sdague | clarkb: no rcs yet | 20:41 |
| dtroyer | pcm_: FWIW, jamielennox is going to be very interested in this topic, he introduced httpretty… | 20:41 |
| pcm_ | sdague: thanks. How will I know that reuqirements has been cut? I work on neutron, so not sure how tell. | 20:41 |
| sdague | pcm_: it will be once there are stable/* trees for all projects | 20:42 |
| fungi | clarkb: should be fine to run manage-projects now. please run it with the -d option and tee/redirect to a file so we can figure out what breaks (if anything) | 20:42 |
| sdague | which is probably 2 - 3 weeks, I forget when in the cycle we go from mp -> stable | 20:43 |
| *** prad has quit IRC | 20:43 | |
| pcm_ | sdague: OK. I'll wait and then raise on the ML. Would love to see discussion on it and hopefully acceptance. | 20:43 |
| *** rpodolyaka1 has quit IRC | 20:44 | |
| sdague | yeh, if dtroyer's assessment is accurate, I think it shouldn't be an issue. | 20:44 |
| sdague | it's just a timing problem right now | 20:44 |
| pcm_ | sdague: sure np. Appreciate the attention to this. | 20:44 |
| clarkb | fungi: ok doing that now | 20:44 |
| *** vhoward has left #openstack-infra | 20:45 | |
| clarkb | fungi: jeblair it is running in a root screen and is dumping logs to a file via tee | 20:46 |
| sdague | speaking of requirements - dhellmann you around? | 20:47 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Make sure dig is installed everywhere https://review.openstack.org/82193 | 20:47 |
| jeblair | clarkb, fungi: brown bag fix pls ^ | 20:47 |
| clarkb | jeblair: +2'd | 20:48 |
| jeblair | (that is installed on hpcloud but not rax, btw) | 20:48 |
| clarkb | fungi: feel free to approve | 20:48 |
| sdague | jeblair: "tests and radical idea like schema migrations" :) | 20:49 |
| jeblair | :) | 20:50 |
| clarkb | devananda: you have been added to the ironic-ptl group | 20:52 |
| openstackgerrit | A change was merged to openstack-infra/config: Make sure dig is installed everywhere https://review.openstack.org/82193 | 20:52 |
| clarkb | devananda: and the agent's core group. Freel free to add members to that core group | 20:52 |
| clarkb | I suppose I can add JayF too | 20:53 |
| devananda | clarkb: thanks. is there a substantive difference between -ptl and -milestone groups? | 20:53 |
| clarkb | devananda: ptl group is for ptl tasks eg doing releases | 20:53 |
| clarkb | milestone is for other ttx related things iirc | 20:53 |
| devananda | ah, gotcha | 20:53 |
| fungi | devananda: -milestone groups get to pick what's approved to the milestone-proposed backport branch during release freeze | 20:53 |
| devananda | fungi: ahhh. does that also apply post-release, eg. for who gets to pick backports? | 20:54 |
| clarkb | JayF: what is your gerrit account number? you can find it on the gerrit settings page | 20:54 |
| fungi | devananda: that said, right now openstack/python-ironicclient can get tags pushed by any members of ironic-core, which probably does merit solving | 20:54 |
| JayF | clarkb: 10342 | 20:54 |
| devananda | fungi: indeed | 20:54 |
| clarkb | JayF: you are now ironic agent core too | 20:54 |
| JayF | tyvm | 20:55 |
| *** prad has joined #openstack-infra | 20:55 | |
| devananda | thanks guys! | 20:55 |
| *** prad has quit IRC | 20:55 | |
| *** SpamapS has quit IRC | 20:58 | |
| jimmyjohns | b 10 | 20:59 |
| jimmyjohns | oops | 20:59 |
| *** julim has quit IRC | 20:59 | |
| *** SpamapS has joined #openstack-infra | 20:59 | |
| *** SpamapS has quit IRC | 20:59 | |
| *** SpamapS has joined #openstack-infra | 20:59 | |
| *** pdmars has quit IRC | 21:00 | |
| dhellmann | sdague: here | 21:00 |
| *** mfer has quit IRC | 21:00 | |
| jeblair | INFO:nodepool.image.build.hpcloud-az1.bare-centos6:Exception: Failed to clone stackforge/openstackdroid | 21:00 |
| jeblair | clarkb: lol building images on new project friday :) | 21:00 |
| clarkb | jeblair: whoops | 21:01 |
| clarkb | jeblair: manage-projects is done now so you should be clear to try again >_> | 21:01 |
| jeblair | cool | 21:01 |
| *** thuc has joined #openstack-infra | 21:02 | |
| clarkb | I have seeded the initial group members for all projects except for the ceph one as that is piggy backing on existing chef cookbook groups | 21:03 |
| *** dkehn__ has quit IRC | 21:03 | |
| *** dkehn_ has quit IRC | 21:04 | |
| sdague | dhellmann: as req core, you want to provide a +2 on this one - https://review.openstack.org/#/c/79817/ | 21:04 |
| clarkb | I don't see anything out of place with the new projects | 21:05 |
| clarkb | I am pretty confident in manage-projects now | 21:05 |
| sdague | dhellmann: it's one of the last 2 req changes I think we need to land pre release | 21:05 |
| dhellmann | sdague: we've tested that? it's so late in the cycle... | 21:05 |
| sdague | it's self testing | 21:05 |
| *** dkehn_ has joined #openstack-infra | 21:05 | |
| dhellmann | sdague: famous last words? :-) | 21:06 |
| sdague | 0.9 is used on all those test runs | 21:06 |
| sdague | you can look for yourself | 21:06 |
| dhellmann | sdague: +2 a | 21:06 |
| dhellmann | sdague: no, I just meant we've seen things pass tests before and then introduce instability | 21:06 |
| sdague | also note markmc's comments - all the distros are going to use it anyway | 21:06 |
| sdague | we're not helping them by not running it upstream | 21:06 |
| clarkb | huh did I tee manage-projects wrong? the log file is zero bytes | 21:07 |
| clarkb | fungi: does that go to stderr instead of stdout? | 21:07 |
| dhellmann | sdague: yeah | 21:07 |
| *** hashar has quit IRC | 21:07 | |
| clarkb | well good thing it looks like everything is happy | 21:07 |
| dhellmann | sdague: you're a gerrit guru, maybe you can answer this: I want to put together a list of gerrit ids for the cores in the requirements review group, but all I have are names and email addresses. How do I get the id value I need for the reviewstatus app rustlebee wrote? | 21:07 |
| fungi | clarkb: oh, yeah i usually do 2>&1 | tee foofile | 21:08 |
| clarkb | dhellmann: there is a hack to get that number. click on the persons name in gerrit and the url it goes to ends in that number | 21:08 |
| fungi | clarkb: sorry 'bout that | 21:08 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Correct invocation order of install_devstack_dependencies https://review.openstack.org/82208 | 21:08 |
| clarkb | dhellmann: not sure if there is a programmatic way to get it that doesn't suck | 21:08 |
| clarkb | fungi: its ok, I now know :) | 21:08 |
| dhellmann | clarkb: the other example files have usernames not numbers, does the tool work with numbers? | 21:08 |
| clarkb | oh it wants usernames why did I read that as account numbers | 21:09 |
| jeblair | clarkb, fungi: https://review.openstack.org/82208 is urgent | 21:09 |
| jeblair | dhellmann: usernames show up in gerrit queries over ssh | 21:09 |
| jeblair | dhellmann: (i wrote that patch actually) | 21:09 |
| sdague | dhellmann: mostly just gets done by knowing those things... I've never reversed it out beyond that | 21:09 |
| jeblair | dhellmann: so if you run "ssh review gerrit query 82193" | 21:10 |
| clarkb | fungi: in any case manage-projects seems to have gone smoothly. I am pretty confident that the only remaining problem is the orchestration across nodes | 21:10 |
| jeblair | dhellmann: you can see my username as the owner | 21:10 |
| dhellmann | jeblair: is there a way to dump a group list over ssh? | 21:10 |
| jeblair | dhellmann: so if you run a similar query for changes owned or reviewed by the people you're interested in, you can get them | 21:10 |
| * dhellmann should read some gerrit docs | 21:10 | |
| clarkb | not with our old gerrit | 21:10 |
| dhellmann | jeblair: ah, ok | 21:10 |
| clarkb | mikal got it added to new gerrit though | 21:10 |
| clarkb | so around end of april you should be able to do it | 21:11 |
| openstackgerrit | A change was merged to openstack-infra/config: Correct invocation order of install_devstack_dependencies https://review.openstack.org/82208 | 21:13 |
| *** alff_ has joined #openstack-infra | 21:17 | |
| jeblair | 3 new images are ready (a bare-precise and 2 bare centos6) all in rax | 21:18 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: [WIP] Comments UI https://review.openstack.org/82213 | 21:19 |
| *** rcleere has quit IRC | 21:22 | |
| *** mbacchi has quit IRC | 21:22 | |
| *** nicedice has joined #openstack-infra | 21:24 | |
| clarkb | jeblair: when you are done https://review.openstack.org/#/c/81792/ >_> | 21:25 |
| *** talluri has quit IRC | 21:26 | |
| *** talluri has joined #openstack-infra | 21:26 | |
| *** afazekas has quit IRC | 21:26 | |
| *** sandywalsh has quit IRC | 21:27 | |
| jeblair | done. but it won't get built anyway since tripleo thinks we're over quota | 21:27 |
| *** e0ne has quit IRC | 21:27 | |
| clarkb | oh right | 21:29 |
| *** jepoy_ has quit IRC | 21:29 | |
| clarkb | was lifeless leaving that as in order to debug? | 21:29 |
| *** jepoy has joined #openstack-infra | 21:29 | |
| clarkb | lifeless: ^ I saw the bug you submitted to nova about it but that isn't tacking the state of broken in your cloud | 21:29 |
| clarkb | oh right lifeless is weekending | 21:29 |
| *** dstanek_afk has joined #openstack-infra | 21:31 | |
| *** alff_ has quit IRC | 21:31 | |
| jeblair | 8/33 images | 21:33 |
| jeblair | (still only unit test nodes) | 21:33 |
| *** jepoy has quit IRC | 21:33 | |
| clarkb | guys I am braving the hipchats for glance access | 21:34 |
| jeblair | clarkb: good luck! | 21:35 |
| *** jamielennox|away is now known as jamielennox | 21:38 | |
| clarkb | jeblair: I am told the request is being submitted | 21:39 |
| openstackgerrit | Adam Gandelman proposed a change to openstack-infra/config: Be diligent in unmounting /proc/xen for Ironic https://review.openstack.org/82191 | 21:39 |
| *** rfolco has quit IRC | 21:39 | |
| *** rpodolyaka1 has joined #openstack-infra | 21:40 | |
| fungi | i look forward to glancing all the things | 21:41 |
| fungi | gotta run to dinner though--will catch up in an hour or so | 21:41 |
| *** dstanek_afk is now known as dstanek | 21:42 | |
| jeblair | fungi: enjoy! | 21:42 |
| jeblair | clarkb: cool. if i don't see a response on the rax ticket next week i'll send emails | 21:42 |
| *** thedodd has quit IRC | 21:43 | |
| *** rfolco has joined #openstack-infra | 21:44 | |
| *** rpodolyaka1 has quit IRC | 21:45 | |
| *** torgomatic has joined #openstack-infra | 21:46 | |
| clarkb | jeblair: fungi: the access I am requesting will make the images private to our project which I figured is a good first step | 21:46 |
| *** _nadya_ has joined #openstack-infra | 21:47 | |
| clarkb | there may be ways to make them more public if we want that at some point but that is apparently more access and trickier | 21:47 |
| clarkb | so I figured we can worry about that later once we figure out what we should do with these images uploads | 21:47 |
| *** jeckersb is now known as jeckersb_gone | 21:49 | |
| *** packet has quit IRC | 21:49 | |
| *** mrodden1 is now known as mrodden | 21:50 | |
| jeblair | clarkb: yeah, and we can just drop these on static.o.o to serve to people | 21:50 |
| jeblair | clarkb: though we may want to run a public glance too, just so we're that cool. | 21:51 |
| marun | what exactly does DEVSTACK_GATE_NEUTRON configure? | 21:52 |
| sdague | marun: it turns on the neutron services and doesn't turn on n-net | 21:52 |
| jeblair | marun: http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate-wrap.sh#n150 | 21:52 |
| marun | sdague: perfect, thanks! | 21:52 |
| marun | jeblair: ah, cool. | 21:52 |
| marun | jeblair: my grepping foo is weak :/ | 21:53 |
| marun | i was only looking in the config repo | 21:53 |
| sdague | http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate.sh | 21:53 |
| jeblair | marun: and several points in http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate.sh#n52 for the gory details | 21:53 |
| jeblair | (depending on branch) | 21:53 |
| marun | jeblair: good to know where to look, thanks. | 21:54 |
| jeblair | marun: np | 21:54 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/reviewstats: Add requirements project config file https://review.openstack.org/82219 | 21:54 |
| jeblair | INFO:nodepool.ImageUpdater:Image devstack-precise-1395437561.template.openstack.org in rax-dfw is ready | 21:55 |
| *** pcm_ has quit IRC | 21:55 | |
| jeblair | thats the first new devstack image ^ | 21:55 |
| pleia2 | oh, I didn't even notice, jenkins decided to ignore https://review.openstack.org/#/c/81792/ | 21:55 |
| pleia2 | no verified | 21:56 |
| clarkb | pleia2: recheck no bug it | 21:56 |
| pleia2 | k | 21:56 |
| clarkb | the zuul machinery is looking for a -1 or +1 check vote but there is no vote | 21:56 |
| clarkb | we can force one with a recheck no bug | 21:56 |
| pleia2 | gotcha, thanks | 21:56 |
| pleia2 | there it goes | 21:57 |
| *** jgrimm has joined #openstack-infra | 21:58 | |
| *** e0ne has joined #openstack-infra | 21:58 | |
| openstackgerrit | Maru Newby proposed a change to openstack-infra/config: Add a neutron job for api testing https://review.openstack.org/82226 | 21:59 |
| *** e0ne has quit IRC | 22:00 | |
| *** torgomatic has left #openstack-infra | 22:00 | |
| openstackgerrit | Maru Newby proposed a change to openstack-infra/config: Add a neutron job for api testing https://review.openstack.org/82226 | 22:00 |
| *** e0ne has joined #openstack-infra | 22:00 | |
| clarkb | marun: out of curiousity why are those tests not in tempest directly? | 22:00 |
| marun | clarkb: because tempest doesn't want them | 22:00 |
| clarkb | that'll do it | 22:00 |
| marun | clarkb: Plus, I'm trying to be clever. | 22:01 |
| openstackgerrit | A change was merged to openstack-infra/config: Setup the tripleo-test-cloud as a provider for Fedora instances https://review.openstack.org/81792 | 22:01 |
| marun | clarkb: I'm writing functional tests that directly target neutron plugins through an extraction layer. | 22:01 |
| marun | oops 'abstraction layer' | 22:01 |
| *** _nadya_ has quit IRC | 22:01 | |
| marun | clarkb: and then the same tests can use a different abstraction layer to target deployed neutron via the tempest rest client | 22:01 |
| marun | clarkb: birds, stones, etc | 22:02 |
| marun | sdague: reviews welcome https://review.openstack.org/#/c/82226 :) | 22:03 |
| *** dstanek has quit IRC | 22:04 | |
| marun | clarkb: hmm, I guess I can't use bash variables like that? | 22:05 |
| *** e0ne has quit IRC | 22:05 | |
| openstackgerrit | Maru Newby proposed a change to openstack-infra/config: Add a neutron job for api testing https://review.openstack.org/82226 | 22:05 |
| clarkb | marun: no you can't I commented on how to fix it | 22:05 |
| clarkb | you can also just go with $CMD_PREFIX | 22:06 |
| *** jepoy has joined #openstack-infra | 22:06 | |
| marun | ah, I just reverted to copy/paste | 22:06 |
| marun | is that ok? | 22:06 |
| clarkb | copy paste is fine too | 22:06 |
| jeblair | clarkb: btw, i think with the nodepool label change, we can build 'check' nodes that aren't based on different images | 22:06 |
| marun | clarkb: great, fixed. | 22:07 |
| clarkb | jeblair: oh man | 22:07 |
| jeblair | clarkb: so testing a provider/image combo will be easier | 22:07 |
| clarkb | jeblair: right we can just add a label and use the existing image when we are happy with a thing | 22:07 |
| sdague | marun: so is the only reason you are pulling in tempest is for our rest client? | 22:08 |
| sdague | you are going to need to do a lot more configuring of tempest than a pip install | 22:09 |
| sdague | to make it run | 22:09 |
| *** bknudson has quit IRC | 22:09 | |
| marun | sdague: that's devstack's job | 22:13 |
| marun | sdague: or am I preventing that from happening… hmmmm | 22:13 |
| marun | sdague: is there to ensure that tempest gets configure but doesn't run? | 22:14 |
| *** dstanek has joined #openstack-infra | 22:14 | |
| sdague | hmm... maybe this works | 22:14 |
| sdague | the pip install is weird though | 22:14 |
| marun | sdague: yeah, I don't know a better way to do it | 22:14 |
| flashgordon | dumb question: why can't I hit http://logs.openstack.org/57/45057/1/gate/gate-tempest-devstack-vm-postgres-full/13d43c1/console.html | 22:15 |
| sdague | flashgordon: because it's been gzed | 22:15 |
| marun | sdague: we'll want to add an explicit dependency but I'm hoping to delay doing so until tempest libification is complete | 22:15 |
| flashgordon | sdague: http://logs.openstack.org/57/45057/1/gate/gate-tempest-devstack-vm-postgres-full/13d43c1/console.html.gz doesn't work either | 22:15 |
| marun | sdague: so a manual install in the tox venv was the best I could come up with | 22:15 |
| marun | sdague: do you have an alternative suggestion? | 22:15 |
| flashgordon | neither does http://logs.openstack.org/57/45057/1/gate/gate-tempest-devstack-vm-postgres-full/13d43c1/ | 22:15 |
| sdague | marun: nope | 22:15 |
| sdague | marun: who thinks they are working on the libification. That we should definitely have a qa-spec proposed for | 22:16 |
| flashgordon | clarkb: ^ | 22:16 |
| marun | sdague: Maybe I'm just extrapolating in my head on some other effort | 22:16 |
| clarkb | flashgordon: files didn't upload then | 22:17 |
| clarkb | flashgordon: where is the chagne you are seeing that for? | 22:17 |
| marun | sdague: jaypipes has mentioned libification more than a couple of times to me but maybe that doesn't encompass the rest client and its configuration. | 22:17 |
| sdague | because there are lots of demands on tempest, especially with defcore coming up, so I really don't want people to think that's something that's emminent or even a priority | 22:17 |
| clarkb | flashgordon: derp | 22:17 |
| clarkb | its 45057 | 22:17 |
| flashgordon | clarkb: https://bugs.launchpad.net/nova/+bug/1221922 | 22:17 |
| uvirtbot | Launchpad bug 1221922 in nova "test_change_password_for_alt_account_fails timed out in swift gate job" [Undecided,Incomplete] | 22:17 |
| clarkb | flashgordon: we only keep 6 months of logs | 22:17 |
| clarkb | flashgordon: I think september was more than that ago | 22:18 |
| flashgordon | clarkb: ahh sept 2013 yeah | 22:18 |
| flashgordon | is it possible to get 1 year of logs? | 22:18 |
| clarkb | flashgordon: not without twice as much disk | 22:18 |
| marun | sdague: apologies, don't mean to imply its necessary for this job, just that it was one direction things could go. do you want me to change the comment in the job? | 22:18 |
| clarkb | /dev/mapper/main-logs 11T 11T 711G 94% /srv/static/logs | 22:18 |
| clarkb | jeblair: fungi ^ we are getting close to our limits there | 22:19 |
| marun | sdague: the alternative would be to add an explicit test dependency in neutron, I'd like to prove the concept before I propose that to neutron folk | 22:19 |
| clarkb | flashgordon: so if you have 22TB of disk laying around for 22TB of compressed logs sure | 22:19 |
| marun | sdague: explicit dependency -> on tempest | 22:19 |
| flashgordon | mordred: can I get more disk please? | 22:19 |
| flashgordon | clarkb: 22TB whoa | 22:19 |
| clarkb | sdague: flashgordon I definitely think that has been a large part of recent testing | 22:20 |
| clarkb | since it was 8.5TB last I looked not long ago | 22:20 |
| flashgordon | ahh makes sense | 22:20 |
| clarkb | flashgordon: yes it is insane | 22:20 |
| sdague | marun: so I'm totally cool with you doing things as this, just realize the tempest interfaces are fluid, so it will break | 22:20 |
| marun | sdague: fair enough, it'll probably stay non-voting for the immediate future anyway. | 22:20 |
| sdague | yep | 22:21 |
| sdague | it will be good data | 22:21 |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=717&rra_id=all | 22:21 |
| sdague | to try out the idea | 22:21 |
| *** dhellmann is now known as dhellmann_ | 22:21 | |
| marun | sdague: I'll update the comment for clarity at least, then feel free to +2 ;) | 22:21 |
| sdague | well, I'm only a +1 there | 22:21 |
| sdague | but I did at that | 22:21 |
| clarkb | we are due for that number to fall slightly looks like so I don't think this is an emergency prior to the weekend | 22:21 |
| openstackgerrit | Maru Newby proposed a change to openstack-infra/config: Add a neutron job for api testing https://review.openstack.org/82226 | 22:21 |
| clarkb | flashgordon: this is why we want to get logs into swift | 22:22 |
| clarkb | flashgordon: apparently we have more terabytes in swift | 22:22 |
| marun | sdague: danke | 22:22 |
| *** gokrokve has quit IRC | 22:24 | |
| flashgordon | clarkb: :) | 22:24 |
| openstackgerrit | A change was merged to openstack/requirements: allow sqla 0.9.x https://review.openstack.org/79817 | 22:24 |
| *** gokrokve has joined #openstack-infra | 22:24 | |
| flashgordon | sdague: have you seen this before: http://logs.openstack.org/12/80412/8/check/check-tempest-dsvm-postgres-full/f9f6158/logs/screen-n-cpu.txt.gz?level=TRACE#_2014-03-21_17_45_12_490 | 22:27 |
| flashgordon | no space left on disk | 22:27 |
| clarkb | flashgordon: which disk? the / disk is really small on those images | 22:28 |
| clarkb | flashgordon: we may need to tell more things to use /opt | 22:28 |
| flashgordon | clarkb: not sure which disk | 22:28 |
| flashgordon | but yeah | 22:28 |
| *** gokrokve has quit IRC | 22:29 | |
| clarkb | huh it is trying to do it with cgroups but cgroups are a virtual file system | 22:30 |
| clarkb | they live under /sys | 22:30 |
| *** BadCub01 has quit IRC | 22:30 | |
| flashgordon | ohh I think that is a recent patch | 22:30 |
| clarkb | to do cgroups? | 22:30 |
| *** jepoy has quit IRC | 22:32 | |
| flashgordon | I am mistaken | 22:32 |
| sdague | flashgordon: nope, that's new to me | 22:32 |
| sdague | it's nice of libvirt to tell us which device there was no space on .... | 22:32 |
| sdague | terrible logs are terrible | 22:33 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Fix syntax error in fedora nodepool config https://review.openstack.org/82235 | 22:33 |
| jeblair | clarkb: urgent ^ | 22:33 |
| flashgordon | haha is that your new mantra | 22:33 |
| flashgordon | sdague: like counting is hard | 22:33 |
| flashgordon | (which it is) | 22:33 |
| *** harlowja_still_a has quit IRC | 22:33 | |
| *** wchrisj has quit IRC | 22:34 | |
| flashgordon | message:"cgroup\: No space left on device" AND filename:logs*screen-n-cpu.txt | 22:34 |
| flashgordon | looks like a new bug | 22:34 |
| clarkb | jeblair: looking | 22:34 |
| flashgordon | my guess is new libvirt patch | 22:34 |
| flashgordon | based on rough timing | 22:34 |
| clarkb | jeblair: did you catch the comment about the logs fs? it is getting near full again | 22:34 |
| sdague | flashgordon: did it show up with the new libvirt? | 22:34 |
| *** enikanorov__ has quit IRC | 22:34 | |
| sdague | basically did we trade one bug for another | 22:34 |
| clarkb | jeblair: approved | 22:35 |
| jeblair | clarkb: that slope definitely seems to have increased | 22:35 |
| flashgordon | sdague: only has hits in last 24 hours | 22:35 |
| sdague | and, honestly, since when were we running cgroups | 22:35 |
| sdague | flashgordon: then probably yes | 22:36 |
| clarkb | jeblair: yeah I think it may be related to the recent logstash size increase too | 22:36 |
| sdague | flashgordon: so target it as an ubuntu bug as well | 22:36 |
| * flashgordon files a bug | 22:36 | |
| clarkb | jeblair: I swaer I looked not long ago and it was 8.5TB | 22:36 |
| flashgordon | dims: ^^^ | 22:36 |
| sdague | and explain we're running serge's hot fix | 22:36 |
| clarkb | about 2 weeks ago | 22:36 |
| jeblair | clarkb, sdague, flashgordon: did you get to the bottom of/fix the 'we have lots more logs possibly related to ceilometer' issue? | 22:37 |
| clarkb | jeblair: sort of | 22:37 |
| clarkb | it is still doing it though | 22:37 |
| clarkb | because ceilometer polls swift | 22:37 |
| sdague | clarkb: is this the log server, or logstash? | 22:37 |
| clarkb | sdague: log server | 22:37 |
| clarkb | sdague: went from 8.5TB to 11TB in two weeks | 22:37 |
| sdague | ok, so the log server is definitely being pummled by ceilometer | 22:37 |
| clarkb | I think so /me looks at more numbers | 22:37 |
| flashgordon | jeblair: it was a two fold issue. ceilometer middleware in swift was spamming things. and secondly ceilometer is polling swift every 15 seconds | 22:38 |
| flashgordon | we still have the 15 second poll from ceilo, so swift is generating a lot more logs | 22:38 |
| clarkb | sdague: see http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=717&rra_id=all | 22:38 |
| sdague | http://logs.openstack.org/12/80412/8/check/check-tempest-dsvm-postgres-full/f9f6158/logs/?C=S;O=A | 22:38 |
| sdague | the conductor logs are way up again as well | 22:38 |
| clarkb | are we also grabbing /etc/ stuff? | 22:39 |
| clarkb | I expect those to be small but maybe some are large | 22:39 |
| jeblair | clarkb: do you recall the per-host cinder limit? | 22:40 |
| sdague | etc is super small | 22:40 |
| jeblair | clarkb: this page says 14 volumes http://www.rackspace.com/knowledge_center/article/cloud-block-storage-overview#limits-of-cbs | 22:40 |
| jeblair | we have 13 | 22:40 |
| flashgordon | sdague: for conductor logs anything pop out? | 22:40 |
| clarkb | jeblair: yeah we are right up against the limits right now | 22:40 |
| sdague | flashgordon: all of it | 22:40 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix syntax error in fedora nodepool config https://review.openstack.org/82235 | 22:41 |
| sdague | the conductor logs are basically a trace of all of amqp | 22:41 |
| sdague | they really need a diet | 22:41 |
| sdague | it's 99.99999% debug | 22:41 |
| *** rpodolyaka1 has joined #openstack-infra | 22:41 | |
| sdague | so the top 6 logs by size | 22:43 |
| sdague | [ ]screen-ceilometer-anotification.txt.gz21-Mar-2014 18:101.2M | 22:43 |
| sdague | [ ]tempest.txt.gz21-Mar-2014 18:101.7M | 22:43 |
| sdague | [ ]screen-n-api.txt.gz21-Mar-2014 18:102.1M | 22:43 |
| sdague | [ ]screen-ceilometer-acentral.txt.gz21-Mar-2014 18:102.3M | 22:43 |
| sdague | [ ]screen-n-cpu.txt.gz21-Mar-2014 18:102.4M | 22:43 |
| sdague | [ ]screen-n-cond.txt.gz21-Mar-2014 18:105.2M | 22:43 |
| flashgordon | sdague: do we need to just turn off amqp debug log evel? | 22:43 |
| sdague | [ ]screen-ceilometer-collector.txt.gz21-Mar-2014 18:106.3M | 22:43 |
| sdague | 7 actually | 22:43 |
| sdague | flashgordon: we already do | 22:43 |
| flashgordon | https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1295876 | 22:43 |
| uvirtbot | Launchpad bug 1295876 in libvirt "libvirtError: internal error unable to add domain xxx to cgroup: No space left on device" [Undecided,New] | 22:43 |
| sdague | then conductor just logs it itself again | 22:43 |
| sdague | honestly, some sifting would be good | 22:43 |
| sdague | my tempest patch series that's up cuts the tempest log in half | 22:43 |
| dims | flashgordon, this was the diff of the patched libvirt - http://launchpadlibrarian.net/169379991/libvirt_0.9.8-2ubuntu17.16_0.9.8-2ubuntu17.18.diff.gz | 22:44 |
| sdague | clarkb: I need to call it a week, my wife is home, and it's been a long one | 22:44 |
| jeblair | clarkb: i'm adding another (presumably the last) volume to static | 22:44 |
| clarkb | jeblair: roger | 22:44 |
| jeblair | clarkb: we also have about 250g of unallocated space on existing volumes | 22:44 |
| sdague | however if you wanted to start an ML thread on this issue | 22:44 |
| clarkb | sdague: sure why don't I do that while jeblair does the technical stuff | 22:45 |
| sdague | especially calling out the worst offenders | 22:45 |
| flashgordon | dims: that patch means nothing to me I don't know the context for the code | 22:45 |
| sdague | and saying "big logs kill puppies, or why you can't have nice things if you don't help us trim these down." | 22:45 |
| jeblair | clarkb, sdague: next step for us is likely reducing retention to < 6 mo; at least until the swift stuff is in place | 22:45 |
| jeblair | (which is close) | 22:45 |
| jeblair | (but is going to take a lot of testing and work so still isn't a near term fix) | 22:46 |
| sdague | jeblair: well, dealing with the compressed retrevial is still unsolved right? | 22:46 |
| sdague | I feel like that's not a solved problem yet | 22:46 |
| flashgordon | dims: so it looks like this new bug is much less frequent (I think) | 22:46 |
| clarkb | https://etherpad.openstack.org/p/test-logs-and-retention | 22:46 |
| jeblair | sdague: that's correct | 22:47 |
| *** jasondotstar has joined #openstack-infra | 22:47 | |
| *** rpodolyaka1 has quit IRC | 22:47 | |
| flashgordon | dims: in last 12 hours only 10 builds hit this | 22:47 |
| jeblair | sdague: enjoy your weekend! | 22:48 |
| sdague | you too | 22:48 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1295876 https://review.openstack.org/82245 | 22:49 |
| uvirtbot | Launchpad bug 1295876 in libvirt "libvirtError: internal error unable to add domain xxx to cgroup: No space left on device" [Undecided,New] https://launchpad.net/bugs/1295876 | 22:49 |
| flashgordon | clarkb: can you include a pretty graph in your email? | 22:49 |
| flashgordon | dims so the new bug is much less frequebnt | 22:50 |
| flashgordon | clarkb: thanks! | 22:51 |
| *** alexpilotti has quit IRC | 22:52 | |
| dims | flashgordon, be with you in 10 mins or so | 22:52 |
| *** mallow123 has quit IRC | 22:54 | |
| jeblair | clarkb: resizing logs volume (adding 1024g) | 22:54 |
| *** nati_ueno has joined #openstack-infra | 22:54 | |
| clarkb | oh also, the subunit logs for some projects are huge I bet | 22:55 |
| clarkb | flashgordon: can you spot check nova, neutron, cinder, etc subunit log files for me? | 22:55 |
| flashgordon | clarkb: sure what do you want me to check for? | 22:57 |
| *** jasondotstar has quit IRC | 22:57 | |
| clarkb | flashgordon: file size | 22:57 |
| flashgordon | btw just cut hacking 0.8.1 | 22:57 |
| clarkb | a list like sdague's aboev for the different projects would be great | 22:57 |
| flashgordon | hopefully that didn't break anything ... | 22:58 |
| flashgordon | http://logs.openstack.org/04/72004/5/gate/gate-tempest-dsvm-neutron/5f46ee5/logs/?C=S;O=D | 22:58 |
| flashgordon | from top of gate | 22:58 |
| *** dims has quit IRC | 22:59 | |
| flashgordon | nothing is that big in there | 22:59 |
| clarkb | flashgordon: not for tempest | 22:59 |
| clarkb | for unittests | 22:59 |
| flashgordon | checking full | 22:59 |
| *** zhiwei has joined #openstack-infra | 22:59 | |
| flashgordon | ohh | 22:59 |
| clarkb | so python26 and python27 jobs | 23:00 |
| *** e0ne has joined #openstack-infra | 23:00 | |
| flashgordon | nova: http://logs.openstack.org/81/81681/3/check/gate-nova-python27/38fda42/ 14MB | 23:01 |
| flashgordon | that was my fault too:) | 23:01 |
| flashgordon | I enabled debug logs | 23:01 |
| flashgordon | sorry :/ | 23:01 |
| jeblair | wow, watching the used space on static.o.o is like watching the national debt clock | 23:01 |
| flashgordon | 14 MB for gate-nova-python26 | 23:02 |
| clarkb | flashgordon: lolololol | 23:02 |
| clarkb | jeblair: :( | 23:02 |
| *** CaptTofu has joined #openstack-infra | 23:02 | |
| flashgordon | cinder: <1MB | 23:02 |
| clarkb | flashgordon: I think neutron will be massive | 23:02 |
| *** gokrokve has joined #openstack-infra | 23:03 | |
| flashgordon | 7.8 MB | 23:03 |
| flashgordon | http://logs.openstack.org/41/82141/1/gate/gate-neutron-python27/4a5f410/ | 23:03 |
| flashgordon | for both 26 27 | 23:04 |
| flashgordon | glance 86K | 23:04 |
| flashgordon | http://logs.openstack.org/67/77167/3/gate/gate-glance-python27/f6c20b8/ | 23:04 |
| flashgordon | keysone 4.8MB http://logs.openstack.org/70/65870/9/gate/gate-keystone-python27/dadd343/ | 23:04 |
| clarkb | flashgordon: yeah so I think neutron and nova are the offenders here | 23:04 |
| *** e0ne has quit IRC | 23:05 | |
| flashgordon | ceilo 158K | 23:05 |
| flashgordon | swift 1.4 | 23:05 |
| flashgordon | M | 23:05 |
| flashgordon | http://logs.openstack.org/65/82165/1/gate/gate-swift-python27/8bab6bd/ | 23:05 |
| flashgordon | clarkb: agreed | 23:05 |
| flashgordon | with keystone in #3 | 23:05 |
| flashgordon | Ijust checked integrated projects | 23:05 |
| clarkb | how does that etherpad look? | 23:08 |
| flashgordon | looks good | 23:09 |
| flashgordon | maybe mention icehouse release | 23:10 |
| lifeless | clarkb: jeblair indicated it wasn't affecting nodepool other than not having as many slaves as desired so I ignored in favor of family | 23:10 |
| *** esker has joined #openstack-infra | 23:10 | |
| *** BadCub01 has joined #openstack-infra | 23:10 | |
| lifeless | clarkb: also we need to capture data for nova to fix it | 23:10 |
| clarkb | lifeless: ++ to ignoring for family. It does also slightly affect our ability ot build images though whcih isn't a huge deal wither | 23:10 |
| flashgordon | http://jogo.github.io/gate/ | 23:10 |
| flashgordon | pretty impressive | 23:10 |
| *** jasondotstar has joined #openstack-infra | 23:11 | |
| clarkb | jeblair: flashgordon I am calling that etherpad good and will send it shortly if you guys don't have anything to add | 23:11 |
| flashgordon | clarkb: maybe add a source for file sizes | 23:12 |
| flashgordon | I was using http://logs.openstack.org/52/81252/2/gate/gate-tempest-dsvm-full/488bc4e/logs/?C=S;O=D | 23:12 |
| flashgordon | which has similar results | 23:12 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add ready-script and multi-node support https://review.openstack.org/82252 | 23:13 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add the ability to create subnodes https://review.openstack.org/81939 | 23:13 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add 'labels' as a configuration primitive https://review.openstack.org/82147 | 23:13 |
| jeblair | /dev/mapper/main-logs 12T 11T 1.7T 86% /srv/static/logs | 23:14 |
| jeblair | clarkb: complete ^ | 23:14 |
| jeblair | clarkb: we still have 250g unallocated | 23:14 |
| *** dims has joined #openstack-infra | 23:14 | |
| *** denis_makogon_ has joined #openstack-infra | 23:15 | |
| *** vkozhukalov has quit IRC | 23:15 | |
| flashgordon | clarkb: for nova I am thinking turn off oslo.messaging debug logs? | 23:16 |
| dims | flashgordon, 1295876 and the large logs are unrelated, right? | 23:16 |
| flashgordon | dims: yes | 23:16 |
| clarkb | jeblair: thanks | 23:16 |
| clarkb | jeblair: does the email look good to you? | 23:17 |
| jeblair | clarkb: just confirmed that we can't add more volumes to static | 23:17 |
| jeblair | clarkb: lgtm | 23:18 |
| * clarkb does last minute formatting and will send | 23:19 | |
| *** gokrokve has quit IRC | 23:21 | |
| flashgordon | clarkb: gonna see what https://review.openstack.org/82255 does to nova logs | 23:21 |
| *** gokrokve has joined #openstack-infra | 23:21 | |
| *** mriedem has quit IRC | 23:22 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Add ready-script and multi-node support https://review.openstack.org/82252 | 23:22 |
| jeblair | okay, that's the whole patch series for nodepool multi-node | 23:23 |
| clarkb | flashgordon: jeblair sdague sent the message | 23:24 |
| clarkb | jeblair: that includes the third step now? | 23:24 |
| clarkb | jeblair: I will try to review the stack again after beer thirty which I am being called away to | 23:24 |
| jeblair | clarkb: yep | 23:24 |
| jeblair | yeah, i'm going to go run around in circles for a while | 23:24 |
| jeblair | we have 31/33 new images | 23:25 |
| *** gokrokve has quit IRC | 23:25 | |
| clarkb | jeblair: looking back at the cacti graph, I wonder if maybe some things aren't getting deleted? flashgordon's example from september above doesn't support that theory though | 23:26 |
| clarkb | since it was properly deleted | 23:26 |
| clarkb | but time for me to afk for a bit | 23:26 |
| *** mrodden has quit IRC | 23:26 | |
| *** caleb_ has joined #openstack-infra | 23:29 | |
| *** hogepodge has left #openstack-infra | 23:33 | |
| *** CaptTofu has quit IRC | 23:38 | |
| *** wchrisj has joined #openstack-infra | 23:38 | |
| *** asselin has joined #openstack-infra | 23:38 | |
| *** jepoy has joined #openstack-infra | 23:41 | |
| *** jasondotstar has quit IRC | 23:43 | |
| fungi | clarkb: jeblair: caught up now... yes, 14 cinder volumes because the version of xen rackspace is running can't present more than 16 virtual block devices to a domu (and root and swap are already occupying two of those) | 23:43 |
| *** david_lyle_ has quit IRC | 23:44 | |
| *** rpodolyaka1 has joined #openstack-infra | 23:44 | |
| fungi | and their cinder volume size maximum is 1tb, so 14tb of cinder space max per instance | 23:44 |
| fungi | also, the retention reduction would have to be substantially less than 6 months because out growth rate means that something like 75% of the logs will be from just the past few months | 23:45 |
| fungi | we could run some estimates, but i expect we'd need to cut our retention to something like 2-3 months | 23:45 |
| *** khyati_ has quit IRC | 23:45 | |
| clarkb | yup | 23:47 |
| *** CaptTofu has joined #openstack-infra | 23:47 | |
| fungi | because remember that what you're looking at with ethe filesystem graph is a derivative (sliding window of 6 months data total size) | 23:47 |
| fungi | 6 months ago, 6 months of data was ~3tib. so the rate at which we accumulate data has more than tripled in size in 6 months | 23:48 |
| *** rpodolyaka1 has quit IRC | 23:48 | |
| *** dstanek has quit IRC | 23:48 | |
| *** thuc_ has joined #openstack-infra | 23:50 | |
| *** denis_makogon_ has quit IRC | 23:51 | |
| *** Ryan_Lane has joined #openstack-infra | 23:52 | |
| dims | flashgordon, nothing pops out, and it does not seem to be causing a lot of harm to the gate. let's wait a bit and see if it gets worse. ok? | 23:53 |
| *** thuc has quit IRC | 23:53 | |
| dims | i was worried that the libvirt change caused a flood of logs. | 23:54 |
| *** thuc_ has quit IRC | 23:54 | |
| *** yamahata has quit IRC | 23:55 | |
| *** yamahata has joined #openstack-infra | 23:55 | |
| *** zzelle has quit IRC | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!