| *** rossella_s has quit IRC | 00:00 | |
| *** lezbar has quit IRC | 00:00 | |
| *** lezbar has joined #openstack-infra | 00:00 | |
| *** rossella_s has joined #openstack-infra | 00:00 | |
| *** karthik__ has quit IRC | 00:00 | |
| *** jamielennox|away is now known as jamielennox | 00:01 | |
| *** amitgandhinz has quit IRC | 00:02 | |
| *** EricGonczer_ has joined #openstack-infra | 00:02 | |
| *** mriedem has joined #openstack-infra | 00:06 | |
| *** zul has quit IRC | 00:07 | |
| *** zul has joined #openstack-infra | 00:09 | |
| *** EricGonczer_ has quit IRC | 00:10 | |
| *** r-mibu has quit IRC | 00:10 | |
| *** sputnik13 has quit IRC | 00:10 | |
| *** sdake has quit IRC | 00:10 | |
| *** sputnik13 has joined #openstack-infra | 00:12 | |
| *** baoli has joined #openstack-infra | 00:13 | |
| *** mtanino has quit IRC | 00:14 | |
| *** rbuzatu has joined #openstack-infra | 00:14 | |
| *** rwsu has quit IRC | 00:14 | |
| JayF | pabelanger: getting another set of new failures now too http://logs.openstack.org/81/119981/33/check/gate-ironic-python-agent-python27-ubuntu-xenial/fc6fb6b/console.html#_2016-08-03_22_46_15_817261 | 00:16 |
|---|---|---|
| *** sputnik13 has quit IRC | 00:16 | |
| JayF | pabelanger: that appears to be well before it runs anything specific to my patch, and I have multiple failures of that nature :x | 00:17 |
| *** baoli has quit IRC | 00:18 | |
| *** rbuzatu has quit IRC | 00:18 | |
| * JayF leaving for the evening | 00:19 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/puppet-openstack_infra_spec_helper: Pin json_pure gem for ruby1.9 support https://review.openstack.org/350835 | 00:23 |
| *** shashank_hegde has quit IRC | 00:30 | |
| *** tonytan4ever has joined #openstack-infra | 00:33 | |
| *** r-mibu has joined #openstack-infra | 00:35 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add support for Growl plugin https://review.openstack.org/350838 | 00:37 |
| *** tonytan4ever has quit IRC | 00:38 | |
| *** Na3iL has joined #openstack-infra | 00:39 | |
| *** xarses has quit IRC | 00:39 | |
| *** thorst has joined #openstack-infra | 00:45 | |
| jlk | mordred: so your cluster I believe is back in action, and ready for infra to take it on again. | 00:47 |
| jlk | (I forgot to say something last week) | 00:47 |
| *** pvaneck has quit IRC | 00:50 | |
| *** gildub has joined #openstack-infra | 00:50 | |
| *** amitgandhinz has joined #openstack-infra | 00:58 | |
| *** xarses has joined #openstack-infra | 00:59 | |
| *** amitgandhinz has quit IRC | 01:02 | |
| *** thorst has quit IRC | 01:04 | |
| *** thorst has joined #openstack-infra | 01:05 | |
| *** zhurong has joined #openstack-infra | 01:06 | |
| *** elo has quit IRC | 01:07 | |
| *** Sukhdev has quit IRC | 01:12 | |
| *** thorst has quit IRC | 01:13 | |
| *** yuanying has quit IRC | 01:14 | |
| *** yuanying has joined #openstack-infra | 01:14 | |
| *** yuanying has quit IRC | 01:15 | |
| *** mmandel has joined #openstack-infra | 01:15 | |
| *** Apoorva_ has joined #openstack-infra | 01:16 | |
| *** zhurong has quit IRC | 01:16 | |
| *** zhurong has joined #openstack-infra | 01:17 | |
| *** yuanying has joined #openstack-infra | 01:18 | |
| *** Apoorva has quit IRC | 01:18 | |
| *** gouthamr has joined #openstack-infra | 01:19 | |
| *** tphummel has joined #openstack-infra | 01:19 | |
| *** mmandel has quit IRC | 01:19 | |
| *** Apoorva_ has quit IRC | 01:20 | |
| *** Na3iL has quit IRC | 01:20 | |
| *** asettle has joined #openstack-infra | 01:24 | |
| *** asettle has quit IRC | 01:28 | |
| *** EricGonczer_ has joined #openstack-infra | 01:34 | |
| *** jamielennox is now known as jamielennox|away | 01:34 | |
| *** weshay has quit IRC | 01:40 | |
| *** gouthamr has quit IRC | 01:41 | |
| *** tqtran_ has quit IRC | 01:42 | |
| *** gouthamr has joined #openstack-infra | 01:42 | |
| mordred | jlk: woot! | 01:43 |
| mordred | jlk: tomorrow I will make it do things | 01:43 |
| mordred | jlk: tomorrow I'm also going ot land the change that updates shade to use neutron directly for fips | 01:43 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: workaround for puppet-ceph https://review.openstack.org/350860 | 01:43 |
| jlk | oh shoot, I was supposed to test that out for you | 01:44 |
| jlk | crap on a stick | 01:44 |
| mordred | jlk: it's ok ... I'll double check it on the bb cloud tomorrow before I release | 01:44 |
| mordred | jlk: there are a few things still in the queue that need to land | 01:45 |
| jlk | I'm going to test it quickly now. | 01:45 |
| jlk | is it on shade master? | 01:45 |
| mordred | yah | 01:45 |
| jlk | okay, I'll clone/install | 01:45 |
| mordred | jlk: create_server auto_ip should work in the way you expect | 01:45 |
| mordred | if it doesn't, something is sad | 01:45 |
| *** thorst has joined #openstack-infra | 01:46 | |
| *** thorst has quit IRC | 01:47 | |
| *** thorst has joined #openstack-infra | 01:47 | |
| openstackgerrit | Joshua Harlow proposed openstack-dev/pbr: Expose deb version to match exposing rpm version https://review.openstack.org/350861 | 01:48 |
| jlk | my quick test will just be using os_floating_ip | 01:48 |
| jlk | oh haha, gotta install this on the controller side. haderp. | 01:48 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: networking-midonet: Update grafana dashboard for stable periodic jobs https://review.openstack.org/350862 | 01:51 |
| *** shashank_hegde has joined #openstack-infra | 01:55 | |
| *** dimtruck is now known as zz_dimtruck | 01:55 | |
| jlk | mordred: well the quick test seems to have been okay | 01:55 |
| *** thorst has quit IRC | 01:56 | |
| *** baoli has joined #openstack-infra | 01:56 | |
| *** dmellado has quit IRC | 01:57 | |
| *** migi has quit IRC | 01:57 | |
| *** ajo has quit IRC | 01:57 | |
| *** migi_ has joined #openstack-infra | 01:57 | |
| *** amitgandhinz has joined #openstack-infra | 01:59 | |
| *** dmellado has joined #openstack-infra | 02:02 | |
| *** ajo has joined #openstack-infra | 02:03 | |
| *** amitgandhinz has quit IRC | 02:03 | |
| *** amotoki has joined #openstack-infra | 02:10 | |
| *** thorst has joined #openstack-infra | 02:18 | |
| *** mriedem has quit IRC | 02:20 | |
| *** sarob has joined #openstack-infra | 02:23 | |
| *** zz_dimtruck is now known as dimtruck | 02:24 | |
| *** markvoelker has joined #openstack-infra | 02:26 | |
| *** baoli has quit IRC | 02:26 | |
| *** sarob has quit IRC | 02:27 | |
| *** thorst has quit IRC | 02:27 | |
| *** thorst has joined #openstack-infra | 02:28 | |
| *** furlongm_ has joined #openstack-infra | 02:28 | |
| *** furlongm has quit IRC | 02:29 | |
| *** dkehn_ has quit IRC | 02:31 | |
| *** dkehn has quit IRC | 02:32 | |
| *** ddieterly has joined #openstack-infra | 02:33 | |
| *** ddieterly has quit IRC | 02:36 | |
| *** zhurong has quit IRC | 02:36 | |
| *** jamielennox|away is now known as jamielennox | 02:36 | |
| *** thorst has quit IRC | 02:37 | |
| *** zhurong has joined #openstack-infra | 02:38 | |
| *** dkehn has joined #openstack-infra | 02:39 | |
| *** dkehn_ has joined #openstack-infra | 02:43 | |
| *** liusheng has quit IRC | 02:49 | |
| *** yuanying has quit IRC | 02:51 | |
| *** sarob has joined #openstack-infra | 02:51 | |
| *** gouthamr has quit IRC | 02:51 | |
| *** yuanying has joined #openstack-infra | 02:54 | |
| *** sarob has quit IRC | 02:55 | |
| *** hichihara has joined #openstack-infra | 02:56 | |
| *** yuanying has quit IRC | 02:56 | |
| *** amitgandhinz has joined #openstack-infra | 03:00 | |
| *** shashank_hegde has quit IRC | 03:02 | |
| *** yamahata has quit IRC | 03:03 | |
| *** sdake has joined #openstack-infra | 03:04 | |
| *** amitgandhinz has quit IRC | 03:04 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Add support for Cloud Foundry plugin https://review.openstack.org/350940 | 03:08 |
| *** binbincong has joined #openstack-infra | 03:08 | |
| *** yanyanhu has joined #openstack-infra | 03:11 | |
| *** binbincong has quit IRC | 03:13 | |
| *** mrmartin has joined #openstack-infra | 03:22 | |
| *** coolsvap has joined #openstack-infra | 03:22 | |
| *** sdake has quit IRC | 03:23 | |
| *** afred312_ has quit IRC | 03:29 | |
| *** persia has quit IRC | 03:30 | |
| *** persia has joined #openstack-infra | 03:33 | |
| *** shashank_hegde has joined #openstack-infra | 03:34 | |
| *** thorst has joined #openstack-infra | 03:35 | |
| *** thorst has quit IRC | 03:42 | |
| *** markvoelker has quit IRC | 03:46 | |
| *** gongysh has joined #openstack-infra | 03:47 | |
| *** sdake has joined #openstack-infra | 03:50 | |
| *** yuanying has joined #openstack-infra | 03:51 | |
| *** yuanying has quit IRC | 03:51 | |
| *** yuanying has joined #openstack-infra | 03:52 | |
| *** amotoki has quit IRC | 03:54 | |
| *** amotoki has joined #openstack-infra | 03:55 | |
| *** EricGonczer_ has quit IRC | 03:56 | |
| *** rossella_s has quit IRC | 04:00 | |
| *** amitgandhinz has joined #openstack-infra | 04:00 | |
| *** asselin_ has joined #openstack-infra | 04:00 | |
| *** rossella_s has joined #openstack-infra | 04:00 | |
| *** sarob has joined #openstack-infra | 04:01 | |
| *** asselin has quit IRC | 04:02 | |
| *** Sukhdev has joined #openstack-infra | 04:03 | |
| *** gongysh has quit IRC | 04:04 | |
| *** Sukhdev has quit IRC | 04:04 | |
| *** Sukhdev has joined #openstack-infra | 04:04 | |
| *** gongysh has joined #openstack-infra | 04:05 | |
| *** amitgandhinz has quit IRC | 04:06 | |
| *** sarob has quit IRC | 04:06 | |
| *** ayoung has quit IRC | 04:08 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/gerritbot: Add gerrit stream over MQTT support to gerritbot https://review.openstack.org/350651 | 04:14 |
| *** julim has quit IRC | 04:16 | |
| *** roxanaghe has joined #openstack-infra | 04:18 | |
| *** shashank_hegde has quit IRC | 04:21 | |
| *** markvoelker has joined #openstack-infra | 04:22 | |
| *** markvoelker has quit IRC | 04:26 | |
| *** links has joined #openstack-infra | 04:28 | |
| *** asselin has joined #openstack-infra | 04:29 | |
| *** kien-ha has quit IRC | 04:31 | |
| *** asselin_ has quit IRC | 04:31 | |
| odyssey4me | we're getting intermittent POST_FAILURES which appear to be due to an auth issue when uploading logs: http://logs.openstack.org/11/349611/2/check/gate-openstack-ansible-openstack-ansible-aio-ubuntu-trusty/8c18f82/_zuul_ansible/ansible_log.txt | 04:33 |
| odyssey4me | jhesketh ^ | 04:33 |
| *** sdake has quit IRC | 04:37 | |
| *** Qiming has quit IRC | 04:39 | |
| *** thorst has joined #openstack-infra | 04:41 | |
| *** tqtran has joined #openstack-infra | 04:42 | |
| *** Qiming has joined #openstack-infra | 04:44 | |
| jhesketh | odyssey4me: hmm, is your job touching any file perms on the logs per chance? | 04:45 |
| *** tqtran has quit IRC | 04:46 | |
| odyssey4me | jhesketh hmm, lemme check - we had an issue with that some time ago and I thought we had addressed it | 04:46 |
| *** shashank_hegde has joined #openstack-infra | 04:46 | |
| jhesketh | odyssey4me: if I were to guess either that log is copied or symlinked into that position and the jenkins user doesn't have the right permission to read it | 04:47 |
| *** thorst has quit IRC | 04:47 | |
| *** psachin has joined #openstack-infra | 04:48 | |
| odyssey4me | hmm, our job runs with sudo - but the directory containing all the data is traversable and all files world readable | 04:48 |
| odyssey4me | jhesketh ok, I see it's the swift logs specifically | 04:50 |
| odyssey4me | heh, ok - those are not world readable | 04:50 |
| jhesketh | right, maybe they have different permissions for some reason? | 04:50 |
| jhesketh | ah there we go | 04:50 |
| odyssey4me | thanks - I'll look into what's happening there | 04:50 |
| jhesketh | so doesn't need to be world readable, just readable by the user 'jenkins' | 04:50 |
| jhesketh | (although easier to be world readable, particularly if we rename the user to zuul one day - but that'll be a v3 thing which has other changes0 | 04:51 |
| *** gildub has quit IRC | 04:51 | |
| odyssey4me | yeah, all our other logs are world readable so I think we'll likely just go with that | 04:51 |
| jhesketh | :-) | 04:52 |
| *** gongysh has quit IRC | 04:53 | |
| *** icey has quit IRC | 04:57 | |
| *** javeriak has joined #openstack-infra | 04:59 | |
| *** afred312 has joined #openstack-infra | 04:59 | |
| *** rbuzatu has joined #openstack-infra | 05:00 | |
| *** amitgandhinz has joined #openstack-infra | 05:01 | |
| *** afred312 has quit IRC | 05:04 | |
| *** javeriak has quit IRC | 05:04 | |
| *** jraju has joined #openstack-infra | 05:04 | |
| *** icey has joined #openstack-infra | 05:04 | |
| *** amitgandhinz has quit IRC | 05:06 | |
| *** links has quit IRC | 05:06 | |
| *** narayrak has joined #openstack-infra | 05:06 | |
| *** shashank_hegde has quit IRC | 05:07 | |
| *** tphummel has quit IRC | 05:10 | |
| *** rcernin has joined #openstack-infra | 05:16 | |
| *** markvoelker has joined #openstack-infra | 05:16 | |
| *** aeng has joined #openstack-infra | 05:19 | |
| openstackgerrit | Merged openstack/diskimage-builder: Optionally remove portage files https://review.openstack.org/348536 | 05:20 |
| *** markvoelker has quit IRC | 05:22 | |
| *** shashank_hegde has joined #openstack-infra | 05:22 | |
| *** tphummel has joined #openstack-infra | 05:25 | |
| *** roxanaghe has quit IRC | 05:26 | |
| *** aeng has quit IRC | 05:32 | |
| *** Qiming has quit IRC | 05:32 | |
| *** Qiming has joined #openstack-infra | 05:33 | |
| openstackgerrit | Alexey Weyl proposed openstack-infra/project-config: Vitrage tempests https://review.openstack.org/350987 | 05:35 |
| *** rbuzatu has quit IRC | 05:35 | |
| *** thorst has joined #openstack-infra | 05:44 | |
| *** roxanaghe has joined #openstack-infra | 05:45 | |
| *** roxanaghe has quit IRC | 05:50 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/diskimage-builder: Nothing to see here https://review.openstack.org/345539 | 05:50 |
| *** thorst has quit IRC | 05:52 | |
| *** apetrich has quit IRC | 05:55 | |
| openstackgerrit | Ben Kero proposed openstack-infra/project-config: Add channel #diskimage-builder https://review.openstack.org/350997 | 05:59 |
| bkero | greghaynes: ^ | 05:59 |
| *** oanson has joined #openstack-infra | 06:00 | |
| *** afred312 has joined #openstack-infra | 06:00 | |
| *** apetrich has joined #openstack-infra | 06:00 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Fix dib PROJECTS path https://review.openstack.org/350998 | 06:01 |
| *** javeriak has joined #openstack-infra | 06:02 | |
| *** amitgandhinz has joined #openstack-infra | 06:02 | |
| *** sarob has joined #openstack-infra | 06:02 | |
| openstackgerrit | Ben Kero proposed openstack-infra/project-config: Add channel #diskimage-builder https://review.openstack.org/350997 | 06:02 |
| bkero | (rebase) | 06:02 |
| ianw | pabelanger / jeblair : ^^ see my comment in above (350998), this failure should have bailed out setup_workspace but it didn't? maybe ansible ignoring it? need to dig further | 06:03 |
| *** Srinu has joined #openstack-infra | 06:04 | |
| *** afred312 has quit IRC | 06:06 | |
| *** pt_15 has quit IRC | 06:06 | |
| *** amitgandhinz has quit IRC | 06:06 | |
| *** sarob has quit IRC | 06:07 | |
| *** camunoz has quit IRC | 06:09 | |
| *** gongysh has joined #openstack-infra | 06:09 | |
| *** asettle has joined #openstack-infra | 06:09 | |
| *** jtomasek has quit IRC | 06:09 | |
| *** pcaruana has joined #openstack-infra | 06:10 | |
| *** itisha has quit IRC | 06:10 | |
| *** Ravikiran_K has joined #openstack-infra | 06:12 | |
| *** gongysh has quit IRC | 06:13 | |
| *** rbuzatu has joined #openstack-infra | 06:13 | |
| *** asettle has quit IRC | 06:14 | |
| *** bhavik1 has joined #openstack-infra | 06:17 | |
| *** jaosorior has joined #openstack-infra | 06:20 | |
| openstackgerrit | Ben Kero proposed openstack-infra/project-config: Add channel #diskimage-builder https://review.openstack.org/350997 | 06:26 |
| bkero | womp womp, gerritbot is at its channel limit. It says to ask in here about setting up another gerritbot. | 06:28 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: grafana: Update midonet unit test jobs after xenial renames https://review.openstack.org/351016 | 06:28 |
| bkero | Anybody know what I'd need to do, or what the status of that is? | 06:28 |
| bkero | AJaeger: ^ | 06:29 |
| *** gildub has joined #openstack-infra | 06:30 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: grafana: Update ovn unit test jobs after xenial renames https://review.openstack.org/351019 | 06:31 |
| *** pgadiya has joined #openstack-infra | 06:32 | |
| *** gongysh has joined #openstack-infra | 06:35 | |
| *** Sukhdev has quit IRC | 06:38 | |
| *** karthikp_ has joined #openstack-infra | 06:38 | |
| *** javeriak has quit IRC | 06:39 | |
| *** yamamoto_ has quit IRC | 06:41 | |
| *** tqtran has joined #openstack-infra | 06:42 | |
| *** karthikp_ has quit IRC | 06:43 | |
| *** jlanoux has joined #openstack-infra | 06:45 | |
| *** roxanaghe has joined #openstack-infra | 06:46 | |
| yolanda | good morning | 06:46 |
| *** pahuang has quit IRC | 06:47 | |
| *** tqtran has quit IRC | 06:47 | |
| *** jed56 has joined #openstack-infra | 06:47 | |
| AJaeger | bkero: we need to run a second gerritbot. Not sure what needs done exactly. Basically running a second one, figure out how to distribute load over those two so that each stays under the limit of jobs... I cannot give details on that. | 06:47 |
| bkero | AJaeger: It would be very simple to just have a second yaml file inside gerritbot/ | 06:48 |
| AJaeger | morning, yolanda ! | 06:48 |
| bkero | although maybe someone could come up with something cleaner | 06:48 |
| AJaeger | bkero: Yeah, that would work... | 06:48 |
| * bkero not a rooter, so likely can't set up the box, but I can do the puppet work necessary for it | 06:48 | |
| bkero | if we agree that is the correct approach. | 06:48 |
| AJaeger | bkero: Best talk with fungi later today - this would be appreciated! | 06:49 |
| bkero | Sure :) | 06:49 |
| bkero | I'll talk with him during normal US hours. | 06:49 |
| *** roxanaghe has quit IRC | 06:50 | |
| *** armax has quit IRC | 06:51 | |
| *** thorst has joined #openstack-infra | 06:51 | |
| AJaeger | thanks, bkero ! | 06:53 |
| *** tesseract- has joined #openstack-infra | 06:53 | |
| *** permalac has quit IRC | 06:54 | |
| openstackgerrit | Merged openstack-infra/project-config: Implement tripleo-ci-centos-7-nonha-multinode-upgrades job https://review.openstack.org/350580 | 06:55 |
| openstackgerrit | Merged openstack-infra/project-config: Add searchlight api-ref jobs https://review.openstack.org/350264 | 06:56 |
| *** thorst has quit IRC | 06:57 | |
| openstackgerrit | Alexey Weyl proposed openstack-infra/project-config: Vitrage tempests https://review.openstack.org/350987 | 06:57 |
| *** rbuzatu_ has joined #openstack-infra | 06:57 | |
| *** rbuzatu has quit IRC | 07:01 | |
| *** ihrachys has joined #openstack-infra | 07:02 | |
| *** afred312 has joined #openstack-infra | 07:02 | |
| *** amitgandhinz has joined #openstack-infra | 07:03 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix dib PROJECTS path https://review.openstack.org/350998 | 07:05 |
| *** javeriak has joined #openstack-infra | 07:05 | |
| *** amitgandhinz has quit IRC | 07:07 | |
| *** afred312 has quit IRC | 07:08 | |
| openstackgerrit | Merged openstack-infra/project-config: puppet: workaround for puppet-ceph https://review.openstack.org/350860 | 07:10 |
| openstackgerrit | Merged openstack-infra/project-config: networking-midonet: Update grafana dashboard for stable periodic jobs https://review.openstack.org/350862 | 07:10 |
| *** mikelk has joined #openstack-infra | 07:10 | |
| *** florianf has joined #openstack-infra | 07:11 | |
| *** tphummel has quit IRC | 07:11 | |
| openstackgerrit | Merged openstack-infra/system-config: Fix emaint sync call https://review.openstack.org/348514 | 07:11 |
| openstackgerrit | Merged openstack-infra/project-config: Forbid ironic grenade job from building ramdisk https://review.openstack.org/350791 | 07:12 |
| *** mikelk has quit IRC | 07:14 | |
| openstackgerrit | Merged openstack-infra/project-config: Add OpenStack Infra internal repositories to APT source lists https://review.openstack.org/350056 | 07:14 |
| *** ccamacho has quit IRC | 07:16 | |
| *** gildub has quit IRC | 07:16 | |
| *** yamahata has joined #openstack-infra | 07:18 | |
| *** ccamacho has joined #openstack-infra | 07:18 | |
| *** nmagnezi has joined #openstack-infra | 07:19 | |
| *** witek has left #openstack-infra | 07:22 | |
| openstackgerrit | Merged openstack-infra/project-config: Move {name}-docs to ubuntu-xenial https://review.openstack.org/347941 | 07:23 |
| openstackgerrit | Merged openstack-infra/project-config: Make bindep Python 3.5 unit test jobs vote https://review.openstack.org/350594 | 07:23 |
| *** HeOS has joined #openstack-infra | 07:26 | |
| openstackgerrit | Merged openstack-infra/project-config: Disable services for tempest gate https://review.openstack.org/350454 | 07:27 |
| openstackgerrit | Merged openstack-infra/project-config: Add common testing repo to OpenStack-Ansible https://review.openstack.org/350503 | 07:27 |
| *** gongysh has quit IRC | 07:28 | |
| *** berendt has quit IRC | 07:30 | |
| *** dimtruck is now known as zz_dimtruck | 07:32 | |
| *** gongysh has joined #openstack-infra | 07:33 | |
| *** openstackgerrit has quit IRC | 07:33 | |
| *** ggnel_t has joined #openstack-infra | 07:33 | |
| *** openstackgerrit has joined #openstack-infra | 07:33 | |
| *** javeriak has quit IRC | 07:38 | |
| *** ihrachys has quit IRC | 07:40 | |
| *** acoles_ is now known as acoles | 07:45 | |
| *** vincentll has joined #openstack-infra | 07:47 | |
| *** shardy has joined #openstack-infra | 07:48 | |
| *** ihrachys has joined #openstack-infra | 07:48 | |
| *** thorst has joined #openstack-infra | 07:54 | |
| *** furlongm_ has quit IRC | 07:54 | |
| *** e0ne has joined #openstack-infra | 07:55 | |
| *** e0ne has quit IRC | 07:55 | |
| *** e0ne has joined #openstack-infra | 07:56 | |
| *** ihrachys has quit IRC | 07:57 | |
| *** markvoelker has joined #openstack-infra | 07:58 | |
| *** ihrachys has joined #openstack-infra | 07:59 | |
| *** gildub has joined #openstack-infra | 07:59 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** rossella_s has quit IRC | 08:00 | |
| *** sflanigan has quit IRC | 08:00 | |
| *** jpich has joined #openstack-infra | 08:00 | |
| *** rossella_s has joined #openstack-infra | 08:00 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/351049 | 08:00 |
| *** zzzeek has joined #openstack-infra | 08:01 | |
| *** thorst has quit IRC | 08:02 | |
| *** afred312 has joined #openstack-infra | 08:03 | |
| *** markvoelker has quit IRC | 08:03 | |
| *** amitgandhinz has joined #openstack-infra | 08:03 | |
| *** ihrachys has quit IRC | 08:06 | |
| *** amitgandhinz has quit IRC | 08:08 | |
| *** afred312 has quit IRC | 08:08 | |
| *** ihrachys has joined #openstack-infra | 08:10 | |
| openstackgerrit | Merged openstack-infra/puppet-openstackci: Pinning JJB and Zuul for 3rd party CI https://review.openstack.org/348035 | 08:10 |
| *** jaosorior has quit IRC | 08:13 | |
| *** jaosorior has joined #openstack-infra | 08:13 | |
| *** hichihara has quit IRC | 08:14 | |
| *** e0ne has quit IRC | 08:14 | |
| *** e0ne has joined #openstack-infra | 08:15 | |
| *** Goneri has joined #openstack-infra | 08:15 | |
| *** rbuzatu_ has quit IRC | 08:16 | |
| *** dtantsur|afk is now known as dtantsur | 08:16 | |
| *** shashank_hegde has quit IRC | 08:17 | |
| *** shashank_hegde has joined #openstack-infra | 08:18 | |
| *** shashank_hegde has quit IRC | 08:19 | |
| *** rbuzatu has joined #openstack-infra | 08:20 | |
| *** yamahata has quit IRC | 08:22 | |
| *** dtantsur is now known as dtantsur|bbl | 08:23 | |
| *** lucas-dinner is now known as lucasagomes | 08:24 | |
| *** yaume has joined #openstack-infra | 08:25 | |
| *** yaume has quit IRC | 08:26 | |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/351049 | 08:29 |
| *** javeriak has joined #openstack-infra | 08:30 | |
| *** roxanaghe has joined #openstack-infra | 08:34 | |
| *** chem has joined #openstack-infra | 08:35 | |
| *** yaume has joined #openstack-infra | 08:36 | |
| *** roxanaghe has quit IRC | 08:38 | |
| *** asettle has joined #openstack-infra | 08:40 | |
| *** javeriak has quit IRC | 08:41 | |
| *** tqtran has joined #openstack-infra | 08:44 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Don't run undercloud idempotent on tempest jobs https://review.openstack.org/351066 | 08:47 |
| *** aviau has quit IRC | 08:47 | |
| *** aviau has joined #openstack-infra | 08:47 | |
| *** tqtran has quit IRC | 08:49 | |
| *** apetrich has quit IRC | 08:51 | |
| *** zz_dimtruck is now known as dimtruck | 08:52 | |
| *** Ahharu has joined #openstack-infra | 08:52 | |
| Ahharu | Hello | 08:52 |
| *** mhickey has joined #openstack-infra | 08:53 | |
| *** markvoelker has joined #openstack-infra | 08:53 | |
| Ahharu | strating today zuul cloner is not getting the right parameters on our module | 08:53 |
| Ahharu | it just gets this | 08:53 |
| Ahharu | 2016-08-04 08:48:58.090984 | + /usr/zuul-env/bin/zuul-cloner -m /tmp/tmp.ZnCQR3pEad | 08:53 |
| *** jtomasek has joined #openstack-infra | 08:53 | |
| Ahharu | even though we have it setup on the spec helper acceptance | 08:53 |
| *** dingyichen has quit IRC | 08:54 | |
| *** Na3iL has joined #openstack-infra | 08:55 | |
| *** markvoelker has quit IRC | 08:57 | |
| *** yamamoto_ has joined #openstack-infra | 08:59 | |
| *** thorst has joined #openstack-infra | 09:00 | |
| *** dimtruck is now known as zz_dimtruck | 09:01 | |
| *** yamamoto_ has quit IRC | 09:02 | |
| *** e0ne has quit IRC | 09:03 | |
| *** yamamoto has joined #openstack-infra | 09:03 | |
| *** e0ne has joined #openstack-infra | 09:04 | |
| *** afred312 has joined #openstack-infra | 09:04 | |
| *** amitgandhinz has joined #openstack-infra | 09:04 | |
| *** thorst has quit IRC | 09:06 | |
| *** yamamoto has quit IRC | 09:08 | |
| *** afred312 has quit IRC | 09:08 | |
| *** javeriak has joined #openstack-infra | 09:09 | |
| *** amitgandhinz has quit IRC | 09:09 | |
| *** gildub has quit IRC | 09:12 | |
| *** berendt has joined #openstack-infra | 09:17 | |
| *** _degorenko|afk is now known as degorenko | 09:22 | |
| *** dulek has quit IRC | 09:25 | |
| *** dulek has joined #openstack-infra | 09:27 | |
| *** skipp has quit IRC | 09:27 | |
| *** stewie925 has quit IRC | 09:28 | |
| *** eeiden has quit IRC | 09:28 | |
| *** eeiden has joined #openstack-infra | 09:31 | |
| *** markmcd has joined #openstack-infra | 09:34 | |
| *** electrofelix has joined #openstack-infra | 09:35 | |
| AJaeger | Ahharu: can you point me to job and logfile? | 09:37 |
| Ahharu | yep | 09:39 |
| *** apetrich has joined #openstack-infra | 09:39 | |
| Ahharu | https://review.openstack.org/#/c/350710/ | 09:39 |
| Ahharu | http://logs.openstack.org/10/350710/1/check/gate-puppet-midonet-puppet-beaker-rspec-centos-7/74ea382/console.html | 09:39 |
| Ahharu | and puppet swift for example > http://logs.openstack.org/90/350790/4/check/gate-puppet-swift-puppet-beaker-rspec-centos-7/1b945dd/console.html | 09:40 |
| AJaeger | Ahharu: I see the problem, working on a fix... | 09:41 |
| Ahharu | thanks! | 09:42 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add missing slash to puppet-module-jobs https://review.openstack.org/351086 | 09:42 |
| AJaeger | Ahharu: ^ | 09:42 |
| AJaeger | yolanda, jhesketh, ianw : If you're around, could you review this, please? | 09:43 |
| Ahharu | good ! | 09:43 |
| AJaeger | Ahharu: Sorry, I missed that in reviewing the original change. | 09:43 |
| yolanda | sure | 09:43 |
| *** gongysh has quit IRC | 09:44 | |
| AJaeger | thanks. | 09:44 |
| AJaeger | Ahharu: with current load of gate: might take two hours to get this merged and then max 1h to be deployed. | 09:46 |
| *** stewie925 has joined #openstack-infra | 09:46 | |
| *** markvoelker has joined #openstack-infra | 09:47 | |
| Ahharu | well .. i will do a workaround by copying my module with the changes for testing until then | 09:48 |
| Ahharu | :) | 09:48 |
| *** permalac has joined #openstack-infra | 09:49 | |
| *** markvoelker has quit IRC | 09:52 | |
| *** skipp has joined #openstack-infra | 09:53 | |
| *** yamamoto has joined #openstack-infra | 09:53 | |
| *** sambetts|afk is now known as sambetts | 09:55 | |
| *** nadya has joined #openstack-infra | 09:55 | |
| *** nadya is now known as _nadya_ | 09:56 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: grafana: Update ovn unit test jobs after xenial renames https://review.openstack.org/351019 | 09:57 |
| *** ams__ has joined #openstack-infra | 10:02 | |
| * AJaeger pushes 351086 through to unbreak puppet | 10:02 | |
| *** sarob has joined #openstack-infra | 10:03 | |
| *** zhurong has quit IRC | 10:04 | |
| *** afred312 has joined #openstack-infra | 10:04 | |
| *** amitgandhinz has joined #openstack-infra | 10:05 | |
| vponomaryov | who knows what have changed to trusty images? | 10:07 |
| *** sarob has quit IRC | 10:08 | |
| vponomaryov | apt-get upgrade falls into endless loop on update of grub that started making dialog | 10:08 |
| vponomaryov | like this - http://paste.openstack.org/show/547902/ | 10:09 |
| *** afred312 has quit IRC | 10:09 | |
| *** amitgandhinz has quit IRC | 10:10 | |
| *** ihrachys has quit IRC | 10:11 | |
| *** furlongm_ has joined #openstack-infra | 10:12 | |
| *** dtantsur|bbl is now known as dtantsur | 10:12 | |
| *** yaume has quit IRC | 10:13 | |
| *** kzaitsev_mb has joined #openstack-infra | 10:13 | |
| openstackgerrit | Merged openstack-infra/project-config: Add missing slash to puppet-module-jobs https://review.openstack.org/351086 | 10:20 |
| *** roxanaghe has joined #openstack-infra | 10:22 | |
| *** roxanaghe has quit IRC | 10:26 | |
| *** chlong has quit IRC | 10:28 | |
| *** yamamoto has quit IRC | 10:29 | |
| *** yamamoto has joined #openstack-infra | 10:30 | |
| *** jimbaker has quit IRC | 10:30 | |
| *** thorst has joined #openstack-infra | 10:31 | |
| *** amotoki has quit IRC | 10:31 | |
| *** tosky has joined #openstack-infra | 10:31 | |
| *** yamamoto has quit IRC | 10:32 | |
| *** ihrachys has joined #openstack-infra | 10:33 | |
| *** yanyanhu has quit IRC | 10:33 | |
| *** jimbaker has joined #openstack-infra | 10:34 | |
| *** jimbaker has quit IRC | 10:34 | |
| *** jimbaker has joined #openstack-infra | 10:34 | |
| *** thorst has quit IRC | 10:36 | |
| AJaeger | vponomaryov: was there an upstream update of grub-common ? I suggest to check the history of that package... | 10:36 |
| AJaeger | Or of grub-pc... | 10:36 |
| *** kzaitsev_mb has quit IRC | 10:37 | |
| *** markvoelker has joined #openstack-infra | 10:41 | |
| vponomaryov | AJaeger: tried just to avoid its update like here - https://review.openstack.org/#/c/351110/ | 10:41 |
| vponomaryov | AJaeger: will see whether it helps or not | 10:41 |
| *** chlong has joined #openstack-infra | 10:41 | |
| *** csomerville has joined #openstack-infra | 10:43 | |
| *** cody-somerville has quit IRC | 10:43 | |
| AJaeger | I suggest to inform upstream about this if that fixes the issue. | 10:44 |
| *** apetrich has quit IRC | 10:45 | |
| *** markvoelker has quit IRC | 10:46 | |
| vponomaryov | AJaeger: upstream of what? | 10:46 |
| vponomaryov | AJaeger: grub project upstream? | 10:46 |
| *** gildub has joined #openstack-infra | 10:47 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/project-config: Return tripleo tempest job to experimental pipe https://review.openstack.org/351120 | 10:48 |
| AJaeger | vponomaryov: That's an Ubuntu package - os upstream Ubuntu | 10:48 |
| vponomaryov | https://launchpad.net/ubuntu/trusty/amd64/grub-common/ | 10:48 |
| *** gongysh has joined #openstack-infra | 10:48 | |
| AJaeger | yeah, released 4 hours ago ;( | 10:49 |
| *** Ahharu has quit IRC | 10:49 | |
| *** kaisers has quit IRC | 10:50 | |
| *** kaisers has joined #openstack-infra | 10:51 | |
| *** kaisers has quit IRC | 10:51 | |
| *** kzaitsev_mb has joined #openstack-infra | 10:53 | |
| *** kaisers has joined #openstack-infra | 10:57 | |
| *** ihrachys has quit IRC | 11:02 | |
| sshnaidm | jeblair, hi, we talked recently about new gear release, just curious if everybody agreed to this and we can proceed? thanks! | 11:03 |
| *** thorst has joined #openstack-infra | 11:03 | |
| *** Ravikiran_K has quit IRC | 11:04 | |
| *** afred312 has joined #openstack-infra | 11:05 | |
| *** amitgandhinz has joined #openstack-infra | 11:06 | |
| *** rbuzatu has quit IRC | 11:06 | |
| *** ihrachys has joined #openstack-infra | 11:06 | |
| *** rbuzatu has joined #openstack-infra | 11:09 | |
| *** jlanoux has quit IRC | 11:09 | |
| *** afred312 has quit IRC | 11:10 | |
| *** bhavik1 has quit IRC | 11:10 | |
| *** amitgandhinz has quit IRC | 11:10 | |
| AJaeger | EmilienM: your change is still broken, see http://logs.openstack.org/10/350710/1/check/gate-puppet-midonet-puppet-beaker-rspec-centos-7/cfd3bad/console.html | 11:12 |
| *** mat128|afk is now known as mat128 | 11:13 | |
| *** Mmike has quit IRC | 11:16 | |
| *** Mmike has joined #openstack-infra | 11:17 | |
| *** Mmike has joined #openstack-infra | 11:17 | |
| *** amotoki has joined #openstack-infra | 11:17 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Fix puppet-openstack-integration-checkout https://review.openstack.org/351127 | 11:18 |
| AJaeger | yolanda: we overlooked another issue in EmilienM's change ;( WE should have reverted instead - please check that my fix is really good, otherwise I'm for reverting. | 11:19 |
| yolanda | ah, i misunderstood the logic there, but yes, that fix makes sense | 11:21 |
| *** sdague has joined #openstack-infra | 11:21 | |
| *** lucasagomes is now known as lucas-hungry | 11:21 | |
| AJaeger | yolanda: so did I ;( | 11:22 |
| *** yamamoto has joined #openstack-infra | 11:23 | |
| yolanda | i gave +2, the shame is that it will take long to merge, right? | 11:23 |
| AJaeger | yolanda: last one took less than an hour to merge. Let me self-approve again... | 11:25 |
| *** chlong has quit IRC | 11:27 | |
| *** chlong has joined #openstack-infra | 11:28 | |
| *** sshnaidm is now known as sshnaidm|lnch | 11:31 | |
| *** ldnunes has joined #openstack-infra | 11:34 | |
| *** kzaitsev_mb has quit IRC | 11:34 | |
| *** ericksonsantos has quit IRC | 11:34 | |
| *** rhallisey has joined #openstack-infra | 11:34 | |
| *** raildo has quit IRC | 11:34 | |
| *** markvoelker has joined #openstack-infra | 11:35 | |
| *** jlanoux has joined #openstack-infra | 11:35 | |
| *** gordc has joined #openstack-infra | 11:39 | |
| *** markvoelker has quit IRC | 11:39 | |
| *** jkilpatr has joined #openstack-infra | 11:40 | |
| *** dulek has quit IRC | 11:43 | |
| *** dulek has joined #openstack-infra | 11:45 | |
| *** acoles is now known as acoles_ | 11:49 | |
| *** amotoki has quit IRC | 11:50 | |
| openstackgerrit | Tang Chen proposed openstack-infra/project-config: Export openstack/ironic/ to $PROJECT https://review.openstack.org/351138 | 11:52 |
| EmilienM | AJaeger: good morning | 11:53 |
| EmilienM | AJaeger: and thank you :) | 11:53 |
| AJaeger | morning, EmilienM. We were a bit eager merging one of your changes and overlooked a few things, hope that 351127 fixes everything. | 11:53 |
| *** psilvad has joined #openstack-infra | 11:54 | |
| EmilienM | AJaeger: ok I'll watch is closely this morning | 11:54 |
| AJaeger | please slowly recheck once that is in - we have a long queue and don't want endless rechecks (so, recheck one change and wait for results, please) | 11:54 |
| *** weshay has joined #openstack-infra | 11:56 | |
| EmilienM | AJaeger: sure thing | 11:56 |
| *** asettle has quit IRC | 11:57 | |
| *** hieulq_ has joined #openstack-infra | 11:57 | |
| *** asettle has joined #openstack-infra | 11:57 | |
| *** lucas-hungry is now known as lucasagomes | 11:58 | |
| *** gouthamr has joined #openstack-infra | 12:01 | |
| *** asettle has quit IRC | 12:02 | |
| *** eharney has quit IRC | 12:02 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton https://review.openstack.org/346995 | 12:05 |
| *** markvoelker has joined #openstack-infra | 12:05 | |
| *** afred312 has joined #openstack-infra | 12:06 | |
| *** amitgandhinz has joined #openstack-infra | 12:06 | |
| *** zul has quit IRC | 12:07 | |
| *** sshnaidm|lnch is now known as sshnaidm | 12:08 | |
| *** hieulq_ has quit IRC | 12:09 | |
| *** roxanaghe has joined #openstack-infra | 12:10 | |
| *** afred312 has quit IRC | 12:11 | |
| *** amitgandhinz has quit IRC | 12:11 | |
| *** zhurong has joined #openstack-infra | 12:12 | |
| *** apetrich has joined #openstack-infra | 12:13 | |
| *** tpsilva has joined #openstack-infra | 12:14 | |
| *** roxanaghe has quit IRC | 12:15 | |
| *** Goneri has quit IRC | 12:17 | |
| *** baoli has joined #openstack-infra | 12:20 | |
| *** zul has joined #openstack-infra | 12:20 | |
| *** dkranz has joined #openstack-infra | 12:22 | |
| *** eharney has joined #openstack-infra | 12:25 | |
| *** Goneri has joined #openstack-infra | 12:25 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix puppet-openstack-integration-checkout https://review.openstack.org/351127 | 12:25 |
| *** rfolco has joined #openstack-infra | 12:25 | |
| *** Srinu has quit IRC | 12:25 | |
| *** kzaitsev_mb has joined #openstack-infra | 12:25 | |
| *** baoli_ has joined #openstack-infra | 12:25 | |
| *** asettle has joined #openstack-infra | 12:26 | |
| *** xyang1 has joined #openstack-infra | 12:27 | |
| *** dprince has joined #openstack-infra | 12:28 | |
| *** baoli has quit IRC | 12:29 | |
| pabelanger | morning | 12:30 |
| pabelanger | AJaeger: Do we want to remove gate-{name}-tox-{envlist} and replace it with gate-{name}-tox-{envlist}-{node}? | 12:30 |
| AJaeger | pabelanger: that'S my understanding. | 12:31 |
| pabelanger | AJaeger: okay, let me do that this morning | 12:31 |
| AJaeger | pabelanger: There have been so far only some steps done in this direction but not the complete move... | 12:31 |
| *** ramishra has quit IRC | 12:31 | |
| AJaeger | pabelanger: thanks! | 12:31 |
| *** ramishra has joined #openstack-infra | 12:33 | |
| openstackgerrit | Merged openstack-infra/system-config: Switch the floating ip source to neutron for rh1 https://review.openstack.org/348778 | 12:33 |
| *** pradk_ has quit IRC | 12:35 | |
| *** kgiusti has joined #openstack-infra | 12:41 | |
| *** amotoki has joined #openstack-infra | 12:43 | |
| *** tqtran has joined #openstack-infra | 12:45 | |
| *** burgerk has joined #openstack-infra | 12:48 | |
| *** raildo has joined #openstack-infra | 12:48 | |
| EmilienM | AJaeger: it seems to work, I continue to check | 12:48 |
| AJaeger | EmilienM: so, first success? | 12:49 |
| *** psilvad has quit IRC | 12:50 | |
| EmilienM | AJaeger: yes :) | 12:50 |
| *** tqtran has quit IRC | 12:50 | |
| AJaeger | great! | 12:50 |
| *** dkranz has quit IRC | 12:50 | |
| AJaeger | yeah, looks like we're fine now... | 12:51 |
| *** zz_dimtruck is now known as dimtruck | 12:53 | |
| Zara | someone posted a story, 'zuul consumes too much memory' and has asked if there are any updates. I'm not sure if people are subscribed to it so I'm acting as storyboardbot just in case. https://storyboard.openstack.org/#!/story/2000640 | 12:53 |
| *** wanghua has joined #openstack-infra | 12:53 | |
| *** javeriak has quit IRC | 12:54 | |
| *** dtantsur is now known as dtantsur|mtg | 12:54 | |
| wanghua | Hi all, as I know, zuul merger prepares a tmp repo for zuul cloner, why zuul cloner still need git_base_url? I think zuul cloner can clone the repo from the zuul merger directly. Can anyone help me? | 12:54 |
| *** psilvad has joined #openstack-infra | 12:55 | |
| *** rlandy has joined #openstack-infra | 12:56 | |
| EmilienM | AJaeger: I still have my problem with puppet-ceph though (not critical at this time) but I'm working on it | 12:56 |
| *** jsavak has joined #openstack-infra | 12:56 | |
| *** gongysh has quit IRC | 12:56 | |
| *** javeriak has joined #openstack-infra | 12:57 | |
| openstackgerrit | Merged openstack-infra/ciwatch: Remove discover from test-requirements https://review.openstack.org/345793 | 12:58 |
| *** psilvad has quit IRC | 13:00 | |
| *** jpich has quit IRC | 13:03 | |
| *** dimtruck is now known as zz_dimtruck | 13:03 | |
| *** pradk has quit IRC | 13:04 | |
| *** psilvad has joined #openstack-infra | 13:04 | |
| *** afred312 has joined #openstack-infra | 13:07 | |
| *** amitgandhinz has joined #openstack-infra | 13:07 | |
| *** zul has quit IRC | 13:07 | |
| *** mdrabe has joined #openstack-infra | 13:09 | |
| *** sdake has joined #openstack-infra | 13:11 | |
| *** afred312 has quit IRC | 13:11 | |
| *** amitgandhinz has quit IRC | 13:12 | |
| openstackgerrit | Ivan Kolodyazhny proposed openstack-infra/project-config: Disable test_volume_from_snapshot tests for Cinder BDD job https://review.openstack.org/351178 | 13:13 |
| *** markvoelker has quit IRC | 13:13 | |
| *** matt-borland has joined #openstack-infra | 13:15 | |
| *** gildub has quit IRC | 13:16 | |
| *** Liuqing has joined #openstack-infra | 13:17 | |
| *** amitgandhinz has joined #openstack-infra | 13:17 | |
| *** ihrachys has quit IRC | 13:18 | |
| *** zul has joined #openstack-infra | 13:22 | |
| *** acoles_ is now known as acoles | 13:22 | |
| *** timl0101 has quit IRC | 13:22 | |
| *** sdake has quit IRC | 13:24 | |
| *** thiagop has joined #openstack-infra | 13:25 | |
| *** amitgandhinz has quit IRC | 13:27 | |
| *** amitgandhinz has joined #openstack-infra | 13:28 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add ability to upload duplicate images https://review.openstack.org/349710 | 13:28 |
| *** matrohon has joined #openstack-infra | 13:30 | |
| *** julim has joined #openstack-infra | 13:31 | |
| asselin | pabelanger, I'm reviewing https://review.openstack.org/#/c/333532/1. Is this expected output? http://paste.openstack.org/show/549229/ | 13:32 |
| *** _ari_ has joined #openstack-infra | 13:32 | |
| EmilienM | AJaeger: ok so I confirm that the problem is still here for puppet-ceph only though | 13:33 |
| pabelanger | asselin: Yup, means you don't have puppet installed locally | 13:33 |
| EmilienM | I'm trying to undercloud why zuul-cloner doesn't checkout the repo at the commit in Depends-On | 13:33 |
| EmilienM | http://logs.openstack.org/11/350511/2/check/gate-puppet-openstack-integration-3-scenario001-tempest-centos-7/273ff5b/console.html#_2016-08-04_13_03_57_923133 | 13:33 |
| EmilienM | the patch is here: https://review.openstack.org/#/c/350511/ | 13:33 |
| *** Jeffrey4l_ has quit IRC | 13:34 | |
| *** esberglu has joined #openstack-infra | 13:35 | |
| *** hieulq_ has joined #openstack-infra | 13:35 | |
| anteaya | EmilienM: the job prepared the parent of the ceph backport patch, which is the depends on | 13:37 |
| *** timl0101 has joined #openstack-infra | 13:38 | |
| EmilienM | anteaya: c13a4187aa2725d593f6e3f1c752fd1550665465 is the latest patch in stable/mitaka, no the commit id in dependson | 13:38 |
| EmilienM | so the depends-on is not adressed | 13:38 |
| anteaya | EmilienM: ah sorry | 13:39 |
| EmilienM | anteaya: no prob | 13:39 |
| EmilienM | thx for looking | 13:39 |
| *** mriedem has joined #openstack-infra | 13:39 | |
| anteaya | welcome, still looking | 13:40 |
| *** hieulq_ has quit IRC | 13:41 | |
| *** burgerk has quit IRC | 13:41 | |
| asselin | pabelanger, what exactly is it looking for? I have puppet installed: http://paste.openstack.org/show/549230/ | 13:41 |
| *** acoles is now known as acoles_ | 13:44 | |
| *** jpich has joined #openstack-infra | 13:45 | |
| *** EricGonczer_ has joined #openstack-infra | 13:45 | |
| *** aeng has joined #openstack-infra | 13:46 | |
| *** jraju has quit IRC | 13:46 | |
| *** yamamoto has quit IRC | 13:46 | |
| anteaya | EmilienM: looks to me like zuul cloner is looking for a stable/hammer branch in puppet-openstack-integration and not finding such a branch: http://logs.openstack.org/11/350511/2/check/gate-puppet-openstack-integration-3-scenario001-tempest-centos-7/273ff5b/console.html#_2016-08-04_13_07_24_826799 | 13:47 |
| EmilienM | anteaya: in fact, we force it to stable/mitaka https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/puppet-module-jobs.yaml#L11-L13 | 13:48 |
| *** matrohon has quit IRC | 13:48 | |
| *** yamamoto has joined #openstack-infra | 13:49 | |
| Zara | hm, question-- I want to use review-dev so I can see how gerrit-storyboard integration is doing and test things (I think we need to set an additional parameter in some config first, but I'm not sure, and I'd rather be able to check before I ask about things). I think the `git review` command will default to talking to review.o.o. so how would I go about talking to review-dev? | 13:49 |
| *** javeriak has quit IRC | 13:49 | |
| *** ddieterly has joined #openstack-infra | 13:50 | |
| *** pgadiya has quit IRC | 13:51 | |
| *** markvoelker has joined #openstack-infra | 13:51 | |
| *** javeriak has joined #openstack-infra | 13:52 | |
| mordred | Zara: for the second thing, the .gitreview file in the git repo points to review.openstack.org which will set up the 'gerrit' remote in git | 13:53 |
| *** mmandel has joined #openstack-infra | 13:53 | |
| *** sdake has joined #openstack-infra | 13:54 | |
| Zara | mordred: ahhhh, right, I see | 13:54 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/infra-manual: Clarify post job log URLs for merge commits https://review.openstack.org/351199 | 13:54 |
| mordred | Zara: there are a couple of projects that only exist in review-dev that have their .gitreview set up ... but if you want to use a project that is in both, you can edit the remote entry in .git/config | 13:55 |
| Zara | okay, thanks. I may just be lazy and clone again so I don't forget where I'm sending things. :) | 13:56 |
| *** yamamoto has quit IRC | 13:56 | |
| *** rbuzatu_ has joined #openstack-infra | 13:57 | |
| anteaya | EmilienM: I'm questioning whether or not it is accepting the force | 13:57 |
| *** PollyZ has joined #openstack-infra | 13:58 | |
| *** xarses has quit IRC | 13:58 | |
| *** hieulq_ has joined #openstack-infra | 13:58 | |
| *** baoli_ has quit IRC | 13:59 | |
| *** shamail has joined #openstack-infra | 14:00 | |
| *** rbuzatu has quit IRC | 14:00 | |
| *** hieulq_ has quit IRC | 14:00 | |
| *** ddieterly has quit IRC | 14:01 | |
| *** ddieterly has joined #openstack-infra | 14:01 | |
| *** vincentll has quit IRC | 14:01 | |
| *** vincentll has joined #openstack-infra | 14:01 | |
| *** PollyZ has quit IRC | 14:02 | |
| *** edmondsw has quit IRC | 14:02 | |
| *** ZZelle has quit IRC | 14:03 | |
| *** PollyZ has joined #openstack-infra | 14:03 | |
| *** rbuzatu_ has quit IRC | 14:04 | |
| *** rbuzatu has joined #openstack-infra | 14:05 | |
| *** matrohon has joined #openstack-infra | 14:05 | |
| *** matrohon has quit IRC | 14:05 | |
| openstackgerrit | Brad P. Crochet proposed openstack-infra/tripleo-ci: Use tripleo-build-images for CI https://review.openstack.org/336312 | 14:06 |
| *** pradk has joined #openstack-infra | 14:06 | |
| *** zul has quit IRC | 14:07 | |
| *** zul has joined #openstack-infra | 14:07 | |
| *** rbrndt has joined #openstack-infra | 14:07 | |
| *** afred312 has joined #openstack-infra | 14:07 | |
| *** coolsvap has quit IRC | 14:11 | |
| AJaeger | fungi, do you have time today to look at constraints and release/post queue, please? https://review.openstack.org/#/c/349627/ is the review | 14:11 |
| *** afred312 has quit IRC | 14:12 | |
| *** pcrews has quit IRC | 14:12 | |
| *** apetrich has quit IRC | 14:12 | |
| *** rockyg has joined #openstack-infra | 14:12 | |
| *** pcrews has joined #openstack-infra | 14:14 | |
| EmilienM | AJaeger: do you have an idea why zuul-cloner checkout the last commit in the branch and not the commit in depends-on in http://logs.openstack.org/11/350511/2/check/gate-puppet-openstack-integration-3-scenario001-tempest-centos-7/273ff5b/console.html#_2016-08-04_13_03_57_923133 ? | 14:14 |
| *** jsavak has quit IRC | 14:14 | |
| *** dtantsur|mtg is now known as dtantsur | 14:14 | |
| AJaeger | EmilienM: not right now, that needs some more digging than I can do right now. | 14:15 |
| AJaeger | EmilienM: the --branch parameter might be the problem... | 14:16 |
| EmilienM | AJaeger: ok | 14:16 |
| *** aeng has quit IRC | 14:16 | |
| fungi | EmilienM: the branch names don't seem to match? | 14:17 |
| fungi | you have a "stable/hammer" change depending on a "stable/mitaka" change | 14:17 |
| fungi | zuul-cloner looks for matching branch names | 14:17 |
| fungi | i think that will be very hard to work around | 14:17 |
| *** jaosorior has quit IRC | 14:17 | |
| *** javeriak has quit IRC | 14:18 | |
| EmilienM | fungi: yes, our puppet-ceph module has a stable/hammer branch that require to work with stable/mitaka | 14:18 |
| fungi | i don't think we have any good solutions to let you integration test that without writing separate integration test tooling | 14:19 |
| EmilienM | ok | 14:19 |
| *** apetrich has joined #openstack-infra | 14:20 | |
| fungi | using a stable/mitaka branch on puppet-ceph too would make the current model feasible, as would just using the master branch of puppet-ceph, but i'm guessing those options are incompatible with your release model for that repo | 14:21 |
| *** aeng has joined #openstack-infra | 14:21 | |
| *** matrohon has joined #openstack-infra | 14:22 | |
| *** PollyZ has quit IRC | 14:22 | |
| *** amotoki has quit IRC | 14:22 | |
| *** PollyZ has joined #openstack-infra | 14:23 | |
| *** signed8bit has joined #openstack-infra | 14:24 | |
| greghaynes | cinerama: ianw You all ok with https://review.openstack.org/#/c/350997/3 ? | 14:24 |
| *** afred312 has joined #openstack-infra | 14:24 | |
| *** aeng has quit IRC | 14:25 | |
| *** zul has quit IRC | 14:25 | |
| *** zul has joined #openstack-infra | 14:25 | |
| *** gongysh has joined #openstack-infra | 14:27 | |
| Zara | hm, actually, are there projects that are in both review-dev and review.o.o? I looked for a dummy project to send commits to to start with, and couldn't find overlap. looked in git.openstack.org and couldn't see anything that matched up with a project in review-dev; tried cloning from review-dev, but I got 'server certificate verification failed'. I'm not sure how to add a project to it, if I need to | 14:27 |
| Zara | do that. | 14:27 |
| *** burgerk has joined #openstack-infra | 14:28 | |
| *** mtanino has joined #openstack-infra | 14:31 | |
| *** zz_dimtruck is now known as dimtruck | 14:31 | |
| *** vincentll has quit IRC | 14:32 | |
| *** gongysh has quit IRC | 14:32 | |
| *** matrohon has quit IRC | 14:32 | |
| anteaya | Zara: I think you need to be a gerrit admin to be able to add a project to our review-dev | 14:32 |
| pabelanger | asselin: it looks at the local package manager IIRC, how did you install puppet? | 14:33 |
| anteaya | can you get what you need with just being able to add a patch to a current project? | 14:33 |
| anteaya | or do you need to add a project? | 14:33 |
| fungi | (or add a corresponding project to storyboard-dev?) | 14:33 |
| asselin | pabelanger, don't remember...but most likely with install_puppet.sh | 14:33 |
| *** mdrabe has quit IRC | 14:34 | |
| EmilienM | fungi: right, it make things weird, as Ceph & OpenStack don't have the same release cycles | 14:34 |
| *** javeriak has joined #openstack-infra | 14:34 | |
| Zara | just adding a patch should be enough to begin with, I just couldn't find a way to clone a project to send patches for it. Ideally I'd like a project in there with a corresponding storyboard-dev project. | 14:34 |
| *** Goneri has quit IRC | 14:35 | |
| Zara | (though I think the config to say 'update this storyboard project' is a matter for a future patch, but yeah, that's something I want to find out by having a wander around) | 14:35 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Migate tox to node https://review.openstack.org/351219 | 14:35 |
| pabelanger | asselin: dpkg -l | grep puppet | 14:36 |
| pabelanger | asselin: anything listed? | 14:36 |
| asselin | pabelanger, http://paste.openstack.org/show/549241/ | 14:37 |
| pabelanger | asselin: ya, you don't have puppet install | 14:38 |
| pabelanger | the puppet package | 14:38 |
| *** asettle has quit IRC | 14:38 | |
| pabelanger | you mind have the binary | 14:38 |
| pabelanger | what version of ubuntu are you using? | 14:38 |
| *** asettle has joined #openstack-infra | 14:39 | |
| *** xarses has joined #openstack-infra | 14:39 | |
| pabelanger | Hmm | 14:39 |
| pabelanger | actually | 14:39 |
| pabelanger | http://packages.ubuntu.com/xenial/all/puppet-common/filelist | 14:39 |
| asselin | pabelanger, ok thanks for clarifying. I see it here in a test-vm, so my error. http://paste.openstack.org/show/549242/ | 14:39 |
| *** mdrabe has joined #openstack-infra | 14:39 | |
| pabelanger | asselin: looks like we should be using puppet-common for ubuntu | 14:39 |
| asselin | i'm on trusty | 14:39 |
| pabelanger | but puppet package has a dependency on puppet-common | 14:40 |
| pabelanger | asselin: ya, I would expect it to work in your test-vm | 14:40 |
| pabelanger | since puppet package is listed | 14:40 |
| asselin | ok, so just an issue w/ my workstation | 14:40 |
| asselin | so fails correctly | 14:41 |
| anteaya | Zara: try some version of git clone ssh://username@review-dev.openstack.org:29418/Project-Foo && cd Project-Foo && git fetch origin refs/meta/config && git checkout FETCH_HEAD | 14:42 |
| *** jcoufal has joined #openstack-infra | 14:45 | |
| *** Jeffrey4l has joined #openstack-infra | 14:45 | |
| *** mhayden has joined #openstack-infra | 14:46 | |
| mhayden | fungi: are you the right person to ask about bindep releases? | 14:47 |
| *** Liuqing has quit IRC | 14:47 | |
| fungi | mhayden: yeah, did i break something with 2.0.1 yesterday? | 14:47 |
| *** Liuqing has joined #openstack-infra | 14:48 | |
| mhayden | ah, i must have missed the release :) | 14:48 |
| *** matrohon has joined #openstack-infra | 14:49 | |
| fungi | it was a minor bugfix release | 14:49 |
| fungi | a handful of cosmetic patches and one change allowing _ in package names (which seems to be an rpmism) | 14:49 |
| mhayden | ah, the package name one the patch i was concerned about :) | 14:49 |
| mhayden | wow that was terrible grammar | 14:50 |
| mhayden | perhaps i need more coffee | 14:50 |
| fungi | if our image uploads succeed today, we should have 2.0.1 in the bindep virtualenv on them | 14:50 |
| *** berendt has quit IRC | 14:50 | |
| fungi | and then any jobs trying to introduce package names with an underscore should pass | 14:51 |
| fungi | er, any changes | 14:51 |
| mhayden | that's good news for me -- thanks, fungi | 14:51 |
| *** Teresita-Warrior has joined #openstack-infra | 14:51 | |
| jeblair | EmilienM: something like this approach may work: http://paste.openstack.org/show/549244/ (cc: fungi) | 14:53 |
| *** javeriak has quit IRC | 14:53 | |
| EmilienM | jeblair: https://review.openstack.org/#/c/350845/ | 14:54 |
| EmilienM | that's what I did :) | 14:54 |
| *** edmondsw has joined #openstack-infra | 14:54 | |
| fungi | jeblair: EmilienM: ahh, yep that's similar i think to what gnocchi does for their nonstandard stable branches too | 14:54 |
| *** vincentll has joined #openstack-infra | 14:54 | |
| fungi | it requires maintaining a branch mapping, so not entirely automagic, but better than nothing | 14:55 |
| *** karthikp_ has joined #openstack-infra | 14:55 | |
| *** annegentle has joined #openstack-infra | 14:55 | |
| fungi | also tricky if you're wanting to mix and match branch names in one zuul-cloner call | 14:55 |
| jeblair | EmilienM: cool. so if you stuck the --project-branch argument on the first zuul-cloner command, you can drop the second one, and it should make Ivan happy too :) | 14:55 |
| *** vvalderrv has joined #openstack-infra | 14:55 | |
| fungi | since ZUUL_BRANCH can only ever have one value per zuul-cloner call | 14:56 |
| jeblair | fungi: right, but --project-branch can have any number | 14:56 |
| jeblair | (you can add multiple --project-branch arguments) | 14:56 |
| EmilienM | oh wouah | 14:56 |
| EmilienM | I can do that? | 14:56 |
| EmilienM | and it will try all of them in a specific order? | 14:57 |
| fungi | true. you just need to pick one to be your $ZUUL_BRANCH default | 14:57 |
| jeblair | EmilienM: look closely at the last line of the paste http://paste.openstack.org/show/549244/ | 14:57 |
| fungi | so preferably whichever branch is used by the greatest number of changes | 14:57 |
| *** yamamoto has joined #openstack-infra | 14:57 | |
| EmilienM | jeblair: ok, I didn't know this feature | 14:58 |
| jeblair | (i commented on 350845) | 14:58 |
| EmilienM | jeblair: have you seen this workaround ? https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/puppet-module-jobs.yaml#L7-L28 | 14:58 |
| EmilienM | jeblair: we also need that to checkout puppet-openstack-integration repo to the right branch but it doesn't checkout the commit in DependsOn | 14:59 |
| *** annegentle has quit IRC | 15:00 | |
| jeblair | yeah, i saw that in the log; i think that's probably inevitable | 15:00 |
| EmilienM | ok | 15:01 |
| EmilienM | so when we'll patch puppet-ceph stable/hammer, we won't be able to have a Depends-On in puppet-openstack-integration, that's all. Right? | 15:01 |
| * EmilienM making sure to understand :-) | 15:01 | |
| jeblair | i think if you make the mapping bi-directional, and have it in both places, it should work | 15:02 |
| Zara | anteaya: added ssh keys to review-dev (same ones as on review.o.o, so they should work) and tried that approach for the clone. still getting 'permission denied (publickey)'. I noticed the 'download' link had the username set to 'null', so I tried that, too, but it didn't work. | 15:02 |
| EmilienM | jeblair: I tested it and it didn't work | 15:02 |
| pabelanger | EmilienM: I would think it works, if you updated your zuul-cloner logic within puppet-openstack-integration | 15:02 |
| EmilienM | jeblair: http://logs.openstack.org/11/350511/2/check/gate-puppet-openstack-integration-3-scenario001-tempest-centos-7/273ff5b/console.html#_2016-08-04_13_03_57_923133 | 15:02 |
| anteaya | Zara: :( | 15:02 |
| anteaya | Zara: is your username the same on review and review-dev? | 15:03 |
| pabelanger | EmilienM: what commit are you expecting? | 15:03 |
| EmilienM | pabelanger: I think the problem is not in p-o-i now, see the log | 15:03 |
| jeblair | EmilienM: to clarify: have the mapping support both 'stable/mitaka' and 'stable/hammer' as input, and put it both in the puppet-openstack-integration-checkout and the main checkout... and do the zuul-cloner stuff like in http://paste.openstack.org/show/549244/ i think it should work | 15:03 |
| EmilienM | so there is "Depends-On: I004ee95e6bd97992aeebd08447396e3b8a29e09f" in https://review.openstack.org/#/c/350511/ | 15:03 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Add docs for firehose.o.o https://review.openstack.org/350764 | 15:03 |
| EmilienM | the parent is c13a4187aa2725d593f6e3f1c752fd1550665465 (and that's what is checked out) but the actual commit I wanted is a83add1a2d401c24ec8f8a419b26d0618620d550 | 15:03 |
| *** acoles_ is now known as acoles | 15:04 | |
| *** rockyg has quit IRC | 15:04 | |
| Zara | yes, but hm, for some reason it doesn't associate the ICLA | 15:04 |
| Zara | not sure if that might cause it or not. | 15:04 |
| anteaya | hmmm, review-dev shouldn't block you on the icla | 15:04 |
| Zara | yeah | 15:04 |
| pabelanger | EmilienM: right, I think what me and jeblair are saying. in the line you lined, you are not cloning puppet-ceph, you are only cloning puppet-opentack-integration. I believe you have logic in puppet-openstack-integraton to clone your other puppet modules | 15:04 |
| anteaya | it might, but if it does it shouldn't | 15:04 |
| *** Liuqing has quit IRC | 15:05 | |
| EmilienM | pabelanger: ok | 15:05 |
| pabelanger | EmilienM: isn't http://logs.openstack.org/11/350511/2/check/gate-puppet-openstack-integration-3-scenario001-tempest-centos-7/273ff5b/console.html#_2016-08-04_13_06_41_936149 the commit you are looking for? | 15:05 |
| *** rockyg has joined #openstack-infra | 15:05 | |
| *** Liuqing has joined #openstack-infra | 15:05 | |
| *** yamamoto has quit IRC | 15:05 | |
| fungi | Zara: anteaya: it depends on what the project is configured for. some of them were set for testing cla integration (and we deploy a dummy foundation api server on review-dev in the form of a cgi that always returns the desired result) | 15:05 |
| EmilienM | pabelanger: it is but not http://logs.openstack.org/11/350511/2/check/gate-puppet-openstack-integration-3-scenario001-tempest-centos-7/273ff5b/console.html#_2016-08-04_13_03_57_923133 | 15:06 |
| EmilienM | pabelanger: ok I'll give a try, maybe I'll use some help :) | 15:06 |
| Zara | oh, actually, hang on, it's not my username, I thought that was full name but apparently it's not the same and it wasn't set | 15:06 |
| fungi | Zara: anteaya: should just be able to click through the cla agreement and add dummy contact info if you're using one that has cla enforcement set | 15:06 |
| *** moravec has quit IRC | 15:07 | |
| *** oanson has quit IRC | 15:07 | |
| EmilienM | jeblair: iiuc, I need http://paste.openstack.org/show/549244/ logic on both sides | 15:07 |
| *** vhosakot has joined #openstack-infra | 15:07 | |
| *** moravec has joined #openstack-infra | 15:07 | |
| Zara | ugh, there we go, sorry for noise. thanks. | 15:07 |
| anteaya | Zara: fungi http://paste.openstack.org/show/549246/ works for me | 15:07 |
| *** akshai has joined #openstack-infra | 15:07 | |
| anteaya | Zara: ah good, glad you are on your way | 15:07 |
| *** _nadya_ has quit IRC | 15:08 | |
| anteaya | fungi: thank you, I didn't know that | 15:08 |
| *** EricGonczer_ has quit IRC | 15:08 | |
| Zara | (yeah, it needed me to manually set the username, but it was displaying 'Zara' in the top-right and as my full name so I thought it was in there.) | 15:08 |
| anteaya | ah | 15:08 |
| *** apetrich has quit IRC | 15:09 | |
| *** rcernin has quit IRC | 15:09 | |
| *** EricGonczer_ has joined #openstack-infra | 15:09 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Replace gate-{name}-tox-{envlist} with gate-{name}-tox-{envlist}-{node} https://review.openstack.org/351219 | 15:12 |
| *** akshai has quit IRC | 15:12 | |
| pabelanger | AJaeger: ^ I think got everything in that | 15:12 |
| pabelanger | AJaeger: too longer then expected | 15:12 |
| *** vincentll has quit IRC | 15:13 | |
| *** psachin has quit IRC | 15:13 | |
| *** mhickey has quit IRC | 15:13 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Setup constraints for release https://review.openstack.org/349627 | 15:13 |
| AJaeger | pabelanger: Thanks! Will review later today | 15:13 |
| AJaeger | fungi, corvus: I've updated 349627 addressing fungi's suggestions. Could you review again, please? | 15:14 |
| AJaeger | pabelanger: only 10 conflicts for your change - that's good ;) | 15:15 |
| *** e0ne has quit IRC | 15:15 | |
| pabelanger | AJaeger: I see some issues, pushing up a new patch | 15:15 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Replace gate-{name}-tox-{envlist} with gate-{name}-tox-{envlist}-{node} https://review.openstack.org/351219 | 15:16 |
| *** ayoung has joined #openstack-infra | 15:17 | |
| AJaeger | pabelanger: I just commented on zuul/layout - there're a few places that have branch conditions to run only on master or future stable branches - for those we could use only xenial and no trusty. | 15:18 |
| EmilienM | jeblair: is it --project-branch puppet-ceph=$BRANCH or --project-branch openstack/puppet-ceph=$BRANCH ? | 15:18 |
| *** yaume has joined #openstack-infra | 15:18 | |
| EmilienM | jeblair: also, in <big list of projects>: should I remove puppet-ceph from it? or it's fine to keep it? | 15:18 |
| AJaeger | pabelanger: So, If you want to remove some of these, have a look - note that I wouldn't do it when it means splitting up templates! | 15:19 |
| AJaeger | pabelanger: Will review later again, now cycling home... The general approach is fine! | 15:19 |
| *** baoli has joined #openstack-infra | 15:19 | |
| fungi | AJaeger: thanks, having another look | 15:19 |
| jeblair | EmilienM: it's --project-branch openstack/puppet-ceph=$BRANCH and keep it in the big list | 15:19 |
| EmilienM | jeblair: ack, thanks a lot for your help | 15:20 |
| jeblair | n4 | 15:20 |
| jeblair | np | 15:20 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Replace gate-{name}-tox-{envlist} with gate-{name}-tox-{envlist}-{node} https://review.openstack.org/351219 | 15:22 |
| *** baoli_ has joined #openstack-infra | 15:23 | |
| *** baoli has quit IRC | 15:23 | |
| *** matrohon has quit IRC | 15:24 | |
| *** armax has joined #openstack-infra | 15:24 | |
| Zara | in the name of testing interesting gerrit things, would anyone like to merge: https://review-dev.openstack.org/#/c/5456/1 ? I don't have the privileges. | 15:25 |
| Zara | (I'm double-checking task-statuses change as expected) | 15:26 |
| *** rwsu has joined #openstack-infra | 15:26 | |
| EmilienM | jeblair: I followed your suggestion, https://review.openstack.org/#/c/350760/ | 15:26 |
| *** karthikp_ has quit IRC | 15:27 | |
| * AJaeger has to get reacquainted with jeblair again and his switching personality ;) | 15:27 | |
| AJaeger | jeblair: could you review 349627 as well, please? | 15:28 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add tripleo-test-cloud grafana dashboard https://review.openstack.org/351251 | 15:28 |
| *** karthikp_ has joined #openstack-infra | 15:28 | |
| *** armax has quit IRC | 15:29 | |
| *** baoli_ has quit IRC | 15:29 | |
| *** timl0101 has quit IRC | 15:29 | |
| *** armax has joined #openstack-infra | 15:29 | |
| *** baoli has joined #openstack-infra | 15:29 | |
| *** timl0101 has joined #openstack-infra | 15:29 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Replace gate-{name}-tox-{envlist} with gate-{name}-tox-{envlist}-{node} https://review.openstack.org/351219 | 15:30 |
| *** ggnel_t has quit IRC | 15:30 | |
| *** zhurong has quit IRC | 15:30 | |
| anteaya | Zara: you may need to beckon to an infra-root or zaro for that request | 15:31 |
| *** baoli_ has joined #openstack-infra | 15:31 | |
| fungi | i thought we had those set to allow anyone to do the submit button | 15:31 |
| pabelanger | jlk: mordred: let me know if I can help bring up bluebox again in nodepool | 15:31 |
| pabelanger | jlk: mordred: excited to have it back! | 15:32 |
| fungi | anteaya: Zara: but i can check the acl for that project on review-dev once my current meeting ends | 15:32 |
| Zara | fungi: thank you. :) it's not urgent and I'm happy to try with another project if that works better. | 15:34 |
| *** baoli has quit IRC | 15:34 | |
| anteaya | thank you | 15:34 |
| *** amotoki has joined #openstack-infra | 15:35 | |
| mordred | pabelanger: I've updated the admin password information in the passwords file | 15:35 |
| *** EricGonczer_ has quit IRC | 15:35 | |
| mordred | pabelanger: i'm currently working on making a patch/playbook to create and setup all of the things we want in it | 15:35 |
| EmilienM | jeblair: I wonder what we are missing in https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/puppet-module-jobs.yaml#L11-L27 though | 15:35 |
| *** EricGonczer_ has joined #openstack-infra | 15:35 | |
| sdague | mirror break - http://logs.openstack.org/50/350750/4/check/gate-grenade-dsvm-multinode/4353704/logs/grenade.sh.txt.gz#_2016-08-04_15_26_53_334 ? | 15:36 |
| *** ddieterly is now known as ddieterly[away] | 15:37 | |
| *** ddieterly[away] is now known as ddieterly | 15:37 | |
| jeblair | AJaeger: lgtm | 15:37 |
| *** ddieterly is now known as ddieterly[away] | 15:37 | |
| *** EricGonczer_ has quit IRC | 15:37 | |
| jeblair | EmilienM: i think that may have everything it needs | 15:37 |
| EmilienM | jeblair: but like I said before, it doesn't work right. I tested it and it checkout the latest commit in stable/mitaka and not the patch in Depends-On | 15:39 |
| jpmaxman | fungi: So it seems this nocaptcha thing might be a bug in the current version of mediawiki | 15:39 |
| dhellmann | does anyone know if the civs election system has moved URLs? http://www.cs.cornell.edu/w8/~andru/civs/ is reporting a 404 | 15:39 |
| jpmaxman | I talked to some people on the mediawiki IRC and filed a bug | 15:39 |
| jpmaxman | https://phabricator.wikimedia.org/T142096 | 15:39 |
| EmilienM | jeblair: I will try again | 15:39 |
| jeblair | sdague: using wget on that url, i get the correct md5, so probably an error in transport that pip masked | 15:39 |
| jpmaxman | I'm going to try the latest 1.26 version to see if it is still there | 15:39 |
| dhellmann | nevermind, anteaya found it | 15:39 |
| jpmaxman | can you drop a fresh dump of the database | 15:39 |
| jpmaxman | on the wiki-upgrade-test.openstack.org | 15:40 |
| anteaya | http://civs.cs.cornell.edu/civs_create.html is the new url | 15:40 |
| jpmaxman | server ? | 15:40 |
| sdague | jeblair: ok, it's happing with some regularity - http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22THESE%20PACKAGES%20DO%20NOT%20MATCH%20THE%20HASHES%20FROM%20THE%20REQUIREMENTS%20FILE.%5C%22 | 15:40 |
| anteaya | I'll update the wiki in a few minutes with the new url | 15:40 |
| *** ddieterly[away] is now known as ddieterly | 15:40 | |
| mordred | anteaya: ++ | 15:40 |
| anteaya | :) | 15:40 |
| *** matrohon has joined #openstack-infra | 15:40 | |
| anteaya | mordred: dhellmann wondered if they might be trying to hide from you | 15:40 |
| jpmaxman | ok fungi - might not be around will check in later :D | 15:40 |
| anteaya | you breaker of polls you | 15:41 |
| sdague | always in rax-ord it seems | 15:41 |
| mordred | anteaya: it would be a good strategy on their part I think | 15:41 |
| *** priteau has joined #openstack-infra | 15:41 | |
| anteaya | he he he | 15:42 |
| *** narayrak has quit IRC | 15:49 | |
| *** shashank_hegde has joined #openstack-infra | 15:50 | |
| fungi | jpmaxman: i'll look in a bit, in the middle of a meeting right now, sorry | 15:50 |
| *** yamahata has joined #openstack-infra | 15:50 | |
| mtreinish | sdague: was that the region that's had the recent network issues? | 15:51 |
| sdague | mtreinish: I feel like it always has network issues | 15:52 |
| jlk | welcome to RAX.... | 15:53 |
| *** karthikp_ has quit IRC | 15:55 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Position archived items at the bottom of worklists https://review.openstack.org/351265 | 15:57 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Complex priorities UI in stories https://review.openstack.org/312666 | 15:57 |
| *** rossella_s has quit IRC | 16:00 | |
| *** matrohon has quit IRC | 16:00 | |
| *** rossella_s has joined #openstack-infra | 16:00 | |
| *** shashank_hegde has quit IRC | 16:01 | |
| *** karthikp_ has joined #openstack-infra | 16:02 | |
| Zara | gerrit things work nicely for changing task-status to/from todo and review (can't check 'merged') yet. first reactions suggest people find the comment links a bit noisy; we might want to disable them if people are already getting email notifications from gerrit itself. | 16:02 |
| Zara | huh, early closing bracket is early | 16:02 |
| Zara | but \o/, review-dev.o.o is talking nicely to storyboard-dev.o.o | 16:03 |
| *** Liuqing has quit IRC | 16:04 | |
| Zara | (see events timeline changes from today for an example: https://storyboard-dev.openstack.org/#!/story/6 ) | 16:05 |
| *** yottatsa has joined #openstack-infra | 16:05 | |
| *** Hal has quit IRC | 16:06 | |
| *** osnexus has quit IRC | 16:06 | |
| *** apetrich has joined #openstack-infra | 16:06 | |
| *** fernnest has joined #openstack-infra | 16:06 | |
| zaro | morning | 16:07 |
| *** jlanoux has quit IRC | 16:07 | |
| zaro | Zara: cross comment will not be enabled on review.o.o | 16:08 |
| *** baoli_ has quit IRC | 16:08 | |
| zaro | https://review.openstack.org/#/c/330925/18/modules/openstack_project/manifests/review.pp@205 | 16:09 |
| *** ddieterly is now known as ddieterly[away] | 16:09 | |
| zaro | it's only enabled on review-dev.o.o for testing. i'm thinking maybe changing setting on review-dev.o.o to same as review.o.o (turn auto comments off) | 16:10 |
| *** raildo has quit IRC | 16:10 | |
| Zara | ohhhh, I missed that it was only enabled there. yeah, that's probably a good idea, so people can get an accurate idea of what it will look like | 16:11 |
| *** raildo has joined #openstack-infra | 16:12 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/system-config: Start indexing ansible_log.txt in logstash https://review.openstack.org/351269 | 16:12 |
| mriedem | clarkb: mtreinish: sdague: ^ hope that's not terrible | 16:12 |
| zaro | hmm, it doesn't look like puppet is keeping review-dev.o.o up to date. was wondering if an infra-root can help? | 16:13 |
| jeblair | sshnaidm: i have tagged gear 0.7.0 (cc: fungi, mordred, notmorgan, SpamapS) | 16:13 |
| *** eharney has quit IRC | 16:13 | |
| jeblair | (should show up on pypi in a little while) | 16:13 |
| sshnaidm | jeblair, \o/ great news, thanks! | 16:13 |
| zaro | this change should have re-enabled puppet on review-dev https://review.openstack.org/#/c/347036/ | 16:14 |
| *** aorourke has joined #openstack-infra | 16:14 | |
| *** baoli has joined #openstack-infra | 16:14 | |
| fungi | zaro: we also had it listed in the emergency disable file because of the bug we were tracking down that was causing ansible to ignore other groups | 16:14 |
| *** raildo has quit IRC | 16:15 | |
| fungi | zaro: i've removed it just now | 16:15 |
| *** ddieterly[away] is now known as ddieterly | 16:15 | |
| fungi | should get hit with puppet on the next pulse | 16:15 |
| *** mat128 is now known as mat128|lunch | 16:15 | |
| fungi | maube in 20-30 minutes | 16:15 |
| *** seldenr has joined #openstack-infra | 16:15 | |
| zaro | fungi: cool, thanks! | 16:15 |
| *** dtantsur is now known as dtantsur|afk | 16:16 | |
| *** nmagnezi has quit IRC | 16:17 | |
| *** matrohon has joined #openstack-infra | 16:17 | |
| *** raildo has joined #openstack-infra | 16:18 | |
| jeblair | Shrews, mordred, notmorgan: this is an interesting ansible error: http://logs.openstack.org/26/350726/2/check/gate-zuul-python27-db-ubuntu-xenial/f79a6c7/console.html#_2016-08-03_22_51_57_444367 | 16:19 |
| mordred | jeblair: I don't lke the word interesting in this context | 16:19 |
| jeblair | i think it may be a behavior change in 2.1.1.0 -- at least, that code was touched recently | 16:19 |
| mordred | jeblair: I agree | 16:19 |
| jeblair | the underlying issue is that it's two ansible processes racing to create the local tmp dir | 16:19 |
| mordred | jeblair: I think we should poke our friends in #ansible-devel | 16:20 |
| jeblair | (and apparently python's os.makedirs is suceptible to such races!) | 16:20 |
| jeblair | it only happened once in my stack of 5 changes | 16:20 |
| *** _nadya_ has joined #openstack-infra | 16:20 | |
| *** weshay is now known as weshay_food | 16:21 | |
| mordred | jeblair: which two ansible processes is it do you think? | 16:21 |
| mordred | like, the async thing and then the next task in the list? | 16:21 |
| jlk | so... | 16:21 |
| jlk | we had to back away from 2.1.1.0 :( | 16:21 |
| jlk | but for unrelated reasons | 16:21 |
| jeblair | mordred: i think it's probably two parallel test runs | 16:21 |
| jeblair | mordred: but that's just a guess | 16:21 |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Update its-storyboard actions on review-dev.o.o https://review.openstack.org/351273 | 16:22 |
| zaro | Zara: ^ | 16:22 |
| mordred | jeblair: oh - like, that's ~/.ansible/tmp on the launcher host? | 16:22 |
| Zara | zaro: thanks, taking a look now :) | 16:22 |
| mordred | jeblair: I thought we made a change to put that into the launcher temp dir we made already - I guess we did not do that? | 16:22 |
| mordred | jlk: yah - we have two fixes we made in upstream that landed in 2.1.1.0 - so moving to it allowed us to not run patched locally | 16:23 |
| jeblair | mordred: hrm, let me look into some things | 16:23 |
| *** e0ne has joined #openstack-infra | 16:23 | |
| mordred | jlk: it'll be sad panda if we also have to back away | 16:23 |
| mordred | jeblair: (seems that if we didn't do that, doing that should potentially ameliorate this) | 16:23 |
| jlk | something changed in the way variables are resolved that led to weird scenarios in our playbook tree | 16:23 |
| SpamapS | jeblair: sweet | 16:23 |
| *** matrohon has quit IRC | 16:23 | |
| jlk | for us, we just had to back away while we do a deeper investigation. | 16:23 |
| mordred | jlk: oh - I think I saw you talking about that | 16:23 |
| mordred | and yah - you have some complex variable overlays going on | 16:24 |
| Zara | zaro: thank you, looks fine to me :) | 16:24 |
| jeblair | mordred: i believe we are running with the zuul-launcher change that sets remote_tmp to something inside of /tmp so that it will not collide what what a job is running. so i think that we have removed the zuul v2.5 ansible launcher from the equation. | 16:25 |
| jlk | 2.1.1.0 also got more strict about some syntax things that previously I think were ignored. So, yay for fixing bugs, and oops on us for having bugs | 16:26 |
| jeblair | mordred: so i suspect that this is just two parallel zuul testr runs trying to create the _local_ ansible tmp dir | 16:26 |
| SpamapS | jlk: hopefully only things that were warnings before? | 16:26 |
| jlk | SpamapS: I think it was silently ignored | 16:27 |
| jlk | it was clearly a syntax issue | 16:27 |
| *** karthikp_ has quit IRC | 16:27 | |
| jlk | with_items: "{{ groups["controller"] }}" | 16:27 |
| SpamapS | jlk: that's kind of b.s. though | 16:27 |
| jlk | that was the line that now is a immediate fail | 16:27 |
| jlk | double use of " is bad | 16:28 |
| SpamapS | I see why it would fail | 16:28 |
| SpamapS | but I'm kind of bullish on always being backward compatible | 16:28 |
| jlk | yeah I'm not really upset about this one. | 16:28 |
| jlk | I think if this file were ever hit by previous versions it would have failed too, it just happens that this was in an unused code path | 16:29 |
| SpamapS | wait | 16:29 |
| SpamapS | how did that ever parse? | 16:29 |
| SpamapS | that's totally invalid yaml? | 16:29 |
| *** Apoorva has joined #openstack-infra | 16:29 | |
| SpamapS | jlk: you guys need our super awesome yaml file syntax checker. :) | 16:29 |
| jeblair | mordred: this is the apparently new code path: https://github.com/ansible/ansible/commit/dcc5dfdf#diff-b77962b6b54a830ec373de0602918318R69 | 16:30 |
| jlk | SpamapS: I think the code path was never parsed before | 16:30 |
| jlk | and now every code path is parsed? | 16:30 |
| jeblair | mordred: i think the actual solution to this is to forward-port the zuulv2.5 ansible tmp path stuff to zuulv3 | 16:31 |
| mordred | jeblair: ++ | 16:31 |
| *** _nadya_ has quit IRC | 16:31 | |
| mordred | jeblair: forward-porting the zuulv2.5 ansible stuff was coming up soon on my TDL | 16:31 |
| *** Apoorva has quit IRC | 16:31 | |
| jeblair | mordred: cool; i can probably recheck through this for now | 16:31 |
| mordred | cool | 16:31 |
| *** Apoorva has joined #openstack-infra | 16:32 | |
| *** ddieterly is now known as ddieterly[away] | 16:32 | |
| *** kro_focused is now known as krotscheck | 16:32 | |
| jeblair | mordred: *maybe* ansible could do: try: os.makedirs() except OSError e: if e.errno==17: pass | 16:32 |
| jeblair | mordred: that seems weird to me since os.makedirs is already supposed to not do that :) | 16:33 |
| jeblair | mordred: i think i'll mention this in -devel in case they want to run with it. but i wouldn't blame them for not. | 16:33 |
| mordred | ++ | 16:33 |
| *** sandanar has joined #openstack-infra | 16:33 | |
| *** tphummel has joined #openstack-infra | 16:34 | |
| *** shashank_hegde has joined #openstack-infra | 16:36 | |
| *** lucasagomes is now known as lucas|dinner | 16:39 | |
| *** SumitNaiksatam has joined #openstack-infra | 16:39 | |
| *** notmyname has joined #openstack-infra | 16:40 | |
| *** matrohon has joined #openstack-infra | 16:40 | |
| *** pt_15 has joined #openstack-infra | 16:42 | |
| *** gomarivera has joined #openstack-infra | 16:43 | |
| *** weshay_food is now known as weshay | 16:43 | |
| *** e0ne has quit IRC | 16:43 | |
| *** yaume has quit IRC | 16:44 | |
| *** florianf is now known as florianf|biab | 16:44 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add gate-python-tripleoclient-buildimage jobs https://review.openstack.org/351278 | 16:45 |
| *** tqtran has joined #openstack-infra | 16:47 | |
| *** Swami has joined #openstack-infra | 16:48 | |
| *** Sukhdev has joined #openstack-infra | 16:50 | |
| *** itisha has joined #openstack-infra | 16:51 | |
| *** tqtran has quit IRC | 16:51 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add gate-python-tripleoclient-buildimage jobs https://review.openstack.org/351278 | 16:52 |
| *** david-lyle has quit IRC | 16:53 | |
| *** david-lyle has joined #openstack-infra | 16:53 | |
| *** ddieterly[away] is now known as ddieterly | 16:54 | |
| *** tosky has quit IRC | 16:55 | |
| *** jtomasek has quit IRC | 16:55 | |
| *** berendt has joined #openstack-infra | 16:55 | |
| *** asettle has quit IRC | 16:55 | |
| *** jpich has quit IRC | 16:57 | |
| jeblair | fungi: the release management team's mascot is an anteater (pangolin). and infra are the ants? | 16:57 |
| jeblair | fungi: er, sorry, security team, not relmgt | 16:58 |
| * anteaya thinks about jumping ship | 16:58 | |
| jeblair | "What if one team's mascot's primary diet is that of a mascot of another team?" is not in the FAQ. | 16:58 |
| anteaya | or did fungi set it up that way? | 16:59 |
| anteaya | ha ha ha | 16:59 |
| anteaya | it should be | 16:59 |
| jeblair | anteaya: indeed -- how many people are on both teams? | 16:59 |
| anteaya | ha ha ha | 16:59 |
| anteaya | a dozen to over 200 | 16:59 |
| * anteaya looks pensively into the distance | 16:59 | |
| jeblair | oh right our team is big :) | 16:59 |
| jlk | lots of ants in the hill | 17:00 |
| anteaya | yeah, so many repos | 17:00 |
| anteaya | jlk: you also thinking security is looking good? | 17:00 |
| jlk | oh hell no | 17:00 |
| anteaya | ha ha ha | 17:00 |
| jlk | talk about tilting at windmills | 17:00 |
| anteaya | ha ha ha | 17:00 |
| anteaya | you are maintaining clouds | 17:00 |
| *** piet_ has joined #openstack-infra | 17:01 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add automatic backports https://review.openstack.org/346095 | 17:02 |
| pabelanger | zigo: ^ should pass now | 17:02 |
| jeblair | anteaya: what's worse is trying to build a platform on a cloud. have you ever tried that? it doesn't work well. | 17:02 |
| *** sambetts is now known as sambetts|afk | 17:03 | |
| anteaya | ha ha ha | 17:03 |
| jeblair | joists fall right through | 17:03 |
| anteaya | that was what I was thinking | 17:03 |
| *** flaper87 has quit IRC | 17:03 | |
| anteaya | so hard to get those nails in place in time | 17:03 |
| openstackgerrit | Merged openstack-infra/project-config: grafana: Update ovn unit test jobs after xenial renames https://review.openstack.org/351019 | 17:04 |
| jeblair | anteaya, fungi: i have sent a query to heidi :) | 17:05 |
| anteaya | jeblair: please share any reply | 17:08 |
| jeblair | will do | 17:10 |
| *** sarob has joined #openstack-infra | 17:11 | |
| *** matrohon has quit IRC | 17:11 | |
| *** gomarivera has quit IRC | 17:11 | |
| *** permalac has quit IRC | 17:11 | |
| *** sandanar has quit IRC | 17:11 | |
| *** eharney has joined #openstack-infra | 17:12 | |
| *** _sarob has joined #openstack-infra | 17:13 | |
| fungi | yeah, i hadn't thought about that. might be a good excuse to go with the honeycomb which came in second | 17:14 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Update iLO IP for baremetal00 https://review.openstack.org/351291 | 17:14 |
| * rcarrillocruz dances ^ | 17:15 | |
| mtreinish | fungi: isn't that what refstack picked... | 17:15 |
| *** sarob has quit IRC | 17:16 | |
| *** signed8bit is now known as signed8bit_Zzz | 17:16 | |
| *** SumitNaiksatam has quit IRC | 17:16 | |
| *** PollyZ has quit IRC | 17:18 | |
| mordred | rcarrillocruz: ooh! that makes it look like baremetal00 might exist again! | 17:19 |
| rcarrillocruz | \o/ | 17:19 |
| rcarrillocruz | so now consolidation reboot 2.0 begins | 17:19 |
| rcarrillocruz | i need to reboot all servers | 17:19 |
| rcarrillocruz | enter on rescue mode | 17:19 |
| rcarrillocruz | figure out what server is by comparing it's "old public ip" to our hiera | 17:20 |
| rcarrillocruz | so i can update the right entry on the hiera infracloud.yaml | 17:20 |
| fungi | mtreinish: maybe? i have heidijoy a ranked list of 23 options, so she'll mark them off until she reaches one we're able to use | 17:20 |
| rcarrillocruz | it will take time, but hey, our servers are reachable on the iLOs at least | 17:20 |
| *** shashank_hegde has quit IRC | 17:20 | |
| fungi | er, s/have/gave/ | 17:20 |
| *** karthikp_ has joined #openstack-infra | 17:21 | |
| *** rbuzatu has quit IRC | 17:22 | |
| *** mat128|lunch is now known as mat128 | 17:23 | |
| *** karthikp_ has quit IRC | 17:24 | |
| *** jcoufal_ has joined #openstack-infra | 17:24 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Enable net-iso for ovb jobs https://review.openstack.org/343017 | 17:24 |
| mordred | fungi: I'd like to express my sadness that "pink fairy armadillo" wasn't the winner | 17:24 |
| *** dprince has quit IRC | 17:24 | |
| *** vhosakot has quit IRC | 17:25 | |
| fungi | mordred: right after i complain that almost nobody wanted a tanuki | 17:25 |
| *** vhosakot has joined #openstack-infra | 17:25 | |
| jeblair | fungi: refstack got the honeycomb+bee | 17:25 |
| anteaya | that is two | 17:26 |
| *** yottatsa_ has joined #openstack-infra | 17:26 | |
| *** jcoufal has quit IRC | 17:26 | |
| anteaya | it is tough enough to not conflict on one | 17:26 |
| *** daemontool has joined #openstack-infra | 17:28 | |
| anteaya | mordred: well if someone else has beaver we might be down to the pink fairy armadillo | 17:28 |
| anteaya | I'm nost sure if others have selected the pink fairy armadillo | 17:28 |
| pabelanger | rcarrillocruz: nice | 17:29 |
| *** yottatsa has quit IRC | 17:29 | |
| *** _nadya_ has joined #openstack-infra | 17:30 | |
| *** karthikp_ has joined #openstack-infra | 17:30 | |
| *** javeriak has joined #openstack-infra | 17:32 | |
| *** degorenko is now known as _degorenko|afk | 17:32 | |
| *** rbuzatu has joined #openstack-infra | 17:32 | |
| *** Na3iL has quit IRC | 17:36 | |
| *** amitgandhinz has quit IRC | 17:36 | |
| *** zul has quit IRC | 17:36 | |
| *** amitgandhinz has joined #openstack-infra | 17:36 | |
| fungi | jpmaxman: there's now a copy of this morning's production wiki db backup available for you at wiki-upgrade-test.openstack.org:~maxwell/wiki_2016-08-04.sql | 17:38 |
| AJaeger | fungi, https://review.openstack.org/#/c/349627/ - the constraints change - has passed check jobs. Do you want to approve it? Then I'll give a heads up to the release team... | 17:38 |
| jeblair | anteaya: i think i would summarize heidi's reply as: good question, not one they had considered in that context, most projects chose mascots to reflect strengths. iow, i don't think she's worried about the symbolism. (i'm not either, fwiw) | 17:38 |
| *** rbuzatu has quit IRC | 17:38 | |
| Zara | (pink fairy armadillo for president) | 17:38 |
| fungi | AJaeger: i'll check with them real quick just to make sure we're not risking major disruption for them | 17:39 |
| *** shardy has quit IRC | 17:39 | |
| anteaya | jeblair: thank you for sharing that | 17:39 |
| AJaeger | thanks, fungi | 17:39 |
| *** rbuzatu has joined #openstack-infra | 17:39 | |
| jeblair | but i think it's good for us to be aware of it and not be surprised. beyond that, i leave things in fungi's capable hands :) | 17:39 |
| *** rockyg has quit IRC | 17:40 | |
| *** acoles is now known as acoles_ | 17:41 | |
| *** rbuzatu has quit IRC | 17:41 | |
| *** rbuzatu has joined #openstack-infra | 17:41 | |
| *** dprince has joined #openstack-infra | 17:41 | |
| *** SumitNaiksatam has joined #openstack-infra | 17:42 | |
| *** vhosakot has quit IRC | 17:42 | |
| *** HeOS has quit IRC | 17:42 | |
| harlowja | fungi u got some time for some historical questions, i'm sorta interested in why zuul got created (vs just jenkins) and the whole drop jenkins --> ansible and how this all happened | 17:43 |
| harlowja | maybe i'll write about all this in a book someday, ha | 17:44 |
| fungi | harlowja: jeblair can do it more justice than i, though i'm happy to take a stab at it | 17:44 |
| harlowja | ah, either is fine | 17:44 |
| harlowja | either/all | 17:44 |
| * harlowja more of just curios | 17:44 | |
| harlowja | *curious | 17:44 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add gate-python-tripleoclient-buildimage jobs https://review.openstack.org/351278 | 17:44 |
| *** Sukhdev has quit IRC | 17:45 | |
| harlowja | as i start trying to get some internal folks to get more involved with the openstack-jenkins-cloud plugin | 17:45 |
| *** apetrich has quit IRC | 17:45 | |
| harlowja | it seems to be a interesting question, of why doesn't the openstack infra folks also use that (or have created it or ...) | 17:45 |
| fungi | harlowja: it sort of grew out of the limited featureset in the gerrit-trigger plugin for jenkins and our desire to be able to test changes in a serialized context before allowing them to merge | 17:45 |
| *** apetrich has joined #openstack-infra | 17:45 | |
| harlowja | and that gets into the whole jenkins, zuul, history, ansible. ... | 17:45 |
| EmilienM | jeblair: when you have time, can you +1 this puppet CI patch please? https://review.openstack.org/#/c/351247/ | 17:45 |
| EmilienM | jeblair: just to make sure I'm doing the right way | 17:45 |
| harlowja | fungi do u know if that got better (in jenkins itself?) | 17:46 |
| harlowja | (or in the gerrit plugin in jenkins) | 17:46 |
| jeblair | harlowja: HP heavily sponsored the jclouds plugin https://wiki.jenkins-ci.org/display/JENKINS/JClouds+Plugin on behalf of the openstack project | 17:46 |
| fungi | harlowja: jenkins has just in the last few months grew a concept of "pipelining" which i think might be closer to that | 17:46 |
| harlowja | jeblair right, its interesting to see https://issues.jenkins-ci.org/browse/JENKINS-30628 which that other plugin dropped the jclouds one | 17:46 |
| * harlowja has been talking (sorta, via issues, ha) with the openstack-cloud-plugin author | 17:47 | |
| jeblair | fungi: i don't think jenkins pipeline is like zuul | 17:47 |
| jeblair | harlowja: jenkins and zuul do different things :) | 17:47 |
| fungi | ahh, i probably completely misunderstood the point of jenkins pipelines | 17:47 |
| fungi | limitations in the jclouds plugin (which we tried to use for a while) eventually drove the invention of nodepool though | 17:48 |
| jeblair | fungi: it's more like chained jobs | 17:48 |
| harlowja | jeblair jenkins does tasks/jobs in certain orders, so does zuul? | 17:48 |
| harlowja | (really simplified view?) ha | 17:48 |
| mordred | harlowja: it's the combination of different proposed repository states that zuul does that jenkins has no concept of | 17:48 |
| mordred | harlowja: so, thikn depends-on in patches | 17:49 |
| fungi | zuul (well, excluding zuul-launcher) doesn't really do any tasks/jobs itself. it's mainly a scheduler | 17:49 |
| mordred | harlowja: and the way nova and keystone and glance and devstack all are interrelated | 17:49 |
| *** shashank_hegde has joined #openstack-infra | 17:49 | |
| *** zul has joined #openstack-infra | 17:49 | |
| harlowja | fungi kk | 17:49 |
| *** karthikp_ has quit IRC | 17:49 | |
| jeblair | harlowja: zuul is a project gating system, at heart: http://docs.openstack.org/infra/zuul/gating.html | 17:49 |
| mordred | it's a completely different basic world view | 17:49 |
| *** signed8bit_Zzz is now known as signed8bit | 17:49 | |
| *** ddieterly is now known as ddieterly[away] | 17:49 | |
| *** javeriak has quit IRC | 17:49 | |
| harlowja | interesting, i wonder if jenkins could have changed (?) to have shared views, (just theorizing) | 17:50 |
| *** sputnik13 has joined #openstack-infra | 17:50 | |
| jeblair | fungi: yes, the scheduling aspect is key | 17:50 |
| fungi | to get a similar model in jenkins would, at a minimum, have required interested parties who didn't mind trying to implement that in java ;) | 17:51 |
| mordred | harlowja: anything is possible, but it would essential be a rewrite of the fundamental concepts of jenkins | 17:51 |
| *** vhosakot has joined #openstack-infra | 17:51 | |
| harlowja | fair, i guess thus zuul | 17:51 |
| harlowja | (due to that desire to not try that rewrite and java) | 17:52 |
| mordred | harlowja: as for drop jenkins for ansible ... by the time we had nodepool creating nodes and attaching them to a set of jenkins masters and had zuul scheduling the running of jobs - jenkins had turned in to essentially a remote execution engine for us | 17:52 |
| harlowja | mordred right | 17:52 |
| openstackgerrit | Merged openstack-infra/project-config: Setup constraints for release https://review.openstack.org/349627 | 17:53 |
| AJaeger | Isn't the scaling part also a reason for zuul? We used in the end 8 different jenkins instances and needed to distribute work to them. | 17:53 |
| mordred | AJaeger: yup | 17:53 |
| *** e0ne has joined #openstack-infra | 17:53 | |
| * harlowja gets confused to as how other people (companies) use jenkisn though | 17:53 | |
| harlowja | does everyone not use jenkins at scale, ha | 17:53 |
| mordred | harlowja: we have VERY intense scale needs that dwarf what other people tend to need | 17:53 |
| *** tesseract- has quit IRC | 17:53 | |
| fungi | scaling more drove zuul 2.0 and its use of gearman so that we could tie in multiple jenkins masters | 17:54 |
| mordred | harlowja: a single jenkins master for us dies on the altar of internal mutexes | 17:54 |
| harlowja | fair, though it makes me wonder how travis is doing it | 17:54 |
| harlowja | (forget if they are using jenkins) | 17:54 |
| clarkb | not to mention the crippling slave thread leak bug that requires weekly master restarts | 17:54 |
| fungi | zuul 1.0 was just associating jobs with one jenkins master | 17:54 |
| mordred | well, travis also does their own thing and does not use jenkins | 17:54 |
| harlowja | kk | 17:54 |
| * clarkb goes back ti not being here | 17:54 | |
| mordred | clarkb: ++ | 17:54 |
| fungi | clarkb: you're terrible at not being here | 17:54 |
| mordred | harlowja: also, last time we compared, which was a while ago, we were a bit larger than travis :) | 17:55 |
| *** thorst has quit IRC | 17:55 | |
| harlowja | bigger and badder | 17:55 |
| harlowja | ha | 17:55 |
| *** thorst has joined #openstack-infra | 17:55 | |
| mordred | they've been growing a lot though, so they may have caught up ... they're also openvz container based rather than vm based, so it's not entirely an apples to apples game | 17:55 |
| fungi | also what's scarier, an eldritch demigod with a taste for devouring whole civilizations, or some guy named "travis"? | 17:56 |
| anteaya | travis | 17:56 |
| fungi | ooh, anteaya may have a point | 17:56 |
| anteaya | watch out for travis | 17:56 |
| harlowja | u never expect travis | 17:56 |
| mordred | but also they have the benefit of easy use case - since they dont' to interdependent repos, everything for them is like our check queue | 17:56 |
| fungi | he's like the spanish inquisition? | 17:56 |
| harlowja | (vs u do expect zuul to do bad things) | 17:56 |
| harlowja | ha | 17:57 |
| *** _sarob has quit IRC | 17:57 | |
| anteaya | you have your eye on the demigod, and travis? he is just doing whatever | 17:57 |
| harlowja | :-P | 17:57 |
| harlowja | mordred fair | 17:58 |
| *** jkilpatr has quit IRC | 17:58 | |
| harlowja | someone should write all this history down, ha | 17:58 |
| *** kzaitsev_mb has quit IRC | 17:58 | |
| jeblair | mordred: though, our check queue has interdependent repos these days :) | 17:58 |
| *** mdrabe_ has joined #openstack-infra | 17:58 | |
| openstackgerrit | Khai Do proposed openstack-infra/puppet-gerrit: fix its_rules parameterization in template https://review.openstack.org/351323 | 17:59 |
| jeblair | so, er, it's like our easy use case a couple years ago | 17:59 |
| zaro | Zara: ^ | 17:59 |
| mordred | jeblair: (ssh, you're making my simplistic story more copmlex) | 17:59 |
| mordred | mmm. copmlex | 17:59 |
| mordred | jeblair: but yeah | 17:59 |
| *** berendt has quit IRC | 18:00 | |
| mordred | harlowja: in any case, we did spend a decent amount of effort making things work with jenkins in each case before we discarded any of its pieces - which has involved actually writing several jenkins plugins, funding the writing of more we never could use and whatnot | 18:00 |
| *** mdrabe_ has quit IRC | 18:00 | |
| *** mdrabe_ has joined #openstack-infra | 18:00 | |
| harlowja | mordred ya, its interesting to know the trials and attempts and such | 18:00 |
| mordred | harlowja: my biggest concern with writing down a history is that it would be taken as a negative at jenkins, which would not be the intent | 18:01 |
| *** thorst_ has joined #openstack-infra | 18:01 | |
| fungi | harlowja: also the jenkins-jclouds and openstack-jenkins-cloud plugins are more along the lines of nodepool than zuul i think (though that distinction is soon to become somewhat moot) | 18:01 |
| *** rbrndt has quit IRC | 18:01 | |
| mordred | harlowja: but I agree, the trials and attempts are potentially good info for people | 18:01 |
| harlowja | mordred well maybe just don't frame it like that | 18:01 |
| *** ccamacho has quit IRC | 18:01 | |
| harlowja | (like a negative, vs just not a good fit at the time) | 18:01 |
| mordred | harlowja: yah. that would be quite key | 18:01 |
| jeblair | jenkins is very good at what it does, it has served us very well, and the authors are a pleasure to work with. | 18:01 |
| jeblair | we just do something fundamentally different | 18:02 |
| fungi | also i think a big driver in all of this is our use case for open infrastructure is not well aligned with the design of jenkins that assumes a lot of access control and running things behind firewalls on internal networks | 18:02 |
| *** mdrabe has quit IRC | 18:02 | |
| *** julim has quit IRC | 18:02 | |
| phschwartz | jeblair: so it looks like we are having a nodepool demand calculation issue and it would be appreciated if you could point me at the right place to look. http://paste.openstack.org/show/549283/ | 18:02 |
| harlowja | ya, it does make me wonder though if they (the jenkins folks) knew about more our ' trials and attempts' that the paths could align at some point | 18:02 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add gate-python-tripleoclient-buildimage jobs https://review.openstack.org/351278 | 18:02 |
| phschwartz | jeblair: all the jobs queued in that paste are all on the bare-trusty label. | 18:02 |
| *** jkilpatr has joined #openstack-infra | 18:02 | |
| jeblair | harlowja: i gave a talk on zuul at a jenkins user conference | 18:03 |
| harlowja | cools | 18:03 |
| jeblair | harlowja: we talk to them a lot in various ways :) | 18:03 |
| harlowja | BFFs? | 18:03 |
| harlowja | lol | 18:03 |
| *** thorst has quit IRC | 18:03 | |
| harlowja | ex-BFFs? | 18:03 |
| harlowja | lol | 18:03 |
| fungi | we talked to them a lot more when we were actually running jenkins, granted | 18:03 |
| jeblair | fungi: true, i have not been as involved *lately* :) | 18:03 |
| *** rockyg has joined #openstack-infra | 18:04 | |
| mordred | jeblair: going to the jenkins uc for us _now_ might be a bit strange | 18:04 |
| fungi | harlowja: situations like https://issues.jenkins-ci.org/browse/JENKINS-27514 were pretty frustrating though up to the end | 18:04 |
| jeblair | i have not submitted a zuulv3 talk, no. | 18:04 |
| harlowja | fungi ha, nice | 18:05 |
| jeblair | phschwartz: have you changed the labels for those jobs? | 18:05 |
| *** julim has joined #openstack-infra | 18:05 | |
| jeblair | phschwartz: if so, nodepool may not be mapping them correctly, to fix, restart the gearman server (which may mean restart zuul) | 18:05 |
| jeblair | phschwartz: (the job-label mapping is made through gearman function names, and those stick around permanently until the gearman server is restarted) | 18:06 |
| phschwartz | jeblair: so the label name hasn't changed in 6 months, all that we have done is add jobs using it | 18:06 |
| fungi | harlowja: we basically ended up with ansible in cron automating rolling restarts of all 8 of our jenkins masters weekly, but even that was about half as often as it needed to be after a while | 18:07 |
| phschwartz | Zuul was restarted yesterday and it still has this issue. But I don't mind restarting it again. | 18:07 |
| jeblair | phschwartz: can you paste the output of "echo status|nc $ZUUL_SERVER 4730" | 18:08 |
| phschwartz | jeblair: sec, grabbing it | 18:09 |
| harlowja | fungi ya, seems to be a pita thread junk | 18:09 |
| openstackgerrit | Merged openstack-infra/nodepool: Include subnodes in updateStats function https://review.openstack.org/349119 | 18:09 |
| *** _nadya_ has quit IRC | 18:09 | |
| phschwartz | jeblair: http://paste.openstack.org/show/549285/ It also seems we get image-upload and image-delete that hang around in there till restart of zuul. | 18:11 |
| jeblair | phschwartz: yep. they're harmless but annoying. | 18:11 |
| *** tpeoples has quit IRC | 18:12 | |
| *** rbuzatu has quit IRC | 18:12 | |
| phschwartz | jeblair: The reason why I am asking for help is one of our jobs is a test deployment with full tempest being run and takes upwards of 2 hours. Right now, we have 8 of them taking up all 8 nodes that were biult because of the min-ready: 8 | 18:12 |
| phschwartz | jeblair: We have a large capacity, we just can't get it to boot more nodes. | 18:12 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for nova resource tracker/scheduler race https://review.openstack.org/316364 | 18:14 |
| jeblair | phschwartz: gimme a minute while i run the demand calculation on the back of a napkin | 18:16 |
| *** tqtran has joined #openstack-infra | 18:17 | |
| openstackgerrit | Anita Kuno proposed openstack-infra/infra-specs: Making space for new lead https://review.openstack.org/351326 | 18:17 |
| *** tpsilva has quit IRC | 18:18 | |
| *** PollyZ has joined #openstack-infra | 18:18 | |
| *** tonytan4ever has joined #openstack-infra | 18:21 | |
| *** Sukhdev has joined #openstack-infra | 18:22 | |
| jeblair | phschwartz: how many bare-trusty nodes do you have in the each state? (according to 'nodepool list') ? | 18:22 |
| *** javeriak has joined #openstack-infra | 18:22 | |
| *** PollyZ has quit IRC | 18:24 | |
| jeblair | phschwartz: also, can you give me the "Demand from gearman:" log line that goes along with these log entries? http://paste.openstack.org/show/549283/ | 18:24 |
| *** seldenr has left #openstack-infra | 18:24 | |
| phschwartz | jeblair: 8 in used state | 18:25 |
| *** karthikp_ has joined #openstack-infra | 18:25 | |
| *** sarob has joined #openstack-infra | 18:26 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Make the index format string configurable in the conf file. https://review.openstack.org/306360 | 18:26 |
| phschwartz | jeblair: there is no demand from gearman line in the logs | 18:27 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/germqtt: Add config option to set QOS of published messages https://review.openstack.org/350169 | 18:27 |
| AJaeger | fungi, I think 349627 is live now - can you confirm, please? | 18:27 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton https://review.openstack.org/346995 | 18:28 |
| jeblair | phschwartz: what version of nodepool are you running? | 18:29 |
| fungi | AJaeger: thanks was just about to confirm | 18:30 |
| *** xnox has joined #openstack-infra | 18:30 | |
| phschwartz | jeblair: the tag before your major changes | 18:30 |
| AJaeger | fungi, I checked puppetboard and it should be applied but can't actually confirm | 18:31 |
| jeblair | phschwartz: can you confirm that you're running 0.3.0 at commit 3a2ed8a755737f34bf4d9842bb44404cc40ecee5 ? | 18:31 |
| phschwartz | jeblair: when I do a --version it is 0.3.0 | 18:32 |
| phschwartz | jeblair: we pinned the tag in our puppet | 18:32 |
| *** whoops has quit IRC | 18:32 | |
| fungi | AJaeger: oh right, this is job configuration, so i need to see if i can spot how zuul-launcher logs its reconfiguration events | 18:33 |
| *** vhosakot has quit IRC | 18:33 | |
| fungi | AJaeger: from zl01: 2016-08-04 18:11:04,761 DEBUG zuul.LaunchServer: Reconfiguration complete | 18:33 |
| *** whoops has joined #openstack-infra | 18:33 | |
| jeblair | phschwartz: i think that will always print the latest tag, but not if there are any patches on top of it | 18:33 |
| fungi | AJaeger: so i think we're good | 18:33 |
| *** vhosakot has joined #openstack-infra | 18:33 | |
| jeblair | phschwartz: how about pbr freeze, pip freeze, or checking the HEAD commit in /opt/nodepool | 18:34 |
| AJaeger | fungi, I agree. Let's tell the release team... | 18:34 |
| *** florianf|biab is now known as florianf | 18:34 | |
| *** karthikp_ has quit IRC | 18:35 | |
| phschwartz | jeblair: looking | 18:35 |
| *** karthikp_ has joined #openstack-infra | 18:35 | |
| *** yottatsa_ has quit IRC | 18:36 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement non-ovb overcloud update job - Newton -> Newton https://review.openstack.org/351330 | 18:36 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add controller01.vanilla metadata for better tracking https://review.openstack.org/351331 | 18:36 |
| *** ayoung has quit IRC | 18:38 | |
| phschwartz | jeblair: pip freeze shows 0.3.0 | 18:38 |
| phschwartz | jeblair: pschwar@nodepool:/opt/nodepool$ git status | 18:38 |
| phschwartz | On branch tag/0.3.0 | 18:38 |
| fungi | wow, was troubleshooting stackalytics.o.o (had become unresponsive and needed a uswgi restart)... in skimming to make sure it was working, i noticed it says that project-config has more commits during newton than any other repo, 1338 so far edging out nova at 1209 (granted the commits involved are not comparable at all, but still an interesting statement on volume) | 18:40 |
| jeblair | phschwartz: when i run the code from 0.3.0 that gets the demand from gearman on the gear output you gave me, i get: Demand from gearman: bare-trusty: 23 | 18:41 |
| *** rbrndt has joined #openstack-infra | 18:42 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute040.vanilla metadata for better tracking https://review.openstack.org/351332 | 18:42 |
| phschwartz | jeblair: I grepped the nodepool logs for demand with `grep -i demand` and got no return | 18:42 |
| AJaeger | fungi: looking at http://stackalytics.com/?metric=commits it's 1408 for nova and 1388 for project-config | 18:42 |
| phschwartz | jeblair: is there a chance that an old version is running in memory since the last install? | 18:42 |
| AJaeger | Let's merge some more ;) | 18:42 |
| fungi | AJaeger: i wonder why they're giving different results | 18:43 |
| jeblair | phschwartz: absolutely. did you run something after 0.3.0 before you pinned it to 0.3.0? | 18:43 |
| jeblair | phschwartz: what's the start time for nodepoold in ps? | 18:43 |
| AJaeger | fungi, so do I... | 18:44 |
| *** florianf has quit IRC | 18:44 | |
| phschwartz | jeblair: Jul19 1602:27 | 18:44 |
| phschwartz | but that should be after 0.3.0 | 18:44 |
| AJaeger | are both using the same calculation for start of newton? | 18:44 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add gate-python-tripleoclient-buildimage jobs https://review.openstack.org/351278 | 18:45 |
| jeblair | phschwartz: yes, but did you pin it yourself, or is it pinned because I289186b7a25755c9258897202f01d278d0cc5a75 landed this morning? | 18:45 |
| phschwartz | jeblair: pin'ed in our system-config that uses puppet-openstackci | 18:45 |
| jeblair | phschwartz: when did that take effect? | 18:45 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute042.vanilla metadata for better tracking https://review.openstack.org/351334 | 18:45 |
| phschwartz | jeblair: Spencer Krum 2016-07-21 11:51:24 -0700 147) revision => '0.3.0', | 18:46 |
| nibalizer | uh oh | 18:46 |
| phschwartz | jeblair: so looks like nibalizer pinned it after that. Let me restart nodepool | 18:46 |
| nibalizer | getting called out | 18:46 |
| phschwartz | nibalizer: lol | 18:47 |
| phschwartz | nibalizer: jeblair is helping me track down our demand issue | 18:47 |
| *** Goneri has joined #openstack-infra | 18:47 | |
| fungi | AJaeger: they should be using the same configuration entirely. they both get it from the stackalytics git repo | 18:47 |
| *** ddieterly[away] is now known as ddieterly | 18:47 | |
| phschwartz | jeblair: ok, now I see the demand line in the logs | 18:48 |
| AJaeger | strange, fungi | 18:48 |
| phschwartz | jeblair: 2016-08-04 18:46:22,499 DEBUG nodepool.NodePool: Demand from gearman: bare-trusty: 7 | 18:48 |
| phschwartz | jeblair: but the queue has cleared to alomst nothing since earlier. | 18:48 |
| phschwartz | jeblair: I might have to wait for it to get loaded down again | 18:48 |
| jeblair | phschwartz, nibalizer: yeah, you were running a version that had a malfunctioning demand calculation. | 18:49 |
| phschwartz | jeblair: oh, so this was known? | 18:49 |
| *** yottatsa has joined #openstack-infra | 18:49 | |
| fungi | (and later fixed) | 18:49 |
| jeblair | phschwartz: yep, we fixed it on jul 20. | 18:49 |
| fungi | it hit us too, which is how it was discovered | 18:49 |
| fungi | if memory serves | 18:49 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute002.vanilla metadata for better tracking https://review.openstack.org/351335 | 18:49 |
| phschwartz | ah, ok. so it should work now. awesome | 18:50 |
| jeblair | from jul 11 to jul 20, demand calcs were wrong-ish in various forms. | 18:50 |
| fungi | phschwartz: nibalizer: i'm guessing it got pinned there because of the database schema change disruption? | 18:50 |
| *** daemontool has quit IRC | 18:51 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Fix setup of server certificate for its-storyboard https://review.openstack.org/351336 | 18:51 |
| jeblair | fungi: the tag they pinned to was before this, but the pin was applied after they started running the buggy version | 18:51 |
| jeblair | so, yeah, everything should be good now :) | 18:51 |
| fungi | aha | 18:51 |
| *** yottatsa has quit IRC | 18:52 | |
| jeblair | (iow, they have reverted to before the bug was introduced) | 18:52 |
| *** daemontool has joined #openstack-infra | 18:52 | |
| AJaeger | pabelanger: I mean: https://git.openstack.org/cgit/openstack-infra/project-config/tree/grafana/nodepool-tripleo-test-cloud.yaml is a file. The sentence reads Check the grafana directory in project-config - and then gives a file URL instead of an URL for the grafana directory | 18:53 |
| AJaeger | pabelanger: but I just noticed it's wrong everywhere, so don't bother... | 18:54 |
| jeblair | well, it's not *wrong*... | 18:54 |
| jeblair | it's just extra helpful | 18:54 |
| *** rbuzatu has joined #openstack-infra | 18:55 | |
| AJaeger | jeblair: ;) | 18:55 |
| *** ddieterly is now known as ddieterly[away] | 18:55 | |
| *** pvaneck has joined #openstack-infra | 18:56 | |
| zaro | nibalizer: what's up with Vinz? you don't like the new gerrit UI either? | 18:57 |
| phschwartz | zaro: gerrit ui is bad, always has been. lol | 18:57 |
| zaro | phschwartz: i'm talking about the new new UI | 18:58 |
| phschwartz | zaro: the new new ui is still grouped in that. lol | 18:58 |
| mordred | zaro: there is a new new ui? | 18:58 |
| nibalizer | zaro: nerp | 18:59 |
| jeblair | i think zaro may be asking about the polygerrit ui | 18:59 |
| jeblair | (aka the new new *new* gerrit ui) | 18:59 |
| zaro | yes, you can try it with this chrom plugin https://chrome.google.com/webstore/detail/gerrit-ui-switcher/fdjpjnlhaohkkbjnglcfehbmcmpojklf | 18:59 |
| *** cody-somerville has joined #openstack-infra | 18:59 | |
| zaro | try it against gerrit's gerrit at https://gerrit-review.googlesource.com | 19:00 |
| *** stewie925 has quit IRC | 19:00 | |
| fungi | anteaya: who was the kaizen that you added as a wiki admin on may 13? they just mass deleted a whole bunch of user pages claiming they were spam | 19:01 |
| pabelanger | AJaeger: Oh, ya. We can fix that in a follow up patch | 19:01 |
| *** csomerville has quit IRC | 19:02 | |
| *** mdrabe_ has quit IRC | 19:03 | |
| *** mdrabe__ has joined #openstack-infra | 19:03 | |
| openstackgerrit | Richard Theis proposed openstack-infra/project-config: Add release-notes-jobs to osc-lib https://review.openstack.org/351342 | 19:03 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add tripleo-test-cloud grafana dashboard https://review.openstack.org/351251 | 19:04 |
| *** ccamacho has joined #openstack-infra | 19:05 | |
| AJaeger | pabelanger: leave it as is, seeing it everywhere, let's not change the URL - and I'm not creative in a better text... | 19:06 |
| mordred | fungi, pleia2: so - it seems that gitiles does the README rendering people keep asking for: https://chromium.googlesource.com/aosp/platform/system/bt/+/master | 19:07 |
| *** ldnunes has quit IRC | 19:07 | |
| *** fifieldt has quit IRC | 19:07 | |
| mordred | (I got sucked down a hole by looking at polygerrit) | 19:07 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-logstash: WIP: Add mqtt output plugin to logstash indexers https://review.openstack.org/351344 | 19:08 |
| mtreinish | fungi, clarkb, rcarrillocruz: ^^^ a random thing we could add to firehose | 19:08 |
| *** Goneri has quit IRC | 19:08 | |
| mtreinish | although that depends on finishing the tls bits | 19:08 |
| mtreinish | it also won't work because all the logstash docs say run 'bin/plugin' but never say what the cwd is.... | 19:09 |
| rcarrillocruz | oh nice... | 19:09 |
| mtreinish | I'm still not sure the best way to generate the certs | 19:09 |
| *** armax_ has joined #openstack-infra | 19:11 | |
| fungi | mtreinish: generate the certs. you mean for an ssl/tls server cert? | 19:11 |
| *** daemontool_ has joined #openstack-infra | 19:11 | |
| AJaeger | fungi, for constraints I consider as next steps: | 19:12 |
| *** armax has quit IRC | 19:12 | |
| *** armax_ is now known as armax | 19:12 | |
| AJaeger | 1) enable constraints for releasenotes https://review.openstack.org/#/c/349181/ We could use the new zuul-release-git-prep-upper-constraints instead of zuul-git-branch-prep-upper-constraints. | 19:12 |
| AJaeger | 2) Rename zuul-git-branch-prep-upper-constraints to zuul-release-git-prep-upper-constraints. | 19:12 |
| AJaeger | 3) Use constraints for translation sync: https://review.openstack.org/348764 | 19:12 |
| mtreinish | fungi: yeah | 19:12 |
| AJaeger | fungi, for that I need to update the changes - but first would like to hear whether you agree with the plan | 19:12 |
| fungi | mtreinish: i usually just buy them and then expense them to the foundation | 19:13 |
| *** sdake_ has joined #openstack-infra | 19:13 | |
| mtreinish | fungi: I dug through my history and when I setup my moquitto at home I used: https://github.com/owntracks/tools/blob/master/TLS/generate-CA.sh | 19:13 |
| *** javeriak has quit IRC | 19:13 | |
| zaro | mordred: ohh, i didn't even realize it but looks like there's even single page diff view on new UI. | 19:14 |
| fungi | AJaeger: i don't think we can combine zuul-git-branch-prep-upper-constraints with zuul-release-git-prep-upper-constraints (or did you mean rename it to zuul-release-git-branch-prep-upper-constraints?) | 19:14 |
| mordred | zaro: there is? how do I see it? | 19:14 |
| *** chem has quit IRC | 19:14 | |
| mordred | zaro: I went to https://gerrit-review.googlesource.com/c/79762/ with that extensoin installed | 19:14 |
| zaro | 'show diffs' link | 19:14 |
| *** daemontool has quit IRC | 19:15 | |
| mordred | AH | 19:15 |
| *** maishsk has joined #openstack-infra | 19:15 | |
| mordred | that is not obvious to me :) | 19:15 |
| *** mdrabe__ is now known as mdrabe | 19:15 | |
| mordred | but thank you - that's great! | 19:15 |
| zaro | wow, it's fast | 19:15 |
| AJaeger | fungi, I meant "mv", so basically use the new zuul-release-git-prep-upper-constraints whereever currently zuul-git-branch-prep-upper-constraints is used. | 19:15 |
| *** sdake has quit IRC | 19:16 | |
| *** thorst_ is now known as thorst | 19:16 | |
| *** javeriak has joined #openstack-infra | 19:16 | |
| mordred | zaro: that is an improvement | 19:16 |
| AJaeger | and rename it as part of that so that we only have zuul-git-branch-prep-upper-constraints | 19:16 |
| mordred | it would be great if it wasn't tied just to googlesource.com urls | 19:16 |
| fungi | AJaeger: i don't think that's possible. the zuul-git-branch-prep-upper-constraints macro is meant for periodic pipelines where we have to supply a {branch} parameter to tell it what branch to run on | 19:17 |
| mordred | zaro: I have a feeling that there are people in our community who might be interseted in running the browser extension just to get single-page-diff | 19:17 |
| *** rbuzatu has quit IRC | 19:17 | |
| AJaeger | fungi, oooh, indeed. | 19:17 |
| fungi | AJaeger: hence the "branch" in its name | 19:18 |
| zaro | mordred: it should work with any gerrit 2.12. | 19:18 |
| mordred | oh. we're not on 2.12 yet | 19:18 |
| mordred | gotit | 19:18 |
| zaro | actually googlesource.com is actually 2.13 | 19:18 |
| *** fifieldt has joined #openstack-infra | 19:18 | |
| mordred | nibalizer: if you haven't looked at polygerrit through the chrome extension zaro linked above - it's worth looking at to see what might be learned for vinz | 19:19 |
| AJaeger | fungi, yeah ;( I got confused | 19:19 |
| clarkb | do you still have to run a separate service for it? | 19:19 |
| clarkb | if so that seems like a gerrit api bug | 19:20 |
| mordred | actually | 19:20 |
| mordred | you do not need to install the chrome extensoin | 19:21 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Replace gate-{name}-tox-{envlist} with gate-{name}-tox-{envlist}-{node} https://review.openstack.org/351219 | 19:21 |
| AJaeger | fungi, checked again your comment - it was "agreed on dropping --branch in the non-branch versions of the zuul-cloner wrapper macros. i too suspected they were there as a workaround for older z-c behavior" - and I misremembered ;( | 19:21 |
| mordred | if you go to: https://canary-gerrit-review.googlesource.com/?polygerrit=1 | 19:21 |
| mordred | it will enable the polygerrit ui | 19:21 |
| fungi | AJaeger: right, we can't drop --branch from the branch parameterized versions of the macro, but we should be able to stop using it in the ones which don't take a branch parameter | 19:22 |
| AJaeger | let me propose that change then... | 19:22 |
| fungi | AJaeger: and ideally that would be the only difference i think (or we could have it set ZUUL_BRANCH instead of passing --branch to zuul-cloner) | 19:22 |
| *** ddieterly[away] is now known as ddieterly | 19:23 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute021.vanilla metadata for better tracking https://review.openstack.org/351348 | 19:23 |
| nibalizer | chorme extesinon? | 19:24 |
| nibalizer | thats funny because vinz lived for a little while as a greasemonkey script | 19:24 |
| *** thorst has quit IRC | 19:25 | |
| *** ihrachys has joined #openstack-infra | 19:25 | |
| *** rbuzatu has joined #openstack-infra | 19:25 | |
| AJaeger | fungi, that will be the case after my change | 19:26 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove unneeded --branch from zuul-cloner https://review.openstack.org/351349 | 19:27 |
| AJaeger | fungi, ^ | 19:27 |
| *** thorst has joined #openstack-infra | 19:28 | |
| *** thorst has quit IRC | 19:29 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute018.vanilla metadata for better tracking https://review.openstack.org/351350 | 19:29 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Enable constraints for releasenotes https://review.openstack.org/349181 | 19:30 |
| AJaeger | fungi, would appreciate a review of this as well ^ | 19:31 |
| *** thorst has joined #openstack-infra | 19:32 | |
| *** bhunter71 has quit IRC | 19:33 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute027.vanilla metadata for better tracking https://review.openstack.org/351352 | 19:33 |
| *** _nadya_ has joined #openstack-infra | 19:33 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Remove pinning of delorean https://review.openstack.org/349995 | 19:35 |
| AJaeger | fungi, and once https://review.openstack.org/348764 is in, we have everything ready for constraints that I'm aware of | 19:35 |
| *** javeriak has quit IRC | 19:35 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Add undercloud job in cistatus https://review.openstack.org/350763 | 19:35 |
| *** berendt has joined #openstack-infra | 19:35 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute028.vanilla metadata for better tracking https://review.openstack.org/351354 | 19:35 |
| *** karthikp_ has quit IRC | 19:36 | |
| *** tonytan_brb has joined #openstack-infra | 19:36 | |
| *** PollyZ has joined #openstack-infra | 19:36 | |
| *** Apoorva has quit IRC | 19:37 | |
| *** burgerk has quit IRC | 19:37 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute029.vanilla metadata for better tracking https://review.openstack.org/351357 | 19:38 |
| *** baoli has quit IRC | 19:38 | |
| *** tonytan4ever has quit IRC | 19:38 | |
| *** Sukhdev has quit IRC | 19:39 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add gate-tripleo-quickstart-base-centos-7-nv job https://review.openstack.org/350301 | 19:39 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute020.vanilla metadata for better tracking https://review.openstack.org/351359 | 19:40 |
| *** xnox has quit IRC | 19:41 | |
| *** baoli has joined #openstack-infra | 19:42 | |
| *** karthikp_ has joined #openstack-infra | 19:44 | |
| *** kzaitsev_mb has joined #openstack-infra | 19:45 | |
| *** kzaitsev_mb has quit IRC | 19:50 | |
| *** kzaitsev_mb has joined #openstack-infra | 19:52 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute026.vanilla metadata for better tracking https://review.openstack.org/351365 | 19:52 |
| *** maishsk has quit IRC | 19:53 | |
| *** rakhmerov has quit IRC | 19:53 | |
| *** vhosakot has quit IRC | 19:55 | |
| openstackgerrit | Richard Theis proposed openstack-infra/project-config: Add release-notes-jobs to osc-lib https://review.openstack.org/351342 | 19:55 |
| mgagne | so I'm trying to use voluptuous, thanks to grafyaml, I found good example of how to use it. Now I'm trying to find a way so one of 2 dict keys needs to be provided, maybe both I suppose but not none. | 19:55 |
| *** shamail has quit IRC | 19:56 | |
| fungi | anteaya: nevermind my question from earlier... kaizen on the wiki is apparently shamail on freenode so we've straightened it all out | 19:56 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute022.vanilla metadata for better tracking https://review.openstack.org/351366 | 19:56 |
| *** maishsk has joined #openstack-infra | 19:56 | |
| *** ayoung has joined #openstack-infra | 19:56 | |
| *** rakhmerov has joined #openstack-infra | 19:57 | |
| *** rbuzatu has quit IRC | 19:58 | |
| *** matrohon has joined #openstack-infra | 19:58 | |
| *** maishsk has quit IRC | 19:59 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute013.vanilla metadata for better tracking https://review.openstack.org/351369 | 19:59 |
| *** rockyg has quit IRC | 19:59 | |
| pabelanger | mgagne: you'll need to write your own validation function. Something like: http://git.openstack.org/cgit/openstack-infra/grafyaml/tree/grafana_dashboards/schema/panel/__init__.py#n30 but check if None | 19:59 |
| *** asettle has joined #openstack-infra | 20:00 | |
| mgagne | pabelanger: yea, I'm depth into writing custom validator atm. I guess it will be a matter of detecting the presence of those keys and updating the schema accordingly. | 20:00 |
| *** eharney has quit IRC | 20:02 | |
| ianw | greghaynes / cinerama : we meeting, or did i get my weeks messed around? | 20:02 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute010.vanilla metadata for better tracking https://review.openstack.org/351371 | 20:02 |
| greghaynes | ianw: I think next week | 20:02 |
| *** maishsk has joined #openstack-infra | 20:03 | |
| *** maishsk has quit IRC | 20:03 | |
| pabelanger | mgagne: but, v.Any() should be work too | 20:03 |
| *** mmandel has quit IRC | 20:03 | |
| cinerama | ianw: we're alternating, i still need to submit the change :P | 20:04 |
| mgagne | pabelanger: oh, I see what you mean, tyvm for the hint! | 20:04 |
| *** asettle has quit IRC | 20:04 | |
| cinerama | dib appears to be on summer holidays at the mo :) | 20:05 |
| *** cody-somerville has quit IRC | 20:05 | |
| cinerama | *(summer subject to availability in your region) | 20:05 |
| *** zul has quit IRC | 20:05 | |
| ianw | greghaynes: for https://review.openstack.org/#/c/350997/3 wouldn't it go under #openstack- ? | 20:06 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute019.vanilla metadata for better tracking https://review.openstack.org/351372 | 20:06 |
| *** rbuzatu has joined #openstack-infra | 20:06 | |
| cinerama | greghaynes: oh cool thanks for putting that forward | 20:07 |
| cinerama | also, i agree | 20:07 |
| ianw | cinerama: yeah, i've been slack on reviews, trying to get f24 working | 20:07 |
| *** vhosakot has joined #openstack-infra | 20:08 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add gate-tripleo-quickstart-base-centos-7-nv job https://review.openstack.org/350301 | 20:08 |
| *** gomarivera has joined #openstack-infra | 20:09 | |
| ianw | pabelanger: which, i just realised my cunning plan to install a custom kernel from copr might blow up as dib writes out the final image and doesn't pick up my new kernel ... have to see | 20:09 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute006.vanilla metadata for better tracking https://review.openstack.org/351374 | 20:09 |
| *** Apoorva has joined #openstack-infra | 20:10 | |
| greghaynes | cinerama: bkero did it, not I :) | 20:10 |
| *** Apoorva has quit IRC | 20:10 | |
| *** Apoorva has joined #openstack-infra | 20:10 | |
| *** vhosakot has quit IRC | 20:10 | |
| *** gomarivera has left #openstack-infra | 20:10 | |
| greghaynes | ianw: cinerama I am fine with #openstack-dib, or the proposed one, either way. One issue with #openstack-dib is someone squatted it after we mentioned that channel in our meeting | 20:11 |
| greghaynes | so well need to deal with freenode opers | 20:11 |
| ianw | ohh, presumably it's not a person we can ask to leave? | 20:12 |
| bkero | greghaynes: what did I do now? | 20:12 |
| *** vvalderrv has left #openstack-infra | 20:12 | |
| greghaynes | the channel patch | 20:12 |
| greghaynes | ianw: I asked once and didnt get a reply | 20:13 |
| bkero | oh | 20:13 |
| bkero | Yeah, I did. | 20:13 |
| greghaynes | ianw: next step is probably to figure out their email and ask that way, actually | 20:13 |
| bkero | It doesn't have gerritbot though because gerritgot has maxed channels | 20:13 |
| *** baoli has quit IRC | 20:13 | |
| *** markvoelker has quit IRC | 20:13 | |
| *** baoli has joined #openstack-infra | 20:14 | |
| *** rbuzatu has quit IRC | 20:14 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute000.vanilla metadata for better tracking https://review.openstack.org/351377 | 20:15 |
| pabelanger | jeblair: fungi: any objections to restart nodepool to pick up 349119? Updates stats to include subnodes | 20:16 |
| jeblair | greghaynes: you deal with me for that | 20:17 |
| jeblair | greghaynes: openstack-infra has the group registration for #openstack-* channels, so we can fix it. | 20:17 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute001.vanilla metadata for better tracking https://review.openstack.org/351378 | 20:18 |
| *** eharney has joined #openstack-infra | 20:19 | |
| fungi | pabelanger: i think a bunch of people were hoping to have bindep 2.0.1 from yesterday make it into centos 7 images, so maybe make sure those have all uploaded first? | 20:19 |
| *** vhosakot has joined #openstack-infra | 20:19 | |
| pabelanger | fungi: Ya, I want to let nodepool-builder finish its uploads first too | 20:20 |
| fungi | AJaeger: i'll take a look. can you set the topic on them to something consistent like "constraints"? | 20:20 |
| jeblair | pabelanger: no objection | 20:20 |
| *** sarob has quit IRC | 20:20 | |
| phschwartz | jeblair: fungi: that was defn the issue. I have 34 instances spawned by demand now. | 20:20 |
| fungi | bkero: saw your gerritbot discussion earlier. i think "sharding" could be as simple as tweaking the puppet module to run more than one gerritbot, using separate channel lists and freenode accounts. doesn't (i think) need a change to the gerritbot code at least | 20:21 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute034.vanilla metadata for better tracking https://review.openstack.org/351379 | 20:22 |
| greghaynes | jeblair: ah, awesome | 20:22 |
| pabelanger | jeblair: fungi: actually, it looks like DIBs didn't build today. Looks to be a hung process for disk-image-create is blocking things | 20:22 |
| jeblair | greghaynes: shall i start the process? | 20:22 |
| pabelanger | if you want to check nodepool.o.o | 20:22 |
| greghaynes | jeblair: That'd be great, thanks | 20:22 |
| jeblair | greghaynes: will do | 20:22 |
| *** kzaitsev_mb has quit IRC | 20:23 | |
| pabelanger | fungi: jeblair: actually, I might be wrong about that. let me check something | 20:23 |
| pabelanger | Hmm, disk-image-create is running but nodepool dib-image-list doesn't show it | 20:24 |
| greghaynes | pabelanger: I think that happens when the builder isnt grabbing the job - if it doesnt know to register as a worker for that image name, etc | 20:24 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute017.vanilla metadata for better tracking https://review.openstack.org/351380 | 20:25 |
| greghaynes | I may have foot-gunned that a day ago when adding a new image and forgetting to restart the builder ;) | 20:25 |
| fungi | could the zuul geard restart a couple days ago have caused that? | 20:25 |
| fungi | oh, got it | 20:26 |
| *** HeOS has joined #openstack-infra | 20:26 | |
| fungi | yeah, i forgot changes to the list of images needs a builder restart right now | 20:26 |
| greghaynes | yep | 20:26 |
| fungi | i guess we never did get otf reloads going for that | 20:26 |
| *** rbrndt has quit IRC | 20:28 | |
| *** ihrachys has quit IRC | 20:29 | |
| *** PollyZ has quit IRC | 20:29 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Add compute039.vanilla metadata for better tracking https://review.openstack.org/351383 | 20:30 |
| pabelanger | fungi: okay, so we can restart both nodepool and nodepool-builder I guess | 20:31 |
| *** e0ne has quit IRC | 20:31 | |
| fungi | rcarrillocruz: remember as you start bringing servers back up that we want to use a separate subdomain zone since rackspace limits the number of records we can have in a single dns zone and we're pushing pretty close to that in openstack.org already | 20:31 |
| openstackgerrit | Timothy R. Chavez proposed openstack-infra/jenkins-job-builder: Add support for the random string parameter https://review.openstack.org/351384 | 20:31 |
| *** e0ne has joined #openstack-infra | 20:32 | |
| timrc | I'm not proud of this^^ | 20:32 |
| rcarrillocruz | k | 20:33 |
| bkero | fungi: I haven't read the puppet module, but I'm pretty sure this should be very simple. I just wanted confirmation that 'gerritbot' and 'gerritbot2' sharding would be okay. | 20:33 |
| rcarrillocruz | i have a long way still to make metadata and ilo ips consolidation | 20:33 |
| rcarrillocruz | i have like...100 and something servers to go | 20:33 |
| fungi | rcarrillocruz: we have an ic.openstack.org zone already created for these, and right now only has a baremetal00.vanilla address record in it | 20:33 |
| timrc | rcarrillocruz: If you have another child, I'm going to recommend naming them ilo :) | 20:34 |
| rcarrillocruz | yeah, i think that was nibalizer that added it | 20:34 |
| *** jkilpatr has quit IRC | 20:34 | |
| rcarrillocruz | timrc: lulz | 20:34 |
| fungi | rcarrillocruz: it probably was (or maybe it was me, my memory is terrible) | 20:34 |
| fungi | i know we discussed it at length anyway | 20:34 |
| rcarrillocruz | otoh, i could call him grunt, for *grunt work* | 20:34 |
| rcarrillocruz | :D | 20:34 |
| *** Jeffrey4l has quit IRC | 20:34 | |
| *** Jeffrey4l_ has joined #openstack-infra | 20:35 | |
| jeblair | greghaynes: #openstack-dib is all set up; you have ops-level perms with chanserv | 20:36 |
| fungi | bkero: sure, making the second conffile, pidfile, logfile, et cetera be gerritbot2 and leaving the current gerritbot instance alone is fine by me. once it's puppeted we can split the current channel list in half and just let people add to which ever one and rebalance them if we ever start to fill up again before we get around to making a gerritbot replacement | 20:36 |
| bkero | fungi: sounds good to me | 20:37 |
| fungi | harlowja: that brings up an interesting irc bot challenge... we're exceeding the freenode channel max on our gerritbot account, curious if errbot has solutions for this other than "run multiple errbots" | 20:37 |
| harlowja | hmmmm | 20:37 |
| harlowja | not quite sure | 20:38 |
| fungi | gerritbot is our most popular one, but wouldn't be surprised if meetbot and eventually statusbot experience similar issues | 20:38 |
| harlowja | my guess is run multiple :-P | 20:38 |
| *** _ari_ has quit IRC | 20:39 | |
| harlowja | https://github.com/errbotio/errbot/blob/master/errbot/backends/irc.py is that backend | 20:39 |
| jpmaxman | ok fungi | 20:39 |
| fungi | like, an errbot feature allowing it to connect to multiple accounts under different nicks and somehow separate actions/triggers between them | 20:39 |
| harlowja | oh maybe, not sure | 20:39 |
| jpmaxman | the mediwiki team helped it turned out it was a stupid mis-named key on my part | 20:39 |
| fungi | jpmaxman: saw my comment earlier about the database dump being there for you? | 20:39 |
| jpmaxman | so 1.27 is working | 20:40 |
| jpmaxman | yes | 20:40 |
| fungi | awesome | 20:40 |
| jpmaxman | https://wiki-upgrade-test.openstack.org/wiki/Main_Page | 20:40 |
| jpmaxman | in the troubleshooting I ended up installing a real cert from letsencrypt | 20:40 |
| fungi | wfm | 20:40 |
| jpmaxman | the mobile view is the only extension that isn' working | 20:40 |
| *** hemanthm is now known as hemanthm|afk | 20:40 | |
| jpmaxman | but I sort of feel like we shouldn't use it anyway | 20:40 |
| *** jkilpatr has joined #openstack-infra | 20:40 | |
| jpmaxman | as it opens up a login vulnerability | 20:40 |
| fungi | i wonder if wikipedia obsoleted that for something else? | 20:40 |
| jpmaxman | maybe I can dig into it more for sure | 20:41 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: test if overcloud update can actually fail https://review.openstack.org/351391 | 20:41 |
| fungi | and yeah, i don't want more surface area to have to cover spam mitigation across | 20:41 |
| jpmaxman | but for the first time I feel like what we have on the test server *might* be better | 20:41 |
| jpmaxman | than what we have now | 20:41 |
| timrc | rcarrillocruz: You working US hours right now? Awfully late there isn't it? | 20:41 |
| jpmaxman | so I thought I'd discuss next steps | 20:41 |
| fungi | jpmaxman: i assume you had to do an in-place distro upgrade from trusty to xenial? | 20:41 |
| rcarrillocruz | well, not really | 20:41 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Update iLO IP for compute016.vanilla https://review.openstack.org/351392 | 20:41 |
| fungi | jpmaxman: between 1.26 and 1.27? | 20:42 |
| timrc | I guess only 10PMish? | 20:42 |
| rcarrillocruz | it's just that i had today an appointment at the child hospital | 20:42 |
| rcarrillocruz | and got back home at 6PM | 20:42 |
| *** dimtruck is now known as zz_dimtruck | 20:42 | |
| greghaynes | jeblair: wooo ty! | 20:42 |
| rcarrillocruz | so working a bit till my eyes close | 20:42 |
| *** mikelk has joined #openstack-infra | 20:42 | |
| timrc | rcarrillocruz: Oh noes. Hope things went well at the appt. | 20:42 |
| jpmaxman | I did an in place from precise to trusty | 20:42 |
| fungi | jpmaxman: oh, right, that's what i meant | 20:42 |
| jpmaxman | which was actually required | 20:42 |
| jpmaxman | even for 1.26 | 20:43 |
| fungi | jpmaxman: but between the 1.26 and 1.27 upgrades presumably? | 20:43 |
| rcarrillocruz | it went ok, we have another check next week | 20:43 |
| fungi | oh, before 1.26? | 20:43 |
| jpmaxman | 1.25 --> 1.26 required trusty | 20:43 |
| fungi | got it | 20:43 |
| jpmaxman | to get on the right version of php | 20:43 |
| jpmaxman | there were work arounds | 20:43 |
| fungi | just trying to get the sequence straight in my head | 20:43 |
| jpmaxman | but I figured just push forward | 20:43 |
| *** pvaneck_ has joined #openstack-infra | 20:43 | |
| *** vhosakot has quit IRC | 20:43 | |
| fungi | jpmaxman: so do you have a summary of the steps you had to follow, such that we could repeat them on the production deployment? | 20:43 |
| jpmaxman | so yeah we're running on trusty | 20:43 |
| jpmaxman | I tried to follow the conventions in place | 20:44 |
| jpmaxman | so there was slot0 and slot1 | 20:44 |
| jpmaxman | with wiki installs | 20:44 |
| *** e0ne has quit IRC | 20:44 | |
| fungi | yeah, i think mediawiki does red/blue upgrades on their servers, so that's how Ryan_Lane set it up | 20:44 |
| *** vhosakot has joined #openstack-infra | 20:44 | |
| jpmaxman | I put slot2 --> 1.25 latest, slot3 --> 1.26 latest, slot 4 1.27 latest | 20:45 |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config: Revert "Fix precedence for pass-in options" https://review.openstack.org/351394 | 20:45 |
| fungi | jpmaxman: oh, cool that works | 20:45 |
| jpmaxman | I then rain the maintenance/update.php | 20:46 |
| jpmaxman | for each version | 20:46 |
| *** pvaneck has quit IRC | 20:46 | |
| jpmaxman | to upgrade the database in steps | 20:46 |
| fungi | and did teh distro upgrade between 1.25 latest and 1.26 latest? | 20:46 |
| jpmaxman | y | 20:46 |
| jpmaxman | I moved the extensions and skins directory to /srv/mediawiki | 20:46 |
| jpmaxman | and put in symbolic links for the ones we used | 20:47 |
| fungi | did you have to do anything with the static file tree it uses for uploaded content or did it just reuse something central? | 20:47 |
| jpmaxman | into /srv/mediawiki/slot4/skins /msr/mediawiki/slot4/extensions | 20:47 |
| jpmaxman | I had to make the cache directory writable | 20:47 |
| *** tonytan_brb has quit IRC | 20:48 | |
| jpmaxman | I think the uploads go here | 20:48 |
| jpmaxman | into /srv/mediawiki/images | 20:48 |
| jpmaxman | which was already aliased and writable | 20:48 |
| jpmaxman | but we should probably test - we disabled file uploads | 20:48 |
| jpmaxman | if we want to reanable we can | 20:49 |
| thiagolib | how do I set up the cinder-volume active / passive | 20:49 |
| fungi | jpmaxman: i'm more worried about existing files still showing up. checking around now | 20:49 |
| openstackgerrit | Ben Kero proposed openstack-infra/project-config: Add channel #diskimage-builder https://review.openstack.org/350997 | 20:49 |
| jpmaxman | ok cool - I can't say I did a deep QA dive so a good idea to poke on i | 20:49 |
| jpmaxman | *it | 20:49 |
| fungi | jpmaxman: though yeah, finding out whether the new recaptcha also covers file uploads in 1.27 would be great | 20:49 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Update iLO IP for compute004.vanilla https://review.openstack.org/351395 | 20:50 |
| fungi | jpmaxman: mostly i think i just want to look around wiki-upgrade-test and then figure out some time when we can announce the wiki going offline for a while to take a fresh server snapshot (as an insurance policy) and repeat the upgrade steps on wiki.o.o | 20:50 |
| *** zz_dimtruck is now known as dimtruck | 20:50 | |
| openstackgerrit | Ben Kero proposed openstack-infra/project-config: Add channel #openstack-dib for diskimage-builder https://review.openstack.org/350997 | 20:51 |
| *** vhosakot has quit IRC | 20:51 | |
| pabelanger | jeblair: fungi: I've restarted nodepool, however I haven't see an Allocation requests since restarting. Did I miss something? | 20:51 |
| jpmaxman | I guess in the interest of getting the production wiki functional without spam as quickly as we can I'd probably leave it disabled and then iterate forward | 20:51 |
| fungi | jpmaxman: once we have production running 1.27 that gives us a stable target to try and hit with our configuration management work rather than trying to have to deal with multiple mediawiki versions we're not running | 20:51 |
| jpmaxman | yeah I agree | 20:51 |
| pabelanger | last node calculation was at 20:33UTC | 20:51 |
| jpmaxman | I'm familiar enough with it now | 20:52 |
| jpmaxman | I think we could do a fresh trusty | 20:52 |
| jpmaxman | if you didn't want to do an in place upgrade | 20:52 |
| jpmaxman | it's up to you though | 20:52 |
| fungi | jpmaxman: oh? i'd be happy to deploy a fresh trusty replacement, sure | 20:52 |
| fungi | jpmaxman: that does make me a little more comfortable than having a server upgraded from precise to trusty (i'm less concerned about the in-place upgrade of the mw pieces) | 20:53 |
| jpmaxman | It's basically a LAMP stack and copy what's in /srv/mediawiki + Apache config + MySQL | 20:53 |
| fungi | jpmaxman: awesome. i have a number of other plans to improve this (move database to trove, static content files to a cinder volume, et cetera) to make redeployments smoother but this is a great first step | 20:54 |
| jpmaxman | yeah in place MW is somewhat necessary to migrate the db | 20:54 |
| jpmaxman | cool - biggest take away is just stay on top of keeping it updated | 20:55 |
| *** raildo has quit IRC | 20:55 | |
| *** rbrndt has joined #openstack-infra | 20:55 | |
| fungi | jpmaxman: yes, agreed. that's the main reason we want to have it under configuration management. the existing deployment was not well documented and nobody'd found the time to dig into it and document it. in our case configuraiton management serves well as a summary documentation of the important parts of the system, but we didn't even have that | 20:56 |
| *** _nadya_ has quit IRC | 20:57 | |
| *** matrohon has quit IRC | 20:57 | |
| *** gomarivera has joined #openstack-infra | 20:57 | |
| fungi | jpmaxman: so anyway, this is awesome--thank you so much. let's sync up next week if you're around and figure out what sort of timeframe we should be looking at for an outage to lock down production so people aren't updating it while we upgrade/switch to the replacement server | 20:59 |
| fungi | you presumably have some feel for how long the db migrations take to complete at each stage | 20:59 |
| pabelanger | fungi: jeblair: nodes are launching now. Took almost 30mins for nodepool to start doing that after the restart | 20:59 |
| *** piet_ has quit IRC | 20:59 | |
| jpmaxman | ok great - in the man time if anyone wants to poke around on the test server and let me know if there are deal breakers for making that version the production version, it'd be great | 20:59 |
| fungi | jpmaxman: absolutely. i'll round up some test subjects | 21:00 |
| fungi | thanks again! | 21:00 |
| jpmaxman | The db migrations were fast | 21:00 |
| jpmaxman | less than a minute | 21:00 |
| jpmaxman | for each | 21:00 |
| jeblair | pabelanger: wait, why? | 21:00 |
| fungi | jpmaxman: oh, then we can probably schedule the cut-over on pretty short notice | 21:00 |
| *** jkilpatr has quit IRC | 21:00 | |
| jpmaxman | the majority of time was me troubleshooting ;) | 21:00 |
| jpmaxman | yeah | 21:00 |
| *** rhallisey has quit IRC | 21:01 | |
| pabelanger | jeblair: I am still trying to figure that out | 21:01 |
| jpmaxman | I'd give us an hour or so to be safe | 21:01 |
| *** psilvad has quit IRC | 21:01 | |
| fungi | jpmaxman: i agree, more time is better just in case of the unexpected | 21:01 |
| *** sputnik13 has quit IRC | 21:02 | |
| jpmaxman | ok I'm going to call it a night for now - will be around tomorrow if you need anything | 21:02 |
| *** baoli_ has joined #openstack-infra | 21:02 | |
| notmorgan | jeblair: *sigh* i just wasted hours wondering why my change was conflicting with everything... git branch -v shows I was on master not feature/zuu;v3 ugh | 21:03 |
| notmorgan | jeblair: just sharing, cause i'm sure that's never happened to you or anyone else :P ever | 21:03 |
| *** baoli has quit IRC | 21:04 | |
| fungi | jpmaxman: have a good night--thanks again | 21:05 |
| pabelanger | jeblair: I think I man have done something incorrectly and could use some help. In the past, we would restart nodepool and then clean up any left over nodes from jenkins. Should I have done something different with zuul-launchers? | 21:05 |
| *** yamamoto has joined #openstack-infra | 21:06 | |
| jeblair | notmorgan: nevar! :) | 21:07 |
| jeblair | pabelanger: nothing beyond restarting nodepool should be necessary | 21:08 |
| pabelanger | jeblair: or maybe our demand calculation is out of sync again? We have about 200+ ready ubuntu-trusty nodes | 21:09 |
| *** gomarivera has quit IRC | 21:09 | |
| *** thorst has quit IRC | 21:10 | |
| jeblair | pabelanger: they may be running jobs with nodepool having missed the start event | 21:11 |
| *** PollyZ has joined #openstack-infra | 21:12 | |
| *** baoli_ has quit IRC | 21:12 | |
| *** baoli has joined #openstack-infra | 21:12 | |
| *** yamamoto has quit IRC | 21:13 | |
| *** gordc has quit IRC | 21:13 | |
| pabelanger | jeblair: okay, as new jobs come online they should get picked up? or is zuul-launcher release needed? | 21:13 |
| jeblair | pabelanger: if they are running jobs, nodepool will get the complete event and delete them. release is only for idle nodes, and there should be none with our current load | 21:14 |
| pabelanger | okay thanks. I'll sit back and watch for a bit to see what happens | 21:15 |
| *** thorst has joined #openstack-infra | 21:15 | |
| notmorgan | jeblair: so.. i now have zuul running this test case but it always hits SIGLARM *and* always just spins forever even when i put my capture log hack in | 21:16 |
| notmorgan | jeblair: i am unsure where to go next here. | 21:16 |
| jeblair | notmorgan: i usually run in the foreground with python -m testtools.run tests.test_scheduler.TestScheduler.test_parallel_changes | 21:16 |
| *** ddieterly is now known as ddieterly[away] | 21:17 | |
| *** signed8bit is now known as signed8bit_Zzz | 21:17 | |
| *** baoli has quit IRC | 21:18 | |
| *** julim has quit IRC | 21:18 | |
| notmorgan | jeblair: thnx. | 21:18 |
| *** julim has joined #openstack-infra | 21:18 | |
| *** baoli has joined #openstack-infra | 21:18 | |
| *** Sukhdev has joined #openstack-infra | 21:18 | |
| *** eharney has quit IRC | 21:19 | |
| *** thorst has quit IRC | 21:19 | |
| *** signed8bit_Zzz is now known as signed8bit | 21:21 | |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config: Pass the argparse data into to validate_auth https://review.openstack.org/351405 | 21:22 |
| *** julim has quit IRC | 21:23 | |
| thiagolib | geguileo: hi i am trying to set the cinder - volume active / passive , and would like to know what I need to configure in cinder.conf and also how should I set the cinder - volume service to start on both nodes . | 21:24 |
| pabelanger | jeblair: I'm starting to think we have a problem with nodepool.o.o, we have 123 ubuntu-xenial nodes that are ready, with pending jobs in the check and gate queues | 21:25 |
| *** baoli has quit IRC | 21:26 | |
| *** sarob has joined #openstack-infra | 21:26 | |
| *** mikelk has quit IRC | 21:26 | |
| *** amotoki has quit IRC | 21:27 | |
| fungi | pabelanger: log into a couple and see if there's anything in ~jenkins/workspace/? | 21:28 |
| *** priteau has quit IRC | 21:28 | |
| pabelanger | fungi: yes, it appears jobs are running on some of them | 21:29 |
| fungi | might be zeromq is letting us down again? | 21:29 |
| *** yamahata has quit IRC | 21:30 | |
| pabelanger | how can I check that? | 21:30 |
| *** xnox has joined #openstack-infra | 21:30 | |
| *** lbeliveau has joined #openstack-infra | 21:30 | |
| pabelanger | ya, a random sample of 5 nodes, show they are all running / ran jobs | 21:30 |
| *** sarob has quit IRC | 21:31 | |
| *** matt-borland has quit IRC | 21:31 | |
| fungi | the nodepool debug log claims to be setting some to used or delete | 21:31 |
| fungi | so maybe it's just some launchers | 21:31 |
| fungi | i'm running some quick stats across them | 21:31 |
| *** pcarver has joined #openstack-infra | 21:32 | |
| *** sputnik13 has joined #openstack-infra | 21:32 | |
| jeblair | i see zmq events from all 7 | 21:32 |
| fungi | ready node distribution across launchers looks pretty consistent too | 21:32 |
| pabelanger | 3194612 is running a job | 21:33 |
| pabelanger | but nodepool still thinks it is ready | 21:33 |
| jeblair | pabelanger: if what i suggested happened earlier has happened, i would not expect it to correct itself for about an hour | 21:33 |
| pabelanger | jeblair: okay | 21:33 |
| jeblair | pabelanger: they will have to run and complete jobs before nodepool updates their status | 21:34 |
| openstackgerrit | Paul Carver proposed openstack-infra/project-config: Enable api-ref publishing for networking-sfc https://review.openstack.org/351407 | 21:34 |
| fungi | yeah, chances are we can just consider the ready node count to be part of the used node count for the next little while | 21:34 |
| pabelanger | sure, if both are good with that, I am too | 21:34 |
| jeblair | pabelanger: if you want to confirm this, you could parse the zuul status json and see how many jobs are actually running, and see if that number approximates ready+used | 21:34 |
| *** xnox has quit IRC | 21:35 | |
| jeblair | oh, the gearman graph does tell us that | 21:35 |
| jeblair | it says 266 are running | 21:36 |
| fungi | where we were flat at around 500 before the restart | 21:36 |
| *** dimtruck is now known as zz_dimtruck | 21:36 | |
| jeblair | so there may be something else at play | 21:36 |
| jeblair | fungi: i'd probably go with 450 for than number | 21:37 |
| fungi | sure, i was looking at the tiny graph where a 10% deviation is tough to spot | 21:37 |
| jeblair | ah heh | 21:37 |
| notmorgan | jeblair: so I'm feeling really dumb right now.did zuul in v2 just roundrobin the connections to the same gerrit? i'm looking at testConnections and don't see how it used the second connection (civoter) previously since both "approvals" were sent with self.fake_review_gerrit | 21:39 |
| pabelanger | We restart at 20:33UTC so were an hour in right now | 21:39 |
| pabelanger | jeblair: fungi: I'll have to step away shortly for a family dinner but will be able to follow up once I get back | 21:39 |
| fungi | the other disturbing behavior i see is, running a count of ready nodes reported by nodepool list, the total number across all types isn't really decreasing (hovering between 280 and 285) | 21:40 |
| fungi | no, wait, it just took a largeish drop to about 270 | 21:40 |
| fungi | now it's falling somewhat steadily so maybe it is self-correcting | 21:41 |
| notmorgan | oh nvm i see it now | 21:41 |
| notmorgan | sigh | 21:41 |
| notmorgan | brain... not connecting some strings. | 21:41 |
| openstackgerrit | Merged openstack-infra/infra-specs: Making space for new lead https://review.openstack.org/351326 | 21:41 |
| pabelanger | fungi: ya, used nodes is starting to tick up | 21:41 |
| stevemar | can we get views to set up release notes setup for osc-lib https://review.openstack.org/#/c/351342/2 we're eager to release it soon | 21:42 |
| pabelanger | okay, have to run. Will check back in and see the results | 21:42 |
| fungi | thanks pabelanger | 21:42 |
| jeblair | notmorgan: the approvals go via different connections (cause they are specified in the layout) -- the only thing the test does is create the fake changes, it doesn't leave any approvals in that one | 21:42 |
| *** ayoung has quit IRC | 21:43 | |
| *** zz_dimtruck is now known as dimtruck | 21:43 | |
| notmorgan | jeblair: yeah now i just need to figure out why the failure is getting "event.type = data.get('type') with a NoneType object has no attribute.get | 21:43 |
| *** kzaitsev_mb has joined #openstack-infra | 21:48 | |
| *** ddieterly[away] is now known as ddieterly | 21:48 | |
| *** yamahata has joined #openstack-infra | 21:49 | |
| *** amotoki has joined #openstack-infra | 21:53 | |
| *** esberglu has quit IRC | 21:54 | |
| *** thorst has joined #openstack-infra | 21:56 | |
| *** dprince has quit IRC | 21:59 | |
| *** amitgandhinz has quit IRC | 21:59 | |
| *** mdrabe has quit IRC | 22:02 | |
| *** thorst has quit IRC | 22:03 | |
| asselin | rcarrillocruz, you around? I'm having issues with ansible-role-cloud-launcher linters: ERROR! the role 'ansible-role-cloud-launcher' was not found in http://paste.openstack.org/show/549308/ | 22:03 |
| *** gouthamr has quit IRC | 22:05 | |
| *** thorst has joined #openstack-infra | 22:05 | |
| *** gouthamr has joined #openstack-infra | 22:05 | |
| *** gouthamr has quit IRC | 22:05 | |
| jeblair | i think the disk on logstash.openstack.org is full | 22:05 |
| *** PollyZ has quit IRC | 22:05 | |
| *** sarob has joined #openstack-infra | 22:07 | |
| *** thorst has quit IRC | 22:07 | |
| *** thorst has joined #openstack-infra | 22:08 | |
| fungi | after looking at df, i am inclined to believe you | 22:08 |
| fungi | looks like logs | 22:09 |
| fungi | ot/var/log/logprocessor/log-client-debug.log* | 22:09 |
| fungi | er | 22:10 |
| fungi | it's /var/log/logprocessor/log-client-debug.log* | 22:10 |
| *** karthikp_ has quit IRC | 22:10 | |
| fungi | looks like it may have ceased rotating | 22:10 |
| fungi | yeah, hasn't rotated since july 12 | 22:10 |
| fungi | up to then it was rotating daily | 22:11 |
| *** signed8bit has quit IRC | 22:11 | |
| asselin | rcarrillocruz, this rolename is unexpected: http://logs.openstack.org/23/322823/13/gate/gate-ansible-role-cloud-launcher-linters/2893e4b/console.html.gz#_2016-07-20_15_13_29_041705 | 22:12 |
| fungi | clarkb: i know you're not here, but should i try to preserve the last several weeks of log-client-debug.log? | 22:13 |
| clarkb | fungi no its not really valuable | 22:14 |
| fungi | okay, i'll blow it away and see if it starts logging anything new and alarming | 22:14 |
| fungi | i have a feeling something caused it to fail to rotate briefly and then it got too big to rotate | 22:14 |
| fungi | i'm rebooting it now since when / fills up you never really know what else can go sideways | 22:15 |
| fungi | will try to keep tabs on the logs and make sure they go back to rotating normally and not growing crazily | 22:16 |
| *** karthikp_ has joined #openstack-infra | 22:16 | |
| *** thorst has quit IRC | 22:16 | |
| *** PollyZ has joined #openstack-infra | 22:17 | |
| anteaya | fungi: glad you got the wiki username sorted out | 22:18 |
| anteaya | fungi: I was offline | 22:18 |
| *** piet_ has joined #openstack-infra | 22:18 | |
| fungi | no worries. it's all good. thanks for checking in! | 22:18 |
| anteaya | should he not have deleted those pages? | 22:18 |
| *** jcoufal_ has quit IRC | 22:19 | |
| fungi | anteaya: well, not without further discussion at least. for example i was surprised to see my user page suddenly deleted. others may have been as well | 22:19 |
| anteaya | fungi: fair enough, did you want them reinstated? | 22:19 |
| fungi | i took care of it already | 22:20 |
| anteaya | he was uncertain and had been looking for direciton previously | 22:20 |
| anteaya | sorry I didn't read all the backscroll | 22:20 |
| fungi | i gave him some tips on how i've been checking for spammers, and how to easily block and clean up after them | 22:20 |
| fungi | it was in privmsg, he wasn't in channel | 22:21 |
| *** mriedem has quit IRC | 22:21 | |
| anteaya | ah okay thanks for the status update | 22:21 |
| anteaya | I appreciate it | 22:21 |
| *** xyang1 has quit IRC | 22:22 | |
| asselin | rcarrillocruz, weird...I don't know why I need to use ../.. here: http://git.openstack.org/cgit/openstack/ansible-role-cloud-launcher/tree/tests/ansible.cfg#n3 Isn't tox -e linters run from the top-level folder of the repo? | 22:23 |
| *** xnox has joined #openstack-infra | 22:24 | |
| *** ayoung has joined #openstack-infra | 22:26 | |
| openstackgerrit | Aaron Rosen proposed openstack-dev/cookiecutter: sync latest requirements.txt https://review.openstack.org/351417 | 22:26 |
| jhesketh | Morning | 22:27 |
| anteaya | morning jhesketh | 22:27 |
| *** kzaitsev_mb has quit IRC | 22:27 | |
| *** zul has joined #openstack-infra | 22:28 | |
| jeblair | fungi, pabelanger: i'm seeing a lot of errors regarding deleting subnodes from the db. i'm going to assume something is wrong with that patch, revert it, and restart nodepool | 22:30 |
| *** furlongm_ has quit IRC | 22:30 | |
| *** furlongm_ has joined #openstack-infra | 22:31 | |
| fungi | jeblair: 349119? yeah, agreed that seems likely | 22:31 |
| jeblair | the rate of "Exception deleting subnode" errors is much higher than yesterday | 22:32 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Revert "Include subnodes in updateStats function" https://review.openstack.org/351419 | 22:32 |
| fungi | also the ready count has only fallen to ~200 so far | 22:32 |
| *** rfolco has quit IRC | 22:32 | |
| openstackgerrit | Merged openstack-infra/nodepool: Revert "Include subnodes in updateStats function" https://review.openstack.org/351419 | 22:32 |
| rhedlind | Updated nodepool to the latest from an ancient verison. Anyone willing to share an example of a working nodepool secure.conf? This one causes nodepool to throw an exception on the dburi line: http://paste.openstack.org/show/549312/ | 22:33 |
| asselin | rhedlind, are you using puppet-openstackci? | 22:34 |
| *** PollyZ has quit IRC | 22:34 | |
| *** dimtruck is now known as zz_dimtruck | 22:34 | |
| jeblair | restarted nodepool with the revert in place | 22:34 |
| *** pradk has quit IRC | 22:34 | |
| *** PollyZ has joined #openstack-infra | 22:36 | |
| fungi | rhedlind: in production we use dburi=mysql+pymysql://someuser:somepassword@localhost/nodepool | 22:37 |
| fungi | rhedlind: obviously with a different username and password | 22:37 |
| rhedlind | asselin: No, the Ci was installed a while back. I got it back to a working state earlier this week. Everything ran great until zuul was killed due to an out of memory issue. After that zuul refused to restart. I suspect it was because the apscheduler was not the right version. | 22:37 |
| rhedlind | upgrading apscheduler caused nodepool to freak out instead, so I decided to upgrade nodepool as well | 22:38 |
| rhedlind | and now I am hitting the issue with secure.conf | 22:38 |
| *** edmondsw has quit IRC | 22:40 | |
| fungi | rhedlind: you're likely running into the (not particularly recent) shift from mysql-python to pymysql | 22:40 |
| asselin | rhedlind, and some database schema changes if I recall | 22:40 |
| asselin | rhedlind, secure.conf is here:http://git.openstack.org/cgit/openstack-infra/puppet-nodepool/tree/templates/secure.conf.erb | 22:40 |
| asselin | rhedlind, but I suggest you try to downgrade your nodepool back to where it was before | 22:41 |
| asselin | rhedlind, try one of the tags: http://git.openstack.org/cgit/openstack-infra/nodepool | 22:42 |
| fungi | yeah, and if you do go with a significant jump in nodepool versions you'll likely just end up having to drop your tables and let nodepoold recreate them. and if you do that you'll likely need to manually clean up old images (and floating ips too if you use them) | 22:43 |
| *** camunoz has joined #openstack-infra | 22:43 | |
| fungi | and probably old server instances if the old version od nodepool you were using predates the automatic alien cleanup metadata on instances | 22:44 |
| rhedlind | removing the ' ' helped! I followed the instructions on http://docs.openstack.org/infra/nodepool/configuration.html#configuration that includes ' ' | 22:44 |
| rhedlind | I needed to upgrade nodepool so I can run a newer version of apscheduler | 22:44 |
| asselin | rhedlind, if I remember right...secure.conf isn't required and the old way still works... | 22:45 |
| rhedlind | asselin: ok | 22:45 |
| zaro | fungi: i've added the force online reindex change to 'gerrit-upgrade' topic. https://review.openstack.org/#/c/351418 | 22:46 |
| fungi | zaro: oh, did they finally agree to that solution? | 22:47 |
| fungi | the upstream change you've linked in the commit message goes to a 404 page | 22:47 |
| zaro | surprisingly yes | 22:47 |
| rhedlind | Do the newer versions of nodepool only support building images using disk-image builder? http://paste.openstack.org/show/549313/ | 22:47 |
| *** rbrndt has quit IRC | 22:47 | |
| asselin | rhedlind, probably...there was discussion about that in the mailing list. 3rd party ci folks are supposed to pin to known versions. | 22:48 |
| asselin | rhedlind, http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/contrib/single_node_ci_data.yaml | 22:49 |
| asselin | rhedlind, there's also other changes going on for zuulv3 that you'll want to pin to what's listed here ^^ | 22:49 |
| fungi | rhedlind: asselin: http://lists.openstack.org/pipermail/openstack-infra/2016-July/004526.html | 22:50 |
| zaro | fungi: oops, just fixed it | 22:50 |
| fungi | thanks zaro! | 22:50 |
| zaro | that was a feature and a bug fix in one | 22:50 |
| fungi | indeed! | 22:51 |
| anteaya | zaro: have you tested online force reindex yet? | 22:51 |
| *** sdague has quit IRC | 22:52 | |
| zaro | anteaya: i've tested on my local machine | 22:52 |
| fungi | i guess if we approve 351418 then we can upgrade review-dev to the new build and exercise it there too | 22:52 |
| rhedlind | asselin: fungi: thanks for your help! I will look into the links above and see if I can straighten this out. | 22:52 |
| zaro | anteaya: fungi has the right plan | 22:52 |
| *** kien-ha has joined #openstack-infra | 22:52 | |
| fungi | zaro: that will make our gerrit maintenances _soooo_ much smoother | 22:54 |
| fungi | thanks for hacking out a solution upstream would agree to! | 22:54 |
| *** kzaitsev_mb has joined #openstack-infra | 22:54 | |
| *** kien-ha has quit IRC | 22:55 | |
| zaro | np, i think we all are happier with less downtime. | 22:55 |
| *** kgiusti has left #openstack-infra | 22:56 | |
| zaro | now if we can get that memory issue under control... | 22:56 |
| *** kien-ha has joined #openstack-infra | 22:56 | |
| *** ddieterly is now known as ddieterly[away] | 22:58 | |
| *** ddieterly[away] has quit IRC | 22:58 | |
| openstackgerrit | Craige McWhirter proposed openstack-infra/infra-specs: Initial commit of the Pholio spec https://review.openstack.org/340641 | 22:58 |
| anteaya | fungi: zaro ah wonderful | 22:59 |
| anteaya | less downtime ++ | 22:59 |
| anteaya | zaro: nice work here | 22:59 |
| anteaya | you are so persistent :) | 22:59 |
| fungi | especially for the upcoming storyboard-its plugin reindexing, but also for teh pending project rename request we've been putting off because we don't want a 4-hour downtime just yet | 23:00 |
| *** Teresita-Warrior has quit IRC | 23:00 | |
| *** amitgandhinz has joined #openstack-infra | 23:00 | |
| *** kzaitsev_mb has quit IRC | 23:01 | |
| zaro | anteaya: yeah, don't give me a reason to come after you. it won't end.. :) | 23:02 |
| *** sarob has quit IRC | 23:02 | |
| anteaya | ha ha ha | 23:02 |
| anteaya | I don't think I ever have | 23:02 |
| *** pahuang has joined #openstack-infra | 23:02 | |
| anteaya | nor is it on my todo list | 23:02 |
| anteaya | :) | 23:02 |
| *** asettle has joined #openstack-infra | 23:03 | |
| *** ams__ has quit IRC | 23:04 | |
| *** amitgandhinz has quit IRC | 23:05 | |
| *** ayoung has quit IRC | 23:06 | |
| *** asettle has quit IRC | 23:07 | |
| *** Swami has quit IRC | 23:09 | |
| *** kzaitsev_mb has joined #openstack-infra | 23:13 | |
| *** thorst has joined #openstack-infra | 23:15 | |
| pabelanger | jeblair: fungi: thanks, I'll dig into the patch tomorrow to see why it would be causing problems with deleting sub nodes. | 23:17 |
| *** dingyichen has joined #openstack-infra | 23:18 | |
| *** kzaitsev_mb has quit IRC | 23:19 | |
| *** thorst has quit IRC | 23:22 | |
| *** thiagop has quit IRC | 23:23 | |
| fungi | though we do still seem to be running a glut of ready nodes | 23:27 |
| jeblair | yeah just starting to look again | 23:27 |
| jeblair | however, none of them have a recent age | 23:28 |
| jeblair | so there's a possibility they are all from the buggy time. | 23:28 |
| *** markvoelker has joined #openstack-infra | 23:28 | |
| *** amotoki has quit IRC | 23:28 | |
| *** PollyZ has quit IRC | 23:28 | |
| pabelanger | grep "deleting subnode" /var/log/nodepool/nodepool.log.2016-08-04_08 | wc -l | 23:29 |
| pabelanger | returned 471 instances | 23:29 |
| pabelanger | I think our exceptions for deleting subnodes might be something other then our recent patch | 23:29 |
| jeblair | i see 2 nodes that have been ready for < 50 minutes. | 23:29 |
| jeblair | pabelanger: deleting subnodes is normal | 23:29 |
| jeblair | pabelanger: Exceptions deleting subnodes is less normal | 23:30 |
| *** zz_dimtruck is now known as dimtruck | 23:30 | |
| *** jamielennox is now known as jamielennox|away | 23:30 | |
| jeblair | i'm going to assume all ready nodes with an age > 1 hour are done and delete them | 23:31 |
| pabelanger | jeblair: right, our exceptions have been high today, 471+ exceptions even before we landed our patch | 23:31 |
| pabelanger | going to see if I can find a timeframe it started at | 23:31 |
| fungi | jeblair: if they happen to actually be running jobs, zuul will presumably reschedule them anyway, right? | 23:31 |
| jeblair | fungi: yeah | 23:31 |
| fungi | seems like a safe measure in that case | 23:31 |
| *** hichihara has joined #openstack-infra | 23:32 | |
| jeblair | pabelanger: ok. i will just have a lot more confidence in your result if your grep is for " Exception deleting subnode" not "deleting subnode". | 23:32 |
| fungi | i was looking at long-running ready nodes as a signal of similar loss of state changes and there are a handful (~30) running for 3+ hours | 23:32 |
| pabelanger | jeblair: agreed | 23:32 |
| jeblair | before the restart, i found ready nodes with an age of ~10 minutes which had started running jobs | 23:33 |
| jeblair | so somehow nodepool was missing state changes | 23:33 |
| fungi | actually, up to 43 now running >3 hours | 23:33 |
| fungi | so i tend to concur | 23:33 |
| *** mrmartin has quit IRC | 23:34 | |
| fungi | i suggest that in addition to nodes ready for >1h, nodes in any state >3h are likely showing stale states | 23:34 |
| pabelanger | Ya, I'm at a lose to explain why that could be | 23:34 |
| pabelanger | http://paste.openstack.org/show/549318/ first 8 hours of subnode deleting exceptions | 23:34 |
| jeblair | i spot checked one, and z-l claimed to have sent the zmq message, but nodepool did not log receiving it. yet nodepool did log other messages from the same host around the same time. | 23:35 |
| pabelanger | way up from previous days | 23:35 |
| jeblair | it doesn't make sense, but with so many database errors, i'm not assuming sense at the moment. | 23:35 |
| fungi | pabelanger: jeblair: there was also a network incident in dfw today, which could account for some earlier failures (though not the more recent loss of state transitions) | 23:36 |
| fungi | https://status.rackspace.com/index/viewincidents?group=19&start=1470283200 | 23:36 |
| pabelanger | fungi: ack | 23:37 |
| fungi | 04:20-11:15 cdt, so 09:20-16:15 utc | 23:37 |
| *** markvoelker_ has joined #openstack-infra | 23:37 | |
| *** berendt has quit IRC | 23:38 | |
| fungi | though that looks more like it would have impacted build tasks rather than delete tasks | 23:38 |
| jeblair | okay, there are only 2 ready nodes now, and they are at 56 minutes | 23:39 |
| jeblair | i'll delete them in 10 mins or so. we'll see if there are any more leaks | 23:39 |
| *** markvoelker has quit IRC | 23:41 | |
| *** xarses has quit IRC | 23:41 | |
| *** jamielennox|away is now known as jamielennox | 23:41 | |
| fungi | shall i delete nodes in a used state >3 hours? | 23:41 |
| jeblair | fungi: ++ | 23:42 |
| fungi | we're up to 45 of them now | 23:42 |
| *** Jeffrey4l_ has quit IRC | 23:42 | |
| fungi | and here they go | 23:43 |
| fungi | they too seemed to be of relatively expected type breakdown and evenly distributed across launchers | 23:43 |
| jeblair | pabelanger: i wonder if len(subnodes) takes a long time doing sql | 23:44 |
| *** sdake has joined #openstack-infra | 23:44 | |
| pabelanger | jeblair: Ya, I was thinking that too | 23:44 |
| jeblair | pabelanger: could probably just use the configuration instead of doing that -- say, this node is type 'devstack-trusty-2-node' and that label has one subnode, so add 2. | 23:45 |
| *** Jeffrey4l has joined #openstack-infra | 23:45 | |
| *** sdake_ has quit IRC | 23:46 | |
| pabelanger | jeblair: sure, let me see how easy that would be | 23:47 |
| zaro | fungi: where is that db script we use to update project names? | 23:48 |
| jeblair | zaro: http://docs.openstack.org/infra/system-config/gerrit.html#renaming-a-project | 23:48 |
| zaro | thanks | 23:48 |
| *** kzaitsev_mb has joined #openstack-infra | 23:51 | |
| pabelanger | jeblair: I think we could do: states[key] += 1 + self.config.labels[node.label_name].subnodes | 23:52 |
| pabelanger | seems to be the way other functions calculate subnodes counts | 23:53 |
| *** sflanigan has joined #openstack-infra | 23:54 | |
| *** zhurong has joined #openstack-infra | 23:55 | |
| *** thorst has joined #openstack-infra | 23:56 | |
| *** weshay has quit IRC | 23:56 | |
| *** piet_ has quit IRC | 23:56 | |
| *** thorst has quit IRC | 23:57 | |
| *** thorst has joined #openstack-infra | 23:57 | |
| *** esberglu has joined #openstack-infra | 23:59 | |
| *** zul has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!