| *** Apoorva has quit IRC | 00:01 | |
| *** Apoorva_ has quit IRC | 00:02 | |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify User Update https://review.openstack.org/499284 | 00:04 |
|---|---|---|
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: WIP Add wheel mirror jobs https://review.openstack.org/499843 | 00:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Ansiblify the wheel build a little more https://review.openstack.org/499857 | 00:04 |
| mordred | jeblair, pabelanger: ^^ there is that taking a single-playbook multi-host approach :) | 00:05 |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify Add User to Group https://review.openstack.org/499345 | 00:06 |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify Check User in Group https://review.openstack.org/499357 | 00:06 |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify Remove User from Group https://review.openstack.org/499360 | 00:07 |
| *** sflanigan has joined #openstack-infra | 00:07 | |
| *** sflanigan has joined #openstack-infra | 00:07 | |
| *** rlandy|bbl is now known as rlandy | 00:08 | |
| *** rlandy has quit IRC | 00:10 | |
| *** bh526r has quit IRC | 00:17 | |
| *** xarses has quit IRC | 00:18 | |
| *** gouthamr has joined #openstack-infra | 00:18 | |
| *** kjackal has quit IRC | 00:19 | |
| *** kjackal has joined #openstack-infra | 00:19 | |
| *** kjackal has quit IRC | 00:24 | |
| *** kjackal has joined #openstack-infra | 00:24 | |
| *** Julien-z_ has quit IRC | 00:25 | |
| *** ijw has joined #openstack-infra | 00:26 | |
| *** Julien-z_ has joined #openstack-infra | 00:31 | |
| *** ijw has quit IRC | 00:31 | |
| *** ijw has joined #openstack-infra | 00:32 | |
| *** slaweq has joined #openstack-infra | 00:34 | |
| *** slaweq has quit IRC | 00:39 | |
| *** thorst_afk has joined #openstack-infra | 00:39 | |
| ianw | is there a reason we're not using ansible 2.3 on hosts? | 00:44 |
| *** thorst_afk has quit IRC | 00:44 | |
| *** iyamahat has quit IRC | 00:45 | |
| clarkb | ianw: for puppeting? in thr past its been each ansible release has broken ansible-puppet | 00:45 |
| clarkb | probably a good idea to test the kick script with 2.3 and upgrade if it works | 00:46 |
| ianw | clarkb: no i mean for devstack-gating in /tmp/ansible virtualenv | 00:46 |
| *** Swami has quit IRC | 00:46 | |
| *** LindaWang has joined #openstack-infra | 00:46 | |
| ianw | just the parted module is new in there, and i'd prefer to not write custom parted stuff if possible | 00:46 |
| *** thorst_afk has joined #openstack-infra | 00:47 | |
| clarkb | dmsimard: had changes up already to upgrade there | 00:47 |
| jeblair | ianw: dmsimard wanted to upgrade recently, though i asked him to hold off since it wasn't necessary for the v3 transition, which would obviate the need shortly anyway. if there's a legit reason we need to, i don't object. | 00:47 |
| *** thorst_afk has quit IRC | 00:47 | |
| clarkb | have they not merged? | 00:47 |
| clarkb | ah | 00:47 |
| ianw | well yeah, i'm somewhat a way off having all this swap stuff really ansibalised (is that the word we're using) but would be helpful | 00:48 |
| *** thorst_afk has joined #openstack-infra | 00:49 | |
| *** thorst_afk has quit IRC | 00:49 | |
| *** mriedem has quit IRC | 00:51 | |
| *** baoli has joined #openstack-infra | 00:53 | |
| *** baoli has quit IRC | 00:54 | |
| *** baoli has joined #openstack-infra | 00:55 | |
| *** vhosakot_ has quit IRC | 00:57 | |
| *** mriedem has joined #openstack-infra | 01:02 | |
| *** mat128 has joined #openstack-infra | 01:04 | |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: add a devstack-legacy job https://review.openstack.org/497699 | 01:08 |
| *** ijw has quit IRC | 01:08 | |
| *** zhurong has joined #openstack-infra | 01:12 | |
| *** hongbin has joined #openstack-infra | 01:13 | |
| dmsimard | clarkb, ianw: we're getting occurrences of xvde1 and xvde2 device busy errors (seen one of each) so I sent https://review.openstack.org/#/c/499687/ | 01:20 |
| ianw | urgh | 01:20 |
| dmsimard | I have a logstash query somewhere | 01:21 |
| dmsimard | message:"/dev/xvde" AND message:"Device or resource busy while setting up superblock" | 01:22 |
| dmsimard | That's for the ext partition | 01:23 |
| ianw | dmsimard: you know i'm going to ask ... why? :) | 01:24 |
| dmsimard | Error: Error informing the kernel about modifications to partition /dev/xvde1 -- Device or resource busy. This means Linux won't know about any changes you made to /dev/xvde1 until you reboot -- so you shouldn't mount it or use it in any way before rebooting. | 01:24 |
| dmsimard | Error: Failed to add partition 1 (Device or resource busy) | 01:24 |
| dmsimard | That's the other signature | 01:24 |
| ianw | devicemapper maybe? | 01:24 |
| dmsimard | I dunno. I guess we may have been overconfident in taking out sync and partprobe | 01:25 |
| dmsimard | Hence why I'd like to add them back | 01:25 |
| dmsimard | That's why I wanted to recheck a few times :( | 01:27 |
| dmsimard | But not seeing any issues spanning 16 nodes on 3 distros... Bleh | 01:28 |
| *** slaweq has joined #openstack-infra | 01:35 | |
| *** slaweq has quit IRC | 01:40 | |
| *** rama_y has joined #openstack-infra | 01:46 | |
| *** rama_y has quit IRC | 01:47 | |
| *** rama_y has joined #openstack-infra | 01:47 | |
| *** priteau has joined #openstack-infra | 01:50 | |
| *** thorst_afk has joined #openstack-infra | 01:54 | |
| *** priteau has quit IRC | 01:55 | |
| *** thorst_afk has quit IRC | 01:59 | |
| *** yamahata has joined #openstack-infra | 02:02 | |
| *** thorst_afk has joined #openstack-infra | 02:05 | |
| *** thorst_afk has quit IRC | 02:05 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add an experimental functional job for Qinling https://review.openstack.org/499567 | 02:06 |
| *** ykarel has joined #openstack-infra | 02:15 | |
| *** rama_y has quit IRC | 02:20 | |
| *** esberglu has joined #openstack-infra | 02:20 | |
| *** esberglu has quit IRC | 02:20 | |
| *** zhurong has quit IRC | 02:28 | |
| *** thegreenhundred has joined #openstack-infra | 02:31 | |
| *** thorst_afk has joined #openstack-infra | 02:36 | |
| *** thorst_afk has quit IRC | 02:41 | |
| *** slaweq has joined #openstack-infra | 02:49 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Copy contents inside html folder for python-docs job https://review.openstack.org/499876 | 02:51 |
| *** zhurong has joined #openstack-infra | 02:53 | |
| *** ykarel has quit IRC | 02:53 | |
| *** slaweq has quit IRC | 02:55 | |
| *** ykarel has joined #openstack-infra | 03:03 | |
| *** nicolasbock has quit IRC | 03:07 | |
| *** ramishra has joined #openstack-infra | 03:08 | |
| *** esberglu has joined #openstack-infra | 03:14 | |
| *** esberglu has quit IRC | 03:18 | |
| *** ykarel has quit IRC | 03:18 | |
| *** mriedem has quit IRC | 03:23 | |
| *** baoli has quit IRC | 03:26 | |
| *** baoli has joined #openstack-infra | 03:27 | |
| *** baoli has quit IRC | 03:31 | |
| *** links has joined #openstack-infra | 03:32 | |
| *** iyamahat has joined #openstack-infra | 03:33 | |
| *** slaweq has joined #openstack-infra | 03:37 | |
| *** udesale has joined #openstack-infra | 03:37 | |
| *** thorst_afk has joined #openstack-infra | 03:38 | |
| *** psachin has joined #openstack-infra | 03:41 | |
| *** slaweq has quit IRC | 03:42 | |
| *** thorst_afk has quit IRC | 03:42 | |
| *** baoli has joined #openstack-infra | 03:51 | |
| *** priteau has joined #openstack-infra | 03:51 | |
| *** gouthamr has quit IRC | 03:53 | |
| *** iyamahat_ has joined #openstack-infra | 03:55 | |
| *** priteau has quit IRC | 03:56 | |
| *** iyamahat has quit IRC | 03:59 | |
| *** hongbin has quit IRC | 04:07 | |
| *** ykarel has joined #openstack-infra | 04:12 | |
| *** claudiub has joined #openstack-infra | 04:17 | |
| *** yamamot__ has quit IRC | 04:28 | |
| *** iyamahat_ has quit IRC | 04:28 | |
| *** iyamahat_ has joined #openstack-infra | 04:29 | |
| *** baoli has quit IRC | 04:34 | |
| *** yamamoto_ has joined #openstack-infra | 04:34 | |
| *** slaweq has joined #openstack-infra | 04:38 | |
| *** thorst_afk has joined #openstack-infra | 04:38 | |
| *** dhajare has joined #openstack-infra | 04:41 | |
| *** slaweq has quit IRC | 04:43 | |
| *** thorst_afk has quit IRC | 04:43 | |
| *** mat128 has quit IRC | 05:07 | |
| *** iyamahat_ has quit IRC | 05:13 | |
| *** yamamoto_ has quit IRC | 05:17 | |
| *** yamamoto_ has joined #openstack-infra | 05:19 | |
| *** Douhet has quit IRC | 05:27 | |
| *** Douhet has joined #openstack-infra | 05:28 | |
| *** thorst_afk has joined #openstack-infra | 05:39 | |
| *** thorst_afk has quit IRC | 05:44 | |
| *** links has quit IRC | 05:50 | |
| *** priteau has joined #openstack-infra | 05:52 | |
| *** marst has joined #openstack-infra | 05:52 | |
| *** cuongnv has joined #openstack-infra | 05:56 | |
| *** priteau has quit IRC | 05:57 | |
| *** marst has quit IRC | 05:57 | |
| *** slaweq has joined #openstack-infra | 05:58 | |
| *** esberglu has joined #openstack-infra | 05:59 | |
| *** links has joined #openstack-infra | 06:02 | |
| *** esberglu has quit IRC | 06:03 | |
| *** slaweq has quit IRC | 06:12 | |
| *** pgadiya has joined #openstack-infra | 06:15 | |
| *** pcaruana has joined #openstack-infra | 06:17 | |
| *** jtomasek has joined #openstack-infra | 06:20 | |
| *** rcernin has joined #openstack-infra | 06:33 | |
| *** thorst_afk has joined #openstack-infra | 06:40 | |
| *** pgadiya has quit IRC | 06:42 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate master: [WIP] Ansiblise ephemeral/swap setup https://review.openstack.org/499467 | 06:44 |
| *** apetrich has joined #openstack-infra | 06:45 | |
| *** thorst_afk has quit IRC | 06:45 | |
| *** xarses has joined #openstack-infra | 06:46 | |
| *** esberglu has joined #openstack-infra | 06:52 | |
| *** esberglu has quit IRC | 06:53 | |
| *** esberglu has joined #openstack-infra | 06:53 | |
| *** esberglu has quit IRC | 06:53 | |
| *** bhavik1 has joined #openstack-infra | 06:53 | |
| *** esberglu has joined #openstack-infra | 06:54 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate master: [WIP] Ansiblise ephemeral/swap setup https://review.openstack.org/499467 | 06:57 |
| *** xarses has quit IRC | 06:57 | |
| *** esberglu has quit IRC | 06:58 | |
| *** slaweq has joined #openstack-infra | 07:01 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate master: [WIP] Ansiblise ephemeral/swap setup https://review.openstack.org/499467 | 07:02 |
| *** slaweq has quit IRC | 07:06 | |
| *** shardy has joined #openstack-infra | 07:09 | |
| *** tesseract has joined #openstack-infra | 07:12 | |
| *** slaweq has joined #openstack-infra | 07:13 | |
| openstackgerrit | zhurong proposed openstack-infra/project-config master: Register murano-tempest-plugin project and jobs https://review.openstack.org/499916 | 07:18 |
| *** d0ugal has quit IRC | 07:20 | |
| openstackgerrit | zhurong proposed openstack-infra/project-config master: Register murano-tempest-plugin project and jobs https://review.openstack.org/499916 | 07:23 |
| *** jpich has joined #openstack-infra | 07:38 | |
| *** jpena|off is now known as jpena | 07:39 | |
| *** lifeless_ is now known as lifeless | 07:39 | |
| *** thorst_afk has joined #openstack-infra | 07:41 | |
| *** ykarel is now known as ykarel|lunch | 07:43 | |
| *** thorst_afk has quit IRC | 07:45 | |
| *** florianf has joined #openstack-infra | 07:47 | |
| *** ralonsoh has joined #openstack-infra | 07:49 | |
| *** egonzalez has joined #openstack-infra | 07:51 | |
| *** priteau has joined #openstack-infra | 07:53 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Revert "Stop trying to build networking-bagpipe with DLRN" https://review.openstack.org/499744 | 07:54 |
| *** priteau has quit IRC | 07:58 | |
| *** sflanigan has quit IRC | 07:58 | |
| *** jaosorior has quit IRC | 07:59 | |
| *** jaosorior has joined #openstack-infra | 08:00 | |
| *** slaweq has quit IRC | 08:00 | |
| *** slaweq has joined #openstack-infra | 08:01 | |
| frickler | does everybody have their fireworks ready to celebrate review #0.5M? :) | 08:04 |
| *** lucas-hungry is now known as lucasagomes | 08:04 | |
| *** slaweq has quit IRC | 08:09 | |
| *** slaweq has joined #openstack-infra | 08:09 | |
| *** s-shiono has quit IRC | 08:31 | |
| *** efoley has joined #openstack-infra | 08:36 | |
| frickler | fungi: I did setup smokeping monitoring for ask.o.o and it shows an outage again this morning between 06:55 and 07:10 (5 min resolution), just in case you want to investigate further | 08:39 |
| *** thorst_afk has joined #openstack-infra | 08:42 | |
| *** ykarel|lunch is now known as ykarel | 08:42 | |
| *** thorst_afk has quit IRC | 08:46 | |
| *** adisky__ has quit IRC | 08:49 | |
| *** electrofelix has joined #openstack-infra | 08:50 | |
| *** psachin has quit IRC | 09:00 | |
| *** priteau has joined #openstack-infra | 09:02 | |
| *** e0ne has joined #openstack-infra | 09:09 | |
| *** sambetts|afk is now known as sambetts | 09:11 | |
| *** efoley has quit IRC | 09:15 | |
| *** efoley_ has joined #openstack-infra | 09:15 | |
| *** salv-orlando has quit IRC | 09:24 | |
| *** kjackal has quit IRC | 09:24 | |
| *** kjackal__ has joined #openstack-infra | 09:25 | |
| *** udesale__ has joined #openstack-infra | 09:27 | |
| *** udesale has quit IRC | 09:27 | |
| *** udesale has joined #openstack-infra | 09:29 | |
| *** salv-orlando has joined #openstack-infra | 09:29 | |
| *** udesale__ has quit IRC | 09:31 | |
| *** nicolasbock has joined #openstack-infra | 09:33 | |
| *** kjackal has joined #openstack-infra | 09:33 | |
| *** kjackal__ has quit IRC | 09:36 | |
| *** e0ne has quit IRC | 09:38 | |
| *** e0ne has joined #openstack-infra | 09:38 | |
| *** udesale has quit IRC | 09:40 | |
| *** udesale has joined #openstack-infra | 09:41 | |
| *** jascott1 has quit IRC | 09:41 | |
| *** jascott1 has joined #openstack-infra | 09:41 | |
| *** e0ne has quit IRC | 09:42 | |
| *** e0ne has joined #openstack-infra | 09:42 | |
| *** thorst_afk has joined #openstack-infra | 09:42 | |
| *** udesale has quit IRC | 09:44 | |
| *** jascott1 has quit IRC | 09:46 | |
| *** thorst_afk has quit IRC | 09:47 | |
| *** salv-orlando has quit IRC | 10:01 | |
| *** cuongnv has quit IRC | 10:02 | |
| *** salv-orlando has joined #openstack-infra | 10:04 | |
| *** d0ugal has joined #openstack-infra | 10:18 | |
| *** LindaWang has quit IRC | 10:21 | |
| *** Julien-z_ has quit IRC | 10:24 | |
| *** LindaWang has joined #openstack-infra | 10:27 | |
| *** Julien-z_ has joined #openstack-infra | 10:30 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Preparing TripleO CI to test upgrades to Queens https://review.openstack.org/499359 | 10:30 |
| *** efoley_ has quit IRC | 10:34 | |
| *** thorst_afk has joined #openstack-infra | 10:43 | |
| *** psachin has joined #openstack-infra | 10:45 | |
| *** thorst_afk has quit IRC | 10:48 | |
| *** jascott1 has joined #openstack-infra | 10:55 | |
| *** tosky has joined #openstack-infra | 10:55 | |
| *** bhavik1 has quit IRC | 10:57 | |
| *** zhurong has quit IRC | 10:59 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Add /nodes and /nodes.json to webapp https://review.openstack.org/499969 | 11:00 |
| *** hashar has joined #openstack-infra | 11:01 | |
| *** hashar is now known as hasharAway | 11:02 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Add /node-list to the webapp https://review.openstack.org/499969 | 11:03 |
| *** lucasagomes is now known as lucas-afk | 11:06 | |
| *** stakeda has quit IRC | 11:08 | |
| *** thorst_afk has joined #openstack-infra | 11:09 | |
| *** _ari_|conf is now known as _ari_ | 11:10 | |
| *** thorst_afk has quit IRC | 11:11 | |
| *** pbourke has quit IRC | 11:23 | |
| *** pbourke has joined #openstack-infra | 11:25 | |
| *** esberglu has joined #openstack-infra | 11:26 | |
| *** esberglu has quit IRC | 11:26 | |
| *** wolverineav has joined #openstack-infra | 11:28 | |
| *** wolverineav has quit IRC | 11:28 | |
| *** wolverineav has joined #openstack-infra | 11:29 | |
| *** dulek has joined #openstack-infra | 11:29 | |
| *** salv-orlando has quit IRC | 11:29 | |
| *** mat128 has joined #openstack-infra | 11:35 | |
| *** ldnunes has joined #openstack-infra | 11:42 | |
| *** sdague has joined #openstack-infra | 11:43 | |
| *** dprince has joined #openstack-infra | 11:53 | |
| *** srobert has joined #openstack-infra | 11:56 | |
| *** srobert has quit IRC | 11:56 | |
| *** srobert has joined #openstack-infra | 11:57 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Implement a static driver for Nodepool https://review.openstack.org/468624 | 12:02 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Implement an OpenContainer driver https://review.openstack.org/468753 | 12:02 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Refactor provider config to driver module https://review.openstack.org/488384 | 12:02 |
| *** thorst_afk has joined #openstack-infra | 12:04 | |
| *** trown|outtypewww is now known as trown | 12:08 | |
| *** Julien-z_ has quit IRC | 12:09 | |
| *** askb has quit IRC | 12:13 | |
| *** esberglu has joined #openstack-infra | 12:19 | |
| *** esberglu has quit IRC | 12:23 | |
| *** gcerami has joined #openstack-infra | 12:28 | |
| *** udesale has joined #openstack-infra | 12:29 | |
| gcerami | when specifying jobs for a pipeline in zuul layout, I can specify a child job that will only run if the parent succeeds, but can a child job have a child job itself ? | 12:30 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/shade master: Updated from global requirements https://review.openstack.org/499983 | 12:34 |
| *** sbezverk has quit IRC | 12:38 | |
| frickler | \o/ proposal bot almost beat me to #500000 :D | 12:38 |
| *** krtaylor has joined #openstack-infra | 12:39 | |
| mnaser | haha congrats frickler | 12:43 |
| *** rlandy has joined #openstack-infra | 12:43 | |
| *** sbezverk has joined #openstack-infra | 12:45 | |
| *** jpena is now known as jpena|lunch | 12:46 | |
| *** psachin has quit IRC | 12:46 | |
| *** pkovar has joined #openstack-infra | 12:51 | |
| *** jcoufal has joined #openstack-infra | 12:51 | |
| pkovar | clarkb: hi, the docs team was wondering if the infra team could send a representative to our ptg docs session on monday when we plan to discuss retention policy, eol docs (un)publishing etc. | 12:53 |
| pkovar | it's scheduled for Monday 04:30-06:00, see https://etherpad.openstack.org/p/docs-i18n-ptg-queens | 12:54 |
| mordred | frickler: nicely done! | 12:58 |
| fungi | gcerami: yes, you can nest/sequence them as deep as you like. in zuul 2.x you're limited to one direct parent for any job though (so you can't make a job than only runs if a group of other jobs all succeed first) | 13:00 |
| fungi | gcerami: but you can certainly have job c which only runs if b passes which only runs if a passes | 13:01 |
| gcerami | fungi: great, thanks. | 13:01 |
| *** thegreenhundred has quit IRC | 13:02 | |
| *** kgiusti has joined #openstack-infra | 13:02 | |
| andreaf | yolanda, sdague, clarkb: looking for reviews on https://review.openstack.org/#/c/499832/? (add reno gate to hacking) | 13:03 |
| *** esberglu has joined #openstack-infra | 13:07 | |
| *** esberglu has quit IRC | 13:07 | |
| * sc` prods openstackgerrit with a stick | 13:09 | |
| sc` | change 500k didn't pop a channel notify | 13:09 |
| *** markvoelker has joined #openstack-infra | 13:11 | |
| sc` | project-config appears to check out, but it's still early | 13:11 |
| *** mriedem has joined #openstack-infra | 13:11 | |
| fungi | pkovar: i expect we'll have plenty of experienced infra folk ready to pop into the docs room for that, but i'll make a note on our planning etherpad so we don't forget | 13:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Fix typo in tox.ini https://review.openstack.org/500045 | 13:13 |
| *** baoli has joined #openstack-infra | 13:14 | |
| pkovar | fungi: thanks! | 13:14 |
| *** scottda_ has joined #openstack-infra | 13:18 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Handle logging ansible python errors properly https://review.openstack.org/499397 | 13:19 |
| *** efried is now known as fried_rice | 13:20 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove -vvv from playbook invocation for log streaming test https://review.openstack.org/499753 | 13:21 |
| openstackgerrit | Jose Luis Franco proposed openstack-infra/tripleo-ci master: WIP: Disable properly services api_cfn and api_cloudwatch. https://review.openstack.org/499182 | 13:21 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add support for debug statements https://review.openstack.org/499608 | 13:21 |
| *** bauzas is now known as bauwser | 13:22 | |
| *** Goneri has joined #openstack-infra | 13:24 | |
| *** adrianofr_ has joined #openstack-infra | 13:24 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Don't output complex items in the summary line https://review.openstack.org/499755 | 13:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Make log streaming test three node https://review.openstack.org/500049 | 13:31 |
| *** dave-mccowan has joined #openstack-infra | 13:32 | |
| *** esberglu has joined #openstack-infra | 13:36 | |
| *** esberglu has quit IRC | 13:37 | |
| *** esberglu has joined #openstack-infra | 13:39 | |
| *** e0ne has quit IRC | 13:40 | |
| *** dulek has left #openstack-infra | 13:40 | |
| *** e0ne has joined #openstack-infra | 13:41 | |
| *** udesale has quit IRC | 13:42 | |
| *** jpena|lunch is now known as jpena | 13:43 | |
| *** links has quit IRC | 13:44 | |
| *** LindaWang has quit IRC | 13:45 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Make log streaming test three node https://review.openstack.org/500049 | 13:45 |
| *** shardy has quit IRC | 13:49 | |
| *** hongbin has joined #openstack-infra | 13:54 | |
| *** shardy has joined #openstack-infra | 13:56 | |
| *** gouthamr has joined #openstack-infra | 13:57 | |
| *** salv-orlando has joined #openstack-infra | 13:58 | |
| *** baoli has quit IRC | 13:59 | |
| *** LindaWang has joined #openstack-infra | 14:01 | |
| clarkb | pkovar: looks like that maps to the idea someone posted at https://etherpad.openstack.org/p/infra-ptg-queens I will update the time and location on our etherpad | 14:03 |
| clarkb | oh and fungi added it to the scheduled list | 14:04 |
| fungi | seemed like the most relevant location for time-oriented reminders | 14:05 |
| *** laurelm has joined #openstack-infra | 14:07 | |
| clarkb | ++ I just hadn't gotten that far down in sb before responding. | 14:10 |
| *** hasharAway has quit IRC | 14:11 | |
| *** LindaWang has quit IRC | 14:13 | |
| *** srobert has quit IRC | 14:18 | |
| *** eharney has joined #openstack-infra | 14:19 | |
| openstackgerrit | Merged openstack-infra/shade master: Updated from global requirements https://review.openstack.org/499983 | 14:29 |
| *** marst has joined #openstack-infra | 14:29 | |
| zoli | ping pabelanger - Paul, are you around? If yes, could you check my PM, pls? | 14:30 |
| *** armax_ has joined #openstack-infra | 14:32 | |
| *** armax has quit IRC | 14:35 | |
| *** armax_ is now known as armax | 14:35 | |
| *** jamesmcarthur has joined #openstack-infra | 14:35 | |
| clarkb | zoli: feel free to ask the channel too if it is something the cahnnel can help with | 14:37 |
| dmsimard | clarkb: Need +3 on https://review.openstack.org/#/c/499687/ to try and see if it resolves the device busy issue we're seeing. | 14:38 |
| *** baoli has joined #openstack-infra | 14:39 | |
| clarkb | dmsimard: done | 14:39 |
| *** jaosorior has quit IRC | 14:40 | |
| dmsimard | clarkb: thanks, I'll add an e-r query but I might need a bit of help (a bit of a elk noob) | 14:41 |
| dmsimard | clarkb: I wanted to edit the existing one https://github.com/openstack-infra/elastic-recheck/blob/dd5c6d25134b3a65e6425441471b1bf63f6899d8/queries/1706103.yaml | 14:41 |
| dmsimard | to add the xvde1 busy and xvde2 busy errors | 14:41 |
| zoli | clarkb, thanks. I just tried to replicate this setup on a test machine: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/mirror.vhost.erb | 14:41 |
| zoli | with small modifications (forward proxy instead of a reverse one and manual config, no Puppet - it's a PoC) | 14:42 |
| zoli | it is proxying but not caching :( | 14:42 |
| zoli | I am afraid that there is some config mistake laughing at me and I do not see it | 14:43 |
| clarkb | dmsimard: what I like to do is use kibana at http://logstash.openstack.org and edit queries there until I like the results. Then edit the e-r yaml | 14:44 |
| clarkb | dmsimard: generally you'd want (message:"foo" OR message:"bar" OR message:"baz") AND tags:"console" to do what you describe | 14:44 |
| clarkb | dmsimard: just replace the message strings with the several different items you have | 14:44 |
| clarkb | zoli: I would double check that you have mod_cache enabled | 14:45 |
| dmsimard | clarkb: oh, yeah -- that's what I do too except the query doesn't work :) | 14:45 |
| clarkb | zoli: but the nature of the http traffic may also prevent it from being cached so you might need to check if your http is cacheable | 14:46 |
| clarkb | zoli: the apache docs for mod cache have a section on what things are cacheable | 14:46 |
| clarkb | dmsimard: can you share your query string? | 14:46 |
| *** LindaWang has joined #openstack-infra | 14:47 | |
| zoli | clarkb, thanks for hints. mod_cache is enabled: | 14:47 |
| *** LindaWang has quit IRC | 14:47 | |
| zoli | /etc/httpd/conf.modules.d/00-base.conf:LoadModule cache_disk_module modules/mod_cache_disk.so | 14:47 |
| zoli | /etc/httpd/conf.modules.d/00-base.conf:LoadModule cache_disk_module modules/mod_cache_disk.so | 14:47 |
| *** jamesmcarthur has quit IRC | 14:48 | |
| dmsimard | clarkb: oh, nevermind, it works now .. I guess I queried before the logs ended up in logstash yesterday | 14:49 |
| clarkb | dmsimard: ya there is some lag. We can't index until after jobs finish and then all the jobs go through a sometimes relatively small straw | 14:49 |
| dmsimard | clarkb: there's definitely some occurrences | 14:50 |
| dmsimard | (message:"/dev/xvde1: No such file or directory") OR (message:"/dev/xvde" AND message:"Device or resource busy") AND tags:"console" | 14:50 |
| * dmsimard edits e-r | 14:50 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/elastic-recheck master: Add additional signature for RAX xvde partitioning issues https://review.openstack.org/500088 | 14:51 |
| dmsimard | pabelanger: ^ | 14:52 |
| clarkb | dmsimard: the vast majority of them are on centos-7 | 14:52 |
| *** bobh has joined #openstack-infra | 14:52 | |
| dmsimard | clarkb: so kernel thing perhaps. | 14:52 |
| dmsimard | clarkb: kernel is old on el7 :( | 14:52 |
| clarkb | the one xenial hit on the first page of results is a misparse and includes centos too (yay logstash bugs) | 14:52 |
| clarkb | so ya I think this is likely to be a kernel behavior on centos-7 | 14:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Remove errant print https://review.openstack.org/500090 | 14:53 |
| dmsimard | clarkb: ok, hopefully the sync/partprobe resolves it -- landing that e-r patch will help us track it down | 14:53 |
| clarkb | ya | 14:54 |
| clarkb | this could also explain why the first chagne was able to sneak through easily | 14:54 |
| *** slaweq has quit IRC | 14:54 | |
| clarkb | we only run one centos 7 job on devstack-gate and if that doesn't run on rax the code path isn't exercised | 14:54 |
| clarkb | dmsimard: I think you may need to wrap the OR'd messages in an outer () due to AND precendence | 14:55 |
| dmsimard | clarkb: makes sense | 14:56 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/elastic-recheck master: Add additional signature for RAX xvde partitioning issues https://review.openstack.org/500088 | 14:57 |
| clarkb | ya I seem to get more results that way | 14:57 |
| clarkb | pabelanger: ^ I'll approve that shortly but want to give you a chance to take a look if you like | 14:58 |
| *** dhajare has quit IRC | 14:58 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Use new keystoneauth version discovery https://review.openstack.org/493582 | 14:59 |
| *** egonzalez has quit IRC | 15:00 | |
| mtreinish | infra-root: it looks like health.o.o is down. It was probably my change to switch the api service install to use system python instead of a venv. Can someone take a look? | 15:02 |
| *** jpena is now known as jpena|away | 15:03 | |
| clarkb | mtreinish: ya I can poke at it after caffeine | 15:04 |
| fungi | or i can take a look after the release team meeting | 15:04 |
| mtreinish | cool, thanks | 15:04 |
| *** camunoz has joined #openstack-infra | 15:05 | |
| jeblair | or i can look after breakfast | 15:06 |
| *** iyamahat has joined #openstack-infra | 15:09 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Use new keystoneauth version discovery https://review.openstack.org/493582 | 15:10 |
| *** iyamahat has quit IRC | 15:10 | |
| *** iyamahat has joined #openstack-infra | 15:11 | |
| *** EmilienM has quit IRC | 15:12 | |
| *** EmilienM has joined #openstack-infra | 15:12 | |
| *** iyamahat has quit IRC | 15:16 | |
| jeblair | mtreinish: when did the change merge? | 15:16 |
| jeblair | looks like Aug 31 21:57:12 is when puppet made the change. i don't see any puppet errors or anything interesting | 15:18 |
| *** links has joined #openstack-infra | 15:19 | |
| *** dmsimard has quit IRC | 15:19 | |
| *** dmsimard has joined #openstack-infra | 15:19 | |
| jeblair | [Fri Sep 01 15:18:23.622601 2017] [:error] [pid 19906:tid 139819439027968] [client 24.69.129.67:38678] Target WSGI script not found or unable to stat: /usr/local/bin/openstack-health | 15:19 |
| *** iyamahat has joined #openstack-infra | 15:19 | |
| jeblair | mtreinish: do you have a link to the change handy? | 15:20 |
| jeblair | perhaps there was no triggering event to cause the install to happen | 15:20 |
| jeblair | i'll manually install it | 15:20 |
| jeblair | mtreinish: done. it loads for me; let me know if you see further problems | 15:21 |
| *** rama_y has joined #openstack-infra | 15:21 | |
| jeblair | mtreinish: and next time a change lands, let's keep an eye out for whether it auto-updates | 15:22 |
| *** thegreenhundred has joined #openstack-infra | 15:22 | |
| *** gmann has quit IRC | 15:23 | |
| openstackgerrit | Attila Darazs proposed openstack-infra/tripleo-ci master: Fix hash_info.sh path for the DLRN API reporting https://review.openstack.org/500101 | 15:23 |
| *** gmann has joined #openstack-infra | 15:23 | |
| openstackgerrit | Ken Dreyer proposed openstack/python-jenkins master: add jenkins_urlopen() method https://review.openstack.org/500103 | 15:24 |
| openstackgerrit | Ken Dreyer proposed openstack/python-jenkins master: build_job: return queue item identifier https://review.openstack.org/500104 | 15:24 |
| openstackgerrit | Ken Dreyer proposed openstack/python-jenkins master: add get_queue_item() method https://review.openstack.org/500105 | 15:24 |
| openstackgerrit | Ken Dreyer proposed openstack/python-jenkins master: better get_build_info() example https://review.openstack.org/500106 | 15:24 |
| mtreinish | jeblair: https://review.openstack.org/490123 was the change | 15:25 |
| mtreinish | jeblair: hmm, it doesn't look like the elastic-recheck install got triggered either | 15:25 |
| mtreinish | the status page is saying it isn't able to import e-r http://health.openstack.org/status | 15:26 |
| *** ihrachys has joined #openstack-infra | 15:26 | |
| jeblair | mtreinish: ah yep, both of those are triggered by vcrrepo updates, so this looks like it should be a one-time error. i'll manually install e-r as well | 15:27 |
| mtreinish | jeblair: cool, thanks | 15:28 |
| jeblair | mtreinish: okay, installed e-r and restarted apache (to clear out cached workers). status json says er is there now | 15:29 |
| *** rcernin has quit IRC | 15:29 | |
| *** pcaruana has quit IRC | 15:30 | |
| mtreinish | ok, great. Everything looks good now | 15:32 |
| mtreinish | and we're finally on the latest version of the api server | 15:32 |
| pabelanger | zoli: clarkb: ya, I would recommend increasing debug logging in apache and see what is happening, I had to do that when trying to proxy cache docker.io. Possible you could do the same process | 15:32 |
| pabelanger | clarkb: dmsimard: wfm | 15:32 |
| openstackgerrit | Merged openstack-infra/shade master: Fix typo in tox.ini https://review.openstack.org/500045 | 15:33 |
| *** ramishra has quit IRC | 15:34 | |
| pabelanger | infra-root: If you didn't see, afs-docs on zuulv3 is now working: https://docs.openstack.org/sandbox/ however, I'd like to land https://review.openstack.org/499876/ of address the html folder issue. Once landed, zuul can be changed over to use this, and I'll start working to clean up run-docs role which will be needed for shade an other big tent projects | 15:37 |
| jeblair | pabelanger: wasn't that supposed to be the default for that role? | 15:39 |
| *** rama_y has quit IRC | 15:42 | |
| pabelanger | jeblair: right now no, but we can change that. Today, we just sync the html directory into log_src on the executor. We can change this to act as above, and first create log_src/html on executor and copy contents of html folder into it. | 15:43 |
| pabelanger | I can push up a few patches now for that | 15:43 |
| jeblair | pabelanger: sphinx_output_src: "src/{{ zuul.project.canonical_name }}/doc/build/html" | 15:43 |
| pabelanger | jeblair: right, that's just the html folder | 15:43 |
| jeblair | pabelanger: what am i missing? that's from the role defaults/main.yaml isn't that the default? | 15:44 |
| *** armax_ has joined #openstack-infra | 15:44 | |
| pabelanger | trailing slash | 15:44 |
| jeblair | pabelanger: yeah, so why aren't you fixing the default? | 15:44 |
| pabelanger | src/{{ zuul.project.canonical_name }}/doc/build/html/ | 15:44 |
| pabelanger | jeblair: I can, but more roles need to change | 15:44 |
| pabelanger | let me push up those patches | 15:44 |
| *** armax has quit IRC | 15:45 | |
| *** armax_ is now known as armax | 15:45 | |
| *** wolverineav has quit IRC | 15:46 | |
| dmsimard | Is there a way to *remove* a queued job from the zuul queue ? | 15:47 |
| *** wolverineav has joined #openstack-infra | 15:47 | |
| dmsimard | There seems to be a bug with one of our jobs where there is one parent job, one child job -- the child job is queued but not the parent and the child ends up being queued forever | 15:47 |
| dmsimard | I've been cleaning those up by restarting zuul-server when the queue is otherwise empty but that doesn't seem ideal | 15:47 |
| clarkb | pushing a new patchset is the only way to clear out today I think | 15:48 |
| jeblair | does the parent have matchers that don't match the item, but the child does? | 15:49 |
| *** jamesmcarthur has joined #openstack-infra | 15:49 | |
| *** salv-orlando has quit IRC | 15:50 | |
| *** mugsie has joined #openstack-infra | 15:51 | |
| fungi | you might try abandoning and restoring the change? | 15:52 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add zuul.timeout variable to jobs https://review.openstack.org/500114 | 15:52 |
| pabelanger | fungi: ya, I have done that before | 15:52 |
| fungi | once abandoned, i think zuul notices it's unable to merge and aborts | 15:52 |
| *** ykarel has quit IRC | 15:53 | |
| stephenfin | Is there something off with the GitHub mirrors? I see that a load of tags, including the recent 16.0.0 tag, are missing for nova | 15:53 |
| *** salv-orlando has joined #openstack-infra | 15:53 | |
| stephenfin | https://github.com/openstack/nova/tree/16.0.0.0 vs. https://github.com/openstack/nova/tree/16.0.0.0rc2 | 15:53 |
| *** yamahata has quit IRC | 15:54 | |
| stephenfin | Maybe I shouldn't be expecting those. idk | 15:54 |
| fungi | stephenfin: check the stable/pike branch | 15:54 |
| clarkb | the tag is 16.0.0 not 16.0.0.0 | 15:54 |
| fungi | right, that too | 15:55 |
| clarkb | and is in github | 15:55 |
| fungi | 16.0.0.0rc2 is the 0rc2 (because of pep 440 reasons) release candidate for 16.0.0 | 15:55 |
| stephenfin | Right. Damn it. Thanks clarkb, fungi | 15:55 |
| stephenfin | Does not appear in the tag dropdown. That's GitHub's issue though | 15:56 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Update fetch-sphinx-output to collect contents of directory https://review.openstack.org/500116 | 15:56 |
| pabelanger | jeblair: ^would be the change needed | 15:57 |
| jeblair | pabelanger: why are you changing executor dest? | 15:57 |
| *** Apoorva has joined #openstack-infra | 15:57 | |
| *** yamahata has joined #openstack-infra | 15:58 | |
| pabelanger | jeblair: because if not, we'll just copy the sphinx build htmls files into the log_src folder, on html subfolder | 15:58 |
| jeblair | pabelanger: i can't parse that sentence | 15:59 |
| pabelanger | s/on html/not html | 15:59 |
| pabelanger | 1 sec, let me get a log URL from zuulv3 job | 15:59 |
| jeblair | and also s/log_src/log_root/ ? | 16:00 |
| pabelanger | oh, yes. sorry | 16:00 |
| jeblair | okay, so i thought you *wanted* to get rid of the html/ directory in the output? | 16:00 |
| pabelanger | so, I'd like to keep the following URL structure on docs jobs: http://logs.openstack.org/16/500116/1/check/openstack-doc-build/92ce82e/html/ | 16:00 |
| pabelanger | jeblair: only for afs-docs, because we copy into artifacts folder on executor, with non afs docs jobs, they go into log_root/html | 16:01 |
| pabelanger | same role, different destinations on executor | 16:01 |
| jeblair | i feel like this role is becoming more confusing than if it didn't exist | 16:02 |
| *** ralonsoh has quit IRC | 16:02 | |
| jeblair | pabelanger: why not make two roles? | 16:02 |
| *** annegentle has joined #openstack-infra | 16:02 | |
| pabelanger | jeblair: we can, but these roles would almost be copypasta | 16:02 |
| *** david-lyle has quit IRC | 16:03 | |
| pabelanger | we really need a generic role to copy source dir on node to target dir on executor, and have the ability to change out the varibles for each | 16:03 |
| jeblair | pabelanger: the roles would be nearly identical because the roles are nothing more than a single rsync command. i think the potential value of these roles is dealing with the paths (which is tricky) for you. | 16:05 |
| *** baoli_ has joined #openstack-infra | 16:06 | |
| jeblair | (if we end up making this no easier than, or even harder than, simply adding an rsync task, then we've missed an opportunity) | 16:06 |
| *** jpich has quit IRC | 16:06 | |
| *** baoli has quit IRC | 16:08 | |
| jeblair | pabelanger: my preference would be to have two different roles so that we only have to think about this question once. failing that, i think i like your earlier suggestion (499876) better than 500116. | 16:08 |
| jeblair | pabelanger: let's ask mordred for input | 16:08 |
| *** annegentle has quit IRC | 16:08 | |
| pabelanger | jeblair: well, it will require some understand of the role, based on input variables. If we create another role with the same rsync command but different ansible variable defaults, I don't think that is much better either, at least from an ansible POV | 16:08 |
| pabelanger | sure, np | 16:09 |
| *** annegentle has joined #openstack-infra | 16:09 | |
| *** jpich has joined #openstack-infra | 16:09 | |
| jeblair | pabelanger: i don't have an ansible pov, i have a user pov, where "fetch-draft-sphinx-output" and "fetch-final-sphinx-output" are easier to deal with than trying to decide whether sphinx_output_src should have a traling / in each individual job. | 16:09 |
| *** annegentle has quit IRC | 16:10 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add additional signature for RAX xvde partitioning issues https://review.openstack.org/500088 | 16:11 |
| mordred | jeblair, pabelanger: reading | 16:12 |
| dmsimard | jeblair: re: parent/child issue (sorry, got sidetracked).. it happens intermittently, I haven't dug too much to figure out why it happens yet | 16:12 |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade master: De-clientify baremetal create/delete https://review.openstack.org/500119 | 16:14 |
| *** rama_y has joined #openstack-infra | 16:14 | |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade master: De-client-ify baremetal machine port list https://review.openstack.org/500120 | 16:14 |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade master: De-client-ify machine patch operations https://review.openstack.org/500121 | 16:14 |
| *** annegentle has joined #openstack-infra | 16:14 | |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: add a devstack-legacy job https://review.openstack.org/497699 | 16:15 |
| dmsimard | jeblair: see: http://i.imgur.com/qDGzCVO.png DLRN-rpmbuild is the parent job | 16:16 |
| dmsimard | It might be a matcher issue, that's a good pointer | 16:16 |
| mordred | jeblair, pabelanger: do we have cases where we expect people to make child jobs that use the fetch-sphinx-output and will want to provide alternate paths? or do we expect the total set of path combinations to be encompassed in a couple of base jobs? | 16:16 |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Re-introduce sync/partprobe in fix_disk_layout https://review.openstack.org/499687 | 16:17 |
| dmsimard | jeblair: yeah, that's exactly it.. missing a branch match | 16:17 |
| mordred | jeblair, pabelanger: it's basically the two, right? "fetch sphinx-output from {dir}" (dir is important because that could be anywhere) and publish to (AFS, logs) | 16:17 |
| jeblair | mordred: the fetch is different for each of the (afs, logs) publishes as well; it's just done with one role | 16:19 |
| mordred | so we always need the first parameter no matter how we organize other things (and it defaults to src/{{ zuul.project.canonical_name }}/doc/build/html/ currently | 16:19 |
| mordred | jeblair: right- I'm just stepping back and making sure I understand the situation, regardless of existing roles | 16:19 |
| jeblair | mordred: well, it defaults to src/{{ zuul.project.canonical_name }}/doc/build/html currently which is correct for logs, but src/{{ zuul.project.canonical_name }}/doc/build/html/ would be correct for afs | 16:19 |
| pabelanger | yes, that is the main issue | 16:20 |
| jeblair | mordred: so conceptually, we have (fetch-for-afs, publish-to-afs) and (fetch-for-logs, publish-to-logs) | 16:20 |
| jeblair | all four of those things are different | 16:20 |
| *** dave-mccowan has quit IRC | 16:20 | |
| mordred | nod. one more wrinkle - the current fetch assumes we only ever want to fetch html made by sphinx, but sphinx has more than one type of output | 16:21 |
| jeblair | right, thought that can (again, currently) be overridden with the src parameter | 16:22 |
| mordred | yah | 16:22 |
| mordred | what if fetch always just fetched src/{{ zuul.project.canonical_name }}/doc/build/ to somewhere on the executor, like 'docs' - and then publish-to-logs moved docs/html to logs and publish-to-afs moved docs/html/* to /afs/.openstack ... ? | 16:23 |
| zoli | have a good weekend | 16:23 |
| jeblair | mordred: come to think of it, why actually have a separate fetch role at all? | 16:24 |
| *** esberglu has quit IRC | 16:24 | |
| jeblair | pabelanger: ? | 16:24 |
| mordred | jeblair: I think it made a lot of sense when the majority of what it was doing was fetching docs into the logs dir so that the base job would publish the logs | 16:24 |
| jeblair | yeah, so maybe morph the current one into 'save-sphinx-output-to-logs' or something, and then put the appropriate synchronize task into 'publish-to-afs' ? | 16:26 |
| *** ykarel has joined #openstack-infra | 16:26 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Validate zuul_stream func test output https://review.openstack.org/499732 | 16:27 |
| *** srobert has joined #openstack-infra | 16:29 | |
| *** aarefiev is now known as aarefiev_afk | 16:29 | |
| *** e0ne has quit IRC | 16:29 | |
| jeblair | mordred, pabelanger: you went quiet? where are we? do we want to change anything? or are you tired of thinking about it? if so, i'll just reverse my -1 on https://review.openstack.org/499876 | 16:30 |
| mordred | jeblair: thinking | 16:30 |
| mordred | and looking at code - sorry | 16:31 |
| jeblair | ok | 16:31 |
| jeblair | np | 16:31 |
| pabelanger | jeblair: ya, thinking | 16:31 |
| pabelanger | trying to compose a reply | 16:31 |
| jeblair | okay. just didn't want to leave thing hanging/blocking | 16:31 |
| *** links has quit IRC | 16:31 | |
| pabelanger | k, here goes :) | 16:31 |
| pabelanger | with, 499876 afs-docs will be done. and I can move onto wheel builds. However, if we want to refactor like above, I don't mind doing it, but that is going to be time not spend on wheel builder. Because PTG timelines and such, will need help on which direction you want me to spend time out | 16:33 |
| pabelanger | I also need to spend time on run-docs refactor for big-tent jobs also, so lots of things to pick from | 16:34 |
| pabelanger | that is: https://review.openstack.org/499778/ | 16:34 |
| mordred | pabelanger, jeblair: ok. how about we land that and move on for today - but put a pin in it for post-ptg because I do think I'm coming around to like two roles or two playbooks or whatnot - but I'm happy for us to do that later | 16:36 |
| *** jpena|away is now known as jpena|off | 16:38 | |
| pabelanger | mordred: I'll defer to jeblair on what he like to do | 16:40 |
| jeblair | pabelanger: okay, let's land 876 because it's simple, and will let us evolve forward to the other thing most easily. maybe we can do that in a spare moment before the ptg even. :) | 16:43 |
| *** ramishra has joined #openstack-infra | 16:44 | |
| *** jpich has quit IRC | 16:45 | |
| pabelanger | for sure | 16:45 |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify User Update https://review.openstack.org/499284 | 16:46 |
| jamesmcarthur | pkovar: hogepodge told me you need to get the docs description updated on the PTG page. | 16:47 |
| pkovar | jamesmcarthur: ha, just started composing an email :) | 16:47 |
| jamesmcarthur | pkovar: That content is in our CMS, so the fastest thing would be to just email it to me: jimmy@openstack.org | 16:47 |
| *** esberglu has joined #openstack-infra | 16:47 | |
| jamesmcarthur | pkovar: awesome :) | 16:47 |
| pkovar | will send it in a bit, thans, jamesmcarthur | 16:47 |
| pkovar | thanks, rather :) | 16:48 |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify Add User to Group https://review.openstack.org/499345 | 16:48 |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify Check User in Group https://review.openstack.org/499357 | 16:50 |
| *** dprince has quit IRC | 16:50 | |
| *** esberglu has quit IRC | 16:51 | |
| *** iyamahat has quit IRC | 16:52 | |
| jamesmcarthur | pkovar: happy to help! | 16:52 |
| *** iyamahat has joined #openstack-infra | 16:52 | |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify Remove User from Group https://review.openstack.org/499360 | 16:54 |
| *** iyamahat_ has joined #openstack-infra | 16:55 | |
| *** gcerami has quit IRC | 16:55 | |
| *** Goneri has quit IRC | 16:55 | |
| *** iyamahat has quit IRC | 16:58 | |
| *** Apoorva has quit IRC | 17:01 | |
| *** electrofelix has quit IRC | 17:01 | |
| *** Apoorva has joined #openstack-infra | 17:02 | |
| *** xarses has joined #openstack-infra | 17:04 | |
| *** Goneri has joined #openstack-infra | 17:05 | |
| openstackgerrit | Merged openstack-infra/project-config master: Copy contents inside html folder for python-docs job https://review.openstack.org/499876 | 17:05 |
| *** annegentle has quit IRC | 17:06 | |
| *** jascott1 has quit IRC | 17:07 | |
| *** jascott1 has joined #openstack-infra | 17:07 | |
| *** jamesmcarthur has quit IRC | 17:11 | |
| *** e0ne has joined #openstack-infra | 17:15 | |
| *** salv-orlando has quit IRC | 17:16 | |
| *** Goneri has quit IRC | 17:17 | |
| *** tosky has quit IRC | 17:17 | |
| *** ramishra has quit IRC | 17:18 | |
| *** sambetts is now known as sambetts|afk | 17:19 | |
| *** esberglu has joined #openstack-infra | 17:20 | |
| *** thegreenhundred has quit IRC | 17:24 | |
| *** jamesmcarthur has joined #openstack-infra | 17:24 | |
| *** jamesmcarthur has quit IRC | 17:25 | |
| *** ykarel has quit IRC | 17:25 | |
| *** jamesmcarthur has joined #openstack-infra | 17:25 | |
| *** trown is now known as trown|lunch | 17:27 | |
| *** harlowja has quit IRC | 17:30 | |
| *** vhosakot has joined #openstack-infra | 17:32 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Fix python3 encoding issues with zuul_afs https://review.openstack.org/500137 | 17:36 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Validate zuul_stream func test output https://review.openstack.org/499732 | 17:40 |
| openstackgerrit | Terry Wilson proposed openstack-infra/project-config master: Add neutron tempest test to ovsdbapp gate https://review.openstack.org/499238 | 17:40 |
| *** jascott1 has quit IRC | 17:40 | |
| openstackgerrit | Merged openstack-infra/shade master: Use new keystoneauth version discovery https://review.openstack.org/493582 | 17:40 |
| *** pkovar has quit IRC | 17:47 | |
| *** jascott1 has joined #openstack-infra | 17:55 | |
| *** bnemec is now known as beekneemech | 17:56 | |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: add a devstack-legacy job https://review.openstack.org/497699 | 17:59 |
| *** tosky has joined #openstack-infra | 18:01 | |
| *** rhallisey has quit IRC | 18:07 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Fix python3 encoding issues with zuul_afs https://review.openstack.org/500137 | 18:09 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: WIP: Refactor run-docs role to use tox role https://review.openstack.org/500155 | 18:10 |
| *** harlowja has joined #openstack-infra | 18:12 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul feature/zuulv3: Always generate the ARA report, even on failure https://review.openstack.org/500159 | 18:16 |
| pabelanger | jeblair: mordred: https://docs.openstack.org/sandbox/ displays properly now | 18:16 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul feature/zuulv3: Always generate the ARA report, even on failure https://review.openstack.org/500159 | 18:17 |
| *** Swami has joined #openstack-infra | 18:17 | |
| jeblair | pabelanger: w00t! | 18:17 |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate master: Create aligned partitions in disk fixing https://review.openstack.org/499314 | 18:17 |
| *** salv-orlando has joined #openstack-infra | 18:17 | |
| clarkb | dmsimard: ^ there is that rebase | 18:17 |
| pabelanger | jeblair: mordred: so, if we wanted to have a job do vos release docs volume, would it make sense to to use a mutex like we did for debian package jobs? | 18:18 |
| jeblair | pabelanger: let's leave that alone for now; it's complicated | 18:20 |
| *** Yogi1 has joined #openstack-infra | 18:20 | |
| clarkb | did that feature to only run one post job get in? | 18:20 |
| clarkb | because that would be even better than a mutex | 18:20 |
| pabelanger | jeblair: sure, didn't expect us to get that before PTG | 18:20 |
| jeblair | clarkb: nope, i'm waiting until after ptg to write that | 18:20 |
| dmsimard | clarkb: wow, you ended up needing to do all that math after all ? | 18:20 |
| clarkb | ok | 18:20 |
| clarkb | dmsimard: ya I couldn't get it to work just with parted | 18:20 |
| clarkb | dmsimard: thankfully the math is simple enough it works in bash | 18:20 |
| dmsimard | clarkb: that's disgusting :( | 18:21 |
| dmsimard | not your fault | 18:21 |
| dmsimard | but :( | 18:21 |
| clarkb | ya seems like partition tools should do that for us | 18:21 |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: add a devstack-legacy job https://review.openstack.org/497699 | 18:21 |
| clarkb | I recently discovered how difficult it is to get the ubuntu installer to make a bigger /boot | 18:21 |
| clarkb | you have to partition it completely manually otherwise its 512MB hope you don't want 2 kernels | 18:21 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: WIP: Refactor run-docs role to use tox role https://review.openstack.org/500155 | 18:21 |
| *** salv-orlando has quit IRC | 18:22 | |
| *** rkukura_ has joined #openstack-infra | 18:26 | |
| dmsimard | clarkb: lol | 18:27 |
| *** kjackal has quit IRC | 18:27 | |
| *** jamesmcarthur has quit IRC | 18:28 | |
| *** rkukura has quit IRC | 18:28 | |
| *** rkukura_ is now known as rkukura | 18:28 | |
| Yogi1 | Hi, I hit with error with tox - https://github.com/tox-dev/tox/issues/595 | 18:28 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add helper script for doing local log streaming tests https://review.openstack.org/500161 | 18:28 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Fix spacer-lines to work with multi-node and items https://review.openstack.org/500162 | 18:28 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Stop output start and end times for each task https://review.openstack.org/500163 | 18:28 |
| Yogi1 | wonder if anyone aware/noticed of it | 18:28 |
| *** trown|lunch is now known as trown | 18:29 | |
| clarkb | Yogi1: I haven't seen it break our jobs yet | 18:30 |
| clarkb | http://logs.openstack.org/79/498979/1/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/b918da4/logs/pip2-freeze.txt.gz looks like we are on the last release still | 18:31 |
| clarkb | tox is not in constraints so I expect we'll start using the newer version soon | 18:32 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add zuul.timeout variable to jobs https://review.openstack.org/500114 | 18:32 |
| *** yamahata has quit IRC | 18:32 | |
| Yogi1 | clarkb Our tox is 2.8.0 | 18:32 |
| *** david-lyle has joined #openstack-infra | 18:33 | |
| Yogi1 | clarkb http://logs.openstack.netapp.com/ci-logs/logs/65/499765/5/upstream-check/cinder-cDOT-FCP/3566c47/logs/pip2-freeze.txt.gz | 18:33 |
| *** iyamahat_ has quit IRC | 18:33 | |
| openstackgerrit | Merged openstack-infra/project-config master: Zuul v3: Use mirrored git repos in base job https://review.openstack.org/499854 | 18:33 |
| clarkb | Yogi1: I have reproduced it locally. Going to poke at it more with pdb | 18:36 |
| fungi | oh joy, new tox today i guess | 18:37 |
| fungi | reading the release notes now | 18:37 |
| clarkb | also python2 and python3 both fail... in different ways :( | 18:37 |
| clarkb | oh maybe its same way just different error message, but still | 18:38 |
| *** e0ne has quit IRC | 18:38 | |
| fungi | the fix for https://github.com/tox-dev/tox/issues/301 seems to be the most significant behavior change there | 18:38 |
| fungi | i know parameter expansion has bitten us in the past | 18:38 |
| fungi | or, rather, changes to default behaviors of tox's parameter substitution/expansion | 18:39 |
| fungi | too easy to wind up relying on unintentional bugs in its behavior | 18:39 |
| clarkb | it seems to dislike https://git.openstack.org/cgit/openstack/tempest/tree/tox.ini#n46 | 18:42 |
| clarkb | it is expecting to be able to split the setenv line on = | 18:42 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Only run zuul-stream testing on callback changes https://review.openstack.org/500168 | 18:42 |
| clarkb | ya thats it | 18:43 |
| *** annegentle has joined #openstack-infra | 18:43 | |
| clarkb | or at least that behavior could explain it | 18:43 |
| fungi | that seems to support to my suspicions as to which change was at fault in this release | 18:44 |
| fungi | so i guess they broke setenv inheritence that way | 18:45 |
| *** hashar has joined #openstack-infra | 18:46 | |
| *** hashar has quit IRC | 18:46 | |
| *** hashar has joined #openstack-infra | 18:46 | |
| fungi | though skimming https://github.com/tox-dev/tox/commit/2c863a7 it's not immediately apparent to me where the bug is if that was actually it | 18:47 |
| clarkb | ya me either, also by moving whitespace I am able to get it to fail on different testenv sections... | 18:48 |
| fungi | gah | 18:48 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config master: add cells v1 job to devstack experimental queue https://review.openstack.org/500169 | 18:49 |
| sdague | clarkb / fungi ^^^ | 18:49 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Allow domain_id for roles https://review.openstack.org/496992 | 18:49 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Move role normalization to normalize.py https://review.openstack.org/500170 | 18:49 |
| clarkb | fungi: that may be because configparser doesn't preserve order? | 18:49 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: WIP: Refactor run-docs role to use tox role https://review.openstack.org/500155 | 18:50 |
| fungi | clarkb: oh, yeah it's just a dict of dicts basically | 18:50 |
| *** Goneri has joined #openstack-infra | 18:50 | |
| fungi | we're likely going to need to hold image updates until this gets fixed (we do still bake tox into the images, right?) | 18:52 |
| clarkb | fungi: that may explain why the pip_install tox in devstack/lib/tempest isn't upgrading since tox is already installed | 18:53 |
| clarkb | heh there is a handy commented out print in tox/config.py in the string replacer /me uncomments it | 18:56 |
| *** jamesmcarthur has joined #openstack-infra | 18:59 | |
| clarkb | we are not entering https://github.com/tox-dev/tox/commit/6889c81a4c31fedeb22af056e9f2b7e8f86f00cf#diff-2c047dd01e630cb8ae1845ec82f1a89eL1018 that block | 19:00 |
| fungi | yep, the traceback also seemed to indicate we weren't at least bubbling up from in tox.config.Replacer.do_replace() anyway | 19:02 |
| fungi | casting a wider net in the commit history now | 19:03 |
| *** ijw has joined #openstack-infra | 19:03 | |
| fungi | there was one other related to env handling | 19:04 |
| *** rossella_s has joined #openstack-infra | 19:04 | |
| fungi | "#515: Don’t require environment variables in test environments where they are not used." | 19:04 |
| *** esberglu has quit IRC | 19:04 | |
| clarkb | replace=name in config.envlist <- is the check resulting in false that skips the replacement | 19:04 |
| *** esberglu has joined #openstack-infra | 19:05 | |
| fungi | https://github.com/tox-dev/tox/pull/521 | 19:05 |
| fungi | ahh, yeah that gets pretty deep into altering env: handling | 19:06 |
| clarkb | looking at config.envlist it appears to only be a partial list | 19:07 |
| clarkb | so it is doing all-plugin in ['full'] | 19:07 |
| clarkb | ok I think I understand | 19:08 |
| fungi | the changes done for that pr do seem to be altering setenv to pass in replace=True in a number of places | 19:08 |
| clarkb | you invoke tox with tox -efull that means envlist is ['full'] but the config parser is going to parse the all-plugin env and break because it won't do replacement on that | 19:08 |
| *** jamesmcarthur has quit IRC | 19:08 | |
| clarkb | if you invoke it with all the envs -efoo -ebar etc it should work | 19:09 |
| clarkb | so I think the trick here is it shouldn't parse envs that aren't in the env list's dep tree | 19:09 |
| fungi | and [testenv:all] doesn't explicitly depend on [tempestenv] | 19:10 |
| fungi | but still tries to substitute/inherit from it | 19:11 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: WIP: Refactor run-docs role to use tox role https://review.openstack.org/500155 | 19:11 |
| *** Guest89418 has joined #openstack-infra | 19:11 | |
| clarkb | fungi: ah | 19:11 |
| clarkb | ya lets test replacing testenv: to tempestenv: | 19:11 |
| clarkb | ok testenv is magical | 19:13 |
| clarkb | you have to use testenv | 19:13 |
| fungi | magically tragical | 19:13 |
| clarkb | but switching everything to testenv does not fix it | 19:15 |
| fungi | i'm trying to think of a way to accomplish what this was doing while using more straightforward inheritence | 19:15 |
| fungi | mtreinish: you may want a piece of this action too ^ | 19:16 |
| fungi | this is clearly an intentional behavior change in tox so i doubt we're going to get very far in asking them to undo it | 19:17 |
| clarkb | I've responded to the bug which has people looking into it | 19:17 |
| *** Guest89418 has quit IRC | 19:17 | |
| *** nicolasbock has quit IRC | 19:18 | |
| clarkb | also someone totally put their aws access key in the bug content | 19:18 |
| *** salv-orlando has joined #openstack-infra | 19:18 | |
| fungi | wow | 19:18 |
| jeblair | yay new cloud! | 19:19 |
| fungi | maybe those are throwaway, or substitutions | 19:19 |
| * fungi probably has too much faith in humanity | 19:20 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Pin tox to 2.7.0 https://review.openstack.org/500175 | 19:21 |
| clarkb | fungi: ^ that should pin our tox install in dib | 19:21 |
| clarkb | reading the bug causing PR the intent was to have this behavior | 19:23 |
| *** salv-orlando has quit IRC | 19:24 | |
| fungi | yeah, odds are we need to find an alternative configuration for tempest (and whatever else was using similar non-inherited substitution patterns) | 19:25 |
| clarkb | though reading https://github.com/tox-dev/tox/issues/515 there may be another fix that is reasonable | 19:25 |
| clarkb | can another infra-root or project config core review 500175 so that we can hopefully cut off the tox brokeness before it happens to us? | 19:28 |
| pabelanger | +2, so others can look | 19:29 |
| mordred | clarkb: done | 19:29 |
| clarkb | tyty | 19:29 |
| mordred | oh - I just +Ad it :) | 19:29 |
| clarkb | Yogi1: thanks for the notice on that, 500175 should fix it for you if using our dib elements | 19:32 |
| Yogi1 | clarkb Thanks! | 19:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos https://review.openstack.org/489719 | 19:34 |
| mordred | clarkb, pabelanger: interesting food for thought - in the v3 base tox job we have a check to make sure tox is installed and if it's not we pip install --user tox ... obviously the network is happier when we have tox pre-installed on the machine, but bugs like this one make me wonder whether we should consider not installing tox in the base images once we're on v3, as it w'd have a more flexible way | 19:37 |
| mordred | to respond if we didn't | 19:37 |
| mordred | TOTALLY not important rightnow, nor a thing we have to deal with soon - just came to mind from today's bug/pin | 19:37 |
| *** rossella_s has quit IRC | 19:38 | |
| openstackgerrit | Merged openstack-infra/project-config master: Pin tox to 2.7.0 https://review.openstack.org/500175 | 19:38 |
| *** rossella_s has joined #openstack-infra | 19:39 | |
| pabelanger | mordred: clarkb: ++ I wanted to bikeshed on that topic at PTG also | 19:39 |
| openstackgerrit | Merged openstack-infra/project-config master: add cells v1 job to devstack experimental queue https://review.openstack.org/500169 | 19:42 |
| *** annegentle has quit IRC | 19:43 | |
| clarkb | heh that github bug totally updated with details on my project-config change beacuse I put the url in there | 19:44 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos https://review.openstack.org/489719 | 19:44 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Validate zuul_stream func test output https://review.openstack.org/499732 | 19:44 |
| *** gouthamr has quit IRC | 19:51 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos https://review.openstack.org/489719 | 19:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Rename tox_command_line in docs to tox_extra_args https://review.openstack.org/489758 | 19:53 |
| *** annegentle has joined #openstack-infra | 19:53 | |
| *** annegentle has quit IRC | 19:54 | |
| mtreinish | fungi: a new tox release broke the var substition in tempest's tox.ini? | 19:56 |
| mtreinish | it took me forever to figure out how to make that work short of duplicating everything in every job deinition | 19:56 |
| clarkb | mtreinish: yes | 19:56 |
| clarkb | https://github.com/tox-dev/tox/issues/595 is bug | 19:57 |
| *** jamesmcarthur has joined #openstack-infra | 19:59 | |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify Check User in Group https://review.openstack.org/499357 | 20:02 |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: De-client-ify Remove User from Group https://review.openstack.org/499360 | 20:02 |
| *** jamesmcarthur has quit IRC | 20:03 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Override tox requirments with zuul git repos https://review.openstack.org/489719 | 20:03 |
| *** baoli_ has quit IRC | 20:04 | |
| fungi | mtreinish: moral of the story is, if you attempt to do anything mildly complex with tox parameters and inheritence, they will eventually break it because they see the behavior you ended up relying on as a bug to be fixed | 20:04 |
| mtreinish | yep, that's kinda how I ended up with all that weird tempestenv stuff in the first place... | 20:06 |
| *** mat128 has quit IRC | 20:06 | |
| clarkb | it does look like they are treating this as a bug rather than a fixed bug though | 20:06 |
| *** mat128 has joined #openstack-infra | 20:06 | |
| pabelanger | clarkb: dmsimard: I haven't been following swap issue much, but is http://logs.openstack.org/57/499457/1/gate/gate-tripleo-ci-centos-7-containers-multinode/ccbe914/logs/devstack-gate-setup-host.txt known? | 20:07 |
| pabelanger | Error: Error informing the kernel about modifications to partition /dev/xvde1 -- Device or resource busy. This means Linux won't know about any changes you made to /dev/xvde1 until you reboot -- so you shouldn't mount it or use it in any way before rebooting. | 20:07 |
| mtreinish | fungi: what I really wanted to do was just define 2 'testenv' things and have the different types of jobs depend on one or the other | 20:07 |
| mtreinish | which you couldn't do | 20:07 |
| clarkb | pabelanger: new to me, also these problems are only happening on centos7 | 20:08 |
| clarkb | pabelanger: that is why tripleo is affected more than anything else | 20:08 |
| pabelanger | clarkb: that is possible, let me check logstash and see if centos is failing more | 20:09 |
| clarkb | pabelanger: it is or at least when I looekd it is 100% centos7 | 20:09 |
| clarkb | the only xenial hit was a misparse by logstash | 20:09 |
| *** mat128 has quit IRC | 20:10 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Rename tox_command_line in docs to tox_extra_args https://review.openstack.org/489758 | 20:10 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add publish-to-pypi-quietly project-template https://review.openstack.org/499364 | 20:11 |
| *** Yogi1 has quit IRC | 20:12 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Make log streaming test three node https://review.openstack.org/500049 | 20:15 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add helper script for doing local log streaming tests https://review.openstack.org/500161 | 20:15 |
| openstackgerrit | Paul Belanger proposed openstack-infra/elastic-recheck master: Add query for bug 1714586 https://review.openstack.org/500184 | 20:18 |
| openstack | bug 1714586 in tripleo "Add missing BarbicanApi service" [Critical,Triaged] https://launchpad.net/bugs/1714586 | 20:18 |
| *** rossella_s has quit IRC | 20:18 | |
| *** salv-orlando has joined #openstack-infra | 20:21 | |
| *** rossella_s has joined #openstack-infra | 20:21 | |
| jeblair | pabelanger, clarkb: i thought that error was one that prompted dmsimard to want to add back the sync/partprobe. did that change land? | 20:21 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Use direct calls to get_<resource>_by_id https://review.openstack.org/494535 | 20:21 |
| pabelanger | jeblair: clarkb: not sure, I haven't really been following. looking for it now | 20:22 |
| *** dhill_ is now known as dhill|brb | 20:23 | |
| clarkb | oh that is the device busy error. Except that happened before the sync | 20:23 |
| pabelanger | ya, it looks like dmsimard patch is merged | 20:23 |
| *** salv-orlando has quit IRC | 20:26 | |
| pabelanger | jeblair: dmsimard: clarkb: so, do we need to move sync up a bit? follow up question, would we consider a revert until next week until tripleo releases pike? I believe EmilienM is hoping to tag everything then | 20:27 |
| jeblair | pabelanger: that error happened with the sync/partprobe patch in place? | 20:27 |
| clarkb | pabelanger: that log is from ~18 hours ago. We should probably check if anything has changed since dmsimard chagne went in | 20:28 |
| *** rama_y has quit IRC | 20:28 | |
| *** rama_y_ has joined #openstack-infra | 20:28 | |
| clarkb | the e-r query was updated as well | 20:28 |
| *** gouthamr has joined #openstack-infra | 20:29 | |
| clarkb | there are recent hits in lgostash though | 20:29 |
| clarkb | http://logs.openstack.org/90/487390/2/gate/gate-tripleo-ci-centos-7-scenario004-multinode-oooq-container/68f6fde/logs/devstack-gate-setup-host.txt | 20:29 |
| clarkb | internets seem to think this could happen if something on that device is already mounted when partitioned | 20:30 |
| clarkb | our ansible facts say that isn't the case though | 20:31 |
| *** rcernin has joined #openstack-infra | 20:31 | |
| pabelanger | clarkb: http://logs.openstack.org/90/487390/2/gate/gate-tripleo-ci-centos-7-scenario004-multinode-oooq-container/68f6fde/logs/devstack-gate-setup-host.txt is from a few moments ago | 20:32 |
| clarkb | ya the one I linked is recent too | 20:32 |
| pabelanger | jeblair: yes, it looks like it happens before we get to sync/partprobe now | 20:32 |
| pabelanger | should we sync between each parted command? | 20:32 |
| clarkb | no I doubt sync will do much since it is supposed to work on filesystems not partitions or devices | 20:33 |
| clarkb | (that was mostly a belts and suspenders thing initially) | 20:33 |
| *** Goneri has quit IRC | 20:34 | |
| clarkb | but ya we may want to delay between each parted command | 20:34 |
| clarkb | so move the udevadm command for each partition to right after we partition it | 20:35 |
| *** camunoz has quit IRC | 20:35 | |
| pabelanger | okay, I don't think I have bandwidth to work on that right now, but happy to review patches | 20:35 |
| pabelanger | but think this might be an issue for tripleo next week on the final push to release pike | 20:35 |
| jeblair | clarkb, pabelanger, dmsimard: what if the parted mklabel command is triggering some udev work which interferes with the parted mkpart command? can we combine those into one command? | 20:36 |
| *** marst has quit IRC | 20:37 | |
| jeblair | clarkb: aha | 20:37 |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate master: Check udevadm early https://review.openstack.org/500186 | 20:37 |
| jeblair | clarkb: i think you are thinking along similar lines :) | 20:37 |
| clarkb | jeblair: something like ^ ? | 20:37 |
| * clarkb grabs lunch | 20:37 | |
| jeblair | clarkb: oh i thought it was DEV1 that was erroring | 20:38 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for bug 1714586 https://review.openstack.org/500184 | 20:39 |
| openstack | bug 1714586 in tripleo "Add missing BarbicanApi service" [Critical,Triaged] https://launchpad.net/bugs/1714586 | 20:39 |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Combine parted script into one command https://review.openstack.org/500188 | 20:41 |
| jeblair | clarkb, pabelanger, dmsimard: ^ alternative | 20:41 |
| pabelanger | ya, that might be better | 20:41 |
| *** thorst_afk has quit IRC | 20:46 | |
| *** esberglu has quit IRC | 20:49 | |
| *** kgiusti has left #openstack-infra | 20:49 | |
| *** esberglu has joined #openstack-infra | 20:50 | |
| *** ijw has quit IRC | 20:50 | |
| *** slaweq has joined #openstack-infra | 20:52 | |
| *** ijw has joined #openstack-infra | 20:52 | |
| *** jcoufal has quit IRC | 20:53 | |
| *** rossella_s has quit IRC | 20:54 | |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: put repos into $BASE/new https://review.openstack.org/496958 | 20:54 |
| *** esberglu has quit IRC | 20:54 | |
| *** rcernin has quit IRC | 20:55 | |
| *** ijw has quit IRC | 20:56 | |
| pabelanger | jeblair: not to distract on devstack-gate front, but is you get a chance to look at 500155 this afternoon, it will help towards getting afs-docs for shade. | 20:56 |
| *** ijw has joined #openstack-infra | 20:56 | |
| pabelanger | need to get some food, but then going to add branch support to afs-docs | 20:56 |
| pabelanger | we only support master ATM | 20:56 |
| jeblair | pabelanger: oh sorry, i had a question | 20:56 |
| jeblair | pabelanger: why are you changing the quotation marks? | 20:56 |
| *** tinwood has quit IRC | 20:57 | |
| *** tinwood has joined #openstack-infra | 20:58 | |
| jeblair | pabelanger: also -- doesn't that change maken that the openstack-doc-build job will run "tox -e docs" instead of "python steup.py build_sphinx" ? | 20:59 |
| *** slaweq has quit IRC | 21:00 | |
| *** srobert has quit IRC | 21:01 | |
| *** iyamahat has joined #openstack-infra | 21:02 | |
| *** Apoorva has quit IRC | 21:03 | |
| *** ijw has quit IRC | 21:05 | |
| clarkb | devstack is now failing on the tox problem | 21:07 |
| mordred | jeblair: no- tox_extra_args is set in zuul.yaml | 21:07 |
| clarkb | I'm going to track down the images that have 2.8 on them and delete them | 21:07 |
| clarkb | just as soon as I load my ssh key apparently | 21:08 |
| mordred | jeblair: I left a note about neeting tox_envlist: venv in there though | 21:08 |
| mordred | clarkb: minor details | 21:08 |
| jeblair | mordred: i see that now; re-evaluating | 21:08 |
| jeblair | mordred, pabelanger: okay got it. then yeah, i like that with some minor changes | 21:09 |
| *** Apoorva has joined #openstack-infra | 21:10 | |
| *** akscram1 has quit IRC | 21:10 | |
| mordred | pabelanger: I also left two notes - which are things I'm 100% sure you already know and have planned for your next patch | 21:11 |
| *** yamahata has joined #openstack-infra | 21:12 | |
| clarkb | ok dib images are marked deleted which transitively deletes uploads iirc | 21:13 |
| *** akscram1 has joined #openstack-infra | 21:13 | |
| fungi | had some just started coming online i guess (we're not seeing devstack auto-upgrading tox on its own or anything, right)? | 21:14 |
| clarkb | fungi: ya image builds all finished just before my change merged and now they are coming online | 21:15 |
| *** ldnunes has quit IRC | 21:15 | |
| clarkb | (uploads completing) | 21:16 |
| fungi | gah, okay | 21:16 |
| *** gouthamr has quit IRC | 21:16 | |
| mordred | jeblair, pabelanger: what did theprocess windup being for getting afs keytabs into secrets? | 21:19 |
| *** hashar has quit IRC | 21:21 | |
| *** salv-orlando has joined #openstack-infra | 21:21 | |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: create stack and tempest users https://review.openstack.org/496959 | 21:21 |
| jeblair | mordred: i think you base64 encode them, then have ansible base64decode them when writing (using jinja |base64decode filter) | 21:23 |
| mordred | jeblair: cool - thanks | 21:24 |
| jeblair | clarkb, fungi: do either of you want to review devstack-legacy or should i push it through? https://review.openstack.org/497699 | 21:25 |
| *** salv-orlando has quit IRC | 21:26 | |
| pabelanger | jeblair: the quote changes were to fix an ansible bug I think, I was getting jinja2 failures, but I can look again at it: http://logs.openstack.org/55/500155/2/check/tox-linters/be8982c/job-output.txt.gz#_2017-09-01_18_24_04_809606 | 21:26 |
| pabelanger | mordred: jeblair: looking at comments now | 21:27 |
| jeblair | pabelanger: that makes sense; you probably only need to change the one in the comment, since the others are matched. | 21:27 |
| pabelanger | jeblair: agree | 21:27 |
| jeblair | pabelanger: i thought that script was already in use, which is why i was asking :) | 21:27 |
| pabelanger | jeblair: ya, actually don't have the issue with 499778, but I can try and figure out why | 21:28 |
| clarkb | jeblair: looks like it ran tempest. I don't know that I need to get into the ansible details. That said do we know why it took so long? | 21:28 |
| jeblair | clarkb: infracloud | 21:29 |
| *** jtomasek has quit IRC | 21:32 | |
| *** thorst_afk has joined #openstack-infra | 21:32 | |
| jeblair | clarkb: it spent 10m downloading qemu packages from the mirror | 21:33 |
| clarkb | nice | 21:33 |
| *** jamesmcarthur has joined #openstack-infra | 21:34 | |
| jeblair | clarkb: the rest of the 30m devstack setup seems like a trickle of things. | 21:34 |
| jeblair | http://logs.openstack.org/99/497699/30/check/devstack-legacy-tempest-dsvm-neutron-full/98b1560/logs/devstacklog.txt.gz#_2017-09-01_19_02_47_506 | 21:34 |
| clarkb | osc alone is over 5 minutes :( | 21:34 |
| jeblair | what is osc? | 21:35 |
| jeblair | i don't see "time_start osc" | 21:35 |
| clarkb | openstackclient commands | 21:35 |
| clarkb | that is setting up users and networks and images and flavors | 21:35 |
| clarkb | problem there is entrypoints is slow as molasses | 21:35 |
| jeblair | okay, so that shouldn't be any external network stuff | 21:35 |
| *** ijw has joined #openstack-infra | 21:36 | |
| clarkb | no, that is largely disk io bound as pkg resources has to stat all the places looking for entrypoints | 21:36 |
| jeblair | so it seems like a general combo of network issues and io slowness; the known weak points of infracloud | 21:36 |
| *** thorst_afk has quit IRC | 21:36 | |
| mordred | jeblair: pabelanger's new afs token role aklog's for us - is that safe to use for vos release or do we need to use a keytab for that? | 21:37 |
| mordred | (wondering about timeouts and I have details paged out of my mindgrapes) | 21:37 |
| *** jamesmcarthur has quit IRC | 21:38 | |
| jeblair | mordred: ah, i think you're asking should we use "k5start <command>" to handle token renewing (since pabelanger *does* use a keytab, he just uses it with aklog rather than k5start) | 21:39 |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade master: De-client-ify baremetal node_set_provision_state https://review.openstack.org/499773 | 21:39 |
| mordred | jeblair: yes - sorry - inexact words | 21:39 |
| jeblair | mordred: i think the aklog thing is fine -- default token lifetime is something like 10 or 12 hours | 21:39 |
| mordred | ok. cool | 21:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add role to do a vos release on an AFS volume https://review.openstack.org/500200 | 21:39 |
| *** priteau has quit IRC | 21:39 | |
| mordred | gah. bad README | 21:39 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Refactor run-docs role to use tox role https://review.openstack.org/500155 | 21:40 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add role to do a vos release on an AFS volume https://review.openstack.org/500200 | 21:40 |
| mordred | pabelanger: thanks for doing the hard work for the afs stuff :) | 21:40 |
| pabelanger | mordred: I think jeblair did the hard work with bwrap, I just ran the commands :D | 21:41 |
| pabelanger | but awesome that is just works | 21:41 |
| *** EricGonczer_ has joined #openstack-infra | 21:43 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/shade master: WIP: Add openstack-doc-build to shade https://review.openstack.org/500201 | 21:44 |
| fungi | jeblair: oh! i didn't have devstack-gate included in my zuulv3 dashboard. thanks for the heads up | 21:45 |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: run devstack https://review.openstack.org/500202 | 21:45 |
| fungi | revising now | 21:45 |
| mordred | jeblair, pabelanger: ok.you ready for a FUN patch? | 21:45 |
| *** EricGonc_ has joined #openstack-infra | 21:45 | |
| mordred | clarkb, fungi: you too - this next one is fun | 21:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add wheel mirror jobs https://review.openstack.org/499843 | 21:46 |
| mordred | boom | 21:46 |
| pabelanger | giggles | 21:46 |
| mordred | periodic pipeline, afs release, replacing a pile of jobs witha single multi-node job, use of final and allowed-projects - basically that's all the bells and whistles right there | 21:46 |
| mordred | and it's in merge conflict - lemme fix | 21:47 |
| fungi | i expanded my topic:zuulv3 to project:^openstack-infra/.* and now i have quite a few additional things to review | 21:47 |
| fungi | (including the devstack-gate legacy bits) | 21:47 |
| jeblair | mordred: nice! | 21:47 |
| *** EricGonczer_ has quit IRC | 21:48 | |
| *** thorst_afk has joined #openstack-infra | 21:48 | |
| pabelanger | mordred: that is pretty nice, I think I see a minor issue. Let me think for a moment | 21:49 |
| mordred | there's definitely an issue - I just found one myself | 21:49 |
| *** thorst_afk has quit IRC | 21:50 | |
| *** ihrachys has quit IRC | 21:50 | |
| pabelanger | mordred: exciting! 2 comments left | 21:53 |
| fungi | i'm still getting up to speed on some of these ansible instruction patterns... for example we have a "Create nodepool subnodes file" task which copies an empty string to /etc/nodepool/sub_nodes_private and then a "Populate nodepool subnodes file" task which writes lines into that file. i have to assume there's a reason for that separation... is a lineinfile task not able to create its target file | 21:58 |
| fungi | when it starts writing? | 21:58 |
| *** slaweq has joined #openstack-infra | 21:59 | |
| fungi | or is this for improved error handling, so that we get an explicit error when attempting to create the empty file there, rather than conflating it with errors related to the data it's about to write into it in the subsequent task? | 22:00 |
| *** slaweq has quit IRC | 22:00 | |
| *** bobh has quit IRC | 22:00 | |
| pabelanger | fungi: clarkb: mordred: dmsimard: would you like to review jeblair patch at https://review.openstack.org/500188/ in an effort to help reduce parted failures we are now seeing in devstack-gate | 22:01 |
| jeblair | fungi: i'm under the impression it's the former (unable to create) unless an ansible expert corrects me | 22:01 |
| jeblair | fungi: i have a memory of testing that and running into problems | 22:01 |
| fungi | or does lineinfile maybe have an unfortunate behavior of not creating the file unless it has content to write, and so we want to make sure we have an empty file in that situation? | 22:01 |
| jeblair | fungi: oh maybe that's it :) | 22:01 |
| jeblair | (and sorry, if i'd known i was going to forget this, i would have left a comment!) | 22:02 |
| pabelanger | I don't use lineinfile much myself, so not sure the behavior | 22:02 |
| fungi | mainly wanting to know under what circumstances i should be wary when a task isn't creating a file another task needs to write into | 22:02 |
| jeblair | fungi: i'm going to guess it's actually your third thing -- we want the empty file in the single-node case. | 22:03 |
| fungi | jeblair: thanks, seems like a good reason ;) | 22:03 |
| clarkb | pabelanger: I wanted to see it work before I approved it, that is how I found the tox errors (it hit them) | 22:04 |
| fungi | eep | 22:04 |
| pabelanger | clarkb: sure, have we started building new images? | 22:04 |
| dmsimard | pabelanger, jeblair, clarkb: man that xvde thing is a <expletive>ing curse | 22:04 |
| clarkb | pabelanger: yes, but we don't need new iamges we have yesterday's | 22:05 |
| pabelanger | clarkb: okay, haven't been following, is the plan to revert / pause to yesterday until we build replacements? | 22:06 |
| pabelanger | or just delete todays images and build new ones | 22:06 |
| dmsimard | pabelanger, jeblair, clarkb: there are definitely still hits, even after re-introducing partprobe/sync so we're technically not only on par with pure bash but we've added udevadm settle on top of it | 22:08 |
| pabelanger | okay, it looks like we just deleted the latest DIB, and in the process of new builds | 22:08 |
| dmsimard | According to http://status.openstack.org/elastic-recheck/index.html#1706103 -- the issue is only affecting CentOS nodes | 22:08 |
| *** tesseract has quit IRC | 22:09 | |
| pabelanger | dmsimard: yes, some discussion in back scroll from jeblair and clarkb | 22:10 |
| *** jamesmcarthur has joined #openstack-infra | 22:10 | |
| jeblair | pabelanger: i'm confused about why you're talking about images -- we're building new images due to the tox problem. the device issue doesn't need new images. | 22:10 |
| *** rlandy has quit IRC | 22:10 | |
| clarkb | jeblair: your parted change ran into errors due to tox | 22:10 |
| jeblair | aaah, thanks | 22:10 |
| clarkb | so I'm getting that sorted so we can properly review your change | 22:11 |
| pabelanger | jeblair: sorry, crossing streams. Just seen tox issue for a tripleo job and your patch, so I wasnt' sure if we rolled new images yet | 22:11 |
| clarkb | pabelanger: we don't need new images | 22:11 |
| clarkb | pabelanger: we are using yesterday's images until new images are made | 22:11 |
| jeblair | well, either new or old :) | 22:11 |
| pabelanger | clarkb: okay, then this might be a different issue, tripleo is seeing failures | 22:11 |
| pabelanger | tox failures* | 22:12 |
| clarkb | pabelanger: it could be those jobs started/ran before I deleted the new images | 22:12 |
| pabelanger | clarkb: okay, that is possible | 22:12 |
| pabelanger | let me confirm | 22:12 |
| fungi | pabelanger: have a link to one? they're relatively easy to recognize... exceptions raised in tox/config.py? | 22:13 |
| pabelanger | http://logs.openstack.org/64/496564/3/check/gate-instack-undercloud-python27-ubuntu-xenial/77b1d19/console.html | 22:13 |
| pabelanger | 2017-09-01 19:40:51.536306 is when that job ran | 22:13 |
| *** jamesmcarthur has quit IRC | 22:14 | |
| fungi | pabelanger: yup, that looks like https://github.com/tox-dev/tox/issues/595 | 22:14 |
| pabelanger | 500175 merged 10mins before that, so it is possible we still had images online | 22:14 |
| pabelanger | I've asked them to recheck and see if still an issue | 22:14 |
| *** xarses has quit IRC | 22:15 | |
| fungi | pabelanger: that job failed in roughly one minute, so should be pretty quick to confirm if it's still being hit | 22:15 |
| pabelanger | fungi: great, thanks for looking | 22:15 |
| clarkb | I think I may have the hackiest of hack fixes for tox, will push that up once tests are done | 22:17 |
| pabelanger | okay, 3 more swap failures in gate for centos. So ya, hopefully 500188 is our fix, otherwise, I think we should consider a revert: http://status.openstack.org/elastic-recheck/#1706103 is very spiky | 22:17 |
| clarkb | pabelanger: might also want to file a bug with centos | 22:17 |
| clarkb | is this 7.4 too? | 22:18 |
| dmsimard | 7.4 is not out yet. | 22:18 |
| pabelanger | 7.3 | 22:18 |
| clarkb | ok so not a new regression at least | 22:18 |
| clarkb | (likely not anyways) | 22:18 |
| fungi | has anybody pointed out before that poppy has a copy of war and peace (literally and literarily) in their repo? http://git.openstack.org/cgit/openstack/poppy/tree/docker/e2e_test/flasksite/static/pg2600.txt | 22:20 |
| *** baoli has joined #openstack-infra | 22:22 | |
| *** salv-orlando has joined #openstack-infra | 22:22 | |
| clarkb | bah there is no way to change anything in tox without something else failing | 22:22 |
| clarkb | too many conflicting ideas of what behavior should be | 22:22 |
| dmsimard | I have a suggestion for logstash, not sure how to make it happen | 22:23 |
| dmsimard | but I feel like build_node should contain the node name, not the actual nodepool name | 22:23 |
| dmsimard | like, it should be "centos-7" or "centos-7-2-node" | 22:23 |
| dmsimard | so that we can easily trend pattern by node type | 22:23 |
| dmsimard | and then for the other things, you can have the nodepool cloud provider, cloud region, as fields too. | 22:24 |
| jeblair | dmsimard: that's either the 'label' or 'diskimage' in nodepool terminology (depending on a distinction that we can't actually make until nodepool v3) | 22:24 |
| dmsimard | jeblair: correct | 22:25 |
| jeblair | (diskimage being something like 'centos-7' and label being something like 'centos-7-8gb' or 'centos-7-2gb'; a label is an image+flavor combo) | 22:25 |
| jeblair | (there's only one flavor for each image right now, so label and diskimage are currently 1:1) | 22:25 |
| *** EricGonc_ has quit IRC | 22:26 | |
| *** baoli has quit IRC | 22:26 | |
| jeblair | dmsimard: my suggestion -- wait until after zuulv3 where we will have all of that information plumbed through right up to the point where we hand it off to the logstash parser; then it will be a small change to add it there. | 22:26 |
| *** salv-orlando has quit IRC | 22:27 | |
| dmsimard | sure | 22:27 |
| dmsimard | I was just trying to see if there was a pattern for the xvde stuff other than centos@rax | 22:27 |
| dmsimard | https://review.openstack.org/#/c/500188/ is not passing for the time being, I'll keep looking in the meantime | 22:28 |
| *** thorst_afk has joined #openstack-infra | 22:29 | |
| mordred | jeblair: ZOMG I have been laboring under a false assumption on these wheel build jobs - largely because I quite simply cannot read | 22:29 |
| openstackgerrit | Ben Nemec proposed openstack/diskimage-builder master: Remove nested quotes from TAROPTS https://review.openstack.org/500212 | 22:30 |
| jeblair | dmsimard, clarkb: er, is that because one of us needs to type "recheck" on 500188 ? | 22:30 |
| mordred | jeblair: turns out - we do NOT keep a long-lived cache of things on the wheel build nodes | 22:31 |
| jeblair | mordred: that alters some assumptions indeed | 22:31 |
| mordred | jeblair: which, I believe, means we actually can just use normal nodes | 22:31 |
| mordred | (I mean, this has been a fun exercise and all) | 22:31 |
| *** thorst_afk has quit IRC | 22:32 | |
| pabelanger | Yay | 22:32 |
| dmsimard | clarkb: stumbling on more parted stuff when looking for the device busy error lol.. looks like we're not the only ones getting that kind of issue https://blog.hqcodeshop.fi/archives/273-GNU-Parted-Solving-the-dreaded-The-resulting-partition-is-not-properly-aligned-for-best-performance.html | 22:32 |
| clarkb | jeblair: yes | 22:32 |
| fungi | mordred: i think we can probably speed it up heavily in general too... if we can work out a sane check to cause it to skip building wheels for things that already have appropriate wheels on pypi for the given platform | 22:32 |
| jeblair | clarkb, dmsimard: i have rechecked that change :) | 22:32 |
| mordred | fungi: indeed. do we have an idea of long the job is taking currently? | 22:33 |
| fungi | mordred: now that a lot of the sdists which take a long time to build wheels have started pushing manylinux1 amd64 wheels up to pypi for each release | 22:33 |
| fungi | mordred: how about i get you a number, just a sec | 22:33 |
| *** ijw has quit IRC | 22:33 | |
| dmsimard | jeblair: ack, I'm searching the internet for some clues as to why we're only seeing this behavior on centos | 22:34 |
| openstackgerrit | Paul Belanger proposed openstack-infra/shade master: Add openstack-doc-build to shade https://review.openstack.org/500201 | 22:34 |
| mordred | fungi: cool - I'm going to fix this patch to just use throwaway nodes and not the existing wheel build nodes so we can just delete those nodes (which will be nice) | 22:34 |
| mordred | oh - I say that ... | 22:34 |
| pabelanger | jeblair: mordred: not to distract too much, but are we okay with openstack-doc-build job name, or should be bikeshed more on it? | 22:34 |
| mordred | jeblair, fungi: since we've got the 'use the wheel builder slaves' code pretty much done ... how about we keep it that way til post-ptg at least | 22:34 |
| mordred | pabelanger: I like it | 22:34 |
| clarkb | ok fix pushed to https://github.com/tox-dev/tox/issues/595 | 22:35 |
| pabelanger | mordred: I am excited we can use nodepool for wheel builds | 22:35 |
| *** ijw has joined #openstack-infra | 22:36 | |
| mordred | pabelanger, jeblair, fungi: so - the reason I think we maybe should hold off until post-ptg for that ... is that it'll mean wheel mirror buidlers will shift from being in the same DC as the afs cell to potentially across the ocean | 22:36 |
| mordred | pabelanger, fungi, jeblair: which should work - but maybe isn't a thing we want to debug just now if it is an issue | 22:37 |
| clarkb | except it doesn't fix tempest quite yet | 22:37 |
| pabelanger | mordred: ya, post PTG for that works too and a valid issue. | 22:37 |
| jeblair | mordred: wfm | 22:38 |
| jeblair | mordred: also maybe we implement fungi's optimization then maybe the ocean isn't as big a deal. | 22:39 |
| jeblair | i mean, the ocean is a *big deal*. but still. | 22:39 |
| mordred | ++ | 22:39 |
| fungi | mordred: i looked at today's runs for centos-7 and ubuntu-xenial, python2 and python3 | 22:41 |
| fungi | they ranged from 0.98 hours to 2.12 hours | 22:42 |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-manual master: Add more content to Zuul v3 migration guide https://review.openstack.org/500213 | 22:42 |
| fungi | i should check that all of those actually succeeded | 22:42 |
| mordred | ok. so we may want to set the job timeout to 4.5 hours ... we'll be running python2 and python3 back to back in a single job | 22:42 |
| mordred | but centos-7, ubuntu-xenial and ubuntu-trusty in parallel | 22:42 |
| fungi | i didn't bother to check ubuntu-trusty times | 22:43 |
| dmsimard | clarkb, jeblair: hang on for just a sec.. the "device busy" error occurs on the mkpart step: udevadm settle didn't even get a chance to run yet. The device is there, but it's "busy" and the only thing it could possibly be doing is the umount that occurs the line just before. | 22:43 |
| dmsimard | I can reliably reproduce that error even on a local VM with a while loop | 22:44 |
| dmsimard | while true; do mount -o loop=/dev/loop0 /root/diskimage /mnt; umount /mnt; done | 22:44 |
| pabelanger | mordred: fungi: Ya, we had to split python2 and python3 into different job runs because of timeout issues. But happy to try a single job | 22:44 |
| fungi | confirmed, all four of the ones i checked were successful runs | 22:44 |
| dmsimard | Ran it for just 5 seconds: http://paste.openstack.org/raw/620243/ | 22:45 |
| dmsimard | Going to try and reproduce on a ubuntu VM. | 22:45 |
| mordred | pabelanger: hrm. WELL - we could split 2 and 3 into separate jobs ... as long as they're using the same semaphore it should be fine | 22:45 |
| clarkb | dmsimard: ya that is why jeblair was thinking of collapsing the mkpart script | 22:45 |
| clarkb | dmsimard: so that we in theory don't ahve to do external waiting? | 22:46 |
| dmsimard | clarkb: collapsing wouldn't resolve the issue of the umount that occurs just before | 22:46 |
| fungi | strangely centos-7 python3 was the fastest wheel job, ubuntu-trusty python3 was the slowest | 22:46 |
| fungi | and the two python2 jobs fell in the middle | 22:46 |
| dmsimard | clarkb: I don't do a mkpart in my while loop, I can trigger it easily just by doing umount/mount | 22:46 |
| pabelanger | mordred: ya, happy to play with it post PTG | 22:46 |
| dmsimard | testing on a ubuntu VM right now.. sec | 22:47 |
| clarkb | dmsimard: umount? | 22:49 |
| clarkb | dmsimard: we don't unmount anything in our script do we? | 22:49 |
| dmsimard | clarkb: the line just before | 22:49 |
| dmsimard | clarkb: https://github.com/openstack-infra/devstack-gate/blob/47b2b4d31cabf54633afe97d069cb1e30e50580d/playbooks/roles/fix_disk_layout/tasks/main.yaml#L39 | 22:50 |
| clarkb | dmsimard: do we execute that? | 22:50 |
| clarkb | dmsimard: our ansible facts said it wasn't mounted | 22:50 |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-manual master: Zuul v3: minor changes to quickstart https://review.openstack.org/500214 | 22:50 |
| dmsimard | clarkb: ah, perhaps | 22:50 |
| *** adrianofr_ has quit IRC | 22:53 | |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-manual master: Zuul v3: Add a section on Ansible variables https://review.openstack.org/500216 | 22:53 |
| clarkb | with https://github.com/tox-dev/tox/pull/597 I can now run `tox --notest -efull` in tempest | 22:55 |
| pabelanger | nice | 22:55 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: WIP https://review.openstack.org/498102 | 22:56 |
| *** EricGonczer_ has joined #openstack-infra | 22:57 | |
| clarkb | dmsimard: ya checking an ubuntu job at least it does mount | grep xvde then straight to parted | 22:57 |
| *** Swami has quit IRC | 22:57 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Ansiblify the wheel build a little more https://review.openstack.org/499857 | 22:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add wheel mirror jobs https://review.openstack.org/499843 | 22:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Treat wheel builders more like build nodes https://review.openstack.org/500217 | 22:57 |
| clarkb | dmsimard: is it possible that centos is mounting it automagically for some reason? | 22:57 |
| dmsimard | clarkb: nah I think you're right -- I don't see the "echo" commands in stdout | 22:58 |
| mordred | pabelanger, jeblair, fungi: ^^ ok, there's a new stab at it - I should probably squash that some, but I figured I'd let you watch me make mistakes :) | 22:58 |
| dmsimard | trying to more reliably reproduce the issue with the mkpart commands instead of the umount | 22:58 |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-manual master: Zuul v3: add a section on web-based log streaming https://review.openstack.org/500218 | 22:58 |
| jeblair | mordred, fungi, Shrews: with that ^ there are no more TODO items in the zuulv3 migration page | 22:58 |
| jeblair | (though, once we have the migration script in hand, i'm sure we'll add more. but for now, that covers all the things that i think would be handy for folks to know about v2 -> v3) | 22:59 |
| *** tosky has quit IRC | 22:59 | |
| jeblair | mordred: zuul is grumpy about dependencies of those patches | 22:59 |
| mordred | jeblair: woot! | 22:59 |
| mordred | jeblair: I'm sure it is | 22:59 |
| jeblair | i mean ... s/ about dependencies of those patches/./ | 23:00 |
| mordred | jeblair, pabelanger: https://review.openstack.org/#/c/500200 is an easy one | 23:00 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Ansiblify the wheel build a little more https://review.openstack.org/499857 | 23:01 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add wheel mirror jobs https://review.openstack.org/499843 | 23:01 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Treat wheel builders more like build nodes https://review.openstack.org/500217 | 23:01 |
| mordred | jeblair: it was layout.yaml :) | 23:02 |
| *** rama_y_ has quit IRC | 23:02 | |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: run devstack https://review.openstack.org/500202 | 23:02 |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate master: Zuul v3: run devstack https://review.openstack.org/500202 | 23:05 |
| pabelanger | mordred: -1 on 500200 with question | 23:06 |
| clarkb | dmsimard: hopefully because I assume parted can operate with a single open on the block device the single script will avoid the busy errors. But I dunno | 23:07 |
| clarkb | infra-root thoughts on direct enqueing 500188 into the gate? it tripped over a straggler node and failed with tox on the python35 job. I don't expect it to fail again in the gate and there is logged evidence the parted stuff works | 23:09 |
| jeblair | clarkb: i agree with your analysis; +1 from me | 23:10 |
| mordred | clarkb: ++ | 23:10 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove project stanza https://review.openstack.org/499278 | 23:10 |
| clarkb | ok I'll do that now | 23:11 |
| mordred | pabelanger: responded on the patch - but tl;dr - I coped that straight from the current job | 23:11 |
| mordred | pabelanger: the afs-release buider in jenkins/jobs/wheel-mirror.yaml | 23:11 |
| *** sdague has quit IRC | 23:11 | |
| pabelanger | mordred: flipped to +2 :) | 23:12 |
| jeblair | mordred, clarkb, pabelanger: all afs command options can be reduced to their shortest unambiguous form. therefore "-v" is "-verbose" | 23:12 |
| clarkb | hrm that didn't work, would it have failed because of the verified -1? | 23:12 |
| *** bobh has joined #openstack-infra | 23:12 | |
| clarkb | `zuul enqueue --trigger gerrit --pipeline gate --project openstack-infra/devstack-gate --change 500188,1` is what I ran | 23:12 |
| pabelanger | jeblair: cool, thanks | 23:13 |
| *** gouthamr has joined #openstack-infra | 23:13 | |
| andreaf | anyone for a project-config review? adding release job to hacking: https://review.openstack.org/#/c/499832/ | 23:13 |
| jeblair | clarkb: 2017-09-01 23:11:51,227 DEBUG zuul.DependentPipelineManager: Change <Change 0x7fa6aec99e90 500188,1> does not match pipeline requirement <ChangeishFilter open: True current-patchset: True required_approvals: [{'username': <_sre.SRE_Pattern object at 0x7fa7209be118>, 'verified': [1, 2]}, {'workflow': 1}]> | 23:14 |
| jeblair | clarkb: assuming it had a w+1 at the time, then yeah, i reckon so. | 23:15 |
| clarkb | jeblair: ya I added +W first | 23:15 |
| clarkb | I will use gerrit powers to amend the verified vote | 23:15 |
| jeblair | ++ | 23:15 |
| mordred | clarkb: you could ... yah, that | 23:15 |
| *** thegreenhundred has joined #openstack-infra | 23:16 | |
| openstackgerrit | Merged openstack-dev/hacking master: Release notes for 0.14.0 https://review.openstack.org/499814 | 23:16 |
| clarkb | ok it is queued now so that must've been it | 23:17 |
| *** wolverineav has quit IRC | 23:17 | |
| dmsimard | clarkb: I've found a way to reproduce the error locally and jeblair's change doesn't fix it for me :( | 23:17 |
| *** wolverineav has joined #openstack-infra | 23:17 | |
| clarkb | dmsimard: :( but at least we've got a reproducer | 23:18 |
| pabelanger | :( | 23:18 |
| pabelanger | Yay for reproduing | 23:18 |
| dmsimard | clarkb, pabelanger: https://gist.github.com/dmsimard/bdea83a76d6cc049f808c1110b6c1ca3 | 23:18 |
| dmsimard | disregard attempt #1, I messed up and didn't clean up from a previously interrupted run | 23:18 |
| dmsimard | the mklabel will always fail | 23:19 |
| dmsimard | the mkpart work though | 23:19 |
| pabelanger | dmsimard: Hmm, the commit for jeblair was a single parted command, you are still using 3 | 23:19 |
| *** rwsu has quit IRC | 23:20 | |
| dmsimard | pabelanger: right, and I mentioned his single parted command doesn't fix it and in fact it seems to make matters worse because the mklabel error cascades to the mkpart and prevents them from running | 23:20 |
| dmsimard | so, what I think about this whole mess is that those mklabel/mkpart errors are not fatal | 23:20 |
| pabelanger | dmsimard: ah, okay. y.sh in gist was confusing me | 23:20 |
| *** xarses_ has quit IRC | 23:21 | |
| dmsimard | *but*, when we migrated the script to ansible from functions.sh | 23:21 |
| dmsimard | we added a set -ex as per jeblair's suggestion in a comment | 23:21 |
| dmsimard | I have a feeling that we did not necessarily have an explicit "set -e" before, leading the mklabel/mkpart commands to perhaps print that out but not being fatal | 23:21 |
| dmsimard | I'm trying to look in logstash for previous events | 23:22 |
| jeblair | dmsimard: may we see the script you ran with the single parted command? | 23:22 |
| *** wolverineav has quit IRC | 23:22 | |
| dmsimard | it's the same one but with the single command instead of 3 | 23:23 |
| dmsimard | hang on | 23:23 |
| *** salv-orlando has joined #openstack-infra | 23:23 | |
| *** thegreenhundred has quit IRC | 23:24 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add role to do a vos release on an AFS volume https://review.openstack.org/500200 | 23:24 |
| dmsimard | jeblair: https://gist.github.com/dmsimard/bdea83a76d6cc049f808c1110b6c1ca3#file-x-sh | 23:25 |
| *** rama_y has joined #openstack-infra | 23:25 | |
| clarkb | dmsimard: except that logstash would show us the old failures too | 23:25 |
| clarkb | dmsimard: it definitely wasn't happening before | 23:25 |
| dmsimard | clarkb: yeah, trying to find if there's anything | 23:25 |
| clarkb | (and the more minimal fail case that happened did cause teh job to fail) | 23:25 |
| *** hongbin has quit IRC | 23:26 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Ansiblify the wheel build a little more https://review.openstack.org/499857 | 23:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add wheel mirror jobs https://review.openstack.org/499843 | 23:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Treat wheel builders more like build nodes https://review.openstack.org/500217 | 23:26 |
| jeblair | dmsimard: what does the output look like with the one line script? and what if you put all the sync and udev wait stuff in there? | 23:27 |
| clarkb | the fact that this only hits centos whereas the old fails hit ubuntu too make me think its likely more related to an interaction between the script changes and the distro than just the script changes | 23:27 |
| *** salv-orlando has quit IRC | 23:27 | |
| jeblair | also, this test is with the loopback device; that could be a significantly different code path in the kernel | 23:29 |
| jeblair | it's interesting that the test appears to always fail | 23:29 |
| dmsimard | clarkb: don't mix the two issues | 23:30 |
| dmsimard | clarkb: it's two different problems | 23:30 |
| clarkb | dmsimard: there are? | 23:30 |
| clarkb | dmsimard: mkpart fails | 23:30 |
| clarkb | it only happens on centos | 23:30 |
| clarkb | and became worse after moving the script into ansible | 23:30 |
| dmsimard | clarkb: old (functions.sh): sometimes xvde devices are missing ... new (ansible): device or resource busy | 23:30 |
| dmsimard | missing xvde devices never occured at the mkpart step, only at the mkwap step | 23:31 |
| clarkb | dmsimard: right my point is that I would expect new(ansible) device or resource busy to affect both distros (ubuntu more because we run more jobs on ubuntu I think) if it were just the move to ansible | 23:31 |
| fungi | i suppose if it _is_ representative of the failure we're seeing with non-loop devices, then it's possible losetup calls returning quickly are causing it to consistently lose whatever race this is | 23:31 |
| dmsimard | while now the busy device is only occuring at the mkpart step | 23:31 |
| *** vhosakot has quit IRC | 23:32 | |
| dmsimard | fungi: yeah loopback devices are perhaps not representative of the issue but I don't have a better way :) | 23:32 |
| dmsimard | clarkb: oh, don't get me wrong, http://my1.fr/files/emilien-right-now.jpg | 23:32 |
| dmsimard | I have absolutely no idea why things are different from bash to ansible | 23:33 |
| dmsimard | It should not be, I simply don't understand | 23:33 |
| jeblair | dmsimard: didn't the move to ansible also move it earlier in the process? | 23:33 |
| jeblair | (from setup_workspace to setup_host?) | 23:33 |
| dmsimard | The only thing I could think of, and I don't see why it would make a difference -- and then again, I don't understand to begin with, would be to move the shell task to an actual standalone script instead of embedding it in a shell task | 23:34 |
| clarkb | jeblair: yes | 23:34 |
| *** EricGonczer_ has quit IRC | 23:34 | |
| dmsimard | jeblair: sure, but does anything that occurs before setup_workspace make that kind of difference ? | 23:34 |
| clarkb | dmsimard: could be load related or similar | 23:34 |
| jeblair | dmsimard: if this is a race condition, then it means the system is in a different state with different things happening around the same time (as clarkb says) | 23:35 |
| dmsimard | let me see what else runs in setup_host.. | 23:35 |
| *** vhosakot has joined #openstack-infra | 23:35 | |
| jeblair | (it could have more io to flush, or less) | 23:35 |
| *** markvoelker has quit IRC | 23:36 | |
| *** olaph has quit IRC | 23:36 | |
| dmsimard | I guess, if anything, jeblair's patch can't possibly make matters worse | 23:36 |
| *** olaph has joined #openstack-infra | 23:37 | |
| dmsimard | but it drives me nuts that I can't find why this is happening | 23:37 |
| dmsimard | This is almost exclusively occuring on 1) CentOS 2) TripleO jobs http://i.imgur.com/x6N4Qal.png | 23:42 |
| dmsimard | Yet we run CentOS based jobs elsewhere as well, puppet-openstack has a bunch of integration tests ran all over the place | 23:42 |
| dmsimard | Oh, puppet-openstack doesn't use devstack-gate, duh | 23:43 |
| clarkb | hrm py35 still failing I wonder if we install tox under python35 so it doesn' | 23:45 |
| clarkb | t see the existing one | 23:45 |
| pabelanger | clarkb: http://logs.openstack.org/88/500188/1/gate/gate-tempest-dsvm-py35-ubuntu-xenial/3b2a747/logs/devstacklog.txt.gz#_2017-09-01_23_27_26_436 | 23:48 |
| pabelanger | think that is the issue | 23:49 |
| pabelanger | pulls in 2.8 | 23:49 |
| fungi | oh, yep | 23:49 |
| mordred | ++ | 23:49 |
| dmsimard | jeblair, clarkb: the other weird thing is that, like I've mentioned, that mklabel command (in the 3 command example) returning "device busy" is not fatal because the mkpart and mkfs work and the devices are available | 23:50 |
| clarkb | pabelanger: fungi ya but only on py35 | 23:50 |
| clarkb | I think due to python versions? | 23:50 |
| mordred | we could drop a pip==2.7 into upper-constraints.txt | 23:50 |
| fungi | i was worried we were going to have some jobs installing or upgrading tox on their own at runtime | 23:50 |
| mordred | gah. tox==2.7 | 23:51 |
| dmsimard | pip 2.7 ? that seems ancient | 23:51 |
| dmsimard | okay :) | 23:51 |
| mordred | since that install is going through upper-constraints already | 23:51 |
| mordred | as will a majority of installs in the gate | 23:51 |
| pabelanger | mordred: good idea | 23:51 |
| fungi | i agree, that'll likely solve it in the short term | 23:51 |
| mordred | what's the actual version? | 23:51 |
| fungi | 2.7.0 is what we want | 23:51 |
| mordred | cool; I'll push up a patch | 23:51 |
| fungi | 2.8.0 was the next non-prerelease and is broken | 23:51 |
| mordred | remote: https://review.openstack.org/500221 Add tox to upper-constraints to help pin the gate | 23:53 |
| mordred | I put it in the g-r file too so that the gate jobs for requirements woulnd't barf | 23:53 |
| mordred | gah | 23:54 |
| mordred | remote: https://review.openstack.org/500221 Add tox to upper-constraints to help pin the gate | 23:54 |
| mordred | updated | 23:54 |
| mordred | it's three ='s | 23:54 |
| pabelanger | http://logs.openstack.org/46/493246/2/gate/gate-tripleo-ci-centos-7-nonha-multinode-oooq/582afe5/console.html | 23:55 |
| pabelanger | anybody able to help see why we timed out on ^ | 23:55 |
| pabelanger | I don't see anything in jobs | 23:55 |
| pabelanger | and can only thing a hung git operation? | 23:55 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/devstack-gate master: Revert "Re-introduce sync/partprobe in fix_disk_layout" https://review.openstack.org/500222 | 23:56 |
| dmsimard | clarkb, pabelanger: just in case jeblair's patch doesn't resolve the intermittent issues ^ we'll have a revert patch ready to go | 23:56 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/hacking master: Updated from global requirements https://review.openstack.org/440029 | 23:58 |
| jeblair | dmsimard: wait -- sync + partprobe are what fail? | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
