| ianw | fungi: ^ yep, should help in future | 00:00 |
|---|---|---|
| *** yamamoto has quit IRC | 00:00 | |
| *** sdake has quit IRC | 00:00 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Support requests for specific providers https://review.openstack.org/639418 | 00:00 |
| *** jesusaur has joined #openstack-infra | 00:00 | |
| fungi | i wonder if that explains the nodejs discrepancy between openstackid-dev01 and openstackid01 | 00:01 |
| *** sdake has joined #openstack-infra | 00:01 | |
| fungi | like, did i forget to add --image initially on one of them? i really don't remember, i've screwed that up a few times since we switched the default to bionic | 00:01 |
| *** tosky has quit IRC | 00:02 | |
| ianw | could be ... it's always fascinating how many corner cases we can hit :) | 00:02 |
| openstackgerrit | Merged openstack-infra/system-config master: Revert "Add new health01 server to our inventory" https://review.openstack.org/639451 | 00:02 |
| *** ekultails has quit IRC | 00:02 | |
| *** sdake has quit IRC | 00:05 | |
| *** sdake has joined #openstack-infra | 00:06 | |
| *** markvoelker has joined #openstack-infra | 00:06 | |
| *** sdake has quit IRC | 00:10 | |
| ianw | clarkb: is it just me, or is https://review.openstack.org/639464 not in the gate? | 00:14 |
| clarkb | ianw: it isn't, it failed on that drush thing above | 00:14 |
| clarkb | the puppet module git repo failed to be cloned from. I will need ot recheck it | 00:15 |
| *** mriedem has quit IRC | 00:15 | |
| clarkb | just as soon as it finished (~2 minutes says zuul) | 00:15 |
| ianw | oh doh, sorry, missed that | 00:15 |
| ianw | oh it was me anyway, i missed it on the status page too | 00:16 |
| *** mriedem has joined #openstack-infra | 00:17 | |
| *** eernst has joined #openstack-infra | 00:17 | |
| *** edmondsw has quit IRC | 00:17 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add graphite01.opendev.org https://review.openstack.org/639466 | 00:18 |
| clarkb | I've just now rechecked it | 00:19 |
| *** eernst has quit IRC | 00:19 | |
| ianw | cool, the graphite one's ontop just to avoid confusion | 00:22 |
| *** paladox has quit IRC | 00:23 | |
| ianw | if someone could +2 the dns change https://review.openstack.org/#/c/639469/ so that can be ready, that would be great | 00:23 |
| ianw | i'm pretty hopeful puppet will just work and we can just schedule a minor downtime to cutover the volumes | 00:24 |
| fungi | looking | 00:24 |
| *** agopi has joined #openstack-infra | 00:25 | |
| openstackgerrit | Merged openstack-infra/zone-opendev.org master: Add graphite01.opendev.org, take 2 https://review.openstack.org/639469 | 00:25 |
| *** nsmeds has quit IRC | 00:27 | |
| *** hwoarang has quit IRC | 00:27 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP failing test case for multiple provides/requires https://review.openstack.org/639472 | 00:28 |
| *** hwoarang has joined #openstack-infra | 00:29 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-openstack_health master: Fully root pip path https://review.openstack.org/639442 | 00:30 |
| *** sdake has joined #openstack-infra | 00:33 | |
| guilhermesp | corvus: sorry not yet. We are facing some issues with neutron-dhcp agent. Me and mnaser are working on it and I'll let you know asap! | 00:36 |
| mnaser | duplicate ips/macs in dhcp unfortunate | 00:36 |
| clarkb | oh hey we've seen that in other clouds | 00:37 |
| clarkb | Does make me wonder if it is rooted in a neutron flaw | 00:37 |
| mnaser | we're digging.. | 00:38 |
| mnaser | i havent seen this happen in other regions oddly enough | 00:38 |
| clarkb | I'm going to pop out for a bit while I'm waiting for my inventory change ot merge | 00:39 |
| clarkb | I'll pop back in to make sure puppet applies on the new host as expected then probably flip over dns tomorrow moring | 00:39 |
| *** markvoelker has quit IRC | 00:40 | |
| *** wolverineav has quit IRC | 00:42 | |
| *** jamesmcarthur has joined #openstack-infra | 00:47 | |
| *** mriedem has quit IRC | 00:47 | |
| *** paladox_ has joined #openstack-infra | 00:49 | |
| *** rascasoft has joined #openstack-infra | 00:51 | |
| *** paladox_ is now known as paladox | 00:52 | |
| gmann | clarkb: corvus I do not think any scope wise restriction of going beyond openstack but we have less resource/maintainer issue for o-h. only masayuki as of now and mtreinish for few update only. | 00:57 |
| *** sdake has quit IRC | 00:59 | |
| *** jamesmcarthur has quit IRC | 01:01 | |
| openstackgerrit | Merged openstack-infra/system-config master: Apply package updates before we reboot in launch-node https://review.openstack.org/639454 | 01:01 |
| *** jamesmcarthur has joined #openstack-infra | 01:02 | |
| *** rascasoft has quit IRC | 01:03 | |
| *** Swami has quit IRC | 01:08 | |
| *** yamamoto has joined #openstack-infra | 01:09 | |
| *** hwoarang has quit IRC | 01:11 | |
| *** hwoarang has joined #openstack-infra | 01:13 | |
| *** yamamoto has quit IRC | 01:13 | |
| *** nsmeds has joined #openstack-infra | 01:14 | |
| *** slaweq has joined #openstack-infra | 01:26 | |
| *** slaweq has quit IRC | 01:31 | |
| *** sdake has joined #openstack-infra | 01:34 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Add python-path option to label https://review.openstack.org/637338 | 01:36 |
| *** markvoelker has joined #openstack-infra | 01:37 | |
| *** takamatsu_ has quit IRC | 01:39 | |
| *** edmondsw has joined #openstack-infra | 01:43 | |
| *** takamatsu_ has joined #openstack-infra | 01:45 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Add python-path option to label https://review.openstack.org/637338 | 01:55 |
| openstackgerrit | Merged openstack-infra/system-config master: Add health01.openstack.org server https://review.openstack.org/639464 | 02:00 |
| *** whoami-rajat has joined #openstack-infra | 02:01 | |
| *** sdake has quit IRC | 02:09 | |
| *** ijw has quit IRC | 02:09 | |
| *** markvoelker has quit IRC | 02:10 | |
| *** jesusaur has quit IRC | 02:11 | |
| *** sdake has joined #openstack-infra | 02:11 | |
| *** sdake has quit IRC | 02:13 | |
| *** gyee has quit IRC | 02:15 | |
| *** sdake has joined #openstack-infra | 02:15 | |
| *** jesusaur has joined #openstack-infra | 02:15 | |
| *** sdake has quit IRC | 02:16 | |
| *** jamesmcarthur has quit IRC | 02:18 | |
| *** jamesmcarthur has joined #openstack-infra | 02:20 | |
| *** jamesmcarthur has quit IRC | 02:25 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Add python-path option to label https://review.openstack.org/637338 | 02:29 |
| *** ijw has joined #openstack-infra | 02:29 | |
| *** rcernin has quit IRC | 02:32 | |
| *** hongbin has joined #openstack-infra | 02:38 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:38 | |
| *** sdake has joined #openstack-infra | 02:40 | |
| *** hongbin has quit IRC | 02:43 | |
| *** sdake has quit IRC | 02:45 | |
| *** sdake has joined #openstack-infra | 02:48 | |
| *** rascasoft has joined #openstack-infra | 02:53 | |
| *** takamatsu_ has quit IRC | 02:53 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Add API endpoint to get frozen jobs https://review.openstack.org/607077 | 02:54 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Get executor job params https://review.openstack.org/607078 | 02:54 |
| *** jamesmcarthur has joined #openstack-infra | 02:55 | |
| *** sdake has quit IRC | 02:55 | |
| *** sdake has joined #openstack-infra | 02:56 | |
| *** yamamoto has joined #openstack-infra | 02:57 | |
| *** psachin has joined #openstack-infra | 03:00 | |
| *** sdake has quit IRC | 03:01 | |
| *** jamesmcarthur has quit IRC | 03:01 | |
| *** rascasoft has quit IRC | 03:01 | |
| *** sdake_ has joined #openstack-infra | 03:02 | |
| *** yamamoto has quit IRC | 03:02 | |
| *** markvoelker has joined #openstack-infra | 03:04 | |
| *** sdake_ has quit IRC | 03:10 | |
| *** sdake has joined #openstack-infra | 03:10 | |
| *** hwoarang has quit IRC | 03:13 | |
| *** apetrich has quit IRC | 03:15 | |
| *** sdake has quit IRC | 03:15 | |
| *** yamamoto has joined #openstack-infra | 03:15 | |
| *** sdake has joined #openstack-infra | 03:15 | |
| *** hwoarang has joined #openstack-infra | 03:17 | |
| *** wolverineav has joined #openstack-infra | 03:18 | |
| *** sdake has quit IRC | 03:23 | |
| *** sdake_ has joined #openstack-infra | 03:23 | |
| *** yamamoto has quit IRC | 03:25 | |
| *** yamamoto has joined #openstack-infra | 03:25 | |
| *** sdake_ has quit IRC | 03:25 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /connections route https://review.openstack.org/631703 | 03:27 |
| *** sdake has joined #openstack-infra | 03:28 | |
| *** sdake has quit IRC | 03:30 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Add API endpoint to get frozen jobs https://review.openstack.org/607077 | 03:31 |
| *** sdake has joined #openstack-infra | 03:32 | |
| *** sdake has quit IRC | 03:35 | |
| *** sdake has joined #openstack-infra | 03:36 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Add python-path option to node https://review.openstack.org/637338 | 03:44 |
| *** sdake has quit IRC | 03:44 | |
| *** ijw has quit IRC | 03:44 | |
| *** sdake_ has joined #openstack-infra | 03:45 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Get executor job params https://review.openstack.org/607078 | 03:48 |
| *** sdake_ has quit IRC | 03:51 | |
| *** sdake has joined #openstack-infra | 03:52 | |
| *** wolverineav has quit IRC | 03:55 | |
| *** sdake has quit IRC | 03:55 | |
| *** hwoarang has quit IRC | 03:55 | |
| *** ramishra has joined #openstack-infra | 03:57 | |
| *** sdake has joined #openstack-infra | 03:58 | |
| *** sdake has quit IRC | 04:00 | |
| *** hwoarang has joined #openstack-infra | 04:00 | |
| *** sdake has joined #openstack-infra | 04:00 | |
| *** sdake has quit IRC | 04:03 | |
| *** rascasoft has joined #openstack-infra | 04:06 | |
| *** sdake has joined #openstack-infra | 04:06 | |
| *** hwoarang has quit IRC | 04:09 | |
| *** sdake has quit IRC | 04:11 | |
| *** hwoarang has joined #openstack-infra | 04:11 | |
| *** sdake_ has joined #openstack-infra | 04:12 | |
| *** spsurya has joined #openstack-infra | 04:15 | |
| *** rascasoft has quit IRC | 04:16 | |
| *** sdake_ has quit IRC | 04:17 | |
| *** armax has joined #openstack-infra | 04:17 | |
| *** ykarel has joined #openstack-infra | 04:19 | |
| *** janki has joined #openstack-infra | 04:33 | |
| *** hwoarang has quit IRC | 04:34 | |
| *** hwoarang has joined #openstack-infra | 04:36 | |
| *** raukadah is now known as chandankumar | 04:40 | |
| *** jamesmcarthur has joined #openstack-infra | 04:46 | |
| *** wolverineav has joined #openstack-infra | 05:01 | |
| *** wolverineav has quit IRC | 05:02 | |
| *** dave-mccowan has quit IRC | 05:05 | |
| *** hwoarang has quit IRC | 05:05 | |
| *** hwoarang has joined #openstack-infra | 05:07 | |
| *** jamesmcarthur has quit IRC | 05:19 | |
| *** ricolin has joined #openstack-infra | 05:21 | |
| *** jesusaur has quit IRC | 05:30 | |
| *** ricolin has quit IRC | 05:35 | |
| *** jesusaur has joined #openstack-infra | 05:36 | |
| *** rlandy has quit IRC | 05:37 | |
| *** udesale has joined #openstack-infra | 05:42 | |
| *** ramishra has quit IRC | 05:43 | |
| *** ramishra has joined #openstack-infra | 05:45 | |
| *** ykarel has quit IRC | 05:54 | |
| *** ykarel has joined #openstack-infra | 05:55 | |
| *** ramishra has quit IRC | 05:55 | |
| *** ramishra has joined #openstack-infra | 06:03 | |
| *** hurgleburgler has quit IRC | 06:17 | |
| *** gregoryo has joined #openstack-infra | 06:17 | |
| *** wolverineav has joined #openstack-infra | 06:21 | |
| *** hwoarang has quit IRC | 06:34 | |
| *** hwoarang has joined #openstack-infra | 06:35 | |
| *** roman_g has quit IRC | 06:37 | |
| *** wolverineav has quit IRC | 06:38 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Add python-path option to node https://review.openstack.org/637338 | 06:38 |
| *** wolverineav has joined #openstack-infra | 06:39 | |
| *** jamesmcarthur has joined #openstack-infra | 06:49 | |
| *** rascasoft has joined #openstack-infra | 06:50 | |
| *** markvoelker has quit IRC | 06:51 | |
| *** quiquell|off is now known as quiquell | 06:52 | |
| *** jtomasek has joined #openstack-infra | 06:53 | |
| *** wolverineav has quit IRC | 06:54 | |
| *** jamesmcarthur has quit IRC | 06:54 | |
| *** slaweq has joined #openstack-infra | 06:57 | |
| *** ramishra has quit IRC | 06:58 | |
| *** rascasoft has quit IRC | 06:59 | |
| *** ramishra has joined #openstack-infra | 07:01 | |
| *** e0ne has joined #openstack-infra | 07:09 | |
| *** rascasoft has joined #openstack-infra | 07:10 | |
| *** aojea has joined #openstack-infra | 07:16 | |
| *** e0ne has quit IRC | 07:16 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Add rsd-virt-for-nova project https://review.openstack.org/638633 | 07:18 |
| *** takamatsu_ has joined #openstack-infra | 07:21 | |
| *** apetrich has joined #openstack-infra | 07:21 | |
| *** dpawlik has joined #openstack-infra | 07:25 | |
| *** yamamoto has quit IRC | 07:25 | |
| *** ccamacho has joined #openstack-infra | 07:29 | |
| *** pgaxatte has joined #openstack-infra | 07:37 | |
| *** wolverineav has joined #openstack-infra | 07:44 | |
| *** armax has quit IRC | 07:48 | |
| *** bhavikdbavishi has quit IRC | 07:49 | |
| *** jamesmcarthur has joined #openstack-infra | 07:51 | |
| *** markvoelker has joined #openstack-infra | 07:52 | |
| *** jamesmcarthur has quit IRC | 07:55 | |
| *** ccamacho has quit IRC | 07:56 | |
| *** e0ne has joined #openstack-infra | 07:57 | |
| *** yamamoto has joined #openstack-infra | 07:59 | |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/project-config master: Add openstack-tox-py37 job to Neutron dashboard https://review.openstack.org/639588 | 07:59 |
| *** zhangfei has joined #openstack-infra | 08:01 | |
| *** kopecmartin|off is now known as kopecmartin | 08:05 | |
| *** ccamacho has joined #openstack-infra | 08:07 | |
| *** roman_g has joined #openstack-infra | 08:12 | |
| *** yboaron_ has joined #openstack-infra | 08:13 | |
| *** kjackal has joined #openstack-infra | 08:13 | |
| *** pcaruana has joined #openstack-infra | 08:13 | |
| *** bhavikdbavishi has joined #openstack-infra | 08:14 | |
| *** yboaron_ has quit IRC | 08:18 | |
| *** wolverineav has quit IRC | 08:18 | |
| *** ginopc has joined #openstack-infra | 08:20 | |
| *** zbr|ssbarnea has joined #openstack-infra | 08:20 | |
| *** zbr|ssbarnea is now known as zbr | 08:22 | |
| *** yamamoto has quit IRC | 08:23 | |
| *** markvoelker has quit IRC | 08:25 | |
| *** pcaruana has quit IRC | 08:28 | |
| *** jesusaur has quit IRC | 08:30 | |
| *** takamatsu_ has quit IRC | 08:31 | |
| *** tkajinam has quit IRC | 08:33 | |
| *** jesusaur has joined #openstack-infra | 08:36 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: doc: add provider driver development instructions https://review.openstack.org/639595 | 08:37 |
| *** helenafm has joined #openstack-infra | 08:39 | |
| *** tosky has joined #openstack-infra | 08:41 | |
| *** pcaruana has joined #openstack-infra | 08:42 | |
| *** wolverineav has joined #openstack-infra | 08:51 | |
| *** gregoryo has quit IRC | 08:51 | |
| *** pcaruana has quit IRC | 08:51 | |
| *** jamesmcarthur has joined #openstack-infra | 08:51 | |
| *** jamesmcarthur has quit IRC | 08:56 | |
| *** jpena|off is now known as jpena | 08:57 | |
| *** e0ne has quit IRC | 08:57 | |
| *** pcaruana has joined #openstack-infra | 08:58 | |
| *** pcaruana|afk| has joined #openstack-infra | 09:01 | |
| *** pcaruana has quit IRC | 09:03 | |
| *** jpich has joined #openstack-infra | 09:04 | |
| *** takamatsu has joined #openstack-infra | 09:15 | |
| *** markvoelker has joined #openstack-infra | 09:23 | |
| *** wolverineav has quit IRC | 09:23 | |
| *** shachar has quit IRC | 09:28 | |
| *** janki has quit IRC | 09:30 | |
| *** janki has joined #openstack-infra | 09:30 | |
| *** jesusaur has quit IRC | 09:31 | |
| *** jesusaur has joined #openstack-infra | 09:40 | |
| *** derekh has joined #openstack-infra | 09:41 | |
| *** gtema has joined #openstack-infra | 09:42 | |
| *** tojuvone has joined #openstack-infra | 09:43 | |
| *** takamatsu has quit IRC | 09:44 | |
| *** ykarel is now known as ykarel|lunch | 09:44 | |
| tojuvone | Hi, I am making DevStack multinode job where I upgrade from Queens to Rocky | 09:45 |
| tojuvone | I have modifications to to UPGRADE flag | 09:45 |
| tojuvone | should I continue this, or look into Grenade? | 09:45 |
| *** jesusaur has quit IRC | 09:46 | |
| tojuvone | This work aims to make rolling upgrade job to Fenix project | 09:46 |
| *** zhangfei has quit IRC | 09:46 | |
| tojuvone | and to allow any developer to test changes over upgrade (with DevStack) | 09:47 |
| AJaeger | tojuvone: better ask on #openstack-qa for DevSTack and Grenade | 09:48 |
| tojuvone | AJaeger, thanks :) | 09:49 |
| gtema | am I the only one who have performance problems with git? last days it takes quite long for `git remote update`, `git review`. Pushing new change took around 4 min | 09:50 |
| *** jesusaur has joined #openstack-infra | 09:50 | |
| openstackgerrit | Gorka Eguileor proposed openstack-infra/devstack-gate master: Include openstack/cinderlib in PROJECTS https://review.openstack.org/639614 | 09:51 |
| *** jamesmcarthur has joined #openstack-infra | 09:52 | |
| gtema | `time git remote update origin` => 2m17. My remaining internet connection is ok | 09:54 |
| *** markvoelker has quit IRC | 09:57 | |
| *** jamesmcarthur has quit IRC | 09:57 | |
| *** owalsh_ has joined #openstack-infra | 09:57 | |
| *** owalsh has quit IRC | 09:58 | |
| *** zhangfei has joined #openstack-infra | 09:58 | |
| *** kjackal has quit IRC | 10:01 | |
| *** sdake has joined #openstack-infra | 10:03 | |
| *** kjackal has joined #openstack-infra | 10:04 | |
| *** sdake has quit IRC | 10:05 | |
| *** priteau has joined #openstack-infra | 10:06 | |
| *** kjackal has quit IRC | 10:07 | |
| *** snapiri has joined #openstack-infra | 10:07 | |
| *** kjackal has joined #openstack-infra | 10:07 | |
| *** sdake has joined #openstack-infra | 10:08 | |
| *** owalsh has joined #openstack-infra | 10:09 | |
| *** yamamoto has joined #openstack-infra | 10:09 | |
| *** owalsh_ has quit IRC | 10:10 | |
| *** sdake has quit IRC | 10:10 | |
| *** yamamoto has quit IRC | 10:14 | |
| *** sdake has joined #openstack-infra | 10:16 | |
| *** yamamoto has joined #openstack-infra | 10:16 | |
| *** sdake has quit IRC | 10:17 | |
| *** tojuvone has left #openstack-infra | 10:18 | |
| *** dpawlik has quit IRC | 10:20 | |
| *** wolverineav has joined #openstack-infra | 10:20 | |
| *** sdake has joined #openstack-infra | 10:22 | |
| *** panda|ruck|off is now known as panda | 10:23 | |
| *** panda is now known as panda|ruck | 10:23 | |
| *** ykarel|lunch is now known as ykarel | 10:23 | |
| *** sdake has quit IRC | 10:25 | |
| *** sdake has joined #openstack-infra | 10:26 | |
| *** sdake has quit IRC | 10:30 | |
| *** sdake has joined #openstack-infra | 10:31 | |
| *** dpawlik has joined #openstack-infra | 10:35 | |
| *** sdake has quit IRC | 10:35 | |
| *** sdake has joined #openstack-infra | 10:35 | |
| *** priteau has quit IRC | 10:40 | |
| *** sdake has quit IRC | 10:41 | |
| *** sdake_ has joined #openstack-infra | 10:41 | |
| *** priteau has joined #openstack-infra | 10:44 | |
| *** sdake_ has quit IRC | 10:45 | |
| *** bhavikdbavishi has quit IRC | 10:47 | |
| *** sdake has joined #openstack-infra | 10:47 | |
| *** gtema has quit IRC | 10:49 | |
| *** sdake has quit IRC | 10:50 | |
| *** sdake has joined #openstack-infra | 10:51 | |
| *** markvoelker has joined #openstack-infra | 10:53 | |
| *** udesale has quit IRC | 10:53 | |
| *** wolverineav has quit IRC | 10:54 | |
| *** sdake has quit IRC | 10:55 | |
| *** sdake has joined #openstack-infra | 10:56 | |
| *** yamamoto has quit IRC | 10:58 | |
| *** sdake has quit IRC | 11:00 | |
| *** yamamoto has joined #openstack-infra | 11:00 | |
| *** sdake has joined #openstack-infra | 11:01 | |
| *** dpawlik has quit IRC | 11:04 | |
| *** e0ne has joined #openstack-infra | 11:16 | |
| *** gtema has joined #openstack-infra | 11:19 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: wip: add a standalone launcher object https://review.openstack.org/639632 | 11:21 |
| *** takamatsu has joined #openstack-infra | 11:24 | |
| *** markvoelker has quit IRC | 11:26 | |
| *** dpawlik has joined #openstack-infra | 11:35 | |
| *** kjackal has quit IRC | 11:39 | |
| *** dpawlik has quit IRC | 11:39 | |
| *** dpawlik has joined #openstack-infra | 11:41 | |
| *** sshnaidm|afk is now known as sshnaidm | 11:49 | |
| *** wolverineav has joined #openstack-infra | 11:51 | |
| *** jamesmcarthur has joined #openstack-infra | 11:54 | |
| *** bhavikdbavishi has joined #openstack-infra | 11:58 | |
| *** jamesmcarthur has quit IRC | 11:59 | |
| *** ramishra has quit IRC | 12:04 | |
| *** markvoelker has joined #openstack-infra | 12:22 | |
| *** wolverineav has quit IRC | 12:24 | |
| *** bhavikdbavishi has quit IRC | 12:25 | |
| *** bhavikdbavishi has joined #openstack-infra | 12:26 | |
| *** ramishra has joined #openstack-infra | 12:26 | |
| *** udesale has joined #openstack-infra | 12:26 | |
| *** gtema has quit IRC | 12:27 | |
| *** kjackal has joined #openstack-infra | 12:31 | |
| openstackgerrit | Elod Illes proposed openstack-infra/zuul-jobs master: Explanatory comment for Editable Git install https://review.openstack.org/639649 | 12:33 |
| *** luizbag has joined #openstack-infra | 12:34 | |
| *** jpena is now known as jpena|lunch | 12:36 | |
| *** bhavikdbavishi has quit IRC | 12:37 | |
| *** bhavikdbavishi has joined #openstack-infra | 12:45 | |
| *** kjackal has quit IRC | 12:50 | |
| *** jamesmcarthur has joined #openstack-infra | 12:55 | |
| *** markvoelker has quit IRC | 12:56 | |
| *** agopi has quit IRC | 12:59 | |
| *** jamesmcarthur has quit IRC | 12:59 | |
| *** yamamoto has quit IRC | 13:00 | |
| *** dave-mccowan has joined #openstack-infra | 13:02 | |
| *** ekultails has joined #openstack-infra | 13:08 | |
| *** pcaruana|afk| has quit IRC | 13:09 | |
| *** bhavikdbavishi has quit IRC | 13:11 | |
| *** sdake has quit IRC | 13:16 | |
| evrardjp | hey | 13:16 |
| evrardjp | I would like to debug the promote pipeline failure, which failed on https://github.com/openstack-infra/zuul-jobs/blob/master/roles/promote-docker-image/tasks/main.yaml#L9-L17 (the upload of the image on merge succeeded) , does someone have a few minutes to check that with me? | 13:17 |
| evrardjp | (promote failure is http://logs.openstack.org/34/639334/2/promote/openstack-helm-images-promote-ceph-config-helper/7d49a84/ara-report/ ) | 13:18 |
| evrardjp | (this pr converts existing things into the new pipeline) | 13:18 |
| *** priteau has quit IRC | 13:19 | |
| *** kgiusti has joined #openstack-infra | 13:20 | |
| evrardjp | it seems I can't speak but I hope you got the gist, and are happy for a pair debugging if someone is available | 13:20 |
| *** wolverineav has joined #openstack-infra | 13:21 | |
| *** derekh has quit IRC | 13:22 | |
| *** zhangfei has quit IRC | 13:22 | |
| *** quiquell is now known as quiquell|lunch | 13:23 | |
| openstackgerrit | Gorka Eguileor proposed openstack-infra/devstack-gate master: Include openstack/cinderlib in PROJECTS https://review.openstack.org/639614 | 13:23 |
| pabelanger | evrardjp: I believe corvus might be best help there, hard to say without logs what happend. But maybe was a blip in the network since we are using uri module. | 13:31 |
| openstackgerrit | Hervé Beraud proposed openstack-dev/pbr master: Fix error when keywords are defined as a list in cfg https://review.openstack.org/639661 | 13:31 |
| pabelanger | maybe we need to add some retries to the task | 13:31 |
| *** gtema has joined #openstack-infra | 13:33 | |
| evrardjp | pabelanger: yup I thought of that. Or alternatively, because our username is also encrypted, it might be a failure to decrypt -- but that sounds weird. | 13:35 |
| *** agopi has joined #openstack-infra | 13:35 | |
| evrardjp | or it's just a misconfiguration, which I believe is the more likely of the 3. | 13:35 |
| *** yamamoto has joined #openstack-infra | 13:36 | |
| pabelanger | evrardjp: one thing we have seen in past is encryption failing because of whitespacing in secrets, make sure to use the --strip flag on the zuul encrypt_secrets tool | 13:38 |
| *** rlandy has joined #openstack-infra | 13:39 | |
| *** jpena|lunch is now known as jpena | 13:40 | |
| *** agopi has quit IRC | 13:40 | |
| *** yamamoto has quit IRC | 13:41 | |
| slaweq | hi frickler, clarkb: can You maybe set to hold nodes where job neutron-tempest-dvr-ha-multinode-full from patch https://review.openstack.org/633979 will be run? | 13:41 |
| *** yamamoto has joined #openstack-infra | 13:42 | |
| slaweq | frickler: clarkb: I'm still trying to make it working with zuulv3 and it looks that something is wrong with connectivity to vms when I set l3_ha in this job | 13:42 |
| slaweq | frickler: clarkb I would like to log to those nodes and check what is going wrong there | 13:42 |
| slaweq | thx in advance | 13:42 |
| evrardjp | pabelanger: I haven't generated it myself. But I expect the upload itself would have failed and not the promote if the secret was wrong | 13:44 |
| *** mriedem has joined #openstack-infra | 13:44 | |
| *** jamesmcarthur has joined #openstack-infra | 13:48 | |
| openstackgerrit | Hervé Beraud proposed openstack-dev/pbr master: Fix error when keywords are defined as a list in cfg https://review.openstack.org/639661 | 13:48 |
| *** e0ne has quit IRC | 13:50 | |
| *** markvoelker has joined #openstack-infra | 13:53 | |
| frickler | slaweq: looking | 13:53 |
| slaweq | frickler: thx a lot | 13:53 |
| *** janki has quit IRC | 13:53 | |
| pabelanger | evrardjp: ah, good point. I'm not familiar enough with those playbooks yet, I assumed each project had their own credentials | 13:54 |
| *** wolverineav has quit IRC | 13:55 | |
| frickler | slaweq: ok, autohold set, should catch the run from your current patchset | 13:56 |
| slaweq | frickler: thx a lot once again | 13:57 |
| *** quiquell|lunch is now known as quiquell | 13:57 | |
| *** derekh has joined #openstack-infra | 13:57 | |
| fungi | gtema: it's interesting that both your git remote update origin and git review invocations are slow. those should be using completely different remotes on different servers. what repository is it? | 13:59 |
| *** yamamoto has quit IRC | 14:00 | |
| gtema | fungi: openstacksdk | 14:02 |
| *** yamamoto has joined #openstack-infra | 14:03 | |
| *** yamamoto has quit IRC | 14:03 | |
| fungi | do you know if you're reaching git.openstack.org and review.openstack.org over ipv4 or ipv6? | 14:04 |
| *** eharney has joined #openstack-infra | 14:04 | |
| *** e0ne has joined #openstack-infra | 14:04 | |
| gtema | hmm, I would really guess v4. Can I somehow double-check that? | 14:04 |
| *** jcoufal has joined #openstack-infra | 14:05 | |
| gtema | fungi: however my router has both v4 and v6, so can't say for sure | 14:06 |
| gtema | fungi: ok, most likely v6, since ping tries v6 now (what doesn't work) | 14:08 |
| fungi | if you're able to reach the ipv6 addresses of those, then your git client is probably choosing v6. can you `ping6 git.openstack.org` or maybe `ping -6 git.openstack.org` (might depend on your ping implementation) | 14:08 |
| *** agopi has joined #openstack-infra | 14:08 | |
| fungi | ahh, yeah, so because we make ipv6 available on all our systems it's *possible* the reason you're seeing performance issues is something is degrading your ipv6 connectivity but whatever else you're doing on the internet is predominately v4 | 14:08 |
| gtema | oh god, do we really have this issue now here? ;-/ | 14:09 |
| fungi | for the record, i have a very stale openstacksdk clone i just did a remote update in and it completed in ~5 seconds over ipv6 | 14:10 |
| gtema | yeah, seems something to be wrong in my net with v6 connectivity, since ping -6 google.com is not working as well | 14:10 |
| fungi | so i doubt any network performance issues in the vicinity of the server, probably something closer to the client | 14:11 |
| *** sdake has joined #openstack-infra | 14:12 | |
| *** sdake has quit IRC | 14:12 | |
| gtema | fungi: seem so. After disabling local v6 (v4 to the router) I start at least getting responses, that address unreachable (from the further provider point) | 14:13 |
| gtema | fungi, thanks for hint | 14:13 |
| *** cmoura has joined #openstack-infra | 14:13 | |
| *** jamesmcarthur has quit IRC | 14:14 | |
| fungi | no problem, glad we at least figured out the underlying issue | 14:14 |
| fungi | and that it isn't something *i'm* going to need to spend half my day trying to fix ;) | 14:15 |
| *** priteau has joined #openstack-infra | 14:16 | |
| *** sdake has joined #openstack-infra | 14:18 | |
| *** sdake has quit IRC | 14:19 | |
| *** eharney_ has joined #openstack-infra | 14:19 | |
| *** ykarel is now known as ykarel|afk | 14:20 | |
| *** eharney has quit IRC | 14:22 | |
| *** markvoelker has quit IRC | 14:25 | |
| *** psachin has quit IRC | 14:27 | |
| *** yamamoto has joined #openstack-infra | 14:38 | |
| frickler | slaweq: tests are still running, but there's a failure in tempest already. nodes have your key now: 38.108.68.77, 38.108.68.115 and 38.108.68.31 | 14:42 |
| slaweq | frickler: thx a lot | 14:43 |
| slaweq | I'm not sure if I will be able to test them just now but I will do it today for sure | 14:43 |
| *** Adri2000 has quit IRC | 14:43 | |
| slaweq | will ping You guys here when I will finish | 14:43 |
| *** yamamoto has quit IRC | 14:43 | |
| *** sdake has joined #openstack-infra | 14:49 | |
| *** wolverineav has joined #openstack-infra | 14:52 | |
| *** sdake has quit IRC | 14:55 | |
| *** bhavikdbavishi has joined #openstack-infra | 14:55 | |
| *** pcaruana has joined #openstack-infra | 14:57 | |
| *** jpena is now known as jpena|brb | 15:02 | |
| *** sdake has joined #openstack-infra | 15:02 | |
| *** jamesmcarthur has joined #openstack-infra | 15:03 | |
| *** dpawlik has quit IRC | 15:03 | |
| *** yamamoto has joined #openstack-infra | 15:05 | |
| *** ykarel|afk is now known as ykarel | 15:06 | |
| *** bogdando has joined #openstack-infra | 15:06 | |
| bogdando | hi folks. I'm looking for greates of zuul gurus | 15:06 |
| bogdando | https://review.openstack.org/#/c/568543 , for some reason this job seems not included for that graph freezer?.. any ideas how to make it included w/o duplicating its definition https://github.com/openstack-infra/openstack-zuul-jobs/blob/165eaf6cb630a53edfc07307318215061ae088db/zuul.d/jobs.yaml#L467 here?.. | 15:06 |
| bogdando | greatest** | 15:06 |
| *** yamamoto has quit IRC | 15:09 | |
| *** jcoufal has quit IRC | 15:09 | |
| *** anteaya has joined #openstack-infra | 15:11 | |
| *** jcoufal has joined #openstack-infra | 15:13 | |
| *** eharney_ is now known as eharney | 15:16 | |
| openstackgerrit | Hervé Beraud proposed openstack-dev/pbr master: Fix error when keywords are defined as a list in cfg https://review.openstack.org/639661 | 15:21 |
| *** markvoelker has joined #openstack-infra | 15:22 | |
| *** wolverineav has quit IRC | 15:24 | |
| *** sdake has quit IRC | 15:33 | |
| *** ccamacho has quit IRC | 15:35 | |
| *** priteau has quit IRC | 15:35 | |
| *** ccamacho has joined #openstack-infra | 15:35 | |
| *** sdake has joined #openstack-infra | 15:35 | |
| corvus | bogdando: please don't do what you're trying to do. | 15:37 |
| openstackgerrit | Hervé Beraud proposed openstack-dev/pbr master: Fix error when keywords are defined as a list in cfg https://review.openstack.org/639661 | 15:39 |
| *** ccamacho has quit IRC | 15:41 | |
| *** ccamacho has joined #openstack-infra | 15:41 | |
| *** dmellado has quit IRC | 15:42 | |
| *** helenafm has quit IRC | 15:42 | |
| *** dmellado has joined #openstack-infra | 15:43 | |
| bogdando | corvus: only proposing for review. tripleo infra and dev teams will decide how to proceed further | 15:44 |
| bogdando | if there is infra team to get in the loop, please join the review folks | 15:45 |
| *** jpena|brb is now known as jpena | 15:45 | |
| openstackgerrit | Camila Moura proposed openstack-infra/storyboard master: Fixes the dependencies for Python 3 https://review.openstack.org/639715 | 15:45 |
| openstackgerrit | Camila Moura proposed openstack-infra/storyboard master: Fixes LauchpadReader iterator https://review.openstack.org/639716 | 15:45 |
| *** quiquell is now known as quiquell|off | 15:46 | |
| corvus | bogdando: almost everyone from the infra team has responded to that mailing list thread explaining why, after looking very closely at the numbers, we think that this is not going to reduce resource usage and will only make the system less useful for developers. clarkb has produced statistics showing where real improvement can be made, effort would be better spent there. | 15:47 |
| corvus | bogdando: we're frankly tired of saying the same thing repeatedly, so please don't expect us to spend any more time doing so | 15:48 |
| evrardjp | corvus, could you ping me about the promote job failures for my new change when you have time? | 15:48 |
| bogdando | corvus: oh, it seems I missed the update from clarkb... | 15:48 |
| bogdando | thanks for letting me know | 15:48 |
| *** ccamacho has quit IRC | 15:48 | |
| corvus | evrardjp: yep, sorry, i'ave actually been working on it for the past 10 mins or so :) | 15:48 |
| *** ccamacho has joined #openstack-infra | 15:48 | |
| evrardjp | oh! | 15:48 |
| corvus | bogdando: no prob, it was a long thread :) | 15:48 |
| evrardjp | well sir, please teach me how to fish | 15:48 |
| evrardjp | because no_log wouldn't help, I ruled out a network issue, and I probably am lost in the wiring of the vars. | 15:49 |
| evrardjp | I took another job as reference, which _seem_ to work | 15:49 |
| corvus | evrardjp: i've gotten sucked into a small rabbit hole trying to figure out why there's no zuul-info/ directory in the logs for that job | 15:49 |
| corvus | it's probably not important or related to the failure | 15:50 |
| corvus | but it's usually how i start debugging things | 15:50 |
| *** armax has joined #openstack-infra | 15:50 | |
| evrardjp | I think promote is quite new, and I am happy to help. | 15:50 |
| *** ccamacho has quit IRC | 15:51 | |
| *** ccamacho has joined #openstack-infra | 15:51 | |
| evrardjp | (FYI I used http://git.openstack.org/cgit/openstack-infra/system-config/tree/.zuul.yaml as inspiration, so the dict structure should be relatively intact, and IIRC, those jobs passed recently) | 15:51 |
| *** ccamacho has quit IRC | 15:51 | |
| *** ccamacho has joined #openstack-infra | 15:52 | |
| corvus | evrardjp: yep, those and the jobs for zuul itself all have been working | 15:52 |
| evrardjp | last time is Jan16th | 15:52 |
| evrardjp | but I suppose it's fine | 15:52 |
| evrardjp | for jinja-init | 15:52 |
| evrardjp | anyway | 15:52 |
| evrardjp | green there! | 15:53 |
| *** bogdando has left #openstack-infra | 15:53 | |
| corvus | evrardjp: http://logs.openstack.org/54/638054/1/promote/zuul-promote-image/ee859d6/ is more recent | 15:53 |
| evrardjp | I guess it's impossible to change to no_log: false temporarily? | 15:53 |
| *** ccamacho has quit IRC | 15:54 | |
| *** ccamacho has joined #openstack-infra | 15:54 | |
| *** stevebaker has quit IRC | 15:54 | |
| evrardjp | oh interesting, and make sense to promote zuul's image on commit ... :D | 15:55 |
| *** markvoelker has quit IRC | 15:56 | |
| fungi | the no_log is presumably a safety measure to prevent us from exposing sensitive data and secrets in errors which could arise form the commands using them | 15:56 |
| fungi | so turning it off, even temporarily, seems potentially dangerous | 15:56 |
| *** stevebaker has joined #openstack-infra | 15:57 | |
| evrardjp | fungi: yes, I thought of a variable that defaults to false, and then we override it in my temp testing | 15:57 |
| evrardjp | but that sounds like a bad idea | 15:57 |
| evrardjp | if we can avoid it let's do it | 15:57 |
| evrardjp | so my expectations are that docker_credentials.username and password are fine for the gate job, as upload-docker-image works . But maybe the promote doesn't get the right docker_credentials, and some credentials are overriden? | 15:58 |
| *** iurygregory is now known as iurygregory_mtg | 15:59 | |
| clarkb | http://23.253.253.28/status looks like http://health.openstack.org/status so I'll be swapping dns over this morning | 15:59 |
| dhellmann | config-core: there's a SIG governance-related patch that would be good to land when you have a minute for the review. https://review.openstack.org/#/c/637125/ | 16:01 |
| corvus | evrardjp: afaict, the job config looks right | 16:02 |
| *** ykarel is now known as ykarel|away | 16:03 | |
| openstackgerrit | Merged openstack-infra/storyboard master: Update development installation documentation https://review.openstack.org/633354 | 16:04 |
| corvus | pabelanger: aha! the log-inventory role is under a "hosts: all" play -- is there any reason it shouldn't be under the localhost play? | 16:04 |
| *** ykarel|away is now known as ykarel | 16:04 | |
| *** ijw has joined #openstack-infra | 16:06 | |
| pabelanger | corvus: from base job? I am unsure, in ansible-network use do use it under only localhost | 16:07 |
| openstackgerrit | Camila Moura proposed openstack-infra/storyboard master: Fixes the dependencies for Python 3 https://review.openstack.org/639716 | 16:08 |
| *** takamatsu has quit IRC | 16:09 | |
| *** udesale has quit IRC | 16:10 | |
| corvus | evrardjp: can you run this ansible playbook locally with your dockerhub credentials? http://paste.openstack.org/show/746442/ | 16:10 |
| *** quiquell|off is now known as quiquell|r-- | 16:10 | |
| clarkb | infra-root https://review.openstack.org/#/c/639470/1 is a launch-node.py update to not use the ansible fact cache when launching new nodes (fixes an issue where we mistake xenial nodes as bionic and potentially other similar mixups) | 16:10 |
| evrardjp | sadly I cannot, as these are not my credentials, let me ask the ptl of that project | 16:10 |
| pabelanger | corvus: looking at the role, we could move it to localhost play, and then consider removing the delegated_to tasks from log-inventory role | 16:10 |
| *** chandankumar is now known as raukadah | 16:11 | |
| evrardjp | portdirect: could you run this: http://paste.openstack.org/show/746442/ ? | 16:12 |
| clarkb | fungi: I think https://review.openstack.org/#/c/639442/ confirms my suspicions about pip paths on test nodes vs production. That change fixes it, but is a bit complicated. I'm not sure if there is a better method as I don't think we can test paths from within pupept and set a variable value based on that (would have to be a fact instead which is more complicated than this change) | 16:12 |
| corvus | evrardjp, portdirect: (there's an extra quote on username -- that was a typo, sorry) | 16:12 |
| evrardjp | corvus portdirect I suppose there is a typo with the double " on the username of that paste, but you got the gist (pun intended) | 16:12 |
| evrardjp | corvus: :D | 16:13 |
| corvus | clarkb: why isn't pip in the path after installing pip? | 16:16 |
| evrardjp | corvus: please note said credentials work on gate pipeline which means the push works | 16:16 |
| clarkb | corvus: because when you run get-pip.py it removes the pip executable from the apt installed path. This leaves pip in /usr/local/bin/pip and not /usr/bin/pip on our test images | 16:16 |
| evrardjp | except if you expect the gate pipeline to not use the defined credentials, but infra's -- but that sounds weird | 16:16 |
| corvus | evrardjp: yeah, which is making me wonder if there's something about those credentials and the jwt token code path, which i don't think is used in the upload task | 16:17 |
| clarkb | infra-root I'm killing the gpg agent on bridge. I'll push up a change to add a script to run emacs for us with the right gpg wrapper after I get dns updated for health | 16:17 |
| portdirect | corvus / evrardjp : results below | 16:18 |
| portdirect | https://www.irccloud.com/pastebin/rUPRZH6P/ | 16:18 |
| corvus | portdirect: thanks! | 16:18 |
| evrardjp | that clarifies something :) | 16:19 |
| corvus | evrardjp, portdirect: that looked like it worked... which may mean we're just looking at a dockerhub network error? is that the only run of that job? | 16:19 |
| evrardjp | that PR has started many jobs | 16:20 |
| evrardjp | multiple promotes so I would expect not all of those to fail if that was a network issue | 16:20 |
| *** aojea has quit IRC | 16:20 | |
| corvus | evrardjp: though, they all ran at the same time, so it's not out of the question | 16:20 |
| evrardjp | corvus: this is the patch: https://review.openstack.org/#/c/639334/ | 16:20 |
| evrardjp | corvus: totally | 16:21 |
| evrardjp | should have some retries? | 16:21 |
| *** sreejithp has joined #openstack-infra | 16:21 | |
| corvus | it'd have to be more of an "outage" rather than a "blip", because it would be 10 requests over about 10 seconds, and most likely from different hosts. | 16:21 |
| corvus | but still possible | 16:22 |
| *** wolverineav has joined #openstack-infra | 16:22 | |
| evrardjp | totally possible indeed | 16:22 |
| evrardjp | but that sounded very unlikely to me | 16:23 |
| corvus | evrardjp: let's see what happens when another change merges... do you have one ready to approve, or can you merge a simple one? and yes, let's add retries as well, but i don't think we need to wait for that. | 16:23 |
| evrardjp | I sadly do not have anything yet, but I will work on a new change, and see how it goes. | 16:24 |
| corvus | evrardjp: (btw, the job should remove the unused change_ tags older than 24h on dockerhub, so those will get cleaned up eventually) | 16:24 |
| *** ramishra has quit IRC | 16:25 | |
| *** gtema has left #openstack-infra | 16:25 | |
| corvus | evrardjp: oh you know what, i could probably enqueue this manually | 16:25 |
| evrardjp | yeah, I am sure that would be okay, I don't see a problem in any case | 16:25 |
| evrardjp | corvus: can you? | 16:25 |
| evrardjp | that's cool | 16:25 |
| corvus | it will take a bit, because the inventory file is missing, but i should be able to construct the command | 16:25 |
| evrardjp | oh yeah, that's why you wanted to have that folder present :D | 16:25 |
| corvus | okay, it's enqueued | 16:26 |
| clarkb | the health.openstack.org A/AAAA records have been deleted and replaced with a CNAME to health01.openstack.org. The TTL on that was an hour so will be some time before I can fully verify that the health dashboard is happy with the switch, but I have done what I can to verify the api server is working | 16:27 |
| corvus | evrardjp: still failed, so that wasn't it. :| | 16:28 |
| evrardjp | corvus: but you have more logs? | 16:30 |
| corvus | evrardjp: no... i could turn on more verbosity, but i think no_log still defeats even that | 16:31 |
| evrardjp | yeah | 16:31 |
| evrardjp | can you override some variables? | 16:31 |
| corvus | evrardjp, portdirect: do you mind if i attempt to decrypt the zuul secret with your dockerhub creds? | 16:31 |
| *** dmellado has quit IRC | 16:32 | |
| portdirect | corvus: go for it | 16:32 |
| evrardjp | for example temporarily overriding the repo, to see if the credentials are infra ones or openstackhelm | 16:32 |
| portdirect | i can rotate when your done | 16:32 |
| *** quiquell|r-- is now known as quiquell|off | 16:32 | |
| *** yamamoto has joined #openstack-infra | 16:32 | |
| corvus | evrardjp: not easily -- i think the only way to do that would be to make a new role/playbook/job | 16:33 |
| corvus | fungi: i don't suppose you saved/documented that decryption command? | 16:34 |
| *** dmellado has joined #openstack-infra | 16:34 | |
| corvus | fungi: found it https://review.openstack.org/637969 | 16:34 |
| fungi | oh, has that not merged yet? thanks for trying it out! | 16:36 |
| *** yamamoto has quit IRC | 16:37 | |
| roman_g | Hello team. I would like to ask if there is an API for the <openstackgerrit> IRC bot, so that I can ask it to shoot a build-related message to the channel from third-party system (jenkins in my case). I could use just some other IRC account, but would prefer to re-use existing infra, if possible. | 16:38 |
| mordred | roman_g: there is not - it is driven only from gerrit itself. | 16:42 |
| *** ykarel_ has joined #openstack-infra | 16:43 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Add edit-secrets script to bridge.o.o https://review.openstack.org/639740 | 16:45 |
| clarkb | infra-root ^ that should install a wrapper script that will address our gpg agent woes if we use it | 16:45 |
| *** gouthamr_ has joined #openstack-infra | 16:46 | |
| *** ykarel has quit IRC | 16:46 | |
| prometheanfire | are we able to have a dashboard/view in storyboard that does not include openstories with tasks in my project that have been merged? ( https://storyboard.openstack.org/#!/story/1659641 ) | 16:47 |
| *** ijw has quit IRC | 16:47 | |
| roman_g | mordred: thank you | 16:47 |
| *** sreejithp has quit IRC | 16:48 | |
| prometheanfire | might just end up deleting my project from it if not | 16:48 |
| clarkb | prometheanfire: iirc no, but that feature has been requested. fungi SotK and diablo_rojo would be the best people to answer though | 16:48 |
| *** gouthamr has quit IRC | 16:48 | |
| prometheanfire | clarkb: yarp, iirc I was the one requesting :P | 16:48 |
| *** sreejithp has joined #openstack-infra | 16:49 | |
| *** gouthamr has joined #openstack-infra | 16:50 | |
| *** gouthamr_ has quit IRC | 16:50 | |
| *** ijw has joined #openstack-infra | 16:50 | |
| *** sdake has quit IRC | 16:52 | |
| *** markvoelker has joined #openstack-infra | 16:53 | |
| openstackgerrit | Merged openstack-infra/system-config master: Docs addition on decrypting Zuul secrets https://review.openstack.org/637969 | 16:53 |
| fungi | prometheanfire: i believe it's a matter of someone altering the query those project views use to join on the project being displayed. there is likely a story open for that request but current focus is on getting attachments working | 16:54 |
| *** sdake has joined #openstack-infra | 16:54 | |
| fungi | er, not join on the project being displayed but alter the task state query to filter on the project being displayed | 16:55 |
| prometheanfire | fungi: ok | 16:55 |
| *** wolverineav has quit IRC | 16:55 | |
| fungi | prometheanfire: an alternative is likely a task-based worklist which filters on projects+state | 16:56 |
| fungi | i can't remember if you've tried that solution yet | 16:56 |
| prometheanfire | I can see, think it didn't work at the time | 16:56 |
| *** pcaruana has quit IRC | 16:57 | |
| fungi | prometheanfire: i think it would be a task list not a story list, since the criteria you want to filter on is task states and projects rather than the story states (as those aren't project-specific) | 16:58 |
| *** gyee has joined #openstack-infra | 16:59 | |
| *** ykarel_ is now known as ykarel|away | 16:59 | |
| prometheanfire | what does automatic mean for 'new worklist'? | 16:59 |
| fungi | that's what you want, allows you to base it on queries | 17:00 |
| prometheanfire | hover tooltip :D | 17:00 |
| fungi | as opposed to "manual" where you select things to add to/remove from it manually | 17:00 |
| prometheanfire | yarp | 17:00 |
| *** rtjure has quit IRC | 17:00 | |
| *** e0ne has quit IRC | 17:00 | |
| *** ijw_ has joined #openstack-infra | 17:00 | |
| *** ijw has quit IRC | 17:01 | |
| *** mriedem is now known as mriedem_away | 17:01 | |
| fungi | automatic worklists (and board lanes) are basically query views | 17:01 |
| fungi | though the project-filtered story view feature does seem like a possibly good option for an intern | 17:02 |
| *** sdake has quit IRC | 17:02 | |
| fungi | diablo_rojo: SotK: ^ ? | 17:02 |
| fungi | not sure if that requires too deep of an understanding of the data model. guess it would depend on the intern | 17:03 |
| clarkb | the health dashboard is working for me locally and dig shows I am using the new dns records now. | 17:04 |
| prometheanfire | not sure I can filter on task state, I can filter on tasks themselves though (it looks like) | 17:04 |
| SotK | fungi: depends how urgently we want it. Looking into it a bit it's likely only a day's work for me but would probably take a while longer for an intern to get up to speed with how the data model around status works | 17:10 |
| *** ijw_ has quit IRC | 17:10 | |
| SotK | prometheanfire: when creating filters for a workout there should be a drop-down to swap between filters for Stories and filters for Tasks | 17:11 |
| corvus | evrardjp: i've decrypted the secrets, which looks like they're fine (i don't see any extra whitespace) and repeated the test i asked portdirect to do, and that seems to work as well. i'm trying to figure out what the next useful thing to do would be. i think we've run out of the easy things. | 17:11 |
| prometheanfire | SotK: there is, but I want to filter for task:openstack-requiermnts and task-state:!merged | 17:12 |
| prometheanfire | and not invalid | 17:12 |
| evrardjp | corvus: I am not surprised -- precedence issues and other vars? | 17:13 |
| *** jamesmcarthur_ has joined #openstack-infra | 17:14 | |
| SotK | Unless something has changed you should be able to express that using the matching/not matching drop-down and the "and" button | 17:16 |
| *** wolverineav has joined #openstack-infra | 17:16 | |
| SotK | Give me 15 minutes to get to a better internet connection and I'll make an example on storyboard-dev | 17:17 |
| *** dpawlik has joined #openstack-infra | 17:17 | |
| *** jamesmcarthur has quit IRC | 17:17 | |
| prometheanfire | thanks, working from https://storyboard.openstack.org/#!/worklist/567 for now | 17:17 |
| *** ijw has joined #openstack-infra | 17:18 | |
| fungi | prometheanfire: if you want them broken out by task state, you could do something similar with automatic lanes in a board | 17:19 |
| fungi | so have a todo lane, a review lane, and so on | 17:20 |
| prometheanfire | 500: POST /api/v1/worklists: | 17:21 |
| corvus | evrardjp: "unfortunately" we were rather thorough when we designed how the executor uses secrets. they are only written to a tmpfs that is only mounted in a container running the ansible process, so it's impossible to see exactly what value zuul gave to ansible | 17:21 |
| prometheanfire | creating worklists and lanes at the same time | 17:22 |
| corvus | evrardjp: i can tell zuul to keep the build directory around, which would allow us to alter and run the playbook again in the same environment, execpt that we'd have to supply the secrets ourselves | 17:22 |
| evrardjp | corvus: tapping into ansible strategies? | 17:22 |
| evrardjp | corvus: is that easy to do? That could help make hypothesis | 17:23 |
| corvus | that still might be worth doing, because it would either tell us there's something about the execution environment causing the issue, or there's something about zuul decrypting the secret. | 17:23 |
| prometheanfire | fungi: still don't see a way to filter based on task state | 17:23 |
| evrardjp | corvus: yeah | 17:23 |
| evrardjp | if it doesn't take too much of your time | 17:24 |
| *** dpawlik has quit IRC | 17:24 | |
| corvus | evrardjp: i'll tell all of the zuul executors to keep their build dirs, retrigger the jobs, turn off the option, then go try to find one :) | 17:24 |
| evrardjp | (sorry for that btw, I am feeling a little clueless...) | 17:24 |
| corvus | evrardjp: no worries, i wrote this, i expect to fix it :) | 17:24 |
| evrardjp | interesting. | 17:24 |
| evrardjp | hahah | 17:24 |
| evrardjp | fair point | 17:24 |
| *** ijw has quit IRC | 17:25 | |
| *** markvoelker has quit IRC | 17:25 | |
| *** trown is now known as trown|lunch | 17:28 | |
| *** jpich has quit IRC | 17:30 | |
| *** ijw has joined #openstack-infra | 17:31 | |
| *** dims has quit IRC | 17:35 | |
| *** takamatsu has joined #openstack-infra | 17:41 | |
| *** yamamoto has joined #openstack-infra | 17:41 | |
| *** wolverineav has quit IRC | 17:46 | |
| *** yamamoto has quit IRC | 17:46 | |
| *** efried is now known as efried_afk | 17:47 | |
| zbr | corvus: fungi : can i get some help merning https://review.openstack.org/#/c/636163/ on bindep? | 17:47 |
| *** dims has joined #openstack-infra | 17:48 | |
| zbr | background: pabelanger asked me not to use bindep-fallback to test bindep in other change, so i have do this before making the other change: https://review.openstack.org/#/c/635590/ | 17:48 |
| clarkb | re cffi and cryptography from yesterday, I've just found some jobs on my edit-secrets change failed because there is or was no cryptography wheel for the recent release today. There are currently manylinux wheels for python2.7 and 3.4 but not 3.5 or 3.6 | 17:49 |
| clarkb | I wonder if we can convince cffi/cryptography to upload the wheels before sdists | 17:50 |
| *** ykarel|away has quit IRC | 17:51 | |
| corvus | evrardjp: by manually running the commands, i promoted opensuse_15-latest. | 17:58 |
| corvus | evrardjp: first -- is it okay that's the only one which has been promoted? | 17:59 |
| evrardjp | mmm | 17:59 |
| evrardjp | I would need context on the job you used there | 18:00 |
| evrardjp | because there was a stack | 18:00 |
| evrardjp | heap of those I mean | 18:00 |
| corvus | evrardjp: actually, i think there's an error, and i think we might have prometed the ubuntu_xenial image to opensuse_15-latest | 18:00 |
| corvus | so let's start by undoing this | 18:01 |
| evrardjp | I would say with the latest change, it's most likely that two images should be promoted | 18:01 |
| *** ginopc has quit IRC | 18:01 | |
| evrardjp | yeah that sounds bad | 18:01 |
| evrardjp | corvus: you want me to resplit per distro? | 18:01 |
| evrardjp | resplit jobs per distro | 18:01 |
| *** takamatsu has quit IRC | 18:01 | |
| evrardjp | (and which job was it?) | 18:02 |
| corvus | evrardjp: ceph-config-helper | 18:02 |
| evrardjp | ok let me check | 18:04 |
| *** Vadmacs has joined #openstack-infra | 18:04 | |
| evrardjp | yes indeed, two images there -- ubuntu_xenial-latest and opensuse_15-latest | 18:04 |
| corvus | evrardjp: is this image new? https://hub.docker.com/r/openstackhelm/ceph-config-helper/tags | 18:05 |
| evrardjp | it's a good example, as this has two images,like many others but not all | 18:05 |
| corvus | evrardjp: i don't see the other tag there | 18:05 |
| evrardjp | mmm | 18:05 |
| evrardjp | that's very weird. | 18:05 |
| corvus | and dockerhub says "6 downloads" | 18:06 |
| evrardjp | all the images are in port namespace | 18:06 |
| evrardjp | that's fine | 18:06 |
| evrardjp | don't worry | 18:06 |
| evrardjp | it's good | 18:06 |
| evrardjp | :) | 18:06 |
| evrardjp | I am glad we worked on that one :p | 18:06 |
| evrardjp | that could have been bad. | 18:06 |
| corvus | in "port namespace"? | 18:06 |
| evrardjp | but there was a change_<changeid> one before, right? | 18:06 |
| corvus | evrardjp: yes, that got cleaned up when it promoted | 18:06 |
| evrardjp | yeah but I mean it will be harder to test now :p | 18:07 |
| corvus | yes | 18:07 |
| SotK | prometheanfire: https://storyboard-dev.openstack.org/#!/worklist/506 | 18:07 |
| evrardjp | for the "port" thing, sorry for lack of clarity: in openstack-helm we are consuming a ceph-config-helper image taht's stored on dockerhub in /port/ceph-config-helper | 18:07 |
| *** yamamoto has joined #openstack-infra | 18:07 | |
| evrardjp | therefore it's normal :p | 18:07 |
| corvus | evrardjp: okay, so this repository is new | 18:08 |
| corvus | at least, under this name | 18:08 |
| evrardjp | yeah | 18:08 |
| corvus | evrardjp: so the error is that the promote role isn't expecting two images with the same repository | 18:08 |
| evrardjp | vbmc is not the same for example: https://hub.docker.com/r/openstackhelm/vbmc/tags | 18:08 |
| evrardjp | corvus: oh | 18:08 |
| evrardjp | I didn't think of that. | 18:09 |
| corvus | evrardjp: so it pushed the suse image up as "ceph-config-helper:change_639334" then it overwrote that with the xenial image as "ceph-config-helper:change_639334" | 18:09 |
| *** panda|ruck is now known as panda|ruck|off | 18:09 | |
| SotK | prometheanfire: I clicked "or", then switched the story dropdown to "Tasks", "matching" to "not matching", then typed Merged in the box and waited for the criteria options to load (which took ages for some reason) | 18:09 |
| SotK | then clicked "and" and repeated the process for the other two criteria | 18:09 |
| corvus | then it promoted ceph-config-helper:change_639334 (the xenial image) to opensuse_15-latest, then deleted ceph-config-helper:change_639334, then tried to promote ceph-config-helper:change_639334 to ubuntu_xenial-latest and failed since it had just deleted the change_ tag | 18:09 |
| corvus | evrardjp: i didn't think of that either when i wrote the role :( | 18:10 |
| prometheanfire | SotK: ok, thanks for the detailed description | 18:10 |
| corvus | evrardjp: so i think the "change_1234" tag needs to be "change_1234_ubuntu_xenial-latest" | 18:11 |
| evrardjp | I find it not really elegant tbh, so if you think I should change the approach that could be done too. | 18:11 |
| evrardjp | but it's closer to what OSH already consumes. | 18:11 |
| *** jpena is now known as jpena|off | 18:11 | |
| evrardjp | corvus: yeah so a change in the publish and the promote | 18:12 |
| evrardjp | ok | 18:12 |
| *** bhavikdbavishi has quit IRC | 18:12 | |
| *** yamamoto has quit IRC | 18:12 | |
| corvus | evrardjp: well, at the very least, the role should probably not corrupt the image like this :) so i think we should fix that, and it's up to you which way you want to structure it. | 18:12 |
| corvus | evrardjp: but, i still don't know why this didn't work when zuul ran it | 18:12 |
| corvus | evrardjp: i'm going to look closer into how zuul is decrypting these secrets | 18:13 |
| evrardjp | yeah because the issue is definitely about that uri | 18:13 |
| evrardjp | well it looks like it in the log at least | 18:13 |
| prometheanfire | SotK: heh, can't remove something once you click 'and', so have to start over | 18:13 |
| prometheanfire | results in 500 | 18:13 |
| evrardjp | that might interest you that username is also encrypted, vs what is done in other jobs. | 18:13 |
| corvus | evrardjp: yeah... it *should* work :) | 18:13 |
| prometheanfire | SotK: https://gist.github.com/prometheanfire/ddc4e91b285285f1221ed0d68743e42e | 18:13 |
| *** jtomasek has quit IRC | 18:15 | |
| fungi | SotK: yeah, similar delay experience for me... i fired up `top` on storyboard-dev and watched mysqld eating >350% of a vcpu for a long while | 18:15 |
| prometheanfire | SotK: working now though :D https://storyboard.openstack.org/#!/worklist/567 | 18:15 |
| fungi | (so basically 87.5% of the 4 vcpus for that vm) | 18:15 |
| prometheanfire | fungi: yourwelcome :D | 18:16 |
| fungi | prometheanfire: it was probably several of us at once | 18:16 |
| prometheanfire | ya | 18:16 |
| SotK | prometheanfire: oh, that's an ugly bug | 18:16 |
| fungi | also this was storyboard-dev, not sure if that's where you were trying | 18:16 |
| SotK | glad its working for you now | 18:16 |
| prometheanfire | no, I was on prod, same thing | 18:16 |
| prometheanfire | type, wait 5-10 sec, click | 18:16 |
| *** kopecmartin is now known as kopecmartin|off | 18:16 | |
| * SotK guesses we don't have a non-fulltext index on the title column and probably should add one | 18:17 | |
| fungi | prometheanfire: so just to compare, on the staging server it was more like a 20-30 second delay for me | 18:18 |
| prometheanfire | ah | 18:19 |
| fungi | and we don't have the "working..." spinny icon on those fields like we do normal search fields that i can see, which made it extra confusing for me at first | 18:19 |
| clarkb | https://github.com/pyca/cryptography/pull/4790 is the upstream fix for cryptography wheels | 18:20 |
| clarkb | that needs to merge then they need to build new wheels | 18:20 |
| clarkb | until then anything installing cryptography without python-dev and libssl-dev headers will fail | 18:20 |
| clarkb | er anything doing that with python3 | 18:20 |
| SotK | fungi: apparently not, I feel like that whole part of the UI could do with some polishing | 18:20 |
| clarkb | though I guess our test jobs will use our wheel mirror, this is limited to problems where we don't use that mirror (like withe the tests for the bridge server) | 18:21 |
| fungi | SotK: thanks for confirming, i'll add a story for that too | 18:21 |
| prometheanfire | so, that's a filtered task view, can you do something like that for story (list stories with tasks in PROJECT and task in project status !=) | 18:22 |
| *** markvoelker has joined #openstack-infra | 18:22 | |
| *** trown|lunch is now known as trown | 18:25 | |
| *** sdake has joined #openstack-infra | 18:25 | |
| *** wolverineav has joined #openstack-infra | 18:26 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fixed isocode country validator https://review.openstack.org/639766 | 18:28 |
| *** jesusaur has quit IRC | 18:28 | |
| *** sshnaidm is now known as sshnaidm|afk | 18:28 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fixed isocode country validator https://review.openstack.org/639766 | 18:32 |
| evrardjp | corvus: I will have to go for today. I will check the irc log tomorrow morning. I will speak in the osh channel to see what we do for the repo's name so that we know if a change is needed in the role to, for example, take the first tag of the image to append on the change_<number> pushed image. | 18:32 |
| *** efried_afk is now known as efried | 18:32 | |
| fungi | prometheanfire: yeah, that's another requested feature. right now you can filter stories by story status and tasks by task status but filtering stories transitively by status of associated tasks isn't yet implemented | 18:32 |
| evrardjp | in the meantime I guess the uri failure is quite a pickle :p | 18:32 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Cleanup old health server https://review.openstack.org/639767 | 18:33 |
| corvus | evrardjp: k, see you tomorrow :) | 18:33 |
| *** sdake has quit IRC | 18:33 | |
| evrardjp | thanks for your time already! | 18:33 |
| clarkb | infra-root ^ ok I think that is ready and I'll delete the old server if you all indicate you agree in review | 18:33 |
| prometheanfire | fungi: ok | 18:34 |
| *** jesusaur has joined #openstack-infra | 18:34 | |
| clarkb | reading the cryptography bug and associated pip bug I wonder if our wheel builds don't have to do similar if we want general python3 support. Anyone know what --build-option --py-limited-api=cp34 does for cryptography? | 18:37 |
| *** sdake has joined #openstack-infra | 18:38 | |
| clarkb | oh I think I know. They build their wheel with a single version of python3 then produce wheels for other versions of python3 | 18:40 |
| clarkb | we do universal wheels so should be fine | 18:40 |
| corvus | evrardjp, fungi: i think the problem *is* encoded newlines in the username and password. i didn't realize that the decoding process fungi documented would not produce its own newlines, so i didn't see that they contained newlines when i did that. i have decrypted it using another process and see that both the username and password have newlines in them. | 18:41 |
| corvus | evrardjp: my guess is that the upload task is not sensitive to extra whitespace, but the promote task is | 18:42 |
| evrardjp | corvus: haha ok "classic" | 18:42 |
| evrardjp | should we -%} ? | 18:42 |
| evrardjp | I think it's fine, as long as portdirect generates the things properly then :p | 18:43 |
| corvus | evrardjp, portdirect: so we should do two things: fix the roles as discussed earlier, and then re-encrypt the secrets without whitespace (either use "--strip" or some method of providing the input that doesn't include newlines) | 18:43 |
| clarkb | should we make --strip default in the encode script | 18:43 |
| evrardjp | corvus: yeah I will fix the roles if that's what portdirect wants to do or if that's what anyone wants to do. | 18:44 |
| *** sdake has quit IRC | 18:44 | |
| evrardjp | clarkb: probably wise | 18:44 |
| evrardjp | ttyl everyone, and thanks for the help! | 18:44 |
| corvus | clarkb: the opposite is probably desired when you're encrypting a binary private key :/ | 18:45 |
| evrardjp | oh yeah totally forgot that | 18:45 |
| *** sdake has joined #openstack-infra | 18:45 | |
| clarkb | corvus: ya but you're more likely to use the file input in that case maybe? | 18:45 |
| fungi | corvus: yeah, to me the indicator of lack of newline in the output of the command i documented was that my new shell prompt continued on the same line immediately following the last character it output | 18:46 |
| corvus | fungi: yep. i understand that now. :( | 18:46 |
| fungi | i agree we need a less confusing story around trailing newlines in zuul secrets | 18:47 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add a tool to decrypt a secret https://review.openstack.org/639771 | 18:47 |
| fungi | if it helps, i had the same confusion when comparing output, until i actually encoded the same secret with --strip (or a similar method which wouldn't send a trailing newline into the pipe) and then noticed the difference | 18:48 |
| corvus | clarkb: that's probably fine | 18:48 |
| corvus | anyway, that tool ^ will decrypt every secret in a yaml file and python repr() it | 18:49 |
| fungi | neat! | 18:49 |
| corvus | it uses zuul functions, since i mostly wrote it to see if there was a problem with zuul itself | 18:49 |
| *** mriedem_away is now known as mriedem | 18:50 | |
| fungi | part of my concern with documenting how to decrypt zuul secrets is that i didn't want to provide unnecessary ammunition to users challenging the note at https://docs.openstack.org/infra/manual/zuulv3.html#secret-variables | 18:50 |
| corvus | fungi: yep | 18:51 |
| *** sdake has quit IRC | 18:51 | |
| *** takamatsu has joined #openstack-infra | 18:52 | |
| *** markvoelker has quit IRC | 18:56 | |
| *** sdake has joined #openstack-infra | 18:56 | |
| *** e0ne has joined #openstack-infra | 18:58 | |
| *** wolverineav has quit IRC | 18:59 | |
| clarkb | fungi: corvus https://review.openstack.org/#/c/639767/ qucik review on that to cleanup the old health server in our configs? | 19:01 |
| *** luizbag has quit IRC | 19:06 | |
| *** luizbag has joined #openstack-infra | 19:06 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Handle multiple docker images with the same repository https://review.openstack.org/639779 | 19:12 |
| corvus | evrardjp, portdirect: ^ | 19:12 |
| *** wolverineav has joined #openstack-infra | 19:16 | |
| *** wolverineav has quit IRC | 19:16 | |
| *** wolverineav has joined #openstack-infra | 19:16 | |
| *** electrofelix has quit IRC | 19:17 | |
| *** ijw has quit IRC | 19:19 | |
| *** wolverineav has quit IRC | 19:24 | |
| *** wolverineav has joined #openstack-infra | 19:29 | |
| *** betherly has joined #openstack-infra | 19:30 | |
| *** yamamoto has joined #openstack-infra | 19:32 | |
| *** betherly has quit IRC | 19:34 | |
| *** yamamoto has quit IRC | 19:37 | |
| clarkb | infra-root so its been about two days since I flipped kdc01 kerberos master to kdc03. As far as I can tell afs is fine with that switch and I have bene able to add and delete a test principal. Anything else you think I should do before deleting kdc01 and do we want to snapshot that server prior to deletion? | 19:40 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Fixed cloud assets config https://review.openstack.org/639792 | 19:40 |
| clarkb | fungi: ^ you've asked for snapshot of the puppetmaster, do you think it would be nice to have for this server too? | 19:40 |
| fungi | for kdc01? maybe... | 19:42 |
| clarkb | ya kdc01 | 19:42 |
| clarkb | I'd like to delete all three servers I've got queued after lunch, health.o.o, kdc01.o.o, puppetmaster.o.o | 19:43 |
| fungi | i wouldn't have had any files on there but maybe corvus or mordred would want to weigh in since they were the main ones to build it out | 19:43 |
| clarkb | I guess I can always make a snapshot and delete it later if we decide we dont' need it | 19:43 |
| openstackgerrit | Merged openstack-infra/system-config master: Cleanup old health server https://review.openstack.org/639767 | 19:44 |
| fungi | clarkb: that's what i've done when in doubt | 19:45 |
| clarkb | ok I'll do that then | 19:45 |
| clarkb | I tend to avoid snapshots if I can because they can take a lot of time, but easy enough to queue those up | 19:45 |
| fungi | yeah, they can be time-consuming, particularly on older flavors that had huge rootfs | 19:45 |
| *** spsurya has quit IRC | 19:52 | |
| *** markvoelker has joined #openstack-infra | 19:53 | |
| *** Vadmacs has quit IRC | 19:58 | |
| *** e0ne has quit IRC | 19:59 | |
| *** ccamacho has quit IRC | 20:00 | |
| *** wolverineav has quit IRC | 20:00 | |
| *** wolverineav has joined #openstack-infra | 20:01 | |
| *** quiquell|off is now known as quiquell | 20:02 | |
| *** luizbag has quit IRC | 20:04 | |
| openstackgerrit | Merged openstack-infra/bindep master: Replace deprecated bindep-fallback testing https://review.openstack.org/636163 | 20:04 |
| *** wolverineav has quit IRC | 20:06 | |
| *** sreejithp has quit IRC | 20:07 | |
| *** e0ne has joined #openstack-infra | 20:09 | |
| corvus | fungi, clarkb: i'm fine with deleting | 20:09 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/bindep master: Adds job to test bindep on Fedora https://review.openstack.org/635590 | 20:10 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/bindep master: Adds job to test bindep on Fedora https://review.openstack.org/635590 | 20:12 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/bindep master: Adds job to test bindep on Fedora https://review.openstack.org/635590 | 20:16 |
| openstackgerrit | James E. Blair proposed openstack/boartty master: Fix opening a story by ID https://review.openstack.org/639807 | 20:17 |
| corvus | i wrote that while waiting for a story to load :( | 20:17 |
| *** e0ne has quit IRC | 20:17 | |
| openstackgerrit | James E. Blair proposed openstack/boartty master: Increase width of task number column in story view https://review.openstack.org/639808 | 20:20 |
| *** jamesmcarthur_ has quit IRC | 20:20 | |
| mordred | fungi, clarkb: I am also fine with deleting | 20:21 |
| ianw | infra-root: if i can get an eye on adding graphite01.opendev.org to inventory in -> https://review.openstack.org/#/c/639466/ i'll monitor today and make sure looks ok & can organise a cut-over for storage and dns | 20:22 |
| clarkb | great I'm cooking up some lunch but will proceed with deletes after | 20:22 |
| corvus | heh, so all of the tasks in https://storyboard.openstack.org/#!/story/2004627 have notes now. some of them are blank. sorry, that appears to be a boartty bug. | 20:22 |
| corvus | but the good news is that adding task notes *does* work :) | 20:23 |
| corvus | or, maybe. i dunno. there's a triangle, but there's no edit button. | 20:23 |
| corvus | anyway, check the task notes before you work on stuff, i left breadcrumbs in there that may help. | 20:24 |
| corvus | ianw, clarkb, fungi, mordred: ^ re https://storyboard.openstack.org/#!/story/2004627 | 20:24 |
| fungi | thanks!!! | 20:25 |
| openstackgerrit | Merged openstack/boartty master: Fix opening a story by ID https://review.openstack.org/639807 | 20:25 |
| openstackgerrit | Merged openstack/boartty master: Increase width of task number column in story view https://review.openstack.org/639808 | 20:25 |
| *** markvoelker has quit IRC | 20:26 | |
| ianw | prometheanfire: per http://grafana.openstack.org/d/NcFtwKriz/nodepool-dib-status?orgId=1 it seems gentoo is in currently in a state of permanent failure | 20:27 |
| ianw | 2019-02-27 20:26:32.446 | + /opt/dib_tmp/dib_build.8o3Mrsev/hooks/root.d/10-gentoo-image:main:88 : gpgv --keyring /opt/dib_tmp/tmp.F8esaDGroi/gentookeys.gpg /opt/dib_cache/gentoo-stage4-systemd.asc | 20:28 |
| ianw | 2019-02-27 20:26:32.448 | gpgv: Can't check signature: public key not found | 20:28 |
| ianw | appears to be the root cause | 20:28 |
| fungi | sounds like we probably need to update the keyring? | 20:29 |
| fungi | gut hunch says they rotated signing keys and the new key isn't in the keyring copy we're embedding | 20:30 |
| *** e0ne has joined #openstack-infra | 20:30 | |
| prometheanfire | fungi: need to look and see, we did rotate a while ago | 20:31 |
| prometheanfire | like a month | 20:31 |
| prometheanfire | reqs meeting right now though | 20:31 |
| fungi | yeah, i expect it's something that can wait ;) | 20:31 |
| mordred | clarkb: you recently launch-node'd ... it's currently in good shape? | 20:32 |
| * mordred is going to do the "boot up some servers for gitea" task in just a little bit | 20:32 | |
| prometheanfire | fungi: I've been working upstream on some keysigning stuff for portage, also, just build gentoo images from dib master on the 21st | 20:33 |
| prometheanfire | and that verifies | 20:33 |
| *** e0ne has quit IRC | 20:33 | |
| mordred | also - I guess we should call them gitea0[x].opendev.org yeah? | 20:33 |
| openstackgerrit | Merged openstack/boartty master: Change openstack-dev to openstack-discuss https://review.openstack.org/621938 | 20:33 |
| clarkb | mordred: you want ianws change to flush cache but that will only affect you if we've previohsly booted nodes with those names | 20:33 |
| clarkb | mordred: +1 on naming | 20:33 |
| mordred | ok. yeah - I donm't think I need that this time | 20:33 |
| fungi | clarkb: and also less likely to be an issue if you're sticking with the default image | 20:34 |
| *** e0ne has joined #openstack-infra | 20:35 | |
| clarkb | mordred: my change for package updates before reboots is in so that should be handled | 20:35 |
| mordred | ++ | 20:35 |
| *** wolverineav has joined #openstack-infra | 20:35 | |
| mordred | clarkb: do we have a venv with osc in it? | 20:36 |
| *** eharney has quit IRC | 20:36 | |
| *** tosky has quit IRC | 20:36 | |
| prometheanfire | ianw: kicking off a new gentoo build should work? | 20:37 |
| mordred | nevermind. foundone | 20:37 |
| *** dave-mccowan has quit IRC | 20:38 | |
| ianw | prometheanfire: well it's running in a loop at the moment http://nl01.openstack.org/dib-image-list | 20:38 |
| clarkb | mordred: I have one in my homedir | 20:39 |
| mordred | clarkb: yeah - I apparently have one in ~ too | 20:39 |
| prometheanfire | ianw: ack | 20:39 |
| mordred | clarkb: we should maybe normalize that at some point :) | 20:39 |
| *** wolverineav has quit IRC | 20:40 | |
| clarkb | changes to the bridge playbooks currently failing testing ehile we wait on cryptogrqphy to push new wheels | 20:40 |
| *** wolverineav has joined #openstack-infra | 20:41 | |
| *** wolverineav has quit IRC | 20:41 | |
| clarkb | but I agree maybe a change to install osc globally | 20:41 |
| *** wolverineav has joined #openstack-infra | 20:41 | |
| *** jillr has quit IRC | 20:42 | |
| *** jillr has joined #openstack-infra | 20:42 | |
| *** betherly has joined #openstack-infra | 20:43 | |
| *** dave-mccowan has joined #openstack-infra | 20:44 | |
| *** wolverineav has quit IRC | 20:45 | |
| mordred | yeah. | 20:48 |
| *** betherly has quit IRC | 20:48 | |
| prometheanfire | reproduced at least | 20:49 |
| mordred | clarkb: the unattended-upgrades playbook had a sad | 20:54 |
| openstackgerrit | Merged openstack-infra/system-config master: Add graphite01.opendev.org https://review.openstack.org/639466 | 20:54 |
| mordred | clarkb: "Cache has broken packages, exiting" | 20:54 |
| mordred | investigating more | 20:54 |
| clarkb | mordred: hrm we might need an apt get update first? | 20:55 |
| mordred | you have one | 20:55 |
| clarkb | and I think unattended upgrades does one internally too | 20:56 |
| mordred | "stderr_lines": ["Exception: E:Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages."] | 20:56 |
| clarkb | oh interesting its like a real packaging error | 20:56 |
| mordred | I'mma try again with --keep | 20:57 |
| mordred | see if the error is more readable in person | 20:57 |
| *** jamesmcarthur has joined #openstack-infra | 20:57 | |
| *** jaosorior has quit IRC | 21:08 | |
| mordred | clarkb: ok. I'm stumped | 21:09 |
| mordred | clarkb: unattended-upgrade fails consistently with that error (no new info running it by hand) | 21:09 |
| mordred | clarkb: but apt-get dist-upgrade works fine | 21:10 |
| mordred | clarkb: apt-get upgrade *did* show packages held back - kernels and also netplan.io | 21:10 |
| prometheanfire | ianw: new signing subkey, testing | 21:12 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Fix multiple jobs with provides/requires https://review.openstack.org/639472 | 21:13 |
| openstackgerrit | Alex Litvinov proposed openstack-infra/project-config master: add cinder-backup-swift charm and interface https://review.openstack.org/639831 | 21:19 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: Update gentoo-releng gpg key https://review.openstack.org/639833 | 21:19 |
| prometheanfire | ianw: fungi ^ | 21:19 |
| *** yamamoto has joined #openstack-infra | 21:20 | |
| *** markvoelker has joined #openstack-infra | 21:23 | |
| *** jamesmcarthur has quit IRC | 21:24 | |
| *** jamesmcarthur has joined #openstack-infra | 21:24 | |
| clarkb | mordred: is this on xenial or bionic? | 21:25 |
| *** yamamoto has quit IRC | 21:25 | |
| clarkb | mordred: and is this in vexxhost? we've seen similar on the vexxhost ns servers with some packages iirc (re held back) | 21:25 |
| clarkb | fungi: ^ do you recall which packages those were on bionic in vexxhost? | 21:25 |
| clarkb | mordred: its possible that is a base image thing | 21:25 |
| *** betherly has joined #openstack-infra | 21:25 | |
| corvus | this should be bionic on vexxhost | 21:25 |
| mordred | clarkb: netplan.io seems to be the issue | 21:25 |
| *** redrobot has quit IRC | 21:25 | |
| mordred | and yes, it's bionic on vexxhost | 21:26 |
| mordred | I'm trying doing it with an apt-get dist-upgrade -y first | 21:26 |
| mordred | just to see if the hammer fixes it | 21:26 |
| clarkb | ya we had to convince that to install properly for one of the ns servers too | 21:26 |
| *** wolverineav has joined #openstack-infra | 21:27 | |
| *** jcoufal has quit IRC | 21:27 | |
| mordred | maybe we should just do an apt-get install -y netplan.io | 21:27 |
| clarkb | if you add it to the package list will that be sufficient? | 21:27 |
| mordred | maybe? | 21:27 |
| * mordred is trying various things | 21:27 | |
| pabelanger | is there any discussion about recent cryptography failures? I guess there is a missing wheel some place? | 21:28 |
| clarkb | pabelanger: yes I linked to their bug tracker in scrollback. Basically pip broke their python3.5/6/7 builds | 21:29 |
| clarkb | pabelanger: they have a fix in cryptography to tell pip what they really want is ok | 21:29 |
| pabelanger | clarkb: ack, thanks | 21:29 |
| clarkb | but it needs to be reviewed and merged then new wheels built and uploaded | 21:29 |
| clarkb | if you need to you can pin to < 2.6 or have python-dev and libssl-dev (or similar for python3 and your distro) | 21:29 |
| clarkb | then pip will build it from source | 21:29 |
| *** wolverineav has quit IRC | 21:30 | |
| *** wolverineav has joined #openstack-infra | 21:30 | |
| pabelanger | ++ | 21:30 |
| pabelanger | thanks for info | 21:30 |
| *** betherly has quit IRC | 21:30 | |
| clarkb | #status log Deleted Old health.openstack.org (9adaa457-16ab-48ea-9618-54af6edd798b) as it has been replaced by health01.openstack.org | 21:30 |
| openstackgerrit | Alex Litvinov proposed openstack-infra/project-config master: add cinder-backup-swift charm and interface https://review.openstack.org/639831 | 21:30 |
| openstackstatus | clarkb: finished logging | 21:30 |
| *** Guest66940 has joined #openstack-infra | 21:37 | |
| *** Guest66940 is now known as redrobot | 21:38 | |
| slaweq | clarkb: hi | 21:43 |
| slaweq | clarkb: frickler hold for me today nodes 38.108.68.77, 38.108.68.115 and 38.108.68.31 | 21:43 |
| slaweq | clarkb: I finished debugging on them - You can remove them now | 21:44 |
| slaweq | clarkb: frickler: thx a lot for Your help | 21:44 |
| clarkb | slaweq: thank you for letting us know | 21:44 |
| clarkb | I'll delete them now | 21:45 |
| *** wolverineav has quit IRC | 21:45 | |
| *** wolverineav has joined #openstack-infra | 21:45 | |
| slaweq | clarkb: thank You :) | 21:45 |
| *** eharney has joined #openstack-infra | 21:47 | |
| *** benj_- has quit IRC | 21:48 | |
| *** benj_ has joined #openstack-infra | 21:48 | |
| fungi | clarkb: mordred: i vaguely remember it being a problem (re: packages not updating cleanly on the ns servers) but i don't recall the details | 21:51 |
| fungi | cooking dinner at the moment but can follow up on it soon | 21:51 |
| *** e0ne has quit IRC | 21:54 | |
| clarkb | oh huh did we already delete the puppetmaster? | 21:56 |
| *** markvoelker has quit IRC | 21:56 | |
| clarkb | it does not ping and I don't see it in a server list | 21:56 |
| *** rh-jelabarre has quit IRC | 21:58 | |
| fungi | check the status log, but maybe so | 21:59 |
| clarkb | #status log Deleted Old kdc01.openstack.org (859d5e9c-193c-4c1b-8cb3-4da8316d060c) as it has been replaced by kdc03.openstack.org | 22:00 |
| openstackstatus | clarkb: finished logging | 22:00 |
| clarkb | 2018-12-19 20:57:05 UTC deleted old puppetmaster.openstack.org and review.openstack.org servers in rackspace dfw after creating final snapshots | 22:00 |
| clarkb | we did do that apparently | 22:00 |
| clarkb | I'll clean up dns in that case | 22:00 |
| ianw | prometheanfire: is this something that changes regularly? could it not be got during build from keyservers? | 22:01 |
| prometheanfire | ianw: no, still want to know why it happened and all | 22:01 |
| prometheanfire | ianw: https://bugs.gentoo.org/678660 no mention of why the subkey was updated | 22:03 |
| clarkb | #status log Removed kdc01.openstack.org and puppetmaster.openstack.org A/AAAA records from DNS | 22:05 |
| openstackstatus | clarkb: finished logging | 22:05 |
| *** betherly has joined #openstack-infra | 22:08 | |
| *** derekh has quit IRC | 22:09 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] Add AuthorizationRule Configuration Object, claim validation https://review.openstack.org/639855 | 22:11 |
| *** betherly has quit IRC | 22:12 | |
| *** trown is now known as trown|outtypewww | 22:15 | |
| *** agopi has quit IRC | 22:16 | |
| tonyb | Hmm gerrit seems to be re-applying my -2 each time I upload a patchset even thought I'm manually removing it is that a feature or bug? | 22:20 |
| *** ekultails has quit IRC | 22:20 | |
| corvus | tonyb: it only does that if your change is *really bad* :) | 22:20 |
| *** agopi has joined #openstack-infra | 22:20 | |
| corvus | tonyb: gerrit normally does re-apply -2 on new patchsets, but if you've removed it, it shouldn't | 22:21 |
| corvus | tonyb: can you link to the change? | 22:21 |
| tonyb | https://review.openstack.org/#/c/639011 | 22:22 |
| tonyb | I -2'd PS1 and then switched to 0,-W in patchset 2 but every new patchset has a -2,0 applied | 22:23 |
| corvus | tonyb: that is weird. it seems like a bug, but not likely one that we'll be able to diagnose or fix. the only thing i can think of is maybe the workflow vote is confusing it, so maybe try clearing that as well right before you upload the next patchset, then set it afterwords? | 22:25 |
| clarkb | am I blind? I don't see the -2? | 22:26 |
| tonyb | Patch Set 1: Code-Review-2 Workflow-1 | 22:26 |
| tonyb | Patch Set 3: -Code-Review Workflow-1 | 22:26 |
| tonyb | etc | 22:26 |
| tonyb | clarkb: I remopved it from PS5 already | 22:26 |
| clarkb | I see that is why it says -Code-Review | 22:27 |
| tonyb | clarkb: Yup it's adding it (Code-Review-2) with each PS | 22:27 |
| tonyb | I'll try removing the -W befoer I publish the next version | 22:29 |
| clarkb | (also if anyone is wondering I am pretty blind) | 22:29 |
| clarkb | though this last visit to the optometrist was the first time in about a decade my prescription didn't get worse | 22:29 |
| *** agopi has quit IRC | 22:31 | |
| *** kgiusti has left #openstack-infra | 22:34 | |
| *** jamesmcarthur has quit IRC | 22:36 | |
| *** sdake has quit IRC | 22:36 | |
| *** jamesmcarthur has joined #openstack-infra | 22:36 | |
| corvus | mordred, clarkb: it looks like gitea will redirect if you rename a project, but not if you transfer it from one org to another | 22:37 |
| mordred | corvus: oh poo | 22:38 |
| mordred | that seems unfortunate. maybe we should file a bug? | 22:38 |
| corvus | mordred: yeah... i'm looking into how easy it is to fix | 22:38 |
| clarkb | huh | 22:40 |
| clarkb | fwiw I'm trying to spin up a storyboard dev env in a bionic container really quickly to see if there are any obvious gotcha | 22:40 |
| clarkb | we have so much interest this outreachy round that making this work for people seems important | 22:41 |
| fungi | clarkb: also SotK has a dockerfile for it which might be worth starting from (if that's not what you're doing already) | 22:42 |
| clarkb | nah I'm starting from base bionic just to get a sense of what all is required (install git python pip then tox etc) | 22:42 |
| clarkb | the dockerfile and resulting image is likely better for real world use, I just won't be able to evaluate it until I haev better understanding of what is needed | 22:42 |
| fungi | k | 22:44 |
| clarkb | learning things like: cryptography issue affects storyboard :) | 22:46 |
| clarkb | this is probably captured in the bindep file if I wasn't going in so blind but it is good to know | 22:47 |
| *** sreejithp has joined #openstack-infra | 22:52 | |
| *** sdake has joined #openstack-infra | 22:53 | |
| clarkb | the systems dep list is not small | 22:57 |
| corvus | the good news is that i have sucessfully modified gitea, built a docker image, and am running it locally. the bad news is that my extremely naive attempt to add a redirect did not work. | 22:59 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Run dist-upgrade in launch-node https://review.openstack.org/639862 | 22:59 |
| *** sreejithp_ has joined #openstack-infra | 22:59 | |
| mordred | clarkb: ^^ this is what I'm I'm trying now | 23:00 |
| mordred | corvus: yay! and doh! | 23:00 |
| *** yamamoto has joined #openstack-infra | 23:00 | |
| clarkb | mordred: fun | 23:00 |
| *** tkajinam has joined #openstack-infra | 23:00 | |
| *** sreejithp has quit IRC | 23:02 | |
| *** rcernin has joined #openstack-infra | 23:06 | |
| *** rascasoft has quit IRC | 23:10 | |
| *** dave-mccowan has quit IRC | 23:14 | |
| clarkb | TIL about docker container prune. Now you don't have to manually delete all of those containers you started that were one shot? | 23:17 |
| mordred | clarkb: that's the best thing I've learned all day | 23:20 |
| clarkb | mordred: right? | 23:20 |
| mordred | clarkb: fwiw - you know about --rm right? | 23:20 |
| mordred | (I'm running container prune right now quite happily) | 23:21 |
| *** dave-mccowan has joined #openstack-infra | 23:21 | |
| clarkb | I don't know about --rm what is that? | 23:21 |
| clarkb | is that a docker run -it --rm and it automaticaly removes it afterwards? | 23:21 |
| clarkb | this is also revolutionizing my usage of this tool | 23:21 |
| mordred | clarkb: yes | 23:22 |
| mordred | that's how I run all my dockers | 23:22 |
| mordred | but sometimes stuff goes south somewhere - so thanks for the trade - container prune for run --rm | 23:22 |
| *** sreejithp_ has quit IRC | 23:23 | |
| mordred | clarkb: "Total reclaimed space: 5.844GB" | 23:23 |
| clarkb | nice | 23:24 |
| corvus | do you still need to prune volumes after running container prune? | 23:24 |
| clarkb | corvus: yes I think so | 23:24 |
| clarkb | the container prune seems to be just for the containers themselves (eg stuff that show up in docker ps) | 23:24 |
| clarkb | and volumes are meant to store long lived data between container instances | 23:26 |
| * mordred leaves volumes around much less frequently than volumes | 23:26 | |
| mordred | gah | 23:26 |
| mordred | clearly it's been a day | 23:26 |
| * mordred leaves volumes around much less frequently than containers | 23:26 | |
| *** rlandy is now known as rlandy|bbl | 23:30 | |
| mwhahaha | how are the master tarballs supposed to be updated every commit? | 23:32 |
| mwhahaha | cause the requirements tarbal is way out of date | 23:32 |
| *** betherly has joined #openstack-infra | 23:32 | |
| clarkb | mwhahaha: there is a post job that sdists the repo and copies that repo to the tarballs site iirc | 23:32 |
| mwhahaha | it seems to be broken because the requirements tarball has neutron-lib==1.22.0 but current is 1.24.0 so it's breaking kolla source installs | 23:33 |
| clarkb | http://zuul.openstack.org/build/1e0e52ac3ffb4075b4551b5328c83a0b is a recent run that says it was successful | 23:33 |
| mwhahaha | http://tarballs.openstack.org/requirements/ | 23:33 |
| mwhahaha | was last updated on 1/23 | 23:33 |
| mwhahaha | which would match that | 23:34 |
| clarkb | oh ya I can't read dates | 23:34 |
| clarkb | looking at the other post jobs there is a post failure on the publish-loci-requirements job | 23:34 |
| mordred | we make an sdist tarball of requirements? | 23:34 |
| clarkb | I think this is fallout from the zuul security change that went in last month | 23:34 |
| clarkb | similar to how cinder stopped running post things ( Ithink ti was cinder) | 23:35 |
| clarkb | prometheanfire: ^ the tl;dr is that zuul decided it wasn't safe to run jobs from another repo against a repo when secrets were involved | 23:35 |
| mordred | oh yeah. that | 23:35 |
| *** harlowja has joined #openstack-infra | 23:35 | |
| clarkb | prometheanfire: so running a loci job with loci secrets in the post pipeline for say requirements or cinder is verboten and zuul runs nothing in that case | 23:35 |
| clarkb | I think the requirements team will need to remove the loci job from their pipeline | 23:36 |
| fungi | pretty sure that's what cinder did for now as well | 23:36 |
| prometheanfire | clarkb: ? | 23:36 |
| prometheanfire | backlog? | 23:36 |
| clarkb | prometheanfire: we stopped running requirements post jobs because you have a loci job in your post pipeline that uses secrets from a loci repo | 23:36 |
| fungi | we wanted to sync up with hogepodge to figure out if those jobs are still desirable and if so then work out a solution | 23:37 |
| *** betherly has quit IRC | 23:37 | |
| clarkb | prometheanfire: this happened about a month ago when zuul decided that could not be done securely and so as a failsafe refuses to run any jobs in that situation | 23:37 |
| prometheanfire | clarkb: right, folow that | 23:37 |
| clarkb | prometheanfire: this means that tarballs.o.o isn't getting updated with requirements tarballs | 23:37 |
| clarkb | prometheanfire: which is what mwhahaha discovered | 23:37 |
| fungi | prometheanfire: end result is that no post pipeline jobs are being run for requirements changes | 23:37 |
| clarkb | prometheanfire: the fix is to remove the publish-loci-requirements job from your post pipeline | 23:37 |
| *** sdake has quit IRC | 23:38 | |
| clarkb | (I think) | 23:38 |
| prometheanfire | ah | 23:38 |
| clarkb | then we can circle back and see what needs to be done to meet loci's use case | 23:38 |
| prometheanfire | need to look at that (what post jobs we have) | 23:38 |
| prometheanfire | ah, our only post job (I think) | 23:39 |
| mwhahaha | kill it with promethanfire? :D | 23:40 |
| * mwhahaha shows himself out | 23:40 | |
| fungi | prometheanfire: it's your only post job which isn't from a project template | 23:40 |
| fungi | i think the branch tarball post job comes from the openstack-python-jobs template | 23:40 |
| prometheanfire | ya, not sure where the it comes from | 23:41 |
| prometheanfire | trying to find a refrence | 23:41 |
| fungi | but yeah, removing that post section at https://git.openstack.org/cgit/openstack/requirements/tree/.zuul.d/project.yaml#n112 should take care of the current problem | 23:41 |
| *** sdake has joined #openstack-infra | 23:41 | |
| weshay | seeing this in a few jobs fyi.. not sure if you guys can do anything about it | 23:41 |
| weshay | Errors: | 23:41 |
| weshay | 2019-02-27 21:34:07.698725 | ubuntu-xenial | Failed building wheel for cryptography | 23:41 |
| weshay | 2019-02-27 21:34:07.698906 | ubuntu-xenial | Could not build wheels for cryptography which use PEP 517 and cannot be installed directly | 23:41 |
| *** sdake has quit IRC | 23:42 | |
| fungi | weshay: yeah, the cryptography maintainers didn't upload wheels for py35 or py36 | 23:42 |
| prometheanfire | ^ | 23:42 |
| fungi | when they made a new release a few days ago | 23:42 |
| clarkb | fungi: just a few hours ago ( was early today) | 23:42 |
| fungi | oh, another one? | 23:42 |
| clarkb | cffi was the last one | 23:42 |
| clarkb | I imagine they were related | 23:43 |
| clarkb | weshay: but it may still be our problem if you are trying to use our mirrors which should have a working wheel | 23:43 |
| fungi | right, yes you're right | 23:43 |
| clarkb | weshay: if installing from pypi or our pypi mirror it will break. If you have our wheel mirror in your index list then I would expect it to work | 23:43 |
| fungi | http://mirror.iad.rax.openstack.org/wheel/ubuntu-16.04-x86_64/cryptography/ | 23:44 |
| weshay | openstack-tox-linter | 23:44 |
| weshay | is the job http://logs.openstack.org/35/639835/1/check/openstack-tox-linters/16062ff/ | 23:44 |
| weshay | oh.. hrm | 23:44 |
| fungi | clarkb: i only see cp27 and cp34 for 2.5 wheels | 23:45 |
| weshay | I think sorin may have a patch on that | 23:45 |
| mwhahaha | build/temp.linux-x86_64-3.5/_openssl.c:498:30: fatal error: openssl/opensslv.h: No such file or directory | 23:45 |
| mwhahaha | missing dep in bindep? | 23:45 |
| weshay | https://review.openstack.org/#/c/639746/2/bindep.txt | 23:45 |
| mwhahaha | ah | 23:46 |
| clarkb | ya if you have libssl-dev (or equiavlent) then pip will build from source on the spot | 23:46 |
| clarkb | fungi: 2.6 is the issue version | 23:46 |
| clarkb | fungi: so not having an 2.6 would lead to this problem | 23:46 |
| openstackgerrit | Logan V proposed openstack/diskimage-builder master: Add DIB_APT_MINIMAL_CREATE_INTERFACES toggle https://review.openstack.org/639865 | 23:46 |
| weshay | sorry for the noise | 23:47 |
| weshay | thanks :) | 23:47 |
| clarkb | I wonder why we haven't built a 2.6 wheel. I wonder if our builds fail to build it too? | 23:47 |
| clarkb | looks like for 2.5 we just copied the upstream one | 23:47 |
| openstackgerrit | Logan V proposed openstack/diskimage-builder master: Add DIB_APT_MINIMAL_CREATE_INTERFACES toggle https://review.openstack.org/639865 | 23:48 |
| prometheanfire | publish-openstack-python-branch-tarball is the source then? | 23:49 |
| clarkb | prometheanfire: that is the job you want to keep. publish-loci-requirements is the one that breaks publish-openstack-python-branch-tarball | 23:49 |
| prometheanfire | clarkb: right | 23:49 |
| clarkb | if you remove publish-loci-requirements I expect things to work again | 23:49 |
| prometheanfire | I just can't tell what publish-loci-requirements actually does | 23:50 |
| clarkb | (you may need to remove this job on all branches, not sure if it will work after fixing only master) | 23:50 |
| prometheanfire | https://github.com/openstack/loci/blob/master/.zuul.d/requirements.yaml#L10 too? | 23:50 |
| prometheanfire | ok, that's the source | 23:51 |
| clarkb | prometheanfire: ya it is ok in the source | 23:51 |
| clarkb | because of how zuul secrets work that is the right way to do it, the issue is when the job jumps repo boundaries | 23:51 |
| corvus | mordred, clarkb: https://github.com/go-gitea/gitea/issues/6210 https://github.com/go-gitea/gitea/pull/6211 | 23:53 |
| corvus | creating the patch only took 2x the time needed to github it. | 23:55 |
| *** jamesmcarthur has quit IRC | 23:55 | |
| clarkb | does that mean you are getting better at github or better at Go? | 23:56 |
| corvus | i'm going to go take a shower now. | 23:56 |
| prometheanfire | clarkb: is this correct? wanted to make a bug for tracking https://storyboard.openstack.org/#!/story/2005108 | 23:57 |
| corvus | oh, also, i have most of the work done to rename a project, i should be able to finalize that tomorrow. | 23:57 |
| corvus | (but i at least have demonstration code for both forms of renames) | 23:57 |
| mordred | I - on the other hand - have ALMOST booted a vm | 23:57 |
| clarkb | prometheanfire: it is more accurate to say zuul won't run job that are defined in another repo and have secrets. But that is close enough | 23:58 |
| clarkb | prometheanfire: you can use secrets from your own repo | 23:58 |
| prometheanfire | right, using secrets from an external repo? | 23:58 |
| prometheanfire | not local secrets | 23:59 |
| fungi | and zuul *will* run jobs that are defined in another repo and have secrets as long as that other repo is a trusted repo | 23:59 |
| corvus | prometheanfire: i added a comment | 23:59 |
| fungi | (which is why you can have jobs like that in project-config for example) | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!