| fungi | clarkb: sure thing | 00:03 |
|---|---|---|
| *** sthussey has quit IRC | 00:04 | |
| *** rlandy is now known as rlandy|bbl | 00:04 | |
| *** threestrands has joined #openstack-infra | 00:05 | |
| *** threestrands has joined #openstack-infra | 00:05 | |
| *** tosky has quit IRC | 00:07 | |
| clarkb | the downside to using the env var for ansible forks value is now I can't jus ask ps for the info on whether or not we are using default or 50 :) | 00:11 |
| clarkb | I don't think it is used on the current run | 00:11 |
| *** owalsh has joined #openstack-infra | 00:11 | |
| *** longkb has joined #openstack-infra | 00:12 | |
| ianw | if you caught ps -aef | grep ssh | wc -l at > 20 that would indicate it? | 00:13 |
| *** owalsh- has quit IRC | 00:15 | |
| clarkb | ianw: when it was on the command line -f included it, but now that it is an env var I think you hvae to check /proc/$pid/environ | 00:15 |
| clarkb | I don't think ps will dump that but maybe I am wrong | 00:15 |
| ianw | oh i just meant if you saw more than 20 ssh processes, that would indicate the limit was > 20 and ergo @ 50 | 00:16 |
| clarkb | oh | 00:17 |
| clarkb | looks like there are plenty, I think due to control persist? not sure that works | 00:19 |
| clarkb | at least cross checking against /proc doesn't seem to be set to 50 yet | 00:19 |
| *** dchen has joined #openstack-infra | 00:20 | |
| clarkb | nodepool fix should be just a few minutes away now | 00:25 |
| *** owalsh_ has joined #openstack-infra | 00:25 | |
| *** owalsh has quit IRC | 00:28 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/devstack-gate master: Update grenade settings for stable/rocky https://review.openstack.org/591594 | 00:29 |
| *** spartakos has quit IRC | 00:30 | |
| *** jamesmcarthur has joined #openstack-infra | 00:33 | |
| *** ijw has quit IRC | 00:34 | |
| *** ijw has joined #openstack-infra | 00:34 | |
| *** ijw has quit IRC | 00:35 | |
| *** ijw has joined #openstack-infra | 00:35 | |
| *** owalsh has joined #openstack-infra | 00:36 | |
| clarkb | remote puppet else is running now, when that is done we should pick up both the forks change and the nodepool fix in the next run (assuming nodepool chnage merges in the next few minutes) | 00:36 |
| *** owalsh_ has quit IRC | 00:40 | |
| *** owalsh_ has joined #openstack-infra | 00:40 | |
| *** owalsh- has joined #openstack-infra | 00:41 | |
| *** owalsh has quit IRC | 00:43 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add a run-nodepool job https://review.openstack.org/598403 | 00:44 |
| *** owalsh_ has quit IRC | 00:45 | |
| *** owalsh has joined #openstack-infra | 00:47 | |
| *** owalsh_ has joined #openstack-infra | 00:50 | |
| *** rcernin has quit IRC | 00:50 | |
| *** owalsh- has quit IRC | 00:51 | |
| *** rcernin has joined #openstack-infra | 00:51 | |
| *** owalsh has quit IRC | 00:52 | |
| *** gyee has quit IRC | 00:53 | |
| clarkb | next ansible run in ~6 minutes | 00:54 |
| *** owalsh has joined #openstack-infra | 00:55 | |
| *** yamamoto has joined #openstack-infra | 00:55 | |
| *** owalsh_ has quit IRC | 00:56 | |
| *** agopi|brb is now known as agopi | 00:56 | |
| *** owalsh_ has joined #openstack-infra | 00:57 | |
| *** threestrands has quit IRC | 00:57 | |
| *** threestrands has joined #openstack-infra | 00:58 | |
| *** threestrands has joined #openstack-infra | 00:58 | |
| *** owalsh- has joined #openstack-infra | 01:00 | |
| *** owalsh has quit IRC | 01:01 | |
| *** owalsh has joined #openstack-infra | 01:02 | |
| openstackgerrit | Kaifeng Wang proposed openstack-infra/project-config master: Add api-ref job to ironic-inspector https://review.openstack.org/599890 | 01:03 |
| *** owalsh_ has quit IRC | 01:03 | |
| *** owalsh_ has joined #openstack-infra | 01:04 | |
| *** owalsh- has quit IRC | 01:05 | |
| *** owalsh- has joined #openstack-infra | 01:07 | |
| *** tonyb has quit IRC | 01:07 | |
| *** owalsh has quit IRC | 01:07 | |
| *** tonyb has joined #openstack-infra | 01:09 | |
| *** owalsh_ has quit IRC | 01:10 | |
| *** owalsh has joined #openstack-infra | 01:10 | |
| *** slaweq has joined #openstack-infra | 01:11 | |
| *** owalsh_ has joined #openstack-infra | 01:12 | |
| *** owalsh- has quit IRC | 01:13 | |
| *** owalsh- has joined #openstack-infra | 01:15 | |
| *** slaweq has quit IRC | 01:16 | |
| *** owalsh has quit IRC | 01:16 | |
| *** owalsh has joined #openstack-infra | 01:17 | |
| *** owalsh_ has quit IRC | 01:18 | |
| *** fnordahl has quit IRC | 01:19 | |
| *** fnordahl has joined #openstack-infra | 01:20 | |
| *** timothyb89 has joined #openstack-infra | 01:20 | |
| *** owalsh_ has joined #openstack-infra | 01:21 | |
| *** tonyb_ has joined #openstack-infra | 01:21 | |
| *** eernst has quit IRC | 01:21 | |
| *** owalsh- has quit IRC | 01:21 | |
| *** owalsh has quit IRC | 01:24 | |
| *** owalsh has joined #openstack-infra | 01:25 | |
| *** eernst has joined #openstack-infra | 01:26 | |
| *** tonyb_ has quit IRC | 01:27 | |
| *** tonyb has quit IRC | 01:27 | |
| *** owalsh- has joined #openstack-infra | 01:27 | |
| *** eernst has quit IRC | 01:27 | |
| *** eernst has joined #openstack-infra | 01:28 | |
| *** owalsh_ has quit IRC | 01:28 | |
| beisner | hi all - seems like the post jobs are forever-stuck. is there a known issue? | 01:30 |
| *** owalsh_ has joined #openstack-infra | 01:30 | |
| *** owalsh has quit IRC | 01:30 | |
| clarkb | beisner: post jobs have the lowest priority behindgate and check | 01:30 |
| clarkb | beisner: this means that with the check and gate queues being busy they are getting all of the nodes starving post. this is normal | 01:31 |
| beisner | hi clarkb - ah ok, thx | 01:31 |
| *** tonyb has joined #openstack-infra | 01:31 | |
| *** eernst has quit IRC | 01:32 | |
| *** owalsh- has quit IRC | 01:34 | |
| *** owalsh has joined #openstack-infra | 01:39 | |
| *** owalsh_ has quit IRC | 01:40 | |
| *** aeng has quit IRC | 01:42 | |
| *** lbragstad has quit IRC | 01:43 | |
| *** jamesmcarthur has quit IRC | 01:43 | |
| *** lbragstad has joined #openstack-infra | 01:45 | |
| *** aeng has joined #openstack-infra | 01:45 | |
| *** jamesmcarthur has joined #openstack-infra | 01:45 | |
| *** annp has joined #openstack-infra | 01:48 | |
| *** hongbin has joined #openstack-infra | 01:51 | |
| *** rlandy|bbl is now known as rlandy | 01:54 | |
| *** armax has quit IRC | 01:54 | |
| *** camunoz has quit IRC | 01:55 | |
| *** jamesmcarthur has quit IRC | 01:56 | |
| *** owalsh_ has joined #openstack-infra | 01:56 | |
| *** aeng has quit IRC | 01:59 | |
| *** owalsh has quit IRC | 01:59 | |
| *** lbragstad has quit IRC | 02:09 | |
| *** spartakos has joined #openstack-infra | 02:10 | |
| *** slaweq has joined #openstack-infra | 02:11 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add guidelines on variable naming https://review.openstack.org/595920 | 02:12 |
| *** slaweq has quit IRC | 02:15 | |
| *** nicolasbock has quit IRC | 02:18 | |
| *** anteaya has quit IRC | 02:18 | |
| *** anteaya has joined #openstack-infra | 02:18 | |
| *** armax has joined #openstack-infra | 02:30 | |
| *** ijw has quit IRC | 02:34 | |
| *** diablo_rojo has quit IRC | 02:35 | |
| *** ijw_ has joined #openstack-infra | 02:37 | |
| *** ijw_ has quit IRC | 02:41 | |
| *** psachin has joined #openstack-infra | 02:45 | |
| *** ijw has joined #openstack-infra | 03:05 | |
| *** ijw has quit IRC | 03:10 | |
| *** slaweq has joined #openstack-infra | 03:11 | |
| *** rlandy has quit IRC | 03:13 | |
| *** gcb_ has quit IRC | 03:14 | |
| *** slaweq has quit IRC | 03:15 | |
| *** anteaya has quit IRC | 03:16 | |
| *** gcb_ has joined #openstack-infra | 03:17 | |
| *** ramishra has joined #openstack-infra | 03:19 | |
| *** ijw has joined #openstack-infra | 03:26 | |
| *** ijw has quit IRC | 03:31 | |
| *** spartakos has quit IRC | 03:35 | |
| *** ramishra has quit IRC | 03:38 | |
| *** ramishra has joined #openstack-infra | 03:39 | |
| *** eernst has joined #openstack-infra | 03:46 | |
| *** jamesmcarthur has joined #openstack-infra | 03:56 | |
| *** hongbin has quit IRC | 04:01 | |
| ianw | corvus / clarkb : https://github.com/ansible/ansible/pull/45265 ... fixes a bunch of profile issues and adds option to just print profile ... off the to great pull request dumping ground in the sky. see if anyone bites | 04:01 |
| *** yamamoto has quit IRC | 04:05 | |
| *** yamamoto has joined #openstack-infra | 04:05 | |
| *** markvoelker has quit IRC | 04:06 | |
| *** markvoelker has joined #openstack-infra | 04:06 | |
| *** threestrands has quit IRC | 04:10 | |
| *** markvoelker has quit IRC | 04:11 | |
| *** janki has joined #openstack-infra | 04:13 | |
| *** yamamoto has quit IRC | 04:16 | |
| *** gcb_ has quit IRC | 04:17 | |
| *** threestrands has joined #openstack-infra | 04:21 | |
| *** threestrands has joined #openstack-infra | 04:21 | |
| mordred | ianw: that pr adds 'import logging' but doesn't seem to use it | 04:26 |
| mordred | otherwise lgtm | 04:27 |
| ianw | oh, yep, sorry, i found the internal logging, let me repush | 04:27 |
| ianw | mordred: if you want a fun one, that i think we'll be pretty unique in triggering -> https://github.com/ansible/ansible/pull/45099 | 04:31 |
| *** ijw has joined #openstack-infra | 04:33 | |
| *** yboaron_ has joined #openstack-infra | 04:34 | |
| *** dave-mccowan has quit IRC | 04:36 | |
| *** eernst has quit IRC | 04:36 | |
| *** ijw has quit IRC | 04:37 | |
| *** longkb has quit IRC | 04:40 | |
| *** longkb has joined #openstack-infra | 04:40 | |
| *** stakeda has joined #openstack-infra | 04:42 | |
| *** Bhujay has joined #openstack-infra | 04:45 | |
| *** gcb_ has joined #openstack-infra | 04:53 | |
| *** kjackal has joined #openstack-infra | 05:04 | |
| *** markvoelker has joined #openstack-infra | 05:07 | |
| *** ijw has joined #openstack-infra | 05:11 | |
| *** slaweq has joined #openstack-infra | 05:11 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] Separate out ansible logs in puppet apply-test https://review.openstack.org/600331 | 05:14 |
| *** ijw has quit IRC | 05:15 | |
| *** slaweq has quit IRC | 05:15 | |
| openstackgerrit | Simon Westphahl proposed openstack-infra/nodepool master: Implement liveness probe for static driver https://review.openstack.org/600083 | 05:18 |
| openstackgerrit | Simon Westphahl proposed openstack-infra/nodepool master: Cleanup of leaked resource for static driver https://review.openstack.org/600084 | 05:18 |
| *** r-mibu has joined #openstack-infra | 05:28 | |
| *** ijw has joined #openstack-infra | 05:31 | |
| *** noama has joined #openstack-infra | 05:32 | |
| *** links has joined #openstack-infra | 05:33 | |
| *** hashar has joined #openstack-infra | 05:35 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove trigger-readthedocs job https://review.openstack.org/583836 | 05:38 |
| *** graphene has joined #openstack-infra | 05:38 | |
| *** udesale has joined #openstack-infra | 05:38 | |
| *** ijw has quit IRC | 05:39 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] Separate out ansible logs in puppet apply-test https://review.openstack.org/600331 | 05:40 |
| AJaeger | config-core, please review https://review.openstack.org/598383 and https://review.openstack.org/581000 https://review.openstack.org/#/c/600240/ | 05:44 |
| *** quiquell has joined #openstack-infra | 05:46 | |
| *** d0ugal_ has quit IRC | 05:49 | |
| *** d0ugal_ has joined #openstack-infra | 05:53 | |
| *** gcb_ has quit IRC | 05:53 | |
| *** links has quit IRC | 05:57 | |
| *** links has joined #openstack-infra | 05:57 | |
| *** quiquell is now known as quiquell|rover | 05:58 | |
| *** chandankumar is now known as chkumar|ruck | 05:58 | |
| openstackgerrit | Merged openstack-infra/zuul master: Remove restriction on add_host https://review.openstack.org/600205 | 05:59 |
| *** d0ugal_ has quit IRC | 06:05 | |
| *** Bhujay has quit IRC | 06:07 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] Separate out ansible logs in puppet apply-test https://review.openstack.org/600331 | 06:16 |
| *** gcb_ has joined #openstack-infra | 06:17 | |
| *** Bhujay has joined #openstack-infra | 06:18 | |
| openstackgerrit | Merged openstack-infra/project-config master: Change neutron grafana dashboard to 2-column format https://review.openstack.org/600240 | 06:22 |
| *** dchen has quit IRC | 06:24 | |
| *** seongsoocho has joined #openstack-infra | 06:25 | |
| *** pcaruana has joined #openstack-infra | 06:25 | |
| *** udesale_ has joined #openstack-infra | 06:27 | |
| *** udesale_ has quit IRC | 06:27 | |
| *** udesale_ has joined #openstack-infra | 06:29 | |
| *** udesale has quit IRC | 06:29 | |
| *** jtomasek has joined #openstack-infra | 06:32 | |
| *** d0ugal_ has joined #openstack-infra | 06:32 | |
| *** udesale_ has quit IRC | 06:39 | |
| *** aojea has joined #openstack-infra | 06:40 | |
| *** udesale has joined #openstack-infra | 06:43 | |
| AJaeger | infra-root, I had usermode +R - only registred users can talk privately to me and removed that and got spammed privately this time ;( So, spam is still around - but might be different modus operandi | 06:44 |
| *** markvoelker has quit IRC | 06:45 | |
| *** apetrich has quit IRC | 06:48 | |
| *** caphrim007_ has quit IRC | 06:50 | |
| *** slaweq has joined #openstack-infra | 06:50 | |
| *** caphrim007 has joined #openstack-infra | 06:50 | |
| *** alexchadin has joined #openstack-infra | 06:54 | |
| *** e0ne has joined #openstack-infra | 06:58 | |
| *** rcernin has quit IRC | 07:00 | |
| *** Miouge is now known as miouge | 07:00 | |
| *** graphene has quit IRC | 07:02 | |
| *** tmorin has joined #openstack-infra | 07:03 | |
| *** graphene has joined #openstack-infra | 07:04 | |
| *** shardy has joined #openstack-infra | 07:04 | |
| *** sshnaidm|off is now known as sshnaidm | 07:06 | |
| *** quiquell|rover has quit IRC | 07:06 | |
| *** Emine has joined #openstack-infra | 07:06 | |
| *** quiquell has joined #openstack-infra | 07:07 | |
| *** pguimaraes has quit IRC | 07:07 | |
| *** d0ugal_ has quit IRC | 07:07 | |
| *** priteau has joined #openstack-infra | 07:09 | |
| *** quiquell|rover has joined #openstack-infra | 07:09 | |
| *** quiquell|rover has joined #openstack-infra | 07:10 | |
| *** alexchadin has quit IRC | 07:10 | |
| *** quiquell|rover has quit IRC | 07:11 | |
| *** quiquell|rover has joined #openstack-infra | 07:12 | |
| *** quiquell has joined #openstack-infra | 07:12 | |
| *** quiquell has joined #openstack-infra | 07:13 | |
| *** quiquell has quit IRC | 07:13 | |
| *** ccamacho has joined #openstack-infra | 07:14 | |
| *** quiquell has joined #openstack-infra | 07:14 | |
| *** quiquell has quit IRC | 07:14 | |
| *** quiquell has joined #openstack-infra | 07:15 | |
| *** masber has joined #openstack-infra | 07:16 | |
| *** quiquell is now known as quiquell|rover | 07:16 | |
| *** masber has quit IRC | 07:16 | |
| *** masber has joined #openstack-infra | 07:17 | |
| *** zhangfei has joined #openstack-infra | 07:18 | |
| *** tesseract has joined #openstack-infra | 07:21 | |
| dirk | infra-root: anyone around who can tell me what/how/why the opensuse-mirror-update.sh is failing? | 07:23 |
| dirk | http://mirror.gra1.ovh.openstack.org/opensuse/repositories/Cloud:/OpenStack:/Master/openSUSE_Leap_42.3/repodata/ is about a week old, so it doesn't sync from the master (which is more current) | 07:23 |
| *** pguimaraes has joined #openstack-infra | 07:24 | |
| *** alexchadin has joined #openstack-infra | 07:27 | |
| *** udesale has quit IRC | 07:27 | |
| evrardjp | dirk: what do you call master? Is that the repo infra is mirroring from, or the original repos of opensuse.org? | 07:27 |
| *** udesale has joined #openstack-infra | 07:27 | |
| *** ssbarnea|bkp2 has quit IRC | 07:27 | |
| evrardjp | dirk: let me rephrase, is kernel.org/gwdg.de up to date? | 07:28 |
| hwoarang | yes | 07:28 |
| evrardjp | ok so that last change didn't hurt. | 07:29 |
| *** d0ugal_ has joined #openstack-infra | 07:34 | |
| AJaeger | infra-root, http://logs.openstack.org/51/576351/3/gate/openstack-infra-base-integration-ubuntu-bionic/9734484/job-output.txt.gz#_2018-09-06_06_41_34_575058 has "Failed to fetch http://mirror.mtl01.inap.openstack.org/ ...." - any problems with that mirror? | 07:35 |
| evrardjp | hwoarang: I am concerned that mirror.pypi releasetimes is now 2 months ago | 07:35 |
| *** pcaruana has quit IRC | 07:36 | |
| hwoarang | evrardjp: i don't understand :) | 07:36 |
| evrardjp | I see on grafana.openstack.org that the latest mirror.debian/ubuntu was done an hour ago, mirror.opensuse 11 days ago, mirror.pypi 2 months ago | 07:37 |
| evrardjp | except if I can't read the dashboard properly, which is possible too | 07:38 |
| *** ssbarnea|bkp has joined #openstack-infra | 07:40 | |
| jhesketh | dirk: rsync: failed to connect to rsync.opensuse.org (62.146.92.202): Connection timed out (110). I'm also struggling to connect to rsync.opensuse.org from my laptop, so I'm not sure if there's something wrong there? | 07:41 |
| *** tosky has joined #openstack-infra | 07:41 | |
| ianw | evrardjp: we've stopped using that mirror and now just use a reverse proxy to upstream pypi | 07:42 |
| ianw | it started to exceed the 2tb limit of the underlying AFS volume | 07:42 |
| ianw | and we had no practical way to prune it | 07:42 |
| dirk | evrardjp: gwdg.de is up to date | 07:42 |
| dirk | jhesketh: but thats the openSUSE Tubmbleweed rsync.. the rsync for repositories should happen before | 07:42 |
| dirk | jhesketh: rsync.opensuse.org has a hardware defect, so its down atm | 07:43 |
| *** markvoelker has joined #openstack-infra | 07:43 | |
| dirk | it is known to be down atm | 07:43 |
| dirk | jhesketh: the /repositories (e.g. OBS_MIRROR in opensuse-mirror-update.sh ) runs before that code path though.. | 07:43 |
| dirk | jhesketh: or does it only release the new stuff when *all* rsyncs finished successfully? | 07:43 |
| evrardjp | ianw: oh that's the reason. Maybe it should be removed from the AFS dashboard? | 07:43 |
| ianw | didn't we switch mirror.opensuse to kernel.org? or back from it or something ... istr we've had trouble finding a stable upstream | 07:43 |
| *** holser_ has joined #openstack-infra | 07:44 | |
| evrardjp | ianw: yes, but that's for a part of those, the OBS hasn't changed | 07:44 |
| ianw | evrardjp: we should clean up the volume and do that. i think we were in a bit of a fire-drill moving away from it, but things seem ok with the reverse proxy | 07:44 |
| evrardjp | so we are wondering how come :) | 07:44 |
| jhesketh | dirk: ah, right.. it looks like the repositories have sync'd to the update-mirror.. have you checked other region mirrors? | 07:45 |
| dirk | jhesketh: no, I don't know how to look them up | 07:45 |
| dirk | ovh is the only one I know and its out dated | 07:45 |
| openstackgerrit | Dirk Mueller proposed openstack-infra/system-config master: Skip occassional issues with mirroring openSUSE:Factory:Update https://review.openstack.org/600360 | 07:45 |
| evrardjp | ianw: so all references to bandersnatch will also get removed in the future? | 07:46 |
| jhesketh | dirk: looks like others are equally out of date http://mirror.ord.rax.openstack.org/opensuse/repositories/Cloud:/OpenStack:/Master/openSUSE_Leap_42.3/repodata/ | 07:46 |
| evrardjp | jhesketh: yeah on AFS it looks like mirror is 11 days out, but I am not sure how to read that | 07:46 |
| *** jpich has joined #openstack-infra | 07:47 | |
| evrardjp | s/days out/days old/ | 07:47 |
| *** pcaruana has joined #openstack-infra | 07:48 | |
| *** jpena|off is now known as jpena | 07:49 | |
| jhesketh | unfortuantely I was just on my way out for the day so I can't look any further right now. I should be back in a few hours though | 07:49 |
| jhesketh | maybe ianw or somebody else can take a quick look if not | 07:49 |
| evrardjp | yeah it must be late jhesketh ! we'll deal with that with the people available, do not worry ;) | 07:51 |
| *** priteau has quit IRC | 07:52 | |
| ianw | the date on that page is the last time the volume released, and the volume releases when the script works | 07:56 |
| ianw | + k5start -t -f /etc/opensuse.keytab service/opensuse-mirror -- timeout -k 2m 30m rsync -rlptDvz --delete --stats --delete-excluded --exclude=i586 rsync://rsync.opensuse.org/buildservice-repos-main/openSUSE:/Factory:/Update/standard/ /afs/.openstack.org/mirror/opensuse/update/tumbleweed | 07:57 |
| ianw | rsync: failed to connect to rsync.opensuse.org (62.146.92.202): Connection timed out (110) | 07:57 |
| openstackgerrit | Dirk Mueller proposed openstack-infra/system-config master: Skip occassional issues with mirroring openSUSE:Factory:Update https://review.openstack.org/600360 | 07:57 |
| ianw | that's the failure (which i guess we already knew?) | 07:57 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/project-config master: Remove references to pypi mirroring from dashboard https://review.openstack.org/600364 | 07:57 |
| dirk | ianw: see the patch above that I just proposed. that would keep the script going to the release | 07:57 |
| evrardjp | thanks dirk | 07:58 |
| hwoarang | sneaky problem :) | 08:02 |
| hwoarang | thanks dirk | 08:02 |
| evrardjp | ianw: I posted a patch about the grafana afs dashboard, but I can help further if required. https://review.openstack.org/#/c/600364/1 | 08:02 |
| evrardjp | hwoarang: sneaky is the right term indeed | 08:02 |
| evrardjp | dirk: on our side, when will that hardware be fixed? :D | 08:03 |
| *** gcb_ has quit IRC | 08:10 | |
| *** david-lyle has quit IRC | 08:14 | |
| *** dklyle has joined #openstack-infra | 08:15 | |
| *** kaisers has quit IRC | 08:19 | |
| *** longkb has quit IRC | 08:19 | |
| *** longkb has joined #openstack-infra | 08:21 | |
| *** olivierb has joined #openstack-infra | 08:22 | |
| *** kaisers has joined #openstack-infra | 08:25 | |
| *** dtantsur|afk is now known as dtantsur | 08:25 | |
| *** sambetts|afk has joined #openstack-infra | 08:27 | |
| dirk | ianw: do you have suggestions on how to detect "systemic failure" ? | 08:29 |
| dirk | ianw: for FActory:Update I really don't care actually.. | 08:29 |
| dirk | want me to try to come up with a code soluton or just change the sloppy comment in the commit message? | 08:30 |
| *** sshnaidm is now known as sshnaidm|afk | 08:31 | |
| hwoarang | i guess for systemic failure we always get that info straight from opensuse so we know when that happens | 08:36 |
| *** udesale has quit IRC | 08:47 | |
| *** udesale has joined #openstack-infra | 08:47 | |
| *** udesale has quit IRC | 08:48 | |
| *** udesale has joined #openstack-infra | 08:48 | |
| *** electrofelix has joined #openstack-infra | 08:48 | |
| *** d0ugal_ has quit IRC | 08:49 | |
| evrardjp | hwoarang: wouldn't that be the 'occasional failure' ? | 08:49 |
| evrardjp | english is hard :p | 08:49 |
| *** d0ugal has joined #openstack-infra | 08:49 | |
| evrardjp | anyway ... I am not sure this matters right now? | 08:50 |
| hwoarang | oh i read 'systematic' | 08:51 |
| cgoncalves | AJaeger, dhellmann: hi. considering py3 goal, would it make sense to rename jobs from *py35* to just *py3*? octavia is an example where we have py35 in job names but we don't force v3.5 explicitly | 08:56 |
| *** alexchadin has quit IRC | 08:58 | |
| ianw | hwoarang / evrardjp : i don't think i care that much :) i mean how do you tell the difference between an occasional failure and "repo moved and is never coming back". i don't think you can, really | 09:04 |
| evrardjp | ianw: that's how I understood it -- and yes, with this code we can't, and I think whatever code we will produce will not replace a good ol' human communication... "this has failed for x days, because of y" | 09:05 |
| evrardjp | cgoncalves: curious about how you'll later see the difference with py36... but wasn't there a thread about it? | 09:06 |
| *** gfidente has joined #openstack-infra | 09:07 | |
| *** hwoarang_ has joined #openstack-infra | 09:11 | |
| cgoncalves | evrardjp, I'd say that if the job doesn't explicitely set a python 3 version that it should be named just 'py3'. if 3.6 is set then 'py36' | 09:12 |
| *** udesale_ has joined #openstack-infra | 09:13 | |
| *** alexchadin has joined #openstack-infra | 09:14 | |
| *** hwoarang has quit IRC | 09:15 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] Separate out ansible logs in puppet apply-test https://review.openstack.org/600331 | 09:15 |
| *** udesale has quit IRC | 09:16 | |
| *** sshnaidm|afk is now known as sshnaidm | 09:16 | |
| *** threestrands has quit IRC | 09:16 | |
| evrardjp | cgoncalves: on yeah I didn't mean mutually exclusive things -- I just wanted to raise the fact that there was maybe a discussion about those :) | 09:20 |
| cmurphy | zuul has quite the backlog today | 09:20 |
| *** cshastri has joined #openstack-infra | 09:21 | |
| *** derekh has joined #openstack-infra | 09:21 | |
| ianw | cmurphy: yeah, it's been busy all .au day too | 09:21 |
| *** hwoarang_ is now known as hwoarang | 09:23 | |
| *** alexchadin has quit IRC | 09:24 | |
| cgoncalves | evrardjp, ah, got it! thanks for letting me know. I will try to look for that email thread | 09:27 |
| *** mgoddard has joined #openstack-infra | 09:27 | |
| mgoddard | hi, seeing connection failures to http://mirror.mtl01.inap.openstack.org/pypi/simple in some jobs. Is it a transient issue or should I hold off rechecking? | 09:28 |
| *** pblaho has joined #openstack-infra | 09:29 | |
| evrardjp | cgoncalves: hoping my memory doesn't play me tricks though :D | 09:29 |
| ianw | mgoddard: hrm, it doesn't seem to be responding | 09:33 |
| *** markmcd has joined #openstack-infra | 09:34 | |
| ianw | mgoddard: interesting, the entire server appears to have disappeared | 09:36 |
| openstackgerrit | Merged openstack/ptgbot master: Add channel feedback on 'book' command https://review.openstack.org/600100 | 09:37 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Revert "Revert "Disable inap-mtl01 provider"" https://review.openstack.org/600395 | 09:40 |
| *** dabukalam has joined #openstack-infra | 09:41 | |
| ianw | project-config : ^ that hurts with current backup, but i don't have time to rebuild the mirror right now | 09:41 |
| *** panda has quit IRC | 09:48 | |
| mgoddard | thanks for checking ianw | 09:49 |
| ianw | mgagne: ^^ unless you have any other thoughts? | 09:49 |
| *** d0ugal has quit IRC | 09:51 | |
| openstackgerrit | Mark Goddard proposed openstack-infra/project-config master: Add new project config for tenks https://review.openstack.org/600397 | 09:51 |
| *** udesale_ has quit IRC | 09:52 | |
| *** udesale has joined #openstack-infra | 09:53 | |
| *** hashar is now known as hasharAway | 09:53 | |
| *** aojea has quit IRC | 09:54 | |
| *** quiquell|rover is now known as quiquell|rover|a | 09:57 | |
| *** quiquell|rover|a is now known as quique|rover|afk | 09:57 | |
| *** xinliang has quit IRC | 09:58 | |
| *** xinliang has joined #openstack-infra | 09:58 | |
| *** annp has quit IRC | 10:00 | |
| *** d0ugal has joined #openstack-infra | 10:01 | |
| *** xinliang has quit IRC | 10:03 | |
| *** xinliang has joined #openstack-infra | 10:03 | |
| *** xinliang has quit IRC | 10:05 | |
| *** wolverineav has joined #openstack-infra | 10:05 | |
| *** xinliang has joined #openstack-infra | 10:05 | |
| *** xinliang has joined #openstack-infra | 10:05 | |
| *** xinliang has left #openstack-infra | 10:07 | |
| *** xinliang has joined #openstack-infra | 10:08 | |
| *** xinliang has joined #openstack-infra | 10:08 | |
| *** xinliang has quit IRC | 10:12 | |
| *** d0ugal has quit IRC | 10:14 | |
| *** xinliang has joined #openstack-infra | 10:14 | |
| *** xinliang has joined #openstack-infra | 10:14 | |
| *** e0ne has quit IRC | 10:15 | |
| *** xinliang has joined #openstack-infra | 10:15 | |
| *** xinliang has joined #openstack-infra | 10:15 | |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Revert "Disable inap-mtl01 provider"" https://review.openstack.org/600395 | 10:17 |
| *** owalsh_ is now known as owalsh | 10:19 | |
| *** quique|rover|afk is now known as quiquell|rover | 10:19 | |
| *** d0ugal has joined #openstack-infra | 10:22 | |
| *** graphene has quit IRC | 10:26 | |
| *** graphene has joined #openstack-infra | 10:28 | |
| *** nicolasbock has joined #openstack-infra | 10:31 | |
| *** stakeda has quit IRC | 10:31 | |
| *** e0ne has joined #openstack-infra | 10:39 | |
| *** dave-mccowan has joined #openstack-infra | 10:46 | |
| *** wolverineav has quit IRC | 10:54 | |
| *** pbourke has quit IRC | 10:59 | |
| *** pbourke has joined #openstack-infra | 11:00 | |
| *** jpena is now known as jpena|lunch | 11:02 | |
| *** dhajare has joined #openstack-infra | 11:03 | |
| *** ccamacho has quit IRC | 11:07 | |
| *** kjackal_ has joined #openstack-infra | 11:12 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Fix bootloader packages for aarch64 https://review.openstack.org/598595 | 11:17 |
| * jhesketh is back for a little bit | 11:21 | |
| jhesketh | +w on the skipping factory update issues, hopefully that fixes the rest of the repo being out of date | 11:21 |
| openstackgerrit | Merged openstack/diskimage-builder master: Replace assertRaisesRegexp with assertRaisesRegex https://review.openstack.org/596774 | 11:22 |
| *** Emine has quit IRC | 11:23 | |
| *** mgoddard has left #openstack-infra | 11:24 | |
| *** tmorin has left #openstack-infra | 11:24 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Fix CentOS image build failure when dib runs on system where audit disabled https://review.openstack.org/559485 | 11:25 |
| openstackgerrit | Merged openstack/diskimage-builder master: Remove legacy-opensuse-423 nodeset https://review.openstack.org/591788 | 11:25 |
| *** Emine has joined #openstack-infra | 11:25 | |
| *** aojea has joined #openstack-infra | 11:26 | |
| *** longkb has quit IRC | 11:28 | |
| *** zhangfei has quit IRC | 11:29 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Remove unsued opensuse jobs https://review.openstack.org/591797 | 11:29 |
| openstackgerrit | Mark Goddard proposed openstack-infra/project-config master: Add new project config for tenks https://review.openstack.org/600397 | 11:34 |
| *** rh-jelabarre has joined #openstack-infra | 11:38 | |
| *** eharney has quit IRC | 11:39 | |
| AJaeger | config-core, senlin is ready for python3-first - please review https://review.openstack.org/#/c/597502/ | 11:41 |
| *** hasharAway is now known as hashar | 11:49 | |
| *** psachin` has joined #openstack-infra | 11:50 | |
| *** psachin has quit IRC | 11:51 | |
| *** rfolco has joined #openstack-infra | 11:52 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove unused job legacy-grenade-dsvm-sahara https://review.openstack.org/600420 | 11:57 |
| *** jpena|lunch is now known as jpena | 11:59 | |
| *** dhajare has quit IRC | 12:00 | |
| *** trown|outtypewww is now known as trown | 12:00 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove unused job legacy-grenade-dsvm-sahara https://review.openstack.org/600420 | 12:04 |
| *** alexchadin has joined #openstack-infra | 12:11 | |
| *** boden has joined #openstack-infra | 12:12 | |
| *** bharath_t has joined #openstack-infra | 12:12 | |
| *** rh-jelabarre has quit IRC | 12:14 | |
| openstackgerrit | Merged openstack-infra/system-config master: Skip occassional issues with mirroring openSUSE:Factory:Update https://review.openstack.org/600360 | 12:18 |
| *** links has quit IRC | 12:22 | |
| *** markvoelker has quit IRC | 12:22 | |
| *** markvoelker has joined #openstack-infra | 12:23 | |
| *** links has joined #openstack-infra | 12:28 | |
| *** yboaron_ has quit IRC | 12:32 | |
| *** yboaron_ has joined #openstack-infra | 12:32 | |
| *** ccamacho has joined #openstack-infra | 12:38 | |
| *** ccamacho has quit IRC | 12:38 | |
| *** rlandy has joined #openstack-infra | 12:38 | |
| openstackgerrit | Nguyen Hai proposed openstack-infra/project-config master: remove job settings for senlin repositories https://review.openstack.org/597502 | 12:38 |
| *** ccamacho has joined #openstack-infra | 12:38 | |
| *** eharney has joined #openstack-infra | 12:38 | |
| *** alexchadin has quit IRC | 12:41 | |
| *** e0ne has quit IRC | 12:41 | |
| *** ansmith has joined #openstack-infra | 12:43 | |
| *** kgiusti has joined #openstack-infra | 12:46 | |
| *** dhajare has joined #openstack-infra | 12:47 | |
| *** r-mibu has quit IRC | 12:47 | |
| *** sthussey has joined #openstack-infra | 12:49 | |
| *** alexchadin has joined #openstack-infra | 12:49 | |
| *** jamesmcarthur has quit IRC | 12:52 | |
| *** jamesmcarthur has joined #openstack-infra | 12:52 | |
| *** lbragstad has joined #openstack-infra | 12:54 | |
| *** e0ne has joined #openstack-infra | 12:59 | |
| *** camunoz has joined #openstack-infra | 13:01 | |
| *** jamesmcarthur has quit IRC | 13:03 | |
| *** bobh has joined #openstack-infra | 13:05 | |
| *** links has quit IRC | 13:08 | |
| *** mriedem has joined #openstack-infra | 13:08 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Match tag items against containing branches https://review.openstack.org/578557 | 13:08 |
| *** lbragstad has quit IRC | 13:09 | |
| *** bobh has quit IRC | 13:09 | |
| *** auristor has quit IRC | 13:12 | |
| *** links has joined #openstack-infra | 13:13 | |
| *** yboaron_ has quit IRC | 13:14 | |
| *** links has quit IRC | 13:14 | |
| *** lbragstad has joined #openstack-infra | 13:15 | |
| *** links has joined #openstack-infra | 13:15 | |
| *** caphrim007 has quit IRC | 13:20 | |
| fungi | dirk: evrardjp: hwoarang: ianw: we'd need to put those in separate afs volumes if we want to release updates for different suse mirrors independently | 13:21 |
| efried | Good UGT morning. Can anyone (AJaeger) tell me how to add zuul doc jobs? My first two attempts in https://review.openstack.org/#/c/600268/ were merely guesses, clearly wrong. | 13:25 |
| fungi | mgagne: huh, our inap-17301 project in mtl01 is indicating there are no servers. we should have a mirror instance there | 13:28 |
| fungi | any idea what's happened? | 13:28 |
| fungi | ahh, i see ianw has already turned us back off there | 13:29 |
| *** auristor has joined #openstack-infra | 13:32 | |
| *** kjackal has quit IRC | 13:33 | |
| *** kjackal_ has quit IRC | 13:34 | |
| *** bobh has joined #openstack-infra | 13:39 | |
| *** kjackal has joined #openstack-infra | 13:40 | |
| *** kjackal_ has joined #openstack-infra | 13:41 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul-jobs master: ensure-sphinx: do not attempt a gettext install if exists https://review.openstack.org/599028 | 13:42 |
| *** eharney has quit IRC | 13:43 | |
| *** chkumar|ruck is now known as chkumar|out | 13:44 | |
| *** bobh has quit IRC | 13:46 | |
| *** caphrim007 has joined #openstack-infra | 13:46 | |
| *** dhajare has quit IRC | 13:47 | |
| *** sshnaidm is now known as sshnaidm|mtg | 13:49 | |
| *** caphrim007 has quit IRC | 13:51 | |
| *** rh-jelabarre has joined #openstack-infra | 13:53 | |
| *** anteaya has joined #openstack-infra | 13:57 | |
| *** eharney has joined #openstack-infra | 13:58 | |
| *** psachin` has quit IRC | 13:59 | |
| *** hashar is now known as hasharAway | 13:59 | |
| *** kjackal_ has quit IRC | 14:02 | |
| *** kjackal has quit IRC | 14:02 | |
| *** d0ugal has quit IRC | 14:05 | |
| *** d0ugal has joined #openstack-infra | 14:09 | |
| ssbarnea1 | corvus: https://review.openstack.org/#/c/570546/ oneliner on devstack. thanks. | 14:13 |
| *** bobh has joined #openstack-infra | 14:18 | |
| AJaeger | efried: reviewed and commented | 14:19 |
| efried | AJaeger: Thanks! | 14:19 |
| *** bharath_t has quit IRC | 14:27 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual master: Mention that docs-on-readthedocs should stay in project-config https://review.openstack.org/600453 | 14:28 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Small edits for descriptions https://review.openstack.org/600454 | 14:33 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-zuul master: Add docker-py dependency for ansible-playbook https://review.openstack.org/600456 | 14:34 |
| *** Bhujay has quit IRC | 14:34 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Use tox-docs for specs repositories https://review.openstack.org/600457 | 14:35 |
| pabelanger | infra-root: To follow up on topic in #zuul, ^ is needed to allow jobs to use docker modules with zuul-executor. I am not sure how best to manage ansible dependencies in the long run | 14:35 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Use tox-docs for specs publishing https://review.openstack.org/600458 | 14:38 |
| *** cshastri has quit IRC | 14:39 | |
| AJaeger | efried: let me send a followup change to use templates ;) | 14:40 |
| efried | AJaeger: That would be great, thank you. | 14:40 |
| *** graphene has quit IRC | 14:44 | |
| AJaeger | efried: done... | 14:44 |
| mgagne | fungi: bad news, looks like someone deleted the mirror VM by mistake in inap-mtl01. good news, volumes still exist. | 14:45 |
| AJaeger | efried: don't you want lower-constraints? | 14:45 |
| *** graphene has joined #openstack-infra | 14:46 | |
| fungi | mgagne: no worries, we'll get a fresh one built | 14:46 |
| mgagne | clarkb: as for boot timeout, could it be that since we updated (in fact, reinstalled some of) the compute nodes, all images need to be downloaded, cached and used to boot new VMs? | 14:46 |
| fungi | mgagne: "accidentally deleted" is way better than "just vanished" at least ;) | 14:47 |
| clarkb | mgagne: possibly | 14:47 |
| mgagne | fungi: someone assumed that "inap-mtl01 is disabled" means there is no VMs whatsoever left. | 14:47 |
| clarkb | fungi mgagne we can delete the volume and start over too if necessary, that is all cache data | 14:47 |
| mgagne | clarkb: whatever works for you, I'm just gonna shame the guy even more =) | 14:48 |
| clarkb | might be good for someone other than me to go through launvh node process on bridge.o.o to see what I missed in trying to get that working | 14:49 |
| efried | AJaeger: You mean the gate job? Yes, I would think so. | 14:49 |
| fungi | mgagne: sure, that's also one of the reasons we have two separate tenants/projects so that persistent control plane stuff can be isolated to one and the ephemeral stuff which is safe to wipe goes in the other | 14:49 |
| AJaeger | efried: patches sent, I'm good now ;) | 14:49 |
| efried | AJaeger: Our l-c file is still nova's, but I still don't understand how to trim it tbh because all the deps seem to be transitive. | 14:49 |
| AJaeger | efried: best ask dhellmann | 14:50 |
| efried | ack | 14:50 |
| *** d0ugal has quit IRC | 14:50 | |
| fungi | mgagne: also i accidentally delete things with some regularity, don't give him too hard of a time ;) | 14:50 |
| *** Swami has joined #openstack-infra | 14:50 | |
| mgagne | fungi: but it's the only pleasure I can get these times! ;) | 14:51 |
| openstackgerrit | Markus Hosch proposed openstack-infra/zuul master: Add --check-config option to zuul scheduler https://review.openstack.org/542160 | 14:51 |
| *** d0ugal has joined #openstack-infra | 14:52 | |
| AJaeger | efried: I think having the job will make that job easier ;) | 14:54 |
| AJaeger | dhellmann: I think kolla is ready for python3-first, want to remove WIP from https://review.openstack.org/#/c/597590/ ? | 14:55 |
| AJaeger | config-core, two cleanups: https://review.openstack.org/#/c/600420/ removes migrated template and senlin is migrated with python3-first https://review.openstack.org/597502 . please review | 14:58 |
| *** quiquell|rover has quit IRC | 15:02 | |
| *** jamesmcarthur has joined #openstack-infra | 15:03 | |
| *** janki has quit IRC | 15:04 | |
| *** hamzy has quit IRC | 15:04 | |
| *** jamesmcarthur has quit IRC | 15:08 | |
| *** kjackal has joined #openstack-infra | 15:09 | |
| *** kjackal_ has joined #openstack-infra | 15:09 | |
| *** dhajare has joined #openstack-infra | 15:10 | |
| *** gyee has joined #openstack-infra | 15:11 | |
| *** r-daneel has quit IRC | 15:17 | |
| *** sshnaidm|mtg is now known as sshnaidm | 15:20 | |
| *** d0ugal has quit IRC | 15:21 | |
| AJaeger | dhellmann: I think ec2-api and kolla are ready for python3-first, want to remove WIP from the project-config changes? | 15:22 |
| *** spartakos has joined #openstack-infra | 15:24 | |
| *** hamzy has joined #openstack-infra | 15:27 | |
| *** bobh has quit IRC | 15:29 | |
| *** bobh has joined #openstack-infra | 15:29 | |
| *** psachin` has joined #openstack-infra | 15:29 | |
| *** graphene has quit IRC | 15:30 | |
| *** psachin` has quit IRC | 15:31 | |
| *** ssbarnea|bkp has quit IRC | 15:32 | |
| *** graphene has joined #openstack-infra | 15:32 | |
| *** d0ugal has joined #openstack-infra | 15:34 | |
| *** udesale has quit IRC | 15:35 | |
| *** tosky has quit IRC | 15:36 | |
| *** alexchadin has quit IRC | 15:38 | |
| *** ccamacho has quit IRC | 15:38 | |
| *** ccamacho1 has joined #openstack-infra | 15:38 | |
| *** pblaho has quit IRC | 15:39 | |
| *** e0ne has quit IRC | 15:42 | |
| *** pcaruana has quit IRC | 15:42 | |
| *** r-daneel has joined #openstack-infra | 15:44 | |
| corvus | infra-root: it looks like we may have broken iptables on adns1 which is impacting dns for zuul-ci.org | 15:44 |
| fungi | corvus: so were we previously allowing 53/tcp+udp to adns1 from the v4 and v6 addresses of ns1 and ns2? | 15:45 |
| *** r-daneel has quit IRC | 15:45 | |
| corvus | fungi: that's my recollection | 15:45 |
| *** anteaya has quit IRC | 15:46 | |
| corvus | i'm going to try to find the old puppet config | 15:46 |
| pabelanger | yes, that looks right: http://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/site.pp?id=dde24421d0ce2053d525db601594a9e3dcca9c44#n858 | 15:47 |
| corvus | pabelanger: ++ | 15:47 |
| corvus | there's a really interesting thing we're doing there with hostnames, but that's beside the point for now :) | 15:48 |
| pabelanger | so agree, adding missing firewall rules will like be fix | 15:48 |
| corvus | i don't see any rules for adns1 | 15:49 |
| corvus | i'll work on a patch to add | 15:49 |
| *** jamesmcarthur has joined #openstack-infra | 15:49 | |
| fungi | yeah, i guess it would be in playbooks/group_vars/adns.yaml which doesn't exist? | 15:49 |
| corvus | fungi: yep | 15:49 |
| fungi | and just needs an iptables_extra_allowed_hosts list i guess to get this working | 15:51 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Allow ns servers to connect to adns https://review.openstack.org/600469 | 15:51 |
| corvus | fungi, clarkb, pabelanger: ^ | 15:51 |
| *** caphrim007 has joined #openstack-infra | 15:52 | |
| clarkb | is adns an group already? | 15:52 |
| clarkb | it is | 15:53 |
| clarkb | and we don't need udp because the transfers are too big for udp right? | 15:53 |
| corvus | clarkb: yeah, the transfers use tcp and that's all we had open before | 15:53 |
| fungi | it's axfr which will always use tcp | 15:53 |
| clarkb | related: is there a way to tell dig to return the actual records and not do validation and fudge the results for you? I guess +trace +additional and going through the entire recursive list is that | 15:53 |
| fungi | i was going to ask about udp as well until i looked at the old puppet manifest | 15:54 |
| fungi | clarkb: and +nodnssec potentially | 15:54 |
| corvus | clarkb: well, an individual query will do that. just "dig +norec ns1.zuul-ci.org @something" | 15:54 |
| clarkb | corvus: I had done dig NS zuul-ci.org last night and it returned no results | 15:55 |
| *** vdrok has joined #openstack-infra | 15:55 | |
| clarkb | even though the NS record was there, dig was excluding those that failed dnssec validation apparently | 15:55 |
| clarkb | or maybe that was the server I was digging against? | 15:55 |
| fungi | if you were relying on your recursive resolver it was likely filtering expired signed results | 15:55 |
| corvus | clarkb: right, but that was either to a recursive resolver which was validating, or it was to a server that wasn't authoritative | 15:56 |
| corvus | so i started with dig @a.root-servers.net and worked down from there | 15:56 |
| corvus | apparently there's a +sigchase option that no one has compiled in by default | 15:56 |
| corvus | clarkb: maybe tumbleweed does? :) | 15:57 |
| *** ramishra has quit IRC | 15:57 | |
| clarkb | corvus: tumbleweed manpage says that feature is deprecated in favor of delv(1) | 15:57 |
| corvus | hrm. debian says i can get delv by installing 'bind9'. :( | 15:58 |
| fungi | yeah, dig is being pruned back and delv is the new thing | 15:58 |
| *** Swami has quit IRC | 15:58 | |
| fungi | debian provides delv in its dnsutils package | 15:58 |
| fungi | so same package as dig | 15:59 |
| corvus | oh maybe i need to upgrade | 15:59 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-zuul master: Add docker dependency for ansible-playbook https://review.openstack.org/600456 | 15:59 |
| fungi | i'm on unstable. i haven't checked stable | 15:59 |
| corvus | it looks like we're rather backed up, should i direct-enqueue that? | 15:59 |
| fungi | yes, that seems prudent | 15:59 |
| corvus | done | 16:00 |
| *** pcaruana has joined #openstack-infra | 16:01 | |
| fungi | thanks! | 16:01 |
| clarkb | corvus: ianw followup on the ansible forks change, looks like we run ansible every hour and 15 minutes with the full ansible run taking just over an hour | 16:03 |
| corvus | clarkb: that's after the fix landed? | 16:04 |
| clarkb | corvus: yes, this is the last handful of runs and I double checked the fix is landed on disk. I haven't confirmed that ansible is using that config though | 16:04 |
| corvus | clarkb: since that's no change, i think we should revert back to "-f 20" and check the time. then change that to "-f 50" | 16:05 |
| clarkb | `grep 'Make sure system-config repo is up to date' /var/log/ansible/ansible.log` is how I'm checking the period on this | 16:05 |
| clarkb | corvus: that seems reasonable | 16:05 |
| corvus | i'll do it | 16:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Revert "Fix ansible forks env variable" https://review.openstack.org/600471 | 16:07 |
| *** hasharAway is now known as hashar | 16:08 | |
| *** sshnaidm is now known as sshnaidm|off | 16:10 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard master: Rationalise worklist ordering code https://review.openstack.org/600472 | 16:10 |
| clarkb | infra-root please consider https://review.openstack.org/#/c/600251/ removes +rf from all our irc channels since we have yet to see spam in this channel | 16:10 |
| fungi | clarkb: AJaeger mentioned in scrollback that when he undid his usermode +R he started getting privmsg spam. not sure if that should be taken as a sign to delay or not | 16:11 |
| *** ccamacho has joined #openstack-infra | 16:12 | |
| *** verdurin has quit IRC | 16:13 | |
| corvus | what is legacy-system-config-puppet-syntax-3 ? | 16:13 |
| *** links has quit IRC | 16:14 | |
| AJaeger | clarkb: I approved 600251 - we can quickly revert... | 16:14 |
| *** jamesmcarthur has quit IRC | 16:14 | |
| clarkb | corvus: it runs `bundle exec rake syntax` whcih I thin runs the puppet linter? | 16:14 |
| *** ssbarnea|bkp has joined #openstack-infra | 16:14 | |
| clarkb | corvus: pep8 for puppet3 | 16:15 |
| corvus | i guess we still need that until we're fully puppet4? | 16:15 |
| AJaeger | dhellmann: we need to merge https://review.openstack.org/#/c/597598/ first to finish storlets - ocata change is not ready for py35, see https://review.openstack.org/#/c/596183 | 16:15 |
| clarkb | ya probably worth keeping | 16:15 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Move project config in repo https://review.openstack.org/600475 | 16:16 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Remove system-config project config https://review.openstack.org/600476 | 16:17 |
| clarkb | fungi: ya we should watch it, overall though I think +r has been problematic for many users and at least in testing here so far things are quiet | 16:17 |
| clarkb | we aren't going to avoid all spam but as long as it isn't persistent we are probably fine | 16:17 |
| *** verdurin has joined #openstack-infra | 16:18 | |
| *** mriedem is now known as mriedem_away | 16:19 | |
| openstackgerrit | Merged openstack-infra/project-config master: remove job settings for senlin repositories https://review.openstack.org/597502 | 16:20 |
| *** eernst has joined #openstack-infra | 16:20 | |
| *** jamesmcarthur has joined #openstack-infra | 16:21 | |
| *** ccamacho has quit IRC | 16:22 | |
| openstackgerrit | Merged openstack-infra/system-config master: Allow ns servers to connect to adns https://review.openstack.org/600469 | 16:24 |
| clarkb | another followup on ansible related items from yseterday: we are writing clouds.yaml on nodepool nodes with ansible now | 16:25 |
| clarkb | seems to be working fine | 16:25 |
| fungi | excellent | 16:25 |
| clarkb | if another infra-root can review https://review.openstack.org/#/c/597728/ (thanks corvus) I can watch that go in and make sure we are cloud launching properly | 16:26 |
| openstackgerrit | Merged openstack-infra/project-config master: Stop requiring users be registered on openstack channels https://review.openstack.org/600251 | 16:26 |
| clarkb | infra-root: also with the merging of the nodepool testinfra tests I think we have a good framework in place for testing arbitrary nodes (even with secrets) as we make changes. I think we should adopt a practice of requiring tests for these chagnes as they go in | 16:27 |
| fungi | why is there an uncommitted playbooks/conn-check.yaml file in /opt/system-config on bridge.o.o? | 16:27 |
| Shrews | i thought we had already adopted that practice :) | 16:28 |
| clarkb | Shrews: we sort of were but sort of weren't (and maybe we discussed it more in the meeting I missed due to vacation), calling it out as I know I failed at it myself | 16:28 |
| clarkb | fungi: I'm guessing monty was using that to test ansible | 16:28 |
| ssbarnea1 | clarkb: I am wondering if you found a workaround for https://github.com/pypa/pip/issues/4759 bug as seeing this error in build logs is really anoying. your pip bug is almost one year old.... | 16:29 |
| fungi | clarkb: have an example? like, how would you recommend testing adns1.o.o allows 53/tcp from ns1 and ns2 now that 600469 has merged? | 16:29 |
| clarkb | ssbarnea1: to be honest I gave up on it, I tried to push that PR through responded to reviewers then 9 months later get an email asking why I ignored the reviewer | 16:30 |
| clarkb | ssbarnea1: if it was an actual breaking bug I would put more effort into it but it is just confusing noise so meh. If you want to pick it up and address the comments feel free | 16:31 |
| clarkb | fungi: https://review.openstack.org/#/c/597728/8/testinfra/test_bridge.py is the test I added for checking the cron entry was added to the node properly. But you would add a test_dns.py file and either a new job or node to existing nodeset with dns group'd nodes | 16:31 |
| clarkb | then in the testinfra test_dns.py file you can shell out to iptables -L -n (though there may be more native primitives) | 16:32 |
| fungi | clarkb: cool, thanls for the example! | 16:32 |
| clarkb | https://testinfra.readthedocs.io/en/latest/modules.html#iptables there is actually an iptables type you can use to check things there | 16:32 |
| corvus | fungi: i was about to write that job/test... do you want me to do that, or did you want to try your hand on it? | 16:32 |
| *** akhila has joined #openstack-infra | 16:32 | |
| *** florianf has quit IRC | 16:32 | |
| fungi | i can give it a shot | 16:33 |
| openstackgerrit | Merged openstack-infra/project-config master: Add stx-distcloud and stx-distcloud-client repos https://review.openstack.org/598383 | 16:33 |
| corvus | fungi: cool, i'd suggest a new 2-node job like run-nodepool with nodes named 'adns1.openstack.org' and 'ns1.openstack.org' | 16:33 |
| corvus | fungi: er, well, a 3 node job cause you also need bridge.openstack.org | 16:33 |
| Shrews | clarkb: what is installing run_cloud_launcher.sh into /opt/system-config ? | 16:33 |
| corvus | Shrews: it's actually in the repo | 16:34 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard master: Rationalise worklist ordering code https://review.openstack.org/600472 | 16:34 |
| corvus | Shrews: /opt/system-config is a checkout of the system-config repo, and it's here: http://git.openstack.org/cgit/openstack-infra/system-config/tree/run_cloud_launcher.sh | 16:35 |
| fungi | clarkb: corvus: a new 3-node job just to test dns transfers? i wonder how many jobs will be running against system-config changes in the future | 16:35 |
| clarkb | Shrews: https://git.openstack.org/cgit/openstack-infra/system-config/tree/playbooks/update-system-config.yaml runs as part of https://git.openstack.org/cgit/openstack-infra/system-config/tree/run_all.sh#n36 | 16:35 |
| clarkb | fungi: we've been using file matches to scope them to the files that are relevant, the nodepool job should have examples of that as well | 16:36 |
| corvus | fungi: i've been pretty agressive with file matchers for these. | 16:36 |
| *** jpena is now known as jpena|off | 16:36 | |
| fungi | got it | 16:36 |
| fungi | concerns alleviated in that case | 16:36 |
| corvus | the file organization makes it really nice actually | 16:36 |
| pabelanger | clarkb: +1 to testinfra for system-config. Think it has been working great so far | 16:37 |
| *** aojea has quit IRC | 16:37 | |
| *** jamesmcarthur has quit IRC | 16:37 | |
| *** jamesmcarthur has joined #openstack-infra | 16:39 | |
| corvus | fungi, clarkb: related, i proposed https://review.openstack.org/600475 so that when we're making new jobs, we can temporarily stop running the 20 other puppet jobs which aren't relevant. maybe we should go ahead and get that moving? | 16:39 |
| Shrews | clarkb: +Ad | 16:40 |
| clarkb | Shrews: thanks | 16:40 |
| fungi | corvus: yep, i was looking to see if i needed to base this job on that change | 16:40 |
| fungi | but other than risking merge conflicts, i suppose not | 16:40 |
| clarkb | corvus: we could also add file matchers to run the puppet jobs when .pp files and some of the support things change | 16:40 |
| corvus | fungi: yeah, not much point since it requires the corresponding project-config change to be useful | 16:41 |
| corvus | clarkb: yeah, that sounds good, though i can't wrap my head around that myself just yet :) | 16:41 |
| *** jpich has quit IRC | 16:41 | |
| * clarkb wanders off to find tea while cloud launcher change goes through the gate | 16:43 | |
| *** trown is now known as trown|lunch | 16:47 | |
| AJaeger | config-core, some reviews for your consideration, please: a small update for descriptions in ozj https://review.openstack.org/600454 removal of unused job https://review.openstack.org/600420 and removal of unused irc channel https://review.openstack.org/598477 | 16:48 |
| *** dtantsur is now known as dtantsur|afk | 16:49 | |
| *** gfidente has quit IRC | 16:50 | |
| clarkb | AJaeger: that clarification on the docs jobs is useful. Do we keep that job around for older branches? | 16:51 |
| *** derekh has quit IRC | 16:55 | |
| *** apetrich has joined #openstack-infra | 16:55 | |
| *** holser_ has quit IRC | 16:56 | |
| *** olivierb has quit IRC | 16:59 | |
| *** shardy has quit IRC | 17:00 | |
| *** jaosorior has quit IRC | 17:00 | |
| *** jaosorior has joined #openstack-infra | 17:01 | |
| AJaeger | thanks, clarkb and corvus | 17:02 |
| AJaeger | clarkb: yes, we keep it around for older branches | 17:02 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove #openstack-product https://review.openstack.org/598477 | 17:03 |
| AJaeger | clarkb: I also switched specs jobs to use the new jobs see https://review.openstack.org/600458 and https://review.openstack.org/600457 . I don't think it's worth for those to add a new template and then rename templates for each repo... | 17:04 |
| *** jamesmcarthur has quit IRC | 17:04 | |
| *** ccamacho has joined #openstack-infra | 17:05 | |
| clarkb | AJaeger: I'm trying to think if they are generally compatible. We don't use branches on the spec repos so if they aren't the fix doesn't have to go to multiple repos at least | 17:05 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Small edits for descriptions https://review.openstack.org/600454 | 17:06 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove unused job legacy-grenade-dsvm-sahara https://review.openstack.org/600420 | 17:06 |
| clarkb | AJaeger: the changes lgtm, we should probably send email to the dev lsit about it when the changes go in | 17:07 |
| AJaeger | clarkb: the difference is that we in the sphinx job "tox -e venv python setup.py build_sphinx" - while tox-docs runs "tox -e docs". Yes, there are no branches.. | 17:07 |
| clarkb | AJaeger: ya so requires that there be a docs target in the specs repos (not sure if they all have one currently) | 17:08 |
| AJaeger | clarkb: yes, wanted to send an email once I had some thumbs up here ;) Will do. | 17:08 |
| AJaeger | clarkb: let me check quicly... | 17:08 |
| *** noama has quit IRC | 17:09 | |
| AJaeger | ... all have docs - and manual checking of first 20 it all looks fine. | 17:09 |
| *** spartakos has quit IRC | 17:11 | |
| *** r-daneel has joined #openstack-infra | 17:11 | |
| clarkb | does anyone have time to work on the inap mirror rebuild today? I'm happy to help, but do think a second set of eyes on the process would be valuable (also 190 more test nodes would help with our backlogs) | 17:13 |
| *** bobh has quit IRC | 17:14 | |
| *** jamesmcarthur has joined #openstack-infra | 17:17 | |
| *** jamesmcarthur_ has joined #openstack-infra | 17:17 | |
| openstackgerrit | Merged openstack-infra/system-config master: Run cloud launcher on bridge.o.o https://review.openstack.org/597728 | 17:17 |
| corvus | clarkb: what's required? | 17:17 |
| clarkb | the channel mode change has been applied globally now looks like | 17:17 |
| clarkb | corvus: running through the launch-node steps on bridge.openstack.org and working out where things don't quite work according to the docs. One case will be needing root/sudo | 17:18 |
| *** psachin has joined #openstack-infra | 17:18 | |
| clarkb | corvus: I think I fixed the python bugs in the process last pass through and we have up to date clouds.yaml content so it should be very close now | 17:18 |
| fungi | i think my dance card is full, also worried about the remaining unknown-unknowns with launch-node right now leading me down even deeper rabbit holes | 17:19 |
| clarkb | mostly I want to avoid the case where it only works for monty and I because we've worked through whatever bugs and failed to push all the fixes back despite our best efforts :) | 17:19 |
| *** bhavikdbavishi has joined #openstack-infra | 17:21 | |
| *** jamesmcarthur has quit IRC | 17:21 | |
| *** tesseract has quit IRC | 17:21 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/bindep master: Update docs building https://review.openstack.org/599755 | 17:23 |
| *** jcoufal has joined #openstack-infra | 17:23 | |
| *** bobh has joined #openstack-infra | 17:24 | |
| *** psachin has quit IRC | 17:26 | |
| *** pcaruana has quit IRC | 17:27 | |
| corvus | clarkb: what's the basic task? why are we rebuilding the server and does it currentyl exist? | 17:30 |
| *** jcoufal has quit IRC | 17:31 | |
| clarkb | corvus: we are rebuilding the server and it does not exist (inap accidently/mistakenly deleted it). The server's old volume does still exist | 17:31 |
| corvus | clarkb: i don't know anything about volumes on mirrors -- is that documented? | 17:32 |
| clarkb | corvus: a quick scan of system config docs says it is not. We use a volume if flavor sizes are too small to provide an afs cache and a apache cache | 17:33 |
| fungi | i think it's not documented particularly because not all our mirrors need the extra space | 17:33 |
| EmilienM | one of the proxies (the cbs.centos) is out to date, what can we do? see: https://cbs.centos.org/repos/virt7-container-common-candidate/x86_64/os/Packages/atomic-registries-1.22.1-25.git5a342e3.el7.x86_64.rpm and http://mirror.bhs1.ovh.openstack.org:8080/cbs.centos/virt7-container-common-candidate/x86_64/os/Packages/atomic-registries-1.22.1-23.git5a342e3.el7.x86_64.rpm | 17:34 |
| EmilienM | mwhahaha: ^ | 17:34 |
| fungi | we started adding volumes to existing mirrors as our data storage needs across them grew and outstripped available free space on the rootfs of various flavors | 17:34 |
| corvus | clarkb, fungi: where should i mount it? | 17:34 |
| *** e0ne has joined #openstack-infra | 17:34 | |
| clarkb | corvus: looking at mirror.dfw.rax.openstack.org we have /dev/mapper/main-proxycache on /var/cache/apache2 and /dev/mapper/main-afscache on /var/cache/openafs each ~100GB total | 17:35 |
| corvus | okay, i can pattern after that | 17:35 |
| clarkb | the lvs share the same pv according to pvs there | 17:35 |
| *** spartakos has joined #openstack-infra | 17:35 | |
| clarkb | corvus: in this case if hyou want to reuse the same volume you should be able to import the existing lvs into lvm then update fstab? | 17:36 |
| *** prad_ has joined #openstack-infra | 17:36 | |
| fungi | yeah, because we added the afs and apache caches at different times, some mirrors have/had them as separate cinder volumes and others as logical volumes on a single cinder volume | 17:36 |
| fungi | started out with some of them having a volume mounted at /var/cache/openafs so the second volume ended up at /var/cache/apache2 | 17:37 |
| clarkb | EmilienM: http://mirror.bhs1.ovh.openstack.org:8080/cbs.centos/virt7-container-common-candidate/x86_64/os/Packages/atomic-registries-1.22.1-25.git5a342e3.el7.x86_64.rpm is there too since its just a proxy | 17:37 |
| clarkb | EmilienM: is the problem that some index hasn't updated yet? | 17:37 |
| fungi | i've considered just doing a single volume at /var/cache instead, but it can be nice to avoid runaway apache proxy cache starving afs cache and causing bizarre errors | 17:38 |
| *** links has joined #openstack-infra | 17:38 | |
| clarkb | if that is the problem you likely have to wait for the ttl to run out on the cached data so that apache will check if it has changed and refetch it. We have a limit of a day on those ttls iirc | 17:38 |
| *** links has quit IRC | 17:38 | |
| *** jamesmcarthur_ has quit IRC | 17:39 | |
| EmilienM | clarkb: most probably yes | 17:39 |
| corvus | clarkb: ok i'll give it a shot | 17:39 |
| EmilienM | clarkb: ok gotcha | 17:39 |
| EmilienM | can we reduce the ttl? | 17:39 |
| EmilienM | well once a day should be fine | 17:39 |
| clarkb | EmilienM: those values are provided by the source server, we just set a sanity check of a day max on our end | 17:39 |
| clarkb | it may be shorter if the source says so, but cannot be longer | 17:39 |
| *** diablo_rojo has joined #openstack-infra | 17:40 | |
| EmilienM | ack | 17:41 |
| EmilienM | mwhahaha: ^* | 17:41 |
| *** mriedem_away is now known as mriedem | 17:41 | |
| *** auristor has quit IRC | 17:43 | |
| *** auristor has joined #openstack-infra | 17:43 | |
| fungi | looks like the dns fix merged at 16:24z, latest ansible run started at 17:30 | 17:44 |
| fungi | so should be taking effect any time now | 17:44 |
| fungi | previous run started at 16:00 so was already underway when the fix went in | 17:45 |
| corvus | clarkb: do you have any WIP documentation changes related to launching? | 17:46 |
| clarkb | corvus: no it looks like they merged, I think one issue is that I didn't update the docs to assume root but to assume you could run as a normal user since that is what we had done and ianw's change was proposed to fix that | 17:47 |
| corvus | clarkb: so https://git.openstack.org/cgit/openstack-infra/system-config/tree/launch/README but without the sudos? what about the rax dns stuff? | 17:48 |
| *** prad_ has quit IRC | 17:49 | |
| clarkb | oh dns, that is a good question. I punted on that because we don't have python2 on bridge.o.o and the rackdns client doesn't do python3 currently | 17:49 |
| clarkb | corvus: ya you don't need the group stuff at the beginning but will need sudo to run launch node iirc | 17:50 |
| corvus | clarkb: rackdns-venv has a python2 | 17:50 |
| corvus | so i guess that's expected to work | 17:50 |
| *** eharney has quit IRC | 17:51 | |
| clarkb | corvus: I want to say that was just rsynced off of puppetmaster and doesn't work? | 17:51 |
| clarkb | I ended up fiddling with it for a bit then just did it through the web ui because I couldn't figure it out with the current system constraints | 17:51 |
| corvus | oh yeah, it doesn't work | 17:51 |
| corvus | clarkb: okay, so what's the goal here? launch the server, or figure out how to use bridge to launch servers? | 17:52 |
| corvus | because bridge is clearly not ready to launch servers, and doing so may be a very long process | 17:52 |
| corvus | if we just want to launch the server, we should use puppetmaster or something | 17:52 |
| *** prad_ has joined #openstack-infra | 17:52 | |
| clarkb | a little of both? this is already useful as it reminds us that we havne't figured out rackdns on bridge. We can install python2 | 17:53 |
| corvus | i feel like a lot of effort has gone into not installing python2 | 17:53 |
| clarkb | if we install python2 we likely need to rebuild the rackdns venv as python versions will have changed between trusty and xenial and venvs tend to not be portable across those changes | 17:53 |
| corvus | so if the goal is figure out how to launch servers on bridge, i think we need to figure something else out. | 17:53 |
| clarkb | corvus: is edit dns manually by hand a reasonable short term something else? | 17:54 |
| clarkb | (I think our other options are not use rackspace dns or make rackdns run under pytho3n) | 17:54 |
| *** prad_ has quit IRC | 17:54 | |
| corvus | clarkb: using the web ui is not an option in my mind | 17:54 |
| fungi | we probably have our work cut out for us redesigning our dns cut out for us anyway once we pick a winterscale domain and start hosting it on our dns infrastructure | 17:55 |
| corvus | fungi: yes, though i feel confident we can find a way to automate updates :) | 17:55 |
| * fungi apparently had a split thought in the middle of writing that not-sentence, but you get my meaning | 17:55 | |
| fungi | corvus: agreed. there are protocols with words like that in them, even | 17:56 |
| *** jamesmcarthur has joined #openstack-infra | 17:56 | |
| fungi | not that we necessarily want to use them, but anyway | 17:56 |
| clarkb | https://github.com/linickx/rsdns is potentially another option | 17:56 |
| corvus | clarkb: if you want me to try to figure out how to use rax dns on bridge, i can work on that | 17:57 |
| clarkb | seems to only require a handful of standardish linux/gnu tools | 17:57 |
| corvus | i was mostly trying to figure out if we wanted to get the server up today to get those 200 nodes, or we wanted to leave it for next week after we figure all this out | 17:57 |
| *** olivierb has joined #openstack-infra | 17:57 | |
| clarkb | I think both things would be nice :) zuul does have a fairly large backlog, but we are also likely heading into a quiet period with the ptg happening | 17:58 |
| clarkb | if we had to prioritize, fixing the systems to launch nodes on bridge.o.o is probably more important and will be most useful during the ptg | 17:58 |
| corvus | ok, will do | 17:58 |
| *** electrofelix has quit IRC | 17:59 | |
| corvus | um. how do i make a virtualenv? | 18:00 |
| fungi | heh, i had the same question | 18:00 |
| clarkb | corvus: I did document that https://git.openstack.org/cgit/openstack-infra/system-config/tree/launch/README#n19 | 18:00 |
| zxiiro | anyone have permissions for JJB ReadTheDocs? none of the current maintainers have access so we cannot do the action required http://lists.openstack.org/pipermail/openstack-dev/2018-August/132836.html | 18:00 |
| fungi | i tried `python3 -m venv foo` and it told me i don't have the necessary additional packages installed because ubuntu apparently likes to strip ensure-pip out of their interpreter package | 18:01 |
| clarkb | fungi: see link above, that did work for me | 18:01 |
| *** e0ne has quit IRC | 18:01 | |
| clarkb | zxiiro: hrm let me see if I can find out | 18:01 |
| corvus | zxiiro: does rtd say who the owners are? | 18:02 |
| clarkb | corvus: zxiiro it does, hashar and zaro | 18:02 |
| *** pcaruana has joined #openstack-infra | 18:02 | |
| clarkb | zxiiro: maybe send them email and feel free to cc me if you like, but I think you can ask them to add you as maintainer | 18:02 |
| zxiiro | Yeah | 18:02 |
| clarkb | (I would +1 that) | 18:02 |
| hashar | hi. I will be around in roughly 1 hour and a half | 18:02 |
| zxiiro | ok i'll dig up their emails from gerrit i guess | 18:02 |
| *** links has joined #openstack-infra | 18:02 | |
| fungi | clarkb: wow, okay so we explicitly install the python3-virtualenv deb ubuntu provides, i guess? | 18:03 |
| hashar | will be happy to add more poeple to rtd as needed (if I can do that) | 18:03 |
| zxiiro | hashar: oh great, you're here. sounds good to me | 18:03 |
| clarkb | hashar: hello can you add zxiiro as maintainer on https://readthedocs.org/projects/jenkins-job-builder/ ? | 18:03 |
| * hashar tries | 18:03 | |
| clarkb | fungi: yup and that doesn't install a command in a path so you have to run the .py directly | 18:03 |
| fungi | clarkb: i guess we could install the python3-venv package too/instead | 18:03 |
| fungi | which would allow `python3 -m venv ...` to work | 18:04 |
| clarkb | corvus: fungi reading rsnds the shell scripts are pretty straightforward if we want to give that a go. Not a spaghetti mess from my reading so far | 18:04 |
| *** openstackgerrit has quit IRC | 18:05 | |
| hashar | zxiiro: I have added https://readthedocs.org/profiles/zxiiro/ :) | 18:05 |
| clarkb | not quite as readable as python probably since it is large curl commands with fun tr involved | 18:05 |
| hashar | zxiiro: feel free to add more maintainers as may fit. I have not been active on openstack projects for quite a while unfortunately :\\ | 18:06 |
| zxiiro | hashar: can confirm. Thanks! | 18:06 |
| zxiiro | hashar: i'll add the other folks | 18:06 |
| *** e0ne has joined #openstack-infra | 18:06 | |
| hashar | thank you !! | 18:06 |
| *** olivierb has quit IRC | 18:06 | |
| clarkb | hashar: thanks | 18:06 |
| fungi | corvus: clarkb: pabelanger: looks like the adns1 iptables fix was applied in the last minute or two, and zuul-ci.org records are resolving for me normally again | 18:08 |
| *** e0ne has quit IRC | 18:08 | |
| corvus | fungi: me too, thx | 18:08 |
| clarkb | fungi: I confirm the new iptables rules in place | 18:08 |
| clarkb | as well as A record resoultion for zuul-ci.org | 18:08 |
| *** openstackgerrit has joined #openstack-infra | 18:10 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Move coverage jobs to non-voting gate jobs https://review.openstack.org/600495 | 18:10 |
| clarkb | rsdns lacks a license :/ | 18:10 |
| *** links has quit IRC | 18:11 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Move coverage jobs to non-voting gate jobs https://review.openstack.org/432836 | 18:11 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Move coverage jobs to non-voting gate jobs https://review.openstack.org/432836 | 18:13 |
| AJaeger | ianw: I updated your change ^ | 18:13 |
| *** bharath_t has joined #openstack-infra | 18:13 | |
| *** gfidente has joined #openstack-infra | 18:14 | |
| bharath_t | can you please review https://review.openstack.org/#/c/599709/ ? | 18:15 |
| bharath_t | And also please let me know the next steps for creating new project | 18:15 |
| *** spartakos has quit IRC | 18:15 | |
| clarkb | bharath_t: that will create 4 repositories for you whose only contents are the .gitreview configuration file to talk to our gerrit server. Once created you can clone them, make changes and push them back | 18:17 |
| clarkb | bharath_t: you will want to request that you get added to the initial groups as well so that you can approve cahnges | 18:17 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Move coverage jobs to non-voting gate jobs https://review.openstack.org/432836 | 18:17 |
| fungi | bharath_t: the next steps are also outlined in https://docs.openstack.org/infra/manual/creators.html | 18:18 |
| *** melwitt is now known as melwitt_awaysick | 18:18 | |
| AJaeger | bharath_t: I found two small problems - could you update quickly and then I'll give my +2? | 18:19 |
| bharath_t | clarkb, we need to raise separate request to add me to that group? | 18:21 |
| bharath_t | fungi, sure :) will check the guide | 18:21 |
| bharath_t | AJaeger, will update it now | 18:22 |
| AJaeger | bharath_t: yes, you need to ask here to get added, this is a manual task - as explained in the guide | 18:22 |
| openstackgerrit | bharaththiruveedula proposed openstack-infra/project-config master: Add new project Gyan https://review.openstack.org/599709 | 18:26 |
| *** jaosorior has quit IRC | 18:27 | |
| *** ccamacho has quit IRC | 18:28 | |
| AJaeger | bharath_t: LGTM - clarkb want to +2A? ^ | 18:28 |
| clarkb | ya I'll take another look | 18:29 |
| corvus | clarkb: i think python3ifying rackdns is feasible. i'm going to do that to minimize the delta. i'd rather save a full overhaul for when we have our new dns system | 18:29 |
| *** bharath_t has quit IRC | 18:29 | |
| clarkb | corvus: ok | 18:30 |
| clarkb | corvus: assuming licenses allow us, we may want to consider making PRs upstream or forking it on our gerrit | 18:30 |
| clarkb | corvus: just so that the delta isn't stuck on bridge.o.o | 18:30 |
| corvus | clarkb: i'd rather just continue to pretend it doesn't exist for a few more months and then do a real thing | 18:31 |
| *** bharath_t has joined #openstack-infra | 18:31 | |
| bharath_t | AJaeger, okay | 18:31 |
| AJaeger | bharath_t: our CI system is really busy, might take two hours or so until this merges (assuming it passes tests) | 18:33 |
| bharath_t | AJaeger, sure | 18:34 |
| bharath_t | AJaeger, few questions ... as you said to add people to the group, we need to request manually. How can I see the groups members, and if we need to add new members to the group, we follow the same procedure | 18:37 |
| *** Swami has joined #openstack-infra | 18:37 | |
| AJaeger | bharath_t: all done via gerrit, find your project - e.g. https://review.openstack.org/#/admin/projects/openstack-dev/pbr and then click on access and then on the groups | 18:38 |
| AJaeger | bharath_t: we will add you as initial member and then it's self-maintenance | 18:39 |
| *** jamesmcarthur has quit IRC | 18:40 | |
| *** akhila has quit IRC | 18:40 | |
| kmalloc | clarkb: did we remove the forward to #openstack-unregistered recently or something similar? I noticed a change in channel modes in -keystone and checking that it was intended (aka, spam flood has died off?) | 18:42 |
| kmalloc | (or anyone else who knows) | 18:43 |
| clarkb | kmalloc: yes | 18:43 |
| kmalloc | clarkb: cool. :) | 18:43 |
| bharath_t | AJaeger, got it, thank you | 18:43 |
| clarkb | kmalloc: https://review.openstack.org/#/c/600251/ was the change to do it. We ran this channel for two days without spam so figured we would extend ti globally | 18:43 |
| clarkb | kmalloc: if we start seeing spam again we can revert | 18:43 |
| AJaeger | config-core, https://review.openstack.org/597590 and https://review.openstack.org/597582 are two repos that have migrated jobs now as part of python3-first, please +2A | 18:47 |
| *** dhajare has quit IRC | 18:48 | |
| kmalloc | clarkb: yeah i'm not worried, i'm pleased the spam stopped | 18:49 |
| *** jamesmcarthur has joined #openstack-infra | 18:50 | |
| *** bhavikdbavishi has quit IRC | 18:51 | |
| *** prad_ has joined #openstack-infra | 18:56 | |
| *** prad_ has quit IRC | 18:56 | |
| *** camunoz has quit IRC | 18:57 | |
| *** graphene has quit IRC | 18:57 | |
| *** trown|lunch is now known as trown | 18:57 | |
| *** graphene has joined #openstack-infra | 18:59 | |
| *** AJaeger_ has joined #openstack-infra | 19:00 | |
| openstackgerrit | Kendall Nelson proposed openstack-infra/puppet-ptgbot master: Unschedule FC-SIG https://review.openstack.org/600516 | 19:00 |
| *** pblaho has joined #openstack-infra | 19:00 | |
| *** AJaeger has quit IRC | 19:02 | |
| *** AJaeger_ is now known as AJaeger | 19:03 | |
| *** e0ne has joined #openstack-infra | 19:05 | |
| *** eharney has joined #openstack-infra | 19:06 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/devstack-gate master: Update grenade settings for stable/rocky https://review.openstack.org/591594 | 19:09 |
| *** olivierb has joined #openstack-infra | 19:09 | |
| *** camunoz has joined #openstack-infra | 19:12 | |
| openstackgerrit | Kendall Nelson proposed openstack/ptgbot master: Update unbook documentation https://review.openstack.org/600520 | 19:15 |
| clarkb | cloud launcher is running now fyi, but failing. Something about a key hash not matching so it won't update (have to delete first) | 19:16 |
| clarkb | I'll dig into that after lunch | 19:16 |
| openstackgerrit | Merged openstack-infra/system-config master: Revert "Fix ansible forks env variable" https://review.openstack.org/600471 | 19:17 |
| *** jtomasek has quit IRC | 19:20 | |
| AJaeger | dhellmann: we need to merge https://review.openstack.org/#/c/597598/ first to finish storlets - ocata change is not ready for py35, see https://review.openstack.org/#/c/596183 - could you un-WIP 597598, please? | 19:23 |
| dhellmann | AJaeger : done | 19:24 |
| AJaeger | thanks | 19:24 |
| AJaeger | config-core, https://review.openstack.org/#/c/597598/ https://review.openstack.org/597590 and https://review.openstack.org/597582 are repos that have migrated jobs now as part of python3-first, please +2A | 19:25 |
| *** spartakos has joined #openstack-infra | 19:30 | |
| *** bharath_t has quit IRC | 19:31 | |
| corvus | AJaeger, timburke: regarding 591528 -- it looks like the idea is that we want to run (1) swift's functional tests with a git checkout of swiftclient, and (2) swiftclient's functional tests with a git checkout of swiftclient. both of those jobs inherit from the basic swift functional job which normally uses released swiftclient and runs swift's functional test. is all that correct? | 19:32 |
| *** bharath_t has joined #openstack-infra | 19:33 | |
| AJaeger | corvus: yes that's my understanding | 19:33 |
| timburke | yep | 19:33 |
| *** pblaho has quit IRC | 19:33 | |
| *** hamzy has quit IRC | 19:34 | |
| corvus | AJaeger, timburke: i think the change as written is the best approach. you can override both required-projects and vars within the project stanza (just like 'voting' is on the job that's removed). but you can't do that twice for the same job with different values. really these new jobs are doing new things compared to the parent, so they merit their own names, even just for the sanity of folks | 19:35 |
| corvus | looking at them (so you know what the job does). | 19:35 |
| *** pcaruana has quit IRC | 19:35 | |
| timburke | and they should both be using tip-of-$branch swift and keystone, and the functional tests (whether swift's or swiftclient's) should be run once with tempauth and once with keystone | 19:35 |
| *** olivierb has quit IRC | 19:35 | |
| corvus | AJaeger, timburke: on that note, i do suggest adding a 'description:' field to them so you can communicate that to folks looking at them. it took me a few minutes of cross-referencing to determine the intent of the jobs. | 19:36 |
| AJaeger | corvus: I just wonder whether the first one is needed - do we need to explicitely add "required-projects" for swiftclient if this is run on swiftclient | 19:36 |
| *** Emine has quit IRC | 19:37 | |
| corvus | AJaeger: oh i see. 1 sec. | 19:37 |
| AJaeger | corvus: or could we use the parent as is directly? | 19:37 |
| *** Emine has joined #openstack-infra | 19:38 | |
| timburke | as i recall, i saw the parent job pass on patchset 1 of https://review.openstack.org/#/c/591530/ (which should have caused everything except docs and pep8 to fail) | 19:40 |
| *** camunoz has quit IRC | 19:43 | |
| corvus | AJaeger: there is a distinction. devstack will add a project to LIBS_FROM_GIT if it is in required-projects *or* it's the project under test. so a similar job using devstack shouldn't need to make a new job for that. tox-siblings will install sibling packages only if they are in required-projects, not if they are the project under test. of course, typically, the project under test is always | 19:44 |
| corvus | installed from source in a tox job, so it's effectively the same thing. the only difference would be if some job used tox but did not install the project under test, yet did use tox-siblings. | 19:44 |
| AJaeger | timburke: could you add descriptions to the jobs in change 591528, please? | 19:44 |
| corvus | AJaeger, timburke: i don't know how swift-dsvm-functional works, so i don't know whether it should be expected to make a difference or not. | 19:45 |
| AJaeger | corvus: swift-dsvm-functional has devstack-minimal as parent - instead of devstack. So, that might be the difference here.... | 19:45 |
| timburke | AJaeger: sure | 19:46 |
| corvus | AJaeger: i think it still does the same libsfromgit thing | 19:46 |
| corvus | AJaeger: in which case, i *think* the first job isn't necessary. | 19:47 |
| timburke | corvus: it *does* use tox, and it uses it inside of *swift* while testing swiftclient. i think that might be the trouble. maybe the definition in swift should include swiftclient as a required-project... | 19:47 |
| AJaeger | corvus: checking devstack, I agree it should not be needed. | 19:48 |
| AJaeger | corvus, timburke, let me write a DNM test change to check... | 19:48 |
| johnsom | Now that we are using a bunch of templates for py27, etc. can I set a nodeset on a template? I want to kick the tires on ubuntu-bionic nodes, but it's not clear to me I can specify a nodeset for the template jobs. | 19:50 |
| corvus | timburke: hrm, yeah, if the tox inside the functional job is pulling in its own dependencies outside of what devstack set up (seems quite likely), then that might be it. | 19:51 |
| corvus | timburke, AJaeger: so devstack would be installing swiftclient from source, but without swiftclient being in required-projects, tox-siblings (which is being run because i see that it's using the tox role) won't be installing swiftclient from source within the tox env | 19:52 |
| AJaeger | corvus: AH! | 19:52 |
| corvus | timburke, AJaeger: it is exactly the 'run tox in a repo other than the repo under test' scenario i outlined above | 19:52 |
| timburke | aaaand that's why i hate legacy-swift-dsvm-functional :-( | 19:53 |
| timburke | (inside of swiftclient, that is) | 19:53 |
| corvus | timburke: as for adding swiftclient to the base job -- that's an option, but it depends on whether you always want to test swiftclient from source. many projects want to test against released libs as well. so some run two jobs: one that tests against released libs, and a second that tests the lib from git. | 19:54 |
| AJaeger | timburke: all those legacy jobs are legacy and need to be replaced... | 19:54 |
| AJaeger | corvus: I've pushed https://review.openstack.org/600540 - I really want to double check... | 19:55 |
| corvus | timburke, AJaeger: so aside from any consideration of changing swift-dsvm-functional, i think the current patchset is a good solution. and probably warrants a note why the project-under-test is being added to required-projects. :) | 19:55 |
| AJaeger | timburke: will you update - or shall I? | 19:55 |
| timburke | corvus: yeah, that's part of why i took the path i did -- i see the value in using the released lib, but i *definitely* want to be making sure that some swiftclient release doesn't break swift's gate | 19:56 |
| *** anteaya has joined #openstack-infra | 19:56 | |
| corvus | AJaeger: good idea. i mean, i've spent a whole 10 minutes digging into a pretty complicated combination of things and making a bunch of assumptions. i could easily be wrong about something. :) | 19:56 |
| timburke | AJaeger: i can update it, but it might be a bit. i'm buried in other things at the moment | 19:56 |
| AJaeger | timburke: ok, I'll do it quickly - if you disagree, please take over again ;) | 19:57 |
| AJaeger | corvus: thanks a lot for all the digging and explanation! | 19:57 |
| timburke | sounds like a plan. thanks AJaeger! | 19:57 |
| *** johanssone has joined #openstack-infra | 19:59 | |
| *** e0ne has quit IRC | 20:02 | |
| corvus | AJaeger, timburke: i left a summary comment on 591528. | 20:02 |
| pabelanger | corvus: clarkb: fungi: any concerns about pip installing docker onto zuul-executor? Context is have a jobs that would like to use docker modules in ansible, but we need them on executor side: https://review.openstack.org/600456/ | 20:05 |
| AJaeger | corvus, thanks - I've updated the change with description and some comments | 20:07 |
| clarkb | pabelanger: I think we need a better long term plan for how to manage that but in the short term that is probably fine to add the one offs like this as we run into them | 20:08 |
| clarkb | openstackci-vexxhost-mtl1 seems to be the problem with run_cloud_launcher. I'm going to run it against only that cloud region manually and see if I can sort it out (may end up deleting the existing infra-root-keys and let it redo it | 20:09 |
| fungi | pabelanger: "use docker modules in ansible..." to do what? that change doesn't really outline the use case | 20:10 |
| clarkb | fungi: use the docker module in ansible to manage containers instead of execing docker cli | 20:10 |
| fungi | containers on server instances, or somewhere else? | 20:10 |
| corvus | what jobs? | 20:10 |
| pabelanger | sure, I can update which jobs. | 20:11 |
| * AJaeger waves good night | 20:11 | |
| clarkb | AJaeger: good night | 20:11 |
| fungi | have a good evening AJaeger! | 20:11 |
| AJaeger | thanks | 20:11 |
| corvus | i'm confused because this would only affect stuff running on the executor | 20:11 |
| corvus | and i feel like i've missed a conversation if we're talking about trusted jobs running docker | 20:12 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-zuul master: Add docker dependency for ansible-playbook https://review.openstack.org/600456 | 20:13 |
| pabelanger | ^ updated commit with example job and log | 20:13 |
| pabelanger | but yes, not running docker on the executor side, just the client for use on remote nodes | 20:14 |
| fungi | pabelanger: why can't the client be executed _on_ the remote notes? | 20:14 |
| pabelanger | fungi: I don't know, https://docs.ansible.com/ansible/latest/modules/docker_image_module.html#requirements is more info about dependency | 20:15 |
| pabelanger | I can look at module and make sure not an issue with documentation | 20:15 |
| pabelanger | or, maybe I am reading this wrong | 20:16 |
| pabelanger | host that executes this module, could be the nodepool node | 20:16 |
| pabelanger | I can test that easyily | 20:16 |
| pabelanger | easily* | 20:16 |
| corvus | yes that's what it should be | 20:16 |
| corvus | you'll need to install the requirements on the remote node | 20:16 |
| pabelanger | okay, in that case, I can abandon puppet-zuul patch | 20:18 |
| clarkb | corvus: pabelanger oh so ansible isn't vendoring over those libs from the local node to the remote? for some reason I thought it did that | 20:19 |
| *** tosky has joined #openstack-infra | 20:19 | |
| pabelanger | testing, but I thought it was doing that too | 20:19 |
| pabelanger | but, believe corvus is right | 20:20 |
| clarkb | it appears that the two accounts in vexxhost mtl1 exhibit the sshe keys error but toehr clouds don't with cloud launcher | 20:20 |
| clarkb | I'm going to see if I can find out more info about the keys in that region | 20:20 |
| mnaser | clarkb: anything our side? | 20:22 |
| corvus | clarkb, pabelanger: it does not do that, and it's a very good thing too. otherwise the requirements on the controller node would be immense and unresolvable (how could we have the necessary dependencies for all possible remote os versions). this is much easier to reason about. need to run docker on remote node? install docker on remote node. | 20:22 |
| clarkb | mnaser: no I think it has more to do with how the ansible modules won't update an existing key by deleting it and creating it again | 20:22 |
| clarkb | the valeus do differ | 20:22 |
| clarkb | infra-root I am going to delete the infra-root-keys value in vexxhost-mtl1 manually then rerun cloud launcher against that region to make sure it is happy now | 20:23 |
| *** caphrim007 has quit IRC | 20:23 | |
| clarkb | then automated cron will rerun at top of hour and we can check it is happy still | 20:23 |
| clarkb | mnaser: thank you for checking though | 20:27 |
| *** jamesmcarthur has quit IRC | 20:27 | |
| pabelanger | clarkb: +1 to delete infra-root-keys | 20:28 |
| clarkb | I've gone ahead and done that, both accounts in that region should have up to date infra-root-keys | 20:28 |
| clarkb | I'll check in half an hour if the full run of cloud launcher succeeds now | 20:28 |
| pabelanger | clarkb: https://github.com/ansible/ansible/pull/30565 likely is our fix for keys | 20:29 |
| clarkb | pabelanger: we'll have to update the cloud launcher playbooks too but ya that looks like what we want. Also there are a bunch of warnings emitted about our use of jinja in when states we should clean up | 20:30 |
| pabelanger | ack | 20:30 |
| clarkb | hopefully that is one more item checked off the migration list though | 20:31 |
| *** nicolasbock has quit IRC | 20:34 | |
| clarkb | I'm going to go ahead and look at fixing the warnings while they are top of mind | 20:34 |
| corvus | clarkb: mirrors need to be xenial? | 20:37 |
| openstackgerrit | Merged openstack-infra/project-config master: Add new project Gyan https://review.openstack.org/599709 | 20:38 |
| clarkb | corvus: yes, they are puppeted currently | 20:38 |
| corvus | clarkb: i should be creating an 8g xenial node in openstackci-inap region mtl01, right? | 20:40 |
| clarkb | corvus: that sounds right to me. We use the test node flavor in a few clouds because they only have that flavor. | 20:41 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Log output and exception when traceroute fails https://review.openstack.org/576351 | 20:41 |
| corvus | mgagne: i just got an error creating that^. uuid 759283c0-c1ce-4c4f-90bd-9d5c0742af10 {u'message': u'No valid host was found. There are not enough hosts available.', u'code': 500, u'created': u'2018-09-06T20:38:10Z'} | 20:41 |
| clarkb | corvus: and that seems to be plenty big as far as resources go other than disk in most cases | 20:41 |
| mgagne | corvus: was it for the mirror instance? | 20:41 |
| corvus | mgagne: yes | 20:42 |
| *** tonytan4ever has quit IRC | 20:43 | |
| *** kgiusti has left #openstack-infra | 20:44 | |
| mgagne | corvus: I freed one node for the other project/tenant. | 20:44 |
| mgagne | all nodes are part of an aggregate limited to the nodepool tenant and none were left for the other projects. | 20:44 |
| mgagne | should be ok now | 20:44 |
| corvus | mgagne: thanks, retrying | 20:44 |
| corvus | clarkb: i see 4 volumes. i assume i should attach mirror.mtl01.inap.openstack.org/main01 and main02. verify it looks okay, and then delete mirror.mtl01.internap.openstack.org/main01 and main02. sound right? | 20:45 |
| corvus | clarkb: (note inap/internap names) | 20:45 |
| *** akhila has joined #openstack-infra | 20:46 | |
| *** ansmith has quit IRC | 20:47 | |
| mgagne | b5beb67b-e231-4f45-8aa2-1371f854553a and ae5fd3fb-bcdd-45bf-9de9-9eeafb471da9 were used according to deleted rows in block_device_mapping | 20:48 |
| clarkb | corvus: yes inap should be the new volume and internap volumes likely can go away | 20:48 |
| mgagne | internap volumes were used with instance UUID b0953052-6dd6-48e3-b333-90a11e276edd which got deleted on 2017-08-17 16:08:43 | 20:49 |
| clarkb | TIL ansible with_items flattened lists so if you move to loop well you have to flatten to maintain compat | 20:49 |
| openstackgerrit | Clark Boylan proposed openstack/ansible-role-cloud-launcher master: Don't use deprecated when jinja2 syntax https://review.openstack.org/600551 | 20:53 |
| clarkb | pabelanger: mordred ^ I haven't tested that however reading docs I think the fix is as simple as that | 20:53 |
| *** nicolasbock has joined #openstack-infra | 20:54 | |
| *** anteaya has quit IRC | 20:55 | |
| *** dims has quit IRC | 20:55 | |
| *** rfolco has quit IRC | 21:00 | |
| *** rfolco has joined #openstack-infra | 21:00 | |
| *** rh-jelabarre has quit IRC | 21:02 | |
| mnaser | clarkb: no problem :) | 21:02 |
| mnaser | fwiw | 21:03 |
| mnaser | cloud launcher ci is broken, i tried to do some stuff but yeah | 21:03 |
| *** kjackal has quit IRC | 21:03 | |
| clarkb | mnaser: hrm is that going to be the new item on the thread I am pulling on? | 21:04 |
| *** kjackal_ has quit IRC | 21:04 | |
| mnaser | clarkb: yup -- https://review.openstack.org/#/q/project:openstack/ansible-role-cloud-launcher | 21:04 |
| mnaser | so os_router has an issue which we fixed upstream | 21:04 |
| mnaser | but i kinda ran out of steam and time | 21:05 |
| *** bobh has quit IRC | 21:05 | |
| *** bharath_t has quit IRC | 21:05 | |
| mnaser | https://github.com/ansible/ansible/pull/44448 | 21:05 |
| mnaser | just merged to 2.6 | 21:05 |
| clarkb | mnaser: running the cloud launcher ci against ansible 2.6 should fix it then? | 21:06 |
| mnaser | clarkb: i dont know if that fix is released or not yet as it was only 2 days ago | 21:06 |
| clarkb | ah | 21:06 |
| mnaser | clarkb: but there was some other ci changes too like actually deploying devstack to run it | 21:06 |
| clarkb | so 2.6.next | 21:06 |
| mnaser | in the bottom of the stack | 21:06 |
| mnaser | yeah | 21:06 |
| mnaser | i commented out the os_router stuff but it failed for some other stuff and yeah | 21:06 |
| clarkb | mnaser: is the floating ip failue the os_router problem? have an example of the failures after commenting things out? | 21:10 |
| mnaser | clarkb: nope, after commenting it out i'm not sure why that timeout happened | 21:11 |
| mnaser | it could really just be a transient thing | 21:11 |
| mnaser | OR because there is no router it cant allocate a floating ip | 21:11 |
| mnaser | so maybe that has to be commented out too | 21:11 |
| *** dims has joined #openstack-infra | 21:11 | |
| clarkb | ya routers are where fips are terminated I think those may be related | 21:11 |
| *** trown is now known as trown|outtypewww | 21:12 | |
| clarkb | the next ansible run should run with the explicit -f20 and we can compare runtimes against that baseline | 21:17 |
| corvus | mgagne: https://identity.api.cloud.iweb.com isn't responding for me | 21:23 |
| clarkb | new cloud launcher error fatal: [localhost]: FAILED! => {"changed": false, "msg": "Multiple matches found for default"} when setting up default security groups in packethost (I'll debug this one too) | 21:24 |
| mgagne | corvus: ok, looks like some people are fixing stuff atm, it should be back now. | 21:25 |
| mgagne | corvus: same guy as last time :D | 21:26 |
| ianw | hey, i see we're on the internap issue, lmn if i can help | 21:27 |
| clarkb | there are indeed many default security groups visible to openstackci-packethost. I believe the reason is openstackci-packethost is an admin account | 21:31 |
| mgagne | corvus: are you seeing problems on your side with API endpoints? | 21:33 |
| mgagne | I'm getting conflicted reports on the system status | 21:33 |
| *** eernst_ has joined #openstack-infra | 21:34 | |
| *** eernst has quit IRC | 21:34 | |
| clarkb | mnaser: ^ have you used cloud launcher with admin accounts? any idea if there is a workaround for this problem? | 21:34 |
| mnaser | clarkb: i have not used cloud launcher with admin accounts because the idea terrifies me :D | 21:34 |
| mnaser | but if you have an admin account, maybe you can use that to make a non-admin one and give it member role for the project and call it a day :) | 21:34 |
| mnaser | because the only other workaround i can imagine is referencing by uuid | 21:35 |
| corvus | mgagne: yes, compute endpoint now | 21:35 |
| *** sshnaidm|off has quit IRC | 21:36 | |
| corvus | mgagne: host='compute.api.mtl01.cloud.iweb.com' | 21:36 |
| mgagne | ok, people on working on it right now. | 21:37 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Don't set security groups in openstackci-packethost https://review.openstack.org/600557 | 21:38 |
| clarkb | mnaser: its kind of annoying that the default api behavior for an admin account is to show everything rather than account specific info | 21:38 |
| clarkb | I'm guessing we can't change that behavior easily at this point so meh | 21:38 |
| mnaser | clarkb: top that -- this is neutron specific behaviour | 21:39 |
| mgagne | unless you introduce the all_tenants=1 parameter =) | 21:39 |
| mnaser | nova doesn't do that | 21:39 |
| mnaser | nope, neutron always sends it all out | 21:39 |
| clarkb | ugh | 21:39 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Update launch README for bridge https://review.openstack.org/600558 | 21:39 |
| clarkb | corvus: did you get rackdns speaking python3? | 21:40 |
| *** ijw has joined #openstack-infra | 21:43 | |
| *** gfidente has quit IRC | 21:44 | |
| *** bobh has joined #openstack-infra | 21:45 | |
| mgagne | corvus: so people are telling me it's back online, can you confirm? | 21:45 |
| corvus | clarkb: yes | 21:46 |
| clarkb | neat | 21:46 |
| corvus | mgagne: yes, server list works for me now | 21:46 |
| corvus | clarkb: 2to3 did most of the work | 21:46 |
| mgagne | corvus: good, sorry for the inconveniences. I don't have much excuses. :-/ | 21:47 |
| corvus | mgagne: heh, that's okay. i ran out of those a long time ago too. | 21:47 |
| openstackgerrit | Merged openstack-infra/system-config master: Move project config in repo https://review.openstack.org/600475 | 21:49 |
| *** ijw has quit IRC | 21:50 | |
| *** eernst_ has quit IRC | 21:50 | |
| *** eernst has joined #openstack-infra | 21:50 | |
| *** bobh has quit IRC | 21:50 | |
| corvus | i think ansible is doing its thing on the new mirror now | 21:52 |
| clarkb | corvus: thsi run started with -f20 | 21:52 |
| clarkb | `grep 'Playbook run took' /var/log/ansible/ansible.log | tail -50` is my rough comparison grep | 21:53 |
| *** boden has quit IRC | 21:54 | |
| corvus | clarkb: you can also use ianw's script in /var/log/ansible to see the whole time (should be ~45 mins if the theory is correct) | 21:54 |
| corvus | once it's finished | 21:55 |
| clarkb | ya the above grep gives you intermediate playbook results | 21:58 |
| clarkb | the really short playbooks are shorted so far | 21:58 |
| clarkb | but I'm not sure their run time is long enough to be reliable | 21:58 |
| *** ijw has joined #openstack-infra | 21:58 | |
| *** dave-mccowan has quit IRC | 22:00 | |
| clarkb | corvus: I think -f20 took base.yaml from ~32-33 minutes down to 27 minutes | 22:01 |
| clarkb | seems to confirm the effort to use 50 threads was not working | 22:01 |
| *** mriedem is now known as mriedem_away | 22:02 | |
| *** mriedem_away has quit IRC | 22:02 | |
| fungi | default value if no -f specified is 10, right? | 22:06 |
| corvus | fungi: 5 | 22:06 |
| fungi | ahh | 22:06 |
| corvus | next we should do -f 50 | 22:06 |
| fungi | so possible most of those playbooks were actually running the equivalent of -f5 because ~handwave~ envvar not being seen/used | 22:07 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Run puppet apply test serially on centos https://review.openstack.org/600270 | 22:07 |
| clarkb | ianw: ^ fyi updated based on teh suggestion to serialize only centos | 22:07 |
| clarkb | fungi: yup | 22:08 |
| corvus | yeah, that's the theory. but we should see about the whole run. the difference was very noticeable (45->65 mins) | 22:08 |
| ianw | clarkb: yeah, i think that's probably the way to go, i don't think we want to expend a lot of effort debugging. i also started to try and clean up the jumbled mess you get running all those playbooks in parallel with https://review.openstack.org/#/c/600331/ | 22:11 |
| *** ijw has quit IRC | 22:11 | |
| ianw | something is, however, being rather persistent at evading my shell redirects | 22:11 |
| ianw | very much complicated by running it under xargs | 22:12 |
| *** ijw has joined #openstack-infra | 22:12 | |
| clarkb | ianw: we do get different stdout for each apply written to disk iirc | 22:13 |
| clarkb | http://logs.openstack.org/70/600270/1/check/infra-puppet-apply-3-centos-7/52847d7/applytest/puppetapplytest26.final.out for example | 22:14 |
| ianw | yeah, but the ansible around the puppet all gets jumbled in console.log | 22:15 |
| clarkb | ah the ansible itself | 22:15 |
| clarkb | which is where we see the race I guess that happens before we run puppet | 22:15 |
| ianw | http://logs.openstack.org/31/600331/4/check/infra-puppet-apply-4-ubuntu-xenial/b05b84d/job-output.txt.gz#_2018-09-06_10_12_07_013789 like all that stuff that is somehow still coming out | 22:16 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Write ansible log to fileout in apply test https://review.openstack.org/600562 | 22:16 |
| corvus | clarkb, ianw: ^ think that might work? | 22:16 |
| *** gema has quit IRC | 22:18 | |
| ianw | could do. turning off "-x" in the script probably worth it for signal:noise ratio too | 22:19 |
| clarkb | that is a unique config per run so should work | 22:19 |
| *** gema has joined #openstack-infra | 22:20 | |
| *** gema has quit IRC | 22:20 | |
| *** gema has joined #openstack-infra | 22:20 | |
| *** ijw has quit IRC | 22:22 | |
| *** dave-mccowan has joined #openstack-infra | 22:23 | |
| *** ijw has joined #openstack-infra | 22:27 | |
| *** rcernin has joined #openstack-infra | 22:29 | |
| *** hashar has quit IRC | 22:29 | |
| *** ijw has quit IRC | 22:33 | |
| *** jamesdenton has quit IRC | 22:34 | |
| openstackgerrit | Thanh Ha (zxiiro) proposed openstack-infra/project-config master: Add JJB rtd webhook id https://review.openstack.org/600569 | 22:35 |
| *** ijw has joined #openstack-infra | 22:36 | |
| clarkb | zxiiro: do you need to add the rtd job as well? | 22:37 |
| zxiiro | clarkb: selective reading, missed that part thanks :) | 22:37 |
| zxiiro | fix that in a bit | 22:37 |
| openstackgerrit | Thanh Ha (zxiiro) proposed openstack-infra/project-config master: Add JJB rtd webhook id https://review.openstack.org/600569 | 22:38 |
| zxiiro | there we go | 22:38 |
| corvus | clarkb, ianw: does mirror01.mtl01.inap.openstack.org look right now? | 22:44 |
| clarkb | corvus: http://mirror01.mtl01.inap.openstack.org/ubuntu/ ya that lgtm | 22:46 |
| clarkb | I think I've found one of the problems with our ansible runtime, we run remote_puppet_afs against groups and hosts matching afs* and that runs with -f 1 now | 22:48 |
| clarkb | problem is mirrors and zuul executors and others all match afs* | 22:48 |
| clarkb | I'm working on a patch now | 22:48 |
| corvus | oof | 22:48 |
| *** ijw has quit IRC | 22:48 | |
| clarkb | corvus: the comment in run_all.sh says we run with -f 1 to ensure we maintain afs quorum. Do you know which hosts we need to run one at a time to maintain quorum? | 22:49 |
| clarkb | corvus: is it afs0X.region.openstack.org and afsdb0X.openstack.org ? | 22:50 |
| corvus | clarkb: yeah. the db servers to keep the ubiq database available, but also we probably don't want to do anything on more than one fileserver at a time, just out of caution. | 22:51 |
| clarkb | thanks | 22:51 |
| *** diablo_rojo has quit IRC | 22:54 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Serialize puppet on afs servers properly https://review.openstack.org/600575 | 22:56 |
| clarkb | corvus: ianw ^ I think I got that right and expect it to speed things up a bit | 22:56 |
| clarkb | (if anyone wants to know how I figured that out I was running ps against ansible while running and basically doing my best to follow it manually while in progress. Noticed that ze04 was being ansibled during remote_puppet_afs which seemed weird and went from there) | 22:56 |
| corvus | clarkb: am i ready to delete the old volumes and put mtl01 back in service? | 22:57 |
| corvus | clarkb: ah. hopefully when we get ara it'll be easier to see what's run where... | 22:57 |
| *** elod has quit IRC | 22:58 | |
| clarkb | corvus: yes I think so | 22:59 |
| clarkb | let me ssh in real quick | 22:59 |
| *** tosky has quit IRC | 22:59 | |
| clarkb | ya new mirror lgtm thanks | 22:59 |
| clarkb | as far as methods of debugging go I'm not sure its the best one :) but reading the log file was becoming difficult | 23:00 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Use git-servers group in remote_puppet_else https://review.openstack.org/600576 | 23:00 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Revert "Revert "Revert "Disable inap-mtl01 provider""" https://review.openstack.org/600577 | 23:00 |
| clarkb | this chagne is a related cleanup, we were inconsistent in our selection of groups | 23:00 |
| clarkb | I don't think that one fixes a production issue though, just a sanity fix | 23:01 |
| *** dave-mccowan has quit IRC | 23:01 | |
| corvus | #status log rebuilt mirror.mtl01.inap.openstack.org and removed unused volumes mirror.mtl01.internap.openstack.org/main02 and mirror.mtl01.internap.openstack.org/main01 | 23:01 |
| openstackstatus | corvus: finished logging | 23:01 |
| clarkb | ianw can you review https://review.openstack.org/600575 as I am sure it intersects with some of your afs work and want to make sure that works for you too (though we should find somethign that works for only serializing the bare minimum set of nodes) | 23:02 |
| corvus | ianw, clarkb, mgagne: should be ready to re-enable inap in https://review.openstack.org/600577 | 23:02 |
| *** pcrews has joined #openstack-infra | 23:02 | |
| clarkb | corvus: I +2d thanks | 23:02 |
| *** slaweq_ has joined #openstack-infra | 23:02 | |
| *** tpsilva has quit IRC | 23:02 | |
| *** slaweq has quit IRC | 23:03 | |
| clarkb | re Ara our ansible.log file contains a lot of data that shouldn't be public I've noticed | 23:03 |
| corvus | clarkb: the afs change looks good; we might make an 'afs-server' group that holds afs and afsdb. but that's borderline bikeshedding right now. | 23:03 |
| clarkb | we should be careful ara doesn't expose those details | 23:04 |
| ianw | corvus: is it 198.72.125.8 mirror.mtl01.inap.openstack.org ? or do i have old dns | 23:04 |
| corvus | clarkb: it does? (aside from the openstacksdk logging?) | 23:04 |
| corvus | ianw: i think that's old. did that come from dns? | 23:04 |
| corvus | i may need to check the web dashboard and delete an old a record | 23:05 |
| clarkb | corvus: ya the openstacksdk stuff | 23:05 |
| ianw | corvus: yep, just tried pinging it from here | 23:05 |
| corvus | clarkb: that shouldn't end up in ara. that also should be removed from the log; i think mordred ran out of time for that before he left. | 23:05 |
| clarkb | corvus: ianw looks like we have an A record instead of a cname at mirror.mtl01.inap.openstack.org | 23:05 |
| clarkb | corvus: gotcha | 23:06 |
| ianw | oh, old fashioned | 23:06 |
| clarkb | corvus: ianw I think we can fix this by replacing that A record with a cname to new 01 servers | 23:06 |
| corvus | clarkb: i will do that | 23:06 |
| corvus | old record had a ttl of 3600; the cname is in place now and should be propagated by 00:10 utc | 23:11 |
| clarkb | I'm going to pop out for a break now, but if we get that afs change in this evening I can help watching it | 23:18 |
| clarkb | *help watch it | 23:18 |
| *** elod has joined #openstack-infra | 23:18 | |
| clarkb | then tomorrow is likely a day of errands for me to help prep family for being out of town for a week | 23:18 |
| *** diablo_rojo has joined #openstack-infra | 23:20 | |
| *** bharath_t has joined #openstack-infra | 23:26 | |
| ianw | "close & reopen your pull request" seems a pretty sucky way to get around an unrelated CI failure | 23:30 |
| bharath_t | AJaeger, regarding https://review.openstack.org/#/c/599709/, can you please me to those groups? | 23:30 |
| clarkb | bharath_t: I'll get them | 23:36 |
| bharath_t | thanks clarkb | 23:37 |
| clarkb | bharath_t: done | 23:38 |
| *** Swami has quit IRC | 23:48 | |
| *** bharath_t has quit IRC | 23:48 | |
| openstackgerrit | Merged openstack-infra/puppet-ptgbot master: Unschedule FC-SIG https://review.openstack.org/600516 | 23:53 |
| *** camunoz has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!