| *** felipemonteiro has joined #openstack-infra | 00:00 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove python-fenixclient template https://review.openstack.org/602032 | 00:00 |
|---|---|---|
| *** dpawlik has quit IRC | 00:01 | |
| *** jamesmcarthur has joined #openstack-infra | 00:02 | |
| *** felipemonteiro has quit IRC | 00:04 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove Nova gerrit patch notices from placement https://review.openstack.org/601816 | 00:04 |
| openstackgerrit | Merged openstack-infra/project-config master: Move Shaker jobs to project repository https://review.openstack.org/602561 | 00:04 |
| *** fresta_ has quit IRC | 00:05 | |
| *** jamesmcarthur has quit IRC | 00:08 | |
| *** openstackgerrit has quit IRC | 00:09 | |
| *** felipemonteiro has joined #openstack-infra | 00:11 | |
| *** jamesmcarthur has joined #openstack-infra | 00:12 | |
| *** felipemonteiro has quit IRC | 00:14 | |
| *** fresta has joined #openstack-infra | 00:24 | |
| *** jamesmcarthur has quit IRC | 00:27 | |
| *** felipemonteiro has joined #openstack-infra | 00:32 | |
| *** felipemonteiro has quit IRC | 00:34 | |
| *** fresta_ has joined #openstack-infra | 00:47 | |
| *** longkb has joined #openstack-infra | 00:47 | |
| *** fresta has quit IRC | 00:50 | |
| *** jamesmcarthur has joined #openstack-infra | 00:53 | |
| *** felipemonteiro has joined #openstack-infra | 00:56 | |
| *** felipemonteiro has quit IRC | 00:57 | |
| *** jamesmcarthur has quit IRC | 00:57 | |
| *** hongbin has joined #openstack-infra | 01:09 | |
| *** jamesmcarthur has joined #openstack-infra | 01:16 | |
| *** bobh has joined #openstack-infra | 01:17 | |
| *** openstackgerrit has joined #openstack-infra | 01:33 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy-devstack-plugin-container-dsvm https://review.openstack.org/601346 | 01:33 |
| ianw | pro tip; no matter how expedient it may seem, running tox tests that end up doing docker magic as root is a bad idea | 01:34 |
| openstackgerrit | Merged openstack-infra/project-config master: remove job settings for dragonflow repositories https://review.openstack.org/597475 | 01:36 |
| *** mrsoul has quit IRC | 01:44 | |
| *** felipemonteiro has joined #openstack-infra | 01:44 | |
| *** haleyb has quit IRC | 01:48 | |
| *** _ari_ has quit IRC | 01:48 | |
| *** jmccrory has quit IRC | 01:54 | |
| *** jmccrory has joined #openstack-infra | 01:57 | |
| *** fresta_ has quit IRC | 02:00 | |
| *** fresta has joined #openstack-infra | 02:02 | |
| *** lbragstad has joined #openstack-infra | 02:07 | |
| *** jmorgan1 has joined #openstack-infra | 02:13 | |
| *** jamesmcarthur has quit IRC | 02:21 | |
| openstackgerrit | Melvin Hillsman proposed openstack-infra/irc-meetings master: Update the UC meeting time https://review.openstack.org/603005 | 02:26 |
| *** felipemonteiro has quit IRC | 02:29 | |
| *** e0ne has joined #openstack-infra | 02:33 | |
| *** lbragstad has quit IRC | 02:36 | |
| *** jamesmcarthur has joined #openstack-infra | 02:39 | |
| *** lbragstad has joined #openstack-infra | 02:41 | |
| *** jamesmcarthur has quit IRC | 02:56 | |
| *** jamesmcarthur has joined #openstack-infra | 02:56 | |
| *** jamesmcarthur has quit IRC | 03:10 | |
| *** jamesmcarthur has joined #openstack-infra | 03:11 | |
| *** vivsoni has joined #openstack-infra | 03:12 | |
| *** e0ne has quit IRC | 03:21 | |
| *** annp has joined #openstack-infra | 03:21 | |
| *** ykarel has joined #openstack-infra | 03:31 | |
| *** udesale has joined #openstack-infra | 03:34 | |
| *** jamesmcarthur has quit IRC | 03:39 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:39 | |
| *** ramishra has joined #openstack-infra | 03:40 | |
| *** jamesmcarthur has joined #openstack-infra | 03:40 | |
| *** shachar has quit IRC | 03:44 | |
| *** oanson has quit IRC | 03:44 | |
| *** jamesmcarthur has quit IRC | 03:47 | |
| *** e0ne has joined #openstack-infra | 03:54 | |
| *** dpawlik has joined #openstack-infra | 03:57 | |
| *** dpawlik has quit IRC | 04:02 | |
| *** udesale has quit IRC | 04:03 | |
| *** udesale has joined #openstack-infra | 04:24 | |
| *** udesale has quit IRC | 04:25 | |
| *** udesale has joined #openstack-infra | 04:25 | |
| *** jamesmcarthur has joined #openstack-infra | 04:34 | |
| *** dtruong has quit IRC | 04:36 | |
| *** bobh has quit IRC | 04:41 | |
| *** e0ne has quit IRC | 04:55 | |
| *** jamesmcarthur has quit IRC | 05:00 | |
| *** snapiri has joined #openstack-infra | 05:10 | |
| *** felipemonteiro has joined #openstack-infra | 05:13 | |
| *** snapiri has quit IRC | 05:18 | |
| *** ykarel has quit IRC | 05:22 | |
| *** jbadiapa has joined #openstack-infra | 05:25 | |
| *** quique|rover|off is now known as quiquell|rover | 05:30 | |
| *** janki has joined #openstack-infra | 05:32 | |
| *** dayou_ has joined #openstack-infra | 05:33 | |
| *** jtomasek has joined #openstack-infra | 05:34 | |
| *** e0ne has joined #openstack-infra | 05:42 | |
| *** ykarel has joined #openstack-infra | 05:42 | |
| *** jtomasek has quit IRC | 05:44 | |
| *** jtomasek has joined #openstack-infra | 05:45 | |
| *** quiquell|rover has quit IRC | 05:56 | |
| *** quiquell has joined #openstack-infra | 05:56 | |
| *** quiquell is now known as quiquell|rover | 05:56 | |
| *** hongbin has quit IRC | 06:00 | |
| *** quiquell|rover is now known as quique|rover | 06:01 | |
| *** quique|rover is now known as quiquell|rover | 06:01 | |
| *** holser|ptg has joined #openstack-infra | 06:03 | |
| *** holser|p_ has joined #openstack-infra | 06:05 | |
| *** holser|ptg has quit IRC | 06:05 | |
| *** dpawlik has joined #openstack-infra | 06:07 | |
| *** felipemonteiro has quit IRC | 06:16 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix unreachable nodes detection https://review.openstack.org/602829 | 06:16 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Also retry the job if a post job failed with unreachable https://review.openstack.org/602830 | 06:16 |
| *** apetrich has joined #openstack-infra | 06:18 | |
| AJaeger | config-core, please review https://review.openstack.org/599760 and https://review.openstack.org/601460 and https://review.openstack.org/601319 | 06:23 |
| *** aojea has joined #openstack-infra | 06:27 | |
| *** quiquell|rover is now known as quique|rover|brb | 06:37 | |
| *** udesale has quit IRC | 06:38 | |
| *** udesale has joined #openstack-infra | 06:40 | |
| *** udesale has quit IRC | 06:45 | |
| *** Seb-Solon has joined #openstack-infra | 06:47 | |
| *** vivsoni has quit IRC | 06:48 | |
| *** e0ne_ has joined #openstack-infra | 06:50 | |
| *** e0ne has quit IRC | 06:53 | |
| *** e0ne_ has quit IRC | 06:54 | |
| egonzalez | Hi, is the epel repository mirror different for each branch? I see jobs in kolla failing to build with missing packages in queens branch and not in rocky or master | 06:59 |
| *** slaweq has joined #openstack-infra | 07:00 | |
| *** olivierb has joined #openstack-infra | 07:01 | |
| *** quique|rover|brb is now known as quiquell|rover | 07:07 | |
| *** rcernin has quit IRC | 07:11 | |
| *** shardy has joined #openstack-infra | 07:15 | |
| *** holser|p_ is now known as holser_ | 07:24 | |
| *** priteau has joined #openstack-infra | 07:30 | |
| *** snapiri has joined #openstack-infra | 07:37 | |
| *** ykarel is now known as ykarel|lunch | 07:45 | |
| *** jpena|off is now known as jpena | 07:50 | |
| *** jpich has joined #openstack-infra | 07:52 | |
| *** slaweq has quit IRC | 08:00 | |
| quiquell|rover | Good morning | 08:01 |
| quiquell|rover | Looks like AFS mirrors are out of sync | 08:01 |
| quiquell|rover | ceph-ansible package for example here http://mirror.dfw.rax.openstack.org/centos/7/storage/x86_64/ceph-luminous/ | 08:01 |
| quiquell|rover | and here http://mirror.centos.org/centos/7/storage/x86_64/ceph-luminous/ | 08:01 |
| *** jamesmcarthur has joined #openstack-infra | 08:02 | |
| quiquell|rover | egonzalez: We have similar issues with sync at mirrors | 08:02 |
| *** florianf has quit IRC | 08:02 | |
| quiquell|rover | AJaeger: ^ Do you know if there is any issues at openstack infra with mirrors ^ ? | 08:03 |
| *** slaweq has joined #openstack-infra | 08:03 | |
| *** vivsoni has joined #openstack-infra | 08:03 | |
| *** vivsoni_ has joined #openstack-infra | 08:03 | |
| *** florianf has joined #openstack-infra | 08:05 | |
| *** jamesmcarthur has quit IRC | 08:06 | |
| *** rpittau has joined #openstack-infra | 08:23 | |
| *** udesale has joined #openstack-infra | 08:25 | |
| *** hashar has joined #openstack-infra | 08:27 | |
| *** udesale has quit IRC | 08:27 | |
| *** udesale has joined #openstack-infra | 08:28 | |
| *** udesale has quit IRC | 08:29 | |
| *** udesale has joined #openstack-infra | 08:29 | |
| *** udesale has quit IRC | 08:29 | |
| *** udesale has joined #openstack-infra | 08:30 | |
| *** roman_g has joined #openstack-infra | 08:38 | |
| *** fresta_ has joined #openstack-infra | 08:40 | |
| *** tosky has joined #openstack-infra | 08:41 | |
| *** fresta has quit IRC | 08:43 | |
| *** udesale has quit IRC | 08:47 | |
| ianw | quiquell|rover: yeah, you can check roughly @ http://grafana.openstack.org/d/ACtl1JSmz/afs?orgId=1 ... let me see what's in logs | 08:47 |
| ianw | rsync: did not see server greeting | 08:48 |
| ianw | looks like the limestone mirror is down | 08:53 |
| ianw | egonzalez / quiquell|rover : i've resynced it to a different mirror manually, let's look into it | 08:55 |
| *** udesale has joined #openstack-infra | 08:55 | |
| quiquell|rover | chkumar|ruck: ^ | 08:58 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Switch centos mirror https://review.openstack.org/603054 | 08:58 |
| ianw | logan-: see ^ ... maybe you know something about rsync://mirror.lstn.net ? | 08:58 |
| quiquell|rover | ianw: Can you add a Closes-Bug to https://bugs.launchpad.net/tripleo/+bug/1792296 | 08:58 |
| openstack | Launchpad bug 1792296 in tripleo "Overcloude deploy error:Timed out waiting for messages from Execution" [Critical,In progress] - Assigned to Quique Llorente (quiquell) | 08:58 |
| *** udesale has quit IRC | 09:01 | |
| *** electrofelix has joined #openstack-infra | 09:01 | |
| *** mrsoul has joined #openstack-infra | 09:02 | |
| quiquell|rover | ianw: So should be synced now ? | 09:04 |
| quiquell|rover | Ahh I see it's sync | 09:05 |
| tosky | ianw, quiquell|rover: are you talking about the error 'BadZipfile("Bad magic number for central directory")' which is popping out in various jobs? | 09:08 |
| quiquell|rover | tosky: What jobs ? | 09:09 |
| tosky | http://logs.openstack.org/23/602223/1/gate/requirements-integration/8435e4f/job-output.txt.gz#_2018-09-17_07_22_50_603447 | 09:10 |
| tosky | and various others | 09:10 |
| *** vivsoni_ has quit IRC | 09:10 | |
| *** florianf has quit IRC | 09:11 | |
| tosky | like http://logs.openstack.org/86/600686/3/check/sahara-tests-tempest/9c65dae/job-output.txt.gz#_2018-09-16_09_11_25_377862 | 09:12 |
| *** ykarel|lunch is now known as ykarel | 09:12 | |
| ianw | tosky: no ... hrm that would be something different again | 09:14 |
| ianw | is http://mirror.iad.rax.openstack.org/wheel/ubuntu-16.04-x86_64/sqlalchemy-utils/SQLAlchemy_Utils-0.33.4-py2.py3-none-any.whl corrupt, or just copying it? | 09:15 |
| tosky | I don't know :) | 09:16 |
| ianw | i get the same error locally, so it looks corrupt | 09:16 |
| tosky | I just noticed the issue | 09:16 |
| tosky | indeed, unzip complains | 09:16 |
| ianw | tonyb: ^ urgh | 09:16 |
| ianw | you can check the logs of the wheel build jobs, and see what's up with that, i don't have time right now :/ | 09:17 |
| *** owalsh_ is now known as owalsh | 09:18 | |
| *** tobiash has quit IRC | 09:20 | |
| *** tobiash has joined #openstack-infra | 09:20 | |
| tosky | ianw: build-wheel-mirror-ubuntu-xenial? It does not seem to show anything weird: http://zuul.openstack.org/builds.html?job_name=build-wheel-mirror-ubuntu-xenial | 09:22 |
| *** priteau has quit IRC | 09:23 | |
| *** priteau has joined #openstack-infra | 09:24 | |
| *** gfidente has joined #openstack-infra | 09:28 | |
| *** priteau has quit IRC | 09:30 | |
| ianw | tosky: no, the periodic job that publishes them to the mirror | 09:33 |
| ianw | http://logs.openstack.org/periodic/git.openstack.org/openstack/requirements/master/publish-wheel-mirror-ubuntu-xenial/ca6ec04/python2/failed.txt | 09:33 |
| ianw | *** FAILED BUILDS FOR BRANCH master | 09:33 |
| ianw | Command | 09:33 |
| ianw | SQLAlchemy-Utils===0.33.4 | 09:33 |
| ianw | perhaps unsurprising | 09:33 |
| *** udesale has joined #openstack-infra | 09:38 | |
| tosky | oh, oki | 09:41 |
| tosky | ianw: so what is the way? Retrigger the job? | 09:41 |
| ianw | i think maybe we have a bit of recursive problem | 09:43 |
| ianw | IIRC these jobs should not be using the wheel mirror to build the wheel, but i can't see where we're disabling that | 09:43 |
| ianw | it seems like building the wheel has downloaded the corrupt wheel and just repopulated it | 09:43 |
| *** priteau has joined #openstack-infra | 09:44 | |
| *** ginopc has quit IRC | 09:47 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Prevent wheel builds using the existing mirror https://review.openstack.org/603064 | 09:53 |
| ianw | tosky / tonyb: that's my first intuition ... but probably worth some investigation ... | 09:54 |
| ianw | pabelanger: ^ this might ring some bells too ... | 09:57 |
| *** oanson has joined #openstack-infra | 10:05 | |
| *** alexchadin has joined #openstack-infra | 10:10 | |
| *** dtantsur|afk is now known as dtantsur | 10:18 | |
| *** quiquell|rover is now known as quique|rover|afk | 10:19 | |
| *** bhavikdbavishi has quit IRC | 10:24 | |
| *** finucannot is now known as stephenfin | 10:34 | |
| *** rh-jelabarre has joined #openstack-infra | 10:43 | |
| *** alexchadin has quit IRC | 10:43 | |
| *** quique|rover|afk is now known as quiquell|rover | 10:49 | |
| *** alexchadin has joined #openstack-infra | 10:53 | |
| *** panda has joined #openstack-infra | 10:58 | |
| *** ginopc has joined #openstack-infra | 11:03 | |
| *** pbourke has quit IRC | 11:18 | |
| *** pbourke has joined #openstack-infra | 11:20 | |
| *** longkb has quit IRC | 11:20 | |
| *** jpena is now known as jpena|lunch | 11:20 | |
| mnasiadka | Hi, is limestone mirror down? http://logs.openstack.org/70/558770/31/check/kolla-ansible-centos-source/d407065/job-output.txt.gz#_2018-09-17_10_04_53_378070 | 11:32 |
| *** felipemonteiro has joined #openstack-infra | 11:33 | |
| *** ldnunes has joined #openstack-infra | 11:35 | |
| ianw | mnasiadka: hrm, yes i can't ping it. i looked and the server appears active, i guess networking issue | 11:38 |
| *** psachin has joined #openstack-infra | 11:43 | |
| *** zul has joined #openstack-infra | 11:47 | |
| *** alexchadin has quit IRC | 11:48 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix unreachable nodes detection https://review.openstack.org/602829 | 11:50 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Also retry the job if a post job failed with unreachable https://review.openstack.org/602830 | 11:51 |
| *** ansmith has quit IRC | 11:53 | |
| *** dave-mccowan has joined #openstack-infra | 11:54 | |
| *** felipemonteiro has quit IRC | 11:56 | |
| *** dhill_ has joined #openstack-infra | 11:57 | |
| *** felipemonteiro has joined #openstack-infra | 11:58 | |
| *** panda has quit IRC | 12:00 | |
| *** rfolco has joined #openstack-infra | 12:00 | |
| *** jamesmcarthur has joined #openstack-infra | 12:02 | |
| *** boden has joined #openstack-infra | 12:03 | |
| *** alexchadin has joined #openstack-infra | 12:05 | |
| *** trown|outtypewww is now known as trown | 12:06 | |
| *** alexchadin has quit IRC | 12:06 | |
| *** jamesmcarthur has quit IRC | 12:07 | |
| *** alexchadin has joined #openstack-infra | 12:07 | |
| *** bobh has joined #openstack-infra | 12:07 | |
| *** olivierb has quit IRC | 12:09 | |
| *** olivierb has joined #openstack-infra | 12:09 | |
| *** bobh has quit IRC | 12:12 | |
| *** felipemonteiro has quit IRC | 12:17 | |
| *** felipemonteiro has joined #openstack-infra | 12:18 | |
| *** felipemonteiro has quit IRC | 12:21 | |
| *** jpena|lunch is now known as jpena | 12:22 | |
| *** tpsilva has joined #openstack-infra | 12:24 | |
| *** janki has quit IRC | 12:25 | |
| *** felipemonteiro has joined #openstack-infra | 12:27 | |
| *** rlandy has joined #openstack-infra | 12:33 | |
| *** quiquell|rover is now known as quique|rover|lch | 12:33 | |
| logan- | mnasiadka ianw: yeah, not sure.. it was refusing connections but seemed to be booted. I rebooted the instance and now it is back up | 12:37 |
| roman_g | Hello all. | 12:46 |
| *** aidin has joined #openstack-infra | 12:47 | |
| roman_g | Readthedocs defaults to the documentation being placed in docs/ directory, and Zuul job for readthedocs is default to the doc/ directory. | 12:47 |
| *** aidin has quit IRC | 12:48 | |
| roman_g | Could Zuul job be changed, or it's not a big deal and possibly would break people's builds? | 12:48 |
| roman_g | highlight ianw | 12:48 |
| slaweq | mordred: hi | 12:48 |
| slaweq | mordred: please ping me when You will be available for few minutes :) | 12:49 |
| *** felipemonteiro has quit IRC | 12:52 | |
| *** udesale has quit IRC | 12:54 | |
| *** kgiusti has joined #openstack-infra | 12:57 | |
| *** felipemonteiro has joined #openstack-infra | 12:57 | |
| mnaser | clarkb: sjc1 has vmx now again | 12:58 |
| *** bobh has joined #openstack-infra | 13:02 | |
| *** quique|rover|lch is now known as quiquell|rover | 13:05 | |
| *** ansmith has joined #openstack-infra | 13:07 | |
| AJaeger | dhellmann: woot, zaqar should be done! | 13:13 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.openstack.org/576907 | 13:14 |
| *** alexchadin has quit IRC | 13:14 | |
| prometheanfire | pip mirror/repo errors? https://review.openstack.org/601487 | 13:15 |
| AJaeger | dhellmann: https://review.openstack.org/#/c/597604/ should be ready - config-core, care to +2, please? | 13:16 |
| *** mriedem has joined #openstack-infra | 13:18 | |
| tosky | prometheanfire: see above and https://review.openstack.org/603064 | 13:18 |
| *** felipemonteiro has quit IRC | 13:18 | |
| *** nhicher has joined #openstack-infra | 13:18 | |
| *** nhicher has quit IRC | 13:20 | |
| *** nhicher has joined #openstack-infra | 13:22 | |
| prometheanfire | just got a ton of 400 errors while creating a story on storyboard :| | 13:23 |
| prometheanfire | https://i.imgur.com/NmDuUtd.png | 13:23 |
| *** alexchadin has joined #openstack-infra | 13:29 | |
| *** psachin has quit IRC | 13:29 | |
| *** felipemonteiro has joined #openstack-infra | 13:31 | |
| *** felipemonteiro has quit IRC | 13:33 | |
| *** jcoufal has joined #openstack-infra | 13:35 | |
| *** alexchadin has quit IRC | 13:38 | |
| *** bdodd has joined #openstack-infra | 13:42 | |
| *** sthussey has joined #openstack-infra | 13:42 | |
| *** ramishra has quit IRC | 13:44 | |
| *** jento has joined #openstack-infra | 13:46 | |
| fungi | prometheanfire: huh, is it repeatable? | 13:46 |
| fungi | might be rabbitmq out to lunch yet again | 13:46 |
| *** alexchadin has joined #openstack-infra | 13:47 | |
| fungi | prometheanfire: oh! i recognize that error | 13:47 |
| prometheanfire | seems to be | 13:47 |
| prometheanfire | :D | 13:48 |
| fungi | prometheanfire: https://storyboard.openstack.org/#!/story/2003650 | 13:49 |
| SotK | that is https://storyboard.openstack.org/#!/story/2003650 | 13:49 |
| SotK | snap | 13:49 |
| fungi | i'm about to add the workarounds in a comment | 13:50 |
| prometheanfire | ty | 13:50 |
| mpeterson_ | hey, a patch that we need to merge to unblock our CI is failing with what seems to be a pip repo problem, can you check it just in case to see if it's in or out the openstack control? https://review.openstack.org/601487 | 13:53 |
| prometheanfire | mpeterson_: https://review.openstack.org/603064 | 13:54 |
| prometheanfire | it's known | 13:54 |
| fungi | prometheanfire: updated with workarounds | 13:55 |
| prometheanfire | ty | 13:55 |
| *** anteaya has joined #openstack-infra | 13:55 | |
| *** bobh has quit IRC | 13:55 | |
| *** alexchadin has quit IRC | 13:56 | |
| dhellmann | is anyone else having trouble with etherpad this morning? | 13:56 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Doc: executor operations - explain jobs will be restarted at restart https://review.openstack.org/603136 | 13:58 |
| *** alexchadin has joined #openstack-infra | 14:01 | |
| mpeterson_ | prometheanfire: a good! perhaps a infra status annonuncment is in place? I checked it before asking here and didn't see anything | 14:03 |
| *** bobh has joined #openstack-infra | 14:13 | |
| *** mpeterson_ is now known as mpeterson | 14:14 | |
| *** eernst has joined #openstack-infra | 14:17 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.openstack.org/576907 | 14:18 |
| *** bobh has quit IRC | 14:18 | |
| *** eernst has quit IRC | 14:19 | |
| *** eernst has joined #openstack-infra | 14:19 | |
| *** hashar is now known as hasharAway | 14:22 | |
| *** quiquell|rover is now known as quique|rover|off | 14:22 | |
| *** eernst has quit IRC | 14:23 | |
| *** eernst_ has joined #openstack-infra | 14:23 | |
| *** e0ne has joined #openstack-infra | 14:30 | |
| *** ykarel is now known as ykarel|away | 14:33 | |
| *** bobh has joined #openstack-infra | 14:35 | |
| *** e0ne has quit IRC | 14:38 | |
| *** vivsoni has quit IRC | 14:44 | |
| *** vivsoni has joined #openstack-infra | 14:44 | |
| *** dayou_ has quit IRC | 14:47 | |
| *** armax has joined #openstack-infra | 14:49 | |
| *** ykarel|away has quit IRC | 14:53 | |
| clarkb | mnaser: thanks | 14:53 |
| *** _d34dh0r53_ is now known as d34dh0r53 | 14:54 | |
| clarkb | infra-root AJaeger re https://review.openstack.org/#/c/603064/1 do we know which wheel(s) are corrupted? do we need to consider deleting them from the afs volume? | 14:54 |
| AJaeger | clarkb: sorry, did not follow along ;( | 14:54 |
| clarkb | scrollback points at SQLAlchemy-Utils maybe | 14:56 |
| mnaser | clarkb: http://logs.openstack.org/74/74/a77fc6b82dab0368f1cdc4d4d39ef6390c7a9526/third-party-check/kata-runsh/0f721e6/ job in sjc1 without vmx problems :) | 14:56 |
| prometheanfire | clarkb: http://logs.openstack.org/87/601487/3/check/requirements-tox-py27-check-uc/337e101/job-output.txt.gz#_2018-09-17_08_58_35_420667 | 14:56 |
| prometheanfire | yep | 14:57 |
| *** janki has joined #openstack-infra | 14:58 | |
| clarkb | ok it will be a few minutes while I spin up afs on my desktop but I'll see if I can delete that file | 14:58 |
| *** bnemec has joined #openstack-infra | 14:59 | |
| *** sgw has quit IRC | 15:00 | |
| openstackgerrit | Merged openstack-infra/project-config master: Prevent wheel builds using the existing mirror https://review.openstack.org/603064 | 15:01 |
| *** roman_g has quit IRC | 15:01 | |
| clarkb | hrm I need to reboot because afs modules are for newer kernel than I booted with | 15:02 |
| fungi | clarkb: that exact case is the reason i wrote the "deleting a file from afs" example in our runbook ;) | 15:02 |
| clarkb | fungi: any chance you are closer to being able to delete that file? | 15:02 |
| clarkb | I'm probably ~5-10 minutes away now that I need a reboot | 15:02 |
| * clarkb proceeds with reboot and will check if it still needs doing after | 15:03 | |
| *** betherly has quit IRC | 15:03 | |
| fungi | sure, i'll take care of it (not really here today but won't take me long) | 15:03 |
| *** roman_g has joined #openstack-infra | 15:04 | |
| clarkb | ok I have afsclient loaded into the kernel now | 15:07 |
| *** alexchadin has quit IRC | 15:08 | |
| clarkb | fungi: should I proceed with delete attempt or do you have it? | 15:08 |
| openstackgerrit | Merged openstack-infra/project-config master: remove job settings for zaqar repositories https://review.openstack.org/597604 | 15:10 |
| *** josephrsandoval has joined #openstack-infra | 15:12 | |
| clarkb | fungi: ls makes it look like you deleted it on the RW volume | 15:12 |
| fungi | #status log manually deleted /afs/.openstack.org/mirror/wheel/ubuntu-16.04-x86_64/s/sqlalchemy-utils/SQLAlchemy_Utils-0.33.4-py2.py3-none-any.whl and released the mirror.wheel.xenialx64 volume | 15:12 |
| openstackstatus | fungi: finished logging | 15:12 |
| clarkb | and now I don't see it on the RO volume. Thanks! | 15:12 |
| fungi | yep, took me a minute to figure out the file path and what the correct volume name was | 15:12 |
| fungi | and now i've gotten in trouble for being spotted doing openstack things when i'm supposed to be taking plywood off the windows. gotta go! | 15:13 |
| AJaeger | fungi: run ;) Thanks! | 15:14 |
| clarkb | fungi: thank you | 15:15 |
| clarkb | AJaeger: https://review.openstack.org/#/c/601855/ is a small afs docs update for those of us on non Debuntu distros maybe you can review that? | 15:16 |
| clarkb | dhellmann: re etherpad it came up over in #zuul as well. Looking at cacti http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=116&rra_id=all there was a spike in connections and a gap in the the snmp data | 15:17 |
| clarkb | it seems fine now, maybe a blip in networking at rax possibly related to the spike in connections? | 15:18 |
| dhellmann | clarkb : thanks. it seems OK now. I had an error connecting earlier and then when it did the client reconnected after a few minutes. | 15:18 |
| dhellmann | so yeah, maybe that's all | 15:18 |
| *** alexchadin has joined #openstack-infra | 15:18 | |
| *** jcoufal has quit IRC | 15:21 | |
| AJaeger | clarkb: +1. | 15:21 |
| *** anteaya has quit IRC | 15:23 | |
| *** alexchadin has quit IRC | 15:23 | |
| *** _ari_ has joined #openstack-infra | 15:28 | |
| clarkb | mnaser: is the kata disk device question a difference in boot from volume (sjc1) and boot from whatever it is in ymq1? | 15:29 |
| clarkb | smcginnis: see email to dev list about removing driverfixes/ocata branch. If you can read that over just to double check we haven't missed anything and let me know you are still happy I will delete the brnach | 15:37 |
| * clarkb finds breakfast | 15:39 | |
| *** alexchadin has joined #openstack-infra | 15:40 | |
| smcginnis | Looking now, thanks clarkb | 15:41 |
| *** josephrsandoval has quit IRC | 15:43 | |
| *** bhavikdbavishi has joined #openstack-infra | 15:43 | |
| *** alexchadin has quit IRC | 15:44 | |
| *** jamesmcarthur has joined #openstack-infra | 15:45 | |
| prometheanfire | still getting sqlalchemy-utils errors | 15:45 |
| prometheanfire | http://logs.openstack.org/83/602983/1/check/requirements-tox-py27-check-uc/b7f1b88/job-output.txt.gz#_2018-09-17_13_20_37_893676 | 15:46 |
| prometheanfire | clarkb: ^ | 15:46 |
| clarkb | prometheanfire: that timestamp is ~2 hours ago? | 15:46 |
| prometheanfire | ah, utc, right | 15:48 |
| *** ginopc has quit IRC | 15:48 | |
| *** aidin has joined #openstack-infra | 15:48 | |
| *** dklyle has quit IRC | 15:49 | |
| *** josephrsandoval has joined #openstack-infra | 15:50 | |
| *** dklyle has joined #openstack-infra | 15:50 | |
| *** aidin has quit IRC | 15:53 | |
| clarkb | #status log Removed openstack/cinder driverfixes/ocata branch with HEAD a37cc259f197e1a515cf82deb342739a125b65c6 | 15:56 |
| openstackstatus | clarkb: finished logging | 15:56 |
| *** icey has quit IRC | 15:56 | |
| *** icey has joined #openstack-infra | 15:57 | |
| smcginnis | clarkb: Thanks | 15:57 |
| *** ykarel has joined #openstack-infra | 15:59 | |
| *** josephrsandoval has quit IRC | 15:59 | |
| *** jpich has quit IRC | 16:01 | |
| mnaser | clarkb: this is because we have a custom flavors that boots /dev/vdb for kata in there, and ideally, i would like to have kata not use it's own custom flavor | 16:01 |
| clarkb | mnaser: gotcha, I'll let you explain that then :) | 16:01 |
| mnaser | ill reply to the github issue shortly | 16:01 |
| *** bhavikdbavishi has quit IRC | 16:08 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: OVH BHS1 Maintenance https://review.openstack.org/603173 | 16:08 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Revert "OVH BHS1 Maintenance" https://review.openstack.org/603174 | 16:08 |
| clarkb | config-core ^ fyi that in a day and a half or so ovh is doing maintenance in BHS1. AJaeger I can approve the max-servers: 0 change tomorrow evening, maybe you can approve the revert once 1200UTC on the 19th is past? | 16:09 |
| * clarkb adds the GRA1 changes | 16:09 | |
| *** bhavikdbavishi has joined #openstack-infra | 16:10 | |
| *** gyee has joined #openstack-infra | 16:12 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: OVH GRA1 Maintenance https://review.openstack.org/603177 | 16:13 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Revert "OVH GRA1 Maintenance" https://review.openstack.org/603178 | 16:13 |
| *** anteaya has joined #openstack-infra | 16:13 | |
| clarkb | GRA1 is in october. I will go ahead and and WIP the GRA1 changes | 16:14 |
| *** holser_ has quit IRC | 16:14 | |
| *** olivierb has quit IRC | 16:19 | |
| clarkb | mnaser: what was the vmx problem out of curiousity | 16:19 |
| mnaser | clarkb: turn it off and on again type of thing | 16:19 |
| mnaser | i guess even if we modprobe -r and add kvm_intel again with nested virt | 16:20 |
| mnaser | it just doesnt accept it | 16:20 |
| *** dtruong has joined #openstack-infra | 16:25 | |
| *** psachin has joined #openstack-infra | 16:28 | |
| *** psachin has quit IRC | 16:43 | |
| *** psachin has joined #openstack-infra | 16:48 | |
| *** ykarel is now known as ykarel|away | 16:50 | |
| *** aojea has quit IRC | 16:53 | |
| *** dtantsur is now known as dtantsur|afk | 16:54 | |
| *** anteaya has quit IRC | 16:56 | |
| *** anteaya has joined #openstack-infra | 16:58 | |
| AJaeger | clarkb: will do if I'm around... | 17:05 |
| AJaeger | clarkb: would be nice to have time and date as git header - let me do that with revert | 17:05 |
| *** josephrsandoval has joined #openstack-infra | 17:05 | |
| AJaeger | ... with all changes... | 17:06 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: OVH BHS1 Maintenance 2018-09-19 0700 UTC https://review.openstack.org/603173 | 17:06 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Revert "OVH BHS1 Maintenance" - 2018-09-19 1200UTC https://review.openstack.org/603174 | 17:07 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Revert "OVH BHS1 Maintenance" - 2018-09-19 1200UTC https://review.openstack.org/603174 | 17:07 |
| *** efried has joined #openstack-infra | 17:08 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: OVH GRA1 Maintenance 2018-10-10 1900 UTC https://review.openstack.org/603177 | 17:08 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: OVH GRA1 Maintenance 2018-10-10 1900 UTC https://review.openstack.org/603177 | 17:08 |
| AJaeger | clarkb: want to WIP https://review.openstack.org/#/c/603177/ again? | 17:08 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Revert "OVH GRA1 Maintenance" - 2018-10-11 0000UTC https://review.openstack.org/603178 | 17:08 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Revert "OVH GRA1 Maintenance" - 2018-10-11 0000UTC https://review.openstack.org/603178 | 17:08 |
| *** jpena is now known as jpena|off | 17:11 | |
| *** efried is now known as efried_off | 17:12 | |
| clarkb | AJaeger: ya I will remark them | 17:15 |
| AJaeger | clarkb: I did already ^ | 17:15 |
| clarkb | oh thabks | 17:16 |
| AJaeger | clarkb: so, just WIP 603177, please | 17:16 |
| AJaeger | config-core, the following are ready for review if anybody has time: https://review.openstack.org/602553 https://review.openstack.org/599760 https://review.openstack.org/601460 https://review.openstack.org/601319 https://review.openstack.org/602493 https://review.openstack.org/602871 | 17:17 |
| *** trown is now known as trown|lunch | 17:18 | |
| AJaeger | clarkb: push and pushMerge belong into " [access "refs/for/refs/*"]" for ACLs, don't they? And do we really hand push out, see https://review.openstack.org/#/c/602377/2/gerrit/acls/openstack/tripleo-quickstart.config | 17:21 |
| clarkb | refs/for is to push changes, I think we can allow push on refs/foo as long as its temporary to get a merge commit in between branches and that foo ref isnt already used | 17:22 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove fuxi jobs https://review.openstack.org/595273 | 17:23 |
| AJaeger | clarkb: I see - but I'll let others review that... | 17:24 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove fuxi jobs https://review.openstack.org/595273 | 17:26 |
| *** josephrsandoval has quit IRC | 17:27 | |
| *** josephrsandoval has joined #openstack-infra | 17:28 | |
| *** devananda has joined #openstack-infra | 17:28 | |
| *** sgw has joined #openstack-infra | 17:28 | |
| *** yumiriam has joined #openstack-infra | 17:29 | |
| *** eernst_ has quit IRC | 17:35 | |
| clarkb | smcginnis: fyi I've made ha-guide active in gerrit and approved the change to update the ACLs on it | 17:37 |
| smcginnis | Awesome, thanks again clarkb | 17:38 |
| *** psachin has quit IRC | 17:40 | |
| clarkb | prometheanfire: re gentoo image spin up are there still issues? I think I'm in a spot whee I can help debug that nowish | 17:41 |
| *** tbarron_ is now known as tbarron | 17:41 | |
| prometheanfire | clarkb: sure, I was told it was network bringup? | 17:44 |
| openstackgerrit | Merged openstack-infra/project-config master: Add new project config for kayobe-config-dev https://review.openstack.org/599760 | 17:44 |
| openstackgerrit | Merged openstack-infra/project-config master: Set launchpad group to neutron-tempest-plugin https://review.openstack.org/601460 | 17:44 |
| * prometheanfire hasn't had problems | 17:44 | |
| clarkb | prometheanfire: are there logs somewhere? (trying to avoid relearning what is already learned) | 17:45 |
| prometheanfire | clarkb: no, someone else mentioned it here | 17:45 |
| prometheanfire | maybe AJaeger | 17:45 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added new endpoints to seed tags on summit tracks https://review.openstack.org/603193 | 17:45 |
| AJaeger | clarkb, this is from earlier: looks like networking setup problem, see http://logs.openstack.org/39/602439/3/check/openstack-infra-base-integration-gentoo-17-0-systemd/f33cf00/job-output.txt.gz#_2018-09-15_10_32_12_158076 . I'm not an expert in these images, please work with rest of the team on this ;) | 17:46 |
| AJaeger | clarkb: thanks for reviewing | 17:46 |
| *** shardy has quit IRC | 17:46 | |
| clarkb | AJaeger: thanks, that helps. | 17:47 |
| clarkb | that indicates the instances are booting but then dns lookups are failing /me goes to see why | 17:47 |
| clarkb | looks like we don't have any gentoo servers currently ready, I'll wait for one in that case | 17:48 |
| prometheanfire | ya, this is part of the problem | 17:49 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added new endpoints to seed tags on summit tracks https://review.openstack.org/603193 | 17:49 |
| prometheanfire | chicken meet egg | 17:49 |
| prometheanfire | booting a systemd image now (since it has glean), will check how it's doing dns resolution | 17:50 |
| openstackgerrit | Merged openstack-infra/project-config master: Add openstack/ha-guide to zuul https://review.openstack.org/601319 | 17:50 |
| prometheanfire | /etc/resolv.conf is populated | 17:51 |
| clarkb | well our images should boot with working unbound config | 17:51 |
| clarkb | shouldn't be any chicken and egg for that (and this is intentional) | 17:51 |
| prometheanfire | ya, the review should include the fix for it | 17:51 |
| prometheanfire | maybe the unbound fix needs to merge, then a new image be built? | 17:51 |
| clarkb | there is no running unbound service | 17:51 |
| prometheanfire | is unbound installed? | 17:52 |
| clarkb | prometheanfire: which change? | 17:52 |
| clarkb | /usr/sbin/unbound is on the system | 17:52 |
| clarkb | I'm going to guess it isn't enabled | 17:52 |
| openstackgerrit | Merged openstack-infra/project-config master: Add oslo.upgradecheck https://review.openstack.org/602493 | 17:52 |
| openstackgerrit | Merged openstack-infra/project-config master: Create puppet-placement https://review.openstack.org/602871 | 17:52 |
| prometheanfire | gentoo doesn't enable by default | 17:52 |
| clarkb | ● unbound.service loaded failed failed Unbound recursive Domain Name Server | 17:53 |
| clarkb | systemctl says it failed to start | 17:53 |
| prometheanfire | https://review.openstack.org/#/c/602439/ | 17:53 |
| clarkb | Sep 17 17:50:42 gentoo-17-0-systemd-rax-ord-0002069208 unbound-checkconf[5360]: input in flex scanner failed | 17:53 |
| clarkb | prometheanfire: you need to fix that on the image | 17:53 |
| prometheanfire | ok | 17:54 |
| clarkb | (for precisely this reason, we need working dns on boot, not later) | 17:54 |
| prometheanfire | what does unbound-checkconf return? | 17:54 |
| *** studarus_ has joined #openstack-infra | 17:54 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Use tox-docs directly https://review.openstack.org/602253 | 17:54 |
| *** david-lyle has joined #openstack-infra | 17:54 | |
| clarkb | server has gone away so can't check :/ | 17:54 |
| prometheanfire | is https://review.openstack.org/#/c/602439/3/roles/configure-unbound/tasks/main.yaml in the unbound config? | 17:54 |
| prometheanfire | oh | 17:54 |
| *** priteau has quit IRC | 17:55 | |
| prometheanfire | I'm downloading the image from https://nb01.openstack.org/images/ to test | 17:55 |
| clarkb | ok | 17:56 |
| *** sgw has quit IRC | 17:56 | |
| *** jmorgan1 has quit IRC | 17:56 | |
| * prometheanfire couldn't do this at the ptg because large files... | 17:57 | |
| *** dklyle has quit IRC | 17:57 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Do not abort node launch if failed node cannot be deleted https://review.openstack.org/589854 | 17:58 |
| *** eernst has joined #openstack-infra | 18:00 | |
| *** jmorgan1 has joined #openstack-infra | 18:00 | |
| *** eernst has quit IRC | 18:00 | |
| *** eernst has joined #openstack-infra | 18:01 | |
| clarkb | goign to try and catch the next one myself | 18:03 |
| prometheanfire | ? | 18:03 |
| clarkb | prometheanfire: to see if I can get the checkconf output | 18:04 |
| *** janki has quit IRC | 18:04 | |
| prometheanfire | oh, thanks | 18:04 |
| prometheanfire | so, recheck for you? | 18:04 |
| *** eernst has quit IRC | 18:04 | |
| clarkb | no don't recheck | 18:04 |
| prometheanfire | k | 18:04 |
| clarkb | it is trying to build a new server and the job running and failing is what causes it to delete | 18:05 |
| clarkb | (we can hold on the job if we really need to but I expect that if we can get the check conf output the solution will be clear) | 18:05 |
| prometheanfire | ya, I'm guessing my ansible insertion wasn't quite correct | 18:05 |
| clarkb | well the ansible insertion happens after boot | 18:06 |
| clarkb | I think the two are unrelated? | 18:06 |
| clarkb | (I' | 18:06 |
| clarkb | er | 18:06 |
| clarkb | I'm pretty sure I searched the journal and that was the first unbound problem found | 18:07 |
| prometheanfire | initial dns should be working just from config-drive/glean | 18:07 |
| prometheanfire | right? | 18:07 |
| prometheanfire | then ansible goes and configures / starts unbound | 18:07 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove migrated keystone jobs https://review.openstack.org/603199 | 18:08 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.openstack.org/576907 | 18:08 |
| clarkb | prometheanfire: no, glean shouldn't be configuring dns for us. We explicitly disable this because some clouds blacklist IPs then reuse them. We rely on unbound on boot | 18:08 |
| prometheanfire | ok, and since gentoo doesn't enable the service on install it's not running | 18:09 |
| prometheanfire | I guess | 18:09 |
| *** openstackgerrit has quit IRC | 18:10 | |
| clarkb | prometheanfire: include: "/etc/unbound/conf.d" and include: /etc/unbound/forwarding.conf are already int he config file at the end of the file | 18:10 |
| clarkb | it was not at the beginning of the file | 18:10 |
| clarkb | the hceck conf command doesn't really say what the issue is | 18:10 |
| *** jiapei has joined #openstack-infra | 18:11 | |
| prometheanfire | just 'failed' | 18:11 |
| prometheanfire | can you copy the conf file out | 18:11 |
| prometheanfire | so I can play | 18:11 |
| clarkb | prometheanfire: you should have it in the image you are downloading | 18:11 |
| clarkb | I've already lsot connectivity to the current host :/ | 18:11 |
| prometheanfire | k | 18:12 |
| prometheanfire | ya, just downloaded | 18:12 |
| prometheanfire | at work now, will upload image when I get home | 18:12 |
| prometheanfire | may have to expand my /var/lib/glance/images partition... | 18:12 |
| clarkb | the error we get is 'input in flex scanner failed' which is a bison error apparently and indicates an io error | 18:13 |
| *** aspiers has quit IRC | 18:13 | |
| *** anteaya has quit IRC | 18:14 | |
| prometheanfire | huh | 18:14 |
| prometheanfire | and checkconf returned bad without saying why? | 18:14 |
| *** sgw has joined #openstack-infra | 18:15 | |
| clarkb | it said that above | 18:15 |
| clarkb | input in flex scanner failed | 18:15 |
| prometheanfire | k | 18:15 |
| *** anteaya has joined #openstack-infra | 18:17 | |
| prometheanfire | reloacting to upload over not-bad connection | 18:18 |
| clarkb | I wonder if the issue is our include of a dir rather than a file | 18:19 |
| prometheanfire | I did include: "/etc/unbound/conf.d/*.conf" and validated correctly locallt | 18:20 |
| prometheanfire | just deleted that instance though... | 18:20 |
| clarkb | prometheanfire: https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/nodepool-base/finalise.d/89-unbound#n59 I think you need to use the glob on that line too | 18:22 |
| clarkb | the manpage for unbound.conf says that you specifcy file paths and that globs can be used. So I think the dir ent is what must be the problem | 18:22 |
| clarkb | prometheanfire: if you want to write that change we can probably get it in between mnaser AJaeger and myself then I can trigger a gentoo image build once the config updtes on the builders | 18:24 |
| prometheanfire | ok | 18:24 |
| prometheanfire | will in a sec | 18:24 |
| *** openstackgerrit has joined #openstack-infra | 18:28 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/project-config master: include conf files for unbound via glob https://review.openstack.org/603204 | 18:28 |
| *** aspiers has joined #openstack-infra | 18:29 | |
| prometheanfire | clarkb: ^ | 18:29 |
| clarkb | +2 thanks | 18:30 |
| prometheanfire | cool | 18:30 |
| prometheanfire | clarkb: does that mean I don't need https://review.openstack.org/#/c/602439/3/roles/configure-unbound/tasks/main.yaml ? | 18:30 |
| AJaeger | prometheanfire: +2A | 18:31 |
| clarkb | prometheanfire: ya if this works then I think that change will be unnecesary | 18:31 |
| *** trown|lunch is now known as trown | 18:32 | |
| *** eernst has joined #openstack-infra | 18:35 | |
| *** eernst has quit IRC | 18:37 | |
| *** bhavikdbavishi has quit IRC | 18:37 | |
| prometheanfire | k | 18:41 |
| *** aspiers has quit IRC | 18:42 | |
| *** studarus_ has quit IRC | 18:46 | |
| *** aspiers has joined #openstack-infra | 18:49 | |
| *** ykarel|away has quit IRC | 18:53 | |
| *** hemna_ has quit IRC | 18:55 | |
| *** hemna_ has joined #openstack-infra | 18:57 | |
| *** alexchadin has joined #openstack-infra | 18:59 | |
| * clarkb pops out for lunch things, will be back after to trigger gentoo image rebuild | 19:05 | |
| *** josephrsandoval has quit IRC | 19:05 | |
| *** zul has quit IRC | 19:08 | |
| *** jamesmcarthur has quit IRC | 19:11 | |
| *** jamesmcarthur has joined #openstack-infra | 19:11 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Add linaro-london to nodepool https://review.openstack.org/602553 | 19:16 |
| *** jamesmcarthur has quit IRC | 19:17 | |
| *** jtomasek has quit IRC | 19:17 | |
| AJaeger | config-core, small cleanup to delete now obsolete jobs - https://review.openstack.org/595273. Please review | 19:18 |
| prometheanfire | yep, waiting for merge and nodepool update | 19:18 |
| *** jtomasek has joined #openstack-infra | 19:20 | |
| *** felipemonteiro has joined #openstack-infra | 19:23 | |
| *** sgw has quit IRC | 19:24 | |
| *** zul has joined #openstack-infra | 19:26 | |
| *** anteaya has quit IRC | 19:31 | |
| *** alexchadin has quit IRC | 19:32 | |
| *** felipemonteiro has quit IRC | 19:32 | |
| *** jtomasek has quit IRC | 19:37 | |
| *** jtomasek has joined #openstack-infra | 19:40 | |
| AJaeger | and https://review.openstack.org/603199 has it's dependency approved, so can be reviewed as well. | 19:41 |
| slaweq | mordred: hi, are You around? | 19:42 |
| *** haleyb has joined #openstack-infra | 19:45 | |
| *** felipemonteiro has joined #openstack-infra | 19:46 | |
| *** aspiers has quit IRC | 19:48 | |
| openstackgerrit | Merged openstack-infra/project-config master: include conf files for unbound via glob https://review.openstack.org/603204 | 19:53 |
| openstackgerrit | Merged openstack-infra/project-config master: Add linaro-london to nodepool https://review.openstack.org/602553 | 19:53 |
| prometheanfire | now to wait for project-config update and then new build | 19:54 |
| rlandy | mnaser: we're seeing a failure with jobs referring to the vexxhost mirror - http://logs.openstack.org/45/602445/5/check/tripleo-ci-centos-7-containers-multinode/c01eace/logs/subnode-2/home/zuul/repo_setup.log.txt.gz#_2018-09-17_19_22_10 | 19:58 |
| rlandy | are you aware of any issues? | 19:58 |
| mnaser | hi | 19:59 |
| rlandy | nhicher: ^^ fyi | 19:59 |
| *** aspiers has joined #openstack-infra | 19:59 | |
| mnaser | rlandy: if it's giving a 404, then i cant really control much | 19:59 |
| mnaser | the vm is up, but for some reason it's giving a 404, so maybe check with any infra-root's here | 19:59 |
| pabelanger | keep in mind, that isn't a mirror but reverse proxy cache. So 404 is really coming from centos cbs | 19:59 |
| pabelanger | so, was indexes deleted on centos.cbs side between yum update and access? | 20:00 |
| *** felipemonteiro has quit IRC | 20:01 | |
| weshay | pabelanger, asking alan to join | 20:01 |
| prometheanfire | every time I hear 'alan'... https://www.youtube.com/watch?v=XgvR3y5JCXg | 20:02 |
| *** diablo_rojo has joined #openstack-infra | 20:03 | |
| weshay | prometheanfire, that just made my day | 20:06 |
| *** yumiriam has left #openstack-infra | 20:06 | |
| prometheanfire | it like lana from archer | 20:08 |
| smcginnis | :D | 20:10 |
| prometheanfire | that one tended to go too far | 20:12 |
| clarkb | AJaeger: 603199 doesnt remove the jobs from ozj? | 20:13 |
| *** rh-jelabarre has quit IRC | 20:18 | |
| *** jtomasek has quit IRC | 20:22 | |
| *** rh-jelabarre has joined #openstack-infra | 20:24 | |
| *** anteaya has joined #openstack-infra | 20:26 | |
| *** lbragstad has quit IRC | 20:26 | |
| *** lbragstad has joined #openstack-infra | 20:27 | |
| *** eernst has joined #openstack-infra | 20:28 | |
| *** zul has quit IRC | 20:32 | |
| *** ansmith has quit IRC | 20:38 | |
| *** jamesmcarthur has joined #openstack-infra | 20:39 | |
| *** kgiusti has left #openstack-infra | 20:42 | |
| *** hasharAway has quit IRC | 20:48 | |
| *** bobh has quit IRC | 20:49 | |
| *** trown is now known as trown|outtypewww | 21:07 | |
| *** ldnunes has quit IRC | 21:15 | |
| clarkb | prometheanfire: new gentoo image is building now | 21:17 |
| clarkb | prometheanfire: on nb01 if you want to follow the logs | 21:17 |
| prometheanfire | cool | 21:20 |
| prometheanfire | yep | 21:20 |
| clarkb | if any infra roots are around and want to review https://review.openstack.org/#/c/601348/3 and its children (puppet 4 parser updates) I can watch those go in | 21:23 |
| *** bobh has joined #openstack-infra | 21:25 | |
| *** bobh has quit IRC | 21:30 | |
| clarkb | I'm trying to figure out why ansible isn't happy about our use of add_host now | 21:35 |
| clarkb | it seems that maybe we aren't running the same pre steps we do in other jobs (I'm guessing that is why the ssh key isn't set up) | 21:39 |
| *** felipemonteiro has joined #openstack-infra | 21:39 | |
| clarkb | I wonder if the abstract attribute affects the inheritance there | 21:41 |
| *** eernst has quit IRC | 21:43 | |
| *** eernst has joined #openstack-infra | 21:44 | |
| clarkb | http://logs.openstack.org/76/601476/1/gate/system-config-run-base/abeb6d3/zuul-info/inventory.yaml shows another job that has a similar inheritance path with abstract: true job as a parent and that one is fine | 21:45 |
| *** eernst has quit IRC | 21:46 | |
| *** eernst_ has joined #openstack-infra | 21:46 | |
| *** eernst_ has quit IRC | 21:50 | |
| *** hemna_ has quit IRC | 21:51 | |
| clarkb | has anyone seen this sort of behavior again with inheritance? I am quite confused | 21:51 |
| *** boden has quit IRC | 21:53 | |
| clarkb | oh I get it now | 21:53 |
| clarkb | the problem is our lack of a nodeset and the pre tasks in base run on all | 21:53 |
| clarkb | except all doesn't include localhost so they do not run | 21:54 |
| *** gfidente has quit IRC | 21:57 | |
| openstackgerrit | Merged openstack-infra/system-config master: Turn on the future parser for translate.o.o https://review.openstack.org/601345 | 21:58 |
| *** bdodd has quit IRC | 22:00 | |
| clarkb | infra-root reading the add-build-sshkey role in zuul-jobs I'm not immediatel sure how to address this. We sort of depend on the all behavior not including localhost there to avoid configuring things we don't want on the executor | 22:02 |
| clarkb | one thought I have is a new base jobs for executor only jobs that run roles that are trimmed down versions of their larger cousins that explicitly only do the things we want on localhost | 22:03 |
| clarkb | though I think we may only want emit-job-header, add-build-sshkey, and log-inventory equivalents? | 22:04 |
| * clarkb hacks a thing that we can at least think about | 22:05 | |
| *** rcernin has joined #openstack-infra | 22:08 | |
| *** slaweq has quit IRC | 22:09 | |
| *** slaweq has joined #openstack-infra | 22:11 | |
| *** felipemonteiro has quit IRC | 22:15 | |
| *** slaweq has quit IRC | 22:15 | |
| *** felipemonteiro has joined #openstack-infra | 22:18 | |
| *** jamesmcarthur has quit IRC | 22:18 | |
| *** jamesmcarthur has joined #openstack-infra | 22:20 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul-jobs master: Make ssh key clearning a distinct role https://review.openstack.org/603240 | 22:22 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Run localhost specific roles/tasks on localhost https://review.openstack.org/603241 | 22:24 |
| openstackgerrit | Gaetan Semet proposed openstack-dev/pbr master: cmd: 'pbr boostrap' create setup.[cfg,py] automatically https://review.openstack.org/559484 | 22:30 |
| clarkb | ok in writing that I realized that we don't actually remove the ssh key at all on nil nodeset jobs so it should work. From that I realized that our sshd config is such that we only allow ssh as root to the bridge from known hosts | 22:30 |
| clarkb | that is the actual error here | 22:30 |
| clarkb | pabelanger: ^ is that localhost and empty nodeset case something you've thought about or run into yet? | 22:33 |
| clarkb | I think it is worth a minute of thought and I'm sure I've approached it poorly there | 22:33 |
| *** felipemonteiro has quit IRC | 22:34 | |
| openstackgerrit | Gaetan Semet proposed openstack-dev/pbr master: option to print only the version of a package https://review.openstack.org/577614 | 22:40 |
| clarkb | prometheanfire: the image is uploading now | 22:41 |
| *** josephrsandoval has joined #openstack-infra | 22:43 | |
| *** felipemonteiro has joined #openstack-infra | 22:44 | |
| clarkb | for addressing the zuul executor to bridge.o.o ssh problem, maybe we should add a zuul user to bridge.o.o instead of allowing root? that may be easier since we can already add non root users pretty easily | 22:45 |
| *** rlandy is now known as rlandy|bbl | 22:50 | |
| *** rockyg has joined #openstack-infra | 22:50 | |
| *** rockyg has quit IRC | 22:51 | |
| *** slaweq has joined #openstack-infra | 23:11 | |
| *** devananda has quit IRC | 23:13 | |
| *** slaweq has quit IRC | 23:16 | |
| *** josephrsandoval has quit IRC | 23:17 | |
| clarkb | prometheanfire: looks like the next problem is Sep 17 23:15:13 gentoo-17-0-systemd-rax-iad-0002072721 unbound-checkconf[5461]: /etc/unbound/var/log: No such file or directory | 23:18 |
| clarkb | I have no idea why it is treating the logfile as relative to the /etc/unbound dir | 23:18 |
| *** josephrsandoval has joined #openstack-infra | 23:19 | |
| clarkb | its definitely doing that to the logfile directive though (I've modified it to be /var/clarkb/log and it complains about /etc/unbound/var/clarkb/log) | 23:19 |
| clarkb | ok figured it out, gentoo seems to have a chroot configured by default for unbound | 23:22 |
| *** mriedem is now known as mriedem_away | 23:23 | |
| *** tosky has quit IRC | 23:23 | |
| *** josephrsandoval has quit IRC | 23:24 | |
| *** jamesmcarthur has quit IRC | 23:24 | |
| clarkb | on ubuntu we don't appear to run chrooted unbound | 23:30 |
| clarkb | but its also not in the config and the config implies it runs chrooted by default | 23:30 |
| *** jiapei has quit IRC | 23:30 | |
| clarkb | centos explicitly unchroots | 23:32 |
| *** jamesmcarthur has joined #openstack-infra | 23:34 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Disable unbound chroot on test node images https://review.openstack.org/603247 | 23:36 |
| clarkb | prometheanfire: ^ Ithink that will fix it. I have a test node with unbound running now that that is in along with the include fix | 23:36 |
| clarkb | in fact if you recheck your job it should use that hand hacked test node | 23:37 |
| *** jamesmcarthur has quit IRC | 23:38 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Upload arm64 images to linaro-london cloud https://review.openstack.org/603249 | 23:45 |
| clarkb | ianw: ^ is that a known issue | 23:46 |
| clarkb | there are no conflicts with listed in gerrit so I will assume not. Hopefully that gets the new london region running | 23:46 |
| *** sthussey has quit IRC | 23:49 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
