| *** jascott1 has quit IRC | 00:00 | |
| *** jistr has quit IRC | 00:00 | |
| *** aeng is now known as aeng_mtg | 00:00 | |
| *** jistr has joined #openstack-infra | 00:01 | |
| *** bobh has joined #openstack-infra | 00:01 | |
| *** jascott1 has joined #openstack-infra | 00:02 | |
| *** bobh has quit IRC | 00:02 | |
| *** pahuang has quit IRC | 00:04 | |
| *** bobh has joined #openstack-infra | 00:04 | |
| *** mat128 has quit IRC | 00:05 | |
| pabelanger | o/ | 00:05 |
|---|---|---|
| *** slaweq_ has joined #openstack-infra | 00:05 | |
| pabelanger | catching up on backscroll | 00:05 |
| *** sree has joined #openstack-infra | 00:08 | |
| pabelanger | jeblair: it does look like reloads are faster on zuulv3.o.o, atleast by timestamps in debug.log | 00:09 |
| SamYaple | pabelanger: yea 7 seconds instead of 50s according to jeblair | 00:09 |
| pabelanger | yup, I can confirm that | 00:09 |
| pabelanger | we still get a spike in CPU, but that is to be expected | 00:11 |
| *** lnxnut has joined #openstack-infra | 00:11 | |
| pabelanger | I did just recheck 509182, and not sure why but it seems to have done 3 dynamic reloads | 00:12 |
| *** sree has quit IRC | 00:12 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/elastic-recheck master: Add support for Zuulv3-specific parameters in elastic-recheck https://review.openstack.org/509313 | 00:13 |
| dmsimard | clarkb, pabelanger ^ first stab at it | 00:13 |
| *** kzaitsev_pi has quit IRC | 00:16 | |
| *** kzaitsev_pi has joined #openstack-infra | 00:16 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/elastic-recheck master: Add support for Zuulv3-specific parameters in elastic-recheck https://review.openstack.org/509313 | 00:16 |
| *** yamamoto has joined #openstack-infra | 00:18 | |
| *** bobh has quit IRC | 00:20 | |
| jeblair | i re-enqueued a bunch of changes from earlier to drive up load | 00:20 |
| *** pahuang has joined #openstack-infra | 00:21 | |
| pabelanger | jeblair: I'm trying to see why 509182 was reloaded 3 times, could it be once per pipeline? | 00:21 |
| *** pt_15 has quit IRC | 00:21 | |
| pabelanger | I see 509182 was added to check, infra-check and tripleo-check | 00:21 |
| *** lnxnut has quit IRC | 00:21 | |
| pabelanger | but only made it into 1 pipeline | 00:21 |
| jeblair | pabelanger: likely. | 00:22 |
| pabelanger | jeblair: okay, thanks. | 00:22 |
| *** yamamoto has quit IRC | 00:23 | |
| dmsimard | jeblair: the failures in https://review.openstack.org/#/c/509254/ don't make sense ? http://logs.openstack.org/54/509254/4/check/tox-pep8/a32022f/ara/result/b06f1fcf-c890-4e6f-b2a0-6b07fbb09af6/ | 00:23 |
| dmsimard | tox-pep8 failing on missing toxenv infra-docs | 00:24 |
| dmsimard | I'm on my phone right now, I'll investigate later if need be, I thought I'd bring it up in case it's related to the config rework | 00:25 |
| *** caphrim007 has quit IRC | 00:25 | |
| *** ijw has quit IRC | 00:29 | |
| *** numans has quit IRC | 00:30 | |
| *** ijw has joined #openstack-infra | 00:30 | |
| pabelanger | http://logs.openstack.org/54/509254/4/check/tox-pep8/a32022f/zuul-info/inventory.yaml | 00:31 |
| pabelanger | tox_envlist: infra-docs | 00:31 |
| pabelanger | I don't know why that is however | 00:31 |
| pabelanger | tox_extra_args: -vv python setup.py build_sphinx also | 00:31 |
| openstackgerrit | Merged openstack/diskimage-builder master: Move fstrim to block device layer https://review.openstack.org/504579 | 00:31 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Speed configuration building https://review.openstack.org/509309 | 00:32 |
| jeblair | dmsimard: yes, you were right. we need a test for that. | 00:32 |
| dmsimard | Oops | 00:33 |
| dmsimard | Feel free to recheck the patch once zuul is reloaded with PS#2 | 00:33 |
| jeblair | done | 00:34 |
| jeblair | dmsimard: thanks | 00:34 |
| *** dprince has quit IRC | 00:34 | |
| *** ijw has quit IRC | 00:35 | |
| *** caphrim007 has joined #openstack-infra | 00:35 | |
| *** numans has joined #openstack-infra | 00:35 | |
| *** ijw has joined #openstack-infra | 00:36 | |
| *** bobh has joined #openstack-infra | 00:36 | |
| *** slaweq_ has quit IRC | 00:37 | |
| *** numans has quit IRC | 00:40 | |
| *** slaweq_ has joined #openstack-infra | 00:41 | |
| *** numans has joined #openstack-infra | 00:46 | |
| *** ijw has quit IRC | 00:48 | |
| *** hongbin has joined #openstack-infra | 00:50 | |
| *** ijw_ has joined #openstack-infra | 00:50 | |
| *** bobh has quit IRC | 00:53 | |
| *** ijw_ has quit IRC | 00:53 | |
| *** bobh has joined #openstack-infra | 00:55 | |
| jeblair | hrm, that still did the same thing | 00:56 |
| *** kiennt26 has joined #openstack-infra | 00:57 | |
| jeblair | i'm going to stop zuulv3 | 00:57 |
| kazsh | AJaeger: Zuul gave -1 for my new change & is saying that my new project must first be added to the main configuration file by the Zuul administrator. I guess the file is zuul/main.yaml, should I add my new project to this in my change or wait for your cooperation ? | 00:58 |
| pabelanger | okay, EOD for me. looking forward to the morning | 00:59 |
| openstackgerrit | Merged openstack-infra/release-tools master: Fix abandoning reviews in eol_branch https://review.openstack.org/508067 | 01:00 |
| *** markvoelker has joined #openstack-infra | 01:01 | |
| *** kiennt26 has quit IRC | 01:02 | |
| *** kiennt26 has joined #openstack-infra | 01:03 | |
| *** lathiat_ is now known as lathiat | 01:03 | |
| *** cshastri has joined #openstack-infra | 01:05 | |
| *** slaweq_ has quit IRC | 01:13 | |
| SamYaple | does 'recheck' not trigger a zuulv3 recheck? | 01:15 |
| SamYaple | do a need to do a 'recheck zuul' or something? | 01:15 |
| *** lnxnut has joined #openstack-infra | 01:18 | |
| jeblair | SamYaple: i stopped it; i'm restarting it now | 01:18 |
| *** slaweq_ has joined #openstack-infra | 01:18 | |
| SamYaple | ah. cool. | 01:19 |
| jeblair | SamYaple: when running recheck will work | 01:19 |
| *** cuongnv has joined #openstack-infra | 01:23 | |
| *** numans has quit IRC | 01:25 | |
| *** Mutter_ has joined #openstack-infra | 01:25 | |
| *** numans has joined #openstack-infra | 01:25 | |
| *** nicolasbock_ has quit IRC | 01:26 | |
| *** Kevin_Zheng has joined #openstack-infra | 01:27 | |
| *** lnxnut has quit IRC | 01:29 | |
| *** ekcs has quit IRC | 01:29 | |
| *** numans has quit IRC | 01:30 | |
| jeblair | okay, that looks better (i did not reinstall correctly) | 01:33 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Speed configuration building https://review.openstack.org/509309 | 01:33 |
| jeblair | that's with a test for the bug | 01:33 |
| *** Mutter_ has quit IRC | 01:34 | |
| *** bobh has quit IRC | 01:34 | |
| *** sree has joined #openstack-infra | 01:34 | |
| *** Mutter_ has joined #openstack-infra | 01:35 | |
| *** numans has joined #openstack-infra | 01:36 | |
| *** Mutter_ has quit IRC | 01:37 | |
| *** gouthamr has quit IRC | 01:37 | |
| *** iyamahat__ has quit IRC | 01:37 | |
| *** yamahata has quit IRC | 01:39 | |
| *** sree has quit IRC | 01:42 | |
| *** stakeda has joined #openstack-infra | 01:44 | |
| *** pahuang has quit IRC | 01:45 | |
| mordred | clarkb: new fetch-tox-output bug? | 01:51 |
| clarkb | mordred: ya look at docs ans pep8 etc jobs | 01:51 |
| clarkb | they are unhappy | 01:51 |
| clarkb | on v3 | 01:51 |
| *** slaweq_ has quit IRC | 01:51 | |
| mnaser | mordred clarkb i made a change that was accepted could i have broken it? | 01:52 |
| mnaser | for releasenotes at least https://review.openstack.org/#/c/509045/ | 01:52 |
| *** slaweq_ has joined #openstack-infra | 01:54 | |
| mnaser | i could swear there was an email out about the effect of any zuul -1's on a review | 01:56 |
| mordred | mnaser: nah - I don't think that broke anything | 01:56 |
| mnaser | i just approved a change that had a -1 from zuul(v3) and +1 from jenkins and i didnt see it show up in the gate | 01:56 |
| mordred | clarkb: pep8 and docs are passing in v3 now | 01:57 |
| mnaser | https://review.openstack.org/#/c/509122 .. is that expected? | 01:57 |
| *** pahuang has joined #openstack-infra | 01:57 | |
| mordred | mnaser: what do you mean? | 01:57 |
| mnaser | mordred i approved that change, but i dont see it in zuul.openstack.org | 01:58 |
| mnaser | of course it appears now. | 01:58 |
| mnaser | i am sorry | 01:58 |
| mordred | :) | 01:58 |
| mnaser | i promise to be more patient | 01:58 |
| * mordred helped! | 01:58 | |
| mnaser | (in my defense, it was ~7 minutes before it popped up :p) | 01:58 |
| mordred | dude - no defense needed | 01:59 |
| mnaser | if any infra folks have spare cycles now that zuulv3 stuff is a bit quiet... https://review.openstack.org/#/c/507864/ | 02:06 |
| *** kiennt26 has quit IRC | 02:12 | |
| *** kiennt26 has joined #openstack-infra | 02:13 | |
| *** baoli has joined #openstack-infra | 02:13 | |
| *** verdurin has quit IRC | 02:17 | |
| *** SumitNaiksatam has joined #openstack-infra | 02:21 | |
| *** lnxnut has joined #openstack-infra | 02:26 | |
| *** slaweq_ has quit IRC | 02:28 | |
| *** yamamoto has joined #openstack-infra | 02:28 | |
| *** pcrews has quit IRC | 02:28 | |
| *** verdurin has joined #openstack-infra | 02:30 | |
| *** slaweq_ has joined #openstack-infra | 02:32 | |
| *** aeng_mtg is now known as aeng | 02:35 | |
| *** lnxnut has quit IRC | 02:36 | |
| yamamoto | what's 22348 in "Removed Verified-2 by Zuul (22348)" ? | 02:41 |
| SamYaple | yamamoto: when infra turned off zuulv3 jobs and switched back to v2.5/jenkins, went through and removed all the -2 votes from zuul | 02:43 |
| SamYaple | can be safely ignored | 02:43 |
| yamamoto | i'm just curious what the number means. | 02:53 |
| *** MasterOfBugs has quit IRC | 02:56 | |
| SamYaple | -2 is a DO NOT MERGE flag. if you get a -2 on anything, you can't merge it (unless you are infra, they sometimes force merge -2 stuff to make things work again) | 02:56 |
| SamYaple | basically with -2 in place, there is no way to automatically merge that patch | 02:57 |
| *** ihrachys has quit IRC | 03:01 | |
| *** erlon has quit IRC | 03:01 | |
| fungi | yamamoto: i hadn't noticed the (22348) in those, but those comment messages were automatically added by gerrit when i deleted the -2 votes from zuul v3 earlier | 03:03 |
| fungi | i have no idea why it added that number as part of the comment | 03:04 |
| yamamoto | i see. thank you. | 03:04 |
| yamamoto | are there metrics available with zuulv3 jobs? | 03:04 |
| yamamoto | eg. an equivalent of stats_counts.zuul.pipeline.check.job.gate-networking-midonet-python27-ubuntu-xenial.FAILURE for the new "openstack-tox-py27" | 03:06 |
| *** numans has quit IRC | 03:11 | |
| *** numans has joined #openstack-infra | 03:11 | |
| *** sshnaidm|afk is now known as sshnaidm|pto | 03:19 | |
| *** ramishra has joined #openstack-infra | 03:20 | |
| *** baoli has quit IRC | 03:30 | |
| *** pahuang has quit IRC | 03:32 | |
| *** kiennt26 has quit IRC | 03:33 | |
| *** lnxnut has joined #openstack-infra | 03:34 | |
| *** leifmadsen has quit IRC | 03:34 | |
| *** leifmadsen has joined #openstack-infra | 03:35 | |
| *** hongbin has quit IRC | 03:36 | |
| *** slaweq_ has quit IRC | 03:36 | |
| *** Kevin_Zheng has quit IRC | 03:37 | |
| *** slaweq_ has joined #openstack-infra | 03:38 | |
| *** udesale has joined #openstack-infra | 03:44 | |
| *** lnxnut has quit IRC | 03:44 | |
| *** boris_42_ has quit IRC | 03:44 | |
| *** gildub has joined #openstack-infra | 03:48 | |
| *** pahuang has joined #openstack-infra | 03:49 | |
| *** coolsvap has joined #openstack-infra | 03:53 | |
| *** dbecker has quit IRC | 03:54 | |
| openstackgerrit | Omer Anson proposed openstack-infra/project-config master: Add neutron to required-projects for dragonflow https://review.openstack.org/508785 | 03:57 |
| *** numans has quit IRC | 03:58 | |
| *** numans has joined #openstack-infra | 04:00 | |
| *** armax has quit IRC | 04:05 | |
| *** numans has quit IRC | 04:05 | |
| *** dbecker has joined #openstack-infra | 04:08 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Use new infra pipelines https://review.openstack.org/509224 | 04:10 |
| *** slaweq_ has quit IRC | 04:10 | |
| *** numans has joined #openstack-infra | 04:12 | |
| *** ykarel has joined #openstack-infra | 04:12 | |
| *** mnaser has quit IRC | 04:15 | |
| *** slaweq_ has joined #openstack-infra | 04:16 | |
| *** tristanC has quit IRC | 04:16 | |
| *** iyamahat has joined #openstack-infra | 04:17 | |
| yamamoto | /etc/nodepool/provider is not available in case of zuul v3? | 04:21 |
| *** iyamahat_ has joined #openstack-infra | 04:22 | |
| *** aeng has quit IRC | 04:23 | |
| *** pahuang has quit IRC | 04:23 | |
| *** tristanC has joined #openstack-infra | 04:23 | |
| *** mnaser has joined #openstack-infra | 04:24 | |
| *** iyamahat has quit IRC | 04:25 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Print environment for legacy-grenade-dsvm-ironic https://review.openstack.org/509007 | 04:25 |
| *** aeng has joined #openstack-infra | 04:26 | |
| *** iyamahat_ has quit IRC | 04:30 | |
| *** claudiub has joined #openstack-infra | 04:30 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add some Zuul v2 checking to project-config https://review.openstack.org/509244 | 04:32 |
| openstackgerrit | Merged openstack-infra/project-config master: Move infra-specs and infra-manual to v3 only https://review.openstack.org/509258 | 04:32 |
| openstackgerrit | Merged openstack-infra/project-config master: remove usage of check pipeline from zuul-jobs https://review.openstack.org/509274 | 04:32 |
| openstackgerrit | Merged openstack-infra/project-config master: finish releasenotes job conversion https://review.openstack.org/509145 | 04:34 |
| *** pahuang has joined #openstack-infra | 04:36 | |
| *** pcrews has joined #openstack-infra | 04:37 | |
| *** yamahata has joined #openstack-infra | 04:37 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Set default on fetch-tox-output to venv https://review.openstack.org/509177 | 04:38 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add TODO note about reworking fetch-tox-output https://review.openstack.org/509237 | 04:38 |
| *** numans has quit IRC | 04:39 | |
| *** lnxnut has joined #openstack-infra | 04:40 | |
| *** numans has joined #openstack-infra | 04:42 | |
| *** ekcs has joined #openstack-infra | 04:43 | |
| openstackgerrit | Merged openstack-infra/zuul master: Use new infra pipelines https://review.openstack.org/509223 | 04:43 |
| *** jascott1 has quit IRC | 04:45 | |
| *** jascott1 has joined #openstack-infra | 04:45 | |
| *** jascott1 has quit IRC | 04:49 | |
| *** slaweq_ has quit IRC | 04:49 | |
| *** slaweq_ has joined #openstack-infra | 04:53 | |
| *** jascott1 has joined #openstack-infra | 04:53 | |
| *** pgadiya has joined #openstack-infra | 04:56 | |
| *** sree has joined #openstack-infra | 05:04 | |
| openstackgerrit | Merged openstack-infra/project-config master: Fail if Zuul v2 frozen files are touched https://review.openstack.org/509158 | 05:04 |
| *** lnxnut has quit IRC | 05:06 | |
| *** psachin has joined #openstack-infra | 05:15 | |
| *** aeng has quit IRC | 05:17 | |
| *** ekcs has quit IRC | 05:19 | |
| *** aeng has joined #openstack-infra | 05:19 | |
| *** eumel8 has joined #openstack-infra | 05:20 | |
| *** markvoelker has quit IRC | 05:20 | |
| *** eumel8 has quit IRC | 05:20 | |
| openstackgerrit | Hiroaki Kobayashi proposed openstack-infra/project-config master: Add required-projects to the blazar-dashboard jobs https://review.openstack.org/509078 | 05:22 |
| *** slaweq_ has quit IRC | 05:26 | |
| *** sree has quit IRC | 05:31 | |
| *** slaweq_ has joined #openstack-infra | 05:32 | |
| *** sree has joined #openstack-infra | 05:32 | |
| *** lnxnut has joined #openstack-infra | 05:33 | |
| *** zz_ja has quit IRC | 05:35 | |
| *** sree has quit IRC | 05:36 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/openstack-zuul-jobs master: Really only copy logs dir for chef-rake-integration https://review.openstack.org/508841 | 05:36 |
| *** zz_ja has joined #openstack-infra | 05:37 | |
| *** lnxnut has quit IRC | 05:37 | |
| SpamapS | yamamoto: the stats will be available at some point. IIRC there were some firewall rules blocking them before. | 05:44 |
| *** quite has quit IRC | 05:48 | |
| *** numans has quit IRC | 05:50 | |
| *** numans has joined #openstack-infra | 05:51 | |
| *** otherwiseguy has quit IRC | 05:53 | |
| *** apetrich has joined #openstack-infra | 05:54 | |
| *** _ari_ has quit IRC | 05:54 | |
| *** kashyap has quit IRC | 05:55 | |
| *** rnoriega has quit IRC | 05:55 | |
| *** migi has quit IRC | 05:55 | |
| *** weshay has quit IRC | 05:55 | |
| *** haleyb has quit IRC | 05:55 | |
| *** pabelanger has quit IRC | 05:56 | |
| *** pahuang has quit IRC | 06:03 | |
| *** udesale__ has joined #openstack-infra | 06:05 | |
| *** slaweq_ has quit IRC | 06:05 | |
| *** udesale has quit IRC | 06:06 | |
| yamamoto | SpamapS: thank you. do you know how the path of "openstack-tox-py27 for this specific project" will look like? | 06:06 |
| *** sree has joined #openstack-infra | 06:07 | |
| *** dhajare has joined #openstack-infra | 06:07 | |
| *** slaweq_ has joined #openstack-infra | 06:11 | |
| *** rm_work has quit IRC | 06:11 | |
| *** aeng has quit IRC | 06:12 | |
| *** rm_work has joined #openstack-infra | 06:14 | |
| *** numans has quit IRC | 06:15 | |
| *** Apoorva has joined #openstack-infra | 06:15 | |
| openstackgerrit | yatin proposed openstack-infra/openstack-zuul-jobs master: Remove legacy magnum jobs https://review.openstack.org/509341 | 06:18 |
| *** numans has joined #openstack-infra | 06:18 | |
| *** Apoorva has quit IRC | 06:20 | |
| *** pahuang has joined #openstack-infra | 06:20 | |
| *** rnoriega has joined #openstack-infra | 06:20 | |
| *** weshay has joined #openstack-infra | 06:20 | |
| *** kashyap has joined #openstack-infra | 06:20 | |
| *** haleyb has joined #openstack-infra | 06:20 | |
| *** _ari_ has joined #openstack-infra | 06:20 | |
| *** pabelanger has joined #openstack-infra | 06:21 | |
| openstackgerrit | yatin proposed openstack-infra/project-config master: Remove legacy magnum jobs from pipeline https://review.openstack.org/508804 | 06:21 |
| *** ykarel_ has joined #openstack-infra | 06:23 | |
| *** otherwiseguy has joined #openstack-infra | 06:24 | |
| *** udesale__ has quit IRC | 06:25 | |
| *** udesale__ has joined #openstack-infra | 06:25 | |
| *** ykarel has quit IRC | 06:25 | |
| AJaeger | infra-root, I do not see the infra pipelines anymore on zuulv3.openstack.org ;( | 06:26 |
| *** migi has joined #openstack-infra | 06:27 | |
| * AJaeger will not approve anything for now | 06:30 | |
| *** rcernin has joined #openstack-infra | 06:30 | |
| AJaeger | yolanda, ianw, let's abstain from approving anything | 06:30 |
| AJaeger | #status log The special infra pipelines in zuul v3 have disappared | 06:31 |
| openstackstatus | AJaeger: finished logging | 06:31 |
| ianw | ++ | 06:31 |
| openstackgerrit | yatin proposed openstack-infra/project-config master: Remove legacy magnum jobs from pipeline https://review.openstack.org/508804 | 06:32 |
| *** edmondsw has joined #openstack-infra | 06:33 | |
| yolanda | ok, is that related with the rollback email i just saw? | 06:34 |
| AJaeger | infra-root, these pipelines exists before at 4:00 UTC, then I rechecked and things merged around 4:30 - and now I'm back online and see them missing. | 06:34 |
| *** lnxnut has joined #openstack-infra | 06:34 | |
| openstackgerrit | yatin proposed openstack-infra/openstack-zuul-jobs master: Remove legacy magnum jobs https://review.openstack.org/509341 | 06:35 |
| AJaeger | yolanda: yes. We run now both zuul v3 and v2 - v3 on project-config, openstack-zuul-jobs, infra-manual, infra-specs - and v2 on everything. v3 runs check on all repos but does not gate. | 06:35 |
| AJaeger | yolanda: for that mordred created some specific pipelines for merging of the v3 only repos and they are not visible in zuulv3.openstack.org anymore (but were there 90 mins ago). | 06:36 |
| *** eumel8 has joined #openstack-infra | 06:36 | |
| AJaeger | yolanda: So, I'll abstain from touching anything for now and let the US folks figure out which of the changes merged was wrong (I double checked and don't see anything obvious) and needs a revert. | 06:37 |
| yolanda | sure | 06:37 |
| AJaeger | yolanda: if you review: We only merge project-config v2 changes if there is an update to v3 as well. | 06:37 |
| *** edmondsw has quit IRC | 06:37 | |
| yolanda | and needs to have green light for zuul v3 as well right? | 06:38 |
| AJaeger | So, no touching of jenkins/jobs or zuul/layout.yaml without doing the change elsewhere in the same page or before | 06:38 |
| AJaeger | project-config et al will only merge with zuul v3 +1. All other repos: Zuul v3 is treated as third-party CI, so only Jenkins (Zuul v2) +1 is needed. | 06:38 |
| *** pahuang has quit IRC | 06:39 | |
| eumel8 | morning | 06:39 |
| eumel8 | is there an extra command for recheck zuul in gerrit? | 06:40 |
| AJaeger | just "recheck" | 06:40 |
| AJaeger | morning, eumel8 ! | 06:40 |
| yolanda | AJaeger, i understand. Only the changes that want to change the project-config layout, need zuul v2/v3 | 06:40 |
| yolanda | thanks | 06:40 |
| AJaeger | yolanda: yes. | 06:42 |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate master: Use pip list for saved output https://review.openstack.org/509349 | 06:42 |
| eumel8 | AJaeger: That works for Jenkins but not for Zuul in this: https://review.openstack.org/#/c/506795/ | 06:42 |
| AJaeger | eumel8: http://zuulv3.openstack.org/ shows 506795 | 06:43 |
| *** slaweq_ has quit IRC | 06:43 | |
| *** jgriffit1 has joined #openstack-infra | 06:44 | |
| *** namnh has joined #openstack-infra | 06:44 | |
| AJaeger | eumel8: or in other words: zuul v3 has less nodes and might be slower, or get restarted while we debug issues. So, have patience and double check, please | 06:45 |
| *** numans has quit IRC | 06:45 | |
| * AJaeger goes offline for a while | 06:45 | |
| eumel8 | AJaeger: ah, it's still running but indeed very very slow | 06:45 |
| frickler | was there a change to zuul-cloner for v3 that has not been reverted? chef jobs seem to be failing now on v2, didn't have that earlier http://logs.openstack.org/63/508463/4/check/gate-openstack-chef-repo-chef-rake/2f98257/console.html#_2017-10-04_05_42_29_357015 | 06:46 |
| frickler | eumel8: yes, zuulv3 has only 20% of the total nodepool capacity | 06:46 |
| *** numans has joined #openstack-infra | 06:48 | |
| *** pcaruana has joined #openstack-infra | 06:48 | |
| ianw | frickler: sudo -E /usr/zuul-env/bin/zuul-cloner -m clonemap.yaml --cache-dir /opt/git --workspace /home/jenkins/workspace/ git://git.openstack.org openstack/ | 06:48 |
| ianw | ^ openstack/ what? seems to be missing something at the end there | 06:48 |
| *** jtomasek has joined #openstack-infra | 06:50 | |
| *** slaweq_ has joined #openstack-infra | 06:51 | |
| *** udesale has joined #openstack-infra | 06:51 | |
| *** ykarel__ has joined #openstack-infra | 06:52 | |
| *** pahuang has joined #openstack-infra | 06:52 | |
| *** udesale__ has quit IRC | 06:53 | |
| *** vsaienk0 has joined #openstack-infra | 06:53 | |
| *** spectr has quit IRC | 06:54 | |
| *** andreas_s has joined #openstack-infra | 06:54 | |
| *** ykarel_ has quit IRC | 06:54 | |
| *** spectr has joined #openstack-infra | 06:55 | |
| *** lnxnut has quit IRC | 06:57 | |
| *** shardy_afk is now known as shardy | 07:01 | |
| *** ykarel__ is now known as ykarel | 07:01 | |
| *** udesale__ has joined #openstack-infra | 07:02 | |
| *** udesale has quit IRC | 07:04 | |
| *** florianf has joined #openstack-infra | 07:06 | |
| *** martinkopec has joined #openstack-infra | 07:07 | |
| *** ccamacho has joined #openstack-infra | 07:07 | |
| *** hashar has joined #openstack-infra | 07:08 | |
| *** ykarel is now known as ykarel|meeting | 07:08 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Print status of playbook run in console https://review.openstack.org/507912 | 07:08 |
| *** Hal has joined #openstack-infra | 07:10 | |
| *** Hal is now known as Guest41920 | 07:11 | |
| *** martinkopec has quit IRC | 07:12 | |
| openstackgerrit | Frank Kloeker proposed openstack-infra/openstack-zuul-jobs master: Add another missing repo to legacy-openstackci-beaker jobs https://review.openstack.org/509358 | 07:13 |
| eumel8 | a new one... | 07:13 |
| *** tesseract has joined #openstack-infra | 07:16 | |
| *** sdake_ has joined #openstack-infra | 07:16 | |
| *** sdake_ is now known as Guest52279 | 07:17 | |
| openstackgerrit | Anshul Jain proposed openstack/diskimage-builder master: Update proliant-tools to support Gen10 Proliant servers https://review.openstack.org/509359 | 07:17 |
| *** Guest52279 has quit IRC | 07:17 | |
| *** aviau has quit IRC | 07:19 | |
| *** aviau has joined #openstack-infra | 07:19 | |
| *** markvoelker has joined #openstack-infra | 07:21 | |
| *** kiennt26 has joined #openstack-infra | 07:22 | |
| *** slaweq_ has quit IRC | 07:24 | |
| *** martinkopec has joined #openstack-infra | 07:25 | |
| *** thorre has quit IRC | 07:28 | |
| *** slaweq_ has joined #openstack-infra | 07:28 | |
| *** thorre has joined #openstack-infra | 07:29 | |
| ttx | After the v2 revert there seems to be node attrition on -upstream-translation-update and -rtfd post-jobs | 07:30 |
| *** gildub has quit IRC | 07:30 | |
| ttx | same for the release-post jobs. I suspect we lost the special hosts | 07:31 |
| frickler | ianw: ah, right, our parsing of ZUUL_CHANGES fails for a dependency that is not openstack/*, will fix, thx for checking | 07:34 |
| *** jascott1 has quit IRC | 07:38 | |
| *** jascott1 has joined #openstack-infra | 07:39 | |
| *** jascott1 has quit IRC | 07:43 | |
| *** jpena|off is now known as jpena | 07:44 | |
| *** kzaitsev1pi has joined #openstack-infra | 07:44 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: WIP: Fix TripleO CI jobs https://review.openstack.org/508660 | 07:44 |
| *** kzaitsev_pi has quit IRC | 07:47 | |
| *** ralonsoh has joined #openstack-infra | 07:47 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/openstack-zuul-jobs master: Fix puppet jobs by adding integration parent https://review.openstack.org/509173 | 07:48 |
| *** egonzalez has joined #openstack-infra | 07:52 | |
| *** markvoelker has quit IRC | 07:55 | |
| *** jpich has joined #openstack-infra | 07:55 | |
| *** ykarel|meeting is now known as ykarel | 07:57 | |
| *** isaacb has joined #openstack-infra | 07:57 | |
| *** pratik has joined #openstack-infra | 07:59 | |
| pratik | @yolanda can you please review https://review.openstack.org/#/c/509159/ when you get a chance? | 08:01 |
| *** udesale has joined #openstack-infra | 08:02 | |
| *** udesale__ has quit IRC | 08:03 | |
| openstackgerrit | Thomas Morin proposed openstack-infra/project-config master: Add projects to required-projects for networking-(bagpipe|bgpvpn) https://review.openstack.org/508825 | 08:06 |
| yolanda | hi pratik , we have stopped reviews for today, until the issues with zuul v3 have been resolved, that is likely to happen this afternoon | 08:07 |
| pratik | okay | 08:07 |
| frickler | infra-root: this nova-gate job has a post-failure, maybe more needs to be reverted for docs jobs? can't find any useful logs there http://logs.openstack.org/03/505903/1/gate/gate-nova-docs-ubuntu-xenial/2fa9ce7/ | 08:12 |
| *** namnh has quit IRC | 08:14 | |
| *** pas-ha has left #openstack-infra | 08:15 | |
| *** numans has quit IRC | 08:16 | |
| *** numans has joined #openstack-infra | 08:17 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Make sure we always have /etc/machine-id available. https://review.openstack.org/508226 | 08:18 |
| dmellado | yolanda: so what happened in the end with the revert? | 08:21 |
| dmellado | full revert in the end? | 08:21 |
| *** edmondsw has joined #openstack-infra | 08:21 | |
| AJaeger | dmellado: We rolled the Zuul v3 move partially back - please read http://lists.openstack.org/pipermail/openstack-dev/2017-October/123049.html and http://lists.openstack.org/pipermail/openstack-dev/2017-October/123064.html | 08:22 |
| AJaeger | infra-root, strange thing: zuul comments on changes, including on some zuul projects. But I do not see the queues in zuulv3 status page. | 08:23 |
| dmellado | thanks AJaeger | 08:23 |
| *** AJaeger has quit IRC | 08:24 | |
| dmellado | how we can make sure that zuulv3 runs the project? just wait or do you plan to add a new keyword | 08:25 |
| dmellado | ? | 08:25 |
| *** edmondsw has quit IRC | 08:26 | |
| *** lucas-afk is now known as lucasagomes | 08:27 | |
| *** lnxnut has joined #openstack-infra | 08:28 | |
| *** AJaeger has joined #openstack-infra | 08:28 | |
| tobiash | AJaeger: the pipelines didn't vanish, the ui seems to be broken: http://paste.openstack.org/show/622611/ | 08:29 |
| AJaeger | tobiash: what a relief - I was suspecting that already since zuul was commenting but didn't had your means to figure it out ;) | 08:30 |
| AJaeger | infra-root ^ | 08:30 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/openstack-zuul-jobs master: Add release-notes-jobs-neutron https://review.openstack.org/509165 | 08:32 |
| *** isaacb has quit IRC | 08:32 | |
| tobiash | AJaeger: jq is a pretty cool json status processor which can be part of a pipe :) | 08:33 |
| tobiash | curl -s http://zuulv3.openstack.org/status.json status.json| jq '.pipelines[].name' | 08:33 |
| *** slaweq_ has quit IRC | 08:33 | |
| tobiash | s/json status processor/json processor/ | 08:34 |
| AJaeger | tobiash, infra-root: still raises the questions what happened with https://review.openstack.org/509057 , https://review.openstack.org/509007 https://review.openstack.org/509182 https://review.openstack.org/509221 - I miss voting/merging on these ;( | 08:34 |
| AJaeger | tobiash: neat | 08:34 |
| tobiash | AJaeger: according to status.json your first review is still in infra-gate queue at first position | 08:37 |
| AJaeger | for ages - since 4 hours ;( | 08:37 |
| AJaeger | reading json output is hard ;( | 08:38 |
| tobiash | AJaeger: that's the job that is hanging/waiting: http://paste.openstack.org/show/622612/ | 08:38 |
| *** pratik has quit IRC | 08:38 | |
| *** stakeda has quit IRC | 08:39 | |
| tobiash | it possibly waits for a node from nodepool or something getting the node for this job is stuck | 08:39 |
| AJaeger | I let it hang and ask jeblair or others to check why it hangs, seen that pattern several times where somehow a node is not allocated/started. With the partial revert, the experts might be able to debug this... | 08:40 |
| tobiash | hm, worker should be a ze*.o.o | 08:40 |
| tobiash | so maybe an executor grabbed the job and got stuck | 08:40 |
| AJaeger | and broke the webui with it ;( | 08:42 |
| AJaeger | tobiash: what's the jq filter to just get a queue like infra-gate? | 08:42 |
| tobiash | AJaeger: did 'jq . | less' and searched for the change number | 08:43 |
| tobiash | was too lazy to find out the correct query | 08:43 |
| *** slaweq_ has joined #openstack-infra | 08:43 | |
| tobiash | I'm sure it's possible with a search query but jq can be hard sometimes ;) | 08:43 |
| AJaeger | tobiash: yeah, that works as well - I had hoped you would know the short cut ;) | 08:43 |
| *** numans has quit IRC | 08:45 | |
| *** lnxnut has quit IRC | 08:47 | |
| *** numans has joined #openstack-infra | 08:48 | |
| tobiash | AJaeger: that's the zuul queue of the infra-gate pipeline: curl -s http://zuulv3.openstack.org/status.json status.json | jq '.pipelines | map(select(.name == "infra-gate")) | .[].change_queues | map(select(.name == "openstack-infra/zuul"))' | 08:49 |
| *** dfflanders has joined #openstack-infra | 08:50 | |
| AJaeger | wow! Thanks, tobiash ! | 08:50 |
| *** jaosorior has quit IRC | 08:51 | |
| *** ykarel has quit IRC | 08:52 | |
| *** Swimingly has quit IRC | 08:52 | |
| *** derekh has joined #openstack-infra | 08:52 | |
| *** markvoelker has joined #openstack-infra | 08:52 | |
| *** tmorin has joined #openstack-infra | 08:55 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/project-config master: Support trigger to recheck zuulv3 only https://review.openstack.org/509379 | 08:55 |
| *** jaosorior has joined #openstack-infra | 08:56 | |
| *** electrofelix has joined #openstack-infra | 08:57 | |
| tmorin | hi infraroot (AJaeger?) : could someone have a look at https://review.openstack.org/508825 ? I think it is ready now | 08:58 |
| *** Swimingly has joined #openstack-infra | 08:58 | |
| *** Swimingly has quit IRC | 08:58 | |
| *** Swimingly has joined #openstack-infra | 08:58 | |
| tmorin | infraroot: https://review.openstack.org/#/c/509167/ is ready as well (has got AJaeger +2 blessing already) | 08:58 |
| AJaeger | tmorin: we'll review later, first need to debug something with Zuul v3 | 08:59 |
| * AJaeger goes offline again | 08:59 | |
| *** lukebrowning_ has joined #openstack-infra | 09:00 | |
| tmorin | AJaeger: would you suggest someone else I could ask ? | 09:00 |
| *** dizquierdo has joined #openstack-infra | 09:00 | |
| *** yamamoto has quit IRC | 09:01 | |
| tmorin | yolanda: would you have time to at https://review.openstack.org/#/c/509167/ and https://review.openstack.org/508825 ? (required to unbreak CI for networking-bagpipe and networking-bgpvpn) | 09:01 |
| yolanda | hi tmorin , no reviews for today as Andreas said... we need for zuul v3 to work before approving any more changes | 09:02 |
| yolanda | sorry | 09:02 |
| *** lukebrowning has quit IRC | 09:02 | |
| tmorin | yolanda: hi, ok, I wasn't aware this was for the whole infra team... thanks! | 09:03 |
| *** tmorin has quit IRC | 09:08 | |
| *** slaweq_ has quit IRC | 09:16 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul-jobs master: Set zuul_log_path for a periodic job https://review.openstack.org/509384 | 09:17 |
| *** pgadiya has quit IRC | 09:21 | |
| ttx | so main issue at this point is that post jobs are stuck with some jobs never getting a host to run ? | 09:21 |
| *** e0ne has joined #openstack-infra | 09:22 | |
| sshnaidm|pto | does anybody know where are logs of legacy periodic jobs now? like legacy-periodic-... | 09:22 |
| *** slaweq_ has joined #openstack-infra | 09:23 | |
| frickler | sshnaidm|pto: there are logs for new v2 period jobs running today at http://logs.openstack.org/periodic/, I haven't found anything for zuulv3 periodic jobs, looked to me like they might have still been broken | 09:25 |
| andreaf | good morning! | 09:25 |
| sshnaidm|pto | frickler, thanks, didn't find them too.. | 09:25 |
| andreaf | something may be wrong on test images - http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%20%5C%22c%2F_cffi_backend.c%3A15%3A17%3A%20fatal%20error%3A%20ffi.h%3A%20No%20such%20file%20or%20directory%5C%22 | 09:26 |
| *** markvoelker has quit IRC | 09:26 | |
| andreaf | http://logs.openstack.org/06/508906/5/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/90ae2e3/console.html#_2017-10-04_09_08_35_431852 | 09:26 |
| frickler | ttx: yes, that seems to be the major remaining issue with the rolled-back zuul v2 setup now | 09:26 |
| andreaf | c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory | 09:26 |
| *** yamamoto has joined #openstack-infra | 09:27 | |
| kaisers1 | andreaf: aah, just came here to ask about that. That was a quick answer, i got it before i even asked... :) | 09:31 |
| andreaf | yolanda: ^^^ do you know if anything changed? Could it be that the images were not built properly this morning? | 09:31 |
| frickler | andreaf: kaisers1: do you also see this error on other patches? | 09:32 |
| andreaf | 500 hits since 10am UTC this morning on different jobs, mostly on ubuntu-xenial | 09:32 |
| *** pgadiya has joined #openstack-infra | 09:33 | |
| *** pgadiya has quit IRC | 09:33 | |
| frickler | andreaf: ah, missed your logstash link, right | 09:33 |
| andreaf | and both console.html and job-output so it's not zuul version specific :) | 09:33 |
| * andreaf afk for 30min | 09:34 | |
| *** iyamahat has joined #openstack-infra | 09:35 | |
| *** dizquierdo has quit IRC | 09:36 | |
| *** yamahata has quit IRC | 09:38 | |
| jzerebecki | the version of cffi that fails is new. before the failure it would download a wheel cffi-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl , afterwards cffi-1.11.1.tar.gz and build that. locally that builds fine. | 09:38 |
| *** sdague has joined #openstack-infra | 09:38 | |
| AJaeger | so, devstack misses a dependency on the ffi devel package - libffi-dev on Ubuntu. Anybody wants to patch devstack for that? | 09:39 |
| *** hashar is now known as hasharAway | 09:39 | |
| *** iyamahat has quit IRC | 09:40 | |
| *** jascott1 has joined #openstack-infra | 09:42 | |
| *** pblaho has joined #openstack-infra | 09:42 | |
| cmurphy | AJaeger: it's not just devstack, it happens when zuul/jenkins try to install ansible on the node before devstack even starts | 09:43 |
| *** tosky has joined #openstack-infra | 09:44 | |
| *** lnxnut has joined #openstack-infra | 09:45 | |
| *** lnxnut has quit IRC | 09:56 | |
| *** slaweq_ has quit IRC | 09:56 | |
| *** tikitavi has joined #openstack-infra | 09:57 | |
| *** pbourke has quit IRC | 09:57 | |
| *** pbourke has joined #openstack-infra | 09:59 | |
| tikitavi | hi! can anybody help with https://review.openstack.org/#/c/500091/ ? There is a problem with instances that are stuck in pending state | 10:00 |
| *** kiennt26 has quit IRC | 10:01 | |
| frickler | so the bug is that cffi-1.11.0 was released only as tar.gz, that should have nothing to do with our images. except that they are (and for some longer time have been) missing proper libs in order to install cffi from source. | 10:01 |
| frickler | the failing command is http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate-wrap.sh#n479 , maybe as a workaround we can install cffi=1.10.0 before that | 10:02 |
| *** cuongnv has quit IRC | 10:06 | |
| jzerebecki | frickler: that will not work, it would still try to install the newer cffi afterwards | 10:07 |
| jzerebecki | however installing libffi-dev before would work. but how? does that script only need to work on ubuntu? is bindep available at that point? | 10:09 |
| *** edmondsw has joined #openstack-infra | 10:09 | |
| frickler | jzerebecki: it does work for me if I test it locally | 10:11 |
| *** pt_15 has joined #openstack-infra | 10:13 | |
| *** edmondsw has quit IRC | 10:13 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/devstack-gate master: Work around cffi install error https://review.openstack.org/509394 | 10:14 |
| andreaf | unless we have an install that forces update, installing 1.11.0 before should be enough to prevent 1.11.1 from being pulled in | 10:15 |
| andreaf | frickler: thanks | 10:16 |
| jzerebecki | mh it didn't work here unless I had it together on the same pip install command | 10:17 |
| frickler | looks like others are affected, too, so hopefully this will get fixed by upstream https://bitbucket.org/cffi/cffi/issues/339/no-wheel-packages-for-1111-version | 10:20 |
| *** markvoelker has joined #openstack-infra | 10:23 | |
| *** ricky_ has quit IRC | 10:26 | |
| *** yamamoto has quit IRC | 10:30 | |
| *** slaweq_ has joined #openstack-infra | 10:31 | |
| *** udesale has quit IRC | 10:32 | |
| *** andreas_s_ has joined #openstack-infra | 10:34 | |
| *** jkilpatr has quit IRC | 10:35 | |
| *** florianf has quit IRC | 10:36 | |
| *** sanket_ has joined #openstack-infra | 10:36 | |
| *** andreas_s has quit IRC | 10:37 | |
| *** tesseract has quit IRC | 10:40 | |
| *** numans has quit IRC | 10:40 | |
| ianw | hmm, actually ffi dev packages look like they're in there ... is this before | 10:41 |
| *** numans has joined #openstack-infra | 10:42 | |
| frickler | ianw: this is before almost everything in "pip install ansible" | 10:42 |
| ianw | yeah ok | 10:42 |
| *** tesseract has joined #openstack-infra | 10:44 | |
| dims | ianw : frickler : so we go with https://review.openstack.org/#/c/509394/ ? | 10:48 |
| ianw | i'm just thinking about it | 10:49 |
| *** dfflanders has quit IRC | 10:49 | |
| *** sambetts|afk is now known as sambetts | 10:50 | |
| *** yamamoto has joined #openstack-infra | 10:51 | |
| *** lucasagomes is now known as lucas-hungry | 10:51 | |
| *** vsaienk0 has quit IRC | 10:52 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Install libffi-dev packages in base image https://review.openstack.org/509399 | 10:52 |
| *** sree has quit IRC | 10:53 | |
| ianw | dims / fricker : ^ there's a longer term idea | 10:53 |
| *** sree has joined #openstack-infra | 10:53 | |
| *** lnxnut has joined #openstack-infra | 10:53 | |
| *** mrunge_ has joined #openstack-infra | 10:54 | |
| ianw | we could get ansible to install that | 10:54 |
| *** mrunge has quit IRC | 10:55 | |
| *** markvoelker has quit IRC | 10:55 | |
| *** vsaienk0 has joined #openstack-infra | 10:56 | |
| jzerebecki | ianw: is there ansible available at that point with zuul v2? | 10:57 |
| *** sree has quit IRC | 10:57 | |
| dims | ianw : +1 on 509399 | 10:57 |
| ianw | no, that's right | 10:57 |
| *** spectr-RH has joined #openstack-infra | 10:59 | |
| *** mat128 has joined #openstack-infra | 11:00 | |
| *** dave-mccowan has joined #openstack-infra | 11:01 | |
| *** spectr has quit IRC | 11:02 | |
| ianw | frickler: ok, thoughts posted on 509394; see if you agree. i'm out sorry, but someone will be here shortly to move it along i'm sure | 11:02 |
| *** lnxnut has quit IRC | 11:03 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add launcher ID to log messages https://review.openstack.org/509406 | 11:03 |
| *** nicolasbock_ has joined #openstack-infra | 11:03 | |
| *** dizquierdo has joined #openstack-infra | 11:05 | |
| *** vsaienk0 has quit IRC | 11:06 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add launcher ID to log messages https://review.openstack.org/509406 | 11:06 |
| *** nicolasbock_ has quit IRC | 11:08 | |
| *** vsaienk0 has joined #openstack-infra | 11:10 | |
| *** jkilpatr has joined #openstack-infra | 11:10 | |
| *** dave-mcc_ has joined #openstack-infra | 11:11 | |
| *** cshastri has quit IRC | 11:11 | |
| *** ociuhandu has quit IRC | 11:11 | |
| *** shardy is now known as shardy_afk | 11:12 | |
| *** hamzy has joined #openstack-infra | 11:12 | |
| *** dave-mccowan has quit IRC | 11:13 | |
| *** vponomaryov1 has joined #openstack-infra | 11:15 | |
| *** vponomaryov1 has quit IRC | 11:16 | |
| *** vponomaryov has quit IRC | 11:16 | |
| *** vponomaryov1 has joined #openstack-infra | 11:16 | |
| *** quite has joined #openstack-infra | 11:17 | |
| *** mrunge_ has quit IRC | 11:18 | |
| *** mrunge has joined #openstack-infra | 11:20 | |
| *** armaan has joined #openstack-infra | 11:21 | |
| *** dtantsur|afk is now known as dtantsur | 11:22 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add launcher ID to log messages https://review.openstack.org/509406 | 11:23 |
| sdague | ianw: is there a way to trigger our mirror to build these wheels? | 11:23 |
| sdague | because you know, gate being dead again after just getting back to working is sad panda | 11:24 |
| *** quite has quit IRC | 11:24 | |
| *** quite has joined #openstack-infra | 11:25 | |
| *** ccamacho is now known as ccamacho|lunch | 11:28 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Allow to pass drivers to enable on dracut https://review.openstack.org/509411 | 11:29 |
| panda | uh ? | 11:29 |
| *** numans has quit IRC | 11:30 | |
| *** numans has joined #openstack-infra | 11:33 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/project-config master: Install libffi-dev packages in base image https://review.openstack.org/509399 | 11:33 |
| sdague | frickler: is there a reason we don't bake ansible itself into the base images? | 11:36 |
| chandankumar | AJaeger: please have a look on this review https://review.openstack.org/#/c/508502/ i am not sure what i am missing for this | 11:37 |
| *** ociuhandu has joined #openstack-infra | 11:37 | |
| sdague | frickler: I'm going to approve the work around patch you proposed. Getting people back up and running is high priority. | 11:38 |
| frickler | sdague: I don't know, other than maybe the generic argument of having as few things as possible in there. | 11:39 |
| *** wolverineav has joined #openstack-infra | 11:39 | |
| sdague | yeh, but ansible is really hard requirement now, and it seems like it should just be available in the base image, not run the risk of install going wrong | 11:39 |
| sdague | but... anyway, +A on https://review.openstack.org/#/c/509394 | 11:40 |
| *** numans has quit IRC | 11:40 | |
| andreaf | sdague: +1 :) | 11:41 |
| andreaf | sdague: by the time I hit sent there was already a reply from you on the email | 11:42 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Allow to pass drivers to enable on dracut https://review.openstack.org/509411 | 11:42 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Allow to pass drivers to enable on dracut https://review.openstack.org/509411 | 11:45 |
| *** numans has joined #openstack-infra | 11:46 | |
| *** ldnunes has joined #openstack-infra | 11:46 | |
| sdague | andreaf: it passed all the voting tests, so as soon as the nv ones get through it will move to gate | 11:47 |
| *** Guest41920 has quit IRC | 11:47 | |
| tikitavi | sorry, once more. Can anybody help with https://review.openstack.org/#/c/500091/ ? There is a problem with instances that are stuck in pending state. We have this problem in this review in this for or another for a month... stable/ocata branch | 11:47 |
| jzerebecki | perhave a different base image with ansible for devstack and still use the slim base for everything else? | 11:47 |
| andreaf | sdague: yay | 11:47 |
| jzerebecki | tikitavi: one of the errors will be fixed by https://review.openstack.org/#/c/509394/1 | 11:48 |
| *** dizquierdo has quit IRC | 11:48 | |
| andreaf | tikitavi: I'm not sure what you mean exactly - instances spawn as part of the test run? do you have a link to the relevant log? | 11:48 |
| *** markvoelker has joined #openstack-infra | 11:53 | |
| sdague | tikitavi: the ec2-api team should get to the bottom of that, it's the functional tests from ec2. pending is not even a nova state. | 11:53 |
| *** jaypipes has joined #openstack-infra | 11:56 | |
| ttx | infra-root: main issue at this point is that post jobs / release-post jobs are stuck with some jobs never getting a host to run | 11:58 |
| ttx | (trying to summarize the backlog) | 11:58 |
| *** rhallisey has joined #openstack-infra | 12:00 | |
| *** lnxnut has joined #openstack-infra | 12:00 | |
| *** ralonsoh has quit IRC | 12:02 | |
| *** lucas-hungry is now known as lucasagomes | 12:03 | |
| *** ldnunes has quit IRC | 12:06 | |
| *** jpena is now known as jpena|lunch | 12:06 | |
| *** ldnunes has joined #openstack-infra | 12:06 | |
| *** trown|outtypewww is now known as trown | 12:07 | |
| *** dizquierdo has joined #openstack-infra | 12:08 | |
| *** slaweq_ has quit IRC | 12:09 | |
| *** dprince has joined #openstack-infra | 12:09 | |
| *** lnxnut has quit IRC | 12:11 | |
| *** tmorin has joined #openstack-infra | 12:11 | |
| *** dbecker has quit IRC | 12:11 | |
| *** dbecker has joined #openstack-infra | 12:12 | |
| *** dizquierdo has quit IRC | 12:13 | |
| *** slaweq_ has joined #openstack-infra | 12:13 | |
| *** dizquierdo has joined #openstack-infra | 12:13 | |
| frickler | ianw: I was thinking about doing the complete ansible venv installation there instead of in devstack-gate, similar to nodepool/elements/infra-package-needs/install.d/40-install-bindep | 12:14 |
| *** tpsilva has joined #openstack-infra | 12:15 | |
| frickler | ah, bad scrollback | 12:15 |
| *** edmondsw has joined #openstack-infra | 12:16 | |
| frickler | clarkb: jeblair: AJaeger: regarding queues not showing up on zuulv3 status page, I'm getting "TypeError: change.id is null" on the JS console: http://paste.openstack.org/show/622646/ , will try to locate the broken change in status.json | 12:17 |
| *** edmondsw_ has joined #openstack-infra | 12:17 | |
| *** tmorin has quit IRC | 12:19 | |
| *** dizquierdo has quit IRC | 12:19 | |
| pabelanger | ttx: AJaeger: this is because we didn't restart zlstatic01.o.o yesterday. Jobs are running now. | 12:20 |
| *** edmondsw has quit IRC | 12:21 | |
| *** udesale has joined #openstack-infra | 12:23 | |
| andreaf | sdague, frickler: https://review.openstack.org/#/c/509394 is not merged yet, and wheels for cffi 1.11.1 are now available | 12:24 |
| andreaf | I think we should stop the patch from merging as the issue should be gone | 12:24 |
| andreaf | https://groups.google.com/forum/#!topic/python-cffi/Up_QyG0HIdc | 12:25 |
| *** ianychoi has quit IRC | 12:25 | |
| frickler | andreaf: I think I'd wait with that until we see a successful run, I'll put up a revert in parallel | 12:26 |
| *** hemna__ has joined #openstack-infra | 12:26 | |
| andreaf | frickler ok | 12:26 |
| pabelanger | why not make sure libffi-dev is installed? | 12:26 |
| frickler | andreaf: just to be on the safe side and avoid another couple of hours gate time if the wheels turn out to be non-working for whatever reason | 12:26 |
| *** ianychoi has joined #openstack-infra | 12:26 | |
| *** markvoelker has quit IRC | 12:26 | |
| pabelanger | we have that in bindep.txt for ansible things for this reason | 12:26 |
| *** markvoelker has joined #openstack-infra | 12:27 | |
| frickler | pabelanger: that is the longterm approach in https://review.openstack.org/#/q/Ia588c55b73643d27bdc3dff57cb00c1748498b37 , we break when installing ansible, so bindep.txt is too late | 12:27 |
| andreaf | pabelanger, frickler: I'd rather stick to binaries and use the --only-binary flags as proposed on the ML - so we can avoid building the packages as part of our CI... it takes long enough already to run integration tests | 12:28 |
| pabelanger | andreaf: well, it will only build if there wheel is missing. Which should only happen on new releases | 12:29 |
| andreaf | pabelanger: fair enough - but we can stick to the older version until the wheels are up | 12:29 |
| pabelanger | sure | 12:30 |
| *** e0ne has quit IRC | 12:31 | |
| *** e0ne has joined #openstack-infra | 12:32 | |
| *** rlandy|bbl is now known as rlandy | 12:34 | |
| frickler | hmm, does gerrit show the "revert" button only after a change has been merged? | 12:34 |
| sdague | frickler: yes | 12:36 |
| sdague | andreaf: here is the thing, the patch does no harm, and can be reverted afterwards | 12:36 |
| sdague | and there is a mirror delay window | 12:36 |
| andreaf | sdague: heh right, good point | 12:37 |
| sdague | I'm always in the camp of shortest-path-to-get-everyone-back-work | 12:37 |
| sdague | then best solution done in parallel after | 12:37 |
| *** camunoz has joined #openstack-infra | 12:38 | |
| *** numans has quit IRC | 12:40 | |
| *** numans has joined #openstack-infra | 12:42 | |
| *** gridinv1 has joined #openstack-infra | 12:43 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/devstack-gate master: Install only wheels for ansible https://review.openstack.org/509429 | 12:45 |
| andreaf | pabelanger: why not https://review.openstack.org/#/c/509379/? it seems like a rather simple change to do and it would really help - now every change in repos with a .zuul.yaml runs all jobs on v2, all legacy- jobs on v3 plus v3 native jobs on v3. A recheck because of a zuulv3 failure require a massive amount of nodes to run | 12:45 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Add launcher ID to log messages https://review.openstack.org/509406 | 12:46 |
| *** slaweq_ has quit IRC | 12:46 | |
| *** gridinv1 has quit IRC | 12:47 | |
| ttx | pabelanger: great, thanks! | 12:48 |
| *** derekh has quit IRC | 12:49 | |
| *** kiennt26 has joined #openstack-infra | 12:49 | |
| *** esberglu has joined #openstack-infra | 12:51 | |
| *** hemna__ has quit IRC | 12:54 | |
| *** rodrigods has quit IRC | 12:54 | |
| *** sree has joined #openstack-infra | 12:55 | |
| *** witek has quit IRC | 12:56 | |
| *** slaweq_ has joined #openstack-infra | 12:57 | |
| *** eharney has joined #openstack-infra | 12:59 | |
| *** spectr-RH has quit IRC | 13:00 | |
| *** mriedem has joined #openstack-infra | 13:02 | |
| *** pcaruana has quit IRC | 13:02 | |
| *** ccamacho|lunch is now known as ccamacho | 13:03 | |
| *** lbragstad has joined #openstack-infra | 13:05 | |
| *** witek has joined #openstack-infra | 13:07 | |
| *** Goneri has joined #openstack-infra | 13:08 | |
| *** lnxnut has joined #openstack-infra | 13:08 | |
| pabelanger | andreaf: right, when we rolled back we specifically wanted jobs to both run on zuulv2.5 and zuulv3, as for nodes, we have 80/20 split between zuulv2.5 and zuulv3. So, even if we did start doing zuulv3 rechecks only, it doesn't have the same capacity and likey just result in the same wait times. But mostly, I'd rather just spend the effort working towards migrating back to zuulv3 then implement | 13:09 |
| pabelanger | things to run 2 zuuls | 13:09 |
| *** krtaylor has joined #openstack-infra | 13:09 | |
| *** hemna_ has joined #openstack-infra | 13:10 | |
| *** mat128 has quit IRC | 13:11 | |
| *** spectr-RH has joined #openstack-infra | 13:12 | |
| *** jpena|lunch is now known as jpena | 13:13 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/zuul feature/zuulv3: Avoid JS error when a change has no id https://review.openstack.org/509435 | 13:14 |
| *** pcaruana has joined #openstack-infra | 13:15 | |
| *** spectr-RH has quit IRC | 13:16 | |
| *** hasharAway has quit IRC | 13:16 | |
| *** rodrigods has joined #openstack-infra | 13:17 | |
| *** lnxnut has quit IRC | 13:17 | |
| *** spectr-RH has joined #openstack-infra | 13:17 | |
| *** rodrigods has quit IRC | 13:17 | |
| *** rodrigods has joined #openstack-infra | 13:17 | |
| *** jcoufal has joined #openstack-infra | 13:17 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/zuul-jobs master: Add a generic stage-artifacts role https://review.openstack.org/509233 | 13:17 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/zuul-jobs master: Add compress capabilities to stage artifacts https://review.openstack.org/509234 | 13:17 |
| *** sree has quit IRC | 13:18 | |
| *** sree has joined #openstack-infra | 13:18 | |
| *** kgiusti has joined #openstack-infra | 13:19 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Add zuul.pipeline and zuul.executor.hostname to job header https://review.openstack.org/509436 | 13:20 |
| *** chlong_ has joined #openstack-infra | 13:22 | |
| *** sree has quit IRC | 13:22 | |
| *** lnxnut has joined #openstack-infra | 13:23 | |
| *** eumel8 has quit IRC | 13:23 | |
| *** sanket_ has quit IRC | 13:25 | |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Work around cffi install error https://review.openstack.org/509394 | 13:25 |
| *** yamamoto has quit IRC | 13:27 | |
| *** mat128 has joined #openstack-infra | 13:27 | |
| *** slaweq_ has quit IRC | 13:29 | |
| *** alexchadin has joined #openstack-infra | 13:30 | |
| *** baoli has joined #openstack-infra | 13:30 | |
| *** numans has quit IRC | 13:31 | |
| *** yee37928 has joined #openstack-infra | 13:31 | |
| *** numans has joined #openstack-infra | 13:31 | |
| *** ralonsoh has joined #openstack-infra | 13:32 | |
| *** yee379 has quit IRC | 13:32 | |
| *** martinkopec has quit IRC | 13:35 | |
| *** coolsvap has quit IRC | 13:39 | |
| *** gmann is now known as gmann_sleep | 13:40 | |
| *** slaweq_ has joined #openstack-infra | 13:40 | |
| *** shardy_afk is now known as shardy | 13:42 | |
| *** rodrigods has quit IRC | 13:43 | |
| *** Hal has joined #openstack-infra | 13:43 | |
| *** Hal is now known as Guest14475 | 13:44 | |
| *** armax has joined #openstack-infra | 13:45 | |
| *** sree has joined #openstack-infra | 13:48 | |
| *** numans has quit IRC | 13:50 | |
| dmsimard | pabelanger, mgagne: nodepool graphs are updated with new metric names now fyi, patch merged http://grafana.openstack.org/dashboard/db/nodepool | 13:51 |
| mgagne | thanks! | 13:52 |
| dmsimard | ^ we're actually seeing a fairly even split of nodes between v2 and v3 ? | 13:52 |
| *** numans has joined #openstack-infra | 13:52 | |
| mgagne | stats look to be computed differently? http://grafana.openstack.org/dashboard/db/nodepool-inap?panelId=13&fullscreen | 13:52 |
| mgagne | I mean, Max should be constant no? | 13:53 |
| *** rodrigods has joined #openstack-infra | 13:53 | |
| *** gouthamr has joined #openstack-infra | 13:53 | |
| pabelanger | nodepool.o.o and nl01.o.o/nl02.o.o will be fighting each other | 13:54 |
| mgagne | this explains everything | 13:54 |
| pabelanger | so, stats won't be consistant until we fully migrate | 13:54 |
| mgagne | alright | 13:54 |
| mgagne | pabelanger: I suppose that now (or this week) would be a bad timing for a maintenance in inap-mtl01? =) | 13:55 |
| *** kiennt26 has quit IRC | 13:55 | |
| *** armaan has quit IRC | 13:55 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Use new post pipelines https://review.openstack.org/509221 | 13:55 |
| mgagne | looking to plan an upgrade to mitaka (from kilo), no rush here, just looking for the right timing | 13:55 |
| *** armaan has joined #openstack-infra | 13:55 | |
| mgagne | we can do it now like we can do it in 4 weeks if needed | 13:56 |
| dmsimard | pabelanger: did we bring infracloud back up yet ? ^ | 13:56 |
| pabelanger | mgagne: I'm sure we can work around your schedule | 13:56 |
| pabelanger | dmsimard: should on nodepool.o.o, think we need to do it for nodepool-launcher still | 13:57 |
| mgagne | pabelanger: as mentioned, there is no rush, I don't want to disturb CI more than it is now :P | 13:57 |
| *** jcoufal_ has joined #openstack-infra | 13:57 | |
| pabelanger | mgagne: Ya, your call, we are early into the cycle | 13:58 |
| mgagne | surely not today, tomorrow would be kind of possible or next week(s). | 13:59 |
| *** felipemonteiro_ has joined #openstack-infra | 14:00 | |
| *** jcoufal has quit IRC | 14:00 | |
| *** stephenfin has quit IRC | 14:00 | |
| pabelanger | ++ | 14:01 |
| *** stephenfin has joined #openstack-infra | 14:01 | |
| *** felipemonteiro__ has joined #openstack-infra | 14:01 | |
| AJaeger | wow, finally two changes merged with zuul v3 - that were triggered 8 hours ago | 14:02 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/zuul-jobs master: Add a generic process-test-results role https://review.openstack.org/509459 | 14:03 |
| *** yamamoto has joined #openstack-infra | 14:03 | |
| dmsimard | AJaeger: v3 is not very healthy right now, some pipelines are missing due to unknown reasons | 14:03 |
| *** yamamoto has quit IRC | 14:03 | |
| dmsimard | we're waiting for the ghostbuster in chief to look at things | 14:03 |
| *** hongbin has joined #openstack-infra | 14:03 | |
| AJaeger | dmsimard: they are there (and I was the one noticing it ;), tobiash found them | 14:04 |
| AJaeger | read backscroll - the webui is broken | 14:04 |
| dmsimard | AJaeger: huh | 14:04 |
| dmsimard | mordred: ^ fyi | 14:04 |
| AJaeger | dmsimard: see also https://review.openstack.org/509435 | 14:04 |
| *** felipemonteiro_ has quit IRC | 14:05 | |
| dmsimard | interesting | 14:05 |
| pabelanger | AJaeger: Ya, we have 2 zuul-executors that were stuck (mergers hung on git operations) once we killed that, the executors started processing requests again | 14:05 |
| AJaeger | dmsimard: run | 14:05 |
| jeblair | fungi, (yamamoto): 22348 is zuul's gerrit user id | 14:05 |
| pabelanger | zuulv3 should be happy now | 14:05 |
| AJaeger | dmsimard: run the curl that tobiash showed earlier... | 14:06 |
| AJaeger | pabelanger: ah, interesting - how can we avoid that? I saw that happen several times the last days... | 14:06 |
| AJaeger | morning, jeblair | 14:06 |
| dmsimard | AJaeger: yup, looking now. I can only go through so much backscroll, when I wake up people have typically been chatting for >6hrs already :P | 14:06 |
| frickler | dmsimard: AJaeger: https://review.openstack.org/509435 fixed zuulv3 pipeline display | 14:07 |
| *** edmondsw_ is now known as edmondsw | 14:07 | |
| AJaeger | dmsimard: yeah, know that feeling | 14:07 |
| frickler | s/fixed/will fix/ | 14:07 |
| pabelanger | AJaeger: need to discuss with jeblair, but I think the idea is to wrap the git operations with a timeout. But is a priority to fix IMO before we rollout zuulv3 again | 14:07 |
| AJaeger | pabelanger: yes, please | 14:07 |
| *** ihrachys has joined #openstack-infra | 14:07 | |
| AJaeger | jeblair: could you put 5095435 on your review list? That should fix my earlier problem that the infra pipelines are missing | 14:08 |
| *** rodrigods has quit IRC | 14:08 | |
| *** rodrigods has joined #openstack-infra | 14:09 | |
| *** d0ugal has quit IRC | 14:10 | |
| frickler | jeblair: AJaeger: on my local test I can also spot the offending changes: everything in the periodic pipeline has id "NA" | 14:10 |
| *** bobh has joined #openstack-infra | 14:11 | |
| *** d0ugal has joined #openstack-infra | 14:13 | |
| *** d0ugal has quit IRC | 14:13 | |
| *** d0ugal has joined #openstack-infra | 14:13 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Log exceptions from MODULE FAILURE more consistently https://review.openstack.org/509484 | 14:14 |
| *** slaweq_ has quit IRC | 14:14 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Set log after we have launcher_id https://review.openstack.org/509485 | 14:14 |
| jeblair | AJaeger: 5095435 has too many digits! | 14:14 |
| jeblair | 509435 i think | 14:14 |
| AJaeger | jeblair: try 509435 | 14:14 |
| openstackgerrit | Merged openstack-infra/system-config master: Update StoryBoard Docs https://review.openstack.org/368797 | 14:15 |
| jdandrea | Do I need to do anything additional for the meeting calendar to be updated? https://review.openstack.org/#/c/508933/ merged. It might be waiting for https://review.openstack.org/#/c/508924/ though (?). | 14:17 |
| *** dizquierdo has joined #openstack-infra | 14:18 | |
| *** slaweq_ has joined #openstack-infra | 14:20 | |
| *** srobert has joined #openstack-infra | 14:22 | |
| *** rbrndt has joined #openstack-infra | 14:22 | |
| openstackgerrit | Merged openstack-infra/project-config master: tap-as-a-service: Add neutron requirement for tox jobs https://review.openstack.org/509134 | 14:24 |
| openstackgerrit | Merged openstack-infra/project-config master: neutron-vpnaas: Add neutron requirement for tox jobs https://review.openstack.org/509135 | 14:24 |
| *** kiennt26 has joined #openstack-infra | 14:26 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Avoid JS error when a change has no id https://review.openstack.org/509435 | 14:27 |
| *** eharney has quit IRC | 14:29 | |
| chandankumar | AJaeger: please have a look on this review https://review.openstack.org/#/c/508502/ i am not sure what i am missing for this | 14:30 |
| chandankumar | ? | 14:30 |
| *** armaan has quit IRC | 14:30 | |
| *** armaan has joined #openstack-infra | 14:30 | |
| AJaeger | chandankumar: with the partial rollback to v2, you would need to add both v2 and v3 files ;( Or wait for the next switch to v3 - hopefully soon | 14:31 |
| clarkb | wow so cryptography did the thing we hoped they wouldnt do and pushed a release without a wheel? | 14:31 |
| *** rhallisey has quit IRC | 14:31 | |
| dmsimard | clarkb: really.. | 14:31 |
| *** iyamahat has joined #openstack-infra | 14:32 | |
| clarkb | We identified it as a risk when we removed the paramiko pin :) should just treat those things as eventualities I guess :) | 14:32 |
| *** udesale has quit IRC | 14:32 | |
| *** pcaruana has quit IRC | 14:33 | |
| dmsimard | clarkb: I see wheels here: https://pypi.python.org/pypi/cryptography | 14:33 |
| dmsimard | clarkb: is it a mirror issue ? | 14:33 |
| *** iyamahat_ has joined #openstack-infra | 14:33 | |
| dmsimard | also dates back to august 23rd ? | 14:33 |
| clarkb | dmsimard: see scrollback, something released without a wheel during ansible install requiring libbffi-dev | 14:34 |
| clarkb | pretty aure that something is cryptography? | 14:34 |
| clarkb | but could be something else | 14:34 |
| *** iyamahat__ has joined #openstack-infra | 14:34 | |
| *** rodrigods has quit IRC | 14:34 | |
| frickler | clarkb: dmsimard: that was cffi | 14:34 |
| dmsimard | clarkb: I'm mostly up to date on scrollback, how long ago was that ? | 14:35 |
| clarkb | frickler: yes which was pulled by cryptography? iirc the only thing in the ansible tree that needs it is cryptography | 14:35 |
| frickler | probably just a timing issue on their side. release tar, build wheels, release those a couple of hours later when ready | 14:35 |
| clarkb | dmsimard: 2 hours ago | 14:35 |
| frickler | "Collecting cffi>=1.4.1 (from pynacl>=1.0.1->paramiko->ansible==2.3.2.0)" | 14:36 |
| *** rodrigods has joined #openstack-infra | 14:36 | |
| dmsimard | I don't remember that pip install cffi line being in d-g in the first place | 14:36 |
| clarkb | ah so pynacl (a different paramiko dep) | 14:36 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Move test_model.test_job_inheritance_configloader https://review.openstack.org/509495 | 14:36 |
| *** iyamahat has quit IRC | 14:36 | |
| dmsimard | oh, there you go https://github.com/openstack-infra/devstack-gate/commit/e473fdec13445191d1acf1c77d19c87bbda512c8 | 14:36 |
| dmsimard | also, yeah, wheels seem to be there for cffi | 14:37 |
| clarkb | frickler: we used to pin paramiko to a version that didnt require compiles (was all native python crypto aiui) | 14:38 |
| *** iyamahat_ has quit IRC | 14:38 | |
| frickler | dmsimard: clarkb: yes and based on the comment on the ML I proposed https://review.openstack.org/509429 as more permanent solution. so should we binary-install paramiko, too? | 14:38 |
| clarkb | frickler: but we removed that pin because upstream published wheela for everything and identified this particular case (new release, no wheels) as the risk involved | 14:38 |
| *** kiennt26 has quit IRC | 14:38 | |
| dmsimard | the worst part of it is that we don't even use paramiko | 14:39 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Fix TripleO CI jobs https://review.openstack.org/508660 | 14:39 |
| dmsimard | but it's pulled by ansible anyway | 14:39 |
| openstackgerrit | Merged openstack-infra/system-config master: Switch storyboard-dev to login.ubuntu.com https://review.openstack.org/410327 | 14:39 |
| clarkb | dmsimard: yup | 14:39 |
| *** armaan has quit IRC | 14:39 | |
| clarkb | frickler: paramiko is an ansible dep so I think you change is fine as is | 14:40 |
| *** armaan has joined #openstack-infra | 14:40 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Speed configuration building https://review.openstack.org/509309 | 14:41 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Move test_model.test_job_inheritance_configloader https://review.openstack.org/509495 | 14:41 |
| frickler | clarkb: so we are sure to always have self-built wheels for ansible? they don't seem to exist on pypi | 14:41 |
| *** spectr-RH has quit IRC | 14:41 | |
| clarkb | frickler: ansible itself may not require a wheel since it doesnt link to C stuff | 14:42 |
| *** spectr has joined #openstack-infra | 14:42 | |
| frickler | clarkb: but if I want to run "pip install --only-binary :all: ansible" I need wheels for everything. otherwise I need to s/:all:/cffi/ and possible other critical libraries | 14:43 |
| *** jcoufal has joined #openstack-infra | 14:43 | |
| *** eharney has joined #openstack-infra | 14:45 | |
| clarkb | yes | 14:45 |
| *** jcoufal_ has quit IRC | 14:45 | |
| *** rodrigods has quit IRC | 14:45 | |
| *** rodrigods has joined #openstack-infra | 14:45 | |
| *** pcaruana has joined #openstack-infra | 14:45 | |
| clarkb | it is curious that they dont publish wheels | 14:46 |
| clarkb | rbergeron: ^ something you might be able to address? | 14:46 |
| dmsimard | clarkb: well there's nothing to really compile in Ansible /me shrug | 14:47 |
| *** sree has quit IRC | 14:47 | |
| clarkb | doesnt matter, we publish wheels for all our things and none of them link to C | 14:47 |
| clarkb | and the reason for it is as frickler points out | 14:48 |
| clarkb | its simplicity and conaistency | 14:48 |
| *** sree has joined #openstack-infra | 14:48 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/devstack-gate master: Install cffi only from wheels for ansible https://review.openstack.org/509429 | 14:49 |
| clarkb | frickler: can you add cryptography too? | 14:50 |
| fungi | dmsimard: clarkb: we _could_ ask ansible to move paramiko to a setup extras so you do pip install ansible[pssh] or something along those lines when you actually want ansible to provide an ssh client for you | 14:50 |
| *** alexchadin has quit IRC | 14:50 | |
| *** sdague has quit IRC | 14:50 | |
| *** dhajare has quit IRC | 14:51 | |
| dmsimard | fungi: that makes sense | 14:51 |
| clarkb | I think they have been wary of that because not linux requires it | 14:52 |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/devstack-gate master: Install critical libraries only from wheels for ansible https://review.openstack.org/509429 | 14:52 |
| clarkb | so its an extra for some subset of the install base but hard dep for the rest | 14:52 |
| fungi | with the justification being "you can use ansible perfectly well without paramiko as long as you provide a supported ssh client separately, so hard-depending on a completely optional non-pure python library is somewhat user hostile" | 14:52 |
| *** sree has quit IRC | 14:52 | |
| mriedem | if i want to move a job to the experimental queue, that happens in openstack-zuul-jobs now? | 14:53 |
| mriedem | and in project-config? | 14:53 |
| fungi | the alternative is to make the "ansible" sdist/wheel a dummy package which depends on paramiko and ansible-nossh or some such | 14:53 |
| *** slaweq_ has quit IRC | 14:53 | |
| fungi | so that people on unfortunate platforms with no supported ssh client can also ignorantly just pip install ansible and get a working system, but people who know they don't need ansible to provide an ssh client for them can install the ansible-nossh package | 14:54 |
| pabelanger | mriedem: project-config, but we are under a freeze currently. | 14:55 |
| fungi | heck, ansible-base could have a pssh extra, so you could have the ansible dummy package declare ansible-base[pssh] as an install_requires | 14:55 |
| *** bobh has quit IRC | 14:56 | |
| pabelanger | mriedem: if we need to do it, we'll need to make the change to both zuulv2.5 and zuulv3 | 14:56 |
| frickler | jeblair: could you also apply https://review.openstack.org/509435 manually on zuulv3.o.o or will that happen automatically at some stage? | 14:56 |
| *** lnxnut_ has joined #openstack-infra | 14:56 | |
| *** iyamahat_ has joined #openstack-infra | 14:56 | |
| pabelanger | frickler: puppet should be applying it shortly | 14:56 |
| *** xarses has joined #openstack-infra | 14:56 | |
| *** slaweq_ has joined #openstack-infra | 14:56 | |
| *** eumel8 has joined #openstack-infra | 14:57 | |
| fungi | and i guess that won't even require restarting anything | 14:57 |
| pabelanger | might need ctrl-F5 on your browser | 14:57 |
| fungi | right | 14:58 |
| pabelanger | which I just did and it works | 14:58 |
| pabelanger | so, looks like applied now | 14:58 |
| *** lnxnut has quit IRC | 14:58 | |
| frickler | pabelanger: fungi: thx, that worked indeed, closing and reopening the window wasn't enough it seems | 14:59 |
| *** iyamahat__ has quit IRC | 14:59 | |
| fungi | confirmed, my browser had cached the js outside the tab itself, but opening a tab to it and force-refreshing the page worked | 14:59 |
| *** numans has quit IRC | 15:00 | |
| *** iyamahat has joined #openstack-infra | 15:01 | |
| *** leifmadsen has quit IRC | 15:01 | |
| *** matbu_ is now known as matbu | 15:01 | |
| *** numans has joined #openstack-infra | 15:02 | |
| *** iyamahat__ has joined #openstack-infra | 15:02 | |
| *** tikitavi has quit IRC | 15:03 | |
| *** yamamoto has joined #openstack-infra | 15:04 | |
| *** iyamahat_ has quit IRC | 15:04 | |
| *** ramishra has quit IRC | 15:06 | |
| *** iyamahat_ has joined #openstack-infra | 15:06 | |
| *** leifmadsen has joined #openstack-infra | 15:06 | |
| *** leifmadsen has quit IRC | 15:06 | |
| *** leifmadsen has joined #openstack-infra | 15:06 | |
| *** iyamahat has quit IRC | 15:06 | |
| *** iyamahat__ has quit IRC | 15:07 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config master: Move tempest-dsvm-ceph-nv jobs to experimental queue https://review.openstack.org/509500 | 15:09 |
| mriedem | pabelanger: ok, posting anyway so i don't forget, | 15:09 |
| mriedem | but can WPI it | 15:09 |
| mriedem | *WIP | 15:09 |
| *** armaan_ has joined #openstack-infra | 15:11 | |
| *** armaan has quit IRC | 15:11 | |
| *** yamamoto has quit IRC | 15:13 | |
| *** rcernin has quit IRC | 15:13 | |
| *** sdague has joined #openstack-infra | 15:14 | |
| *** andreas_s_ has quit IRC | 15:16 | |
| persia | pleia2: Looking at https://review.openstack.org/#/c/410328/1 , I wondered if you were still following closely enough to want to be a superuser on openstack's storyboard. No worries either way: just different data than I expected. | 15:18 |
| rbergeron | clarkb, fungi, dmsimard: aiui -- the main reason behind not publishing wheels is that we have dependencies that don't have wheels. | 15:19 |
| pleia2 | persia: I was in 2016, I am not anymore | 15:19 |
| rbergeron | https://github.com/ansible/ansible/issues/18714 -- as bcoca ever so sweetly, ahem, stated, So we have looked into this before and since there are no wheels for our dependencies and we are not going to start doing packaging for other projects this is a non starter. | 15:19 |
| persia | pleia2: That was my thought :) Thanks for the confirmation. | 15:20 |
| clarkb | rbergeron: that seems like a good way to never have wheels :) | 15:20 |
| clarkb | rbergeron: also you can totally publish as a wheel even when your deps don't have wheels | 15:20 |
| clarkb | it still cuts down on installation time... | 15:20 |
| fungi | persia: yeah, i didn't get around to cleaning up the sb superusers list when i swept up the infra-root cobwebs a few months back | 15:21 |
| fungi | rbergeron: also, that issue is from december, and was closed due to circumstances which are likely to change as time marches on... i guess the expectation is to just reopen that issue (or a new one) once those deps have wheels? | 15:22 |
| rbergeron | there have been some movements around pycrypto stuff where wheel-y things have come up and people have come up with interesting ways around things -- https://github.com/ansible/ansible/pull/20566 ... https://github.com/ansible/ansible/issues/13075 | 15:22 |
| rbergeron | fungi: possibly -- my recommendation would be... | 15:22 |
| rbergeron | (hold while i get links) | 15:22 |
| clarkb | rbergeron: fwiw cryptography does publish wheels now, as does cffi (though more slowly than sdsists apparnetly) | 15:22 |
| fungi | rbergeron: it would have been nice too if issue 18714 mentioned which of "our dependencies" are still lacking wheels | 15:23 |
| clarkb | and that is a relatively recent change so ya may be worht looking at again | 15:23 |
| fungi | (or at least linked to where that research was published) | 15:23 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Set log after we have launcher_id https://review.openstack.org/509485 | 15:23 |
| fungi | i'll follow up to the issue | 15:24 |
| *** erlon has joined #openstack-infra | 15:25 | |
| *** psachin has quit IRC | 15:25 | |
| rbergeron | so: #1 -- I'd just open an issue in ansible/ansible (and maybe link the "yes we saw previous "nope, non-starter"" issues) -- | 15:25 |
| rbergeron | and then ... The core team has an irc meeting on tuesdays and thursdays - | 15:26 |
| rbergeron | and you can add that issue to the agenda -- https://github.com/ansible/community/issues/263 | 15:26 |
| fungi | well, we'll see if anyone follows up to my comment in issue 18714, but i can always generate a transitive deps list for ansible and update with the relevant status i suppose | 15:27 |
| jeblair | in #ansible-meeting of course | 15:27 |
| rbergeron | and then, voila, maybe just chat through it. i suspect abadger1999 will be of particular help / kindness, esp. since we've all now bonded over zuul things and python stuff at ansiblefest -- | 15:27 |
| rbergeron | multiple times | 15:27 |
| rbergeron | ;) | 15:27 |
| rbergeron | (/me goes to take a kid to school, brb) | 15:27 |
| clarkb | fungi: I'm doing that listing now | 15:27 |
| *** slaweq_ has quit IRC | 15:27 | |
| *** slaweq_ has joined #openstack-infra | 15:28 | |
| fungi | rbergeron: though separate from that, it would be nice to come up with a solution for pip installing ansible which doesn't unnecessarily drag in paramiko (and perhaps other optional deps) | 15:28 |
| *** pcaruana has quit IRC | 15:28 | |
| smcginnis | What's up with the zuul2 post queue? Jobs sitting queued for over 9 hours now? | 15:28 |
| *** tesseract has quit IRC | 15:29 | |
| fungi | there are definitely ways for that to still play nicely with people on ssh-less platforms to not have to know more than `pip install ansible` and put the onus on people who actually know they don't need ansible to bring its own ssh client to use more complex install commands | 15:29 |
| *** vsaienk0 has quit IRC | 15:30 | |
| eumel8 | pabelanger: pls WF for another missing repo in zuul check: https://review.openstack.org/#/c/509358/ | 15:30 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove test_job_inheritance https://review.openstack.org/509509 | 15:30 |
| fungi | smcginnis: we failed to bring up the static nodes job launcher during rollback yesterday, so jobs depending on long-lived privileged nodes were queuing up until a couple hours ago | 15:30 |
| fungi | it should me moving now | 15:31 |
| smcginnis | fungi: So that should start moving soon? First in the queue is still sitting there. | 15:31 |
| *** igordc has quit IRC | 15:32 | |
| clarkb | http://paste.openstack.org/show/622677/ pycparser and markupsafe lack wheels. Everything else has them | 15:32 |
| *** dizquierdo has quit IRC | 15:33 | |
| *** igordc has joined #openstack-infra | 15:33 | |
| clarkb | fungi: ^ | 15:33 |
| fungi | smcginnis: looks like maybe the {name}-upstream-translation-update jobs still aren't getting a node assignment. i'll take a look at those. everything else appears to be running, but i'll see why those are still pending | 15:33 |
| *** yamahata has joined #openstack-infra | 15:34 | |
| smcginnis | fungi: Thanks | 15:34 |
| fungi | also some hook-{name}-rtfd jobs, but those should just use normal nodes i think so they may simply be waiting on available resources | 15:35 |
| AJaeger | pabelanger, mordred, do you know why the new project-config-frozen-files job gives " project-config-frozen-files : ERROR No valid playbook found (non-voting)" - see https://review.openstack.org/504870 ? | 15:37 |
| AJaeger | what did I do wrong? | 15:37 |
| openstackgerrit | Corey Bryant proposed openstack-infra/project-config master: snaps: add snap-gnocchi definitions https://review.openstack.org/500964 | 15:38 |
| AJaeger | mordred, pabelanger change introducing this was https://review.openstack.org/#/c/509158/ | 15:38 |
| fungi | smcginnis: move along, nothing to see here ;) those are the proposal node jobs which got scheduled after the periodic pipeline jobs, so they're waiting for periodic to burn down | 15:39 |
| smcginnis | fungi: OK, good. Thanks for checking. :) | 15:40 |
| AJaeger | frickler: thanks for fixing the zuulv3 page! | 15:40 |
| fungi | smcginnis: the proposal node is definitely running jobs, but it's focused on the periodic ones at the moment since periodic and post have the same priority therefore get lumped together for first-come-first-served | 15:40 |
| clarkb | AJaeger: drop the .yaml in the run: line | 15:40 |
| clarkb | AJaeger: I imagine that is going to get a lot of us :/ | 15:41 |
| clarkb | AJaeger: so its run and not run.yaml | 15:41 |
| AJaeger | smcginnis, fungi : Those translation jobs are currently run in both post and experimental. I saw them moving forward... | 15:41 |
| AJaeger | clarkb: Ah! Thanks, will do. | 15:42 |
| smcginnis | AJaeger, fungi: OK, I'll just be patient. | 15:42 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix project-config-frozen-files https://review.openstack.org/509513 | 15:43 |
| AJaeger | clarkb, mordred, pabelanger : Here's the fix ^ | 15:43 |
| AJaeger | smcginnis: periodic now has 79 projects - it had 89 when I checked last. those projects all have translation jobs waiting... | 15:44 |
| AJaeger | infra-root, we do have periodic jobs in zuul v3. Shouldn't we disable all of them? | 15:45 |
| * AJaeger prepares a change | 15:45 | |
| fungi | clarkb: so looks like ansible is (perhaps unnecessarily) blocking its decision on whether or not to publish wheels on a resolution to https://github.com/pallets/markupsafe/issues/45 and https://github.com/eliben/pycparser/issues/147 | 15:46 |
| * fungi "fondly" remembers when pycparser tried to start publishing wheels | 15:48 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add git timeout https://review.openstack.org/509517 | 15:48 |
| fungi | clarkb: looks like yesterday was the one-year anniversary! ;) | 15:48 |
| clarkb | one of the comments on markupsafe sort of hits at why ansible should just do it, it is significantly faster to install even for pure python packages | 15:48 |
| clarkb | and that isn't dependent on your deps having wheels too | 15:48 |
| fungi | er, i guess it was monday actually | 15:49 |
| clarkb | it obviously helps if everything has a wheel but you realize the speedups independently | 15:49 |
| *** jcoufal_ has joined #openstack-infra | 15:49 | |
| sdague | it's always seemed a little nutty to me that wheels need to be built by the uploader | 15:49 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Disable Zuul v3 Periodic jobs https://review.openstack.org/509518 | 15:49 |
| *** slaweq_ has quit IRC | 15:49 | |
| *** rhallisey has joined #openstack-infra | 15:49 | |
| fungi | sdague: they don't need to be. pip will build wheels automatically if it ends up having to download an sdist | 15:49 |
| AJaeger | team, change to disable periodic jobs ^ | 15:49 |
| *** slaweq_ has joined #openstack-infra | 15:50 | |
| rbergeron | fungi, clarkb: you can always also pop into #ansible-devel and chat, but on something like this where there is history and the historical causes keep changing about -- in a meeting at least there will be enough folks around to have some broad consensus or "next steps to think on for a path forward", etc. | 15:50 |
| *** slaweq_ has quit IRC | 15:50 | |
| sdague | fungi: no, I mean, in pypi | 15:50 |
| fungi | rbergeron: yep, though i expect async discussion will garner input from more people | 15:50 |
| rajinir | I have an error building the dib image, fatal: unable to connect to git.openstack.org:, setting GIT_BASE didn't solve it. I only restarted the nodepool-builder, can someone help ? | 15:51 |
| clarkb | I'll be traveling during the next meeting I think | 15:51 |
| *** slaweq_ has joined #openstack-infra | 15:51 | |
| fungi | sdague: as in like have the psf fund a build farm of various platforms to autogenerate wheels for each sdist? | 15:51 |
| sdague | fungi: yes | 15:51 |
| clarkb | rajinir: you should independently verify connectivity to git.openstack.org | 15:51 |
| clarkb | rajinir: are you behind a firewall or proxy? basically make sure it is working outside of dib firstthen look at making it work wiht dib | 15:51 |
| rajinir | https://www.irccloud.com/pastebin/NElFtXm3/Ping%20works | 15:51 |
| rajinir | clarkb: I can ping to it | 15:52 |
| fungi | sdague: it's come up many times before on the distutils-sig ml... main blocker is funding a team of sysadmins to take responsibility for it, or finding sufficient volunteer labor to do the same | 15:52 |
| clarkb | rajinir: ping isn't what git speaks though. You want to check http:// https:// and/or git:// | 15:52 |
| *** jcoufal has quit IRC | 15:52 | |
| clarkb | rajinir: also ping is an ipv4 utility and won't check ipv6 (which the other protocols may run over) | 15:52 |
| jeblair | AJaeger: why disable periodic jobs? | 15:52 |
| rbergeron | it may also be a "maintain a unified / consistent front" type of thing for a bit as we get absorbed into certain distros that may be part or may not be part of our corporate overlord's offerings and sort out that drama (or avoid drama) | 15:53 |
| rbergeron | but that is just my cynical speculation | 15:53 |
| AJaeger | jeblair: have a look at the current status page. They still run - due to our small quota - and run the translations jobs that are broken. Is that really useful? | 15:54 |
| sdague | fungi: I wonder what their build load is relative to openstack's normal load. Like how much of openstack infra compute capacity would be required to fullfill it | 15:54 |
| sdague | noting there is a platform bit as well | 15:54 |
| sdague | but just a thought | 15:54 |
| jeblair | AJaeger: of course -- someone should be fixing them. :) | 15:54 |
| fungi | rbergeron: yes, i can imagine a general distro-centric mindset asking why you should bother making it possible to install ansible through means other than your distro's package manager | 15:54 |
| jeblair | AJaeger: we're running lots of broken stuff. that's the point, to see that it's broken. | 15:54 |
| AJaeger | jeblair: ;) | 15:54 |
| clarkb | sdague: ya the biggest problem is likely having windows and os x to build on | 15:54 |
| rbergeron | but i'm sure folks will be willing to discuss it, particularly with other folks who have to deal in the same types of dependency hells that exist in community projects and also know and love python | 15:54 |
| sdague | also as it might be an interesting olive branch back from openstack to the rest of python | 15:54 |
| rbergeron | ;) | 15:54 |
| clarkb | sdague: linux is easy as the markupsafe thread points out | 15:55 |
| AJaeger | jeblair: does it make sense with current quota? if you think it does, I'll abandon... | 15:55 |
| *** slaweq_ has quit IRC | 15:55 | |
| *** eranrom_ has joined #openstack-infra | 15:55 | |
| rbergeron | fungi: on the bright side we do cater to and support officially with the monies in product land other distros, which helps us to push back on the non-switzerland-like things. | 15:56 |
| *** ldnunes has quit IRC | 15:56 | |
| *** armaan_ has quit IRC | 15:56 | |
| *** bobh has joined #openstack-infra | 15:56 | |
| pabelanger | AJaeger: looking at No valid playbook issue | 15:56 |
| rbergeron | i am not trying to make humans look terrible, but i also got to be in fedora land for 5 years watching the packaging standards thing be drama, drama, drama for a long time | 15:56 |
| AJaeger | pabelanger: bug found already | 15:56 |
| *** ldnunes has joined #openstack-infra | 15:56 | |
| pabelanger | Ya, see that now | 15:56 |
| fungi | sdague: yeah, i think for universal wheels it's just a matter configuring your setup.py accordingly and knowing to run an extra (or slightly longer) command to build and upload wheels in addition to sdists. for "manylinux1" non-universal binary wheels, there are docker images with the required build environment available, and travis build services already doing that they can reuse but we could | 15:57 |
| *** eroux has joined #openstack-infra | 15:57 | |
| fungi | certainly look into similar functionality on our end | 15:57 |
| AJaeger | pabelanger: https://review.openstack.org/509513 - clarkb pointed out the problem | 15:57 |
| *** armaan has joined #openstack-infra | 15:57 | |
| AJaeger | thanks, pabelanger | 15:57 |
| *** eranrom__ has joined #openstack-infra | 15:57 | |
| rbergeron | and know that it's a constant back and forth historically with other products -- but at least there are plenty of folks who get upstream and know that's where magic happens, and y'all are good with that | 15:57 |
| rbergeron | anyway | 15:57 |
| * rbergeron waxes community poetic | 15:57 | |
| pabelanger | FWIW: I think we should support .yaml extensions in run :) I actually like how it looks | 15:57 |
| AJaeger | ;) | 15:57 |
| *** lucasagomes is now known as lucas-afk | 15:57 | |
| fungi | rbergeron: completely agree! | 15:58 |
| rajinir | clarkb: http://git.openstack.org/ and https://git.openstack.org/ works. git://git.openstack.org doesn't , is there a way to force the image builder to use http or https? | 15:58 |
| melwitt | has anyone seen a problem before in a job where keystone can't start up because "ImportError: cannot import name deploy"? http://logs.openstack.org/87/508187/3/check/legacy-tempest-dsvm-full-devstack-plugin-ceph/761fadb/logs/screen-keystone.txt.gz | 15:58 |
| fungi | rbergeron: basically, i expect what's missing is someone embedded in the ansible dev community with an interest in improving its packaging situation on pypi. that's probably all it takes really | 15:58 |
| *** eranrom has quit IRC | 15:58 | |
| pabelanger | rajinir: git://git.openstack.org/openstack-infra/project-config should be a valid project to clone via git protocol | 15:59 |
| clarkb | rajinir: that is what GIT_BASE is for did you try setting that to https://git.openstack.org? | 15:59 |
| clarkb | oh ya also we don't serve at the root of git.o.o via git:// | 16:00 |
| *** eranrom_ has quit IRC | 16:00 | |
| fungi | melwitt: that's a new one on me, but maybe someone in #openstack-qa is familiar with that error condition | 16:00 |
| *** hashar has joined #openstack-infra | 16:00 | |
| clarkb | rajinir: so oyu'll need to test a specific repo like pabelanger points out | 16:00 |
| melwitt | thanks fungi | 16:00 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Collect /etc/snmp in quickstart jobs https://review.openstack.org/509525 | 16:01 |
| rajinir | <pabelanger: clarkb yep, git clone git://git.openstack.org/openstack-infra/project-config with git protocol failed | 16:01 |
| *** egonzalez has quit IRC | 16:01 | |
| fungi | melwitt: also i find this: https://ask.openstack.org/question/11458 | 16:01 |
| fungi | havana. wow | 16:01 |
| *** bobh has quit IRC | 16:01 | |
| fungi | sugegsts missing routes and/or paste-deploy deps | 16:01 |
| clarkb | melwitt: fungi possibly fallout from deleting v2 api things? | 16:02 |
| rajinir | pabelanger: clarkb , GIT_BASE is set in the env, but image builder is not picking it up | 16:02 |
| pabelanger | rajinir: which image builder? is this for DIBs? | 16:02 |
| clarkb | rajinir: what is it set to? and you set it in the nodepool.yaml file under the image env? | 16:02 |
| rbergeron | fungi: agree -- I think right now the focus has largely been on "keep the user experience largely as simple / painless as possible" -- don't raise the bar too high for people to actually get to experience the awesome. combined with: maintaining anything beyond what we have now is potentially a pit of never-ending hell. esp. since lots of use of virtual environments and whatnot. | 16:02 |
| melwitt | fungi: I found that too but was thinking how could something be missing from the gate env? sorry I'm ignorant about a lot of these things | 16:02 |
| clarkb | rajinir: oh I bet I know why it maybe isn't working, because the repos already are cached? | 16:02 |
| rajinir | pabelanger: clarkb ; It is set in the env, but will try to set in the nodepool.yaml and try. Thanks | 16:03 |
| fungi | melwitt: if something dropped paste-deploy as a requirement or test-requirement, that could explain it i guess | 16:03 |
| rbergeron | but i have to think a solid "give me the advanced install or options or etc." method is possible, esp. if geared towards the people who are the most likely to actually be invested in figuring out how to make it work well and reliably | 16:03 |
| clarkb | rajinir: since nodepool is calling out to dib it curates its environment iirc and you set that env in the nodepool.yaml under the image config | 16:03 |
| rajinir | pabelanger: clarkb: yes, possibly already cached, because this is old system. How do I clear the cache? | 16:03 |
| rajinir | pabelanger: clarkb or should I even clear the cache? | 16:04 |
| clarkb | rajinir: start by fixing the env | 16:04 |
| clarkb | I think dib may be smart enough to handle the different remotes | 16:04 |
| rajinir | clarkb: ok, will try. Thanks | 16:04 |
| rbergeron | my python-fu is ... non-existent (consider me in the "just make it work and i hope someday to care about other things like even faster installs" camp) so i can't really speculate beyond that, but i think approaching it from a "experimental, could we try and just make it very clear that this way is not for many people and not officially supported... yet" ;) | 16:04 |
| pabelanger | /opt/dib_cache i think, at least on our builders | 16:04 |
| rbergeron | aspect might go over fine. since... yay community and all | 16:05 |
| *** armaan has quit IRC | 16:14 | |
| *** armaan has joined #openstack-infra | 16:14 | |
| *** armaan has quit IRC | 16:15 | |
| *** armaan has joined #openstack-infra | 16:15 | |
| *** hemna_ has quit IRC | 16:18 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool feature/zuulv3: Provide resource_name in logging as structured data https://review.openstack.org/509531 | 16:20 |
| *** Rockyg has joined #openstack-infra | 16:22 | |
| *** ldnunes has quit IRC | 16:30 | |
| *** vsaienk0 has joined #openstack-infra | 16:30 | |
| *** ldnunes has joined #openstack-infra | 16:30 | |
| *** dizquierdo has joined #openstack-infra | 16:31 | |
| *** jpich has quit IRC | 16:32 | |
| *** larainema has quit IRC | 16:32 | |
| *** gridinv1 has joined #openstack-infra | 16:33 | |
| *** jgriffith is now known as Guest73111 | 16:39 | |
| *** jgriffit1 is now known as jgriffith | 16:39 | |
| *** jgriffith_ has joined #openstack-infra | 16:39 | |
| *** vsaienk0 has quit IRC | 16:40 | |
| *** e0ne has quit IRC | 16:40 | |
| AJaeger | pabelanger: regarding https://review.openstack.org/509182 - zuul v3 is very slow, no need to recheck, let's figure out why it's not showing up | 16:44 |
| *** felipemonteiro__ has quit IRC | 16:44 | |
| *** jascott1 has quit IRC | 16:45 | |
| *** jascott1 has joined #openstack-infra | 16:45 | |
| *** hashar is now known as hasharAway | 16:46 | |
| pabelanger | AJaeger: it needs a recheck for new pipelines | 16:47 |
| *** shardy has quit IRC | 16:47 | |
| *** dtantsur is now known as dtantsur|afk | 16:47 | |
| pabelanger | AJaeger: openstack-zuul-jobs should be using infra-check / infra-gate now | 16:47 |
| pabelanger | at least i think it merged | 16:48 |
| pabelanger | Ya, it did | 16:48 |
| AJaeger | pabelanger: yes, that merged - and the previous recheck run already | 16:49 |
| pabelanger | AJaeger: however, zuulv3 is slow right now because of memory debugging jeblair is doing | 16:49 |
| pabelanger | AJaeger: last recheck was just before 509182 merged, so we only had check pipelines | 16:50 |
| AJaeger | pabelanger: ah... | 16:50 |
| *** jamesmcarthur has joined #openstack-infra | 16:51 | |
| *** sambetts is now known as sambetts|afk | 16:55 | |
| Shrews | pabelanger: do you still need the v3 held node? | 16:56 |
| pabelanger | Shrews: let me chcek | 16:56 |
| pabelanger | Shrews: no, I can delete it now | 16:57 |
| Shrews | k | 16:57 |
| pabelanger | done, thanks | 16:57 |
| *** Swami has joined #openstack-infra | 16:58 | |
| *** jpena is now known as jpena|off | 16:58 | |
| *** numans has quit IRC | 16:59 | |
| *** SumitNaiksatam has quit IRC | 17:01 | |
| *** numans has joined #openstack-infra | 17:02 | |
| *** rwsu has quit IRC | 17:03 | |
| mordred | AJaeger, pabelanger, clarkb: if you have a sec: https://review.openstack.org/#/c/509212 | 17:03 |
| pabelanger | +2 | 17:05 |
| AJaeger | +2a | 17:06 |
| *** yamahata has quit IRC | 17:08 | |
| *** iyamahat_ has quit IRC | 17:08 | |
| mnaser | mordred: the release note jobs are currently broken even with the small bit of work we did (code paths are weird because exit 0, old code tries to do sudo stuff but revoke-sudo already ran) | 17:12 |
| mnaser | i am working on refactoring them (assuning no one else is) | 17:12 |
| *** hemna_ has joined #openstack-infra | 17:13 | |
| *** trown is now known as trown|lunch | 17:13 | |
| AJaeger | mnaser: did we merge all bits and pieces or are you aware of open bits to approve? | 17:15 |
| mnaser | AJaeger i believe we merged most of the stuff and there was still issues that came out of it | 17:16 |
| mnaser | i dont think there is anything open in the work that mordred did at least | 17:16 |
| AJaeger | mnaser: some cleanups are still not merged - but functionality is indeed in. So, ask mordred whether he's on it or appreciates your help ;) | 17:18 |
| mnaser | AJaeger cool ill wait to hear :) | 17:19 |
| *** ccamacho has quit IRC | 17:19 | |
| mordred | mnaser: don't refactor releasenotes jobs | 17:20 |
| AJaeger | pabelanger, clarkb, could one of you +2A https://review.openstack.org/#/c/508766/ - cleanup of releasenotes | 17:20 |
| mordred | mnaser: (sorry - that's a 'hang on, I want to make sure we're not duplicating effort) | 17:21 |
| mnaser | mordred absolutely, no problem, please let me know, i dont mind picking it up (i was thinking of doing a stage by stage small refactor + fix things on the way) .. to get it to pass at least | 17:21 |
| mordred | mnaser: ok - nevermind - go for it (assuming you're refactoring the new one) | 17:21 |
| AJaeger | pabelanger, clarkb : I just ask for recheck - it should be good ;) | 17:22 |
| mnaser | mordred yes, the 'releasenotes' role, i'll try to keep smaller/simpler patches to make review easier :> | 17:22 |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Add python-tempestconf project https://review.openstack.org/508502 | 17:22 |
| chandankumar | AJaeger: ^^ | 17:23 |
| *** jascott1 has quit IRC | 17:24 | |
| *** jascott1 has joined #openstack-infra | 17:24 | |
| mordred | mnaser: woot - thanks! | 17:25 |
| *** vsaienk0 has joined #openstack-infra | 17:26 | |
| dims | frickler : fungi : sdague : is this what we are going with for the cffi? https://review.openstack.org/#/c/509429/ | 17:27 |
| *** iyamahat has joined #openstack-infra | 17:27 | |
| dims | hmm looks like https://review.openstack.org/#/c/509394/ was merged... | 17:27 |
| *** SumitNaiksatam has joined #openstack-infra | 17:28 | |
| sdague | dims: I merged the thing that was guarunteed to get us working again, and letting other folks figure out what they want long term | 17:28 |
| dims | sdague : cool, was going to do some rechecks, hence asking. thanks! | 17:29 |
| *** jascott1 has quit IRC | 17:29 | |
| *** ralonsoh has quit IRC | 17:30 | |
| *** markvoelker has quit IRC | 17:30 | |
| *** markvoelker has joined #openstack-infra | 17:30 | |
| *** dprince has quit IRC | 17:31 | |
| openstackgerrit | Mohammed Naser proposed openstack-infra/openstack-zuul-jobs master: Refactor releasenote job to have one path for tox building https://review.openstack.org/509558 | 17:33 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/openstack-zuul-jobs master: Refactor tox to use the role https://review.openstack.org/509559 | 17:33 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/openstack-zuul-jobs master: Switch translated release notes conditionals https://review.openstack.org/509560 | 17:33 |
| *** lbragstad has quit IRC | 17:34 | |
| *** felipemonteiro_ has joined #openstack-infra | 17:35 | |
| *** vsaienk0 has quit IRC | 17:35 | |
| *** felipemonteiro__ has joined #openstack-infra | 17:36 | |
| *** rbrndt has quit IRC | 17:36 | |
| *** anticw has quit IRC | 17:38 | |
| *** tosky has quit IRC | 17:38 | |
| *** felipemonteiro_ has quit IRC | 17:40 | |
| *** zoli has quit IRC | 17:41 | |
| *** esberglu has quit IRC | 17:41 | |
| *** jamesmcarthur has quit IRC | 17:42 | |
| *** esberglu has joined #openstack-infra | 17:42 | |
| *** gridinv1 has quit IRC | 17:42 | |
| *** esberglu has quit IRC | 17:42 | |
| *** jamesmcarthur has joined #openstack-infra | 17:42 | |
| *** esberglu has joined #openstack-infra | 17:42 | |
| *** yamahata has joined #openstack-infra | 17:45 | |
| *** jgriffith has quit IRC | 17:45 | |
| *** jgriffith_ is now known as jgriffith | 17:45 | |
| *** dizquierdo has quit IRC | 17:46 | |
| *** zoli has joined #openstack-infra | 17:46 | |
| *** jamesmcarthur has quit IRC | 17:47 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Update node requests after nodes https://review.openstack.org/509571 | 17:47 |
| mnaser | ^ that's an attempt im testing in https://review.openstack.org/#/c/509563/1 ... i'll give it a bit of time before getting back to it because of current zuulv3 memory debugging slowdowns | 17:48 |
| pabelanger | I'm stepping away to run an errand, get some lunch. Back in 90mins or so | 17:49 |
| *** jamesmcarthur has joined #openstack-infra | 17:53 | |
| *** lbragstad has joined #openstack-infra | 17:57 | |
| *** tosky has joined #openstack-infra | 17:58 | |
| *** SumitNaiksatam has quit IRC | 17:58 | |
| *** MasterOfBugs has joined #openstack-infra | 18:00 | |
| *** SumitNaiksatam has joined #openstack-infra | 18:01 | |
| *** numans has quit IRC | 18:04 | |
| jdandrea | *peeks in* ... are there known delays with Zuul? I have six checks stuck in queue for over two hours. https://review.openstack.org/#/c/508924/ | 18:04 |
| *** ijw has joined #openstack-infra | 18:04 | |
| jdandrea | If so, np - want to be sure (or if perhaps I need to do a recheck, f'rinstance) | 18:05 |
| clarkb | jdandrea: zuulv2 just has a lot of changes to work with, 217 in check, 38 in gate. zuulv3 is being actively debugged for memory issues which slows it down | 18:05 |
| clarkb | now up to 219 :) | 18:05 |
| jdandrea | clarkb Whee... | 18:05 |
| *** numans has joined #openstack-infra | 18:06 | |
| *** trown|lunch is now known as trown | 18:09 | |
| AJaeger | cleanup review request for https://review.openstack.org/#/c/509091/ , please | 18:10 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: v3: Use openstack-tox-validate https://review.openstack.org/509578 | 18:13 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove obsolete validate jobs https://review.openstack.org/509579 | 18:13 |
| AJaeger | mordred: used the new job you defined ^ | 18:14 |
| AJaeger | mordred: will rebase to avoid merge conflict... | 18:15 |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/openstack-zuul-jobs master: Really only copy logs dir for chef-rake-integration https://review.openstack.org/508841 | 18:15 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: v3: Use openstack-tox-validate https://review.openstack.org/509578 | 18:19 |
| jdandrea | Yaay, this passed. Can someone please review when able? https://review.openstack.org/#/c/508924/ | 18:20 |
| *** edmondsw has quit IRC | 18:20 | |
| *** ociuhandu has quit IRC | 18:21 | |
| jdandrea | Thank you AJaeger! | 18:21 |
| AJaeger | jdandrea: we can only merge this when no meetings occur in any of the channels. so, might take some time. You might want to ask Friday for review and point out that you have a irc channel change for system-config so that the right people can help... | 18:23 |
| *** edmondsw has joined #openstack-infra | 18:23 | |
| jdandrea | AJaeger That sounds good. I will look for the agenda page (for infra, I take it?). | 18:24 |
| AJaeger | jdandrea: any scheduled meeting... | 18:25 |
| *** e0ne has joined #openstack-infra | 18:25 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove obsolete validate jobs https://review.openstack.org/509579 | 18:25 |
| jdandrea | AJaeger Oh, I see what you mean. I will pop in here on Friday and ask for that. | 18:25 |
| *** edmondsw has quit IRC | 18:28 | |
| *** rbrndt has joined #openstack-infra | 18:29 | |
| *** eharney has quit IRC | 18:29 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/bindep master: Add new syntax to allow matching multiple profile https://review.openstack.org/506502 | 18:30 |
| *** eharney has joined #openstack-infra | 18:30 | |
| SamYaple | seems to be a bit laggy today | 18:30 |
| *** vhosakot has joined #openstack-infra | 18:31 | |
| * AJaeger signs off for the day | 18:31 | |
| openstackgerrit | Brian Haley proposed openstack-infra/project-config master: WIP: Fix neutron-failure-rate grafana dashboard https://review.openstack.org/509291 | 18:34 |
| *** dprince has joined #openstack-infra | 18:38 | |
| fungi | SamYaple: you're not alone... i'm feeling a bit laggy today myself | 18:41 |
| *** gridinv1 has joined #openstack-infra | 18:42 | |
| SamYaple | i left my windows open lastnight before i passed out and woke up shivering and sneezing. i have made mistakes | 18:42 |
| *** caphrim007_ has joined #openstack-infra | 18:43 | |
| *** caphrim007 has quit IRC | 18:44 | |
| *** armaan has quit IRC | 18:45 | |
| *** armaan has joined #openstack-infra | 18:46 | |
| *** eharney has quit IRC | 18:47 | |
| *** edmondsw has joined #openstack-infra | 18:48 | |
| clarkb | mordred: have you seen https://setuptools.readthedocs.io/en/latest/setuptools.html#configuring-setup-using-setup-cfg-files | 18:50 |
| inc0 | hey guys, looking at hostvars of zuulv3: "primary" node is always going to be there and one instance of it? | 18:51 |
| inc0 | http://paste.openstack.org/show/622689/ | 18:51 |
| clarkb | inc0: not necessarily, it depends on how you set up your nodeset | 18:51 |
| clarkb | inc0: that is how we've done the legacy nodesets though | 18:51 |
| inc0 | ahh ok I see it now, cool, that's ok | 18:52 |
| clarkb | mordred: if we ignore the sphinx and test and versioning stuff in pbr, could we potentially just use that instead? | 18:52 |
| *** pt_15 has quit IRC | 18:53 | |
| clarkb | also pbr made it into the setuptools docs!? | 18:54 |
| *** jamesmcarthur has quit IRC | 18:54 | |
| clarkb | oh entry points is not supported | 18:54 |
| * clarkb wonders why not its all just parameter passing | 18:55 | |
| *** jcoufal_ has quit IRC | 18:58 | |
| *** gouthamr has quit IRC | 18:58 | |
| clarkb | that is amazingness | 18:58 |
| *** gouthamr_ has joined #openstack-infra | 18:59 | |
| *** MasterOfBugs has quit IRC | 18:59 | |
| clarkb | oh it can do entrypoints just as options.entrypoints? in any case something to think about possible | 18:59 |
| *** vhosakot has quit IRC | 19:00 | |
| fungi | yeah, could simplify pbr somewhat (though only in conjunction with very recent setuptools) | 19:02 |
| *** baoli has quit IRC | 19:02 | |
| *** baoli has joined #openstack-infra | 19:03 | |
| *** eharney has joined #openstack-infra | 19:03 | |
| *** felipemonteiro__ has quit IRC | 19:04 | |
| *** felipemonteiro__ has joined #openstack-infra | 19:04 | |
| *** oomichi is now known as oomichi_afk | 19:05 | |
| *** edmondsw has quit IRC | 19:06 | |
| *** edmondsw has joined #openstack-infra | 19:06 | |
| *** jcoufal has joined #openstack-infra | 19:08 | |
| mordred | clarkb: wow - no I haven't | 19:10 |
| *** gridinv1 has quit IRC | 19:10 | |
| mordred | fungi: we PRETTY much require recent setuptools anyway | 19:10 |
| fungi | fair, though that's _really_ recent setuptools | 19:11 |
| fungi | like less than a year new | 19:11 |
| mordred | yah - it's exciting though - biggest question remaining for me (especially if we adopt my proposal for changing docs builds) | 19:11 |
| mordred | will be versioning and requirements processing ... although we could just shift to using the new Pipfile / Pipfile.lock stuff for requirements | 19:12 |
| clarkb | ya though thats still in flux | 19:12 |
| clarkb | however pipenv is apparently blessed as official now and people are using it? | 19:13 |
| mordred | which would leave us with versioning - and at that point doing that as a setuptools plugin like the setuptools_scm plugin is (if setuptools_scm is not sufficient for us) | 19:13 |
| clarkb | (it deps on pipfile/pipfile.lock) | 19:13 |
| mordred | clarkb: yah - mainly just saying, if pipenv and pipfile are the new overlords, I don't know that there is a ton of value in us maintaining a similar-but-different thing in pbr ... | 19:13 |
| clarkb | yup | 19:14 |
| *** slaweq_ has joined #openstack-infra | 19:14 | |
| clarkb | I'm just pointing out that pipfile isn't technically stable/ready/usable yet | 19:15 |
| clarkb | but we can probably get away with that | 19:15 |
| clarkb | esepcially if pipenv is using it | 19:15 |
| mordred | clarkb: us using pipfile is a GREAT way to help find all of the bugs | 19:15 |
| mordred | I'm sure dstufft would love us :) | 19:15 |
| clarkb | ha | 19:16 |
| *** chlong_ has quit IRC | 19:16 | |
| fungi | i smell a new community goal brewing | 19:17 |
| *** Rockyg has quit IRC | 19:21 | |
| *** bnemec is now known as bnemec-backlater | 19:24 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard master: Check story permissions when populating automatic worklists https://review.openstack.org/509601 | 19:25 |
| *** camunoz has quit IRC | 19:27 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Handle double node locking snafu https://review.openstack.org/509603 | 19:27 |
| dmsimard | fungi: speaking of community goal, that reminds me I meant to ask at the last -infra meeting if there was a formal "sign up" or "onboarding" process for https://governance.openstack.org/tc/reference/top-5-help-wanted.html#community-infrastructure-sysadmins | 19:29 |
| dmsimard | Or if it's just a matter of helping, time, trust etc | 19:29 |
| *** hemna_ has quit IRC | 19:29 | |
| fungi | dmsimard: the latter, though we'll of course report on new additions as we get them | 19:31 |
| fungi | and if we ever end up flush with volunteers, we can recommend striking it off the top 5 help wanted list | 19:32 |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Add python-tempestconf project https://review.openstack.org/508502 | 19:34 |
| *** aviau has quit IRC | 19:35 | |
| *** aviau has joined #openstack-infra | 19:35 | |
| *** lbragstad has quit IRC | 19:35 | |
| dmsimard | fungi: makes sense, I'll keep on helping and see what happens :) | 19:36 |
| fungi | thanks! we appreciate all the help | 19:38 |
| *** baoli_ has joined #openstack-infra | 19:39 | |
| *** Sukhdev has joined #openstack-infra | 19:39 | |
| *** baoli has quit IRC | 19:41 | |
| *** bobh has joined #openstack-infra | 19:41 | |
| *** ijw has quit IRC | 19:46 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Handle double node locking snafu https://review.openstack.org/509603 | 19:47 |
| *** Goneri has quit IRC | 19:48 | |
| *** jtomasek has quit IRC | 19:48 | |
| *** Sukhdev has quit IRC | 19:51 | |
| *** nicolasbock has joined #openstack-infra | 19:52 | |
| rajinir | clarkb: Somehow, the git problem was solved. There was a REPOLOCATION hardcoded in one of the nodepool elements | 20:01 |
| *** vhosakot has joined #openstack-infra | 20:01 | |
| rajinir | https://www.irccloud.com/pastebin/SwsDMEMX/new%20error | 20:01 |
| rajinir | clarkb or anyone, the dib-image is not building | 20:02 |
| *** ianw is now known as ianw|pto | 20:04 | |
| ianw|pto | rajinir: that should be added by project-config:nodepool/elements/jenkins-slave/install.d/20-jenkins-slave ... are you using the right elements? | 20:07 |
| *** armaan has quit IRC | 20:07 | |
| *** felipemonteiro_ has joined #openstack-infra | 20:08 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/nodepool feature/zuulv3: Bring back per label groups in Openstack https://review.openstack.org/509620 | 20:08 |
| *** felipemonteiro__ has quit IRC | 20:11 | |
| rajinir | ianw|pto> May be I have to get the new elements | 20:13 |
| *** jcoufal has quit IRC | 20:14 | |
| openstackgerrit | Anup Navare proposed openstack-infra/project-config master: [test] DNM Checking gate jobs https://review.openstack.org/509627 | 20:16 |
| *** ijw has joined #openstack-infra | 20:17 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add mysql reporter to the other pipelines https://review.openstack.org/509212 | 20:18 |
| *** baoli_ has quit IRC | 20:20 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy-releases-python35 https://review.openstack.org/509091 | 20:25 |
| *** lbragstad has joined #openstack-infra | 20:27 | |
| *** eroux has quit IRC | 20:30 | |
| *** Goneri has joined #openstack-infra | 20:35 | |
| mordred | clarkb: feel like pulling the trigger on https://review.openstack.org/#/c/410328/ ? | 20:38 |
| mordred | clarkb: fungi just +2d it this morning - maybe it'll fix my can't-log-in issues :) | 20:38 |
| *** jbadiapa_ has quit IRC | 20:39 | |
| *** zzzeek has quit IRC | 20:39 | |
| *** jascott1 has joined #openstack-infra | 20:40 | |
| * clarkb looks | 20:40 | |
| *** ijw has quit IRC | 20:40 | |
| clarkb | fungi: mordred does that need a db migration step too? or did we apply that already? | 20:41 |
| mordred | clarkb: I do not know the answer to that question - our admin users are certain specified using login.launchpad.net openid identifiers | 20:43 |
| clarkb | mordred: fungi I've approved it fwiw, I think worst case we just update the db after hte fact | 20:43 |
| mordred | clarkb: cool | 20:44 |
| *** jkilpatr has quit IRC | 20:44 | |
| *** ijw has joined #openstack-infra | 20:45 | |
| *** gouthamr_ has quit IRC | 20:46 | |
| *** ijw has quit IRC | 20:46 | |
| *** mat128 has quit IRC | 20:53 | |
| *** ijw has joined #openstack-infra | 20:55 | |
| *** ijw has quit IRC | 20:56 | |
| *** ijw has joined #openstack-infra | 20:56 | |
| *** srobert_ has joined #openstack-infra | 20:57 | |
| *** zzzeek has joined #openstack-infra | 20:57 | |
| *** hasharAway has quit IRC | 20:58 | |
| *** srobert has quit IRC | 20:58 | |
| fungi | yeah, odds are one of us will need to just do a mass update query across all the existing users as soon as that gets deployed | 21:00 |
| fungi | something like the etl mordred did to gerrit | 21:00 |
| *** trown is now known as trown|outtypewww | 21:01 | |
| *** rwsu has joined #openstack-infra | 21:02 | |
| openstackgerrit | Merged openstack-infra/system-config master: Switch storyboard to login.ubuntu.com https://review.openstack.org/410328 | 21:03 |
| *** armaan has joined #openstack-infra | 21:06 | |
| tonyb | evrardjp: No need, I just need to understand why this is a new prob;em and weather it's tied up in the "new" zuul-cloner. | 21:07 |
| tonyb | ooops | 21:07 |
| *** eumel8 has quit IRC | 21:11 | |
| *** Goneri has quit IRC | 21:12 | |
| *** jcoufal has joined #openstack-infra | 21:12 | |
| *** jcoufal has quit IRC | 21:14 | |
| *** ldnunes has quit IRC | 21:16 | |
| *** e0ne has quit IRC | 21:18 | |
| openstackgerrit | Anup Navare proposed openstack-infra/devstack-gate master: [Test] DNM Checking if tinyIPA builds with py3 https://review.openstack.org/509641 | 21:18 |
| *** felipemonteiro_ has quit IRC | 21:19 | |
| *** felipemonteiro_ has joined #openstack-infra | 21:19 | |
| *** zzzeek has quit IRC | 21:19 | |
| *** e0ne has joined #openstack-infra | 21:19 | |
| *** e0ne has quit IRC | 21:19 | |
| *** e0ne has joined #openstack-infra | 21:20 | |
| *** ociuhandu has joined #openstack-infra | 21:20 | |
| *** e0ne has quit IRC | 21:20 | |
| *** zzzeek has joined #openstack-infra | 21:20 | |
| *** kgiusti has left #openstack-infra | 21:21 | |
| *** e0ne has joined #openstack-infra | 21:21 | |
| *** e0ne has quit IRC | 21:22 | |
| *** e0ne has joined #openstack-infra | 21:22 | |
| *** e0ne has quit IRC | 21:23 | |
| *** e0ne has joined #openstack-infra | 21:23 | |
| *** e0ne has quit IRC | 21:23 | |
| *** zzzeek has quit IRC | 21:23 | |
| *** zzzeek has joined #openstack-infra | 21:25 | |
| *** jkilpatr has joined #openstack-infra | 21:29 | |
| *** edmondsw has quit IRC | 21:32 | |
| *** jascott1 has quit IRC | 21:32 | |
| *** edmondsw has joined #openstack-infra | 21:32 | |
| *** slaweq_ has quit IRC | 21:34 | |
| *** edmondsw has quit IRC | 21:37 | |
| *** jascott1 has joined #openstack-infra | 21:42 | |
| *** srobert_ has quit IRC | 21:42 | |
| *** jascott1 has quit IRC | 21:43 | |
| *** jascott1 has joined #openstack-infra | 21:44 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Don't store pipeline references on builds https://review.openstack.org/509653 | 21:46 |
| *** sree has joined #openstack-infra | 21:49 | |
| *** ijw has quit IRC | 21:53 | |
| *** sree has quit IRC | 21:53 | |
| *** ijw has joined #openstack-infra | 21:54 | |
| *** ijw has quit IRC | 21:58 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Don't store pipeline references on builds https://review.openstack.org/509653 | 22:00 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Update node requests after nodes https://review.openstack.org/509571 | 22:00 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Move test_model.test_job_inheritance_configloader https://review.openstack.org/509495 | 22:00 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove test_job_inheritance https://review.openstack.org/509509 | 22:00 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add git timeout https://review.openstack.org/509517 | 22:00 |
| *** rlandy has quit IRC | 22:02 | |
| *** dprince has quit IRC | 22:02 | |
| jeblair | i'm restarting zuulv3 with 509653 applied locally; it's expected to fix the memory leak problems | 22:04 |
| *** slaweq_ has joined #openstack-infra | 22:04 | |
| *** threestrands has joined #openstack-infra | 22:08 | |
| *** eharney has quit IRC | 22:11 | |
| mnaser | ^ woo | 22:13 |
| *** felipemonteiro_ has quit IRC | 22:15 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Don't store pipeline references on builds https://review.openstack.org/509653 | 22:21 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Update node requests after nodes https://review.openstack.org/509571 | 22:21 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Move test_model.test_job_inheritance_configloader https://review.openstack.org/509495 | 22:21 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove test_job_inheritance https://review.openstack.org/509509 | 22:21 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add git timeout https://review.openstack.org/509517 | 22:21 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add debug messages and exception handling to stack dump handler https://review.openstack.org/508786 | 22:22 |
| *** gouthamr has joined #openstack-infra | 22:26 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/nodepool feature/zuulv3: Bring back per label groups in Openstack https://review.openstack.org/509620 | 22:27 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Speed configuration building https://review.openstack.org/509309 | 22:27 |
| clarkb | jeblair: I'm assuming its been restarted at this point? | 22:28 |
| jeblair | clarkb: yes | 22:29 |
| jeblair | clarkb: i think it did those 2 merges | 22:30 |
| SpamapS | jeblair: wow, that must have been quite a spelunking expedition into the pipeline<->layout plumbing | 22:30 |
| jeblair | SpamapS: https://i.imgur.com/JqtYeG5.jpg | 22:31 |
| jeblair | i *almost* walked down to the copy center to have that printed, but then i saw enough of the pattern with the help of xdot that i could start to pull apart the threads | 22:31 |
| *** ociuhandu has quit IRC | 22:32 | |
| mordred | mnaser: you got new patches up for the requirements jobs, yeah? | 22:33 |
| mnaser | mordred releasenotes? yes | 22:33 |
| mordred | mnaser: oh - no - you were working on relasenotes | 22:33 |
| SpamapS | jeblair: I would have been printing it too. :) | 22:34 |
| mnaser | those seem to be working but i need to test bumping test-requirements to 2.5.1 for reno | 22:34 |
| SpamapS | jeblair: so the good news is that while it was growing somewhat out of control.. the growth wasn't actually infinite. | 22:35 |
| *** tpsilva has quit IRC | 22:35 | |
| mordred | mnaser: kk. cool - I just added a thing to do the todo list for moving the requirements job to the requirmeents repo before we re-roll-forward - since I think we'll be able to get it done more quickly before than after | 22:35 |
| clarkb | mordred: because I am terrible at keeping track of anything, was the tox fetch output thing sortedout? | 22:35 |
| mordred | clarkb: yes - the primary problem was a zuul bug that jeblair fixed | 22:35 |
| SpamapS | jeblair: so if we had just resized onto a 32G instance, we might have been ok. ;) | 22:35 |
| mordred | clarkb: and not actually a job issue | 22:36 |
| mordred | clarkb: I've gotta run - but when I get back and/or first thing in the morning I think we can crank through the rest of the issues pretty quickly | 22:36 |
| *** lbragstad has quit IRC | 22:38 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Change yappi bytesio to stringio https://review.openstack.org/508787 | 22:38 |
| *** slaweq_ has quit IRC | 22:39 | |
| *** rbrndt has quit IRC | 22:41 | |
| *** edmondsw has joined #openstack-infra | 22:41 | |
| openstackgerrit | Merged openstack-infra/zuul-sphinx master: Update exception message to include directories https://review.openstack.org/505400 | 22:42 |
| *** slaweq_ has joined #openstack-infra | 22:44 | |
| *** edmondsw has quit IRC | 22:46 | |
| *** bobh has quit IRC | 22:51 | |
| *** jdg has joined #openstack-infra | 22:53 | |
| *** ijw has joined #openstack-infra | 22:55 | |
| *** jdg has quit IRC | 22:55 | |
| *** xarses has quit IRC | 22:57 | |
| andreaf | jeblair, mordred: in case you have time for reviews, my proposed zuul-jobs and tempest roles this far https://review.openstack.org/#/q/status:open+branch:master+topic:zuulv3-tempest | 22:59 |
| *** sdague has quit IRC | 23:00 | |
| *** esberglu has quit IRC | 23:00 | |
| *** ijw has quit IRC | 23:01 | |
| *** esberglu has joined #openstack-infra | 23:01 | |
| *** armax has quit IRC | 23:01 | |
| clarkb | mordred: for your first thing in the morning http://logs.openstack.org/59/509659/1/check/build-openstack-releasenotes/6127917/job-output.txt.gz#_2017-10-04_22_23_07_256421 looks like release notes is still unhappy | 23:02 |
| clarkb | mordred: I think that is probably one of the biggest outstanding job issues that we want to sort out before hitting go again | 23:03 |
| * clarkb checks up on the tripleo situation | 23:03 | |
| mnaser | mordred ive been working on that | 23:04 |
| mnaser | err, i mean clarkb | 23:04 |
| clarkb | mnaser: the release notes stuff? | 23:05 |
| mnaser | clarkb i believe that failure is because nova is pinned to 2.2.1 and the fix for that issue is in 2.5.1 (i think)2 | 23:05 |
| *** esberglu has quit IRC | 23:05 | |
| clarkb | reno is pinned you mean? | 23:06 |
| mnaser | *but* there is another failure that will happen after that which we caught in puppet gates, which im testing a fix for .. at this stack https://review.openstack.org/#/c/509558/ | 23:06 |
| clarkb | http://git.openstack.org/cgit/openstack/requirements/tree/upper-constraints.txt#n151 says 2.5.1 | 23:06 |
| mnaser | yes, reno is pinned | 23:06 |
| mnaser | the logs of that run shows 2.2.1 | 23:06 |
| clarkb | mnaser: oh could be stable branch above | 23:06 |
| mnaser | maybe this ran in a stable branch? | 23:07 |
| mnaser | :P | 23:07 |
| clarkb | except I thought release notes only ever run against master | 23:07 |
| clarkb | so that coul djust be a branch exclusion bug | 23:07 |
| mnaser | (regardless, there are a few other issues other than the older reno version which im fixing at that stack .. given how awesome zuul has been performing right now its easy to iterate right now) | 23:07 |
| mnaser | clarkb afaik there is no limiting in project-config on release note jobs running in master only | 23:08 |
| mnaser | maybe we need to update the job definition to add it to master only | 23:08 |
| jlvillal | I had a question on the logfile display code: For example on this one: http://logs.openstack.org/73/507373/1/check/gate-tempest-dsvm-ironic-ipa-wholedisk-agent_ipmitool-tinyipa-multinode-ubuntu-xenial/53a4b0f/logs/subnode-2/screen-g-api.txt.gz?level=ERROR | 23:08 |
| jlvillal | How would we get it to also show "CRITICAL" errors | 23:08 |
| clarkb | jlvillal: I don't believe CRITICAL is a valid python logging level /me checks | 23:09 |
| clarkb | oh it is | 23:09 |
| jlvillal | clarkb: Well I see it in some logfiles :) | 23:09 |
| clarkb | cool, you'll need to update os-loganalyze (ya in the past openstack has added its own log levels which are ugh and I'd like to see us not facilitate that furhter but CRITICAL is a proper level) | 23:10 |
| jlvillal | clarkb: Thanks. I'll go look for it! | 23:10 |
| clarkb | mnaser: ya lets double check with dhellmann but I seem to recall that it was said only master updates the release notes | 23:10 |
| *** SumitNaiksatam has quit IRC | 23:11 | |
| mnaser | clarkb cool, on my side ill work to make sure the jobs are fixed with master and we can double check that detail | 23:11 |
| clarkb | jlvillal: https://git.openstack.org/cgit/openstack-infra/os-loganalyze/tree/os_loganalyze/filter.py#n52 | 23:11 |
| jlvillal | clarkb: Awesome :) | 23:11 |
| *** edmondsw has joined #openstack-infra | 23:13 | |
| *** slaweq__ has joined #openstack-infra | 23:16 | |
| *** slaweq_ has quit IRC | 23:17 | |
| *** hongbin has quit IRC | 23:18 | |
| SamYaple | If I am building and installing wheels, will I ever use .pydistutils.cfg ? I think it is just for easy_install, but im not sure | 23:18 |
| clarkb | SamYaple: .pydistutils.cfg is only for easy_install, it pops up in unexpected situations occasionally even when using pip | 23:19 |
| clarkb | SamYaple: a common one is setup_requires which are avoided by using wheels | 23:19 |
| SamYaple | got it. so in my case i dont need to setup that for mirroring. | 23:20 |
| SamYaple | thanks | 23:20 |
| *** felipemonteiro_ has joined #openstack-infra | 23:20 | |
| clarkb | assuming all your packages always push wheels before sdists on new releases, sure | 23:21 |
| *** slaweq__ has quit IRC | 23:21 | |
| clarkb | (as we dicsovered with cffi today that isn't always the case ) | 23:21 |
| *** felipemonteiro__ has joined #openstack-infra | 23:21 | |
| SamYaple | im just building dockerimages and produce only .whl's and consume them in such a way shouldn't that be possible for a file without the extention .whl to slip through | 23:23 |
| SamYaple | it would break from missing a package before it would fetch it from upstream | 23:23 |
| SamYaple | now i am trying to track down what $AFS_SLUG is to setup the extra-index-url . Where is this set? | 23:24 |
| clarkb | I want to say thats a docs only thing? I forget | 23:24 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/os-loganalyze master: Support the log level "CRITICAL" https://review.openstack.org/509670 | 23:25 |
| SamYaple | is it? hmmm then i must be having a different failure | 23:25 |
| jlvillal | clarkb: ^^ Not totally sure I did it all correctly. But I hope I did :) | 23:25 |
| *** felipemonteiro_ has quit IRC | 23:25 | |
| SamYaple | HTTPError: 404 Client Error: Not Found for url: http://mirror.gra1.ovh.openstack.org/pypi/simple/pkg-resources/ | 23:25 |
| clarkb | jlvillal: maybe add a test for it too? | 23:26 |
| jlvillal | clarkb: Yeah. good idea... | 23:26 |
| clarkb | SamYaple: pkg-resources is not an independent package, its part of setuptools iirc | 23:26 |
| SamYaple | so maybe i have an out-of-date setuptools or something | 23:26 |
| SamYaple | oh wait. im pip installing setuptools to get the latest after dropping the pip.conf with the mirrors | 23:27 |
| SamYaple | strange this only breaks me on ubuntu and not centos though | 23:27 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Don't store pipeline references on builds https://review.openstack.org/509653 | 23:27 |
| clarkb | centos and debuntu install python packages from system and pip differently | 23:27 |
| SamYaple | youre refering to /usr/local on debian ? | 23:28 |
| clarkb | SamYaple: yes | 23:29 |
| SamYaple | hmmm. ok. thats a good starting point to troubleshoot the issue further | 23:29 |
| SamYaple | hoping i dont have to switch back to get-pip.py | 23:29 |
| * jlvillal grumbles that the sample log files are gzipped | 23:30 | |
| SamYaple | jlvillal: really? i love that. firefox can handle gzipped logs in browser and they load much quicker | 23:30 |
| jlvillal | SamYaple: The ones in the test cases are gzipped. I think the web server decompresses them. When I wget the files they are unzipped | 23:31 |
| *** gmann_sleep is now known as gmann | 23:32 | |
| SamYaple | i wonder if because you arent passing the gzip header its just serving you uncompressed ones | 23:32 |
| clarkb | SamYaple: that is exactly what it is | 23:32 |
| clarkb | jlvillal: I think they are gzipped to test the gzip facilities in os-loganalyze because we store them compressed as well | 23:32 |
| clarkb | jlvillal: since even compressed can only store about 4 weeks of data | 23:33 |
| *** edmondsw has quit IRC | 23:35 | |
| *** wolverineav has quit IRC | 23:43 | |
| *** slaweq_ has joined #openstack-infra | 23:46 | |
| SamYaple | I am trying to find clarity here on what --extra-index-url does, but coming up short. Ill just trust your knowledge that that should only be used for docs | 23:49 |
| SamYaple | my working understanding is its for local package installs (that do not exist upstream) | 23:50 |
| clarkb | SamYaple: https://pip.pypa.io/en/stable/reference/pip_wheel/#extra-index-url | 23:50 |
| clarkb | SamYaple: we use it for the wheel mirror, and now I remeber that afs slug is a poorly named thing that figures out the location of the mirror or something | 23:50 |
| SamYaple | so i *should* be setting it? | 23:51 |
| clarkb | AFS_SLUG="$OS_TYPE-$OS_VERSION-$OS_ARCH" | 23:51 |
| clarkb | its the tuple that identifies which wheel mirror is appropriate for the current platform | 23:51 |
| clarkb | (so is poorly named) | 23:51 |
| clarkb | SamYaple: it shouldn't be required but could speed up jobs / avoid needing C dev libs in some cases | 23:52 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/os-loganalyze master: Support the log level "CRITICAL" https://review.openstack.org/509670 | 23:52 |
| jlvillal | clarkb: Now with unit tests :) | 23:52 |
| SamYaple | clarkb: my entire goal from one build is to just output wheels, if i can grab already built wheels that would be ideal | 23:52 |
| *** ijw has joined #openstack-infra | 23:53 | |
| clarkb | SamYaple: the wheel mirror is wheels that we have built for platforms like ubuntu 16.04 amd64 and centos7.4 x86-64 | 23:53 |
| clarkb | SamYaple: http://mirror.dfw.rax.openstack.org/wheel/ | 23:54 |
| clarkb | SamYaple: /etc/ci/mirror_info.sh has NODEPOOL_WHEEL_MIRROR defined in it and will have the correct url for the current platform | 23:55 |
| clarkb | (you can source that file) | 23:55 |
| clarkb | though I think if you are looking for non legacy jobs you may want to directly construct that from ansible facts | 23:55 |
| clarkb | (to be less legacy) | 23:55 |
| SamYaple | yea i already construct from ansible facts (and this happens inside a container too) | 23:55 |
| SamYaple | so the question for me is, should I be consuming those, or is the output of those wheels non-deterministic | 23:56 |
| *** jdg has joined #openstack-infra | 23:56 | |
| clarkb | I don't think any wheel buidl can be considered deterministic | 23:56 |
| clarkb | because python | 23:56 |
| clarkb | the biggest trouble you might run into is anything that links against places like UCA | 23:56 |
| SpamapS | very few builds of anything anywhere are deterministic | 23:57 |
| SamYaple | fair enough, but they do pull in dev libs so can the wheels go out of date if a new lib is released (for say a security fix) | 23:57 |
| pabelanger | o/ | 23:57 |
| SpamapS | see: repeatable builds in debian effort | 23:57 |
| clarkb | beacuse then you've got the base distro package and hte UCA package and you can only wheel against one of them in cases | 23:57 |
| pabelanger | I have internets again | 23:57 |
| clarkb | SamYaple: yes | 23:57 |
| clarkb | SamYaple: its more pronounced with eg UCA because they don't guarantee api compat with the original distro pacakages | 23:57 |
| clarkb | SpamapS: security updates should however | 23:57 |
| clarkb | er SamYaple ^ | 23:57 |
| SamYaple | best advice then, should i consume these wheels in the container builds (that will eventually get pushed to dokcerhub) or not? | 23:58 |
| *** baoli has joined #openstack-infra | 23:58 | |
| SpamapS | IIRC, UCA isn't handled by the ubuntu security team directly. | 23:58 |
| clarkb | if you're testing they work it should be fine I think | 23:58 |
| *** armax has joined #openstack-infra | 23:58 | |
| clarkb | if not then you might want to build your own just to be sure the wheel in the container lines up with the packages in the container | 23:58 |
| *** ijw has quit IRC | 23:59 | |
| SamYaple | ok. all good info. ill sync with the team and see what they think | 23:59 |
| *** baoli has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
