Friday, 2020-10-30

« Thursday, 2020-10-29 Index Saturday, 2020-10-31 »
*** zbr has quit IRC00:11
*** zbr has joined #oooq00:11
*** tosky has quit IRC00:27
*** saneax has quit IRC00:57
*** dsneddon has quit IRC02:24
*** dsneddon has joined #oooq02:25
*** dsneddon has quit IRC02:31
*** dsneddon has joined #oooq02:40
*** jmasud has joined #oooq03:22
*** udesale has joined #oooq04:08
*** ysandeep|away is now known as ysandeep|ruck04:16
*** matbu has quit IRC04:22
*** matbu has joined #oooq04:24
*** irclogbot_0 has quit IRC04:27
*** irclogbot_1 has joined #oooq04:31
*** dsneddon has quit IRC04:52
*** dsneddon has joined #oooq05:10
*** ykarel has joined #oooq05:10
*** dsneddon has quit IRC05:15
*** akahat is now known as akahat|PTO05:20
*** ratailor has joined #oooq05:28
*** marios has joined #oooq06:22
*** ysandeep|ruck is now known as ysandeep|afk06:33
*** skramaja has joined #oooq06:47
*** amoralej|off is now known as amoralej07:54
*** saneax has joined #oooq08:02
*** ysandeep|afk is now known as ysandeep08:02
*** ysandeep is now known as ysandeep|ruck08:06
marios10:15 #tripleo: < openstackgerrit> Marios Andreou proposed openstack/tripleo-quickstart master: Restore centos7 master release config for content providers  https://review.opendev.org/76050908:16
*** frenzy_friday has joined #oooq08:19
*** frenzy_friday has quit IRC08:20
*** frenzy_friday has joined #oooq08:21
mariosykarel: o/08:33
marioswana talk on phone 5 mins ? about https://review.opendev.org/#/c/760438/08:34
ykarelmarios, hi08:35
marioso/08:35
ykarellink?08:35
marioshttps://meet.google.com/uzh-rshu-cix08:36
mariosykarel: ^^08:36
*** tosky has joined #oooq08:50
*** jpena|off is now known as jpena08:58
ysandeep|rucksshnaidm|rover, if you are working today, fyi.. we hit an issue in master - https://bugs.launchpad.net/tripleo/+bug/1902190 , I am trying to check for cause09:00
openstackLaunchpad bug 1902190 in tripleo "Master periodic jobs are failing with "msg": "Error running container image prepare: failed"" [Critical,New]09:00
sshnaidm|roverysandeep|ruck, ack, I'm off in Fri, will look later09:06
*** sshnaidm|rover is now known as sshnaidm|roff09:06
*** sshnaidm|roff is now known as sshnaidm|off09:06
ysandeep|rucksshnaidm|off, nw please enjoy your pto, i will take care of this.. i got confused by your irc nick09:08
*** dtantsur|afk is now known as dtantsur09:08
*** ysandeep|ruck is now known as ysandeep|lunch09:11
*** slaweq has joined #oooq09:18
*** derekh has joined #oooq09:30
*** dtantsur is now known as dtantsur|afk10:06
*** dtantsur|afk is now known as dtantsur|brb10:06
*** ysandeep|lunch is now known as ysandeep|ruck10:27
*** ykarel has quit IRC10:52
*** ykarel has joined #oooq10:53
*** dtantsur|brb is now known as dtantsur11:01
*** pojadhav is now known as pojadhav|afk11:21
*** ysandeep|ruck is now known as ysandeep|brb11:32
*** anbanerj_ has joined #oooq11:41
*** frenzy_friday has quit IRC11:43
*** ysandeep|brb is now known as ysandeep|ruck11:49
mariosweshay|ruck: please vote and we can merge https://review.opendev.org/#/c/759680/11:49
*** rlandy has joined #oooq12:07
*** udesale_ has joined #oooq12:08
*** ratailor has quit IRC12:09
*** udesale has quit IRC12:11
*** anbanerj_ is now known as frenzy_friday12:23
weshay|ruckdone12:26
mariosweshay|ruck: thank you12:26
weshay|ruckysandeep|ruck,12:28
weshay|ruck FATAL | Run yum_update.sh | localhost | error={"changed": true, "cmd": ["buildah", "run", "--volume", "/tmp/ansible.zgerf8ty:/tmp/yum_update.sh", "--volume", "/etc/yum.repos.d:/etc/yum.repos.d", "--volume", "/etc/pki:/etc/pki", "--user", "root", "--net", "host", "openstack-cron-working-container", "/tmp/yum_update.sh", "delorean-current"], "delta": "0:00:00.611050", "end": "2020-10-30 01:37:23.728489", "msg": "non-zero return code",12:29
weshay|ruck"rc": 1, "start": "2020-10-30 01:37:23.117439", "stderr": "standard_init_linux.go:211: exec user process caused \"exec format error\"", "stderr_lines": ["standard_init_linux.go:211: exec user process caused \"exec format error\""], "stdout": "", "stdout_lines": []}12:29
ysandeep|ruckweshay|ruck, hi12:29
*** dsneddon has joined #oooq12:29
weshay|ruckhey..12:29
ysandeep|ruckweshay|ruck, https://bugs.launchpad.net/tripleo/+bug/190219012:29
openstackLaunchpad bug 1902190 in tripleo "Master periodic jobs are failing with "msg": "Error running container image prepare: failed"" [Critical,New]12:29
ysandeep|ruckweshay|ruck, but i didn't see the same error in last execution12:29
weshay|ruckso .. I think we should be able to turn off container updates in child jobs..12:29
ysandeep|ruckweshay|ruck, where are you seeing this?12:30
weshay|ruckysandeep|ruck, https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-master/21a7770/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz12:30
weshay|ruckhttps://bugs.launchpad.net/tripleo/+bug/190219012:30
weshay|ruckfrom your bug12:30
ysandeep|ruckweshay|ruck, last run which is in zuul queue passed12:30
weshay|ruckContainerImagePrepare:12:31
weshay|ruck    -   modify_append_tag: -updated-2020103001220212:31
weshay|ruck        modify_only_with_source:12:31
weshay|ruck        - kolla12:31
weshay|ruck        - tripleo12:31
weshay|ruck        modify_role: tripleo-modify-image12:31
weshay|ruck        modify_vars:12:31
weshay|ruck            tasks_from: yum_update.yml12:31
weshay|ruck            update_repo: delorean-current12:31
weshay|ruck            yum_repos_dir_path: /etc/yum.repos.d12:31
weshay|ruckhttps://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-master/21a7770/logs/undercloud/home/zuul/containers-prepare-parameters.yaml.txt.gz12:31
weshay|ruckthose are the settings to turn off container updates12:31
weshay|ruckwhich we should consider for child jobs12:32
weshay|ruckmarios, think that's possible idea?12:32
weshay|ruckmarios, re: https://review.opendev.org/#/c/760438/12:32
weshay|ruckmarios, skipped jobs fail the gate12:33
weshay|ruckthere is no reason to have content providers vote.. they don't test anything12:33
mariosweshay|ruck: joining us on content provider call?12:33
weshay|ruckaye12:33
*** jpena is now known as jpena|lunch12:34
*** saneax has quit IRC12:34
ysandeep|ruckweshay|ruck, mind if i hop in too, RR this sprint .. will be easy for me to tackle bugs next week12:34
ysandeep|ruckmarios, ^^12:34
mariosweshay|ruck: rlandy: https://review.opendev.org/75965412:38
mariosysandeep|ruck: of course :D you don't need to ask for permission! :D12:38
ysandeep|ruckmarios, could you please pass me the meeting link12:39
ysandeep|ruckgot it from yesterday's chat12:40
*** rfolco has joined #oooq12:41
marioshttps://review.opendev.org/#/c/76008912:47
marioshttps://review.opendev.org/#/c/76043812:47
rfolcoweshay|ruck, rlandy please lets merge this one https://review.opendev.org/#/c/758627/12:52
rfolcoand https://review.opendev.org/#/c/759429/12:53
mariosweshay|ruck: ysandeep|ruck: rlandy: https://review.opendev.org/#/c/759654/ https://review.opendev.org/#/c/759702/3 https://review.opendev.org/#/c/760323/212:58
mariosreviews please ^^^12:59
rlandymarios: Add content provider jobs for tripleo based jobs12:59
weshay|ruckmarios, https://review.opendev.org/#/c/759654/3/zuul.d/content-provider.yaml13:00
mariosarxcruz: please check that ^^^ make it just about content provider victoria so we can merge please13:01
*** udesale_ has quit IRC13:01
arxcruzmarios: ack13:07
mariosarxcruz: lets do the move in a new patch after we're done13:08
*** derekh has quit IRC13:13
arxcruzmarios: done13:14
mariosweshay|ruck: https://review.opendev.org/#/c/759985/20//COMMIT_MSG13:16
mariosarxcruz: thanks13:17
arxcruzforgot to update the commit message13:17
weshay|ruckhttps://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-master/21a7770/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz13:20
ysandeep|ruckweshay|ruck, 2020-10-30 01:42:53.809122 | fa163eff-949c-feba-7e4b-000000000db1 |      FATAL | Container image prepare | undercloud | error={"changed": false, "error": "Modifying image 192.168.24.1:8787/tripleomaster/openstack-glance-api:938acab2d960d1e9b3ede6e4d35983a8 failed", "msg": "Error running container image prepare: Modifying image 192.168.24.1:8787/tripleomaster/openstack-glance-api:938acab2d960d1e9b3ede6e4d35983a8 failed",13:21
ysandeep|ruck"params": {}, "success": false}13:21
ysandeep|ruckweshay|ruck, https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-master/c425eec/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz13:21
arxcruzweshay|ruck: 1-1?13:30
weshay|ruckarxcruz, in content provider call still13:30
weshay|rucklots o fun13:30
*** jpena|lunch is now known as jpena13:30
*** amoralej is now known as amoralej|lunch13:33
mariosposted https://review.opendev.org/76036013:34
marioshttps://review.opendev.org/76035913:34
*** skramaja has quit IRC13:35
*** TrevorV has joined #oooq13:36
weshay|ruckarxcruz, /me gets coffee then we can chat13:41
arxcruzweshay|ruck: ack13:41
*** TrevorV has quit IRC13:43
*** TrevorV has joined #oooq13:44
*** pojadhav|afk is now known as pojadhav13:45
weshay|ruckarxcruz, http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=113:58
weshay|ruckarxcruz, https://github.com/DomiStyle/docker-idrac614:10
ysandeep|ruckweshay|ruck, do you have a free slot today, want to talk about a bug..14:12
weshay|ruckysandeep|ruck, sure.. let's chat now14:13
weshay|ruckysandeep|ruck, actually one sec14:13
weshay|ruckmatbu, hey14:13
weshay|ruckmatbu, want to 1-1? or you good?14:13
rlandyweshay|ruck: hmmm ... this is the same   "update_repo": standalone_container_prep_update_repo ~ add_repos_update | default('')14:14
rlandythere is some other issue14:14
rlandy        modify_vars:14:14
rlandy            tasks_from: yum_update.yml14:15
rlandy            update_repo: delorean-current14:15
rlandy            yum_repos_dir_path: /etc/yum.repos.d14:15
weshay|ruckysandeep|ruck, https://meet.google.com/snn-kgdw-vtq?authuser=114:15
rlandy^^ correct14:15
rlandyysandeep|ruck: ^^ those params moved but they are correct14:15
rlandyupdate_repo: delorean-current14:15
ysandeep|ruckrlandy, ack14:15
* rlandy looks for issue14:15
weshay|ruckysandeep|ruck, https://opendev.org/openstack/openstack-tempest-skiplist/src/branch/master/roles/validate-tempest/vars/tempest_skip.yml#L642-L65414:19
ysandeep|ruckweshay|ruck, https://review.opendev.org/#/c/758430/14:21
rlandyysandeep|ruck: weshay|ruck: the container update stuff looks right to me - it's the same - it just moved ...14:22
mariosarxcruz: https://review.opendev.org/#/c/759654/6/zuul.d/content-provider.yaml when you have time please14:22
rlandyhttps://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario002-standalone-master14:22
rlandyran again and passed14:22
weshay|ruckarxcruz, how much time is left in your day today?14:22
mariosweshay|ruck: which patch did you update with branches so we can point arxcruz for the example (i commented for him at https://review.opendev.org/#/c/759654/6/zuul.d/content-provider.yaml)14:22
rlandylooks like we had an issue accessing delorean-current14:22
weshay|ruckysandeep|ruck, has some questions but can wait til monday14:22
arxcruzweshay|ruck: around 2 and half hour14:22
rlandyweshay|ruck: ysandeep|ruck: I can look again if you see that failure repeated14:22
rlandybut the vars are correct14:23
rlandyyum.repos.d is just the repo path14:23
rlandynot the collection of update repos used14:23
mariosarxcruz: can you fix branches at https://review.opendev.org/#/c/759654/6/zuul.d/content-provider.yaml14:24
mariosthen i'll rebase https://review.opendev.org/#/c/759702/ & https://review.opendev.org/#/c/760323/ onto it14:25
mariosarxcruz: ^^14:25
arxcruzmarios: ack just a sec14:25
mariosarxcruz: thanks14:25
*** amoralej|lunch is now known as amoralej14:27
rlandy'https://review.rdoproject.org/zuul/builds?pipeline=%09openstack-periodic-integration-main14:27
rlandywe did fine there14:27
ysandeep|ruckarxcruz, hey o/ basically just want to understand why this test is triggering on master and not other branch - neutron_tempest_plugin.scenario.test_floatingip.FloatingIPQosTest14:30
ysandeep|ruckhttps://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/07ac197/logs/undercloud/var/log/tempest/stestr_results.html.gz14:30
ysandeep|ruckhttps://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-ussuri/9b5908c/logs/undercloud/var/log/tempest/stestr_results.html.gz14:30
ysandeep|ruckmaster and ussuri log ^^ for comparision14:30
ysandeep|ruckarxcruz, history.. this is regarding https://bugs.launchpad.net/tripleo/+bug/1900357 , earlier some tests were triggering on periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-queens and failing.. But these tests are not triggered anymore.. trying to understand what changed.14:33
openstackLaunchpad bug 1900357 in tripleo "fs020 queens is failing tempest tests - test_floatingip.FloatingIPQosTest and test_server_multi_create_auto_allocate" [Critical,Triaged] - Assigned to Soniya Murlidhar Vyas (svyas)14:33
arxcruzmarios: "^(master|stable/victoria).*$" ?14:34
arxcruzysandeep|ruck: checking just a sec14:34
ysandeep|ruckarxcruz, thanks! :) if you are occupied with other work i can pick it up with you on monday14:35
mariosarxcruz: no14:38
mariosarxcruz: like branches: ^(master|stable/victoria).*$ should do it yeah actually what you have is fine14:39
mariosarxcruz: i was thinking if we need stable/(victoria, foo) but we don't14:39
mariosonly victoria14:39
mariosthanks14:39
arxcruzmarios: done14:40
mariosthen i think weshay|ruck will remove his -2 when you update https://review.opendev.org/#/c/759654/14:40
mariosarxcruz: no14:40
mariosarxcruz: branches:14:40
weshay|ruckarxcruz, so.. I agree we should move stuff around14:40
weshay|ruckbut we have some changes into the original zuul defs atm14:41
mariosarxcruz: pleas fix?14:41
weshay|ruckto help limit some damage we're seeing w/ failing jobs14:41
mariosarxcruz: https://review.opendev.org/#/c/759654/7/zuul.d/content-provider.yaml14:41
mariosarxcruz: add branches:14:41
weshay|ruckarxcruz, rebase on top of my two patches14:41
mariosarxcruz: leave the override as it was14:41
mariosarxcruz: we want it to match for master and victoria changes only14:41
mariosarxcruz: is it clear now?14:41
weshay|ruckI mean.. wouldn't it just be easier to rebase.. on top.. and move14:42
weshay|ruck?14:42
arxcruzmarios: yes, sorry, check it now14:42
weshay|ruckarxcruz, https://review.opendev.org/#/c/760437/14:42
weshay|ruckhttps://review.opendev.org/#/c/760438/14:42
mariosarxcruz: ack thanks looks good now weshay|ruck please remove -2 https://review.opendev.org/#/c/759654/8/zuul.d/content-provider.yaml14:43
weshay|ruckk. /me looks14:43
weshay|ruckdo we want quotes?14:43
weshay|ruck branches: "^(master|stable/victoria).*$"14:43
mariosweshay|ruck: don't think it makes any difference14:43
weshay|ruckya.. let's keep it consistent..14:44
mariosweshay|ruck: but sure lets remove arxcruz please?14:44
marios:D14:44
mariosarxcruz: can you update one last time? https://review.opendev.org/#/c/759654/8/zuul.d/content-provider.yaml14:44
mariosvielen dank herr. cruz14:48
weshay|ruckarxcruz++14:49
arxcruzmarios: gerne14:49
weshay|ruckrlandy, need eyes?14:49
*** ysandeep|ruck is now known as ysandeep|brb14:49
weshay|ruckrfolco, 0/ ping if anything is needed14:50
rfolcoweshay|ruck, well, c7 provider is broken still14:50
weshay|ruckmarios, don't disappear on me14:51
mariosweshay|ruck: not yet ;)14:51
weshay|ruckmarios, https://e6a4aa5ab901b319f49c-fd65c414e871aa674eb3d52f029874e9.ssl.cf1.rackcdn.com/757829/3/check/tripleo-ci-centos-8-undercloud-upgrade-ussuri/41a7594/logs/undercloud/home/zuul/undercloud_upgrade.log14:52
mariosweshay|ruck: gimme couple mins rebasing onto arx patch14:52
mariosconflicts14:52
rlandyweshay|ruck: on what part?14:52
mariosweshay|ruck: hmm weird we already fixed that... or why are we seeing it now?14:53
weshay|ruckrlandy, we'll chat later .. re: container updates14:53
mariosweshay|ruck: will have better look in few14:53
weshay|ruckarxcruz, this is interesting https://662cba25091b41168cd8-c6d0ba205acfdaf07abcc75d255ce0b7.ssl.cf5.rackcdn.com/757544/10/check/tripleo-ci-centos-8-standalone-upgrade-ussuri/6c3b61f/job-output.txt14:53
rlandyweshay|ruck: I left a string of comments on irc14:53
rlandyit's fine14:53
weshay|ruckjust so you folks know...14:53
weshay|ruckUSSURI IS TOTALLY F'd14:53
rlandythere is no problem with container_updates14:53
weshay|ruckw/ content provider14:53
rlandythe vars are correct14:53
marios:(14:53
rlandywhat's the ussuri problem? need help there?14:54
weshay|ruckarxcruz, another one https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d21/757821/3/check/tripleo-ci-centos-8-standalone-upgrade-ussuri/d21ac17/job-output.txt14:54
weshay|ruckmarios, it's looking like we're going to mark standalone-upgrade nv in ussuri14:54
weshay|ruckunless we can get a handle on these14:55
mariosweshay|ruck: ack :/14:55
weshay|ruckmarios, plesae look.. 2nd hit V14:55
weshay|ruckhttps://6ee53134f7f98d7ec741-43939889d4a60991ed49de41534ef9b0.ssl.cf1.rackcdn.com/757836/6/check/tripleo-ci-centos-8-undercloud-upgrade-ussuri/1325d50/logs/undercloud/home/zuul/undercloud_upgrade.log14:55
arxcruzweshay|ruck: checking14:56
mariosweshay|ruck: question is 'why now' i mean we were running that as voting for days now14:56
mariosweshay|ruck: something was 'fixed' in the last few days that causes it14:56
marios:54 < weshay|ruck> marios, it's looking like we're going to mark standalone-upgrade nv in ussuri14:57
mariosweshay|ruck: undercloud ^^^ not standalone-upgrade14:57
weshay|ruckwell.. perhaps both14:57
mariosweshay|ruck: the standalone looks like it fails on tempest workspace14:57
marios16:54 < weshay|ruck> arxcruz, another one14:58
marioshttps://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d21/757821/3/check/tripleo-ci-centos-8-standalone-upgrade-ussuri/d21ac17/job-output.txt14:58
mariosweshay|ruck: that one i mean ^14:58
mariosweshay|ruck: so different issue14:58
rlandyrfolco: replied to your comment on https://review.opendev.org/#/c/757643/14:59
rlandydo you want to remove your -1?14:59
weshay|ruckrfolco, what are you seeing? or is rlandy taking care of it?15:00
arxcruzmarios: weshay|ruck I know the problem, I'll work on it15:00
arxcruzthis piece of code is a PITA15:00
arxcruzmarios: weshay|ruck but please, mark as nv for now15:01
mariosarxcruz: the workspace one you mean?15:01
arxcruzmarios: yes15:01
mariosarxcruz: but which job? does it run in all jobs?15:01
mariosweshay|ruck: arxcruz: lets not mark the stnadalone-upgrade non voting for something that happens on all jobs please15:01
weshay|ruckmarios, it's blocking ussuri content-provider15:01
arxcruzmarios: only on upgrade job is happening15:01
marios:(15:02
rfolcoweshay|ruck, 2 problems... 1) for some reason container builds in c7 provider is failing to find gating packages, and 2) in spite of use_buildah is true in zuul inventory, the build script does not use it15:02
weshay|ruckmarios, tell me how to fix the yum issue15:02
weshay|ruckshould be simple15:02
mariosweshay|ruck: i will dig there in a sec ... must be something very recent anyway15:02
mariosweshay|ruck: we fixed that ages ago and it was running as voting for weeks so...15:02
mariosweshay|ruck: something fixed it back ;)15:02
marioswell, days anyway15:03
weshay|ruckheh15:03
mariosactually undercloud one was weeks15:03
weshay|ruckrfolco, so where are you stuck?15:03
rlandyweshay|ruck: this was rfolco's comment -https://review.opendev.org/#/c/757643/20/zuul.d/multinode-jobs.yaml  I think it's fine .. I want to merge15:03
rfolcoweshay|ruck, trying to find c7 provider error while building containers15:03
rlandyrfolco: there is a patch to sort the c7 provider error15:04
rfolcorlandy, if makes no sense train there, merge it15:04
weshay|ruckrlandy, I wf15:04
rfolcorlandy, which one ?15:04
rlandycool15:04
rlandyrfolco: https://review.opendev.org/#/c/759985/20/zuul.d/undercloud-jobs.yaml15:04
weshay|ruckrfolco,  log?15:05
weshay|ruckor review please15:05
rlandynote that c7 will only run on queens stein15:05
rlandynot master15:05
rfolcoc7 provider on train is failing15:05
rfolcoeven with your patch for use_buildah15:06
rlandyrfolco: it was looking for a  master c7 release file15:06
rfolcoweshay|ruck, example https://review.opendev.org/#/c/759737/15:06
rlandywhich does not exits15:06
rlandycan you post the logs15:06
rlandywe can check if it's the same error15:06
arxcruzmarios: this is on os_tempest side, i'll work on this, for now, if it's blocking, mark as nv, when the job reachs this, it's because it was fully upgraded, and is trying to run tempest again15:06
arxcruzweshay|ruck: *15:07
rfolcorlandy, the error I see is file:///opt/gating_repo/repodata/repomd.xml: [Errno 14] curl#37 - "Couldn't open file /opt/gating_repo/repodata/repomd.xml"15:07
arxcruzweshay|ruck: ^15:07
rfolcohttps://48076276ef55ded18bdd-ee6f79bcfc63e36433e8a3999f9ecff9.ssl.cf5.rackcdn.com/759737/6/check/tripleo-ci-centos-7-content-provider/3f0f6cb/logs/undercloud/home/zuul/workspace/build.log15:07
rfolcoweshay|ruck, rlandy ^15:07
rfolcotalking about c7 provider in train, not master15:07
* rlandy looks15:08
rlandytripleo-ci-centos-7-content-provider15:10
rlandysorry - bad paste15:10
weshay|ruckrfolco, same shit as yesterday15:10
weshay|ruckrfolco, it's not using buildah15:10
rfolcolook at inventory15:10
rfolcohttps://48076276ef55ded18bdd-ee6f79bcfc63e36433e8a3999f9ecff9.ssl.cf5.rackcdn.com/759737/6/check/tripleo-ci-centos-7-content-provider/3f0f6cb/zuul-info/inventory.yaml15:11
rfolco    use_buildah: true15:11
rfolcoand the error above makes me think there is something wrong with the gating repo itself15:12
rlandyk - I see ... Couldn't open file /opt/gating_repo/repodata/repomd.xml"15:12
rlandyon sec - still looking15:12
rfolcocompare to my testproject15:12
rlandyhold on15:12
rfolcohttps://logserver.rdoproject.org/89/30689/2/check/tripleo-ci-centos-7-content-provider/bf61bf2/logs/undercloud/home/zuul/workspace/build.log.txt.gz15:12
rfolcoin this testproject I changed defaults to use_buildah15:13
weshay|ruckrfolco, ya.. I see it in the inventory15:13
rfolcosuper weird, tried to find if use_buildah has changed in the job.. can't spot any15:14
weshay|ruckrfolco, but compare https://48076276ef55ded18bdd-ee6f79bcfc63e36433e8a3999f9ecff9.ssl.cf5.rackcdn.com/759737/6/check/tripleo-ci-centos-7-content-provider/3f0f6cb/logs/undercloud/home/zuul/workspace/build_containers.sh w/ https://logserver.rdoproject.org/89/30689/2/check/tripleo-ci-centos-7-content-provider/bf61bf2/logs/undercloud/home/zuul/workspace/build_containers.sh.txt.gz15:14
rlandy- name: Check if /opt/gating_repo exists15:14
rlandy  stat:15:14
rlandy    path: /opt/gating_repo15:14
rlandy  register: gating_stat15:14
rlandyit should check if the repo exists or nor15:14
weshay|ruckrfolco, buildah is not passed15:14
rlandynot15:14
rfolcoweshay|ruck, not passed I saw...15:15
rfolcobut why?15:15
*** derekh has joined #oooq15:16
rlandydepends what consumes the var15:17
rlandyif the var is consumed by toci15:17
rlandywe need to define it elsewhere15:17
rlandyrfolco: ^^15:18
weshay|ruckrfolco, it's not in https://48076276ef55ded18bdd-ee6f79bcfc63e36433e8a3999f9ecff9.ssl.cf5.rackcdn.com/759737/6/check/tripleo-ci-centos-7-content-provider/3f0f6cb/logs/zuul-variables.yaml15:18
rfolcowhy did you say its wrong setting use_buildah as true in defaults ?15:18
weshay|ruckrfolco, because it is wrong15:19
rfolcoweshay|ruck, maybe beacuse you confused zuul with 2 job definitions15:19
rfolcolol15:19
rfolcojust saying15:19
weshay|ruckrfolco, read thishttps://48076276ef55ded18bdd-ee6f79bcfc63e36433e8a3999f9ecff9.ssl.cf5.rackcdn.com/759737/6/check/tripleo-ci-centos-7-content-provider/3f0f6cb/logs/quickstart_files/playbook_executions.log15:19
weshay|ruckhttps://48076276ef55ded18bdd-ee6f79bcfc63e36433e8a3999f9ecff9.ssl.cf5.rackcdn.com/759737/6/check/tripleo-ci-centos-7-content-provider/3f0f6cb/logs/quickstart_files/playbook_executions.log15:19
rfolcoI still did not swallow yesterday fall15:19
weshay|ruckrfolco, then you should understand15:19
rlandyweshay|ruck: rfolco: we could set it in the train c7 release file15:20
weshay|ruckrfolco, the variable use_buildah IS NOT PASSED15:20
weshay|ruckrlandy, we could do that15:20
rlandyzuul vars are not passed if the role is called by toci15:20
rlandyso either we pass it in the release file15:20
rlandyor15:20
weshay|ruckoh wait.. see it here https://48076276ef55ded18bdd-ee6f79bcfc63e36433e8a3999f9ecff9.ssl.cf5.rackcdn.com/759737/6/check/tripleo-ci-centos-7-content-provider/3f0f6cb/logs/hostvars-variables.yaml15:20
rlandywe add job.use_buildah15:20
*** ysandeep|brb is now known as ysandeep15:21
rlandyand set use_buildah to job.use_buildah15:21
weshay|ruckya.. that would be job.use_buildah15:21
rlandyrfolco: weshay|ruck: ^^ those are twp options15:21
weshay|ruckI agree w/ rlandy15:21
ysandeeparxcruz, hey o/ lets pick it up of monday, its getting late for me.. i will ping you monday morning.. thank you and happy weekend15:21
rlandyweshay|ruck: imho the release file is easier15:21
ysandeeps/of/on*15:21
arxcruzysandeep: yes please, i'm checking the upgrade stuff failing15:21
weshay|ruckso rfolco15:21
weshay|ruckhttps://opendev.org/openstack/tripleo-ci/src/branch/master/roles/build-containers/defaults/main.yaml#L1315:21
arxcruzupgrade job*15:21
rlandyunless you require it to be job based15:21
weshay|ruckset that to read from job.use_buildah if job.use_buildah is defined15:22
*** ysandeep is now known as ysandeep|away15:22
ysandeep|awayhappy weekend guys o/15:22
rfolcoweshay|ruck, k, got it. Patch coming rlandy weshay|ruck thanks15:22
rlandyrfolco: weshay|ruck: I'd be careful with that change15:23
rlandychanging the role defaults15:24
rlandyhas implications for current usage15:24
rlandyeither add an or case15:24
rlandyor set in release file15:24
rlandyor set_fact15:24
rfolcorlandy, weshay|ruck let me know if this is not safe https://review.opendev.org/#/c/760580/115:31
rfolcobrb lunch15:31
*** ksambor has quit IRC15:39
weshay|ruckrlandy, https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/build-containers/templates/build.sh.j2#L3415:41
rlandyrfolco: weshay|ruck and I updated https://review.opendev.org/#/c/760580/15:43
rlandyrest looks good15:43
weshay|ruckrfolco, minor change..15:43
weshay|ruckmarios, arxcruz should we chat about ussuri?15:44
rfolcoweshay|ruck, what else15:44
weshay|ruckrfolco, take a look, you can't assume job is defined either.. so just added a little snip15:45
mariosweshay|ruck: sure15:45
weshay|ruckhttps://meet.google.com/net-anbz-ntv?pli=1&authuser=015:45
mariosweshay|ruck: i see the issue, but a bit confused cos we also have green runs today see https://review.opendev.org/#/c/760519/15:45
mariosweshay|ruck: joining15:47
arxcruzweshay|ruck: marios https://review.opendev.org/#/c/753393/15:49
arxcruzrlandy: ^15:49
*** ykarel is now known as ykarel|away15:58
mariosarxcruz: https://opendev.org/openstack/tripleo-quickstart-extras/src/commit/a4a5e888ea873342891a30de2540a4f3dc70521c/playbooks/multinode-standalone-upgrade.yml16:04
weshay|ruckarxcruz, https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/playbooks/multinode-standalone-upgrade.yml#L64-L7616:08
marioshttps://review.opendev.org/#/c/760519/16:11
*** ykarel|away has quit IRC16:22
arxcruzweshay|ruck: marios rlandy https://review.opendev.org/#/c/760626/2/playbooks/multinode-standalone-upgrade.yml16:23
weshay|ruckrlandy, https://hackmd.io/MlbZ_izSTEuZsCWTJvu_Kg16:27
rfolcoweshay|ruck, rlandy marios can we merge https://review.opendev.org/#/c/758627/16:29
rfolcoand https://review.opendev.org/#/c/759429/16:29
rfolcoweshay|ruck, rlandy marios ans also neutron ussuri/victoria are green --> https://review.opendev.org/#/c/760212/ | https://review.opendev.org/#/c/760216/16:34
weshay|ruckrfolco, email slavek16:35
weshay|rucker.. /me gets the right nick/email16:35
rfolcoweshay|ruck, did recheck on master16:35
rfolcoits all good, except master failing on non-related jobs16:36
rfolcoand train which depends on our new patch for use_buildah16:36
rfolcoweshay|ruck, no need to email16:36
weshay|ruckrfolco, Slawomir KaplonskiEngineerNetwork vNESslaweq16:36
rlandyk .. lunch brb16:37
*** amoralej is now known as amoralej|off16:45
rlandyweshay|ruck: I'm w+'ing https://review.opendev.org/#/c/758627/17:01
rlandyrfolco: is https://review.opendev.org/#/c/760580/working for you now?17:03
rfolcorlandy, watching close my testproject and my c7 train patch in zuul upstream17:04
rfolcorlandy, will know soon17:04
rfolcorlandy, my testproject console running just fyi https://review.rdoproject.org/zuul/stream/c3c4b6878bb347608a4edb4d10d8e89a?logfile=console.log17:05
rfolcorlandy, I am going to update all c7/train patches upstream as soon as I get a green17:06
rfolconot before, so I don't add more noise to upstream queues17:06
rlandyok17:08
weshay|ruckrlandy, hrm.. where did we fix the centos-7 contentent provider to only run w/ train and queens.. today.. can't find that review17:15
*** marios is now known as marios|out17:16
rlandylooking17:17
rfolcohttps://review.opendev.org/#/c/760428/217:18
rfolcoweshay|ruck, rlandy ^17:18
rfolcogerrit is slow for me17:18
rlandyhttps://review.opendev.org/#/c/759985/21/zuul.d/undercloud-jobs.yaml17:19
weshay|ruckthanks17:21
weshay|ruckrlandy, rfolco please review this list of prioritized patches.. ping me if we need to add anyting https://hackmd.io/MlbZ_izSTEuZsCWTJvu_Kg?view17:34
rlandyk17:35
rfolcoweshay|ruck, rlandy c7 provider worked with that patch, now updating all c7 train patches17:43
weshay|ruckrfolco++17:43
*** marios|out has quit IRC17:44
rlandygood17:46
*** jpena is now known as jpena|off17:57
*** derekh has quit IRC18:05
*** jmasud has quit IRC18:11
*** dtantsur is now known as dtantsur|afk18:15
rlandyrfolco: weshay|ruck: is this patch still needed? duplicated elsewhere?18:32
weshay|ruckwhich?18:32
rfolcowhich18:32
rlandyhttps://review.opendev.org/#/c/760428/18:35
rlandysorry18:35
rlandythought I pasted18:35
weshay|ruckrlandy, I think that's needed18:37
weshay|ruckrlandy, probably should depend on folco's change to update the buildah settings though18:38
rlandyweshay|ruck: ack - those patches should probably go on the merge list then18:38
weshay|ruckadded18:39
weshay|ruckyour patch was.. folco's was not.. please update depends-18:39
rlandyweshay|ruck: do we need to look at the 16.2 stuff more closely or is that sorted now?18:39
weshay|ruckrlandy, which part of 16.2? the stuff they are talking about in channel?18:40
weshay|ruckwe can ignore it18:40
rlandyweshay|ruck: great - all about ignoring people these days18:40
weshay|ruckrlandy, revote.. added depends-on https://review.opendev.org/#/c/759912/18:45
rfolconooo18:46
rfolcodon't do that foks18:46
weshay|rucksorry?18:46
rlandyweshay|ruck: rebased https://review.opendev.org/#/c/760221/, https://review.opendev.org/#/c/760244/18:46
rlandy^^ need votes there18:46
rfolcosagi->rlandy->folco18:47
weshay|ruckrfolco, it was failing in the gate18:47
weshay|ruckyou would have been blocked.. anyway..18:47
rfolcohttps://review.opendev.org/#/c/760580/ parents off https://review.opendev.org/#/c/760428/18:47
rfolconow I reset all my c7 patches18:47
weshay|ruckrfolco, they would have reset anyway18:47
weshay|ruckit's important to merge the these patches in the right order at this point18:48
weshay|ruckor else ur just rechecking to warm the earth18:48
rfolcook I'll have to rebase https://review.opendev.org/#/c/760580/18:50
rfolcobecause now its parent https://review.opendev.org/#/c/760428 has a depends-on to it18:51
rfolcoyou could just depends-on 760580, it was in the right order... and all in tripleo-ci (same repo)18:53
*** frenzy_friday has quit IRC18:58
*** frenzy_friday has joined #oooq18:59
*** frenzy_friday has quit IRC19:01
rfolcoweshay|ruck, before: 760333 -> 760428 -> 760580   |   now: 760333 -> 760428 (depends-on: 760580)19:05
* weshay|ruck looks19:06
weshay|ruckrfolco, looks good to me..19:07
rfolcok19:07
*** TrevorV has quit IRC20:10
rfolcodammn c7 provider failed weshay|ruck20:20
rfolcohttps://c6f66df5178af5c9e794-0fd7aeaffe35682d1011363ed54bba64.ssl.cf2.rackcdn.com/759737/8/check/tripleo-ci-centos-7-content-provider/beec3b0/logs/undercloud/home/zuul/workspace/build.log20:20
weshay|rucksec20:22
*** yolanda has quit IRC20:24
*** yolanda has joined #oooq20:25
rlandygrep: /tmp/container-builds/6df51c3a-7e13-463c-af9e-9ea8b7624708/docker/base/base-build.log: Permission denied20:36
rlandyrfolco: weshay|ruck: ^^ become?20:36
rfolcoredherring20:37
rfolcothe real error is in build itself20:37
rfolco2020-10-30 20:08:34 | Stderr: u'error building at STEP "RUN yum -y install centos-release-nfs-ganesha28 centos-release-openstack-train centos-release-opstools centos-release-ceph-nautilus centos-release-qemu-ev yum-plugin-priorities centos-release-opstools && rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Cloud && rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-OpsTools && rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Storage && rpm20:37
rfolco--import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Virtualization && yum -y update --security --sec-severity=Important --sec-severity=Critical     && yum clean all && rm -rf /var/cache/yum": error while running runtime: exit status 1\n'20:37
*** slaweq has quit IRC20:38
rlandyyou can't log the results20:39
rlandyso it's failing20:40
rfolcothe build failure has nothing to do with log, its in a different level of execution20:41
rfolcopermission denied will tell us what failed w/ build logs, but its not the rrot cause20:42
rfolcoroot*20:42
*** jfrancoa has quit IRC20:42
rfolcoam I crazy or none of c7 branchful provider -train are using buildah ? https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-content-provider-train20:52
weshay|rucklooking20:57
weshay|ruckrfolco, ur a little crazy.. afaict the only c7 train provider that has reported since the change is https://c6f66df5178af5c9e794-0fd7aeaffe35682d1011363ed54bba64.ssl.cf2.rackcdn.com/759737/8/check/tripleo-ci-centos-7-content-provider/beec3b0/logs/undercloud/home/zuul/workspace/index.html20:59
rfolcoweshay|ruck, thats what I am trying to say... branchful -train jobs were passing even without using buildah21:01
rfolcobefore the change21:01
weshay|ruckya.. go look at c7 train check jobs21:02
weshay|ruckfor container builds21:02
rfolcohttps://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-content-provider-train21:03
rfolcothese are all branchful21:03
rfolcoand are green21:03
rfolcoand don't use buildah21:03
weshay|ruckcan you do what I asked21:04
rfolcoweshay|ruck, I do not follow you. I did look at branchful train jobs/21:06
rfolcodidn't I?21:06
weshay|ruckno21:06
rfolcocan you explain me21:07
weshay|ruckrfolco, yes.. a branchful job runs because the zuul_branch != the branch designated in the branchful job21:08
weshay|rucka check job runs w/ the zuul_branch21:08
weshay|ruckI need to plug back into my hardline..21:09
rfolcoI know what a branchful job is :)21:10
weshay|ruckok.. I asked you to look at a check job.. not a branchful job21:10
rfolcobut I am speaking about branchful, that has this https://github.com/openstack/tripleo-ci/blob/master/zuul.d/undercloud-jobs.yaml#L23021:11
rfolcoand works well even though it does NOT use buildah21:11
weshay|ruckrfolco, well yes.. I know21:12
weshay|ruckthe branchful check jobs should follow the pattern of the check jobs for that branch21:12
weshay|ruckyou are implying that use_buildah does NOT work21:12
weshay|ruckso I asked you to refer the check jobs21:12
rfolco2 separate things, ok21:12
rfolcoone is the check job not working even using buildah21:13
rfolcothe other thing I was referring was the branchful green jobs w/ use_buildah not actually using21:13
weshay|ruckrfolco, prove it21:14
weshay|ruckrfolco, prove to me the c7 train containers build check job .. not branchful is not working21:15
weshay|ruckshow me21:15
rfolcohttps://zuul.openstack.org/status21:15
rfolcoin flight21:15
rfolco75973721:15
rfolcotripleo-ci-centos-7-content-provider  failure21:15
rfolcohttps://c6f66df5178af5c9e794-0fd7aeaffe35682d1011363ed54bba64.ssl.cf2.rackcdn.com/759737/8/check/tripleo-ci-centos-7-content-provider/beec3b0/21:15
weshay|ruckthat's a content provider21:16
weshay|ruckthat's not the containers-build check job21:16
rfolcoyes content provider21:17
rfolco2020-10-30 20:08:34 | Stderr: u'error building at STEP "RUN yum -y install centos-release-nfs-ganesha28 centos-release-openstack-train centos-release-opstools centos-release-ceph-nautilus centos-release-qemu-ev yum-plugin-priorities centos-release-opstools && rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Cloud && rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-OpsTools && rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Storage && rpm21:17
rfolco--import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Virtualization && yum -y update --security --sec-severity=Important --sec-severity=Critical     && yum clean all && rm -rf /var/cache/yum": error while running runtime: exit status 1\n'21:17
weshay|ruckrfolco, a content-provider is NOT the containers-build check job21:18
rfolcodid I say that?21:18
weshay|ruckso show me one21:18
weshay|ruckrfolco, can you do what I asked please21:19
rfolcowes whats going on with you?21:19
rfolco:)21:19
weshay|ruckI'm asking for a simple thing21:19
rlandy [FAILED] python3-metalsmith-0.15.1-0.20200720234727.0afed74.el8ost.noarch.rpm: No more mirrors to try - All mirrors were already tried without success21:19
weshay|ruckand you are all over the map21:19
rlandy^^ killing rhos-release21:19
weshay|ruckcan you please find me a c7 train container-build check job.. not a content-provider21:19
rlandyone patch merged - one left21:19
weshay|rucknot a branchful21:19
rfolcoweshay|ruck, show me one what ?21:20
weshay|ruck[15:19:52] <weshay|ruck> can you please find me a c7 train container-build check job.. not a content-provider21:20
rfolcoweshay|ruck, I said: c7 provider is failing... who mentioned container build ?21:20
rfolconevermind, I'll investigate whats going on21:21
rfolcothx21:21
weshay|ruckthat's failing in the container build right?21:21
weshay|ruckrfolco, no no21:21
weshay|ruckur not off the hook21:21
weshay|ruckyet21:21
rfolcoweshay|ruck, yes failing in the container build21:21
rfolcoand?21:21
weshay|ruckhttps://meet.google.com/landing?authuser=121:22
rfolcosec21:22
rfolcoweshay|ruck, which mtg21:23
rfolcoweshay|ruck, open one please21:24
weshay|ruckhttps://meet.google.com/dym-tcep-hxs?authuser=1 again21:24
weshay|ruckah.. pasted incorrectly21:24
rfolcoyes21:24
*** rlandy has quit IRC21:35
weshay|ruckhttps://0589223ea5b3f61fa18e-fe304f23412f432b70850b4eb068f885.ssl.cf5.rackcdn.com/756514/3/gate/tripleo-build-containers-centos-7-buildah/d6c4998/job-output.txt21:38
weshay|ruckhttps://c6f66df5178af5c9e794-0fd7aeaffe35682d1011363ed54bba64.ssl.cf2.rackcdn.com/759737/8/check/tripleo-ci-centos-7-content-provider/beec3b0/job-output.txt21:38
weshay|ruckrfolco, https://opendev.org/openstack/tripleo-ci/src/branch/master/playbooks/tripleo-buildcontainers/post.yaml21:50
weshay|ruckrfolco, https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/build-containers/tasks/post.yaml#L20-L2121:52
weshay|ruckrfolco, https://review.opendev.org/76067022:05
rfolcoweshay|ruck, k will send you in a few22:06
weshay|ruckrfolco, I think this is only a c7 problem22:08
weshay|ruckc8 has https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6b2/759429/6/check/tripleo-ci-centos-8-content-provider/6b238ae/logs/undercloud/home/zuul/container-builds/2284aeb1-99f2-4eca-b476-4c1c986307da/base/index.html22:08
weshay|ruckwhich is why we're finding this now22:08
rfolcoweshay|ruck, hmm not sure on top of which change I rebase22:12
weshay|ruckrfolco, it's not clear to how to use gerrit to find the open reviews that touch $file22:13
weshay|ruckrfolco, I know of this one https://review.opendev.org/#/c/760438/4/zuul.d/undercloud-jobs.yaml22:14
rfolcofor c7 changes, not looking at the ones that modify the file....22:15
rfolcowe have 3 in tripleo-ci22:15
rfolcoso I need to make a logical chain there22:15
rfolcoweshay|ruck, testing here https://review.opendev.org/#/c/759737/22:20
rfolcoand here https://review.rdoproject.org/r/#/c/30700/22:21
rfolcoweird that c7 provider worked before in testproject....22:21
rfolcook bbl22:21
*** rfolco is now known as rfolco|bbl22:21
weshay|ruckrfolco|bbl, k.. thanks22:23
weshay|ruckCRAP https://review.opendev.org/#/c/760438/422:29
*** jmasud has joined #oooq23:08
*** yolanda has quit IRC23:33
*** saneax has joined #oooq23:51
« Thursday, 2020-10-29 Index Saturday, 2020-10-31 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!