| *** slaweq has joined #oooq | 00:11 | |
| *** slaweq has quit IRC | 00:15 | |
| *** rfolco_l8r is now known as rfolco | 00:49 | |
| *** slaweq has joined #oooq | 01:11 | |
| *** slaweq has quit IRC | 01:16 | |
| *** apetrich has quit IRC | 01:58 | |
| *** dsneddon has quit IRC | 02:57 | |
| *** dsneddon has joined #oooq | 03:02 | |
| *** dsneddon has quit IRC | 03:08 | |
| *** slaweq has joined #oooq | 03:11 | |
| *** ykarel has joined #oooq | 03:13 | |
| *** slaweq has quit IRC | 03:16 | |
| *** jbadiapa has quit IRC | 03:21 | |
| *** dsneddon has joined #oooq | 03:21 | |
| *** jbadiapa has joined #oooq | 03:21 | |
| *** dsneddon has quit IRC | 03:26 | |
| *** gkadam has joined #oooq | 03:40 | |
| *** gkadam has quit IRC | 03:45 | |
| *** skramaja has joined #oooq | 03:55 | |
| *** dsneddon has joined #oooq | 03:57 | |
| *** udesale has joined #oooq | 04:03 | |
| *** slaweq has joined #oooq | 04:11 | |
| *** ykarel has quit IRC | 04:13 | |
| *** dsneddon has quit IRC | 04:15 | |
| *** slaweq has quit IRC | 04:16 | |
| *** ykarel has joined #oooq | 04:36 | |
| *** dsneddon has joined #oooq | 04:39 | |
| *** saneax has joined #oooq | 04:46 | |
| *** holser_ has joined #oooq | 04:48 | |
| *** kopecmartin|off is now known as kopecmartin | 04:54 | |
| *** marios has joined #oooq | 05:09 | |
| *** slaweq has joined #oooq | 05:11 | |
| *** slaweq has quit IRC | 05:17 | |
| *** skramaja_ has joined #oooq | 05:18 | |
| *** skramaja has quit IRC | 05:19 | |
| *** ratailor has joined #oooq | 05:33 | |
| *** udesale has quit IRC | 05:35 | |
| *** udesale has joined #oooq | 05:35 | |
| *** ratailor_ has joined #oooq | 05:36 | |
| *** udesale has quit IRC | 05:36 | |
| *** udesale has joined #oooq | 05:36 | |
| *** ykarel_ has joined #oooq | 05:36 | |
| *** ratailor has quit IRC | 05:38 | |
| *** ykarel has quit IRC | 05:38 | |
| *** holser_ has quit IRC | 06:00 | |
| *** jfrancoa has joined #oooq | 06:03 | |
| *** slaweq has joined #oooq | 06:11 | |
| *** ksambor|away is now known as ksambor | 06:11 | |
| *** slaweq has quit IRC | 06:16 | |
| *** raukadah is now known as chandankumar | 06:19 | |
| chandankumar | zbr: Good morning | 06:21 |
|---|---|---|
| chandankumar | zbr: let me know when around, I need to some help to fix http://logs.openstack.org/57/669057/34/check/openstack-tox-molecule/a1a4369/tox/reports.html molecule failure | 06:21 |
| *** udesale has quit IRC | 06:22 | |
| *** udesale has joined #oooq | 06:22 | |
| *** udesale has quit IRC | 06:24 | |
| *** udesale has joined #oooq | 06:25 | |
| chandankumar | sshnaidm|rover: morning | 06:25 |
| chandankumar | sshnaidm|rover: regarding novajoin test https://logs.rdoproject.org/24/639324/41/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039/232fbce/logs/undercloud/home/zuul/.config/openstack/clouds.yaml.txt.gz | 06:25 |
| chandankumar | sshnaidm|rover: auth_url: https://overcloud.ooo.test:13000 is getting overriden that's why it is not working | 06:25 |
| *** slaweq has joined #oooq | 06:28 | |
| *** slaweq has quit IRC | 06:33 | |
| sshnaidm|rover | chandankumar, ooo.test is domain managed by IPA server | 06:35 |
| sshnaidm|rover | chandankumar, dns server is https://logs.rdoproject.org/24/639324/41/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039/232fbce/logs/undercloud/etc/resolv.conf.txt.gz | 06:36 |
| sshnaidm|rover | chandankumar, and in supplemental node as dns: https://logs.rdoproject.org/24/639324/41/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039/232fbce/logs/supplemental/etc/resolv.conf.txt.gz | 06:37 |
| sshnaidm|rover | chandankumar, so if it's not resolved - need to ask security dfg folks to debug it | 06:37 |
| *** apetrich has joined #oooq | 06:40 | |
| *** jtomasek has joined #oooq | 06:48 | |
| *** amoralej|off is now known as amoralej | 06:57 | |
| *** slaweq has joined #oooq | 07:02 | |
| *** matbu has quit IRC | 07:16 | |
| *** sshnaidm|rover is now known as sshnaidm|afk | 07:18 | |
| *** matbu has joined #oooq | 07:19 | |
| *** bogdando has joined #oooq | 07:19 | |
| chandankumar | marios: does release http://logs.openstack.org/87/669487/42/check/tripleo-build-containers-centos-7/9ace6e6/job-output.txt.gz#_2019-07-17_04_49_46_931827 comes from release file? | 07:20 |
| marios | chandankumar: i think its set on the jobs definition | 07:20 |
| marios | chandankumar: https://codesearch.rdoproject.org/?q=release: | 07:21 |
| chandankumar | marios: the job is upstream one | 07:21 |
| marios | chandankumar: e.g. there https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/build-containers.yaml | 07:22 |
| marios | chandankumar: yeah but we do refer to release in upstream, like checking if it is set and doing stuff, but afaik we don't _set_ it upstream | 07:22 |
| marios | chandankumar: more examples e.g. https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/multinode-jobs.yaml | 07:23 |
| chandankumar | ok checking | 07:23 |
| marios | chandankumar: ah so we do set it there in the config files | 07:23 |
| marios | chandankumar: i mean https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/release/centosci/master-current-tripleo.yml | 07:23 |
| marios | chandankumar: e.g. ^ | 07:23 |
| marios | https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/general_config/featureset045.yml | 07:24 |
| marios | chandankumar: also some featureset apparently like ^ | 07:24 |
| marios | chandankumar: so maybe you should put it into the release file in this case | 07:25 |
| *** tosky has joined #oooq | 07:26 | |
| marios | chandankumar: looks like almost all of those have release there but i'd only noticed it from the d/stream job definitions before | 07:27 |
| chandankumar | marios: I will address your comments on RHEL-8 later once release file depends on works or not | 07:27 |
| marios | chandankumar: you mean there https://review.opendev.org/#/c/669487/ ? it was mostly for that become: true why was it necessary for the container build | 07:29 |
| chandankumar | marios: I will comment there | 07:30 |
| *** dsneddon has quit IRC | 07:30 | |
| *** dsneddon has joined #oooq | 07:40 | |
| *** ykarel_ is now known as ykarel|meeting | 07:42 | |
| *** ratailor__ has joined #oooq | 07:52 | |
| *** ratailor_ has quit IRC | 07:55 | |
| *** ykarel has joined #oooq | 07:55 | |
| *** udesale has quit IRC | 07:56 | |
| *** ykarel|meeting has quit IRC | 07:56 | |
| *** ykarel is now known as ykarel|lunch | 07:57 | |
| *** yolanda has quit IRC | 08:12 | |
| *** yolanda has joined #oooq | 08:12 | |
| *** yolanda has quit IRC | 08:12 | |
| chandankumar | marios: zbr panda invite sent for task https://tree.taiga.io/project/tripleo-ci-board/task/1154 sync | 08:14 |
| marios | chandankumar: thanks | 08:14 |
| chandankumar | feel free to update the timing based on your suitability | 08:14 |
| marios | wfm | 08:14 |
| marios | sshnaidm|afk: arxcruz|ruck is that scen3 standalone a known thing http://logs.openstack.org/29/669329/4/check/tripleo-ci-centos-7-scenario003-standalone/3e37c09/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz its blocking me here https://review.opendev.org/#/c/669329/ ['/usr/bin/podman', 'start', '-a', u'container-puppet-designate'] run failed after + mkdir -p /etc/puppet", | 08:20 |
| *** holser_ has joined #oooq | 08:20 | |
| marios | 2019-07-17 05:11:13 | "puppet-user: Error: Execution of '/usr/sbin/named-checkconf /etc/named.conf20190717-16-1398a2i' returned 1: | 08:20 |
| *** dtantsur|afk is now known as dtantsur | 08:23 | |
| arxcruz|ruck | marios, yeah, it was a change in puppet-dns | 08:24 |
| arxcruz|ruck | marios, https://github.com/theforeman/puppet-dns/commit/526d4db3e3f3106f95dffdd47e8b34a8a6278eca | 08:24 |
| marios | arxcruz|ruck: thanks is there something in progress that i might depends-on | 08:25 |
| marios | arxcruz|ruck: (do we have bug i can look there thanks for checking) | 08:25 |
| arxcruz|ruck | marios, https://bugs.launchpad.net/tripleo/+bug/1836812 | 08:26 |
| openstack | Launchpad bug 1836812 in tripleo " /etc/named/options.conf:1: change directory to '/var/named-persistent' failed: file not found"" [Critical,Fix released] - Assigned to Alex Schultz (alex-schultz) | 08:26 |
| marios | ty arxcruz|ruck | 08:26 |
| marios | arxcruz|ruck: looks like a fix merged 1 hour ago | 08:27 |
| * marios rechecks away | 08:27 | |
| *** ykarel|lunch is now known as ykarel | 08:39 | |
| marios | easy review if someone has few 2 mins thanks https://review.opendev.org/#/c/663018/ | 08:42 |
| zbr | chandankumar: i am back | 08:42 |
| *** udesale has joined #oooq | 08:46 | |
| chandankumar | panda: I have updated all the patches | 08:47 |
| arxcruz|ruck | panda, sshnaidm|afk hey, do you guys know how to manually promote a hash? wes is asking to promote stein before the meeting | 08:48 |
| chandankumar | zbr: please have a look at the release file molecule failure | 08:56 |
| chandankumar | is there a way to by pass that | 08:56 |
| *** derekh has joined #oooq | 08:58 | |
| chandankumar | zbr: panda meeting time | 09:00 |
| arxcruz|ruck | chandankumar, interview meeting ? | 09:01 |
| chandankumar | arxcruz|ruck: nope | 09:01 |
| arxcruz|ruck | ok | 09:02 |
| arxcruz|ruck | panda, let me know when you have time | 09:02 |
| chandankumar | arxcruz|ruck: it is another meeting | 09:02 |
| chandankumar | zbr: panda Changed reviews | 09:03 |
| chandankumar | zbr: panda https://redhat.bluejeans.com/u/chkumar/ | 09:03 |
| marios | https://bluejeans.com/1571313919 ^^^ | 09:03 |
| *** bogdando has quit IRC | 09:03 | |
| *** bogdando has joined #oooq | 09:10 | |
| zbr | too late? | 09:23 |
| arxcruz|ruck | panda, sshnaidm|afk | 09:25 |
| *** holser_ has quit IRC | 09:31 | |
| chandankumar | zbr: Hello | 09:31 |
| chandankumar | zbr: please have a look at this review https://review.opendev.org/#/c/669057/ -> http://logs.openstack.org/57/669057/34/check/openstack-tox-molecule/a1a4369/tox/reports.html | 09:31 |
| *** holser_ has joined #oooq | 09:33 | |
| zbr | chandankumar: yep. fix it | 09:35 |
| zbr | wes added an non routed ip there, so is normal to fail | 09:35 |
| zbr | chandankumar: so if you want to make it pass, you need to put an URL that passes the testing | 09:38 |
| zbr | this needs to be publicly accesible, for both zuul and users. | 09:38 |
| *** holser_ has quit IRC | 09:40 | |
| *** holser_ has joined #oooq | 09:40 | |
| zbr | chandankumar: i am afraid I cannot help with that as it is above my paygrade ;) | 09:44 |
| marios | zbr: see comment athttps://review.opendev.org/#/c/669057/34 | 09:51 |
| marios | zbr: i don't think we have a choice here as that rcm guest is what was provided for us and it is only accessible from rdo nodes | 09:52 |
| marios | zbr: so the only thing i can think of is to run your test on rdo too | 09:52 |
| *** bogdando has quit IRC | 09:52 | |
| zbr | yeah. i know that is why i am pissed. | 09:52 |
| marios | 12:36 < zbr> chandankumar: yep. fix it | 09:52 |
| marios | zbr: we can't ^ | 09:52 |
| zbr | that URL is not accesible even for me when I am connected to redhat VPN! | 09:52 |
| marios | zbr: right nothing to do with rh/vpn | 09:53 |
| marios | zbr: its rdo internal network | 09:53 |
| marios | zbr: only rdo vms | 09:53 |
| zbr | i know, that is why i say we need to involve the boss | 09:53 |
| zbr | we have some tests, we need to be able to run them. | 09:53 |
| *** bogdando has joined #oooq | 09:54 | |
| zbr | I can enable a molecule job on rdo, that is not the issue. but I do not want to disable it on upstream, or to lose the ability to run molecule when i am not on rdo network. | 09:55 |
| zbr | you see my point? is not only about upstream-zuul, but is also about developer-machines. | 09:55 |
| marios | zbr: yeah well either if you can hack skipping that one line or using a mock url there, or we find a public space . the latter may take longer though don't know | 09:56 |
| zbr | marios: do you know how to detect if I am on rdo network from inside a running job? | 09:56 |
| marios | zbr: well if its rdo job then you will be :D but don't know if thats what you mean | 09:56 |
| zbr | i could try to make that test to skip when outside rdo | 09:56 |
| marios | zbr: i mean, one simple test is to curl that rcm guest :D | 09:56 |
| marios | zbr: if it fails, you're not | 09:56 |
| zbr | lets bj a bit https://bluejeans.com/2655417928 | 09:57 |
| marios | zbr: can't right now | 09:57 |
| marios | brb | 09:57 |
| arxcruz|ruck | panda, around? | 09:58 |
| zbr | maybe there is an environment variable which is defined when a job runs under RDO. | 09:59 |
| *** dsneddon has quit IRC | 10:00 | |
| *** sshnaidm|afk is now known as sshnaidm | 10:03 | |
| *** sshnaidm is now known as sshnaidm|rover | 10:03 | |
| panda | sorry guys, meetgin plalend 2 hours before are fdifficult | 10:05 |
| panda | arxcruz|ruck: need help with the promotion ? | 10:05 |
| arxcruz|ruck | panda, yes please! | 10:06 |
| panda | arxcruz|ruck: wanna chat ? | 10:07 |
| arxcruz|ruck | panda, yes | 10:07 |
| panda | arxcruz|ruck: I'll join your bj | 10:08 |
| panda | arxcruz|ruck: what's the number ? :) | 10:08 |
| arxcruz|ruck | panda, let me check | 10:09 |
| arxcruz|ruck | panda, 7071866728 | 10:09 |
| panda | arxcruz|ruck: I was kidding, I use rover to get the bj id usually | 10:10 |
| sshnaidm|rover | arxcruz|ruck, looking | 10:12 |
| *** bogdando has quit IRC | 10:13 | |
| arxcruz|ruck | sshnaidm|rover, i'm checking right now with panda | 10:13 |
| sshnaidm|rover | arxcruz|ruck, I'm looking too :) | 10:13 |
| ykarel | arxcruz|ruck, sshnaidm|rover, are u guys looking for stein phase1 promotion? | 10:15 |
| sshnaidm|rover | ykarel, yes | 10:15 |
| arxcruz|ruck | ykarel, yes | 10:15 |
| ykarel | sshnaidm|rover, arxcruz|ruck ack | 10:15 |
| ykarel | fyi i reran the job few hours back | 10:15 |
| ykarel | in ci.centos | 10:16 |
| sshnaidm|rover | ykarel, yeah, I ran it in morning, it failed again.. | 10:16 |
| sshnaidm|rover | ykarel, but it should have been promoted previous hash | 10:16 |
| ykarel | sshnaidm|rover, ack, last fail seems environmeent related | 10:16 |
| ykarel | so i reran | 10:16 |
| *** bogdando has joined #oooq | 10:18 | |
| *** bogdando has quit IRC | 10:20 | |
| sshnaidm|rover | ykarel, do you know which hash this job ran? https://centos.logs.rdoproject.org/weirdo-stein-promote-puppet-openstack-scenario004/103/ | 10:21 |
| ykarel | sshnaidm|rover, d4eea6fb6258bd7baf828c5565cc03a7c7583871_e4d61247 | 10:21 |
| ykarel | https://centos.logs.rdoproject.org/weirdo-stein-promote-puppet-openstack-scenario004/103/repolist.txt.gz | 10:21 |
| ykarel | u can also check Parameters in job build | 10:22 |
| sshnaidm|rover | ykarel, damn, tripleo job ran in different hash: baseurl=https://trunk.rdoproject.org/centos7-stein/a8/49/a849dcf1c70b3d4653d208a16c33f727a956231f_e4d61247/ | 10:22 |
| sshnaidm|rover | arxcruz|ruck, ^ | 10:22 |
| sshnaidm|rover | arxcruz|ruck, that's why didn't promote | 10:22 |
| ykarel | :( how? | 10:22 |
| sshnaidm|rover | idk.. this build didn't promote: https://ci.centos.org/job/rdo_trunk-promote-stein-current-tripleo/119/ | 10:23 |
| ykarel | ohhk possible when it reran when there was new promotion from tripleo-ci | 10:23 |
| ykarel | but Parameters should not change | 10:23 |
| ykarel | sshnaidm|rover, i see same Hash in Parameters in both tripleo and weirdo job in 119 run | 10:24 |
| ykarel | i see d4eea6fb6258bd7baf828c5565cc03a7c7583871_e4d61247 | 10:24 |
| sshnaidm|rover | ykarel, well, it takes from tripleo-testing I think, I'm not sure it take hash from parameters.. | 10:25 |
| ykarel | it should be either parametrs or current-tripleo | 10:25 |
| sshnaidm|rover | need to check, hardly remember ci centos jobs | 10:25 |
| sshnaidm|rover | arxcruz|ruck, I think we can promote without it, the job ran with different hash.. | 10:26 |
| ykarel | hmm it promoting current-tripleo https://trunk.rdoproject.org/centos7-stein/current-tripleo | 10:27 |
| ykarel | which might have changed when job ran or reran | 10:27 |
| arxcruz|ruck | the delorean_current_hash parameter on tripleo-quickstart-promote-stein-current-tripleo-delorean-minimal is d4/ee/d4eea6fb6258bd7baf828c5565cc03a7c7583871_e4d61247 | 10:28 |
| sshnaidm|rover | ykarel, yeah, seems like race condition | 10:28 |
| arxcruz|ruck | but at the end it's getting a different one | 10:28 |
| ykarel | hmm right | 10:28 |
| ykarel | ideally it should use Parameters as those are static, current-tripleo symlink can change | 10:29 |
| sshnaidm|rover | ykarel, but weirdo jobs ran with same.. weird weirdo jobs | 10:30 |
| sshnaidm|rover | arxcruz|ruck, because we reran it and it took the newets current-tripleo hash | 10:32 |
| *** dsneddon has joined #oooq | 10:32 | |
| sshnaidm|rover | arxcruz|ruck, ykarel the problem is that we promote tripleo-ci too fast, we aren't used to | 10:32 |
| arxcruz|ruck | lol | 10:32 |
| arxcruz|ruck | sshnaidm|rover, what we do now then ? | 10:33 |
| ykarel | :D | 10:33 |
| sshnaidm|rover | arxcruz|ruck, if it passed with next hash, I think it's fine to release it with previous.. but need to file a bug for this race condition | 10:33 |
| *** dsneddon has quit IRC | 10:36 | |
| chandankumar | panda: please have a look at this patch https://review.rdoproject.org/r/#/c/21495/ | 10:38 |
| chandankumar | I have updated it | 10:38 |
| *** udesale has quit IRC | 10:38 | |
| panda | chandankumar: sorry for the meeting , do you want to share a summary ? | 10:40 |
| *** slaweq has quit IRC | 10:43 | |
| panda | chandankumar: do you know if ipa image needs overcloud images to be built first ? | 10:43 |
| panda | marios: ^ ? | 10:43 |
| *** slaweq has joined #oooq | 10:44 | |
| marios | panda: no there isn't a hard dependency so we shouldn't have to use that dependency (you mean in chandankumar patch for the pipeline right?. | 10:45 |
| panda | marios: yes | 10:45 |
| marios | panda: it might make sense in something like check to help on resources/no need to run if overcluod-full didn't build but even then you can argue we want them to run independently so | 10:45 |
| chandankumar | panda: i think it would be good to have image there for first run and then from second run onwards image should be uploaded there | 11:00 |
| chandankumar | panda: regarding summary: I have updated the patches with their proper parents to avoid merge conflict | 11:01 |
| panda | chandankumar: ok, what dependencies are missing ? | 11:01 |
| panda | chandankumar: I see zuul is not complaining ? | 11:01 |
| chandankumar | panda: https://review.rdoproject.org/r/#/c/21495/ | 11:02 |
| chandankumar | panda: and https://review.rdoproject.org/r/#/c/21270/ | 11:02 |
| chandankumar | panda: yes, zuul is not complaining as I rebased all the patches | 11:03 |
| chandankumar | and the sequence is written in commit message | 11:03 |
| chandankumar | panda: main thing to check https://review.rdoproject.org/r/#/c/21495/7/zuul.d/tripleo.yaml@133 -> job dependencies | 11:04 |
| chandankumar | panda: apart from that we discussed about container build job | 11:04 |
| panda | chandankumar: why are you adding a periodic job to the check pipeline in 21270 | 11:05 |
| panda | chandankumar: I see it was there before, not sure why | 11:06 |
| chandankumar | panda: that was just for testing | 11:06 |
| chandankumar | panda: let me remove that | 11:06 |
| panda | chandankumar: ok | 11:07 |
| chandankumar | arxcruz|ruck: panda I am putting the interview planing meeting tomorrow if it is ok? | 11:10 |
| panda | chandankumar: ok | 11:10 |
| *** dsneddon has joined #oooq | 11:10 | |
| chandankumar | panda: arxcruz|ruck kopecmartin invite sent for interview | 11:11 |
| chandankumar | prep | 11:11 |
| chandankumar | and 21270 review updated | 11:13 |
| *** bogdando has joined #oooq | 11:16 | |
| *** dsneddon has quit IRC | 11:26 | |
| *** ykarel is now known as ykarel|afk | 11:27 | |
| *** tesseract has joined #oooq | 11:28 | |
| kopecmartin | chandankumar, do you have time for a quick update meeting? | 11:32 |
| chandankumar | kopecmartin: join my bj | 11:32 |
| chandankumar | kopecmartin: redhat.bluejeans.com/chkumar | 11:32 |
| *** jfrancoa has quit IRC | 11:37 | |
| *** jfrancoa has joined #oooq | 11:38 | |
| *** panda is now known as panda|lunch | 11:44 | |
| weshay | sshnaidm++ | 11:55 |
| weshay | arxcruz++ | 11:55 |
| weshay | thanks arx | 11:58 |
| *** dsneddon has joined #oooq | 11:59 | |
| arxcruz|ruck | weshay, i did nothing, sshnaidm|rover is da man! | 12:01 |
| chandankumar | weshay: rfolco https://logs.rdoproject.org/95/21295/49/check/tripleo-build-containers-rhel-8-master/3450ca5/logs/buildah-builds/kolla-w7fbj9cc/docker/tempest/tempest-build.log.txt.gz | 12:02 |
| chandankumar | weshay: rfolco except tempest all are built now | 12:02 |
| chandankumar | weshay: rfolco tempest one is afailing with random selinux stuff | 12:03 |
| * weshay looks | 12:03 | |
| weshay | get out of town | 12:03 |
| weshay | chandankumar btw.. selinux should be perm | 12:03 |
| chandankumar | weshay: we fixed that | 12:03 |
| *** dsneddon has quit IRC | 12:03 | |
| chandankumar | in release file | 12:03 |
| weshay | aye.. we may want to check that in fact it's set to perm | 12:04 |
| weshay | chandankumar so Alan got us the virt rpms | 12:05 |
| chandankumar | weshay: yes | 12:05 |
| weshay | what fixed the other errors like etcd | 12:05 |
| chandankumar | amoralej: have all the magic for missing rpms | 12:06 |
| *** dsneddon has joined #oooq | 12:06 | |
| chandankumar | weshay: you can find patches iteration here | 12:06 |
| chandankumar | https://review.opendev.org/#/c/671183/ | 12:06 |
| weshay | woooooooooot | 12:06 |
| weshay | chandankumar rfolco OK!!! let's turn push on | 12:06 |
| rfolco | :) | 12:07 |
| chandankumar | rfolco: if you update this patch https://review.opendev.org/#/c/671183/ donot forgot to rebase it https://review.opendev.org/669471 | 12:07 |
| chandankumar | rfolco: virt:rhel is disabled now | 12:07 |
| chandankumar | in dockerfile | 12:08 |
| chandankumar | so that rhui virt gets installed | 12:08 |
| chandankumar | one more things swift container is still py2 | 12:09 |
| rfolco | chandankumar, adv virt was enabled in rhui ? I don't see any other fixes | 12:09 |
| chandankumar | rfolco: yes | 12:10 |
| rfolco | \o/ | 12:10 |
| chandankumar | rfolco: and check swift proxy container fix | 12:10 |
| chandankumar | rfolco: and i think we donot need sslverfy and gpg check disabled | 12:11 |
| rfolco | chandankumar, I did not include it on purpose, we cannot rely on py2 | 12:11 |
| chandankumar | there is no python3-ceilometermiddleware | 12:12 |
| rfolco | chandankumar, I thought this was in dib already | 12:12 |
| *** dsneddon has quit IRC | 12:12 | |
| chandankumar | rfolco: one min | 12:12 |
| rfolco | chandankumar, and tempest does not fail... what is wrong? | 12:12 |
| chandankumar | rfolco: https://softwarefactory-project.io/r/#/c/15940/ please add more info here | 12:13 |
| chandankumar | rfolco: selinux linux error | 12:13 |
| rfolco | hmm | 12:13 |
| chandankumar | rfolco: libsemanage.semanage_commit_sandbox: Error while renaming /var/lib/selinux/targeted/active to /var/lib/selinux/targeted/previous. (Invalid cross-device link) | 12:13 |
| weshay | chandankumar that is across a bunch of containers | 12:14 |
| rfolco | chandankumar, and note the mirror hack I included last night, few hours ago | 12:14 |
| rfolco | we need to fix that | 12:14 |
| chandankumar | weshay: then I need to build that container locally and see what is the issue | 12:16 |
| rfolco | yeah its not failing, but the container does not build | 12:17 |
| weshay | rfolco let's make sure selinux is put into permissive | 12:17 |
| weshay | maybe it's done in the image build | 12:17 |
| weshay | chandankumar rfolco it's also unclear to me what selinux's impact would be on the containers... afaik.. selinux is not in play on the containers themselves.. just the base node | 12:18 |
| *** apetrich has quit IRC | 12:21 | |
| rfolco | chandankumar, weshay: can I take over the patch and turn push on? | 12:23 |
| weshay | rfolco ya man | 12:23 |
| rfolco | k | 12:23 |
| weshay | panda|lunch where are we at w/ logging into the rdo registry | 12:24 |
| weshay | panda|lunch is this still *the* patch? https://review.rdoproject.org/r/#/c/21215/ | 12:25 |
| *** ykarel|afk is now known as ykarel | 12:25 | |
| weshay | arxcruz|ruck you can remove the comment about beta | 12:26 |
| weshay | just speak to our bits | 12:26 |
| *** ratailor__ has quit IRC | 12:26 | |
| weshay | arxcruz|ruck lolz :) | 12:27 |
| arxcruz|ruck | O.o | 12:27 |
| weshay | it's all good | 12:27 |
| arxcruz|ruck | weshay, did you hear me? | 12:27 |
| arxcruz|ruck | not sure if my mic was working i'm having problems with bj today | 12:27 |
| weshay | arxcruz|ruck no.. but it's fine | 12:27 |
| arxcruz|ruck | I was speaking lol | 12:28 |
| *** dsneddon has joined #oooq | 12:28 | |
| marios | rfolco: see comment in v42 fyi incase it is lost especially the become: true https://review.opendev.org/#/c/669487/42/roles/build-containers/tasks/main.yaml | 12:29 |
| weshay | rfolco chandankumar now that we have all the rpms.. and builds working.. noting selinux errors.. I'm comfortable closing the three bugs while we continue to work on kolla patches and carry what ever we need | 12:29 |
| rfolco | marios, its an old version of buildah/podman | 12:29 |
| rfolco | marios, does not work with normal user | 12:30 |
| marios | rfolco: so it is necessary? :( maybe we need two tasks then? | 12:30 |
| *** sshnaidm_ has joined #oooq | 12:30 | |
| weshay | rfolco chandankumar you guys comfortable w/ me closing the bugs? | 12:30 |
| marios | rfolco: i mean one with one without and condition. don't know it just seems wrong to run all the container build with sudo | 12:30 |
| rfolco | marios, yeah before merging we'll need to polish the entire thin | 12:30 |
| rfolco | thing | 12:30 |
| marios | rfolco: but ack ok yeah it isn't the most important thing just spotted it this morning | 12:30 |
| marios | rfolco: thanks | 12:30 |
| *** sshnaidm|rover has quit IRC | 12:30 | |
| *** sshnaidm_ is now known as sshnaidm|rover | 12:30 | |
| *** apetrich has joined #oooq | 12:31 | |
| rfolco | marios, your comment will be taken in consideration, I just need to push the containers now, then we fix the patch for merging | 12:31 |
| marios | rfolco: ack | 12:31 |
| weshay | start thinking about retro cards | 12:32 |
| *** dsneddon has quit IRC | 12:32 | |
| *** udesale has joined #oooq | 12:33 | |
| amoralej | chandankumar, sorry i was in a call, how is it going? | 12:36 |
| amoralej | are all containers building? | 12:36 |
| amoralej | just tempest failing | 12:37 |
| *** panda|lunch is now known as panda | 12:37 | |
| weshay | amoralej thanks for the help man!! very clutch :) Anyway.. chandankumar saw an selinux message in tempest, the other containers have the same issue. I think we're going to test w/ a simple standalone deployment and see how it goes | 12:45 |
| weshay | after we confirm selinux is in permissive | 12:45 |
| weshay | amoralej++ | 12:45 |
| amoralej | where did you see the selinux error? | 12:45 |
| amoralej | i couldn't find it | 12:45 |
| weshay | chandankumar rfolco marios rfolco periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master should be non-voting | 12:45 |
| amoralej | ah, ok, so the message could be not the actual error | 12:46 |
| amoralej | ? | 12:46 |
| weshay | libsemanage.semanage_commit_sandbox: Error while renaming /var/lib/selinux/targeted/active to /var/lib/selinux/targeted/previous. (Invalid cross-device link). | 12:46 |
| weshay | https://logs.rdoproject.org/95/21295/49/check/tripleo-build-containers-rhel-8-master/3450ca5/logs/containers-consolidated-builds.log.txt.gz | 12:46 |
| weshay | you see it across multiple containers | 12:46 |
| amoralej | yeah | 12:47 |
| weshay | sounds like selinux relabeling failing on a container.. probably fine | 12:47 |
| amoralej | i think sometime ago there was a problem with tempest container with old buildah | 12:47 |
| amoralej | because of duplicate workdir | 12:47 |
| amoralej | ykarel, ^ | 12:47 |
| amoralej | was it tempest? | 12:47 |
| chandankumar | amoralej: yes | 12:48 |
| chandankumar | amoralej: it appeared to be a buildah bug | 12:48 |
| amoralej | we may be using old buildah with el8 | 12:48 |
| amoralej | you remember where did we see the error message? | 12:48 |
| amoralej | apparently container builds | 12:49 |
| chandankumar | ykarel: ^^ | 12:49 |
| amoralej | but then is listed as failed | 12:49 |
| ykarel | amoralej, yes there was issue with buildah + tempest, it got fixed in buildah-1.8.1 and workaround is in tempest | 12:49 |
| ykarel | s/tempest/kolla | 12:49 |
| amoralej | ah, ok, so there is workaround in tempest | 12:49 |
| amoralej | it may be missing in el8 patch? | 12:49 |
| amoralej | ykarel, what was the workaround? | 12:50 |
| chandankumar | amoralej: weshay ykarel: https://review.opendev.org/#/c/652937/ | 12:50 |
| ykarel | amoralej, https://review.opendev.org/#/c/652937/ | 12:50 |
| chandankumar | amoralej: this patch is already included in tempest container | 12:50 |
| chandankumar | amoralej: https://logs.rdoproject.org/95/21295/49/check/tripleo-build-containers-rhel-8-master/3450ca5/logs/buildah-builds/kolla-w7fbj9cc/docker/tempest/Dockerfile.j2.txt.gz | 12:52 |
| amoralej | yep | 12:53 |
| amoralej | it's weird that tehre is no error log in the container build log but is listed as failed | 12:54 |
| ykarel | amoralej, yes it's related to buildah | 12:54 |
| ykarel | https://logs.rdoproject.org/95/21295/49/check/tripleo-build-containers-rhel-8-master/3450ca5/logs/buildah-builds/kolla-w7fbj9cc/docker/tempest/tempest-build.log.txt.gz | 12:54 |
| chandankumar | rfolco: can we add a task to collect all the rpm installed on the buildah vm? | 12:54 |
| chandankumar | it will be easier while debugging to know what packages are used | 12:55 |
| amoralej | in the docker file there is a last statement "USER tempest" | 12:55 |
| amoralej | which is not logged | 12:55 |
| ykarel | WORKDIR /tempest --> WORKDIR /var/lib/tempest should workaround it | 12:56 |
| chandankumar | rfolco: ykarel https://review.opendev.org/#/c/671183/ updating this patch itself | 12:56 |
| ykarel | ack | 12:56 |
| rfolco | cool | 12:56 |
| amoralej | ykarel, so https://review.opendev.org/#/c/652937/2/docker/tempest/Dockerfile.j2 is not enough? | 12:57 |
| *** holser_ has quit IRC | 12:58 | |
| amoralej | need to also update WORKDIR? | 12:58 |
| chandankumar | amoralej: ykarel updated and rechecked | 12:58 |
| amoralej | chandankumar, i think last ps is wrong | 12:59 |
| amoralej | ln -sf /var/lib/tempest /var/lib/tempest ? | 12:59 |
| amoralej | same file | 12:59 |
| amoralej | iiuc you need to keep that as is and update WORKDIR | 12:59 |
| rfolco | marios, sshnaidm, weshay, panda, rlandy, arxcruz, rfolco, chandankumar, zbr, kopecmartin | 12:59 |
| rfolco | marios, sshnaidm, weshay, panda, rlandy, arxcruz, rfolco, chandankumar, zbr, kopecmartin | 12:59 |
| rfolco | marios, sshnaidm, weshay, panda, rlandy, arxcruz, rfolco, chandankumar, zbr, kopecmartin | 12:59 |
| amoralej | ykarel, ^ | 13:00 |
| rfolco | retro meeting starting at https://bluejeans.com/5878458097 | 13:00 |
| amoralej | going for lunch | 13:00 |
| rfolco | https://etherpad.openstack.org/p/tripleo-ci-squad-meeting | 13:00 |
| kopecmartin | rfolco, I'll be one hour late, i have a team meeting now | 13:00 |
| rfolco | ack kopecmartin | 13:00 |
| ykarel | amoralej, right | 13:00 |
| ykarel | need WORKDIR /tempest --> WORKDIR /var/lib/tempest | 13:00 |
| rfolco | https://trello.com/b/0VFswmht/rdo-infra-retrospective?menu=filter&filter=label:UniSprint12 | 13:00 |
| ykarel | amoralej, and yes https://review.opendev.org/#/c/652937/2/docker/tempest/Dockerfile.j2 was not enough | 13:00 |
| amoralej | ok | 13:01 |
| amoralej | then it makes sense | 13:01 |
| *** amoralej is now known as amoralej|lunch | 13:01 | |
| *** hamzy has quit IRC | 13:05 | |
| *** dsneddon has joined #oooq | 13:07 | |
| chandankumar | ykarel: trying to rebase with different project review | 13:11 |
| chandankumar | that's why it was not working | 13:11 |
| ykarel | ack | 13:11 |
| sshnaidm|rover | zbr, can you please remove all molecule bits from https://review.rdoproject.org/r/#/c/21463/ ? I'd like to test this change and merge | 13:13 |
| sshnaidm|rover | zbr, let's do it before any molecule work | 13:13 |
| *** dsneddon has quit IRC | 13:18 | |
| *** holser_ has joined #oooq | 13:23 | |
| *** ykarel is now known as ykarel|afk | 13:25 | |
| arxcruz|ruck | weshay, rfolco my screen just frozen | 13:29 |
| rfolco | arxcruz|ruck, back? | 13:30 |
| rfolco | reconnect if its just you | 13:30 |
| *** dsneddon has joined #oooq | 13:30 | |
| *** ykarel|afk has quit IRC | 13:32 | |
| zbr | sshnaidm|rover: i am against changing the role without testing around it, especially complex changes like making it run on a new platform. but I agree that molecule should done separately, but before not after. | 13:32 |
| sshnaidm|rover | zbr, I test this role on live zuul system | 13:33 |
| sshnaidm|rover | zbr, with running ovb job, so it's most reliable test atm.. | 13:33 |
| zbr | sshnaidm|rover: here is my proposal: make sure that I see a job running when I touch any file under roles/ovb-manage/ -- this should be priority. | 13:34 |
| sshnaidm|rover | zbr, you can't, it's trusted repo | 13:34 |
| zbr | at least we would know that changes we make do not break current use. | 13:34 |
| zbr | we move it in another place, the trusted repo is not an excuse for not testing our code | 13:34 |
| sshnaidm|rover | zbr, there are reasons for it to be there | 13:35 |
| zbr | we need to find a way to test changes made to out code | 13:35 |
| *** dsneddon has quit IRC | 13:35 | |
| sshnaidm|rover | zbr, for testing it we'll need to redesign the whole role, moving parts of it to different repo, leaving parts of it in trusted, it uses secrets | 13:35 |
| sshnaidm|rover | zbr, the secrets leave us no choice | 13:35 |
| *** hamzy has joined #oooq | 13:36 | |
| *** aakarsh has quit IRC | 13:36 | |
| sshnaidm|rover | zbr, anyway it's big chunk of work to do, and not worth to start it in this patch at all | 13:37 |
| *** dsneddon has joined #oooq | 13:37 | |
| *** amoralej|lunch is now known as amoralej | 13:42 | |
| *** dsneddon has quit IRC | 13:42 | |
| *** udesale has quit IRC | 13:43 | |
| *** udesale has joined #oooq | 13:43 | |
| *** aakarsh has joined #oooq | 14:04 | |
| *** vinaykns has joined #oooq | 14:08 | |
| *** altlogbot_0 has quit IRC | 14:09 | |
| *** aakarsh|2 has joined #oooq | 14:10 | |
| *** aakarsh has quit IRC | 14:13 | |
| *** dsneddon has joined #oooq | 14:14 | |
| *** altlogbot_1 has joined #oooq | 14:17 | |
| *** irclogbot_1 has quit IRC | 14:17 | |
| *** ykarel has joined #oooq | 14:19 | |
| *** altlogbot_1 has quit IRC | 14:22 | |
| amoralej | chandankumar, i see all containers built fine but it still reports failure | 14:22 |
| amoralej | http://logs.rdoproject.org/95/21295/50/check/tripleo-build-containers-rhel-8-master/f20386a/logs/build.log.txt.gz | 14:22 |
| *** altlogbot_3 has joined #oooq | 14:25 | |
| *** udesale has quit IRC | 14:25 | |
| *** altlogbot_3 has quit IRC | 14:26 | |
| *** altlogbot_0 has joined #oooq | 14:26 | |
| chandankumar | amoralej: yes, commented, I think we need to fix returncode part in tripleo ci rhel 8 patch | 14:27 |
| *** ykarel has quit IRC | 14:31 | |
| *** altlogbot_0 has quit IRC | 14:32 | |
| *** dsneddon has quit IRC | 14:32 | |
| *** altlogbot_0 has joined #oooq | 14:33 | |
| *** dsneddon has joined #oooq | 14:35 | |
| *** Vorrtex has joined #oooq | 14:36 | |
| *** holser_ is now known as justanothernick | 14:37 | |
| *** altlogbot_0 has quit IRC | 14:38 | |
| *** altlogbot_1 has joined #oooq | 14:39 | |
| *** altlogbot_1 has quit IRC | 14:44 | |
| *** dsneddon has quit IRC | 14:44 | |
| weshay | arxcruz++ | 14:45 |
| weshay | sshnaidm++ | 14:45 |
| *** altlogbot_3 has joined #oooq | 14:45 | |
| *** altlogbot_3 has quit IRC | 14:50 | |
| *** altlogbot_0 has joined #oooq | 14:51 | |
| *** justanothernick is now known as holser_ | 14:54 | |
| *** ykarel has joined #oooq | 14:55 | |
| *** altlogbot_0 has quit IRC | 14:56 | |
| *** altlogbot_2 has joined #oooq | 14:57 | |
| *** dsneddon has joined #oooq | 15:00 | |
| *** skramaja_ has quit IRC | 15:01 | |
| *** altlogbot_2 has quit IRC | 15:02 | |
| *** holser_ has quit IRC | 15:02 | |
| *** sgolovat has joined #oooq | 15:02 | |
| *** altlogbot_2 has joined #oooq | 15:03 | |
| *** sgolovat is now known as holser | 15:03 | |
| *** holser has quit IRC | 15:04 | |
| *** jfrancoa has quit IRC | 15:04 | |
| *** holser has joined #oooq | 15:04 | |
| *** dsneddon has quit IRC | 15:06 | |
| *** altlogbot_2 has quit IRC | 15:08 | |
| *** altlogbot_0 has joined #oooq | 15:11 | |
| *** altlogbot_0 has quit IRC | 15:12 | |
| weshay | sshnaidm|rover https://review.rdoproject.org/r/#/c/21483/ | 15:12 |
| *** altlogbot_2 has joined #oooq | 15:13 | |
| ykarel | chandankumar, can u share that manual regarding container push that we discussed while i was leaving | 15:13 |
| chandankumar | ykarel: sharing | 15:13 |
| *** tosky has quit IRC | 15:13 | |
| chandankumar | ykarel: https://tree.taiga.io/project/tripleo-ci-board/task/1166?kanban-status=1447275 | 15:14 |
| ykarel | chandankumar, but ^^ looks empty, no detail | 15:15 |
| chandankumar | ykarel: https://tree.taiga.io/project/tripleo-ci-board/task/1162?kanban-status=1447277 | 15:15 |
| ykarel | chandankumar, Thanks | 15:16 |
| *** altlogbot_2 has quit IRC | 15:17 | |
| *** altlogbot_2 has joined #oooq | 15:19 | |
| *** dsneddon has joined #oooq | 15:20 | |
| *** dsneddon has quit IRC | 15:25 | |
| zbr | molecule presentation proposal: https://www.openstack.org/summit/shanghai-2019/vote-for-speakers#/24059 | 15:27 |
| *** altlogbot_2 has quit IRC | 15:28 | |
| *** altlogbot_0 has joined #oooq | 15:31 | |
| *** altlogbot_0 has quit IRC | 15:32 | |
| *** altlogbot_0 has joined #oooq | 15:33 | |
| *** dsneddon has joined #oooq | 15:34 | |
| *** ykarel is now known as ykarel|afk | 15:36 | |
| zbr | weshay: we have a problem with the rhui-cds on docker, because rdo dns does not resolve it and it shoudl. | 15:38 |
| *** altlogbot_0 has quit IRC | 15:38 | |
| zbr | to make use of the containers rhui-cds must be resolved by the local dns. | 15:38 |
| *** altlogbot_2 has joined #oooq | 15:39 | |
| zbr | even if we manually put the IP address of the server in repo files we will fail because the server redirects requests to https://rhui-cds/ | 15:39 |
| zbr | mainly I made it work correctly on my lan by adding an entry to my router DNS. but it still fails on rdo jobs. | 15:40 |
| zbr | i am not sure which DNS servers are used by our nodepool images, if they are not rdo ones, we may need to update the images. | 15:41 |
| *** dsneddon has quit IRC | 15:41 | |
| *** altlogbot_2 has quit IRC | 15:44 | |
| marios | rfolco: am almost out.. i will sync with sshnaidm|rover / arxcruz|ruck if i need to tomorrow morning after checking their etherpad. then you and me can sync when you come in tomorrow? | 15:45 |
| *** altlogbot_3 has joined #oooq | 15:45 | |
| rfolco | marios, sounds good, I'll be working on manual push today... | 15:45 |
| rfolco | thx | 15:46 |
| rfolco | :) | 15:46 |
| marios | rfolco: ack | 15:46 |
| *** ykarel|afk is now known as ykarel|away | 15:47 | |
| *** altlogbot_3 has quit IRC | 15:50 | |
| *** altlogbot_3 has joined #oooq | 15:51 | |
| rfolco | chandankumar, we still did not find the root cause of tempest container ? | 15:51 |
| *** marios is now known as marios|out | 15:56 | |
| *** holser has quit IRC | 15:56 | |
| *** holser has joined #oooq | 15:57 | |
| *** altlogbot_3 has quit IRC | 15:58 | |
| *** holser is now known as justanothernick | 16:02 | |
| *** altlogbot_3 has joined #oooq | 16:03 | |
| *** justanothernick is now known as sgolovat | 16:07 | |
| *** altlogbot_3 has quit IRC | 16:08 | |
| *** sgolovat has quit IRC | 16:09 | |
| *** altlogbot_2 has joined #oooq | 16:09 | |
| *** irclogbot_1 has joined #oooq | 16:13 | |
| *** marios|out has quit IRC | 16:13 | |
| *** altlogbot_2 has quit IRC | 16:14 | |
| *** altlogbot_3 has joined #oooq | 16:14 | |
| *** dsneddon has joined #oooq | 16:14 | |
| *** sshnaidm|rover is now known as sshnaidm|afk | 16:15 | |
| *** sabedevops has joined #oooq | 16:15 | |
| *** irclogbot_1 has quit IRC | 16:17 | |
| *** holser has joined #oooq | 16:20 | |
| *** holser has quit IRC | 16:22 | |
| *** irclogbot_0 has joined #oooq | 16:22 | |
| *** holser has joined #oooq | 16:22 | |
| *** holser has quit IRC | 16:23 | |
| *** dsneddon has quit IRC | 16:23 | |
| *** holser has joined #oooq | 16:23 | |
| *** holser has quit IRC | 16:24 | |
| *** holser has joined #oooq | 16:24 | |
| *** holser has quit IRC | 16:25 | |
| *** holser has joined #oooq | 16:25 | |
| *** irclogbot_0 has quit IRC | 16:26 | |
| weshay | rfolco what is the tempest error, I haven't seen it http://logs.rdoproject.org/95/21295/49/check/tripleo-build-containers-rhel-8-master/3450ca5/logs/buildah-builds/kolla-w7fbj9cc/docker/tempest/ | 16:26 |
| *** irclogbot_3 has joined #oooq | 16:26 | |
| weshay | although I don't see the signature written in that log | 16:27 |
| weshay | Storing signatures | 16:27 |
| weshay | --> 94526a499c3f855bed2dbde53eaadc1956eb37c6ca14805d691a3632076560ca | 16:27 |
| weshay | rfolco fyi.. ovh mirror is still not right.. | 16:28 |
| weshay | -07-17 16:17:58.452 14795 ERROR root [ ] Image prepare failed: 401 Client Error: Unauthorized for url: http://mirror.bhs1.ovh.openstack.org:8082/v2/tripleomaster/centos-binary-nova-libvirt/blobs/sha256:4c3152bda61158839e096bc06c363fb74f80e1b4c8a568f4365207f4542a65c9 | 16:28 |
| *** irclogbot_3 has quit IRC | 16:30 | |
| *** irclogbot_0 has joined #oooq | 16:30 | |
| *** irclogbot_0 has quit IRC | 16:34 | |
| *** altlogbot_3 has quit IRC | 16:34 | |
| *** irclogbot_1 has joined #oooq | 16:34 | |
| *** altlogbot_2 has joined #oooq | 16:35 | |
| *** irclogbot_1 has quit IRC | 16:38 | |
| *** irclogbot_0 has joined #oooq | 16:38 | |
| chandankumar | weshay: rfolco tempest image passed | 16:38 |
| weshay | ya.. cool | 16:39 |
| chandankumar | weshay: rfolco in current buildah there is a issue with symlink | 16:39 |
| weshay | symlink? | 16:39 |
| chandankumar | detection | 16:39 |
| weshay | ah | 16:39 |
| chandankumar | weshay: that's why we added a workaround in tempest image | 16:39 |
| chandankumar | last time | 16:39 |
| weshay | chandankumar RUN ln -sf /var/lib/tempest /tempest | 16:40 |
| chandankumar | ykarel|away: ^^ do you have the buildah symlink issue handy | 16:40 |
| ykarel|away | chandankumar, /me finds | 16:41 |
| *** irclogbot_0 has quit IRC | 16:42 | |
| *** irclogbot_1 has joined #oooq | 16:42 | |
| ykarel|away | chandankumar, https://bugs.launchpad.net/tripleo/+bug/1824952 | 16:42 |
| openstack | Launchpad bug 1824952 in tripleo "Tempest Container Image is not building successfully using buildah" [High,Triaged] | 16:42 |
| chandankumar | weshay: ^^ | 16:43 |
| weshay | https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/scenario003-standalone.yaml | 16:43 |
| weshay | oops | 16:43 |
| weshay | ykarel|away thanks | 16:43 |
| weshay | + '[' -s build-errors.log ']' │· | 16:43 |
| weshay | + '[' 128 -le 128 ']' │· | 16:43 |
| weshay | + echo 'FAILURE: I'\''m sorry. Some containers failed to build :(' │· | 16:43 |
| weshay | FAILURE: I'm sorry. Some containers failed to build :( │· | 16:43 |
| weshay | + exit 1 │· | 16:43 |
| weshay | ^C | 16:43 |
| weshay | rfolco ^ | 16:43 |
| *** irclogbot_1 has quit IRC | 16:46 | |
| *** irclogbot_3 has joined #oooq | 16:46 | |
| *** dsneddon has joined #oooq | 16:49 | |
| weshay | rfolco chandankumar https://review.opendev.org/#/c/669487/45..46/roles/build-containers/templates/build.sh.j2 | 16:49 |
| weshay | fyi.. fixes the shell error | 16:49 |
| chandankumar | rfolco: weshay what to do with the two kolla patches | 16:50 |
| *** irclogbot_3 has quit IRC | 16:50 | |
| weshay | chandankumar we'll spend time there.. for now we can carry the patches as we get the pipeline up | 16:50 |
| weshay | and check jobs | 16:50 |
| *** irclogbot_1 has joined #oooq | 16:50 | |
| weshay | chandankumar ok by you? | 16:50 |
| chandankumar | weshay: ack | 16:50 |
| weshay | chandankumar rfolco ok.. the manual build.. I think worked... | 16:51 |
| weshay | we just need to turn on push.. and rerun | 16:51 |
| weshay | but will wait for rfolco | 16:51 |
| chandankumar | weshay: rfolco I rechecked rdo patch | 16:52 |
| weshay | chandankumar ok.. so in ci we should see it pass I think | 16:52 |
| *** irclogbot_1 has quit IRC | 16:54 | |
| *** irclogbot_3 has joined #oooq | 16:54 | |
| *** dsneddon has quit IRC | 16:54 | |
| *** ykarel|away has quit IRC | 16:55 | |
| *** jbadiapa has quit IRC | 16:55 | |
| *** irclogbot_3 has quit IRC | 16:58 | |
| *** irclogbot_2 has joined #oooq | 16:58 | |
| weshay | chandankumar have you seen this? http://logs.openstack.org/94/668994/5/gate/tripleo-ci-centos-7-standalone-os-tempest/356b6a6/job-output.txt.gz#_2019-07-17_16_21_24_771008 | 17:00 |
| *** derekh has quit IRC | 17:00 | |
| weshay | no changes in there for a while | 17:01 |
| chandankumar | weshay: in config_template, few changes landed | 17:02 |
| *** irclogbot_2 has quit IRC | 17:02 | |
| chandankumar | weshay: https://github.com/openstack/ansible-config_template | 17:02 |
| weshay | chandankumar dang it | 17:02 |
| *** irclogbot_3 has joined #oooq | 17:02 | |
| weshay | chandankumar standalone-os-tempest is running in gate but check? https://review.opendev.org/#/c/668994/ | 17:02 |
| weshay | how did that happen | 17:02 |
| weshay | oh I see it | 17:03 |
| chandankumar | weshay: https://review.rdoproject.org/r/#/c/21519/ -> coming to rdo as a package | 17:03 |
| zbr | weshay: you can review again chandan addition of rhel8 release config, i updated it to bypass verification. see https://review.opendev.org/#/c/669057/ | 17:03 |
| *** amoralej is now known as amoralej|off | 17:05 | |
| weshay | zbr chandankumar I think this is fine for now | 17:05 |
| weshay | but we'll want to set selinux permissive explicitly as we get into fs001 | 17:05 |
| weshay | http://codesearch.openstack.org/?q=selinux&i=nope&files=&repos=openstack/tripleo-quickstart,openstack/tripleo-quickstart-extras | 17:05 |
| *** irclogbot_3 has quit IRC | 17:06 | |
| *** irclogbot_1 has joined #oooq | 17:06 | |
| chandankumar | weshay: you mean in rdocloud rhel8 nodepool image itself? | 17:06 |
| weshay | chandankumar atm.. in the release config that zbr pointed to.. https://review.opendev.org/#/c/669057/35/config/release/tripleo-ci/RedHat-8/master.yml | 17:07 |
| weshay | chandankumar review http://codesearch.openstack.org/?q=selinux&i=nope&files=&repos=openstack/tripleo-quickstart,openstack/tripleo-quickstart-extras in your time | 17:07 |
| weshay | chandankumar more immidiately.. have you seen http://logs.openstack.org/94/668994/5/gate/tripleo-ci-centos-7-standalone-os-tempest/356b6a6/job-output.txt.gz#_2019-07-17_16_21_24_771008 | 17:07 |
| weshay | previously | 17:08 |
| weshay | not sure how syntax issue could be a one off failure | 17:08 |
| weshay | maybe I'm not understanding something | 17:08 |
| weshay | yet | 17:08 |
| chandankumar | weshay: config_template is not there i think | 17:09 |
| chandankumar | weshay: https://github.com/openstack/ansible-config_template/commit/99d8c3cdaec63378047b32eb6f52386e2cd44669 change broke the stuff | 17:09 |
| *** irclogbot_1 has quit IRC | 17:10 | |
| weshay | oh k | 17:10 |
| *** irclogbot_2 has joined #oooq | 17:10 | |
| weshay | chandankumar that's a gate failure | 17:10 |
| weshay | we need a boog | 17:10 |
| weshay | rfolco ^ | 17:11 |
| *** dsneddon has joined #oooq | 17:13 | |
| *** sabedevops has quit IRC | 17:13 | |
| *** irclogbot_2 has quit IRC | 17:14 | |
| *** irclogbot_1 has joined #oooq | 17:14 | |
| rfolco | weshay, I am back from lunch | 17:15 |
| rfolco | weshay, 1) will re-run with push=true | 17:15 |
| weshay | rfolco thanks.. I upodated build.sh in the review as well | 17:15 |
| weshay | rfolco job is running .. may pass https://review.rdoproject.org/zuul/stream/e07d5c011ec745daacf34c6d29f13ac8?logfile=console.log | 17:15 |
| rfolco | weshay, build.sh in the review was not the same you saw in the box | 17:16 |
| rfolco | ok I trust on your judgement | 17:16 |
| weshay | rfolco https://review.opendev.org/#/c/669487/45..46/roles/build-containers/templates/build.sh.j2 | 17:16 |
| *** dtantsur is now known as dtantsur|afk | 17:16 | |
| rfolco | weshay, oh dammn equal is success ! stupid me | 17:17 |
| weshay | meh | 17:17 |
| rfolco | weshay, thx | 17:17 |
| *** bogdando has quit IRC | 17:17 | |
| *** irclogbot_1 has quit IRC | 17:18 | |
| chandankumar | rfolco: weshay we need to increase coverage on ansible-config_template project side | 17:18 |
| *** irclogbot_2 has joined #oooq | 17:18 | |
| *** sabedevops has joined #oooq | 17:18 | |
| chandankumar | it will break other scenario jobs also soon | 17:18 |
| weshay | zbr chandankumar fyi.. https://review.opendev.org/#/c/669057/ is failing in check | 17:18 |
| weshay | 2019-07-17 17:08:16.153448 | primary | changed: Repository 'rdo-qemu-ev': Error parsing config: Error parsing "baseurl = '$DISTRO_MIRROR_HOST/centos/7/virt/x86_64/kvm-common/'": URL must be http, ftp, file or https not "" | 17:19 |
| weshay | 2019-07-17 17:10:05.729114 | primary | "msg": "AnsibleUndefinedVariable: 'release' is undefined" | 17:19 |
| weshay | hrm.. we broke that build image job in some other patch | 17:19 |
| chandankumar | weshay: i thought release file fixes the stuff | 17:19 |
| weshay | chandankumar the release file is not in play for the container build.. or the image build | 17:20 |
| * chandankumar needs to check that part tomorrow then | 17:20 | |
| *** irclogbot_2 has quit IRC | 17:22 | |
| *** irclogbot_3 has joined #oooq | 17:22 | |
| rfolco | weshay, you want to check anything before I push the button ? | 17:23 |
| weshay | rfolco ya.. let's blue for a minute | 17:23 |
| weshay | rfolco might as well double check a few things before pushing | 17:24 |
| chandankumar | weshay: can we wait till job gives a green light? | 17:24 |
| rfolco | weshay, I'm on your blue | 17:24 |
| weshay | rfolco no ur not :) https://bluejeans.com/u/whayutin/ | 17:25 |
| rfolco | oh shit I am on mine | 17:25 |
| rfolco | what did I eat ? | 17:25 |
| rfolco | or smoke | 17:25 |
| rfolco | lolz | 17:25 |
| *** irclogbot_3 has quit IRC | 17:26 | |
| chandankumar | see ya tomorrow, | 17:26 |
| *** chandankumar is now known as raukadah | 17:26 | |
| weshay | raukadah thanks man | 17:26 |
| *** irclogbot_2 has joined #oooq | 17:26 | |
| *** irclogbot_2 has quit IRC | 17:30 | |
| *** irclogbot_3 has joined #oooq | 17:30 | |
| *** holser has quit IRC | 17:31 | |
| *** irclogbot_3 has quit IRC | 17:34 | |
| *** irclogbot_0 has joined #oooq | 17:34 | |
| *** irclogbot_0 has quit IRC | 17:38 | |
| *** irclogbot_3 has joined #oooq | 17:38 | |
| *** irclogbot_3 has quit IRC | 17:42 | |
| *** irclogbot_1 has joined #oooq | 17:42 | |
| *** irclogbot_1 has quit IRC | 17:46 | |
| *** irclogbot_1 has joined #oooq | 17:46 | |
| weshay | raukadah rfolco check job passed for rhel8 ocntainers | 17:49 |
| rfolco | weshay, one thing that I remembered is that osp15-deps.repo needs to be renamed like we did for fedora | 17:50 |
| *** irclogbot_1 has quit IRC | 17:50 | |
| *** irclogbot_0 has joined #oooq | 17:50 | |
| rfolco | to delorean-deps.repo | 17:50 |
| raukadah | rfolco: we cannot rename it | 17:50 |
| raukadah | it is for osp15 deps | 17:50 |
| raukadah | rfolco: for 16 it is coming soon | 17:51 |
| raukadah | once apevec back from pto | 17:51 |
| rfolco | I remember deployment expects delorean* repos | 17:51 |
| raukadah | one thing we can do is to compare version of from rhel/centos packages so that we can have a track what is missing | 17:51 |
| rfolco | somewhere... ok if you hit this problem, ping me | 17:51 |
| raukadah | rfolco: I will bug you anytime | 17:52 |
| rfolco | I don't work on weekends :) | 17:52 |
| rfolco | kidding | 17:52 |
| raukadah | no problem enjoy :-) | 17:53 |
| weshay | raukadah rfolco ya.. I'll fix raukadah's review | 17:53 |
| weshay | actually looks ok except for deps | 17:54 |
| raukadah | zbr: did you get a chance to look at ovb-manage stuff | 17:54 |
| *** irclogbot_0 has quit IRC | 17:54 | |
| rfolco | I am talking about this | 17:54 |
| rfolco | https://github.com/openstack/tripleo-ci/blob/master/roles/tripleo-repos/tasks/main.yml#L94 | 17:54 |
| rfolco | raukadah, weshay ^ | 17:54 |
| raukadah | it will speed up fs01 review | 17:54 |
| *** irclogbot_0 has joined #oooq | 17:54 | |
| rfolco | not sure that reuqirement is still true | 17:55 |
| raukadah | rfolco: above task got skipped | 17:56 |
| rfolco | raukadah, we had to do this for fedora... for rhel, we have osp15.repo... not sure we need an equivalent task for rhel | 17:57 |
| *** irclogbot_0 has quit IRC | 17:58 | |
| *** irclogbot_3 has joined #oooq | 17:58 | |
| weshay | rfolco what part of the deployment expects dlrn-deps? | 17:59 |
| rfolco | weshay, sorry, not deployment, container build. Its kolla config rpm_setup_config | 17:59 |
| rfolco | weshay, in periodic | 17:59 |
| rfolco | in check that config was "" | 17:59 |
| rfolco | weshay, look | 18:00 |
| rfolco | http://logs.openstack.org/25/624325/10/check/tripleo-build-containers-centos-7-buildah/814e8a4/conf/kolla-build.conf.txt.gz | 18:00 |
| rfolco | rpm_setup_config="http://172.17.0.1/delorean.repo,http://172.17.0.1/delorean-master-testing.repo" | 18:00 |
| rfolco | expects these two files | 18:00 |
| rfolco | delorean.repo and delorean-master-testing | 18:01 |
| weshay | rfolco so it could be adjusted in the config right? | 18:01 |
| *** irclogbot_3 has quit IRC | 18:02 | |
| rfolco | weshay, just saying... we injected delorean into containers. If we continue the way I did, we don't need this | 18:02 |
| *** irclogbot_3 has joined #oooq | 18:02 | |
| rfolco | weshay, ignore me for now. I'll shut up. | 18:03 |
| rfolco | back to container push | 18:03 |
| rfolco | :) | 18:03 |
| *** irclogbot_3 has quit IRC | 18:06 | |
| *** irclogbot_3 has joined #oooq | 18:06 | |
| *** irclogbot_3 has quit IRC | 18:10 | |
| *** irclogbot_1 has joined #oooq | 18:10 | |
| *** irclogbot_1 has quit IRC | 18:14 | |
| *** irclogbot_0 has joined #oooq | 18:14 | |
| *** irclogbot_0 has quit IRC | 18:18 | |
| *** irclogbot_2 has joined #oooq | 18:18 | |
| rfolco | weshay, around? | 18:21 |
| *** irclogbot_2 has quit IRC | 18:22 | |
| *** irclogbot_0 has joined #oooq | 18:22 | |
| *** irclogbot_0 has quit IRC | 18:26 | |
| *** irclogbot_2 has joined #oooq | 18:26 | |
| raukadah | weshay: rfolco https://review.opendev.org/#/c/669057/ time to merge it | 18:29 |
| *** irclogbot_2 has quit IRC | 18:30 | |
| *** holser has joined #oooq | 18:30 | |
| *** irclogbot_2 has joined #oooq | 18:30 | |
| *** irclogbot_2 has quit IRC | 18:34 | |
| *** irclogbot_0 has joined #oooq | 18:34 | |
| *** irclogbot_0 has quit IRC | 18:38 | |
| *** irclogbot_2 has joined #oooq | 18:38 | |
| *** irclogbot_2 has quit IRC | 18:42 | |
| *** irclogbot_0 has joined #oooq | 18:42 | |
| *** sbroderick has joined #oooq | 18:45 | |
| *** sabedevops has quit IRC | 18:45 | |
| *** irclogbot_0 has quit IRC | 18:46 | |
| *** irclogbot_0 has joined #oooq | 18:46 | |
| rfolco | weshay, buildah push does not work, but docker push works. How to make buildah get my login to push ? | 18:47 |
| weshay | rfolco https://review.rdoproject.org/r/#/c/21215/2/playbooks/tripleo-rdo-base/pre.yaml | 18:49 |
| *** irclogbot_0 has quit IRC | 18:50 | |
| rfolco | weshay, login works | 18:50 |
| *** irclogbot_1 has joined #oooq | 18:50 | |
| rfolco | weshay, but buildah push does not | 18:50 |
| rfolco | docker push worked for rhel-binary-base | 18:50 |
| rfolco | did manually | 18:50 |
| *** sabedevops has joined #oooq | 18:51 | |
| *** sbroderick has quit IRC | 18:52 | |
| *** irclogbot_1 has quit IRC | 18:54 | |
| *** irclogbot_0 has joined #oooq | 18:54 | |
| *** irclogbot_0 has quit IRC | 18:58 | |
| *** irclogbot_3 has joined #oooq | 18:58 | |
| rfolco | weshay, see in tmux upper panel | 18:58 |
| rfolco | weshay, login works, push works, with docker. | 18:58 |
| rfolco | but with buildah it does not work... whats the trick there? | 19:00 |
| *** irclogbot_3 has quit IRC | 19:02 | |
| *** irclogbot_1 has joined #oooq | 19:02 | |
| *** holser has quit IRC | 19:05 | |
| *** irclogbot_1 has quit IRC | 19:06 | |
| *** irclogbot_2 has joined #oooq | 19:06 | |
| *** irclogbot_2 has quit IRC | 19:10 | |
| *** irclogbot_2 has joined #oooq | 19:10 | |
| *** irclogbot_2 has quit IRC | 19:14 | |
| *** irclogbot_0 has joined #oooq | 19:14 | |
| *** irclogbot_0 has quit IRC | 19:18 | |
| *** irclogbot_0 has joined #oooq | 19:18 | |
| weshay | rfolco you can login w/ docker | 19:20 |
| weshay | and push w/ docker | 19:21 |
| rfolco | no | 19:21 |
| weshay | no? | 19:21 |
| rfolco | need to use buildah | 19:21 |
| * weshay looks at our push yml | 19:21 | |
| weshay | maybe we dont | 19:21 |
| * weshay gets | 19:21 | |
| rfolco | coz rhel does not have docker service | 19:21 |
| rfolco | to build w/ kolla using docker we need docker full, including service | 19:21 |
| rfolco | I can build w/ buildah and push with docker manually if you like | 19:22 |
| rfolco | but for the job this won't work | 19:22 |
| *** irclogbot_0 has quit IRC | 19:22 | |
| *** irclogbot_0 has joined #oooq | 19:22 | |
| rfolco | as it is an atomic operation under the covers of openstack overcloud container image build command using kolla | 19:22 |
| raukadah | weshay: rfolco https://review.opendev.org/#/c/671336/ please get this merged otherwise check jobs will fail | 19:23 |
| raukadah | oh no tripleo people already +w it | 19:23 |
| raukadah | but ci coverage is missing there | 19:24 |
| rfolco | weshay, I tried to remove --use-buildah | 19:24 |
| rfolco | it does not work, it requires docker.service to build w/ docker | 19:24 |
| raukadah | time to sleep now, see ya | 19:25 |
| weshay | rfolco one can always change the physics | 19:25 |
| weshay | if one needs to :) | 19:25 |
| weshay | install docker from pip or src | 19:25 |
| raukadah | rfolco: feel free to drop notes, I will be away for few hours tomorrow | 19:25 |
| *** irclogbot_0 has quit IRC | 19:26 | |
| *** irclogbot_3 has joined #oooq | 19:26 | |
| weshay | rfolco /me looks at the current post | 19:26 |
| *** irclogbot_3 has quit IRC | 19:30 | |
| *** irclogbot_2 has joined #oooq | 19:30 | |
| weshay | rfolco didn't you already push some containers/ | 19:31 |
| weshay | ? | 19:31 |
| rfolco | weshay, no I pushed just one rhel-binary-base w/ docker push command to test | 19:31 |
| rfolco | weshay, look at rdo registry on tripleomaster namespace | 19:31 |
| rfolco | weshay, look at tmux and I'll show you the error w/ docker build | 19:32 |
| weshay | rfolco blue? | 19:32 |
| rfolco | k | 19:32 |
| *** irclogbot_2 has quit IRC | 19:34 | |
| *** irclogbot_0 has joined #oooq | 19:34 | |
| *** kopecmartin is now known as kopecmartin|off | 19:36 | |
| *** irclogbot_0 has quit IRC | 19:38 | |
| *** irclogbot_3 has joined #oooq | 19:38 | |
| *** irclogbot_3 has quit IRC | 19:42 | |
| *** irclogbot_2 has joined #oooq | 19:42 | |
| *** irclogbot_2 has quit IRC | 19:46 | |
| *** irclogbot_1 has joined #oooq | 19:46 | |
| *** sabedevops has quit IRC | 19:48 | |
| *** irclogbot_1 has quit IRC | 19:50 | |
| *** irclogbot_1 has joined #oooq | 19:50 | |
| *** sabedevops has joined #oooq | 19:51 | |
| *** sabedevops has quit IRC | 19:53 | |
| *** irclogbot_1 has quit IRC | 19:54 | |
| *** irclogbot_1 has joined #oooq | 19:54 | |
| *** irclogbot_1 has quit IRC | 19:58 | |
| *** irclogbot_0 has joined #oooq | 19:58 | |
| *** saneax has quit IRC | 19:59 | |
| *** jdennis has joined #oooq | 20:00 | |
| *** irclogbot_0 has quit IRC | 20:02 | |
| *** irclogbot_2 has joined #oooq | 20:02 | |
| *** irclogbot_2 has quit IRC | 20:06 | |
| *** irclogbot_2 has joined #oooq | 20:06 | |
| *** Vorrtex has quit IRC | 20:07 | |
| *** sabedevops has joined #oooq | 20:08 | |
| *** irclogbot_2 has quit IRC | 20:09 | |
| *** irclogbot_0 has joined #oooq | 20:11 | |
| *** irclogbot_0 has quit IRC | 20:14 | |
| *** vinaykns has quit IRC | 20:30 | |
| *** saneax has joined #oooq | 20:39 | |
| *** irclogbot_3 has joined #oooq | 20:43 | |
| *** irclogbot_3 has quit IRC | 20:46 | |
| *** aakarsh|2 has quit IRC | 20:49 | |
| *** holser has joined #oooq | 21:10 | |
| rfolco | weshay, ./build.sh: line 65: [: -l: binary operator expected | 21:17 |
| rfolco | need to fix this in the patch | 21:17 |
| *** irclogbot_2 has joined #oooq | 21:22 | |
| *** saneax has quit IRC | 21:23 | |
| *** holser has quit IRC | 21:29 | |
| *** holser has joined #oooq | 21:29 | |
| *** panda has quit IRC | 21:46 | |
| *** panda has joined #oooq | 21:48 | |
| *** aakarsh|2 has joined #oooq | 21:50 | |
| *** sabedevops has quit IRC | 22:02 | |
| *** slaweq has quit IRC | 22:23 | |
| *** tesseract has quit IRC | 22:40 | |
| *** sabedevops has joined #oooq | 22:59 | |
| *** holser has quit IRC | 23:03 | |
| *** sabedevops has quit IRC | 23:04 | |
| *** jtomasek has quit IRC | 23:17 | |
| *** jtomasek has joined #oooq | 23:19 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!