| *** holser_ has quit IRC | 00:24 | |
| *** jtomasek has joined #oooq | 00:38 | |
| *** jtomasek has quit IRC | 00:43 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp- (1 more message) | 00:53 |
|---|---|---|
| *** jjoyce|gone has joined #oooq | 01:25 | |
| *** jjoyce has quit IRC | 01:25 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.openstack.org/564291 | 02:54 |
| *** ykarel has joined #oooq | 03:17 | |
| *** jtomasek has joined #oooq | 03:46 | |
| *** skramaja has joined #oooq | 03:50 | |
| *** udesale has joined #oooq | 04:02 | |
| *** chandankumar is now known as chkumar|ruck | 04:32 | |
| *** jtomasek has quit IRC | 04:48 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ (1 more message) | 04:54 |
| *** ykarel is now known as ykarel|afk | 05:48 | |
| *** ratailor has joined #oooq | 05:52 | |
| *** ykarel|afk has quit IRC | 05:53 | |
| *** ykarel|afk has joined #oooq | 06:14 | |
| *** ykarel|afk is now known as ykarel | 06:14 | |
| *** sshnaidm|off is now known as sshnaidm|afk | 06:45 | |
| *** quiquell|off is now known as quiquell|rover | 06:48 | |
| *** saneax has joined #oooq | 06:53 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ (1 more message) | 06:54 |
| *** jbadiapa has joined #oooq | 07:04 | |
| *** kopecmartin|off is now known as kopecmartin | 07:07 | |
| quiquell|rover | chkumar|ruck: o/ | 07:14 |
| chkumar|ruck | quiquell|rover: \o/ | 07:18 |
| chkumar|ruck | quiquell|rover: testing gnocchi pr https://review.rdoproject.org/r/#/c/13943/ here let's how it goes | 07:18 |
| quiquell|rover | chkumar|ruck: that job is working fine | 07:21 |
| quiquell|rover | https://softwarefactory-project.io/zuul/t/rdoproject.org/builds?job_name=periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master | 07:21 |
| chkumar|ruck | quiquell|rover: because telemetry tests were in skipp list | 07:21 |
| chkumar|ruck | quiquell|rover: https://bugs.launchpad.net/tripleo/+bug/1792862 | 07:21 |
| openstack | Launchpad bug 1792862 in tripleo "[master] Telemetry Tempest integration tests failed giving Unable to complete operation on subnet b13c76ec-c851-48d0-91a9-245a2fdcad9b: One or more ports have an IP allocation from this subnet." [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 07:21 |
| quiquell|rover | chkumar|ruck: ack | 07:22 |
| quiquell|rover | let's see then | 07:22 |
| chkumar|ruck | quiquell|rover: https://trello.com/c/OpWmOkAZ/742-cixlp1792862tripleociproa-master-telemetry-tempest-integration-tests-failed-giving-unable-to-complete-operation-on-subnet-b13c76 | 07:22 |
| chkumar|ruck | quiquell|rover: master promotion blocked due to this https://bugs.launchpad.net/tripleo/+bug/1815411 | 07:23 |
| openstack | Launchpad bug 1815411 in tripleo "[periodic/check][master] multinode fs016/019 & scenario-10 job failed while deploying overcloud with Permission denied: '/var/lib/mistral/overcloud/ceph-ansible" [Critical,Triaged] | 07:23 |
| chkumar|ruck | quiquell|rover: https://review.openstack.org/#/c/636086/ might fix the issue | 07:23 |
| *** jfrancoa has joined #oooq | 07:25 | |
| quiquell|rover | chkumar|ruck: something is no good at the cockpit they don't appear at periodic failures | 07:26 |
| *** saneax is now known as saneax|lunch | 07:29 | |
| quiquell|rover | chkumar|ruck: Also we can merge this https://review.openstack.org/#/q/topic:bug/1814872+(status:open+OR+status:merged) | 07:30 |
| chkumar|ruck | quiquell|rover: yes, | 07:30 |
| ykarel | ack will check post lunch | 07:31 |
| chkumar|ruck | marios: Hello | 07:39 |
| quiquell|rover | Go back in few | 07:39 |
| *** quiquell|rover is now known as quique|rover|brb | 07:39 | |
| chkumar|ruck | marios: regarding this https://review.openstack.org/#/c/634904/ I need to rework on this yum_update review sp that jinja will parse and update it otherwise it is complaing | 07:39 |
| quique|rover|brb | chkumar|ruck: btw are youi exercising the role locally ? | 07:40 |
| quique|rover|brb | It's going to be faster | 07:40 |
| marios | chkumar|ruck: ack i was reviewing it before you -1 | 07:41 |
| *** jtomasek has joined #oooq | 07:42 | |
| *** jtomasek has quit IRC | 07:43 | |
| *** jtomasek has joined #oooq | 07:54 | |
| *** chem has joined #oooq | 07:56 | |
| *** ykarel is now known as ykarel|lunch | 07:59 | |
| *** gkadam has joined #oooq | 08:01 | |
| *** saneax|lunch is now known as saneax | 08:01 | |
| *** holser_ has joined #oooq | 08:02 | |
| *** ccamacho has joined #oooq | 08:06 | |
| *** rascasoft has joined #oooq | 08:07 | |
| *** holser_ has quit IRC | 08:08 | |
| *** holser_ has joined #oooq | 08:19 | |
| *** holser_ has quit IRC | 08:22 | |
| *** jpena|off is now known as jpena | 08:22 | |
| *** holser_ has joined #oooq | 08:23 | |
| *** jaosorior has joined #oooq | 08:23 | |
| chkumar|ruck | quique|rover|brb: marios I am reusing the playbook defined in readme | 08:30 |
| chkumar|ruck | for tripleo modify imae | 08:30 |
| chkumar|ruck | *image | 08:30 |
| *** quique|rover|brb is now known as quiquell|rover | 08:39 | |
| *** ykarel|lunch is now known as ykarel | 08:46 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ (1 more message) | 08:54 |
| *** tosky has joined #oooq | 08:54 | |
| *** bogdando has joined #oooq | 09:12 | |
| *** sshnaidm|afk is now known as sshnaidm | 09:26 | |
| *** d0ugal has quit IRC | 09:28 | |
| quiquell|rover | chkumar|ruck: cockpit fixes now we see the failing periodic jobs | 09:32 |
| *** d0ugal has joined #oooq | 09:34 | |
| *** derekh has joined #oooq | 09:35 | |
| chkumar|ruck | quiquell|rover: cool! | 09:42 |
| arxcruz | panda: sshnaidm around? | 09:43 |
| sshnaidm | arxcruz, yes | 09:43 |
| arxcruz | panda: where do i need to add a job that will run fs020 only when tempest_skip_$release.yml file get updated ? | 09:43 |
| arxcruz | not sure if is in tripleo-ci or rdo-infra/rdo-jobs | 09:43 |
| arxcruz | and how to trigger it | 09:43 |
| quiquell|rover | chkumar|ruck: have updated steve baker review https://review.openstack.org/#/c/636086/ | 09:44 |
| quiquell|rover | chkumar|ruck: to include sceanrio010 there | 09:44 |
| arxcruz | i know i need to add files: ^roles/validate-tempest/vars/tempest_skip_$release | 09:44 |
| sshnaidm | arxcruz, add regexp ^roles/validate-tempest/vars/tempest_skip_.*$ | 09:46 |
| sshnaidm | arxcruz, add it to https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/tripleo.yaml#L346 | 09:47 |
| sshnaidm | arxcruz, look at fs039 for example | 09:47 |
| arxcruz | aha! | 09:47 |
| arxcruz | sshnaidm: תודה לך | 09:48 |
| *** dtantsur|afk is now known as dtantsur | 09:48 | |
| sshnaidm | arxcruz, אין על מה | 09:49 |
| sshnaidm | arxcruz, but if it will run 5 hours it may be a problem.. | 09:49 |
| arxcruz | sshnaidm: that's why i want to run only when tempest_skip_$release get touched | 09:49 |
| panda | arxcruz: where is the standalone cloud set up ? http://logs.openstack.org/25/634725/3/check/tripleo-ci-centos-7-scenario012-standalone/c2a185a/logs/undercloud/home/zuul/tempest.log.txt.gz | 10:04 |
| arxcruz | panda: I believe it was chkumar|ruck who work on this for standalone, iirc it should have a file called cloud.yaml with the credentials | 10:05 |
| arxcruz | since standalone job doesn't create the overcloudrc file | 10:05 |
| arxcruz | chkumar|ruck: is that correct? | 10:06 |
| chkumar|ruck | panda: arxcruz tht cares of that | 10:06 |
| arxcruz | chkumar|ruck: but the name is clouds.yaml right ? | 10:06 |
| chkumar|ruck | panda: arxcruz https://github.com/openstack/tripleo-heat-templates/blob/a535dd6e9cb70f120ca117480c45d04a31569089/extraconfig/post_deploy/clouds_yaml.py | 10:07 |
| chkumar|ruck | arxcruz: yes the name will be same | 10:07 |
| panda | mmhh, extraconfig | 10:07 |
| chkumar|ruck | arxcruz: it will look for clouds.yaml file in ~/.config or /etc/openstac | 10:07 |
| arxcruz | chkumar|ruck: export OS_CLOUD=standalone so, this is wrong right? it should be clouds.yaml | 10:07 |
| arxcruz | or inside clouds.yaml must have a definition for standalone ? | 10:07 |
| chkumar|ruck | arxcruz: panda now overcloud , undercloud has got the support of clouds.yaml | 10:08 |
| chkumar|ruck | arxcruz: we donot need to export os_cloud | 10:08 |
| chkumar|ruck | arxcruz: you mean in validate-tempest? | 10:08 |
| arxcruz | chkumar|ruck: i guess... | 10:08 |
| chkumar|ruck | arxcruz: for validate-tempest we need to export as needed during network creation | 10:09 |
| panda | arxcruz: OS_CLOUD specify the cloud name not, the configuration file | 10:09 |
| panda | not sure why it is not created, I don';t see anything different in scenario001 that calls that template | 10:09 |
| chkumar|ruck | panda: arxcruz os_tempest takes cares of that | 10:10 |
| arxcruz | chkumar|ruck: we are talking about validate-tempest | 10:10 |
| arxcruz | i guess | 10:10 |
| panda | marios: do you remember if you had to add something so that standalone would create the cloud.yaml for tempest ? | 10:11 |
| chkumar|ruck | arxcruz: panda https://github.com/openstack/tripleo-quickstart-extras/commit/5a6ba2889373912e2c9475082d9ab8a8d3499faa commit added | 10:11 |
| marios | panda: no but you can check the existing 1-4 env files all the tempest stuff is in the job definitions | 10:11 |
| marios | panda: but no don't recall having to do something for clouds.yaml | 10:11 |
| zbr | rfolco: hi! let me know when you can sync about tasks | 10:11 |
| panda | marios: yep, taht shwat I was doing without much success, thanks anyway. | 10:12 |
| marios | panda: k sorry :/ | 10:12 |
| panda | I'm dyslexic, but only when I write. | 10:12 |
| chkumar|ruck | panda: during standalone deployment clouds.yaml got created | 10:13 |
| panda | chkumar|ruck: in that log I posted ? | 10:15 |
| chkumar|ruck | panda: let me check that log | 10:16 |
| chkumar|ruck | panda: http://logs.openstack.org/25/634725/3/check/tripleo-ci-centos-7-scenario012-standalone/c2a185a/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2019-02-08_17_45_11 | 10:17 |
| chkumar|ruck | panda: now I need to check why it is not getting exported | 10:17 |
| panda | chkumar|ruck: I don't see it specifically, I see the deployment was successful, not if that file was created. | 10:18 |
| panda | chkumar|ruck: I think Its' not created, and I found the reason | 10:21 |
| chkumar|ruck | panda: where? | 10:21 |
| chkumar|ruck | panda: where can i find the logs for these http://logs.openstack.org/25/634725/3/check/tripleo-ci-centos-7-scenario012-standalone/c2a185a/logs/undercloud/home/zuul/tripleo-heat-installer-templates/extraconfig/ / | 10:21 |
| chkumar|ruck | ? | 10:21 |
| *** dsneddon has quit IRC | 10:22 | |
| chkumar|ruck | http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/environments/undercloud.yaml#n5 -> takes care of clouds.yaml generation | 10:22 |
| *** jtomasek has quit IRC | 10:22 | |
| panda | ChanServ: https://review.openstack.org/#/c/634723/7/ci/environments/scenario012-standalone.yaml@19 | 10:22 |
| panda | chkumar|ruck: ^ | 10:23 |
| panda | I'm overriding the post | 10:23 |
| panda | chkumar|ruck: thanks for pointing me in the right direction | 10:23 |
| chkumar|ruck | panda: yes | 10:23 |
| *** dsneddon has joined #oooq | 10:23 | |
| chkumar|ruck | panda: you are welcome :-) | 10:23 |
| *** jtomasek has joined #oooq | 10:25 | |
| *** dsneddon has quit IRC | 10:27 | |
| *** dsneddon has joined #oooq | 10:48 | |
| *** dsneddon has quit IRC | 10:52 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ (1 more message) | 10:54 |
| chkumar|ruck | quiquell|rover: once this review gets in https://review.openstack.org/#/c/636114/ we can tests gnoochi prs in periodic jobs in rdo | 11:14 |
| quiquell|rover | chkumar|ruck: ack | 11:15 |
| *** jtomasek has quit IRC | 11:17 | |
| *** jtomasek has joined #oooq | 11:17 | |
| *** dsneddon has joined #oooq | 11:22 | |
| panda | marios: you invitation arrived 1 minute before the end of the meeting ... | 11:25 |
| marios | panda: oh no its now i scheduled it wrong sec will update | 11:25 |
| marios | i mean in 5 mins ssb | 11:25 |
| marios | zbr: ^ | 11:25 |
| marios | & panda ^ | 11:26 |
| panda | marios: ok | 11:26 |
| rfolco | zbr, hi! let me prepare the patches so we can walk through them and review together | 11:26 |
| zbr | panda: much better than the voting for the corp merge, got my forms about o month after the vote. | 11:26 |
| *** dsneddon has quit IRC | 11:27 | |
| marios | zbr: you can't hear us | 11:32 |
| marios | zbr: we can hear you | 11:32 |
| *** holser_ is now known as holser|lunch | 11:37 | |
| marios | *=* 11:07:00 *=*=*= "Allow distro to be auto detected" https://review.openstack.org/#/c/635475/4 I4a0807133dcf7c3081f228d1b6b2119e3cd29300 | 11:38 |
| marios | panda: ^T | 11:38 |
| marios | kthxbai panda zbr | 11:44 |
| marios | panda: 14 mins that's my kind of meeting | 11:44 |
| panda | marios: I saw some bubble starting to pop on you cheek. | 11:45 |
| marios | panda: :D | 11:45 |
| zbr | rfolco: also look at ^ it should clearly help with your story. | 11:46 |
| rfolco | zbr, https://review.openstack.org/#/c/635475? | 11:46 |
| zbr | rfolco: yes | 11:47 |
| panda | zbr: in general, if you look at what's around that build containers job for f28, you can start making sure that it's supporting multiple distribution | 11:47 |
| rfolco | zbr, indeed | 11:47 |
| panda | zbr: even before having the job itself | 11:47 |
| *** dsneddon has joined #oooq | 12:02 | |
| chkumar|ruck | panda: hello | 12:02 |
| panda | chkumar|ruck: O_oo | 12:03 |
| chkumar|ruck | panda: https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/build-test-packages/tasks/dlrn-build.yml#L71 | 12:03 |
| chkumar|ruck | panda: How we can chnage this path artg_change.host to github.com | 12:04 |
| chkumar|ruck | ? | 12:04 |
| chkumar|ruck | panda: it is related to this review http://logs.rdoproject.org/43/13943/78/check/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/8f64dc8/logs/ara_oooq/result/97bb01db-04b9-4efc-8e52-d4844813ee6d/ | 12:04 |
| chkumar|ruck | panda: https://review.openstack.org/#/c/636114/ in order to tes gnocchi pr in periodic jobs | 12:04 |
| chkumar|ruck | to unblock telemetry tempest tests | 12:04 |
| *** udesale has quit IRC | 12:05 | |
| *** udesale has joined #oooq | 12:06 | |
| panda | chkumar|ruck: for where doest the change come ? | 12:06 |
| panda | chkumar|ruck: gerrit ? | 12:06 |
| chkumar|ruck | panda: the change is coming from github pr | 12:06 |
| chkumar|ruck | panda: https://review.rdoproject.org/r/18791 -> added this so that zuul knows about gnocchi project | 12:07 |
| *** dsneddon has quit IRC | 12:07 | |
| chkumar|ruck | panda: https://review.rdoproject.org/r/#/c/13943/ where we are testing by pointing gnocchi pr as adepends-on | 12:07 |
| panda | chkumar|ruck: not supported. That role supports zuull or gerrit | 12:08 |
| chkumar|ruck | hmm | 12:10 |
| panda | hhmm indeed. | 12:11 |
| chkumar|ruck | :-) | 12:11 |
| chkumar|ruck | panda: let me find someworkaround if possible | 12:11 |
| chkumar|ruck | panda: let me try with this change https://review.openstack.org/#/c/636093/ it will fix the host | 12:14 |
| chkumar|ruck | it will not work | 12:15 |
| panda | where is that change fixing the host ? | 12:16 |
| *** dsneddon has joined #oooq | 12:17 | |
| panda | chkumar|ruck: arxcruz I have a question about validate tempest and configure_tempest.sh template. Who has time ? | 12:21 |
| arxcruz | panda: shoot | 12:21 |
| *** dsneddon has quit IRC | 12:21 | |
| panda | arxcruz: in this change https://review.openstack.org/#/c/509728/36/roles/validate-tempest/templates/configure-tempest.sh.j2 we need to add configuration specific for ironic. NOw I realize that I have to add even more pre configuration to make ironic ready to be tested. If I add it here, how will it be ported to OSA then ? Is this even the right place to do configurations for single services ? | 12:23 |
| panda | arxcruz: I'll have to add commands that install epel and libvirt and then create two fake vbmc nodess to use with ironic | 12:23 |
| arxcruz | hmmmmmm | 12:23 |
| arxcruz | need to add on osa as well in the future when we migrate | 12:24 |
| panda | you tempesteers are high on the humming today :) | 12:24 |
| panda | arxcruz: is that possible/easy ? | 12:24 |
| arxcruz | panda: we don't have it on osa side, and i don't think it's hard, but I'll be against shell script because you might already realize how messed it is on validate-tempest, and i don't want to make the same mistakes again | 12:25 |
| panda | arxcruz: ok, how can I add further services configuration in a way that is easy to migrate to osa ? | 12:26 |
| arxcruz | panda: depending on how you want to do it | 12:26 |
| arxcruz | if it's something that menas just change an option on tempest, it's very easy | 12:27 |
| panda | arxcruz: this is what I have to run before tempest http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/ci/common/vbmc_setup.yaml | 12:27 |
| panda | arxcruz: it's currenlty at heat stack level, but it doesn't have to be | 12:27 |
| chkumar|ruck | panda: I am thinking on how to fix this part https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/build-test-packages/library/zuul_deps.py#L56 | 12:27 |
| chkumar|ruck | so that github will be get appended | 12:28 |
| arxcruz | panda: can't it be a separated playbook/role ? | 12:28 |
| arxcruz | does it must be inside validate-tempest and in the future in osa ? | 12:28 |
| panda | marios: do you know if it's possible to specify two different templates for the same OS::SOMETHING::WHATEVER: services ? | 12:28 |
| panda | arxcruz: no it doesn't have to, by I'm trying to find the correct place for it | 12:29 |
| panda | arxcruz: it's for CI only, it's not part of the product, it's needed for testing, but it's current position is overriding other important parts in the workflow | 12:29 |
| panda | arxcruz: I either find a new place in heat templates, or ... or .. | 12:30 |
| quiquell|rover | chkumar|ruck: I see again the tripleo-container-image-prepare failing | 12:32 |
| quiquell|rover | at OVB jobs | 12:32 |
| marios | panda: you mean same value in different env files yes | 12:32 |
| marios | panda: last one wins | 12:32 |
| chkumar|ruck | quiquell|rover: stuck at libselinux issue trying to find right solution there | 12:32 |
| *** jpena is now known as jpena|lunch | 12:32 | |
| quiquell|rover | chkumar|ruck: are you using virtualenv ? | 12:33 |
| chkumar|ruck | quiquell|rover: yes | 12:33 |
| quiquell|rover | chkumar|ruck: have you try with --system-site-packages | 12:33 |
| marios | panda: i mean if you have os::tripleo::service foo in env1 and then service bar in another env2 depends on the order you pas the env files in the call to deploy/heat | 12:33 |
| chkumar|ruck | quiquell|rover: in my local setup I am using the tox venv then there I am trying to run but the same issue is coming above also | 12:33 |
| quiquell|rover | after installing the python-selinux RPM ? | 12:34 |
| panda | marios: no, I want to run both templates. | 12:34 |
| quiquell|rover | chkumar|ruck: you can pass virtualenv args at tox | 12:34 |
| quiquell|rover | chkumar|ruck: pass --system-site-packages | 12:34 |
| marios | panda: no then you need new services, or makes one include the other like we do for the service base stuff | 12:34 |
| quiquell|rover | chkumar|ruck: also at ansible I think ou have to override ansible_python_interpreter to python3 | 12:34 |
| panda | marios: can the include be made conditional ? | 12:34 |
| chkumar|ruck | quiquell|rover: ok | 12:35 |
| marios | panda:e.g. here https://github.com/openstack/tripleo-heat-templates/blob/4bc7de4e44fd437f83ff46a70cfc8366769429a7/deployment/nova/nova-consoleauth-container-puppet.yaml#L70-L71 not sure about that there is an if heat statement not sure if you can wrap that resource declaration into a conditional | 12:36 |
| panda | marios: mmhh, thanks, it gets complicated really quickly then. arxcruz I think some role/playbooks just before tempest is the best option. | 12:39 |
| arxcruz | panda: i agree | 12:42 |
| arxcruz | sorry, i was lunching | 12:42 |
| *** holser|lunch is now known as holser_ | 12:43 | |
| *** weshay has joined #oooq | 12:46 | |
| panda | arxcruz: are there other projects taht need to make independent setup before tempest ? where do thety put them ? If I don't use configure_tempest.sh, I'll need to create another hook point on some playbook | 12:47 |
| arxcruz | panda: not that i am aware, osa has some installer that might do some setup, and just pass the variables to os_tempest | 12:48 |
| arxcruz | just like we do | 12:48 |
| *** ratailor has quit IRC | 12:49 | |
| zbr | @oooq https://review.openstack.org/#/c/635576/ on bindep, it could ease our new repro logic. | 12:51 |
| zbr | https://review.openstack.org/#/c/635590/ as well | 12:51 |
| *** dsneddon has joined #oooq | 12:52 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.openstack.org/564291 | 12:54 |
| *** dsneddon has quit IRC | 12:57 | |
| *** derekh has quit IRC | 12:58 | |
| *** skramaja has quit IRC | 13:03 | |
| *** skramaja has joined #oooq | 13:04 | |
| *** amoralej is now known as amoralej|lunch | 13:07 | |
| *** rlandy has joined #oooq | 13:10 | |
| weshay | rlandy, good news.. successful repro tests on a fresh system for libvirt and rdo | 13:12 |
| weshay | quiquell|rover, sshnaidm ^ | 13:12 |
| weshay | f28 | 13:13 |
| rlandy | weshay: great - thanks for merging that | 13:13 |
| sshnaidm | \o/ | 13:13 |
| *** trown|outtypewww is now known as trown | 13:13 | |
| quiquell|rover | weshay: very good news :-) | 13:13 |
| weshay | couple WIP changes in the works | 13:15 |
| weshay | https://review.openstack.org/#/c/636073/ | 13:15 |
| weshay | https://review.openstack.org/#/c/636028/ | 13:16 |
| *** dsneddon has joined #oooq | 13:29 | |
| weshay | quiquell|rover, chkumar|ruck need someone in the cix call | 13:31 |
| * chkumar|ruck joining | 13:31 | |
| quiquell|rover | chkumar|ruck: ack, thanks | 13:31 |
| *** dsneddon has quit IRC | 13:33 | |
| chkumar|ruck | panda: please have a look at this card https://trello.com/c/xUy82HzR/869-cixlp1813913tripleociproa-periodic-tripleo-fedora-28-master-containers-build-failing-with-unauthorized-access-to-rdo-registry related to this bug https://bugs.launchpad.net/tripleo/+bug/1813913 | 13:34 |
| openstack | Launchpad bug 1813913 in tripleo "periodic-tripleo-fedora-28-master-containers-build failing with unauthorized access to rdo registry" [Wishlist,Triaged] - Assigned to Gabriele Cerami (gcerami) | 13:34 |
| chkumar|ruck | panda: feel free to update the trello when free | 13:34 |
| *** jpena|lunch is now known as jpena | 13:35 | |
| chkumar|ruck | panda: I have copied the last comment from your last bug comment | 13:37 |
| panda | chkumar|ruck: why is that in escalation ? | 13:38 |
| chkumar|ruck | panda: https://bugs.launchpad.net/tripleo/+bug/1813913 got added to promotion blocker | 13:39 |
| openstack | Launchpad bug 1813913 in tripleo "periodic-tripleo-fedora-28-master-containers-build failing with unauthorized access to rdo registry" [Wishlist,Triaged] - Assigned to Gabriele Cerami (gcerami) | 13:39 |
| chkumar|ruck | from last sprint | 13:39 |
| chkumar|ruck | panda: I will ask to remove that from there | 13:39 |
| panda | chkumar|ruck: was added by error I think, we are supposed to CIX after this sprint as DOD for the promotion. We may resurrect if it's still there. | 13:40 |
| chkumar|ruck | panda: removed from cix alerts | 13:40 |
| panda | chkumar|ruck: thanks. I'll make sure it's readded if at the end of the sprint the issue is not solved. | 13:45 |
| chkumar|ruck | quiquell|rover: weshay https://bugs.launchpad.net/tripleo/+bug/1815411 for this bug cix card does not got created | 13:46 |
| openstack | Launchpad bug 1815411 in tripleo "[periodic/check][master] multinode fs016/019 & scenario-10 job failed while deploying overcloud with Permission denied: '/var/lib/mistral/overcloud/ceph-ansible" [Critical,Triaged] | 13:46 |
| quiquell|rover | chkumar|ruck: About that I am working with steve baker review to fix it | 13:47 |
| chkumar|ruck | quiquell|rover: yes, seen, :-) | 13:47 |
| rlandy | panda: can we chat re: fs059? | 13:49 |
| *** derekh has joined #oooq | 13:49 | |
| panda | rlandy: yes, after the scrum ? | 13:49 |
| *** quiquell|rover is now known as quique|rover|eat | 13:49 | |
| rlandy | panda; ack - thanks - not clear to me why some settings have been left out | 13:49 |
| weshay | marios, this fixed my chrony issue https://review.openstack.org/#/c/636073/2 | 13:53 |
| *** amoralej|lunch is now known as amoralej | 13:53 | |
| weshay | when using libvirt, I did not hit the issue using rdo | 13:53 |
| marios | weshay: looking | 13:54 |
| zbr | weshay: i am no longer able to read our kb articles, lacking subscription. i wonder if mine expired or if i am doing somethign wrong. | 13:54 |
| weshay | zbr, hrm.. every employee should have a login | 13:54 |
| marios | weshay: ack in my case it was beaker specific it seems all 8.8.8.8 and 1.1.1. blocked | 13:55 |
| *** ykarel is now known as ykarel|away | 13:55 | |
| weshay | 1.1.1.1 is blocked? | 13:55 |
| weshay | jebus | 13:55 |
| rlandy | internal blocks outside | 13:55 |
| weshay | marios, rlandy makes me wonder how to override these settings for the repro? | 13:55 |
| rlandy | same with baremetal | 13:55 |
| rlandy | weshay: rdo reaches outside | 13:56 |
| rlandy | weshay: for libvirt, we can override in the job definition | 13:56 |
| rlandy | we have dealt with this issue in bm for some time | 13:56 |
| rlandy | and internal ovb | 13:56 |
| rlandy | not the custom_nameserver settings | 13:57 |
| rlandy | that's the override | 13:57 |
| marios | weshay: rlandy lets talk in 3 mins :) | 13:57 |
| marios | brb | 13:57 |
| weshay | rlandy, btw.. internal sf mtg right after our scrum | 13:58 |
| *** skramaja has quit IRC | 13:59 | |
| rlandy | weshay: ack - I got up at 6am to attend the meeting - only to see it had moved :) | 13:59 |
| weshay | rlandy, ugh.. sorry | 13:59 |
| rlandy | weshay: np - now I know how you feel | 13:59 |
| *** skramaja has joined #oooq | 13:59 | |
| rfolco | ping scrum marios, quiquell, sshnaidm, weshay, panda, rlandy, arxcruz, mwhahaha, rfolco, chkumar, ssbarnea, kopecmartin | 14:00 |
| *** ykarel|away has quit IRC | 14:01 | |
| *** dsneddon has joined #oooq | 14:02 | |
| *** bogdando has quit IRC | 14:03 | |
| *** bogdando has joined #oooq | 14:03 | |
| quique|rover|eat | weshay, sshnaidm: cockpit with noops from gerrit and using mariadb https://review.rdoproject.org/r/18777 | 14:04 |
| weshay | aye.. thanks | 14:04 |
| weshay | quique|rover|eat, jk... WHERE'S the screenshot ;P | 14:05 |
| * weshay looking | 14:06 | |
| quique|rover|eat | amending | 14:06 |
| weshay | don't post I'm kidding | 14:06 |
| quique|rover|eat | now is there | 14:06 |
| weshay | quique|rover|eat, ^ | 14:06 |
| weshay | ha ha | 14:06 |
| weshay | k | 14:06 |
| quique|rover|eat | noop where almost never accurate | 14:07 |
| quique|rover|eat | s/where/were/ | 14:07 |
| *** dsneddon has quit IRC | 14:07 | |
| *** quique|rover|eat is now known as quiquell|rover | 14:17 | |
| panda | arxcruz: how much time until you EOD ? | 14:19 |
| arxcruz | 2 hours, i have german exam today :/ | 14:19 |
| arxcruz | i can be online after the exam | 14:19 |
| arxcruz | or tomorrow i can wake earlier :) | 14:19 |
| arxcruz | did sorin change his nickname ? | 14:20 |
| panda | arxcruz: ok, tomorrow at 6 am | 14:20 |
| arxcruz | panda: it's a date ;) | 14:20 |
| zbr | arxcruz: yes, arx, i am zbr on freenode now. (but i get notification if you use old nick) | 14:22 |
| arxcruz | zbr: why? why y want to complicate? | 14:22 |
| zbr | arxcruz: just to avoid overlapping with sshnaidm -- too often people were misstagging us due to common "ss" in id. | 14:24 |
| panda | arxcruz: you have a meeting after this call ? | 14:27 |
| *** ykarel has joined #oooq | 14:27 | |
| arxcruz | panda: nope | 14:27 |
| panda | arxcruz: weshay can we talk about scenario12 immediately after this call ? | 14:28 |
| zbr | sshnaidm or panda: https://review.openstack.org/#/c/634060/ -- build-containers to vote. | 14:28 |
| arxcruz | panda: sure so, canceling our date tomorrow 6 am? | 14:29 |
| * arxcruz inserts a heartbreaking meme here | 14:29 | |
| panda | arxcruz: hold on until weshay's ack. | 14:32 |
| chkumar|ruck | quiquell|rover: marios can you take a look at this patch while running yum-update getting retry error | 14:38 |
| chkumar|ruck | quiquell|rover: marios https://review.openstack.org/#/c/634904/ | 14:38 |
| *** skramaja has quit IRC | 14:39 | |
| *** vinaykns has joined #oooq | 14:40 | |
| *** dsneddon has joined #oooq | 14:40 | |
| quiquell|rover | chkumar|ruck: do you have a pointer to the error ? | 14:40 |
| chkumar|ruck | quiquell|rover: http://paste.openstack.org/show/744844/ | 14:40 |
| weshay | chkumar|ruck, quiquell|rover fyi.. have you guys seen https://tree.taiga.io/project/tripleo-ci-board/us/702?milestone=217491 | 14:40 |
| weshay | chkumar|ruck, quiquell|rover if you have questions about this user story.. I can provide context | 14:41 |
| chkumar|ruck | quiquell|rover: update my code in review.openstack.org | 14:41 |
| chkumar|ruck | weshay: https://bugs.launchpad.net/tripleo/+bug/1813546 -> is in progress | 14:42 |
| openstack | Launchpad bug 1813546 in tripleo "ansible-role-modify-image is not py3 and fedora 28+ ready" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 14:42 |
| quiquell|rover | weshay: nope, how is this task generated ? | 14:42 |
| quiquell|rover | weshay: is like result of bug triage ? | 14:42 |
| *** agopi has joined #oooq | 14:42 | |
| chkumar|ruck | arxcruz: https://bugs.launchpad.net/tripleo/+bug/1813224 is still work in progress from your side? | 14:42 |
| openstack | Launchpad bug 1813224 in tripleo "fedora28 standalone failing on tempest" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 14:42 |
| arxcruz | chkumar|ruck: hmmmm need to check who was working on it... | 14:43 |
| *** dsneddon has quit IRC | 14:44 | |
| rlandy | panda: can we bj now for a few? | 14:48 |
| arxcruz | chkumar|ruck: I wasn't working on it, I remember on CI escalation meeting, someone said it would take a look (on neutron side) | 14:48 |
| panda | rlandy: yes | 14:48 |
| arxcruz | rlandy: no, panda is mine after this meeting | 14:48 |
| arxcruz | :P | 14:48 |
| panda | arxcruz: weshay didn't ack yet. | 14:49 |
| arxcruz | (╯°□°）╯︵ ┻━┻ | 14:49 |
| panda | rlandy: joining your channel | 14:49 |
| rlandy | arxcruz: you can join as well - won't be long | 14:49 |
| quiquell|rover | chkumar|ruck: scenarios are passing now at https://review.openstack.org/#/c/636086 | 14:51 |
| quiquell|rover | chkumar|ruck: going to put a test review on periodic to see if it works now | 14:51 |
| ykarel | quiquell|rover, i see scenarios failing on ^^ | 14:54 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.openstack.org/564291 | 14:54 |
| ykarel | quiquell|rover, atleast sceanrio010 which is what we need to get passed | 14:54 |
| quiquell|rover | ykarel: scenario010 is 038 | 14:55 |
| weshay | panda, need me to review something? | 14:55 |
| quiquell|rover | ykarel: and we don't have it activated at criteria | 14:55 |
| quiquell|rover | ykarel: going to throw a test review with periodics | 14:55 |
| quiquell|rover | ykarel: Is this allrigh ? https://review.rdoproject.org/r/18802 | 14:55 |
| ykarel | quiquell|rover, but if it's failing how testreview will help? | 14:55 |
| ykarel | i see it's failing for ceph-ansible | 14:56 |
| ykarel | so scenario010. 001, 004 all have same issue | 14:57 |
| ykarel | scenario010 is failing in your review | 14:58 |
| quiquell|rover | ykarel: Ahh damn ok | 14:58 |
| weshay | chkumar|ruck, quiquell|rover, ya.. it's a result of a bug triage.. so I would say... those bugs are second priority to any new issues you are finding in the pipeline | 14:58 |
| weshay | chkumar|ruck, quiquell|rover does that make sense? | 14:58 |
| quiquell|rover | weshay: yep | 14:59 |
| quiquell|rover | ykarel: at least we have progress on it | 14:59 |
| ykarel | yup agree on that part | 14:59 |
| quiquell|rover | ykarel: so looks like standalone scenarios are passing there | 15:00 |
| quiquell|rover | ykarel: so we have here a bug not catch by them :-( | 15:00 |
| ykarel | standalone were already good | 15:00 |
| ykarel | yup exactly | 15:00 |
| quiquell|rover | We cannot have it all | 15:00 |
| panda | weshay: no, I wanted to understand if you have time to talk about my blocker in ironic | 15:07 |
| panda | weshay: arxcruz is available | 15:07 |
| weshay | panda, I'm in an internal sf meeting for the next hour | 15:08 |
| weshay | panda, avail on irc | 15:08 |
| arxcruz | panda: i can later when i get back from my german class or tomorrow, up to you :) | 15:09 |
| panda | weshay: arxcruz ok let's try to kick off the discussion here. Restating the problem, I need to find a place for this bad boy http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/ci/common/vbmc_setup.yaml | 15:10 |
| rlandy | sshnaidm: added card https://tree.taiga.io/project/tripleo-ci-board/task/736 | 15:10 |
| rlandy | sshnaidm: if you can add some details in there as to what is needed, I'll put up a review for OVB jobs | 15:10 |
| panda | marios: you were suggesting to add an additional service to the scenario ? | 15:10 |
| sshnaidm | rlandy, ack | 15:11 |
| marios | panda: well if you need one... i was more replying to your ' i have config but don't know where to put it' answer is in tht/ci/environments/ | 15:14 |
| panda | marios: it's already in tht/ci, I don't know if I can add just a line in the scenario and call it in some way | 15:15 |
| panda | marios: I don't know how to make an effective mapping between some service and the template that implements it | 15:16 |
| marios | panda: what does it define? you can just declare an instance of it and set its type in the env. registry | 15:16 |
| marios | panda: k do you want to have a call about this tomorrow morning? | 15:16 |
| *** dsneddon has joined #oooq | 15:16 | |
| rlandy | weshay: did I miss anything? | 15:18 |
| panda | marios: tomorrow morning could be too late, people could get hurt in the meantime. | 15:18 |
| marios | panda: trying to find example for you sec | 15:18 |
| weshay | rlandy, ? | 15:18 |
| panda | marios: jk tomorrow morning is fine if you don't have time now | 15:18 |
| rlandy | weshay: in the first 15 mins of the SF meeting - was chatting with panda | 15:18 |
| weshay | rlandy, haven't seen anything that interests you.. | 15:20 |
| weshay | rlandy, added you to the agenda | 15:20 |
| weshay | https://docs.google.com/document/d/16NgqyuGMLj4ECqlW52ATasNvSrYUq7eipBcdGrs-hxY/edit# | 15:21 |
| rlandy | weshay: thanks - see that item | 15:21 |
| *** dsneddon has quit IRC | 15:21 | |
| marios | panda: simplest one is userdata like declaration https://github.com/openstack/tripleo-heat-templates/blob/4bc7de4e44fd437f83ff46a70cfc8366769429a7/overcloud.j2.yaml#L411-L412 and then mapped via resource reg https://github.com/openstack/tripleo-heat-templates/blob/4bc7de4e44fd437f83ff46a70cfc8366769429a7/overcloud-resource-registry-puppet.j2.yaml#L45 to https://github.com/openstack/tripleo-heat-templates/blob/master/firstboot/userdata_default.yam | 15:22 |
| panda | marios: ok so http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/ci/environments/scenario012-multinode-containers.yaml#n19 points to http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/ci/common/vbmc_setup.yaml but I have to make it point to http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/extraconfig/post_deploy/standalone_post.yaml instead. And then I have | 15:22 |
| panda | to find another place to put the vbmc setup because it's needed for the service to be tested properly | 15:22 |
| panda | marios: looking | 15:22 |
| marios | panda: that last thing you point to can be some softwareconfig/deployment pair that runs eg a bash script | 15:22 |
| marios | or ansible or whatever you want to run | 15:22 |
| panda | marios: ok, so in summary I could create a NodeExtraPostIronic: somewhere with type OS::Tripleo::SetupIronic as type, then add a OS::Tripleo::SetupIronic: vbmc_setup.yaml line in the scenario file ? is it that simple ? Where I should add the NodeExtraPostIronic to be effectively working in the flow ? | 15:26 |
| marios | panda: yah something like that will be easier to comment once its in the review and specific :) but yah like the default is OS::Heat::None in the resource registry and you can activate with the scenario environmetnf ile | 15:27 |
| marios | with a mapping to the real thing you want | 15:27 |
| *** saneax has quit IRC | 15:33 | |
| *** quiquell|rover is now known as quiquell|off | 15:39 | |
| *** agopi has quit IRC | 15:39 | |
| panda | marios: mmhh, I'm looking around, I have no idea where to put the intial ExtraIronicVBMCConfig. I suppose I have to find an environment that matches standalone, and a place in that file that is in the right order in the workflow. ... | 15:42 |
| marios | panda: ack (pvt and we talk tomorrow) | 15:49 |
| *** gkadam has quit IRC | 15:49 | |
| *** dsneddon has joined #oooq | 15:55 | |
| weshay | panda, rfolco I'll be a few minutes late to the triage | 15:59 |
| rfolco | weshay, ack | 15:59 |
| *** dsneddon has quit IRC | 15:59 | |
| panda | rfolco: please join | 16:01 |
| chkumar|ruck | zbr: marios: weshay sshnaidm panda need some help on this review https://review.openstack.org/#/c/634904/ hitting this error http://logs.openstack.org/04/634904/12/check/tripleo-ci-centos-7-undercloud-containers/c0e5ebf/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz locally as well | 16:01 |
| chkumar|ruck | no clue what I am doing wrong | 16:01 |
| sshnaidm | chkumar|ruck, is it? https://github.com/containers/buildah/issues/475#issuecomment-397861089 | 16:04 |
| marios | chkumar|ruck: ack noting for next reviews | 16:09 |
| chkumar|ruck | sshnaidm: updating with sshnaidm comment | 16:09 |
| panda | sshnaidm: can you add to the bug the patch you're working on to fix https://bugs.launchpad.net/tripleo/+bug/1815048 ? | 16:12 |
| openstack | Launchpad bug 1815048 in tripleo "[ovb][collect_logs] ovb collect logs job runs twice " [High,Triaged] | 16:12 |
| sshnaidm | panda, this one? https://review.openstack.org/#/c/635510/ | 16:14 |
| *** udesale has quit IRC | 16:15 | |
| sshnaidm | panda, tripleo-ci doesn't connect LP to patches, as it's in openstack-infra namespace.. | 16:15 |
| *** ykarel is now known as ykarel|away | 16:19 | |
| *** chkumar|ruck is now known as chandankumar | 16:24 | |
| panda | sshnaidm: I see you updated , thanks. | 16:24 |
| *** dsneddon has joined #oooq | 16:30 | |
| weshay | rlandy, https://review.openstack.org/#/c/635567/ | 16:36 |
| *** bogdando has quit IRC | 16:38 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.openstack.org/564291 | 16:54 |
| rfolco | zbr, don't see you review comment on https://review.openstack.org/636160... feel free to commit it and co-author the patch | 16:55 |
| weshay | panda, rfolco sorry about the overlap.. did you guys do any bug triage? | 16:55 |
| rfolco | weshay, we kind of started... closed one bug | 16:56 |
| weshay | rfolco, panda should we reschedule? | 16:56 |
| panda | weshay: lots of bugs on upgrades in the list, we tried look at them not much background there. Close one bug on tq gates, and pushed for closure on one ovb collect logs. | 16:58 |
| rfolco | there was one new bug that is being worked on... | 16:58 |
| weshay | rfolco, imho.. let's look for a slot in the next few days | 16:58 |
| weshay | anyone against this? https://review.openstack.org/#/c/636073/ adding a dns server to multinode? | 17:00 |
| panda | weshay: we are pointing to 127.0.0.1 probably because of a local dnsmasq. I don't know if it does only forwarding, or has some additional configuration that woul be bypassed by using directly an external DNS. | 17:05 |
| weshay | localhost points to unbound | 17:05 |
| weshay | panda, not a local dnsmasq | 17:05 |
| *** dtrainor has quit IRC | 17:08 | |
| *** ykarel|away has quit IRC | 17:08 | |
| weshay | panda, there is no cache on local libvirt images | 17:08 |
| weshay | only nodepool | 17:08 |
| *** dtrainor has joined #oooq | 17:10 | |
| weshay | panda, fyi https://review.openstack.org/#/c/494545/ | 17:10 |
| chandankumar | sshnaidm: not worked adding entrypoint http://logs.openstack.org/04/634904/13/check/tripleo-ci-centos-7-undercloud-containers/0974a1c/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz | 17:19 |
| chandankumar | sshnaidm: I am 100% sure I am missing something here https://review.openstack.org/#/c/634904/ | 17:19 |
| *** jpena is now known as jpena|off | 17:35 | |
| rlandy | trown: hello ... I'd like to chat with you re: scenario009 - we are building an equivalent standalone job | 17:37 |
| panda | weshay: voted/commented. also https://review.rdoproject.org/r/#/c/18502/ should be fixed now , waiting for CI to stabilize | 17:37 |
| weshay | thanks looking | 17:38 |
| trown | rlandy: sure | 17:40 |
| rlandy | trown: thanks - let's start with the multinode job - which is currently failing on master | 17:41 |
| *** panda is now known as panda|off | 17:41 | |
| rlandy | trown: it's easiest to compare when we have a working multinode job ... | 17:41 |
| rlandy | http://logs.openstack.org/64/635464/4/check/tripleo-ci-centos-7-scenario009-multinode-oooq-container/2803dce/ | 17:41 |
| trown | rlandy: can you point me to a failing job... oh thanks :) | 17:42 |
| rlandy | the job fails out on node prep and doesn't even get to deployment | 17:42 |
| rlandy | trown: rocky multinode passes - if that's a better release to go after | 17:43 |
| trown | rlandy: sure http://logs.openstack.org/64/635464/4/check/tripleo-ci-centos-7-scenario009-multinode-oooq-container/2803dce/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz#_2019-02-11_15_02_37 doesnt look related to scenario009 at a glance | 17:44 |
| rlandy | trown: ack - I agree but since the job does not get to deploy, hard to know if deploy should work on master at all ... | 17:44 |
| trown | rlandy: it looks like that `podman ps | grep executor` is returning null | 17:44 |
| rlandy | so we like to start with a working multinode | 17:45 |
| trown | rlandy: cool, lets start with rocky then | 17:45 |
| rlandy | trown: ok - so ... going with the rocky angle ... | 17:45 |
| rlandy | let's take a look at the featureset and tht reviews | 17:46 |
| rlandy | thanks to panda, we have the following to start with: | 17:46 |
| rlandy | https://review.openstack.org/#/c/635461/5/config/general_config/featureset059.yml | 17:47 |
| rlandy | ( compare with fs 033) | 17:47 |
| rlandy | and | 17:47 |
| rlandy | https://review.openstack.org/#/c/635208/5/ci/environments/scenario009-standalone.yaml | 17:47 |
| rlandy | compare with https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/scenario009-multinode.yaml | 17:47 |
| rlandy | trown: clearly we are missing stuff here | 17:47 |
| rlandy | and the tht review needs to be rebased on rocky | 17:48 |
| rlandy | here is the latest log from standalone: | 17:48 |
| rlandy | http://logs.openstack.org/64/635464/4/check/tripleo-ci-centos-7-scenario009-standalone/e5cc40b/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz | 17:49 |
| rlandy | trown: I'd like you take on what services are really required and not required - so we know whether these failures are legit or we just have the templates incorrectly configured | 17:49 |
| trown | rlandy: hmm the error in that log seems like mysql is not even setup | 17:52 |
| trown | http://logs.openstack.org/64/635464/4/check/tripleo-ci-centos-7-scenario009-standalone/e5cc40b/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2019-02-11_15_00_46 | 17:52 |
| rlandy | trown: yep - so looks like we are missing that | 17:52 |
| *** saneax has joined #oooq | 17:53 | |
| rlandy | trown: iiuc, we would likely need all the services listed in multinode tht config ... https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/scenario009-multinode.yaml#L12 | 17:54 |
| rlandy | trown: ok - let me cherry pick onto rocky - add the services I think we are missing - and I'll ping you again with those results | 17:56 |
| trown | rlandy: it is more about the resource_registry I think https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/scenario009-multinode.yaml#L5-L7 | 17:56 |
| trown | I am not sure ControllerServices does anything with standalone | 17:56 |
| rlandy | trown: correct they don't ... | 17:56 |
| rlandy | https://review.openstack.org/#/c/635208/5/ci/environments/scenario009-standalone.yaml | 17:57 |
| rlandy | but what is in ^^ is clearly missing stuff | 17:57 |
| trown | oh right those are there | 17:57 |
| rlandy | that is what we are trying to narrow down | 17:57 |
| trown | rlandy: ya I guess first try could be just adding in https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/scenario004-standalone.yaml#L26 to that | 17:57 |
| trown | rlandy: that should get mysql... then see what the next error is | 17:58 |
| trown | maybe that is all that is missing | 17:58 |
| rlandy | trown: yep - will go with rocky and master concurrently for the moment | 17:58 |
| *** derekh has quit IRC | 17:58 | |
| trown | rlandy: ok, feel free to ping me if you get new errors and need help | 18:03 |
| *** holser_ has quit IRC | 18:03 | |
| *** trown is now known as trown|lunch | 18:03 | |
| weshay | welcome back.. | 18:03 |
| weshay | :) | 18:03 |
| *** dtantsur is now known as dtantsur|afk | 18:07 | |
| *** kopecmartin is now known as kopecmartin|off | 18:12 | |
| *** amoralej is now known as amoralej|off | 18:17 | |
| *** fultonj has quit IRC | 18:26 | |
| zbr | weshay: https://review.openstack.org/#/c/632156/ | 18:39 |
| *** rlandy is now known as rlandy|brb | 18:52 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.openstack.org/564291 | 18:54 |
| *** fultonj has joined #oooq | 19:05 | |
| *** fultonj has quit IRC | 19:06 | |
| *** rlandy|brb is now known as rlandy | 19:09 | |
| *** jfrancoa has quit IRC | 19:11 | |
| *** fultonj has joined #oooq | 19:18 | |
| weshay | rlandy, this is interesting... | 19:19 |
| weshay | task path: /var/tmp/RECREATE/playbooks/tripleo-ci-reproducer/pre.yaml:85 | 19:19 |
| weshay | changed: [localhost] => {"append": true, "changed": true, "comment": "weshayutin", "group": 1000, "groups": "dockerroot", "home": "/home/weshayutin", "move_home": false, "name": "weshayutin", "shell": "/bin/bash", "state": "present", "uid": 1000} | 19:19 |
| weshay | ERROR! Cannot reset connection: | 19:19 |
| weshay | Control socket connect(/home/weshayutin/.ansible/cp/276d6f8b70): No such file or directory | 19:19 |
| weshay | rlandy, quiquell|off sshnaidm the connection reset fails on centos-7.6 | 19:19 |
| rlandy | hmmm | 19:20 |
| *** trown|lunch is now known as trown | 19:20 | |
| rlandy | and we didn;t prevent that if you had eveything set up | 19:21 |
| weshay | rpm -q ansible | 19:21 |
| weshay | ansible-2.4.2.0-2.el7.noarch | 19:21 |
| weshay | ya.. I put up a patch to skip it.. | 19:22 |
| rlandy | collect logs takes forever :( | 19:23 |
| rlandy | "Warning: Scope(Class[Tripleo::Profile::Base::Neutron::Dhcp_agent_wrappers]): Docker runtime is deprecated. Consider switching container_cli to podman", | 19:25 |
| rlandy | panda|off: ^^ | 19:25 |
| weshay | rlandy, easy https://review.rdoproject.org/r/#/c/18804/ | 19:26 |
| rlandy | weshay: what does platform: rpm resolve to? | 19:26 |
| weshay | rlandy, centos, fedora, rhel | 19:27 |
| rlandy | so it's being skipped on centos currently? | 19:27 |
| weshay | no.. w/o any profile.. it will run on anythign | 19:27 |
| weshay | anything | 19:27 |
| weshay | like.. ubuntu, debian etc | 19:27 |
| weshay | and those packages will not be found | 19:27 |
| weshay | make sense? | 19:27 |
| rlandy | weshay: waiting on ci to vote on https://review.rdoproject.org/r/#/c/18804/ - I tried that before and it failed ci - so let's see. if ci is ok with it, +2 | 19:28 |
| weshay | rlandy, worked locally | 19:29 |
| * weshay runs tox | 19:29 | |
| weshay | ya.. should pass | 19:29 |
| rlandy | weshay: k - will +2 - if zuul fails it - we will find out | 19:29 |
| *** saneax has quit IRC | 19:29 | |
| weshay | dear lord | 19:29 |
| weshay | check this out.. OH EPEL | 19:30 |
| weshay | cat /etc/docker/daemon.json | 19:30 |
| weshay | {"group": "dockerroot"}[weshayutin@localhost RECREATE]$ | 19:30 |
| weshay | epel did that ^ | 19:30 |
| rlandy | OMG | 19:30 |
| rlandy | I give up!! | 19:30 |
| weshay | lolz | 19:30 |
| rlandy | weshay: we can relax the check | 19:30 |
| * weshay wonders if I can just remove epel after python-pip is installed | 19:31 | |
| rlandy | we do so in libvirt | 19:31 |
| rlandy | install epel and kick it | 19:31 |
| rlandy | weshay: unrelated standalone question ... | 19:31 |
| rlandy | pacemaker in standlone | 19:32 |
| rlandy | how does that make sense? | 19:32 |
| rlandy | I see it's needed | 19:32 |
| weshay | can you show me? | 19:32 |
| rlandy | http://logs.openstack.org/64/635464/4/check/tripleo-ci-centos-7-scenario009-standalone/9d5381f/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2019-02-11_19_05_01 | 19:32 |
| rlandy | 2019-02-11 19:05:01 | "Error: Evaluation Error: Error while evaluating a Function Call, The 'hacluster_pwd' hiera key is undefined, did you forget to include ::tripleo::profile::base::pacemaker in your role? (file: /etc/puppet/modules/tripleo/manifests/profile/base/pacemaker.pp, line: 94, column: 5) on node standalone.localdomain", | 19:33 |
| weshay | 2019-02-11 19:05:01 | "stderr: Error: unable to get cib", | 19:33 |
| weshay | 2019-02-11 19:05:01 | "Error: unable to get cib", | 19:33 |
| rlandy | it is here ... | 19:33 |
| rlandy | https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/scenario004-standalone.yaml#L22 | 19:33 |
| weshay | I don't think it's doing anything | 19:34 |
| rlandy | also - we need to make sure we are running podman | 19:34 |
| rlandy | was not in fs | 19:34 |
| rlandy | adding | 19:34 |
| weshay | khttp://logs.openstack.org/44/636144/3/check/tripleo-ci-centos-7-scenario001-standalone/f8e689b/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz | 19:35 |
| rlandy | ?? | 19:35 |
| weshay | rlandy, pacemaker is in the other standalone jobs | 19:35 |
| weshay | but not doing anything afaik | 19:36 |
| rlandy | ack | 19:36 |
| rlandy | seems strange though | 19:36 |
| weshay | I don't think that's ur problem | 19:36 |
| rlandy | weshay: ? | 19:37 |
| weshay | rlandy, it's probably a good question for marios | 19:39 |
| rlandy | weshay: well, starting my ensuring podman and adding pacemaker - will see what shakes out then | 19:39 |
| rlandy | weshay: hmm ... https://logs.rdoproject.org/04/18804/1/check/tripleo-ci-reproducer-fedora-28-host/0dd87db/job-output.txt.gz#_2019-02-11_19_38_49_024152 | 19:56 |
| rlandy | weshay: may be better to figure out why rest fails | 19:57 |
| *** holser_ has joined #oooq | 20:00 | |
| weshay | rlandy, task does not support when conditional" | 20:02 |
| weshay | is the real error | 20:02 |
| weshay | rlandy, ok.. I think it can work as is.. I just need epel installed first | 20:08 |
| rlandy | weshay; k - better | 20:18 |
| rlandy | weshay: wrt internal sf ... it's still not clear to me where I would submit job definition | 20:25 |
| rlandy | I see where I would add a nodepool static driver | 20:25 |
| weshay | rlandy, don't we want a new repo for our jobs there? | 20:25 |
| weshay | like rdo-jobs? | 20:26 |
| rlandy | weshay: also - I didn;t get of we could call playbooks from upstream | 20:26 |
| rlandy | I don;t want to redefine everything here | 20:26 |
| rlandy | we would need rdo repos | 20:26 |
| rlandy | can't say I am clear after today's meeting about our next steps | 20:27 |
| weshay | rlandy, I would think that ... the internal sf.. could have the same relationship w/ upstream zuul | 20:27 |
| weshay | as rdo has | 20:27 |
| weshay | rlandy, that's fine.. we're only doing research atm | 20:27 |
| rlandy | maybe I'll ping rdo guys separately | 20:27 |
| rlandy | I know -but I'd like the roadmap to be clear | 20:27 |
| weshay | +1 | 20:28 |
| weshay | rlandy, can you check if rdo is up for you? | 20:29 |
| weshay | nvrmind | 20:29 |
| rlandy | weshay: yeah, no worries, I will compile my list of questions and send them | 20:31 |
| * rlandy just needs something to wry about while I wait for standalone results | 20:31 | |
| rlandy | worry | 20:31 |
| weshay | talk to nhicher | 20:32 |
| *** quiquell|off is now known as quiquell|rover | 20:35 | |
| weshay | quiquell|rover, hey hey.. afaict.. on centos.. the docker group is changing to dockerroot in the latest rpms | 20:37 |
| quiquell|rover | weshay: I remember someting yes | 20:37 |
| quiquell|rover | weshay: But we should not change the group if daemon.json have already one | 20:38 |
| weshay | we may need to host the rrcockpit outside of rdo.. lolz | 20:38 |
| weshay | quiquell|rover, agree | 20:38 |
| weshay | taking care of that now | 20:38 |
| quiquell|rover | weshay: wait, you talking about rrcockpit or reproducer ? | 20:38 |
| *** gouthamr has quit IRC | 20:38 | |
| quiquell|rover | weshay: well we can use alpine directly | 20:38 |
| quiquell|rover | weshay: requirements are minimal | 20:38 |
| *** dmellado has quit IRC | 20:39 | |
| weshay | quiquell|rover, both.. two diff threads | 20:39 |
| *** hubbot1 has quit IRC | 20:39 | |
| weshay | reproducer, the docker group can be docker or dockerroot | 20:40 |
| weshay | quiquell|rover, and rdo-cloud just went down | 20:40 |
| quiquell|rover | arggg | 20:40 |
| quiquell|rover | make sense to put rrcockpit elsewere we cannot monitor broken stuff from broken stuff | 20:40 |
| quiquell|rover | about group we just have to get he already set user at daemon.json | 20:40 |
| quiquell|rover | and use that | 20:40 |
| quiquell|rover | ansible "slurp" the file and get the value it have | 20:41 |
| quiquell|rover | weshay: what options do we have to put rrcockpit ? It have to be nice for devs | 20:41 |
| weshay | it's tooooooo late | 20:41 |
| weshay | if we do it in ansible | 20:41 |
| weshay | will have to be in both | 20:42 |
| weshay | quiquell|rover, good question re: where else | 20:42 |
| weshay | not sure atm | 20:42 |
| quiquell|rover | what do you mean both ? | 20:42 |
| weshay | the user script and in pre.yaml | 20:43 |
| weshay | anyway... | 20:43 |
| quiquell|rover | ack | 20:43 |
| weshay | bbiab | 20:43 |
| *** weshay has quit IRC | 20:43 | |
| quiquell|rover | wes | 20:46 |
| quiquell|rover | well people can run rrockpit locally | 20:46 |
| quiquell|rover | if RDO is down | 20:46 |
| quiquell|rover | docker-compose up --build | 20:46 |
| quiquell|rover | 255920 | 20:46 |
| rlandy | weshay ... fyi - ... | 20:48 |
| rlandy | <nhicher> rlandy: we have to configure a gerrit connection to review.rdo, when done, jobs define on the review.rdo config repo will be available on sf.hosted zuul | 20:48 |
| rlandy | <nhicher> rlandy: the config repo is https://code.engineering.redhat.com/gerrit/openstack/sf-config | 20:48 |
| rlandy | <nhicher> rlandy: I can't add the connection now, rdocloud outage | 20:48 |
| *** quiquell|rover is now known as quiquell|off | 21:00 | |
| *** weshay has joined #oooq | 21:02 | |
| *** gouthamr has joined #oooq | 21:39 | |
| *** dmellado has joined #oooq | 21:43 | |
| *** hubbot1 has joined #oooq | 21:48 | |
| *** trown is now known as trown|outtypewww | 22:00 | |
| *** holser_ has quit IRC | 22:18 | |
| *** holser_ has joined #oooq | 22:24 | |
| *** holser_ has quit IRC | 22:48 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.openstack.org/564291 | 22:55 |
| *** rascasoft has quit IRC | 23:10 | |
| weshay | rlandy, fyi.. | 23:14 |
| weshay | https://review.openstack.org/#/c/636028 | 23:14 |
| weshay | https://review.rdoproject.org/r/#/c/18804/ | 23:15 |
| weshay | https://review.openstack.org/#/c/636230/ | 23:15 |
| *** dtantsur|afk has quit IRC | 23:46 | |
| *** honza has quit IRC | 23:46 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!