Wednesday, 2021-07-28

*** ykarel|away is now known as ykarel04:44
*** marios is now known as marios|ruck05:49
marios|ruckneeds reviews please https://review.opendev.org/c/openstack/tripleo-ansible/+/80161506:45
soniya29|roverpojadhav has power failure issues hence she will resume the work once power is back06:47
*** amoralej|off is now known as amoralej07:07
*** ykarel is now known as ykarel|lunch08:34
soniya|roverkopecmartin, chandankumar, do you have anything to discuss for today's tempest meeting?09:06
*** zbr is now known as Guest254409:09
marios|ruckneed vote here please https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/801736 test at https://review.opendev.org/c/openstack/tripleo-heat-templates/+/802618 (scen4 tested)09:37
marios|ruckthank you sshnaidm|afk chandankumar 09:40
pojadhavmarios|ruck, 0/09:44
*** sshnaidm|afk is now known as sshnaidm09:45
pojadhavfor the c7 updates ... do we are care about any release other than train and queens? I got comments on patch : https://review.rdoproject.org/r/c/rdo-jobs/+/34632 to avoid changes of job names for pike and rocky as its EOL.09:46
*** pojadhav is now known as pojadhav|brb09:53
marios|ruckpojadhav|brb: queens is also eol now :)09:59
kopecmartinsoniya|rover: nothing specific10:00
marios|ruckpojadhav|brb: https://opendev.org/openstack/releases/commit/37d51ffe3b28b8680572a9cbc91971bd93d82b2d 10:00
marios|ruckpojadhav|brb: so just train 10:00
*** ykarel|lunch is now known as ykarel10:17
soniya|roverkopecmartin, chandankumar, we are cancelling today's tempest meeting since weshay|ruck, ysandeep|PTO, and arxcruz are on PTO and nothing much to discuss.10:52
chandankumarsoniya|rover: ack10:52
*** pojadhav|brb is now known as pojadhav11:03
pojadhavmarios|ruck, so i will revert the change made for pike, rocky and queens and only keep train job name changes.11:04
pojadhavfor c-7 updates11:05
marios|ruckpojadhav: which change? 11:10
marios|ruckpojadhav: maybe don't need revert just fixup with new patch? 11:10
pojadhavmarios|ruck, https://review.rdoproject.org/r/q/topic:%2522refactor_job_names%2522+(status:open+OR+status:open)11:11
pojadhavc-7 job names to match with upstream (FS37, FS30 and FS10)11:11
pojadhavmarios|ruck, yeah.. will upload new patchset.11:12
pojadhavmarios|ruck, also i got zuul error : https://review.rdoproject.org/r/c/rdo-jobs/+/34632/1#message-283f0496b1a9dd0868d2c3951d04b847b606851a11:12
pojadhavwhat should be the job name after refactoring.. actually I didnt get it.11:13
marios|ruckpojadhav: you're using the same name exactly 11:14
marios|ruckpojadhav: that is the problem 11:14
pojadhavmarios|ruck, okay11:16
marios|ruckme afk few mins brb11:26
*** rlandy is now known as rlandy|ruck11:52
rlandy|ruckmarios|ruck: soniya|rover: hey - all set for the program call?11:53
marios|ruckrlandy|ruck: o/ 11:53
rlandy|ruckI'm joining but I'll let you talk11:53
marios|ruckrlandy|ruck: sure, ok will do11:53
marios|ruckrlandy|ruck: thanks for following through on train 11:53
marios|ruck\o/11:53
marios|ruckrlandy|ruck: i mental high fived you this morning ;) 11:53
rlandy|ruckmarios|ruck: wrt downstream - I marked 16.2 yellow because we got a promotion just befire outage11:53
marios|ruckrlandy|ruck: why don't you give status?11:54
rlandy|ruck17 is red as we didn't get a  promotion on friday and now it's out of water11:54
marios|ruckrlandy|ruck: upstream not much to mention so you can talk about dstream more11:54
marios|ruckrlandy|ruck: if you prefer not to i will fine no prob11:54
rlandy|ruckmarios|ruck: ^^ that's really all there is to say11:54
rlandy|ruckyou go ahead11:54
marios|ruckrlandy|ruck: ack 11:54
rlandy|ruckI'll say something if need be11:54
marios|ruckrlandy|ruck: k 11:54
soniya|roverrlandy|ruck, hello11:58
rlandy|rucksoniya|rover: hey - all ok in ruck/rover land?11:59
chandankumarrlandy|ruck: bhagyashris https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34688 and https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34273 When free. thanks !11:59
soniya|roverrlandy|ruck, yeah11:59
chandankumarmarios|ruck: against this https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/790926 https://338e746b16b3007d4bcf-acf78f12e623b559f4ddc7a30082b17f.ssl.cf5.rackcdn.com/790926/13/check/tripleo-ci-centos-8-scenario007-multinode-oooq-container/a47b348/logs/undercloud/var/log/tempest/tempest_run.log tempest test failure unrelated from this patch.12:01
chandankumarseen this?12:01
soniya|rovermarios|ruck, I have fixed the grafana issue, https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3471412:01
chandankumar{0} tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops [18.447043s] ... FAILED -     Details: {'code': 400, 'created': '2021-07-28T10:14:17Z', 'message': "Host 'centos-8-stream-ovh-bhs1-0025672773.localdomain' is not mapped to any cell"}12:01
zbrsshnaidm: do you remember which was the problem that made molecule-podman fail to gather facts?12:02
sshnaidmzbr, yes, container wasn't running12:03
zbrand why it was not running?12:03
sshnaidmzbr, because of various reasons12:04
zbrtake a look at https://paste.openstack.org/show/807759/12:04
zbrbase on previous task I assume container started, so create suceeded.12:04
zbrbut i guess, that it could be the case were a container quits in between12:05
sshnaidmzbr, check if it's running12:05
zbris not, no container running.12:05
sshnaidmso that's it, "podman logs"12:06
sshnaidmpodman logs --latest12:07
zbrnada, but i see that the image was downloaded.12:07
zbrinteresting, running that image manually seems to work as expected. maybe i found another bug in molecule.12:08
pojadhavmarios|ruck, when I search the occurances of "tripleo-ci-centos-7-multinode-1ctlr-featureset010" in rdo-jobs, but this job is not used anywhere, so I think i should remove that job definition. whats your opinion?12:11
pojadhavmarios|ruck, https://review.rdoproject.org/codesearch/?q=tripleo-ci-centos-7-multinode-1ctlr-featureset010&i=nope&files=&repos=12:12
marios|ruckpojadhav: in call will check later12:14
pojadhavmarios|ruck, yeah sure..12:14
zbrsshnaidm: very weird thing, i identified the exact command used by podman driver and run it myself, it starts a container as I get its long id but when I do "podman ps", nothing, even using logs and container numebr show nothing, is like that execution never happened.12:24
zbrpodman run -d --name centos8 --hostname=centos8 quay.io/pycontribs/centos:stream812:26
zbrwhen I try to run again it fails complaining that name is already in used by the container that is no longer running.12:26
zbrhttps://paste.openstack.org/show/807760/12:27
sshnaidmpodman ps -a ?12:29
sshnaidmprobably one with same name failed, but didn't remove12:30
zbrwell, i did clean them manually but they still fail to really start. even in foreground.12:30
zbri am now trying another image12:31
*** pojadhav is now known as pojadhav|afk12:31
zbrconclusion so far is that if another container with the same name previously run, it will not run again. not sure why the error is not seen when using molecule.12:38
zbrwhen i run podman from cli, i can see the error like  error creating container storage: the container name "centos8" is already in use by12:38
sshnaidmto remove all existing containers: podman rm -fa12:38
sshnaidmprobably in molecule it's not processed right12:39
sshnaidmif it checks if container exists - it does12:39
sshnaidmbut not running12:39
zbri suspect that this line may not do what it is supposed to do https://github.com/ansible-community/molecule-podman/blob/master/lib/molecule_podman/playbooks/create.yml#L9912:40
zbrnow i know, local version of the driver is 0.3.0, which does not have this cleanup code.12:42
zbrhttps://github.com/ansible-community/molecule-podman/blob/0.3.0/lib/molecule_podman/playbooks/create.yml12:42
zbri do not see any cleaning here.12:42
zbrbut master seems to have it, time to check if it works with that.12:42
zbrsshnaidm: basically what happens is that podman reports container as starting, reporting its it but that one instantly disappears. when I try to use log on it, it does list nothing.12:53
zbrthe stopped container is there but i have no explanation why it stopped.12:53
zbrand i see no reason why this container should run for less than second: https://gist.github.com/ssbarnea/80941778d05dbd9060f6a446a8ca951c#file-gistfile1-txt-L82 -- i also included inspect report on it.12:57
sshnaidmzbr, maybe in journal something you can see13:00
sshnaidmzbr, look at exit code in inspection "podman inspect centos8"13:01
rlandy|rucksoniya|rover: still having the tempest meeting?13:01
soniya|roverrlandy|ruck, nope13:01
soniya|roverwe have cancelled it13:01
rlandy|ruckah I see email13:01
*** amoralej is now known as amoralej|lunch13:16
rlandy|ruckmarios|ruck: hey - gate stuck?13:22
rlandy|rucktripleo-ci-centos-8-scenario000-multinode-oooq-container-updates13:22
rlandy|ruckone job queued13:23
rlandy|ruckholding all the jobs that are done13:23
marios|ruckrlandy|ruck: looking13:23
rlandy|ruckping openstack-infra?13:23
marios|ruckrlandy|ruck: yeah... but they will likely just tesll us to abandon it... 13:24
marios|ruckrlandy|ruck: urgh... didn't notice it. it has been in the gate since last night 13:24
marios|ruckrlandy|ruck: if i abandon it will kill all the things below it but... 13:24
marios|ruckwe may just have to do that 13:24
marios|ruckrlandy|ruck: man that is horrible there are a lot of green runs below it :(13:25
rlandy|ruckmarios|ruck: I don't think we have choice unless infra can help us out13:25
rlandy|ruckgo4it13:26
rlandy|ruckoh I see you chatting with infra13:26
marios|ruckrlandy|ruck: well lets see if folks say something give it couple mins13:27
marios|ruckrlandy|ruck: otherwise we can reset :/13:27
rlandy|ruckack13:27
rlandy|ruckno other way out13:27
rlandy|ruckzbr: did you get the infrared issue sorted?13:33
marios|ruckthanks rlandy|ruck++ completely missed that ;)13:33
*** amoralej|lunch is now known as amoralej13:38
rlandy|ruckugh ... here we go ... https://artifacts.ci.centos.org/rdo/jenkins-tripleo-quickstart-promote-victoria-current-tripleo-delorean-minimal-100/undercloud/home/stack/undercloud_install.log13:43
rlandy|ruckThe 'validation_id' argument should be a List13:44
rlandy|ruckdidn't we fix that??13:44
rlandy|ruckmarios|ruck: ^^ that error look familiar?13:45
rlandy|ruckchandankumar: ^^ maybe from your shift13:45
rlandy|ruckon ci.centos jobs13:45
rlandy|ruckfailing victoria undercloud install there13:46
chandankumarrlandy|ruck: it is from validation 13:46
chandankumarwe fixed it13:46
rlandy|ruckyeah - still failing ci.centos - weird13:46
marios|ruckrlandy|ruck: havent come across that one13:46
rlandy|ruckmarios|ruck: yeah - I remember it from last week13:46
rlandy|ruckbut it's in the ci.centos line13:47
rlandy|ruckso we are missing something there13:47
chandankumarmarios|ruck: https://review.opendev.org/q/efdefd92de54d2dd0803f0367bf89bb57792703713:47
rlandy|ruckussuri also failing13:47
chandankumarrlandy|ruck: ^^13:47
chandankumarhttps://bugs.launchpad.net/tripleo/+bug/193646513:47
rlandy|ruckchandankumar: thank ou13:47
rlandy|ruckyou13:47
rlandy|ruckhttps://ci.centos.org/job/tripleo-quickstart-promote-ussuri-current-tripleo-delorean-minimal/187/console ussuri is progressing13:50
rlandy|ruckmaybe victoria needs an update13:50
rlandy|ruckchecking 13:50
marios|ruckthanks chandankumar 13:52
rlandy|ruckpython3-tripleoclient.noarch                  14.2.2-0.20210702232255.79637b0.el8       @delorean-component-tripleo13:57
rlandy|ruck^^ old13:57
rlandy|ruckpython-tripleoclient-14.2.2-0.20210702232255.79637b0.el8.src.rpm2021-07-02 23:24 330K 14:02
rlandy|ruckpython-tripleoclient-14.2.3-0.20210722165810.9b3bd5f.el8.src.rpm2021-07-22 16:59 331K 14:02
rlandy|ruckci.centos is probably not taking current on tripleo14:02
rlandy|ruckso we need to check component line there14:02
chandankumarsshnaidm: rlandy|ruck marios|ruck https://review.opendev.org/q/topic:%22remove_uneeded_tq_requirements%22+(status:open%20OR%20status:merged) testreport is here: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/790926/13#message-ac7c5f22f60b1ffac1969c0982c904ad522ed36714:03
chandankumarif more needed, please add comments before merging14:03
chandankumarwe can do the testing14:03
rlandy|ruckthanks - will look in a bit14:03
marios|ruckchandankumar: ack adding to list14:05
sshnaidmchandankumar, commented on https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/79092614:07
chandankumarsshnaidm: done https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/790926/13/playbooks/ovb-setup.yml#1714:08
chandankumarsshnaidm: may be I need to put a tripleo-common changes and also trigger fs02 change there?14:10
sshnaidmchandankumar, yeah, a good idea14:10
chandankumarsshnaidm: let me propose one14:11
sshnaidmI'm fine with this patch, but more tests the better14:11
chandankumarmarios|ruck: rlandy|ruck putting one more changes 14:11
rlandy|ruckhttps://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario004-standalone-tripleo-victoria/25e331a/logs/undercloud/var/log/tempest/stestr_results.html.gz14:11
chandankumarto test more14:11
rlandy|ruckstill failing14:11
rlandy|ruckhttps://opendev.org/openstack/openstack-tempest-skiplist/commit/3bcf24eb9d0e11a0a054944d1d0832468820872b14:11
rlandy|ruckwe need that back14:11
marios|ruckrlandy|ruck: looking14:12
rlandy|ruckmarios|ruck: victoria tripleo pipeline14:13
marios|ruckrlandy|ruck: hm that is from yesterday though https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario004-standalone-tripleo-victoria/25e331a/logs/undercloud/var/log/tempest/14:13
marios|ruckrlandy|ruck: checking when did fix merge 14:13
rlandy|ruckbeen failing for a while14:13
rlandy|ruckthe component job was never in the skip14:15
rlandy|ruckmarios|ruck: ^^ that is blocking ci.centis14:15
rlandy|ruckcentos14:15
marios|ruckrlandy|ruck: fixes merged 26/27 so i think we need to see what happens with today run ? 14:15
rlandy|ruckmarios|ruck: if we promote victoria, it may help14:16
rlandy|ruckcheck will never see it14:16
rlandy|ruckas it takes consistent14:16
marios|ruckrlandy|ruck: well v has been promoting pretty consistently last few days14:17
marios|ruckevery day for a week now :)14:17
marios|ruckrlandy|ruck: let's move the cix back from done though and we can discuss in few mins on the call 14:18
marios|ruckrlandy|ruck: https://trello.com/c/85yCXxqV/2017-cixlp1934879tripleociproa-victoriaussuritrain-c8tempestlibexceptionstimeoutexception-request-timed-out-failing-in-scenario004-st#comment-61016763c9dcb82538ab3bbc 14:19
marios|ruckrlandy|ruck: oh that is a component job? 14:19
marios|ruckrlandy|ruck: i missed that bit14:19
marios|ruckrlandy|ruck: that's why it was failing14:19
marios|ruckrlandy|ruck: i mean thats why it wasn't skipping the test and failing ... 14:21
rlandy|ruckmarios|ruck: so we have two options ...14:24
rlandy|ruckput the skip back in the DNM review14:24
rlandy|ruckrerun the component test14:24
rlandy|ruckpromote component14:24
rlandy|ruckand see if that helps14:24
rlandy|ruckor put the skip back premanently14:24
rlandy|ruckmarios|ruck: ^^ voting for the first to see if we get by that way14:25
marios|ruckrlandy|ruck: yeah we could just put the skip back for the component 14:25
* rlandy|ruck puts in review14:25
marios|ruckrlandy|ruck: will have a look after the cix14:25
marios|ruckrlandy|ruck: ah thank you ronelle !14:25
*** ykarel is now known as ykarel|away14:45
rlandy|ruckfrenzy_friday: hey - you all set with tox/mocking?15:18
frenzy_fridayrlandy|ruck, yeah for now. I'll worry about secrets after pushing the patch with mock15:19
rlandy|ruckfrenzy_friday: sure - ping when ready to work on that15:19
rlandy|ruckok  - so back to ussuri failures15:19
*** chem is now known as Guest259815:36
*** amoralej is now known as amoralej|off15:57
marios|ruckrlandy|ruck: just chasing some ptl stuff so it doesn't slip but will go in a bit16:21
marios|ruckrlandy|ruck: thank you very much for all your help as usual 16:21
rlandy|ruckmarios|ruck: sure - will keep watch :)16:21
*** marios|ruck is now known as marios|out16:38
pojadhavrlandy|ruck, I have updated most of the patches about c7 updates.. we will look first for FS037 and FS10.. please have a look on those patches when you have bandwidth : https://review.rdoproject.org/r/q/topic:%2522refactor_job_names%2522+(status:open+OR+status:open)16:53
pojadhavrlandy|ruck, discussed with marios about releases that needs to care for C-7. he suggested me to care about train only. we will avoid to make change to pike, rocky and queens.16:55
pojadhavso made changes to the patches accordingly.16:56
*** pojadhav is now known as pojadhav|out16:57
rlandy|ruckpojadhav|out: k - I agree17:16
rlandy|ruckwill look later17:16
dviroelrlandy|ruck: hi o/19:16
dviroelrlandy|ruck: when you have a moment19:16
dviroeli was working on https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3461219:17
rlandy|rucksure - in 5 min19:17
dviroelafter some PS, seems to work as I was expecting19:17
dviroelsome code changes were needed to work19:17
dviroelin PS12, i copied the artifacts to the output log dir, just to check, and this was the result: https://logserver.rdoproject.org/12/34612/8/check/tripleo-ci-compose-promote-latest-compose-to-tripleo-ci-testing/32d72d7/19:18
dviroels/PS12/PS819:19
dviroellooking at ci-config repo, i saw that the right thing is to add a molecule job, to have a job testing this code19:20
dviroelwhich will replace the current job that I added in layout.yaml for now19:20
dviroelmy guess is that it will work with secrets too, according to some tests that I did in my local zuul, using a 'ssh-add' to add a private-key to the ssh-agent19:21
dviroelwhich looks like the process that other jobs in 'config' are doing19:22
* dviroel needs to study how molecule jobs works19:22
* rlandy|ruck reads back19:23
rlandy|ruckhttps://logserver.rdoproject.org/12/34612/8/check/tripleo-ci-compose-promote-latest-compose-to-tripleo-ci-testing/32d72d7/centos8/19:24
rlandy|ruckdviroel: ^^ result looks good!19:24
dviroel:)19:24
dviroelwe can add more artifacts as needed when we start to consume 19:24
rlandy|ruckdviroel: so you're ready for the config/secrets job?19:25
rlandy|ruckmolecule job is definitely preferred19:26
rlandy|rucknot required to start but good to have19:26
rlandy|ruckdviroel: do you think this is ready to merge?19:27
rlandy|rucknot that I would do that now19:27
rlandy|ruckjust checking for approx readiness19:27
dviroelthe only thing that is missing for config/secrets is to do the ssh key configuration for the job, like this 19:27
dviroelhttps://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/config/update_local.yaml#L3-L2019:27
dviroelrlandy|ruck: i would think on adding a couple of unit tests and the molecule job to get it merged19:28
rlandy|ruckdviroel: suee19:28
rlandy|rucksure19:28
rlandy|ruckgo there first19:28
rlandy|ruckthen we can merge19:28
rlandy|ruckand do the secrets bit19:28
rlandy|ruckbut looking good - thank you19:28
dviroelthis part of adding secret keys, is something that shoud be at 'config' right?19:29
rlandy|rucklooking at the ssh key piece ...19:29
rlandy|ruckhttps://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/config/update_local.yaml#L3-L2019:30
rlandy|ruckwe'd probably want to switch case that 19:30
rlandy|ruckif not secret is defined19:30
rlandy|ruckif it is, go with the secret19:30
dviroelat 'ci-config'?19:30
rlandy|ruckyeah19:31
rlandy|ruckand we will create another job in config19:31
rlandy|ruckparented off the one in cic-onfig19:31
dviroelan then in 'config' we just use the as parent job and add the secret?19:31
rlandy|ruckci-config19:31
dviroelack19:31
rlandy|ruckwith the secret added19:31
rlandy|ruckI think that will work19:31
rlandy|ruckbut I think we will need the job merged to try that in config19:32
rlandy|ruckbecause of the no speculative testing condition19:32
rlandy|ruckwe'll have to code, merge, try19:32
dviroelok, as long as I have everything in ci-config, i can use molecule jobs to help on validate19:33
rlandy|ruckyeah - that;s the safest approach19:33
dviroelrlandy|ruck: great, thanks19:33
rlandy|ruckwhat we were doing before, with a script19:33
rlandy|ruckis what wes wanted to get away from19:34
dviroel+119:34
rlandy|ruckto do eaxctly what you are suggested19:34
rlandy|ruckself contained, testable19:34
rlandy|ruckand min related to a real server upload19:34
rlandy|ruckwe can touch base on this again at tomorrow's scrum19:35
dviroelsure, good idea, i'll need a feedback from the team for the ansible/roles/module approach too19:36
dviroelwill update the task with some links19:37
rlandy|ruck++19:45

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!