Thursday, 2021-06-10

*** bhagyashris_ has joined #oooq01:12
*** bhagyashris has quit IRC01:20
*** whayutin_ has joined #oooq01:31
*** weshay|ruck has quit IRC01:38
*** ysandeep has joined #oooq02:38
*** ysandeep has quit IRC02:52
*** whayutin_ has quit IRC03:27
*** whayutin has joined #oooq03:31
*** ykarel has joined #oooq04:11
*** marios has joined #oooq05:00
*** marios is now known as marios|ruck05:01
*** whayutin has quit IRC05:03
pojadhavmarios|ruck, when you free pls have a look.. https://review.opendev.org/c/openstack/tripleo-ci/+/794571 thanks :)05:23
pojadhavmarios|ruck, and do we need to test each upgrades job added in periodic..?05:24
marios|ruckpojadhav: i voted, cant spot any more nits lgtm05:31
marios|ruckpojadhav: don't think you need to post testprojects for that if this is what you mean (though it won't hurt to do it) ... but after it merges you should chck that they are indeed all running in the periodic line then fix accordingly if there are issues05:32
*** ysandeep has joined #oooq05:42
*** jpodivin has joined #oooq05:51
bhagyashris_marios|ruck, hi, i see you made all the changes using centos user06:05
bhagyashris_but we use the promoter user for promotions06:06
bhagyashris_marios|ruck, fyi i am doing the promotions one by one06:06
bhagyashris_currently06:06
bhagyashris_i am doing the ussuri promotion06:06
bhagyashris_once it finishes then i will start the c8 train promotion06:07
*** bhagyashris_ is now known as bhagyashris06:08
marios|ruckack ok thanks so you mean we have a user called 'promoter' which we should use instaed of centos?06:09
marios|ruckbhagyashris: ^^06:09
bhagyashrismarios|ruck, yes06:10
marios|ruckbhagyashris: ack06:10
bhagyashrismarios|ruck, have you restarted the service06:10
bhagyashris?06:10
marios|ruckbhagyashris: no i did not06:10
bhagyashrismarios|ruck, thanks06:10
marios|ruckbhagyashris: i just changed the criteria files06:10
bhagyashrisok np06:11
marios|ruckbhagyashris: thank you06:11
bhagyashrismarios|ruck, if you want to see the changes just ssh to promoter server and switch to promoter user 1. $ sudo su - 2.  $su - promoter 3. join tmux session $tmux a06:12
bhagyashrisi doing all the stuff there06:12
marios|ruckbhagyashris: thanks06:12
bhagyashrisfyi ussuri promotion is goin on here : http://38.102.83.109/promoter_logs/centos8_ussuri.log , http://38.102.83.109/promoter_logs/container-push/20210610-053218.log06:14
marios|ruckbhagyashris:ack06:15
*** soniya29 has joined #oooq06:16
*** jpena|off is now known as jpena06:58
*** jbadiapa has quit IRC07:02
*** jbadiapa has joined #oooq07:02
*** tosky has joined #oooq07:03
*** amoralej|off has joined #oooq07:04
*** frenzy_friday is now known as anbanerj|rover07:04
*** amoralej|off is now known as amoralej07:06
marios|ruckanbanerj|rover: o/ so i am catching up still07:10
marios|ruckanbanerj|rover: if you want to do the same and we can sync ina  couple of hours once you have a chance to check things out?07:11
anbanerj|roverHey marios|ruck, I am going through the CIX board then to see if there are updates on cards/patches07:11
anbanerj|roveryep, sure07:11
marios|ruckanbanerj|rover: ack yeah CIX sounds good07:11
marios|ruckanbanerj|rover: just started looking at the fs2 wallaby issue07:11
marios|ruckanbanerj|rover: far as i can see upstream seems to be a bit better today, although long queue but nothing stuck as far as i can see07:12
*** soniya29 has quit IRC07:13
*** soniya29 has joined #oooq07:14
anbanerj|rovermarios|ruck, for bug https://bugs.launchpad.net/tripleo/+bug/1921400 should we add TestNetworkBasicOps.test_connectivity_between_vms_on_different_networks to skiplist or tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_mtu_sized_frames? Are they both same? (https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/795725/2/roles/validate-tempest/vars/tempest_skip.yml)07:44
opendevmeetLaunchpad bug 1921400 in tripleo "TestNetworkBasicOps.test_connectivity_between_vms_on_different_networks in master fails on auth" [Critical,Triaged]07:44
marios|ruckanbanerj|rover: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/79572507:49
anbanerj|rovermarios|ruck, here we have mtu_sized_frames in skiplist. Do we need  TestNetworkBasicOps.test_connectivity_between_vms_on_different_networks as well?07:50
marios|ruckanbanerj|rover: i don't think so, that bug was filed a long time ago and the test_connectivity_between_vms_on_different_networks was failing then but not any more?07:52
marios|ruckanbanerj|rover: i guess you can check that to confirm ?07:52
anbanerj|roveryep, lemme testP07:52
*** derekh has joined #oooq08:06
anbanerj|rovermarios|ruck, I am checking https://trello.com/c/wfZF9hvY/1991-cixlp1931428tripleociproa-sc10-sc10-kvm-sc010-ovn-provider-failing-with-pymysqlerrprogrammingerror-1146-table-octaviaquotas-does with testP08:47
marios|ruckanbanerj|rover: ack08:48
marios|ruckanbanerj|rover: filed fs2 wallaby new bug there https://bugs.launchpad.net/tripleo/+bug/1931545 (testing fix with https://review.rdoproject.org/r/c/testproject/+/34057 )08:49
opendevmeetLaunchpad bug 1931545 in tripleo "periodic featureset 002 failing tempest "Non subunit content"" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)08:49
anbanerj|rovermarios|ruck, ack08:49
*** frenzy_friday has joined #oooq09:51
*** frenzy_friday has left #oooq09:51
*** er-bot has joined #oooq09:52
*** er-bot has left #oooq09:53
*** pojadhav is now known as pojadhav|brb10:48
*** jpena is now known as jpena|lunch11:37
*** ysandeep is now known as ysandeep|afk11:44
*** rlandy has joined #oooq11:45
*** soniya has joined #oooq11:47
*** pojadhav|brb is now known as pojadhav11:51
*** soniya29 has quit IRC11:53
*** ysandeep|afk is now known as ysandeep12:11
rlandymarios|ruck: anbanerj|rover: just FYI .. I'd like to merge https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/79458412:12
rlandyjust fyi - in case you see fallout12:12
anbanerj|rovermarios|ruck, https://bugs.launchpad.net/tripleo/+bug/1930811 - this has a fix, but a lot of gate jobs are still failing with "Could not clean up: 'ClientManager' object has no attribute 'sdk_connection'"12:12
opendevmeetLaunchpad bug 1930811 in tripleo "After successfull deploy we are noticing message: "Could not clean up: 'ClientManager' object has no attribute 'sdk_connection'"" [Low,Triaged]12:12
rlandyI don;t expect any12:12
rlandybut pls take look just in case12:12
marios|ruckrlandy: thanks12:13
*** whayutin has joined #oooq12:14
marios|ruckanbanerj|rover: can you please add comment on the bug with pointers to the gate fails? I didn't see those (I saw some other gate fails but didn't come across that one)12:14
anbanerj|rovermarios|ruck, ok12:14
rlandyysandeep: so ... 16.2 is back in action12:14
rlandyysandeep: working on the multinode failure there12:15
ysandeeprlandy: ack o/12:16
marios|ruckanbanerj|rover: can you please file some new bugs - I am digging into promotions (started with train, since we need to promote it before wednesday for program call)12:23
marios|ruckanbanerj|rover: in particular those two fs35 https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train12:24
marios|ruckanbanerj|rover: seems to consistently time out12:24
anbanerj|rovermarios|ruck, ack12:24
marios|ruckanbanerj|rover: and then another bug for scenario010 https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-train https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario010-standalone-train12:24
marios|ruckanbanerj|rover: they are both promotion blockers since those jobs are in criteria (not sure how we promoted 6 days ago for train though but maybe they were waived?)12:25
marios|ruckanbanerj|rover: i can't find bugs for those so I think we need 2 new promotion blockers (fs35, scen10)12:25
marios|ruckanbanerj|rover: thank you12:25
anbanerj|rovermarios|ruck, ack. Adding them12:25
marios|ruckanbanerj|rover: thank you12:26
zbri do remember that at some point we accidentally exposed some secrets on CI due to lack of no_log on a task. Can someone point me to a place where this happened?12:28
*** jpena|lunch is now known as jpena12:34
*** amoralej is now known as amoralej|lunch12:34
rlandyhmmm .. I remember we did that once in a bug report12:34
marios|ruckysandeep: very nice job the ruck_rover.py is quite useful thank you!12:48
marios|ruckysandeep++12:48
marios|ruckwhayutin: ^^^ we need to hire this guy!12:48
ysandeepmarios|ruck, are you using that?12:49
marios|ruckysandeep: i used it today to check the failing criteria jobs yes12:49
ysandeepmarios|ruck, improvement coming, now that will give logs also of failing job in console itself12:49
marios|ruckysandeep: nice12:50
rlandymarios|ruck: talking with pojadhav .. ok to merge https://review.opendev.org/c/openstack/tripleo-ci/+/794571?12:51
marios|ruckrlandy: checking12:51
marios|ruckrlandy: no ysandeep is right we should add the update jobs too didn't notice them missing12:52
marios|ruckrlandy: we could merge and followup if pojadhav prefers12:52
marios|ruckrlandy: i can revote if so let me know i just -1 it12:53
rlandymarios|ruck: I think so - so we can get this in12:53
anbanerj|roverhey marios|ruck I added this https://bugs.launchpad.net/tripleo/+bug/1931573 . Could you pls add me to the group so that I can change it to Triaged?12:53
opendevmeetLaunchpad bug 1931573 in tripleo "periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train fails with No matching distribution found for ara==1.0.0" [Undecided,New]12:53
rlandywill comment12:53
marios|ruckrlandy: k re-voted12:54
marios|ruckanbanerj|rover: looking12:55
marios|ruckanbanerj|rover: do we have more than one example of that?12:55
marios|ruckanbanerj|rover: far as i can see that happened once on the 8th https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train in general we don't file bugs unless we have at least 2 examples12:56
bhagyashrisarxcruz, dviroel anbanerj|rover pojadhav marios|ruck soniya zbr sshnaidm rlandy ysandeep whayutin12:57
marios|ruckanbanerj|rover: if there are more examples can you please add to the bug12:57
bhagyashrisSprint 46 Planning meeting12:57
marios|ruckanbanerj|rover: and ack i will dig about the group thing12:57
bhagyashrishttps://hackmd.io/P5sTPMSBREe5kBS_4jDhIg#2021-06-10-Unified-Sprint-46-Planning @line 16012:57
anbanerj|rovermarios|ruck, lemme check12:57
marios|ruckanbanerj|rover: for fs35 i mean we need a new bug for the *timeout* that happens last few runs12:58
marios|ruckanbanerj|rover: generally though when filing a bug you should always have at least 2 examples there (otherwise we can't say if it is just a one-off or soemthing passing that was fixed)12:59
anbanerj|rovermarios|ruck, ah, ok. No it seems the ara thing was one time. Can I delete a bug? Or should I mark invalid?12:59
marios|ruckanbanerj|rover: yes please12:59
marios|ruckanbanerj|rover: lets mark invalid (add some comment about it being one off though please)12:59
anbanerj|rovermarios|ruck, ok, thanks12:59
bhagyashrishey folks meeting time13:00
bhagyashrisrlandy, whayutin pojadhav13:02
bhagyashris^13:02
*** ysandeep is now known as ysandeep|mtg13:08
*** amoralej|lunch is now known as amoralej13:23
rlandywhayutin: no go on multinode with new network component on 16.214:01
amoralejrlandy, last centos9 compose is broken, in case you notices something weird14:01
rlandyrunning again now14:01
whayutinya.. I saw :(14:02
anbanerj|rovermarios|ruck, periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-train and periodic-tripleo-ci-centos-8-scenario010-standalone-train are hitting https://bugs.launchpad.net/tripleo/+bug/188108714:02
opendevmeetLaunchpad bug 1881087 in tripleo "tripleo-ci-centos-8-scenario010-standalone timing out frequently" [High,Triaged]14:02
rlandyamoralej: thanks for heads up14:02
anbanerj|rovershould I create a new bug or just comment on ^ ?14:02
marios|ruckanbanerj|rover: let me check gimm few14:06
marios|ruckanbanerj|rover: so the issues I saw today weren't timeout it was a fail so I am leaning towards new bug14:07
marios|ruckanbanerj|rover: examples there14:07
marios|ruckanbanerj|rover:         * https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-train/1f41f96/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz14:07
marios|ruck        * https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-train/d47863d/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz14:07
marios|ruck17:02 < anbanerj|rover> marios|ruck, periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-train and periodic-tripleo-ci-centos-8-scenario010-standalone-train are hitting https://bugs.launchpad.net/tripleo/+bug/188108714:08
opendevmeetLaunchpad bug 1881087 in tripleo "tripleo-ci-centos-8-scenario010-standalone timing out frequently" [High,Triaged]14:08
marios|ruckanbanerj|rover: what are you basing that on ^^^ did you see somehing in the logs ?14:08
anbanerj|rovermarios|ruck, https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-train/1f41f96/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz and https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-t14:10
anbanerj|roverrain/d47863d/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz14:10
anbanerj|roverIn job output it fails saying standalone deploy failed. In standalone deploy log it looks like stuck at  Configure octavia on overcloud14:11
cgoncalvesanbanerj|rover, possibly https://bugs.launchpad.net/tripleo/+bug/188108714:11
opendevmeetLaunchpad bug 1881087 in tripleo "tripleo-ci-centos-8-scenario010-standalone timing out frequently" [High,Triaged]14:11
anbanerj|roverIn octavia ansible log https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-train/1f41f96/logs/undercloud/home/zuul/standalone-ansible-sqcp9dzp/octavia-ansible/octavia-ansible.log.txt.gz it looks like https://bugs.launchpad.net/tripleo/+bug/188108714:11
marios|ruckanbanerj|rover: ok then comment on the bug and let's add promotion blocker to it14:12
marios|ruckanbanerj|rover: thank you14:12
anbanerj|rovermarios|ruck, Sorry, where do I add promotion blocker?14:12
marios|ruckanbanerj|rover: np in tags14:13
marios|ruckanbanerj|rover: see under the description it says "Tags: alert"14:13
marios|ruckanbanerj|rover: add promotion-blocker to that14:13
anbanerj|roverthanks, putting a comment and adding14:13
marios|ruckanbanerj|rover: thank you14:14
whayutinmarios|ruck, reran the script... the two promotion-blocker bugs were picked up14:18
whayutinyou can now add comments14:19
marios|ruckwhayutin: thanks14:26
ysandeep|mtgbhagyashris: pm14:38
*** soniya has quit IRC14:48
marios|ruckwhayutin: anbanerj|rover: ready whenever you are14:49
anbanerj|rovermarios|ruck, we are on the same meet14:49
marios|ruckanbanerj|rover: k re-joining14:49
marios|ruckwhayutin: anbanerj|rover: https://review.rdoproject.org/r/c/testproject/+/3405715:05
whayutinanbanerj|rover, please vote https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/79574215:06
marios|ruckhttps://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train/0c69bf7/job-output.txt https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train15:15
rlandyError: Unable to find a match: openstack-ironic-python-agent - baremetal component error15:17
rlandyfixing15:28
*** anbanerj|rover has quit IRC15:32
*** frenzy_friday has joined #oooq15:33
*** frenzy_friday is now known as anbanerj|rover15:33
*** ysandeep|mtg is now known as ysandeep15:37
ysandeepwhayutin rlandy: you need hand on 16.2? on df call i heard auto import was extended till this friday only, may be we can confirm that with delivery..15:41
rlandyysandeep: I have the fix for train baremetal15:41
rlandytestprojecting it now15:41
ysandeepwhayutin, rlandy: thanks! please drop me an email if I need to pick something up tomorrow on 16.215:42
whayutink15:42
rlandyysandeep: will do ...15:42
rlandywe may need help on multinode if we don;t crack that today15:43
rlandywill leave you notes15:43
ysandeepthanks!15:43
*** jpodivin has quit IRC15:56
anbanerj|roverwhayutin, you want to discuss abt tempest tests and health dashboard?15:57
whayutinsure15:57
dviroelwhayutin: we have a 1-1s in 2 min15:58
anbanerj|roverhttps://meet.google.com/knh-crwd-vpw15:58
whayutindviroel, can I push like 15min?15:58
anbanerj|roverIf you want we can do it tomorrow as well15:59
dviroelwhayutin: hum, will conflict with lunch and other stuff, i can push to the end of the day - my end of the day :)15:59
*** marios|ruck is now known as marios|out16:02
ysandeepanbanerj|rover, https://review.opendev.org/c/openstack/python-tripleoclient/+/795520 merged i hope ovb will be fixed soon, my test patch passed with depends-on that this morning.16:10
whayutindviroel, ok.. ready16:10
*** jpena is now known as jpena|off16:17
*** marios|out has quit IRC16:21
*** ysandeep is now known as ysandeep|out16:38
rlandywhayutin: you around?16:45
rlandywe're looking at the clients component on 16.216:45
whayutinaye16:47
whayutinrlandy, ^16:47
rlandywhayutin: pls see current-tripleo16:47
rlandyansible-collections-openstack-1.4.1-2.20210519064815.6b3bf3b.el8.src.rpm2021-05-19 02:52 256K16:48
rlandyas opposed to tripleo-ci-testing16:48
rlandyansible-collections-openstack-1.4.1-2.20210608074808.4a1b092.el8.src.rpm2021-06-08 03:52 257K16:48
rlandyhow re the two so far apart?16:48
rlandywhayutin: we're chatting on #rhos-delivery16:49
rlandyand request is for updated clients16:49
rlandybut alan says pinned in rdoinfo16:49
whayutinhttps://github.com/redhat-openstack/rdoinfo/blob/master/tags/train.yml#L216:51
whayutindoesn't seem to be16:51
whayutinrlandy, is something failing?16:51
rlandyclients16:51
rlandywhayutin: no16:51
whayutinI don't think that package is even used in 1616:52
rlandyidk if 16.2 is promotable16:52
whayutinhttps://meet.google.com/knh-crwd-vpw?pli=1&authuser=116:52
whayutin?16:52
*** amoralej is now known as amoralej|off16:56
*** derekh has quit IRC16:59
*** amoralej|off has quit IRC16:59
rlandyhttps://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/79583817:06
whayutinrlandy, forget bm17:20
whayutinsee my notes in the doc...  at least for now17:20
rlandythe not built note?17:21
rlandywhayutin: either way we need to fix the image build with component17:21
rlandyso goo dwe found it17:21
*** ykarel has quit IRC17:22
whayutin++17:22
rlandy2021-06-10 14:36:12.528 176641 CRITICAL tempest [-] Unhandled error: config_tempest.services.base.ServiceError: Request on service 'load-balancer' with url 'http://192.168.24.1:9876' failed with code 50017:40
rlandyweird17:40
*** ysandeep|out has quit IRC17:43
rlandyhttps://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario010-standalone-train17:44
rlandy^^ hasn't passed in ages17:44
rlandy021-05-21 07:59:1917:44
whayutinrlandy, so unworkflow https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/795838 right?17:51
rlandywhayutin: I think so 0 until we see both tests go through image build17:52
rlandywill ping you when the second one does17:52
whayutinrlandy, ur looking at the wrong job17:52
rlandywhayutin: re scenario010 failure ...17:52
whayutinno17:52
whayutinforget scenario01017:52
whayutinI think you are confusing.. featureset010 and containers-multinode17:52
rlandyhttps://bugs.launchpad.net/tripleo/+bug/186610717:52
opendevmeetLaunchpad bug 1866107 in tripleo "tripleo-ci-centos-7-scenario010-standalone jobs are failing tesmpest tests "ServiceError: Request on service 'load-balancer' with url 'http://192.168.24.1:9876' failed with code 500"" [Critical,Fix released]17:52
rlandyno - I know they are both failing17:53
whayutincentos-717:53
rlandywe have seen that error before17:53
rlandyanyways17:53
whayutinrlandy, it's not voting17:53
* rlandy carries on to multinode17:53
rlandyecho '/etc/yum.repos.d/*-component.repo is not a valid yum repo configuration file.' yep - need a fix18:11
rlandyupdated patch under test18:12
rlandywhayutin: hey21:51
rlandyso some investigation on 16.2 multinode22:01
*** yamamoto has quit IRC22:22

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!