Thursday, 2020-10-01

« Wednesday, 2020-09-30 Index Friday, 2020-10-02 »
*** rfolco has joined #oooq00:15
*** rfolco has quit IRC00:41
*** rfolco has joined #oooq01:12
*** Goneri has quit IRC01:23
*** rfolco has quit IRC02:15
*** skramaja has joined #oooq03:27
*** jmasud has joined #oooq03:27
*** ykarel|away has joined #oooq04:12
*** ykarel|away is now known as ykare04:12
*** ykare is now known as ykarel04:12
*** ykarel_ has joined #oooq04:34
*** ykarel has quit IRC04:36
*** marios has joined #oooq04:48
*** ysandeep|away is now known as ysandeep04:51
*** ratailor has joined #oooq05:04
*** ratailor_ has joined #oooq05:10
*** ykarel_ is now known as ykarel05:10
*** ratailor has quit IRC05:12
*** udesale has joined #oooq05:45
*** bhagyashri is now known as bhagyashris06:02
*** udesale has quit IRC06:23
*** udesale has joined #oooq06:23
*** ykarel_ has joined #oooq06:27
*** ykarel has quit IRC06:29
soniya29|ruckbhagyashri centos-8-ovb-3ctlr_1comp-featureset035-train has once got timed_out and skipped, should I wait for the re-run?06:47
soniya29|ruckbhagyashris ^^06:48
bhagyashrissoniya29|ruck, log link?06:50
soniya29|ruckbhagyashris, https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train/2fd8cc5/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz06:55
*** jtomasek has joined #oooq06:56
bhagyashrissoniya29|ruck, looks like it's not consistent wait for next run or you can check on testproject patch06:58
soniya29|ruckbhagyashris, lets wait for next run06:59
bhagyashrissoniya29|ruck, fyi openstack-periodic-integration-stable2 is currently running06:59
bhagyashrisyou will get result soon06:59
bhagyashrishttps://review.rdoproject.org/zuul/status06:59
bhagyashrissoniya29|ruck, fyi it's timed out at Execute tempest test task07:01
*** ykarel_ is now known as ykarel07:07
*** jfrancoa has joined #oooq07:08
*** ratailor_ has quit IRC07:15
*** amoralej|off is now known as amoralej07:17
soniya29|ruckbhagyashris, ack07:29
*** tosky has joined #oooq07:39
*** jbadiapa has joined #oooq07:47
*** jpena|off is now known as jpena07:56
*** akahat has quit IRC08:06
*** akahat has joined #oooq08:08
*** jfrancoa has quit IRC08:14
*** ratailor has joined #oooq08:20
holsermorning08:29
*** jfrancoa has joined #oooq08:31
*** apetrich has joined #oooq08:36
*** derekh has joined #oooq08:42
*** jfrancoa has quit IRC08:43
holsermarios thanks for reviews08:46
holserI have corrected https://review.opendev.org/#/c/755402/08:46
holserand will update bug in a moment08:46
*** ratailor has quit IRC08:46
marioso/ ack thanks08:47
mariosholser: ^08:47
holserack08:47
chemmarios: hey, I'm trying to run a reproducer as a background task ...08:51
chemmarios: so I though I go through https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_7ce/755336/4/check/tripleo-ci-centos-8-standalone-upgrade/7ce8d65/logs/README-reproducer.html08:51
chemmarios: but it raised some question08:51
chemmarios: would you have a bit of time at some point, or today isn't a good day ?08:52
*** ratailor has joined #oooq08:52
marioschem: not surprised that hasn't been updated in ages.08:53
marioschem: generally a bit busy but we can find some time, or you can ask here. to be honest i haven't really touched reproducer in a while ... holser was very recently successful with it i should introduce you two ;)08:53
holserchem https://hackmd.io/W0InkA__SsSXmgNmnILFCw08:54
holserchem I don't use cloud for reproducer ... I am too old08:55
chemmarios: holser oki, ... this ssh key business is kinda aweful, why should I have to upload key (and by default new keys) each time I run reproducer08:58
chemmarios: holser this makes no-sense on the usability side of thing08:58
chemby the way, hi holser I'm chem :)08:58
marioschem: holser: i knew you 2 would hit it off08:59
chemmarios: but now, I'm curious, what do you use to debug stuff on live env?08:59
marioschem: so, you could re-use your own keys i mean instead of creating new ones for the zuul reproducer user08:59
marioschem: you only have to re-upload them if you are creating a new user from scratch09:00
chemmarios: well, everything is public, I don't get why it needs key at all09:00
marioschem: have used reproducer for getting jobs setup but for most tasks i just get vm in upshift (used to use rdo cloud)09:01
*** ysandeep is now known as ysandeep|away09:01
chemmarios: if I want to start this from a vm ... I need to create vm keys (my private keys won't go there)09:01
chemmarios: so basically I need to create a new set of key that give write access to my account and put that in a vm that I'm going to forget at some point :)09:02
*** ratailor has quit IRC09:02
chemwhy, why, why09:02
*** ratailor has joined #oooq09:02
holserchem I created keys for reproducer only... all it can do is submit patch to review.opendev.orf09:03
holserorg09:03
chemholser: yeah, that key has the same "power" than your usual key ... and as if the cost of entry wasn't high enought :)09:04
holserwell... you can remove key from review.opendev.org once you are done09:05
chemholser: with another review :)09:05
chemholser: and then another09:05
holsergenerate a new key09:05
chemyeah, classy09:05
chemmarios: holser the other one thing that bothered me is the mention of "docker" all over the place ... is the prefered bootstrap env ubuntu  ?09:05
marioschem: yes09:06
holserchem docker used to bootstrap zuul, nodepool and friends09:06
holseryou can use ubuntu09:06
holseror rhel8 with podman09:06
chemholser: is the reproducer script working on rhel8 ?09:06
holseryes it does09:07
chemholser: ah, so no need of docker09:07
chemoki09:07
marioschem: (i was being sarcastic about ubuntu it isn't clear in irc ;) as i said it hasn't been updated in a long time that readme09:07
marioschem: so it was written when we had docker and centos7 etc09:07
chemmarios: I knew you were :)09:07
chemmarios: by the way, I was too :)09:08
marioschem: OH!09:08
marios;)09:08
chemholser: nevertheless the need for private key upload on rdo/gerrit is a serious show stopper I think ... especially when everything is available without any key09:09
marioschem: i would need to dig but i believe it is needed by zuul to be able to talk to gerrit? sshnaidm around?09:09
mariossshnaidm: there are complaints about the necessity for ssh key in reproducer ^^^09:10
mariossshnaidm: this french guy is complaining about it i don't know what to say09:10
* marios shrugs09:10
sshnaidmsorry, what is the problem?09:10
chemheheh09:10
mariossshnaidm: 12:02 < chem> marios: so basically I need to create a new set of key that give write access to my account and put that in a vm that I'm going to forget at some point :)09:11
mariossshnaidm: 12:09 < chem> holser: nevertheless the need for private key upload on rdo/gerrit is a serious show stopper I think ... especially when everything is available without any key09:11
mariossshnaidm: so basically why do we need to upload the key is it needed for zuul<-->gerrit09:11
sshnaidmwhy do you need to put any keys on vm..?09:11
sshnaidmupload where?09:12
mariossshnaidm: if you are using a new vm to run the reproducer on that and you want to use your own/existing keys instead of creating new ones09:12
sshnaidmoh, I see09:12
chemsshnaidm: I'm dreaming of a working cloud configuration, a script, brew some coffee and get the error in a vm somewhere09:12
sshnaidmwell, when zuul scheduler will have option to get updates by http, we'll switch to it, but afaik it has ssh only09:12
sshnaidmtristanC, am I right? ^09:13
chemsshnaidm: hum ... oki, I haven't been around for a long time ... so excuse the dumb question ... why do I need zuul scheduler to deploy standalone tripleo ...09:15
chemoki -> [ ] :)09:15
sshnaidmchem, if we're talking about zuul reproducer, we set up a full zuul system there09:15
sshnaidmchem, but for standalone only it's really overkill09:15
holserzuul, nodepool :D09:16
marioschem: the idea is once you have a setup you can re-use it and submit jobs to your own local zuul09:16
marioschem: for a 'throwaway' it makes less sense09:16
marioschem: for something you keep it makes more sense09:16
sshnaidmand check also ovb and multinode as well ^09:16
holseronce script for ALL is way better than million throwaway scripts09:17
sshnaidmfor standalone maybe we can come up with something very simple..09:17
holsersshnaidm is it worth?09:18
sshnaidmholser, yeah, but only for standalone, as it doesn't require any other vms to set up09:18
sshnaidmI saw people create various scripts for it, for me it's easier to download standalone configs from any job logs and run the script09:19
holserwell standalone is full of compromises... it doesn't have ha, own upgrade path... so it have very little similar with real deployments09:20
chemsshnaidm: oh, I don't want to add my own request around. I think I miss something here, that's all.  Maybe setting up a gerrit/nodepool/zuul in a semi-permanent vm is worth the trouble if I can reuse easily09:20
chemsshnaidm: holser as for standalone it was just because the vast majority of gerrit upstream are standalone09:20
sshnaidmholser, yeah, we also use it in bootcamps etc to show minimal tripleo deployment09:20
chemsshnaidm: holser  ...  I think (I have the feeling I'm wrong :))09:21
sshnaidmchem, yeah, but as holser mentioned it miss some features09:21
sshnaidmso it depend on your goals09:21
sshnaidmfor full experience I'd rather set up a vm with a zuul-reproducer there and use it for any job, it will be your own CI system :)09:22
chemsshnaidm: oki, now I get the spirit09:22
chemsshnaidm: I think a video teaser would be awesome :)09:23
sshnaidmchem, and you can reuse it always for any job, no need to set up it more than once09:23
sshnaidmchem, yeah, it's a good idea :)09:23
chemsshnaidm: ah, that's sound better and better09:24
chemsshnaidm: so "./reproducer-fresh-box-setup.sh -u marios -p password -c reprozuul" is run only once and then "curl -Lo reproducer-zuul-based-quickstart.tar"; "./reproducer-zuul-based-quickstart.sh --workspace" for each reproducer I want09:25
chemsshnaidm: then everything will be visible in my own private gerrit/zuul experience09:26
chemsshnaidm: oki .... let's give it a shot09:26
sshnaidmchem, after the system is up and running, you need just to send patches to installed gerrit, that's it09:26
marioschem: if you setup your vm (port forwarding etc) you can access you local zuul/gerrit in the browser09:26
chemsshnaidm: oh wait what ?09:27
chemsshnaidm: like i just upload my review to the private gerrit and then it will run the test suite on it ?09:27
sshnaidmgerrit is available on http://localhost:808009:27
chemsshnaidm: is that what reproducer-zuul-based-quickstart.sh  does ?09:28
chemsshnaidm: I think irc has reached its teaser limit, time to get dirty ... I'll set this up and see how it goes09:29
chemsshnaidm++ marios++ holser++ thanks for the feeback ...09:29
chemman no bot here09:30
holserchem yw09:30
sshnaidmchem, ack, also please take a look: https://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer/blob/master/README.md09:30
sshnaidmthere are configs for internal PSI cloud as well09:30
marioschem: np09:30
*** udesale_ has joined #oooq09:30
chemsshnaidm: is this https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_7ce/755336/4/check/tripleo-ci-centos-8-standalone-upgrade/7ce8d65/logs/README-reproducer.html still ok as well ?09:31
sshnaidmchem, yeah, but I like this way less..09:31
*** udesale has quit IRC09:33
*** ratailor has quit IRC10:04
*** ratailor has joined #oooq10:07
*** ysandeep|away is now known as ysandeep|sick10:09
*** ratailor has quit IRC10:09
*** holser has quit IRC10:14
*** ratailor has joined #oooq10:14
*** ratailor has quit IRC10:15
*** ratailor has joined #oooq10:15
*** holser has joined #oooq10:16
*** chem has quit IRC10:52
* pojadhav biab11:16
*** jpena is now known as jpena|lunch11:27
*** ratailor has quit IRC11:42
*** ratailor has joined #oooq11:43
*** ratailor has quit IRC11:44
*** ratailor has joined #oooq11:45
*** ratailor has quit IRC11:49
*** rfolco has joined #oooq11:57
*** rlandy has joined #oooq11:58
*** rlandy is now known as rlandy|rover11:59
rlandy|roversoniya29|ruck: hey - how's it going?12:00
soniya29|ruckrlandy|rover, third party master ussuri centos-8 are all good12:01
rlandy|roversoniya29|ruck: k - have patch in to fix queens12:01
*** amoralej is now known as amoralej|lunch12:01
soniya29|ruckrlandy|rover, train ovb jobs were inconsitently failing..hence waiting for the next run12:02
soniya29|ruckrlandy|rover, ack12:02
* rlandy|rover looks into stein12:02
rlandy|roversoniya29|ruck: rekicked failing ussuri jobs12:08
rlandy|roverhttps://review.rdoproject.org/r/#/c/18953/12:08
rlandy|roverweshay|ruck: can we rekick stein?12:09
rlandy|roverthe whole line12:09
soniya29|ruckrlandy|rover, ussuri is already good12:11
rlandy|rovercurrent run has two failing jobs12:11
rlandy|roverchandankumar: hi ... wrt https://review.rdoproject.org/r/#/c/29809/ - do we still need this review?12:13
rlandy|roveroctavia fix12:13
chandankumarrlandy|rover: nope12:13
rlandy|roverchandankumar: abandon?12:13
chandankumarrlandy|rover: https://review.rdoproject.org/r/#/c/29827/12:13
chandankumarykarel: can you help on rlandy|rover on this ?12:14
rlandy|roverchandankumar++12:14
rlandy|roverthanks12:14
soniya29|ruckrlandy|rover centos-8-ovb-3ctlr_1comp_1supp-featureset039-train is failing consistently and is a promotion blocker as well12:14
chandankumarrlandy|rover: one more patch you need to send12:14
chandankumarrlandy|rover: https://review.rdoproject.org/r/#/c/29827/12:14
chandankumarsorry12:15
chandankumarrlandy|rover: https://github.com/rdo-packages/octavia-tempest-plugin-distgit/blob/rpm-master/python-octavia-tests-tempest.spec#L6912:15
chandankumaradd Requires:       python3-httpx12:15
ykarelchandankumar, i commented on review12:15
chandankumarand then it will be fixed12:15
ykarelit needs more deps12:15
rlandy|roverzbr: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e2a/753778/5/check/tripleo-ansible-centos-8-molecule-tripleo-container-manage/e2ad6d5/job-output.txt12:16
rlandy|roverknow anything about failing molecule12:16
soniya29|ruckrlandy|rover, it is failing to establish authenticated ssh connection to cirros12:16
rlandy|rover "msg": "Failed to validate GPG signature for containerd.io-1.2.6-3.3.el7.x86_64"12:16
rlandy|roversoniya29|ruck: k - I'll look into it after molecule and octavia12:17
zbrrlandy|rover: is not molecule specific, is docker related.12:18
rlandy|roverzbr: yep - Emilien found the issue12:18
zbri seen it yesterday too12:19
zbrhttps://review.opendev.org/#/c/752630/ should fix it but is incomplete12:19
rlandy|roverchandankumar: sorry - got distracted ...12:25
*** jpena|lunch is now known as jpena12:29
akahatmarios, i'll be skipping Promoter meeting.12:31
akahatdo you have something new to discuss?12:32
weshay|ruckcentos-8 train auth is failing to the cirros image https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/9da05da/logs/undercloud/var/log/tempest/stestr_results.html.gz12:34
rlandy|roverweshay|ruck: hey- can we rekick the stein line?12:35
weshay|ruckakahat, are the patches merged or do we need to send folks12:35
soniya29|ruckweshay|ruck, rlandy|rover, where to find more details for failure due to selinux denials tempest test failed issues12:35
weshay|ruckrlandy|rover, if marios thinks standalone-upgrade stein is ready / working12:35
weshay|rucksoniya29|ruck, /me sent an email12:35
rlandy|roversoniya29|ruck: see email12:35
rlandy|roveryep12:35
rlandy|rovermarios: waiting on your call pls ^^12:36
weshay|rucksoniya29|ruck, https://hackmd.io/07z0xroHTFi2IbX93P5ZfQ#Selinux-denials12:36
soniya29|ruckweshay|ruck, ohh yes12:36
weshay|rucksoniya29|ruck, I opened two bugs on it.. if you see a different issue ping12:36
weshay|ruckrfolco, we ready w/ the boards etc?12:37
soniya29|ruckweshay|ruck centos-8 featureset39 train is failing due to it consistently12:37
rlandy|roverweshay|ruck: soniya29|ruck: just putting in octavia review - then will work on fs03912:37
rfolcoweshay|ruck, I'll move to 34 now12:37
mariosweshay|ruck: rlandy|rover: didn't check the stein job but holser said he found something related (the cirros image changes) i need to check not sure it is fixed/merged yet12:37
rfolcoweshay|ruck, agenda is ready12:37
weshay|rucksoniya29|ruck, what's the cause12:37
rlandy|rovermarios: ack - thanks12:38
soniya29|ruckweshay|ruck, selinux denials12:38
weshay|ruckrfolco, can you join #rhos-df  Emilien wants to move the DF call and it overlaps our scrum12:39
weshay|ruckrfolco, our scrum also overlaps the infrared teams scrum12:39
weshay|ruckperhaps we can propose a move12:39
weshay|rucksoniya29|ruck, read my email again12:39
holsermarios tempest failed due to bad image12:39
akahatweshay|ruck, no patches are yet merged: https://review.rdoproject.org/r/#/c/28081/12:40
holserwe need to merge https://review.opendev.org/#/c/755401/  https://review.opendev.org/#/c/755402/12:40
holserthen I'll recheck https://review.opendev.org/#/c/755220/12:40
holserand then stein and train jobs should be fixed12:40
holserussuri is a different story12:41
holserwe mess up networking during upgrade as we run deploy script from ansible operator and upgrade script from quickstart-extra12:42
weshay|ruckholser, ya.. waiting for a +1 from ci12:42
weshay|ruckrlandy|rover, ^ track that stein work.. for a stein promotion.. standalone-upgrade stein is criteria12:43
rlandy|roverweshay|ruck: ack - adding to hackmd12:43
weshay|rucksoniya29|ruck, can you poke at the tempest failures in periodic stable-2 train centos-812:45
weshay|ruckhttps://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/57d77c6/logs/undercloud/var/log/tempest/stestr_results.html.gz12:46
weshay|rucksoniya29|ruck, I'll look at fs03912:46
soniya29|ruckweshay|ruck, even we have fs20 train with same cause12:46
weshay|rucksoniya29|ruck, fyi.. selinux is permissive in any upstream / rdo job..12:46
weshay|rucksoniya29|ruck, what cause?12:46
mariosweshay|ruck: akahat: just got off call with panda going to cancel the daily syncs from tomorrow12:46
weshay|rucksoniya29|ruck, when we're rucking.. please be very specific12:46
mariosweshay|ruck: akahat: rfolco: we can setup a new daily sync if needed this sprint for promoter work12:47
weshay|ruckmarios, I merged the first patch12:47
soniya29|ruckweshay|ruck, selinux denials12:47
mariosweshay|ruck: yeah i was hovering over the +A button and you beat me to it :D12:47
mariosweshay|ruck: we were discussing with panda12:47
weshay|rucksoniya29|ruck, ok.. you understand how that is impossible?12:47
rfolcomarios, I am ok with that12:48
soniya29|ruckweshay|ruck, I dont think so12:48
soniya29|ruckweshay|ruck, I will look into the tempest failure till then12:48
weshay|rucksoniya29|ruck, selinux... has three modes... enforcing, permissive, off12:49
*** udesale__ has joined #oooq12:49
soniya29|ruckweshay|ruck, yeah12:49
soniya29|ruckweshay|ruck, so?12:49
weshay|rucksoniya29|ruck, all tripleo, rdo jobs run in permissive.. which means.. we'll flag the error.. you will find denied in the audit log.. but it's NEVER fatal12:49
weshay|rucksoniya29|ruck, it's just a warning.. that IF we ran in enforcing.. it would fail12:50
weshay|ruckso we can get them fixed before they hit OSP12:50
weshay|rucksoniya29|ruck, is that clear now?12:50
soniya29|ruckweshay|ruck, okay..got it now12:50
*** udesale_ has quit IRC12:51
soniya29|ruckweshay|ruck, so its not something we have to get worried for the moment12:51
weshay|ruckwe should open bugs on it.. but it's not why jobs are failing... and we'll see that warning on most jobs until it's fixed12:52
weshay|rucksoniya29|ruck, give me a log to the fs039 job you were looking at please12:52
soniya29|ruckweshay|ruck, https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train/2dc2ddc/logs/_selinux_denials_found_Tempest_test_failed.log12:53
weshay|rucksoniya29|ruck, 2020-10-01 07:39:07.591462 | primary | undercloud                 : ok=27   changed=12   unreachable=0    failed=1    skipped=70   rescued=0    ignored=112:55
weshay|ruckhttps://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train/2dc2ddc/job-output.txt12:55
weshay|rucksoniya29|ruck, https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train/2dc2ddc/logs/undercloud/home/zuul/tempest/tempest.html.gz12:55
weshay|ruckthat's the failure..12:56
soniya29|ruckweshay|ruck, ack12:56
weshay|ruckperhaps I need add a sova reason for tempest12:56
weshay|rucksshnaidm, fyi ^12:56
rlandy|roverykarel: wrt https://review.rdoproject.org/r/#/c/29827/ ...12:56
rlandy|roverworth adding the change to https://github.com/rdo-packages/octavia-tempest-plugin-distgit/blob/rpm-master/python-octavia-tests-tempest.spec#L6912:56
rlandy|roveror not until the deps have all been worked out?12:56
rfolcotripleo ci planning meeting starting in 2 min >> arxcruz, rfolco, zbr, sshnaidm, rlandy, marios, ysandeep, bhagyashris, svyas, pojadhav, akahat, weshay, chandankumar12:57
rfolcohttps://hackmd.io/IhMCTNMBSF6xtqiEd9Z0Kw#2020-10-01-Unified-Sprint-34-Planning12:58
rfolcohttps://projects.engineering.redhat.com/secure/RapidBoard.jspa?rapidView=428512:58
sshnaidmweshay|ruck, it worked fine: https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train/2dc2ddc/logs/_selinux_denials_found_Tempest_test_failed.log12:59
*** Goneri has joined #oooq12:59
rlandy|rover020-10-01 07:22:30,554 385096 INFO     [tempest.lib.common.ssh] Creating ssh connection to '10.0.0.121:22' as 'cirros' with public key authentication12:59
rlandy|rover2020-10-01 07:22:30,559 385096 WARNING  [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@10.0.0.121 ([Errno None] Unable to connect to port 22 on 10.0.0.121). Number attempts: 1. Retry after 2 seconds.12:59
rlandy|rover2020-10-01 07:22:33,065 385096 WARNING  [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@10.0.0.121 ([Errno None] Unable to connect to port 22 on 10.0.0.121). Number attempts: 2. Retry after 3 seconds12:59
rlandy|rover^^ failure on all OVB12:59
weshay|rucksshnaidm, ah.. so it did!! thanks13:00
weshay|rucksshnaidm++13:00
sshnaidmweshay|ruck, and I think need to separate selinux from it..13:00
sshnaidmcreate confusions13:00
weshay|rucksshnaidm, let's chat in planning13:01
weshay|ruckhttp://dashboard-ci.tripleo.org/d/3-DYSmOGk/jobs-exploration?orgId=1&var-influxdb_filter=job_name%7C%3D%7Cperiodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train13:01
weshay|ruckseems ok to me13:01
weshay|ruck:)13:01
rlandy|rover2020-10-01 07:39:01 |     2020-10-01 07:38:40,496 38 ERROR    [tempest.scenario.manager] Public network connectivity check failed13:02
*** udesale_ has joined #oooq13:02
weshay|ruckzbr,13:02
rlandy|roversoniya29|ruck: k - so it looks like all the OVB tests are failing tempest in train13:03
rlandy|rovernot just fs03913:03
soniya29|ruckrlandy|rover, yeah now all ovb jobs are failing13:03
rlandy|roverand it started yesterday13:04
rlandy|roversoniya29|ruck: is there a bug for that yet?13:04
soniya29|ruckrlandy|rover, nope13:04
rlandy|roversoniya29|ruck: k - creating one13:04
*** udesale__ has quit IRC13:05
soniya29|ruckrlandy|ruck, okay13:05
ykarelrlandy|rover, need to add deps first then only it can be added in https://github.com/rdo-packages/octavia-tempest-plugin-distgit/blob/rpm-master/python-octavia-tests-tempest.spec#L6913:05
rlandy|roverykarel: ack13:07
holserweshay|ruck https://review.opendev.org/#/c/755401/ and https://review.opendev.org/#/c/755402/ passed CI13:09
holsergood to merge13:09
*** amoralej|lunch is now known as amoralej13:11
rlandy|roversoniya29|ruck: https://bugs.launchpad.net/tripleo/+bug/189804613:21
openstackLaunchpad bug 1898046 in tripleo "Centos-8 Train OVB jobs are failing tempest - Failed to establish authenticated ssh connection to cirros" [Undecided,New]13:21
soniya29|ruckrlandy|rover, ack13:21
rlandy|roversoniya29|ruck: looking at changes in train https://trunk.rdoproject.org/centos8-train/report.html13:38
soniya29|ruckrlandy|rover ack13:40
*** chem has joined #oooq13:51
rlandy|roverhttps://logserver.rdoproject.org/53/18953https://logserver.rdoproject.org/53/18953/16/check/periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-ussuri/c1c236b/job-output.txt/16/check/periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-ussuri/c1c236b/job-output.txt13:51
rlandy|roverfailure there13:51
rlandy|roverhttps://logserver.rdoproject.org/53/18953/16/check/periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-ussuri/c1c236b/logs/undercloud/home/zuul/overcloud_update_run-Controller.log.txt.gz13:52
*** rascasoft has joined #oooq14:01
zbr10 conn/secon? AFAIK, docker pull does concurrent pulls by default anyway! ... on a single image pull.14:05
zbreach layer is GET, and they are downloaded in parallel.14:05
weshay|ruckhttps://hackmd.io/vRMVeZXZRgK5Vxqi6ENUDg14:08
rlandy|roverykarel: iff the top of your head :) ... any thoughts on https://bugs.launchpad.net/tripleo/+bug/1898046? train only, ovb only ssh auth fail14:12
openstackLaunchpad bug 1898046 in tripleo "Centos-8 Train OVB jobs are failing tempest - Failed to establish authenticated ssh connection to cirros" [Critical,Triaged]14:12
rlandy|roveroff14:12
rlandy|roversince yesterday14:13
rlandy|roverhttps://opendev.org/openstack/networking-ovn/commit/dd31ea507d777cfb5cf81677b347b582aa511c2c14:15
*** TrevorV has joined #oooq14:16
ykarelrlandy|rover, i see errors Name or service not known14:20
ykarelhttps://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/57d77c6/logs/overcloud-novacompute-0/var/log/containers/neutron/ovn-metadata-agent.log.txt.gz14:20
rlandy|roverykarel: was looking at:14:21
rlandy|rover2020-09-28 11:15:58 2020-09-28 11:11:48 python-networking-ovn14:21
rlandy|roverbbfe8feb5ebcfe548bb701007e8a1f78535d9396 network14:21
rlandy|roverSUCCESS14:21
rlandy|roverrepo build log14:21
rlandy|rover2020-09-28 11:12:59 2020-09-28 11:11:47 python-networking-ovn14:21
rlandy|roverdd31ea507d777cfb5cf81677b347b582aa511c2c network14:21
rlandy|roverSUCCESS14:21
rlandy|roverrepo build log14:21
rlandy|rovercurrent tripleo rev'ed on python-networking-ovn-7.3.1-0.20200928111618.bbfe8fe.el8.src.rpm2020-09-28 11:16 1.2M14:21
rlandy|roverykarel: thanks14:23
rlandy|roveradding details to LP14:23
ykarelrlandy|rover, seems issue is caused with https://review.opendev.org/#/c/749955/314:37
ykarelneutron patch14:37
rlandy|roverykarel: thanks - see the comments there14:38
ykarellooking14:38
rlandy|roversorry - ... I see the comments on the error when using the patch14:39
ykarelbut i see u mentioned networking-ovn not neutron14:39
rlandy|roverjust debug notes14:43
weshay|ruckrfolco, let's meet later today for a bit and review the board for clarity14:50
weshay|ruckrlandy|rover, should we add fs001 to http://dashboard-ci.tripleo.org/d/mOvYIiOMk/component-pipeline-train?orgId=1&fullscreen&panelId=42914:54
rlandy|roverweshay|ruck: probably - there is some stuff only OVB hits14:55
rlandy|roverweshay|ruck: will add that14:56
weshay|ruckrlandy|rover++14:56
rlandy|roverweshay|ruck: when did we branch ussuri?14:58
rlandy|roverafter Feb this year?14:58
* weshay|ruck looks14:58
weshay|ruckrlandy|rover, https://releases.openstack.org/ussuri/schedule.html14:59
weshay|ruckapril14:59
weshay|ruckhttps://hackmd.io/n7eBGiPVR4yoFbz9o-V2bQ14:59
rlandy|roverweshay|ruck: k - ack  - explains missing backport14:59
rlandy|roversoniya29|ruck: we have a ussuri updates failure ...15:02
rlandy|roverperiodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-ussuritestprojectmastercheck18953,161 hr 19 mins 14 secs2020-10-01 12:11:01FAILURE15:02
rlandy|roverperiodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-ussuriopenstack/tripleo-cimasteropenstack-periodic-integration-stable1master1 hr 21 mins 9 secs2020-10-01 09:56:58FAILURE15:02
rlandy|roverchecking that next15:02
holsermarios I guess you forgot files: in https://review.opendev.org/#/c/739457/3115:03
mariosholser: what do you mean?15:04
holserlook what we tested15:04
holsermarios nevermind my bad15:05
* marios confused stare15:05
mariosholser: ack :D15:05
mariosakahat: rfolco: weshay|ruck: so lets merge the next one then https://review.rdoproject.org/r/#/c/28019/15:09
akahatmarios, rfolco weshay|ruck, if possible let's merge our all the patches today..15:11
mariosakahat: k i rebased the one ontop too just now https://review.rdoproject.org/r/2802015:15
akahatmarios, ack15:15
rlandy|roverweshay|ruck: soniya29|ruck: https://review.rdoproject.org/r/29832 Add fs001 jobs to network component15:33
rlandy|roverweshay|ruck: soniya29|ruck: I guess master promoted?15:36
weshay|ruckmarios, https://review.opendev.org/#/c/755402/15:37
mariosholser: will you add some info on the bug please https://bugs.launchpad.net/tripleo/+bug/189653715:38
openstackLaunchpad bug 1896537 in tripleo "[stein] standalone-upgrade failing tempest" [Critical,Triaged]15:38
holsermarios sure15:38
mariosholser: do you have a test for it somewhere? i.e. how do we know this will fix the tempest fail for the bug15:39
mariosweshay|ruck: checking15:39
holsermarios it will fix because the 0.3.5 won't boot up15:40
holsermarios look here https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_993/755220/2/check/tripleo-ci-centos-7-standalone-upgrade-stein/99353c0/logs/undercloud/home/zuul/tempest/etc/tempest.conf15:40
holserit's config of failed job https://review.opendev.org/#/dashboard/self15:41
mariosholser: but where is the green run that is using 0.4.0 that was my request...15:42
mariosholser: but I +A on 18:37 < weshay|ruck> marios, https://review.opendev.org/#/c/755402/15:42
mariosholser: especially because it was green run /bumping the version didn't break somethign else15:42
weshay|ruckmarios, and when chandan and arx get back.. I have to find out why these jobs are not usiing os-tempest15:43
holserWhat was green? I am nopt following15:43
holsermarios commented15:47
*** pojadhav is now known as pojadhav|afk15:48
weshay|ruckrlandy|rover, on the rdo-jobs.. zuul.d/component-jobs.yaml:685:8: [error] wrong indentation: expected 6 but found 7 (indentation)15:48
*** ykarel is now known as ykarel|away15:48
weshay|ruckrlandy|rover, I fixed it15:50
rlandy|roverweshay|ruck: thanks - saw15:50
*** ykarel|away has quit IRC15:53
rlandy|rover, "Error: error pulling image \"docker.io/tripleou/centos-binary-rabbitmq:current-tripleo\": unable to pull docker.io/tripleou/centos-binary-rabbitmq:current-tripleo: unable to pull image: Error writing blob: error storing blob to file \"/var/tmp/storage940813393/1\": read tcp 192.168.103.246:35498->104.18.122.25:443: read: connection reset by peer"], "stdout": "", "stdout_lines": []}15:57
rlandy|rover^^ general rate limiting?15:57
*** jmasud has quit IRC15:59
*** jmasud has joined #oooq15:59
rlandy|roverhttps://hub.docker.com/layers/tripleou/centos-binary-rabbitmq/current-tripleo/images/sha256-864d57d58fdc6d3850b5115552f92381b332d70bb577b0ea2aa742220a9ca3a9?context=explore16:00
rlandy|roverexists16:00
*** dsneddon has quit IRC16:01
weshay|ruckrlandy|rover, don't think so16:03
weshay|ruckhiccup16:03
rlandy|rovertwice over ... hmmm ... rerun , rerun16:04
*** rascasoft has quit IRC16:10
rlandy|roverweshay|ruck: wrt train-centos-7...16:13
rlandy|roverafaict, it should promote16:13
rlandy|roverfrom the run today16:13
weshay|ruckhrm k.. /me looks16:13
rlandy|roverchecking the criteria16:13
rlandy|roverit looks like we have passing system16:13
rlandy|rover2020-10-01 15:54:09,812 25375 INFO     promoter No more candidates16:13
rlandy|roverwhat am I missing here?16:14
rlandy|rover2020-10-01 15:54:03,932 25375 INFO     promoter Current "current-tripleo" hash is commit: 75bd92ac34ad64e0a44f46b0399d063e392289e5, distro: 003a41ddaa880da0c4344ed0a94ba6a5106c1b6e, timestamp=160154648016:15
rlandy|roverweshay|ruck: k - commit has not rev'ed16:15
rlandy|rover2020-10-01 11:41:12.120751 | primary |   "aggregate_hash": null,16:15
rlandy|rover2020-10-01 11:41:12.120819 | primary |   "commit_hash": "75bd92ac34ad64e0a44f46b0399d063e392289e5",16:15
rlandy|rover2020-10-01 11:41:12.120854 | primary |   "component": null,16:15
rlandy|rover2020-10-01 11:41:12.120916 | primary |   "distro_hash": "003a41ddaa880da0c4344ed0a94ba6a5106c1b6e",16:15
weshay|ruckrlandy|rover, I think standalone-upgrade is criteria16:15
weshay|ruckshoot.. it's not16:16
rlandy|roverweshay|ruck: negative according to http://38.102.83.109/config/CentOS-7/train.ini16:16
weshay|ruckya16:16
weshay|ruckrlandy|rover, ya. k. https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-standalone-upgrade-train16:17
weshay|ruckI'm wrong16:17
rlandy|roverweshay|ruck: either way - no new hash to promote16:18
weshay|ruckrlandy|rover, ya.. promoter.. says nothing new for c7 train16:19
weshay|ruckconsistent/2020-09-17 01:31-16:20
weshay|ruck[DIR]current-passed-ci/2020-09-17 01:31-16:20
weshay|ruck[DIR]current-tripleo-rdo/2020-09-17 01:31-16:20
weshay|ruck[DIR]current-tripleo/2020-09-17 01:3116:20
weshay|ruckand it's right16:20
rlandy|roverhttp://tripleo-cockpit.usersys.redhat.com/d/9DmvErfZz/cockpit?orgId=116:23
rlandy|roverwe only have 5 upstream fails16:23
rlandy|roverhand clap for upstream16:23
soniya29|ruckrlandy|rover, yeah just 5 of them :)16:24
*** udesale_ has quit IRC16:25
*** amoralej is now known as amoralej|off16:25
weshay|rucksoniya29|ruck, rlandy|rover yup.. pretty good.. a closer inspection16:33
rlandy|rovera step up from yesterday's 2216:33
weshay|ruckmaster gate 95.6% pass, ussuri 98.7% pass and train 91.7% pass16:33
weshay|ruckall the gate failures have merged last I checked16:34
weshay|ruckthere were 15 retries... so if tripleo pass > zuul pre.yamls.. I'll take it16:34
rlandy|roverSuccess40%rdo_trunk-promote-master-current-tripleo16:35
rlandy|rovereven a ci.centos pass16:35
rlandy|roverparty16:35
rlandy|roverSuccess20%rdo_trunk-promote-ussuri-current-tripleo16:36
weshay|ruck294 passed gate jobs16:36
weshay|ruckin 24hrs16:36
rlandy|roverweshay|ruck: what was that you said about not listening to ruck/rover prayers????16:36
weshay|ruck:)16:37
*** skramaja has quit IRC16:46
*** marios is now known as marios|out16:55
*** jtomasek has quit IRC16:57
*** derekh has quit IRC17:03
*** jpena is now known as jpena|off17:05
weshay|ruckrlandy|rover, https://review.rdoproject.org/r/2983317:07
weshay|rucksoniya29|ruck, ^ fyi.. so that neutron doesn't break ovb jobs in integration17:08
rlandy|rover   "success": true,17:21
rlandy|roverussuri should meet criteria now17:21
rlandy|roverlunch  biab17:22
*** dsneddon has joined #oooq17:23
*** ykarel|away has joined #oooq17:43
*** saneax has quit IRC17:59
*** b3nt_pin is now known as beagles18:00
rlandy|roverweshay|ruck: how often does the promoter master/ussuri/train8 trigger per day?18:08
weshay|rucktwice18:08
*** marios|out has quit IRC18:09
*** ykarel|away has quit IRC18:20
weshay|ruckrlandy|rover, our motto for the next three weeks.. "success": true,18:28
weshay|ruckrlandy|rover, well done18:28
*** gchamoul has quit IRC18:31
*** gchamoul has joined #oooq18:31
weshay|ruckof course holser's patch is failing in the gate :( I'll get it going :)18:40
weshay|ruckit's always the patches you want the most18:40
weshay|ruckLOT'S O retries18:41
rlandy|roverweshay|ruck: yeah - was just looking at https://review.opendev.org/#/c/755220/18:41
weshay|rucksomething is up w/ infra18:41
rlandy|roverbut that looks like a legit failure18:41
rlandy|rover2020-10-01 10:02:35 | 2020-10-01 10:02:35.862 280557 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] fatal: [standalone]: FAILED! => {"changed": false, "msg": "You can't upgrade cinder_volume without staged upgrade.  You need to use the limit option in order to do so.\n"}[00m18:41
rlandy|roverweshay|ruck: which patch are you looking at?18:42
weshay|ruck755401,318:42
rlandy|roverweshay|ruck: hmmm ... we are getting more gate failures now18:42
rlandy|roverlooking18:42
rlandy|roveron that patch18:43
weshay|ruckya.. it happens..18:43
rlandy|rover2020-10-01 16:07:36.665222 | primary | TASK [build-test-packages : Run DLRN] ******************************************18:44
rlandy|roverError in build_rpm_wrapper for openstack-tripleo-heat-templates: HTTP Error 403: Forbidden18:44
rlandy|rover^^ seen that before?18:45
rlandy|roverhttps://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_27d/755401/3/gate/tripleo-ci-centos-8-scenario003-standalone/27dc155/logs/delorean_logs/component/tripleo/fb/c0/fbc0649446985ace7375b759280bc4f7653a5040_dev/18:45
rlandy|roverugh - why is upgrade train still in the gate??18:45
weshay|ruckrlandy|rover, ya.. so I think that's been left there in hopes we can get it wroking18:46
rlandy|roverweshay|ruck: abandon restore on https://review.opendev.org/#/c/755401/?18:47
rlandy|roverfailing everywhere now18:47
rlandy|rover2020-10-01 18:16:46.687432 | primary | TASK [os_tempest : Execute tempest tests] **************************************18:47
rlandy|roverdiff failures on each job18:48
weshay|ruckwe could.. or just recheck when it fails18:49
weshay|ruckno opinion18:49
weshay|ruckam I jaded?18:49
rlandy|roverweshay|ruck: no - it's almost done and then we will have a record18:50
*** rlandy|rover is now known as rlandy|rover|mtg19:00
*** jmasud has quit IRC19:03
holserweshay|ruck thanks19:20
weshay|rucknp :)19:21
*** rlandy|rover|mtg is now known as rlandy|rover19:30
rlandy|roverweshay|ruck: thanks for rechecking 75540119:31
weshay|rucknp19:32
rlandy|roverthe rest of the gate looks not bad19:32
rlandy|rovermerge party19:32
weshay|ruckya.. we're good rlandy|rover I'll let you know when we need something there19:32
weshay|ruckrlandy|rover, you can do the 10 other things on your plate :)19:33
weshay|ruckthat is a LONG zuul queue19:33
rlandy|roverweshay|ruck: yep - started looking at the installed rpm list19:33
rlandy|roveryesterday19:33
weshay|ruckmeh.. and it just reset.. lolz19:34
rlandy|roverugh19:34
weshay|ruckoh well..19:34
rlandy|roverwhatever19:34
weshay|rucksuch is life19:34
rlandy|rovergum zu le'tovah19:34
*** jmasud has joined #oooq19:35
*** jmasud has quit IRC19:39
*** jmasud has joined #oooq19:44
*** jmasud has quit IRC19:55
rfolcoweshay|ruck, could you please clarify in the wire-up tasks what you mean by templates in standalone, multinode, etc ? I see the standalone jobs parent off content provider, not sure what needs to be done in those tasks. Also I started this https://hackmd.io/vRMVeZXZRgK5Vxqi6ENUDg#QampA to help clarifying questions20:20
rlandy|roverweshay|ruck: think that's an exception20:21
weshay|ruckya.. Q&A is a solid idea20:21
rlandy|roverother gate failures have node logs20:21
weshay|ruckya..20:21
weshay|ruckrfolco, yes.. helping the security team out20:22
weshay|ruckso will be a few20:22
weshay|ruckrfolco, let's do the board tomorrow morning.. out of time today20:52
rfolcosure20:53
rfolcothanks20:53
* rfolco goes to barber shop in a few20:53
*** jmasud has joined #oooq20:57
*** rfolco has quit IRC21:07
rlandy|roverhttps://review.opendev.org/#/c/755368/ - why did that review kick so many tests in third party??21:20
*** holser has quit IRC21:25
*** holser has joined #oooq21:42
*** jmasud has quit IRC22:30
*** jmasud has joined #oooq22:31
*** rlandy|rover has quit IRC22:34
*** tosky has quit IRC22:45
*** holser has quit IRC22:54
*** TrevorV has quit IRC23:02
*** matbu has quit IRC23:03
*** rfolco has joined #oooq23:19
*** rfolco has quit IRC23:24
*** rfolco has joined #oooq23:25
*** rfolco has quit IRC23:29
*** Goneri has quit IRC23:55
« Wednesday, 2020-09-30 Index Friday, 2020-10-02 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!