Thursday, 2020-03-19

*** ysandeep|afk is now known as ysandeep00:10
*** tosky has quit IRC00:50
*** jmasud has quit IRC00:54
*** jmasud has joined #oooq00:55
*** rlandy|bbl is now known as rlandy02:22
*** rlandy has quit IRC02:45
*** apetrich has quit IRC03:09
*** udesale has joined #oooq04:32
*** dsneddon_ has quit IRC04:41
*** skramaja has joined #oooq04:47
*** ratailor has joined #oooq04:54
*** dsneddon_ has joined #oooq06:21
*** dsneddon_ has quit IRC06:26
*** marios has joined #oooq06:40
*** skramaja_ has joined #oooq06:51
*** skramaja has quit IRC06:51
*** dpawlik has joined #oooq06:52
*** dpawlik has quit IRC06:53
*** dpawlik has joined #oooq06:53
*** jfrancoa has joined #oooq07:29
*** amoralej|off is now known as amoralej07:32
*** apetrich has joined #oooq07:36
*** tesseract has joined #oooq07:51
amoralejchandankumar, wrt https://bugs.launchpad.net/tripleo/+bug/1867931 are you around?08:03
openstackLaunchpad bug 1867931 in tripleo "periodic-tripleo-ci-centos-8-scenario010-standalone-network-master deploy fails with package conflict - "openstack-neutron-common-1:16.0.0-0.20200318094753.8c7ca77.el8.noarch conflicts"" [Critical,Triaged]08:03
amoraleji'd like to create a reproducer locally, as i couldn't reproduced the issue out of the env08:04
*** bogdando has joined #oooq08:04
chandankumaramoralej, I think we need to waive the criteria to make it passing https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario010-standalone-network-master08:11
chandankumarlet me propose a patch08:12
amoralejchandankumar, i'm trying to do minimal reproducer using tripleo-modify-image role08:17
chandankumarok08:17
chandankumaramoralej, marios https://review.rdoproject.org/r/2599608:19
*** tosky has joined #oooq08:21
marioschandankumar: k08:21
mariosreading bug08:21
marioschandankumar: but that means we're promoting a bad neutron version?08:22
marioschandankumar: how will it be fixed?08:23
chandankumarmarios, once we build the fresh container and then upgrade issues will be vanished08:23
chandankumarit will work in integration pipeline08:23
chandankumarany further issue can be caught in integration pipeline08:24
marioschandankumar: ok good point on the integration pipeline , this is component promotion. that's a bit better then...08:24
chandankumarmarios, I will wait amoralej last reproducer work08:24
*** jbadiapa has joined #oooq08:33
mariosfellow tripleo-ci folks please check https://review.opendev.org/#/c/710556/2 when you get a chance (I just added you) cgoncalves is adding a new FS but the only diff is 'standalone-upgrade' so IMO we should just add to allowed_overrides08:40
mariosi know panda is particularly fond of this approach and in his words 'lets just override everything why note'08:42
marios(not)08:42
* marios twists needle in panda doll some more08:42
amoralejchandankumar, i've reproduced the issue08:44
amoralejlocally08:44
amoralejwith the role08:44
chandankumaramoralej, awesome08:44
amoralejthere is some problem in yum_update.sh08:44
amoralejwhen building the list of packages to update08:44
amoralejthat's wrong /usr/bin/dnf -y update openstack-neutron-common openstack-tripleo-common-container-base puppet-concat puppet-glance puppet-kafka puppet-keystone puppet-midonet puppet-vcsrepo python3-neutron"08:45
chandankumaramoralej, sorry you mean it does not include python3-ovn-networking package?08:46
*** jpena|off is now known as jpena08:48
amoralejchandankumar, it does not include openstack-neutron which provides/obsoletes python3-networking-ovn08:52
amoralejwe could move provides/obsoletes to openstack-neutron-common08:52
amoralejas workaround08:52
chandankumaramoralej, next question is does openstack-neutron gets updated in the current repo?08:52
amoralejchandankumar, from repo pov is fine08:53
amoralejthe problem is08:53
amoralejas currently, openstack-neutron is not installed in the container08:53
amoralejit's not detected to be updated08:53
chandankumaramoralej, ah08:53
amoralejit may happen with any package renaming08:53
amoralejyum_update.sh only tries to update package which *are already* installed08:54
amoralejif a package is updated by a different one using provides, it will fail08:54
*** jmasud has quit IRC08:54
chandankumaramoralej, ok got it08:56
chandankumaramoralej, let's get the workaround in neutron-common and then we can re-run the job08:56
*** jmasud has joined #oooq08:57
*** skramaja_ has quit IRC09:12
*** skramaja has joined #oooq09:14
amoralejchandankumar, i explained what the problem is in https://bugs.launchpad.net/tripleo/+bug/186793109:17
openstackLaunchpad bug 1867931 in tripleo "periodic-tripleo-ci-centos-8-scenario010-standalone-network-master deploy fails with package conflict - "openstack-neutron-common-1:16.0.0-0.20200318094753.8c7ca77.el8.noarch conflicts"" [Critical,Triaged]09:17
amoraleji'll propose workaround09:17
zbrpanda: https://review.rdoproject.org/r/#/c/25989/09:29
pandamarios: this morning I woke up suddenly with a sharp pain in my back.09:37
arxcruz|roverpanda: are you okay man?09:37
mariospokerface09:37
arxcruz|roverI'm getting very stressed, every day, if my daughter cof, I got freaking nervous09:38
mariosarxcruz|rover: ( 10:42  * marios twists needle in panda doll some more09:38
arxcruz|rovermarios: ahhh hehehe, sorry09:38
mariosarxcruz|rover: yes agree this is very s***y situation :( and no one knows how long it will last09:38
arxcruz|roverit's being hard, 2 weeks already without leave home09:38
mariosarxcruz|rover: so in germany there is measures for staying at home like italy?09:39
*** dtantsur|afk is now known as dtantsur09:39
arxcruz|rovermarios: yes, even churches are closed09:40
arxcruz|roveronly essential09:40
mariosarxcruz|rover: here most things are (ordered to) close but there is no law about moving freely yet, even though like 70% people are just at home roads are empty09:41
arxcruz|roverwell, it started this monday, but I started not leaving home one week before, just in case09:41
mariosarxcruz|rover: it is suspected they will make it that we can't leave the house in the next few hours, but thre is no announcement yet09:41
arxcruz|rovermarios: you can move free around berlin, but you can't go to crowded places09:41
arxcruz|roverin fact, crouded places are prohibited here09:42
mariosyah here too09:42
arxcruz|roverbut they say, if you can, stay at home09:42
mariossame09:42
mariossurreal09:42
arxcruz|roverand in my building there is a couple that is positive to corona09:42
arxcruz|roveryeah man, i thought I would be killing zombies in the apocalipse09:43
* marios twists needle in panda doll some more09:43
mariosoh well we'll find ways to cope i guess09:43
*** derekh has joined #oooq09:43
marios:(09:43
*** rascasoft has joined #oooq09:47
*** sshnaidm|afk is now known as sshnaidm09:49
*** dsneddon_ has joined #oooq09:55
*** dsneddon_ has quit IRC10:00
zbrmarios or panda https://review.rdoproject.org/r/#/c/25987/10:04
zbrsshnaidm: should we make this voting? https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-zuul-runner-reproducer-fedora-30-host10:06
zbror is too soon?10:06
zbror maybe adding it to a weekly pipeline?10:07
sshnaidmzbr, where do you see this job?10:08
zbrhttps://review.rdoproject.org/r/#/c/25987/10:08
zbri see non-voting there10:09
zbri have no knowledge about the project, but i fixed the skip_missing_interpreters, and i was surprised about the nv10:09
sshnaidmzbr, yeah, ok to do it voting10:09
sshnaidmzbr, it's not a developing project right now..10:10
zbrsshnaidm: you can also add it to weekend pipeline at the same time, it could prove practical.10:10
sshnaidmzbr, this is stalled, no need10:10
zbrat least we will know if it gets broken faster instead of two months later. ok, if you say so.10:10
marioszbr: ack10:11
*** amoralej is now known as amoralej|afk10:13
arxcruz|roverchandankumar: https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-master/3e5f3ca/logs/undercloud/var/log/containers/stdouts/container-puppet-neutron.log.txt.gz10:16
arxcruz|roverhave you seen this error ?10:16
arxcruz|rover2020-03-18T17:40:18.540481168+00:00 stderr F <13>Mar 18 17:40:18 puppet-user: No such file or directory @ rb_sysopen - /etc/neutron/plugins/networking-ovn/networking-ovn-metadata-agent.ini10:16
arxcruz|roveropening a promotion blocker bug10:16
marioschandankumar: nice green baremeratl fs1 run(s)10:29
*** holser has joined #oooq10:36
*** tesseract has quit IRC10:36
*** tesseract has joined #oooq10:42
*** tesseract has quit IRC10:46
*** tesseract has joined #oooq10:48
weshay|ruckarxcruz|rover, https://review.opendev.org/#/c/713676/3/roles/build-containers/tasks/main.yaml10:50
arxcruz|roverweshay|ruck: weird, i'm pretty sure i added the become :/10:51
arxcruz|roverweshay|ruck: doing right now10:51
weshay|ruck:)10:51
weshay|ruckarxcruz|rover, chandankumar let me grab a change for you re neutron ovn10:51
arxcruz|roverweshay|ruck: https://bugs.launchpad.net/tripleo/+bug/186807910:53
openstackLaunchpad bug 1868079 in tripleo "puppet-neutron container failed" [Critical,Triaged]10:53
arxcruz|roverthe bug10:53
weshay|ruckchandankumar, ugh10:54
weshay|ruckarxcruz|rover, make chandankumar fix it :)10:55
weshay|ruckchandankumar, I don't think we should have waived https://bugs.launchpad.net/tripleo/+bug/186793110:55
openstackLaunchpad bug 1867931 in tripleo "periodic-tripleo-ci-centos-8-scenario010-standalone-network-master deploy fails with package conflict - "openstack-neutron-common-1:16.0.0-0.20200318094753.8c7ca77.el8.noarch conflicts"" [Critical,Triaged]10:55
*** tesseract-RH has joined #oooq11:00
weshay|ruckchandankumar, amoralej|afk we have a held node for the neutron issue11:00
weshay|ruckholding the waive11:00
weshay|ruckarxcruz|rover, ignore centos-8 pipeline in integration for now11:01
*** yolanda has quit IRC11:01
arxcruz|roverok11:01
pandamarios: not sure if you're interested at this point, added explanations on https://review.rdoproject.org/r/2593811:02
*** tesseract-RH has left #oooq11:02
*** tesseract has quit IRC11:03
pandamarios: and for heaven's sake, put down that doll.11:04
mariospanda: thanks pandaman i will check again. what doll?11:06
marioschandankumar: did you see the weird tempest error (and no logs) at https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-compute-master/d55ae94/job-output.txt11:07
marioshttps://logserver.rdoproject.org/openstack-component-tempest/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-tempest-master/de4fa04/job-output.txt11:07
marios        * 2020-03-18 20:24:46.925453 | primary | Data could not be sent to remote host "127.0.0.2". Make sure this host can be reached over ssh: kex_exchange_identification: Connection closed by remote host11:07
*** holser has quit IRC11:08
*** holser has joined #oooq11:10
chandankumarweshay|ruck, amoralej|afk fixed it11:11
weshay|ruckchandankumar, patch?11:11
*** holser has quit IRC11:11
chandankumarweshay|ruck, https://review.rdoproject.org/r/#/c/25997/11:11
*** holser has joined #oooq11:12
weshay|ruck:)11:12
weshay|rucknot a ci bug :)11:12
chandankumarweshay|ruck, rechecked here https://review.rdoproject.org/r/#/c/25325/11:12
weshay|ruckchandankumar++11:12
* weshay|ruck looks11:12
*** yolanda has joined #oooq11:13
*** dtantsur is now known as dtantsur|afk11:20
chandankumararxcruz|rover, weshay|ruck https://review.opendev.org/71380911:21
*** ysandeep is now known as ysandeep|afk11:28
chandankumarmarios, yes in the morning11:34
chandankumarmarios, I have seen this earlier also may due to network issue11:34
marioschandankumar: ack hard to see whats up without logs though not sure how to proceed there11:36
*** ccamacho has quit IRC11:38
*** ccamacho has joined #oooq11:44
weshay|ruckwow... two full promotions from baremetal :)11:54
*** rlandy has joined #oooq11:57
chandankumaramoralej|afk, Hello11:58
chandankumaramoralej|afk, still not helping https://logserver.rdoproject.org/25/25325/17/check/periodic-tripleo-ci-centos-8-scenario010-standalone-network-master/42b1497/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz11:58
weshay|ruckchandankumar, amoralej|afk we have a held node.. and I also set it up on my local env.. just rpm though11:59
weshay|ruckchandankumar, amoralej|afk /me notes that internal osp packaging uses the same role to update containers12:00
* weshay|ruck wonders if the issue is that the config file has changed content?12:00
weshay|rucknot default from rpm12:00
* weshay|ruck checks w/ my env12:00
sshnaidmweshay|ruck, chandankumar hi, so wrt renaming, are you good with such an approach? https://review.rdoproject.org/r/#/q/topic:f027+(status:open+OR+status:merged)12:02
weshay|ruckI am now that we're out of the weeds.. marios ^?12:03
chandankumarweshay|ruck, need to include openstack-neutron-common package12:04
chandankumarsorry python3-neutron12:04
weshay|ruckamoralej|afk, chandankumar if it turns out the only time this package would upgrade from this version is right now.. I think we can waive the job12:04
weshay|ruckchandankumar, that's in the neutron component no?12:05
* weshay|ruck looks12:05
chandankumarweshay|ruck, please merge waive patch12:05
rlandythis could happen in other situations in general12:05
weshay|ruckchandankumar, need more than that12:05
chandankumarweshay|ruck, loike12:06
chandankumarlike12:06
chandankumarsshnaidm, checking12:06
weshay|ruckchandankumar, so note.. if this would not repeat in osp container builds12:07
rlandychandankumar: looks enough to me to continue12:07
rlandywhat else would we need to get through criteria?12:07
weshay|ruckchandankumar, /me notes they ALSO use tripleo-modify-image to update their containers12:07
weshay|ruckchandankumar, /me sets up locally.. give me a few12:08
rlandychandankumar: how often does a package get obsoleted by a different component?12:08
*** holser has quit IRC12:08
chandankumarrlandy, if obseletes keyword defined in spec file12:08
chandankumarrlandy, https://review.rdoproject.org/r/#/c/25997/12:09
rlandy"but doing it at the python module level is more accurate as it's the package"12:10
*** ysandeep|afk is now known as ysandeep12:10
rlandy^^ what are the chances we see that again?12:10
*** holser has joined #oooq12:11
weshay|rucksshnaidm, need a patch to update criteria https://review.rdoproject.org/r/#/q/topic:f027+(status:open+OR+status:merged)12:11
chandankumarrlandy, we obeselete a package when there is a rename of the package or some context exists in both project12:12
chandankumar*same12:12
sshnaidmweshay|ruck, this one? https://review.rdoproject.org/r/#/c/25835/12:12
mariosweshay|ruck: reading back12:12
mariosweshay|ruck: k will add that to my reviews12:14
chandankumarrlandy, let me check whether the fix landed in the consistent repo or not12:14
weshay|ruckchandankumar++12:15
chandankumarwe need to run network component consitent then network standalone jobs na?12:15
chandankumarrlandy, ^^12:15
chandankumarrlandy, https://review.rdoproject.org/r/25325 here we donot promoted the fix so it is not there na?12:15
rlandychandankumar: we need to rerun the promote job12:16
chandankumarrlandy, yes that's why fix is not there12:16
chandankumarlet's rerun it and see12:16
rlandychandankumar: it ran the first time12:16
rlandybut let me rerun in testproject - sec12:16
*** udesale_ has joined #oooq12:19
rlandychandankumar: weshay|ruck: https://review.rdoproject.org/r/#/c/25325/12:19
rlandythat will promote consistent-to-component-ci-testing and run scenario010 again12:20
*** udesale has quit IRC12:20
chandankumarrlandy, aye, hope to see it working12:20
rlandychandankumar: let's hope ...12:20
chandankumararxcruz|rover, weshay|ruck https://github.com/raukadah/openstack-tempest-skiplist12:20
rlandybut we need to address this issue in general  - in scrum12:21
chandankumararxcruz|rover, weshay|ruck I am keeping train and master skip list as rest will go away in sometime12:22
weshay|ruckchandankumar, amoralej|afk so.. I'll rolled back.. updated the config file... /etc/neutron/plugins/networking-ovn/networking-ovn.ini  then ran an update12:22
weshay|ruckand it worked fine12:22
weshay|ruckchandankumar, anything else to try?12:22
chandankumarweshay|ruck, I am out of ideas12:23
weshay|ruckchandankumar, k... /me removes -212:23
weshay|ruckrlandy, ?12:23
chandankumarweshay|ruck, please wait for rlandy patch result12:23
weshay|ruckany further thoughts?12:23
weshay|ruckk12:23
rlandyweshay|ruck: in general ,lots12:23
rlandyfor this, let it go if the testproject passes12:24
weshay|ruckk12:24
rlandythis is our basic component pipeline mechanism12:24
rlandyit needs to work for ALL update cases12:24
rlandyhow ever packages get changes/obsoleted/12:25
*** amoralej|afk is now known as amoralej12:28
*** jpena is now known as jpena|lunch12:30
amoralejweshay|ruck, chandankumar let me try with the new packages12:30
weshay|ruckk.. thanks for your patience on this one amoralej12:30
amoralejweshay|ruck, n p12:31
weshay|ruckarxcruz|rover, you need anything?12:31
amoralejweshay|ruck, what do you mean by "I'll rolled back.. updated the config file... /etc/neutron/plugins/networking-ovn/networking-ovn.ini  then ran an update" ?12:31
arxcruz|roverweshay|ruck: toilette paper12:32
weshay|ruckLOLZ12:32
arxcruz|roverlol just kidding, no, everything is fine12:32
chandankumarIndian does not need to worry about toilet papers12:32
arxcruz|roverI am scary to ask...12:32
chandankumararxcruz|rover, water is everywhere12:32
amoralejchandankumar, weshay|ruck in my test, it works fine after https://review.rdoproject.org/r/#/c/25997/12:33
amoralejwith the role, i mean12:33
chandankumaramoralej, after rlandy https://review.rdoproject.org/r/#/c/25325/ review we can see real result that time we donot promoted consistent repo to component repo12:34
amoralejchandankumar, i've tested again with the same reproducer playbook12:40
amoralejthat was reproducing the issue12:40
amoralejand this time worked fine12:40
amoralejafter moving the provides/obsoletes12:40
rlandychandankumar: weshay|ruck: if the testproject works, you won't need the patch to remove criteria12:41
rlandyit will promote on the next round12:41
rlandyevery two hours12:41
chandankumarrlandy, aye12:43
rlandychandankumar: if this passes, I'll run the rest of  the tests12:43
rlandyand we will get a promotion to promoted-components12:43
rlandylet's discuss more at scrum12:43
amoralejrlandy, that patch has any "trick" applied?12:49
amoralejor it's regular component promotion job?12:49
weshay|ruckchandankumar, rlandy marios we ready to make podman default for master? https://review.opendev.org/#/c/636441/12:51
weshay|ruckthe only thing I can think of is the upgrade job.. which should not block I think12:51
mariosweshay|ruck: looking (does updates work ok?)12:51
mariosyeah looks like12:52
mariosweshay|ruck: even standalone-upgrade green there12:52
mariosweshay|ruck: but at this point there shouldn't be any errors i mean master/nothing is using docker12:52
rlandyamoralej: no tricks :)12:53
* marios recalls an issue for scenario4 and manila/no docker sthing sthing but that job is also green 12:53
rlandyrerunning the other jobs so we can clean promote12:54
*** jmasud has quit IRC12:54
amoralejack12:54
*** jmasud has joined #oooq12:56
rlandyhttps://review.rdoproject.org/r/#/c/25412/13/.zuul.yaml12:57
rlandychandankumar: amoralej: weshay|ruck  ^^ that will run the rest of the network component pipeline jobs so we will get a promotion12:58
weshay|ruck:)12:58
chandankumarwhY the virus color is always RED?13:00
rlandyarxcruz|rover: zbr: chandankumar: scrum13:01
weshay|ruckzbr, ur going to be sorry you missed13:03
zbrjoining...13:09
*** ysandeep is now known as ysandeep|away13:11
bogdandohttps://review.opendev.org/636441 https://sfw.so/uploads/posts/2009-07/1247041812_4.jpg13:12
weshay|ruckhttps://docs.google.com/document/d/1jhHadIXwGDeR53TAdXTp3Ug8UkjeGhzXltpVFrBZmWY/edit#13:12
mariosrlandy: did she put you on the federal stop list? ;)13:14
rlandymarios: no - but she did ban me from going to the grocery store13:14
rlandyso I'll just starve to death13:15
rlandychandankumar: https://review.rdoproject.org/zuul/stream/ab13178e6c464533873d6facda00433a?logfile=console.log - no dice13:15
*** derekh has quit IRC13:26
amoralejrlandy, it seems it failed with different issue this time?13:27
rlandyamoralej: yes - must have passed update13:27
rlandyon meeting - will debug afterwards13:27
*** rascasoft_ has joined #oooq13:28
*** rascasoft has quit IRC13:28
*** rascasoft_ has quit IRC13:28
amoralejrlandy, ack, yes it passed TASK [tripleo_container_image_prepare : Run tripleo_container_image_prepare this time13:29
*** jpena|lunch is now known as jpena13:32
*** rascasoft has joined #oooq13:32
rascasofthey folks hi! Just wanted to know if you old fellas are all good, hope this moment passes by quickly, take care folks!13:39
weshay|ruckeveryone wave to rascasoft 0/13:43
apetricharxcruz|rover, so I have a patch up https://review.opendev.org/#/c/713873/ for the https://bugs.launchpad.net/tripleo/+bug/1866621 but I don't think automation has caught up with that yet13:46
openstackLaunchpad bug 1866621 in tripleo "Can't run container mistral_db_sync" [High,Triaged]13:46
apetrichrascasoft, o/13:46
arxcruz|roverapetrich: ack13:47
weshay|ruckrlandy, chandankumar same error13:48
weshay|ruck  Error: Transaction check error:\n      file /etc/neutron/plugins/networking-ovn/networking-ovn.ini from install of openstack-neutron-common-1:16.0.0-0.20200318094753.8c7ca77.el8.noarch conflicts with file from package python3-networking-ovn-7.0.0-0.20200109131747.eda5d7f.el8.noarch\n  stderr_lines13:48
weshay|ruckhttps://logserver.rdoproject.org/25/25325/17/check/periodic-tripleo-ci-centos-8-scenario010-standalone-network-master/42b1497/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz13:48
*** amoralej is now known as amoralej|lunch13:53
marios        * https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-compute-master/d55ae94/job-output.txt13:56
marios        * 2020-03-18 11:07:29.190587 | primary |     "cmd": "set -e\nif [ -d /openstack/venvs/tempest-untagged/bin ];\nthen\n. /openstack/venvs/tempest-untagged/bin/activate\nfi\ntempest run   --concurrency 4   --blacklist-file /home/zuul/tempest/etc/tempest_blacklist.txt   --whitelist-file /home/zuul13:56
marios        * https://logserver.rdoproject.org/openstack-component-tempest/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-tempest-master/de4fa04/job-output.txt13:56
marios        * 2020-03-18 20:24:46.925453 | primary | Data could not be sent to remote host "127.0.0.2". Make sure this host can be reached over ssh: kex_exchange_identification: Connection closed by remote host13:56
marios        * 2020-03-18 20:24:46.924876 | primary | fatal: [undercloud]: UNREACHABLE! => {13:57
chandankumarmarios, rlandy https://github.com/ansible/ansible/issues/1340113:59
*** udesale_ has quit IRC14:00
*** derekh has joined #oooq14:00
rlandyweshay|ruck: chandankumar: no - it's a different issue14:06
chandankumarrlandy, sorry what issue?14:06
rlandychandankumar: why scenario010 standalone on network component failed14:09
chandankumarrlandy, /me looks at logs14:10
chandankumarrlandy, weshay|ruck https://logserver.rdoproject.org/25/25325/18/check/periodic-tripleo-ci-centos-8-scenario010-standalone-network-master/ab13178/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz14:11
chandankumarit is a new issue all girbris lang14:12
chandankumarcgoncalves, Helllo mate14:12
chandankumarcgoncalves, please have a look we got some issue here https://logserver.rdoproject.org/25/25325/18/check/periodic-tripleo-ci-centos-8-scenario010-standalone-network-master/ab13178/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz14:12
chandankumarcgoncalves, Configure octavia on overcloud14:12
rlandychecking the other jobs on that hash14:13
chandankumarrlandy, weshay|ruck since new team mates will be joining our next retro, can we put a card for short intro ?14:13
*** ysandeep|away is now known as ysandeep14:13
rlandychandankumar: ack - ...14:14
rlandychandankumar: rfolco will be back next week also14:14
chandankumarit will help them to know what everyone does in the team14:14
chandankumarrlandy, rfoloco is on vacation14:14
rlandyso he'll be back running14:14
chandankumarrlandy, aye14:14
rlandychandankumar: ack - he has been the whole sprint :)14:14
chandankumarrlandy, I will let him know since he is coming on monday :-)14:15
weshay|ruckchandankumar, aye14:15
rlandycool14:16
chandankumarrlandy, Thank you for doing TC work, It was awesome :-)14:16
weshay|ruckrlandy++14:17
chandankumarrlandy++14:17
*** amoralej|lunch is now known as amoralej14:22
cgoncalveschandankumar, hey. so, looking at the log, the error comes from Glance.14:24
*** jmasud has quit IRC14:24
cgoncalvesstderr: 'Error finding address for http://192.168.24.1:9292/v2/images/d6f38bda-61a6-41c8-b509-53f0799ee1b2/file: Remote end closed connection without response'14:24
*** jmasud has joined #oooq14:25
chandankumarrlandy, cgoncalves can we get the bug up?14:27
*** ysandeep is now known as ysandeep|afk14:27
cgoncalveschandankumar, I can open one, sure14:28
*** TrevorV has joined #oooq14:30
rlandychandankumar: wonder if that is legit/repeatable14:36
rlandyother network jobs passed - fs010 is still running14:37
rlandychandankumar, marios: (assuming I am not on the do-not-fly list) I have a flight tomorrow morning so I will miss daily sync - will be on in US afternoon15:05
chandankumarrlandy, stay safe15:06
mariosrlandy: ack15:07
*** skramaja has quit IRC15:08
*** holser has quit IRC15:08
*** holser has joined #oooq15:09
rlandyhttps://review.rdoproject.org/r/#/c/25412/ - all passed15:11
rlandychandankumar: rekicked https://review.rdoproject.org/r/#/c/25325/ to be sure15:11
rlandycgoncalves: hi - did you create a bug yet, if not I'll do that now, rekicked ^^ to ensure the error is repro15:12
mariosweshay|ruck: rlandy: chandankumar: when you get a chance please https://review.opendev.org/#/c/713659/15:15
chandankumarmarios, one query in longer term, we are going to move to ansible operator, can we make it as a part of ansible operator?15:18
chandankumarweshay|ruck, 6615:18
chandankumarweshay|ruck, ^^15:18
rlandymarios: pl see commnet15:18
rlandycomment15:18
*** ratailor has quit IRC15:21
mariosrlandy: ack thanks done15:21
marioschandankumar: if you mean move the deployment of free-ipa to use an operator then yes that sounds like a fine idea15:22
chandankumarmarios, I am thinking forward to redo the work, since we have undercloud deployment working from operator working, we may want to take a look there15:23
chandankumar*instead of redoing the work15:23
chandankumar*from operator15:24
*** holser has quit IRC15:28
*** rlandy is now known as rlandy|brb15:28
*** holser has joined #oooq15:29
*** Trevor_V has joined #oooq15:37
cgoncalvesrlandy|brb, sorry, I have not yet. I've been in meetings for the past hour15:38
cgoncalvesgo ahead. thank you15:39
*** TrevorV has quit IRC15:40
*** rlandy|brb is now known as rlandy15:50
*** bogdando has quit IRC15:50
*** ysandeep|afk is now known as ysandeep15:52
chandankumarrlandy, weshay|ruck marios see ya tomorrow, stay safe16:02
*** chandankumar is now known as raukadah16:02
rlandyraukadah: you too - stay safe16:03
marioso/ rlandy16:03
mariossorry raukadah ^16:03
mariosbut also rlandy16:03
marioso/ to all the people16:03
rlandymarios: hey16:03
pandamarios: kthxbye16:05
rlandyhttps://review.rdoproject.org/zuul/stream/466beed43e984cdc946cc4edc16da522?logfile=console.log16:06
rlandyhttps://logserver.rdoproject.org/25/25325/19/check/periodic-tripleo-ci-centos-8-scenario010-standalone-network-master/466beed/job-output.txt16:12
rlandyraukadah: weshay|ruck: amoralej: ^^ this next run passed deploy and failed in tempest16:12
rlandyso we're getting somewhere now16:13
rlandyinvsetigating16:13
rlandyinvestigating16:13
amoralej2020-03-19 16:05:23 | 2020-03-19 16:05:23.740 128585 CRITICAL tempest [-] Unhandled error: config_tempest.services.base.ServiceError: Request on service 'load-balancer' with url 'http://192.168.24.1:9876' failed with code 50016:14
rlandyamoralej: may be timing16:14
raukadahamoralej, meed to check whether we have octavia support or not16:14
raukadahin tempestconf16:14
weshay|ruckrlandy, ah it worked the second time :)16:14
amoralej2020-03-19 16:07:19.022 26 ERROR octavia.api.drivers.driver_factory [-] Unable to load provider driver ovn due to: No 'octavia.api.drivers' driver found, looking for 'ovn': stevedore.exception.NoMatches: No 'octavia.api.drivers' driver found, looking for 'ovn'16:15
amoralejnot sure if that's an actual issue16:15
rlandywhat log?16:15
amoralejhttps://logserver.rdoproject.org/25/25325/19/check/periodic-tripleo-ci-centos-8-scenario010-standalone-network-master/466beed/logs/undercloud/var/log/containers/octavia/octavia.log.txt.gz16:15
amoralejthat's probably expected until we have the package for the new ovn-octavia-provider16:16
amoralejit's wip https://review.rdoproject.org/r/#/c/25972/16:17
amoralejbut i'd expect we could live with amphora without ovn16:17
amoralejcgoncalves, ^16:17
amoraleji mean, without ovn driver in octavia16:17
amoralejapi_settings.enabled_provider_drivers = {'amphora': 'The Octavia Amphora driver.', 'octavia': 'Deprecated alias of the Octavia Amphora driver.', 'ovn': 'Octavia OVN driver.'}16:18
*** johnsom has joined #oooq16:18
cgoncalvesthe OVN provider driver is auto-enabled if deployment is ML2/OVN16:18
amoralejmmmm16:18
amoralejis there any option in tripleo to disable it?16:19
cgoncalvessome folks wanted OVN driver enabled out of the box...16:19
amoralejyeah, make sense to me16:19
amoralejit's just timing issue16:19
cgoncalvesI wish there was an option16:19
amoraleji think it's fine, but *right now* there is no package for ovn-octavia-provider yet16:19
cgoncalvesis there an ETA for having it?16:20
amoralejcgoncalves, https://review.rdoproject.org/r/#/c/25972/ maciej is working on it16:20
cgoncalvesI see. thanks!16:22
cgoncalvesI guess/hope it can be merged still this week16:22
amoralejcgoncalves, would it work if we install the old networking-ovn there?16:23
amoralejmmm16:23
cgoncalvesamoralej, it would (assuming the driver is still there in the master branch)16:23
cgoncalveshttps://github.com/openstack/networking-ovn/tree/master/networking_ovn/octavia16:23
cgoncalveslooks like so16:24
amoralejhttps://github.com/openstack/networking-ovn/blob/master/setup.cfg#L61-L6216:24
amoralejit's this, right?16:24
amoralejrlandy, raukadah would it make sense to remove this job from promotion criteria until we have the new package?16:25
amoralejdo we have coverage for non-octavia services in other jobs for network component?16:26
raukadahamoralej, I am totally ok with that16:26
raukadahweshay|ruck, rlandy ^^16:26
rlandyamoralej: we could ...16:26
rlandyit will fail integration though16:26
rlandybut we might as well let this content hit integration16:26
rlandywe still have https://review.rdoproject.org/r/#/c/25996/16:27
weshay|ruckraukadah, if the rpm passed this time.. yes16:27
rlandyweshay|ruck: raukadah: let's do this16:27
weshay|ruckwas there another change?16:27
rlandyhttps://review.rdoproject.org/r/#/c/25996/16:28
rlandyweshay|ruck: ^^ pls remove your -216:28
rlandyI will revert as soon as well merge16:28
rlandyweshay|ruck: no other chang16:28
rlandyprevious error was a 50016:28
amoralejweshay|ruck, missing new package in octavia image https://review.rdoproject.org/r/#/c/25972/16:28
amoralejcollateral damage of ovn moved to neutron16:29
rlandyand the ETA on that is this week??16:29
amoralejwill also need patch in kolla btw16:29
amoraleji push quite much to ovn team16:29
rlandywe're failing everything now in integration - my vote is to let it go16:29
rlandyand deal with OVN16:29
rlandymuch smaller set of failures16:29
amoralejit's in their shoulders, i think they need an upstream patch too16:30
rlandyI +2'ed https://review.rdoproject.org/r/#/c/25996/16:30
weshay|ruckamoralej, ah. that is something update won't handle :)16:30
amoralejhttps://review.opendev.org/#/c/713685/16:30
amoralejweshay|ruck, no, not update case :)16:30
rlandyI assume we don't need another bug?16:30
amoralejnet new package16:30
rlandythis is expected16:30
amoralejworkaround would be to disable ovn provider16:30
amoralejwhich i'm not sure if it's possible16:31
amoralejin octavia16:31
amoralejapparently not16:31
amoralejbeagles, ^16:32
johnsomYou can in the Octavia.conf (upstream it is not enabled until that driver is installed). Maybe this is a limitation of the configuration tooling?16:32
amoralejjohnsom, what option in octavia.conf?16:32
johnsomamoralej https://github.com/openstack/octavia/blob/master/etc/octavia.conf#L4716:33
*** dsneddon_ has joined #oooq16:38
amoralejthere is no option to disable it in puppet-tripleo16:42
*** dsneddon_ has quit IRC16:44
* marios out see you tomorrow folks 16:47
mariosenjoy day off those that chose friday ;)16:47
beaglescorrect we auto-enable the driver agent and setup the config when deploying ovn16:50
beaglesamoralej, johnsom ^16:50
weshay|ruckamoralej, going to try w/ just changing ansible first https://review.rdoproject.org/r/2600916:51
weshay|ruckamoralej, feel free to update that16:51
amoralejweshay|ruck, ok16:51
beaglescgoncalves, ^ you should probs be aware of this as well16:51
*** marios is now known as marios|out16:51
cgoncalvesbeagles, your msg of ovn being auto-enabled or weshay|ruck msg?16:52
amoralejweshay|ruck, i think https://bugs.launchpad.net/tripleo/+bug/1867608 is unrelated to ceph-ansible but good to test16:52
openstackLaunchpad bug 1867608 in tripleo "Error in keystone resources in tripleo-ansible with ansible 2.9" [Undecided,New]16:52
beaglescgoncalves,  do we need a heat parameter to allow us to enable/disable this so instead of removing scenario010 from voting or knocking it out of periodic/promotion when the ovn provider breaks stuff16:53
beaglescgoncalves, just that its currently messed really16:53
beaglescgoncalves, I assume you already know we automagically enable the driver agent if ovn16:53
beagles ;)16:53
cgoncalvesbeagles, yeah, I know. I commented earlier :)16:53
* beagles scrolls up 16:54
beagleswhoops sorry16:54
beaglesdoh16:54
*** jmasud has quit IRC16:54
amoralejweshay|ruck, i'll add more jobs to your review16:54
cgoncalveshmm. parameter to disable auto-enabled providers like amphora and now ovn makes sense16:54
cgoncalveslike if someone is on a commercial LB solution, probably doesn't care about amphora and ovn16:55
beaglescgoncalves,  true16:55
beaglescgoncalves, it would also be an option for the case we were discussing earlier this week16:55
beagless/option/solution16:55
*** jmasud has joined #oooq16:56
cgoncalvescorrect, even though I can think of one person that is strongly against taking that path even if temporary16:56
beagles:)16:56
beaglesregardless of context - it seems like it might be a mechanism that would be useful in times like this - we'd just tweak scenario environment file16:57
cgoncalves+116:59
rlandywhat's the timeframe on ^^?17:00
rlandywe're deciding whether to let the current code hit the integration pipeline17:01
beaglesrlandy, I'm working on it now.. should have a patch proposed in the next hour or so17:01
rlandybeagles: great - we can test it as soon as you have it17:02
*** marios|out has quit IRC17:07
weshay|ruckarxcruz|rover, you have to follow up your own patches man https://review.opendev.org/#/c/713676/17:08
arxcruz|roverweshay|ruck: my bad, I saw the error, but got distracted by something else17:08
arxcruz|rover:/17:08
weshay|ruckarxcruz|rover, do a check-rdo17:08
rlandyperiodic-tripleo-ci-centos-8-multinode-1ctlr-featureset030-master - why is that job waiting?17:26
*** rascasoft has quit IRC17:29
rlandyarxcruz|rover: weshay|ruck: did zuul have a stroke on rdo?17:30
rlandy17 hr 2017:30
rlandywaiting for fs030 to kick'17:30
weshay|ruckhttp://dashboard-ci.tripleo.org/d/cockpit/cockpit?orgId=1&fullscreen&panelId=39917:31
* weshay|ruck looks at rdo17:31
weshay|ruckah.. the periodic job17:32
rlandyweshay|ruck: asking on #rhos-ops17:33
rlandyit's hanging there17:33
rlandyI can't tell why from our view17:34
amoralejweshay|ruck, scenario001 fails because ceph-ansible desn't support 2.8:17:51
amoralej2020-03-19 17:38:21 |   - 'fatal: [standalone]: FAILED! => changed=false '17:51
amoralej2020-03-19 17:38:21 |   - '  msg: Ansible version must be 2.8!'17:51
amoralejthe others are failing with same issue17:52
sshnaidmraukadah, fyi https://github.com/containers/libpod/pull/544917:55
amoralejsame issue == https://bugs.launchpad.net/tripleo/+bug/186760817:55
openstackLaunchpad bug 1867608 in tripleo "Error in keystone resources in tripleo-ansible with ansible 2.9" [Undecided,New]17:55
*** derekh has quit IRC18:05
weshay|ruckrlandy, https://review.opendev.org/#/c/712665/18:15
*** jpena is now known as jpena|off18:16
*** holser has quit IRC18:18
*** dsneddon_ has joined #oooq18:19
rlandyfails on openstack/tripleo-heat-templates only18:20
*** dsneddon has quit IRC18:20
*** dsneddon has joined #oooq18:21
*** dsneddon has quit IRC18:22
*** dsneddon has joined #oooq18:22
*** dsneddon has quit IRC18:22
*** dsneddon has joined #oooq18:22
*** holser has joined #oooq18:39
zbrhttps://review.opendev.org/#/c/713963/18:55
beaglesrlandy, this is the patch I'm testing atm https://review.opendev.org/#/c/713960/19:09
beaglesrlandy, so far I've only been able to pep8 it so it might bomb miserably19:09
beagles(having some trouble getting an env up and running)19:09
*** dsneddon has quit IRC19:15
*** dsneddon has joined #oooq19:16
rlandybeagles: let's give it a try19:27
rlandyas a depends on19:27
*** holser has quit IRC19:32
*** holser has joined #oooq19:33
*** sshnaidm is now known as sshnaidm|off19:33
*** Trevor_V has quit IRC19:41
rlandybeagles: this is the deploy failure the test returned so far: https://logserver.rdoproject.org/25/25325/20/check/periodic-tripleo-ci-centos-8-scenario010-standalone-network-master/fcc7285/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz20:00
rlandyhttps://review.rdoproject.org/r/#/c/25325/20:00
rlandy^^ testproject review20:00
rlandyfeel free to rekick it if you want to test another change to that patch20:00
beaglesrlandy, thanks!20:08
*** chem has quit IRC20:09
rlandyweshay|ruck: https://review.rdoproject.org/r/26015 Dlrn-role condition should be switched on os version20:11
rlandy^^ quick review20:12
rlandyfor bm jobs20:12
*** TrevorV has joined #oooq20:12
*** chem has joined #oooq20:13
*** TrevorV has quit IRC20:18
*** jmasud has quit IRC20:54
*** jmasud has joined #oooq20:56
*** dsneddon_ has quit IRC20:57
weshay|ruckrlandy, added +220:59
rlandyweshay|ruck: thanks ...21:01
rlandybaremetal in play ...21:01
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/stream/5da6ff2792784407916407ef0ef0f025?logfile=console.log21:01
weshay|ruckreallly.. nice21:01
rlandylet's see how far we get21:01
beaglesrlandy, I have to step away for a bit - I'm getting stumbled by some heat template syntax atm but will get bak at it later21:40
*** holser has quit IRC21:43
rlandybeagles: no worries21:47
rlandyif we need to, we can skip this test and merge the content21:48
*** jfrancoa has quit IRC22:01
*** holser has joined #oooq22:11
*** holser has quit IRC22:12
*** holser has joined #oooq22:13
*** dpawlik has quit IRC22:38
*** jbadiapa has quit IRC22:50
*** apetrich has quit IRC22:50
*** holser has quit IRC23:04

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!