Thursday, 2018-12-20

« Wednesday, 2018-12-19 Index Friday, 2018-12-21 »
*** tosky has quit IRC00:43
*** ykarel|away has joined #oooq02:12
*** ykarel|away has quit IRC02:24
*** ykarel|away has joined #oooq02:25
*** saneax has joined #oooq02:27
*** ykarel|away has quit IRC02:29
*** rlandy|rover|bbl is now known as rlandy|rover02:55
*** rlandy|rover has quit IRC03:04
*** ykarel|away has joined #oooq03:14
*** ykarel|away has quit IRC03:18
*** ykarel|away has joined #oooq03:18
*** ykarel|away has quit IRC03:29
*** ykarel has joined #oooq03:41
*** ykarel has quit IRC05:10
*** ratailor has joined #oooq05:29
*** ykarel has joined #oooq05:36
*** chkumar|vanished is now known as chandankumare05:42
*** chandankumare is now known as chandankumar05:42
*** ccamacho has quit IRC06:47
*** quiquell|off is now known as quiquell07:02
*** holser_ has joined #oooq07:41
*** quiquell is now known as quiquell|brb07:49
*** gkadam has joined #oooq07:49
*** rascasoft has joined #oooq08:03
*** ccamacho has joined #oooq08:03
*** jfrancoa has joined #oooq08:05
*** kopecmartin|off is now known as kopecmartin08:06
*** ykarel is now known as ykarel|lunch08:11
*** ccamacho has quit IRC08:13
*** ccamacho has joined #oooq08:13
*** quiquell|brb is now known as quiquell08:21
*** tosky has joined #oooq08:24
*** jpena|off is now known as jpena08:31
*** amoralej|off is now known as amoralej08:38
*** ccamacho has quit IRC08:49
*** ykarel|lunch is now known as ykarel08:59
*** rascasoft has quit IRC09:04
*** ccamacho has joined #oooq09:13
*** derekh has joined #oooq09:29
*** brault has joined #oooq09:36
*** skramaja has joined #oooq09:38
*** rascasoft has joined #oooq09:49
quiquellpanda, weshay|ruck, sshnaid: Talked trying to gate podman with tripleo, amoralej, ykarel are doing it09:53
*** skramaja_ has joined #oooq09:54
*** skramaja has quit IRC09:54
*** skramaja_ has quit IRC09:59
*** skramaja_ has joined #oooq09:59
mariosquiquell: here?10:29
*** dsneddon has quit IRC10:32
sshnaidmquiquell, in github?10:46
quiquellsshnaidm: They are doing at RDO using deps.yml10:58
quiquellsshnaidm: so we take podman from there10:58
quiquellsshnaidm: ebst solution directly at centos packaging, but this will take time10:59
mariosquiquell: o/10:59
quiquellmarios: \o10:59
mariosquiquell: i am looking at rdo jobs for the upgrade extra_tags k/? just checking if you already did?10:59
mariosjfrancoa: ^ ?10:59
*** dsneddon has joined #oooq11:00
quiquellmarios: nope, wask busy looking at gating podman somewhere11:02
*** kopecmartin is now known as kopecmartin|off11:02
quiquellmarios: need help to find the jobs at RDO repos ?11:02
mariosquiquell: no lemme post some stuf and see whats missing11:03
mariosquiquell: thanks11:03
marios(just checking to prevent duplicate)11:03
quiquellmarios: ack11:03
*** dsneddon has quit IRC11:05
*** dsneddon has joined #oooq11:06
jfrancoamarios: ack, I didn't start with it either11:09
mariosthanks jfrancoa just checking11:09
mariosquiquell: vote at https://review.openstack.org/#/c/626546/1 https://review.openstack.org/#/c/626511/211:10
mariosjfrancoa: you too man your mistake showing up here ^11:10
mariosplease thanks if you have time and anyone else lurking here please do it ^11:11
*** dsneddon has quit IRC11:11
chandankumarrf0lc0: panda https://code.engineering.redhat.com/gerrit/#/c/159204/ removing rally from our bug list11:15
*** saneax has quit IRC11:29
mariosjfrancoa: quiquell fyi https://review.rdoproject.org/r/#/c/17962/ thanks for reviews when you next have time for that11:29
*** saneax has joined #oooq11:29
jfrancoamarios: thanks, I'll have a look at it11:29
mariosty jfrancoa11:32
*** dsneddon has joined #oooq11:40
*** dsneddon has quit IRC11:46
*** udesale has joined #oooq11:56
*** jpena is now known as jpena|lunch11:59
*** ratailor has quit IRC12:07
*** holser_ is now known as holser|lunch12:09
*** udesale has quit IRC12:11
*** udesale has joined #oooq12:12
*** ccamacho has quit IRC12:14
*** udesale has quit IRC12:15
*** udesale has joined #oooq12:16
*** dsneddon has joined #oooq12:16
*** dsneddon has quit IRC12:20
*** udesale has quit IRC12:21
*** udesale has joined #oooq12:21
quiquellsshnaidm: project for the repro https://review.rdoproject.org/r/#/c/17968/12:33
*** rf0lc0 is now known as rfolco12:39
rfolcochem, o/ did you get any progress on the standalone-update ? I may have some cycles to work on it, so I'd like to help.12:40
quiquellrf0lc0 is the 00fline folco ?12:42
rfolcoquiquell, 2nd nick option :)12:43
*** dsneddon has joined #oooq12:48
*** udesale has quit IRC12:49
*** udesale has joined #oooq12:50
*** udesale has quit IRC12:52
*** udesale has joined #oooq12:52
*** dsneddon has quit IRC12:53
*** udesale has quit IRC12:53
*** rlandy has joined #oooq12:53
*** udesale has joined #oooq12:53
*** udesale has quit IRC12:54
weshay|ruckchandankumar, do you have a minute to look at https://bugs.launchpad.net/tripleo/+bug/180924812:54
openstackLaunchpad bug 1809248 in tripleo "Tempest server tests for standalone job fail because dhcp-agent failures" [Critical,Triaged] - Assigned to Ronelle Landy (rlandy)12:54
weshay|ruckplease12:54
*** udesale has joined #oooq12:55
quiquellweshay|ruck: Asked at RDO about gating podman with tripleo12:55
weshay|ruckquiquell, where can I see that gating?12:55
quiquellweshay|ruck: amoralej is working on it, will add it to rdoinfo12:55
quiquellweshay|ruck: It means we have to maintain it12:55
weshay|ruckquiquell, maintain it? or just pin it?12:56
quiquellpin it12:56
weshay|ruckya.. that's different12:56
quiquellLet me find the review amoralej is working on12:56
chandankumarweshay|ruck: looking12:56
quiquellweshay|ruck: https://review.rdoproject.org/r/#/q/topic:gate-podman12:57
rlandyweshay|ruck: hello12:57
quiquellBut final solution is podman packagers at centos gate it with tripleo12:57
weshay|ruckhttps://review.rdoproject.org/r/#/c/17960/12:57
weshay|ruckquiquell, ya12:57
quiquellMeanwhile good to do some tripleo before podman arrives12:57
rfolcoweshay|ruck, scrum is by mistake on today's cal12:58
*** ccamacho has joined #oooq12:58
*** ccamacho has quit IRC12:58
amoralejquiquell, weshay|ruck see https://trello.com/c/21zwXgcP/695-ci-gate-podman-updates-with-tripleo-job12:58
*** ccamacho has joined #oooq12:58
rfolcoweshay|ruck, you own scrum mtg btw12:58
amoraleji linked both reviews there12:58
quiquellamoralej: thanks so much12:59
weshay|ruckamoralej++12:59
amoralejquiquell, where should it fail now?12:59
amoralejdeploy overcloud?12:59
quiquellamoralej: let me check12:59
* weshay|ruck gets coffee13:00
quiquellrlandy: o/13:00
quiquellrlandy: I am creating a new project at RDO for the repro ansible role https://review.rdoproject.org/r/#/c/17968/13:01
mariosscrum?13:01
mariosplanning call13:01
marios?13:01
mariosrfolco: ?13:01
mariospanda: ?13:01
quiquellDamn... haven't lunch13:01
* marios foreveralone13:01
quiquellWE have the nodepool images stuff today13:02
rlandyquiquell: yes - in about 30 mins13:02
quiquellrlandy: forgot about it13:02
rlandymarios: https://review.openstack.org/#/c/626400/13:03
rlandyyou ok with that?13:03
mariosrlandy: looking13:04
rfolcomarios, planning call starts in one hour from now13:04
mariosrfolco: ack there was also scrum in the calendar anyway me and quiquell dropped just now13:04
rfolcoscrum should not be on cal, we'll need to fix this13:04
rfolcoweshay|ruck, ^13:04
mariosrlandy: i haven't reviewed that before, will add to check in a bit/later is that ok? or is it something urgetn?13:05
rfolcoignore scrum that is on planning day for now13:06
rlandymarios: sure13:06
*** ccamacho has quit IRC13:06
weshay|ruckamoralej, quiquell what prompted you guys to pin podman?13:08
chandankumarweshay|ruck: I am not sure http://logs.openstack.org/71/625971/2/gate/tripleo-ci-centos-7-standalone/f98cd76/logs/undercloud/var/log/containers/neutron/l3-agent.log.txt.gz#_2018-12-20_09_34_43_609 it is related to this13:09
quiquellweshay|ruck: https://bugs.launchpad.net/tripleo/+bug/180921813:09
openstackLaunchpad bug 1809218 in tripleo "podman jobs failing after new podman(0.12.1) and container-selinux(2.77) are released" [Undecided,In progress] - Assigned to yatin (yatinkarel)13:09
amoralejweshay|ruck, quiquell propted me :)13:09
quiquellNew podman version breaking everything13:09
weshay|ruckheh.. so does the old one13:09
amoralejit seems last update broke again, iiur13:09
mariosrlandy: weshay|ruck forgot to say i added note at https://review.rdoproject.org/etherpad/p/ruckrover-sprint23 for the selinux bug cos it blocks the promotion blocker13:10
mariosrlandy: wasn't sure if you were using a new etherpad? couldn't see one in the ci scrum ether13:10
weshay|ruckmarios, thanks will look in a second13:10
weshay|ruckamoralej, quiquell I think we have to consider turning podman off again13:10
amoraleji don't like pin podman in deps.yml but i understand it's mitigation13:10
weshay|ruck.11 was broken, .12 is broken13:10
ykarellast time it was container-selinux iirc13:11
amoralejweshay|ruck, i will not complaint if you do :)13:11
chandankumarweshay|ruck: sshnaidm it is failing due to podman https://bugs.launchpad.net/tripleo/+bug/180921813:11
openstackLaunchpad bug 1809218 in tripleo "podman jobs failing after new podman(0.12.1) and container-selinux(2.77) are released" [Undecided,In progress] - Assigned to yatin (yatinkarel)13:11
weshay|ruckamoralej, pinning only really helps if there is a known good version13:11
rlandymarios: k - will transfer to current etherpad13:11
chandankumarweshay|ruck: quiquell and marios seen it in the morning13:11
weshay|ruckk.. sec13:11
amoralejafaik the build before last update was working fine, quiquell am i right?13:11
quiquellamoralej: yep13:12
ykarelamoralej, it was working, but have random failures13:12
weshay|ruckamoralej, we'll have to wait.. Emilien is PTO and it's too early for Alex13:12
ykarelso for that 0.12.1 is proposed13:12
weshay|ruckykarel++13:12
ykarelweshay|ruck, i heared Alex is also out till Jan13:12
weshay|ruckyes, random failures resetting the gate all the time13:12
*** ccamacho has joined #oooq13:12
weshay|ruckdang it13:12
weshay|ruckok.. guys give me a few minutes13:13
quiquellweshay|ruck: For podman we need to test upgrades of it before adding it to the mix13:13
quiquellweshay|ruck: are going back and forth podman/docker13:13
quiquellwe a lot of energy in lose in the middle13:13
*** rlandy is now known as rlandy|rover13:14
quiquellbetter gate new versions, that was my line of thinking13:14
sshnaidmchandankumar, cool, I'll close my as dup13:14
chandankumarsshnaidm: thanks!13:14
quiquellccccccjgkgvcjjlkvfvrikkcgkbdenvhgehckgurfecg13:14
quiquellups13:15
mariosquiquell: time to change keys?13:15
marios:D13:15
sshnaidmchandankumar, so podman doesn't support selinux relabel?13:15
sshnaidmquiquell, is it a cat?13:15
quiquellmy yubi just arrived, broke the other one13:15
*** trown|outtypewww is now known as trown13:15
marioshaha quiquell never tried one of those maybe i should in the new year like a resolution :)13:15
mariosi mean at most i have to do that like 2 x per day?13:16
quiquellmarios: they are super convenient13:16
mariosso i have never been entirely sold i guess if you use it a lot more then yes absolutely makes sense no doubt13:16
*** jpena|lunch is now known as jpena13:16
quiquellmarios: well... I am kind of early adopter on things, cannot help it13:17
sshnaidmquiquell, so, how do rdo folks gate podman with tripleo jobs?13:17
weshay|ruckmarios, the yubi keys are awesome .. cheers quiquell  ccccccdlvuefehdihfutjcjjlfjlukfdjdulibfhcvjg13:17
marios:)13:17
weshay|rucksshnaidm, quiquell it's more complicated than that13:18
mariosi wish i could spew out random nonsense at will13:18
weshay|ruckyou guys dont want to know13:18
mariosall i want for christmas13:18
weshay|ruckwish, you do it all the time13:18
weshay|ruckZING13:18
marioswell i'd settle for the laptop first :)13:18
weshay|ruckmarios, any news there?13:18
mariosweshay|ruck: no13:18
mariospokerface13:18
marioslets see13:18
mariosi hope its just 'its xmas'13:19
chandankumarweshay|ruck: feel free to check comment https://tree.taiga.io/project/tripleo-ci-board/task/52413:19
mariosweshay|ruck: looks like the workflowy stuff is happening in the request anyway13:19
quiquellweshay|ruck: so we cannot gate podman ?13:20
*** amoralej is now known as amoralej|lunch13:24
*** gkadam has quit IRC13:25
*** dsneddon has joined #oooq13:26
weshay|ruckquiquell, amoralej|lunch sshnaidm if you want to chat about it13:26
weshay|ruckI'll be on my blue13:26
rlandy|rovermarios: about the zaqar failure, the patch failed fs01013:27
sshnaidmweshay|ruck, I have a different mtg now13:27
weshay|rucksshnaidm, ah ya13:27
rlandy|roverweshay|ruck: we are meeting re: nodepool images13:27
weshay|ruckso do I13:27
weshay|rucksshnaidm, quiquell I'll chat w/ everyone about dep packages and gating after the planning13:27
quiquellweshay|ruck: ack13:28
mariosrlandy|rover: thanks will check ... previous run was failing on the docker/selinux issue for both standalone and containers-multinode  i commented on the bug13:28
rlandy|rovermarios: looks like same failure to me - only required after promotion?13:28
weshay|ruckundercloud_container_cli: >-13:33
weshay|ruck  {% if release in ['rocky'] -%}13:33
weshay|ruck  docker13:33
weshay|ruck  {%- else -%}13:33
weshay|ruck  podman13:33
weshay|ruck  {%- endif -%}13:33
weshay|ruckso.. 226container_cli="{{ undercloud_container_cli | default('podman') }}"13:34
*** dsneddon has quit IRC13:34
*** holser|lunch is now known as holser_13:51
* chandankumar will be late for meeting for 5-10 mins13:52
*** dsneddon has joined #oooq13:53
rlandy|rovermarios: maybe diff error ... https://review.openstack.org/#/c/625256/ ... https://logs.rdoproject.org/56/625256/8/openstack-check/tripleo-ci-centos-7-multinode-1ctlr-featureset010/98bf1cb/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-12-20_00_45_3214:01
*** dsneddon has quit IRC14:01
rlandy|rover2018-12-20 00:45:30 |         "2018-12-20 00:45:30,409 ERROR: 25923 -- ERROR configuring mysql",14:01
rlandy|rover2018-12-20 00:45:30 |         "2018-12-20 00:45:30,410 ERROR: 25923 -- ERROR configuring mistral14:01
quiquellrfolco: do you have the blue link ?14:01
rfolcoquiquell, https://bluejeans.com/587845809714:02
rlandy|roverweshay|ruck: ^^ we still have an issue with fs010 in the patch to fix zaqar14:02
weshay|ruckrlandy|rover, did the patch merge?14:02
weshay|ruckrlandy|rover, check the etherpad14:02
rlandy|roverweshay|ruck: no - failed standlone - checking that14:03
*** udesale has quit IRC14:05
*** jrist has quit IRC14:05
*** udesale has joined #oooq14:06
*** ykarel is now known as ykarel|away14:07
*** skramaja_ has quit IRC14:08
*** amoralej|lunch is now known as amoralej14:10
*** jrist has joined #oooq14:22
*** ykarel|away has quit IRC14:26
*** dsneddon has joined #oooq14:38
*** dsneddon has quit IRC14:42
*** ykarel|away has joined #oooq14:55
rlandy|rovermarios: sorry to keep bugging you ... https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-singlenode-featureset050-upgrades-master/68f5fcc/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz#_2018-12-20_12_32_2215:02
rlandy|roverdidn't see anything related on the sprint 23 etherpad15:02
rlandy|rovernew?15:03
mariosrlandy|rover: yeah, i have seen that once (looking for the comment to point at) but i didn't file it15:04
rlandy|rovermarios: k - looking through the history - thanks15:04
*** dsneddon has joined #oooq15:18
rlandy|roverweshay|ruck: fyi ... creating bug for ^^15:21
*** dsneddon has quit IRC15:22
rlandy|roverchem: does this look familiar to you https://bugs.launchpad.net/tripleo/+bug/1809268? otherwise will look into it more15:27
openstackLaunchpad bug 1809268 in tripleo "Promotion failure: periodic-tripleo-ci-centos-7-singlenode-featureset050-upgrades-master is failing on installing docker packages - templating error" [Critical,Triaged]15:27
chemrlandy|rover: hey, looking15:28
chemrlandy|rover: looking like bad ansible15:30
jfrancoachem: rlandy|rover it should be fixed by this https://review.openstack.org/#/c/625903/15:30
rlandy|roverjfrancoa: thanks - will mark the bug as duplicate15:31
jfrancoarlandy|rover: no problem15:31
*** dsneddon has joined #oooq15:51
*** dsneddon has quit IRC15:56
*** dsneddon has joined #oooq16:00
quiquellweshay|ruck: So we don't continue with adding podman to rdoinfo and test it there with tripleo ?16:08
*** udesale has quit IRC16:11
*** quiquell is now known as quiquell|off16:12
*** ccamacho has quit IRC16:24
rlandy|roverweshay|ruck: pls see line 13 on https://review.rdoproject.org/etherpad/p/ruckrover-sprint24 - comments on failures on that review - worth rechecking?16:27
rlandy|roverchandankumar: looking into https://bugs.launchpad.net/tripleo/+bug/1806495 - afaict the only fix associated is to try track this better. do you know anything more about this tempestmail error?16:49
openstackLaunchpad bug 1806495 in tripleo "validate-tempest role erroring trying to send mail" [High,In progress] - Assigned to Sorin Sbarnea (ssbarnea)16:49
rlandy|roverthe bug is all about tracking - not about fixing - if that is possible16:50
*** jfrancoa has quit IRC17:05
*** ccamacho has joined #oooq17:09
rlandy|roversshnaidm: do you have access to tempestmail?17:11
sshnaidmrlandy|rover, nope17:11
rlandy|roversshnaidm: there are a lot of tempest mail access issues17:11
rlandy|roverthe instance is in a strange state17:11
rlandy|rover17:11
rlandy|roverConfirm or Revert Resize/Migrate17:11
rlandy|rovernot sure what's going on there17:12
rlandy|roverpublic ip is pingable17:12
sshnaidmrlandy|rover, I saw some, but it doesn't seems as an issue17:12
rlandy|roversshnaidm: it's failing a lot now17:12
rlandy|rovergate failures17:12
rlandy|roverthe only bug I can find i=has info on tracking the issue17:12
rlandy|rovernot fixing it17:12
sshnaidmrlandy|rover, afaik it can't fail gate17:12
rlandy|roverhttps://bugs.launchpad.net/tripleo/+bug/180649517:13
openstackLaunchpad bug 1806495 in tripleo "validate-tempest role erroring trying to send mail" [High,In progress] - Assigned to Sorin Sbarnea (ssbarnea)17:13
sshnaidmrlandy|rover, I think the errors are ignored17:14
sshnaidmrlandy|rover, anyway, can you login to: ssh centos@38.145.34.221 ?17:14
rlandy|roversshnaidm: you are right - that is not causing direct failure but  we are still seeing it17:15
chandankumarrlandy|rover: I also donot have access to the instance17:16
chandankumarrlandy|rover: but I will take a look at the bug17:16
sshnaidmrlandy|rover, added you key there, you can login17:16
rlandy|roverchandankumar: thanks17:17
rlandy|rovermaybe we should just increase timeout17:17
rlandy|roverchandankumar: looks like a lot of the times that error shows up, is because tempest failed previously17:22
rlandy|rovermaybe not worth spending too much time on?17:22
chandankumarrlandy|rover: when atest fail it tries to send email17:23
rlandy|roverchandankumar: ok - so maybe that is the cause17:23
* rlandy|rover has been out of ruck/rovering for too long17:23
chandankumarrlandy|rover: yes not worthit to spend too much time currently17:23
rlandy|roverthansk - just following up on repeating failures17:24
*** trown is now known as trown|lunch17:29
*** gouthamr has quit IRC17:31
*** dmellado has quit IRC17:31
weshay|ruckrfolco, please schedule a follow up on the packaging discussion for tomorrow17:37
*** gouthamr has joined #oooq17:40
rlandy|roverrdo cloud outage17:43
*** jpena is now known as jpena|off17:44
weshay|ruckrlandy|rover, srsly?17:51
rlandy|roverweshay|ruck: do I usually joke about things like this?17:51
rlandy|roverweshay|ruck: and I left some notes in the now unaccessible etherpad17:51
*** derekh has quit IRC17:52
weshay|ruckrlandy|rover, I was going to mention in your goals, you need to joke more often17:52
weshay|ruckloz17:52
weshay|rucklolz17:52
rlandy|roverduly noted17:52
*** panda is now known as panda|off17:53
rlandy|roverok - while the great reboot is going on - need to run out for a few17:55
rlandy|roverbiab17:55
rlandy|roverweshay|ruck: ^^17:55
weshay|ruckrlandy|rover, no prob17:55
*** rlandy|rover is now known as rlandy|rover|bia17:55
weshay|ruckget lunch too :)17:55
weshay|ruckpanda|off, rfolco this is closer to reality now https://tree.taiga.io/project/tripleo-ci-board/epics18:10
*** ykarel|away has quit IRC18:11
*** sshnaidm is now known as sshnaidm|off18:19
*** holser_ has quit IRC18:22
weshay|ruckrfolco, https://drive.google.com/drive/u/1/folders/1duziyA5IpQLrzs4JncjKm6FHLOlDh5_-18:29
weshay|ruckrfolco, don't forget :)18:29
rfolcoweshay|ruck, yes, working on it18:36
*** gouthamr_ has joined #oooq18:37
*** gouthamr has quit IRC18:37
*** dmellado has joined #oooq18:40
*** trown|lunch is now known as trown18:43
*** hubbot1 has joined #oooq18:44
*** rlandy|rover|bia is now known as rlandy|rover18:46
rlandy|roverah rdocloud back  - nice18:47
rlandy|rover19 h queue18:55
rlandy|roverweshay|ruck: any thoughts on     https://bugs.launchpad.net/tripleo/+bug/1809280 Periodic jobs are not using podman on the OverCloud?19:00
openstackLaunchpad bug 1809280 in tripleo "Periodic jobs are not using podman on the OverCloud" [High,In progress] - Assigned to Ronelle Landy (rlandy)19:00
rlandy|rovernot sure this is the right fix even19:00
rlandy|roverhave testproject review in but will fail prior to that point19:01
weshay|ruckrlandy|rover wait what?19:01
rlandy|roverweshay|ruck: I kind if came in late to this discussion ... getting chat copy19:02
weshay|rucknot EVERYTHING should be an extra_args:  :(19:03
rlandy|roverhttp://pastebin.test.redhat.com/68685419:03
* weshay|ruck reads19:03
rlandy|roverweshay|ruck: ^^ I agree about extra_vars19:03
rlandy|roverit's a trap19:03
rlandy|roverpeople are throwing everything in there19:03
rlandy|roverand overriding settings not intended19:04
rlandy|roverwe should limit it19:04
rlandy|roverlike featureset_override19:04
weshay|ruckrlandy|rover, agree19:04
rlandy|roverweshay|ruck: the proposed solution has almost the same problem19:05
weshay|ruckrlandy|rover, anyway yatin and alfreado are wrong19:05
rlandy|roverin that it overrides19:05
weshay|ruckrlandy|rover, the only fs w/ it enabled on the overcloud is fs01019:05
weshay|ruck# extra_args: >-19:05
weshay|ruck#   {% if release not in ['newton', 'ocata', 'pike', 'queens', 'rocky'] -%}19:05
weshay|ruck#   -e {{ overcloud_templates_path }}/environments/podman.yaml19:05
weshay|ruck#   {%- endif -%}19:05
weshay|ruckoh crud19:06
* weshay|ruck reading bug19:06
rlandy|roverthey are commenting on fs01019:06
weshay|ruckrlandy|rover, well.. now they are19:06
weshay|ruckwe disabled podman19:06
rlandy|roverI know19:06
weshay|ruckrlandy|rover, so what we need now is a containers settings in overcloud_prep19:06
weshay|ruckand defaults19:06
rlandy|roverwhich is why it will be impossible to test19:07
rlandy|roverwe had those, I thought19:07
rlandy|roverwhen we only had overcloud containers19:07
* rlandy|rover looks into that19:07
rlandy|roveroh shoot - back to create_failed stacks19:08
weshay|ruckrlandy|rover, no ci review https://review.openstack.org/#/c/612526/19:08
rlandy|roverprobably outage19:09
rlandy|roverlovely19:09
rlandy|roverwell are they wrong or right?19:10
rlandy|roveris the bug really valid?19:10
weshay|ruckrlandy|rover, they are right.. but only because Emilien was lazy19:10
rlandy|roverfine19:10
weshay|ruckrlandy|rover, may be able to use https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-deploy/defaults/main.yml#L11819:11
weshay|ruckrlandy|rover, HA19:11
weshay|ruckor just fix it here19:11
weshay|ruckhttps://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-deploy/tasks/pre-deploy.yml#L18819:11
weshay|ruckfakers19:11
weshay|ruckrlandy|rover, doesn't need to go into args at all19:11
rlandy|roverthere is not stein/master there19:12
weshay|ruckjust add an else19:12
weshay|ruckright19:12
weshay|ruckjust need an else statement19:12
rlandy|rover {% if release in ['pike', 'queens', 'rocky'] %}19:12
rlandy|roverstupid condition19:12
* rlandy|rover fixes19:12
weshay|ruckand we'll remove it from the featureset19:12
weshay|ruckproblems w/ open source19:13
weshay|ruckit's hard to be every where all the time19:13
* rlandy|rover thinks about that impacts undercloud19:13
rlandy|roveralso - if we have another podman issue, we will have to disable it here19:15
rlandy|rovermaybe better to have a default19:15
rlandy|roverand override in the fs19:15
weshay|ruckrlandy|rover, oh wait19:15
weshay|ruckit's not ok to have the default be podman yet of course..19:16
weshay|ruckso we need override the entire setting19:16
weshay|ruckcontainer_args: >-19:16
weshay|ruck      {% if release in ['pike', 'queens', 'rocky'] %}Jiri Stransky, 2 months ago: • Stop using docker.yaml after it was deprecate…19:16
weshay|ruck      -e {{ overcloud_templates_path }}/environments/docker.yamlWes Hayutin, 2 years ago: • deployment updates for containerized compute19:16
weshay|ruck      {% endif %}Jiri Stransky, 2 months ago: • Stop using docker.yaml after it was deprecate…19:16
weshay|ruck      {% if enable_pacemaker|bool or osp_release is defined%}John Trowbridge, 1 year ago: • Always pass docker-ha env when passing docker…19:16
weshay|ruck      -e {{ overcloud_templates_path }}/environments/docker-ha.yamlJiri Stransky, 1 year ago: • Containerized deployment with pacemaker via O…19:16
weshay|ruck      {% endif %}19:16
weshay|ruck      {% if release in ['ocata', 'pike', 'queens', 'rocky'] %}Steve Baker, 7 months ago: • Switch to workflow driven image prepare19:16
weshay|ruck      -e {{ working_dir }}/containers-default-parameters.yamlWes Hayutin, 2 years ago: • deployment updates for containerized compute19:16
weshay|ruck      {% else %}Steve Baker, 7 months ago: • Switch to workflow driven image prepare19:16
weshay|ruck      -e {{ working_dir }}/containers-prepare-parameter.yaml19:16
weshay|ruck      {% endif %}19:16
rlandy|roverthat's kind of a confusing copy19:18
* rlandy|rover puts in a review - easier to work from there19:18
rlandy|rover      {% if enable_pacemaker|bool or osp_release is defined%}19:19
rlandy|rover      -e {{ overcloud_templates_path }}/environments/docker-ha.yaml19:19
rlandy|roverwrong as well19:19
rlandy|roverweshay|ruck: sec - just cleaning up create_failed mess in tenant19:20
weshay|ruckrlandy|rover, kk.. sec19:21
*** saneax has quit IRC19:23
rlandy|roverok  better19:24
weshay|ruckrlandy|rover, https://review.openstack.org/62666119:24
rlandy|roverdo we run newton at all now?19:25
rlandy|roveranything before pike?19:26
rlandy|roverleft some comments19:28
weshay|ruckrlandy|rover, no.. newton is dead19:40
rlandy|roverweshay|ruck: ok - so then go forth19:40
weshay|ruckya.. we'll see what CI says19:40
rlandy|roverwe will have to rethink it when osp gets podman19:41
rlandy|rover   {% if enable_pacemaker|bool or osp_release is defined%}19:41
rlandy|rover      -e {{ overcloud_templates_path }}/environments/docker-ha.yaml19:41
weshay|ruckrlandy|rover, ya..19:42
* rlandy|rover wonders why that condition actually exists19:43
rlandy|roverie: why osp would always use docker_ha19:44
rlandy|roverhttps://github.com/openstack/tripleo-quickstart-extras/commit/caa8a6287195c1f97bb9d7cf1f1fa2c8e927841819:44
rlandy|roverweird19:45
rlandy|rover"This does not break anything"19:45
rlandy|roverfamous last words19:45
rlandy|roverit always breaks something19:45
rlandy|rover Resource CREATE failed: resources.private_network: Request limit exceeded: You have reached the maximum stacks per tenant, 100. Please delete some stacks.20:01
weshay|ruckrfolco, I should be reviewing https://docs.google.com/document/d/1_daWFP5q7KKEeM2kZ5CE_4MWr0A2nXZGvL28oyTiriI/edit#heading=h.w6l1oa7hmg7u20:01
weshay|ruckrfolco, the title says 2220:01
rfolcohttps://docs.google.com/document/d/1_daWFP5q7KKEeM2kZ5CE_4MWr0A2nXZGvL28oyTiriI/edit#20:01
rfolcoweshay|ruck, ^20:02
weshay|ruckrfolco, ya.. 22 or 23 :)20:02
rlandy|roverwe are below the limit20:02
weshay|rucknote the title20:02
weshay|ruckrlandy|rover, 74 stacks?20:02
rlandy|roverweshay|ruck: yeah - I think that error was from earlier20:03
rlandy|roverI think ok now20:03
rlandy|roverwatching20:03
rfolcoweshay|ruck, oh the title.... fixed20:03
rlandy|roverwas just talking to myself on chat20:03
weshay|ruckrfolco, thanks20:03
weshay|ruckrlandy|rover, heh20:03
weshay|ruckrfolco, show me the disabled jobs20:07
rfolcoweshay|ruck, example scen3 https://review.openstack.org/#/c/623260/20:07
weshay|ruckrfolco, let's just do this on blue20:07
rfolcok20:07
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-standalone, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-containers-multinode @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7 (1 more message)20:08
rfolcoweshay|ruck, frozen20:09
weshay|ruckrlandy|rover, do you know the bug # on fedora 2820:10
rlandy|roverweshay|ruck: which f28?20:11
rlandy|roverah tripleo-ci-fedora-28-standalone20:12
rlandy|roverlooking20:12
*** gouthamr_ is now known as gouthamr20:13
rlandy|rover    why is the fedora 28 job failing in post? http://zuul.openstack.org/builds?job_name=tripleo-ci-fedora-28-standalone < wes >20:13
rlandy|roveronly mention of it in previous etherpad20:14
rlandy|roverweshay|ruck: I don't see a bug we carried for that20:19
rlandy|rovercan create one20:19
rlandy|roverhttps://bugs.launchpad.net/tripleo/+bug/1808118 should be completed20:20
openstackLaunchpad bug 1808118 in tripleo "f28 standalon fails at container prep" [Critical,In progress] - Assigned to Quique Llorente (quiquell)20:20
rlandy|roverah same bug20:22
rlandy|roverweshay|ruck: ^^20:22
rlandy|roversame error20:22
rlandy|roverbut now it fails in deploy20:22
rlandy|roverfixed should have been merged20:25
rlandy|roverChange has been successfully merged by Zuul20:25
rlandy|roverDec 19 8:48 AM20:25
rlandy|roverha - f28 still fails on the change20:26
rlandy|roverweshay|ruck: ok - so fix does not work20:27
weshay|ruckrfolco, https://tree.taiga.io/project/tripleo-ci-board/epic/41620:37
weshay|ruckso.. it's unclear to me at this time that anything was completed in tempest20:37
weshay|ruckhttps://tree.taiga.io/project/tripleo-ci-board/us/1920:38
rlandy|roverhttps://bugs.launchpad.net/tripleo/+bug/1808118 marked promotion-blocker20:50
openstackLaunchpad bug 1808118 in tripleo "f28 standalon fails at container prep" [Critical,In progress] - Assigned to Quique Llorente (quiquell)20:50
rlandy|roverdon;t think that is correct20:50
rlandy|roverweshay|ruck: is f28 job fix a rover or a f28 sprint task? doesn't look like the job has passed21:14
*** dtrainor has quit IRC21:18
*** dtrainor has joined #oooq21:21
weshay|ruckrlandy|rover, f28 is a sprint task, old recreate tools is ruck/rover21:23
rlandy|roverk21:24
rlandy|roverweshay|ruck++ - nice - works ... https://logs.rdoproject.org/61/626661/1/openstack-check/tripleo-ci-centos-7-multinode-1ctlr-featureset010/5baa57d/logs/undercloud/home/zuul/overcloud-deploy.sh.txt.gz21:55
hubbot1rlandy|rover: weshay|ruck's karma is now 121:55
weshay|ruckrlandy|rover, cool22:00
*** trown is now known as trown|outtypewww22:02
weshay|ruckweird tempest failed22:03
rlandy|roverweshay|ruck: same failure22:05
* rlandy|rover gets22:05
rlandy|roverLaunchpad bug 180924822:05
openstackLaunchpad bug 1809218 in tripleo "duplicate for #1809248 podman jobs failing after new podman(0.12.1) and container-selinux(2.77) are released" [Undecided,In progress] https://launchpad.net/bugs/1809218 - Assigned to yatin (yatinkarel)22:05
weshay|ruckah22:06
weshay|ruckweirdness /me looks22:06
rlandy|rover<chandankumar> weshay|ruck: sshnaidm it is failing due to podman https://bugs.launchpad.net/tripleo/+bug/180921822:07
openstackLaunchpad bug 1809218 in tripleo "podman jobs failing after new podman(0.12.1) and container-selinux(2.77) are released" [Undecided,In progress] - Assigned to yatin (yatinkarel)22:07
rlandy|roveraborted: Failed to allocate the network(s), not rescheduling.', u'code': 500, u'created': u'2018-12-20T00:47:43Z'}22:07
weshay|ruckya.. but that run should have been w/ docker22:07
rlandy|roverthis error is everywhere22:07
weshay|ruckhttps://review.openstack.org/#/c/626661/22:08
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-standalone, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-containers-multinode @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7 (1 more message)22:08
weshay|ruckrlandy|rover, oh boy22:10
weshay|ruckthis is fun22:10
weshay|ruckhttp://logs.openstack.org/61/626661/1/check/tripleo-ci-centos-7-containers-multinode/ef1ff35/logs/undercloud/var/log/containers/neutron/dhcp-agent.log.txt.gz#_2018-12-20_21_00_57_88022:10
weshay|ruckERROR neutron.agent.dhcp.agent relabel22:10
weshay|ruckrlandy|rover, ur right... same bug22:10
weshay|ruckhowever cedric's patch wont fix docker22:11
weshay|ruckI just created an upstream bug: https://github.com/containers/libpod/issues/203422:11
rlandy|roverthis really is a blast22:11
weshay|ruckhttps://review.openstack.org/#/c/626583/22:11
weshay|ruckthat might fix it22:11
weshay|ruckrlandy|rover, but doesn't that kind of prove it's not related to podman22:11
weshay|ruckhttps://review.openstack.org/#/c/626583/ fs10 works22:12
weshay|rucklet's see where that is in the queue22:12
rlandy|roverwell https://review.openstack.org/#/c/626583/3/config/general_config/featureset010.yml turns podman off22:12
weshay|ruckbogdan had a tool to download logs22:12
weshay|ruckI couldnt find it22:12
rlandy|roverhttps://review.openstack.org/#/c/626661/1/config/general_config/featureset010.yml makes sure podman is used in the overcloud22:12
weshay|rucker.. meant https://review.openstack.org/#/c/626546/22:13
weshay|rucklet's see if alex is here22:14
rlandy|roverso merge https://review.openstack.org/#/c/626546/ and forget the rest22:15
rlandy|roverweshay|ruck: what is the expectation for the old reproducer?22:31
rlandy|roverthat people manually test it?22:32
rlandy|roverthat rover checks it daily?22:32
weshay|ruckrlandy|rover, so going into the new year.. if we don't have a working reproducer that is bad22:32
weshay|ruckrlandy|rover, good news is I think the patches are fairly easy22:32
rlandy|roveroh yes it is22:32
weshay|ruckrlandy|rover, I think we can just tear out a couple lines in toci_gate_test.sh22:32
* weshay|ruck gets22:32
weshay|ruckor submits something22:32
weshay|ruckif I remember from yesterday22:33
rlandy|roverwhatever - can work on it22:33
weshay|ruckrlandy|rover, https://review.openstack.org/62675822:37
weshay|ruckis what got me going yesterday22:37
* rlandy|rover looks22:38
rlandy|roverinteresting22:38
rlandy|roverreproducer never ran with te-broker to begin with22:39
rlandy|rover! -e $WORKSPACE/instackenv.json22:39
openstackrlandy|rover: Error: "-e" is not a valid command.22:39
rlandy|roverugh22:39
rlandy|rovernothing really changed there22:42
rlandy|roverbut ok22:42
weshay|ruckrlandy|rover, ya.. any thoughts on ci for the new stuff?22:42
weshay|ruckmaybe chat about it tomorrow22:43
rlandy|rovernew stuff - yea - we should22:43
rlandy|roversf should be tested unto itself22:43
rlandy|roverso we should be checking the file that gets generated from jobs22:43
rlandy|roverie from the user side22:43
rlandy|roveralso the testproject jobs that would get generated22:43
rlandy|roverwhich I think is possible22:44
weshay|ruckI really need to get that working22:44
weshay|ruckI feel blind22:44
rlandy|roverlike a dlrn specific job on ovb22:44
rlandy|roverblind leading the blind22:44
rlandy|roverbut really what we will land up giving people is a set up22:44
rlandy|roverwhether we can test the sf zuul set up in ci, idk22:45
rlandy|roverbut the testproject file generated, we can22:45
rlandy|roverwe can submit that job to rdocloud22:45
rlandy|roverexact job22:45
rlandy|roverand verify stuff22:45
rlandy|rovernow22:45
rlandy|roverovb should run the same way22:45
weshay|ruckya.. in theory we should be able run the setup on a nodepool node22:45
rlandy|roverlibvirt is another story22:46
weshay|ruckjust need to run the setup22:46
weshay|ruckeven with libvirt22:46
weshay|ruckwe might be able to get it22:46
rlandy|roverzuul on zuul22:46
weshay|rucklolz22:46
rlandy|roverzoz22:46
weshay|rucktripleoZ22:46
weshay|rucktripleZ22:46
rlandy|roverkind of cool22:46
weshay|rucktripleOZ22:46
rlandy|roverthe wizard of tripleOZ22:47
rlandy|roverok - will test reproducer off https://review.openstack.org/#/c/626758/ if it generates - otherwise will need dep on https://review.openstack.org/#/c/626400/22:49
weshay|ruckoh ya.. I think we need both22:50
weshay|ruckrlandy|rover, let's get out of here :)22:50
weshay|ruckwork on it tomorrow22:50
weshay|ruckI'll watch the two tripleo patches22:50
rlandy|roverthanks22:50
*** rascasoft has quit IRC23:11
*** rlandy|rover has quit IRC23:48
« Wednesday, 2018-12-19 Index Friday, 2018-12-21 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!