| *** Goneri has quit IRC | 00:09 | |
| *** Goneri has joined #oooq | 00:15 | |
| *** Goneri has quit IRC | 00:29 | |
| *** yolanda has quit IRC | 00:38 | |
| *** yolanda has joined #oooq | 00:39 | |
| *** rfolco|ruck has quit IRC | 01:14 | |
| *** chem has quit IRC | 01:38 | |
| *** chem has joined #oooq | 01:40 | |
| *** yolanda has quit IRC | 02:06 | |
| *** yolanda has joined #oooq | 02:13 | |
| *** skramaja has joined #oooq | 03:31 | |
| *** ykarel|away has joined #oooq | 03:54 | |
| *** ykarel_ has joined #oooq | 04:11 | |
| *** ykarel|away has quit IRC | 04:14 | |
| *** ykarel_ is now known as ykarel | 04:16 | |
| *** saneax has joined #oooq | 04:58 | |
| *** udesale has joined #oooq | 05:07 | |
| *** ratailor has joined #oooq | 05:08 | |
| *** marios has joined #oooq | 05:17 | |
| *** jfrancoa has joined #oooq | 05:19 | |
| *** jfrancoa has quit IRC | 05:24 | |
| *** jtomasek has joined #oooq | 05:57 | |
| *** yolanda has quit IRC | 06:10 | |
| *** ysandeep|away is now known as ysandeep | 06:24 | |
| bhagyashri|rover | marios, hi good morning ! is standalone-upgrade master and ussuri passing and stable ? | 06:33 |
|---|---|---|
| marios | bhagyashri|rover: master not sure but it shouldn't be voting? or i missed that it became voting. for ussuri no there is a known bug | 06:33 |
| *** jfrancoa has joined #oooq | 06:34 | |
| bhagyashri|rover | marios, actually the periodic jobs are added in the integration pipeline and those are failing https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-upgrade-ussuri&job_name=periodic-tripleo-ci-centos-8-standalone-upgrade-master | 06:35 |
| bhagyashri|rover | since yesterday i saw those are running in integration pipeline | 06:36 |
| bhagyashri|rover | though those are not in the promotion criteria | 06:36 |
| marios | bhagyashri|rover: ack yeah they should not be in criteria if we aren't gating on them | 06:41 |
| bhagyashri|rover | marios, ok | 06:42 |
| *** chem has quit IRC | 06:49 | |
| *** chem has joined #oooq | 06:50 | |
| *** yolanda has joined #oooq | 07:05 | |
| *** amoralej|off is now known as amoralej | 07:10 | |
| ykarel | bhagyashri|rover, have u created bug for standalone-upgrade master? | 07:21 |
| bhagyashri|rover | ykarel, nope | 07:22 |
| bhagyashri|rover | i will do it | 07:22 |
| ykarel | bhagyashri|rover, ack | 07:23 |
| *** jtomasek has quit IRC | 07:30 | |
| *** tosky has joined #oooq | 07:30 | |
| *** jpena|off is now known as jpena | 07:32 | |
| *** jtomasek has joined #oooq | 07:33 | |
| bhagyashri|rover | ykarel, marios FYI https://bugs.launchpad.net/tripleo/+bug/1897863 | 07:37 |
| openstack | Launchpad bug 1897863 in tripleo "ERROR oslo_db.sqlalchemy.engines [-] Database connection was found disconnected is failing on standalone upgrade master" [High,Triaged] | 07:37 |
| marios | bhagyashri|rover: thanks | 07:39 |
| ykarel | bhagyashri|rover, ack, /me will update bug as actual issue is something else | 08:04 |
| bhagyashri|rover | ykarel, ok | 08:05 |
| bhagyashri|rover | thanks | 08:07 |
| ykarel | bhagyashri|rover, updated | 08:11 |
| bhagyashri|rover | ykarel, thanks | 08:14 |
| *** jbadiapa has joined #oooq | 08:37 | |
| *** derekh has joined #oooq | 08:45 | |
| sshnaidm | chandankumar, as I see, the docker service is not started on queens: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d0a/752421/31/check/tripleo-ci-centos-7-content-provider-queens/d0aa2f4/logs/undercloud/home/zuul/workspace/build.log | 08:57 |
| sshnaidm | chandankumar, idk why, but need to start it | 08:58 |
| chandankumar | sshnaidm: will fix that | 08:59 |
| chandankumar | sshnaidm: for train https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_1df/752421/31/check/tripleo-ci-centos-7-content-provider-train/1dff934/logs/undercloud/home/zuul/workspace/containers-consolidated-builds.log | 09:00 |
| chandankumar | there might be some repos are not available | 09:00 |
| chandankumar | No package erlang-hipe available. | 09:00 |
| sshnaidm | chandankumar, there are varios non available packages | 09:00 |
| sshnaidm | chandankumar, do we still build train containers on c7 anywhere? | 09:01 |
| chandankumar | sshnaidm: yes, in periodic pipeline https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable2-centos7&job_name=periodic-tripleo-centos-7-train-containers-build-push | 09:02 |
| sshnaidm | No package python2-iso8601 available No package openvswitch available | 09:02 |
| sshnaidm | chandankumar, maybe problems with repos | 09:02 |
| chandankumar | https://review.opendev.org/#/c/752419/13/playbooks/multinode-standalone-pre.yml@62 | 09:03 |
| chandankumar | in config repo side, it is set to empty | 09:03 |
| sshnaidm | chandankumar, yeah, seems like it ignores all openstack containers.. | 09:04 |
| chandankumar | https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/build-containers/defaults/main.yaml#L15 | 09:04 |
| chandankumar | there is no testing.repo -testing.repo | 09:04 |
| chandankumar | let me switch to drln.repo and start docker service and see how it goes | 09:05 |
| sshnaidm | chandankumar, also should be gating.repo there | 09:05 |
| chandankumar | arxcruz: akahat soniya29 kopecmartin weshay|ruck do we want to cancel today's tempest meeting due to retro? | 09:19 |
| akahat | chandankumar, i move it to Friday (only for this week) | 09:22 |
| chandankumar | akahat: on friday I am not available | 09:28 |
| akahat | chandankumar, ah.. me too.. forgot it.. | 09:30 |
| akahat | well i don't have much to discuss. | 09:30 |
| *** derekh has quit IRC | 09:32 | |
| *** derekh has joined #oooq | 09:33 | |
| kopecmartin | chandankumar: the only thing i would discuss are the tempest bumps for train and ussuri .. can you have a look at the ussuri bump and it's dependency? it's ready https://review.rdoproject.org/r/#/c/29616/ | 10:09 |
| kopecmartin | chandankumar: and regarding the train bump i don't know why, but centos-7 jobs are failing, most likely not due to tempest bump: https://review.rdoproject.org/r/#/c/29611/ | 10:10 |
| chandankumar | kopecmartin: can we cherry-pick this one https://review.rdoproject.org/r/#/c/27372/ for ussuri | 10:17 |
| chandankumar | ? | 10:17 |
| chandankumar | instead of creating a new one | 10:17 |
| chandankumar | kopecmartin: for ussuri /usr/bin part stays while building packages? | 10:20 |
| kopecmartin | chandankumar: no idea if the 'rm -rf of /usr/bin ..' is required or not, i just copied the logic from last time when we removed verify-tempest-config https://review.rdoproject.org/r/#/c/27712/ | 10:24 |
| chandankumar | kopecmartin: can you try with cherry-picking | 10:24 |
| chandankumar | if /usr/bin comes then we will abandon cherry-picking | 10:26 |
| kopecmartin | sure why not | 10:28 |
| * marios food biab | 10:55 | |
| *** marios has quit IRC | 10:55 | |
| *** ysandeep is now known as ysandeep|afk | 11:11 | |
| *** dtantsur|afk is now known as dtantsur | 11:11 | |
| *** apetrich has joined #oooq | 11:12 | |
| amoralej | wrt https://bugs.launchpad.net/tripleo/+bug/1897863, where is repo_setup_upgrades.sh script coming from in upgrades job? | 11:28 |
| openstack | Launchpad bug 1897863 in tripleo "[master][standalone-upgrade] Fails while applying ovn controller puppet config with Error: Failed to apply catalog: Execution of '/usr/bin/ovs-vsctl list Open_vSwitch .' returned 1: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)" [High,Triaged] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 11:28 |
| amoralej | chem, ^ i couldn't find it | 11:28 |
| amoralej | chandankumar, ^ any help is appreciated | 11:28 |
| *** jpena is now known as jpena|lunch | 11:28 | |
| chandankumar | looking | 11:29 |
| chem | amoralej++ ok, thanks for looking into it | 11:30 |
| chem | chandankumar: hey, let me know when you/if find it, I'll be helping/doing from the upgrade squad standpoint on this. | 11:31 |
| chandankumar | sure, I am trying to find out repo_setup_upgrades.sh script | 11:31 |
| chandankumar | chem: https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/playbooks/multinode-standalone-upgrade.yml#L22 where script is getting generated | 11:36 |
| chandankumar | https://opendev.org/openstack/tripleo-quickstart/src/branch/master/roles/repo-setup/templates/repo_setup.sh.j2 from which it gets generared | 11:37 |
| chandankumar | *generated | 11:38 |
| chandankumar | https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-master/90bb089/logs/undercloud/home/zuul/repo_setup_upgrade.sh.txt.gz generted from the job | 11:39 |
| chandankumar | amoralej: ^^ | 11:40 |
| *** marios has joined #oooq | 11:40 | |
| amoralej | chem, ^ | 11:41 |
| chem | chem: chem ^ | 11:42 |
| chem | we got to the end of the recursion :) | 11:42 |
| amoralej | :) | 11:42 |
| amoralej | it must be setting repo_cmd_after or dep_repo_cmd_after somewhere | 11:43 |
| chem | chandankumar: ack will look into it, but i'm in a mtg right now | 11:43 |
| amoralej | chem, https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/release/tripleo-ci/CentOS-8/master.yml#L186 | 11:44 |
| amoralej | not sure what's the release config used in the upgrade | 11:44 |
| amoralej | but be careful as that yum update is needed in other cases | 11:45 |
| amoralej | set repo_setup_run_update to false in second execution for upgrade | 11:45 |
| *** ysandeep|afk is now known as ysandeep | 11:51 | |
| *** rfolco has joined #oooq | 11:53 | |
| *** rfolco is now known as rfolco|ruck|bbl | 11:53 | |
| *** rlandy has joined #oooq | 11:58 | |
| weshay|ruck | bhagyashri|rover, you can come back to the retro after your status update | 12:05 |
| bhagyashri|rover | weshay|ruck, sure | 12:05 |
| *** amoralej is now known as amoralej|lunch | 12:06 | |
| *** Goneri has joined #oooq | 12:17 | |
| *** udesale_ has joined #oooq | 12:18 | |
| zbr | weshay|ruck: i find no option to edit or delete labels on trello, i suspect you need to give me some extra permissions | 12:19 |
| *** udesale has quit IRC | 12:21 | |
| *** jpena|lunch is now known as jpena | 12:36 | |
| *** ratailor has quit IRC | 12:37 | |
| chem | amoralej|lunch: so that's rdo-openvswitch that trigger the stop ? if so, yeah, we have nothing in tripleo_ovs_upgrade that will be triggered given that ovs doesn't have a restart | 12:44 |
| chem | marios: hey o/ | 12:55 |
| chem | marios: I would need help/pointer to set "repo_setup_run_update: false" for standalone_upgrade | 12:57 |
| chem | I need to eat something. | 12:57 |
| *** chem is now known as chem|lunch | 12:57 | |
| *** derekh has quit IRC | 13:01 | |
| *** derekh has joined #oooq | 13:03 | |
| zbr | weshay|ruck: hint to improve trello: just change color to gray for old sprints, this sorts them to the bottom. current sprint should use green or blue color. | 13:03 |
| zbr | labels are ordered by their color, not name, gray being last almost some kind of hidden. | 13:04 |
| *** pojadhav is now known as pojadhav|brb | 13:05 | |
| *** amoralej|lunch is now known as amoralej | 13:06 | |
| ysandeep | rlandy: hi 16.2 integration line is running right now, if today's run go well.. I am thinking we should probably promote it.. last promotion was 1 week before | 13:06 |
| chandankumar | https://7b4f6579770a76590ca3-628ff6cd1ebc53a96e6edaebcdf80a30.ssl.cf2.rackcdn.com/752421/33/check/tripleo-ci-centos-7-content-provider-queens/2a6ee13/logs/undercloud/home/zuul/workspace/build.log | 13:07 |
| rlandy | ysandeep: sure - you want me to promote this afternoon? | 13:07 |
| ysandeep | rlandy, yes please if you can that will be great.. that ovb job will run for another 5 hours.. so that will be mid night for me | 13:08 |
| rlandy | ysandeep: if the ovb job fails, should we still promote? | 13:08 |
| ysandeep | rlandy: checking previous week build history | 13:09 |
| apetrich | arxcruz, congrats 6 years of red hat or so linkedin says :) | 13:10 |
| rlandy | rfolco|ruck|bbl: bhagyashri|rover: soniya29: should we sync on ruck/rover? | 13:12 |
| * rlandy has prod chain in a few | 13:13 | |
| rlandy | rfolco|ruck|bbl: maybe this afternoon? | 13:13 |
| bhagyashri|rover | rlandy, sure | 13:13 |
| soniya29 | rlandy, sure | 13:13 |
| ysandeep | rlandy: if it fails with timeout on tempest, i think we can promote.. this job randomly times out on tempest. | 13:15 |
| holser | marios weshay|ruck I made some progress for https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-7-standalone-upgrade-stein | 13:17 |
| amoralej | chem|lunch, the stop is triggered by openvswitch uninstall | 13:17 |
| *** pojadhav|brb is now known as pojadhav | 13:17 | |
| amoralej | not by rdo-openvswitch | 13:17 |
| holser | so the issue is cirros or something ... tempest cannot boot up VMs | 13:18 |
| weshay|ruck | holser, nice.. looking | 13:18 |
| holser | weshay|ruck I still have env | 13:19 |
| holser | weshay|ruck https://imgur.com/a/rvI0HFp | 13:20 |
| holser | it's VM tempest tried to create | 13:20 |
| holser | However, the problem is we tried to download cirros 0.4.0 instead of cirros 0.3.5 and got the same result | 13:21 |
| weshay|ruck | cool.. | 13:21 |
| weshay|ruck | some of the folks are in a mtg right now fyi | 13:21 |
| rlandy | weshay|ruck: bhagyashri|rover: soniya29: rfolco|ruck|bbl: https://hackmd.io/1qxCqYzATfudl1cKvaQ8-w sprint 34 ruck/rover hackmd | 13:21 |
| holser | weshay|ruck sorry... I didn't know | 13:21 |
| *** rlandy is now known as rlandy|rover | 13:22 | |
| weshay|ruck | holser, ok.. re: cirros.. arx is pto but chandankumar may be able to take a look at the cirros issue | 13:22 |
| weshay|ruck | I know that's been a thing | 13:22 |
| weshay|ruck | holser, the latest run I see.. | 13:22 |
| weshay|ruck | 2020-09-30 11:31:39 | 2020-09-30 11:31:39.580 315857 WARNING tripleoclient.v1.tripleo_upgrade.Upgrade [-] fatal: [standalone]: FAILED! => {"changed": false, "msg": "You can't upgrade cinder_volume without staged upgrade. You need to use the limit option in order to do so.\n"}[00m | 13:22 |
| weshay|ruck | is that not happening in w/ the node you held | 13:22 |
| holser | weshay|ruck thanks a lot | 13:23 |
| *** soniya29 is now known as soniya29|ruck | 13:24 | |
| *** skramaja has quit IRC | 13:24 | |
| *** chem|lunch is now known as chem | 13:26 | |
| chem | amoralej: ho ... but then I misred what you put there http://paste.openstack.org/show/798571/ ? | 13:27 |
| amoralej | chem, that's openvswitch, not rdo-openvswitch | 13:28 |
| chem | amoralej: well, but you just wrote that openvswitch has the uninstall script to stop the service | 13:28 |
| * chem is so confused | 13:28 | |
| * amoralej too | 13:29 | |
| amoralej | :) | 13:29 |
| chem | amoralej: oh ... | 13:29 |
| chem | amoralej: it's preuninstall | 13:29 |
| marios | holser: in a call will check in a bit thanks | 13:29 |
| amoralej | yes | 13:29 |
| amoralej | so, when going from openvswitch to openvswitch2.13 it executes that preuninstall scriptlet | 13:30 |
| amoralej | which stop and disables | 13:30 |
| chandankumar | holser: Hello | 13:30 |
| chem | amoralej: and in our stuff we check something "restart" after postuninstall | 13:30 |
| chem | amoralej: which was the fragile test we settle on years ago | 13:30 |
| chem | amoralej: oki, time to update the test I guess | 13:30 |
| amoralej | chem, that's why in the "special ovs upgrade procedure" it remove the package with rpm -e --noscripts | 13:31 |
| *** rlandy|rover is now known as rlandy|rover|mtg | 13:31 | |
| chem | amoralej: yeah, so this https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/ansible_plugins/modules/tripleo_ovs_upgrade.py#L199-L201 | 13:32 |
| chem | amoralej: need to adjust to the new package taste | 13:32 |
| chem | amoralej: until taste change ... man | 13:32 |
| amoralej | yes, it's a mess | 13:32 |
| rlandy|rover|mtg | soniya29|ruck: I'll take care of rerunning ci.centos | 13:33 |
| chem | amoralej: oki, so with that fixed, we should be able to move from non-layered to layered safely | 13:33 |
| soniya29|ruck | rlandy|rover|mtg okay | 13:34 |
| holser | chandankumar hey | 13:35 |
| chandankumar | holser: Anything I can help on cirros issue | 13:35 |
| chem | amoralej: so that and finding where to set repo_setup_run_update: false for upgrade jobs and we're done :) | 13:35 |
| amoralej | chem, in https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/playbooks/multinode-standalone-upgrade.yml#L22 ? | 13:36 |
| chem | amoralej: oh ... yeah why not | 13:37 |
| *** b3nt_pin has joined #oooq | 13:37 | |
| chem | amoralej: I though that we maybe would put that in job definition, but you're right it could belong there | 13:37 |
| chem | amoralej: oki, coding time :) | 13:37 |
| chem | amoralej: hum ... fixing time :) | 13:37 |
| amoralej | chem, note that https://review.opendev.org/#/c/755238/ seems to fix the issue as collateral damage :) | 13:40 |
| chem | amoralej: first time we caught an issue in ovs upgrade before it hits downstream ... I want to cry, so much joy | 13:40 |
| amoralej | with https://review.opendev.org/#/c/754974/ it makes os-net-config to start ovs | 13:40 |
| amoralej | chem, you'd be happy of catching issues upstream :) | 13:40 |
| chem | amoralej: yeah, as I put it, joy ... no blocker bug, no (as much) stress, I like upstream :) | 13:41 |
| chem | amoralej: I absolutely don't get how that review could fix it, but look so unrelated that my brain started to convulse ... should I deep dive in that review ? | 13:42 |
| chem | amoralej: that "systemctl --no-reload disable --now openvswitch.service" requires some pair of balls in a critical system like ovs | 13:43 |
| amoralej | chem, https://bugs.launchpad.net/tripleo/+bug/1897863/comments/4 | 13:44 |
| openstack | Launchpad bug 1897863 in tripleo "[master][standalone-upgrade] Fails while applying ovn controller puppet config with Error: Failed to apply catalog: Execution of '/usr/bin/ovs-vsctl list Open_vSwitch .' returned 1: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)" [High,Triaged] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 13:44 |
| amoralej | ykarel explained there about the os-net-config | 13:44 |
| * chem convulse a lot | 13:46 | |
| *** udesale_ has quit IRC | 13:47 | |
| chem | oki, I'll stick to simple thing for today, already a long day | 13:48 |
| rlandy|rover|mtg | ysandeep: fyi .. after promotion, we will ping rel del to update rho-release | 14:02 |
| rlandy|rover|mtg | if you see failures tomorrow morning, pls work with them to revert | 14:02 |
| rlandy|rover|mtg | also we could set up a four job testproject to allow to test against these versions | 14:03 |
| ysandeep | rlandy|rover|mtg, sure , just to be sure - we are talking about this one right? - https://code.engineering.redhat.com/gerrit/#/c/213592/ | 14:05 |
| rlandy|rover|mtg | ysandeep: ack | 14:05 |
| rlandy|rover|mtg | testing container builds against that | 14:05 |
| weshay|ruck | rfolco|ruck|bbl, bhagyashri|rover we'll need one of you to join the cix call today for a proper transition :) | 14:06 |
| weshay|ruck | and then ur done.. minus a transition | 14:06 |
| bhagyashri|rover | weshay|ruck, ok i will join | 14:07 |
| ysandeep | rlandy|rover|mtg: can we add depends-On on rhos-release gerrit patches for our testprojects? | 14:09 |
| rlandy|rover|mtg | ysandeep: setting up job | 14:09 |
| rlandy|rover|mtg | sec | 14:09 |
| *** rfolco|ruck|bbl is now known as rfolco | 14:15 | |
| *** rlandy|rover|mtg is now known as rlandy|rover | 14:17 | |
| ysandeep | got it from your patch, thx | 14:18 |
| *** ysandeep is now known as ysandeep|away | 14:19 | |
| *** bhagyashri|rover is now known as bhagyashri | 14:20 | |
| rlandy|rover | weshay|ruck: rfolco: will one of you be attending CIX? | 14:21 |
| rlandy|rover | will join but catching up on status | 14:21 |
| weshay|ruck | aye.. I'll join | 14:22 |
| rfolco | rlandy|rover, I can attend | 14:22 |
| rfolco | will be there in cases there is any remaining question | 14:22 |
| *** TrevorV has joined #oooq | 14:23 | |
| rlandy|rover | thanks | 14:23 |
| rfolco | standalone upgrade variances | 14:46 |
| rfolco | https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-upgrade-master&job_name=periodic-tripleo-ci-centos-8-standalone-upgrade-ussuri&job_name=periodic-tripleo-ci-centos-8-standalone-upgrade-train | 14:46 |
| rfolco | may need to check if this additional bug is valid | 14:46 |
| rfolco | https://bugs.launchpad.net/tripleo/+bug/1895705 | 14:46 |
| openstack | Launchpad bug 1895705 in tripleo "[train|ussuri] scen10-ovn-provider-standalone: insert-headers is not a valid option for a TCP protocol listener" [Critical,Triaged] | 14:46 |
| rfolco | rlandy|rover, ^ | 14:46 |
| rfolco | soniya29|ruck, weshay|ruck fyi ^ | 14:46 |
| rlandy|rover | rfolco: let's meet after this | 14:47 |
| rlandy|rover | to sync | 14:47 |
| rfolco | k | 14:47 |
| weshay|ruck | rfolco, isn't that an old bug? | 14:47 |
| weshay|ruck | ya | 14:47 |
| rfolco | weshay|ruck, just don't know if this one is dup or not.... note this is a diff bug (not the https not supported one) | 14:48 |
| weshay|ruck | rfolco, this one was fixed w/ the additional tempest config right? | 14:48 |
| rfolco | weshay|ruck, https one should be fixed. Not sure about this one | 14:49 |
| rfolco | weshay|ruck, maybe its the same issue or even collateral | 14:49 |
| *** rlandy|rover is now known as rlandy|rover|mtg | 14:50 | |
| weshay|ruck | rfolco, rlandy|rover|mtg https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-train%09 | 14:52 |
| rlandy|rover|mtg | weshay|ruck: going to touch base with rfolco after this meeting to catch up re; ruck/rover | 14:53 |
| rlandy|rover|mtg | https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-train-centos7-current-tripleo/ | 14:53 |
| rlandy|rover|mtg | never starts | 14:54 |
| rfolco | weshay|ruck, was referring to this tripleo-ci-centos-7-standalone-upgrade-steinFAILURE in 2h 08m 47s | 15:00 |
| weshay|ruck | I know :) | 15:00 |
| rfolco | k there is no point in rechecking :) | 15:00 |
| rfolco | Sergii Golovatiuk yesterday at 11:29 AM | 15:01 |
| rfolco | We cannot merge patch as tempest is failing on all standalone-stein jobs | 15:01 |
| rlandy|rover|mtg | rfolco: meet now? | 15:02 |
| rlandy|rover|mtg | soniya29|ruck: ^^ | 15:02 |
| rfolco | okay | 15:02 |
| rlandy|rover|mtg | https://meet.google.com/vps-omwf-nbo?pli=1&authuser=0 | 15:03 |
| rlandy|rover|mtg | rfolco: bhagyashri: soniya29|ruck: weshay|ruck: ^^ | 15:03 |
| rlandy|rover|mtg | ruck/rover sync | 15:03 |
| rlandy|rover|mtg | rfolco: https://hackmd.io/1qxCqYzATfudl1cKvaQ8-w | 15:04 |
| chem | omg ... so I would like to trigger tripleo-ci-centos-8-standalone-upgrade in tripleo-ansible | 15:25 |
| chem | and why not include it in the test tripleo-ansible run ... but my usual copy/paste foo failed me | 15:25 |
| chem | could somebody help me out here ? | 15:26 |
| chem | I have the feeling that it happen in the template def of the zuul.d/layout.yaml, but where are those template ? | 15:26 |
| chem | marios: holser maybe you know ^ | 15:27 |
| marios | chem: looking | 15:27 |
| chem | s/foo/fu/ | 15:27 |
| holser | chem trying to understand your foooo | 15:27 |
| chem | hehe, everything is foo :) | 15:27 |
| marios | chem: looks like they aren't even running that job actually https://opendev.org/openstack/tripleo-ansible/src/branch/master/zuul.d/layout.yaml | 15:27 |
| marios | chem: so you can add it to the layout there ... is this what you mean though? | 15:28 |
| marios | chem: 'how can i make it run there' ? | 15:28 |
| chem | marios: yeap | 15:28 |
| marios | chem: fetching you pointer sec | 15:28 |
| chem | marios: but I tried to look into tht layout.yaml (as it has the job) and my grep just failed me | 15:29 |
| weshay|ruck | rfolco, can we conclude if we can close https://bugs.launchpad.net/tripleo/+bug/1895705 | 15:29 |
| openstack | Launchpad bug 1896277 in tripleo "duplicate for #1895705 [scen10-ovn] OVN provider does not support HTTPS protocol" [Critical,Triaged] | 15:29 |
| marios | chem: hold on | 15:29 |
| marios | chem: tht or tripleo-ansible what are we talking about | 15:29 |
| chem | holser: by the way I have fixes for upstream jobs https://bugs.launchpad.net/tripleo/+bug/1897863/comments/12 | 15:29 |
| openstack | Launchpad bug 1897863 in tripleo "[master][standalone-upgrade] Fails while applying ovn controller puppet config with Error: Failed to apply catalog: Execution of '/usr/bin/ovs-vsctl list Open_vSwitch .' returned 1: ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)" [High,In progress] - Assigned to Sofer Athlan-Guyot (sofer-athlan-guyot) | 15:29 |
| rfolco | weshay|ruck, marked as dup, the original bug is fixed and closed: fix_commit | 15:30 |
| marios | chem: nice (ovs just keeps on giving ;)) | 15:30 |
| marios | chem: so ... you want to jump in a meet? | 15:30 |
| chem | marios: yeah, it's a constant stream of joy | 15:30 |
| marios | chem: maybe faster than irc ... but maybe not necessary... are you talking about tht? | 15:30 |
| chem | marios: oh, yeah, gimmi a link | 15:30 |
| marios | chem: or tripleo-ansible? | 15:30 |
| chem | thtpleo-ansible | 15:31 |
| marios | chem: ack https://meet.google.com/_meet/pyh-gmqm-iwb?ijlm=1601479871504&adhoc=1&hs=125 | 15:31 |
| holser | weshay|ruck I fixed tempest on my reproducer environment | 15:33 |
| holser | marios | 15:33 |
| holser | I don't know why but cirros-0.3.5 is not going to boot up | 15:33 |
| holser | 0.4.0 works fine | 15:33 |
| holser | so I would bump cirros to 0.4.0 across all templates in tripleo-ci | 15:34 |
| weshay|ruck | rlandy|rover|mtg, so here's an issue we need to fix | 15:36 |
| weshay|ruck | https://review.opendev.org/#/c/752230/6 | 15:36 |
| weshay|ruck | rlandy|rover|mtg, /me notices tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-train is running in gate but not check | 15:36 |
| holser | weshay|ruck marios https://paste.centos.org/view/f39cc369 | 15:38 |
| chandankumar | holser: awesome :-) | 15:38 |
| chandankumar | weshay|ruck: this sprint we can prioritize cirros mess | 15:38 |
| weshay|ruck | chandankumar, holser :) | 15:38 |
| marios | holser: ack thanks great | 15:39 |
| * weshay|ruck really wants to see the cirros images on the local filesystem as they should be | 15:39 | |
| weshay|ruck | in the node build | 15:39 |
| *** udesale_ has joined #oooq | 15:39 | |
| holser | weshay|ruck yeah | 15:39 |
| *** ykarel is now known as ykarel|away | 15:40 | |
| holser | so stein and train tempest fdailures are due to cirros image | 15:40 |
| chem | marios: I was still plugged to openstack-infra/tripleo-ci ... man I haven't been around since a long time :) | 15:41 |
| holser | chem you've been around in openstack since cactus | 15:41 |
| holser | you are tooo old | 15:41 |
| chem | holser: man, there was a cactus version !! yike | 15:42 |
| marios | chem: ;) | 15:43 |
| *** rlandy|rover|mtg is now known as rlandy|rover | 15:44 | |
| chem | marios: first time upstream is getting this kind of issue (even if due to an overall misconfiguration), so I'm really celebrating :) | 15:45 |
| weshay|ruck | rlandy|rover, same thing here: https://review.opendev.org/#/c/755209/ | 15:53 |
| rlandy|rover | weshay|ruck: k - looking into that | 16:02 |
| weshay|ruck | rlandy|rover, looks like it's working correctly here: https://review.opendev.org/#/c/754829/ so it's some nuance in zuul.. or on the projects zuul | 16:03 |
| rlandy|rover | weshay|ruck: must be some places use a template and some do not - checking | 16:03 |
| rlandy|rover | weshay|ruck: there is no tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-train on https://review.opendev.org/#/c/754829/ | 16:04 |
| rlandy|rover | what am I missing? | 16:04 |
| rlandy|rover | tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-ussuri | 16:05 |
| weshay|ruck | rlandy|rover, this is one to look at https://review.opendev.org/#/c/752230/ | 16:06 |
| weshay|ruck | updates-train in gate but not check | 16:06 |
| weshay|ruck | perhaps.. the file trigger | 16:06 |
| weshay|ruck | not sure why this is happening | 16:06 |
| chem | marios: going to be epic https://review.opendev.org/755336 | 16:07 |
| chem | marios: that was fast :) | 16:07 |
| rlandy|rover | ok - checking | 16:07 |
| weshay|ruck | marios, https://review.opendev.org/#/c/755336/ not sure the state of the content provider for non-ci repos | 16:08 |
| weshay|ruck | you would know better than me | 16:08 |
| marios | chem: i see it... k in that case try adding the provider tempalte i mean https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/multinode-jobs.yaml#L219 | 16:10 |
| marios | chem: include that template i thought you wanted the standalone-upgrade but it fails (zuul) trying to get the update job | 16:10 |
| marios | chem: if you add that template name: tripleo-multinode-container-minimal-pipeline should work | 16:10 |
| rlandy|rover | weshay|ruck: k - here's the difference ... | 16:11 |
| rlandy|rover | https://github.com/openstack/tripleo-heat-templates/blob/stable/train/zuul.d/layout.yaml#L54 | 16:11 |
| rlandy|rover | ^^ only in check | 16:11 |
| chem | marios: I simplified it, but yeah, next step will be to add a template | 16:11 |
| marios | weshay|ruck: ack commented | 16:11 |
| rlandy|rover | but the template - tripleo-multinode-container-minimal | 16:11 |
| rlandy|rover | has gate defined as well | 16:11 |
| marios | weshay|ruck: but we havn't added it anywhere yet only triupleo-ci | 16:11 |
| weshay|ruck | marios, thanks | 16:11 |
| rlandy|rover | so I assume that the check definition is overriding the template | 16:12 |
| rlandy|rover | if we remove the check def, the template should be used in both cases | 16:12 |
| * rlandy|rover tries | 16:12 | |
| *** marios is now known as marios|out | 16:13 | |
| *** rfolco has left #oooq | 16:15 | |
| *** amoralej is now known as amoralej|off | 16:17 | |
| *** tosky has quit IRC | 16:19 | |
| rlandy|rover | weshay|ruck: k - we have two options here ... | 16:19 |
| rlandy|rover | https://github.com/openstack/tripleo-heat-templates/blob/stable/train/zuul.d/layout.yaml#L54 | 16:20 |
| rlandy|rover | remove that | 16:20 |
| rlandy|rover | or duplicate it in gate | 16:20 |
| rlandy|rover | preference? | 16:20 |
| rlandy|rover | if we remove it, the template will rule | 16:20 |
| weshay|ruck | rlandy|rover, ya.. templates > individual jobs | 16:22 |
| weshay|ruck | so try that first | 16:22 |
| weshay|ruck | thanks | 16:22 |
| *** marios|out has quit IRC | 16:24 | |
| weshay|ruck | be back in 1 hr | 16:36 |
| weshay|ruck | scratch that | 16:40 |
| *** udesale_ has quit IRC | 16:41 | |
| *** jpena is now known as jpena|off | 16:57 | |
| *** derekh has quit IRC | 17:00 | |
| *** ykarel|away has quit IRC | 17:08 | |
| rlandy|rover | 2020-09-30 14:58:31.311636 | primary | ModuleNotFoundError: No module named 'httpx' | 17:12 |
| rlandy|rover | arxcruz: hey ^^ | 17:13 |
| rlandy|rover | https://logserver.rdoproject.org/openstack-component-octavia/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-octavia-master/1022167/job-output.txt | 17:13 |
| rlandy|rover | thoughts on htpx in tempest? | 17:13 |
| rlandy|rover | http://codesearch.openstack.org/?q=httpx&i=nope&files=&repos= | 17:13 |
| johnsom | It's needed for HTTP/2 testing | 17:14 |
| *** dtantsur is now known as dtantsur|afk | 17:23 | |
| weshay|ruck | rlandy|rover, arxcruz is out for the rest of the week | 17:31 |
| * weshay|ruck looks | 17:31 | |
| rlandy|rover | weshay|ruck: k - 1-on-1? | 17:31 |
| rlandy|rover | weshay|ruck: logged bug ... | 17:31 |
| rlandy|rover | https://bugs.launchpad.net/tripleo/+bug/1897947 | 17:31 |
| openstack | Launchpad bug 1897947 in tripleo "periodic-tripleo-ci-centos-8-scenario010-standalone-octavia-master fails tempest with 'ModuleNotFoundError: No module named 'httpx''" [Critical,Triaged] | 17:31 |
| weshay|ruck | rlandy|rover, k.. let's 1-1 a bit later.. very noisy here | 17:31 |
| weshay|ruck | atm | 17:31 |
| *** rfolco has joined #oooq | 17:36 | |
| rlandy|rover | promoting 16.2 | 17:38 |
| rlandy|rover | ysandeep|away: ^^ | 17:38 |
| rlandy|rover | weshay|ruck: https://bugs.launchpad.net/tripleo/+bug/1897947 - we assign that to tempest or octavia? | 17:39 |
| openstack | Launchpad bug 1897947 in tripleo "periodic-tripleo-ci-centos-8-scenario010-standalone-octavia-master fails tempest with 'ModuleNotFoundError: No module named 'httpx''" [Critical,Triaged] | 17:39 |
| weshay|ruck | rlandy|rover, Failed to import test module: octavia_tempest_plugin. | 17:40 |
| weshay|ruck | so I would say octavia | 17:40 |
| johnsom | Why isn't the octavia-tempest-plugin requirements.txt being used there? | 17:40 |
| johnsom | httpx is in there | 17:40 |
| * rlandy|rover checks what's used | 17:42 | |
| chandankumar | johnsom: we need to package httpx in fedora first | 17:43 |
| chandankumar | then we can bring it in rdo via centos | 17:43 |
| chandankumar | for current scenario, we might want to pin the octavia tempest plugin in rdoinfo | 17:43 |
| rlandy|rover | chandankumar: or exclude from criteria | 17:44 |
| chandankumar | rlandy|rover: nope | 17:44 |
| rlandy|rover | how long will the packaging take? | 17:44 |
| chandankumar | rlandy|rover: max a week | 17:44 |
| chandankumar | rlandy|rover: min 2-3 days | 17:44 |
| rlandy|rover | chandankumar: k - let's pin then | 17:44 |
| chandankumar | short term solution pin it in rdoinfo | 17:44 |
| rlandy|rover | k | 17:44 |
| chandankumar | johnsom: I will take care of httpx packaging | 17:44 |
| johnsom | Ok, thank you! | 17:45 |
| rlandy|rover | chandankumar: what file do we use for master: https://github.com/redhat-openstack/rdoinfo/tree/master/tags | 17:46 |
| chandankumar | rlandy|rover: https://github.com/redhat-openstack/rdoinfo/blob/master/tags/victoria-uc.yml#L231 | 17:48 |
| chandankumar | rlandy|rover: pin to this commit https://opendev.org/openstack/octavia-tempest-plugin/commit/f7718ef862c568f7c761c17ac9736e650ececb73 | 17:49 |
| rlandy|rover | ack - review coming up | 17:49 |
| chandankumar | source-branch: above tag | 17:50 |
| chandankumar | *commit | 17:50 |
| rlandy|rover | chandankumar: https://review.rdoproject.org/r/29809 Pin octavia-tempest-plugin to avoid httpx import error | 17:57 |
| weshay|ruck | rfolco, https://meet.google.com/tcv-xsmn-izc?authuser=1 | 18:01 |
| rlandy|rover | hmm - are we expecting queens to be blood red? | 18:05 |
| rlandy|rover | rfolco: weshay|ruck: ^^? | 18:06 |
| rlandy|rover | integration | 18:06 |
| weshay|ruck | upstream? | 18:08 |
| weshay|ruck | integration.. ah.. look at the criteria | 18:08 |
| rlandy|rover | 2020-09-30 15:25:15 | + --tag c42b81cb7c7739d097d50a45ffc16e5ccbe38df8_424c8702 --prefix centos-binary --suffix '' --set ceph_namespace=docker.io/ceph --set ceph_image=daemon --set ceph_tag=v3.2.14-stable-3.2-luminous-centos-7-x86_64 --push-destination 192.168.24.1:8787 | 18:09 |
| rlandy|rover | 2020-09-30 15:25:15 | /home/zuul/overcloud-prep-containers.sh: line 31: --tag: command not found | 18:09 |
| rlandy|rover | weshay|ruck: rdocloud | 18:09 |
| weshay|ruck | rlandy|rover, can 1-1 here in a sec | 18:16 |
| weshay|ruck | rlandy|rover, https://meet.google.com/tcv-xsmn-izc?authuser=1 | 18:17 |
| weshay|ruck | rlandy|rover, https://review.opendev.org/#/c/750989/1/tripleoclient/v1/container_image.py | 18:26 |
| weshay|ruck | rlandy|rover, https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable4-5&job_name=periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens%09 | 18:29 |
| weshay|ruck | rlandy|rover, https://review.opendev.org/#/q/status:merged+tripleo+branch:stable/queens | 18:30 |
| weshay|ruck | https://review.opendev.org/#/q/project:openstack/tripleo-quickstart-extras+status:merged+age:2days | 18:33 |
| weshay|ruck | rlandy|rover, ^ | 18:33 |
| *** jfrancoa has quit IRC | 18:34 | |
| rlandy|rover | weshay|ruck: https://logserver.rdoproject.org/openstack-periodic-integration-stable4-5/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/a4da29d/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz | 18:37 |
| weshay|ruck | rlandy|rover, https://logserver.rdoproject.org/openstack-periodic-integration-stable4-5/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/7850d85/logs/undercloud/home/zuul/overcloud-prep-containers.sh.txt.gz | 18:38 |
| weshay|ruck | rlandy|rover, https://review.opendev.org/#/c/753406/7/roles/overcloud-prep-containers/templates/overcloud-prep-containers.sh.j2 | 18:39 |
| rlandy|rover | weshay|ruck: https://review.rdoproject.org/r/#/c/29809/ | 18:49 |
| *** tosky has joined #oooq | 19:32 | |
| weshay|ruck | \0/ for i in `podman images | awk '{print $3}'`; do echo $i; podman run --net=host $i rpm -qa; done | 19:33 |
| *** jtomasek has quit IRC | 20:15 | |
| rlandy|rover | weshay|ruck: should I look at the ci.centos failures or just leave them? | 20:23 |
| rlandy|rover | master has a deploy failure | 20:24 |
| rlandy|rover | 2020-09-30 16:29:08 | 2020-09-30 16:29:08.038847 | 00559b12-3313-7ffb-081a-000000006be1 | FATAL | Check containers status | overcloud-novacompute-0 | error={"changed": false, "msg": "Failed container(s): ['nova_wait_for_compute_service'], check logs in /var/log/containers/stdouts/"} | 20:24 |
| weshay|ruck | rlandy|rover, so.. anything executed in ci.centos is already passed in tripleo-ci | 20:24 |
| weshay|ruck | rlandy|rover, it's just a matter of getting a ci.centos node that is powerful enough to run tripleo | 20:24 |
| rlandy|rover | k - rekick | 20:24 |
| weshay|ruck | so.. it's just a rerun rerun.. | 20:25 |
| weshay|ruck | ya | 20:25 |
| rlandy|rover | k - done | 20:25 |
| weshay|ruck | thanks | 20:26 |
| weshay|ruck | rlandy|rover, master promoted | 20:26 |
| rlandy|rover | k - good | 20:27 |
| weshay|ruck | ussuri is promoting now | 20:27 |
| rlandy|rover | new rhos-release looks ok | 20:52 |
| sshnaidm | rlandy|rover, do you know maybe what is purpose of this task? https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/build-test-packages/tasks/main.yml#L173 | 20:59 |
| sshnaidm | rlandy|rover, it fails in internal job 'sed: can't read packages.yml: No such file or directory' : https://sf.hosted.upshift.rdu2.redhat.com/logs/78/213478/8/check/tripleo-ci-rhel-7-multinode-1ctlr-featureset010/e676b2e/job-output.txt | 20:59 |
| rlandy|rover | sshnaidm: probably due to the way downstream repos are accessed | 21:01 |
| rlandy|rover | see git clone instructions http://git.app.eng.bos.redhat.com/git/openstack-tripleo-common.git | 21:02 |
| rlandy|rover | sshnaidm: how's it going with rhos-13 release file? | 21:02 |
| rlandy|rover | picked up ruck/rover so haven't done work there | 21:03 |
| sshnaidm | rlandy|rover, currently fails because of 'sed: can't read packages.yml: No such file or directory' | 21:03 |
| rlandy|rover | sshnaidm: try taking it out | 21:03 |
| sshnaidm | yeah, will do | 21:03 |
| rlandy|rover | we rarely pick up downstream changes | 21:03 |
| rlandy|rover | most jobs are running periodically | 21:03 |
| sshnaidm | rlandy|rover, but overall have some progress, at least job starts on rhel7 :) | 21:05 |
| rlandy|rover | k - good | 21:05 |
| *** jbadiapa has quit IRC | 21:35 | |
| *** apetrich has quit IRC | 21:57 | |
| *** rfolco has quit IRC | 22:18 | |
| *** tosky has quit IRC | 22:54 | |
| *** chem has quit IRC | 23:12 | |
| *** chem has joined #oooq | 23:14 | |
| *** TrevorV has quit IRC | 23:20 | |
| *** rlandy|rover has quit IRC | 23:38 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!