| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike-branch, legacy-tripleo-ci- (3 more messages) | 00:47 |
|---|---|---|
| *** weshay has joined #oooq | 01:57 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike-branch, legacy-tripleo-ci- (3 more messages) | 02:47 |
| *** skramaja has joined #oooq | 02:52 | |
| *** ykarel|away has joined #oooq | 03:39 | |
| *** ykarel|away is now known as ykarel | 03:49 | |
| *** udesale has joined #oooq | 03:57 | |
| *** ratailor has joined #oooq | 04:25 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, master: legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike-branch, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates- (2 more messages) | 04:47 |
| *** agopi_ has joined #oooq | 04:50 | |
| *** ykarel has quit IRC | 04:51 | |
| *** agopi|training has quit IRC | 04:53 | |
| *** sanjayu_ has quit IRC | 04:53 | |
| *** ykarel has joined #oooq | 05:07 | |
| *** jaosorior has joined #oooq | 05:20 | |
| *** chkumar|off is now known as chkumar|ruck | 05:42 | |
| *** quique|rover|off is now known as quiquell|rover | 05:45 | |
| *** jfrancoa has joined #oooq | 06:02 | |
| *** jtomasek has joined #oooq | 06:27 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, master: legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike-branch, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates- (2 more messages) | 06:47 |
| *** sanjayu_ has joined #oooq | 07:05 | |
| *** quiquell|rover is now known as quique|rover|brb | 07:10 | |
| *** holser_ has joined #oooq | 07:10 | |
| *** ykarel is now known as ykarel|lunch | 07:25 | |
| *** quique|rover|brb is now known as quiquell|rover | 07:42 | |
| *** agopi__ has joined #oooq | 07:46 | |
| *** agopi_ has quit IRC | 07:49 | |
| *** tosky has joined #oooq | 07:54 | |
| *** bogdando has joined #oooq | 07:58 | |
| bogdando | PTAL https://review.openstack.org/#/c/595265/ https://review.openstack.org/#/c/592996/ | 08:05 |
| bogdando | and https://review.openstack.org/#/c/576746/ | 08:09 |
| chkumar|ruck | brb | 08:09 |
| *** ykarel|lunch is now known as ykarel | 08:35 | |
| *** dtantsur|afk is now known as dtantsur | 08:38 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, master: legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike-branch, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates- (2 more messages) | 08:47 |
| *** panda|off is now known as panda | 09:01 | |
| *** sshnaidm has joined #oooq | 09:07 | |
| quiquell|rover | sshnaidm: http://dashboard-ci.tripleo.org/d/poOr-d0mk/ansible-exploration?orgId=1 | 09:25 |
| sshnaidm | quiquell|rover, nice | 09:27 |
| sshnaidm | quiquell|rover, let's sync about that tomorrow | 09:28 |
| sshnaidm | quiquell|rover, need to think how to do it more clear and "exploratable" | 09:29 |
| *** sshnaidm is now known as sshnaidm|afk | 09:29 | |
| quiquell|rover | sshnaidm: sure, btw can you take a look at this ARA issue https://bugs.launchpad.net/tripleo/+bug/1794238 ? | 09:29 |
| openstack | Launchpad bug 1794238 in tripleo "overcloud ARA is the same as undercloud ARA" [High,In progress] - Assigned to Quique Llorente (quiquell) | 09:29 |
| quiquell|rover | Don't want to spend time there, want to go back to timeouts | 09:30 |
| sshnaidm|afk | quiquell|rover, yeah, will handle this | 09:34 |
| quiquell|rover | sshnaidm|afk: I need to gather mor info, have add a "offset" parameter to telegraf python to get old jobs | 09:34 |
| sshnaidm|afk | quiquell|rover, but we just started to collect data for undercloud/overcloud | 09:35 |
| quiquell|rover | sshnaidm|afk: undercloud is all | 09:35 |
| quiquell|rover | sshnaidm|afk: undercloud was in place before August | 09:35 |
| quiquell|rover | sshnaidm|afk: and global ara is older | 09:35 |
| quiquell|rover | sshnaidm|afk: I am using the json, not the influxdb line there (I don't need that though) | 09:36 |
| sshnaidm|afk | quiquell|rover, uh, ok | 09:36 |
| *** sshnaidm|afk is now known as sshnaidm|off | 09:37 | |
| *** sshnaidm|off has quit IRC | 09:42 | |
| *** jfrancoa has quit IRC | 09:43 | |
| *** jfrancoa has joined #oooq | 09:46 | |
| *** sanjayu__ has joined #oooq | 10:15 | |
| *** sanjayu_ has quit IRC | 10:18 | |
| quiquell|rover | chkumar|ruck: Can you cover CIX today too ? | 10:21 |
| chkumar|ruck | quiquell|rover: yes | 10:33 |
| chkumar|ruck | quiquell|rover: but I need some help on few bugs | 10:34 |
| chkumar|ruck | quiquell|rover: pulling the card | 10:34 |
| quiquell|rover | chkumar|ruck: Let's review CIX | 10:34 |
| quiquell|rover | chkumar|ruck: Going to add info to cards I know of | 10:34 |
| chkumar|ruck | quiquell|rover: https://trello.com/c/dfAprCuv/761-cixlp1794251tripleociproa-master-undercloud-reinstall-failed-with-invalid-selinux-context-errno-95-operation-not-supported | 10:34 |
| chkumar|ruck | quiquell|rover: https://bugs.launchpad.net/tripleo/+bug/1794251 | 10:36 |
| openstack | Launchpad bug 1794251 in tripleo "[master] undercloud reinstall failed with invalid selinux context: [Errno 95] Operation not supported" [Critical,Triaged] | 10:36 |
| * quiquell|rover checking | 10:36 | |
| chkumar|ruck | Does anyone tried standalone on fs028? | 10:37 |
| chkumar|ruck | *fedora 28 | 10:41 |
| panda | chkumar|ruck: there are patches to make it work | 10:42 |
| panda | chkumar|ruck: testing and landing thos patches is part of the next sprint | 10:42 |
| *** ratailor_ has joined #oooq | 10:43 | |
| *** ratailor has quit IRC | 10:44 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates @ https://review.openstack.org/604298, master: legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike-branch, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates- (2 more messages) | 10:47 |
| quiquell|rover | chkumar|ruck: You have stuff about this https://trello.com/c/LNHvG9LW/760-cixlp1794228tripleociproa-queenspike-no-package-uwsgi-plugin-python2-available-during-barbican-api-container-build ? | 10:48 |
| chkumar|ruck | quiquell|rover: yes | 10:48 |
| chkumar|ruck | quiquell|rover: fixes are up | 10:48 |
| quiquell|rover | chkumar|ruck: Going to update the review for https://trello.com/c/ocIzDwCI/750-cixlp1793293tripleociproa-pike-promotion-failing-trying-to-use-ipv6-to-get-buildlogs | 10:49 |
| quiquell|rover | chkumar|ruck: To remove -1 of the fixing review | 10:50 |
| chkumar|ruck | quiquell|rover: ack | 10:50 |
| chkumar|ruck | quiquell|rover: need to check fs-16 and fs016 does telemetry tests are fixed or not | 10:52 |
| quiquell|rover | chkumar|ruck: CIX updated, you are covered now | 10:58 |
| chkumar|ruck | quiquell|rover: thanks :-) | 10:59 |
| quiquell|rover | chkumar|ruck: about timeouts, we still have just validation, we have a disable validations review, and also created a tool to navigate ansible between builds | 10:59 |
| quiquell|rover | chkumar|ruck: what's up with fs016 ? | 10:59 |
| chkumar|ruck | quiquell|rover: yes | 10:59 |
| chkumar|ruck | quiquell|rover: fs016 tempest tests failed due to ssh timed out error, but I runned again in test-project but it passed | 11:00 |
| chkumar|ruck | quiquell|rover: I am waiting for next periodic run if it is visible then I need to investigation in ciner or neutron logs | 11:00 |
| quiquell|rover | chkumar|ruck: so it fails at periodics but not manually ? | 11:00 |
| chkumar|ruck | quiquell|rover: yes, just one time failure | 11:00 |
| quiquell|rover | chkumar|ruck: But we disable tempest there | 11:01 |
| quiquell|rover | chkumar|ruck: It's new temptes failure ? | 11:01 |
| chkumar|ruck | quiquell|rover: we disable telemetry test failure, not volume boot test | 11:01 |
| quiquell|rover | chkumar|ruck: now volume is failing ? | 11:01 |
| quiquell|rover | holy sh... | 11:01 |
| chkumar|ruck | volume boot test cannot be diasbled it is the minmail tempest tests running in multiple fs | 11:01 |
| quiquell|rover | chkumar|ruck: so you talk about disable fs016 at promotions ? | 11:02 |
| chkumar|ruck | quiquell|rover: no once server is boot, user tryied to ssh into the instance then ssh happened, then agin they tried to ssh into another ssh that time it was timed out after 22 attempest | 11:02 |
| chkumar|ruck | quiquell|rover: nope, let's wait for next run | 11:02 |
| chkumar|ruck | quiquell|rover: that's why I have not opened a bug | 11:03 |
| quiquell|rover | chkumar|ruck: ack | 11:13 |
| *** udesale has quit IRC | 11:17 | |
| *** panda is now known as panda|afk | 11:19 | |
| *** dtantsur is now known as dtantsur|bbl | 11:25 | |
| chkumar|ruck | brb back before CIX call | 11:26 |
| *** jfrancoa has quit IRC | 11:42 | |
| *** jfrancoa has joined #oooq | 11:43 | |
| weshay | chkumar|ruck, quiquell|rover howdy | 11:45 |
| quiquell|rover | weshay: ansible explorer http://dashboard-ci.tripleo.org/d/poOr-d0mk/ansible-exploration?orgId=1 | 11:46 |
| quiquell|rover | weshay: filter by timed_out for example http://dashboard-ci.tripleo.org/d/poOr-d0mk/ansible-exploration?orgId=1&var-influxdb_filter=job_result%7C%3D%7CTIMED_OUT | 11:47 |
| quiquell|rover | weshay: btw, what do you mean with validations not designed for upstream ci ? | 11:48 |
| quiquell|rover | weshay: the disabling patch is controversial | 11:48 |
| *** quiquell|rover is now known as quique|rover|lch | 11:49 | |
| chkumar|ruck | weshay: \o/ | 11:49 |
| chkumar|ruck | weshay: panda|afk fs021 is completely broken https://bugs.launchpad.net/tripleo/+bug/1794258 | 11:49 |
| openstack | Launchpad bug 1794258 in tripleo "[master][Rocky] fs021 periodic job is failing with different issues" [Critical,Triaged] | 11:49 |
| chkumar|ruck | need some help here | 11:49 |
| weshay | quique|rover|lch,++ | 12:02 |
| weshay | quique++ | 12:02 |
| weshay | chkumar|ruck, feel free to comment | 12:03 |
| chkumar|ruck | weshay: today we have rocky promotion in the morning | 12:06 |
| weshay | I see that now :) | 12:06 |
| weshay | chkumar++ | 12:06 |
| chkumar|ruck | weshay: there are few blockers for master fixes are in progress, we will get for master also | 12:07 |
| weshay | quique|rover|lch, nice job diagnosing the timeouts | 12:08 |
| weshay | quique|rover|lch, are you available after lunch to give me an update? | 12:08 |
| ssbarnea|bkp | fyi fc28 public image is now available on rdo, no need to upload it anymore. | 12:13 |
| ykarel | weshay, i looked a little to timeouts and it doesn't look related to overcloud deployment timing out | 12:19 |
| ykarel | it something to do with mistral/tripleoclient | 12:19 |
| ykarel | the overcloudrc create just stucks and raises exception | 12:19 |
| ykarel | there me be already more findings on ^^, i just looked so shared | 12:20 |
| ykarel | if not already involved, good to involve someone from mistral | 12:21 |
| ykarel | chkumar|ruck, quique|rover|lch in case u have noticed ^^ | 12:22 |
| weshay | chkumar|ruck, quique|rover|lch do you guys have a bug on legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch | 12:23 |
| *** trown|outtypewww is now known as trown | 12:23 | |
| weshay | chkumar|ruck, quique|rover|lch and legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master | 12:24 |
| ykarel | weshay, chkumar|ruck quique|rover|lch and if the timeout issue is just master, i really doubt https://review.openstack.org/#/c/603802/ | 12:24 |
| chkumar|ruck | weshay: checking | 12:25 |
| chkumar|ruck | weshay: https://bugs.launchpad.net/tripleo/+bug/1793073 it might be related to that | 12:28 |
| openstack | Launchpad bug 1793073 in tripleo "[queens] fs01 noop job failed with Stderr: u'/usr/bin/ironic-inspector-rootwrap: Unauthorized command: systemctl start openstack-ironic-inspector-dnsmasq.service (no filter matched)" [Critical,Triaged] - Assigned to Quique Llorente (quiquell) | 12:28 |
| *** agopi__ has quit IRC | 12:34 | |
| *** ratailor_ has quit IRC | 12:37 | |
| weshay | chkumar|ruck, thanks | 12:38 |
| weshay | chkumar|ruck, quique|rover|lch https://review.rdoproject.org/r/16449 | 12:41 |
| weshay | ssbarnea, make any progress on fedora? | 12:41 |
| ssbarnea | weshay: yep, just made a minor change to https://review.openstack.org/#/c/602492/ | 12:42 |
| ssbarnea | i wanted to ask you few things, do you have some time? | 12:42 |
| ssbarnea | can be BJ or irc | 12:43 |
| weshay | ssbarnea, sure man | 12:43 |
| weshay | let's blue | 12:43 |
| weshay | quicker | 12:43 |
| ssbarnea | zzbj | 12:44 |
| weshay | https://bluejeans.com/u/whayutin/ | 12:44 |
| ssbarnea|bkp | weshay please send me you meeting it, i cannot use the browse bj. sound is not working in it. | 12:46 |
| ssbarnea|bkp | standalone works but i need number | 12:46 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, master: legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master, legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike-branch, legacy-tripleo-ci-centos-7-container-to-container-upgrades- (2 more messages) | 12:47 |
| *** panda|afk is now known as panda | 12:49 | |
| weshay | ssbarnea|bkp, http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/roles/libvirt/setup/overcloud/tasks/vars/libvirt_nodepool_vars.yml | 12:52 |
| *** ssbarnea has quit IRC | 12:56 | |
| *** agopi__ has joined #oooq | 13:05 | |
| *** ssbarnea has joined #oooq | 13:06 | |
| *** quique|rover|lch is now known as quiquell|rover | 13:09 | |
| quiquell|rover | weshay: I am up to talk about timeouts | 13:11 |
| weshay | quiquell|rover, ok I am ready in 3min | 13:12 |
| quiquell|rover | weshay: let me know when you are free | 13:12 |
| *** skramaja has quit IRC | 13:12 | |
| ykarel | quiquell|rover, is there some place where these updates(timeouts) are tracked, i too wanted to see | 13:13 |
| weshay | https://bluejeans.com/u/whayutin/ | 13:13 |
| ykarel | in case i looked something wrong, just correct me | 13:16 |
| *** vinaykns has joined #oooq | 13:18 | |
| quiquell|rover | ykarel: we have a job's explorer http://dashboard-ci.tripleo.org/d/FEdraO0ik/jobs-exploration?orgId=1 | 13:42 |
| quiquell|rover | ykarel: from there you can filter by /update/ regex and result TIMED_OUT | 13:43 |
| *** sanjayu__ has quit IRC | 13:43 | |
| *** saneax has joined #oooq | 13:43 | |
| quiquell|rover | ykarel: for example this http://dashboard-ci.tripleo.org/d/FEdraO0ik/jobs-exploration?orgId=1&var-influxdb_filter=result%7C%3D%7CTIMED_OUT&var-influxdb_filter=job_name%7C%3D~%7C%2Fupdates%2F | 13:43 |
| quiquell|rover | panda: We have to watch this https://review.openstack.org/#/c/604706/ | 13:47 |
| quiquell|rover | panda: It's introducing new bash | 13:48 |
| weshay | anybody need a 1-1 that I've had to skip recently? | 13:54 |
| weshay | quiquell|rover, chkumar|ruck fyi.. so for bugs that block the gate.. the current best way to handle that .. is to create a bug w/ alert && promotion blocker | 13:55 |
| weshay | quiquell|rover, chkumar|ruck once the card is created in trello.. add all the upstream tags... tripleo-master, tripleo-rocky, tripleo-queens | 13:56 |
| weshay | I'll try to add that to the ruck / rover docs so it's more clear | 13:56 |
| panda | quiquell|rover: ouch | 13:57 |
| quiquell|rover | panda: yep, take a look at the review, we have to help with that | 13:58 |
| weshay | tripleo mtg | 14:00 |
| weshay | rfolco, you around buddy? | 14:01 |
| rfolco | weshay, o/ | 14:01 |
| weshay | rfolco, hey.. so let's chat for a minute during this tripleo mtg | 14:01 |
| rfolco | bj? | 14:01 |
| weshay | rfolco, https://bluejeans.com/4113567798 | 14:02 |
| *** dtantsur|bbl is now known as dtantsur | 14:04 | |
| weshay | rfolco, https://etherpad.openstack.org/p/tripleo-meeting-items | 14:12 |
| ykarel | quiquell|rover, currently in meeting, will check in few minutes | 14:18 |
| chkumar|ruck | weshay: sure | 14:19 |
| *** quiquell|rover is now known as quique|rover|off | 14:20 | |
| weshay | rfolco, https://drive.google.com/drive/u/1/folders/1duziyA5IpQLrzs4JncjKm6FHLOlDh5_- | 14:23 |
| *** fultonj has joined #oooq | 14:27 | |
| *** fhubik has joined #oooq | 14:28 | |
| ykarel | quique|rover|off, ok got that, is timeout issue clear(what caused that), can discuss tomorrow, i think overcloud deployment is completed, just overcloudrc creation is stuck, actually i wanted to know what releases are having issue and when exactly it started to narrow down the problem, if it started failing on 21st then tripleoclient patch seems issue, if not then can check other parts | 14:29 |
| ykarel | and validation parts doesn't seem to be an actual issue that's just adding extra time to job | 14:30 |
| ykarel | but yes good to clear for ci | 14:30 |
| *** agopi__ is now known as agopi | 14:33 | |
| weshay | marios, panda https://review.openstack.org/#/c/600517/31 | 14:35 |
| marios | weshay: ack | 14:40 |
| *** jfrancoa has quit IRC | 14:45 | |
| *** fhubik is now known as fhubik|brb | 14:46 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, master: legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master, legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike-branch, legacy-tripleo-ci-centos-7-container-to-container-upgrades- (2 more messages) | 14:47 |
| *** ykarel is now known as ykarel|away | 14:57 | |
| weshay | anyone know what is going on here? https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset021-master/093617f/job-output.txt.gz#_2018-09-25_00_51_59_315241 | 14:58 |
| ykarel|away | weshay, following seems actual issue | 15:02 |
| ykarel|away | 2018-09-25 00:52:48.274 | rsync: change_dir "/home/zuul/src/*/openstack/tripleo-upgrade" failed: No such file or directory (2) | 15:02 |
| ykarel|away | 2018-09-25 00:52:48.366 | rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]non-zero return code | 15:02 |
| *** kopecmartin is now known as kopecmartin|off | 15:11 | |
| *** udesale has joined #oooq | 15:12 | |
| *** chkumar|ruck is now known as chkumar|off | 15:17 | |
| weshay | ykarel|away, thanks! | 15:20 |
| *** ykarel|away has quit IRC | 15:22 | |
| *** fhubik|brb is now known as fhubik | 15:23 | |
| *** jtomasek has quit IRC | 15:26 | |
| chem | hi I've got "Failed to parse dlrn hash" when I re-run repo-setup in the standalone-upgrade jobs, anybody familiar with this kind of error ? | 15:28 |
| weshay | chem, hey.. let's see the log | 15:30 |
| weshay | chem, probably a blip | 15:30 |
| chem | weshay: http://logs.openstack.org/06/604706/5/check/tripleo-ci-centos-7-standalone-upgrade/5477a4b/logs/undercloud/home/zuul/repo_setup_upgrade.log.txt.gz | 15:30 |
| chem | weshay: hum ... anyway it's the wrong hash | 15:32 |
| chem | weshay: according to http://logs.openstack.org/06/604706/5/check/tripleo-ci-centos-7-standalone-upgrade/5477a4b/logs/emit_releases_file.log | 15:32 |
| *** sanjayu_ has joined #oooq | 15:33 | |
| chem | weshay: I should be using standalone_target_hash not standalone_deploy_hash at that time, so I must be missing something | 15:34 |
| *** saneax has quit IRC | 15:35 | |
| weshay | seems ritht http://logs.openstack.org/06/604706/5/check/tripleo-ci-centos-7-standalone-upgrade/5477a4b/logs/emit_releases_file.log | 15:35 |
| weshay | panda, can you help me look at this w/ chem | 15:36 |
| weshay | I think you are more familiar w/ it | 15:36 |
| chem | it is but this is the repo call before the upgrade so I should be using the target hash (48a0d56e4ba547ab10a35888138370fb1ec74a97_31a62456) not the deploy hash (e89450c44e41ec2ddada7909e63f1edc1aa1afdd) | 15:37 |
| panda | chem: maybe you're not passing the correct release on the second playbook call | 15:38 |
| chem | panda: is that not enough https://review.openstack.org/#/c/604706/5/playbooks/tripleo-ci/templates/toci_quickstart.sh.j2 ? | 15:38 |
| panda | chem: looking | 15:38 |
| weshay | chem, I wonder if container-prep is messing this up http://logs.openstack.org/06/604706/5/check/tripleo-ci-centos-7-standalone-upgrade/5477a4b/logs/emit_releases_file.log | 15:39 |
| weshay | oops | 15:39 |
| weshay | https://review.openstack.org/#/c/604736/5/roles/standalone-upgrade/meta/main.yml | 15:39 |
| weshay | nope | 15:39 |
| chem | weshay: hum copy/pasta, but in the end It's not used I believe | 15:40 |
| weshay | panda, do we have no_log set of repo setup for any particular reason? | 15:41 |
| panda | weshay: not sure, maybe credentials ? | 15:43 |
| panda | that's usually why we used in the other tasks | 15:43 |
| chem | panda: weshay so the cli is indeed correct and use the target hash -> http://logs.openstack.org/06/604706/5/check/tripleo-ci-centos-7-standalone-upgrade/5477a4b/job-output.txt.gz#_2018-09-25_14_46_19_101312 | 15:46 |
| panda | chem: the invocation looks good, let me take a look at the logs | 15:47 |
| weshay | chem, panda the error is from config/release/tripleo-ci/master.yml line 40 | 15:48 |
| weshay | if [[ -z "$rdo_dlrn" || -z "$tripleo_dlrn" ]]; then | 15:48 |
| weshay | chem, I supsect both curl's failed? | 15:48 |
| chem | weshay: yes | 15:48 |
| weshay | https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/master.yml#L40 | 15:49 |
| weshay | panda, chem maybe we could log that call vs.. silent | 15:49 |
| weshay | oh wait | 15:50 |
| marios | /win 7 | 15:51 |
| weshay | chem, panda we should see dlrn_repo_curl_errors.log | 15:51 |
| weshay | maybe ~ is not working | 15:52 |
| bogdando | PTAL https://review.openstack.org/#/c/595265/ https://review.openstack.org/#/c/592996/ and https://review.openstack.org/#/c/576746/ | 15:52 |
| chem | weshay: panda I've got to call it a day, will look again later, let me know if you get something in the review https://review.openstack.org/#/c/604706/ or somewhere :) | 15:52 |
| chem | weshay: panda thanks for the help | 15:52 |
| bogdando | those have +2 from you, weshay so please do not bother :) | 15:52 |
| bogdando | there is also https://review.openstack.org/#/c/605021/ | 15:53 |
| panda | bogdando: I have a question pending on 595265 | 15:55 |
| weshay | chem, panda when in doubt.. http://mirror.iad.rax.openstack.org:8080/rdo/centos7/59/88 | 15:55 |
| weshay | lol | 15:55 |
| weshay | vs.. | 15:55 |
| weshay | chem, panda when in doubt.. http://mirror.iad.rax.openstack.org:8080/rdo/centos7/59/ | 15:55 |
| weshay | chem, you hit an infra error | 15:55 |
| weshay | suprise suprsie | 15:55 |
| panda | heh | 15:56 |
| chem | weshay: ha ... how do we solve this ? | 15:56 |
| weshay | chem, I enjoy weeping | 15:56 |
| panda | retry until you consume the hard disk heads | 15:56 |
| chem | ah, nice :) will do then ... | 15:56 |
| weshay | chem, you could bug infra.. but they will tell you to pound sand probably | 15:57 |
| weshay | rfolco, ssbarnea|bkp ^ | 15:57 |
| weshay | rfolco, ssbarnea|bkp chkumar|off FYI.. opportuity to check on the status of upstream proxies http://mirror.iad.rax.openstack.org:8080/rdo/centos7/59/ | 15:58 |
| weshay | works | 15:58 |
| weshay | but not http://mirror.iad.rax.openstack.org:8080/rdo/centos7/59/88 | 15:58 |
| chem | weshay: hum, nice. need to go | 15:58 |
| chem | weshay: panda thanks | 15:58 |
| *** ykarel|away has joined #oooq | 15:59 | |
| *** ykarel|away is now known as ykarel | 15:59 | |
| panda | weshay: why 59/88 ? | 15:59 |
| panda | weshay: I see the hash is 48/a0 | 15:59 |
| weshay | panda, that is in the log | 15:59 |
| weshay | really? | 15:59 |
| weshay | I just closed everything | 16:00 |
| * weshay looks again | 16:00 | |
| weshay | curl --silent http://mirror.iad.rax.openstack.org:8080/rdo/centos7/59/88/59889fc7fc8e3390d9bd4f0bf01d3a8cdaab6785_fd06359b/delorean.repo -S | 16:00 |
| weshay | curl --silent http://mirror.iad.rax.openstack.org:8080/rdo/centos7/e8/94/e89450c44e41ec2ddada7909e63f1edc1aa1afdd_c4b405a1/delorean.repo -S | 16:00 |
| weshay | ur right | 16:01 |
| weshay | panda, same story | 16:01 |
| weshay | http://mirror.iad.rax.openstack.org:8080/rdo/centos7/e8/ works | 16:01 |
| panda | http://mirror.iad.rax.openstack.org:8080/rdo/centos7/48/a0/48a0d56e4ba547ab10a35888138370fb1ec74a97_31a62456/ | 16:01 |
| weshay | http://mirror.iad.rax.openstack.org:8080/rdo/centos7/e8/94/ does not | 16:01 |
| weshay | panda, look at the bottom of http://logs.openstack.org/06/604706/5/check/tripleo-ci-centos-7-standalone-upgrade/5477a4b/logs/undercloud/home/zuul/repo_setup_upgrade.log.txt.gz#_2018-09-25_14_46_31 | 16:01 |
| weshay | ur wrong | 16:02 |
| *** sshnaidm|off has joined #oooq | 16:02 | |
| panda | weshay: I was looking at http://logs.openstack.org/06/604706/5/check/tripleo-ci-centos-7-standalone-upgrade/5477a4b/job-output.txt.gz#_2018-09-25_14_46_19_101312 | 16:02 |
| ssbarnea|bkp | i have no knowledge on how the mirrors are configured (or synced). | 16:03 |
| weshay | ssbarnea|bkp, ya.. that is an area where we need help | 16:03 |
| weshay | a lot | 16:03 |
| weshay | ssbarnea|bkp, would be a +100 to find out | 16:03 |
| bogdando | panda: I cannot see a case one wants to chown not being a root. That would fail miserably with a permission denied, won't it? | 16:03 |
| weshay | https://specs.openstack.org/openstack-infra/infra-specs/specs/unified_mirrors.html | 16:03 |
| bogdando | and when it won't fail, there is nothing to chown | 16:04 |
| bogdando | correction to chown as a+x | 16:04 |
| bogdando | and being not a non_root_user | 16:05 |
| *** fhubik has quit IRC | 16:06 | |
| bogdando | sorry, I mean if non_root_user:non_root_user wants to chown something else, it needs to be sudo | 16:06 |
| ssbarnea|bkp | (reading now about it, i would personally have went for a squid approach, which doesn't even need sync and also avoids the need to mention which mirrors to use, you only need to configure the proxy to use) | 16:06 |
| bogdando | otherwise that is a pointless | 16:06 |
| bogdando | panda: does that makes any sense? | 16:06 |
| *** sshnaidm|off has quit IRC | 16:06 | |
| *** sshnaidm has joined #oooq | 16:06 | |
| bogdando | panda: ok, I got it, perhaps I should fix the name as well, as I needed to run that being non root! | 16:07 |
| panda | weshay: so basically the repo setup role is ignoring the dlrn_hash we're passing, and is trying to download tripleo-current anyway ? | 16:07 |
| weshay | panda, sorry? | 16:08 |
| weshay | ssbarnea|bkp, rfolco join sf-ops please | 16:08 |
| weshay | internal | 16:08 |
| weshay | panda, ya.. https://trunk.rdoproject.org/centos7/e8/94/ | 16:12 |
| weshay | and https://trunk.rdoproject.org/centos7/59/88 | 16:12 |
| weshay | don't exist | 16:12 |
| weshay | ssbarnea|bkp, sorry it's not a mirror issue as you saw jpena point out | 16:12 |
| ssbarnea | ok, seen. | 16:13 |
| panda | weshay: e89450c44e41ec2ddada7909e63f1edc1aa1afdd_c4b405a1 is the hash we're using in the standalone install , 48a0d56e4ba547ab10a35888138370fb1ec74a97_31a62456/ is the one to update to. The install works, the upgrade fails IIUC. Indeed look at http://logs.openstack.org/06/604706/5/check/tripleo-ci-centos-7-standalone-upgrade/5477a4b/logs/undercloud/home/zuul/repo_setup.log.txt.gz | 16:14 |
| panda | but in the repo_setup_upgrade we're using the same hashes, and for some reason they now fail. | 16:14 |
| weshay | panda, k.. I guess he should be installing w/ previous_current_tripleo | 16:14 |
| weshay | and upgrading to... current_tripleo | 16:14 |
| weshay | chem, ^ | 16:14 |
| panda | weshay: HTEY ARE FOR ROCKY | 16:14 |
| panda | weshay: that's why they are not there | 16:15 |
| panda | weshay: the hash you're looking for is for rocky | 16:15 |
| weshay | yup | 16:15 |
| weshay | https://trunk.rdoproject.org/centos7-rocky/59/88/ | 16:15 |
| weshay | mutha fucka | 16:15 |
| weshay | panda, he's using a master release file though | 16:15 |
| panda | but with the wrong hashehs | 16:16 |
| panda | even if we are passing the right ones | 16:16 |
| bogdando | panda: commented | 16:16 |
| panda | bogdando: thanks sorry I've been cauht in the middle of upgrade scandal | 16:17 |
| weshay | panda, http://logs.openstack.org/06/604706/5/check/tripleo-ci-centos-7-standalone-upgrade/5477a4b/job-output.txt.gz#_2018-09-25_14_46_19_091665 | 16:17 |
| weshay | panda, there is a bug here https://trunk.rdoproject.org/centos7-master/48/a0/ | 16:17 |
| weshay | some where | 16:17 |
| weshay | ssbarnea, rfolco for future ref: https://github.com/openstack-infra/system-config/blob/master/modules/openstack_project/manifests/mirror.pp#L3 | 16:18 |
| weshay | panda, https://review.openstack.org/#/c/604706/5/scripts/emit_releases_file/emit_releases_file.py | 16:20 |
| weshay | shouldn't the release dict include more than just the hash | 16:21 |
| panda | weshay: how are the extra-vars we pass in command line handled in the repo_setup ? | 16:21 |
| weshay | panda, I think the issue is that emit releases does not emit a release | 16:22 |
| *** jfrancoa has joined #oooq | 16:22 | |
| weshay | lol | 16:22 |
| weshay | hrm.. ok.. emit releases does | 16:22 |
| weshay | but we're not converting the data in http://logs.openstack.org/06/604706/5/check/tripleo-ci-centos-7-standalone-upgrade/5477a4b/logs/emit_releases_file.log | 16:23 |
| weshay | to the release vars we need | 16:23 |
| panda | weshay: http://logs.openstack.org/06/604706/5/check/tripleo-ci-centos-7-standalone-upgrade/5477a4b/logs/releases.sh | 16:24 |
| panda | this is the output | 16:24 |
| weshay | I guess that is baked into the relase file | 16:24 |
| panda | I can't find any reference of dlrn_hash in the release file. We have all sorts of dlrn_* | 16:24 |
| panda | tags, paths, newest path and tags | 16:24 |
| weshay | panda, k .. that looks good | 16:24 |
| ssbarnea|bkp | need to go, will be back in ~45mins. | 16:25 |
| *** udesale has quit IRC | 16:27 | |
| weshay | panda, I dont see a bug :( | 16:29 |
| weshay | +(./toci_quickstart.sh:140): echo '--extra-vars @/home/zuul/workspace/.quickstart/config/release/tripleo-ci/master.yml -e dlrn_hash=48a0d56e4ba547ab10a35888138370fb1ec74a97_31a62456 -e get_build_command=48a0d56e4ba547ab10a35888138370fb1ec74a97_31a62456' | 16:29 |
| weshay | 2018-09-25 14:46:19.092084 | primary | --extra-vars @/home/zuul/workspace/.quickstart/config/release/tripleo-ci/master.yml -e dlrn_hash=48a0d56e4ba547ab10a35888138370fb1ec74a97_31a62456 -e get_build_command=48a0d56e4ba547ab10a35888138370fb1ec74a97_31a62456 | 16:29 |
| weshay | that seems right | 16:29 |
| panda | yes, that part is good | 16:30 |
| panda | the part that fails is that we are not using those hashes when setting up the repo | 16:30 |
| panda | we are using the hashes from rocky | 16:31 |
| panda | repo_setup is using e89450c44e41ec2ddada7909e63f1edc1aa1afdd_c on master | 16:31 |
| panda | and fails | 16:31 |
| panda | there's nothing passing the hash we specified in command line to the setup repo | 16:32 |
| weshay | http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/roles/repo-setup/tasks/get-dlrn-hash-newest.yml#n30 | 16:32 |
| weshay | panda, maybe we couldn't override release ? | 16:32 |
| panda | no mention od dlrn_hash in these tasks | 16:33 |
| weshay | "release": "master", | 16:34 |
| weshay | http://logs.openstack.org/06/604706/5/check/tripleo-ci-centos-7-standalone-upgrade/5477a4b/logs/undercloud/var/log/extra/dump_variables_vars.json.txt.gz | 16:34 |
| panda | release is correct, dlrn_hash is not. | 16:35 |
| weshay | "dlrn_hash": "e89450c44e41ec2ddada7909e63f1edc1aa1afdd_c4b405a1", | 16:35 |
| weshay | "dlrn_hash_newest": "59889fc7fc8e3390d9bd4f0bf01d3a8cdaab6785_fd06359b", | 16:35 |
| weshay | "dlrn_hash_path": "e8/94/e89450c44e41ec2ddada7909e63f1edc1aa1afdd_c4b405a1", | 16:35 |
| weshay | "dlrn_hash_path_newest": "59/88/59889fc7fc8e3390d9bd4f0bf01d3a8cdaab6785_fd06359b", | 16:35 |
| weshay | http://logs.openstack.org/06/604706/5/check/tripleo-ci-centos-7-standalone-upgrade/5477a4b/logs/undercloud/var/log/extra/dump_variables_vars.json.txt.gz | 16:35 |
| weshay | yup | 16:35 |
| weshay | that's the issue | 16:35 |
| weshay | panda, maybe the order of the args? | 16:36 |
| *** bogdando has quit IRC | 16:36 | |
| weshay | panda, I think I see why | 16:37 |
| panda | weshay: if you look at where we are failing https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/master.yml#L38 we are taking the has value from dlrn_hash_path_newest or dlrn_hash_tag_newest | 16:37 |
| weshay | panda, https://github.com/openstack/tripleo-quickstart/blob/master/roles/repo-setup/tasks/main.yml#L3 | 16:38 |
| weshay | panda, the fact caching kills us here | 16:38 |
| weshay | panda, those always need to run when it's an upgrade | 16:39 |
| weshay | chem, ^ | 16:39 |
| *** sanjayu_ has quit IRC | 16:40 | |
| panda | that may be correct. I still don't see how -e dlrn_hash becomes dlrn_hash_path_newest or dlrn_hash_tag_newest | 16:40 |
| weshay | upgrade_type: standalone-upgrade | 16:41 |
| weshay | panda, http://codesearch.openstack.org/?q=dlrn_hash_path_newest&i=nope&files=&repos= | 16:41 |
| weshay | panda, http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/roles/repo-setup/tasks/get-dlrn-hash-newest.yml#n32 | 16:41 |
| *** jtomasek has joined #oooq | 16:42 | |
| weshay | panda, there is no ansible variable indicating it's an upgrade | 16:43 |
| weshay | it's in the zuul inventory | 16:44 |
| weshay | IMHO.. the upgrade playbooks should set some fact | 16:44 |
| weshay | about what it's doing | 16:44 |
| weshay | http://logs.openstack.org/06/604706/5/check/tripleo-ci-centos-7-standalone-upgrade/5477a4b/logs/undercloud/var/log/extra/dump_variables_vars.json.txt.gz | 16:44 |
| *** dtantsur is now known as dtantsur|afk | 16:45 | |
| panda | I still can't find any place taht takes that dlrn_hash, and transform it into something that is usable by that curl | 16:45 |
| panda | like in here http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/config/release/centosci/master-current-tripleo.yml#n4 | 16:45 |
| panda | we are clearly using "dlrn_hash" | 16:46 |
| panda | can't find anything similar for the curl command | 16:46 |
| panda | or any path of transformation from dlrn_has to dlrn_hash_path | 16:46 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, master: legacy-tripleo-ci-centos-7-containers-multinode-upgrades-pike-branch, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates- (2 more messages) | 16:47 |
| panda | but yeah, fact caching may be an issue here anyway, depending on how we deal with thos variables | 16:47 |
| *** dtantsur|afk has quit IRC | 16:51 | |
| *** panda is now known as panda|off | 16:52 | |
| weshay | panda|off, chem /me pushing a patch to the playbook | 16:54 |
| weshay | instead of including the repo-setup role | 16:54 |
| weshay | I think it should only run the repo-setup/task/get-dlrn-hash.yml | 16:54 |
| *** dtantsur has joined #oooq | 16:58 | |
| *** dtantsur is now known as dtantsur|afk | 16:58 | |
| panda|off | weshay: no repo for the new releases ? so you'll just download the containers with the new tags? is that enough ? | 16:59 |
| weshay | panda|off, I think https://review.openstack.org/#/c/605149/1/playbooks/multinode-standalone-upgrade.yml | 17:00 |
| weshay | will do the trick | 17:00 |
| weshay | lines 2-11 | 17:01 |
| weshay | panda|off, it's the logic in main.yml that is fucking it up | 17:01 |
| *** ykarel is now known as ykarel|away | 17:01 | |
| weshay | panda|off, chem trying it again w/ new depends-on https://review.openstack.org/#/c/604706/ | 17:02 |
| *** trown is now known as trown|lunch | 17:08 | |
| *** vinaykns has quit IRC | 17:27 | |
| *** panda|off has quit IRC | 17:41 | |
| *** holser_ has quit IRC | 17:43 | |
| *** jfrancoa has quit IRC | 17:44 | |
| *** panda has joined #oooq | 17:45 | |
| *** panda is now known as panda|off | 17:47 | |
| *** ykarel has joined #oooq | 17:49 | |
| *** ykarel|away has quit IRC | 17:49 | |
| *** trown|lunch is now known as trown | 18:36 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, stable/pike: legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike, legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-pike, legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022-pike @ (3 more messages) | 18:47 |
| *** hubbot1 has quit IRC | 18:52 | |
| chem | weshay: hey, not sure, but I think include_roles is a tasks not a playbook parameter | 18:52 |
| chem | weshay: meaning this new patchset should fail | 18:52 |
| *** gouthamr has quit IRC | 18:52 | |
| *** dmellado has quit IRC | 18:53 | |
| chem | weshay: but I silent myself and see how it goes, maybe I didn't get it right :) | 18:53 |
| weshay | chem, the main.yml in repo-setup is wonky | 18:58 |
| chem | weshay: oki, I'm pushing yet another review I nearly sure we have to use include_role at the tasks level so in the yum upgrade should do | 19:00 |
| chem | weshay: unless you're 100% positive this should work as is | 19:01 |
| chem | weshay: or I'll wait tomorrow, I'm a bit wonky myself. | 19:01 |
| weshay | chem, if you just take the two tasks you'll be fine | 19:02 |
| weshay | you can not use main.yml | 19:02 |
| *** ykarel has quit IRC | 19:05 | |
| *** gouthamr has joined #oooq | 19:12 | |
| rfolco | weshay, can you pls quick look at https://review.openstack.org/#/c/594511 ? why gate job did not trigger | 19:12 |
| *** ssbarnea|bkp has quit IRC | 19:14 | |
| *** ssbarnea|bkp has joined #oooq | 19:14 | |
| chem | weshay: ack | 19:32 |
| *** trown is now known as trown|outtypewww | 20:03 | |
| *** gouthamr_ has joined #oooq | 20:05 | |
| *** dmellado has joined #oooq | 20:07 | |
| *** chem has quit IRC | 20:26 | |
| *** chem has joined #oooq | 20:26 | |
| *** gouthamr has quit IRC | 20:38 | |
| *** gouthamr_ is now known as gouthamr | 20:39 | |
| *** chem has quit IRC | 20:48 | |
| *** jtomasek has quit IRC | 21:00 | |
| *** hubbot1 has joined #oooq | 21:12 | |
| *** agopi has quit IRC | 21:52 | |
| *** agopi has joined #oooq | 22:32 | |
| *** agopi has quit IRC | 22:37 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates @ https://review.openstack.org/604298, stable/pike: legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike, legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-pike, legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset022-pike @ (3 more messages) | 22:48 |
| *** tosky has quit IRC | 23:11 | |
| *** agopi has joined #oooq | 23:22 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!