| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000 (1 more message) | 00:36 |
|---|---|---|
| *** links has joined #oooq | 01:17 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, (1 more message) | 02:36 |
| *** brault has quit IRC | 02:53 | |
| *** brault has joined #oooq | 02:57 | |
| *** vinaykns has quit IRC | 03:28 | |
| *** holser_ has joined #oooq | 03:35 | |
| *** saneax has quit IRC | 03:53 | |
| *** ykarel has joined #oooq | 04:03 | |
| *** holser_ has quit IRC | 04:26 | |
| *** holser_ has joined #oooq | 04:29 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.open | 04:36 |
| *** jaganathan has quit IRC | 04:42 | |
| *** jaganathan has joined #oooq | 04:42 | |
| *** honza has quit IRC | 04:47 | |
| *** honza has joined #oooq | 04:48 | |
| *** honza is now known as Guest56850 | 04:49 | |
| *** holser_ has quit IRC | 04:56 | |
| *** ratailor has joined #oooq | 05:19 | |
| *** skramaja has joined #oooq | 05:21 | |
| quiquell|off | panda|off: Who watches the watchmen | 05:33 |
| *** quiquell|off is now known as quiquell | 05:38 | |
| quiquell | sshnaidm|ruck: Good morning | 05:41 |
| *** ccamacho has joined #oooq | 05:44 | |
| *** gvrangan has joined #oooq | 05:52 | |
| quiquell | chkumar|rover, sshnaidm|ruck: Do we have a bug for this ? http://logs.openstack.org/61/584761/1/check/tripleo-ci-centos-7-scenario003-multinode-oooq-container/a7f88f8/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-07-23_10_31_22 | 05:56 |
| quiquell | marios: https://review.openstack.org/#/c/579587/ merged :-) | 05:58 |
| quiquell | sshnaidm|ruck, marios: Can workflow Folco's sprint change ? https://review.openstack.org/#/c/582385 | 06:02 |
| quiquell | just missing the +1w | 06:02 |
| chkumar|rover | quiquell: currently seeing | 06:04 |
| ykarel | chkumar|rover, is there bug for RDO phase1 issues? | 06:09 |
| ykarel | i can see tripleo jobs failing there | 06:09 |
| chkumar|rover | ykarel: I have just started looking | 06:09 |
| ykarel | chkumar|rover, ack btw it's failing for few days | 06:10 |
| ykarel | so asked | 06:10 |
| chkumar|rover | ykarel: specific job | 06:10 |
| ykarel | tripleo jobs in all releases in phase 1 | 06:10 |
| ykarel | https://ci.centos.org/view/rdo/view/promotion-pipeline/ | 06:10 |
| ykarel | so it's happeing after https://review.openstack.org/#/c/583736/, attempt to fix:- https://review.openstack.org/#/c/584644/ and https://review.openstack.org/#/c/584801/ | 06:13 |
| chkumar|rover | ykarel: failing bug | 06:20 |
| *** saneax has joined #oooq | 06:26 | |
| chkumar|rover | sshnaidm|ruck: ykarel https://bugs.launchpad.net/tripleo/+bug/1783262 | 06:27 |
| openstack | Launchpad bug 1783262 in tripleo "[All branches] tripleo quickstart promotion jobs are failing while pip install giving OSError: Command /home/jenkins/worksp...ld-images/bin/python - setuptools pip wheel failed with error code 1" [Critical,Triaged] | 06:27 |
| chkumar|rover | quiquell: about above failure does the issue coming only in one patch? | 06:27 |
| chkumar|rover | quiquell: I am looking into it why it is coming | 06:28 |
| quiquell | chkumar|rover: Could be, just asking if you have seen it around | 06:28 |
| quiquell | it could be legit | 06:28 |
| marios | o/ quiquell morning sorry having some laptop issues getting to it in bit sure | 06:30 |
| marios | quiquell: but why you -1 it? | 06:31 |
| marios | quiquell: ah waiting for another run | 06:31 |
| quiquell | marios: Show after asking, that the test review has fails, sure they are not related, but want to ensure | 06:33 |
| quiquell | Let's wait for the results | 06:34 |
| quiquell | marios: Do you need any help with the laptop issues ? | 06:34 |
| quiquell | marios: btw, your workspace check reproducer change is merged :-) | 06:35 |
| quiquell | marios: Let's celebrate btween ssbarnea and you we will get a good reproducer | 06:35 |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.open | 06:36 |
| marios | quiquell: thanks... not sure whatsup i have stack traces from gnome-shell ... i tried x-org instead of wayland and it seems to be ok now. problem only when i dock, so i can boot fine but once i dock for monitors it freezes. on xorg so far so good but it is a very recent/last night software updates | 06:37 |
| *** skramaja_ has joined #oooq | 06:37 | |
| marios | quiquell: woo \o/ on the workspace thing thanks | 06:37 |
| *** skramaja has quit IRC | 06:37 | |
| quiquell | marios: Fedora ? I will not update | 06:38 |
| marios | quiquell: man whats up here https://review.openstack.org/#/c/578081/ y zuul so reddid we have problems last night? | 06:38 |
| marios | quiquell: am on 28 | 06:38 |
| marios | quiquell: but it has been really great so far | 06:38 |
| *** bogdando has joined #oooq | 06:39 | |
| *** skramaja_ is now known as skramaja | 06:39 | |
| chkumar|rover | quiquell: http://logs.openstack.org/61/584761/1/check/tripleo-ci-centos-7-scenario003-multinode-oooq-container/a7f88f8/logs/subnode-2/var/log/containers/designate/designate-manage.log.txt.gz#_2018-07-23_10_29_35_867 | 06:40 |
| chkumar|rover | quiquell: I need some help why mysql lost connection | 06:40 |
| quiquell | marios: Done a recheck on that, reproducer cannot break any job, well... if the jinja template is not broken. | 06:40 |
| chkumar|rover | for designate | 06:40 |
| quiquell | chkumar|rover: Looks like ha hiccup, have send a recheck, let's wait for it. | 06:41 |
| chkumar|rover | quiquell: what is InnoDB: Online DDL in mysql log? | 06:44 |
| ykarel | chkumar|rover, second review is still in gate | 06:44 |
| ykarel | so please update the descripton | 06:44 |
| ykarel | second review should fix it | 06:44 |
| *** kopecmartin has joined #oooq | 06:46 | |
| *** jfrancoa has joined #oooq | 06:47 | |
| marios | thanks quiquell | 06:50 |
| *** quiquell is now known as quiquell|bbl | 06:54 | |
| *** skramaja_ has joined #oooq | 06:56 | |
| *** skramaja has quit IRC | 06:56 | |
| *** udesale has joined #oooq | 07:06 | |
| *** amoralej|off is now known as amoralej | 07:13 | |
| *** brault has quit IRC | 07:14 | |
| *** brault has joined #oooq | 07:16 | |
| *** tesseract has joined #oooq | 07:18 | |
| *** ykarel is now known as ykarel|lunch | 07:33 | |
| *** tosky has joined #oooq | 07:36 | |
| *** holser_ has joined #oooq | 07:36 | |
| *** holser_ has quit IRC | 07:46 | |
| *** holser_ has joined #oooq | 07:49 | |
| quiquell|bbl | chkumar|rover: Need to do some testing with the rrcockpit, will be out for a few | 07:50 |
| quiquell|bbl | sshnaidm|ruck: ^ | 07:50 |
| *** quiquell|bbl is now known as quiquell | 07:52 | |
| chkumar|rover | quiquell: sure | 07:52 |
| *** holser_ has quit IRC | 07:52 | |
| *** holser_ has joined #oooq | 07:55 | |
| *** tcw1 has joined #oooq | 08:02 | |
| *** tcw has quit IRC | 08:03 | |
| *** gkadam has joined #oooq | 08:13 | |
| *** ykarel|lunch is now known as ykarel | 08:21 | |
| sshnaidm|ruck | chkumar|rover, https://review.openstack.org/#/c/584801/ should solve ci.centos problem | 08:23 |
| *** holser_ has quit IRC | 08:27 | |
| *** holser_ has joined #oooq | 08:30 | |
| *** gvrangan has quit IRC | 08:32 | |
| *** dtantsur|afk is now known as dtantsur | 08:33 | |
| *** jfrancoa has quit IRC | 08:33 | |
| *** florianf has joined #oooq | 08:33 | |
| *** dtrainor has quit IRC | 08:34 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.open | 08:36 |
| *** florianf has quit IRC | 08:39 | |
| sshnaidm|ruck | quiquell, commented on https://review.rdoproject.org/r/#/c/14632/ | 08:46 |
| *** sshnaidm|ruck is now known as sshnaidm|afk | 08:46 | |
| quiquell | sshnaidm|ruck: Ok, implementing the fixes, rrcockpit will be down today | 08:46 |
| quiquell | sshnaidm|afk: thanks | 08:46 |
| chkumar|rover | sshnaidm|afk: thanks :-) | 08:50 |
| *** jfrancoa has joined #oooq | 08:53 | |
| *** gvrangan has joined #oooq | 08:53 | |
| *** jfrancoa has quit IRC | 08:57 | |
| chkumar|rover | sshnaidm|afk: quiquell post failure has started coming, rdo sites are down | 09:01 |
| *** hubbot has quit IRC | 09:02 | |
| quiquell | chkumar|rover: Yep just found it... :-/ what a week | 09:03 |
| *** dmellado has quit IRC | 09:03 | |
| quiquell | RDO is down | 09:04 |
| chkumar|rover | rdo cloud is down | 09:04 |
| *** fhubik has joined #oooq | 09:05 | |
| *** udesale has quit IRC | 09:06 | |
| fhubik | Hi, we have issues with 1 failing upstream gate blocking us, https://review.openstack.org/#/c/583552/, tripleo-ci-centos-7-scenario004-multinode-oooq-container gate is failing for reasons not clear to us? Can anyone give us a directon? | 09:09 |
| chkumar|rover | fhubik: http://logs.openstack.org/52/583552/10/check/tripleo-ci-centos-7-scenario004-multinode-oooq-container/e60e7fa/logs/subnode-2/var/log/containers/nova/nova-compute.log.txt.gz#_2018-07-24_01_34_16_272 | 09:12 |
| *** jfrancoa has joined #oooq | 09:15 | |
| chkumar|rover | fhubik: it appears to be a timed out issue | 09:15 |
| quiquell | marios: Can you help me debug, this non voting upgrade job ? http://logs.openstack.org/61/584761/1/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/745f895/ | 09:15 |
| marios | quiquell: looking | 09:16 |
| quiquell | marios: fatal: [undercloud]: FAILED! => {"msg": "'controllers' is undefined"} | 09:16 |
| fhubik | chkumar|rover: thx, do you think is it infra or product issue? Anyway, I added you to https://trello.com/c/zqTCZplK/658-cixbz1601382ospphase1osp14ceph-ansibleunreachable-overcloud-nodes-during-run-nodes-uuid-task | 09:16 |
| quiquell | marios: ccmacho already told me that doens't looks related, but want to double check | 09:16 |
| chkumar|rover | quiquell: the same error is seen by Tengu | 09:16 |
| quiquell | chkumar|rover: Humm... ok so it's not related | 09:16 |
| fhubik | chkumar|rover: is there way to retrigger this gate? Recheck doesn't seem to work there for ceph-dfg folks... | 09:16 |
| quiquell | chkumar|rover: Do we have a bug for it ? | 09:17 |
| chkumar|rover | fhubik: 1 hour back recheck is added | 09:17 |
| *** ratailor has quit IRC | 09:18 | |
| quiquell | marios: If you see it's ok, then we can workflow this https://review.openstack.org/#/c/582385/ and continue with the rest of patches | 09:18 |
| chkumar|rover | quiquell: nope | 09:19 |
| chkumar|rover | fhubik: I am not sure this is also related http://logs.openstack.org/52/583552/10/check/tripleo-ci-centos-7-scenario004-multinode-oooq-container/e60e7fa/logs/subnode-2/var/log/extra/docker/containers/manila_scheduler/log/manila/manila-share.log.txt.gz#_2018-07-24_01_28_39_456 | 09:20 |
| *** ratailor has joined #oooq | 09:20 | |
| fhubik | chkumar|rover: hmm, so it could be product (ceph) issue anyway | 09:21 |
| chkumar|rover | fhubik: not sure need an ceph expert | 09:21 |
| Tengu | quiquell: I didn't open a big, as I wasn't sure if it was a temporary error or something more... err. "present". | 09:22 |
| chkumar|rover | Tengu: is it the same run? or we hitted again in another run? | 09:22 |
| Tengu | quiquell: and I didn't find interesting information in the logs, maybe (probably) I overlooked something. | 09:22 |
| marios | quiquell: so i am trying to find the error in http://logs.openstack.org/61/584761/1/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/745f895/logs/undercloud/home/zuul/ | 09:22 |
| Tengu | chkumar|rover: http://logs.openstack.org/14/584314/6/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/41f570c/job-output.txt.gz#_2018-07-24_07_07_01_701729 another one from quiquell | 09:23 |
| marios | quiquell: nm i see it in console now | 09:23 |
| fhubik | chkumar|rover: ok, thanks, I'll update the escalation card and ask Ceph people to comment | 09:23 |
| quiquell | marios: Where ? | 09:23 |
| marios | http://logs.openstack.org/61/584761/1/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/745f895/job-output.txt.gz#_2018-07-24_07_38_33_243002 | 09:23 |
| marios | quiquell: ^ | 09:23 |
| fhubik | chkumar|rover: (with your findings) | 09:24 |
| *** panda|off is now known as panda | 09:25 | |
| *** udesale has joined #oooq | 09:28 | |
| quiquell | chkumar|rover: So you have already see at upstream the "Controllers undefined error ? " | 09:28 |
| quiquell | marios: Looks like at posterior commit the job is working fine | 09:29 |
| quiquell | So looks unrelated | 09:29 |
| quiquell | let's workflow the review | 09:29 |
| quiquell | marios, panda: We can workflow https://review.openstack.org/#/c/582385/ | 09:30 |
| *** dtrainor has joined #oooq | 09:31 | |
| marios | quiquell: ack lemme poke a bit more at the logs and will do | 09:32 |
| chkumar|rover | quiquell: I am opening a bug, I thought it is coming only in one run | 09:33 |
| quiquell | marios: Thanks so much | 09:33 |
| chkumar|rover | quiquell: I checked scenario000 passed /failed jobs, both of them case that part was skipped | 09:33 |
| chkumar|rover | quiquell: since we have two logs, now i can open a bug | 09:33 |
| quiquell | chkumar|rover: Cool thanks | 09:34 |
| *** skramaja_ is now known as skramaja | 09:34 | |
| *** dsneddon has quit IRC | 09:36 | |
| quiquell | marios, panda: also 'nodes' one looks ok to workflow https://review.openstack.org/#/c/582386/ | 09:38 |
| chkumar|rover | quiquell: need some help here 2018-07-24 09:08:34 | /home/zuul/tempest-setup.sh: line 142: [: ==: unary operator expected | 09:38 |
| chkumar|rover | quiquell: https://review.openstack.org/#/c/584771/8/roles/validate-tempest/templates/run-tempest.sh.j2@63 | 09:39 |
| Tengu | chkumar|rover: empty var on one side I guess | 09:39 |
| quiquell | chkumar|rover: Add some " around that command | 09:40 |
| chkumar|rover | Tengu: docker inspect will return nothing if user is not there | 09:40 |
| Tengu | so you might prefix with "x" and check for "xtempest" | 09:40 |
| quiquell | ^ that's it | 09:40 |
| Tengu | :) | 09:41 |
| marios | quiquell: panda beat me to it but added a note just now anyway | 09:42 |
| quiquell | marios: No problem, chkumar|rover Is openning a bug for it I think | 09:45 |
| quiquell | marios, panda: The test review for ÂÂ'periodic' anddryrun' working fine but zuul on the change has fail | 09:45 |
| quiquell | let's recheck it to be sure | 09:45 |
| quiquell | It was some keystone connectivity issue | 09:46 |
| *** jaosorior has quit IRC | 09:47 | |
| quiquell | btw this review is testing the whole "enchilada" and is working fine | 09:47 |
| quiquell | https://review.openstack.org/#/c/583874/ | 09:47 |
| quiquell | I think we can start workflowing everything | 09:48 |
| panda | is scenario008 broken again ? | 09:49 |
| quiquell | panda: Has being broken all the sprint, was fixed at some point ? | 09:49 |
| panda | no | 09:50 |
| *** ratailor has quit IRC | 09:55 | |
| chkumar|rover | quiquell: Tengu https://bugs.launchpad.net/tripleo/+bug/1783303 | 09:56 |
| openstack | Launchpad bug 1783303 in tripleo "[master] scenario000 multinode upgrade job is failing at tripleo-upgrade include task giving controllers' is undefined"}" [Critical,Triaged] | 09:56 |
| *** ratailor has joined #oooq | 09:56 | |
| chkumar|rover | quiquell: Tengu https://github.com/openstack/tripleo-upgrade/blob/4572055ffb52b9fe6ff3567afe709a0a68cf1d6b/tasks/common/controller_post_scripts.yml#L5 | 09:56 |
| chkumar|rover | i think here it is failing | 09:56 |
| Tengu | chkumar|rover: that's the task registering controllers - so it exists, but might be empty... ? | 09:57 |
| rook | are the logs for rdoproject timingout for everyone? | 10:01 |
| rook | https://review.rdoproject.org/zuul/stream.html?uuid=d11218fad81e449b9c21078cd4872071&logfile=console.log | 10:01 |
| quiquell | rook: RDO cloud is down, https://semaphore.cloud.upshift.engineering.redhat.com/ | 10:01 |
| *** zoli is now known as zoli|lunch | 10:01 | |
| Tengu | rook: there's an announce on outage for the rdo cloud. | 10:02 |
| Tengu | for about 1 our. | 10:03 |
| rook | ack thanks | 10:07 |
| rook | Tengu: where can i get those? | 10:07 |
| * rook will subscribe to the outage list | 10:08 | |
| Tengu | rook: exactly :) | 10:08 |
| Tengu | rook: there's a lot of "noise", but sometime it's valuable :) | 10:09 |
| *** matbu has quit IRC | 10:09 | |
| tbarron | chkumar|rover: fhubik - w.r.t. http://logs.openstack.org/52/583552/10/check/tripleo-ci-centos-7-scenario004-multinode-oooq-container/e60e7fa/logs/subnode-2/var/log/extra/docker/containers/manila_scheduler/log/manila/manila-share.log.txt.gz#_2018-07-24_01_28_39_456 | 10:10 |
| tbarron | chkumar|rover: fhubik - this should fix it ^^ https://review.openstack.org/#/c/584949/ | 10:11 |
| *** jaosorior has joined #oooq | 10:22 | |
| *** dmellado has joined #oooq | 10:24 | |
| rook | so when rdocloud restarts how does one view the console log? it seems to be empty for me | 10:26 |
| marios | https://review.openstack.org/#/c/583195/8/zuul.d/base.yaml@53 panda this what you had in mind? | 10:28 |
| panda | rook: what's the change you'r looking ? | 10:28 |
| *** hubbot has joined #oooq | 10:31 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master @ https://review.open | 10:36 |
| panda | marios: yes, but you're missing one bit | 10:36 |
| panda | marios: remove the tripleo.sh invocation from non templated toci_gate_test.sh | 10:37 |
| marios | panda: i thought i did | 10:37 |
| marios | panda: https://review.openstack.org/#/c/583195/8/toci_gate_test.sh | 10:37 |
| marios | panda: ^ | 10:37 |
| panda | marios: sometimes gertty is difficult to read :) | 10:39 |
| marios | panda: ha :) np thanks for checking so quickly! | 10:39 |
| * marios brb | 10:39 | |
| panda | marios: we need to find a way to test the old workflow | 10:41 |
| rook | panda: just took a while, it is back now | 10:42 |
| * rook is not patient enough | 10:42 | |
| *** quiquell has quit IRC | 10:48 | |
| Tengu | rook: do like me: eat an icecream :D | 10:50 |
| panda | Tengu: if I eat an icecream every time I'm waiting for a job to finish ... | 10:50 |
| Tengu | panda: well. good point. | 10:51 |
| *** quiquell has joined #oooq | 10:55 | |
| quiquell | panda: Have started to create an ansible module for emit_releases_file.py https://review.openstack.org/585200 | 10:56 |
| quiquell | panda: So we can cleanup this part from toci_gate_test | 10:57 |
| panda | quiquell is already on Sprint 18 | 11:01 |
| agopi | more progress rook https://logs.rdoproject.org/17/583717/16/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053-master/d11218f/job-output.txt.gz | 11:01 |
| quiquell | panda: 17 you mean | 11:02 |
| panda | quiquell: 18 | 11:03 |
| sshnaidm|afk | chkumar|rover, fyi https://bugs.launchpad.net/tripleo/+bug/1783310 | 11:04 |
| openstack | Launchpad bug 1783310 in tripleo "Tripleo periodic jobs fail because /etc/nodepool/id_rsa permission denied" [Critical,Triaged] - Assigned to chandan kumar (chkumar246) | 11:04 |
| quiquell | panda: where ? | 11:05 |
| quiquell | panda: So it's ok that I continue with the review ? | 11:05 |
| panda | quiquell: let's have a chat | 11:08 |
| quiquell | panda: ok | 11:08 |
| panda | ssbarnea: what's with the skip_ansible_lint tags in https://review.openstack.org/582163. Where are they handled ? | 11:08 |
| quiquell | panda: https://bluejeans.com/7891065232 | 11:09 |
| panda | ssbarnea: ah, mmhh, | 11:09 |
| ssbarnea | panda: yep, it disables the linter error on shell task | 11:09 |
| panda | quiquell: in ~1h ok ? | 11:09 |
| panda | quiquell: need to run for lunch | 11:10 |
| *** panda is now known as panda|lunch | 11:10 | |
| quiquell | panda|lunch: no problem | 11:10 |
| ssbarnea | because same task can be written using ansible module (or at least this is what linter believes) | 11:10 |
| ssbarnea | but sadly in LOTS of case is wrong because the module is either broken or doesn't have the same behavior :D | 11:11 |
| *** fhubik is now known as fhubik|brb | 11:22 | |
| *** udesale has quit IRC | 11:28 | |
| ykarel | chkumar|rover, --clean issue is fixed, now https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-master-current-tripleo-delorean-minimal-389/undercloud/home/stack/overcloud_prep_images.log.gz | 11:28 |
| *** atoth has joined #oooq | 11:29 | |
| ykarel | seems like it's failing since few days | 11:29 |
| *** sshnaidm|afk is now known as sshnaidm|ruck | 11:29 | |
| chkumar|rover | ykarel: thanks :-) | 11:29 |
| *** fhubik|brb is now known as fhubik | 11:30 | |
| *** zoli|lunch is now known as zoli | 11:31 | |
| sshnaidm|ruck | panda|lunch, marios, rfolco, quiquell do you know maybe if some patch from sprint could break it? https://bugs.launchpad.net/tripleo/+bug/1783310 | 11:34 |
| openstack | Launchpad bug 1783310 in tripleo "Tripleo periodic jobs fail because /etc/nodepool/id_rsa permission denied" [Critical,Triaged] - Assigned to chandan kumar (chkumar246) | 11:34 |
| quiquell | sshnaidm|ruck: Going to check | 11:35 |
| *** agopi is now known as agopi|brb | 11:36 | |
| quiquell | sshnaidm|ruck: It's working fine at non periodic https://logs.rdoproject.org/84/582384/9/openstack-check/legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master/3ab7cab/job-output.txt.gz#_2018-07-19_19_50_04_574631 | 11:38 |
| sshnaidm|ruck | quiquell, yeah, weird | 11:39 |
| quiquell | At periodics we do something with releases file, to select the correct one | 11:40 |
| quiquell | sshnaidm|ruck: There is no success periodic after merge stuff from our sprint ? | 11:41 |
| *** agopi|brb has quit IRC | 11:41 | |
| sshnaidm|ruck | quiquell, I think it's only first time it happens.. | 11:43 |
| chkumar|rover | panda|lunch: sshnaidm|ruck quiquell please have a look https://review.openstack.org/#/c/584771/ undercloud tempest passed | 11:43 |
| chkumar|rover | sshnaidm|ruck: I am adding the check whether it exist or not then check for permission | 11:43 |
| sshnaidm|ruck | maybe it's because of rdo cloud outahge, who knows.. | 11:43 |
| sshnaidm|ruck | chkumar|rover, https://review.openstack.org/#/c/585321/ | 11:44 |
| quiquell | sshnaidm|ruck, chkumar|rover: I see some success here https://softwarefactory-project.io/zuul/t/rdoproject.org/builds.html?job_name=legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset018-queens | 11:44 |
| sshnaidm|ruck | quiquell, yes, it doesn't include current run | 11:45 |
| *** fhubik is now known as fhubik|brb | 11:46 | |
| *** amoralej is now known as amoralej|lunch | 11:48 | |
| quiquell | sshnaidm|ruck: Can I retrieve the tripleo-ci commit id from https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset018-queens/9ada9f5/ ? | 11:51 |
| quiquell | To check if the changes where already there ? | 11:51 |
| quiquell | I can look at the logs too | 11:51 |
| sshnaidm|ruck | quiquell, you have patch and patchset number there | 11:52 |
| sshnaidm|ruck | quiquell, oops, it's periodic | 11:52 |
| quiquell | Humm it's running devstack things... | 11:52 |
| quiquell | sshnaidm|ruck: But we run periodic over our new stuff too, isn't it ¿ | 11:53 |
| sshnaidm|ruck | quiquell, it runs on rdo cloud, w/o run-v3.yml | 11:53 |
| quiquell | Ahhh | 11:53 |
| quiquell | ok | 11:53 |
| sshnaidm|ruck | quiquell, we shouldn't | 11:53 |
| quiquell | We haven't merge anything afecting old workflow yet | 11:53 |
| quiquell | Only the extra vars at the jobs | 11:54 |
| chkumar|rover | sshnaidm|ruck: quiquell https://review.rdoproject.org/r/#/c/14971 after this it statrted failing | 11:54 |
| quiquell | sshnaidm|ruck: Where is the patchset number ? | 11:54 |
| quiquell | chkumar|rover: Feels like the real candidate | 11:55 |
| sshnaidm|ruck | chkumar|rover, yeah, seems that | 11:56 |
| *** fhubik|brb is now known as fhubik | 12:07 | |
| *** trown|outtypewww is now known as trown | 12:11 | |
| bogdando | which fs is it, folks? in https://bugs.launchpad.net/tripleo/+bug/1782267 ? | 12:11 |
| openstack | Launchpad bug 1782267 in tripleo "Stderr: u'Set Chassis Power Control to Up/On failed: Command not supported in present state\n': ProcessExecutionError: Unexpected error while running command." [Critical,Triaged] | 12:12 |
| bogdando | how can we alter the netconfig used for that job mentioned in https://bugs.launchpad.net/tripleo/+bug/1782267/comments/7 > | 12:12 |
| bogdando | ? | 12:12 |
| bogdando | in quickstart featuresets I mean, or something | 12:13 |
| sshnaidm|ruck | chkumar|rover, so, you prepare a patch? | 12:13 |
| sshnaidm|ruck | bogdando, this happens in libvirt only afaik | 12:14 |
| bogdando | the thing is we need to go with ci/common/net-config-simple-bridge.yaml for multi-node only jobs, w/o introscpection | 12:14 |
| *** hubbot has quit IRC | 12:14 | |
| bogdando | for the latter we have to use t-h-t defaults | 12:14 |
| bogdando | and the former is only needed to do VXLAN magic for CI cases, IIRC | 12:15 |
| bogdando | so I'm not sure which is the right place to fix that job's defaults for netconfig | 12:15 |
| sshnaidm|ruck | bogdando, neither me. ML? | 12:17 |
| *** hubbot has joined #oooq | 12:18 | |
| bogdando | I hoped to get answers here for that jenkins-tripleo-quickstart-promote-master-current-tripleo-delorean-minimal-379 job | 12:18 |
| *** ratailor has quit IRC | 12:19 | |
| quiquell | sshnaidm|ruck: internal ip address can only being use in the same tenant ? | 12:24 |
| chkumar|rover | sshnaidm|ruck: https://review.openstack.org/#/c/585336/ | 12:27 |
| sshnaidm|ruck | quiquell, not sure I understand | 12:28 |
| sshnaidm|ruck | chkumar|rover, no, I mean patch that fixes it | 12:29 |
| sshnaidm|ruck | chkumar|rover, look at #sf-dfg discussion | 12:29 |
| quiquell | chkumar|rover, sshnaidm|ruck: Any issue like "Failed connect to mirror.us-west-1.packethost.openstack.org:8080; No route to host\\"" ? | 12:29 |
| quiquell | http://logs.openstack.org/87/582387/11/check/tripleo-ci-centos-7-containers-multinode/861e47b/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz | 12:30 |
| sshnaidm|ruck | quiquell, hmm.. it's really unavailable | 12:31 |
| quiquell | sshnaidm|ruck: rrcockpit fixed, try to run it at your tenant now | 12:35 |
| sshnaidm|ruck | quiquell, ok | 12:35 |
| quiquell | sshnaidm|ruck: btw to do modifications you can do docker-compose up directly at your fedora | 12:35 |
| sshnaidm|ruck | quiquell, sure | 12:35 |
| *** panda|lunch is now known as panda | 12:36 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ (1 more message) | 12:36 |
| chkumar|rover | sshnaidm|ruck: https://review.rdoproject.org/r/#/c/14987/ | 12:36 |
| sshnaidm|ruck | chkumar|rover, commented, the problem is that files are created as root | 12:38 |
| Tengu | chkumar|rover: hm, 0755 already allows user to read | 12:38 |
| sshnaidm|ruck | chkumar|rover, need to create the dir with root only, all the files should be created as regular user | 12:38 |
| *** rlandy has joined #oooq | 12:39 | |
| panda | quiquell: have time to chat now ? | 12:40 |
| rlandy | marios: hello | 12:40 |
| quiquell | panda: sure | 12:41 |
| quiquell | sshnaidm|ruck, chkumar|rover: new rrcockpit url http://38.145.34.131/d/pVVquHFik/cockpit?orgId=1 | 12:42 |
| marios | rlandy: o/ | 12:43 |
| rlandy | marios: hey - just wanted to touch base re: bootstrap | 12:44 |
| marios | rlandy: ack thanks for the test review | 12:44 |
| marios | rlandy: i updated https://review.openstack.org/#/c/583195/ v8 today, and did a recheck on the test reveiw | 12:44 |
| sshnaidm|ruck | quiquell, oh, dashboard is back 👏 | 12:44 |
| marios | rlandy: but, sshnaidm|ruck -1 it so i am updating now for comments left a note on the review just now | 12:44 |
| rlandy | marios: yep, you add files to the review to remove the bootstrap references | 12:44 |
| rlandy | I am looking at all that now | 12:45 |
| marios | rlandy: thanks, gonna update it and hopefully get a good ci run | 12:45 |
| chkumar|rover | sshnaidm|ruck: done | 12:45 |
| rlandy | marios: because it's the last day, just want to check in if we need other testing. | 12:45 |
| marios | rlandy: but open question re https://review.openstack.org/#/c/583195/8/zuul.d/base.yaml@55 sshnaidm|ruck what should we do there | 12:45 |
| marios | rlandy: i think we're ok as long as we're not breaking all the things so far seems ok :) | 12:45 |
| marios | rlandy: i think a big part of this is removing what is actually redundant code (the bootstrap-subnodes thing, with the exception of the loop device) | 12:46 |
| marios | rlandy: not sure how to handle the rdo cloud thing that sshnaidm|ruck commented on though | 12:46 |
| rlandy | marios: the rdocloud problem is that we still have a bunch legacy mess there | 12:46 |
| rlandy | and it's generated | 12:46 |
| marios | rlandy: sshnaidm|ruck one idea is to have it in the run.yaml directly | 12:46 |
| marios | for the legacy case | 12:46 |
| rlandy | and duplicated everywhere | 12:47 |
| rlandy | but run.yaml is diff for each job | 12:47 |
| * rlandy gets | 12:47 | |
| marios | rlandy: i see | 12:47 |
| marios | rlandy: well, not really but i see it is generated so we can't just add to it | 12:47 |
| rlandy | https://github.com/rdo-infra/rdo-jobs/blob/master/playbooks/legacy/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset018-queens/run.yaml | 12:48 |
| rlandy | example | 12:48 |
| * rlandy reads the remarks again | 12:48 | |
| sshnaidm|ruck | marios, let me try to add it to rdo cloud.. | 12:48 |
| marios | sshnaidm|ruck: rlandy i meant here https://github.com/openstack-infra/tripleo-ci/blob/master/playbooks/tripleo-ci/run.yaml#L27 | 12:48 |
| marios | sshnaidm|ruck: ok will you post a review on top? | 12:48 |
| marios | sshnaidm|ruck: i am updating it now for your other comments | 12:49 |
| sshnaidm|ruck | marios, well, when this cloud will start work | 12:49 |
| sshnaidm|ruck | marios, not sure when it will be.. | 12:49 |
| marios | sshnaidm|ruck: ok fine i just meant so we don't conflict but i will post myupdate in a sec anyway thanks | 12:49 |
| chkumar|rover | brb | 12:49 |
| rlandy | sshnaidm|ruck, well since bootstrap is only involved with ceph now, maybe not needed everywhere | 12:50 |
| rlandy | sshnaidm|ruck: it is my understanding that this gets overriden in legacy jobs https://github.com/openstack-infra/tripleo-ci/blob/master/playbooks/tripleo-ci/run.yaml#L27 | 12:51 |
| sshnaidm|ruck | rlandy, what exactly overriden? | 12:55 |
| sshnaidm|ruck | rlandy, I hope it won't, but I can't test it really with jobs when rdo cloud is down or close to down.. :( | 12:55 |
| rlandy | sshnaidm|ruck:run.yaml | 12:55 |
| sshnaidm|ruck | rlandy, yeah, it has its own run.yaml | 12:55 |
| rlandy | maybe I ma talking at cross-purposes | 12:55 |
| rlandy | but each job has its own run | 12:55 |
| sshnaidm|ruck | rlandy, yep | 12:56 |
| rlandy | so hence your pre? suggestion | 12:56 |
| rlandy | the rdocloud situation is so confusing | 12:57 |
| *** fhubik is now known as fhubik|brb | 12:58 | |
| *** skramaja has quit IRC | 12:59 | |
| sshnaidm|ruck | chkumar|rover, commente | 13:02 |
| sshnaidm|ruck | d | 13:02 |
| sshnaidm|ruck | rlandy, we can add this pre.yml to all multinode jobs in rdo cloud | 13:03 |
| sshnaidm|ruck | otherwise periodic multinode jobs that use ceph device will fail.. | 13:03 |
| sshnaidm|ruck | actually only ceph jobs will be fine too | 13:03 |
| rlandy | sshnaidm|ruck: ack | 13:07 |
| quiquell | rfolco: We are near to merge the "enchilada" | 13:08 |
| rlandy | sshnaidm|ruck: all - or just those requiring ceph? | 13:08 |
| quiquell | rfolco: The test review of it is working https://review.openstack.org/#/c/583874/ | 13:08 |
| *** gvrangan has quit IRC | 13:11 | |
| sshnaidm|ruck | rlandy, good question, if we use it for ceph only, why is it applied to all upstream jobs? | 13:11 |
| sshnaidm|ruck | marios, panda ^^ | 13:11 |
| panda | lol | 13:12 |
| marios | sshnaidm|ruck: rlandy well it was previously doing packages and repo setup too, but indeed all that is left now is ceph loop device so we may need to rescope it :/ | 13:12 |
| panda | legacy can really bring you in all sort of places before you realize most of it is unnecessary | 13:13 |
| marios | panda: which bit :) the fact that this keeps changing :) | 13:13 |
| sshnaidm|ruck | generally it won't hurt to have unused loopback device, but still.. | 13:13 |
| marios | sshnaidm|ruck: yeah but how can we scope it. i mean do we move it to the scenario then? | 13:14 |
| marios | sshnaidm|ruck: scope it so it is ceph only i mean | 13:14 |
| marios | rlandy: panda ^^^ | 13:14 |
| sshnaidm|ruck | marios, by scenario you mean..? | 13:14 |
| marios | sshnaidm|ruck: the list of services | 13:14 |
| panda | marios: no, lol about aving a goal at the start of the sprint, and thnking to bring it to the end without asking why too much | 13:14 |
| marios | sshnaidm|ruck: so ceph would be there | 13:14 |
| rlandy | I don't mind it being everywhere | 13:14 |
| rlandy | but if we are doing specific jobs, it's unnecessary duplication | 13:15 |
| *** agopi|brb has joined #oooq | 13:15 | |
| panda | why is this not done by ceph role in tripleO ? | 13:15 |
| *** agopi|brb is now known as agopi | 13:15 | |
| marios | sshnaidm|ruck: like https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/scenario000-multinode-containers.yaml | 13:15 |
| rlandy | All multinode jobs is probably safer | 13:16 |
| marios | panda: well discovering we didn't need th package setup and repos was part of the journey :) | 13:16 |
| rlandy | but there will be some unnecessary dup | 13:16 |
| marios | panda: and we are still on that journey apparently but hopefully the end is near | 13:16 |
| panda | marios: absolutely agree | 13:16 |
| Tengu | bandini: tiny reminder for the two reviews in order to avoid keepalived errors with haproxy :) | 13:16 |
| panda | Tengu: reviews involving CI ? | 13:17 |
| Tengu | panda: hmmm. was intended on #tripleo, sorry -.-' | 13:17 |
| Tengu | messed up my tags. | 13:17 |
| bandini | Tengu: aye, not forgotten. just in calls all day and swamped in bugs | 13:17 |
| Tengu | bandini: :] | 13:18 |
| Tengu | bandini: need some help maybe? | 13:18 |
| panda | "swamped in bugs" is not at all a nice image | 13:18 |
| sshnaidm|ruck | marios, ideally it would be configured in featureset whether to use ceph or not. But I think for now it's safer to have it for all. I can' think now about good way to decide in pre.yml if we need ceph device | 13:19 |
| Tengu | panda: guess it explains the situation pretty well :] | 13:19 |
| rlandy | sshnaidm|ruck: marios: ^^ I'd agree .. put the same comment above | 13:19 |
| sshnaidm|ruck | marios, it was before so, so we don't change here a lot | 13:19 |
| rlandy | about being safer in all | 13:19 |
| *** agopi is now known as agopi|brb | 13:25 | |
| rfolco | quiquell, can we sync on bj ? | 13:28 |
| quiquell | rfolco: Sure, have a few | 13:28 |
| *** tcw1 has quit IRC | 13:29 | |
| *** tcw has joined #oooq | 13:29 | |
| *** agopi|brb has quit IRC | 13:29 | |
| *** agopi|brb has joined #oooq | 13:31 | |
| panda | ssbarnea: https://review.openstack.org/582963. Patchsets diff is not helping, what did you change from ps 3 to ps 4 ? | 13:32 |
| rfolco | quiquell, https://review.openstack.org/#/c/584465/ | 13:32 |
| rfolco | https://review.openstack.org/#/c/584508/ | 13:33 |
| panda | ssbarnea: ah, the cherry pick ? | 13:33 |
| *** tcw1 has joined #oooq | 13:34 | |
| *** tcw has quit IRC | 13:34 | |
| ssbarnea | panda: yep, Wes asked to use git only instead of git-review tool, to avoid adding a new requirements. | 13:34 |
| panda | lol again, I had an entire tool ready once that synced two repositories in their topological order | 13:34 |
| *** tcw1 is now known as tcw | 13:35 | |
| panda | ssbarnea: ok | 13:35 |
| *** fhubik|brb is now known as fhubik | 13:36 | |
| *** fhubik is now known as fhubik|mtg | 13:36 | |
| *** amoralej|lunch is now known as amoralej | 13:36 | |
| panda | ssbarnea: Have you put some thouhts for when ZUUL_CHANGES will be obsoleted ? like, next week ? :) | 13:36 |
| chkumar|rover | the best way to find sshnaidm|ruck on gerrit just type ssh and you will find it :-) | 13:38 |
| ssbarnea | i don't know about that move, one way or another we will still need some form of it for the reproducer. | 13:38 |
| ssbarnea | reproducer needs to know how to clone the right versions of these two repos. | 13:39 |
| rlandy | rook: agopi|brb: how's it going? | 13:39 |
| rlandy | need some help to get by the current errors or doing ok? | 13:40 |
| *** vinaykns has joined #oooq | 13:42 | |
| *** ccamacho1 has joined #oooq | 13:43 | |
| *** ccamacho has quit IRC | 13:43 | |
| panda | ssbarnea: the reproducer will probably work on the zuul inventory file e.g. http://logs.openstack.org/85/582885/25/check/tripleo-ci-centos-7-containers-multinode/42bad00/zuul-info/inventory.yaml the ZUUL_CHANGES is being replaced by zuul.items list | 13:45 |
| sshnaidm|ruck | marios, another question, are you sure you can use multiple pre-run playbooks? https://review.openstack.org/#/c/583195/9/zuul.d/base.yaml | 13:46 |
| panda | sshnaidm|ruck: yes, we can, let me find the zuul documentation | 13:46 |
| ssbarnea | i think we are missing a critical point here: reproducer was supposed to be downloaded alone and run locally by the user, without the need for any other files. | 13:46 |
| quiquell | panda: Thinking about removing QUICKSTART_RELEASE can we run already emit_releases_file.py at all featuresets ? | 13:47 |
| quiquell | panda: I can set a review to test if we break something | 13:47 |
| ssbarnea | if you tell me that now i need to download another file is like... killing the usability of the reproducer. | 13:47 |
| panda | sshnaidm|ruck: https://zuul-ci.org/docs/zuul/user/config.html#attr-job.pre-run | 13:47 |
| sshnaidm|ruck | panda, great | 13:48 |
| panda | ssbarnea: it needs to be discussed. we can make the script download the file, or add it in the file | 13:48 |
| quiquell | panda, sshnaidm|ruck: we can download the file from a command inside the reproducer | 13:48 |
| quiquell | panda, ssbarnea: ^ | 13:49 |
| panda | ssbarnea: quiquell food for design session :) | 13:49 |
| panda | ssbarnea: did you address the comment from wes, that the regexp was not working for him ? | 13:49 |
| quiquell | panda, ssbarnea: All the stuff needed from logs server can be downloaded, from the same reproducer script | 13:49 |
| panda | quiquell: maybe we can just gather only the variables we need and put them in bash env variables | 13:50 |
| ssbarnea | yep, i was thinking about the same thing, mostly. with few variants: a) include a curl in reproucer to get extra files from build, b) inject a recreated ZUUL_CHANGES into reproducer in jinja2. | 13:50 |
| sshnaidm|ruck | marios, are there any CI jobs in your patch..? I can't see anything running there.. | 13:51 |
| ssbarnea | i like more b) because it allows me to easily hack the file, like removing/adding new change on the list. If I get a curl inside it, it would beugly as I would have to first download the file path it, and disable the curl to avoid override. | 13:51 |
| ssbarnea | i like the self-contained design of repro', lets try to keep it, if possible. | 13:51 |
| marios | sshnaidm|ruck: yeah rlandy posted https://review.openstack.org/#/c/585032 to check ceph jobs | 13:51 |
| marios | sshnaidm|ruck: is that what you meant? | 13:52 |
| sshnaidm|ruck | marios, yeah, cool | 13:53 |
| quiquell | ssbarnea, panda: agree on self containing repro, we can embedd stuff there, not much problem | 13:53 |
| sshnaidm|ruck | marios, should run on your patch also, seems like CI is slow.. | 13:53 |
| *** bogdando has quit IRC | 13:54 | |
| rlandy | sshnaidm|ruck: what's going on with rdocloud? | 13:54 |
| *** quiquell has quit IRC | 13:54 | |
| rlandy | <kforde> node is back and nearly back to normal | 13:54 |
| panda | ssbarnea: the main problem with ZUUL_CHANGE is that gives less informations than the zuul.items. Gerrit hostname for example. So we'll be missing some features if we continue to use that format. | 13:55 |
| sshnaidm|ruck | rlandy, as usual :D | 13:55 |
| panda | ssbarnea: anyway, for the scope of this review, did you address the regexp problem from wes ? | 13:55 |
| marios | sshnaidm|ruck: oh yeah i pointed to some logs sec | 13:55 |
| sshnaidm|ruck | rlandy, it's making problems last two weeks, so everything is possible.. | 13:56 |
| marios | sshnaidm|ruck: see my comment at https://review.openstack.org/#/c/583195/7/playbooks/tripleo-ci/run-v3.yaml@29 | 13:56 |
| ssbarnea | panda: what regex problem? i don't remember mentioning any problem with regex. | 13:56 |
| panda | ssbarnea: comment to patchset 3 from wes | 13:57 |
| ssbarnea | panda: if i remember well he wrote me that he made a mistake and thus why it didn't work, not the script fault. but obviously I can double check. | 13:58 |
| marios | tripleo time | 13:59 |
| *** ykarel is now known as ykarel|away | 14:04 | |
| panda | ssbarnea: ok, no need then. | 14:04 |
| sshnaidm|ruck | marios, another question, - hosts: subnodes - where is "subnodes" defined? | 14:06 |
| sshnaidm|ruck | marios, is it coming from zuul? | 14:06 |
| *** hamzy_ has quit IRC | 14:07 | |
| marios | sshnaidm|ruck: https://github.com/openstack-infra/tripleo-ci/blob/master/zuul.d/nodesets.yaml | 14:08 |
| sshnaidm|ruck | marios, ack | 14:08 |
| marios | sshnaidm|ruck: you can see in the jobs inventory like http://logs.openstack.org/95/583195/7/check/tripleo-ci-centos-7-3nodes-multinode/324ccdf/zuul-info/inventory.yaml | 14:09 |
| myoung | if anyone has a hot sec, could you please join the community meeting now/early? i have a new headset and want to level check :) | 14:11 |
| *** ykarel|away has quit IRC | 14:12 | |
| *** kopecmartin has quit IRC | 14:14 | |
| *** dtrainor has quit IRC | 14:17 | |
| panda | chkumar|rover: do you really need to run docker run to chmod a directory that ws originally mounted from the host ? would a chmod in the host work ?\ | 14:18 |
| panda | chkumar|rover: or is there some problem with UIDs ? | 14:18 |
| *** dtrainor has joined #oooq | 14:19 | |
| panda | chkumar|rover: talking about (https://review.openstack.org/584771) | 14:27 |
| marios | /win 11 | 14:29 |
| panda | /lose 20 | 14:30 |
| *** links has quit IRC | 14:31 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ (1 more message) | 14:36 |
| chkumar|rover | panda: yes as it is discussed with mandre for the same | 14:37 |
| panda | chkumar|rover: ok sorry | 14:38 |
| chkumar|rover | panda: the thing is that we need to ublock the promotion then I need to come up with a better soltuion | 14:38 |
| *** hamzy_ has joined #oooq | 14:50 | |
| *** hamzy_ has quit IRC | 14:52 | |
| *** hamzy_ has joined #oooq | 14:52 | |
| *** udesale has joined #oooq | 14:56 | |
| *** saneax has quit IRC | 14:58 | |
| *** dtrainor has quit IRC | 15:04 | |
| *** dtrainor has joined #oooq | 15:05 | |
| agopi|brb | rlandy, yes we're getting there, https://logs.rdoproject.org/17/583717/16/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053-master/d11218f/job-output.txt.gz hit this error on browbeat side, rook might've an idea. | 15:17 |
| rlandy | good | 15:17 |
| *** fhubik|mtg has left #oooq | 15:17 | |
| *** ccamacho1 has quit IRC | 15:18 | |
| *** udesale has quit IRC | 15:18 | |
| *** rlandy is now known as rlandy|brb | 15:42 | |
| vinaykns | Hello channel...I have a question...I was trying to deploy quickstart using devmode.sh on the rdo-cloud....but it is failing at getting the network deployment file. | 15:48 |
| *** jfrancoa has quit IRC | 15:58 | |
| *** holser_ has quit IRC | 16:02 | |
| *** agopi|brb has quit IRC | 16:03 | |
| *** agopi|brb has joined #oooq | 16:03 | |
| *** trown is now known as trown|lunch | 16:16 | |
| *** dsneddon has joined #oooq | 16:17 | |
| *** panda is now known as panda|off | 16:18 | |
| *** zoli is now known as zoli|gone | 16:19 | |
| *** zoli|gone is now known as zoli | 16:19 | |
| myoung | panda|off, chkumar|rover: FYI https://trello.com/c/FQaGNhq9/49-raise-ansible-deprecation-issue-top-level-fact-vars | 16:22 |
| *** lucasagomes is now known as lucas-afk | 16:24 | |
| *** ccamacho has joined #oooq | 16:24 | |
| chkumar|rover | myoung: ack | 16:25 |
| *** ccamacho has quit IRC | 16:25 | |
| *** ccamacho has joined #oooq | 16:25 | |
| *** jaganathan has quit IRC | 16:27 | |
| *** rlandy|brb is now known as rlandy | 16:28 | |
| *** vinaykns has quit IRC | 16:34 | |
| *** vinaykns has joined #oooq | 16:36 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ (1 more message) | 16:36 |
| *** sshnaidm|ruck is now known as sshnaidm|bbl | 16:37 | |
| *** dtantsur is now known as dtantsur|afk | 16:39 | |
| rlandy | need to run out for a bit | 16:49 |
| *** rlandy is now known as rlandy|afk | 16:49 | |
| *** myoung is now known as myoung|lunch | 16:58 | |
| *** agopi|brb has quit IRC | 17:03 | |
| *** holser_ has joined #oooq | 17:15 | |
| *** tesseract has quit IRC | 17:17 | |
| *** ccamacho has quit IRC | 17:20 | |
| *** vinaykns has quit IRC | 17:31 | |
| *** amoralej is now known as amoralej|off | 17:37 | |
| *** trown|lunch is now known as trown | 17:37 | |
| *** hamzy_ has quit IRC | 17:55 | |
| *** jaganathan has joined #oooq | 17:57 | |
| *** vinaykns has joined #oooq | 18:07 | |
| *** gkadam has quit IRC | 18:17 | |
| *** agopi|brb has joined #oooq | 18:23 | |
| *** agopi|brb is now known as agopi | 18:27 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp- (2 more messages) | 18:37 |
| *** myoung|lunch is now known as myoung | 18:37 | |
| *** hamzy_ has joined #oooq | 18:39 | |
| *** holser_ has quit IRC | 18:59 | |
| agopi | rook, myoung rlandy|afk are we still on? | 19:00 |
| myoung | agopi: i'm joining, think rlandy|afk isn't back yuet | 19:01 |
| myoung | agopi: what's the meeting id? | 19:01 |
| myoung | # | 19:02 |
| agopi | ack myoung https://bluejeans.com/u/jtaleric/ | 19:02 |
| *** rlandy|afk is now known as rlandy | 19:03 | |
| *** vinaykns has quit IRC | 19:21 | |
| *** vinaykns has joined #oooq | 19:33 | |
| *** agopi is now known as agopi|brb | 19:42 | |
| *** agopi|brb is now known as agopi | 20:02 | |
| *** sshnaidm|bbl is now known as sshnaidm|ruck | 20:11 | |
| *** jtomasek has quit IRC | 20:20 | |
| *** chem has quit IRC | 20:22 | |
| rook | rlandy: what do we do about a RETRY_LIMIT? | 20:26 |
| * rook tries to insert more coins. | 20:27 | |
| agopi | rook, why did it fail btw? | 20:28 |
| myoung | chkumar|rover, sshnaidm|ruck, rlandy: going thru reviews for https://trello.com/c/yxnO7GGf/871-create-a-job-to-test-browbeat, for the misc failures from yesterday, are we good to recheck? | 20:28 |
| myoung | chkumar|rover / sshnaidm|ruck ^^ are we still expecting misc infra errors and not clear for rechecks? | 20:28 |
| rook | 2018-07-24 20:25:17.757850 | tripleo-ovb-centos-7 | ok: 1 changed: 0 unreachable: 0 failed: 1 | 20:28 |
| rook | agopi: ^ | 20:28 |
| rook | maybe due to rdo cloud | 20:29 |
| rook | ? | 20:29 |
| rlandy | RETRY_LIMIT? | 20:29 |
| rook | 2018-07-24 20:25:24.529051 | Timeout exception waiting for the logger. Please check connectivity to [38.145.32.181:19885] | 20:29 |
| rook | https://review.openstack.org/#/c/583717/ rlandy | 20:29 |
| agopi | https://logs.rdoproject.org/17/583717/17/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053-master/026bb64/job-output.txt.gz | 20:29 |
| rlandy | recheck? | 20:31 |
| rlandy | I would think that's an rdocloud glitch | 20:31 |
| rlandy | looking more | 20:31 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph- (2 more messages) | 20:37 |
| sshnaidm|ruck | rlandy, we have problem with linters | 20:39 |
| sshnaidm|ruck | rlandy, rook when we include some external role/playbook in ours, linter starts to check them too | 20:40 |
| rlandy | sshnaidm|ruck: hmmm ... because of the requirements? | 20:40 |
| sshnaidm|ruck | rlandy, we need either to skip them in linter, or to change playbooks. The same problem was with tripleo-upgrade repo | 20:40 |
| sshnaidm|ruck | rlandy, it clones it and linter goes on all code | 20:41 |
| rlandy | sshnaidm|ruck, what did we decide to do for upgrades | 20:42 |
| rlandy | we should do the same | 20:42 |
| sshnaidm|ruck | rlandy, we fixed lint issues there | 20:43 |
| rlandy | how bad is it? | 20:43 |
| rlandy | link? | 20:43 |
| *** d0ugal has quit IRC | 20:44 | |
| sshnaidm|ruck | rlandy, hmm.. actually linter failed on your code in nodepool role | 20:46 |
| rlandy | nodepool role as in the reproducer | 20:46 |
| rlandy | sshnaidm|ruck" I think, ideally, we should fix the issues | 20:47 |
| sshnaidm|ruck | rlandy, linter checks all code | 20:47 |
| rlandy | but it depends how bad it is for now | 20:47 |
| sshnaidm|ruck | rlandy, commented there | 20:47 |
| sshnaidm|ruck | rlandy, http://logs.openstack.org/88/581488/21/check/openstack-tox-linters/01c3499/job-output.txt.gz | 20:48 |
| rlandy | sec- looking | 20:49 |
| *** agopi is now known as agopi|brb | 20:51 | |
| rlandy | ok - I see | 20:51 |
| *** hamzy_ has quit IRC | 20:55 | |
| *** trown is now known as trown|outtypewww | 21:02 | |
| rlandy | sshnaidm|ruck: thanks - fixing | 21:02 |
| *** brault has quit IRC | 21:17 | |
| rlandy | sshnaidm|ruck: so in this case, the copy from line 31 confuses matters | 21:24 |
| rlandy | it is my understanding that the copy will overwrite the existing requirements file | 21:24 |
| rlandy | the linter is correct - I made a typo | 21:26 |
| *** agopi|brb is now known as agopi | 21:35 | |
| agopi | rlandy, so how do we get the job back to running like before? | 21:35 |
| agopi | what do we need to do? | 21:36 |
| * rlandy looks | 21:37 | |
| rlandy | 2018-07-24 20:46:44.690640 | | 21:38 |
| rlandy | 2018-07-24 20:46:44.690784 | PLAY RECAP | 21:38 |
| rlandy | 2018-07-24 20:46:44.690891 | tripleo-ovb-centos-7 | ok: 1 changed: 0 unreachable: 0 failed: 1 | 21:38 |
| rlandy | 2018-07-24 20:46:44.690955 | | 21:38 |
| * rlandy doesn't think that's your doing - looking at other jobs | 21:39 | |
| agopi | ack rlandy | 21:39 |
| rlandy | prep-workspace | 21:40 |
| rlandy | ok - looking | 21:40 |
| *** brault has joined #oooq | 21:41 | |
| rlandy | - name: Synchronize src repos to workspace directory. | 21:42 |
| rlandy | synchronize: | 21:42 |
| rlandy | dest: "{{ zuul_workspace_root }}" | 21:42 |
| rlandy | src: "{{ zuul.executor.src_root }}" | 21:42 |
| rlandy | ^^ error | 21:42 |
| *** agopi is now known as agopi|brv | 21:48 | |
| *** agopi|brv has quit IRC | 21:53 | |
| rlandy | https://logs.rdoproject.org/88/581488/22/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/c4e8910/job-output.txt.gz#_2018-07-24_21_39_23_354006 | 21:58 |
| rlandy | looks like the same error - unrelated to your job | 21:58 |
| rlandy | https://review.rdoproject.org/zuul/stream.html?uuid=2efe5131bb294b7da08818694a51b67a&logfile=console.log | 22:01 |
| rlandy | looks better | 22:01 |
| rlandy | rook; ^^ | 22:01 |
| *** tosky has quit IRC | 22:07 | |
| rlandy | sshnaidm|ruck: still around? | 22:07 |
| rlandy | | ReadTimeoutError: HTTPConnectionPool(host='mirror.regionone.rdo-cloud-tripleo.rdoproject.org', port=80): Read timed out. | 22:07 |
| rlandy | more fun and games on rdocloud | 22:08 |
| rlandy | ttps://logs.rdoproject.org/17/583717/18/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053-master/2efe513/job-output.txt.gz#_2018-07-24_22_03_38_671781 | 22:08 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-ocata, legacy-tripleo-ci-centos-7 (3 more messages) | 22:37 |
| *** rlandy has quit IRC | 22:37 | |
| *** rfolco has quit IRC | 23:04 | |
| *** rfolco has joined #oooq | 23:05 | |
| *** vinaykns has quit IRC | 23:19 | |
| *** agopi has joined #oooq | 23:25 | |
| *** yolanda_ has joined #oooq | 23:56 | |
| *** yolanda has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!