| *** rfolco has joined #oooq | 00:01 | |
| *** matbu has quit IRC | 00:13 | |
| *** matbu has joined #oooq | 00:18 | |
| rlandy|afk | ok - rdocloud should be clean now | 00:26 |
|---|---|---|
| rlandy|afk | sshnaidm|afk: ^^ | 00:26 |
| *** rfolco has quit IRC | 00:44 | |
| *** rlandy|afk has quit IRC | 01:03 | |
| *** rfolco has joined #oooq | 01:13 | |
| *** rfolco has quit IRC | 01:33 | |
| *** rfolco has joined #oooq | 01:33 | |
| *** rfolco has quit IRC | 01:38 | |
| *** surpatil has joined #oooq | 03:15 | |
| *** surpatil is now known as surpatil||127_0_ | 03:28 | |
| *** surpatil||127_0_ is now known as surpatil | 03:29 | |
| *** surpatil is now known as surpatil|127_0_0 | 03:29 | |
| *** surpatil|127_0_0 is now known as surpatil | 03:29 | |
| *** surpatil is now known as surpatil|wrf | 03:32 | |
| *** surpatil|wrf is now known as surpatil|wfr | 03:32 | |
| *** surpatil|wfr is now known as surpatil|wfh | 03:33 | |
| *** openstackstatus has joined #oooq | 03:41 | |
| *** ChanServ sets mode: +v openstackstatus | 03:41 | |
| *** udesale has joined #oooq | 04:00 | |
| *** ykarel|away has joined #oooq | 04:01 | |
| *** bhagyashris has joined #oooq | 04:03 | |
| *** ykarel|away is now known as ykarel | 04:21 | |
| *** bhagyashris has quit IRC | 04:43 | |
| *** bhagyashris has joined #oooq | 04:49 | |
| *** soniya29 has joined #oooq | 05:03 | |
| *** raukadah is now known as chkumar|ruck | 05:20 | |
| *** skramaja has joined #oooq | 05:31 | |
| *** surpatil|wfh has quit IRC | 05:45 | |
| *** epoojad1 has joined #oooq | 05:52 | |
| *** jtomasek has joined #oooq | 06:01 | |
| *** jtomasek has quit IRC | 06:06 | |
| *** jtomasek has joined #oooq | 06:12 | |
| *** marios|rover has joined #oooq | 06:29 | |
| *** soniya29 has quit IRC | 06:31 | |
| *** soniya29 has joined #oooq | 06:39 | |
| *** dsneddon has quit IRC | 06:39 | |
| *** d0ugal has quit IRC | 06:48 | |
| *** d0ugal has joined #oooq | 06:51 | |
| *** dsneddon has joined #oooq | 07:04 | |
| *** epoojad1 has quit IRC | 07:06 | |
| *** epoojad1 has joined #oooq | 07:07 | |
| *** ykarel is now known as ykarel|lunch | 07:46 | |
| panda | marios|rover: enqueue https://review.rdoproject.org/r/23931 | 08:21 |
| marios|rover | needs votes and +A if the zuul is ready please https://review.opendev.org/#/c/696874/ https://review.opendev.org/#/c/696871/ https://review.opendev.org/#/c/696870/ https://review.opendev.org/#/c/696872/ | 08:24 |
| marios|rover | thanks and happy friday! | 08:25 |
| marios|rover | panda: ack | 08:25 |
| *** saneax has joined #oooq | 08:31 | |
| *** tesseract has joined #oooq | 08:31 | |
| marios|rover | anyone feels like merging that please thanks https://review.opendev.org/#/c/695878/ | 08:48 |
| chkumar|ruck | marios|rover: I can but no rights | 08:51 |
| marios|rover | chkumar|ruck: o/ i thought you are travelling today | 08:52 |
| chkumar|ruck | marios|rover: yes in another 30 mins | 08:52 |
| chkumar|ruck | marios|rover: regarding current master promotion, rdo master dlrn current is not consistent due to failure caught in rdo spec file | 08:56 |
| *** ykarel|lunch is now known as ykarel | 08:57 | |
| marios|rover | chkumar|ruck: i saw the pipelines are a mess | 08:57 |
| marios|rover | chkumar|ruck: i am ignoring it for now hoping it will go away | 08:57 |
| marios|rover | chkumar|ruck: also rhel8 promotion seems fcucked... like i see | 08:58 |
| marios|rover | * http://logs.rdoproject.org/19/23919/2/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/c21429e/logs/undercloud/home/zuul/undercloud_install.log.txt.gz | 08:58 |
| marios|rover | * #TODO bad promotion? we didn't have one! | 08:58 |
| marios|rover | the test for https://bugs.launchpad.net/tripleo/+bug/1853652 debug patch @ http://logs.rdoproject.org/19/23919/2/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/c21429e/logs/undercloud/home/zuul/undercloud_install.log.txt.gz * 2019-12-05 12:50:34 | tripleo_common.image.exception.ImageNotFoundException: Not found image: | 08:58 |
| openstack | Launchpad bug 1853652 in tripleo "openstack overcloud node provide --all-manageable timing out and failing periodic rhel-8-ovb-3ctlr_1comp-featureset001-master" [Critical,Triaged] | 08:58 |
| marios|rover | docker://trunk.registry.rdoproject.org/tripleomaster/rhel-binary-cron:36a84820e51bad57c6bbb92429f3afb3d9da29c2_6e3b098e | 08:58 |
| marios|rover | chkumar|ruck: anyway ack & safe travels i will review the pipelines in due course | 09:07 |
| chkumar|ruck | marios|rover: let me fix that for you | 09:09 |
| *** dsneddon has quit IRC | 09:11 | |
| chkumar|ruck | marios|rover: may be updating this https://review.rdoproject.org/r/#/c/23919/2/.zuul.yaml with container build dependency will help it to get the container | 09:13 |
| chkumar|ruck | marios|rover: somethign like this https://review.rdoproject.org/r/#/c/23825/10/zuul.d/projects.yaml and also include depends on https://review.opendev.org/#/c/697236/ container build | 09:14 |
| marios|rover | thanks chkumar|ruck | 09:18 |
| *** derekh has joined #oooq | 09:32 | |
| *** dsneddon has joined #oooq | 09:47 | |
| *** dsneddon has quit IRC | 09:53 | |
| panda | marios|rover: I gave all I can in those reviews, I'm not allowed (nor feel comfortable) gving +2 | 10:07 |
| *** dsneddon has joined #oooq | 10:09 | |
| marios|rover | panda: ack thanks | 10:10 |
| marios|rover | panda: not allowed? you mean cos other repos? yeay but if you referring to the 'make undercloud upgrade voting' then surely you are! | 10:11 |
| marios|rover | panda: but thakns will check thn again in bit | 10:11 |
| panda | marios|rover: I am ? | 10:12 |
| panda | marios|rover: Ill +W every single mf there. | 10:12 |
| *** derekh has quit IRC | 10:12 | |
| marios|rover | panda: not sure what you're referring to but since it is about jobs.. then we own those so... | 10:12 |
| *** dsneddon has quit IRC | 10:13 | |
| *** dsneddon has joined #oooq | 10:14 | |
| *** derekh has joined #oooq | 10:18 | |
| *** dsneddon has quit IRC | 10:19 | |
| *** d0ugal has quit IRC | 10:24 | |
| *** d0ugal has joined #oooq | 10:27 | |
| *** dtantsur|afk is now known as dtantsur | 10:28 | |
| *** rfolco has joined #oooq | 10:28 | |
| panda | rfolco: enqueue https://review.rdoproject.org/r/23931 | 10:30 |
| *** d0ugal has quit IRC | 10:31 | |
| rfolco | panda, ack, will do asap | 10:31 |
| *** holser has joined #oooq | 10:47 | |
| *** dsneddon has joined #oooq | 10:52 | |
| *** derekh has quit IRC | 10:53 | |
| *** derekh has joined #oooq | 10:55 | |
| *** dsneddon has quit IRC | 10:56 | |
| *** derekh has quit IRC | 10:59 | |
| *** derekh has joined #oooq | 10:59 | |
| zbr | panda: marios|rover : easy review: https://review.rdoproject.org/r/#/c/23661/ | 11:03 |
| marios|rover | zbr: make me | 11:05 |
| *** dsneddon has joined #oooq | 11:05 | |
| zbr | thanks! | 11:05 |
| *** dsneddon has quit IRC | 11:09 | |
| *** dsneddon has joined #oooq | 11:10 | |
| *** dsneddon has quit IRC | 11:15 | |
| *** dsneddon has joined #oooq | 11:19 | |
| *** dsneddon has quit IRC | 11:24 | |
| *** sshnaidm|afk is now known as sshnaidm|off | 11:28 | |
| *** zbr has quit IRC | 11:47 | |
| *** zbr has joined #oooq | 11:47 | |
| *** epoojad1 has quit IRC | 11:47 | |
| marios|rover | needs merge please https://review.opendev.org/#/c/697413/ | 11:50 |
| *** tosky has joined #oooq | 11:50 | |
| *** dsneddon has joined #oooq | 11:52 | |
| *** udesale has quit IRC | 12:00 | |
| *** dsneddon has quit IRC | 12:02 | |
| *** rfolco is now known as rfolco|bbl | 12:03 | |
| *** dsneddon has joined #oooq | 12:34 | |
| *** saneax has quit IRC | 12:43 | |
| *** rlandy has joined #oooq | 13:03 | |
| rlandy | marios|rover: hi - how are we doing ruck/rover wise? | 13:11 |
| rlandy | did I kill anything yesterday? | 13:11 |
| rlandy | marios|rover: also there are two reviews from stevebaker | 13:11 |
| marios|rover | rlandy: o/ i updated https://etherpad.openstack.org/p/ruckroversprint19 as usual... ongoing promotion blocker master/train are the headline i guess | 13:12 |
| marios|rover | rlandy: not aware of any new fires or fallout from your merges | 13:12 |
| rlandy | marios|rover: ok - then corrected them all yesterday | 13:12 |
| rlandy | marios|rover: pls see https://review.opendev.org/#/c/680571 and https://review.opendev.org/#/c/680573/ | 13:12 |
| marios|rover | rlandy: ack in a few finish current thing | 13:13 |
| rlandy | https://review.rdoproject.org/zuul/builds?pipeline=openstack-component-compute | 13:17 |
| rlandy | rfolco|bbl: ^^ look what we have | 13:17 |
| marios|rover | rlandy: looks like we may have issues with rdo cloud | 13:18 |
| marios|rover | rlandy: currently train pipeline is runing and already some jobs hit RETRY_LIMIT | 13:18 |
| rlandy | marios|rover: ack - spent some time of that yesterday | 13:18 |
| rlandy | marios|rover: we had growing stacks | 13:18 |
| rlandy | not being deleted | 13:19 |
| marios|rover | rlandy: i already saw that today stein today run https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-24hr lots of jobs RETRY_LIMIT & no logs | 13:19 |
| marios|rover | rlandy: ah ack didn't know it was ongoing from yesterday | 13:19 |
| rlandy | marios|rover: we need to catch one job with that before it bails and see | 13:19 |
| rlandy | let me look at the tenant | 13:20 |
| marios|rover | rlandy: few jobs still running in that pipeline | 13:21 |
| rlandy | https://review.opendev.org/gitweb?p=openstack/tripleo-ci.git;a=commitdiff;h=None in the component pipeline | 13:21 |
| marios|rover | rlandy: https://review.rdoproject.org/zuul/stream/58eef101a4e740ac8f14076b2ed84d76?logfile=console.log fs 20 | 13:21 |
| marios|rover | ack | 13:21 |
| rlandy | marios|rover: ^^ what is this console - we expect a retry limit here? | 13:22 |
| marios|rover | rlandy: well i don't know ... just that some of the jobs there hit that | 13:23 |
| marios|rover | rlandy: and i just pikced one of th eongoing ones. i don't know of a way to predict which one will hit that :D | 13:23 |
| rlandy | 2019-12-06 13:17:58.967060 | primary | "msg": "Data could not be sent to remote host \"38.145.32.110\". Make sure this host can be reached over ssh: ssh: connect to host 38.145.32.110 port 22: No route to host\r\n", | 13:23 |
| rlandy | 2019-12-06 13:17:58.967353 | primary | "unreachable": true | 13:23 |
| rlandy | 2019-12-06 13:17:58.955826 | [Zuul] Log Stream did not terminate | 13:23 |
| rlandy | marios|rover: http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-train/e0cc63a/ - is marked retry-limit | 13:25 |
| rlandy | but has logs | 13:25 |
| rlandy | with errors posted above | 13:25 |
| marios|rover | rlandy: ack good thanks noting | 13:26 |
| rlandy | marios|rover: looks like kforde isn't around | 13:26 |
| rlandy | looking through https://review.rdoproject.org/zuul/builds?result=RETRY_LIMIT | 13:27 |
| rlandy | 2019-12-06 12:04:35.850539 | [Zuul] Log Stream did not terminate | 13:28 |
| rlandy | 2019-12-06 12:04:35.850890 | primary | ERROR | 13:28 |
| rlandy | 2019-12-06 12:04:35.851050 | primary | { | 13:28 |
| rlandy | 2019-12-06 12:04:35.851113 | primary | "msg": "Data could not be sent to remote host \"38.145.34.71\". Make sure this host can be reached over ssh: ssh: connect to host 38.145.34.71 port 22: No route to host\r\n", | 13:28 |
| rlandy | 2019-12-06 12:04:35.851167 | primary | "unreachable": true | 13:28 |
| rlandy | ^^ that's a retry problem | 13:28 |
| *** skramaja has quit IRC | 13:36 | |
| rlandy | issue with collect logs connection | 13:43 |
| rlandy | marios|rover: ^^ | 13:45 |
| marios|rover | rlandy: looks like chkumar|ruck noted that on the etherpad before he left this morning " https://review.rdoproject.org/zuul/builds?result=RETRY_LIMIT it again arrived" | 13:45 |
| rlandy | yeah but the job run fine | 13:45 |
| rlandy | and fails collecting logs | 13:45 |
| rlandy | you'll note that there are no logs | 13:46 |
| rlandy | marios|rover: when was the first time you saw the RETRY_LIMIT error? | 13:51 |
| rlandy | I think I may know the reason | 13:51 |
| marios|rover | rlandy: me today | 13:53 |
| rlandy | marios|rover: may be the change to the clean up script | 13:54 |
| rlandy | I will try reverse it | 13:54 |
| marios|rover | rlandy: via http://dashboard-ci.tripleo.org/d/YRJtmtNWk/cockpit?orgId=1&fullscreen&panelId=231 it seems to have peaked already today | 13:55 |
| marios|rover | rlandy: maybe it should be better... but still of the order of 40 undeleted stacks | 13:55 |
| marios|rover | rlandy: makes sense then 15:54 < rlandy> marios|rover: may be the change to the clean up script | 13:55 |
| *** soniya29 has quit IRC | 13:55 | |
| rlandy | marios|rover: pls see #sf-ops | 13:57 |
| marios|rover | ack rlandy | 13:58 |
| *** bhagyashris has quit IRC | 14:07 | |
| rlandy | sshnaidm|off: wrt cleanup scripts ... why move port deletion to the end? https://github.com/rdo-infra/ci-config/commit/55e4095c3a4d9a5d8d474482d7423557d68b632c | 14:12 |
| rlandy | stacks fail deletion due to ports | 14:12 |
| rlandy | if we get rid of the ports first, the stacks deletw | 14:12 |
| rlandy | otherwise they become DELETE_FAILED | 14:13 |
| sshnaidm|off | rlandy, heat actually should delete all its resources, including ports | 14:15 |
| sshnaidm|off | rlandy, are you sure it's because of ports? and which ports exactly? | 14:15 |
| rlandy | sshnaidm|off: heat should but when heat doesn't the clean up does | 14:16 |
| rlandy | yes - I'm sure | 14:16 |
| sshnaidm|off | rlandy, the only port that can't be deleted is port on undercloud that is connected to provision_ net of heat | 14:16 |
| sshnaidm|off | rlandy, we run stack delete before and heat should clean it up | 14:16 |
| rlandy | sshnaidm|off: correct but a lot of stacks get stuck due to ports that can't be deleted | 14:17 |
| sshnaidm|off | rlandy, mmm.. but maybe these undercloud ports doesn't allow him to delete, it makes sense | 14:17 |
| rlandy | sshnaidm|off: so the order was important | 14:17 |
| sshnaidm|off | rlandy, yeah, need to separate these ports from others.. I think we can revert it for now, will work on that later | 14:17 |
| rlandy | sshnaidm|off: I'd be ok with it, if and only if we reran the stack deletion after the port deletion | 14:18 |
| rlandy | sshnaidm|off: we are debugging a sporadic retry failure | 14:18 |
| rlandy | where at the point we reach collect logs, the node can;t be reached | 14:18 |
| sshnaidm|off | rlandy, just previous version worked not good as well, deleting ports which was mostly unnecessary was taking a long time and stacks weren't deleted | 14:18 |
| rlandy | sshnaidm|off: agreed that was not perfect | 14:19 |
| rlandy | sshnaidm|off: the better way to do it is to isolate the ports | 14:19 |
| rlandy | or | 14:19 |
| rlandy | run stack deletion again after port deletion | 14:19 |
| rlandy | so delete stacks - those that can delete will | 14:19 |
| rlandy | those that fail, will wait for port deletion | 14:20 |
| sshnaidm|off | well, stacks can't be deleted if these ports are up | 14:20 |
| rlandy | and then get a second chance | 14:20 |
| rlandy | heat can take care of it in some cases | 14:20 |
| rlandy | some not | 14:20 |
| sshnaidm|off | rlandy, wait, why not if undercloud host is gone actually | 14:20 |
| sshnaidm|off | well, it's still not a part of heat stack.. | 14:21 |
| rlandy | we have a script o hit servers as well - different one | 14:21 |
| sshnaidm|off | need to look for these ports exactly, delete them first, delete stacks, and then all the rest of ports | 14:21 |
| rlandy | yeah - I'd agree with looking more closely for the ports | 14:22 |
| rlandy | I suspect that the retry getting to the nods in collect logs has something to do with deleting ports | 14:23 |
| rlandy | we should only delete donw ports | 14:23 |
| rlandy | down | 14:23 |
| rlandy | though | 14:23 |
| rlandy | but maybe if the old nodes remain, there is the ip reuse | 14:23 |
| rlandy | floating ip as we have seen before | 14:24 |
| rlandy | sshnaidm|off: so yeah - let's revert the change, rethink how to define the ports better | 14:24 |
| sshnaidm|off | we do delete only down ports | 14:24 |
| sshnaidm|off | need to look at ports devices or networks | 14:25 |
| sshnaidm|off | rlandy, ack to revert | 14:25 |
| rlandy | we delete subnets and networks | 14:27 |
| rlandy | afterwards | 14:27 |
| rlandy | so we can be more selective | 14:27 |
| *** fmount has quit IRC | 14:28 | |
| rlandy | sshnaidm|off: k - proposed revert but needs rebase - fixing | 14:29 |
| rlandy | we can put back the 6 hours | 14:30 |
| *** fmount has joined #oooq | 14:31 | |
| *** Goneri has joined #oooq | 14:38 | |
| rlandy | marios|rover: sshnaidm|off: https://review.rdoproject.org/r/#/c/23994/ | 14:48 |
| *** ykarel is now known as ykarel|afk | 14:50 | |
| marios|rover | rlandy: ack checking | 14:55 |
| rlandy | sorry - I need to put back the 360 | 14:56 |
| rlandy | done | 14:57 |
| chkumar|ruck | marios|rover: I thinl dlrn builds for master is now consistent https://review.rdoproject.org/r/#/q/topic:pin-py2+(status:open+OR+status:merged) got merged 4 hours ago | 14:58 |
| marios|rover | chkumar|ruck: ack cool ... you arrived ... somewhere? | 14:59 |
| chkumar|ruck | marios|rover: yes | 15:00 |
| chkumar|ruck | in my hotel | 15:00 |
| marios|rover | rlandy: ah rlandy i should have checked irc first i commented on that just now but +2 anyway | 15:00 |
| marios|rover | chkumar|ruck: \o/ | 15:00 |
| marios|rover | chkumar|ruck: ok it's all yours | 15:00 |
| * marios|rover runs | 15:00 | |
| marios|rover | (joke) | 15:01 |
| chkumar|ruck | hehe | 15:01 |
| chkumar|ruck | rlandy: is here , so no need to worry | 15:01 |
| *** chkumar|ruck is now known as raukadah | 15:01 | |
| *** bhagyashris has joined #oooq | 15:02 | |
| rlandy | raukadah: it's fine - relax in the hotel ... I agreed to this | 15:02 |
| rlandy | marios|rover: so fs020 is running longer than 5 hours | 15:03 |
| rlandy | so could get deleted | 15:03 |
| marios|rover | rlandy: train? | 15:03 |
| *** bhagyashris has quit IRC | 15:04 | |
| rlandy | train and master | 15:04 |
| rlandy | checking retry limits | 15:04 |
| rlandy | 2019-12-06 14:37:08.326548 | primary | "msg": "Data could not be sent to remote host \"38.145.32.108\". Make sure this host can be reached over ssh: ssh: connect to host 38.145.32.108 port 22: No route to host\r\n", | 15:04 |
| rlandy | 2019-12-06 14:37:08.326620 | primary | "unreachable": true | 15:04 |
| rlandy | 2019-12-06 14:37:08.326710 | primary | } | 15:04 |
| rlandy | same errpr | 15:04 |
| rlandy | marios|rover: ok fine- let | 15:05 |
| rlandy | s do a clean revert | 15:05 |
| mjturek | anyone seen an error like this before? https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/1811/logs/logs/build.log at 12:25:50 | 15:05 |
| mjturek | I'm assuming it's the container asking the host for the delorean repo? | 15:05 |
| rlandy | marios|rover: sorry - last time | 15:08 |
| rlandy | clean revert | 15:08 |
| marios|rover | mjturek: not seen that you sure it isn't network issue like it couldn't talk to 172 delorean.repo? | 15:11 |
| marios|rover | rlandy: ack looking | 15:11 |
| *** TrevorV has joined #oooq | 15:15 | |
| marios|rover | if someone has couple mins that needs merge please https://review.opendev.org/#/c/697413/ thanks | 15:20 |
| marios|rover | and that one https://review.opendev.org/#/c/695878/ | 15:22 |
| *** dsneddon has quit IRC | 15:24 | |
| rlandy | done | 15:24 |
| mjturek | marios|rover sure seems like it :-\ | 15:26 |
| mjturek | is 172.0.10.0 the docker bridge or something? | 15:27 |
| mjturek | where does it get set up? | 15:27 |
| marios|rover | mjturek: i don't know i would have to go dig but it would be somewhere in tripleo-common | 15:30 |
| mjturek | cool thanks | 15:31 |
| marios|rover | mjturek: sorry i am not getting any luck with grep on 'delorean.repo' under tripleo_common/image/* :/ | 15:33 |
| *** dsneddon has joined #oooq | 15:35 | |
| mjturek | marios|rover yeaaah I'm not seeing the IP anywhere in tripleo-common | 15:35 |
| mjturek | grep -rnE "172\.17\.0\.1" | 15:35 |
| mjturek | returns nothing | 15:35 |
| marios|rover | mjturek: so on the repo setup part... at least i can point you to the upstream job ... we setup repos in pre | 15:35 |
| marios|rover | mjturek: https://github.com/openstack/tripleo-ci/blob/7679b71817aa385ee35003ef7ca569f91bf5fe6f/playbooks/tripleo-buildcontainers/pre.yaml#L6-L7 | 15:36 |
| marios|rover | mjturek: but that seems to happen during the build in your case? | 15:36 |
| mjturek | marios|rover I mean we run the pre tasks as well | 15:37 |
| rlandy | marios|rover: ok -so the cleanup should be back - the down ports are mimimal now - no failed stacks | 15:37 |
| rlandy | rerunning locally | 15:37 |
| rlandy | hopefully we should be done with retry_limite | 15:37 |
| rlandy | let's see | 15:37 |
| marios|rover | rlandy: k going to recheck that in a bit https://review.rdoproject.org/r/23986 it didn't report yet butsome of them hit retry limit | 15:38 |
| marios|rover | rlandy: i posted it cos stein hit retry limits | 15:38 |
| marios|rover | rlandy: well may as well recheck it now? will that work maybe i should click rebase instead | 15:38 |
| *** ykarel|afk is now known as ykarel|away | 15:38 | |
| rlandy | marios|rover: k - either | 15:39 |
| rlandy | rebase it | 15:39 |
| rlandy | and rerun | 15:39 |
| marios|rover | rlandy: cant rebase testproject | 15:39 |
| rlandy | right | 15:39 |
| marios|rover | rlandy: just hit recheck | 15:39 |
| marios|rover | typed it anyway | 15:39 |
| *** dsneddon has quit IRC | 15:39 | |
| rlandy | change the commit message if it won't rerun immediately | 15:39 |
| marios|rover | rlandy: abandon restore :D | 15:40 |
| marios|rover | rlandy: that kills the queue | 15:40 |
| rlandy | hack hack | 15:40 |
| marios|rover | cool they are already queued again | 15:41 |
| marios|rover | rlandy: posting the missing train jobs then gimme few | 15:42 |
| marios|rover | rlandy: i mean the ones that hit retry maybe we can get train today too that would be very nice of a friday | 15:42 |
| rlandy | rechecking https://review.opendev.org/#/c/697236 | 15:42 |
| rlandy | sure | 15:42 |
| rlandy | good idea | 15:42 |
| mjturek | marios|rover: https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/1811/logs/consoleText.txt if you want to see the pre-tasks run | 15:42 |
| marios|rover | rlandy: ack that is rhel8 17:42 < rlandy> rechecking https://review.opendev.org/#/c/697236 | 15:43 |
| rlandy | correct | 15:43 |
| *** dsneddon has joined #oooq | 15:44 | |
| *** ykarel|away has quit IRC | 15:49 | |
| marios|rover | rlandy: https://review.rdoproject.org/r/23995 current train criteria like http://paste.openstack.org/raw/787253/ | 15:54 |
| rlandy | marios|rover: ack -ok | 15:55 |
| rlandy | marios|rover: and we are not expecting a master promotion | 15:55 |
| marios|rover | rlandy: not until we get consistent build see comment #7 https://bugs.launchpad.net/tripleo/+bug/1855063 | 15:56 |
| openstack | Launchpad bug 1855063 in tripleo "Master standalone deploy failed with Table 'ovn_revision_numbers' already exists while performing neutron sync" [Critical,Confirmed] | 15:56 |
| marios|rover | rlandy: and related ping from ykarel in rdo earlier 15:16 < ykarel> marios|rover, fyi periodic job started but repo is still not consistent, still some packages pending to build https://trunk.rdoproject.org/centos7-master/queue.html, next run should be good | 15:56 |
| marios|rover | rlandy: 'earlier' like 3 hours ago | 15:56 |
| rlandy | marios|rover: ok so next master run maybe - only centos though | 15:57 |
| rlandy | still waiting on review for rhel | 15:57 |
| marios|rover | rlandy: yes. rhel is blocked on melanx | 15:57 |
| rlandy | marios|rover: rhel train> | 15:57 |
| marios|rover | rlandy: well waiting for either chandan fix or dropping the container but not for us to make that call | 15:57 |
| marios|rover | rlandy: yes | 15:57 |
| marios|rover | rlandy: no | 15:57 |
| marios|rover | rlandy: checking but i thought master | 15:57 |
| rlandy | yes, no - to which? | 15:58 |
| marios|rover | 17:57 < rlandy> marios|rover: rhel train> | 15:58 |
| marios|rover | https://bugs.launchpad.net/tripleo/+bug/1855050 references master afaics | 15:58 |
| openstack | Launchpad bug 1855050 in tripleo "RHEL 8 container build failed while building neutron-mlnx-agent due to missing libvirt-python python-ethtool python-networking-mlnx" [Critical,Confirmed] | 15:58 |
| marios|rover | rlandy: panda: owns train promotions on rhel for now | 15:58 |
| marios|rover | rlandy: via the 'new' promoter fyi | 15:58 |
| rlandy | marios|rover: amazing - not our problem | 15:58 |
| marios|rover | (as an aside since you brought up rhel train) | 15:58 |
| marios|rover | rlandy: see 'promotion pipelines status' in etherpad i hav it updated on all branches with pointers | 15:59 |
| *** dsneddon has quit IRC | 15:59 | |
| *** fmount has quit IRC | 16:10 | |
| mjturek | marios|rover just a heads up the next ppc64le run is in about 2 hours. Planning to jump on the node and investigate a bit more | 16:11 |
| *** fmount has joined #oooq | 16:12 | |
| *** rfolco|bbl is now known as rfolco | 16:12 | |
| mjturek | if you have any advice on what to look for, let me kno! | 16:12 |
| rfolco | rlandy, have you found the root cause of retry_limit ? | 16:12 |
| rlandy | maybe | 16:13 |
| *** fmount has quit IRC | 16:13 | |
| *** apetrich has quit IRC | 16:15 | |
| *** fmount has joined #oooq | 16:18 | |
| rfolco | rlandy, did you run it again in testproject to confirm? | 16:20 |
| rlandy | rfolco: run what against testproject? | 16:20 |
| rfolco | standlone retry_limit job | 16:20 |
| rfolco | rlandy, what is preventing us to merge https://review.rdoproject.org/r/#/c/23875/ | 16:21 |
| rlandy | rfolco: nothing - pls ask another core to vote | 16:22 |
| rlandy | marios|rover: ^^ pls | 16:22 |
| rlandy | there is duplication | 16:22 |
| rlandy | other than that ok | 16:22 |
| *** dtantsur is now known as dtantsur|afk | 16:22 | |
| marios|rover | rlandy: rfolco: don't think i'll do that review justice i will have a look on monday if still around sorry brain is done | 16:26 |
| * marios|rover gets ready to go | 16:26 | |
| marios|rover | mjturek: o/ it is my eod very soon now... not sure what to suggest for that rlandy fyi mjturek had some build container fail ... error trying to fetch delorean.repo | 16:26 |
| rlandy | panda? | 16:26 |
| *** dsneddon has joined #oooq | 16:28 | |
| rfolco | mjturek, link ? | 16:33 |
| rfolco | mjturek, for the error fetching delorean.repo | 16:33 |
| *** dsneddon has quit IRC | 16:33 | |
| rfolco | got it | 16:34 |
| rfolco | https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/1811/logs/logs/build.log | 16:34 |
| marios|rover | rfolco: 17:05 < mjturek> anyone seen an error like this before? https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/1811/logs/logs/build.log at 12:25:50 | 16:34 |
| *** marios|rover is now known as marios|out | 16:34 | |
| *** d0ugal has joined #oooq | 16:35 | |
| *** dsneddon has joined #oooq | 16:36 | |
| *** dsneddon has quit IRC | 16:40 | |
| *** marios|out has quit IRC | 16:42 | |
| *** rlandy is now known as rlandy|ruck | 16:44 | |
| rlandy|ruck | rfolco: ^^ covering for ruck/rover now | 16:44 |
| rlandy|ruck | rfolco: I am ok with merging that patch | 16:45 |
| rfolco | rlandy|ruck, ok, if no other cores to review, we'll do it on monday | 16:45 |
| rlandy|ruck | rfolco: cores or no cores - monday we merge | 16:45 |
| rfolco | k | 16:45 |
| rlandy|ruck | rfolco: pretty minimal impact | 16:46 |
| rlandy|ruck | panda: ping ^^ could you take a look at rfolco patch? | 16:46 |
| rlandy|ruck | we would like another core to look at | 16:47 |
| rfolco | mjturek, looks like the containers cannot access the host for some reason -- http://172.17.0.1/delorean.repo | 16:49 |
| *** dsneddon has joined #oooq | 16:57 | |
| *** dsneddon has quit IRC | 17:02 | |
| *** derekh has quit IRC | 17:03 | |
| rlandy|ruck | baseurl=https://trunk-staging.rdoproject.org/centos7/component/compute/22/30/2230ec836ba41337e1fa870eeece971649e8bbf7_c9bfa013 | 17:05 |
| *** ykarel|away has joined #oooq | 17:16 | |
| *** apetrich has joined #oooq | 17:18 | |
| *** rlandy|ruck is now known as rlandy|ruck|brb | 17:29 | |
| *** dsneddon has joined #oooq | 17:30 | |
| *** d0ugal has quit IRC | 17:32 | |
| *** tesseract has quit IRC | 17:35 | |
| *** d0ugal has joined #oooq | 17:36 | |
| zbr | rlandy|ruck|brb or rfolco: https://review.rdoproject.org/r/#/c/23984/ (easy) | 17:48 |
| *** rlandy|ruck|brb is now known as rlandy|ruck | 17:54 | |
| rlandy|ruck | zbr: ^^ in return, pls review https://review.rdoproject.org/r/#/c/23875/ for rfolco | 17:58 |
| zbr | rlandy|ruck: sure but i see it WIP with lots of pending changes. | 17:59 |
| zbr | i guess is should wait until these are addressed and review it then. | 18:00 |
| *** fmount has quit IRC | 18:01 | |
| *** holser has quit IRC | 18:01 | |
| rlandy|ruck | zbr: ok - panda looked at - we're good | 18:03 |
| zbr | rfolco: added few comments, but i still have few questions like why not using environment instead of defining vars inside shell snippets. | 18:09 |
| rfolco | zbr, nice thank you | 18:23 |
| zbr | rfolco: did you have something preventing you from using ansible environment to declare these vars? | 18:24 |
| rfolco | zbr, I am following the pattern, not sure how I should pass the env vars | 18:24 |
| zbr | https://hackmd.io/7Aj-hWlvSeyqYCUaxqnqoA | 18:31 |
| zbr | i am not asking you to do it like this, just to consider it. | 18:32 |
| rfolco | zbr, thanks for sharing | 18:32 |
| zbr | if the environment way helps us reduce number of lines, easy maintenance, we shoudl use it. | 18:32 |
| zbr | otherwise we can stick to bash mode. | 18:33 |
| rfolco | zbr, one more avise | 18:33 |
| rfolco | advise | 18:33 |
| mjturek | rfolco yeah just some stuff in the cico node during the run and I'm stumped | 18:35 |
| rfolco | how you would search some words in an output from dlrn for example? I run dlrnapi and it returns stdout. Then in a follow up task, I grep a few items with with_item list stdout_lines | 18:35 |
| rfolco | zbr, ^ | 18:35 |
| mjturek | the docker bridge is there, is pingable, and I can curl the delorean file | 18:35 |
| rfolco | mjturek, selinux ? | 18:35 |
| zbr | what format is the dlrnapi result? | 18:35 |
| zbr | json? | 18:35 |
| mjturek | rfolco: but that's all tried locally within the cico node | 18:35 |
| rfolco | yes it returns { } json format | 18:36 |
| rfolco | zbr, ^ | 18:36 |
| mjturek | rfolco: that's a theory, gonna try locally and see if I get the same result | 18:36 |
| zbr | rfolco: i would use URI if possible because it can load JSON into variable, bypassing bash/grep. | 18:37 |
| rfolco | zbr, I convert stdout w/ toniceyaml function? | 18:38 |
| zbr | another option is to use from_json, like advertised on https://stackoverflow.com/a/40844916/99834 -- but likely will force you to do it in two tasks. | 18:38 |
| rfolco | interesting... let me try something | 18:39 |
| zbr | either talk with dlrn api directly using uri module (probably is easy), or if you want to still call its cli, load the json in ansible | 18:39 |
| zbr | but always play locally, save result to a file and play with a local playbook to get it right | 18:39 |
| zbr | with ansible you almost never get it right from first attempt, at least me. | 18:39 |
| rfolco | oh retrieve from dlrnapi with uri | 18:40 |
| rfolco | now I got the idea | 18:41 |
| rfolco | zbr, thx | 18:41 |
| rfolco | we always did with client using bash script | 18:41 |
| zbr | if i remember its API is REST based and simple enough, sometimes this means that you could save time bypassing it. but first check, i may be wrong. | 18:42 |
| rfolco | zbr, maybe its a silly question, but how do I get something from https://trunk-staging.rdoproject.org/api-centos-master-uc/ | 18:56 |
| rfolco | dlrnapi --url https://staging.rdoproject.org/api-centos-master-uc repo-status --commit-hash bf577e5a999f7db4cb9b790664ad596e1926d9a0 --distro-hash 67a09fe97aa40ef05a73a3a7681700d2c25a58dd --success true | 18:56 |
| rfolco | zbr, how do I transform the args in a url | 18:57 |
| rfolco | zbr, I think I'll stick with cli :) | 18:58 |
| zbr | look at examples from https://docs.ansible.com/ansible/latest/modules/uri_module.html | 18:58 |
| zbr | in fact you can even do body: "{{ some_dict | to_json }}" to send them. | 18:58 |
| zbr | the key is to use POST, and not GET to avoid having to encode the URL | 18:59 |
| zbr | is also safer | 18:59 |
| zbr | look at example: Login to a form based webpage | 18:59 |
| rfolco | hmm cool | 18:59 |
| zbr | in fact, i would not be surprised to see someone already doing this with dlnr, use codesearch | 19:00 |
| zbr | maybe you can find an example and save few minutes | 19:00 |
| rfolco | zbr, panda gave me this dlrn module as example... | 19:01 |
| rfolco | https://github.com/softwarefactory-project/dlrnapi_client/blob/master/dlrnapi_client/ansible/example_playbook.yaml | 19:01 |
| zbr | sure, use dlnr_api module! | 19:02 |
| zbr | i did not know about it | 19:02 |
| rfolco | zbr, would have to install it from source I suppose... https://github.com/softwarefactory-project/dlrnapi_client | 19:04 |
| *** dsneddon has quit IRC | 19:05 | |
| zbr | rfolco: not really, we should be able to define it as an ansible dependency | 19:08 |
| zbr | not sure if it is already published on galaxy as a collection but this can be done easily, even without we can create an requirements.yml file for that. | 19:09 |
| zbr | i can help you next week on that. | 19:09 |
| mjturek | rfolco the plot thickens, locally the containers are building fine and selinux is enabled | 19:17 |
| *** tosky has quit IRC | 19:18 | |
| mjturek | any other ideas? | 19:20 |
| *** jtomasek has quit IRC | 19:28 | |
| *** dsneddon has joined #oooq | 19:34 | |
| rfolco | mjturek, maybe your job is missing any pre playbook ? did compare ? | 19:39 |
| mjturek | rfolco we run build-containers pre.yaml | 19:50 |
| mjturek | any other ones you can think of? | 19:50 |
| mjturek | I mean the only difference between the local env and the remote one is that we don't enable push locally? | 19:53 |
| rfolco | I've seen this error before but can't find the bug | 19:56 |
| mjturek | dang :( | 19:57 |
| rfolco | mjturek, in last case, bring this to next community call, where everyone is around and hopefully one will remember | 19:58 |
| mjturek | yeaaaah will do | 19:59 |
| rfolco | mjturek, searching in lp... | 19:59 |
| mjturek | rfolco: do you have the etherpad link for the community call? | 20:05 |
| rfolco | https://hackmd.io/IhMCTNMBSF6xtqiEd9Z0Kw | 20:05 |
| rfolco | mjturek, add your item at the top in the appropriate section | 20:06 |
| rfolco | mjturek, then this is copied over to the agenda before the mtg starts | 20:06 |
| mjturek | will do! | 20:06 |
| mjturek | alright I have the agenda item in | 20:10 |
| *** jtomasek has joined #oooq | 20:13 | |
| rfolco | mjturek, thanks. Sorry man, no luck in finding the bug. I cannot spot anything from the logs. If selinux and iptables are good... ran out of ideas. | 20:20 |
| rfolco | mjturek, but you had this working before, when this started ? | 20:20 |
| *** TrevorV has quit IRC | 20:27 | |
| rlandy|ruck | cleanup script | 20:34 |
| mjturek | once we switched to docker | 20:51 |
| mjturek | rfolco ^ | 20:51 |
| mjturek | but I wonder if the silent failure and this are related | 20:51 |
| *** dsneddon has quit IRC | 20:53 | |
| *** dsneddon has joined #oooq | 20:53 | |
| *** dsneddon has quit IRC | 20:58 | |
| *** dsneddon has joined #oooq | 20:59 | |
| *** dsneddon has quit IRC | 21:03 | |
| *** rlandy|ruck has quit IRC | 21:11 | |
| *** dsneddon has joined #oooq | 21:31 | |
| *** rfolco has quit IRC | 21:33 | |
| *** irclogbot_1 has quit IRC | 21:45 | |
| *** d0ugal has quit IRC | 21:51 | |
| *** ykarel|away has quit IRC | 22:11 | |
| *** jtomasek has quit IRC | 22:12 | |
| *** tosky has joined #oooq | 22:35 | |
| *** jbadiapa has quit IRC | 23:51 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!