| rlandy | sshnaidm|afk: you still around? | 00:13 |
|---|---|---|
| sshnaidm|afk | rlandy, yep | 00:13 |
| rlandy | sshnaidm|afk: can you help me clean up promoter? | 00:13 |
| rlandy | failed: [localhost] (item=rsyslog) => {"ansible_loop_var": "item", "changed": false, "item": "rsyslog", "msg": "Error pulling trunk.registry.rdoproject.org/tripleotrain/centos-binary-rsyslog - code: None message: failed to register layer: Error processing tar file(exit status 1): write /var/lib/rpm/Packages: no space left on device"} | 00:13 |
| rlandy | failing pulling containers | 00:13 |
| rlandy | sshnaidm|afk: what did you delete before? | 00:14 |
| sshnaidm|afk | lemme look | 00:14 |
| rlandy | dev/vdb1 80G 69G 12G 86% /var/lib/docker | 00:15 |
| sshnaidm|afk | rlandy, yeah, there are 2 months old images, removing them | 00:16 |
| rlandy | sshnaidm|afk: from where ? so I know what to do next time | 00:17 |
| sshnaidm|afk | docker image ls | 00:17 |
| sshnaidm|afk | in the bottom you can see '2 months ago' | 00:18 |
| sshnaidm|afk | we don't need them | 00:18 |
| rlandy | yeah I see | 00:19 |
| sshnaidm|afk | for i in $(docker image ls | grep ab34c90c15f5cb5efb7236fdf4b22020e6f3cd04_67a09fe9 | awk {'print $1'}); do docker rmi ${i}:ab34c90c15f5cb5efb7236fdf4b22020e6f3cd04_67a09fe9; done | 00:19 |
| sshnaidm|afk | so we remove this tag only | 00:20 |
| rlandy | 6bd5835cd379bcf9824bba7e9f81f2489d19e506_bada63cf 4537d04ac5a7 6 days ago | 00:20 |
| rlandy | nah - ^^ that can go as well | 00:20 |
| rlandy | if we need it, it will download again | 00:20 |
| *** rfolco|bbl has joined #oooq | 00:21 | |
| sshnaidm|afk | yeah | 00:21 |
| rlandy | thanks - master, train, stein,rocky could all promote today | 00:22 |
| sshnaidm|afk | wow | 00:23 |
| rlandy | see the logs, they all tried | 00:23 |
| rlandy | I figure this may have been going on for a few days | 00:23 |
| sshnaidm|afk | a big promotion day | 00:23 |
| rlandy | rhel is bad shape though | 00:24 |
| rlandy | can't get fs001 on train or master through | 00:24 |
| rlandy | collectd/healthcheck hang | 00:24 |
| rlandy | git one tht patch that may clear train eventually | 00:24 |
| rlandy | got | 00:25 |
| sshnaidm|afk | which one? | 00:25 |
| rlandy | https://review.opendev.org/#/c/699438/ | 00:27 |
| sshnaidm|afk | k, we'll see | 00:29 |
| sshnaidm|afk | I'll keep an eye next week, at least first days :) | 00:29 |
| rlandy | sshnaidm|afk: thanks | 01:10 |
| *** Goneri has quit IRC | 01:36 | |
| *** Goneri has joined #oooq | 01:38 | |
| *** ysandeep has joined #oooq | 01:43 | |
| *** rlandy has quit IRC | 01:51 | |
| *** ysandeep has quit IRC | 02:05 | |
| *** rfolco|bbl has quit IRC | 03:31 | |
| *** rfolco|bbl has joined #oooq | 03:32 | |
| *** rfolco|bbl has quit IRC | 03:42 | |
| *** ykarel|away has joined #oooq | 03:45 | |
| *** d0ugal has quit IRC | 03:47 | |
| *** d0ugal has joined #oooq | 03:48 | |
| *** ykarel|away has quit IRC | 04:24 | |
| *** dsneddon has quit IRC | 04:24 | |
| *** ykarel|away has joined #oooq | 04:24 | |
| *** bhagyashris has joined #oooq | 04:37 | |
| *** whoami-rajat has joined #oooq | 04:40 | |
| *** ykarel|away has quit IRC | 04:51 | |
| *** dsneddon has joined #oooq | 04:57 | |
| *** dsneddon has quit IRC | 05:09 | |
| *** ykarel|away has joined #oooq | 05:10 | |
| *** ykarel|away is now known as ykarel | 05:14 | |
| *** holser has joined #oooq | 05:19 | |
| *** whoami-rajat has quit IRC | 05:32 | |
| *** holser has quit IRC | 05:32 | |
| *** bhagyashris has quit IRC | 05:34 | |
| *** dsneddon has joined #oooq | 05:46 | |
| *** dsneddon has quit IRC | 05:51 | |
| *** bhagyashris has joined #oooq | 05:59 | |
| *** surpatil has joined #oooq | 06:01 | |
| *** surpatil has quit IRC | 06:16 | |
| *** surpatil has joined #oooq | 06:17 | |
| *** dsneddon has joined #oooq | 06:20 | |
| *** dsneddon has quit IRC | 06:25 | |
| *** skramaja has joined #oooq | 06:31 | |
| *** dsneddon has joined #oooq | 06:54 | |
| *** holser has joined #oooq | 06:57 | |
| *** dsneddon has quit IRC | 07:08 | |
| *** holser has quit IRC | 07:38 | |
| *** apetrich has joined #oooq | 07:41 | |
| *** dsneddon has joined #oooq | 07:42 | |
| *** marios has joined #oooq | 07:43 | |
| *** saneax has joined #oooq | 07:48 | |
| *** dsneddon has quit IRC | 07:51 | |
| *** dsneddon has joined #oooq | 07:54 | |
| *** bhagyashris has quit IRC | 07:57 | |
| *** dsneddon has quit IRC | 08:05 | |
| *** amoralej|off is now known as amoralej | 08:08 | |
| *** dsneddon has joined #oooq | 08:12 | |
| *** dsneddon has quit IRC | 08:16 | |
| *** tosky has joined #oooq | 08:19 | |
| *** dsneddon has joined #oooq | 08:22 | |
| *** holser has joined #oooq | 08:38 | |
| *** jpena|off is now known as jpena | 08:57 | |
| marios | 2019-12-20 04:29:44,459 30179 INFO promoter Promoting the container images for dlrn hash 99d58382abb0d6ad8ee126835cd40edf3ec07d6c on master to current-tripleo | 08:58 |
| *** dsneddon has quit IRC | 08:58 | |
| *** ccamacho has joined #oooq | 09:03 | |
| *** Tengu has quit IRC | 09:04 | |
| *** bhagyashris has joined #oooq | 09:08 | |
| *** yolanda has quit IRC | 09:10 | |
| *** SurajPatil has joined #oooq | 09:21 | |
| *** surpatil has quit IRC | 09:24 | |
| *** yolanda has joined #oooq | 09:28 | |
| *** dsneddon has joined #oooq | 09:31 | |
| *** derekh has joined #oooq | 09:42 | |
| *** dsneddon has quit IRC | 09:46 | |
| *** surpatil has joined #oooq | 10:10 | |
| *** SurajPatil has quit IRC | 10:12 | |
| *** dsneddon has joined #oooq | 10:13 | |
| *** dsneddon has quit IRC | 10:18 | |
| *** ykarel is now known as ykarel|afk | 10:25 | |
| *** dsneddon has joined #oooq | 10:42 | |
| *** Tengu has joined #oooq | 10:47 | |
| *** Tengu has quit IRC | 10:47 | |
| *** bhagyashris has quit IRC | 10:48 | |
| *** Tengu has joined #oooq | 10:49 | |
| *** dsneddon has quit IRC | 11:20 | |
| *** bhagyashris has joined #oooq | 11:22 | |
| *** ykarel|afk is now known as ykarel | 11:26 | |
| *** dsneddon has joined #oooq | 11:39 | |
| *** dsneddon has quit IRC | 11:46 | |
| *** rfolco|bbl has joined #oooq | 12:11 | |
| *** rfolco|bbl is now known as rfolco | 12:12 | |
| *** SurajPatil has joined #oooq | 12:15 | |
| *** surpatil has quit IRC | 12:15 | |
| *** dsneddon has joined #oooq | 12:18 | |
| *** bhagyashris_ has joined #oooq | 12:20 | |
| *** bhagyashris has quit IRC | 12:22 | |
| *** bhagyashris_ is now known as bhagyashris | 12:32 | |
| *** dsneddon has quit IRC | 12:36 | |
| rfolco | marios, panda weshay need this fix https://review.rdoproject.org/r/#/c/24273/ for the component promotion job | 12:36 |
| *** jpena is now known as jpena|lunch | 12:38 | |
| *** dsneddon has joined #oooq | 12:38 | |
| *** surpatil has joined #oooq | 12:40 | |
| marios | rfolco: k let me know if you want merge | 12:40 |
| rfolco | I do | 12:40 |
| marios | and so it was done | 12:42 |
| marios | the rfolco willed it so | 12:42 |
| marios | and it was | 12:42 |
| marios | # coolstorybro | 12:42 |
| *** SurajPatil has quit IRC | 12:42 | |
| *** SurajPatil has joined #oooq | 12:44 | |
| *** surpatil has quit IRC | 12:47 | |
| rfolco | marios, thank you. Do I have more 2 wishes, my lamp genius | 12:47 |
| rfolco | ? | 12:47 |
| marios | no | 12:49 |
| *** bhagyashris has quit IRC | 12:53 | |
| weshay | morning | 12:58 |
| *** rlandy has joined #oooq | 12:58 | |
| *** dsneddon has quit IRC | 13:00 | |
| rlandy | hey weshay | 13:01 |
| rlandy | welcome back | 13:01 |
| *** dsneddon has joined #oooq | 13:01 | |
| marios | o/ | 13:01 |
| marios | weshay: rlandy: have you ever had to add swap to the promoter | 13:02 |
| marios | weshay: rlandy: i added 4gb but not really seeing it be used at all | 13:02 |
| marios | i think i should disable it | 13:02 |
| rlandy | marios: no - but it keeps running out of space on device | 13:02 |
| rlandy | needed clean up yesterday | 13:03 |
| marios | rlandy: yeah i saw that... so i thought it was completely borked. everything, i mean all the things promoted today | 13:03 |
| marios | or trying to... | 13:03 |
| rlandy | and then all releases promoted | 13:03 |
| rlandy | and now it's out of space again | 13:03 |
| rlandy | to pormote phase 1 | 13:03 |
| rlandy | cleaning | 13:03 |
| marios | rlandy: so i saw a no space for stein. master was stuck for a veeeery long time | 13:03 |
| marios | rlandy: k going to remove that swap then if it isn't used then no point taking space | 13:03 |
| rlandy | marios: yep - figured that out late my time | 13:03 |
| marios | rlandy: maybe i should leave 2 gb? wdyt? | 13:04 |
| rlandy | probably harmless | 13:04 |
| marios | rlandy: but really i've been looking at top and i dno't see swap being used. but memory is maxed out | 13:04 |
| rlandy | it's the docker containers | 13:04 |
| rlandy | they are not being cleaned up | 13:04 |
| rlandy | dev/vdb1 80G 53G 28G 67% /var/lib/docker | 13:04 |
| weshay | marios, ah.. seeing those traces? | 13:06 |
| marios | weshay: which bit | 13:06 |
| rlandy | marios: queens is promoting now | 13:06 |
| weshay | promoter | 13:06 |
| weshay | sec in tempest call | 13:06 |
| rlandy | so leaving those containers | 13:06 |
| weshay | SurajPatil, Sorin ( zbr ) | 13:07 |
| rlandy | marios: also looks like there is a legit issue with the weirdo jobs | 13:07 |
| marios | rlandy: k made it 2 gb swap | 13:08 |
| marios | rlandy: see https://etherpad.openstack.org/p/ruckroversprint19 i added logs there | 13:08 |
| marios | rlandy: they're all promoting | 13:08 |
| rlandy | ykarel: ^^ https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-master-current-tripleo/ | 13:08 |
| rlandy | the weirdo jobs have been failing on master for the last few runs | 13:08 |
| marios | rlandy: train and stein left queens already promoted curre-tripleo maybe you mean queens promoting phase1 | 13:09 |
| rlandy | weirdo-master-promote-puppet-openstack-scenario00{1-4} | 13:09 |
| ykarel | rlandy, it need promotion of current-tripleo | 13:09 |
| ykarel | there was a fix recently in glance | 13:09 |
| rlandy | promoter Promoting the container images for dlrn hash 2c6d1578fef7087aa440544293190eca80ae4f9b on queens to current-tripleo-rdo | 13:09 |
| marios | weshay: basically all the things are promoting and it is straining the promoter server. | 13:09 |
| rlandy | ykarel: any action needed from us or the fix will work its way through? | 13:09 |
| *** dsneddon has quit IRC | 13:09 | |
| marios | kind of a nice problem to have but rlandy do we need to move the server to new node | 13:09 |
| marios | but never on a friday ... :/ | 13:10 |
| ykarel | rlandy, with next master promotion to current-tripleo , phase1 should clear | 13:10 |
| rlandy | marios: containers should be cleaned up | 13:10 |
| ykarel | rlandy, any known blockers for master promotion | 13:10 |
| rlandy | ykarel: k - thanks | 13:10 |
| marios | rlandy: having said that, i thought master was borked earlier but then it managed to finish container push | 13:10 |
| rlandy | it takes a long time | 13:10 |
| marios | rlandy: but next time lets get a bigger node. i mean why don't we just max that out | 13:10 |
| marios | weshay: ^^ | 13:10 |
| rlandy | because it had not promoted in a while | 13:10 |
| marios | rlandy: went though and closed bunch of things out and removed them so we have a shorter list at https://etherpad.openstack.org/p/ruckroversprint19 for ongoing things | 13:11 |
| rlandy | marios: also, as you say, not usual that all releases promote both phases on the same day | 13:12 |
| rlandy | but that was my goal | 13:12 |
| rlandy | to clear the board | 13:12 |
| marios | it's a christmas miracle | 13:12 |
| marios | \o/ | 13:12 |
| marios | ;) | 13:12 |
| rlandy | marios: rhel is still in trouble | 13:12 |
| rlandy | both train and master | 13:12 |
| rlandy | looking through current issues on the etherpad | 13:12 |
| marios | i man we've been chasing train for a looong time. i think its close | 13:12 |
| rlandy | we merged the supposed fix for train | 13:12 |
| marios | if we got train today/monday rlandy it would be icing on cake | 13:12 |
| rlandy | I didn;t see a pass yesterday | 13:12 |
| marios | rlandy: train failed in different way i saw one today sec | 13:12 |
| marios | * http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-train/4d09b8c/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 13:13 |
| marios | * 2019-12-19 18:32:45 | 2019-12-19 18:27:53 131238 INFO tripleo_common.image.image_uploader [ ] stderr: 'Error: Unknown repo: ''gating-repo''' | 13:13 |
| weshay | SurajPatil, https://github.com/pwittchen/learn-python-the-hard-way#unit-testing | 13:13 |
| rlandy | marios: oh gee | 13:14 |
| rlandy | that sounds legit | 13:14 |
| marios | weshay: relevant bug is there https://bugs.launchpad.net/tripleo/+bug/1853978/comments/27 i mean 'the rhel8 train bug' | 13:14 |
| openstack | Launchpad bug 1853978 in tripleo "periodic train rhel8 ovb overcloud deployment failed with Could not find class ::tripleo::profile::base::neutron::ovn_metadata_agent_wrappers" [Critical,Fix released] | 13:14 |
| rlandy | gating-repo is the update | 13:14 |
| rlandy | I think the promoter is running now | 13:15 |
| rlandy | space is at 61% | 13:15 |
| rlandy | which should be functional | 13:15 |
| rlandy | will watch if queens containers go away after promotion | 13:15 |
| marios | rlandy: k but for christmas i want a bigger promoter server please | 13:15 |
| rlandy | marios: I'll ping santa | 13:16 |
| marios | \o/ | 13:16 |
| *** jfrancoa has joined #oooq | 13:16 | |
| rlandy | marios: it's actually within our power | 13:16 |
| rlandy | our infra node | 13:16 |
| rlandy | we can add more storage | 13:17 |
| * marios not sure he likes the way this is heading. | 13:17 | |
| marios | rlandy: i'd just like to remind you today is friday | 13:17 |
| rlandy | marios; I don;t have a death wish | 13:17 |
| marios | rlandy: wait its just an rdo vm though right? | 13:18 |
| rlandy | yes | 13:18 |
| marios | rlandy: i see what you mean. but we have to pause it first no? | 13:18 |
| marios | rlandy: or we can just resize it | 13:18 |
| marios | rlandy: well attach a new bigger volume? | 13:18 |
| rlandy | periodic-tripleo-ci-rhel-8-scenario001-standalone-master - passed | 13:18 |
| marios | rlandy: but still means disruption | 13:18 |
| rlandy | and | 13:18 |
| rlandy | periodic-tripleo-ci-rhel-8-scenario001-standalone-train | 13:18 |
| marios | rlandy: nice | 13:18 |
| marios | and nice | 13:19 |
| rlandy | so we got one bug down | 13:19 |
| weshay | k.. | 13:19 |
| rlandy | that is the fix we merged | 13:19 |
| * weshay says hi to my family for a bit | 13:19 | |
| rlandy | now we have fs001 bugs | 13:19 |
| weshay | ah NICE on rhel 8 :) | 13:19 |
| rlandy | marios: ack we can | 13:19 |
| marios | rlandy: yeah i closed the train bug out earlier https://bugs.launchpad.net/tripleo/+bug/1853978/comments/27 based on the 18th green run | 13:19 |
| openstack | Launchpad bug 1853978 in tripleo "periodic train rhel8 ovb overcloud deployment failed with Could not find class ::tripleo::profile::base::neutron::ovn_metadata_agent_wrappers" [Critical,Fix released] | 13:19 |
| rlandy | cool | 13:19 |
| rlandy | did we move the CIX? | 13:19 |
| marios | rlandy: oh fs1 master excellent cos https://bugs.launchpad.net/tripleo/+bug/1853652/comments/34 | 13:20 |
| openstack | Launchpad bug 1853652 in tripleo "openstack overcloud node provide --all-manageable timing out and failing periodic rhel-8-ovb-3ctlr_1comp-featureset001-master" [Critical,In progress] - Assigned to Cédric Jeanneret (cjeanner) | 13:20 |
| rlandy | marios: no, no | 13:20 |
| rlandy | https://bugs.launchpad.net/tripleo/+bug/1856278 | 13:20 |
| openstack | Launchpad bug 1856278 in tripleo "RHEL8 scenario 1 standalone deployment failed with The following containers failed validations and were not started: collectd" for master and train" [Critical,Fix released] - Assigned to Cédric Jeanneret (cjeanner) | 13:20 |
| marios | rlandy: so looks like the very latest run included the selinux fixes | 13:20 |
| rlandy | marios: fs001 master and train are still failing | 13:21 |
| rlandy | this https://trello.com/c/r8g1gG41/1270-cixlp1856278tripleociproa-rhel8-scenario-1-standalone-deployment-failed-with-the-following-containers-failed-validations-and-wer is fixed | 13:21 |
| marios | rlandy: ah scen1 sorry mixed them ack with you now | 13:21 |
| rlandy | ok - moving that card to done | 13:21 |
| rlandy | then let's look at the two fs001 | 13:21 |
| marios | rlandy: i even commented there earlier https://bugs.launchpad.net/tripleo/+bug/1856278/comments/14 | 13:22 |
| openstack | Launchpad bug 1856278 in tripleo "RHEL8 scenario 1 standalone deployment failed with The following containers failed validations and were not started: collectd" for master and train" [Critical,Fix released] - Assigned to Cédric Jeanneret (cjeanner) | 13:22 |
| marios | rlandy: (standalone i mean ) | 13:22 |
| marios | rlandy: so fs1 still failing on timeout | 13:22 |
| marios | rlandy: i commented there https://bugs.launchpad.net/tripleo/+bug/1853652/comments/34 | 13:22 |
| openstack | Launchpad bug 1853652 in tripleo "openstack overcloud node provide --all-manageable timing out and failing periodic rhel-8-ovb-3ctlr_1comp-featureset001-master" [Critical,In progress] - Assigned to Cédric Jeanneret (cjeanner) | 13:22 |
| marios | rlandy: still neutron dhcp agent issue but not selinux this time cos definitely permissive | 13:22 |
| marios | rlandy: so we need to ping back to slaweq/neutron folks but | 13:22 |
| marios | holidays | 13:22 |
| rlandy | marios: I'm confused :) | 13:22 |
| rfolco | rlandy, please https://review.rdoproject.org/r/24284 | 13:23 |
| rlandy | did I move the wrong card? | 13:23 |
| marios | rlandy: periodic rhel-8-ovb-3ctlr_1comp-featureset001-master | 13:23 |
| rlandy | rfolco: done | 13:23 |
| marios | rlandy: want to bluejeans? | 13:23 |
| rlandy | marios: I closed the one about scenario001 | 13:23 |
| marios | rlandy: yes right | 13:23 |
| marios | rlandy: scen1 done | 13:23 |
| marios | rlandy: now i am talking about fs1 | 13:23 |
| marios | 15:21 < rlandy> then let's look at the two fs001 | 13:24 |
| rlandy | marios: yep - let's chat on meet | 13:24 |
| marios | 15:22 < marios> rlandy: so fs1 still failing on timeout | 13:24 |
| marios | 15:22 < marios> rlandy: i commented there https://bugs.launchpad.net/tripleo/+bug/1853652/comments/34 | 13:24 |
| openstack | Launchpad bug 1853652 in tripleo "openstack overcloud node provide --all-manageable timing out and failing periodic rhel-8-ovb-3ctlr_1comp-featureset001-master" [Critical,In progress] - Assigned to Cédric Jeanneret (cjeanner) | 13:24 |
| marios | rlandy: k plugging headphones | 13:24 |
| marios | impromptu scrum welcome all https://meet.google.com/evp-nyur-dhb | 13:24 |
| marios | rlandy: ^ | 13:24 |
| rlandy | joined | 13:25 |
| *** amoralej is now known as amoralej|lunch | 13:35 | |
| *** jpena|lunch is now known as jpena | 13:36 | |
| marios | 2019-12-20 13:26:41,621 3382 INFO promoter Promoting the container images for dlrn hash 80cc8ed4b1d2e10e7c4a518dd4ca1a5305ed8ede on train to current-tripleo-rdo | 13:38 |
| marios | https://bugs.launchpad.net/tripleo/+bug/1853652/comments/34 | 13:42 |
| openstack | Launchpad bug 1853652 in tripleo "openstack overcloud node provide --all-manageable timing out and failing periodic rhel-8-ovb-3ctlr_1comp-featureset001-master" [Critical,In progress] - Assigned to Cédric Jeanneret (cjeanner) | 13:42 |
| *** dsneddon has joined #oooq | 13:43 | |
| *** ykarel has quit IRC | 13:48 | |
| weshay | back | 13:54 |
| weshay | sorry .. so tired.. got it at 2am last night | 13:55 |
| weshay | anything I can help w/? | 13:56 |
| marios | https://bugs.launchpad.net/tripleo/+bug/1853978/comments/27 | 13:56 |
| openstack | Launchpad bug 1853978 in tripleo "periodic train rhel8 ovb overcloud deployment failed with Could not find class ::tripleo::profile::base::neutron::ovn_metadata_agent_wrappers" [Critical,Fix released] | 13:56 |
| weshay | ah ya | 13:57 |
| * weshay put up a puppet-tripleo review.. and then I'll work the release | 13:57 | |
| marios | weshay: sorry that was for rlandy | 13:57 |
| rlandy | weshay: train rhel 8 is still legit failure | 13:57 |
| marios | weshay: not asking for help | 13:57 |
| weshay | rlandy, marios ok.. so this merged https://review.opendev.org/#/c/700045/ | 13:58 |
| weshay | marios, oh.. I thought train needed a release for rdo for fixing that | 13:58 |
| rlandy | weshay: so looking at latest run | 13:59 |
| marios | weshay: perhaps i am really not sure | 13:59 |
| marios | weshay: rlandy: we are looking at latest i think its a new issue but we aren't sure ... :/ | 13:59 |
| rlandy | http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-train/253a697/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 13:59 |
| rlandy | latest logs | 13:59 |
| *** skramaja has quit IRC | 14:00 | |
| marios | weshay: btw we don't know how but the promoter is promoting all the things at the same time | 14:00 |
| marios | promoter on steroids | 14:00 |
| rlandy | weshay: the promoter is outsourcing | 14:00 |
| marios | weshay: we decided to let it finish and then do a reboot/clear all the services (might be duplicate services) | 14:01 |
| weshay | ya.. that's been going on for the last couple weeks | 14:01 |
| weshay | marios, rlandy has anyone tried killing all the python processes, stopping the service and restarting | 14:01 |
| weshay | ? | 14:01 |
| rlandy | weshay: no - because it's actually promoting | 14:01 |
| marios | weshay: we don't want to interrupt it. cos it is like a christmas tree its promoting EVEERYTHING | 14:01 |
| rlandy | once it stops | 14:01 |
| weshay | lolz | 14:01 |
| weshay | like right now? | 14:02 |
| marios | weshay: whole day | 14:02 |
| weshay | lolz | 14:02 |
| weshay | k | 14:02 |
| marios | weshay: we've had master queens rocky already | 14:02 |
| marios | train happening | 14:02 |
| marios | #happening | 14:02 |
| rlandy | on both phases | 14:02 |
| rlandy | and let us say amen | 14:02 |
| weshay | #happy_xmas_from_your_local_promoter | 14:02 |
| weshay | from our promoter to g-d's ears? | 14:03 |
| weshay | ok.. I'll work on it later | 14:03 |
| marios | weshay: so yeah we want to reboot that at some point and maybe add some more storage | 14:03 |
| rlandy | oprah special ... you get a promotion, and you get a promotion, you all get a promotion | 14:03 |
| marios | weshay: but maybe very late in your day | 14:04 |
| weshay | lolz | 14:04 |
| weshay | rlandy++ | 14:04 |
| weshay | marios, not as late as yours | 14:04 |
| marios | oprah: "promotioooooooooooons " | 14:04 |
| marios | weshay: i mean 'later on in your day before you leave maybe please reboot the promoter" | 14:05 |
| rlandy | weshay: marios: ^^ ack in our afternoon | 14:05 |
| weshay | aye.. not sure if I'll reboot, but I'll def kill everything and restart the service | 14:05 |
| marios | rlandy: yes "once it stops" | 14:05 |
| marios | ;) | 14:05 |
| weshay | it's not windoze | 14:05 |
| rlandy | maybe we should just stop it now | 14:05 |
| rlandy | take the hit | 14:06 |
| *** dsneddon has quit IRC | 14:06 | |
| rlandy | it keeps running out of space | 14:06 |
| weshay | oh | 14:06 |
| weshay | that's not great | 14:06 |
| * weshay checks it out | 14:06 | |
| *** dsneddon has joined #oooq | 14:06 | |
| rlandy | weshay: see docker image ls | 14:06 |
| rlandy | you will see two sets of containers | 14:07 |
| rlandy | and ps -u cenos | 14:07 |
| rlandy | centos | 14:07 |
| weshay | disk space is ok right now | 14:07 |
| rlandy | will show you two ansible-playbook processes | 14:07 |
| rlandy | weshay: ack because we cleaned up | 14:07 |
| rlandy | it was at 80% | 14:07 |
| rlandy | it's parallel processing | 14:07 |
| rlandy | which could be a neat trick or a disaster | 14:07 |
| weshay | ya.. not good.. but I think what happens is that a new push starts and kills the running one | 14:09 |
| weshay | it looks like both are running in ps | 14:09 |
| weshay | but it's only one | 14:09 |
| weshay | not sure | 14:10 |
| weshay | I saw this happen before I left for nyc | 14:10 |
| *** d0ugal has quit IRC | 14:10 | |
| weshay | wow.. 95% pass rate upstream | 14:11 |
| *** dsneddon has quit IRC | 14:11 | |
| rlandy | 2019-12-20 13:26:41,621 3382 INFO | 14:11 |
| rlandy | so the train run just started | 14:11 |
| weshay | oh ha.. that my local dev env | 14:11 |
| *** dsneddon has joined #oooq | 14:12 | |
| weshay | k.. /me is on the tmux | 14:13 |
| *** d0ugal has joined #oooq | 14:13 | |
| rlandy | marios: kicked a rerun on rhel8 train | 14:14 |
| rlandy | weshay: you on the promoter? | 14:15 |
| rlandy | weshay: maybe we should just kill and restart | 14:16 |
| rlandy | and clean up these containers images | 14:16 |
| marios | rlandy: ack thanks | 14:16 |
| weshay | rlandy, ya | 14:16 |
| *** dsneddon has quit IRC | 14:17 | |
| *** SurajPatil has quit IRC | 14:17 | |
| rlandy | 2019-12-20 10:06:08,242 7877 | 14:17 |
| rlandy | queens one has been going for ages | 14:17 |
| rlandy | weshay: sorry ya to which part | 14:17 |
| weshay | I'm on the promoter | 14:18 |
| weshay | I'm sorry.. very sluggish | 14:18 |
| weshay | marios, rlandy sshnaidm|afk fyi.. I booked my flight etc | 14:22 |
| marios | weshay: fyi _manifests looks good updated that https://tree.taiga.io/project/tripleo-ci-board/task/1315 with some logs moving to ready-for-review though we're not merging anything there | 14:22 |
| weshay | k | 14:22 |
| weshay | waiting on the pandalorian | 14:22 |
| marios | weshay: looks like it deploys the _manifest 'containers' fine | 14:23 |
| marios | weshay: even though they're manifests | 14:23 |
| marios | weshay: ack on flight i will likely do that next week | 14:23 |
| weshay | very cool | 14:23 |
| rlandy | weshay: yeah - got to get on that | 14:23 |
| weshay | marios, I'll check out those test jobs in a bit | 14:23 |
| *** TrevorV has joined #oooq | 14:24 | |
| marios | weshay: ack if you mean _manifests would be nice to have a sanity check i added pointers to logs there in taiga. Once we switch to the new code it will be easier we can also test on master too for example | 14:24 |
| weshay | very cool.. thanks for pushing on that | 14:25 |
| marios | weshay: ack nice to finally close that card out | 14:26 |
| rlandy | weshay: on your tmux | 14:26 |
| *** epoojad1 has joined #oooq | 14:26 | |
| rlandy | i can't operate with these multiple windows :( | 14:27 |
| weshay | marios, rlandy ok.. usually tox fails but.. https://review.opendev.org/700179 | 14:29 |
| rlandy | and now stein is also promoting | 14:29 |
| rlandy | weshay: thanks | 14:30 |
| rlandy | weshay: pls join me on https://meet.google.com/xnz-rpyd-tfq | 14:31 |
| rlandy | marios: ^^ feel free to join us if you want | 14:31 |
| marios | rlandy: sure plugging in | 14:32 |
| *** amoralej|lunch is now known as amoralej | 14:39 | |
| *** dsneddon has joined #oooq | 14:46 | |
| *** ccamacho has quit IRC | 14:57 | |
| *** dsneddon has quit IRC | 15:11 | |
| *** EmilienM is now known as EvilienM | 15:16 | |
| *** epoojad1 has quit IRC | 15:25 | |
| *** dsneddon has joined #oooq | 15:47 | |
| *** derekh has quit IRC | 15:51 | |
| weshay | rlandy, the component pipeline is very green | 15:53 |
| *** dsneddon has quit IRC | 15:54 | |
| rlandy | weshay: downstream or upstream? | 15:54 |
| weshay | rlandy, I only have eyes on upstream atm | 15:54 |
| marios | weshay: rlandy: getting ready to go need something before i do? | 15:55 |
| rlandy | weshay: ok - so downstream standalone compute fails tempest | 15:55 |
| rlandy | mario: nah - have some time off | 15:56 |
| rlandy | marios: ^^ | 15:56 |
| marios | rlandy: ack around monday anyway will take time after that :D | 15:56 |
| weshay | rlandy, ya.. let migi handle that :) | 15:56 |
| rlandy | weshay: wrt upstream, it should track standalone master | 15:56 |
| rlandy | which is also very green | 15:56 |
| rlandy | weshay: the downstream failure is legit | 15:57 |
| rlandy | osp17 component with rhos-16 product base | 15:57 |
| weshay | aye.. | 15:59 |
| *** marios is now known as marios|out | 15:59 | |
| rlandy | weshay: interestingly enough the mix keystone job works | 16:05 |
| rlandy | train promotion completed | 16:08 |
| rlandy | stein in progress | 16:09 |
| rlandy | weshay: ^^ cleaning up train container images | 16:10 |
| rlandy | dev/vdb1 80G 50G 31G 62% /var/lib/docker | 16:10 |
| weshay | k | 16:14 |
| *** epoojad1 has joined #oooq | 16:15 | |
| *** ykarel has joined #oooq | 16:21 | |
| *** dsneddon has joined #oooq | 16:24 | |
| rlandy | dev/vdb1 80G 36G 45G 45% /var/lib/docker | 16:27 |
| rlandy | better | 16:27 |
| rlandy | images area just stein and queens now | 16:27 |
| rlandy | 2019-12-20 10:36:42 | ResourceInError: resources.Controller: Went to status ERROR due to "Message: No valid host was found. , Code: 500" | 16:37 |
| rlandy | shoot | 16:37 |
| rlandy | tenant cleanup may be needed | 16:38 |
| *** dsneddon has quit IRC | 16:38 | |
| *** Goneri has quit IRC | 16:41 | |
| *** marios|out has quit IRC | 16:45 | |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-centos-7-bm_envA-3ctlr_1comp-featureset001-master | 16:48 |
| rlandy | wow = can see how many times master promoted from BM logs | 16:48 |
| *** dsneddon has joined #oooq | 17:00 | |
| *** Trevor_V has joined #oooq | 17:00 | |
| *** TrevorV has quit IRC | 17:04 | |
| rfolco | rlandy, https://review.rdoproject.org/r/#/c/24287/ - tested this one now locally, with / it works as expected | 17:04 |
| weshay | rlandy, ya.. lolz | 17:15 |
| weshay | rlandy, I asked Sagi to stand up an internal cockpit on some hardware I gave him | 17:15 |
| rlandy | weshay: thanks | 17:16 |
| rlandy | done | 17:16 |
| weshay | rlandy, rfolco check it out :) https://trunk.rdoproject.org/centos8-master/component/ | 17:16 |
| rfolco | cool | 17:16 |
| rlandy | ok | 17:17 |
| weshay | rfolco, rlandy as we're standing up centos-8 next year.. we'll have to chat about if it makes sense to build out all the component jobs and have them feed the integration jobs | 17:18 |
| rlandy | we're ready for that | 17:18 |
| rlandy | weshay: who has +2 on https://review.opendev.org/#/c/700179? | 17:19 |
| rlandy | periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-train still fails | 17:20 |
| rlandy | periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master - that is the job to watch now | 17:20 |
| rlandy | weshay: if cockpit is internal - where should the logs post? | 17:21 |
| *** holser has quit IRC | 17:21 | |
| weshay | rlandy, the folks in #openstack-release | 17:22 |
| weshay | rlandy, want to chat about that for 5? | 17:22 |
| rlandy | ok | 17:22 |
| weshay | https://meet.google.com/iiq-zhpf-msq | 17:23 |
| *** ykarel is now known as ykarel|away | 17:33 | |
| *** Goneri has joined #oooq | 17:37 | |
| weshay | rlandy, https://softwarefactory-project.io/zuul/api/tenant/rdoproject.org/builds | 17:37 |
| weshay | rlandy, https://sf.hosted.upshift.rdu2.redhat.com/zuul/api/tenant/tripleo-ci-internal/builds | 17:43 |
| *** ykarel|away has quit IRC | 17:45 | |
| weshay | rlandy, just to document the path https://review.rdoproject.org/r/#/c/24291/1/ci-scripts/infra-setup/roles/rrcockpit/files/telegraf/telegraf.d/zuulv3_job_builds.conf | 17:48 |
| rlandy | ack | 17:49 |
| *** jpena is now known as jpena|off | 17:51 | |
| *** jpena|off has quit IRC | 17:52 | |
| *** amoralej has quit IRC | 17:57 | |
| rlandy | great - stein done | 17:57 |
| rlandy | promoter is looking healthy again | 17:58 |
| weshay | rfolco, https://review.rdoproject.org/zuul/builds?pipeline=github-check | 18:05 |
| weshay | https://review.rdoproject.org/zuul/builds?pipeline=github-manual | 18:06 |
| weshay | rfolco, saying we closed it out.. when we don't yet have the job that runs the tests done is probably not as accurate as we want to be :) | 18:06 |
| *** jfrancoa has quit IRC | 18:07 | |
| rfolco | weshay, ok, almost closed it out, the poc I meant | 18:12 |
| rfolco | ok | 18:12 |
| weshay | rfolco, ya.. I just don't have a good handle on the actual check jobs yet | 18:13 |
| weshay | how much more work we have there | 18:13 |
| weshay | 3rd party deps was a hot topic at the manager meetup | 18:13 |
| rfolco | for the poc we just need to fix the job that promotes, I fix one issue, another shows up | 18:13 |
| weshay | it's really nice we have this stuff going | 18:14 |
| rfolco | latest issue - http://logs.rdoproject.org/99/24199/5/check/periodic-tripleo-centos-7-master-component-compute-promote-to-current-tripleo/4bf6b95/job-output.txt | 18:14 |
| weshay | rfolco, ? | 18:14 |
| weshay | rfolco, we're not talking about components | 18:14 |
| weshay | we're talking deps | 18:14 |
| weshay | podman, ceph-ansible | 18:14 |
| rfolco | ah | 18:15 |
| rfolco | sorry | 18:15 |
| rfolco | well, need to rephrase that then | 18:16 |
| *** saneax has quit IRC | 18:20 | |
| *** rfolco is now known as rfolco|dentist | 18:21 | |
| rlandy | weshay: looked into the gate failures | 18:37 |
| rlandy | we are running at 91% pass | 18:37 |
| rlandy | but it record 15 failures | 18:37 |
| weshay | ya.. for check and gate | 18:37 |
| weshay | aye | 18:37 |
| rlandy | only one though | 18:37 |
| rlandy | in the last couple hours | 18:37 |
| rlandy | so I think it's ok | 18:37 |
| weshay | :) | 18:39 |
| weshay | thanks for checking | 18:39 |
| *** hamzy_ has quit IRC | 18:42 | |
| *** tosky has quit IRC | 18:45 | |
| rlandy | still out n rhel fs001 | 19:01 |
| rlandy | 2019-12-20 13:47:31 | ResourceInError: resources.Controller: Went to status ERROR due to "Message: No valid host was found. , Code: 500" | 19:01 |
| rlandy | ugh | 19:01 |
| rlandy | can't get enough resources to run these tests | 19:01 |
| rlandy | weshay: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master | 19:03 |
| rlandy | ^^ has not passed since 12/11 | 19:03 |
| rlandy | maybe even before | 19:04 |
| rlandy | should we try promote w/o it? | 19:04 |
| rlandy | checking cockpit stats | 19:04 |
| *** hamzy has joined #oooq | 19:07 | |
| rlandy | since 12/07 | 19:13 |
| *** brault has quit IRC | 19:17 | |
| rlandy | No such file or directory: '/var/log/pcsd/pcsd.log' | 19:35 |
| rlandy | we have a pacemaker issue | 19:36 |
| *** rfolco|dentist has quit IRC | 19:48 | |
| *** rfolco has joined #oooq | 19:49 | |
| weshay | rlandy, hrm that master job? | 19:58 |
| rlandy | weshay: yes | 19:59 |
| rlandy | not the latest run | 19:59 |
| rlandy | that one dies on resources | 19:59 |
| rlandy | I am rerunning with https://review.opendev.org/#/c/699318/ | 19:59 |
| rlandy | see if it helps | 19:59 |
| weshay | k.. thanks.. I guess that job is busted anyway | 20:01 |
| rlandy | rocky promoted | 20:03 |
| rlandy | and queens is up | 20:03 |
| rlandy | getting there | 20:03 |
| rlandy | weshay: finally http://rhos-release.virt.bos.redhat.com:3030/rhosp | 20:20 |
| rlandy | one round of mass promoting done | 20:20 |
| *** rlandy has quit IRC | 21:13 | |
| *** Trevor_V has quit IRC | 21:26 | |
| weshay | nice | 22:25 |
| *** EvilienM is now known as EmilienM | 22:36 | |
| *** dtrainor has quit IRC | 23:32 | |
| *** Goneri has quit IRC | 23:40 | |
| *** dsneddon has quit IRC | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!