| *** aakarsh has joined #oooq | 01:12 | |
| *** rlandy|rover|bbl is now known as rlandy|rover | 01:45 | |
| *** rlandy|rover has quit IRC | 01:50 | |
| *** bhagyashris has joined #oooq | 01:57 | |
| *** vinaykns has quit IRC | 02:08 | |
| *** apetrich has quit IRC | 02:08 | |
| *** gkadam has joined #oooq | 03:31 | |
| *** surpatil has joined #oooq | 03:33 | |
| *** gkadam has quit IRC | 03:40 | |
| *** raukadah is now known as chkumar|ruck | 03:49 | |
| *** udesale has joined #oooq | 04:09 | |
| *** skramaja has joined #oooq | 04:13 | |
| *** aakarsh has quit IRC | 04:43 | |
| *** aakarsh has joined #oooq | 04:44 | |
| *** ykarel|away has joined #oooq | 04:46 | |
| *** ykarel|away is now known as ykarel | 04:51 | |
| *** jdennis has quit IRC | 04:53 | |
| *** jdennis has joined #oooq | 05:04 | |
| *** marios has joined #oooq | 05:11 | |
| *** ykarel has quit IRC | 05:19 | |
| *** ykarel has joined #oooq | 05:20 | |
| *** ratailor has joined #oooq | 05:44 | |
| *** jaosorior has quit IRC | 05:56 | |
| *** ccamacho has quit IRC | 06:03 | |
| *** ykarel is now known as ykarel|afk | 06:04 | |
| *** jaosorior has joined #oooq | 06:10 | |
| *** bogdando has joined #oooq | 06:18 | |
| *** ykarel|afk is now known as ykarel | 06:22 | |
| *** jaosorior has quit IRC | 06:29 | |
| *** jfrancoa has joined #oooq | 06:32 | |
| *** jbadiapa has joined #oooq | 06:33 | |
| *** jfrancoa has quit IRC | 06:36 | |
| *** udesale has quit IRC | 06:41 | |
| *** udesale has joined #oooq | 06:42 | |
| marios | chkumar|ruck: o/ hey do we have bug for the scenario10 standalone "tripleo_octavia_api_healthcheck.service loaded failed failed octavia_api | 06:48 |
|---|---|---|
| marios | chkumar|ruck: https://logs.opendev.org/28/674228/7/check/tripleo-ci-centos-7-scenario010-standalone/e1c1d86/job-output.txt.gz | 06:49 |
| marios | http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-scenario010-standalone | 06:49 |
| *** jfrancoa has joined #oooq | 06:49 | |
| chkumar|ruck | marios: nope | 06:50 |
| marios | chkumar|ruck: ack added note for you in https://etherpad.openstack.org/p/ruckroversprint14 | 06:51 |
| marios | chkumar|ruck: hows it going :D... are you settling in for the next 6 weeks ruck? | 06:53 |
| * marios runs | 06:53 | |
| chkumar|ruck | marios: I will try, let me complete first 3 weeks, then we can think about another 3 weeks | 06:54 |
| chkumar|ruck | baby steps. | 06:54 |
| marios | chkumar|ruck: :D | 06:54 |
| *** ykarel is now known as ykarel|pto | 07:00 | |
| *** jtomasek has joined #oooq | 07:01 | |
| *** udesale has quit IRC | 07:03 | |
| *** ccamacho has joined #oooq | 07:03 | |
| *** udesale has joined #oooq | 07:03 | |
| *** ykarel|pto has quit IRC | 07:07 | |
| *** amoralej|off is now known as amoralej | 07:12 | |
| *** jaosorior has joined #oooq | 07:16 | |
| *** tosky has joined #oooq | 07:17 | |
| *** suraj_ has joined #oooq | 07:24 | |
| *** surpatil has quit IRC | 07:27 | |
| *** surpatil has joined #oooq | 07:29 | |
| *** tesseract has joined #oooq | 07:31 | |
| *** suraj_ has quit IRC | 07:32 | |
| *** suraj_ has joined #oooq | 07:35 | |
| *** panda has quit IRC | 07:35 | |
| *** surpatil has quit IRC | 07:36 | |
| *** suraj__ has joined #oooq | 07:37 | |
| chkumar|ruck | marios: Do we have a healthcheck logs somewhere? | 07:38 |
| *** panda has joined #oooq | 07:38 | |
| *** suraj_ has quit IRC | 07:38 | |
| *** surpatil has joined #oooq | 07:41 | |
| marios | chkumar|ruck: dont know | 07:42 |
| *** suraj__ has quit IRC | 07:42 | |
| *** dtantsur|afk is now known as dtantsur | 07:43 | |
| *** apetrich has joined #oooq | 07:44 | |
| marios | chkumar|ruck: poking aroudn i can't find something https://logs.opendev.org/28/674228/7/check/tripleo-ci-centos-7-scenario010-standalone/e1c1d86/logs/undercloud/home/ | 07:45 |
| *** surpatil has quit IRC | 07:46 | |
| *** surpatil has joined #oooq | 07:49 | |
| *** suraj_ has joined #oooq | 07:54 | |
| *** surpatil has quit IRC | 07:57 | |
| *** udesale has quit IRC | 07:57 | |
| *** udesale has joined #oooq | 07:58 | |
| *** kopecmartin|off is now known as kopecmartin | 07:59 | |
| *** brault has joined #oooq | 07:59 | |
| chkumar|ruck | marios: fs10 on each failure, we have different issue] | 08:06 |
| chkumar|ruck | marios: https://review.opendev.org/#/c/675263/ will fix scenario 10 | 08:17 |
| marios | chkumar|ruck: thanks will check in a bit | 08:26 |
| marios | chkumar|ruck: ack lets see how scenario10 does there and will vote thanks | 08:27 |
| *** derekh has joined #oooq | 08:30 | |
| *** ksambor has quit IRC | 08:36 | |
| sshnaidm | marios, https://review.opendev.org/#/c/675225/ - it fixes rhel8 buildimages jobs in check | 08:36 |
| *** jbadiapa has quit IRC | 08:36 | |
| sshnaidm | marios, I remove then "buildimages" playbooks from config repo, they shouldn't be there | 08:37 |
| sshnaidm | panda, ^^ | 08:37 |
| marios | sshnaidm: ack | 08:39 |
| *** ksambor has joined #oooq | 08:39 | |
| sshnaidm | zbr, ping | 08:44 |
| zbr | morning | 08:44 |
| sshnaidm | zbr, can you please check something wrt warnings in ansible? | 08:48 |
| zbr | sure. send me link | 08:48 |
| sshnaidm | zbr, I'm trying to suppress this warning: https://logs.opendev.org/19/667219/17/check/tripleo-ci-centos-7-standalone/c299907/job-output.txt.gz#_2019-08-08_00_28_57_553535 | 08:49 |
| sshnaidm | zbr, in patch https://review.opendev.org/675223 when using it with ansible 2.8 here: https://review.opendev.org/#/c/667219/ | 08:49 |
| sshnaidm | zbr, but seems like it didn't work | 08:49 |
| sshnaidm | zbr, maybe it's possible to disable warnings on playbook level? | 08:49 |
| sshnaidm | zbr, we run collect-logs with -vvvv | 08:50 |
| zbr | even if possible, i would do my best not do disable them, fixing them instead. | 08:51 |
| zbr | now I do not understand the nature/source of that warning | 08:51 |
| zbr | note sure where this string comes from but my guess is that we need !unsafe in that place. see https://docs.ansible.com/ansible/latest/user_guide/playbooks_advanced_syntax.html#unsafe-or-raw-strings | 08:54 |
| zbr | where does this comes from? | 08:55 |
| zbr | does it come from one of the loaded yaml file? | 08:55 |
| sshnaidm | zbr, no, it's from shell output: https://github.com/openstack/ansible-role-collect-logs/blob/master/tasks/ara_influxdb.yml#L3 | 08:58 |
| zbr | sshnaidm: well... doing that is bit insane due to its spam potential. | 09:00 |
| sshnaidm | zbr, doing what? | 09:00 |
| zbr | i would not be surprised to see this crashing ansible in some cases. | 09:00 |
| sshnaidm | zbr, I have no idea why it should crash ansible, works like a charm a few years | 09:01 |
| sshnaidm | zbr, can you help disable this warning? | 09:01 |
| sshnaidm | zbr, ok, nvrmind | 09:07 |
| zbr | i am still looking | 09:07 |
| zbr | sshnaidm: add a "| string" when you pass the parameter to ara module. | 09:08 |
| zbr | not sure why it is needed because it is supposed to be a string already. | 09:08 |
| *** skramaja has quit IRC | 09:10 | |
| zbr | this ara_influxdb could have being written better by allowing it to read the ara_data from a file instead of only allowing passing data as variable. we could avoid involving ansible loading and sending what could be a huge amount of data. | 09:12 |
| marios | afk few mins errand | 09:12 |
| zbr | like a param ara_data_file param | 09:12 |
| *** bhagyashris has quit IRC | 09:44 | |
| *** dmellado has quit IRC | 09:53 | |
| *** dmellado has joined #oooq | 09:55 | |
| *** dsneddon has quit IRC | 09:55 | |
| *** suraj_ has quit IRC | 09:57 | |
| *** surpatil has joined #oooq | 09:58 | |
| *** udesale has quit IRC | 10:01 | |
| *** udesale has joined #oooq | 10:01 | |
| *** udesale has quit IRC | 10:02 | |
| *** udesale has joined #oooq | 10:03 | |
| *** jaosorior has quit IRC | 10:26 | |
| *** udesale has quit IRC | 10:38 | |
| chkumar|ruck | sshnaidm: Since kolla broke RHEL-8 container build twice | 10:43 |
| chkumar|ruck | sshnaidm: do we want to gate it with RHEL-8 third party container build job> | 10:44 |
| chkumar|ruck | ? | 10:44 |
| sshnaidm | chkumar|ruck, yeah, when it'll start to work | 10:46 |
| sshnaidm | chkumar|ruck, is it the same container problem - collectd? | 10:46 |
| chkumar|ruck | sshnaidm: yes | 10:47 |
| sshnaidm | chkumar|ruck, but is it possible to skip it completely? | 10:47 |
| chkumar|ruck | sshnaidm: not sure, It is used in deployment I think | 10:47 |
| sshnaidm | chkumar|ruck, I don't think it's important part tbh | 10:47 |
| chkumar|ruck | sshnaidm: but because of RHEL8 job, we discovered this issue and the good part is that package was not in RHEL-8 | 10:48 |
| chkumar|ruck | it is finaly started paying off | 10:48 |
| sshnaidm | chkumar|ruck, that's right | 10:49 |
| sshnaidm | weshay, ptal https://review.opendev.org/#/c/675225/ | 10:51 |
| chkumar|ruck | sshnaidm: we need this https://review.rdoproject.org/r/#/c/21771/ in order to apply tripleo=common patch | 11:07 |
| chkumar|ruck | in container push job | 11:07 |
| *** jaosorior has joined #oooq | 11:10 | |
| sshnaidm | chkumar|ruck, it's already here: https://github.com/openstack/tripleo-ci/blob/master/zuul.d/base.yaml#L198-L204 | 11:11 |
| marios | sshnaidm: did sthing change in grafana if you know? getting strange invalid api key trying to export sthing | 11:37 |
| sshnaidm | marios, sthing.. | 11:39 |
| sshnaidm | ? | 11:39 |
| marios | sshnaidm: sec sorry might be pebcak (wondering if something changed.. sthing == something) | 11:39 |
| sshnaidm | marios, pebcak..? is it sthing in greek? :) | 11:40 |
| marios | sshnaidm: haha pebcak == Problem Exists Between Keyboard and Chair (user error) | 11:40 |
| sshnaidm | marios, so sthing is pebkac mb hpnd | 11:41 |
| marios | sshnaidm: right | 11:41 |
| marios | sshnaidm: anyway , i tried creating new key but still cant export. if nothing changed maybe i'm doing something wrong (tried http/https nop) | 11:41 |
| sshnaidm | marios, you don't need the new key, if you use the same setup | 11:42 |
| sshnaidm | marios, only if you redeployed ething from scratch | 11:42 |
| marios | sshnaidm: hmm so you mean if i deleted that key i need to redeploy it | 11:42 |
| marios | development -c clean | 11:43 |
| marios | etc | 11:43 |
| sshnaidm | marios, did you delete it? | 11:43 |
| marios | sshnaidm: yeah | 11:43 |
| marios | sshnaidm: it has worked for me before though, to recreate one and re-export | 11:43 |
| sshnaidm | marios, well, then creating a new one should work | 11:43 |
| marios | sshnaidm: right. | 11:43 |
| sshnaidm | marios, maybe different name? | 11:43 |
| marios | sshnaidm: k let me see what else it might be | 11:43 |
| marios | sshnaidm: thx will try | 11:43 |
| sshnaidm | marios, you don't need to delete it | 11:44 |
| marios | sshnaidm: hah new name worked thanks :D | 11:44 |
| sshnaidm | marios, np | 11:44 |
| sshnaidm | marios, gtwfy | 11:44 |
| marios | sshnaidm: ackl lgtm | 11:45 |
| sshnaidm | marios, npagin | 11:45 |
| weshay | pebcak! | 11:49 |
| *** dsneddon has joined #oooq | 11:51 | |
| chkumar|ruck | sshnaidm: How can I replace this default branch with https://github.com/openstack/tripleo-ci/blob/master/roles/build-containers/tasks/main.yaml#L74 with branch originaly checked out by zuul? | 11:52 |
| weshay | marios sshnaidm has rfolco popped up today? | 11:58 |
| marios | weshay: haven't seen him | 11:58 |
| sshnaidm | chkumar|ruck, not sure I understand.. | 12:01 |
| *** ratailor has quit IRC | 12:02 | |
| sshnaidm | chkumar|ruck, branch_override is actual branch defined in zuul job vars, if not defined - use master | 12:02 |
| *** amoralej is now known as amoralej|lunch | 12:02 | |
| chkumar|ruck | sshnaidm: I want to replace this https://github.com/openstack/tripleo-ci/blob/master/roles/build-containers/tasks/main.yaml#L74 branch_override with ref coming zuul inventory otherwise it will not work in kolla check jobs | 12:05 |
| chkumar|ruck | I think I got the solution for the same | 12:05 |
| *** akahat has joined #oooq | 12:07 | |
| chkumar|ruck | brb for 30 mins | 12:11 |
| *** rfolco has joined #oooq | 12:12 | |
| weshay | chkumar|ruck oof.. was there an infra problem or bad patch /me looking at the gate jobs | 12:16 |
| marios | weshay: scen10 blocking som stuff, don't know what else | 12:16 |
| weshay | marios thansks... chkumar|ruck remove tripleo-ci-centos-7-scenario009-multinode-oooq-containerfailure (non-voting) from any gate queues please | 12:16 |
| marios | weshay: octavia health check https://bugs.launchpad.net/tripleo/+bug/1839436 | 12:17 |
| openstack | Launchpad bug 1839436 in tripleo "octavia healthcheck fails in scenario010" [Critical,In progress] - Assigned to Rabi Mishra (rabi) | 12:17 |
| *** jfrancoa has quit IRC | 12:17 | |
| sshnaidm | weshay, there are problems with trunk.rdoproject.org that failed a lot of gates | 12:18 |
| weshay | sshnaidm w/ the server itself or proxies? | 12:18 |
| sshnaidm | weshay, server | 12:18 |
| weshay | chkumar|ruck I see bugs for octavia, and collectd | 12:19 |
| weshay | 1839423 | 12:19 |
| weshay | link to bug | 12:19 |
| weshay | Triaged | 12:19 |
| weshay | promotion-blocker rdo | 12:19 |
| weshay | "'[RHEL-8][container-build]No match for argument: collectd-sensubility'" | 12:19 |
| weshay | 1839436In Progressalert promotion-blocker"'octavia healthcheck fails in scenario010'" | 12:19 |
| weshay | sshnaidm is chkumar|ruck aware of the issues w/ dlrn? | 12:20 |
| sshnaidm | weshay, yes | 12:20 |
| weshay | k | 12:20 |
| weshay | thanks | 12:20 |
| weshay | rfolco thanks for dealing w/ power issues... /me notes I'll have to drop half way through the planning mtg to demo some prod chain work | 12:21 |
| rfolco | weshay, ok, I just created the agenda, feel free to own it | 12:21 |
| rfolco | weshay, https://etherpad.openstack.org/p/tripleo-ci-squad-meeting @L53 | 12:22 |
| *** jaosorior has quit IRC | 12:22 | |
| *** ccamacho has quit IRC | 12:24 | |
| *** rlandy has joined #oooq | 12:28 | |
| *** udesale has joined #oooq | 12:28 | |
| *** rlandy is now known as rlandy|rover | 12:28 | |
| chkumar|ruck | rlandy|rover: Good morning | 12:29 |
| chkumar|ruck | rlandy|rover: want to sync? | 12:29 |
| rlandy|rover | chkumar|ruck: hey - sure | 12:29 |
| chkumar|ruck | https://redhat.bluejeans.com/u/chkumar/ | 12:30 |
| chkumar|ruck | rlandy|rover: ^^ | 12:31 |
| chkumar|ruck | weshay: ^^ do you want to join, I have something for you | 12:31 |
| weshay | rlandy|rover we hear you | 12:34 |
| weshay | rlandy|rover can you hear us? | 12:35 |
| *** amoralej|lunch is now known as amoralej | 12:47 | |
| sshnaidm | weshay, rlandy|rover chkumar|ruck I think we're good to go: https://review.opendev.org/#/c/667219/ | 12:52 |
| weshay | sshnaidm fyi 2019-08-08 08:46:50.557365 | primary | ImportError: No module named keystoneauth1.identity | 12:59 |
| rfolco | ping planning mtg | 13:00 |
| rfolco | marios, sshnaidm, weshay, panda, rlandy, arxcruz, rfolco, chandankumar, zbr, kopecmartin | 13:00 |
| *** bogdando has quit IRC | 13:01 | |
| weshay | zbr please join | 13:03 |
| weshay | kopecmartin can skip | 13:03 |
| zbr | sorry. local emergency. joined. | 13:09 |
| rlandy|rover | chkumar|ruck: https://review.rdoproject.org/r/21773 | 13:19 |
| chkumar|ruck | rlandy|rover: https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/dlrn-promoter-service.sh#L23 | 13:22 |
| *** ccamacho has joined #oooq | 13:23 | |
| *** jfrancoa has joined #oooq | 13:29 | |
| *** bogdando has joined #oooq | 13:29 | |
| chkumar|ruck | rlandy|rover: https://review.opendev.org/#/c/675263/ | 13:31 |
| *** surpatil has quit IRC | 13:45 | |
| *** dsneddon has quit IRC | 13:50 | |
| *** ade_lee_ has quit IRC | 13:53 | |
| chkumar|ruck | rlandy|rover: periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-stein | 13:53 |
| *** vinaykns has joined #oooq | 13:54 | |
| rlandy|rover | chkumar|ruck: https://review.rdoproject.org/r/21774 | 13:56 |
| *** akahat has quit IRC | 14:01 | |
| *** brault has quit IRC | 14:05 | |
| *** jbadiapa has joined #oooq | 14:11 | |
| *** dsneddon has joined #oooq | 14:11 | |
| *** dsneddon has quit IRC | 14:16 | |
| rlandy|rover | chkumar|ruck: woohoo ... 2019-08-08 14:11:31,411 7669 INFO promoter Promoting the container images for dlrn hash 416ce369c68996166acbbc3874de1f55440cf59a on stein to current-tripleo | 14:18 |
| *** ade_lee_ has joined #oooq | 14:26 | |
| *** ade_lee__ has joined #oooq | 14:28 | |
| *** dsneddon has joined #oooq | 14:30 | |
| *** ade_lee_ has quit IRC | 14:31 | |
| *** rlandy|rover is now known as rlandy|rover|mtg | 14:31 | |
| *** dsneddon has quit IRC | 14:35 | |
| chkumar|ruck | rlandy|rover|mtg: cool | 15:04 |
| chkumar|ruck | rlandy|rover|mtg: weshay sshnaidm https://review.opendev.org/#/c/675342/ let's merge it once zuul gives +1 | 15:05 |
| rlandy|rover|mtg | chkumar|ruck: ack - pls vote | 15:05 |
| chkumar|ruck | done | 15:05 |
| *** dsneddon has joined #oooq | 15:09 | |
| *** jbadiapa has quit IRC | 15:18 | |
| *** rlandy|rover|mtg is now known as rlandy|rover | 15:20 | |
| rlandy|rover | weshay: pls review https://code.engineering.redhat.com/gerrit/#/c/178072/ - removing pike | 15:20 |
| *** bogdando has quit IRC | 15:29 | |
| *** jfrancoa has quit IRC | 15:29 | |
| *** jbadiapa has joined #oooq | 15:30 | |
| *** jbadiapa has quit IRC | 15:35 | |
| weshay | rlandy|rover few more lines | 15:38 |
| zbr | i am back | 15:43 |
| *** marios has quit IRC | 15:48 | |
| weshay | rlandy|rover looks like stein is trying to promote now | 15:48 |
| rlandy|rover | weshay: ack failed once - hopefully this time | 15:57 |
| weshay | rlandy|rover was there anything broken w/ the promoter? | 15:57 |
| *** amoralej is now known as amoralej|off | 15:58 | |
| rlandy|rover | weshay: no _ I duplicated one of the criteria so it failed | 15:59 |
| weshay | rlandy|rover ok.. easy.. thank you! | 15:59 |
| rlandy|rover | we fixed that - so it doesn;t skip stein anymore | 15:59 |
| *** brault has joined #oooq | 16:15 | |
| *** brault has quit IRC | 16:19 | |
| rlandy|rover | weshay: https://code.engineering.redhat.com/gerrit/#/c/178072/ - updated | 16:35 |
| rlandy|rover | oy vey gate failures | 16:35 |
| rlandy|rover | sshnaidm: weshay: ok - to merge https://review.opendev.org/#/c/675225/1? | 16:39 |
| sshnaidm | rlandy|rover, of course | 16:40 |
| weshay | rlandy|rover one active failure in the gate | 16:41 |
| * weshay checks | 16:41 | |
| chkumar|ruck | see ya tomorrow | 16:41 |
| *** chkumar|ruck is now known as raukadah | 16:42 | |
| weshay | rlandy|rover probably https://logs.opendev.org/63/674363/1/gate/tripleo-ci-centos-7-undercloud-containers/9285c4c/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2019-08-08_15_01_20 | 16:42 |
| sshnaidm | rlandy|rover, after it's merged you can merge https://review.rdoproject.org/r/21776 | 16:42 |
| sshnaidm | rlandy|rover, so that it won't duplicate in periodic jobs | 16:42 |
| weshay | keepalive restarts probably can't be ignored | 16:42 |
| rlandy|rover | sshnaidm: ack | 16:43 |
| rlandy|rover | weshay: we're looking at 30 gate failures now :( | 16:43 |
| rlandy|rover | o such container"], "stdout": "", "stdout_lines": []} | 16:44 |
| rlandy|rover | 2019-08-08 15:01:20 | ...ignoring | 16:44 |
| rlandy|rover | ^^ that is ignored | 16:44 |
| rlandy|rover | weshay: ^^ | 16:44 |
| weshay | rlandy|rover I think most of it was scen010 | 16:44 |
| * weshay looks | 16:44 | |
| rlandy|rover | scenario00 | 16:44 |
| rlandy|rover | 2019-08-08 15:02:50 | TASK [Run tripleo-container-image-prepare logged to /var/log/tripleo-container-image-prepare.log] *** | 16:45 |
| weshay | rlandy|rover hrm.. new cloud.. | 16:45 |
| weshay | fortnebula | 16:45 |
| rlandy|rover | that is our real failures | 16:45 |
| rlandy|rover | and ovh | 16:45 |
| weshay | dam.. ovh | 16:45 |
| weshay | get ur sh*t together | 16:45 |
| rlandy|rover | we need a cloud-non-grata status | 16:45 |
| rlandy|rover | can we exclude clouds? | 16:46 |
| rlandy|rover | three strikes and you're out | 16:46 |
| rfolco | weshay, raukadah: sprint 14 board is organized as follows: rhel8 at the top, tempest, then staging env user stories in the same order they show up in the epic. Let me know if anything missing. | 16:48 |
| weshay | rlandy|rover ERROR tripleo_common.image.image_export [ ] Error while writing blob /var/lib/image-serve/v2/tripleomaster/centos-binary-ironic-neutron-agent/blobs/sha256:ddb3d05480efcff3def93ff067a40ce4179f72959550c364568c00b05354f0a6.gz | 16:49 |
| *** tosky has quit IRC | 16:49 | |
| raukadah | rfolco: you are back :-) | 16:49 |
| weshay | rlandy|rover may want to find a signature and pop it into logstash | 16:49 |
| weshay | and see how often this is happening | 16:49 |
| * raukadah checks taiga | 16:49 | |
| rlandy|rover | ack - there has got to be some pattern in the madness | 16:49 |
| rfolco | raukadah, no, I'm sorin. I'm folco | 16:49 |
| rfolco | :) | 16:49 |
| raukadah | Did we have taken notes during planning on user story? | 16:50 |
| raukadah | rfolco: we need to move this us https://tree.taiga.io/project/tripleo-ci-board/us/952?milestone=236933 back to sprint 13 | 16:51 |
| raukadah | it is done | 16:51 |
| rfolco | raukadah, I don't understand why taiga did not close this out | 16:52 |
| rfolco | raukadah, its gone | 16:52 |
| raukadah | rfolco: :-(, np | 16:52 |
| rfolco | raukadah, will try to move | 16:52 |
| rfolco | panda, zbr raukadah: can we merge this one ? https://review.opendev.org/#/c/673005 | 16:53 |
| raukadah | weshay: rfolco anything needed from prodchain infra for this sprint, I will be syncing with jpena next week? | 16:53 |
| weshay | raukadah just rhel8 rpms if and when needed | 16:54 |
| raukadah | rfolco: yes | 16:54 |
| raukadah | weshay: yes, will get it covered | 16:55 |
| weshay | raukadah rfolco we should spend some time tomorrow.. making sure the board has the right order | 16:55 |
| rlandy|rover | rfolco: raukadah: weshay: I think we can close https://tree.taiga.io/project/tripleo-ci-board/us/1031 - the team could not get the lab sorted and are going in another direction | 16:55 |
| zbr | rfolco: looking now | 16:56 |
| *** dtantsur is now known as dtantsur|afk | 16:57 | |
| rfolco | rlandy|rover, thx | 16:57 |
| raukadah | weshay: do we want to get this done https://tree.taiga.io/project/morucci-software-factory/us/2703 remove anonymous pull access from rdo registry this sprint from prod chain, I will request jpena for the same? | 16:57 |
| raukadah | as logging work depends on that | 16:57 |
| rfolco | rlandy|rover, archived | 16:57 |
| zbr | rfolco: yep, GTG with https://review.opendev.org/#/c/673005/7 <-- rlandy please give a wf on it. | 16:58 |
| rlandy|rover | thanks | 16:58 |
| weshay | raukadah that is blocked on the login work from panda | 16:58 |
| weshay | and zbr | 16:58 |
| rlandy|rover | weshay: wrt gate failures - a lot of mirror access issues - I am not sure there is much we can do about that in other clouds | 16:59 |
| weshay | rlandy|rover we should get a query if possible and check it into elastic-recheck.. perhaps zbr can help us get that merged | 16:59 |
| rlandy|rover | zbr: rfolco: w+'ed https://review.opendev.org/#/c/673005/ | 17:00 |
| zbr | i would help but i need warn you: only if is something likely to happen again, high impact. | 17:00 |
| rfolco | rlandy|rover, \o/ | 17:00 |
| rfolco | rlandy|rover, thx | 17:00 |
| *** derekh has quit IRC | 17:00 | |
| zbr | small issues will not be merged by upstream. they want to keep the list small. | 17:00 |
| raukadah | rfolco: weshay http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2019-08-08.log.html#t2019-08-08T13:51:00 | 17:00 |
| zbr | we could run our own instance with our own rules if we want. | 17:01 |
| raukadah | this conversation might help to dignose the issue | 17:01 |
| * raukadah needs to re-read to understand the stuff | 17:01 | |
| raukadah | related to mirror issue | 17:01 |
| zbr | weshay: rfolco: placeholder mo' for tripleo-repos: https://review.opendev.org/#/c/674571/3 | 17:03 |
| raukadah | I will check with amoralej|off and jpena|off regarding dlrn repos traffic and how we can fix it | 17:04 |
| raukadah | next week | 17:04 |
| rfolco | zbr, will review it asap | 17:04 |
| rfolco | its in my backlog | 17:04 |
| zbr | weshay: how about https://review.opendev.org/#/c/668015/ ? abandon? | 17:05 |
| zbr | same mo placeholder job for tempest: https://review.opendev.org/#/c/673859/ -- once in I can add tests. | 17:07 |
| *** udesale has quit IRC | 17:08 | |
| raukadah | zbr: does these molecule work tracked in taiga if not please create it, thanks! | 17:09 |
| *** radez has joined #oooq | 17:09 | |
| *** sshnaidm is now known as sshnaidm|off | 17:10 | |
| *** Vorrtex has joined #oooq | 17:14 | |
| zbr | raukadah: sure, I added ticket and comment linking to it. i did not update commit message to avoid wasting another day waiting for check to run. | 17:16 |
| raukadah | zbr: thanks! | 17:16 |
| weshay | raukadah++ | 17:21 |
| weshay | zbr ya | 17:22 |
| * raukadah is watching Lukka chuppi (hide and seek) movie. | 17:22 | |
| raukadah | please let's hold recheck till we merge standalone scenario10 non voting patch merged | 17:23 |
| *** tesseract has quit IRC | 17:27 | |
| rfolco | rlandy|rover, raukadah: https://tree.taiga.io/project/tripleo-ci-board/task/1229 | 17:28 |
| rfolco | this is a request from my ruck week ^ | 17:28 |
| raukadah | rfolco: can we wait till we get rhel8 jobs get healthy? | 17:28 |
| *** kopecmartin is now known as kopecmartin|off | 17:31 | |
| weshay | raukadah no point... | 17:34 |
| weshay | raukadah f28 provides no value anymore | 17:34 |
| raukadah | weshay: aye sir, we will take care of that. :-) | 17:34 |
| rlandy|rover | rfolco: your ruck weeK/ | 17:40 |
| rlandy|rover | or our ruck week? | 17:40 |
| rfolco | rlandy|rover, leftovers from my ruck | 17:40 |
| rfolco | rlandy|rover, I submitted, please help it to merge when appropriate | 17:41 |
| raukadah | rlandy|rover: we need ruck/rover handoff tomorrow with marios and rfolco ? | 17:41 |
| rlandy|rover | raukadah: that would have been helpful yesterday | 17:41 |
| rlandy|rover | now, not really | 17:41 |
| rfolco | rlandy|rover, I'm sorry about that, please let me know if any quesations | 17:42 |
| rfolco | questions* | 17:42 |
| raukadah | rfolco: I am going to catch you and marios tomorrow | 17:42 |
| * rlandy|rover think we need to get standalone -10 out of gate as well | 17:42 | |
| rlandy|rover | not just non-voting | 17:42 |
| raukadah | one more question, earlier we have featureset files, we know what is configured there, now we have standalone scenario, we can make a matrix for easy understanding what is there? | 17:43 |
| rlandy|rover | weshay: the review that moved standalone-10 to non-voting - should we also remove it from gate? | 17:51 |
| rlandy|rover | or leave it because it's juts temp disabled | 17:52 |
| weshay | if it's just for a day or two.. I would think it's ok | 17:52 |
| * rlandy|rover doesn't know if a non-voting gate failure resets the queue | 17:52 | |
| rlandy|rover | ok | 17:52 |
| rlandy|rover | we can always follow up | 17:52 |
| weshay | rlandy|rover non-voting does not reset | 17:52 |
| *** panda has quit IRC | 17:53 | |
| rlandy|rover | rfolco: -1'ed your review to remove the job definition for f28 until all the references to the jobs are removed in previous reviews | 17:54 |
| rlandy|rover | otherwise zuul will object | 17:54 |
| *** panda has joined #oooq | 17:54 | |
| rfolco | rlandy|rover, it was moved to deprecated-jobs.yaml, not removed. | 17:55 |
| rfolco | rlandy|rover, Feel free to change the approach though | 17:55 |
| rlandy|rover | rfolco: afaict the job name changed ... tripleo-ci-deprecated-standalone | 18:00 |
| rlandy|rover | oh nvm | 18:00 |
| rlandy|rover | scroll down :) | 18:00 |
| zbr | weshay: here is an idea to think about: what you start tacking the number of openreviews on our repos every week (or sprint), eventually even the number of merges per interval. it would be an interesting metric to track over time. | 18:10 |
| zbr | sadly we are too busy with other stuff | 18:10 |
| weshay | $blessed w/ work | 18:11 |
| zbr | this could make a good poster | 18:12 |
| rlandy|rover | zbr: think I need some help querying logstash | 18:13 |
| rlandy|rover | http://logstash.openstack.org/#/dashboard/file/logstash.json' | 18:13 |
| rlandy|rover | ^^ there | 18:13 |
| rlandy|rover | I type the error into the query box | 18:13 |
| weshay | rlandy|rover I have some instructions in the cheat-sheet dir | 18:13 |
| rlandy|rover | set the time interval to 24 hrs | 18:13 |
| zbr | what are you trying to search for. | 18:13 |
| rlandy|rover | no return | 18:13 |
| rlandy|rover | gate failure signatures | 18:14 |
| rlandy|rover | using @message | 18:14 |
| zbr | rlandy|rover: i personally use the context search in firefox, like on https://tree.taiga.io/project/tripleo-ci-board/task/1230 | 18:14 |
| rlandy|rover | https://tree.taiga.io/project/tripleo-ci-board/task/1230/?? | 18:15 |
| zbr | to search on it it needs proper escaping, which could be tricky. thus is why i use the extension: i only select the text in browser, right-click select logstash and enjoy | 18:15 |
| zbr | (after a number of seocnds) | 18:15 |
| rlandy|rover | is the taiga project link correct? | 18:16 |
| rlandy|rover | 1230enable placeholder molecule job in all repos | 18:16 |
| zbr | haha, wrong link. correction: https://github.com/openstack/coats#context-search-for-firefox | 18:17 |
| rlandy|rover | 2019-08-08 15:28:45.090 19255 ERROR root [ ] Image prepare failed: {"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"tripleomaster/centos-binary-nova-compute-ironic","Action":"pull"}]}]} | 18:25 |
| rlandy|rover | do we require authorization suddenly?? | 18:26 |
| rlandy|rover | https://logs.opendev.org/63/674363/1/gate/tripleo-ci-centos-7-undercloud-containers/9285c4c/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz?level=ERROR | 18:26 |
| rlandy|rover | rfolco: and other container upload people | 18:26 |
| weshay | rlandy|rover that is from a gate job right? | 18:30 |
| weshay | rlandy|rover docker.io | 18:30 |
| rlandy|rover | weshay: ack | 18:31 |
| rlandy|rover | https://logs.opendev.org/85/674485/1/gate/tripleo-ci-centos-7-undercloud-containers/d70a317/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz?level=ERROR | 18:31 |
| * weshay looks | 18:31 | |
| rlandy|rover | ^^ same failure | 18:31 |
| rlandy|rover | 2019-08-08 08:56:59.435 31121 ERROR root [ ] Image prepare failed: {"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"tripleomaster/centos-binary-mistral-api","Action":"pull"}]}]} | 18:31 |
| weshay | think it's another infra error | 18:32 |
| weshay | rlandy|rover we want to find a signature.. or a copule | 18:32 |
| weshay | couple | 18:32 |
| weshay | and get them into elastic recheck | 18:32 |
| weshay | rlandy|rover want help w/ that? | 18:32 |
| weshay | rlandy|rover this is a trap.. chasing them one by one | 18:33 |
| rlandy|rover | weshay: ^^ that is the first repeatable one I found | 18:33 |
| weshay | ugh k | 18:33 |
| rlandy|rover | no point n adding one-offs | 18:33 |
| weshay | correct | 18:33 |
| rlandy|rover | that is what I am looking for - repeated patterns | 18:33 |
| * weshay checks this pattern | 18:34 | |
| weshay | "UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"tripleomaster/centos-binary | 18:34 |
| rlandy|rover | <weshay> rlandy|rover this is a trap - HA totally | 18:34 |
| rlandy|rover | tomorow, I move on | 18:35 |
| rlandy|rover | logstash is thinking about it | 18:36 |
| weshay | rlandy|rover join my blue | 18:40 |
| weshay | zbr message:""UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"tripleomaster/centos-binary" AND tags:"console" | 18:43 |
| weshay | oops | 18:43 |
| weshay | zbr http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Name%5C%5C%5C%22%3A%5C%5C%5C%22tripleomaster%2Fcentos-binary%5C%22%20AND%20tags%3A%5C%22console%5C%22 | 18:43 |
| weshay | zbr can you drive this w/ infra tomorrow | 18:45 |
| rlandy|rover | zbr: can you join weshay's bj? | 18:46 |
| rlandy|rover | we have an infra | 18:46 |
| zbr | yeah. file a bug, or send me an email but with link to log containing the line. I want to see if my context search works. | 18:46 |
| zbr | i got tired of trying to escape the string myself. | 18:46 |
| weshay | rlandy|rover message:"Name\":\"tripleomaster/centos-binary" AND tags:"console" | 18:49 |
| *** rfolco has quit IRC | 18:51 | |
| weshay | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Name%5C%5C%5C%22%3A%5C%5C%5C%22tripleomaster%2Fcentos-binary%5C%22%20AND%20tags%3A%5C%22console%5C%22 | 19:00 |
| *** rfolco has joined #oooq | 19:00 | |
| rlandy|rover | weshay: OMG - stein finally promoted | 19:02 |
| weshay | \0/ | 19:08 |
| *** dsneddon has quit IRC | 19:11 | |
| rlandy|rover | zbr: weshay: https://bugs.launchpad.net/tripleo/+bug/1839532 the start of the bug - I assigned it to the tripleo component - although that maybe some infra component? | 19:14 |
| openstack | Launchpad bug 1839532 in tripleo "tripleo gate jobs are failing to pull containers when running on ovh provider with "UNAUTHORIZED" error" [Undecided,New] | 19:14 |
| rlandy|rover | any tags we add here? | 19:14 |
| weshay | rlandy|rover I would add an alert tag to get it the cockpit | 19:15 |
| weshay | rlandy|rover now that we have a bug.. we can submit a review to eslastic recheck | 19:15 |
| rlandy|rover | weshay: ack - doing that | 19:16 |
| weshay | rlandy|rover name the yaml file.. after the bug .. eg https://review.opendev.org/#/c/675408/ | 19:16 |
| weshay | Aqueries/1839515.yaml | 19:16 |
| *** chem has quit IRC | 19:28 | |
| rlandy|rover | https://review.opendev.org/#/c/675453/ | 19:30 |
| rlandy|rover | zbr: weshay: ^^ pls check me | 19:30 |
| weshay | rlandy|rover run it locally | 19:31 |
| rlandy|rover | k | 19:31 |
| rlandy|rover | k - fixing errors | 19:47 |
| rlandy|rover | TypeError: '<' not supported between instances of 'list' and 'str' | 19:52 |
| rlandy|rover | ugh what am I missing?? | 19:52 |
| rlandy|rover | zbr: weshay: pls dig me out of my misery ... ^^ fails when run locally | 19:54 |
| rlandy|rover | what am I missing> | 19:54 |
| weshay | rlandy|rover elastic-recheck-query | 19:54 |
| rlandy|rover | yes | 19:55 |
| rlandy|rover | that is what is failing | 19:55 |
| weshay | hrm? | 19:55 |
| weshay | it works for me locally.. I tried it too :) | 19:55 |
| weshay | elastic-recheck-query queries/test.yaml | 19:55 |
| *** dsneddon has joined #oooq | 19:55 | |
| rlandy|rover | it's failing for me locally | 19:55 |
| weshay | tmate? | 19:55 |
| *** panda has quit IRC | 19:57 | |
| *** panda has joined #oooq | 19:59 | |
| *** jbadiapa has joined #oooq | 20:00 | |
| *** ankit08 has joined #oooq | 20:06 | |
| rlandy|rover | weshay: git one more for you ... https://code.engineering.redhat.com/gerrit/#/c/178072/ | 20:10 |
| weshay | rlandy|rover k.. all I have is +1 | 20:11 |
| weshay | jjoyce may be able to merge | 20:11 |
| rlandy|rover | k - will ping him - thanks | 20:12 |
| rlandy|rover | just want to check the patch is correct | 20:12 |
| *** jbadiapa has quit IRC | 20:12 | |
| weshay | <clarkb> weshay: ideally e-r queries will only match on jobs that have failed due to the bug tied to the query. If we are searching for the name of an image that is used in every job are we going to match every job that runs regardless of whether or not it failed? | 20:17 |
| weshay | 14:15 <weshay> http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Name%5C%5C%5C%22%3A%5C%5C%5C%22tripleomaster%2Fcentos-binary%5C%22%20AND%20tags%3A%5C%22console%5C%22 | 20:17 |
| weshay | 14:15 <weshay> we can refine it a bit more.. | 20:17 |
| weshay | 14:15 <weshay> I get your point now | 20:17 |
| rlandy|rover | k | 20:20 |
| rlandy|rover | weshay: k .. trying the suggested <clarkb> '"code":"UNAUTHORIZED","message":"authentication required"' looks like the string we may want to match on | 20:22 |
| weshay | rlandy|rover https://review.opendev.org/#/c/675453/2/queries/1839532.yaml | 20:23 |
| weshay | rlandy|rover so I think that will find failures when we're pulling | 20:23 |
| weshay | just tested | 20:23 |
| weshay | wdyt | 20:24 |
| rlandy|rover | well I won;t offer to test :) | 20:24 |
| rlandy|rover | let me try on GUI | 20:24 |
| rlandy|rover | I want to see the diff between the two | 20:24 |
| weshay | rlandy|rover k | 20:25 |
| rlandy|rover | logstash is thinking about it | 20:26 |
| rlandy|rover | I didn;t know you could add message twice | 20:27 |
| rlandy|rover | weshay: k - works on http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Action%5C%5C%5C%22%3A%5C%5C%5C%22pull%5C%22%20AND%20message%3A%5C%22ERROR%20%2Fvar%2Flog%2Ftripleo-container-image-prepare.log%5C%22%20AND%20tags%3A%5C | 20:27 |
| rlandy|rover | let's try it if clark is happier with that | 20:28 |
| *** jtomasek has quit IRC | 20:31 | |
| rlandy|rover | weshay: lol jjoyce is PTO - will find someone tomorrow to look at that review again | 20:32 |
| weshay | rlandy|rover mburns can merge it | 20:33 |
| rlandy|rover | ack - asked him | 20:34 |
| rlandy|rover | weshay: zbr: thanks for the lesson in elastic recheck | 20:35 |
| *** ade_lee__ has quit IRC | 20:59 | |
| weshay | rlandy|rover no prob | 21:03 |
| weshay | rlandy|rover they merged it | 21:07 |
| rlandy|rover | mburns? | 21:07 |
| rlandy|rover | yes | 21:07 |
| rlandy|rover | I am waiting for https://review.opendev.org/#/c/675342/ to get through check so I can w+ it | 21:08 |
| *** Vorrtex has quit IRC | 21:11 | |
| rlandy|rover | oh https://review.opendev.org/#/c/675453/ merged | 21:12 |
| rlandy|rover | yay | 21:12 |
| rlandy|rover | zbr will be shocked | 21:12 |
| rlandy|rover | 29 fails - we're out of the 30 range | 21:15 |
| *** rfolco has quit IRC | 21:18 | |
| *** dsneddon has quit IRC | 21:20 | |
| *** dsneddon has joined #oooq | 21:29 | |
| *** jtomasek has joined #oooq | 21:36 | |
| *** dsneddon has quit IRC | 21:39 | |
| *** ankit08 has quit IRC | 21:44 | |
| *** dsneddon has joined #oooq | 21:48 | |
| *** panda has quit IRC | 22:03 | |
| *** dsneddon has quit IRC | 22:03 | |
| *** panda has joined #oooq | 22:03 | |
| *** vinaykns has quit IRC | 22:15 | |
| *** panda has quit IRC | 22:16 | |
| *** panda has joined #oooq | 22:20 | |
| *** dsneddon has joined #oooq | 22:24 | |
| *** dsneddon has quit IRC | 22:29 | |
| *** dsneddon has joined #oooq | 22:51 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!