Friday, 2021-10-01

« Thursday, 2021-09-30 Index Monday, 2021-10-04 »
*** ysandeep|out is now known as ysandeep|sick05:11
* ysandeep|sick on sick leave today05:12
*** poojajadhav is now known as pojadhav05:14
*** pojadhav- is now known as pojadhav05:52
*** amoralej|off is now known as amoralej06:28
*** jpena|off is now known as jpena06:58
dpawliktake care ysandeep|sick!07:06
toskyarxcruz|ruck: hi! Not sure how to do that, but would it be possible to boost the priority for the fix to ansible-role-collect-logs (https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/809979)? 08:23
toskyI have a pending patch which solves some issue and it's blocked by that08:23
arxcruz|rucktosky: sure, i'm ruck rovering, that's the reason it's taking too long08:31
arxcruz|rucktosky: if it's something urgent, maybe put as non voting for now 08:31
toskyarxcruz|ruck: if you are the ruck, shouldn't the rover help solving those issues, if I remember how it works?08:44
*** ykarel is now known as ykarel|lunch08:50
arxcruz|rucktosky: too many things happening, but i'll try to work on this :) 08:56
* bhagyashris|rover lunch brb08:57
amoralejis there any known issue with the containers for last promotion in master in quay.io ?09:07
amoraleji got "Error running container image prepare: Not found image: https://quay.io/v2/tripleomaster/openstack-placement-api/manifests/623c4c284547ff6bb7d50aa1d5eed839", "params": {}, "success": false}09:08
amoralejarxcruz|ruck, ^09:08
amoralejin tripleo job in https://review.opendev.org/c/openstack/puppet-swift/+/79583209:09
arxcruz|ruckamoralej: checking 09:09
amoralejin https://quay.io/repository/tripleomaster/openstack-placement-api?tab=tags i see current-tripleo is 10 days old09:10
amoralejbut https://trunk.rdoproject.org/centos8-master/current-tripleo/ is from yesterday09:10
arxcruz|ruckamoralej: that's weird, i'm checking the logs, and should be tagged properly 09:12
arxcruz|ruckamoralej: give me a few minutes09:12
amoralejack09:12
amoralejwhere are the promoter logs?09:12
amoralejif i can take a look09:12
amoralejthere may be something wrong in job definition09:13
arxcruz|ruckamoralej: not a problem with the job, the problem was the toolbox, it was without dns (don't know why) and so it wasn't able to tag it properly 09:24
arxcruz|rucktime="2021-10-01T08:00:03Z" level=error msg="The HTTP request failed with error  Get \"https://review.rdoproject.org/zuul/api/builds?job_name=periodic-tripleo-ci-build-containers-ubi-8-push\": dial tcp: lookup review.rdoproject.org on 192.168.100.4:53: server misbehaving"                       09:25
amoralejso, it's affecting other tripleo jobs?09:25
amoralejups09:25
arxcruz|ruckamoralej: I'm running manually the tool, i can't even see the html logs, something is wrong 09:25
amoralejack, thanks09:26
arxcruz|ruckamoralej: logs should be here https://38.102.83.131/quay/tag/09:26
arxcruz|ruckgut i can't access :( 09:26
arxcruz|ruck38.102.83.131/quay/tag/master-report.html09:26
arxcruz|ruckamoralej: 10-20 min and it will be done 09:27
dpawlikrlandy|testbox: FYI, One CI job https://softwarefactory-project.io/r/gitweb?p=software-factory/sf-infra.git;a=blob;f=roles/log-gearman/tasks/main.yml#l77 is pushing logs into opensearch:  https://opensearch.rdoproject.org/rdoproject_org_autologin . Can it be in rdo tenant or I need to create dedicated new tenant for it?10:03
dpawlikcc weshay and anbanerj10:04
*** ykarel|lunch is now known as ykarel10:20
arxcruz|ruckamoralej: fixed, for master, working on the other branches10:22
amoralejthanks arxcruz|ruck, i'm rechecking10:22
*** jpodivin is now known as jpodivin|ruck10:27
*** dviroel|out is now known as dviroel10:47
erbotRequired files not ready after 301.38148283958435s for tripleo-stream9-development-buildimage-overcloud-hardened-uefi-full-master 35465,7,bfabcc1dc0b74566a3e4afd792cb098110:51
frenzy_fridayHey dpawlik, I saw https://softwarefactory-project.io/r/c/software-factory/sf-infra/+/22669 is merged, thabks! Are there other patches still required to get the logs from containers-multinode upstream to rdo elasticsearch?10:54
*** pojadhav is now known as pojadhav|afk11:01
* pojadhav|afk away for sometime11:01
*** rlandy is now known as rlandy|ruck11:03
rlandy|ruckarxcruz|ruck: bhagyashris|rover: hey - how are things today?11:06
rlandy|ruckneed help with anything?11:06
rlandy|ruckOVB still hosed?11:06
rlandy|ruckyep - look slike it11:06
rlandy|ruckdviroel: hey - did you work out the c7 https molecule error?11:07
dviroelrlandy|ruck: no, still getting the same error11:10
rlandy|ruckbhagyashris|rover: hey - when you have a moment, let's chat about moving ussuri to PSI11:10
rlandy|ruckmarios: hey ... booked some time with you later to go through the PTG stuff for CI.11:11
mariosack rlandy|ruck sounds good11:12
rlandy|ruckdviroel: did you chat with marios about the  c7 error?11:13
* rlandy|ruck remembers there were some issues zbr broke/fixed accessing https11:14
rlandy|ruckvexx is still hosed11:14
dviroelrlandy|ruck: no not yet11:14
dviroelmarios: fyi  https://zuul.opendev.org/t/openstack/build/833f2fed093542eeb46bd1308e3de3b8/log/job-output.txt#100411:15
dviroelmarios: get_hash started to fail yesterday on c7 molecule job11:16
mariosdviroel: ouch thanks11:16
mariosdviroel: maybe we can disable if it blocks until we work out/fix it11:16
rlandy|ruckmarios: just fyi - vexx added new DNS to networks that is hosing OVB11:16
rlandy|ruckmay be hitting that 11:16
rlandy|ruckbut in the mean time11:16
bhagyashris|roverrlandy|ruck, OVB jobs are still failing 11:17
rlandy|ruckbhagyashris|rover: let's check up with infra11:19
rlandy|ruckbhagyashris|rover: going to #rhos-ops11:19
bhagyashris|roveri asked dpawlik but no updates 11:19
bhagyashris|roverrlandy|ruck, ack11:20
arxcruz|ruckrlandy|ruck: toolbox was without dns, some containers weren't coppied, i'm working on that 11:20
rlandy|ruckarxcruz|ruck: ok - thank you 11:21
rlandy|ruckarxcruz|ruck: did the servers we added get removed?11:21
rlandy|ruckarxcruz|ruck: is that specific to the toolbox or did it get hit by the vexx fun we're having with OVB?11:22
arxcruz|ruckrlandy|ruck: it was without dns, i added the 1.1.1.1 and now i'm copping the current-tripleo tags 11:23
rlandy|ruckarxcruz|ruck: k - that happens from time to time when vexx gets restarted11:24
*** jpena is now known as jpena|lunch11:27
bhagyashris|roverrlandy|ruck, we can chat about moving ussuri to PSI11:33
bhagyashris|roveri am free11:33
rlandy|ruckbhagyashris|rover: yep - let's talk about PSI https://meet.google.com/iah-afpu-evr?pli=1&authuser=011:34
*** jpena|lunch is now known as jpena12:26
mariosrlandy|ruck: https://etherpad.opendev.org/p/tripleo-xena-topics https://etherpad.opendev.org/p/tripleo-ptg-retrospective https://etherpad.opendev.org/p/tripleo-ptg-xena 12:34
mariosrlandy|ruck: https://etherpad.opendev.org/p/tripleo-ci-yoga-topics12:36
mariosrlandy|ruck: https://etherpad.opendev.org/p/centos-stream-9-upstream12:42
*** ykarel is now known as ykarel|afk12:43
*** amoralej is now known as amoralej|lunch12:53
bhagyashris|roverrlandy|ruck, arxcruz|ruck leaving for the day 12:56
rlandy|ruckbhagyashris|rover: have a good weekend13:02
*** pojadhav|afk is now known as pojadhav13:16
pojadhavrlandy|ruck, hey13:17
rlandy|ruckpojadhav: hey - will ping you in a few13:19
pojadhavrlandy|ruck, sure13:19
rlandy|ruckarxcruz|ruck: hey - cockpit change ready to merge?13:20
arxcruz|ruckrlandy|ruck: yes, i update this morning 13:25
*** ykarel|afk is now known as ykarel13:29
rlandy|ruckarxcruz|ruck: k - w'ed13:30
rlandy|ruckonce it merges will pull the change and restart cockpit13:30
*** amoralej|lunch is now known as amoralej13:35
rlandy|ruckarxcruz|ruck: k - pulling in your change - hope it restarts this time :)13:49
* arxcruz|ruck cross his fingers13:49
rlandy|ruckon the tmux13:49
rlandy|ruckarxcruz|ruck: we ok :)13:52
arxcruz|ruck\o/13:52
*** ysandeep|sick is now known as ysandeep13:55
* ysandeep is around now.13:56
rlandy|ruckysandeep: hey - feeling better?13:58
ysandeeprlandy|ruck: Hi.. yeah much better than morning.. thanks!13:59
*** pojadhav is now known as pojadhav|afk14:17
*** ykarel is now known as ykarel|away14:26
*** ysandeep is now known as ysandeep|out15:00
frenzy_fridayrlandy|ruck, call ?15:01
rlandy|ruckyep15:01
ade_leedviroel, hey good mornign15:14
ade_leedviroel, so -- still trying to figure out the fips jobs ..15:15
ade_leedviroel, a few of them seem to be failing with "Can't run container rabbitmq_wait_bundle"15:16
ade_leeas in here:  https://zuul.opendev.org/t/openstack/build/2abd56a6ca034dc6b12c9d63e8f77eff/log/logs/undercloud/home/zuul/standalone_deploy.log15:16
ade_leeis this new and seen elsewhere?  its certainly new to this job ..15:17
dviroelade_lee: hum, lets take a look on var/log/extras to see if there is a specific issue from this one15:17
ade_leeError: 'rabbitmqctl eval "lists:keymember(rabbit, 1, application:which_applications())." | grep -q true' returned 1 instead of one of [0]15:17
ade_leethere are at least a couple jobs that fail because of this one15:17
ade_leedviroel, the other thing is trying to get the testproject runs going - but they seem to be failing some merge thing -- its not clear whats failing though --15:19
ade_leehttps://review.rdoproject.org/r/c/testproject/+/3591115:19
ade_leehttps://review.rdoproject.org/r/c/testproject/+/3591115:19
ade_leehttps://review.rdoproject.org/r/c/testproject/+/3512115:19
ade_leetrying to rebase everything ..15:21
dviroelyeah, would be my first tentative15:22
rlandy|ruckade_lee: dviroel: hey - seems like you got the first step here15:32
ade_leerlandy|ruck, eh?15:32
rlandy|ruckrebase15:32
ade_leeI tried rebasing everything but it looks like its still failing ..15:33
rlandy|ruckhttps://review.rdoproject.org/r/c/testproject/+/35911?15:33
* rlandy|ruck checks15:34
ade_leerlandy|ruck, yup15:34
ade_leeI rechecked and instantaneously got merge failure15:34
rlandy|ruckgive me one sec15:34
rlandy|ruckjust nede to restart ovb15:34
rlandy|ruckwill rework this in a few15:34
ade_leerlandy|ruck, ok thanks15:35
*** dviroel is now known as dviroel|lunhc15:42
*** dviroel|lunhc is now known as dviroel|lunch15:42
*** marios is now known as marios|out15:46
rlandy|ruckade_lee: https://review.opendev.org/c/openstack/tripleo-ci/+/81078615:50
rlandy|ruck Rebase on top of the master branch (breaks relation chain) 15:50
rlandy|ruckWIP/DNM - Add fips job 15:51
rlandy|ruckrebased on that15:51
rlandy|ruckhttps://review.opendev.org/c/openstack/tripleo-ci/+/808215/ would need rebase15:51
rlandy|ruckwhat's the right relationship there?15:52
rlandy|ruck https://review.opendev.org/c/openstack/tripleo-ci/+/810786 - what should that patch be rebased on?15:52
rlandy|ruckade_lee: ^^?15:56
ade_leerlandy|ruck, looking16:05
ade_leerlandy|ruck, sorry -- I'm not entirely sure I understand your question16:07
rlandy|ruckhttps://review.rdoproject.org/r/c/testproject/+/3591116:08
rlandy|ruckdepends on16:08
rlandy|ruckhttps://review.opendev.org/c/openstack/tripleo-ci/+/81078616:08
rlandy|ruck^^ patch is not rebased16:08
rlandy|ruckif I rebase that on master - reports to break chain16:09
rlandy|ruckmust be rebased on 'WIP/DNM - Add fips job', right?16:09
*** jpena is now known as jpena|off16:09
ade_leeyes - but  it is already as far as I can tell16:09
ade_leeDNM - Testing FIPS in multinode jobs too is based on "WIP/DNM - Add fips job" which is based on master16:10
rlandy|ruckhttps://review.opendev.org/c/openstack/tripleo-ci/+/80821516:10
rlandy|ruckis not rebased16:10
rlandy|ruckade_lee: quick meeting to sort this out?16:11
ade_leerlandy|ruck, sounds good16:11
rlandy|ruckhttps://meet.google.com/nwm-prfe-vnw?pli=1&authuser=016:11
*** dviroel|lunch is now known as dviroel16:26
ade_leerlandy|ruck, ok - so removing depends-on seems to allow it to continue.  going to add them back one by one and see what happens16:43
rlandy|ruckade_lee: k - at least we know16:43
*** amoralej is now known as amoralej|off16:57
dviroelade_lee: did you solve the depends-on mystery?17:05
dviroeloh, still see that you are working with them17:06
ade_leedviroel, rlandy|ruck yeah -- it seems that rdo really doesn't like https://review.opendev.org/c/openstack/barbican-tempest-plugin/+/81078217:09
ade_leethats the one that is breaking the merge17:10
ade_leedviroel, rlandy|ruck so yeah -- rdo doesn't like something about barbican-tempest-plugin17:11
rlandy|ruckout of meeting in a few17:11
ade_leerlandy|ruck, thanks17:12
dviroelade_lee: and the multi-node one?17:16
dviroelhttps://review.rdoproject.org/r/c/testproject/+/3591117:16
dviroelhave few dependencies17:17
*** ysandeep|out is now known as ysandeep17:17
ade_leeyeah -- that depends on the patch I'm looking at dependencies for17:17
ade_leedviroel, I think if I fix whats wrong with https://review.rdoproject.org/r/c/testproject/+/35121 it will fix https://review.rdoproject.org/r/c/testproject/+/3591117:18
rlandy|ruckade_lee: ok - out of meeting17:19
frenzy_fridayIs https://review.rdoproject.org/analytics/app/ down? I cant log in17:19
dviroelade_lee: yeah, because it has another tripleo-ci patch as parent17:19
ade_leerlandy|ruck, gmeet?  might be faster ..17:20
rlandy|rucksure17:20
ade_leerlandy|ruck, same as before ?17:20
rlandy|ruckk17:20
ade_leerlandy|ruck, https://meet.google.com/nwm-prfe-vnw?pli=1&authuser=017:21
ade_leedviroel, ^^ if you want to join17:21
ade_leedviroel, started 3591117:34
ade_leedviroel, seems to be running17:36
ade_leedviroel, can we keep the node for that one?17:36
ade_leenodes ..17:36
ade_leeso we can try to diagnose the ssh issue17:37
dviroelsure, we just need to ask on #rhos-ops17:38
ade_leedviroel, do we need to wait till the actual job starts , so they know which nodes to hold?17:44
dviroelade_lee: yes17:44
ysandeephappy weekend guys! See you on Monday!18:17
rlandy|ruckysandeep: have a good weekend18:17
*** ysandeep is now known as ysandeep|out18:17
rlandy|ruckade_lee: dviroel; I might be able to hold those nodes18:20
rlandy|rucklet me check18:20
dviroelrlandy|ruck: don't we need to wait the job get a node first? containers-multinode didn't started yet18:21
rlandy|ruckyes18:21
rlandy|ruckif nobody response on #rhos-ops and the job starts, I'll try then18:22
dviroel++18:22
* rlandy|ruck goes back to fixing OVB18:22
dviroelrlandy|ruck: is the fix only to install ca-certificates?18:23
dviroel^ bmc 18:23
dviroelrlandy|ruck: the ssl issue?18:23
* dviroel try that in molecule job18:23
dviroelupdating ca-certificates fix the issue on c7 container18:29
rlandy|ruckdviroel: ack18:30
rlandy|ruckwe updated the bmc-template image 18:30
dviroelyeah, read some comments in the irc18:31
dviroelgreat :)18:31
rlandy|ruckI need to rename the images on vexxhost so OVB uses the new one18:31
dviroelack18:32
ade_leerlandy|ruck, dviroel hey -- looks like the job just started18:37
ade_leerlandy|ruck, dviroel gotta go grab some late lunch and then pick up kids from school -- can one of you guys make sure the nodes are reserved?18:38
rlandy|ruckyou want the scenario002 node or the content provider node?18:38
rlandy|ruckboth nodes?18:38
ade_leetripleo-ci-centos-8-containers-multinode18:38
rlandy|ruckah18:39
rlandy|rucklet me try18:39
ade_leerlandy|ruck, this is for https://review.rdoproject.org/r/c/testproject/+/3591118:39
* ade_lee gotta nosh ..18:41
rlandy|ruck+------------+----------------+-----------------------------------+------------------------------------------+--------------------------+-----------+---------------+18:42
rlandy|ruck|     ID     |     Tenant     |              Project              |                   Job                    |        Ref Filter        | Max Count |     Reason    |18:42
rlandy|ruck+------------+----------------+-----------------------------------+------------------------------------------+--------------------------+-----------+---------------+18:42
rlandy|ruck| 0000000948 | rdoproject.org | review.rdoproject.org/testproject | tripleo-ci-centos-8-containers-multinode | refs/changes/11/35911/.* |     1     | debug ade_lee |18:42
rlandy|ruck+------------+----------------+-----------------------------------+------------------------------------------+--------------------------+-----------+---------------+18:42
rlandy|ruckon autohold list18:42
rlandy|ruckhopefully that worked18:42
* rlandy|ruck never sure18:42
ade_leerlandy|ruck, thanks -- sent you my key to add to nodes if it worked 18:43
rlandy|ruckdepends if my keys get on that node :)18:43
rlandy|ruckwe will see18:43
rlandy|ruckI'm half competent at this :)18:44
ade_leethanks -  back in a little bit18:44
rlandy|ruckade_lee: so I could hold the node (I think) but I don;t have access to add your keys19:00
-opendevstatus- NOTICE: The review.opendev.org Gerrit server has become unreachable as of approximately 19:10 UTC due to a networking issue in the provider, but should be reachable again shortly19:43
dviroelrlandy|ruck: https://review.opendev.org/c/openstack/tripleo-repos/+/81221120:01
dviroelreview is back :)20:01
rlandy|rucknice20:01
rlandy|ruckif that gets through tox, great20:01
dviroelzuul already voted20:03
rlandy|rucklet's merge then20:05
*** dviroel is now known as dviroel|out21:24
dviroel|outhave a good weekend o/21:24
rlandy|ruckdviroel|out: you too21:24
weshayrlandy|ruck, this was funny today21:36
weshaygot into the quay vs. key pronunciation thing today..21:36
rlandy|rucklol21:36
weshayapparently.. I asked Brandon Phillips (Then CoreOS CTO) on a call how they said it. He was quite frank it was pronounced "kway" and not "key", much to the chagrin of our Aussie coworker who still calls it "key". I've called it "kway" since.21:36
weshayhttps://news.ycombinator.com/item?id=2151656521:36
rlandy|ruckhysterical21:36
weshayif you google around you will find the coreOS team pronounces it QUAY.. not KEY21:37
rlandy|ruckwhen engineers fight over pronunciation, you know you've hit a sensitive spot21:37
weshayhttps://www.redhat.com/en/blog/floating-kwaaaay-podman-and-systemd21:37
rlandy|ruckThey're note using Slack!21:38
rlandy|ruck^^ best comment21:38
rlandy|ruckOMG this runs deep21:38
« Thursday, 2021-09-30 Index Monday, 2021-10-04 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!