*** ysandeep|out is now known as ysandeep | 02:08 | |
*** ysandeep is now known as ysandeep|afk | 03:40 | |
*** ysandeep|afk is now known as ysandeep | 05:20 | |
*** akahat_ is now known as akahat | 05:39 | |
jm1 | o/ | 06:25 |
---|---|---|
marios | \o | 06:33 |
*** ykarel_ is now known as ykarel | 06:41 | |
*** ysandeep is now known as ysandeep|afk | 06:51 | |
*** amoralej|off is now known as amoralej | 07:06 | |
marios | please add to your reviews thank you https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/851427 | 07:10 |
reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 07:10 |
*** ysandeep|afk is now known as ysandeep | 07:18 | |
*** jpena|off is now known as jpena | 07:32 | |
*** ysandeep is now known as ysandeep|lunch | 07:50 | |
*** soniya29 is now known as soniya29|afk | 08:13 | |
*** ysandeep|lunch is now known as ysandeep | 09:40 | |
* jm1 bbl, rebooting | 09:48 | |
*** rlandy_ is now known as rlandy | 10:33 | |
rlandy | bhagyashris|ruck: frenzyfriday|rover: hello | 10:34 |
frenzyfriday|rover | rlandy, hi | 10:35 |
rlandy | bhagyashris|ruck: frenzyfriday|rover: let's sync when you are both around | 10:35 |
bhagyashris|ruck | rlandy, hi | 10:35 |
frenzyfriday|rover | rlandy, regarding https://bugs.launchpad.net/tripleo/+bug/1982744/ : there is a suggestioon to add sudo dnf reinstall openstack-selinux container-selinux | 10:35 |
frenzyfriday|rover | sudo rpm -V openstack-selinux as workaround | 10:35 |
frenzyfriday|rover | do you know where we can add it? release files? | 10:35 |
rlandy | bhagyashris|ruck: frenzyfriday|rover: https://meet.google.com/diq-dpvh-wcq | 10:35 |
bhagyashris|ruck | give me min | 10:36 |
rlandy | bhagyashris|ruck: https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/pipeline/job/pipeline_integration-pcci-16.2_dlrn-rhel-8.4-virthost-3cont_2comp_3ceph-ipv6-geneve-ceph/ | 10:55 |
rlandy | arxcruz: hi - what is the story with https://bugs.launchpad.net/tripleo/+bug/1982874? | 11:09 |
rlandy | can you update the CIX card | 11:09 |
rlandy | I think you added to skip list? | 11:09 |
rlandy | bhagyashris|ruck: frenzyfriday|rover: ^^ | 11:10 |
rlandy | arxcruz: hey - can you also explain what's the final word on https://trello.com/c/8tGYExhe/2603-cixlp1980255tripleociproa-tripleo-ci-centos-9-standalone-and-multinode-ipa-are-failing-the-testminimumbasicinstancehardrebootaft? | 11:12 |
arxcruz | rlandy it's a new but, with tempest update, this test started to run (it was being skipped before because it was marked as unstable) | 11:14 |
arxcruz | rlandy the second one, it passes, but we need to check when/how we will update tempest to pin on it, because it might bring other failures as well | 11:14 |
arxcruz | but the patch was merged already | 11:15 |
rlandy | arxcruz: pls join https://meet.google.com/diq-dpvh-wcq | 11:15 |
marios | review call starting now | 11:17 |
rlandy | arxcruz: for the first one should we add to skiplist | 11:17 |
rlandy | marios: working with rr - pls continue w/o me | 11:17 |
arxcruz | rlandy it is already on the skiplist | 11:17 |
ysandeep | frenzyfriday|rover, bhagyashris|ruck, rlandy review time | 11:17 |
rlandy | ysandeep: we are working on rr stuff | 11:17 |
rlandy | pls start w/o us | 11:17 |
ysandeep | okay o/ | 11:17 |
rlandy | arxcruz: can you join the meeting pls | 11:18 |
*** pojadhav is now known as pojadhav|afk | 11:33 | |
*** dviroel_ is now known as dviroel | 11:45 | |
rlandy | frenzyfriday|rover: hmmm periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby failed again | 11:50 |
rlandy | you can skip promote | 11:50 |
rlandy | we know it's a transient issue | 11:50 |
frenzyfriday|rover | rlandy, yes, it is a genuine failure. https://bugs.launchpad.net/tripleo/+bug/1982464 | 11:50 |
frenzyfriday|rover | I have held the node | 11:50 |
rlandy | ok - great | 11:50 |
frenzyfriday|rover | 39 is transient | 11:50 |
rlandy | we don;t need to hole the line | 11:50 |
rlandy | frenzyfriday|rover: ok - you can prep a patch to skip kvm and those two test from master and wallaby | 11:51 |
rlandy | and let's promote those | 11:51 |
rlandy | and clear that from your plate | 11:51 |
frenzyfriday|rover | rlandy, ack for the master. But for the wallaby fs64 has an active bug https://bugs.launchpad.net/tripleo/+bug/1982464 Should we still promote? | 11:52 |
frenzyfriday|rover | https://bugs.launchpad.net/tripleo/+bug/1982464 is related to the ipa issue | 11:52 |
rlandy | frenzyfriday|rover: is amoralej going to look into it? | 11:52 |
rlandy | ie: the held node? | 11:52 |
rlandy | frenzyfriday|rover: if so, let's see after the investigation | 11:53 |
amoralej | what should i look at? | 11:53 |
rlandy | amoralej: https://bugs.launchpad.net/tripleo/+bug/1982464/comments/1 | 11:53 |
amoralej | do we have some held node? | 11:53 |
amoralej | that'd be cool | 11:53 |
frenzyfriday|rover | amoralej, yep, lemme ping you the IP | 11:53 |
rlandy | responding to the comment to hold the node to check dns for package install for ipa | 11:53 |
rlandy | frenzyfriday|rover: ^^ can you pass the held node? | 11:53 |
rlandy | great | 11:53 |
amoralej | please add my pub key | 11:53 |
amoralej | https://github.com/amoralej.keys | 11:54 |
rlandy | bhagyashris|ruck: we can ask dasm or I can run scrum today - since you'll be on CIX | 11:57 |
bhagyashris|ruck | rlandy, yes i will ask him | 11:57 |
rlandy | bhagyashris|ruck: also when we sync, let's chat about retro schedule | 11:57 |
rlandy | keeping it on wed or moving to thurs | 11:57 |
rlandy | will confirm at scrum today | 11:58 |
rlandy | and let you know | 11:58 |
marios | o/ tripleo-ci please add to yr reviews when you have time thank you https://review.opendev.org/c/openstack/python-tripleoclient/+/851384 (and depends-on) | 12:16 |
reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 12:16 |
*** ysandeep is now known as ysandeep|afk | 12:23 | |
rlandy | frenzyfriday|rover, I kicked the fs001 train rerun: https://review.rdoproject.org/r/c/testproject/+/36254 | 12:24 |
frenzyfriday|rover | rlandy, thanks! | 12:25 |
bhagyashris|ruck | rlandy, ack | 12:25 |
rlandy | fs039 same error: ansible-macros-2021.1.2-2.el9s.noarch: Cannot download, all mirrors were already tried without success | 12:27 |
chandankumar | ysandeep|afk++ adding port to security group fixed the ibm log issue | 12:29 |
frenzyfriday|rover | rlandy, https://logserver.rdoproject.org/34/44334/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/8e4a00d/job-output.txt and https://logserver.rdoproject.org/37/44337/1/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/20ea2bb/job-output.txt | 12:31 |
frenzyfriday|rover | one failed with cannot download... the other failed in Run the tripleo-ipa preparation script - same as fs64 | 12:32 |
rlandy | tripleo-ipa preparation script - same as fs64 - yep same as 64 | 12:32 |
rlandy | bhagyashris|ruck: need any help? | 12:33 |
bhagyashris|ruck | rlandy, not now thank you :) | 12:35 |
frenzyfriday|rover | rlandy, the testprojs to check if workaround for the selinux works are up - lets see if they pass | 13:00 |
rlandy | frenzyfriday|rover: great - thank you | 13:10 |
*** pojadhav|afk is now known as pojadhav | 13:15 | |
rlandy | frenzyfriday|rover: tripleo component on master looks like legit error | 13:16 |
rlandy | 13>Jul 31 20:38:12 puppet-user: Error: Evaluation Error: Error while evaluating a Resource Statement, Apache::Vhost[placement_wsgi]: | 13:16 |
rlandy | <13>Jul 31 20:38:12 puppet-user: parameter 'port' expects a value of type Undef, Array[Stdlib::Port = Integer[0, 65535]], or Stdlib::Port = Integer[0, 65535], got String | 13:16 |
rlandy | <13>Jul 31 20:38:12 puppet-user: parameter 'priority' expects a value of type Undef, Integer, or Boolean, got String | 13:16 |
rlandy | <13>Jul 31 20:38:12 puppet-user: parameter 'access_log_file' expects a value of type Undef or String, got Boolean | 13:16 |
rlandy | <13>Jul 31 20:38:12 puppet-user: parameter 'access_log_pipe' expects a value of type Undef or String, got Boolean (file: /etc/puppet/modules/openstacklib/manifests/wsgi/apache.pp, line: 357) on node standalone.localdomain | 13:17 |
rlandy | + rc=1 | 13:17 |
rlandy | + '[' false = false ']' | 13:17 |
rlandy | + set +x | 13:17 |
rlandy | 2022-07-31 16:38:21.001651 | fa163ecf-73d9-df2e-85fe-00000000211a | FATAL | Manage Puppet containers (generate config) for step 1 | standalone | error={"changed": false, "msg": "Failed containers: container-puppet-horizon, container-puppet-keystone, container-puppet-manila, container-puppet-neutron, container-puppet-nova, container-puppet-nova_metadata, container-puppet-placement"} | 13:17 |
rlandy | pls LP bug and ping tkajinum | 13:17 |
rlandy | frenzyfriday|rover: ^^ | 13:17 |
rlandy | frenzyfriday|rover: bhagyashris|ruck: let's sync again at half past the hour | 13:18 |
rlandy | ie: in 12 minutes | 13:18 |
rlandy | bhagyashris|ruck: pls add integration line info plus any testproject reruns for downstream on hackmd | 13:19 |
frenzyfriday|rover | rlandy, ack, checking | 13:20 |
frenzyfriday|rover | rlandy, I am creating a bug for tripleo master | 13:23 |
bhagyashris|ruck | rlandy, ack | 13:29 |
bhagyashris|ruck | rlandy, yes | 13:29 |
*** ysandeep|afk is now known as ysandeep | 13:33 | |
ysandeep | chandankumar: great! | 13:34 |
frenzyfriday|rover | bhagyashris|ruck, rlandy do you want to sync now? | 13:34 |
rlandy | frenzyfriday|rover: bhagyashris|ruck: ack https://meet.google.com/imr-duwb-csh?pli=1&authuser=0 | 13:34 |
akahat | rlandy, I've closed this bug just now.. https://bugzilla.redhat.com/show_bug.cgi?id=2110535, I just saw your comments on the CIX card. | 13:35 |
akahat | rlandy, should i keep this bug open for Infrared? | 13:35 |
*** amoralej is now known as amoralej|lunch | 13:37 | |
rlandy | akahat; yeah - was keeping it open for infrared | 13:58 |
rlandy | but ok if you closed it | 13:58 |
rlandy | the CIX is still open | 13:58 |
rlandy | and they can reopen if need be | 13:58 |
akahat | rlandy, ok. | 13:59 |
*** Guest6564 is now known as dasm | 14:00 | |
dasm | o/ | 14:00 |
rlandy | dasm: you on rbac? | 14:01 |
dasm | about to join | 14:01 |
rlandy | arxcruz: scrum time | 14:01 |
dasm | > "there is no way to detect if srbac is enabled in the cloud" | 14:03 |
*** amoralej|lunch is now known as amoralej | 14:23 | |
dasm | dviroel: i'm seeing different branches for tripleo-ci-base-ovb-rdo-centos-9 (multinode) and tripleo-ci-ovb-rdo-centos-9 (multinode-fips) | 14:38 |
dviroel | dasm: let me check, we can have everything on multinode-fips, with fips disabled, and just enabling on specific jobs | 14:41 |
dasm | k. i'm a bit confused with that | 14:43 |
dviroel | dasm: it is because tripleo-ci-base-ovb-rdo-centos-9 ir parenting off a centos-8 job, and centos-8 will not have fips-enabled for now | 14:47 |
dasm | dviroel: should we clean that up? instead of cs8 -> cs9, create a separate branch for just cs9? | 14:49 |
dasm | dviroel: that's helpful. so basically i need to just change *-fips branch. the other one i need to verify if it uses ovb stable/2.0 | 14:50 |
*** beagles_ is now known as beagles | 14:58 | |
ysandeep | rlandy, marios, chandankumar, dasm: udi and pini wants to meet today, they have some questions around podifed stuff.. you are welcome to join meet.google.com/cmc-xtfi-twq | 15:00 |
ysandeep | dviroel, ^^ | 15:00 |
ysandeep | we are meeting now ^^ | 15:00 |
dviroel | ack | 15:01 |
*** frenzyfriday|rover is now known as frenzyfriday|rover|lunch | 15:01 | |
rlandy | ysandeep: in prod chain | 15:02 |
bhagyashris|ruck | rlandy, updated hackmd leaving for the day | 15:02 |
rlandy | bhagyashris|ruck++ thank you | 15:03 |
marios | thanks ysandeep will skip this one | 15:05 |
*** amoralej is now known as amoralej|off | 15:17 | |
rlandy | frenzyfriday|rover|lunch: look slike both your jobs are hanging on repo setup | 15:21 |
rlandy | sudo dnf reinstall openstack-selinux container-selinux | 15:24 |
rlandy | needs a -y | 15:24 |
rlandy | at end | 15:24 |
rlandy | frenzyfriday|rover|lunch: pls update and recheck | 15:25 |
rlandy | frenzyfriday|rover|lunch: fs064 must have passed on wallaby | 15:26 |
rlandy | master 39 running now | 15:26 |
rlandy | will w+ your patch to skip kvm if that happens | 15:26 |
rlandy | frenzyfriday|rover|lunch: train should promote now | 15:36 |
rlandy | and 16.2 | 15:36 |
*** rlandy is now known as rlandy|brb | 15:40 | |
*** frenzyfriday|rover|lunch is now known as frenzyfriday|rover | 15:51 | |
*** rlandy|brb is now known as rlandy | 15:52 | |
*** tweining is now known as tweining|off | 15:52 | |
frenzyfriday|rover | rlandy, ack, updated -y | 15:54 |
frenzyfriday|rover | rechecked testproj | 15:55 |
rlandy | frenzyfriday|rover++ thanks - let's see if that does it | 15:57 |
rlandy | frenzyfriday|rover: fs039 failed now on tempest | 15:58 |
rlandy | giving it one last try | 15:58 |
rlandy | TASK [os_tempest : Executing python-tempestconf] | 15:59 |
rlandy | frenzyfriday|rover: ^^ seen that error before? | 15:59 |
frenzyfriday|rover | rlandy, tempest sounds good. At least the ipa mirror thing is resolved. I''' recheck another fs64 to be sure | 15:59 |
frenzyfriday|rover | no, checking | 15:59 |
rlandy | k - it's a pain | 16:05 |
rlandy | every now and again | 16:05 |
rlandy | + sudo rpm -V openstack-selinux", "2022-08-01 12:07:05 | package openstack-selinux is not installed" | 16:07 |
rlandy | fails | 16:07 |
rlandy | frenzyfriday|rover: ok ... so .. | 16:08 |
rlandy | sudo dnf reinstall openstack-selinux container-selinux -y ... needs a ; at the end | 16:08 |
rlandy | prevent all the output from printing | 16:09 |
rlandy | and sudo rpm -V openstack-selinux fails | 16:09 |
rlandy | maybe we need an || true type statement | 16:09 |
rlandy | frenzyfriday|rover: ^^ what were you hoping to get from that line? | 16:10 |
frenzyfriday|rover | rlandy, not sure, but probably it will restart and reload the configuration | 16:11 |
frenzyfriday|rover | Maybe I should ask Cedric | 16:11 |
* dviroel lunch | 16:21 | |
rlandy | frenzyfriday|rover: yeah - and hold the node | 16:22 |
*** dviroel is now known as dviroel|lunch | 16:22 | |
rlandy | and try the commands manually | 16:22 |
*** marios is now known as marios|out | 16:24 | |
*** ysandeep is now known as ysandeep|out | 16:26 | |
rlandy | frenzyfriday|rover: rekick the job | 16:28 |
rlandy | hold the node | 16:28 |
rlandy | get on the node | 16:28 |
rlandy | try the command | 16:28 |
frenzyfriday|rover | yep | 16:28 |
rlandy | frenzyfriday|rover; determine what output you want | 16:28 |
frenzyfriday|rover | rlandy, ack, node is help. I am creating a bug for triplo component meanwhile | 16:43 |
frenzyfriday|rover | rlandy, I am eod in a few min. New status on network component master on hackmd | 17:02 |
frenzyfriday|rover | I will check the common issue on the node tomorrow | 17:03 |
rlandy | frenzyfriday|rover: ok - anything I should watch? | 17:07 |
frenzyfriday|rover | rlandy, the network testproj pls https://review.rdoproject.org/r/c/testproject/+/44342 | 17:11 |
rlandy | thank you | 17:11 |
frenzyfriday|rover | If https://review.rdoproject.org/r/c/testproject/+/44342 passes network will promote. Otherwise if it fails with different set of tempest tests we can skip and promote | 17:11 |
frenzyfriday|rover | https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-master/13984cc/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz are the tests that failed last time | 17:12 |
*** jpena is now known as jpena|off | 17:14 | |
*** dviroel|lunch is now known as dviroel | 17:24 | |
rlandy | lunch brb | 17:32 |
rlandy | dviroel: rcastillo: is the downstream cockpit resolving for you? | 18:21 |
dviroel | rlandy: no, just pings | 18:23 |
rlandy | ugh - wonder what happened there | 18:28 |
rlandy | was up this morning | 18:28 |
rlandy | psi? | 18:28 |
dviroel | hum, i was able to ssh | 18:36 |
dviroel | curl returns a page | 18:36 |
jm1 | rlandy, dviroel: checking downstream cockpit | 18:41 |
dviroel | tks jm1 | 18:42 |
rlandy | jm1: thank you | 18:43 |
rlandy | will look again after meeting | 18:43 |
jm1 | rlandy, dviroel: ruck_rover.py is raising an exception | 18:44 |
rlandy | did we merge something there recently? | 18:44 |
rlandy | maybe it's falling over on the 17.1 release? | 18:44 |
rlandy | dasm: ^^? | 18:44 |
dasm | jm1: what kind of exception? | 18:46 |
jm1 | rlandy, dasm: yeah gimme a sec to paste it ^^ | 18:46 |
dasm | rlandy: we did hcange few things. one of them might be the culprit | 18:47 |
rlandy | dasm: may be unrelated - just guessing | 18:47 |
dviroel | lets see the message | 18:47 |
jm1 | rlandy, dasm, dviroel: posted on internal irc | 18:47 |
dasm | i see | 18:48 |
rlandy | meeting - will check back afterwards | 18:48 |
dasm | jm1: worked for me | 18:49 |
dasm | jm1: https://paste.opendev.org/show/bbMJ5XHfFWI76vfNmQpM/ | 18:49 |
dasm | jm1: can you sry again? | 18:49 |
dasm | *try | 18:49 |
dasm | jm1: you can add "--verbose" to see more details | 18:49 |
jm1 | dasm: writing on internal irc due to potentially sensitive data | 18:54 |
dasm | there is nothing sensitive, tbh | 18:55 |
jm1 | dasm: it is an internal system so by default we should not paste hostnames and stuff here.. but yeah, the issue itself is not very sensitive | 18:59 |
jm1 | dasm, dviroel, rlandy: Error: Invalid value for '--release': 'victoria' is not one of 'wallaby', 'train', 'master', 'osp17', 'osp16-2', 'osp17-1'. | 19:12 |
dasm | jm1: victoria is not valid value. | 19:13 |
dasm | jm1: are you reporting journalctl logs? | 19:13 |
jm1 | dasm: docker logs --since "2022-08-01" telegraf_py3 2>&1 | less | 19:13 |
jm1 | dasm: there a several errors | 19:14 |
dasm | some leftovers: https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/rrcockpit/files/telegraf_py3/telegraf.d/jobs_blocking_promotion.conf#L57 | 19:15 |
jm1 | dasm: looks like most of the downstream cockpit is working though | 19:15 |
dasm | jm1: dviroel https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44355 | 19:16 |
dasm | jm1: i don't know what do you mean by "downtstream cockpit is rowking" | 19:16 |
dasm | *working | 19:16 |
dasm | (sorry for my spelling today) | 19:16 |
jm1 | dasm: i can open downstream cockpit and view downstream component pipelines etc. | 19:16 |
dasm | was there an issue with that? | 19:17 |
dasm | i have it opened in the background, i haven't checked it in last few days | 19:17 |
jm1 | dasm: rlandy and I had issues accessing it, but dviroel could open it. i did not get a response via http but after rebooting the vm its working fine | 19:18 |
dasm | ack, that's good | 19:18 |
jm1 | maybe we should ask someone to look for those errors in the telegraf_py3 container. but at least they are not preventing us from using (most) of downstream cockpit | 19:19 |
jm1 | anyway, i am off for today | 19:20 |
dasm | jm1: k | 19:20 |
* jm1 have a nice evening 🥂 | 19:21 | |
dasm | jm1: btw, i see what you meant by errors. in the contaier it's failing on json | 19:21 |
dasm | but when i'm in the container, the same command works w/o issues | 19:22 |
jm1 | dasm: yep and lots more errors ^^ but nice how robust it is. nobody noticed it 😆 | 19:23 |
jm1 | dasm: looks like it requires deeper investigation | 19:23 |
dasm | indeed | 19:23 |
jm1 | dasm: not the most pressuring task i guess | 19:23 |
dasm | i would say, it's pretty high on the priority list | 19:24 |
dasm | esp, that's the main thing which allows us to check for failures | 19:24 |
jm1 | dasm: feel free to create a jira card under infra for it and raise awareness in scrum call | 19:25 |
dasm | yup | 19:25 |
jm1 | dasm: not that i could get too much attention on the other outstanding infra issues though 🤷 | 19:26 |
dasm | jm1: maybe we should tag team? :) | 19:26 |
jm1 | dasm: team := Toll, Ein Anderer Machts 😉 | 19:26 |
dasm | jm1: it sounds like some terrifying spells. I hope you didn't curse me! | 19:27 |
jm1 | dasm: it is something along "great, someone else will do it!" | 19:27 |
dasm | xD | 19:28 |
dasm | it's so called "delegation of work" ;) | 19:28 |
jm1 | ^^ | 19:28 |
dasm | jm1: i just "delegated" two tasks to Jira :P | 19:29 |
* jm1 off for today, now for real 🙈 | 19:29 | |
dasm | take care | 19:29 |
dviroel | jm1: o/ tks | 19:29 |
dviroel | see you all tomorrow o/ | 21:26 |
*** dviroel is now known as dviroel|out | 21:26 | |
dasm | dviroel|out: o/ | 21:26 |
* dasm => off | 21:41 | |
*** dasm is now known as dasm|off | 21:41 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!