Monday, 2022-08-01

*** ysandeep|out is now known as ysandeep02:08
*** ysandeep is now known as ysandeep|afk03:40
*** ysandeep|afk is now known as ysandeep05:20
*** akahat_ is now known as akahat05:39
jm1o/06:25
marios\o06:33
*** ykarel_ is now known as ykarel06:41
*** ysandeep is now known as ysandeep|afk06:51
*** amoralej|off is now known as amoralej07:06
mariosplease add to your reviews thank you https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/851427 07:10
reviewbotDo you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks.07:10
*** ysandeep|afk is now known as ysandeep07:18
*** jpena|off is now known as jpena07:32
*** ysandeep is now known as ysandeep|lunch07:50
*** soniya29 is now known as soniya29|afk08:13
*** ysandeep|lunch is now known as ysandeep09:40
* jm1 bbl, rebooting09:48
*** rlandy_ is now known as rlandy10:33
rlandybhagyashris|ruck: frenzyfriday|rover: hello10:34
frenzyfriday|roverrlandy, hi10:35
rlandybhagyashris|ruck: frenzyfriday|rover: let's sync when you are both around10:35
bhagyashris|ruckrlandy, hi10:35
frenzyfriday|roverrlandy, regarding https://bugs.launchpad.net/tripleo/+bug/1982744/ : there is a suggestioon to add sudo dnf reinstall openstack-selinux container-selinux10:35
frenzyfriday|roversudo rpm -V openstack-selinux as workaround10:35
frenzyfriday|roverdo you know where we can add it? release files?10:35
rlandybhagyashris|ruck: frenzyfriday|rover: https://meet.google.com/diq-dpvh-wcq10:35
bhagyashris|ruckgive me min10:36
rlandybhagyashris|ruck: https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/pipeline/job/pipeline_integration-pcci-16.2_dlrn-rhel-8.4-virthost-3cont_2comp_3ceph-ipv6-geneve-ceph/10:55
rlandyarxcruz: hi - what is the story with https://bugs.launchpad.net/tripleo/+bug/1982874?11:09
rlandycan you update the CIX card11:09
rlandyI think you added to skip list?11:09
rlandybhagyashris|ruck: frenzyfriday|rover: ^^11:10
rlandyarxcruz: hey - can you also explain what's the final word on https://trello.com/c/8tGYExhe/2603-cixlp1980255tripleociproa-tripleo-ci-centos-9-standalone-and-multinode-ipa-are-failing-the-testminimumbasicinstancehardrebootaft?11:12
arxcruzrlandy it's a new but, with tempest update, this test started to run (it was being skipped before because it was marked as unstable)11:14
arxcruzrlandy the second one, it passes, but we need to check when/how we will update tempest to pin on it, because it might bring other failures as well 11:14
arxcruzbut the patch was merged already11:15
rlandyarxcruz: pls join https://meet.google.com/diq-dpvh-wcq11:15
mariosreview call starting now11:17
rlandyarxcruz: for the first one should we add to skiplist11:17
rlandymarios: working with rr - pls continue w/o me11:17
arxcruzrlandy it is already on the skiplist 11:17
ysandeepfrenzyfriday|rover, bhagyashris|ruck, rlandy review time11:17
rlandyysandeep: we are working on rr stuff11:17
rlandypls start w/o us11:17
ysandeepokay o/11:17
rlandyarxcruz: can you join the meeting pls11:18
*** pojadhav is now known as pojadhav|afk11:33
*** dviroel_ is now known as dviroel11:45
rlandyfrenzyfriday|rover: hmmm periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby failed again11:50
rlandyyou can skip promote11:50
rlandywe know it's a transient issue11:50
frenzyfriday|roverrlandy, yes, it is a genuine failure. https://bugs.launchpad.net/tripleo/+bug/198246411:50
frenzyfriday|roverI have held the node11:50
rlandyok - great11:50
frenzyfriday|rover39 is transient11:50
rlandywe don;t need to hole the line11:50
rlandyfrenzyfriday|rover: ok - you can prep a patch to skip kvm and those two test from master and wallaby11:51
rlandyand let's promote those11:51
rlandyand clear that from your plate11:51
frenzyfriday|roverrlandy, ack for the master. But for the wallaby fs64 has an active bug https://bugs.launchpad.net/tripleo/+bug/1982464 Should we still promote?11:52
frenzyfriday|roverhttps://bugs.launchpad.net/tripleo/+bug/1982464 is related to the ipa issue11:52
rlandyfrenzyfriday|rover: is amoralej going to look into it?11:52
rlandyie: the held node?11:52
rlandyfrenzyfriday|rover: if so, let's see after the investigation11:53
amoralejwhat should i look at?11:53
rlandyamoralej: https://bugs.launchpad.net/tripleo/+bug/1982464/comments/111:53
amoralejdo we have some held node?11:53
amoralejthat'd be cool11:53
frenzyfriday|roveramoralej, yep, lemme ping you the IP11:53
rlandyresponding to the comment to hold the node to check dns for package install for ipa11:53
rlandyfrenzyfriday|rover: ^^ can you pass the held node?11:53
rlandygreat11:53
amoralejplease add my pub key11:53
amoralejhttps://github.com/amoralej.keys11:54
rlandybhagyashris|ruck: we can ask dasm or I can run scrum today - since you'll be on CIX11:57
bhagyashris|ruckrlandy, yes i will ask him11:57
rlandybhagyashris|ruck: also when we sync, let's chat about retro schedule11:57
rlandykeeping it on wed or moving to thurs11:57
rlandywill confirm at scrum today11:58
rlandyand let you know11:58
marioso/ tripleo-ci please add to yr reviews when you have time thank you https://review.opendev.org/c/openstack/python-tripleoclient/+/851384 (and depends-on)12:16
reviewbotDo you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks.12:16
*** ysandeep is now known as ysandeep|afk12:23
rlandyfrenzyfriday|rover, I kicked the fs001 train rerun: https://review.rdoproject.org/r/c/testproject/+/3625412:24
frenzyfriday|roverrlandy, thanks!12:25
bhagyashris|ruckrlandy, ack12:25
rlandyfs039 same error: ansible-macros-2021.1.2-2.el9s.noarch: Cannot download, all mirrors were already tried without success12:27
chandankumarysandeep|afk++ adding port to security group fixed the ibm log issue12:29
frenzyfriday|roverrlandy, https://logserver.rdoproject.org/34/44334/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/8e4a00d/job-output.txt and https://logserver.rdoproject.org/37/44337/1/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/20ea2bb/job-output.txt12:31
frenzyfriday|roverone failed with cannot download... the other failed in Run the tripleo-ipa preparation script - same as fs6412:32
rlandytripleo-ipa preparation script - same as fs64 - yep same as 6412:32
rlandybhagyashris|ruck: need any help?12:33
bhagyashris|ruckrlandy, not now thank you :)12:35
frenzyfriday|roverrlandy, the testprojs to check if workaround for the selinux works are up - lets see if they pass13:00
rlandyfrenzyfriday|rover: great - thank you13:10
*** pojadhav|afk is now known as pojadhav13:15
rlandyfrenzyfriday|rover: tripleo component on master looks like legit error13:16
rlandy13>Jul 31 20:38:12 puppet-user: Error: Evaluation Error: Error while evaluating a Resource Statement, Apache::Vhost[placement_wsgi]:13:16
rlandy<13>Jul 31 20:38:12 puppet-user:   parameter 'port' expects a value of type Undef, Array[Stdlib::Port = Integer[0, 65535]], or Stdlib::Port = Integer[0, 65535], got String13:16
rlandy<13>Jul 31 20:38:12 puppet-user:   parameter 'priority' expects a value of type Undef, Integer, or Boolean, got String13:16
rlandy<13>Jul 31 20:38:12 puppet-user:   parameter 'access_log_file' expects a value of type Undef or String, got Boolean13:16
rlandy<13>Jul 31 20:38:12 puppet-user:   parameter 'access_log_pipe' expects a value of type Undef or String, got Boolean (file: /etc/puppet/modules/openstacklib/manifests/wsgi/apache.pp, line: 357) on node standalone.localdomain13:17
rlandy+ rc=113:17
rlandy+ '[' false = false ']'13:17
rlandy+ set +x13:17
rlandy2022-07-31 16:38:21.001651 | fa163ecf-73d9-df2e-85fe-00000000211a |      FATAL | Manage Puppet containers (generate config) for step 1 | standalone | error={"changed": false, "msg": "Failed containers: container-puppet-horizon, container-puppet-keystone, container-puppet-manila, container-puppet-neutron, container-puppet-nova, container-puppet-nova_metadata, container-puppet-placement"}13:17
rlandypls LP bug and ping tkajinum13:17
rlandyfrenzyfriday|rover: ^^13:17
rlandyfrenzyfriday|rover: bhagyashris|ruck: let's sync again at half past the hour13:18
rlandyie: in 12 minutes13:18
rlandybhagyashris|ruck: pls add integration line info plus any testproject reruns for downstream on hackmd13:19
frenzyfriday|roverrlandy, ack, checking13:20
frenzyfriday|roverrlandy, I am creating a bug for tripleo  master13:23
bhagyashris|ruckrlandy, ack13:29
bhagyashris|ruckrlandy, yes13:29
*** ysandeep|afk is now known as ysandeep13:33
ysandeepchandankumar: great!13:34
frenzyfriday|roverbhagyashris|ruck, rlandy do you want to sync now?13:34
rlandyfrenzyfriday|rover: bhagyashris|ruck: ack https://meet.google.com/imr-duwb-csh?pli=1&authuser=013:34
akahatrlandy, I've closed this bug just now.. https://bugzilla.redhat.com/show_bug.cgi?id=2110535, I just saw your comments on the CIX card. 13:35
akahatrlandy, should i keep this bug open for Infrared?13:35
*** amoralej is now known as amoralej|lunch13:37
rlandyakahat; yeah - was keeping it open for infrared13:58
rlandybut ok if you closed it13:58
rlandythe CIX is still open13:58
rlandyand they can reopen if need be13:58
akahatrlandy, ok.13:59
*** Guest6564 is now known as dasm14:00
dasmo/14:00
rlandydasm: you on rbac?14:01
dasmabout to join14:01
rlandyarxcruz: scrum time14:01
dasm> "there is no way to detect if srbac is enabled in the cloud"14:03
*** amoralej|lunch is now known as amoralej14:23
dasmdviroel: i'm seeing different branches for tripleo-ci-base-ovb-rdo-centos-9 (multinode) and tripleo-ci-ovb-rdo-centos-9 (multinode-fips)14:38
dviroeldasm: let me check, we can have everything on multinode-fips, with fips disabled, and just enabling on specific jobs14:41
dasmk. i'm a bit confused with that14:43
dviroeldasm: it is because tripleo-ci-base-ovb-rdo-centos-9 ir parenting off a centos-8 job, and centos-8 will not have fips-enabled for now14:47
dasmdviroel: should we clean that up? instead of cs8 -> cs9, create a separate branch for just cs9?14:49
dasmdviroel: that's helpful. so basically i need to just change *-fips branch. the other one i need to verify if it uses ovb stable/2.014:50
*** beagles_ is now known as beagles14:58
ysandeeprlandy, marios, chandankumar, dasm:  udi and pini wants to meet today, they have some questions around podifed stuff.. you are welcome to join meet.google.com/cmc-xtfi-twq 15:00
ysandeepdviroel, ^^15:00
ysandeepwe are meeting now ^^15:00
dviroelack15:01
*** frenzyfriday|rover is now known as frenzyfriday|rover|lunch15:01
rlandyysandeep: in prod chain15:02
bhagyashris|ruckrlandy, updated hackmd leaving for the day 15:02
rlandybhagyashris|ruck++ thank you15:03
mariosthanks ysandeep will skip this one 15:05
*** amoralej is now known as amoralej|off15:17
rlandyfrenzyfriday|rover|lunch: look slike both your jobs are hanging on repo setup15:21
rlandysudo dnf reinstall openstack-selinux container-selinux15:24
rlandyneeds a -y15:24
rlandyat end15:24
rlandyfrenzyfriday|rover|lunch: pls update and recheck15:25
rlandyfrenzyfriday|rover|lunch: fs064 must have passed on wallaby15:26
rlandymaster 39 running now15:26
rlandywill w+ your patch to skip kvm if that happens15:26
rlandyfrenzyfriday|rover|lunch: train should promote now15:36
rlandyand 16.215:36
*** rlandy is now known as rlandy|brb15:40
*** frenzyfriday|rover|lunch is now known as frenzyfriday|rover15:51
*** rlandy|brb is now known as rlandy15:52
*** tweining is now known as tweining|off15:52
frenzyfriday|roverrlandy, ack, updated -y15:54
frenzyfriday|roverrechecked testproj15:55
rlandyfrenzyfriday|rover++ thanks - let's see if that does it15:57
rlandyfrenzyfriday|rover: fs039 failed  now on tempest15:58
rlandygiving it one last try15:58
rlandyTASK [os_tempest : Executing python-tempestconf] 15:59
rlandyfrenzyfriday|rover: ^^ seen that error before?15:59
frenzyfriday|roverrlandy, tempest sounds good. At least the ipa mirror thing is resolved. I''' recheck another fs64 to be sure15:59
frenzyfriday|roverno, checking15:59
rlandyk - it's a pain16:05
rlandyevery now and again16:05
rlandy+ sudo rpm -V openstack-selinux", "2022-08-01 12:07:05 | package openstack-selinux is not installed"16:07
rlandyfails16:07
rlandyfrenzyfriday|rover: ok ... so ..16:08
rlandysudo dnf reinstall openstack-selinux container-selinux -y ... needs a ; at the end16:08
rlandyprevent all the output from printing16:09
rlandyand   sudo rpm -V openstack-selinux fails16:09
rlandymaybe we need an || true type statement16:09
rlandyfrenzyfriday|rover: ^^ what were you hoping to get from that line?16:10
frenzyfriday|roverrlandy, not sure, but probably it will restart and reload the configuration 16:11
frenzyfriday|roverMaybe I should ask Cedric 16:11
* dviroel lunch16:21
rlandyfrenzyfriday|rover: yeah - and hold the node 16:22
*** dviroel is now known as dviroel|lunch16:22
rlandyand try the commands manually16:22
*** marios is now known as marios|out16:24
*** ysandeep is now known as ysandeep|out16:26
rlandyfrenzyfriday|rover: rekick the job16:28
rlandyhold the node16:28
rlandyget on the node16:28
rlandytry the command16:28
frenzyfriday|roveryep16:28
rlandyfrenzyfriday|rover; determine what output you want16:28
frenzyfriday|roverrlandy, ack, node is help. I am creating a bug for triplo component meanwhile16:43
frenzyfriday|roverrlandy, I am eod in a few min. New status on network component master on hackmd17:02
frenzyfriday|roverI will check the common issue on the node tomorrow17:03
rlandyfrenzyfriday|rover: ok - anything I should watch?17:07
frenzyfriday|roverrlandy, the network testproj pls https://review.rdoproject.org/r/c/testproject/+/4434217:11
rlandythank you17:11
frenzyfriday|roverIf https://review.rdoproject.org/r/c/testproject/+/44342 passes network will promote. Otherwise if it fails with different set of tempest tests we can skip and promote17:11
frenzyfriday|roverhttps://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-master/13984cc/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz are the tests that failed last time17:12
*** jpena is now known as jpena|off17:14
*** dviroel|lunch is now known as dviroel17:24
rlandylunch brb17:32
rlandydviroel: rcastillo: is the downstream cockpit resolving for you?18:21
dviroelrlandy: no, just pings18:23
rlandyugh - wonder what happened there18:28
rlandywas up this morning18:28
rlandypsi?18:28
dviroelhum, i was able to ssh18:36
dviroelcurl returns a page18:36
jm1rlandy, dviroel: checking downstream cockpit18:41
dviroeltks jm1 18:42
rlandyjm1: thank you18:43
rlandywill look again after meeting18:43
jm1rlandy, dviroel: ruck_rover.py is raising an exception18:44
rlandydid we merge something there recently?18:44
rlandymaybe it's falling over on the 17.1 release?18:44
rlandydasm: ^^?18:44
dasmjm1: what kind of exception?18:46
jm1rlandy, dasm: yeah gimme a sec to paste it ^^18:46
dasmrlandy: we did hcange few things. one of them might be the culprit18:47
rlandydasm: may be unrelated - just guessing18:47
dviroellets see the message18:47
jm1rlandy, dasm, dviroel: posted on internal irc18:47
dasmi see18:48
rlandymeeting - will check back afterwards18:48
dasmjm1: worked for me18:49
dasmjm1: https://paste.opendev.org/show/bbMJ5XHfFWI76vfNmQpM/18:49
dasmjm1: can you sry again?18:49
dasm*try18:49
dasmjm1: you can add "--verbose" to see more details18:49
jm1dasm: writing on internal irc due to potentially sensitive data18:54
dasmthere is nothing sensitive, tbh18:55
jm1dasm: it is an internal system so by default we should not paste hostnames and stuff here.. but yeah, the issue itself is not very sensitive18:59
jm1dasm, dviroel, rlandy: Error: Invalid value for '--release': 'victoria' is not one of 'wallaby', 'train', 'master', 'osp17', 'osp16-2', 'osp17-1'.19:12
dasmjm1: victoria is not valid value.19:13
dasmjm1: are you reporting journalctl logs?19:13
jm1dasm: docker logs --since "2022-08-01" telegraf_py3 2>&1 | less19:13
jm1dasm: there a several errors19:14
dasmsome leftovers: https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/rrcockpit/files/telegraf_py3/telegraf.d/jobs_blocking_promotion.conf#L5719:15
jm1dasm: looks like most of the downstream cockpit is working though19:15
dasmjm1: dviroel https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4435519:16
dasmjm1: i don't know what do you mean by "downtstream cockpit is rowking"19:16
dasm*working19:16
dasm(sorry for my spelling today)19:16
jm1dasm: i can open downstream cockpit and view downstream component pipelines etc. 19:16
dasmwas there an issue with that?19:17
dasmi have it opened in the background, i haven't checked it in last few days19:17
jm1dasm: rlandy and I had issues accessing it, but dviroel could open it. i did not get a response via http but after rebooting the vm its working fine19:18
dasmack, that's good19:18
jm1maybe we should ask someone to look for those errors in the telegraf_py3 container. but at least they are not preventing us from using (most) of downstream cockpit19:19
jm1anyway, i am off for today19:20
dasmjm1: k19:20
* jm1 have a nice evening 🥂19:21
dasmjm1: btw, i see what you meant by errors. in the contaier it's failing on json19:21
dasmbut when i'm in the container, the same command works w/o issues19:22
jm1dasm: yep and lots more errors ^^ but nice how robust it is. nobody noticed it 😆19:23
jm1dasm: looks like it requires deeper investigation19:23
dasmindeed19:23
jm1dasm: not the most pressuring task i guess19:23
dasmi would say, it's pretty high on the priority list19:24
dasmesp, that's the main thing which allows us to check for failures19:24
jm1dasm: feel free to create a jira card under infra for it and raise awareness in scrum call19:25
dasmyup19:25
jm1dasm: not that i could get too much attention on the other outstanding infra issues though 🤷19:26
dasmjm1: maybe we should tag team? :)19:26
jm1dasm: team := Toll, Ein Anderer Machts 😉19:26
dasmjm1: it sounds like some terrifying spells. I hope you didn't curse me!19:27
jm1dasm: it is something along "great, someone else will do it!"19:27
dasmxD19:28
dasmit's so called "delegation of work" ;)19:28
jm1^^19:28
dasmjm1: i just "delegated" two tasks to Jira :P19:29
* jm1 off for today, now for real 🙈19:29
dasmtake care19:29
dviroeljm1: o/ tks19:29
dviroelsee you all tomorrow o/21:26
*** dviroel is now known as dviroel|out21:26
dasmdviroel|out: o/21:26
* dasm => off21:41
*** dasm is now known as dasm|off21:41

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!