| *** ysandeep|out is now known as ysandeep | 01:54 | |
| ysandeep | dasm|off, o/ still slow for me: http://pastebin.test.redhat.com/1068548 | 02:48 |
|---|---|---|
| ysandeep | replied here: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43161/17#message-af254741d5431408ec72565fb39f27932cbb04e3 | 02:59 |
| *** ysandeep is now known as ysandeep|afk | 03:24 | |
| dasm|off | ysandeep|afk: it takes 2s per logs to download. hm... | 03:24 |
| dasm|off | ysandeep|afk: i reworked patches. can you check other ones? i'm gonna see how to solve this issue. | 03:25 |
| dasm|off | i | 03:25 |
| dasm|off | i'm not sure if it can be done more efficient. | 03:26 |
| dasm|off | ysandeep|afk: how long does it take for you to open any logserver? | 03:26 |
| dasm|off | ysandeep|afk: i recently had issues when tried to use ipv6 over vpn. it took a long time. | 03:26 |
| dasm|off | ysandeep|afk: could you try again w/o vpn? | 03:27 |
| *** ysandeep|afk is now known as ysandeep | 03:38 | |
| ysandeep | dasm|off, took 1m 59s without vpn.. so looks like vpn is not a factor here | 03:39 |
| dasm|off | ack | 03:39 |
| ysandeep | dasm|off, please check my comment here: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43161/17#message-af254741d5431408ec72565fb39f27932cbb04e3 , download task is not taking long but query_zuul_job_details | 03:39 |
| ysandeep | dasm|off, yes I am reviewing your other patches.. left a comment here as well: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/42149/6#message-4d63cf5908f8944723baa187f6dfa7a0418b4a74 | 03:41 |
| dasm|off | ysandeep: so query_zuul takes long time? | 03:41 |
| ysandeep | yeah it took ~29 s | 03:41 |
| dasm|off | so it's not fetching failure_reason? | 03:41 |
| dasm|off | i thought it was failure reason before | 03:42 |
| ysandeep | yes in this case - downloading didn't take time because there was nothing to download :) that file itself was not present | 03:42 |
| dasm|off | mhm | 03:42 |
| dasm|off | ysandeep: and what happens wnen you're opening links to logserver and review? does it take so long too? | 03:43 |
| dasm|off | i had issues with ipv6 | 03:45 |
| ysandeep | dasm|off, I don't feel any slowness in general when using webui | 03:45 |
| dasm|off | ysandeep: can you disable ipv6 on your network and try again via console? | 03:46 |
| ysandeep | net.ipv6.conf.all.disable_ipv6 = 1 ? | 03:48 |
| dasm|off | i changed via gui in network manager. i'm not sure about cli option | 03:48 |
| * dasm|off checking | 03:48 | |
| dasm|off | > sudo sysctl -w net.ipv6.conf.eth0.disable_ipv6=1 | 03:49 |
| dasm|off | ysandeep: yes, you're right | 03:49 |
| * ysandeep doing some tests | 03:53 | |
| ysandeep | dasm|off, http://pastebin.test.redhat.com/1068554 1 m 14s | 03:54 |
| dasm|off | hmm, faster, but no cake | 03:54 |
| * dasm|off can't check pastebin -- i'm on my private pc now | 03:54 | |
| ysandeep | let me share in openstack pastebin.. no secrets here :D | 03:55 |
| ysandeep | https://paste.openstack.org/show/bRgyMBrGlMFZXEK3fdjJ/ 1m 20 s in latest test | 03:56 |
| dasm|off | checking | 03:56 |
| dasm|off | 4s for promotion info | 03:57 |
| dasm|off | job details = 22s | 03:57 |
| dasm|off | so 'query_zuul_jobs_details" takes long time | 03:58 |
| ysandeep | yes.. fetching things from zuul takes time.. | 03:58 |
| dasm|off | and it's about 45s total. which is over half of overall time execution | 03:59 |
| dasm|off | checking the same on my config | 03:59 |
| dasm|off | 4s for fetching jobs details | 04:00 |
| ysandeep | so now I am skeptical about removing compare_upstream flag here.. https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43307/14#message-b4f8280c0ebd45b4e106ccad4eca145ab6693706 :D | 04:00 |
| ysandeep | that will increase the time even more :D | 04:00 |
| ysandeep | imho.. we should keep separate function for influx and for print | 04:00 |
| ysandeep | +1 to unification but if that decrease performance and we don't use that info during print.. | 04:01 |
| dasm|off | we're sharing too many variables between those two to keep separate functions | 04:01 |
| dasm|off | currently, when i want to fetch info about last 5 promotion candidates, it's difficult | 04:02 |
| ysandeep | true.. but currently we are only doing what we have to for print.. no extra things | 04:03 |
| dasm|off | i would need to rerun entire script 5 times to get the same info. even though, we should be able to fetch all of that at once, maybe twice | 04:03 |
| dasm|off | ues, that's currently the case. | 04:04 |
| dasm|off | but the unification is not the source of problem here | 04:04 |
| dasm|off | for me, running on master branch takes the same time like running with changes. | 04:05 |
| dasm|off | hence i tried to understand if the issue mignt be caused by network | 04:05 |
| ysandeep | I think you are closer to zuul server and logserver | 04:06 |
| dasm|off | i know what i can try. i can try connecting through some vpn over in Asia | 04:06 |
| dasm|off | ysandeep: that would be the case, if you would see the same slowness when connecting through webgyi | 04:06 |
| dasm|off | if you're experiencing that only through the console, i'm not so sure anymore | 04:07 |
| ysandeep | lets both try to download a file from rdo logserver and see the difference in timing | 04:08 |
| dasm|off | ysandeep: can you add "logging.debug(response.url)" to query_zuul_job_details? it should give you the link. you might try opening that in the browser and see how long toes it take | 04:08 |
| dasm|off | ysandeep: ok | 04:09 |
| dasm|off | ysandeep: time curl https://review.rdoproject.org/zuul/api/builds?job_name=periodic-tripleo-ci-centos-9-scenario004-standalone-common-master 1>/dev/null 2>&1 | 04:11 |
| dasm|off | try this | 04:11 |
| dasm|off | > real 0m1.225s | 04:11 |
| dasm|off | it takes for me about 1.6s top | 04:11 |
| ysandeep | taking ~4s here | 04:12 |
| dasm|off | so 4x O/ | 04:12 |
| ysandeep | real0m3.820s | 04:12 |
| dasm|off | :/ | 04:12 |
| dasm|off | can you open the same link in the browser and see how long does it take | 04:12 |
| dasm|off | in the browser it took me 962ms | 04:14 |
| ysandeep | dasm|off, how are you calculating that? | 04:14 |
| dasm|off | fx or chrome have Developer Tools | 04:14 |
| dasm|off | for fx it's called "Inspect" | 04:15 |
| dasm|off | there is a Network tab with info about the time | 04:15 |
| ysandeep | thanks, let me take a look | 04:16 |
| ysandeep | dasm|off, yeah around same time | 04:17 |
| dasm|off | > Finish: 2.39 s | 04:17 |
| dasm|off | this is what i got right now | 04:17 |
| dasm|off | ysandeep: the same, you mean to the console? | 04:17 |
| ysandeep | yes with the console - ~ 4 - 5 secs | 04:18 |
| dasm|off | ack | 04:18 |
| dasm|off | ysandeep: so, your hypothesis about server proximity is the right one | 04:18 |
| dasm|off | ysandeep: can you see how long does it take for master cs9 now? | 04:19 |
| ysandeep | yeah looks like latency is a factor here | 04:19 |
| ysandeep | dasm|off, integration line? | 04:19 |
| dasm|off | yes | 04:20 |
| ysandeep | dasm|off, are you not late for sleep? :D | 04:21 |
| dasm|off | ysandeep: watching "Logan" | 04:21 |
| dasm|off | it's 21:21 here | 04:21 |
| ysandeep | dasm|off, took real1m9.727s with your reduce parametes patch | 04:21 |
| dasm|off | > real 0m23.288s | 04:22 |
| dasm|off | 3$ time CURL_CA_BUNDLE="" ./ruck_rover.py 1>/dev/null 2>&1 | 04:22 |
| dasm|off | real 0m21.220s | 04:23 |
| ysandeep | real0m32.295s with master branch | 04:23 |
| dasm|off | ysandeep: i'm gonna revisit what i can do with that. | 04:23 |
| ysandeep | dasm|off, I know its late for you.. have a good rest | 04:23 |
| dasm|off | ysandeep: so, you're saying -- UX >> simplicity? :) | 04:24 |
| ysandeep | yeah.. imho.. user experience >> simplicity of code | 04:24 |
| ysandeep | i don't mind devs writing complex code as far as the end product is simple to use and UX is great | 04:25 |
| dasm|off | ysandeep: would you be able to accept temporary (couple weeks) slowdown of the script, to improve the code, and later allow to reduce its complexity? | 04:27 |
| dasm|off | ysandeep: kinda like 2-step approach. one -- make it easier, and later decouple it into smaller functions? | 04:27 |
| dasm|off | like i mentioned -- right now there is a lot of interdependencies, which i'm trying to untangle | 04:28 |
| ysandeep | absolutely that works for me | 04:28 |
| ysandeep | what do you think of https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43161/17 as a temporary workaround? | 04:29 |
| dasm|off | ysandeep: keeping "influx"? but you mentioned that bigger issue is with "job details" | 04:30 |
| ysandeep | can we not toggle "job details" based on "influx" | 04:31 |
| dasm|off | i see | 04:31 |
| dasm|off | ysandeep: i'll prepare the change (tomorrow) | 04:32 |
| ysandeep | dasm|off, o/ thanks | 04:32 |
| ysandeep | good night | 04:32 |
| dasm|off | good day! | 04:32 |
| *** ysandeep is now known as ysandeep|break | 04:32 | |
| bhagyashris | akahat|ruck, arxcruz|rover let me know once you login for rr sync | 05:19 |
| *** ysandeep|break is now known as ysandeep | 05:36 | |
| *** bhagyashris is now known as bhagyashris|ruck | 06:05 | |
| jm1 | happy friday :) | 06:15 |
| ysandeep | jm1: good morning and happy friday o/ | 06:25 |
| soniya29 | jm1, ysandeep happy friday :) | 06:32 |
| ysandeep | soniya29: o/ you too | 06:32 |
| jm1 | soniya29, ysandeep: o/ 🍻 😄 | 06:37 |
| akahat|ruck | bhagyashris|ruck, o/ | 06:42 |
| akahat|ruck | bhagyashris|ruck, do you want to sync now? | 06:43 |
| bhagyashris|ruck | akahat|ruck, may be after 30 mins | 06:47 |
| bhagyashris|ruck | going for lunch | 06:47 |
| *** ysandeep is now known as ysandeep|lunch | 07:14 | |
| arxcruz|rover | bhagyashris|ruck i need 10-20 min | 07:15 |
| bhagyashris|ruck | arxcruz|rover, akahat|ruck i am back | 07:15 |
| bhagyashris|ruck | let me know when we meet? | 07:21 |
| arxcruz|rover | bhagyashris|ruck akahat|ruck https://meet.google.com/hgt-rfdb-qgt?authuser=1 | 07:30 |
| bhagyashris|ruck | arxcruz|rover, hey i can't hear you... | 07:32 |
| arxcruz|rover | akahat|ruck around? | 07:38 |
| arxcruz|rover | akahat|ruck around? | 07:58 |
| marios | may disconnect for a bit ... isp is upgrading my connection hopefully back soon ;) | 08:15 |
| *** jpena|off is now known as jpena | 08:45 | |
| *** akahat|ruck is now known as akahat | 09:12 | |
| arxcruz|rover | bhagyashris|ruck periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby passed, one less to go | 09:17 |
| bhagyashris|ruck | arxcruz|rover, yoo thanks! | 09:17 |
| arxcruz|rover | fs39 fails... :( | 09:17 |
| arxcruz|rover | if rlandy|out agrees, we can skip fs39 and promote wallaby c9 | 09:18 |
| arxcruz|rover | i'll wait fs001 finishes then recheck anyway | 09:19 |
| bhagyashris|ruck | arxcruz|rover, ack thanks | 09:20 |
| marios | thanks ysandeep|lunch for early recheck on my non-voting patch it got through | 09:30 |
| *** ysandeep|lunch is now known as ysandeep | 10:21 | |
| ysandeep | marios :D | 10:23 |
| *** rlandy|out is now known as rlandy | 10:35 | |
| rlandy | bhagyashris|ruck: arxcruz|rover: want to sync? | 10:38 |
| rlandy | bhagyashris|ruck: is dviroel|afk covering today? | 10:38 |
| rlandy | ananya still out? | 10:38 |
| rlandy | bhagyashris|ruck: looking at failing rhel-9 components | 10:57 |
| rlandy | also pls fill in promotion status on rr hackmd | 10:58 |
| rlandy | yes - you can skip fs039 | 10:58 |
| rlandy | is it related to bug on cix ? | 10:58 |
| rlandy | bhagyashris|ruck: https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/421973 Fix parameter format for registry proxy in 17 and 17.1 | 11:03 |
| marios | amoralej: (when you have time please) can you check https://review.opendev.org/c/openstack/python-tripleoclient/+/851366 do you think it would cause any problems for rdo package builds (removed version specific classifiers) | 11:12 |
| bhagyashris|ruck | rlandy, give me min | 11:12 |
| amoralej | lemme check | 11:12 |
| amoralej | marios, i think it's not a problem from rdo packaging point of view | 11:13 |
| amoralej | but i'm not sure if it's a good practice | 11:14 |
| amoralej | lemme take a look | 11:14 |
| rlandy | bhagyashris|ruck: fixed the failing jobs I think | 11:14 |
| rlandy | rekicked what was missing in therhle-9 line | 11:15 |
| amoralej | btw, you also publish it in pypi, i'm not sure if it can be a problem for pip install it | 11:15 |
| rlandy | need to join review time | 11:15 |
| rlandy | will sync with you afterwards | 11:15 |
| *** dviroel|afk is now known as dviroel | 11:15 | |
| marios | thanks amoralej | 11:15 |
| amoralej | marios, btw, that will not fix https://bugs.launchpad.net/tripleo/+bug/1983004 | 11:16 |
| rlandy | review time | 11:16 |
| amoralej | thar's related to mixing centos8 and centos9 builds | 11:17 |
| dviroel | rlandy: bhagyashris|ruck - yes i will cover ananya today | 11:18 |
| *** dviroel is now known as dviroel|rover | 11:18 | |
| rlandy | dviroel|rover: bhagyashris|ruck: on review time | 11:18 |
| bhagyashris|ruck | dviroel|rover, ack | 11:18 |
| rlandy | need to get ananya's reviews in | 11:18 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 11:18 |
| rlandy | let's sync in a bit | 11:19 |
| bhagyashris|ruck | sure | 11:19 |
| rlandy | bhagyashris|ruck: in the mean time, pls update the rr hackmd | 11:19 |
| rlandy | with promotion status etc. | 11:19 |
| marios | amoralej: thanks i am in call but ... grateful if you have some info add into the bug please? | 11:24 |
| marios | amoralej: re (that will not fix ) | 11:24 |
| marios | amoralej: thanks for checking | 11:24 |
| amoralej | marios, i replied in the review. | 11:24 |
| amoralej | but i'll also update the bz | 11:25 |
| amoralej | lemme see what those jobs are doing ... | 11:25 |
| marios | thank you amoralej i can check the review | 11:25 |
| marios | amoralej: thanks but it is c9 there replied https://review.opendev.org/c/openstack/python-tripleoclient/+/851366/4#message-7a357ceeef32a16e3b0f215059367c5ba31ba154 | 11:27 |
| marios | amoralej: like the job has 3.9 installed | 11:27 |
| amoralej | marios, exactly, the host is centos9 and the package .el8 | 11:28 |
| amoralej | so for cs8 | 11:28 |
| marios | oh.. .didn't notice that amoralej | 11:29 |
| marios | :) | 11:29 |
| amoralej | i mean https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_2bc/851240/1/check/tripleo-ci-centos-8-9-multinode-mixed-os/2bcce46/logs/undercloud/home/zuul/install_packages.sh.log | 11:29 |
| marios | python3-tripleoclient-16.4.1-0.20220728031741.bce8cb7.el8.noarch | 11:29 |
| amoralej | i left a comment in the bug also | 11:29 |
| amoralej | yep, that | 11:29 |
| marios | i dind't notice it was pulling an el8 package | 11:29 |
| marios | that is the problem then | 11:29 |
| amoralej | right | 11:29 |
| marios | thank you amoralej :) i will dig at that then | 11:29 |
| amoralej | some issue with repos definition i guess ... | 11:29 |
| amoralej | what are those jobs trying to do, btw? :) | 11:29 |
| amoralej | compute nodes in cs8 and controllers in cs9 i guess? | 11:30 |
| marios | amoralej: yeah c8 for compute only | 11:33 |
| marios | amoralej: but undercloud delorean-current is correct https://7e2d13ca2835f2277cf7-527af0e460b15f09b47abe14cd6cea3c.ssl.cf5.rackcdn.com/851366/3/check/tripleo-ci-centos-8-9-multinode-mixed-os/a63148b/logs/undercloud/etc/yum.repos.d/delorean-current.repo | 11:34 |
| marios | and it has the el9 repo | 11:34 |
| marios | amoralej: will dig after this call | 11:34 |
| marios | amoralej: not sure why it is trying to pull el8 (or even which repo ) | 11:34 |
| marios | (and only on tripleoclient :/) | 11:34 |
| amoralej | mmm maybe it's built by dlrn in the same job? | 11:36 |
| amoralej | it seems it doesn't | 11:39 |
| amoralej | weird | 11:39 |
| rlandy | bhagyashris|ruck: dviroel|rover: rr sync https://meet.google.com/xow-evuj-adf | 11:49 |
| *** soniya is now known as soniya|ruck | 11:52 | |
| *** soniya|ruck is now known as soniya|afk | 11:52 | |
| ysandeep | rlandy, fyi.. As discussed left a comment for dasm|off here: https://review.rdoproject.org/r/c/rdo-jobs/+/44153/4#message-ea7b7002218ad1c1106e05a8d0644ed664daaae1 | 11:52 |
| *** frenzy_friday is now known as frenzyfriday|rover | 11:54 | |
| frenzyfriday|rover | hey bhagyashris|ruck , I am back | 11:59 |
| frenzyfriday|rover | anything I should look at first? | 12:00 |
| bhagyashris|ruck | may be downstream | 12:00 |
| bhagyashris|ruck | i just checked the failing jobs and rekicked the jobs | 12:00 |
| bhagyashris|ruck | haven't get chance to looks deeply | 12:00 |
| bhagyashris|ruck | frenzyfriday|rover, ^ | 12:01 |
| bhagyashris|ruck | frenzyfriday|rover, here is hackmd https://hackmd.io/H9CSoXvlTm6nTZ4bsJkeRg | 12:01 |
| dviroel|rover | frenzyfriday|rover: hey, I just started :) | 12:01 |
| bhagyashris|ruck | updated | 12:01 |
| frenzyfriday|rover | cool :) | 12:02 |
| frenzyfriday|rover | dviroel|rover, 0/ no worries, I am back now | 12:02 |
| dviroel|rover | it is a rover party \o/ arxcruz|rover frenzyfriday|rover | 12:02 |
| frenzyfriday|rover | XD !! | 12:03 |
| akahat | bhagyashris|ruck, https://bugzilla.redhat.com/show_bug.cgi?id=2112344 | 12:25 |
| rlandy | ysandeep: thank you | 12:34 |
| rlandy | dviroel|rover: frenzyfriday|rover: you guys tag-teaming? | 12:35 |
| rlandy | bhagyashris|ruck; frenzyfriday|rover;just did a sync with dviroel|rover | 12:35 |
| rlandy | chandankumar: hey ... still want to meet? | 12:36 |
| rlandy | chandankumar: have 20 mins now | 12:37 |
| rlandy | let's try spring you free | 12:37 |
| chandankumar | rlandy: yes | 12:37 |
| frenzyfriday|rover | rlandy, yep, dviroel|rover was covering me. I am back now | 12:38 |
| dviroel|rover | yeah, will stay rover for a bit | 12:38 |
| rlandy | chandankumar: https://meet.google.com/jju-xyba-jkz?pli=1&authuser=0 | 12:39 |
| akahat | dviroel|rover, o/ | 12:41 |
| akahat | dviroel|rover, https://sf.hosted.upshift.rdu2.redhat.com/logs/71/421971/1/check/periodic-tripleo-ci-rhel-9-scenario010-standalone-octavia-rhos-17/3504f78/logs/undercloud/home/zuul/containers-prepare-parameters.yaml | 12:42 |
| akahat | dviroel|rover, may be this caused it: https://opendev.org/openstack/tripleo-quickstart-extras/commit/df634354a6cd79c93d1cb0dd984ae766c687834f | 12:42 |
| bhagyashris|ruck | rlandy, ack | 12:42 |
| akahat | all rhel-9 jobs are affected by this | 12:42 |
| dviroel|rover | akahat: let me check | 12:42 |
| akahat | this is invalid yaml:ceph_namespace: docker_ceph_namespace: registry-proxy.engineering.redhat.com/rh-osbs | 12:43 |
| ysandeep | akahat, rlandy merged a patch to fix ^^ | 12:45 |
| ysandeep | akahat, https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/421973 | 12:45 |
| akahat | oh.. | 12:46 |
| akahat | ysandeep, thanks | 12:47 |
| dviroel|rover | ysandeep: akahat: yeah, looks like this was the issue, lets wait | 12:48 |
| akahat | dviroel|rover, yes... i was searching upstream. | 12:48 |
| ysandeep | rlandy, dasm|off found something interesting.. https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/environments/enable-secure-rbac.yaml#L2 "EnforceSecureRbac: false" when talking with Pranali (she reached out to me - she wanted to enable some rbac related tempest tests) | 12:49 |
| ysandeep | Are we really enabling srbac completely in fs020-rbac job? | 12:49 |
| rlandy | ysandeep: so we have an overall switch | 12:49 |
| rlandy | to enable rbac per job | 12:49 |
| rlandy | per tempest test | 12:49 |
| *** amoralej is now known as amoralej|lunch | 12:50 | |
| rlandy | there may be an special switch | 12:50 |
| rlandy | as some tempest test are rbac-specifc | 12:50 |
| rlandy | or ill totally fail | 12:50 |
| rlandy | to answer your question - dasm|off would know more updated info - as long as the test won't fail, we can def enable it | 12:51 |
| rlandy | she can testproject check that | 12:51 |
| ysandeep | rlandy, she already did - deployment failed on cinder trying to enable rbac with "EnforceSecureRbac: true" | 12:51 |
| ysandeep | https://logserver.rdoproject.org/04/40004/13/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-rbac-master/e16c411/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 12:52 |
| rlandy | did she try wallaby? | 12:52 |
| rlandy | ysandeep: let's wait for dasm|off to come on line | 12:52 |
| rlandy | then dicuss | 12:52 |
| rlandy | discuss | 12:52 |
| ysandeep | she tried on master | 12:52 |
| ysandeep | fyi.. this was the test project: https://review.rdoproject.org/r/c/rdo-jobs/+/40004/13//COMMIT_MSG | 12:52 |
| ysandeep | anyway I have requested to rerun the job once more to confirm its legit bug(looks like it is from traceback) and raise the failure in SRBAC mtg | 12:54 |
| ysandeep | we will need security/cinder folks help | 12:54 |
| rlandy | ok | 12:54 |
| ysandeep | dasm|off, we need to discuss with security why "EnforceSecureRbac" is still false here: https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/environments/enable-secure-rbac.yaml#L2 | 12:55 |
| ysandeep | this switch ^^ toggle rbac for different services | 12:55 |
| ysandeep | to me looks like today we are just adding some custom policies via enable-secure-rbac.yam for various services but not actually enabling srbac completely. | 12:56 |
| rlandy | bhagyashris|ruck: frenzyfriday|rover: when did a sync with dviroel|rover, we discussed some RDO components that were lagging ... pls will one of you look at master, train, wallby c8 and c9 components and see what's failing/debug/rerun? | 12:58 |
| rlandy | bhagyashris|ruck: also - maybe real issue on 16.2 for common - can you look? | 12:59 |
| dviroel|rover | rlandy: i am looking at rdo, frenzyfriday|rover is looking downstream | 12:59 |
| rlandy | great | 12:59 |
| rlandy | thank you | 12:59 |
| *** ysandeep is now known as ysandeep|out | 13:02 | |
| rlandy | chandankumar: can you revote: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44261/2/ci-scripts/infra-setup/tenant_vars/common.yml | 13:05 |
| bhagyashris|ruck | rlandy, will check i am rerunning failing component jobs at downstream | 13:05 |
| dviroel|rover | bhagyashris|ruck: https://review.rdoproject.org/zuul/buildset/72dc549e3ea347d680c105e6be50637f - common component in master and wallaby | 13:12 |
| dviroel|rover | bhagyashris|ruck: frenzyfriday|rover: rlandy: need to take the kid to the doc in 10 minutes, he is not felling well since yesterday | 13:19 |
| rlandy | dviroel|rover: np - frenzyfriday|rover is here now | 13:19 |
| rlandy | hope he feels better | 13:19 |
| rlandy | you can roll off rr | 13:19 |
| frenzyfriday|rover | dviroel|rover, yeah no problem | 13:19 |
| *** dviroel|rover is now known as dviroel | 13:20 | |
| *** dviroel is now known as dviroel|afk | 13:25 | |
| bhagyashris|ruck | dviroel|afk, ack | 13:26 |
| * bhagyashris|ruck brb | 13:26 | |
| chandankumar | rlandy: done, | 13:27 |
| rlandy | bhagyashris|ruck: if you get a chance, pls see comments on https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/402101 | 13:28 |
| rlandy | also needs rebase | 13:28 |
| arxcruz|rover | bhagyashris|ruck rlandy sorry, i went to take the 4th vaccine and took more time than expected | 13:41 |
| arxcruz|rover | but i'm back | 13:41 |
| rlandy | arxcruz|rover: np - how are you feeling?? | 13:42 |
| rlandy | arxcruz|rover: you can come off rover :) | 13:43 |
| rlandy | you're done, sir | 13:43 |
| arxcruz|rover | rlandy so far i'm good | 13:43 |
| arxcruz|rover | rlandy i was just rovering to support bhagyashris|ruck since she is alone | 13:43 |
| rlandy | arxcruz|rover: ok now frenzyfriday|rover is back | 13:43 |
| arxcruz|rover | ok | 13:43 |
| *** arxcruz|rover is now known as arxcruz | 13:43 | |
| rlandy | thanks for covering | 13:43 |
| rlandy | dviroel|afk rekciked wallaby fs039 | 13:44 |
| rlandy | will skip promo if no luck there | 13:44 |
| rlandy | and fs001 on c8 | 13:44 |
| rlandy | those are rekicked | 13:44 |
| chandankumar | marios: hello | 13:44 |
| marios | chandankumar: o/ | 13:44 |
| chandankumar | marios: https://review.opendev.org/c/openstack/tripleo-ansible/+/850562/2#message-6c74081af5176ffc8e26cc41bf300e05db57489d | 13:45 |
| chandankumar | marios: I think we need to make tripleo-ci-centos-8-9-multinode-mixed-os https://zuul.opendev.org/t/openstack/build/32a55e3ebeeb4b4da8b273a4aaa2c66a : FAILURE in 1h 38m 23s non voting in tripleo-ci | 13:45 |
| chandankumar | ? | 13:45 |
| marios | chandankumar: looking | 13:46 |
| chandankumar | stderr": "/bin/sh: os-net-config: command not found\n", "stderr_lines": ["/bin/sh: os-net-config: command not found"], "stdout": "", "stdout_lines": []} | 13:46 |
| chandankumar | not sure it is a real issue | 13:46 |
| marios | chandankumar: but it was green there in tripleo-ansible i mean same review see https://review.opendev.org/c/openstack/tripleo-ansible/+/850562/2#message-610956498d685b80cf7485ed5067dcaf6af98083 | 13:46 |
| marios | chandankumar: i know there is a problem with client (and it is non voting there) | 13:47 |
| marios | chandankumar: but it was green everywhere else at least so far | 13:47 |
| chandankumar | marios: sorry, didnot checked the lastr log | 13:47 |
| chandankumar | let me recheck it | 13:47 |
| marios | chandankumar: k .. that os-net-config thing is weird though | 13:48 |
| marios | chandankumar: well so far only once there though see https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-9-multinode-mixed-os&project=openstack%2Ftripleo-ansible&skip=0 | 13:49 |
| marios | chandankumar: i mean 'there' tripleo-ansible | 13:49 |
| marios | 1 fail today | 13:49 |
| marios | so if it is a bug its a new one | 13:49 |
| marios | chandankumar: ^^ | 13:49 |
| chandankumar | marios: thanks for looking into that, my bad, I will let you know the next run result | 13:49 |
| marios | chandankumar: ack thanks | 13:50 |
| marios | but the os-net-config this is worrying anyway might be a new bug chandankumar | 13:50 |
| marios | :/ | 13:50 |
| marios | lets see | 13:50 |
| frenzyfriday|rover | bhagyashris|ruck, do you know if we have a bug for Parse container params failure? I can see this in tripleo component dwnstream. | 13:51 |
| frenzyfriday|rover | containers-prepare-parameters.yaml seems to be missing stuff when i compare it to passing jobs | 13:51 |
| *** dasm|off is now known as dasm | 13:59 | |
| rlandy | are we not doing happy friday anymore? | 14:02 |
| rlandy | dasm: hello | 14:02 |
| *** amoralej|lunch is now known as amoralej | 14:03 | |
| rlandy | dasm: ysandeep|out left some comments on the OVB patch | 14:04 |
| rlandy | consensus was as we discussed on review time yesterday | 14:04 |
| rlandy | if wallaby works with master of OVB | 14:04 |
| rlandy | let's just do ahead | 14:04 |
| rlandy | and make it a c8, c9 split | 14:04 |
| rlandy | all c9 will go with master OVB | 14:04 |
| rlandy | branchful and branchless | 14:04 |
| dasm | rlandy: ysandeep|out sweet! i'm gonna go over his comments | 14:05 |
| rlandy | dasm: so you'll have to change the branchful ones to use master as well | 14:05 |
| rlandy | and test | 14:05 |
| rlandy | and then let us know if that works | 14:06 |
| dasm | ack | 14:06 |
| *** pojadhav is now known as pojadhav|out | 14:08 | |
| rlandy | frenzyfriday|rover: you;ll need to add the 17.1 scenario004 jobs to the skiplist | 14:08 |
| rlandy | https://opendev.org/openstack/openstack-tempest-skiplist/src/branch/master/roles/validate-tempest/vars/tempest_skip.yml#L1294 | 14:10 |
| rlandy | frenzyfriday|rover: ^^ currently failing | 14:10 |
| rlandy | and then rerun with that skip | 14:10 |
| frenzyfriday|rover | rlandy, ack, adding. I am creating a bug for multinode ipa 17.0 on 9 | 14:10 |
| frenzyfriday|rover | rlandy, bhagyashris|ruck could you pls check if https://bugzilla.redhat.com/show_bug.cgi?id=2112374 looks okay? I havent filed many downstream bugs | 14:16 |
| bhagyashris|ruck | frenzyfriday|rover, severity i think you can mentioned | 14:27 |
| frenzyfriday|rover | ack, updated | 14:29 |
| bhagyashris|ruck | frenzyfriday|rover, rlandy leaving for the day ... | 14:38 |
| frenzyfriday|rover | bhagyashris|ruck, ack - anything I should keep an eye on? | 14:38 |
| rlandy | bhagyashris|ruck: thanks | 14:39 |
| rlandy | pls leave notes what we should watch | 14:39 |
| bhagyashris|ruck | may be downstream specially 17 on rhel9 | 14:39 |
| frenzyfriday|rover | bhagyashris|ruck, ack, I'll check the downstream | 14:40 |
| rlandy | frenzyfriday|rover: checking bug | 14:40 |
| frenzyfriday|rover | rlandy, https://code.engineering.redhat.com/gerrit/c/testproject/+/422009 - rerun sc004 17.1 with skiplist | 14:40 |
| rlandy | frenzyfriday|rover: skiplist looks good - merging - ty | 14:40 |
| chandankumar | see ya! | 14:41 |
| rlandy | frenzyfriday|rover: wrt periodic-tripleo-ci-rhel-9-standalone-on-multinode-ipa-tripleo-rhos-17 | 14:42 |
| rlandy | frenzyfriday|rover: I think that should be fixed | 14:42 |
| rlandy | it was my mistake in the envs file | 14:43 |
| frenzyfriday|rover | rlandy, oh, cool. It should be green after your testproj finishes then. Closing the bug | 14:43 |
| rlandy | frenzyfriday|rover: you should be able to testproject recheck it now | 14:44 |
| rlandy | I merged the tripleo-environments change | 14:44 |
| rlandy | I am putting the bug on distribution component while you investigate | 14:44 |
| frenzyfriday|rover | rlandy, ack https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-8-standalone-on-multinode-ipa-tripleo-rhos-17&skip=0 is green again | 14:44 |
| rlandy | frenzyfriday|rover: you logged the bug on rhel-9 | 14:45 |
| rlandy | frenzyfriday|rover: also - I can't edit it | 14:46 |
| rlandy | can you move the component to distribution? | 14:46 |
| rlandy | frenzyfriday|rover: ok - nvm - you closed it | 14:47 |
| rlandy | all good | 14:47 |
| rlandy | frenzyfriday|rover: sorry - that was my fault - I messed up one of the settings values | 14:47 |
| frenzyfriday|rover | rlandy, yep, closed it np. the 9 version of it should also be green in the next run | 14:48 |
| rlandy | yep - hope so | 14:48 |
| * frenzyfriday|rover checking the rest of the components | 14:48 | |
| rlandy | frenzyfriday|rover: I think the real investigation is common on 16.2 | 14:48 |
| rlandy | those failures may be legit | 14:48 |
| rlandy | can you check there? | 14:48 |
| frenzyfriday|rover | aye , checking | 14:50 |
| * jm1 bbl | 14:50 | |
| frenzyfriday|rover | also periodic-tripleo-rhel-8-rhos-17-component-compute-promote-to-promoted-components and some other promote jobs are red | 14:51 |
| frenzyfriday|rover | oh ok, those are criteria failures, sorry | 14:52 |
| rlandy | frenzyfriday|rover: sooo ,,, bhagyashris|ruck and you and both rr this week | 14:56 |
| rlandy | when should I book the CRE team meeting | 14:56 |
| rlandy | maybe thursday when you roll off? | 14:56 |
| frenzyfriday|rover | rlandy, thursday or friday? | 14:57 |
| rlandy | arie doesn;t work on friday | 14:57 |
| frenzyfriday|rover | thursday it might be late for Bhagyashri | 14:57 |
| rlandy | let me see if anything is available on thursday | 14:57 |
| rlandy | hmm ... we have retro and planning | 14:58 |
| * rlandy will need to ask bhagyashri if we do retro wed or thursday | 14:58 | |
| rlandy | marios: ^^ do you remember what we chose? | 14:58 |
| marios | rlandy: yeah we said thursday | 14:58 |
| marios | rlandy: cancel scrum on thursday and use the slot for retro | 14:59 |
| marios | rlandy: then do planning immediately after it | 14:59 |
| rlandy | k - let me see if I can find a slot on wed | 14:59 |
| rlandy | frenzyfriday|rover: ^^ just for kickoff | 15:00 |
| frenzyfriday|rover | rlandy, cool | 15:00 |
| frenzyfriday|rover | rlandy, what is pipeline_component-common-pcci-16.2_dlrn-rhel-8.4-virthost-3cont_2comp_3ceph-ip job? I dont see it in https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/jobs | 15:11 |
| rlandy | frenzyfriday|rover: jenkins jobs | 15:12 |
| rlandy | kciked by QE | 15:12 |
| frenzyfriday|rover | oh | 15:12 |
| rlandy | they report to dlrn | 15:12 |
| rlandy | if you need them rerun, I can show you how | 15:12 |
| rlandy | we discuss those on #openstack-pcci | 15:12 |
| frenzyfriday|rover | it hasnt passed in a while, checking what happened | 15:13 |
| frenzyfriday|rover | what is the downstream jenkings url? | 15:13 |
| rlandy | chatting on internal | 15:14 |
| marios | rlandy: chandankumar: ysandeep|out: dviroel|afk: o/ please add to your reviews when you have time https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/851427 thanks | 15:26 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 15:26 |
| frenzyfriday|rover | sc001 and 2 16.2 common has a common standlone deploy failure: Cannot find config file: /etc/puppet/hiera.yaml | 15:36 |
| rlandy | frenzyfriday|rover: looks legit | 15:36 |
| rlandy | are you seeing the same error in train common component? | 15:37 |
| rlandy | frenzyfriday|rover: all the downstream lines have/will promote today | 15:52 |
| *** jpena is now known as jpena|off | 15:52 | |
| rlandy | waiting to see current run on fs001 in wallaby c8 | 15:53 |
| rlandy | then we can decide if worth it to promo | 15:53 |
| frenzyfriday|rover | rlandy, cool! I updated the rr hackmd with the failures I found for 16.2 common | 15:53 |
| rlandy | cool | 15:53 |
| frenzyfriday|rover | creating bugs in some time | 15:53 |
| rlandy | frenzyfriday|rover: if you don;t see those in train, it can downstream bug it | 15:53 |
| rlandy | the CIX | 15:54 |
| rlandy | then | 15:54 |
| rlandy | lunch - brb | 15:57 |
| rlandy | frenzyfriday|rover: ^^ | 15:57 |
| *** marios is now known as marios|out | 16:02 | |
| *** dviroel|afk is now known as dviroel | 16:14 | |
| *** amoralej is now known as amoralej|off | 16:21 | |
| rlandy | back | 16:26 |
| rlandy | dasm: need anything from me re: ovb? | 16:27 |
| rlandy | rcastillo: you all set with virt? | 16:27 |
| dasm | rlandy: i'm good. thanks | 16:28 |
| rcastillo | rlandy: testing here | 16:32 |
| rcastillo | I couldn't figure out how to get my keys in there yesterday. Is there a specific role I need? | 16:32 |
| rcastillo | forgot link https://code.engineering.redhat.com/gerrit/c/testproject/+/421866 | 16:32 |
| rlandy | rcastillo: sent you example on pvt | 16:41 |
| rlandy | if keys are not there | 16:41 |
| rcastillo | rlandy: thanks | 16:42 |
| frenzyfriday|rover | rlandy, I am leaving for the day. I'll check the components and create bugs for common if they still fail on monday | 17:24 |
| rlandy | frenzyfriday|rover: ok - thanks | 17:24 |
| rlandy | frenzyfriday|rover: may skip prmo wallaby c8 | 17:24 |
| rlandy | and c9 | 17:24 |
| rlandy | theyare both lagging | 17:24 |
| rlandy | frenzyfriday|rover: wallaby c9 is 4 days out | 17:25 |
| rlandy | we need to promote that | 17:25 |
| rlandy | frenzyfriday|rover: dviroel: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44307 Temp remove critera to promo wallaby c8 and c9 | 17:36 |
| rlandy | fs039 for c9 and fs001 for c8 | 17:37 |
| frenzyfriday|rover | aye | 17:37 |
| rlandy | pls vote | 17:37 |
| rlandy | will revert | 17:38 |
| frenzyfriday|rover | done | 17:38 |
| rlandy | thanks | 17:39 |
| dviroel | rlandy: rdo openstack-component-common seems to be a real issue | 18:24 |
| dviroel | rlandy: i can open a LP bug | 18:25 |
| dviroel | "stdout": "package openvswitch is not installed | 18:26 |
| dviroel | why only common? | 18:26 |
| dviroel | affects master and wallaby (8 and 9) | 18:27 |
| rlandy | dviroel: you checking components in RDO? | 18:27 |
| rlandy | ananya was looking at 16.2 common | 18:27 |
| rlandy | interesting | 18:28 |
| rlandy | looks like both | 18:28 |
| rlandy | if it's common only | 18:28 |
| rlandy | we should compare packages in component-ci-testing and tripleo-ci-testing | 18:28 |
| dviroel | rlandy: i saw this error earlier, and triggered common again, to test a new commit hash | 18:29 |
| dviroel | rlandy: the issue remains | 18:30 |
| rlandy | yep | 18:30 |
| rlandy | ok - let's look at the diff | 18:30 |
| rlandy | ┃ current-tripleo ┃ component-ci-testing ┃ | 18:32 |
| rlandy | ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ | 18:32 |
| rlandy | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ | 18:32 |
| rlandy | │ diskimage-builder-3.22.1-0.20220712164406.ba88a12.el9 │ diskimage-builder-3.22.1-0.20220725044337.50390d0.el9 │ | 18:32 |
| rlandy | │ openstack-selinux-0.8.34-0.20220711202850.a241718.el9 │ openstack-selinux-0.8.34-0.20220727173345.c3061f5.el9 │ | 18:32 |
| rlandy | │ openstack-tobiko-0.6.1-0.20220721215903.648f963.el9 │ openstack-tobiko-0.6.1-0.20220727131843.b303fa4.el9 | 18:32 |
| rlandy | dviroel: best guess is selinux | 18:32 |
| rlandy | Tengu mentioned we may be in for problems there | 18:32 |
| rlandy | checking if master has the same | 18:33 |
| dviroel | ok | 18:33 |
| rlandy | │ openstack-selinux-0.8.34-0.20220711202841.a241718.el9 │ openstack-selinux-0.8.34-0.20220727174553.c3061f5.el9 │ | 18:33 |
| rlandy | │ openstack-tobiko-0.6.1-0.20220721215850.648f963.el9 │ openstack-tobiko-0.6.1-0.20220727131836.b303fa4.el9 │ | 18:33 |
| rlandy | └───────────────────────────────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────── | 18:33 |
| rlandy | tobiko and selinux | 18:33 |
| rlandy | in both master and wallaby c9 | 18:33 |
| rlandy | │ openstack-selinux-0.8.34-2.20220712085402.a241718.el8osttrunk │ openstack-selinux-0.8.34-2.20220728085240.c3061f5.el8osttrunk | 18:34 |
| rlandy | same in 16.2 | 18:34 |
| Tengu | apparently it's an issue related to a not up-to-date system | 18:34 |
| Tengu | rlandy: -^ | 18:34 |
| rlandy | Tengu: Oh my gosh | 18:34 |
| rlandy | so late on a friday | 18:35 |
| Tengu | so if you update everything, it should be fine. | 18:35 |
| Tengu | rlandy: and on PTO | 18:35 |
| Tengu | :) | 18:35 |
| rlandy | go, go | 18:35 |
| rlandy | I was just talking about you - not to you :) | 18:35 |
| Tengu | :D | 18:35 |
| rlandy | dviroel: ^^ we probably need a bug with this info | 18:36 |
| Tengu | sorry, my 6th sense got that small tickling :) | 18:36 |
| rlandy | np | 18:36 |
| Tengu | rlandy: there's one already | 18:36 |
| rlandy | idk what no up to date system means exactly | 18:36 |
| rlandy | oh | 18:36 |
| rlandy | ok | 18:36 |
| rlandy | can you pass and we'll read the details | 18:36 |
| Tengu | rlandy: https://bugs.launchpad.net/tripleo/+bug/1982744/ | 18:36 |
| * rlandy remembers you warning us | 18:36 | |
| rlandy | Tengu++ thank you | 18:37 |
| Tengu | np | 18:37 |
| rlandy | now pls PTO | 18:37 |
| Tengu | back on Tuesday! | 18:37 |
| rlandy | see you then :) | 18:37 |
| Tengu | Monday is Swiss national day ;) | 18:37 |
| Tengu | see you around ;) | 18:37 |
| rlandy | dviroel: adding to rr hackmd | 18:37 |
| dviroel | Tengu: nice, thanks o/ | 18:37 |
| dviroel | rlandy: ack | 18:38 |
| * rlandy reads bug | 18:39 | |
| rlandy | ah comment 2 address the openvswitch thing | 18:39 |
| dviroel | lots of comments there | 18:43 |
| dviroel | and moved to won't fix | 18:44 |
| rlandy | dviroel: there is a workaround | 18:46 |
| rlandy | wdyt? | 18:46 |
| rlandy | try it? | 18:46 |
| rlandy | sudo dnf reinstall openstack-selinux container-selinux | 18:47 |
| rlandy | sudo rpm -V openstack-selinux | 18:47 |
| rlandy | Just ensure your system is fully up2date before you install container-selinux and then move to to use standalone. | 18:47 |
| dviroel | hum, maybe | 18:59 |
| dviroel | seems to be an selinux issue, so pinning might work too | 19:02 |
| jm1 | have a nice weekend #oooq :) | 19:17 |
| * rlandy thinks about this | 19:18 | |
| rlandy | reverted ci-config patch | 20:04 |
| dviroel | both promoted :) | 20:05 |
| rlandy | trying to hold node and try the update | 20:24 |
| * dasm => offline | 20:46 | |
| dasm | talk to you Monday! | 20:46 |
| *** dasm is now known as dasm|off | 20:46 | |
| dviroel | o/ | 20:46 |
| rlandy | selinux seems updated | 21:06 |
| rlandy | let's see | 21:06 |
| rlandy | Failed: | 21:23 |
| rlandy | openstack-selinux-0.8.34-0.20220727174553.c3061f5.el9.noarch | 21:23 |
| rlandy | ugh - I give up | 21:23 |
| rlandy | dviroel: ^^ will pick this up another time | 21:23 |
| rlandy | logging out | 21:23 |
| rlandy | have a great weekend | 21:23 |
| rlandy | and thanks for covering today | 21:23 |
| dviroel | rlandy: have a great weekend o/ | 21:25 |
| * dviroel too late | 21:25 | |
| rcastillo | have a great weekend dviroel | 21:29 |
| * rcastillo out | 21:29 | |
| dviroel | rcastillo: you too | 21:30 |
| *** dviroel is now known as dviroel|out | 21:33 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!