| *** rlandy|bbl is now known as rlandy | 01:02 | |
| *** rlandy is now known as rlandy|out | 01:07 | |
| *** ysandeep|PTO is now known as ysandeep | 03:50 | |
| *** chandankumar is now known as raukadah | 04:03 | |
| akahat | ysandeep, o/ | 06:29 |
|---|---|---|
| akahat | ysandeep, there are jobs stuck in the zuul queue.. from last 6 hours. | 06:30 |
| ysandeep | hello akahat | 06:30 |
| * ysandeep looking | 06:30 | |
| akahat | https://review.rdoproject.org/zuul/status | 06:30 |
| *** amoralej|off is now known as amoralej | 06:31 | |
| ysandeep | akahat, check/gate queue? | 06:32 |
| ysandeep | I am looking at top gate queue - looks like its progressing: https://zuul.openstack.org/stream/2f49b3996a3e40c1a79920916bb1b1a3?logfile=console.log | 06:32 |
| ysandeep | top job* in gate queue | 06:32 |
| akahat | ysandeep, yes check gate and other queues also. | 06:32 |
| akahat | ysandeep, rdo one. | 06:33 |
| ysandeep | akahat, want to meet on gmeet to cross check we are looking at same thing? | 06:34 |
| akahat | ysandeep, yeah.. sure. | 06:34 |
| akahat | one min | 06:34 |
| akahat | ysandeep, https://meet.google.com/gdk-dcuk-usv | 06:35 |
| *** akahat is now known as akahat|ruck | 06:47 | |
| *** ysandeep is now known as ysandeep|afk | 06:49 | |
| *** jm1|ruck is now known as jm1|rover | 07:00 | |
| jm1 | hello everyone :) | 07:02 |
| *** jpena|off is now known as jpena | 07:10 | |
| akahat|ruck | jm1, o/ | 07:17 |
| jm1 | akahat|ruck: o/ we have yet another promotion blocker? 🙄 | 07:21 |
| akahat|ruck | jm1, yes.. it's in component jobs. | 07:21 |
| akahat|ruck | We are not able to provision overcloud network. | 07:21 |
| jm1 | akahat|ruck: that must be something intermittent if it fails once and passes once with the same hash? | 07:28 |
| akahat|ruck | jm1, yeah.. we got one job passed. and many filed.. didn't checked hash. | 07:29 |
| jm1 | akahat|ruck: updated rr notes about hash which failed and passed | 07:31 |
| jm1 | akahat|ruck: moved that bug to "known bugs | 07:40 |
| jm1 | " in rr hackmd | 07:40 |
| akahat|ruck | jm1, thanks! | 07:41 |
| jm1 | marios: o/ i have a testproject which depends-on a change in rdoproject config repo. can that possibly work? or how can we test config project changes? https://review.rdoproject.org/r/c/testproject/+/43806 | 08:38 |
| marios | o/ jm1 looking (cant test config project though sec lemme check) | 08:40 |
| marios | jm1: yah cant test that https://review.rdoproject.org/r/c/config/+/43805 speculatively we have to merge to test it | 08:42 |
| marios | jm1: so you can try to confirm that url is OK (i.e. you can authenticate with that fine) then merge/test and revert if needed | 08:42 |
| jm1 | marios: ah, ok. thanks for clarification! then i will wait for rlandy to get permission before playing with config repo :) | 08:44 |
| marios | jm1: k :) - do you have something more to point to about the deprecated url *(left a comment) | 08:44 |
| jm1 | marios: there was a discussion on #vexxhost yesterday but afaik that channel has no logs, so nothing to point to | 08:45 |
| marios | jm1: k, maybe file a bug on our side then - i assume it broke the job? | 08:51 |
| marios | jm1: at least to have some record of what/why we had to change that URL | 08:51 |
| jm1 | marios: will check with rlandy later. atm i am digging through all the other open bug reports | 08:53 |
| marios | jm1: what will you check with rlandy if we need a bug? | 08:54 |
| marios | jm1: k | 08:54 |
| jm1 | marios: mnaser mentioned that he notified teams a while ago about switching to new endpoint, so maybe there is a mail in the openstack/rdo archives somewhere which we could quote/reference | 08:55 |
| marios | jm1: ok but imo we need a bug on our side to say 'look this broke us' | 08:56 |
| marios | jm1: and in that bug you can add any other context you might find | 08:56 |
| jm1 | marios: ack, so no way around that bug?! ^^ | 08:56 |
| marios | jm1: if i were ruck|rover i'd file one for sure but i am not so ultimately it is up to you and akahat|ruck | 08:57 |
| marios | jm1: few more hours to go hang in there ;) | 08:57 |
| jm1 | marios: will create a new bug just for you XD | 09:00 |
| marios | i will frame it and put in on the wall ;) | 09:00 |
| jm1 | marios: yes please https://bugs.launchpad.net/tripleo/+bug/1980343 😂 | 09:08 |
| marios | jm1: thanks :) | 09:09 |
| marios | fires up printer | 09:09 |
| jm1 | internal irc down? | 09:56 |
| marios | yeah looks like it jm1 12:53 -!- Irssi: Connection lost | 09:56 |
| marios | like ~3 mins ago | 09:56 |
| jm1 | it took akahat|ruck down as well 😠| 09:57 |
| *** akahat is now known as akahat|ruck | 10:08 | |
| akahat|ruck | jm1, hey.. are you able to connect internal irc? | 10:12 |
| jm1 | akahat|ruck: nope, still down | 10:12 |
| jm1 | akahat|ruck: but glad we got you back :) | 10:12 |
| akahat|ruck | oh.. Okay. | 10:12 |
| akahat|ruck | jm1, i'm not going anywhere. :P | 10:13 |
| * jm1 lunch, bbl | 10:20 | |
| *** bhagyashris_ is now known as bhagyashris | 10:25 | |
| marios | do we have missing containers jm1 akahat|ruck fatal: [undercloud]: FAILED! => {"changed": false, "msg": "Failed to pull following containers: ['quay.rdoproject.org/tripleowallabycentos9/daemon:current-ceph']"} | 10:28 |
| marios | perhaps related to the login issue? https://bugs.launchpad.net/tripleo/+bug/1980343 | 10:28 |
| marios | mm not sure if related | 10:29 |
| akahat|ruck | marios, container is exists in the repo. I also think it is not related. | 10:37 |
| marios | akahat|ruck: yeah its something strange... this is *8* content provider and it failed pulling a *9* container | 10:39 |
| marios | akahat|ruck: so might be some new issue, trying to understand if related to https://review.opendev.org/c/openstack/tripleo-ci/+/843787 not sure yet | 10:39 |
| akahat|ruck | that's interesting.. | 10:39 |
| *** rlandy|out is now known as rlandy | 10:39 | |
| rlandy | helo | 10:40 |
| rlandy | hello | 10:40 |
| rlandy | jm1: akahat|ruck: want to sync? | 10:40 |
| rlandy | need to change config | 10:40 |
| akahat|ruck | marios, could you please share logs where you seen this issue? | 10:41 |
| akahat|ruck | rlandy, yes | 10:41 |
| rlandy | akahat|ruck: jm1: https://meet.google.com/nvk-exzk-wic?pli=1&authuser=0 | 10:42 |
| marios | akahat|ruck: there https://review.rdoproject.org/zuul/build/153ebca694324eaf81bd141aeda56ab7 but... | 10:44 |
| marios | akahat|ruck: not sure if this is transient or what is going on yet ... i don't see it anywhere else like * https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-content-provider | 10:44 |
| marios | akahat|ruck: maybe just keep in mind for now in case it hits elsewhere but i dind't make any changes to content provider in my tests (so weird if this is just transient) lets see | 10:45 |
| *** ysandeep|afk is now known as ysandeep | 10:50 | |
| *** dviroel_ is now known as dviroel | 11:20 | |
| * marios going pickup kid from summer school back in a bit | 11:24 | |
| *** marios is now known as marios|afk | 11:24 | |
| rlandy | pojadhav: ysandeep: forwarded you both email about ntp servers | 11:41 |
| rlandy | we need to mass change our entries | 11:41 |
| rlandy | we have dozens of them | 11:42 |
| ysandeep | rlandy, reading | 11:42 |
| rlandy | going to start putting in some changes now | 11:42 |
| rlandy | ysandeep: ^^ | 11:42 |
| rlandy | pojadhav: ysandeep: I grepped tripleo-envs tripleo-ci-internal-jobs and config | 11:42 |
| rlandy | pojadhav: ysandeep: I'm starting with patches for internal jobs and config | 11:43 |
| ysandeep | rlandy++ I can patch tripleo-environment | 11:43 |
| Tengu | anything I can help? /me reads "mass change". | 11:44 |
| rlandy | Tengu: internal ntp server | 11:46 |
| ysandeep | Tengu: thanks, all okay - just change in NTP in internal | 11:46 |
| Tengu | 'k. if help is needed, lemme know. | 11:46 |
| Tengu | btw, how's the gate for master and wallaby? We found an issue earlier today, a patch is already pushed (https://bugs.launchpad.net/tripleo/+bug/1980320 ) | 11:46 |
| Tengu | issue created by a lack of test in tripleo-ansible - correction for this: https://review.opendev.org/c/openstack/tripleo-ci/+/848244 | 11:47 |
| rlandy | ysandeep: let's use a single topic | 11:50 |
| rlandy | corp-ntp? | 11:50 |
| ysandeep | ack | 11:50 |
| jm1 | rlandy, akahat|ruck: why is https://bugs.launchpad.net/tripleo/+bug/1980185 already marked done on cix card? its wallaby patch has not been merged yet | 11:51 |
| * ysandeep just ran sed magic | 11:52 | |
| *** marios|afk is now known as marios | 12:00 | |
| ysandeep | rlandy pojadhav https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/417984 | 12:02 |
| rlandy | ysandeep: pojadhav: the whole list: https://code.engineering.redhat.com/gerrit/q/topic:corp-ntp | 12:06 |
| rlandy | ysandeep: 417986 (ie: the tripleo-ci-internal-jobs) one will actually run a test | 12:07 |
| rlandy | if that passes, let's merge the whole series | 12:07 |
| ysandeep | rlandy, your patches looks good, we covered all the references. I have left my votes - ack to +w the series once zuul reports | 12:09 |
| ysandeep | sed++ thanks for making our life easy :) | 12:10 |
| ysandeep | rlandy, Is it okay with you if I add tripleo-environment patch as depends-on for your -jobs patch? We can test release files change in same patch. | 12:13 |
| rlandy | ysandeep: sure | 12:13 |
| ysandeep | ++ thanks, updated | 12:14 |
| *** amoralej is now known as amoralej|lunch | 12:26 | |
| marios | akahat|ruck: just to close - the content provider 8 thing was transient at least I didn't hit it on re-runs sorry for noise | 12:51 |
| akahat|ruck | marios, oh.. that's great!! | 12:52 |
| akahat|ruck | thanks for info! | 12:52 |
| jm1 | rlandy, akahat|ruck: since i did not see a response, let me ask again 🙊 why is https://bugs.launchpad.net/tripleo/+bug/1980185 already marked done on cix card? its wallaby patch has not been merged yet | 12:52 |
| akahat|ruck | jm1, oh.. it's okay.. it will merge. | 12:52 |
| akahat|ruck | We have +w on that. | 12:52 |
| marios | dviroel: o/ hey is this a bug? content provider 8 pulls in tripleowallabycentos9/daemon:current-ceph (i.e. c9/wallaby?) | 12:53 |
| jm1 | akahat|ruck: ack. we left the other cards open till the patches got merged. so i was just wondering why its already marked as Done. But okay | 12:53 |
| marios | dviroel: eg there last one from the gate https://c9a83ffc038379da48fb-c97a7f6b7423ddbb4161c783b6bd4b21.ssl.cf2.rackcdn.com/848111/2/gate/tripleo-ci-centos-8-content-provider/28ded72/job-output.txt | 12:53 |
| rlandy | jm1: looking | 12:54 |
| marios | dviroel: the c8 content provider job passes but i noticed this as it failed in my job earlier (transient) | 12:54 |
| rlandy | jm1: you are right - the wallaby patch has not merged | 12:54 |
| dviroel | marios: it is expected, we didn't promote for c8, but the ceph daemon image is the same | 12:55 |
| rlandy | probably the bug was marked fix-released when the master patch merged | 12:55 |
| marios | dviroel: ah great was hoping for something like that thanks for sanity check then | 12:55 |
| dviroel | sure, np | 12:56 |
| bhagyashris | arxcruz, rcastillo dviroel ysandeep | 13:01 |
| bhagyashris | scrum time | 13:01 |
| bhagyashris | soniya29, | 13:01 |
| dviroel | joinning | 13:02 |
| marios | nice shirt dasm|off (doom?) also renick :D | 13:03 |
| marios | and good morning | 13:03 |
| *** dasm|off is now known as dasm | 13:03 | |
| dasm | o/ | 13:03 |
| dasm | marios: it says "Not in the MooD" ;) | 13:03 |
| jm1 | dasm: oh wow, i read Doom as well 😂 | 13:04 |
| soniya29 | bhagyashris, i will join by 5-10 min | 13:04 |
| dasm | but it uses "DooM'. Now it should be visible ^^ | 13:04 |
| jm1 | dasm: yeah, nice ^^ | 13:04 |
| arxcruz | indeed nice shirt dasm :D | 13:04 |
| dasm | jm1: I know. It looks very similar :) | 13:04 |
| marios | dasm: yeah i saw it was mood but i assumed it was in the style of doom | 13:05 |
| dasm | thank you :) | 13:05 |
| marios | i thought it was intentional | 13:05 |
| dasm | marios: :) | 13:05 |
| dasm | now my head looks like i'm bald xD | 13:05 |
| dasm | https://review.rdoproject.org/r/q/topic:skiplist+owner:dasm | 13:16 |
| dasm | skiplist patch ^ | 13:16 |
| bhagyashris | https://review.rdoproject.org/r/q/topic:skiplist+owner:dasm | 13:16 |
| *** amoralej|lunch is now known as amoralej | 13:24 | |
| jm1 | frenzy_friday: reviewbot down? | 13:29 |
| frenzy_friday | checkimg | 13:29 |
| frenzy_friday | jm1, bot is back | 13:32 |
| jm1 | frenzy_friday: oh great, thanks :) | 13:33 |
| * frenzy_friday checking something with the bot | 13:34 | |
| jm1 | rlandy: which friday afternoon will you be out? this or next week? | 13:37 |
| dasm | jm1: iiuc next. | 13:39 |
| * frenzy_friday has stoped playing with the bot now | 13:46 | |
| Tengu | /13 | 13:49 |
| ysandeep | rcastillo, hey o/ just want to confirm - Are you out on Monday? | 13:57 |
| rcastillo | ysandeep: nope | 13:58 |
| ysandeep | rcastillo: ack, good for me :) | 14:01 |
| ysandeep | rlandy, ntp patch - zuul gave +1 but.. we are still using old ntp server | 14:19 |
| ysandeep | https://sf.hosted.upshift.rdu2.redhat.com/logs/86/417986/3/check/tripleo-ci-rhel-8-standalone-rhos-16.2/2ead1b7/logs/undercloud/var/log/extra/chrony.txt | 14:19 |
| ysandeep | rlandy, i think that because we have ntp in config repo: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/417983/1/playbooks/set-internal-vars-standalone.yaml | 14:19 |
| ysandeep | rlandy, we need to merge the config patch first to test | 14:19 |
| rlandy | ysandeep: ack - that is in config | 14:20 |
| rlandy | go ahead and merge that one | 14:20 |
| rlandy | and we can see | 14:20 |
| ysandeep | ack | 14:20 |
| rlandy | ysandeep: jm1: akahat|ruck: rcastillo: rr sync in 4 mins? | 14:26 |
| ysandeep | sure , works for me | 14:27 |
| rlandy | https://meet.google.com/syn-twgn-ppb?pli=1&authuser=0 | 14:28 |
| rlandy | let's see if we get more responses | 14:28 |
| jm1 | rlandy: ack, gimme a sec | 14:28 |
| ysandeep | rcastillo, akahat|ruck ^^ please join if around | 14:29 |
| rlandy | ysandeep: periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-master | 14:43 |
| ysandeep | rcastillo, jm1 created new rr hackmd for us: https://hackmd.io/pRC9TDaoQLWqRSQioTUl5g | 14:47 |
| rlandy | rcastillo: https://review.opendev.org/c/openstack/tripleo-quickstart/+/846194/3/config/release/tripleo-ci/CentOS-9/master.yml | 14:48 |
| jm1 | rlandy, akahat|ruck, ysandeep: eod now | 14:50 |
| rlandy | jm1: thanks - have a good night | 14:52 |
| ysandeep | jm1, akahat|ruck thanks for keeping things in shape and for very good handover :) | 14:53 |
| ysandeep | rcastillo, rlandy: I am eod too, drop me notes on https://hackmd.io/pRC9TDaoQLWqRSQioTUl5g if you want me to pick anything. | 14:54 |
| *** jgilaber_ is now known as jgilaber|away | 14:54 | |
| rlandy | ysandeep: ack - see you tomorrow | 14:55 |
| ysandeep | bhagyashris, hey o/ https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status#417047 could you please check why 16.2 jobs are triggering on ceph 5 related changes. | 14:57 |
| ysandeep | bhagyashris, we use ceph 4 in 16.2 jobs | 14:57 |
| * dviroel lunch + manila community meeting | 14:58 | |
| ysandeep | bhagyashris, We need to optimize the resources we trigger against rhos-release now - looks like ton of jobs triggered on this patch. | 14:58 |
| *** ysandeep is now known as ysandeep|out | 14:59 | |
| rlandy | ysandeep|out: periodic-tripleo-ci-rhel-8-bm_envD-3ctlr_1comp-featureset035-rhos-16.2 - need some work again - maybe another reboot | 15:06 |
| rlandy | rerunning the other two failures | 15:07 |
| ysandeep|out | rlandy, yeah same issue - reboot will solve - I can try to rca tomorrow(last time we didn't find the cause) - If its urgent I can reboot the node | 15:37 |
| rlandy | ysandeep|out: not urgent | 15:37 |
| rlandy | I am trying to fix other failures | 15:38 |
| rlandy | so you can deal with that one in your morning | 15:38 |
| ysandeep|out | ack | 15:38 |
| rcastillo | rlandy: libvirt exclude revert. Waiting on results https://review.opendev.org/c/openstack/tripleo-quickstart/+/848273 | 15:46 |
| rlandy | rcastillo: cool - let me know | 15:47 |
| *** marios is now known as marios|out | 15:50 | |
| *** jpena is now known as jpena|off | 16:28 | |
| *** amoralej is now known as amoralej|off | 16:58 | |
| rlandy | lunch - brb | 17:17 |
| rlandy | rcastillo: failures in master line need tkajinam's fix | 18:06 |
| rlandy | will rerun after that merges | 18:06 |
| rcastillo | rlandy: do we need to kick tripleo component when that merges or just directly integration lines? | 18:25 |
| rlandy | integration | 18:38 |
| rlandy | tripleo-ansible | 18:38 |
| rlandy | the tripleo master line is very behind | 18:38 |
| rlandy | I'm trying to rerun the failed job there now | 18:38 |
| rlandy | fix for scen001 is second from top in gate | 18:39 |
| rcastillo | I saw | 18:46 |
| rcastillo | small tool I wrote to run scripts when zuul jobs change status https://github.com/eltrufas/zuul-alarm | 18:52 |
| rcastillo | using it to get desktop and phone notifications when jobs finish | 18:53 |
| rcastillo | rlandy: latest libvirt works | 18:55 |
| rlandy | ha - fun | 18:57 |
| rlandy | ok - you can put the revert on review list - we'll vote in a few | 18:57 |
| rcastillo | sure | 18:58 |
| rlandy | dviroel: dasm: review time? | 19:01 |
| dviroel | oh, joinning | 19:01 |
| *** beagles_ is now known as beagles | 20:12 | |
| rlandy | rcastillo: rhos-17 on rhel-8 just promoted | 20:19 |
| rlandy | rcastillo: fs001 in network component master - passed in testproject but the hash changed | 20:21 |
| rlandy | the whole component is running now | 20:21 |
| rlandy | may need another rerun | 20:21 |
| rcastillo | rlandy: ahh, was gonna ask if I needed to rekick, I'll watch it | 20:21 |
| rlandy | rcastillo: wallby c8 should promote now ... https://trunk.rdoproject.org/api-centos8-wallaby/api/civotes_agg_detail.html?ref_hash=783b77d47c10a8b29f79cd10a56b3a66 | 20:29 |
| rcastillo | rlandy: nice. rerrunning c9 wallaby fs035 | 20:34 |
| rlandy | rcastillo: chatting on #rhos-ops | 20:39 |
| rlandy | asking to get the last of the hanging check jobs cleared | 20:39 |
| * dasm signs off | 20:41 | |
| dasm | o/ | 20:41 |
| *** dasm is now known as dasm|off | 20:42 | |
| * dviroel afk | 20:42 | |
| *** dviroel is now known as dviroel|afk | 20:42 | |
| rcastillo | rlandy: takashi's patch merged, kicking master | 22:19 |
| rlandy | rcastillo: nice | 22:28 |
| rlandy | rcastillo: ugh - we're very backed up on rdo zuul | 22:28 |
| rlandy | hopefully it will start to clear out | 22:28 |
| rlandy | stepping out - will check back later | 22:29 |
| *** rlandy is now known as rlandy|bbl | 22:29 | |
| rcastillo | ok, ttyl | 22:29 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!