| *** pojadhav- is now known as pojadhav | 04:55 | |
| *** marios is now known as marios|ruck | 04:57 | |
| marios|ruck | \o/ | 04:57 |
|---|---|---|
| marios|ruck | soniya29: o/ | 04:57 |
| marios|ruck | soniya29: morning anything to report today? lets sync in a bit? | 05:01 |
| marios|ruck | can you also renick? soniya29 | 05:01 |
| marios|ruck | thank you raukadah and ysandeep|out for fixing the blocker on friday o/ | 05:05 |
| *** soniya29 is now known as soniya29|ruck | 05:35 | |
| soniya29|ruck | marios|ruck, done | 05:35 |
| soniya29|ruck | marios|ruck, hello | 05:36 |
| marios|ruck | soniya29|ruck: hi - just catching up a bit lets do a call? can we meet in like 1 hour is that OK for you? | 05:36 |
| soniya29|ruck | marios|ruck, yeah..thats okay | 05:37 |
| *** soniya29|ruck is now known as soniya29|rover | 05:37 | |
| marios|ruck | soniya29|ruck: k sent invitation | 05:38 |
| soniya29|rover | marios|ruck, ack | 05:39 |
| marios|ruck | soniya29|rover: o/ do you have a link to the failing rhos17 multinode please | 06:18 |
| soniya29|rover | marios|ruck, https://sf.hosted.upshift.rdu2.redhat.com/logs/43/211643/129/check/periodic-tripleo-ci-rhel-9-bm_envB-3ctlr_1comp-featureset001-rhos-17/1094b43/job-output.txt | 06:18 |
| marios|ruck | soniya29|rover: yeah found it thanks added it to the hackmd | 06:20 |
| *** ysandeep|out is now known as ysandeep | 06:23 | |
| ysandeep | soniya29|rover, hey o/ I am still working on 17 baremetal job - periodic-tripleo-ci-rhel-9-bm_envB-3ctlr_1comp-featureset001-rhos-17 , Its not yet in criteria. | 06:26 |
| ysandeep | Infact it is not in pipeline too | 06:26 |
| ysandeep | marios|ruck: my pleasure o/ | 06:26 |
| marios|ruck | soniya29|rover: o/ lets quick sync? | 06:31 |
| soniya29|rover | sure | 06:31 |
| soniya29|rover | just a min | 06:31 |
| soniya29|rover | ysandeep, okay, it was failing hence i made a record of it | 06:33 |
| pojadhav | marios|ruck, 0/ hello do you have few mins ?? to understand ussuri tear down stuff requested by ronelle. | 06:36 |
| marios|ruck | pojadhav: sure lets talk on phone | 06:43 |
| marios|ruck | pojadhav: https://meet.google.com/udy-msip-eoj?hs=49 | 06:43 |
| pojadhav | marios|ruck, ack | 06:43 |
| marios|ruck | https://review.opendev.org/c/openstack/releases/+/834049 pojadhav | 06:45 |
| jpodivin | marios|ruck: I see the https://bugs.launchpad.net/tripleo/+bug/1966382 is nearing resolution. | 06:58 |
| marios|ruck | o/ jpodivin | 07:06 |
| -opendevstatus- NOTICE: zuul isn't executing check jobs at the moment, investigation is ongoing, please be patient | 07:17 | |
| ysandeep | soniya29|rover, marios|ruck looks like downstream jobs are stuck for more than 33 hours. | 07:17 |
| * ysandeep pining infra in internal | 07:17 | |
| ysandeep | pinging* | 07:17 |
| marios|ruck | soniya29|rover: can you please follow up with ysandeep ^^^ | 07:18 |
| marios|ruck | thanks ysandeep | 07:18 |
| soniya29|rover | marios|ruck, sure | 07:19 |
| marios|ruck | soniya29|rover: fyi i added collapsible to the hackmd - can you do the same for the rest of them (you can click to expand the details) | 07:20 |
| soniya29|rover | marios|ruck, ack | 07:20 |
| *** jpena|off is now known as jpena | 07:39 | |
| * soniya29|rover lunch | 07:40 | |
| *** soniya29|rover is now known as soniya29|rover|lunch | 07:41 | |
| jm1 | o/ | 07:46 |
| jm1 | whats up with zuul? not one job is running https://zuul.opendev.org/t/openstack/status ?!? | 07:46 |
| *** ysandeep is now known as ysandeep|lunch | 07:49 | |
| marios|ruck | soniya29|rover|lunch: i am updating the hackmd for the collapsible | 07:49 |
| marios|ruck | jm1: 10:17 -opendevstatus:#opendev- NOTICE: zuul isn’t executing check jobs at the moment, investigation is ongoing, please be patient | 07:57 |
| jm1 | marios|ruck: oh, did not know that channel. thanks marios :) | 07:58 |
| soniya29|rover|lunch | marios|ruck, ack | 08:02 |
| marios|ruck | jm1: ack thanks ysandeep|lunch who noticed it first ;) | 08:04 |
| jm1 | ysandeep|lunch++ ;) | 08:06 |
| *** ysandeep|lunch is now known as ysandeep | 08:45 | |
| jpodivin | marios|ruck: regarding the workaround for #1966382 I'm wondering which projects should need it. Last week when we are triaging the issue, it appeared that only repos without "traditional" python module structure should need it. Is that still the case? | 09:02 |
| marios|ruck | jpodivin: are you seeing it somewhere else? brian is working on a couple see #tripleo | 09:03 |
| marios|ruck | jpodivin: otherwise i think we are ok | 09:03 |
| marios|ruck | jpodivin: if you know of more please go ahead | 09:04 |
| marios|ruck | sorry jpodivin not brian brenadn | 09:04 |
| marios|ruck | :) brendan | 09:04 |
| jpodivin | marios|ruck: I know who you meant :) | 09:04 |
| ysandeep | jpodivin: easy to answer which repo need same fix with quick testing, pip install setuptools=61.0 , clone repo, cd into repo, pip install . | 09:05 |
| jpodivin | marios|ruck: but regarding your other question. I think that on the whole there shouldn't be that many projects with this issue. | 09:05 |
| jpodivin | ysandeep: exactly | 09:05 |
| marios|ruck | but 12:03 < marios|ruck> jpodivin: are you seeing it somewhere else? brian is working on a | 09:05 |
| marios|ruck | jpodivin: hmm actually the gate was stuck this morning so perhaps we will still see more tripleo repos | 09:06 |
| marios|ruck | jpodivin: k if you post anything can you add me to reviews please | 09:06 |
| jpodivin | marios|ruck: ack. But imho the gate might be unrelated to the original issue. | 09:07 |
| marios|ruck | jpodivin: how you mean? | 09:07 |
| marios|ruck | jpodivin: ah you mean "just cos the gate isn't broken? " - sure but i am ruck and my job is to watch the queues this week so i have to focus on this | 09:07 |
| jpodivin | marios|ruck: understood, all I'm saying is that two failures might not have the same cause. | 09:08 |
| jpodivin | the original setuptools problem was made more complex by the freeze graph issue. | 09:08 |
| jpodivin | otherwise we would have more patches merged on Friday. | 09:09 |
| marios|ruck | jpodivin: yeah that freeze graph issue may be related to recent reparenting efforts but as noted should be fixable by adding the setup.py to irrelevant files | 09:09 |
| jpodivin | marios|ruck: I'm ... well I know it enables the CI to run and that we need it ... But I'm wondering if we shouldn't flip it around. Rather than to stop one job from triggering make the other one run as well. | 09:11 |
| jpodivin | It's too late now, and I don't want to spam the pipelines with DNMs though | 09:11 |
| jpodivin | In any case it's a minor thing. Arguably the setup.* files are barely touched so we should be fine. | 09:12 |
| marios|ruck | jpodivin: not sure we need that minor update job for changes to setup.py though i get your point but in this case i think irrelevant files fits | 09:13 |
| jpodivin | As I said, we should be fine. There are other jobs, most of them fairly low level, that should catch setup file issues.. | 09:15 |
| *** soniya29|rover|lunch is now known as soniya29|rover | 09:18 | |
| jpodivin | anyway, I'm going to search the projects for more potential problems, see if anything comes up we need to patch. | 09:33 |
| ysandeep | jpodivin, me patching tht | 09:34 |
| marios|ruck | thanks jpodivin ysandeep | 09:34 |
| marios|ruck | soniya29|rover: fyi new master/9 tempest issue filed (on the hackmd) | 09:35 |
| soniya29|rover | marios|ruck, ack | 09:35 |
| jpodivin | ysandeep: btw thanks for the test suggestion, I'm automating that. | 09:39 |
| ysandeep | jpodivin, please see https://review.opendev.org/q/topic:lp%252F1966382 to track projects that already have fix | 09:40 |
| * ysandeep checking tripleoclient now | 09:40 | |
| ysandeep | tripleoclient is okay without fix | 09:42 |
| * ysandeep checking tripleo-common now | 09:42 | |
| ysandeep | tripleo-common is okay as well without fix | 09:43 |
| ysandeep | tripleo-image-elements need fix, me proposing | 09:46 |
| ysandeep | folks o/ anyone knows if we use https://github.com/openstack/tripleo-ha-utils anymore? | 09:58 |
| jpodivin | ysandeep: ack, I'm going alphabetically. | 09:59 |
| ysandeep | jpodvin ack i have checked everything tripleo-* and pythontripleoclient | 10:03 |
| rlandy | marios|ruck: soniya29|rover: hello ... need help with anything? | 10:23 |
| soniya29|rover | rlandy, hello | 10:23 |
| ysandeep | we need to request ananya to put reviewbot on some server | 10:24 |
| soniya29|rover | rlandy, not as such | 10:24 |
| rlandy | marios|ruck: soniya29|rover: let me know if you want to sync | 10:24 |
| ysandeep | folks o/ please visit https://review.opendev.org/q/topic:lp/1966382+status:open when time allows. | 10:26 |
| jpodivin | ysandeep: so far I don't have much in a way of findings. Except for devstack plugins, I'll have to verify them though. | 10:28 |
| rlandy | jpodivin: can you adjust the commit message length here: https://review.opendev.org/c/openstack/ansible-role-atos-hsm/+/835432 | 10:29 |
| jpodivin | omg | 10:29 |
| jpodivin | sorry | 10:29 |
| ysandeep | jpodivin, no worries, thanks for checking, atleast https://review.opendev.org/q/topic:lp/1966382+status:open will take care of projects we are more interested in. | 10:29 |
| marios|ruck | o/ rlandy | 10:30 |
| marios|ruck | rlandy: sure we can catchup on reviews call ... you can check https://hackmd.io/mXb6zsGsTNWWO6Y6UpUNAg?view for status | 10:30 |
| rlandy | ok | 10:30 |
| marios|ruck | rlandy: couple new bugs there | 10:30 |
| rlandy | marios|ruck: ok - so you can just let me know if there is something you want help with | 10:30 |
| marios|ruck | rlandy: will do | 10:31 |
| rlandy | marios|ruck: left a patch to get fs035 out of ussuri and promote | 10:32 |
| rlandy | soniya29|rover: you don;t need to rerun the rbac test - it's not in criteria | 10:33 |
| soniya29|rover | rlandy, ack | 10:34 |
| rlandy | save you some resources | 10:34 |
| jpodivin | I might have to slow down with the testing, laptop is getting hot :D | 10:34 |
| jpodivin | I might have to slow down with the testing, laptop is getting hot :D | 10:34 |
| jpodivin | Well devstack plugins are fine after all. They can't be installed in the first place. Devstack itself on the other hand is not. | 10:37 |
| marios|ruck | rlandy: yes i merged it this morning | 10:39 |
| marios|ruck | rlandy: thanks | 10:39 |
| rlandy | ok - checking if it promoted | 10:40 |
| rlandy | current-tripleo/2022-03-21 13:26 | 10:40 |
| rlandy | um no | 10:40 |
| rlandy | hash may be too old | 10:41 |
| marios|ruck | rlandy: k i will check again later but.. ussuri ;) | 10:42 |
| marios|ruck | rlandy: next week we can nuke | 10:42 |
| rlandy | marios|ruck: no worries | 10:42 |
| rlandy | was just trying to get it out of your hair | 10:43 |
| * rlandy checking components | 10:43 | |
| marios|ruck | rlandy: thx | 10:43 |
| rlandy | soniya29|rover: ^^ can you check components downstream? I'm checking upstream | 10:50 |
| soniya29|rover | rlandy, i am upto it | 10:50 |
| rlandy | soniya29|rover: thank you - pls make a list of any that are lagging and we can review | 10:50 |
| soniya29|rover | rlandy, sure | 10:51 |
| rlandy | soniya29|rover: on rr hackmd | 10:53 |
| rlandy | at line 99 added a section to log components that need attention | 10:53 |
| marios|ruck | rlandy: thanks we synced earlier with soniya29|rover and she is watching the d/stream & components for now and I will ping her with any other specific things | 10:59 |
| marios|ruck | rlandy: you can click on the d/stream fails on the hackmd i made them expandables there is inline trace and soniya29|rover will file bugs for the things that are consistent there (she posted test projects) | 10:59 |
| rlandy | marios|ruck: ack - I read that | 10:59 |
| ysandeep | rlandy, https://review.opendev.org/c/openstack/tripleo-upgrade/+/835409/1#message-979e53a95249af917d784d91586ce4b8d0c657ca | 11:47 |
| ysandeep | rlandy, can you +w this one too | 11:48 |
| *** dviroel|pto is now known as dviroel | 11:52 | |
| marios|ruck | soniya29|rover: are you digging into components upstream that rlandy pointed to ? | 11:57 |
| marios|ruck | soniya29|rover: especially wallaby 8 network needs some attention please | 11:58 |
| soniya29|rover | marios|ruck, i am filing a bug for 16.2, please give me few mins | 11:58 |
| marios|ruck | soniya29|rover: ack thanks | 11:58 |
| marios|ruck | soniya29|rover: what is the bug/do we have it at least 2x? (please don't file if it is only seen 1x) | 11:58 |
| *** pojadhav is now known as pojadhav|afk | 11:59 | |
| soniya29|rover | marios|ruck, it is consistent failure | 12:00 |
| marios|ruck | thanks soniya29|rover | 12:01 |
| ysandeep | pojadhav|afk, minor comment: https://code.engineering.redhat.com/gerrit/c/openstack/rrcockpit/+/315574 | 12:06 |
| rlandy | pojadhav|afk: soniya29|rover: bhagyashris: [rhos-dev] Berlin OpenInfra Summit Self-Nomination Process | 12:10 |
| rlandy | see email | 12:10 |
| rlandy | arxcruz: you're local and approved to apply | 12:11 |
| rlandy | pojadhav|afk: soniya29|rover: bhagyashris: everyone else is business need only | 12:11 |
| rlandy | pls see https://openinfra.dev/summit-covid/ | 12:12 |
| arxcruz | rlandy ack, this means i don't need to fill the form ? | 12:14 |
| rlandy | arxcruz: please fill out the form - explaining that you live local to the area | 12:15 |
| rlandy | info I got from management is that they have green-lighted local people | 12:15 |
| arxcruz | rlandy done | 12:15 |
| *** ysandeep is now known as ysandeep|afk | 12:25 | |
| rlandy | marios|ruck: when chandankumar returns, will add your and his keys to the ci creation script | 12:26 |
| rlandy | so you both can kick it | 12:26 |
| rlandy | running script now | 12:28 |
| marios|ruck | rlandy: ack thanks | 12:28 |
| rlandy | I think that is the last thing I exclusively own | 12:29 |
| rlandy | every other infra piece others should have access to | 12:29 |
| rlandy | ysandeep|afk: you are well | 12:30 |
| rlandy | will add your keys to trello create | 12:30 |
| rlandy | you will all need to create trello tokens | 12:31 |
| marios|ruck | soniya29|rover: o/ let me know if you need something wrt the components (are you still filing the bugzilla?) | 12:38 |
| rlandy | marios|ruck: new cards are on CIX now | 12:42 |
| marios|ruck | thank you rlandy | 12:43 |
| *** dviroel is now known as dviroel|brb | 12:43 | |
| rlandy | dviroel|brb: https://review.opendev.org/c/openstack/tripleo-docs/+/834957 - left comments but I think you updated the patch in the mean time | 12:47 |
| soniya29|rover | marios|ruck, filed the bug, now moving to components :) | 12:54 |
| marios|ruck | ok soniya29|rover | 12:56 |
| arxcruz | rlandy i'm trying to autohold a node, but it's the third time it's cleaned | 13:00 |
| rlandy | arxcruz: which cloud? | 13:03 |
| arxcruz | rdo | 13:03 |
| arxcruz | i'm trying to debug the webdownload test failure | 13:03 |
| arxcruz | https://review.rdoproject.org/r/c/testproject/+/37916 | 13:03 |
| rlandy | arxcruz: looking | 13:03 |
| rlandy | is the job running now? | 13:03 |
| arxcruz | rlandy finish a few minutes ago, and the node was already cleaned | 13:04 |
| arxcruz | i'm asking on rhos-ops | 13:04 |
| rlandy | arxcruz: ok - going to rekick and try hols | 13:04 |
| rlandy | hold | 13:04 |
| rlandy | arxcruz: periodic-tripleo-ci-centos-9-standalone-master | 13:05 |
| rlandy | force_periodic: true? | 13:05 |
| rlandy | can I add that? | 13:05 |
| arxcruz | rlandy i'll add here, needs that to hold ? | 13:09 |
| rlandy | arxcruz: I added | 13:09 |
| rlandy | no - irrelevant | 13:09 |
| rlandy | but needed to check test | 13:09 |
| rlandy | arxcruz: do you also want a depends-on skiplist revert | 13:09 |
| rlandy | before we restart job? | 13:09 |
| arxcruz | rlandy no, i'll manually run it | 13:11 |
| rlandy | ok - adding new hold when test starts | 13:11 |
| rlandy | idk if it holds if the test passes | 13:11 |
| arxcruz | really? it only holds if fails ? | 13:14 |
| *** ysandeep|afk is now known as ysandeep | 13:14 | |
| marios|ruck | arxcruz: yeah we have a var for it like force_fail or something would have to dig for that | 13:14 |
| marios|ruck | arxcruz: not sure if we can ask for that on the hold request | 13:15 |
| * marios|ruck brb | 13:15 | |
| marios|ruck | arxcruz: there fyi rlandy https://opendev.org/openstack/tripleo-ci/src/commit/cc3f50ba85cbe02df5ebc4111631d82baea8fe73/playbooks/tripleo-buildimages/run-v3.yaml#L27 | 13:23 |
| marios|ruck | add it to the zuul job vars in testproject | 13:23 |
| arxcruz | marios|ruck ty | 13:24 |
| marios|ruck | arxcruz: ack that one is wrong pointer but right var (for general case that clause ) https://opendev.org/openstack/tripleo-ci/src/commit/cc3f50ba85cbe02df5ebc4111631d82baea8fe73/roles/run-test/templates/toci_quickstart.sh.j2#L187 | 13:27 |
| *** dviroel|brb is now known as dviroel | 13:27 | |
| rlandy | arxcruz: try your hold command again once the job starts | 13:28 |
| rlandy | and you see a node shown on the console output | 13:28 |
| rlandy | marios|ruck: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-master two green runs here? | 13:35 |
| dviroel | rlandy: looking your comments in https://review.opendev.org/c/openstack/tripleo-docs/+/834957 - lets remove everything related to modules here? | 13:38 |
| marios|ruck | rlandy: ack they reported after i filed the bug | 13:42 |
| marios|ruck | rlandy: i'll track and update the bug if we don't see again will close out tomorrow | 13:43 |
| rlandy | dviroel: ack | 13:43 |
| rlandy | c8 only | 13:43 |
| marios|ruck | rlandy: we should get master promotion (one of those green was from my https://review.rdoproject.org/r/c/testproject/+/41031 ) | 13:44 |
| rlandy | marios|ruck++ yay | 13:44 |
| rlandy | bhagyashris: hey - why is ysandeep on the rr three times in May? | 13:53 |
| ysandeep | marios|ruck, rlandy: fyi.. I have added an item on checkpoint agenda(in CI section - we should get out turn early) to discuss c8 wallaby coverage, reading https://review.opendev.org/c/openstack/tripleo-specs/+/834076/1..2/specs/wallaby/mixed-operating-system-versions.rst#b218 looks like James was under impression we will keep wallaby coverage on both cs8 and cs9 , let's see how discussion turns out. | 13:53 |
| ysandeep | rlandy: I requested her, I prefer that way if that's not a problem for anyone. | 13:53 |
| marios|ruck | rlandy: thanks ysandeep | 13:53 |
| rlandy | ysandeep: k - let us know | 13:54 |
| rlandy | I did bring that up - but was shouted down at last meeting | 13:55 |
| *** dasm|off is now known as dasm | 13:57 | |
| dasm | o/ | 13:57 |
| ysandeep | rlandy: fyi.. I am getting 1 less turn as upstream ruck/rover in comparison to all, because I have monitored downstream line for first 2 months. | 13:58 |
| * dasm will join standup after rbac meeting | 14:00 | |
| bhagyashris | akahat, chandankumar rlandy soniya29|rover pojadhav|afk | 14:00 |
| bhagyashris | marios|ruck, | 14:01 |
| bhagyashris | scrum time | 14:01 |
| bhagyashris | dasm, ^ | 14:01 |
| *** pojadhav|afk is now known as pojadhav | 14:02 | |
| marios|ruck | soniya29|rover: please let me know where you get to before you go so i can have a look regarding components ... should be just 2 to check mainly network wallaby 8 and master compute components were like 11 days old | 14:14 |
| rlandy | rcastillo; you might also be interested in https://review.opendev.org/c/openstack/openstack-virtual-baremetal/+/834813 | 14:14 |
| rlandy | uefi boot for OVB | 14:15 |
| soniya29|rover | marios|ruck, sure | 14:15 |
| marios|ruck | rlandy: ysandeep: soniya29|rover: fyi https://review.rdoproject.org/r/c/config/+/41041 | 14:15 |
| marios|ruck | (once we get the go-ahead for c8 wallaby teardown today ) | 14:15 |
| ysandeep | marios|ruck: ack | 14:16 |
| * soniya29|rover brb | 14:31 | |
| dpawlik | marios|ruck, arxcruz: hey, if you have a min, please check https://review.opendev.org/c/openstack/ci-log-processing/+/835455 | 14:38 |
| marios|ruck | dpawlik: k will add to my next reviews | 14:40 |
| dpawlik | thanks | 14:40 |
| dpawlik | marios|ruck: this one would be also awesome to review https://review.opendev.org/c/openstack/ci-log-processing/+/833011 | 14:45 |
| *** amoralej|off is now known as amoralej | 14:53 | |
| marios|ruck | rlandy: lets get 5 mins break before next call :) | 14:54 |
| dasm | https://www.youtube.com/watch?v=L2zqTYgcpfg | 14:57 |
| ysandeep | marios|ruck, just confirming will you join checkpoint or production call? | 15:02 |
| soniya29|rover | marios|ruck, rlandy, do we have some different job name conventions like in openstack-tempest-component we have name as 'periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-tempest-train' and in criteria file we have https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config_environments/rdo/CentOS-8/train.yaml#L32 | 15:19 |
| soniya29|rover | marios|ruck: rlandy, it is bit confusing me | 15:19 |
| rlandy | soniya29|rover: diff between integration and component lines | 15:20 |
| rlandy | so the 'periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-tempest-train' job runs in the tempest line | 15:20 |
| rlandy | while the criteria you are looking at would refer to the integration line | 15:21 |
| rlandy | the component line criteria are in: https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/dlrnapi_promoter/config | 15:21 |
| marios|ruck | soniya29|rover: (what rlandy said :) ) - sorry we are in a call | 15:22 |
| *** marios|ruck is now known as marios|ruck|call | 15:23 | |
| soniya29|rover | marios|ruck|call, okay | 15:23 |
| soniya29|rover | rlandy, thanks :) | 15:23 |
| marios|ruck|call | soniya29|rover: are you filing bugs for the component lines or you think inconsistent issue? | 15:34 |
| soniya29|rover | marios|ruck|call, i see full-tempest-scenario train is on tempest tests and is in promotion criteria | 15:39 |
| soniya29|rover | marios|ruck|call, i am investigating the issue | 15:39 |
| marios|ruck|call | soniya29|rover: train should be ok per rlandy notes there... so | 15:39 |
| marios|ruck|call | soniya29|rover: mainly can you please check network wallaby 8 and master compute components | 15:39 |
| soniya29|rover | marios|ruck|call, okay | 15:40 |
| soniya29|rover | marios|ruck|call, i am going first with network wallaby 8 | 15:40 |
| marios|ruck|call | soniya29|rover: please see private and let me know before you go what you have done thanks | 15:42 |
| soniya29|rover | marios|ruck|call, i will let you know | 15:43 |
| rlandy | soniya29|rover: marios|ruck|call: | 15:44 |
| rlandy | periodic-tripleo-ci-centos-8-standalone-full-tempest-api-network-wallabytestprojectmastercheck39960,17 | 15:45 |
| rlandy | 2 hrs 19 mins 59 secs2022-03-28 11:53:35SUCCESS | 15:45 |
| rlandy | periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-wallabytestprojectmastercheck39960,17 | 15:45 |
| rlandy | 3 hrs 22 mins 6 secs2022-03-28 11:53:05FAILURE | 15:45 |
| rlandy | last testproject on that | 15:45 |
| marios|ruck|call | soniya29|rover: i will leave very soon after this call so please update the hackmd with anytihng you have on those components please for handoff to rlandy | 15:58 |
| soniya29|rover | marios|ruck|call, sure | 15:58 |
| marios|ruck|call | soniya29|rover: if you didn't have time to dig into those then you should do it first thing tomorrow but let rlandy know either way before you go please (its very late in pune now?) | 15:59 |
| rlandy | marios|ruck|call: do you want me to look at/do anything? | 15:59 |
| marios|ruck|call | rlandy: no i think we have all the things... info/latest status on the hackmd, otherwise soniya29|rover has been digging at components ^^ | 15:59 |
| rlandy | ok - leaving it | 16:00 |
| soniya29|rover | marios|ruck|call, thanks for your so kind concern, i know its late in pune, but before i leave for today i will update the hackmd and will properly handoff to rlandy | 16:02 |
| soniya29|rover | marios|ruck|call, thanks | 16:02 |
| marios|ruck|call | soniya29|rover: k | 16:02 |
| *** marios|ruck|call is now known as marios|ruck | 16:03 | |
| marios|ruck | rlandy: off in a minute need sthing ? | 16:10 |
| marios|ruck | soniya29|rover: ^^ | 16:10 |
| soniya29|rover | marios|ruck, nope | 16:11 |
| rlandy | marios|ruck: no - have a good night | 16:12 |
| *** dviroel is now known as dviroel|lunch | 16:13 | |
| marios|ruck | rlandy: looks like we did get the master c9 promotion | 16:14 |
| marios|ruck | rlandy: will pickup tomorrow then bai | 16:14 |
| *** marios|ruck is now known as marios|out | 16:15 | |
| rlandy | great | 16:16 |
| *** ysandeep is now known as ysandeep|out | 16:20 | |
| ysandeep|out | regarding --> I have discussed c8 wallaby coverage in checkpoint call today, the suggestion was to keep one job which really test overcloud (and not only undercloud) as minimum set for c8 wallaby coverage. People on call have suggestion that atleast vanilla standalone job should run for both c8 and c9 for wallaby in check/gate. | 16:21 |
| ysandeep|out | marios|out, rlandy ^^ lets chat more tomorrow | 16:22 |
| marios|out | ysandeep|out: ack | 16:22 |
| * ysandeep|out out for the day | 16:22 | |
| soniya29|rover | rlandy, for fs001-network-wallaby the testproject is failing on TestNetworkBasicOps test and latest run fails on overcloud deployment failure | 16:27 |
| soniya29|rover | both are failing with different failures | 16:27 |
| soniya29|rover | and is promotion blocker | 16:27 |
| soniya29|rover | rlandy, i think mentioning both the failures in bug can be a good idea | 16:28 |
| *** jbadiapa is now known as jbadiapa|off | 16:30 | |
| rlandy | soniya29|rover: look at the history | 16:30 |
| rlandy | and rerun latest | 16:31 |
| rlandy | to see if it's still an issue when it gets there | 16:31 |
| rlandy | but the bug is worth it at this point | 16:31 |
| soniya29|rover | rlandy, i think we require a bug since it is failing from 26 | 16:32 |
| rlandy | soniya29|rover: yes - go ahead | 16:32 |
| rlandy | soniya29|rover: you can discuss with marios|out in morning | 16:32 |
| soniya29|rover | rlandy, i will rerun the job and also file the bug with both the failures | 16:32 |
| soniya29|rover | rlandy, sure | 16:32 |
| rlandy | ok | 16:32 |
| rlandy | great | 16:33 |
| soniya29|rover | rlandy, https://bugs.launchpad.net/tripleo/+bug/1966797 | 16:42 |
| rlandy | thanks - will check after calls | 16:42 |
| soniya29|rover | rlandy, with network wallaby8 other things are good | 16:48 |
| rlandy | great | 16:49 |
| soniya29|rover | rlandy, moving to master compute components | 16:49 |
| *** jpena is now known as jpena|off | 16:52 | |
| *** dviroel|lunch is now known as dviroel | 16:55 | |
| soniya29|rover | rlandy, https://bugs.launchpad.net/tripleo/+bug/1966799 | 17:11 |
| rlandy | soniya29|rover: pls make the importnace of those bugs critical | 17:13 |
| soniya29|rover | rlandy, thanks, done | 17:14 |
| rlandy | soniya29|rover: late for you | 17:19 |
| rlandy | you can pick up the rest tomorrow | 17:19 |
| soniya29|rover | rlandy, not a problem, i will complete the master compute components and after that i will be leaving | 17:20 |
| soniya29|rover | rlandy, cs9 full tempest api is also failing with different failures 1. tempest tests 2. ansible execution failed | 17:21 |
| soniya29|rover | is a promotion blocker as well | 17:21 |
| rlandy | soniya29|rover: only bug if there is a consistent failure | 17:22 |
| soniya29|rover | rlandy, it is failing from 24th march | 17:23 |
| soniya29|rover | rlandy, if we see the latest failures | 17:23 |
| * rlandy looks | 17:24 | |
| soniya29|rover | rlandy, one of them fails over tempest tests and other one fails for ansible execution | 17:24 |
| soniya29|rover | rlandy, https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-compute-master/ec94fe7/logs/undercloud/var/log/tempest/stestr_results.html.gz | 17:24 |
| rlandy | soniya29|rover: ansible exceution of what? deploy | 17:24 |
| soniya29|rover | rlandy, yes | 17:24 |
| soniya29|rover | rlandy, https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-compute-master/9ad9150/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz | 17:26 |
| rlandy | I am looking through all the failures - sec' | 17:26 |
| rlandy | 22-03-28 08:02:20,386 58701 INFO tripleo_common.image.image_uploader [ ] Non-2xx: id 21bb289a5d2c5578fa4bef7305af3d2834b00261, status 404, reason Not Found, text {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"c16775607f2619e1086d495b6ddb8612"}}]} | 17:27 |
| rlandy | latest error is an infra hitch | 17:27 |
| rlandy | 2022-03-28 04:02:20.522409 | fa163ed2-2176-a182-cbdb-000000001504 | FATAL | Container image prepare | undercloud | error={"changed": false, "error": "Not found image: https://trunk.registry.rdoproject.org/v2/tripleomastercentos9/openstack-cinder-volume/manifests/c16775607f2619e1086d495b6ddb8612", | 17:28 |
| rlandy | you can ignore that one | 17:28 |
| soniya29|rover | it is promotion blocker | 17:29 |
| rlandy | 2022-03-26 04:32:54.009817 | fa163eaf-0c9b-4f6e-e97b-000000001504 | FATAL | Container image prepare | undercloud | error={"changed": false, "error": "Modifying image 192.168.24.1:8787/tripleomastercentos9/openstack-cinder-volume:c16775607f2619e1086d495b6ddb8612 failed", "msg": "Error running container image prepare: Modifying image 192.168.24.1:8787/tripleomastercentos9/openstack-cinder-volume:c16775607f2619e1086d495b6ddb8612 | 17:30 |
| rlandy | failed", "params": {}, "success": false} | 17:30 |
| rlandy | ^6 error I see twice | 17:30 |
| rlandy | but it's different from latest | 17:30 |
| rlandy | soniya29|rover: I would rerun here | 17:30 |
| rlandy | to see what the real problem is | 17:30 |
| rlandy | you'll need more debug | 17:30 |
| rlandy | otherwise the bug will send people in the wring direction | 17:31 |
| soniya29|rover | rlandy, tommorrow morning i will have look over it to see if we have consistent failures and will have more time to debug as well | 17:34 |
| soniya29|rover | rlandy, i have made a note of it | 17:34 |
| rlandy | soniya29|rover: sounds good | 17:36 |
| rlandy | have a good night | 17:36 |
| soniya29|rover | rlandy, same thing is with cs9 standalone compute master | 17:42 |
| soniya29|rover | rlandy, its confusing | 17:42 |
| soniya29|rover | its also a promotion blocker | 17:42 |
| soniya29|rover | rlandy, these are all the issues with master compute | 17:44 |
| *** amoralej is now known as amoralej|off | 17:51 | |
| soniya29|rover | rlandy, should we sync? | 17:52 |
| rlandy | you can add them to the same bug | 17:52 |
| rlandy | if it's the same issue | 17:52 |
| rlandy | soniya29|rover: yep - and component line blockers are promotion blockers | 17:53 |
| rlandy | soniya29|rover: putting in a testproject to rerun periodic-tripleo-ci-centos-8-scenario007-standalone-ussuri | 17:55 |
| rlandy | so ussuri can promote | 17:55 |
| soniya29|rover | rlandy: okay, i will add them to the bug | 17:56 |
| soniya29|rover | rlandy, i have commented on the bug here:- https://bugs.launchpad.net/tripleo/+bug/1966797 | 17:59 |
| soniya29|rover | rlandy, leaving for the day | 18:00 |
| soniya29|rover | anything else you want me to look? | 18:01 |
| rlandy | soniya29|rover: that's all thanks | 18:06 |
| rlandy | have a good night | 18:06 |
| soniya29|rover | rlandy, thanks | 18:07 |
| soniya29|rover | rlandy, :) | 18:07 |
| rlandy | dasm: vexxhost is looking better now | 18:24 |
| rlandy | master promoted | 18:25 |
| rlandy | so you can try ovb there | 18:25 |
| dasm | ack, thanks | 18:29 |
| *** dviroel is now known as dviroel|out | 20:30 | |
| rcastillo | rlandy: couple centosci c8 wallaby issues I saw. May be related to my changes in quickstart. I'll see if I can reproduce in my env | 21:58 |
| rlandy | rcastillo: just in the job or did we hose something else? | 22:00 |
| rcastillo | just these jobs afaict | 22:00 |
| rlandy | ok - good | 22:23 |
| *** rlandy is now known as rlandy|out | 22:55 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!