Friday, 2021-12-17

*** rlandy|ruck is now known as rlandy|out00:12
*** ysandeep|out is now known as ysandeep01:39
*** sshnaidm is now known as sshnaidm|afk02:45
*** ysandeep is now known as ysandeep|brb06:19
*** ysandeep|brb is now known as ysandeep06:57
ysandeepchandankumar, marios hey I see one comment from rlandy here that she prefer to merge this patch on monday: https://review.opendev.org/c/openstack/tripleo-ci/+/816991/17#message-f3e8fe7b20e68fd2c9d36a3b16b94e27dcc07bfd08:14
mariosysandeep: o/ 08:14
ysandeepchandankumar, looks like -w didn't worked , job is already in gate08:14
marioschandankumar: ysandeep: i think its ok... its early enough 08:14
marioschandankumar: ysandeep: if there is aproblem i will revert it08:14
marios'its early enough' i mean it is going to merge by lunchtime hopefully :)08:15
ysandeepmarios, ack, patch lgtm08:15
mariosysandeep: thanks, yeah lgtm last time too but it had a nit :D let's see 08:15
mariosysandeep: i don't know08:16
mariosysandeep: maybe it wont merge? 08:16
mariosysandeep: like after it runs in gate it may not merge cos it doesn't have the w+ anymore 08:16
mariosbut lets see if it merges also fine chandankumar ysandeep 08:16
ysandeepmarios: okay, sounds good 08:17
chandankumarack08:21
*** amoralej|off is now known as amoralej08:23
*** chem is now known as Guest869008:59
*** ysandeep is now known as ysandeep|lunch09:11
*** ysandeep|lunch is now known as ysandeep09:42
mariosbiab10:12
ysandeepakahat, are you still hitting that standalone failure which faced yesterday? 10:15
ysandeepakahat, I can chat about that now if that is still an issue10:15
ysandeepfolks o/ Please add in your review list: "Add Automation around node hold" - https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/37025 10:41
chandankumaranbanerj|ruck: please abandon this patch https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/82139210:57
*** rlandy_ is now known as rlandy|ruck11:11
pojadhavfolks, pls review : https://review.rdoproject.org/r/c/rdo-jobs/+/3700011:12
*** dviroel|out is now known as dviroel|rover11:14
rlandy|ruckpojadhav: pls put it on the review hackmd11:14
pojadhavrlandy|ruck, sure11:14
rlandy|ruckanyone see ananya today?11:15
rlandy|ruckdviroel|rover: ^^11:15
dviroel|roverrlandy|ruck: just log in, didn't talk with anyone yet =/\11:16
rlandy|ruckchandankumar: hey - you see ananya today?11:16
chandankumarrlandy|ruck: nope11:17
chandankumarrlandy|ruck: based on tripleo-ci gchat, she was active 55 mins back11:17
rlandy|ruckhmmm - maybe she got disconnected11:18
rlandy|ruckdviroel|rover: there are a ton of tox failures around11:19
rlandy|rucknot just tripleo repos11:19
rlandy|ruckleft note11:19
rlandy|ruckchandankumar: review time?11:20
chandankumarrlandy|ruck: based on #opendev https://status.python.org/ there was Service Interruption due to database upgrades on pypi11:20
* frenzy_friday has bad internet11:27
dviroel|roverfrenzy_friday: o/11:27
*** frenzy_friday is now known as anbanerj|ruck11:27
anbanerj|ruckdviroel|rover, Hey 0/11:27
dviroel|roveranbanerj|ruck: ussuri will promoted, tks11:31
rlandy|ruckanbanerj|ruck: hey - good to see you11:32
rlandy|ruckdviroel|rover: anbanerj|ruck: can you take a look at recent gate failures11:32
rlandy|ruckand see if we have any real issues there11:32
anbanerj|ruckrlandy|ruck, a few gate failures had conflicting pip packages11:36
anbanerj|ruckshould I create a bug for those?11:36
dviroel|roverat least, the last 5 failures are related to pip package install11:37
dviroel|roveri think we should open a bug11:38
ysandeepchandankumar, thanks for review, I have updated the patch https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/37025/9..10/ci-scripts/node_holder/node_holder.py 11:42
ysandeepreplied about your other comment.11:43
rlandy|ruckanbanerj|ruck: and idea on the failing master line: https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-main+11:44
rlandy|ruckcdn issue11:47
rlandy|ruckok11:47
anbanerj|ruckrlandy|ruck, ^ again conflicting dependencies.11:50
*** ysandeep is now known as ysandeep|afk11:51
anbanerj|ruckShould I testproject in case they pass?11:51
rlandy|ruckakahat: hey - can you take a look at the comments in https://review.rdoproject.org/r/c/rdo-jobs/+/36490 and rabase - so we can merge?11:53
akahatrlandy|ruck, yes.. already on it.11:53
rlandy|ruckmarios: hey - can you add dviroel|rover and rcastillo pinning patches to your review list?11:55
rlandy|ruckCompose pinning:11:55
rlandy|ruck    https://review.opendev.org/q/topic:"compose_repos_dep"+(status:open OR status:merged)11:55
rlandy|ruck    https://review.rdoproject.org/r/q/topic:"compose_repos_dep"+(status:open OR status:merged)11:55
rlandy|ruck    testproject: https://review.rdoproject.org/r/c/testproject/+/3672711:55
rlandy|ruckchandankumar: ^^11:55
rlandy|rucklet's try merge this on monday if no objections11:55
chandankumarhttps://review.opendev.org/c/openstack/tripleo-ci/+/82169811:56
mariosrlandy|ruck: o/ sure i already reviewd some today 12:00
mariosrlandy|ruck: adding topic to my list12:01
dviroel|rovertks team12:02
rlandy|ruckanbanerj|ruck: dviroel|rover: do we need to ruck/rover sync?12:03
rlandy|ruckanything other than the red upstream?12:03
rlandy|ruckysandeep|afk: hey - if you are not out yet for the year - pls ping and let's sync on what12:03
rlandy|rucks left of your stuff12:03
anbanerj|ruckrlandy|ruck, not much happened today. Majority are due to pypi cdn issue. Should I raise it somewhere?12:04
chandankumarrlandy|ruck: missed this https://review.opendev.org/c/openstack/tripleo-ci/+/82179912:04
rlandy|ruckpojadhav: ysandeep|afk will be out for two weeks  ps keep an eye on downstream12:04
rlandy|ruckanbanerj|ruck: it was raised on opendev ... chandankumar passed the suggested patch infra has12:05
marioschandankumar: new issues? https://review.rdoproject.org/r/c/testproject/+/36267/43#message-883fb31ba0bd36175b0c38ff82ed60ccf8d7ceb1 12:05
anbanerj|ruckcool, me checks. Bad internet connection today :( 12:05
marioschandankumar: 2021-12-17 09:49:36.673836 | LOOP [build-containers : Install deps from upper-constraints]12:05
marios2021-12-17 09:49:38.647624 | primary | ERROR: Item: tripleo-common12:05
dviroel|roverrlandy|ruck: anbanerj|ruck yeah, i can't see any other issue for now12:05
marioswth 12:05
marios:/12:05
rlandy|ruckchandankumar: ok to merge? https://review.opendev.org/c/openstack/tripleo-ci/+/82179912:06
chandankumarmarios: it's everywhere12:06
marios:/12:06
mariosthanks chandankumar 12:06
chandankumarrlandy|ruck: and anbanerj|ruck are talking about the same12:06
mariosi see12:06
rlandy|ruckmarios: you are not alone12:06
chandankumarrlandy|ruck: yes12:06
rlandy|ruckit's all of openstack12:06
* marios feels slightly better its not just more 9-stream randomness12:06
rlandy|ruckhttps://review.opendev.org/c/opendev/system-config/+/82209512:06
rlandy|ruckthanks to chandankumar for pointing us to ^^12:06
mariosthanks rlandy|ruck chandankumar 12:08
*** ysandeep|afk is now known as ysandeep12:13
ysandeeprlandy|ruck, hey o/ i am available to chat 12:13
rlandy|ruckysandeep: https://meet.google.com/nqn-crpj-kfn?pli=1&authuser=012:17
ysandeeprlandy|ruck, https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3702512:21
*** frenzy_friday is now known as anbanerj|ruck12:21
rlandy|ruckchandankumar: can you revote on https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3702512:21
rlandy|ruckthen can merge before ysandeep goes12:21
rlandy|ruckstandalone script12:21
chandankumarrlandy|ruck: marios https://review.opendev.org/c/openstack/tripleo-ci/+/821698/ and https://review.opendev.org/c/openstack/tripleo-ci/+/822101 please have a look when free, thanks :-)12:27
*** beagles is now known as beagles_pto12:28
marioschandankumar: added to list 12:31
ysandeeprlandy|ruck, After seeing all those sc010 triggering patches, I just want to say - computer engineering is about finding solution to problem that never existed. :D lol! 12:35
marios:)12:37
dviroel|roveranbanerj|ruck: running failed jobs for train, victoria and wallaby (previous hash)12:38
pojadhavrlandy|ruck, ack12:38
rlandy|ruckysandeep: :)12:38
rlandy|ruckysandeep: hack, code and pray12:39
ysandeeprlandy|ruck: lets merge that https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/37025, I think people will have more feedback once they use the tool - I can fix any improvements (if any) when I get back.12:45
bhagyashrisakahat, hey i think you missed this comment https://review.rdoproject.org/r/c/rdo-jobs/+/36490/10#message-e72b15a153f30d94630f59ae2e069c15d9b5d3ba12:58
bhagyashrisand above one from chandan12:58
akahatbhagyashris, sorry.. i forgot to mention.. I'm creating separate patch for it.13:04
bhagyashrisakahat, ack i think chandan'scomment you can fix in the same patch 13:05
akahatbhagyashris, yes. already did.13:05
bhagyashrisakahat, ack13:05
bhagyashrisakahat, thank you :)13:05
akahat:)13:06
bhagyashrisrlandy|ruck, chandankumar kindly review this patch https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/37225 when you have free time. thanks :)13:06
rlandy|ruckbhagyashris: is victoria already set up on the new promoter?13:17
rlandy|ruckbhagyashris: I'm ok with this - but can you merge the patch on monday morning so we don;t lose weekend promotions?13:17
rlandy|ruckchandankumar: ^^?13:17
bhagyashrisrlandy|ruck, yeah i did changes manually on old promoter server13:18
rlandy|ruckbhagyashris: k - great13:18
rlandy|ruckso I +2'ed it13:18
bhagyashrisrlandy|ruck, up that will be great if we merge that change on my monday morning 13:18
rlandy|ruckbhagyashris: when you get in, chandankumar can merge that for you  - and you can run13:19
rlandy|ruckand test over the week13:19
*** amoralej is now known as amoralej|lunch13:19
rlandy|ruckbhagyashris++ thank you for taking on this work13:19
bhagyashrisi will reach out to ysandeep or chandankumar in our Monday morning 13:19
rlandy|ruckysandeep is out13:19
bhagyashrisohh ok13:19
bhagyashrisi will check with chandankumar than13:19
rlandy|ruckyep13:19
bhagyashristhanks :)13:20
rlandy|ruckbhagyashris: hopefully we will promote over the weekend - then you will have a few days grace to get the logs working etc.13:20
* ysandeep reading back13:20
rlandy|ruckysandeep: no worries13:20
rlandy|ruckjust mentioned you are out13:20
bhagyashrisrlandy|ruck, yeah hopefully ...13:21
*** pojadhav is now known as pojadhav|brb13:24
rlandy|ruckakahat: ysandeep; here's the last of the downstream side triggers: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/299570/1/zuul.d/projects.yaml13:29
rlandy|ruckpls vote13:29
rlandy|ruckadding rdo triggers13:29
ysandeeprlandy|ruck, i just did sanity check there, looks good +2 , thanks!13:31
rlandy|ruckadding rdo triggers13:36
*** jpena|off is now known as jpena13:49
rlandy|ruckamoralej|lunch: thanks for fixin the weirdo jobs13:50
rlandy|rucknow of course our jobs failed this run - but re-triggering13:50
rlandy|ruckdviroel|rover: anbanerj|ruck: hey - I have an outage today from around 10 today to 12ish - dr's appointment13:59
anbanerj|ruckrlandy|ruck, in 30 mins right? In that case shall we do the rr hand off now?14:00
rlandy|ruckone hour14:00
anbanerj|ruckoh, then cool14:01
*** amoralej|lunch is now known as amoralej14:01
rlandy|ruckactually 10: 1514:01
rlandy|ruckI guess they moved my appointment 14:02
dviroel|roverrlandy|ruck: ok, np :)14:02
*** ysandeep is now known as ysandeep|dinner14:08
*** pojadhav|brb is now known as pojadhav14:19
dviroel|roverhttps://bugs.launchpad.net/tripleo/+bug/195491814:34
rlandy|ruckchandankumar: akahat: https://review.rdoproject.org/r/c/rdo-jobs/+/37276 Add downstream triggers for component jobs14:46
rlandy|ruckhttps://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/299570/1/zuul.d/projects.yaml#4714:46
*** ysandeep|dinner is now known as ysandeep14:52
dasmo/15:00
chandankumarsee ya, Have a nice weekend and Happy holidays ysandeep :-)15:02
ysandeepchandankumar, happy new year!15:02
ysandeepchandankumar, we are on happy friday call if you want to join15:02
*** rlandy|ruck is now known as rlandy|dr_appt15:07
dasmrcastillo: ysandeep asked if we could start KT earlier. are you around?15:08
*** dviroel|rover is now known as dviroel|rover|lunch15:09
ysandeeprcastillo, hey you around, I see you haven't accepted the new invite?15:30
ade_leejpodivin, I see you narrowed it down to the tempest patches16:11
*** dviroel|rover|lunch is now known as dviroel|rover16:33
ysandeepdasm, rcastillo everything we discusssed about component lines and integration line is here: https://hackmd.io/5uYRmLaOTI2raTbHWsaiSQ 16:35
ysandeepdasm, rcastillo some obselete stuff here but you will find this useful: https://docs.openstack.org/tripleo-docs/latest/ci/dlrn-promoter-overview.html 16:36
rcastilloysandeep: hadn't seen that doc, thanks!16:36
dasmysandeep: thanks! enjoy your vacation16:36
weshayHave a good break all ya'll16:36
ysandeepHappy new year and wonderful vacation to all!16:37
*** ysandeep is now known as ysandeep|out16:39
dviroel|roverweshay: hey, tks, you too o/16:49
*** jpena is now known as jpena|off17:13
*** rlandy|dr_appt is now known as rlandy|ruck17:34
rlandy|ruckweshay: hey - same to you17:37
*** amoralej is now known as amoralej|off17:55
rlandy|ruckdviroel|rover: hey - all ok here?18:02
dviroel|roverit seems so18:03
rlandy|ruckwoohoo ...18:03
rlandy|rucktrigger success 18:03
rlandy|ruckperiodic-tripleo-ci-centos-8-scenario010-kvm-internal-standalone-wallabytripleo-downstream-trigger-nested-virtmastercheck-rdo37280,118:03
rlandy|ruck1 hr 31 mins 12 secs2021-12-17 16:31:02SUCCESS18:03
dviroel|rovervictoria and train promoted18:03
rlandy|ruckgreat18:04
* rlandy|ruck moving car - brb18:04
dviroel|roverrlandy|ruck:  Error: error adding Infra Container: Error initializing source docker://registry.access.redhat.com/ubi8/pause:latest: error pinging docker registry registry.access.redhat.com: invalid status code from registry 504 (Gateway Timeout)18:12
dviroel|roverwallaby jobs failing on overcloud deploy ^18:12
dviroel|rovermay be a temporary issue with redhat registry18:13
* rlandy|ruck looks18:14
rlandy|ruck docker://registry.access.redhat.com/ubi8/pause:latest ha18:16
rlandy|ruckdoes it always pull from there?18:16
dviroel|rovernot sure18:16
rlandy|rucklet's check18:16
dviroel|roverhttps://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/tripleo_podman/defaults/main.yml#L8318:17
rlandy|ruckok - so then yeah18:18
rlandy|rucknot a fall back18:18
rlandy|ruckI woudl assume some registry issue18:18
dviroel|roverdocker pull give me 504 too18:19
rlandy|ruckchecking if there is a reported outage18:20
rlandy|ruckregistry.access.redhat.com is in https://access.redhat.com/RegistryAuthentication#red-hat-registries-118:21
rlandy|rucklet's try run again in a bit18:21
rlandy|ruckimage and container builds passed18:22
rlandy|ruckso they must have gotten by18:22
rlandy|ruckdviroel|rover: wouldn't worry too much - let's recheck in about an hour18:23
dviroel|roversure18:23
dviroel|rover"Status: Downloaded newer image for registry.access.redhat.com/ubi8/pause:latest"19:11
dviroel|rovertrying again19:11
dviroel|rover"rror response from daemon: received unexpected HTTP status: 504 Gateway Time-out"19:12
dviroel|roverstill unstable19:12
rlandy|ruckdviroel|rover: promoted yesterday - so we have time19:14
rlandy|ruckI'll tell the next ruck rovers you didn't just ignore the error :)19:14
dviroel|rover:)19:15
ade_leedviroel|rover, rlandy|ruck hey19:23
ade_leedviroel|rover, rlandy|ruck I'd like to do an experiment on scenario 4 , but not quite sure how to do it ..19:24
rlandy|ruckexperiment ... ok ...19:24
ade_leeso in scenario 4 , we know that there is some issue that causes ceph_rgw to fail19:25
ade_leeso fultonj suggested that until we get that fixed we try run scenario 4 with a different env file19:25
ade_leefultonj> change this19:26
ade_lee<fultonj> -e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml19:26
ade_lee<fultonj> to this19:26
ade_lee<fultonj>   -e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm-rbd-only.yaml19:26
ade_leeso - trying to figure out how to do that for scenario 4 ..19:26
ade_leerlandy|ruck, here is one example of scenario 4 -- https://zuul.opendev.org/t/openstack/build/db53dd4a73e94d62927717589976774b/logs19:27
ade_leecant seem to find the config that tells it to use ceph19:27
rlandy|ruckok ... so scenario004 has some ceph parameters ... 19:29
ade_leerlandy|ruck, it seems to be saying to use /usr/share/openstack-tripleo-heat-templates/ci/environments/scenario004-standalone.yaml19:29
rlandy|ruckhttps://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/scenario004-standalone.yaml19:29
ade_leeah :)19:29
rlandy|ruck  OS::TripleO::Services::CephClient: ../../deployment/cephadm/ceph-client.yaml19:30
rlandy|ruckyou can hack up that file19:30
rlandy|rucksubmit it DNM19:30
rlandy|ruckand see19:30
ade_leeack - looking19:31
rlandy|ruckdviroel|rover: you can do your compliance training while registery is unstable :)19:33
rlandy|ruckthat's why HR gives us these things19:34
rlandy|ruckin case our infra is down19:34
dviroel|roverrlandy|ruck: lolz - thats true19:34
dviroel|rovergah, there are 4 to complete 19:37
rlandy|ruckdviroel|rover: the first one is just a confirmation you read it19:42
rlandy|ruckso really it's three19:42
rlandy|ruckMerged openstack/tripleo-quickstart-extras master: Set pcs version to 0.10.11-1 in all releases  https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/82187419:43
rlandy|ruck^^ merged :)19:43
dviroel|roveryep19:43
dviroel|rover3 to go19:49
dviroel|rover2 to go20:18
dviroel|rovermaybe another day 20:18
dviroel|rover:)20:18
dviroel|roverregistry seems more stable now20:18
dviroel|roverwallaby line will start in less than one hour, so lets wait20:19
* dviroel|rover brb20:34
*** dviroel|rover is now known as dviroel|rover|brb20:34
rlandy|ruckdviroel|rover|brb: ack - should be ok now20:38
rlandy|ruckdviroel|rover|brb: if it's not the container reg instability it's https://review.opendev.org/c/openstack/tripleo-ci/+/821799/5/roles/build-containers/vars/main.yaml20:52
rlandy|ruck^^ that merged today20:52
*** dviroel|rover|brb is now known as dviroel|rover21:20
dviroel|roverno lucky, wallaby looks bad21:21
dviroel|roverbut failing on pre, doesn't look to be the same issue21:22
dviroel|rovermaybe, the lines went for the weekend already21:22
dviroel|rovergoing out, happy weekend for you all, see you on monday21:29
*** dviroel|rover is now known as dviroel|out21:29
dasm\o21:32
* dasm => out22:00
-opendevstatus- NOTICE: The review.opendev.org server is being rebooted to validate a routing configuration update, and should return to service shortly22:27

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!