Wednesday, 2020-12-23

« Tuesday, 2020-12-22 Index Thursday, 2020-12-24 »
*** rlandy has quit IRC00:27
*** sanjayu_ has quit IRC00:27
*** jmasud has quit IRC00:38
*** rfolco has joined #oooq00:48
*** jmasud has joined #oooq01:01
*** ysandeep|away is now known as ysandeep01:06
*** rfolco has quit IRC02:21
*** jmasud has quit IRC02:55
*** holser has quit IRC04:21
*** holser has joined #oooq04:23
*** ykarel has joined #oooq04:42
*** sanjayu_ has joined #oooq04:42
*** holser has quit IRC04:44
*** udesale has joined #oooq05:15
*** jmasud has joined #oooq05:41
*** skramaja has joined #oooq06:08
*** holser has joined #oooq06:09
*** holser has quit IRC06:34
*** holser has joined #oooq06:34
*** holser has quit IRC06:40
*** marios has joined #oooq06:42
*** holser has joined #oooq09:25
*** derekh has joined #oooq09:36
*** tosky has joined #oooq10:12
*** jmasud has quit IRC10:13
* bhagyashris|ruck brb10:26
*** derekh has quit IRC10:30
*** jbadiapa has joined #oooq10:34
zbrbhagyashris|ruck: have you checked https://review.rdoproject.org/r/#/c/31400/ ?10:36
*** ykarel has quit IRC10:37
*** ykarel has joined #oooq10:40
*** holser_ has joined #oooq10:42
*** holser has quit IRC10:43
*** derekh has joined #oooq10:44
soniya29|roverbhagyashris|ruck, I am stepping out for 1 hr10:45
bhagyashris|ruckzbr, hey sorry quite busy with rr i will look into it thanks !10:58
bhagyashris|rucksoniya29|rover, ok10:59
zbrapparently nobody reviewed https://review.opendev.org/c/openstack/tripleo-common/+/767754 -- which is essential for ruck/rovering buildah issues11:01
zbrthat change also has a backport to train11:02
bhagyashris|ruckzbr, is there any bug reported for ^11:02
bhagyashris|ruckohh ok11:02
bhagyashris|ruckhttps://bugs.launchpad.net/tripleo/+bug/190827611:02
openstackLaunchpad bug 1908276 in tripleo "tracker: buildah returns 125 error in train/ussuri container builds" [Critical,Fix released]11:02
zbrnot directly, that enables logging of retries made, so when something fails11:02
zbrcurrently we have no proof that any retry happened or not11:03
zbrbecause we did not had logging enabled on thenacity.retry11:03
zbrthat adds a line for each retry being made, making possible to identify how often it happens.11:04
zbrif things are ok from start, nothing is logged.11:04
bhagyashris|ruckzbr, ack,11:05
derekh0011:09
*** dtantsur|afk is now known as dtantsur11:28
*** ysandeep is now known as ysandeep|afk11:30
*** udesale_ has joined #oooq11:36
*** udesale has quit IRC11:39
mariosbiab12:19
*** marios has quit IRC12:19
*** rfolco has joined #oooq12:24
* bhagyashris|ruck brb12:38
* pojadhav brb12:58
*** rlandy has joined #oooq13:08
*** Tengu has quit IRC13:09
weshay|ruck0/13:16
weshay|ruckzbr++ https://review.rdoproject.org/r/#/c/31400/13:17
weshay|rucknice13:17
zbrweshay|ruck: that is one of the things I was referring in retro meeting.13:19
zbris ok to pin *test* tests if we do it like this, and we can easily bump them with "tox -e deps".13:20
zbrhaving to manually find a mix of deps that work, is boring. pip-compile does a decent job.13:20
zbralso adds comments from where each one comes from13:20
*** marios has joined #oooq13:23
weshay|ruckbhagyashris|ruck, soniya29|rover probably makes sense to add a voting / gating tripleo-ci-build-containers-ubi-8 job back to master for tripleo-common and python-tripleoclient. Apparently tripleo-ci-centos-8-content-provider was not sufficient in catching https://bugs.launchpad.net/tripleo/+bug/190910513:31
openstackLaunchpad bug 1909105 in tripleo "ERROR openstack Stderr: 'level=debug msg="Pull Policy for pull [PullIfNewer]"\nerror building at STEP "RUN ln -s /usr/share/openstack-tripleo-common/healthcheck/swift-account failing on build-containers-ubi-8-push master" [Critical,Triaged]13:31
weshay|ruckprobably due to the repos13:31
*** Tengu has joined #oooq13:31
bhagyashris|ruckweshay|ruck, ack13:32
weshay|ruckcontent-provider uses tq/config/release and tripleo-ci-build-containers-ubi-8 uses tripleo-repos.. an inconsistentcy I hope to address early next year13:32
weshay|ruckbhagyashris|ruck, thanks for catching that! very well done13:32
bhagyashris|ruckweshay|ruck, :)13:32
bhagyashris|ruckweshay|ruck, do we need to merge this https://review.opendev.org/c/openstack/tripleo-common/+/76826413:33
weshay|ruckya.. I'm doing a little homework on it13:33
bhagyashris|ruckweshay|ruck, ack13:34
weshay|ruckbhagyashris|ruck, ya.. we'll need to merge it13:36
bhagyashris|ruckweshay|ruck, ok , i keep DNM tag as is so will need to remove that one13:36
weshay|ruckmarios, you here here today? https://review.opendev.org/c/openstack/tripleo-common/+/76826413:36
weshay|ruckbhagyashris|ruck, I nuked it13:36
bhagyashris|ruckweshay|ruck, thanks !13:37
weshay|ruckbhagyashris|ruck, soniya29|rover getting a job up asap.. so we can prevent breaking changes to the container build is a ++13:37
mariosweshay|ruck: o/13:37
weshay|ruckI think we made a mistake in assuming content providers would protect us13:37
weshay|ruckhowdy!13:38
bhagyashris|ruckok13:39
mariosweshay|ruck: looks like it has enough votes now sorry was dealing with sthing on the phone 15:36 < weshay|ruck> marios, you here here today? https://review.opendev.org/c/openstack/tripleo-common/+/76826413:43
weshay|ruckmarios, no worries.. /me was just hunting for cores13:45
bhagyashris|ruckweshay|ruck, marios hi i revert the change https://review.opendev.org/c/openstack/python-tripleoclient/+/761862 to get back  tripleo-ci-build-containers-ubi-8 job as voting here the https://review.opendev.org/c/openstack/python-tripleoclient/+/76826813:51
weshay|rucksshnaidm, so what should we do w/ the pin's there on container tools... it's working on rhel8 vs. 2.0 for train and ussuri now...13:51
weshay|rucksshnaidm, perhaps.. leave that patch as is.. perhaps change the vars in that patch and tq?13:52
weshay|ruckhard to say13:52
bhagyashris|ruckmarios, weshay|ruck, https://review.opendev.org/c/openstack/tripleo-common/+/768269 and this one for tripleo-common13:53
sshnaidmweshay|ruck, it works for train and ussuri - for which c.tools version?13:53
weshay|ruckbhagyashris|ruck, so.. would it be better to add tripleo-ci-build-containers-ubi-8  to a template in tripleo-ci.. so we have it more centralized?  not sure13:53
weshay|rucksshnaidm, afaict.. rhel813:53
weshay|rucklast I checked13:53
weshay|ruckbhagyashris|ruck, I was going to say.. let's see what marios thinks.. but he's +213:54
weshay|ruckbhagyashris|ruck, k.. so let's merge the two .. and readdress this gap in january13:54
* weshay|ruck takes a note13:54
sshnaidmweshay|ruck, so we don't need anymore?13:55
sshnaidmthis patch13:55
*** ysandeep|afk is now known as ysandeep13:55
weshay|rucksshnaidm, ya.. I would say we don't NEED IT, we still may WANT it.. imho aligning container tools between what we have in tq and the container-build role in toci is critical13:56
sshnaidmweshay|ruck, mm.. so let's remove from tq as well?13:57
weshay|rucksshnaidm, ya.. so imho.. aligning the modules is MOST important, having a way to pin both is also a very good idea13:58
weshay|ruckeven if we're not actually pinning at this moment13:59
weshay|ruckdoes that ring tru to you?13:59
sshnaidmweshay|ruck, I'm not sure if we need pinning though if it works, do we expect another problems with container tools?14:00
weshay|rucksshnaidm, lolz.. yes.. given history... rlandy ^ I would expect future issues w/ container-tools14:01
sshnaidmweshay|ruck, maybe we can make some general pinning mechanism instead of hacks14:01
rlandyweshay|ruck: sshnaidm: we need to settle on what version of container tools runs where14:02
rlandyonly ever pin in one place14:02
rlandyotherwise we will trip eachother up14:02
weshay|rucksshnaidm, rlandy well.. yes.. but I think that may be part of the tripleo-repos work for next year14:02
rlandyweshay|ruck: fine as long as its the one agreed on place14:02
weshay|ruckit's a fairly large chunk of work afaict14:02
rlandyyep14:03
rlandystream is rhel814:03
sshnaidmweshay|ruck, why can't we get just appropriate version from rdo?14:03
rlandysshnaidm: based of os14:03
rlandyoff14:03
weshay|ruckrlandy, sshnaidm for now.. I would ack.. changing tq ussuri/train to rhel8 container-tools and removing the vars for train/ussuri in https://review.opendev.org/c/openstack/tripleo-ci/+/767918/14:03
weshay|rucksorin has some nice patches on retries for container builds14:04
rlandyweshay|ruck: I would get the pins out of the release files first14:04
*** PagliaccisCloud has joined #oooq14:04
weshay|ruckand rfolco is getting us a nice report14:04
weshay|ruckrlandy, ya.. to see IF that works well enough14:04
weshay|ruckit's not just that it works once.. it's also that it works and passes at a high pass rate.. > 90%14:04
rlandyweshay|ruck: tbh, I am -1 on https://review.opendev.org/c/openstack/tripleo-ci/+/767918/14:05
weshay|ruckwe have to be careful w/ these content-providers14:05
rlandyset that in the job14:05
weshay|ruckrlandy, zuul vars?14:05
rlandyalso - not clear who is taking this - ci or the df14:05
weshay|ruckwhat do you mean .. taking it?14:06
rlandydigging through all the places and removing the pins14:06
weshay|ruckrlandy, that's something for next year14:06
weshay|ruckno one should be making BIG changes atm14:07
weshay|ruckwe're just keeping the lights on until jan14:07
rlandyyes well +1 on no big change before shutdown14:07
rlandyweshay|ruck: do we need https://review.opendev.org/c/openstack/tripleo-ci/+/767918/ to make the jobs pass?14:07
weshay|ruckrlandy, centralizing the pins is something we'll have to coordinate w/ df14:07
rlandycorrect14:07
rlandyare we sure we want to lock to container tools 2.014:08
weshay|ruckrlandy, sshnaidm I thought we were going to have to do it... to get train and ussuri back online.. but Alex had two patches that ended up fixing it.. he didn't know why :)14:08
weshay|ruckrlandy, compare it to tq config14:08
weshay|ruckrlandy, for ussuri and train14:08
weshay|ruckrlandy, note that content-providers use tq release files14:08
rlandyweshay|ruck: do you see how the rhel module are set on the base image14:08
rlandywe should do that for centos14:09
weshay|ruckI don't see how a zuul var > then a var in containers-build14:09
weshay|ruckI see it as equiv14:09
weshay|ruckrlandy, well...14:09
rlandyone place14:09
weshay|ruckI guess I'll take that back14:09
rlandywe have it in zuul vars already14:10
rlandythat is all14:10
weshay|ruckrlandy, well.. right.. I would argue that zuul is a bad place to centralize vars14:10
weshay|ruckbut.. it does make dep jobs easier.. if the pin is done in zuul14:10
weshay|ruckso I'll give the edge there to zuul vars14:10
weshay|ruckrlandy, but in no way.. do I think moving forward.. zuul is the right place to centralize these kinds of pins14:10
rlandyweshay|ruck: iiuc, we don;t need any emergency pin14:10
weshay|ruckzuul can read a var we set elsewhere14:10
weshay|ruckrlandy, we do in fact14:11
weshay|ruckrlandy, because atm... container-builds the ubi-8 jobs and content-providers are pinning at two different levels14:11
rlandy<weshay|ruck> rlandy, sshnaidm I thought we were going to have to do it... to get train and ussuri back online.. but Alex had two patches that ended up fixing it.. he didn't know why :)14:11
weshay|ruckrlandy, We need to ALIGN first...14:11
rlandy^^ so we still need your patch with Alex's change?14:11
* rlandy is confused now14:12
weshay|ruckrlandy, We need to ALIGN first...14:12
weshay|ruckrlandy, We need to ALIGN first...14:12
weshay|ruckrlandy, check the release files14:12
weshay|ruckhttps://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/release/tripleo-ci/CentOS-8/train.yml#L192-L19314:13
weshay|ruckthat is NOT what the ubi-8 container build job is doing14:13
weshay|ruckand that's a problem14:13
rlandyunderstood14:13
*** PagliaccisCloud has quit IRC14:13
rlandyweshay|ruck: wrt aligning, I would remove the release file pin rather than pin the container-builds job14:15
rlandyunless you need to pin back to 2.014:15
rlandywe should be on rhel814:15
rlandy8.3 is rhel814:15
weshay|ruckrlandy, right.. agreed.. and a side note... until we fix this the right way..  I feel more comfortable having a way to pin container-tools in the container-build role.. just in case there is another fire14:18
weshay|ruckrlandy, if we want to do that in zuul.. as you have internally14:18
weshay|ruckplease put up a patch.. for review..14:18
rlandyweshay|ruck: for upstream, I could go with tripleo-ci rather than zuul as df developers are less comfortable with zuul than repo changes14:20
weshay|ruckrlandy, ok.. put something up so we can see how we want to proceed in the interim14:21
rlandyweshay|ruck: if ok by you, let's keep the 1-on-1 slot we have and can confirm how we cant to go forward with this14:22
rlandytoo many people in the mix here14:23
* bhagyashris|ruck dinner brb14:35
weshay|ruckbhagyashris|ruck, soniya29|rover /me canceled the cix call15:20
weshay|rucksorry... I thought Jan had done that15:20
bhagyashris|ruckweshay|ruck, ack. but i see Jan has canceled that in my mail box15:22
weshay|ruckk.. was still on the rhos-dev cal15:22
* weshay|ruck nuked it15:22
bhagyashris|ruckit's canceled for 23 Dec , 28 Dec and 30 Dec15:23
rfolco'buildah images <image_name>' does not work in content provider jobs, but it works fine in build containers job.... any buildah expert around to tell me why ?15:28
rfolcoweshay|ruck, you know ^ ?15:28
rfolcoI don't want to use 'buildah images | grep <image_name>', but I'll have to15:29
rfolcomarios, maybe you know ^?15:29
rfolcozbr, ^15:30
weshay|ruckrfolco, you can use podman to list images15:30
weshay|ruckthe overllap of buildah and podman is strange15:31
rfolcohmm, for any job ? content provider, build containers, downstream?15:31
rfolcoI'll install it in the venv with pip then15:32
rfolcogood idea15:32
weshay|ruckrfolco, could just use the rpm man15:32
weshay|ruckrpm > pip install15:33
weshay|ruckbuildah and podman break all the time15:33
rfolcook, I just wanted to avoid env differences, downstream vs upstream...15:33
weshay|ruckrfolco, use rpms15:33
rfolcoits so weird that buildah worked fine on build containers15:33
rfolcoand works in my local env15:33
weshay|ruckwe can have another difference in how buildah and podman enter the workflow15:33
weshay|ruckrlandy, ^15:33
rfolcobut in content providers it does not find the image15:33
weshay|ruckrfolco, what does not find the image?15:34
* rlandy reads back15:34
rfolcobuildah images <image>15:34
rfolcoyou have to grep to find it15:34
weshay|ruckright.. which is why I'm saying try podman15:34
rfolcobuildah images | grep <image>15:34
weshay|ruckbut don't pip install15:34
rlandyright ... so15:34
rfolcok k15:34
rfolcothanks for the tip15:34
weshay|ruck:)15:35
weshay|ruckthanks for asking :)))))15:35
rlandythere is the os buildah/podman, the one we lay down in pre15:35
rlandyand the one base container specifies15:35
rlandyanyone could mess with you15:35
rfolcolife messes with me15:38
mariosrfolco: reading15:42
rfolcomarios, summary - will use podman to avoid buildah issues in listing images15:42
mariosrfolco: i don't know why but i also havent used that before I always use podman image list15:43
rfolcomarios, cool thx my beloved ptl15:43
rfolcoI should have started beer later15:44
rfolcololz15:44
*** udesale_ has quit IRC15:48
mariosrfolco: :)15:48
mariosrfolco: i mean its beer oclock somewhere right?15:49
mariosit must be!15:49
mariosalmost beer oclock here in fact15:49
rfolco:)15:49
*** skramaja has quit IRC15:49
pojadhavweshay|ruck, rlandy, sshnaidm : please review when free https://review.rdoproject.org/r/#/c/30492/15:55
weshay|ruckk.. thanks pojadhav15:56
pojadhavjumped to next script15:57
*** pojadhav is now known as pojadhav|away15:57
weshay|rucksoniya29|rover, FYI.. the tempest componet in train hasn't promoted in a long time16:04
weshay|rucksoniya29|rover, http://dashboard-ci.tripleo.org/d/mOvYIiOMk/component-pipeline-train?orgId=116:04
weshay|ruckthe rest of the components now appear to be healthy16:04
soniya29|roverweshay|ruck, ack16:04
weshay|rucksoniya29|rover, these failures indicate an issue w/ the latest patches from tempest16:04
weshay|rucksoniya29|rover, in your wheel house.. so please investigate :)16:05
weshay|rucksoniya29|rover, looking at https://logserver.rdoproject.org/openstack-periodic-integration-stable3/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-train/6402c00/logs/undercloud/var/log/tempest/stestr_results.html.gz16:06
weshay|ruckI suspect it's .. /me gets16:06
weshay|rucksoniya29|rover, https://trello.com/c/w6ShXhXW/1789-cixlp1908976tripleociproa-ovsdbappbackendovsidlidlutilsrownotfound-cannot-find-logicalrouter-with-nameneutron-uuid-is-failing-on16:06
weshay|rucksoniya29|rover, probably should read through that card first.. there actually may be nothing to do in tempest itself16:07
soniya29|roverweshay|rover, let me check16:09
weshay|ruckbhagyashris|ruck, soniya29|rover all the component lines.. master, victoria, ussuri look healthy :))  and looking more closely at train.. tempest is either pinned or no updates backported to train https://trunk.rdoproject.org/centos8-train/component/tempest/16:10
weshay|ruckall the pinned dates are the same16:10
bhagyashris|ruckweshay|ruck, ack16:17
soniya29|roverweshay|ruck, ack16:17
weshay|ruckrlandy, centos-8-stream jobs are now visible on the front page :) http://dashboard-ci.tripleo.org/d/Z4vLSmOGk/cockpit?orgId=116:19
rlandyweshay|ruck: woohoo - front page news16:19
rlandyso exciting16:19
*** jmasud has joined #oooq16:19
weshay|ruckTHIS JUST IN .. boop beep boop16:19
weshay|ruckpojadhav|away, +2 on your patch16:19
* rlandy is trying to get compliant ...16:20
rlandyalmost done16:20
*** marios is now known as marios|out16:34
*** ysandeep is now known as ysandeep|away16:43
*** marios|out has quit IRC16:50
*** jmasud has quit IRC17:02
*** ykarel has quit IRC17:10
*** jmasud has joined #oooq17:31
zbrrlandy: please check backports https://review.opendev.org/q/topic:%22buildah-retries%22+(status:open%20OR%20status:merged) -- wes already did.17:35
zbrmaster merged17:35
*** dtantsur is now known as dtantsur|afk17:38
rlandylooking17:59
rlandyzbr: thanks for taking care of that18:00
*** derekh has quit IRC18:00
* rlandy takes for car inspection - brb18:01
*** rlandy is now known as rlandy|brb18:01
*** rlandy|brb is now known as rlandy18:16
*** jmasud has quit IRC18:17
*** jmasud has joined #oooq18:26
*** jmasud has quit IRC18:55
*** jbadiapa has quit IRC19:01
*** jmasud has joined #oooq19:22
*** jmasud has quit IRC19:55
*** jfrancoa has joined #oooq20:19
*** jfrancoa has quit IRC20:26
rlandyweshay|ruck: https://review.rdoproject.org/r/31428 Add c8 branch jobs for c8stream dependency line20:32
rlandy^^ testing out20:32
*** rfolco has quit IRC21:01
*** rfolco has joined #oooq22:41
*** rfolco has quit IRC22:46
*** rlandy has quit IRC23:30
« Tuesday, 2020-12-22 Index Thursday, 2020-12-24 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!