Friday, 2020-12-18

*** josefwells has quit IRC00:58
*** mgoddard has quit IRC01:55
*** ikhan has quit IRC02:48
*** ikhan has joined #zuul02:48
*** hamalq_ has quit IRC02:50
*** ikhan has quit IRC02:53
*** bhavikdbavishi has joined #zuul03:11
*** bhavikdbavishi has quit IRC03:46
*** bhavikdbavishi has joined #zuul03:59
*** mgoddard has joined #zuul04:12
*** bhavikdbavishi has quit IRC04:42
*** wuchunyang has joined #zuul04:46
*** bhavikdbavishi has joined #zuul04:53
*** mgoddard has quit IRC04:53
*** evrardjp has quit IRC05:33
*** evrardjp has joined #zuul05:33
*** wuchunyang has quit IRC05:57
*** bhavikdbavishi has quit IRC06:00
*** jfoufas1 has joined #zuul06:11
*** mgoddard has joined #zuul06:11
*** bhavikdbavishi has joined #zuul06:15
openstackgerrityatin proposed zuul/zuul-jobs master: Rename CentOS8 repo files to CentOS-Linux  https://review.opendev.org/c/zuul/zuul-jobs/+/76766806:43
*** vishalmanchanda has joined #zuul06:48
*** mgoddard has quit IRC06:53
*** bhavikdbavishi has quit IRC06:56
*** piotrowskim has joined #zuul07:08
openstackgerritDinesh Garg proposed zuul/zuul-jobs master: Allow customization of pip and helm binary  https://review.opendev.org/c/zuul/zuul-jobs/+/76735407:22
openstackgerritDinesh Garg proposed zuul/zuul-jobs master: Allow customization of pip and helm charts  https://review.opendev.org/c/zuul/zuul-jobs/+/76735407:35
*** bhavikdbavishi has joined #zuul07:45
zbr|roverdid anyone found a way to force chrome to open build failure links from gerrit in the same tab instead of creating a new one?07:46
zbr|roveri tried two extensions so far and none was able to make a simple click open in the same page, even if they were supposed to override the target behavior.07:47
*** mgoddard has joined #zuul08:05
*** jcapitao has joined #zuul08:11
*** jpena|off is now known as jpena08:17
*** rpittau|afk is now known as rpittau08:30
*** hashar has joined #zuul08:50
*** mgoddard has quit IRC08:53
openstackgerrityatin proposed zuul/zuul-jobs master: Rename CentOS8 repo files to CentOS-Linux  https://review.opendev.org/c/zuul/zuul-jobs/+/76766808:53
*** smyers has quit IRC08:56
*** smyers has joined #zuul09:00
*** mgoddard has joined #zuul09:05
openstackgerritTobias Henkel proposed zuul/nodepool master: Bump openshift dep  https://review.opendev.org/c/zuul/nodepool/+/76587309:18
*** nils has joined #zuul09:24
mhufelixedel, I'm probably late to ask this, but what was the reason to have different stores depending on contexts (prod, dev, test)?09:49
mhuin zuul-webui I mean09:49
avassIs anyone caching docker images in dib and has a good solution for that?10:07
*** bhavikdbavishi has quit IRC10:14
openstackgerritClément Mondion proposed zuul/zuul master: [api][cors] Access-Control-Allow-Origin * for all routes  https://review.opendev.org/c/zuul/zuul/+/76769110:18
openstackgerritTobias Henkel proposed zuul/nodepool master: Bump openshift dep  https://review.opendev.org/c/zuul/nodepool/+/76587310:31
openstackgerritTobias Henkel proposed zuul/nodepool master: Replace call to deprecated Thread.isAlive  https://review.opendev.org/c/zuul/nodepool/+/76675711:20
openstackgerritTobias Henkel proposed zuul/nodepool master: Bump openshift dep  https://review.opendev.org/c/zuul/nodepool/+/76587311:20
*** jcapitao has quit IRC11:26
*** hashar is now known as hasharLunch11:27
*** bhavikdbavishi has joined #zuul11:51
*** bhavikdbavishi has quit IRC12:04
openstackgerritTobias Henkel proposed zuul/nodepool master: Bump openshift dep  https://review.opendev.org/c/zuul/nodepool/+/76587312:09
tobiashavass: we do, but it's a really ugly hack12:11
tobiashavass: it depends, are you using docker or podman?12:12
tobiashpodman would be much easier12:12
tobiashI can try to push up an element for the docker use case as poc12:13
avassdocker but I'm gonna check if the tools that uses docker can use podman instead12:13
*** bhavikdbavishi has joined #zuul12:13
avassAll I can think of is ugly hacks or importing image archives after boot12:13
tobiashavass: I'll decouple that from our environment specific cruft and upload an element to dib12:15
avassthanks :)12:15
avassit would be nice if docker supported doing things without being completely dependent on it's daemon12:16
tobiashfor that you need podman ;)12:17
avassyeah.... :)12:17
tobiashremote: https://review.opendev.org/c/openstack/diskimage-builder/+/767706 POC: Pre-pull docker images12:20
tobiashavass: ^12:20
tobiashit depends on pre-installed docker and bwrap into the image12:20
tobiashtldr, it runs the into the image installed docker with downstripped options and inside bwrap to do the pulls12:21
tobiashit also works within docker btw12:21
tobiashso in our deployment it's docker in bwrap in docker12:21
avasswe're running nodepool-builder in k8s currently but I'm starting to feel like we need to run that outside of the cluster12:22
*** hasharLunch is now known as hashar12:22
tobiashwe as well so that element is targeted to work within k8s12:22
avassah good :)12:22
tobiashdocs are missing so if you have questions about the vars ask me today :D12:23
tobiashit btw also supports authenticated custom registries12:24
avassauthenticated private registries are exactly what we need :D12:24
avassI wanted a pull through cache but that's apparently not supported for private registries12:24
avassand we got a way too slow network connection to a registry containing a 6.6Gb image. takes like 10min to just pull that12:25
tobiashyeah I know that problem, we at times had 100 parallel jobs that pulled a 12gb image at the same time12:27
tobiashyou can imagine how that ended12:27
avassthat's what I'm trying to avoid :)12:28
avassprobably won't have time to really look at in until after new year. Last day before vacation today. but it's good to know there's a solution at least :)12:30
*** rfolco has joined #zuul12:37
avasstobiash: so what are the requirements for running that in a kubernetes pod?12:51
tobiashavass: the nodepool-builder pod must be privileged12:52
avass+ MKNOD like we got already I guess12:52
tobiashand docker and brap must be installed by different elements12:52
tobiashyeah, mknod should be part of privileged I guess12:53
avassit wasn't12:53
*** tosky has joined #zuul12:58
*** jpena is now known as jpena|lunch13:00
openstackgerritJonas Sticha proposed zuul/nodepool master: WIP: aws: add support for uploading diskimages  https://review.opendev.org/c/zuul/nodepool/+/73521713:09
*** bhavikdbavishi has quit IRC13:16
*** bhavikdbavishi has joined #zuul13:17
openstackgerritClément Mondion proposed zuul/zuul master: [api][cors] Access-Control-Allow-Origin * for all routes  https://review.opendev.org/c/zuul/zuul/+/76769113:26
*** sanjayu_ has quit IRC13:29
openstackgerritTobias Henkel proposed zuul/nodepool master: Bump openshift dep  https://review.opendev.org/c/zuul/nodepool/+/76587313:34
openstackgerritClément Mondion proposed zuul/zuul master: [api][cors] Access-Control-Allow-Origin * for all routes  https://review.opendev.org/c/zuul/zuul/+/76769113:39
*** bhavikdbavishi has quit IRC13:54
avasstobiash: I think podman actually works so might stick with that. could still be good to have docker as an alternative13:55
openstackgerritClément Mondion proposed zuul/zuul master: [api][cors] Access-Control-Allow-Origin * for all routes  https://review.opendev.org/c/zuul/zuul/+/76769113:59
*** jpena|lunch is now known as jpena14:00
openstackgerritTobias Henkel proposed zuul/nodepool master: Bump openshift dep  https://review.opendev.org/c/zuul/nodepool/+/76587314:02
openstackgerritClément Mondion proposed zuul/zuul master: [api][cors] Access-Control-Allow-Origin * for all routes  https://review.opendev.org/c/zuul/zuul/+/76769114:11
*** bhavikdbavishi has joined #zuul14:34
*** iurygregory has joined #zuul14:38
openstackgerritTobias Henkel proposed zuul/nodepool master: Bump openshift dep  https://review.opendev.org/c/zuul/nodepool/+/76587315:00
*** ykarel has joined #zuul15:12
openstackgerritMatthieu Huin proposed zuul/zuul master: web UI: user login with OpenID Connect  https://review.opendev.org/c/zuul/zuul/+/73408215:12
iceyis it possible to set the timeout for a cleanup-job? it looks like they just timeout at 15 minutes regardless?15:12
fungiykarel has brought to our attention that centos has reorganized their mirror structure, and the paths hard-coded in the configure-mirrors role no longer work: https://review.opendev.org/76766815:14
ykarelThanks fungi for raising it here15:14
fungiquick review of that would be appreciated15:15
tobiashdo we need to keep backwards compatibility here?15:15
fungithat's a great question. i wonder if there are organizations maintaining centos 8.2 mirrors who won't update them to 8.3 right away15:16
openstackgerritMatthieu Huin proposed zuul/zuul master: Add authentication-realm attribute to tenants  https://review.opendev.org/c/zuul/zuul/+/73558615:16
ykareltobiash, you mean people still using centos8.2 images(and don't run dnf update) instead of centos8.3 images15:20
ykarelatleast the nodepool images are updated to 8.3 and have new repo structure15:22
corvusicey: do you mean cleanup playbooks?  timeout is currently hardcoded to 5 minutes.15:31
iceyyeah, cleanup playbooks15:31
iceycorvus: I have a task that I want to run as a cleanup, but it might take a while (10-15 minutes is possible)15:32
corvusicey: it should be a straightforward change to zuul to make that configurable15:32
iceycorvus: it should be OK to put it into a post-run as well, but not quite as reliable, as I understand it?15:33
iceyie: won't get run on cancellation?15:33
corvusicey: that's right15:35
zbr|roverimho 767668 is good to go.15:36
*** rpittau is now known as rpittau|afk15:38
avasstobiash: wha'ts your podman solution then if I may ask? :)15:45
tobiashavass: we don't (yet) use podman ;)15:46
avassaww15:46
avassmaybe I can come up with something then. it should probably be easier to fix than docker15:47
*** hashar has quit IRC15:53
*** jfoufas1 has quit IRC16:00
mordredavass: skopeo should be able to download images and save them locally. I don't know if it can save them in a local docker image cache without a daemon running... but - you could save them in the image in a native oci format, and then run a script on node boot once docker is running to copy the cached images using skopeo from the oci format into the docker image cache. involves a runtime step - but could avoid the need to have the docker daemon running16:04
mordredduring image build. you could maybe write a systemd unit to do the copy and tag it to run after docker is running16:04
avassmordred: yeah that's my backup strategy right now. but since podman seems to be working great with that tool I'm sticking with that for now16:05
tobiashmordred, avass: when using docker skopeo is not really an option since it needs the docker daemon as well for docker images (at least a year ago when I looked). However I think using skopeo can be the right choice for caching images for podman since I think the filesystem layout of skopeo and podman are the same16:10
openstackgerritMerged zuul/zuul-operator master: Replace Fedora 31 with Fedora 32 in zuul-operator-functional  https://review.opendev.org/c/zuul/zuul-operator/+/76747316:10
tobiash(and copying many images during boot time is probably not really scalable)16:11
mordredjust get faster disks? :)16:11
tobiashit's still preferable to have the images where they should be at boot time :)16:11
mordredbut yeah - ultimately migrating the workloads to not-docker and using podman and/or crio end-to-end opens up many more flexible doors16:12
tobiashespecially with the crazy large images16:12
zbr|roveris there a way to request an extra role to be installed on a standard job without defining a new one? example: adding ensure-yarn to a tox-linters job for example?16:12
avasstobiash: yep exactly that16:12
zbr|roverquite often I face the need to add some ensure roles to existing jobs.16:12
tobiashzbr|rover: you still can add pre-playbooks in the project-pipeline16:13
tobiashzbr|rover: like https://opendev.org/zuul/zuul/src/branch/master/.zuul.yaml#L223 which is just an implicit new job16:14
zbr|rovertobiash: i tried adding roles: to it but it does not pick them from there.16:14
avassalso downloading 30Gb images in a tmpfs was a bad idea16:15
tobiashzbr|rover: you need to add a pre-playbook there that calls the role16:15
zbr|rovertobiash: that it bit sad as it creates a new file to maintain.16:16
zbr|rovermaybe we should add a roles variable where you declare which roles to be imported and allow zuul to loop over them.16:17
corvuszbr|rover: we made a design decision for zuulv3 that we would not re-implement ansible in zuul; so the execution framework is all playbooks.16:18
avasszbr|rover:  pre-playbook is sort of how you do that. otherwise you'd just add another way to tell zuul in which step or before which playbook a role should be executed16:18
tobiashwell you already can do that when you create a pre playbook that does that in your job16:18
tobiashthere is no need to adapt zuul for this16:18
corvuszbr|rover: having said that, if you really wanted to do something like that, you could have a (base?) job with a pre-playbook that included roles based on variables16:18
corvusthat might be what tobiash is getting at16:18
tobiashyes16:18
zbr|roverbut I want to add role foo to job 1 and role bar to job 2, i would have to create two extra playbooks just for that.16:19
corvuspersonally, i don't think we'd want to structure jobs in zuul-jobs like that though16:19
corvuszbr|rover: that's correct16:19
avassthat could probably lead to nasty errors16:20
corvusavass: i agree16:20
corvusavass: it could be very confusing for folks because you're building an ansible system on top of ansible16:20
tobiashzbr|rover: you could have a pre playbook like this in your base job: http://paste.openstack.org/show/801160/16:20
zbr|rovertobiash: yep, i was already considering it :D16:21
tobiashif it's in your base job it would not even have to be part of a zuul-jobs job16:21
corvusto further clarify, i don't think we would do this in opendev base jobs either.  but if zbr|rover wanted to do something like that for, say, tripleo jobs, that would work.16:22
corvus(i just want to be clear about what one *can* do in zuul vs what i think we would be willing to do in specific installations of zuul)16:23
tobiash++16:23
zbr|roverwhat i was trying to underline is that at this moment zuul does not allow you to alter https://zuul-ci.org/docs/zuul/reference/job_def.html?highlight=job#attr-job.roles when you add the job to your project pipeline. It just ignores the presence of "roles:" under the job name. Other fields can be overriden like voting, vars,...16:24
tobiashzbr|rover: roles only adds the roles into the namespace of the job so they can be called from playbooks, are you saying this is not working?16:26
tobiashzuul-maint: a review on https://review.opendev.org/c/zuul/nodepool/+/766757 would be appreciated to fix nodepool with python 3.916:29
zbr|rovertobiash: right... i guess I should end the week, i am starting to get confused.16:30
openstackgerritMerged zuul/zuul-jobs master: Rename CentOS8 repo files to CentOS-Linux  https://review.opendev.org/c/zuul/zuul-jobs/+/76766816:31
*** hashar has joined #zuul16:31
zbr|rovertobiash: look at https://review.opendev.org/c/zuul/zuul/+/766460/ and its dependency.16:32
fungido we know why nodepool-functional-openstack-src is taking so long to run?16:33
zbr|rovermainly i need to assure that zuul-tox-docs has yarn installed, so I added it to the job definition and forgot to call it.16:33
*** jfoufas1 has joined #zuul16:33
tobiashzbr|rover: adding the role is not needed there since it's part of zuul-jobs which is already added in a parent job. Instead just add a pre playbook that calls this role.16:36
tobiashfungi: I don't know, but likely unrelated16:37
openstackgerritSorin Sbârnea proposed zuul/project-config master: Add yarn to zuul-tox-docs  https://review.opendev.org/c/zuul/project-config/+/76712216:37
tobiashfungi: typically it's much faster so might be a slow node or problems in the mirror infrastructure16:38
fungitobiash: maybe. it ran quickly two days ago, but twice in a row today it timed out16:39
openstackgerritSorin Sbârnea proposed zuul/zuul master: Document tox environments  https://review.opendev.org/c/zuul/zuul/+/76646016:40
fungitobiash: could just be coincidence, but i'll see if i can spot any commonality between the two builds16:40
tobiashfungi: where is the second? I see only one timeout in the recent builds: https://zuul.opendev.org/t/zuul/builds?job_name=nodepool-functional-openstack-src&project=zuul/nodepool16:40
fungitobiash: oh! you're right, the previous one was nodepool-functional-openstack16:41
funginot -src16:41
tobiashright, so two functional openstack: https://zuul.opendev.org/t/zuul/builds?job_name=nodepool-functional-openstack-src&job_name=nodepool-functional-openstack16:42
tobiashprobably regardless of src or not16:42
fungii agree, likely whatever's slow in some environments is the same in both versions of the job16:42
tobiashfungi: there is a huge difference in devstack setup16:44
tobiashhttp://paste.openstack.org/show/801161/16:45
tobiashfast one vs timeouted16:45
tobiashso setting up devstack took more than an hour16:45
tobiashapt-get was ten times slower16:46
*** piotrowskim has quit IRC16:46
tobiashboth timeouts were in vexxhost-ca-ymq-116:48
zbr|rovercorvus: let me guess: yarn is not installed as being in protected repo? so we cannot test the change.16:54
corvuszbr|rover: correct16:54
zbr|roverhttps://ea507024840f9c0decff-897daedd12d6711e791474c4dbb73e9e.ssl.cf1.rackcdn.com/766460/10/check/zuul-tox-docs/fec4ea3/job-output.txt failed without any attempt to run ensure-yarn16:54
corvuszbr|rover: you may want to override that in just the zuul repo16:54
corvusit's a more targeted change anyway, since no other docs job needs it16:55
zbr|roversadly buntu does not have yarn right available, so bindep cannot save me.16:56
corvusi'm comfortable either way though.  i +2d 767122 with comment.16:56
*** rlandy has joined #zuul16:58
openstackgerritSorin Sbârnea proposed zuul/zuul master: Document tox environments  https://review.opendev.org/c/zuul/zuul/+/76646017:01
fungii can see both sides of it. it's nice to have a consistent docs job across the zuul projects, but unless we expect nodepool or zuul-client et cetera to have javascript content at some point it's a bit of additional bloat in their docs builds to install a tool which won't be run17:02
zbr|roveri wonder how big are the images used by github actions, as they took a very different path than us: try to fit most build tools on images, which make me believe they are quite fat.17:04
zbr|roverbut they avoid having a "pre" stage that is compute and network intensive.17:05
corvuszbr|rover: (us==opendev in that, not zuul).17:05
corvuszbr|rover: but yes, opendev used to have fat images, then decided to take a different path and has moved to [mostly] slim images.17:06
zbr|rovercorvus: and we know which approach is better?17:06
corvuszbr|rover: no right answer generally, but so far, i think in opendev's case the slim images have been an improvement for that environment.17:07
zbr|roverclearly for system testing we cannot use fat images. but for unittesting17:07
*** jpena is now known as jpena|off17:16
*** hamalq has joined #zuul17:19
*** vishalmanchanda has quit IRC17:19
zbr|rovercorvus: apparenly installing yarn does not install it in PATH, https://6f670868c142d4e8c5b3-0595cf8ce23c3a62b4c5e95f9113455a.ssl.cf5.rackcdn.com/766460/11/check/zuul-tox-docs/9f85aa8/job-output.txt17:20
*** hamalq_ has joined #zuul17:21
zbr|roverbit weird as the task uses default path, but the tox job fails to find it.17:22
zbr|rovermaybe PATH is not passed in tox.ini...17:22
openstackgerritSorin Sbârnea proposed zuul/zuul master: Document tox environments  https://review.opendev.org/c/zuul/zuul/+/76646017:24
*** hamalq has quit IRC17:24
zbr|roverwhat is weird is why i was able to run it locally w/o having to edit PATH17:25
tristanCzbr|rover: we do use container images with the tool pre-installed, effectively skipping zuul-jobs pre phase17:30
openstackgerritMatthieu Huin proposed zuul/zuul master: web UI: allow a privileged user to dequeue a change  https://review.opendev.org/c/zuul/zuul/+/73485017:31
tristanCeven using a curated node_modules bundle to enable javascript zuul jobs to run under 2 minutes17:33
corvusand that's why all the ensure- zuul-jobs roles are designed to skip their work if tools are pre-installed17:33
tristanCcorvus: that's not the case for yarn at least, it uses `state: latests`17:37
tristanCand also install repositories unconditionnaly17:37
corvusthat's a bug; that's how they're supposed to work17:38
corvusthat's why we named them 'ensure' vs 'install'17:39
openstackgerritMerged zuul/nodepool master: Replace call to deprecated Thread.isAlive  https://review.opendev.org/c/zuul/nodepool/+/76675717:40
zbr|rovercorvus: still broken, i don't know why and i need to leave. feel free to look, if you have time.17:48
*** jfoufas1 has quit IRC17:48
zbr|roversphinx output is not very useful here17:48
*** bhavikdbavishi has quit IRC18:01
*** ykarel has quit IRC18:28
pabelangerI'd appreciate if I could get a +A on https://review.opendev.org/c/zuul/zuul-jobs/+/76736118:29
pabelangerfor a friday review18:30
*** hashar is now known as hasharDinner18:31
pabelangeravass: tobiash: so, I'm going to be playing with podman cache more over the break.  On of our teams, doesn't like we rebuild all the layers each time a new PR is done.  This is bascially podman build . --no-cache.  So, I want to see, if there is a way we can share the cache, between PRs when possible for this project. I'm un sure if podman pull foo.bar is more then enough, to then do podman build .18:31
pabelanger-t foo.bar and the cache used, if nothing changed18:31
pabelangeravass: tobiash: right now, this team is using static nodes, and I want to get them off that.  But speed of builds is the block right now18:32
*** armstrongs has joined #zuul18:32
openstackgerritMerged zuul/zuul-jobs master: Create tox_package_name for tox role  https://review.opendev.org/c/zuul/zuul-jobs/+/76736118:47
avasspabelanger: maybe https://review.opendev.org/c/zuul/zuul-jobs/+/764808 is a good fit for that18:50
avassthat way you don't rebuild all the layers but you need to download/upload them all the time18:51
pabelangeravass: ah, neat. That could just work18:51
*** hasharDinner is now known as hashar19:02
avasspabelanger: maybe it could be optimized to work with image layers efficiently somehow. because if you just cache, say: .local/share/containers/storage/overlay-images/ that could easily grow over time19:18
openstackgerritMatthieu Huin proposed zuul/zuul master: web UI: user login with OpenID Connect  https://review.opendev.org/c/zuul/zuul/+/73408219:18
openstackgerritMatthieu Huin proposed zuul/zuul master: Add authentication-realm attribute to tenants  https://review.opendev.org/c/zuul/zuul/+/73558619:18
avassand then it wouldn't be very efficient anymore19:19
openstackgerritMatthieu Huin proposed zuul/zuul master: web UI: allow a privileged user to dequeue a change  https://review.opendev.org/c/zuul/zuul/+/73485019:25
*** armstrongs has joined #zuul19:33
armstrongsHey with override-checkout option is there an option to configure a fork?19:34
armstrongsLooking to do this on required-projects if possible19:34
fungiarmstrongs: you'll have to be more specific about what you mean by "configure a fork"19:37
fungiif you mean checkout a different project in place of a listed required project, i don't think so. you'd need to add the fork to the projects in the zuul tenant and then reference it explicitly in an altered required-projects list19:38
avassyeah the fork is technically a completely different repo isn't it?19:41
avassso that would be a different remote with the same content and override-checkout only handles refs19:42
*** armstrongs has quit IRC19:47
*** rlandy has quit IRC20:14
*** rfolco has quit IRC20:40
*** rfolco has joined #zuul20:40
*** _erlon_ has quit IRC20:49
*** nils has quit IRC21:30
*** zenkuro has joined #zuul21:43
*** bodgix has left #zuul22:08
*** sduthil has quit IRC22:16
*** hashar has quit IRC22:45
*** dmsimard5 has joined #zuul23:50
*** dmsimard has quit IRC23:52
*** dmsimard5 is now known as dmsimard23:52

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!