| *** dkehn has quit IRC | 01:29 | |
| *** dkehn has joined #zuul | 01:36 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [wip] Add dogpile.cache master to the -src tests https://review.openstack.org/625457 | 01:49 |
|---|---|---|
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [wip] Add dogpile.cache master to the -src tests https://review.openstack.org/625457 | 03:07 |
| *** bjackman has joined #zuul | 03:36 | |
| *** bhavikdbavishi has joined #zuul | 03:41 | |
| *** bjackman has quit IRC | 03:50 | |
| *** chandan_kumar is now known as chandankumar | 04:55 | |
| *** bhavikdbavishi1 has joined #zuul | 05:59 | |
| *** bhavikdbavishi has quit IRC | 06:01 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 06:01 | |
| *** quiquell|off has quit IRC | 06:09 | |
| *** threestrands has quit IRC | 06:21 | |
| *** AJaeger has quit IRC | 07:00 | |
| *** AJaeger has joined #zuul | 07:09 | |
| *** jpena|off is now known as jpena | 07:42 | |
| *** yolanda has joined #zuul | 07:50 | |
| *** pcaruana has joined #zuul | 08:18 | |
| *** jpena is now known as jpena|away | 08:21 | |
| *** fbo has joined #zuul | 08:44 | |
| *** quiquell has joined #zuul | 08:49 | |
| quiquell | Good morning | 08:49 |
| quiquell | Could be beneficial to ignore "non-voting" jobs and start gate jobs before they finish ? | 08:58 |
| tobiash | quiquell: zuul only reports once after all jobs are finished. So if you want to 'ignore' non-voting jobs you can move them into a comment triggered pipeline like check-experimental in openstack land. | 09:08 |
| quiquell | tobiash: ack, ok so change pipelines should do that | 09:09 |
| tobiash | yes, you also could make it trigger automatically but not require the vote of that pipeline for gate | 09:09 |
| quiquell | tobiash: this is nice, voting jobs that run at experimental, don't count ? | 09:11 |
| quiquell | tobiash: like third party ones I suppose | 09:11 |
| tobiash | quiquell: that depends on your pipeline definition, that controls votes of it. And you can let it vote on a non-required label or even just comment without any votes. | 09:12 |
| quiquell | tobiash: btw after adding some mergers to my local zuul It takes just 2,5 minute to startup connecting to git.review.org and rdo :-) | 09:18 |
| *** bjackman has joined #zuul | 09:27 | |
| *** themroc has joined #zuul | 09:28 | |
| *** bhavikdbavishi has quit IRC | 09:36 | |
| *** ssbarnea|rover has joined #zuul | 09:47 | |
| *** jpena|away is now known as jpena | 10:12 | |
| *** electrofelix has joined #zuul | 10:52 | |
| *** bhavikdbavishi has joined #zuul | 11:30 | |
| *** rfolco has joined #zuul | 11:37 | |
| *** dkehn has quit IRC | 11:57 | |
| *** gtema has joined #zuul | 12:07 | |
| *** bhavikdbavishi has quit IRC | 12:09 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/zuul-jobs master: Remove world writable umask from /src folder https://review.openstack.org/625576 | 12:14 |
| *** jpena is now known as jpena|lunch | 12:30 | |
| *** bjackman has quit IRC | 12:31 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Delay Github fileschanges workaround to pipeline processing https://review.openstack.org/625584 | 12:50 |
| *** rlandy has joined #zuul | 12:57 | |
| *** gtema has quit IRC | 12:58 | |
| sean-k-mooney | o/ | 13:00 |
| sean-k-mooney | quick question. i deployed the container form the adming demo docker compose using kubernetes at the weekend | 13:01 |
| sean-k-mooney | i have each of the service deploy but zuul is not picking up chages from gerrit | 13:01 |
| sean-k-mooney | any suggestions on how to debug | 13:01 |
| sean-k-mooney | can i check if zuul is connected to gerrit in some way via the zuul cli? | 13:02 |
| tobiash | sean-k-mooney: do you have the scheduler log? | 13:08 |
| sean-k-mooney | am not to hand but i have access to them yes | 13:10 |
| sean-k-mooney | they were almost empty but thinking about it i assume the container did not default to debug loginin that would probaly be a good place to start | 13:10 |
| openstackgerrit | Quique Llorente proposed openstack/pbrx master: Add tag to container images https://review.openstack.org/625590 | 13:12 |
| sean-k-mooney | i peiced the deploy logic together form the docker compose file by had. since i know relitilly little about kubernets its higly proably i messed something up but i think the container are correct but i could eaisily have missed something. | 13:12 |
| *** bhavikdbavishi has joined #zuul | 13:13 | |
| sean-k-mooney | im buying a new server to use fo teh ci and will likely redeploy in a vm when it arrive so this is not urgent. | 13:14 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Delay Github fileschanges workaround to pipeline processing https://review.openstack.org/625584 | 13:26 |
| *** jpena|lunch is now known as jpena | 13:27 | |
| quiquell | tobiash: Has created this to tag zuul docker images https://review.openstack.org/#/c/625590/ | 13:34 |
| quiquell | pbrx change | 13:35 |
| tobiash | quiquell: I'm no core in pbrx, Shrews, mordred ^ | 13:36 |
| quiquell | tobiash: ack | 13:37 |
| *** bjackman has joined #zuul | 13:38 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/zuul-jobs master: Alwas use pathless docker mirror URI https://review.openstack.org/625596 | 13:43 |
| *** dkehn has joined #zuul | 13:45 | |
| *** bhavikdbavishi has quit IRC | 13:49 | |
| *** quiquell is now known as quiquell|lunch | 13:49 | |
| *** pcaruana has quit IRC | 13:50 | |
| *** bjackman has quit IRC | 13:55 | |
| quiquell|lunch | tobiash: do we have access to the commit id in at zuul job "vars" ? | 13:55 |
| quiquell|lunch | tobiash: like expanding at job var to the commit id ? | 13:56 |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/nodepool master: Switch devstack jobs to Xenial https://review.openstack.org/624855 | 13:57 |
| tobiash | quiquell|lunch: check out any inventory file of a job result: http://logs.openstack.org/84/625584/2/check/tox-py36/82b809e/zuul-info/inventory.yaml | 13:59 |
| tobiash | quiquell|lunch: that contains all vars zuul supplies | 13:59 |
| sean-k-mooney | tobiash: quiquell|lunch is that where the zuul docker images are defiend? | 13:59 |
| sean-k-mooney | i was trying to figure that out but there is no info in the dockerhub site or the docs or the zuul git repo to figure it out | 14:00 |
| tobiash | sean-k-mooney: pbrx is the tool used to build the zuul docker images | 14:00 |
| sean-k-mooney | im guessign it stands for pbr extentions | 14:01 |
| sean-k-mooney | hum so this is how you define packages to be installed https://github.com/openstack-infra/zuul/blob/master/setup.cfg#L48-L58 and it generate a contaier for each console script ... | 14:05 |
| tobiash | yes | 14:05 |
| *** pcaruana has joined #zuul | 14:05 | |
| sean-k-mooney | while that seams conviniant it is rther obscure. it would be nice to call that out in the docs | 14:06 |
| sean-k-mooney | im kind of surprised it does not use bindep.txt | 14:07 |
| *** quiquell|lunch has quit IRC | 14:15 | |
| *** quiquell has joined #zuul | 14:20 | |
| quiquell | toabctl: zuul.build is the commit id ? | 14:21 |
| mordred | quiquell, tobiash: lgtm - Shrews, SpamapS wanna take a look? | 14:22 |
| openstackgerrit | Quique Llorente proposed openstack/pbrx master: Add tag to container images https://review.openstack.org/625590 | 14:22 |
| quiquell | mordred: ^ updated also the job | 14:23 |
| mordred | quiquell: what if we used zuul.tag instead of zuul.build? that way it'll do the right thing in release pipelines, but we won't be pushing tags to dockerhub just for normal builds | 14:26 |
| tobiash | hrm, zuul-quick-start seems to be broken: http://logs.openstack.org/84/625584/2/check/zuul-quick-start/f53553d/job-output.txt.gz#_2018-12-17_13_30_04_409474 | 14:29 |
| fungi | tobiash: https://review.openstack.org/625596 seems to fix it for zuul.openstack.org, but likely needs some discussion still | 14:30 |
| tobiash | fungi: thanks! | 14:31 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Delay Github fileschanges workaround to pipeline processing https://review.openstack.org/625584 | 14:32 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/zuul-jobs master: Remove world writable umask from /src folder https://review.openstack.org/625576 | 14:33 |
| quiquell | mugsie: I don't see zuul.tag http://logs.openstack.org/84/625584/2/check/tox-py36/82b809e/zuul-info/inventory.yaml | 14:34 |
| quiquell | mordred: ^ | 14:34 |
| quiquell | wrong mfoobar | 14:34 |
| mordred | quiquell: :) yah - its only set if a job has been run in a pipeline triggered by a tag. check out http://logs.openstack.org/e1/e1c7f9f254c817f31cfa5a3195f60b3f2c4ae37a/release/propose-update-constraints/55e4a69/zuul-info/inventory.yaml | 14:36 |
| quiquell | mordred: I think we need both | 14:38 |
| quiquell | mordred: not every commit id increase the semver | 14:38 |
| quiquell | mordred: I suppose the pinning granularity is commit id | 14:38 |
| mordred | yah - that's right ... but wouldn't it get messy to push a tag to dockerhub for every commit? | 14:38 |
| quiquell | mordred: how often is semver increased ? | 14:39 |
| mordred | usually every 2 weeks or so | 14:39 |
| quiquell | mordred: ack, what are the reqs ? | 14:40 |
| quiquell | sshnaidm: ^ we are going to pin zuul to semver instead of commit id, the update it every two weeks | 14:40 |
| mordred | tends to be when people say "hey, we should tag a new release" - but also, we update openstack's zuul and run it for a little bit to make sure it's good before we tag | 14:40 |
| mordred | quiquell: well - I'm not sure tagging commits is a bad idea ... totally asking | 14:41 |
| mordred | I mean, we should definitely tag tags when they exist | 14:41 |
| mordred | but I could see a benefit in tagging git shas - I just don't know if there is a downside | 14:41 |
| quiquell | mordred: https://container-solutions.com/tagging-docker-images-the-right-way/ | 14:41 |
| quiquell | reading stuff | 14:41 |
| tobiash | mordred, frickler, fungi: I think we should rethink the docker mirror thing in the zuul-jobs/install-docker role | 14:42 |
| quiquell | look like having commit id and semver is the way to go | 14:42 |
| tobiash | mordred: do you know if there are more users of the install-docker role that use the mirror thing? | 14:42 |
| quiquell | mordred: tagging with commit id, can render on space problemas at dockerhub ? | 14:42 |
| tobiash | mordred: that still looks quite openstack specific to me: https://review.openstack.org/#/c/625596/1/roles/install-docker/tasks/mirror.yaml | 14:43 |
| mordred | tobiash: yeah- I agree, that's pretty openstack specific looking | 14:43 |
| mordred | tobiash: I'm not sure if anyone else is using it - but seeing how it's so openstack specific, I doubt it :) | 14:44 |
| tobiash | mordred: I think on the long run we should change that to take the mirror url instead (and a flag if it's insecure or not) | 14:44 |
| mordred | tobiash: ++ | 14:44 |
| tobiash | mordred: yes, so with that reasoning I'd be +2 on the change to unblock things | 14:44 |
| frickler | this is where our mirror setup for this is defined http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/mirror.vhost.erb#n387 | 14:44 |
| frickler | but I agree that this shouldn't belong into the generic job | 14:45 |
| mordred | tobiash: that mirror_fqdn is being set if zuul_site_mirror_fqdn is set - but given this whole running-on-a-port setup, I think basing it off of zuul_site_mirror_fqdn may not really work here | 14:47 |
| mordred | maybe a new variable, like zuul_site_docker_mirror_url that the role consumes for defaults - so it's easy for an admin to setup a mirror? | 14:48 |
| tobiash | mordred: ++ | 14:48 |
| mordred | and we can get the "construct url with port from fqdn" logic out | 14:48 |
| tobiash | That makes totally sense | 14:49 |
| mordred | frickler: that "use_upstream_docker" behavior difference with the mirror is actually about "is docker old or new" | 14:49 |
| fungi | so should we merge 625596 without any warning as a short-term fix and then refactor the role to be more generic longer-term? | 14:50 |
| mordred | frickler: I'm not sure if we should rework the logic to expose the old mirror url as a choice? or maybe just roll with it for now and only support newer mirrors if we're seeing newer docker everywhere now? | 14:50 |
| mordred | fungi: yeah - I think it's fine as a fix for now and just has brought to mind we can improve it | 14:51 |
| fungi | i just didn't want to see us merge a potentially breaking change to zuul-jobs without some discussion outside of #openstack-infra | 14:51 |
| mordred | ++ | 14:51 |
| mordred | fungi: I think we're pretty safe from this being used outside of infra given how openstack-specific the mirror setup is in the first place | 14:51 |
| *** toabctl has quit IRC | 14:57 | |
| *** toabctl has joined #zuul | 14:57 | |
| tobiash | mordred: ah now I see, there really was a docker with registry v1 api in use? | 15:00 |
| tobiash | that feels a little bit like stone age ;) | 15:01 |
| mordred | tobiash: yah - and in fact it was in heavy usage by openstack jobs :( | 15:02 |
| tobiash | mordred: ok, so if we still want to support v1 we should switch on the docker version instead of upstream docker or not | 15:02 |
| mordred | I think the docker that was being installed in centos needed it | 15:02 |
| tobiash | ah ok | 15:02 |
| mordred | tobiash: I agree. although I think I'm not sure how to sanely know what version got installed ... so I think I'm fine just defaulting this one to sanity and providing a variable people can set at invocation time if they need it | 15:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Update install-docker to use docker site variable https://review.openstack.org/625617 | 15:03 |
| mordred | tobiash, frickler, fungi: ^^ there's a stab at updating the role to use the site variable defined in the infra project-config patch | 15:03 |
| tobiash | mordred: we should ask the installed docker probabky | 15:03 |
| mordred | tobiash: yah - but that's going to be very distro specific ... not sure the effort it worth it for what should be a shrinking number of jobs | 15:04 |
| mordred | lemme ping the tripleo folks about it though | 15:04 |
| tobiash | mordred: that should do it: docker version --format '{{.Server.Version}}' | 15:08 |
| tobiash | at least for current versions | 15:09 |
| mordred | tobiash: oh - neat! | 15:10 |
| *** themroc has quit IRC | 15:11 | |
| mordred | tobiash: now to figure out which version of docker needs v1 registry | 15:11 |
| openstackgerrit | Quique Llorente proposed openstack/pbrx master: Add tag to container images https://review.openstack.org/625590 | 15:11 |
| tobiash | mordred: registry v2 is supported since 1.6 | 15:12 |
| mordred | I was just reading that it returns v1 style by default for 1.9 - but if 1.6 supports v2, maybe that's the right one | 15:13 |
| tobiash | centos 7 has 1.13.1 currently | 15:14 |
| tobiash | mordred: trusty has 1.6.1 and doesn't support the --format parameter | 15:16 |
| tobiash | mordred: but docker --version still seems to look the same: | 15:16 |
| tobiash | Docker version 1.6.2, build 7c8fca2 | 15:16 |
| tobiash | Docker version 17.09.1-ce, build 19e2cf6 | 15:17 |
| tobiash | that might be parsable | 15:17 |
| mordred | tobiash: well - if trusty is the only thing that has 1.6 | 15:17 |
| mordred | tobiash: then maybe actually just not bothering is better? I don't think we're supporting trusty anymore in infra - right frickler? - and I'm guessing we'd be the most likely people to need to still support build nodes that old | 15:18 |
| tobiash | xenial has Docker version 18.06.1-ce, build e68fc7a | 15:18 |
| frickler | at least for nodes running jobs, trusty should be gone, yes | 15:19 |
| openstackgerrit | Quique Llorente proposed openstack/pbrx master: Add tag to container images https://review.openstack.org/625590 | 15:23 |
| *** themroc has joined #zuul | 15:23 | |
| *** chandankumar is now known as chkumar|out | 15:26 | |
| *** pcaruana has quit IRC | 15:29 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/nodepool master: [wip] Add dogpile.cache master to the -src tests https://review.openstack.org/625457 | 15:42 |
| openstackgerrit | Quique Llorente proposed openstack/pbrx master: Add tag to container images https://review.openstack.org/625590 | 15:42 |
| openstackgerrit | Quique Llorente proposed openstack/pbrx master: Add tag to container images https://review.openstack.org/625590 | 15:43 |
| *** pcaruana has joined #zuul | 15:51 | |
| tobiash | mordred: do you think the dogpile cache problems will break swift upload too? | 15:54 |
| mordred | tobiash: hrm. that's a really good question. they might, since it's a decorator that happens at import time. HOWEVER - we do a null wrap if you don't have per-resource caching configured | 15:55 |
| mordred | tobiash: so it might not | 15:56 |
| tobiash | mordred: ah so this is only a problem if I really enable caching in clouds.yaml? | 15:57 |
| mordred | tobiash: I *think* so - but I'm not 100% sure since it's a code-level issue that happens at decorator time | 15:58 |
| mordred | tobiash: kmalloc has a patch up to fix the dogpile issue though | 15:58 |
| kmalloc | Yes | 15:59 |
| kmalloc | Only happens with caching. The fix is pending another fix though. | 15:59 |
| tobiash | kmalloc: thanks, I'm not using caching on the executors that do swift upload | 16:00 |
| kmalloc | Yeah it should be fine, since we don't do the wrap with real dogpile | 16:01 |
| kmalloc | With caching disabled | 16:01 |
| tobiash | cool, so I'll be able to update my zuul later :) | 16:04 |
| quiquell | mordred: damn I think we need to run also without tags to have "latest" :-/ | 16:05 |
| quiquell | mordred: will look it tomorrow have to drop | 16:05 |
| *** quiquell is now known as quiquell|off | 16:06 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Alwas use pathless docker mirror URI https://review.openstack.org/625596 | 16:07 |
| mordred | quiquell|off: oh yeah. good call - thanks for looking at it so far - I may poke at that patch some while you're afk - it's been on my todo list for a while so I appreciate you getting it going | 16:09 |
| *** themr0c has joined #zuul | 16:16 | |
| *** themroc has quit IRC | 16:19 | |
| *** pcaruana has quit IRC | 16:28 | |
| *** bhavikdbavishi has joined #zuul | 16:38 | |
| corvus | SpamapS: can you look at https://review.openstack.org/619156 if you have a moment? | 16:49 |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/nodepool master: [wip] Add dogpile.cache master to the -src tests https://review.openstack.org/625457 | 17:00 |
| AJaeger | here's a change for zuul-jobs, could you put that on your review queue, please? https://review.openstack.org/#/c/624575/ as test change and then https://review.openstack.org/#/c/621840 as real one. Together with a note for testing: https://review.openstack.org/624578 | 17:15 |
| AJaeger | all looks fine to me - but I'd like broader review so didn't +2 yet | 17:15 |
| openstackgerrit | Merged openstack-infra/zuul master: Use combined status for Github status checks https://review.openstack.org/623417 | 17:21 |
| *** themr0c has quit IRC | 17:27 | |
| SpamapS | corvus: looking. When you've got a moment, I'd love to walk through an issue I'm having that I don't understand. | 17:33 |
| tobiash | AJaeger: commented on the testing note | 17:34 |
| corvus | SpamapS: i'm available now | 17:35 |
| SpamapS | corvus: perfect, give me a second to get a +2 on that auth review. ;) | 17:35 |
| SpamapS | (Reviewing the nodepool side too) | 17:37 |
| *** bhavikdbavishi has quit IRC | 17:37 | |
| SpamapS | corvus: ok, so I have this in my repo (GoodMoney/tech) http://paste.openstack.org/show/737498/ and it is tied to this pipeline: http://paste.openstack.org/show/737499/ | 17:40 |
| AJaeger | thanks, tobiash ! | 17:40 |
| SpamapS | corvus: tech has two branches, 'master', and 'prod'. The idea is that you only run gmapiprod-post on merge to prod, and only run gmapidev-post on merge to master.... | 17:41 |
| SpamapS | corvus: what actually ends up happening is, both copies of gmapidev-post run on merges to master, the one from master, and the one from prod. | 17:41 |
| SpamapS | Now, I can delete either/or from the two respective branches, and solve this problem for now.. but I want to understand a) is there a way to keep the configs consistent between? b) is there an idiomatic example of what I'm trying to do? | 17:42 |
| corvus | SpamapS: oh, the contents of that paste are in the repo itself, and on both branches? | 17:43 |
| openstackgerrit | Merged openstack-infra/nodepool master: Switch devstack jobs to Xenial https://review.openstack.org/624855 | 17:44 |
| SpamapS | corvus: correct | 17:45 |
| SpamapS | gmapidev-post puts anything that lands on master into a staging env for UAT type things.. then people PR to prod to publish forealz. | 17:46 |
| SpamapS | I'm actually thinking of changing to a 'tag for prod' model. | 17:46 |
| SpamapS | But for now, this is how it works. | 17:46 |
| SpamapS | (tags would make it easier to reuse the container images) | 17:46 |
| SpamapS | (FYI, this is how https://goodmoney.com is published) | 17:47 |
| corvus | SpamapS: based on what i see as the differences there -- a couple of variables and the run playbook, i think you could make them one job | 17:49 |
| corvus | SpamapS: you could then have the "master" and "prod" branches just have implied branch variants with those differences. | 17:50 |
| corvus | SpamapS: or, you could define the job in master with a branch matcher of ".*" and then add explicit branch variants in either the branches, or even on the project-pipeline definition | 17:51 |
| SpamapS | corvus: Ok, that was one avenue I'd thought of too... | 17:52 |
| corvus | SpamapS: the latter option is probably the best way to have a single definititon with the bulk of the settings, then one or two more little definitions with the delta. | 17:52 |
| SpamapS | corvus: this was.. extremely confusing and frustrating to debug. The first evidence I had it was not working well was that the slack notifications were duplicated. But it really broke when I landed a backward-incompatible change in the job's ansible in master, and then the prod ansible ran and it cost me 7 minutes of downtime. :-/ | 17:53 |
| corvus | SpamapS: the former presents the most localized view (you just look in the prod branch to see the complete prod definition of the job) | 17:53 |
| SpamapS | (I actually caught it in staging, but we were under a regulatory time box which forced me to manually deploy.. and manual deploy screwed things up for 7 minutes. ;-) | 17:53 |
| SpamapS | One job with branch variant seems like a good plan. | 17:55 |
| SpamapS | But the really weird thing, the thing that really blew my mind, was that it ran every piece of the job, one from master, one from prod, when I landed stuff in each branch. | 17:55 |
| SpamapS | So like, two pre's, two runs, two posts. | 17:55 |
| corvus | SpamapS: i understand -- i think perhaps as we work on narrative documentation for setting up jobs, we might want to come up with a rule of thumb and highlight it -- something like "don't use explicit branch matchers on in-repo jobs in branched repos". it's not a hard-and-fast rule (i'm suggesting you violate it in my option #2) -- but it works most of the time, including in my option #1. and the point is | 17:55 |
| corvus | just to avoid foot-guns. | 17:55 |
| SpamapS | Ok, this makes some sense now. My expectation was that only the jobs defined in the branch I was in, would run, which was false. | 17:57 |
| SpamapS | The explicit branch matching made the config match despite being from a different branch. | 17:57 |
| SpamapS | corvus: I think I'll just accelerate my move to having prod pushes just take tags... that way there's just one branch, and two distinct jobs. | 18:02 |
| *** jpena is now known as jpena|off | 18:14 | |
| *** electrofelix has quit IRC | 18:32 | |
| fungi | fun discovery in #openstack-infra, if you upgrade ansible on an active executor, jinja2 templating can fail somewhat cryptically on an inability to find ansible's core filter plugin | 18:57 |
| fungi | the more you know! | 18:58 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/zuul-jobs master: Remove unsecure chmod which makes src world writable https://review.openstack.org/625694 | 19:27 |
| openstackgerrit | Merged openstack-infra/zuul master: Modify some file content errors https://review.openstack.org/624278 | 20:00 |
| *** sshnaidm is now known as sshnaidm|off | 20:07 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Add a note on testing https://review.openstack.org/624578 | 20:39 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: Add a note on testing trusted roles https://review.openstack.org/624578 | 20:39 |
| *** hashar has joined #zuul | 21:02 | |
| *** dmellado has quit IRC | 21:05 | |
| *** gouthamr has quit IRC | 21:06 | |
| *** gouthamr has joined #zuul | 21:17 | |
| *** yolanda has quit IRC | 21:18 | |
| fungi | another osf newsletter is going out wednesday. can anyone think of important zuul news which isn't covered in corvus's latest project update on the zuul-discuss ml or in published release notes for recent releases of zuul and nodepool? | 21:28 |
| tobiash | fungi: I'm not aware of further topics | 21:29 |
| fungi | i figured those would be a good base source of news items at any rate | 21:29 |
| *** gouthamr_ has joined #zuul | 21:30 | |
| tobiash | yay, ubuntu removed the setuid bit from bwrap in the latest package version. This breaks dockerized executors running on centos based hosts | 21:30 |
| clarkb | tobiash: this is a case of the userland having different expectations from the kernel than it got? | 21:31 |
| clarkb | probably beacuse ubuntu kernels allow user namespacing | 21:31 |
| tobiash | yes, that's probably the reason they removed it | 21:31 |
| tobiash | but centos kernels don't allow user namespacing | 21:31 |
| tobiash | and thus bwrap needs setuid | 21:31 |
| tobiash | just did an executor upgrade and nothing worked | 21:32 |
| tobiash | everything went into retry limit | 21:32 |
| clarkb | ouch | 21:32 |
| *** hashar has quit IRC | 21:32 | |
| clarkb | tobiash: any idea if rhel8 beta allows user namespacing (mostly just curious if this general problem goes away in the future) | 21:32 |
| tobiash | clarkb: I don't know | 21:33 |
| corvus | tobiash: this is if you build an executor image based on ubuntu, right? can you switch to the upstream alpine images? (or build your own alpine images using pbrx?) | 21:33 |
| tobiash | corvus: pbrx doesn't really fit into my current deployment process so I'm building my own images based on ubuntu (as this is currently kind of the default platform for testing) | 21:35 |
| mordred | ah - I didn't realize you'd moved off of alpine images - good to know | 21:35 |
| mordred | (Also good to know about the setuid bit removal) | 21:36 |
| tobiash | mordred: yes, I moved to ubuntu two months ago | 21:36 |
| mordred | cool. | 21:36 |
| *** gouthamr_ has quit IRC | 21:37 | |
| tobiash | I switched mostly because of python weels, ubuntu as default testing platform and potentially faster python on ubuntu because of glibc | 21:39 |
| mordred | wheels existing for ubuntu is a big one | 21:40 |
| clarkb | zuul's wheels should be universal though? | 21:40 |
| mordred | tobiash: do you have any data on faster python with glibc? | 21:40 |
| mordred | clarkb: for dependencies | 21:40 |
| clarkb | mordred: well for deps pypi doesn't distinguish distros for wheels | 21:40 |
| clarkb | you get the rhel5 like universal wheel option or build it yourself | 21:41 |
| *** gouthamr_ has joined #zuul | 21:41 | |
| tobiash | mordred: I've seen a python in docker benchmark for different base images on phoronix half a year ago | 21:41 |
| tobiash | and alpine was much slower in almost every test case | 21:41 |
| clarkb | tobiash: mordred re performance glibc throws memory at performance but musl is memory conscious | 21:41 |
| mordred | clarkb: right - but alpine runs a different libc, so I think there are wheels that publish binaries linked with glibc, no? | 21:42 |
| clarkb | I thought the rhel5 base image was intentionally chosen because it should be compatible across the newer libc options | 21:43 |
| clarkb | its possible that doens't work as well as expected | 21:43 |
| tobiash | found it: https://www.phoronix.com/scan.php?page=article&item=docker-summer-2018&num=4 | 21:43 |
| tobiash | so they ran pybench | 21:44 |
| *** gouthamr_ has quit IRC | 21:46 | |
| tobiash | corvus: the reason for pbrx not fitting for me is that I'm currently using openshift build configs that only support s2i or plain old dockerfiles so I'm using a plain old dockerfile | 21:48 |
| mordred | tobiash: do they support multi-stage dockerfiles? | 21:49 |
| tobiash | unfortunately not | 21:49 |
| tobiash | I learned about multi-stage dockerfiles a few months ago and this was awesome, then tried them in openshift but it didn't like them | 21:51 |
| clarkb | is openshift using buildah | 21:51 |
| *** gouthamr_ has joined #zuul | 21:51 | |
| tobiash | I have no idea, I just know it's not using docker | 21:54 |
| *** gouthamr_ has quit IRC | 21:56 | |
| mordred | the openshift-y way of doing the multi-stage build approach would be builder images for s2i ... and in making those following the guidance for compiled languages rather than the guidance for interpreted languages | 22:00 |
| mordred | I was toying with the idea of making one of those for zuul - or for re-imagining pbrx as a generic builder image | 22:00 |
| mordred | but haven't gotten around to playing with it yet | 22:00 |
| tobiash | I think so | 22:01 |
| clarkb | it likely isworth noting that the theoretical "kernel and user space won't agree" problem is no longer theoretical here. I wonder if that does make an argument for running containers within the same family as the host kernel | 22:01 |
| tobiash | but I anyway want to switch to images built by zuul but then I have some kind of bootstrap problem | 22:01 |
| clarkb | (I'm trying to be open to the idea that "just build one image because it doesn't matter" is maybe not the best approach and kolla et al may have something in their approach) | 22:02 |
| tobiash | clarkb: I think zuul is a special case here because of sandboxing things | 22:04 |
| tobiash | oh and nodepool too because of dib ;) | 22:05 |
| clarkb | tobiash: ya its possible that its such a corner case that it doesn't matter in general. Mostly I'v eargued against the kolla approach of container on every platform possible, because it seems like a waste of effort and shouldn't matter if containers are being useful | 22:05 |
| clarkb | but maybe they aren't quite as strong and abstraction as we might assume :) | 22:06 |
| tobiash | yes, if you aren't using fancy new or not widely adopted kernel features I think the approach having one base image is sufficient | 22:07 |
| clarkb | tobiash: as far as possible terrible hacks go can you setuid the binary in your image build and it works? | 22:08 |
| tobiash | yes, that was my fix, chmod 4755 at the end and everything worked again | 22:09 |
| tobiash | it was just unexpected that ubuntu bionic as an lts release made such a change without even changing the version number of bwrap | 22:10 |
| clarkb | ya | 22:10 |
| SpamapS | tobiash: sorry I said in #openstack-infra, but, yeah, Alpine-- IMO. I don't think there's much reason to use them now that we have good solid slim Debian/Ubuntu images. | 22:35 |
| tobiash | SpamapS: yes, the most important reason for alpine in the past for me was the size | 22:36 |
| SpamapS | And I'm very dubious about who the Alpine devs are. | 22:36 |
| tobiash | but the zuul images are large anyway | 22:36 |
| SpamapS | Not that I think they're bad people, but they just appeared out of nowhere, their bug tracker is hard to join, and I'm just not sure they're ready to power as much as they're powering. | 22:37 |
| SpamapS | Yeah my zuul alpine images are not tiny. | 22:37 |
| * SpamapS has a todo to switch to Debian. ;) | 22:37 | |
| SpamapS | zuul-base latest 80dca46bd93d 2 weeks ago 265MB | 22:37 |
| tobiash | in fact my scheduler image based on bionic is just 250mb | 22:37 |
| SpamapS | 265MB isn't so bad I guess. | 22:37 |
| SpamapS | Anything under 500MB is a win really. | 22:38 |
| tobiash | my alpine based zuul was around 600mb | 22:38 |
| SpamapS | tobiash: interesting | 22:38 |
| SpamapS | What were you putting in there? ;) | 22:38 |
| SpamapS | zuul-scheduler latest 914c4f94bfda 2 weeks ago 265MB | 22:38 |
| SpamapS | (The base and scheduler are the same image w/ diff cmds) | 22:38 |
| tobiash | probably because of something not cleaned up during installing gcc world, compiling stuff and deinstalling gcc world in a single docker task... | 22:39 |
| SpamapS | tobiash: multi-stage builds FTW | 22:39 |
| SpamapS | wheel it up in a build stage, then just copy the wheels out into the runtime stage. | 22:39 |
| tobiash | SpamapS: tell that the openshift devs ;) | 22:39 |
| SpamapS | I've not used openshift. Not sure what the fuss is about. K8S+Artifactory is pretty nice.. :-P | 22:40 |
| SpamapS | Or in my case right now, K8S + AWS ECR | 22:40 |
| SpamapS | But I'm sure there's some "Enterprise" thing I'm not doing that I should be. :) | 22:40 |
| tobiash | SpamapS: yes, I also plan to switch to something like this and not using buildconfigs anymore | 22:41 |
| SpamapS | I'm pretty happy with Spruce for templating things btw. | 22:41 |
| SpamapS | https://github.com/geofffranks/spruce | 22:41 |
| SpamapS | I was going to make Helm charts for Zuul but Helm is.. not inspiring IMO. | 22:42 |
| SpamapS | But I might open source our spruce based zuul env if I get time. | 22:42 |
| tobiash | interesting, never heard about spruce | 22:43 |
| clarkb | 256MB is about 1/3 the size of an image on centos to run rsyslog :P | 22:46 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add governance document https://review.openstack.org/622439 | 22:49 |
| SpamapS | Indeed. But to give context, it's likely only about 50MB bigger to run it on Debian or Ubuntu. | 22:49 |
| SpamapS | (as long as you start with the -slim containers which are basically just a bare debootstrap) | 22:50 |
| clarkb | also alpine was around in the embedded space for a long time aiui (that is why musl too). The come out of nowhere was docker pushing them as a lightweight base alternative | 22:50 |
| mordred | SpamapS: so - the other reason I was doing alpine was that bubblewrap on debian was ... old, and it was the good version in alpine | 23:02 |
| mordred | but - that's not really a long-term-design reason :) | 23:02 |
| mordred | SpamapS: and yeah - starting with python:slim instead of python:alpine would not be super onerous or anything | 23:03 |
| SpamapS | Did Bionic release with old too? | 23:04 |
| SpamapS | 0.2.1 .. | 23:05 |
| mordred | python:slim is on top of debian | 23:05 |
| SpamapS | Ah yeah, 0.1.7 | 23:06 |
| mordred | SpamapS: which has 0.1.7-1 | 23:06 |
| SpamapS | looks like backports has 0.3.1 | 23:07 |
| mordred | yeah. kind of unfortunate. then I went down the rabbit hole of trying to figure out how to express ppas in bindep files | 23:07 |
| SpamapS | any carrots down there? | 23:07 |
| mordred | just insanity | 23:07 |
| SpamapS | A very merry unbirthday, to Stretch. | 23:08 |
| SpamapS | corvus: ok, so would this work the way I described I wanted it working (only run one version in either master, or prod)? http://paste.openstack.org/show/737515/ | 23:18 |
| SpamapS | Hopefully that only creates two variants in the config, one for all branches, and one for prod. | 23:18 |
| SpamapS | (and hopefully in prod, it only runs the prod one?) | 23:19 |
| corvus | SpamapS: that would work if it were in a branchless repo (eg, project-config, or tempest). but in a branch repo, you'll need to add 'branches: .*' to the first job def. other than that should be good. | 23:20 |
| SpamapS | Ohhh | 23:24 |
| SpamapS | Ok you said that before | 23:24 |
| SpamapS | now I get why | 23:24 |
| SpamapS | implicitness biting me ;) | 23:25 |
| *** dkehn has quit IRC | 23:37 | |
| *** dkehn has joined #zuul | 23:38 | |
| corvus | SpamapS: if you think this is the way everything should behave for this project, you may want to set https://zuul-ci.org/docs/zuul/user/config.html#attr-pragma.implied-branch-matchers to false | 23:39 |
| corvus | SpamapS: that will turn off implied branch matchers altogether in that repo. then your example would work as you wrote it. | 23:39 |
| corvus | SpamapS: i should have thought of that earlier, but was probably trying to be too minimal in suggesting changes. | 23:40 |
| SpamapS | corvus: Hm, I'll think about it. I am seriously pondering de-branching anyway. | 23:42 |
| SpamapS | corvus: thanks for the help. My mind is fully unbent again. | 23:42 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [wip] Add dogpile.cache master to the -src tests https://review.openstack.org/625457 | 23:43 |
| corvus | SpamapS: no prob. i *think* we should be able to support either thing. we've just learned another thing to say in the to-be-written narrative docs: if you're doing a dev/prod sort of branching, turn of implied-branch-matchers. :) | 23:44 |
| corvus | s/turn of/turn off/ | 23:44 |
| SpamapS | Yeah I think that's right. | 23:46 |
| SpamapS | implied matchers are really there for more of a stable branch setup, I presume. | 23:46 |
| corvus | yep | 23:54 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!