Monday, 2020-04-06

openstackgerritIan Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role  https://review.opendev.org/71763900:41
openstackgerritIan Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role  https://review.opendev.org/71763900:48
openstackgerritIan Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role  https://review.opendev.org/71763900:52
mnaseravass: woops i didnt see that because of the typo in my name :p00:54
mnaseryeah, i mean i'm open to whatever, but it seems to fit the theme of these days :)00:54
openstackgerritIan Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role  https://review.opendev.org/71763901:01
openstackgerritIan Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role  https://review.opendev.org/71763901:06
openstackgerritIan Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role  https://review.opendev.org/71763901:14
*** rfolco has joined #zuul01:24
*** swest has quit IRC01:28
*** ysandeep|away is now known as ysandeep|rover01:30
*** ysandeep|rover is now known as ysandeep|away01:31
openstackgerritIan Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role  https://review.opendev.org/71763901:33
*** swest has joined #zuul01:44
*** rfolco has quit IRC01:45
openstackgerritIan Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role  https://review.opendev.org/71763901:57
*** Goneri has quit IRC02:15
openstackgerritIan Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role  https://review.opendev.org/71763902:44
openstackgerritIan Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role  https://review.opendev.org/71763902:52
openstackgerritIan Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role  https://review.opendev.org/71763903:03
*** ysandeep|away is now known as ysandeep|rover03:33
*** bhavikdbavishi has joined #zuul03:57
*** bhavikdbavishi1 has joined #zuul04:02
*** bhavikdbavishi has quit IRC04:04
*** bhavikdbavishi1 is now known as bhavikdbavishi04:04
openstackgerritIan Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role  https://review.opendev.org/71763904:27
openstackgerritIan Wienand proposed zuul/zuul-jobs master: Add bionic-plain testing  https://review.opendev.org/71765504:27
openstackgerritIan Wienand proposed zuul/zuul-jobs master: test jobs: fixup autogeneration header  https://review.opendev.org/71765604:27
openstackgerritIan Wienand proposed zuul/zuul-jobs master: Update Fedora to 31  https://review.opendev.org/71765704:29
*** evrardjp has quit IRC04:36
*** evrardjp has joined #zuul04:37
openstackgerritIan Wienand proposed zuul/zuul-jobs master: Add bionic-plain non-voting testing  https://review.opendev.org/71765505:44
openstackgerritIan Wienand proposed zuul/zuul-jobs master: test jobs: fixup autogeneration header  https://review.opendev.org/71765605:44
openstackgerritIan Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role  https://review.opendev.org/71763905:44
openstackgerritIan Wienand proposed zuul/zuul-jobs master: Update Fedora to 31  https://review.opendev.org/71765705:44
openstackgerritIan Wienand proposed zuul/zuul-jobs master: [wip] ensure-tox: preinstall pip  https://review.opendev.org/71766305:44
*** dpawlik has joined #zuul06:28
*** rpittau|afk is now known as rpittau06:35
*** bhavikdbavishi has quit IRC06:37
*** bhavikdbavishi has joined #zuul06:37
tobiashclarkb: re leaked dib processes: I've found one as well. What's weird about that is that it has been dropped from nodepool in the process hierarchy. What I also found is that this image faced a build timeout in nodepool. So maybe canceling timeouted builds in nodepool has an issue.06:41
*** avass is now known as Guest6280806:44
*** avass has joined #zuul06:44
*** bhavikdbavishi has quit IRC06:45
*** saneax has joined #zuul06:53
*** yolanda has joined #zuul06:56
openstackgerritIan Wienand proposed zuul/zuul-jobs master: test jobs: fixup autogeneration header  https://review.opendev.org/71765606:57
openstackgerritIan Wienand proposed zuul/zuul-jobs master: [wip] ensure pip role  https://review.opendev.org/71763906:57
openstackgerritIan Wienand proposed zuul/zuul-jobs master: [wip] ensure-tox: preinstall pip  https://review.opendev.org/71766306:57
openstackgerritIan Wienand proposed zuul/zuul-jobs master: Update Fedora to 31  https://review.opendev.org/71765706:58
openstackgerritIan Wienand proposed zuul/zuul-jobs master: ensure-tox: update Python version documentation  https://review.opendev.org/71766907:05
*** jcapitao has joined #zuul07:15
openstackgerritMerged zuul/zuul-jobs master: ensure-tox: update Python version documentation  https://review.opendev.org/71766907:24
*** gtema has joined #zuul07:49
*** bhavikdbavishi has joined #zuul07:51
*** bhavikdbavishi1 has joined #zuul07:54
*** bhavikdbavishi has quit IRC07:55
*** bhavikdbavishi1 is now known as bhavikdbavishi07:55
*** tosky has joined #zuul07:58
*** spsurya_ has joined #zuul07:59
fricklerso what's with these "Comments left for invalid file" in https://review.opendev.org/717533 ?08:04
fricklerwould be great if one could at least see from which job they come instead of having to check them all08:04
*** gtema has quit IRC08:05
*** gtema has joined #zuul08:06
*** gtema has quit IRC08:06
*** gtema has joined #zuul08:07
*** sshnaidm|afk is now known as sshnaidm08:07
AJaegerfrickler: they come from https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/tox/library/tox_parse_output.py - the work that tobiash, mnaser, mordred did.08:14
AJaegerI think the invalid file is a reaction of gerrit to notify about a file that is not in the commit - but I'm not completly sure about that.08:16
fricklerAJaeger: that's what I'm assuming, yes, therefore I'm asking whether we can make those comments more easily consumable, otherwise zuul should stop submitting them IMO08:17
AJaegerfrickler: I proposed in https://review.opendev.org/#/c/717486/ to check which files are part of the change and only report on those (iterate over list of files in git change) - 717486 was a quick fix08:19
fricklerAJaeger: that was twice the same patch for ignoring .tox, I cannot find the other one08:27
AJaegerfrickler: it was meant to be the same08:27
AJaegerThe commit message explained what was the better solution but I did the quick fix only08:28
*** dpawlik has quit IRC08:29
*** ysandeep|rover is now known as ysandeep|lunch08:36
*** dpawlik has joined #zuul08:38
*** ysandeep|lunch is now known as ysandeep09:18
*** ysandeep is now known as ysandeep|rover09:19
openstackgerritAlbin Vass proposed zuul/zuul master: WIP: Add support for ansible callbacks  https://review.opendev.org/71726009:31
openstackgerritAlbin Vass proposed zuul/zuul master: WIP: Enables whitelisting and configuring callbacks  https://review.opendev.org/71726009:36
openstackgerritAlbin Vass proposed zuul/zuul master: WIP: Enables whitelisting and configuring callbacks  https://review.opendev.org/71726009:37
*** gtema has quit IRC10:32
*** bhavikdbavishi has quit IRC10:54
*** bhavikdbavishi has joined #zuul11:00
*** gtema has joined #zuul11:03
*** jcapitao is now known as jcapitao_lunch11:05
openstackgerritJan Kundrát proposed zuul/zuul-jobs master: docs: fix a typo in `run-test-command`  https://review.opendev.org/71771311:18
*** bhavikdbavishi has quit IRC12:11
*** rlandy has joined #zuul12:12
*** jcapitao_lunch is now known as jcapitao12:25
openstackgerritFabien Boucher proposed zuul/zuul master: pagure: Make use of the new project webhook/token endpoint  https://review.opendev.org/71773212:30
*** Goneri has joined #zuul12:31
*** rfolco has joined #zuul12:35
mnaserAJaeger, frickler: we're working on cleaning all that up..12:44
AJaegermnaser: thanks! who's we?12:45
mnaserAJaeger: i think the collective we of zuul-ers as we discover more "invalid file" comments :p12:45
AJaegermnaser: Ah ;)12:46
openstackgerritAlbin Vass proposed zuul/zuul master: WIP: Enables whitelisting and configuring callbacks  https://review.opendev.org/71726013:03
*** hashar has joined #zuul13:06
*** Goneri has quit IRC13:08
zbrmnaser: clarkb: i suppose today we merge the s/install/ensure/ stuff, right?13:13
zbrasking because i want to see them in as i have several other patches that would need to be rebased after that.13:14
fungitoday's when we announced we would given no objections, if i'm remembering accurately13:14
*** Goneri has joined #zuul13:15
*** avass is now known as Guest4005413:24
*** Guest62808 is now known as avass13:24
openstackgerritTristan Cacqueray proposed zuul/zuul-registry master: config: add environment variable substitution  https://review.opendev.org/71064413:25
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Add zuul-registry deployment  https://review.opendev.org/71065013:30
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Update dhall-kubernetes version to v4.0.0  https://review.opendev.org/71734213:42
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Add zuul-registry deployment  https://review.opendev.org/71065013:42
AJaegerzbr: yes, I'll slowly start merging that one now13:52
AJaegerconfig-core, infra-root, I slowly start merging the stack starting at https://review.opendev.org/#/c/716663 for renaming from roles from install to ensure. This should not cause problems - but let's play it safe...13:53
tristanCmordred: heh, so opendev is going to deploy zuul using docker-compose ? ( i just saw https://review.opendev.org/717620 )13:53
mordredtristanC: it's how we're doing other container things atm - so I worked it up keeping the same pattern13:56
mordredwe haven't actually had an opendev-level conversation about updating the zuul deployment yet- I was just a bit bored this weekend13:57
*** zxiiro has joined #zuul13:57
tristanCmordred: oh ok, then i guess you gave up on using k8s?13:58
avassAJaeger: oh, nice13:59
*** cdearborn has joined #zuul14:00
mordredtristanC: I don't know that we gave up on k8s as much as we haven't reached a point where we've decided to try using k8s in the first place ;)14:02
mordredtristanC: so far it hasn't been a road we've been ready to go down because it involves adding "running a k8s" to the list of things we need to do.  at least so far, nothing we've been running in containers would be made so much easier by k8s that the cost of running k8s would be worth it.14:04
*** lennyb has quit IRC14:04
*** gtema has quit IRC14:10
*** gtema has joined #zuul14:11
tristanCmordred: ok fair enough. though it is surprising to see yet another zuul deployment recipe14:11
tristanCmordred: if you are bored, those zuul-operator changes are waiting for review https://review.opendev.org/707193 and https://review.opendev.org/71065014:12
mordredtristanC: fair - I don't think we're aiming to make this one re-usable by others as much as "stop using puppet" :)14:13
*** sshnaidm is now known as sshnaidm|afk14:24
mordredtristanC: in other news though: https://review.opendev.org/#/c/717057/ - we're almost done getting rid of our cron job!14:25
openstackgerritTristan Cacqueray proposed zuul/zuul-registry master: config: add environment variable substitution  https://review.opendev.org/71064414:26
openstackgerritTristan Cacqueray proposed zuul/zuul-registry master: Use explicit provides/requires for container jobs  https://review.opendev.org/71776714:26
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Add zuul-registry deployment  https://review.opendev.org/71065014:27
tristanCmordred: iiuc the cron job is what applies puppet without being driven by ansible?14:27
openstackgerritMerged zuul/zuul-jobs master: Rename install-docker to ensure-docker for consistency  https://review.opendev.org/71666314:30
*** ysandeep|rover is now known as ysandeep|afk14:31
*** bhavikdbavishi has joined #zuul14:33
shanemcdHi everyone. I recently started seeing "Something went wrong." when trying to look at live logs in Safari.14:34
shanemcdNot sure if this is helpful: https://imgur.com/a/YT99B5O14:34
shanemcdSeems ok in other browsers I've tried14:34
*** bhavikdbavishi1 has joined #zuul14:35
tristanCshanemcd: it's probably related to the Content-Security-Policy setup by software-factory. for somereason, safari is more picky about it. i think we can fix that server side14:35
AJaegeravass: will you update other repos as well? For example, check http://codesearch.openstack.org/?q=name%3A%20install-docker&i=nope&files=&repos=14:36
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: docker-install: workaround for centos-8 conflicts  https://review.opendev.org/70305314:36
avassAJaeger: sure I'll take a look at it later14:37
*** bhavikdbavishi has quit IRC14:37
*** bhavikdbavishi1 is now known as bhavikdbavishi14:37
AJaegerthanks, avass !14:37
mordredtristanC: the cron job is what currently runs ansible. ansible drives everything for us - including both native ansible things as well as puppet14:38
mordredtristanC: those updates shift us to having ansible driven by zuul \o/14:38
tristanCmordred: oh i see, thus you are replacing cron with zuul?14:38
mordredyup14:39
fungimordred: we *did* try deploying gitea in kubernetes, right?14:45
fungiso not entirely fair to say we haven't tried yet14:45
tristanCfungi: i do remember discussion about using ceph on k8s too14:45
mordredyeah - but that's also a bit of a special case14:45
mordredthe gitea needs for that k8s deployment were quite intensive14:45
mordredI'm not sure that's indicative of what a more "normal" use might look like14:46
mordredfungi: but that it definitely true14:46
*** bhavikdbavishi has quit IRC14:46
fungiexcept the hope was that kubernetes would make it easier to deploy and manage a complex service like gitea. for "normal" (simple) services it seems like swatting a fly with a buick14:47
*** sshnaidm|afk is now known as sshnaidm14:51
*** sgw has quit IRC14:53
*** bhavikdbavishi has joined #zuul14:55
openstackgerritMerged zuul/zuul-jobs master: Rename install-kubernetes to ensure-kubernetes for consistency  https://review.opendev.org/71666714:56
mordredfungi: well - that wasn't the 'extra complex' part I mean. the running ceph and hooking it in as a k8s storage provider was the bit that made things a bit more heavyweight ... and even then, honestly, it didn't do a bad job, it was just figuring out how to get rook working for us took a while :)14:57
fungigot it14:57
openstackgerritMonty Taylor proposed zuul/zuul master: Install skopeo in executors  https://review.opendev.org/71761315:04
openstackgerritMonty Taylor proposed zuul/zuul master: Add gear to the ansible envs  https://review.opendev.org/71761615:04
*** jcapitao is now known as jcapitao_afk15:05
*** sgw has joined #zuul15:12
*** ysandeep|afk is now known as ysandeep15:13
clarkbfor us I think the biggest unsolved problem is simply operating a k8s15:16
clarkbturns out the hosted options aren't great, and then we've tried a couple versions of the self managed k8s15:17
clarkband docker compose has been a really good transition point because the underlying structure doesn't change, just the "packaging" really15:18
clarkband from there you can take the same images then change the structure15:18
tristanCclarkb: that makes sense. thanks for the details.15:20
openstackgerritMerged zuul/zuul-jobs master: Rename install-openshift to ensure-openshift for consistency  https://review.opendev.org/71667515:20
openstackgerritMerged zuul/zuul-jobs master: Rename install-podman to ensure-podman for consistency  https://review.opendev.org/71668215:20
tristanCmakes me wonder if such docker-compose based scenario could be implemented in a zuul project so that it can be re-used by other admin facing the same issue?15:21
clarkbtobiash: interesting, ours don't ever seem to hit the build timeout according to logs15:21
clarkbtobiash: my theory is/was that we are hitting a subprocess wait() call deadlock as warned against in the subprocess module docs15:21
*** gtema has quit IRC15:22
mordredtristanC: I think it would not be hard to extract what I've started on there once we're happy with it15:29
mordredtristanC: I kind of purposefully didn't parameterize a few things just for expediency15:29
mordredbut - as a followup if we decided it was valuable, I don't think it would be hard to accomplish15:29
mordredtristanC: could also see about adding it as an option to windmill ... since what we've got there is really just using compose as a replacement for pip/apt/rpm and systemd - the rest if all ansible15:30
mordred(again, I didn't want to overcomplicate things this time)15:33
corvusmordred, tristanC, mnaser: i left a note on https://review.opendev.org/717616 -- i just want to be clear that i don't think "opendev uses this" is necessarily a sufficient reason to add something to the zuul images.  but if we still want to add it, i'm ok.15:34
tristanCmordred: that sounds like a good plan. Shouldn't windmill be moved to the zuul/ namespace then?15:36
*** jcapitao_afk is now known as jcapitao15:37
*** ysandeep is now known as ysandeep|away15:39
mordredtristanC: well - so far it's been paul - if we get to a point where we're looking at doing that and supporting it more directly we could certainly talk about it - but I think one thing at a time ;)15:42
mordredcorvus: I added "remove zuul-migrate" to https://etherpad.openstack.org/p/zuulv4 - seems like maybe a good time to cut that loose?15:43
corvusmordred: heh, yeah :)15:44
tristanCmordred: opendev system-config not leveraging/contributing to windmill doesn't help either... i kind of hoped puppet-zuul would have been replaced by one of the existing zuul deployment project.15:47
*** bhavikdbavishi has quit IRC15:52
tobiashtristanC, corvus: there is already a way to install more things into the ansible venvs (ANSIBLE_EXTRA_PACKAGES env var when running zuul-manage-ansible)16:00
mordredtobiash: yeah - but that's run at container build time16:00
tobiasha zuul.conf entry might confuse especially users who use the official docker images as the ansible vens are created during build time, not runtime16:00
tobiashmordred: configuring via zuul.conf would be the same16:01
mordredtobiash: oh - I might be responding to a different thing ... what's the context of your thing?16:01
tobiashhttps://review.opendev.org/71761616:01
mordredtobiash: ah! I hadn't seen the comments there16:03
mordredyeah - I think this is a build-time vs runtime thing - and a zuul.conf entry wouldn't help so much16:03
tobiashso actually I see two ways to put extra stuff there for container users, add into the official image or create a new image based from the official one and re-run zuul-manage-ansible there16:03
mordredyah16:03
tobiashI guess the question is which to choose for gear16:04
tristanCcorvus: this wasn't part of the operator spec, but https://review.opendev.org/#/c/710650/ adds a registry component to deploy the zuul-registry. and it seems to work, integration tests shows `Login Succeeded!`16:04
tobiashbut at least there seem two users needing this16:04
mordredyeah - I mean - it's a small depend, and it's already installed by zuul components anyway, so it's not a _big_ burden. otoh - process-wise it's maybe a little odd if we don't have the consuming code for it in zuul-jobs16:05
tristanCmordred: gear almost made it into zuul-jobs with https://review.opendev.org/#/c/537847/16:06
*** rpittau is now known as rpittau|afk16:08
openstackgerritMerged zuul/zuul-jobs master: Rename install-devstack to ensure-devstack for consistency  https://review.opendev.org/71668516:08
mordredtristanC: maybe we should revive that? clarkb ^^ (we have more of a pattern now for putting in some roles that require more infrastructure around them along with some docs on how to set that infrastructure up - like zuul-registry)16:10
clarkbmordred: tristanC I never even knew that was a thing. I think its a fine idea. I also really don't see the tooling openstack uses as openstack specific16:11
clarkbyes it was made to meet openstack's demands but nothing about the indexing pipeline is openstack specific16:11
clarkbOn the visualization side there are openstack things that need to be decoupled, but the indexing backend is literally "take logs and index tehm"16:12
openstackgerritMerged zuul/zuul-jobs master: Rename install-javascript-packages to ensure-javascript-packages for consistency  https://review.opendev.org/71668716:17
mnasercorvus: yes, i agree with you, it's a +1 from me in the perspective of useful for zuul deployers and not opendev16:26
fungiwell, useful for opendev too16:32
fungiwe could potentially align with it at some point16:32
corvusmnaser, mordred, tobiash, tristanC, clarkb, fungi: okay, i just didn't want to establish an "opendev needs it so we'll add it to the image" precedent.  but it seems like we feel it's generally useful and low-cost enough to add it.  i'm happy with the outcome.16:35
tobiash++16:35
*** bhavikdbavishi has joined #zuul16:36
corvusmordred: though, apparently zuul has other thoughts :)  https://zuul.opendev.org/t/zuul/build/d176984ce7814f91a319938601856a9516:36
*** evrardjp has quit IRC16:36
corvusmordred: ah, it's the skopeo change that's failing16:36
*** evrardjp has joined #zuul16:37
*** bhavikdbavishi1 has joined #zuul16:41
*** bhavikdbavishi has quit IRC16:42
*** bhavikdbavishi1 is now known as bhavikdbavishi16:42
openstackgerritMerged zuul/zuul-jobs master: Rename install-nodejs to ensure-nodejs for consistency  https://review.opendev.org/71669216:50
clarkbavass: tristanC https://review.opendev.org/#/c/716298/2 and https://review.opendev.org/#/c/708871/ conflcit with each other. Is there a prefered merge order?16:53
clarkbtristanC: I left a question/note on https://review.opendev.org/#/c/716298/216:54
openstackgerritMerged zuul/zuul-jobs master: Rename install-yarn to ensure-yarn for consistency  https://review.opendev.org/71669316:56
clarkbAJaeger: ^ fyi since you asked about reviews on those16:56
openstackgerritTobias Henkel proposed zuul/zuul master: Defer setting build pause to event queue  https://review.opendev.org/71293916:57
mordredcorvus: oh. blerg. gpg isn't installed16:58
*** jcapitao has quit IRC16:58
mordredcorvus: I feel like we fixed this somewhere recently16:58
mordredbut it certainly seems like the sort of thing that shoudl be installed in the excutor :)16:58
corvusmordred: we fixed it in virtual machine images16:58
mordredah16:58
*** hashar is now known as hasharAway16:58
clarkbmordred: gnupg2 or gnupg are the packages on debuntu. The first gets you the gpg2 aliases and is transitional16:59
AJaegerthanks, clarkb17:01
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: docker-install: workaround for centos-8 conflicts  https://review.opendev.org/70305317:03
openstackgerritMonty Taylor proposed zuul/zuul master: Add gear to the ansible envs  https://review.opendev.org/71761617:03
tobiashmordred: commented ^17:05
tristanCclarkb: thanks, i replied. no prefered merge order. I guess adding kubectl support to zuul-base-jobs can wait17:12
AJaegerzuul-jobs experts, there's a larger refactor of our jobs on how we handle repositories by mnaser, please review the stack starting at https://review.opendev.org/#/c/717507/217:13
clarkbtristanC: avass ok both chagnes lgtm but I didn't approve in case there was a strong opinion on ordering17:14
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: Add support for RedHat platforms on ensure-podman  https://review.opendev.org/71657817:14
*** rlandy is now known as rlandy|lunch17:14
mordredtobiash: ++17:15
mnaserAJaeger: thanks for that heads up for others :)17:15
mnaseri think that'll be an excellent cleanup/improvement17:15
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: tox: allow tox to be upgraded  https://review.opendev.org/69005717:18
openstackgerritMonty Taylor proposed zuul/zuul master: Add gear to the ansible envs  https://review.opendev.org/71761617:18
tobiashmordred, corvus: do we want that different refresh button in https://review.opendev.org/716305 ?17:19
openstackgerritMerged zuul/zuul-jobs master: Rename install-pdk-dependencies to ensure-pdk-dependencies for consistency  https://review.opendev.org/71669517:20
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: Add support for RedHat platforms on ensure-podman  https://review.opendev.org/71657817:20
mordredtobiash: I think so far folks have been ok with the change - but I think I'd be fine if someone knew how to make it look the other way (I don't)17:20
AJaegermnaser: did you test that with some depends-on?17:21
tobiashI don't either ;)17:21
mordredtobiash: :)17:21
*** gtema has joined #zuul17:23
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: Add support for RedHat platforms on ensure-podman  https://review.opendev.org/71657817:24
corvustobiash: yeah, i can live with it for now :)17:25
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: Add support for RedHat platforms on ensure-podman  https://review.opendev.org/71657817:27
*** gtema has quit IRC17:27
mnaserAJaeger: i think a lot of those are self testing inside zuul-jobs17:32
tobiashmordred: is there some issue with an opendev mirror? I've seen this failure more than once today: https://fceab12b277662d76f26-6eadae60e9a1b03c6789f42d51c17d3a.ssl.cf1.rackcdn.com/717602/1/gate/zuul-tox-remote/1fbac99/job-output.txt17:35
clarkbtobiash: thats downloading from node's source repo17:37
clarkbtobiash: not our mirror17:37
clarkbbasically the package is being downloaded from the mirror successfully then we install the package and that triggers a download of node itself. I've looked into making that more reliable but they hard code the source location in the package17:37
clarkbso there isn't really anything we can do unless we use a transparent proxy (which we don't due to potential for abuse)17:38
tobiashoh right17:38
tobiashand I guess it's hard to mirror that17:38
clarkbya the issue is in telling the package to download from somewhere else17:38
clarkbits been a while since I read the package source but I'm pretty sure it unconditionally does that download from its upstream17:38
mordredtobiash: we could put the install in a retry loop17:42
mordred(which would be stupid ugly - but might be a thing we want to consider?)17:42
mnaserfwiw im pretty sure ansible-lint encourages retries for remote operations17:47
mnaserthats what we do a lot of in OSA17:47
zbryes it does, but is not a genius, it will not detect network operations with shell :)17:48
mnaseroh that's inside shell17:48
mnaseri didnt dig into the context but yeah17:48
mnaserwe should probably just do that then imho.17:48
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: tox: allow tox to be upgraded  https://review.opendev.org/69005717:48
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: tox: allow tox to be upgraded  https://review.opendev.org/69005717:50
*** dpawlik has quit IRC17:51
mnaserensure-yarn on it's own fails because it requires ensure-nodejs.  should we document this or just add include_role ensure-nodejs ?17:51
mordredmnaser: echo "APT::Acquire::Retries \"3\";" > /etc/apt/apt.conf.d/80-retries17:51
* mordred learned something today17:51
mnaseri'm in favour of #2 because it follows a similar pattern in things like ensure-kubernetes which does ensure-docker for example17:51
mordredmnaser: #217:52
mordredmnaser: I would expect ensure-yarn or ensure-k8s to do what they need to do to ensure that17:52
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: Add support for RedHat platforms on ensure-podman  https://review.opendev.org/71657817:52
mnaseri'm terrified of putting a patch on top of the others which will make me carpet bomb zuul17:53
zbrnot sure if anyone cares but next version of ansible-lint will require ansible 2.8+ and py3+17:53
clarkbfwiw retrying network operations with shell is super easy. Use until17:53
clarkbuntil install_cmd; do bump_incr; if [ incr > threshold ] ; then break ; fi ; done17:54
mordredzbr: I do not think that will break anything here - but thanks!17:54
AJaegeravass: I pushed https://review.opendev.org/717813 and https://review.opendev.org/717814 https://review.opendev.org/717815 for the install -> ensure rename. Will you do the rest, please?17:54
zbrclarkb: i agree, rerunning entire shell block with until/retries is the nice and clean way of doing it.17:55
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: ensure-yarn: run ensure-nodejs before  https://review.opendev.org/71781717:55
mnaseroh i didnt do anything17:56
mnasermordred: ^ as discussed, and made it self testing17:56
*** saneax has quit IRC17:56
AJaegeravass: what about preparing a change to remove the now obsolete install- roles as well? We should mark that as WIP.17:57
AJaegeravass: so, your stack is approved - thanks!17:57
tristanCzuul-maint: a few zuul-operator changes are waiting for reviews, in particular these two are causing rebase issue and it would be nice to have them merged to avoid that: https://review.opendev.org/717342 and https://review.opendev.org/707193 . Thanks in advance!17:59
openstackgerritMerged zuul/zuul-jobs master: Rename install-if-python to ensure-if-python for consistency  https://review.opendev.org/71669818:02
*** rlandy|lunch is now known as rlandy18:03
avassAJaeger: I can see what I can do :)18:06
*** lseki_ has joined #zuul18:07
*** Miouge has quit IRC18:07
*** lseki has quit IRC18:08
*** lseki_ is now known as lseki18:08
openstackgerritTobias Henkel proposed zuul/zuul master: Retry nodeenv creation  https://review.opendev.org/71782018:08
zbrmnaser: clarkb done rebase https://review.opendev.org/#/c/716578/18:09
openstackgerritMohammed Naser proposed zuul/zuul-website master: WIP: switch website to Gatsby  https://review.opendev.org/71737118:09
*** Miouge has joined #zuul18:10
tobiashavass: in case you miss it because of the other +2, I commented on https://review.opendev.org/70887118:14
zbrdoes anyone know if become: "{{ item }}" works? i have the impression that it does not.18:15
AJaegerthanks, avass - and if you need help, please ask!18:16
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: ensure-repositories: Add role  https://review.opendev.org/71750718:17
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: ensure-nodejs: refactor to use ensure-repositories  https://review.opendev.org/71750818:17
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: ensure-yarn: refactor to use ensure-repositories  https://review.opendev.org/71750918:17
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: ensure-yarn: run ensure-nodejs before  https://review.opendev.org/71781718:17
mnaserAJaeger: lol.  the _only_ role that wasn't self testing, had been broken :)18:18
openstackgerritAlbin Vass proposed zuul/nodepool master: Use ensure-* roles  https://review.opendev.org/71782218:18
mnaseri swapped debian/redhat accidentally in the vars file.  found out in fixing ensure-yarn18:18
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: tox: allow tox to be upgraded  https://review.opendev.org/69005718:19
AJaegermnaser: ;)18:20
AJaegermnaser: can you add a self-test?18:20
mnaserAJaeger: if i add self-test, it will break.  therefore, the run ensure-nodejs fixes the role *and* adds self test (aka 717817)18:20
tristanCtobiash: thank you for the review, it's really appreciated!18:20
tobiashtristanC: no proble :)18:21
openstackgerritAlbin Vass proposed zuul/zuul-base-jobs master: Remove ssh key in base cleanup run.  https://review.opendev.org/70887118:22
openstackgerritAlbin Vass proposed zuul/zuul-base-jobs master: Remove ssh key in base cleanup run.  https://review.opendev.org/70887118:23
openstackgerritAlbin Vass proposed zuul/zuul-base-jobs master: Remove ssh key in base cleanup run.  https://review.opendev.org/71782718:24
openstackgerritTobias Henkel proposed zuul/zuul master: DNM: Test doc file comments  https://review.opendev.org/71668018:24
openstackgerritAlbin Vass proposed zuul/zuul-base-jobs master: Remove ssh key in base-test cleanup run.  https://review.opendev.org/70887118:25
openstackgerritAlbin Vass proposed zuul/zuul-base-jobs master: Remove ssh key in base cleanup run.  https://review.opendev.org/71782718:25
avasstobiash: fixed :)18:26
tobiashavass: thanks!18:26
openstackgerritMohammed Naser proposed zuul/zuul-website master: WIP: switch website to Gatsby  https://review.opendev.org/71737118:26
openstackgerritMerged zuul/zuul master: Make github file annotation levels configurable via zuul return  https://review.opendev.org/71117918:28
*** bhavikdbavishi has quit IRC18:28
*** bhavikdbavishi has joined #zuul18:29
openstackgerritMerged zuul/zuul master: Strip by default in tools/encrypt_secret  https://review.opendev.org/71450818:29
avassAJaeger: should I push changes to all repos on review.opendev.org for s/install/ensure/ or just stick to opendev/openstack/zuul?18:32
avassI mean, I guess everything needs to be updated if we're to remove the install- roles18:32
*** bhavikdbavishi has quit IRC18:34
openstackgerritMohammed Naser proposed zuul/zuul-website master: WIP: switch website to Gatsby  https://review.opendev.org/71737118:35
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: tox: allow tox to be upgraded  https://review.opendev.org/69005718:38
mnaseri always manage to find fun rabbit holes18:43
mnaserwe're installing nodejs 6.x by default in zuul-jobs from what it seems18:43
mnaserwhich has been EOL'd since april 201918:44
mnaserthe most recent current active LTS is nodejs 1018:45
mnaserwell, v10 and v12 are both in active LTS, v10 is the older, v12 is the newer18:45
mnaserv13 seems like it should be entering LTS in a few days ago but not listed on the website18:46
mnaserhttps://nodejs.org/en/about/releases/18:46
mnaserhow do we go about updating our roles? :\18:46
openstackgerritMerged zuul/zuul-operator master: Update dhall-kubernetes version to v4.0.0  https://review.opendev.org/71734218:47
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Add job_volumes CR spec attribute  https://review.opendev.org/70664218:47
mnaseri guess we can change the default behaviour and send a 2 week notice to ML18:51
tristanCclarkb: maybe you'd be interested in reviewing https://review.opendev.org/706639 (adding job volumes to the operator spec)18:58
avassAJaeger: there are still two changes for renaming the roles left: https://review.opendev.org/#/c/716689/ https://review.opendev.org/#/c/716752/19:02
avassmnaser: I think you accidentally changed the topic on the changes except for those two :)19:03
*** hasharAway is now known as hashar19:08
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-go to ensure-go for consistency  https://review.opendev.org/71668919:10
AJaegerconfig-core, please review https://review.opendev.org/#/c/716752/ and https://review.opendev.org/#/c/716689/19:17
openstackgerritMohammed Naser proposed zuul/zuul-website master: WIP: switch website to Gatsby  https://review.opendev.org/71737119:31
avassAJaeger: I'll have to go, so I'll continue working on changing the install- role sto ensure- roles tomorrow19:34
AJaegeravass: enjoy your evening - I'm signing off now as well...19:35
AJaegeravass: and thanks!19:35
avassno problem! bye :)19:35
openstackgerritMohammed Naser proposed zuul/zuul-website master: WIP: switch website to Gatsby  https://review.opendev.org/71737120:15
openstackgerritMohammed Naser proposed zuul/zuul-website master: WIP: switch website to Gatsby  https://review.opendev.org/71737120:25
mnaseravass: oops, i did do that, my bad20:28
openstackgerritMohammed Naser proposed zuul/zuul-website master: WIP: switch website to Gatsby  https://review.opendev.org/71737120:39
*** Guest40054 has quit IRC20:42
mnasercorvus, mordred: does zuul-preview run in opendev somewhere right?20:52
mnaseri ran into the issue with gatsby-ifying the website.  https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_2c9/717371/11/check/gatsby-build/2c9d101/npm/html/20:52
mnaserthing are breaking because gatsby needs/depends on root urls20:52
corvusmnaser: oh, there was a security problem with it and we shut it down; i need to fix it;  shouldn't be hard, just didn't have any pressing need.  maybe tomorrow?20:53
mnasercorvus: oh cool, that wfm, i will start "refactoring" the gatsby jobs into zuul/zuul-jobs in the meantime given that i have it successfully building20:58
mnaserit kinda sucks that it's not actually _technically_ successfully building though :\20:58
mnaserso i don't know if it's worth pushing that up there20:59
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Update attributes to camelCase  https://review.opendev.org/70719320:59
*** jkt has quit IRC21:04
*** jkt has joined #zuul21:04
openstackgerritMohammed Naser proposed zuul/zuul-website master: WIP: switch website to Gatsby  https://review.opendev.org/71737121:04
mordredmnaser: http://localhost:8000/blog/zuul-is-cool/ is pretty awesome21:08
mordredcorvus: grab mnaser's change, run "yarn install ; yarn develop" and check out the magic21:09
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: ensure-yarn: run ensure-nodejs before  https://review.opendev.org/71781721:10
mnasermordred: aha, it's actually based off our (vexxhost) blog so i kinda had to change it or otherwise it would have been articles from our site =P21:12
mordredmnaser: :)21:12
mnasermordred: i was thinking of splitting off into two changes, one adding the blog later, but eh.21:12
mordredmnaser: Zuul: "let vexxhost power your cloud journey!"21:12
mnaseroh no did we leave that there lol21:12
mnaseri git grep'd21:12
mordredmnaser: no - I just thought that would be funny21:12
mnasermordred: aha, okay, fair nuff :p21:15
mnasermordred: btw given you're part of the masochist group of people who do some js-y stuff, what are your thoughts about earlier wrt the node version we have in zuul-jobs ?21:22
mnasertl;dr: we're installing node_version: 6 which has been EOL a year ago, wonder if we should bump it and how we go about that21:22
mordredwe should _totally_ bump the default21:25
mordredprobably should send a mailing list message warning that we're going to bump it in 2 weeks21:26
mordredshould we default it to 10 or to 12?21:26
*** hashar has quit IRC21:33
fungii updated some of our stuff to nodejs 10.x because it's on ubuntu xenial still and didn't see any nodesource packages of 12.x for xenial21:37
fungibut that may not be a concern here21:37
mordredfungi: shouldn't be ... and one of the nice things here is that we don't need matching node versions on the running servers - only on build hosts21:38
mordredso we can build on bionic with upstream node 13 - and still deploy on xenial21:39
mordredwould be more of an issue if people were running node as a server21:39
fungimakes sense21:40
*** rlandy is now known as rlandy|afk21:50
*** y2kenny has joined #zuul21:57
openstackgerritJames E. Blair proposed zuul/zuul-preview master: Don't proxy urls that don't appear in the map  https://review.opendev.org/71787022:02
corvusmnaser: ^ that should get us moving again22:03
corvusmordred: ^22:03
*** rfolco has quit IRC22:03
openstackgerritMerged zuul/zuul-jobs master: Rename install-go to ensure-go for consistency  https://review.opendev.org/71668922:03
mordredcorvus: seems reasonable22:04
fungioh, yep, i had forgotten about all that22:05
mnasermordred: I think we can do 13?  It’s supppsed to have entered LTS a few days ago22:08
mnaserI assume that’s just not updated yet: https://nodejs.org/en/about/releases/22:08
mordredmnaser: if we're going to bump a default, seems like a good target22:10
fungicorvus: +2 on 717870 but also a question, just trying to make sure i grasp how that's working22:10
y2kennyDoes zuul scheduler automatically recover when ssh connection to gerrit timed out?22:15
mnasery2kenny: I believe it should22:18
y2kennyok22:18
fungiwe restart gerrit without restarting zuul, and it continues to work22:18
funginow depending on how it "times out" that behavior may vary22:19
clarkbit won't discover events that happened during the outage though22:19
y2kennyok22:19
y2kennyI missed some events and then turn out the connection timed out.22:19
y2kennyI ended up restarting the scheduler22:20
fungicertainly if the ssh session is disconnected via normal tcp/ip means (peer connection reset via rst packet, connection shutdown via fin/fin+ack or the like) then it works22:20
y2kennyI think I saw an  paramiko error22:20
y2kennyor the time out came from paramiko error22:21
fungiif the connection spontaneously goes away and zuul is never informed (maybe silently dropped by a firewall mid-session), then whether it finds out the connection is dead may depend on txp keepalive/dead peer detection settings in other places22:21
y2kennyI didn't see any indication about reconnecting so I wasn't sure if it recovered or not.22:21
fungibut those are more general challenges of maintaining long-lived tcp socket connections22:22
fungii don't think there's any ssh-level keepalive or dpd implemented, but that would depend on both paramiko(zuul side) and mina-sshd(gerrit side) i think22:23
y2kennyI see22:25
*** hashar has joined #zuul22:26
fungiy2kenny: aha! well-trodden ground: https://zuul-ci.org/docs/zuul/reference/drivers/gerrit.html#attr-%3Cgerrit%20connection%3E.keepalive22:26
y2kennycool!22:27
fungiby default it does ssh keepalive with an ssh protocol zero-content chunk every 60 seconds, looks like22:27
*** sshnaidm is now known as sshnaidm|afk22:28
y2kennyI will keep an eye on it and see if this happens again22:29
fungicool22:31
y2kennyfor the upload-log, is it correct to say that it is an ansible job being run on the log server to rsync from the executor to the log server?22:31
y2kennyI got the upload-log working last week but for some reason it stopped working (I got ssh: connect to host <log server> port 22: Connection refused)22:31
y2kennyeven though the other jobs (like add-fileserver and set zuul-log-path) seems to be working...22:32
y2kennyI can see the directories like the "zuul_log_path" are created... the failure seems to be on the rsync bit.22:33
y2kennyand I can rsync from my workstation to the log server using the ssh key that's encrypted... um...22:34
fungiwe stopped using it in favor of log storage, but i thought it ran rsync from the executor to push content to the logserver22:35
fungier, stopped using it in favor of object storage (in opendev's zuul deployment)22:35
y2kennyyea... this is my intermediate setup22:36
y2kennysomething simple and quick and it was working last week... I just need to traceback and figure out what I changed22:36
fungiso can rsync on the executor connect to the logserver?22:36
y2kennyI can ssh  but I haven't tried rsync... I should give that a try22:37
fungiconnection refused implies it's either ssh not listening on 22/tcp on the logserver or maybe some firewall rules not allowing it22:37
fungiwell, rsync seems to be trying to use ssh protocol there, so ssh seems like a reasonable test22:37
fungiif rsync is getting a connection refused on 22/tcp then presumably so would ssh (or anything trying to connect to that port for that matter)22:38
openstackgerritMerged zuul/zuul-preview master: Don't proxy urls that don't appear in the map  https://review.opendev.org/71787023:07
fungicorvus: thanks for the answer on 717870, that confirmed it's working the way i thought it was at least. i agree both solutions are roughly equivalent23:11
fungicorvus: also i think the example here basically matches what you wound up doing, so seems like as fine a choice as they offer: http://httpd.apache.org/docs/2.4/rewrite/rewritemap.html23:13
fungiyou merely took the additional step to make sure the default couldn't accidentally be shadowed by a local file23:14
fungiwhich seems like a reasonable precaution23:15
*** tosky has quit IRC23:28
*** hashar has quit IRC23:32

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!