Friday, 2019-12-20

mnaserdoes the executor do anything funky that can result in a big increase in disk space usage?00:15
mnaser(reported in df -h)(00:15
mnaserit seems like when jobs are running, the disk usage is increasing significantly on this node to the point of hitting 85%+ which makes k8s decide to evacuate it00:16
clarkbmnaser: yes each job has the repos oncifugred in it. if that is a shared fs it will hardlink though. also log collection happens on the executor00:18
mnaserclarkb: yeah in this case these are very lightweight jobs00:19
clarkbhow big are tge repos?00:19
clarkbyou might not be hardlinking00:19
mnaserdu says 18 megabytes00:20
*** rfolco|bbl has joined #zuul00:21
mnaseroh00:21
mnaseri think this might be totally unrelated00:21
mnaseri think nginx is buffering uploads on disk instead of sending straight through so its filling up the disk00:21
mnasergah00:21
*** jamesmcarthur has joined #zuul00:22
*** mattw4 has quit IRC00:23
*** jamesmcarthur has quit IRC00:27
*** jamesmcarthur has joined #zuul00:32
*** jamesmcarthur has quit IRC00:35
*** jamesmcarthur has joined #zuul00:50
*** jamesmcarthur has quit IRC00:54
*** sgw has quit IRC01:02
*** jamesmcarthur has joined #zuul01:22
*** jamesmcarthur has quit IRC01:27
*** jamesmcarthur has joined #zuul01:28
*** jamesmcarthur has quit IRC01:30
*** Goneri has quit IRC01:36
*** Goneri has joined #zuul01:38
*** avass has quit IRC01:39
*** rlandy has quit IRC01:51
*** sgw has joined #zuul02:28
*** jamesmcarthur has joined #zuul02:33
*** jamesmcarthur has quit IRC02:38
*** bhavikdbavishi has joined #zuul03:12
*** bhavikdbavishi1 has joined #zuul03:15
*** jamesmcarthur has joined #zuul03:16
*** bhavikdbavishi has quit IRC03:16
*** bhavikdbavishi1 is now known as bhavikdbavishi03:16
*** jamesmcarthur has quit IRC03:18
*** rfolco|bbl has quit IRC03:31
*** rfolco|bbl has joined #zuul03:32
*** rfolco|bbl has quit IRC03:42
*** decimuscorvinus has quit IRC05:05
*** decimuscorvinus has joined #zuul05:06
*** pabelanger has quit IRC05:19
*** pcaruana has joined #zuul05:52
*** jamesmcarthur has joined #zuul06:01
*** jamesmcarthur has quit IRC06:06
*** avass has joined #zuul07:07
*** themroc has joined #zuul07:20
*** saneax has joined #zuul07:48
*** jcapitao has joined #zuul07:58
*** avass has quit IRC08:06
*** tosky has joined #zuul08:19
*** themroc has quit IRC08:42
*** coldtom has joined #zuul08:51
*** jpena|off is now known as jpena08:57
*** yolanda has quit IRC09:10
*** mhu has joined #zuul09:27
mhuall I want for christmas is this change to be merged :) https://review.opendev.org/#/c/642408/09:27
*** yolanda has joined #zuul09:28
*** hashar has joined #zuul10:14
*** bhavikdbavishi has quit IRC10:24
*** avass has joined #zuul11:19
*** hashar has quit IRC11:56
*** rfolco|bbl has joined #zuul12:11
*** rfolco|bbl is now known as rfolco12:12
*** jcapitao is now known as jcapitao|afk12:13
*** hashar has joined #zuul12:37
*** jpena is now known as jpena|lunch12:38
*** rlandy has joined #zuul12:58
*** jcapitao|afk is now known as jcapitao13:01
*** armstrongs has joined #zuul13:24
*** armstrongs has quit IRC13:33
*** jpena|lunch is now known as jpena13:36
openstackgerritMohammed Naser proposed zuul/zuul master: Fix documentation typo  https://review.opendev.org/70017814:18
*** mhu has quit IRC14:50
*** EmilienM is now known as EvilienM15:16
*** pcaruana has quit IRC15:25
*** hashar has quit IRC15:55
*** jcapitao is now known as jcapitao|afk16:23
*** mattw4 has joined #zuul16:26
*** jcapitao|afk is now known as jcapitao16:32
*** Goneri has quit IRC16:41
corvushttps://gerrit-review.googlesource.com/admin/repos/q/filter:zuul17:19
corvusthe gerrit admins creates some repos for gerrit's upcoming zuul instance17:19
corvuscreated17:20
clarkbcorvus: and sounds like in the new year there will be gcp resources for hosting things?17:21
corvusi think they're available now17:21
corvusthere's one existing project; they're looking into making a second one (so we can have the same control/data plane separation we have in opendev)17:22
corvusbut since that's not strictly necessary, i think we can proceed17:22
corvusthe only real blocker is a gerrit account17:22
corvusbut i think i can create a k8s cluster and start spinning up an idle zuul17:23
corvusnow i have to pick a google cloud region17:23
SpamapSPick brazil!17:28
SpamapSOo or maybe Singapore17:28
corvusthe default is iowa17:28
*** jcapitao has quit IRC17:29
clarkbcorvus: I see17:35
*** Goneri has joined #zuul17:37
corvusmnaser: do you have your helm charts published anywhere (even to a temp location?)17:41
* SpamapS is down to collaborate on helm charts.. even more than operators17:44
corvusSpamapS: do you have some?17:45
clarkbcorvus: the issue with a gerrit account being google credentials, cla, and privs?17:45
clarkbdo they issue system accounts to bypass that google account business?17:45
corvusno, the opendev zuul is a "regular" google account that i made with our root email address, and confirmed using a mint mobile sim :|17:46
corvusso i'd like to personally avoid doing that for the gerrit project's zuul account; if that is what google and/or the maintainers want to do, that's fine, but i do think it should be at least one of the maintainers doing it in that case, not me17:47
clarkb++17:47
corvusre helm, i figure if someone has some helm charts, i can probably scale that down real quick to a minimial zuul install.  doesn't even need to work all that well, but if that's the first content we put in the zuul/ops repo, it'll be easier to collab going forward.17:48
*** jpena is now known as jpena|off17:51
*** jpena|off has quit IRC17:52
clarkbcorvus: gcp publishes cpu platforms available and defaults for their various regions17:52
clarkbcentral has some of the oldest available. Other regions might be more performant (though I have no real world evidence for that)17:53
corvusclarkb: oh interesting...well i went ahead and created a cluster in central, but it's easy to migrate or destroy/recreate if we want17:55
corvusit looks like i get to bootstrap perms on the repos too.  i'm in the "zuul-maintainers" group, which is self-owned and has the "owner" perm, but that's it.  the rest is greenfield.17:56
mnasercorvus: i have not yet unfortunately, i need to "de-clutter" them a bit because they have some secrets embedded right now17:57
mnaserso maybe i could get that done by monday?17:57
corvusmnaser: no rush; i'll see if i can learn enough helm to say "create a scheduler pod"17:57
mnasercorvus: if you want to create zuul/zuul-helm or zuul/helm -- i can have something there by monday17:58
mnaserthere are a few tight dependencies to some operators we use (i.e. mysql) that might need to be cleaned up17:58
corvusmnaser: good idea, i'll push up a repo creation change soon17:58
clarkbyou'll probably want to use https://cloud.google.com/sql/docs/ for that install17:59
clarkbat least I expect that will be the most googly version of the thing17:59
mnaserya we use presslabs/mysql-operator to run databases17:59
mnaserit's been nice, has backups and stuff built in, can upload to swift17:59
corvusclarkb: oh good point; though i might need to ask for a new service account for that.18:00
*** jamesmcarthur has joined #zuul18:05
mnaserok, i have a python script that can now entirely go through a charts folder, look at all the requirements, check if the exist in the designated repos, if they do, pass, if they don't, check if they exist in the local charts folder, if they exist and the requested version *matches* the one stored locally, build *the local one*, publish it to the helm registry, and then update the dependencies to point towards the helm18:09
mnaserregistry with the locally-built version18:09
mnaserthe only bits that aren't done are the actual "build" bits, its mostly a print statement, but the logic is there18:09
mnaserwhich means we can actually do speculative testing of helm charts with a buildset repo, which is proven to be WAY too complicated but the helm ecosystem has some questionable things wrt versioning18:09
mnasernow i just need to make it actaully build things and then make it an ansible module, then we can have a buildset helm repo that other jobs can use18:10
mnaserand some code to munge the requirements.yaml to point towards that repo *only* for the stuff we built locally18:11
corvusmnaser: do you need a repository to use helm?18:13
mnasercorvus: technically you could run it locally, but for dependencies, you need to point towards a repo (or there is a very 'frowned upon' way of pointing to things via file:///)18:13
mnaseri guess its similar to requirements.txt pointing at things which exist in pypi18:14
corvusmnaser: gotcha18:14
mnaserexcept its possible for a git repo to have multiple charts, which depend on each other -- which is why it becomes tricky18:14
mnaserif you say: split your work into two commits, one that bumps chart A version, then one that bumps chart B dependency, your chart A bump could effectively break chart B without you knowing18:15
corvusmnaser: i was batting around the idea of having zuul run argo to do the helm apply18:15
mnaserif you try and merge a change that bumps chart A version and dependency of chart B at the same time, helm will complain because it cant find chart A version :<18:16
mnasercorvus: https://fosdem.org/2020/schedule/event/safe_gated_and_integrated_gitops_for_kubernetes/18:16
mnaseri know someone who's doing it :p18:16
corvusmnaser: \o/18:17
mnaserand argocd has some cool concept which is 'application of applications'18:18
clarkbmnaser: doesn't that sort of defeat the purpose of having versioning? (not being able to make new versions)18:18
mnaserhttps://argoproj.github.io/argo-cd/operator-manual/declarative-setup/18:18
mnaserclarkb: so 'chart-tesing', the helm linters for the big helm/charts repo implies that you must bump version on every single change to a chart18:19
*** saneax has quit IRC18:20
mnaserso assuming i have these 3 charts: keystone 1.0.0, nova 1.0.0, openstack 3.0.0 (which requires keystone 1.0.0 and nova 1.0.0).  if i make a change to keystone, i'd bump it to keystone 1.0.1 -- however, i'm deploying openstack 3.0.0 -- so i need to bump openstack dependency to keystone 1.0.1 and then bump openstack version to 3.0.1)18:20
*** rfolco is now known as rfolco|dentist18:21
mnaseri risk releasing a keystone 1.0.1 that is *not* properly integrated with openstack 3.0.0 and i'll only find that out after i bump the requirement/dependency18:21
clarkband when you create keystone 1.0.1 that breaks the existing openstack 3.0.018:21
clarkbI guess the issue here is that 1.0.0 isn't findable by the tooling even though it exists in some form in the git repo history18:22
mnaserno because the 1.0.0 artifact still exists in the repo somewehre18:22
clarkbright, that is how I would expect it to work but you said "helm will complain because it cant find chart A version"18:22
mnaserright -- that is if i tried to bump both at the same time18:22
mnaser"will keystone 1.0.1 work with openstack 3.0.0" is not something that i can answer when testing 1.0.118:23
clarkbbeacuse you must create a 3.0.1 ?18:23
clarkbfun18:23
mnaseryep, because you must bump version when you make changes in helm (or at least that's what their linter wants)18:24
corvusso either speculative registries, or don't use their linter?18:24
mnaseryeah, that's pretty much where i'm at right now18:24
clarkbdo they allow you to specify version dep ranges?18:25
clarkbyou could reserve the last set of point versions for speculative testing if so18:25
mnaserhmm, they do, i think they try to enforce semver for the version field18:26
mnaserhmm18:26
mnaseras silly as it sounds, i could also go ahead and update the repo path to be relative file paths when running an 'integration' test18:27
mnaserand not touch it when running the linter18:27
mnaserand just set version to 0.0.0 for all charts18:27
mnasernone of it feels clean really18:27
corvusmnaser: i don't think that's a bad idea; i think similar ideas have come up wrt to rust, go, etc.18:28
corvus(also, not all that different from python siblings)18:28
mnaseroo18:30
mnaserso they use https://github.com/Masterminds/semver to do semver vhrvkd18:30
mnaserchecks*18:30
mnaserwhich seems like it might be happy accepting something like .beta18:31
corvusi guess that's an option as long as it doesn't conflict with an actual '.beta'?18:32
corvus\o/ https://gerrit-review.googlesource.com/c/zuul/ops/+/24913318:32
corvuspaladox: ^18:32
clarkbsemver needs a .speculative18:33
clarkbwhcih should only ever show up in a Ci system :)18:34
*** jamesmcarthur has quit IRC18:34
corvus++18:35
corvusclarkb: what's the status of the wikipedia page?18:35
*** jamesmcarthur has joined #zuul18:36
clarkbcorvus: I trailed off on the etherpad I started after realizing how difficult it is to write encyclopedia articles18:36
clarkbI think what I had was a reasonable enough outline but actually writing that type of material is not easy18:36
corvusclarkb: i think your outline was good, and probably had enough content for a starter article...18:37
clarkbperhaps. https://etherpad.openstack.org/p/zuul-wikipedia is that content18:38
clarkbjimmy isn't here but I can ask him if he wants to update his draft article request with that18:38
corvusclarkb, jamesmcarthur: i think it would be better to work from clarkb's outline rather than starting from the wikimedia one... i think it would be best to go ahead and drop the wikimedia copy, and start fresh with that.  i think then the main question is, is that enough to go ahead and request review, or do we want to fill it out more.18:38
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: Add basic Helm jobs  https://review.opendev.org/70022218:38
clarkboh jimmy is here18:38
mnaser^ so that basic job is working for me for now in terms of running basic linting18:39
clarkbcorvus: jamesmcarthur: I know the OSUOSL article got delisted recently as it was deemd not noteworthy enough. The editors seemed to base that on the articles cited. We may want to include news articles about zuul or other external docs as much as possible. However looking at the content I have there again I think it probably is complete enough as is if we add those citations18:41
*** jamesmcarthur has quit IRC18:41
corvusclarkb: that sounds important.  i started a todo at the top so we'll remember to do that18:42
clarkb++18:42
*** tosky has quit IRC18:45
corvusremote:   https://review.opendev.org/700224 Add zuul-helm repo18:46
corvusmnaser, clarkb: ^18:46
clarkbI went ahead and approved that since it seems likely no one else will be around before monday to do so18:47
corvussounds reasonable, thx18:47
mnaserok great, i'll push up small bits18:47
clarkband I won't be here monday :)18:47
mnaseryou can point argocd towards a git repo so we don't have to publish charts18:48
mnaserbut for better user experience we should probably publish them somewhere18:48
paladoxCorvus :O18:52
paladoxAwesome!18:52
*** jamesmcarthur has joined #zuul18:59
clarkbcorvus: I put a couple links under the todo lost. The first is definitely a good one I think18:59
mnasercorvus: how do you feel about a reconfigure_on_tenant_config_change option in zuul?18:59
mnaserright now these charts restart the entire world if the tenant config file changes19:00
mnaseri can write a small sidecar, or i can implement this right into zuul.. both work..19:00
clarkbmnaser: why not trigger the config update within zuul as per usual?19:01
clarkbor maybe that is what you mean19:01
corvusmnaser: did you see the smart reconfig option that just landed?19:01
mnaserclarkb: yeah, i am currently restaring the entire pod on change instead of triggering a config update19:01
corvusmnaser: https://zuul-ci.org/docs/zuul/releasenotes.html#new-features19:02
mnasercorvus: yeah, something still needs to trigger it though afaik19:02
mnaserand with helm, there isn't an easy way of saying "run X on every time value Y changes" -- we can write a job but it can get pretty iffy19:02
corvusmnaser: ah, so like inotify on the file in zuul and run smart-reconfigure when it changes?19:02
mnaseryep19:02
corvusmnaser: i think that is a fine thing to do (with an enable or disable option; i guess it should be disabled by default, at least at first).  we could also have an api endpoint for that (if we don't already).  those options are not exclusive.  :)19:03
mnaserok cool, i can try and work on something inside zuul for autoreload, because that's a big drawback right now, a full restart of $world everytime the tenant config changes19:04
mnaserbtw if anyone wants to pad stats = https://review.opendev.org/#/c/700178/19:05
corvusgibbidygubbidy.19:06
clarkbmnaser: could you say "always run X" then have X decide if Z action should be taken based on Y state?19:07
clarkbthat said having an inotify watch in zuul for its configs is probably fine19:07
clarkb(inotify gets slow when watching many files but zuul's tenant config tends to not be many files)19:07
mnaserclarkb: yeah, that would be a sidecar that watches the configmap and it can do things after, but it would also involve writing code19:07
mnasermatter of "do we want it to live in-tree or not" :p19:08
mnaserwell that turned out to be simpler/easier than expected19:23
mnaseri just have to write up the tests19:23
*** jamesmcarthur has quit IRC19:25
*** openstackgerrit has quit IRC19:27
*** openstackgerrit has joined #zuul19:41
openstackgerritMerged zuul/zuul master: authentication config: add optional max_validity_time, skew  https://review.opendev.org/64240819:41
corvusi hope mhu has a merry christmas ^ :)19:41
*** rfolco|dentist has quit IRC19:48
*** rfolco has joined #zuul19:49
openstackgerritMohammed Naser proposed zuul/zuul master: Add auto reconfigure option  https://review.opendev.org/70023519:50
mnaser^ i need some help in figuring out how/where to test this best19:51
* mnaser was quickly overwhelmed inside test_scheduler.py :\19:51
corvusit's a bit of a catch-all19:53
clarkbtest_live_reconfiguration and the other ~2 tests like it are probably closest19:54
mnaserthe zuultestcase seems to setup fixtures in advance but isn't exactly meant for having a config file fixture swept under it19:54
clarkbbut it drives that directly via sched.reconfigure19:55
mnaserright but to properly test this, i would have to modify the fixture file and ensure that reconfigure is being called19:57
corvusmnaser: see TestSchedulerSmartReconfiguration19:57
corvusmnaser: it uses newTenantConfig to swap out the tenant config  file19:57
openstackgerritMerged zuul/zuul master: Fix documentation typo  https://review.opendev.org/70017819:58
corvusmnaser: and the "config_file" class attribute can be used to specify a different zuul.ini file (with the auto_reconfigure option set to true19:59
mnasercorvus: hm, swapping out the file would probably still not hit a "modify" event though with inotify, would it19:59
mnasernewTenantConfig seem to write things in a new namedtemporaryfile20:00
corvusmnaser: oh you're write, that method changes the yep that20:00
mnaseri guess the only thing would be create my own fixture file, and edit it during tests, then restore it in a clean up20:01
mnaseri guess the only alternative would be to manually fire an inotify event with a mock20:02
mnaserand make sure i ran config there20:02
clarkbmnaser: testtools has a tmpfile fixture20:04
clarkbI would use that, splat a zuul config into it and use it.20:04
clarkband you might use a generic config fixture to start, copy the contents of that file to tmpfile, switch config file to the tmpfile, then edit tmpfile20:05
corvus++20:05
corvusi think you could probably just copy the first part of the code in newtenantconfig for that20:05
corvusthat's step 1, then step 2 is just copy without creating the new tmpfile20:06
mnaserso actually i technically could use newtenantconfig right off the bat20:06
mnasermanipulate the file which is in self.config (which is a temp path)20:06
mnaserand then it will clean up after me20:06
corvusoh yep20:07
mnaseri guess ill just have to test this in gate20:27
mnaseri guess ill just have to test this in gate20:27
mnasercause it seems to want zookeeper running20:27
clarkbmnaser: ya you need a zk. I run it out of the tarball they publish20:27
clarkbzookeeper/bin/zkServer.sh start|stop20:28
openstackgerritMohammed Naser proposed zuul/zuul master: Add auto reconfigure option  https://review.opendev.org/70023520:29
mnaseri have to run but i guess i can throw that up and we'll see how it all pans out..20:29
clarkbthat also works20:29
openstackgerritMohammed Naser proposed zuul/zuul master: Add auto reconfigure option  https://review.opendev.org/70023520:30
openstackgerritMohammed Naser proposed zuul/zuul master: Add auto reconfigure option  https://review.opendev.org/70023520:31
*** armstrongs has joined #zuul20:32
*** armstrongs has quit IRC20:41
*** rlandy has quit IRC21:13
openstackgerritJames E. Blair proposed zuul/zuul master: WIP: auto reconfig updates  https://review.opendev.org/70024921:49
corvusmnaser: that causes the full option to work21:49
corvusmnaser: smart needs a slightly different method of testing (since it noops on just adding a blank line)21:50
corvusmnaser: but also, i think we need to change it to watch the whole directory and pick out the files it needs -- since an in place atomic rename ("mv old new") doesn't show up as a MODIFY, but rather a MOVED_TO21:51
corvusmnaser: (also, we should actually use MODIFY_CLOSE rather than MODIFY)21:51
*** dustinc is now known as dustinc_inandout22:17
corvusoh, we might be able to watch the file for DELETE_SELF to catch that case22:30
*** EvilienM is now known as EmilienM22:36
*** avass has quit IRC22:42
*** jamesmcarthur has joined #zuul23:03
*** mattw4 has quit IRC23:10
openstackgerritJames E. Blair proposed zuul/zuul master: Add auto reconfigure option  https://review.opendev.org/70023523:13
openstackgerritJames E. Blair proposed zuul/zuul master: Add auto reconfigure option  https://review.opendev.org/70023523:17
corvusmnaser: ^ i think that should be gtg.  let me know what you think.23:18
*** jamesmcarthur has quit IRC23:39
*** Goneri has quit IRC23:40
*** nhicher has quit IRC23:42
clarkbcorvus: mnaser left some comments on ^23:43
clarkbI think there is a small race23:43
clarkbchances of hitting it in the real world seem slim particularly since we are talking about IO here, but...23:44
clarkbseems I've learned that if they exist we will eventually hit them23:44
corvusclarkb: well, we need to account for the file path changing23:45
*** jamesmcarthur has joined #zuul23:45
clarkbah so we need to check the old watch against the new one?23:46
clarkbseems we can still do that in an add first rm after order. We may have to track more state though23:46
corvusclarkb: yep23:47
corvusclarkb: if you want to handle that, you should go ahead and leave a -1 on that, i won't be able to fix that immediately (maybe mnaser can when he returns)23:48
clarkbI can't either. I'm about to pop out to go visit the parents23:48
clarkband I expect next week to be very slow23:49
clarkbIf I have chunk of time to poke at it I'll try23:49

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!