| *** spsurya has joined #zuul | 00:16 | |
| *** openstack has joined #zuul | 01:20 | |
| *** ChanServ sets mode: +o openstack | 01:20 | |
| *** nguyenhai has quit IRC | 05:07 | |
| *** Rohaan has joined #zuul | 05:15 | |
| *** nguyenhai has joined #zuul | 05:15 | |
| *** nchakrab has joined #zuul | 05:27 | |
| *** nchakrab_ has joined #zuul | 05:28 | |
| *** nchakrab has quit IRC | 05:32 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: log-inventory: add missing zuul_info_dir prep https://review.openstack.org/577674 | 05:33 |
|---|---|---|
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Improve logging of GithubTriggerEvents https://review.openstack.org/575014 | 05:40 |
| *** nchakrab_ has quit IRC | 05:45 | |
| *** nchakrab has joined #zuul | 05:48 | |
| *** Rohaan has quit IRC | 05:49 | |
| *** Rohaan has joined #zuul | 05:49 | |
| *** yolanda has joined #zuul | 05:53 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement a Kubernetes driver https://review.openstack.org/535557 | 05:59 |
| *** nchakrab has quit IRC | 06:02 | |
| *** nchakrab has joined #zuul | 06:03 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement a Runc driver https://review.openstack.org/535556 | 06:03 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenShift resource provider https://review.openstack.org/570667 | 06:09 |
| goern | tristanC, github app send the webhook | 06:10 |
| goern | pabelanger, I saw that regex, but from what I see it does not support semver.org?! | 06:10 |
| goern | pabelanger, tristanC what consuses me is that I see there is a job created and finished for the release pipeline, but it doesne seem to do anything | 06:12 |
| goern | I'll get the executor logs... | 06:12 |
| tristanC | goern: where is that ref coming from? by default we set this one: https://softwarefactory-project.io/cgit/software-factory/sf-config/tree/ansible/roles/sf-repos/templates/config/zuul.d/_pipelines.yaml.j2#n249 | 06:13 |
| tristanC | and openstack-infra is using http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/pipelines.yaml#n146 | 06:13 |
| goern | tristanC, I took the regex from semver.org it should match?! | 06:14 |
| goern | https://regex101.com/ seems to match what I want with the regexp from https://github.com/thoth-station/zuul-test-config/blob/master/zuul.d/_pipelines.yaml#L125 | 06:16 |
| tristanC | isn't this one missing \. ? | 06:17 |
| goern | ja, there is always somewhere a \. missing in a regex :/ | 06:18 |
| goern | tristanC, which part do you mean? | 06:18 |
| tristanC | goern: ^refs/tags/[0-9a-zA-Z\.]+$ | 06:20 |
| goern | pabelanger, tristanC here is the executor log, it really says that much... https://paste.fedoraproject.org/paste/1bHBVn6tM77JdRuzrxly7A | 06:20 |
| tristanC | goern: scheduler log may be more interesting | 06:21 |
| goern | tristanC, please reload https://github.com/thoth-station/zuul-test-config/blob/master/zuul.d/_pipelines.yaml#L125 | 06:21 |
| tristanC | did you add a job to the release pipeline? e.g. in https://github.com/thoth-station/package-extract/blob/master/.zuul.yaml | 06:22 |
| goern | https://paste.fedoraproject.org/paste/Hp3bFb0orSotfZKcBS3TUA | 06:22 |
| goern | tristanC, ja https://github.com/thoth-station/srcops-testing/blob/master/.zuul.yaml#L12 | 06:22 |
| tristanC | goern: the scheduler creates a queue item for each event before it figures out if there is any job to run | 06:22 |
| goern | tristanC, what is that ID? Removed job b69b6d1571534382a61e2cbdbfa1b30b | 06:23 |
| goern | it does not relate to any other thing i see?! | 06:23 |
| tristanC | that seems like an internal id of the apscheduler library | 06:24 |
| goern | tristanC, ok | 06:25 |
| tristanC | is the scheduler running in DEBUG mode ? | 06:25 |
| goern | is gearman involved in a release job? I see an error there... https://paste.fedoraproject.org/paste/mwOG~rCb5CyEHJMavIUluw | 06:26 |
| goern | tristanC, nearly all levels in scheduler-log.yaml are DEBUG, alembic and gerrit is info | 06:27 |
| tristanC | goern: hum, that reminds me this discussion, i don't think you can attach secrets in project pipeline configuration: http://lists.zuul-ci.org/pipermail/zuul-discuss/2018-January/000215.html | 06:27 |
| tristanC | the scheduler and the executor are communicating through gearman jobs | 06:28 |
| tristanC | that ssl exception is benign iirc, and the timestamp doesn't match with your scheduler logs | 06:29 |
| goern | tristanC, right, thats why I ignored it, just wanted to make sure you know all :) | 06:29 |
| tristanC | goern: there should more logs after the "INFO zuul.Pipeline.local.release: Adding change" line | 06:30 |
| goern | so... the 215.html, I cant understand what Jim is saying... basically... I need to create a upload-pypi job with the secret attached and use that new job in my project's pipeline?! | 06:30 |
| tristanC | yes, put the secret in the same project as the playbooks | 06:31 |
| tristanC | e.g. in your config repo, copy those files: https://softwarefactory-project.io/cgit/config/tree/zuul.d/_jobs-pypi.yaml | 06:31 |
| goern | tristanC, is there a reason to use the _ in the file name? | 06:34 |
| tristanC | goern: nop | 06:34 |
| *** nchakrab has quit IRC | 06:36 | |
| *** nchakrab_ has joined #zuul | 06:36 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: executor: add support for resource connection type https://review.openstack.org/570668 | 06:56 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-base-jobs master: Add openshift-base job https://review.openstack.org/570669 | 07:01 |
| *** pcaruana has joined #zuul | 07:07 | |
| *** nchakrab_ has quit IRC | 07:10 | |
| *** nchakrab has joined #zuul | 07:27 | |
| *** gtema has joined #zuul | 07:30 | |
| *** hashar has joined #zuul | 07:34 | |
| *** jpena|off is now known as jpena | 07:41 | |
| *** toabctl has quit IRC | 07:58 | |
| *** Rohaan has quit IRC | 08:00 | |
| *** Rohaan has joined #zuul | 08:09 | |
| *** snapiri- has joined #zuul | 08:11 | |
| tristanC | corvus: can we hope for a nodepool tag today? | 08:13 |
| *** snapiri has quit IRC | 08:13 | |
| *** toabctl has joined #zuul | 08:29 | |
| *** sshnaidm|off is now known as sshnaidm | 08:39 | |
| *** Rohaan has quit IRC | 08:42 | |
| *** Rohaan has joined #zuul | 08:44 | |
| *** electrofelix has joined #zuul | 08:58 | |
| *** snapiri- has quit IRC | 08:59 | |
| *** snapiri has joined #zuul | 08:59 | |
| *** Rohaan___ has joined #zuul | 10:54 | |
| *** Rohaan has quit IRC | 10:55 | |
| *** sarad has joined #zuul | 10:59 | |
| *** jpena is now known as jpena|lunch | 11:00 | |
| *** nchakrab has quit IRC | 11:55 | |
| *** worldchat has joined #zuul | 11:55 | |
| *** nchakrab has joined #zuul | 11:58 | |
| *** jpena|lunch is now known as jpena | 12:00 | |
| *** dkranz has quit IRC | 12:02 | |
| *** dkranz has joined #zuul | 12:03 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: build-python-release: ensure LC_ALL is set to en_US.utf8 https://review.openstack.org/577800 | 12:13 |
| *** sarad has quit IRC | 12:16 | |
| goern | tristanC++ | 12:21 |
| goern | uh, no karma bot :/ | 12:21 |
| *** weshay_ is now known as weshay|ruck | 12:22 | |
| *** nchakrab has quit IRC | 12:29 | |
| *** nchakrab has joined #zuul | 12:30 | |
| *** rlandy has joined #zuul | 12:32 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix missing dequeue on synchronized pull request variant one https://review.openstack.org/577354 | 12:32 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix missing dequeue on synchronized pull request variant two https://review.openstack.org/577355 | 12:32 |
| *** nchakrab has quit IRC | 12:35 | |
| *** gtema has quit IRC | 12:45 | |
| *** nchakrab has joined #zuul | 12:48 | |
| goern | what does `2018-06-25 12:47:56,883 DEBUG zuul.layout: Pipeline variant <Job upload-test-pypi branches: {MatchAny:{BranchMatcher:master}} source: thoth-station/kebechet/.zuul.yaml@master#0> did not match <Tag 0x7ff258609b70 creates refs/tags/1.0.0-rc.4 on 7d02538c7f692d9dec3977a0009183c05f174a3e>` try to tell me? the tag '1.0.0-rc.4' is pointing to exactly that commit?! | 12:49 |
| fungi | <Tag 0x7ff258609b70 creates refs/tags/1.0.0-rc.4 on 7d02538c7f692d9dec3977a0009183c05f174a3e> is a specific reference object (note the enclosing <>) | 12:53 |
| fungi | the log entry is saying that object didn't match thoth-station/kebechet/.zuul.yaml branch master | 12:54 |
| fungi | for a variant of the upload-test-pypi job with a branch matcher of master | 12:54 |
| goern | got that, I created a tag via `git tag 1.0.0-rc.4; git push --tags` that should have a master and tag pointing at the same commit, no?! | 12:55 |
| fungi | i don't believe tag objects are expected to match any branch, so that's probably benign | 12:55 |
| *** Rohaan___ has quit IRC | 12:56 | |
| fungi | branches are git heads, tags are a different sort of git object | 12:56 |
| goern | confusing, the same pileline for a different project works with the same git commands | 12:57 |
| goern | I start reckoning why you called it zuul... https://i.kinja-img.com/gawker-media/image/upload/s--xhYIuHnC--/c_scale,f_auto,fl_progressive,q_80,w_800/v6ugnufcmyncheeaxug1.jpg | 12:58 |
| fungi | does a upload-test-pypi job variant in thoth-station/kebechet/.zuul.yaml include an explicit branch matcher for master? wondering whether this is an explicit or implicit matcher it's logging (i don't spend enough time staring at zuul debug logs) | 13:00 |
| goern | its the matcher from the pipeline not the project | 13:01 |
| goern | fungi, I think you asked about https://github.com/thoth-station/zuul-test-config/blob/master/zuul.d/_pipelines.yaml#L125 ? | 13:02 |
| fungi | the debug log entry you were asking about is specific to the configuration at thoth-station/kebechet/.zuul.yaml | 13:03 |
| goern | ja, and there is just jobs https://github.com/thoth-station/kebechet/blob/master/.zuul.yaml#L14 nothing that defines a matcher/regex | 13:04 |
| fungi | got it, so yeah must have been in reference to an implicit matcher (zuul attempts to match configuration based on branches on which it's found since you can have different zuul configuration on different branches of the same project) | 13:05 |
| goern | fungi, ja, thats what tristanC explained, so I made sure no other branch/tag/pr is having a .zuul.yaml | 13:06 |
| *** acozine1 has joined #zuul | 13:08 | |
| fungi | looking through our debug logs, we have lots of tag events and i don't see any mismatches logged like what you have there, so perhaps it is related somehow. where's your job definition for upload-test-pypi reside? | 13:11 |
| goern | fungi, https://github.com/thoth-station/zuul-test-jobs/blob/master/zuul.d/jobs-upload-test-pypi.yaml I spend the morning with tristanC debugging that stuff, generally it works, it seems to be related to that repo | 13:12 |
| fungi | yeah, nothing out of the ordinary in the job definition | 13:13 |
| goern | ja, I'm at newbie level :) | 13:13 |
| fungi | me too, apparently ;) | 13:14 |
| tobiash | goern: maybe the post-review flag is guilty: https://zuul-ci.org/docs/zuul/user/config.html#attr-pipeline.post-review | 13:14 |
| tobiash | goern: check if your pipeline is also marked as post-review | 13:14 |
| fungi | it is, i checked that earlier | 13:15 |
| fungi | and as he said, it's working for one of his other repos | 13:15 |
| goern | hmm... no branch protection on that kebechet repo, I will enable it and repush the tag | 13:15 |
| fungi | tobiash: post-review is set for the release pipeline at https://github.com/thoth-station/zuul-test-config/blob/master/zuul.d/_pipelines.yaml#L117 | 13:16 |
| tobiash | ah ok | 13:16 |
| goern | branch protection enabled doesnt change anything | 13:19 |
| fungi | i'm out of ideas, but more people should be waking up soon too | 13:20 |
| tobiash | goern: you need at least a reconfiguration to refresh branch protection in zuul | 13:21 |
| tobiash | goern: is your main.yaml public? | 13:21 |
| *** acozine1 has quit IRC | 13:21 | |
| fungi | i wonder whether toggling branch protection fires any events the scheduler can trigger on to know to perform a reconfiguration | 13:22 |
| tobiash | fungi: it doesn't... | 13:22 |
| tobiash | fungi: that was my plan last week until I noticed that there is no way of being notified currently | 13:23 |
| fungi | i see, so have to send a manual reconfigure signal to the scheduler or wait for another alteration which would trigger one i guess | 13:23 |
| tobiash | fungi: we can still improve that by rereading the branch protection on push events and pull_request events and trigger tenant reconfigurations if needed | 13:24 |
| fungi | okay, so eventually consistent still, but yes i can see some opportunities to shrink the "eventually" | 13:24 |
| tobiash | yes, a bad compromise but at least some improvement | 13:25 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Static driver: fix node registration https://review.openstack.org/577464 | 13:25 |
| Shrews | tobiash: ^^ decided to add some exception handling to that one | 13:26 |
| tobiash | Shrews: lgtm | 13:27 |
| tobiash | goern: is that your tenant config: https://github.com/thoth-station/zuul-test-config/blob/master/zuul/_main.yaml ? | 13:28 |
| tobiash | goern: that looks at least suspicious because it defines two tenants with the same name | 13:28 |
| tristanC | tobiash: all the zuul/*.yaml files are manually merged | 13:29 |
| tristanC | tobiash: goern: here is the final zuul tenant config: https://paste.fedoraproject.org/paste/iFdX2uFfWfQoACppimLo2Q | 13:30 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Add tenant yaml validation option to scheduler https://review.openstack.org/574265 | 13:32 |
| *** fbo has joined #zuul | 13:33 | |
| *** gtema has joined #zuul | 13:46 | |
| tobiash | goern, tristanC: in that case protected branches don't matter as they are not excluded | 13:47 |
| goern | right | 13:48 |
| goern | tristanC, that _main.yaml seems to be a left over... | 13:49 |
| tristanC | goern: i just pulled it from the configuration endpoint | 13:50 |
| tristanC | goern: it should be the one that is copied to /etc/zuul/main.yaml | 13:50 |
| goern | shall I copy that to the config repo?! | 13:51 |
| goern | /etc/zuul/zuul.yaml looks good | 13:51 |
| tristanC | goern: no need, it's generated from the resources, fwiw here is the /manage/v2/configurations/zuul code that software factory uses to generate the zuul tenant configuration: https://softwarefactory-project.io/cgit/software-factory/managesf/tree/managesf/controllers/api/v2/configurations.py#n49 | 13:52 |
| *** acozine1 has joined #zuul | 13:53 | |
| * goern needs to be afk for a few... hrs | 13:55 | |
| goern | thanks for the help... brb | 13:55 |
| *** acozine1 has quit IRC | 14:03 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Add pause function to executor https://review.openstack.org/577827 | 14:04 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Add pause function to executor https://review.openstack.org/577827 | 14:15 |
| Shrews | anyone want to push this static driver fix through? https://review.openstack.org/577464 | 14:24 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add pause function to executor https://review.openstack.org/577827 | 14:25 |
| *** nchakrab_ has joined #zuul | 14:27 | |
| *** nchakra__ has joined #zuul | 14:27 | |
| *** nchakrab has quit IRC | 14:29 | |
| *** nchakrab_ has quit IRC | 14:31 | |
| Shrews | oh yay, pm spam | 14:38 |
| *** acozine1 has joined #zuul | 14:47 | |
| *** acozine1 has quit IRC | 14:48 | |
| *** acozine1 has joined #zuul | 14:48 | |
| clarkb | tobiash: did you see that I had a preference for variant two? that was one of my original suggestions and I think it owrks here | 14:49 |
| clarkb | its a little hacky but less so than variant one I think | 14:49 |
| tobiash | clarkb: yes I agree | 14:50 |
| openstackgerrit | Merged openstack-infra/nodepool master: Static driver: fix node registration https://review.openstack.org/577464 | 14:51 |
| *** nchakra__ has quit IRC | 14:54 | |
| *** acozine1 has quit IRC | 15:00 | |
| *** worldchat has quit IRC | 15:02 | |
| *** pcaruana has quit IRC | 15:13 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Add tenant yaml validation option to scheduler https://review.openstack.org/574265 | 15:27 |
| *** electrofelix has quit IRC | 15:44 | |
| corvus | clarkb, fungi, Shrews, tobiash: what do folks think about a nodepool release? should we restart openstack's nodepool and burn current master in for a bit? | 16:05 |
| clarkb | ya I think we should burn it in a bit especially given the fun of the last set of changes we restarted for | 16:06 |
| Shrews | corvus: we def need a restart to get a bug fix. wanted to do it last week but was bit hampered by pain | 16:06 |
| fungi | sounds like a good idea | 16:06 |
| fungi | not the pain, but the burn-in | 16:07 |
| Shrews | corvus: do we want a release note for the static driver pre-register node change? | 16:08 |
| corvus | Shrews: you up for doing the restarts today? and yeah, i think that's note-worthy (desirable but probably not required) | 16:09 |
| Shrews | corvus: yep, can type a bit better today thx to drugs. will work on a release note first | 16:10 |
| corvus | tobiash: i agree with clarkb i like variant two better, but i think we need a note in the source code to prevent us from "fixing" it later :) | 16:14 |
| tobiash | corvus: I agree :) | 16:15 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add release note for static driver changes https://review.openstack.org/577858 | 16:16 |
| mordred | corvus: ++ to restart, burn in and release | 16:16 |
| mordred | corvus: whenever you feel in the mood to look at things, https://review.openstack.org/#/c/551989/ and https://review.openstack.org/#/c/572588 are ready to go | 16:21 |
| mordred | corvus: no rush, obviously | 16:21 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix missing dequeue on synchronized pull request variant two https://review.openstack.org/577355 | 16:23 |
| tobiash | corvus: added the note ^ | 16:23 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix missing dequeue on synchronized pull request https://review.openstack.org/577355 | 16:24 |
| tobiash | and updated the title now that we've chosen a variant | 16:24 |
| corvus | mordred: \o/ on my list :) | 16:25 |
| tobiash | corvus: ++ for restart, burn... | 16:26 |
| *** gtema has quit IRC | 16:41 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Add release note for static driver changes https://review.openstack.org/577858 | 16:43 |
| *** jpena is now known as jpena|off | 16:43 | |
| corvus | tobiash, tristanC, goern: is it possible that at the time of the "Pipeline variant ... did not match" message that there was more than one branch in kebechat? (including potentially a throwaway PR branch?) | 17:05 |
| corvus | (i'm beginning to suspect a bug where we automatically add branch matchers to pipeline variants where we should not) | 17:08 |
| corvus | one way to tell might be to see if there were any other "Pipeline variant ... did not match" messages nearby | 17:16 |
| goern | corvus, hmm, might be, there have been branches, but none with a .zuul.yaml | 17:29 |
| *** myoung is now known as myoung|biab | 17:33 | |
| corvus | goern: that's enough to change zuul's behavior. do you have access to more of those logs so we can see if there's another "Pipeline variant" message near the one you pasted? | 17:34 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Don't add implied branch matchers to project-pipeline variants https://review.openstack.org/577881 | 17:34 |
| corvus | goern: ^ i suspect that's the issue, but those messages would help confirm | 17:35 |
| corvus | goern: oh, actually... | 17:35 |
| corvus | goern: if there were other branches, but no .zuul.yaml files in them, then we won't see any other "Pipeline variant" messages. | 17:35 |
| corvus | i think the only way to confirm would be to go back to the last reconfiguration and see what branches zuul saw then. that's more difficult; may not be worth it. | 17:36 |
| corvus | tristanC, tobiash, fungi: https://review.openstack.org/577881 may be fix for issue goern reported | 17:37 |
| fungi | corvus: thanks! | 17:42 |
| pabelanger | q: have we discuss about adding a hard cap to the number of jobs an executor can run at once, outside of the load_multiplier / min_avail_mem settings? The use case is disk_limit_per_job is capped at X , but number of jobs X disk_limit_per_job is greater then local HDD and don't see a way to stop launching more jobs | 17:50 |
| goern | corvus, coolio! thx | 17:52 |
| corvus | pabelanger: are you saying that you are seeing your executors use up all their disks? | 17:53 |
| pabelanger | corvus: yes, single executor in this case | 17:54 |
| goern | oh man... `Project thoth-station/zuul-test-jobs not in pipeline <Pipeline post> for change <Branch 0x7f3ef05e13c8 refs/heads/master updated 7d380222da963403bab1182ac4dc60fe1b6fb935..df9057443fa8e165303aeace76fee94619eb692a>` :) | 17:54 |
| goern | not in pipeline? that means that zuul-test-jobs is not in /etc/zuul/main.yaml? | 17:55 |
| *** harlowja has joined #zuul | 17:55 | |
| goern | see https://paste.fedoraproject.org/paste/KEtyP6zAgHs2kVxntq~u-A | 17:56 |
| corvus | pabelanger: i don't believe we have a governor that looks at total available space. maybe write one? | 17:56 |
| corvus | pabelanger: (something that unregisters for work when free disk is < X %) | 17:56 |
| corvus | goern: that means there's no "post:" section in https://github.com/thoth-station/kebechet/blob/7d02538c7f692d9dec3977a0009183c05f174a3e/.zuul.yaml | 17:57 |
| pabelanger | corvus: yes, that would also work too | 17:58 |
| goern | corvus, thats ok, its a different repo. the config repo: https://github.com/thoth-station/zuul-test-jobs | 17:58 |
| goern | tristanC and I fiddled round with the shadow of the config repo I think | 17:58 |
| tobiash | pabelanger: if you want to implement a disk space governer you should stick that on top of 549275 | 17:59 |
| pabelanger | tobiash: sure | 18:00 |
| corvus | goern: ah, well if there's a pipeline definition for that repo (i don't see one), then that message just means there are no post jobs for it. | 18:03 |
| goern | maybe that is a software-factory related issue... I will talk to tristanC | 18:04 |
| corvus | goern: it also may not be an issue | 18:05 |
| corvus | goern: unless you wanted something to happen when you merged a change to that repo | 18:06 |
| goern | corvus, ja, that was the case... pushes to that repo reconfigured zuul ;) | 18:06 |
| corvus | goern: that's expected and should be automatic with no configuration needed. but, for example, the only actual post jobs we have on our "zuul-jobs" repos in openstack are jobs which build and publish the job documentation. since those repos don't appear to be set up to build documentation, i'm not sure there's any jobs which should be run under those circumstances. | 18:08 |
| goern | ah understood, than there is something broken :/ | 18:10 |
| corvus | goern: i don't know -- let me back up and ask for a more complete bug report about what did or did not happen as you expected :) | 18:11 |
| goern | and I think its related to what we did on the server | 18:11 |
| goern | tristanC, asked me to rsync the openstack-jobs repo and commit it back to the local git repo, I guess that is out of sync now with that we got on github | 18:12 |
| pabelanger | tobiash: how large is your zuul-merger by chance? | 18:13 |
| pabelanger | SpamapS: ^ | 18:14 |
| pabelanger | curious if anybody is using 1GB mergers | 18:14 |
| pabelanger | (ram) | 18:14 |
| SpamapS | pabelanger: I have no dedicated merger | 18:15 |
| SpamapS | I just have the one 16GB box that does all except zk | 18:15 |
| corvus | goern: i may not have all the necessary information -- but perhaps this will help -- the 'git driver' which is probably what is being used to talk to the 'local git repo' only checks for updates every 2 hours by default. manually triggering a full reconfiguration would force an update though. | 18:15 |
| pabelanger | SpamapS: ack, thanks | 18:15 |
| SpamapS | and merges in executor | 18:15 |
| goern | corvus, ack | 18:17 |
| tobiash | pabelanger: have to look it up, just a sec | 18:18 |
| pabelanger | I actually wonder if we could get away with 512Mb for a dedicated merger | 18:20 |
| pabelanger | looking at http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=1522&rra_id=all could be | 18:20 |
| pabelanger | will have to test and see | 18:20 |
| SpamapS | I think you could | 18:21 |
| SpamapS | executor itself is pretty light on RAM usage. | 18:22 |
| SpamapS | it's just the ansible's that use up RAM | 18:22 |
| tobiash | pabelanger: I have three merger pods running with a limit of 1500mb each | 18:22 |
| tobiash | they currently take 400, 600 and 1400 mb | 18:22 |
| clarkb | pabelanger: the complexity and size of the git merge itself will determine memory usage of that process for the most part | 18:23 |
| pabelanger | tobiash: thanks for the info | 18:26 |
| pabelanger | clarkb: agree | 18:26 |
| pabelanger | I think I'll ask for 1GB per new zuul-merger and see how that works | 18:26 |
| SpamapS | I've never charted that, but git's memory uage during a complex merge may not show up on aggregate graphs if it spikes up. | 18:30 |
| *** sshnaidm is now known as sshnaidm|afk | 18:41 | |
| *** pcaruana has joined #zuul | 19:27 | |
| *** yolanda has quit IRC | 20:01 | |
| *** yolanda has joined #zuul | 20:01 | |
| *** myoung|biab is now known as myoung | 20:17 | |
| *** pcaruana has quit IRC | 20:43 | |
| fungi | pabelanger: corvus: nice thing about the disk space % governor idea is that you don't need to fiddle with matching caps to various disks if your executors aren't homogeneous, and with hotplug disks and online resize you also don't need to make config adjustments and restart the executors when you give them more space | 20:47 |
| *** goern has quit IRC | 21:01 | |
| *** goern has joined #zuul | 21:01 | |
| *** EmilienM has joined #zuul | 21:03 | |
| *** tobiash has quit IRC | 21:03 | |
| *** EmilienM_PTO has quit IRC | 21:04 | |
| *** harlowja has quit IRC | 21:04 | |
| *** EmilienM has quit IRC | 21:04 | |
| *** EmilienM has joined #zuul | 21:04 | |
| *** tobiash has joined #zuul | 21:05 | |
| corvus | fungi: yep, the goal for the governors is to be as automatic as possible (let computers do the tuning so sysadmins can sit on the beach, or whatever it is we do) | 21:17 |
| *** hashar has quit IRC | 21:48 | |
| * fungi was actually sitting by the water, slow-cooking hot dogs and corn in the smoker | 21:51 | |
| fungi | unfortunately it's now time to do battle with the lawn | 21:51 |
| *** harlowja has joined #zuul | 22:14 | |
| *** yolanda_ has joined #zuul | 22:21 | |
| *** yolanda has quit IRC | 22:22 | |
| *** yolanda__ has joined #zuul | 22:28 | |
| *** yolanda_ has quit IRC | 22:31 | |
| tristanC | pabelanger: i think there is another issue regarding to the DiskAccountantDiskAccountant, it seems like it shouldn't be running for merge jobs | 22:46 |
| tristanC | where the disk_limit_per_job value is actually used for all the git data cached by merger job. Once the cache pass the limit, then the Accountant constantly printing info messages. | 22:48 |
| fungi | i seem to recall the zuul project had some boards in sb which were very slow to load. was https://storyboard.openstack.org/#!/board/53 one of them? | 23:21 |
| fungi | dhellmann just contributed a db indexing fix for storytags | 23:22 |
| fungi | several orders of magnitude speedup on the primary query | 23:22 |
| clarkb | nice and quick | 23:34 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
