| *** JasonCL has joined #zuul | 00:13 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Remove .json suffix from web routes https://review.openstack.org/537010 | 00:16 |
|---|---|---|
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Port per-change status to zuul-web https://review.openstack.org/535903 | 00:16 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Re-enable test of returning 404 on unknown tenant https://review.openstack.org/545644 | 00:16 |
| *** JasonCL has quit IRC | 00:18 | |
| *** JasonCL has joined #zuul | 00:50 | |
| *** JasonCL has quit IRC | 00:54 | |
| *** JasonCL has joined #zuul | 01:31 | |
| *** JasonCL has quit IRC | 01:40 | |
| *** Wei_Liu has joined #zuul | 01:51 | |
| *** Wei_Liu has quit IRC | 01:56 | |
| *** JasonCL has joined #zuul | 04:18 | |
| *** JasonCL has quit IRC | 04:23 | |
| *** JasonCL has joined #zuul | 06:57 | |
| *** JasonCL has quit IRC | 07:02 | |
| *** JasonCL has joined #zuul | 07:09 | |
| *** JasonCL has quit IRC | 07:14 | |
| *** JasonCL has joined #zuul | 07:30 | |
| *** JasonCL has quit IRC | 07:35 | |
| *** JasonCL has joined #zuul | 08:09 | |
| *** JasonCL has quit IRC | 08:14 | |
| *** JasonCL has joined #zuul | 08:44 | |
| *** JasonCL has quit IRC | 08:49 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Port per-change status to zuul-web https://review.openstack.org/535903 | 09:09 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Re-enable test of returning 404 on unknown tenant https://review.openstack.org/545644 | 09:09 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add /info and /{tenant}/info route to zuul-web https://review.openstack.org/537011 | 09:09 |
| *** JasonCL has joined #zuul | 09:27 | |
| *** JasonCL has quit IRC | 09:31 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add /info and /{tenant}/info route to zuul-web https://review.openstack.org/537011 | 09:42 |
| mordred | corvus, tobiash: those four ^^ should be ready to go now | 09:43 |
| *** JasonCL has joined #zuul | 10:22 | |
| *** sshnaidm|off has quit IRC | 10:26 | |
| *** JasonCL has quit IRC | 10:27 | |
| *** sshnaidm|off has joined #zuul | 10:30 | |
| mordred | tristanC: ^^ (you too) | 10:31 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: WIP Use yarn and webpack to manage zuul-web javascript https://review.openstack.org/538099 | 10:47 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add babel transpiling enabling use of ES6 features https://review.openstack.org/538125 | 10:47 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add StandardJS eslint config for linting https://review.openstack.org/538126 | 10:47 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add bundle analysis to the lint target https://review.openstack.org/538129 | 10:47 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: WIP Inject url endpoint information https://review.openstack.org/538130 | 10:47 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Make bundle of built web content https://review.openstack.org/538131 | 10:47 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Remove use strict https://review.openstack.org/538132 | 10:47 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/jobs/{job_name} route https://review.openstack.org/535545 | 10:47 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add jobs graph rendering https://review.openstack.org/537869 | 10:47 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: /{tenant}/projects routes https://review.openstack.org/537870 | 10:47 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add project pipeline rendering https://review.openstack.org/537871 | 10:47 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/pipelines route https://review.openstack.org/541521 | 10:47 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Fix indentation and renable the eslint rule https://review.openstack.org/545671 | 10:47 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: WIP Use yarn and webpack to manage zuul-web javascript https://review.openstack.org/538099 | 11:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add babel transpiling enabling use of ES6 features https://review.openstack.org/538125 | 11:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add StandardJS eslint config for linting https://review.openstack.org/538126 | 11:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add bundle analysis to the lint target https://review.openstack.org/538129 | 11:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: WIP Inject url endpoint information https://review.openstack.org/538130 | 11:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Make bundle of built web content https://review.openstack.org/538131 | 11:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Remove use strict https://review.openstack.org/538132 | 11:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/jobs/{job_name} route https://review.openstack.org/535545 | 11:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add jobs graph rendering https://review.openstack.org/537869 | 11:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: /{tenant}/projects routes https://review.openstack.org/537870 | 11:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add project pipeline rendering https://review.openstack.org/537871 | 11:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/pipelines route https://review.openstack.org/541521 | 11:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Fix indentation and renable the eslint rule https://review.openstack.org/545671 | 11:38 |
| *** elyezer has quit IRC | 12:15 | |
| *** JasonCL has joined #zuul | 12:33 | |
| *** JasonCL has quit IRC | 12:37 | |
| *** JasonCL has joined #zuul | 12:38 | |
| *** JasonCL has quit IRC | 13:09 | |
| *** JasonCL has joined #zuul | 13:10 | |
| *** JasonCL has quit IRC | 13:14 | |
| *** JasonCL has joined #zuul | 13:41 | |
| *** JasonCL has quit IRC | 13:46 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: WIP Use yarn and webpack to manage zuul-web javascript https://review.openstack.org/538099 | 14:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add babel transpiling enabling use of ES6 features https://review.openstack.org/538125 | 14:50 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add StandardJS eslint config for linting https://review.openstack.org/538126 | 14:50 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add bundle analysis to the lint target https://review.openstack.org/538129 | 14:50 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: WIP Inject url endpoint information https://review.openstack.org/538130 | 14:50 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Make bundle of built web content https://review.openstack.org/538131 | 14:50 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Remove use strict https://review.openstack.org/538132 | 14:50 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/jobs/{job_name} route https://review.openstack.org/535545 | 14:50 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add jobs graph rendering https://review.openstack.org/537869 | 14:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: /{tenant}/projects routes https://review.openstack.org/537870 | 14:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add project pipeline rendering https://review.openstack.org/537871 | 14:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/pipelines route https://review.openstack.org/541521 | 14:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Fix indentation and renable the eslint rule https://review.openstack.org/545671 | 14:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: WIP Use yarn and webpack to manage zuul-web javascript https://review.openstack.org/538099 | 14:51 |
| *** JasonCL has joined #zuul | 15:28 | |
| *** JasonCL has quit IRC | 15:33 | |
| *** elyezer has joined #zuul | 15:35 | |
| -openstackstatus- NOTICE: Zuul has been restarted and queues were saved. However, patches uploaded after 14:40UTC may have been missed. Please recheck your patchsets where needed. | 15:56 | |
| *** JasonCL has joined #zuul | 16:05 | |
| *** JasonCL has quit IRC | 16:10 | |
| openstackgerrit | Merged openstack-infra/zuul master: Make PipelineParser a regular class https://review.openstack.org/545151 | 16:17 |
| *** bhavik1 has joined #zuul | 16:20 | |
| *** bhavik1 has quit IRC | 16:25 | |
| fungi | 545448 (pcontext stuff) is one of those deep changes which makes me glad we have such extensive testing | 16:26 |
| pabelanger | yah, I actually fine it a little easier to understand now too | 16:28 |
| pabelanger | but ++ to testing, still complicated stuff | 16:28 |
| AJaeger | so, whole stack is approved now - with exception of the bottom which is https://review.openstack.org/#/c/545610 | 16:29 |
| fungi | yeah, i just caught up through that one and +2'd it | 16:31 |
| *** JasonCL has joined #zuul | 16:31 | |
| *** rlandy has joined #zuul | 16:34 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: WIP Use yarn and webpack to manage zuul-web javascript https://review.openstack.org/538099 | 16:40 |
| *** JasonCL has quit IRC | 16:45 | |
| *** JasonCL has joined #zuul | 17:05 | |
| *** JasonCL has quit IRC | 17:11 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add babel transpiling enabling use of ES6 features https://review.openstack.org/538125 | 17:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add StandardJS eslint config for linting https://review.openstack.org/538126 | 17:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add bundle analysis to the lint target https://review.openstack.org/538129 | 17:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: WIP Inject url endpoint information https://review.openstack.org/538130 | 17:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Make bundle of built web content https://review.openstack.org/538131 | 17:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Remove use strict https://review.openstack.org/538132 | 17:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/jobs/{job_name} route https://review.openstack.org/535545 | 17:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add jobs graph rendering https://review.openstack.org/537869 | 17:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: /{tenant}/projects routes https://review.openstack.org/537870 | 17:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add project pipeline rendering https://review.openstack.org/537871 | 17:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/pipelines route https://review.openstack.org/541521 | 17:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Fix indentation and renable the eslint rule https://review.openstack.org/545671 | 17:11 |
| andreaf | mordred, corvus, tobiash: uhm I've never seen this kind of failure before http://logs.openstack.org/96/545696/1/check/devstack-multinode/0061572/job-output.txt.gz#_2018-02-18_16_46_30_863505 | 17:14 |
| andreaf | the class in the log is log_ara.CallbackModule - could it be that ara causes a failure *during* a job run? | 17:15 |
| andreaf | Also the error Not a boolean value: 'True' is not obviously related in any way I can see to my patch | 17:16 |
| andreaf | but my patch does use the latest zuul feature of setting group_vars in the job: https://review.openstack.org/#/c/545696/1 | 17:16 |
| mordred | andreaf: thebool issue is just a warning | 17:42 |
| mordred | andreaf: the ansible parse error is the main issue - except I think dmsimard and I found out that exit code that indicates parse error can also sometimes be related to connection issues | 17:43 |
| mordred | there's a connection issue in the middle of the output, so I *think* that's what went wrong | 17:44 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Reorgnize the output bundle split https://review.openstack.org/545706 | 17:49 |
| mordred | tristanC: ^^ that's in response to your review quesiton | 17:49 |
| andreaf | mordred: oh ok - when zuul does a retry of pre, it uses fresh nodes right? I hit a retry limit which makes me think a connection issue three times in a row is not so probabable | 17:55 |
| andreaf | mordred: I can try a recheck anyways | 17:55 |
| andreaf | fungi pabelanger mordred adding stable-queens checks to d-g https://review.openstack.org/#/c/545144/ | 18:00 |
| mnaser | hmm, im not sure if this is a zuul bug or expected behavior - i had a stack of 4 changes, i was asked to squash one of the changes, so i squashed them making the stack now 3 changes. gerrit reported 'change was rebased', zuul just queued the jobs (without removing the -1 however) | 18:02 |
| mnaser | so right now the rebased change is going though check queue, but zuul didn't remove the -1 | 18:03 |
| pabelanger | 2018-02-18 14:30:03,910 WARNING kazoo.client: Connection dropped: socket connection broken | 18:32 |
| pabelanger | andreaf: corvus: tobiash: ^zuul lost access to zookeeper for 545696, which deleted nodes under running jobs. you can see SSH errors in the log above | 18:33 |
| pabelanger | looks like it happen a few times today | 18:33 |
| pabelanger | http://grafana.openstack.org/dashboard/db/nodepool?from=1518971013089&to=1518973259706 shows the excitement | 18:34 |
| pabelanger | in-use nodes | 18:34 |
| fungi | dmsimard: zuul is configured to not remove any existing verify vote when enqueuing into openstack-infra's check pipeline | 18:42 |
| *** jimi|ansible has quit IRC | 18:43 | |
| fungi | zuul _can_ be configured to unset votes when enqueuing, that particular deployment just isn't configured to do it in check (but does when enqueuing into its gate pipeline) | 18:44 |
| mordred | mnaser: ^^ I think fungi was replying to you | 18:47 |
| fungi | oh, indeed i was | 18:48 |
| fungi | sorry dmsimard! | 18:48 |
| mordred | mnaser: also - gerrit's rebase-detection will also cause votes to stick around after a new patchset | 18:48 |
| fungi | right, which has nothing to do with zuul | 18:48 |
| mnaser | ah, i could swear when proposing an update, zuul would drop it's own -1 | 18:48 |
| mordred | yah | 18:48 |
| mordred | mnaser: it's usually gerrit that clears those out | 18:48 |
| mnaser | okay i see | 18:48 |
| mnaser | so because gerrit is the one who decided it was just a rebase, it didnt remove them | 18:49 |
| mordred | mnaser: however, when a change shifts into gate, zuul *does* remove the original +1 | 18:49 |
| mordred | mnaser: exactly | 18:49 |
| mnaser | so it isn't zuul which removes it's own -1 when a patchset updates, okay i see | 18:49 |
| mnaser | til | 18:49 |
| fungi | mnaser: gerrit used to clear verify votes on rebase, seems like 2.13 may have subtly changed some behavior there | 18:49 |
| mnaser | fungi: generally it doesn't remove it, i think because this was a squashed rebase, it might have affected it | 18:49 |
| mnaser | i guess it figured it was squashing the change below it so it was "still ok" | 18:49 |
| fungi | perhaps, but earlier (<=2.11?) gerrit used to clear verify with our config if so much as the commit id changed, which any alteration of its parents should cause | 18:51 |
| openstackgerrit | Merged openstack-infra/zuul master: Make SemaphoreParser a regular class https://review.openstack.org/545152 | 19:02 |
| openstackgerrit | Merged openstack-infra/zuul master: Make TenantParser a regular class https://review.openstack.org/545153 | 19:02 |
| openstackgerrit | Merged openstack-infra/zuul master: Add ConfigLoader initializer https://review.openstack.org/545180 | 19:02 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add StandardJS eslint config for linting https://review.openstack.org/538126 | 19:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add bundle analysis to the lint target https://review.openstack.org/538129 | 19:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: WIP Inject url endpoint information https://review.openstack.org/538130 | 19:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Make bundle of built web content https://review.openstack.org/538131 | 19:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Remove use strict https://review.openstack.org/538132 | 19:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/jobs/{job_name} route https://review.openstack.org/535545 | 19:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add jobs graph rendering https://review.openstack.org/537869 | 19:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: /{tenant}/projects routes https://review.openstack.org/537870 | 19:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add project pipeline rendering https://review.openstack.org/537871 | 19:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/pipelines route https://review.openstack.org/541521 | 19:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Fix indentation and renable the eslint rule https://review.openstack.org/545671 | 19:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Reorgnize the output bundle split https://review.openstack.org/545706 | 19:06 |
| openstackgerrit | Merged openstack-infra/zuul master: Remove tenant argument from UnparsedTenantConfig https://review.openstack.org/545181 | 19:09 |
| openstackgerrit | Merged openstack-infra/zuul master: Add ParseContext class https://review.openstack.org/545448 | 19:09 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix typo in PragmaParser https://review.openstack.org/545449 | 19:09 |
| openstackgerrit | Merged openstack-infra/zuul master: Remove project.unparsed_config https://review.openstack.org/545485 | 19:09 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Remove .json suffix from web routes https://review.openstack.org/537010 | 19:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Port per-change status to zuul-web https://review.openstack.org/535903 | 19:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Re-enable test of returning 404 on unknown tenant https://review.openstack.org/545644 | 19:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add /info and /{tenant}/info route to zuul-web https://review.openstack.org/537011 | 19:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: WIP Use yarn and webpack to manage zuul-web javascript https://review.openstack.org/538099 | 19:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add babel transpiling enabling use of ES6 features https://review.openstack.org/538125 | 19:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add StandardJS eslint config for linting https://review.openstack.org/538126 | 19:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add bundle analysis to the lint target https://review.openstack.org/538129 | 19:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: WIP Inject url endpoint information https://review.openstack.org/538130 | 19:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Make bundle of built web content https://review.openstack.org/538131 | 19:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Remove use strict https://review.openstack.org/538132 | 19:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/jobs/{job_name} route https://review.openstack.org/535545 | 19:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add jobs graph rendering https://review.openstack.org/537869 | 19:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: /{tenant}/projects routes https://review.openstack.org/537870 | 19:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add project pipeline rendering https://review.openstack.org/537871 | 19:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: add /{tenant}/pipelines route https://review.openstack.org/541521 | 19:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Fix indentation and renable the eslint rule https://review.openstack.org/545671 | 19:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Reorgnize the output bundle split https://review.openstack.org/545706 | 19:24 |
| *** kmalloc has quit IRC | 19:48 | |
| *** elyezer has quit IRC | 20:21 | |
| *** elyezer has joined #zuul | 20:34 | |
| *** sshnaidm has joined #zuul | 20:53 | |
| *** sshnaidm has quit IRC | 20:53 | |
| SpamapS | Hmmm. I need to figure out how to do something a little weird. | 21:28 |
| SpamapS | I have a local repo that I use off 'master', but I want to use another project in a specific branch ('godaddy'). I've been using that other project with override-checkout in required-projects, but now I want to start running the same job when we PR into the godaddy branch (and once that is stable, gate on that with a shared queue). | 21:29 |
| SpamapS | I feel like the branches being different might get confusing. Hrm. | 21:29 |
| *** elyezer has quit IRC | 21:33 | |
| *** elyezer has joined #zuul | 21:38 | |
| SpamapS | btw, in debugging why my jobs stay queued for a while, I noticed this | 21:52 |
| SpamapS | http://paste.openstack.org/show/676626/ | 21:52 |
| SpamapS | no mention of the request for 5 minutes | 21:52 |
| SpamapS | not sure what nodepool was doing in that 5 minutes | 21:53 |
| *** elyezer has quit IRC | 21:57 | |
| *** elyezer has joined #zuul | 21:58 | |
| SpamapS | oh that's interesting, I think it was spinning them up. | 22:12 |
| SpamapS | logs are not very clear on that.. which is surprising. | 22:12 |
| pabelanger | SpamapS: do you have log handler for shade enabled? | 22:13 |
| pabelanger | should see some openstack API requests | 22:13 |
| SpamapS | pabelanger: probably not | 22:13 |
| pabelanger | http://git.openstack.org/cgit/openstack-infra/puppet-nodepool/tree/templates/nodepool-launcher.logging.conf.erb | 22:14 |
| pabelanger | we setup WARNING on shade | 22:14 |
| pabelanger | kazoo is helpful too | 22:14 |
| SpamapS | I don't even have a logging.conf .. hm | 22:14 |
| SpamapS | 2018-02-18 22:14:49.340577 | map | fatal: [map -> localhost]: FAILED! => {"changed": false, "checksum": "6eef6648406c333a4035cd5e60d0bf2ecf2606d7", "msg": "Aborting, target uses selinux but python bindings (libselinux-python) aren't installed!"} | 22:15 |
| SpamapS | >:| | 22:15 |
| SpamapS | even though libselinux-python is installed | 22:15 |
| SpamapS | and selinux is disabled | 22:15 |
| SpamapS | :-| | 22:15 |
| SpamapS | ah I do have a logging.conf and it does not have a shade component | 22:16 |
| SpamapS | INFO seems like it would be what I want. | 22:16 |
| *** threestrands has joined #zuul | 22:17 | |
| *** threestrands has quit IRC | 22:17 | |
| *** threestrands has joined #zuul | 22:17 | |
| *** jesusaur has quit IRC | 22:20 | |
| SpamapS | I'm starting to wonder if there's a way I can get nodepool to allow multi-node jobs to span AZ | 22:26 |
| SpamapS | I have 3 az's in my cloud, and min-ready: 15, and a 5 node job still has to spin up new nodes to fill whatever random AZ it chose about 1/3 of the time. | 22:27 |
| SpamapS | Like right now I have 9 nodes of the label I use available in AZ 1 | 22:27 |
| SpamapS | but it's spinning up 4 in AZ 2 | 22:27 |
| pabelanger | SpamapS: single launcher right? | 22:28 |
| SpamapS | pabelanger: yes | 22:29 |
| pabelanger | I'm not 100%, never used AZs with nodepool, but believe they don't span them. | 22:30 |
| pabelanger | I seem to recall citycloud having different AZ, and we launched nodes. But they are currently disabled in nodepool | 22:31 |
| SpamapS | Yeah nodepool ensures a single request is in a single AZ. | 22:35 |
| SpamapS | And I could just contain myself to one AZ | 22:35 |
| SpamapS | But that would make me a bad citizen of that cloud. ;) | 22:36 |
| SpamapS | So perhaps I'll look and see if nodepool can somehow favor AZ's that have more ready nodes. | 22:36 |
| SpamapS | Overall it works out. | 22:37 |
| SpamapS | Like now that there are 9 available, I get a fast one the next time | 22:37 |
| SpamapS | But then there are only 4.... and the next one that hits az1 will still have to wait for 1 node to boot. | 22:37 |
| pabelanger | Yah, I think there is some optimizations that could be made to help speed up node usage | 22:37 |
| SpamapS | I think 5-node jobs are probably not the usual case that nodepool was optimized for | 22:38 |
| SpamapS | so I'm willing to dive in and see if we can handle it better. | 22:38 |
| SpamapS | Because I think it's a logarithmic worsening. 4 nodes would be a lot better than 5. 3 nodes would be even better. 2 would be impossible to notice. | 22:39 |
| SpamapS | So my naive 15 min-ready is just not sufficient. Need 30 to be more likely to have 5 available at any time. | 22:39 |
| SpamapS | and probably really I need 75 min-ready to be 100% sure. | 22:39 |
| *** jesusaur has joined #zuul | 22:40 | |
| pabelanger | have you tried using a shared common ansible_host address in your inventory? eg: https://docs.openstack.org/infra/zuul/user/config.html?highlight=nodeset#attr-nodeset.nodes.name (list of string). Where you just use a single node from nodepool, but ansible see it as 5 hosts | 22:41 |
| SpamapS | because I think the probability of getting 5 node ready in any az's is 1/(n_az*req_size**n_az) | 22:42 |
| pabelanger | mind you, not sure what your jobs are doing | 22:42 |
| SpamapS | I need them separate. | 22:42 |
| pabelanger | kk | 22:42 |
| SpamapS | Point is making sure the things that aren't on the same host can talk to eachother. :) | 22:43 |
| SpamapS | Though actualy I do think we can collapse two nodes into one now.. we've factored some stuff out. | 22:44 |
| SpamapS | so that does make my calc much better | 22:44 |
| *** jimi|ansible has joined #zuul | 22:45 | |
| SpamapS | instead of 1 / 3*(5**3) we'd be at 1 / 3*(4**3) ... so 1 / 375 with 5 nodes, vs. 1 / 192 with 4 nodes. | 22:46 |
| SpamapS | still I think we need to change that equation a little. | 22:46 |
| SpamapS | like if nothing else, just do a naive "if req_size > 1 then search_for_az_with_enough() | 22:47 |
| SpamapS | that would at least have us always hit the az that has enough | 22:47 |
| SpamapS | then with good random distribution on min-ready reqs, it should be more like 1/n_az | 22:48 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!