| tristanC | Shrews: 568704 looks good to me, though why it would cause havoc? it seems like just a rename of pollLauncher into launchComplete... | 00:49 |
|---|---|---|
| SpamapS | I'd swear my zuul stopped logging shell commands | 01:24 |
| SpamapS | 2018-05-17 01:16:36.720906 | LOOP [kick_kolla_ansible : Kicking kolla-ansible] | 01:24 |
| SpamapS | 2018-05-17 01:17:22.845876 | [map] Waiting on logger | 01:24 |
| SpamapS | and started just saying "Waiting on logger" | 01:24 |
| SpamapS | did we change the port or something? | 01:24 |
| SpamapS | although the play directly prior to this one did log | 01:24 |
| SpamapS | 2018-05-17 01:14:11.872681 | TASK [kick_kolla_ansible : Run system prep playbook] | 01:24 |
| SpamapS | 2018-05-17 01:14:14.397297 | map | [WARNING]: Could not match supplied host pattern, ignoring: rabbitmq | 01:24 |
| SpamapS | task just before even | 01:24 |
| tobiash | SpamapS: do you reboot nodes during the job? | 03:51 |
| *** jesusaur has quit IRC | 04:39 | |
| *** jesusaur has joined #zuul | 04:42 | |
| SpamapS | tobiash: no | 04:56 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: WIP: Cleanup leaked images https://review.openstack.org/568937 | 05:17 |
| *** ssbarnea_ has joined #zuul | 05:47 | |
| *** rlandy|bbl has quit IRC | 05:56 | |
| *** ssbarnea_ has quit IRC | 06:27 | |
| *** gtema has joined #zuul | 06:48 | |
| SpamapS | You know.. it would be pretty cool if nodepool-builder could be on-demand. | 06:56 |
| *** gtema has quit IRC | 07:13 | |
| *** sshnaidm|off is now known as sshnaidm|rover | 07:15 | |
| *** gtema has joined #zuul | 07:24 | |
| *** ssbarnea_ has joined #zuul | 07:34 | |
| *** jpena|off is now known as jpena | 07:46 | |
| *** gtema has quit IRC | 07:49 | |
| *** gtema has joined #zuul | 07:52 | |
| tobiash | SpamapS: but nodepool-builder also holds the final images locally (e.g. for later upload to new providers without rebuild) | 08:22 |
| SpamapS | tobiash: that seems like an outside use case, but ok. | 08:25 |
| tobiash | well that's how it works currently | 08:26 |
| SpamapS | Yeah, I can see the efficiency. | 08:26 |
| SpamapS | perhaps a cache somewhere then | 08:27 |
| SpamapS | object store or volume | 08:27 |
| SpamapS | just thinking about how you could have nodepool-launcher basically boot a builder whenever it needs one. | 08:28 |
| SpamapS | let the image build/upload happen, then shut that VM down again. | 08:28 |
| SpamapS | anyway it was just a thought as I was looking at the brief period where my allinone-except-zk is at a load of 10 because of image builds. | 08:29 |
| tobiash | that would indeed be pretty cool | 08:29 |
| * SpamapS needs sleep | 08:29 | |
| tobiash | n8 | 08:29 |
| SpamapS | ty on the debug assist btw.. taking a look at zk and the other components was really what I needed to hear | 08:29 |
| tobiash | :) | 08:30 |
| SpamapS | splitting zk into its own VM really streamlined this zuul.. everyone noticed today jobs were starting faster | 08:30 |
| SpamapS | so.. listen up boys and girls.. make sure your zk is on its own node! | 08:30 |
| SpamapS | Next up is a second executor | 08:30 |
| * SpamapS sleeps | 08:31 | |
| *** dims has quit IRC | 09:20 | |
| *** xinliang has quit IRC | 09:46 | |
| *** sshnaidm|rover has quit IRC | 09:52 | |
| *** sshnaidm|rover has joined #zuul | 09:55 | |
| *** xinliang has joined #zuul | 09:58 | |
| *** xinliang has joined #zuul | 09:58 | |
| *** ssbarnea_ has quit IRC | 09:59 | |
| *** ssbarnea_ has joined #zuul | 10:02 | |
| *** ssbarnea_ has quit IRC | 10:03 | |
| *** dims has joined #zuul | 10:07 | |
| *** ssbarnea_ has joined #zuul | 10:14 | |
| *** dims has quit IRC | 10:17 | |
| *** jpena is now known as jpena|lunch | 10:59 | |
| *** electrofelix has joined #zuul | 11:30 | |
| *** ssbarnea_ has quit IRC | 11:31 | |
| *** ssbarnea_ has joined #zuul | 11:42 | |
| *** ssbarnea_ has quit IRC | 11:44 | |
| *** ssbarnea_ has joined #zuul | 11:44 | |
| *** jpena|lunch is now known as jpena | 12:24 | |
| *** rlandy has joined #zuul | 12:33 | |
| *** dims has joined #zuul | 12:43 | |
| pabelanger | that is good news | 12:48 |
| *** swest has quit IRC | 13:29 | |
| *** sdake_ is now known as sdake | 13:39 | |
| *** ssbarnea_ has quit IRC | 13:52 | |
| *** ssbarnea_ has joined #zuul | 13:58 | |
| *** johanssone has quit IRC | 14:16 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Move SQL web handler to driver https://review.openstack.org/568028 | 14:35 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Replace use of aiohttp with cherrypy https://review.openstack.org/567959 | 14:35 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Convert streaming unit test to ws4py and remove aiohttp https://review.openstack.org/568335 | 14:35 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Revert "Revert "Switch to stestr"" https://review.openstack.org/568949 | 14:35 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Fix race in test_reconfigure_window_fixed https://review.openstack.org/569129 | 14:35 |
| *** dkranz has joined #zuul | 15:06 | |
| *** pwhalen_ has joined #zuul | 15:08 | |
| *** pwhalen has quit IRC | 15:10 | |
| *** pwhalen_ is now known as pwhalen | 15:18 | |
| *** pwhalen has quit IRC | 15:18 | |
| *** pwhalen has joined #zuul | 15:18 | |
| *** jpena is now known as jpena|off | 15:22 | |
| *** jpena|off is now known as jpena | 15:33 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Revert "Revert "Switch to stestr"" https://review.openstack.org/568949 | 15:33 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Move SQL web handler to driver https://review.openstack.org/568028 | 15:33 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Replace use of aiohttp with cherrypy https://review.openstack.org/567959 | 15:33 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Convert streaming unit test to ws4py and remove aiohttp https://review.openstack.org/568335 | 15:33 |
| *** sshnaidm|rover is now known as sshnaidm|bbl | 15:41 | |
| *** ssbarnea_ has quit IRC | 15:44 | |
| *** sshnaidm|bbl has quit IRC | 15:46 | |
| fdegir | i suppose changes add8ng instructions to install zuul on centos are ready to go in | 15:49 |
| fdegir | the question i have now is that which version of ubuntu i should be adding the docs? | 15:49 |
| fdegir | thinking of doing for 18.04 | 15:49 |
| pabelanger | I like the idea of 18.04, to get bwrap from OS over the openstack-infra PPA | 15:55 |
| pabelanger | been testing it for a while, and it just works (tm) | 15:55 |
| fdegir | pabelanger: ok - will send the patches based on the structure on centos instructions | 15:59 |
| SpamapS | Hm | 16:11 |
| SpamapS | I notice right before the next play that doesn't show streaming, I see this | 16:11 |
| SpamapS | 2018-05-17 15:59:38.155575 | TASK [kick_kolla_ansible : Kicking kolla-ansible prechecks] | 16:11 |
| SpamapS | 2018-05-17 16:00:58.510037 | [Zuul] Log Stream did not terminate | 16:11 |
| SpamapS | so that task doesn't stream, and we get the 'did not terminate" | 16:12 |
| SpamapS | I wonder if the zuul console daemon is crashing | 16:12 |
| SpamapS | How would I check that? | 16:12 |
| pabelanger | SpamapS: anything in executor-debug.log? I've seem cases where we don't long a task propelry | 16:13 |
| SpamapS | pabelanger: it's consistently happening every time at this point in this job. | 16:14 |
| SpamapS | The previous task is logged, and then no further tasks are. | 16:14 |
| SpamapS | It may have something to do with the fact that we're running ansible playbooks... I dunno. | 16:14 |
| *** AJaeger has quit IRC | 16:15 | |
| *** sshnaidm|bbl has joined #zuul | 16:35 | |
| fungi | SpamapS: and nothing in dmesg? for a while we had memory pressure killing streamers on the executors, and that tends to leave no trace in the zuul logs | 16:36 |
| *** gtema has quit IRC | 16:42 | |
| corvus | SpamapS: there should be a zuul_console process running on the worker, and it should be listening on port 19885 on all interfaces | 16:47 |
| corvus | SpamapS: as long as you don't have an ensure:something-other-than-present attribute set on the zuul_console ansible module, it should continue running even past the end of the job. so you can hold the node and inspect it. | 16:47 |
| corvus | SpamapS: the command output is written to /tmp/console-{uuid}.log so you can ls /tmp and pick a uuid and then "echo uuid | nc host 19885" to ask it for one of those logs to see if it's working | 16:50 |
| corvus | SpamapS: (note that the log uuids are not the build uuid -- each command invocation gets its own uuid for its own log file) | 16:50 |
| corvus | SpamapS: (to be clear, the /tmp directory with log files is also on the worker node) | 16:51 |
| *** jpena is now known as jpena|off | 17:01 | |
| SpamapS | corvus: good tips, I'll hold a node and inspect | 17:20 |
| *** StaceyF has joined #zuul | 17:30 | |
| *** gtema has joined #zuul | 18:00 | |
| *** ssbarnea_ has joined #zuul | 18:02 | |
| *** gtema has quit IRC | 18:17 | |
| *** acozine1 has joined #zuul | 18:21 | |
| *** rlandy is now known as rlandy|brb | 19:01 | |
| *** gtema has joined #zuul | 19:07 | |
| *** rlandy|brb is now known as rlandy | 19:27 | |
| *** gtema has quit IRC | 19:52 | |
| *** dkranz has quit IRC | 20:39 | |
| *** ssbarnea_ has quit IRC | 20:44 | |
| *** hughsaunders has quit IRC | 21:44 | |
| *** hughsaunders has joined #zuul | 21:46 | |
| *** StaceyF has quit IRC | 21:56 | |
| *** acozine1 has quit IRC | 22:12 | |
| *** acozine1 has joined #zuul | 22:19 | |
| *** myoung|ruck is now known as myoung|ruck|off | 22:31 | |
| *** acozine1 has quit IRC | 22:35 | |
| *** threestrands has joined #zuul | 22:37 | |
| *** rlandy is now known as rlandy|biab | 22:38 | |
| SpamapS | Hm, trolling through my builds db.. I find that we're running 10 - 30 jobs per hour and jobs are taking about 20,000 - 30,000 seconds per hour (meaning we have a lot of concurrency ;) | 23:30 |
| SpamapS | I need to turn on statsd so I can get node stats | 23:30 |
| *** rlandy|biab is now known as rlandy | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!