| corvus | tobiash, mordred, jhesketh: i think the sys.exit(0) which was removed in https://review.openstack.org/530205 was critical for actually getting the scheduler to exit | 00:43 |
|---|---|---|
| corvus | since that landed, it doesn't seem to actually stop | 00:43 |
| corvus | when changing things like that, it's important to actually run the daemons manually. the tests don't start and stop them the way they are started in stopped in service. | 00:45 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Omnibus scheduler stop fixes https://review.openstack.org/535997 | 01:07 |
| corvus | tobiash, mordred, jhesketh, pabelanger: ^ that makes 'zuul-scheduler stop' work for me | 01:07 |
| clarkb | corvus: does the omnibus include volumes 7 and 8? | 01:08 |
| corvus | everything but the last volume | 01:10 |
| corvus | 'zuul-fingergw stop' works -- which jives with what happened on zuulv3.o.o -- when i ran that manually it stopped. so it seems like an init problem. | 01:10 |
| corvus | i'm going to poke at that on zuul01 since i bet no one is using it right now | 01:13 |
| corvus | okay it works | 01:14 |
| corvus | i guess something to look out for next time then | 01:14 |
| jhesketh | corvus: so is it working with master or were you testing with your patch? | 01:17 |
| corvus | jhesketh: fingergw works on master, scheduler needs my patch | 01:18 |
| jhesketh | ack, thanks for the clarificationn | 01:19 |
| *** jkilpatr_ has quit IRC | 01:37 | |
| mnaser | so i take it with nodepoolv3, the zmq event publisher cannot be really used anymore? | 02:46 |
| mnaser | i'm trying to work with a some folks that do a lot of CI but currently depend on the openstack jenkins plugin (which has a lot of setbacks, such as images v1 support only) | 02:46 |
| clarkb | mnaser: correct it uses the zookeeper based node request system | 02:47 |
| pabelanger | yah, no longer needed | 02:47 |
| clarkb | rather than inferred demand and cleanups based on zmq data | 02:47 |
| mnaser | leaping to zuulv3 would be too much operationally but going to nodepool v2 to later have to go to v3 might be a hassle | 02:47 |
| mnaser | so i guess jenkins + nodepoolv3 is currently something that would require work to do | 02:47 |
| clarkb | yes. It should be possibly to implement a node requestor to nodepool for jenkins but I am not aware of that being done yet | 02:48 |
| pabelanger | I think mordred has some notes on an etherpad too | 02:48 |
| corvus | a java dev should be able to implement it within a few days | 02:49 |
| mnaser | corvus: i guess zuuls source code would be the best reference point for a dev | 02:50 |
| mnaser | given i will assume documentation on this sort of thing is probably not super set-in-stone | 02:50 |
| corvus | mnaser: zuul+nodepool probably, yeah. the request framework is pretty simple. | 02:51 |
| mnaser | https://github.com/openstack-infra/zuul/blob/master/zuul/nodepool.py | 02:52 |
| mnaser | so i guess this is where "the magic happens" | 02:52 |
| corvus | yep, and zk.py | 02:52 |
| corvus | er | 02:52 |
| corvus | sorry, there's some stuff in model.py | 02:52 |
| corvus | mnaser: mordred's write up of how do do it is really good. but i don't know how to find it right now. | 02:54 |
| mnaser | hopefully all this highlighting means he'll see it at some point :p | 02:54 |
| mnaser | i'll go over the write up, contact the ML and see if folks want to put together and effort and then i'll try to bring in some resources as well | 02:56 |
| mnaser | probably best to keep it the openstack-infra umbrella too | 02:57 |
| mnaser | i think it would be extremely beneficial for third party ci who are comfortable sticking to jenkins | 02:57 |
| corvus | oh, i thought you were looking at it as an intermediate step | 02:58 |
| mnaser | corvus: semi-permanent i would say. i think for folks that don't have people who understand zuulv3 very well, it's really hard to operate right now | 03:00 |
| *** harlowja has quit IRC | 03:00 | |
| mnaser | as much as i'd love to see them run zuulv3, not everyone has the luxury of having a zuul devs on their team like infra does :D | 03:01 |
| corvus | mnaser: oh, yeah, no one should be running it *now*. but in a couple months, we will have actually released it, and then i think it's a great thing for folks to run. and it won't be hard. :) | 03:01 |
| mnaser | corvus: i'm just as excited to see people run it and i think especially as users consume openstack for ci, it's honestly the best tooling out tehre | 03:02 |
| mnaser | the openstack jenkins plugin really bridges a huge gap and does a great job but its hard to maintain java :( | 03:02 |
| clarkb | corvus this got me looking, amy ide why zuul's zk.py callback closure updates that local request_nodes variable it seems to be an unused value? | 03:13 |
| clarkb | oh it may be side effecting the data in the node request because the contents of the list are referneces to objects in the request | 03:14 |
| *** harlowja has joined #zuul | 03:37 | |
| *** bhavik1 has joined #zuul | 04:17 | |
| *** bhavik1 has quit IRC | 04:23 | |
| *** harlowja has quit IRC | 05:32 | |
| mordred | mnaser: https://etherpad.openstack.org/p/zuulv3-jenkins-integration | 15:51 |
| mrhillsman | what makes nodepool-builder continue to try building an image not in zookeeper | 16:11 |
| clarkb | mrhillsman: thsts built in behavior it needs at least one image built for each of the configured disk images | 16:24 |
| mrhillsman | so they are there | 16:25 |
| mrhillsman | actually 2 of each | 16:25 |
| mrhillsman | there's this other one though that just continues to try building | 16:25 |
| mrhillsman | i just sat and watched the logs | 16:26 |
| mrhillsman | http://paste.openstack.org/show/648803/ | 16:26 |
| mrhillsman | 2018-01-20 16:24:28,943 INFO nodepool.image.build.ubuntu-xenial: b"2018-01-20 16:24:28.943 | aiohttp requires Python '>=3.4.2' but the running Python is 2.7.12" | 16:26 |
| mrhillsman | i think that is the problem | 16:26 |
| mrhillsman | which is interesting because ps faux shows nodepool-builder running under python3 | 16:27 |
| clarkb | that is the log from the dib build itself | 16:28 |
| mrhillsman | which looks like it is cause of http://paste.openstack.org/show/648807/ | 16:29 |
| clarkb | oh we may have broken image builds with the branch merge on thursday | 16:30 |
| clarkb | cool | 16:30 |
| mrhillsman | so i need to pin aiohttp probably? | 16:30 |
| mrhillsman | looking at pypi confirms that 3.4.2 requirement | 16:31 |
| mrhillsman | i'm not sure if the devs pulled in the latest yet | 16:31 |
| mrhillsman | but i do see some images that have been build for what looks like a couple days at least | 16:32 |
| mrhillsman | http://paste.openstack.org/show/648810/ | 16:33 |
| clarkb | aiohttp is python3 only. We need to pin the version of zuul being installed in that venv to last v2 tag | 16:33 |
| mrhillsman | ah ok | 16:34 |
| mrhillsman | i wonder if it is actually an issue with just our environment because disk-image-create is using /usr/bin/python | 16:45 |
| mrhillsman | sorry, not a developer :) | 16:45 |
| mrhillsman | maybe it should be /usr/bin/python3 | 16:45 |
| clarkb | no the image build is installing zuul into a virtualenv on the image. It does this as backward compat way to install zuul-cloner | 16:49 |
| clarkb | we merged feature/zuulv3 into master on thursday which broke that | 16:50 |
| mrhillsman | got it | 16:50 |
| *** cinerama has quit IRC | 18:22 | |
| SpamapS | oh I should check my image builds I think I was still pulling in cloner | 19:06 |
| SpamapS | even though I don't need it | 19:06 |
| SpamapS | Weird, so I just dropped an element from the list for an image, and nodepool responded by deleting all the images. Is that normal? | 19:40 |
| SpamapS | Oh wait, I didn't even do that. | 19:40 |
| SpamapS | just changed one of the elements itself | 19:41 |
| *** _ari_ is now known as _ari_|DevConf | 19:49 | |
| pabelanger | mrhillsman: clarkb: yah, zuul-env should pip install 2.6.0 now, no longer master. SF also had this issue on Friday | 19:52 |
| mrhillsman | ok, will update, i was trying to find a 2.X.XdevXXXX tag | 19:54 |
| mrhillsman | since i never used anything other than master and would see something like that post install stdout | 19:54 |
| SpamapS | weird... this is weird.. | 19:55 |
| SpamapS | nodepool-builder has built images, but it isn't uploading to providers. :-P | 19:55 |
| SpamapS | It's just sitting there not uploading them. | 19:56 |
| SpamapS | n/m | 19:57 |
| SpamapS | found the problem | 19:57 |
| SpamapS | forgot to list the disk image in the providers | 19:57 |
| * SpamapS finally ends the fight to have two clouds and just uses the big less-used one. :-P | 20:16 | |
| SpamapS | we need to figure out how to make nodepool work better with smaller clouds/quota providers. | 20:16 |
| SpamapS | maybe have a thing that schedules requests rather than pulling as a best effort | 20:17 |
| pabelanger | I'm sure it will take time, even now we are moving from 2 launcher to 4 launcher to see if that improves the performance of launching nodes in nodepool. | 20:19 |
| pabelanger | but, because each launcher is able to request min-ready nodes, I suspect we'll start seeing more ready nodes online. Possible that logic might need to change too | 20:21 |
| clarkb | min ready should be synchronized by zookeeper right? | 20:24 |
| clarkb | SpamapS: and you mean scheduling diskimage uploads? | 20:24 |
| clarkb | and yes no images in the list for the provider means remove the images | 20:25 |
| pabelanger | clarkb: I think so, but each launcher sends the request, so with min-ready 1, and 3 launchers, we can at least 3 nodes created | 20:26 |
| pabelanger | clarkb: I'd need to confirm with Shrews again | 20:26 |
| pabelanger | we have 4 opensuse nodes ready now, but min-ready is 1 | 20:27 |
| SpamapS | clarkb: two different problems | 20:42 |
| SpamapS | clarkb: the image thing was just a think-o | 20:42 |
| SpamapS | clarkb: but the bigger problem is that I had a cloud with max-servers: 8 and one with max-servers: 50 and the 8 one would grab requests for 5-nodes and sit forever. | 20:43 |
| SpamapS | I tried just putting the image that serves the 5-node request in one cloud but that didn't seem to work either. Then realized I just don't need the 8 nodes. | 20:44 |
| SpamapS | as far as min-ready.. that should probably be something with a semaphore so only one launcher requests min-ready at a time. | 20:45 |
| * SpamapS goes to Satuday a bit more | 20:45 | |
| *** toabctl has quit IRC | 21:29 | |
| *** clarkb has quit IRC | 22:05 | |
| *** weshay|rover has quit IRC | 22:05 | |
| *** kklimonda has quit IRC | 22:05 | |
| *** adam_g has quit IRC | 22:05 | |
| *** mgagne has quit IRC | 22:05 | |
| *** adam_g has joined #zuul | 22:05 | |
| *** clarkb has joined #zuul | 22:05 | |
| *** mgagne has joined #zuul | 22:06 | |
| *** mgagne is now known as Guest28399 | 22:06 | |
| *** kklimonda has joined #zuul | 22:06 | |
| *** weshay has joined #zuul | 22:07 | |
| *** fungi has quit IRC | 23:01 | |
| *** fungi has joined #zuul | 23:04 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!