| openstackgerrit | Joshua Hesketh proposed openstack-infra/nodepool: Merge branch 'master' into feature/zuulv3 https://review.openstack.org/400536 | 00:19 |
|---|---|---|
| *** jamielennox is now known as jamielennox|away | 01:26 | |
| *** jamielennox|away is now known as jamielennox | 02:15 | |
| *** saneax-_-|AFK is now known as saneax | 06:16 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul: Ignore .idea directory https://review.openstack.org/403522 | 06:44 |
| *** willthames has quit IRC | 07:04 | |
| *** abregman has joined #zuul | 07:24 | |
| *** hashar has joined #zuul | 08:27 | |
| *** hashar has quit IRC | 08:28 | |
| *** hashar has joined #zuul | 08:42 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul: Cloner: More robust infrastrastructure detection https://review.openstack.org/403559 | 08:46 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul: Cloner: Better infrastructure failure handling https://review.openstack.org/403559 | 08:56 |
| *** Cibo has joined #zuul | 10:56 | |
| *** Cibo_ has quit IRC | 10:57 | |
| *** hashar has quit IRC | 11:02 | |
| *** hashar has joined #zuul | 11:03 | |
| *** hashar has quit IRC | 11:48 | |
| *** hashar has joined #zuul | 13:05 | |
| openstackgerrit | Merged openstack-infra/zuul: Ignore .idea directory https://review.openstack.org/403522 | 13:33 |
| openstackgerrit | Merged openstack-infra/nodepool: Re-enable test: test_alien_list_fail https://review.openstack.org/400422 | 13:37 |
| openstackgerrit | Merged openstack-infra/nodepool: Re-enable TestWebApp tests https://review.openstack.org/399716 | 13:37 |
| openstackgerrit | Merged openstack-infra/nodepool: Remove waitForBuiltImages() / JobTracker() from nodepool.py https://review.openstack.org/399727 | 13:40 |
| openstackgerrit | Merged openstack-infra/nodepool: Enable test_delete / test_delete_now tests https://review.openstack.org/399734 | 13:40 |
| Shrews | pabelanger: odd. for the json issue, how exactly did you "delete" it? | 13:55 |
| pabelanger | Shrews: used the CLI, dib-image-delete centos-7-0000000001 | 13:57 |
| pabelanger | let me see if I can reproduce it | 13:58 |
| Shrews | pabelanger: ah. so, this smells like another race where one thread has one view of the zk nodes, while another deletes one of them out from under it | 13:58 |
| pabelanger | Shrews: Yup, think you are right. Just reproduced it again | 14:00 |
| pabelanger | dib-image-delete existing built image | 14:01 |
| Shrews | oh, actually... | 14:01 |
| pabelanger | dib-image-list a few times | 14:01 |
| pabelanger | exception | 14:01 |
| *** Cibo has quit IRC | 14:05 | |
| Shrews | i'm actually not sure how that is happening. perhaps after more coffee | 14:07 |
| Shrews | pabelanger: so, in order to get that json failure, there would have to be a path where a build node could exist w/o any data. I cannot conceive of such a path. When it's created, it's created with a value. When it's updated, it's updated with a new value. This one has me perplexed | 14:25 |
| Shrews | because that path obviously exists, as you've demonstrated | 14:26 |
| pabelanger | Shrews: some more history, uploads aren't actually working right now. Because my clouds.yaml file is not correct | 14:27 |
| pabelanger | so upload workers are just failing | 14:27 |
| pabelanger | let me see if I can get some debug logs | 14:27 |
| Shrews | that's something else we should do something about (bad configured cloud). not sure how we should handle that | 14:30 |
| Shrews | pabelanger: what's wrong with your clouds.yaml? | 14:30 |
| pabelanger | Shrews: bad username /passwork | 14:30 |
| pabelanger | so uploads fail | 14:31 |
| pabelanger | I'm going to disable uploads | 14:31 |
| pabelanger | to see if that might be the issue | 14:31 |
| pabelanger | ok, rebuilding image | 14:32 |
| Shrews | maybe we need a new ProviderManager method to validate connection before we do stuff | 14:33 |
| Shrews | *shrug* | 14:34 |
| pabelanger | Going to work on pause field this morning too | 14:35 |
| pabelanger | easier then deleting 1/2 my nodepool.yaml file | 14:35 |
| Shrews | i'm going to complete my RH training this morning so they quit bugging me about it | 14:38 |
| pabelanger | Ha | 14:42 |
| pabelanger | I should do that too | 14:42 |
| *** saneax is now known as saneax-_-|AFK | 15:11 | |
| *** herlo has joined #zuul | 15:21 | |
| *** herlo has joined #zuul | 15:21 | |
| *** phschwartz has quit IRC | 16:01 | |
| *** gothicmindfood has quit IRC | 16:01 | |
| *** leifmadsen has quit IRC | 16:01 | |
| *** gothicmindfood has joined #zuul | 16:02 | |
| *** phschwartz has joined #zuul | 16:02 | |
| *** leifmadsen has joined #zuul | 16:02 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Add pause support for diskimages https://review.openstack.org/403780 | 16:05 |
| pabelanger | Shrews: jeblair: ^ first pass at pass support for diskimages. | 16:06 |
| pabelanger | happy to rework the test_dib_image_pause test too | 16:06 |
| pabelanger | need a better way to detect the image isn't actually built | 16:07 |
| pabelanger | since waitForBuild() loops for ever | 16:07 |
| *** abregman has quit IRC | 16:12 | |
| timrc | pabelanger: did you mean test_db_image_delete? | 16:24 |
| timrc | dib* | 16:25 |
| timrc | Oh oops nm... | 16:25 |
| pabelanger | I named it test_dib_image_pause | 16:25 |
| pabelanger | but that can change | 16:25 |
| timrc | I confused waitForBuild with waitForImage. Nevermind me! | 16:26 |
| timrc | I see what you're saying now about the implementation of test_dib_image_pause. | 16:26 |
| *** hashar has quit IRC | 17:10 | |
| *** Shuo has joined #zuul | 17:13 | |
| Shrews | pabelanger: left a suggestion on the pause change | 17:18 |
| Shrews | ooh, had another thought, too | 17:20 |
| Shrews | left another comment | 17:21 |
| pabelanger | Shrews: re: manual build requests, where is that done? I found the only 2 places we called _buildImage(), _checkImageForScheduledImageUpdates() and _checkImageForManualBuildRequest() | 17:41 |
| Shrews | pabelanger: by the client. cmd/nodepoolcmd.py | 17:42 |
| pabelanger | ah | 17:42 |
| pabelanger | checking | 17:42 |
| Shrews | pabelanger: if you prevent the client from submitting a request, you probably don't need the check in checkImageForManualBuildRequest | 17:44 |
| Shrews | since hasBuildRequest() would return false. but, it also doesn't hurt | 17:45 |
| Shrews | but i do think we should tell the user "hey, this image is paused so you can't ask for a build right now" | 17:46 |
| pabelanger | I'll keep paused just to be safe | 17:46 |
| pabelanger | and ya, added a log statement | 17:46 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Add pause support for diskimages https://review.openstack.org/403780 | 17:47 |
| Shrews | pabelanger: that message doesn't get back to the user though, does it? | 17:49 |
| pabelanger | should, depending on logging settings | 17:49 |
| pabelanger | let me confirm the defaults | 17:49 |
| Shrews | i ask b/c some things use print, others use log | 17:50 |
| *** abregman has joined #zuul | 17:52 | |
| *** abregman has quit IRC | 17:53 | |
| *** abregman has joined #zuul | 17:54 | |
| pabelanger | Shrews: yup, works. But problem with code | 17:57 |
| pabelanger | I'll create a test too | 17:57 |
| pabelanger | Also | 17:57 |
| SpamapS | zomg | 17:57 |
| pabelanger | actually, never mind | 17:57 |
| SpamapS | I'm healthy | 17:57 |
| SpamapS | my kids are healthy | 17:58 |
| SpamapS | I might actually like.. get work done | 17:58 |
| auggy | What time is the zuul meeting today? | 18:06 |
| SpamapS | 1400 PST | 18:07 |
| auggy | I keep mixing up the times so I want to double check so I don't miss it | 18:07 |
| SpamapS | auggy: I highly recommend subscribing to the openstack meetings ical. | 18:07 |
| auggy | Ok! That's what I had put in my calendar! *whew* | 18:07 |
| SpamapS | It's a nice overlay.. and if you use google cla, it's really easy to copy things onto your personal calendar. | 18:07 |
| SpamapS | s/cla/cal | 18:08 |
| auggy | Yeah, it's not always up to date or something | 18:08 |
| auggy | I had some issues with it | 18:08 |
| auggy | Or maybe I subscribed to the wrong OpenStack ical | 18:08 |
| auggy | I like to put things on my calendar and set alarms :) | 18:09 |
| SpamapS | Yeah I do that too, the ical just serves as a source of truth when I'm not sure my calendar is right. | 18:11 |
| morgan_ | SpamapS: i ma jealous of "healthy"... i'm still recovering from an awful cold. | 18:20 |
| SpamapS | morgan_: me, wife, and all 4 children, all got a stomach bug, in sequence. | 18:22 |
| SpamapS | actually the sequence was baby (who felt it the least), 4 year old, wife, 7 year old, me, 13 year old | 18:23 |
| SpamapS | luckily was 24 hours, but still.. brutal | 18:23 |
| morgan_ | SpamapS: ah my cold was ~7 days... started weekend before thanksgiving... | 18:24 |
| *** abregman has quit IRC | 18:28 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Add pause support for diskimages https://review.openstack.org/403780 | 18:41 |
| pabelanger | Shrews: ^ decided to raise an exception, it does make testing easier | 18:42 |
| pabelanger | jeblair: Shrews: so, even though nodepool-builder doesn't need database access, nodepoolcmd still does. Which is causing some issues on nb01.o.o, since we are missing a database. How do we want to handle that? | 19:25 |
| Shrews | what's it need a db for? | 19:26 |
| pabelanger | we still use it for online nodes | 19:26 |
| pabelanger | so, nodepool list | 19:26 |
| jeblair | pabelanger: can we just not run those commands on nb01? | 19:27 |
| jeblair | i'm okay if it just errors out there for those commands | 19:28 |
| mordred | ++ | 19:28 |
| pabelanger | issue is, we first try to configureDatabase(), which fails because our secret.conf is using localhost. This is from nb01 | 19:29 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: [WIP] Re-enable test: test_image_upload_fail https://review.openstack.org/400970 | 19:29 |
| jeblair | pabelanger: oh, maybe we can move that call to only happen if we need to? | 19:29 |
| pabelanger | Ya, that is what I am doing now | 19:30 |
| Shrews | i've just spent the last few hours trying to get that test ^^^ to work. the provider manager uploadImage() still doesn't get the updated fake client | 19:30 |
| Shrews | imma bout to give up on it | 19:30 |
| jeblair | Shrews: ok, i'll take a look later, unless mordred beats me to it | 19:34 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Use database for non-zookeeper commands https://review.openstack.org/403869 | 19:38 |
| pabelanger | jeblair: Shrews: ^ should do what we need | 19:39 |
| pabelanger | nb01 also has 1024 SATA drive for /opt too | 19:40 |
| pabelanger | So, we should be able to start our first builds shortly | 19:40 |
| pabelanger | okay, just added nb01.o.o to puppetmaster.o.o, puppet should start to running on the server shortly | 20:04 |
| *** hashar has joined #zuul | 20:23 | |
| openstackgerrit | Adam Gandelman proposed openstack-infra/zuul: Re-enable test_success_pattern as test_success_url https://review.openstack.org/400455 | 20:28 |
| pabelanger | jeblair: mordred: Shrews: do you mind taking a quick peak at https://review.openstack.org/#/c/403869/ is needed for nb01.o.o | 20:33 |
| mordred | Shrews: wow, that really is calling REAL CREATE IMAGE isn't it? | 20:40 |
| clarkb | pabelanger: looks like that broke a bunch of tests | 20:41 |
| pabelanger | clarkb: boo | 20:41 |
| pabelanger | checking | 20:41 |
| clarkb | pabelanger: so my +2 was probably too early :) but I think generally that idea is what we want | 20:41 |
| pabelanger | okay, let me clean up | 20:42 |
| Shrews | mordred: yeah. the manager's uploadImage() method uses self._client, which is always a FakeOpenStackCloud (i.e., it never gets set to a FakeUploadFailCloud) | 20:42 |
| Shrews | mordred: that's the part that confuses me b/c resetClient() is supposed to set it. so my current theory is that we're working with copies of managers which don't permanently store the new client | 20:43 |
| mordred | Shrews: so - we're monkey patching FakeProviderManager - but we do that after the provider exists | 20:46 |
| mordred | Shrews: I think you'd need to monkeypatch the instant of ProviderManager at that point in the flow | 20:46 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Use database for non-zookeeper commands https://review.openstack.org/403869 | 20:46 |
| pabelanger | clarkb: ^local tests happy now | 20:47 |
| mordred | Shrews: like, instead of monkeypatching the class, after provider = pool.config.providers['fake-provider'] - do a provider._getClient = get_fake_client perhaps? | 20:47 |
| Shrews | mordred: i think so, but we don't have a way to get the real instance now. i think using 'pool' there is wrong, b/c it's now the BuildWorkers which have the config things | 20:48 |
| clarkb | pabelanger: thanks will look again | 20:48 |
| mordred | Shrews: is it possible to do the monkey patch before starting the pool? | 20:48 |
| Shrews | mordred: no, because we need the first upload (happens automagically) to succeed | 20:48 |
| Shrews | mordred: we are trying to fail the second upload | 20:49 |
| Shrews | mordred: oh wait... maybe we don't need that first upload. lemme try | 20:49 |
| mordred | oh - right - but also yeah | 20:49 |
| mordred | Shrews: also, this is the first thing I was trying to say: http://paste.openstack.org/show/590711/ | 20:51 |
| mordred | if it turns out we do need the first thing | 20:51 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Clean up zookeeper usage in nodepoolcmd https://review.openstack.org/403902 | 20:58 |
| pabelanger | Shrews: ^what do you think about using nodepool.getZK() for nodepoolcmd.py? Help reduce some duplicate logic | 20:59 |
| clarkb | Shrews: mordred if you are patching and patching the class, any existing objects of that class will already have the old thing iirc | 21:00 |
| clarkb | because each object has its own __dict__ in python | 21:00 |
| Shrews | mordred: got it working with the "patch before starting" thing (and discovering that we weren't properly recording failed uploads) | 21:04 |
| Shrews | mordred: thx | 21:04 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Re-enable test: test_image_upload_fail https://review.openstack.org/400970 | 21:05 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Re-enable test: test_image_upload_fail https://review.openstack.org/400970 | 21:07 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Re-enable test: test_image_upload_fail https://review.openstack.org/400970 | 21:10 |
| Shrews | pabelanger: let me make some tea before i look at your reviews... sorry, got a bit wrapped up there | 21:11 |
| jeblair | jamielennox: i left a reply to your comment on https://review.openstack.org/391681 | 21:12 |
| mordred | Shrews: yay! | 21:13 |
| openstackgerrit | Merged openstack-infra/zuul: Update webapp status json to support tenants https://review.openstack.org/391681 | 21:14 |
| pabelanger | Shrews: sure, np | 21:15 |
| openstackgerrit | Merged openstack-infra/zuul: Add tenant support to zuul client https://review.openstack.org/393850 | 21:16 |
| openstackgerrit | Merged openstack-infra/zuul: Remove includes and parameter functions https://review.openstack.org/394644 | 21:18 |
| Shrews | pabelanger: I don't think we want to do that. It would tie any use of the client to having a nodepoold daemon running locally. In the new world order (where the pieces don't have to share a local DB), we may have builders, uploaders, and nodepool daemons all separated | 21:25 |
| clarkb | isnt it explicitly decoupling from the db | 21:26 |
| pabelanger | Hmm | 21:28 |
| pabelanger | I thought nodepoold wasn't running when I tested | 21:28 |
| pabelanger | just a dependency on nodepool/nodepool.py | 21:29 |
| Shrews | pabelanger: oh, maybe i misread the code there... 1 sec | 21:29 |
| pabelanger | not a major deal if we want to abandon it | 21:29 |
| Shrews | pabelanger: yeah, i misread (looked too quickly). that seems ok | 21:31 |
| clarkb | is this not related to the fix nodepoolcmd change? | 21:31 |
| Shrews | clarkb: not sure what change you're refering to | 21:32 |
| pabelanger | 403869 is what I was talking about | 21:32 |
| clarkb | 403902 | 21:32 |
| pabelanger | which I just rechecked for database drop issue | 21:32 |
| pabelanger | Oh | 21:32 |
| pabelanger | 403902 | 21:33 |
| pabelanger | ya | 21:33 |
| jeblair | i think we may eventually want to consider that class to just be the nodepool daemon, but we can reorg that later | 21:33 |
| Shrews | pabelanger: also, fyi, 970 will change the status of the failed uploads you've been seeing in testing to 'failed' | 21:40 |
| pabelanger | looking | 21:40 |
| pabelanger | Shrews: okay, cool | 21:41 |
| pabelanger | I'll test again once that merges | 21:41 |
| jeblair | pabelanger, clarkb: https://review.openstack.org/399775 could use a couple +2s | 21:43 |
| openstackgerrit | Merged openstack-infra/nodepool: Add test to validate when a node build is disabled https://review.openstack.org/399642 | 21:44 |
| jeblair | Shrews: re our zk sequence number conversation -- did we come to this conclusion: do not assume anything about the seqno in code, okay to assume it starts at 1 in tests for now, but likely remove those assumptions after our initial production push (and probably around the time we make a decision about what to do about build/upload ids in the user interface) | 21:46 |
| openstackgerrit | Merged openstack-infra/nodepool: Re-enable test_hold test https://review.openstack.org/399758 | 21:47 |
| Shrews | jeblair: i don't remember the "remove after init prod" part, but pretty much, yeah | 21:47 |
| clarkb | jeblair: before I approve, should name be of string and not int type? it doesnt matter here but that could be confusing later | 21:47 |
| jeblair | Shrews: okay. do you want to modify your vote on https://review.openstack.org/399974 then? | 21:48 |
| Shrews | jeblair: done | 21:49 |
| jeblair | clarkb: i don't think we need to assume it's an int -- i think anything that's stringable is okay | 21:49 |
| clarkb | well its set to int values in that change eg 0 | 21:50 |
| jeblair | (now i want to write a generator that returns 'alice', 'bob', ....) | 21:50 |
| jamielennox | jeblair: yep, and i agree, with a lot of tenants just generating a full status.json would get too heavy, was just looking to point out our current case and if we find anything we really want we can add it later | 21:52 |
| jeblair | jamielennox: cool. also, let me know if my comments on https://review.openstack.org/401019 are confusing | 21:55 |
| openstackgerrit | Merged openstack-infra/nodepool: Log each worker thread separately https://review.openstack.org/399775 | 21:56 |
| *** hashar has quit IRC | 21:57 | |
| openstackgerrit | Merged openstack-infra/nodepool: Add test to validate rebuilds are scheduled https://review.openstack.org/399974 | 22:00 |
| jlk | Is there a zuul meeting happening today? | 22:01 |
| jeblair | i think there is! | 22:01 |
| jeblair | over in #openstack-meeting-alt | 22:01 |
| jhesketh | Morning | 22:01 |
| jlk | seems there is another meeting happening there | 22:01 |
| jlk | oh they just ended | 22:01 |
| openstackgerrit | Merged openstack-infra/nodepool: Delete test_handle_dib_build_gear_disconnect test https://review.openstack.org/398655 | 22:07 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Re-enable test: test_list_nodes https://review.openstack.org/400955 | 22:31 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Re-enable alien-image-list command and tests https://review.openstack.org/400836 | 22:31 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Re-enable test: test_image_upload_fail https://review.openstack.org/400970 | 22:31 |
| jamielennox | ianw: it's not that, we're running nodepool via systemd unit which only points at the application entry, there's no way (i could see) to source the activate file first | 22:46 |
| jamielennox | ianw: so PATH is hard set to systemd's defaults at that point | 22:46 |
| jamielennox | we can set our own path, symlink things into real path, a whole bunch of options | 22:47 |
| clarkb | ianw: pabelanger jamielennox iirc the issues are dib specific | 22:48 |
| clarkb | so potentially you could fix those bugs in dib | 22:48 |
| ianw | jamielennox: hmm, ok ... i guess calling an app via it's entry-point like /path/to/venv/bin/app doesn't set the PATH inside the app | 22:49 |
| clarkb | iirc you can't reliably do path/to/venv/bin/disk-image-create either | 22:49 |
| clarkb | (regardless of whether or not nodepool is over the top of it) | 22:49 |
| ianw | so, with dib v2, disk-image-create has moved to a standard entry point | 22:49 |
| ianw | which means instead of calling it just as an external program, we could import diskimage_builder and call it from python | 22:50 |
| ianw | that would ensure we get the version in the virtualenv | 22:50 |
| jamielennox | clarkb: oh, i hadn't noticed the problem go that far - if you can't call /path/to/venv/bin/disk-image-create then there's no point | 22:50 |
| jamielennox | however i'm inclined to say that if dib v2 has a python API then nodepool should start using that and the problem goes away | 22:50 |
| clarkb | jamielennox: it may not be a problem any longer (I would have to double check) but I know in the past that I have just given up and installed dib globally because of this | 22:50 |
| clarkb | jamielennox: you ahve to source the venv and run dib or install globally to make it work | 22:51 |
| clarkb | and thats a dib thing nothing to do with nodepool | 22:51 |
| pabelanger | jamielennox: clarkb: ianw: http://git.openstack.org/cgit/openstack/windmill/tree/playbooks/templates/nodepool-builder/usr/local/bin/disk-image-create.j2 is how I solved it. Just created a disk-image-create wrapper script, dropped into /usr/local/bin. Since I didn't source venv for nodepool-builder in systemd | 22:51 |
| ianw | clarkb: that is probably right, it maybe couldn't find it's elements. in v2, we have moved all the elements under diskimage_builder python, rather than have them as data files | 22:52 |
| ianw | clarkb: that means that we can find them always by using pkg_resources | 22:52 |
| ianw | at the moment, it does horrible, horrible hacks trying to figure out where it's installed relative to the elements | 22:52 |
| jamielennox | pabelanger: we just hacked path into /etc/default/nodepool https://github.com/BonnyCI/hoist/blob/master/roles/nodepool/templates/etc/default/nodepool#L10 | 22:53 |
| pabelanger | oh, it was dib-run-parts that fails too, with symlinks | 22:53 |
| pabelanger | that's why it didn't work | 22:53 |
| ianw | pabelanger: hmm, that's an interesting one. the shell would still be calling out to that. i wonder if we should be setting the PATH to the current virtualenv in disk-image-create in https://git.openstack.org/cgit/openstack/diskimage-builder/tree/diskimage_builder/disk_image_create.py?h=feature/v2 | 22:55 |
| ianw | ahh, no maybe more to the point, disk-image-create should make sure it activates the virtualenv if it is one ... | 22:57 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Re-enable test: test_list_nodes https://review.openstack.org/400955 | 22:59 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Re-enable test: test_image_upload_fail https://review.openstack.org/400970 | 23:00 |
| SpamapS | jeblair: should we be tracking nodepool-only stories on the board too? | 23:03 |
| SpamapS | feels like we should | 23:03 |
| jlk | ianw: having the app activate the venv itself it it happens to be in one feels somewhat wrong. | 23:04 |
| SpamapS | explains why I haven't necessarily seen much work going on if those tasks are happening under the radar. | 23:04 |
| SpamapS | jlk: agreed | 23:04 |
| jeblair | SpamapS: yes -- i think the current effort just came in at a place where it didn't make sense to make stories for it | 23:04 |
| ianw | jlk: yeah, on second thought that's not necessary. because diskimage_builder.disk_image_create:main() calls sys.exec() with the current environ, the virtualenv PATH will be correct for the invoked script, so it will find dib-run-parts (and anything else in the environ) | 23:04 |
| jeblair | SpamapS: nothing is happening under the radar | 23:05 |
| SpamapS | jeblair: that's fine. I just want to make sure people can find meaningful work todo | 23:05 |
| SpamapS | eh, under the radar was a poor choice of words | 23:05 |
| jeblair | SpamapS: yeah, there isn't much fruit there, it's mopping up now | 23:05 |
| SpamapS | happening outside the view of sauron's evil eye? ;) | 23:05 |
| SpamapS | jeblair: ok, if there's not much for people to pick up, I'll just keep watch for more | 23:06 |
| pabelanger | ianw: Ya, it works for now. But could be a better way to do it | 23:06 |
| pabelanger | ianw: but I only spent 5mins on solving it | 23:06 |
| jeblair | SpamapS: ++ | 23:06 |
| SpamapS | apparently the storyboard folks are working on work boards that automatically build themselves from stories of a specific tag, btw, and have a lane per task status. | 23:06 |
| SpamapS | which would be _amazing_ | 23:06 |
| jeblair | SpamapS: oh yes | 23:06 |
| jeblair | that would be great | 23:07 |
| SpamapS | I've promised to be their first beta customer. :) | 23:07 |
| pabelanger | okay, if we can land https://review.openstack.org/#/c/403869/ we'll have nb01.o.o closer to working. But, now I run away for some food | 23:07 |
| ianw | pabelanger: i think calling it from python directly will work, but obviously exec'ing the nodepool-builder process won't. so we'll have to provide some sort of forking wrapper that gives access to the output | 23:07 |
| pabelanger | ianw: yup | 23:07 |
| jeblair | cause i think the main deficiency here is new stories or tasks not showing up in the board automatically | 23:07 |
| SpamapS | though one thing that lacks is there's no "on hold" status in tasks. | 23:07 |
| SpamapS | jeblair: yeah I'm having to just police the stories in the zuulv3 tag manually. No fun. | 23:08 |
| jeblair | i have to run an errand; back to finish reviews later | 23:09 |
| SpamapS | anyway, hopefully people can use what's there to find work. | 23:09 |
| *** saneax-_-|AFK is now known as saneax | 23:09 | |
| clarkb | jamielennox: ya would be good to confirm dib in general works if you just start the executable out of a venv before merging that | 23:13 |
| clarkb | jamielennox: but also if v2 is going to do python imports I think that negates a need for it in the first place too | 23:13 |
| jamielennox | clarkb: yea, i'm inclined to just wait for dib2 and to have it all in python | 23:14 |
| ianw | clarkb: i imagine it won't, because it won't find dib-run-parts unless it's in the regular path | 23:14 |
| ianw | sourcing the venv, then running will though | 23:14 |
| ianw | which i realise is the whole problem :) | 23:14 |
| clarkb | ianw: ya thats my experience | 23:14 |
| SpamapS | honestly, sourcing the venv is always required for anything more "scripty" than "library-y" | 23:15 |
| ianw | jamielennox: yeah, i'll look at making some sort of forking interface nodepool can use to invoke | 23:15 |
| jamielennox | unless someone wants to teach systemd about venvs :) | 23:16 |
| ianw | SpamapS: that's why my changes were on a topic "pythonic-dib" :) trying to make it more library-y | 23:16 |
| SpamapS | jamielennox: systemd can bash -c '. /venv/bin/activate ; program' | 23:16 |
| clarkb | Shrews: starting to review your nodepool stack at https://review.openstack.org/#/c/400836/2 and the way to create an alien image is to have the fake image list return an image called foo with $uuid | 23:17 |
| clarkb | Shrews: if the image doesn't have the right name/metadata nodepool treats it as alien | 23:17 |
| jamielennox | yea, that's true, anyway we do have this solved by setting VIRTUAL_ENV and PATH in the /etc/default file so it's not a big deal | 23:17 |
| jlk | Is there anyway to externally poke at gearman to see if it's running properly? | 23:27 |
| pabelanger | jamielennox: ya, best I did today was PREFIX in /etc/default, which is updated to point to venv/bin/nodepool-builder, then wrapper script for /usr/local/bin/disk-image-create. But going to revisit some of the other suggestions here | 23:30 |
| pabelanger | then ExecStart=/bin/sh -c "${PREFIX}/bin/nodepool-builder -d ${ARGS}" | 23:31 |
| pabelanger | for systemd | 23:31 |
| jamielennox | jlk: yea, nc 127.0.0.1 4730 then status | 23:31 |
| jamielennox | pabelanger: so all venv should do is set the PATH and the VIRTUAL_ENV env vars, so i think setting those in /etc/default is reasonable | 23:33 |
| clarkb | Shrews: ok managed to review that change | 23:33 |
| jamielennox | you can't do PATH=/path/to/venv/bin:$PATH in systemd, but PATH is static so you can know in advance what it is and set the whole line | 23:33 |
| clarkb | jlk: yes telnet/nc to it then you can run the admin commands | 23:34 |
| clarkb | jlk: 'status' and 'workers' are the two I know off the top of my head | 23:34 |
| clarkb | jlk: should be documented inthe protocol docs though | 23:34 |
| pabelanger | jamielennox: yup, I've also floated the idea to myself to just call a big ole wrapper script to start nodepool-builder, over trying to write things the systemd way. But, I'm trying to give systemd a chance and do the right(?) thing | 23:35 |
| *** saneax is now known as saneax-_-|AFK | 23:39 | |
| clarkb | Shrews: ok I have reviewed those three changes, comments on two of them though I only -1'd the one | 23:53 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!