| clarkb | trying to run stestr directly from the tox venv says that there is no subunit installed and fails. So now I'm passing all the args through tox | 00:00 |
|---|---|---|
| clarkb | I think this is mostly working except for the it fails part :) | 00:01 |
| clarkb | maybe I need tox -r to pull in the new stuff | 00:01 |
| clarkb | that was it I can reproduce now | 00:04 |
| clarkb | if the openstack sdk integration tests run before tests that wait for image we fail beacuse there is a real-cloud thread running. Now t ofigure out where/why that thread is running | 00:04 |
| clarkb | ok I think I see the bug | 00:07 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool master: Don't let openstacksdk run a taskmanager during testing https://review.openstack.org/612835 | 00:16 |
| clarkb | that should fix this bug | 00:16 |
| clarkb | one thing that kept tripping me up is that sdk runs a taskmanager using the cloud name in cloud config as the thread name | 00:22 |
| clarkb | nodepool uses the provider name | 00:22 |
| clarkb | thats a difference we'll want to sort out if we convert to the sdk taskmanager | 00:22 |
| clarkb | (I think cloud name is fine, except that it isn't region specific. That should probably be a more unique key) | 00:22 |
| clarkb | py35 and py36 both pass on ^ which is a rarity (so I think this does fix it) I will now go find dinner | 00:28 |
| ianw | clarkb: ++ great find! | 00:29 |
| SpamapS | http://logs.openstack.org/24/612824/1/check/zuul-quick-start/5757b3a/job-output.txt.gz#_2018-10-23_23_08_00_561428 | 00:36 |
| SpamapS | common fail? | 00:36 |
| SpamapS | Request failed on fetching the status page during zuul-quick-start | 00:36 |
| clarkb | SpamapS: that is unrelated to the bug I just fixed in nodepool. I'm not sure if that is a common failure or not | 00:37 |
| SpamapS | hrm | 00:37 |
| SpamapS | definitely seems unrelated to the patch itself. | 00:37 |
| clarkb | SpamapS: if zuul-web couldn't start due to config read failure, that might cause it? | 00:38 |
| SpamapS | true | 00:39 |
| clarkb | SpamapS: that example conf file is likely used by the docker compose setup | 00:39 |
| SpamapS | ah so maybe my messing with it breaks that | 00:39 |
| clarkb | or at least it is possible that is the case. A lot of the xample stuff is used by docker compose | 00:39 |
| * SpamapS looks closer | 00:39 | |
| SpamapS | ah cool I think I actually just need to set the password | 00:43 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul master: Add the process environment to zuul.conf parser https://review.openstack.org/612824 | 00:44 |
| * SpamapS squints at zookeeper and remembers it has no auth and then huffs and walks away for a minute | 00:46 | |
| SpamapS | >:| | 00:46 |
| dmsimard | is there anything locking ourselves into zookeeper ? genuinely curious | 00:48 |
| SpamapS | Just a painful backward-incompatible upgrade | 00:50 |
| SpamapS | we *could* test with zetcd and have a transition period | 00:50 |
| SpamapS | and to be clear, zk *does* have auth options | 00:51 |
| SpamapS | they just aren't simple or awesome. | 00:51 |
| dmsimard | had never heard about zetcd before, interesting | 00:52 |
| clarkb | I really like zk... | 00:55 |
| clarkb | etcd has a lot of problems with python granted they dont affect zuul mich | 00:55 |
| clarkb | yay for not using asyncio or eventlet I guess | 00:55 |
| dmsimard | sometimes the grass isn't always greener | 00:56 |
| dmsimard | :) | 00:56 |
| clarkb | etcd v2 lacks features compared to zk and v3 is grpc which makes some python stuff weird because they only support real threads | 00:57 |
| clarkb | which zuul and nodepool do so less an issue here now that the asyncion was removed | 00:57 |
| SpamapS | yeah I don't love how grpc requires you to buy in to all of these things in a single implementation | 00:59 |
| SpamapS | like, protobufs, k, http2, k, oauth2, k... but.. they forced all 3 of those things into a custom protobuf compiler so now if you want to just use your own http2 or oauth2... nope. | 00:59 |
| dmsimard | yay golang | 01:00 |
| SpamapS | actually IIRC grpc came from C++ folks | 01:05 |
| SpamapS | and then python | 01:05 |
| SpamapS | and then go | 01:05 |
| dmsimard | Protobufs was go I thought | 01:18 |
| dmsimard | Nevermind | 01:18 |
| mordred | dmsimard: yeah - we used protobuf back in drizzle | 02:56 |
| *** bhavikdbavishi has joined #zuul | 02:56 | |
| mordred | clarkb: +A on the nodepool test fix ... we actually have a few patches in flight which should simply that whole thing and actually get rid of the task manager thread completely | 03:00 |
| mordred | clarkb: https://review.openstack.org/#/c/605043/ is the proposed replacement implementation - next up for that is writing tests | 03:00 |
| openstackgerrit | Merged openstack-infra/nodepool master: Don't let openstacksdk run a taskmanager during testing https://review.openstack.org/612835 | 03:21 |
| *** spsurya has joined #zuul | 04:24 | |
| *** nilashishc has joined #zuul | 04:35 | |
| *** bhavikdbavishi has quit IRC | 04:35 | |
| *** bhavikdbavishi has joined #zuul | 04:36 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Normalise more of the API stats calls https://review.openstack.org/605898 | 04:57 |
| *** chkumar|off is now known as chandankumar | 05:03 | |
| *** rlandy|bbl has quit IRC | 05:21 | |
| *** sshnaidm|pto is now known as sshnaidm|ruck | 05:45 | |
| tristanC | corvus: regarding zuul configuration, maybe we should consider being able to add connections using the rest api, similarly for projects, pipelines, jobs, ... | 06:51 |
| tristanC | that way we could have a web wizard and it may simplify automated post-conf, whatever the deployment method used | 06:52 |
| tristanC | clarkb: regarding zuul in openshift, iiuc pods are further restricted and may not be able to perform regular user operation, like creating nested namespaces | 06:54 |
| *** bhavikdbavishi has quit IRC | 07:02 | |
| *** bhavikdbavishi has joined #zuul | 07:03 | |
| *** pcaruana has joined #zuul | 07:06 | |
| *** threestrands has quit IRC | 07:13 | |
| *** bhavikdbavishi has quit IRC | 07:25 | |
| *** themroc has joined #zuul | 07:48 | |
| *** jpena|off is now known as jpena | 07:48 | |
| *** ianychoi_ is now known as ianychoi | 08:03 | |
| *** bhavikdbavishi has joined #zuul | 08:08 | |
| *** hashar has joined #zuul | 08:18 | |
| *** jesusaur has quit IRC | 09:45 | |
| *** jesusaur has joined #zuul | 09:49 | |
| *** panda has quit IRC | 09:54 | |
| *** panda has joined #zuul | 09:55 | |
| *** bhavikdbavishi has quit IRC | 11:18 | |
| openstackgerrit | Simon Westphahl proposed openstack-infra/zuul master: Fix issue in Github connection with large diffs https://review.openstack.org/612989 | 11:27 |
| *** jpena is now known as jpena|lunch | 11:36 | |
| openstackgerrit | Simon Westphahl proposed openstack-infra/zuul master: Fix issue in Github connection with large diffs https://review.openstack.org/612989 | 11:45 |
| openstackgerrit | Merged openstack-infra/nodepool master: Reduce socket connect timeout in nodescan https://review.openstack.org/608897 | 12:14 |
| openstackgerrit | Simon Westphahl proposed openstack-infra/zuul master: Fix issue in Github connection with large diffs https://review.openstack.org/612989 | 12:14 |
| *** rlandy has joined #zuul | 12:31 | |
| *** rfolco|rucker is now known as rfolco|rover | 12:34 | |
| *** jpena|lunch is now known as jpena | 12:37 | |
| Shrews | clarkb: looks like the tmpfs change did not have any impact on the zookeeper test failures, but i think it's still worth keeping | 12:39 |
| Shrews | dstat graphs don't show anything concerning either | 12:47 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: DNM: testing zookeeper oddities https://review.openstack.org/612750 | 12:50 |
| *** hashar is now known as hasharAway | 13:07 | |
| corvus | tristanC: i think it's fine for someone to write a web wizard which then writes out the config file. i don't think zuul needs to do that. | 13:09 |
| Shrews | corvus: is the --change option to zuul's autohold the change id (e.g., Ie92...), or the review # (e.g, 612750)? | 13:10 |
| Shrews | i could not discern that from the code | 13:11 |
| corvus | Shrews: number | 13:12 |
| Shrews | i can't tell you how often that confuses me :( | 13:13 |
| tristanC | corvus: that implies that zuul-web run on the scheduler node... wouldn't it be better if such configuration happens over gearman? | 13:13 |
| corvus | tristanC: no i'm saying that i don't think zuul needs to handle this. it's very deliberately a git-driven system. if someone wants to hide that behind a web interface, that's fine, they can write that and it can output config files that zuul can read. | 13:16 |
| corvus | put another way: the api for zuul's config is git | 13:17 |
| corvus | anything can write to that api -- human or machine | 13:18 |
| tristanC | corvus: how about connections defined in zuul.conf and tenant configurations in zuul.yaml? | 13:20 |
| corvus | tristanC: we already landed the change which lets zuul.yaml come from an external process. | 13:23 |
| tristanC | corvus: yes, but it doesn't seems to be enough for SpamapS use-case... | 13:25 |
| corvus | i thought we found a solution for SpamapS's use case | 13:25 |
| tobiash | corvus: do you cache normal git workspaces or bare repos in the nodes? | 13:32 |
| tobiash | corvus: I'm currently working on a generic way to use git to push repos to the nodes | 13:32 |
| tobiash | and have a working proof of concept | 13:33 |
| corvus | tobiash: normal i believe | 13:33 |
| corvus | i'll double check | 13:33 |
| corvus | yes, normal | 13:34 |
| tobiash | corvus: I used bare repos for start, but I think that won't matter as long as we look in the right path for the repo | 13:34 |
| tobiash | corvus, mordred: I also noticed that the buildlog seems to be buggy with loops | 13:35 |
| tobiash | it worked on two repos but the log only contains one: http://paste.openstack.org/show/732963/ | 13:37 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Collect docker logs after quick-start run https://review.openstack.org/613027 | 13:46 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add the process environment to zuul.conf parser https://review.openstack.org/612824 | 13:46 |
| corvus | SpamapS, clarkb: ^ maybe that will help | 13:46 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul-jobs master: Add prepare-workspace-git role https://review.openstack.org/613036 | 14:04 |
| tobiash | corvus: this should work generically ^ | 14:06 |
| tobiash | corvus: a change that is currently incompatible to openstack is that this uses the canonical name for the cached repos | 14:07 |
| tobiash | but other than that it should be compatible as a drop-in replacement of the 'use-cached-repos/mirror-workspace-git-repos' dance | 14:08 |
| tobiash | and it also should be compatible as a drop-in replacement of prepare-workspace so users can choose to cache or not to cache just by placing the repos into the image or not | 14:10 |
| corvus | tobiash: great, thanks! | 14:28 |
| corvus | i thought we did use the canonical name on our images... at any rate, that should be easy for us to fix | 14:28 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: DNM: testing zookeeper oddities https://review.openstack.org/612750 | 14:42 |
| clarkb | Shrews: ya I found the bug in the nodepool test suite and fixed it | 14:49 |
| clarkb | Shrews: https://review.openstack.org/#/c/612835/ should fix a large number of nodepool test errors | 14:49 |
| *** pbrobinson has joined #zuul | 14:49 | |
| Shrews | clarkb: i don't see how that affects zk connections though | 14:51 |
| clarkb | Shrews: it doesn't, but I think the zk connection suspending was just noise | 14:51 |
| clarkb | I dug into the test failures I had run into and they were all this wait for image timeout happening beacuse wait_for_threads() was stuck on an extra thread | 14:51 |
| clarkb | Shrews: https://review.openstack.org/#/c/612828/ was the last major piece of my debugging puzzle | 14:53 |
| clarkb | Shrews: then I was able to reproduce by running the sdk integrations test before the webapp tests | 14:53 |
| corvus | maybe the suspending was just part of the normal shutdown (which then was halted waiting for the openstacksdk thread) | 14:55 |
| corvus | nonetheless, i'm very happy to have zk running on tmpfs in the gate. i do that locally for tests :) | 14:55 |
| Shrews | corvus: as do i | 14:56 |
| Shrews | mostly because i forget to shut it down | 14:56 |
| clarkb | Should I rebase https://review.openstack.org/#/c/612816/ so that it doesn't depend on the dstat work? | 14:57 |
| Shrews | i'd say no. i want both | 14:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Merger: automatically add new hosts to the known_hosts file https://review.openstack.org/608453 | 14:59 |
| corvus | Shrews, clarkb: the dstat stuff isn't mergable as-is -- it clones something from github. we need to either find a real way to do that (package and release it?) or switch to sar. i propose we un-base on dstat, wait for tobiash to push up his sar stuff, and see if we can incorporate that. if that doesn't work, i'll look into solutions for dstat | 15:02 |
| clarkb | ok I'll rebase then | 15:03 |
| tobiash | corvus: I'll see if I can push up the sar stuff later today | 15:10 |
| *** bjackman has quit IRC | 15:19 | |
| *** themroc has quit IRC | 15:45 | |
| *** pcaruana has quit IRC | 15:49 | |
| *** samccann_ is now known as samccann | 15:56 | |
| *** hasharAway is now known as hashar | 16:02 | |
| *** bhavikdbavishi has joined #zuul | 16:19 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool master: Run test zookeeper on top of tmpfs https://review.openstack.org/612816 | 16:20 |
| clarkb | rebased without dstat stuff | 16:20 |
| *** pcaruana has joined #zuul | 16:46 | |
| SpamapS | dstat? | 16:48 |
| SpamapS | I think I have commit access to that. | 16:48 |
| clarkb | SpamapS: changes to zuul and nodepool to run dstat and graph the results per job, but tobiash has some other thing they run on all jobs that apparently works great we'll look at too | 16:49 |
| SpamapS | dstat is pretty gross inside | 16:53 |
| SpamapS | I wined to dag a couple years back and he granted me and some other people commit access so we could fix it | 16:54 |
| SpamapS | but by that time I had moved on | 16:54 |
| *** panda is now known as panda|off | 16:59 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul-jobs master: WIP: Add sar logging roles https://review.openstack.org/613112 | 17:07 |
| *** irclogbot_8 has joined #zuul | 17:08 | |
| *** hashar is now known as hasharDinner | 17:08 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: Enable sar logging for unit tests https://review.openstack.org/613117 | 17:13 |
| tobiash | let's see if that works, had to rework that a bit to make it compatible ^ | 17:15 |
| *** jpena is now known as jpena|off | 17:15 | |
| tobiash | hrm, I guess I have to wait quite some time to get a node to see if it's working... | 17:17 |
| *** irclogbot_8 has quit IRC | 17:21 | |
| *** irclogbot_8 has joined #zuul | 17:22 | |
| tobiash | corvus: any idea how I could share here an svg example file of ^ ? | 17:23 |
| clarkb | tobiash: imgur? we had ti stoo rendering in gerrit iirc | 17:28 |
| tobiash | hrm, didn't work | 17:29 |
| *** bhavikdbavishi has quit IRC | 17:41 | |
| *** bhavikdbavishi1 has joined #zuul | 17:41 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Collect docker logs after quick-start run https://review.openstack.org/613027 | 18:03 |
| *** electrofelix has quit IRC | 18:05 | |
| corvus | hrm... that change, which touches no python code, encountered 283 pep8 errors | 18:07 |
| corvus | i guess there was a new flake8 release | 18:08 |
| tobiash | yes | 18:11 |
| corvus | wow there's some really bad rules | 18:14 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Collect docker logs after quick-start run https://review.openstack.org/613027 | 18:19 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add the process environment to zuul.conf parser https://review.openstack.org/612824 | 18:19 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Pin flake8 https://review.openstack.org/613134 | 18:19 |
| *** nilashishc has quit IRC | 18:22 | |
| Shrews | ooh, this new X1 laptop is soooo much lighter than the T450s it's replacing | 18:37 |
| *** ssbarnea|bkp2 has joined #zuul | 18:39 | |
| *** dkehn_ has joined #zuul | 18:42 | |
| *** D0han_ has joined #zuul | 18:43 | |
| *** tobiash_ has joined #zuul | 18:43 | |
| *** jhesketh_ has joined #zuul | 18:44 | |
| *** jesusaur has quit IRC | 18:48 | |
| *** tobiash has quit IRC | 18:48 | |
| *** ssbarnea has quit IRC | 18:48 | |
| *** jlk has quit IRC | 18:48 | |
| *** dkehn has quit IRC | 18:48 | |
| *** SpamapS has quit IRC | 18:48 | |
| *** SotK has quit IRC | 18:48 | |
| *** D0han has quit IRC | 18:48 | |
| *** Diabelko has quit IRC | 18:48 | |
| *** jhesketh has quit IRC | 18:48 | |
| *** dkehn_ is now known as dkehn | 18:48 | |
| *** Diabelko has joined #zuul | 18:49 | |
| *** D0han_ is now known as D0han | 18:49 | |
| *** jesusaur has joined #zuul | 18:49 | |
| *** SpamapS has joined #zuul | 18:50 | |
| *** jlk has joined #zuul | 18:56 | |
| *** pwhalen has joined #zuul | 19:05 | |
| *** openstackgerrit has quit IRC | 19:06 | |
| *** openstackgerrit has joined #zuul | 19:10 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: support foreign required-projects https://review.openstack.org/613143 | 19:10 |
| mordred | corvus: ^^ ooh | 19:12 |
| corvus | when i get that fleshed out, i'll try doing a third-party ci setup using the docker-compose stuff and see if it's less terrible | 19:14 |
| *** irclogbot_8 has quit IRC | 19:25 | |
| *** irclogbot_8 has joined #zuul | 19:25 | |
| *** irclogbot_8 has quit IRC | 19:26 | |
| *** irclogbot_8 has joined #zuul | 19:27 | |
| *** irclogbot_8 has quit IRC | 19:31 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Run test zookeeper on top of tmpfs https://review.openstack.org/612816 | 20:06 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Filter file comments for existing files https://review.openstack.org/613161 | 20:49 |
| corvus | mordred: ^ | 20:50 |
| *** pcaruana has quit IRC | 20:50 | |
| *** openstack has quit IRC | 21:03 | |
| *** openstack has joined #zuul | 21:07 | |
| *** ChanServ sets mode: +o openstack | 21:07 | |
| *** spsurya has quit IRC | 21:21 | |
| *** hasharDinner has quit IRC | 21:48 | |
| *** rlandy is now known as rlandy|bbl | 22:31 | |
| openstackgerrit | Merged openstack-infra/zuul master: Pin flake8 https://review.openstack.org/613134 | 23:15 |
| *** openstackgerrit has quit IRC | 23:20 | |
| *** jesusaur has quit IRC | 23:34 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!