| *** logan- has quit IRC | 00:02 | |
| *** logan- has joined #zuul | 00:02 | |
| *** JasonCL has quit IRC | 00:03 | |
| *** JasonCL has joined #zuul | 00:08 | |
| *** JasonCL has quit IRC | 00:17 | |
| *** odyssey4me has quit IRC | 00:34 | |
| *** odyssey4me has joined #zuul | 00:34 | |
| dmsimard | I wrote a blog post about Zuul, ARA and OpenStack: https://dmsimard.com/2018/04/09/scaling-ara-to-a-million-ansible-playbooks-a-month/ | 00:52 |
|---|---|---|
| dmsimard | (and Ansible of course) | 00:53 |
| dmsimard | Superuser would like to feature it on their blog. Is that okay ? | 00:53 |
| dmsimard | clarkb, fungi, mordred, corvus: ^ | 00:54 |
| pabelanger | seems like good publicly for ARA | 01:06 |
| pabelanger | I see a link to zuul-ci.org too :) | 01:07 |
| *** JasonCL has joined #zuul | 01:12 | |
| *** JasonCL has quit IRC | 01:14 | |
| Shrews | corvus: ooh, https://review.openstack.org/559326 merging will cause tests to flap i believe. i needed to fix the port in use by the new test to not conflict, but dinner and EOD got in the way | 01:16 |
| Shrews | i'll try to toss something up now | 01:16 |
| *** JasonCL has joined #zuul | 01:20 | |
| *** JasonCL has quit IRC | 01:27 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Fix streaming test ports https://review.openstack.org/559875 | 01:45 |
| Shrews | ^^ temporary fix. asyncio doesn't seem to like me passing it port 0 for some odd reason | 01:45 |
| Shrews | if it becomes urgent, just merge that. otherwise, i'll look at a better solution tomorrow unless someone beats me to it | 01:47 |
| *** trishnag has quit IRC | 02:13 | |
| *** trishnag has joined #zuul | 02:16 | |
| *** trishnag has quit IRC | 02:16 | |
| *** trishnag has joined #zuul | 02:16 | |
| *** jimi|ansible has quit IRC | 03:10 | |
| *** jimi|ansible has joined #zuul | 03:12 | |
| *** harlowja has quit IRC | 03:16 | |
| *** Canaimero-15d has joined #zuul | 03:27 | |
| Canaimero-15d | hola | 03:28 |
| *** Canaimero-15d has quit IRC | 03:32 | |
| *** harlowja has joined #zuul | 03:57 | |
| *** bhavik1 has joined #zuul | 04:03 | |
| *** bhavik1 has quit IRC | 04:08 | |
| ianw | pabelanger: the nodepool boot tests are seeing a lot of issues with zk connection losses causing randomish failures; i think due to slow nodes. maybe it's time to turn it into a two-node test ... | 04:28 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Update Github3.py to 1.1.0 https://review.openstack.org/559798 | 05:01 |
| openstackgerrit | Merged openstack-infra/zuul master: Add license and downgrade exception to alembic template https://review.openstack.org/554055 | 05:29 |
| mrhillsman | sean-k-mooney if you have not figured out the implicit role not found issue i think i may know what's going on but it is late for me so i will see if my theory is correct in the morning and check with you if you have figured it out | 05:36 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Prefix existing indexes https://review.openstack.org/559910 | 06:30 |
| *** harlowja has quit IRC | 06:40 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Also prefix the indexes if needed https://review.openstack.org/535718 | 06:48 |
| *** hashar has joined #zuul | 07:07 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Prefix existing indexes https://review.openstack.org/559910 | 07:17 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Also prefix the indexes if needed https://review.openstack.org/535718 | 07:17 |
| *** AJaeger has quit IRC | 07:31 | |
| *** jhesketh_ has joined #zuul | 07:31 | |
| *** AJaeger has joined #zuul | 07:36 | |
| *** jhesketh has quit IRC | 07:37 | |
| *** jpena|off is now known as jpena | 07:45 | |
| *** electrofelix has joined #zuul | 07:47 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Prefix existing indexes https://review.openstack.org/559910 | 08:05 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Also prefix the indexes if needed https://review.openstack.org/535718 | 08:05 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Prefix existing indexes https://review.openstack.org/559910 | 08:22 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Also prefix the indexes if needed https://review.openstack.org/535718 | 08:22 |
| *** jesusaur has quit IRC | 08:23 | |
| *** jesusaur has joined #zuul | 08:26 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix sql reporting with postgres https://review.openstack.org/535720 | 08:26 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add tests for postgresql https://review.openstack.org/549518 | 08:26 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add tests for postgresql https://review.openstack.org/549518 | 08:28 |
| *** JasonCL has joined #zuul | 08:31 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Prefix existing indexes https://review.openstack.org/559910 | 08:33 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Also prefix the indexes if needed https://review.openstack.org/535718 | 08:33 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix sql reporting with postgres https://review.openstack.org/535720 | 08:33 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add tests for postgresql https://review.openstack.org/549518 | 08:33 |
| *** ssbarnea_ has joined #zuul | 08:43 | |
| *** ssbarnea_ has quit IRC | 08:51 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Prefix existing indexes https://review.openstack.org/559910 | 09:05 |
| openstackgerrit | Felix Schmidt proposed openstack-infra/zuul-sphinx master: Make the yaml parser aware of '!encrypted/' tags https://review.openstack.org/559963 | 09:08 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Prefix existing indexes https://review.openstack.org/559910 | 09:19 |
| sean-k-mooney | mrhillsman: hi i have not had time to valideate the issues yet. reading back through http://eavesdrop.openstack.org/irclogs/%23zuul/%23zuul.2018-03-27.log.html#t2018-03-27T14:07:45 and https://etherpad.openstack.org/p/sqJ1DzLmSU i think this is at least part of my issue | 09:23 |
| sean-k-mooney | following the zuul from scratch docs gets you 90% of the way there. at least in my case i was able to get to the point that the noop jobs worked. the last 10% will is the tricky bit :) | 09:28 |
| *** jesusaur has quit IRC | 10:02 | |
| *** jesusaur has joined #zuul | 10:15 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Prefix existing indexes https://review.openstack.org/559910 | 10:30 |
| *** JasonCL has quit IRC | 10:35 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Prefix existing indexes https://review.openstack.org/559910 | 10:42 |
| *** sshnaidm is now known as sshnaidm|bbl | 10:47 | |
| *** sshnaidm|bbl has quit IRC | 10:47 | |
| *** JasonCL has joined #zuul | 10:50 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Prefix existing indexes https://review.openstack.org/559910 | 10:55 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Prefix existing indexes https://review.openstack.org/559910 | 10:57 |
| *** sshnaidm|bbl has joined #zuul | 11:27 | |
| *** JasonCL has quit IRC | 11:29 | |
| sigmavirus24 | jlk: tobiash github3.py 1.1.0 is released to fix the bug y'all were running into | 11:33 |
| tobiash | sigmavirus24: yay, thank you :) | 11:34 |
| sigmavirus24 | Also if anyone has a good way of getting in touch with the Bandit devs that isn't openstack-dev, I'd greatly appreciate it. They wanted to move bandit to github.com/pycqa | 11:34 |
| sigmavirus24 | (lhinds isn't anwering in #openstack-security and Eric hasn't answered a single email since February :( ) | 11:34 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Prefix existing indexes https://review.openstack.org/559910 | 11:36 |
| AJaeger | sigmavirus24: That might be more for the tc channel. One suggestion: openstack-dev mailing list gives a record for reference, I would use that as additional means. | 11:36 |
| sigmavirus24 | Ah good point. I had forgotten to check if the TC was even considering this :) | 11:37 |
| sigmavirus24 | Amazing what a year away will atrophy in one's brain ;) | 11:39 |
| AJaeger | sigmavirus24: if a PTL - and lhinds is PTL - is not reacting, then yes, I would reach out to TC. | 11:42 |
| *** jpena is now known as jpena|lunch | 11:45 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Prefix existing indexes https://review.openstack.org/559910 | 11:52 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Also prefix the indexes if needed https://review.openstack.org/535718 | 11:52 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix sql reporting with postgres https://review.openstack.org/535720 | 11:52 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add tests for postgresql https://review.openstack.org/549518 | 11:52 |
| openstackgerrit | Benedikt Löffler proposed openstack-infra/zuul master: Add role information to task in zuul_json callback https://review.openstack.org/559998 | 11:52 |
| *** sshnaidm|bbl is now known as sshnaidm | 11:53 | |
| *** odyssey4me has quit IRC | 12:00 | |
| *** odyssey4me has joined #zuul | 12:00 | |
| *** weshay_pto is now known as weshay | 12:03 | |
| *** openstackgerrit has quit IRC | 12:04 | |
| *** dkranz has joined #zuul | 12:20 | |
| *** gouthamr has joined #zuul | 12:23 | |
| *** rlandy has joined #zuul | 12:27 | |
| *** _ari_ is now known as _ari_|conf | 12:29 | |
| *** gouthamr has quit IRC | 12:43 | |
| *** openstackgerrit has joined #zuul | 12:43 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Prefix existing indexes https://review.openstack.org/559910 | 12:43 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Also prefix the indexes if needed https://review.openstack.org/535718 | 12:43 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix sql reporting with postgres https://review.openstack.org/535720 | 12:43 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add tests for postgresql https://review.openstack.org/549518 | 12:43 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: test https://review.openstack.org/560013 | 12:43 |
| *** gouthamr has joined #zuul | 12:44 | |
| *** gouthamr has quit IRC | 12:45 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: test2 https://review.openstack.org/560014 | 12:46 |
| openstackgerrit | Felix Schmidt proposed openstack-infra/zuul-sphinx master: Make the yaml parser aware of '!encrypted/' tags https://review.openstack.org/559963 | 12:47 |
| *** jpena|lunch is now known as jpena | 12:53 | |
| *** logan- has quit IRC | 12:55 | |
| *** logan- has joined #zuul | 12:56 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: test https://review.openstack.org/560013 | 12:58 |
| *** nhicher has quit IRC | 13:00 | |
| *** nhicher has joined #zuul | 13:00 | |
| *** JasonCL has joined #zuul | 13:08 | |
| fungi | dmsimard: excellent article! i'd be thrilled for it to get syndicated on su | 13:12 |
| *** JasonCL has quit IRC | 13:13 | |
| dmsimard | The intent was to give a little bit of advertising to Zuul, hope it didn't come across otherwise | 13:13 |
| fungi | sigmavirus24: the bandit move has been getting discussed at the weekly openstack security sig meeting (1500utc in the #openstack-meeting channel) | 13:13 |
| fungi | sigmavirus24: also there's a #openstack-security channel though it's not terribly active | 13:14 |
| fungi | heh, and now i see you've been trying to reach them in there already | 13:15 |
| *** JasonCL has joined #zuul | 13:18 | |
| *** Wei_Liu has quit IRC | 13:23 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Prefix existing indexes https://review.openstack.org/559910 | 13:24 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Also prefix the indexes if needed https://review.openstack.org/535718 | 13:24 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix sql reporting with postgres https://review.openstack.org/535720 | 13:24 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add tests for postgresql https://review.openstack.org/549518 | 13:24 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Fix streaming test ports https://review.openstack.org/559875 | 13:29 |
| tobiash | Shrews: added a (nit) comment ^ | 13:34 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Make gearman calls async in ZuulWeb https://review.openstack.org/560026 | 13:34 |
| Shrews | tobiash: oh, oops | 13:35 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Fix streaming test ports https://review.openstack.org/559875 | 13:35 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Make gearman calls async in ZuulWeb https://review.openstack.org/560026 | 13:36 |
| tobiash | corvus, clarkb: I think the postgres stack should be ready for review now. I created the index rename change as first change which tests the mysql index renaming and stacked the other postgres changes on top of it. That tests the renamed index and also the whole thing with postgres. | 13:44 |
| tobiash | mordred: don | 13:47 |
| tobiash | mordred: don't we have jobs on zuul-sphinx? | 13:48 |
| mordred | tobiash: I thought we did | 13:50 |
| mordred | tobiash: nope | 13:51 |
| mordred | tobiash: we have publish jobs defined, but no test jobs | 13:52 |
| tobiash | nothing seems to be triggered there and there are only post jobs defined | 13:52 |
| corvus | and i'm not sure the publish jobs work (unsurprising if the check jobs don't exist) | 13:53 |
| mordred | tobiash: maybe we should put together a job that builds zuul-jobs and openstack-zuul-jobs docs jobs against zuul-sphinx patches | 13:53 |
| mordred | of course, that would require me fixing siblings support for sphinx jos | 13:53 |
| mordred | jobs | 13:53 |
| mordred | s/me// | 13:53 |
| mordred | tobiash: pg patches look good to me - I left a comment on the first one. I don't think it *actually* matters given sizes of current data sets | 13:54 |
| *** fdegir has quit IRC | 13:54 | |
| *** fdegir has joined #zuul | 13:54 | |
| corvus | mordred: it might to openstack-infra? | 13:55 |
| mordred | corvus: even with openstack-infra I'm not sure it's super important - it would only impact people browsing builds in the dashboard | 13:56 |
| corvus | true | 13:56 |
| mordred | or, rather, searching for specific builds | 13:56 |
| mordred | (browsing being a table scan anyway) | 13:56 |
| Shrews | trying to run the test_web test suite in Zuul locally, but getting: AssertionError: False is not true : Static web assets are missing, be sure to run 'npm run build' | 13:57 |
| corvus | Shrews: you want https://zuul-ci.org/docs/zuul/developer/javascript.html | 13:58 |
| corvus | ooh, we should change that exception message to include that link :) | 13:59 |
| dmsimard | mordred, corvus: thoughts on my late night ping I did last night about superuser.openstack.org ? | 13:59 |
| mordred | ++ | 13:59 |
| tobiash | mordred, corvus: changed the ordering locally. I can push this up or wait for further reviews first. | 14:00 |
| corvus | dmsimard: great idea :) | 14:00 |
| Shrews | well that looks like a lot of work | 14:00 |
| corvus | tobiash: i'm on change #2 right now | 14:00 |
| corvus | Shrews: do you have docker installed? if so, it's just "npm run build:docker" | 14:00 |
| Shrews | corvus: no | 14:01 |
| corvus | er, rather, if you have npm installed | 14:01 |
| mordred | dmsimard: I love it - although if I were to provide any feedback on the content, I'd remove the words 'on AWS' from 'How much does 300 000 minutes (or 208 days) of compute costs you at AWS ?' | 14:01 |
| dmsimard | mordred: that's totally fair | 14:01 |
| dmsimard | mordred: It was a tongue in cheek thing but the effect is the same with AWS taken out | 14:02 |
| corvus | Shrews: simplest thing is to install docker and run the second command there. | 14:02 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Make gearman calls async in ZuulWeb https://review.openstack.org/560026 | 14:02 |
| mordred | dmsimard: it's a fantastic post though - you should totally transform it into a conference talk | 14:02 |
| Shrews | corvus: or abuse our zuul :) | 14:02 |
| dmsimard | mordred: well it's not like I submitted a talk about ara in vancouver or anything :( | 14:02 |
| corvus | Shrews: or, if you want a bunch of stuff installed locally on your workstation, run tools/install-js-tools.sh (that's what i did) | 14:02 |
| mordred | dmsimard: because there are great lessons in there about the costs/benefits of static site generation vs. dynamic backends | 14:02 |
| corvus | mordred: well, there's a lesson about using static site generation for a *database* | 14:03 |
| mordred | corvus, Shrews: yarn and npm (from the apt/yum repos in install-js-tools.sh are the only things that actually need to be installed on your laptop - everything else is self-contained inside of the source dir | 14:03 |
| dmsimard | mordred: I fixed the aws part, thanks for that | 14:03 |
| *** swest has quit IRC | 14:05 | |
| mordred | corvus: yah - I think I'd say though that it can be extrapolated. the underlying point being that neither pre-generating all content nor dynamically generating all content are silver bullets and that analyzing actual data size, costs and usage patterns can be valuable | 14:05 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix streaming test ports https://review.openstack.org/559875 | 14:05 |
| mordred | dmsimard: also :( re: vancouver | 14:06 |
| sigmavirus24 | fungi: thanks for tat. Just noticed your ping after bugging the -tc | 14:07 |
| corvus | tobiash: all pg changes lgtm. i think we can quickly re-review after you push up the change to the first. | 14:08 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Prefix existing indexes https://review.openstack.org/559910 | 14:09 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Also prefix the indexes if needed https://review.openstack.org/535718 | 14:09 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix sql reporting with postgres https://review.openstack.org/535720 | 14:09 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add tests for postgresql https://review.openstack.org/549518 | 14:09 |
| tobiash | corvus: ok | 14:09 |
| corvus | mordred: https://review.openstack.org/559910 | 14:11 |
| *** patrickeast has quit IRC | 14:11 | |
| *** patrickeast has joined #zuul | 14:11 | |
| corvus | tobiash: you can unwip and recheck https://review.openstack.org/559798 now | 14:12 |
| tobiash | corvus: let's see if the release made it into the mirrors now | 14:12 |
| corvus | oh that | 14:12 |
| corvus | tobiash: well, you can at least unwip it :) | 14:13 |
| tobiash | the last recheck was already after the gh3.py release | 14:13 |
| *** mnaser has quit IRC | 14:15 | |
| tobiash | ok, live log looks good | 14:16 |
| *** mnaser has joined #zuul | 14:16 | |
| corvus | tobiash: you may also be interested in https://review.openstack.org/559340 | 14:17 |
| corvus | that might be good to sneak into the next release | 14:17 |
| tobiash | looking | 14:18 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Make gearman calls async in ZuulWeb https://review.openstack.org/560026 | 14:18 |
| *** elyezer has joined #zuul | 14:18 | |
| tobiash | corvus: do we want a reno entry there is is this considered to just align implementation to the docs? | 14:18 |
| *** gouthamr has joined #zuul | 14:19 | |
| corvus | tobiash: i'm okay skipping reno since it's aligning with the docs, but don't object if folks want to add one | 14:19 |
| tobiash | I'm ok with that, just wanted to ask :) | 14:20 |
| *** sdoran has quit IRC | 14:23 | |
| tobiash | corvus: lgtm | 14:24 |
| *** sdoran has joined #zuul | 14:24 | |
| tobiash | github bump is green now \o/ | 14:24 |
| *** gregdek has quit IRC | 14:25 | |
| *** gregdek has joined #zuul | 14:26 | |
| corvus | tobiash: comment on https://review.openstack.org/559745 | 14:27 |
| *** zhuli has quit IRC | 14:28 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Remove initial release note https://review.openstack.org/560037 | 14:28 |
| *** zhuli has joined #zuul | 14:28 | |
| tobiash | corvus: thanks, will change that later when I'm at home | 14:29 |
| corvus | tobiash: do you want to drop in a reno about postgres? | 14:36 |
| openstackgerrit | Merged openstack-infra/zuul master: Make reject a mirror of require for github https://review.openstack.org/559340 | 14:36 |
| *** maxamillion has quit IRC | 14:38 | |
| *** maxamillion has joined #zuul | 14:39 | |
| *** gouthamr has quit IRC | 14:40 | |
| rcarrillocruz | hey folks | 14:41 |
| rcarrillocruz | dropping https://github.com/ansible/ansible-runner in case you didn't see it | 14:41 |
| rcarrillocruz | it's a new project from tower folks, for better interfacing (and consuming) ansible | 14:41 |
| *** elyezer has quit IRC | 14:48 | |
| corvus | rcarrillocruz: thanks! | 14:49 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Make gearman queries asynchronous in github driver https://review.openstack.org/560044 | 14:54 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Make gearman queries asynchronous in github driver https://review.openstack.org/560044 | 14:55 |
| openstackgerrit | Merged openstack-infra/zuul master: Prefix existing indexes https://review.openstack.org/559910 | 14:57 |
| openstackgerrit | Merged openstack-infra/zuul master: Also prefix the indexes if needed https://review.openstack.org/535718 | 14:57 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix sql reporting with postgres https://review.openstack.org/535720 | 14:57 |
| openstackgerrit | Merged openstack-infra/zuul master: Add tests for postgresql https://review.openstack.org/549518 | 14:57 |
| *** elyezer has joined #zuul | 15:00 | |
| mordred | corvus: if you have a sec, https://review.openstack.org/#/c/551987 has 2x +2 but I've been holding it for you in case you wanted to review | 15:07 |
| mordred | corvus: I need to rebase the rest of the stack though due to merge conflict, so mostly trying to avoid full rebase - although I just realized that the rebase won't touch the first patch and it'll keep its votes, so nevermind, I'll just rebase the whole thing | 15:08 |
| dmsimard | At what point are we upgrading to Ansible >=2.4 ? 2.3 is officially EOL since 2.5 was released: https://github.com/ansible/ansible/blob/devel/docs/docsite/rst/reference_appendices/release_and_maintenance.rst | 15:12 |
| dmsimard | corvus: ^ my only comment towards the release of a 3.1/3.0.1 | 15:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Upgrade to webpack 4 https://review.openstack.org/551987 | 15:14 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Upgrade from angularjs (v1) to angular (v5) https://review.openstack.org/551989 | 15:14 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Translate zuulStartStream into typescript https://review.openstack.org/558618 | 15:14 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Shift log streaming code into StreamComponent https://review.openstack.org/558619 | 15:14 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use glyphicons for status balls https://review.openstack.org/551992 | 15:14 |
| corvus | dmsimard: i suppose we ought to do that soon. | 15:16 |
| corvus | probably not tomorrow, but maybe it's the next thing we should focus on | 15:16 |
| dmsimard | corvus: do we have some sort of strategy for testing user playbooks and roles prior to upgrade ? Seeing as Ansible minor releases notoriously introduce regressions (or behavioral changes) | 15:17 |
| corvus | dmsimard: nope. what we have so far is "deal with the fallout" from upgrading to 2.4, then after that, work on supporting multiple versions (which would let folks opt-in to ansible versions per-job) | 15:18 |
| dmsimard | ouch :/ | 15:18 |
| Shrews | i think maybe we should bypass the 2.4 goal and go straight to 2.5. that would give us more breathing room to implement the multiple version support | 15:21 |
| Shrews | unless, of course, something is preventing us from going straight to 2.5 | 15:22 |
| corvus | Shrews: you and tobiash are the experts :) | 15:22 |
| fungi | we probably would also benefit from deciding on an ansible version support policy. like whether we get to stop caring about eol ansible versions or not | 15:22 |
| Shrews | pffft | 15:22 |
| corvus | fungi: indeed, especially once we have multiple version support, i'm not sure we will care that much -- at least, i imagine we should be able to carry "eol" versions for some time. | 15:23 |
| tobiash | we at least depend on >=2.4 | 15:23 |
| corvus | Shrews, tobiash: you want to see if jumping to 2.5 will work? | 15:24 |
| *** Diabelko has quit IRC | 15:24 | |
| corvus | it's a trade-off -- i think the 2.4 support is about ready in zuul, so we could merge it RSN. but if it's not that much more work to use 2.5, it may be worth it. | 15:24 |
| tobiash | I could try that (possibly next week) | 15:25 |
| corvus | we need someone to look at the action plugins, etc, for any changes we should know about | 15:25 |
| *** acozine1 has joined #zuul | 15:26 | |
| tobiash | corvus: regarding pg I think I'll write a reno note | 15:26 |
| corvus | ack | 15:27 |
| pabelanger | clarkb: Shrews: nodepool dsvm patch to use AFS mirrors for debian images: https://review.openstack.org/558633/ should help speed up testing on glean / diskimage-builder | 15:37 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Update in-development releasenotes title https://review.openstack.org/560057 | 15:40 |
| *** Diabelko has joined #zuul | 15:41 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Upgrade from angularjs (v1) to angular (v5) https://review.openstack.org/551989 | 15:54 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Translate zuulStartStream into typescript https://review.openstack.org/558618 | 15:54 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Shift log streaming code into StreamComponent https://review.openstack.org/558619 | 15:54 |
| openstackgerrit | Merged openstack-infra/zuul master: Remove initial release note https://review.openstack.org/560037 | 15:57 |
| mrhillsman | sean-k-mooney my theory did not pan out though i have been able to get jobs running | 16:00 |
| mrhillsman | other than noop :) | 16:00 |
| clarkb | mordred: I'm attempting to write a test for the builds/ route in zuul-web so that I can make sure my asyncio chagnes don't break that. I'm getting AssertionError: False is not true : Static web assets are missing, be sure to run 'npm run build' how is it that this works under tox normally? (I've subclassed the base zuul web test class too) | 16:01 |
| clarkb | mordred: maybe I've just gotten the url wrong /me looks more | 16:02 |
| clarkb | aha api/tenant/builds maybe | 16:02 |
| clarkb | no that isn't it and the assertion is happening in set up. /me tries to run the other web tests | 16:04 |
| corvus | clarkb: you want https://zuul-ci.org/docs/zuul/developer/javascript.html | 16:05 |
| corvus | we should make that exception output that url | 16:05 |
| clarkb | corvus: but when we run tox -epy35 in the gate we aren't doing that right? | 16:05 |
| clarkb | we're just running tox | 16:05 |
| corvus | clarkb: no, we build javascript first. we have a special tox job | 16:06 |
| corvus | clarkb: http://git.zuul-ci.org/cgit/zuul/tree/.zuul.yaml#n32 | 16:07 |
| clarkb | we might want to consider using nodeenv and making it all automagic with tox? | 16:07 |
| jlk | sigmavirus24: oh, hey thanks! | 16:07 |
| corvus | clarkb: i feel certain mordred must have considered that, but i don't recall the outcome | 16:08 |
| *** maxamillion has quit IRC | 16:08 | |
| *** maxamillion has joined #zuul | 16:08 | |
| clarkb | fwiw the docker command is failing with docker: Error response from daemon: invalid volume spec ":/usr/src/app:z": invalid volume specification: ':/usr/src/app:z' I don't think $(PWD) is getting expanded properly for some reason | 16:10 |
| clarkb | because its trying to execute PWD as a comamnd in $(PWD) | 16:11 |
| * clarkb changes to ${PWD} and will push docs fix if that works | 16:11 | |
| mordred | corvus, clarkb: yes, actually, 've been thinking we should add nodeenv to zuul's tox | 16:14 |
| mordred | the biggest issue is that we're using yarn currently, which people tend to prefer installing globally - but I just checked and it works as an npm install as well | 16:16 |
| corvus | it's probably worth noting that if you're going to develop on the js stuff, you're still probably better off running the install script. | 16:17 |
| corvus | but 'how do i make the tests work' is definitely now a FAQ :) | 16:17 |
| clarkb | progress! I now get a 404 | 16:17 |
| clarkb | from my test, the existing tests work properly | 16:17 |
| mordred | corvus: maybe we should hook up tox to use nodeenv if it doesn't find yarn/npm available globally | 16:18 |
| corvus | mordred: that sounds desirable, but i've forgotten the mechanism of nodeenv, so i'm vague on the mechanics of that :) | 16:18 |
| mordred | but it turns out the nodeenv route is less efficient than just installing yarn, so it would behoove folks doing more regular dev to just install the tools | 16:19 |
| clarkb | corvus: its a python package that builds node and npm in your virtualenv | 16:19 |
| mordred | corvus: it piggybacks on virtualenv | 16:19 |
| mordred | yah | 16:19 |
| clarkb | corvus: so you get a self contained nodejs and npm within the virtualenv like self contained python and pip | 16:19 |
| corvus | ah cool -- how would we get tox to 'switch' on that? | 16:20 |
| corvus | i can just wait for the patch if you want :) | 16:20 |
| mordred | corvus: patch coming | 16:20 |
| mordred | corvus: but the tl;dr is thatit uses the bin dir in the virtualenv structure to put npm and friends, so tox's venv switching just works | 16:21 |
| pabelanger | I like that vs global npm installs :) | 16:21 |
| * mordred still advocates for installing npm globally | 16:22 | |
| mordred | it's no worse a tool to install globally than pip is | 16:22 |
| clarkb | mordred: the problem for me historically hasn't been npm its been node | 16:22 |
| mordred | BUT - nodeenv patch coming | 16:22 |
| clarkb | because distro node is ancient in many cases | 16:22 |
| mordred | right. not from distros | 16:22 |
| clarkb | and so you have this weird bootstrapping problem of runtime too old to function | 16:23 |
| mordred | node installed from the node repos | 16:23 |
| corvus | i've had it installed for weeks and have shown no symptoms of discomfort aside from minor blemishes | 16:23 |
| clarkb | but once bootstrapped you are fine | 16:23 |
| mordred | same here - except for months | 16:23 |
| mordred | I would not use distro node to bootstrap npm - I would install the apt/yum repos from node and use them exclusively | 16:23 |
| mordred | (as well as the apt/yum repos for yarn) | 16:23 |
| clarkb | ya that helps if you yum or apt :P | 16:24 |
| clarkb | in my case I'm probably fine beacuse rolling distro | 16:24 |
| mordred | yah | 16:25 |
| mordred | clarkb: does tumbleweed have yarn? | 16:25 |
| *** ttx has joined #zuul | 16:27 | |
| clarkb | mordred: not in base repos | 16:29 |
| clarkb | but does have nodejs 8 and 6 so in theory Ican just npm my way to a yarn? | 16:29 |
| mordred | clarkb: you can - although it's recommended to just install it globally via package management because it's a top-level thing you'd interact with | 16:34 |
| mordred | that said - 'npm install yarn ; npx yarn install' will totally work | 16:34 |
| mordred | incidentally, 'npx' will let you run any command that was installed into node_modules/.bin in the current dir without needing to muck with paths | 16:35 |
| mordred | it's a worthy command to know about in the toolbox | 16:35 |
| openstackgerrit | Merged openstack-infra/zuul master: Update Github3.py to 1.1.0 https://review.openstack.org/559798 | 16:44 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add postgresql release note https://review.openstack.org/560079 | 16:45 |
| tobiash | corvus: ok or too short? ^ | 16:45 |
| corvus | tobiash: perfect | 16:46 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix missing semaphore release on zk error https://review.openstack.org/559745 | 16:49 |
| tobiash | corvus: ^ | 16:49 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add postgresql release note https://review.openstack.org/560079 | 17:08 |
| tobiash | corvus: what's the expected process on 559963 as there are no check/gate jobs there? | 17:10 |
| pabelanger | tobiash: left comment on 560079 | 17:12 |
| tobiash | pabelanger: you mean linking to https://zuul-ci.org/docs/zuul/admin/drivers/sql.html ? | 17:13 |
| pabelanger | tobiash: yah, just to help point users to which setting supports postgresql | 17:14 |
| pabelanger | maybe even an example, does look like we provide on that page | 17:15 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-sphinx master: Add build-sphinx check/gate jobs https://review.openstack.org/560090 | 17:20 |
| corvus | tobiash: ^ merge that first, i think | 17:20 |
| tobiash | corvus: thanks, lgtm | 17:21 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add postgresql release note https://review.openstack.org/560079 | 17:23 |
| tobiash | pabelanger: ^ | 17:23 |
| *** sean-k-mooney has quit IRC | 17:24 | |
| pabelanger | tobiash: I want to say we can use sphinx anchors to point to sql docs, over using a URL. | 17:24 |
| tobiash | oh, right | 17:25 |
| pabelanger | attr:: <sql connection> maybe? | 17:25 |
| *** sshnaidm is now known as sshnaidm|bbl | 17:27 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add postgresql release note https://review.openstack.org/560079 | 17:28 |
| tobiash | pabelanger: ok, will try that | 17:30 |
| clarkb | woo finally got it to 200 | 17:34 |
| *** openstackgerrit has quit IRC | 17:34 | |
| clarkb | I have learned a lot about zuul web and the test framework mocking out databases | 17:34 |
| clarkb | or rather fixturing databases | 17:35 |
| corvus | whew | 17:35 |
| clarkb | now I just have to make fixtures that won't break everyone else | 17:36 |
| *** openstackgerrit has joined #zuul | 17:36 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add postgresql release note https://review.openstack.org/560079 | 17:36 |
| clarkb | it is probably a problem if the main single-tenant tenant config has a database configured since many tests that use it don't use the db test class right? | 17:37 |
| clarkb | sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from string '$MYSQL_FIXTURE_DBURI$' confirmed | 17:37 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Update pypi metadata https://review.openstack.org/560098 | 17:37 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Remove reference to zuul-migrate in extras https://review.openstack.org/560099 | 17:37 |
| corvus | clarkb: yeah, it's heavy-weight, so we only use it where needed | 17:37 |
| clarkb | separately the mysql I ended up with doesn't do dns lookups by default. any objections to replacing localhost with 127.0.0.1 in the mysql fixture to accomodate that? | 17:38 |
| pabelanger | clarkb: corvus: tobiash: ^maybe pickup for next release too? | 17:38 |
| corvus | clarkb: no objection | 17:39 |
| corvus | pabelanger: left comments | 17:40 |
| *** jpena is now known as jpena|off | 17:40 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Update pypi metadata https://review.openstack.org/560098 | 17:42 |
| pabelanger | corvus: okay, I've -W 560099 until we decide how to move forward. I thought we removed it a while back, but maybe it was just disabling of testing cc mordred | 17:43 |
| openstackgerrit | Zane Bitter proposed openstack-infra/zuul-jobs master: Default warning-is-error to True for non-legacy Sphinx projects https://review.openstack.org/559348 | 17:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Upgrade from angularjs (v1) to angular (v5) https://review.openstack.org/551989 | 18:00 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Translate zuulStartStream into typescript https://review.openstack.org/558618 | 18:00 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Shift log streaming code into StreamComponent https://review.openstack.org/558619 | 18:00 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use nodeenv for npm and yarn in tox https://review.openstack.org/560104 | 18:00 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Remove docker instructions and build:docker helper command https://review.openstack.org/560105 | 18:00 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Clean up developer javascript instructions https://review.openstack.org/560106 | 18:00 |
| mordred | corvus, clarkb, tobiash: ^^ added nodeenv support, then did some doc cleanup | 18:00 |
| mordred | corvus: although, come to think of it, with that patch I should be able to remove the zuul-tox-py35 job and go back to using stock jobs | 18:06 |
| clarkb | whats a typical full test suite run take locally? I am making sure to run the whole thing since I've chagned some db related stuff | 18:07 |
| clarkb | its not as quick as I seem to remember it, I guess that means we've added a lot more testing | 18:07 |
| *** electrofelix has quit IRC | 18:09 | |
| clarkb | mordred: also you may know how to deal with this, the db fixtures grant access to the user from localhost but I'm running mysql in a container so I can delete it when I'm done and it sees the nat address not the localhost ip so I've just granted all foo@'' to deal with that currently | 18:10 |
| clarkb | I don't think we want to do that in the actual test suite code | 18:10 |
| clarkb | but unsure if there is a better way to handle that | 18:10 |
| tobiash | clarkb: in january I think the full tests took around 8 to 10 minutes on my machine | 18:12 |
| *** gouthamr has joined #zuul | 18:13 | |
| corvus | last time i ran them was about 6m i think? | 18:13 |
| mordred | clarkb: we could make it foo@% instead of foo@localhost | 18:13 |
| clarkb | mordred: that is what I did locally. I guess I can push that up as part of the use real ip addrs change too | 18:13 |
| clarkb | something like 4 chagnes are about to come up out of this and I realize now I didn't rebase on the postgres work so hopefully I don't have to also sort that out | 18:14 |
| tobiash | clarkb: ya, that would help me run the sql tests locally too | 18:14 |
| clarkb | tobiash: cool if it isn't just me it makes me feel better about doing it :) | 18:15 |
| tobiash | clarkb: maybe we should also make the target address configurable like with zk | 18:15 |
| clarkb | tobiash: ya though currently thats beyond what I've done as I've just done a port forwadr from localhost to container | 18:15 |
| clarkb | Ran 515 (+514) tests in 911.563s (+903.689s) | 18:16 |
| clarkb | that was my local wall clock about 15 minutes | 18:16 |
| tobiash | I'm running zk and mysql with docker compose | 18:16 |
| tobiash | And due to mac with docket in vm host networking is no option | 18:17 |
| tobiash | So db and zk are really different hosts in my local testing | 18:17 |
| *** gouthamr has quit IRC | 18:19 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Make db queries asynchronous in zuul-web https://review.openstack.org/559852 | 18:22 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Update mysql fixture for modern mysql https://review.openstack.org/560116 | 18:22 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Add test for builds/ web route https://review.openstack.org/560117 | 18:22 |
| clarkb | I've pushed the test as the parent change to the chagne in zuul-web so that we can see that zuul-web works correctly before and after | 18:23 |
| clarkb | now to write the docs update change | 18:23 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Fix docker command in javascript docs https://review.openstack.org/560118 | 18:24 |
| clarkb | ok I think ^ is all ready for review now | 18:24 |
| clarkb | tobiash: and maybe we build specifying the db address a change on top of that change | 18:25 |
| *** harlowja has joined #zuul | 18:25 | |
| openstackgerrit | Merged openstack-infra/zuul master: Update in-development releasenotes title https://review.openstack.org/560057 | 18:28 |
| clarkb | Shrews: ^ sorry just noticing now that your gearman change was based on my sql change. Do you want me to rebase it for you too? | 18:36 |
| Shrews | clarkb: sure, thx | 18:37 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Make gearman queries asynchronous in github driver https://review.openstack.org/560044 | 18:37 |
| clarkb | done | 18:37 |
| *** gouthamr has joined #zuul | 18:38 | |
| clarkb | argh I forgot to git add some things | 18:43 |
| *** gouthamr has quit IRC | 18:43 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Add test for builds/ web route https://review.openstack.org/560117 | 18:44 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Make db queries asynchronous in zuul-web https://review.openstack.org/559852 | 18:44 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Make gearman queries asynchronous in github driver https://review.openstack.org/560044 | 18:44 |
| clarkb | I think that should be happy now with the git add done | 18:44 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Support databases on other hosts during tests https://review.openstack.org/560127 | 18:45 |
| tobiash | clarkb: ^ | 18:45 |
| tobiash | that might do it, have to create a postgres container before I can validate this locally | 18:46 |
| *** sshnaidm|bbl is now known as sshnaidm | 18:48 | |
| corvus | without the pg tests, i'm currently at "Ran 522 (+12) tests in 364.334s" locally | 18:48 |
| clarkb | corvus: your cpu is likely much faster than mine :) | 18:48 |
| clarkb | I've got an older haswell era low power thing in a micropc | 18:49 |
| clarkb | maybe I should upgrade | 18:49 |
| corvus | clarkb: Intel(R) Core(TM) i7-3770S CPU @ 3.10GHz. i run the test suite entirely in tmpfs though. | 18:49 |
| clarkb | oh that will help too | 18:49 |
| clarkb | I run it on my ssd | 18:49 |
| corvus | i decided to spare my ssd those flash cycles; it's really io heavy | 18:50 |
| clarkb | mean time to failure seems so incredibly high these days that I don't worry about it too much but also try to keep these disks largely ephemeral in content | 18:51 |
| clarkb | so they have git repos I can always reclone | 18:51 |
| clarkb | but real data lives on the nas | 18:51 |
| clarkb | which runs zfs which totally had a scary bug recently :( thankfully I don't appear to ahve been affected because ubuntu zfs is too old | 18:51 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Support databases on other hosts during tests https://review.openstack.org/560127 | 19:02 |
| tobiash | clarkb: that makes sql (both mysql and postgres) work for me ^ | 19:02 |
| tobiash | corvus: what do you specify to run that in tmpfs? Just ZUUL_TEST_ROOT? | 19:04 |
| *** openstackgerrit has quit IRC | 19:04 | |
| corvus | tobiash: yep | 19:05 |
| *** JasonCL has quit IRC | 19:24 | |
| SpamapS | Hm, I get this a lot in my nodepool-launcher log | 19:28 |
| SpamapS | http://paste.openstack.org/show/718853/ | 19:28 |
| *** JasonCL has joined #zuul | 19:30 | |
| tobiash | SpamapS: that looks like zuul deletes a request (e.g. due to dequeueing an item) and nodepool then tries to update it and fails | 19:31 |
| tobiash | SpamapS: do you have an issue or is this just spam in the logs? | 19:32 |
| *** JasonCL_ has joined #zuul | 19:32 | |
| SpamapS | tobiash: takes a *very* long time for a job to claim ready nodes. | 19:32 |
| SpamapS | I'm not sure this is the cause, or even a symptom | 19:32 |
| SpamapS | but I have 5 ready nodes and it takes > 2 minutes to start a job | 19:33 |
| SpamapS | http://paste.openstack.org/show/718854/ | 19:34 |
| SpamapS | I'd expect that request to get served pretty fast | 19:34 |
| SpamapS | it's been sitting there for 2 minutes now | 19:34 |
| *** openstackgerrit has joined #zuul | 19:35 | |
| openstackgerrit | Merged openstack-infra/zuul master: Fix docker command in javascript docs https://review.openstack.org/560118 | 19:35 |
| *** JasonCL has quit IRC | 19:35 | |
| SpamapS | and the worker I expect to pick them up, 'a', is just spinning on fetching servers/ports | 19:36 |
| SpamapS | 4 minutes queued now | 19:36 |
| *** JasonCL_ has quit IRC | 19:36 | |
| tobiash | SpamapS: what's strange is that it's only processing the min-ready request | 19:37 |
| tobiash | I would assume that it processes both at the same time | 19:37 |
| tobiash | is there quota pressure? | 19:37 |
| SpamapS | nope | 19:38 |
| SpamapS | centos@zuul[zuul-gd]:~$ sudo -u nodepool /opt/venvs/nodepool/bin/nodepool list|wc -l | 19:38 |
| SpamapS | 47 | 19:38 |
| SpamapS | max is 75 | 19:38 |
| openstackgerrit | Merged openstack-infra/zuul master: Update mysql fixture for modern mysql https://review.openstack.org/560116 | 19:39 |
| SpamapS | I do have one worker, 'p', that has max-nodes of 0 | 19:39 |
| SpamapS | But I'd expect it to just decline it and quickly move on | 19:39 |
| SpamapS | It's like nodepool just doesn't want to see that request | 19:40 |
| tobiash | SpamapS: did you configure max-concurrency in your provider? | 19:40 |
| SpamapS | tobiash: no | 19:42 |
| SpamapS | is the default low? | 19:43 |
| tobiash | the default is unlimited | 19:43 |
| Shrews | SpamapS: use the --detail option to 'list' to see if those nodes are assigned | 19:43 |
| Shrews | might provide more insight | 19:43 |
| Shrews | fwiw, i don't expect unlocked nodes to be assigned, but just curious | 19:45 |
| SpamapS | Shrews: ok, it took about 11 minutes to grab a node and start the job btw | 19:46 |
| SpamapS | Shrews: http://paste.openstack.org/show/718856/ | 19:55 |
| SpamapS | http://paste.openstack.org/show/718857/ | 19:56 |
| SpamapS | requests just sitting there unserved | 19:56 |
| SpamapS | Seems like what finally makes it work is that for some reason the launcher deletes one of the nodes, and creates a new one | 19:58 |
| Shrews | SpamapS: that seems like a capacity thing. i can't help you much w/o access to your system, but the first thing i always do is grep the request number in the launcher logs (and maybe zuul logs) and follow the trail | 20:00 |
| SpamapS | Shrews: capacity is about 50% used | 20:00 |
| Shrews | and also make sure you're running the latest version of nodepool | 20:00 |
| SpamapS | centos@zuul[zuul-gd]:~$ sudo -u nodepool /opt/venvs/nodepool/bin/nodepool --version | 20:01 |
| SpamapS | 3.0.1 | 20:01 |
| Shrews | SpamapS: maybe max-servers is, but there's other capacity things in play. | 20:02 |
| Shrews | anyway, follow the log trail | 20:02 |
| clarkb | hrm my unittests are still failing, I did rebase and didn't rerun tests so I'm guessing it is realted to that and am debugging | 20:03 |
| SpamapS | No, that cloud is at 50% | 20:03 |
| SpamapS | I understand what you're saying | 20:03 |
| SpamapS | but there were *5* ready nodes | 20:04 |
| SpamapS | So not sure how capacity would even come into play. | 20:04 |
| SpamapS | To me it looks like the 'a' worker doesn't pick the request up until something happens to trigger it | 20:04 |
| SpamapS | Once it grabs the request, all works well | 20:04 |
| SpamapS | also it's not consistent | 20:06 |
| SpamapS | sometimes it fires right up and runs the job | 20:06 |
| SpamapS | tobiash: to your point, I wonder if I *SHOULD* make a concurrency limit | 20:08 |
| openstackgerrit | Merged openstack-infra/zuul master: Upgrade to webpack 4 https://review.openstack.org/551987 | 20:09 |
| tobiash | SpamapS: my question was targeted in the other direction | 20:10 |
| tobiash | I think you shouldn't make a limit | 20:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use nodeenv for npm and yarn in tox https://review.openstack.org/560104 | 20:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Clean up developer javascript instructions https://review.openstack.org/560106 | 20:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Upgrade from angularjs (v1) to angular (v5) https://review.openstack.org/551989 | 20:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Translate zuulStartStream into typescript https://review.openstack.org/558618 | 20:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Shift log streaming code into StreamComponent https://review.openstack.org/558619 | 20:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use glyphicons for status balls https://review.openstack.org/551992 | 20:11 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Add test for builds/ web route https://review.openstack.org/560117 | 20:11 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Make db queries asynchronous in zuul-web https://review.openstack.org/559852 | 20:11 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Make gearman queries asynchronous in github driver https://review.openstack.org/560044 | 20:11 |
| clarkb | ok I think I got it there | 20:12 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: Add container spec https://review.openstack.org/560136 | 20:12 |
| SpamapS | tobiash: well, if I have too many threads.. I may be suffering GIL problems | 20:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Remove docker instructions and build:docker helper command https://review.openstack.org/560105 | 20:13 |
| SpamapS | BTW, is it intentional that zuul-jobs rely on virtualenv being installed? | 20:13 |
| *** JasonCL has joined #zuul | 20:13 | |
| *** JasonCL has joined #zuul | 20:13 | |
| corvus | SpamapS: we haven't seen gil starvation until we have about 1000 simultaneous launches | 20:14 |
| SpamapS | corvus: ok, that's good to know. :) | 20:14 |
| SpamapS | then I dunno what's causing it | 20:14 |
| SpamapS | it has ceased happening at this moment | 20:14 |
| tobiash | SpamapS: yah so maybe you want a limit but more like 100 | 20:14 |
| SpamapS | I have max-servers at 75.. so it would be hard to get to 1000 ;) | 20:15 |
| clarkb | SpamapS: I think the tox ones likely rely on tox being installed too? | 20:15 |
| SpamapS | clarkb: they do not | 20:15 |
| clarkb | oh neat | 20:15 |
| SpamapS | ensure-tox is a thing :) | 20:15 |
| SpamapS | and I've just written an ensure-pip and ensure-virtualenv | 20:16 |
| SpamapS | which I will gladly contribute | 20:16 |
| SpamapS | just want to make sure that is wanted | 20:16 |
| corvus | SpamapS: i think so | 20:16 |
| SpamapS | (and do it in such a way where it is a virtual noop when they're already present) | 20:16 |
| tobiash | corvus: I wonder if it would be worth/possible to use asyncio instead of a thread per node | 20:16 |
| SpamapS | This way I can use our base images unmodified | 20:16 |
| SpamapS | which is good for testing if things can work on those images. :) | 20:16 |
| clarkb | pip10 makes stuff like that more and more dangerous if we don't noop properly if already present | 20:16 |
| clarkb | so ya lets get that correct :) | 20:16 |
| *** JasonCL has quit IRC | 20:18 | |
| *** JasonCL has joined #zuul | 20:22 | |
| mordred | clarkb, Shrews, SpamapS, pabelanger: https://review.openstack.org/#/c/560104 could use review by anybody who doesn't have yarn/npm installed globally but still wants to run zuul tests locally | 20:26 |
| *** JasonCL has quit IRC | 20:27 | |
| clarkb | topic:asyncio-db-queries is now ready for review and passes testing | 20:31 |
| corvus | i still have a review backlog of about 110 changes, so i won't be reviewing non-urgent recent changes for a while | 20:32 |
| *** JasonCL has joined #zuul | 20:35 | |
| mordred | clarkb: lgtm - I think there's an opportunity to refactor a smidge (as a followup) and shift the run_in_executor and await asyncio.wait_for stuff into rpc.submitJob so that the pattern isn't needed at each callsite - but what's there lgtm | 20:36 |
| clarkb | mordred: on the gearman one? I'll have to defer to shrews for that one | 20:37 |
| corvus | Shrews: you decided not to use gear's own async handling like i suggested yesterday? | 20:38 |
| clarkb | (I just rebased it because it depended on my change initially and I was pushing too many patchsets) | 20:38 |
| mordred | clarkb: oh - that is Shrews isn't it | 20:38 |
| corvus | mordred: the rpc class is not only used by zuul web, so we shouldn't leak asyncio stuff into it | 20:39 |
| *** JasonCL has quit IRC | 20:39 | |
| Shrews | yeah, i mean, we don't always want submitJob() to do asyncio things | 20:39 |
| corvus | but it's also the case that we don't need to have the task wait inside the executor, since it's already async | 20:40 |
| Shrews | corvus: does gear support asyncio? | 20:40 |
| *** JasonCL has joined #zuul | 20:40 | |
| corvus | Shrews: no, but it *is* asynchronous | 20:40 |
| corvus | (asyncio is not the only way to write an asynchronous program) | 20:41 |
| corvus | zuul itself is an asynchronous single-threaded program. | 20:41 |
| corvus | (the scheduler at least) | 20:41 |
| corvus | so the same model that the zuul scheduler uses to deal with gear asynchronously can be used by zuul-web+asyncio | 20:42 |
| Shrews | does it matter which way we do it? | 20:42 |
| corvus | the only caveat for that is that submitjob still does wait on the network, so we might want to put that in an executor for now, and consider a change to gear to make it async. | 20:43 |
| corvus | Shrews: the change was for efficiency, yeah? this has a thread for every pending job which is only there to wait for the result from another thread. this way works, but we could make significantly less use of threads with the other approach. | 20:44 |
| Shrews | corvus: i thought asyncio used coroutines, not threads | 20:46 |
| corvus | Shrews: run_in_executor uses a threadpool | 20:46 |
| Shrews | ah right. i think i remember that now | 20:46 |
| corvus | it seems like the escape hatch :) | 20:47 |
| clarkb | right it basically throws the action onto a thread in a threadpool os that it can wait for it to finish without blocking | 20:50 |
| clarkb | then when you get a result your await can return and return to normal coordinating coroutines | 20:50 |
| *** JasonCL has quit IRC | 20:51 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Reduce strictness of growroot test https://review.openstack.org/560150 | 20:56 |
| *** JasonCL has joined #zuul | 21:00 | |
| *** JasonCL has quit IRC | 21:03 | |
| *** acozine1 has quit IRC | 21:03 | |
| *** JasonCL has joined #zuul | 21:04 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use nodeenv for npm and yarn in tox https://review.openstack.org/560104 | 21:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Clean up developer javascript instructions https://review.openstack.org/560106 | 21:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Upgrade from angularjs (v1) to angular (v5) https://review.openstack.org/551989 | 21:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Translate zuulStartStream into typescript https://review.openstack.org/558618 | 21:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Shift log streaming code into StreamComponent https://review.openstack.org/558619 | 21:06 |
| *** hashar has quit IRC | 21:08 | |
| *** JasonCL has quit IRC | 21:08 | |
| corvus | mordred, tobiash, pabelanger: do we want to hyperlink to documentation inside of release notes? | 21:21 |
| corvus | i'm looking at https://review.openstack.org/560079 | 21:21 |
| corvus | and wondering if that's the pattern we want to establish. it could get quite verbose. do we want to do it for every note? just new features? | 21:21 |
| corvus | should we add it as a second paragraph like that, or just inline links? | 21:21 |
| corvus | mordred: are there patterns from other reno usages that you think have been successful? | 21:22 |
| *** elyezer has quit IRC | 21:24 | |
| mordred | corvus: I think using doc references like that can be nice, given that we're building the releasenotes along with our docs ... | 21:28 |
| mordred | the pattern doen't work for the regular openstack projects because the releasenotes are built/published separately (I've wanted to cross-reference in shade/sdk reno before though, so I think this one is neat) | 21:29 |
| mordred | I like how http://logs.openstack.org/79/560079/5/check/build-sphinx-docs/c631c1d/html/releasenotes.html winds up looking | 21:30 |
| *** JasonCL has joined #zuul | 21:30 | |
| corvus | okay, i'm mildly worried about brevity in release notes. maybe we should just combine those 2 sentences into one paragraph as a compromise :) | 21:30 |
| mordred | ah - yah. I think that would be fine | 21:30 |
| mordred | corvus: the other entry in the bug fixes section already has some references too - and i lik ethey way they flow | 21:31 |
| mordred | corvus: so I guess my vote is "it's ok to do - but may not always be necessary" | 21:31 |
| *** JasonCL has quit IRC | 21:32 | |
| clarkb | the final rendering has all of the bug fixes in their own section at the end of the notes right? | 21:33 |
| pabelanger | I'm okay with new features linking to docs, but don't have a strong opinion is people opposite it | 21:33 |
| clarkb | so they don't overpower the typically more relevant info for end uers higher up in the list | 21:33 |
| corvus | clarkb: yes, that's visible in the link mordred pasted | 21:33 |
| corvus | i'm not sure bug fixes are less revelant for users :) | 21:34 |
| corvus | if it's not important, don't add a release note. if it is, do. :) | 21:34 |
| clarkb | thats fair | 21:34 |
| mordred | yah - and there are a few different sections you can put the note in | 21:34 |
| corvus | tobiash: what's the status of https://review.openstack.org/535680 ? | 21:43 |
| corvus | tobiash: (it says WIP, but is green and hasn't been updated in a while) | 21:43 |
| corvus | clarkb: your irc question was answered on https://review.openstack.org/535878 | 21:51 |
| *** elyezer has joined #zuul | 21:54 | |
| clarkb | "If in addition in the future we get a way to trigger a reload via a Zuul admin protected API then it makes the process really straightforward." makes me wonder if maybe instead of running arbitrary code to get that data we could POST it as part of that command (and have it imply a reload too)? | 21:54 |
| clarkb | that might be a cleaner api overall | 21:55 |
| clarkb | (however completely lacking such abilities today) | 21:55 |
| corvus | yeah. maybe we move toward that in baby steps? | 21:57 |
| clarkb | overall my biggest concern is that this seems to fail relatively quietly and unload the config if it fails because it returns [] (but you'd have to double check me on that) | 21:57 |
| clarkb | and so as a user you may not really notice then have a hard time debugging why tenants stopped working | 21:58 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Reformat tenant_config_script release note https://review.openstack.org/560164 | 21:58 |
| corvus | clarkb: that's my understanding | 22:01 |
| corvus | clarkb: maybe we could improve that with a warning | 22:01 |
| corvus | clarkb: but if you're still +2 on that, could you also +2 https://review.openstack.org/560164 real quick? that way we don't have to respin that since it has 2x2 already | 22:02 |
| clarkb | corvus: looking | 22:02 |
| clarkb | and ya I'm generally +2 on the idea if it is helpful just awnt ot make sure errors are handled in a way that makes sense to people debugging them | 22:02 |
| clarkb | corvus: done | 22:05 |
| *** dkranz has quit IRC | 22:06 | |
| openstackgerrit | Merged openstack-infra/zuul master: Tenant config can be read from an external script https://review.openstack.org/535878 | 22:19 |
| *** JasonCL has joined #zuul | 22:22 | |
| *** JasonCL has quit IRC | 22:23 | |
| *** JasonCL has joined #zuul | 22:25 | |
| openstackgerrit | Merged openstack-infra/zuul master: Allow using remote refs to find commits for change https://review.openstack.org/544964 | 22:25 |
| openstackgerrit | Merged openstack-infra/zuul master: Reformat tenant_config_script release note https://review.openstack.org/560164 | 22:26 |
| corvus | mordred: you have a bunch of topic:javascript stuff that's stuck on outdated/abandoned dependencies | 22:29 |
| *** JasonCL has quit IRC | 22:33 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Reduce strictness of growroot test https://review.openstack.org/560150 | 22:37 |
| *** acozine1 has joined #zuul | 23:03 | |
| openstackgerrit | Merged openstack-infra/zuul master: autohold: prevent using ',' in change argument https://review.openstack.org/556329 | 23:05 |
| openstackgerrit | Merged openstack-infra/zuul master: Add clarity to *Zuul from Scratch* instructions https://review.openstack.org/556344 | 23:12 |
| corvus | my review backlog is down to 63 at EOD | 23:22 |
| *** JasonCL has joined #zuul | 23:28 | |
| *** JasonCL has quit IRC | 23:33 | |
| *** JasonCL has joined #zuul | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!