| *** jpenag has joined #zuul | 00:56 | |
| *** tflink has quit IRC | 00:57 | |
| *** Shrews has quit IRC | 00:57 | |
| *** jpena|off has quit IRC | 00:57 | |
| *** lennyb has quit IRC | 00:57 | |
| *** panda|ruck has quit IRC | 00:59 | |
| *** panda has joined #zuul | 01:02 | |
| *** tflink has joined #zuul | 01:04 | |
| *** Shrews has joined #zuul | 01:04 | |
| *** panda has quit IRC | 01:06 | |
| *** openstackgerrit has quit IRC | 01:06 | |
| *** panda has joined #zuul | 01:07 | |
| *** openstackgerrit has joined #zuul | 01:32 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: executor: enable add_host for trusted play and update inventory https://review.openstack.org/590092 | 01:32 |
|---|---|---|
| *** jiapei has joined #zuul | 01:49 | |
| *** elyezer has quit IRC | 02:00 | |
| *** elyezer has joined #zuul | 02:03 | |
| *** elyezer has quit IRC | 02:35 | |
| *** elyezer has joined #zuul | 02:38 | |
| *** elyezer has quit IRC | 02:52 | |
| *** elyezer has joined #zuul | 02:53 | |
| *** elyezer has quit IRC | 03:20 | |
| *** elyezer has joined #zuul | 03:21 | |
| *** elyezer has quit IRC | 03:40 | |
| *** elyezer has joined #zuul | 03:44 | |
| *** jiapei has quit IRC | 03:59 | |
| *** elyezer has quit IRC | 05:02 | |
| *** elyezer has joined #zuul | 05:05 | |
| *** odyssey4me has quit IRC | 06:14 | |
| *** odyssey4me has joined #zuul | 06:14 | |
| *** elyezer has quit IRC | 06:40 | |
| *** elyezer has joined #zuul | 06:43 | |
| *** pcaruana has joined #zuul | 06:44 | |
| openstackgerrit | Markus Hosch proposed openstack-infra/nodepool master: Move sphinx + deps to doc/requirements.txt https://review.openstack.org/591565 | 06:47 |
| SpamapS | woooot.. AnsibleFest talk accepted! | 06:56 |
| *** elyezer has quit IRC | 06:57 | |
| *** elyezer has joined #zuul | 07:01 | |
| *** gtema has joined #zuul | 07:18 | |
| SpamapS | tobiash: when you're around.. I'm curious why you have zuul-executor as a statefulset. | 07:48 |
| SpamapS | seems like it would work fine with just a deployment+volume | 07:48 |
| gtema | SpamapS: yes, it should work also fine. I think it was just to keep "cached" info for each individual instance | 07:49 |
| gtema | statefulset is a more recommended approach if there is a state in place and i.e. instance name is important (second is not very relevant here, compared to zookeeper) | 07:51 |
| tobiash | SpamapS: we're using the cinder driver for provisioning volumes. These are single attach. Thus a deployment would need the recreate strategy and rolling restarts/upgrades would not be possible. | 07:54 |
| tobiash | SpamapS: the second reason is that the executors need a real identity which can be reached by zuul-web for log streaming | 07:54 |
| tobiash | SpamapS: also having a constant identity is what we want with regards to statsd statistics | 07:56 |
| SpamapS | ah interesting | 07:59 |
| SpamapS | I think having the caches sitting around as volumes is a good thing too. | 08:00 |
| SpamapS | I'm about 70% of the way through converting the openshift stuff to pure k8s | 08:00 |
| SpamapS | Now I'm dealing mostly with like, how to set up AWS's EKS and ECR with the right policies/users/etc. | 08:01 |
| SpamapS | another weird thing is that kubectl apply doesn't like the yamls to be lists of objects | 08:04 |
| SpamapS | so I'm having to strip them down to single objects | 08:04 |
| *** openstackstatus has quit IRC | 08:12 | |
| *** jpenag is now known as jpena | 08:27 | |
| *** sshnaidm|off is now known as sshnaidm | 08:48 | |
| *** elyezer has quit IRC | 08:53 | |
| *** elyezer has joined #zuul | 08:58 | |
| *** electrofelix has joined #zuul | 09:04 | |
| *** elyezer has quit IRC | 09:38 | |
| *** openstackstatus has joined #zuul | 09:42 | |
| *** ChanServ sets mode: +v openstackstatus | 09:42 | |
| *** elyezer has joined #zuul | 09:44 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: [wip] web: rewrite interface in react https://review.openstack.org/591604 | 09:56 |
| tristanC | mordred: i updated the react-zuul poc and proposed it in 591604. this version only includes the builds page, though the plumbing for white-label/multi-tenant should be working | 10:12 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Do not abort node launch if failed node cannot be deleted https://review.openstack.org/589854 | 10:13 |
| *** panda is now known as panda|ruck | 10:24 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: [wip] web: rewrite interface in react https://review.openstack.org/591604 | 10:40 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: [wip] web: rewrite interface in react https://review.openstack.org/591604 | 10:58 |
| tristanC | mordred: i shifted the base of the js things to the web directory so that it will be easier to move to a dedicated project, though that will raise the challenge of cross npm projects testing | 11:02 |
| *** jpena is now known as jpena|lunch | 11:15 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: [wip] web: rewrite interface in react https://review.openstack.org/591604 | 11:32 |
| *** elyezer has quit IRC | 11:39 | |
| *** elyezer has joined #zuul | 11:43 | |
| *** ssbarnea has quit IRC | 11:44 | |
| tristanC | here is what zuul webui can looks like with patternfly-react: logs.openstack.org/04/591604/4/check/zuul-build-dashboard-multi-tenant/7e7e6b8/npm/html/ | 11:47 |
| *** ssbarnea has joined #zuul | 11:51 | |
| *** jpena|lunch is now known as jpena | 12:21 | |
| *** rlandy has joined #zuul | 12:48 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Do not abort node launch if failed node cannot be deleted https://review.openstack.org/589854 | 12:55 |
| Shrews | FYI, out for a bit this morning to get my stitches removed. bbl | 12:57 |
| mordred | tristanC: interesting - it's gonna take me a bit to wrap my head around the differences in the javascript enough to be able to comment - but the builds and jobs pages seem to be working on both single and multi tenant | 13:02 |
| *** elyezer has quit IRC | 13:10 | |
| tristanC | mordred: cool, a couple of notes. it's using HashRouter as it seems easier to deploy, but we can use the BrowserRouter instead to keep the previous url path | 13:21 |
| *** elyezer has joined #zuul | 13:21 | |
| tristanC | mordred: and i used a redux store with a reducer for the initial /api/info call, but i don't know the logic exactly, so the rest of api calls are using '.then()' instead of the dispatch logic | 13:22 |
| mordred | tristanC: yeah - just like the angular I think it's likely there are a few things we'd need to learn | 13:43 |
| mordred | tristanC: I'm otherwise leaving some comments on the patch itself | 13:44 |
| openstackgerrit | Markus Hosch proposed openstack-infra/nodepool master: Add list of metrics provided to statsd https://review.openstack.org/590233 | 13:49 |
| *** elyezer has quit IRC | 13:52 | |
| *** elyezer has joined #zuul | 13:55 | |
| mordred | tristanC: silly question - in javascript and typescript the trailing ;'s are optional - does the jsx parser allow you to leave them out? (asking because I've come to enjoy the ;-less code) | 13:56 |
| tristanC | mordred: i think so, i put them because rjsx-mode warn about them by default | 13:58 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Do not abort node launch if failed node cannot be deleted https://review.openstack.org/589854 | 14:01 |
| tristanC | mordred: thanks a lot for the prompt review. I'll try to add a status and stream page implementation tomorrow. | 14:08 |
| mordred | tristanC: cool. and ok - don't put TOO much work into it until we've had a chance to circle back around with corvus - and we've got a couple of infra things today that might dominate our time | 14:14 |
| mordred | tristanC: I think the builds/jobs pages have enough code in them to be able to look at what's going on to assess the differences | 14:15 |
| corvus | mnaser: ^ fyi because you're interested in this too | 14:33 |
| * mnaser reads | 14:33 | |
| *** bstinson_ is now known as bstinson | 14:55 | |
| tobiash | mhu: commented on 589854 | 14:58 |
| panda|ruck | how are the variables added to a zuul job configuration passed to the ansible playbook ? are they passed through inventory only ? Is there any way to pass them as extra-vars so they can't be overridden in the playbook itself ? | 15:05 |
| mordred | tobiash: aren't you supposed to be on vacation? | 15:11 |
| clarkb | mordred: panda|ruck I want to say secrets are treated as extra vars | 15:11 |
| mordred | clarkb, panda|ruck: yes - and also I think pabelanger was working on ability to pass arbitrary extra_vars | 15:11 |
| mordred | I do not remember if we landed that yet or not | 15:11 |
| pabelanger | https://zuul-ci.org/docs/zuul/user/config.html#attr-job.extra-vars | 15:11 |
| pabelanger | already landed | 15:11 |
| pabelanger | but, not an file, need to add them into zuul.yaml for now | 15:12 |
| panda|ruck | mmmmhh | 15:12 |
| mordred | https://zuul-ci.org/docs/zuul/user/jobs.html#job-extra-variables | 15:12 |
| mordred | pabelanger beat me to it | 15:12 |
| panda|ruck | using a file may certainly be convenient, but this is a start, thanks :) | 15:13 |
| tobiash | mordred: yes, I'm on vacation, but sometimes I like doing some reviews ;) | 15:13 |
| pabelanger | I think we could eventually use an extra-vars file from the repo too | 15:13 |
| mordred | tobiash: :) | 15:13 |
| panda|ruck | mordred: clarkb pabelanger thanks all | 15:13 |
| pabelanger | panda|ruck: https://zuul-ci.org/docs/zuul/user/jobs.html#variables is a good read for zuul variable precedence | 15:15 |
| clarkb | without knowing what you are trying to do, I think one of the ideas with zuul job inheritance is that you can modify these things | 15:16 |
| clarkb | the devstack base job and all of its children are a great example of this | 15:16 |
| openstackgerrit | Markus Hosch proposed openstack-infra/nodepool master: Add list of metrics provided to statsd https://review.openstack.org/590233 | 15:20 |
| pabelanger | tobiash: do you have any check jobs in place to validate a github project needs to have repo.change_merge_setting setup? | 15:24 |
| pabelanger | I'm struggling to find that setting in our zuul docs | 15:24 |
| tobiash | pabelanger: no, we don't have this as a check job | 15:25 |
| tobiash | but yes, that's needed if you want to do gating on github | 15:26 |
| pabelanger | tobiash: also, are you managing your github projects via yaml at all? Or do users just use the webui to create them? | 15:38 |
| tobiash | pabelanger: they just use the webui | 15:39 |
| *** darkwisebear has joined #zuul | 15:40 | |
| pabelanger | tobiash: and they follow docs to make sure branch protection / merge commits is properly setup? | 15:43 |
| tobiash | yes | 15:43 |
| pabelanger | k | 15:44 |
| openstackgerrit | Markus Hosch proposed openstack-infra/nodepool master: Add metric for image build result https://review.openstack.org/590412 | 15:44 |
| tobiash | that's one reason why we gate our tenant config | 15:45 |
| pabelanger | tobiash: yah, I want to do the same for sf.io | 15:45 |
| pabelanger | I'm sure with winterscale, we'd want to do the same, if supporting github | 15:46 |
| *** pcaruana has quit IRC | 16:02 | |
| pabelanger | any chance of landing https://review.openstack.org/591457/ today, adds nodepool info into emit-job-header role | 16:07 |
| *** jpena is now known as jpena|off | 16:13 | |
| *** elyezer has quit IRC | 16:18 | |
| *** elyezer has joined #zuul | 16:19 | |
| darkwisebear | tobiash: Is https://review.openstack.org/591565 sufficient to separate doc requirements from test requirements? It's the new prerequisite so that https://review.openstack.org/590233 may eventually land | 16:22 |
| tobiash | darkwisebear: I think you also need to adapt tox.ini to not break local doc builds | 16:23 |
| *** gtema has quit IRC | 16:32 | |
| *** darkwisebear has quit IRC | 16:35 | |
| *** rlandy is now known as rlandy|brb | 16:46 | |
| *** rlandy|brb is now known as rlandy | 17:24 | |
| *** Shrews has quit IRC | 17:52 | |
| pabelanger | tristanC: corvus: do you have a moment to look at http://paste.openstack.org/show/728030/ | 18:09 |
| pabelanger | I am wondering if we have a bug in zuul, when we have the same project in 2 different connections | 18:09 |
| pabelanger | https://softwarefactory-project.io/r/13256/ was the change in question | 18:09 |
| corvus | tristanC, mordred: the react patch looks promising. i think i can follow it fairly well so far (i think it may be easier to follow than angular). i think if you wanted to look at getting the jquery status page in there along with streaming, so we can see if there are potential problems with that, it would be a good use of time. also, maybe just so we can see some more navigation and html, maybe you could | 18:09 |
| corvus | look at doing the "job" page (the thing we were talking about over the weekend) | 18:09 |
| corvus | pabelanger: biab. | 18:11 |
| fungi | so if we wanted to add some temporary info pointing people to the summit from https://zuul-ci.org/ where would that best fit into the page? i think i have a blurb on the way from the events marketing team to let people know about zuul content scheduled for berlin | 18:11 |
| pabelanger | corvus: tristanC: okay, I think it is a syntax issue in our tenant configuration | 18:14 |
| corvus | pabelanger: you found it? (it's included twice under the same source (line 884 and 904) | 18:18 |
| pabelanger | corvus: I did thanks, missed it the first time looking | 18:18 |
| pabelanger | remote: https://softwarefactory-project.io/r/13414 Remove duplicate openstack-infra/tripleo-ci | 18:18 |
| pabelanger | going to see how to best add a check job to protect users from making the mistake again | 18:18 |
| corvus | fungi: i don't think we have something ready-made for that (we already used all the tricks arcana had). we could re-do the shaded banner at https://html5up.net/arcana fairly easily ("a responsive site template...") or we could make another call-to-action bar ("are you ready to continue your quest"). or we could do something new. | 18:24 |
| fungi | cool, i'll see what i can come up with to propose | 18:24 |
| fungi | thanks for the insights! | 18:25 |
| *** electrofelix has quit IRC | 18:26 | |
| *** Shrews has joined #zuul | 18:48 | |
| *** elyezer has quit IRC | 19:02 | |
| *** elyezer has joined #zuul | 19:05 | |
| *** elyezer has quit IRC | 19:21 | |
| *** abelur has quit IRC | 19:27 | |
| *** abelur has joined #zuul | 19:27 | |
| *** zxiiro has quit IRC | 19:27 | |
| *** zxiiro has joined #zuul | 19:27 | |
| *** myoung has joined #zuul | 19:32 | |
| *** elyezer has joined #zuul | 19:41 | |
| pabelanger | corvus: tristanC: decided to try zuul tenant-conf-check for the invalid rdo.yaml configuration from before, and it seems to pass validation. I'll look more into tomorrow | 23:24 |
| SpamapS | well we finally got too many parallel jobs for our single-node scheduler+executor to handle | 23:40 |
| SpamapS | Pretty sure the governor isn't working for that box. | 23:41 |
| fungi | the load average governor? | 23:41 |
| SpamapS | It's spiking up to a load of 10, and then tapers off to 3-4 as jobs finish.. but it never says "whoa that's too many" despite subsequently timing out jobs because the load is too high. | 23:41 |
| SpamapS | Yeah it's just a cumulative load thing | 23:42 |
| SpamapS | It's not sustaining enough that it doesn't pull in all the jobs we have (around 10) whenever somebody submits a PR | 23:42 |
| SpamapS | so it starts 10 jobs | 23:42 |
| SpamapS | then they just battle for the CPU/RAM/Disk until the quick ones finish | 23:43 |
| SpamapS | I'm deploying 3 new executors to combat the problem. :) | 23:43 |
| fungi | i dream of a day when zuul can request additional executors from nodepool and then delete them again once it no longer needs as many | 23:44 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: WIP: fix tenant-conf-check for duplicate projects https://review.openstack.org/591868 | 23:45 |
| pabelanger | tristanC: corvus: working example of issue, I'd expect validation to fail | 23:45 |
| pabelanger | will dig more into it in the morning | 23:45 |
| SpamapS | fungi: aye | 23:46 |
| SpamapS | Well I actually think we can also just let out of band auto scalers do that | 23:46 |
| SpamapS | I intend to set up Kubernetes that way. | 23:46 |
| pabelanger | SpamapS: make sure you update to latest hwe kernel, if you can. We seen a large performance boot in openstack-infra | 23:47 |
| SpamapS | pabelanger: dunno what that means. :) | 23:47 |
| SpamapS | I'm on CentOS7 | 23:47 |
| SpamapS | so .. :( | 23:47 |
| pabelanger | ah | 23:47 |
| SpamapS | and not really interested in trying to upkernel | 23:48 |
| pabelanger | yah, not sure what the story on centos is. haven't updated kernels for sf.io either | 23:48 |
| SpamapS | bad enough I have to compile python3.5 ;) | 23:48 |
| pabelanger | ++ | 23:48 |
| fungi | right, i think it was more that our executors seemed to get a performance boost from kernels that no longer have such a severe performance penalty from spectre/meltdown mitigated microcode | 23:48 |
| SpamapS | ah | 23:49 |
| SpamapS | Seems like you'd want spectre/meltdown mitigations. | 23:49 |
| SpamapS | or you're saying they improved the patch? | 23:49 |
| clarkb | SpamapS: yes, basically from 4.13 to 4.15 they made the performance pain go away largely | 23:50 |
| clarkb | SpamapS: there was a blog post by gkh iirc that basically said so many performance improvements in 4.15 too bad they they basically get us back to where we were pre meltdown | 23:50 |
| fungi | newer kernels are able to take advantage of some of the options the microcode is able to expose to avoid taking as much of a performance hit | 23:51 |
| fungi | <citation needed> | 23:51 |
| SpamapS | Oh yeah I remember that | 23:51 |
| fungi | basically we grew enough that we needed to add an additional executor, then we did a kernel upgrade on our executors and overall load average dropped enough that we could delete the extra executor again | 23:53 |
| fungi | so not a huge boost (given our scale) but still certainly noticeable | 23:54 |
| fungi | i guess that suggests we saw at least a 10% improvement anyway | 23:54 |
| clarkb | basically 4.14 was the released version at time of meltdown so it got the fix it fast patches | 23:55 |
| clarkb | 4.15 and beyond are more complete aiui | 23:55 |
| fungi | centos 7 kernels... who knows | 23:55 |
| SpamapS | yeah | 23:55 |
| SpamapS | patchypatchy | 23:55 |
| clarkb | I actually need to reboot to pick up 4.17 Ithink | 23:57 |
| fungi | for some reason 4.17 seems to have broken my ability to negotiate voltages higher than 5v over usb type-c on the gpd pocket | 23:59 |
| fungi | and 4.18 rc4 isn't fixing it | 23:59 |
| fungi | so i've had to stick to 4.16 on those until i can dig deeper | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!