| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: config: add statsd-server config parameter https://review.openstack.org/528969 | 00:33 |
|---|---|---|
| *** dtruong2 has joined #zuul | 01:27 | |
| *** threestrands_ has joined #zuul | 01:40 | |
| *** threestrands has quit IRC | 01:43 | |
| *** dtruong2 has quit IRC | 01:45 | |
| *** dkranz has quit IRC | 02:03 | |
| *** jappleii__ has joined #zuul | 03:12 | |
| *** jappleii__ has quit IRC | 03:13 | |
| *** jappleii__ has joined #zuul | 03:13 | |
| *** threestrands_ has quit IRC | 03:15 | |
| *** JasonCL has quit IRC | 03:48 | |
| *** bhavik1 has joined #zuul | 04:10 | |
| *** bhavik1 has quit IRC | 04:16 | |
| *** xinliang has quit IRC | 05:46 | |
| *** xinliang has joined #zuul | 06:01 | |
| *** xinliang has quit IRC | 06:01 | |
| *** xinliang has joined #zuul | 06:01 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: ansible: honor command no_log module attribute https://review.openstack.org/533509 | 06:49 |
| *** jappleii__ has quit IRC | 07:02 | |
| *** saop has joined #zuul | 07:39 | |
| saop | hello all | 07:39 |
| saop | I setup zuul v3 in my CI, and zuul-web is not showing tenants.html page, any reason , what i need to do? | 07:52 |
| tristanC | saop: have you checked zuul/web/static/README ? | 08:06 |
| saop | tristanC, yes | 08:06 |
| saop | tristanC, I did same thing in apache config | 08:07 |
| saop | tristanC, and it also showing in status.html page status.json not found | 08:07 |
| saop | tristanC, Thanks i found one error | 08:10 |
| saop | tristanC, it solved | 08:10 |
| *** hashar has joined #zuul | 08:13 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add buildset-artifacts-location and fetch roles https://review.openstack.org/530679 | 08:25 |
| *** sshnaidm is now known as sshnaidm|afk | 08:41 | |
| *** jpena|off is now known as jpena | 08:49 | |
| *** jaianshu has joined #zuul | 08:59 | |
| *** xinliang has quit IRC | 09:52 | |
| *** sshnaidm|afk has quit IRC | 09:55 | |
| *** xinliang has joined #zuul | 10:05 | |
| *** xinliang has quit IRC | 10:05 | |
| *** xinliang has joined #zuul | 10:05 | |
| *** sshnaidm|afk has joined #zuul | 10:52 | |
| *** sshnaidm|afk is now known as sshnaidm | 10:58 | |
| *** electrofelix has joined #zuul | 11:06 | |
| *** JasonCL has joined #zuul | 12:05 | |
| *** jkilpatr has joined #zuul | 12:14 | |
| *** jkilpatr has quit IRC | 12:19 | |
| *** jkilpatr has joined #zuul | 12:19 | |
| *** saop has quit IRC | 12:30 | |
| *** jpena is now known as jpena|lunch | 12:38 | |
| *** nhicher has joined #zuul | 12:57 | |
| Shrews | FYI, I am unavailable today. | 13:02 |
| *** openstackgerrit has quit IRC | 13:18 | |
| *** jpena|lunch is now known as jpena | 13:29 | |
| *** jaianshu has quit IRC | 13:31 | |
| *** lennyb_ has joined #zuul | 13:33 | |
| *** lennyb_ has quit IRC | 13:34 | |
| *** rlandy has joined #zuul | 13:41 | |
| *** jkilpatr has quit IRC | 14:03 | |
| mhu | Hello, is it possible to find out the nodepool id(s) of the node(s) used by zuul to build a specific job? I've looked at status.json but I can only find references to zuul executors. The use case would be, for example, given that I have access to nodepool, I could hold a job's node(s) before the build completes so I can investigate the build env post mortem | 14:04 |
| *** jkilpatr has joined #zuul | 14:06 | |
| *** JasonCL has quit IRC | 14:11 | |
| *** JasonCL has joined #zuul | 14:11 | |
| *** JasonCL has quit IRC | 14:11 | |
| *** JasonCL has joined #zuul | 14:12 | |
| *** JasonCL has quit IRC | 14:12 | |
| *** JasonCL has joined #zuul | 14:12 | |
| *** JasonCL has quit IRC | 14:19 | |
| *** JasonCL has joined #zuul | 14:24 | |
| *** JasonCL has quit IRC | 14:25 | |
| lennyb | mhu, we usually check in Jenkins where job is executed and then hold it | 14:40 |
| mhu | lennyb, I forgot to mention I was talking about zuulv3 | 14:40 |
| *** jkilpatr has quit IRC | 14:54 | |
| *** jkilpatr has joined #zuul | 14:55 | |
| lennyb | mhu, sorry, I cant help you with this one yet | 14:58 |
| corvus | mhu: check out the 'zuul autohold' command | 15:12 |
| corvus | mhu: in v3, the hold functionality has been moved to zuul | 15:13 |
| *** dkranz has joined #zuul | 15:14 | |
| mhu | corvus, ok, looking, thx | 15:19 |
| *** AJaeger has joined #zuul | 15:34 | |
| AJaeger | zuul team, the zuul tests fail on zuul-jobs, see http://logs.openstack.org/36/531936/6/check/tox-py35-on-zuul/f5f94b0/job-output.txt.gz#_2018-01-15_15_25_49_593331 | 15:34 |
| *** dkranz has quit IRC | 15:50 | |
| *** dkranz has joined #zuul | 16:03 | |
| corvus | AJaeger: https://github.com/aio-libs/aiohttp/issues/2662 is the issue | 16:28 |
| corvus | we may need to add yarl to requirements ahead of aiohttp until they fix that | 16:29 |
| AJaeger | corvus: I see ;( | 16:31 |
| corvus | adding "yarl>=0.11,<1.0" above aiohttp in requirements.txt should do it i think | 16:32 |
| corvus | i'll do that real quick | 16:32 |
| *** openstackgerrit has joined #zuul | 16:34 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Temporarily pin yarl while aiohttp is broken https://review.openstack.org/533726 | 16:34 |
| corvus | AJaeger: ^ | 16:34 |
| AJaeger | thanks, corvus | 16:35 |
| *** jpena is now known as jpena|brb | 16:44 | |
| *** jpena|brb is now known as jpena | 17:27 | |
| clarkb | pabelanger: testing this little bit of code is proving to be difficult | 17:37 |
| clarkb | pabelanger: the problem is we hand node boot failures just fine and there is a bit of an assumption that all external failures would happen there. Problem is when the handler itself fails (due to quota or whatever) is the place where we are fixing | 17:37 |
| clarkb | and there isn't any way to order these node request handler run_handlers in a reliable way that I have found | 17:38 |
| clarkb | oh wait there is a paused_handler. Maybe that is the magic I need | 17:39 |
| clarkb | oh except thats an actual state and not just a flag | 17:39 |
| clarkb | I think there may be a bug in quota checking too | 17:43 |
| clarkb | OpenStackNodeRequestHandler._waitForNodeSet() checks if it should pause itself after we check the quota in run_handler() so we can bail out early if we are at quota instead of pausing | 17:45 |
| clarkb | corvus: ^ | 17:45 |
| clarkb | though I'm not entirely sure how the state machine would properly represent that situation and keep jobs happy | 17:46 |
| *** jkilpatr has quit IRC | 18:05 | |
| *** jpena is now known as jpena|off | 18:18 | |
| *** jkilpatr has joined #zuul | 18:18 | |
| *** kmalloc has joined #zuul | 18:19 | |
| *** corvus is now known as jeblair | 18:21 | |
| *** jeblair is now known as corvus | 18:21 | |
| *** myoung|pto is now known as myoung | 18:39 | |
| mnaser | corvus: in oom related stuff, im not sure if this data point helps but i approved a large set of patches today in puppet just before things fell through (and i wonder if the first time a few days ago it fell over was because of the large amounts of patches pushed at once) | 18:45 |
| mnaser | https://review.openstack.org/#/q/status:open+topic:use_journal | 18:45 |
| corvus | mnaser: it should only significantly affect zuul memory usage if it changes zuul configuration, those don't look like they should, so probably not a direct cause | 18:46 |
| mnaser | corvus: yeah did notice that they were not zuul reconfigs but i figured i'd let you know as a data point just in case | 18:46 |
| corvus | mnaser: thx | 18:49 |
| *** corvus is now known as jeblair | 18:58 | |
| *** jeblair is now known as corvus | 18:58 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Temporarily pin yarl while aiohttp is broken https://review.openstack.org/533726 | 19:04 |
| pabelanger | https://review.openstack.org/511986/ and https://review.openstack.org/532615/ cleans up old zuulv3-dev.o.o and zuul-dev.o.o servers in site.pp | 19:07 |
| *** electrofelix has quit IRC | 19:17 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Revert "Revert "Add zuul.{pipeline,nodepool.provider,executor.hostname} to job header"" https://review.openstack.org/514489 | 19:31 |
| corvus | let's try to keep openstack-infra specific topics in #openstack-infra | 19:37 |
| SpamapS | Oy, porting pyre2 to Py3 is going to be a bit of a bear | 19:50 |
| SpamapS | unicode.. bytes... lots of assumptions. :-P | 19:50 |
| SpamapS | I'm about 30% of the way through I think, but having to double back to make it py2+py3. Also the original author is MIA, so we may have to adopt it if we want to use it. But it might be worth it.. really nice RE library. | 19:51 |
| SpamapS | Even better would be if we could convince *python* to adopt it :) | 19:51 |
| *** jkilpatr has quit IRC | 19:55 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool feature/zuulv3: Add test_launcher test https://review.openstack.org/533771 | 19:56 |
| clarkb | pabelanger: ^ thats a half working test for the thing you wanted. It doesn't work reliably enough to merge it though and I really have no idea how to make it reliable without putting a bunch of test specific code in the launch handler. | 19:57 |
| clarkb | The problem here is we basically need to pause the main run loop of the second provider until after the first provider fails | 19:58 |
| clarkb | But using the built in pause mechanism doesn't seem to work because a paused request doesn't update its config (so if you use max-servers to pause a handler it won't see any max-server updates to unpause it) | 19:58 |
| clarkb | the other I dea I have that I haven't had a chance to test yet is getting both providers to pause due to max-servers being met. Then make a third node request. Delete provider1's existing node now there is room but it will fail first, then delete provider2's existing node which should succeed | 20:01 |
| tobiash | SpamapS: what's the problem with regular expressions in python? | 20:02 |
| clarkb | the problem with this is that third request could be in a paused state on either provider. Which means I'm back to the same problem as before of other provider getting it first | 20:03 |
| clarkb | Shrews: ^ any ideas on how to test this? | 20:06 |
| clarkb | the real problem here seems to be that paused handlers are in a super exceptional state where we can't really affect them | 20:07 |
| corvus | clarkb: Shrews is afk today | 20:07 |
| clarkb | anyways, I believe that code does work as intended. The test passes when the assignHandlers race goes in the order we want | 20:07 |
| clarkb | it fails when it goes in the other order | 20:08 |
| SpamapS | tobiash: try timing the regex '(x+x+)+y' against an ever increasing string of 'x' ... | 20:08 |
| SpamapS | tobiash: around 100 x's the permutations exceed seconds until the sun will supernova ;-) | 20:09 |
| SpamapS | re2 doesn't allow backrefs | 20:10 |
| SpamapS | But it's a C lib and needs glue. The glue is 2.7 only at the moment | 20:10 |
| tobiash | ah, ok | 20:13 |
| tobiash | that would be a great way to dos zuul... | 20:13 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Use setup.py if we can't get the name from setup.cfg https://review.openstack.org/531936 | 20:14 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool feature/zuulv3: Add test_launcher test https://review.openstack.org/533771 | 20:14 |
| clarkb | now with missing file add | 20:14 |
| dmsimard | btw I formalized something I may have discussed before -- needing a feature to "autokeep" build directories: https://storyboard.openstack.org/#!/story/2001470 | 20:16 |
| SpamapS | tobiash: have done it to my stage env. 😋 | 20:19 |
| SpamapS | will hopefully have re2 ported soon, but with author MIA.. it gets complicated | 20:20 |
| dmsimard | Is the procedure to enable/disable keep on a live executor documented somewhere ? I think it can be enabled through the command socket without restarting the executor but I can't find where | 20:22 |
| SpamapS | It can and I think it is in the help for zuul-executor | 20:23 |
| dmsimard | yeah I see the "keep" and "nokeep" commands, as well as the --keep-jobdir arg | 20:26 |
| dmsimard | Just not 100% positive that running "zuul-executor keep" and "zuul-executor nokeep" (while zuul-executor is running) does what I expect | 20:26 |
| dmsimard | I need to add RAM to my home lab so I can set up zuul/nodepool :/ | 20:27 |
| tobiash | dmsimard: it does what you expect | 20:33 |
| tobiash | the same works with verbose | 20:33 |
| dmsimard | tobiash: ok, so if I want to troubleshoot something, I can safely use "zuul-executor keep" to toggle keep on and then when I'm done "zuul-executor nokeep". | 20:35 |
| tobiash | dmsimard: yes | 20:38 |
| dmsimard | Great, thanks. | 20:38 |
| tobiash | but you have to delete the dirs manually afterwards | 20:38 |
| *** jkilpatr has joined #zuul | 20:53 | |
| *** sshnaidm is now known as sshnaidm|afk | 21:23 | |
| rcarrillocruz | heya folks | 21:24 |
| rcarrillocruz | so looking at post queue on zuul github | 21:24 |
| rcarrillocruz | checking docs, it's not clear to me what event is triggered when a PR is merged | 21:24 |
| rcarrillocruz | i.e. what's the gerrit equivalent ref-updated on GH | 21:26 |
| *** jappleii__ has joined #zuul | 21:32 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add consolidated role for processing subunit https://review.openstack.org/529339 | 21:33 |
| corvus | rcarrillocruz: oh i think it's 'push' | 21:36 |
| rcarrillocruz | really? gah, was assuming it would be an action on the pull_request event | 21:39 |
| rcarrillocruz | k thx corvus | 21:39 |
| dmsimard | Zuul meeting in 5 minutes | 21:55 |
| clarkb | woo I think I finally figured out a test for this cloud error issue | 22:00 |
| clarkb | just in time for the meeting | 22:00 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool feature/zuulv3: Add test_launcher test https://review.openstack.org/533771 | 22:07 |
| clarkb | I think ^ should work now | 22:08 |
| clarkb | that was brain melting | 22:08 |
| *** dkranz has quit IRC | 22:17 | |
| *** hashar has quit IRC | 22:18 | |
| dmsimard | Right now we whitelist "/var/lib/zuul/builds/uuid/work" (my understanding of '~'): http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/ansible/paths.py?h=feature/zuulv3#n25 | 22:44 |
| dmsimard | But we implicitely prepare roles in /var/lib/zuul/builds/uuid/ansible, which is not allowed by that directive | 22:44 |
| dmsimard | Example of role preparation: 2018-01-15 21:20:15,318 DEBUG zuul.AnsibleJob: [build: 406ded37f2f5469fbe0ca8af3011fd5d] Adding role path /var/lib/zuul/builds/406ded37f2f5469fbe0ca8af3011fd5d/ansible/playbook_0/role_2/zuul-jobs/roles | 22:45 |
| dmsimard | This seems to be problematic when loading content from within the /var/lib/zuul/builds/uuid/ansible, see here for example: http://logs.openstack.org/89/514489/3/check/openstack-infra-base-integration-opensuse423/406ded3/job-output.txt.gz#_2018-01-15_21_20_48_120762 | 22:46 |
| dmsimard | Is the fix to authorize "../ansible" (/var/lib/zuul/builds/uuid/ansible) in http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/ansible/paths.py?h=feature/zuulv3#n25 ? | 22:47 |
| pabelanger | why does an untrusted job need access to ansible folder? | 22:49 |
| dmsimard | pabelanger: that's an include_vars task which wants to include vars from that role... | 22:51 |
| dmsimard | pabelanger: i.e, configure-unbound is in /var/lib/zuul/builds/uuid/ansible/roles/configure-unbound and it wants to include /var/lib/zuul/builds/uuid/ansible/roles/configure-unbound/vars/something.yml | 22:51 |
| dmsimard | I think it has to do with using "{{ role_path }}" instead of doing a relative include | 22:52 |
| dmsimard | See the include here http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/roles/configure-unbound/tasks/main.yaml#n41 | 22:53 |
| dmsimard | The role_path is required in the include_vars because Ansible uses precedence when finding relative paths -- so if you have a "vars" folder inside your playbook directory, it's going to look there before looking inside your role's vars directory. | 22:54 |
| corvus | dmsimard: let's chat with mordred about this when he's back | 23:11 |
| mordred | corvus, dmsimard: ohai! sorry - was stuck in a dark hole all day - just now coming up for air - looks like I missed a fun day though :( | 23:12 |
| * mordred reads scrollback | 23:12 | |
| dmsimard | mordred: I have a kept build of our role path issue, summary is above | 23:14 |
| mordred | dmsimard: yah, thank you for the writeup - that at least makes it make more sense... | 23:14 |
| mordred | dmsimard, corvus: as for what to do about it ... I'm not super-crazy about opening up that dir explicitly - since whitelisting it whitelists it for all the things | 23:17 |
| corvus | that could allow untrusted playbooks to overwrite trusted ones. we would need to be very careful with that. | 23:18 |
| mordred | yah | 23:18 |
| mordred | I think it's a topic worthy of some serious pondering ... I think that's our only current use of explicit {{ role_path }} at the moment | 23:19 |
| corvus | mordred: dhellman may have a use case, but it may not be necessary | 23:20 |
| dmsimard | fwiw we can work around the issue by naming one of the directories something else than "vars" | 23:22 |
| dmsimard | But it's a workaround | 23:22 |
| mordred | dmsimard: yah - I think that's a good workaround for now until we have time to sort out a more fundamental approach we're all happy with | 23:22 |
| dmsimard | Either that or we convince upstream that their precedence for include_vars is wrong (i.e, when used from inside a role, the role should have precedence) | 23:23 |
| dmsimard | Which I could try but that's not going to help us in the short term | 23:23 |
| mordred | no, and I don't think we're likely to convince them - I think the current precedence is that way to allow folks to have playbook associated overrides | 23:24 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!