| *** swest has quit IRC | 01:55 | |
| *** swest has joined #zuul | 02:11 | |
| *** bstinson has quit IRC | 05:01 | |
| *** bstinson has joined #zuul | 05:18 | |
| *** gtema has joined #zuul | 05:45 | |
| *** jhesketh has quit IRC | 05:58 | |
| *** xinliang has joined #zuul | 06:03 | |
| *** gtema has quit IRC | 06:37 | |
| *** pcaruana has joined #zuul | 06:51 | |
| *** threestrands_ has quit IRC | 06:58 | |
| *** jpena|off is now known as jpena | 07:40 | |
| *** sshnaidm|off is now known as sshnaidm|rover | 08:03 | |
| *** toabctl has quit IRC | 08:04 | |
| *** toabctl has joined #zuul | 08:08 | |
| *** ssbarnea_ has joined #zuul | 08:20 | |
| openstackgerrit | Stephen Finucane proposed openstack-infra/zuul-jobs master: Default warning-is-error to True for non-legacy Sphinx projects https://review.openstack.org/559348 | 08:24 |
|---|---|---|
| *** jhesketh has joined #zuul | 08:29 | |
| *** jhesketh has quit IRC | 08:47 | |
| *** jhesketh has joined #zuul | 09:06 | |
| *** jpena is now known as jpena|lunch | 12:06 | |
| *** rlandy has joined #zuul | 12:33 | |
| *** pcaruana|afk| has joined #zuul | 13:03 | |
| *** NotPcaruana has joined #zuul | 13:04 | |
| *** dkranz has joined #zuul | 13:28 | |
| *** NotPcaruana has quit IRC | 13:34 | |
| *** pcaruana|afk| has quit IRC | 13:34 | |
| *** jpena|lunch is now known as jpena | 13:47 | |
| *** dmellado has joined #zuul | 13:47 | |
| openstackgerrit | Merged openstack-infra/zuul master: Improve logging around project branches https://review.openstack.org/567128 | 14:32 |
| *** mario-goulart has quit IRC | 15:29 | |
| *** ssbarnea_ has quit IRC | 15:35 | |
| *** pcaruana has quit IRC | 15:52 | |
| clarkb | corvus: any reason to not approve https://review.openstack.org/#/c/566186/2 ? I just approved the aprent change which set up opensans font | 15:58 |
| corvus | clarkb: nope i think we can go for it | 15:59 |
| openstackgerrit | Merged openstack-infra/zuul-website master: Switch to Open Sans font https://review.openstack.org/566185 | 16:01 |
| *** acozine1 has joined #zuul | 16:01 | |
| *** ssbarnea_ has joined #zuul | 16:01 | |
| openstackgerrit | Merged openstack-infra/zuul-website master: Change tagline to dark blue https://review.openstack.org/566186 | 16:03 |
| openstackgerrit | Merged openstack-infra/zuul-website master: Update tagline https://review.openstack.org/567295 | 16:03 |
| corvus | https://zuul-ci.org/ is up to date | 16:20 |
| pabelanger | nice | 16:26 |
| pabelanger | looking at the following, it doesn't seem clear to me: | 16:26 |
| pabelanger | Use the same Ansible to deploy your system and run your tests. | 16:26 |
| pabelanger | should that maybe be Use the same Ansible playbooks... | 16:26 |
| corvus | perhaps? | 16:26 |
| pabelanger | or is it the same version of ansible | 16:28 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-website master: Clarify usage of Ansible for users https://review.openstack.org/567640 | 16:32 |
| *** pcaruana has joined #zuul | 16:36 | |
| *** pcaruana has quit IRC | 16:44 | |
| *** sshnaidm|rover is now known as sshnaidm|off | 16:52 | |
| mordred | pabelanger: I just rebased https://review.openstack.org/#/c/564507/ so it picks up the fixes for the pep8 mess ... you wanna +A it again real quick? | 17:10 |
| *** ssbarnea_ has quit IRC | 17:17 | |
| pabelanger | great | 17:19 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-website master: Clarify usage of Ansible for users https://review.openstack.org/567640 | 17:21 |
| *** ssbarnea_ has joined #zuul | 17:25 | |
| *** ssbarnea_ has quit IRC | 17:28 | |
| pabelanger | I think we might want to consider a 3.0.3 release of zuul to pick up: http://git.openstack.org/cgit/openstack-infra/zuul/commit/?id=4b1d92d59bb0848e3aacc748147f46482ad36100 | 17:55 |
| pabelanger | right now, zuul from pypi can't be installed on ubuntu-xenial because of it | 17:55 |
| clarkb | I was thinking about that and should we use env markers to allow newer asyncio libs on 3.6? | 17:57 |
| clarkb | we have tests for 3.6 now right? | 17:57 |
| pabelanger | clarkb: yah, I think that's what was done | 17:59 |
| pabelanger | async-timeout<3.0.0;python_version<='3.5.2' | 17:59 |
| clarkb | ah cool I thought one of the pins was just a < and called it good | 17:59 |
| pabelanger | aiohttp maybe? | 18:00 |
| clarkb | pabelanger: ya aiohttp<3.0.0 | 18:00 |
| clarkb | it should have the same env marker likely | 18:00 |
| pabelanger | k, let me fix quickly | 18:00 |
| clarkb | unless it doesn't work at all | 18:00 |
| pabelanger | not sure | 18:00 |
| clarkb | but testing should cover that? there are zuul-web tests at least | 18:00 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Only cap aiohttp for python 3.5.2 and below https://review.openstack.org/567663 | 18:03 |
| pabelanger | clarkb: looks like we need a +3 on https://review.openstack.org/565881/ for py36 jobs | 18:03 |
| *** ssbarnea_ has joined #zuul | 18:03 | |
| *** electrofelix has quit IRC | 18:04 | |
| clarkb | the zuul docs have fedora directions which is 3.6 too right? So probably a good thing to add 3.6 testing | 18:04 |
| clarkb | corvus: ^ do you want to rview that at all? | 18:04 |
| clarkb | pabelanger: we can probably remove the env marker from uvloop;python_version>='3.5' too if we want to clean that up (since I think 3.5 is required anyways) | 18:07 |
| pabelanger | clarkb: sure | 18:07 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Remove env marker from uvloop https://review.openstack.org/567665 | 18:08 |
| fungi | i'm increasingly happy with the https://zuul-ci.org/ site | 18:14 |
| Shrews | i don't think it uses <blink> nearly enough | 18:15 |
| clarkb | pabelanger: on 567663 I think you need a second aiohttp for >3.5.2 | 18:15 |
| fungi | Shrews: i was also thinking we needed some flaming text and spinning skulls. also where is the barbed wire horizontal rule? | 18:16 |
| Shrews | fungi: ikr?? | 18:16 |
| pabelanger | clarkb: a 2nd entry? | 18:16 |
| * fungi briefly wonders if geocities still exists, then decides it's better not to check | 18:17 | |
| clarkb | pabelanger: ya as is you won't install aiohttp on python 3.6 for example | 18:17 |
| pabelanger | clarkb: wouldn't that also be an issue for async-timeout and yarl? | 18:17 |
| clarkb | pabelanger: no they are aiohttp dependencies aiui so we only pull them in transitively | 18:17 |
| *** ssbarnea_ has quit IRC | 18:17 | |
| clarkb | whereas we directly use aiohttp | 18:17 |
| pabelanger | clarkb: k, so can I just do aiohttp or do I need to also include env filter | 18:18 |
| clarkb | pabelanger: you need an env filter I think otherwise it will apply to <= 3.5.2 as well | 18:18 |
| pabelanger | okay | 18:19 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Only cap aiohttp for python 3.5.2 and below https://review.openstack.org/567663 | 18:23 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Remove env marker from uvloop https://review.openstack.org/567665 | 18:23 |
| pabelanger | clarkb: thanks, haven't really used markers much before | 18:23 |
| clarkb | cool, I think if we can get the 3.6 jobs in then we recheck and go fro mthere | 18:25 |
| *** jpena is now known as jpena|off | 18:25 | |
| clarkb | I don't want to add new jobs to zuul without corvus input though (which is probably why that changes wasn't approved by other reviewers) | 18:25 |
| clarkb | that said I do think it is a good idea here espceially since our how to docs have people using 3.6 | 18:25 |
| corvus | i had a previous +W on it, i just reapplied it | 18:25 |
| clarkb | ah cool | 18:25 |
| corvus | clarkb, pabelanger: i was going to suggest we release what openstack is running today, but maybe we should cheat and release current master to get the requirements change too? | 18:26 |
| corvus | the additional changes in master that we aren't running are *probably* okay :) | 18:26 |
| corvus | the only thing that actually changes anything in zuul is the start/end timestamp in zuul_json change | 18:27 |
| clarkb | 7816dfc3aefe6958dcbc00c3d5e99b2a8b346d57 is what openstack is running right? | 18:27 |
| corvus | yep | 18:27 |
| clarkb | then ya the callback stuff is all I would be concerned about | 18:27 |
| clarkb | and its just adding a new field to the results dicts? that seems relatively safe. Has a test too | 18:30 |
| pabelanger | wfm | 18:30 |
| corvus | so how about i tag 1e3e5d33fb140d5dc993eeed6f8b13248105647a as 3.0.3 ? | 18:35 |
| corvus | or do we need any more requirements changes in? | 18:35 |
| pabelanger | I think they could wait, we are just uncapping and cleaning up env markers | 18:38 |
| pabelanger | 1e3e5d33fb140d5dc993eeed6f8b13248105647a +1 | 18:38 |
| clarkb | wfm | 18:45 |
| *** sshnaidm|off has quit IRC | 18:47 | |
| openstackgerrit | Merged openstack-infra/zuul master: Add tox-py36 jobs https://review.openstack.org/565881 | 18:49 |
| fungi | looks like a good 3.0.3 candidate to me | 19:09 |
| fungi | 7816dfc has been running in production for us ~36 hours, looks like | 19:10 |
| fungi | oh, wait, you said 1e3e5d3 | 19:11 |
| fungi | 1b747ca6 is now master branch tip | 19:12 |
| dmsimard | huh, I'm seeing results from git.zuul-ci.org on google searches. | 19:12 |
| fungi | dmsimard: is that bad? | 19:12 |
| dmsimard | No, I suppose I just don't remember seeing results from git.o.o | 19:13 |
| fungi | is it returning hits for non-zuul repos or anything? | 19:13 |
| dmsimard | not that I can see, this one is zuul itself | 19:14 |
| clarkb | I noticed recently that google is returning hits for gerrit changes too (not zuul specific) | 19:14 |
| fungi | so after 7816dfc3, 1e3e5d33 seems to add its own py36 testing, the async-timeout pin, a logo for docs, start/end timestamp in the json callback, skips attempting python3-devel install on centos, and adds systemd drop-in files for centos | 19:16 |
| fungi | those seem fairly non-intrusive | 19:16 |
| fungi | no, wait, the tox-py36 jobs come even later in 1b747ca6 | 19:16 |
| fungi | along with logging improvements related to project branches | 19:16 |
| fungi | so 1b747ca6 seems at least as good a 3.0.3 release candidate to me | 19:17 |
| corvus | yeah, just didn't want to include more un-beta-tested code changes than necessary | 19:47 |
| *** sshnaidm|off has joined #zuul | 19:49 | |
| corvus | pushed 3.0.3 | 20:21 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Add favicon https://review.openstack.org/567685 | 20:30 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: DNM: test favicon https://review.openstack.org/567686 | 20:30 |
| *** dkranz has quit IRC | 20:34 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Add favicon https://review.openstack.org/567685 | 20:39 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: DNM: test favicon https://review.openstack.org/567686 | 20:39 |
| *** sc68cal_ is now known as sc68cal | 20:44 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: DNM: test favicon https://review.openstack.org/567686 | 20:51 |
| *** acozine1 has quit IRC | 21:20 | |
| ianw | what do you think the chances are that it's something to do with using with_items: that the logging on the *second* an subsequent script calls in ... | 21:23 |
| ianw | http://logs.openstack.org/79/566779/38/check/dib-functests-python3/e164153/job-output.txt.gz#_2018-05-10_06_13_30_260438 | 21:23 |
| ianw | goes missing | 21:23 |
| ianw | it's "run dib functional tests" in https://review.openstack.org/#/c/566779/38/roles/dib-functests/tasks/main.yaml | 21:24 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Add favicon https://review.openstack.org/567685 | 21:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: DNM: test favicon https://review.openstack.org/567686 | 21:25 |
| corvus | ianw: a with_items + zuul_console problem seems likely. the good news is that we can actually regression test that now with the zuul-remote tests | 21:26 |
| ianw | corvus: ok, i'll dig a little deeper. i'm wondering if the essentially "1> >(logfilter.py)" we do in dib is part of it | 21:29 |
| ianw | but i'll make up some test scripts and see what goes wrong | 21:30 |
| *** dkranz has joined #zuul | 21:40 | |
| clarkb | corvus: that latest test worked on firefox | 21:50 |
| corvus | clarkb: ah, so it must have been grabbing one of the other items | 21:52 |
| corvus | i do think i like it better without the word 'zuul' | 21:52 |
| corvus | http://logs.openstack.org/86/567686/4/check/zuul-website-build/1afdfb7/html/ vs http://logs.openstack.org/86/567686/3/check/zuul-website-build/97055ef/html/ | 21:53 |
| clarkb | ++ | 21:53 |
| clarkb | its a bit more distinct on my tab panel that way. Also it says Zuul: Stop merging broken code just to the right of the icon | 21:54 |
| corvus | heh. maybe we should use something other than the tagline for the page title. there's an ambiguous reading when you just read it like that. sort of like "zuul, stop hitting yourself". | 21:55 |
| clarkb | maybe just "Zuul" | 21:57 |
| corvus | always a classic | 21:57 |
| SpamapS | So I have an interesting problem today | 22:02 |
| SpamapS | Using our corporate images is proving really difficult | 22:02 |
| SpamapS | One thing they do is join to our AD domain and restart sshd to enable AD users to ssh in... | 22:02 |
| SpamapS | What that means is nodepool marks the node as `ready` and then sshd stops working for a few seconds some time later (~1 minute) | 22:03 |
| SpamapS | Wondering if anybody has any quick fixes for this. | 22:03 |
| clarkb | SpamapS: I thought sshd restarts were supposed to be safe for that | 22:03 |
| SpamapS | What's happening is that jobs are retry_limit'ing regularly. | 22:03 |
| SpamapS | 2018-05-10 14:31:02,579 DEBUG zuul.AnsibleJob: [build: da602c022cbc48d8878a3feef28d7c9a] Ansible output: b' "msg": "SSH Error: data could not be sent to remote host \\"10.26.15.53\\". Make sure this host can be reached over ssh",' | 22:04 |
| SpamapS | One way others in the company fix this is by setting a piece of nova metadata that avoids the domain join. | 22:04 |
| SpamapS | but I don't see any way to configure nodepool to do that. | 22:05 |
| SpamapS | So before I add that to nodepool, just looking for something that maybe we can do to ansible/zuul to make it be a little more forgiving. | 22:05 |
| SpamapS | one thought I've had is to try and detect if a box has joined to the domain in our base pre-playbook. | 22:05 |
| SpamapS | And if it hasn't, just spin for a while waiting for sshd to go away and then come back. | 22:06 |
| clarkb | ya you can add metadata to images that are uploaded to glance but I don't think you can do per instance metadata other than ssh keys? | 22:06 |
| clarkb | and if you are using preuploaded images that won't help | 22:06 |
| mordred | yah - we don't currently have support for cloud-init metadata because eww | 22:07 |
| mordred | SpamapS: I thnk the pre-playbook idea is a thing you should be able to hack up quickly and should do the job | 22:08 |
| mordred | SpamapS: and at least provide a bit of space for figuring out a possible better solution | 22:08 |
| SpamapS | It's not cloud-init metadata | 22:08 |
| SpamapS | it's nova metadata | 22:08 |
| SpamapS | like the same place we set 'groups' | 22:08 |
| mordred | oh - like just an arbitrary metadata? | 22:09 |
| mordred | gotcha | 22:09 |
| mordred | that seems easier to add support for :) | 22:09 |
| SpamapS | Yeah I found the lines where those are set | 22:09 |
| SpamapS | so I can just plumb a thing in from the label. | 22:09 |
| corvus | yeah, adding metadata sounds like a good idea. i think otherwise, the risk is that the window of non-functionality is racy. | 22:10 |
| clarkb | SpamapS: ya reading the unit for sshd locally it a sighup and should reload config like that gracefully. You might also want to check if your images are doing the right thing | 22:10 |
| SpamapS | clarkb: oh, they're not. ;) | 22:11 |
| SpamapS | but actually I believe it has to be restarted to pick up the PBIS nss driver changes. | 22:11 |
| corvus | also perhaps it would be possible to avoid starting sshd until that config was in place? then nodepool will wait longer before declaring it ready. | 22:11 |
| clarkb | SpamapS: I believe it does a full new process but does socket handoff | 22:14 |
| clarkb | SpamapS: there is a good chance that it would work | 22:14 |
| clarkb | "sshd rereads its configuration file when it receives a hangup signal, SIGHUP, by executing itself with the name and options it was started with, e.g. /usr/sbin/sshd." | 22:14 |
| SpamapS | The process for me to change the image is far harder than the process to change nodepool. | 22:15 |
| SpamapS | but that is encouraging | 22:15 |
| SpamapS | and maybe I could fix that | 22:15 |
| clarkb | additionally (at least on xenial) there is ssh.service enabled by default but also ssh.socket. ssh.socket will use systemd socket activation to queue up connections. This may also help | 22:18 |
| SpamapS | We're on CentOS 7 | 22:20 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Allow upload-logs to toggle compression of console logs https://review.openstack.org/567696 | 22:20 |
| pabelanger | so, ^ is a breaking change and likely needs an email to zuul-discuss, but believe make upload-logs more usable by default, when we disable compression of console logs | 22:21 |
| clarkb | pabelanger: looks like zuul web tests did say it doesn't like newer aiohttp | 22:23 |
| clarkb | I guess that is a good thing to know | 22:23 |
| pabelanger | clarkb: ah, I haven't look at results yet | 22:23 |
| pabelanger | not much help in logs | 22:25 |
| clarkb | ya, I think it might be failing a bit more catastrophically than the test framework can pull log sout of | 22:26 |
| clarkb | SpamapS: and I'm guessing you don't want to use your own images because part of what you are testing is that the corp images work? | 22:30 |
| SpamapS | clarkb: right, we want to be using the same image that we'll be deploying to. | 22:33 |
| pabelanger | okay, sent email to ML about upload-logs | 22:34 |
| SpamapS | clarkb: we do a lot of custom images too, when speed is more important than accuracy. | 22:34 |
| SpamapS | but like, a lot of these are 2 minute tests | 22:34 |
| SpamapS | and it's nice to know that your unittests work on the image. | 22:34 |
| SpamapS | or that your deployment code works on it | 22:34 |
| clarkb | pabelanger: what is the motivation or background on the logs change/ | 22:35 |
| pabelanger | clarkb: lets configuration need in apache / nginx out of the box | 22:35 |
| clarkb | pabelanger: I'm not sure I'm a fan of that one. Compressed logs by default is a major win and its super simple to make your browser inflate them | 22:35 |
| clarkb | and its only like one extra click if it doesn't | 22:37 |
| pabelanger | sorry, click where? | 22:38 |
| clarkb | pabelanger: your browser will open a "open with" dialog and it should default to archive viewr which then opens the file | 22:38 |
| clarkb | at least with firefox | 22:38 |
| pabelanger | chrome forces me to download it | 22:39 |
| pabelanger | then open it locally with editor | 22:39 |
| clarkb | oh interesting ff gives you the option | 22:39 |
| clarkb | its the same dialog with two choices, open with tool or download | 22:39 |
| pabelanger | yah, I imagine there is a why in chrome to do it, but don't think out of box | 22:40 |
| pabelanger | I don't have a strong option if we compress by default, but would like a way to toggle it | 22:41 |
| pabelanger | i just figured, it would be one less thing to configure (web server) to view logs in a browser | 22:41 |
| clarkb | in this particular case I think the rlatively simple web server config (which you hvae to configure anyways) is worth it compared to having lots of easily compressable files that aren't compressed | 22:42 |
| clarkb | if we were serving the files without needing to configure the web server at all I'd probably agree with you, but you have to configure something | 22:42 |
| pabelanger | I agree, I could just place logs into /var/www/html/ and use the default vhost setup by the package also. I do think, we are recommending gzip of console logs out of the box, we'll also want to add documentation around how to configure that. | 22:46 |
| clarkb | pabelanger: I think it may be as simple as 'AddEncoding x-gzip .gz' 'AddType text/plain job-output.txt.gz' ? | 23:01 |
| clarkb | 'AddType text/plain .txt.gz' might do it too | 23:01 |
| clarkb | er still need the addencoding though | 23:02 |
| *** rlandy has quit IRC | 23:45 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!