*** tosky has quit IRC | 00:18 | |
*** zenkuro has quit IRC | 00:23 | |
*** zenkuro has joined #zuul | 00:23 | |
*** wuchunyang has joined #zuul | 00:24 | |
*** wuchunyang has quit IRC | 00:28 | |
pabelanger | corvus: \o/ | 00:57 |
---|---|---|
pabelanger | thank you again | 00:57 |
*** sean-k-mooney1 has joined #zuul | 01:01 | |
*** sean-k-mooney has quit IRC | 01:03 | |
*** zenkuro has quit IRC | 01:53 | |
*** sean-k-mooney2 has joined #zuul | 02:15 | |
*** sean-k-mooney1 has quit IRC | 02:17 | |
*** jhesketh has quit IRC | 02:27 | |
*** jhesketh has joined #zuul | 02:33 | |
*** bhavikdbavishi has joined #zuul | 02:49 | |
openstackgerrit | Ian Wienand proposed zuul/zuul master: web: Add optional link prop to title-with-icon and results https://review.opendev.org/763471 | 02:58 |
*** bhavikdbavishi1 has joined #zuul | 03:06 | |
*** bhavikdbavishi has quit IRC | 03:07 | |
*** bhavikdbavishi1 is now known as bhavikdbavishi | 03:07 | |
*** rlandy|rover has quit IRC | 03:42 | |
*** iurygregory has quit IRC | 03:42 | |
*** hamalq has quit IRC | 03:56 | |
*** wuchunyang has joined #zuul | 04:09 | |
*** wuchunyang has quit IRC | 04:14 | |
*** bhavikdbavishi has quit IRC | 04:23 | |
*** bhavikdbavishi has joined #zuul | 04:23 | |
*** evrardjp has joined #zuul | 05:33 | |
*** bhavikdbavishi has quit IRC | 05:55 | |
*** bhavikdbavishi1 has joined #zuul | 05:55 | |
*** bhavikdbavishi has joined #zuul | 05:58 | |
*** bhavikdbavishi1 has quit IRC | 05:59 | |
*** rpittau|afk is now known as rpittau | 06:52 | |
*** bhavikdbavishi has quit IRC | 06:55 | |
*** bhavikdbavishi has joined #zuul | 07:15 | |
felixedel | ianw: I like that one. Maybe you could directly include your follow-up commits into my original change and remove the parts that are not needed anymore? So we don't end up with unnecessary CSS classes and other stuff in the first place. | 07:19 |
*** iurygregory has joined #zuul | 07:39 | |
ianw | felixedel: ok, i don't quite have time right now, but feel free if you like to squash them :) otherwise i'll do it later | 07:57 |
*** jcapitao has joined #zuul | 08:14 | |
openstackgerrit | Matt Kucia proposed zuul/zuul master: tox: Show line length style errors on linters https://review.opendev.org/758622 | 08:23 |
openstackgerrit | Matt Kucia proposed zuul/zuul master: configloader: Log more error information https://review.opendev.org/758660 | 08:23 |
openstackgerrit | Matt Kucia proposed zuul/zuul master: drivers: Bitbucket skeleton https://review.opendev.org/757532 | 08:23 |
openstackgerrit | Matt Kucia proposed zuul/zuul master: doc: Bitbucket driver https://review.opendev.org/758661 | 08:23 |
openstackgerrit | Matt Kucia proposed zuul/zuul master: examples: Sample configuration for Bitbucket driver https://review.opendev.org/758623 | 08:23 |
openstackgerrit | Matt Kucia proposed zuul/zuul master: drivers: Bitbucket - Source https://review.opendev.org/758662 | 08:23 |
*** bhavikdbavishi has quit IRC | 08:36 | |
*** saneax has joined #zuul | 08:36 | |
*** jpena|off is now known as jpena | 08:37 | |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: Handle the yaml merge operator https://review.opendev.org/763532 | 09:15 |
*** hashar has joined #zuul | 09:26 | |
zbr | ianw: are you around? related to the artifact details/expansion.... | 09:30 |
openstackgerrit | zbr proposed zuul/zuul master: Consolidate js/jsx indentation https://review.opendev.org/747377 | 09:40 |
*** nils has joined #zuul | 09:41 | |
*** tosky has joined #zuul | 09:57 | |
avass | does zuul work well with gerrit replicas? | 10:08 |
avass | I would guess opendev is using that | 10:08 |
*** openstackgerrit has quit IRC | 10:25 | |
*** saneax has quit IRC | 10:51 | |
*** zenkuro has joined #zuul | 11:13 | |
zbr | avass: yes | 11:19 |
zbr | avass: felixedel ianw tobiash: can you please help me deliver the ^ js/jsx indentation fix? | 11:20 |
*** bhavikdbavishi has joined #zuul | 11:25 | |
*** bhavikdbavishi1 has joined #zuul | 11:29 | |
*** bhavikdbavishi has quit IRC | 11:31 | |
*** bhavikdbavishi1 is now known as bhavikdbavishi | 11:31 | |
*** sean-k-mooney2 is now known as sean-k-mooney | 11:50 | |
*** jcapitao is now known as jcapitao_lunch | 11:57 | |
tobiash | zuul-maint: current nodepool master spams our logs due to multi ssh key handling: http://paste.openstack.org/show/800251/ | 12:01 |
*** jfoufas1 has joined #zuul | 12:13 | |
*** zenkuro has quit IRC | 12:22 | |
*** zenkuro has joined #zuul | 12:23 | |
*** sassyn has joined #zuul | 12:26 | |
sassyn | hi everyone. I have trouble this morning to make zuul working again. | 12:26 |
sassyn | I have a power issue at my Data Center. Zuul server were booted... all was working fine for few days now | 12:27 |
sassyn | no change in the syste, | 12:27 |
*** jpena is now known as jpena|lunch | 12:27 | |
sassyn | system* | 12:27 |
*** hashar has quit IRC | 12:27 | |
sassyn | however today all jobs get the Waiting on logger | 12:27 |
sassyn | and get into a RETRY_LIMIT | 12:28 |
sassyn | I can see using TCP dump that the Executor trying to open a connection to Zuul Streaming Console port @ 19885 TCP | 12:29 |
sassyn | like this | 12:29 |
sassyn | 14:29:33.573735 IP Executor01.57276 > 172.17.0.2.19885: Flags [S], seq 2834551981, win 65495, options [mss 65495,sackOK,TS val 616387407 ecr 0,nop,wscale 7], length 0 | 12:30 |
sassyn | ack 1, win 0, length 0 | 12:30 |
sassyn | there is no firewall and the Executor01 can access the 172.17.0.2 machine and all ports | 12:30 |
sassyn | in the worker I can't see anything that is bind to the port 19885 | 12:30 |
sassyn | what do I miss? | 12:30 |
avass | sassyn: the executor needs access to that port for live log streaming | 12:32 |
sassyn | the executor do have access to the worker nodes of the nodepool | 12:33 |
sassyn | in this port | 12:33 |
avass | and this role starts that daemon on the node: https://zuul-ci.org/docs/zuul-jobs/general-roles.html?highlight=stream#role-start-zuul-console | 12:33 |
avass | so if it's never started it won't be able to connect to it | 12:33 |
sassyn | start-zuul-console | 12:34 |
sassyn | I don't have this role setup | 12:34 |
avass | that's probably why you're getting 'Waiting on logger' | 12:34 |
sassyn | i will add this to my pre jobs in the trusted zuul repo | 12:34 |
sassyn | but How it was working before? | 12:35 |
avass | it shouldn't have unless that had been started :) | 12:35 |
avass | but zuul uploads the logs after the jobs finishes as well | 12:35 |
sassyn | ok | 12:36 |
sassyn | so it should come after the prepare-workspace-git? | 12:36 |
avass | as long as you do that in a post/cleanup run | 12:36 |
sassyn | i have log-inventory, add-build-sshkey and prepare-workspace-git in my pre roles | 12:36 |
avass | we do it after add-build-host-key, so our second role | 12:37 |
avass | the earlier the better I'd say | 12:37 |
tobiash | sassyn: typically the waiting for logger is non-fatal in most cases, so you might have a different issue as well | 12:45 |
sassyn | I add the tole | 12:46 |
sassyn | role* seems to be working fine | 12:46 |
sassyn | now thanks! | 12:46 |
*** rlandy has joined #zuul | 12:53 | |
*** rlandy is now known as rlandy|rover | 12:53 | |
*** hashar has joined #zuul | 12:53 | |
fungi | avass: zbr: opendev doesn't connect its zuul to a git replica nor clustered gerrit, so regardless of what you mean by gerrit replica i think the answer is no, opendev's zuul connects to their primary (and only) gerrit server to receive events and pull change refs, mainly because we don't want it to race replication delays | 12:55 |
fungi | tobiash: should we catch paramiko.ssh_exception.SSHException i guess? and log something at info level (or not at all)? | 12:56 |
tobiash | fungi: yes I think so | 12:57 |
*** bhavikdbavishi has quit IRC | 13:01 | |
zbr | fungi: i guess i confused it with gitea, which was the distributed one. | 13:02 |
fungi | yeah, our zuul doesn't interact with our gitea cluster at all | 13:03 |
zbr | fungi: can you please +W https://review.opendev.org/#/c/747377/ before it needs another rebase? | 13:03 |
*** zenkuro has quit IRC | 13:03 | |
*** zenkuro has joined #zuul | 13:03 | |
-openstackstatus- NOTICE: The Gerrit service at review.opendev.org will be offline starting at 15:00 UTC (roughly two hours from now) for a weekend upgrade maintenance: http://lists.opendev.org/pipermail/service-announce/2020-October/000012.html | 13:04 | |
*** jcapitao_lunch is now known as jcapitao | 13:04 | |
tobiash | remote: https://review.opendev.org/763553 Don't spam logs due to non-existing hostkeys | 13:09 |
tobiash | looks like the gerrit bot is already offline ;) | 13:09 |
fungi | tobiash: oh, i also just pushed https://review.opendev.org/763555 Silence paramiko exceptions for incompatible keys | 13:18 |
fungi | and no, the openstackgerrit irc bot has probably just fallen in a netsplit again | 13:19 |
tobiash | fungi: hrm, that if actually should have catched this already | 13:20 |
fungi | tobiash: it didn't have an else condition though | 13:21 |
fungi | so it fell through to raising anyway i think? | 13:21 |
tobiash | fungi: but the message would not match the if | 13:21 |
tobiash | but the exception clause doesn't match | 13:22 |
tobiash | paramiko is throwing paramiko.ssh_exception.SSHException and we catch paramiko.SSHException | 13:22 |
fungi | ahh, yeah i wonder if paramiko changed where it keeps its exceptions | 13:23 |
tobiash | fungi: oh wait, I think that log comes from some thread within paramiko itself | 13:23 |
tobiash | since the stack trace goes straight to a run method and no zuul path is in there | 13:24 |
fungi | it does lack nodepool source context in the traceback | 13:24 |
fungi | maybe nodepool can't catch these directly? | 13:24 |
tobiash | looks like that | 13:24 |
tobiash | I guess we need to look into paramiko to see if there's a way to inhibit that | 13:25 |
tobiash | and if not I'll maybe exclude paramiko from our log config :D | 13:25 |
fungi | we don't see those in our launcher debug logs at least | 13:27 |
tobiash | do run them already with https://review.opendev.org/761229 included? | 13:27 |
fungi | yes, we added it so that nodes could be rebooted into "fips mode" during job run time, which will change what host keys ssh can use | 13:29 |
tobiash | so your log config already excludes paramiko? | 13:29 |
fungi | i suppose so. checking now | 13:29 |
tobiash | confirmed in paramiko, it explicitly logs the exception | 13:31 |
tobiash | so the only way is to explicitly exclude paramiko.transport from the log config | 13:31 |
*** jpena|lunch is now known as jpena | 13:32 | |
fungi | tobiash: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/nodepool-launcher/files/logging.conf | 13:32 |
fungi | that's ours | 13:33 |
tobiash | so in your case it won't be in the debug log, but in the console log only | 13:33 |
*** zenkuro has quit IRC | 13:40 | |
*** zenkuro has joined #zuul | 13:41 | |
-openstackstatus- NOTICE: The Gerrit service at review.opendev.org will be offline starting at 15:00 UTC (roughly one hour from now) for a weekend upgrade maintenance: http://lists.opendev.org/pipermail/service-announce/2020-October/000012.html | 14:02 | |
avass | fungi: thanks, good to know | 14:16 |
avass | fungi: we've been growing a lot lately so we've got a lot of load on our gerrit and we're exploring solutions | 14:17 |
corvus | fungi, zbr, tristanC: the js indentation patch is incompatible with the default style in emacs, which is used by at least tristan and myself | 14:19 |
corvus | tobiash: ^ | 14:19 |
tobiash | corvus: oh, can we abort it still? | 14:20 |
corvus | fungi, zbr, tobiash, tristanC: honestly, i would prefer that we *not* enforce indentation in the linter because i think it's a waste of time for us to even talk about it. it was all readable before -- we worked on that code for years with no problems | 14:20 |
corvus | tobiash: https://review.opendev.org/747377 merged | 14:20 |
corvus | i did not look at the change because i have a policy of not reviewing whitespace changes | 14:20 |
corvus | i only just now noticed that it also *enforces* whitespace in the linter | 14:21 |
corvus | i propose we choose one of the following options: a) just back out the linting enforcement (i still don't care about the whitespace, so i'm fine with all the 'consolidation' changes). if that isn't acceptable, then b) someone should figure out the eslintrc settings to make it compatible with the editors we use (or figure out the editor settings to make them compatible with the linter). there's like a | 14:24 |
corvus | whole page of eslint settings for whitespace https://eslint.org/docs/rules/indent ; considering my feelings on whitespace, i don't personally plan on doing (b) | 14:24 |
tobiash | I think I'd probably prefer first a) then b) (eslint settings) as I'd prefer to have some validation (we have pep8 for python as well) so we have a somewhat consistent indentation (in the past there were some changes that introduced inconsistent indentation that I noticed in reviews which lead to that stack in the end) | 14:28 |
zbr | i am back | 14:28 |
zbr | that identation is standard, is not imposed by me. if a particular editor does not work by default, is a bug fix for it, imho | 14:29 |
corvus | tobiash: i hold differing opinions on python/js. indentation is part of the code in python, so it's important to the actual execution for the code. in js, it's purely stylistic. i continue to hold to the belief that it's not worth spending our time arguing about stylistic aspects like indentation. | 14:30 |
zbr | the whole point was to avoid random indentation, zuul used to be quite consistent but during the last year the js(x) become....very relaxed. | 14:30 |
corvus | zbr: if zuul js was consistent at all, it was to the emacs default which you say is non-standard | 14:31 |
corvus | (because most of the original authors of the js use emacs) | 14:31 |
zbr | i guess we could add a helper config for emacs? | 14:31 |
corvus | tbh, i kind of like the emacs style, and i disagree that it's non-standard. even the eslint page lists different styles used by different projects/communities | 14:32 |
corvus | you could say the zuul community had a standard (which i suspect can be expressed with an eslint config) | 14:32 |
corvus | you just changed it | 14:32 |
corvus | and i *still* don't care | 14:32 |
corvus | i don't mind if it changes | 14:33 |
corvus | the only thing i care about is not spending any more time talking about this | 14:33 |
corvus | this is a decision we made very early on in the zuul project | 14:33 |
zbr | if the project wants it own standard, it is possible to configure the eslint to match custom project preferences. | 14:33 |
corvus | to not nitpick about style | 14:33 |
zbr | nope | 14:33 |
zbr | i am ok as long we pick one style, not if we avoid picking one | 14:34 |
corvus | zbr: we're going to have to disagree on that one. i'm not going to spend any time trying to convince you. | 14:34 |
corvus | remote: https://review.opendev.org/763568 Don't enforce js whitespace | 14:35 |
corvus | tobiash, fungi: ^ that's (a) | 14:35 |
corvus | i still will agree with (b) if someone wants to research and implement that and a sufficient number of maintainers do as well, though i would prefer not to spend any more time on it. | 14:36 |
corvus | all of the zuul maintainers are a bit strapped for time right now and i think we should spend it wisely | 14:37 |
corvus | tobiash: do we need to change zuul's default logging config for paramiko? | 14:39 |
tobiash | corvus: maybe, we have a custom one as well | 14:39 |
tobiash | at least I saw that paramiko wants to have a configured logger | 14:40 |
tobiash | otherwise it seems to configure its own handler | 14:40 |
tobiash | in our deployment we use stdout as main log output so that's bad and I had to put paramiko explicitly on a NullHandler | 14:41 |
zbr | corvus: tobiash: what about the first change on the summary page? i managed to get the copy uuid working as described on the ml | 14:44 |
avass | I don't really care much about style as long as the code is readable. | 14:47 |
tobiash | remote: https://review.opendev.org/763573 Prevent paramiki.transport log spam | 14:47 |
tobiash | corvus: I did something similar to this ^ | 14:47 |
corvus | tobiash: yeah, that's what i was thinking we might need based on what i've read | 14:47 |
tristanC | corvus: i agree with not changing indentation, i think we should use eslint more to detect code defect, and less for code style | 14:48 |
corvus | tristanC: oof, it's error level? ouch, that seems potentially problematic | 14:49 |
corvus | tobiash: ^ | 14:49 |
corvus | tristanC: sorry tabfail | 14:49 |
tobiash | corvus: yes, that's outputting those exceptions with error level | 14:49 |
tobiash | and there seems to be no way other than this or avoid the exceptions in the first place | 14:50 |
clarkb | its an interesting choice on their part to log an error then raise an exception too in a library | 14:50 |
clarkb | seems like they should just raise the exception. Maybe w can convince them to change that upstream? | 14:50 |
corvus | tobiash: do you think it's worth making a custom handler that filters it or something? | 14:50 |
tobiash | corvus: it might not be as problematic since nodepool itself logs any exception with the full trace | 14:50 |
corvus | tobiash: ok, so if we really encounter an error, we'll get it in our thread and log it? | 14:51 |
clarkb | corvus: yes we should, because they raise too | 14:51 |
tobiash | corvus: yes, since nodepool logs any errors by itself in the nodepool.* packages | 14:51 |
corvus | ok, then i think 763573 is probably ok | 14:51 |
tobiash | that's only about the stuff paramiko logs by itself | 14:51 |
corvus | littly scary but ok :) | 14:51 |
*** Goneri has joined #zuul | 14:51 | |
*** rpittau is now known as rpittau|afk | 14:53 | |
fungi | i'm fine backing out the linter enforcement, i find it pointless personally, i'm just tired of people pestering me to review changes like that but i should probably adopt a policy like yours and just tell them sorry i'm not reviewing that (or -1 it because i find indentation checking for languages without semantic whitespace unnecessary) | 14:55 |
fungi | happy to revert the indentation change next week after gerrit maintenance but i need to focus on that now. sorry i approved it :/ | 14:56 |
corvus | zbr: left review; i think you should solicit more comments on the points ianw raises | 14:58 |
zbr | if i remember well python has advanced ability to filter logging messages (so we could eliminate the noisy one) | 14:58 |
-openstackstatus- NOTICE: The Gerrit service at review.opendev.org is offline for a weekend upgrade maintenance, updates will be provided once it's available again: http://lists.opendev.org/pipermail/service-announce/2020-October/000012.html | 15:05 | |
*** rlandy|rover is now known as rlandy|rover|brb | 15:31 | |
zbr | corvus: thanks, for both. | 15:39 |
*** jfoufas1 has quit IRC | 15:48 | |
*** rlandy|rover|brb is now known as rlandy|rover | 15:53 | |
*** ttx has quit IRC | 16:39 | |
*** ttx has joined #zuul | 16:45 | |
*** tosky has quit IRC | 16:56 | |
*** sassyn has quit IRC | 17:18 | |
*** hashar has quit IRC | 17:46 | |
*** slaweq has quit IRC | 17:49 | |
*** hamalq has joined #zuul | 17:51 | |
*** jpena is now known as jpena|off | 17:51 | |
*** jcapitao has quit IRC | 17:53 | |
*** hamalq has quit IRC | 18:01 | |
*** hamalq has joined #zuul | 18:02 | |
*** tosky has joined #zuul | 18:10 | |
*** gouthamr_ has quit IRC | 18:30 | |
*** sassyn has joined #zuul | 18:31 | |
sassyn | Hi again | 18:31 |
sassyn | Still issues with my setup :-( | 18:31 |
sassyn | I'm getting OSError: [Errno 39] Directory not empty: 'logs' | 18:32 |
sassyn | https://pastebin.pl/view/d4ddef35 | 18:32 |
*** yoctozepto has quit IRC | 18:37 | |
*** yoctozepto has joined #zuul | 18:38 | |
*** gouthamr_ has joined #zuul | 18:46 | |
sassyn | anyone? | 19:10 |
clarkb | sorry we're distracted by a giant gerrit upgrade today and the weekend | 19:11 |
*** tosky has quit IRC | 19:13 | |
clarkb | sassyn: that is really curious, I don't know why rmtree isfailing to rm things | 19:16 |
clarkb | sassyn: on our hosts the paths to check are /var/lib/zuul/builds/$uuid | 19:17 |
clarkb | sassyn: if the zuul executor isn't running you can try manually rm'ing those $uuid dirs and see if that fixes it | 19:17 |
clarkb | or maybe if that fails you'll get a better idea for why | 19:17 |
tristanC | sassyn: are you using 3.19.1 ? | 19:22 |
tristanC | if i recall correct, a couple of fix for build log cleanup are present in master, but not released yet | 19:23 |
sassyn | clarkb thank u , when i shutdown and the executor and switching to the zuul user and doing the rm no error, files deteled | 19:26 |
sassyn | tristanC I'm using version .3.19.0 | 19:26 |
tristanC | sassyn: then the fix for your issue might be: https://opendev.org/zuul/zuul/commit/d82ff8a755edfd2bde86f9eb21c95d1a3e36712c | 19:34 |
sassyn | tristanC thank u | 19:35 |
sassyn | I think it is NFS issue | 19:35 |
sassyn | since the /var/lib/zuul/builds/$uuid is located on a NetApp storage | 19:36 |
sassyn | https://stackoverflow.com/questions/58943374/shutil-rmtree-error-when-trying-to-remove-nfs-mounted-directory | 19:37 |
sassyn | see this | 19:37 |
clarkb | I think you may be able to disable those cleanups. But then when you restart zuul-executor you can leak those build dirs | 19:40 |
clarkb | another option is to clean them up prior to starting zuul-executor with an init script or similar | 19:40 |
clarkb | and maybe we should consider an update that does manual recursion to avoid issues? | 19:40 |
tristanC | clarkb: the clean on start was already fixed for that issue with https://opendev.org/zuul/zuul/commit/98238ede6418ae4d3b3cdc26b7c8ef0214303f84 | 19:41 |
clarkb | ah | 19:41 |
sassyn | this issue is in line shutil.rmtree | 19:51 |
*** nils has quit IRC | 20:29 | |
fungi | is it the classic race between recursively deleting parents before their children? | 20:31 |
fungi | shutil.rmtree has suffered that for as long as i can recall | 20:31 |
*** iurygregory has quit IRC | 20:42 | |
corvus | fungi: i think it's due to having an open file handle to something in the directory | 20:44 |
fungi | oh, yep | 20:45 |
corvus | i don't know what in zuul would have a file open; i think it warrants some debugging | 20:45 |
fungi | that'll quickly derail recursive deletion | 20:45 |
corvus | (under nfs; not otherwise) | 20:45 |
corvus | (because the nfs client keeps a .nfs file around in order to preserve the file on the nfs server) | 20:46 |
fungi | yup | 20:47 |
corvus | i'd probably debug that by catching that exception then running an lsof or something | 20:47 |
corvus | oh; quick guess: ssh control persist socket? | 20:47 |
corvus | that sticks around for 60 seconds after the job ends i think | 20:48 |
*** iurygregory has joined #zuul | 20:59 | |
*** rlandy|rover has quit IRC | 21:14 | |
*** sassyn has quit IRC | 23:09 | |
*** iurygregory has quit IRC | 23:22 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!