Friday, 2020-11-20

*** tosky has quit IRC00:18
*** zenkuro has quit IRC00:23
*** zenkuro has joined #zuul00:23
*** wuchunyang has joined #zuul00:24
*** wuchunyang has quit IRC00:28
pabelangercorvus: \o/00:57
pabelangerthank you again00:57
*** sean-k-mooney1 has joined #zuul01:01
*** sean-k-mooney has quit IRC01:03
*** zenkuro has quit IRC01:53
*** sean-k-mooney2 has joined #zuul02:15
*** sean-k-mooney1 has quit IRC02:17
*** jhesketh has quit IRC02:27
*** jhesketh has joined #zuul02:33
*** bhavikdbavishi has joined #zuul02:49
openstackgerritIan Wienand proposed zuul/zuul master: web: Add optional link prop to title-with-icon and results  https://review.opendev.org/76347102:58
*** bhavikdbavishi1 has joined #zuul03:06
*** bhavikdbavishi has quit IRC03:07
*** bhavikdbavishi1 is now known as bhavikdbavishi03:07
*** rlandy|rover has quit IRC03:42
*** iurygregory has quit IRC03:42
*** hamalq has quit IRC03:56
*** wuchunyang has joined #zuul04:09
*** wuchunyang has quit IRC04:14
*** bhavikdbavishi has quit IRC04:23
*** bhavikdbavishi has joined #zuul04:23
*** evrardjp has joined #zuul05:33
*** bhavikdbavishi has quit IRC05:55
*** bhavikdbavishi1 has joined #zuul05:55
*** bhavikdbavishi has joined #zuul05:58
*** bhavikdbavishi1 has quit IRC05:59
*** rpittau|afk is now known as rpittau06:52
*** bhavikdbavishi has quit IRC06:55
*** bhavikdbavishi has joined #zuul07:15
felixedelianw: I like that one. Maybe you could directly include your follow-up commits into my original change and remove the parts that are not needed anymore? So we don't end up with unnecessary CSS classes and other stuff in the first place.07:19
*** iurygregory has joined #zuul07:39
ianwfelixedel: ok, i don't quite have time right now, but feel free if you like to squash them :)  otherwise i'll do it later07:57
*** jcapitao has joined #zuul08:14
openstackgerritMatt Kucia proposed zuul/zuul master: tox: Show line length style errors on linters  https://review.opendev.org/75862208:23
openstackgerritMatt Kucia proposed zuul/zuul master: configloader: Log more error information  https://review.opendev.org/75866008:23
openstackgerritMatt Kucia proposed zuul/zuul master: drivers: Bitbucket skeleton  https://review.opendev.org/75753208:23
openstackgerritMatt Kucia proposed zuul/zuul master: doc: Bitbucket driver  https://review.opendev.org/75866108:23
openstackgerritMatt Kucia proposed zuul/zuul master: examples: Sample configuration for Bitbucket driver  https://review.opendev.org/75862308:23
openstackgerritMatt Kucia proposed zuul/zuul master: drivers: Bitbucket - Source  https://review.opendev.org/75866208:23
*** bhavikdbavishi has quit IRC08:36
*** saneax has joined #zuul08:36
*** jpena|off is now known as jpena08:37
openstackgerritTobias Henkel proposed zuul/zuul master: Handle the yaml merge operator  https://review.opendev.org/76353209:15
*** hashar has joined #zuul09:26
zbrianw: are you around? related to the artifact details/expansion....09:30
openstackgerritzbr proposed zuul/zuul master: Consolidate js/jsx indentation  https://review.opendev.org/74737709:40
*** nils has joined #zuul09:41
*** tosky has joined #zuul09:57
avassdoes zuul work well with gerrit replicas?10:08
avassI would guess opendev is using that10:08
*** openstackgerrit has quit IRC10:25
*** saneax has quit IRC10:51
*** zenkuro has joined #zuul11:13
zbravass: yes11:19
zbravass: felixedel ianw tobiash: can you please help me deliver the ^ js/jsx indentation fix?11:20
*** bhavikdbavishi has joined #zuul11:25
*** bhavikdbavishi1 has joined #zuul11:29
*** bhavikdbavishi has quit IRC11:31
*** bhavikdbavishi1 is now known as bhavikdbavishi11:31
*** sean-k-mooney2 is now known as sean-k-mooney11:50
*** jcapitao is now known as jcapitao_lunch11:57
tobiashzuul-maint: current nodepool master spams our logs due to multi ssh key handling: http://paste.openstack.org/show/800251/12:01
*** jfoufas1 has joined #zuul12:13
*** zenkuro has quit IRC12:22
*** zenkuro has joined #zuul12:23
*** sassyn has joined #zuul12:26
sassynhi everyone. I have trouble this morning to make zuul working again.12:26
sassynI have a power issue at my Data Center. Zuul server were booted... all was working fine for few days now12:27
sassynno change in the syste,12:27
*** jpena is now known as jpena|lunch12:27
sassynsystem*12:27
*** hashar has quit IRC12:27
sassynhowever today all jobs get the  Waiting on logger12:27
sassynand get into a RETRY_LIMIT12:28
sassynI can see using TCP dump that the Executor trying to open a connection to Zuul Streaming Console port @ 19885 TCP12:29
sassynlike this12:29
sassyn14:29:33.573735 IP Executor01.57276 > 172.17.0.2.19885: Flags [S], seq 2834551981, win 65495, options [mss 65495,sackOK,TS val 616387407 ecr 0,nop,wscale 7], length 012:30
sassynack 1, win 0, length 012:30
sassynthere is no firewall and the Executor01 can access the 172.17.0.2 machine and all ports12:30
sassynin the worker I can't see anything that is bind to the port 1988512:30
sassynwhat do I miss?12:30
avasssassyn: the executor needs access to that port for live log streaming12:32
sassynthe executor  do have access to the worker nodes of the nodepool12:33
sassynin this port12:33
avassand this role starts that daemon on the node: https://zuul-ci.org/docs/zuul-jobs/general-roles.html?highlight=stream#role-start-zuul-console12:33
avassso if it's never started it won't be able to connect to it12:33
sassynstart-zuul-console12:34
sassynI don't have this role setup12:34
avassthat's probably why you're getting 'Waiting on logger'12:34
sassyni will add this to my pre jobs in the trusted zuul repo12:34
sassynbut How it was working before?12:35
avassit shouldn't have unless that had been started :)12:35
avassbut zuul uploads the logs after the jobs finishes as well12:35
sassynok12:36
sassynso it should come after the prepare-workspace-git?12:36
avassas long as you do that in a post/cleanup run12:36
sassyni have log-inventory, add-build-sshkey and prepare-workspace-git in my pre roles12:36
avasswe do it after add-build-host-key, so our second role12:37
avassthe earlier the better I'd say12:37
tobiashsassyn: typically the waiting for logger is non-fatal in most cases, so you might have a different issue as well12:45
sassynI add the tole12:46
sassynrole* seems to be working fine12:46
sassynnow thanks!12:46
*** rlandy has joined #zuul12:53
*** rlandy is now known as rlandy|rover12:53
*** hashar has joined #zuul12:53
fungiavass: zbr: opendev doesn't connect its zuul to a git replica nor clustered gerrit, so regardless of what you mean by gerrit replica i think the answer is no, opendev's zuul connects to their primary (and only) gerrit server to receive events and pull change refs, mainly because we don't want it to race replication delays12:55
fungitobiash: should we catch paramiko.ssh_exception.SSHException i guess? and log something at info level (or not at all)?12:56
tobiashfungi: yes I think so12:57
*** bhavikdbavishi has quit IRC13:01
zbrfungi: i guess i confused it with gitea, which was the distributed one.13:02
fungiyeah, our zuul doesn't interact with our gitea cluster at all13:03
zbrfungi: can you please +W https://review.opendev.org/#/c/747377/ before it needs another rebase?13:03
*** zenkuro has quit IRC13:03
*** zenkuro has joined #zuul13:03
-openstackstatus- NOTICE: The Gerrit service at review.opendev.org will be offline starting at 15:00 UTC (roughly two hours from now) for a weekend upgrade maintenance: http://lists.opendev.org/pipermail/service-announce/2020-October/000012.html13:04
*** jcapitao_lunch is now known as jcapitao13:04
tobiashremote: https://review.opendev.org/763553 Don't spam logs due to non-existing hostkeys13:09
tobiashlooks like the gerrit bot is already offline ;)13:09
fungitobiash: oh, i also just pushed https://review.opendev.org/763555 Silence paramiko exceptions for incompatible keys13:18
fungiand no, the openstackgerrit irc bot has probably just fallen in a netsplit again13:19
tobiashfungi: hrm, that if actually should have catched this already13:20
fungitobiash: it didn't have an else condition though13:21
fungiso it fell through to raising anyway i think?13:21
tobiashfungi: but the message would not match the if13:21
tobiashbut the exception clause doesn't match13:22
tobiashparamiko is throwing paramiko.ssh_exception.SSHException and we catch paramiko.SSHException13:22
fungiahh, yeah i wonder if paramiko changed where it keeps its exceptions13:23
tobiashfungi: oh wait, I think that log comes from some thread within paramiko itself13:23
tobiashsince the stack trace goes straight to a run method and no zuul path is in there13:24
fungiit does lack nodepool source context in the traceback13:24
fungimaybe nodepool can't catch these directly?13:24
tobiashlooks like that13:24
tobiashI guess we need to look into paramiko to see if there's a way to inhibit that13:25
tobiashand if not I'll maybe exclude paramiko from our log config :D13:25
fungiwe don't see those in our launcher debug logs at least13:27
tobiashdo run them already with https://review.opendev.org/761229 included?13:27
fungiyes, we added it so that nodes could be rebooted into "fips mode" during job run time, which will change what host keys ssh can use13:29
tobiashso your log config already excludes paramiko?13:29
fungii suppose so. checking now13:29
tobiashconfirmed in paramiko, it explicitly logs the exception13:31
tobiashso the only way is to explicitly exclude paramiko.transport from the log config13:31
*** jpena|lunch is now known as jpena13:32
fungitobiash: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/nodepool-launcher/files/logging.conf13:32
fungithat's ours13:33
tobiashso in your case it won't be in the debug log, but in the console log only13:33
*** zenkuro has quit IRC13:40
*** zenkuro has joined #zuul13:41
-openstackstatus- NOTICE: The Gerrit service at review.opendev.org will be offline starting at 15:00 UTC (roughly one hour from now) for a weekend upgrade maintenance: http://lists.opendev.org/pipermail/service-announce/2020-October/000012.html14:02
avassfungi: thanks, good to know14:16
avassfungi: we've been growing a lot lately so we've got a lot of load on our gerrit and we're exploring solutions14:17
corvusfungi, zbr, tristanC: the js indentation patch is incompatible with the default style in emacs, which is used by at least tristan and myself14:19
corvustobiash: ^14:19
tobiashcorvus: oh, can we abort it still?14:20
corvusfungi, zbr, tobiash, tristanC: honestly, i would prefer that we *not* enforce indentation in the linter because i think it's a waste of time for us to even talk about it.  it was all readable before -- we worked on that code for years with no problems14:20
corvustobiash: https://review.opendev.org/747377 merged14:20
corvusi did not look at the change because i have a policy of not reviewing whitespace changes14:20
corvusi only just now noticed that it also *enforces* whitespace in the linter14:21
corvusi propose we choose one of the following options: a) just back out the linting enforcement  (i still don't care about the whitespace, so i'm fine with all the 'consolidation' changes).  if that isn't acceptable, then b) someone should figure out the eslintrc settings to make it compatible with the editors we use (or figure out the editor settings to make them compatible with the linter).  there's like a14:24
corvuswhole page of eslint settings for whitespace https://eslint.org/docs/rules/indent ; considering my feelings on whitespace, i don't personally plan on doing (b)14:24
tobiashI think I'd probably prefer first a) then b) (eslint settings) as I'd prefer to have some validation (we have pep8 for python as well) so we have a somewhat consistent indentation (in the past there were some changes that introduced inconsistent indentation that I noticed in reviews which lead to that stack in the end)14:28
zbri am back14:28
zbrthat identation is standard, is not imposed by me. if a particular editor does not work by default, is a bug fix for it, imho14:29
corvustobiash: i hold differing opinions on python/js.  indentation is part of the code in python, so it's important to the actual execution for the code.  in js, it's purely stylistic.  i continue to hold to the belief that it's not worth spending our time arguing about stylistic aspects like indentation.14:30
zbrthe whole point was to avoid random indentation, zuul used to be quite consistent but during the last year the js(x) become....very relaxed.14:30
corvuszbr: if zuul js was consistent at all, it was to the emacs default which you say is non-standard14:31
corvus(because most of the original authors of the js use emacs)14:31
zbri guess we could add a helper config for emacs?14:31
corvustbh, i kind of like the emacs style, and i disagree that it's non-standard.  even the eslint page lists different styles used by different projects/communities14:32
corvusyou could say the zuul community had a standard (which i suspect can be expressed with an eslint config)14:32
corvusyou just changed it14:32
corvusand i *still* don't care14:32
corvusi don't mind if it changes14:33
corvusthe only thing i care about is not spending any more time talking about this14:33
corvusthis is a decision we made very early on in the zuul project14:33
zbrif the project wants it own standard, it is possible to configure the eslint to match custom project preferences.14:33
corvusto not nitpick about style14:33
zbrnope14:33
zbri am ok as long we pick one style, not if we avoid picking one14:34
corvuszbr: we're going to have to disagree on that one.  i'm not going to spend any time trying to convince you.14:34
corvusremote:   https://review.opendev.org/763568 Don't enforce js whitespace14:35
corvustobiash, fungi: ^ that's (a)14:35
corvusi still will agree with (b) if someone wants to research and implement that and a sufficient number of maintainers do as well, though i would prefer not to spend any more time on it.14:36
corvusall of the zuul maintainers are a bit strapped for time right now and i think we should spend it wisely14:37
corvustobiash: do we need to change zuul's default logging config for paramiko?14:39
tobiashcorvus: maybe, we have a custom one as well14:39
tobiashat least I saw that paramiko wants to have a configured logger14:40
tobiashotherwise it seems to configure its own handler14:40
tobiashin our deployment we use stdout as main log output so that's bad and I had to put paramiko explicitly on a NullHandler14:41
zbrcorvus: tobiash: what about the first change on the summary page? i managed to get the copy uuid working as described on the ml14:44
avassI don't really care much about style as long as the code is readable.14:47
tobiashremote: https://review.opendev.org/763573 Prevent paramiki.transport log spam14:47
tobiashcorvus: I did something similar to this ^14:47
corvustobiash: yeah, that's what i was thinking we might need based on what i've read14:47
tristanCcorvus: i agree with not changing indentation, i think we should use eslint more to detect code defect, and less for code style14:48
corvustristanC: oof, it's error level?  ouch, that seems potentially problematic14:49
corvustobiash: ^14:49
corvustristanC: sorry tabfail14:49
tobiashcorvus: yes, that's outputting those exceptions with error level14:49
tobiashand there seems to be no way other than this or avoid the exceptions in the first place14:50
clarkbits an interesting choice on their part to log an error then raise an exception too in a library14:50
clarkbseems like they should just raise the exception. Maybe w can convince them to change that upstream?14:50
corvustobiash: do you think it's worth making a custom handler that filters it or something?14:50
tobiashcorvus: it might not be as problematic since nodepool itself logs any exception with the full trace14:50
corvustobiash: ok, so if we really encounter an error, we'll get it in our thread and log it?14:51
clarkbcorvus: yes we should, because they raise too14:51
tobiashcorvus: yes, since nodepool logs any errors by itself in the nodepool.* packages14:51
corvusok, then i think 763573 is probably ok14:51
tobiashthat's only about the stuff paramiko logs by itself14:51
corvuslittly scary but ok :)14:51
*** Goneri has joined #zuul14:51
*** rpittau is now known as rpittau|afk14:53
fungii'm fine backing out the linter enforcement, i find it pointless personally, i'm just tired of people pestering me to review changes like that but i should probably adopt a policy like yours and just tell them sorry i'm not reviewing that (or -1 it because i find indentation checking for languages without semantic whitespace unnecessary)14:55
fungihappy to revert the indentation change next week after gerrit maintenance but i need to focus on that now. sorry i approved it :/14:56
corvuszbr: left review; i think you should solicit more comments on the points ianw raises14:58
zbrif i remember well python has advanced ability to filter logging messages (so we could eliminate the noisy one)14:58
-openstackstatus- NOTICE: The Gerrit service at review.opendev.org is offline for a weekend upgrade maintenance, updates will be provided once it's available again: http://lists.opendev.org/pipermail/service-announce/2020-October/000012.html15:05
*** rlandy|rover is now known as rlandy|rover|brb15:31
zbrcorvus: thanks, for both.15:39
*** jfoufas1 has quit IRC15:48
*** rlandy|rover|brb is now known as rlandy|rover15:53
*** ttx has quit IRC16:39
*** ttx has joined #zuul16:45
*** tosky has quit IRC16:56
*** sassyn has quit IRC17:18
*** hashar has quit IRC17:46
*** slaweq has quit IRC17:49
*** hamalq has joined #zuul17:51
*** jpena is now known as jpena|off17:51
*** jcapitao has quit IRC17:53
*** hamalq has quit IRC18:01
*** hamalq has joined #zuul18:02
*** tosky has joined #zuul18:10
*** gouthamr_ has quit IRC18:30
*** sassyn has joined #zuul18:31
sassynHi again18:31
sassynStill issues with my setup :-(18:31
sassynI'm getting OSError: [Errno 39] Directory not empty: 'logs'18:32
sassynhttps://pastebin.pl/view/d4ddef3518:32
*** yoctozepto has quit IRC18:37
*** yoctozepto has joined #zuul18:38
*** gouthamr_ has joined #zuul18:46
sassynanyone?19:10
clarkbsorry we're distracted by a giant gerrit upgrade today and the weekend19:11
*** tosky has quit IRC19:13
clarkbsassyn: that is really curious, I don't know why rmtree isfailing to rm things19:16
clarkbsassyn: on our hosts the paths to check are /var/lib/zuul/builds/$uuid19:17
clarkbsassyn: if the zuul executor isn't running you can try manually rm'ing those $uuid dirs and see if that fixes it19:17
clarkbor maybe if that fails you'll get a better idea for why19:17
tristanCsassyn: are you using 3.19.1 ?19:22
tristanCif i recall correct, a couple of fix for build log cleanup are present in master, but not released yet19:23
sassynclarkb thank u , when i shutdown and the executor  and switching to the zuul user and doing the rm no error, files deteled19:26
sassyntristanC I'm using version .3.19.019:26
tristanCsassyn: then the fix for your issue might be: https://opendev.org/zuul/zuul/commit/d82ff8a755edfd2bde86f9eb21c95d1a3e36712c19:34
sassyntristanC thank u19:35
sassynI think it is NFS issue19:35
sassynsince the /var/lib/zuul/builds/$uuid is located on a NetApp storage19:36
sassynhttps://stackoverflow.com/questions/58943374/shutil-rmtree-error-when-trying-to-remove-nfs-mounted-directory19:37
sassynsee this19:37
clarkbI think you may be able to disable those cleanups. But then when you restart zuul-executor you can leak those build dirs19:40
clarkbanother option is to clean them up prior to starting zuul-executor with an init script or similar19:40
clarkband maybe we should consider an update that does manual recursion to avoid issues?19:40
tristanCclarkb: the clean on start was already fixed for that issue with https://opendev.org/zuul/zuul/commit/98238ede6418ae4d3b3cdc26b7c8ef0214303f8419:41
clarkbah19:41
sassynthis issue is in line shutil.rmtree19:51
*** nils has quit IRC20:29
fungiis it the classic race between recursively deleting parents before their children?20:31
fungishutil.rmtree has suffered that for as long as i can recall20:31
*** iurygregory has quit IRC20:42
corvusfungi: i think it's due to having an open file handle to something in the directory20:44
fungioh, yep20:45
corvusi don't know what in zuul would have a file open; i think it warrants some debugging20:45
fungithat'll quickly derail recursive deletion20:45
corvus(under nfs; not otherwise)20:45
corvus(because the nfs client keeps a .nfs file around in order to preserve the file on the nfs server)20:46
fungiyup20:47
corvusi'd probably debug that by catching that exception then running an lsof or something20:47
corvusoh; quick guess: ssh control persist socket?20:47
corvusthat sticks around for 60 seconds after the job ends i think20:48
*** iurygregory has joined #zuul20:59
*** rlandy|rover has quit IRC21:14
*** sassyn has quit IRC23:09
*** iurygregory has quit IRC23:22

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!