Tuesday, 2019-05-21

*** jamesmcarthur has joined #zuul00:04
*** sshnaidm|afk has quit IRC00:04
*** sshnaidm has joined #zuul00:06
*** panda|rover has quit IRC00:10
*** panda has joined #zuul00:11
openstackgerritJimmy McArthur proposed zuul/zuul-website master: Revert "Revert promotional message banner and event list"  https://review.opendev.org/66023200:19
pabelangerSpamapS: thanks, testing that out too00:22
tristanCmordred: http://8a4226c6403443a892ae9a1e00578e69.openstack.zuul-preview.opendev.org/ seems to be stuck, so00:24
mordredtristanC: one sec00:25
tristanCif you go to http://logs.openstack.org/31/660231/2/check/zuul-build-dashboard-multi-tenant/7c90924/npm/html/  then click rdoproject's list and DLRN, you'll have a "Something went wrong." error00:25
tristanCand in the console.log, there should be a: TypeError: "n is undefined"  stacktrace with a link to the source file: ProjectVariant.jsx:3100:25
mordredtristanC: yes - I agree.00:26
mordredtristanC: this is what I would expect to happen on an error - and is much nicer to deal with :)00:26
tristanCmordred: (not sure if zuul-preview can work with regular log_url when the build has no artifact)00:27
mordredtristanC: so - maybe what we should do is make sure you're online next time we're going to deploy the react2 update00:27
mordredand deploy it - and then if it falls over we can capture the issue better00:27
tristanCalso, the yarn.lock update was pretty old, i'll submit a new generation that hopefully contains more fix than regression :-)00:27
clarkbhttp://paste.openstack.org/show/751488/ is what pabelanger got from the console log fwiw00:28
tristanCthat's kind of helpful, the "value at" line points at the actual source code file00:31
openstackgerritTristan Cacqueray proposed zuul/zuul master: Revert "Revert "web: upgrade react and react-scripts to ^2.0.0""  https://review.opendev.org/65999100:32
clarkbtristanC: mordred in that stacktrace I see things about config errors00:32
clarkbiirc we did have config errors back then00:32
clarkbmaybe related to that? we could induce a config error (maybe?) and check if the code works?00:33
tristanCclarkb: indeed, the stack trace also refer to App.jsx:152 which is where the config errors are fetched00:34
*** jamesmcarthur has quit IRC00:35
clarkbbtw don't try to open the yarn.lock diff in gerrit web ui00:35
clarkbit makes gerrit stop working in your browser for a bit00:35
clarkbI've got to run now, but that seems to be a thread we can pull on00:37
fungigertty (on a 1gb vm with a bunch of other stuff also running) pauses for several seconds when you request that diff00:37
funginot an understatement either. it was maybe 3 seconds after hitting the d key before i could start scrolling full-speed through it00:38
*** mattw4 has quit IRC00:46
tristanChmm, i can't reproduce with both PS on a setup with config-errors...00:50
*** jamesmcarthur has joined #zuul01:05
*** jamesmcarthur_ has joined #zuul01:10
*** jamesmcarthur has quit IRC01:11
openstackgerritTristan Cacqueray proposed zuul/zuul master: Revert "Revert "web: upgrade react and react-scripts to ^2.0.0""  https://review.opendev.org/65999101:53
tristanCnew PS adds a react test to ensure config errors are rendered properly (it even simulates a mouse click event :)01:54
openstackgerritTristan Cacqueray proposed zuul/zuul master: Revert "Revert "web: upgrade react and react-scripts to ^2.0.0""  https://review.opendev.org/65999101:58
openstackgerritTristan Cacqueray proposed zuul/zuul master: DNM: this should cause a React exception  https://review.opendev.org/66023102:51
*** saneax has quit IRC03:09
*** jamesmcarthur_ has quit IRC03:23
*** bhavikdbavishi has joined #zuul03:42
*** yolanda_ has quit IRC03:49
*** jamesmcarthur has joined #zuul03:54
*** jamesmcarthur has quit IRC03:59
*** jamesmcarthur has joined #zuul05:01
*** pcaruana has joined #zuul05:02
*** raukadah is now known as chandankumar05:06
*** jamesmcarthur has quit IRC05:07
*** pcaruana has quit IRC05:11
*** bhavikdbavishi1 has joined #zuul05:28
*** bhavikdbavishi has quit IRC05:28
*** bhavikdbavishi1 is now known as bhavikdbavishi05:28
openstackgerritTobias Henkel proposed zuul/zuul master: WIP: Fix broken setRefs whith missing objects  https://review.opendev.org/62166705:43
openstackgerritTobias Henkel proposed zuul/zuul master: Match tag items against containing branches  https://review.opendev.org/57855705:46
openstackgerritTobias Henkel proposed zuul/zuul master: Use implied branch matcher for implied branches  https://review.opendev.org/64027205:47
*** saneax has joined #zuul05:51
openstackgerritTobias Henkel proposed zuul/zuul master: Update cached repo during job startup only if needed  https://review.opendev.org/64822905:53
openstackgerritTobias Henkel proposed zuul/zuul master: Add support for smart reconfigurations  https://review.opendev.org/65211405:57
openstackgerritTobias Henkel proposed zuul/zuul master: Add --check-config option to zuul scheduler  https://review.opendev.org/54216005:57
*** zbr_ has quit IRC06:05
*** bhavikdbavishi has quit IRC06:10
*** bhavikdbavishi has joined #zuul06:12
openstackgerritTobias Henkel proposed zuul/zuul master: Match tag items against containing branches  https://review.opendev.org/57855706:20
openstackgerritTobias Henkel proposed zuul/zuul master: Use implied branch matcher for implied branches  https://review.opendev.org/64027206:20
openstackgerritTobias Henkel proposed zuul/zuul master: Update cached repo during job startup only if needed  https://review.opendev.org/64822906:26
*** jamesmcarthur has joined #zuul06:35
*** jamesmcarthur has quit IRC06:39
*** gtema has joined #zuul07:00
openstackgerritTristan Cacqueray proposed zuul/zuul master: config: add tenant.toDict() method and REST endpoint  https://review.opendev.org/62134407:18
*** pcaruana has joined #zuul07:21
*** jamesmcarthur has joined #zuul07:25
*** jamesmcarthur has quit IRC07:30
*** pcaruana has quit IRC07:39
*** jangutter has joined #zuul07:44
*** bhavikdbavishi has quit IRC07:51
*** themroc has joined #zuul07:52
*** jpena|off is now known as jpena07:56
*** hashar has joined #zuul08:07
*** bhavikdbavishi has joined #zuul08:56
openstackgerritMatthieu Huin proposed zuul/zuul master: web: add tenant and project scoped, JWT-protected actions  https://review.opendev.org/57690709:02
openstackgerritMatthieu Huin proposed zuul/zuul master: Allow operator to generate auth tokens through the CLI  https://review.opendev.org/63619709:02
openstackgerritMatthieu Huin proposed zuul/zuul master: Zuul CLI: allow access via REST  https://review.opendev.org/63631509:02
openstackgerritMatthieu Huin proposed zuul/zuul master: Add Authorization Rules configuration  https://review.opendev.org/63985509:02
openstackgerritMatthieu Huin proposed zuul/zuul master: Web: plug the authorization engine  https://review.opendev.org/64088409:02
openstackgerritMatthieu Huin proposed zuul/zuul master: Zuul Web: add /api/user/authorizations endpoint  https://review.opendev.org/64109909:03
openstackgerritMatthieu Huin proposed zuul/zuul master: authentication config: add optional token_expiry  https://review.opendev.org/64240809:03
*** openstackstatus has quit IRC09:13
*** openstackstatus has joined #zuul09:15
*** ChanServ sets mode: +v openstackstatus09:15
*** jamesmcarthur has joined #zuul09:26
*** electrofelix has joined #zuul09:30
*** jamesmcarthur has quit IRC09:31
*** gtema has quit IRC11:03
*** gtema has joined #zuul11:03
*** snapiri has quit IRC11:03
*** panda is now known as panda|rover|eat11:13
*** markwork has joined #zuul11:15
markworkWoohoo, our demo went well, and we've got some feedback from the wider organization :)11:15
*** jamesmcarthur has joined #zuul11:27
*** jamesmcarthur has quit IRC11:33
*** jamesmcarthur has joined #zuul11:43
*** jamesmcarthur has quit IRC11:55
*** jamesmcarthur has joined #zuul12:00
mordredmarkwork: congrats, that's exciting!12:00
*** jpena is now known as jpena|lunch12:00
*** themroc has quit IRC12:15
*** jamesmcarthur has quit IRC12:16
*** jamesmcarthur has joined #zuul12:16
*** bhavikdbavishi has quit IRC12:23
*** panda|rover|eat is now known as panda|rover12:27
*** themroc has joined #zuul12:30
*** jamesmcarthur has quit IRC12:32
*** rlandy has joined #zuul12:37
*** flepied has joined #zuul12:38
*** jamesmcarthur has joined #zuul12:42
*** jpena|lunch is now known as jpena12:52
*** saneax has quit IRC12:53
fungimarkwork: we're happy to hear any feedback you like to pass along too13:26
*** jamesmcarthur has quit IRC13:39
*** themroc has quit IRC13:42
*** bhavikdbavishi has joined #zuul13:45
*** jamesmcarthur has joined #zuul13:48
*** themroc has joined #zuul13:48
*** rf0lc0 has joined #zuul13:57
*** jangutter_ has joined #zuul13:57
*** jamesmcarthur_ has joined #zuul13:59
*** tobiash_ has joined #zuul14:01
*** jangutter has quit IRC14:06
*** SpamapS has quit IRC14:06
*** rfolco has quit IRC14:06
*** jamesmcarthur has quit IRC14:06
*** electrofelix has quit IRC14:06
*** sshnaidm has quit IRC14:06
*** toabctl has quit IRC14:06
*** swest has quit IRC14:06
*** aspiers has quit IRC14:06
*** smcginnis has quit IRC14:06
*** nickx-intel has quit IRC14:06
*** edmondsw_ has quit IRC14:06
*** tobiash has quit IRC14:06
*** electrofelix has joined #zuul14:07
*** rf0lc0 is now known as rfolco14:12
*** mattw4 has joined #zuul14:12
*** sshnaidm has joined #zuul14:12
*** SpamapS has joined #zuul14:13
*** swest has joined #zuul14:14
*** swest has quit IRC14:15
*** aspiers has joined #zuul14:15
*** themr0c has joined #zuul14:27
*** bhavikdbavishi has quit IRC14:28
*** hashar is now known as hasharAway14:29
*** themroc has quit IRC14:30
*** mattw4 has quit IRC14:35
*** hasharAway is now known as hashar14:41
*** chandankumar is now known as raukadah14:46
*** markwork has quit IRC14:55
*** markwork has joined #zuul15:02
markworkHey ho, one of our Python core dev guys gave my code in my changeset a cosmetics makeover, can I just push that as an additional commit?15:02
markworkLike, sorting hashes into alphabetical order, re-ordering imports, that kind of stuff.15:03
fungiif your change hasn't merged yet, you can just push a revision of it with those improvements included15:09
fungigerrit will show reviewers the difference between the previous patch and the new one15:09
fungigit commit --amend15:09
clarkbbut you can also have a series of changes in gerrit with dependency chains if it makes more sense to split things out logically or if you'd rather not recet reviews on existing changes15:13
*** themr0c has quit IRC15:13
*** gtema has quit IRC15:22
clarkbs/recet/reset/15:23
clarkbas a heads up skopeo appears to have rebuilt ppa packages with tristanC's fix so fungi is goign to remove our pin of that package. Please say something if you see oddities from the zuul docker image related jobs15:28
*** hashar has quit IRC15:47
*** smcginnis has joined #zuul15:57
*** panda|rover has quit IRC16:09
*** panda has joined #zuul16:10
openstackgerritDavid Shrewsbury proposed zuul/nodepool master: Use py3 pathlib in DibImageFile  https://review.opendev.org/66019116:17
openstackgerritDavid Shrewsbury proposed zuul/nodepool master: Use py3 pathlib in DibImageFile  https://review.opendev.org/66019116:19
openstackgerritDavid Shrewsbury proposed zuul/nodepool master: Use py3 pathlib in DibImageFile  https://review.opendev.org/66019116:21
*** mgoddard has quit IRC16:25
*** mgoddard has joined #zuul16:28
*** mattw4 has joined #zuul16:36
*** mgoddard has quit IRC16:42
*** mgoddard has joined #zuul16:43
*** mattw4 has quit IRC17:07
*** markwork has quit IRC17:10
*** hashar has joined #zuul17:12
*** jangutter_ has quit IRC17:14
mordredShrews: if you get bored ... the admin experience when setting an autohold on a multi-node job isn't stellar17:20
mordredShrews: (although it's possible there's no good way to fix it) ...17:20
clarkbmordred: how it multinode different from single node?17:20
*** mattw4 has joined #zuul17:20
mordredShrews: main issues are - when you go to nodepool list, you just get nodes with no indication of which node it might be17:21
mordredfor instance, when doing a hold on system-config-run-gitea - one of the nodes is gitea01.opendev.org and the others are not - so to debug thigs on that node, I need to trial-and-error to find the right one17:21
clarkbah yup you have to cross check against the job inventory17:22
mordredthen - when I'm done debugging, I need to nodepool delete each of them - there is no way to say "thanks, I'm done with all of the held nodes for this job"17:22
Shrewsmordred: does —detail not show a comment field with the hold reason?17:22
Shrews I’m afk for lunch atm17:22
mordredShrews: it does - but the comment is the same for all nodes in a nodeset17:22
ShrewsI do not have ideas for improving that experience atm17:23
mordredShrews: so for the second, I can do a grep for all the things with that comment - but then I have to do 4 individual deletes17:23
mordredShrews: me neither17:23
*** jpena is now known as jpena|off17:24
mordredmostly just wanted to mention it to shove it into your brainhole17:24
clarkbmordred: for the first one I've had people take the inventory from the job that resulted in the hold then they can look at names/IPs to sort out what is what17:24
mordredclarkb: yeah - fair enough - and it's probably not a terrible idea to just look at the inventory instead of doing the nodepool list to find the node17:25
fungithat's been the way i've tackled it17:26
mordredhowever - that means I've got to leave shell land and go to browser land - so it's still not *ideal* - but is workable17:26
mordred(also, if I wasn't tying the autohold it to a specific change, I'd then have to figure out which change has the failing job - which I can do from the nodepool list --detail output since it mentions refs/changes/{change}17:28
fungimordred: many command-line tools are able to fetch and display yaml served over http ;)17:31
*** saneax has joined #zuul17:32
openstackgerritTobias Henkel proposed zuul/zuul master: Add example statsd_exporter mapping  https://review.opendev.org/66047217:33
mordredfungi: sure - but wow that would be a lot of typing- and I'd have to figure out the URL to fetch :)17:33
tobiash_mordred: ^17:34
mordredtobiash_: awesome, thanks!17:34
tobiash_I'll upload the nodepool mapping in a sec17:34
*** tobiash_ is now known as tobiash17:34
*** electrofelix has quit IRC17:36
fungimordred: ...zuul builds api?17:36
fungibut yes, i agree a browser is probably easuer17:36
fungier, easier too17:36
openstackgerritTobias Henkel proposed zuul/nodepool master: Add statsd_exporter mapping  https://review.opendev.org/66047317:37
*** saneax has quit IRC17:41
*** jamesmcarthur_ has quit IRC17:42
*** saneax has joined #zuul17:42
mordredfungi: yeah - mostly - my workflow for this is "ssh to zuul01.openstack.org ; set autohold ; wait ; check to see if autohold on zuul goes away ; logout ; ssh to nl01.openstack.org ; nodepool list --detail | grep $comment ; try to ssh into nodes"17:45
mordredwhich otherwise doesn't involve api calls or browsers17:45
fungiinstead of waiting for the autohold to go away i just switch to checking nodepool17:49
fungibut otherwise yes17:49
mordredfungi: you're frequently smarter than me17:53
*** jamesmcarthur has joined #zuul17:58
funginah, checking zuul until the autohold goes away is probably smarter since querying nodepool is a slower operation17:58
fungii'm just lazy17:58
fungiand also tend to not start checking for held nodes immediately17:58
Shrewsmordred: perhaps what is needed is for zuul to begin storing autohold info in zk rather than memory, and all actions, including node release/delete be done through the zuul client.18:01
Shrews(as one possible bad/good/otherwise solution)18:02
*** jamesmcarthur has quit IRC18:02
Shrewszuul could then go crazy with updating the hold request with whatever info it has and a user may need18:04
fungialso that would allow it to maintain autohold states across scheduler restarts18:06
fungiright now you have to readd them on restart if you still want them18:06
fungiso seems like a generally good idea18:06
Shrewszuul image builds seem b0rked again?18:07
clarkbShrews: I just looked and we had successfully built a trusty image within about the last hour18:08
clarkbShrews: do you have more info?18:08
Shrewsdocker images18:08
ShrewsService 'node' failed to build: Get https://registry-1.docker.io/v2/rastasheep/ubuntu-sshd/manifests/latest: unauthorized: incorrect username or password18:08
clarkboh those image builds sorry. Thats the known issue18:08
clarkbchances are that job ran in limestone where we have ipv6 as primary ip connectivity and docker hub has ipv4 only so we have to go over shared NAT with the buildset registry and that is flaky18:09
Shrewsclarkb: so many images  :)18:09
clarkblogan-: ^ it might be good to figure out why that is flaky if you have time18:09
clarkblogan-: maybe the table is too small or retention too long18:09
Shrewsit was limestone18:09
clarkbShrews: ianw has been working to spin up new opendev mirror nodes with tls which will allow those jobs to talk to our tls'd docker hub proxy with a dedicated ipv4 floating IP instead of through the flaky docker registry18:10
logan-i was just seeing some weirdness with git.openstack.org https: http://paste.openstack.org/raw/751897/18:10
clarkbs/flake docker registry/flaky buildset registry proxy/18:10
clarkbfungi: ^ your bad host is hitting files.o.o fwiw18:12
clarkbbut load average there is still low ~1 and that is over ipv4 not 618:12
logan-guess thats unrelated to the docker.io issue there18:12
*** jamesmcarthur has joined #zuul18:12
clarkblogan-: the docker issue in the job is from limestone test node to docker hub via a registry proxy. This fails so then it tries to access it directly which then fails because the creds are only valid in the registry proxy not docker hub itself18:13
clarkblogan-: I believe that the shared NAT for the test nodes may be to blame18:13
logan-i can add some monitoring for registry-1.docker.io externally from the nodepool cloud to see if it is an issue of the nodepool cloud failing to hit that endpoint, or our network overall failing to hit that18:13
logan-yeah NAT port contention might be the case18:13
clarkbit is also possible there is an ipv4 routing issue between those cloud nodes (though i would expect to see other failures likefor multinode testing if that was the case)18:14
logan-but it also says "unauthorized", so it did connect and get a response, right?18:14
*** panda has quit IRC18:14
clarkblogan-: yes. This is due to a quirk of how docker works. If you tell docker that a mirror is a mirror of dockerhub then it will always try to talk to dockerhub directly if the mirror fails18:15
logan-ah ok18:15
clarkblogan-: so in this case it is first talking to the in cloud docker registry proxy to dockerhub. That fails (possibly due to nat) then it tries to connect directly and that succeeds18:15
*** panda has joined #zuul18:15
clarkbbut it then fails at layer 7 doing auth that is only valid on the in cloud registry proxy18:15
logan-where is the docker registry proxy located? on the mirror? or on a different nodepool node?18:16
fungiclarkb: hah, well at least it's not really a vulnerable system and can absorb the load. the problem with them hitting wiki.o.o was all the php scripts they were calling into blowing up cpu utilization18:16
clarkblogan-: it is located on a single use test node. In zuuls case a different test node in the same cloud region18:17
logan-got it18:17
clarkblogan-: this is why we think replacing that functionality with the in cloud mirror which has a fip will help18:17
clarkbbut before we do that we want tls on those mirror nodes18:17
clarkb(because we end up publishing the result of these image builds back to dockerhub)18:18
clarkbone thing that makes this extra frustrating is docker hub seems to be hosted by an aws elb18:18
clarkbwhich supports ipv6 termination18:18
logan-yeah NAT port contention at the gateway is one possibility, ML2 population issues causing node-to-node connectivity issues is another possibility. but iirc nodepool tests multi-node jobs for that type of working connectivity before running the job so the latter is less likely I think.18:18
clarkbif they just terminated an ipv6 addr on their load balancer we'd be golden18:19
*** jamesmcarthur_ has joined #zuul18:22
fungiipv6: it's not for containering18:22
*** jamesmcarthur has quit IRC18:24
*** mattw4 has quit IRC18:24
*** mattw4 has joined #zuul18:28
*** bhavikdbavishi has joined #zuul18:42
*** sshnaidm is now known as sshnaidm|afk18:56
*** bhavikdbavishi has quit IRC19:27
*** jamesmcarthur_ has quit IRC19:44
*** saneax has quit IRC19:51
*** logan- has quit IRC19:55
*** jamesmcarthur has joined #zuul20:02
*** rlandy is now known as rlandy|biab20:06
*** rlandy|biab is now known as rlandy20:39
SpamapSI will forgive random websites and even some web applications for not doing ipv6. But infrastructure like docker? >:|21:06
*** flepied has quit IRC21:07
fungisee also: ipv6 in kubernetes21:07
fungihttps://github.com/kubernetes/enhancements/issues/56321:10
fungi[spoiler: it's coming... real soon now]21:10
*** jamesmcarthur has quit IRC21:11
*** logan- has joined #zuul21:16
*** hashar has quit IRC21:20
*** flepied has joined #zuul21:26
*** flepied has quit IRC21:33
*** tjgresha has joined #zuul22:51
*** mattw4 has quit IRC23:11
*** mattw4 has joined #zuul23:12
*** rlandy has quit IRC23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!