Friday, 2021-01-22

*** jamesmcarthur has quit IRC00:01
*** jamesmcarthur has joined #zuul00:03
*** sshnaidm|ruck is now known as sshnaidm|afk00:08
*** tosky has quit IRC00:09
*** jamesmcarthur has quit IRC00:15
*** jamesmcarthur has joined #zuul00:15
*** jamesmcarthur has quit IRC00:22
*** jamesmcarthur has joined #zuul00:22
*** jamesmcarthur has quit IRC00:27
*** jamesmcarthur has joined #zuul00:28
*** jamesmcarthur has quit IRC00:30
*** jamesmcarthur has joined #zuul00:37
*** jamesmcarthur has quit IRC00:38
*** jamesmcarthur has joined #zuul00:38
*** jamesmcarthur has quit IRC00:38
*** jamesmcarthur has joined #zuul01:22
*** jamesmcarthur has quit IRC01:23
*** jamesmcarthur has joined #zuul01:23
*** jamesmcarthur has quit IRC01:34
*** jamesmcarthur has joined #zuul01:35
*** hamalq has quit IRC02:24
*** jamesmcarthur has quit IRC02:28
*** jamesmcarthur has joined #zuul02:30
*** jamesmcarthur has quit IRC02:35
*** tflink_ is now known as tflink02:40
*** jamesmcarthur has joined #zuul02:58
*** ianw is now known as ianw_pto03:12
*** bhavikdbavishi has joined #zuul03:22
*** bhavikdbavishi1 has joined #zuul03:25
*** bhavikdbavishi has quit IRC03:26
*** bhavikdbavishi1 is now known as bhavikdbavishi03:26
*** bhavikdbavishi has quit IRC03:38
*** zenkuro has quit IRC03:40
*** bhavikdbavishi has joined #zuul03:40
*** bhavikdbavishi1 has joined #zuul03:43
*** bhavikdbavishi has quit IRC03:45
*** bhavikdbavishi1 is now known as bhavikdbavishi03:45
*** bhavikdbavishi has quit IRC03:48
*** bhavikdbavishi has joined #zuul04:00
*** bhavikdbavishi1 has joined #zuul04:03
*** jamesmcarthur has quit IRC04:03
*** jamesmcarthur has joined #zuul04:03
*** bhavikdbavishi has quit IRC04:04
*** bhavikdbavishi1 is now known as bhavikdbavishi04:04
*** bhavikdbavishi has quit IRC04:41
*** vishalmanchanda has joined #zuul04:47
*** ykarel has joined #zuul04:59
*** evrardjp has quit IRC05:33
*** evrardjp has joined #zuul05:33
*** jamesmcarthur has quit IRC05:51
*** jamesmcarthur has joined #zuul05:52
*** jfoufas1 has joined #zuul05:53
*** jamesmcarthur has quit IRC05:57
*** jamesmcarthur has joined #zuul06:01
*** jamesmcarthur has quit IRC06:01
*** rpittau|afk is now known as rpittau07:34
*** jcapitao has joined #zuul07:37
*** hashar has joined #zuul07:41
*** jamesmcarthur has joined #zuul08:02
*** harrymichal has joined #zuul08:06
*** jamesmcarthur has quit IRC08:06
*** mgoddard has quit IRC08:11
*** mgoddard has joined #zuul08:11
openstackgerritMerged zuul/zuul-jobs master: Rename config repos file config for CentOS Stream  https://review.opendev.org/c/zuul/zuul-jobs/+/77081508:15
*** tosky has joined #zuul08:44
*** nils has joined #zuul08:50
*** jpena|off is now known as jpena08:57
*** jamesmcarthur has joined #zuul09:13
*** dcastellani has quit IRC09:14
*** iamweswilson has quit IRC09:14
*** PrinzElvis has quit IRC09:14
*** dcastellani has joined #zuul09:15
*** iamweswilson has joined #zuul09:15
*** PrinzElvis has joined #zuul09:15
*** masterpe has quit IRC09:17
*** jamesmcarthur has quit IRC09:20
*** jamesmcarthur has joined #zuul09:21
*** mordred has quit IRC09:21
*** Eighth_Doctor has quit IRC09:21
*** jamesmcarthur has quit IRC09:26
tobiashcorvus: re nodepool optimization: looking at the sdk code nodepool is already using the correct method within the sdk and I think I can increase the cache duration directly via clouds.yaml if needed09:30
tobiashcorvus: however I've seen that also our providers that basically do nothing have a constant cpu usage of >50% so I'll switch the target and see if there is any improvement there09:34
*** Eighth_Doctor has joined #zuul10:08
*** bhavikdbavishi has joined #zuul10:27
*** mordred has joined #zuul10:29
*** masterpe has joined #zuul10:29
*** systemc is now known as systemb10:44
*** hashar is now known as hasharAway10:44
*** bhavikdbavishi has quit IRC10:48
openstackgerritTobias Henkel proposed zuul/nodepool master: Use libyaml parsing when available  https://review.opendev.org/c/zuul/nodepool/+/77196110:53
*** bhavikdbavishi has joined #zuul10:55
openstackgerritMatthieu Huin proposed zuul/zuul master: zuul-client: test "show running-jobs" command  https://review.opendev.org/c/zuul/zuul/+/77196211:07
*** ykarel_ has joined #zuul11:08
*** ykarel has quit IRC11:08
*** bhavikdbavishi1 has joined #zuul11:12
*** bhavikdbavishi has quit IRC11:13
*** bhavikdbavishi1 is now known as bhavikdbavishi11:13
*** harrymichal has quit IRC11:15
*** ykarel__ has joined #zuul11:16
*** harrymichal has joined #zuul11:16
*** ykarel_ has quit IRC11:19
*** jcapitao is now known as jcapitao_lunch11:23
*** mgoddard has quit IRC11:30
*** msuszko has quit IRC11:50
*** msuszko has joined #zuul11:51
*** zenkuro has joined #zuul11:54
*** dry has joined #zuul12:19
*** msuszko has quit IRC12:20
*** rlandy has joined #zuul12:25
openstackgerritTobias Henkel proposed zuul/nodepool master: Handle ports exceeded error as quota error  https://review.opendev.org/c/zuul/nodepool/+/77196812:25
*** iurygregory_ has joined #zuul12:28
*** iurygregory has quit IRC12:28
*** iurygregory_ is now known as iurygregory12:29
*** jpena is now known as jpena|lunch12:35
*** bhavikdbavishi has quit IRC12:40
*** bhavikdbavishi has joined #zuul12:42
*** bhavikdbavishi has quit IRC12:47
*** hasharAway is now known as hashar12:52
*** AJaeger has joined #zuul12:54
*** jcapitao_lunch is now known as jcapitao13:01
*** bhavikdbavishi has joined #zuul13:28
*** mgoddard has joined #zuul13:32
*** jpena|lunch is now known as jpena13:34
*** ykarel__ is now known as ykarel13:44
*** AJaeger has quit IRC13:53
openstackgerritJavier Peña proposed zuul/zuul-jobs master: Fix BaseOS repo file name for CentOS Stream  https://review.opendev.org/c/zuul/zuul-jobs/+/77198013:54
*** bhavikdbavishi has quit IRC14:00
*** slaweq has joined #zuul14:03
slaweqhi zuul cores14:03
slaweqI have a question about patch https://review.opendev.org/c/zuul/zuul-jobs/+/762650 to zuul-jobs14:03
slaweqI neutron ovn jobs, where ovn and ovs is installed from source by devstack plugin we need to "restore" br-infra connectity between nodes14:04
slaweqso I proposed that patch which creates simply script which will do exactly the same thing as ansible role is doing to configure connectivity through this br-infra bridge14:05
slaweqbut I recently had some feedback that it's now 2 places which are doing the same and can be quickly "not in sync"14:06
slaweqso wdyt if I would change that zuul role so it would create script and call it then? I know it's not perfect ansible way solution but that was suggestion already14:06
mordredslaweq: I was actually thinking that when you did your original patch - but probably an idea that should get more than one opinion on :)14:26
slaweqmordred: Your opinion isn't the only one already :)14:27
mordred\o.14:27
mordred\o/14:27
* mordred can't type14:27
slaweq:)14:27
slaweqso if there's more than one voices to do it that way, I will update my patch :)14:28
slaweqthx a lot14:28
*** jamesmcarthur has joined #zuul14:48
openstackgerritMatthieu Huin proposed zuul/zuul master: zuul-client: test "show running-jobs" command  https://review.opendev.org/c/zuul/zuul/+/77196214:57
*** rpittau is now known as rpittau|afk14:58
*** mugsie has joined #zuul15:09
*** GomathiselviS has joined #zuul15:14
*** vishalmanchanda has quit IRC15:15
*** mugsie has quit IRC15:18
*** maxamillion has joined #zuul15:21
*** zbr3 has joined #zuul15:22
*** zbr has quit IRC15:24
*** zbr3 is now known as zbr15:24
*** mugsie has joined #zuul15:51
openstackgerritMatthieu Huin proposed zuul/zuul master: REST API: add api/tenant/{tenant}/jobs/running endpoint  https://review.opendev.org/c/zuul/zuul/+/77164816:02
openstackgerritMatthieu Huin proposed zuul/zuul master: zuul-client: test "show running-jobs" command  https://review.opendev.org/c/zuul/zuul/+/77196216:03
openstackgerritJavier Peña proposed zuul/zuul-jobs master: Fix repo files for CentOS Stream  https://review.opendev.org/c/zuul/zuul-jobs/+/77198016:22
openstackgerritMatthieu Huin proposed zuul/zuul-client master: Add show running-jobs subcommand  https://review.opendev.org/c/zuul/zuul-client/+/77185316:29
mhuhello zuul-maint, thse two bug fixes just need the final +3: https://review.opendev.org/c/zuul/zuul/+/768312 and https://review.opendev.org/c/zuul/zuul/+/76822916:33
*** armstrongs has joined #zuul16:40
*** jfoufas1 has quit IRC16:40
armstrongshey, I have enabled zookeeper encryption and have 3 vm nodes now and they are all accessible on http with the services running. However, the zuul dashboard now appears as a blank page on the load balancer i have in front of zuul. When i inspect the page it says Uncaught SyntaxError: Unexpected token '<' 2.1f35f3c2.chunk.js:116:43
armstrongsso it looks like the static pages on each node maybe can't be accessed now zookeeper encryption is on? any help would be appreciated16:44
corvusarmstrongs: zk encryption shouldn't affect that (the web servers don't do anything with zk yet (but they will in the future).  that sounds like a js error, so is likely completely in the domain of zuul-web or the load balancer16:45
*** ykarel has quit IRC16:47
openstackgerritJavier Peña proposed zuul/zuul-jobs master: Fix repo files for CentOS Stream  https://review.opendev.org/c/zuul/zuul-jobs/+/77198016:47
armstrongscorvus the weird thing is every web node individually is fine and i can access on http16:48
corvusarmstrongs: so maybe something getting munged by the load balancer?16:48
armstrongsthe dashboard loads ok on the servers. Its been running in production for 2 years now with no load balancer changes and the only difference is the zookeeper change thats why i reached out. its very odd. Have tried a couple of different load balancers as well.16:50
corvusmaybe just the upgrade or restart for the zk change triggered the behavior?16:51
armstrongswe are pinned to a ref so code hasn't changed16:51
armstrongsits when we have went from 2 to 3 nodes same config16:52
armstrongsas we needed extra executors as we have added more jobs16:53
corvusarmstrongs: i don't have any suggestions other than to continue to dig into web side of things.  maybe look at the content returned for various requests (maybe an error page is being returned instead of an actual js file).  sorry i don't have more specific ideas.16:56
corvusbut at least, from where i sit now, i don't see how any zk related changes could have immediately caused this16:56
armstrongshttp://paste.openstack.org/show/801890/ thats my config incase anything glaring16:56
corvusyour web server is colocated on one of your zk servers?16:58
armstrongsyeah have all services running on each node. 1 scheduler enabled on 1 box16:59
armstrongsfollows exactly zuul from scratch16:59
armstrongsbut automated16:59
corvusarmstrongs: are you running more than one zuul-web instance?17:01
corvusand load balancing to all of them?17:01
*** reiterative has quit IRC17:01
*** reiterative has joined #zuul17:02
armstrongsyes running 317:03
armstrongs1 web service on each of the 3 boxes17:03
armstrongswith a config for web pointing at local ip17:03
armstrongsso have 3 boxes behind the load balancer17:03
armstrongspointing to port 9000 on the ip17:03
armstrongsserving on the https ssl cert for root17:03
*** harrymichal has quit IRC17:04
armstrongsyeah17:04
corvusso line 28 is different for each of those boxes?17:05
armstrongsyeah that matches the ip of each box17:06
armstrongsand all of the dashboards work if i hit them on http not using the load balancer17:06
armstrongsi just get a blank page and javascript issue when accessing them via the load balancer and i tried a couple17:07
corvusarmstrongs: the ks error said unexpected token in 2.1f35f3c2.chunk.js -- what happens if you look at the network tab in the browser tools and see the content for 2.1f35f3c2.chunk.js ?17:08
corvuss/ks error/js error/17:08
armstrongswill check 2secs17:09
armstrongsDevTools failed to load SourceMap: Could not parse content for https://zuul.paas.je-labs.com/static/js/2.f003d247.chunk.js.map: Unexpected token < in JSON at position 017:12
corvusi don't know what a "js.map" file is; that doesn't show up in opendev's production site17:15
*** GomathiselviS has quit IRC17:15
corvusarmstrongs: can you see the actual payload for that file?17:16
armstrongshttp://paste.openstack.org/show/801891/17:17
*** jamesmcarthur has quit IRC17:17
*** jamesmcarthur has joined #zuul17:18
corvusarmstrongs: that looks like a different url, and it just looks like the headers, not the actual body.  but i still see something interesting there: "Status Code: 200  (from disk cache)"  which suggests something in your web setup has a cache.  maybe that needs to be cleared?  it may not be interacting well with the cach-busting urls generated by webpack?  (that's what the '2.1f35f3c2' stuff is about)17:20
armstrongsso the web setup is just following the zuul from scratch guide is there anything there that would be caching17:22
*** jamesmcarthur has quit IRC17:23
corvusarmstrongs: no.  i'd suggest looking closely at your load balancer.17:24
armstrongsok thanks for the help will have a look17:24
*** hashar has quit IRC17:25
armstrongsim using an alb with standard settings for reference17:26
armstrongsif there is any gotchas there when you have more web nodes17:26
*** jamesmcarthur has joined #zuul17:29
mordredfwiw - js.map is a thing that webpack can produce to allow for in-browser debugging of minified code17:29
armstrongsquick different question based on my config file that i provided corvus what would i require just to run additional executor services on their own in terms of zuul.conf?17:29
armstrongsmordred thanks for the info17:30
*** harrymichal has joined #zuul17:33
*** jamesmcarthur has quit IRC17:34
corvusarmstrongs: the following sections: gearman, statsd, zookeeper, executor, connection*17:38
*** GomathiselviS has joined #zuul17:46
armstrongsthanks17:50
*** jamesmcarthur has joined #zuul17:50
*** jpena is now known as jpena|off17:55
*** jcapitao has quit IRC17:59
*** harrymichal has quit IRC18:06
*** jamesmcarthur has quit IRC18:10
*** slaweq has quit IRC18:12
*** ikhan has joined #zuul18:16
*** jamesmcarthur has joined #zuul18:25
openstackgerritMatthieu Huin proposed zuul/zuul master: zuul-client: test "show running-jobs" command  https://review.opendev.org/c/zuul/zuul/+/77196218:58
avasscorvus, armstrongs that can happen if the zuul-web services are different versions and you're requesting a file that exists in one version and not another19:16
avassI think the main.js (?) has a slightly different name when react generates it19:18
avassoh actually this: 2.1f35f3c2.chunk.js is different if you're running different versions19:18
*** nils has quit IRC19:25
*** ikhan has quit IRC19:25
*** jamesmcarthur has quit IRC19:28
corvusright, it does that for cache busting19:28
avasswhich means requesting 2.1f35f3c2.chunk.js can give you a 404 most likely19:34
*** jamesmcarthur has joined #zuul19:43
*** slaweq has joined #zuul19:46
*** jamesmcarthur has quit IRC19:48
openstackgerritMatthieu Huin proposed zuul/zuul master: zuul-client: test "show running-jobs" command  https://review.opendev.org/c/zuul/zuul/+/77196219:51
*** harrymichal has joined #zuul19:58
*** jamesmcarthur has joined #zuul20:00
*** slaweq has quit IRC20:01
*** harrymichal has quit IRC20:01
*** harrymichal has joined #zuul20:02
openstackgerritMatthieu Huin proposed zuul/zuul master: zuul-client: test "show running-jobs" command  https://review.opendev.org/c/zuul/zuul/+/77196220:13
*** GomathiselviS has quit IRC20:22
*** jamesmcarthur has quit IRC20:34
*** jamesmcarthur has joined #zuul20:34
*** jamesmcarthur has quit IRC20:48
*** armstrongs has quit IRC21:01
*** jamesmcarthur has joined #zuul21:15
*** jamesmcarthur has quit IRC21:23
*** jamesmcarthur has joined #zuul21:29
*** rlandy has quit IRC21:40
openstackgerritArtom Lifshitz proposed zuul/zuul-jobs master: WIP: Generate Zuul temp key in PEM format  https://review.opendev.org/c/zuul/zuul-jobs/+/77212221:52
*** jamesmcarthur has quit IRC22:16
*** jamesmcarthur has joined #zuul22:17
*** sduthil has quit IRC22:27
*** akrpan-pure has joined #zuul22:39
akrpan-pureGuessing there isn't since it doesn't seem very "Traditional YAML"-y, but is there a way to refer to other variables in a job definition? e.g. to say that vars.val2 should have the value of vars.val1?22:40
akrpan-pureOr I guess another way to ask it is "do ansible variable interpolations work in job variable definitions"?22:40
fungiansible doesn't have variable interpolations, you might be thinking of jinja2 substitution which ansible performs over its own yaml files, or you may be thinking of anchors/pointers in yaml which allow you to repeat the same data structure in additional locations within a single file22:45
fungier, i mean yaml doesn't have variable interpolations22:45
akrpan-pureYeah, I was thinking of jinja2. I'm guessing that won't work on job YAMLs?22:45
fungizuul job definitions aren't ansible playbooks22:45
fungiso right, that won't work22:45
akrpan-pureHeck22:46
akrpan-pureMight have a workaround, doing more research22:46
fungithe zuulish way to be DRY is to use inheritence22:46
fungiif it's just a matter of avoiding lots of repeated patterns22:47
akrpan-pureBasically, my two uses are 1. I need to generate a random name prefix for one variable, and 2. There's a few things I'd like to set in one place that have to go in multiple, or I'd at least like to give them more sane constants22:47
akrpan-pureAnd this would definitely help job inheritance22:47
akrpan-pureI'm gonna see if I can set them in a pre-run playbook22:47
fungiyeah, generating a random value in a zuul job definition is something i can't think of an easy solution for. if possible punt that to a playbook/role/module22:48
akrpan-pureYeah, I had the same thing. I know how to do it in shell, just not in a job22:48
fungiyou could rely on fact caching between playbooks, yep22:48
akrpan-pureI'm trying to set variables for the various devstack jobs is my end goal22:49
fungigenerate the value in a pre-run playbook, set it as a fact, then refer to that fact in run22:49
akrpan-pureBut I think since they're just variables, they get applied to the playbook before it's run? I need to find where they actually set them and generate the config file22:49
fungimy ansible-fu is not strong, but you can probably assign the local variable in the playbook by referencing the fact you set in the earlier phase22:50
fungisomeone else in here almost certainly has a better grasp of how to go about that22:51
akrpan-pureWell, time to make a mock playbook and see what happens22:53
fungiyeah, you could likely mock it up locally just making sure fact caching is on, then running one playbook which sets a fact, followed by running another playbook which assigns that fact as a role variable value and then check the value in the role22:54
fungiassuming i understand what it is you're trying to plumb22:54
fungiakrpan-pure: these are what you're wanting to set mostly, right? https://docs.openstack.org/devstack/latest/zuul_roles.html#rolevar-write-devstack-local-conf.devstack_local_conf22:56
akrpan-pureOh! Actually, that's a good point, I forgot you could set vars in the playbook where you add the role, that actually nails exactly what I needed22:56
akrpan-pureYep, that looks like it22:56
corvusalso, just fyi ansible/jinja will often interpolate variables into other varibles, so you frequently can do var: "{{ othervar }}".  zuul would pass that straight through to ansible.22:57
akrpan-pureThat's also super useful to know, thanks!22:58
fungiuseful, and yeah that's what i was thinking you could do with cached facts if you wanted to feed them into a rolevar22:58
fungibut if you don't actually need to generate the values in a different phase, then all the easier22:59
akrpan-pureAwesome, yeah, this should all simplify things. My goal was to have one "pure" dictionary variable that's the source of truth, and then all the other things are set based on those variables22:59
akrpan-pureWhich I think is in reach now22:59
*** ikhan has joined #zuul23:01
fungii love it when a plan comes together23:02
*** paladox has quit IRC23:14
*** paladox has joined #zuul23:17
*** jamesmcarthur has quit IRC23:34
*** paladox has quit IRC23:40
*** paladox has joined #zuul23:55

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!