Tuesday, 2019-04-02

*** timburke has joined #zuul00:00
*** jamesmcarthur has joined #zuul00:01
*** mgoddard has quit IRC00:06
*** mgoddard has joined #zuul00:22
*** rlandy has quit IRC00:25
*** jamesmcarthur has quit IRC00:31
*** jamesmcarthur has joined #zuul00:54
*** jamesmcarthur has quit IRC00:58
*** jamesmcarthur has joined #zuul01:27
*** jamesmcarthur has quit IRC01:34
*** mrhillsman_afk is now known as mrhillsman02:10
*** bhavikdbavishi has joined #zuul02:33
*** jamesmcarthur has joined #zuul02:44
*** jamesmcarthur has quit IRC03:04
*** spsurya has joined #zuul03:31
*** openstackgerrit has joined #zuul03:33
openstackgerritIan Wienand proposed openstack-infra/nodepool master: Add Debian Buster boot tests  https://review.openstack.org/64921403:33
*** bjackman has joined #zuul03:37
*** jamesmcarthur has joined #zuul03:56
*** jamesmcarthur has quit IRC03:59
*** jamesmcarthur has joined #zuul04:01
*** jamesmcarthur has quit IRC04:05
*** jamesmcarthur has joined #zuul04:08
*** jamesmcarthur has joined #zuul04:08
*** jamesmcarthur has quit IRC04:17
*** jamesmcarthur has joined #zuul04:18
*** jamesmcarthur has quit IRC04:25
*** jamesmcarthur_ has joined #zuul04:26
*** jamesmcarthur_ has quit IRC04:39
*** jamesmcarthur has joined #zuul04:40
*** jamesmcarthur has quit IRC04:46
*** pcaruana has joined #zuul05:35
*** jamesmcarthur has joined #zuul05:41
*** jamesmcarthur has quit IRC05:46
*** quiquell|off is now known as quiquell05:49
*** jamesmcarthur has joined #zuul05:57
*** jamesmcarthur has quit IRC06:01
*** openstackgerrit has quit IRC06:09
*** jamesmcarthur has joined #zuul06:28
*** gtema has joined #zuul07:11
*** themroc has joined #zuul07:44
*** jpena|off is now known as jpena07:58
*** zbr|pto is now known as zbr08:23
*** openstackgerrit has joined #zuul08:47
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: Add Authorization Rules configuration  https://review.openstack.org/63985508:48
*** hashar has joined #zuul08:48
*** manjeets has quit IRC08:56
*** manjeets has joined #zuul08:57
*** manjeets has quit IRC09:05
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: Add Authorization Rules configuration  https://review.openstack.org/63985509:28
openstackgerritMonty Taylor proposed openstack-infra/nodepool master: Remove TaskManager and just use keystoneauth  https://review.openstack.org/64064309:30
mordredtobiash: heads up - we released a new openstacksdk that has the taskmanager removed. if you encounter any issues you may want to pin openstacksdk to <= 0.26.009:32
tobiashmordred: thanks for the hint, I'll try it in a test environment first :)09:32
mordredactually - you know - why don't I push up a <= 0.26.0 patch to nodepool in front of the patch to actually remove taskmanager from nodepool09:33
*** electrofelix has joined #zuul09:34
openstackgerritMonty Taylor proposed openstack-infra/nodepool master: Remove TaskManager and just use keystoneauth  https://review.openstack.org/64064309:36
openstackgerritMonty Taylor proposed openstack-infra/nodepool master: Pin openstacsksdk to <0.27.0  https://review.openstack.org/64925409:36
mordredtobiash: ^^ maybe that's a safer way to step this one forward09:36
tobiashmordred: so 640643 is needed to work with the new openstacksdk?09:37
mordredtobiash: I think it SHOULD work fine with new openstacksdk and the stuff in 640643 is mostly just cleanup09:38
mordredalthough the statsd reporting keys will change09:38
tobiashah, ok09:38
*** manjeets has joined #zuul09:38
mordredas documented in the https://review.openstack.org/#/c/640643/6/releasenotes/notes/task-manager-replaced-12e4b3a0108f9358.yaml09:38
tobiashah ok, I don't care about the reporting keys :)09:39
mordred\o/09:40
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: WIP: Test Single provider reload  https://review.openstack.org/64925909:41
openstackgerritFabien Boucher proposed openstack-infra/zuul master: WIP - Pagure driver  https://review.openstack.org/60440409:52
mordredfbo: oh cool - all the needed changes have been landed in pagure?10:02
mordredfbo: ah - reading the outstanding issues list in https://review.openstack.org/#/c/604404/19/zuul/driver/pagure/pagureconnection.py10:05
mordredfbo: I agree, the current blocking issue is an issue for sure10:06
tristanCmordred: here is a PoC for the operator we talked about: https://github.com/TristanCacqueray/zuul-operator10:08
tristanCmordred: not sure how far you are from the spec, thus i went ahead to start a prototype10:09
mordredtristanC: cool! working on the spec is actually my next todo list item - as soon as I stop being sick10:11
mordredtristanC: incidentally: https://github.com/operator-framework/operator-lifecycle-manager/blob/master/Documentation/design/building-your-csv.md#required-crds is the relevant docs to read about declaring inter-operator crd dependencies10:14
mordredtristanC: so I believe we should be able to use that to pull in the zookeeper crd10:15
tristanCoh yes, that's an early poc to test an ansible based reconcile loop10:25
tristanCalso, adding crd is pretty neat, they are available from the cli and you can do: "oc get zuul" :-)10:26
*** bhavikdbavishi has quit IRC10:30
mordredtristanC: neat!10:35
*** yolanda has quit IRC10:39
*** gtema has quit IRC10:41
bjackmanWith my version of Gerrit I am getting some unexpected pipeline triggers. I have "Workflow: 1" in my pipeline.trigger.gerrit.event.approval for a "comment-added" event. But rather than only triggering on comments that _add_ the approval, it seems to trigger for any comment on a change that already has the approval10:47
bjackmanCould it be that the Gerrit event stream format has changed?10:48
bjackmanThe data received by the driver looks like this: https://paste.gnome.org/pl7l9udi210:48
bjackmanThe "approvals" field in the object contains "{'description': 'Workflow', 'type': 'Workflow', 'value': '1'}" even though the comment did not add the Workflow +1 - it was already there10:49
bjackmanInterestingly it includes the "Code-Review +2" and "Workflow +1" that I added, but does not include the "Verified -2" added by Zuul10:50
bjackmanAh yes, looking at the Gerrit docs it seems like Zuul should be checking the "oldValue" field in the approvals to see if the approval was newly created by the comment: https://gerrit-review.googlesource.com/Documentation/json.html#approval10:55
mordredbjackman: I'm not an expert in the gerrit event stream - but I know on older gerrit zuul does the right thing for the sequence of events I beleive you are describing10:59
mordredtobiash: ^^ does this sound like a gerrit behavior change to you?10:59
mordredbjackman: I asumme you're running latest gerrit?11:00
tobiashmordred: unfortunately I'm not very familiar with the gerrit event stream anymore as I don't run a recent gerrit myself since quite some time now11:01
mordreddarn. none of us are running recent gerrit then I guess11:02
mordredtobiash: in other news - yay, the "remove taskmanager" patch https://review.openstack.org/#/c/640643/ is green11:03
tobiashmordred: cool :)11:03
*** bjackman has quit IRC11:06
sshnaidm|roverhow can I define zuul reconnection retries and timeout?11:17
*** gtema has joined #zuul11:23
*** yolanda has joined #zuul11:26
openstackgerritMerged openstack-infra/zuul-jobs master: Add trailing slash for log url  https://review.openstack.org/64883711:27
*** jpena is now known as jpena|lunch11:29
*** bhavikdbavishi has joined #zuul11:32
mordredShrews: whence you make the awakening, https://review.openstack.org/#/q/topic:task-manager+status:open are just for you!12:04
*** bhavikdbavishi has quit IRC12:09
openstackgerritMonty Taylor proposed openstack-infra/nodepool master: Remove unused use_taskmanager flag  https://review.openstack.org/64929312:13
openstackgerritMonty Taylor proposed openstack-infra/nodepool master: Remove reference to TaskManagerStopped exception  https://review.openstack.org/64929412:13
*** manjeets has quit IRC12:16
*** jamesmcarthur has quit IRC12:23
*** jamesmcarthur has joined #zuul12:23
*** jpena|lunch is now known as jpena12:27
*** jamesmcarthur has quit IRC12:33
*** jamesmcarthur has joined #zuul12:35
*** rlandy has joined #zuul12:39
*** jamesmcarthur has quit IRC12:40
*** jamesmcarthur has joined #zuul12:44
*** hashar has quit IRC12:54
*** quiquell is now known as quiquell|lunch13:00
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: Web: plug the authorization engine  https://review.openstack.org/64088413:03
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: Zuul Web: add /api/user/actions endpoint  https://review.openstack.org/64109913:04
*** quiquell|lunch is now known as quiquell13:24
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: Zuul Web: add /api/user/authorizations endpoint  https://review.openstack.org/64109913:41
*** quiquell is now known as quiquell|off14:11
Shrewsmordred: fyi, those changes totally undo ianw's change from https://review.openstack.org/605898  That may or may not be a bad thing????14:14
*** smcginnis_pto is now known as smcginnis14:16
Shrewsfluctuating back and forth on the stats api is probably not going to be well received14:18
openstackgerritFabien Boucher proposed openstack-infra/zuul master: WIP - Pagure driver - https://pagure.io/pagure/  https://review.openstack.org/60440414:19
corvusyay!  https://zuul-ci.org/docs/zuul-sphinx/0.4.0/ exists14:41
corvusi approved a change which will re-exercise the promote job too14:43
*** hashar has joined #zuul14:46
*** gtema has quit IRC14:49
openstackgerritFabien Boucher proposed openstack-infra/zuul master: WIP - Pagure driver - https://pagure.io/pagure/  https://review.openstack.org/60440414:50
openstackgerritMerged openstack-infra/zuul-sphinx master: Raise an error if a file in zuul.d is empty  https://review.openstack.org/60606214:50
*** yolanda has quit IRC14:55
openstackgerritJames E. Blair proposed zuul/project-config master: Fix docs promote job  https://review.openstack.org/64935514:55
openstackgerritMonty Taylor proposed openstack-infra/zuul-website master: Add a slide presentation for Zuul Graduation  https://review.openstack.org/64935614:55
corvusmordred: ^ quick fix if you have a sec14:55
mordredcorvus: ^ first draft of graduation slides14:55
mordredthere are bits I'm sure that are wrong, missing, terrible, etc14:55
corvusmordred: \o/14:55
corvusthat's gonna be way easier to review when the docs jobs come back :)14:56
openstackgerritMerged zuul/project-config master: Fix docs promote job  https://review.openstack.org/64935515:05
*** yolanda has joined #zuul15:13
mordredcorvus: nah - it doesn't need them: http://logs.openstack.org/56/649356/1/check/zuul-website-build/6d486de/html/graduation/#(1)15:15
corvusmordred: sorry, that's what i meant -- when the job reports15:17
mordredcorvus: ah :)15:18
clarkbmordred: quickly skimming through and on the architecture diagram is it accurate that the finger service talks to gearman? I thought it talked directly to the executors over the finger protocol (on a higher port)15:20
mordredclarkb: I agree - fwiw, that architecture diagram is actually a direct link to the one from the docs15:21
clarkb++ to using that one and if it needs fixing, fixing it in the docs is a good place :)15:22
corvusit talks to gearman to find out which executor to talk to15:24
clarkbon the engagement side of things not sure if we want to mention zuul's work with ansible and openlab but those were other examples that came to mind15:24
clarkbcorvus: ah15:24
mordredcorvus: oh right. good point15:24
corvusclarkb: ++15:24
openstackgerritPaul Belanger proposed openstack-infra/zuul master: Add web / fingergw connections for components graph  https://review.openstack.org/64585215:24
*** jamesmcarthur has quit IRC15:24
mordredclarkb: ++15:24
*** jamesmcarthur has joined #zuul15:25
*** themroc has quit IRC15:26
clarkbone last thought before I find tea: Under the use cases heading would it be useful to call out that zuul is used in a variety of target applications (that is not the right phrase, but basically we use it for open source project development with gerrit, ansible uses github, bmw does it for their internal development and none of this assumes a specific programming language or target deployment platform [openstack15:29
clarkbk8s baremetal etc])15:29
clarkbprobably something along the lines of "zuul is flexible and tries to avoid assumptions about software being tested and target platforms"15:30
clarkbthis sets it apart from current trends in the space too imo. As there is a big rush towards all in on containers as an example15:30
clarkbwhich neglects a set of users15:31
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: Web: plug the authorization engine  https://review.openstack.org/64088415:32
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: Zuul Web: add /api/user/authorizations endpoint  https://review.openstack.org/64109915:33
openstackgerritPaul Belanger proposed openstack-infra/zuul master: Add database info for components graph  https://review.openstack.org/64937115:34
pabelanger^should give deployers a good map of the required connections needed between zuul services.  Was helpful when setting up firewall permissions15:35
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: authentication config: add optional token_expiry  https://review.openstack.org/64240815:37
*** jamesmcarthur has quit IRC15:44
*** yolanda has quit IRC15:46
*** yolanda has joined #zuul15:47
*** bhavikdbavishi has joined #zuul15:47
mordredclarkb: ++16:00
*** jamesmcarthur has joined #zuul16:04
openstackgerritMerged openstack-infra/zuul master: Use xterm.js for live log streaming  https://review.openstack.org/64883816:04
*** hashar has quit IRC16:07
pabelangertobiash: PgUp / PgDn don't seem to work on ^ now. At one point shift+PgUp / shift+PgDn did. Any idea why?16:08
pabelangerI find it helpful to use them to move up and down, without using mouse16:08
tobiashpabelanger: is that already live?16:09
clarkbthe web updates go live after a puppet pulse iirc16:09
tobiashpabelanger: I guess it needs the focus16:10
clarkbwhich has not yet happened16:10
pabelangertobiash: I was using zuul-build-dashboard job to validate16:10
mordredclarkb: ok. I took a stab at that.16:11
pabelangerfor example: http://logs.openstack.org/65/648865/3/check/zuul-build-dashboard/4864887/npm/html/stream/15b3cf82b54f46db8bcb4c45b2456c19?logfile=console.log16:11
pabelangeroh, guess can link that16:11
pabelangercan't*16:13
openstackgerritMonty Taylor proposed openstack-infra/zuul-website master: Add a slide presentation for Zuul Graduation  https://review.openstack.org/64935616:16
mordredclarkb: ^^ included your suggestions and rearranged the use-cases slide a smidge16:17
clarkbcool taking a look16:17
mordredclarkb: (written in python3 isn't really a use case)16:17
clarkbooh I like the comment about network switches because thats a thing :)16:17
mordredclarkb: yah - I just rewrote that line - tell me if you like this better:16:19
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Increase timeout of zuul-quick-start  https://review.openstack.org/64938416:19
*** yolanda has quit IRC16:19
mordredclarkb: http://paste.openstack.org/show/748733/16:19
*** jamesmcarthur has quit IRC16:21
clarkbI would list k8s and openshift as different things as they use different drivers. But the rewrite in the paste flows a little better I think16:21
clarkbs|/|and| type replacement16:21
SpamapSopenshift is a superset of k8s, IIRC16:25
SpamapSlike, they do actually use many of the same underlying API's and python modules16:25
openstackgerritJames E. Blair proposed zuul/project-config master: Set a default for docs path templates  https://review.openstack.org/64938916:26
corvusclarkb, mordred: ^ is what i'm gathering from this error: http://logs.openstack.org/62/606062/2/promote/zuul-promote-docs/83e4331/ara-report/result/f10d43cd-8cc6-4136-b71a-de035981d39e/16:26
*** jamesmcarthur has joined #zuul16:27
mordredSpamapS: it's a superset - but also a stricter execution context - so you really can't assume something that works on k8s will work on openshift16:27
corvus(^ *optionally)16:27
mordredSpamapS: but also - I believe we use the openshift client library to talk to vanialla k8s because it's better at being a client library16:27
clarkbmordred: the kubernetes python lib is gerneated code and openshift lib is built on top of that making it more humanly consumable16:28
SpamapSmordred: indeed16:28
* corvus reviews http://logs.openstack.org/56/649356/2/check/zuul-website-build/37f4b4d/html/graduation16:28
SpamapSI've been kind of stuck with the kubernetes driver... but I'm hoping to poke at it more this week.16:29
mordredactually - how about I remove OpenShift from the line - that's an impl detail - I think kuberenetes is sufficient to explain COE without having to explain what a COE is to board members16:29
openstackgerritMonty Taylor proposed openstack-infra/zuul-website master: Add a slide presentation for Zuul Graduation  https://review.openstack.org/64935616:30
corvusmordred: on slide 5 you mention cloud provider drivers for openstack and aws; should k8s be there?16:31
Shrewsand static nodes?16:31
corvus++16:32
mordred++16:32
openstackgerritMonty Taylor proposed openstack-infra/zuul-website master: Add a slide presentation for Zuul Graduation  https://review.openstack.org/64935616:33
pabelangerI have a few patches out for components diagram update, so far: http://logs.openstack.org/71/649371/1/check/tox-docs/5aec25a/html/admin/components.html is the latest version.  If we wanted more moving bits for that slide.16:33
corvusmordred: looks good!  i'd vote for us.  :)16:34
mordredcorvus: yay!16:35
mordredpabelanger: I like the updated diagram16:35
mordredpabelanger: although nodepool also talks to statsd16:35
pabelangermordred: yah, I didn't include nodepool bits, as we likely want to split into nodepool-launcher / nodepool-builder16:36
pabelangerbut if we want that, we can totally do16:36
mordredgood point. as it's in a single bubble, yeah - leaving it like that makes maybe more sense16:36
mordredI wish there was a graphviz shape that could be used to indicate "this is a scale-out component so there are potentially many of this type"16:39
pabelangerYES!16:39
pabelangerthat would be awesome16:39
mordredbut none of the existing ones listed on https://www.graphviz.org/doc/info/shapes.html would do that without some sort of explanation16:39
corvuswe could switch to blockdiag: http://blockdiag.com/en/blockdiag/examples.html#shape-of-nodes16:40
corvusit has 'stacked' which i think would work16:40
corvushttp://blockdiag.com/en/blockdiag/examples.html#using-stacked-attribute16:40
corvuswe use blockdiag elsewhere in the manual; i think that diagram is the only place we use graphviz16:40
mordredyeah- stacked is the thing my brain wants16:41
mordredwe could also just make an eps file with a stacked box in it16:41
corvusis that easier than switching to blockdiag?16:41
mordredif we didn't want to switch that diagram from graphviz to blockdiag16:41
mordredI don't know - you're the main person I know who can write eps by hand16:41
corvusoh, it also has dots: http://blockdiag.com/en/blockdiag/examples.html#using-dots-shape16:41
mordredcorvus: switching to blockdiag doesn't look like it would be super harder16:42
pabelanger+1 I can push up a patch if we want to see what it looks like16:44
corvus0 0 10 10 rectstroke 2 2 10 10 rectstroke 1 setgray 4 4 10 10 rectfill 0 setgray 4 4 10 10 rectfill16:44
corvusor something like that :)16:45
clarkbmordred: seems like the presentation lacks an ending16:50
clarkbmaybe future goals16:51
mordredclarkb: do we have any other than "take over the world"?16:51
corvushrm, is future goals what the board is looking for?  i thought this was more retrospective16:51
clarkbcorvus: I think it is mostly retrospective. Just trying to come up with some way to send it off16:52
clarkbhttps://wiki.openstack.org/w/index.php?title=Governance/Foundation/OSFProjectConfirmationGuidelines are the guidelines which I am not rereading16:52
clarkbfuture goals might be best captured under strategic focus under those guidelines16:53
*** jamesmcarthur_ has joined #zuul16:53
corvuscould send it off with links to website, etc16:54
clarkb*now rereading16:54
corvusor find something to summarize16:55
corvusclarkb, mordred: also remember this exists: https://zuul-ci.org/media/zuul_solution_brief.pdf16:56
*** jamesmcarthur has quit IRC16:57
corvusmordred: you should set the <title>16:57
mordredcorvus: I should set the title16:59
clarkbcorvus: ya a "find us at $locations" bookend slide might work best given the guidelines17:00
clarkbwe should stay focused on the areas they are interested in17:00
clarkbor a "questions" bookend17:00
clarkbthough given historical board meetings I expect they will just interrupt and ask quetsions as we go17:01
openstackgerritMerged zuul/project-config master: Set a default for docs path templates  https://review.openstack.org/64938917:02
fungispeaking of zuul's impending oip confirmation (i left a couple comments on the current deck patchset btw), here is the raw feedback from the openstack community the tc will be working into a foundation ml post to fulfil its obligation to the board with providing input on oip confirmation applications: https://etherpad.openstack.org/p/openstack-tc-zuul-confirmation-feedback17:11
fungiin summary, it's overwhelmingly positive17:12
fungi(we requested input from the whole openstack community, but in the end it was mostly just tc members providing feedback on the zuul confirmation, we had a few more non-tc folks weigh in on the kata confirmation feedback etherpad)17:14
openstackgerritMerged openstack-infra/zuul master: Add search function to web streaming  https://review.openstack.org/64886217:18
*** jamesmcarthur_ has quit IRC17:28
*** jpena is now known as jpena|off17:28
corvusyay! https://zuul-ci.org/docs/zuul-sphinx/.root-marker is correct17:34
pabelangercorvus: clarkb: so, I was able to reproduce ara issue running ansible-playbook from ./venv/bin/ansible-playbook locally, PATH did not include venv/bin.17:37
pabelangerI don't see any logic to add that in zuul-executor17:37
corvustobiash: ^ do you have thoughts on this?  istr that we're including ara in the virtualenvs, but it looks like the virtualenv doesn't end up in the path17:38
mordredcorvus, clarkb, fungi: I'm done hacking for the day - so if anyone wants to make updates to the slides directly, please feel free to push up new changes - my feelings will certainly not be hurt17:46
mordredoh - we should also add jimmy as a co-authored-by - he did all of the css and the first pass at those which I then imported and edited - would be remiss to not credit him17:46
fungithanks mordred! and i wish you a swift recovery from your travel plague17:46
mordredfungi: my travel plague shakes its fists at you - but I thank you17:46
clarkbI guess the gear install works via python path not path path?17:58
*** bhavikdbavishi has quit IRC17:58
*** gtema has joined #zuul17:59
corvusclarkb: yeah, i don't think we run any gear binaries17:59
SpamapSheh.. so.. I just noticed.. I think for the second time but I forgot last time.. the aws driver completely ignores max-servers18:01
tobiashcorvus: the virtualenv normally is in the path, but it might be because of the localhost connection of ansible18:06
tobiashcorvus: we don't set ansible_python_interpreter for localhost do we?18:08
tobiashcorvus: I can reproduce that locally by creating a virtualenv with ansible and ara and running './ansible/bin/ansible localhost -m shell -a ara'18:10
tobiashcorvus: looks like the path is only set if the virtualenv is getting activated18:12
pabelangermordred: Shrews: happen to notice this traceback from nodepool today: http://paste.openstack.org/show/748735/ any thoughts?18:12
tobiashcorvus: so one easy fix could be to just add the dir containing the ansible executable to the path of the ansible process18:12
tobiashthat's exactly what happens when the virtualenv gets activated18:13
corvustobiash: that sounds reasonable18:14
pabelanger+118:14
tobiashvirtualenv prepends the path, I'll prepare a change that does the same18:15
tobiashconfirmed that this works: PATH=`pwd`/ansible/bin:$PATH ./ansible/bin/ansible localhost -m shell -a ara18:16
tobiashso that will work18:16
Shrewspabelanger: uh, hrm. that's odd18:17
pabelangertobiash: yah, that was my workaround locally too18:18
*** electrofelix has quit IRC18:18
*** rlandy is now known as rlandy|brb18:19
openstackgerritPaul Belanger proposed openstack-infra/zuul master: Add database info for components graph  https://review.openstack.org/64937118:20
Shrewspabelanger: unfortunately, nothing springs to mind as to what could be happening there18:22
pabelangerShrews: k, will keep an eye out for it. I've deleted instances manually to silence the exception for now18:23
SpamapSoh fun, and I also think the aws driver doesn't actually respect the region passed in via the provider config18:30
SpamapSseems it uses the .aws/config file only18:30
*** spsurya has quit IRC18:32
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/nodepool master: Fix AWS provider ignoring region in providers  https://review.openstack.org/64942318:36
*** rlandy|brb is now known as rlandy18:36
*** gtema has quit IRC18:40
ShrewsSpamapS: maybe a new test for that ^^?  you can use pool.getProviderManager() to get at that object18:44
SpamapSDoesn't fix the problem anyway. :(18:45
fungiin light of mordred's earlier comments in here, any objection to me pushing an update to the zuul confirmation slide deck change addressing my comments and adding jimmy as a co-author?18:46
SpamapSanother weird thing is that the driver is choosing an explicit availability zone, even though it never actually mentions availability zones. Makes me think this may be a boto3 bug18:47
openstackgerritNicolas Hicher proposed openstack-infra/zuul-jobs master: Propose submit-logstash role to export logs  https://review.openstack.org/64942618:48
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Prepend path with bin dir of ansible virtualenv  https://review.openstack.org/64942718:48
tobiashcorvus, pabelanger: ^18:48
corvusfungi: sounds good18:50
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Cleanup executor specific requirements  https://review.openstack.org/64942818:50
pabelangertobiash: that should mean openstack-infra, which installs ara globally on executor, now pick it up from virtualenv.18:52
pabelangerclarkb: ^18:52
tobiashpabelanger: yes, I'd expect that to happen18:52
clarkbI think we've carefully controlled the ara version but I'm not sure that matters? Then we could clean up our puppet for that18:53
tobiashclarkb: did you carefully control it in puppet or as a zuul requirement in the past?18:53
clarkbtobiash: its in puppet let me dig it up18:54
clarkbI don't think it is necessary though18:54
clarkbhttps://git.openstack.org/cgit/openstack-infra/puppet-zuul/tree/manifests/executor.pp#n53 its that nasty exec. And actually we install latest ara just not latest everything else18:54
clarkbso ya this chagne should be totally fine18:54
tobiashcorvus: responded on 64938419:01
corvustobiash: ++19:02
tobiashcorvus: we throw the builder image away right? So we could copy in the correct proxy config without having to worry about contaminating the final image?19:05
pabelangercould I get a review on https://review.openstack.org/648815/ to update prepare-workspace for static nodes? I've included log file to downstream testing19:07
corvustobiash: yes, though we do install packages in the final image too (bubblewrap)19:07
tobiashhrm, ok, so we don't have to worry about that in the builder step and need to find a proper solution in the final step19:08
pabelangertobiash: SpamapS: how have you setup your database for zuul? Are you doing any sort of HA with it today, if so, what components are you using? Currently, I just have a single db server running and trying to see steps to maybe to HA without tomuch down the rabbit hole19:13
pabelangermy current cloud doesn't offer dbaas19:13
tobiashpabelanger: I offloaded that to the database department so I don't have to worry about that19:13
tobiashbecause yes, HA database is hard and you need to know what you're doing19:14
tobiashand I don't19:14
pabelanger++19:14
tobiash(at least for databases ;) )19:14
pabelangeryah, so far zookeeper hasn't been terrible. It just works.19:15
pabelanger(HA)19:15
tobiashyeah, because it has HA builtin19:15
tobiashthat's not the case for traditional sql databases where you need to deal with replication, failover, recovery...19:16
tobiashpabelanger: maybe you want to try this one: https://www.cockroachlabs.com/19:16
tobiashcloud native ha containerized sql database19:16
tobiashand supports the postgres wire format19:16
pabelangertobiash: thanks, will find some time to test it out19:18
tobiashthat could be the 'just works' solution if it works good19:18
pabelangeryah19:18
clarkbdoes zuul gracefully degrade if the db is down?19:19
pabelangergood question19:20
*** gtema has joined #zuul19:23
SpamapSpabelanger: I'm using AWS RDS, with no HA. We just back it up and if it goes down we'll restore from snap and move on.19:24
SpamapSclarkb: I don't think it does.19:24
SpamapSI think scheduler keeps working, and running jobs tho19:25
SpamapSjust throws exceptions19:25
SpamapSso .. maybe sort of gracefully19:25
tobiashcorvus: is debian mirrored completely in your infrastructure? looking at http://mirror.sjc1.vexxhost.openstack.org/debian/dists/stretch/ this is missing the signature of the Release file compared to http://ftp.debian.org/debian/dists/stretch/19:26
pabelangertobiash: it isn't a 100% mirror, we dont' sign artifacts because we keep some packages a few hours longer before removing. The window is about 4 hours behind upstream, in an effort to not break runnings jobs.19:27
tobiashpabelanger: apt-get complains about the missing signature: W: The repository 'http://mirror.sjc1.vexxhost.openstack.org/debian stretch Release' is not signed.19:28
pabelangertobiash: yah, we need to disable signed packages19:29
pabelangerwhere do you see that?19:29
tobiashtrying locally to add the mirrors to the docker image build19:29
clarkbcorrect we use reprepro to build a new repository instead of mirroring directly. The reason for this is deb mirrors are updated in a manner that breaks clients often19:29
clarkbso instead we build a working repo then publish it via afs only when it is viable19:29
*** gtema has quit IRC19:29
pabelangertobiash: http://git.zuul-ci.org/cgit/zuul-jobs/tree/roles/configure-mirrors/templates/etc/apt/apt.conf.d/99unauthenticated.j219:30
pabelangerthat is what we set on all nodes19:30
pabelangeryou'll have to do the same19:30
pabelanger(all test nodes)19:30
pabelangerclarkb: what was the tool your bother created again, in an attempt to fix this issue?19:31
clarkbhttps://github.com/squidboylan/apt-package-mirror19:32
*** ianychoi has quit IRC19:32
pabelangerthanks19:32
*** ianychoi has joined #zuul19:33
SpamapSBTW...19:34
SpamapSapt has been fixed19:34
tobiashpabelanger: thanks, that seems to do the trick19:34
SpamapSnewer versions do not have the cache expiration bug that old versions had19:34
clarkbSpamapS: ya you need bionic though iirc19:35
clarkbxenial isn't new enough. So soon we may have a better story there (openstack still using a lot of xenial)19:35
tobiashpabelanger, clarkb: how do other openstack projects publish docker images? I don't feel very comfortable installing unsigned packages into official images.19:35
clarkbtobiash: many of them don't use our mirrors to build the images19:36
SpamapSprobably in a year or so19:36
SpamapSright? :)19:36
tobiashok, so we might want two variants, one that uses the mirror which is used for zuul-quick-start and one that doesn't use the mirrors in the gate (zuul-build-images)?19:37
SpamapSOk I was wrong, boto wasn't choosing an AZ for me. I forgot I had an explicit VPC/subnet and the subnet was AZ specific.19:37
pabelangertobiash: yah, I wouldn't want to use our mirrors for production builds19:44
pabelangerSpamapS: is that fix in bionic?19:44
pabelangerwowo19:46
pabelangerwoah*19:46
pabelangernew xterm.js is live on zuul.o.o19:46
mhuwhat does it do?19:47
pabelangermhu: faster loading of logs19:47
mhusweet!19:47
mhuI'll put it to the test right now!19:48
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: Web: plug the authorization engine  https://review.openstack.org/64088419:48
*** hashar has joined #zuul19:49
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: Zuul Web: add /api/user/authorizations endpoint  https://review.openstack.org/64109919:50
daniel2When you specify multiple targets in  nodepool, such as 2 jenkins servers, how do you specify where to launch the nodes?19:50
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: Zuul Web: add /api/user/authorizations endpoint  https://review.openstack.org/64109919:52
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: authentication config: add optional token_expiry  https://review.openstack.org/64240819:53
pabelangerdaniel2: usually you kept all your jenkins in sync, so jobs could run on any of them, using something like JJB.19:56
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: authentication config: add optional token_expiry  https://review.openstack.org/64240819:56
daniel2pabelanger: the nodes should show up as available on both jenkins then I am assuming?19:57
pabelangerdaniel2: nodepool would dynamically attach a node to any one of the jenkins master you had configured. No 2 masters would have the same single node.19:59
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: web: add tenant and project scoped, JWT-protected actions  https://review.openstack.org/57690720:01
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: [WIP] Allow operator to generate auth tokens through the CLI  https://review.openstack.org/63619720:01
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: [WIP] Zuul CLI: allow access via REST  https://review.openstack.org/63631520:01
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: [WIP] Add Authorization Rules configuration  https://review.openstack.org/63985520:01
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: [WIP] Web: plug the authorization engine  https://review.openstack.org/64088420:01
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: Zuul Web: add /api/user/actions endpoint  https://review.openstack.org/64109920:01
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: authentication config: add optional token_expiry  https://review.openstack.org/64240820:01
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: [WIP] admin REST API: docker-compose PoC, frontend  https://review.openstack.org/64353620:01
mhuwoops20:01
mhuwhat's the easiest way to return to a previous patchset?20:02
daniel2pabelanger: can we specify availability per target?20:03
pabelangerdaniel2: IIRC, no.20:03
pabelangerdaniel2: http://git.zuul-ci.org/cgit/nodepool/tree/doc/source/configuration.rst?h=0.5.0#n56320:04
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: web: add tenant and project scoped, JWT-protected actions  https://review.openstack.org/57690720:05
daniel2okay thanks20:05
clarkbmhu: Gerrit enforces that the most recent patchset be the one that can merge. So you have to checkout and repush older patchsets if you want them20:06
mhuclarkb, that's what I was afraid of, thanks20:06
pabelangerdaniel2: with zuul 3.0, we do support executor zones: https://zuul-ci.org/docs/zuul/admin/components.html#attr-executor.zone20:08
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: Allow operator to generate auth tokens through the CLI  https://review.openstack.org/63619720:08
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: Zuul CLI: allow access via REST  https://review.openstack.org/63631520:10
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: Add Authorization Rules configuration  https://review.openstack.org/63985520:11
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: Web: plug the authorization engine  https://review.openstack.org/64088420:12
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: Zuul Web: add /api/user/authorizations endpoint  https://review.openstack.org/64109920:12
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: authentication config: add optional token_expiry  https://review.openstack.org/64240820:13
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: [WIP] admin REST API: docker-compose PoC, frontend  https://review.openstack.org/64353620:14
openstackgerritFabien Boucher proposed openstack-infra/zuul master: A reporter for Elasticsearch with the capability to index build and buildset results in an index.  https://review.openstack.org/64492720:16
openstackgerritMerged openstack-infra/zuul master: Document how to run eslint locally  https://review.openstack.org/64885520:16
openstackgerritFabien Boucher proposed openstack-infra/zuul master: A reporter for Elasticsearch with the capability to index build and buildset results in an index.  https://review.openstack.org/64492720:17
openstackgerritTobias Henkel proposed openstack-infra/zuul master: WIP: Use mirror infrastructure during zuul-quick-start  https://review.openstack.org/64944820:18
openstackgerritTobias Henkel proposed openstack-infra/zuul master: WIP: Use mirror infrastructure during zuul-quick-start  https://review.openstack.org/64944820:21
openstackgerritJeremy Stanley proposed openstack-infra/zuul-website master: Add a slide presentation for Zuul Graduation  https://review.openstack.org/64935620:22
openstackgerritJeremy Stanley proposed openstack-infra/zuul-website master: Slide presentation for Zuul's OIP Confirmation  https://review.openstack.org/64935620:26
*** pcaruana has quit IRC20:30
*** jamesmcarthur has joined #zuul20:47
*** jamesmcarthur has joined #zuul20:47
*** adam_g has quit IRC20:53
corvustobiash, mordred, tristanC: oops: https://review.openstack.org/60936621:02
tobiashoops, sorry tristanC21:04
*** jamesmcarthur has quit IRC21:19
openstackgerritMerged openstack-infra/zuul-jobs master: Add parameter to upload-git-mirror to specify ssh host key  https://review.openstack.org/64903321:22
openstackgerritNicolas Hicher proposed openstack-infra/zuul-jobs master: Propose submit-logstash role to export logs  https://review.openstack.org/64942621:24
corvusfungi: thanks for sprinkling those periods on the confirmation presentation; i was about to do that myself :)21:27
fungiheh, i held back on some of my more nit-picky tendencies21:29
fungibut figured at least consistent punctuation was worth having21:30
openstackgerritTim Burke proposed openstack-infra/zuul-jobs master: Exclude linux_x86_64 wheels from upload to PyPI  https://review.openstack.org/64945521:32
fungithat's ^ a problem we spotted in #openstack-release a couple hours ago21:34
fungi(or rather a possible fix to a problem we spotted)21:34
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: [WIP] admin REST API: docker-compose PoC, frontend  https://review.openstack.org/64353621:38
*** jamesmcarthur has joined #zuul21:45
*** hashar has quit IRC21:46
*** jamesmcarthur has quit IRC21:50
openstackgerritJames E. Blair proposed openstack-infra/zuul-sphinx master: Run opendev-release-python on release  https://review.openstack.org/64945621:54
corvusmhu: that change reminds me that we need to upgrade zuul's docker jobs to the new system which respects depends-on21:55
corvusmhu: i'll do that real quick, then your stack should be fully self-testing without needing to modify build contexts21:56
corvusmhu: oh, er, actually i'm a little confused -- the quick-start job always builds all the images from scratch... so why are you setting the build attribute in docker-compose?22:02
openstackgerritMerged openstack-infra/zuul-jobs master: Exclude linux_x86_64 wheels from upload to PyPI  https://review.openstack.org/64945522:19
*** jamesmcarthur has joined #zuul22:25
openstackgerritJames E. Blair proposed openstack-infra/nodepool master: Use opendev docker jobs  https://review.openstack.org/64946522:28
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Use opendev docker jobs  https://review.openstack.org/64946622:28
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Use the buildset registry in zuul-quick-start  https://review.openstack.org/64946722:28
openstackgerritJames E. Blair proposed openstack-infra/nodepool master: Use the buildset registry in zuul-quick-start  https://review.openstack.org/64946822:30
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Use the buildset registry in zuul-quick-start  https://review.openstack.org/64946722:31
*** rlandy has quit IRC22:49
*** jamesmcarthur has quit IRC22:50
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/nodepool master: Implement max-servers for AWS driver  https://review.openstack.org/64947423:02
SpamapSFYI ^^ I had a *really* hard time getting any tests to work23:02
SpamapSI don't really understand the way zk works with unit tests23:02
SpamapSShrews: ^ maybe you can explain to me how we could test that the quota bits work there?23:03
corvusSpamapS: i think you have to have a zk running (could be in a docker image)23:05
corvusi think stephenfin has started some tox-docker changes to make that easier23:05
corvushttps://review.openstack.org/649041 not ready for prime time23:06
SpamapSI have zk running23:12
SpamapSwhat happens is, if I add a second request, nothing ever handles it. It remains in 'pending' state forever.23:13
SpamapSAnd I don't know if that's just normal.. or there's something I have to kick to make the provider handle it.. or what23:13
SpamapSI was thinking I'd like to test adding a node that would push things over quota and making sure it got rejected by the provider.23:14
*** jamesmcarthur has joined #zuul23:14
SpamapSTO do that I need one active request with a node, so the second one would see 1 running, with a max-servers of 1, and in theory.. the provider would reject it. But instead it just stays pending forever.23:15
corvusSpamapS: oh sorry, i misunderstood23:17
SpamapSoh wait.. hm23:18
corvusSpamapS: i would expect a test like you describe to work23:18
SpamapSreading the actual code23:18
SpamapSWe dont' reject when hasRemainingQuota returns false23:18
SpamapSwe "pause"23:18
SpamapSand that seems to be an internal only state, so the test will have to peek inside NodeRequestHandler to verify that23:20
SpamapScorvus: I think you were just my rubber ducky23:20
openstackgerritJames E. Blair proposed openstack-infra/zuul-jobs master: Fix error in pull-from-intermediate-registry  https://review.openstack.org/64947523:22
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/nodepool master: Implement max-servers for AWS driver  https://review.openstack.org/64947423:38
SpamapScorvus: indeed, quack has been the knack.. ^^ now with test coverage!23:38
openstackgerritMerged openstack-infra/zuul-jobs master: Fix error in pull-from-intermediate-registry  https://review.openstack.org/64947523:39
corvusSpamapS: quack i could quack!23:41
*** jamesmcarthur has quit IRC23:43
*** jamesmcarthur has joined #zuul23:43

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!