Tuesday, 2017-10-24

SpamapSjeblair: yeah I suspect it may not be talking to that project right just yet.00:00
jeblairSpamapS: ah.00:00
SpamapSI did just verify zuul's user has write access00:00
SpamapSso not 100% sure yet00:00
jeblairSpamapS: if there is no config for that project, it won't report00:00
jeblairSpamapS: you can get it to do so by adding a project stanza in a config project with "check: {jobs: []}"00:01
jeblairSpamapS: then zuul thinks that project is attached to check and should report config syntax errors on the initial in-repo add.00:01
SpamapSpabelanger: thanks that worked00:03
SpamapShttps://photos.app.goo.gl/IXr5GfCWrMUWxheG200:03
jlkSpamapS: I hadn't seen that, but Ansible's ansibot author says he's seen it00:03
SpamapSjeblair: it's reporting to the PR now, so maybe the old zuul.yaml was bad syntax'd too?00:03
pabelangerSpamapS: yay00:03
SpamapSjlk: my VM is running in the same DC as the github enterprise, so latency is crazy low00:03
jlkhuh, maybe we need to introduce some delay?00:04
SpamapSjlk: or some kind of cache invalidating00:04
jlkYou'd think the etag would change00:04
SpamapSlike maybe there's an if-none-match or if-modified-since we can add00:04
SpamapSI haven't looked at what we send for etag00:05
jlkthe etag is supposed to be an indicator that something has changed00:05
SpamapSso we're sending if-non-match already00:05
SpamapSnone00:05
jlkif we get the notice before the API has the event, I gotta wonder if that's a bug on the GHE side00:05
SpamapSjeblair: ah yeah, it wasn't in any pipelines yet.. so that makes sense.00:05
SpamapSjlk: Or maybe we're getting it _while the transaction is still uncommitted_00:06
jlkyeah, that seems pretty poor from GHE00:06
SpamapSlike, with txn: { send_notifications; commit_stuff }00:06
SpamapSinstead of getting the hooks after00:07
jlkthat seems dangerous00:07
SpamapSYeah it doesn't sound right. I'm guessing it has to do with etags.00:08
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: experiment with late-binding inheritance  https://review.openstack.org/51135200:13
SpamapSjlk: the rate limiting work you did was all on the app side, eyah?00:16
SpamapSyeah00:16
jlkyeah.00:17
jlknon-app chews through API faster00:17
jlkwas hoping that graphql would come online soon, and be available to apps00:17
jlkstill missing ONE API call for fetching thus far, aside from, you know, ALL the calls being available to apps.00:18
SpamapSour version of GHE doesn't have apps yet anyway00:18
jlkyeah, that's coming in the next release I believe00:18
SpamapSThat's what I hear00:18
jlkoooh! I got a webhook sent through, only my request signature is wrong. Hrm.00:28
*** artgon has joined #zuul00:30
*** artgon has left #zuul00:31
jlkoh sweet hot goodness. Got past that, only to have my client access token not work. Interesting.00:31
jlkdisconcerting. Github doesn't want to auth me.00:51
jlkah, works as a user, just not an app. I'm not sure I ever had that working.00:59
ianwcan you use with_items / with_nested on a job definition?04:32
ianwhttps://review.openstack.org/#/c/512450/15/.zuul.d/jobs.yaml was what i was trying04:34
*** bhavik1 has joined #zuul04:42
ianwno being the answer, as i guess configloader doesn't grok that04:44
tobiashianw: the job definition is before ansible so that doesn't work04:58
ianwtobiash: yeah, i thought we might have copied the behavior.  might be fun to play with one day05:01
*** dmellado has quit IRC05:21
*** dmellado has joined #zuul05:22
*** bhavik1 has quit IRC05:31
*** dmellado has quit IRC05:37
*** dmellado has joined #zuul05:40
*** yolanda has joined #zuul05:44
openstackgerritAndreas Jaeger proposed openstack-infra/zuul-jobs master: Silence ansible-lint  https://review.openstack.org/51452606:29
openstackgerritAndreas Jaeger proposed openstack-infra/zuul-jobs master: Silence ansible-lint  https://review.openstack.org/51452606:37
*** yolanda has quit IRC06:40
*** hashar has joined #zuul07:42
*** yolanda has joined #zuul07:46
SpamapShrm.. Depends-On not working in github.07:47
SpamapSoh n/m, they check the body of the _pr_, not the commits inside it07:53
SpamapShrm.. and it leaves the gate status pending for some reason08:16
openstackgerritKien Nguyen proposed openstack-infra/zuul-jobs master: [DNM] Fix generic process-test-results role  https://review.openstack.org/51455908:23
*** electrofelix has joined #zuul08:44
openstackgerritJens Harbott (frickler) proposed openstack-infra/zuul-jobs master: Follow redirects when triggering readthedocs build  https://review.openstack.org/51457008:58
*** sambetts|afk is now known as sambetts09:43
*** yolanda has quit IRC10:55
Shrewsclarkb: thx for the comments on 512637. left you a reply, but the short answer is "not going to change anything about the jobs themselves until after successful migration"11:28
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: Create job-output.txt together with JobDir  https://review.openstack.org/51461711:50
tobiashthat fixes the 'Log not found for build ID 94f4e27243a44e73a19808c328499a28' when opening the live log stream too early ^11:51
kklimondadmsimard: as I said on #ansible, I have a more of a workflow question - you can see a job I've written that's used to "validate" yaml files for any parser errors: https://signal.klimonda.com/single-task/11:58
kklimondabut It's unreadable in this format11:58
kklimondadmsimard: so I had this brilliant idea that I could use "include_tasks" and have automatic per-yaml file task like here: https://signal.klimonda.com/multiple-tasks-ignored-errors/11:59
dmsimardkklimonda: you'd prefer to have one task per file or something like that ?11:59
kklimondabut now ansible and ARA have no way of reporting back to me what failed11:59
dmsimardah, that's that I was going to suggest11:59
kklimondaso I started thinking that perhaps, in my first example, ARA should be reporting status on per-item basis when task is being looped over (by using with_items)12:00
dmsimardkklimonda: in 2.4 there's include_tasks and import_tasks, each have a different behavior12:01
dmsimardand by the way, Zuul doesn't provide 2.4(.1) yet12:01
dmsimardkklimonda: can you try import_tasks instead of include_tasks, out of curiosity ?12:01
dmsimardI believe import_tasks is the equivalent of 2.3 'include' with 'static: no'12:01
kklimonda"ERROR! You cannot use loops on 'import_tasks' statements. You should use 'include_tasks' instead."12:02
dmsimardhm12:03
dmsimardstruggling to understand the difference, haven't played a lot with the new stuff in 2.4 a lot yet tbh12:04
dmsimardkklimonda: in your multi-task example, the error is showing up as if you had ignore_errors: yes https://signal.klimonda.com/multiple-tasks-ignored-errors/result/857eb9fa-bacb-4c0d-be74-8173e842d6cd/12:05
kklimondaI'm not even sure if that approach has merit, perhaps a better way would be to generate txt/html report, upload it to the logs server, and make the task generate the report and fail if need be12:05
kklimondayes, I do have ignore_errors: yes here12:05
kklimondahttps://signal.klimonda.com/multiple-tasks-with-errors/12:05
kklimondathis one doesn't have ignore_errors, but in that case ansible aborts prematurely12:05
kklimonda(well, not from ansible's point of view)12:06
dmsimardkklimonda: oh, what's the problem then ? sorry I'm not quite awake yet :D12:06
kklimondadmsimard: so I'd like ARA to generate a report where I can see parsing status (FAILED/OK) on a per-file basis12:06
kklimondabut I just don't think that's possible, and that begs the question if I'm moving in the right direction - perhaps I should be validating those files in a single task, generating a report out of that run and have ansible pass/fail based on the result12:08
dmsimardhmm, just thinking out loud here12:09
dmsimardbut maybe register the result of your single task, iterate over it to print a line "file: status"12:10
dmsimardlet me get out some pseudocode12:13
dmsimardkklimonda: that surely doesn't work but http://paste.openstack.org/raw/624460/12:15
dmsimardor use a template file, something, dunno12:16
dmsimard¯\_(ツ)_/¯12:16
* dmsimard off to get caffeine12:16
kklimondadmsimard: oh wow, can you use templates like that?12:39
dmsimardkklimonda: inline jinja? sure http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/multi-node-firewall/tasks/main.yaml12:40
*** yolanda has joined #zuul12:56
kklimondadmsimard: ara_report is failing with MODULE FAILURE, I had to add ignore errors to the validate yaml task as it wouldn’t run next tasks anyway13:25
kklimondaI’ll debug it some more once I get home13:25
kklimondaWhat would be a result anyway? Does ARA render variables set with ara_record differently?13:26
openstackgerritDavid Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Add log streaming logging and exception handling  https://review.openstack.org/51381113:30
Shrewsugh. foiled yet again by py 3.5 vs 3.6 changes13:30
dmsimardkklimonda: it's just a way to associate arbitrary data with your report, see example here (expand "records") http://logs.openstack.org/92/511992/9/check/legacy-ara-integration-py27-latest-centos-7/e6c8e23/logs/build-playbook/13:47
dmsimardkklimonda: http://ara.readthedocs.io/en/latest/usage.html#using-the-ara-record-module13:47
dmsimardkklimonda: note that it requires the ara modules to be in the ansible library path -- which isn't the case for zuul v313:58
clarkbShrews: I think the dropping of the "legacy" in the job names had me confused as to what the goal was there. I was reading it as migrate jobs to nodepool and zuulify them14:52
clarkbbut I guess that is because you can't have jobs with the same name defined in multiple places (so job name had to change)14:52
Shrewsclarkb: ah, no. sorry, should have probably been more explicit with the commit message. just a simple move into the nodepool repo. the renaming is following the "job rename" guidelines14:53
Shrewshttps://docs.openstack.org/infra/manual/drivers.html#naming-with-zuul-v314:54
openstackgerritMerged openstack-infra/nodepool master: Migrate legacy jobs  https://review.openstack.org/51263715:10
Shrewsjeblair: a job defined in branch A of a project should be able to be referenced in branch B of the same project, right?15:25
Shrewsbranch A == master, in this scenario15:25
jeblairShrews: yes15:32
Shrewsjeblair: ok. i must have something else wrong then.15:34
jeblairShrews: not necessarily.  things are weird with branch stuff right now.15:35
jeblairShrews: that's what i'm working on.15:36
Shrewsjeblair: ok. https://review.openstack.org/513766 keeps failing on the situation i described. no logs, just "error" from the referenced jobs15:37
Shrewsjeblair: i'll be patient as you dig into such black magic and assume this to be a zuul bug15:38
jeblairShrews: i think you should copy that job and those playbooks into the feature branch15:38
Shrewsjeblair: yeah, i can do that. that was going to be my plan if you answered "no"15:39
jeblairShrews: i think that will work now and in the future (but not 100% on that yet)15:39
Shrewsjeblair: same job names? or should i modify them?15:40
jeblairShrews: same15:40
* Shrews tries15:40
jeblairShrews: you might then run up against another bug or two, which are the things i'm working on now.15:40
Shrewsneat15:41
jeblairi will not have a solution for those.15:41
jeblairyet.15:41
Shrewsunderstood15:41
SpamapSmmmm Zuul doing work for me.15:43
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Migrate legacy jobs for feature/zuulv3 branch  https://review.openstack.org/51376615:44
SpamapSneed to figure out this github labels issue though15:44
SpamapSI _think_ we might just have to add a small delay handling webhooks15:44
pabelangerSpamapS: awesome! Have you tried using any of zuul-jobs roles yet? Would love to see another zuul using them and giving feedback. I know jamielennox tried on bonny15:53
*** yolanda has quit IRC15:54
*** hashar is now known as hasharAway15:54
tobiashpabelanger: I started today with a new setup using zuul-jobs15:56
SpamapSpabelanger: I'm using a couple yes.15:56
* SpamapS looks15:56
pabelangerYay15:56
pabelangerthat is awesome :)15:56
tobiashone thing I learned that the cached git handling is a bit scattered throughout project-config and zuul-jobs15:57
tobiashmaybe it is possible to solve this in a generic way directly in zuul-jobs15:58
SpamapSpabelanger: I plan to contribute some as well. :)15:58
tobiashbut I have to think about more deeply15:58
SpamapSCurrently working on a thing that validates ansible playbooks for people.15:58
*** jeblair has quit IRC16:00
*** jeblair has joined #zuul16:01
SpamapSand kubernetes resources as well16:03
SpamapSpabelanger: are there any specific roles in there you wish people used more?16:08
SpamapSI'm not likely to setup AFS any time soon :)16:08
SpamapSBut I do want to move my logs off my all in one zuul and into object storage at some point.16:08
pabelangerSpamapS: nothing specific, mostly just want to get some more coverage.  Something like prepare-workspace should work for everbody16:09
jeblairSpamapS: zuul-jobs certainly does not require afs16:12
jeblairSpamapS: all of the roles *and jobs* in zuul-jobs are meant to be reusable by anyone, so please do :)16:13
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix branch ordering on dynamic reconfiguration  https://review.openstack.org/51474416:27
*** sambetts is now known as sambetts|afk17:00
jeblairclarkb: ^ can you review that?17:04
jeblairtobiash, SpamapS: ^ you also reviewed the earlier related change17:04
* tobiash looking17:07
clarkbnow that I am looking at this change and the original one again I wonder how safe editing a list is when iterating it17:11
jeblairclarkb: sorted returns a copy17:12
clarkbjeblair: right but you assign that copy to branches then iterate over branches and modify branches in that iteration17:13
jeblairclarkb: where do i modify branches in the iteration?17:13
clarkbjeblair: it looks like prepending (like in your changes) is safe17:13
clarkboh thats an if17:14
clarkbnevermind17:14
clarkbI need more caffeine17:14
jeblairwhew17:14
clarkbshould've just trusted my previous review more :)17:14
SpamapSpabelanger: yeah, prepare-workspace is working fine17:18
SpamapSas are the SSH key pieces17:18
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Fix branch ordering on dynamic reconfiguration  https://review.openstack.org/51474417:27
jeblairi sent this email about checking out tags: http://lists.openstack.org/pipermail/openstack-infra/2017-October/005631.html17:28
openstackgerritMerged openstack-infra/zuul-jobs master: Follow redirects when triggering readthedocs build  https://review.openstack.org/51457018:02
Shrewsjeblair: fyi, your suggestion to just copy the master branch jobs to the feature branch worked without any hiccups in https://review.openstack.org/51376618:03
Shrewspabelanger: clarkb: if you both could re-review that (513766) when you get a moment, we'll have the feature/zuulv3 properly testing again18:05
jeblairShrews: cool, don't look too hard :)18:08
SpamapSso, I think my github problem is in fact just that my GHE is so close to my zuul.18:09
SpamapStesting with a delay whenever receiging label updated events now18:09
*** electrofelix has quit IRC18:23
*** __zeus__ has joined #zuul18:31
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: DNM: test bash script  https://review.openstack.org/51478818:35
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: DNM: test bash script  https://review.openstack.org/51478818:40
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: DNM: test bash script  https://review.openstack.org/51478818:45
pabelangerShrews: +318:46
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: DNM: test bash script  https://review.openstack.org/51478818:48
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Migrate legacy jobs for feature/zuulv3 branch  https://review.openstack.org/51376618:56
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: DNM: test bash script  https://review.openstack.org/51478819:02
dmsimardI have a question that confuses my brain. Is it a bad assumption that Zuul pipeline precedence does not take into account nodepool at all ?19:14
dmsimardMy problem is the following: I have pipelines A B C (served by nodepool region #1) and X Y Z (served by nodepool region #2), if I set pipeline Z on a high precedence, will it prevent jobs from A B and C from being enqueued even though there could be available capacity ?19:15
mrhillsmanif they are being served by two separate regions why would it matter?19:16
mrhillsmanregion1 could care less about region2 jobs and vice versa no?19:17
mrhillsmani am a novice so pardon my response if too elementary19:18
dmsimardmrhillsman: zuul pipeline precedence makes it so if you have a pipeline set to 'high', no job with a 'normal' precedence pipeline start until all the jobs in 'high' have been served19:18
dmsimardmrhillsman: this is why there is a backlog of jobs in the 'post' pipeline in openstack sometimes, it's because it has a lower priority than, for example, 'gate'19:19
Shrewsdmsimard: priority changes which requests are processed first by nodepool. if they are separate nodepool processes, one won't affect the other because region#1 will decline any requests that can only be served by region#219:20
Shrewsand vice versa19:20
dmsimardShrews: they're not separate nodepool processes, it's one zuul and one nodepool (v2)19:20
Shrewsdmsimard: separate pool definitions?19:20
dmsimardShrews: yeah, like OVH and RAX are two different clouds19:21
Shrewsi'd have to assume "yes". same concept. one pool will decline requests only the other can server19:21
Shrewsserve19:21
dmsimardShrews: basically my pipeline A B C is served out of OVH, my pipeline X Y Z is served out of RAX.. if I have a 'high' precedence on Z, would it prevent nodepool from serving OVH nodes to A B and C19:22
dmsimardShrews: ok, I guess we can try it and see what happens..19:22
Shrewsdmsimard: no. each pool handles requests independently (one thread per pool)19:22
dmsimardShrews: that's true for v2 as well ?19:22
Shrewsdmsimard: oh, no. v2 is totally different and soooo 201619:23
Shrews:)19:23
dmsimardSorry I'm not in october 2017 yet19:23
dmsimardWe'll be in october sometime late next month probably :)19:23
Shrewsdmsimard: apologies then. thought we were discussing v3. ignore what i said :)19:23
*** rcarrillocruz has quit IRC20:08
*** rcarrill1 has joined #zuul20:08
*** __zeus__ has quit IRC20:26
SpamapShrm.. still not getting a success status when gate passes.20:32
SpamapSoh n/m .. weird20:32
SpamapSmust have been a permissions issue20:32
pabelangerdmsimard: are they the same labels (for images) in A B C and X Y Z?20:53
*** hasharAway has quit IRC20:53
dmsimardpabelanger: in this case, no20:53
dmsimardpabelanger: in RDO context, it's the RDO and the TripleO tenants on RDO cloud which are two different "clouds"20:54
pabelangerdmsimard: okay, so yoi should be fine20:54
dmsimardthey might want to set the periodic jobs to have a 'high' priority, I just want to make sure that doesn't screw up the RDO gate.20:54
pabelangerIf the same label is across your clouds, then it might be possible for Z to starve resources in other pipelines20:55
dmsimardpabelanger: they happen to use a different label/image so we should be fine then20:55
pabelangerdmsimard: this is how tripleo-test-cloud-rh1 works today, if gate pipeline is full of jobs, nodepool will still process tripleo-centos-7 images properly20:57
dmsimardpabelanger: great20:57
*** fdegir has joined #zuul21:12
dmsimardthe madness that is merging things in openshift-ansible in a world without gerrit and zuul https://openshift-ansible-sq-status-ci.svc.ci.openshift.org/#/info?prDisplay=4256&historyDisplay=425621:44
dmsimardOops, that URL was filted to my PR, this one isn't: https://openshift-ansible-sq-status-ci.svc.ci.openshift.org21:46
pabelangerdmsimard: that looks a lot like kubernetes bot, I wonder if they pulled that downstream to use21:49
dmsimardpabelanger: that's what they use for origin and openshift-ansible so maybe21:50
pabelangeryah, seems to be the same21:50
dmsimardThe one for origin is https://origin-sq-status-ci.svc.ci.openshift.org/21:50
dmsimardneed to introduce them to zuul and gerrit :)21:51
pabelangerWas just in sig-testing meeting for kubernetes today, they are officially moving off jenkins into prow21:51
dmsimardpabelanger: off jenkins to what ?21:51
pabelangerprow21:51
dmsimardwth is prow21:51
pabelangerthey build the tool themself21:51
* dmsimard google21:51
dmsimardhttps://github.com/kubernetes/test-infra/tree/master/prow21:52
pabelangeryup21:52
dmsimardeh https://prow.k8s.io/#21:52
dmsimardpabelanger: cmd/jenkins-operator is the controller that manages jobs running in Jenkins.21:54
dmsimardpabelanger: are they at the zuul v2 stage where they're using jenkins as a dumb runner ?21:54
pabelangeryah, they are at the point where they can remove it now21:55
pabelangerand implement there own task execution21:55
dmsimardokay21:56
pabelangerThe interesting piece of info I learned today, is google wants to decrease their commitment to the sig-testing group, both humans and hardware. So there is an effort to have the CNCF be more involved in running / manging testing.  Basically start moving things into AWS21:56
jeblairi wrote an email about some stable branch issues: http://lists.openstack.org/pipermail/openstack-infra/2017-October/005634.html22:26
SpamapSI just discovered an amazing paradigm for testing stuff23:24
SpamapSI may need to encode this into a tool23:24
SpamapSI created an entire fake zuul yaml file23:24
SpamapSwith two projects23:24
SpamapSand now I can run  ANSIBLE_ROLES_PATH=thoserepos/roles ansible-playbook -i localhost, -e @thatfile playbooks/foo.yaml23:25
SpamapSand it like, works23:25
SpamapSI can test my zuul playbooks before I push them up23:25
pabelangeryah, I've been doing that locally lately. Works pretty well23:28
SpamapSThere are just times where I need to work out the kinks before waiting 5 - 20 minutes to find out if it works :)23:29
pabelangeryah, also working on https://review.openstack.org/512715/ for a shared linter / syntax-check job across all our projects. Plan on iterating on that, maybe drop tox as a dependency. But uses ANSIBLE_ROLES_PATH23:31
pabelangerI should see how to do that in 2.4, since that apparently changed23:32
*** weshay|ruck is now known as weshay|PTO23:38
SpamapSHrm, so when a Depends-On'd PR is merged, the dependent PR's aren't automatically sent into gate for some reason.23:52
jlkthis is for github?23:54
SpamapSjlk: yeah23:54
jlkIt should have an idea of changes that are dependent23:54
jlkthere's a spot in the code that mucks with that.23:55
jlkcrap, I have to run.23:55
SpamapSdo they have to share queues?23:56
SpamapS2017-10-24 16:52:16,521 DEBUG zuul.Pipeline.GoDaddy.gate:   Change <Change 0x7f64fc181278 3,af5ede37fa61a409376f5ae157e6c6b9a4964913> in project cloudplatform/godaddy-zuul-jobs does not share a change queue with <Change 0x7f64fc1e0278 18,3d019b588b29f1d15d1c08efde10d70af24f6e9c> in project cloudplatform/k8s-addons23:56
jlkoh hrm.23:56
jlkmaybe?23:56
SpamapS2017-10-24 16:52:16,521 DEBUG zuul.Pipeline.GoDaddy.gate: Failed to enqueue changes ahead of <Change 0x7f64fc1e0278 18,3d019b588b29f1d15d1c08efde10d70af24f6e9c>23:56

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!