| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Cleanup pipeline requirements https://review.openstack.org/487618 | 00:00 |
|---|---|---|
| jeblair | clarkb: updated; thanks ^ | 00:00 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove ZUUL_CHANGES https://review.openstack.org/486245 | 00:21 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove ZUUL_BRANCH https://review.openstack.org/486246 | 00:24 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove ZUUL_VOTING and add zuul.voting https://review.openstack.org/486247 | 00:24 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove ZUUL_URL https://review.openstack.org/486249 | 00:24 |
| *** dmsimard has quit IRC | 01:03 | |
| *** jamielennox has quit IRC | 01:08 | |
| *** jamielennox has joined #zuul | 01:15 | |
| *** dmsimard has joined #zuul | 01:17 | |
| *** harlowja has quit IRC | 01:19 | |
| *** jkilpatr has quit IRC | 01:20 | |
| *** jamielennox has quit IRC | 02:17 | |
| *** jamielennox has joined #zuul | 02:24 | |
| *** harlowja has joined #zuul | 03:31 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove extra GC debug info https://review.openstack.org/487622 | 03:48 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove ZUUL_PIPELINE https://review.openstack.org/486250 | 03:50 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove ZUUL_PROJECT https://review.openstack.org/486251 | 03:56 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove ZUUL_UUID https://review.openstack.org/486252 | 03:56 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Case sensitive label matching https://review.openstack.org/469946 | 04:09 |
| jeblair | tobiash, SpamapS: i think i have a log from the test_periodic_override hang. i don't understand it yet, but i'll look closer tomorrow | 04:09 |
| *** harlowja has quit IRC | 04:52 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Case sensitive label matching https://review.openstack.org/469946 | 05:50 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Use correct label casing in tests https://review.openstack.org/487703 | 05:50 |
| *** amoralej|off is now known as amoralej | 07:04 | |
| *** hashar has joined #zuul | 07:59 | |
| *** yolanda has quit IRC | 09:11 | |
| *** yolanda has joined #zuul | 09:12 | |
| *** kklimonda has joined #zuul | 09:38 | |
| *** jkilpatr has joined #zuul | 11:10 | |
| *** smyers has quit IRC | 11:27 | |
| *** amoralej is now known as amoralej|lunch | 12:35 | |
| *** AJaeger has joined #zuul | 13:23 | |
| AJaeger | FYI, I'm proposing a naming for Zuulv3 jobs as discussed in the last infra meeting: ttps://review.openstack.org/487848 | 13:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Collect logging information into ara callback https://review.openstack.org/487853 | 13:32 |
| *** amoralej|lunch is now known as amoralej | 13:34 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Collect logging information into ara callback https://review.openstack.org/487853 | 13:43 |
| mordred | pabelanger, dmsimard: ^^ that's what I was thinking for ara integration | 13:48 |
| mordred | dmsimard: I have an ara feature request - when we execute a playbook in zuul v3, the filesystem path it's in is ... ugly | 13:50 |
| *** dkranz_ has joined #zuul | 13:51 | |
| dmsimard | mordred, pabelanger: so in the spec I've ended up not re-writing because ENOTIME https://review.openstack.org/#/c/444088/ | 13:51 |
| mordred | dmsimard: we pass in a value to use in logging for our other callback that has a cleaned up path to report for the playbook | 13:51 |
| dmsimard | (Which was also discussed during a Zuul meeting) | 13:51 |
| mordred | dmsimard: it would be neat if we could maybe set an env var or something and have the ara callback also pick up the same thing | 13:52 |
| dmsimard | We sort of had consensus on not tight-coupling Zuul and ARA together but making it easy for users to enable it in their jobs if they needed | 13:52 |
| dmsimard | Or enabling callbacks in general | 13:52 |
| dmsimard | jeblair commented an example on the spec review of what this might look like, I don't know if this still translates well today | 13:53 |
| dmsimard | mordred: I'm not sure I understand your question | 13:54 |
| dmsimard | mordred: you mean the file tab ends up ugly ? | 13:54 |
| mordred | dmsimard: jeblair's comment can still apply, but I think writing the plumbing for that is going to take a little longer - so between now and denver I think just hardcoding it in, then refactoring it out before final v3 release will get us further | 13:55 |
| mordred | dmsimard: (I'm mostly pushing on this right now because I'm about to start poking at devstack-gate jobs which use ara - and without this there's no way to get it enabled in those jobs :) ) | 13:56 |
| mordred | dmsimard: for the other thing ... | 13:56 |
| mordred | dmsimard: if you look at: http://logs.openstack.org/39/487539/5/check/tox-py35/135ad35/job-output.txt.gz | 13:56 |
| mordred | you'll see it lists the playbook being run as git.openstack.org/openstack-infra/project-config/playbooks/base/pre | 13:57 |
| mordred | then as git.openstack.org/openstack-infra/zuul-jobs/playbooks/unittests/pre | 13:57 |
| mordred | etc | 13:57 |
| mordred | that's not ACTUALLY their real filesystem path | 13:57 |
| mordred | their real filesystem path is like /tmp/27255ad449624904bbf745fd2706ddf7/trusted/project_0/git.openstack.org/openstack-infra/project-config/playbooks/base/pre.yaml | 13:58 |
| mordred | but the /tmp dir and project_0 and other bits are impl details and super not-interesting to an openstack develop | 13:59 |
| mordred | developer | 13:59 |
| mordred | dmsimard: so we basically set a variable with a "friendly" name for th eplaybok that we use in logging to override the playbook name that's found in the playbook object | 13:59 |
| mordred | dmsimard: anyway - my 'feature request' is to be able to set a similar value - hopefully via an env var - and have ara use that in it's view of where a given playbook file lives | 14:01 |
| mordred | dmsimard: it's possible that doesn't make sense, which is fine | 14:01 |
| dmsimard | mordred: hm, ultimately ara picks up whatever ansible throws at it -- if the playbook path is foobar, then it picks up foobar. Are you using the vanilla ansible-playbook command or are you overriding it (like, say, the 'command' module) ? | 14:01 |
| *** smyers has joined #zuul | 14:02 | |
| dmsimard | mordred: probably silly but "ln -s /some/ugly/path/playbook.yml git.o.o/playbook.yml; ansible-playbook git.o.o/playbook.yml" would probably work too | 14:03 |
| mordred | dmsimard: we use the vanilla ansible-playbook command - and we started by just picking up what ansible threw at it | 14:03 |
| dmsimard | mordred: are the roles in similarly ugly paths ? | 14:03 |
| mordred | yup | 14:04 |
| mordred | I mean - honestly, I think having the report will be a large enough win that it's less important that we solve this any time in the short-term | 14:04 |
| Shrews | anyone know why a tox run would report "db type could not be determined" ? | 14:04 |
| mordred | I think the 'ugly' path won't be as much of a problem in the ara web ui | 14:04 |
| mordred | Shrews: yes | 14:05 |
| mordred | Shrews: it's due to py2 vs. py3 impls of the dbm module - usually from running testr with py2 before py3 | 14:05 |
| mordred | Shrews: just rm -rf .testrepository | 14:05 |
| dmsimard | mordred: yeah, like I said if you want a "clean" path, you can always substitute with symlinks and set a proper ANSIBLE_ROLES_PATH or something | 14:05 |
| Shrews | mordred: danke | 14:05 |
| dmsimard | mordred: added some comments in your review | 14:06 |
| mordred | dmsimard: sweet - thanks! | 14:06 |
| dmsimard | mordred: also it seems like ara is installed late in the process (it's installed by ansible itself?) so you're ultimately missing out on some bits that happen beforehand | 14:07 |
| dmsimard | probably not a big deal, but worth mentioning | 14:08 |
| mordred | dmsimard: I'm not sure what you mean by that? with this patch ara is installed alongside zuul itself | 14:09 |
| mordred | dmsimard: and added to the ansible.cfg before we run any ansible | 14:09 |
| dmsimard | mordred: that's some ansible running things: http://logs.openstack.org/53/487853/2/check/tox-cover/ed12ee2/job-output.txt.gz#_2017-07-27_13_43_45_450535 | 14:10 |
| dmsimard | and then later ara is installed: http://logs.openstack.org/53/487853/2/check/tox-cover/ed12ee2/job-output.txt.gz#_2017-07-27_13_45_53_535161 | 14:10 |
| mordred | dmsimard: right. that's because that's a tox test of zuul itself | 14:10 |
| mordred | dmsimard: if the zuul running that tox test had this patch applied, the playbook that runs tox would be running with an ara installed | 14:11 |
| dmsimard | ahhh, makes sense | 14:11 |
| mordred | but then on the test node the tox test will install ara | 14:11 |
| mordred | dmsimard: it's turtles all the way down around here | 14:11 |
| dmsimard | mordred: oh, for FWIW, one of the things I'm waiting on for the next release is this thing: https://review.openstack.org/#/c/480716/ | 14:19 |
| pabelanger | dmsimard: mordred: left comment / questions on 487853 | 14:19 |
| dmsimard | So ideally we'll be able to export subunit from ARA | 14:19 |
| dmsimard | and then ship it to openstack-health or something | 14:19 |
| mordred | dmsimard: neat | 14:20 |
| mordred | pabelanger: looking | 14:21 |
| mordred | pabelanger: well - if we did it in a trusted playbook we'd be installing ara and its depends every job, and we'd be having a playbook modify the ansible.cfg - and we'd miss the logging for the playbook that had the installation of ara | 14:22 |
| pabelanger | mordred: right, chicken and egg. | 14:23 |
| mordred | pabelanger: which is why I think having it directly in zuul gets us where we need to be - I do agree that making it optional is important, but I don't want to fall down that rabbithole too much as I mostly just want to be able to refactor devstack-gate jobs and not lose the ara :) | 14:23 |
| mordred | pabelanger: I think we should _definitely_ clean that up to make it optional for an admin to install before we cut a v3 release though | 14:24 |
| mordred | pabelanger: that said - I'm about ready to start hammering on more jobs with you (sorry, asia trip put a damper on my ability to help wit that task) | 14:24 |
| mordred | pabelanger: thanks for getting the shade jobs started going! | 14:25 |
| pabelanger | ya, longer term, I think parsing job-output.txt would be an option too, for optional install. | 14:25 |
| mordred | pabelanger: yah - I agree | 14:25 |
| pabelanger | mordred: np! I think I'm going to look at github connection stuff this morning | 14:25 |
| mordred | in fact - that would also solve the thing I was mentoined ot dmsimard above which is that we rework the playbook name in our callback | 14:25 |
| mordred | so if ara did grow an 'import json' feature that would be neat | 14:26 |
| pabelanger | ++ | 14:26 |
| dmsimard | yeah, definitely something we can do | 14:26 |
| dmsimard | need some prior hacking first | 14:27 |
| dmsimard | I think async json importing is a better option for our context | 14:27 |
| Shrews | anyone know offhand if there is a pathway to get tenant and project names from a zuul Build object? | 14:30 |
| mordred | uhm | 14:33 |
| mordred | Shrews: I do not - but you saying that makes me realize the thing I've been wanting all this time | 14:34 |
| mordred | Shrews: which is an ER diagram | 14:34 |
| Shrews | mordred: yeah, that would be very helpful | 14:34 |
| Shrews | like... VERY | 14:34 |
| mordred | turns out that's how my brain works | 14:34 |
| mordred | and without actual data types in python... | 14:34 |
| mordred | jeblair: ^^ not super urgent, but next time we think about zuul developer/internals docs, I think something similar to an ER diagram for the model objects could be quite helpful. I have no useful suggestions on how to accomplish that | 14:35 |
| mordred | ooh! https://pypi.python.org/pypi/ERAlchemy <-- that's neat | 14:36 |
| Shrews | being a very visual learner, i'm very on board with such a document | 14:36 |
| mordred | also - once we're there - python 3.6 type annotations will make me a happy camper | 14:40 |
| *** hashar has quit IRC | 14:41 | |
| Shrews | i am saddened that my unit test has no nodes for autoholding fun :( | 14:50 |
| mordred | Shrews: boo | 14:53 |
| mordred | Shrews: also - it turns out I am a bad python person - we have type hints in 3.5 already | 14:54 |
| mordred | https://www.python.org/dev/peps/pep-0484/ - which also includes a suggested grammar for type hints in comments for variables | 14:54 |
| Shrews | oh neat | 14:56 |
| Shrews | i just tried it myself following https://docs.python.org/3/library/typing.html | 14:56 |
| Shrews | i (correctly) cannot greet an integer! | 14:57 |
| Shrews | TypeError: must be str, not int | 14:57 |
| Shrews | mordred: balancing the comment type hints with pep8 rules for line length will be fun | 15:00 |
| mordred | yah. 3.6 is where variable annotations come in | 15:01 |
| mordred | Shrews: how did you get it to TypeError? | 15:01 |
| Shrews | mordred: greeting(1) | 15:01 |
| Shrews | vs greeting("david") | 15:02 |
| Shrews | i typed it directly into the interpreter. i assume (would hope) the same happens in a .py file | 15:02 |
| mordred | hrm. I did the same and didn't get a TypeError | 15:03 |
| mordred | are you on 3.6 maybe? | 15:03 |
| pabelanger | mordred: jeblair: Do I need to restart zuul to pick up new github credentials or just reload? | 15:04 |
| pabelanger | I guess I can try a reload first | 15:04 |
| Shrews | mordred: oh, hrm. that's failing on the + operation | 15:04 |
| Shrews | mordred: 3.5 | 15:05 |
| Shrews | mordred: oops, nope. 3.6 | 15:05 |
| mordred | well - in any case, I like that we can use them | 15:06 |
| Shrews | mordred: if i change it to "return 'Hello %s' % name", it will not error at all. I don't understand now | 15:06 |
| mordred | I think it's just that they're informational so you can use static type checkers | 15:06 |
| Shrews | oh! | 15:06 |
| mordred | http://mypy.readthedocs.io/en/latest/ for instance, will do a static analysis pass of your code | 15:07 |
| jeblair | Shrews: Build.pipeline.layout.tenant | 15:08 |
| Shrews | jeblair: ah ha. i looked at Pipeline but missed that. thank you | 15:10 |
| jeblair | Shrews: regarding nodes -- you may need to make a new zuul config for that test with a job that requests nodes. easiest way is to stick the whole zuul.yaml in tests/fixtures/layouts/foo.yaml and then decorate the test with @simple_layout('foo.yaml') | 15:11 |
| Shrews | ok. thx again | 15:14 |
| *** dmsimard is now known as dmsimard|afk | 15:14 | |
| Shrews | jeblair: hints for getting at project name from Build? | 15:16 |
| jeblair | Shrews: yep. gotta get to the QueueItem because that holds that context | 15:17 |
| *** AJaeger has left #zuul | 15:17 | |
| jeblair | Shrews: Build.build_set.item | 15:17 |
| jeblair | mordred, dmsimard|afk: suggestion on 487853 to avoid the giant dep list pabelanger mentioned. | 15:18 |
| jeblair | mordred: unfortunately, it will *double* the number of lines of code in your patch. :) | 15:19 |
| jeblair | but at least it won't double our dependencies :) | 15:19 |
| pabelanger | pip install ara[callback-only] would work too | 15:19 |
| pabelanger | but like try/import too | 15:20 |
| jeblair | pabelanger: that seems like the thing we should add to our puppet then | 15:20 |
| pabelanger | ya | 15:20 |
| Shrews | jeblair: ah, found it. would have taken me forever to figure that out | 15:24 |
| mordred | jeblair: ohno! double the lines | 15:32 |
| mordred | jeblair: ah - yes. easy fix | 15:33 |
| clarkb | tobiash: thank you for the update to case sensitivity change I have +2'd | 15:47 |
| SpamapS | jeblair: any luck so far? | 15:50 |
| SpamapS | those "the process just died" bugs are awful | 15:50 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix race in test_periodic_override https://review.openstack.org/487917 | 15:54 |
| jeblair | SpamapS, tobiash: ^ yes, in fact! | 15:54 |
| jeblair | i could not see that yesterday, but it took me about 5 minutes today. this is why we stop work and go do other things. :) | 15:55 |
| SpamapS | brains like rest | 15:57 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: Simplify run tox task https://review.openstack.org/487551 | 16:08 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Don't request empty nodesets https://review.openstack.org/487243 | 16:10 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix race in test_periodic_override https://review.openstack.org/487917 | 16:14 |
| jeblair | mordred, SpamapS, pabelanger, fungi: i think it's decision time for tobiash's case-sensitive label change: https://review.openstack.org/469946 | 16:16 |
| jeblair | clarkb and I are +2 on that. mordred is +2 on a previous version. but it's a significant change, so i want to make sure folks are cool. | 16:17 |
| fungi | seems like it was necessitated at least by our change in database collation, unsure if it was gerrit 2.13-specific, right? | 16:18 |
| jeblair | fungi: that's my understanding | 16:18 |
| jeblair | fungi: could be one, the other, or both. | 16:18 |
| fungi | either way, seems like a reasonable change and 3.0 is the time to make such backward-incompatible breaking changes | 16:19 |
| clarkb | considering it is new behavior with postgres backend I think it may be gerrit delegating that behavior to the db whereas previously it may have normalized internally? Though I haven't found any evidence of that in the gerrit log | 16:19 |
| jeblair | (we don't have a convenient way to test <2.13 + binary collation, so we can't narrow it down further without spinning up another gerrit) | 16:19 |
| fungi | well, regardless, we have conclusive evidence that at least _some_ gerrit deployments need it | 16:19 |
| jeblair | clarkb: i agree that seems (somehow) likely. | 16:20 |
| fungi | no need to split hairs, i guess | 16:20 |
| clarkb | fungi: right I think that is the important info here. Some gerrits not our own have run into this | 16:20 |
| clarkb | fungi: and we should support them too | 16:20 |
| jeblair | fungi: yeah, that's kind of my thinking. long term, capitalizing and/or quoting some things in the config is going to annoy users less than hitting this error and us being like "oh, yeah, run gerrit differently" | 16:20 |
| mordred | jeblair: ++ | 16:21 |
| fungi | it's far from being the most significant behavior change in v3 ;) | 16:21 |
| mordred | jeblair: I went ahead and put a +2 on this version of the patch | 16:21 |
| clarkb | I'm going to work on testing zuulv2 today with tobiash's change on that branch just to be double sure both versions are happy | 16:22 |
| jeblair | i mean, it will annoy me Because Every Time I Hit The Shift Key I Swear. But I Wil Get Over It. | 16:22 |
| jeblair | clarkb: ++ | 16:22 |
| fungi | switch emacs into title caps mode (i'm sure it has one!) | 16:22 |
| jeblair | fungi: probably a minor mode | 16:23 |
| SpamapS | jeblair: I think I remember seeing it and thinking "a minor version did this?" | 16:24 |
| SpamapS | but I think I'm +2.. let me refresh, it's been a few weeks | 16:24 |
| clarkb | SpamapS: well they don't actually use any of this code base upstream aiui so its easy for these sorts of problems to get in | 16:24 |
| clarkb | SpamapS: they use a bunch of googly services that they patch in with guice (its the only reason to really have guice in gerrit) upstream | 16:25 |
| clarkb | git repos don't live on disk, they don't use a sql database server, etc | 16:25 |
| fungi | including proprietary google database backend apparently | 16:26 |
| SpamapS | clarkb: perhaps we should ask if we can gate for them.... ;-) | 16:26 |
| SpamapS | Google's relationship with open source and free software is so weird.. I want to trust them. I want to think they get it.. but...... every time I get below the surface... | 16:27 |
| SpamapS | added my (somewhat unfortunate) +2 ;) | 16:28 |
| jeblair | to be fair, a lot of the reviewdb work is so they can stop using their proprietary database and just use their, um, proprietary disk. | 16:28 |
| SpamapS | That makes sense | 16:30 |
| SpamapS | I recall reading the description and thinking "I thought that's already how it worked..." | 16:30 |
| clarkb | SpamapS: in gerrits case it seems to be tension between google production deployment requirements and what is feasible using open source software | 16:30 |
| clarkb | SpamapS: google requires N+2 region redundancy or something crazy and good luck doing that without their googly bits | 16:31 |
| SpamapS | Yeah | 16:32 |
| fungi | s/reviewdb/notedb/ but yeah | 16:34 |
| * SpamapS shutters at the similarity to Notes | 16:35 | |
| SpamapS | shudders? | 16:35 |
| SpamapS | either way | 16:35 |
| SpamapS | both | 16:35 |
| fungi | though the accountpatchreviewsdb is a stepping stone to notedb aiui | 16:36 |
| tobiash | o/ | 16:42 |
| jeblair | clarkb: want to go ahead and +3 https://review.openstack.org/487703 since it's a rebase magnet? | 16:57 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: Simplify run tox task https://review.openstack.org/487551 | 17:02 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Case sensitive label matching https://review.openstack.org/469946 | 17:05 |
| tobiash | yay | 17:06 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: Simplify run tox task https://review.openstack.org/487551 | 17:07 |
| clarkb | jeblair: will review | 17:07 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: Simplify run tox task https://review.openstack.org/487551 | 17:09 |
| *** harlowja has joined #zuul | 17:09 | |
| clarkb | and done | 17:11 |
| tobiash | :) | 17:11 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: Simplify run tox task https://review.openstack.org/487551 | 17:16 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Use correct label casing in tests https://review.openstack.org/487703 | 17:23 |
| jeblair | pabelanger, mordred, clarkb: so, erm, i reckon we need to go capitalize some things in our zuul config now :) i'll write a patch | 17:25 |
| mordred | jeblair: :) | 17:25 |
| jeblair | remote: https://review.openstack.org/487955 Capitalize gerrit labels in zuulv3 config | 17:26 |
| pabelanger | jeblair: mordred: I'm going to stop / start zuulv3 to pickup github changes in zuul.conf. I don't think a reload is enough | 17:35 |
| jeblair | pabelanger: should probably wait until 487955 lands | 17:38 |
| pabelanger | ah, I just stop / started | 17:39 |
| pabelanger | let me stop it again | 17:39 |
| jeblair | pabelanger: well, it may or may not error when trying to leave reports in gerrit. i'm not actually sure in this case. :) | 17:40 |
| pabelanger | jeblair: Hmm, I actually think zuul-scheduler is hung ATM. Do you want to take a quick look? | 17:41 |
| pabelanger | why steps where, stop scheduler, the start (with new github setting). Then do the same on zuul-executor | 17:41 |
| pabelanger | I then stopped scheduler to wait for 487955 but didn't seem to respond | 17:42 |
| *** amoralej is now known as amoralej|off | 17:44 | |
| jeblair | pabelanger: no, just do what you need to kill it | 17:46 |
| pabelanger | jeblair: k | 17:47 |
| mordred | jeblair: do we need restarts for pipeline reconfig? | 17:55 |
| mordred | that's live now, yeah? | 17:55 |
| jeblair | mordred: correct, updates on landing | 17:55 |
| jeblair | mordred: and you should get error messages in the speculative phase (even though the config isn't actually used) | 17:56 |
| mordred | jeblair: cool | 17:57 |
| mordred | jeblair: so only changes to zuul.conf and main.yaml require restarts now, yeah? | 17:57 |
| jeblair | mordred: yep. theoretically they should only require reconfigurations. pabelanger said that didn't work, but it sounds like there may have been other problems. | 17:58 |
| mordred | jeblair: nod. | 17:58 |
| pabelanger | Ya, I was seeing a 'github' keyerror for connections, you can see it in debug.log | 17:58 |
| jeblair | i'd like us to assume that they should only require reconfiguration, and then, if it doesn't work, we investigate that as a bug | 17:58 |
| mordred | jeblair: ++ | 17:59 |
| tobiash | jeblair: I wonder how nodepool fits into the multi tenant structure | 18:08 |
| jeblair | tobiash: there isn't a way to restrict labels to certain tenants, but we may want to add that. | 18:09 |
| tobiash | jeblair: do you see any possibility to isolate tenant specific images from each other? | 18:09 |
| tobiash | jeblair: how would you do that? Nodepool knowing the allowed tenants for an image and declining node requests from not allowed? | 18:10 |
| jeblair | tobiash: i think labels is the way to go (labels have associated images) | 18:10 |
| jeblair | tobiash: that would require tenant-specific labels | 18:11 |
| tobiash | jeblair: or would you specify the allowed labels in the tenant config? | 18:11 |
| jeblair | tobiash: if that's not acceptable, then yeah, we'd need to make nodepool tenant-aware. | 18:11 |
| jeblair | tobiash: yeah, that was my first thought. allowed nodepool labels in zuul tenant config file. | 18:12 |
| tobiash | jeblair: sound easy and would fullfill my use case | 18:12 |
| clarkb | +1 to having zuul restrict that | 18:15 |
| tobiash | will look into that tomorrow or next week | 18:19 |
| pabelanger | okay, zuulv3 changes on disk, going to bring it back online | 18:24 |
| pabelanger | jeblair: mordred: okay, figured out the issue. For some reason we have mysql connection settings in zuulv3.o.o, I take it we don't actually need them? They looked to be blocking zuul from starting, I can only assume because it is firewalled? | 18:36 |
| pabelanger | jeblair: mordred: with github connection, we did get an exception: http://paste.openstack.org/show/616770/ | 18:37 |
| mordred | pabelanger: we have them in so that we can do this: https://review.openstack.org/#/c/487950/ | 18:37 |
| mordred | pabelanger: I'd love to figure out why it didn't work - but we can also defer that for a second if you want | 18:37 |
| pabelanger | mordred: okay, possible our settings are not correct? Because once I removed the connection in zuul.conf the scheduler started properly | 18:38 |
| mordred | pabelanger: lemme poke for a sec | 18:41 |
| pabelanger | mordred: sure | 18:41 |
| mordred | well - that would explain that :) | 18:43 |
| mordred | no PyMySQL lib installed | 18:43 |
| pabelanger | ah | 18:43 |
| *** hashar has joined #zuul | 18:43 | |
| mordred | pabelanger: https://review.openstack.org/487974 | 18:44 |
| pabelanger | mordred: actually, do we need that just for scheduler? | 18:46 |
| pabelanger | or does executor need it too | 18:46 |
| Shrews | jeblair: You mentioned that in nodepool v0, you tell it how many nodes to accumulate for a given job (defaulting to 3). How does that work for multinode jobs? Trying to figure out how to manage the held node count for multinode jobs. | 18:48 |
| mordred | pabelanger: just need it for the scheduler - want me to update? | 18:50 |
| pabelanger | mordred: If you don't mind | 18:50 |
| Shrews | jeblair: I'd sort of expect if someone said "hold 3 nodes for job X", and job X used 2 nodes, then it would hold 6 nodes total (so count is really the number of runs for job X). I can't imagine it making any sense any other way | 18:53 |
| mordred | Shrews, jeblair: that makes sense to me: "hold me nodes for X runs of job" | 18:56 |
| pabelanger | I don't see how github connection driver is working for bonnyci, if I am reading this correctly: http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/driver/github/githubconnection.py?h=feature/zuulv3#n397 won't actually have a session yet | 18:56 |
| Shrews | mordred: yeah, that's the only thing that makes sense to me | 18:57 |
| pabelanger | Ah, I think I see the issue | 19:01 |
| pabelanger | we don't seem to have the right version of github3.py installed | 19:01 |
| mordred | pabelanger: oh - maybe we installed it a while back and never updated? | 19:01 |
| pabelanger | possible | 19:01 |
| mordred | pabelanger: cause I was just about to say - I don't see from the github lib how session could not exist | 19:01 |
| pabelanger | mordred: going to stop scheduler, uninstall it and pip install /opt/zuul again | 19:02 |
| mordred | ++ | 19:02 |
| mordred | pabelanger: you might need to either do a -U or an explicit of that one, since we're tracking a git url for that lib | 19:02 |
| mordred | pabelanger: also- I updated the puppet-zuul patch and added a system-config patch to use it | 19:03 |
| pabelanger | I was able to reproduce the issue locally, but had pypi release | 19:03 |
| mordred | pabelanger: https://review.openstack.org/487974 and https://review.openstack.org/487975 | 19:03 |
| *** dmsimard|afk is now known as dmsimard | 19:04 | |
| jeblair | Shrews, mordred: agree with what mordred said. i think that's actually the way it works in nodepool v0 too (incidentally due to the way subnodes work) | 19:05 |
| Shrews | k | 19:05 |
| pabelanger | mordred: I had to run: sudo pip3 install -U /opt/zuul/ -r /opt/zuul/requirements.txt just using pip3 install -U /opt/zuul was using github3.py from pypi | 19:07 |
| pabelanger | mordred: do you know why that would be? | 19:07 |
| mordred | pabelanger: no - but maybe something weird with the git url | 19:09 |
| pabelanger | github3.exceptions.AuthenticationFailed: 401 Bad credentials | 19:11 |
| pabelanger | yaya | 19:11 |
| pabelanger | that is better | 19:11 |
| mordred | pabelanger: it is? why our credentials not right? :) | 19:12 |
| pabelanger | mordred: I have no idea :D | 19:12 |
| pabelanger | checking token now | 19:13 |
| mordred | pabelanger: I odn't see that error in the log ... | 19:13 |
| pabelanger | mordred: debug.log on zuulv3.o.o | 19:14 |
| dmsimard | pabelanger: you remember the series of patches we landed in review.r.o in order to namespace the gerrit instance repositories ? | 19:14 |
| mordred | ah - I was grepping bad | 19:14 |
| dmsimard | pabelanger: i.e, /var/lib/zuul/git/<review.rdo> and /var/lib/zuul/git/<review.openstack> | 19:14 |
| pabelanger | dmsimard: yes | 19:15 |
| mordred | pabelanger: something is running from your home directory | 19:15 |
| dmsimard | pabelanger: so these are aggregated here https://github.com/softwarefactory-project/zuul-distgit/blob/master/Fix-Third-party-CI-conflict.patch | 19:15 |
| dmsimard | pabelanger: it looks like this works well for gerrit triggers but periodic triggers end up using the review.rdo namespace :/ | 19:15 |
| pabelanger | mordred: Oh, that is old | 19:15 |
| jeblair | dmsimard: that should not be necessary for zuulv3, fwiw | 19:16 |
| pabelanger | mordred: let me check path to github3.py | 19:16 |
| dmsimard | pabelanger: we have https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/jobs/_default_jobs.yaml#L36-L43 to determine which git server to zuul clone from based on the pipeline | 19:16 |
| mordred | pabelanger: yah - it's installed globlaly from your home dir :) | 19:16 |
| dmsimard | pabelanger: but what's currently happening is that zuul cloner, albeit trying to clone from git.o.o, will use the review.rdo namespace and thus doesn't find the repo | 19:16 |
| dmsimard | (only for periodic jobs) | 19:17 |
| dmsimard | scratching my head what to do for this one | 19:17 |
| pabelanger | mordred: Oh, I know why. Because I did pip install -U /opt/zuul -r /opt/zuul/requirements.txt from my homedir | 19:17 |
| dmsimard | I guess for periodic jobs they could use a builder that does a regular git clone instead of zuul clone | 19:17 |
| mordred | pabelanger: hahaha | 19:17 |
| pabelanger | and pip git cloned github3.py | 19:17 |
| pabelanger | mordred: maybe should do it from /home/zuul ? | 19:18 |
| pabelanger | or just see why I have to pass -r requirements.txt | 19:18 |
| pabelanger | dmsimard: let me look at it in a bit, head down on zuulv3 ATM | 19:19 |
| mordred | pabelanger: ok - I see the issue with the auth things | 19:20 |
| mordred | the docs are incomplete - patches coming | 19:21 |
| pabelanger | cool | 19:21 |
| mordred | pabelanger: ok - I have updated the settings both in hiera on puppetmaster and directly in the zuul.conf file -so you can try restarting | 19:26 |
| pabelanger | sure | 19:27 |
| clarkb | tobiash: jeblair zuulv2 case sensitive against gerrit 2.13 doesn't seem to work quite right http://paste.openstack.org/show/616780/ | 19:28 |
| clarkb | --Verified is not a valid option | 19:28 |
| pabelanger | mordred: ha, puppet just reverted it | 19:29 |
| mordred | pabelanger: heh | 19:29 |
| mordred | pabelanger: will fix again real quick | 19:29 |
| pabelanger | I'll kick it from puppetmaster in a minute | 19:29 |
| pabelanger | sure | 19:29 |
| mordred | it should do it righ tnext time- it's updated in hiera | 19:29 |
| mordred | I musthave just managed to hit the window beween hiera copy and puppet run :) | 19:29 |
| mordred | pabelanger: k. done | 19:30 |
| pabelanger | mordred: okay, I seen github event | 19:31 |
| mordred | woot! | 19:32 |
| mordred | now to fix docs ... | 19:32 |
| mordred | oh - I have another patch to send up ... | 19:33 |
| clarkb | do v2 and v3 talk to gerrit different afterall? | 19:33 |
| clarkb | this is weird | 19:33 |
| mordred | jeblair: so - this morning my chat with Shrews about type annotations and diagrams of object models led me down a little rathole of learning and I encoded it in a patch... | 19:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Use mypy to do static type checking https://review.openstack.org/488161 | 19:34 |
| mordred | jeblair: please feel free to -2 or ignore that | 19:34 |
| Shrews | lol. mypy is my bash alias for activating my python virtualenv | 19:34 |
| mordred | haha | 19:35 |
| Shrews | i also have a mypy3 for python 3 | 19:35 |
| Shrews | mordred: it's interesting you mention ratholes from this morning convo: http://paste.openstack.org/show/616781/ | 19:37 |
| Shrews | that was just to help me get my head straight | 19:37 |
| clarkb | the gerrit review code in zuul looks the same between the two branches. I am really confused | 19:39 |
| jeblair | mordred: thanks. i'm not sure how i feel about that. but i'm deep into debugging an error condition with how the git directory is set up in a post job. i don't think i have the mental space for it at the moment. | 19:39 |
| jeblair | clarkb: yes, nothing should have changed | 19:39 |
| jeblair | Shrews: zuul has some block diagram stuff in its sphinx docs (gating.rst) which you could use if you felt like adding it to the dev guide | 19:41 |
| Shrews | jeblair: oh, that's just a tiny portion of the entire ERD. doing the full thing would be exhausting | 19:41 |
| jeblair | Shrews: yes, it also has been changing a lot recently | 19:42 |
| jeblair | Shrews: heh, that was definitely not a request. just trying to enable your ocd. :) | 19:43 |
| clarkb | jeblair: tobiash --label Verified=1 works | 19:43 |
| clarkb | which we seem to do in v3 | 19:43 |
| jeblair | oh wow, did we change that? | 19:43 |
| clarkb | yup | 19:44 |
| tobiash | clarkb: there is somewhere such a patch for v2 lying around | 19:44 |
| tobiash | (which I have in my v2 deployment) | 19:44 |
| jeblair | clarkb: Ia9d3e3280995163ae2caee8ed25c7e987fcb00a4 | 19:44 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Update docs on github connection settings https://review.openstack.org/488165 | 19:44 |
| jeblair | clarkb: and Ib591997141b68708bee0b291bb88a93cd3294301 | 19:44 |
| mordred | ok. there's the docs update | 19:44 |
| clarkb | jeblair: zuul/connection/gerrit.py is where the code lives in master now from what I can tell | 19:46 |
| clarkb | jeblair: possible that got lost when connections went in? | 19:46 |
| tobiash | clarkb, jeblair: this was merged in master and got lost somehow | 19:46 |
| tobiash | I2f3a338f18583ac7256b8a2a484cf52be4592fc3 | 19:46 |
| tobiash | ^ re-adds it back to v2 | 19:46 |
| clarkb | tobiash: so I think we want to depeonds on ^ in your master change and also update your master change to remove the normalization function and usage like we did in v3. I can do that if you like or happy to have you do it | 19:47 |
| mordred | pabelanger: https://review.openstack.org/488167 | 19:47 |
| clarkb | er not depends on but rebase on that change | 19:47 |
| pabelanger | mordred: +2 | 19:47 |
| jeblair | clarkb: i suppose that calls into question the results from earlier (inasmuch as this is another input variable), but maybe not the conclusion -- this is still something we should support. | 19:47 |
| clarkb | jeblair: yes I think that is the case. and v3 does it all properly | 19:48 |
| clarkb | jeblair: I also think we should probably just merge I2f3a338f18583ac7256b8a2a484cf52be4592fc3 since it is a regression and doesn't change behavior in a backward incompatible manner. Then we can worry about how to make the case sensitive change to master separately | 19:49 |
| clarkb | I've +2'd the change to master to use --label but not approved it in case you want to think about it more | 19:50 |
| tobiash | clarkb: I could update the patch tomorrow or monday, but feel free if you like to do it earlier | 19:52 |
| clarkb | tobiash: I'll do a local rebase update just to make sure it solves the problem but can push up the update when that is working | 19:53 |
| tobiash | clarkb: ok | 19:54 |
| tobiash | clarkb, jeblair: should the label case patch ever land in v2 or is it just for the case gerrit2.13 coming before zuulv3 | 19:58 |
| tobiash | this could/would possibly break many existing 3rd party v2 deployments | 19:58 |
| Shrews | hrm, our 'nodepool list' output is really far too wide. i think we should scale it back, maybe add a --detail option | 20:00 |
| tobiash | Shrews: it fits well on my 4k display ;) | 20:00 |
| tobiash | Shrews: but yes, it is too wide... | 20:00 |
| clarkb | tobiash: ya thats the big question. I think we want to make the patch available and working but maybe not merge it. This way existing deployments aren't affected unless they intentionally go and upgrade gerrit | 20:01 |
| clarkb | tobiash: in which case hopefully they ask us why it doesn't work and we can point them to v2 patch they can apply | 20:02 |
| clarkb | yup with I2f3a338f18583ac7256b8a2a484cf52be4592fc3 and case sensitive patch zuulv2 works fine | 20:02 |
| tobiash | clarkb: are 3rd party installations also affected by openstack gerrit upgrade? | 20:02 |
| * Shrews makes mental note to change the command... and to steal tobiash's 4k display | 20:03 | |
| jeblair | clarkb: maybe send out a notice saying: pin to 2.5.2, switch your layout.yaml to match case, then upgrade to 2.5.3 | 20:03 |
| tobiash | clarkb: in other words, are there 3rd party installations involved in gating? | 20:03 |
| clarkb | tobiash: yes they will be, but there are also people running their own gerrits so this is a fun one | 20:03 |
| clarkb | tobiash: they aren't involved in gating but the weirdness around clearing votes will affect them | 20:03 |
| * tobiash makes a mental note to buy a kensington lock... | 20:04 | |
| jeblair | mordred: zuul aborted 2 jobs on 488165; any idea why? | 20:04 |
| clarkb | there are actually about ~20 changes between --label fix and the case sensitive fix so might just easiest to merge --label fix in order to avoid rebase madness in reviews | 20:04 |
| clarkb | jeblair: any opposition to approving 424806? | 20:05 |
| jeblair | clarkb: nope done | 20:06 |
| jeblair | i'll abandon the conflict patch | 20:07 |
| mordred | jeblair: looking | 20:07 |
| mordred | jeblair: no - I do not - I'll look in the logs | 20:08 |
| mordred | jeblair: I will say that we've got some conflicting/confusing words in a log message: | 20:11 |
| mordred | 2017-07-27 19:44:59,288 DEBUG zuul.AnsibleJob: [build: 5f0413ab86834c339111cad0a2f7ef82] Job 5f0413ab86834c339111cad0a2f7ef82: updating playbook or role {'connection': 'gerrit', 'type': 'zuul', 'target_name': 'project-config', 'implicit': True, 'project': 'openstack-infra/project-config'} | 20:11 |
| mordred | jeblair: (we can fix later of course- just noticing that the 5f0413ab86834c339111cad0a2f7ef82 id is showing up twice in the line and once prefixed by 'build' and once with 'job' | 20:11 |
| jeblair | mordred: yeah. build is now, job is old and can go | 20:18 |
| mordred | jeblair: remote: https://review.openstack.org/488183 Stop double-logging the build id | 20:18 |
| mordred | it didn't report in here- but I have made the cleanup patch while I was looking at it | 20:18 |
| jeblair | mordred: (if we were to be super-hyper-technical they're actually both right in this case but let's not go down that road :) | 20:18 |
| mordred | jeblair: :) | 20:19 |
| jeblair | mordred: small -1 on that | 20:20 |
| mordred | jeblair: oh - the aborts are because the executor got restarted | 20:21 |
| *** jkilpatr has quit IRC | 20:21 | |
| jeblair | mordred: kk | 20:22 |
| mordred | jeblair: sholdn't the scheduler re-scheduler those in that case? | 20:22 |
| mordred | jeblair: fixed | 20:23 |
| jeblair | mordred: theoretically | 20:23 |
| mordred | jeblair: I'm sure we'll be motivated at some point to investigate that theory | 20:24 |
| jeblair | mordred: it's on my list :) | 20:32 |
| jeblair | (which is to say, it's on my list of things to make stories about) | 20:33 |
| mordred | I feel like I should have a story to manage my lists of lists of things I need to make stories about | 20:34 |
| *** jkilpatr has joined #zuul | 20:40 | |
| jeblair | mordred: you'll just end up with an emacs buffer of things to add to that story | 20:42 |
| mordred | jeblair: then I have to remember which emacs buffer it's in | 20:42 |
| SpamapS | perhaps you could keep track in a story | 20:48 |
| Shrews | should i feel like a bad programmer because I'm placing Kinks lyrics in my code comments? | 21:09 |
| mordred | Shrews: nope. you shold feel like a good programmer | 21:10 |
| *** hashar has quit IRC | 21:12 | |
| *** dkranz_ has quit IRC | 21:15 | |
| jeblair | mordred, pabelanger: i'm reminded that i think we should change the schema for the sql reporter before we get too far down the api/interface path. i think we should replace score with the buildset result (which is actually a thing already -- it's what drives the success/failure reporter decision making; it's just not directly exposed anywhere) | 21:17 |
| jeblair | mordred, pabelanger: however, we should totally be able to do that with an alembic revesion and forward port the existing 'score' values | 21:17 |
| jeblair | so i +3d 487950 and figure we can test that on ourselves :) | 21:18 |
| jeblair | (to elaborate, 'score' makes sense when you see the sql reporter as "a log of what i did to gerrit", but less so in a multi-driver world or as a necessary backend for a dashboard) | 21:18 |
| mordred | jeblair: ++ | 21:19 |
| pabelanger | wfm! | 21:19 |
| mordred | jeblair: so potentially the pipeline config would just not have any arguments | 21:20 |
| mordred | (and for a transition, we can have it just ignore the score argument for a bit) | 21:20 |
| mordred | jeblair, pabelanger: so - I think the next thing I want to crank on is actually translating over the infra-publish-jobs jobs for docs publication, and the publish-to-pypi jobs, getting gate, post, release pipelines defined in v3, then shifting zuul, zuul-jobs, openstack-zuul-jobs, openstack-zuul-roles and sphinx to be gating in v3 and not configured at all in v2 | 21:22 |
| mordred | I'd like to think I can get all of that done by EOD tomorrow | 21:23 |
| mordred | that way next week we can hammer hard on devstack-gate | 21:23 |
| pabelanger | mordred: Ya, I just need to finish up https://review.openstack.org/487551 and I think we'll be in a good spot for tox playbook / role | 21:25 |
| jeblair | mordred: sounds good. i'm working on what i'm fairly confident is a problem with post pipelines, but not one that would block that (they work "good enough" atm) and the fix won't impact you | 21:31 |
| mordred | pabelanger: awesome | 21:35 |
| mordred | jeblair: \o/ | 21:36 |
| mordred | jeblair: about to push up a fix to the ara patch based on your review - although I'm keeping ara in the test-requirements so that we make sure it doesn't bork ansible-playbook runs somehow - thatwork for you? | 21:38 |
| jeblair | mordred: i was kind of not looking forward to having all those extra things installed :| | 21:42 |
| jeblair | mordred: normally i'd say if we do what we had discussed, we would just create a dummy callback plugin for testing | 21:42 |
| mordred | jeblair: oh - right - it's just it's the extra things being installed is the only thing I'm actualy worried about | 21:43 |
| mordred | jeblair: I'm not worried about the callback plugin interface itself | 21:43 |
| mordred | jeblair: but - we can totally leave it out of test-requirements - we don't test that stuff super heavily there yet anyway | 21:44 |
| mordred | jeblair: and come back to it if we discover a need | 21:44 |
| dmsimard | fwiw ara packaging is approved for fedora and I'd eventually like to bribe someone with stickers to do ubuntu packaging work (if that can help with keeping dependencies in check) | 21:46 |
| mordred | dmsimard: doesn't really help - all of our normal dev/test is via pip anyway | 21:47 |
| jeblair | mordred: i'd like that. i really want this to be a facility for anyone to be able to add their own callback plugins; i'd like this to work well with ara, but i'd also like to avoid special casing it more than necessary (because i don't want to have to have built-in support for everything) | 21:47 |
| mordred | jeblair: ++ | 21:47 |
| mordred | jeblair: I also left a todo note about generalizing it | 21:47 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Collect logging information into ara callback https://review.openstack.org/487853 | 21:48 |
| jeblair | kk | 21:48 |
| dmsimard | mordred: what about inside the bubblewrap environment ? | 21:48 |
| mordred | dmsimard: whatcha mean? | 21:48 |
| dmsimard | mordred: I'm not sure how bubblewrap is being used tbh (didn't keep up with that line of work) but I mean it's used to isolate each job inside the runner, right ? | 21:49 |
| dmsimard | Is ara installed "globally", or inside each job/context ? | 21:49 |
| mordred | dmsimard: yes - so the python libs are bind-mounted in - so if it's instlaled on the executor host it'll be accessible from the bubblewrap | 21:49 |
| mordred | dmsimard: we treat each bubblewrap job context also like its own $HOME - so each job should get its own sqlite db into which its playbooks can be recorded | 21:50 |
| mordred | so it should work out pretty smoothly | 21:50 |
| dmsimard | right, okay | 21:50 |
| dmsimard | mordred: so, one thing about that callback_whitelist config I see in that review | 21:51 |
| dmsimard | mordred: anything in the callback_plugins path will be included automatically and do not need to be whitelisted | 21:52 |
| dmsimard | mordred: the whitelist feature is only for callbacks that are bundled in ansible proper | 21:52 |
| dmsimard | so, for example, profile_tasks out of https://github.com/ansible/ansible/tree/devel/lib/ansible/plugins/callback | 21:52 |
| mordred | ah - gotcha. good to know | 21:52 |
| mordred | I may make an ansible docs patch for that - it is not clear in the documentation | 21:53 |
| dmsimard | mordred: https://github.com/ansible/ansible/blob/devel/lib/ansible/config/data/config.yml#L511 :P | 21:54 |
| mordred | dmsimard: :) | 21:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Collect logging information into ara callback https://review.openstack.org/487853 | 22:00 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Remove callback_whitelist setting https://review.openstack.org/488214 | 22:00 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Ensure ref-updated jobs run with their ref https://review.openstack.org/488216 | 22:03 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Ensure ref-updated jobs run with their ref https://review.openstack.org/488216 | 22:04 |
| dmsimard | oh man | 22:04 |
| dmsimard | the new web console is super sweet | 22:04 |
| mordred | dmsimard: \o/ | 22:05 |
| clarkb | its the old web console right just with streaming logs added in? /me making sure he isn't missing something else | 22:06 |
| dmsimard | I was looking at http://zuulv3.openstack.org/static/stream.html?uuid=5b8bf54dd9bd455197771585c2d7da20&logfile=console.log | 22:06 |
| dmsimard | I had never seen it before | 22:06 |
| clarkb | mordred: one thing I noticed with the logs is we have the node names in the lines which is great but not for things that run on the executor? might want to have the zuul executor hostname in there so that you can see the switch more clearly? | 22:07 |
| clarkb | oh its just that the task headers don't have the name got it | 22:07 |
| dmsimard | There's also the challenge around multinode, are the log lines just interleaved ? | 22:07 |
| jeblair | dmsimard: they are interleaved and tagged with the node name | 22:08 |
| pabelanger | clarkb: ya, I mentioned that already too. I think mordred has a patch up to fix that | 22:08 |
| dmsimard | jeblair: that's less than ideal and mostly the reason why upstream Ansible hasn't adopted data streaming iirc | 22:09 |
| jeblair | clarkb: ha. "just with streaming logs". if you mean "just with the first ever instance of streaming console output from arbitrary ansible tasks on multiple nodes simultaneously", yes. *just* that. :) | 22:09 |
| dmsimard | but thankfully ara provides a solution for making sense of multi-host playbook runs \o/ | 22:09 |
| jeblair | dmsimard: i disagree! | 22:09 |
| clarkb | I seem to recall ansible rejecting related patches not because they are a problem but bceause it would cut into tower profits | 22:10 |
| pabelanger | jeblair: mordred: neat, a gerrit change attempted to report against github: http://paste.openstack.org/show/616792/ | 22:10 |
| clarkb | pabelanger: don't cross the streams! | 22:10 |
| pabelanger | :) | 22:10 |
| dmsimard | jeblair: eh, I don't know, you think it's good UX to interleave lines from different hosts ? | 22:10 |
| jeblair | dmsimard: we have visibility into everything that's going on now, and we have plans (and have laid the foundation to support) letting folks request a subset of that, or even other log files. | 22:11 |
| jeblair | dmsimard: yep | 22:11 |
| clarkb | dmsimard: similar to how journald integartion with devstack is a big win but on a process level | 22:11 |
| jeblair | dmsimard: for a live stream of what's happening with a job, absolutely. | 22:11 |
| clarkb | dmsimard: being able to only see a tiny subset of a problem space makes it hard to understand what is happening at any given time | 22:11 |
| jeblair | dmsimard: if i want to see everything, *that* is everything. if i want to see less, i can ask for less. | 22:11 |
| dmsimard | jeblair: you'd ask for less how ? is there filtering built in somehow ? | 22:12 |
| jeblair | dmsimard: but you can't ask for less if you don't have it to start with | 22:12 |
| dmsimard | jeblair: ah. | 22:12 |
| dmsimard | I missed the part where filtering/subsets would eventually be made available | 22:12 |
| clarkb | jeblair: sorry I meant "just" as in the console itself hasn't changed other than to add a feature. Not that the feature is inconsequential | 22:13 |
| clarkb | eg there aren't 10 featuers I missed | 22:13 |
| jeblair | clarkb: i know, i tease. :) | 22:13 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Monitor job root and kill over limit jobs https://review.openstack.org/485902 | 22:15 |
| jeblair | dmsimard, clarkb: also, we chatted with upstream about streaming based on our work and they had some ideas about how ansible could be changed to support it in the future | 22:16 |
| jeblair | so there's definitely no ideological or profit reason at the moment (regardless of whether there was in the past -- i don't know). only technical ones. | 22:16 |
| dmsimard | jeblair: ah, the fact that they're at least open about opportunities is interesting | 22:16 |
| clarkb | jeblair: ya this was pre red hat purchase so things could be completely different today | 22:17 |
| dmsimard | They seem to be opening up and asking for feedback in general actually, I've surprisingly been chatting with bcoca a lot over some of the new stuff they're trying to land in 2.4 | 22:17 |
| jeblair | dmsimard: ya. this is probably not an ansible 2.4 thing. :) | 22:17 |
| dmsimard | They're planning an interesting approach for letting plugins configure themselves automatically -- the new configuration manager in general is going to be great in fact | 22:20 |
| dmsimard | Say, completely changing topic but is toggling zuul v3 support for ara something we can consider if you are planning to use it ? I'd like to start self-testing ara with zuul v3 so that I can make sure I don't break stuff. | 22:22 |
| dmsimard | As in, getting gate jobs to run with zuul v3 on ara code reviews | 22:23 |
| jeblair | dmsimard: we can definitely set up a cross repo job for that. i don't think we're quite at the point of being ready to add ara to the v3 config yet, but you make a compelling case for doing so early when we are ready. :) | 22:25 |
| dmsimard | jeblair: I'm totally down for being an early adopter/beta tester :D | 22:25 |
| jeblair | dmsimard: it should be a v3 job only, because switching zuul itself to v3 is coming up soon, so we wouldn't be running it for long with v2. | 22:26 |
| dmsimard | Yeah, I mean the same thing that you are currently doing with the feature/v3 branch, some jobs out of v2 and some jobs out of v3 | 22:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Update SQL reporter to store results https://review.openstack.org/488221 | 22:28 |
| mordred | jeblair: ^^ there's the sql reporter update to use result | 22:29 |
| dmsimard | mordred: I was actually discussing something along those lines with tristanC this week | 22:29 |
| dmsimard | He was telling me how the result is actually an aggregate result, say, if one job out of the buildset fails, it's marked as failed, even though some jobs may have passed | 22:30 |
| dmsimard | We were discussing how we could display this in terms of UX in the eventual job dashboard | 22:30 |
| jeblair | mordred: looks great! zuul doesn't like it. | 22:30 |
| mordred | jeblair: zuul never does | 22:31 |
| mordred | dmsimard: there also should be a record for each job, yeah? so with buldset result and build.result you should be alble to show it nicely? | 22:31 |
| dmsimard | mordred: maybe, but are builds and buildsets recorded separately ? | 22:32 |
| mordred | dmsimard: they're associated | 22:32 |
| dmsimard | oh, so a build is always the children of a buildset basically | 22:32 |
| mordred | dmsimard: every build in the build table has a reference to a buildset | 22:32 |
| mordred | yup | 22:32 |
| clarkb | and whether or not a buildset is failing depends on voting attributes and the status | 22:33 |
| dmsimard | okay, so passing the result instead of the score sounds like what we need to indeed | 22:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Update SQL reporter to store results https://review.openstack.org/488221 | 22:34 |
| mordred | ok. that should be better | 22:34 |
| mordred | dmsimard: yup - also should be more consistent across triggering mechanisms | 22:34 |
| *** openstack has joined #zuul | 22:46 | |
| *** pleia2 has joined #zuul | 22:48 | |
| *** xinliang has quit IRC | 22:51 | |
| *** xinliang has joined #zuul | 23:04 | |
| *** xinliang has quit IRC | 23:04 | |
| *** xinliang has joined #zuul | 23:04 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Ensure ref-updated jobs run with their ref https://review.openstack.org/488216 | 23:22 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!