Monday, 2018-10-15

*** smyers has joined #zuul00:46
*** bhavikdbavishi has joined #zuul02:14
SpamapSwho is giving this talk? https://www.meetup.com/OpenStack-LA/events/25539064202:31
pabelangerSpamapS: I believe John Studarus, who is running packethost for openstack-infra02:36
pabelangervia packet.net02:36
SpamapSahhh02:52
dmsimardHe's an openstack community ambassador iirc :)02:56
dmsimardYup: https://groups.openstack.org/ambassador-program02:56
*** bhavikdbavishi has quit IRC02:59
SpamapSsweet03:04
*** jesusaur has quit IRC03:21
*** jesusaur has joined #zuul03:27
*** bhavikdbavishi has joined #zuul03:41
*** bhavikdbavishi1 has joined #zuul03:44
*** bhavikdbavishi has quit IRC03:46
*** bhavikdbavishi1 is now known as bhavikdbavishi03:46
*** quiquell|off is now known as quiquell05:37
*** pcaruana has quit IRC06:37
*** slaweq has joined #zuul06:43
*** bhavikdbavishi1 has joined #zuul06:49
*** bhavikdbavishi has quit IRC06:51
*** bhavikdbavishi1 has quit IRC06:54
*** quiquell is now known as quiquell|brb06:59
*** pcaruana has joined #zuul07:02
*** threestrands has quit IRC07:20
*** evrardjp_ is now known as evrardjp07:21
*** fbo has joined #zuul07:32
*** themroc has joined #zuul07:33
*** quiquell|brb is now known as quiquell07:42
*** ssbarnea has joined #zuul07:43
*** AJaeger has quit IRC07:45
*** AJaeger has joined #zuul07:48
*** hashar has joined #zuul07:49
*** bhavikdbavishi has joined #zuul07:55
*** bhavikdbavishi has quit IRC07:55
*** bhavikdbavishi has joined #zuul07:55
*** anticw has quit IRC08:07
*** hashar is now known as hasharAway08:15
*** goern has joined #zuul09:15
*** rfolco has joined #zuul09:31
*** electrofelix has joined #zuul10:25
*** hasharAway is now known as hashar11:08
quiquellHello11:19
quiquellIt's possible to override ansible_python_interpreter from jobs "vars: "11:20
*** jesusaur has quit IRC11:31
*** jesusaur has joined #zuul11:37
tobiashI don't think so, that's currently hard coded in zuul11:49
*** quiquell is now known as quiuell|lunch12:00
*** bhavikdbavishi1 has joined #zuul12:07
*** bhavikdbavishi has quit IRC12:09
*** bhavikdbavishi1 is now known as bhavikdbavishi12:09
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Use quota handling code for min-ready declines  https://review.openstack.org/61049512:17
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Remove unneeded todo comment  https://review.openstack.org/61049612:22
pabelangertobiash: quiuell|lunch: still hardcoded to python2 right now12:25
*** bhavikdbavishi1 has joined #zuul12:28
*** bhavikdbavishi has quit IRC12:29
*** bhavikdbavishi1 is now known as bhavikdbavishi12:29
*** rlandy has joined #zuul12:34
*** quiuell|lunch is now known as quiquell12:41
quiquellpabelanger: ack12:41
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Ensure that completed handlers are removed frequently  https://review.openstack.org/61002912:41
*** EmilienM has quit IRC12:46
*** EmilienM has joined #zuul12:47
*** bhavikdbavishi has quit IRC13:04
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Use quota handling code for min-ready declines  https://review.openstack.org/61049513:07
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Remove unneeded todo comment  https://review.openstack.org/61049613:07
*** themroc has quit IRC13:29
*** themroc has joined #zuul13:30
*** sshnaidm_ has joined #zuul13:53
*** sshnaidm has quit IRC13:54
*** jesusaur has quit IRC14:00
mordredpabelanger: we should probably swing back around and revisit that at some point - modules should be py3 safe at this point14:03
pabelanger+114:04
*** jesusaur has joined #zuul14:07
tobiashmordred, pabelanger: how do you shard the providers in nodepool to different launcher instances?14:18
tobiashmordred, pabelanger: our nodepool (single instance atm) is currently struggling to keep the usage at quota with two cloud providers (with two pools each) and a few static providers14:20
tobiashand it's accepting node requests very slowly14:20
*** sshnaidm_ has quit IRC14:25
*** quiquell is now known as quiquell|off14:26
*** jamielennox has quit IRC14:27
*** jamielennox has joined #zuul14:29
mordredtobiash: right now it's just manual - we have different nodepool.yaml files for different launchers14:32
tobiashmordred: how many providers do you have per launcher?14:32
mordredtobiash: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool the nl*.yaml files are the config files14:33
tobiashthx14:33
mordredtobiash: it varies - we've got some with 3 - some with 614:33
mordredI think we've been trying to keep the total max-servers count somewhat even across launchers14:34
pabelangertobiash: yup, what mordred said. I've been toying with the idea of just doing a launcher per provider / region, but haven't made that switch yet14:34
pabelangerI'd love to have multiple launchers per provider, but will wait until that is supported14:35
tobiashin the code I found that it does a 10s sleep in the provider loop even if there are quite some requests to fulfill (and there is quota left)14:37
tobiashhttps://git.zuul-ci.org/cgit/nodepool/tree/nodepool/launcher.py#n31414:37
tobiashbecause we probably skip many/several requests per loop because they are locked by other providers14:38
*** sshnaidm_ has joined #zuul14:45
openstackgerritMerged openstack-infra/nodepool master: Update pypi metadata  https://review.openstack.org/55773814:53
tobiashmaybe we should skip that sleep under some conditions14:54
openstackgerritMerged openstack-infra/zuul master: Include enqueue and report timestamps in MQTT msg  https://review.openstack.org/60935515:03
*** jimi|ansible has quit IRC15:10
openstackgerritMerged openstack-infra/zuul master: Retry jobs failed with MERGER_FAILURE  https://review.openstack.org/60994515:15
openstackgerritMerged openstack-infra/zuul master: docs: add Project Testing Interface guide  https://review.openstack.org/57142015:15
*** ssbarnea has quit IRC15:16
*** chandankumar is now known as chkumar|off15:35
*** themroc has quit IRC15:41
*** sdoran has joined #zuul15:44
*** openstackgerrit has quit IRC15:47
*** openstackgerrit has joined #zuul15:49
corvustristanC, mordred: it looks like the "builds" button at the top of the screen still displays even if there is no sql connection... i can't remember if that's a regression or not, but it does seem like something that could be improved... do we expose that in the info endpoint?16:16
mordredcorvus: we either do - or there is a patch floating out there to expose it - it's not a regression, it's been on the "we should improve this" list16:17
mordredcorvus: job_history in the Capabilities object is the flag that is supposed to indicate this16:18
mordredcorvus: oh - but I don't think that's wired up16:19
corvusmordred: ok, i guess it was one of those things where i imagined the future was here today :)16:19
mordredcorvus: I frequently imagine that16:19
mordredso we have a thing in the api response that is the flag that should be checked, but we are not yet setting it16:19
mordredor reading it16:19
mordredcorvus: we do validateWebConfig on all of the Connection plugins in zuul.web - but we don't do a similar thing with the info object ... perhaps we should add a method to Connection that a plugin can use to update an Info object?16:22
mordredcorvus: that way we can have the sqlconnection plugin set job_history to True16:23
*** sshnaidm_ is now known as sshnaidm|afk16:24
mrhillsmando we have a way to ignore a pr via commit message or another mechanism for github driver16:27
mrhillsmanlike the depends-on syntax has special implications16:28
mrhillsmanuse case is an update to like readme for example that there is no need to run jobs on16:29
mrhillsmanor is that just a matter of needing better config-fu16:29
corvusmordred: that sounds like a good idea16:30
corvusmrhillsman: jobs can have 'files' or 'irrelevant-files' matchers, so you can configure jobs to ignore changes that only touch certain files with, for example, "irrelevant-files: README"16:31
mrhillsmanahhh...did not know that, awesome, ty sir16:31
corvusmrhillsman: https://zuul-ci.org/docs/zuul/user/config.html#attr-job.irrelevant-files has more info16:32
mrhillsmanreading is fundamental hehe16:33
mrhillsmanasking is quicker16:33
mrhillsman:)16:33
* corvus is a documentation oracle16:33
mrhillsman++16:34
mrhillsmani was just looking in the wrong place :(16:34
corvusyeah, that's the heart of the documentation, but it's a bit too buried.  i need to figure out a way to make that easier to find and work with.16:37
*** ssbarnea has joined #zuul16:38
clarkbcorvus: https://review.openstack.org/#/c/610159/ adds zuul quick start job to nodepool since that job  builds nodepool from source it could break on the zuul side without gating them together16:39
corvusclarkb: thanks!16:40
mordredcorvus: speaking of that - in the #openstack-infra scrollback, there was a discussion about the gerrit used in the git-review testing and I brought up the zuul quick-start job as a potential model16:41
pabelangerq: For zuul / github, I'd like to write a pipeline trigger when somebody 'edits' a specific comment (eg: 1st in the PR) zuul reacts to it. Looking at https://developer.github.com/v3/activity/events/types/#issuecommentevent I should be able to collect all the information, but unsure how to best add this to zuul today: https://git.zuul-ci.org/cgit/zuul/tree/zuul/driver/github/githubconnection.py#n317 This is all16:43
pabelangeraround if / when somebody updated commit 1 to add depends-on header. Today, zuul doesn't see that an a user needs to then do a 'recheck'.16:43
*** hashar is now known as hasharAway16:46
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Add status page to the quick-start  https://review.openstack.org/61066016:48
clarkbpabelanger: you could probably do a comment matcher like for recheck that matched on depends on? I don't think that will give you a delta though so it may run more than you like16:50
corvusmordred, clarkb, Shrews: ^ that closes the circle on the status stuff16:51
*** ssbarnea has quit IRC16:51
pabelangerclarkb: right now, it seems we hardcode issue_comment event to only be created. Guess we'd have to allow edit16:52
*** jimi|ansible has joined #zuul16:52
corvuspabelanger: the "body" of the pull request is what is consulted for depends-on headers... is that the same thing as the first comment?16:55
clarkbcorvus: aiui yes16:55
clarkbthis tripped me up a bunch getting zuul with kata going16:55
corvuspabelanger: in that case, i think you're on the right track.  does github provide any special indication that a comment edit is for the first comment/pr body?  is it a special kind of event, or are any special flags set in the event?16:56
clarkbto update a depends on I have to go back and edit the first commit. A small concern is I'm not sure if that means other people can fix or set depends on or if it is PR owner only16:56
corvusclarkb: did you meant 'comment' rather than 'commit'?16:57
clarkbyes sorry16:57
corvusok good :)16:57
pabelangerI don't believe there is a special event for the first commit. We'd need to somehow map that out using issue object I think, it has number field, which I assume is related to which comment in a PR16:58
pabelangerI haven't tested that yet16:58
corvuspabelanger: did you mean "first comment" rather than "first commit"?16:58
pabelangeryes, sorry first comment16:58
corvusok good :)16:58
pabelangerbut, this is just a guess. I need to actually test16:59
corvuspabelanger: so yeah, i think the question is whether there's some combination of existing data that we can just expose (like "triggers: github: comment-edited: sequence:1") or whether we need a new synthetic event ("triggers: github: pr-body-edited").17:01
corvuspabelanger: it sounds probable that we can work something out and it's worth continuing to explore.  tobiash and jlk may be of help.17:02
pabelangercorvus: okay good to know, not a major issue, but small annoyance in github UI when needing to add depends-on. When I have time, I'll dig more into it17:04
corvus++17:04
*** ssbarnea has joined #zuul17:06
corvusclarkb: you found a fascinating edge case in depends-on: https://review.openstack.org/61015917:07
corvusclarkb: that worked as long as the target change had not merged, but once it had merged, it didn't work any more :)17:07
clarkbcorvus: do you know why?17:08
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Add Zuul to zuul-quick-start  https://review.openstack.org/61066417:08
clarkbis it because the job def changes?17:08
corvusclarkb: it's because while the depends-on change was unmerged, zuul included zuul in the projects list (due to the dependency), but once it had merged, it wasn't necessary, so it wasn't there.17:09
corvusclarkb: 610664 fixes the issue17:09
openstackgerritJames E. Blair proposed openstack-infra/nodepool master: Run zuul-quick-start job  https://review.openstack.org/61015917:09
clarkboh you know what I chceked that inr everse. I made sure nodepool was in the required projects list for zuul17:09
corvusclarkb: yep.  me too.  :)17:10
corvusi don't have an easy solution to the zuul config puzzle.  it's a fun one.  :)17:10
*** ssbarnea_ has joined #zuul17:19
*** ssbarnea has quit IRC17:23
corvusi was thinking it might be good to add mysql to the quickstart, so that the builds page works anyway...17:23
*** ssbarnea_ has quit IRC17:31
*** panda is now known as panda|off17:37
mordredcorvus: ++17:57
*** ssbarnea has joined #zuul17:58
corvusmordred: http://paste.openstack.org/show/732166/17:59
corvusmordred: that appears to be a mysql deadlock, but i'm not sure how.  it's entirely possible the second connection is another zuul component racing with the first.  they may both be trying to create the tables.  but one of them says "SLEEP" so it's hard to confirm.18:00
corvusmordred: should i just assume that's the case and try to avoid the deadlock, or is there something else i can do to confirm/deny that and see if there's something else going on?18:00
corvusmordred: i ran 'kill 8' (the sleeping one) and deadlock was broken18:01
corvusso i guess that's confirmation of a sort :)18:01
*** dmellado has quit IRC18:05
mordredcorvus: I agree with all of your words18:05
*** dmellado has joined #zuul18:05
mordredcorvus: although this doesn't come up super-often - it does show a potential gotcha with our auto-sqlalchemy approach when starting up a pile of things18:07
corvusmordred: yeah... i'm not sure how to deal with this...18:07
corvusin zuul v4, i'd suggest we lock a znode and let one component create tables18:07
corvusbut i'm not sure how to avoid it in zuul v3.  we could probably only have the scheduler do it, since it's a singleton; but there's some complexity to that.18:08
mordredmaybe in zuul v3 we do $something so that only the scheduler process tries to create tables?18:08
corvusmordred: i don't suppose mysql has a nice way to avoid this?18:08
mordredmysql has a named lock facility - one sec, I get you link18:09
mordredcorvus: https://dev.mysql.com/doc/refman/8.0/en/miscellaneous-functions.html#function_get-lock18:09
mordredcorvus: I'm also not 100% sure how to wire that in - or whether that would be more or less work than using the scheduler as a singleton18:11
corvuscool that might work.  i'll poke at that and the scheduler-only idea and see which is easiest.18:11
corvusexactly :)18:11
pabelangerjlk: do you happen to know what type of event an email reply for comment triggers?18:11
mordredpabelanger: I assume jlk knows all things18:11
pabelangerjlk: for exmaple: https://github.com/ansible-network/vyos/pull/31#issuecomment-42995618018:11
pabelangerrandomly tried 'recheck' via email, didn't work18:12
jlkpabelanger: an email reply? I thought emails were just a way to get a comment in there, and it would generate a comment event.18:12
jlkissue comment18:12
pabelangerokay, I'll ask to look at zuul logs in a bit18:13
*** electrofelix has quit IRC18:13
jlk99.9% certain it'd be https://developer.github.com/v3/activity/events/types/#issuecommentevent18:13
pabelangerack18:13
*** ssbarnea|bkp2 has joined #zuul18:13
jlkIf you have access to the admin page of the app in question, you would be able to see a listing of all the events that have come in18:14
pabelangerAgree, sadly not an admin18:14
corvusmordred: oh, right, we support postgres.18:16
mordredcorvus: yah18:16
pabelangermordred: mnaser: here is a weird traceback from upload-logs-swift with vexxhost: https://softwarefactory-project.io/paste/show/1260/18:16
corvusmordred: i guess i'll do the singleton.18:17
pabelangerI don't think the issue is cloud side, but something weird with zuul-executor and DNS maybe?18:17
mordredpabelanger: yeah - that looks like you're trying to hit vexxhost but instead hitting softwarefactory-project.io18:17
mordredpabelanger: so I'd vote for environmental in sf.io - and I don't have much insight into what might cause that18:18
pabelangermordred: yes, same, so far only once. Was our first post_failure for swift in almost a month of using18:19
mordred\o/18:23
mordredpabelanger: that's great new18:23
pabelangerYah, so far been happy how well it is working18:26
openstackgerritJames E. Blair proposed openstack-infra/zuul master: SQL: only create tables in scheduler  https://review.openstack.org/61069618:30
openstackgerritJames E. Blair proposed openstack-infra/zuul master: WIP: add mysql to quick-start  https://review.openstack.org/61069718:31
corvusmordred: ^ that might work.  i think i need to figure out how to build images locally so i can test locally18:32
corvus(i mean, that shouldn't be hard, i have the commands in a playbook :)18:32
corvusbut maybe zuul will get around to running the quick-start test on the second patch first :)18:32
mordredcorvus: yes - being able to build images locally might wind up being a thing you find yourself wanting to do18:35
*** ssbarnea has quit IRC18:50
corvusthat was pretty easy18:56
*** sshnaidm|afk has quit IRC19:06
*** jimi|ansible has quit IRC19:30
*** hasharAway has quit IRC19:31
openstackgerritMerged openstack-infra/zuul master: Add status page to the quick-start  https://review.openstack.org/61066020:07
*** j^2 has quit IRC20:12
*** j^2 has joined #zuul20:18
*** j^2 has quit IRC20:20
openstackgerritMerged openstack-infra/zuul master: Add Zuul to zuul-quick-start  https://review.openstack.org/61066420:35
openstackgerritMonty Taylor proposed openstack/pbrx master: Allow mounting replacement source in a volume  https://review.openstack.org/61072620:48
corvusi filled up my disk with docker images20:57
jlkthumbsup20:58
mordredcorvus: you're not cloud native enough21:03
mordredcorvus: if you were more cloud native, all your storage would be ephemeral21:03
clarkbthere is a docker rmi prune command21:04
clarkbsomething like that at least, that should remove images that aren't needed for any defined containers21:04
mordredclarkb: maybe it should have been called "vacuum"21:05
corvus"docker image prune"21:09
corvusalso "docker volume prune"21:11
corvusthat was 6.784 GB21:11
clarkbvolume prune is scarier if you run real services on docker as you may want the volume to persist across time even when not associated with a container today21:12
clarkbbut ya in the context of the zuul quickstart probably fine21:12
mordredcorvus: Total reclaimed space: 8.868GB21:15
mordredcorvus: yeah. it adds up21:16
clarkbI noticed something is creating "temporary" volumes when doing zuul quick start in addition to the specified volumes in the composefile21:16
corvusclarkb: yeah.  "docker rm -v" will apparently "remove container and its volumes"21:16
corvusclarkb: well, some images specify they use volumes internally (gerrit for example)21:17
corvusit has like 5 volumes21:17
clarkbah21:17
clarkbthat is probably it then21:17
clarkbcorvus: still plan to fix https://review.openstack.org/#/c/605856/2 ? I'm curious to start using that logging data21:35
corvusmordred:   RAN: /usr/bin/docker exec b64f8c1d2529d17b008cfa0ac1faf056fefc2ffd83a43c662aeb790ed858f025 sh -c mkdir /usr/src21:44
corvusmordred: mkdir: cannot create directory ‘/usr/src’: File exists21:44
corvusmordred: that seems like a minor design flaw21:44
corvusclarkb: yeah... but i'm knee deep right now.  i can get to it in a day or two, or if you want to fix it sooner that's ok with me21:45
clarkbok21:45
clarkbI'll probably throw up a patch later today21:46
*** openstackgerrit has quit IRC21:56
*** openstackgerrit has joined #zuul21:58
*** openstackgerrit has quit IRC21:58
*** openstackgerrit has joined #zuul22:01
*** slaweq has quit IRC22:05
mordredcorvus: oh - piddle. yeah.22:06
*** openstackgerrit has quit IRC22:12
*** openstackgerrit has joined #zuul22:15
*** openstackgerrit has quit IRC22:43
*** jesusaur has quit IRC22:45
*** jesusaur has joined #zuul22:53
*** caphrim007 has joined #zuul22:53
*** openstackgerrit has joined #zuul23:24
openstackgerritClark Boylan proposed openstack-infra/zuul master: Provide some accounting of node usage in logs  https://review.openstack.org/60585623:24
openstackgerritMonty Taylor proposed openstack/pbrx master: Allow mounting replacement source in a volume  https://review.openstack.org/61072623:26
*** caphrim007 has quit IRC23:30
dmsimardcorvus: got a good genuine laugh out of "i filled up my disk with docker images", thanks for that23:32
dmsimardIt's the modern equivalent of what would've been on bash.org23:33
mordreddmsimard: :)23:35
dmsimardmordred: it's the "that was pretty easy" that came a few hours before that made it good23:37
mordreddmsimard: ++23:53

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!