Monday, 2017-08-21

jamielennoxis there a reason to expose an executor-only host to the public internet?00:29
jamielennox(incoming)00:30
jamielennoxlog streaming is via the proxy00:30
jamielennoxi don't see a reason a regular user would hit an executor directly00:31
mordredjamielennox: well - finger-protocol streaming if people want that doesn't have a central proxy yet - but I think you could certainly skip public internet on executor if you wanted at this point00:46
jamielennoxmordred: is a central proxy for that coming?00:47
mordredand there is an intent to have a central finger proxy/gateway too00:47
mordredyah00:47
jamielennoxmordred: ok, great. i think it's easiest to set it up with that assumption and see if there are any problems00:48
jamielennoxjust wanted to make sure i wasn't going to do something obviously wrong00:48
mordrednope - I think that should totally work00:48
dmsimard|offjamielennox: fyi https://review.openstack.org/#/c/495561/01:05
jamielennoxdmsimard|off: oh - cool, i saw your tweet but mostly just saw a bunch of cache variables i wasn't sure were public01:06
jamielennoxif that works i'm happy01:07
dmsimard|offreverse engineering power01:07
jamielennoxit was always a hack that seemed unnecessary01:07
dmsimard|offasked upstream about it, they said they don't feel callbacks should "need" access to the inventory01:08
dmsimard|offso there's no proper way to access it and thus is very awkward01:08
jamielennoxi'd probably agree with them in most cases, but zuul..01:09
dmsimard|off¯\_(ツ)_/¯01:09
dmsimard|offI ran into the same issue for ara and was going to run with your hack (which I had also found) but though there *had* to be a better way01:09
dmsimard|offand only came up with something slightly better :/01:10
jamielennoxi'd say way better, but 'all' is special cased in a number of places so it's not that weird here01:11
*** dkranz has quit IRC01:19
jlkIf you mentioned me in the past week, I've lost it to the scrollback gods.03:24
*** bhavik1 has joined #zuul04:56
*** bhavik1 has quit IRC05:03
*** isaacb has joined #zuul07:16
*** amoralej|off is now known as amoralej07:20
*** pbelamge has joined #zuul07:43
*** xinliang has joined #zuul07:45
*** xinliang has quit IRC07:45
*** xinliang has joined #zuul07:45
*** isaacb has quit IRC08:13
*** isaacb has joined #zuul08:22
*** electrofelix has joined #zuul09:19
*** isaacb has quit IRC10:27
*** jkilpatr has quit IRC10:41
*** pbelamge has quit IRC10:47
*** jkilpatr has joined #zuul11:01
*** openstackgerrit has quit IRC11:17
*** dkranz has joined #zuul11:59
*** isaacb has joined #zuul12:16
*** amoralej is now known as amoralej|lunch12:27
*** amoralej|lunch is now known as amoralej13:11
*** dmsimard|off is now known as dmsimard13:59
*** isaacb has quit IRC14:57
jeblairdmsimard: i think having more of the playbook path and having the entries be in chronological order would be a huge ux win in the zuul context (at least -- as a sysadmin, i would also like the same for my plays).  what do you think of this mockup as a solution to the problem of displaying the long names?  http://imgur.com/jetBnYC15:02
jeblairmordred, pabelanger: ^ what do you think of that?15:02
pabelangerjeblair: ++ Ya, that would be perfect I think15:04
dmsimardjeblair: hey o/15:05
pabelangerreally shows nicely to users the order of playbook execution15:05
jeblairdmsimard: welcome back! :)15:05
dmsimardjeblair: that mockup looks good, I was at the very least going to prefix the basedir of the playbook based on pabelanger's feedback15:06
dmsimardfrom your comment it looks like you read the backlog15:06
dmsimardfor chronological order, the only problem is that it makes sense so long as you don't have a number of playbooks that exceeds pagination15:06
dmsimardi.e, your latest playbooks ends up on page 5015:07
jeblairdmsimard: note that the paths to our playbooks are even longer than those in my mockup -- but they have lots of useless stuff at the beginning /var/lib/zuul/builds/blahblahblahblah/ which i stripped out.  that actually suggests an idea for implementation -- what if we supplied a variable to ara to indicate what the playbook "display name" should be?15:07
dmsimardI can add a configuration toggle to control the order in which things are displayed15:07
dmsimardjeblair: oh, wait, my brain might be asleep still -- did you mean the *playbook* ordering or the *tasks* ordering ?15:08
jeblairdmsimard: i'm only looking at the index page now, so playbook ordering15:08
jeblairdmsimard: does static generation use pagination?15:09
dmsimardjeblair: ok, I'll add a new toggle to my todo, pabelanger also asked to reverse the sorting order for tasks (which is consistent with the playbook order, last task first)15:09
jeblairdmsimard: it probably makes sense to keep those the same, so if you reverse one, the other is reversed as well, yeah?15:10
dmsimardjeblair: static generation is 100% parity with dynamic webapp, so yes, it has pagination if required. Default is 10 playbooks in a page but is configurable -- you can see an example in ARA's own integration tests where I force a 3 report pagination: http://logs.openstack.org/74/495474/3/check/gate-ara-integration-py27-latest-centos-7/afc1db6/logs/build/15:10
pabelangerdmsimard: sorry, playbooks like jeblair listed in his mock. That is what I was looking for yesterday15:10
pabelangertask order is correct now15:11
dmsimardpabelanger: you wanted task order in the other direction no ?15:11
pabelangerdmsimard: I'd want what jeblair mock up does15:11
dmsimardjeblair: ah actually the task order is inconsistent with the playbook order right now, tasks are in chronological order while playbooks are reversed15:11
dmsimardjeblair: I'll add two configuration toggles for each ordering -- not hard at all and can't please everyone with a single setting :D15:13
*** isaacb has joined #zuul15:13
jeblairdmsimard: that'll be great.  :)15:13
dmsimardfor the path, there is the problem of overly long paths but I found that the max path is usually 255 characters long (longer is unsupported by most filesystem implementations)15:14
dmsimardso it can be long but we can fairly assume that there is a limit15:14
jeblairdmsimard: it's 4096 on linux15:14
dmsimardjeblair: It's not a linux filesystem limit though, is it ? It's a filesystem one iirc15:14
dmsimarder, not a linux limit*15:15
dmsimardah, unless it's about any single filename, not necessarily it's path -- but I did run into errors relatively recently where files ended up nested deep and would error out15:16
dmsimardhttps://en.wikipedia.org/wiki/Comparison_of_file_systems#Limits15:16
dmsimardyup, looks like there's a distinction between file and path names15:17
jeblairdmsimard: i believe 4096 is a linux limit; some filesystems may be shorter... i'm not sure.  one interesting thing is that shebang (#!/bin/...) paths are capped at 128.  in practice i think planning for a max of 255 seems reasonable for most non-critical applications.15:17
dmsimardjeblair: so, in a previous implementation of the UI, the compromise that we had was to truncate the file path (from the beginning) if it was too long and prefix "..."15:17
jeblairdmsimard: that seems reasonable. i'd say doing that at even 128 would be fine.  though also, playing with my mockup, it seems to fold the path nicely too.15:18
jeblairdmsimard: our paths from zuul are very long, and it would be useful to have, more than just the base dir, but less than the whole path.  what do you think about the idea of a playbook path display name variable?  so we can have "git.openstack.org/openstack-infra/project-config/playbooks/base-test/pre.yaml" ?15:19
jeblair(the full path is /var/lib/zuul/builds/ff8cbf919ab847b3ba6afb6b0cd134ac/trusted/project_0/git.openstack.org/openstack-infra/project-config/playbooks/base-test/pre.yaml">git.openstack.org/openstack-infra/project-config/playbooks/base-test/pre.yaml which only zuul admins can love)15:20
jeblairer sorry15:20
jeblair/var/lib/zuul/builds/ff8cbf919ab847b3ba6afb6b0cd134ac/trusted/project_0/git.openstack.org/openstack-infra/project-config/playbooks/base-test/pre.yaml15:20
dmsimardhow would you see that variable working ?15:21
dmsimardI think we can strike a good (simple) middle ground by giving more room to the playbook path and truncate from the beginning if it's too long, the length before truncating could be made configurable and default to a sane, "good enough" value15:21
jeblairdmsimard: we currently do something similar for the zuul_stream callback... lemme get a link15:22
jeblairdmsimard: http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/executor/server.py?h=feature/zuulv3#n160215:22
jeblairdmsimard: see the "zuul_execution_phase" variable we add -- that's specifically for zuul_stream to include extra info in its output15:23
jeblairdmsimard: so we could add something similar (ara_playbook_name?) and if ara sees that variable, it uses it instead of the filename15:23
jeblairat least as the display name15:23
dmsimardjeblair: oh, I see what you mean now15:23
dmsimardjeblair: that's interesting.15:24
dmsimardI believe harlowja asked for that before but in a way I didn't understand back at the time15:24
dmsimardThat's a good idea, let me add that to the todo.15:24
jeblairdmsimard: here's where it's used: http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/ansible/callback/zuul_stream.py?h=feature/zuulv3#n17315:25
dmsimardyep, I totally get what you mean15:25
dmsimardthere would be nothing preventing you from naming your playbook "dancing banana" :D15:26
jeblairi would run that playbook :)15:26
dmsimardpeanut_butter_jelly_time.gif15:26
jeblairdmsimard: but yeah, so i'd keep the mouseover with the actual full path (don't want to lose any data, this should just be about preseting the most useful form of the data)15:27
jeblairs/preseting/presenting/15:28
dmsimardjeblair: yup, and the full paths will always be available when a) clicking on the file b) looking at the file tab15:30
jeblair++15:30
dmsimardjeblair: that's great feedback, thanks :)15:31
jeblairdmsimard: yay!  thank you!  this is going to be really sweet.  :)15:31
pabelangerYay15:36
dmsimardI'm still making good progress towards 1.0, I'm confident to have something in shape for a beta by the PTG15:40
dmsimardIf you've missed it, I've highlighted what's coming recently in https://dmsimard.com/2017/08/16/whats-coming-in-ara-1.0/15:41
dmsimardWhich reminds me, I've been meaning to ask -- now that ARA is a default in zuul, I'd like to have a v3 gate job in ARA's gate to make sure I don't break things.15:42
dmsimardA *LOT* of code is being merged and while I'm happy with the extent of the coverage we already have, I'd hate to break something that could've been prevented15:44
dmsimardI mean, the testing matrix is fairly exhaustive already.. Spanning unit and integration tests, we're testing py2, py3, ansible 2.2, ansible 2.3, ansible 2.4, fedora, centos, ubuntu and have an implementation of openstack-ansible jobs as well.15:45
* dmsimard should probably document that test matrix15:45
jeblairdmsimard: we don't have any tests which exercise ara yet, so we'll need to add those, but we should be able to add a job like that pretty soon15:46
dmsimardjeblair: the roles aren't integration tested themselves ?15:47
jeblairdmsimard: the ara generation role is in a base job which means it doesn't get run speculatively -- mordred added it to the base-test job (which isn't normally used) so that it could be tested specially before being merged.  short version: it's a bit low-level to be tested in the normal way for zuul jobs.  there are likely ways for us to live-test changes to ara in zuulv3 (though that will require some care), or we can look at adding zuul ...15:51
jeblair... integration tests.15:51
*** isaacb has quit IRC16:08
jeblairSpamapS, clarkb: interested in reviewing 495439 and 495449 -- secrets on tmpfs?16:21
SpamapSjeblair: I'll start with those in a few minutes.16:21
jeblairSpamapS: thanks!16:21
jeblairpabelanger, mordred: anything i should review for upload/publish jobs?16:21
clarkbuh the moon has begun to cover the sun, you should be outside :P though now sure how apparently it is in california yet16:22
pabelangerjeblair: mordred: 494672 is our patch that needs reviewed and testing16:22
pabelangerYa, I'll likely be outside for the next few hours with kids looking at science16:23
jeblairclarkb: it's going to look the same no matter what: fog.16:23
jeblairi mean, i could go to the other side of the hills but.... then i'd have to go to the other side of the hills.16:24
jeblairpabelanger: small -1 on that change16:24
jeblairno solar eclipse on the planet krikkit: http://static.lawrencehallofscience.org/scienceview/scienceview.berkeley.edu/html/view/index.php16:27
pabelangerlol, there are 2 storm troopers right line on nasa TV feed16:28
pabelangerjeblair: thanks, I'll upload shortly16:28
*** openstackgerrit has joined #zuul16:32
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Document execution_wrapper setting.  https://review.openstack.org/49544016:32
mordredjeblair: from this weekend's work, I'm fairly sure we want to figure out some sort of integration test for the playbooks in the base job - or at the very least for the post-logs playbook, since it's impossile to debug syntax errors in it without executor debug logs - and sometimes wihtout keep turned on16:38
mordredjeblair: I've noodled on HOW to do that a little, but don't have an answer I'm super excited abotyet -it's also not URGENT since that's unlikely to change frequently - but I think long-term it'll be valuable16:39
jeblairmordred: that's fair.  i think pabelanger had a proposal to run ansible-lint on project-config.  that seems like something easy we can do now.  but since we don't expect the post playbook to change much between now and ptg, maybe we can tackle that later; possibly in conjuction with the ara job dmsimard requested.  (indeed, this seems like a great conversation to have *at* the ptg)16:40
mordredyup. totally16:40
mordredI _think_ even some synthetic and specific jobs - that actually just set up  avars file witha fileserver variable pointing at a second node and then run ansible-playbook on a couple of playbooks would give us enough to be able to test theplaybooks/roles themselves, ansible versions and ara versions  - but I totally agree it's a great PTG conversation16:42
mordred"hey everybody, look at how this hangs together - what test jobs can we write now to avoid foot-gun?"16:42
jeblairyup16:43
mordredspeaking of - I'd like to restart zuul to pick up the secret naming patch - anything else we should make sure lands first?16:43
jeblairmordred: nah, go for it16:44
mordredjeblair: I think that only needs scheduler restart, yeah?16:44
jeblairmordred: i think that's correct16:45
mordredok. scheduler restarted16:47
dmsimardmordred, jeblair: I'd be happy to help/contribute to such integration jobs :)16:57
jeblairdmsimard: thanks! :)16:57
jeblairmordred, pabelanger: i have 2 things on my plate today -- 1) add the ssh key without it hitting disk, and then 2) start working on the devstack-gate localconf module16:58
mordredjeblair: ++17:00
mordredjeblair: I noodled just a bit over the weekend at splitting out / reworking a couple of d-g things as playbooks - lemme push that stack up real quick17:02
openstackgerritJames E. Blair proposed openstack-infra/zuul-jobs master: Add TODO about improving key security  https://review.openstack.org/49593117:05
jeblairmordred: er actually, i'm going to leave task #1 as a TODO ^ since that gets a lot better in ansible 2.417:06
mordredjeblair: cool17:06
Shrewsjeblair: just curious... when a job inherits from another job in a different project, what's the expected format for referencing that job? <tenant>.<project>.<job>?17:10
dmsimarddid someone mention that the concept of pipeline precedence is going away in v3 ? or is my brain playing tricks on me ?17:10
mordredShrews: <job> - it's a flat namespace17:11
jeblairdmsimard: nope, precedence stays17:11
dmsimardjeblair: okay, so I was trying to understand what precedence (or priority, if you will?) translated to in practice. For example, would jobs in a low precedence remain queued for as long as there are no jobs queued in pipelines with normal and high precedence ?17:12
jeblairdmsimard: that's correct17:13
jeblairdmsimard: we put check at normal, gate and publish at high, post and periodic at low.17:13
Shrewsmordred: ok. i think we may have conflicting info in https://docs.openstack.org/infra/manual/drivers.html#v3-naming17:14
dmsimardjeblair: so I find it hard to believe that post and periodic jobs run at all ?17:14
jeblairdmsimard: that way things people tend to wait around for get handled first17:14
dmsimardjeblair: I mean, at least in the openstack zuul, there's always stuff in the check and gate queue17:14
Shrews"There is in general no need to give the name of the repository as part of the job as it was done with Zuul v2."   ...and then a bit below that...  "If you define a job in a specific repo, the name of the job should use the repository name as prefix or as first part of it."17:15
mordredShrews: right. that's a recommendation for openstack jobs so that people don't step on each other17:15
Shrewsmordred: it just feels like that first sentence needs a "however" portion to it17:16
mordredso shade could define a 'shade-devstack-nodepool' job and not worry aboutotherprojects maybe truing to define a job of the same name17:16
Shrewsbecause we're saying "you don't need to, but you really need to"17:17
mordredyah- and I think there is a distinction between use and define here that likely needs captured17:17
mordredShrews: like - in v2 you needed to define (using a template) a job called gate-{project}-python27,then use gate-shade-python27 in the zuul config17:18
mordredwhereas in v3 there is already a job called tox-py27 andyou don't need to define anything to use it17:18
mordredthere's also possibly two different audiences here (that we never had before)17:19
jeblairShrews, mordred: yeah, i think what's attempting to be conveyed is: not all jobs need to have the project name now (eg, tox-py27).  but if you make a project-specific job, it *should* have the project name.17:19
Shrewsjeblair: yes, exactly. that wording is much more clearerer17:19
jeblairdmsimard: yes, though precedence affects when jobs start (it doesn't wait until they finish).  so once all the jobs in check have *started* then jobs in post begin to start (in the order they were enqueued).  if new jobs show up in check, the ones in post which haven't started yet will sit a bit longer.17:26
jeblairdmsimard: with our current constrained resources, post jobs do indeed wait for a long time (sometimes all day)17:27
*** harlowja has joined #zuul17:29
dmsimardjeblair: okay, so I'm trying to figure out something tangentially related where we have a batch of periodic jobs that we might not want to trigger all at once17:33
dmsimardsay the schedule is every 24hrs, like upstream, does that mean they will all queue at the same time -- or is it possible to space them out in a way that they still run once every 24hrs but with some randomness17:34
dmsimardI'd hate to have to artificially space jobs out into different pipelines to make sure they are somewhat spaced out17:35
dmsimards/artificially space jobs out/artificially split jobs out/17:35
*** amoralej is now known as amoralej|off17:45
jlko/17:47
jeblairdmsimard: there isn't an easy facility to do that.  why do you want them spaced out though?17:57
dmsimardjeblair: mostly the amount of jobs involved17:57
jeblairdmsimard: right, but if they're at the bottom of the precedence list, does it matter?  or did you want them at a higher precedence?17:58
dmsimardjeblair: in jenkins, you can set the schedule of a periodic job with a "shuffle", like "H H(0-7) * * *"17:59
dmsimardjeblair: they'd likely be at a higher precedence, yes, these would be important periodic jobs17:59
jeblairdmsimard: what's less important than them?17:59
dmsimardIt's not so much about the periodic jobs being triggered before another kind of job in terms of priority. It's more about, given a set of jobs in the same periodic pipeline, don't necessarily trigger all of them at once and potentially generate a temporary resource shortage18:01
dmsimardBasically. I'm more interested in "make this job run once a day" than "make this job run at 8AM every morning"18:02
dmsimardIt's a subtle nuance18:02
jeblairdmsimard: right; it's just that when faced with this problem, we found that scheduling things by time wasn't the most direct solution to the problem.  a schedule can't adapt to the current resource situation, but queue precedence can.  our daily jobs get scheduled at some arbitrary time, and then they end up running whetever there is spare capacity.  if the goal is to avoid resource constraints, then prioritizing periodic jobs at the bottom ...18:03
jeblair... seems like the best approach.18:03
jeblairdmsimard: if, after doing that, those jobs don't ever run, then the system just plain doesn't have enough resources -- no time of day shuffling will solve that.  :)18:04
jeblairs/running whetever/running whenever/18:04
dmsimardjeblair: so do things potentially overlap if they don't get started within the time window ? I'm assuming 24hrs is not so problematic but I'm looking at other frequencies (every 4, 6 or 8 hours, for example)18:05
jeblairdmsimard: duplicate entries shouldn't end up in the same pipeline, so if a periodic job is still in the pipeline 24h later, a second one should not be enqueued.  i think.  :)18:06
jeblairor, X hours later, as it were18:07
dmsimardOk!18:07
jeblairdmsimard: tangentially related -- i'm proposing a new kind of pipeline manager that allows for at most 2 of the same project items in a pipeline.  this is to be used in the post pipeline so that we publish the branch tip tarball/docs/etc after a change, but if 5 more changes land while it sits in the queue, we skip the intervening changes and the next publish run will be for the latest change.18:08
jeblairdmsimard: but i'm being a team player and not writing that until after ptg.  :)18:09
jeblairdmsimard: but it may be interesting to explore the interactions with that and periodic pipelines18:09
jlkI got travel approval for PTG. Time to book some flights.18:09
dmsimardjlk: yay!18:09
dmsimardjeblair: makes sense for some kind of jobs, yeah18:10
jlkthe plan is still to cut over that week, right? Nothing caught on fire last week?18:10
dmsimardOtherwise, I'm basically helping with the process of migrating some "legacy" workloads from Jenkins to Zuul and a lot of those are based on that periodic shuffle trigger I mentioned. It's an interesting experiment -- I have to admit that periodic jobs are probably a weakness of Zuul right now, both in terms of scheduling and in reporting, at least when compared to the defacto Jenkins. So hopefully we can18:12
jeblairjlk: so far, yes.  i'm nervous about whether we will have everything ready in time, but that's not a new feeling, and no plans have changed.18:12
dmsimardimprove that.18:12
jlkjeblair: is there a high value target punch list somewhere, that somebody like me could chip away at this week?18:12
dmsimardI think the upcoming job dashboard will improve the periodic job visibility a lot, so there's that18:13
*** electrofelix has quit IRC18:13
jeblairjlk: this is the root etherpad with links to children: https://etherpad.openstack.org/p/zuulv3-pre-ptg18:13
jeblairspeaking of which18:13
jeblairjlk, mordred, pabelanger, SpamapS, Shrews: can everyone attend the meeting today?18:14
jlksigns point to yes.18:14
* Shrews shakes jlk vigorously for an answer18:15
jlkyour answer appears to be vomit.18:15
ShrewsI think that means "It is certain"18:16
dmsimardwhat time is the meeting? I'll add it to my agenda18:16
* Shrews walks out to gaze at the sun18:17
jeblairdmsimard: 2200 utc18:17
dmsimardoh, yeah, the eclipse is in like 10 minutes here.18:17
jlkjust finished watching the 94% here.18:18
SpamapSjeblair: I'll be there.18:20
jeblairit got darker here.  then it got brighter.18:20
* SpamapS spent the morning running a science experiment. :)18:21
SpamapShttps://twitter.com/spamaps/status/89968767970453913618:21
SpamapSjeblair: reviewing now.18:22
jeblairSpamapS: science!18:27
dmsimardjeblair: lol, are you like the guy in this comic? http://i0.kym-cdn.com/photos/images/newsfeed/000/915/770/e6c.jpg :p18:28
pabelangerjeblair: yes18:34
openstackgerritMerged openstack-infra/zuul-jobs master: Add fetch-stestr-output role  https://review.openstack.org/49155918:45
pabelangerjeblair: mordred: https://review.openstack.org/495463/ as ansible-lint to project-config. However, i think we might need to create a shared ansible-lint job between all 4 project (zuul, zuul-jobs, project-config, openstack-zuul-jobs) and zuul-clone each to ensure we have all the right playbooks / roles.  As you can see in 495463, it is failing because of a role issue in another project18:53
jeblairpabelanger: maybe add that to zuulv3?18:54
pabelangerjeblair: yes, I think that would be a great idea18:55
*** weshay has joined #zuul19:01
weshayhowdy19:02
SpamapSjeblair: tmpfs secrets should merge shortly19:35
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Add wrapper driver execution context  https://review.openstack.org/49543919:35
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Write secrets to tmpfs  https://review.openstack.org/49544919:37
openstackgerritPaul Belanger proposed openstack-infra/zuul-jobs master: WIP: Add upload-twine role  https://review.openstack.org/49597219:54
Shrewsjeblair: mordred: first pass at a quickstart guide: https://review.openstack.org/49597119:54
Shrewsand anyone else interested in it, really19:55
leifmadsen*yoink*20:02
mordredShrews: looks great! comments within20:13
Shrewsmordred: thx20:21
*** jkilpatr has quit IRC20:35
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Retrieve filtered list of hosts for a task, not all hosts  https://review.openstack.org/49556120:45
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Fix documentation nits  https://review.openstack.org/49431021:13
*** dkranz has quit IRC21:37
*** jkilpatr has joined #zuul21:53
jeblairit's zuul meeting time in #openstack-meeting-alt22:01
jeblairjlk: ^22:01
jlkoh hi22:01
leifmadseno/22:47

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!