Friday, 2017-02-10

*** Shuo has joined #zuul00:19
*** saneax-_-|AFK is now known as saneax00:25
jeblairrbergeron, mordred: well, i'm planning on writing some stuff up for the ptg tomorrow, so we can see where we are after that :)00:27
mordredjeblair: woot00:29
SpamapSquestion: how would one create a similar animation to the one that is used in zuul presentations?00:57
clarkbSpamapS: I believe its all just javascript00:58
clarkbSpamapS: so you edit the javascript things and run it in your browser00:59
SpamapSclarkb: ok cool.. I want to try and show it on a larger scale with humans approving and reviewing on a time line.01:05
SpamapSso you get a visual sense of why you want automated gating in the first place01:06
*** jamielennox is now known as jamielennox|away01:07
clarkbthat should be possible. the animation is graph manipulation via javascript so should be generatable from logs01:08
*** jamielennox|away is now known as jamielennox01:21
*** Shuo has quit IRC01:49
SpamapSmaybe I could abuse gource again02:15
*** harlowja has quit IRC02:53
*** saneax is now known as saneax-_-|AFK04:07
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: Support swift logserver without Send-Temp-Url-Key  https://review.openstack.org/27033804:48
*** bhavik1 has joined #zuul06:26
*** bhavik1 has quit IRC06:35
*** Cibo_ has joined #zuul07:57
*** saneax-_-|AFK is now known as saneax08:19
*** hashar has joined #zuul08:47
*** Cibo_ has quit IRC09:14
*** Cibo_ has joined #zuul09:27
*** openstackgerrit has quit IRC10:18
*** saneax is now known as saneax-_-|AFK11:49
*** saneax-_-|AFK is now known as saneax11:58
*** hashar is now known as hasharLunch12:19
*** saneax is now known as saneax-_-|AFK12:19
*** saneax-_-|AFK is now known as saneax12:29
*** hasharLunch is now known as hashar12:47
*** openstackgerrit has joined #zuul14:11
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: WIP: Implement node launching  https://review.openstack.org/43152314:11
Shrewswith any luck, min-ready nodes will now be launched ^^^14:11
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: WIP: Implement node launching  https://review.openstack.org/43152314:15
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: DNM: Re-enable devstack test  https://review.openstack.org/43164914:27
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Update nodepool 'list' command  https://review.openstack.org/43164714:27
*** Shrews has quit IRC14:29
*** Shrews has joined #zuul14:32
*** saneax is now known as saneax-_-|AFK14:39
openstackgerritMonty Taylor proposed openstack-infra/zuul feature/zuulv3: Add action plugins to restrict untrusted execution  https://review.openstack.org/42879815:23
mordredjeblair: I think that should be about everytihng until we need to filter plugins from roles - but that should be fairly easy with the playbook plugin filtering already there15:24
mordredjeblair: at least, that should be everything minus adding container execution wrapping15:25
*** yolanda has quit IRC15:38
*** yolanda has joined #zuul15:38
pabelangermorning15:44
rbergeronyes it is. ughhhh15:44
pabelangerShrews: are we ready to update nodedeleter for zookeeper?15:46
pabelangerfrom what I see, it is still using the database15:47
Shrewspabelanger: you can update the command to set node.state=DELETE, but there is nothing running to actually delete though15:48
pabelangerShrews: ya, that's what I've been doing now15:49
pabelangerwas looking to plumb waitForNodeDeletion(), but will hold off for now15:50
jeblairmordred: w00t, i'll resume work on roles next week15:55
Shrewsjeblair: getting closer. min-ready support added: http://logs.openstack.org/49/431649/4/check/gate-dsvm-nodepool/e598bf9/logs/screen-nodepool.txt.gz#_2017-02-10_14_46_15_36516:00
Shrewsjeblair: still fails. i suspect the request is being declined b/c the images are not yet uploaded and ready. will have to add a check for that16:01
jeblairShrews: cool, i will start poking after breakfast16:01
Shrewsjeblair: no need yet, unless you just want to. i have to head out for an appointment in a bit. will add the check for ready images when i return16:02
pabelangerShrews: Nice16:02
Shrewsoh, i just thought of something clever to try...16:06
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: DNM: Re-enable devstack test  https://review.openstack.org/43164916:07
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Update nodepool 'list' command  https://review.openstack.org/43164716:07
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: WIP: Implement node launching  https://review.openstack.org/43152316:07
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: DNM: Re-enable devstack test  https://review.openstack.org/43164916:10
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Update nodepool 'list' command  https://review.openstack.org/43164716:10
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: WIP: Implement node launching  https://review.openstack.org/43152316:10
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Inherit playbooks and modify job variance  https://review.openstack.org/43048316:12
Shrewspabelanger: we're going to need some sort of 'wait' test method that will handle the nodes launched by the min-ready path. i haven't had time to put much thought into that yet16:12
Shrewspabelanger: that's the reason the 'list' change is failing, atm16:13
pabelangerShrews: ya16:13
jeblairmordred:  https://review.openstack.org/43048316:14
jeblairmordred: that should be ready for review now16:14
pabelangerwe should be able to update waitfornodes16:14
mordredjeblair: is it going to hurt my head?16:14
jeblairmordred: a little, but you've already seen half of it16:14
Shrewspabelanger: we can change our tests to not submit node requests and just wait on the min-ready nodes, which is probably the way we should do it. If you have time to put some thought into it, would be very helpful16:15
Shrewspabelanger: yes, update waitfornodes sounds like the thing16:15
pabelangerShrews: sure, I can trying and get it working today16:16
Shrewspabelanger: feel free to post a PS to my 'list' change, if you want16:17
pabelangerkk16:17
Shrewsgotta step away for a bit. bbl16:18
*** hashar has quit IRC16:24
jeblairclarkb, pabelanger, SpamapS, jlk: any of you want to look at 430483 too?16:24
mordredjeblair: makes sense / looks good16:37
* Shrews notes from a remote location that nodes are being created: http://logs.openstack.org/49/431649/6/check/gate-dsvm-nodepool/c796757/16:51
mordredShrews: woot!16:55
pabelangerwoah, I think I just got an allocation from nodepool for a zookeeper node16:58
*** Shuo has joined #zuul17:00
mordredpabelanger: zomg17:01
pabelangerDEBUG:nodepool.NodePool:  Deficit: fake-label: 0 (start: 0 min-ready: 1 ready: 1 capacity: 95)17:02
pabelangerI am sure, my code is wrong17:03
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: [WIP] Allocation a node from zookeeper  https://review.openstack.org/43238617:03
pabelangerShrews: ^ when you are back, would be interested in talking about above17:04
pabelangerspecifically, I guess we don't have subnodes any more17:04
*** harlowja has joined #zuul17:05
*** harlowja has quit IRC17:05
*** harlowja has joined #zuul17:05
pabelangerjeblair: ^ left few comments on above too17:06
*** hashar has joined #zuul17:10
jeblairpabelanger: ah, subnodes are gone.17:11
pabelangerthat is what I thought17:11
jeblairpabelanger: zuul asks for nodes as groups now, so we don't need subnodes anymore17:11
pabelangerack17:12
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Add 'hostname-format' to provider config section  https://review.openstack.org/42984117:16
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Unallocate new nodes if request is pulled  https://review.openstack.org/43045317:17
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Add new Node ZK model attributes.  https://review.openstack.org/43099217:17
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Remove 'template-' from image name format  https://review.openstack.org/43156417:17
mordredjeblair: btw - on the ansible action plugin patch - once we've landed the GH work I think we should set up a test job that cross-tests tip of ansible with tip of zuul - and add a job that checks to see if there are any new actoin plugins in tip-of-ansible17:20
jeblairmordred: ooooh17:20
mordredjeblair: that way we can at least be aware of someone submitting a PR to ansible that would add a new thing17:20
mordredjeblair: (honestly, this would also be a good way to protect ourselves against our necessary use of private ansible variables in places too)17:21
jeblairya17:21
pabelangerjeblair: Shrews: any objections if I start removing the SubNodeLauncher from nodepool?17:22
mordredso it seems we have at least 2 cross-repo relationships that are interesting with ansible -  ansible/zuul and ansible/shade (to test that shade changes don't break os_ modules)17:22
SpamapSjeblair: holy mother of all commit messages17:23
* SpamapS reads17:23
rbergerondfa17:24
* mordred waves to rbergeron and SpamapS17:25
jeblairSpamapS: yeah, sorry.  it's really two changes, but i wrote them together in the wrong order because i didn't realize the other was needed.  i could separate them, but it'd probably take me a few hours.  so i'm hoping i can get away with it.  :)17:25
SpamapSNo I think they're intertwined17:29
SpamapSI mean yeah, in a perfect world. But I don't mind two related changes in one commit when one is feeding the other.17:30
jeblairyeah, i might end up spending a lot of time trying to get tests to temporarily pass were i to try.17:31
jeblairpabelanger: see comment on 43175617:34
SpamapSjeblair: exactly why I think the change is fine as-is17:35
SpamapSI found the tests to be extremely helpful in understanding the code17:36
SpamapStried to look out for uncovered branches but it looks pretty solid17:36
jeblairheh, me too :)  [that was my yesterday]17:36
* rbergeron waves back at mordred17:38
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: Remove subnodes from nodepool  https://review.openstack.org/43240317:38
* mordred loves it when the tests are useful for undestanding :)17:39
pabelangerjeblair: WFM, I can do a follow up17:39
clarkbjeblair: reviewed17:40
*** Cibo_ has quit IRC17:46
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: Update nodepool hold to use zookeeper  https://review.openstack.org/43175617:48
*** hashar has quit IRC17:52
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: [WIP] Allocation a node from zookeeper  https://review.openstack.org/43238617:55
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Inherit playbooks and modify job variance  https://review.openstack.org/43048317:59
jeblairclarkb, SpamapS, mordred: thanks.  responded and updated ^18:00
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: Update nodepool hold to use zookeeper  https://review.openstack.org/43175618:08
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: Implement nodepool 'list' command  https://review.openstack.org/43164718:08
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: WIP: Implement node launching  https://review.openstack.org/43152318:08
pabelangerUGh18:08
pabelangerjeblair: sorry18:08
pabelangerI incorrectly rebased there18:08
*** yolanda has quit IRC18:14
*** yolanda has joined #zuul18:16
Shrewspabelanger: umm18:21
Shrewscan you undo what you did there? it lost a lot of stuff18:21
Shrewss/lost/reverted/18:21
pabelangerShrews: yes, I'll revert18:22
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: WIP: Implement node launching  https://review.openstack.org/43152318:23
Shrewsjeblair: yeah, sorry for the size of 431523. it's the culmination of days of work to get something functional18:24
Shrewsjeblair: and it's not even complete. but i think it's a good base to add the "rest of the things"18:24
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: Update nodepool 'list' command  https://review.openstack.org/43164718:25
pabelangerShrews: okay, reverted to your patch18:26
Shrewspabelanger: many thanks18:26
pabelangerYa, sorry for the noise18:26
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Implement node launching  https://review.openstack.org/43152318:28
jeblairShrews: no apology necessary.  i'm in the same boat.  :)18:29
jeblairclarkb: did you want take another pass over 430483, or should i go ahead and approve it?  (i only made the minor changes you and SpamapS suggested)18:32
clarkbjeblair: let me take a quick look at interpatch diff18:32
ShrewsIf anyone else wants to review https://review.openstack.org/431523, I'm happy to do some hand holding and walk you through the important pieces. Would probably make it easier.18:34
jeblairi've restored my +2 on it18:35
clarkbjeblair: re self.playbook where is that being used? It isn't used internally to that class at least18:35
jeblairclarkb: in runPlaybooks18:35
jeblair        job_status, job_code = self.runAnsiblePlaybook(self.jobdir.playbook)18:35
*** Shuo has quit IRC18:36
jeblairthat happens after preparePlaybookRepos which chooses which of the candidate playbooks to run18:37
jeblair(via preparePlaybookRepo and findPlaybook)18:37
jeblair[we check out all of the repos which might have a playbook, and examine each in the order supplied by zuul.  the first one we find we set jobdir.playbook to, and that is the one we run]18:38
clarkblooks like line 593 ish?18:39
jeblairclarkb: yep18:39
pabelangerShrews: sure, I can take a look. If you mind looking at 432386 to see how I updated waitForNodes()18:44
Shrewspabelanger: ooh, yeah. looking18:44
pabelanger432403 also removed subnodes18:44
Shrewspabelanger: oh, that won't work18:45
pabelangerYa, figured there would be some issues18:46
Shrewspabelanger: you're attempting to use the old Nodepool stuff. There is no allocation history, and getNeededNodes() doesn't exist in the new one18:46
jesusaurShrews: i'd like to take a look at 431523, it seems pretty meaty18:49
Shrewsjesusaur: cool. yeah, it's got some meat to it18:50
Shrewsjesusaur: let me know if you need a quick walkthrough18:51
jesusaurShrews: will do18:52
pabelangerShrews: right, I replaced the database calls with zookeeper. But, I am not yet familiar with the post v3 status of how allocation looks18:55
Shrewspabelanger: for the subnode removal stuff, I had planned to do a bulk code removal to get rid of all the unused code (everything you're changing there is not active code now). That would avoid the modifications you have to make to dead code that you had to do there.18:55
pabelangerShrews: okay, that works for me18:55
Shrewspabelanger: but that removal is ok. probably don't want to do any more just yet18:56
pabelangermaybe I'll hold off until I see how you implement the allocation of nodes18:56
Shrewspabelanger: allocation? could you give me more words as to what you need there?18:57
jesusaurShrews: im a bit confused by removeCompletedRequests, when is that called?19:00
Shrewspabelanger: like, how nodes are assigned to providers, or ...  ?19:00
Shrewsjesusaur: ah, that's part of the min-ready code path. The main NodePool thread code sits in a loop and makes repeated calls to createMinReady() (which calls removeCompletedRequests)19:02
pabelangerShrews: right, the logic where nodepool decided to start building more nodes.  if I understand, 431647, you manually allocated the node request with the _createNodeRequest() function, over nodepool launching it19:03
pabelangerI _think_ that is the last step to bring on devstack gate job back online19:03
Shrewspabelanger: ah, we needed the manual request BEFORE i added support for min-ready (which happened this morning)19:04
Shrewspabelanger: now we don't need the manual request19:04
Shrewsso i should update that.... but i was hung up and getting the automatically created node19:05
pabelangerokay, so does this mean nodepool/allocation.py will eventually be deleted?19:05
Shrewsyep19:05
pabelangerOkay19:05
pabelangernow I understand19:05
pabelangercool19:05
pabelangerso ya, my patch is wrong :)19:06
Shrewspabelanger: capacity is calculated by querying ZK now (see _wouldExceedQuota method)19:06
jesusaurShrews: ah, ok. the function name made search to see if it would deallocate nodes created in the main body of NodeRequestHandler19:09
Shrewsjesusaur: so, fwiw, the main pieces of that change are NodeLauncher._launchNode() which is an almost copy-paste of the original node launching code. And the createMinReady() method (which starts the min-ready support code path). Everything else is just fodder in support of those19:14
Shrewsjesusaur: and keep in mind that it isn't quite complete. it's just the base to get a minimal working nodepoold. i'll build on it in future reviews19:17
Shrewsjesusaur: this log shows it actually working, if that helps: http://logs.openstack.org/49/431649/6/check/gate-dsvm-nodepool/c796757/console.html#_2017-02-10_16_38_09_80611819:18
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: Update waitForNodes() to support zookeeper  https://review.openstack.org/43242919:20
pabelangerShrews: okay, have a review of ^19:20
pabelangerthat should pass your test_list_nodes now19:20
pabelangerhowever, don't know how to properly get getNeededNodes() now19:21
Shrewspabelanger: that might be a bit racey with the node state there. maybe we should have waitForNodes take a label to wait for, and return when state==READY?19:25
Shrewsif node.type == label and node.state == zk.READY: return19:25
pabelangerShrews: Seems sane, we can add a secondary check19:25
Shrewsto be more complete19:25
jesusaurShrews: my concern is around what happens when NodeRequestHandler sees that no nodes are available and launches a nodeset; can't that fulfilled request then get picked up in removeCompletedRequests and then that nodeset is no longer associated with the request?19:33
jesusaurI guess I'm confused about the path from state READY to state USED19:34
Shrewsjesusaur: That's EXACTLY what we want to happen with min-ready. We don't want the nodes associated with a request. We're just preparing them for some other request down the line.19:35
Shrewsjesusaur: What I'm doing there is usurping the node request pipeline to prime the ready nodes19:36
Shrewsjesusaur: once I unset allocated_to on the nodes, they'll be available for the requests from zuul19:36
jesusaurShrews: right, but aren't there two different paths? there are nodes booted for min-ready, and there are nodes booted for a specific request?19:37
Shrewsjesusaur: no. they both go through NodeRequestHandler19:37
jesusauror do all nodes go into the available pool before getting associated with a specific job?19:37
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: Update waitForNode() to support zookeeper  https://review.openstack.org/43242919:38
pabelangerShrews: ^took inspiration from waitForBuild()19:38
Shrewsjesusaur: for the min-ready nodes, they are _temporarily_ associated with the request from nodepool. but once I delete that request and reset the node assignment, any zuul request will see them as available for us19:39
Shrewsuse19:39
jesusauraha, ok19:39
Shrewspabelanger: except we won't have the node_id to pass in. specifying the label would help to get around that. we just have to iterate through all nodes looking for that label19:43
pabelangerShrews: ya, there is a little bit of guess work there19:45
pabelangerneed some coffee19:46
Shrewspabelanger: i think if you just add a getNodes() call, loop through those, use the type/state check you already have, then that might work19:46
*** hashar has joined #zuul19:59
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: Update waitForNodes() to support zookeeper  https://review.openstack.org/43242920:07
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: Update waitForNodes() to support zookeeper  https://review.openstack.org/43242920:26
Shrewspabelanger: left you a comment on PS3. I think we should rebase my 'list' change on top of yours, too.20:27
pabelangerShrews: sure, just testing the patch with dib_pause tests, ATM20:30
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: Re-enable test_dib_image_pause / test_dib_image_upload_pause  https://review.openstack.org/43244720:32
pabelangerShrews: left a comment, also added some more code in ps420:34
Shrewspabelanger: yes, it would be a bit problematic if we have configs that launch more than one node of that type. I'm not sure how else to handle it, TBH.20:35
pabelangerI think ps4 adds the 2nd check you are looking for20:36
pabelangerthis lets us limit things to a specific label20:36
pabelanger432447 is a follow up to see it in action20:36
Shrewspabelanger: your PS4 is doing pretty much the exact same thing as what I suggested. In fact, you could reduce PS4 to just the 2nd while-loop20:39
Shrewsall of our test .yaml files have min-ready = 1, fwiw. except for the subnode.yaml which is useless now20:41
pabelangerokay20:42
pabelangerlet me do that20:42
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: Update waitForNodes() to support zookeeper  https://review.openstack.org/43242920:43
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: Update waitForNodes() to support zookeeper  https://review.openstack.org/43242920:48
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: Update nodepool 'list' command  https://review.openstack.org/43164720:48
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: WIP: Implement node launching  https://review.openstack.org/43152320:48
Shrewsjeblair: So, we right now assume that all of the node types given in a NodeRequest can be handled by a single provider. If that's not what we wanted, I might have designed us into a corner here.20:48
jeblairShrews: nope, that's correct.  we want that (and further, we actually want them in the same az, but like i said earlier, that can wait)20:49
pabelangerokay, I think I rebased correctly20:49
jeblairShrews: that's desirable so that, for example, devstack multinode tests which require close network proximity will work20:49
Shrewspabelanger: uh oh. no, you undid my last PS to 431523 (which had two +2's and a +1)  :(20:50
pabelangergrr20:50
pabelangernot sure why that happened20:50
Shrewsjeblair: oh  *phew*20:50
pabelangerOh: https://review.openstack.org/#/c/431523/13..14/nodepool/nodepool.py20:50
pabelangerI have no idea how I pulled that in20:50
Shrewspabelanger: perhaps we should undo that and get it merged to prevent this again  ;)20:51
jeblairShrews: (however, we'll probably want to tweak the min-ready algorithm once we do that so we don't end up making giant requests that all get filled from the same place)20:51
pabelangerShrews: ya20:51
pabelangerlets do that20:51
Shrewsjeblair: same place? it should be random which provider satisfies the request20:52
openstackgerritPaul Belanger proposed openstack-infra/nodepool feature/zuulv3: Implement node launching  https://review.openstack.org/43152320:52
pabelangerreverts to PS1320:52
Shrewsjeblair: oh, you mean if like min-ready is 50 or something20:52
jeblairShrews: exactly20:52
Shrewsdidn't think of that. we could break up the requests later20:53
jeblairShrews: ours are on the order of about 2020:53
jeblairShrews: yeah.  it's not urgent.20:53
Shrewspabelanger: thx again20:53
pabelangerI'll learn git one day20:54
Shrewspabelanger: feel free to +A it  :)20:54
pabelangerjeblair: have a few minutes to look at 432429, Shrews and I have been going back and forth on that. I think we have it right now, but side affect is we'd potentially only launch 1 ready node, if we had more then one20:54
jeblairpabelanger: i'll look at it a bit later, i have to afk for a bit20:55
pabelangerack20:55
Shrewspabelanger: i think your test is fooling you21:00
Shrewsit's not actually being called21:00
Shrewsleft comments21:00
pabelangerShrews: ya, i'll squash it21:02
openstackgerritAntoine Musso proposed openstack-infra/zuul master: (WIP) mimic clone when directory exists  https://review.openstack.org/43247722:03
openstackgerritAntoine Musso proposed openstack-infra/zuul master: (WIP) mimic clone when directory exists  https://review.openstack.org/43247722:06
hashar^ ignore my patches :)22:10
*** Guest34532 has joined #zuul22:30
*** Guest34532 has quit IRC22:33
*** hashar has quit IRC22:38
*** jamielennox is now known as jamielennox|away22:58
*** nt has quit IRC23:06
*** nt has joined #zuul23:10
*** jamielennox|away is now known as jamielennox23:50

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!