| *** pall is now known as pabelanger | 00:18 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Update devstack test to Fedora 28 https://review.openstack.org/614375 | 00:21 |
|---|---|---|
| pabelanger | ianw: https://review.openstack.org/559211 might be of interest, I could restore if you wanted to take over | 00:23 |
| ianw | pabelanger: oh, heh sorry missed that. sorry it sat so long, i had it in my head to update dib to native v3 jobs before migrating everything | 00:27 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Update devstack test to Fedora 28 https://review.openstack.org/614375 | 00:30 |
| *** pabelanger is now known as pall | 00:33 | |
| *** ssbarnea has quit IRC | 00:52 | |
| *** bhavikdbavishi has joined #zuul | 03:31 | |
| *** bhavikdbavishi has quit IRC | 03:31 | |
| *** bhavikdbavishi has joined #zuul | 03:33 | |
| tobiash | ianw: left a comment on https://review.openstack.org/614074 | 04:56 |
| ianw | tobiash: yeah, happy to do more levels. we have *some* camelcase as the openstacksdk stats get converted to that, but yeah, i think you're right we probably shouldn't introduce new ones | 04:59 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul-jobs master: Add prepare-workspace-git role https://review.openstack.org/613036 | 05:11 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Updated bindep to cover for MacOS requirements https://review.openstack.org/613727 | 05:19 |
| *** pcaruana|elisa| has joined #zuul | 05:29 | |
| *** pcaruana|elisa| has quit IRC | 05:37 | |
| *** swest has joined #zuul | 05:49 | |
| *** bhavikdbavishi has quit IRC | 05:56 | |
| *** bhavikdbavishi has joined #zuul | 06:08 | |
| *** bhavikdbavishi has quit IRC | 06:38 | |
| *** bhavikdbavishi has joined #zuul | 06:39 | |
| *** chkumar|off is now known as chandankumar | 07:13 | |
| openstackgerrit | Merged openstack-infra/zuul master: Collect docker logs after quick-start run https://review.openstack.org/613027 | 07:25 |
| *** pcaruana|elisa| has joined #zuul | 07:45 | |
| *** sshnaidm|off is now known as sshnaidm|ruck | 08:03 | |
| *** bhavikdbavishi has quit IRC | 08:16 | |
| *** bhavikdbavishi has joined #zuul | 08:28 | |
| *** ssbarnea has joined #zuul | 08:40 | |
| *** jpena|off is now known as jpena | 08:52 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Logs stats for nodepool automated cleanup https://review.openstack.org/614074 | 08:54 |
| *** panda|off is now known as panda | 09:01 | |
| *** hasharAway is now known as hashar | 09:03 | |
| *** pcaruana|elisa| has quit IRC | 09:13 | |
| *** pcaruana|elisa| has joined #zuul | 09:15 | |
| *** bhavikdbavishi has quit IRC | 09:15 | |
| *** bhavikdbavishi has joined #zuul | 09:16 | |
| *** bhavikdbavishi has quit IRC | 09:47 | |
| *** bhavikdbavishi has joined #zuul | 09:48 | |
| *** lennyb has quit IRC | 09:48 | |
| *** lennyb has joined #zuul | 09:49 | |
| *** ianychoi has quit IRC | 09:51 | |
| *** ianychoi has joined #zuul | 09:52 | |
| *** pbrobinson has quit IRC | 10:05 | |
| *** electrofelix has joined #zuul | 10:07 | |
| *** bhavikdbavishi has quit IRC | 10:11 | |
| *** hashar is now known as hasharLunch | 11:28 | |
| *** bhavikdbavishi has joined #zuul | 11:43 | |
| *** pbrobinson has joined #zuul | 11:54 | |
| *** panda is now known as panda|lunch | 12:15 | |
| *** themroc has joined #zuul | 12:20 | |
| *** rlandy has joined #zuul | 12:38 | |
| *** jpena is now known as jpena|lunch | 12:46 | |
| *** sshnaidm|ruck is now known as sshnaidm|bbl | 12:50 | |
| *** bhavikdbavishi has quit IRC | 12:57 | |
| *** panda|lunch is now known as panda | 13:02 | |
| *** lennyb has quit IRC | 13:10 | |
| ssbarnea | any zuul guru around? i am wondering how hard it would be to make zuul able to deal with priority queues as described in https://review.openstack.org/#/c/613753/ | 13:24 |
| SpamapS | ssbarnea: I believe that's already a thing. | 13:35 |
| ssbarnea | SpamapS any hints would be welcomed, preferably as comments on that proposal. At this moment it describes only the reasoning and expected behaviour, I didn't check the technical aspects yet. | 13:37 |
| ssbarnea | if this is possible using proposed idea, or simething similar it would be *super* | 13:37 |
| SpamapS | ssbarnea: I think right now, there's a way for admins to bump changes up | 13:39 |
| * SpamapS is looking for how that's done | 13:39 | |
| SpamapS | IIRC there is something, because the scenario described in the spec happens a lot in OpenStack. | 13:40 |
| SpamapS | almost once per 6 month cycle. | 13:40 |
| *** goern has quit IRC | 13:46 | |
| SpamapS | ssbarnea: https://zuul-ci.org/docs/zuul/admin/client.html#promote | 13:46 |
| *** jpena|lunch is now known as jpena | 13:48 | |
| *** goern has joined #zuul | 13:56 | |
| *** hasharLunch is now known as hashar | 14:17 | |
| corvus | hi! anything interesting happen while i was away? | 14:31 |
| Shrews | corvus: tristanC has the k8s func tests working | 14:35 |
| corvus | yay! | 14:36 |
| *** themroc has quit IRC | 14:37 | |
| Shrews | well, func "test" (singular). it's a start | 14:37 |
| *** themroc has joined #zuul | 14:38 | |
| fungi | corvus: zuul.openstack.org is running great. should be all clear to tag the pending release! | 14:43 |
| mordred | corvus: we moved to the zk cluster | 14:46 |
| mordred | corvus: and also, a bug in openstacksdk combined with the cleanup leaked ports patch in nodepool caused ALL of the ports in one of tobiash's tenants to get deleted (oops) | 14:47 |
| tobiash | fortunately it killed only 300 jobs and nothing of the control plane (which I wisely separated from nodepool) | 14:49 |
| fungi | that cleanup feature got reverted though, right? | 14:51 |
| mordred | it did - but a re-revert is up for review ... https://review.openstack.org/614370 | 14:53 |
| *** ianychoi has quit IRC | 15:00 | |
| *** bhavikdbavishi has joined #zuul | 15:26 | |
| tobiash | mordred: this time I'll leave it to you to try this out ;) | 15:49 |
| mordred | tobiash: :) | 15:57 |
| *** nilashishc has joined #zuul | 16:09 | |
| *** goern has quit IRC | 16:09 | |
| *** bhavikdbavishi has quit IRC | 16:18 | |
| *** bhavikdbavishi has joined #zuul | 16:19 | |
| *** hashar is now known as hasharAway | 16:27 | |
| *** pcaruana|elisa| has quit IRC | 16:37 | |
| openstackgerrit | Merged openstack-infra/zuul master: Filter file comments for existing files https://review.openstack.org/613161 | 16:37 |
| *** themroc has quit IRC | 16:49 | |
| *** chandankumar is now known as chkumar|off | 16:53 | |
| *** bhavikdbavishi1 has joined #zuul | 16:57 | |
| *** bhavikdbavishi has quit IRC | 16:59 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 16:59 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: support foreign required-projects https://review.openstack.org/613143 | 17:26 |
| *** pwhalen has quit IRC | 17:30 | |
| *** electrofelix has quit IRC | 17:38 | |
| Shrews | how unfortunate that our experimental nodepool jobs do not capture nodepool logs | 17:50 |
| Shrews | oh, they do! just named weirdly | 17:51 |
| *** panda is now known as panda|off | 17:53 | |
| clarkb | Shrews: are they screen-nodepool? | 17:53 |
| clarkb | if so thats devstack log ahndling sillyness | 17:53 |
| Shrews | clarkb: yeah | 17:54 |
| *** jpena is now known as jpena|off | 18:04 | |
| *** bhavikdbavishi has quit IRC | 18:30 | |
| manjeets | Hi, I am trying to run standalone zuul with just zuul.conf and tenant_conf, but zuul service start is giving me an error | 18:40 |
| manjeets | Traceback (most recent call last): | 18:40 |
| manjeets | File "/usr/local/bin/zuul-executor", line 6, in <module> | 18:40 |
| manjeets | from zuul.cmd.executor import main | 18:40 |
| manjeets | File "/usr/local/lib/python2.7/dist-packages/zuul/cmd/__init__.py", line 158 | 18:40 |
| manjeets | class ZuulDaemonApp(ZuulApp, metaclass=abc.ABCMeta): | 18:40 |
| manjeets | seems like py3 issue ? | 18:41 |
| corvus | manjeets: yes, zuul only supports python 3 | 18:41 |
| corvus | specifically, i think we require at least version 3.5 | 18:41 |
| manjeets | corvus, I have py2 and py3 both on vm, how do I force it to run on 3 ? | 18:42 |
| clarkb | yup, there are annotations in the code which 3.4 doesn't understand | 18:42 |
| corvus | manjeets: if you install it with python3, it should work correctly. you may need to uninstall it, then install it with "pip3" (depending on your distro) | 18:43 |
| manjeets | thanks corvus, I think a note should be added to documentation either on quick start where docker-compose example is shown or in zuul setup | 18:44 |
| manjeets | that would help | 18:44 |
| corvus | manjeets: yes, thanks, we should put something here: https://zuul-ci.org/docs/zuul/admin/installation.html | 18:45 |
| corvus | the docker-compose case should be handled though -- it runs from images with python3 | 18:45 |
| manjeets | corvus, docker-compose gave me so many errors so I'm trying simple zuul first https://zuul-ci.org/docs/zuul/admin/installation.html | 18:46 |
| corvus | manjeets: what errors did you get? | 18:47 |
| manjeets | I created files manaully on host from where i fired docker-compose | 18:47 |
| manjeets | even though those files exists but give me an error http://paste.openstack.org/show/733716/ | 18:48 |
| manjeets | Exception: Unable to read tenant config file at /home/ubuntu/zuul/doc/source/admin/examples/etc_zuul/main.yaml | 18:48 |
| manjeets | corvus, I also changed the zuul.conf to point at https://review.openstack.org | 18:48 |
| corvus | manjeets: if you look at http://git.zuul-ci.org/cgit/zuul/tree/doc/source/admin/examples/etc_zuul/zuul.conf you'll see that the path for the tenant config file is /etc/zuul/main.yaml. | 18:49 |
| manjeets | other issue is sshkey path, I defined the path to key from host which exists but its not able to find that | 18:49 |
| manjeets | http://paste.openstack.org/show/733719/ | 18:49 |
| corvus | manjeets: that file is http://git.zuul-ci.org/cgit/zuul/tree/doc/source/admin/examples/etc_zuul/main.yaml | 18:49 |
| corvus | manjeets: the docker-compose file maps that directory into the container, so you don't need to update the path | 18:50 |
| corvus | manjeets: just edit the file in place. | 18:50 |
| corvus | manjeets: (in other words, /home/ubuntu/zuul/doc/source/admin/examples/etc_zuul/ on your VM is mapped into /etc/zuul/ inside the container) | 18:50 |
| manjeets | oh so I just use /etc/zuul/zuul.conf in conf because of mapping of container path | 18:51 |
| corvus | yep | 18:51 |
| manjeets | and what about sshkey path ? how's that mapping work ? | 18:51 |
| corvus | manjeets: the ssh key is probably a similar problem, you'll want to put it in a directory that's mapped into the containers. | 18:51 |
| corvus | manjeets: you can probably use the etc_zuul or etc_nodepool directories to hold your ssh key | 18:52 |
| corvus | manjeets: if you look at http://git.zuul-ci.org/cgit/zuul/tree/doc/source/admin/examples/docker-compose.yaml you can see the mappings in there. for example "./etc_zuul/:/etc/zuul/:z" means "map etc_zuul on the host to /etc/zuul in the container" | 18:53 |
| manjeets | corvus, so I just put in sshkey in etc_zuul or etc_nodepool and it will be ~/.ssh/id_rsa in container ? | 18:53 |
| corvus | manjeets: no it would be '/etc/zuul/id_rsa' | 18:54 |
| corvus | manjeets: (or /etc/nodepool/id_rsa) | 18:54 |
| manjeets | ohk thanks corvus I spent whole yesterday assuming those scripts might be creating another user and key may not exists in that user's ~/.ssh | 18:55 |
| manjeets | my bad | 18:55 |
| corvus | manjeets: you're probably the first person to go past the end of the docker-compose tutorial :) | 18:56 |
| *** sshnaidm|bbl is now known as sshnaidm|ruck | 18:57 | |
| manjeets | corvus, I think if I figure this docker compose thing, it will make things easy in future, third part ci would be matter of one command, we have 3rd part ci but it has millions of scripts and very hard for new comers to setup and debug | 18:59 |
| manjeets | thanks for your help | 19:00 |
| corvus | manjeets: yeah, i think that'll be a good use. let us know how it goes :) | 19:01 |
| *** nilashishc has quit IRC | 19:09 | |
| *** rcarrillocruz has joined #zuul | 19:39 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul-jobs master: Add role to install kubernetes https://review.openstack.org/605823 | 19:49 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add tox functional testing for drivers https://review.openstack.org/609515 | 19:55 |
| *** dmsimard has quit IRC | 20:05 | |
| *** hasharAway has quit IRC | 20:20 | |
| manjeets | corvus, I think I just passed those errors and saw in logs it tried to connect gerrit | 20:33 |
| manjeets | would it work if I don't define config-project ? | 20:34 |
| *** caphrim007_ has joined #zuul | 20:36 | |
| manjeets | I was able to see gerrit stream from openstack.org this is cool thanks corvus and clarkb | 20:42 |
| corvus | manjeets: you'll need a config-project in order to set up your pipelines (ie, the third-party check pipeline you will define) | 20:43 |
| manjeets | corvus, I have question on that, since that project is separate from review.openstack.org and not available there, can i make pointers to config-project from local repo that's running all this set up | 20:44 |
| corvus | manjeets: yes, you can use the 'git' driver to do that. | 20:45 |
| manjeets | cool thanks corvus | 20:45 |
| corvus | manjeets: https://zuul-ci.org/docs/zuul/admin/drivers/git.html | 20:45 |
| corvus | manjeets: you set it up like a github or gerrit connection, but you can give it a url. it can be a "file:///" url. | 20:46 |
| manjeets | got it thanks ! | 20:47 |
| manjeets | so the comment part goes to main.yaml (like posting comments to a gerrit patch) ? or it should be added test job ? | 20:51 |
| manjeets | corvus, sorry for asking so many questions but what is this section in zuul.conf for ? http://git.zuul-ci.org/cgit/zuul/tree/doc/source/admin/examples/etc_zuul/zuul.conf#n23 | 21:10 |
| clarkb | ianw: for https://review.openstack.org/#/c/613196/ is there a sibling change in zuul for consistency? | 21:11 |
| corvus | manjeets: the comment part goes in the pipeline definition -- like http://git.zuul-ci.org/cgit/zuul/tree/doc/source/admin/examples/zuul-config/zuul.d/pipelines.yaml#n17 | 21:12 |
| caphrim007_ | hey folks. should this url be working? https://git.zuul-ci.org/openstack-infra/zuul-jobs/ | 21:12 |
| corvus | manjeets: that section is so we can import the zuul-jobs repo. it isn't used in the docker-compose example yet, but we plan on adding documentation about how to use existing jobs from that repo | 21:12 |
| clarkb | caphrim007_: https://git.zuul-ci.org/cgit/zuul-jobs/ is the url you are looking for (note the cgit/ not opensatck-infra/ prefix) | 21:13 |
| corvus | caphrim007_: https://git.zuul-ci.org/zuul-jobs/ should work (no openstack-infra); where did you see that so we can update? | 21:13 |
| caphrim007_ | corvus: good question. lemme see if i can find where i saw it | 21:14 |
| corvus | (clarkb and i are both right, dependeng on whether you want to browse or clone) | 21:14 |
| ianw | clarkb: no, i'm not sure zuul is really doing anything similar where the external exception is very closely tied to the internal request | 21:17 |
| clarkb | ianw: ah, well I +2'd but didn't approve on the basis of not wanting the two to diverge. Maybe corvus knows if this would be useful in zuul too? | 21:18 |
| corvus | ianw: can you catch me up on that? | 21:18 |
| caphrim007_ | corvus: I dont think this is where I saw it...perhaps it was some old config I was reading that has since been updated. the only reference i see now is here https://zuul-ci.org/docs/zuul/admin/drivers/git.html in the first sentence | 21:18 |
| clarkb | https://review.openstack.org/#/c/613196/ is the change in question | 21:18 |
| caphrim007_ | and maybe that ref is correct | 21:18 |
| ianw | corvus: https://review.openstack.org/#/c/613196/ ; i think the change explains the reasoning in my usual too much detail :) | 21:18 |
| corvus | yes zuul has something like that which i would like to see ported elsewhere in zuul and possibly nodepool | 21:19 |
| corvus | let me dig it up | 21:19 |
| corvus | ianw, clarkb: http://git.zuul-ci.org/cgit/zuul/tree/zuul/executor/server.py#n614 | 21:20 |
| corvus | ianw, clarkb: the executor creates a logging adaptor for every job, so that you can untangle the output for different jobs | 21:20 |
| corvus | ianw, clarkb: we pass that to the merger too, so that merger output is tied to the job as well | 21:21 |
| manjeets | corvus, http://paste.openstack.org/show/733738/ if I want to run a job on ci-sandbox | 21:21 |
| manjeets | i think i need to add config-project here in this file | 21:21 |
| clarkb | corvus: in that case sounds like zuul already does this where helpful so this change to nodepool makes nodepool more consistent | 21:21 |
| corvus | clarkb: but it does it a different way | 21:22 |
| ianw | corvus: hrm, i'll have to take a closer look, does that handle the multi-line issue thought? is every line passed to the adapter? | 21:22 |
| corvus | it doesn't specifically change the exception format | 21:22 |
| corvus | er, let me check :) | 21:23 |
| clarkb | ya zuul puts it on top of the existing logging formatters, whereas nodepool change uses a special formatter | 21:23 |
| ianw | i was surprised it was such a pain TBH, but i couldn't find a better way to catch each line of the exception traceback | 21:24 |
| ianw | i don't think it quite does -> http://paste.openstack.org/show/733739/ | 21:26 |
| ianw | although, that doesn't have the [build] bit ... | 21:26 |
| corvus | that's not using it | 21:26 |
| corvus | nope, it's like the standard logging: http://paste.openstack.org/show/733740/ | 21:28 |
| ianw | yeah, i can see where it does the ansible output thing, but not where it catches and prefixes an exception. my feeling is it wouldn't, because the lines are added in the formatter | 21:28 |
| corvus | i think there are two issues -- identifying different threads/processes in log lines, and having any identifiers on exception lines | 21:29 |
| ianw | yeah. i think for nodepool the threads/processes thing is fine, the info is in the logging record that comes in | 21:30 |
| ianw | it's just quite annoying that grepping for the request doesn't show you any of the related openstacksdk failures | 21:30 |
| ianw | which tell you if it's nova, neutron, misconfiguration, etc | 21:30 |
| corvus | the exception thing is also nice for when you are grepping timestamps out of logs (i do that frequently and only get the first line of the traceback) | 21:30 |
| ianw | yep, since it's the same record it has the same timestamp | 21:31 |
| corvus | it might be nice to have both things... maybe we can combine them | 21:33 |
| corvus | something like the adaptor which lets you specify the prefix and pass around the logger between objects, and then the formatter which ensures the exception is formatted | 21:34 |
| clarkb | ianw: ya I do a lot of grep -A 10 -B 10 | 21:34 |
| corvus | put another way: the thing in nodepool, but with an explicit prefix rather than a magic one based on the thread name | 21:34 |
| ianw | at least for the launchers, the thread is fairly helpful | 21:35 |
| ianw | 2018-10-25 05:11:53,515 ERROR nodepool.NodeLauncher-0003412786 | 21:35 |
| ianw | e.g. | 21:35 |
| clarkb | ianw: that is the node id? | 21:37 |
| corvus | yep, though with an explicit identifier, we could have just the node id (without the unecessary text) and other info like request number, tec | 21:37 |
| corvus | etc | 21:37 |
| ianw | clarkb: yep | 21:38 |
| corvus | it looks like the two things we're talking about really do need two implementations. so to combine them, i think we'd need to make an adaptor class to handle the explicit prefix, then a formatter class to attach it to the exceptions | 21:39 |
| corvus | but i think if we did that, we could use the same thing pretty much everywhere in nodepool and zuul | 21:41 |
| caphrim007_ | anyone encountered this when viewing a job in the zuul web ui? http://paste.openstack.org/show/733742/ | 21:43 |
| ianw | coruvs: i gotta go for a bit, i'll have to context switch in the logging adapter stuff. if you'd like to put some breadcrumb comments on 613196 i can look | 21:44 |
| corvus | ianw: cool, done | 21:52 |
| caphrim007_ | corvus: does zuul periodically run merger to update the zuul configs it knows about in the various untrusted-projects it is configured for? or how does that happen? | 22:01 |
| corvus | caphrim007_: for github/gerrit it watches for merge events and updates immediately; for the plain git driver, it polls periodically | 22:07 |
| *** rlandy is now known as rlandy|bbl | 22:26 | |
| caphrim007_ | corvus: any idea what i might look at to resolve this? http://paste.openstack.org/show/733744/ | 22:28 |
| caphrim007_ | i'm stumped on the 'NoneType' object has no attribute 'project' | 22:29 |
| corvus | caphrim007_: neat. that's an unhandled error. my guess is that it's due to the name "noop" which is a special internal-only job in zuul (it's the only such job). | 22:32 |
| corvus | caphrim007_: i think if you change the name, it'll work. and we should add a test / fix that error reporting :) | 22:32 |
| caphrim007_ | oh goodie. i just picked that name. go me | 22:32 |
| caphrim007_ | lemme try a name change | 22:32 |
| corvus | caphrim007_: great minds think alike | 22:32 |
| caphrim007_ | corvus: is noop-jobs also reserved like seen here? http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/projects.yaml#n8789 | 22:33 |
| caphrim007_ | or is that a safe name to use | 22:33 |
| corvus | caphrim007_: on a related note, that job is defined internally and does what the description says, so you don't have to define it. :) (depending on whether your goal is to execute a noop job, or you're working through the process of making incrementally more complex jobs) | 22:34 |
| corvus | caphrim007_: that template name is fine to use | 22:34 |
| caphrim007_ | corvus: at this point I think I'm still fumbling. i just want to create a periodic noop job so that i can see it doing "stuff" | 22:34 |
| corvus | caphrim007_: (that template is not already defined, so if you want something like that, you'll need to define it yourself. ours is defined here: http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/project-templates.yaml#n18 ) | 22:35 |
| caphrim007_ | okie doke | 22:36 |
| corvus | caphrim007_: the internal noop job really does nothing, so it's useful for exercising triggering / reporting, but doesn't generate any logs or anything. | 22:36 |
| caphrim007_ | ok. good to know | 22:37 |
| caphrim007_ | corvus: should project-templates be in config-projects? or untrusted-projects? | 22:40 |
| corvus | caphrim007_: either one at your convenience (there's no functional difference for templates) | 22:42 |
| caphrim007_ | corvus: if i want to shared it across projects, the path of least resistence is config-project, aye? | 22:42 |
| corvus | caphrim007_: all the config items within a tenant are globally accessible, so even that will work either way | 22:43 |
| caphrim007_ | oh. ok, was no aware | 22:43 |
| caphrim007_ | not* | 22:43 |
| *** caphrim007_ has quit IRC | 22:45 | |
| kmalloc | corvus: you're going to be in berlin, right? | 22:46 |
| * kmalloc assumes so... but. | 22:46 | |
| corvus | kmalloc: yes | 22:46 |
| kmalloc | corvus: cool | 22:46 |
| kmalloc | :) | 22:46 |
| *** ianychoi has joined #zuul | 23:35 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!