*** threestrands has joined #zuul | 00:00 | |
clarkb | wow so they just removed it | 00:02 |
---|---|---|
openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add initial withCertManager input toggle https://review.opendev.org/718840 | 00:04 |
openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add gearman tls secret provided by cert-manager https://review.opendev.org/719110 | 00:08 |
openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add registry tls secret provided by cert-manager https://review.opendev.org/719185 | 00:16 |
*** cdearborn has quit IRC | 00:23 | |
*** rfolco has quit IRC | 00:30 | |
openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Document output variables https://review.opendev.org/719704 | 00:48 |
ianw | tristanC: there was a couple in the stack https://review.opendev.org/#/q/topic:ensure-pip you didn't vote on -- was that on purpose or just didn't get to? | 01:00 |
openstackgerrit | Merged zuul/zuul-jobs master: ensure-pip: Add role https://review.opendev.org/717639 | 01:09 |
*** swest has quit IRC | 01:18 | |
openstackgerrit | Merged zuul/nodepool master: Stop checking user_data in func test https://review.opendev.org/720088 | 01:27 |
openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Python roles: misc doc updates https://review.opendev.org/720111 | 01:27 |
*** swest has joined #zuul | 01:33 | |
*** y2kenny has quit IRC | 01:36 | |
*** Goneri has quit IRC | 01:37 | |
openstackgerrit | Merged zuul/nodepool master: Add ZooKeeper TLS support https://review.opendev.org/712733 | 01:41 |
*** ysandeep|away is now known as ysandeep|rover | 02:11 | |
*** zxiiro has quit IRC | 02:39 | |
*** sanjayu__ has quit IRC | 02:47 | |
*** ysandeep|rover is now known as ysandeep|BRB | 04:08 | |
*** ysandeep|BRB is now known as ysandeep|rover | 04:23 | |
*** bhavikdbavishi has joined #zuul | 04:26 | |
*** bhavikdbavishi1 has joined #zuul | 04:29 | |
*** bhavikdbavishi has quit IRC | 04:30 | |
*** bhavikdbavishi1 is now known as bhavikdbavishi | 04:30 | |
*** evrardjp has quit IRC | 04:37 | |
*** evrardjp has joined #zuul | 04:37 | |
openstackgerrit | Ian Wienand proposed zuul/nodepool master: Test alternative username in unit tests https://review.opendev.org/720101 | 05:32 |
*** tobberydberg has joined #zuul | 05:45 | |
*** threestrands has quit IRC | 05:54 | |
*** bhavikdbavishi has quit IRC | 06:18 | |
*** hashar has joined #zuul | 06:26 | |
*** bhavikdbavishi has joined #zuul | 06:35 | |
*** hashar has quit IRC | 06:42 | |
*** gtema has joined #zuul | 06:43 | |
*** dpawlik has joined #zuul | 06:49 | |
*** sanjayu__ has joined #zuul | 06:55 | |
*** hashar has joined #zuul | 07:07 | |
*** jcapitao has joined #zuul | 07:14 | |
*** tosky has joined #zuul | 07:23 | |
*** rpittau|afk is now known as rpittau | 07:28 | |
*** bhavikdbavishi has quit IRC | 07:47 | |
*** ysandeep|rover is now known as ysandeep|lunch | 07:48 | |
*** jpena|off is now known as jpena | 07:53 | |
*** bhavikdbavishi has joined #zuul | 07:59 | |
*** hashar has quit IRC | 08:02 | |
msuszko | ianw: Excuse me for tests missing from c/719191 I'm speeding though environment and process setup at the moment. (behind air gap) | 08:03 |
*** bhavikdbavishi has quit IRC | 08:22 | |
*** bhavikdbavishi has joined #zuul | 08:30 | |
*** bhavikdbavishi has quit IRC | 08:35 | |
zbr | AJaeger: re suse support on zuul-roles, i wonder who can commit to keep suse supported? | 08:39 |
*** ysandeep|lunch is now known as ysandeep|rover | 08:40 | |
*** bhavikdbavishi has joined #zuul | 08:40 | |
zbr | i would personally be happy to avoid having to spend extra time making roles work on suse. | 08:40 |
*** gtema has left #zuul | 08:40 | |
AJaeger | zbr: dirk and cmurphy (both not in this channel) are the best to talk to | 08:44 |
openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Improve 404 error message on download-logs.sh https://review.opendev.org/720035 | 08:52 |
openstackgerrit | Sorin Sbarnea proposed zuul/zuul master: For pre blocks to wrap text https://review.opendev.org/681532 | 09:03 |
openstackgerrit | Sorin Sbarnea proposed zuul/zuul master: Force pre blocks to wrap text https://review.opendev.org/681532 | 09:03 |
swest | zuul-maint: anyone up for a review of https://review.opendev.org/#/c/701531/ ? Besides the issues that this solves it's also improving the startup time quite a bit :) | 09:10 |
tobiash | tristanC or mordred: this would be an easy review which improves logging in nodepool: https://review.opendev.org/716566 | 09:12 |
*** sgw has quit IRC | 09:12 | |
*** hashar has joined #zuul | 09:14 | |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add --validate-tenants option to zuul scheduler https://review.opendev.org/542160 | 09:19 |
avass | tobiash: re https://review.opendev.org/#/c/653712/ | 09:33 |
avass | tobiash: I have a vague memory doing something like that and ansible_host/ansible_user would use the delegate hosts variables instead of the remote host. | 09:34 |
avass | I could be wrong though | 09:34 |
tobiash | avass: ansible_host stays the remote host | 09:34 |
avass | alright | 09:34 |
tobiash | we use it like that in prod since a long time ;) | 09:35 |
avass | oh :) | 09:35 |
avass | Couldn't get a clear answer when looking around either | 09:35 |
avass | tobiash: We can't test that in zuul-jobs right? I guess we don't have any ssh-enabled windows nodes | 09:36 |
tobiash | avass: unfortunately no and we couldn't setup a third party ci yet | 09:37 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Support per branch change queues https://review.opendev.org/718531 | 09:55 |
openstackgerrit | Merged zuul/zuul-jobs master: Support ssh-enabled windows hosts in add-build-sshkey https://review.opendev.org/653712 | 10:00 |
openstackgerrit | Merged zuul/zuul master: Use ZK TLS in quickstart https://review.opendev.org/712817 | 10:05 |
*** rpittau is now known as rpittau|bbl | 10:23 | |
*** jcapitao is now known as jcapitao_lunch | 10:38 | |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Move queue from pipeline to project https://review.opendev.org/720182 | 10:40 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Move queue from pipeline to project https://review.opendev.org/720182 | 10:42 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Move queue from pipeline to project https://review.opendev.org/720182 | 10:45 |
openstackgerrit | Merged zuul/zuul-jobs master: Improve 404 error message on download-logs.sh https://review.opendev.org/720035 | 10:46 |
*** ysandeep|rover is now known as ysandeep|coffee | 11:00 | |
*** cdearborn has joined #zuul | 11:15 | |
*** ysandeep|coffee is now known as ysandeep|rover | 11:17 | |
*** jpena is now known as jpena|lunch | 11:30 | |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: Support per branch change queues https://review.opendev.org/718531 | 11:39 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Move queue from pipeline to project https://review.opendev.org/720182 | 11:39 |
*** jcapitao_lunch is now known as jcapitao | 11:58 | |
tristanC | swest: tobiash: done and done :-) | 12:01 |
*** gtema has joined #zuul | 12:01 | |
*** bhavikdbavishi has quit IRC | 12:03 | |
tobiash | tristanC: thanks! | 12:04 |
openstackgerrit | Jan Kubovy proposed zuul/zuul master: Connect merger to Zookeeper https://review.opendev.org/716221 | 12:07 |
openstackgerrit | Jan Kubovy proposed zuul/zuul master: Connect executor to Zookeeper https://review.opendev.org/716262 | 12:07 |
openstackgerrit | Jan Kubovy proposed zuul/zuul master: Connect fingergw to Zookeeper https://review.opendev.org/716875 | 12:07 |
tristanC | ianw: oops i misread the ensure-pip role implementation, it seems like it install pip however pip is already there | 12:07 |
tristanC | corvus: it seems like a case of an ensure-* role that is not a noop when the thing it ensure is already there | 12:08 |
tristanC | left a comment about it in https://review.opendev.org/#/c/717663/24 | 12:10 |
tobiash | corvus: since I implemented a small part of https://review.opendev.org/701531 (as I implemented the py35 workaround) I've reduced my vote on that to +1, it would be great if you could have a second look on that | 12:13 |
*** bhavikdbavishi has joined #zuul | 12:15 | |
tobiash | zuul-maint: I'd have an easy review that fixes timeouts waiting for logger issues with windows jobs: https://review.opendev.org/717973 | 12:15 |
swest | tristanC: thanks a lot! | 12:16 |
*** rfolco has joined #zuul | 12:17 | |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: Support per branch change queues https://review.opendev.org/718531 | 12:19 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Move queue from pipeline to project https://review.opendev.org/720182 | 12:19 |
*** Goneri has joined #zuul | 12:21 | |
tobiash | tristanC: this would be an easy review as well: https://review.opendev.org/718397 (fixes disabled tls verification for test environments) | 12:28 |
tobiash | tristanC: thanks! | 12:32 |
*** jpena|lunch is now known as jpena | 12:33 | |
*** bhavikdbavishi has quit IRC | 12:36 | |
openstackgerrit | Merged zuul/nodepool master: Add debug info on json parse error of image upload https://review.opendev.org/716566 | 12:39 |
openstackgerrit | Tobias Henkel proposed zuul/nodepool master: Logs stats for nodepool automated cleanup https://review.opendev.org/614074 | 12:42 |
*** yolanda has quit IRC | 12:43 | |
openstackgerrit | Merged zuul/zuul master: Ensure correct cleanup on repo update and reset https://review.opendev.org/701531 | 12:46 |
openstackgerrit | Jan Kubovy proposed zuul/zuul master: Scheduler's pause/resume functionality https://review.opendev.org/709735 | 12:47 |
openstackgerrit | Jan Kubovy proposed zuul/zuul master: Separate connection registries in tests https://review.opendev.org/712958 | 12:47 |
openstackgerrit | Jan Kubovy proposed zuul/zuul master: Prepare Zookeeper for scale-out scheduler https://review.opendev.org/717269 | 12:47 |
openstackgerrit | Jan Kubovy proposed zuul/zuul master: Improve typings for driver event ingestion https://review.opendev.org/717297 | 12:47 |
openstackgerrit | Jan Kubovy proposed zuul/zuul master: WIP: Driver event ingestion https://review.opendev.org/717299 | 12:47 |
*** bhavikdbavishi has joined #zuul | 12:49 | |
*** rpittau|bbl is now known as rpittau | 12:49 | |
openstackgerrit | Tobias Henkel proposed zuul/nodepool master: Logs stats for nodepool automated cleanup https://review.opendev.org/614074 | 12:51 |
*** gtema has quit IRC | 13:14 | |
openstackgerrit | Merged zuul/nodepool master: k8s/OKD Provider: Don't Set ca_cert if TLS verification is skipped https://review.opendev.org/718397 | 13:15 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add --validate-tenants option to zuul scheduler https://review.opendev.org/542160 | 13:22 |
andreykurilin | hi folks! I have a question regarding secrets: if the secret should be defined in a project where it is used, why the name of secret should be unique per all projects? | 13:24 |
andreykurilin | I mean there is project A that defines and uses secret with name XXX and there is another project B that would like to setup the secret with name XXX for it own purpose... zuul currently returns 2 errors to me (for project B): first one says that XXX secret is defined in another project and the second one says that I cannot use secret XXX since it should be defined in the same project... | 13:27 |
*** gtema has joined #zuul | 13:29 | |
*** sgw has joined #zuul | 13:32 | |
*** ysandeep|rover is now known as ysandeep|away | 13:32 | |
*** gtema_ has joined #zuul | 13:32 | |
*** gtema has quit IRC | 13:36 | |
mordred | andreykurilin: that's an excellent question. I think the first part is simply that all the other objects share a global namespace - I'm not sure how feasible it would be at the moment in the configparser to scope them to a project ... although your reasoning about their usage certainly makes logical sense. we'll probably want to ask corvus about that when he wakes up | 13:37 |
*** bhavikdbavishi has quit IRC | 13:37 | |
corvus | clarkb, AJaeger: see comments on https://review.opendev.org/717663 | 13:43 |
andreykurilin | mordred: good. at least that is not I'm doing something completely wrong :) thanks! PS: I reserved dockerhub_credentials secret name (devil) at openstack tenant | 13:43 |
AJaeger | corvus: agreed, let's point it out to ianw as well ^ | 13:44 |
corvus | andreykurilin, mordred: i think that covers it -- they're just not special. andreykurilin, but since they aren't special, you might consider sticking to the openstack naming scheme and prefixing that with your project name: https://docs.opendev.org/opendev/infra-manual/latest/drivers.html#consistent-naming-for-zuul-jobs | 13:45 |
corvus | andreykurilin: for example, here's zuul's own dockerhub secret: https://opendev.org/zuul/zuul/src/branch/master/.zuul.yaml#L111 | 13:47 |
corvus | tobiash: i don't remember the problem you mention on 701531 | 13:49 |
andreykurilin | corvus: It's mine! I was first! my precious... | 13:49 |
andreykurilin | PS: I will rename it for sure to avoid confusions. I would name it right from the beginning but made an assumption that if it can be used only by project where it is defined, it should not follow naming convention since it is inner thing | 13:49 |
andreykurilin | thanks | 13:50 |
*** gtema_ has left #zuul | 13:50 | |
tobiash | corvus: you mean the problem this attempts to fix or the problem the old git version had? | 13:50 |
corvus | andreykurilin: yeah, makes sense | 13:50 |
corvus | tobiash: something about breaking the py35 job? | 13:50 |
tobiash | corvus: the problem with the py35 job was that it used an old git versions that leaks empty directories, hence the workaround: https://review.opendev.org/#/c/701531/8/zuul/merger/merger.py line 315 (which is the part I added) | 13:51 |
corvus | andreykurilin, mordred: it may be more intuitive to scope them to the project, but then we'll have two different kinds of config objects (scoped and unscoped), which is also a bit of complexity for users to know. and if we ever figure out a way to safely let projects share secrets, that could be a problem. all of that to say, i think we'd want to think carefully about scoping it. | 13:51 |
corvus | tobiash: ooooooooooooh! i get it, you're saying you don't feel entitled to a +2 vote on that because of your contribution. thanks for letting us know, but i'm not too worried about that. :) | 13:53 |
tobiash | corvus: yes, exactly ;) | 13:53 |
corvus | (for a minute, i thought there was some other change with a py35 fix) | 13:54 |
tristanC | yeah me too, got me worried for a bit :-) | 13:55 |
tobiash | I just wanted to comply with the 2x +2 vote rule and my +2 in this case would be actually +1.7 or so ;) | 13:56 |
corvus | it rounds up ;) | 13:56 |
corvus | tobiash: speaking of -- https://review.opendev.org/718708 could use a +3 | 13:56 |
tobiash | corvus: yeah, I rechecked it with the intention to +3 afterwards :) | 13:57 |
corvus | and hey, it looks like the zk tls stuff merged :) | 13:57 |
tobiash | yes :) | 13:57 |
corvus | i think maybe the next steps are that we should start using that in opendev, then release a 3.x with that, and ask people to start using it | 13:58 |
*** y2kenny has joined #zuul | 13:58 | |
tobiash | ++ | 13:58 |
mordred | corvus: ++ | 13:59 |
tobiash | corvus: re moving queue to project, this is a first stab which is a surprisingly small change and passes all but one tests so far: https://review.opendev.org/720182 | 13:59 |
mordred | corvus: do you think it's worth trying to ansiblify our install? we've got the docker+afs thing blocking using docker, but I could update my ansible patch to just pip install zuul on the executor | 14:00 |
mordred | corvus: or would you rather not conflate the two goals for now? | 14:00 |
tobiash | corvus: we would then rebase the cyclic dep change on that and adapt it to leverage that | 14:00 |
zbr | part of ensure rename: https://review.opendev.org/#/c/720145/ | 14:04 |
corvus | mordred: hrm; if we're going to be stuck at the afs question for a while, maybe that is worth it | 14:08 |
corvus | mordred: -> #opendev | 14:08 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Move queue from pipeline to project https://review.opendev.org/720182 | 14:10 |
y2kenny | is there a way to post a URL when a job is queued? I see the success and failure-url but not sure if there's one for job queued or started. I noticed there's the web.status_url in zuul.conf but setting it doesn't seem to do what I thought it would do. | 14:18 |
corvus | y2kenny: do you want to leave a message in gerrit when a job is queued? | 14:19 |
y2kenny | yes | 14:19 |
y2kenny | pretty much pointing folks to zuul-web so that people can look at the streamer if they want | 14:21 |
AJaeger | y2kenny: you want an URL like http://zuul.opendev.org/t/zuul/status/change/720182,6 for https://review.opendev.org/720182 ? | 14:22 |
y2kenny | AJaeger: oh yes! I actually didn't know that type of pages exists | 14:23 |
y2kenny | I always click into the stream log but the example you gave is even better | 14:23 |
AJaeger | y2kenny: click on the icon to the left of the change in the webui | 14:23 |
AJaeger | y2kenny: so, on http://zuul.opendev.org/t/zuul/status there's a Green circle with a tick mark besides 720182, click on it ;) | 14:24 |
AJaeger | unfortunately not easy to find ;( | 14:24 |
openstackgerrit | Merged zuul/zuul master: OIDCAuthenticator: add capabilities, scope option https://review.opendev.org/702275 | 14:25 |
corvus | y2kenny: zuul only reports to gerrit on entire items (changes), not individual jobs; so for things like that, you'll want to look at the pipeline config rather than the job config. https://zuul-ci.org/docs/zuul/reference/pipeline_def.html#attr-pipeline.start should be close to what you describe | 14:28 |
corvus | y2kenny: note also the one above "enqueue" though it will run that even if no jobs are run, so it may not be quite what you're looking for. | 14:28 |
y2kenny | corvus: great thanks. I will take a look | 14:30 |
*** ysandeep|away is now known as ysandeep | 14:31 | |
y2kenny | AJaeger: ah yes, I see it now. | 14:34 |
y2kenny | thanks for the tip | 14:35 |
openstackgerrit | Merged zuul/nodepool master: Add requires to zuul-quick-start job https://review.opendev.org/718708 | 14:36 |
openstackgerrit | Sorin Sbarnea proposed zuul/zuul master: Force pre blocks to wrap text https://review.opendev.org/681532 | 14:53 |
openstackgerrit | Sorin Sbarnea proposed zuul/zuul master: Force pre blocks to wrap text https://review.opendev.org/681532 | 14:55 |
zbr | does anyone know why we print the stdout/stderr abbreviated output using <span> intead of <pre> inside renderFailedTask ? | 15:00 |
corvus | tristanC: ^ | 15:01 |
zbr | printing console output as "text" seems weird to me, as we also loose the fixed-width font due to this. | 15:01 |
zbr | is quite funny because msg and exc are printed with pre, but not stdout/stderr. | 15:03 |
zbr | asking because if there is no particular reason, I would like to make it a use <pre>. | 15:03 |
tristanC | zbr: corvus: no particular reason | 15:04 |
openstackgerrit | Sorin Sbarnea proposed zuul/zuul master: WIP: Improve display of stdout/stderr task errors https://review.opendev.org/681532 | 15:09 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: Support per branch change queues https://review.opendev.org/718531 | 15:11 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Move queue from pipeline to project https://review.opendev.org/720182 | 15:11 |
*** ianychoi has quit IRC | 15:13 | |
zbr | tristanC: corvus: tranks, i will ping you when I have the desired rendering achieved. | 15:15 |
openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add zuul-preview service https://review.opendev.org/718419 | 15:16 |
y2kenny | Are there any documentations on replacement fields like {change.number} {build.uuid}, etc? I tried searching and couldn't find any. I think I sumbled upon it buy searching the source and finding some test case | 15:17 |
y2kenny | by* | 15:17 |
openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add zuul-operator-upload-image job https://review.opendev.org/708860 | 15:17 |
zbr | it is really "env" lookup plugin banned from being used on zuul? what would a workaround for {{ lookup('env', 'HOME') }} ? | 15:22 |
clarkb | corvus: tristanC I guess the earlier ensure-pip changes didn't catch it because sf ci isn't yet using ensure-pip but does use ensure-tox? | 15:24 |
tristanC | clarkb: that is correct | 15:24 |
openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: test: refactor run tasks to include file https://review.opendev.org/718158 | 15:25 |
clarkb | tristanC: ok just making sure we weren't missing something else there. Thanks for confirming | 15:25 |
clarkb | zbr: I think it would only be blocked on localhost but should work on remotes? Unless lookup plugins are somethign that always runs within the ansible process on localhost | 15:26 |
zbr | lookup env always runs on controller, well documented. | 15:26 |
tristanC | clarkb: ftr here is the current list of files triggering tox job test: https://softwarefactory-project.io/cgit/third-party-ci-jobs/tree/zuul.d/legacy.yaml#n139 | 15:26 |
tristanC | and we are going to improve the third-party-ci-jobs we do on zuul-jobs shortly to improve and increase test coverage with kubernetes pods | 15:27 |
zbr | i am trying to make a role compatible with zuul, already got rid of a custom filter, but i reached the env one now. | 15:27 |
avass | zbr: why do you need the home directory on the controll node? | 15:28 |
tristanC | zbr: when would you need to know the HOME environment of the executor user? | 15:28 |
zbr | long story.... https://opendev.org/openstack/ansible-role-collect-logs/src/branch/master/defaults/main.yml#L3 | 15:29 |
*** Goneri has quit IRC | 15:30 | |
zbr | i guess i need to use a zuul exposed var when present to avoid it. | 15:30 |
clarkb | zbr: ya beacuse that won't be the correct path from the zuul executor standpoint necessarily | 15:30 |
clarkb | I think in our case currently it would be if the lookup was allowed to run but nothing requires zuul's executor process to run with a user that has the same homedir as the remote test nodes | 15:31 |
*** Goneri has joined #zuul | 15:33 | |
avass | This is ready if anyone want to take a look at it: https://review.opendev.org/#/c/709292/23 | 15:42 |
openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: dhall-diff: add new job https://review.opendev.org/718694 | 15:44 |
zbr | clarkb: for that part case it was easy to use zuul_work_dir but i am afraid that there is a lot of lookup(env,...) usage along the role, not sure how to address the others. | 15:47 |
*** ianychoi has joined #zuul | 15:47 | |
zbr | probably it would be easier if env would always return nothing instead of raising an exception. | 15:48 |
zbr | i already have cased with fallback to a default value. | 15:48 |
clarkb | zbr: the issue is one of security concern so its better to error I think | 15:51 |
zbr | clarkb: i will probably find out soon the extend of the issue. I wonder if ansible_env can be used instead. | 15:54 |
clarkb | zbr: if ansible_env is populated as a fact from the remote host it should work | 15:55 |
zbr | yep, but the intended use was to load vars from the controller, as the playbook logic can vary based on them. | 15:56 |
clarkb | zbr: in a zuul context that won't be valid | 15:57 |
zbr | if it does not work, i will probably have to contain all unsafe lookups into a vars file which is loaded only when outside zuul. | 15:57 |
clarkb | the homedir of the controller is completely independent of the remote | 15:58 |
zbr | homedir issue already sorted, i was worried about the rest, but i think I have an idea about what can be done | 15:59 |
corvus | i don't recall the security concern with env; if there isn't one, we may be able to allow it... | 16:00 |
zbr | it can be a security issue if we populate stuff like PYPI_PASSWORD, or similar. | 16:00 |
avass | corvus: api keys | 16:00 |
openstackgerrit | Jan Kubovy proposed zuul/zuul master: Scheduler's pause/resume functionality https://review.opendev.org/709735 | 16:01 |
openstackgerrit | Merged zuul/zuul-jobs master: Adds roles to install and run hashicorp packer https://review.opendev.org/709292 | 16:01 |
corvus | avass: like running zuul-executor with that set globally? | 16:01 |
zbr | maybe a more friendly approach would be to whitelist/blacklist specific variables, so we would allow some to be exposed but not all. | 16:02 |
corvus | i'd be scared to do that; i'd rather rely on trusted_ro_path to get private information to trusted jobs | 16:02 |
corvus | (basically, the blocked ansible plugins are our first line of defense, but they're also our weakest) | 16:03 |
avass | corvus: right, that's probably only set for the executor process | 16:03 |
corvus | yeah, i don't enven know if we pass through env vars to bwrap and ansible | 16:03 |
openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: test: refactor run tasks to include file https://review.opendev.org/718158 | 16:04 |
corvus | oh, yep we do pass them through | 16:04 |
corvus | so yeah, you could run "SECRET_ENV_VAR=apikey zuul-executor", and it'll show up in jobs. but i wouldn't recommend it, i'd recommend writing the secret to a file and using trusted_ro_paths to pull that file into the bwrap for trusted playbooks | 16:05 |
corvus | at the very least, if we did want to consider allowing env, that means we should certainly discuss/announce it well in advance :) | 16:06 |
avass | corvus: I think we're using config interpolation with environment variables for an api key to gitlab :) | 16:06 |
tobiash | I think if we want to allow that we should filter the env vars for a known set of vars before running ansible | 16:08 |
tristanC | avass: corvus: that's all the case for ZUUL_DB_URI | 16:08 |
corvus | avass: oh, so you're accidentally making your environment available to the executor | 16:08 |
corvus | or rather to ansible | 16:08 |
corvus | you just want the executor to see that env var, but it's propagating into the job too, which is farther than you want | 16:08 |
tristanC | also* , it's actually a supported feature of zuul to support replacing secret in zuul.conf with environment variables | 16:08 |
avass | yep | 16:09 |
tobiash | we're already setting up a distinct set of env vars for each ansible run: https://opendev.org/zuul/zuul/src/branch/master/zuul/executor/server.py#L2120 | 16:09 |
corvus | right, but we start with what the executor sees | 16:09 |
tobiash | so we should filter them or start from a clean set probably | 16:09 |
corvus | yep | 16:09 |
corvus | all of the variables we interpolate into the config are prefixed with ZUUL_ right? | 16:10 |
avass | I think so | 16:10 |
openstackgerrit | Tristan Cacqueray proposed zuul/nodepool master: config: add missing voluptuous required attributes https://review.opendev.org/719094 | 16:10 |
corvus | so we could filter out ZUUL_ variables there to prevent env-config-var-interpolation from leaking into jobs | 16:11 |
tristanC | avass: corvus: yes, at least those need to be filter out | 16:11 |
corvus | ideally before someone figures out how to work around the env plugin restriction since that's our only line of defense for that right now :) | 16:12 |
avass | I can take a look at it later, doesn't sound like it should be hard to fix :) | 16:14 |
corvus | avass: cool, thanks :) | 16:14 |
avass | But that does mean that we're probably able to configure ansible callbacks with env vars heh | 16:16 |
*** gtema has joined #zuul | 16:17 | |
*** jcapitao has quit IRC | 16:20 | |
*** gtema has quit IRC | 16:22 | |
openstackgerrit | Albin Vass proposed zuul/zuul master: Filter out environment config variables https://review.opendev.org/720238 | 16:23 |
avass | corvus: send you a private message | 16:31 |
*** rpittau is now known as rpittau|afk | 16:33 | |
openstackgerrit | Sorin Sbarnea proposed zuul/zuul master: WIP: Improve display of stdout/stderr task errors https://review.opendev.org/681532 | 16:36 |
*** evrardjp has quit IRC | 16:37 | |
*** evrardjp has joined #zuul | 16:37 | |
*** jpena is now known as jpena|off | 17:10 | |
openstackgerrit | Albin Vass proposed zuul/zuul master: Filter secret ZUUL_ env variables from ansible env https://review.opendev.org/720238 | 17:11 |
mnaser | should we warn users if they push up a change where they have both zuul.d/ and .zuul.yaml in there? maybe refuse to land those? | 17:13 |
mnaser | i've been hit by this | 17:13 |
SpamapS | mnaser: isn't there a defined order? | 17:15 |
SpamapS | (that's what I would have assumed) | 17:15 |
mnaser | SpamapS: maybe? but i think if you have zuul.d folder, .zuul.yaml seems to be completely ignored from my experience | 17:16 |
zbr | clarkb: apparently we need to do something about env, take a look at https://zuul.opendev.org/t/openstack/build/81c14f81eadc4ba68cda4e8be11e782e | 17:17 |
zbr | An unhandled exception occurred while templating '{{ zuul_work_dir | default(lookup('env', 'HOME')) }}/.quickstart'. | 17:17 |
zbr | if this failed with exception, i have no idea how to make it work, apparently the lazy-rendering does not work in that case. | 17:18 |
tobiash | mnaser, SpamapS: yes, there is a defined order but no notification when one file is ignored afaik | 17:18 |
fungi | SpamapS: mnaser: yes, the documentation states the order they're considered in, but i agree it's potentially confusing | 17:18 |
tobiash | I guess a warning in this case might make sense | 17:18 |
mnaser | there is a warning, but only in the zuul log | 17:18 |
mnaser | "2020-04-14 11:25:21,394 WARNING zuul.TenantParser: Multiple configuration files in ...." | 17:19 |
tobiash | mnaser: then it's probably easy to plumb this warning through to the user | 17:19 |
fungi | i think so far we've avoided providing non-fatal warnings in feedback on proposed changes | 17:19 |
tobiash | we have the mechanisms for that in place | 17:19 |
tobiash | (see warnings attribute on the queue item or buildset) | 17:20 |
fungi | ahh, right that. i don't think we've generally propagated those into review comments though | 17:20 |
* mnaser is a little ENOTIME these days | 17:21 | |
mnaser | i can't get into this, but something to keep in mind i guess | 17:21 |
fungi | i could see making it a tenant configuration option though, whether to reject changes which would shadow a zuul config with another zuul config at a different path | 17:22 |
SpamapS | A warning in the UI just like the job config errors would be ideal IMO. | 17:26 |
*** yolanda has joined #zuul | 17:27 | |
openstackgerrit | James E. Blair proposed zuul/nodepool master: Mark dib as deleting before erasing files from disk https://review.opendev.org/720247 | 17:29 |
corvus | clarkb, fungi, mordred: ^ | 17:29 |
fungi | thanks corvus! | 17:30 |
fungi | SpamapS: you mean the alarm bell notification on the dashboard, that could be reasonable too yes | 17:30 |
*** y2kenny has quit IRC | 17:30 | |
fungi | er, that was meant to be a question | 17:31 |
SpamapS | fungi: yeah, I found those to be massively useful in cleaning up trash. | 17:31 |
tristanC | corvus: not sure why, but i now get a cryptic `This change depends on a change with an invalid configuration.` on https://review.opendev.org/718158 . Have this reach a limit of acceptable speculative change ? | 17:32 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Warn user when dynamic layout ignores zuul config https://review.opendev.org/720249 | 17:32 |
corvus | tristanC: you're probably hitting a long-standing bug we've been trying to track down | 17:32 |
tobiash | mnaser, SpamapSthat should do it ^ | 17:32 |
corvus | we added some more debugging logs for it; i'm unsure if opendev is running with that though | 17:32 |
corvus | tristanC: i have reached the limit of the number of fires i can fight at one time right now, so i can't check the logs :( | 17:33 |
tristanC | corvus: no worries, but i think this time it's different, previously it was 'unknown error' iirc | 17:33 |
tristanC | oh and one more recheck made it through, it's building again | 17:34 |
openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Check if pip is preinstalled before installing it https://review.opendev.org/720254 | 17:36 |
corvus | tristanC: ^ can you look at that | 17:37 |
tristanC | sure | 17:38 |
*** gtema has joined #zuul | 17:47 | |
*** dpawlik has quit IRC | 17:48 | |
*** gtema has quit IRC | 17:51 | |
clarkb | tristanC: apparently python3-pip on centos8 does not install `pip` | 17:58 |
clarkb | so that assumption is bad | 17:58 |
clarkb | working on a fix now | 17:58 |
openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Check if pip is preinstalled before installing it https://review.opendev.org/720254 | 18:00 |
clarkb | our testing actually has pretty good coverage of that stuff across distros looks like. Except that most of our images come preinstalled with pip | 18:00 |
openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: test: refactor run tasks to include file https://review.opendev.org/718158 | 18:05 |
openstackgerrit | Merged zuul/nodepool master: Mark dib as deleting before erasing files from disk https://review.opendev.org/720247 | 18:26 |
AJaeger | clarkb: is https://review.opendev.org/#/c/720257/1 addressing corvus' comment from https://review.opendev.org/#/c/717663/ ? | 18:34 |
openstackgerrit | Tristan Cacqueray proposed zuul/nodepool master: config: add missing voluptuous required attributes https://review.opendev.org/719094 | 18:35 |
mordred | AJaeger: https://review.opendev.org/720254 is | 18:35 |
AJaeger | mordred: wrong paste in my change - thanks | 18:36 |
openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: merger: replace custom message logger https://review.opendev.org/720266 | 18:54 |
openstackgerrit | Tristan Cacqueray proposed zuul/nodepool master: config: add environment variable substitution https://review.opendev.org/719599 | 18:54 |
*** sassyn has joined #zuul | 19:02 | |
sassyn | Hi All Again, | 19:02 |
openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: URLTrigger driver time based https://review.opendev.org/635567 | 19:03 |
*** hashar has quit IRC | 19:04 | |
sassyn | I have a question on the pipeline. Consider the following: User X push a change to x, the zuul trigger this change, run the job and on success will mark it as Verified +1. User Y go to the gerrit web interface and then do a code review +2 on this patch of user X. Zuul then will run the JOB and if all goes OK will merge the code. | 19:05 |
fungi | go on | 19:06 |
sassyn | hi fungi :-) | 19:06 |
fungi | hi ;) | 19:07 |
sassyn | my questions: 1. the test as you can see here running two times, for the verified and for the merge. If I have patch X1, X2, X3, X4, X5 , it means I run it 5 times for verified, and then X1, X1+X2, X1+X2+X3, X1+X2+X3+X4, X1+X2+X3+X4+X5 = total 10 times | 19:08 |
fungi | that sounds right. zuul is designed to test each patch so it can identify if one of them has a problem | 19:09 |
fungi | and you can't assume that the test from when the patch was proposed will yield the same result later when it's approved | 19:10 |
*** bhavikdbavishi has joined #zuul | 19:10 | |
sassyn | But why I should run the X1...X5 in stand alone if I run it after | 19:10 |
fungi | because the state of the target branch in the repository may have changed over that time, and also any patches approved before it which haven't merged yet could introduce something which breaks that patch | 19:10 |
fungi | you don't *have* to run changes when they're first proposed, it's usually just done as feedback to reviewers | 19:11 |
fungi | er, don't have to run tests for patches when they're proposed | 19:11 |
fungi | for most code review based workflows, reviewers do want to know if a proposed patch is passing tests though | 19:12 |
sassyn | my problem is that my jobs takes a lot of time ~ 45 min. | 19:12 |
clarkb | fwiw this is all configurable | 19:12 |
fungi | our average job runtime in opendev is around an hour, and we have some jobs which take 3-4 hours to complete | 19:13 |
clarkb | you should be able to run the jobs when it makes the most sense for your repos | 19:13 |
fungi | some of our tenants are also configured for our "gate" pipeline to supplant our "check" pipeline, so if jobs haven't completed in check yet when a change gets enqueued into gate, then the check jobs are aborted to free up resources | 19:13 |
AJaeger | corvus: want to review https://review.opendev.org/#/c/720254/ and then change your vote on https://review.opendev.org/#/c/717663/? (the pip ensure zuul-jobs changes)? | 19:14 |
*** sanjayu__ has quit IRC | 19:14 | |
fungi | sassyn: also we configure our gate pipeline to have a higher priority than check, so gate always gets all the available resources if it needs them, and check uses what's still available | 19:14 |
sassyn | but this mean the gate will never check patches which not being verifed | 19:15 |
fungi | how does it mean that? | 19:15 |
sassyn | gate will check only +2 reviewed +1 verified | 19:16 |
fungi | as clarkb says, that's configurable | 19:16 |
sassyn | others patch in the Q, will wait | 19:16 |
clarkb | AJaeger: corvus tristanC I can also depends on or rebase it such that tristanC's jobs will run. Though I think we can land the ensure-pip change first then recheck ensure-tox as nothing is using ensure-pip yet | 19:16 |
fungi | some of our tenants have a gate pipeline which does not depend on there being an existing verified +1 | 19:16 |
fungi | sassyn: ^ | 19:16 |
fungi | so changes get enqueued straight into it the moment they're approved, start running jobs there, and abort any builds for that change in the check pipeline as they're no longer relevant | 19:17 |
fungi | sassyn: here's an example... https://opendev.org/opendev/project-config/src/branch/master/zuul.d/pipelines.yaml#L30-L67 | 19:19 |
sassyn | the critical issue, is that when a user X push a patch X, the job is running... and get verified. Then or in parallel other user Y also pushed a patch Y which got verified. Now I need to submit patch X, and Reasbe patch Y (which will trigger again a verified JOB), and then do the submit. | 19:19 |
fungi | note the absence of a check for verified:1 and also the supercedes:check directive | 19:19 |
sassyn | Yes | 19:20 |
sassyn | I read about it... | 19:20 |
fungi | sassyn: why do you need to rebase? | 19:20 |
sassyn | cause I'm in fast forward git repo | 19:20 |
AJaeger | clarkb: let's land and recheck. I'll +2 and give corvus a chance to review... | 19:20 |
fungi | sassyn: oh, that sounds like a self-imposed choice for inefficiency in that case. you get a prettier git log, but at the expense of constant rebasing | 19:21 |
AJaeger | clarkb: but go ahead and +A if you want | 19:21 |
fungi | sassyn: it's one of the reasons we use merge-if-necessary strategy for our projects | 19:21 |
fungi | the history is still effectively linear if you ignore the single-change merge commit glue (and git log --no-merges is an easy way to filter those out when skimming history) | 19:22 |
sassyn | U know too much :-) | 19:23 |
sassyn | but what if I want to keep fast forward | 19:23 |
sassyn | what would u recommend? | 19:24 |
fungi | sassyn: then you'll run a lot of additional jobs and get very little benefit from dependent pipelines | 19:24 |
sassyn | can u please explain more? | 19:25 |
fungi | i would consider fast-forward mode in gerrit to be a (poor) alternative to what zuul does for you by ensuring changes are tested in the order they merge | 19:25 |
fungi | can you explain what perceived benefits you gain from gerrit forcing you to rebase changes if they merge in a different order than you propose them? | 19:26 |
sassyn | well, I can't say | 19:27 |
sassyn | I need to ask the HEAD of the R&D | 19:27 |
sassyn | this was his call | 19:27 |
fungi | ahh, basically most of zuul's features aren't much use if you're forced to use gerrit in that mode | 19:28 |
AJaeger | sassyn: education him/her | 19:28 |
fungi | like for example if a change fails in a dependent pipeline, you're going to have to rebase everything, zuul won't be able to just restart testing without the failing commit | 19:28 |
*** bhavikdbavishi has quit IRC | 19:29 | |
fungi | fast forward mode in gerrit seems like it would be a hindrance for any project with more than one developer working on more than one series of changes at a time | 19:30 |
fungi | zuul is designed to multiplex your development activity | 19:30 |
*** bhavikdbavishi has joined #zuul | 19:31 | |
fungi | this is effectively the entire reason it was developed | 19:31 |
sassyn | fungi I agree with u | 19:32 |
fungi | gerrit and git were also designed with that in mind, which is really why merge commits exist in the first place | 19:32 |
sassyn | true, fast-forward is like SVN | 19:33 |
sassyn | old school | 19:33 |
openstackgerrit | Merged zuul/nodepool master: diskimage.username setting was not read from configuration file https://review.opendev.org/719191 | 19:35 |
sassyn | but If we will leave it aside for a moment, the verified +1 for patch A, and then Review +2 will trigger again the Job in merge if needed. not for fast forwarder | 19:35 |
sassyn | this is what I understand | 19:37 |
openstackgerrit | Merged zuul/nodepool master: Test alternative username in unit tests https://review.opendev.org/720101 | 19:44 |
*** yolanda has quit IRC | 19:50 | |
fungi | sassyn: yes, on zuul's side, it will try to merge the change to (its copy of) the branch and run jobs on it | 19:55 |
fungi | i know it also consults gerrit to find out if the change meets gerrit's requirements to submit, so it's possible that in ff-only mode gerrit may say any patchset which isn't parented to the branch tip isn't fit to submit, in which case zuul may just leave a verify -2 or similar and say something like "this change cannot be merged to the repository" (like it would for an actual merge conflict) | 19:57 |
mordred | once we have zuul-push, setting merge-mode to cherry-pick should allow both "clean tree" and taking advantage of zuul's multiplexing | 20:02 |
*** yolanda has joined #zuul | 20:03 | |
fungi | gerrit's configured merge-mode becomes irrelevant once zuul is pushing the branch state into it directly, doesn't it? | 20:04 |
mordred | fungi: a bit yeah | 20:08 |
mordred | fungi: although if you have gerrit set to ff-only and zuul to merge, gerrit is going to grump at you | 20:09 |
*** jkt has quit IRC | 20:09 | |
*** jkt has joined #zuul | 20:10 | |
fungi | yeah | 20:13 |
*** harrymichal has joined #zuul | 20:16 | |
openstackgerrit | Albin Vass proposed zuul/zuul master: Filter secret ZUUL_ env variables from ansible env https://review.opendev.org/720238 | 20:37 |
avass | ^ wanted to test that without running ansible to make it a bit faster but I couldn't figure out a good way of doing it | 20:39 |
sassyn | fungi : Thank You! | 20:39 |
sassyn | is it possible to do the a reverse check? so if A, B, C, D patch are pushed, I will run a job on A + B + C +D, and not on A, A+B, A+B+C? so if A+B+C+D will pass, it will be merge? and if it fails, then I will run A+B+C... and so on.. | 20:41 |
fungi | sassyn: i don't think there's any support for that in zuul, it's designed to make sure that every point in your git history passes configured jobs, and not skip testing some commits just because they're approved around the same point in time | 20:45 |
fungi | this is so that you can more reliably git bisect and expect that every point you're checking was tested as it merged | 20:46 |
fungi | and also so that if someone pulls from the repo in the middle of that batch of changes merging, they don't wind up with broken/untested source code | 20:47 |
clarkb | ya zuul doesn't do batching | 20:48 |
clarkb | it is likely possible to write a pipeline manager that does though | 20:49 |
clarkb | I'm not sure how you'd know when to create a batch. Maybe on a time interval? Basically every minute clump changes together or something (probably makes sense to make that user configurable) | 20:51 |
openstackgerrit | Albin Vass proposed zuul/zuul master: Filter secret ZUUL_ env variables from ansible env https://review.opendev.org/720238 | 21:00 |
*** harrymichal has quit IRC | 21:07 | |
openstackgerrit | Albin Vass proposed zuul/zuul master: Filter secret ZUUL_ env variables from ansible env https://review.opendev.org/720238 | 21:10 |
*** bhavikdbavishi has quit IRC | 21:28 | |
mordred | sassyn: it's worth noting, in that case, zuul will run tests on a, a+b, a+b+c and a+b+c+d in parallel, so if a+b+c passes but a+b+c+d does not, you'll get a+b+c merged in the same amount of time (actually more quickly, since you don't have to wait for a+b+c+d to run,then fail, then a+b+c to run) | 21:34 |
avass | mordred: that's if you have the resources to do that | 22:05 |
openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: merger: replace custom message logger https://review.opendev.org/720266 | 22:05 |
clarkb | corvus: ianw did you want to weigh in on https://review.opendev.org/#/c/720254/ and whether we should rebase ianw's stack on it or just merge and recheck ianw's stack? | 22:11 |
mordred | avass: totally. but - you know - just buy more computers ;) | 22:13 |
corvus | if the computers are cheaper than engineers, that approach may have some success with folks in control of budget. however, sometimes computers are more expensive than people. | 22:14 |
corvus | clarkb: did that get tested in sf.io? | 22:15 |
clarkb | corvus: not yet. There are two ways we can do that I think, rebase it into ianw's stack (or depends on it) or merge it then recheck ianw's change | 22:16 |
clarkb | corvus: ensure-pip has landed but none of the consumer role changes have so its semi safe to land it then test. But it means we may have ot make more changes later | 22:16 |
corvus | i like the depends-on approach | 22:16 |
clarkb | k | 22:16 |
clarkb | working on that now | 22:16 |
corvus | clarkb: +1 on process grounds; i don't have enough stuff paged in to +2 the code, but trust you and the other reviewers on that. | 22:18 |
openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: ensure-tox: use ensure-pip role https://review.opendev.org/717663 | 22:18 |
openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Update Fedora to 31 https://review.opendev.org/717657 | 22:18 |
openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Make ubuntu-plain jobs voting https://review.opendev.org/719701 | 22:18 |
openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Document output variables https://review.opendev.org/719704 | 22:18 |
openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Python roles: misc doc updates https://review.opendev.org/720111 | 22:18 |
clarkb | tristanC: ianw corvus AJaeger ^ fyi 3pci from sf.io should test 717663 with the don't install pip if it is installed checks now | 22:18 |
tristanC | clarkb: we could also add ensure-pip to the files that trigger sf.io tox test job | 22:21 |
clarkb | tristanC: ya until its consumed its probably less useful but that would be a good change once the rest of the stack there lands | 22:21 |
*** rfolco has quit IRC | 22:24 | |
openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: ensure-pip: export ensure_pip_virtualenv_command https://review.opendev.org/718224 | 22:34 |
openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: fetch-zuul-cloner: use ensure-pip https://review.opendev.org/717882 | 22:34 |
openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: fetch-subunit-output test: use ensure-pip https://review.opendev.org/718225 | 22:34 |
openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: ensure-tox: use ensure-pip role https://review.opendev.org/717663 | 22:34 |
openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Update Fedora to 31 https://review.opendev.org/717657 | 22:34 |
openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Make ubuntu-plain jobs voting https://review.opendev.org/719701 | 22:34 |
openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Document output variables https://review.opendev.org/719704 | 22:34 |
openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Python roles: misc doc updates https://review.opendev.org/720111 | 22:34 |
*** Goneri has quit IRC | 22:47 | |
openstackgerrit | Merged zuul/zuul-jobs master: Check if pip is preinstalled before installing it https://review.opendev.org/720254 | 23:26 |
*** tosky has quit IRC | 23:39 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!