| *** jbadiapa has quit IRC | 00:04 | |
| *** holser has joined #oooq | 00:50 | |
| *** holser has quit IRC | 00:57 | |
| *** d0ugal has quit IRC | 01:38 | |
| *** rlandy has quit IRC | 01:50 | |
| *** d0ugal has joined #oooq | 01:53 | |
| *** holser has joined #oooq | 03:06 | |
| *** ykarel|away has joined #oooq | 03:08 | |
| *** holser has quit IRC | 03:10 | |
| *** ccamacho has quit IRC | 03:11 | |
| *** rfolco has quit IRC | 03:32 | |
| *** dsneddon has quit IRC | 03:46 | |
| *** udesale has joined #oooq | 04:00 | |
| *** ykarel|away has quit IRC | 04:08 | |
| *** surpatil has joined #oooq | 04:09 | |
| *** dsneddon has joined #oooq | 04:16 | |
| *** ykarel|away has joined #oooq | 04:25 | |
| *** ykarel|away is now known as ykarel | 04:25 | |
| *** soniya29 has joined #oooq | 04:29 | |
| *** dsneddon has quit IRC | 04:30 | |
| *** skramaja has joined #oooq | 04:47 | |
| *** marios has joined #oooq | 04:49 | |
| *** brault has joined #oooq | 05:00 | |
| *** dsneddon has joined #oooq | 05:03 | |
| *** saneax has joined #oooq | 05:03 | |
| *** brault has quit IRC | 05:05 | |
| *** ratailor has joined #oooq | 05:06 | |
| *** dsneddon has quit IRC | 05:08 | |
| *** jtomasek has joined #oooq | 05:13 | |
| *** redrobot has quit IRC | 05:25 | |
| * marios relocate + (work related) errand biab | 05:33 | |
| *** marios has quit IRC | 05:33 | |
| *** dsneddon has joined #oooq | 05:41 | |
| *** kopecmartin|off is now known as kopecmartin | 05:46 | |
| *** dsneddon has quit IRC | 05:47 | |
| *** jfrancoa has joined #oooq | 06:17 | |
| *** dsneddon has joined #oooq | 06:21 | |
| *** holser has joined #oooq | 06:22 | |
| *** dsneddon has quit IRC | 06:26 | |
| chandankumar | sshnaidm|afk: zbr https://hackmd.io/XtQpBZXxR4qd7Iine3WU-A?view I have updated the etherpad based on the comments given | 06:41 |
|---|---|---|
| chandankumar | please have a look | 06:41 |
| *** dsneddon has joined #oooq | 06:55 | |
| *** ykarel is now known as ykarel|lunch | 06:57 | |
| *** brault has joined #oooq | 06:57 | |
| *** dsneddon has quit IRC | 07:00 | |
| *** tesseract has joined #oooq | 07:11 | |
| *** dsneddon has joined #oooq | 07:26 | |
| *** dsneddon has quit IRC | 07:30 | |
| *** jpena|off is now known as jpena | 07:37 | |
| *** ccamacho has joined #oooq | 07:38 | |
| jfrancoa | chandankumar: Hello! we are observing this error in the Undercloud upgrades jobs in master -> http://pastebin.test.redhat.com/796179 , is this a known issue? I thought we already had Ansible 2.8 upstream.. | 07:39 |
| bogdando | hi folks | 07:41 |
| bogdando | I've managed to deploy tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates multinode by the new CI reproducer \o/, but only at the half (˚Õ˚)ر ~~~~╚╩╩╝ | 07:42 |
| bogdando | overcloud deployment seems blocked by https://bugs.launchpad.net/tripleo/+bug/1779263 http://104.130.124.113/irclogs/%23rdo/%23rdo.2018-06-29.log.html#t2018-06-29T05:55:57 | 07:42 |
| openstack | Launchpad bug 1779263 in tripleo ""AnsibleUndefinedVariable: 'dict object' has no attribute 'overcloud'"} at periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-master" [Critical,Fix released] - Assigned to Quique Llorente (quiquell) | 07:42 |
| bogdando | same stuff | 07:42 |
| bogdando | any help with that? | 07:42 |
| *** ykarel|lunch is now known as ykarel | 07:45 | |
| bogdando | ykarel: hi | 07:46 |
| bogdando | ^^ seems like you have had that issue in the past? | 07:46 |
| ykarel | bogdando, hi | 07:46 |
| bogdando | my nodepool.yaml is https://pastebin.com/yRaUTLsh | 07:47 |
| ykarel | looking | 07:47 |
| bogdando | it seems some inventories of configs are missing overcloud group? | 07:47 |
| *** dsneddon has joined #oooq | 07:48 | |
| bogdando | ykarel: some logs snippets from the failed job https://pastebin.com/kFyDhVfC | 07:49 |
| bogdando | ykarel: the repro script I'm using was from https://ed9e5f51095c33770faf-957e0323e7657520e3a7352c7d8ac713.ssl.cf2.rackcdn.com/677852/4/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/ed7c811/logs/reproducer-quickstart/ | 07:53 |
| *** dsneddon has quit IRC | 07:53 | |
| bogdando | at least it deployed undercloud... | 07:54 |
| bogdando | so I'm almost happy | 07:54 |
| bogdando | https://review.opendev.org/#/c/681182/ has resolved the multinode problems with CI repro | 07:54 |
| bogdando | that I used to have for the past week | 07:55 |
| bogdando | w/o that the job couldn't event get started | 07:55 |
| bogdando | please also consider merging https://review.opendev.org/#/q/topic:ci-repro+(status:open+OR+status:merged) | 07:56 |
| ykarel | bogdando, /me looks now, was looking into other issues | 07:59 |
| bogdando | ykarel: thanks. I'm just wondering where does that missing overcloud dict/group thingy come from | 08:00 |
| bogdando | My /etc/nodepool/sub_nodes_private is empty is also empty | 08:01 |
| *** jbadiapa has joined #oooq | 08:01 | |
| bogdando | the static nodepool provider uses the fallback to public ipv4 | 08:01 |
| bogdando | VMs has only a single eth0 running with IP | 08:02 |
| bogdando | may be that's somehow related, tho I'm not bought on that completly | 08:02 |
| ykarel | bogdando, can u share job definition, | 08:04 |
| ykarel | what nodeset is being used | 08:04 |
| bogdando | ykarel: sure, where does it sit, in executor container? | 08:04 |
| ykarel | bogdando, in local gerrit also u can see it | 08:05 |
| bogdando | oh indeed | 08:05 |
| ykarel | also in logs | 08:05 |
| *** marios_ has joined #oooq | 08:05 | |
| bogdando | ykarel: https://pastebin.com/uZWQ71DT | 08:06 |
| bogdando | could it be nodepool.yaml is also missing labels.subnodes:1 like it was in https://review.rdoproject.org/r/#/c/14520/2/nodepool/nodepool.yaml? | 08:06 |
| bogdando | ykarel: my nodepool.yaml is https://pastebin.com/yRaUTLsh | 08:06 |
| ykarel | bogdando, yes subnodes: 1 | 08:07 |
| ykarel | if it's not set, it's the issue | 08:07 |
| bogdando | hm, so need to adjust that jinja? | 08:07 |
| bogdando | it also doesn't provide diskimages ... is that bad? :) | 08:07 |
| bogdando | ok, I'll tweak that j2 for quickstart extras and retry, thanks! | 08:08 |
| ykarel | diskimage is for preparing image afaik, but u using already prepared images | 08:08 |
| bogdando | yea | 08:08 |
| bogdando | so labels... | 08:08 |
| bogdando | oh, that's in ansible-role-tripleo-ci-reproducer | 08:09 |
| *** sshnaidm|afk is now known as sshnaidm|ruck | 08:09 | |
| bogdando | sshnaidm|ruck: hi, new patch incoming! | 08:10 |
| bogdando | ^^ | 08:10 |
| sshnaidm|ruck | bogdando, ack | 08:10 |
| *** ykarel is now known as ykarel|meeting | 08:12 | |
| chandankumar | jfrancoa: I think ansible 2.8 is available in master , is the upgrade job is from rocky to master? | 08:13 |
| *** ykarel_ has joined #oooq | 08:14 | |
| *** ratailor_ has joined #oooq | 08:14 | |
| *** ratailor has quit IRC | 08:16 | |
| *** ykarel|meeting has quit IRC | 08:16 | |
| bogdando | sshnaidm|ruck, ykarel_: https://review.rdoproject.org/r/#/c/22187/ | 08:18 |
| bogdando | testing it locally now | 08:18 |
| *** ykarel_ is now known as ykarel|meeting | 08:18 | |
| *** derekh has joined #oooq | 08:20 | |
| *** dsneddon has joined #oooq | 08:22 | |
| *** dsneddon has quit IRC | 08:28 | |
| jfrancoa | chandankumar: sorry, I was looking for the log. It's appearing in Stein in my case (i'm not sure if the same is occuring in master) -> https://ed7fd98c829b7e882de3-db749892da1a15bc69477ce2379d5ff9.ssl.cf1.rackcdn.com/681123/2/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/40b7109/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz | 08:32 |
| *** akahat has joined #oooq | 08:35 | |
| *** marios_ has quit IRC | 08:38 | |
| sshnaidm|ruck | bogdando, is it ok to merge? https://review.rdoproject.org/r/#/c/22154/ | 08:40 |
| *** marios_ has joined #oooq | 08:45 | |
| *** dsneddon has joined #oooq | 08:55 | |
| *** dsneddon has quit IRC | 08:59 | |
| sshnaidm|ruck | chandankumar, where is tempest.html in jobs? Don't see it lately | 09:05 |
| *** marios_ is now known as marios | 09:14 | |
| chandankumar | sshnaidm|ruck: the file got missing from logs | 09:16 |
| chandankumar | sshnaidm|ruck: not sure why | 09:16 |
| chandankumar | it was suppose to be in logs directory | 09:16 |
| *** soniya29 has quit IRC | 09:18 | |
| chandankumar | sshnaidm|ruck: https://github.com/openstack/tripleo-ci/blob/master/toci-quickstart/config/collect-logs.yml#L281 | 09:19 |
| chandankumar | might be this was causing | 09:19 |
| chandankumar | proposing a patch | 09:19 |
| *** surpatil has quit IRC | 09:22 | |
| chandankumar | sshnaidm|ruck: https://review.opendev.org/#/c/681421/ | 09:25 |
| *** dtantsur|afk is now known as dtantsur | 09:30 | |
| *** dsneddon has joined #oooq | 09:31 | |
| *** ykarel_ has joined #oooq | 09:32 | |
| *** ratailor_ has quit IRC | 09:32 | |
| bogdando | sshnaidm|ruck: double checking that | 09:32 |
| *** ratailor_ has joined #oooq | 09:32 | |
| *** ykarel|meeting has quit IRC | 09:34 | |
| arxcruz | sshnaidm|ruck: still not working, do i need to change the invocation? | 09:35 |
| arxcruz | ./reproducer-zuul-based-quickstart.sh -ok arxcruz_pwl -f -w /tmp/arx-repro | 09:36 |
| arxcruz | that's how i'm calling the reproducer | 09:36 |
| *** dsneddon has quit IRC | 09:36 | |
| *** ykarel_ is now known as ykarel | 09:39 | |
| sshnaidm|ruck | arxcruz, not working - what? | 09:41 |
| arxcruz | sshnaidm|ruck: http://paste.openstack.org/show/775099/ | 09:41 |
| *** surpatil has joined #oooq | 09:42 | |
| sshnaidm|ruck | arxcruz, have you waited? | 09:47 |
| sshnaidm|ruck | arxcruz, are all containers up? | 09:47 |
| zbr | sshnaidm|ruck: can you please help me with https://review.opendev.org/#/c/681251/ ? mainly to confirm that it makes the report visible in zuul UI. | 09:47 |
| arxcruz | sshnaidm|ruck: yes, i waited, containers are up | 09:47 |
| sshnaidm|ruck | arxcruz, what is in http://localhost:9000 | 09:48 |
| arxcruz | sshnaidm|ruck: just the zuul page, with nothing in queue | 09:48 |
| sshnaidm|ruck | arxcruz, are you in tripleo-ci-testing tenant there? | 09:49 |
| sshnaidm|ruck | arxcruz, then reproducer worked, job itself doesn't run, check in gerrit patch what is the message | 09:49 |
| arxcruz | sshnaidm|ruck: tripleo-ci-reproducer | 09:49 |
| sshnaidm|ruck | arxcruz, check in http://localhost:8080/ | 09:50 |
| arxcruz | sshnaidm|ruck: yup, no message from zuul, just the commit | 09:50 |
| sshnaidm|ruck | arxcruz, what is your zuul.yaml (w/o passwords) | 09:50 |
| arxcruz | sshnaidm|ruck: checking here tripleo-ci-reproducer_gerritconfig_1 is not running | 09:51 |
| sshnaidm|ruck | it's ok | 09:51 |
| arxcruz | sshnaidm|ruck: http://paste.openstack.org/show/775100/ | 09:51 |
| arxcruz | sshnaidm|ruck: it's an ovb job btw | 09:52 |
| sshnaidm|ruck | zbr, how can I confirm? | 09:52 |
| zbr | sshnaidm|ruck: click on needed-by, go to molecule job and you should see a new artifact listed below called "molecule report". if you can load it, it works. | 09:56 |
| sshnaidm|ruck | arxcruz, do you have a key arxcruz_pwl? | 09:56 |
| arxcruz | sshnaidm|ruck: yes | 09:57 |
| zbr | sshnaidm|ruck: personally I don't like the new UI, even with my fix the report is far less visible than what it used to be but is much better than being totally hidden among log files. | 09:57 |
| arxcruz | sshnaidm|ruck: and it's on my rdo and opendev account as well | 09:58 |
| zbr | I would have preferred to make it visible as a tab like we have: summary, logs, and consoe. | 09:58 |
| sshnaidm|ruck | arxcruz, check that you use right gerrit usernames for both rdo gerrit and upstream | 09:58 |
| sshnaidm|ruck | arxcruz, look at scheduler logs if it has errors | 09:58 |
| arxcruz | sshnaidm|ruck: I do, it's my username | 09:59 |
| arxcruz | sshnaidm|ruck: sec | 09:59 |
| sshnaidm|ruck | zbr, yeah, I see molecule report | 09:59 |
| sshnaidm|ruck | arxcruz, save your zuul.yaml aside, create zuul.yaml like that: http://paste.openstack.org/show/775101/ and commit and merge it via gerrit | 10:02 |
| sshnaidm|ruck | arxcruz, then create a patch with you previous zuul.yaml and submit, that way you'll see zuul errors report | 10:03 |
| arxcruz | sshnaidm|ruck: weird, i'm not seeing a zuul service running, what's the name of the zuul service? executor or merger ? | 10:05 |
| zbr | sshnaidm|ruck: panda: in general is very useful to get an extra confirmation (vote) from another team mate on changes I propose upstream, so openstack-infra can see that is not a "personal" request from sorin ;) -- every request that had such support went in much faster, so thanks for the help. | 10:05 |
| sshnaidm|ruck | arxcruz, what do you mean? | 10:06 |
| *** dsneddon has joined #oooq | 10:07 | |
| arxcruz | sshnaidm|ruck: http://paste.openstack.org/show/775102/ | 10:07 |
| sshnaidm|ruck | zbr, np, voted | 10:07 |
| sshnaidm|ruck | arxcruz, what are you trying to do? | 10:07 |
| arxcruz | sshnaidm|ruck: trying to find where's failing | 10:08 |
| sshnaidm|ruck | arxcruz, it can be in scheduler, launcher and executor | 10:08 |
| sshnaidm|ruck | arxcruz, better to do as I wrote above | 10:08 |
| *** saneax has quit IRC | 10:11 | |
| arxcruz | sshnaidm|ruck: i just did, approve it ready for commit, but nothing happens | 10:11 |
| *** dsneddon has quit IRC | 10:12 | |
| arxcruz | sshnaidm|ruck: there are a lot of notifications like this http://paste.openstack.org/show/775104/ | 10:12 |
| *** saneax has joined #oooq | 10:12 | |
| *** dsneddon has joined #oooq | 10:15 | |
| *** d0ugal has quit IRC | 10:16 | |
| sshnaidm|ruck | arxcruz, hmm.. maybe last change broke it.. did you press submit button? | 10:18 |
| arxcruz | sshnaidm|ruck: yup | 10:18 |
| sshnaidm|ruck | arxcruz, and prepared a patch with your previous zuul.yaml? | 10:19 |
| arxcruz | sshnaidm|ruck: yes, but i believe it won't work anyway, because the base job has a failure, so zuul will not continue | 10:19 |
| arxcruz | Zuul encountered a syntax error while parsing its configuration in the | 10:19 |
| arxcruz | repo openstack/tripleo-ci on branch master. The error was: | 10:19 |
| zbr | i also raised https://bugzilla.redhat.com/show_bug.cgi?id=1751163 related to py36 from rhel7.7/centos7.7 missing selinux, maybe we have a chance to make it happen. | 10:19 |
| arxcruz | extra keys not allowed @ data['ansible-version'] | 10:19 |
| openstack | bugzilla.redhat.com bug 1751163 in python3 "python36 is missing selinux python bindings (preventing ansible use)" [Urgent,New] - Assigned to python-maint | 10:20 |
| sshnaidm|ruck | arxcruz, maybe, but I'd like to see zuul message in patch | 10:20 |
| arxcruz | sshnaidm|ruck: there is no message :( | 10:21 |
| sshnaidm|ruck | arxcruz, ok, I'll update zuul containers to newest version and let's see.. | 10:21 |
| arxcruz | sshnaidm|ruck: ok | 10:22 |
| arxcruz | sshnaidm|ruck: Zuul version: 3.3.2.dev60 5f5032cf for me btw | 10:22 |
| sshnaidm|ruck | zbr, do you know if this version supports "ansible_version" in jobs? ^^ | 10:23 |
| zbr | sshnaidm|ruck: not underline, ansible-version: 2.8 | 10:26 |
| zbr | look at other options, all uses dashes | 10:26 |
| zbr | afaik all decently recent versions of zuul support it, and clearly works on rdo and upstream. | 10:28 |
| *** d0ugal has joined #oooq | 10:30 | |
| chandankumar | marios: sshnaidm|ruck panda kopecmartin please have a look at another proposal https://hackmd.io/7iRphp5SR3G_GszH2bjswA?view | 10:32 |
| chandankumar | and have your comments also | 10:32 |
| chandankumar | related to skip list management | 10:32 |
| sshnaidm|ruck | zbr, I see, but I'm asking about 3.3.2 | 10:32 |
| sshnaidm|ruck | chandankumar, you take xml, not html | 10:33 |
| chandankumar | sshnaidm|ruck: ok updating it | 10:35 |
| sshnaidm|ruck | chandankumar, oops, it's exclude list | 10:35 |
| sshnaidm|ruck | chandankumar, but anyway, xml looks fine to be collected too | 10:35 |
| sshnaidm|ruck | it's not what overloads logs server | 10:36 |
| zbr | sshnaidm|ruck: based on https://zuul-ci.org/docs/zuul/releasenotes.html is clear does *not*, you need 3.7.0 or newer. | 10:36 |
| zbr | considering that current is 3.10.... | 10:36 |
| sshnaidm|ruck | zbr, ok, that explains | 10:37 |
| sshnaidm|ruck | arxcruz, ^^ | 10:37 |
| zbr | arxcruz: in fact you need minimum 3.9.0, based on release notes, as this is the one introducing support for ansible .28 | 10:37 |
| arxcruz | zbr: yeah, i remember see that | 10:37 |
| arxcruz | so we need to upgrade the zuul container | 10:38 |
| chandankumar | sshnaidm|ruck: ok | 10:38 |
| arxcruz | if we want to run ovb jobs | 10:38 |
| arxcruz | remind to myself, check the zuul notifications | 10:38 |
| marios | chandankumar: ack reading it but why not make a gerrit proposal for tripleo-specs | 10:39 |
| marios | chandankumar: or a story maybe better | 10:39 |
| marios | chandankumar: like i have to sign up there for commenting | 10:39 |
| marios | chandankumar: so the job that runs the skip list, is expected to fail all jobs. that is 'success'. if instead it passes some job, then it needs to 'fail' so we are alerted that a job is now passing from the skip list and we should remove it? | 10:39 |
| marios | chandankumar: do we really need to automate that? i don't think that part is really necessary | 10:40 |
| marios | chandankumar: i mean the bot to submit the patch for remove skip etc | 10:40 |
| sshnaidm|ruck | and all this because we don't have periodic update of containers for reproducer | 10:40 |
| marios | chandankumar: how often does this happen | 10:40 |
| marios | chandankumar: it is a great step that we have an alert if a test is now passing and it can be moved from skiplist | 10:40 |
| zbr | @oooq: https://bugzilla.redhat.com/show_bug.cgi?id=1719978 (private) nut sure what else I can say, maybe a CC on memo list.... | 10:42 |
| openstack | zbr: Error: Error getting bugzilla.redhat.com bug #1719978: NotPermitted | 10:42 |
| chandankumar | marios: in 2-3 months, if we looked at the fs021 then we update the tests. | 10:42 |
| chandankumar | marios: it is all manual | 10:42 |
| arxcruz | sshnaidm|ruck: job is running now :) | 10:55 |
| sshnaidm|ruck | arxcruz, \o/ | 10:56 |
| sshnaidm|ruck | arxcruz, with new containers? | 10:56 |
| arxcruz | sshnaidm|ruck: hold on... it seems it fails... | 10:57 |
| arxcruz | sshnaidm|ruck: nah, fails again, the test has Zuul version: 3.5.0 | 10:57 |
| arxcruz | sshnaidm|ruck: needs zuul 3.9 | 10:58 |
| sshnaidm|ruck | hmm.. | 10:58 |
| sshnaidm|ruck | it means zuul didn't update their containers | 10:59 |
| arxcruz | marios: are you the guest Jacobs in https://hackmd.io/7iRphp5SR3G_GszH2bjswA?view ? | 11:10 |
| *** dsneddon has quit IRC | 11:18 | |
| *** sshnaidm|ruck is now known as sshnaidm|bbl | 11:20 | |
| *** ykarel is now known as ykarel|afk | 11:20 | |
| *** udesale has quit IRC | 11:31 | |
| marios | arxcruz: i am not | 11:35 |
| *** jpena is now known as jpena|lunch | 11:40 | |
| *** dsneddon has joined #oooq | 11:46 | |
| *** dsneddon has quit IRC | 11:51 | |
| zbr | sshnaidm|bbl: or marios: can you either abandon or workflow https://review.opendev.org/#/c/578687/2 ? | 11:53 |
| *** brault has quit IRC | 11:54 | |
| *** brault has joined #oooq | 11:55 | |
| *** brault has quit IRC | 11:55 | |
| *** brault has joined #oooq | 11:56 | |
| weshay | arxcruz sshnaidm|bbl nice.. you guys updating the containers in the repro? | 12:04 |
| *** rfolco has joined #oooq | 12:06 | |
| weshay | panda I thought the board http://rhos-release.virt.bos.redhat.com:3030/rhosp would have flipped to 1 day ( tripleo) 2 days ( rdo ) by mtg time.. | 12:15 |
| arxcruz | weshay: well... trying | 12:15 |
| weshay | that's why the numbers are off | 12:15 |
| *** derekh has quit IRC | 12:16 | |
| weshay | arxcruz ya.. cool rlandy and I were looking at that last week | 12:16 |
| panda | weshay: you want me to revert ? | 12:16 |
| weshay | panda doesn't matter | 12:16 |
| weshay | she'll skp us | 12:16 |
| weshay | skip | 12:16 |
| panda | ok | 12:16 |
| weshay | because we're green | 12:16 |
| arxcruz | weshay: problem now with the reproducer, is that ovb job have ansible 2.8, that is supported only on 3.9 and forward | 12:16 |
| panda | all right, I'm out. | 12:16 |
| weshay | panda thanks!! | 12:17 |
| *** rlandy has joined #oooq | 12:23 | |
| *** dsneddon has joined #oooq | 12:24 | |
| *** dsneddon has quit IRC | 12:28 | |
| *** jpena|lunch is now known as jpena | 12:32 | |
| rfolco | marios, o/ | 12:34 |
| rfolco | marios, please check the drawings at https://tree.taiga.io/project/tripleo-ci-board/task/1245, let me know if any questions :) | 12:36 |
| marios | rfolco: ack | 12:36 |
| *** ykarel|afk is now known as ykarel | 12:38 | |
| rlandy | marios: panda: please see https://tree.taiga.io/project/tripleo-ci-board/us/1211?milestone=238595 | 12:45 |
| rlandy | add if anything is missing | 12:45 |
| rlandy | starting on the test script | 12:45 |
| marios | rlandy: ack thanks | 12:46 |
| chandankumar | rfolco: Hello | 12:49 |
| rfolco | chandankumar, hello from the other side | 12:49 |
| rfolco | I must have called thousand times | 12:50 |
| chandankumar | rfolco: can we use the same etherpad https://etherpad.openstack.org/p/tripleo-ci-squad-meeting for intern projects planning meeting? | 12:50 |
| rfolco | to tell you... I'm Sorrrin | 12:50 |
| rfolco | ok I stop | 12:50 |
| rfolco | chandankumar, indeed, I'm ok with that | 12:50 |
| rlandy | also need to syn con what is left for the containers push work | 12:52 |
| chandankumar | rfolco: rlandy arxcruz https://etherpad.openstack.org/p/tripleo-ci-squad-meeting I have added the agenda at line 55 | 12:53 |
| rfolco | rlandy, yes, I'm confused on that part. I had a patch for pushing mock containers but I suspect this is being worked somwehere else... as dynamic ones, not static. | 12:53 |
| rlandy | rfolco: there is a commented promotion_run work | 12:54 |
| rlandy | and the container work in https://review.rdoproject.org/r/#/c/21895/ | 12:54 |
| rfolco | rlandy, this is not needed anymore? https://review.rdoproject.org/r/21934 | 12:55 |
| rfolco | rlandy, I suspect the mock containers are dynamic now, implemented somewhere else | 12:55 |
| rfolco | rlandy, marios panda we need to sync and fill the gaps, gather the lego pieces | 12:56 |
| rlandy | rfolco: let's discuss at sync meeting | 12:56 |
| rfolco | rlandy, ack, pls re-think about your comment at https://review.rdoproject.org/r/#/c/21872/ | 12:57 |
| chandankumar | rlandy: rfolco marios sshnaidm|bbl panda weshay zbr arxcruz surpatil kopecmartin interns planning meeting https://bluejeans.com/1571313919 in another 3 mins | 12:58 |
| rfolco | rlandy, and perhaps a quick look at https://review.rdoproject.org/r/#/c/22182/ | 12:59 |
| marios | rfolco: rlandy yeah we have a call in an hour | 13:00 |
| *** dsneddon has joined #oooq | 13:00 | |
| rfolco | marios, yes, I am just annoying rlandy :) | 13:00 |
| chandankumar | rfolco: rlandy marios sshnaidm|bbl we are waiting for you if you can | 13:01 |
| marios | chandankumar: opening blue | 13:01 |
| *** derekh has joined #oooq | 13:01 | |
| *** ratailor_ has quit IRC | 13:02 | |
| *** dsneddon has quit IRC | 13:05 | |
| panda | chandankumar: water gun! water gun! | 13:06 |
| chandankumar | panda: forgot to buy that | 13:07 |
| *** Goneri has joined #oooq | 13:08 | |
| *** jaosorior has quit IRC | 13:26 | |
| *** dsneddon has joined #oooq | 13:30 | |
| *** skramaja has quit IRC | 13:32 | |
| *** akahat has quit IRC | 13:33 | |
| bogdando | ykarel: https://review.rdoproject.org/r/#/c/14520/2/nodepool/nodepool.yaml should it be the number of subnodes total here or overcloud (Secondary)? | 13:36 |
| bogdando | not sure how to adjust labels for a 1crlr multinode | 13:36 |
| bogdando | whereas I have 2 subnodes, as primary for UC and secondary for OC | 13:37 |
| bogdando | I'll test it with subnodes_total -1 | 13:37 |
| *** aakarsh|2 has quit IRC | 13:39 | |
| weshay | panda are you ruck today? | 13:39 |
| panda | weshay: we didn't discuss, but usually this is the switch day yes. | 13:40 |
| *** panda is now known as panda|ruck | 13:40 | |
| weshay | panda|ruck thanks.. just fyi.. I have a conflict w/ the cix mtg today.. please attend it | 13:40 |
| ykarel | bogdando, it should be extra nodes | 13:41 |
| ykarel | not including undercloud | 13:41 |
| bogdando | ok thanks | 13:41 |
| ykarel | bogdando, btw so u faced new issue now? | 13:41 |
| panda|ruck | weshay: I was starting to prepare for it | 13:41 |
| bogdando | still testing that labels change ykarel | 13:42 |
| ykarel | bogdando, ack | 13:42 |
| bogdando | having a local copy of openstack infra on my laptop is so cool | 13:42 |
| bogdando | I like the new reproducer :) | 13:42 |
| bogdando | kudos folks, rlandy | 13:43 |
| bogdando | did you read https://model-engineers.com/en/company/references/success-stories/volvo-cars/ perchance? | 13:43 |
| bogdando | and that was only for zuul v2 | 13:43 |
| *** sshnaidm|bbl is now known as sshnaidm|ruck | 13:44 | |
| rlandy | bogdando: happy it works for you. | 13:44 |
| weshay | I'll take bogdando endorsement .. that is awesome | 13:44 |
| bogdando | I can imagine how happy they would be of adopting v3 | 13:44 |
| weshay | \0/ | 13:44 |
| panda|ruck | sshnaidm|ruck: do you want to switch ? | 13:44 |
| bogdando | rlandy: I saw you had a multinode private_ipv4 problem in the past | 13:44 |
| sshnaidm|ruck | panda|ruck, yeah, after todays cix call | 13:44 |
| bogdando | So did I | 13:45 |
| panda|ruck | sshnaidm|ruck: ok | 13:45 |
| *** panda|ruck is now known as panda|rover | 13:45 | |
| bogdando | https://review.opendev.org/#/c/681182/ unlocked it | 13:45 |
| rlandy | bogdando: yes - when we started using v3 on rdocloud | 13:45 |
| rlandy | and then on downstream zuul | 13:46 |
| bogdando | for now, I'm onlt truing upstream multinode | 13:46 |
| bogdando | rdo / ovb is on todo maybe | 13:46 |
| rlandy | right - so we run across three platforms | 13:46 |
| *** redrobot has joined #oooq | 13:50 | |
| rlandy | bogdando: that volvo article takes me back to my previous life MATLAb/dSpace/control :) | 13:57 |
| bogdando | :) | 13:57 |
| sshnaidm|ruck | zbr, how do you see zuul version of CI? | 14:01 |
| marios | rfolco: rlandy sync? | 14:02 |
| rfolco | yep | 14:02 |
| rlandy | marios: ack | 14:02 |
| chandankumar | panda|rover: marios: rfolco rlandy weshay arxcruz zbr thanks for joining the meeting, it was productive discussion :-) | 14:02 |
| rfolco | chandankumar, thank you | 14:03 |
| zbr | sshnaidm|ruck: ask on #zuul or #openstack-infra --- i asked myself the same question. apparently the web page does not include that in formation in the footer, not the cool looking https://zuul.opendev.org/openapi | 14:03 |
| zbr | these is also a cool page https://zuul.opendev.org/manifest.json -- also useless | 14:04 |
| zbr | maybe that is on puprose, like on nginx/apache where they hide server version for security reasons ;) | 14:05 |
| *** surpatil has quit IRC | 14:06 | |
| *** Vorrtex has joined #oooq | 14:10 | |
| *** aakarsh|2 has joined #oooq | 14:16 | |
| zbr | i think I know a way to get centos-8 before others and I bet it will work. hint, we are still able to run jobs on their jenkins server. | 14:21 |
| zbr | weshay: should I create a job to steal that image for which we get 403? | 14:21 |
| weshay | zbr so what are you proposing.. create a job on ci.centos to check tq and ooo on centos8? | 14:23 |
| zbr | weshay: nope, one that downloads the image and uploads it to rdo :D | 14:23 |
| weshay | zbr and use it from nodepool? | 14:24 |
| zbr | bug probably it would be useful without repos to install packages from | 14:24 |
| zbr | s/useful/useless | 14:24 |
| zbr | weshay: but one thing is clear: c8 is delayed due the work being done on c77, which we already know is useless to us. | 14:25 |
| zbr | but the comment from Carl on https://bugs.centos.org/view.php?id=16389 is really interesting, adding selinux is more about removing a skip. | 14:26 |
| *** dsneddon has quit IRC | 14:33 | |
| *** pierreprinetti has joined #oooq | 14:33 | |
| zbr | who can help me with a 2nd pair of eyes on a change? | 14:36 |
| zbr | i have a job where zuul silently skips running tox, runs pre-run and after this it runs post-run,... but there is no run! | 14:37 |
| zbr | http://logs.rdoproject.org/86/21486/26/check/rdo-tox-molecule-rhel8-delegated/c46b27d/job-output.txt.gz | 14:37 |
| *** ykarel is now known as ykarel|away | 14:37 | |
| zbr | everything seems fine, no error, but neither tox is executed. | 14:38 |
| zbr | i am not sure how is possible to see "PRE-RUN END RESULT_NORMAL" and next line being "POST-RUN START" | 14:39 |
| sshnaidm|ruck | panda|rover, let's switch | 14:40 |
| *** sshnaidm|ruck is now known as sshnaidm|rover | 14:40 | |
| sshnaidm|rover | arxcruz, can you try with new :test containers? They contain a new zuul now | 14:41 |
| sshnaidm|rover | arxcruz, but remove them locally, seems like docker cache behaves weird | 14:42 |
| *** ykarel|away has quit IRC | 14:46 | |
| *** panda|rover is now known as panda|ruck | 14:52 | |
| panda|ruck | rlandy: marios rfolco https://github.com/openstack/ansible-role-container-registry/blob/master/molecule/login/prepare.yml | 14:52 |
| sshnaidm|rover | weshay, should we make it promotion blocker? https://bugs.launchpad.net/tripleo/+bug/1843422 | 14:53 |
| openstack | Launchpad bug 1843422 in tripleo "OVB featureset039 all-TLS job fails in master" [Critical,Triaged] | 14:53 |
| weshay | sshnaidm|rover ya.. we'll at least get some attention on it | 14:53 |
| *** jbadiapa has quit IRC | 14:54 | |
| bogdando | please merge https://review.opendev.org/#/c/679975/ and review's needed https://review.opendev.org/#/c/681171/ | 14:59 |
| weshay | panda|ruck available? | 15:02 |
| *** ykarel has joined #oooq | 15:03 | |
| *** dsneddon has joined #oooq | 15:04 | |
| sshnaidm|rover | bogdando, on which OS do you run reproducer? | 15:04 |
| panda|ruck | weshay: on a meeting | 15:06 |
| weshay | k | 15:06 |
| *** dsneddon has quit IRC | 15:08 | |
| *** chandankumar has quit IRC | 15:11 | |
| *** chandankumar has joined #oooq | 15:12 | |
| weshay | sshnaidm|rover /me creating a ticket for the prod-chain infra team re: rcm and rhel 8 | 15:17 |
| weshay | sshnaidm|rover was there a lp or any other info I can reference? | 15:18 |
| weshay | sshnaidm|rover opened a ticket | 15:22 |
| sshnaidm|rover | weshay, rcm? what is it about? | 15:22 |
| weshay | sshnaidm|rover rhel 8 images | 15:23 |
| sshnaidm|rover | weshay, out of space you mean? | 15:24 |
| weshay | yes sir | 15:24 |
| weshay | https://tree.taiga.io/project/morucci-software-factory/issue/2961 | 15:24 |
| sshnaidm|rover | weshay, no, talked with zoli and dmsimard directly | 15:24 |
| *** dsneddon has joined #oooq | 15:36 | |
| *** dsneddon has quit IRC | 15:42 | |
| *** jaosorior has joined #oooq | 15:44 | |
| *** d0ugal has quit IRC | 15:44 | |
| *** ykarel is now known as ykarel|away | 15:51 | |
| *** d0ugal has joined #oooq | 15:59 | |
| *** dtantsur is now known as dtantsur|afk | 16:10 | |
| *** jfrancoa has quit IRC | 16:12 | |
| chandankumar | weshay: thanks to kopecmartin we have now core access in refstack https://review.opendev.org/#/admin/groups/281,members . | 16:15 |
| *** dsneddon has joined #oooq | 16:15 | |
| weshay | kopecmartin++ | 16:16 |
| weshay | kopecmartin nice man | 16:16 |
| *** dsneddon has quit IRC | 16:20 | |
| sshnaidm|rover | marios, rlandy zbr replied in https://review.opendev.org/#/c/681302/ | 16:26 |
| sshnaidm|rover | let's do a progress, I need it for investigations of build images/containers jobs | 16:26 |
| sshnaidm|rover | currently it's unknown completely which hash they use | 16:27 |
| marios | sshnaidm|rover: ack but tomorrow /me out | 16:30 |
| panda|ruck | I don't like the meetings that finish too early | 16:30 |
| panda|ruck | we should stay a bit more next time | 16:30 |
| marios | panda|ruck: i know right. so much more left to say | 16:30 |
| panda|ruck | marios: get out | 16:30 |
| marios | panda|ruck: but seriously that work is at _least_ next sprint. there is a lot of unkown/new | 16:30 |
| panda|ruck | marios: GET OUT WHILE YOU STILL CAN! | 16:31 |
| panda|ruck | marios: I think we are going to finish this sprint. | 16:31 |
| marios | panda|ruck: i will say this for testing. i have ended the day with 0 tests. however precisely because i started the day thinking about how to test the manifests, did i go to the trouble of setting up local registry and manually testing the manifests commands (I mean, i've tested them before on docker.io but pulling from rdo registry etc etc) | 16:31 |
| * chandankumar out, see ya | 16:31 | |
| *** chandankumar is now known as raukadah | 16:31 | |
| panda|ruck | raukadah: see you in 30 minutes | 16:31 |
| marios | panda|ruck: anyway i think we can't use local registry for manifest testing. | 16:31 |
| panda|ruck | nobody ever leaves this place | 16:31 |
| panda|ruck | marios: that's bad :( | 16:32 |
| marios | panda|ruck: not sure how we can use molecule then against dockerhub. i mean, each time it runs, ... i think i have to consider mocking | 16:32 |
| raukadah | panda|ruck: no way, /me is going to sleep today | 16:32 |
| *** marios is now known as marios|out | 16:33 | |
| *** marios|out has quit IRC | 16:37 | |
| *** jpena is now known as jpena|off | 16:44 | |
| *** jaosorior has quit IRC | 16:46 | |
| *** dsneddon has joined #oooq | 16:51 | |
| *** ykarel|away has quit IRC | 16:52 | |
| *** holser has quit IRC | 16:53 | |
| *** dsneddon has quit IRC | 16:56 | |
| *** tesseract has quit IRC | 16:57 | |
| *** derekh has quit IRC | 17:00 | |
| *** d0ugal has quit IRC | 17:04 | |
| *** ykarel|away has joined #oooq | 17:08 | |
| *** d0ugal has joined #oooq | 17:09 | |
| panda|ruck | I don't understand | 17:17 |
| sshnaidm|rover | arxcruz, I've updated containers and have ovb job running fine now, you can try | 17:17 |
| *** dsneddon has joined #oooq | 17:25 | |
| sshnaidm|rover | zbr, I think we're good to go with https://review.opendev.org/#/c/679409/ | 17:26 |
| sshnaidm|rover | zbr, reproducer is fixed, promotion worked, I'm not a ruck.. we can break things | 17:26 |
| *** sshnaidm|rover is now known as sshnaidm|off | 17:28 | |
| zbr | yeah, i would workflow it too. | 17:30 |
| panda|ruck | we should also workflow the login | 17:32 |
| *** brault has quit IRC | 17:33 | |
| *** brault has joined #oooq | 17:35 | |
| *** brault has quit IRC | 17:35 | |
| *** brault has joined #oooq | 17:35 | |
| *** kopecmartin is now known as kopecmartin|off | 17:38 | |
| panda|ruck | well I didn't mean now ... | 17:41 |
| panda|ruck | now that I have to go off | 17:41 |
| rlandy | weshay: who would we ask for the new namespace in rdo registry? jpena|off? | 17:47 |
| weshay | rlandy usually david simard, you could add a ticket to their tiaga.. https://tree.taiga.io/project/morucci-software-factory/issues | 17:48 |
| weshay | new issue | 17:48 |
| rlandy | k | 17:49 |
| *** ykarel|away has quit IRC | 18:00 | |
| *** ykarel|away has joined #oooq | 18:00 | |
| rlandy | panda|ruck: rfolco: weshay: https://tree.taiga.io/project/morucci-software-factory/issue/2962 | 18:03 |
| rfolco | rlandy, thx | 18:04 |
| rlandy | rfolco: I am going to fix https://review.rdoproject.org/r/#/c/21872/ so we can merge | 18:06 |
| panda|ruck | zbr: the login fails, revert | 18:10 |
| panda|ruck | revert | 18:10 |
| panda|ruck | revert | 18:10 |
| panda|ruck | revert | 18:10 |
| panda|ruck | dammit dammit dammit | 18:10 |
| rlandy | panda|ruck: which patch? | 18:10 |
| panda|ruck | https://review.rdoproject.org/r/21547 | 18:11 |
| panda|ruck | rlandy: ^ | 18:11 |
| rlandy | panda|ruck: k - issuing revert job | 18:11 |
| panda|ruck | rlandy: thanks | 18:12 |
| rlandy | panda|ruck: https://review.rdoproject.org/r/#/c/22215/ | 18:12 |
| rlandy | panda|ruck: rfolco: https://review.rdoproject.org/r/#/c/21872/ updated - pls review | 18:15 |
| panda|ruck | rlandy: approved. | 18:15 |
| rlandy | thanks | 18:15 |
| rlandy | putting up review for updated namespace - DNM until we get https://tree.taiga.io/project/morucci-software-factory/issues | 18:16 |
| *** panda|ruck is now known as panda|ruck|off | 18:21 | |
| rfolco | rlandy, whats the user name for the image server ? | 18:22 |
| rlandy | rfolco: which one? | 18:22 |
| rfolco | rlandy, I used test here https://review.rdoproject.org/r/#/c/21872/9/ci-scripts/dlrnapi_promoter/dlrn-promoter.sh | 18:22 |
| rfolco | export IMAGE_SERVER_USER_HOST="test@localhost" | 18:23 |
| rlandy | Gabriele Cerami | 18:24 |
| rlandy | Sep 9 1:30 PM | 18:24 |
| rlandy | in the current fixtures, the user is "foo", the password is "bar" | 18:24 |
| rlandy | ^^ that is for https://review.rdoproject.org/r/#/c/21872/1..9/ci-scripts/dlrnapi_promoter/config/CentOS-7/staging.ini | 18:24 |
| rfolco | rlandy, oh, sorry, | 18:24 |
| rfolco | ok | 18:24 |
| rlandy | if you are talking about where overcloud images are stored ... | 18:25 |
| rlandy | no idea | 18:25 |
| rlandy | I know the regular one | 18:25 |
| rfolco | rlandy, I hope the export works when the dlrnapi script runs | 18:25 |
| rfolco | https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/dlrnapi_promoter.py#L171-L176 | 18:25 |
| rlandy | if not, we will find out | 18:25 |
| rlandy | rfolco: run it locally with -s | 18:25 |
| rlandy | with molecule | 18:25 |
| rlandy | https://review.rdoproject.org/r/#/c/21872/9/ci-scripts/dlrnapi_promoter/dlrn-promoter.sh | 18:26 |
| rlandy | rfolco: ^^ as long as the regular promoter keeps running we are ok | 18:26 |
| *** jbadiapa has joined #oooq | 18:27 | |
| * raukadah is back to square 0 | 18:27 | |
| *** ykarel|away has quit IRC | 18:28 | |
| rfolco | rlandy, pls elaborate your last sentence | 18:31 |
| rfolco | as long as regular promoter keeps running... | 18:31 |
| rlandy | as long as anything we merge does not break the current promoter, we can test it afterwards | 18:32 |
| rlandy | IMAGE_SERVER_USER_HOST="test@localhost" | 18:32 |
| rlandy | let's see what the set up is there | 18:32 |
| rfolco | rlandy, can I fix test@ before we merge this one ? | 18:33 |
| rfolco | rlandy, oh you suggested to add a molecule test first | 18:33 |
| rfolco | a sanity check | 18:33 |
| rlandy | https://review.rdoproject.org/r/#/c/21895/43/ci-scripts/dlrnapi_promoter/tests/staging-setup/staging_environment.py | 18:34 |
| rlandy | doesn;t look like there is a password | 18:34 |
| rfolco | rlandy, will replace test w/ foo and add a molecule test to run this script with -s | 18:36 |
| rlandy | ack | 18:36 |
| rfolco | rlandy, the other part is getting the list of containers from a file... which is still not quite clear to me where this file is. | 18:37 |
| rlandy | rfolco: after which, can we merge https://review.rdoproject.org/r/#/c/22182/? | 18:37 |
| rlandy | rfolco: I am working on that | 18:37 |
| rlandy | the list of containers is here ... | 18:38 |
| rfolco | rlandy, ok so you are taking over task 1251 ? https://tree.taiga.io/project/tripleo-ci-board/task/1251 | 18:38 |
| rlandy | https://review.rdoproject.org/r/#/c/21895/43/ci-scripts/dlrnapi_promoter/tests/staging-setup/samples/docker_images.txt | 18:38 |
| rlandy | just need to adjust namespace | 18:38 |
| rlandy | and enable push | 18:38 |
| rfolco | rlandy, dynamic names too? | 18:39 |
| rlandy | rfolco: I don't know what tasks related to what in this project | 18:39 |
| rfolco | instead of 12345, should be timestamp ? | 18:39 |
| rlandy | yes plus namespace | 18:40 |
| rlandy | after that I want to work on the python testing of the promoter | 18:41 |
| rfolco | rlandy, lets keep syncing closely | 18:41 |
| rlandy | rfolco: ugh read panda|ruck|off's mail | 18:43 |
| * rlandy shoots everyone | 18:44 | |
| rlandy | forwarding to weshay | 18:44 |
| * rlandy tries that locally | 18:45 | |
| *** d0ugal has quit IRC | 18:47 | |
| *** d0ugal has joined #oooq | 18:52 | |
| *** brault has quit IRC | 19:02 | |
| rlandy | create --insecure | 19:04 |
| rlandy | ^^ maybe marios was missing the insecure piece | 19:04 |
| weshay | ok.. you guys would rather use a local registry? | 19:08 |
| weshay | it's hard to tell exactly what kind of registry that is.. because it's just a container | 19:09 |
| rlandy | weshay: so originally I wanted to use a local registry while we were testing the push code | 19:14 |
| rlandy | so we didn't pollute a public registry with a whole lot of junk getting our tests right | 19:14 |
| rlandy | if we get our own namespace maybe it doesn't matter | 19:15 |
| rlandy | as you said registries are all diff | 19:15 |
| rlandy | and dealing with a secure registry is different as well | 19:15 |
| rlandy | really now I have no idea what our direction is | 19:16 |
| *** panda|ruck|off has quit IRC | 19:55 | |
| *** panda has joined #oooq | 19:57 | |
| weshay | I guess it's worth trying.. but I have some concerns but will let it roll for now | 20:28 |
| rlandy | panda: you around | 20:32 |
| rlandy | question about adding timestamps to the container names | 20:32 |
| rlandy | looking at https://review.rdoproject.org/r/#/c/21895/43/ci-scripts/dlrnapi_promoter/tests/staging-setup/staging_environment.py | 20:32 |
| rlandy | rfolco: ^^ do you know which files need to be rendered with the timestamp? | 21:03 |
| rlandy | ci-scripts/dlrnapi_promoter/tests/staging-setup/samples/meta.yaml? | 21:03 |
| rlandy | ci-scripts/dlrnapi_promoter/tests/staging-setup/samples/docker_images.txt | 21:04 |
| rlandy | ci-scripts/dlrnapi_promoter/tests/staging-setup/samples/overcloud_containers.txt? | 21:04 |
| *** Vorrtex has quit IRC | 21:12 | |
| rfolco | rlandy, don't know, I would need to take a closer look as well | 21:19 |
| rlandy | rfolco: np - I am putting up a patch with my best guess | 21:20 |
| rlandy | will make discussion/review easier | 21:20 |
| rfolco | rlandy, great | 21:20 |
| *** saneax has quit IRC | 21:57 | |
| *** panda has quit IRC | 22:00 | |
| *** panda has joined #oooq | 22:03 | |
| *** matbu has quit IRC | 22:06 | |
| *** rlandy is now known as rlandy|bbl | 22:24 | |
| *** aakarsh|2 has quit IRC | 22:42 | |
| *** saneax has joined #oooq | 23:12 | |
| *** sanjayu_ has joined #oooq | 23:18 | |
| *** saneax has quit IRC | 23:20 | |
| *** d0ugal has quit IRC | 23:23 | |
| *** d0ugal has joined #oooq | 23:25 | |
| *** sanjayu_ has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!