| *** sshnaidm has quit IRC | 00:45 | |
| *** sshnaidm has joined #oooq | 01:00 | |
| *** Goneri has quit IRC | 01:00 | |
| *** rlandy|bbl is now known as rlandy | 01:36 | |
| *** caowei99 has joined #oooq | 02:13 | |
| *** rlandy has quit IRC | 02:22 | |
| *** EmilienM|PTO is now known as EmilienM | 02:42 | |
| *** apetrich has quit IRC | 03:09 | |
| *** caowei99_ has joined #oooq | 03:23 | |
| *** caowei99 has quit IRC | 03:28 | |
| *** ysandeep has joined #oooq | 03:40 | |
| *** dsneddon has quit IRC | 03:40 | |
| *** skramaja has joined #oooq | 03:50 | |
| *** sshnaidm_ has joined #oooq | 03:53 | |
| *** sshnaidm has quit IRC | 03:57 | |
| *** epoojad1 has joined #oooq | 04:08 | |
| *** dsneddon has joined #oooq | 04:08 | |
| *** saneax has quit IRC | 04:09 | |
| *** dsneddon has quit IRC | 04:13 | |
| *** udesale has joined #oooq | 04:14 | |
| *** ykarel has joined #oooq | 04:19 | |
| *** sshnaidm__ has joined #oooq | 04:25 | |
| *** sshnaidm_ has quit IRC | 04:26 | |
| *** dtrainor_ is now known as dtrainor | 04:29 | |
| *** caowei99 has joined #oooq | 04:30 | |
| *** caowei99_ has quit IRC | 04:34 | |
| *** caowei99 has quit IRC | 04:39 | |
| *** dsneddon has joined #oooq | 05:02 | |
| *** saneax has joined #oooq | 05:05 | |
| *** skramaja_ has joined #oooq | 05:06 | |
| *** skramaja has quit IRC | 05:07 | |
| *** ratailor has joined #oooq | 05:11 | |
| *** caowei99 has joined #oooq | 05:32 | |
| *** epoojad1 has quit IRC | 05:39 | |
| *** raukadah is now known as chkumar|rover | 05:41 | |
| *** soniya29 has joined #oooq | 05:43 | |
| *** caowei99 has quit IRC | 05:44 | |
| *** bhagyashris has joined #oooq | 05:45 | |
| *** akahat has joined #oooq | 06:09 | |
| *** caowei99 has joined #oooq | 06:15 | |
| *** marios|ruck has joined #oooq | 06:16 | |
| *** epoojad1 has joined #oooq | 06:19 | |
| *** caowei99_ has joined #oooq | 06:26 | |
| *** caowei99 has quit IRC | 06:29 | |
| *** ratailor has quit IRC | 06:38 | |
| *** udesale has quit IRC | 06:40 | |
| *** epoojad1 has quit IRC | 06:40 | |
| *** udesale has joined #oooq | 06:41 | |
| *** ratailor has joined #oooq | 06:42 | |
| *** jtomasek has joined #oooq | 06:44 | |
| *** jfrancoa has joined #oooq | 06:51 | |
| *** jbadiapa has joined #oooq | 07:14 | |
| *** apetrich has joined #oooq | 07:29 | |
| *** apetrich has quit IRC | 07:34 | |
| *** udesale has quit IRC | 07:34 | |
| *** udesale has joined #oooq | 07:35 | |
| *** caowei99 has joined #oooq | 08:12 | |
| *** tesseract has joined #oooq | 08:16 | |
| *** epoojad1 has joined #oooq | 08:16 | |
| *** caowei99_ has quit IRC | 08:16 | |
| *** amoralej|off is now known as amoralej | 08:16 | |
| *** ratailor has quit IRC | 08:17 | |
| *** ratailor_ has joined #oooq | 08:18 | |
| *** tosky has joined #oooq | 08:25 | |
| *** jtomasek has quit IRC | 08:34 | |
| *** jtomasek has joined #oooq | 08:34 | |
| chkumar|rover | marios|ruck: sorry, on os_tempest undercloud, I am trying to make blacklist working so that it does break, I will give a green light for review once it starts working | 08:35 |
|---|---|---|
| *** saneax has quit IRC | 08:36 | |
| *** saneax has joined #oooq | 08:37 | |
| *** bogdando has joined #oooq | 08:40 | |
| *** jpena|off is now known as jpena | 08:44 | |
| *** dsneddon has quit IRC | 08:50 | |
| marios|ruck | chkumar|rover: ack | 08:54 |
| marios|ruck | chkumar|rover: maybe -1 it or set WIP if it isn't ready for reviews | 08:54 |
| *** soniya29 has quit IRC | 09:02 | |
| *** jtomasek has quit IRC | 09:04 | |
| *** apetrich has joined #oooq | 09:04 | |
| *** epoojad1 has quit IRC | 09:04 | |
| *** jtomasek has joined #oooq | 09:09 | |
| *** dsneddon has joined #oooq | 09:25 | |
| Tengu | hello there! | 09:34 |
| Tengu | I'm wondering - we don't have any post-check that ensures there is no selinux issue, right? | 09:35 |
| Tengu | (in the CI) | 09:35 |
| Tengu | i.e. I think it would be good to get something that ensures the /var/log/extra/denials.txt.gz is empty... any thoughts? | 09:36 |
| Tengu | especially since we do have proper selinux support with podman - we can't afford to ignore those issues imho | 09:36 |
| marios|ruck | Tengu: not aware of such a change but i don't know if checking denials.txt is empty. for example just checked that its a green job http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/e3833b0/logs/undercloud/var/log/extra/denials.txt.gz | 09:38 |
| marios|ruck | Tengu: such a change/such a thing i mean which checks selinux | 09:38 |
| Tengu | marios|ruck: my point :) | 09:38 |
| Tengu | I think such a job shouldn't be green, seeing the content of the denials.txt, right? :) | 09:38 |
| marios|ruck | Tengu: hmm also i think we are running permissive right now ? http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/e3833b0/logs/undercloud/var/log/extra/selinux.txt.gz | 09:39 |
| Tengu | nothing will raise any issue before patch hits a user/customer because most of the jobs are in permissive. | 09:39 |
| Tengu | marios|ruck: permissive=1 | 09:39 |
| marios|ruck | Tengu: right so that's the first thing to address i mean consider using enforcing. | 09:40 |
| Tengu | marios|ruck: I think this isn't possible | 09:40 |
| marios|ruck | Tengu: i am not aware of the context or history on that permissive though. maybe its one for the community call today | 09:40 |
| Tengu | marios|ruck: you want to involve weshay|ruck on that - I remember he said that it wasn't possible due to the infra or something | 09:41 |
| Tengu | more over, it's not sure that a denial will make the job fail | 09:41 |
| marios|ruck | Tengu: right see if you can join community call today maybe others have more info on th team. otherwise we can discuss you idea about adding a validation, albeit with permissive | 09:41 |
| Tengu | depending on the thing that is blocked, it won't trigger an actual issue within the job. | 09:41 |
| marios|ruck | Tengu: well it will if it causes an operation to fail but maybe it won't be clear 'it was selinux issue' | 09:42 |
| *** ykarel is now known as ykarel|lunch | 09:42 | |
| Tengu | my take is: checking the file is probably safer | 09:42 |
| Tengu | marios|ruck: nope, as said, it might be triggered by a service (or part of a service) that isn't actually used. | 09:42 |
| Tengu | https://72a8888f4d09f1e72a1e-7c45ade9796b4a439c7ceea8af4ef1aa.ssl.cf2.rackcdn.com/695903/2/check/tripleo-ci-centos-7-standalone/107cbd4/logs/undercloud/var/log/extra/denials.txt.gz <- for instance - job is green | 09:42 |
| Tengu | and here, it's enforcing. | 09:43 |
| Tengu | :) | 09:43 |
| Tengu | "oops" ;) | 09:43 |
| Tengu | and if we do check the denials.txt, we can output a nice thing such as "SELinux is blocking things, please check" | 09:43 |
| Tengu | pointing to the right issue directly. | 09:43 |
| Tengu | .... even providing a .te as an example on how to solve the issue - but it might lead to mistakes and security issues. | 09:44 |
| marios|ruck | Tengu: well i mean its 'only' ovs-vswitchd not like its anything critical in there ! | 09:44 |
| marios|ruck | Tengu: yah just don't know how useful it will be when that file is quite full like in the earlier example | 09:45 |
| marios|ruck | Tengu: but hold on does it mean that job is broken ? or the error was overcome somehow/we did something in a subsequent step of the code | 09:46 |
| chkumar|rover | marios|ruck: I have updated this bug https://bugs.launchpad.net/tripleo/+bug/1853978 | 09:46 |
| openstack | Launchpad bug 1853978 in tripleo "periodic rhel8 ovb overcloud deployment failed with Could not find class ::tripleo::profile::base::neutron::ovn_metadata_agent_wrappers" [Critical,Triaged] - Assigned to chandan kumar (chkumar246) | 09:46 |
| Tengu | marios|ruck: https://bugzilla.redhat.com/show_bug.cgi?id=1776685 for more info on that precise issue. | 09:46 |
| openstack | bugzilla.redhat.com bug 1776685 in openvswitch "SELinux denial for net_broadcast" [Urgent,New] - Assigned to ovs-team | 09:46 |
| Tengu | marios|ruck: brb | 09:47 |
| marios|ruck | chkumar|rover: ack thanks for the update | 09:47 |
| zbr_ | marios|ruck: chkumar|rover see https://review.opendev.org/#/c/696041/1 | 09:48 |
| *** soniya29 has joined #oooq | 09:48 | |
| zbr_ | we may need to add this to few more places, i already updated https://tree.taiga.io/project/tripleo-ci-board/task/1374 | 09:49 |
| *** derekh has joined #oooq | 09:49 | |
| chkumar|rover | marios|ruck: https://review.opendev.org/#/c/696019/ will fix the issue | 09:50 |
| chkumar|rover | checking in rdo -jobs | 09:50 |
| marios|ruck | chkumar|rover: thanks adding comment there then | 09:51 |
| marios|ruck | zbr_: ack but too early to vote there need to see zuul report | 09:54 |
| marios|ruck | zbr_: ping again later? | 09:54 |
| zbr_ | marios|ruck: there is a zuul, report, linters passed, you just did not check it. | 09:54 |
| marios|ruck | zbr_: https://review.opendev.org/#/c/696041/1 ? | 09:54 |
| marios|ruck | zbr_: i don't see it. just refreshed again | 09:55 |
| zbr_ | yeah, https://ead82af4f93b84f6e23c-abb4792a05bf9ff32511cb91e1271a04.ssl.cf1.rackcdn.com/696041/1/check/openstack-tox-linters/bde790e/ | 09:55 |
| marios|ruck | zbr_: ack, so it passed but didn't report on the review yet. | 09:55 |
| chkumar|rover | marios|ruck: I am adding RHEL-8 master job against puppet-tripleo | 09:55 |
| chkumar|rover | fs01 | 09:56 |
| zbr_ | marios|ruck: i guess you are not using https://opendev.org/x/coats#openstack_gerrit_zuul_statususerjs :D | 09:56 |
| marios|ruck | zbr_: nice | 09:58 |
| zbr_ | i still dream about the moment when we will see live zuul status in our gerrit, without hacks. | 09:59 |
| *** bhagyashris has quit IRC | 09:59 | |
| *** bhagyashris has joined #oooq | 09:59 | |
| *** dsneddon has quit IRC | 09:59 | |
| zbr_ | latest versions of gerrit added support for this but someone needs to update zuul and upgrade gerrit(s), which is not easy. | 10:00 |
| arxcruz | zbr_: did you get any progress to install golang to make the podman rpm ? | 10:00 |
| arxcruz | jpena: ykarel|lunch any chance you guys know where to find the spec file for podman rpm? it seems they don't have it in their repo like the ceph-ansible | 10:02 |
| Tengu | marios|ruck: when is the mtg today? just before #tripleo right? | 10:02 |
| zbr_ | arxcruz: give me few mins, i wanted to ping you about, on which job/CR it was this needed? | 10:02 |
| Tengu | marios|ruck: if it's right after, I'll have a conflict :/ | 10:02 |
| arxcruz | zbr_: ok, let me know when :) | 10:02 |
| marios|ruck | Tengu: yeah half hour before tripleo weekly irc | 10:03 |
| Tengu | marios|ruck: perfect, I'll try to show up then :) | 10:03 |
| marios|ruck | Tengu: https://hackmd.io/IhMCTNMBSF6xtqiEd9Z0Kw?both meet.google.com/bqx-xwht-wky | 10:03 |
| Tengu | marios|ruck: should I add an entry for today's meeting, or.. ? | 10:05 |
| *** akahat has quit IRC | 10:06 | |
| marios|ruck | Tengu: maybe use the space at top "Items for next community call" then rfolco will add to today's agenda | 10:06 |
| Tengu | marios|ruck: added an entry :) | 10:08 |
| marios|ruck | ack Tengu | 10:12 |
| *** epoojad1 has joined #oooq | 10:12 | |
| jpena | arxcruz: if it's not part of the repo, I guess we should use the fedora spec -> https://src.fedoraproject.org/rpms/podman/blob/master/f/podman.spec | 10:13 |
| arxcruz | jpena: yeah, i'm seeing this one, it's quite big | 10:13 |
| jpena | Looking at it, I see there's some automation going on, it might be good to contact lsm5 for more info | 10:13 |
| arxcruz | jpena: and there's a bot to move the stuff forward | 10:13 |
| arxcruz | and a lot of dependences with hardcoded commit not sure how to handle it | 10:14 |
| arxcruz | jpena: do you have lsm5 contact? i don't know who he is | 10:14 |
| arxcruz | jpena: what do you think? should i create a task that will download this podman.spec and built from it ? | 10:17 |
| jpena | arxcruz: I would check how the podman team are doing those auto updates, we might reuse them as part of the job (which would eventually download podman.spec and modify it) | 10:18 |
| *** dsneddon has joined #oooq | 10:19 | |
| arxcruz | jpena: so this lsm5 knows that right? | 10:19 |
| arxcruz | i'll email him, cc'ing you and zbr_ | 10:20 |
| *** dsneddon has quit IRC | 10:25 | |
| arxcruz | annnnndd he's on pto | 10:25 |
| chkumar|rover | arxcruz: please keep EmilienM in the loop also | 10:27 |
| *** epoojad1 has quit IRC | 10:27 | |
| arxcruz | chkumar|rover: man, decide which nick you will use, i was trying to ping you this morning, type chan <tab> didn't saw then I assume you weren't here | 10:27 |
| arxcruz | then I try hauk <tab> nothing | 10:28 |
| arxcruz | now is chk <tab> | 10:28 |
| *** ykarel|lunch is now known as ykarel | 10:28 | |
| *** jpich has joined #oooq | 10:29 | |
| chkumar|rover | arxcruz: ah, sorry, How can i help? | 10:31 |
| arxcruz | chkumar|rover: you can't you broke my feelings | 10:32 |
| arxcruz | lol | 10:32 |
| chkumar|rover | hehe | 10:32 |
| arxcruz | just kidding, i was wondering if you know how to build the rpm for podman with the spec file | 10:32 |
| chkumar|rover | arxcruz: I built it last time | 10:32 |
| arxcruz | it seems to be more complicated than i was expecting | 10:32 |
| arxcruz | chkumar|rover: how? where? when? | 10:32 |
| chkumar|rover | arxcruz: https://hackmd.io/cThSvUCiRzeHvi8ekq6zWw#Proposal | 10:33 |
| chkumar|rover | arxcruz: https://github.com/containers/libpod/blob/master/contrib/build_rpm.sh | 10:33 |
| chkumar|rover | run it from podman root directory | 10:33 |
| arxcruz | fuck | 10:33 |
| arxcruz | chkumar|rover: thanks | 10:33 |
| arxcruz | chkumar|rover:++ | 10:34 |
| chkumar|rover | it is written on the hackmd pad | 10:34 |
| *** epoojad1 has joined #oooq | 10:34 | |
| * arxcruz needs a beer | 10:34 | |
| chkumar|rover | arxcruz: if golang is not there, now there is a role in zuul-jobs to install that https://review.opendev.org/#/c/691114/ | 10:36 |
| arxcruz | chkumar|rover: is your patch to add that rpm build role already merged? | 10:39 |
| chkumar|rover | arxcruz: nope, it is on hold, till we add CI | 10:41 |
| arxcruz | chkumar|rover: zbr_ jpena can you guys please review https://review.rdoproject.org/r/#/c/23537/ ? | 10:41 |
| arxcruz | nevermind guys ^ | 10:41 |
| arxcruz | chkumar|rover: ok, i'll do it in the old fashion way for podman than we can move forward with this | 10:41 |
| zbr_ | arxcruz: how to review a change that has a depends-on on a change with W -1? see https://review.opendev.org/#/c/694396/ ? | 10:45 |
| zbr_ | it makes no sense to even read anything else about it | 10:46 |
| arxcruz | zbr_: did you check my comments on my review? | 10:46 |
| arxcruz | the testproject jobs passing that's why i +1 | 10:46 |
| arxcruz | ronellye w-1 because she was testing first | 10:47 |
| arxcruz | she didn't update the w-1 doesn't means i can +1 | 10:47 |
| zbr_ | arxcruz: ronelle blocked any progress on it, I cannot remove her vote. | 10:48 |
| arxcruz | zbr_: so, what do I have to do with this? | 10:49 |
| *** apetrich has quit IRC | 10:49 | |
| zbr_ | arxcruz: but my impression is that this depends-on is not really needed. | 10:49 |
| arxcruz | which depends-on ? | 10:49 |
| zbr_ | https://review.rdoproject.org/r/#/c/23537/ | 10:50 |
| arxcruz | 11:41:24 @arxcruz | nevermind guys ^ | 10:50 |
| arxcruz | zbr_: i said nevermind :) | 10:50 |
| zbr_ | but what I do not like is that I do not see tripleo-ceph-integration being triggered as part of this change, it should. touching pre/run does break jobs, should have trigger on it. | 10:51 |
| zbr_ | so we can see the outcome of the change without having to create changes in test-project | 10:51 |
| zbr_ | using test-project is a bit more like "works for me", or "not writing tests". | 10:52 |
| zbr_ | chkumar|rover: ^ I could update it to trigger it and remove depends-on, what do you think? | 10:53 |
| chkumar|rover | zbr_: sorry | 10:53 |
| chkumar|rover | zbr_: currently ceph ansible related jobs will be triggered based on comments on ceph-ansile pr | 10:55 |
| chkumar|rover | zbr_: https://tree.taiga.io/project/tripleo-ci-board/task/1388 | 10:55 |
| zbr_ | chkumar|rover: yes but as part of implementing it you have to also trigger it when you change its CI definition, that is implied. | 10:55 |
| chkumar|rover | zbr_: there is a task to create a seperate pipeline to run from the good known bits | 10:55 |
| zbr_ | otherwise next change would likely break it | 10:56 |
| chkumar|rover | zbr_: may be modifying git hooks to listen merge events and then trigger a job will help? | 10:56 |
| zbr_ | chkumar|rover: nope, i will update it, it should be two extra lines to trigger job. | 10:56 |
| chkumar|rover | zbr_: ok, please add the info on the task | 10:57 |
| *** dsneddon has joined #oooq | 10:58 | |
| *** udesale has quit IRC | 10:58 | |
| chkumar|rover | arxcruz: did you get a chance to sync with panda and zbr regarding podman job result status? | 10:59 |
| arxcruz | chkumar|rover: the actual job doesn't build the rpm, and no, i did not have the chance, i was on pto last week | 10:59 |
| chkumar|rover | arxcruz: no, problem, the job final status is always returning success either it fails or pass, we have fixed podman side to ignore the zuul jobs vote currnetly | 11:00 |
| arxcruz | zbr_: your patch, why do we need ceotns-7 and centos-8? according weshay|ruck we will care only el8 for now | 11:08 |
| zbr_ | arxcruz: i want/need to see what happens, so I can write the right changes from start. I will make it non-voting if i get blocked by something external. | 11:09 |
| zbr_ | arxcruz: i am adding a new package, why to create more work for the future when I can do the right thing now (when I am touching this code) | 11:10 |
| zbr_ | i see centos-8 as part of test-coverage :D | 11:11 |
| *** jaosorior has joined #oooq | 11:13 | |
| arxcruz | ok | 11:14 |
| arxcruz | let's merge it so i can work on the next steps | 11:14 |
| arxcruz | chkumar|rover: please review zbr_ patches https://review.rdoproject.org/r/#/c/23826/ | 11:14 |
| *** sshnaidm__ is now known as sshnaidm | 11:15 | |
| *** caowei99_ has joined #oooq | 11:16 | |
| *** caowei99 has quit IRC | 11:19 | |
| *** caowei99_ has quit IRC | 11:22 | |
| *** apetrich has joined #oooq | 11:27 | |
| *** apetrich has quit IRC | 11:27 | |
| *** apetrich has joined #oooq | 11:29 | |
| *** dsneddon has quit IRC | 11:30 | |
| marios|ruck | matbu: bogdando: o/ folks thanks for working there https://bugs.launchpad.net/tripleo/+bug/1853812 so we need all the patches ? (just added comment #9) | 11:37 |
| openstack | Launchpad bug 1853812 in tripleo "train containerized-undercloud-upgrades failing with "error creating container the container name is already in use "" [Critical,Triaged] - Assigned to Bogdan Dobrelya (bogdando) | 11:37 |
| marios|ruck | chkumar|rover: \o/ master promoted (https://review.rdoproject.org/r/#/c/23822/ didn't report yet but centos fs01 is green there and that was missing | 11:45 |
| bogdando | marios|ruck: https://review.opendev.org/#/c/695929/ replaces revert | 11:46 |
| marios|ruck | bogdando: ok thanks, lets -1 the revert/add comment there (sorry if i missed it) | 11:47 |
| marios|ruck | bogdando: sorry i see your comments there now. was confused as you have -1 on the other one but not on the revert | 11:48 |
| *** epoojad1 is now known as epoojad1|afk | 11:49 | |
| chkumar|rover | awesome | 11:53 |
| *** epoojad1|afk has quit IRC | 11:53 | |
| chkumar|rover | arxcruz: https://github.com/containers/libpod/blob/master/contrib/build_rpm.sh this script also needs to modified | 11:57 |
| chkumar|rover | espcially this part https://github.com/containers/libpod/blob/master/contrib/build_rpm.sh#L12 | 11:57 |
| chkumar|rover | in RHEL-8, we have yum as well as dnf | 11:58 |
| *** surpatil has joined #oooq | 11:59 | |
| *** caowei99_ has joined #oooq | 12:00 | |
| chkumar|rover | arxcruz: one more thing, we also need to check whether all those packages are availble in RHUI or not | 12:01 |
| *** derekh has quit IRC | 12:01 | |
| *** derekh has joined #oooq | 12:01 | |
| zbr_ | FYI yum is a symlink to dnf | 12:02 |
| *** dsneddon has joined #oooq | 12:03 | |
| chkumar|rover | arxcruz: but that logic needs to be fixed | 12:04 |
| *** akahat has joined #oooq | 12:04 | |
| *** rfolco has joined #oooq | 12:05 | |
| *** kopecmartin has joined #oooq | 12:06 | |
| arxcruz | ok | 12:10 |
| arxcruz | let's merge zbr_ patch | 12:10 |
| *** skramaja_ has quit IRC | 12:12 | |
| *** jpich has quit IRC | 12:12 | |
| *** jpich has joined #oooq | 12:13 | |
| chkumar|rover | arxcruz: zbr_ left one comment | 12:14 |
| marios|ruck | sshnaidm: o/ can you see comment #3 https://bugs.launchpad.net/tripleo/+bug/1853028 were you asked by wes to add promotion-blcoker there? | 12:17 |
| openstack | Launchpad bug 1853028 in tripleo "Build overcloud image for rhel8 fails sometimes on in_target.d/post-install.d/51-enable-network-service" [Critical,Triaged] | 12:17 |
| marios|ruck | sshnaidm: do we have recent examples of that | 12:17 |
| *** skramaja_ has joined #oooq | 12:20 | |
| *** dtantsur|afk is now known as dtantsur | 12:20 | |
| *** udesale has joined #oooq | 12:21 | |
| sshnaidm | marios|ruck, https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-rhel-8-buildimage-overcloud-full-master%20&job_name=tripleo-rhel-8-buildimage-overcloud-full | 12:21 |
| sshnaidm | marios|ruck, it happens both in check and periodics | 12:22 |
| marios|ruck | sshnaidm: ah ok so it was seen in check yesterday maybe? | 12:22 |
| sshnaidm | marios|ruck, yes | 12:22 |
| marios|ruck | sshnaidm: didn't look for that | 12:22 |
| marios|ruck | sshnaidm: ack ok thanks | 12:22 |
| sshnaidm | marios|ruck, it happens for a last month in fact | 12:22 |
| *** apetrich has quit IRC | 12:23 | |
| marios|ruck | sshnaidm: ack, i was just suprised to see it as new in prod chain board but i hadn't seen the bug yet | 12:23 |
| sshnaidm | marios|ruck, it was in our etherpad and I think it was on board in some moment too, maybe it's better to put it in "ci degrade" column | 12:27 |
| marios|ruck | sshnaidm: ack not yet am adding info there and jan can move it /on next call | 12:27 |
| sshnaidm | marios|ruck, ack, I'll join the call tomorrow, can provide more info if needed | 12:29 |
| sshnaidm | but this should be assigned and solved some day | 12:30 |
| marios|ruck | sshnaidm: thanks added some info on trello card | 12:30 |
| *** akahat has quit IRC | 12:39 | |
| *** ccamacho has quit IRC | 12:40 | |
| *** akahat has joined #oooq | 12:41 | |
| *** caowei99_ has quit IRC | 12:47 | |
| *** caowei99_ has joined #oooq | 12:48 | |
| rfolco | chkumar|rover, can you please update tasks under https://tree.taiga.io/project/tripleo-ci-board/us/1342 ? Its hard to take it over without context. These tasks have been started and its impossible to know whats going on without some context info. | 12:52 |
| *** Goneri has joined #oooq | 12:53 | |
| *** ysandeep has quit IRC | 12:54 | |
| *** weshay|ruck is now known as weshay | 13:00 | |
| *** ccamacho has joined #oooq | 13:02 | |
| *** ratailor_ has quit IRC | 13:03 | |
| *** amoralej is now known as amoralej|lunch | 13:07 | |
| *** dsneddon has quit IRC | 13:08 | |
| *** soniya29 has quit IRC | 13:10 | |
| *** caowei99 has joined #oooq | 13:11 | |
| *** rlandy has joined #oooq | 13:11 | |
| arxcruz | ill pick up my daughter in the kindergarden, brb | 13:12 |
| *** epoojad1 has joined #oooq | 13:12 | |
| *** apetrich has joined #oooq | 13:14 | |
| *** caowei99_ has quit IRC | 13:14 | |
| *** surpatil has quit IRC | 13:14 | |
| *** caowei99 has quit IRC | 13:16 | |
| zbr_ | @oooq a headup, i am now working to fix https://bugs.launchpad.net/tripleo/+bug/1852078 -- which directly affects us. zuul bug, hopefully we can see it in today | 13:18 |
| openstack | Launchpad bug 1852078 in tripleo "rdo zuul has broken zuul logger for pre-runs" [High,Triaged] | 13:18 |
| rlandy | rfolco: weshay: ha - think the fs_override thing worked http://logs.rdoproject.org/49/23749/7/check/periodic-tripleo-ci-centos-7-standalone-compute-master/cf815d5/logs/undercloud/etc/yum.repos.d/component.repo.txt.gz | 13:24 |
| chkumar|rover | rfolco: sorry was in a clal | 13:25 |
| weshay | rlandy, nice.. rfolco nice idea ++ | 13:26 |
| rfolco | rlandy++ | 13:26 |
| chkumar|rover | rfolco: will do that right now | 13:26 |
| *** jpena is now known as jpena|lunch | 13:26 | |
| rfolco | ci community mtg starts in 3 min.... place: https://meet.google.com/bqx-xwht-wky | 13:27 |
| rfolco | agenda https://hackmd.io/IhMCTNMBSF6xtqiEd9Z0Kw?both | 13:27 |
| rlandy | we need to have a merge party this afternoon | 13:28 |
| rfolco | rlandy, cool thanks for doing that, I struggled to make it general path for any component | 13:29 |
| rfolco | this simplifies a lot | 13:29 |
| rfolco | one single component file thats all we need | 13:29 |
| chkumar|rover | rlandy: https://review.rdoproject.org/r/#/c/23813/ something not working there | 13:29 |
| rfolco | community mtg starting now | 13:30 |
| rfolco | WARNING | 13:30 |
| rfolco | !!! | 13:30 |
| openstack | rfolco: Error: "!!" is not a valid command. | 13:30 |
| matbu | marios|ruck: Hey man, I don't think we need the revert | 13:30 |
| matbu | bogdando: ^ | 13:31 |
| weshay | rlandy, community call | 13:31 |
| matbu | marios|ruck: bogdando but the one in the paunch ansible module looks correct to me... i saw some strange behavior with the config_id | 13:31 |
| rlandy | chkumar|rover: would depend on timing ... see https://review.rdoproject.org/r/#/c/23811/2/zuul.d/tripleo.yaml | 13:31 |
| rlandy | we will have to think about not building vs. building images | 13:32 |
| chkumar|rover | rlandy: ok, then it is good to go | 13:32 |
| marios|ruck | matbu: thanks | 13:32 |
| rlandy | chkumar|rover: not yet | 13:32 |
| matbu | marios|ruck: i will check the validation stuff now | 13:32 |
| * matbu tries to get clean env to fix the next bug :) | 13:33 | |
| *** soniya29 has joined #oooq | 13:33 | |
| EmilienM | what would be the link on cockpit to see the evolution of tripleo-ci-centos-7-undercloud-containers job runtime? | 13:35 |
| *** dsneddon has joined #oooq | 13:36 | |
| marios|ruck | EmilienM: i don't believe we capture runtime info | 13:38 |
| marios|ruck | sshnaidm: ? ^ 15:35 < EmilienM> what would be the link on cockpit to see the evolution of tripleo-ci-centos-7-undercloud-containers job runtime? | 13:38 |
| *** jaosorior has quit IRC | 13:38 | |
| marios|ruck | EmilienM: well we have duration there i guess ;) but i don't think we are capturing that info and doing progression/comparison etc | 13:39 |
| rlandy | what's wrong with using the zuul link: http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-undercloud-containers | 13:39 |
| rlandy | compare duration | 13:40 |
| EmilienM | rlandy: it's showing results from today & yesterday | 13:40 |
| marios|ruck | rlandy: heh yeah but i think he wants a progression | 13:40 |
| EmilienM | yes | 13:40 |
| rlandy | sova used to do that | 13:40 |
| EmilienM | what's its url? | 13:40 |
| sshnaidm | EmilienM, like that? http://dashboard-ci.tripleo.org/d/si1tipHZk/jobs-exploration?orgId=1&fullscreen&panelId=12&var-influxdb_filter=job_name%7C%3D%7Ctripleo-ci-centos-7-undercloud-containers&from=now-90d&to=now | 13:41 |
| marios|ruck | sshnaidm: nice :) | 13:41 |
| EmilienM | ah | 13:42 |
| EmilienM | perfect, thanks | 13:42 |
| arxcruz | and i'm back | 13:42 |
| zbr_ | weshay: not sure about your epel-feelings but that is a reality: https://review.rdoproject.org/r/#/c/23826/6/playbooks/tripleo-ceph-integration/pre.yaml | 13:45 |
| sshnaidm | EmilienM, the new role takes more time although, we can now enable/disable paunch in any job by featureset_override, and can compare them: https://review.opendev.org/#/c/694393/ | 13:45 |
| EmilienM | sshnaidm: yeah I saw | 13:45 |
| rlandy | chkumar|rover: so baremetal is still failing os_tempest ... https://sf.hosted.upshift.rdu2.redhat.com/logs/59/186259/1/check/periodic-tripleo-ci-centos-7-bm_envA-3ctlr_1comp-featureset001-master/85df776/job-output.txt - I added these settings https://code.engineering.redhat.com/gerrit/#/c/186258/1/hardware_environments/dell_fc430_envA/network_configs/single_nic_vlans/env_settings.yml but clearly I'm missing one | 13:45 |
| EmilienM | sshnaidm: it's on my todo to continue this work | 13:45 |
| rlandy | chkumar|rover: how is it picking https://10.9.120.218 | 13:46 |
| sshnaidm | EmilienM, let's sync in your time, btw: https://review.opendev.org/#/c/695923/ | 13:46 |
| chkumar|rover | rlandy: need to check that, give me a min | 13:47 |
| *** rfolco has quit IRC | 13:47 | |
| rlandy | it should use the allocation_pools | 13:48 |
| *** rfolco has joined #oooq | 13:49 | |
| *** rfolco has quit IRC | 13:50 | |
| chkumar|rover | rlandy: now looking | 13:55 |
| *** ysandeep has joined #oooq | 13:59 | |
| sshnaidm | rlandy, weshay good news, I see cleanup working now, just need to improve it a bit | 14:02 |
| rlandy | k - good | 14:02 |
| sshnaidm | https://review.rdoproject.org/zuul/stream/23f421d4334045bb91416da6a5e0b8be?logfile=console.log | 14:02 |
| *** amoralej|lunch is now known as amoralej | 14:02 | |
| rlandy | anyone see the prod chain council meeting on their calendars? | 14:03 |
| rlandy | marios|ruck: ^^? | 14:03 |
| marios|ruck | rlandy: checking | 14:04 |
| marios|ruck | rlandy: nop but i think weshay wants you to ping him for addition (that's what i heard on the call just now) | 14:05 |
| marios|ruck | weshay: please add me and rlandy :D | 14:05 |
| rlandy | thanks | 14:05 |
| *** epoojad1 has quit IRC | 14:07 | |
| marios|ruck | chkumar|rover: we really need rhel8 promotion just filed that https://bugs.launchpad.net/tripleo/+bug/1854062 | 14:09 |
| openstack | Launchpad bug 1854062 in tripleo "rhel8 jobs failing for missing containers " Exception: Not found image: docker://..."" [Critical,Triaged] | 14:09 |
| chkumar|rover | marios|ruck: cool | 14:09 |
| marios|ruck | rlandy: do you agree with that? i mean that we need a new promotion/something went bad with rhel8 promotion (please if you have 2 mins see https://bugs.launchpad.net/tripleo/+bug/1854062 ) | 14:09 |
| chkumar|rover | marios|ruck: rhel8 fs01 one issue, one still penging | 14:09 |
| rlandy | looking | 14:09 |
| *** epoojad1 has joined #oooq | 14:11 | |
| *** bhagyashris has quit IRC | 14:13 | |
| rlandy | marios|ruck: that can happen if the containers were not all pushed/tagged | 14:14 |
| chkumar|rover | rlandy: sshnaidm https://review.opendev.org/678833 | 14:14 |
| marios|ruck | rlandy: ack yeah that's what i was thinking so really we need a new promotion to fix it and update curent-tripleo | 14:14 |
| * rlandy checks status of rhel-8 promotions | 14:14 | |
| marios|ruck | rlandy: thanks for checking | 14:14 |
| rlandy | you can force it | 14:14 |
| marios|ruck | rlandy: yeah we need fs1 (we have testproject etc) | 14:15 |
| marios|ruck | rlandy: thanks was mainly looking for sanity check on that bug | 14:15 |
| rlandy | marios|ruck: I dont see your testproject job running | 14:15 |
| marios|ruck | rlandy: well it timed out https://review.rdoproject.org/r/#/c/23822/ | 14:16 |
| chkumar|rover | marios|ruck: rhel8 fs1? | 14:16 |
| chkumar|rover | train? | 14:16 |
| *** ysandeep has quit IRC | 14:17 | |
| chkumar|rover | https://review.rdoproject.org/r/#/c/23825/ | 14:17 |
| rlandy | marios|ruck: remove fs01 temp from criteria and promote if you have to | 14:17 |
| rlandy | checking the timeout | 14:17 |
| marios|ruck | rlandy: but via http://38.145.34.55/redhat8_master.log that's the missing job (missing successful jobs: [u'periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master'] | 14:17 |
| marios|ruck | rlandy: ack on removing if we have to will keep in mind if we don't get it by tomorrow chkumar|rover | 14:17 |
| rlandy | marios|ruck: I will watch it this afternoon | 14:18 |
| marios|ruck | chkumar|rover: we need fs01 rhel8 for promotion | 14:18 |
| *** jpena|lunch is now known as jpena | 14:18 | |
| rlandy | and replace the criteria | 14:18 |
| rlandy | checking why you have a timeouw | 14:18 |
| rlandy | timeout | 14:18 |
| chkumar|rover | marios|ruck: ok | 14:18 |
| rlandy | 019-11-26 03:26:14.382266 | primary | TASK [overcloud-prep-images : Prepare the overcloud images for deploy] ********* | 14:18 |
| rlandy | 2019-11-26 03:26:14.399308 | primary | Tuesday 26 November 2019 03:26:14 -0500 (0:00:01.929) 0:00:17.522 ****** | 14:18 |
| rlandy | 2019-11-26 11:34:08.195273 | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/run-v3.yaml@master] | 14:18 |
| rlandy | 2019-11-26 11:34:08.195602 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/tripleo-ci-periodic-base/post.yaml@master] | 14:18 |
| rlandy | 2019-11-26 11:34:10.293151 | | 14:18 |
| marios|ruck | rlandy: thanks was gonna get to that i need to file another bug first | 14:19 |
| rlandy | weord | 14:19 |
| rlandy | 2019-11-26 03:29:37 | Introspection completed. | 14:19 |
| rlandy | 2019-11-26 03:29:37 | + openstack overcloud node provide --all-manageable | 14:19 |
| rlandy | 2019-11-26 03:29:39 | Waiting for messages on queue 'tripleo' with no timeout. | 14:19 |
| *** soniya29 has quit IRC | 14:19 | |
| marios|ruck | rlandy: yeah maybe i'll try a recheck there first | 14:19 |
| marios|ruck | rlandy: well, i can remove the centos job though | 14:19 |
| marios|ruck | it already served it purpose | 14:20 |
| marios|ruck | thanks for your time rlandy | 14:20 |
| rlandy | https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master | 14:20 |
| rlandy | marios|ruck: ^^ have a legit problem there | 14:20 |
| marios|ruck | rlandy: yeah :/ i know | 14:20 |
| rlandy | looking through those errors | 14:21 |
| zbr_ | marios|ruck: give https://review.opendev.org/#/c/696041/ a kick | 14:26 |
| *** rfolco has joined #oooq | 14:28 | |
| weshay | sshnaidm, w/ two minutes to spare | 14:28 |
| sshnaidm | weshay, ok | 14:30 |
| rlandy | marios|ruck: rerunning is not going to help unless you hold the nodes | 14:31 |
| rlandy | checking intro image prep workflow | 14:31 |
| marios|ruck | rlandy: ack yeah i am seeing that now going to kill that job then | 14:31 |
| *** dsneddon has quit IRC | 14:33 | |
| rlandy | openstack overcloud node provide --all-manageable - hang | 14:33 |
| marios|ruck | rlandy: i'll file a bug for it for starters anyway | 14:33 |
| rlandy | marios|ruck: https://bugs.launchpad.net/tripleo/+bug/1846791 | 14:33 |
| openstack | Launchpad bug 1846791 in tripleo "race, openstack overcloud node introspect --all-manageable --provide may hang" [Medium,Triaged] | 14:33 |
| marios|ruck | rlandy: and we might need to do the criteria skip for the promote | 14:33 |
| marios|ruck | rlandy: ah i see /me reads | 14:34 |
| rlandy | I haven't read it yet but the head line was good | 14:34 |
| *** dsneddon has joined #oooq | 14:38 | |
| *** TrevorV has joined #oooq | 14:40 | |
| rlandy | looks like mysq error | 14:40 |
| rlandy | Failed to create action: qinling.webhooks_list: AttributeError: 'NoneType' object has no attribute 'Client' | 14:40 |
| *** Goneri has quit IRC | 14:41 | |
| rlandy | your bmc is fine | 14:41 |
| rlandy | you have IPMI | 14:41 |
| rlandy | the status is not returning | 14:41 |
| marios|ruck | rlandy: where are you seeing that i couldn't see anything wrong in http://logs.rdoproject.org/22/23822/2/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/2052e10/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz looked like introspection completed | 14:42 |
| rlandy | marios|ruck: ack - ironic is doing fine - seeing errors in mistral | 14:42 |
| *** ysandeep has joined #oooq | 14:44 | |
| *** dsneddon has quit IRC | 14:45 | |
| rlandy | marios|ruck: forget that - it's in passing centos 7 job as well | 14:48 |
| marios|ruck | rlandy: k gonna file it and dig from there thanks, don't mean to eat allyr time | 14:49 |
| *** ysandeep has quit IRC | 14:49 | |
| * marios|ruck tea brb | 14:49 | |
| rlandy | marios|ruck: k - will look at it a bit more and leave you notes on etherpad if I see anything | 14:49 |
| chkumar|rover | Few of the check jobs might hitted pypi connection issue Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read)) during dlrn installation | 14:50 |
| rlandy | my best guess is ssl | 14:53 |
| *** Goneri has joined #oooq | 14:53 | |
| rlandy | Enforcing | 14:58 |
| rlandy | http://logs.rdoproject.org/22/23822/2/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/2052e10/logs/undercloud/var/log/extra/selinux.txt.gz | 14:58 |
| rlandy | http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/0f19b3c/logs/undercloud/var/log/extra/selinux.txt.gz | 14:59 |
| rlandy | ^^ permissive in passing | 14:59 |
| rlandy | marios|ruck: ^^ | 14:59 |
| zbr_ | arxcruz: did we resolve the issue with non voting zuul jobs breaking podman? if a job reports green is effectively useless | 14:59 |
| marios|ruck | rlandy: ack great thank you | 15:00 |
| arxcruz | zbr_: I'm working on that | 15:00 |
| zbr_ | arxcruz: asking because I convinced https://review.rdoproject.org/r/#/c/23826/ to run "make rpm", which is a clear progress. should I remove WIP and merge it or remove the forced-success on it and keep it open till we have solution for github? | 15:02 |
| rlandy | periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-masteropenstack/tripleo-cimasteropenstack-periodic-mastermaster22082019-11-18T13:12:45FAILURE | 15:02 |
| rlandy | periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-masteropenstack/tripleo-cimasteropenstack-periodic-mastermaster84422019-11-18T01:02:36SUCCESS | 15:02 |
| rlandy | marios|ruck: ^^ that's the date/tie of the permissive/enforcing change | 15:02 |
| arxcruz | zbr_: no, make rpm doesn't work, keep it because i'm working on that part | 15:02 |
| chkumar|rover | zbr_: that issue got fixed, if zuul job pass or fail it will not blok | 15:03 |
| *** jaosorior has joined #oooq | 15:04 | |
| *** saneax has quit IRC | 15:04 | |
| marios|ruck | rlandy: thanks... denials file looks busy at least ! http://logs.rdoproject.org/22/23822/2/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/2052e10/logs/undercloud/var/log/extra/denials.txt.gz | 15:06 |
| marios|ruck | rlandy++ | 15:06 |
| rlandy | marios|ruck: yeah - we just need to figutre out now what flipped that enforcing switch | 15:06 |
| rlandy | on Nov 18th ish | 15:07 |
| chkumar|rover | rlandy: got the issue https://code.engineering.redhat.com/gerrit/#/c/186258/1 | 15:07 |
| chkumar|rover | check comments | 15:07 |
| chkumar|rover | rlandy: https://opendev.org/openstack/openstack-ansible-os_tempest/src/branch/master/defaults/main.yml#L135 | 15:08 |
| rlandy | chkumar|rover: the errir os still out of range | 15:08 |
| rlandy | error is | 15:08 |
| rlandy | it picks 218 | 15:09 |
| rlandy | but I'll correct that | 15:09 |
| *** dsneddon has joined #oooq | 15:10 | |
| rlandy | https://10.9.120.218:13696/v2.0/s | 15:11 |
| rlandy | outside allocation pools of 214 | 15:11 |
| rlandy | I can change the allocation pools | 15:11 |
| rlandy | but the start is correct | 15:11 |
| rlandy | somewhere it's pulling 218 | 15:11 |
| rlandy | it should choose a value with my allocation pools | 15:12 |
| chkumar|rover | rlandy: yes, here, allocation pool does not got applied may be that's why? | 15:13 |
| *** ykarel is now known as ykarel|pto | 15:14 | |
| *** jpich has quit IRC | 15:15 | |
| chkumar|rover | rlandy: Do we need to redeploy the whole setup again or just changing in tempest.yaml will work | 15:16 |
| rlandy | chkumar|rover: no - you can get on the box and change the values | 15:16 |
| chkumar|rover | rlandy: will take a look, time to head home, if still does not work | 15:17 |
| chkumar|rover | marios|ruck: rlandy see ya | 15:17 |
| *** chkumar|rover is now known as raukadah | 15:17 | |
| rlandy | https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/RedHat-8/promotion-testing-hash-master.yml#L89 | 15:17 |
| rlandy | marios|ruck: ^^ should be permissive | 15:17 |
| rlandy | raukadah: I'll check it out - thanks | 15:17 |
| marios|ruck | rlandy: thank you added it in https://bugs.launchpad.net/tripleo/+bug/1854073 | 15:19 |
| openstack | Launchpad bug 1854073 in tripleo "periodic rhel-8-ovb-3ctlr_1comp-featureset001-master times out during overcloud-prep-images" [Critical,Triaged] | 15:19 |
| marios|ruck | rlandy: checking git blame on that | 15:19 |
| rlandy | it's ages ago | 15:19 |
| rlandy | it's correct | 15:19 |
| marios|ruck | rlandy: oh it is permissive | 15:19 |
| rlandy | something is changing it | 15:19 |
| marios|ruck | rlandy: ack | 15:19 |
| rlandy | # Enable or disable SELinux during the deployment. (boolean value) | 15:30 |
| rlandy | undercloud_enable_selinux = true | 15:30 |
| rlandy | http://logs.rdoproject.org/22/23822/2/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/2052e10/logs/undercloud/home/zuul/undercloud.conf.txt.gz | 15:31 |
| rlandy | ^^ change | 15:31 |
| rlandy | compare to http://logs.rdoproject.org/22/23822/2/check/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/ad7f180/logs/undercloud/home/zuul/undercloud.conf.txt.gz | 15:31 |
| marios|ruck | rlandy: but i think its different one for overcloud | 15:31 |
| rlandy | this is undercloud | 15:31 |
| rlandy | you're intropsecting | 15:31 |
| rlandy | not deploying | 15:31 |
| marios|ruck | rlandy: ah right! | 15:31 |
| rlandy | idk - nothing changed recently | 15:35 |
| * rlandy goes back to fixing baremetal | 15:35 | |
| marios|ruck | rlandy: thats very helpful though i'm sure we'll find it soon enough thanks | 15:35 |
| marios|ruck | rlandy: thanks for your help | 15:35 |
| *** derekh has quit IRC | 15:38 | |
| *** derekh has joined #oooq | 15:38 | |
| zbr_ | arxcruz: libpod has not target 'rpm' in its Makefile | 15:38 |
| zbr_ | but I was able to make it build, after LOTS of changes. | 15:39 |
| marios|ruck | rlandy: fyi enabled by default in rhel https://github.com/openstack/tripleo-quickstart-extras/blob/e1e470ae0e97f9df7ed4341f35cf616977415e41/roles/undercloud-deploy/defaults/main.yml#L87-L92 | 15:39 |
| rlandy | marios|ruck: yes - but something disabled it before | 15:39 |
| rlandy | before Nov 18 | 15:40 |
| marios|ruck | rlandy: i see. k don't mean to suck you back in here was just pointing that out i will refrain from harrassing you further about that :D | 15:40 |
| marios|ruck | (till tomorrow anyway ) | 15:40 |
| rlandy | marios|ruck: I would try a job setting that in fs | 15:41 |
| rlandy | and see what you get | 15:41 |
| marios|ruck | rlandy: ack yeah good idea | 15:41 |
| rlandy | testproject | 15:41 |
| arxcruz | zbr_: I told you i was working on that, there's a more easy way to do it... | 15:47 |
| zbr_ | arxcruz: add me CR that add that, so far I think that by the end of the day I will have the jobs build successfully podman on both centos 7 and 8, a step that was required for building a RPM | 15:56 |
| weshay | marios|ruck, the settings for selinux are different variables for then undercloud and overcloud | 15:58 |
| weshay | marios|ruck, updated the bug.. rhel8 is running w/ selinux enforcing on the undercloud :( | 15:59 |
| marios|ruck | weshay: yeah it defaults to enabled but rlandy reckons it was disabled before ~ 18th | 15:59 |
| marios|ruck | weshay: defaults there 17:39 < marios|ruck> rlandy: fyi enabled by default in rhel | 15:59 |
| marios|ruck | https://github.com/openstack/tripleo-quickstart-extras/blob/e1e470ae0e97f9df7ed4341f35cf616977415e41/roles/undercloud-deploy/defaults/main.yml#L87-L92 | 15:59 |
| rlandy | yep - the default is enforcing for rhel | 16:00 |
| weshay | marios|ruck, ya.. the right place to override that would be the release files for RHEL | 16:00 |
| rlandy | but that setting hasn't changed since sep | 16:00 |
| weshay | so we can be sure we don't mess w/ osp deployments | 16:00 |
| marios|ruck | weshay: basically, we really need promotino to fix that one https://bugs.launchpad.net/tripleo/+bug/1854062 but the one job we need fs1 (via promoter logs it is the only one missing) is failing on this selinux issue now | 16:00 |
| openstack | Launchpad bug 1854062 in tripleo "rhel8 jobs failing for missing containers " Exception: Not found image: docker://..."" [Critical,Triaged] | 16:00 |
| rlandy | thank you for remembering osp | 16:00 |
| marios|ruck | all hands retro thing now | 16:00 |
| rlandy | marios|ruck: if you add the setting in testproject and it passes criteria will be met | 16:01 |
| rlandy | so you may get away with it before it merges | 16:01 |
| marios|ruck | rlandy: ack ok will do that... wondering though can i just add it in vars? or do i need to do it with featureset override addition or something | 16:01 |
| marios|ruck | rlandy: vars should work ... | 16:02 |
| marios|ruck | rlandy: cos is a role | 16:02 |
| *** dtantsur is now known as dtantsur|afk | 16:03 | |
| rlandy | marios|ruck: vars will not work - it doesn;t get passed to any deploy work | 16:03 |
| rlandy | marios|ruck: fs_override will work | 16:03 |
| marios|ruck | rlandy: hmmm but do we have it in the allowed overrides checking | 16:04 |
| rlandy | but weshay is right that the release file is the right way to go | 16:04 |
| weshay | rlandy, marios|ruck https://review.opendev.org/696119 | 16:04 |
| marios|ruck | rlandy: weshay: ack thanks | 16:04 |
| rlandy | interesting | 16:04 |
| * rlandy looks | 16:04 | |
| marios|ruck | rlandy: doesn't look like it i'd have to add https://github.com/openstack/tripleo-ci/blob/2f6cc0b80bf90ebefd7620a57e02ffb8f15a9d48/roles/run-test/tasks/main.yaml#L8 | 16:05 |
| marios|ruck | rlandy: but instead i can just depends-on weshay patch | 16:05 |
| rlandy | marios|ruck: ack - depends on is the way to go | 16:06 |
| marios|ruck | weshay: rlandy: fyi there https://review.rdoproject.org/r/#/c/23822/ | 16:07 |
| rlandy | overcloud_selinux_enabled: permissive | 16:07 |
| rlandy | undercloud_selinux_enabled: false | 16:07 |
| * marios|ruck hoping we get it for program call tomorrow | 16:07 | |
| rlandy | ^^ interesting that the values are different | 16:07 |
| weshay | marios|ruck, raukadah http://rhos-release.virt.bos.redhat.com:3030/rhosp you guys will be green for tomorrows mtg | 16:08 |
| weshay | focuses on train only | 16:08 |
| marios|ruck | weshay: yeah would be nice to get train from today (I know we promoted yesterday) and master did today | 16:09 |
| rlandy | ugh - just lost env | 16:21 |
| *** tesseract has quit IRC | 16:23 | |
| *** akahat has quit IRC | 16:25 | |
| panda | marios|ruck: rfolco try: merge_requests.enqueue("https://review.rdoproject.org/r/22890","https://review.rdoproject.org/r/23717","https://review.rdoproject.org/r/23773") except AlreadyEnqueued: raise_priority() except NotNowPanda: raise_fists() | 16:33 |
| marios|ruck | panda: ack will look forward to those for breakfast | 16:36 |
| rfolco | panda, starting review now just to beat marios|ruck | 16:36 |
| marios|ruck | :( | 16:36 |
| * panda patpat marios|ruck | 16:36 | |
| marios|ruck | why so much hate in your heart rfolco what did i eeeever do toyou | 16:37 |
| panda | "there there" | 16:37 |
| rfolco | this is what call "healthy competition", leverages motivation and productivity | 16:38 |
| rfolco | :) | 16:38 |
| marios|ruck | so now you're calling me unmotivated and uproductive ? | 16:39 |
| rfolco | lol | 16:39 |
| panda | --talking; ++reviewing | 16:44 |
| *** jpena is now known as jpena|brb | 16:45 | |
| rlandy | marios|ruck: raukadah: you're saved from updating the pipelines to use image builds jobs as I can't get https://review.opendev.org/#/c/695575/ through gates | 16:47 |
| marios|ruck | rlandy: ack ok revisit tomorrow then | 16:48 |
| rlandy | if the gate gods agree then | 16:48 |
| zbr_ | rlandy: i am afraid that gods left the planet long time ago ;) | 16:50 |
| rlandy | zbr_: I am a person of deep faith | 16:51 |
| rlandy | {read} I really need this patch to merge | 16:52 |
| rlandy | weshay: rfolco: I'd like to merge some patches for the component pipeline: | 16:55 |
| rlandy | https://review.opendev.org/#/c/695990/ | 16:55 |
| rlandy | https://review.opendev.org/#/c/695992/ | 16:55 |
| rlandy | ^^ (or rfolco's rolled back one) | 16:55 |
| rlandy | then this can go .. https://review.rdoproject.org/r/#/c/23792/ | 16:56 |
| rlandy | per test run here ... https://review.rdoproject.org/r/#/c/23749/ | 16:56 |
| rlandy | will save me carrying those changes in the pipeline tests | 16:56 |
| * marios|ruck out as soon as this call ends | 16:58 | |
| *** marios|ruck has quit IRC | 17:00 | |
| *** sshnaidm is now known as sshnaidm|afk | 17:05 | |
| *** udesale has quit IRC | 17:09 | |
| *** jpena|brb is now known as jpena | 17:21 | |
| weshay | rlandy, fyi https://review.opendev.org/#/c/696081/2 | 17:24 |
| rlandy | looking | 17:25 |
| weshay | dat annoying logger message | 17:26 |
| *** jaosorior has quit IRC | 17:32 | |
| *** ykarel|pto has quit IRC | 17:34 | |
| rlandy | any thought on my merge requests? | 17:42 |
| rlandy | <rlandy> https://review.opendev.org/#/c/695990/ | 17:46 |
| rlandy | <rlandy> https://review.opendev.org/#/c/695992/ | 17:46 |
| rlandy | how about it?? | 17:46 |
| rlandy | weshay: ^^ please | 17:47 |
| rlandy | saves me carrying them on the pipeline review | 17:47 |
| * weshay loooks | 17:47 | |
| rlandy | thank you | 17:47 |
| rlandy | one more | 17:47 |
| rlandy | https://review.rdoproject.org/r/#/c/23792/ | 17:47 |
| rlandy | not sure if I need the var or note | 17:47 |
| rlandy | not | 17:47 |
| weshay | ugh.. allowed_overrides? | 17:47 |
| weshay | is that the only way? | 17:48 |
| weshay | if it saves 50 files | 17:48 |
| weshay | then fine... | 17:48 |
| raukadah | zbr_: Hello | 17:48 |
| rlandy | weshay: open to better suggestions | 17:49 |
| *** skramaja_ has quit IRC | 17:49 | |
| zbr_ | hi | 17:49 |
| raukadah | zbr_: https://review.rdoproject.org/r/#/c/23826/ regarding CentoS7 was not in scope | 17:49 |
| rlandy | weshay: there is another way | 17:49 |
| rlandy | and that is to create an additional repo on the fly | 17:49 |
| rlandy | like gating | 17:49 |
| weshay | rlandy, in this case.. component is a little like " - 'standalone_environment_files' | 17:49 |
| weshay | " | 17:49 |
| zbr_ | raukadah: ok, no problem with that, my focus was c8 anyway. | 17:49 |
| raukadah | zbr_: centos8 is also not inscope | 17:50 |
| raukadah | only RHEL8 for ceph-ansible and podman | 17:50 |
| weshay | rlandy, for recreating a component job.. | 17:50 |
| rlandy | weshay: leave the review | 17:50 |
| weshay | how does one pass it? | 17:50 |
| rfolco | rlandy, you don't need the var anymore | 17:50 |
| rlandy | there are in fact two options | 17:50 |
| weshay | I guess in testproject | 17:50 |
| weshay | k.. | 17:50 |
| raukadah | zbr_: I will go through the review and add comments | 17:50 |
| rfolco | rlandy, only the featureset_override | 17:50 |
| * weshay thinking through it | 17:50 | |
| rlandy | no - there are other ways | 17:50 |
| weshay | rlandy, just on the defensive because it's in allowed_overrides | 17:50 |
| rlandy | I just thought you guys agreed on the release file already | 17:51 |
| weshay | yes.. on the release file | 17:51 |
| rlandy | so - all you need is another repo? | 17:51 |
| weshay | but I probably didn't think through how that var was passed | 17:51 |
| zbr_ | raukadah: i guess it will be fun to get those dependencies for rhel. | 17:51 |
| rfolco | weshay, rlandy thats why I had proposed one file per component | 17:51 |
| rlandy | weshay: rfolco: hold on ... let me rethink this then | 17:51 |
| rfolco | hardcoded in the release file | 17:52 |
| rlandy | we can use the standard release file | 17:52 |
| raukadah | zbr_: on RHEL8 node, after enabling rhui repo, does packages are not there? | 17:52 |
| weshay | rfolco, well.. I think you did the one file per.. because that was my ask first.. and it's quick and easy | 17:52 |
| rlandy | and just add and enable another repo | 17:52 |
| rlandy | that's all you need, right | 17:52 |
| weshay | rfolco, but we have some time now... | 17:52 |
| raukadah | zbr_: https://github.com/containers/libpod/blob/master/contrib/build_rpm.sh#L20 these packages | 17:52 |
| rlandy | repo_setup allows to add additional repos | 17:52 |
| weshay | rfolco, now that we're in the next sprint | 17:52 |
| weshay | I'm 75% fine w/ allowed_overrides | 17:53 |
| rfolco | 75%? | 17:53 |
| weshay | just need to poke at it a bit | 17:53 |
| rlandy | ok -let me try the additional repo out | 17:53 |
| weshay | rlandy, wait wait | 17:53 |
| rlandy | we can get by w/o it | 17:53 |
| rfolco | we are changing all the design just coz one optimization | 17:53 |
| weshay | rlandy, what would the additional repo look like? can you etherpad it? | 17:53 |
| weshay | rfolco, it's fine. .it's a good optimzation | 17:54 |
| weshay | but just need to poke at it | 17:54 |
| weshay | as well | 17:54 |
| rlandy | k - I need to experiment a bit | 17:54 |
| rlandy | also - quick lunch - brb | 17:54 |
| weshay | rlandy, ok.. time cap it.. run it by me later | 17:54 |
| rfolco | pick the override then | 17:54 |
| rlandy | we did it for the ceph-ansibel stuff | 17:54 |
| rfolco | and remove the other var | 17:54 |
| raukadah | rlandy: does baremetal tempest work? | 17:54 |
| rfolco | no need for 2 vars there | 17:54 |
| rlandy | weshay: ack - you can stop me this afternoon | 17:55 |
| weshay | k | 17:55 |
| rlandy | raukadah: not sure - sshnaidm|afk started another job that took my ready env - I restarted the test with train - but I knwo where it takes that value from | 17:55 |
| weshay | rlandy, I think I thought the var would come directly from zuul config for the job, w/o remembering we needed this pass through | 17:55 |
| zbr_ | raukadah: i am going to update the change to run on rhel-8 too, is very easy. | 17:56 |
| raukadah | zbr_: make sure rhui repo is enabled there | 17:57 |
| weshay | rlandy, btw.. component_name may be a better var name.. than comonent_promotion | 17:57 |
| weshay | but naming is hard | 17:57 |
| zbr_ | raukadah: don't we have a nodepool image that already has these requirementes? | 17:57 |
| rlandy | weshay: let's reconsider after we try the additional repo | 17:57 |
| weshay | ack | 17:57 |
| rlandy | overcloud_public_vip: 10.9.121.218 | 17:57 |
| rlandy | ^^ trying to use that valye | 17:58 |
| rlandy | PublicVirtualFixedIPs: [{"ip_address": "10.9.121.218"}] | 17:58 |
| rlandy | we never set anything that specific in our other runs | 17:58 |
| rlandy | but in B< it's required | 17:58 |
| raukadah | zbr_: sometimes it is enabled sometimes not | 17:58 |
| rlandy | so I am retrying with extended range | 17:58 |
| rlandy | lunch - brb | 17:59 |
| *** derekh has quit IRC | 18:00 | |
| raukadah | weshay: arxcruz since we have everywhere standalone, I am going to depreacate os_tempest jobs from all projects | 18:25 |
| weshay | rock on king kong | 18:25 |
| *** jaosorior has joined #oooq | 18:26 | |
| *** jpena is now known as jpena|off | 18:31 | |
| raukadah | rfolco: Hello, https://tree.taiga.io/project/tripleo-ci-board/us/1324?milestone=247128 Does tech debts is for ruck/rover or for sprint members? | 18:35 |
| rlandy | back | 18:37 |
| rfolco | raukadah, I am not quite sure, pls check w/ weshay | 18:37 |
| raukadah | weshay: ^^ | 18:37 |
| *** amoralej is now known as amoralej|off | 18:48 | |
| weshay | raukadah, open to anyone | 18:51 |
| rlandy | arxcruz; hey - can you point me to where you added the additional_ repo for repo_steup in ceph ansible | 18:59 |
| *** apevec has joined #oooq | 19:00 | |
| rlandy | nvm - I think I see an example | 19:03 |
| raukadah | :-) | 19:04 |
| raukadah | paul is doing some existing work related to zuul artifacts, and collections might be reused need to take a look | 19:05 |
| raukadah | rlandy: weshay in mode of merging stuff? | 19:06 |
| raukadah | rlandy: weshay I preassume yes, https://review.opendev.org/#/c/694907/ and https://review.opendev.org/#/c/694719/ | 19:06 |
| rlandy | raukadah: looking | 19:06 |
| * raukadah is done with os_tempest porting | 19:06 | |
| rlandy | rfolco: weshay: have five minutes to discuss component repos? | 19:12 |
| rlandy | new solution | 19:12 |
| weshay | rlandy, sure | 19:13 |
| rfolco | rlandy, where | 19:13 |
| rlandy | now? | 19:15 |
| weshay | sure | 19:16 |
| rlandy | https://meet.google.com/qdi-mwzv-gmj?pli=1&authuser=0 | 19:16 |
| raukadah | rlandy: rfolco please keep the new solution on hackmd | 19:17 |
| rlandy | weshay: ^^ you joining? | 19:18 |
| rlandy | got to pick up shoes - brb | 19:56 |
| *** rlandy is now known as rlandy|brb | 19:56 | |
| *** fmount has quit IRC | 20:02 | |
| *** fmount has joined #oooq | 20:04 | |
| *** rlandy|brb is now known as rlandy | 20:10 | |
| *** jfrancoa has quit IRC | 20:10 | |
| rlandy | https://review.rdoproject.org/zuul/stream/0277d634096146b3ac0c07b99a2c7cdb?logfile=console.log ... not good | 20:16 |
| rlandy | hmmm ... means the permissive solution did not help | 20:17 |
| rlandy | weshay: ^^ fyi | 20:17 |
| rlandy | will check it out when the logs come through | 20:17 |
| rlandy | rfolco: weshay: here is an old review where I tried to do something similar: https://review.opendev.org/#/c/636406/ | 20:21 |
| rlandy | add jobs variables to be available in toci | 20:21 |
| rlandy | ^^ better direct pass through of jab vars to toci | 20:22 |
| *** holser has joined #oooq | 20:25 | |
| rfolco | rlandy, +1 on the vars bit | 20:34 |
| rfolco | rlandy, one suggestion.... | 20:35 |
| rfolco | rlandy, what if we ALWAYS ADD the component repo to the existing release file (don't use a new one), and enabled it at the end only if component_promotion is defined ? | 20:36 |
| rfolco | rlandy, so we would revert the custom release file.... | 20:36 |
| rfolco | we add the repo always | 20:36 |
| rfolco | enable if its a component job inly | 20:36 |
| rfolco | only | 20:36 |
| rfolco | here https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-7/promotion-testing-hash-master.yml#L107-L123 | 20:36 |
| rfolco | rlandy, wtdy ? | 20:36 |
| rfolco | wtyt | 20:36 |
| rfolco | tdyt | 20:37 |
| rfolco | wdyt | 20:37 |
| rfolco | arrgrghrgr | 20:37 |
| rfolco | my fat fingers | 20:37 |
| rfolco | rlandy, let me know if you like it | 20:37 |
| rfolco | just like other custom repos work | 20:38 |
| rlandy | rfolco: that's a fine solution - downside is that each release file has to be edited and we will still need a way of knowing that it is a component job, so ... | 20:38 |
| rlandy | we'd have to have a condition like if component on zuul.job_name | 20:38 |
| rfolco | nooo, I like your idea for the vars | 20:39 |
| rfolco | plus my idea of enabling it in the same config file we use today | 20:39 |
| rfolco | component_promotion: compute | 20:39 |
| rfolco | then the repo is always present there in https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-7/promotion-testing-hash-master.yml#L46 | 20:40 |
| rfolco | and then is conditionally enabled here https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-7/promotion-testing-hash-master.yml#L107 | 20:40 |
| rlandy | we'd need a way of knowing what component we are working with | 20:40 |
| rlandy | it's better than a whole new release filw | 20:41 |
| rlandy | file | 20:41 |
| rfolco | if you pass the var like you suggested, then the release file will have it, so the baseurl can be genetic w/ {{ component_promotion }} translated | 20:42 |
| rfolco | generic*** | 20:42 |
| rfolco | not genetic :) | 20:42 |
| *** ccamacho has quit IRC | 20:46 | |
| *** tosky has quit IRC | 20:46 | |
| *** holser has quit IRC | 20:51 | |
| *** TrevorV has quit IRC | 20:59 | |
| *** jtomasek has quit IRC | 21:00 | |
| *** holser has joined #oooq | 21:01 | |
| *** rfolco has quit IRC | 21:14 | |
| *** holser has quit IRC | 21:36 | |
| rlandy | http://logs.rdoproject.org/22/23822/4/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/0277d63/logs/undercloud/var/log/extra/selinux.txt.gz permissive - still stalled | 21:37 |
| rlandy | http://logs.rdoproject.org/49/23749/7/check/periodic-tripleo-ci-centos-7-standalone-compute-master/7710d4a/logs/undercloud/etc/yum.repos.d/component.repo.txt.gz | 21:38 |
| rlandy | ^^ that wors as a start | 21:38 |
| rlandy | works | 21:38 |
| rlandy | add-Repo is functional | 21:38 |
| *** holser has joined #oooq | 21:41 | |
| weshay | rlandy, you have a minute | 21:50 |
| weshay | or am I too late? | 21:50 |
| rlandy | weshay: ack | 21:50 |
| rlandy | you have two hours | 21:50 |
| weshay | rlandy, k.. https://meet.google.com/snx-vyhj-hyj?authuser=1 | 21:50 |
| weshay | rlandy name: tripleo-ci-base-singlenode-rdo | 21:51 |
| weshay | rlandy, https://review.rdoproject.org/r/#/c/23840/3/.zuul.yaml | 21:52 |
| weshay | rlandy, single-centos-7-node | 21:54 |
| weshay | rlandy, https://softwarefactory-project.io/r/#/c/17091/ | 21:57 |
| *** jaosorior has quit IRC | 21:59 | |
| weshay | rlandy, https://review.rdoproject.org/r/#/c/23840/3/.zuul.yaml | 22:03 |
| *** apevec has quit IRC | 22:03 | |
| rlandy | weshay: does the vexxhost test work now? | 22:17 |
| weshay | rlandy, so far /me checks again | 22:20 |
| weshay | rlandy, needs a few more minutes | 22:20 |
| *** epoojad1 has quit IRC | 22:21 | |
| weshay | ah.. this is cool | 22:26 |
| weshay | https://e2b6e16adf31aa974ee6-a5ee49b92ce983f885f1f8e1b503fb0b.ssl.cf2.rackcdn.com/695575/6/gate/tripleo-ci-centos-7-scenario001-standalone/b914a5b/logs/_Failed_to_start_collectd_container.log | 22:26 |
| weshay | rlandy, that's your gate job from overcloud image builds | 22:26 |
| * weshay rechecks | 22:27 | |
| weshay | hrm.. 4 jobs failed on collectd | 22:28 |
| rlandy | what's this? | 22:28 |
| weshay | rlandy, was just looking at gate failures | 22:28 |
| rlandy | that failed gate again?? | 22:28 |
| weshay | one was your patch.. your attention is not required.. other than the sova failure identifcation is cool | 22:28 |
| rlandy | https://review.opendev.org/#/c/695575/ | 22:28 |
| weshay | ya | 22:28 |
| rlandy | weshay: it is - that's the fourth time | 22:28 |
| rlandy | I better check it's not legit | 22:29 |
| weshay | omg.. we need some excludes on that | 22:29 |
| weshay | i cant be | 22:29 |
| weshay | it can't be | 22:29 |
| weshay | totally unrelated | 22:29 |
| weshay | rlandy, I think this node is working on vexx btw :) | 22:29 |
| rlandy | weshay: it fails a different job each time | 22:29 |
| weshay | rlandy++ | 22:29 |
| weshay | ya.. too many jobs | 22:29 |
| rlandy | very equal opportunity failure | 22:29 |
| weshay | it's like a promotion job | 22:29 |
| rlandy | ugh | 22:29 |
| weshay | jebus | 22:29 |
| rlandy | I can't get the changes to the pipeline in w/o it | 22:30 |
| rlandy | good for the vexx stuff | 22:30 |
| rlandy | https://review.rdoproject.org/zuul/stream/a8219daf4b2c4ce08300de77bcb32892?logfile=console.log | 22:30 |
| rlandy | I got some traction with add_repos | 22:30 |
| weshay | ++ | 22:31 |
| rlandy | ugh - I guess I'll recheck that patch again | 22:31 |
| rlandy | oh - you juts did | 22:31 |
| rlandy | thanks | 22:31 |
| weshay | np | 22:31 |
| rlandy | we need a way just to rerun gate | 22:33 |
| rlandy | this failing high school and going back to first grade is a waste of resources | 22:33 |
| rlandy | no such functionality, right? | 22:34 |
| rlandy | weshay: ^^? | 22:34 |
| weshay | zuul doesn't allow it.. | 22:34 |
| rlandy | picky | 22:35 |
| weshay | rlandy, but.. we have to be much more targetet w/ the jobs we're running in tripleo-ci | 22:35 |
| rlandy | amen to that | 22:35 |
| *** ysandeep has joined #oooq | 23:12 | |
| *** rfolco has joined #oooq | 23:47 | |
| *** rfolco has quit IRC | 23:51 | |
| *** rfolco has joined #oooq | 23:51 | |
| *** rfolco has quit IRC | 23:55 | |
| *** rfolco has joined #oooq | 23:55 | |
| *** rlandy has quit IRC | 23:58 | |
| *** rlandy has joined #oooq | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!