| *** ysandeep|out is now known as ysandeep | 05:58 | |
| *** marios is now known as marios|ruck | 06:02 | |
| *** ysandeep is now known as ysandeep|brb | 06:42 | |
| *** amoralej|off is now known as amoralej | 07:02 | |
| *** ysandeep|brb is now known as ysandeep | 07:22 | |
| Tengu | hello there - can anyone push this in the gate? https://review.opendev.org/c/openstack/tripleo-ci/+/825437 | 07:56 |
|---|---|---|
| Tengu | that would allow to re-use existing container or env when we have a scheduled build.. | 07:57 |
| *** jpena|off is now known as jpena | 08:38 | |
| ysandeep | Tengu, looking ^^ | 09:27 |
| ysandeep | okay, already +wed | 09:28 |
| Tengu | ysandeep: yep :). but thanks for looking | 09:30 |
| amoralej | are you seeing issues with opstools repo? | 09:31 |
| ysandeep | ^^ may be marios|ruck noticed anything? | 09:34 |
| marios|ruck | ysandeep: there are a few issues ongoing but haven't come across opstools issue yet | 09:41 |
| marios|ruck | amoralej: we have issue possibly with appstream there https://bugs.launchpad.net/tripleo/+bug/1959524 | 09:41 |
| amoralej | yeah, that's different | 09:42 |
| ysandeep | marios|ruck: ack, thank you! | 09:42 |
| amoralej | you may start getting issues with opstools | 09:42 |
| amoralej | background https://review.rdoproject.org/r/c/rdo-infra/ansible-role-dlrn/+/38673 | 09:42 |
| marios|ruck | amoralej: what issue? we have one for pacemaker-cluster-libs not sure if that is opstools | 09:42 |
| amoralej | i'm setting manually | 09:42 |
| amoralej | in delorean-deps.repo | 09:42 |
| amoralej | but you may hit it for repos configured via oooq config | 09:43 |
| marios|ruck | amoralej: noting | 09:43 |
| ysandeep | https://logserver.rdoproject.org/openstack-component-glance/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-glance-master/a3dff07/job-output.txt | 09:47 |
| ysandeep | 2022-01-31 09:17:06.521213 | primary | "msg": "Failed to download metadata for repo 'RDO-CentOS8-deps': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried", | 09:47 |
| ysandeep | I noticed some mirror issues as well | 09:47 |
| ysandeep | marios|ruck, amoralej ^^ | 09:48 |
| marios|ruck | ysandeep: ack | 09:48 |
| ysandeep | looks like started recently, happening with lot of running jobs now - https://review.rdoproject.org/zuul/status#openstack-component-glance | 09:48 |
| amoralej | ysandeep, i'm not sure, tbh | 09:53 |
| amoralej | could you look where that repo is created? | 09:55 |
| amoralej | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/multi-node-bridge/templates/zuul-multi-node-bridge-ovs.repo.j2 | 09:56 |
| amoralej | aaahhh | 09:56 |
| amoralej | also related | 09:56 |
| amoralej | sending the patch | 09:56 |
| bhagyashris | folks please add in your review list https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/38667 | 09:57 |
| bhagyashris | frenzyfriday, will need your help here https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/38667 | 09:57 |
| bhagyashris | victoria and ussuri promotion is blocked | 09:57 |
| ysandeep | amoralej, sry missed your ping, https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/multi-node-bridge/templates/zuul-multi-node-bridge-ovs.repo.j2#L17 - need to move to cs8 stream? | 10:01 |
| marios|ruck | amoralej: any ideas about that one https://bugs.launchpad.net/tripleo/+bug/1959526/comments/2 appstream repo | 10:01 |
| marios|ruck | amoralej: but the package is there | 10:01 |
| marios|ruck | so maybe mirror issue not sure | 10:01 |
| *** ysandeep is now known as ysandeep|coffee | 10:04 | |
| amoralej | marios|ruck, we are still landing patches in stable/train, right? | 10:06 |
| marios|ruck | amoralej: yes | 10:06 |
| amoralej | ysandeep|coffee, yes, i'm sengind a patch | 10:06 |
| marios|ruck | amoralej: so then we'll need a patch for quickstart too? | 10:07 |
| amoralej | marios|ruck, quickstart install repos from config or pull delorean-deps.repo? | 10:07 |
| marios|ruck | amoralej: https://opendev.org/openstack/tripleo-quickstart/src/commit/358d8da7877a5cb408bbf79c80ddd42edf9ef553/config/release/tripleo-ci/CentOS-8/promotion-testing-hash-master.yml#L175 | 10:07 |
| marios|ruck | where export NODEPOOL_CENTOS_MIRROR={{ ansible_env['NODEPOOL_CENTOS_MIRROR']|default('http://mirror.centos.org/centos', true) }} | 10:08 |
| amoralej | marios|ruck, we have a couple of different cases | 10:08 |
| amoralej | gimme a while please | 10:08 |
| marios|ruck | amoralej: ack | 10:08 |
| frenzyfriday | bhagyashris, checking | 10:10 |
| amoralej | marios|ruck, ysandeep|coffee https://review.opendev.org/c/zuul/zuul-jobs/+/827032 that is fix for the issue in multi-node-bridge | 10:12 |
| amoralej | marios|ruck, yes, we need to fix that | 10:13 |
| amoralej | until we have collectd properly in the mirrors for cs8, i've created https://trunk.rdoproject.org/centos8-master/deps/collectd/ as workaround | 10:13 |
| frenzyfriday | bhagyashris, https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-centos-8-scenario010-kvm-internal-standalone-ussuri - ussuri sc10 kvm internal seems to be running, but I think it is not reporting to dlrn maybe? | 10:16 |
| bhagyashris | frenzyfriday, yeah | 10:17 |
| frenzyfriday | hm.. checking Meanwhile ++ to the patch ^^ | 10:17 |
| bhagyashris | frenzyfriday, ack thanks :) | 10:17 |
| *** chem is now known as Guest1260 | 10:21 | |
| marios|ruck | amoralej: ack | 10:21 |
| *** ysandeep|coffee is now known as ysandeep | 10:51 | |
| ysandeep | amoralej: ++ thanks for the patch | 10:51 |
| dviroel | o/ | 11:14 |
| *** rlandy is now known as rlandy|ruck | 11:14 | |
| * dviroel will rebase https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/38694 - 1m | 11:19 | |
| rlandy|ruck | slaweq: hi - can you review harold's comments on https://bugs.launchpad.net/tripleo/+bug/1958955/comments/11 and see if you agree. We will try a revert on testproject | 11:21 |
| rlandy|ruck | gthiemonge: hello ... per https://bugs.launchpad.net/tripleo/+bug/1958531 - last comment - your patches are working - thank you - juts waiting on merge there | 11:24 |
| gthiemonge | rlandy|ruck: amazing! thanks for testing them! | 11:26 |
| rlandy|ruck | Tengu: hey ... looking at https://bugs.launchpad.net/tripleo/+bug/1958230 - does proposed patch work for you? | 11:26 |
| rlandy|ruck | akahat: hey is https://review.opendev.org/c/openstack/tripleo-ci/+/823688 ready for review? on review list but DNM | 11:36 |
| marios|ruck | ysandeep: https://review.rdoproject.org/r/c/rdo-jobs/+/38602 & https://review.opendev.org/c/openstack/tripleo-ci/+/826317 | 11:37 |
| marios|ruck | rlandy|ruck: ^^ | 11:39 |
| marios|ruck | rlandy|ruck: ysandeep: & https://review.rdoproject.org/r/c/config/+/38601 | 11:41 |
| akahat | rlandy|ruck, no not yet. | 11:41 |
| bhagyashris | rlandy|ruck, ysandeep kindly add in your review list https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/38667 | 11:42 |
| bhagyashris | added in the review list hackmd | 11:42 |
| ysandeep | bhagyashris, merged | 11:43 |
| bhagyashris | ysandeep, thanks | 11:43 |
| bhagyashris | ananya is looking into that | 11:43 |
| ysandeep | ack, got that from ananya's comment | 11:44 |
| dviroel | rlandy|ruck: hi, fyi, i was pinning c9 ovb to 01/19 compose - got same results - https://logserver.rdoproject.org/74/35174/42/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-compose-master-pinning/2fad731/logs/ | 11:55 |
| Tengu | rlandy|ruck: nope - yet another issue apparently. | 11:56 |
| Tengu | missing file or something, I've added a comment to the gchat + patch | 11:56 |
| Tengu | rlandy|ruck: though - if the overcloud-full.qcow2 is being deprecated, it might be good to know about the proposed timeline for the actual deprecation - and I can switch to the overcloud-hardened-uefi-full instead. | 11:57 |
| Tengu | rlandy|ruck: though I'm hitting some issues while trying to build that hardened-uefi-full image :). cf https://bugs.launchpad.net/tripleo/+bug/1959568 + https://bugs.launchpad.net/tripleo/+bug/1959093 | 11:58 |
| Tengu | would be too easy. | 11:58 |
| Tengu | I'll modify tripleo-lab to allow the hardened-uefi one, apparently it's not delivered as a targz, and there's probably a need for some option such as wholedisk or something. | 11:58 |
| rlandy|ruck | dviroel: thanks | 12:08 |
| slaweq | rlandy|ruck: yes, I already pushed some DNM patch to neutron to add some logs there and to try to understand what is going on there | 12:08 |
| rlandy|ruck | slaweq: awesome - thanks | 12:09 |
| rlandy|ruck | https://trello.com/c/hzuFEfxF/2320-cixlp1958955tripleociproa-cs9-master-fs001-fs002-fs020-fs035-fs039-and-fs064-is-failing-to-provision-overcloud-node-the-last-err | 12:10 |
| rlandy|ruck | https://bugs.launchpad.net/tripleo/+bug/1958955 | 12:10 |
| rlandy|ruck | https://opendev.org/openstack/neutron/commit/1562f9141ba9c90c69c1c851dd9a50c8108ab15f | 12:10 |
| rlandy|ruck | ysandeep: hey pls join review time again | 12:17 |
| amoralej | marios|ruck, wrt opstools change, did you send any patch? | 12:18 |
| ysandeep | rlandy|ruck, ack, joining back in 1 min | 12:18 |
| marios|ruck | amoralej: not yet (for quickstart you mean) sorry just got off a call and chasing sthing else will do in few mins | 12:22 |
| amoralej | no problem, i'm fixing in rdo, in some minutes i can also help on that | 12:23 |
| marios|ruck | amoralej: thx | 12:23 |
| amoralej | btw, isn't this breaking all the cs8 jobs? | 12:23 |
| amoralej | what's the impact in tripleo-ci? | 12:23 |
| marios|ruck | amoralej: haven't seen it in gates... promotions are failing on sthing else so possibly we don't see it at that point? | 12:23 |
| marios|ruck | amoralej: we have 2 centos8 appstream related/deps(?) issues | 12:24 |
| amoralej | marios|ruck, the other one is the gcc one? | 12:24 |
| amoralej | where i can see logs? | 12:24 |
| marios|ruck | amoralej: yeah gcc and pacemaker-cluster-libs logs on bugs @ https://bugs.launchpad.net/tripleo/+bug/1959524 https://bugs.launchpad.net/tripleo/+bug/1959526 | 12:24 |
| amoralej | mmm pacemaker-cluster-libs? | 12:25 |
| amoralej | lemme check | 12:25 |
| marios|ruck | slaweq: can you point me to your dnm patch i don't see it quickly https://review.opendev.org/q/project:openstack/neutron | 12:25 |
| marios|ruck | slaweq: for https://bugs.launchpad.net/tripleo/+bug/1958955/comments/11 | 12:26 |
| slaweq | marios|ruck: https://review.opendev.org/c/openstack/neutron/+/826999 | 12:26 |
| marios|ruck | slaweq: thank you .. ok so it isn't revert you are debugging for now... i may post the revert so i can testproject and see if it is OK? | 12:27 |
| rlandy|ruck | dviroel: sshnaidm: hey - can you guys vote on https://review.opendev.org/c/openstack/tripleo-ci/+/826521 - to spring us on downstream? it fails tox now due to opstools - will recheck later | 12:29 |
| marios|ruck | slaweq: fyi https://review.opendev.org/c/openstack/neutron/+/827014 & https://review.rdoproject.org/r/c/testproject/+/38830 | 12:34 |
| amoralej | marios|ruck, i commented in https://bugs.launchpad.net/tripleo/+bug/1959526/ it should work now | 12:40 |
| amoralej | i proposed a fix | 12:40 |
| amoralej | i mean, didn't send a patch, just in the comment | 12:41 |
| arxcruz | marios|ruck https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/815654 | 12:41 |
| arxcruz | can you review it? | 12:41 |
| arxcruz | you add some comments, but I think we are good to go | 12:41 |
| rlandy|ruck | marios|ruck: so reading back on amoralej's comments on https://bugs.launchpad.net/tripleo/+bug/1959526/ - it looks like we do not disable the centos repos laid down in image builds | 12:42 |
| arxcruz | marios|ruck nevermind | 12:42 |
| amoralej | rlandy|ruck, actually, it seems baseos and appstream are disabled | 12:44 |
| amoralej | but not others as highavailability | 12:44 |
| rlandy|ruck | amoralej: ack - looking into why that is | 12:44 |
| amoralej | rlandy|ruck, https://logserver.rdoproject.org/openstack-periodic-integration-main-centos8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-full-master/9031fc4/yum.repos.d/ | 12:44 |
| slaweq | marios|ruck: 2 things - first, I don't want to revert that patch as it fixes some other CI issue which we had few times | 12:45 |
| slaweq | and 2nd - are You sure it will work with such depends-on? I proposed https://review.rdoproject.org/r/c/testproject/+/38582 which depends on https://review.opendev.org/c/openstack/neutron/+/826999 and I don't see my extra logs in the dhcp agent's logs in the undercloud (that's where the issue is) | 12:46 |
| rlandy|ruck | amoralej: are we looking at the same problem on https://bugs.launchpad.net/tripleo/+bug/1959524? | 12:46 |
| * rlandy|ruck checks those repos | 12:46 | |
| amoralej | lemme check | 12:46 |
| amoralej | rlandy|ruck, that's probably different | 12:47 |
| amoralej | lemme see | 12:47 |
| rlandy|ruck | https://logserver.rdoproject.org/openstack-periodic-integration-main-centos8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-master/204a336/logs/undercloud/etc/yum.repos.d/ | 12:47 |
| rlandy|ruck | ack | 12:47 |
| rlandy|ruck | those are all disabled properly | 12:47 |
| * rlandy|ruck looks into fixing the first one | 12:47 | |
| marios|ruck | slaweq: ack .. more so we can say if it is the root or not... cheap to find out if it works (running now https://review.rdoproject.org/zuul/stream/9478b678c7e646f0a6aba3da67cb9175?logfile=console.log i think it should work lets see it has ZUUL_CHANGE_IDS=827014,1 38830,1 | 12:49 |
| marios|ruck | slaweq: i have DNM and workflow-1 on the neutron patch | 12:49 |
| marios|ruck | slaweq: let me check logs from your job sec | 12:50 |
| slaweq | marios|ruck: I think that depends-on works correctly, but maybe the culprit is different | 12:50 |
| slaweq | I have node on hold now and I'm investigating it | 12:50 |
| marios|ruck | slaweq: should be there (the neutron change i mean) looking at https://logserver.rdoproject.org/82/38582/3/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master/cd78e7f/job-output.txt 2022-01-31 04:11:15.574423 | primary | included: /home/zuul/workspace/.quickstart/usr/local/share/ansible/roles/build-test-packages/tasks/dlrn-build.yml for undercloud => (item={'host': | 12:51 |
| marios|ruck | 'https://review.opendev.org/c/openstack/neutron/+', 'project': 'openstack/neutron', 'branch': 'master', 'refspec': 'refs/changes/99/826999/1'}) | 12:51 |
| marios|ruck | slaweq: ack thanks | 12:51 |
| amoralej | rlandy|ruck, that's strange, i see AFS synchronization for gcc and gcc-c++ happened fine, in the same sync and repos are configured to use AFS mirrors | 12:54 |
| amoralej | it's just that occurrence? | 12:54 |
| amoralej | actually gcc-8.5.0-8 was synced on centos.log.1:2022-01-28T16:43 | 12:55 |
| amoralej | but the log is from 30th | 12:55 |
| rlandy|ruck | amoralej: nah - been going on since friday | 12:55 |
| rlandy|ruck | only that rpm | 12:55 |
| rlandy|ruck | but it's been happening for a while | 12:56 |
| amoralej | it's strange i see in that log that it pulled http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/centos/8-stream/BaseOS/x86_64/os/Packages/libgfortran-8.5.0-7.el8.x86_64.rpm | 13:01 |
| amoralej | but by then, 8.5.0-8 was available in the repo | 13:01 |
| rlandy|ruck | yep - it is | 13:02 |
| amoralej | mmm, has failed 29, 30 and 31 | 13:03 |
| amoralej | rlandy|ruck, only for victoria? | 13:03 |
| Tengu | rlandy|ruck: quick feedback: using the hardened-uefi-full image available on images.rdoproject, I get a successful node provisioning. So, if the overcloud-full.qcow2 is to be deprecated, I can live with that. | 13:06 |
| Tengu | rlandy|ruck: tripleo-lab is able to eat both now. | 13:06 |
| marios|ruck | amoralej: cant we use http://mirror.centos.org/centos/8-stream/opstools/x86_64/collectd-5/ instead of https://trunk.rdoproject.org/centos8-master/deps/collectd or it doesn't have collectd yet in mirror. ? | 13:09 |
| amoralej | marios|ruck, collectd is still not in that repo | 13:10 |
| amoralej | see http://mirror.centos.org/centos/8-stream/opstools/x86_64/collectd-5/Packages/ | 13:10 |
| amoralej | that's the actual problem | 13:10 |
| marios|ruck | amoralej: yes sorry just checked ack | 13:10 |
| marios|ruck | amoralej: k posting patch in a few | 13:10 |
| *** amoralej is now known as amoralej|lunch | 13:10 | |
| marios|ruck | amoralej|lunch: but will you create the same for wallaby... train or just master ? | 13:14 |
| marios|ruck | amoralej|lunch: i mean create teh https://trunk.rdoproject.org/centos8-master/deps/ | 13:14 |
| *** beagles is now known as eagles | 13:16 | |
| *** dasm|off is now known as dasm | 13:19 | |
| dasm | o/ | 13:19 |
| *** dasm is now known as dasm|rover | 13:19 | |
| marios|ruck | amoralej|lunch: no ... stream will be the same regardless of openstack branch i think so you'll probably just use one repo as the temp | 13:22 |
| rlandy|ruck | bhagyashris: hello ... https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/38814/1/ci-scripts/dlrnapi_promoter/dlrn-promoter.sh | 13:24 |
| rlandy|ruck | bhagyashris: merged https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/38814 | 13:25 |
| rlandy|ruck | let me know when we can out it back | 13:26 |
| marios|ruck | rlandy|ruck: amoralej|lunch: i think we aren't seeing the issue with collectd because we aren't enabling that repo anywhere that we are running (the featuresets here aren't used I think) | 13:26 |
| marios|ruck | amoralej|lunch: its why we dind't see it in ci | 13:26 |
| dviroel | rlandy|ruck: hi, https://bugs.launchpad.net/tripleo/+bug/1959582 - for centos9-scenario007-standalone-wallaby - didn't find any related LP | 13:26 |
| marios|ruck | amoralej|lunch: rlandy|ruck: i think we would have seen it in th eintegration line but that is broken earlier so isn't reaching that point | 13:27 |
| marios|ruck | amoralej|lunch: anyway posting the patch and we can discuss there | 13:27 |
| rlandy|ruck | dviroel: there probably wasn't one - thanks | 13:27 |
| marios|ruck | amoralej|lunch: going to point to https://trunk.rdoproject.org/centos8-master/deps/collectd/ for all the branches with TODO | 13:27 |
| Tengu | dang..... some more selinux issues. | 13:30 |
| bhagyashris | rlandy|ruck, ack | 13:37 |
| rlandy|ruck | amoralej|lunch: not just victoria | 13:39 |
| rlandy|ruck | that was just the first place we saw it | 13:40 |
| marios|ruck | amoralej|lunch: what is [1] please i want to reference it too https://review.rdoproject.org/r/c/rdo-infra/ansible-role-dlrn/+/38673/6#message-a01d8c3a9b8dfa9e1ff66e34f99952fb22440fb2 | 13:43 |
| frenzyfriday | Hey marios|ruck 0/ have you seen this error: "Depsolve Error occured: \n Problem: cannot install both gcc-8.5.0-7.el8.x86_64 and gcc-8.5.0-8.el8.x86_64\n - package gcc-c++-8.5.0-7.el8.x86_64 requires gcc = 8.5.0-7.el8, but none of the providers can be installed\n - cannot install the best update candidate for package gcc-8.5.0-8.el8.x86_64\n - cannot install the best candidate for the job" ? | 13:44 |
| frenzyfriday | scenario010-kvm-internal-standalone wallaby and ussuri are hitting this. bhagyashris : Maybe thats what was blocking the promotions? | 13:45 |
| bhagyashris | frenzyfriday, fyi https://trello.com/c/AFB8lJuR/2331-cixlp1959524tripleociproa-ostempest-pythonvenvbuild-failing-with-depsolve-error-occured-n-problem-cannot-install-both-gcc-850-7e | 13:45 |
| marios|ruck | frenzyfriday: yeah | 13:46 |
| marios|ruck | frenzyfriday: it blocks all the c8 lines | 13:46 |
| marios|ruck | frenzyfriday: on cix call | 13:46 |
| frenzyfriday | ack, thanks | 13:47 |
| slaweq | marios|ruck: rlandy|ruck I think that the issue isn't caused by my patch https://opendev.org/openstack/neutron/commit/1562f9141ba9c90c69c1c851dd9a50c8108ab15f but I more or less know what is going on ther | 13:54 |
| slaweq | I will send small patch in few minutes so we can test it in test project | 13:54 |
| slaweq | I hope it will be better with that fix | 13:54 |
| *** amoralej|lunch is now known as amoralej | 13:54 | |
| amoralej | amoralej, that make sense, but i think collectd is used to build containers, didn't we hit the issue in the jobs building containers? | 13:55 |
| slaweq | marios|ruck: rlandy|ruck FYI: https://review.opendev.org/c/openstack/neutron/+/827101 | 13:59 |
| slaweq | and I just updated https://review.rdoproject.org/r/c/testproject/+/38582 to test it | 13:59 |
| slaweq | lets see how it will work now | 13:59 |
| marios|ruck | slaweq: thanks in cix call now will check in a bit | 13:59 |
| slaweq | ykarel_: hi, You can clean the node which You hold for me earlier (IP 38.102.83.85) | 14:00 |
| slaweq | thx a lot, it was very helpful | 14:00 |
| bhagyashris | rlandy|ruck, ysandeep marios|ruck dam dviroel rcastillo | 14:01 |
| bhagyashris | scrum time | 14:01 |
| rlandy|ruck | we;re on cix | 14:01 |
| rlandy|ruck | will join in a few | 14:01 |
| bhagyashris | ack | 14:02 |
| dasm|rover | in srbac. will join in a bit | 14:02 |
| dviroel | joinning | 14:02 |
| ysandeep | sshnaidm: I would like to join today's meeting as well with my UA hat on so that I can understood how much work is there, please invite me as well. | 14:10 |
| amoralej | there may be some way to hold a node for https://bugs.launchpad.net/tripleo/+bug/1959524 ? | 14:36 |
| marios|ruck | amoralej|lunch: what is [1] please i want to reference it too https://review.rdoproject.org/r/c/rdo-infra/ansible-role-dlrn/+/38673/6#message-a01d8c3a9b8dfa9e1ff66e34f99952fb22440fb2 | 14:37 |
| marios|ruck | amoralej: ah thanks you upated | 14:37 |
| marios|ruck | (sorry still in call :D) | 14:37 |
| marios|ruck | [1] https://www.centos.org/centos-linux-eol/ | 14:37 |
| amoralej | np | 14:37 |
| amoralej | i can't reproduce it by myself | 14:37 |
| sshnaidm | ** I created an event for talk today with SDK folks, 16.30 UTC: https://meet.google.com/teo-fhxt-irr - if I didn't add someone and you would like to join - ping me, or just join :) | 14:38 |
| ysandeep | sshnaidm: thanks! | 14:41 |
| * bhagyashris dinner brb | 14:44 | |
| marios|ruck | rlandy|ruck: https://opendev.org/openstack/tripleo-repos/src/branch/master/plugins/module_utils/tripleo_repos/main.py | 14:47 |
| rlandy|ruck | marios|ruck: https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-full-train/8128424/yum.repos.d/ | 14:50 |
| rlandy|ruck | marios|ruck" dasm|rover: https://review.opendev.org/c/openstack/tripleo-ci/+/823688 | 14:53 |
| marios|ruck | dasm|rover: https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/release/tripleo-ci/CentOS-8 | 14:53 |
| marios|ruck | dasm|rover: https://opendev.org/openstack/tripleo-repos/src/branch/master/plugins/module_utils/tripleo_repos/main.py#L41 | 14:55 |
| * frenzyfriday will be back in 2 hrs | 14:57 | |
| rlandy|ruck | dasm|rover: https://review.rdoproject.org/r/c/testproject/+/38065 | 14:57 |
| rlandy|ruck | dasm|rover: https://bugs.launchpad.net/tripleo/+bug/1959524 | 15:00 |
| rlandy|ruck | marios|ruck: https://review.opendev.org/c/openstack/tripleo-quickstart/+/827111 ready to merge? | 15:13 |
| marios|ruck | rlandy|ruck: yeah i think so | 15:19 |
| marios|ruck | rlandy|ruck: i don't think it affects us in ci | 15:19 |
| marios|ruck | rlandy|ruck: because https://opendev.org/openstack/tripleo-quickstart/src/commit/358d8da7877a5cb408bbf79c80ddd42edf9ef553/config/release/tripleo-ci/CentOS-8/master.yml#L202 | 15:19 |
| marios|ruck | rlandy|ruck: and https://codesearch.opendev.org/?q=enable_opstools&i=nope&literal=nope&files=&excludeFiles=&repos= (none of the featuresets are used i think) | 15:19 |
| marios|ruck | rlandy|ruck: but lets merge it anyway since the 8 one is gone and until we get the 8-stream opstools | 15:20 |
| rlandy|ruck | sshnaidm: can you add me to the sdk meeting | 15:21 |
| sshnaidm | rlandy|ruck, done | 15:22 |
| rlandy|ruck | marios|ruck: assuming https://review.opendev.org/c/openstack/tripleo-quickstart/+/827111 won;t fix the tox issues we're seeing | 15:24 |
| marios|ruck | rlandy|ruck: not that am aware of :) | 15:25 |
| marios|ruck | rlandy|ruck: link to tox issue? | 15:25 |
| rlandy|ruck | marios|ruck: also https://review.opendev.org/c/openstack/tripleo-ci/+/823688/ is not passing check | 15:26 |
| rlandy|ruck | dasm|rover: ^^ | 15:26 |
| rlandy|ruck | we need to fix tripleo-repos | 15:26 |
| dasm|rover | rlandy|ruck: do you mean exceeded "retry_limit" on centos-8? | 15:33 |
| dasm|rover | > ERROR: This script does not work on Python 3.6 The minimum supported Python version is 3.7. Please use https://bootstrap.pypa.io/pip/3.6/get-pip.py instead. | 15:36 |
| dasm|rover | https://zuul.opendev.org/t/openstack/build/83bd7f6201114b7fbd3274cd7faa152f | 15:36 |
| dasm|rover | pip released today 22.0 stopped supporting python 3.6 | 15:37 |
| dasm|rover | > Requires: Python >=3.7 | 15:37 |
| dasm|rover | https://pypi.org/project/pip/ | 15:37 |
| beagles | was just about to ask about a missing opstools repo for CentOS.. but I thnk this is being discussed already? | 15:38 |
| marios|ruck | beagles: yeah are you seeing it in ci now? | 15:39 |
| *** dviroel is now known as dviroel|lunch | 15:39 | |
| beagles | marios|ruck: no I'm seeing problems with image configi using local quickstart deployment | 15:39 |
| marios|ruck | beagles: k this might help you https://review.opendev.org/c/openstack/tripleo-quickstart/+/827111 | 15:39 |
| rlandy|ruck | ysandeep: https://review.opendev.org/c/openstack/tripleo-ci/+/826521/8..9/roles/oooci-build-images/add_cert.yaml | 15:40 |
| marios|ruck | beagles: or edit the relevant file in your env | 15:40 |
| rlandy|ruck | d /etc/pki/ca-trust/source/anchors/rh.crt:/etc/pki/ca-trust/source/anchors/rh.crt | 15:40 |
| beagles | marios|ruck: perfect! thanks! | 15:40 |
| marios|ruck | beagles: ack np | 15:40 |
| ysandeep | rlandy|ruck, thanks for highlight that | 15:40 |
| rlandy|ruck | would aonly make sense for path 1 | 15:41 |
| ysandeep | rlandy|ruck, I wanted to move to --upload {{ cert_path }}:{{ cert_path }} but missed that | 15:41 |
| *** jpodivin_ is now known as jpodivin | 15:43 | |
| rlandy|ruck | ysandeep: let's meet in 15 | 15:44 |
| ysandeep | rlandy|ruck, ack | 15:44 |
| ysandeep | rlandy|ruck, trying https://review.opendev.org/c/openstack/tripleo-ci/+/826521/10/roles/oooci-build-images/add_cert.yaml#48 | 15:45 |
| rlandy|ruck | dasm|rover: marios|ruck: ugh - may need to fix tripleo-repos | 15:45 |
| rlandy|ruck | quickest path | 15:45 |
| dasm|rover | rlandy|ruck: you mean "add pip to install"? | 15:45 |
| dasm|rover | it seems that we're checking for the pip presence, which tries to download it, when it's missing. | 15:46 |
| dasm|rover | and it's pulling recent release. | 15:46 |
| *** ysandeep is now known as ysandeep|dinner | 15:46 | |
| * ysandeep|dinner back in ~15 mins | 15:47 | |
| marios|ruck | rlandy|ruck: yeah i see it... not sure if it is the quickest though maybe we can fix the issue with the akahat patch | 15:48 |
| marios|ruck | rlandy|ruck: i think we have another/new issue fs39 train digging a bit | 15:48 |
| rlandy|ruck | marios|ruck: /o\ too man issues | 16:00 |
| rlandy|ruck | many | 16:00 |
| rlandy|ruck | dasm: ack - if you can fix that review | 16:04 |
| rlandy|ruck | ysandeep|dinner: when you are back ... https://meet.google.com/igg-cjeb-fhi?pli=1&authuser=0 | 16:04 |
| marios|ruck | rlandy|ruck: luckily this one is nit/easy fix dealing with now | 16:06 |
| *** ysandeep|dinner is now known as ysandeep | 16:06 | |
| ysandeep | rlandy|ruck, joining | 16:06 |
| *** amoralej is now known as amoralej|off | 16:30 | |
| marios|ruck | rlandy|ruck: dasm|rover: going in a few mins folks need sthing before i do? otherwise same again tomorrow ;) | 16:31 |
| marios|ruck | wew monday | 16:31 |
| rlandy|ruck | marios|ruck: just want we can pick up | 16:32 |
| marios|ruck | rlandy|ruck: should be on the reviews list | 16:32 |
| marios|ruck | lemme check | 16:32 |
| rlandy|ruck | https://bugs.launchpad.net/tripleo/+bug/1959524 - says still investigating | 16:32 |
| rlandy|ruck | so pacemaker-libs fix | 16:33 |
| marios|ruck | rlandy|ruck: dont have any updates on those cix | 16:33 |
| marios|ruck | rlandy|ruck: only https://review.opendev.org/c/openstack/tripleo-quickstart/+/827143 for train | 16:33 |
| marios|ruck | rlandy|ruck: and https://review.opendev.org/c/openstack/tripleo-quickstart/+/827111 but not urgent/should not be seeing that in ci | 16:34 |
| rlandy|ruck | https://review.opendev.org/c/openstack/tripleo-quickstart/+/827143 - thanks for that | 16:36 |
| marios|ruck | rlandy|ruck: will pickup in the morning - try get some progress on the cix :/ | 16:37 |
| rlandy|ruck | marios|ruck: sure dasm|rover and I will leave you nots | 16:38 |
| rlandy|ruck | notes | 16:38 |
| *** dviroel|lunch is now known as dviroel | 16:38 | |
| rlandy|ruck | marios|ruck: no worries - we are all broken on gcc | 16:40 |
| rlandy|ruck | https://bugs.launchpad.net/tripleo/+bug/1959524 | 16:40 |
| rlandy|ruck | amoralej|off is gone already | 16:40 |
| rlandy|ruck | so we'll see tomorrow | 16:40 |
| marios|ruck | rlandy|ruck: good news from slaweq at least ;) | 16:41 |
| dviroel | yeah? | 16:41 |
| marios|ruck | dviroel: for the ovb provision issue see #tripleo just now | 16:41 |
| * marios|ruck out | 16:41 | |
| *** marios|ruck is now known as marios|out | 16:42 | |
| dviroel | will take a look o/ | 16:42 |
| rlandy|ruck | dviroel: https://logserver.rdoproject.org/82/38582/4/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master/ae3536a/ | 16:43 |
| dasm|rover | hmm... marios is out :/ | 16:45 |
| rlandy|ruck | dasm|rover: what do you need? | 16:46 |
| dviroel | interesting, small fix on neutron | 16:46 |
| rlandy|ruck | dasm|rover: still testing amol's patch? | 16:46 |
| dviroel | we should tp against more ovb jobs, just to make sure | 16:46 |
| dasm|rover | rlandy|ruck: i'm not testing amol's patch. the issue is we're pulling latest pip from upstream | 16:47 |
| dasm|rover | marios introduced that change here: https://review.opendev.org/c/openstack/tripleo-ci/+/818222 | 16:47 |
| dasm|rover | and i wanted to double check with him if that's actually what he intended to do | 16:47 |
| dasm|rover | here: https://review.opendev.org/c/openstack/tripleo-ci/+/818222/49/roles/tripleo-build-jobs-repos/tasks/main.yaml#38 | 16:47 |
| dasm|rover | rlandy|ruck: maybe you remember what was the intend here? why he was uninstalling pip from repo and installing from the upstream? | 16:48 |
| rlandy|ruck | dasm|rover: not really | 16:48 |
| dasm|rover | ack | 16:48 |
| dasm|rover | lemme try this change and i'm gonna rerun jobs to see if that's what we need to do | 16:48 |
| rlandy|ruck | dasm|rover: amoralej|off said mirrors should be fixed atm | 16:49 |
| dasm|rover | ack | 16:49 |
| rlandy|ruck | so yeah - maybe line rekick | 16:49 |
| rlandy|ruck | bhagyashris: any new on c7? | 16:49 |
| rlandy|ruck | dasm|rover: so you can rekick openstack-periodic-integration-stable4 | 16:55 |
| rlandy|ruck | and openstack-periodic-integration-stable2 | 16:55 |
| rlandy|ruck | and see if we spring free | 16:55 |
| rlandy|ruck | we need to fix the issue | 16:55 |
| rlandy|ruck | anyways | 16:55 |
| rlandy|ruck | but we may be able to promote | 16:56 |
| dasm|rover | rlandy|ruck: i can do that, but i'm not sure if the issue is going to be fixed. | 16:58 |
| dasm|rover | rlandy|ruck: we're pulling upstream pip, which dropped support for python 3.6 | 16:58 |
| dasm|rover | however centos8 has python 3.6, hence we're unable to install it. | 16:59 |
| dasm|rover | i submitted change https://review.opendev.org/c/openstack/tripleo-ci/+/827157 and i'm going to run it with my fix. | 16:59 |
| rlandy|ruck | ok - dropping meeting - will chat with you | 16:59 |
| dasm|rover | it should fix the issue with c8, but i don't know why marios needed that in first place. in the commit message, he mentioned it as needed for c9 | 17:00 |
| rlandy|ruck | you can switch case | 17:00 |
| dasm|rover | switch case? what do you mean? | 17:00 |
| dasm|rover | hmm.. i see there is another issue, which you just mentioned, about broken mirrors. | 17:01 |
| dasm|rover | https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-containers-multinode-train/4844be3/job-output.txt | 17:01 |
| dasm|rover | brb | 17:02 |
| dasm|rover | back | 17:05 |
| rlandy|ruck | dasm|rover: ok - https://meet.google.com/rrx-nbyd-gfb?pli=1&authuser=0 | 17:05 |
| *** ysandeep is now known as ysandeep|out | 17:30 | |
| *** jpena is now known as jpena|off | 17:32 | |
| ysandeep|out | rlandy|ruck, /me out, please keep an eye on https://review.opendev.org/c/openstack/tripleo-ci/+/826521 thanks! | 17:36 |
| * frenzyfriday afk | 17:41 | |
| rlandy|ruck | 823688,15 | 17:42 |
| dasm|rover | rlandy|ruck: fyi, promotion is failing due to "primary | /bin/sh: /home/zuul/workspace/hash_info.sh: No such file or directory" | 17:50 |
| dasm|rover | i'm gonna look into that as a next thing | 17:50 |
| rlandy|ruck | dasm|rover: taking few minutes for lunch - brb | 17:52 |
| dasm|rover | k, no worries | 17:54 |
| rlandy|ruck | dasm|rover: back - looking into line failures | 18:13 |
| dasm|rover | k | 18:13 |
| rlandy|ruck | dasm|rover: thanks for creating https://review.rdoproject.org/r/c/testproject/+/38844/1/.zuul.yaml | 18:14 |
| dasm|rover | :) i'm looking at its progress now | 18:14 |
| rlandy|ruck | rcastillo: hey ... how about some more tox molecule fun? | 18:18 |
| rlandy|ruck | rcastillo: want to debug this one for us? https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-molecule&skip=0 | 18:18 |
| rcastillo | oh boy | 18:19 |
| rcastillo | looking | 18:19 |
| rlandy|ruck | rcastillo: tripleo-ci as well | 18:20 |
| rlandy|ruck | repo layout there | 18:21 |
| rlandy|ruck | rcastillo: needs to be running on a stream node | 18:27 |
| rcastillo | yeah I saw | 18:28 |
| rcastillo | need to mount the vars I believe | 18:28 |
| rlandy|ruck | 2022-01-31 17:33:22.448391 | localhost | Distro: Ubuntu 20.04 | 18:28 |
| rlandy|ruck | rcastillo: it's defined in a templates | 18:28 |
| rlandy|ruck | rcastillo": https://meet.google.com/exu-fomx-knp?pli=1&authuser=0 | 18:29 |
| dviroel | rlandy|ruck: pls, take a look at this LP later https://bugs.launchpad.net/tripleo/+bug/1956475 - hitting wallaby too, no CIX card openned I think | 18:43 |
| dviroel | rlandy|ruck: tests being skipped: https://opendev.org/openstack/openstack-tempest-skiplist/commit/85a70a8691010b82f1a509a313f4aa2bc829bb7d | 18:43 |
| dasm|rover | rlandy|ruck: fyi, https://bugs.launchpad.net/tripleo/+bug/1959611 | 18:51 |
| rlandy|ruck | dasm|rover: k - need to get on meeting now ... | 18:57 |
| rlandy|ruck | spoke to rcastillo about molecule failures | 18:57 |
| rlandy|ruck | need ti fix linter failure on marios patch | 18:57 |
| rlandy|ruck | dasm|rover: https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-hardened-full-train/abccb60/job-output.txt - need marios patch to fix | 18:59 |
| dasm|rover | i think the issue i just reported is a red herring. i see it being marked as "ignore_errors: true" | 18:59 |
| rlandy|ruck | dasm|rover: it;s ok just close it | 18:59 |
| dasm|rover | k | 18:59 |
| rlandy|ruck | periodic-tripleo-centos-8-buildimage-overcloud-hardened-full-train need to testproject that with https://review.opendev.org/c/openstack/tripleo-quickstart/+/827111/ | 19:00 |
| rlandy|ruck | and fix linter ussue there ^^ | 19:00 |
| rlandy|ruck | dasm|rover: ^^ you can give that a shot | 19:00 |
| rlandy|ruck | ussuri white space | 19:00 |
| dasm|rover | k | 19:02 |
| dasm|rover | fixed linter issue, started test job with ^ | 19:11 |
| dasm|rover | brb | 19:33 |
| rlandy|ruck | dasm|rover: thanks | 19:43 |
| dasm|rover | back | 19:47 |
| rlandy|ruck | ok - out of meeting | 20:18 |
| rlandy|ruck | rcastillo: how are things looking? | 20:18 |
| rlandy|ruck | dasm|rover: here's the next issue: https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-content-provider&skip=0 | 20:19 |
| dasm|rover | mhm | 20:20 |
| dviroel | DEBUG util.py:596: Errors during downloading metadata for repository 'centos-opstools': | 20:21 |
| dviroel | DEBUG util.py:596: - Status code: 404 for http://mirror.centos.org/centos/8/opstools/x86_64/collectd-5/repodata/repomd.xml (IP: 85.236.43.108) | 20:21 |
| rlandy|ruck | - Status code: 404 for http://mirror.centos.org/centos/8/opstools/x86_64/collectd-5/repodata/repomd.xml (IP: 85.236.43.108) | 20:21 |
| rlandy|ruck | ugh | 20:21 |
| rlandy|ruck | we need a bug for the opstools issue | 20:21 |
| rlandy|ruck | see https://review.opendev.org/c/openstack/tripleo-quickstart/+/827111/ | 20:22 |
| * rlandy|ruck logs | 20:23 | |
| rlandy|ruck | dasm|rover: dviroel: rcastillo: at least we have a bug for it now: https://bugs.launchpad.net/tripleo/+bug/1959619 | 20:32 |
| dasm|rover | ack | 20:32 |
| dasm|rover | rlandy|ruck: this one https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-content-provider&skip=0 is also affected by opstools | 20:35 |
| dasm|rover | https://2af8965ad28e49781e91-ba7281a549a418e9d574e043a1b402b5.ssl.cf5.rackcdn.com/827088/2/gate/tripleo-ci-centos-8-content-provider/b05d748/logs/undercloud/home/zuul/DLRN/data/repos/component/tripleo/ec/33/ec33ab40488efacaf98e31852b280947ee73bff8_dev/rpmbuild.log | 20:35 |
| dasm|rover | this particular change: https://review.opendev.org/827088 | 20:35 |
| dasm|rover | checking others | 20:35 |
| rlandy|ruck | dasm|rover: yep - pinging also rhos-ops to get https://review.rdoproject.org/r/c/rdo-infra/ansible-role-dlrn/+/38673 merged | 20:35 |
| dasm|rover | my hunch says -- they're the same | 20:35 |
| rcastillo | molecule job on quickstart failing with a 404 on an image now https://d5816c7856271a4bd827-9262fa59dca60616978c458f4dd8b484.ssl.cf2.rackcdn.com/827166/4/check/tripleo-tox-molecule/e790e3a/ | 20:36 |
| rcastillo | so https://review.opendev.org/c/openstack/tripleo-quickstart/+/827166 clears the repo issue we were getting at least | 20:37 |
| rcastillo | rlandy|ruck: ^^ | 20:38 |
| rlandy|ruck | rcastillo: posting out about gate blocket | 20:39 |
| rlandy|ruck | blocker | 20:39 |
| * rcastillo lunch, brb | 20:40 | |
| rlandy|ruck | oh gee | 20:41 |
| dasm|rover | ? | 20:41 |
| rlandy|ruck | rcastillo: k- let me see if I can fix that | 20:42 |
| rlandy|ruck | maybe we can get this in | 20:42 |
| *** dviroel is now known as dviroel|brb | 20:48 | |
| * rcastillo back | 21:01 | |
| rcastillo | so c8 is actually gone for realsies now huh? | 21:01 |
| rlandy|ruck | rcastillo: no it exists | 21:04 |
| rlandy|ruck | this is what happened | 21:04 |
| rlandy|ruck | there was a space issue on the images server | 21:04 |
| rlandy|ruck | so https://images.rdoproject.org/centos8/wallaby/rdo_trunk/current-tripleo/ promoted empty | 21:05 |
| rlandy|ruck | fs039 is the only test outstanding to promote wallaby | 21:05 |
| rlandy|ruck | so I am rerunning | 21:05 |
| rlandy|ruck | rcastillo: maybe make that test non-voting | 21:06 |
| rlandy|ruck | just to get marios patch in to spring everything else??? | 21:06 |
| rlandy|ruck | dasm|rover: rcastillo: this is literally the craziet ruck/rover day ever | 21:07 |
| dasm|rover | #TODO: [x] - attend craziest day ever | 21:08 |
| dasm|rover | checked! | 21:08 |
| rlandy|ruck | dasm|rover: so sorry | 21:08 |
| rcastillo | geez | 21:08 |
| dasm|rover | no worries, rlandy|ruck! you didn't plan it | 21:09 |
| dasm|rover | (at least i hope so ;P) | 21:09 |
| rlandy|ruck | rcastillo: dasm|rover: let's meet and try get the order of merge here | 21:09 |
| dasm|rover | k | 21:09 |
| rcastillo | sure | 21:10 |
| rlandy|ruck | https://meet.google.com/rcf-veof-car?pli=1&authuser=0 | 21:11 |
| dasm|rover | > Failed to download metadata for repo 'centos-opstools'", "2022-01-31 21:59:08 | Error: Failed to download metadata for repo 'centos-opstools'"]} | 22:13 |
| dasm|rover | https://logserver.rdoproject.org/44/38844/3/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-wallaby/94fff24/job-output.txt | 22:13 |
| dasm|rover | for https://review.rdoproject.org/r/c/testproject/+/38844 | 22:13 |
| rlandy|ruck | off call | 22:27 |
| rlandy|ruck | checking on jobs in a few | 22:27 |
| dasm|rover | rlandy|ruck: i'm out of steam for today. i'm gonna disconnect. | 22:28 |
| rlandy|ruck | dasm|rover: sure - see you tomorrow | 22:28 |
| dasm|rover | rlandy|ruck: sorry for leaving you in the middle of the storm | 22:28 |
| *** dasm|rover is now known as dasm|off | 22:29 | |
| rlandy|ruck | dasm|off; no worries - have a good night | 22:35 |
| rlandy|ruck | rcastillo: looking into failure | 22:38 |
| rlandy|ruck | may need to update the patch | 22:39 |
| rlandy|ruck | rcastillo: merging https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/38856 | 23:05 |
| rlandy|ruck | to promote | 23:05 |
| rcastillo | rlandy|ruck: ack | 23:05 |
| *** ysandeep|out is now known as ysandeep | 23:30 | |
| ysandeep | rlandy|ruck, need help with anything for check/gate till marios comes online? | 23:58 |
| rlandy|ruck | ysandeep: hey | 23:58 |
| rlandy|ruck | ysandeep: we're dealing with so many failures | 23:59 |
| rlandy|ruck | I need to get marios patch though | 23:59 |
| rlandy|ruck | with also disables molecule | 23:59 |
| ysandeep | I can keep an eye if you can share the patch? | 23:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!