*** Goneri has quit IRC | 00:41 | |
*** ysandeep|away is now known as ysandeep | 01:46 | |
ysandeep | rlandy|bbl, ^^ i will try to debug - Invalid database user cinder@ | 02:00 |
---|---|---|
*** weshay|ruck has quit IRC | 02:05 | |
*** weshay has joined #oooq | 02:06 | |
*** apetrich has quit IRC | 02:14 | |
ysandeep | rlandy|bbl, I think i found something, notice - "cinder::keystone::authtoken::auth_uri": "http://%2522%2522:5000", - https://sf.hosted.upshift.rdu2.redhat.com/logs/75/208575/31/check/periodic-tripleo-ci-rhel-8-standalone-rhos-16-2/0e5c0b1/logs/undercloud/etc/puppet/hieradata/service_configs.json | 02:23 |
ysandeep | vs "cinder::keystone::authtoken::auth_uri": "http://192.168.24.1:5000", in good jobs | 02:23 |
*** jmasud has quit IRC | 02:33 | |
*** jmasud has joined #oooq | 02:36 | |
*** rlandy|bbl is now known as rlandy | 02:38 | |
rlandy | ysandeep: hey - yeah ... if you can look into ^^ that would be great | 02:38 |
rlandy | see you in a few hours | 02:38 |
*** rlandy has quit IRC | 02:38 | |
*** rfolco has quit IRC | 02:54 | |
*** ykarel|away has joined #oooq | 03:48 | |
*** ykarel|away is now known as ykarel | 04:14 | |
*** saneax has joined #oooq | 04:18 | |
*** raukadah is now known as chkumar|rover | 04:25 | |
*** skramaja has joined #oooq | 04:37 | |
*** yolanda has quit IRC | 04:41 | |
*** ratailor has joined #oooq | 04:42 | |
ykarel | chkumar|rover, is issue known for centos7 pip? | 04:47 |
ykarel | failure example:- https://f05144106d047809c3b3-8b5cc018a02fed487ca2a6d59a4ee9d5.ssl.cf2.rackcdn.com/746592/1/gate/tripleo-ci-centos-7-containers-multinode-train/12933b7/job-output.txt | 04:47 |
chkumar|rover | ykarel: nope | 04:48 |
ykarel | i seen multiple failures, seems mirror related | 04:48 |
ykarel | with some it pass and some it fail | 04:48 |
chkumar|rover | ykarel: noted on hackmd | 05:02 |
ykarel | chkumar|rover, ack it's affectoing gate job | 05:02 |
ykarel | chkumar|rover, and is train standalone issue known? | 05:03 |
ykarel | The task includes an option with an undefined variable. The error was: the inline if-expression on line 1 evaluated to false and no else section was defined. ? | 05:03 |
ykarel | seen in rdo gate job, should also be hitting in periodic and gate | 05:03 |
chkumar|rover | ykarel: https://0af46641a0193018034c-60d68d0bfd7f8f0178e466da97b3a245.ssl.cf2.rackcdn.com/746160/3/gate/tripleo-ci-centos-8-standalone/053a58b/job-output.txt | 05:04 |
ykarel | chkumar|rover, okk perfect | 05:04 |
ykarel | likely due to control vip patch | 05:05 |
chkumar|rover | ykarel: https://review.opendev.org/725782 ? | 05:06 |
ykarel | chkumar|rover, yes | 05:07 |
ykarel | i see there is test patch from ysandeep on changing this | 05:07 |
ykarel | https://review.opendev.org/#/c/743464/3/roles/standalone/tasks/main.yml | 05:07 |
ykarel | can add else clause to fix it | 05:08 |
chkumar|rover | ykarel: let me open the bug, you can send the fix | 05:09 |
ysandeep | chkumar|rover, i think ronelle already opened the bug | 05:09 |
ysandeep | chkumar|rover, ykarel https://review.opendev.org/746772 | 05:09 |
ysandeep | chkumar|rover, https://bugs.launchpad.net/tripleo/+bug/1892078 | 05:11 |
openstack | Launchpad bug 1892078 in tripleo "Train standalone jobs are failing check/gate - unable to resolve tripleo_deploy_control_virtual_ip" [Critical,In progress] - Assigned to Ronelle Landy (rlandy) | 05:11 |
ykarel | hmm ^^ is it, but it's failing need to check | 05:11 |
ysandeep | my patch was just for my downstream work till this issue is fixed, you can ignore that can we can focus on rlandy patch https://review.opendev.org/#/c/746772/ | 05:12 |
chkumar|rover | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_aeb/746772/4/check/tripleo-ci-centos-8-scenario001-standalone/aeb554e/logs/undercloud/home/zuul/standalone_deploy.log | 05:12 |
chkumar|rover | '\"192.168.24.3\"/32' is not valid under any of the given schemas", | 05:13 |
ykarel | extra quotes are injected | 05:13 |
chkumar|rover | yes | 05:13 |
chkumar|rover | removing quotes will fix it i think | 05:14 |
chkumar|rover | "{{ standalone_ha_control_virtual_ip }}" | 05:14 |
ykarel | yes +1 | 05:14 |
chkumar|rover | let me update that | 05:14 |
ykarel | ack | 05:15 |
ykarel | chkumar|rover, i think also need to handle tripleo_deploy_ha_control_plane not defined case | 05:16 |
ykarel | tripleo_deploy_ha_control_plane|default(false)|bool should handle it | 05:16 |
ykarel | and also standalone_ha_control_virtual_ip --> standalone_ha_control_virtual_ip|default() | 05:17 |
ykarel | chkumar|rover, ^ need to handle ^^ too | 05:19 |
chkumar|rover | on it | 05:20 |
chkumar|rover | now check | 05:22 |
chkumar|rover | ykarel: ^^ | 05:22 |
*** ykarel_ has joined #oooq | 05:22 | |
ykarel_ | chkumar|rover, okk ignore, first one also not necssary as it's always defined | 05:23 |
ykarel_ | just test with train to ensure nothing else is missed | 05:23 |
chkumar|rover | tripleo_deploy_ha_control_plane|default(false)|bool should handle it ? | 05:23 |
ykarel_ | u already send a patch, let me check | 05:24 |
chkumar|rover | ok | 05:24 |
*** ykarel has quit IRC | 05:25 | |
ykarel_ | chkumar|rover, you can revert to previous patch, extra conditions are not needed, if want to keep then replace tripleo_deploy_ha_control_plane|bool and tripleo_deploy_ha_control_plane|default(false)|bool --> tripleo_deploy_ha_control_plane|default(false)|bool | 05:26 |
ykarel_ | as both are duplicated | 05:27 |
chkumar|rover | yes | 05:28 |
chkumar|rover | now check | 05:29 |
ykarel_ | chkumar|rover, now looks better, let's see how ci goes | 05:30 |
ykarel_ | also test in train standalone | 05:30 |
*** ykarel__ has joined #oooq | 05:33 | |
chkumar|rover | ykarel_: sending | 05:35 |
*** ykarel_ has quit IRC | 05:36 | |
*** ykarel__ is now known as ykarel | 05:45 | |
*** marios has joined #oooq | 05:58 | |
*** marios is now known as marios|ruck | 05:58 | |
*** jfrancoa has joined #oooq | 06:02 | |
*** udesale has joined #oooq | 06:02 | |
*** dmellado has quit IRC | 06:11 | |
marios|ruck | chkumar|rover: \o/ | 06:28 |
chkumar|rover | marios|ruck: \o | 06:28 |
chkumar|rover | marios|ruck: https://review.opendev.org/746772 with ref https://launchpad.net/bugs/1892078 please have a look, thanks! | 06:29 |
openstack | Launchpad bug 1892078 in tripleo "Train standalone jobs are failing check/gate - unable to resolve tripleo_deploy_control_virtual_ip" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 06:29 |
marios|ruck | chkumar|rover: ack | 06:29 |
*** apetrich has joined #oooq | 06:29 | |
*** dmellado has joined #oooq | 06:41 | |
*** yolanda has joined #oooq | 06:50 | |
chkumar|rover | marios|ruck: https://review.opendev.org/#/q/fbd575cbe88400bad1ce4d37b914d7ae8a43e15c for https://bugs.launchpad.net/tripleo/+bug/1892008 | 06:50 |
openstack | Launchpad bug 1892008 in tripleo "periodic OVB train centos 7 failing overcloud configuration (config download) Invalid cross-device link" [Critical,In progress] - Assigned to Oliver Walsh (owalsh) | 06:50 |
marios|ruck | chkumar|rover: yeah thanks i saw that one yesterday will do ... was happy to see it was picked up so quickly | 06:52 |
marios|ruck | no retrospective in calendar chkumar|rover ? | 06:54 |
chkumar|rover | marios|ruck: yes, might be rfolco missed it | 06:54 |
chkumar|rover | let me send it | 06:55 |
marios|ruck | ack | 06:55 |
marios|ruck | ysandeep: o/ forgot to share this the other day if you are trying out irssi might find it useful https://quadpoint.org/articles/irssi/#hilight-window | 06:57 |
marios|ruck | akahat: ^ | 06:57 |
ysandeep | marios|ruck, thank you | 06:57 |
*** jbadiapa has joined #oooq | 06:57 | |
ysandeep | marios|ruck, i found it fascinating using irc from terminal :) | 06:58 |
marios|ruck | ysandeep: ack its cool once you have it setup and get used to it but its just a matter of preference some people don't like it | 07:04 |
*** dmellado has quit IRC | 07:07 | |
marios|ruck | chkumar|rover: actually not sure it will interrupt the check run | 07:17 |
marios|ruck | chkumar|rover: nice we got the train promotion (as we expected) last night \o/ | 07:18 |
marios|ruck | chkumar|rover: so very green for prog call | 07:18 |
chkumar|rover | awesome | 07:21 |
chkumar|rover | ykarel: c8 train standalone is now running at os_tempest | 07:22 |
chkumar|rover | marios|ruck: got the retro invite? | 07:28 |
marios|ruck | chkumar|rover: checking | 07:28 |
marios|ruck | gmail being a bit weird here | 07:28 |
marios|ruck | chkumar|rover: ack got it | 07:28 |
chkumar|rover | marios|ruck: cool, thanks! | 07:29 |
marios|ruck | chkumar|rover: hmm same time as program call | 07:29 |
chkumar|rover | marios|ruck: want me to shift a little? | 07:29 |
chkumar|rover | may be 30 mins later | 07:29 |
marios|ruck | chkumar|rover: yeah OK | 07:29 |
marios|ruck | chkumar|rover: i might join a little late then | 07:29 |
marios|ruck | vs missing it | 07:29 |
chkumar|rover | done | 07:30 |
marios|ruck | thanks | 07:30 |
marios|ruck | weshay: will you cancel today's cix (clash with open forum all hands) i think last time it was cancelled | 07:31 |
marios|ruck | weshay: asking you cos jbuchta is out | 07:31 |
ykarel | chkumar|rover, ack | 07:35 |
akahat | marios|ruck, thank you!! | 07:44 |
marios|ruck | chkumar|rover: rate limiting still a thing https://bugs.launchpad.net/tripleo/+bug/1889122/comments/44 | 07:45 |
openstack | Launchpad bug 1889122 in tripleo "mirror timeouts in upstream causing undercloud and standalone failures" [Critical,Triaged] - Assigned to Alex Schultz (alex-schultz) | 07:45 |
marios|ruck | akahat: :) np | 07:45 |
chkumar|rover | marios|ruck: yes | 07:49 |
Tengu | can't we talk to infra and drop that rate limitation? since it's "ours".... | 07:51 |
marios|ruck | chkumar|rover: fyi fix-released @ https://bugs.launchpad.net/tripleo/+bug/1891971/comments/2 | 07:51 |
openstack | Launchpad bug 1891971 in tripleo "ERROR: No matching distribution found for pprint (from -r /home/zuul/src/opendev.org/openstack/tripleo-ci/test-requirements.txt (line 6))" [Critical,Fix released] | 07:51 |
chkumar|rover | marios|ruck: yes | 07:52 |
chkumar|rover | Tengu: Does pep8 glanceclient issue fixed? | 07:52 |
Tengu | chkumar|rover: seems to be fixed yep | 07:53 |
*** dmellado has joined #oooq | 07:53 | |
Tengu | hitting other issues, but not related to that one | 07:53 |
*** jpena|off is now known as jpena | 07:58 | |
*** dtantsur|afk is now known as dtantsur | 08:16 | |
zbr | chkumar|rover: https://review.opendev.org/#/c/731497/ please (very easy) | 08:25 |
chkumar|rover | zbr: thanks! | 08:26 |
*** tosky has joined #oooq | 08:28 | |
chkumar|rover | ysandeep: do we have a dnm patch for rhos-16.2 https://review.opendev.org/#/c/745819/ testing? | 09:01 |
ysandeep | chkumar|rover, i tested it for previous patchset, but 16.2 are hitting number of issues so i have removed for now - Let me try to grab previous logs | 09:04 |
chkumar|rover | ysandeep: I just wanted to make sure package is getting built and installed | 09:05 |
chkumar|rover | ysandeep: can you pass me the log url? | 09:05 |
ysandeep | chkumar|rover, Without https://review.opendev.org/#/c/745819/ it was failing earlier on :- | 09:13 |
ysandeep | https://sf.hosted.upshift.rdu2.redhat.com/logs/75/208575/19/check/periodic-tripleo-ci-rhel-8-standalone-rhos-16-2/24f7a98/job-output.txt | 09:13 |
ysandeep | With https://review.opendev.org/#/c/745819/, it passed earlier step but later hit other issue and failed on some unit tests:- | 09:13 |
ysandeep | https://sf.hosted.upshift.rdu2.redhat.com/logs/75/208575/21/check/periodic-tripleo-ci-rhel-8-standalone-rhos-16-2/b31aec6/logs/delorean_logs/component/tripleo/0d/8d/0d8d7d375f136d7f3d6c9f0ffe68c87fc8ae84b4_dev_dev/mock.log.txt.gz | 09:13 |
ysandeep | Right now, we have removed depends-on which was causing trigger of buildtestpackage role(to avoid above unit test failure issue), Once basic standalone job is green i have plans to retest it. | 09:13 |
ysandeep | we can keep https://review.opendev.org/#/c/745819/ on hold, if you want to see run again | 09:14 |
chkumar|rover | ysandeep: ok. let's test later, since it is not going to affect upstream | 09:15 |
chkumar|rover | ysandeep: thanks for the link | 09:15 |
*** apetrich has quit IRC | 09:15 | |
ysandeep | chkumar|rover, thanks to you for review , I might buzz you for help with buildtestpackage role if I hit more issues. | 09:17 |
chkumar|rover | ysandeep: https://sf.hosted.upshift.rdu2.redhat.com/logs/75/208575/21/check/periodic-tripleo-ci-rhel-8-standalone-rhos-16-2/b31aec6/logs/delorean_logs/projects.ini | 09:19 |
chkumar|rover | downstream_distro_branch=rhos-16.2-rhel-8-trunk | 09:19 |
chkumar|rover | downstream_tag=osp-16.2 | 09:19 |
chkumar|rover | looks ok | 09:19 |
ysandeep | yes | 09:21 |
chkumar|rover | ysandeep: thanks for working on that :-) | 09:21 |
ysandeep | chkumar++ :) | 09:22 |
*** ratailor has quit IRC | 09:27 | |
*** ratailor_ has joined #oooq | 09:27 | |
*** ysandeep is now known as ysandeep|lunch | 09:35 | |
marios|ruck | chkumar|rover: oof bad buildset for train but all known/hitting https://bugs.launchpad.net/tripleo/+bug/1892078 | 09:42 |
openstack | Launchpad bug 1892078 in tripleo "Train standalone jobs are failing check/gate - unable to resolve tripleo_deploy_control_virtual_ip" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 09:42 |
marios|ruck | chkumar|rover: well, also some tempest | 09:42 |
chkumar|rover | marios|ruck: https://review.opendev.org/#/c/746772/ will fix it and tested it on train here https://review.opendev.org/#/c/746835/ | 09:43 |
marios|ruck | chkumar|rover: yeah but anyway we're good for train handoff 3 promotions this week. | 09:43 |
* marios|ruck shakes fist at ussuri | 09:43 | |
chkumar|rover | marios|ruck: yes | 09:44 |
marios|ruck | chkumar|rover: oof bad buildset for train but all known/hitting https://bugs.launchpad.net/tripleo/+bug/1892078 | 09:44 |
openstack | Launchpad bug 1892078 in tripleo "Train standalone jobs are failing check/gate - unable to resolve tripleo_deploy_control_virtual_ip" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 09:44 |
marios|ruck | sorry accidental duplication ^ | 09:44 |
chkumar|rover | marios|ruck: for ussuri we need to merge one https://review.opendev.org/#/c/746533/ | 09:44 |
marios|ruck | master promoting/will promote | 09:44 |
marios|ruck | * https://review.rdoproject.org/zuul/buildset/2b8df6b02eb14d4196a6e6f916337f03 | 09:44 |
chkumar|rover | then all will be green | 09:44 |
chkumar|rover | marios|ruck: cool | 09:45 |
marios|ruck | chkumar|rover: yeah must be the minor update fix | 09:45 |
marios|ruck | looking | 09:45 |
marios|ruck | ack | 09:45 |
marios|ruck | chkumar|rover: so maybe we get lucky and it promotes ussuri tonight/tomorrow morning | 09:45 |
chkumar|rover | https://review.opendev.org/#/c/725782/ introduced regression which brought https://bugs.launchpad.net/tripleo/+bug/1892078 | 09:46 |
openstack | Launchpad bug 1892078 in tripleo "Train standalone jobs are failing check/gate - unable to resolve tripleo_deploy_control_virtual_ip" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 09:46 |
*** ysandeep|lunch is now known as ysandeep | 09:55 | |
marios|ruck | chkumar|rover: fyi new * https://bugs.launchpad.net/tripleo/+bug/1892169 but not a blocker (commented out in criteria) | 10:03 |
openstack | Launchpad bug 1892169 in tripleo "periodic centos 8 train FS20 tempest fails test_server_connectivity_resize" [Undecided,Triaged] | 10:03 |
chkumar|rover | marios|ruck: ack, will take a look | 10:03 |
*** ysandeep is now known as ysandeep|brb | 10:27 | |
chkumar|rover | marios|ruck: so rate limit has arrived it is seen in most of the check jobs now | 10:43 |
chkumar|rover | especially c8 multinode jobs | 10:43 |
marios|ruck | chkumar|rover: yeah https://bugs.launchpad.net/tripleo/+bug/1889122/comments/44 | 10:44 |
openstack | Launchpad bug 1889122 in tripleo "mirror timeouts in upstream causing undercloud and standalone failures" [Critical,Triaged] - Assigned to Alex Schultz (alex-schultz) | 10:44 |
marios|ruck | chkumar|rover: we still have it on our hackmd anyway | 10:44 |
marios|ruck | chkumar|rover: you still working on https://bugs.launchpad.net/tripleo/+bug/1891372 ? i added a comment there it affects stein too looks like | 10:46 |
openstack | Launchpad bug 1891372 in tripleo "rocky periodic jobs are failing with " Error: image tripleorocky/centos-binary-tempest:9801dc7461cbd6cbd73868e72e74d21d586c6708_fbb4de96-updated-20200812131025 not found"" [Critical,In progress] - Assigned to chandan kumar (chkumar246) | 10:46 |
chkumar|rover | marios|ruck: checking | 10:48 |
*** ysandeep|brb is now known as ysandeep | 10:49 | |
zbr | chkumar|rover: weshay https://review.opendev.org/#/c/730389/ (easy) | 11:02 |
*** jpena is now known as jpena|lunch | 11:31 | |
*** apetrich has joined #oooq | 11:46 | |
*** apetrich has quit IRC | 11:52 | |
*** apetrich has joined #oooq | 11:52 | |
weshay | marios|ruck, fyi.. for next sprint https://review.opendev.org/#/c/731100/5 | 11:54 |
weshay | chkumar|rover, I think we can cancel todays retro | 11:55 |
weshay | and just hold it tomorrow | 11:55 |
weshay | that's what rfolco and I discussed anyway | 11:55 |
chkumar|rover | weshay: so tomorrow we have retro and planning both? | 11:58 |
weshay | chkumar|rover, since there is not that much to plan.. I think we can do it | 11:58 |
weshay | chkumar|rover, wdyt? | 11:58 |
chkumar|rover | weshay: sounds ok to me | 11:59 |
chkumar|rover | marios|ruck: bhagyashris arxcruz sshnaidm ysandeep svyas retro cancelled today | 11:59 |
ysandeep | chkumar|rover, ack | 12:00 |
chkumar|rover | we will be doing retro and planning together tomorrow | 12:00 |
marios|ruck | chkumar|rover: ack | 12:01 |
marios|ruck | chkumar|rover: i will add couple cards am out tomorrow | 12:01 |
chkumar|rover | marios|ruck: sure thanks! | 12:01 |
*** rlandy has joined #oooq | 12:02 | |
marios|ruck | weshay: looking (in prog call) | 12:02 |
marios|ruck | weshay: ack ok | 12:02 |
*** rfolco has joined #oooq | 12:03 | |
rlandy | ysandeep: hello | 12:04 |
ysandeep | rlandy, weshay \o/ good news we got our first 16.2 Standalone Green run :) | 12:04 |
rlandy | ysandeep: that's great .. more worried about the control_virtual_ip change | 12:04 |
ysandeep | rlandy chkumar|rover fixed that | 12:04 |
rlandy | chkumar|rover++ | 12:06 |
rlandy | ysandeep: that's great news | 12:07 |
rlandy | well done | 12:07 |
rlandy | ysandeep: ok - so I am closing https://bugzilla.redhat.com/show_bug.cgi?id=1869379 | 12:08 |
openstack | bugzilla.redhat.com bug 1869379 in rabbitmq-server "[OSP 16.2] Containers fail to start for step 2 - mysql_init_bundle error" [High,New] - Assigned to plemenko | 12:08 |
ysandeep | rlandy, go ahead | 12:09 |
ysandeep | rlandy, for standalone - with latest podman i faced some issues(details in email i sent to you), I used older podman to get green run. But from changelog it looks like latest and older podman don't have much difference.. May be we are hitting random issues - I am retriggering more jobs to confirm if podman is the issue. | 12:09 |
arxcruz | rfolco: don't we have sync today ? | 12:09 |
rfolco | nope arxcruz | 12:09 |
ysandeep | rlandy, https://code.engineering.redhat.com/gerrit/#/c/209162/ had to come put with this dirty hack to install older version of podman | 12:10 |
ysandeep | temporary ^^ | 12:10 |
rlandy | ysandeep: that should be set in container tools - let me check that | 12:12 |
ysandeep | s/put/up | 12:13 |
ysandeep | rlandy, ack | 12:13 |
rlandy | oh - the update to rhel 8.3 | 12:13 |
rlandy | rhos-release | 12:13 |
rlandy | appstream | 12:13 |
rlandy | this should be locked in container tools | 12:14 |
ysandeep | yes we are getting new rpm from rhosp-rhel-8.3-appstream | 12:15 |
ysandeep | i have run two job 1 with older version and 1 with new version - https://code.engineering.redhat.com/gerrit/#/c/208575/ & https://code.engineering.redhat.com/gerrit/#/c/201382/ lets see how this goes | 12:16 |
rlandy | ysandeep: so currently youhave two jobs running, right? | 12:16 |
rlandy | yep | 12:16 |
ysandeep | yes | 12:16 |
rlandy | looks like 1 just failed | 12:16 |
rlandy | let's check the versions | 12:16 |
rlandy | I think we have to lock container tools | 12:17 |
rlandy | I though rhos-release was doing that | 12:17 |
ysandeep | failed one was with newer podman version (so in total 4/4 jobs failed with newer podman) | 12:17 |
rlandy | buildah.x86_64 1.11.6-8.module+el8.3.0+7385+02cfa547 @rhosp-rhel-8.3-appstream | 12:18 |
chkumar|rover | marios|ruck: https://review.opendev.org/746894 will fix rocky stein tempest container issue | 12:18 |
rlandy | shoot - they upgraded buildah as well | 12:18 |
rlandy | but at leats not to 1.14 | 12:18 |
rlandy | podman.x86_64 1.6.4-13.module+el8.3.0+7385+02cfa547 @rhosp-rhel-8.3-appstream | 12:19 |
ysandeep | rlandy, i got success with podman.x86_64 1.6.4-12.module+el8.2.0+6670+014d0ff8 | 12:19 |
ysandeep | https://sf.hosted.upshift.rdu2.redhat.com/logs/75/208575/33/check/periodic-tripleo-ci-rhel-8-standalone-rhos-16-2/953874d/logs/undercloud/var/log/extra/package-list-installed.txt | 12:20 |
rlandy | ysandeep: so we definitely need to lock it | 12:20 |
rlandy | the question is hwy is rhos-release not | 12:20 |
ysandeep | rlandy, yup | 12:20 |
ysandeep | rlandy, i think this started after https://code.engineering.redhat.com/gerrit/#/c/209089/ | 12:21 |
rfolco | weshay, pls cancel today's retro | 12:21 |
rfolco | I 'll send a new invite for tomorrow | 12:21 |
ysandeep | rfolco, chandan canceled today's invite | 12:22 |
rlandy | ysandeep: yes it did | 12:22 |
rlandy | # ACTION: dnf -y module disable container-tools:rhel8 | 12:22 |
rlandy | # ACTION: dnf -y module enable container-tools:2.0 | 12:22 |
rlandy | I need to find out how this all works | 12:22 |
rfolco | ysandeep, ok thx | 12:23 |
rfolco | invite sent for tomorrow chkumar|rover | 12:23 |
ysandeep | rlandy, me too otherwise this thing will keep on banging our heads | 12:23 |
rlandy | ykarel: hi - saw your discussion container-tools on #rhos-delivery | 12:23 |
rlandy | I thought that rhos-release locked us to container-tools 2.0? | 12:24 |
rlandy | per lines above | 12:24 |
rlandy | maybe we need to execute that again? | 12:24 |
chkumar|rover | rfolco: thanks! | 12:25 |
rlandy | shreshtha> ykarel, it coming from module-container-tools-rhel8-8020120200601155013-ffd2803a | 12:25 |
rlandy | ysandeep: ^^ we should be 2.0 | 12:25 |
rlandy | ysandeep: let's try another test patch ... | 12:26 |
*** sshnaidm is now known as sshnaidm|afk | 12:26 | |
rlandy | in the release file, in stead of taking the old bets repo, | 12:26 |
rlandy | try this ... | 12:26 |
rlandy | https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-8/master.yml#L162 | 12:28 |
ykarel | rlandy, hi | 12:28 |
ykarel | rlandy, execute again, what? | 12:28 |
rlandy | ykarel: so ... rhos-release prints out ... | 12:29 |
ykarel | in rhos-delivery we were discussing compose failures | 12:29 |
rlandy | # ACTION: dnf -y module disable container-tools:rhel8 | 12:29 |
rlandy | # ACTION: dnf -y module enable container-tools:2.0 | 12:29 |
rlandy | so I thought it locked the container-tools to 2.0 | 12:29 |
ykarel | yes it's locking to container-tools:2.0 | 12:29 |
rlandy | ok - so it is ... | 12:30 |
rlandy | ysandeep's job picked up podman 1.6.4-13.module+el8.3.0+7385+02cfa547 | 12:30 |
rlandy | which fails | 12:30 |
ykarel | ^ part of container-tools:2.0 | 12:31 |
rlandy | since rhos-release was updated to follow latest ... https://code.engineering.redhat.com/gerrit/#/c/209089/2/repos/rhos-release-rhel-8.3.repo | 12:31 |
rlandy | so version are not locked even in container-tools:2.0 | 12:31 |
ykarel | so stream can ship multiple versions | 12:32 |
ykarel | newer wins | 12:32 |
ykarel | unless and until it's excluded | 12:32 |
rlandy | ykarel: what's the right way to go about this? log a bug to podman? | 12:33 |
rlandy | how would we exclude a version? | 12:33 |
rlandy | I see buildah updated as well but that seems ok | 12:34 |
ykarel | rlandy, yes i think bug needs to be created, but you can discuss with Emilien, he used to deal with podman | 12:35 |
ykarel | rlandy, wrt exclude, i just checked rhel8.3 repo, only 1 version is shipped in container-tools:2.0 | 12:35 |
ykarel | so nothing to exclude | 12:36 |
rlandy | ykarel: last question ... how do version get included in container-tools? is there a qualification process? | 12:36 |
*** jpena|lunch is now known as jpena | 12:37 | |
rlandy | https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_podman/vars/redhat.yml | 12:39 |
rlandy | ^^ would it help to lock this more specifically? | 12:40 |
rlandy | ysandeep: we haven't done a container build since the rhos-release update | 12:44 |
rlandy | better try that | 12:44 |
ysandeep | rlandy, okay, rechecking container build | 12:44 |
marios|ruck | rfolco: chkumar|rover: weshay: reminder /me out tomorrow for few days (see mail, also will send status later today ) | 12:45 |
rlandy | ysandeep: wait - it will fail | 12:46 |
rlandy | let me put back a change ... | 12:46 |
ysandeep | rlandy, okay | 12:46 |
chkumar|rover | marios|ruck: Happy holidays :-) | 12:46 |
chkumar|rover | weshay: we need to find new ruck & rovers for next sprint | 12:46 |
marios|ruck | \o/ thanks chkumar|rover i really need a break | 12:46 |
*** weshay is now known as weshay|interview | 12:47 | |
*** Goneri has joined #oooq | 12:47 | |
rlandy | ysandeep: ok - updated and rekicked | 12:49 |
ykarel | rlandy, https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_podman/vars/redhat.yml will only help if enabled module stream have that version | 12:49 |
ykarel | rlandy, what you mean by qualification process? | 12:50 |
ykarel | you mean qa expect? like what tests it's run before shipping | 12:50 |
rlandy | ykarel: run tests before updating the versions | 12:50 |
ykarel | rlandy, no i am not aware of that | 12:50 |
ysandeep | rlandy, ack thanks! | 12:51 |
rlandy | ysandeep: let's get a handle on how different the two versions are | 12:51 |
rlandy | I saw your comparison in the email | 12:52 |
ysandeep | rlandy, I am wondering if we locally extract rpm, we will be able to see the actual differences. I am not aware if we keep track somewhere or other easy way to do this. | 12:55 |
rlandy | ysandeep: short of looking through the commits in between, I don't have a better idea | 12:57 |
rlandy | I want to see what happens with container builds | 12:57 |
rfolco | zbr, arxcruz: would you review https://review.rdoproject.org/r/#/c/29003/ https://review.rdoproject.org/r/#/c/29004 https://review.rdoproject.org/r/#/c/28901/ and https://review.rdoproject.org/r/#/c/28902/ | 13:00 |
rlandy | ysandeep: this changing rhel base is going to make us all crazy! | 13:01 |
rlandy | if the steam is not locked | 13:01 |
rlandy | I don't see how we operate | 13:01 |
ysandeep | rlandy, :( | 13:01 |
rfolco | rlandy, marios|ruck chkumar|rover akahat ysandeep bhagyashris svyas pojadhav: pls add above patches to your review queue if possible. Thanks. | 13:02 |
rfolco | specially you zbr, arxcruz akahat bhagyashris ^ | 13:02 |
akahat | rfolco, ack | 13:03 |
bhagyashris | rfolco, ack | 13:03 |
zbr | rfolco: where is the value of changing the job name for periodic ones? IMHO is bad because I loose ability to see history of a job run on chances and periodi | 13:05 |
rfolco | zbr, I am replacing them with the new ones, discarding legacy ones | 13:05 |
rfolco | zbr, not renaming | 13:05 |
zbr | true and that is why i approved, but my question still stands | 13:06 |
zbr | why another name if no special reason. in fact there was something around file filters in the past, right? | 13:07 |
zbr | but I do not remember exactly | 13:07 |
rlandy | ysandeep: let's ask at the prod chain meeting | 13:07 |
rlandy | forwarding you the invite | 13:07 |
rfolco | zbr, mol-* are the new ones... molecule-delegated-* are the legacy ones. The reason is that the old ones run a totally different workflow. The new ones were just created fresh to not break the existing ones. | 13:07 |
rfolco | zbr, On our design we agreed to create new ones without touching the old ones | 13:08 |
rfolco | except the role itself, but that was necessary | 13:08 |
zbr | sure, question is unrelated to you patch. | 13:08 |
zbr | take a look at https://review.rdoproject.org/zuul/builds?job_name=mol-promoter# | 13:08 |
zbr | do you see gate, check and periodic? right? | 13:09 |
zbr | does it give you good indication if something went wrong, as periodic should had also fail | 13:09 |
rfolco | zbr, I don't understand... trying to process your comment... | 13:10 |
zbr | my question is: why we would still need to create additional child jobs that run periodic, they prevent us from seeing a page like above | 13:10 |
*** weshay|interview is now known as weshay | 13:10 | |
rfolco | aaaah | 13:11 |
rfolco | name should be preserved your mean for periodic ones | 13:11 |
rfolco | ok, I kept the same structure, just that. But yeah, I don't see any reason for having a child job just for periodic, we should be able to use the same names.... and we should be able to define them in ci-config | 13:12 |
rfolco | not in rdo-jobs | 13:12 |
weshay | rlandy, good morning.. | 13:13 |
rfolco | zbr, if that is an impediment, please -1 | 13:13 |
svyas | rfolco, ack | 13:13 |
zbr | rfolco: nope, it is but is unrelated to your change. | 13:13 |
rfolco | zbr, yeah I can propose a change now to consolidate the jobs if you agree | 13:14 |
weshay | rlandy, saw your pings from yesterday.. let's chat about the ansible role path when we have a sec.. but for the prod-chain council meeting.. I think we want to get insight on which repos to use for RHEL deps pipeline right? | 13:14 |
rlandy | weshay: ack ... need to ask about rhel there | 13:18 |
weshay | aye | 13:18 |
rlandy | weshay: ysandeep: I think I finally understand why people has money for stable versions of linux | 13:18 |
rlandy | pay | 13:18 |
weshay | lolz | 13:18 |
rlandy | I'd pay for stable rhel right now | 13:18 |
ysandeep | hahahaa | 13:18 |
rlandy | a lot | 13:18 |
rfolco | chkumar|rover, https://code.engineering.redhat.com/gerrit/#/c/208524/ is green, so the only jo that still fails is fs002 train/centos7, but not my patch's fault from what I see. So, when CI runs on https://review.opendev.org/733659, please help double checking everything is ok. I also did a recheck with the depends-on at https://review.rdoproject.org/r/#/c/28948/. | 13:18 |
weshay | chkumar|rover, marios|ruck so you two have one more morning as ruck/rover | 13:19 |
rfolco | chkumar|rover, thx for your help | 13:19 |
weshay | sorry | 13:19 |
*** ratailor_ has quit IRC | 13:19 | |
marios|ruck | weshay: i'm out tomorrow | 13:20 |
weshay | ah | 13:20 |
weshay | k | 13:20 |
weshay | I'll cover | 13:20 |
weshay | oh man.. I have my nick back | 13:20 |
weshay | \0/ | 13:20 |
weshay | hoooray | 13:21 |
*** weshay is now known as weshay|ruck | 13:21 | |
weshay|ruck | marios|ruck, don't tell anyone but I forgot my login to my znc bouncer | 13:21 |
weshay|ruck | lolz | 13:21 |
* weshay|ruck is dumb | 13:21 | |
weshay|ruck | chkumar|rover, marios|ruck cix canceled | 13:23 |
chkumar|rover | weshay|ruck: ok | 13:24 |
weshay|ruck | pain.. https://review.opendev.org/#/c/746533/ | 13:24 |
weshay|ruck | chkumar|rover, let's chat about tomorrow for a sec | 13:24 |
chkumar|rover | weshay|ruck: sure | 13:25 |
marios|ruck | weshay|ruck: ack | 13:25 |
marios|ruck | chkumar|rover: we can sync after paul call if you like | 13:25 |
marios|ruck | sorry i mean weshay|ruck ^^ | 13:25 |
marios|ruck | late then for chkumar|rover | 13:25 |
weshay|ruck | chkumar|rover, https://meet.google.com/gnd-xirg-bvb?authuser=1 | 13:26 |
weshay|ruck | marios|ruck, I just need to chat w/ him re: UA stuff | 13:26 |
marios|ruck | ack weshay|ruck was wondering ok | 13:26 |
rlandy | well container builds on 17 are completely broken | 13:26 |
ysandeep | rlandy, with base os/ rhel packages? | 13:28 |
rlandy | haven't even looked as to why | 13:30 |
*** udesale_ has joined #oooq | 13:33 | |
*** udesale has quit IRC | 13:36 | |
*** TrevorV has joined #oooq | 13:46 | |
rlandy | weshay|ruck: https://review.opendev.org/#/c/745912/ | 13:46 |
rlandy | weshay|ruck: want to chat now? | 14:12 |
weshay|ruck | rlandy, https://meet.google.com/bbs-bzse-vrj?authuser=1 | 14:13 |
marios|ruck | chkumar|rover: rfolco: fyi there are two UniSprint31 labels on the retrospective board | 14:14 |
marios|ruck | chkumar|rover: rfolco: i am adding both to my cards hope you don't miss them (out tomorrow adding cards now) | 14:14 |
rfolco | marios|ruck, if there is an empty one, please remove | 14:14 |
rfolco | marios|ruck, I mean no cards with that one yet | 14:14 |
marios|ruck | rfolco: both empty afaics | 14:15 |
rfolco | marios|ruck, use one and I'll remove the other | 14:15 |
rfolco | marios|ruck, thx for letting me know | 14:15 |
marios|ruck | rfolco: k feel free to remove one i added both of them | 14:15 |
rfolco | marios|ruck, k | 14:15 |
marios|ruck | rfolco: chkumar|rover: fyi * https://trello.com/c/P1oVcDAd * https://trello.com/c/Hqm11FOI * https://trello.com/c/KC1VcKxY | 14:16 |
marios|ruck | thanks | 14:16 |
chkumar|rover | marios|ruck: thanks! | 14:23 |
*** saneax is now known as saneax_ | 14:26 | |
ysandeep | rlandy, with success on vanila standalone.. from tommorow I will start attempting fs001/component. see you tomorrow o/ | 14:27 |
rlandy | ysandeep: ack - great work today! | 14:28 |
ysandeep | rlandy++ :) bye o/ | 14:28 |
*** ysandeep is now known as ysandeep|away | 14:29 | |
chkumar|rover | marios|ruck: one more down https://review.opendev.org/#/c/746894/2 rocky issue fixed | 14:32 |
marios|ruck | chkumar|rover: nice | 14:32 |
chkumar|rover | it will clear stein queens and rocky all | 14:32 |
rlandy | weshay|ruck: https://code.engineering.redhat.com/gerrit/#/c/209162/1/config/release/promotion-testing-hash-rhos-16.2.yml | 14:40 |
weshay|ruck | ysandeep|away, easy one https://code.engineering.redhat.com/gerrit/#/c/209162/1 | 14:41 |
*** sshnaidm|afk is now known as sshnaidm | 14:42 | |
rlandy | weshay|ruck: https://code.engineering.redhat.com/gerrit/#/c/195884 | 14:44 |
marios|ruck | weshay|ruck: chkumar|rover: so do you want to sync re ruck|rover. any ideas who will be next? | 15:07 |
*** skramaja has quit IRC | 15:10 | |
*** ykarel is now known as ykarel|away | 15:12 | |
chkumar|rover | marios|ruck: all ok from myside | 15:12 |
* chkumar|rover logs out now | 15:14 | |
chkumar|rover | marios|ruck: once again Happy Holidays :-) | 15:14 |
chkumar|rover | see ya | 15:14 |
*** chkumar|rover is now known as raukadah | 15:14 | |
marios|ruck | o/ raukadah | 15:14 |
marios|ruck | raukadah: thanks for your support sprint3 1 | 15:14 |
raukadah | marios|ruck: thank to you too for the same :-) | 15:15 |
weshay|ruck | marios|ruck, meh.. no not sure | 15:16 |
marios|ruck | weshay|ruck: not sure? | 15:16 |
marios|ruck | oh who will be next | 15:17 |
marios|ruck | weshay|ruck: just go in order | 15:17 |
weshay|ruck | marios|ruck, just make sure chandan has the info | 15:17 |
marios|ruck | weshay|ruck: ack yeah me and raukadah are all synced up to time.now() | 15:18 |
*** ykarel|away has quit IRC | 15:26 | |
*** apetrich has quit IRC | 15:46 | |
* marios|ruck shutdown sequence | 15:46 | |
marios|ruck | weshay|ruck: ^ going in few | 15:46 |
*** marios|ruck is now known as marios|out | 15:56 | |
rlandy | weshay|ruck: ysandeep|away: https://code.engineering.redhat.com/gerrit/#/c/209162/ updated | 15:57 |
*** marios|out has quit IRC | 16:06 | |
*** jpena is now known as jpena|off | 16:07 | |
weshay|ruck | rlandy, 0/ | 16:12 |
rlandy | weshay|ruck: hey | 16:12 |
weshay|ruck | rlandy, meant to say.. +2 ^ | 16:13 |
rlandy | weshay|ruck: so I think we should just keep the beta repo option and add to it as needed while rhel 8.3 is in flux | 16:13 |
weshay|ruck | k | 16:16 |
*** udesale_ has quit IRC | 16:18 | |
weshay|ruck | sshnaidm, rlandy arxcruz sorry to kill your week next week w/ interviews | 16:22 |
weshay|ruck | we have two really competitive candidates | 16:22 |
rlandy | that's a good problem | 16:22 |
weshay|ruck | and we can repeat the same questions for both | 16:22 |
weshay|ruck | they have similiar backgrounds.. | 16:23 |
arxcruz | weshay|ruck: cool | 16:30 |
arxcruz | I'm having fun with panda's code it might be a good idea to change a little bit the context, that code is driving me crazy | 16:30 |
*** dtantsur is now known as dtantsur|afk | 17:01 | |
rlandy | weshay|ruck: https://review.rdoproject.org/r/#/c/28715/ | 17:42 |
weshay|ruck | rlandy, https://review.opendev.org/#/q/topic:tcib/train+(status:open+OR+status:merged) | 17:45 |
rlandy | weshay|ruck: https://code.engineering.redhat.com/gerrit/#/c/207802/ | 17:58 |
rlandy | weshay|ruck: https://code.engineering.redhat.com/gerrit/#/c/208575/ | 17:59 |
*** apetrich has joined #oooq | 18:21 | |
weshay|ruck | rlandy, https://opendev.org/openstack/tripleo-common/src/commit/d28fb0be8ec1421b87741615597e9033e82ba803/tripleo_common/actions/ansible.py#L74-L82 | 18:48 |
weshay|ruck | rlandy, same in train and master | 18:49 |
rlandy | weshay|ruck: yeah - no idea why | 18:49 |
weshay|ruck | https://opendev.org/openstack/tripleo-common/src/branch/stable/train/tripleo_common/actions/ansible.py | 18:49 |
rlandy | '/usr/share/ansible/roles:' | 18:49 |
weshay|ruck | rlandy, we could update /etc/ansible/ansible.cfg | 18:50 |
rlandy | http://git.app.eng.bos.redhat.com/git/openstack-tripleo-common.git/tree/tripleo_common/actions/ansible.py?h=rhos-16.2-trunk-patches#n77 | 18:51 |
rlandy | why update - it's there | 18:51 |
weshay|ruck | not the virtual env | 18:52 |
weshay|ruck | yes.. have you copy everything | 18:52 |
rlandy | weshay|ruck: on the upside, my patch didn't break 17 ... here is the periodic job ... | 18:52 |
weshay|ruck | of course it's not logged https://sf.hosted.upshift.rdu2.redhat.com/logs/84/195884/25/check/periodic-tripleo-build-containers-ubi-8-internal-rhel-8-build-push-upload-rhos-16.2/a4d3785/etc/ | 18:52 |
rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/84/195884/26/check/periodic-tripleo-build-containers-ubi-8-internal-rhel-8-build-push-upload-rhos-17/d470db1/logs/build.log | 18:52 |
weshay|ruck | ah good | 18:52 |
rlandy | ^^ legit failure on tempest | 18:53 |
rlandy | maybe we can fix that | 18:53 |
rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/84/195884/26/check/periodic-tripleo-build-containers-ubi-8-internal-rhel-8-build-push-upload-rhos-17/d470db1/logs/buildah-builds/container-builds/4790eedd-72aa-4946-9b21-f5a946e9df54/base/os/tempest/tempest-build.log | 18:53 |
rlandy | weshay|ruck: missing deps | 18:54 |
rlandy | - cannot install the best candidate for the job | 18:54 |
rlandy | - nothing provides python3.6dist(pyasn1-modules) >= 0.2.1 needed by python3-google-auth-1:1.19.1-1.el8ost.noarch | 18:54 |
rlandy | - nothing provides python3.6dist(setuptools) >= 40.3.0 needed by python3-google-auth-1:1.19.1-1.el8ost.noarch | 18:54 |
rlandy | http://download.eng.bos.redhat.com/rel-eng/repos/rhos-17.0-rhel-8-trunk-candidate/tagged-builds.lst | 18:55 |
rlandy | python-google-auth-1.19.1-1.el8ost | 18:56 |
weshay|ruck | https://review.opendev.org/747022 | 18:56 |
weshay|ruck | add ansible config to container builds | 18:56 |
weshay|ruck | rlandy, so.. the move to 8.3 caused those errors? | 18:56 |
weshay|ruck | apparently | 18:56 |
rlandy | weshay|ruck: no - 17 | 18:56 |
rlandy | :) | 18:57 |
weshay|ruck | ya.. but 17 moved to 8.3 as well right? | 18:57 |
rlandy | not yet | 18:57 |
weshay|ruck | I guess we don't know | 18:57 |
rlandy | latest-RHOS_TRUNK-17-RHEL-8.2/ | 18:57 |
rlandy | 8.2 | 18:58 |
rlandy | missing dep | 18:58 |
rlandy | python3-kubernetes noarch 10.0.1-1.el8 delorean-master-testing 1.1 M | 19:03 |
rlandy | weshay|ruck: let's promote 17 | 19:03 |
rlandy | the last one with passing containers | 19:04 |
weshay|ruck | k | 19:04 |
* rlandy gets | 19:04 | |
rlandy | https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?pipeline=openstack-periodic-rhos-17&job_name=periodic-tripleo-build-containers-ubi-8-internal-rhel-8-build-push-upload-rhos-17 | 19:05 |
rlandy | 08-14 | 19:05 |
rlandy | aggregate_hash": "62db6bcf3201a5aa1c8632f62975c6d1" | 19:05 |
rlandy | [DIR]current-tripleo/2020-07-31 12:22 | 19:06 |
rlandy | periodic-tripleo-ci-rhel-8-standalone-rhos-17openstack/tripleo-cimasteropenstack-periodic-rhos-17master37 mins 6 secs2020-08-12 00:44:58FAILURE | 19:13 |
rlandy | periodic-tripleo-ci-rhel-8-standalone-rhos-17openstack/tripleo-cimasteropenstack-periodic-rhos-17master1 hr 28 mins 21 secs2020-08-11 01:50:29SUCCESS | 19:13 |
rlandy | did we stop installing pacemaker? | 19:13 |
weshay|ruck | I hope not.. still on a call though | 19:13 |
weshay|ruck | need a few | 19:13 |
*** jfrancoa has quit IRC | 19:22 | |
*** saneax_ has quit IRC | 19:32 | |
weshay|ruck | rlandy, rhos-17 promotion | 19:58 |
weshay|ruck | need me? | 19:58 |
weshay|ruck | rlandy, ysandeep|away fyihttps://review.opendev.org/#/q/topic:modules+(status:open+OR+status:merged) | 20:07 |
rlandy | weshay|ruck: yes - pls | 20:08 |
rlandy | weshay|ruck: wrt https://review.opendev.org/#/c/747024/ going to try that with a temp downstream change | 20:09 |
weshay|ruck | rlandy, k | 20:10 |
weshay|ruck | rlandy, so tmux on internal? | 20:11 |
rlandy | weshay|ruck: yep | 20:11 |
weshay|ruck | ok.. might as well chat while we do it | 20:11 |
weshay|ruck | https://meet.google.com/aox-rusm-goe?authuser=1 | 20:11 |
weshay|ruck | rlandy, https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/buildset/62fab4171d4f40119d1bbf8885a548b7 | 20:17 |
weshay|ruck | rlandy, d18671cafbf31d038fe085ed42850fcc | 20:21 |
weshay|ruck | commit_hash: 21a722adac9c844d33a07323d111d0fe7a888ab3 | 20:22 |
weshay|ruck | distro_hash: 159f4d19b142dbedd11f895f00905de93b033f12 | 20:22 |
*** rfolco has quit IRC | 21:05 | |
*** jmasud has quit IRC | 21:10 | |
*** jmasud has joined #oooq | 21:19 | |
*** jmasud has quit IRC | 21:30 | |
*** jmasud has joined #oooq | 21:36 | |
*** tosky has quit IRC | 21:47 | |
*** jmasud has quit IRC | 22:14 | |
*** jmasud has joined #oooq | 22:16 | |
*** rlandy is now known as rlandy|bbl | 22:28 | |
*** TrevorV has quit IRC | 22:31 | |
*** apetrich has quit IRC | 22:47 | |
*** Goneri has quit IRC | 23:27 | |
*** jmasud has quit IRC | 23:53 | |
*** jmasud has joined #oooq | 23:59 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!