| *** rlandy|bbl is now known as rlandy|out | 00:55 | |
| rlandy|out | rekicked fs001 and fs035 | 00:55 |
|---|---|---|
| dasm|ruck | k | 01:09 |
| *** dasm|ruck is now known as dasm|ruck|ff | 04:19 | |
| *** dasm|ruck|ff is now known as dasm|ruck|off | 04:19 | |
| *** ysandeep|out is now known as ysandeep | 05:01 | |
| *** pojadhav- is now known as pojadhav | 05:07 | |
| *** marios is now known as marios|ruck | 05:15 | |
| ysandeep | rlandy|out, this job need to be moved to c9 also https://jenkins-cloudsig-ci.apps.ocp.ci.centos.org/job/tripleo-quickstart-extras-gate-master-tripleo-ci-delorean-full-featureset052/ or removed. | 05:34 |
| ysandeep | reviewbot: please add in review list: https://review.opendev.org/q/topic:discover-latest-image | 05:43 |
| reviewbot | I have added your review to the Review list | 05:43 |
| ysandeep | reviewbot: thanks! | 05:45 |
| reviewbot | You are welcome :) | 05:45 |
| dpawlik | frenzy_friday: hey, from #openstack-tc: "<gmann> yes, merging should not leads to delete the master and copy rdo one, it should be both merged" | 06:23 |
| dpawlik | cc arxcruz, rlandy|out | 06:23 |
| ysandeep | marios|ruck, fyi.. https://code.engineering.redhat.com/gerrit/gitweb?p=openstack/sf-config.git;a=tree;f=nodepool/virt_images/roles/discover-latest-image;h=de7e663cc82e870de0a04d070fa1eae52ed459cc;hb=HEAD we already have discover-latest-image role in downstream written by rlandy|out few months back, I made minimum changes to keep the role consistent in upstream and downstream, thanks for review I will brainstorm about your suggestions. | 06:54 |
| marios|ruck | ysandeep: ack - i mean we may as well do a review and improve if there is agreement | 06:54 |
| ysandeep | yeah, make sense | 06:56 |
| marios|ruck | chem: o/ fwiw ;) https://bugs.launchpad.net/tripleo/+bug/1971605/comments/3 | 07:01 |
| chem | marios|ruck: hey, thanks. | 07:12 |
| marios|ruck | chem: o/ np | 07:15 |
| chem | marios|ruck: mumble back something ... don't know if it's even worth the time of back and forth :) | 07:16 |
| marios|ruck | chem: do you prefer not do use the change you proposed you mean? | 07:17 |
| marios|ruck | chem: mumble mumble snort mumble | 07:17 |
| chem | marios|ruck: yeah, I've just got a idea ! | 07:17 |
| marios|ruck | chem: ack replied ... worth a shot but you may find those vars inaccessible because we have 'nested' ansible in our executions | 07:24 |
| *** jgilaber is now known as jgilaber|away | 07:28 | |
| jm1 | chem: do you have an example where job.vars become "prefixed" with "job."? | 07:38 |
| jm1 | chem: *example job | 07:38 |
| *** jpena|off is now known as jpena | 07:39 | |
| *** ykarel_ is now known as ykarel | 07:48 | |
| ykarel | marios|ruck, chem if wanted to run just with job vars to pass to quickstart, can set those vars in run_test_role_vars: | 07:50 |
| ykarel | or other option can be to expose the required vars in featureset override and set those vars in featureset_override | 07:50 |
| ykarel | can update reproducer docs too with this | 07:51 |
| marios|ruck | ykarel: ack featureset_override is another option | 07:51 |
| * marios|ruck fetch coffee brb | 07:51 | |
| chem | ykarel: yeah, I've seen the featureset_override part as well, currently those options are not taken into account, but that could be another way to solve this | 07:52 |
| ykarel | run_test_role_vars is more flexible as would not need reviews for each and every quicstart var | 07:52 |
| ykarel | chem, can you try run_test_role_vars:, this should work for the case u mentioned | 07:52 |
| chem | ykarel: the run_test_role_vars I didn't see that one ... I'll look into it before eod | 07:52 |
| ykarel | chem, ack whatever we set it in ^ var are passed to quickstart run | 07:53 |
| chem | ykarel: but it's taken from job.run_test_roles_vars ? | 07:53 |
| ykarel | no | 07:53 |
| ykarel | actually yes | 07:54 |
| ykarel | let me share the link, | 07:54 |
| chem | ykarel: ah ... it's either that (taken from from job.run ....) or else in featureset override | 07:54 |
| ykarel | chem, https://github.com/openstack/tripleo-ci/blob/d492451c1311e6d83cc7816ca93a5381948dea50/roles/run-test/templates/role-vars.j2#L1 | 07:54 |
| ykarel | the var file is generated from job var run_test_role_vars ^ and is passed to quickstart | 07:55 |
| ykarel | so whatever vars set in run_test_role_vars exist in the generated var file | 07:55 |
| chem | oki, so that won't work unless we can pass it from featureset override, unless I'm missing something | 07:55 |
| chem | I think I need job.run_test_role_vars to become run_test_role_vars somehow | 07:56 |
| ykarel | chem, no not needed | 07:57 |
| ykarel | so what happens is whatever we set in job: vars: is available in tripleo ci playbooks/roles directly | 07:58 |
| ykarel | if want to use in nested quickstart, job.<var> can be used | 07:58 |
| ykarel | run_test_role_vars job var is used in tripleo ci playbooks, so directly accessible and whatever content it has, based on it role-vars.yaml is generated and passed to quickstart | 07:59 |
| chem | ykarel: my use case is to have the left pane working somehow https://review.rdoproject.org/r/c/rdo-infra/ansible-role-tripleo-ci-reproducer/+/42555/2/README.md | 08:00 |
| chem | ykarel: so this will be a job.... variable | 08:00 |
| chem | ykarel: or I hardcode them in the psi file (which kinda make sense as well) | 08:00 |
| chem | ykarel: that's the right pane in that review :) | 08:01 |
| ykarel | chem, https://paste.opendev.org/show/b6sn90bpswGsrqvrSehs/ way should work | 08:01 |
| chem | ykarel: oki... I'll give it a try, overall that would be a powerful way to overcome the featureset override. | 08:02 |
| ykarel | yeap exactly | 08:02 |
| chem | ykarel++ marios++ thanks for the help so far, I'll keep you posted of what I find out | 08:03 |
| marios|ruck | ack np chem thanks ykarel | 08:05 |
| chem | ykarel: I've added your comment to the lp so that my old fish like brain doesn't forget .... testing today at some point ... | 08:06 |
| chem | need to go on an errand, be back in ~1h | 08:06 |
| ykarel | chem, ack | 08:06 |
| *** chem is now known as chem|afk | 08:06 | |
| *** ysandeep is now known as ysandeep|lunch | 08:07 | |
| *** jgilaber|away is now known as jgilaber | 08:27 | |
| jpodivin | marios|ruck: hi. About the failing gate due to missing oslotest from yesterday. | 08:59 |
| jpodivin | marios|ruck: it passed after recheck? | 08:59 |
| marios|ruck | jpodivin: i didn't check can you remind me the link ? are you saying or asking :) | 09:00 |
| jpodivin | marios|ruck: sorry, saying https://trello.com/c/MPWI0Il8/2489-cixbz2080920pipelineosp17dftripleo-heat-templateovercloudrc-have-empty-ospassword | 09:00 |
| jpodivin | marios|ruck: the whole sitch is weird | 09:01 |
| *** pojadhav is now known as pojadhav|lunch | 09:01 | |
| jpodivin | but good news is that it passed, and that I have a fix for what appears to be an intermittent error. | 09:01 |
| marios|ruck | jpodivin: k can you add the info on the card/bug | 09:01 |
| marios|ruck | jpodivin: thanks for udpate | 09:01 |
| jpodivin | marios|ruck: already done | 09:02 |
| jpodivin | VF knows about it and the fix should get merged today | 09:02 |
| marios|ruck | thank you jpodivin | 09:04 |
| jpodivin | marios|ruck: np. | 09:06 |
| *** pojadhav|lunch is now known as pojadhav|afk | 09:27 | |
| *** ysandeep|lunch is now known as ysandeep | 09:36 | |
| marios|ruck | arxcruz: did we remove the ara file from the job logs cant find it any more | 09:49 |
| arxcruz | marios|ruck it seems so long time ago, i always wonder the same... | 09:49 |
| marios|ruck | arxcruz: k thanks... maybe we should reconsider (did it take a lot of time to make it? or space? do you remember why we removed?) | 09:51 |
| arxcruz | marios|ruck i don't remember the reason, but would be nice to have it, sometimes is better to debug errors on ara than checking logs | 09:52 |
| *** chem|afk is now known as chem | 10:25 | |
| *** rlandy|out is now known as rlandy | 10:25 | |
| *** chem is now known as Guest169 | 10:25 | |
| rlandy | marios|ruck: hello ... | 10:26 |
| rlandy | marios|ruck: want to open a ticket to vexx today | 10:26 |
| marios|ruck | rlandy: \o | 10:26 |
| rlandy | based on train now also failing deploy | 10:26 |
| marios|ruck | rlandy: yeah i know, i've been trying to gather info but don't have something major | 10:26 |
| rlandy | marios|ruck: let me know when you have a few to meet | 10:27 |
| marios|ruck | rlandy: sure | 10:27 |
| marios|ruck | rlandy: lets meet | 10:27 |
| rlandy | marios|ruck: give me 5 ... looking for our ticket account | 10:27 |
| rlandy | will ping | 10:27 |
| marios|ruck | k brb | 10:28 |
| rlandy | chandankumar: hey - do you have an account on vexx support? | 10:30 |
| chandankumar | rlandy: not logged in there from long time | 10:33 |
| chandankumar | checking now | 10:33 |
| chandankumar | rlandy: yuo | 10:33 |
| chandankumar | yup I have an account | 10:33 |
| chandankumar | rlandy: Only dasm|ruck|off and me are there in vex contact | 10:35 |
| rlandy | chandankumar: ack - I don;t have it | 10:35 |
| rlandy | chandankumar: waiting for marios|ruck to come back | 10:35 |
| rlandy | then pls join https://meet.google.com/hdc-eofm-niz | 10:35 |
| rlandy | marios|ruck: ^^ pls ping when you are back | 10:35 |
| chandankumar | rlandy: let me tell apevc to add you there | 10:35 |
| marios|ruck | rlandy: joining | 10:36 |
| rlandy | chandankumar: pls | 10:36 |
| *** pojadhav|afk is now known as pojadhav | 10:36 | |
| rlandy | chandankumar: marios|ruck: thanks ... on https://meet.google.com/hdc-eofm-niz now | 10:36 |
| ysandeep | rlandy: I am next ruck/rover, do you mind if I join as well to understand what we are reporting to vexx. | 10:36 |
| rlandy | ysandeep: yeah - join the fun | 10:38 |
| ysandeep | marios|ruck, is this the deployment error we are talking about in train? https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/790544f/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 10:45 |
| ysandeep | after heat stack create, we just get deployment failure | 10:46 |
| marios|ruck | ysandeep: yeah 2022-05-05 09:18:14 | Timed out waiting for port 22 from 192.168.24.19 | 10:46 |
| marios|ruck | ysandeep: we've seen that a couple times, like yesterday as well | 10:47 |
| marios|ruck | ysandeep: so they may all be the same root cause | 10:47 |
| ysandeep | ack, thanks! | 10:47 |
| ysandeep | sounds like network/floating ip /security group related | 10:48 |
| *** dviroel|out is now known as dviroel | 11:21 | |
| rlandy | forwarded initial ticket response | 11:31 |
| marios|ruck | rlandy: where email? | 11:31 |
| marios|ruck | rlandy: yea see it now thanks | 11:31 |
| rlandy | adding to the ticket alan's request to check the L3 | 11:33 |
| ysandeep | I have two testprojects running for master and train (added on hold as well): https://review.rdoproject.org/r/c/testproject/+/31954 and https://review.rdoproject.org/r/c/testproject/+/39357, Hopefully we will get something from reproducer. | 11:40 |
| *** ysandeep is now known as ysandeep|afk | 11:46 | |
| rlandy | marios|ruck: anything else need attention? 16.2 and rhos-17 on rhel-9 promoted yesterday - looking at the rhel-8 one | 12:10 |
| rlandy | ysandeep|afk: ^^ nice to see downstream being the stable platform now | 12:11 |
| rlandy | soniya29: ^^ fyi ... we had the whole vexxhost discussion | 12:11 |
| soniya29 | rlandy, okay | 12:12 |
| rlandy | all; anyone who logged conquer expenses - pls check you were paid | 12:14 |
| rlandy | jm1; ysandeep|afk; ^^ others | 12:14 |
| rlandy | I approved but can't tell if you got reimbursed | 12:15 |
| marios|ruck | rlandy: not aware of some issue atm. did a round on components earlier gate is green | 12:20 |
| jm1 | rlandy: thank you very much. last time it took two weeks after approval for payment to arrive, so plenty of time left :) | 12:21 |
| marios|ruck | rlandy: looks like there may be some new gate issue starting (tht) digging | 12:24 |
| *** soniya29 is now known as soniya29|afk | 12:27 | |
| * soniya29|afk will be back in a hr | 12:27 | |
| *** ysandeep|afk is now known as ysandeep | 12:29 | |
| ysandeep | rlandy: yeah downstream is in good shape, I have retrigger the tripleo component failing ovb job in 17/8 today. | 12:32 |
| ysandeep | rlandy, thanks! I haven't submitted any expense report yet. | 12:33 |
| rlandy | ysandeep: 17/8 is the promotion I am chasing as well | 12:34 |
| bhagyashris | chandankumar, soniya rlandy | 13:01 |
| bhagyashris | scrum time | 13:01 |
| bhagyashris | dasm|ruck|off, | 13:01 |
| bhagyashris | rlandy, ^ | 13:02 |
| rlandy | sorry - joining | 13:02 |
| rlandy | just chatting with vexx | 13:02 |
| bhagyashris | ok np | 13:02 |
| *** dasm|ruck|off is now known as dasm | 13:07 | |
| chandankumar | https://review.rdoproject.org/r/c/config/+/42529 and https://review.opendev.org/c/openstack/tripleo-ci/+/839149 | 13:07 |
| *** dasm is now known as dasm|ruck | 13:08 | |
| dasm|ruck | o/ | 13:08 |
| dviroel | https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/42378 | 13:15 |
| pojadhav | folks, https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/42425 | 13:19 |
| rlandy | rcastillo: not a holiday for you today?? | 13:30 |
| rcastillo | rlandy: apparently not, but the list says I had sunday (may 1st) off :) | 13:31 |
| rlandy | soniya: hello | 13:35 |
| rlandy | you around? | 13:35 |
| rlandy | soniya: you're missing scrum and rr hand off | 13:36 |
| rlandy | 9 days agotempestpromoted-components2 days ago | 13:46 |
| rlandy | ysandeep: dasm|ruck: ^^ rhel-9 component | 13:46 |
| dasm|ruck | ack | 13:49 |
| dasm|ruck | cs9 master promotion skip: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/42580 | 13:49 |
| dasm|ruck | rlandy: marios|ruck ^ | 13:50 |
| marios|ruck | dasm|ruck: ack | 14:00 |
| dasm|ruck | brb | 14:01 |
| ysandeep | marios|ruck, dasm|ruck rlandy thanks! I will check the tht blocker in my morning, I got a reproducer for ovb train issue - trying to debug that before my eod. | 14:02 |
| marios|ruck | ysandeep: ack thanks i'll keep digging at the gate thing | 14:03 |
| marios|ruck | rlandy: http://lists.openstack.org/pipermail/openstack-discuss/2022-May/028421.html | 14:04 |
| rlandy | marios|ruck; thank you | 14:08 |
| rlandy | ysandeep++ on reproducer | 14:08 |
| *** soniya is now known as soniya|afk | 14:15 | |
| ysandeep | marios|ruck, dasm|ruck rlandy hah - ssh service on overcloud node is down (c8 train) | 14:21 |
| ysandeep | that explains why can't we ssh | 14:21 |
| marios|ruck | ysandeep: wtf ... but then that one may just be different bug? | 14:21 |
| marios|ruck | i mean not related to the vexx/issues with tempest etc | 14:21 |
| marios|ruck | ysandeep: or, is it down and should be up by now? (maybe some delay?) | 14:21 |
| ysandeep | marios|ruck, let me pm you the console url - so that you can see as well | 14:21 |
| ysandeep | marios|ruck, failed to restart | 14:22 |
| rlandy | ysandeep: pls bring up at DF call | 14:23 |
| ysandeep | rlandy, marios|ruck dasm|ruck fyi.. https://paste.opensuse.org/70731614 | 14:26 |
| marios|ruck | ysandeep: ack wtf... | 14:27 |
| ysandeep | https://paste.opensuse.org/73624218 no hostkey available | 14:27 |
| dasm|ruck | back | 14:35 |
| dasm|ruck | ysandeep: huh. good find | 14:36 |
| rlandy | pls add to bug | 14:41 |
| ysandeep | reading https://www.xmodulo.com/sshd-error-could-not-load-host-key.html - looks like keys get created on ssh install - "When OpenSSH server is first installed on Linux system, SSH host keys should automatically be generated for subsequent use. " | 14:51 |
| ysandeep | ssh issue solves If I create keys manually, "sudo ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key" "sudo ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key" | 14:52 |
| * ysandeep checking rpm diff between current and last good known | 14:53 | |
| dpawlik | rlandy: wanna join https://bostonu.zoom.us/j/96404797762?pwd=SWpBNW4wekppV0piUmZzS2l6YzRwQT09#success ? | 15:02 |
| rlandy | dpawlik: on DF call | 15:03 |
| dpawlik | or frenzy_friday | 15:03 |
| dpawlik | ah | 15:03 |
| dpawlik | ack. | 15:03 |
| dasm|ruck | brb | 15:04 |
| rlandy | dasm|ruck: ysandeep: soniya|afk: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/840683/ | 15:11 |
| *** soniya|afk is now known as soniya | 15:11 | |
| ysandeep | rlandy, fyi.. trying dib patch revert https://review.opendev.org/c/openstack/diskimage-builder/+/840486 , here: https://review.rdoproject.org/r/c/testproject/+/39357 | 15:23 |
| rlandy | thanks, sandeep | 15:24 |
| frenzy_friday | dpawlik, hey sorry, was afk, are you still on the call? | 15:24 |
| dpawlik | frenzy_friday: yup. It is recorded, just FYI | 15:25 |
| rlandy | ysandeep: marios|ruck: dasm|ruck: response from pidone ... | 15:31 |
| rlandy | <lmiccini> to suggest storage is slow and so the podman monitors take too long to return. you could increase the pacemaker timeouts and also the haproxy checks timeouts but that would be just a workaround and you would still have to trial&error to find the right values | 15:32 |
| rlandy | Tengu: ^^ | 15:32 |
| marios|ruck | rlandy: thanks | 15:32 |
| rlandy | will need to look for those timeouts when out of meetingfs | 15:32 |
| rlandy | if you guy have time to do it before then, pls do | 15:33 |
| rlandy | next hour and a half are meeting booked | 15:33 |
| ysandeep | rlandy, I have a reproducer for master tempest issue as well - let me try changing the timeout | 15:33 |
| ysandeep | its late for me, but let me check this quickly | 15:34 |
| marios|ruck | dasm|ruck: going in a few - need something before i go? | 15:41 |
| dasm|ruck | back | 15:47 |
| dasm|ruck | marios|ruck: no, thanks mate | 15:47 |
| marios|ruck | dasm|ruck: few hours left ;) ... | 15:47 |
| dasm|ruck | \( ﾟヮﾟ)/ | 15:48 |
| * marios|ruck doesnt want to look at zuul logs for a few days please | 15:48 | |
| marios|ruck | how did we ever used to do this for 3 weeks i have no idea | 15:48 |
| marios|ruck | maybe it was easier "when i were a lad" ... | 15:48 |
| dasm|ruck | marios|ruck: we're gonna see after ysandeep will finish his 3-weeks | 15:48 |
| dasm|ruck | marios|ruck: earlier there was no "stream". maybe that's why? | 15:49 |
| marios|ruck | dasm|ruck: yea i was wondering if ysandeep did that on purpose and why | 15:49 |
| dasm|ruck | he did. i asked him. he just want to have it "old way" | 15:49 |
| dasm|ruck | marios|ruck: just noticed your email. have a great long weekend! | 15:50 |
| marios|ruck | thanks dasm|ruck | 15:50 |
| rlandy | marios|ruck: bye - have a good weekend | 15:52 |
| rlandy | ysandeep: pls pass to dasm|ruck or me | 15:52 |
| marios|ruck | rlandy: thanks you too | 15:53 |
| * dviroel lunch | 15:53 | |
| *** dviroel is now known as dviroel|lunch | 15:53 | |
| ysandeep | rlandy, in a debug session with Tengu atm.. will ping back | 15:53 |
| rlandy | ysandeep: going into manager meeting | 15:53 |
| marios|ruck | dasm|ruck: can you please reply to http://lists.openstack.org/pipermail/openstack-discuss/2022-May/028421.html & update later once https://review.opendev.org/c/openstack/tripleo-heat-templates/+/840683 is workflowed? | 15:56 |
| dasm|ruck | sure | 15:56 |
| rlandy | dasm|ruck: also - pls follow ysandeep on this - to see where he is changing the options | 16:00 |
| rlandy | will check in after meeting | 16:00 |
| dasm|ruck | k | 16:00 |
| rlandy | pls join call | 16:00 |
| dasm|ruck | which call? | 16:01 |
| *** marios|ruck is now known as marios|out | 16:01 | |
| ysandeep | rlandy, we are debugginf different issue - c8 train one | 16:01 |
| rlandy | soniya: pls also keep in the loop | 16:01 |
| rlandy | soniya: you around? | 16:01 |
| frenzy_friday | bot pls add to review list https://review.opendev.org/c/openstack/tripleo-ci/+/839323, https://review.opendev.org/c/openstack/tripleo-quickstart/+/839305 | 16:14 |
| reviewbot | I have added your review to the Review list | 16:14 |
| reviewbot | I have added your review to the Review list | 16:14 |
| ysandeep | Tengu++ | 16:17 |
| Tengu | rlandy: cloud-init update was the culprite! ysandeep++ | 16:17 |
| Tengu | and now I'm off :). | 16:17 |
| dasm|ruck | Tengu++ | 16:17 |
| ysandeep | ~~~ | 16:18 |
| ysandeep | < cloud-init-21.1-15.el8.noarch | 16:18 |
| ysandeep | > cloud-init-22.1-1.el8.noarch | 16:18 |
| ysandeep | ~~~ | 16:18 |
| rlandy | Tengu++ thanks for that debug | 16:18 |
| Tengu | np - my pleasure :). the machine-id and FirstBootCondition are out of the picture! | 16:18 |
| rlandy | ysandeep: train only? | 16:18 |
| ysandeep | rlandy, dasm|ruck finally we now what's affecting c8 train issue | 16:18 |
| rlandy | ysandeep: looking at the timeouts | 16:18 |
| rlandy | ysandeep: dasm|ruck: pls bug the train issue separately | 16:18 |
| rlandy | put the info in | 16:19 |
| rlandy | if we plan to lock or reverse? | 16:19 |
| * rlandy looking at timeouts | 16:19 | |
| ysandeep | rlandy, yes, that's the issue we debug today... I can check other one tomorrow | 16:19 |
| dasm|ruck | rlandy afair we have a bug for this already | 16:19 |
| rlandy | for the train specific? dasm|ruck: which one is train specific? | 16:19 |
| rlandy | ysandeep: I will confirm with cloudnull that I have the right settings | 16:20 |
| rlandy | will pass to you tomorrow | 16:20 |
| dasm|ruck | hmm.. no, we had cs9 master: https://bugs.launchpad.net/tripleo/+bug/1970400 | 16:20 |
| rlandy | ack | 16:20 |
| rlandy | no train specific | 16:20 |
| ysandeep | dasm|ruck, its late for me here.. Can you push a patch to pin/downgrade cloud-init. | 16:20 |
| dasm|ruck | so cs8 train is a new thing then? | 16:21 |
| dasm|ruck | ysandeep, i'll try. | 16:21 |
| dasm|ruck | not sure yet where it sits | 16:21 |
| rlandy | dasm|ruck: go to the job | 16:21 |
| rlandy | look where that is installed from | 16:21 |
| * rlandy will get | 16:21 | |
| dasm|ruck | k | 16:22 |
| * ysandeep re-enabling the cleanup script for ovb, and out for the day. | 16:23 | |
| rlandy | https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/790544f/logs/baremetal_0-console.log | 16:24 |
| rlandy | dasm|ruck: ^^ | 16:24 |
| ysandeep | rlandy, I will check with lmiccini tomorrow, if he is available to debug I will pull him as well. | 16:24 |
| rlandy | [ 73.380330] cloud-init[1494]: Cloud-init v. 22.1-1.el8 running 'init' at Thu, 05 May 2022 09:06:57 +0000. Up 72.97 seconds | 16:24 |
| rlandy | ysandeep: will chat with cloudnull about it | 16:24 |
| rlandy | I found some vars | 16:24 |
| rlandy | just checking if they are right | 16:24 |
| ysandeep | ack | 16:24 |
| *** ysandeep is now known as ysandeep|out | 16:28 | |
| rlandy | ysandeep: I only see cloud-init-22.1-1.el8.noarch | 16:28 |
| rlandy | ysandeep|out | 16:28 |
| ysandeep|out | rlandy, where? | 16:28 |
| rlandy | https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/790544f/logs/baremetal_0-console.log | 16:29 |
| rlandy | ysandeep|out: ^^ | 16:29 |
| rlandy | where are you diff'ing | 16:29 |
| ysandeep|out | rlandy, older good job | 16:29 |
| ysandeep|out | check last green run on train | 16:29 |
| dasm|ruck | this is passing job: https://review.rdoproject.org/zuul/build/461dc7789f57441aa2292c223f2b2d55 | 16:29 |
| rlandy | ah | 16:29 |
| dasm|ruck | it has 21.1-15.el8 | 16:30 |
| rlandy | you looked on the console logs as well | 16:30 |
| rlandy | got it | 16:30 |
| rlandy | ysandeep|out: have a good night | 16:30 |
| ysandeep|out | thanks! | 16:30 |
| rlandy | [ 10.722640] cloud-init[1027]: Cloud-init v. 21.1-15.el8 running 'init-local' at Wed, 27 Apr 2022 21:43:54 +0000. Up 9.85 seconds. | 16:30 |
| rlandy | got | 16:31 |
| ysandeep|out | abandoning https://review.opendev.org/c/openstack/diskimage-builder/+/840486 | 16:31 |
| rlandy | cloud-init.noarch 21.1-15.el8 @appstream | 16:32 |
| rlandy | dasm|ruck: see https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/461dc77/logs/overcloud-controller-0/var/log/extra/package-list-installed.txt.gz | 16:32 |
| rlandy | comes from centos 08 | 16:32 |
| rlandy | appstream | 16:32 |
| dasm|ruck | rlandy: what i meant was -- where to dowgrade/pin dependency to get it working. | 16:33 |
| dasm|ruck | currently i'm writing the bug to catch all of it | 16:33 |
| rlandy | dasm|ruck: you'd have to look at two things | 16:33 |
| dasm|ruck | actually, grepping through logs to pin point it | 16:33 |
| rlandy | the image we boot | 16:33 |
| rlandy | and the repos we lay down | 16:33 |
| ysandeep|out | dasm|ruck, fyi.. can try something like https://review.opendev.org/c/openstack/tripleo-quickstart/+/794737/1/config/release/tripleo-ci/CentOS-8/cloudsig/victoria.yml for appstream.. | 16:34 |
| ysandeep|out | but need to check first if cloud-init don't come in baseos itself - in that case - we have to downgrade cloud-init first | 16:35 |
| rlandy | ysandeep|out: may be on the image already | 16:35 |
| rlandy | dasm|ruck: you may need to downgrade | 16:35 |
| rlandy | dasm|ruck: seee http://mirror.centos.org/centos/8-stream/AppStream/x86_64/os/Packages/ | 16:36 |
| * ysandeep|out out for real | 16:36 | |
| dasm|ruck | working build -- last execution: 2022-05-03 22:40:42 (failed due to other issues) | 16:37 |
| rlandy | you'd need to exclude cloud-init-22.* | 16:37 |
| dasm|ruck | non-working: 2022-05-04 07:29:16 | 16:37 |
| rlandy | dasm|ruck: so you would have to check image builds | 16:37 |
| dasm|ruck | we tried to promote it yesterday | 16:37 |
| * rlandy gets | 16:37 | |
| rlandy | https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-full-train/3846db0/build.log | 16:38 |
| rlandy | dasm|ruck: ^^ | 16:38 |
| dasm|ruck | looking | 16:39 |
| rlandy | https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-full-train/3846db0/build.log | 16:39 |
| rlandy | image builds should also use repo setup | 16:39 |
| dasm|ruck | > 2022-05-05 06:29:10.045 | cloud-init noarch 22.1-1.el8 | 16:39 |
| rlandy | so you can try lock it there | 16:39 |
| rlandy | start with image builds | 16:39 |
| dasm|ruck | i'm gonna finish writing bug report | 16:39 |
| rlandy | https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-ironic-python-agent-train/dc45c07/build.log | 16:40 |
| rlandy | lock those | 16:40 |
| dasm|ruck | hmm... we have a promotion of cs8 train 3 days ago. why are we having new overcloud-hardened-full image if it didn't pass criteria of entire promotion? | 16:44 |
| dasm|ruck | (rhetorical question) | 16:44 |
| rlandy | dasm|ruck: irrelevant - issue is in integration line | 16:51 |
| rlandy | before promotion | 16:51 |
| rlandy | dasm|ruck: you want to run an OVB job with current train - fs001 .. but you want to build overcloud images in that job | 16:52 |
| rlandy | and you want to depends on a change to the release file to do two things | 16:52 |
| rlandy | exclude the versions you don't want | 16:52 |
| rlandy | and possibly downgrade | 16:53 |
| * rlandy gets vars | 16:53 | |
| rlandy | to_build: true | 16:54 |
| rlandy | dasm|ruck: https://github.com/rdo-infra/rdo-jobs/blob/2d4517815775546e105fcc9b05c641bde1f0a8e9/zuul.d/component-jobs-master-centos9.yaml#L741 | 16:56 |
| rlandy | you need that var | 16:56 |
| rlandy | dasm|ruck: start by creating a patch to exclude and test with ^^ | 16:59 |
| *** dviroel|lunch is now known as dviroel | 17:01 | |
| dasm|ruck | rlandy: fyi, all cs8 ovb jobs are affected. all overcloud-hardened-full images contain cloud-init 22.1 | 17:10 |
| * dasm|ruck is updating the bug descr | 17:10 | |
| dasm|ruck | brb | 17:17 |
| dviroel | dasm|ruck: rlandy: let me know if you folks need help with something | 17:18 |
| * dviroel see lot of messages | 17:18 | |
| rlandy | dasm|ruck: let's meet so you know what to test | 17:20 |
| rlandy | dviroel: you can join as well if you like | 17:20 |
| dviroel | ok | 17:20 |
| rlandy | dasm|ruck: dviroel: https://meet.google.com/ssk-ssai-eva | 17:22 |
| *** jpena is now known as jpena|off | 17:24 | |
| dasm|ruck | back | 17:31 |
| Tengu | rlandy, ysandeep|out i think cloudinit 21.4 introduced the issue. maybe a config thing leading to a regression. ill check that tomorrow during my morning | 17:39 |
| dasm|ruck | Tengu: 21.5 seems to be still correct. | 17:39 |
| rlandy | Tengu: we are testing with rebuilding the overcloud images for c8 releases | 17:40 |
| dasm|ruck | Tengu: https://bugs.launchpad.net/tripleo/+bug/1971751 | 17:40 |
| Tengu | oh? ok | 17:40 |
| rlandy | idk if c9 is impacted at all | 17:40 |
| dasm|ruck | *21.1-15 not 21.5 | 17:40 |
| Tengu | ah! | 17:40 |
| Tengu | yeah, 21.1 family seems fine | 17:40 |
| rlandy | so only 21.1 is ok | 17:40 |
| Tengu | they introduced changes in ssh and ssh-keygen in later releases | 17:41 |
| rlandy | Tengu: we see the same 22 version on c9 | 17:41 |
| Tengu | woops | 17:41 |
| rlandy | idk if that is imapcting tempest but it does ot fail deploy | 17:41 |
| Tengu | though i didnt hit the issue today when deploying | 17:41 |
| Tengu | ill check again tomorrow | 17:41 |
| rlandy | what release? | 17:41 |
| rlandy | Tengu> though i didnt hit the issue today when deploying | 17:42 |
| rlandy | ^^ train? | 17:42 |
| rlandy | master? | 17:42 |
| Tengu | master on 9 | 17:42 |
| rlandy | yeah - that only has tempest errors | 17:42 |
| rlandy | ad sometimes node provision | 17:42 |
| Tengu | ok | 17:43 |
| Tengu | maybe the issue is on sshd-keygen service, it checks for a cloudinit related service | 17:44 |
| Tengu | as said - ill dig a bit more tomkrrow. for now: Moon Knight on disney! | 17:44 |
| Tengu | see you folks | 17:44 |
| dasm|ruck | Tengu: thanks! have fun! | 17:44 |
| rlandy | dasm|ruck: https://review.rdoproject.org/r/c/testproject/+/36255 | 17:52 |
| rlandy | https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-8/promotion-testing-hash-train.yml#L137 | 17:52 |
| rlandy | https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-8/promotion-testing-hash-wallaby.yml#L155 | 17:53 |
| rlandy | dasm|ruck: dviroel: lunch ... brb | 18:09 |
| dasm|ruck | ack | 18:10 |
| rlandy | dasm|ruck: if by some chance,image do not build - ad override does not work on perioidc, we'll have to run image build jobs as well | 18:11 |
| rlandy | let's see | 18:11 |
| dasm|ruck | k | 18:11 |
| dviroel | "msg": "Depsolve Error occured: \n Problem 1: package libguestfs-1:1.44.0-6.module_el8.7.0+1140+ff0772f9.x86_64 requires libvirt-daemon-kvm >= 8.0.0, but none of the providers can be installed\ | 18:39 |
| dviroel | libvirt* | 18:39 |
| dviroel | which we are excluding for a while in the release file for C8\ | 18:40 |
| dviroel | dasm|ruck: rlandy ^ | 18:40 |
| dasm|ruck | dviroel: where is that? | 18:40 |
| dviroel | wrt to periodic-tripleo-centos-8-buildimage-overcloud-full-wallaby | 18:40 |
| dasm|ruck | i see | 18:41 |
| dviroel | https://logserver.rdoproject.org/55/36255/69/check/periodic-tripleo-centos-8-buildimage-overcloud-full-wallaby/c4f4af9/job-output.txt | 18:41 |
| dviroel | we should try the revert of those other excludes too - I think that we don't try that for a while now | 18:42 |
| dviroel | dasm|ruck: libvirt* exclude bug is https://bugs.launchpad.net/tripleo/+bug/1950916 | 18:43 |
| dasm|ruck | k | 18:43 |
| rlandy | you can try w/o | 18:43 |
| rlandy | or just test this with train for now | 18:43 |
| rlandy | just get one run to prove it | 18:44 |
| dasm|ruck | train is still running | 18:44 |
| dasm|ruck | but if it failed to build for overcloud-image, won't it fail for train as well? | 18:44 |
| dasm|ruck | train: https://review.rdoproject.org/zuul/stream/23fde2b1b67a402289586e7301b328db?logfile=console.log | 18:44 |
| rlandy | not excluded in train | 18:45 |
| dasm|ruck | right. it's wallaby-only excluded | 18:45 |
| dasm|ruck | i'm gonna wait few more on train to get it report. | 18:46 |
| dviroel | right | 18:46 |
| rlandy | besides ... overcloud-hardened-uefi-full | 18:47 |
| rlandy | we're probably using that | 18:47 |
| dasm|ruck | brb | 19:42 |
| rlandy | dasm|ruck: hmmm .... so far no success | 19:58 |
| rlandy | let's see the ones in rerun now | 19:58 |
| dasm|ruck | back | 20:01 |
| dasm|ruck | this one is currently introspecting overcloud: https://review.rdoproject.org/zuul/stream/23fde2b1b67a402289586e7301b328db?logfile=console.log | 20:02 |
| dasm|ruck | i should see the result in next few minutes | 20:02 |
| rlandy | dasm|ruck: looks like you still have cloud-init 22 | 20:10 |
| dasm|ruck | O/ | 20:10 |
| dasm|ruck | :/ | 20:10 |
| dasm|ruck | lemme see why it installed excluded version. maybe i made some mistake with it | 20:11 |
| rlandy | dasm|ruck: trying another version with image build | 20:15 |
| * rlandy thinks maybe the - is a problem | 20:15 | |
| dasm|ruck | link? | 20:15 |
| dasm|ruck | rlandy: link>? | 20:16 |
| rlandy | dasm|ruck: I am on the node | 20:16 |
| dasm|ruck | ah | 20:16 |
| rlandy | cat the image logs and grep | 20:16 |
| dviroel | dasm|ruck: rlandy: looks like quickstart roles does not load release file vars before calling oooci-build-images | 20:22 |
| rlandy | dviroel: not in this case | 20:23 |
| rlandy | trying with image builds directly | 20:23 |
| dasm|ruck | hm | 20:23 |
| dviroel | are we looking at periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train ? | 20:23 |
| dviroel | with to_build: true? | 20:24 |
| rlandy | yep | 20:24 |
| rlandy | if not, we will have to downgrade | 20:24 |
| dviroel | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/build-images/tasks/main.yml#L42-L53 | 20:25 |
| dviroel | there is no release files load - marios change was in a different flow | 20:25 |
| dviroel | marios change was here: https://opendev.org/openstack/tripleo-ci/src/branch/master/playbooks/tripleo-buildimages/run-v3.yaml#L5-L21 | 20:26 |
| dviroel | which load release files before building images | 20:26 |
| dviroel | thats my guess :) | 20:27 |
| dviroel | so, in image-build jobs it is working to exclude: https://logserver.rdoproject.org/55/36255/69/check/periodic-tripleo-centos-8-buildimage-overcloud-full-wallaby/c4f4af9/yum.repos.d/quickstart-centos-appstreams.repo | 20:33 |
| dviroel | but in OVB jobs with "to_build: true", the flow is different and release files may not be loaded - lets wait this fs001 finishes to check appstream.repo | 20:34 |
| dasm|ruck | yes, fs001 is still running | 20:34 |
| dviroel | mnaser> it looks like n-ovs-agent is just hanging, i was stracing trying to find the root cause when it decided to wake up | 20:40 |
| * dviroel needs to go out - dad on duty today | 20:43 | |
| * dviroel o/ | 20:43 | |
| *** dviroel is now known as dviroel|out | 20:43 | |
| dasm|ruck | dviroel|out: thanks. o/ | 20:44 |
| rlandy | ugh - keep missing review time | 20:51 |
| rlandy | dasm|ruck: dviroel|out; do you get pinged? | 20:51 |
| rcastillo | I don't | 20:53 |
| dasm|ruck | rlandy: i'm getting calendar pings. that's all | 20:53 |
| rcastillo | keep missing it as well | 20:53 |
| rcastillo | lunch brb | 20:57 |
| dasm|ruck | > "overcloud_deploy_result": "failed" | 21:12 |
| dasm|ruck | https://logserver.rdoproject.org/55/36255/69/check/periodic-tripleo-centos-8-buildimage-overcloud-full-wallaby/c4f4af9/overcloud-full.log | 21:20 |
| rlandy | dasm|ruck: I don;t think it worked on image build either | 21:21 |
| rlandy | we may have to downgrade | 21:21 |
| dasm|ruck | mhm. so basically like with "cmd", probably? | 21:21 |
| dasm|ruck | checking for similar thing | 21:21 |
| rlandy | added the downgrade | 21:34 |
| rlandy | will see if that helps | 21:34 |
| dasm|ruck | rlandy: isn't that downgrading on undercloud? | 21:36 |
| dasm|ruck | overcloud is fetched here: https://review.opendev.org/c/openstack/tripleo-quickstart/+/840755/2/config/release/tripleo-ci/CentOS-8/promotion-testing-hash-train.yml#b18 | 21:37 |
| rlandy | for the image build role | 21:38 |
| rlandy | it runs repo-setup before the image build | 21:38 |
| dasm|ruck | mhm | 21:39 |
| rlandy | we may need to downgrade it | 21:39 |
| rlandy | in image builds | 21:39 |
| rlandy | dasm|ruck: or better yet | 21:39 |
| rlandy | to virt-customize the overcloud images | 21:39 |
| rlandy | to downgrade it | 21:40 |
| dasm|ruck | i thought we're running jobs to downgrade overcloud-image | 21:40 |
| rlandy | it will - but only after repo setup runs | 21:40 |
| rlandy | so looking at train logs | 21:40 |
| rlandy | dasm|ruck: you are right - this won't help | 21:42 |
| rlandy | repo-setup id not run on overcloud yet | 21:42 |
| rlandy | we will need to change DIB | 21:42 |
| dasm|ruck | mhm | 21:42 |
| rlandy | or virt-customize | 21:42 |
| * rlandy check virt-customize | 21:43 | |
| rcastillo | can add virt-customize calls here https://github.com/openstack/tripleo-ci/blob/master/roles/oooci-build-images/tasks/main.yaml#L116 | 21:45 |
| rlandy | dasm|ruck: rcastillo: we need to virt-customize here: https://github.com/openstack/tripleo-ci/blob/master/roles/oooci-build-images/tasks/main.yaml | 21:46 |
| rlandy | yep | 21:46 |
| rlandy | we did this one before | 21:46 |
| rcastillo | yup | 21:46 |
| rcastillo | we should test if we can get rid of that workaround later too | 21:46 |
| rlandy | https://logserver.rdoproject.org/60/39960/33/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master/4d9d79e/logs/undercloud/var/log/tempest/stestr_results.html.gz | 21:47 |
| rlandy | at leats it was only one test fail this time | 21:48 |
| dasm|ruck | we had one successful test run > 2022-05-05 01:02:47 | 21:50 |
| dasm|ruck | two after that failed 1. overcloud; 2. undercloud. | 21:51 |
| rlandy | dasm|ruck: are working on the virt-customize? if not, I will | 21:54 |
| dasm|ruck | rlandy: i'm working on that atm | 21:55 |
| rlandy | dasm|ruck: k - I updated https://review.opendev.org/c/openstack/tripleo-quickstart/+/840755 to remove the downgrade | 21:57 |
| rlandy | you can reuse the testproject if you want | 21:57 |
| rlandy | dasm|ruck: pls leave notes for sandeep before your EoD | 21:57 |
| rlandy | I am copying vexxhost comments into rr page | 21:57 |
| dasm|ruck | k | 21:58 |
| dasm|ruck | rcastillo: did you think about this? https://review.opendev.org/c/openstack/tripleo-ci/+/840766 cc rlandy | 22:03 |
| rcastillo | dasm|ruck: that's the idea, yeah | 22:04 |
| dasm|ruck | k, starting project job to see if that's gonna help | 22:04 |
| rlandy | dasm|ruck; commenting | 22:05 |
| rlandy | should be for c8 only | 22:05 |
| rcastillo | rlandy: beat you to it ;) | 22:05 |
| dasm|ruck | i'm respinning your test to see if that's addressing the issue | 22:06 |
| rlandy | and probably needed on all images | 22:06 |
| rlandy | ipa as well | 22:06 |
| rcastillo | oh yeah that's true | 22:06 |
| dasm|ruck | rcastillo: if we don't have cloud-init installed, it complains about attempts to downgrade it. | 22:06 |
| dasm|ruck | not sure when we're getting cloud-init | 22:06 |
| rcastillo | hmm, the image dib builds should have it right? | 22:07 |
| dasm|ruck | good question | 22:07 |
| rlandy | dasm|ruck: also dnf downgrade -y cloud-init-21.15 | 22:08 |
| dasm|ruck | rlandy: i just answered it to rcastillo | 22:09 |
| rlandy | .el8 | 22:09 |
| rlandy | like dnf downgrade pcs-0.10.11-1.el8 -y; | 22:09 |
| rcastillo | could do `dnf upgrade || dnf install` to be sure? | 22:09 |
| rlandy | we know it's installed | 22:09 |
| rlandy | fom DIB | 22:09 |
| rcastillo | downgrade it is | 22:09 |
| rlandy | but the || is ok | 22:10 |
| rlandy | but DIB shows it installed | 22:10 |
| dasm|ruck | ok, updated: https://review.opendev.org/c/openstack/tripleo-ci/+/840766/2/roles/oooci-build-images/tasks/main.yaml | 22:11 |
| dasm|ruck | rcastillo: rlandy ^ | 22:11 |
| rlandy | tripleo_image_source | urlsplit('path') | basename | 22:13 |
| rlandy | dasm|ruck: ^^ what does that resolve to? | 22:13 |
| rcastillo | also we need the virt-customizes before we tar the image | 22:14 |
| rcastillo | otherwise they have no effect | 22:14 |
| dasm|ruck | (－‸ლ) | 22:15 |
| dasm|ruck | good call | 22:15 |
| dasm|ruck | rlandy: wrong copy. it should be "overcloud_image_customize" | 22:16 |
| rlandy | dasm|ruck: left comments | 22:17 |
| rlandy | also may need it on ipa as well | 22:17 |
| rlandy | rcastillo: ^^ thoughts? | 22:17 |
| rlandy | I think we do | 22:17 |
| dasm|ruck | how many different images do we build? 4? | 22:17 |
| rlandy | for c8 3 | 22:18 |
| rlandy | dasm|ruck: and that testproject won't work | 22:18 |
| dasm|ruck | why? | 22:18 |
| rcastillo | this is for booting right? Seems like ipa needs it too | 22:18 |
| rlandy | you;ll need to build all images I think | 22:18 |
| dasm|ruck | is it similar to 'config' one? | 22:18 |
| rlandy | no | 22:19 |
| rlandy | try build all the train images first | 22:19 |
| rlandy | and see if it works | 22:19 |
| rlandy | and upload | 22:19 |
| rlandy | then try ovb | 22:20 |
| dasm|ruck | rcastillo: i don't see qcow2 for ironic-python-client. there is only initramfs and kernel/ | 22:23 |
| rcastillo | dasm|ruck: you're right. We could try doing something else for that one, but I'd say we try and skate by with just the oc images | 22:28 |
| dasm|ruck | ++ | 22:28 |
| dasm|ruck | rcastillo: can you give it another check? https://review.opendev.org/c/openstack/tripleo-ci/+/840766/4/roles/oooci-build-images/tasks/main.yaml | 22:28 |
| dasm|ruck | does it look good? | 22:28 |
| rcastillo | you have my +1 :) | 22:30 |
| dasm|ruck | neat | 22:31 |
| dasm|ruck | rcastillo: https://www.youtube.com/watch?v=wvK-iKtkV70&t=200s | 22:32 |
| dasm|ruck | ^^ | 22:32 |
| rcastillo | exactly hahaha | 22:33 |
| rcastillo | hopefully this most of these c8 issues | 22:34 |
| rlandy | OMG | 22:35 |
| rlandy | fs001 master passed | 22:35 |
| rlandy | first time in ages | 22:35 |
| rlandy | dasm|ruck: rcastillo: stepping out for a bit - long day - bbl | 22:36 |
| *** rlandy is now known as rlandy|bbl | 22:36 | |
| dasm|ruck | rlandy|bbl: have a good night! | 22:55 |
| * dasm|ruck is leaving as well | 22:55 | |
| dasm|ruck | rcastillo: take care mate! have a good one! | 22:55 |
| *** dasm|ruck is now known as dasm|off | 22:55 | |
| rcastillo | I'll leave as well o/ | 22:58 |
| dasm|off | good. | 22:58 |
| dasm|off | see you tomorrow! | 22:58 |
| rcastillo | have a good one | 22:58 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!