Tuesday, 2021-01-05

*** tosky has quit IRC00:05
*** MentalSiege has quit IRC00:06
*** ociuhandu has joined #openstack-ironic00:07
*** pmannidi has quit IRC00:11
*** ociuhandu has quit IRC00:14
*** pmannidi has joined #openstack-ironic00:17
*** zzzeek has quit IRC00:30
*** zzzeek has joined #openstack-ironic00:32
jandersTheJulia regarding https://review.opendev.org/c/openstack/bifrost/+/762998 ( disabling inspector discovery by default in bifrost) would you still like me to send an email to ML to feel out what is the operator sentiment about this idea?00:39
*** arne_wiebalck has quit IRC00:53
*** arne_wiebalck has joined #openstack-ironic00:53
*** rcernin has joined #openstack-ironic01:17
*** gyee has quit IRC01:52
*** ociuhandu has joined #openstack-ironic01:55
*** gyee has joined #openstack-ironic02:03
*** ociuhandu has quit IRC02:08
openstackgerritMerged openstack/ironic-python-agent-builder master: Remove firmware from debian based systems  https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/76719202:23
*** rcernin has quit IRC02:26
*** zzzeek has quit IRC02:30
*** zzzeek has joined #openstack-ironic02:31
*** baha has quit IRC02:52
*** baha has joined #openstack-ironic02:53
*** rcernin has joined #openstack-ironic02:58
*** mkrai has joined #openstack-ironic03:04
*** rcernin has quit IRC03:10
*** rh-jelabarre has quit IRC03:15
*** zzzeek has quit IRC03:43
*** zzzeek has joined #openstack-ironic03:45
*** zzzeek has quit IRC03:53
*** zzzeek has joined #openstack-ironic03:55
openstackgerritJacob Anders proposed openstack/ironic master: Use OOB inspection to fetch MACs for IB inspection  https://review.opendev.org/c/openstack/ironic/+/75094304:14
openstackgerritJacob Anders proposed openstack/ironic-inspector master: Use OOB inspection to fetch MACs for IB inspection  https://review.opendev.org/c/openstack/ironic-inspector/+/75899404:15
*** zzzeek has quit IRC04:23
*** pmannidi has quit IRC04:24
*** zzzeek has joined #openstack-ironic04:25
*** mkrai has quit IRC05:16
*** mkrai_ has joined #openstack-ironic05:16
*** ociuhandu has joined #openstack-ironic05:32
*** ociuhandu has quit IRC05:38
*** gyee has quit IRC05:44
*** tzumainn has quit IRC06:01
openstackgerritJacob Anders proposed openstack/ironic master: Use OOB inspection to fetch MACs for IB inspection  https://review.opendev.org/c/openstack/ironic/+/75094306:13
openstackgerritMerged openstack/bifrost master: Disable inspector discovery by default  https://review.opendev.org/c/openstack/bifrost/+/76299806:13
openstackgerritJacob Anders proposed openstack/ironic-inspector master: Use OOB inspection to fetch MACs for IB inspection  https://review.opendev.org/c/openstack/ironic-inspector/+/75899406:25
*** zzzeek has quit IRC06:31
*** zzzeek has joined #openstack-ironic06:33
*** pmannidi has joined #openstack-ironic06:39
*** pmannidi has quit IRC06:40
*** pmannidi has joined #openstack-ironic06:42
*** zzzeek has quit IRC06:46
*** zzzeek has joined #openstack-ironic06:47
*** mkrai_ has quit IRC06:53
*** mkrai_ has joined #openstack-ironic06:58
*** ricolin has joined #openstack-ironic06:59
*** tosin has joined #openstack-ironic07:14
*** zzzeek has quit IRC07:17
*** ricolin has quit IRC07:18
*** zzzeek has joined #openstack-ironic07:19
*** abdysn has joined #openstack-ironic07:26
*** lbotell has quit IRC07:28
*** lbotell has joined #openstack-ironic07:30
openstackgerritMerged openstack/virtualbmc master: Add doc/requirements  https://review.opendev.org/c/openstack/virtualbmc/+/76916007:33
*** rpittau|afk is now known as rpittau07:33
rpittaugood morning ironic! o/07:33
*** Abdallahyas has joined #openstack-ironic07:36
*** abdysn has quit IRC07:40
arne_wiebalckGood morning rpittau and ironic!07:40
rpittauhey arne_wiebalck :)07:41
*** zzzeek has quit IRC07:45
*** zzzeek has joined #openstack-ironic07:45
*** zzzeek has quit IRC07:50
*** zzzeek has joined #openstack-ironic07:53
openstackgerritRiccardo Pittau proposed openstack/ironic-inspector master: Update version of doc8  https://review.opendev.org/c/openstack/ironic-inspector/+/76689608:00
*** abdysn has joined #openstack-ironic08:03
*** Abdallahyas has quit IRC08:06
*** zzzeek has quit IRC08:11
jandersgood morning rpittau arne_wiebalck o/08:11
rpittauhey janders :)08:11
*** zzzeek has joined #openstack-ironic08:11
*** akahat has joined #openstack-ironic08:13
*** zzzeek has quit IRC08:16
*** zzzeek has joined #openstack-ironic08:18
*** dougsz has joined #openstack-ironic08:24
*** Abdallahyas has joined #openstack-ironic08:35
*** abdysn has quit IRC08:39
*** tosky has joined #openstack-ironic08:39
*** zzzeek has quit IRC08:39
*** zzzeek has joined #openstack-ironic08:41
*** ociuhandu has joined #openstack-ironic08:49
arne_wiebalckhey janders o/08:50
*** lucasagomes has joined #openstack-ironic09:09
*** ociuhandu has quit IRC09:09
*** ociuhandu has joined #openstack-ironic09:10
*** zzzeek has quit IRC09:11
*** zzzeek has joined #openstack-ironic09:13
*** fmuyassarov has joined #openstack-ironic09:13
*** ociuhandu has quit IRC09:16
*** mkrai_ has quit IRC09:23
*** zzzeek has quit IRC09:26
*** zzzeek has joined #openstack-ironic09:27
*** ociuhandu has joined #openstack-ironic09:28
openstackgerritzhangtongjian proposed openstack/ironic-specs master: remove unicode from code  https://review.opendev.org/c/openstack/ironic-specs/+/76929409:29
*** mkrai_ has joined #openstack-ironic09:30
*** ociuhandu has quit IRC09:33
*** ociuhandu has joined #openstack-ironic09:36
*** ociuhandu has quit IRC09:42
*** tkajinam has quit IRC09:42
*** zzzeek has quit IRC09:45
openstackgerritVerification of a change to openstack/ironic-python-agent failed: CI: Remove l-c job, set ipa-b to 2.2.0  https://review.opendev.org/c/openstack/ironic-python-agent/+/76762209:47
*** zzzeek has joined #openstack-ironic09:47
*** derekh has joined #openstack-ironic09:54
*** abdysn has joined #openstack-ironic10:06
*** ociuhandu has joined #openstack-ironic10:09
*** Abdallahyas has quit IRC10:09
rpittauI spent some time checking the grenade job for ironic-inspector in stable/victoria10:12
rpittaufor some reason, when the node boots up, it loads the config from boot.ipxe and not from ironic-inspector.ipxe, so it gets the wrong kernel parameters10:12
rpittauit works fine in ussuri and master, if anyone has a minute to help, so far I couldn't find anything wrong :/10:12
*** tosin has quit IRC10:22
*** ociuhandu has quit IRC10:31
*** rpittau is now known as rpittau|bbl10:32
*** Qianbiao has joined #openstack-ironic10:35
*** mkrai_ has quit IRC10:39
*** ociuhandu has joined #openstack-ironic10:40
*** mkrai has joined #openstack-ironic10:41
*** ociuhandu has quit IRC11:13
*** ociuhandu has joined #openstack-ironic11:16
*** anuradha1904 has joined #openstack-ironic11:16
*** mkrai has quit IRC11:16
iurygregorygood morning Ironic11:26
iurygregoryrpittau|bbl, I will try to take a look11:26
*** zzzeek has quit IRC11:26
*** zzzeek has joined #openstack-ironic11:28
viks____my earlier error gone related to networking after changing the network to `flat`. Now when i try to provision the node using11:30
viks____`openstack baremetal node provide 6064e0ca-27fb-42a3-9562-b0317c743a4f`11:30
viks____i see it is stays in `clean wait` state:11:30
viks____https://www.irccloud.com/pastebin/TB41gjWA/11:31
viks____and in `nova-compute.log` i see:11:31
viks____`ERROR nova.compute.manager NoResourceClass: Resource class not found for Ironic node 6064e0ca-27fb-42a3-9562-b0317c743a4f`11:31
viks____i'm using `train` release of openstack.11:32
openstackgerritMerged openstack/ironic stable/victoria: Include HeartbeatMixin in the ramdisk deploy  https://review.opendev.org/c/openstack/ironic/+/76778611:32
viks____what resource class it is looking for.. how do i overcome this error? can someone guide me?11:33
*** zzzeek has quit IRC11:43
*** zzzeek has joined #openstack-ironic11:43
*** ociuhandu has quit IRC11:50
iurygregoryviks____, https://docs.openstack.org/ironic/train/install/configure-nova-flavors.html11:52
*** mkrai has joined #openstack-ironic11:53
viks____iurygregory: thnx.. will check.. but how do i change the state from `clean wait` to manage again?11:53
iurygregoryI think you would need to change, because you will need to set the resource class for the node11:59
viks____iurygregory: ok.. i'm able to bring it to `manageable` state.. i'll check that nw11:59
iurygregorycool12:00
*** abdysn has quit IRC12:03
*** Abdallahyas has joined #openstack-ironic12:03
*** zzzeek has quit IRC12:20
*** ociuhandu has joined #openstack-ironic12:20
*** mkrai has quit IRC12:21
*** abdysn has joined #openstack-ironic12:24
*** zzzeek has joined #openstack-ironic12:24
*** Abdallahyas has quit IRC12:25
*** tosin has joined #openstack-ironic12:28
*** ociuhandu has quit IRC12:30
arne_wiebalckviks____: the clean_wait issue and nova's resource class error should be unrelated12:31
arne_wiebalckviks____: nova does not like nodes which have no resource class set12:31
arne_wiebalckviks____: just set it to something and the error should go away12:31
*** ociuhandu has joined #openstack-ironic12:37
arne_wiebalckviks____: in addition, your node seems to have maintenance set to True; cleaning does not start when maintenance=True and the node will stay in clean_wait until it times out12:38
arne_wiebalckviks____: so, you may want to try to unset maintenance and relaunch cleaning12:39
*** ociuhandu has quit IRC12:41
*** ociuhandu has joined #openstack-ironic12:41
*** ociuhandu has quit IRC12:42
*** ociuhandu has joined #openstack-ironic12:44
*** rpittau|bbl is now known as rpittau12:44
rpittauhey iurygregory, thanks, I really need another set of eyes on that, I was checking the job in https://review.opendev.org/c/openstack/ironic-inspector/+/767536, as you can see it worked until Dec 1712:47
iurygregoryrpittau, ack o/12:47
rpittau I excluded some things, for example it doesn't seem to depend from the infra node, it fails on rax and ovh in the same way12:48
iurygregoryack12:53
*** rh-jelabarre has joined #openstack-ironic12:53
*** ociuhandu has quit IRC13:02
*** ociuhandu has joined #openstack-ironic13:05
*** ociuhandu has quit IRC13:10
*** ociuhandu has joined #openstack-ironic13:13
*** ociuhandu has quit IRC13:20
rpittauoh marvelous! Same issue with dd in vmedia in victoria....13:20
iurygregory"same" from grenade?13:21
rpittauno, sorry, same dd error, permission denied accessing /tmp13:21
iurygregoryoh ok13:26
iurygregory=)13:26
*** mkrai has joined #openstack-ironic13:26
rpittaugot that 2 weeks ago, thought it was fixed https://etherpad.opendev.org/p/hIPC_SjP_6XeZvevEzDZ13:27
*** Qianbiao has quit IRC13:30
iurygregory2021...13:30
*** Qianbiao has joined #openstack-ironic13:30
*** ociuhandu has joined #openstack-ironic13:31
*** ociuhandu has quit IRC13:31
*** ociuhandu has joined #openstack-ironic13:32
rpittau2020v213:32
*** ociuhandu has quit IRC13:38
*** ociuhandu has joined #openstack-ironic13:42
iurygregoryhehehe13:45
rpittauooook I think I found the fix for the dd issue13:47
viks____arne_wiebalck: i did those... also those resource class errors are gone nw... but again it stays in clean wait state.. i also see the below when i execute `openstack baremetal node show $NODE_ID`:13:48
viks____https://www.irccloud.com/pastebin/skocrt5e/13:48
openstackgerritRiccardo Pittau proposed openstack/ironic stable/victoria: Simplify injecting network data into an ISO image  https://review.opendev.org/c/openstack/ironic/+/76919413:48
viks____```maintenance_reason     | Timeout reached while cleaning the node. Please check if the ramdisk responsible for the cleaning is running on the node. Failed on step {}.```13:49
iurygregorywe forgot to backport? D;13:49
rpittauiurygregory: yep13:49
iurygregory+213:49
rpittaummmm we may need to merge a couple of backports, or I can disable vmedia temporary until the backport is merged13:50
rpittauprobably the latter is easier13:50
ajyaviks____: do you see that PXE boot is working? AFAIK there will be nothing in ironic logs if it fails, but cleaning times out.13:51
openstackgerritAija Jauntēva proposed openstack/ironic master: Add 'deploy steps' parameter for provisioning API  https://review.opendev.org/c/openstack/ironic/+/76835313:52
viks____ajya: Not sure hw to check that.. but i see the below error in neutron server log:13:54
viks____https://www.irccloud.com/pastebin/zog2uoVt/13:54
*** paras333 has joined #openstack-ironic13:55
openstackgerritRiccardo Pittau proposed openstack/ironic stable/victoria: Convert last bionic jobs to focal  https://review.opendev.org/c/openstack/ironic/+/76798913:55
rpittau^ let's make this pass and then merge the fix for the vmedia job, fingers crossed it works13:55
arne_wiebalckviks____: the node will need to get the image with the IPA via PXE, can you follow what the node does when ironic moves it into clean_wait (which is the moment Ironic has finished the setup and powers the node on) ... "follow" by looking at the console, for instance13:56
openstackgerritRiccardo Pittau proposed openstack/ironic stable/victoria: Convert last bionic jobs to focal  https://review.opendev.org/c/openstack/ironic/+/76798913:57
iurygregoryrpittau, ack13:57
arne_wiebalckviks____: and, yes, when cleaning times out, the node is set into maintenance (which you will need to remove every time this happens before retrying)13:57
ajyaviks____: that "failed to bind" error I get all the time, it's not indicating cause and clean/deploy still works for me. As Arne said check the console to see what the server is doing.14:01
viks____ajya: which console? or some logs? i do not have horizon installed at the moment14:02
ajyaviks____: I usually have direct access to BMC where I can connect to console.14:07
arne_wiebalckviks____: the KVM console, usually accessible via the BMC14:07
*** tzumainn has joined #openstack-ironic14:10
viks____arne_wiebalck:  ajya14:12
viks____ok... let me check if it is possible for me to check that console..14:12
viks____thnx14:12
*** zbitter is now known as zaneb14:12
arne_wiebalckviks____: the main point here is to check if the node boots and if it actually boots into the IPA14:13
viks____arne_wiebalck: ok14:13
*** tosky has quit IRC14:15
*** tosky_ has joined #openstack-ironic14:16
*** tosky_ is now known as tosky14:16
*** rloo has joined #openstack-ironic14:19
*** fmuyassarov has quit IRC14:31
viks____arne_wiebalck:  i see that booting in to ubuntu with login prompt...also there are some cloudinit errors...14:41
*** abdysn has quit IRC14:41
viks____i mean when i see it via KVM console...14:46
viks____also i had created user image using: `disk-image-create ubuntu baremetal dhcp-all-interfaces grub2 -o my-image`14:49
*** ociuhandu has quit IRC14:50
*** ociuhandu has joined #openstack-ironic14:50
arne_wiebalckviks____: did it boot into the image with the ironic python agent?14:53
*** Qianbiao has quit IRC14:53
*** Qianbiao has joined #openstack-ironic14:54
*** ociuhandu has quit IRC14:54
arne_wiebalckviks____: or the one which may have been on the disk?14:55
*** ociuhandu has joined #openstack-ironic15:03
*** MentalSiege has joined #openstack-ironic15:05
iurygregoryrpittau, if you have a chance can you look at https://review.opendev.org/c/openstack/sushy/+/769174 https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/769196 I'm trying  I've tested adding usedevelop = False for ipa-b and worked fine, on sushy it didn't work but CI is green in both...15:08
rpittauiurygregory: ipa-b is ok, what was the issue with sushy ?15:09
viks____arne_wiebalck: i guess it's not the one which is on disk because i could see cloudinit logs while booting up.. and deploy image i used from https://tarballs.opendev.org/openstack/ironic-python-agent/dib/files/ipa-centos7-stable-train.kernel15:14
iurygregoryif I understood dtantsur|afk comment is the fact we would need apidoc in req15:14
iurygregoryif we add usedevelop = False it would fail, but CI didn't complain15:15
rpittauI see, ok, we might need to add it then15:20
iurygregoryso sphinxcontrib-apidoc would need to go also in requirements.txt in the root dir?15:23
iurygregoryit fails locally if I add have usedevelop = False in tox.ini for docs15:24
*** Qianbiao has quit IRC15:24
iurygregorybut CI is green so I don't understand why we would need15:24
rpittaummm indeed we set usedevelop true in tox.ini for all the jobs, so I'm not sure why that should be necessary15:25
iurygregorysame here ¯\_(ツ)_/¯15:28
iurygregoryso I'm not alone in the boat \o/15:28
*** MentalSiege has quit IRC15:29
*** MentalSiege has joined #openstack-ironic15:30
rpittauyep, I don't see the reason, we'll have to wait for more comments15:31
*** yoctozepto has quit IRC15:43
*** yoctozepto has joined #openstack-ironic15:44
TheJuliagood morning everyone15:46
rpittaugood morning TheJulia :)15:46
TheJuliajanders: just a heads up is likely fine, fwiw since it has alreaddy merged15:48
iurygregorygood morning TheJulia =)15:51
* iurygregory brb lunch15:52
*** gyee has joined #openstack-ironic16:18
*** MentalSiege has quit IRC16:18
*** MentalSiege has joined #openstack-ironic16:19
arne_wiebalckviks____: ok, that is good news then :)16:28
arne_wiebalckviks____: can you log into the node? if you built the image yourself, you may need to rebuild it with your keys inside16:28
arne_wiebalckviks____: the idea would be see why the IPA does not call back home16:29
viks____arne_wiebalck: but node state is still in `clean wait`. i have not added any key inside the image as i have built it as per the above said command.. i'll try to rebuild the image with keys... how to do that.. isn't it possible to set it via cloud-init somehow?16:32
arne_wiebalckviks____: cloud-init can also create users and set passwords, but it needs a source where to get this information from16:33
arne_wiebalckviks____: for the IPA image, I think it is easier to bake user/pw into the image16:33
arne_wiebalckviks____: there is a DIB element for the keys ...16:34
arne_wiebalckviks____: is the node pingable at the moment?16:35
viks____arne_wiebalck: let me check16:39
*** lucasagomes has quit IRC16:51
viks____arne_wiebalck: hw do i know what ip got assigned?16:52
viks____`openstack baremetal node show 6064e0ca-27fb-42a3-9562-b0317c743a4f` not showing up any ip info.. also i tried to ping the ip range i gave in the subnet,  which did nt work16:56
viks____but i see:17:01
viks____https://www.irccloud.com/pastebin/xydJBBAg/17:01
*** mkrai has quit IRC17:03
viks____arne_wiebalck: anyway will try rebuilding the image tomorrow and see if i can login.. thanks17:06
*** rloo has quit IRC17:06
arne_wiebalckviks____: DHCP logs should have it, no? Ok, let me know how this story continues :)17:06
*** rloo has joined #openstack-ironic17:06
rpittaugood night! o/17:08
*** rpittau is now known as rpittau|afk17:08
*** rloo has quit IRC17:08
*** rloo has joined #openstack-ironic17:09
*** ociuhandu has quit IRC17:09
viks____arne_wiebalck: sure... all the 5 ip's i could not ping.. so there may be some other issue which i'll check in logs tomorrow..  thnx17:09
*** rloo has quit IRC17:12
*** rloo has joined #openstack-ironic17:12
*** rloo has quit IRC17:17
*** rloo has joined #openstack-ironic17:18
arne_wiebalckbye everyone o/17:21
*** ociuhandu has joined #openstack-ironic17:21
iurygregorybye arne_wiebalck o/17:23
*** ociuhandu has quit IRC17:26
*** dougsz has quit IRC17:29
*** rloo has quit IRC17:46
*** rloo has joined #openstack-ironic18:00
*** derekh has quit IRC18:02
openstackgerritMerged openstack/ironic master: Document using ramdisks with the ramdisk deploy interface  https://review.opendev.org/c/openstack/ironic/+/76739118:31
*** mgoddard has quit IRC18:45
*** ayoung has joined #openstack-ironic18:58
*** lbotell has quit IRC19:40
*** lbotell has joined #openstack-ironic20:25
*** lbotell has quit IRC20:30
*** lbotell has joined #openstack-ironic20:31
lbotellHi, I am running into an issue, after an instance is deployed, the target node boots up with the hostname "host-192-168-5-130" instead of the name set in the instance name, this also causes to get a login error, my guess is the keypair was not properly injected too, does anybody have any advice on this?20:40
lbotellTarget node is set with the properties: "boot_option:local,boot_mode:uefi" ; after deployment, I can see that the node boots up using local disks20:43
*** trandles has joined #openstack-ironic20:44
TheJulialbotell: so dhcp hostname is asserted via neutron... and if your using ironic in standalone, we don't do anything related to any of that.21:22
TheJulialbotell: w/r/t keypair, depends on how your deploying. I'm wondering if maybe config-drive is not being parsed?21:23
TheJulialbotell: well, good it is booting to local disks :) Unless there is an concern you have on your very last message21:23
openstackgerritMerged openstack/ironic stable/victoria: Convert last bionic jobs to focal  https://review.opendev.org/c/openstack/ironic/+/76798921:24
lbotellTheJulia I am using ironic (ussuri) deployed with kolla-ansible among the other core openstack services21:26
lbotellThing is the deployed image (CentOS8) is the same as the one that I am consuming in other cluster deployed with packstack and there (in packastack cluster) is working pretty fine21:27
lbotellTheJulia, I am using tiny-ipa-ussuri and I can not see any issue21:28
lbotellTheJulia, there are no error messages in the openstack services logs, that is why I am really confused about what is happening21:29
TheJuliaso I think it comes down to exactly how data is being submitted and the requests are being made21:29
TheJuliaeach intermediate layer is going to take different paths even if they are using the same underlying tools21:29
TheJulia"hi, give me a baremetal, oh I'm including this extra data because as an intermediate tool I was written to"21:30
TheJuliadoes that make sense?21:30
lbotellYes, that makes sense, all the instances are created via horizon.21:45
lbotellTheJulia, I can assume the boot loader installation has nothing to do with this problem, right?21:46
TheJuliaoh!21:46
TheJuliaso these are end user instances your requesting from horizon on each?21:46
TheJuliabootloader, umm.. no21:46
lbotellTheJulia, yes all end user instances are requested via horizon, virtual machines are working fine, they get the correct hostname and the keypair is installed properly, but for baremetal nodes, they don't21:49
TheJuliaAHH!21:51
TheJuliaokay21:51
TheJuliaso... it *sounds* like there is no metadata service21:52
TheJuliaNOR a config drive in one scenario21:53
TheJuliaso takea  look at https://docs.openstack.org/ocata/config-reference/compute/config-options.html and search for the option "force_config_drive"21:54
TheJuliait should be set to true for ironic to *always* get a configuration drive and embed it21:54
*** tosin has quit IRC22:07
*** tosky_ has joined #openstack-ironic22:12
*** tosky has quit IRC22:13
*** tosky_ is now known as tosky22:17
*** jamesdenton has quit IRC22:19
*** jamesden_ has joined #openstack-ironic22:19
lbotellTheJulia let me try that22:20
*** tkajinam has joined #openstack-ironic23:01
*** zzzeek has quit IRC23:25
*** zzzeek has joined #openstack-ironic23:27
*** rloo has quit IRC23:27
*** rcernin has joined #openstack-ironic23:33
*** rcernin has quit IRC23:53
*** rcernin has joined #openstack-ironic23:56
lbotellTheJulia it worked like a charm, Thanks a lot!!23:57
TheJulialbotell: seems like we need a troubleshooting docs entry23:57
lbotellYep, that would be really helpful23:58
*** rcernin has quit IRC23:59

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!