| *** spatel has quit IRC | 00:02 | |
| *** k_mouza has joined #openstack-nova | 00:12 | |
| *** k_mouza has quit IRC | 00:17 | |
| *** _alastor_ has quit IRC | 00:17 | |
| *** k_mouza has joined #openstack-nova | 00:20 | |
| *** k_mouza has quit IRC | 00:24 | |
| *** tbachman has joined #openstack-nova | 00:44 | |
| *** tbachman_ has joined #openstack-nova | 00:49 | |
| *** tbachman has quit IRC | 00:49 | |
| *** tbachman_ is now known as tbachman | 00:49 | |
| *** yedongcan has joined #openstack-nova | 01:20 | |
| *** gyee has quit IRC | 01:23 | |
| *** hshiina has joined #openstack-nova | 01:24 | |
| *** _alastor_ has joined #openstack-nova | 01:25 | |
| *** _alastor_ has quit IRC | 01:29 | |
| *** imacdonn has quit IRC | 01:38 | |
| *** imacdonn has joined #openstack-nova | 01:39 | |
| *** cfriesen has quit IRC | 02:18 | |
| *** mrsoul has joined #openstack-nova | 02:37 | |
| *** psachin has joined #openstack-nova | 02:49 | |
| *** mhen has quit IRC | 02:56 | |
| *** mhen has joined #openstack-nova | 02:57 | |
| *** cfriesen has joined #openstack-nova | 03:10 | |
| openstackgerrit | yenai proposed openstack/nova master: Rollback detaching when reserving volume is failed https://review.openstack.org/617861 | 03:17 |
|---|---|---|
| *** Bhujay has joined #openstack-nova | 03:53 | |
| *** udesale has joined #openstack-nova | 04:05 | |
| *** sridharg has joined #openstack-nova | 04:43 | |
| *** udesale has quit IRC | 04:50 | |
| *** lpetrut has joined #openstack-nova | 04:51 | |
| *** udesale has joined #openstack-nova | 05:02 | |
| *** efried has joined #openstack-nova | 05:13 | |
| *** lpetrut has quit IRC | 05:25 | |
| *** Bhujay has quit IRC | 05:29 | |
| *** _alastor_ has joined #openstack-nova | 05:33 | |
| *** poojaj has joined #openstack-nova | 05:33 | |
| *** efried has quit IRC | 05:36 | |
| *** _alastor_ has quit IRC | 05:38 | |
| *** moshele has joined #openstack-nova | 05:42 | |
| *** janki has joined #openstack-nova | 05:44 | |
| *** dave-mccowan has joined #openstack-nova | 05:48 | |
| *** moshele has quit IRC | 05:51 | |
| *** Dinesh_Bhor has joined #openstack-nova | 05:53 | |
| *** jichenjc has joined #openstack-nova | 05:53 | |
| *** dave-mccowan has quit IRC | 05:54 | |
| *** Dinesh_Bhor has quit IRC | 05:58 | |
| *** jding1_ has joined #openstack-nova | 06:04 | |
| *** spatel has joined #openstack-nova | 06:05 | |
| *** poojaj has quit IRC | 06:05 | |
| *** poojaj has joined #openstack-nova | 06:08 | |
| *** jackding has quit IRC | 06:08 | |
| *** poojaj has quit IRC | 06:10 | |
| *** spatel has quit IRC | 06:10 | |
| *** alex_xu has quit IRC | 06:11 | |
| *** alex_xu has joined #openstack-nova | 06:14 | |
| *** poojaj has joined #openstack-nova | 06:15 | |
| *** tetsuro has joined #openstack-nova | 06:30 | |
| *** threestrands has quit IRC | 06:34 | |
| *** ratailor has joined #openstack-nova | 06:35 | |
| *** cfriesen has quit IRC | 06:41 | |
| *** poojajadhav has joined #openstack-nova | 06:50 | |
| *** poojaj has quit IRC | 06:53 | |
| sean-k-mooney | the summit starts in 1 hour right? at least the keynote should? | 06:58 |
| sean-k-mooney | or it could have stared yesterday... | 06:58 |
| *** poojajadhav has quit IRC | 07:01 | |
| tetsuro | sean-k-mooney: The summit started yesterday. The keynote of Day 2 starts in 1 hour. | 07:17 |
| sean-k-mooney | tetsuro: hehe yes i started watching the youtube vids | 07:17 |
| sean-k-mooney | tetsuro: are you at the summit this time | 07:18 |
| tetsuro | Yes I am, but I didn't see you. | 07:18 |
| tetsuro | you aren't this time? | 07:18 |
| sean-k-mooney | no i choose not to travel this time as i have done quite alot of travel, at least for me already this year | 07:19 |
| tetsuro | I see. I hope I can see you next Denver :) | 07:19 |
| tetsuro | Anyway, gotta leave the hotel | 07:19 |
| sean-k-mooney | yes same, im hoping to be in denver. enjoy | 07:20 |
| tetsuro | bye. | 07:20 |
| *** tetsuro has quit IRC | 07:20 | |
| *** lpetrut has joined #openstack-nova | 07:21 | |
| *** brinzhang has joined #openstack-nova | 07:23 | |
| *** annp has joined #openstack-nova | 07:30 | |
| *** pooja_jadhav has joined #openstack-nova | 07:39 | |
| *** mriedem has joined #openstack-nova | 07:40 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add DownCellFixture https://review.openstack.org/614810 | 07:41 |
| *** pcaruana has joined #openstack-nova | 07:42 | |
| *** lpetrut has quit IRC | 07:44 | |
| *** rodolof has joined #openstack-nova | 07:58 | |
| openstackgerrit | Dan Smith proposed openstack/nova master: Add CellsV2 FAQ about API design decisions https://review.openstack.org/617898 | 07:59 |
| *** jichenjc has quit IRC | 08:01 | |
| *** cdent has joined #openstack-nova | 08:04 | |
| *** sahid has joined #openstack-nova | 08:05 | |
| *** rodolof has quit IRC | 08:05 | |
| *** rodolof has joined #openstack-nova | 08:05 | |
| mriedem | dansmith: this is pretty simple, something we talked about at the ptg to fix a few issues in scheduling to a specific node https://review.openstack.org/#/c/603585/ | 08:08 |
| *** lbragstad has joined #openstack-nova | 08:08 | |
| *** ratailor_ has joined #openstack-nova | 08:09 | |
| *** ratailor__ has joined #openstack-nova | 08:11 | |
| *** ratailor has quit IRC | 08:12 | |
| *** ratailor has joined #openstack-nova | 08:14 | |
| *** ratailor_ has quit IRC | 08:14 | |
| *** ratailor__ has quit IRC | 08:16 | |
| dansmith | mriedem: okay I +Wd the spec, but I wish it included more about the actual nova side that actually solves a problem instead of just a placement-side API that _could_ be used to solve a problem | 08:16 |
| *** Dinesh_Bhor has joined #openstack-nova | 08:17 | |
| sean-k-mooney | dansmith: the usecase we discussed at the ptg was related to numa affinity right | 08:18 |
| dansmith | sean-k-mooney: no | 08:19 |
| dansmith | sean-k-mooney: boot or live migration with a target host | 08:19 |
| sean-k-mooney | dansmith: the in_tree paramater? | 08:20 |
| *** ratailor has quit IRC | 08:20 | |
| mriedem | correct | 08:20 |
| mriedem | or rebuild with an image change | 08:20 |
| mriedem | or evac with a target host | 08:20 |
| mriedem | or cold migrate with a target host... :) | 08:21 |
| sean-k-mooney | on the in_tree is set to the compute node uuid | 08:21 |
| sean-k-mooney | ya that makes sense too | 08:21 |
| openstackgerrit | Merged openstack/nova-specs master: Spec: allocation candidates in tree https://review.openstack.org/603585 | 08:22 |
| *** ratailor has joined #openstack-nova | 08:22 | |
| *** erlon has joined #openstack-nova | 08:36 | |
| *** ygk_12345 has joined #openstack-nova | 08:40 | |
| mriedem | jroll: just some updates on https://review.openstack.org/#/c/609709/ and i'm happy | 08:45 |
| *** erlon has quit IRC | 08:48 | |
| *** jpena|off is now known as jpena | 08:54 | |
| openstackgerrit | sean mooney proposed openstack/os-vif master: always create ovs port during plug https://review.openstack.org/602384 | 08:58 |
| openstackgerrit | sean mooney proposed openstack/os-vif master: add isolate_vif config option https://review.openstack.org/612534 | 08:59 |
| openstackgerrit | sean mooney proposed openstack/os-vif stable/rocky: always create ovs port during plug https://review.openstack.org/609850 | 09:02 |
| *** markvoelker has joined #openstack-nova | 09:03 | |
| *** Dinesh_Bhor has quit IRC | 09:03 | |
| *** Dinesh_Bhor has joined #openstack-nova | 09:04 | |
| *** brinzhang has quit IRC | 09:10 | |
| *** pvradu has joined #openstack-nova | 09:13 | |
| *** ralonsoh has joined #openstack-nova | 09:16 | |
| *** Dinesh_Bhor has quit IRC | 09:21 | |
| *** pvradu has quit IRC | 09:23 | |
| *** pvradu has joined #openstack-nova | 09:23 | |
| *** markvoelker has quit IRC | 09:24 | |
| openstackgerrit | sean mooney proposed openstack/os-vif stable/rocky: add isolate_vif config option https://review.openstack.org/616285 | 09:31 |
| openstackgerrit | Ivaylo Mitev proposed openstack/nova master: VMware: Attach volumes using adapter type from instance https://review.openstack.org/616599 | 09:34 |
| *** _alastor_ has joined #openstack-nova | 09:35 | |
| *** lee2 has joined #openstack-nova | 09:38 | |
| *** lee2 is now known as lyarwood | 09:38 | |
| jroll | mriedem: yep, doing it this morning | 09:39 |
| *** _alastor_ has quit IRC | 09:39 | |
| *** derekh has joined #openstack-nova | 09:40 | |
| *** cdent has quit IRC | 09:43 | |
| *** Bhujay has joined #openstack-nova | 09:50 | |
| *** Bhujay has quit IRC | 09:51 | |
| *** Bhujay has joined #openstack-nova | 09:51 | |
| *** dave-mccowan has joined #openstack-nova | 09:54 | |
| mriedem | Kevin_Zheng: when you get a chance, can you update the root volume detach spec with the resize issue i told you about the other night? https://review.openstack.org/#/c/600628/ | 09:57 |
| Kevin_Zheng | mriedem: Oh the spec, go it, I was only thinking about the implementation | 09:58 |
| *** Dinesh_Bhor has joined #openstack-nova | 09:59 | |
| *** lbragstad has quit IRC | 10:03 | |
| mriedem | i just want to make sure it's recorded in the spec so we don't forget about it during review | 10:04 |
| mriedem | since the unshelve/start limitations are listed in the spec | 10:04 |
| openstackgerrit | Jim Rollenhagen proposed openstack/nova-specs master: Use conductor groups to partition nova-compute services for Ironic https://review.openstack.org/609709 | 10:05 |
| mriedem | resize revert/confirm would have the same limitations - fail if there is no root volume | 10:05 |
| jroll | mriedem: ^ should finally be good to go there now | 10:05 |
| *** cdent has joined #openstack-nova | 10:06 | |
| mriedem | ack | 10:07 |
| *** dave-mccowan has quit IRC | 10:07 | |
| mriedem | sean-k-mooney: if you want to follow along https://etherpad.openstack.org/p/BER-nfv-hpc-pain-points | 10:07 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Calculate port_id rp_uuid mapping for binding https://review.openstack.org/616239 | 10:08 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Pass allocations and traits to neturonv2 api https://review.openstack.org/616240 | 10:08 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send RP uuid in the port binding https://review.openstack.org/569459 | 10:08 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317 | 10:08 |
| sean-k-mooney | mriedem: yep i already had it open :) | 10:09 |
| *** ygk_12345 has quit IRC | 10:09 | |
| *** priteau has joined #openstack-nova | 10:14 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317 | 10:21 |
| mriedem | logan-: did you get that resize bug opened? | 10:26 |
| *** ccamacho has joined #openstack-nova | 10:27 | |
| *** ccamacho has quit IRC | 10:27 | |
| *** ccamacho has joined #openstack-nova | 10:27 | |
| *** priteau has quit IRC | 10:39 | |
| *** mriedem has quit IRC | 10:40 | |
| *** Dinesh_Bhor has quit IRC | 10:40 | |
| *** yedongcan has quit IRC | 10:41 | |
| sean-k-mooney | looks like there was a good conversation in https://etherpad.openstack.org/p/BER-nfv-hpc-pain-points | 10:44 |
| *** lbragstad has joined #openstack-nova | 10:49 | |
| *** alexchadin has joined #openstack-nova | 10:49 | |
| *** yedongcan has joined #openstack-nova | 10:50 | |
| *** yedongcan has left #openstack-nova | 10:55 | |
| *** udesale has quit IRC | 10:59 | |
| *** priteau has joined #openstack-nova | 11:02 | |
| *** pooja_jadhav has quit IRC | 11:05 | |
| *** dave-mccowan has joined #openstack-nova | 11:06 | |
| logan- | mriedem: https://bugs.launchpad.net/nova/+bug/1803331 | 11:13 |
| openstack | Launchpad bug 1803331 in OpenStack Compute (nova) "Root disk lost when resizing instance from imagebackend to rbd backed flavor" [Undecided,New] | 11:13 |
| *** alexchadin has quit IRC | 11:16 | |
| *** lbragstad has quit IRC | 11:18 | |
| lyarwood | logan-: https://review.openstack.org/#/c/578846/ might resolve that assuming this is a cause of us cleaning up aggresively after a failure, hard to tell from the trace alone. | 11:20 |
| sean-k-mooney | logan-: how do you have an RBD backed flavor? isnt the storage backend determined by the nova.conf not the flaovor? | 11:22 |
| lyarwood | sean-k-mooney: host aggregates | 11:23 |
| sean-k-mooney | lyarwood: right but just because you are afinities flaovrs to host aggrages and that chnages your network backend does not mean this is related to flaovrs | 11:24 |
| openstackgerrit | Sean McGinnis proposed openstack/nova master: Remove v1 check in Cinder client version lookup https://review.openstack.org/617927 | 11:24 |
| sean-k-mooney | this is simply a bug with rezieing between image backed nodes and rbd nodes | 11:25 |
| lyarwood | sean-k-mooney: imagebackend and yeah agreed I was just passing on what was described in the bug | 11:25 |
| *** mdbooth has joined #openstack-nova | 11:25 | |
| lyarwood | sean-k-mooney: this is all imagebackend, I'm not sure why logan- has called one out as imagebackend and the other as rbd | 11:25 |
| mdbooth | lyarwood: reading | 11:25 |
| sean-k-mooney | reading the bug it seams the issues is with migrating from imagebacked nodes to rbd backed nodes no? | 11:26 |
| logan- | Sorry if it is confusing. Yes these are different nodes with different storage backends configured in nova.conf | 11:27 |
| logan- | sean-k-mooney: correct | 11:27 |
| sean-k-mooney | i would assume the same behavior would exist if you just migrated and did not reszie | 11:27 |
| mdbooth | Sounds like there's 2 issues there | 11:27 |
| mdbooth | Firstly, resizing between compute hosts using different imagebackends isn't supported | 11:27 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Pass allocations and traits to neturonv2 api https://review.openstack.org/616240 | 11:28 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send RP uuid in the port binding https://review.openstack.org/569459 | 11:28 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317 | 11:28 |
| mdbooth | So, I guess that's not an issue, except that we could do better at preventing the attempt | 11:28 |
| mdbooth | I think sean-k-mooney mentioned that the other week. | 11:28 |
| logan- | mdbooth: right, I don’t expect it to be. I just expect it to fail and not destroy the instance :) | 11:28 |
| mdbooth | Secondly, we seem to have lost the root disk, which is extremely bad | 11:28 |
| *** dave-mccowan has quit IRC | 11:28 | |
| sean-k-mooney | logan-: i actully dont think its unresobaly to support we jsut dont today | 11:28 |
| mdbooth | sean-k-mooney: Conceptually it's not unreasonable, it's just that the imagebackend code is utterly awful. | 11:29 |
| * mdbooth has refactored it twice, but it didn't land. | 11:30 | |
| *** ratailor has quit IRC | 11:30 | |
| sean-k-mooney | i was going to suggest makeing https://bugs.launchpad.net/nova/+bug/1803331 more general and simply stating "live migration between image backend types fails with loss of the root disk." | 11:30 |
| openstack | Launchpad bug 1803331 in OpenStack Compute (nova) "Root disk lost when resizing instance from imagebackend to rbd backed flavor" [Undecided,New] | 11:30 |
| mdbooth | sean-k-mooney: This is resize | 11:31 |
| mdbooth | sean-k-mooney: Completely different code path. | 11:31 |
| sean-k-mooney | mdbooth: yes it is but it would also break on migreate right | 11:31 |
| mdbooth | Live migration might actually work. | 11:31 |
| sean-k-mooney | sorry cold migrate | 11:31 |
| *** priteau has quit IRC | 11:32 | |
| sean-k-mooney | cold migrate and resize would both triger this. live migrate who knows | 11:32 |
| mdbooth | logan-: At this point if you were a RH customer I'd ask you for sosreports. | 11:32 |
| mdbooth | Basically I'd want to see full logs from the initial error which caused the root disk to be deleted. | 11:32 |
| mdbooth | The subsequent error after the root disk has been deleted isn't all that interesting. | 11:33 |
| sean-k-mooney | mdbooth: in theory this would alow apply to any missmatch in image backend right e.g. any combinantion lvm or image or rbd where the backend changes. | 11:34 |
| mdbooth | sean-k-mooney: Yes. The code explicitly assumes that the layouts are identical on source and dest. | 11:34 |
| sean-k-mooney | so this seams like something we should add to the host state object and then check in the scheuler in the future | 11:35 |
| logan- | Ok I can pull more logs this afternoon. What would help? nova-compute in debug on both source & dest I’m guessing? | 11:35 |
| mdbooth | logan-: Yep, specifically for the resize operation itself, not just the subsequent failure to boot | 11:36 |
| logan- | Got it | 11:36 |
| logan- | Will do. Thanks | 11:36 |
| *** cdent has quit IRC | 11:36 | |
| mdbooth | logan-: If you look in the instance directory on the destination you should find the root disk there | 11:40 |
| mdbooth | logan-: Also, you should be able to revert resize | 11:40 |
| mdbooth | I don't think we've actually deleted the root disk here, it's just that the dest is looking for it in the wrong place. | 11:41 |
| sean-k-mooney | we might have deleted it on the source node | 11:42 |
| sean-k-mooney | or do we not do that until we confirm and your guessing on revert it tries to find it using the location form the dest | 11:43 |
| mdbooth | Not until confirm | 11:45 |
| mdbooth | Revert will use the copy which remains on the source | 11:45 |
| mdbooth | So actually I think this is just a case of extremely bad user experience. The operation isn't supported, but we could warn earlier rather than going down in flames. I don't believe any data has been lost. | 11:46 |
| mdbooth | logan-: If ^^^ is correct, no need for those logs, btw. | 11:47 |
| openstackgerrit | Merged openstack/nova stable/rocky: Set defult value of num_nvme_discover_tries=5 https://review.openstack.org/608683 | 11:56 |
| openstackgerrit | Martin Midolesov proposed openstack/nova master: Allow driver to specify switch&port for faster lookup https://review.openstack.org/617695 | 12:00 |
| *** ygk_12345 has joined #openstack-nova | 12:00 | |
| *** hshiina has quit IRC | 12:07 | |
| *** tbachman has quit IRC | 12:09 | |
| *** cdent has joined #openstack-nova | 12:12 | |
| *** priteau has joined #openstack-nova | 12:16 | |
| *** mriedem has joined #openstack-nova | 12:19 | |
| *** dave-mccowan has joined #openstack-nova | 12:26 | |
| *** ygk_12345 has quit IRC | 12:27 | |
| *** jpena is now known as jpena|lunch | 12:31 | |
| *** tbachman has joined #openstack-nova | 12:34 | |
| *** priteau has quit IRC | 12:36 | |
| *** tetsuro has joined #openstack-nova | 12:37 | |
| *** tbachman_ has joined #openstack-nova | 12:38 | |
| *** tbachman has quit IRC | 12:38 | |
| *** tbachman_ is now known as tbachman | 12:38 | |
| *** bnemec has joined #openstack-nova | 12:39 | |
| *** panda is now known as panda|rover | 12:42 | |
| *** Dinesh_Bhor has joined #openstack-nova | 12:47 | |
| openstackgerrit | Merged openstack/nova-specs master: Use conductor groups to partition nova-compute services for Ironic https://review.openstack.org/609709 | 12:52 |
| *** tetsuro has quit IRC | 12:57 | |
| *** lbragstad has joined #openstack-nova | 13:01 | |
| *** tetsuro has joined #openstack-nova | 13:01 | |
| *** moshele has joined #openstack-nova | 13:02 | |
| *** bnemec has quit IRC | 13:08 | |
| openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Use external placement in functional tests https://review.openstack.org/617941 | 13:08 |
| *** jaosorior has joined #openstack-nova | 13:14 | |
| *** Bhujay has quit IRC | 13:14 | |
| *** ThomasWhite has quit IRC | 13:16 | |
| *** _alastor_ has joined #openstack-nova | 13:17 | |
| *** moshele has quit IRC | 13:18 | |
| *** jaosorior has quit IRC | 13:18 | |
| *** ivaat has joined #openstack-nova | 13:18 | |
| logan- | updated the bug with additional details mdbooth | 13:19 |
| openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Use external placement in functional tests https://review.openstack.org/617941 | 13:19 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Calculate port_id rp_uuid mapping for binding https://review.openstack.org/616239 | 13:23 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Pass allocations and traits to neturonv2 api https://review.openstack.org/616240 | 13:23 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send RP uuid in the port binding https://review.openstack.org/569459 | 13:23 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317 | 13:23 |
| mdbooth | logan-: Can you retry the revert after setting instance state to 'resize_confirm' ? | 13:26 |
| *** udesale has joined #openstack-nova | 13:27 | |
| ivaat | hi i am seeking help for this: https://bugs.launchpad.net/nova/+bug/1801326 | 13:32 |
| openstack | Launchpad bug 1801326 in OpenStack Compute (nova) "CRITICAL keystonemiddleware.auth_token" [Undecided,New] | 13:32 |
| *** tetsuro has quit IRC | 13:32 | |
| *** tbachman has quit IRC | 13:34 | |
| *** jpena|lunch is now known as jpena | 13:37 | |
| *** tetsuro has joined #openstack-nova | 13:37 | |
| *** tbachman has joined #openstack-nova | 13:37 | |
| *** lbragstad has quit IRC | 13:38 | |
| openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Use external placement in functional tests https://review.openstack.org/617941 | 13:42 |
| *** skatsaounis has quit IRC | 13:45 | |
| *** skatsaounis has joined #openstack-nova | 13:46 | |
| logan- | mdbooth: I guess that will require database shenanigans? "openstack server set: error: argument --state: invalid choice: u'resize_confirm' (choose from 'active', 'error')" | 13:47 |
| mriedem | logan-: heh, reset state won't allow you to set it back to VERIFY_RESIZE | 13:50 |
| mriedem | so yes, db surgery | 13:50 |
| logan- | ok i'll give it a shot | 13:51 |
| *** ccamacho has quit IRC | 13:51 | |
| *** hongda has joined #openstack-nova | 13:51 | |
| *** openstackgerrit has quit IRC | 13:51 | |
| mriedem | because you need to get to this https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3506 | 13:51 |
| *** ccamacho has joined #openstack-nova | 13:53 | |
| ivaat | hi! i am seeking help for this: https://bugs.launchpad.net/nova/+bug/1801326 | 13:53 |
| openstack | Launchpad bug 1801326 in OpenStack Compute (nova) "CRITICAL keystonemiddleware.auth_token" [Undecided,New] | 13:53 |
| ivaat | i am sure it is bug | 13:54 |
| *** tetsuro has quit IRC | 13:55 | |
| *** tetsuro has joined #openstack-nova | 13:55 | |
| *** hongda has quit IRC | 13:55 | |
| *** zigo has joined #openstack-nova | 13:57 | |
| mriedem | ivaat: see my comment in the bug, it's not a bug, unless you can investigate and prove otherwise | 13:58 |
| mriedem | it looks like a misconfiguration | 13:58 |
| ivaat | mriedem, well i attached nova.conf as well. tried user with logging in with same user. works | 13:59 |
| ivaat | if it looks like misconfiguration then could you point out what part in config? i have gone trough config several times. followed official doc | 14:00 |
| mriedem | see my comment on the bug | 14:00 |
| ivaat | bug seems to be deleted | 14:00 |
| ivaat | sry going directly with bug link i can see it now | 14:01 |
| ivaat | mriedem, wtf. i just confirmed i have done this several times.. just take your time when helping. conf was attached to see what is wrong | 14:02 |
| *** dave-mccowan has quit IRC | 14:03 | |
| logan- | i tried hacking instances.vm_state='resized' and now it says "Instance has not been resized." so I guess there is some other state fields that need to be updated to hack it back to a revertable state | 14:03 |
| mriedem | ivaat: i looked at the paste in the bug, i didn't see [keystone_authtoken] in there | 14:05 |
| mriedem | logan-: so what did you do? vm was in error and you fixed the disk and tried to confirm again but that failed b/c it was in error rather than resized status? | 14:07 |
| ivaat | at moment when i installed was no guide to add keystone_authtoken | 14:07 |
| mriedem | ivaat: i don't know what install guide you were looking at | 14:07 |
| *** tetsuro has quit IRC | 14:07 | |
| mriedem | maybe something very old | 14:08 |
| *** aspiers[m] has joined #openstack-nova | 14:08 | |
| *** cfriesen has joined #openstack-nova | 14:09 | |
| ivaat | alright, i will test it out | 14:09 |
| ivaat | thx for now | 14:09 |
| aspiers[m] | mriedem: does it make sense to aim for a quick hallway discussion about SEV while we're here? sean-k-mooney too if you're around | 14:10 |
| ivaat | even nova conf is missing such keyword.. since it is section it should have been there | 14:10 |
| ivaat | i think i got previous version conf | 14:10 |
| *** moshele has joined #openstack-nova | 14:10 | |
| logan- | mriedem: ya, resize -> error -> unable to revert resize due to error state | 14:10 |
| sean-k-mooney | aspiers[m]: im not in berlin unfortunetly but im on irc | 14:11 |
| *** tetsuro has joined #openstack-nova | 14:11 | |
| aspiers[m] | sean-k-mooney: OK thanks. We can continue on the review of course, but feel free to ping me here too if you prefer | 14:12 |
| logan- | So looking at how to revert the resize and return it back to the source node where the disk is still present in /var/lib/nova/instances/<uuid>_resize/* | 14:12 |
| sean-k-mooney | if im needed i could dial in to a googel hangout but im sure there are enough people to sync up with to figure things out there without me | 14:12 |
| aspiers[m] | I'm currently liasing with our SEV engineers and AMD's to get more clarity around the questions you guys raised | 14:12 |
| *** spatel has joined #openstack-nova | 14:12 | |
| *** erlon has joined #openstack-nova | 14:13 | |
| aspiers[m] | guessing a hangout won't be needed but will certainly let you know if I'm wrong | 14:15 |
| *** moshele has quit IRC | 14:15 | |
| sean-k-mooney | aspiers[m]: ok just read your responces. if we can just document what will work and what wont then im more or less happy. | 14:15 |
| sean-k-mooney | aspiers[m]: im also fine with an fix constatin + vm ram + video ram | 14:16 |
| aspiers[m] | sean-k-mooney: absolutely - next patch set should give much more clarity over what is expected to work vs. not | 14:16 |
| sean-k-mooney | but if we go that way i would make the constant a config value | 14:16 |
| *** efried has joined #openstack-nova | 14:16 | |
| aspiers[m] | OK that sounds like a good idea as a short-term workaround until qemu exposes that calculation | 14:16 |
| aspiers[m] | so you mean a fixed configurable value for the overhead per VM, right? so <hard_limit> would be calculated as flavor ram plus that value | 14:17 |
| sean-k-mooney | the issue is qemu cant really know this up front as it depends on several things that can cahgne at runtime and from version to version | 14:17 |
| aspiers[m] | possibly with an adjustment if guest video ram is specified via extra specs or image props | 14:18 |
| *** _alastor_ has quit IRC | 14:18 | |
| *** openstackgerrit has joined #openstack-nova | 14:18 | |
| openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Use external placement in functional tests https://review.openstack.org/617941 | 14:18 |
| sean-k-mooney | aspiers[m]: ya it could be a config value for the over head or a flavor extraspec | 14:18 |
| sean-k-mooney | just some way to reduce or increase the 512 without code change if we mesuer it is too big or two small later | 14:19 |
| sean-k-mooney | so it would be ram + video ram + a configurable constat overhead | 14:19 |
| *** itlinux has joined #openstack-nova | 14:20 | |
| sean-k-mooney | that said i think we do this somewhere already | 14:20 |
| sean-k-mooney | aspiers[m]: we support setting the hardlimit in https://github.com/openstack/nova/blob/88951ca98e1b286b58aa1ad94f9af40b8260c01f/nova/virt/libvirt/config.py#L2102-L2133 and i belive we use it for realtime instances | 14:23 |
| *** lbragstad has joined #openstack-nova | 14:23 | |
| aspiers[m] | sean-k-mooney: ah yeah, I guess probably for the same reason (memory pinning) | 14:25 |
| *** awaugama has joined #openstack-nova | 14:25 | |
| sean-k-mooney | aspiers[m]: yes we use it so we can memlock the memory for realtime instaces. also all hugepage backed instace will also have the hugepages memlocked which we use for ovs-dpdk so it can do dma into the guest memory | 14:25 |
| sean-k-mooney | im trying not to be angry about the fact that apparently we only use this in unit tests.... http://codesearch.openstack.org/?q=hard_limit&i=nope&files=&repos=nova | 14:29 |
| aspiers[m] | haha :) | 14:30 |
| sean-k-mooney | aspiers[m]: well the code is there for you to use... | 14:31 |
| aspiers[m] | indeed. I don't think the actual coding is gonna be too hard | 14:34 |
| *** dklyle has quit IRC | 14:39 | |
| *** tetsuro has quit IRC | 14:44 | |
| *** tetsuro has joined #openstack-nova | 14:48 | |
| *** _alastor_ has joined #openstack-nova | 14:50 | |
| *** tbachman has quit IRC | 14:51 | |
| *** tbachman has joined #openstack-nova | 14:52 | |
| *** _alastor_ has quit IRC | 14:56 | |
| *** tetsuro has quit IRC | 14:56 | |
| *** spatel has quit IRC | 14:59 | |
| *** itlinux has quit IRC | 15:00 | |
| *** itlinux has joined #openstack-nova | 15:02 | |
| *** lbragstad has quit IRC | 15:04 | |
| *** mriedem has quit IRC | 15:06 | |
| *** lbragstad has joined #openstack-nova | 15:06 | |
| *** itlinux has quit IRC | 15:07 | |
| *** Luzi has joined #openstack-nova | 15:08 | |
| *** cdent has quit IRC | 15:09 | |
| *** N3l1x has joined #openstack-nova | 15:10 | |
| *** cdent has joined #openstack-nova | 15:12 | |
| *** erlon has quit IRC | 15:16 | |
| *** mriedem has joined #openstack-nova | 15:20 | |
| mriedem | dansmith: so belmiro gave me another issue to worry about, for which he's going to open a bug. they drop the allocation candidates limit down to 10 to make scheduling faster, but they also at times disable a large number of computes in a cell for $reasons (mnaser also does that for load balancing the cloud); the problem then is b/c of the low alloc candidates limit, placement gives back 10 disabled computes and gets novalidhost | 15:21 |
| mriedem | to workaround it he set reserved = total for inventory in placement on the disabled nodes, | 15:22 |
| mriedem | but that sucks for hundreds of vms | 15:22 |
| dansmith | ouch | 15:22 |
| mriedem | i suggested a pre-request placement filter to somehow filter out disabled nodes | 15:22 |
| mriedem | maybe forbidden trait/aggregate, idk | 15:22 |
| dansmith | hmm | 15:22 |
| dansmith | they disable compute or disable a cell? | 15:22 |
| dansmith | *computes | 15:23 |
| sean-k-mooney | mriedem: i was going to say yes tag with a trait and use forbindin traits but then we are using placmeent to sotre metadata | 15:23 |
| dansmith | mriedem: I think the right thing to do is to make placement results better so they don't need such a tiny result set to mitigate performance issues | 15:24 |
| dansmith | but, regardless of what that number is, the same problem could happen | 15:24 |
| dansmith | so if it's a disabled compute situation, I think having compute decorate its RP in some way is probably the thing to do there, | 15:25 |
| dansmith | although it generates a lot of chatty traffic from api->compute->placement | 15:25 |
| sean-k-mooney | stephenfin: bauzas by the way is there a nova cyborg session at the ptg? they demoed something that does not work upstream in the key note. im reviewing the spec again but it would be good to check in with them. | 15:25 |
| openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Use external placement in functional tests https://review.openstack.org/617941 | 15:25 |
| *** efried has quit IRC | 15:26 | |
| mriedem | i would really want to avoid needing to make placement api calls from the pre-placement filter for sure | 15:28 |
| mriedem | i.e. to rope disabled computes into a resource provider aggregate and then tag a forbidden aggregate filter on the request | 15:28 |
| dansmith | no, | 15:28 |
| dansmith | you can't really do that I don't think | 15:29 |
| mriedem | the compute tagging it's resource provider as disabled with a trait is one way yeah - not really the same for ironic (they set reserved = total) | 15:29 |
| dansmith | I meant have computes decorate their own RP with a trait or reserved=total | 15:29 |
| openstackgerrit | melanie witt proposed openstack/nova-specs master: Propose counting quota usage from placement and API database https://review.openstack.org/509042 | 15:29 |
| mriedem | sean-k-mooney: traits are going to be used as metadata in placement | 15:29 |
| mriedem | dansmith: right | 15:29 |
| mriedem | anyway, i told belmiro to open a bug so we don't lose track of the issue | 15:29 |
| mriedem | we also talked about the upt stuff in compute | 15:30 |
| dansmith | ack | 15:30 |
| mriedem | i think he's ok with what's going on in efried's series to clean that up and sighup on changes | 15:30 |
| mriedem | and my short history lesson on how we got here | 15:30 |
| mriedem | my reward was his other issues | 15:31 |
| *** erlon has joined #openstack-nova | 15:31 | |
| dansmith | heh | 15:34 |
| *** Luzi has quit IRC | 15:35 | |
| *** tetsuro has joined #openstack-nova | 15:37 | |
| sean-k-mooney | mriedem: ya traits are kind of metata but im not sure how peopel would fell about a mainantnce trait or whatever we would use to make the compute node RP as down in placemetn | 15:39 |
| dansmith | I would expect jaypipes to not want that | 15:40 |
| dansmith | since "cant do stuff" isn't a capability | 15:40 |
| dansmith | reserved=total is a lot more accurate | 15:40 |
| dansmith | "these resources aren't schedulable right now" | 15:40 |
| sean-k-mooney | yes that was my original concern with a trait | 15:40 |
| mriedem | yeah, plus that matches what the ironic driver already does today | 15:40 |
| dansmith | yep | 15:40 |
| sean-k-mooney | ya that does seem alot cleaner | 15:41 |
| mriedem | i'm not sure if that happens immediately or in a periodic? when the admin disables the compute service in the api, does the api set reserved=total for all inventory on the compute node resource provider? | 15:41 |
| mriedem | seems it should be immediately mirrored | 15:42 |
| dansmith | disable is an rpc call right? | 15:42 |
| mriedem | no | 15:42 |
| dansmith | because xen used to do stuff when you disabled it I think | 15:42 |
| dansmith | maybe I'm mixing in the maintenance mode call | 15:42 |
| mriedem | it's just a db change | 15:42 |
| mriedem | yeah i think yo uare | 15:42 |
| dansmith | okay, so I was going to say let the compute service do the disable, | 15:43 |
| dansmith | but same difference just having api do it | 15:43 |
| sean-k-mooney | mriedem: well renebaling it would need the virt driver to resore the old reserved value right | 15:43 |
| dansmith | it's more better to let compute do it I think, just because it's the one that owns the resources | 15:43 |
| dansmith | and the RP | 15:43 |
| mriedem | set_host_enabled was the thing you're thinking of | 15:43 |
| mriedem | in the os-hosts api | 15:43 |
| jaypipes | sean-k-mooney, dansmith, mriedem: I'm still reviewing Virgil's Aeneid^W^Wthe Cyborg-nova interaction spec. | 15:44 |
| dansmith | and that doesn't get called when disable? | 15:44 |
| sean-k-mooney | mriedem: what inventor was reserver=total set on by the way | 15:44 |
| mriedem | dansmith: no | 15:44 |
| mriedem | PUT /os-services/{compute_node_id} is just a db change on the services record | 15:44 |
| *** rodolof has quit IRC | 15:44 | |
| sean-k-mooney | jaypipes: im reviewing that currently too | 15:44 |
| sean-k-mooney | jaypipes: this one right https://review.openstack.org/#/c/603955/10/specs/stein/approved/nova-cyborg-interaction.rst,unified | 15:44 |
| dansmith | mriedem: okay, well, it's rather unfortunate to have the api muck with the inventory of the compute node, but mechanically it would work | 15:44 |
| jaypipes | sean-k-mooney: yes. that one. | 15:45 |
| dansmith | mriedem: un-disabling is a good thought though, since you'd really not want to do that from the api I think | 15:45 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Add API ref guideline for body text https://review.openstack.org/605628 | 15:45 |
| sean-k-mooney | jaypipes: i jsut pushed what i had it take a while to get through that end to end | 15:45 |
| mriedem | dansmith: you mean fixing the reserved value once the service is re-enabled? | 15:45 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (7) https://review.openstack.org/571992 | 15:45 |
| dansmith | mriedem: yeah | 15:46 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (8) https://review.openstack.org/571993 | 15:46 |
| mriedem | yeah i don't think we can do that from api, compute knows that info | 15:46 |
| mriedem | some of it is stored in the compute node record, but maybe not all inventory records | 15:46 |
| dansmith | mriedem: right.. also there's a potential race I guess, | 15:46 |
| mriedem | only ram/vcpu/disk | 15:46 |
| mriedem | yeah | 15:46 |
| mriedem | let the compute b/c the source of truth for inventory (unless the API reserves it all...?) | 15:46 |
| *** artom has joined #openstack-nova | 15:46 | |
| dansmith | mriedem: which is api racing with compute to update the inventory.. they could both use generation to catch the race, but probably better to just let one own it | 15:46 |
| dansmith | yeah | 15:46 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Add description of custom resource classes https://review.openstack.org/616721 | 15:47 |
| *** artom has quit IRC | 15:47 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in virt/test_block_device.py https://review.openstack.org/566153 | 15:47 |
| *** artom has joined #openstack-nova | 15:47 | |
| *** pvradu has quit IRC | 15:47 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: api-ref: Add descriptions for vol-backed snapshots https://review.openstack.org/615084 | 15:48 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Use links to placement docs in nova docs https://review.openstack.org/614056 | 15:48 |
| sean-k-mooney | jaypipes: are you at the summit by the way. you ususally only attend the ptgs as of late | 15:48 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove Placement API reference https://review.openstack.org/614437 | 15:48 |
| *** sahid has quit IRC | 15:48 | |
| jaypipes | sean-k-mooney: no. I'm not in Berlin | 15:48 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Transform compute_task notifications https://review.openstack.org/482629 | 15:49 |
| sean-k-mooney | jaypipes: ah ok same. its weird not being there but also i got to review that spec again | 15:49 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Use oslo_db.sqlalchemy.test_fixtures https://review.openstack.org/609352 | 15:49 |
| *** sahid has joined #openstack-nova | 15:49 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix best_match() deprecation warning https://review.openstack.org/611204 | 15:50 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix server query examples https://review.openstack.org/616834 | 15:50 |
| openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Use external placement in functional tests https://review.openstack.org/617941 | 15:51 |
| *** dklyle has joined #openstack-nova | 15:55 | |
| jaypipes | sean-k-mooney: hell of a reward... | 15:56 |
| *** lbragstad has quit IRC | 16:00 | |
| *** dave-mccowan has joined #openstack-nova | 16:03 | |
| mriedem | ha | 16:04 |
| *** cdent has quit IRC | 16:06 | |
| *** _alastor_ has joined #openstack-nova | 16:07 | |
| *** ccamacho has quit IRC | 16:07 | |
| *** sridharg has quit IRC | 16:08 | |
| *** irclogbot_2 has quit IRC | 16:08 | |
| *** mriedem has quit IRC | 16:10 | |
| *** mriedem has joined #openstack-nova | 16:12 | |
| *** tbachman has quit IRC | 16:15 | |
| *** irclogbot_2 has joined #openstack-nova | 16:16 | |
| *** dave-mccowan has quit IRC | 16:20 | |
| *** ccamacho has joined #openstack-nova | 16:23 | |
| *** tbachman has joined #openstack-nova | 16:24 | |
| *** tetsuro has quit IRC | 16:26 | |
| *** pcaruana has quit IRC | 16:29 | |
| *** k_mouza has joined #openstack-nova | 16:30 | |
| *** tbachman has quit IRC | 16:35 | |
| *** jding1_ has quit IRC | 16:35 | |
| *** ccamacho has quit IRC | 16:35 | |
| *** k_mouza has quit IRC | 16:36 | |
| *** udesale has quit IRC | 16:36 | |
| *** jackding has joined #openstack-nova | 16:36 | |
| *** dims has quit IRC | 16:37 | |
| *** dims has joined #openstack-nova | 16:41 | |
| *** tbachman has joined #openstack-nova | 16:41 | |
| *** k_mouza has joined #openstack-nova | 16:44 | |
| *** cdent has joined #openstack-nova | 16:46 | |
| *** k_mouza has quit IRC | 16:48 | |
| openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Use external placement in functional tests https://review.openstack.org/617941 | 16:50 |
| *** jmlowe has joined #openstack-nova | 16:52 | |
| *** mriedem has quit IRC | 16:53 | |
| openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Use external placement in functional tests https://review.openstack.org/617941 | 16:56 |
| *** psachin has quit IRC | 16:58 | |
| *** k_mouza has joined #openstack-nova | 16:58 | |
| *** Dinesh_Bhor has quit IRC | 17:02 | |
| *** k_mouza has quit IRC | 17:03 | |
| *** erlon has quit IRC | 17:04 | |
| *** pcaruana has joined #openstack-nova | 17:11 | |
| *** janki has quit IRC | 17:14 | |
| *** gyee has joined #openstack-nova | 17:14 | |
| *** mrjk_ has quit IRC | 17:16 | |
| *** tbachman has quit IRC | 17:21 | |
| *** k_mouza has joined #openstack-nova | 17:25 | |
| *** k_mouza has quit IRC | 17:29 | |
| *** k_mouza has joined #openstack-nova | 17:29 | |
| *** k_mouza has quit IRC | 17:30 | |
| *** k_mouza_ has joined #openstack-nova | 17:30 | |
| *** k_mouza_ has quit IRC | 17:35 | |
| openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Use external placement in functional tests https://review.openstack.org/617941 | 17:35 |
| *** cdent has quit IRC | 17:38 | |
| *** tbachman has joined #openstack-nova | 17:40 | |
| *** k_mouza has joined #openstack-nova | 17:43 | |
| *** panda|rover is now known as panda|rover|off | 17:47 | |
| *** k_mouza has quit IRC | 17:48 | |
| *** k_mouza has joined #openstack-nova | 17:54 | |
| *** k_mouza has quit IRC | 17:55 | |
| *** derekh has quit IRC | 18:04 | |
| *** k_mouza has joined #openstack-nova | 18:11 | |
| *** psachin has joined #openstack-nova | 18:12 | |
| *** jpena is now known as jpena|off | 18:16 | |
| *** k_mouza has quit IRC | 18:16 | |
| *** k_mouza has joined #openstack-nova | 18:20 | |
| *** k_mouza has quit IRC | 18:24 | |
| *** psachin has quit IRC | 18:25 | |
| *** tbachman has quit IRC | 18:26 | |
| *** sahid has quit IRC | 18:37 | |
| *** ralonsoh has quit IRC | 18:38 | |
| *** k_mouza has joined #openstack-nova | 19:05 | |
| *** k_mouza has quit IRC | 19:09 | |
| *** spatel has joined #openstack-nova | 19:45 | |
| spatel | sean-k-mooney: ^^ | 19:45 |
| spatel | if i have 8 vCPU on instance then its ok to set ethtool -L eth0 combined 8 ? | 19:46 |
| spatel | or i should set 4 (50% of number of cpu) | 19:46 |
| *** jmlowe has quit IRC | 19:50 | |
| *** pcaruana has quit IRC | 19:54 | |
| *** artom has quit IRC | 20:22 | |
| *** tetsuro has joined #openstack-nova | 20:23 | |
| *** tetsuro has quit IRC | 20:25 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Pass allocations and traits to neturonv2 api https://review.openstack.org/616240 | 20:32 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send RP uuid in the port binding https://review.openstack.org/569459 | 20:32 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317 | 20:32 |
| *** fyx has joined #openstack-nova | 20:40 | |
| *** artom has joined #openstack-nova | 20:55 | |
| *** k_mouza has joined #openstack-nova | 21:25 | |
| *** tbachman has joined #openstack-nova | 21:27 | |
| *** k_mouza_ has joined #openstack-nova | 21:29 | |
| *** k_mouza has quit IRC | 21:30 | |
| *** threestrands has joined #openstack-nova | 21:32 | |
| *** k_mouza_ has quit IRC | 21:33 | |
| *** k_mouza has joined #openstack-nova | 21:36 | |
| *** k_mouza_ has joined #openstack-nova | 21:39 | |
| *** k_mouza has quit IRC | 21:39 | |
| *** k_mouza_ has quit IRC | 21:43 | |
| *** ivaat has quit IRC | 22:02 | |
| *** markvoelker has joined #openstack-nova | 22:10 | |
| *** gokhan has quit IRC | 22:15 | |
| *** awaugama has quit IRC | 22:34 | |
| *** spatel has quit IRC | 23:02 | |
| *** dave-mccowan has joined #openstack-nova | 23:12 | |
| *** maciejjozefczyk has joined #openstack-nova | 23:13 | |
| *** maciejjozefczyk has quit IRC | 23:32 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!