Monday, 2018-09-03

openstackgerritTakashi NATSUME proposed openstack/nova master: Fix a failure to format config sample  https://review.openstack.org/59798600:16
*** brinzhang has joined #openstack-nova00:26
*** gbarros has joined #openstack-nova00:44
*** ircuser-1 has joined #openstack-nova00:57
*** jiapei has joined #openstack-nova01:06
*** slaweq has joined #openstack-nova01:11
*** slaweq has quit IRC01:16
*** rnoriega has quit IRC01:29
*** rnoriega has joined #openstack-nova01:30
*** brinzhang has quit IRC01:34
*** brinzhang has joined #openstack-nova01:35
*** Dinesh_Bhor has joined #openstack-nova02:15
*** hongbin has joined #openstack-nova02:26
*** lei-zh has joined #openstack-nova02:31
*** psachin has joined #openstack-nova02:42
*** brinzh has joined #openstack-nova02:43
*** bzhao__ has quit IRC02:46
*** brinzhang has quit IRC02:46
*** vivsoni has quit IRC02:53
*** threestrands has quit IRC02:57
openstackgerritBrin Zhang proposed openstack/nova master: Need further updates, no need to review  https://review.openstack.org/59927602:57
*** sambetts_ has quit IRC03:01
*** sambetts_ has joined #openstack-nova03:02
*** links has joined #openstack-nova03:08
*** slaweq has joined #openstack-nova03:11
*** slaweq has quit IRC03:16
*** Dinesh_Bhor has quit IRC03:18
*** hongbin has quit IRC03:19
*** Dinesh_Bhor has joined #openstack-nova03:30
*** gbarros has quit IRC03:32
openstackgerritChen proposed openstack/nova master: Fix SOFT_DELETE vm_state string  https://review.openstack.org/59848203:33
*** gbarros has joined #openstack-nova03:56
*** kevinbenton has quit IRC04:00
*** kevinbenton has joined #openstack-nova04:00
*** udesale has joined #openstack-nova04:02
*** Dinesh_Bhor has quit IRC04:03
*** udesale has quit IRC04:06
*** udesale has joined #openstack-nova04:06
*** udesale has quit IRC04:06
*** udesale has joined #openstack-nova04:08
*** rcernin has quit IRC04:18
*** rcernin has joined #openstack-nova04:18
*** jiapei has quit IRC04:26
*** hoonetorg has quit IRC04:31
*** tetsuro has joined #openstack-nova04:34
*** stakeda has joined #openstack-nova04:34
*** tetsuro has quit IRC04:38
*** tetsuro_ has joined #openstack-nova04:39
*** lei-zh has quit IRC04:46
*** abhishekk has joined #openstack-nova04:48
*** abhishekk has quit IRC04:51
*** Dinesh_Bhor has joined #openstack-nova05:01
*** gbarros has quit IRC05:04
*** slaweq has joined #openstack-nova05:11
*** slaweq has quit IRC05:15
*** lei-zh has joined #openstack-nova05:23
*** janki has joined #openstack-nova05:26
*** hamzy has quit IRC05:32
*** hamzy has joined #openstack-nova05:32
*** beagles has quit IRC05:35
*** bhagyashris has joined #openstack-nova05:36
*** tetsuro_ has quit IRC05:47
*** hoonetorg has joined #openstack-nova05:57
*** luksky has joined #openstack-nova06:13
*** hoonetorg has quit IRC06:14
*** hoonetorg has joined #openstack-nova06:15
*** prometheanfire has joined #openstack-nova06:29
prometheanfirehttps://github.com/openstack/nova/commit/50658eee4fe26a55854642119beeff46c0d0108a seems to be causing problems06:29
prometheanfire2018-09-03 01:29:14.620 11567 ERROR nova.objects.cell_mapping [req-0b912c05-d2bf-4e68-bcdf-7953c87dcc88 - - - - -] Failed to parse [database]/connection to format cell mapping: ValueError: expected ':' after conversion specifier06:30
prometheanfiretonyb: think I found another one ^06:30
*** moshele has joined #openstack-nova06:30
prometheanfirenot sure what's wrong with my db06:30
prometheanfireI'm guessing my password isn't being escaped06:30
*** bhagyashris has quit IRC06:31
tonybprometheanfire: can you run nova in a venv and import cell_mapping.py and check the parse results with your netloc?06:32
prometheanfiremaybe the root?06:32
prometheanfire2018-09-03 01:29:14.919 11567 ERROR oslo_db.sqlalchemy.exc_filters [req-0b912c05-d2bf-4e68-bcdf-7953c87dcc88 - - - - -] DBAPIError exception wrapped from (psycopg2.ProgrammingError) relation "projects" does not exist06:32
prometheanfiredon't have a venv atm06:32
tonybprometheanfire: venvs are cheap, but if you hav nova installed as a site package you can just use that06:33
prometheanfireanother thing to blame not testing on postgres?06:33
prometheanfireya06:33
tonybprometheanfire: No I doubt that a postgres thing06:33
prometheanfiremy password actually is fine for the one in question06:34
prometheanfireso probably not that06:35
*** Luzi has joined #openstack-nova06:35
tonybprometheanfire: :( Well I guess double check you're pointing at the right DB06:37
prometheanfireit worked before the upgrade06:39
prometheanfirenot seeing anything in upgrade nodes or deprication notes or https://docs.openstack.org/nova/rocky/user/upgrade.html denoting any change needed06:43
tonybprometheanfire: ... Okay but that doesn't mean it wasn't using a deprecated feature that has gone between queens and rocky06:43
tonybprometheanfire: Okay06:43
*** jaosorior has joined #openstack-nova06:51
*** sahid has joined #openstack-nova06:56
prometheanfireproblem seems to be with this06:57
prometheanfirehttps://github.com/openstack/nova/commit/50658eee4fe26a55854642119beeff46c0d0108a#diff-61f32a08cd0554ad482db124dcdc546aR12806:58
prometheanfirethe url.format is failing06:58
prometheanfirehttps://gist.githubusercontent.com/prometheanfire/c98816d477331bcc0a0272adb93cc5e2/raw/c746f1b1780f1202286b9902178ff4e613329f44/gistfile1.txt06:59
*** gnuoy has joined #openstack-nova07:05
*** dulek has joined #openstack-nova07:05
*** slaweq has joined #openstack-nova07:11
*** rcernin has quit IRC07:11
*** ykarel has joined #openstack-nova07:12
*** slaweq has quit IRC07:15
*** alexchadin has joined #openstack-nova07:24
prometheanfiretonyb: it IS my password07:25
*** ykarel is now known as ykarel|lunch07:28
*** Dinesh_Bhor has quit IRC07:35
*** Dinesh_Bhor has joined #openstack-nova07:40
*** jpena|off is now known as jpena07:43
openstackgerritZhenyu Zheng proposed openstack/nova-specs master: Make scheduling weight more granular  https://review.openstack.org/59930807:51
*** psachin has quit IRC07:57
openstackgerritZhenyu Zheng proposed openstack/nova-specs master: Make scheduling weight more granular  https://review.openstack.org/59930808:02
*** giblet_off has quit IRC08:09
*** holser_ has joined #openstack-nova08:10
*** ykarel|lunch is now known as ykarel08:13
prometheanfireok, the nova_api database has a non-escaped database_connection in the cell_mappings table08:14
prometheanfirewhich fixed THAT error at least08:15
*** cdent has joined #openstack-nova08:17
*** pas-ha has joined #openstack-nova08:26
*** derekh has joined #openstack-nova08:35
*** derekh has quit IRC08:36
*** derekh has joined #openstack-nova08:36
bauzasgood morning stackers08:38
*** psachin has joined #openstack-nova08:41
*** ttsiouts has joined #openstack-nova08:41
*** jchhatbar has joined #openstack-nova08:56
*** janki has quit IRC08:56
openstackgerritMatthew Thode proposed openstack/nova stable/rocky: Don't use '_TransactionContextManager._async'  https://review.openstack.org/59742108:57
*** janki has joined #openstack-nova08:59
*** jchhatbar has quit IRC08:59
*** pcaruana has joined #openstack-nova09:09
openstackgerritChen proposed openstack/nova master: Fix SOFT_DELETE vm_state string  https://review.openstack.org/59848209:11
*** tzumainn has joined #openstack-nova09:16
*** tssurya has joined #openstack-nova09:18
*** dtantsur|afk is now known as dtantsur09:19
*** slaweq has joined #openstack-nova09:20
*** lei-zh has quit IRC09:20
*** owalsh has quit IRC09:20
prometheanfirethis isn't helping, for some reason re-running the service isn't using the file I just changed, wtf09:23
prometheanfirehttps://gist.githubusercontent.com/prometheanfire/76b31b592b1780a72b32c289ab1bfb06/raw/b58c80c3d22860397781c53915d7350d6dbe0bab/gistfile1.txt09:24
prometheanfirebut async_ isn't defined at all on that line anymore, I editited it to be async09:25
prometheanfireanyone have any idea where that's cached?09:25
prometheanfireI've removed any pycache files I could find09:28
openstackgerritMerged openstack/nova master: libvirt: guest: introduce blockStats instead of domain.blockStats  https://review.openstack.org/52683309:30
*** ttsiouts has quit IRC09:30
*** owalsh has joined #openstack-nova09:33
*** mdbooth has joined #openstack-nova09:37
*** Dinesh_Bhor has quit IRC09:41
*** ttsiouts has joined #openstack-nova09:46
*** luksky has quit IRC09:51
openstackgerritSurya Seetharaman proposed openstack/nova master: Making instance listing skipping down cells configurable  https://review.openstack.org/59242809:55
*** stakeda has quit IRC09:56
*** Luzi has quit IRC09:59
openstackgerritSujeet Banerjee proposed openstack/nova master: WIP: Added a periodic task for network management.  https://review.openstack.org/59933810:00
*** Luzi has joined #openstack-nova10:15
prometheanfirewell, it looks like https://review.openstack.org/597421 may not be the right fix10:15
*** ttsiouts has quit IRC10:18
*** ttsiouts has joined #openstack-nova10:20
*** ttsiouts has quit IRC10:22
*** Dinesh_Bhor has joined #openstack-nova10:23
*** sahid has quit IRC10:30
*** luksky has joined #openstack-nova10:34
*** alexchadin has quit IRC10:37
*** ccamacho has joined #openstack-nova10:41
*** ccamacho has quit IRC10:41
*** alexchadin has joined #openstack-nova10:41
*** kaliya has joined #openstack-nova10:45
*** alexchadin has quit IRC10:52
*** udesale has quit IRC10:56
*** erlon has joined #openstack-nova10:56
*** sahid has joined #openstack-nova10:59
*** erlon_ has joined #openstack-nova11:00
*** dosaboy has joined #openstack-nova11:00
*** erlon has quit IRC11:03
*** ccamacho has joined #openstack-nova11:06
*** jpena is now known as jpena|lunch11:12
*** ttsiouts has joined #openstack-nova11:13
*** gibi has joined #openstack-nova11:22
gibio/11:23
*** tetsuro has joined #openstack-nova11:26
sean-k-mooneygibi: o/11:27
*** tetsuro has quit IRC11:28
openstackgerritMerged openstack/nova master: hardware: fix memory check usage for small/large pages  https://review.openstack.org/53216811:30
*** Dinesh_Bhor has quit IRC11:33
*** sahid has quit IRC11:40
*** sahid has joined #openstack-nova11:47
*** sahid has quit IRC11:48
*** sahid has joined #openstack-nova11:49
*** ccamacho has quit IRC11:50
*** ccamacho has joined #openstack-nova11:50
*** jpena|lunch is now known as jpena12:10
*** Bhujay has joined #openstack-nova12:14
*** davidsha has joined #openstack-nova12:18
*** amarao has joined #openstack-nova12:20
amaraoHow to access object attributes in policy json? I'd like to forbid some operations based on flavor. Is someone tried that before?12:20
sean-k-mooneyamarao: as in based on the flavor of an instance you want to change what api actions are allowable and or by whom?12:26
sean-k-mooneyamarao: that sound like a nightmare from an interoperablity stand point so i doubt that has been tried/tested before12:27
*** jaosorior has quit IRC12:29
amaraoYep, I want to use  flavor of the instance (or another instance attribute).12:30
*** ykarel is now known as ykarel|away12:30
amaraoBasically, I want to lock instance. I can use nova lock for this, but user still have access to console, and I want to remove it.12:30
*** dtantsur is now known as dtantsur|brb12:32
moshelesean-k-mooney: hi12:34
*** ttsiouts has quit IRC12:35
moshelesean-k-mooney:  in intel driver if spoofchk  is enabled does it allow to set the mac address of the vf to 00:00:00:00:00:0012:35
*** ttsiouts has joined #openstack-nova12:36
*** ttsiouts has quit IRC12:38
*** ttsiouts has joined #openstack-nova12:38
*** udesale has joined #openstack-nova12:39
*** janki has quit IRC12:39
*** pcaruana has quit IRC12:52
sean-k-mooneymoshele: i am not sure. if it did i would expect that to mean if the dest/source mac is not 00:00:00:00:00:00 specically drop the packet e.g. only broadcast traffic would be allowed12:58
sean-k-mooneymoshele: also at least in niantic (x520/x510) cards i know the mac filter is before the vlan filter so if you had 2 vf with that adress that would cause issues13:02
sean-k-mooneyamarao: locking in nova is intended to prevent api acess to an instance not to prevent users form using it13:03
sean-k-mooneyamarao: so a user can still poweroff a locked instance by sshing in and executing power off if they choose too.13:04
openstackgerritSurya Seetharaman proposed openstack/nova master: Making instance/migration listing skipping down cells configurable  https://review.openstack.org/59242813:05
sean-k-mooneyamarao: is there a partical usecase you have in mind. perhaps a dedicated feature would be better then a flavor hack.13:05
amaraosean-k-mooney, in our scenario we want to run it with our own key (user have no private key from it). 'Hidden' key plus 'lock' almost lock instance from user. What is left is console: Ctrl-Alt-Del, Sysrq-b, and (most importantly) grub access at boot time.13:06
*** jaosorior has joined #openstack-nova13:06
amaraoI'd like to take away console from user as well (for specific instances only).13:06
*** janki has joined #openstack-nova13:08
moshelesean-k-mooney: the reason I am asking is that mellanox driver block this and it can cause issues. So the mac address 00:00:00:00:00:00  is just to allow libvirt to clean vf when the  VM with VF  is deleted  see https://www.redhat.com/archives/libvir-list/2017-March/msg00455.html13:10
moshelesean-k-mooney: but now if spoofchk  enabled first nova(libvirt delete the vm) restoring the mac to 00:00:00:00:00:00   and then the sriov agent will disable the spofchk13:11
moshelesean-k-mooney: so the vf mac will not set to 00:00:00:00:00:00  and we can't reuse the mac on the same compute13:12
moshelesean-k-mooney:  I wonder how intel driver behavior to undestand were to fix it, in the driver or in openstack ...13:12
*** ykarel|away has quit IRC13:12
sean-k-mooneymoshele: i really need to get a redhat account at somepoint.. i cant read that linke but anyway. i dont think setting 00:00:00:00:00:00 is a good idea on intel cards either13:14
sean-k-mooneymoshele: i belive that the intel driver generates unique macs automaticaly for the pfs13:14
sean-k-mooney* vfs13:14
sean-k-mooneypersonally the fact that libvirt is seting 00:00:00:00:00:00 on deleteing the vm sound like a libvirt bug to me13:16
sean-k-mooneyor a nova bug if we are doing that in the libvirt vir driver13:16
moshelesean-k-mooney: basically it set the restore the previously  used mac so  if you auto generate it. it will restore the auto generate one. in mellanox case the vf is start with  00:00:00:00:00:00 so we allowed to restore to 00:00:00:00:00:0013:18
sean-k-mooneyamarao: well would a flavor extra-spec for disableing the console on an instace work?13:18
amarao sean-k-mooney are there such property for extra-spec?13:19
sean-k-mooneymoshele: right in anycase when the vf is detached from a instance its linkstate should be down so no trafic will travers it so does it matter if spoof checking is disabled13:19
sean-k-mooneyamarao: ill check you can chages several aspecs of the vm like this via extra-sepcs and or image metadata.13:20
amaraoIf I could disable console via flavor specs, it would be cool.13:22
sean-k-mooneyamarao: you can change the type of gpu in the guest  and ram https://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt-image.json#L45-L6213:23
moshelesean-k-mooney: right, but I think the default links state is auto, (at least for Mellanox)13:23
sean-k-mooneyyou can also set the serial port count https://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt.json#L18-L2313:23
sean-k-mooneyamarao: we could add a display count extra spec13:23
sean-k-mooneyamarao: or extend hw_video_model to allow none to signel no display13:24
amaraoNodisplay wouldn't work with many OSes.13:24
amaraoBut disable_vnc (or disable_access) will be cool.13:25
*** brinzh has quit IRC13:25
sean-k-mooneyamarao: hum the perhaps we should leave the display and allow disabling of the vnc/spice console that is attached to it instead?13:25
*** ttsiouts has quit IRC13:26
*** ttsiouts has joined #openstack-nova13:26
sean-k-mooneyamarao: atleast in terms of libvirt/kvm the vnc/spice console is a sperate device form the gpu/display so we could disable it speraate ly or leave it here but limit its acess to local scope so you cannot connect to it via the novnc_proxy13:27
amaraoFor the best user expirience it's better to return error on get-vnc-console, saying 'VNC console is disabled for this instance'.13:28
amarao... And admins will love that they still can connect to VNC through local IP on libvirt host.13:28
amaraoShould I report wishbug into launchpad?13:29
sean-k-mooneyamarao: we could proably do that based on the extraspec via an api check. amarao this would be a good topic for a blueprint care to write one13:29
sean-k-mooneyamarao: yes. a bug or sepcless blueprint.13:29
*** dtantsur|brb is now known as dtantsur13:30
amaraoI'll start from a bug. I never created a blueprint, so I'll write down a bug and then will try with blueprint.13:30
*** ttsiouts has quit IRC13:31
sean-k-mooneyamarao: ok sound good. feel free to add me to the bug if you like. am will you be attending the PTG next week. if not i can highlight this as an RFE13:31
*** Altabay has joined #openstack-nova13:32
*** Altabay has quit IRC13:34
*** med_ has joined #openstack-nova13:34
amaraoShould I put it into 'hw' namespace?13:34
*** psachin has quit IRC13:34
sean-k-mooneyam you dont have to detail the impmentation in the bug but yes i would think so13:35
amaraoI thought about something like 'hw:disable_consoles: vnc,spice,serial'13:35
*** Luzi has quit IRC13:36
amaraoOh, it's simpler than I thought. https://blueprints.launchpad.net/nova/+spec/flavor-based-access-to-console13:36
sean-k-mooneyamarao: perhapes invert that to hw:consoles=<one of vnc|spice|serial|None>13:37
amarao.. and if this field is absent, everything is permitted. ack.13:38
sean-k-mooneyamarao: yes blueprints are baseicelay jsut what is the problem you would like to fix13:38
sean-k-mooneyamarao: yep if the field is not there just do waht we do today13:38
sean-k-mooneyamarao: i dont think we actully allow more then one console currently hence the one of but that is something we could figure out as we start looking at the implementaion13:39
sean-k-mooneye.g. i dont think you can have an instance with both spice and vnc enabled13:39
amaraoack, updated.13:41
amaraoIt's more about serial/vnc thing, actually.13:41
sean-k-mooneyya serial likely can be mix with vnc as they attach do different virualised hardware13:42
sean-k-mooneyrdp/vnc/spice all share teh virutal gpu13:42
amaraoAnyway, we are still on mitaka (sad, but true), so I'll continue to search some kind of dirty hack for that. Thank you for help.13:43
sean-k-mooneythe other think about inverting it is we could model what consoles are available on each host as traits in the placement api and use this extraspec if present to land on a host that is able to support that console13:43
stephenfinAfternoon, gibi. Think this is something you'd be happy reviewing? https://review.openstack.org/#/c/595592/13:44
sean-k-mooneyamarao:  no worries, you may be able to use policies but that is not a area im familar with unfrotuetly.13:45
amarao... policy.json is like a path in a swamp. Everyone knew it exist, but rarely someone passes through. :)13:46
gibistephenfin: sure, looking13:49
*** jaosorior has quit IRC13:53
*** takashin has joined #openstack-nova13:56
*** Bhujay has quit IRC13:58
gibistephenfin: https://review.openstack.org/#/c/595592/ looks good overall. I'm wondering how we can tests this other than in the unit test that is in the patch13:58
stephenfingibi: I'm not sure how we could do it in the gate due to the hardware dependencies. I do have an SR-IOV machine locally that I could validate the fix with, if you'd like13:59
stephenfingibi: Downstream we will likely look at automating this as part of the whitebox-tempest-plugin Tempest plugin, but how we actually run those tests is still very much in the air14:00
* stephenfin should talk to cfriesen at the PTG about how they ran (run?) their internal CI for Titanium/StarlingX14:01
bauzasgiven we're on a US holiday today, I guess we won't have any scheduler meeting?14:02
gibistephenfin: if it does not take more than 2 hours of your time to locally test it then could you please run that test? It would give sizeable confidence boost to me towards this patch14:02
stephenfingibi: No problem, should only take me 20 minutes14:03
gibibauzas: cdent asked it in openstack-placement where I said it is OK to me to skip14:03
gibistephenfin: thanks a lot14:03
* stephenfin cracks knuckles and SSHs to local box14:03
bauzask14:03
*** ttsiouts has joined #openstack-nova14:04
moshelestephenfin: Mellanox CI was update to test this14:07
*** r-daneel has joined #openstack-nova14:09
*** links has quit IRC14:10
*** ttsiouts has quit IRC14:10
*** takashin has left #openstack-nova14:11
sean-k-mooneystephenfin: gibi moshele  if you are refering to https://review.openstack.org/#/c/595592/3 i think that is a pretty safe change14:16
gibisean-k-mooney: yes, we are talking about that. If stephenfin's timeline is correct he will have test results in 5 minutes anyhow :)14:18
*** sapd1_ has joined #openstack-nova14:19
*** helenafm has joined #openstack-nova14:20
sean-k-mooneystephenfin: if not i can prep an sriov env if needed but ill have to do an os reinstall on the node so it will be more like 2-3 hours before ill be able to check this directly.14:20
sean-k-mooney* or rather if you have issues14:20
stephenfinsean-k-mooney: Think I'm all good. Will known in 60 seconds :P14:23
sean-k-mooneystephenfin: i take it you worked out your sriov issues then?14:23
*** moshele has quit IRC14:24
stephenfingibi: Without the fix http://paste.openstack.org/show/729332/14:24
stephenfinsean-k-mooney: Sure did. It was a really stupid typo14:24
stephenfinsean-k-mooney: and those warnings didn't seem to affect anything.14:25
sean-k-mooneyha they are always the best typos14:25
sean-k-mooneystephenfin: ya they dont14:25
sean-k-mooneylibvirt is  just unhappy that a netdev does not exist for the device14:25
stephenfinsean-k-mooney: I'm not sure why they're warnings actually and not debug (or removed entirely). Must ask moshele that14:25
stephenfinsean-k-mooney: No, we emit those logs14:25
sean-k-mooneystephenfin: in that case we must be trying to read stuff via ip or sysfs that we should not assume is available14:26
stephenfinsean-k-mooney: Yeah, this is what I see http://paste.openstack.org/show/729333/14:27
sean-k-mooneystephenfin: in gerearal we assumt that pci device of type pf or vf are network devices which is not always correct14:27
*** r-daneel has quit IRC14:28
stephenfingibi: and with the fix http://paste.openstack.org/show/729334/14:28
sean-k-mooneyya this is likely coming form the code that tryis to get the netdev feature flags14:28
sean-k-mooneywe shold only do that if a netdev exists for the vf. i dont see why this should be a warning instead of debug as you said14:29
sean-k-mooneystephenfin: do you have the queue lenght set in your conf?14:29
*** r-daneel has joined #openstack-nova14:30
sean-k-mooneystephenfin: if not can you add it an restart the n-cpu agent and try one more time with both vnic-type=macvtap and vnic-type=direct14:30
stephenfinsean-k-mooney: nope http://paste.openstack.org/show/729336/14:31
sean-k-mooney? that is the domain xml for a direct passhtoug device14:32
sean-k-mooneydo you have the queue lenght set in /etc/nova/nova-cpu.conf14:33
sean-k-mooneyif so then that is correct for direct but we should see it populated in the driver section for macvtap14:33
stephenfinsean-k-mooney: Oh, good point. 02:00.0 is the PF. That's a 'direct' attach14:34
stephenfinUsed 'openstack port create --network 84b18250-c0d3-4594-92d1-19328fb37da5 --vnic-type direct sriov-port'14:35
sean-k-mooneywell direct could also be a vf14:35
stephenfinBased on the address, that's the PF. I probably have my whitelist set up wrong14:35
sean-k-mooneystephenfin: yes with that commandline we do not expect the domain xml to contain the queulenght14:36
sean-k-mooneythe fact its a pf or vf is irelevent in this case14:36
stephenfinCool. So that's working as expected14:36
sean-k-mooneyyes but you have not confirms if you set the queue lenght in your config14:36
stephenfinThe macvtap route is interesting though. I'm seeing this14:36
stephenfinsean-k-mooney: Oh, sorry. Yes, I have set it14:37
sean-k-mooneyok cool. so for macvtap what is the result?14:37
stephenfinand as a result, got this error (without the fix) http://paste.openstack.org/show/729332/14:37
stephenfinsean-k-mooney: For macvtap, I'm seeing http://paste.openstack.org/show/729339/14:38
stephenfinso it turns out those warnings are a big deal :)14:38
stephenfinand I didn't think they were simply because it was the PF being attached instead of the VF14:38
sean-k-mooneystephenfin: is the pf still bound?14:39
sean-k-mooneye.g to a vm14:39
stephenfinNo, I killed that VM14:39
stephenfinThere are no VMs running14:39
sean-k-mooneyyou should be able to use the pf for the macvtap if you delete the neturon port that is currectly using it14:39
sean-k-mooneyactullly if you deleted the vm that might be enough14:40
sean-k-mooneystephenfin: it makes sense however if the netdev does not exist then you cant create a macvtap from it14:40
stephenfinsean-k-mooney: Nope, same error after deleting the two ports :/14:42
stephenfinYeah, it does. I've no idea how to create said netdevs though14:42
sean-k-mooneystephenfin: can you try deaclloating the vf by echoing 0 into sriov_num_vf then restart the n-cpu agent so that only the pf is available14:42
sean-k-mooneystephenfin: the pf shows up with ifconfig/ip link right?14:43
stephenfinsean-k-mooney: Yup, along with its VMs. It's in up state14:43
sean-k-mooneystephenfin: VMs?14:44
stephenfinVFs, sorry14:44
stephenfinHeh, "-su: echo: write error: Function not implemented"14:44
* stephenfin goes to find MLNX docs on how to do this with their tools14:44
sean-k-mooneystephenfin: you could just change your whitelisnt to onely have 02:00:00 address14:45
openstackgerritsahid proposed openstack/nova stable/rocky: hardware: fix memory check usage for small/large pages  https://review.openstack.org/59941314:45
sean-k-mooneystephenfin: gibi anyway it sounds like the patch is working as expect in terms of fixing the direct case. the macvtap case likely will work if the nic was correctly configured14:47
*** jiapei has joined #openstack-nova14:47
* gibi was pulled to a discussion locally. reading back now14:50
openstackgerritsahid proposed openstack/nova stable/rocky: libvirt: Always escape IPv6 addresses when used in migration URI  https://review.openstack.org/59941914:51
*** ttsiouts has joined #openstack-nova14:52
gibistephenfin, sean-k-mooney: thanks for the extra check. I'm a lot more confident to +A that patch14:55
*** helenafm has quit IRC14:59
*** gbarros has joined #openstack-nova15:04
*** luksky has quit IRC15:05
bauzasstephenfin: I was looking at https://review.openstack.org/#/c/588570/215:06
bauzasstephenfin: maybe because it's changing a lot of tech debt, that's why people wonder if that's a feature backport15:07
bauzaswhile not, of course15:07
bauzasafter all, the change is hairy, you know15:07
bauzasand https://review.openstack.org/#/c/588570/2/nova/objects/numa.py is a bit concerning15:07
stephenfinbauzas: Yeah, I get that. Unfortunately it's pretty much unavoidable though :(15:08
stephenfinbauzas: Is it? None of those functions are stored in a DB so there's no issues around versioning15:09
bauzasstephenfin: yup, we're not changing the RPC API15:10
sean-k-mooneystephenfin: well we could greatly simply that code by rewriting it in a declarive/fuctional style with filters and transforms instead of the imperitive stily that is currently used for numa affinity15:10
bauzasstephenfin: I meant, when someone provides a backport change asking to modify an o.vo object, we need to look more15:11
* bauzas just trying to explain why you don't have a lot of reviews15:11
sean-k-mooneyso the complexity is not entirely due to the problem statement however that is not backportable in anyway15:11
bauzasand why it's taking me a long time to look at it15:11
stephenfinbauzas: Yup, that's fair :) I figured as much15:11
bauzassean-k-mooney: well, it would be an only-stable change, then ?15:12
bauzasif so, that's difficult15:12
kashyapHey folks, can anyone do a small test for me: Boot a Nova instance with Q35 machine type, and post the complete guest XML somewhere, please15:12
stephenfinHmm, I wonder how I can resolve the issue, in that case. We could stick on more sellotape but I haven't even thought how I'd do that15:12
kashyapYou can set Q35 in two ways: either vai the Nova config: `hw_machine_type=x86_64=q35`, or on the Glance image image: `openstack image set --property hw_machine_type=x86_64=q35 Fedora28Template`15:12
stephenfinkashyap: What properties would I need to set to do that?15:12
stephenfinAh :)15:12
sean-k-mooneybauzas: rewriting the numa code to be fucntional instead of imperitve. i was talking generally not for this specific patch15:13
stephenfinsean-k-mooney: oh, gotcha15:13
sean-k-mooneystephenfin: for this patch the only object change i see is a new property15:13
stephenfinYeah, all that code is crufty af. I might look to take that "track CPUs in placement" spec from jaypipes this cycle, assuming he's oversubscribed as usual15:13
stephenfin...though I don't really want to wade into the great placement war of 201815:14
kashyapstephenfin: Thank you!15:14
*** med_ has quit IRC15:15
sean-k-mooneystephenfin: well we could clean up that code without ever touching placement but thats a different topic. e.g. keep existing functionality jsut do it differently15:15
bauzassean-k-mooney: oh well, sure we could do it15:16
bauzasor use Haskell :p15:16
kashyapstephenfin: I need to head to my Dutch class (starts afresh) in a few; when you get to it, post it here, I'll take a look.  Gracias!15:16
bauzas(FWIW, I *love* Haskell)15:16
sean-k-mooneystephenfin: bauzas anyway back to https://review.openstack.org/#/c/588570/2 the only object change there is a syntasied property that will now be included in teh serialisation of the object so i think this is backportable as there is no rpc change right?15:16
kashyap(s/starts afresh/new level starts today)15:17
bauzassean-k-mooney: I feel so15:17
sean-k-mooneybauzas: that explains a lot ....15:17
* sean-k-mooney takes a step back before bauzas explains what a monad is15:17
bauzasvery simple15:18
stephenfinbauzas: I studied Haskell in NZ and adored it (helped me grasp recursion at long last). Alas, it's not something I've had the chance to work with since15:18
stephenfinProlog, on the other hand. Ick15:18
sean-k-mooneystephenfin: python has really good fuctional programing supprot built in by they way.15:19
bauzassean-k-mooney: since we're adding a new property, that's not chaning what we pass over RPC15:19
bauzassean-k-mooney: my only concern is on the left side15:19
*** mlavalle has joined #openstack-nova15:19
bauzasie.what we remove15:19
stephenfinkashyap: Here's the output of 'virsh dumpxml' http://paste.openstack.org/show/729345/15:20
bauzasanyway, time for wrapping off15:20
*** amotoki has joined #openstack-nova15:20
kashyapstephenfin: Nice, /me clicks15:20
stephenfinkashyap: and here are the commands I ran to get there http://paste.openstack.org/show/729348/15:21
bauzastoday was school's back for my daughters, so I want to take time to discuss with them15:21
bauzasso, calling it a day \o15:21
sean-k-mooneybauzas: as you should. o/15:21
stephenfinkashyap: That's using commit 90cdf807503f94bfbd3ae9a62a9bfe466a18bcad (so about two month old cold)15:21
stephenfinbauzas: Enjoy o/15:21
kashyapstephenfin: Nice.  Can I ask one more, since you have the env?  Also related to PCIe15:21
stephenfinkashyap: shoot15:21
kashyapstephenfin: Set the 'num_pcie_ports=2' in nova.conf, restart the Compute service, and again boot an instance afresh, and post the guest XML15:22
*** gbarros has quit IRC15:22
kashyapThanks a for the quick response :-)15:22
stephenfinkashyap: num_pcie_ports ? That's not a nova.conf option, is it?15:23
sean-k-mooneykashyap: wait why is that a nova conf setting and not a flavor extra spec15:23
stephenfinOh, what do you know, so it is15:23
kashyapsean-k-mooney: Hehe, I haven't checked if there's a flavor extra spec for it or not.15:23
sean-k-mooneystephenfin: really that sucks.15:24
*** gbarros has joined #openstack-nova15:24
sean-k-mooneystephenfin: that is going to make live migration even more of a pain15:24
stephenfinsean-k-mooney: I think it's like TX/RX queue size. The expectation would be that every node would have the same value (managed by Ansible or whatever other tool you're using)15:24
stephenfinkashyap: With or without the q35 flag?15:25
sean-k-mooneystephenfin: i think setting this suff in the nova conf is a serios antipatern15:25
kashyapstephenfin: With, please.15:25
kashyapstephenfin: Also - PCIe goodies are 'q35' _only_, BTW.15:25
sean-k-mooneystephenfin: that kind of assumtion make upgrading a pain as once set you can never safly change it15:25
kashyapsean-k-mooney: Before we forget, perhaps we should file a bug to revisit the PCIe root ports thingie?15:26
*** udesale has quit IRC15:26
* kashyap now really need to step out; will read the scrollback later15:26
*** gbarros has quit IRC15:26
sean-k-mooneywhich one. the auto allocation of a pcie route per numa node?15:26
*** gbarros has joined #openstack-nova15:27
* sean-k-mooney i have so many blueprints / specs to file... i likely wont have time to work on any of them :(15:27
stephenfinkashyap: http://paste.openstack.org/show/729349/15:28
stephenfinsean-k-mooney: Fortunately for you, I have zilch on my plate rn. Send em my way15:28
sean-k-mooneyill create an ether pad with them the issue is i have not got around to filing/writing them all.15:29
sean-k-mooneystephenfin: im not sure that is correct for num_pcie_ports=215:31
*** dtantsur is now known as dtantsur|afk15:32
sean-k-mooneystephenfin: your machine type is pc-i440fx-bionic so you are not enableing q35 chipset so no pcie at all15:33
stephenfinI don't think so either. I was expecting to see a <pci model='pcie-root-port'> element15:33
stephenfinSorry - <controller type='pci' model='pcie-root-port'>15:34
stephenfinSo why didn't that property get applied...15:35
sean-k-mooneywell currently everything in that xml is connect to the pci bus not pcie. also as i sad you are not using the q35 chipset so no pcie is available15:35
stephenfinOh yeah, I meant the 'hw:machine_type' property15:35
sean-k-mooneyi think its only read for arm...15:36
stephenfinwaaaait15:36
sean-k-mooneyit should be read on all archs but just going on the glance metadata docs i think its ignored on x86. i would prefer if it was not15:36
stephenfinI misread kashyap's comments. You can't do this with extra specs15:37
* stephenfin tries again15:37
*** gryf has quit IRC15:40
stephenfinkashyap: I misread your instructions. Here's the q35-only output http://paste.openstack.org/show/729350/15:41
*** gbarros has quit IRC15:41
*** gryf has joined #openstack-nova15:43
stephenfinkashyap: And with the addition of the num_pcie_ports option http://paste.openstack.org/show/729351/15:44
sean-k-mooneystephenfin: kashyap  that also looks incorrect but for different reasons. you asked for 2 pcie-root-ports and have 415:44
stephenfinsean-k-mooney: That looks better ^15:44
stephenfinsean-k-mooney: Nope, first one is without that option15:45
sean-k-mooneystephenfin: that still has 4 pcie-root-ports15:45
stephenfinIndeed. I'd expect to see 315:46
*** gibi has quit IRC15:46
sean-k-mooneywhy 3 didnt you request 2?15:46
*** tzumainn has quit IRC15:46
stephenfinbecause someone can't count :) https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5114-L511915:46
stephenfinSo that option determines how many extra to add above the original amount15:48
stephenfin*above the minimum required15:48
*** moshele has joined #openstack-nova15:48
stephenfinmoshele: Quick question: how does one create a netdev for a connectx-3 VF?15:49
stephenfinAssuming that's something you can do? I'm getting warnings without it15:49
*** helenafm has joined #openstack-nova15:49
moshelestephenfin: https://community.mellanox.com/docs/DOC-1484#jive_content_id_Configuring_8_VFs_on_a_dual_port_NIC_while_all_VFs_are_probed_on_port_115:50
stephenfinmoshele: ta15:51
moshelestephenfin: connectx-3 VF is not working with openstack15:51
stephenfinohhh, really?15:51
sean-k-mooneystephenfin: really? that is not what the melonox docs say15:51
sean-k-mooneyoh wait connext-x 315:52
moshelestephenfin: there is a missing patch https://review.openstack.org/#/c/409526/15:52
moshelestephenfin: because we one PF for 2 ports15:52
moshelestephenfin: basicly it will not work with duel port nic of cx315:53
sean-k-mooneymoshele: oh connet-x3 is that card15:53
stephenfinmoshele: I saw that. Definitely quirky15:53
moshelestephenfin: brb15:53
stephenfinGuess I can revive that patch and see if we can move it forward. Would be nice to be able to use this NIC15:53
*** helenafm has quit IRC15:55
sean-k-mooneystephenfin: that patch is not realated to your issue15:55
sean-k-mooneystephenfin: there is another issue with that card on the neutron side but the failure you are hitting is in nova/libvirt way after teh port is already bound15:57
stephenfinsean-k-mooney: Just realized that code snippet I sent is actually adding two different devices. L5114 is a PCIe root controller while the other is a PCIe root _port_ controller. My bad https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5114-L511915:58
stephenfinSo I'm guessing the first two ports are automatically added by libvirt15:58
sean-k-mooneystephenfin: looking at https://community.mellanox.com/docs/DOC-2365 did you set probe_vf ?15:58
sean-k-mooneystephenfin: you should set probe_vf=115:59
*** gbarros has joined #openstack-nova16:00
sean-k-mooneyactully im not sure..16:01
*** ttsiouts has quit IRC16:02
*** ttsiouts has joined #openstack-nova16:03
*** sahid has quit IRC16:06
*** ttsiouts has quit IRC16:07
moshelestephenfin: yes you should to make it work with a single port vf16:09
moshelestephenfin:  what is the issue exactly?16:10
sean-k-mooneymoshele: the issue stephenfin is having is that there are no netdevs allcoated for the VFs16:12
stephenfinmoshele: Yeah, and I'm seeing this for macvtap as a result http://paste.openstack.org/show/729339/16:13
stephenfinI'll try the probe option now16:13
stephenfinIt wasn't clear what that was needed for16:13
sean-k-mooneymoshele: so while ip link shows the vfs allocated form the pf there are no nedevs for the vfs16:13
*** purplerbot has quit IRC16:14
moshelesean-k-mooney: there should be if you set  probe_vf=X16:15
sean-k-mooneymoshele: currntly i dont think stephen set that so i assume based on the behavior he is seeing that it defaults to 016:15
moshelesean-k-mooney: yes you are right16:15
*** purplerbot has joined #openstack-nova16:15
sean-k-mooneyin which case if stephenfin set probe_vf=1 then vnic-type=macvtap may start working16:16
sean-k-mooneyif there is no netdev then get_ifname_by_pci_address is going to fail16:17
stephenfinmoshele: What's the difference between mlx4.conf and mlx4_core.conf?16:19
*** gbarros has quit IRC16:19
moshelestephenfin: no difference you can use ether one I think the mlx4_core.conf comes from Mellanox OFED16:20
moshelestephenfin: brb16:20
*** ccamacho has quit IRC16:20
*** pvc has joined #openstack-nova16:30
pvcNo compute node record for host when enabling ironic on packstack16:30
*** tonyb has quit IRC16:31
stephenfinmoshele: Yup, that was the issue. I know for next time. I am seeing another issue now though, if you have time to look at it http://paste.openstack.org/show/729352/16:31
stephenfin(the issue is at the end. The rest is the configuration I have)16:32
*** kosamara has quit IRC16:33
sean-k-mooneystephenfin: if you do a virsh nodedev-list can you see net_enp2s1d1_fa_16_3e_38_ea_a616:33
stephenfinI can't. I'm not sure where it got that from16:34
stephenfinActually, it only started happening once I created an instance with macvtap16:34
stephenfinI wonder if this is another reflection of the dual-port issues16:35
stephenfin*another example16:35
pvcNo compute node record for host when enabling ironic on packstack16:36
pvcanyone16:36
pvccan help me16:36
sean-k-mooneypvc: sorry you proably wont find that many people with packstack knowladge here16:37
*** dhellmann has joined #openstack-nova16:37
*** dhellmann has quit IRC16:39
*** dhellmann has joined #openstack-nova16:39
*** patrickeast has left #openstack-nova16:43
pvcbut the main problem is16:43
pvcit cannot find any hypervisor16:43
*** davidsha has quit IRC16:43
pvcbecause i enable the ironic driver instead of libvir16:43
*** jpena is now known as jpena|off16:43
*** med_ has joined #openstack-nova16:44
sean-k-mooneypvc: when you enable ironic it should still run at least one instance of the nova-compute agent that loads the ironic driver. this agent should create at least one compute service record16:44
*** dhellmann has quit IRC16:44
*** dhellmann has joined #openstack-nova16:45
sean-k-mooneyeach node you enrole in ionic should then be exposed as a seperate compute node/ hypervior record mapped to that compute servcice16:45
sean-k-mooneyif the ironic nova-compute agent does not start or fails for some reason you will have no compute node entries16:46
pvchttp://paste.openstack.org/show/729355/16:49
sean-k-mooneypvc: yes.. did you manually add nodes to ironic. also this does not show you the compute noes16:50
*** szaher has quit IRC16:50
pvcahmm no. for now we're not manually adding nodes to ironic16:50
pvcwe just install openstack using packstack16:50
pvci found something simirlar16:50
pvchttp://getvasanth.blogspot.com/2015/09/getting-start-with-ironic.html16:50
sean-k-mooneypvc: what happens if you run openstack compute service list and openstack compute agent list16:51
pvchttp://paste.openstack.org/show/729356/16:52
sean-k-mooneypvc: well it looks like you have 3 registers compute services that said i would have expected to also see them in the agent list16:53
pvcbut the server i'm using have an ipmi16:54
sean-k-mooneyin either case teh service state is up so i would suspec the nova pices is running correctly16:54
pvci guess packstack doesnt install ironic well16:54
*** dhellmann has quit IRC16:54
*** dhellmann has joined #openstack-nova16:54
sean-k-mooneypvc: well installing ironic and enroling node in ironic are two very different things16:54
pvci see16:55
pvcso i need to manually add it16:55
sean-k-mooneypvc: i think packstack installed the ironic compute agent on nova01,nova02 and network hosts but it does not have any baremetal nodes registered for it to contorl16:56
pvccan i use this guide sir? http://getvasanth.blogspot.com/2015/09/getting-start-with-ironic.html16:58
sean-k-mooneypvc: looking at it quickly it seam rather old. it may still work but i would gues several things have changed in the last 3 years. if you do want to deploy with packstack i would recommend following a more up to date guide16:59
*** derekh has quit IRC17:00
pvccan i use the ironic official document?17:03
sean-k-mooneypvc: are you just trying to deploy ironic or do you want an actual packstack install. there are several installer that support ironic that you could use if all you want is to have an ironic enabled cloud17:05
pvcan actual packstack install. i'll just install the ironic manually since my compute server have an IPMI17:05
sean-k-mooneypvc: the ironic offila documtaiton will tell you how to achive that manually17:06
sean-k-mooneypvc: this might be of relevence to you also https://review.openstack.org/#/c/440258/17:06
pvcnoted on this. wait i willl just check it17:06
sean-k-mooneypvc: it looks like packstack ironic deployment was broken in the past depending on what version you are now installing you migh have hit that error17:07
sapd1_pvc: I followed official openstack documents and deployed successful ironic. So you can trust it. :D17:07
*** janki has quit IRC17:10
*** holser_ has quit IRC17:11
*** r-daneel has quit IRC17:13
*** r-daneel has joined #openstack-nova17:13
*** hongbin has joined #openstack-nova17:28
moshelestephenfin: I will talk you tomorrow about this need to put my baby to sleep (It seem like harder task than to debug openstack issues :) )17:31
*** hongbin has quit IRC17:31
*** hongbin has joined #openstack-nova17:33
*** jiapei has quit IRC17:37
*** sapd1_ has quit IRC17:39
*** moshele has quit IRC17:44
*** dave-mccowan has quit IRC17:47
*** tssurya has quit IRC17:48
*** med_ has quit IRC17:53
openstackgerritChris Dent proposed openstack/nova master: Removing pip-missing-reqs from default tox jobs  https://review.openstack.org/59944218:15
*** med_ has joined #openstack-nova18:15
prometheanfirestephenfin: I'm here as well18:37
prometheanfireabout the weird async_ thing18:38
*** med_ has quit IRC18:38
openstackgerritMerged openstack/nova master: libvirt: skip setting rx/tx queue sizes for not virto interfaces  https://review.openstack.org/59559218:40
*** luksky has joined #openstack-nova18:47
*** Gorian has quit IRC19:10
*** kencjohnston_ has quit IRC19:10
*** test222___ has quit IRC19:10
*** csatari has quit IRC19:10
*** raorn has quit IRC19:10
*** kmalloc has quit IRC19:10
*** awestin1 has quit IRC19:10
*** betherly has quit IRC19:10
*** catmando has quit IRC19:10
*** xyang has quit IRC19:10
*** jbryce has quit IRC19:10
*** McNinja has quit IRC19:10
*** toan has quit IRC19:10
*** kencjohnston has joined #openstack-nova19:10
*** Gorian has joined #openstack-nova19:12
*** McNinja has joined #openstack-nova19:14
*** kmalloc has joined #openstack-nova19:15
*** itlinux has joined #openstack-nova19:40
*** med_ has joined #openstack-nova19:41
*** hongbin has quit IRC19:44
*** cfriesen has joined #openstack-nova20:00
prometheanfirestephenfin: around? I get the same error in a venv, even the same backtrace to a non-existant file20:07
*** erlon_ has quit IRC20:07
*** cfriesen has quit IRC20:09
*** med_ has quit IRC20:15
*** hongbin has joined #openstack-nova20:25
*** cdent has quit IRC20:28
*** erlon has joined #openstack-nova20:28
*** Sundar has joined #openstack-nova20:35
*** cdent has joined #openstack-nova20:43
*** cdent has quit IRC20:43
*** tbachman has quit IRC20:45
*** tonyb has joined #openstack-nova20:51
*** imacdonn has quit IRC21:02
*** imacdonn has joined #openstack-nova21:02
*** erlon has quit IRC21:05
*** cfriesen has joined #openstack-nova21:14
*** fried_rice1 has joined #openstack-nova21:19
*** fried_rice has quit IRC21:20
*** fried_rice1 is now known as fried_rice21:20
*** cfriesen has quit IRC21:27
*** fried_rice1 has joined #openstack-nova21:29
*** fried_rice has quit IRC21:30
*** fried_rice has joined #openstack-nova21:31
*** fried_rice1 has quit IRC21:33
*** cfriesen has joined #openstack-nova21:35
*** cfriesen has quit IRC21:40
*** Sundar has left #openstack-nova21:54
*** fried_rice has quit IRC21:57
*** luksky has quit IRC22:03
*** fried_rice has joined #openstack-nova22:22
*** hoonetorg has quit IRC22:29
*** hongbin has quit IRC22:32
*** hongbin has joined #openstack-nova22:34
*** rcernin has joined #openstack-nova22:37
*** hongbin has quit IRC22:50
*** erlon has joined #openstack-nova23:01
*** threestrands has joined #openstack-nova23:03
*** threestrands has quit IRC23:03
*** threestrands has joined #openstack-nova23:04
*** threestrands has quit IRC23:05
*** threestrands has joined #openstack-nova23:05
*** threestrands has quit IRC23:06
*** threestrands has joined #openstack-nova23:07
*** threestrands has quit IRC23:08
*** r-daneel has quit IRC23:15
*** dave-mccowan has joined #openstack-nova23:17
*** slaweq_ has joined #openstack-nova23:30
*** slaweq has quit IRC23:31
*** Dinesh_Bhor has joined #openstack-nova23:40
*** elod has quit IRC23:41
*** med_ has joined #openstack-nova23:43
*** tbachman has joined #openstack-nova23:54

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!