Wednesday, 2019-02-06

*** tetsuro has joined #openstack-nova00:03
*** whoami-rajat has quit IRC00:08
*** macza has quit IRC00:09
*** tetsuro has quit IRC00:16
*** tetsuro_ has joined #openstack-nova00:16
artomWhat takes precedence again? Flavor extra specs or image props?00:28
sean-k-mooneythat depends we generally now raise an exception if there is a conflict but original flavor took prescedence over image properties00:30
sean-k-mooneyim not sure we we have converted all conficts into exections currently but stephenfin did that for several of them00:31
artomHow are you still working o_O!00:31
artomI'm happy you're here though :D00:31
artomI'm asking because mem_page_size is in both00:32
sean-k-mooneyhehe i often take a few hours away around 7 my time and come back online for an hour or two around 10 ish00:32
sean-k-mooneyya00:33
sean-k-mooneyso for mempage size if the flavor has set a spefici page size and the image does not agree its an error00:33
sean-k-mooneyif the flavoer was hw:mem_page_size=large or hw:mem_page_size=any and the image had hw_mem_page_size=1G that should be allow but i dont know if we support it or not00:34
*** tosky has quit IRC00:34
openstackgerritMatt Riedemann proposed openstack/nova master: WIP: Add prep_snapshot_based_resize_at_dest compute method  https://review.openstack.org/63329300:34
openstackgerritMatt Riedemann proposed openstack/nova master: WIP: Add PrepResizeAtDestTask  https://review.openstack.org/62789000:34
openstackgerritMatt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_source compute method  https://review.openstack.org/63483200:34
openstackgerritMatt Riedemann proposed openstack/nova master: WIP: Add PrepResizeAtSourceTask  https://review.openstack.org/62789100:34
openstackgerritMatt Riedemann proposed openstack/nova master: Move finish_resize.(start|end) notifications to helper method  https://review.openstack.org/63507900:34
openstackgerritMatt Riedemann proposed openstack/nova master: WIP: Add finish_snapshot_based_resize_at_dest compute method  https://review.openstack.org/63508000:34
artomsean-k-mooney, so based on unit tests it looks like mem_page_size in image is outright forbidden...00:35
mriedemartom: i've always though image overrides flavor since the user can sometimes upload their own images00:35
mriedemstephenfin might think otherwise as well since i think he's written something up about this brfeore00:35
mriedem*before00:35
sean-k-mooneywell the user should not be able to overide the admin but if ther is no conflict then its fine00:36
artommriedem, hrmm, doesn't look like that's universally true00:36
artomsean-k-mooney, mriedem, https://github.com/openstack/nova/blob/f7252b586b4b9f5a098fedfa27715b8f7e662af6/nova/tests/unit/virt/test_hardware.py#L2253-L226400:36
artomTo me that reads as we bail if there's a page size in image props00:36
sean-k-mooneyartom: https://github.com/openstack/glance/blob/master/etc/metadefs/compute-guest-memory-backing.json00:36
sean-k-mooney artom no in test_get_requested_mempages_pagesize_from_image_forbidden00:37
sean-k-mooneythe flavor is hw:mem_page_size=small so not hugepages but the image asks for 2m hugepages so that is not allowed because teh admin said no00:38
sean-k-mooneytest_get_requested_mempages_pagesize_from_image_forbidden2 fails befause for the image to be allow to ask for hugepages teh flaovr must have hw:mem_page_size=large or any00:39
artomAh, so https://github.com/openstack/nova/blob/f7252b586b4b9f5a098fedfa27715b8f7e662af6/nova/virt/hardware.py#L1120 exists00:39
sean-k-mooneyartom: basically you can only aske for a specific page size via the image if the admin said it was ok in the flavor00:39
sean-k-mooneyat least that was the intent00:39
sean-k-mooney artom https://github.com/openstack/nova/blob/f7252b586b4b9f5a098fedfa27715b8f7e662af6/nova/virt/hardware.py#L1169-L117000:40
*** mlavalle has quit IRC00:41
sean-k-mooneyartom: the idea wa that if the admin wanted to supprot hugepage guest they would only ever set hw:mem_page_size=large. and if the tenant cared about the size of the hugepage they would then say so in the image00:42
artomsean-k-mooney, aha, that makes sense00:42
*** mriedem has quit IRC00:43
*** wolverineav has quit IRC00:43
artomConcretely, in our live migration claim case, it looks like we'll have to override both with the actual pagesize on the source host00:44
artomSomehow00:44
*** wolverineav has joined #openstack-nova00:45
sean-k-mooneyartom: the instance_numa_topology object has the pagesize00:46
sean-k-mooneyhttps://github.com/openstack/nova/blob/f7252b586b4b9f5a098fedfa27715b8f7e662af6/nova/objects/instance_numa_topology.py#L5100:46
*** wolverineav has quit IRC00:46
sean-k-mooneyso when you are claiming the memory for the instance jsut pull it out of that00:46
*** wolverineav has joined #openstack-nova00:46
artomsean-k-mooney, yeah, that wasn't what I was bugging on00:46
*** macza has joined #openstack-nova00:47
artomSo actually, because https://github.com/openstack/nova/blob/f7252b586b4b9f5a098fedfa27715b8f7e662af6/nova/virt/hardware.py#L1168-L1170, if we set a numerical pagesize in the flavor, it'll override anything in the image00:47
artomOK, I'm good00:47
sean-k-mooneyartom: yes but we should not mess with the flavor00:47
sean-k-mooneyor the image00:47
sean-k-mooneyjust keep it as a seperate value00:48
artomsean-k-mooney, not on the original, but if we deepcopy it we can do whatever we want :)00:48
artomI know it's ugly, but I'd rather not dive into the claims code to make it understand a separate pagesize00:48
sean-k-mooneyits a nova object wich is a ovo so you all have to reset the fields to make shour it does not get persited back into the db in some cases00:48
*** lbragsta_ has joined #openstack-nova00:50
sean-k-mooneyartom: well throw up a poc and then we can see what it looks like00:50
artomsean-k-mooney, yep00:50
*** lbragsta_ is now known as lbragstad__00:52
*** _alastor_ has quit IRC00:54
*** Swami has quit IRC00:57
*** wolverineav has quit IRC01:03
*** wolverineav has joined #openstack-nova01:04
*** hamzy has joined #openstack-nova01:06
*** wolverineav has quit IRC01:08
*** tetsuro_ has quit IRC01:08
*** tetsuro has joined #openstack-nova01:09
*** tbachman has quit IRC01:18
*** tbachman has joined #openstack-nova01:20
*** wolverineav has joined #openstack-nova01:24
*** takashin has joined #openstack-nova01:24
*** tbachman_ has joined #openstack-nova01:33
*** tetsuro has quit IRC01:33
*** tbachman has quit IRC01:34
*** tbachman_ is now known as tbachman01:34
sean-k-mooneynight all o/01:39
*** dims has quit IRC01:40
*** _alastor_ has joined #openstack-nova01:42
*** _alastor_ has quit IRC01:47
*** agopi|off has quit IRC01:57
*** wolverineav has quit IRC02:06
*** wolverineav has joined #openstack-nova02:07
*** TxGirlGeek has quit IRC02:08
*** dims has joined #openstack-nova02:11
*** wolverineav has quit IRC02:11
*** Dinesh_Bhor has joined #openstack-nova02:15
*** agopi|off has joined #openstack-nova02:16
*** cfriesen has quit IRC02:20
*** moshele has joined #openstack-nova02:20
openstackgerritMerged openstack/nova master: Move retry from _update to _update_to_placement  https://review.openstack.org/63425202:21
*** dims has quit IRC02:25
*** tetsuro has joined #openstack-nova02:30
*** dims has joined #openstack-nova02:33
*** TxGirlGeek has joined #openstack-nova02:36
*** sdake has quit IRC02:37
*** wolverineav has joined #openstack-nova02:38
*** sdake has joined #openstack-nova02:41
*** wolverineav has quit IRC02:45
*** tetsuro has quit IRC02:46
*** tetsuro_ has joined #openstack-nova02:46
*** wolverineav has joined #openstack-nova02:46
*** wolverineav has quit IRC02:47
*** lbragstad__ has quit IRC02:52
*** gyee has quit IRC03:00
*** psachin has joined #openstack-nova03:04
*** sdake has quit IRC03:23
openstackgerritArtom Lifshitz proposed openstack/nova master: [WIP] Create claims when live migrating  https://review.openstack.org/63460603:26
openstackgerritArtom Lifshitz proposed openstack/nova master: [WIP] New objects to transmit NUMA config from dest to source  https://review.openstack.org/63482703:26
openstackgerritArtom Lifshitz proposed openstack/nova master: [WIP] Send dest NUMA info inside LiveMigrateData  https://review.openstack.org/63482803:26
*** _hemna has joined #openstack-nova03:38
*** ccamacho has quit IRC03:39
*** itlinux has joined #openstack-nova03:41
*** macza has quit IRC03:45
*** tetsuro_ has quit IRC03:53
*** TxGirlGeek has quit IRC03:54
*** igordc has quit IRC04:02
*** moshele has quit IRC04:03
*** _hemna has quit IRC04:09
*** sdake has joined #openstack-nova04:10
*** macza has joined #openstack-nova04:20
*** udesale has joined #openstack-nova04:30
*** macza has quit IRC04:34
*** itlinux has quit IRC04:41
*** itlinux has joined #openstack-nova04:53
*** janki has joined #openstack-nova04:53
*** macza has joined #openstack-nova05:00
*** macza has quit IRC05:05
*** bhagyashris has joined #openstack-nova05:08
*** wolverineav has joined #openstack-nova05:11
*** wolverineav has quit IRC05:16
*** sdake has quit IRC05:25
*** sdake has joined #openstack-nova05:34
*** tkajinam_ has joined #openstack-nova05:41
*** macza has joined #openstack-nova05:42
*** tkajinam has quit IRC05:43
*** udesale has quit IRC05:45
*** macza has quit IRC05:46
*** udesale has joined #openstack-nova05:55
*** macza has joined #openstack-nova06:10
*** ratailor has joined #openstack-nova06:13
*** macza has quit IRC06:15
*** sdake has quit IRC06:20
*** moshele has joined #openstack-nova06:26
*** tetsuro has joined #openstack-nova06:30
*** tkajinam_ is now known as tkajinam06:34
*** sridharg has joined #openstack-nova06:51
*** dpawlik has joined #openstack-nova06:52
*** whoami-rajat has joined #openstack-nova06:53
*** priteau has joined #openstack-nova06:54
*** tkajinam_ has joined #openstack-nova07:03
*** lpetrut has joined #openstack-nova07:03
*** tkajinam has quit IRC07:05
*** slaweq has joined #openstack-nova07:07
openstackgerritTakashi NATSUME proposed openstack/nova master: Replace glance command with openstack command  https://review.openstack.org/63510207:10
openstackgerritTakashi NATSUME proposed openstack/nova master: Replace glance command with openstack command  https://review.openstack.org/63510207:13
*** udesale has quit IRC07:15
*** priteau has quit IRC07:18
*** udesale has joined #openstack-nova07:20
*** pcaruana has joined #openstack-nova07:29
*** abhishekk has joined #openstack-nova07:30
*** takashin has quit IRC07:32
*** markvoelker has joined #openstack-nova07:39
*** bhagyashris has quit IRC07:41
*** takashin has joined #openstack-nova07:50
openstackgerritJan Gutter proposed openstack/nova master: Ignore some PendingDeprecationWarnings for os-vif  https://review.openstack.org/63492807:55
*** takashin has left #openstack-nova08:00
*** _fragatina has joined #openstack-nova08:00
*** belmoreira has joined #openstack-nova08:02
*** rpittau has joined #openstack-nova08:06
*** zul has quit IRC08:07
*** lpetrut has quit IRC08:07
*** dpawlik has quit IRC08:07
*** lpetrut has joined #openstack-nova08:08
*** _fragatina_ has joined #openstack-nova08:08
*** _fragatina_ has quit IRC08:10
*** awalende has joined #openstack-nova08:11
*** _fragatina has quit IRC08:11
*** markvoelker has quit IRC08:12
*** rchurch_ has quit IRC08:14
*** dpawlik has joined #openstack-nova08:14
*** rchurch has joined #openstack-nova08:15
*** tkajinam_ has quit IRC08:15
*** tesseract has joined #openstack-nova08:16
*** _fragatina has joined #openstack-nova08:18
openstackgerritJan Gutter proposed openstack/os-vif master: Convert hardcoded regexes to raw strings for py36  https://review.openstack.org/63511108:20
*** dpawlik has quit IRC08:20
*** yan0s has joined #openstack-nova08:27
*** panda|off is now known as panda08:32
*** dpawlik has joined #openstack-nova08:32
*** ttsiouts has joined #openstack-nova08:33
*** dpawlik has quit IRC08:36
*** ttsiouts has quit IRC08:47
*** ttsiouts has joined #openstack-nova08:48
*** tetsuro has quit IRC08:48
*** ttsiouts has quit IRC08:52
*** tssurya has joined #openstack-nova08:54
*** Luzi has joined #openstack-nova08:54
*** tosky has joined #openstack-nova08:56
*** xek has joined #openstack-nova09:09
*** ociuhandu has quit IRC09:09
*** ttsiouts has joined #openstack-nova09:09
*** markvoelker has joined #openstack-nova09:09
*** ociuhandu has joined #openstack-nova09:10
*** lpetrut has quit IRC09:14
*** ociuhandu has quit IRC09:15
*** derekh has joined #openstack-nova09:34
*** markvoelker has quit IRC09:43
*** takamatsu_ has joined #openstack-nova09:57
*** takamatsu has quit IRC09:57
*** takamatsu_ has quit IRC10:00
*** ociuhandu has joined #openstack-nova10:03
*** takamatsu_ has joined #openstack-nova10:03
*** ociuhandu has quit IRC10:07
openstackgerritSurya Seetharaman proposed openstack/nova master: API microversion 2.68: Handles Down Cells  https://review.openstack.org/59165710:10
openstackgerritSurya Seetharaman proposed openstack/nova master: Modify InstanceMappingList.get_not_deleted_by_cell_and_project()  https://review.openstack.org/63512010:10
openstackgerritSurya Seetharaman proposed openstack/nova master: Plumbing for ignoring list_records_by_skipping_down_cells  https://review.openstack.org/63512110:10
*** psachin has quit IRC10:24
*** cdent has joined #openstack-nova10:25
*** slaweq has quit IRC10:40
*** markvoelker has joined #openstack-nova10:40
*** erlon has joined #openstack-nova10:48
*** psachin has joined #openstack-nova10:50
openstackgerritAndrey Volkov proposed openstack/nova master: Check hosts have no instances for AZ rename  https://review.openstack.org/50920610:53
*** erlon_ has joined #openstack-nova10:55
*** abhishekk has quit IRC10:55
*** slaweq has joined #openstack-nova10:57
*** erlon has quit IRC10:58
*** macza has joined #openstack-nova10:59
*** slaweq has quit IRC11:01
*** macza has quit IRC11:03
*** ttsiouts has quit IRC11:13
*** ccamacho has joined #openstack-nova11:13
*** markvoelker has quit IRC11:13
*** udesale has quit IRC11:17
openstackgerritStephen Finucane proposed openstack/python-novaclient master: Microversion 2.68: Remove 'forced' live migrations, evacuations  https://review.openstack.org/63513111:18
openstackgerritStephen Finucane proposed openstack/nova master: API: Remove evacuate/live-migrate 'force' parameter  https://review.openstack.org/63460011:22
*** wolverineav has joined #openstack-nova11:36
*** slaweq has joined #openstack-nova11:38
*** wolverineav has quit IRC11:41
*** tbachman has quit IRC11:48
*** takamatsu_ has quit IRC11:48
*** janki has quit IRC11:49
*** takamatsu_ has joined #openstack-nova11:52
*** janki has joined #openstack-nova11:54
*** ttsiouts has joined #openstack-nova11:58
*** _fragatina has quit IRC12:00
*** _fragatina has joined #openstack-nova12:01
*** ratailor has quit IRC12:09
*** markvoelker has joined #openstack-nova12:09
*** janki has quit IRC12:23
*** Dinesh_Bhor has quit IRC12:24
*** takamatsu_ has quit IRC12:24
*** takamatsu has joined #openstack-nova12:24
*** cdent has quit IRC12:26
*** ociuhandu has joined #openstack-nova12:35
*** ociuhandu has quit IRC12:35
*** ociuhandu has joined #openstack-nova12:35
*** ociuhandu has quit IRC12:35
*** ociuhandu has joined #openstack-nova12:36
*** Luzi has quit IRC12:37
*** Luzi has joined #openstack-nova12:37
*** Luzi has quit IRC12:38
*** _fragatina has quit IRC12:41
*** sdake has joined #openstack-nova12:41
*** _fragatina has joined #openstack-nova12:41
openstackgerritSurya Seetharaman proposed openstack/nova master: Plumbing for ignoring list_records_by_skipping_down_cells  https://review.openstack.org/63512112:41
openstackgerritSurya Seetharaman proposed openstack/nova master: API microversion 2.68: Handles Down Cells  https://review.openstack.org/59165712:41
openstackgerritSurya Seetharaman proposed openstack/nova master: Plumbing for allowing the all-tenants filter  https://review.openstack.org/63514512:41
openstackgerritSurya Seetharaman proposed openstack/nova master: Plumbing required in ViewBuilder to construct partial results  https://review.openstack.org/63514612:41
openstackgerritSurya Seetharaman proposed openstack/nova master: API microversion 2.68: Handles Down Cells Documentation  https://review.openstack.org/63514712:41
*** markvoelker has quit IRC12:42
*** _fragatina has quit IRC12:44
*** _fragatina has joined #openstack-nova12:44
*** _fragatina has quit IRC12:45
*** _fragatina has joined #openstack-nova12:46
*** _fragatina has quit IRC12:50
*** _fragatina has joined #openstack-nova12:51
*** _fragatina has quit IRC12:52
*** _fragatina has joined #openstack-nova12:53
*** _fragatina has quit IRC12:59
*** _fragatina has joined #openstack-nova12:59
*** _fragatina has quit IRC13:01
*** _fragatina has joined #openstack-nova13:01
*** _fragatina has quit IRC13:03
*** _fragatina has joined #openstack-nova13:04
*** _fragatina has quit IRC13:05
*** _fragatina has joined #openstack-nova13:06
*** tbachman has joined #openstack-nova13:08
*** ociuhandu_ has joined #openstack-nova13:09
*** _fragatina has quit IRC13:10
openstackgerritArtom Lifshitz proposed openstack/nova master: [WIP] Create claims when live migrating  https://review.openstack.org/63460613:10
openstackgerritArtom Lifshitz proposed openstack/nova master: [WIP] New objects to transmit NUMA config from dest to source  https://review.openstack.org/63482713:10
openstackgerritArtom Lifshitz proposed openstack/nova master: [WIP] Send dest NUMA info inside LiveMigrateData  https://review.openstack.org/63482813:10
*** erlon_ has quit IRC13:10
*** _fragatina has joined #openstack-nova13:10
*** mmethot has quit IRC13:11
*** ociuhandu_ has quit IRC13:11
*** ociuhandu_ has joined #openstack-nova13:12
*** ociuhandu has quit IRC13:12
*** mmethot has joined #openstack-nova13:14
*** _fragatina has quit IRC13:17
*** _fragatina has joined #openstack-nova13:18
*** TxGirlGeek has joined #openstack-nova13:18
*** _fragatina has quit IRC13:21
*** sdake has quit IRC13:23
*** sdake has joined #openstack-nova13:23
*** _fragatina has joined #openstack-nova13:25
*** erlon_ has joined #openstack-nova13:26
*** Luzi has joined #openstack-nova13:27
*** sdake has quit IRC13:27
*** tbachman has quit IRC13:29
*** mriedem has joined #openstack-nova13:32
*** TxGirlGeek has quit IRC13:33
*** jamesdenton has quit IRC13:33
*** janki has joined #openstack-nova13:35
*** sdake has joined #openstack-nova13:35
*** slaweq has quit IRC13:35
*** tbachman has joined #openstack-nova13:36
*** sdake has quit IRC13:39
*** markvoelker has joined #openstack-nova13:39
*** tosky has quit IRC13:44
bauzasgood afternoon Nova, now begins my upstream day13:45
sean-k-mooneyo/ hello upstream bauzas :)13:46
bauzassylvainb != bauzas :)13:46
bauzasso 'upstream bauzas' is just a pleonasm13:46
*** sdake has joined #openstack-nova13:47
sean-k-mooneybauzas: the real question is who is the evil one and who is the good one sylvainb or bauzas13:47
bauzasit's schizophrenic, I know13:48
bauzasmwuaaahahahah13:48
*** mvkr has quit IRC13:48
mriedemtssurya: i could be wrong but it seems the commit message and comment in https://review.openstack.org/#/c/635120/ are backward13:49
sean-k-mooneybauzas: the real question is who is the evil one and who is the good one sylvainb or bauzas13:50
bauzasI can't answer this question without the presence of my lawyer13:51
*** jamesdenton has joined #openstack-nova13:51
*** cdent has joined #openstack-nova13:56
*** sdake has quit IRC14:00
*** sdake has joined #openstack-nova14:02
*** _fragatina has quit IRC14:05
*** _fragatina has joined #openstack-nova14:06
*** moshele has quit IRC14:07
*** ttsiouts has quit IRC14:09
*** _fragatina has quit IRC14:09
*** _fragatina has joined #openstack-nova14:10
*** psachin has quit IRC14:10
*** sdake has quit IRC14:12
*** _hemna has joined #openstack-nova14:12
*** markvoelker has quit IRC14:13
*** ttsiouts has joined #openstack-nova14:17
*** mvkr has joined #openstack-nova14:19
*** belmoreira has quit IRC14:20
*** belmoreira has joined #openstack-nova14:21
*** moshele has joined #openstack-nova14:23
tssuryamriedem: um, no I meant for the ones where the data migration is run, it would be False/True while for the new ones its None/NULL at least in the database..14:26
tssuryaI'll update it on the review.14:27
*** eharney has joined #openstack-nova14:27
mriedemi'm not sure what "new ones" means14:29
mriedemnew instance mappings should have a default of false14:29
mriedemexisting records would have None14:29
mriedemand those are what get migrated14:29
tssuryalike fresh instance_mappings from Stein14:29
tssuryathat is what I mean by new14:29
mriedemand those should have queued_for_delete=False in the db by default14:29
tssuryawhich they don't14:29
mriedem?14:30
tssuryaI thought they should have qfd=False by default too14:30
tssuryabut in the new devstack in which I tested they are NULL14:30
tssuryaby default14:30
mriedemif they don't, then we have a problem because we'd never be able to complete that data migration14:30
tssuryawell I am assuming once we run the migration, we would not run it again14:30
mriedembut if you're saying that new instance mappings records don't have queued_for_delete=false in the db, then we'd have to migrate those as well14:31
mriedemb/c the migration is looking for queued_for_delete=None14:31
tssuryawould have to ask dansmith14:31
tssuryabut I am thinking for the newer ones None also means not queued_for_delete14:31
tssuryaand when they get deleted it would become True14:32
dansmithwhy ask me? :)14:32
dansmithif we're writing the qfd flag on object save, then new records should have =False, if not then it depends on whether or not we have a default in the schema14:33
tssuryahttps://review.openstack.org/#/c/584504/5//COMMIT_MSG@1614:33
*** zul has joined #openstack-nova14:34
dansmithoh ;)14:34
dansmithyeah, I remember this being some weird detail when writing that migration but I don't remember why14:34
*** sdake has joined #openstack-nova14:34
tssuryato catch all the None and change them to True/False14:35
tssuryawhich made sense14:35
tssuryaso I am assuming we were ok with have new records that would be None by default right ?14:35
tssuryahaving*14:35
mriedemno14:35
mriedembecause then the data migration will never be complet14:35
mriedem*complete14:35
mriedemnew records should have False in the db when we insert the record14:36
dansmithoh right right14:36
tssuryamriedem: yea the data migration will never be complete, but I am assuming it will be removed eventually and the goal of that migration was to update the deleted instances to True14:36
dansmithbecause they need to be either true or false null means unmigrated14:36
*** _alastor_ has joined #openstack-nova14:36
dansmithtssurya: no, I think the intent is to update them to true or false right?14:36
tssuryaheh ok I didn't think it was bug, but maybe it is ? :D14:36
tssuryadansmith: yea for the old instances14:37
*** mvkr has quit IRC14:37
mriedemthe data migration also sets queued_for_delete=False on instance mappings for non-deleted instances14:37
mriedemhttps://review.openstack.org/#/c/584504/5/nova/objects/instance_mapping.py@17914:37
tssuryabut we don't care for the new ones right ? for them None = False () sort of and when they get deleted it would become True() from None that's all14:37
*** mvkr has joined #openstack-nova14:37
tssuryamriedem: yea I understand14:38
tssuryaif we feel its a bug for the migration to be always incomplete14:38
*** moshele has quit IRC14:38
mriedemwell generally data migrations are something you'd like to have an end state14:38
tssuryathen we can fix.14:39
tssurya(anu suggestions ?)14:39
tssuryaany*14:39
dansmithI'm confused about what the problem is14:39
mriedemstarted with my confusion over the commit message and comments in https://review.openstack.org/#/c/635120/14:40
openstackgerritStephen Finucane proposed openstack/nova master: Remove get_config_vhostuser  https://review.openstack.org/56547114:41
openstackgerritStephen Finucane proposed openstack/nova master: Set bandwidth limits for generic vHost User interfaces  https://review.openstack.org/63517014:41
mriedemand tssurya said that with a new instance mapping in devstack today, the queued_for_delete in the DB is NULL even though we default to False in the object and sqla model code14:41
mriedemwhich ^ doesn't make sense to me14:41
*** _alastor_ has quit IRC14:41
mriedemi get that we don't set False for the server_default14:41
mriedembut inserting new records should have a default value of False in the DB i'd think14:41
stephenfinmriedem, jaypipes: I've split out that bandwidth limits fix to a separate patch, if you fancy taking a quick look over it again https://review.openstack.org/565471 https://review.openstack.org/63517014:41
mriedemstephenfin: only if i can raise my pinky while reviewing it14:42
tssuryamriedem: we don't default to false in the api_models code right ? that was changed for the migration14:42
tssuryawhich is why this is happening I think14:42
dansmithtssurya: right but we should be passing it from the object always14:42
*** BjoernT has joined #openstack-nova14:42
dansmithif not, then that's a problem14:42
* stephenfin hands mriedem a tea cup to complete the look14:42
mriedemok then i missed https://review.openstack.org/#/c/584504/5/nova/db/sqlalchemy/api_models.py14:42
mriedemwe're not passing it from the object b/c the field isn't set14:43
mriedemhttps://review.openstack.org/#/c/584504/5/nova/objects/instance_mapping.py@10214:43
mriedemhttps://review.openstack.org/#/c/584504/5/nova/compute/api.py@93814:43
dansmithwe have a default on the field, but that doesn't mean it gets set,14:44
mriedemright14:44
dansmithso maybe whoever wrote that code thought it would be14:44
mriedemso it's not in the changes14:44
mriedemanyway it's a simple fix14:44
dansmithyeah, so create() should always set that =False and destroy should set it =True14:44
mriedemchanges.setdefault('queued_for_delete', False)14:44
dansmithI'll work up a patch14:45
*** mchlumsky has joined #openstack-nova14:45
dansmithmriedem: I think it'd be better to be explicit about it14:45
dansmithbecause we have to be on delete anyway14:45
tssuryadestroy removes the mapping right ?14:45
dansmithtssurya: got a bug for this?14:45
tssuryadansmith: thanks I will open a bug now14:45
dansmithoh right, no soft delete here14:45
tssuryaand assign it to you14:45
dansmithwhen does it get set =True then?14:46
mriedemright, we set qfd = True in the API https://github.com/openstack/nova/blob/master/nova/compute/api.py#L219214:46
dansmithI clearly don't remember this14:46
dansmithoh righ on instance deete duh14:46
tssuryaduring nova delete's14:46
*** betherly has joined #openstack-nova14:47
dansmithah and we have it in our tests14:49
*** wolverineav has joined #openstack-nova14:49
dansmithlike, explicitly setting it =False14:49
*** sdake has quit IRC14:50
*** awaugama has joined #openstack-nova14:51
*** betherly has quit IRC14:51
tssuryadamsith: https://bugs.launchpad.net/nova/+bug/181491314:52
openstackLaunchpad bug 1814913 in OpenStack Compute (nova) "A new instance_mapping record will have queued_for_delete set to NULL" [Undecided,New] - Assigned to Dan Smith (dansmith)14:52
tssuryadansmith * sorry14:53
dansmiththanks, patch shortly14:53
*** cdent has quit IRC14:53
mriedemhigh fives for team work14:53
*** wolverineav has quit IRC14:53
tssuryaI will change the patch then to no longer have to consider "None" values14:54
*** macza has joined #openstack-nova14:54
bauzasmmm, we don't document the reasons why flavors are immutable, neither in https://docs.openstack.org/nova/latest/admin/flavors.html or https://docs.openstack.org/nova/latest/user/flavors.html14:54
dansmithtssurya: your patch? you still need to handle none values14:54
dansmithtssurya: meaning you can't fail if they're there14:54
tssuryadansmith: I mean this one: https://review.openstack.org/#/c/635120/14:55
bauzasI have a downstream request about modifying the nested flavor by not resizing, so I'd like to explain to the customer why I don't like it14:55
sean-k-mooneybauzas: they are immutable as if they were not existing instaces may exceed available resouces14:55
dansmithtssurya: ah okay14:55
tssuryato get non-deleted I only need to ask for "False()" right ? assuming people ran the migration14:55
bauzasyeah, I know *why*,  but I'd like to see the fact that you can't modify a nested flavor to be documented14:55
dansmithtssurya: no14:55
openstackgerritYury Kulazhenkov proposed openstack/nova master: rename ScaleIO driver to VxFlex Os  https://review.openstack.org/63486614:56
dansmithtssurya: you can't assume the migration was run until it's forced really14:56
sean-k-mooneybauzas: the downstream issue is related to the flavor extra specs not the flaovr14:56
bauzasyeah14:56
*** awaugama has quit IRC14:56
sean-k-mooneybauzas: but yes it would be nice to have that documented14:56
tssuryadansmith: even then now that you are fixing it, it would be wrong for me to have a logic where I am also querying for the None ones thinking they are non-deleted14:57
bauzastbh, I don't like to be able to modify the extra specs but...14:57
*** sdake has joined #openstack-nova14:57
tssuryabecause it could just be that the user has not run the migration14:57
dansmithtssurya: if it's null it means we don't know if it's queued for delete or not, so pick one.. should the user see their instance go away or come back from the dead? it has to be one or the other14:57
*** macza has quit IRC14:58
tssuryadansmith: hah I don't have a strong opinion on that probably "come back from dead" is better14:59
tssuryaso I will just leave my change as it is then14:59
tssuryaand just edit the commit/comment14:59
*** cfriesen has joined #openstack-nova14:59
*** janki has quit IRC15:00
*** agopi|off is now known as agopi15:01
sean-k-mooneybauzas: we proably should update https://github.com/openstack/nova/blob/master/doc/source/admin/flavors.rst#modify-a-flavor15:01
bauzasI'm out of steam for that but yeah15:02
mriedembauzas: so add something about that here https://docs.openstack.org/nova/latest/admin/flavors.html#modify-a-flavor15:02
*** macza has joined #openstack-nova15:03
sean-k-mooneybauzas: i can add a note if you like15:03
mriedemyou can copy what's in https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/flavor-description.html if you want15:04
mriedem" Note The only field that can be updated is the description field. Nova has historically intentionally not included an API to update a flavor because that would be confusing for instances already created with that flavor. Needing to change any other aspect of a flavor requires deleting and/or creating a new flavor. "15:04
sean-k-mooneymriedem: ya that makes sense. bauzas want me to do it since your busy or will i leave it to you.15:05
mriedemwe can also update the osc command note in https://docs.openstack.org/nova/latest/admin/flavors.html#modify-a-flavor to mention that support was added in openstack CLI in 3.16.0: https://review.openstack.org/#/c/575390/15:05
mriedemor just change the example to use osc15:06
mriedemand drop the nonte15:06
mriedem*note15:06
sean-k-mooneysure15:06
bauzasactually, the spec is a good example15:06
sean-k-mooneyill go do that now15:06
bauzasthanks15:06
sean-k-mooneybauzas: ya you can use the spec as a reference for the customer while i update this15:07
bauzasyeah15:07
bauzasbut anyway, I need to provide a hack :(15:07
bauzasand I hate this15:07
sean-k-mooney well you dont15:07
sean-k-mooneywe shoud discuss that downstream15:07
bauzasoh yeah, I have to do it15:07
bauzastrust me15:08
*** awaugama has joined #openstack-nova15:09
*** macza has quit IRC15:09
*** markvoelker has joined #openstack-nova15:10
*** Luzi has quit IRC15:11
stephenfinadrianc: Quick question: how does one identify the "OPN" of a ConnectX-3 NIC?15:12
openstackgerritDan Smith proposed openstack/nova master: Fix InstanceMapping to always default queued_for_delete=False  https://review.openstack.org/63518515:12
stephenfinI can find the PSID via 'mstflint query' but can't see the OPN http://www.mellanox.com/page/firmware_table_ConnectX3EN15:12
sean-k-mooneystephenfin: OPN?15:12
sean-k-mooneyoh its a vendor id15:13
stephenfinsean-k-mooney: The second column in that firmware download list. I assume it's the part ID15:13
sean-k-mooneyya it like intels etag for there firmware15:13
sean-k-mooneyat least it looks like it is anyway15:13
stephenfinsean-k-mooney: I think that's only good to distinguish between e.g. the ConnectX-3 and ConnectX-3 Pro, based on http://www.mellanox.com/page/firmware_HCA_FW_identification15:14
stephenfinand the ID I'm getting, 15b3:1003, doesn't give me any deets on https://www.pcilookup.com/?ven=15b3&dev=1003&action=submit15:14
stephenfinadrianc: I had the PSID and just clicked down through the list of OPNs to find what I was looking for. If there's a better way though, I'd be happy to hear about it15:15
stephenfinsean-k-mooney: ^15:15
openstackgerritBalazs Gibizer proposed openstack/nova master: Transfer port.resource_request to the scheduler  https://review.openstack.org/56726815:17
openstackgerritBalazs Gibizer proposed openstack/nova master: Extend RequestGroup object for mapping  https://review.openstack.org/61952715:17
openstackgerritBalazs Gibizer proposed openstack/nova master: Calculate RequestGroup resource provider mapping  https://review.openstack.org/61623915:17
openstackgerritBalazs Gibizer proposed openstack/nova master: Fill the RequestGroup mapping during schedule  https://review.openstack.org/61952815:17
openstackgerritBalazs Gibizer proposed openstack/nova master: Pass resource provider mapping to neutronv2 api  https://review.openstack.org/61624015:17
openstackgerritBalazs Gibizer proposed openstack/nova master: Recalculate request group - RP mapping during re-schedule  https://review.openstack.org/61952915:17
openstackgerritBalazs Gibizer proposed openstack/nova master: Send RP uuid in the port binding  https://review.openstack.org/56945915:17
openstackgerritBalazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request  https://review.openstack.org/57331715:17
openstackgerritBalazs Gibizer proposed openstack/nova master: Remove port allocation during detach  https://review.openstack.org/62242115:17
openstackgerritBalazs Gibizer proposed openstack/nova master: Record requester in the InstancePCIRequest  https://review.openstack.org/62531015:17
openstackgerritBalazs Gibizer proposed openstack/nova master: Add pf_interface_name tag to passthrough_whitelist  https://review.openstack.org/62531115:17
openstackgerritBalazs Gibizer proposed openstack/nova master: Ensure that bandwidth and VF are from the same PF  https://review.openstack.org/62354315:17
jaypipesstephenfin: ok, will try. (sorry, been out of IRC contact for 6+ days)15:17
stephenfinjaypipes: It's all good. Thanks for even trying15:19
*** macza has joined #openstack-nova15:19
*** mrch has joined #openstack-nova15:20
*** awalende has quit IRC15:23
*** erlon__ has joined #openstack-nova15:23
*** awalende has joined #openstack-nova15:24
*** artom has quit IRC15:26
*** erlon_ has quit IRC15:26
*** awalende has quit IRC15:28
*** ociuhandu_ has quit IRC15:28
*** mlavalle has joined #openstack-nova15:28
*** ociuhandu has joined #openstack-nova15:29
mriedemwhy oh why doesn't the set -x here https://review.openstack.org/#/c/634962/2/nova/tests/live_migration/hooks/run_tests.sh@12 enable tracing?15:30
mriedemhttp://logs.openstack.org/62/634962/2/check/legacy-grenade-dsvm-neutron-multinode-live-migration/f8f245f/logs/devstack-gate-post_test_hook.txt.gz#_2019-02-05_22_35_09_43915:30
*** mrch has quit IRC15:31
mriedemoh i bet something downstream unsets it15:31
mriedemaha http://logs.openstack.org/62/634962/2/check/legacy-grenade-dsvm-neutron-multinode-live-migration/f8f245f/logs/devstack-gate-post_test_hook.txt.gz#_2019-02-05_22_35_09_52515:32
openstackgerritMatt Riedemann proposed openstack/nova master: WIP: Fix legacy-grenade-dsvm-neutron-multinode-live-migration  https://review.openstack.org/63496215:36
bauzasgibi_off: would you mind if I was rebasing https://review.openstack.org/#/c/631559/ on top of the new revision I'm working for https://review.openstack.org/#/c/599208/15:36
bauzas?15:36
openstackgerritEric Fried proposed openstack/nova master: WIP: Allow [pci]alias to take a list  https://review.openstack.org/63519115:37
mriedembauzas: i'm sure he doesn't mind15:37
bauzasack15:37
*** macza has quit IRC15:40
*** markvoelker has quit IRC15:43
*** moshele has joined #openstack-nova15:44
*** moshele has quit IRC15:46
openstackgerritJack Ding proposed openstack/nova master: Improve libvirt image and snapshot handling  https://review.openstack.org/61669215:46
*** udesale has joined #openstack-nova15:48
*** wwriverrat has quit IRC15:54
stephenfinsean-k-mooney: https://that.guru/blog/updating-mellanox-connectx-3/15:55
stephenfinadrianc: ^ if you figure that out, I'll update the above15:55
*** awalende has joined #openstack-nova15:56
*** cdent has joined #openstack-nova15:56
openstackgerritsean mooney proposed openstack/nova master: update flavor admin docs  https://review.openstack.org/63519815:59
*** awalende has quit IRC16:00
*** tbachman has quit IRC16:01
*** _alastor_ has joined #openstack-nova16:04
sean-k-mooneystephenfin: i assume this should not happen http://paste.openstack.org/show/744632/16:04
stephenfinsean-k-mooney: You've got left over pyc files16:05
openstackgerritMerged openstack/nova master: Ignore some PendingDeprecationWarnings for os-vif  https://review.openstack.org/63492816:05
stephenfinand the interpreter is trying to read them16:05
sean-k-mooneyoh even after -r16:05
stephenfin'find . -type f -name "*.pyc" -delete'16:05
melwitto/16:05
mriedemsean-k-mooney: stephenfin: we used to have something in tox.ini to delete those on each run to avoid that but someone had an opinion and removed it16:06
sean-k-mooneymriedem: yes i rememebr i was going to check if it was remvoed16:07
*** tbachman has joined #openstack-nova16:07
stephenfinmriedem: It's still there and that sounds rather passive aggressive. If that's intentional, stop it please. No need for that16:07
stephenfinWe just don't do it for docs16:07
sean-k-mooneystephenfin: anyway that seams to have partly fixed it.16:07
sean-k-mooneyi now get dot command 'dot' cannot be run (needed for graphviz output), check the graphviz_dot setting16:08
stephenfinsean-k-mooney: Yup, if you fancy sticking a call to '{[testenv]commands}' in the commands for the 'docs' target, it'd be an easy fix to prevent that recurring16:08
stephenfinsean-k-mooney: Guess you need to install dot?16:08
sean-k-mooneyya i can16:08
mriedemhuh?16:08
mriedemwe don't delete pyc in the tox docs env16:09
mriedemb/c we override commands16:09
mriedemwhat am i being passive aggressive about?16:09
mriedemi hit this same thing from time to time and have to remember to delete pycs manually which is annoying16:09
sean-k-mooneyill update the docs tox env but i think we need to add graphviz as a docs dep too16:10
mriedemgraphviz is a binary16:10
mriedemit's in bindep16:10
sean-k-mooneyyou are right but why do we now need it for docs16:10
sean-k-mooneywe didnt in the past16:10
mriedemwe've needed it for a long time16:10
mriedemare you on a new system?16:11
sean-k-mooneyim on a system that has never had devstack run16:11
sean-k-mooneyso yes16:11
sean-k-mooneyok ill just install it16:11
jackdingmriedem: runway for https://review.openstack.org/#/c/620706/ ends tomorrow. I believe I have addressed your comments. Do you have sometime to revisit?16:12
mriedemstephenfin: this is what i was referring to http://lists.openstack.org/pipermail/openstack-dev/2014-September/045873.html16:12
stephenfinmriedem: "we used to have something in tox.ini to delete those on each run to avoid that but someone had an opinion and removed it"16:12
stephenfinmriedem: I figured that was aimed at me, in which case it seemed like I was being singled out for what was almost certainly a change made for good reasons16:12
stephenfinas opposed to "opinions"16:12
mriedemi was referring to that thread16:13
mriedemwhich was related to https://review.openstack.org/#/c/338231/16:13
mriedemwhich was yours, but that was just "don't create pycs rather than delete them" which as you can see i was +2 on16:13
stephenfinYeah, we reverted that though 233160644f4f0a17177fffe8a1e6db1341e61ad416:13
mriedemi know16:13
sean-k-mooney anyway ill push a patch to add the delete to the docs job if ye feel like reviewing16:14
stephenfinCool. So this has just been missed is all. Surprised it didn't come up before now, tbh16:14
mriedemsean-k-mooney: just do what stephenfin asked for16:14
mriedem '{[testenv]commands}'16:15
sean-k-mooneyya that was what i was going to do16:15
sean-k-mooneyi was looking up the syntax :)16:15
*** _fragatina_ has joined #openstack-nova16:16
stephenfinmriedem: But, crucially, I didn't remove it (or anything else, I'd like to think) the first day because of "opinions". I gave sound reasons in that commit message. I just missed the tox issue, which was the reason for the later revert16:16
stephenfinThat is all :)16:16
stephenfinSpeaking of, as soon as we support a new enough version of tox, we can revert that revert again16:16
stephenfinhttps://github.com/tox-dev/tox/commit/336f4f6bd8b53223f940fc5cfc43b1bbd78d469916:17
*** _fragatina has quit IRC16:17
*** ttsiouts has quit IRC16:18
*** ttsiouts has joined #openstack-nova16:18
melwittgibi_off: thank you for the note! I hope you are having a nice recovery16:20
*** ttsiouts has quit IRC16:23
*** tbachman has quit IRC16:24
*** agopi is now known as agopi|brb16:25
*** xek has quit IRC16:26
*** xek has joined #openstack-nova16:26
mriedemstephenfin: question inline https://review.openstack.org/#/c/635170/16:27
* stephenfin clicks16:27
stephenfinmriedem: Yeah, I am16:27
stephenfinsean-k-mooney, jangutter: You should probably look at the above too16:28
sean-k-mooneystephenfin: for vhostuser i dont think they work16:28
sean-k-mooneystephenfin: that is why we dont set them today16:28
sean-k-mooneystephenfin: libvirt sets tehm on the tap device but vhost user does not have a tap16:29
janguttersean-k-mooney: yah vhost-user has a socket.16:29
sean-k-mooneystephenfin: jangutter well the main point is that only way to do ratelimiting with vhost-user that im aware of is via dpdk/ovs16:30
janguttersean-k-mooney: ya, there's no netdev to set nifty options like rxqueuelen, etc.16:31
sean-k-mooneytc cant do ratelimiting on a unix socket escpially since that socket is part of the contol plane not the data plane16:31
sean-k-mooneyjangutter: well you can set the queue lenght in the qemu commandline16:31
sean-k-mooneyand we do i think16:32
*** tbachman has joined #openstack-nova16:32
*** gyee has joined #openstack-nova16:32
sean-k-mooneybut qemu does not do the rate limiting tc did i think16:32
*** agopi|brb has quit IRC16:32
openstackgerritsean mooney proposed openstack/nova master: cleanup *.pyc files in docs tox envs  https://review.openstack.org/63521016:33
janguttersean-k-mooney: drat, already running out of caffeine, I'm thinking of txquelen for tc qdiscs (like ip link set txqueuelen 500 dev eth0)16:33
janguttersean-k-mooney: yah, setting things like number of descriptors in vhost-user is something entirely different and doesn't have to do with rate limiting.16:34
mriedemsean-k-mooney: weren't you +1 on https://review.openstack.org/#/c/565471/ previously? this change for bw was split out from that16:34
sean-k-mooneymriedem: no dont think i had ever left a +1 on that one specifrically  but i did remove my -116:37
sean-k-mooneyi have not reviewed it recently16:37
sean-k-mooneymriedem: stephenfin anyway i dont think it would break vhost-user but i think it would have no effect16:38
sean-k-mooneyin anycase we should not set it as it cannot work with vhost-user so its misleading16:38
*** markvoelker has joined #openstack-nova16:40
mriedemsean-k-mooney: ok i remember you asking me to look at it :)16:40
mriedemso maybe i assumed you had16:40
*** artom has joined #openstack-nova16:41
sean-k-mooneymriedem: yes i did sorry i had planned too re review but i didnt get to it yet sorry16:42
mriedemyou're forgiven sean16:44
melwittbauzas: you're not in #openstack-placement, but there's a extraction update meeting at the top of the hour, hangout url to be linked in #openstack-placement16:44
bauzasmelwitt: I'll be there16:45
melwittcool16:45
bauzasmelwitt: for some reason, ZNC dropped me from -placement, but I'll rejoin16:46
bauzasmelwitt: thanks for the notice :)16:46
janguttersean-k-mooney, stephenfin: https://github.com/libvirt/libvirt/blob/568a417224e7b4656b9959b03154d45566480d65/src/conf/netdev_bandwidth_conf.h#L3816:47
sean-k-mooney jangutter right so based on ^ libvirt will ignore it for vhos-user if it is set16:49
janguttersean-k-mooney, stephenfin: yah, in fact, if that's canonical, it's a nice checklist of "supported" interfaces.16:49
openstackgerritMichal Arbet proposed openstack/nova master: Fix python3 compatibilit  https://review.openstack.org/63522016:49
openstackgerritMichal Arbet proposed openstack/nova master: Fix python3 compatibility  https://review.openstack.org/63522016:50
*** agopi|brb has joined #openstack-nova16:52
*** med_ has joined #openstack-nova16:55
jackdingmriedem: just let you know that https://review.openstack.org/#/c/620706/ and https://review.openstack.org/#/c/603844/ is ready to review, thanks16:57
*** agopi|brb is now known as agopi16:58
*** sdake has quit IRC17:00
cdentfor those not in #openstack-placement placement checkin hangout at https://hangouts.google.com/call/C5h9TyVk5BMfSO8vGxZ_AEEE17:01
*** udesale has quit IRC17:04
*** _hemna has quit IRC17:07
*** tssurya has quit IRC17:09
*** macza has joined #openstack-nova17:09
stephenfinjangutter, sean-k-mooney: Right, so we were setting the option for OVS vHost User interfaces but it wasn't doing anything17:10
stephenfinjangutter, sean-k-mooney: In that case, is it fair to fold it back into the following patch and just ignore the fact that it's being set, given that it does nothing?17:11
stephenfinFar as I can see, the only side effect will be this https://review.openstack.org/#/c/565471/7/nova/tests/unit/virt/libvirt/test_vif.py17:11
*** macza has quit IRC17:13
*** sdake has joined #openstack-nova17:13
*** markvoelker has quit IRC17:13
sean-k-mooneystephenfin: we should not set it if its not supported17:16
*** ociuhandu_ has joined #openstack-nova17:16
stephenfinOK,so I guess I should pivot that bug to say "this should *not* be set"?17:16
sean-k-mooneyyes i think so17:17
*** ociuhandu has quit IRC17:17
*** pcaruana has quit IRC17:19
*** mvkr has quit IRC17:22
*** mdbooth_ is now known as mdbooth17:24
*** macza has joined #openstack-nova17:27
-openstackstatus- NOTICE: Any changes failed around 16:30 UTC today with a review comment from Zuul like "ERROR Unable to find playbook" can be safely rechecked; this was an unanticipated side effect of our work to move base job definitions between configuration repositories.17:28
*** sdake has quit IRC17:31
*** macza has quit IRC17:32
*** rpittau has quit IRC17:34
openstackgerritArtom Lifshitz proposed openstack/nova master: [WIP] Create claims when live migrating  https://review.openstack.org/63460617:35
openstackgerritArtom Lifshitz proposed openstack/nova master: [WIP] New objects to transmit NUMA config from dest to source  https://review.openstack.org/63482717:35
openstackgerritArtom Lifshitz proposed openstack/nova master: [WIP] Send dest NUMA info inside LiveMigrateData  https://review.openstack.org/63482817:35
openstackgerritArtom Lifshitz proposed openstack/nova master: [Very WIP] Update NUMA XML during live migration  https://review.openstack.org/63522917:35
artomcfriesen, ^^ I know you wanted to play with early code. So it's still very early, but once that top [Very WIP] patch is just WIP it'll be ready for testing - by which I mean, I'll have enough code written to start actually running it myself in my dev env17:36
*** sdake has joined #openstack-nova17:36
artomSo it'll most likely still be broken, actually :)17:36
artomMy plan is to write some automated tests with a tempest plugin we have, and iterate running those/fixing bugs until the feature works. Then unit tests will get fixed/written.17:36
cfriesenartom: cool.  I'm just finishing up some other downstream stuff, but thanks for the heads-up.17:38
*** macza has joined #openstack-nova17:41
*** mdbooth has quit IRC17:45
*** macza has quit IRC17:45
*** mdbooth has joined #openstack-nova17:45
mriedemspeaking of gibi's bw provider series, i'm +2 on the bottom 2 patches https://review.openstack.org/#/c/567268/17:47
mriedemto help keep that moving17:47
*** mvkr has joined #openstack-nova17:51
*** jmlowe has joined #openstack-nova17:53
*** yan0s has quit IRC17:58
*** jobewan has joined #openstack-nova18:00
*** owalsh has quit IRC18:01
*** derekh has quit IRC18:01
*** xek_ has joined #openstack-nova18:02
*** _fragatina_ has quit IRC18:03
*** xek has quit IRC18:04
*** owalsh has joined #openstack-nova18:06
artomcfriesen, yeah, no huge rush, just an FYI18:10
*** markvoelker has joined #openstack-nova18:10
openstackgerritMatt Riedemann proposed openstack/nova master: WIP: Fix legacy-grenade-dsvm-neutron-multinode-live-migration  https://review.openstack.org/63496218:16
*** tesseract has quit IRC18:16
*** sdake has quit IRC18:17
*** spatel has joined #openstack-nova18:20
*** sdake has joined #openstack-nova18:21
*** _hemna has joined #openstack-nova18:22
openstackgerritStephen Finucane proposed openstack/nova master: docs: Update the PCI passthrough guides  https://review.openstack.org/63524318:31
stephenfinmriedem: You might have some interest in that little doc patch ^18:31
stephenfinInspired by stephenfin's struggles getting a PCI environment set up locally again18:32
*** jmlowe has quit IRC18:35
*** sridharg has quit IRC18:36
*** TxGirlGeek has joined #openstack-nova18:39
cdentmriedem, melwitt, jaypipes: Assuming we all still have jobs and openstack is still a thing come the opening of train, I'd like to work on making shared disk a thing. Do we have anything other than effort that is blocking that?18:40
*** markvoelker has quit IRC18:43
*** ociuhandu_ has quit IRC18:47
*** wolverineav has joined #openstack-nova18:49
*** jmlowe has joined #openstack-nova18:50
*** tbachman has quit IRC18:52
*** spatel has quit IRC18:58
*** wolverineav has quit IRC19:06
*** dpawlik has joined #openstack-nova19:06
*** wolverineav has joined #openstack-nova19:06
*** wolverineav has quit IRC19:07
*** wolverineav has joined #openstack-nova19:07
mordredcdent: I think that also assumes that the rule of law still holds somewhat and the seas haven't risen and swallowed us right?19:08
mriedemcdent: unknown unknowns19:08
mriedemi have a devstack-plugin-ceph patch that sets up a shared DISK_GB provider19:08
mriedembut that job needs to be made multi-node to be interesting19:08
mriedemhttps://review.openstack.org/#/c/586363/19:09
mriedemplus that existing bug which enumerates some known issues and the ptg etherpad19:09
mriedemthe todo from the ptg was write up a spec with the known issues and how we deal with those so we have a clear picture19:09
mriedemthen the rest i guess we find out from integration testing19:09
mriedemcdent: assuming the vmware case is something like multiple vcenters sharing the same datastore?19:11
mriedemor maybe that's already supported albeit poorly in nova19:11
cdentmriedem: that's one. the other is N>1 datastores in the same vcenter19:12
cdentshared can work there too19:12
cdenthttps://github.com/cdent/placement-disk-models19:12
cdentor a mix of both19:12
cdentthe main problem facing vcenters right now is the N>1 situation where it appears like there is a larger free slice than there actually is19:13
cdentright now max_unit is manipulated to cover that, but it is tetchy19:13
cdentmordred: you make a good point. I should have forewarned that I'm unable to enumerate all risks, and past performance is not an indicator19:15
mordredcdent: fair19:15
*** jobewan has quit IRC19:16
openstackgerritsean mooney proposed openstack/nova master: update flavor admin docs  https://review.openstack.org/63519819:26
*** moshele has joined #openstack-nova19:26
mriedemneed another stable core for https://review.openstack.org/#/c/633807/19:28
*** wolverineav has quit IRC19:38
openstackgerritMatt Riedemann proposed openstack/nova master: update flavor admin docs  https://review.openstack.org/63519819:39
*** wolverineav has joined #openstack-nova19:39
*** markvoelker has joined #openstack-nova19:40
*** wolverineav has quit IRC19:41
*** wolverineav has joined #openstack-nova19:41
openstackgerritAndrey Volkov proposed openstack/nova master: Check hosts have no instances for AZ rename  https://review.openstack.org/50920619:43
sean-k-mooneymriedem: thanks for adressing the remaining nits. i dont mind respinning for things like that but docs are not really my strong suit.19:45
openstackgerritsean mooney proposed openstack/nova master: cleanup *.pyc files in docs tox envs  https://review.openstack.org/63521019:46
openstackgerritMatt Riedemann proposed openstack/nova master: api-ref: fix link to flavor extra specs docs  https://review.openstack.org/63525219:48
openstackgerritMatt Riedemann proposed openstack/nova master: api-ref: fix link to flavor extra specs docs  https://review.openstack.org/63525219:49
*** jobewan has joined #openstack-nova19:50
openstackgerritsean mooney proposed openstack/nova stable/ocata: PCI: do not force remove allocated devices  https://review.openstack.org/63507519:59
*** erlon__ has quit IRC20:03
*** wolverineav has quit IRC20:04
*** dpawlik has quit IRC20:05
*** wolverineav has joined #openstack-nova20:06
*** cdent has quit IRC20:06
*** ccamacho has quit IRC20:08
*** wolverineav has quit IRC20:10
*** wolverineav has joined #openstack-nova20:11
*** markvoelker has quit IRC20:12
*** sdake has quit IRC20:13
*** jmlowe has quit IRC20:13
openstackgerritArtom Lifshitz proposed openstack/nova master: [WIP] New objects to transmit NUMA config from dest to source  https://review.openstack.org/63482720:14
openstackgerritArtom Lifshitz proposed openstack/nova master: [WIP] Send dest NUMA info inside LiveMigrateData  https://review.openstack.org/63482820:14
openstackgerritArtom Lifshitz proposed openstack/nova master: [WIP] Update NUMA XML during live migration  https://review.openstack.org/63522920:14
*** wolverineav has quit IRC20:15
artomOK, that's "done". Now to automate some tests.20:18
sean-k-mooneyartom: done as in works?20:19
artomsean-k-mooney, no idea :) But I'm at the point there the XML on the dest should be updated. So I can start writing tests that assert stuff about it20:19
sean-k-mooneyim planning to pull it down in tommorow anyway but i can try it out later if you think it will work20:20
artomIt's an MVP - claims aren't cleaned up where they should be, no rollback handling, etc etc20:20
artomsean-k-mooney, unlikely, I haven't run it myself yet20:20
artom(Unless you count the gate)20:20
sean-k-mooneywell it wont hurt to try. ill spin it up on my ci cloud in the backgound and take a look20:21
artomI fully expect it to blow up immediately :D20:21
sean-k-mooneywell thats what devstack vms are for20:22
*** dpawlik has joined #openstack-nova20:22
*** sdake has joined #openstack-nova20:24
*** dpawlik has quit IRC20:27
openstackgerritMatt Riedemann proposed openstack/nova master: doc: mention description field in user flavors docs  https://review.openstack.org/63526320:29
*** dtantsur is now known as dtantsur|afk20:39
*** panda is now known as panda|off20:41
cfriesenstephenfin: you around?20:41
*** awaugama has quit IRC20:42
sean-k-mooneystephenfin: he normally finishes about 2 hours before now20:43
cfriesensean-k-mooney: mriedem: you guys might know this too...we expect both KVM-based and tcg-based qemu to report a hypervisor type of "QEMU", right?20:43
sean-k-mooneyyes20:43
sean-k-mooneykvm is just a software/kernel accleration for qemu so it does not change the hypervisor type if used20:45
openstackgerritMatt Riedemann proposed openstack/nova master: Fix legacy-grenade-dsvm-neutron-multinode-live-migration  https://review.openstack.org/63496220:45
cfriesenI think there's a bug in stephenfin's "fail live migration if instance has numa topologies" commit then.  https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L16620:45
sean-k-mooneysimilarly if you were to use the tcg multi thread backend "which we dont support yet" it would still be qemu20:45
sean-k-mooney hum maybe lets see if that is the same hypervior type that is in the hyperviors api20:46
sean-k-mooneyso that is refering to https://github.com/openstack/nova/blob/master/nova/objects/compute_node.py#L6920:47
openstackgerritMatt Riedemann proposed openstack/nova master: Set bash xtrace and errexit globally in nova/tests/live_migration/hooks/*  https://review.openstack.org/63529120:48
*** wolverineav has joined #openstack-nova20:50
*** xek__ has joined #openstack-nova20:50
sean-k-mooneyso the fact its not a HVTypeField is not promissing but it proably should be https://github.com/openstack/nova/blob/master/nova/objects/fields.py#L37920:51
openstackgerritMatt Riedemann proposed openstack/nova master: Replace ansible --sudo with --become in live_migration/hooks scripts  https://review.openstack.org/63530820:51
*** xek_ has quit IRC20:52
sean-k-mooneycfriesen: so ya when virt tipe is set to kvm it is still qemu i just check on a local devstack.20:53
cfriesensean-k-mooney: agreed, that's what I was thinking.  QEMU rather than KVM20:53
*** wolverineav has quit IRC20:54
cfriesenthe tricky bit is that I think qemu with TCG doesn't support numa stuff20:54
sean-k-mooneyit does20:55
sean-k-mooneyit just does not support per cpu pinning20:55
sean-k-mooneyyou can create numa nodes and use hugepages20:55
cfriesenah, okay.20:55
cfriesenshould be all right then20:55
sean-k-mooneybut nova's implematnion does not work due to the per cpu pinning it adds on top20:55
sean-k-mooneythe mttcg backend (multi thread TCG) remvoes the cpu pinning limitation too20:56
sean-k-mooneylibvirt will have support for that soon20:56
*** sdake has quit IRC20:57
*** sdake has joined #openstack-nova20:59
cfriesenI'll propose a fix, see what stephen has to say tomorrow.21:00
sean-k-mooneycfriesen: ya looking at the compute node object in the db it does not have the virt type21:00
sean-k-mooneyso i guess just update it to qemu21:01
*** _hemna has quit IRC21:01
sean-k-mooneyat the moment numa stuff wont work with qemu via nova so its fine that this applies to both21:01
sean-k-mooneyif the instance has a numa toplogy and we are migrating it it means it was spawned on a kvm host21:02
*** jmlowe has joined #openstack-nova21:06
*** wolverineav has joined #openstack-nova21:06
sean-k-mooneyartom: fyi this is the most imporant comment out of the one i just left on the review https://review.openstack.org/#/c/634606/9/nova/compute/manager.py@601121:07
artomsean-k-mooney, that explains so much!21:07
artomI hadn't rebased locally, line numbers in CI output didn't match, and _get_resource_tracker() was clearly present21:08
sean-k-mooneyit broke the sriov migrtaion series too21:08
artomThanks!21:08
sean-k-mooneyi fixed it in that yesterday so when i saw the fuction and the filing ci it clicked :)21:08
openstackgerritMerged openstack/nova stable/queens: Fix port dns_name reset  https://review.openstack.org/63380721:08
openstackgerritMerged openstack/nova master: update flavor admin docs  https://review.openstack.org/63519821:09
artomWell, CI was going to fail regardless, but I was hoping it'd fail less21:09
*** markvoelker has joined #openstack-nova21:09
*** slaweq has joined #openstack-nova21:12
openstackgerritMatt Riedemann proposed openstack/nova master: api-ref: warn about changing/unsetting AZ name with instances  https://review.openstack.org/63531521:15
*** ociuhandu has joined #openstack-nova21:16
*** xek__ has quit IRC21:17
openstackgerritMatt Riedemann proposed openstack/nova master: api-ref: mention policy defaults for aggregates  https://review.openstack.org/63531721:18
*** tbachman has joined #openstack-nova21:19
*** moshele has quit IRC21:21
*** sdake has quit IRC21:23
*** tbachman has quit IRC21:24
*** tbachman has joined #openstack-nova21:26
*** wolverineav has quit IRC21:27
*** wolverineav has joined #openstack-nova21:28
*** wolverineav has quit IRC21:32
*** artom has quit IRC21:34
*** _fragatina has joined #openstack-nova21:36
*** ociuhandu has quit IRC21:40
*** markvoelker has quit IRC21:42
mriedemcfriesen: jackding: https://review.openstack.org/#/c/603844/21:48
mriedemi like this approach better, but the patch needs work21:48
mriedemand i think we probably don't want to mess with instances whose task_state is not None21:49
mriedemalso funny that i was looking at a very similar thing in something i was writing yesterday for cross-cell resize https://review.openstack.org/#/c/635080/1/nova/compute/manager.py@502921:50
*** takashin has joined #openstack-nova21:50
*** wolverineav has joined #openstack-nova21:51
jackdingmriedem: thanks. Yeh rollback can be messy.21:55
*** tbachman_ has joined #openstack-nova21:55
*** slaweq has quit IRC21:55
*** tbachman has quit IRC21:56
*** tbachman_ is now known as tbachman21:56
openstackgerritMatt Riedemann proposed openstack/nova master: neutron: make migrate_instance_finish call setup_instance_network_on_host  https://review.openstack.org/63534322:15
openstackgerritMichal Arbet proposed openstack/nova master: Fix python3 compatibility  https://review.openstack.org/63522022:18
*** TxGirlGeek has quit IRC22:34
openstackgerritMichal Arbet proposed openstack/nova master: Fix python3 compatibility  https://review.openstack.org/63522022:36
*** agopi is now known as agopi|brb22:36
*** TxGirlGeek has joined #openstack-nova22:36
*** alex_xu has quit IRC22:38
*** sdake has joined #openstack-nova22:39
openstackgerritMatt Riedemann proposed openstack/nova master: De-cruftify the finish_resize methods  https://review.openstack.org/63534922:39
*** markvoelker has joined #openstack-nova22:40
*** agopi|brb has quit IRC22:40
openstackgerritEric Fried proposed openstack/nova master: Use :oslo-config: role in hypervisor-kvm doc  https://review.openstack.org/63479822:42
*** kevko has joined #openstack-nova22:42
*** mchlumsky has quit IRC22:50
hogepodgeHi, someone just asked if there's a way to explicitly disable live migrations, and after looking at the configuration docs I can see why they're stumped. Anyone have some info on how to do that?22:50
*** BjoernT has quit IRC22:50
sean-k-mooneythere is a way to do it yes ill have to look it up22:50
*** kevko has quit IRC22:51
sean-k-mooneyhogepodge: it is admin only thing so nomally you dont have to expcltly disable it22:51
*** tkajinam has joined #openstack-nova22:53
sean-k-mooneyhogepodge: you might have to do it via the polcy.json22:53
hogepodgeok, thanks. I thought that might be the case. I asked for more context22:54
openstackgerritChris Friesen proposed openstack/nova master: fix up numa-topology live migration hypervisor check  https://review.openstack.org/63535022:54
sean-k-mooneyi think the way this is normally done is to block the api request using the policy.json22:54
sean-k-mooneydansmith: do you recall another way to disable live migrations?22:54
openstackgerritMatt Riedemann proposed openstack/nova master: Delete the obj_as_admin context manager  https://review.openstack.org/63535122:55
mriedemhttps://docs.openstack.org/nova/latest/configuration/policy.html22:55
mriedemos_compute_api:os-migrate-server:migrate_live22:55
mriedemchange that to !22:56
mriedemto disable it for everyone22:56
mriedemhogepodge: ^22:56
mriedemoh hey look legacy-grenade-dsvm-neutron-multinode-live-migration is fixed22:58
mriedemhttps://review.openstack.org/#/c/634962/22:58
mriedemhttp://logs.openstack.org/62/634962/5/check/legacy-grenade-dsvm-neutron-multinode-live-migration/982e90b/22:58
*** TxGirlGeek has quit IRC22:59
*** eharney has quit IRC22:59
*** agopi|brb has joined #openstack-nova23:01
hogepodgemriedem: sean-k-mooney: thanks23:01
sean-k-mooneymriedem: so is that related to the block migration issue where it was complaing it was not on shared shared storage23:03
sean-k-mooneyah it is23:03
sean-k-mooneyi was looking for https://bugs.launchpad.net/nova/+bug/1813216 yesterday23:03
openstackLaunchpad bug 1813216 in OpenStack Compute (nova) "legacy-grenade-dsvm-neutron-multinode-live-migration failing with "is not on shared storage: Shared storage live-migration requires either shared storage or boot-from-volume with no local disks." since Jan 21" [Medium,In progress] - Assigned to Matt Riedemann (mriedem)23:03
*** agopi|brb is now known as agopi23:04
mriedemyeah nova-compute during the ceph run didn't have the rbd config in it23:05
sean-k-mooneymriedem: its not really a block migration if you are using the RBD iamge backend23:05
sean-k-mooneyits not a bfv instance either but when using RBD it really is a shared filesystem.23:07
sean-k-mooneydo we run the block migration tests on any multinode job without ceph/nfs23:08
mriedemyes23:08
mriedemwell, i think so,23:08
mriedemneutron multinode or whatever23:08
*** ttsiouts has joined #openstack-nova23:08
mriedemhttp://logs.openstack.org/44/603844/19/check/tempest-multinode-full/698a95c/job-output.txt.gz#_2019-02-05_22_28_06_65401123:09
sean-k-mooneyah this one https://github.com/openstack/nova/blob/master/.zuul.yaml#L252-L25423:10
mriedemwe don't have anything that tests the crazy environment lyarwood keeps trying to fix which is rbd shared storage for the root disk but local storage for the instance files23:10
sean-k-mooneyok i was wondering if that came in form one of the templates23:10
mriedemnova, for the most part, assumes that if you're using rbd you're shared everywhere23:10
mriedemwhich apparently isn't true for some people23:10
sean-k-mooneyi dont understand the "local storage for the instance files" bit23:11
sean-k-mooneyi can ask lyarwood23:11
mriedemthe stuff under $instances_path23:11
mriedemconsole logs, the image, config drive (i think)23:11
mriedemetc23:11
sean-k-mooneyoh the libivt logs and stuff23:11
mriedemyes this crazy thing https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.ensure_libvirt_rbd_instance_dir_cleanup23:12
sean-k-mooneyi assumed if you set then image backend to ceph those where on ceph too well the config drive and swap at least23:12
mriedemthat's what the libvirt driver assumes as well in some places23:13
mriedembut the rbd image backend just controls the disks, not the other files associated with the instance on the host23:13
*** markvoelker has quit IRC23:13
sean-k-mooney that explain some of my issue on my old newton cloud...23:14
mriedemit's very fun23:14
sean-k-mooneywell the more you know23:14
mriedemcould probably add that to the pile-o-problems for cdent to work on shared storage providers...23:14
sean-k-mooneyi know the file that is created when you suspend an instace was written to the compute node but ya23:15
sean-k-mooneyi wonder how many people over look that as i did23:15
sean-k-mooneyoh i was talking to lyarwood about a multi attach live migration bug earlier in the week23:16
mriedemwell, we don't really help anyone with no mention of this in the docs really, or config options for that matter23:16
mriedemwe should probably have a "things to consider if you plan to use shared storage" section in the admin docs23:17
sean-k-mooneyassuming we dont already have a tempest senario test i think im going to write one and add it to the tempest-full-multinode job23:17
mriedemthere are no tempest tests for live migration + volume multiattach that i'm aware of23:17
mriedemthere are for resize though23:18
sean-k-mooneymriedem: lyarwood was looking at a posibly race. i dont have the context to had but i was thinking a simple test that boots 2 vms with the same multi attach volume and migrate one of the vms back and forth23:19
*** TxGirlGeek has joined #openstack-nova23:19
sean-k-mooneyi think it was related to concurent update of the attachnets23:21
sean-k-mooneymriedem: oh lyarwood filled an upstream bug https://bugs.launchpad.net/nova/+bug/181424523:21
openstackLaunchpad bug 1814245 in OpenStack Compute (nova) "_disconnect_volume incorrectly called for multiattach volumes during post_live_migration" [Undecided,In progress] - Assigned to Lee Yarwood (lyarwood)23:21
*** mlavalle has quit IRC23:22
mriedemso it disconnects the volume from the other server that wasn't live migrated23:24
mriedemif both servers start on the same source host23:24
mriedemyeah?23:24
sean-k-mooneyya23:24
*** awalende has joined #openstack-nova23:25
mriedemi guess that bug says https://review.openstack.org/#/c/551302/ fixes it,23:25
mriedembut ^ needs to be updated23:25
mriedemsean-k-mooney: i don't know that a tempest test would catch this unless that test also ssh's into the guest and tries to verify the volume is still connected on the source host23:27
mriedemand and connected to the live migrated server on the dest host23:27
sean-k-mooneymriedem: yes i planned to ssh in touch a file migrate and check i could read the file on both vms23:27
mriedemthis is similar to https://review.openstack.org/#/c/548356/23:28
mriedemmaybe you want to work on cleaning up ^ first23:28
mriedemi'm sure steve is long gone by now23:28
*** _fragatina has quit IRC23:28
mriedemhe was only on loan from oracle until they got multiattach support in queens23:28
*** awalende has quit IRC23:29
sean-k-mooneyam i can take a look. i have not worked with tempest in a while so i kind of wanted to use this to get familar with it again23:29
*** wolverineav has quit IRC23:29
*** wolverineav has joined #openstack-nova23:30
sean-k-mooneyif that is isshing into the vm it really should be a senario test not an api test... but that is a different issue23:30
sean-k-mooneythe api tests "should" be able to run with the nova fake or libvirt fake driver23:31
*** wolverineav has quit IRC23:34
openstackgerritMerged openstack/nova master: cleanup *.pyc files in docs tox envs  https://review.openstack.org/63521023:42
*** wolverineav has joined #openstack-nova23:45
*** itlinux has quit IRC23:48
*** imacdonn has quit IRC23:49
*** imacdonn has joined #openstack-nova23:49
*** mshaikh99_ has joined #openstack-nova23:50
mshaikh99_I am using OpenStack Queens Release and running Ubuntu 16.04 VM on it, The disk IO seems to be very slow with default cache setting as none for Qemu based virt type23:51
mshaikh99_# time dd if=/dev/zero of=/tmp/test1 bs=20M count=100 conv=fsync;rm /tmp/test1 100+0 records in 100+0 records out 2097152000 bytes (2.1 GB) copied, 296.352 s, 7.1 MB/s  real    4m56.559s user    0m0.100s sys     4m19.716s23:51
mshaikh99_Where is the compute node has the IO as: root@compute1:~# time dd if=/dev/zero of=/tmp/test1 bs=20M count=100 conv=fsync;rm /tmp/test1  100+0 records in 100+0 records out 2097152000 bytes (2.1 GB, 2.0 GiB) copied, 20.724 s, 101 MB/s23:51
mshaikh99_Based on the openstack forum link, I tried to chagne the default cache from none to writeback as suggested https://ask.openstack.org/en/question/6800/qemu-poor-disk-performance/  but it does not change and after the VM hardreboot I still see it as none23:52
mshaikh99_http://paste.openstack.org/show/744655/23:53
openstackgerritMatt Riedemann proposed openstack/nova master: Change live_migration_wait_for_vif_plug=True by default  https://review.openstack.org/63536023:53
mshaikh99_which file and which node?23:54
*** mshaikh99_ has quit IRC23:55
sean-k-mooneymshaikh99_: did you do the hard reboot after restarting nova compute23:56
*** mriedem has quit IRC23:56
*** mshaikh99 has joined #openstack-nova23:56
mshaikh99Hi23:57
mshaikh99Is there an update on the question I asked23:57
sean-k-mooneymshaikh99: if the qcow file is not preaccloated that could be part of the problem23:57
sean-k-mooneymshaikh99: https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.preallocate_images23:58
mshaikh99ok23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!