Monday, 2018-11-19

*** takashin has joined #openstack-nova00:10
*** slaweq has joined #openstack-nova00:11
*** slaweq has quit IRC00:24
*** brinzhang has joined #openstack-nova01:00
*** slaweq has joined #openstack-nova01:16
*** artom has quit IRC01:23
*** slaweq has quit IRC01:24
*** artom has joined #openstack-nova01:28
*** spatel has joined #openstack-nova01:48
*** slaweq has joined #openstack-nova02:16
openstackgerritzhufl proposed openstack/nova master: Add missing ws seperator between words  https://review.openstack.org/61849102:20
*** slaweq has quit IRC02:24
*** tbachman has quit IRC02:27
*** sridharg has joined #openstack-nova02:29
*** sambetts_ has quit IRC02:35
*** mrsoul has quit IRC02:35
*** sambetts_ has joined #openstack-nova02:39
*** owalsh_ has joined #openstack-nova02:44
*** psachin has joined #openstack-nova02:46
*** owalsh has quit IRC02:48
openstackgerritTetsuro Nakamura proposed openstack/nova master: Consider root id is None in the database case  https://review.openstack.org/61330502:52
*** slaweq has joined #openstack-nova03:11
*** tbachman has joined #openstack-nova03:19
*** slaweq has quit IRC03:25
*** udesale has joined #openstack-nova03:51
*** slaweq has joined #openstack-nova04:14
*** janki has joined #openstack-nova04:24
*** slaweq has quit IRC04:25
*** spatel has quit IRC04:26
openstackgerritMerged openstack/nova master: Fix regression in glance client call  https://review.openstack.org/61853404:33
openstackgerritMerged openstack/nova master: doc: Rewrite the console doc  https://review.openstack.org/60614804:34
openstackgerritMerged openstack/nova master: doc: Add minimal documentation for RDP consoles  https://review.openstack.org/60699204:34
openstackgerritMerged openstack/nova master: doc: Add minimal documentation for MKS consoles  https://review.openstack.org/60699304:34
*** ivve has joined #openstack-nova04:37
*** bhagyashris has joined #openstack-nova04:46
*** diga has joined #openstack-nova04:50
*** ircuser-1 has joined #openstack-nova04:51
*** cfriesen has joined #openstack-nova04:55
*** bhagyashris has quit IRC04:59
*** slaweq has joined #openstack-nova05:16
openstackgerritzhufl proposed openstack/nova master: Add missing ws seperator between words  https://review.openstack.org/61849105:17
*** pooja_jadhav has joined #openstack-nova05:22
*** slaweq has quit IRC05:24
*** bhagyashris has joined #openstack-nova05:32
*** ratailor has joined #openstack-nova05:56
*** moshele has joined #openstack-nova06:10
*** slaweq has joined #openstack-nova06:11
*** moshele has quit IRC06:13
*** slaweq has quit IRC06:24
*** Dinesh_Bhor has joined #openstack-nova06:25
*** fanzhang has joined #openstack-nova06:44
*** belmoreira has joined #openstack-nova06:46
*** Luzi has joined #openstack-nova07:03
*** ratailor has quit IRC07:03
*** ratailor has joined #openstack-nova07:05
*** mrsoul has joined #openstack-nova07:18
*** ccamacho has joined #openstack-nova07:20
*** pcaruana has joined #openstack-nova07:24
*** trident has quit IRC07:29
*** belmoreira has quit IRC07:30
*** belmoreira has joined #openstack-nova07:31
*** trident has joined #openstack-nova07:31
*** maciejjozefczyk has joined #openstack-nova07:41
bhagyashrisartom: Hi,07:42
*** hoonetorg has quit IRC07:55
*** alexchadin has joined #openstack-nova07:57
*** sahid has joined #openstack-nova07:58
*** Dinesh_Bhor has quit IRC08:00
*** jangutter has joined #openstack-nova08:06
*** slaweq has joined #openstack-nova08:11
*** hoonetorg has joined #openstack-nova08:13
*** slaweq has quit IRC08:16
*** pooja_jadhav has quit IRC08:38
*** Dinesh_Bhor has joined #openstack-nova08:40
*** mhen has joined #openstack-nova08:45
*** jpena|off is now known as jpena08:48
*** bhdn has quit IRC08:51
*** cfriesen has quit IRC08:56
*** pooja_jadhav has joined #openstack-nova08:58
*** diga has quit IRC09:03
*** s10 has joined #openstack-nova09:10
*** kukacz has quit IRC09:10
*** slaweq has joined #openstack-nova09:11
*** sahid has quit IRC09:13
*** sahid has joined #openstack-nova09:13
*** kukacz has joined #openstack-nova09:13
*** owalsh_ is now known as owalsh09:15
*** alexchadin has quit IRC09:17
melwittfrickler: thanks for jumping on that. agreed we should consider making the ceph job voting09:20
* melwitt returns to PTO09:20
*** takashin has left #openstack-nova09:30
*** jsheeren has joined #openstack-nova09:34
jsheerenhi all I was curious about an issue we ran into09:34
*** derekh has joined #openstack-nova09:34
jsheerenwe have an Opensack Pike env and we're using NFS as shared storage for /var/lib/nova/instances09:35
jsheerenour env consists of 29 compute nodes, each has a mount to the volume for shared storage09:36
kashyapmelwitt: Have fun; and try to disconnect :-)09:36
jsheerenwe noticed the latency on that volume becoming very high and we were getting api timeouts. (with rabbitmq time-outs)09:37
jsheerenour suspicion is the image cache manager as a cause of the latencies.   when we did a rolling restart of nova-compute services (with a 5 min delay between them); the latency was gone09:39
jsheerenand we are not seeing the api time-outs any more.09:39
*** bhagyashris has quit IRC09:40
jsheerenhas anyone had a similar experience?   we increased the image cache manager interval as well, from the default 2400 sec to 4800 sec09:40
*** jaosorior has joined #openstack-nova09:41
*** tssurya has joined #openstack-nova09:42
jsheerenit seems the crawling of the /var/lib/nova tree takes longer than the message time-out.  so we also increased the rcp_response time-out09:43
jsheerenthere are approx 700 instances on the env btw09:43
*** maciejjozefczyk has quit IRC09:50
*** cdent has joined #openstack-nova09:51
openstackgerritMatthew Booth proposed openstack/nova master: Call migrate_volume_completion after cinder migrate  https://review.openstack.org/61871709:55
*** brinzhang has quit IRC09:58
*** brinzhang has joined #openstack-nova09:59
*** brinzhang has quit IRC10:00
*** brinzhang has joined #openstack-nova10:01
*** sridharg has quit IRC10:05
*** k_mouza has joined #openstack-nova10:06
*** brinzhang has quit IRC10:08
*** brinzhang has joined #openstack-nova10:09
*** sean-k-mooney has quit IRC10:15
*** Dinesh_Bhor has quit IRC10:17
*** sean-k-mooney has joined #openstack-nova10:23
*** betherly has joined #openstack-nova10:27
*** toabctl has quit IRC10:33
*** maciejjozefczyk has joined #openstack-nova10:35
*** moshele has joined #openstack-nova10:36
tobias-urdinmelwitt: we did hit this bug in rocky today https://review.openstack.org/#/c/591080/10:39
tobias-urdinwould that be acceptable for backport?10:39
*** panda|rover|off is now known as panda|rover10:49
*** cdent has quit IRC10:57
*** udesale has quit IRC11:06
openstackgerritVlad Gusev proposed openstack/nova stable/rocky: Handle unicode characters in migration params  https://review.openstack.org/61873411:11
*** janki has quit IRC11:18
openstackgerritRadoslav Gerganov proposed openstack/nova master: VMware: implement trigger crash dump  https://review.openstack.org/61873611:20
*** pooja_jadhav has quit IRC11:20
openstackgerritJason SUN proposed openstack/nova master: For instances end up in cell0, set its availability_zone to None.  https://review.openstack.org/61873711:25
*** rodolof has joined #openstack-nova11:29
*** moshele has quit IRC11:34
*** k_mouza has quit IRC11:38
*** k_mouza has joined #openstack-nova11:39
*** moshele has joined #openstack-nova11:40
*** moshele has quit IRC11:44
*** sridharg has joined #openstack-nova11:44
*** janki has joined #openstack-nova11:49
*** pooja_jadhav has joined #openstack-nova12:03
tobias-urdinis it possible to flush the entries in instance_system_metadata for an instance manually without breaking something? there is cached content on user and project which is incorrect12:09
tobias-urdini assume the only other wait to force a rewrite of those values in the xml would be a hard reboot, but not sure that would do the trick either12:09
*** moshele has joined #openstack-nova12:17
*** cdent has joined #openstack-nova12:19
*** moshele has quit IRC12:23
*** brinzhang has quit IRC12:24
*** ratailor has quit IRC12:24
*** sahid has quit IRC12:25
*** ondrejme has joined #openstack-nova12:25
*** jaypipes has joined #openstack-nova12:26
*** rodolof has quit IRC12:29
*** rodolof has joined #openstack-nova12:30
*** jpena is now known as jpena|lunch12:31
*** ccamacho has quit IRC12:31
*** sahid has joined #openstack-nova12:31
*** ccamacho has joined #openstack-nova12:33
*** mriedem has joined #openstack-nova12:55
*** dave-mccowan has joined #openstack-nova12:56
*** moshele has joined #openstack-nova12:56
*** udesale has joined #openstack-nova13:08
*** moshele has quit IRC13:09
*** udesale has quit IRC13:11
*** udesale has joined #openstack-nova13:11
*** jsheeren has quit IRC13:12
*** cdent has quit IRC13:13
*** sapd1 has joined #openstack-nova13:13
*** k_mouza has quit IRC13:14
*** tetsuro has joined #openstack-nova13:20
*** tbachman has quit IRC13:20
*** takashin has joined #openstack-nova13:21
openstackgerritSurya Seetharaman proposed openstack/nova master: Add os_compute_api:servers:create:cell_down policy  https://review.openstack.org/61478313:25
*** jpena|lunch is now known as jpena13:29
*** k_mouza has joined #openstack-nova13:31
*** k_mouza_ has joined #openstack-nova13:33
*** k_mouza has quit IRC13:36
*** dave-mccowan has quit IRC13:36
*** psachin has quit IRC13:37
openstackgerritSurya Seetharaman proposed openstack/nova master: Add os_compute_api:servers:create:cell_down policy  https://review.openstack.org/61478313:41
*** tbachman has joined #openstack-nova13:45
openstackgerritMatt Riedemann proposed openstack/nova master: Add DownCellFixture  https://review.openstack.org/61481013:46
*** mvkr has quit IRC13:47
*** tbachman_ has joined #openstack-nova13:48
*** moshele has joined #openstack-nova13:50
*** cdent has joined #openstack-nova13:50
*** tbachman has quit IRC13:51
*** tbachman_ is now known as tbachman13:51
*** moshele has quit IRC13:54
*** cdent has quit IRC13:55
openstackgerritTakashi NATSUME proposed openstack/nova master: Add API ref guideline for body text  https://review.openstack.org/60562814:03
*** moshele has joined #openstack-nova14:03
*** ccamacho has quit IRC14:04
*** cdent has joined #openstack-nova14:05
*** moshele has quit IRC14:06
*** ccamacho has joined #openstack-nova14:08
*** lbragstad has joined #openstack-nova14:09
*** awaugama has joined #openstack-nova14:11
openstackgerritJosephine Seifert proposed openstack/nova-specs master: Spec for the Nova part of Image Encryption  https://review.openstack.org/60869614:12
openstackgerritTakashi NATSUME proposed openstack/nova master: Fix server query examples  https://review.openstack.org/61683414:32
*** k_mouza has joined #openstack-nova14:34
*** mvkr has joined #openstack-nova14:35
*** k_mouza_ has quit IRC14:37
*** mchlumsky has joined #openstack-nova14:43
*** burt has joined #openstack-nova14:44
*** tetsuro has quit IRC14:55
openstackgerritElod Illes proposed openstack/nova master: Transform scheduler.select_destinations notification  https://review.openstack.org/50850614:56
mriedemjackding: some small changes for https://review.openstack.org/#/c/605902/ and then i'm +214:56
mriedemplus the glance image docs14:56
*** cfriesen has joined #openstack-nova14:58
*** mlavalle has joined #openstack-nova15:00
*** mdbooth has quit IRC15:01
*** k_mouza_ has joined #openstack-nova15:01
*** sapd1 has quit IRC15:02
*** k_mouza__ has joined #openstack-nova15:02
*** Luzi has quit IRC15:03
jackdingmriedem: ok thanks15:04
*** spatel has joined #openstack-nova15:04
spatelsean-k-mooney: morning!!!15:04
*** k_mouza has quit IRC15:04
*** takashin has left #openstack-nova15:05
*** k_mouza_ has quit IRC15:06
*** sapd1 has joined #openstack-nova15:09
*** slaweq_ has joined #openstack-nova15:10
*** slaweq has quit IRC15:12
*** spatel has left #openstack-nova15:13
*** k_mouza__ has quit IRC15:17
*** spatel_ has joined #openstack-nova15:17
sean-k-mooneyhello15:19
spatel_how are you ?15:19
spatel_i had question related multi-queue for virt-io15:20
sean-k-mooneynot too bad. just rootcaused an issue in some logs so feeling quite good15:20
*** sapd1 has quit IRC15:20
spatel_if i have 8 vCPU on instance in that queue what value should be good to set in ethtool -L eth0 combined <N> ?15:21
sean-k-mooney815:22
sean-k-mooneywhen you enable multiqueue openstack will allocate a tx and rx pair per core15:22
spatel_i thought i should keep some CPU core for other work15:24
spatel_This is what i am doing so far may be i am wrong here.. if i create 16 vCPU machine then i assign only 8 queue15:24
spatel_if i create 8 vCPU machine then i create 4 queue15:25
spatel_am i doing something wrong doing that or best practice is to give vCPU = Queue size15:25
sean-k-mooneyassinging a queue does nto mean you are assinging a cpu to manage it15:26
sean-k-mooneythe vswitch cannot know which queue the os is using so it will assume it can enqueue packets to all of them15:26
sean-k-mooneyif you dont config the kernel to pool on all queue you will miss packets15:27
*** jaypipes has quit IRC15:27
sean-k-mooneyyou can have one guess cpu pool all queue if you want15:27
spatel_so best option is to give all queue right?15:28
spatel_8 vCPU = 8 queue ?15:28
sean-k-mooneyby assingin 1 queue per cpus its allows apps to transmit without haveing to take locks as you can use the cpu id as a queue index15:28
sean-k-mooneyyes15:28
sean-k-mooneybut your application that is processign the traffic in the vm can be confied to only 4 cpus if you like15:29
sean-k-mooneyyou just need to configre the app to use 2 queues per core in that case15:29
sean-k-mooneyif you are using the kernel driver this is all handeled internally i belive15:29
spatel_I thought queue bind with vCPU so if you specify 4 queue size then only 4 vCPU parellel work on packet processing..15:30
spatel_that is why i was reserving 4 vCPU for OS and 4vCPU for queue (for networking work)15:31
sean-k-mooneythere are 2 level here. the host level were we will allocate 1 queu per vcpu and the guest level15:31
spatel_https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/libvirt-virtiomq.html15:32
spatel_Host level ?? never heard that15:32
sean-k-mooneywhat im trying to say is that nova will configure qemu with 1 queu per vcpu form the host/hyperviour perspective15:33
sean-k-mooneyprovided you enable multique that is15:33
spatel_hmm!15:34
sean-k-mooneyas such if you do not set ethtool -L <NIC> combined $(nproc)15:34
sean-k-mooneyyou will get incorrect behavior15:35
tobias-urdinmriedem: could i steal some of your time, perhaps you know if it flushed the system metadata cache15:35
tobias-urdin12:58 < tobias-urdin> is it possible to flush the entries in instance_system_metadata for an instance manually without breaking something? there is cached content on user and project which is incorrect15:35
tobias-urdin12:58 < tobias-urdin> i assume the only other wait to force a rewrite of those values in the xml would be a hard reboot, but not sure that would do the trick either15:35
spatel_i think slowly i am following you... what you trying to say.. sean-k-mooney15:35
openstackgerritTobias Urdin proposed openstack/nova stable/rocky: Handle unicode characters in migration params  https://review.openstack.org/61873415:35
sean-k-mooneybut within the guest the application that process packts form that interface can pin to 1-n core independly form the number of quese that there are15:35
spatel_oh!! got it.. what you saying..15:36
spatel_i was wrong about queuing logic...15:36
mriedemtobias-urdin: there is no external REST API for changing things within the instance system_metadata,15:37
spatel_let me change my all instance to use all nproc15:37
mriedemand what goes in there depends on the instance and operation, and manually messing with it is probably a bad idea given it's system data15:37
mriedemtobias-urdin: so you'd have to be more specific about the incorrect sysmeta data that needs to be fixed15:38
spatel_sean-k-mooney: thanks again for saving me!!!15:38
tobias-urdinmriedem: we are affected by https://review.openstack.org/#/c/618734/ but we have renamed the project until then but live migrations fails since it's wrong in the XML15:38
sean-k-mooneyspatel_: no worries this is rather difficult to figure out all at once15:38
tobias-urdinso i was not sure if doing a hard reboot would use the cached instance_system_metadata or not, it does rebuild the XML but perhaps with the wrong data15:38
tobias-urdinperhaps better to just test it out in a test environment15:38
spatel_in document is not very clear! anyway i am totally on it now15:39
tobias-urdinit's probably not a good idea to DELETE those rows in instance_system_metadata15:39
mriedemtobias-urdin: no that's probably not a great idea15:39
mriedemtobias-urdin: what are the bad fields? user name and project name?15:40
*** k_mouza has joined #openstack-nova15:40
mriedemi.e. owner_user_name and owner_project_name in the xml?15:40
sean-k-mooneyspatel_: ya this was a topic at the summit. we need an nfv tuning guide with examples as the docs for these advanced feature are... lacking clarity to put it nicely15:41
tobias-urdinmriedem: yeah the owner project name in the xml15:41
mriedemi'm guessing _get_guest_config_meta in the libvirt driver is the thing putting the bad data in the xml15:42
mriedemif you deleted owner_user_name and owner_project_name from the instance_system_metadata, we would put "N/A" in the xml15:42
sean-k-mooneyspatel_: i as a developer often use the specs to understand what we intended to enable but end users should not needed to read them to consume the features and use them properly.15:42
spatel_++15:43
*** moshele has joined #openstack-nova15:43
mriedemtobias-urdin: and those fields are put into the system_metadata only at server create time, not reboot15:44
mriedemso even if you changed the user name / project name in keystone, that wouldn't be reflected in the instance sysmeta15:44
tobias-urdinmriedem: thanks! then i'll apply the patch manually on the affected nodes until it lands in stable/rocky15:46
*** priteau has joined #openstack-nova15:51
dansmithmriedem: tobias-urdin +W15:52
*** moshele has quit IRC15:52
*** jaypipes has joined #openstack-nova15:53
*** slaweq_ has quit IRC15:54
*** slaweq_ has joined #openstack-nova15:54
dansmithmriedem: jaypipes sean-k-mooney: I think we need to give some clarity to sundar here: https://review.openstack.org/#/c/603955/10/specs/stein/approved/nova-cyborg-interaction.rst15:55
dansmithI am majorly opposed to documenting all the cyborg and os-acc APIs and functions in that spec15:55
*** dklyle has quit IRC15:55
sean-k-mooneydansmith: just hopping on a call in 5 mins but sure what had you in mind15:56
sean-k-mooneydansmith: yes i suggested splitting it into 3 specs a while ago15:56
dansmithL324 specifically15:56
dansmithsean-k-mooney: even if it gets split, those definitions don't belong in one of our specs, IMHO15:56
sean-k-mooney am i highlevel discription of the os-acc that nova will consume would be nice but the cyborg apis dont need to be in the spec for sure15:57
sean-k-mooneyi would like to know the data structure that are passed the the fuction that should be called at diffrent point by nova but the internal can live in cyborg specs15:58
dansmithsean-k-mooney: the data structures are fine, but you asked him to add more api signatures where I asked him to completely remove them15:59
sean-k-mooneydansmith: i think the os-vif spec got it about right https://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/os-vif-library.html15:59
dansmithwhich is why we need to converge so he knows what to do15:59
jaypipessean-k-mooney: I think I was pretty clear in Denver and elsewhere that I think the Cyborg team first needs to have some workable low-level device plumbing library (what I *thought* os-acc was supposed to be) before Nova even becomes part of the conversation. I view os-acc as very similar to os-vif, and I'm disappointed that there is even discussion about REST APIs or anything like that instead of designing an easy to use, hard-to-abuse low-level15:59
jaypipesdevice plugging lib.15:59
sean-k-mooneydansmith: well that was because he was documenting the api and did not discribe the device profile enpoint16:00
sean-k-mooneyso either remove the api enpoitn entirely which im fine with or include that one16:00
sean-k-mooneyjaypipes: yes i agree with that16:00
jaypipessean-k-mooney: totally unnecessary at this point, IMHO.16:00
jaypipessean-k-mooney: (the REST API or anything like that)16:01
sean-k-mooneyjaypipes: ya so let ask for all the cyborg rest apis to be removed and state it will be handeled by os-acc16:01
dansmithjaypipes: can you comment on the spec (L324ish)? I've already voiced there and so another "remove all this" would be good..16:01
jaypipessean-k-mooney: the path to success with Cyborg is to mimick os-vif, not Mogan.16:01
sean-k-mooneyjaypipes: yes i agree16:01
sean-k-mooneyjaypipes: anyway i need to jump on a call16:02
jaypipesdansmith: ok. I've already made a number of those comments but will do so again16:02
dansmithjaypipes: I know, and I appreciate it.. L324 is where there was a conflicting "Do X" and "Don't do X"16:02
sean-k-mooneydansmith: jaypipes feel free to link to this irc convo or just state im +1 on removing the api enpoint too16:02
jaypipesdansmith: ack16:02
dansmithjaypipes: and sundar expressed confusion there, so..16:02
cdentI spoke with sundar briefly in berlin and he felt like he was getting different messages from jaypipes and dansmith. Both efried and I suggested he show up here to be a part of these discussions if/when he had a chance.16:03
*** dklyle has joined #openstack-nova16:03
*** tbachman has quit IRC16:04
*** pcaruana has quit IRC16:07
dansmithcdent: I imagine you mean between me and sean-k-mooney or jaypipes and sean-k-mooney16:08
dansmithhe didn't even acknowledge my comments on the spec,16:08
cdentdansmith: I don't.16:08
dansmithand jaypipes and I didn't comment on most of the same lines16:08
*** moshele has joined #openstack-nova16:08
cdentI'm reporting merely on what sundar said to me, not what's happening on the spec16:09
cdentThe only attention I've given to the spec is to point out where the proposed API is not right16:09
*** s10 has quit IRC16:11
*** moshele has quit IRC16:14
*** udesale has quit IRC16:16
*** Sundar has joined #openstack-nova16:17
*** jamesdenton has joined #openstack-nova16:18
mriedemand i haven't looked at it at all16:20
mriedemi'm kind of waiting for ^ to settle before digging into it16:21
*** janki has quit IRC16:26
*** mchlumsky has quit IRC16:31
*** mchlumsky has joined #openstack-nova16:34
*** tbachman has joined #openstack-nova16:36
jaypipesmriedem: the nova-cyborg thing?16:39
mriedemye16:40
mriedem*yes16:40
*** sapd1 has joined #openstack-nova16:45
*** ccamacho has quit IRC16:46
*** ccamacho has joined #openstack-nova16:46
*** rodolof has quit IRC16:51
sean-k-mooneycdent: there were some mixed messages from me vs dan and jay. i wanted them to either document everything or drop the api stuff. dan wanted the api stuff remvoed and i think jay agreed16:51
*** moshele has joined #openstack-nova16:52
sean-k-mooneycdent: i was fine with removing the cyborg api stuff which sundar may not have picked up on but i dislike have only some of the enpoints int he spec16:52
*** moshele has quit IRC16:53
sean-k-mooneyi think dansmith jaypipes and i all agree  we would like to see the integration follow the os-vif and os-bricks model where os-acc become the nova integration point16:53
cdentsean-k-mooney: I said to him that it would be easier to evaluate the API if it were all in one place16:53
cdentAnd if the API isn't needed at this stage, then cool16:53
cdentI was simply trying to help him navigate what has proven to be a strenuous process16:54
sean-k-mooneycdent: that is true but the one place could be a cyborg spec that we provide input into yes?16:54
cdentyes, if the one place were to happen, then in cyborg would be the place16:55
*** k_mouza_ has joined #openstack-nova16:57
efriedproblem there being that they've had very little input on the cyborg specs from outside the cyborg team.16:58
efriedI think it may have been on my recommendation (which was for that reason ^) that Sundar put this one into nova-specs.16:58
*** k_mouza__ has joined #openstack-nova16:58
*** Sundar has quit IRC16:58
dansmiththere really shouldn't need to be a lot of input from us on cyborg APIs I don't think16:59
dansmithif we keep the discussion on the nova side to the workflow and interaction with the library,16:59
sean-k-mooneyefried: well you and i have both left quite a lot of feedback at different times but i dont think alot of it has been incorperated. some definetly has but we have had the same converstaton several times16:59
dansmiththe server-side stuff should be discernible from that I think16:59
dansmithsean-k-mooney: yeah, definitely feel like we've gone in circles a lot on this, approaching exhaustion16:59
*** k_mouza has quit IRC17:00
*** priteau has quit IRC17:00
openstackgerritElod Illes proposed openstack/nova master: Transform scheduler.select_destinations notification  https://review.openstack.org/50850617:01
*** k_mouza_ has quit IRC17:02
*** k_mouza__ has quit IRC17:04
*** tssurya has quit IRC17:07
*** sapd1 has quit IRC17:07
jaypipesjangutter: I really like the "before and after" sections in https://review.openstack.org/#/c/607610/3/specs/stein/approved/generic-os-vif-offloads.rst (e.g. lines 133-177). makes it much easier to see how the changes affect things. ++17:10
*** itlinux has joined #openstack-nova17:12
*** imacdonn has quit IRC17:17
*** imacdonn has joined #openstack-nova17:17
*** mmethot has joined #openstack-nova17:21
*** k_mouza has joined #openstack-nova17:24
sean-k-mooneyjaypipes: dansmith i responded on the cyborg spec to say please remove the cyborg api section.17:25
*** sahid has quit IRC17:25
sean-k-mooneyjaypipes: dansmith also do we all agree the indirect usecase where the acclerator is not attach to the vm should not be in the spec?17:25
jaypipessean-k-mooney: I just finished up with jangutter's spec on os-vif object model changes. adding some comments to the cyborg spec now...17:26
sean-k-mooneymy understanding is that the instance will not be the consumer of the devices in placement and since they are not attach to the nova instance nova should not care17:26
cdentefried: we (you?) need to come up with a plan for replacing the tests that use rp_object as added in https://review.openstack.org/#/c/560444/17:27
sean-k-mooney jaypipes: oh ya i need to go back to that one17:27
sean-k-mooneyjaypipes: am one point i made on the code which is not in the spec is we should not use the term datapath offlod since the representor netdev is actull part of the control plane.17:28
jaypipessean-k-mooney: I said on the spec that I think "OffloadBase" is fine and removing "Datapath" would be good.17:30
sean-k-mooneyi realise when i clicked the link i have only looked at the code for this so im currently readign the spec for the first tiem17:31
sean-k-mooneyjaypipes: if you have not seen it can you reivew this comment on the code to see if you agree https://review.openstack.org/#/c/572081/9/os_vif/objects/vif.py@376 i will see how that relates to the spec17:35
*** pooja_jadhav has quit IRC17:36
*** rodolof has joined #openstack-nova17:36
*** rodolof has quit IRC17:40
*** rodolof has joined #openstack-nova17:40
efriedcdent: That should be easy peasy. Did you want me to do that in nova, placement, or both?17:41
*** rodolof has quit IRC17:41
cdentefried: it only exists in nova17:41
efriedstand by.17:41
*** rodolof has joined #openstack-nova17:41
* cdent stands on one leg17:42
*** pcaruana has joined #openstack-nova17:44
openstackgerritMerged openstack/nova stable/rocky: Handle unicode characters in migration params  https://review.openstack.org/61873417:44
*** k_mouza has quit IRC17:46
*** sridharg has quit IRC17:49
*** rodolof has quit IRC17:51
openstackgerritEric Fried proposed openstack/nova master: Nix refs to ResourceProvider obj from libvirt UT  https://review.openstack.org/61878617:51
efriedcdent: There ya go ^17:51
cdentrad17:51
*** rodolof has joined #openstack-nova17:51
cdentthanks17:52
efriedThere's more refactoring that could be done there, but this is roughly the easiest.17:52
efriedwithout being too hacky17:52
efriedcdent: BTW, pretty sure the origin of the test code was bhagyashris. This doesn't look like my style at all.17:54
cdentefried: noted. I just went by git blame when raising the "alarm`'17:54
efriedyup, and she's not around atm anyway. Easy enough fix.17:54
*** rodolof has quit IRC17:56
*** rodolof has joined #openstack-nova17:56
*** derekh has quit IRC18:00
efriedjaypipes, mriedem, bauzas: ----^ easy one18:01
*** jpena is now known as jpena|off18:03
openstackgerritChris Dent proposed openstack/nova master: Use external placement in functional tests  https://review.openstack.org/61794118:07
openstackgerritChris Dent proposed openstack/nova master: WIP: Delete the placement code  https://review.openstack.org/61821518:07
*** tbachman has quit IRC18:07
cdentfeh, that's not going to work after a merge18:08
*** dtantsur is now known as dtantsur|afk18:09
*** rodolof has quit IRC18:28
*** tbachman has joined #openstack-nova18:29
*** READ10 has joined #openstack-nova18:40
larsksHey folks.  Should disk-related instance resource quotas (such as quota:disk_write_bytes_sec) be applied when booting from a volume?18:48
*** k_mouza has joined #openstack-nova18:49
*** k_mouza has quit IRC18:54
*** Swami has joined #openstack-nova19:13
*** mvkr has quit IRC19:20
cfriesenlarsks: I think it's supposed to be applied, at least for libvirt/qemu.19:21
sean-k-mooneycfriesen: yes it is but i just tested with devstack and it does not work on master19:26
sean-k-mooneycfriesen: larsks  moved to public pastbin http://paste.openstack.org/show/735692/19:27
sean-k-mooneyim link 95% sure this used to work19:28
cfriesensean-k-mooney: I think it's "quota:write_bytes_sec"19:28
cfriesenno, that webpage was wrong19:29
sean-k-mooneyquota:disk_write_bytes_sec is what horizon set19:29
*** moshele has joined #openstack-nova19:29
sean-k-mooneyill set all the limits and see what hapes perhaps its just that one.  i think this works with the ceph backend but maybe for boot form volume you have to use cinder qos19:30
sean-k-mooneycfriesen: larsks nope still ignored http://paste.openstack.org/show/735693/19:34
openstackgerritJack Ding proposed openstack/nova master: Add HPET timer support for x86 guests  https://review.openstack.org/60590219:40
*** moshele has quit IRC19:43
larskscfriesen: thanks, and sean-k-mooney thanks for checking.  What I'm curious about is whether or not that is expected behavior...19:46
larsksI.e., should I open a bug for this?19:46
sean-k-mooneylarsks: it is expect to work and yes19:46
larsksAck, I'll do that right now, then.19:46
sean-k-mooneybut we also advise peole not to use falvor qouta any more even though we have not deprecated them offically19:47
larsksOh, that's news. Why is that?19:47
sean-k-mooneybecause cinder and neutron both have there own qos apis which are richer and better tested19:47
larsksOkay.  Being able to associate a collection disk/network/etc quotas with a flavor is very convenient.19:48
larsksBut I can see that having these resource quotas managed in different places could lead to confusion and duplication of effort.19:49
sean-k-mooneyit is but its legacy functionality that was supperseed by qos apis in the services19:49
sean-k-mooneythat daid if you are not using cinder or neutron an want that fuctionality the falvor were the only way to do it19:49
larsksFair enough.  Do you know if there is documentation that would suggest avoiding flavor quotas? I may need to convince someone to do so.19:49
sean-k-mooneylarsks: not that im aware of but most people dont know they  exist and we just dont activly publicise that fact to new users19:50
larsksGot it.  Okay, thanks!19:50
cdentIs anyone else seeing HackingTestCase.test_check_doubled_words fail on py36 on ubuntu 18.04? It's not failing on macos19:56
efriedcdent: I just ran it locally on trusty and it ran okay.19:58
efriedI noticed it breaking here: https://review.openstack.org/#/c/618786/  <== is that where you were seeing it, or somewhere else?19:59
*** jobewan has joined #openstack-nova19:59
cdentfirst on my placement removal patches, and then on master19:59
*** rodolof has joined #openstack-nova20:00
*** mvkr has joined #openstack-nova20:09
*** mchlumsky has quit IRC20:10
*** mchlumsky has joined #openstack-nova20:11
*** rodolof has quit IRC20:11
cdentweird, it's a problem in 3.6 but not 3.720:12
*** rodolof has joined #openstack-nova20:12
*** rodolof has quit IRC20:16
*** rodolof has joined #openstack-nova20:17
cdentoh rich, it passes in stestr in 3.7, but not with testtools.run or unittest20:18
* cdent flips a table20:22
*** burt has quit IRC20:26
*** rodolof has quit IRC20:26
*** rodolof has joined #openstack-nova20:27
*** rodolof has quit IRC20:31
*** rodolof has joined #openstack-nova20:32
larskssean-k-mooney: Looks like there is already a bug, from 2016 :). https://bugs.launchpad.net/nova/+bug/144563720:33
openstackLaunchpad bug 1445637 in OpenStack Compute (nova) "Instance resource quota not observed for non-ephemeral storage" [Wishlist,Opinion]20:33
sean-k-mooneylarsks: ah ok let me repriorise it for you20:35
*** READ10 has quit IRC20:35
larsksI don't know, maybe at this point it's just a doc bug and the docs should say "use cinder and neutron resource controls".20:36
sean-k-mooneyi have set it to confirmed and low. we can propse a docs update as a resolution to the but if we like20:37
*** rodolof has quit IRC20:38
*** rodolof has joined #openstack-nova20:39
*** rodolof has quit IRC20:41
*** rodolof has joined #openstack-nova20:42
*** rodolof has quit IRC20:56
larskssean-k-mooney: ...but flavor quotas *are* still the correct mechanism for controlling cpu and memory limits?21:01
sean-k-mooneylarsks: cpu and memory limits?21:03
cdentefried: it's because of this https://bugs.python.org/issue33899 which gets python 3.6.7 and newer versions of python 3.7. I'll make a bug and a patch21:03
larskssean-k-mooney: right, like quota:cpu_quota.21:04
larskssean-k-mooney: so really just cpu.21:04
sean-k-mooneylarsks: that is also something that in general is not used21:04
larsksOkay. Is there an alternative for that, or is that just something that was considered less useful?21:05
sean-k-mooneylarsks: most of those values are from lxc21:05
sean-k-mooneyor vm ware21:05
sean-k-mooneymany of them never work with libvirt/kvm cpu_shares did but not the rest21:06
sean-k-mooneythe same is true fo the memory limits21:06
sean-k-mooneyhttps://docs.openstack.org/nova/latest/user/flavors.html21:06
sean-k-mooneyall of the flavor quota atibute came form the vmware driver originally21:07
larsksOkay. So effectively the best practice is not to use instance resource limits for anything, and rely on the per-service limits (e.g., cinder, neutron) where available. Is that accurate?21:09
sean-k-mooneyif you want to have interop between different public cloud then you could not rely on that feature existing21:10
sean-k-mooneyas an operator you can set them but it may cause issues for your users if you also expose the service qos apis21:10
cdentefried: https://bugs.launchpad.net/nova/+bug/1804062 the thing that I thought would fix it does not21:10
openstackLaunchpad bug 1804062 in OpenStack Compute (nova) "test_hacking fails for python 3.6.7 and newer" [Undecided,New]21:10
cdentso still thinking21:10
larskssean-k-mooney: fair enough. Are there any plans to deprecate the feature in the near future?21:10
cdentmriedem: to distract you, that bug ^ is real fun21:11
sean-k-mooneylarsks: again it kind of a if its not broken dont fix it or delete it situation21:11
sean-k-mooneylarsks: i dont think so but it has been mentioned in the past21:11
sean-k-mooneylarsks: a better way to look at it is it unlikely that we would add more quota: extraspec in the future21:12
larsksOkay, that makes sense. Thanks for spending some time on this!21:13
cfriesenis it just me or is it not intuitive that the way to get a vnc session to a server using the "openstack" command is "openstack console url show" rather than an "openstack server console show" or similar?21:14
sean-k-mooneycfriesen: i did fine it weird too21:15
sean-k-mooneyits there to differenceate between the url or the log21:15
sean-k-mooneybut i would have gon with opesntack server console show and openstack server log show21:15
sean-k-mooneycfriesen: sorry tehy are console log show and console url show21:16
sean-k-mooneybut yes openstack server console show and openstack server log show would have made more sense to me21:17
*** rmart04 has joined #openstack-nova21:18
*** rmart04 has quit IRC21:19
mriedemcfriesen: likely a question for dtroyer in #openstack-sdks21:19
mriedemcdent: so we're intermittently hitting that in py36 jobs?21:20
cdentyes21:20
cdentseems to depend on whether the node has a recently updated python. 3.6.7 was released to ubuntu not long ago21:21
sean-k-mooneycdent: is that happening in nova or in the hacking repo?21:21
cdentnova21:21
cdentit's just the one test21:21
cdentthe root is that having a newline token being added on the end is causing pep8 to think it needs to do a "logical check"21:22
sean-k-mooneycan you past the test name again ill try and run it on python 3.6 on centos21:22
cdentwhich means it bypasses the physical checks that happen otherwise21:22
cdentsean-k-mooney: nova.tests.unit.test_hacking.HackingTestCase.test_check_doubled_words21:22
cdentit ought to pass fine in <3.6.721:22
mriedemhttp://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22testtools.matchers._impl.MismatchError%3A%20%5B(1%2C%200%2C%20'N343')%5D%20!%3D%20%5B%5D%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20build_name%3A%5C%22openstack-tox-py36%5C%22&from=7d21:22
cdentyeah, so always bionic21:23
mriedemyup21:23
mriedemdo we want to skip the test for now?21:23
cdentif you're happy with that, that's probably the easiest way to go21:23
mriedemor at least skip on py3621:24
sean-k-mooneypython 3.6 will be the defult on rhel8 also as per doug hellmann's mail earilier today21:25
cdentyeah, I'll do a skipIf21:26
cdentone moment please21:26
sean-k-mooneyim testing with Python 3.6.5 on centos 7 and it passed21:26
cdentyeah, that's expected21:27
sean-k-mooneyoh it needs to be 3.6.7+ to fail21:28
*** awaugama has quit IRC21:34
openstackgerritChris Dent proposed openstack/nova master: Skip double word hacking test  https://review.openstack.org/61884321:43
cdentmriedem, efried, sean-k-mooney ^21:43
* sean-k-mooney clicks21:43
sean-k-mooneyis that really a nova bug or a hacking bug?21:44
cdentthe failing test is because of changes to the tokenize stdlib. The hacking check is in nova itself, not in hacking21:45
sean-k-mooneyi guess we shoudl make the test support both the old and new api21:45
cdentat least the way nova is experiencing the problem21:45
cdentsean-k-mooney: I tried and failed to do that because it is in the guts of pep8 where the problem happens21:45
cdentall hacking has is a regex21:45
cdentthe presence of the added NEWLINE token causes pep8 to believe it needs to make a logical instead of physical check21:46
sean-k-mooneynot sure i follow but then again i have not looked at the python bug yet21:47
cdentthere's little we can do from our side of the world to make it not add the NEWLINE21:47
cdentlooking at the python bug (and the related tracker for api breakages it causes) will make things much more clear21:47
*** jobewan has quit IRC21:47
sean-k-mooneyso just to be clear on what line in test_chesck_double_words is it addint the new line21:49
sean-k-mooneythe + operator21:49
cdentat the end of the entire string21:49
cdentand it isn't adding a new line21:49
cdentit is adding a NEWLINE token when tokenizing21:49
sean-k-mooneyoh21:49
sean-k-mooneythat seams like a think it should not be doing... but ok21:50
sean-k-mooneyanyway you temp fix makes sense21:50
cdentyeah, you'd think so, but someone decided that the method should behave more like C21:50
cdentthe comments on the python bug are amusing21:50
sean-k-mooneyc ends with \021:50
sean-k-mooneye.g. a 0 byte21:50
sean-k-mooneynot a new line21:50
*** pcaruana has quit IRC21:51
cdentright, but it tokenizes a NEWLINE21:51
cdentapparently21:51
sean-k-mooneyso it does not match c either in terms of string termination of const char* s21:51
cdentit's not about string termination21:51
cdentit is about code tokenizing21:51
cdentplease, read the bug21:51
cdentit will all become clear21:52
sean-k-mooneyyes i was as we were chatting "As was pointed out in https://bugs.python.org/issue33766 there is an edge case in the tokenizer whereby it will implicitly treat the end of input as a newline."21:52
efriedcdent: What is [1] in the commit message?21:53
cdentthat link that sean-k-mooney just pointed at, did I forget to add it? le sigh21:54
efriednot https://bugs.python.org/issue33899 ?21:54
openstackgerritChris Dent proposed openstack/nova master: Skip double word hacking test  https://review.openstack.org/61884321:54
cdentargh too many bugs21:54
sean-k-mooneyefried: the openstack bug points to the python bug21:55
openstackgerritChris Dent proposed openstack/nova master: Skip double word hacking test  https://review.openstack.org/61884321:55
sean-k-mooneybut yes the commit does not have a bug link21:55
sean-k-mooneyhehe ok now it does21:56
efriedso none of those is https://bugs.python.org/issue3376621:56
cdentthere we go, now it links to the two python bugs I wanted to link to, neither of which are the one that sean pointed to (which was an originating bug)21:56
cdentindeed21:56
*** ivve has quit IRC21:56
* cdent can't seem to stay out of 13 hour days21:56
efriedNot having dug any deeper, I'll point out that ``echo 'print("a");print("b")' > test.py`` does indeed end with a newline.21:57
sean-k-mooneyyes https://bugs.python.org/issue35107 is tracking the fallout of fixing https://bugs.python.org/issue33899 which ended up breaking backward compatiblity21:57
efriedefried@efried-ThinkPad-W520:~$ echo 'print("a");print("b")' > /tmp/test.py21:57
efriedefried@efried-ThinkPad-W520:~$ od -c /tmp/test.py21:57
efried0000000   p   r   i   n   t   (   "   a   "   )   ;   p   r   i   n   t21:57
efried0000020   (   "   b   "   )  \n21:57
efried000002621:57
cdenti don't think that's the point efried21:57
efriedI'm sure it's not.21:57
efriedhence the 'not having dug any deeper'21:57
cdentit's the same deal: discussion on that bug revealed the different between a newline character and a NEWLINE token and ways in which tokenizer failed to produce NEWLINE in some cirtical areas21:59
cdentthe fix, however, was too hasty21:59
sean-k-mooneyya and they should not have backporte it to python 2.7 luckally they reverted it on the 2.7 branch22:00
efriedcdent: Did this start appearing because we *just* started testing with py3.6.7?22:01
cdentyes22:01
cdentubuntu bionic got it within the past few days22:01
sean-k-mooneylooking at https://bugs.python.org/issue35107 they are still discusiing what the behavior should be but i agreee with them that you should be able to round trip22:01
cdentand our images updated recently22:01
sean-k-mooneyam i missrembereing or did ubunut ship with a beta python version at lauchch or was that just for 3.722:02
sean-k-mooneyi remembere there was somting about an unreleased python version at one point on ubuntu but dont know if that was 3.6 or 3.722:03
efriedright, cuz we've been testing with 3.7 for a while now, so if it was >=3.6.7 we should have hit it.22:03
sean-k-mooneyefried: well no the original bugfix was merged  2018-07-06 10:2322:04
mriedemefried: i think the 3.6 jobs switched to bionic in the last 24 hours22:05
mriedemaccording to logstash22:05
sean-k-mooneyso it proably took a while to get in the ubuntu package repos and then for use to update the image with it22:05
sean-k-mooneyor what mriedem said22:05
mriedemanyway, it doesn't really matter22:06
mriedemsh*t is f'ed, skip until it's not22:06
mriedemmaking sure we don't have double words in comments is probably not super high on the cloud infrastructure priority list22:07
cdentmy testing box has been bionic for months, but i did an update update and upgrade today (after being gone for the week of berlin) and that's when it started showing up. I reguarly to upgrades on that machine, so I think it is a minor upgrade in bionic22:07
cdentmriedem: no way man, it's the tippy top top22:07
mriedemhow are we going to do flake8 checks AT THE EDGE?!22:08
sean-k-mooneyyou know i have been starting to look into jamspell after the sumit demo22:08
sean-k-mooneyim hopign to see if i can figure out how to make it work with a git commit hook to try and fix may patches22:09
sean-k-mooneymaybe22:09
efriedheh22:09
sean-k-mooneyok i should call it a night o/22:10
cdenti probably need something like that too22:11
cdentI gave up on grammarly, seemed to be making browser too slow, and only works there22:11
sean-k-mooneyi still use it somethime but i have to fix the formating after so it breaks my workflow to copy and past the suff into a broswer22:12
sean-k-mooneyanyway email sent really going this time22:13
cdentI think I should go too. night all22:19
*** cdent has quit IRC22:19
*** aloga_ has joined #openstack-nova22:36
*** dklyle has quit IRC22:37
*** slaweq__ has joined #openstack-nova22:37
*** dklyle has joined #openstack-nova22:38
*** wznoinsk has joined #openstack-nova22:38
*** tbachman has quit IRC22:39
*** jiaopengju has quit IRC22:39
*** mgoddard has quit IRC22:39
*** aloga has quit IRC22:39
*** slaweq_ has quit IRC22:39
*** marvin_mhg has quit IRC22:39
*** bandini has quit IRC22:39
*** imacdonn has quit IRC22:39
*** spatel_ has quit IRC22:39
*** cfriesen has quit IRC22:39
*** mriedem has quit IRC22:39
*** sambetts_ has quit IRC22:39
*** jmlowe has quit IRC22:39
*** manjeets has quit IRC22:39
*** jmlowe has joined #openstack-nova22:41
*** tbachman has joined #openstack-nova22:41
*** jiaopengju has joined #openstack-nova22:41
*** imacdonn has joined #openstack-nova22:41
*** sambetts_ has joined #openstack-nova22:43
*** mgoddard has joined #openstack-nova22:45
*** cfriesen has joined #openstack-nova23:01
efriedjroll: Can I get your nod on https://review.openstack.org/#/c/616721/ please before I merge it?23:05
openstackgerritEric Fried proposed openstack/nova master: Nix refs to ResourceProvider obj from libvirt UT  https://review.openstack.org/61878623:08
*** spatel has joined #openstack-nova23:16
*** mlavalle has quit IRC23:16
*** spatel has quit IRC23:20
*** spatel has joined #openstack-nova23:22
openstackgerritTakashi NATSUME proposed openstack/nova master: Fix server query examples  https://review.openstack.org/61683423:23
*** spatel has quit IRC23:26
*** aloga_ has quit IRC23:33
*** gibi has quit IRC23:34
*** aloga has joined #openstack-nova23:35
*** gibi has joined #openstack-nova23:35
*** manjeets has joined #openstack-nova23:36
*** lbragstad has quit IRC23:40
*** takashin has joined #openstack-nova23:45

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!