Thursday, 2019-08-01

« Wednesday, 2019-07-31 Index Friday, 2019-08-02 »
*** nafiux has quit IRC00:02
*** betherly has joined #openstack-nova00:06
*** nafiux has joined #openstack-nova00:06
*** betherly has quit IRC00:11
*** nafiux has quit IRC00:14
*** brinzhang has joined #openstack-nova00:17
*** brinzhang_ has quit IRC00:20
*** ivve has quit IRC00:24
*** nafiux has joined #openstack-nova00:35
*** threestrands has quit IRC00:49
*** efried has quit IRC00:51
*** bhagyashris has joined #openstack-nova00:51
*** efried has joined #openstack-nova00:59
*** ricolin has joined #openstack-nova01:04
*** betherly has joined #openstack-nova01:08
*** tetsuro has joined #openstack-nova01:12
*** betherly has quit IRC01:13
*** nafiux has quit IRC01:14
*** nafiux has joined #openstack-nova01:21
*** igordc has quit IRC01:31
*** boxiang has joined #openstack-nova01:38
*** nafiux has quit IRC01:38
*** boxiang has quit IRC01:39
*** boxiang has joined #openstack-nova01:39
*** betherly has joined #openstack-nova01:40
*** boxiang has quit IRC01:42
*** betherly has quit IRC01:44
*** gyee has quit IRC02:00
*** betherly has joined #openstack-nova02:01
*** betherly has quit IRC02:05
*** slaweq has joined #openstack-nova02:11
*** slaweq has quit IRC02:15
*** tetsuro has quit IRC02:24
*** betherly has joined #openstack-nova02:25
openstackgerritpengyuesheng proposed openstack/os-vif master: Bump the openstackdocstheme extension to 1.20  https://review.opendev.org/67285702:28
*** betherly has quit IRC02:29
*** BjoernT has joined #openstack-nova02:33
*** tetsuro has joined #openstack-nova02:48
*** threestrands has joined #openstack-nova03:02
*** tetsuro has quit IRC03:21
openstackgerritLi Liu proposed openstack/nova master: Define new exceptions related to device profiles and ARQs.  https://review.opendev.org/67373303:30
openstackgerritLi Liu proposed openstack/nova master: Refactor some methods for reuse by Cyborg code.  https://review.opendev.org/67373403:30
openstackgerritLi Liu proposed openstack/nova master: WIP: Add Cyborg device profile groups to request spec.  https://review.opendev.org/63124303:30
openstackgerritLi Liu proposed openstack/nova master: WIP: Create and bind Cyborg ARQs.  https://review.opendev.org/63124403:30
openstackgerritLi Liu proposed openstack/nova master: fixed merge conflict  https://review.opendev.org/67393803:30
openstackgerritLi Liu proposed openstack/nova master: added cyborg external event  https://review.opendev.org/67393903:31
openstackgerritLi Liu proposed openstack/nova master: WIP: Create and bind Cyborg ARQs.  https://review.opendev.org/63124403:37
*** whoami-rajat has joined #openstack-nova03:41
*** hongbin has joined #openstack-nova03:45
*** hongbin has quit IRC03:46
*** udesale has joined #openstack-nova03:46
*** psachin has joined #openstack-nova03:55
*** slaweq has joined #openstack-nova04:11
*** slaweq has quit IRC04:17
*** mkrai has joined #openstack-nova04:23
*** Luzi has joined #openstack-nova04:25
*** tetsuro has joined #openstack-nova04:28
*** BjoernT has quit IRC04:43
*** bhagyashris has quit IRC04:52
*** tetsuro has quit IRC05:02
*** ratailor has joined #openstack-nova05:09
openstackgerritSundar Nadathur proposed openstack/nova master: ksa auth conf and client for Cyborg access  https://review.opendev.org/63124205:11
openstackgerritSundar Nadathur proposed openstack/nova master: Refactor some methods for reuse by Cyborg-related code.  https://review.opendev.org/67373405:11
openstackgerritSundar Nadathur proposed openstack/nova master: WIP: Add Cyborg device profile groups to request spec.  https://review.opendev.org/63124305:11
openstackgerritSundar Nadathur proposed openstack/nova master: WIP: Create and bind Cyborg ARQs.  https://review.opendev.org/63124405:11
openstackgerritSundar Nadathur proposed openstack/nova master: WIP: Get resolved Cyborg ARQs and add PCI BDFs to VM's domain XML.  https://review.opendev.org/63124505:11
openstackgerritSundar Nadathur proposed openstack/nova master: Delete ARQs for an instance when the instance is deleted.  https://review.opendev.org/67373505:11
*** nafiux has joined #openstack-nova05:11
*** ociuhandu has joined #openstack-nova05:22
*** dpawlik has joined #openstack-nova05:26
*** dansmith has quit IRC05:26
*** ociuhandu has quit IRC05:27
*** dansmith has joined #openstack-nova05:28
*** tetsuro has joined #openstack-nova05:43
*** jaosorior has quit IRC05:46
*** tetsuro has quit IRC05:48
*** belmoreira has joined #openstack-nova05:50
*** belmoreira has quit IRC05:50
*** belmoreira has joined #openstack-nova05:52
*** ccamacho has quit IRC05:54
*** maciejjozefczyk has joined #openstack-nova05:59
*** bhagyashris_ has joined #openstack-nova06:03
*** slaweq has joined #openstack-nova06:04
*** slaweq has quit IRC06:09
*** slaweq has joined #openstack-nova06:11
*** takamatsu has joined #openstack-nova06:13
*** slaweq has quit IRC06:16
*** xek has joined #openstack-nova06:17
*** janki has joined #openstack-nova06:22
*** ricolin_ has joined #openstack-nova06:26
*** xek has quit IRC06:27
*** ricolin has quit IRC06:29
*** aojea has joined #openstack-nova06:31
*** belmoreira has quit IRC07:01
*** kashyap has joined #openstack-nova07:02
*** belmoreira has joined #openstack-nova07:03
kashyapstephenfin: Morning, when you can, mind having a look at this small patch (that fixes two bugs)?  Pinging you because, we've discussed this in the past: https://review.opendev.org/#/c/348394/07:03
*** slaweq has joined #openstack-nova07:04
kashyapstephenfin: For context, you've quoted me in the change, at that time I thought it wasn't worth it07:04
kashyapstephenfin: But seeing that CentOS _and_ SLES are broken, this temporary solution seemed acceptable07:05
kashyapaspiers: https://review.opendev.org/#/c/348394/07:05
*** maciejjozefczyk_ has joined #openstack-nova07:05
*** maciejjozefczyk has quit IRC07:08
*** pcaruana has quit IRC07:12
*** rpittau|afk is now known as rpittau07:13
*** tetsuro has joined #openstack-nova07:13
*** tetsuro has quit IRC07:17
*** ccamacho has joined #openstack-nova07:18
*** tesseract has joined #openstack-nova07:20
*** nafiux has quit IRC07:23
*** belmoreira has quit IRC07:28
*** tssurya has joined #openstack-nova07:32
*** belmoreira has joined #openstack-nova07:35
*** cdent has joined #openstack-nova07:36
*** ralonsoh has joined #openstack-nova07:39
*** ociuhandu has joined #openstack-nova07:39
*** boxiang has joined #openstack-nova07:39
*** boxiang has quit IRC07:39
*** boxiang has joined #openstack-nova07:40
*** ralonsoh has quit IRC07:40
*** ralonsoh has joined #openstack-nova07:40
*** pcaruana has joined #openstack-nova07:42
*** ivve has joined #openstack-nova07:51
*** ociuhandu has quit IRC07:56
*** boxiang has quit IRC07:59
*** boxiang has joined #openstack-nova07:59
*** belmoreira has quit IRC08:01
*** lpetrut has joined #openstack-nova08:03
*** lpetrut has quit IRC08:04
*** lpetrut has joined #openstack-nova08:04
*** belmoreira has joined #openstack-nova08:07
*** belmoreira has quit IRC08:09
*** belmoreira has joined #openstack-nova08:11
kashyapalex_xu: Hi, mind also looking at this? -- https://review.opendev.org/#/c/348394/08:12
*** boxiang has quit IRC08:17
*** maciejjozefczyk_ is now known as maciejjozefczyk08:17
*** derekh has joined #openstack-nova08:25
*** shilpasd has joined #openstack-nova08:25
*** tkajinam has quit IRC08:27
*** nfakhir has joined #openstack-nova08:27
*** lpetrut has quit IRC08:29
*** ivve has quit IRC08:30
stephenfinkashyap: Sure, I'll take a look now08:32
kashyapstephenfin: Thank you.  I also want to do a functional test (as this is not functionally tested in the Gate)08:32
kashyapI'll report back my findings on the patch, too08:32
kashyapThank you.  To save you time, see the last three comments (two from me, one from Adam)08:33
bhagyashris_stephenfin: Hi,08:35
stephenfinbhagyashris_: o.08:35
stephenfin*o/08:35
bhagyashris_stephenfin: I just encountered in one issue and I have posted comment on your patch https://review.opendev.org/#/c/671793/5 .08:35
bhagyashris_stephenfin: Just to info you I am currently working on the upgrade patch  https://review.opendev.org/#/c/672224/1 (Fixing review comments given by you) and also testing those change on top of your changes. Once it’s fixed  I will push the patch soon.08:36
*** purplerbot has quit IRC08:36
*** purplerbot has joined #openstack-nova08:36
stephenfinbhagyashris_: Ack. I hope to have a new revision with that fixed and the WIP flag removed today08:37
*** ivve has joined #openstack-nova08:40
bhagyashris_stephenfin: ok... Me too will push the upgrade patch soon on top of your changes...08:40
stephenfinSweet08:41
bhagyashris_stephenfin: and also one thing that the new syntax "resources:(PCPU|VCPU) = <no of cpus>" is not accepted in the current patch series so that thing also we need to take care...08:43
stephenfinbhagyashris_: Yes. I think I'm going to drop my conversion patch and use yours, if that's okay?08:44
stephenfin(the entire commit, so will keep authorship)08:44
bhagyashris_stephenfin: sure No problem!08:45
*** priteau has joined #openstack-nova08:54
*** belmoreira has quit IRC08:54
openstackgerritBrin Zhang proposed openstack/nova master: nit: fix the test case of migration obj_make_compatible  https://review.opendev.org/67396108:57
*** psachin has quit IRC09:00
*** belmoreira has joined #openstack-nova09:03
*** janki has quit IRC09:08
*** janki has joined #openstack-nova09:09
kashyapaspiers: Let me know when you're about to double-check me on a domainCapabilities parsing thing09:11
kashyaps/"about to"/"about -- to"/09:11
*** jchhatbar has joined #openstack-nova09:17
*** janki has quit IRC09:17
*** belmoreira has quit IRC09:22
*** jchhatbar has quit IRC09:25
*** belmoreira has joined #openstack-nova09:26
kashyapUnrelated...that's some commit message right there: https://github.com/dtschump/CImg/commit/47e57118bc1eb58c009:33
kashyap(And many others in that repo :D)09:33
*** shilpasd has quit IRC09:34
*** ivve has quit IRC09:37
sean-k-mooneybhagyashris_: stephenfin i know efried wants use to support the resouces: syntax but im really not sure if that is a good idea09:38
sean-k-mooneybhagyashris_: stephenfin if we do allow "resources:(PCPU|VCPU) = <no of cpus>" we shoudl only allow it if cpu policy=mixed09:39
sean-k-mooneye.g. it houls be an error to have PCPUs if you policy is shared or VCPUs if the policy is dedicated09:39
stephenfinsean-k-mooney: I was thinking defining the policy along with explicit resources doesn't make sense and should be an error in its own right09:42
*** priteau has quit IRC09:43
sean-k-mooneyif you dont define the policy PCPUs will never be used09:43
*** mkrai has quit IRC09:43
sean-k-mooneywe dont want to infer masks from "resources:(PCPU|VCPU) = <no of cpus>"09:43
*** priteau has joined #openstack-nova09:44
sean-k-mooneyso i find it hard to understand why we would support this at all09:44
sean-k-mooneygive "resources:(PCPU|VCPU) = <no of cpus>" will not affect xml generation in any way09:44
*** priteau has quit IRC09:50
*** priteau has joined #openstack-nova09:52
*** belmoreira has quit IRC09:53
openstackgerritBrin Zhang proposed openstack/nova master: WIP: Add user_id and project_id colume to Migration  https://review.opendev.org/67399009:54
*** ociuhandu has joined #openstack-nova09:57
*** shilpasd has joined #openstack-nova09:57
*** sapd1_x has joined #openstack-nova09:59
*** belmoreira has joined #openstack-nova10:00
*** ociuhandu has quit IRC10:01
*** prometheanfire has quit IRC10:04
*** rpittau is now known as rpittau|bbl10:04
aspierssean-k-mooney: do you know which part of nova-conductor this comes from? http://paste.openstack.org/show/755199/10:12
aspiersI'm seeing it spammed to mysqld at a rate of 2M/s10:12
aspierseven though mysqladmin processlist shows everything as "Sleep"10:12
aspiersI don't know wtf is going on10:12
aspiersit's from devstack@n-super-cond.service10:13
*** bhagyashris_ has quit IRC10:16
sean-k-mooney2M/s as in 2,000,000/s10:16
sean-k-mooneyor 2 messages a second10:17
sean-k-mooneybecause if its the former no wonder you service died10:17
aspiers2Mb/s10:17
aspiersactually no, 2MB/s10:18
aspierseven worse10:18
sean-k-mooneyi dont know if that is higher or lower10:18
aspiersB = bytes10:18
aspiersb = bits10:18
sean-k-mooneyno not that10:18
sean-k-mooney2MB/s of a tiny record10:18
sean-k-mooneyis that more or less the 2 million queries10:18
aspiers?10:18
sean-k-mooneyits a lot in any case10:19
aspiershuh10:19
aspiers2 megabytes / s10:19
aspiersI'm not talking about numbers of queries10:19
sean-k-mooneyi know im trying to convert to quires10:19
aspierswell queries range from 10s to 100s of bytes10:20
aspierslet's say each query is 200 bytes10:20
aspiersthen that would be 100k queries per sec10:20
aspiersno wait10:20
aspiers10kq/s10:20
aspiersa lot anyway10:21
aspiershtf do I debug this?10:21
sean-k-mooneyyep it looks like its trying to determin if a compute service exists10:21
sean-k-mooneyor rather look it up by its id10:22
aspiersbut why in an infinite loop, and why the commit and rollback?10:24
sean-k-mooneyis it alway the same service id10:25
aspiersoh good point10:25
sean-k-mooneyit could be trying to register the conductor service and failing10:25
aspiersit's either id 1 or 210:27
sean-k-mooneythat log should WHERE services.deleted = 0 AND services.id = 510:27
sean-k-mooneybut ok10:27
aspiershrm10:27
sean-k-mooneywhat are service id 1 and 210:27
aspierswhich table do I check in which db?10:28
*** ricolin__ has joined #openstack-nova10:28
aspierskeystone / service?10:29
aspiersno10:29
sean-k-mooneyno nova_cell110:29
aspierswhy not cell0?10:30
sean-k-mooneynova_cell1.services10:30
aspiers1 is nova-conductor10:31
aspiers3 is nova-compute10:31
aspiersthat's it10:31
*** ricolin_ has quit IRC10:31
sean-k-mooneyim not sure where 2 and 5 came form10:31
aspiersnor me10:31
aspiersso n-super-cond seems to use cell1 and n-cond-cell0 uses cell010:32
aspierswhy is that?10:32
aspierscells have 2 tiers of conductors?10:32
sean-k-mooneyyes10:32
aspiersbut in that case what if another cell got added10:33
aspiersthat would be cell2?10:33
aspiersseems weird10:33
sean-k-mooneythe db name is up to you10:33
aspierssurely the top tier super-cond would come first as cell010:33
sean-k-mooneybut cells are ways for sharding your db and messagebus10:33
aspiersthis is devstack so I didn't decide10:33
aspiersif devstack was configured with 3 cells, would n-super-cond be cell3?10:34
aspiersand n-cond-cell[0-2]10:34
aspiers?10:34
sean-k-mooneyi dont know10:34
aspiersok anyway10:34
aspierssomething is super weird10:34
aspiersit's not just conductor10:34
aspierswtf10:35
aspiersOK there is no nova processes running now but mysqladmin processlist still shows a nova_cell1 process in sleep10:36
aspiersand mysqld still spikes to 2MB/s disk writes every few seconds10:36
sean-k-mooneythis kind of sounds like a perodic task10:36
sean-k-mooneyor you have duplicate host names somewer and some other devstack is trying to use yoru db and cause werid things to happen10:37
sean-k-mooneye.g. its either supper weird or it likely just a backroud taks10:37
aspiers 15502 be/4 mysql       0.00 B/s    2.08 M/s  0.00 %  0.12 % mysqld --defaults-file=/etc/my.cnf --user=mysql10:38
sean-k-mooneyhave you tried restarting mysql10:38
aspiersbackground tasks from where?10:38
sean-k-mooneya periodic task from one of the nova services10:38
aspiersthere are no nova services running!10:39
aspiersI stopped them all10:39
aspiers<aspiers> OK there is no nova processes running now but mysqladmin processlist still shows a nova_cell1 process in sleep10:39
aspiersthis is f'ing weird10:39
aspiers| Id   | User        | Host            | db         | Command | Time | State                    | Info             | Progress |10:40
sean-k-mooneyhave you check ps to see if there is a zombie python process10:40
aspiers| 5774 | root        | localhost       | nova_cell1 | Sleep   | 537  |                          |                  | 0.000    |10:40
aspiersbut 5774 is an internal mysqld pid I think10:40
aspiersohhhh no wait10:40
aspiersUID         PID   PPID  C STIME TTY          TIME CMD10:40
aspiersstack      5774   5329  0 03:20 ?        00:00:05 placementuWSGI worker 5010:40
aspierswow10:40
aspiersI forgot about wsgi10:41
aspierswas only looking at systemd services10:41
sean-k-mooneyplacement has a systemd service10:41
aspiersbut apparently also wsgi?10:41
sean-k-mooneyits the same thing10:41
aspiersProxyPass "/placement" "unix:/var/run/uwsgi/placement-api.socket|uwsgi://uwsgi-uds-placement-api/" retry=010:42
aspiersI think I'm gonna unstack.sh this damn thing10:42
aspiersand rebuild10:42
sean-k-mooneydevstack@placement-api.service10:43
*** bbowen has joined #openstack-nova10:44
aspiersyeah I've stopped that and 5774 is still listed in mysqld10:44
aspiersbut not a kernel process any more10:44
*** bbowen has quit IRC10:45
sean-k-mooney./clean.sh && sudo reboot10:45
aspiershah10:45
sean-k-mooneyclean will remove mysql and your problem with it10:45
*** bbowen has joined #openstack-nova10:46
aspiershow is that different from unstack.sh?10:46
aspiers# ``clean.sh`` does its best to eradicate traces of a Grenade10:46
aspiersI'm not using Grenade10:46
sean-k-mooneyclean does more then that10:46
sean-k-mooneyit removes several packages and files created by devstack that unstack does not10:47
aspiersOK I guess I can run both10:47
sean-k-mooneyyou normally only use clean if changing branches e.g. master to stable10:47
sean-k-mooneyclean runs unstack10:47
aspiersoh right10:47
aspierssomeone should fix the misleading comment at the top of clean then10:47
sean-k-mooneywell normally you dont need to use it but when weird stuff happens somethimes its for the best10:48
*** prometheanfire has joined #openstack-nova10:50
sean-k-mooneyevery few days i close all my fire fox windows becaue i have just too many tabs open to go back too that i dont rememebr what tabs i have already dealt with10:51
*** tbachman has quit IRC10:51
aspiersyou should use a tab limiter10:51
sean-k-mooneywell when i had 10 windows im not sure that will help10:52
sean-k-mooneyalso why10:52
aspiersmine stops me from that crazy behaviour10:52
aspiersreminds me not to excessively multi-task10:52
aspiersif I want to remember a URL, I add it to a file of structured notes10:52
aspiersor to my TODO list10:52
aspiersthen I stay focused on fewer things10:52
sean-k-mooneynomally 90% of my tabs are gerrit reviews10:53
sean-k-mooneyand like 20 copys of nova's github10:53
aspierswhy are you wanting to keep them open though?10:53
aspierssounds like your workflow needs a bit of reevaluating10:53
aspiersI find it hard not to multitask which is why the extension helps10:54
sean-k-mooneyoften i plan to check back on the review to see if people replied or i leave it open when i join a meeting10:54
aspiersmtultitasking is so damaging to sanity and productivity10:54
sean-k-mooneyi usually have 1 or two windows that i use mostly on two different monitors and the other are form my poping out bluejeans for a video meeting and all the tabs i opened during that meeting10:56
sean-k-mooneyso i just need to get in the habit of killing the meeting windows after the meeting but i leave it open if i have ot go update a bug or something10:56
*** dpawlik has quit IRC10:58
openstackgerritGhanshyam Mann proposed openstack/nova master: Add test coverage of existing os-services policies  https://review.opendev.org/66918110:58
*** rcernin has quit IRC10:58
*** dpawlik has joined #openstack-nova11:06
*** ivve has joined #openstack-nova11:07
janguttersean-k-mooney: At least the debate hasn't started veering into the tabs-vs-spaces territory.11:12
aspiersjangutter: just you wait11:13
aspierssean-k-mooney: clean.sh didn't remove my OVS bridges11:13
*** tbachman has joined #openstack-nova11:14
sean-k-mooneyya it doesnt11:14
jangutterI have a friend that uses all 16 virtual desktops. One for each separate client engagement and/or project. They have a special ultra-reliable UPS for his workstation in-line with the building UPS.11:14
sean-k-mooneyit also does not unistall and nuke apache config11:15
sean-k-mooneyit would be nice if it it gets close enough11:15
aspiersSSLError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Max retries exceeded with url: /packages/11/fa/0160cd525c62d7abd076a070ff02b2b94de589f1a9789774f17d7c54058e/pyparsing-2.4.2-py2.py3-none-any.whl (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)'),))11:16
sean-k-mooneyjangutter: but still just one power supply in the desktop[11:16
aspiersdevstack hates me today11:16
*** belmoreira has quit IRC11:17
sean-k-mooneyjangutter: did you see https://review.opendev.org/#/c/672834/ if not can you review11:18
sean-k-mooneystephenfin: you too ^11:18
*** belmoreira has joined #openstack-nova11:19
*** ociuhandu has joined #openstack-nova11:20
*** ociuhandu has quit IRC11:21
*** panda is now known as panda|eat11:25
openstackgerritHuachang Wang proposed openstack/nova-specs master: Use PCPU and VCPU in one instance  https://review.opendev.org/66865611:25
*** dpawlik has quit IRC11:27
aspiers      Successfully uninstalled pip-19.2.111:27
aspiersSuccessfully installed pip-9.0.311:27
aspiersthanks devstack, that's really helpful!11:27
*** Conqueror has quit IRC11:27
aspiersX-(11:27
kashyapWish me luck, I'm just about to kick off a fresh DevStack on F30 with a custom patch11:28
sean-k-mooneywell devstack does not support ver 10+11:28
sean-k-mooneypip broke compatiablity with something and i dont think we have fixed it yet11:29
sean-k-mooneyor maybe we have an we just didnt bother to update it11:29
sean-k-mooneywe shoudl at some point11:29
aspierssean-k-mooney: https://github.com/pypa/pip/issues/445911:29
aspiersopen for over 2 years11:30
aspiersand people say Python doesn't suck11:31
sean-k-mooneywell you can just set you corperate proxt as trusted in you /etc/pip.conf11:31
aspierswhat does this have to do with proxies?11:32
aspiersand why did devstack work fine before on the same machine?11:32
aspiersI *never* had to tweak /etc/pip.conf for devstack before11:32
sean-k-mooney"a note about my environment - i'm runing behind cntlm and the corp proxy."11:32
aspiersI didn't write that11:32
aspiersnot sure where you saw that11:33
sean-k-mooneyits the 5 or 6 line in the bug11:33
sean-k-mooneysecond line of the description11:33
aspiersyeah but devstack worked *yesterday* on this same machine11:33
aspiersmaybe clean.sh has removed some certificates?11:33
sean-k-mooneyi doubt it11:33
aspierswell how else do you explain the cert verificiation failing?11:33
sean-k-mooneytry stacking again11:33
aspiersthat's what I tried11:34
aspiersthat's where I'm seeing the error11:34
sean-k-mooneywell i was wondering if the cert expired honestly on the mirror you hit11:34
aspiersexpired in the last 12 hours?11:34
aspierswhen https://github.com/pypa/pip/issues/4459 has been open since May 1 2017?11:34
aspiersI doubt it11:35
aspiersthis is some horrible pip breakage11:35
openstackgerritBrin Zhang proposed openstack/nova master: WIP: Add user_id and project_id colume to Migration  https://review.opendev.org/67399011:35
kashyapaspiers: Hi.  How's DevStack treating you today?11:36
* kashyap hopes he's not rubbing salt on wound11:36
aspierskashyap: not funny11:36
kashyapWhoops.  Sorry11:36
aspiersI'm in the middle of this ridiculous deadline and everything is conspiring against me11:36
* kashyap was about to ask a favor; not a good time11:37
aspiersok, update-ca-certificates fixed it11:37
*** ivve has quit IRC11:37
kashyapaspiers: O, I didn't read the full scrollback, and made a poor joke; disregard me...11:37
sean-k-mooneyya i was debating if that would be related11:37
aspierskashyap: it's fine11:37
aspierskashyap: devstack is now running finally so I have time to kill11:37
sean-k-mooneybut i dont know why it would have been unless a cert expired11:37
aspiersI'm guessing something to do with devstack clashing with SUSE's weird cert handling11:38
kashyapaspiers: I see.  Maybe you want to tackle other bits of your deadline.  I was going to ask, if you want to triple-confirm if I'm not missing anything in  the 'enum' parsing bits here - https://review.opendev.org/#/c/673790/2/nova/virt/libvirt/config.py11:38
aspierskashyap: ok11:38
aspiersthe depressing thing is I've wasted loads of time due to broken hardware, and I strongly suspect the "fix" won't have fixed it11:39
aspiersso most likely this reinstall is a waste of time too11:39
kashyapsean-k-mooney: You're using 'xpath' here: https://review.opendev.org/#/c/666915/6/nova/virt/libvirt/config.py@18311:39
kashyapaspiers: Yeah, broken-hardware--11:40
sean-k-mooneykashyap: yes because i did not want to create lots of small clases for the enums11:40
kashyapsean-k-mooney: It's the first usage in config.py.  I'd say we should be consistent and use 'enum' approach?11:40
sean-k-mooneyto avoid https://review.opendev.org/#/c/673790/2/nova/virt/libvirt/config.py@21411:40
sean-k-mooneywell i stongly prefer xpath so its going to be rather low on my priority list to chage11:41
kashyapsean-k-mooney: Hmm, I'm not "opposed" to it, though.  As I'm a heavy 'xpath' user myself11:41
aspiersImportError: No module named enum11:41
sean-k-mooneyi dont object to other using the domb parsing either11:41
aspiersthis just keeps getting better11:41
sean-k-mooney*dom11:41
kashyapsean-k-mooney: BTW, it's not about our personal preferences; just using something consistently well.  You don't need to create lots of classes11:42
aspierskashyap: we had this conversation yesterday or the day before11:42
kashyapsean-k-mooney: You can parse the enums in the same class, no?11:42
sean-k-mooneyyou can i just really diskile how that module it written11:43
kashyapaspiers: Did we?  Then my bumblebee-like memory forgot, less than 7 seconds11:43
sean-k-mooneyto the point i considered rewringing it to use xpath and decided not to and just use it for my bit11:43
aspiers"we" as in me and sean-k-mooney11:43
aspiershttp://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-07-29.log.html#t2019-07-29T12:34:5811:43
kashyapAah, /me clicks11:43
aspiershttp://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-07-29.log.html#t2019-07-29T12:42:0211:44
aspierspersonally I'm with kashyap on this but I don't have the energy to bikeshed it because the rest of my world is falling apart11:44
* kashyap clicks further11:44
kashyapaspiers: Yeah, go on.  This is not super urgent, but I really dislike if we all codify our random preferences11:45
sean-k-mooney aspiers dont bother updating your code right now11:45
kashyapThat would suck hard11:45
sean-k-mooneyit work and its correct11:45
stephenfinsean-k-mooney: sure11:45
aspiersisn't it amazing how everything magically breaks just before a deadline?11:45
aspiersseen it so many times11:45
kashyapaspiers: Yeah, Murphy dancing on the head, seems like :-(11:45
sean-k-mooneykashyap: sure but i hat the fact that we are califying parts of the code such that we can never use better solution to solve thing11:46
*** priteau has quit IRC11:46
kashyapsean-k-mooney: Again, note - I like 'xpath' myself and happy to use it in the code :-)11:46
sean-k-mooneyit activly makes me want to stop working on nova sometimes11:46
kashyapOh, no...11:46
kashyapsean-k-mooney: Okay, leave it as-is for now.11:46
sean-k-mooneyif the general concenous is to do it the old way i can redo my code its just lots of wasted effort11:47
kashyapsean-k-mooney: I myself might even rewrite to use 'xpath' because ... I am not sure if it has the expected 'enum' property defined: https://review.opendev.org/#/c/673790/2/nova/virt/libvirt/config.py11:48
kashyapAnyway.  Don't want to get hung up on this :-)11:49
*** udesale has quit IRC11:49
*** udesale has joined #openstack-nova11:50
aspiersOK so my install of enum34 was corrupted11:52
aspiersprobably a fight between python module rpms and devstack11:52
sean-k-mooneyis enum34 the python 3.4 version?11:53
aspiersit's the backport11:53
sean-k-mooneyah ok11:53
aspiersBTW I tried python 3.411:53
aspiersit failed horribly11:53
sean-k-mooneyya we never ran nova with it so im not surprised11:53
aspiersbut thinking about it more, maybe that was because I was trying to partially rely on packages11:53
sean-k-mooneysome oslo libs did breifly but most project start with 3.511:53
aspiersactually no11:53
aspiersit was trying to install the latest versions of requests and Flask and neither work with 3.411:54
aspiersprobably lots of modules dropped support for 3.411:54
sean-k-mooney3.5 added a lot of nic features so it was kind of the first good python 3 release11:55
sean-k-mooneyor the first one people agreed was worth porting too11:55
sean-k-mooneyits when python3 got performace parity with py 2.7 and started to pull ahead in some cases11:56
aspiersImportError: No module named requests11:56
aspiersawesome!11:56
aspiersbest day ever!11:56
sean-k-mooneyi think the latest requests package got pulled form pypi by the way11:56
sean-k-mooneythe one in upper constratis nolonger exists11:56
aspiersthey fixed that last week11:56
sean-k-mooneyat least the one that was there yesterday didnt11:56
sean-k-mooneyah ok11:57
sean-k-mooneyhavent updated my requirement repo in a while11:57
aspiersyeah you should11:57
*** dpawlik has joined #openstack-nova11:57
aspiershooray it's finally starting rabbit11:57
aspiersoriginally I was trying to install SOC on this node so getting python modules from packages11:58
aspiersthen I switched to devstack which is why the rpms are fighting with pip11:58
sean-k-mooneyyou know when we update to a new pip in devstack maybe we should add --user11:59
aspierswell isn't it using virtualenv?11:59
sean-k-mooneyno11:59
sean-k-mooneynot by default11:59
sean-k-mooneyit can but its not really tested anywhere11:59
aspiersI see12:00
*** ociuhandu has joined #openstack-nova12:01
*** dpawlik has quit IRC12:01
*** jcosmao has joined #openstack-nova12:04
*** ociuhandu has quit IRC12:05
aspierssean-k-mooney: maybe I misremembered, it was pyparsing not requests https://review.opendev.org/#/c/672395/12:08
aspiersbut requests seems OK now12:08
aspiersor maybe I'm using a local branch of requirements12:09
mordredefried, stephenfin: left a comment on https://review.opendev.org/#/c/66551812:10
sean-k-mooneymordred: we are not going to use the pre-commit stuff in jobs12:11
aspiersthe best bit about devstack is how if it fails near the end you have to rerun the whole thing from the beginning12:11
mordredsean-k-mooney: ok. I'm less worried about it then12:11
aspierssort of guaranteed O(n^2) debug time12:12
sean-k-mooneymordred: it will be just optional for people to use locally and tox will still be used for jobs12:12
mordredsean-k-mooney: we got a patch to zuul-jobs suggesting using pre-commit in our tox.ini as the command tox -elinters runs - which didn't seem like a TERRIBLE idea other than the github cloning ... but having it there just as an option for people who want such a thing is even better12:13
aspiersI also love how pip (at least devstack version) doesn't have any way to verify that modules are intact12:13
sean-k-mooneymordred: ya i noted that we could run it via a tox env12:13
sean-k-mooneybut i at least was not planning to propose it as a job12:13
sean-k-mooneybut i guess i can see why other might12:14
mordredsean-k-mooney: yeah. it's .. you know, I *like* the concept of a declarative file with a list of linters to run. that part is honesty not a terrible design12:14
sean-k-mooneymordred: the removing tabs thing however is really annoying to make work with emacs and i get tired of the fact we enforce it manually12:14
mordredsean-k-mooney: is it really? weird12:15
sean-k-mooneyto do it consitenly across all mode ya12:15
*** ociuhandu has joined #openstack-nova12:15
mordredoh. yeah. that I could see being annoying12:15
sean-k-mooneyi have found no simple way to do it that "just works"12:15
mordredyou didn't want to write your own minor mode in elisp?12:16
sean-k-mooneyi have kind of stopped using emacs again partly because of it12:16
stephenfinmordred: I'm not sure I get you. Like sean-k-mooney says, none of this is integrated into tox or anything. It's purposefully kept separate12:16
mordredstephenfin: yup. I somehow missed that12:16
mordredstephenfin: I now understand better12:16
mordredconcerns withdrawn12:17
sean-k-mooneystephenfin: others have suggested that we could use it for jobs in other repos12:17
stephenfinAh, fair. I think TripleO might be doing something different but I just want this for our personal, non-gating uses12:17
mordredyeah - if we wanted to do that - I think we'd want to think about it systemically - because I do think it would be possible to do sanely - it would just take some work12:17
sean-k-mooneyby the way we could port the remove tabs script in repo12:17
sean-k-mooneybut that one works so effort+mainatince12:18
mordred++12:18
mordredif we were to go with a job, I'd say put the remove tabs script into the hacking repo, then add hacking as a required-project and have the pre-playbook we'd need to edit the pre-commit config know how to do the pre-commit-hooks repo and the hacking repo - then we'd have a generalized solution that was workable ... but there's several pieces of work in there and I don't want to do any of them right now12:19
mordred... so I like your current approach12:20
stephenfinI also like not having to do work12:20
sean-k-mooneyya that could be a good idea going forward12:20
mordrednot doing work so far exceeds doing work in desirability12:20
stephenfinYes. Yes it does.12:21
sean-k-mooneywhen we get support for pre-comit in nova i can see moving it to hacking as step two12:21
*** ociuhandu has quit IRC12:25
sean-k-mooneyi sugessted usign that repo by the way because 1 its mit so its a nice license an 2 the hook is in pyton12:26
*** tbachman has quit IRC12:26
*** ociuhandu has joined #openstack-nova12:27
mordredsean-k-mooney: ok - you've poked at this a bit ... how do you get pre-commit run -a to load in-repo hacking checks?12:28
mordredI just tried the config file from that patch in the openstacksdk repo so I could play with it and understand it better - but we've got an in-repo hacking check that it's barfing on not being able to find12:28
sean-k-mooneylike this https://github.com/SeanMooney/gerio/blob/58c26decf2dfcf1a7deeedf5f016257b89c24149/.pre-commit-config.yaml12:28
*** dpawlik has joined #openstack-nova12:29
aspiersOMG I got devstack to work12:30
sean-k-mooneycongrats you grauate to step two12:31
aspiersI can hardly believe it12:31
* aspiers waits for the kernel oops to reappear12:31
*** ociuhandu has quit IRC12:31
aspiersand for services to randomly segfault12:31
sean-k-mooneytime to get multi node devstack to work or the thing that working in devstack working in not devstack the choice is yours12:31
mordredwell - that's a local script - but in my case just running flake8 which loads hacking finds that I have this: https://opendev.org/openstack/openstacksdk/src/branch/master/tox.ini#L46-L47 defined12:31
*** ociuhandu has joined #openstack-nova12:31
aspierssean-k-mooney: I still see mysqld spinning at 2MB/s12:32
sean-k-mooney-   repo: local12:32
sean-k-mooney    hooks:12:32
mordredsean-k-mooney: and since nova also has local-check-factory = nova.hacking.checks.factory in tox.ini ... I'm wondering how the flake8 task is working12:32
sean-k-mooney    -   id: black12:32
sean-k-mooney        name: black12:32
sean-k-mooney        language: system12:32
sean-k-mooney        entry: sh -c "tools/run_black.sh"12:32
sean-k-mooneyfiles: ''12:32
sean-k-mooneymordred: so you could replace entry with a call to run tox12:32
sean-k-mooneyor just the flake8 command12:32
mordredsure ... but how is the config in that nova patch working since it calls flake8 directly as a hook?12:32
sean-k-mooneyoh i see what you you ment12:33
* mordred decides to try the nova repo and see for himself ... :)12:33
*** panda|eat is now known as panda12:34
sean-k-mooneymordred: well the flak8 check is not readign tox.ini12:35
sean-k-mooneyat least i dont think it is12:35
mordredbut hacking does if it exists to load its own config whetherit's running in tox or not12:35
sean-k-mooneyah ok well maybe this only works if you have ran devstack on that host12:36
*** tbachman has joined #openstack-nova12:36
sean-k-mooneywhich installs all the test requrieemtn system wide12:36
sean-k-mooneyi havent tried it on a clean system too check12:37
*** tssurya_ has joined #openstack-nova12:37
*** tssurya has quit IRC12:37
*** tssurya_ is now known as tssurya12:37
mordredsean-k-mooney: well - inexplicably - the nova one seems to be working12:37
*** artom has quit IRC12:37
mordredso I'll take that to mean there is _Something_ broken _somewhere_ -- which sadly means now I have to figure out what or else I won't be able to sleep12:38
sean-k-mooneywell the flake8 hook is just https://github.com/pre-commit/pre-commit-hooks/blob/master/.pre-commit-hooks.yaml#L134-L14012:39
sean-k-mooneyso does it work in the sdk if you just run flake812:40
mordredsean-k-mooney: it has found a niggly issue!12:40
sean-k-mooneypre-commit++12:41
mordredthe issue is that our flake8 checks have been running in a tox env which means openstacksdk has been installed in the venv ... and since the local hacking hooks are in openstack._hacking.factory, that means openstack itself gets imported - and because of something in openstack/__init__.py it triggers an import appdirs from a subfile which is a dependency12:41
mordredso clearly I need to fix it so that openstack/__init__.py does not trigger that import12:42
mordredsince that's just rude in general12:42
sean-k-mooneyi am a firm beliver that __init__.py shoudl always be empty unless you have a very good reason12:43
sean-k-mooneyi never think to check __init__.py unless something is broken in a weird way12:44
mordredyeah - I'm also a believe in that. in this case it's for factory functions - so you can do "import openstack ; conn = openstack.connect('foo')" ... but the importing there can be greatly improved so as to avoid this and go back to being mostly like being empty12:46
sean-k-mooneythe one usecase i kind of get is where you want to decuple the folder stucture form the structure of the modules in teh public interface12:47
sean-k-mooneyso that if you reange files internally your public api remains the same12:47
*** ratailor has quit IRC12:48
*** belmoreira has quit IRC12:48
sean-k-mooneywhich is ok for a libary to do but non libary code really shouldnt do that12:48
mordredyah. totally12:49
*** tbachman has quit IRC12:51
*** tbachman has joined #openstack-nova12:54
*** Luzi has quit IRC12:55
openstackgerritMerged openstack/nova stable/stein: Avoid logging traceback when detach device not found  https://review.opendev.org/67283312:56
*** rpittau|bbl is now known as rpittau12:57
*** jaosorior has joined #openstack-nova12:58
openstackgerritMerged openstack/nova stable/stein: Fix no propagation of nova context request_id  https://review.opendev.org/67069413:00
openstackgerritMerged openstack/nova stable/stein: Restore RT.old_resources if ComputeNode.save() fails  https://review.opendev.org/67203813:06
openstackgerritMerged openstack/nova stable/stein: Fix GET /servers/detail host_status performance regression  https://review.opendev.org/66995813:06
*** belmoreira has joined #openstack-nova13:06
kashyapaspiers: Did I interpret your comment correctly there? - https://review.opendev.org/#/c/348394/1013:09
*** mchlumsky has joined #openstack-nova13:13
kashyapsean-k-mooney: Or anyone, can you confirm if my notes are correct: to test DevStack with a random change:13:14
*** mdbooth has quit IRC13:14
kashyapHave these two in local.conf:13:14
kashyap  NOVA_REPO=$GIT_BASE/openstack/nova.git13:14
kashyap  NOVA_BRANCH=refs/changes/348394/10/13:14
kashyapNo, the 'refs' are wrong.  It should be:13:16
kashyap- refs/changes/348394/1013:16
kashyap+ refs/changes/94/348394/1013:16
*** mdbooth has joined #openstack-nova13:23
sean-k-mooneyyes13:23
efriedsean-k-mooney, stephenfin: I haven't read all the scrollback, but13:23
efriedif you wanted to *not* support resources:{P|V}CPU, you would have to put an explicit check in place for that.13:23
efriedBecause right now we support arbitrary placement-isms for traits and resources.13:23
efriedThe reason for that syntax would be to support the "overloaded" meaning of PCPU/VCPU for donwstream use cases like high/normal priority.13:23
efriedBut I suppose there's no reason you *need* that syntax to support that -- you could still just use the maskage.13:23
efriedSo I guess I'm fine either way.13:23
sean-k-mooneyalthough you dont need the NOVA_REPO if you going to set it to $GIT_BASE/openstack/nova.git that is the default13:24
sean-k-mooneyefried: there is no need to specify it driectly to do that13:24
sean-k-mooneyyep you can just use the mask13:25
sean-k-mooneywe dont use teh PCPU/VCPU resouce class values to generate teh xml either by the way13:25
sean-k-mooneyso it would not result in them being pinned13:25
kashyapsean-k-mooney: Also, got a latest very minimal local.conf?  Can this be cut down any further? -- http://paste.openstack.org/show/755214/13:25
sean-k-mooneyor not pinned13:25
sean-k-mooneywell ignoreing the fact you are moving where it install13:26
*** ociuhandu has quit IRC13:26
sean-k-mooneyyou dont need VIRT_DRIVER=libvirt13:26
sean-k-mooneythat is the default13:26
kashyapsean-k-mooney: Ah, right13:26
sean-k-mooneyas is your ml2 config section13:26
kashyapsean-k-mooney: Can nuke it?13:27
sean-k-mooneyand the neutron section13:27
kashyapOkido; /me goes to nuke13:27
*** ociuhandu has joined #openstack-nova13:27
*** mriedem has joined #openstack-nova13:27
*** priteau has joined #openstack-nova13:28
*** mchlumsky has quit IRC13:28
kashyapThanks13:29
*** mchlumsky has joined #openstack-nova13:30
sean-k-mooneykashyap: i think you can just do http://paste.openstack.org/show/755215/13:30
kashyapI also need 'FORCE=yes' as I'm testing on F30...13:31
*** tbachman has quit IRC13:31
sean-k-mooney#MULTI_HOST=True13:31
sean-k-mooneythat is for nova networks13:31
sean-k-mooneyso you dont need that13:31
*** ociuhandu has quit IRC13:31
kashyapsean-k-mooney: Yeah, I thought I commented it out13:31
* kashyap should update these https://kashyapc.fedorapeople.org/virt/openstack/multi-node-configs/13:31
sean-k-mooneyyou did13:33
kashyapYep, thx13:33
kashyapsean-k-mooney: Remind me again ... do I need 'force_config_drive = False' for Live Mig -- or is that fixed?  (It is, IIRC)13:34
sean-k-mooneybut there is no reason to keep it commeted we dont support nova net downstream and we shouldnt use it upstream13:34
mriedemefried: dustinc: i'm about to add https://blueprints.launchpad.net/nova/+spec/openstacksdk-in-nova to a runway slot, is it ready?13:34
* kashyap needs to check13:34
sean-k-mooneykashyap: config drive is off by default13:34
kashyapSweet13:34
sean-k-mooneyand in theory new version of libvirt can copy readonly configdrive so it should not be needed13:35
sean-k-mooneygenrally i used to change the configdriver to vfat instaead as a better workaround for the same issue13:35
kashyapOkido, it's on progress.  Let's see, if it goes through13:35
efriedmriedem: I will need to let dustinc answer that. The bits I worked on - the base and the placement cutover - have already merged.13:35
efriedmriedem: dustinc is Pacific, so might be a couple hours.13:36
mriedemok13:36
mriedemthough you're on https://review.opendev.org/#/c/642899/ which is my current hangup13:36
efriedmriedem: oh, yeah, I introduced like the very first PoC PS of that, but dustinc has owned it since then.13:37
sean-k-mooneykashyap: oh by the way you should make it less minimal13:38
sean-k-mooneyadd USE_PYTHON3=True13:38
efriedpretty sure all my touches since the PTG have been rebases (possibly a couple of trivial manual ones in there)13:38
*** tbachman has joined #openstack-nova13:38
sean-k-mooneykashyap: actully on fedora 30 you might only have python 3 so it might do that by default13:39
kashyapsean-k-mooney: Afraid, it's in progress.13:40
openstackgerritMerged openstack/nova stable/stein: Handle Invalid exceptions as expected in attach_interface  https://review.opendev.org/67238413:41
kashyapsean-k-mooney: Indeed, F30 _does_ use PY3.13:41
openstackgerritMerged openstack/nova stable/stein: Add functional regression test for bug 1837955  https://review.opendev.org/67353213:41
openstackbug 1837955 in OpenStack Compute (nova) stein "MaxRetriesExceeded sometime fails with messaging exception" [Medium,In progress] https://launchpad.net/bugs/1837955 - Assigned to Matt Riedemann (mriedem)13:41
sean-k-mooneyits fine it should do the right thing13:41
openstackgerritMerged openstack/nova stable/stein: Cleanup when hitting MaxRetriesExceeded from no host_available  https://review.opendev.org/67353313:41
*** eharney has joined #openstack-nova13:46
*** priteau has quit IRC13:53
openstackgerritGhanshyam Mann proposed openstack/nova master: Pass RequestContext to oslo_policy  https://review.opendev.org/67403813:59
*** lbragstad has joined #openstack-nova13:59
kashyapsean-k-mooney: Sigh, I missed the SUBUNIT_OUTPUT=$DEST/devstack.subunit14:00
sean-k-mooneykashyap: why do you change the dest folder by the way14:01
*** priteau has joined #openstack-nova14:01
sean-k-mooney/opt/stack usually works fine14:01
kashyapsean-k-mooney: Because I want them all in one place, and that's how my muscle memory is fixed, afraid14:01
openstackgerritGhanshyam Mann proposed openstack/nova master: Add test coverage of existing os-services policies  https://review.opendev.org/66918114:01
*** BjoernT has joined #openstack-nova14:01
kashyapOtherwise DevStack drops turds all over the place14:02
kashyap(Not that I can completely escape from it...)14:02
sean-k-mooneyno devstack puts everting in /opt/stack otherwise14:02
openstackgerritGhanshyam Mann proposed openstack/nova master: Pass RequestContext to oslo_policy  https://review.opendev.org/67403814:02
*** mdbooth has quit IRC14:02
openstackgerritGhanshyam Mann proposed openstack/nova master: Add test coverage of existing os-services policies  https://review.opendev.org/66918114:03
sean-k-mooneye.g. DEST default to /opt/stack so all the things that use DEST will be sub directories of it14:03
sean-k-mooneyanyway you do you14:03
kashyapNod :-)14:03
*** mdbooth has joined #openstack-nova14:05
*** spatel has joined #openstack-nova14:06
mordredsean-k-mooney, stephenfin: you nerd-sniped me - https://review.opendev.org/#/c/674040/ . However, word of warning, using the flake8 hook results in a different version of flake8 than what is installed by hacking14:06
*** ociuhandu has joined #openstack-nova14:07
mordred(easy workaround - just a thing to be aware of)14:07
sean-k-mooneyit will use whatever version is on your system path right14:08
*** BjoernT_ has joined #openstack-nova14:09
sean-k-mooneymordred: i think the version we use is generally the same as its teh one install by devstack14:10
*** belmoreira has quit IRC14:10
sean-k-mooneybut ya it could be much newer i guess too depening on how it was installed14:10
*** BjoernT has quit IRC14:10
mriedemslaweq: i wanted to apply https://github.com/openstack/tempest/commit/eb0a2cc5f240d52efa3a58c5a1ba8821bae3147e to the nova-next job, do you think i should wait for the devstack change? otherwise it's easy for me to just throw it in the nova-next zuul yaml config14:10
mriedemoh huzzah https://review.opendev.org/#/c/674025/14:11
mriedemi guess once that lands nova-next will just get it for free14:12
gmannmriedem: yeah tempest one is reverted back.14:14
mriedemyup i saw14:15
mriedemthe devstack patch looks good to me14:15
mriedemhttps://logs.opendev.org/25/674025/1/check/devstack-multinode/48f53e2/controller/logs/etc/nova/nova_conf.txt.gz14:15
mriedem[cache] memcache_servers = localhost:11211 backend = dogpile.cache.memcached enabled = True14:15
*** ociuhandu has quit IRC14:16
mordredsean-k-mooney: if you just do "pip install --user pre-commit ; pre-commit run -a" pre-commit will helpfully install flake8 for you into a virtualenv - resulting in a wildly different version: http://paste.openstack.org/show/755219/14:17
sean-k-mooneymordred: even if flak8 is already installed14:17
sean-k-mooneybut good to know14:17
slaweqmriedem: I hope this devstack change can be merged soon so You should have it "for free" in Your jobs14:17
mriedemyup same14:18
mriedemit only takes clarkb and gmann to merge it :)14:18
mordredsean-k-mooney: yeah - one of the things we've tried to do is make sure the gate runs the same commands that devs do so that in addition to making sure the software is good, we're making sure that devs running commands have a decent expectation of those commands working and only breaking if their code broke something14:18
sean-k-mooneymordred: i personally havent tried using it for flake8 yet14:18
sean-k-mooneymordred: i was usinging it for basic white space managment14:19
mordredyeah - I ran pre-commit run -a in the nova repo with stephenfin's patch and it blew up massively14:19
mordredI actually think basic whitespace management is a great use of it14:19
*** ociuhandu has joined #openstack-nova14:19
mordredlike - flake8 isn't quick to run :)14:19
sean-k-mooneyapparently it is faster with pre-commit as its ment to only run on the files you are modifying14:20
sean-k-mooneylike our fast8 tox env does14:20
sean-k-mooneybut again havent tested stephenfin patch14:20
mordredyeah - if I run without -a it's very quick14:20
sean-k-mooneyyep that was the goal. quick test/hooks that fix the common style stuff before it hits the gate14:21
sean-k-mooneyits always frustrating to come back after a tempest run and notice a trailing whitespace14:22
mordredright?14:22
*** jhesketh has quit IRC14:23
*** ociuhandu has quit IRC14:23
*** artom has joined #openstack-nova14:25
mriedemefried: on this func test from gibi https://review.opendev.org/#/c/667913/ it's not really a regression so i'd like to move it elsewhere. i suggested nova/tests/functional/compute/test_init_host.py which i have added here https://review.opendev.org/#/c/670393/ - i'm just wondering if it's ok to rebase gibi's change on top of mine since the only dependency would be the module name14:33
mriedemand mine hasn't received any core review14:33
mriedemthough artom loves it14:33
artomI do what I can14:34
efriedmriedem: what are you asking?14:42
efriedYou want to rebase gibi's change and also move the test into another module?14:42
efriedand then fast approve it?14:43
*** mlavalle has joined #openstack-nova14:43
*** dpawlik has quit IRC14:43
mriedemidk about fast approve14:44
mriedembut yeah i want to move his test out of regressions since it's not a regression, it's latent behavior14:44
efriedokay, that would be the only part I'd be hesitant about, so go for it.14:44
mriedemack14:44
*** jaosorior has quit IRC14:46
openstackgerritMonty Taylor proposed openstack/nova master: Keep pre-commit inline with hacking and fix whitespace  https://review.opendev.org/67405714:51
mordredsean-k-mooney, stephenfin: ^^ followup that fixes those issues - feel free to squash or ignore or whatever.14:52
sean-k-mooneywe have way more tabs in the repo then i hoped for. e.g. more then 014:54
sean-k-mooneywe could proably exclude svg files form that but on the other hand it wont break them so i guess its fine14:54
*** aojea has quit IRC14:56
mordredsean-k-mooney: want me to re-run with an exclude added for .svg?14:56
*** priteau has quit IRC14:56
sean-k-mooneyam i dont mind but if someone generated tehm with a tool we proably dont want to process them14:56
sean-k-mooneyi doubt the wrote them by hand so its proably for the best to exclude them14:57
stephenfinmordred: I was just going to fix stuff as we went, tbh :) (i.e. not using the '-a' flag)14:57
sean-k-mooneystephenfin: well ignoring the svg files the rest is minor14:58
stephenfinyeah, no merge conflicts either. Easy straight up merge, IMO14:58
sean-k-mooneyyou spoke too soon https://review.opendev.org/#/c/660147/814:59
stephenfinsean-k-mooney: Yeah, ignore that. That patch needs massive rework as-is15:00
openstackgerritMonty Taylor proposed openstack/nova master: Keep pre-commit inline with hacking and fix whitespace  https://review.opendev.org/67405715:02
mordredthat's a much smaller version15:02
*** tbachman has quit IRC15:03
sean-k-mooneyyep it looks sane15:04
stephenfinmordred: Nice. LGTM15:04
stephenfinJust need to find someone to hold their nose and approve the parent patch now15:04
*** ratailor has joined #openstack-nova15:05
mordredheh15:05
*** maciejjozefczyk has quit IRC15:07
openstackgerritmelanie witt proposed openstack/nova stable/rocky: Avoid logging traceback when detach device not found  https://review.opendev.org/67406815:07
*** ociuhandu has joined #openstack-nova15:10
*** tbachman has joined #openstack-nova15:11
*** bbowen has quit IRC15:15
*** bbowen has joined #openstack-nova15:15
openstackgerritsean mooney proposed openstack/nova master: support pci numa affinity policies in flavor and image  https://review.opendev.org/67407215:26
*** gyee has joined #openstack-nova15:27
*** tssurya has quit IRC15:30
*** tbachman has quit IRC15:30
stephenfinCan someone explain wtf is going on here? http://paste.openstack.org/show/755239/15:41
stephenfindoes 'x = y = set([])' mean 'x' and 'y' are pointing to the same thing in memory or something?15:42
* stephenfin hasn't noticed that before, if so15:42
*** ccamacho has quit IRC15:43
stephenfinTIL https://stackoverflow.com/a/1634935615:44
*** ociuhandu has quit IRC15:47
*** cdent has quit IRC15:48
*** priteau has joined #openstack-nova15:50
*** whoami-rajat has quit IRC15:51
*** dpawlik has joined #openstack-nova15:52
*** jcosmao has left #openstack-nova15:53
*** tbachman has joined #openstack-nova15:55
prometheanfiresean-k-mooney: do you need a bug for https://review.opendev.org/673848 ?15:56
*** dpawlik has quit IRC15:59
edleafestephenfin: wow, usually people get bit by that within their first few Python programs.15:59
stephenfinI know. I've seen it before with objects but I didn't think sets would work like that too16:00
* artom has never seen nor felt the need for x = y = foo16:03
sean-k-mooneystephenfin: x an y should both be refernces to the same set yes16:03
* artom tries to write the least clever code possible16:03
openstackgerritMatt Riedemann proposed openstack/nova master: Functional reproduce for bug 1833581  https://review.opendev.org/66791316:04
openstackbug 1833581 in OpenStack Compute (nova) "instance stuck in BUILD state if nova-compute is restarted" [Low,In progress] https://launchpad.net/bugs/1833581 - Assigned to Balazs Gibizer (balazs-gibizer)16:04
openstackgerritMatt Riedemann proposed openstack/nova master: Error out interrupted builds  https://review.opendev.org/66685716:04
*** artom has quit IRC16:04
sean-k-mooneyprometheanfire: am given i forgot about that until you pinged me we proably should file one yes16:05
melwittefried, mriedem: I saw the chat on IRC about RequestContext and logging, so I added a comment on https://review.opendev.org/67392416:07
openstackgerritGhanshyam Mann proposed openstack/nova master: Introduce scope_types in os-services  https://review.opendev.org/64542716:08
mriedemmelwitt: thanks. sounds like you should -2 that?16:08
*** ksdean has joined #openstack-nova16:08
*** igordc has joined #openstack-nova16:08
*** ksdean has quit IRC16:08
prometheanfiresean-k-mooney: :D16:09
*** yaawang has quit IRC16:09
*** ksdean has joined #openstack-nova16:09
*** ksdean has quit IRC16:10
aspiersso it turns out that on a box with 256 CPUs, devstack configures nova scheduler/conductor/metadata/api with 64 workers which spams mysqld to hell and back16:11
sean-k-mooneyaspiers: ya16:11
*** ratailor has quit IRC16:11
sean-k-mooneyi have hit that proablem several times16:11
sean-k-mooneyover the years16:11
aspierssean-k-mooney: bah, but you didn't mention it earlier when I asked ;-p16:11
sean-k-mooneyit used to default to 1 worker per core16:12
*** ksdean has joined #openstack-nova16:12
sean-k-mooneyi see they tried to make it less dumb16:12
sean-k-mooneywhat that it create X works per core16:12
aspiersAPI_WORKERS=${API_WORKERS:=$(( ($(nproc)/4)<2 ? 2 : ($(nproc)/4) ))}16:12
sean-k-mooneyya the ": ($(nproc)/4 " shoudl be ":8"16:13
*** yaawang has joined #openstack-nova16:13
melwittmriedem: I didn't want to be heavy-handed, it's WIP and I was thinking the approach would be changed based on the information16:13
sean-k-mooneybut i used ot hit those issue the whole time at intel and just ened up hardcoding the worker when i did16:14
sean-k-mooneyi just am condition to assume other dont have that many cpus16:14
aspierssean-k-mooney: think there should be a hard cap?16:14
sean-k-mooneyand those that do know about this behavior16:14
sean-k-mooneyaspiers: yes 816:14
aspiersthat low?16:14
sean-k-mooneyfor devstack yes16:14
aspiersI guess16:15
aspiersbut also I wonder if the heartbeating code has a bug16:15
sean-k-mooneyif you want to set it you can use the [[post-config| path/to/file]]16:15
sean-k-mooneysyntax16:15
aspiersit does a COMMIT then ROLLBACK after every SELECT16:15
aspiersI don't even understand how mysqld would interpret that16:15
sean-k-mooneywell the heartbeating code reconnect every 60 seconds to rabbit16:16
aspiersbut that's what the general_log says16:16
sean-k-mooneybut that is a knonw issue with uwsgi and eventlets16:16
sean-k-mooneythat shoudl only effect teh api services16:16
sean-k-mooneyif you are seeing heatbeat issues on other services then that a new issue16:16
aspiersI see this with conductor16:16
openstackgerritGhanshyam Mann proposed openstack/nova master: Add new default roles and mapping in policy base class  https://review.opendev.org/64545216:17
aspiersas soon as it starts, mysqld starts writing at ~2MB/s16:17
sean-k-mooneywhat are the symtoms in the conductor logs16:17
aspiersnothing16:17
sean-k-mooneyno repeating messages16:17
sean-k-mooneywait are you seeing spikes at fixed intervals?16:18
aspiersno repeating messages16:18
sean-k-mooneyactully never mind i was thinking of somethign else you only have 1 compute node16:18
aspiersno it's more or less continuous 2MB/s16:18
aspierssometimes up to 5MB/s16:19
aspiersthat is writes not reads16:19
prometheanfiresean-k-mooney: https://bugs.launchpad.net/nova/+bug/183866616:19
openstackLaunchpad bug 1838666 in OpenStack Compute (nova) "lxml 4.4.0 causes failed tests in nova" [Undecided,New]16:19
aspiersWTF is it writing?!16:19
sean-k-mooneyaspiers: is the file growing?16:19
aspierswhich file?16:19
sean-k-mooneythe size of the mysql data directory16:19
sean-k-mooneye.g. is it updating stuff or writing new data16:19
aspiersupdating16:21
aspiersstays the same size16:21
sean-k-mooneyok and you think its the heartbeat?16:22
aspiersI enabled general_log16:22
aspierswhich is supposed to show *all* queries, right?16:22
aspiersand it just shows this select from services over and over16:23
aspiershttp://paste.openstack.org/show/755199/16:23
sean-k-mooneyand you have tracked it to the conductor16:23
aspiersnot just the conductor16:23
aspiersseveral nova services16:23
aspiersI stopped them all => mysqld goes quiet16:23
aspiersstart one => mysqld goes nuts16:24
aspiersI think it's any which has 64 workers16:24
sean-k-mooneyright the compute agent dont use workers16:24
aspiersmysqladmin processlist shows 853 entries!16:24
sean-k-mooneythe only ones that do are the metadata api, api, shduler and conductor16:24
aspiersyes those are the ones I saw the spam from16:25
sean-k-mooneyactully im not sure we use worker in the conductor16:25
aspiers Main PID: 177005 (nova-conductor)16:25
aspiers    Tasks: 64 (limit: 51200)16:25
aspiersit does16:25
aspiersnova is using 513 db connections16:25
sean-k-mooneythe condoctor can do db acess on behalf of other service like the scheduler i think16:26
aspiersneutron 15516:26
aspiersmaybe but anyway why is it doing COMMIT and ROLLBACK?16:26
aspiersselect should just be reads16:26
sean-k-mooneyno idea16:26
aspierssurely a bug16:26
openstackgerritGhanshyam Mann proposed openstack/nova master: Add new default roles and mapping in policy base class  https://review.opendev.org/64545216:26
sean-k-mooneyit might be using a a read context manager incorrectly16:27
aspiersnova.servicegroup.drivers.db.DbDriver.is_up16:28
openstackgerritGhanshyam Mann proposed openstack/nova master: WIP:Introduce scope_types in servers API  https://review.opendev.org/66296816:28
openstackgerritDongcan Ye proposed openstack/nova master: Docs: Fix launch an instance from a volume  https://review.opendev.org/67408616:29
*** lbragstad has quit IRC16:29
*** artom has joined #openstack-nova16:29
*** yaawang has quit IRC16:30
*** yaawang has joined #openstack-nova16:32
efriedthanks melwitt, that's helpful, ahem, context.16:33
efriedWhat I'm confused about, though, is: isn't a threadlocal... local... to the thread? So like a boot request will be running in a separate thread from a periodic, and thus *should* have its own context and request_id?16:35
efried(I haven't yet gotten back to looking at what happened to my test patch, which ought to be informative.)16:35
*** gregwork has joined #openstack-nova16:36
sean-k-mooneythread locals are typically implemented by kepying a read only value of the intialiastion value16:36
sean-k-mooneyand then copying when a new tread is spawned16:36
sean-k-mooneyat least that is how it work in c/c++16:37
efriedI probably need a better understanding of the lifecycle of the various threads in a nova-compute process.16:38
sean-k-mooneysame but i think im more or less done for today16:39
*** rpittau is now known as rpittau|afk16:40
sean-k-mooneyonce i triage one bug...16:40
gregworkhas anyone seen a situation where nova fails to create the record in placement for a queens/rdo overcloud deployment on baremetal?  we are getting this on a clean undercloud install for all baremetal nodes: "There was a conflict when trying to complete your request.\n\n Unable to allocate inventory: Unable to create allocation for 'CUSTOM_BAREMETAL' on resource provider 'ed51f633-972e-4d72-ac7e-6478d301c57b'. The16:41
gregworkrequested amount would exceed the capacity.  ", "title": "Conflict"}]})16:41
gregworkthe resource tags for vcpu/memory/disk on things like control are all 016:41
gregworkand looking at the nova_api db shows that all bm nodes have values > 016:41
gregworkfor those requisites16:41
efriedgregwork: It's telling you the CUSTOM_BAREMETAL resource is unavailable, so VCPU/MEMORY_MB/DISK_GB should all be n/a. Can you pull the inventory record for that provider?16:43
efriedopenstack resource provider inventory show $node_uuid CUSTOM_BAREMETAL16:44
efriedwhat you're looking for there is reserved==total==116:45
efriedif reserved==0, then look for usage:16:45
*** nafiux has joined #openstack-nova16:45
efriedopenstack resource provider usage show $node_uuid16:45
* efried lunches16:46
gregworki dont have openstack resource provider16:46
gregworki have openstack resource member16:46
*** sapd1_x has quit IRC16:49
sean-k-mooneygregwork: you are missing the osc-placment plugin16:55
*** ivve has joined #openstack-nova16:55
gregworkim installing the package16:56
gregworkseems to not get installed with a base rdo/queens install16:56
gregwork+------------------+-------+17:00
gregwork| resource_class   | usage |17:00
gregwork+------------------+-------+17:00
gregwork| VCPU             |     0 |17:00
gregwork| MEMORY_MB        |     0 |17:00
gregwork| CUSTOM_BAREMETAL |     0 |17:00
gregwork| DISK_GB          |     0 |17:00
gregwork+------------------+-------+17:00
gregworkall of them show that17:00
*** derekh has quit IRC17:01
*** nafiux has quit IRC17:02
*** igordc has quit IRC17:03
*** kdean has joined #openstack-nova17:04
*** priteau has quit IRC17:05
*** ksdean has quit IRC17:06
stephenfinI *think* I finally have all my failing unit tests for cpu-resources fixed17:10
gregworkefried: here was the other command17:11
gregwork+------------------+-------+17:12
gregwork| Field            | Value |17:12
gregwork+------------------+-------+17:12
gregwork| allocation_ratio | 1.0   |17:12
gregwork| max_unit         | 1     |17:12
gregwork| reserved         | 0     |17:12
gregwork| step_size        | 1     |17:12
gregwork| min_unit         | 1     |17:12
gregwork| total            | 1     |17:12
gregwork+------------------+-------+17:12
melwittmriedem: what do you think about backporting the --before command for archive_deleted_rows? https://review.opendev.org/556751 this has come up repeatedly downstream in clouds that run archive_deleted_rows periodically + have outages where local deletes happen. if archive runs at an inopportune time, we end up with "running deleted" guests on the hypervisor that are never cleaned up, along with placement allocations that are never17:13
melwitt cleaned up,17:13
melwittbeing able to use --before could mitigate that17:14
*** udesale has quit IRC17:14
*** whoami-rajat has joined #openstack-nova17:15
stephenfinf*** yeah17:18
stephenfinonto functional tests17:18
*** ralonsoh has quit IRC17:20
*** ashish2307 has joined #openstack-nova17:22
melwittcorrection: I see now that allocations are not cleaned up as part of the periodic. they are instead deleted at local delete time. so --before would help only in the orphaned libvirt guests case17:26
*** igordc has joined #openstack-nova17:27
*** ricolin__ is now known as ricolin17:30
*** igordc has quit IRC17:30
efriedgregwork: That all looks copacetic. Is the issue reproducible?17:40
*** nafiux has joined #openstack-nova17:41
*** slaweq has quit IRC17:42
*** igordc has joined #openstack-nova17:42
zzzeekmriedem: ping17:43
zzzeekmriedem: need a +2 on https://review.opendev.org/#/c/671040/17:43
*** ociuhandu has joined #openstack-nova17:44
*** ociuhandu has quit IRC17:48
*** betherly has joined #openstack-nova17:49
*** betherly has quit IRC17:54
*** slaweq has joined #openstack-nova17:57
openstackgerritDustin Cowles proposed openstack/nova master: Provider config file schema and loader  https://review.opendev.org/67334117:58
efriedmelwitt: are you around?18:02
melwittefried: yes18:02
mriedemmelwitt: orphans ala https://review.opendev.org/#/c/627765/18:03
mriedemmelwitt: idk about backporting https://review.opendev.org/#/c/556751/, it's tied to a blueprint18:03
efriedmelwitt: I've got some clever ideas about how to make global_request_id a) present and b) useful in (at least) the nova-compute logs, but I need some help understanding how the processes use threads. Is that something you know about? (or mriedem)18:03
melwittmriedem: yes those orphans. and yeah, that's why I wasn't sure. it's useful for operations though. on the orphan patch, I'm concerned about it because afaict, it's going to be libvirt-only and I hope that's ok. because to make it work for other drivers would take a lot more change. reason being, all the driver.destroy() require instance objects and when we're dealing with db-record-less guests, we have no real objects in that case18:05
melwittefried: I'd say I know a little. I'm in the middle of writing another comment on the patch based on your question about periodics having their own thread. to correct my comment18:06
mriedemzzzeek: done18:07
stephenfinPatchset incoming (I had to rebase :()18:07
openstackgerritStephen Finucane proposed openstack/nova master: Follow-up for I2936ce8cb293dc80e1a426094fdae6e675461470  https://review.opendev.org/67266918:07
openstackgerritStephen Finucane proposed openstack/nova master: trivial: Remove unused function parameter  https://review.opendev.org/67179618:07
openstackgerritStephen Finucane proposed openstack/nova master: objects: Rename 'nova.objects.instance_numa_topology'  https://review.opendev.org/67178918:07
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Remove unnecessary try-catch around 'getCPUMap'  https://review.opendev.org/67179018:07
openstackgerritStephen Finucane proposed openstack/nova master: objects: Remove legacy '_from_dict' functions  https://review.opendev.org/53741418:07
openstackgerritStephen Finucane proposed openstack/nova master: claims: Remove useless caching  https://review.opendev.org/67179118:07
openstackgerritStephen Finucane proposed openstack/nova master: Add '[compute] cpu_dedicated_set' option  https://review.opendev.org/67179218:07
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Start reporting PCPU inventory to placement  https://review.opendev.org/67179318:07
openstackgerritStephen Finucane proposed openstack/nova master: trivial: Rename exception argument  https://review.opendev.org/67179518:07
openstackgerritStephen Finucane proposed openstack/nova master: Remove 'hardware.get_host_numa_usage_from_instance'  https://review.opendev.org/67179718:07
openstackgerritStephen Finucane proposed openstack/nova master: Remove 'hardware.host_topology_and_format_from_host'  https://review.opendev.org/67179818:07
openstackgerritStephen Finucane proposed openstack/nova master: Remove 'hardware.instance_topology_from_instance'  https://review.opendev.org/67179918:07
openstackgerritStephen Finucane proposed openstack/nova master: Rework 'hardware.numa_usage_from_instances'  https://review.opendev.org/67256518:07
openstackgerritStephen Finucane proposed openstack/nova master: tests: Split NUMA object tests  https://review.opendev.org/67233618:07
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: '_get_(v|p)cpu_total' to '_get_(v|p)cpu_available'  https://review.opendev.org/67269318:07
openstackgerritStephen Finucane proposed openstack/nova master: hardware: Differentiate between shared and dedicated CPUs  https://review.opendev.org/67180018:07
openstackgerritStephen Finucane proposed openstack/nova master: Add support translating CPU policy extra specs, image meta  https://review.opendev.org/67180118:07
openstackgerritStephen Finucane proposed openstack/nova master: objects: Rename 'fields' import to 'obj_fields'  https://review.opendev.org/67410318:07
gregworkefried: the issue is presently 100% reproducible it is blocking all overcloud deploys18:10
zzzeekmriedem: thanks!18:14
*** shilpasd has quit IRC18:20
efriedmelwitt: Is there a way to get at the Service's context from within the virt driver? I was looking to fix the func failure exposed by gibi's poison patch18:23
efriedI could do it by adding a context arg to get_available_resource, but that's a nontrivial impact18:24
melwittefried: not that I know of. have to dig a bit but I doubt it's accessible18:27
*** betherly has joined #openstack-nova18:30
mriedemmelwitt: just a thought on the downstream deployment problem, as a workaround those tools that run the archive cron could check if any compute service are down (that info is in the api) and not do the archive18:31
*** eharney has quit IRC18:32
mriedemsomething like,18:32
mriedemif [ openstack compute service list -f value -c State | grep down ]; then don't run archive18:33
melwittmriedem: thanks. that could mitigate some of it, the other issue is that the reap task does not run immediately during nova-compute start until https://review.opendev.org/657132 landed this cycle. which means reap will not run until 30 min have elapsed since starting nova-compute, by default. and I unfortunately did not create a bug for that change, so does that cause a problem for wanting to backport?18:34
mriedemit's a behavior change so yeah18:35
melwittso prior to that change, would have to check whether it's down and also how long since it was down18:35
*** betherly has quit IRC18:35
melwittyeah. that's why I was going more toward wanting to backport --before18:35
melwittfor archive18:36
mriedemhow long since it was down isn't a thing we record18:36
mriedemin the api anyway18:36
mriedemwe do have last_seen_up in the db18:36
*** eharney has joined #openstack-nova18:36
melwittyeah, I was thinking something like that, if there's a way to see when it started or came up, could do a time delta on that18:37
mriedemthe api does have an updated_at it returns which may or may not help18:37
melwittefried: it looks like a virt driver could get to the compute manager (though not supposed to, I think) by self.virtapi._compute, so if the context were stored not only on service but also on the compute manager, then the driver could get it e.g. self.virtapi._compute.ctxt. I don't know whether that would be a good idea, just saying for the sake of thinking about it18:45
melwittthe service creates the manager, so adding the context to the manager would be simple18:46
efriedmelwitt: cool, thanks. I'm adding it to get_available_resource right now and it's, predictably, a minor PITA. Not sure which is worse though18:47
efriedbut it's certainly not the only driver method to which we pass a context, so...18:48
*** betherly has joined #openstack-nova18:58
nafiuxHi Team, I’m receiving this error: “Received a sync request from an unknown host 'c1.o7k.io'. Re-created its InstanceList.” in nova-scheduler.log, and “The instance sync for host 'c1.o7k.io' did not match. Re-created its InstanceList.”, any hint?18:59
nafiuxActually, nevermind, that isn’t impacting the creationg of the instance… sorry. Thanks!18:59
nafiuxShould I worry for that message at all?19:01
*** betherly has quit IRC19:03
* aspiers wonders why devstack rolls its own ini editor instead of using crudini19:04
openstackgerritEric Fried proposed openstack/nova master: Poison context usage in periodic tasks  https://review.opendev.org/54289119:14
openstackgerritEric Fried proposed openstack/nova master: WIP: Generate and log global_request_id properly  https://review.opendev.org/67392419:14
openstackgerritEric Fried proposed openstack/nova master: Add context param to get_available_resource  https://review.opendev.org/67411219:14
openstackgerritEric Fried proposed openstack/nova master: Pass context to _get_disk_over_committed_size_total  https://review.opendev.org/67411319:14
efriedmelwitt: So I added the context param in one patch, and funneled it through to that one LibvirtDriver method in another ^19:15
efriedreally I think I'm just stalling trying to figure out the right thing to do about oslo.log's threadlocal RequestContext...19:15
mriedemartom: fyi https://review.opendev.org/#/q/topic:story/2006302+(status:open+OR+status:merged)19:16
artommriedem, the osc thing I signed up for yeah :(19:17
artomSorry, I clearly overestimated my availability for such things19:17
melwittefried: ok. I'm not up to speed on the problem, so I'll try to look at the patch later to see what you're trying to do19:19
efriedmelwitt: oh, the thing above is just to fix up the func test failure that gibi uncovered when writing the poison fixture to enforce the "don't get_[admin_]context from periodics" that you wrote.19:20
melwittoh ok19:20
efriedmelwitt: The thing that led me down the rabbit hole is the fact that we're not logging the global request ID19:21
efriedat all19:21
efriedand the (non-global) request ID that we're logging isn't the one from the context we're using to set the global request ID header we're sending to placement19:21
melwittyeah. I guess I was thinking I dunno why logging that should involve needing to overwrite thread local context19:21
efriedso the only way we can possibly correlate what's happening in n-cpu with what's happening in placement is to log the placement (local) request_id it sends back in the response.19:21
efriedwhich sucks19:21
efriedyeah, to solve the general problem I'll need a better understanding of how threads are used throughout19:22
efriedbut for this specific case, I think I have a solution19:22
melwittit would seem like you would just take the global_request_id from the thread local request context (if there is one) and send that in the placement client calls19:23
efriedbingo19:23
efriedwell, there isn't a global_request_id in the thread local request context19:23
efriedwhich I would like to remedy as well19:23
efriedthough separately19:23
*** tesseract has quit IRC19:23
efriedbut context.global_id is a @property that gives you the global_request_id or the request_id if global isn't set19:23
melwittI see. yeah, would want to store that too similarly. is it not one of the normal attrs on RequestContext I guess?19:23
melwittoh19:24
efriedso I can set it up and it'll be an improvement, and then if we can get a real global_request_id into the threadlocal guy, it'll get better without having to change it again.19:24
efriedbut what I'd like to do is be logging *and* sending across the wire a global_request_id that actually makes sense for a given operation's lifecycle.19:24
melwittmakes sense19:24
efriedwhich was the whole purpose of global_request_id to start with.19:25
aspierssean-k-mooney: https://bugs.launchpad.net/devstack/+bug/183868819:27
openstackLaunchpad bug 1838688 in devstack "API_WORKERS default too high on machines with many CPUs" [Undecided,New]19:27
efriedmelwitt: What I'd like to be the case is19:27
efried- When an operation comes into the API, we create one RequestContext for it. We should be initializing global_request_id here, and that guy should be going in the (n-api) threadlocal for logging purposes19:27
efried- Then whenever that n-api thread does a request across the wire, we send the global_request_id along. If over REST, via the header. If over RPC, by serializing the RequestContext itself - and then when we deserialize on the other side, *that* side should overwrite *its* threadlocal so the logs consistently log it.19:27
melwittefried: yeah, that makes sense. being that most of it works that way today (the creation of RequestContext on API request start and the carrying over of it over RPC), the main thing is you'd have to add global_request_id to RequestContext in general. the REST thing would be new though19:31
efriedThe REST thing is easy. It's the one bit I actually understand :)19:31
*** dpawlik has joined #openstack-nova19:32
melwittyeah, we already have normal request_id going across services over rpc, so you get should that part for free. just have to add global_request_id as a new field and make sure it's part of what gets de/serialized19:34
melwittthough I'm already seeing it in oslo.context, there's a global_request_id field. so maybe we're just not setting it19:35
*** nafiux has quit IRC19:38
melwitthttps://github.com/openstack/nova/blob/master/nova/api/auth.py#L95 which will generate a new request_id (as it's not being set) https://github.com/openstack/oslo.context/blob/master/oslo_context/context.py#L426 but global_request_id is not auto-generated19:40
efriedcorrect19:40
melwittjust stays none19:40
melwittso you're thinking grab it out of the header and set it there?19:41
melwitt(if present)19:41
efriedIf it's present in the header, set it.19:42
efriedIf it's absent from the header, I'd like to generate it as early as possible in whatever request flow we're initiating.19:42
efriedWhat I haven't even started digging into yet is where I would do that ^ from.19:42
efriedif it's just one place or multiple19:42
melwittit would be there only, I think19:42
efriednice, that would be cool.19:43
efriedI'll try shoving that change on top of this one I'm working on for the report client19:43
melwittthere's also another place in nova/api/openstack/auth.py where you could also do it for the NoAuth middleware19:43
efriedokay19:43
melwittand then because of https://github.com/openstack/oslo.context/blob/master/oslo_context/context.py#L333 you'll get de/serialization over rpc for free19:43
melwittonce you do that change, you should be able to use context.get_current().global_request_id (check for None first) to get it to send in placement client call https://github.com/openstack/oslo.context/blob/master/oslo_context/context.py#L50219:47
efriedmelwitt: Thanks, this gels with what I've been thinking.19:48
melwittlooks like there's only one other use of it I find in the codebase https://github.com/openstack/nova/blob/master/nova/utils.py#L79019:49
openstackgerritMerged openstack/nova stable/rocky: Add functional recreate test for regression bug 1825537  https://review.opendev.org/66936119:49
openstackbug 1825537 in OpenStack Compute (nova) rocky "finish_resize failures incorrectly revert allocations" [Medium,In progress] https://launchpad.net/bugs/1825537 - Assigned to Matt Riedemann (mriedem)19:49
openstackgerritMerged openstack/nova stable/rocky: Perf: Use dicts for ProviderTree roots  https://review.opendev.org/67018219:49
openstackgerritMerged openstack/nova stable/rocky: doc: Fix a parameter of NotificationPublisher  https://review.opendev.org/67022519:49
openstackgerritMerged openstack/nova stable/rocky: Stabilize unshelve notification sample tests  https://review.opendev.org/66911819:49
openstackgerritMerged openstack/nova stable/rocky: docs: Correct issues with 'openstack quota set' commands  https://review.opendev.org/67009719:49
*** ash2307 has joined #openstack-nova19:54
*** amodi has joined #openstack-nova19:55
*** ash2307 has quit IRC19:58
*** betherly has joined #openstack-nova20:00
*** ash2307 has joined #openstack-nova20:00
*** eharney has quit IRC20:03
*** betherly has quit IRC20:04
*** Luzi has joined #openstack-nova20:06
aspiersany SQLAlchemy experts here? I'm wondering if it's really OK that every heartbeat has a COMMIT *and* ROLLBACK after it http://paste.openstack.org/show/755199/20:19
aspiersI'm seeing mysqld get an endless stream of heartbeats from conductor, and it's also writing constantly at 2--5MB/s which seems insane for a devstack cloud sitting there doing literally nothing20:21
aspiersmysqld is writing constantly, I mean20:21
aspiersefried: does this make any sense to you?20:22
aspiersmaybe I'm just missing something20:22
mriedemzzzeek is the sqlalchemy expert but i'm not sure he'd have context on what nova is doing20:24
aspiersthe weird thing is it only happens on this one devstack, not my others20:24
aspiersone question is whether rollback after every commit makes any sense20:24
efriedI wouldn't think we should be doing that for normal non-error cases, no.20:25
aspiersanother is why there are so many damn selects constantly checking the services table20:25
aspierssurely heartbeats should only happen every few seconds at most20:25
mriedemwhat's your underlying mysql library?20:25
mriedemmysqldb?20:25
mriedempython-mysql?20:25
aspierschecking20:25
mriedemhttps://stackoverflow.com/questions/13287749/should-i-commit-after-a-single-select20:25
aspiersyeah I found that one20:26
aspiersit doesn't explain select -> commit -> rollback though20:26
aspiersonly select -> commit or select -> rollback20:26
mriedemi'm not sure why it's doing a commit at all since it's a select query not changing anything20:27
aspiershrm, looks like this is not related to https://bugs.launchpad.net/devstack/+bug/1838688 after all20:27
openstackLaunchpad bug 1838688 in devstack "API_WORKERS default too high on machines with many CPUs" [Undecided,New]20:27
aspiersmriedem: that's exactly what I thought20:27
mriedemunless the query is in an engine facade transaction context manager (oslo.db thing) and it's just adding that on automatically20:28
aspiersunless there's some kind of locking or transactional thing to avoid race conditions, but that is clutching at straws20:28
aspierswell yeah that would seem more likely20:28
mriedemwould need to identify where the query is coming from, but it's probably the service group API heartbeat stuff20:28
aspiersI just had a look20:28
melwittlooks like heartbeat goes every 10 seconds20:28
aspiersI found the Service object and the last_updated_time, and service_is_up() / is_up()20:28
aspiersmelwitt: where is that? I couldn't find it20:29
openstackgerritEric Fried proposed openstack/nova master: Correct global_request_id sent to Placement  https://review.opendev.org/67412920:29
mriedemhttps://github.com/openstack/nova/blob/master/nova/servicegroup/drivers/db.py20:29
efriedmelwitt: let's see how that shakes out ^20:29
mriedemaspiers: periodic is run here v20:29
mriedemhttps://github.com/openstack/nova/blob/master/nova/servicegroup/drivers/db.py#L5320:29
aspiersoh yeah20:29
aspiersthanks20:29
mriedemhttps://github.com/openstack/nova/blob/master/nova/servicegroup/api.py#L4720:29
mriedemhttps://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.report_interval20:30
mriedemso yeah you'll see that every 10 seconds per service20:30
mriedemhere is the select i think https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L55020:31
melwittand every heartbeat will do a write to the database20:31
mriedemeach report interval saves the report_count increment20:31
mriedemand that service_update does a select first20:31
aspiersI see no writes in general_log though20:31
aspiersonly selects20:31
aspierslike the paste above20:31
melwittwhich conductor is this? super conductor or cell conductor20:32
aspiersboth I think20:32
aspierslet me check20:32
aspiersoriginally I thought this was due to having an insane number of workers20:32
aspiersbut my other SEV boxen have lots and their mysqld is sitting there doing nothing20:33
aspiersthis one is hovering between 2MB/s and 15MB/s total disk write20:33
aspiersand it's a devstack with 0 users :-o20:34
melwittfwiw, it's a well-known operational problem for computes to be hammering the database with heartbeats. that's why long ago yahoo was pushing for fixes to the zookeeper servicegroup driver, since that uses a passive "watch" concept where when the node you're watching goes down, it calls you. IIRC20:34
melwittbut this sounds like something new and out of the ordinary? I dunno20:35
aspierswhy does it have to save the report_count increment?20:35
aspiersand shouldn't just the parent be heartbeating, not the workers?20:35
melwittthe main point is the update the service record, so that looking at it's update time shows you whether it's "down"20:35
melwittsomething like that20:36
melwittI haven't looked at this in a long time20:36
aspiersah OK, I'm confusing two things20:36
aspiers1) a service heartbeating to say it's alive20:36
aspiers2) a service checking the heartbeat of another service20:37
aspiersthe SQL spam I'm seeing has to be 2) I think20:37
aspiersbecause it's only doing SELECTs20:37
aspiersnot updates20:37
aspiersas in http://paste.openstack.org/show/755199/20:37
melwittok. sorry I was not up to date on what you were talking about20:37
aspiersno totally my fault since I was not clear in my head until just now20:38
aspiersIIUC the join() mriedem pointed out is 1) not 2)20:38
melwittso the question is, what is checking for is_up if no requests are happening20:38
aspiersexactly20:38
melwitt*the question that you are asking20:38
melwittyeah, I think the join() is 1)20:38
aspiersalso, does DbDriver.is_up() really read the db?20:38
melwittyes, I think so20:39
aspiersI only see it checking the Service object's last_seen_up20:39
melwittyeah... tracing that20:39
mriedemno it doesn't20:39
mriedem_report_state is what runs in the cron and updates the report_count,20:39
mriedemwhich down in the db api changes last_seen_up20:39
mriedemwhich is what is_up checks20:39
aspiersso where the **** are these selects coming from?20:39
melwittok, my mistake20:40
*** spatel has quit IRC20:41
mriedemthere could be something hitting a service version check20:41
mriedemand not caching it20:41
mriedemtry turning this on? https://docs.openstack.org/nova/latest/configuration/config.html#database.connection_trace20:41
aspierswhoa cool!20:42
melwittwell, ok, is_up(service) tells you whether the service is "down" but you have to pass it a service object, which you most of the time got from a db query. this is again, only during requests though20:43
aspiersoh I guess I need connection_debug too20:44
mriedemdo any of the services in any of the dbs have version=0?20:44
aspiersmriedem: checking20:44
mriedembecause if so, i think you could hit a case where every service save checks the min version, finds 0 and decides not to cache it20:44
mriedemand that might explain why you'd have one busted devstack and not others20:45
aspiersthey're all version 38 AFAICS20:45
mriedemin nova_cell1?20:45
aspiersboth cells20:45
mriedemwell then i guess you're going connection trace debugging20:46
aspierswhich is good because dumbass here didn't even know about connection_debug until just now20:46
aspierswhat's a good value to start with?20:46
aspiershah I guess I'll find out20:46
mriedem...true?20:46
aspiersit's an int20:46
aspiers100=everything20:46
aspiers0=none20:46
mriedemhttps://docs.openstack.org/nova/latest/configuration/config.html#database.connection_trace is a bool20:46
aspiersconnection_debug not _trace20:47
mriedemsure,20:47
mriedembut you want to know where it's coming from right?20:47
aspiersI'm assuming I need _debug on for _trace to have effect?20:47
mriedemidk20:47
aspiersI'll try :)20:47
mriedemif it turns out it's all shitty to configure, then we should update help on those options20:47
aspiersbefore I'd just told mysqld set global general_log=1;20:47
aspiersand then it dumps to /var/lib/mysql/devstack.log20:48
aspiersbut it doesn't give any context, just the internal mysqld process ids20:48
aspierswhich you then have to map back to normal pids via mysqladmin processlist and lsof -i tcp:$port20:48
aspiersthis way will be much better I guess20:48
efriedNova meeting in 12 minutes in #openstack-meeting20:49
mriedemi can't even keep services up and running in my devstack, with API_WORKERS=120:49
mriedemthings just randomly fall over20:49
aspierslol20:49
aspiersoh, on that note20:49
aspiersturns out I wasn't having the same issue as you20:49
aspiersit was a hardware thing20:49
aspiersrandom segfaults all over the place20:49
aspiersI guess when you have 512GB RAM the chances of some of it being bad are quite high20:49
mriedemi just get,20:50
mriedemAug 01 20:36:05 devstack systemd[1]: devstack@n-sch.service: Main process exited, code=dumped, status=11/SEGV20:50
mriedemAug 01 20:36:05 devstack systemd[1]: devstack@n-sch.service: Failed with result 'core-dump'.20:50
aspiersyeah that's what I was getting20:50
aspiersbut it affected non-python stuff too20:50
aspiersand weird kernel oops etc.20:50
aspiersand random system hangs20:50
mriedemthis is an 8 VCPU, 8 GB RAM, 200 GB disk vm and i've never had this kind of persistent problem with a devstack vm that is basically configured like the gate nodes20:50
*** takashin has joined #openstack-nova20:50
aspierswell it could be a hw issue for you too I guess20:51
aspiersalthough that would be a slightly weird coincidence20:51
mriedemthis image is a bit old (18.04 LTS from last year) so i probably should have updated the os20:51
aspiersmaybe do a memory check?20:51
aspiersor install the python gdb extensions and do a coredumpctl gdb and py-bt20:51
mriedem$ free -h20:51
mriedem              total        used        free      shared  buff/cache   available20:51
mriedemMem:           7.8G        3.4G        747M        3.2M        3.7G        4.1G20:51
mriedemSwap:            0B          0B          0B20:51
aspiersI meant a memory hardware soak test20:52
aspierslike those ones you can do from a tiny boot image20:52
aspiersalternatively the python gdb extension sounds like a great way to pin it down, I was about to try that just as I realised it must be a h/w issue20:53
aspiersOK now I'm getting useful info20:57
aspiersit's coming from service_update() in db/api.py20:58
*** betherly has joined #openstack-nova20:59
aspiersbut the trace is only 3 layers deep20:59
aspiersI can add debug20:59
mriedemyeah it's doing a query of the service on each save21:00
efriednova meeting now21:01
mriedembut you need to find what's calling Service.save21:02
melwittisn't it the heartbeats?21:02
melwitt_report_state21:02
mriedemhe said he wasn't seeing writes/updates,21:02
mriedemwhich should happen if report_count changes21:02
melwittok, I thought that's what "it's coming from the service_update" meant21:03
melwittnevermind then21:03
*** betherly has quit IRC21:03
mriedembtw, watching top in devstack cinder-volume seems to be pretty cpu heavy at idle21:06
mriedemwell, not a high % but always running21:06
*** nafiux has joined #openstack-nova21:13
*** betherly has joined #openstack-nova21:19
*** betherly has quit IRC21:24
openstackgerritMerged openstack/nova stable/rocky: Revert resize: wait for events according to hybrid plug  https://review.opendev.org/67064821:31
openstackgerritMerged openstack/nova stable/rocky: libvirt: move checking CONF.my_ip to init_host()  https://review.opendev.org/67215521:31
openstackgerritMerged openstack/nova stable/rocky: Fix type error on call to mount device  https://review.opendev.org/66966421:31
openstackgerritMerged openstack/nova stable/rocky: Avoid crashing while getting libvirt capabilities with unknown arch names  https://review.opendev.org/67274621:31
openstackgerritIgor D.C. proposed openstack/nova master: Libvirt: add nfv job  https://review.opendev.org/65219721:35
*** artom has quit IRC21:39
*** Luzi has quit IRC21:46
openstackgerritTakashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (13)  https://review.opendev.org/57602021:46
openstackgerritTakashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (14)  https://review.opendev.org/57602721:46
openstackgerritTakashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (15)  https://review.opendev.org/57603121:46
openstackgerritTakashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (16)  https://review.opendev.org/57629921:46
openstackgerritTakashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (17)  https://review.opendev.org/57634421:47
openstackgerritTakashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (18)  https://review.opendev.org/57667321:48
mriedemmelwitt: efried: posted https://blueprints.launchpad.net/nova/+spec/policy-rule-for-host-status-unknown21:49
openstackgerritTakashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (19)  https://review.opendev.org/57667621:49
openstackgerritTakashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (20)  https://review.opendev.org/57668921:49
openstackgerritTakashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (21)  https://review.opendev.org/57670921:49
openstackgerritTakashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (22)  https://review.opendev.org/57671221:49
*** betherly has joined #openstack-nova21:50
melwittmriedem: thank you21:50
mriedemand with that i'm going to put on my lawn mowing outfit and hit the yard https://i.ytimg.com/vi/DOwqfxkx6mE/maxresdefault.jpg21:52
mriedemo/21:52
*** mriedem has quit IRC21:53
mnaseri left a review on that spec21:55
*** betherly has quit IRC21:55
* mnaser has a clusterf' of reviews and gerrit emails disabled so pings welcome D:21:55
*** slaweq has quit IRC22:10
*** betherly has joined #openstack-nova22:11
*** slaweq has joined #openstack-nova22:11
openstackgerritEric Fried proposed openstack/nova master: WIP: Always set a global_request_id in RequestContext  https://review.opendev.org/67413822:12
*** betherly has quit IRC22:15
*** slaweq has quit IRC22:16
*** BjoernT_ has quit IRC22:17
*** jbernard_ has joined #openstack-nova22:29
*** fyx_ has joined #openstack-nova22:29
*** dustinc_ has joined #openstack-nova22:29
*** mnasiadka_ has joined #openstack-nova22:30
*** jrosser_ has joined #openstack-nova22:30
*** kmalloc_ has joined #openstack-nova22:30
*** betherly has joined #openstack-nova22:31
*** Ben78 has joined #openstack-nova22:34
Ben78Can a user create a VM and a new volume with a single curl command?22:35
*** betherly has quit IRC22:36
*** fyx has quit IRC22:37
*** jbernard has quit IRC22:37
*** mordred has quit IRC22:37
*** mnasiadka has quit IRC22:37
*** kmalloc has quit IRC22:37
*** dustinc has quit IRC22:37
*** jrosser has quit IRC22:37
*** mnasiadka_ is now known as mnasiadka22:37
*** kmalloc_ is now known as kmalloc22:37
*** fyx_ is now known as fyx22:37
*** dustinc_ is now known as dustinc22:37
*** jrosser_ is now known as jrosser22:37
openstackgerritTakashi NATSUME proposed openstack/nova master: Add database schema upgrade check  https://review.opendev.org/66704722:39
*** panda has quit IRC22:41
*** panda has joined #openstack-nova22:42
*** mordred has joined #openstack-nova22:44
*** mlavalle has quit IRC22:45
*** tkajinam has joined #openstack-nova22:51
*** whoami-rajat has quit IRC22:55
*** nafiux has quit IRC23:00
*** Ben78 has quit IRC23:02
*** ivve has quit IRC23:05
*** bbowen has quit IRC23:06
*** nafiux has joined #openstack-nova23:06
*** igordc has quit IRC23:10
*** betherly has joined #openstack-nova23:12
*** betherly has quit IRC23:18
*** rcernin has joined #openstack-nova23:21
*** artom has joined #openstack-nova23:52
*** betherly has joined #openstack-nova23:54
*** trident has quit IRC23:54
*** betherly has quit IRC23:58
*** trident has joined #openstack-nova23:59
« Wednesday, 2019-07-31 Index Friday, 2019-08-02 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!