Thursday, 2019-10-31

*** luksky has quit IRC00:04
*** gyee has quit IRC00:07
*** slaweq has joined #openstack-nova00:11
*** slaweq has quit IRC00:15
*** rajinir has quit IRC00:18
*** mgoddard has quit IRC00:22
*** mgoddard has joined #openstack-nova00:23
*** rajinir has joined #openstack-nova00:24
*** markvoelker has joined #openstack-nova00:30
*** JamesBenson has joined #openstack-nova00:38
*** jawad_axd has joined #openstack-nova00:39
*** JamesBenson has quit IRC00:39
*** jawad_axd has quit IRC00:43
*** ociuhandu has joined #openstack-nova01:08
*** markvoelker has quit IRC01:09
*** Liang__ has joined #openstack-nova01:11
*** Liang__ is now known as LiangFang01:11
*** brault has quit IRC01:14
*** nanzha has joined #openstack-nova01:14
*** jawad_axd has joined #openstack-nova01:20
*** ociuhandu has quit IRC01:22
*** jawad_axd has quit IRC01:25
openstackgerritMerged openstack/nova master: nova-net: Use deepcopy on value returned by NeutronFixture  https://review.opendev.org/68813901:25
*** brault has joined #openstack-nova01:27
*** ociuhandu has joined #openstack-nova01:51
*** Xuchu has joined #openstack-nova01:52
*** ociuhandu has quit IRC01:57
*** ileixe has quit IRC02:02
openstackgerritmelanie witt proposed openstack/nova master: Replace time.sleep(10) with service forced_down in tests  https://review.opendev.org/69225202:04
*** ricolin has joined #openstack-nova02:05
openstackgerritmelanie witt proposed openstack/nova master: Replace time.sleep(10) with service forced_down in tests  https://review.opendev.org/69225202:07
*** ileixe has joined #openstack-nova02:10
*** slaweq has joined #openstack-nova02:11
*** slaweq has quit IRC02:16
*** jawad_axd has joined #openstack-nova02:22
*** jawad_axd has quit IRC02:27
*** jawad_axd has joined #openstack-nova02:43
*** jawad_axd has quit IRC02:48
*** brault has quit IRC02:53
*** brault has joined #openstack-nova02:53
*** tkajinam has quit IRC03:04
openstackgerritBrin Zhang proposed openstack/nova-specs master: Support re-configure deleted_on_termination in server  https://review.opendev.org/58033603:05
*** markvoelker has joined #openstack-nova03:10
*** markvoelker has quit IRC03:16
*** ociuhandu has joined #openstack-nova03:24
*** ileixe has quit IRC03:24
*** ileixe has joined #openstack-nova03:27
*** ociuhandu has quit IRC03:29
*** psachin has joined #openstack-nova03:33
*** mkrai has joined #openstack-nova03:34
*** ociuhandu has joined #openstack-nova03:45
*** ociuhandu has quit IRC03:50
*** igordc has joined #openstack-nova04:09
*** slaweq has joined #openstack-nova04:11
*** mkrai has quit IRC04:14
*** tkajinam has joined #openstack-nova04:14
*** slaweq has quit IRC04:16
*** artom has quit IRC04:20
*** xXraphXx has quit IRC04:32
*** mkrai has joined #openstack-nova04:35
*** prometheanfire has joined #openstack-nova04:39
*** mdbooth has quit IRC05:00
*** ociuhandu has joined #openstack-nova05:01
*** mdbooth has joined #openstack-nova05:02
*** ociuhandu has quit IRC05:06
openstackgerritMerged openstack/nova master: Update help for image_cache_manager_interval option  https://review.opendev.org/67895405:08
openstackgerritMerged openstack/nova master: Only allow one scheduler service in tests  https://review.opendev.org/68248605:08
*** udesale has joined #openstack-nova05:12
*** nanzha has quit IRC05:30
*** ociuhandu has joined #openstack-nova05:30
*** nanzha has joined #openstack-nova05:31
*** ociuhandu has quit IRC05:35
*** ociuhandu has joined #openstack-nova05:41
*** ociuhandu has quit IRC05:45
*** ratailor has joined #openstack-nova05:47
*** dave-mccowan has quit IRC05:51
*** dpawlik has joined #openstack-nova06:05
*** slaweq has joined #openstack-nova06:11
*** slaweq has quit IRC06:15
*** jawad_axd has joined #openstack-nova06:17
*** janki has joined #openstack-nova06:37
*** links has joined #openstack-nova06:37
*** icarusfactor has quit IRC06:38
*** ccamacho has quit IRC06:39
*** mkrai has quit IRC06:45
*** tesseract has joined #openstack-nova06:45
*** dpawlik has quit IRC06:57
*** nanzha has quit IRC06:57
*** dpawlik has joined #openstack-nova06:58
*** jawad_axd has quit IRC07:05
*** dpawlik has quit IRC07:07
*** nanzha has joined #openstack-nova07:09
*** markvoelker has joined #openstack-nova07:13
*** ratailor_ has joined #openstack-nova07:13
*** ratailor has quit IRC07:15
*** markvoelker has quit IRC07:17
*** dpawlik has joined #openstack-nova07:20
*** jawad_axd has joined #openstack-nova07:22
*** igordc has quit IRC07:34
*** dpawlik has quit IRC07:35
*** luksky has joined #openstack-nova07:39
*** ratailor_ has quit IRC07:42
*** brtknr has quit IRC07:42
*** brtknr has joined #openstack-nova07:46
*** pcaruana has joined #openstack-nova07:50
*** lpetrut has joined #openstack-nova07:51
*** trident has quit IRC07:53
*** damien_r has joined #openstack-nova07:54
*** slaweq has joined #openstack-nova07:55
*** dpawlik has joined #openstack-nova07:56
*** slaweq has quit IRC07:57
*** Xuchu has quit IRC07:57
*** trident has joined #openstack-nova08:00
*** nanzha has quit IRC08:02
*** slaweq has joined #openstack-nova08:06
*** Dinesh_Bhor has joined #openstack-nova08:08
*** abaindur has quit IRC08:10
*** nanzha has joined #openstack-nova08:14
*** tssurya has joined #openstack-nova08:21
*** Xuchu has joined #openstack-nova08:25
*** zbr has quit IRC08:37
*** tkajinam has quit IRC08:39
*** ralonsoh has joined #openstack-nova08:44
openstackgerritBalazs Gibizer proposed openstack/nova master: Use API calls to force down compute in func test  https://review.opendev.org/69229108:47
*** ccamacho has joined #openstack-nova08:51
*** rpittau|afk is now known as rpittau08:52
*** ratailor has joined #openstack-nova08:52
*** ivve has joined #openstack-nova08:53
*** mkrai has joined #openstack-nova08:56
*** tetsuro has quit IRC08:57
*** tetsuro has joined #openstack-nova08:59
*** tetsuro has quit IRC09:04
*** tetsuro has joined #openstack-nova09:05
*** ccamacho has quit IRC09:09
*** brault has quit IRC09:10
*** tetsuro has quit IRC09:10
*** brault has joined #openstack-nova09:10
*** tetsuro has joined #openstack-nova09:11
*** zbr has joined #openstack-nova09:12
*** ociuhandu has joined #openstack-nova09:15
*** LiangFang has quit IRC09:24
*** artom has joined #openstack-nova09:32
openstackgerritAdit Sarfaty proposed openstack/nova master: Avoid fetching metadata when no subnets found  https://review.opendev.org/67924709:33
*** brault has quit IRC09:33
*** priteau has joined #openstack-nova09:33
*** aspiers has joined #openstack-nova09:35
*** jangutter has quit IRC09:37
*** mkrai has quit IRC09:40
*** trident has quit IRC09:42
*** xek_ has joined #openstack-nova09:45
*** trident has joined #openstack-nova09:49
openstackgerritAdit Sarfaty proposed openstack/nova master: Avoid fetching metadata when no subnets found  https://review.opendev.org/67924709:53
*** maciejjozefczyk has joined #openstack-nova09:55
*** nanzha has quit IRC09:57
*** nanzha has joined #openstack-nova09:57
*** sapd1 has joined #openstack-nova10:03
*** tssurya has quit IRC10:06
*** ociuhandu has quit IRC10:19
openstackgerritAdit Sarfaty proposed openstack/nova master: Avoid fetching metadata when no subnets found  https://review.opendev.org/67924710:19
*** dviroel has joined #openstack-nova10:19
*** dtantsur|afk is now known as dtantsur10:20
*** jawad_axd has quit IRC10:21
*** Xuchu has quit IRC10:22
*** sapd1_ has quit IRC10:23
*** sapd1_ has joined #openstack-nova10:24
*** jawad_axd has joined #openstack-nova10:26
openstackgerritAdit Sarfaty proposed openstack/nova master: Avoid fetching metadata when no subnets found  https://review.opendev.org/67924710:29
*** jawad_axd has quit IRC10:31
*** maciejjozefczyk has quit IRC10:34
*** maciejjozefczyk has joined #openstack-nova10:36
*** tbachman has quit IRC10:40
*** jawad_axd has joined #openstack-nova10:42
*** mkrai has joined #openstack-nova10:44
*** sapd1 has quit IRC10:54
*** jangutter has joined #openstack-nova10:57
*** maciejjozefczyk has quit IRC10:58
*** tetsuro has quit IRC10:59
*** rpittau is now known as rpittau|bbl11:00
*** damien_r has quit IRC11:02
openstackgerritMerged openstack/nova master: Add missing parameter  https://review.opendev.org/68288611:09
*** markvoelker has joined #openstack-nova11:15
*** tkajinam has joined #openstack-nova11:17
*** markvoelker has quit IRC11:20
openstackgerritBrin Zhang proposed openstack/nova-specs master: Proposal for a safer noVNC console with password authentication  https://review.opendev.org/62312011:20
openstackgerritBrin Zhang proposed openstack/nova-specs master: Proposal for a safer noVNC console with password authentication  https://review.opendev.org/62312011:22
*** jangutter has quit IRC11:27
*** sapd1_ has quit IRC11:27
*** sapd1 has joined #openstack-nova11:28
*** ccamacho has joined #openstack-nova11:30
*** ratailor has quit IRC11:34
*** jangutter has joined #openstack-nova11:42
*** jangutter has quit IRC11:47
*** maciejjozefczyk has joined #openstack-nova11:51
*** tssurya has joined #openstack-nova11:53
*** arxcruz is now known as arxcruz|lunch12:01
*** tbachman has joined #openstack-nova12:06
*** mkrai has quit IRC12:11
*** brinzhang has joined #openstack-nova12:12
*** FlorianFa has joined #openstack-nova12:13
*** markvoelker has joined #openstack-nova12:13
*** udesale has quit IRC12:14
*** udesale has joined #openstack-nova12:14
*** maciejjozefczyk has quit IRC12:14
*** luksky has quit IRC12:17
*** damien_r has joined #openstack-nova12:18
*** tssurya has quit IRC12:19
*** damien_r has quit IRC12:20
*** damien_r has joined #openstack-nova12:20
*** jangutter has joined #openstack-nova12:23
*** dave-mccowan has joined #openstack-nova12:30
*** sapd1 has quit IRC12:30
*** gibi is now known as gibi_off12:31
*** sapd1 has joined #openstack-nova12:31
*** damien_r has quit IRC12:38
*** sapd1 has quit IRC12:46
*** sapd1 has joined #openstack-nova12:47
*** tssurya has joined #openstack-nova12:52
*** sapd1 has quit IRC12:54
*** sapd1 has joined #openstack-nova12:58
*** mkrai has joined #openstack-nova13:06
*** tkajinam has quit IRC13:06
*** tssurya has quit IRC13:07
openstackgerritMerged openstack/nova stable/train: Switch to opensuse-15 nodeset  https://review.opendev.org/69203213:09
*** arxcruz|lunch is now known as arxcruz13:09
*** mriedem has joined #openstack-nova13:13
mriedemboo13:13
*** rpittau|bbl is now known as rpittau13:16
*** panda is now known as panda|pto13:16
*** sapd1 has quit IRC13:18
*** sapd1 has joined #openstack-nova13:19
*** maciejjozefczyk has joined #openstack-nova13:21
mdboothsean-k-mooney: I was about to start reviewing https://review.opendev.org/#/c/687957/ . Any advice?13:24
*** brinzhang_ has joined #openstack-nova13:24
*** luksky has joined #openstack-nova13:24
sean-k-mooneyam i was going to avoid depending on your refactor to make backporting simpeler13:25
sean-k-mooneyi do think you functest refactor is good but im not sure it will be backported13:25
sean-k-mooneyas such i am wondering if i should avoid intoducing the new test class and use the old way in that patch13:26
mdboothsean-k-mooney: Sure. It's not strictly required either as I think I mentioned, as you can just inherit from the class I ripped everything out of.13:26
mdboothThe additional cruft isn't actually harmful if you're not using it.13:27
sean-k-mooneyya13:27
*** brinzhang has quit IRC13:28
*** dtantsur is now known as dtantsur|brb13:28
*** janki has quit IRC13:28
sean-k-mooneyanyway if you have input on how i should adress this let me know and i can adress it13:28
*** sapd1 has quit IRC13:29
*** sapd1 has joined #openstack-nova13:29
openstackgerritFan Zhang proposed openstack/nova master: Rollback persistent domain xml after DeviceDetachFailed.  https://review.opendev.org/69237013:30
*** jaosorior has joined #openstack-nova13:30
*** abhishekk has joined #openstack-nova13:33
*** psachin has quit IRC13:35
*** sapd1 has quit IRC13:36
efriedmriedem: https://zuul.opendev.org/t/openstack/build/aeac2cbd2fc94751a4a5eb7e86b70981/log/logs/subnode-2/screen-n-cpu.txt.gz?severity=4 I don't see anything rbd in any of our requirements files. Does it have another name?13:39
*** ygk_12345 has joined #openstack-nova13:39
*** sapd1 has joined #openstack-nova13:40
openstackgerritMatt Riedemann proposed openstack/nova master: Add functional test for two-cell scheduler behaviors  https://review.opendev.org/45200613:45
mriedemefried: librados13:46
mriedem?13:46
mriedemit's not on pypi,13:47
efriedthat neiter13:47
mriedemit's a python lib built from the rbd binary13:47
efriedneither13:47
mriedemi think13:47
efriedthat error is only showing up on the "stop testing with py2" patch and the one above it, per logstash. But I don't see how that patch could have affected that job.13:47
mriedemefried: we should do https://review.opendev.org/#/c/691765/ first then13:48
mriedemin case it's related13:48
mriedemthough,13:48
mriedemthat's not the job that's failing in the drop py2 change right?13:48
*** maciejjozefczyk has quit IRC13:49
mriedemthe nova-live-migration and nova-grenade-multinode jobs have a post-test hook script that sets up ceph13:49
efriedcorrect, but I'll approve that change anyway.13:49
mriedemand those likely aren't setting USE_PYTHON313:49
mriedemi can wip something up quick to try to move those to py313:50
efriedthe other one is failing with a weird PyYAML dep13:50
efriedmriedem:  stuff it in the pile underneath 'remove py2'?13:50
mriedemyes13:50
mriedemdoes this call for a "back to the pile" south park reference?!13:50
efriedIt wasn't intentional. My South Park knowledge is slim13:51
mriedemit's nsfw anyway13:51
efriedwhen have you ever cared about that?13:51
mriedem...13:51
mriedemi've gotten better after being burned a few times over the years13:51
mriedemdespite artom still leaving traps for me13:52
efriedThe PyYAML thing would be happening earlier than the rbd fail, so it's possible it would have hit the same thing later anyway.13:52
artommriedem, hey man, I've always had plausible deniability for everything, those traps are in your head13:52
efriedbut again, I don't see how the 'stop testing py2' patch would have changed anything that would make the hook setup change.13:53
efriedthe straight man in the comedy duo13:53
openstackgerritMatt Riedemann proposed openstack/nova master: Convert legacy nova-live-migration and nova-multinode-grenade to py3  https://review.opendev.org/69237413:53
openstackgerritMatt Riedemann proposed openstack/nova master: Convert legacy nova-live-migration and nova-multinode-grenade to py3  https://review.opendev.org/69237413:54
openstackgerritMatt Riedemann proposed openstack/nova master: Stop testing Python 2  https://review.opendev.org/68795413:54
mriedemlet's see how that works13:55
mriedemmels, then mine, then stephens13:55
efriedzuul kicked it13:55
efriednm13:55
efriednova meeting in five minutes13:55
sean-k-mooneyefried: that reminds me i need to recheck your os-vif job change.13:57
sean-k-mooneyi think the devstack patch should now be merged13:57
sean-k-mooneyactully no it faild to merge and is being rechecked so ill wait13:58
efriedthanks for keeping track of that sean-k-mooney14:00
*** dklyle has quit IRC14:00
*** dklyle has joined #openstack-nova14:01
sean-k-mooneywe should wait until after m1 to merge the os-vif change in anycase but ill keep an eye on it till then14:02
*** mkrai has quit IRC14:05
openstackgerritMatt Riedemann proposed openstack/nova master: Default AZ for instance if cross_az_attach=False and checking from API  https://review.opendev.org/46967514:09
mriedemdansmith: updated ^ with a test and fix for a thing alex_xu found (thanks alex)14:09
dansmithI saw14:09
mriedemsean-k-mooney: i want to say this is a duplicate of an existing bug but haven't dug for it, maybe you know? https://bugs.launchpad.net/nova/+bug/185073614:10
openstackLaunchpad bug 1850736 in OpenStack Compute (nova) " MemoryPageSizeInvalid: Invalid memory page size" [Undecided,New]14:10
sean-k-mooneymriedem: i have not looked at that but ill do it now and update it14:11
sean-k-mooneyalso til packstack is still used...14:11
*** JamesBenson has joined #openstack-nova14:14
*** dpawlik has quit IRC14:16
*** ociuhandu has joined #openstack-nova14:17
*** ociuhandu has quit IRC14:22
*** mkrai has joined #openstack-nova14:24
*** brinzhang has joined #openstack-nova14:25
*** brinzhang has quit IRC14:26
*** brinzhang has joined #openstack-nova14:26
*** brinzhang has quit IRC14:27
*** JamesBenson has quit IRC14:28
*** brinzhang has joined #openstack-nova14:28
*** JamesBenson has joined #openstack-nova14:28
*** brinzhang_ has quit IRC14:28
dansmithoh jeez, today is halloween14:29
dansmithtotes forgot14:29
*** sapd1 has quit IRC14:32
*** ygk_12345 has quit IRC14:33
*** tbachman has quit IRC14:36
*** sapd1 has joined #openstack-nova14:38
*** jawad_axd has quit IRC14:39
*** dtantsur|brb is now known as dtantsur14:40
*** jawad_axd has joined #openstack-nova14:40
*** lpetrut has quit IRC14:43
mdboothsean-k-mooney: My concern with the 'Don't clean anything up because we want to backport it' philosophy is that it means nothing ever gets cleaned up, which means we just end up accumulating duplicated crap. You could move the generic parts of NUMAServersTestNew to a reusable location and it would14:43
mdboothnot impact the backportability of this patch at all.14:43
mriedemdansmith: i don't want to alarm you, but you should probably get to your panic room in the basement soon14:43
mriedemthe kids start earlier and earlier14:43
mdboothAt some point you have to clean stuff up. That impacts backports. The alternative is never clean anything up.14:44
mriedemmdbooth: it's not that black and white,14:44
mriedemif you've got 3 refactors leading up to a bug fix that is going to be backported, try to minimize change to make the fix backportable, and stack the refactors on top14:44
dansmithmriedem: remember, my house automatically shuts off all front-facing illumination on 10/3114:44
dansmithit remembered, I didn't14:44
*** jawad_axd has quit IRC14:45
mdboothdansmith: Nice :)14:45
* mdbooth also hates halloween.14:45
*** jawad_axd has joined #openstack-nova14:45
mriedemthe cirros image we're using in the gate (0.4.0) is ~2 years old...14:45
mriedembut there isn't anything newer14:46
* mdbooth blames the USA for exporting it to the UK, btw. Halloween wasn't a thing when I was a kid.14:46
efriedmriedem: here's that rbd thing http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22RuntimeError%3A%20rbd%20python%20libraries%20not%20found%5C%2214:46
sean-k-mooneymdbooth: halloween is an irish holiday14:46
sean-k-mooneywe exported it to the us and they may have exported it to the uk14:46
mdboothsean-k-mooney: What are you doing at work, then?14:46
efriedmriedem: am I reading that correctly, 80 hits in like 12 minutes?14:47
mdboothOk, I may have claimed to have no beef with Ireland in the past, but now I'm mad.14:47
sean-k-mooneywe had the bankholiday on monday14:47
*** jmlowe has quit IRC14:47
sean-k-mooneytrick or treating was not part of the irish holiday by the way14:47
sean-k-mooneythat was added in the us14:48
mdboothmriedem: Sure, but in the case of sean-k-mooney's patch the 'refactor' is just putting the generic stuff somewhere else.14:48
sean-k-mooneyit orgially was a pagan festival with games and music and stuff14:48
mdboothAnd if you invert the order, it's still going to hit the backportability of the next thing14:49
mdboothSo if the cost is low, meh14:49
sean-k-mooneymdbooth: mriedem i have no issue with wriing another patch to refactor my test on top of mdbooth refactor by the way14:49
mdboothsean-k-mooney: My thing is gone14:49
mdboothsean-k-mooney: I'm not using libvirt for that functional test any more14:49
sean-k-mooneyoh ok did not know that14:49
sean-k-mooneywell i dont need libvirt in this case either14:49
sean-k-mooneyits just the only numa test we currently have14:50
*** jawad_axd has quit IRC14:50
mdboothsean-k-mooney: I was just advocating that rather than essentially rewriting a new libvirt functional base class for 1 test, you could actually write one and then use it14:50
mdboothDo it in the same patch if required.14:50
mdboothmriedem: How strongly do you feel about the CastAsCall thing, btw?14:51
mdboothmriedem: That was a significant change when I rewrote my test. I see the benefit, though.14:51
sean-k-mooneywell i coudl write a generic numa functional base class and use that14:51
*** abhishekk has quit IRC14:51
sean-k-mooneyand then later refactor the libvir one to inherit form it14:51
mdboothsean-k-mooney: If you went down the route of doing something like mriedem's new base class, but adding libvirtiness, you'd be losing CastAsCall.14:52
sean-k-mooneywe dont want the castAsCall stuff right?14:52
mdboothWhich I agree with in principal, especially if we're writing a new base class, as it's quite different.14:52
mdboothsean-k-mooney: Right, but your base class here will be using it.14:53
mdboothSo I'm wondering what the strength of feeling is there.14:53
sean-k-mooneyyes but may tests does not care14:53
mdboothsean-k-mooney: Ah, crap. I forgot to get coffee before your meeting ;)14:54
* mdbooth runs14:54
sean-k-mooneyi need the messaging to work so the service can comunicate but i dont depend on the castascall behavior14:54
sean-k-mooneyyep im about to do the same14:54
*** mkrai has quit IRC14:54
*** mkrai has joined #openstack-nova14:55
mriedemefried: logstash isn't showing me hits for that14:55
mriedemin 15 minutes anyway14:55
mriedemefried: it's also only a couple of changes in the last 2 days14:55
mriedemmdbooth: i don't know what sean-k-mooney patch you're referring to14:56
mriedemmdbooth: i feel pretty strongly about not building new tests on anything that uses CastAsCall,14:56
mriedemsince it's bogus for testing realistic API behavior which is the point of functional tests14:57
mriedemi'd like to nuke the castascall fixture unless we have a good reason to use it14:57
dansmithmriedem: I agree it creates unrealistic tests, but don't we use it in a crapton of places currently?14:58
sean-k-mooneymriedem: its the numa rebuild patch. which we discussed before as actully not beeing libvirt specific14:58
sean-k-mooneymriedem: and it does not need castascall14:58
mriedemdansmith: it's used in one of the parent classes for the API samples tests14:58
mriedemdansmith: so lots of functional tests inherit it unknowingly14:58
dansmithyeah14:59
sean-k-mooneyyep most proably would work fine without it14:59
sean-k-mooneywell provied the messageing worked14:59
dansmithmriedem: so that mysql 16m packet patch of yours15:03
*** jmlowe has joined #openstack-nova15:04
dansmithoh, I see you saw the problem even on that eh?15:04
*** links has quit IRC15:04
mriedemnot sure, i haven't dug into the grenade jobs on that one - unless i did and commented15:05
mriedemi did see the aborted connections though15:05
mriedemnot the grenade failure though i don't think15:05
*** nanzha has quit IRC15:05
mriedembut yeah the point is that is supposed to avoid the aborted connections15:05
*** nanzha has joined #openstack-nova15:06
dansmiththe mysql side of the error you mean15:06
mriedemright15:06
dansmithI was going to say, let's just merge that and see,15:06
dansmithbut maybe not worth it if you still saw those15:06
mriedemright15:06
mriedemkind of would like zzzeek to validate that kind of thing15:06
*** prometheanfire has left #openstack-nova15:06
dansmiththose aborted connections definitely seem like they have to be a bad thing15:07
mriedemthe change on top that enables query logging - the gd mysql slow log file is so big you can't open it15:07
dansmithmriedem: we don't capture etc/my.cnf it looks like so we can't validate that you set it properly right?15:08
*** AJaeger has left #openstack-nova15:14
mriedemyeah that's a todo15:18
dansmithmriedem: did you look for other evidence that it was set right? I dunno if it logs anything useful15:19
*** JamesBen_ has joined #openstack-nova15:19
dansmithlooks like we only get error logs15:20
*** mkrai has quit IRC15:20
dansmithI see the sudo log with the sed command at least15:21
*** JamesBenson has quit IRC15:21
dansmithmriedem: ah, that message can come from a client just dropping the connection, which could maybe be happening due to connection pooling or something less problematic15:24
dansmiththe mysql-side error message I mean15:24
mriedemok15:24
mriedemso this mostly hits in master patches, so grenade master == stable/train old and ussuri new, i'll check some stable/stein patches to see if their grenade logs have the same mysql aborted connection messages15:25
dansmithack15:26
mriedemyeah this is a stein patch that merged on 10/26, seeing the same stuff in here https://zuul.opendev.org/t/openstack/build/7fd7bdb35ade439c81023b0fb0846ee8/log/logs/mysql/error.txt.gz15:27
mriedemi wonder if there is some mysql diagnostics command we can run post-upgrade after calling verify_instance (which hits cell1) but before running smoke tests15:27
dansmithwell, I'm kinda thinking that this is just fallout from things we don't control, like the db connection pooling (or even api workers under uwsgi never getting scheduled)15:28
*** macz has joined #openstack-nova15:28
dansmithunrelated to the scheduler thing15:28
dansmithalso, hmm15:29
mriedembut something changed in train/ussuri obviously15:29
dansmithmriedem: those errors  say the user is "root".. aren't the services using actual users to connect to mysql?15:29
mriedemconnection = mysql+pymysql://root:secretmysql@127.0.0.1/nova_api?charset=utf815:31
mriedem...15:31
mriedemi guess not15:31
mriedemhttps://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8ec/688419/2/check/grenade-py3/8ec542b/logs/etc/nova/nova.conf.txt.gz15:31
dansmithhuh.15:31
mriedemsame in stein for a non-grenade job https://zuul.opendev.org/t/openstack/build/6cd13330e99346e4b688f35ec2a7ed36/log/controller/logs/etc/nova/nova_conf.txt.gz15:32
mriedemyeah DATABASE_USER in devstack defaults to MYSQL_USER which defaults to root15:34
dansmithokay always thought it was stack or nova or something15:35
mriedemhttps://docs.openstack.org/nova/train/configuration/config.html#database.connection_recycle_time defaults to 1 hour, i wonder if we should drop that to like 10 minutes or something - not sure it would help anything15:35
*** sapd1 has quit IRC15:36
*** sapd1 has joined #openstack-nova15:37
* dansmith looks to see what that even does15:37
dansmithah, like max_requests15:37
dansmiththe failure happens really early in the run though right?15:37
mriedempost-upgrade?15:38
dansmithyeah15:38
dansmithearly after starting services Imean15:38
dansmithso if you set recycle to 1s maybe it would be doing lots of reconnecting and we could see if that matters, but it doesn't seem like a fix15:39
dansmithconnection+debug=100 maybe?15:39
mriedemyeah i think so. last i checked the funny thing is we run verify_instance post-upgrade which is ok and then grenade tries to create a server or run tempest smoke and things fall over15:39
dansmithyeah, but verify_instance wouldn't be the scheduler,15:39
dansmithso if the scheduler is the only one affected somehow...15:40
dansmithhmm15:41
dansmithis scheduler getting restarted at all?15:41
mriedemit's not only the scheduler, just mostly. some jobs hit it in the api too15:41
mriedemyes scheduler gets restarted in the grenade run15:41
dansmithI don't see it in the log15:41
dansmithoh, I'm looking at a normal job15:42
mriedemi have some notes here https://bugs.launchpad.net/nova/+bug/1844929/comments/315:42
openstackLaunchpad bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed]15:42
dansmithI know, I read those previously, but I'm trying to avoid re-reading them to not get stuck down a given line of thinking15:42
mriedemwe restart the scheduler and it queries compute nodes and services from the cell1 db at that time ok15:42
mriedemack15:42
dansmithI know we should,15:43
dansmithI was just wondering if maybe we don't restart it actually and the first attempt fails15:43
dansmithso,15:44
dansmithhttps://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8ec/688419/2/check/grenade-py3/8ec542b/logs/screen-n-sch.txt.gz15:44
dansmithI see it started at 14:07:00 only15:44
dansmithare we not keeping the log or something maybe?15:44
mriedemit looks like that job failed smoke tests on the old side,15:46
mriedemso never got to upgrade15:46
mriedemhttps://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8ec/688419/2/check/grenade-py3/8ec542b/logs/grenade.sh.summary.txt.gz15:46
dansmithokay but the same error15:46
mriedemoh that's interesting15:46
dansmiththe first instance of that happens well after the last "aborted connection" message in the mysql error log15:51
dansmithwhich makes me think all those errors in the mysql log are from things we're doing during devstack setup15:51
dansmithlike one-shot client calls or something15:52
*** jawad_axd has joined #openstack-nova15:52
dansmithand 18 minutes after the last "ready for connections" message from mysql15:53
*** FlorianFa has quit IRC15:54
*** mkrai has joined #openstack-nova15:56
*** jawad_axd has quit IRC15:57
dansmithdoesn't really look like any sort of unreasonable load or kernel events around that time15:57
*** lpetrut has joined #openstack-nova15:59
*** lpetrut has quit IRC16:00
*** lpetrut has joined #openstack-nova16:00
*** mkrai has quit IRC16:01
*** mkrai has joined #openstack-nova16:01
*** luksky has quit IRC16:09
dansmithmriedem: https://review.opendev.org/#/c/692402/16:11
dansmithmriedem: have any tricks to get a lot of grenade runs on that?16:12
dansmithI know in the past we'd create like ten noop patches that depends-on something like that to get 10x the runs16:12
dansmithseems irresponsible in today's climate :)16:12
*** jawad_axd has joined #openstack-nova16:13
mriedemyou could create a grenade patch that depends-on it16:16
*** jawad_axd has quit IRC16:18
*** ivve has quit IRC16:18
*** nanzha has quit IRC16:20
*** brinzhang_ has joined #openstack-nova16:21
*** brinzhang has quit IRC16:24
*** JamesBen_ has quit IRC16:29
*** tesseract has quit IRC16:33
*** efried has quit IRC16:37
*** markvoelker has quit IRC16:40
*** efried has joined #openstack-nova16:41
*** markvoelker has joined #openstack-nova16:43
*** dtantsur is now known as dtantsur|afk16:45
*** lpetrut has quit IRC16:49
*** gyee has joined #openstack-nova16:50
*** trident has quit IRC16:55
*** jangutter has quit IRC16:57
*** jangutter has joined #openstack-nova16:58
*** trident has joined #openstack-nova17:01
*** ricolin has quit IRC17:03
*** mdbooth has quit IRC17:04
*** igordc has joined #openstack-nova17:05
openstackgerritLee Yarwood proposed openstack/nova master: WIP service: Do not perform version checks for deleted services  https://review.opendev.org/69241217:05
lyarwoodmelwitt: ^ when you're back, I'm likely missing something obvious here.17:06
*** mdbooth has joined #openstack-nova17:06
*** udesale has quit IRC17:11
mriedemlyarwood: didn't mel already fix that bug in the Instance object when lazy-loading services?17:11
mriedemhttps://review.opendev.org/#/c/582408/17:12
*** JamesBenson has joined #openstack-nova17:12
lyarwoodmriedem: that was the original bug when running through this upgrade scenario yeah but instead of ServiceNotFound we now see ServiceTooOld raised with that fix applied17:14
*** rpittau is now known as rpittau|afk17:14
*** jawad_axd has joined #openstack-nova17:14
*** jawad_axd has quit IRC17:19
*** mkrai has quit IRC17:21
mdboothsean-k-mooney: https://review.opendev.org/#/c/687957/17:25
*** brinzhang has joined #openstack-nova17:28
sean-k-mooneymdbooth: thanks for the review. ill go though it tommorow im just about to go have dinner17:29
mdboothsean-k-mooney: ack17:29
sean-k-mooneyby the way i belive the way im comparing the numa toplogy constirant should compare things like pages size too17:30
sean-k-mooneywe dont actully want to try and fit the request to a host and compare that we just want to extrat the contaritns and assert they dont change17:30
*** brinzhang_ has quit IRC17:31
sean-k-mooneyi can add some more test cases however to test different type of constraitn changes17:31
sean-k-mooneyto ensure that change hugpeage also will be detected ectra17:32
sean-k-mooneyi assume that is what you ment by poking at different levels17:32
mdboothsean-k-mooney: Yeah, I wasn't 100% on that, although I *did* go look up 1) the implementation of obj_to_primitive(), 2) the implementation of dict.__eq__ in python ;)17:33
mdboothsean-k-mooney: I think it's ok, I was just thinking for completeness17:33
mdboothBut about the exception, I think it's important to positively assert that your last test case is being rejected for the reason you expect17:34
sean-k-mooneyya i can do that17:34
mdboothRather than for some other, coincidental reason17:34
mdbooth(It is, btw, I checked that too)17:34
sean-k-mooneythe self.assertRaisss returns the exception raised17:34
mdboothIt does17:34
sean-k-mooneyso i can save it to a var and then do more asserts17:34
sean-k-mooneyim pretty sure we can do that anyway17:36
sean-k-mooneymdbooth: https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertRaises17:37
sean-k-mooneyi know we can get teh exception via the context manager apparch but i think the non context manager form just returns the exception17:37
mdboothsean-k-mooney: You can do both17:38
mdboothYou can do:17:38
mdboothex = self.assertRaises(...)17:38
sean-k-mooneyyep17:38
mdboothAnyway, I don't care what colour the bike shed is ;)17:38
sean-k-mooneyso ill leave the exsting test tehn i can assert after on the body17:38
sean-k-mooneywell it was more that there is an easy way to be more specific for that last case17:39
*** jmlowe has quit IRC17:39
sean-k-mooneyi can assert an excetion is raised and then assert what it was as you suggested17:39
*** JamesBenson has quit IRC17:47
*** ivve has joined #openstack-nova17:47
*** JamesBenson has joined #openstack-nova17:48
openstackgerritMatt Riedemann proposed openstack/nova master: Reset vm_state to original value if rebuild claim fails  https://review.opendev.org/69218518:01
*** jaosorior has quit IRC18:04
*** ociuhandu has joined #openstack-nova18:19
*** mvkr has quit IRC18:20
*** ociuhandu has quit IRC18:23
melwittmdbooth, lyarwood: thanx for the fast reviews on the get_hash_str patch!18:24
*** priteau has quit IRC18:28
*** tbachman has joined #openstack-nova18:40
*** sapd1 has quit IRC18:45
*** sapd1 has joined #openstack-nova18:46
*** ralonsoh has quit IRC18:47
efriedmriedem: flavors were previously associated with instances just by ID, right?18:54
efriedBut that was problematic because when you changed a flavor, you could no longer tell what was used to create the instance...18:55
efriedso we started copying the flavor content into the instance18:55
efriedand now we... no longer even store the original flavor ID?18:55
mriedemwe saved off the flavor information for a long time...not sure when that started, but it used to be in system_metadata rather than instance_extra18:59
melwittI think we still do store it, as instance_type column in the instances table19:00
mriedemyes, we do19:00
mriedemthe embedded instance flavor was migrated from system_metadata to instance_extra back in kilo - dan did that work19:00
mriedeminstances.instance_type_id is the flavor.id19:01
mriedemit's set on create and resize19:01
efriedis it used for anything, or just there kind of as a historical artifact?19:01
mriedemquick grok shows the simple tenant usage api uses it19:02
mriedembut probably shouldn't19:02
efriedokay.19:02
mriedemoh it's only used there if the instance.flavor doesn't load19:02
mriedemthat's super old compat code though19:02
mriedemit's used for filtering servers in GET /servers19:02
mriedemand it's in the Migration object19:03
mriedemold/new_instance_type_id19:03
mriedemi think it's mostly vestigial though19:03
efriedAnd all of this (copying the flavor rather than just referring to it) is so things like rebuild behave sanely when the flavor is changed out of band.19:03
efriedright?19:03
mriedemthe flavor doesn't change on rebuild...but you mean in case the flavor extra specs change or something?19:04
mriedemor if the flavor is deleted after the server is created and then you rebuild the server?19:04
mriedemif so, yes19:04
mriedemsame reason we squirrel away the image meta used to create and rebuild the server19:04
mriedemin case the image goes away19:04
efriedright, okay, thanks for confirming.19:04
mriedemare you asking b/c of that spec about changing how flavor extra specs work?19:05
mriedemi don't even want to look at that spec19:05
*** luksky has joined #openstack-nova19:05
dansmithchanging how they work how"19:05
dansmith?19:05
mriedemyou'd have to look...19:05
efriedYes. IMO it's a good idea.19:06
mriedemi couldn't get past the problem statement19:06
efriedhttps://review.opendev.org/#/c/663563/6/specs/backlog/approved/add-flavor-metadata-or-metadata-group.rst19:06
efriedIt's basically composable flavors.19:06
dansmithoh jesus19:06
efriedso you can have a "flavor group" that describes, say, your qos extra specs, and another that describes your numa topo, and when you build an instance, you can specify multiple groups.19:07
efriedSo that you don't have skittles.19:07
efriedkind of like what we're doing for cyborg device profiles, but generically.19:07
mriedembut we don't have dynamic flavors19:09
mriedemand what you described is basically dynamic flavors19:09
dansmithless useful and more complex than dynamic flavors19:09
mriedemcreate me an instance with this cpu/ram/disk but gimme some qos and numa on the side, regardless of how the flavor is mapped to aggregates19:09
efried"flavor mapped to aggregates'?19:10
mriedemyeah, pinning flavors to a subset of hosts in the deployment19:11
mriedemcapacity planning19:11
mriedemisolation based on hw and features19:12
mriedemetc19:12
dansmithcomposing multiple things that define super detailed requirements seems like an invitation for "I can't find any combination of flavors that boot me an instance"19:12
efriedyou mean by having a member_of in the flavor?19:12
efriedthis idea isn't suggesting removing the ability to put extra specs into the flavor afaict19:12
efriedanyway, I've got no skin in this, I'm just reviewing someone's spec. If you guys have major objections, put 'em in the review.19:13
*** dave-mccowan has quit IRC19:16
mriedemi've gotta run to my kids halloween parade thing at the school, be back later19:20
*** mriedem is now known as mriedem_afk19:21
*** abaindur has joined #openstack-nova19:24
*** dave-mccowan has joined #openstack-nova19:32
openstackgerritKobi Samoray proposed openstack/nova master: Avoid fetching metadata when no subnets found  https://review.opendev.org/67924719:46
*** eharney has quit IRC19:54
*** jawad_axd has joined #openstack-nova19:59
*** jawad_axd has quit IRC20:04
*** markvoelker has quit IRC20:06
openstackgerritMerged openstack/nova stable/ocata: Add functional regression test for bug 1669054  https://review.opendev.org/64941920:10
openstackbug 1669054 in OpenStack Compute (nova) ocata "RequestSpec.ignore_hosts from resize is reused in subsequent evacuate" [Medium,In progress] https://launchpad.net/bugs/1669054 - Assigned to Matt Riedemann (mriedem)20:10
openstackgerritMerged openstack/nova stable/ocata: Do not persist RequestSpec.ignore_hosts  https://review.opendev.org/64942020:10
*** jamesdenton has quit IRC20:17
*** jamesden_ has joined #openstack-nova20:17
*** jawad_axd has joined #openstack-nova20:20
*** jawad_axd has quit IRC20:25
*** sapd1 has quit IRC20:37
*** sapd1 has joined #openstack-nova20:39
*** mriedem_afk is now known as mriedem20:44
mriedemholy crap 2 changes merged20:44
mriedemthat's....spooktacular20:44
*** jmlowe has joined #openstack-nova20:45
*** eharney has joined #openstack-nova20:50
*** eharney has quit IRC20:59
*** xek_ has quit IRC21:04
*** JamesBen_ has joined #openstack-nova21:05
*** jawad_axd has joined #openstack-nova21:06
*** JamesBen_ has quit IRC21:08
*** ociuhandu has joined #openstack-nova21:08
*** JamesBenson has quit IRC21:09
*** jawad_axd has quit IRC21:10
*** ociuhandu has quit IRC21:13
*** pcaruana has quit IRC21:31
*** mriedem has quit IRC21:36
*** jawad_axd has joined #openstack-nova21:47
*** brinzhang_ has joined #openstack-nova21:49
*** jawad_axd has quit IRC21:52
*** brinzhang has quit IRC21:52
*** zbr has quit IRC21:58
*** zbr has joined #openstack-nova22:00
*** markvoelker has joined #openstack-nova22:07
*** jawad_axd has joined #openstack-nova22:08
*** markvoelker has quit IRC22:11
*** jawad_axd has quit IRC22:12
*** dviroel has quit IRC22:15
*** lpetrut has joined #openstack-nova22:23
*** threestrands has joined #openstack-nova22:42
*** slaweq has quit IRC22:44
*** ociuhandu has joined #openstack-nova22:48
*** ociuhandu has quit IRC22:53
*** tkajinam has joined #openstack-nova23:00
*** slaweq has joined #openstack-nova23:01
*** slaweq has quit IRC23:05
*** macz has quit IRC23:05
*** igordc has quit IRC23:10
*** jlvillal has left #openstack-nova23:15
*** ociuhandu has joined #openstack-nova23:47
*** luksky has quit IRC23:50
*** ociuhandu has quit IRC23:52

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!