Thursday, 2018-10-25

*** gyee has quit IRC00:14
*** ElCoyote_ has quit IRC00:31
*** trungnv has quit IRC00:35
*** sai_p_ has quit IRC00:36
*** ElCoyote_ has joined #openstack-ironic00:47
*** bfournie has joined #openstack-ironic00:51
*** dtrainor has quit IRC01:04
*** dtrainor has joined #openstack-ironic01:05
*** dtrainor has quit IRC01:24
jiapeiGood morning Ironicers01:31
*** tiendc has joined #openstack-ironic01:33
*** MattMan_ has quit IRC01:47
*** MattMan_ has joined #openstack-ironic01:48
*** openstackgerrit has joined #openstack-ironic02:19
openstackgerritRuby Loo proposed openstack/ironic master: [WIP] Migration step to update objects to latest version  https://review.openstack.org/61318102:19
rloodtantsur|afk: ^^ sorry, i didn't have a chance to work on it; that's what I'm thinking. Haven't tested it at all. Will touch base with you tomorrow.02:20
*** rloo has quit IRC02:21
*** munimeha1 has quit IRC02:30
*** dtrainor has joined #openstack-ironic02:42
*** jesusaur has joined #openstack-ironic03:21
*** jaganathan has quit IRC03:58
*** skazi has quit IRC04:36
openstackgerritTuan Do Anh proposed openstack/ironic master: Add download link apache configuration with mod_wsgi  https://review.openstack.org/61149704:48
*** jaganathan has joined #openstack-ironic05:10
openstackgerritDebayan Ray proposed openstack/sushy master: Follow-up to 27c725c to move up ``cache_clear``  https://review.openstack.org/61319705:28
*** skazi has joined #openstack-ironic05:34
*** skazi has quit IRC05:34
*** adrianc has joined #openstack-ironic05:53
*** jaganathan has quit IRC06:09
*** moshele has joined #openstack-ironic06:22
*** jaganathan has joined #openstack-ironic06:27
openstackgerritMerged openstack/ironic master: Fix DHCPv6 support  https://review.openstack.org/61160006:29
openstackgerritKaifeng Wang proposed openstack/python-ironic-inspector-client master: [DNM/TEST] Zuul migration  https://review.openstack.org/61321606:56
*** rcernin has quit IRC07:00
*** pcaruana has joined #openstack-ironic07:04
iurygregorygood morning all07:04
jiapeio/ morning iurygregory07:07
iurygregory\o07:07
*** SpamapS has quit IRC07:11
etingofgood morning iurygregory & jiapei o/07:21
iurygregory\o07:21
jiapeimorning etingof o/07:21
*** SpamapS has joined #openstack-ironic07:24
*** hjensas has quit IRC07:24
*** yolanda has joined #openstack-ironic07:24
arne_wiebalckGood morning, Ironic!07:26
arne_wiebalckrloo dtantsur: When preparing the upgrade from Q to R, we ran into a situation where 'db-sync upgrade' on R failed due to the object versions, but 'db-sync online_data_migrations' on Q did not find anything to migrate. FWIU, https://review.openstack.org/#/c/613181/ is supposed to fix exactly this?07:26
patchbotpatch 613181 - ironic - [WIP] Migration step to update objects to latest v... - 1 patch set07:26
openstackgerritDebayan Ray proposed openstack/sushy master: Follow-up to 27c725c to move up ``cache_clear``  https://review.openstack.org/61319707:28
*** hjensas has joined #openstack-ironic07:28
*** Emine has joined #openstack-ironic07:46
*** serlex has joined #openstack-ironic07:58
*** dougsz has joined #openstack-ironic08:10
*** moshele has quit IRC08:14
openstackgerritKaifeng Wang proposed openstack/python-ironic-inspector-client master: [DNM/TEST] Zuul migration  https://review.openstack.org/61321608:29
*** e0ne has joined #openstack-ironic08:42
*** stendulker has joined #openstack-ironic08:52
*** stendulker has quit IRC08:58
*** hkominos has joined #openstack-ironic09:05
*** dtantsur|afk is now known as dtantsur09:09
dtantsurmorning ironic09:09
dtantsurarne_wiebalck: yeah, it sounds like the same problem09:09
iurygregorymorning dtantsur09:12
*** jaganathan has quit IRC09:17
arne_wiebalckdtantsur: thx. We moved forward by setting the versions in the DB directly.09:23
dtantsursigh09:24
openstackgerritIlya Etingof proposed openstack/sushy-tools master: Refactor tests to pass without dependencies  https://review.openstack.org/61185909:29
openstackgerritDmitry Tantsur proposed openstack/ironic master: [WIP] Migration step to update objects to latest version  https://review.openstack.org/61318109:31
openstackgerritIlya Etingof proposed openstack/sushy-tools master: Make external dependencies optional  https://review.openstack.org/61185909:35
*** derekh has joined #openstack-ironic09:39
*** adrianc has quit IRC09:46
*** adrianc has joined #openstack-ironic09:51
*** jaganathan has joined #openstack-ironic10:25
openstackgerritIlya Etingof proposed openstack/sushy-tools master: Make external dependencies optional  https://review.openstack.org/61185910:30
*** stendulker has joined #openstack-ironic10:32
*** Emine has quit IRC10:38
*** Emine has joined #openstack-ironic10:39
*** moshele has joined #openstack-ironic11:04
*** stendulker has quit IRC11:25
*** rh-jelabarre has joined #openstack-ironic11:34
openstackgerritwanghui proposed openstack/ironic-specs master: Update tox minversion to 2.0  https://review.openstack.org/61328111:35
*** jesusaur has quit IRC11:43
*** jesusaur has joined #openstack-ironic11:46
*** dnuka has joined #openstack-ironic11:53
openstackgerritDigambar proposed openstack/ironic master: Fix CPU count returned by introspection in Ironic iDRAC driver  https://review.openstack.org/60998212:02
*** jistr_ is now known as jistr12:14
*** dnuka has quit IRC12:16
openstackgerritDmitry Tantsur proposed openstack/ironic-tempest-plugin master: Add API tests for protected nodes  https://review.openstack.org/61301412:20
openstackgerritDmitry Tantsur proposed openstack/ironic master: [WIP] Migration step to update objects to latest version  https://review.openstack.org/61318112:24
*** pcaruana has quit IRC12:26
*** S4ren has joined #openstack-ironic12:30
*** pcaruana has joined #openstack-ironic12:39
dtantsuretingof: hey, what's the current state of the sushy fix? do we have something final?12:42
etingofdtantsur, here is the latest take -- https://review.openstack.org/#/c/613062/12:43
patchbotpatch 613062 - sushy - Allow HTTP session info to be set on authenticator - 2 patch sets12:43
etingofdtantsur, used by this -- https://review.openstack.org/#/c/605408/12:44
patchbotpatch 605408 - ironic - Reuse Redfish session tokens - 11 patch sets12:44
dtantsuretingof: this is not backportable12:44
dtantsurcan we please start with caching Sushy objects and then dive into micro-optimizing it?12:45
dtantsurI'm quite worried about not fixing the problem for Queens and Rocky12:45
etingofdtantsur, could we get away with just disabling session auth there? -- https://review.openstack.org/#/c/603818/12:46
patchbotpatch 603818 - ironic - Add configurable Redfish client authentication - 15 patch sets12:46
dtantsuretingof: and get a feature regression? what about hardware that does not support basic auth?12:48
dtantsurmaybe the fix we need right now is simply to close any sessions we open. then think about caching and stuff.12:49
*** jcoufal has joined #openstack-ironic12:50
dtantsurTheJulia: I suggest we move on with the sushy release. figuring this ^^ out is definitely going to take more time.12:51
etingofdtantsur, so when somebody has issues with session exhaustion at their BMC, they can switch to basic auth via configuration option12:53
*** rloo has joined #openstack-ironic12:53
dtantsuretingof: on master - yes12:53
etingofdtantsur, if the BMC suffers from session exhaustion and it does not support basic auth, then that fix ^ won't help indeed12:54
* etingof is not sure if that qualifies as feature regression12:55
dtantsurwell, it's definitely a bug, and a quite serious one :)12:55
dtantsuralso, I'm not entirely sure that all hardware that requires session auth will support basic auth12:55
etingofbecause in the worst case the above fix does not help, but all the features are still there12:56
dtantsurthis seems not quite intuitive12:56
*** bnemec has joined #openstack-ironic12:56
dtantsuretingof: actually the commit message explicitly answers my question: https://github.com/openstack/sushy/commit/9610c06a179be10b19939b4285f23657b292524512:56
etingofon the other hand, using session for just one API call sort of undermines the purpose of sessions...12:56
dtantsurright, yeah. we need to cache them, we just need to find a way to do it without adding features to the library12:57
etingofdtantsur, that's where we have started more or less - messing with private attributes of sushy authenticator (IIRC)12:58
dtantsursigh12:58
etingofanyway, let's do immediate close12:58
etingofthat should not require sushy changes12:58
dtantsur... or we could cache Sushy objects....12:58
etingofI am afraid of this doubly independent caching we'd create12:59
etingofe.g. we cache auth, urllib caches connections12:59
dtantsurthis sounds much less problematic to me than other solutions13:00
dtantsuron the other hand, closing sessions + caching sounds like a can of worms13:01
etingofthat's what I fair if we talk about caching the entire sushy connection13:02
dtantsurif we talk about caching anything13:03
dtantsurwe need a way to make sure that a session we close is not used by anything right now13:03
etingofalso, we cache request objects that might change pooling behavior in the future potentially colliding with our higher-level caching logic13:04
dtantsurin what way?13:05
etingofI am thinking of leaks here13:06
etingofsince we keep a ref to the whole sushy->requests->urllib->socket chain13:06
etingofwe manage it at the sushy level while requests manage its own cache of the rest independently13:07
etingofperhaps the worst thing could be that requests frees up its stuff while we still hold everything in memory13:08
dtantsuretingof: if requests frees up stuff, it's responsible for recreating it or whatever13:09
dtantsurencapsulations \o/13:09
dtantsur:)13:09
rlooGood morning ironickers, etingof, dtantsur13:10
hkominosmorning all. I need a quick top regarding grub. I have setup a grub.cfg file as defined here. https://docs.openstack.org/ironic/rocky/install/configure-pxe.html. However when the boards boots I am forwarded to a grub menu which does not define any kernel/initrd to boot from. It is basically a text copy of what I have in my grub.cfg13:10
etingofrloo, o/13:10
dtantsurmorning rloo13:10
dtantsurrloo: did you few updates to your patch, I hope you do not mind13:10
rloomorning hkominos13:10
rloodtantsur: i don't mind at all. wanted to you if you wanted to take over :)13:10
hkominosrloo: Good morning to you13:10
rloodtantsur: i hate writing unit tests etc. but i can continue. probably should.13:11
etingofdtantsur, requests will recreate stuff, trouble is that if we still hold something in memory (sockets, whatever) we may exhaust something (memory, sockets)13:11
rloodtantsur: anyway, wanted to give you a chance to have some glory too, if you wanted ;)13:11
dtantsurrloo: I can take it over, but I may not have enough time today (meetings! excitement!!)13:11
hkominos(i.e. configfile /var/lib/tftpboot/$net_default_mac.conf this is what I see in my grub menu )13:11
dtantsuretingof: this sounds like the smallest of our problems :)13:11
rloodtantsur: ok, we can tag team. i'll see how much i can get done today, then will pass to you.13:11
dtantsur++ great plan13:12
etingofdtantsur, I am not sure, if we care about the scale...13:12
dtantsuretingof: if you care about this issue, you have to measure it. my gut feeling is that it's minor compared to other things we do.13:13
dtantsurnow, failing redfish deployment because of session problems IS a real problem13:13
etingofdtantsur, so to summarize: my gut feeling is that by caching sushy sessions we introduce a risk of resource leaks sooner or later13:14
etingofdtantsur, but I can change the patch into caching the entire sushy session to get this done13:14
dtantsurI'd like a better idea, but other ideas don't look necessary better13:15
*** jistr is now known as jistr|call13:29
*** rnoriega has quit IRC13:31
*** d0ugal has quit IRC13:33
*** d0ugal has joined #openstack-ironic13:34
TheJuliathe database migration fix basically lgtm, just needs a release note I think13:36
*** pcaruana has quit IRC13:37
dtantsuryeah, and unit tests13:37
TheJuliayeah13:37
* TheJulia is still sipping coffee13:37
dtantsuralso good morning13:37
etingofenjoy your coffee, TheJulia! o/13:37
*** rnoriega has joined #openstack-ironic13:38
etingofactually, if we cache the entire sushy sessions I am not sure we could invalidate the sessions we'd like to close... because sushy does not do that by itself -- https://github.com/openstack/sushy/blob/master/sushy/auth.py#L19113:40
dtantsuretingof: I really don't know what to do with session invalidation.. if we cache anything, it will be racy.13:40
dtantsurdon't we have something like Sushy.close that also calls self._auth.close?13:41
etingofdtantsur, the thing is that we can't easily tell BMC - "please, forget this session"13:41
etingofdtantsur, I do not see that .close() call -- https://github.com/openstack/sushy/search?q=close&unscoped_q=close13:42
dtantsurmeh13:42
etingofdtantsur, still, if we start reusing sessions, we would be less aggressive wrt requesting new sessions13:43
dtantsurwe should reuse sessions, yes13:43
etingofbut never close at BMC atm13:43
dtantsurso, we cannot even fix it in a simple way by closing sushy objects right after opening? this is.. bad..13:44
dtantsursomething we can fix is close auth/connector in __del__ (yes, I know, __del__..)13:44
etingofconductor restart may cause more pressure to BMC sessions13:44
TheJuliawow https://review.openstack.org/#/c/613169/ got really far all things considered13:44
patchbotpatch 613169 - ironic - WIP: ipv6 ipxe BFV?!? - 2 patch sets13:44
TheJuliaAm I crazy?13:44
etingofdtantsur, do you mean introducing sushy.close() calling auth.close() ?13:45
dtantsuretingof: we probably need to do it, but again - not backportable13:45
etingofdtantsur, yes, so the only option is to reuse sessions but never close them at BMC for the time being13:46
dtantsurright13:46
etingof...or mess with sushy internals13:46
etingofpulling .close() from there13:47
*** jezogwza has joined #openstack-ironic13:47
etingofI mean at runtime13:47
*** mjturek has joined #openstack-ironic13:48
*** jcoufal has quit IRC13:48
*** baha has joined #openstack-ironic13:49
*** d0ugal has quit IRC13:49
etingof...I wonder if Sushy.__del__ would work given the cyclic connector<->authenticator reference it holds...13:50
dtantsurdepends on how we implement it. we can break the cycle13:51
*** jcoufal has joined #openstack-ironic13:51
*** homegrown has left #openstack-ironic13:51
*** rpittau has quit IRC13:52
etingofso the call flow would be like this: GC calls Sushy.__del__ which calls auth.close() which runs HTTP DELETE against BMC. if that all goes well we could unset connector from auth (in a hacky way atm)13:55
etingoffeels shaky because it's __del__13:56
dtantsuryeah13:56
*** skazi has joined #openstack-ironic13:56
*** skazi has quit IRC13:56
dtantsuron the other hand, leaving dangling sessions after Sushy deletion is also bad13:56
dtantsurso we need __del__ and close(), __del__ calling close()13:56
dtantsurthen in ironic we rely on __del__ for the backportable part13:56
dtantsurthen we can switch to close() for master13:56
etingofso is that what I've just described? ^13:57
dtantsurI think yes13:57
etingofthen we need to wait with sushy release a bit ;)13:58
*** mjturek has quit IRC13:59
dtantsurokay, asked them to wait till early next week14:00
etingofack, thanks!14:01
*** mjturek has joined #openstack-ironic14:07
mjturekgm ironic14:08
TheJuliagood morning mjturek14:19
*** moshele has quit IRC14:29
etingofdebugging unittest mocks deprive my mind of happiness14:31
TheJuliaThere are mocks that do the reverse at times :)14:33
TheJuliajust please don't mock 3+ layers down... :(14:33
TheJulialots of our pxe tests are that way14:33
etingofthese unhappy mocks are exactly 3-store mocks14:34
dtantsurheh14:35
* etingof now seeks happiness in coffee14:35
-openstackstatus- NOTICE: Zuul and Nodepool services are being restarted to migrate them to a new Zookeeper cluster. THis brings us an HA database running on newer servers.14:39
TheJuliawheeeee14:41
*** munimeha1 has joined #openstack-ironic14:42
openstackgerritDmitry Tantsur proposed openstack/ironic master: Support for protecting nodes from undeploying and rebuilding  https://review.openstack.org/61166214:43
openstackgerritDmitry Tantsur proposed openstack/ironic-tempest-plugin master: Add API tests for protected nodes  https://review.openstack.org/61301414:44
*** tiendc has quit IRC14:45
*** jistr|call is now known as jistr14:50
*** jtomasek has quit IRC15:01
*** hkominos has quit IRC15:11
-openstackstatus- NOTICE: The Zuul and Nodepool database transition is complete. Changes updated during the Zuul outage may need to be rechecked.15:31
*** jaganathan has quit IRC15:32
*** mjturek has quit IRC15:34
*** mjturek has joined #openstack-ironic15:40
NobodyCamGood Morning Ironic'ers15:47
*** derekh has quit IRC15:58
*** e0ne has quit IRC15:59
openstackgerritIlya Etingof proposed openstack/sushy-tools master: Add memoization to emulator connections  https://review.openstack.org/61275816:14
*** gyee has joined #openstack-ironic16:16
openstackgerritJulia Kreger proposed openstack/ironic master: WIP: ipv6 ipxe BFV?!?  https://review.openstack.org/61316916:18
* TheJulia crosses her fingers16:18
*** emine__ has joined #openstack-ironic16:21
*** dtantsur is now known as dtantsur|afk16:22
dtantsur|afksee you16:22
NobodyCamnight dtantsur|afk16:24
*** Emine has quit IRC16:24
*** jhesketh has joined #openstack-ironic16:25
TheJuliaxclarity logs \o/ http://lxca-logs.electronicjungle.net/62/611662/5/check/lenovo-xclarity-ironic/32a9fc0/16:26
TheJuliagoodnight dtantsur|afk16:26
openstackgerritJulia Kreger proposed openstack/ironic master: Cleanup of remaining pxe focused is_ipxe_enabled  https://review.openstack.org/61193316:27
*** jhesketh_ has quit IRC16:27
rpiosoGood afternoon16:27
*** S4ren has quit IRC16:29
TheJuliaI'm sure arne_wiebalck would be happy if https://review.openstack.org/#/c/592639 got reviews #NotSubtle16:31
patchbotpatch 592639 - ironic-python-agent - Fix multi-device behavior - 5 patch sets16:31
*** trown is now known as trown|lunch16:32
*** mjturek has quit IRC16:40
*** mjturek has joined #openstack-ironic16:49
*** adrianc_ has joined #openstack-ironic16:52
*** moshele has joined #openstack-ironic16:55
*** adrianc has quit IRC16:55
*** adrianc_ has quit IRC16:57
*** pcaruana has joined #openstack-ironic17:03
*** dougsz has quit IRC17:03
*** moshele has quit IRC17:15
openstackgerritBill Dodd proposed openstack/ironic master: WIP: Add BIOS interface to Redfish hardware type  https://review.openstack.org/58231417:26
*** skazi has joined #openstack-ironic17:35
*** mjturek has quit IRC17:36
*** trown|lunch is now known as trown17:54
*** betherly has joined #openstack-ironic17:55
*** mbeierl has quit IRC17:58
*** zzzeek_ has joined #openstack-ironic17:59
*** mbeierl has joined #openstack-ironic17:59
*** betherly has quit IRC17:59
*** serlex has quit IRC18:14
*** bnemec has quit IRC18:29
TheJuliabdodd: nice, I put some comments in18:35
bdoddThanks TheJulia - I'll check them out!18:36
TheJuliaIt is looking in really good shape, the only real issue that I see is a test seems to be missing18:36
*** munimeha1 has quit IRC18:37
*** moshele has joined #openstack-ironic18:38
TheJuliamoshele: Sorry, I've not had a chance to look at the spec again this week18:39
mosheleTheJulia:  no worries18:40
*** dsneddon__ has quit IRC18:43
*** dsneddon__ has joined #openstack-ironic18:43
*** dtrainor has quit IRC18:45
*** dtrainor has joined #openstack-ironic18:45
bdoddTheJulia - for the reboot requested test, the logic is that we only set the 'post_config_reboot_requested' flag in the driver_internal_info after we issue the reboot. The reboot is done from post_configuration() and the flag is set in _set_reboot_requested().18:47
bdoddIs the concern that the flag could be left over or set via some other means?18:48
TheJuliano, the concern is that the second time it is called, it is essentially an independent call so we should have a test that validates that behavior is what we expect18:48
bdoddSo like a call to the service (via sushy) that would verify the reboot had been requested?18:50
TheJuliaNo, I mean like a test that sets the stage as if the node has rebooted already18:51
TheJuliaand apply_configuration is called again18:51
*** moshele has quit IRC18:57
bdoddTheJulia - Are you pointing out the lack of a unit test that tests this stage? (which is definitely missing.) Sorry if I'm being especially dense.18:58
TheJuliai am19:01
TheJuliano worries, we're just using different words19:01
bdoddAh, good. Yes, I was aware of that will fix it. :-)19:02
TheJuliaawesome19:09
*** jcoufal_ has joined #openstack-ironic19:26
*** jcoufal has quit IRC19:27
*** irclogbot_1 has joined #openstack-ironic20:01
*** moshele has joined #openstack-ironic20:01
*** moshele has quit IRC20:04
*** jcoufal_ has quit IRC20:19
*** jcoufal has joined #openstack-ironic20:21
*** irclogbot_1 has quit IRC20:22
*** jcoufal has quit IRC20:54
*** betherly has joined #openstack-ironic20:56
*** munimeha1 has joined #openstack-ironic20:58
*** betherly has quit IRC21:01
openstackgerritJulia Kreger proposed openstack/ironic-python-agent master: zuul-ify primary IPA jobs  https://review.openstack.org/61342621:14
*** irclogbot_1 has joined #openstack-ironic21:15
*** trown is now known as trown|outtypewww21:15
*** betherly has joined #openstack-ironic21:16
openstackgerritJulia Kreger proposed openstack/ironic-python-agent master: zuul-ify primary IPA jobs  https://review.openstack.org/61342621:17
*** betherly has quit IRC21:21
*** betherly has joined #openstack-ironic21:37
*** dsneddon__ has quit IRC21:39
*** dsneddon__ has joined #openstack-ironic21:40
*** betherly has quit IRC21:42
openstackgerritRuby Loo proposed openstack/ironic master: Migration step to update objects to latest version  https://review.openstack.org/61318121:49
rloodtantsur|afk: ^^ that's my contribution for today. unit tests. it is based on the old backfill code so should work :).but didn't have time to put together a test env to try it out.21:52
*** betherly has joined #openstack-ironic21:58
*** betherly has quit IRC22:02
*** betherly has joined #openstack-ironic22:18
*** betherly has quit IRC22:23
*** baha has quit IRC22:24
*** emine__ has quit IRC22:24
*** ElCoyote_ has quit IRC22:34
*** dtrainor has quit IRC22:35
*** betherly has joined #openstack-ironic22:49
*** betherly has quit IRC22:53
*** munimeha1 has quit IRC23:00
*** betherly has joined #openstack-ironic23:09
*** betherly has quit IRC23:14
openstackgerritJulia Kreger proposed openstack/ironic-python-agent master: zuul-ify primary IPA jobs  https://review.openstack.org/61342623:43
*** gyee has quit IRC23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!