Friday, 2018-07-20

*** sdake has quit IRC00:01
*** vkmc has quit IRC00:02
*** rcernin_ has quit IRC00:02
*** sdake has joined #openstack-ironic00:06
*** sdake has quit IRC00:06
*** sdake has joined #openstack-ironic00:06
*** jhesketh has quit IRC00:06
*** vkmc has joined #openstack-ironic00:06
*** lbragstad has quit IRC00:06
*** lucasagomes has quit IRC00:06
*** rnoriega has quit IRC00:06
*** lhinds has quit IRC00:06
*** amotoki has quit IRC00:06
*** gmann has quit IRC00:06
*** jovial[m] has quit IRC00:10
*** retr0h has quit IRC00:11
*** csmart has quit IRC00:11
*** mkrai has quit IRC00:11
*** jhesketh has joined #openstack-ironic00:15
*** rcernin_ has joined #openstack-ironic00:15
*** lbragstad has joined #openstack-ironic00:15
*** lucasagomes has joined #openstack-ironic00:15
*** rnoriega has joined #openstack-ironic00:15
*** lhinds has joined #openstack-ironic00:15
*** amotoki has joined #openstack-ironic00:15
*** gmann has joined #openstack-ironic00:15
*** threestrands has joined #openstack-ironic00:23
*** threestrands has quit IRC00:23
*** threestrands has joined #openstack-ironic00:23
*** livelace2 has quit IRC00:28
*** livelace2 has joined #openstack-ironic00:28
*** rcernin_ has quit IRC00:43
*** rcernin has joined #openstack-ironic00:44
*** phuongnh has joined #openstack-ironic00:45
*** jovial[m] has joined #openstack-ironic00:46
*** exodusftw has quit IRC00:46
*** exodusftw has joined #openstack-ironic00:52
*** r-daneel has quit IRC00:57
*** r-daneel has joined #openstack-ironic01:00
*** harlowja has quit IRC01:09
*** liuzz has joined #openstack-ironic01:14
*** ijw has quit IRC01:18
*** r-daneel has quit IRC01:18
*** ijw has joined #openstack-ironic01:20
*** phuongnh has quit IRC01:30
*** phuongnh has joined #openstack-ironic01:31
*** gcb has quit IRC01:34
*** phuongnh has quit IRC01:39
*** phuongnh has joined #openstack-ironic01:40
*** tiendc has joined #openstack-ironic02:02
*** spartacloud has joined #openstack-ironic02:11
*** spartacloud has quit IRC02:12
*** retr0h[m] has joined #openstack-ironic02:12
*** csmart[m] has joined #openstack-ironic02:12
*** madhuri[m] has joined #openstack-ironic02:12
*** spartacloud has joined #openstack-ironic02:23
*** spartacloud has quit IRC02:27
*** spartacloud has joined #openstack-ironic02:39
*** links has joined #openstack-ironic02:40
*** spartacloud has quit IRC02:43
*** lifeless has quit IRC02:51
*** threestrands has quit IRC02:54
*** lifeless has joined #openstack-ironic02:58
*** rh-jelabarre has quit IRC03:03
*** rh-jelabarre has joined #openstack-ironic03:06
*** ijw has quit IRC03:09
*** threestrands has joined #openstack-ironic03:10
*** threestrands has quit IRC03:10
*** threestrands has joined #openstack-ironic03:10
openstackgerritHa Manh Dong proposed openstack/sushy-tools master: Switch to use stestr for unit test  https://review.openstack.org/58383203:24
*** trungnv has quit IRC04:14
*** ijw has joined #openstack-ironic04:24
*** ijw has quit IRC04:28
*** harlowja has joined #openstack-ironic04:30
*** jaganathan has joined #openstack-ironic04:37
*** threestrands has quit IRC04:40
*** jiapei has joined #openstack-ironic04:44
*** mbeierl has quit IRC04:49
*** harlowja has quit IRC04:52
*** mbeierl has joined #openstack-ironic04:52
*** pcaruana has joined #openstack-ironic05:09
*** spartacloud has joined #openstack-ironic05:28
*** spartacloud has quit IRC05:33
*** dmellado has quit IRC05:48
openstackgerritKaifeng Wang proposed openstack/ironic-inspector master: Supports database as introspection data storage: processing  https://review.openstack.org/51455205:52
*** mjura has joined #openstack-ironic06:02
*** skazi has joined #openstack-ironic06:15
*** AlexeyAbashkin has joined #openstack-ironic06:15
*** ijw has joined #openstack-ironic06:25
*** hjensas has joined #openstack-ironic06:25
*** hjensas has quit IRC06:25
*** hjensas has joined #openstack-ironic06:25
*** dmellado has joined #openstack-ironic06:29
*** ijw has quit IRC06:30
*** tiendc_ has joined #openstack-ironic06:30
*** tiendc has quit IRC06:33
*** AlexeyAbashkin has quit IRC06:41
*** zhangfei has joined #openstack-ironic06:48
*** hoonetorg has quit IRC06:54
*** AlexeyAbashkin has joined #openstack-ironic06:58
*** Alexey_Abashkin has joined #openstack-ironic07:01
*** tesseract has joined #openstack-ironic07:02
*** AlexeyAbashkin has quit IRC07:02
*** Alexey_Abashkin is now known as AlexeyAbashkin07:02
*** rcernin has quit IRC07:02
*** jtomasek has joined #openstack-ironic07:04
*** tiendc has joined #openstack-ironic07:05
*** tiendc_ has quit IRC07:07
*** hoonetorg has joined #openstack-ironic07:11
*** e0ne has joined #openstack-ironic07:13
*** hjensas has quit IRC07:22
*** jiapei has quit IRC07:24
*** hamzy_ has joined #openstack-ironic07:31
*** hamzy has quit IRC07:31
*** vkmc has quit IRC07:32
*** tiendc has quit IRC07:32
*** gkadam has joined #openstack-ironic07:37
openstackgerritKaifeng Wang proposed openstack/python-ironicclient master: Query node by instance uuid  https://review.openstack.org/58426307:39
*** spartacloud has joined #openstack-ironic07:44
*** spartacloud has quit IRC07:49
*** hamzy_ has quit IRC07:50
*** alexpilotti has quit IRC07:54
*** pmannidi has quit IRC07:56
*** spartacloud has joined #openstack-ironic07:58
*** jrcloud has joined #openstack-ironic08:02
*** spartacloud has quit IRC08:03
*** tssurya has joined #openstack-ironic08:05
*** dsneddon has quit IRC08:06
*** jrcloud has quit IRC08:06
*** dsneddon has joined #openstack-ironic08:08
*** hkominos_ has joined #openstack-ironic08:09
*** ajya has joined #openstack-ironic08:10
*** mgoddard has joined #openstack-ironic08:12
*** dougsz has joined #openstack-ironic08:13
*** zhangfei has quit IRC08:14
*** athomas has joined #openstack-ironic08:17
openstackgerritKaifeng Wang proposed openstack/ironic-inspector master: [TEST] unit test failure due to oslo.policy upgrade  https://review.openstack.org/58427308:17
openstackgerritKaifeng Wang proposed openstack/ironic-inspector master: Stop introspection if set boot device failed  https://review.openstack.org/58351908:20
*** zhangfei has joined #openstack-ironic08:26
*** hjensas has joined #openstack-ironic08:50
*** hjensas has quit IRC08:50
*** hjensas has joined #openstack-ironic08:50
*** liuzz has quit IRC09:08
*** liuzz has joined #openstack-ironic09:11
*** liuzz has quit IRC09:13
*** liuzz has joined #openstack-ironic09:13
*** mgoddard has quit IRC09:17
*** alexpilotti has joined #openstack-ironic09:18
*** skazi has quit IRC09:24
*** links has quit IRC09:29
*** hamzy_ has joined #openstack-ironic09:31
openstackgerritKaifeng Wang proposed openstack/ironic-inspector master: Supports database as introspection data storage: processing  https://review.openstack.org/51455209:35
*** csmart[m] has quit IRC09:42
*** madhuri[m] has quit IRC09:42
*** retr0h[m] has quit IRC09:43
*** jovial[m] has quit IRC09:43
*** prasen has joined #openstack-ironic09:48
*** vkmc has joined #openstack-ironic09:49
*** jovial[m] has joined #openstack-ironic09:55
*** liuzz has quit IRC09:56
*** hjensas has quit IRC10:00
*** mgoddard has joined #openstack-ironic10:04
*** AlexeyAbashkin has quit IRC10:08
*** skazi has joined #openstack-ironic10:09
*** AlexeyAbashkin has joined #openstack-ironic10:11
*** spartacloud has joined #openstack-ironic10:17
*** AlexeyAbashkin has quit IRC10:18
*** spartacloud has quit IRC10:22
*** AlexeyAbashkin has joined #openstack-ironic10:31
*** dtantsur|afk is now known as dtantsur10:31
dtantsurmorning ironic10:31
*** phuongnh has quit IRC10:32
etingofgood Friday morning everyone o/10:35
*** retr0h[m] has joined #openstack-ironic11:01
*** madhuri[m] has joined #openstack-ironic11:01
*** csmart[m] has joined #openstack-ironic11:01
*** hjensas has joined #openstack-ironic11:04
*** hjensas has quit IRC11:04
*** hjensas has joined #openstack-ironic11:04
*** ijw has joined #openstack-ironic11:24
*** ijw has quit IRC11:29
*** jaypipes is now known as leakypipes11:32
*** links has joined #openstack-ironic11:32
jrollmorning, happy friday \o/11:37
dtantsurhey jroll11:38
jroll\o dtantsur11:38
dtantsurresponded on the conductor group patch11:41
dtantsurnow I cannot get why you need two hash rings..11:42
jrollright, I will go back over it today11:44
jrollwe use undelimited so that rolling upgrade is smooth11:44
jrolland we agreed that storing the delimited now will allow us a smooth switch to delimited in stein11:44
jrollbut I'm not sure that is true11:45
dtantsuryeah, I don't see how it's true, given that, as you rightfully notice, a hash ring is local11:46
*** kiseok7 has quit IRC11:48
dtantsursrsly, we never implemented this https://review.openstack.org/#/c/584263/ ? O__o11:53
patchbotpatch 584263 - python-ironicclient - Query node by instance uuid11:53
TheJuliaGood morning everyone11:53
TheJuliadtantsur: I believe the nova virt driver code does it via get get nodes with an argument11:54
TheJuliaIt also has the weird abstraction that it uses11:54
dtantsurmorning TheJulia11:54
TheJuliadtantsur: so I got an aarch64 vm running last night (was super painful)11:55
dtantsurTheJulia: \o/ awesome11:56
dtantsurI suggest we add this ironicclient patch to the upcoming release11:56
TheJuliaso I think it wouldn't be too difficult to build testing around both uefi paths ... to assist in ripping out elilo support.11:56
TheJuliadtantsur: agreed11:56
jrollFYI we already have SDK code for it https://github.com/openstack/python-ironicclient/blob/master/ironicclient/v1/node.py#L32611:56
jrollnot sure if we want to use that instead of adding an argument to list()11:57
dtantsurjroll: mmm, right. but we still need to expose it in CLI11:57
jrollhttps://github.com/openstack/python-ironicclient/blob/master/ironicclient/osc/v1/baremetal_node.py#L122411:58
jrolllooks like it's there11:58
jrollopenstack baremetal node show --instance foo11:58
dtantsuroh, and we do, yeah11:58
jroll:)11:58
* dtantsur changes his vote11:58
jrollI had the same O__o reaction, I knew it couldn't be true :P11:58
dtantsurgood catch jroll :)11:59
jroll:)12:00
*** zhangfei has quit IRC12:00
*** jaganathan has quit IRC12:01
TheJuliadtantsur: w/r/t the new option I proposed to add for using macs for uefi. It has a +2 right now, we could land it and deprecate/kill it next cycle. I don't think there is going to be any way to get all of the uefi stuff sorted before we cut this cycle's release, and switching to mac default via removing elilo is likely moderately logically and then write out an ip address for compatibility if neutron is turned on12:01
*** arnewiebalck__ has joined #openstack-ironic12:02
dtantsurTheJulia: is it possible to detect if we're using elilo or not?12:03
TheJuliadtantsur: w/r/t to the two hash rings are needed so api nodes can understand how to handle the transition to delimited once conductors start getting upgraded. Then again, we might be seriously overthinking it, but changing the structure with adding the delimiter is breaking to api->conductor communication ability by for topic selection.12:04
*** trown|outtypewww is now known as trown12:04
dtantsurTheJulia: something like https://github.com/openstack/ironic/blob/master/ironic/common/pxe_utils.py#L23912:04
TheJuliadtantsur: I don't think so, but I've also not even tried to find elilo packages for a long time12:04
dtantsurwhich is one place we're already doing that12:05
TheJuliahttps://www.rpmfind.net/linux/rpm2html/search.php?query=elilo lol12:05
dtantsurTheJulia: wrt the hash rings, I don't think this is actually true12:05
dtantsuras Jim reminded me, any API instance is building its own hash ring, with or without delimiters12:05
TheJuliaI guess it all depends on how smooth we want to make the transition.12:05
dtantsurso I don't quite understand why we care12:05
TheJuliaindeed, and addresses those messages, so if we want things to match up correctly it seems necessary. If we don't care as much *shrugs*12:06
dtantsurI seem to be missing some bit part of it. if any service has its own hash ring copy, why do we care about delimiters?12:06
jrollthere's two goals: keep them in sync as much as possible, and reduce take_over as much as possible12:06
jrollusing the undelimited for this upgrades means the hash ring does not change during the upgrade, which helps with both of these12:06
jroll(because no nodes or conductors have a group yet)12:07
TheJuliaahh, yeah, rloo and us also talked about take_over12:07
*** MattMan has quit IRC12:07
*** prasen has quit IRC12:07
*** MattMan has joined #openstack-ironic12:07
dtantsuroh. anyway, the current code has to be fixed, since one of the rings is never used. or is it?12:08
jrollnow for the upgrade to stein, we do want to switch to using a delimited hash ring, because it reduces the opportunity for conflicts (example group "ip" and hw type "mi" conflicts with group "" and hw type "ipmi")12:08
jrollwe somehow agreed that storing both now helps us do that upgrade more smoothly. but now I really can't remember how.12:09
dtantsuryeah, that's my point. one of the rings is generated (in-memory) and immediately discarded12:10
dtantsuronly the undelimited ring seems used12:10
dtantsurso it does not seem to me that we can avoid take_overs on upgrade to Stein when the rings start being delimited..12:10
jrollright12:10
jrollin which case we should just do it now12:10
dtantsuror even: maybe take_over's are not a problem at all12:11
dtantsuragain, each conductor detects if it manages a host12:11
dtantsurmmm, interesting, I wonder if two conductors (old and new) can start fighting for a host12:12
* dtantsur starts getting headache again12:12
jrollright, that's the concern, the time during rolling restarts to unpin it12:12
dtantsurmmm, okay, no, at each point in time both undelimited and delimited hash rings are identical12:13
dtantsurbecause the database does not know about delimiters...12:14
jrolloh, the hash ring itself only deals with conductor names and node uuids12:15
dtantsurwait the database does know about delimiters12:15
dtantsurjroll: isn't https://review.openstack.org/#/c/579684/13/ironic/db/sqlalchemy/api.py@858 breaking the driver API?12:15
patchbotpatch 579684 - ironic - Use conductor group for hash ring calculations12:15
jrolloof12:16
dtantsurjroll: we're using it in https://github.com/openstack/ironic/blob/master/ironic/api/controllers/v1/driver.py#L38512:16
jrollyeah, I realized that when you said it12:16
dtantsurwtf don't we have tempest tests for these endpoints? :)12:16
jrollnot ones where the conductors are configured to use a group :)12:17
dtantsurriiiiight12:17
jrollsigh, lots of problems here12:17
dtantsurwell, solvable problems12:18
jrollyeah12:18
jrollI'd rather be doing something else though :D12:18
dtantsurheh12:18
dtantsurimagine my mood when fixing unit tests for the driver composition reform? :D12:19
dtantsuryou're at least developing new cool features12:19
jrollyeah, I bet12:19
*** early has quit IRC12:19
dtantsura small request: let's concentrate all the delimiter logic in one place. I nearly missed this dbapi change.12:20
openstackgerritvinay kumar muddu proposed openstack/ironic master: iLO BIOS interface implementation  https://review.openstack.org/57680312:20
jrollyeah, I agree with that12:20
dtantsur"Temperatures will continue to rise and by the end of the week could reach upwards of 35 to 37 degrees." dear weather, why the hell you did not do it when I was on vacation? >_<12:21
jrollouch12:21
dtantsurif next week I have bad internet, you know that I work-from-sea :D12:22
jrollsounds great12:22
*** early has joined #openstack-ironic12:23
*** dtantsur is now known as dtantsur|brb12:23
TheJuliawork-from-sea sounds... interesting12:23
TheJuliawheeeee.... all my weekend plans just imploded12:25
jroll:/12:26
jrollare they suddenly the same as my weekend plans?12:26
jroll"do more work on the house"12:26
TheJuliawell, I guess the one on wheels needs re-packing12:26
*** skazi has quit IRC12:26
jrollah12:27
*** AlexeyAbashkin has quit IRC12:31
* TheJulia finishes first coffee12:31
*** AlexeyAbashkin has joined #openstack-ironic12:35
*** dprince has joined #openstack-ironic12:41
*** d0ugal has quit IRC12:45
*** d0ugal has joined #openstack-ironic12:45
*** d0ugal has quit IRC12:45
*** d0ugal has joined #openstack-ironic12:45
*** edleafe is now known as figleaf12:46
hkominos_Hi all. Can I get some quick input on this error ? https://paste.fedoraproject.org/paste/VktDmaS5FssPdhsPrU30Bg/  I think it implies that a package is missing on my images but I would like to verify12:47
*** arnewiebalck__ has quit IRC12:53
*** rpioso|afk is now known as rpioso12:55
rpiosoGood morning12:55
*** tssurya has quit IRC12:55
*** AlexeyAbashkin has quit IRC12:55
*** AlexeyAbashkin has joined #openstack-ironic12:56
*** d0ugal has quit IRC12:57
TheJuliagood morning rpioso13:00
openstackgerritKaifeng Wang proposed openstack/ironic-inspector master: Fix gate failure due to recent oslo.policy release  https://review.openstack.org/58427313:00
rpiosoTheJulia: o/13:00
openstackgerritJulia Kreger proposed openstack/bifrost master: WIP Arm64  https://review.openstack.org/58436313:01
*** ijw has joined #openstack-ironic13:01
TheJuliahkominos_: that... seems... very... broken13:02
TheJuliaheh you'r using an aarch64 machine, how did you build the image? Asking because I'm going to be approaching the same problem very soon13:03
*** arnewiebalck__ has joined #openstack-ironic13:04
TheJuliahkominos_: do you know if the arm64 grub modules were installed?13:04
*** jcoufal has joined #openstack-ironic13:05
*** ijw has quit IRC13:06
*** d0ugal has joined #openstack-ironic13:09
* TheJulia wonders if people are selling arm gear with BMCs at a reasoable price these days13:13
hkominos_TheJulia: Hi. So the good news is that It kinda works on aarch64. At least my virtual deploy worked fine and I use DIB tobuild the images13:16
hkominos_TheJulia: When I use whole-images the system boots fine.13:16
hkominos_TheJulia: The point is that when I use DIB for a whole disk image I define myself hwo the disk will look like. But If i understand correctly when you provide ironic with rootfs,kernel, and ramdisk, it is ironic which will create the partitions etc..13:18
*** rloo has joined #openstack-ironic13:18
hkominos_TheJulia: So regarding your question: do you know if the arm64 grub modules were installed? -->the question is where ? should grub2 existin in the ironic python agent ramdisk or exist within my rootfs ?13:19
TheJuliahkominos_: That it should as long as your not explicitly set to netboot13:19
TheJuliahkominos_: both I believe13:19
hkominos_TheJulia: Ok I will try with some new images then. The issue is that grub2 package has a weird name on aarch64. and that is probably why he cannto find it13:20
hkominos_TheJulia: Should I assume that if the package is already present in the images, ironic will not complain???13:20
TheJuliahkominos_: it looks like part of it is present, the other part seems not13:21
TheJuliait is when ipa is calling grub-install...13:21
TheJuliaHmm.. here is one, we're lacking aarch64 templates...13:21
TheJuliafor network booting that is13:21
*** arnewiebalck__ has quit IRC13:22
hkominos_TheJulia: Ok. I will install grub2 manually inside the images and pray: thx for your help13:23
TheJuliahkominos_: just make sure you get the grub2 arm efi modules as well13:23
hkominos_TheJulia. I do. I have added this  "grub-efi-aarch64": "grub2-efi-aa64 grub2-efi-aa64-modules", in https://github.com/openstack/diskimage-builder/blob/master/diskimage_builder/elements/bootloader/pkg-map13:26
hkominos_TheJulia: there is this strange naming convention between aarch64 ,armv8, arm64 which I dont really understand.13:27
TheJulia:(13:27
hkominos_TheJulia: I forgot to ask. Does the error above mean that grub2 is missing from the target image or from the Ironic-python image?13:31
TheJuliaI think the modules from the ipa image, but you'll need it in the deployed image if anything needs to be changed after deployment13:31
MrBernsHey guys, what is the proper way to delete instances on bifrost? The openstack API commands arent avaialble as nova isnt really deployed13:36
jrollrloo: morning, sorry to bother you about it again. we can't figure out how keeping two hash rings helps with the upgrade to stein: http://eavesdrop.openstack.org/irclogs/%23openstack-ironic/%23openstack-ironic.2018-07-20.log.html#t2018-07-20T12:04:1613:37
rloogood morning jroll.13:37
*** rloo is now known as waterloo13:38
jrollMrBerns: openstack baremetal node undeploy $node_uuid13:38
jrollthough13:38
jrollMrBerns: there's extra things to take care of, I wonder if bifrost has an all in one command, let me look13:38
waterloolooking, i thought we had discussed later that we didn't need two rings... give me a few min13:38
MrBernsThanks jroll13:38
jrollwaterloo: thanks, wanted to check before I rewrite it all again :/13:39
*** zhangfei has joined #openstack-ironic13:39
TheJuliajroll: ironic deletes instance_info and all13:40
*** TheJulia is now known as needssleep13:40
* needssleep truly needs more sleep today13:40
jrollneedssleep: so what I said works?13:40
needssleepyes, it should just fine13:40
jrollcool, MrBerns ^^13:40
MrBernsThanks guys Ill give that command a go13:41
waterloojroll: after reading that exchange, the short answer is 'i don't know for sure'. When do we need to get that feature in by?13:43
MrBernsjroll, needssleep Thanks, that seems to have done the trick13:43
waterloofeature freeze is july 23. next week.13:43
jrollwaterloo: well, hopefully next week to make feature freeze and client freeze13:43
jrollya13:43
jrollor we could revert the half that landed I guess13:43
needssleepOr we could roll forward13:44
waterloohmm. i feel like i would need to spend 2-3 hours to be sure13:44
jrollyeah I guess13:44
*** jcoufal has quit IRC13:44
needssleepwe've ripped stuff back out and it quite literally shot us in the foot for a long time13:44
* needssleep things tags patches13:44
jrollmark the config with something about no effect13:44
waterloohas it all landed?13:44
jrollthe db/objects code landed, and the conductor config13:44
needssleepjroll: that is reasonable13:44
needssleepbut I'd still like to get minimally viable in if at all possible since we could continue to roll forward13:45
*** jcoufal has joined #openstack-ironic13:45
jrollhonestly, we can probably just go straight to the delimited hash ring and nobody would notice any effects that has on rolling upgrades13:45
jroll(with an rpc version bump)13:45
waterloojust thinking. i wasn't planning on looking at that today. can i look at it on Monday, or i guess you'd prefer i think about it today?13:46
needssleepjroll: the only way I think it could be noticed... is if they had something already weirdly partitioned in an unsupported way13:46
jrollwaterloo: today would be better but it's your call13:46
waterloojroll: i'll look at it later today, ok. but as-is, i don't think we need to roll back. only the config is exposed.13:46
*** mbeierl has quit IRC13:46
jrollneedssleep: idk how one would do that13:46
*** mjura has quit IRC13:46
*** mjura has joined #openstack-ironic13:47
needssleepjroll: hacked driver names13:47
jrolleh, that should be fine13:48
needssleepnot if pools of conductors exist without the other drivers13:49
needssleephacked drivers that is13:49
needssleepbut we're deep in theoretical land13:50
needssleeplets focus on the 90% practical13:50
*** dprince has quit IRC13:54
*** dprince has joined #openstack-ironic13:55
*** ijw has joined #openstack-ironic13:59
*** mjturek has joined #openstack-ironic14:03
*** dansmith is now known as superdan14:04
*** ijw has quit IRC14:04
*** tssurya has joined #openstack-ironic14:04
*** arnewiebalck__ has joined #openstack-ironic14:05
*** markguz has joined #openstack-ironic14:05
*** markguz_ has joined #openstack-ironic14:09
*** mjturek has quit IRC14:10
*** markguz has quit IRC14:12
*** spartacloud has joined #openstack-ironic14:13
*** jcoufal has quit IRC14:14
*** mjturek has joined #openstack-ironic14:14
*** jcoufal has joined #openstack-ironic14:14
*** dtantsur|brb is now known as dtantsur14:15
*** spartacloud has quit IRC14:18
dtantsur++ I think we can land it14:18
arnewiebalck__Hi Ironic, is there any advise on how to deploy the inspector wrt to the api or the conductor? What we did is to create “all-in-one” controller nodes (api, conductor, inspector all in one VM) and have multiple of these. The idea was to easily scale horizontally and add or remove these building blocks when needed. Seems to work fine so far (~1200 nodes). However: we just noticed that the inspector does a14:19
arnewiebalck__regular clean up where it gets all nodes … and this is now done on all controllers :-) No big deal, just unnecessary … and it may cause some issue as we’ll need to scale this up by a factor of ~10, so I was wondering if there were some best practices, for instance if the inspector should be kept separate.14:19
NobodyCamGood Morning Ironic'ers ...14:20
NobodyCamand ..14:20
NobodyCamofc ..14:21
NobodyCamTGIF14:21
*** ijw has joined #openstack-ironic14:24
dtantsurhey hey NobodyCam, TGIF indeed14:24
NobodyCam:) oh ya :) I'm ready for the weekend too.. New pupper has been keeping everyone awake all night :p14:25
NobodyCamHow about you any plans?14:25
dtantsurarnewiebalck__: are you using inspector in HA mode? if not, you need only one instance. otherwise there is not much point to get beyond 3 instances for HA.14:26
dtantsurNobodyCam: heh, puppers are like that :) dunno about plans, would go to the sea (it's going to be HOT here), but the budget is a bit tight.14:26
dtantsuryou?14:26
arnewiebalck__dtantsur: yes, and I agree14:27
arnewiebalck__dtantsur: this is mostly b/c it’s simple and allows to have symmetric controllers14:27
arnewiebalck__dtantsur: we have 5 controllers atm14:27
NobodyCamsleep! and some work around the house. :P not to hot here only about 95 F here this weekend14:28
arnewiebalck__dtantsur: 5 to handle power sync :)14:28
*** jtomasek has quit IRC14:28
*** ijw has quit IRC14:29
dtantsurNobodyCam: 35C? I consider it hot :)14:29
arnewiebalck__dtantsur: I noticed constant load and had a look where this was coming from14:29
arnewiebalck__dtantsur: (inspector clean_up)14:29
dtantsurarnewiebalck__: yeah, power sync is.. problematic14:29
dtantsurhmm, clean_up? it can be tuned if it causes problems14:29
arnewiebalck__dtantsur: yes, we can increase the interval14:29
arnewiebalck__it finishes in 53 secs14:30
*** tssurya has quit IRC14:30
arnewiebalck__takes a break of 7 :-D14:30
arnewiebalck__dtantsur: power sync is the root of all evil :)14:30
arnewiebalck__dtantsur: as mentioned, no big deal, just wondering if the whole setup is  … suboptimal14:31
*** efried is now known as fried_rice14:32
arnewiebalck__dtantsur: the load is coming from policy checking it seems14:32
arnewiebalck__dtantsur: for every node, oslo is checking the policy, after searching for it, so there are quite some stats involved14:33
*** ajya has quit IRC14:33
dtantsurouch14:34
dtantsurI guess it's inevitable since ironic and ironic-inspector are different services..14:34
arnewiebalck__yes14:34
dtantsurarnewiebalck__: if you don't delete nodes too often, you may lower the rate of inspector clean up14:35
*** mjturek has quit IRC14:35
dtantsuras to power sync.. needssleep is doing something to allow splitting periodic tasks into their own processes14:35
arnewiebalck__dtantsur: what does the clean up actually clan up?14:36
needssleepwell, more like be able to run multiple conductors, but yeah14:36
needssleepits a massive change though14:36
dtantsurarnewiebalck__: I think the primary goal is to remove nodes from internal cache that are removed from ironic14:36
arnewiebalck__dtantsur: ok14:36
dtantsurmaybe timeouts check are in the same task, I'll double-check14:36
arnewiebalck__needssleep: the issue with power sync was mostly that nova was hammering the iornic api for 1000+ instances14:38
needssleeparnewiebalck__: Interesting, I wasn't aware that was presenting as a problem for deployments14:38
arnewiebalck__needssleep: got better when we reduced the green threads pool and added 2 more controllers14:39
dtantsurarnewiebalck__: yeah, the same task also checks timeouts. but it should be quicker, it only accesses the DB14:39
arnewiebalck__needssleep: power sync?14:39
needssleeparnewiebalck__: well, power sync from the conductors standpoint while it is walking the individual nodes to check the status14:39
arnewiebalck__needssleep: no, I meant nova getting the power state14:40
needssleeparnewiebalck__: please tell me you've submitted a talk to the berlin summit on your performance issues and actions taken?14:40
needssleephmmmm14:40
arnewiebalck__needssleep: I have!14:40
needssleeparnewiebalck__: AWESOME!14:41
dtantsur\o/14:41
* dtantsur is ready for voting14:41
needssleeparnewiebalck__: I feel the need to look at the virt driver code.... but I'm kind of on this track of building us some better uefi testing14:41
needssleeparm based at that...14:41
arnewiebalck__needssleep: sounds great14:42
arnewiebalck__dtantsur: :) thx!14:43
needssleepsomebody please remind me if I don't look at that in the next couple of weeks, since we won't be able to really improve that performance until next cycle14:44
* needssleep wonders if that is what china telecom was also encountering14:44
needssleeperr14:44
needssleepchina mobile14:44
* dtantsur has this spec next up for review14:45
needssleepdtantsur: conductor?14:47
*** mjturek has joined #openstack-ironic14:48
*** mjturek has quit IRC14:50
*** links has quit IRC14:52
dtantsurperiodics workers14:56
needssleepk14:56
*** rpittau_ has quit IRC14:59
*** e0ne has quit IRC14:59
*** r-daneel has joined #openstack-ironic14:59
*** ijw has joined #openstack-ironic15:00
*** zhangfei has quit IRC15:01
*** lbragstad has quit IRC15:03
*** r-daneel_ has joined #openstack-ironic15:04
*** r-daneel has quit IRC15:04
*** r-daneel_ is now known as r-daneel15:04
*** ijw has quit IRC15:05
*** mjturek has joined #openstack-ironic15:05
*** e0ne has joined #openstack-ironic15:05
*** e0ne has quit IRC15:06
*** e0ne has joined #openstack-ironic15:06
*** mjturek has quit IRC15:07
*** bnemec is now known as beekneemech15:09
*** mjturek has joined #openstack-ironic15:10
*** lbragstad has joined #openstack-ironic15:12
*** arnewiebalck__ has quit IRC15:13
*** mjura has quit IRC15:18
*** arnewiebalck_ has quit IRC15:19
*** hoonetorg has quit IRC15:24
*** hoonetorg has joined #openstack-ironic15:25
openstackgerritMichael Turek proposed openstack/ironic-python-agent master: Install grub to PReP partition when prep_boot_part_uuid is provided  https://review.openstack.org/58296215:51
*** trown is now known as trown|lunch15:51
*** r-daneel has quit IRC15:53
*** r-daneel has joined #openstack-ironic15:53
openstackgerritMichael Turek proposed openstack/ironic-python-agent master: Install grub to PReP partition when prep_boot_part_uuid is provided  https://review.openstack.org/58296216:08
*** AlexeyAbashkin has quit IRC16:13
*** e0ne has quit IRC16:14
openstackgerritDebayan Ray proposed openstack/sushy master: Add system storage resource support  https://review.openstack.org/56186316:16
*** med_ has joined #openstack-ironic16:19
*** med_ has quit IRC16:19
*** med_ has joined #openstack-ironic16:19
*** tesseract has quit IRC16:24
*** fried_rice is now known as fried_rolls16:24
*** gkadam has quit IRC16:29
*** MrBerns has quit IRC16:30
*** mgoddard has quit IRC16:34
*** tssurya has joined #openstack-ironic16:41
openstackgerritMerged openstack/ironic-specs master: The direct deploy interface provisioning with HTTP server  https://review.openstack.org/50403916:51
*** dougsz has quit IRC16:55
*** vmud213 has joined #openstack-ironic17:10
dtantsurokay, have a great weekend everyone!17:11
*** dtantsur is now known as dtantsur|afk17:11
needssleephave a wonderful weekend17:11
*** trown|lunch is now known as trown17:12
*** vmud213 has quit IRC17:20
*** spartacloud has joined #openstack-ironic17:26
*** jtomasek has joined #openstack-ironic17:28
openstackgerritMerged openstack/bifrost master: Remove syslinux usage  https://review.openstack.org/58265517:30
*** racedo has quit IRC17:30
*** r-daneel has quit IRC17:39
*** spartacloud has quit IRC17:40
*** jtomasek has quit IRC17:47
*** jtomasek has joined #openstack-ironic17:49
*** jtomasek has quit IRC17:54
*** med_ has quit IRC18:00
*** tssurya has quit IRC18:02
*** kzaitsev1pi has joined #openstack-ironic18:14
*** tssurya has joined #openstack-ironic18:15
*** kzaitsev_pi has quit IRC18:17
*** e0ne has joined #openstack-ironic18:18
waterloojroll: i commented on https://review.openstack.org/#/c/579684/1318:20
patchbotpatch 579684 - ironic - Use conductor group for hash ring calculations18:20
waterloojroll: so that's what i meant, but i didn't (don't) know if that solves the problem. it was my thought at the time. I will think about it more now, wrt rolling upgrades.18:21
jrollwaterloo: thanks18:22
jrollI'm not sure which would be the trigger to look it up as delimited or not18:22
waterloojroll: the rpc api version. at least, that is what i was thinking of at the time.18:24
jrollwaterloo: that still gives us the same issue when rolling out the unpin18:24
waterloojroll: no, or err, i need to convince myself. but with old code OR pin: no delimiter; old code & new code will return same value.18:25
jrollwaterloo: right, but when conductor1 is unpinned and conductor2 is still pinned, it's going to be different18:26
jrollthat said, we realized the keys there have nothing to do with the data in the ring itself18:26
waterloooh, that's what i have to do. really understand what the rings/data represent.18:27
waterloowe should have documented it :-(18:27
needssleepI might be able to  at some point, but I'll need tequila and more time in the universe18:28
jrollwaterloo: this is the code for the actual ring, we just keep some number of them in ironic's data structure18:29
jrollso that said, I'm not sure the keys actually matter much18:31
openstackgerritDebayan Ray proposed openstack/sushy master: Add storage and simple_storage attr to system  https://review.openstack.org/56305118:31
waterloojroll: i will have to disappear to think about this (to prove to myself one way or the other).18:32
jrollwaterloo: fair enough, I'm going to go back to only using the delimited version for now18:32
waterloojroll: ok, I don't think we want two rings. Whether we have one ring with one key per driver, or one ring with two keys per driver (one w/o conductor group) is what I am wondering about.18:34
jrollright, I see what you were saying now18:34
jrollafaict it's all local so nothing matters :)18:34
waterloojroll: you're right. delimited or not delimited, the value is the same.18:36
*** r-daneel has joined #openstack-ironic18:37
*** fried_rolls is now known as fried_rice18:38
*** markguz has joined #openstack-ironic18:39
*** markguz_ has quit IRC18:43
*** ijw has joined #openstack-ironic18:44
*** ijw has quit IRC18:44
*** athomas has quit IRC18:50
*** tssurya has quit IRC18:58
*** jcoufal has quit IRC19:00
*** tssurya has joined #openstack-ironic19:00
*** e0ne has quit IRC19:08
*** arnewiebalck_ has joined #openstack-ironic19:10
*** ijw has joined #openstack-ironic19:12
*** med_ has joined #openstack-ironic19:13
*** med_ has quit IRC19:13
*** med_ has joined #openstack-ironic19:13
*** tssurya has quit IRC19:16
*** tssurya has joined #openstack-ironic19:16
jrollblah, not sure I can leave the grouping stuff out of dbapi19:24
*** markguz has quit IRC19:24
*** markguz has joined #openstack-ironic19:25
*** e0ne has joined #openstack-ironic19:26
*** markguz has quit IRC19:29
*** markguz has joined #openstack-ironic19:35
*** dprince has quit IRC19:37
*** r-daneel has quit IRC19:39
*** markguz_ has joined #openstack-ironic19:40
*** r-daneel has joined #openstack-ironic19:40
*** markguz_ has quit IRC19:40
*** markguz_ has joined #openstack-ironic19:40
*** markguz__ has joined #openstack-ironic19:42
*** markguz__ has quit IRC19:42
*** markguz has quit IRC19:42
*** markguz has joined #openstack-ironic19:42
*** markguz_ has quit IRC19:45
*** racedo has joined #openstack-ironic19:47
*** med_ has quit IRC19:50
*** arnewiebalck_ has quit IRC20:04
*** hjensas has quit IRC20:04
*** dprince has joined #openstack-ironic20:05
*** racedo has quit IRC20:21
openstackgerritJim Rollenhagen proposed openstack/ironic master: Use conductor group for hash ring calculations  https://review.openstack.org/57968420:22
openstackgerritJim Rollenhagen proposed openstack/ironic master: Expose node.conductor_group in the REST API  https://review.openstack.org/58139120:22
jrollso this goes back to what we originally had for the hash ring keys. I think it should be fine.20:23
needssleepokay20:23
jrollalso solves dtantsur|afk's concerns with the driver API, though doesn't move it out of the dbapi20:23
jrolland now I see waterloo suggested what I did :D20:24
jrollgreat minds and all20:24
*** med_ has joined #openstack-ironic20:26
*** med_ has quit IRC20:26
*** med_ has joined #openstack-ironic20:27
*** racedo has joined #openstack-ironic20:28
*** rh-jelabarre has quit IRC20:29
*** med_ has quit IRC20:30
*** rh-jelabarre has joined #openstack-ironic20:34
*** mjturek has quit IRC20:37
*** r-daneel has quit IRC20:40
*** spartacloud has joined #openstack-ironic20:41
*** spartacloud has quit IRC20:45
*** racedo has quit IRC20:49
waterloojroll: :) and sorry about that20:50
*** arnewiebalck_ has joined #openstack-ironic20:55
*** markguz has quit IRC21:00
*** markguz has joined #openstack-ironic21:01
*** trown is now known as trown|outtypewww21:04
*** markguz_ has joined #openstack-ironic21:09
*** markguz_ has quit IRC21:10
*** markguz has quit IRC21:10
*** markguz has joined #openstack-ironic21:11
*** arnewiebalck_ has quit IRC21:11
*** dprince has quit IRC21:11
*** csmart[m] has quit IRC21:17
*** jovial[m] has quit IRC21:17
*** retr0h[m] has quit IRC21:17
*** madhuri[m] has quit IRC21:17
*** r-daneel has joined #openstack-ironic21:20
*** e0ne has quit IRC21:27
*** e0ne has joined #openstack-ironic21:27
*** spartacloud has joined #openstack-ironic21:28
*** e0ne has quit IRC21:30
*** spartacloud has quit IRC21:33
*** markguz_ has joined #openstack-ironic21:39
*** figleaf is now known as edleafe21:41
*** markguz has quit IRC21:42
*** markguz_ has quit IRC21:44
waterloojroll: i reviewed/commented. we should be able to land this early next week :)22:01
*** fried_rice is now known as efried22:09
*** jovial[m] has joined #openstack-ironic22:11
*** weshay is now known as weshay_PTO22:15
*** racedo has joined #openstack-ironic22:27
*** waterloo has quit IRC22:32
*** tssurya has quit IRC22:52
*** alex_xu has quit IRC22:53
*** alex_xu has joined #openstack-ironic23:00
*** racedo has quit IRC23:00
*** madhuri[m] has joined #openstack-ironic23:16
*** csmart[m] has joined #openstack-ironic23:16
*** retr0h[m] has joined #openstack-ironic23:16
*** r-daneel has quit IRC23:26
*** r-daneel has joined #openstack-ironic23:29
*** markguz has joined #openstack-ironic23:33

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!