| TheJulia | I'm calling it a night. My cat has decided that if I am to continue in front of a computer... that she must block access to at least one monitor at any given time. | 00:08 |
|---|---|---|
| TheJulia | "so sayeth the feline overlordd" | 00:12 |
| *** gyee has quit IRC | 00:17 | |
| *** sthussey has quit IRC | 01:17 | |
| *** betherly has joined #openstack-ironic | 01:40 | |
| codyhill | TheJulia: G’Night! Thanks Again! | 01:41 |
| *** betherly has quit IRC | 01:44 | |
| *** rloo has quit IRC | 01:51 | |
| *** sri_ has quit IRC | 02:23 | |
| *** srini_ has joined #openstack-ironic | 02:23 | |
| openstackgerrit | suichangyin proposed openstack/ironic master: The image size is greater than 2GB，download error https://review.openstack.org/622114 | 02:45 |
| *** betherly has joined #openstack-ironic | 02:51 | |
| *** betherly has quit IRC | 02:55 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Add a prelude for ironic 12.0 https://review.openstack.org/623047 | 03:06 |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Allow disabling instance image cache https://review.openstack.org/616031 | 03:10 |
| *** tssurya has quit IRC | 03:42 | |
| *** codyhill has quit IRC | 03:45 | |
| *** hwoarang has quit IRC | 03:47 | |
| *** hwoarang has joined #openstack-ironic | 03:50 | |
| *** etingof has quit IRC | 04:01 | |
| *** betherly has joined #openstack-ironic | 04:32 | |
| *** betherly has quit IRC | 04:37 | |
| *** rh-jelabarre has quit IRC | 04:41 | |
| *** dsneddon has quit IRC | 05:00 | |
| *** dsneddon has joined #openstack-ironic | 05:04 | |
| *** codyhill has joined #openstack-ironic | 05:25 | |
| *** etingof has joined #openstack-ironic | 05:41 | |
| openstackgerrit | Merged openstack/ironic master: Fix OOB introspection to use pxe_enabled flag in idrac driver https://review.openstack.org/617951 | 05:49 |
| *** codyhill has quit IRC | 05:58 | |
| *** skazi has quit IRC | 06:04 | |
| *** skazi has joined #openstack-ironic | 06:48 | |
| *** jtomasek has joined #openstack-ironic | 07:22 | |
| *** e0ne has joined #openstack-ironic | 07:38 | |
| openstackgerrit | Ilya Etingof proposed openstack/ironic master: Raw image size estimation improved https://review.openstack.org/544839 | 07:40 |
| *** rcernin has quit IRC | 07:56 | |
| *** pcaruana has joined #openstack-ironic | 07:58 | |
| *** pcaruana is now known as muttley | 07:58 | |
| openstackgerrit | Merged openstack/ironic master: Fix "import xxx as xxx" grammar https://review.openstack.org/622907 | 08:02 |
| *** stendulker has joined #openstack-ironic | 08:03 | |
| *** rcernin has joined #openstack-ironic | 08:03 | |
| *** gkadam has joined #openstack-ironic | 08:16 | |
| *** skazi has quit IRC | 08:19 | |
| *** skazi has joined #openstack-ironic | 08:22 | |
| *** skazi has quit IRC | 08:22 | |
| *** gabys has joined #openstack-ironic | 08:23 | |
| *** rcernin has quit IRC | 08:33 | |
| *** gabys has quit IRC | 08:42 | |
| *** gabys has joined #openstack-ironic | 08:43 | |
| *** gabys has quit IRC | 08:48 | |
| *** mkrai has joined #openstack-ironic | 09:00 | |
| mkrai | Hi mgoddard, I tried doing the multitenant setup using this local.conf http://paste.openstack.org/show/736740/ | 09:01 |
| mkrai | But still the boot is unsuccessful | 09:02 |
| mkrai | mgoddard: Which parameter in local.conf tells that it is a multitenant setup? | 09:02 |
| *** iurygregory has joined #openstack-ironic | 09:03 | |
| iurygregory | good morning everyone o/ | 09:04 |
| etingof | o/ iurygregory | 09:12 |
| iurygregory | etingof \o | 09:14 |
| *** dougsz has joined #openstack-ironic | 09:15 | |
| *** S4ren has joined #openstack-ironic | 09:18 | |
| *** dtantsur|afk is now known as dtantsur | 09:29 | |
| dtantsur | morning ironic | 09:29 |
| *** derekh has joined #openstack-ironic | 09:44 | |
| iurygregory | morning dtantsur | 09:57 |
| *** tssurya has joined #openstack-ironic | 09:59 | |
| *** robbbe has joined #openstack-ironic | 10:05 | |
| *** dnuka has joined #openstack-ironic | 10:08 | |
| openstackgerrit | Merged openstack/ironic stable/rocky: Add ipmi_disable_timeout to avoid problematic IPMI command https://review.openstack.org/620455 | 10:08 |
| dnuka | good morning everyone :) | 10:08 |
| iurygregory | =) | 10:08 |
| dnuka | folks, how do I resolve a merge conflict? | 10:11 |
| dnuka | #link https://review.openstack.org/#/c/608563/ | 10:11 |
| patchbot | patch 608563 - sushy - Add support for the `CompositionService` resource - 15 patch sets | 10:11 |
| dtantsur | dnuka: update the master branch of your local git checkout (git pull --ff-only), then rebase your patch on top of master, then reupload. | 10:12 |
| dtantsur | the rebase step will involve git complaining about conflict and you resolving them by hand | 10:13 |
| dtantsur | this can be an interesting experience the first time :) but you'll get used to it | 10:13 |
| dnuka | thank you dtantsur :) | 10:13 |
| iurygregory | merge conflict </3 | 10:14 |
| dnuka | iurygregory: :D | 10:14 |
| dtantsur | nobody likes'em man :) | 10:14 |
| iurygregory | exactly XD | 10:14 |
| openstackgerrit | Dmitry Tantsur proposed openstack/metalsmith master: [WIP] Completely switch to openstacksdk https://review.openstack.org/619558 | 10:16 |
| *** arne_wiebalck_ has joined #openstack-ironic | 10:24 | |
| *** e0ne has quit IRC | 10:24 | |
| *** e0ne has joined #openstack-ironic | 10:27 | |
| *** stendulker has quit IRC | 10:37 | |
| *** iurygregory has quit IRC | 10:38 | |
| *** iurygregory has joined #openstack-ironic | 10:38 | |
| openstackgerrit | Dhanuka Warusadura proposed openstack/sushy master: Add support for the `CompositionService` resource https://review.openstack.org/608563 | 11:00 |
| openstackgerrit | Ilya Etingof proposed openstack/ironic master: Raw image size estimation improved https://review.openstack.org/544839 | 11:08 |
| jroll | morning | 11:09 |
| jroll | thanks for fixing up https://review.openstack.org/#/c/616031 for me, TheJulia :) | 11:09 |
| patchbot | patch 616031 - ironic - Allow disabling instance image cache - 3 patch sets | 11:09 |
| dnuka | morning jroll o/ | 11:09 |
| etingof | jroll dnuka o/ | 11:09 |
| dnuka | hi etingof :) | 11:10 |
| jroll | \o | 11:10 |
| dnuka | dtantsur: I have just noticed, https://review.openstack.org/#/c/608177/ | 11:11 |
| patchbot | patch 608177 - sushy - Add `ChassisCollection` and `Chassis` classes (MERGED) - 11 patch sets | 11:11 |
| dnuka | patch missing a test for the `_chassis_path` | 11:11 |
| dnuka | #link https://review.openstack.org/#/c/608177/11/sushy/tests/unit/test_main.py | 11:11 |
| patchbot | patch 608177 - sushy - Add `ChassisCollection` and `Chassis` classes (MERGED) - 11 patch sets | 11:11 |
| dnuka | dtantsur: if it's important, I can add that in separate patch :) | 11:14 |
| mkrai | dtantsur: Hi, I tried the multitenant setup with http://logs.openstack.org/33/618733/1/check/ironic-tempest-dsvm-ipa-wholedisk-agent_ipmitool-tinyipa-multinode/db83e12/logs/local.conf.txt.gz | 11:18 |
| mkrai | but still not successful to boot a node | 11:18 |
| mkrai | dtantsur: Which parameter in this conf tells that this is a multitenant setup? | 11:18 |
| dtantsur | dnuka: I'm not sure, leaving it up to you | 11:22 |
| dtantsur | mkrai: IRONIC_NETWORK_INTERFACE=neutron | 11:22 |
| dnuka | dtantsur: thank you :) | 11:22 |
| *** jesusaur has quit IRC | 11:27 | |
| *** jesusaur has joined #openstack-ironic | 11:31 | |
| mkrai | dtantsur: does devstack does any other configuration with that parameter? | 11:31 |
| dtantsur | mkrai: well, all other parameters. I don't quite get the question, sorry.. | 11:32 |
| mkrai | My assumption is the nodes' network_interface is set to neutron and also the *_network_interface conf is set in /etc/ironic/ironic.conf | 11:32 |
| mkrai | dtantsur: I meant what different setup is done by devstack when this parameter is defined IRONIC_NETWORK_INTERFACE=neutron | 11:33 |
| dtantsur | mkrai: it's essentially default_network_interface in ironic.conf | 11:33 |
| openstackgerrit | Dmitry Tantsur proposed openstack/metalsmith master: [WIP] Completely switch to openstacksdk https://review.openstack.org/619558 | 11:35 |
| mkrai | dtantsur: Ok and what does this https://github.com/openstack/ironic/blob/master/devstack/lib/ironic#L2454 do? | 11:37 |
| dtantsur | mkrai: enables testing VIF attach/detach in tempest IIRC | 11:37 |
| *** arne_wiebalck_ has quit IRC | 11:39 | |
| mkrai | dtantsur: Thank you | 11:45 |
| *** mkrai has quit IRC | 11:54 | |
| openstackgerrit | Dhanuka Warusadura proposed openstack/sushy master: Add missing tests https://review.openstack.org/623197 | 11:56 |
| *** gkadam_ has joined #openstack-ironic | 12:00 | |
| *** gkadam has quit IRC | 12:03 | |
| *** gkadam_ has quit IRC | 12:03 | |
| *** gkadam has joined #openstack-ironic | 12:05 | |
| *** sambetts_ has quit IRC | 12:13 | |
| openstackgerrit | Merged openstack/ironic master: Fix IPv6 Option Passing https://review.openstack.org/621416 | 12:14 |
| *** bfournie has quit IRC | 12:28 | |
| *** hjensas has quit IRC | 12:49 | |
| *** betherly has joined #openstack-ironic | 12:52 | |
| *** arne_wiebalck_ has joined #openstack-ironic | 12:53 | |
| *** gkadam_ has joined #openstack-ironic | 12:55 | |
| *** rh-jelabarre has joined #openstack-ironic | 12:57 | |
| *** gkadam has quit IRC | 12:58 | |
| *** arne_wiebalck_ has quit IRC | 12:58 | |
| *** MattMan_ has quit IRC | 13:00 | |
| *** MattMan_ has joined #openstack-ironic | 13:01 | |
| *** betherly has quit IRC | 13:01 | |
| *** arne_wiebalck_ has joined #openstack-ironic | 13:03 | |
| *** bfournie has joined #openstack-ironic | 13:08 | |
| *** muttley has quit IRC | 13:08 | |
| *** srini_ has quit IRC | 13:11 | |
| *** sri_ has joined #openstack-ironic | 13:16 | |
| TheJulia | good morning | 13:16 |
| sri_ | TheJulia, Good Morning | 13:16 |
| *** rcernin has joined #openstack-ironic | 13:17 | |
| dnuka | good morning TheJulia :) | 13:17 |
| TheJulia | jroll: no problem :) | 13:19 |
| openstackgerrit | Dmitry Tantsur proposed openstack/metalsmith master: [WIP] Completely switch to openstacksdk https://review.openstack.org/619558 | 13:19 |
| dtantsur | morning TheJulia | 13:19 |
| *** muttley has joined #openstack-ironic | 13:21 | |
| *** muttley has quit IRC | 13:25 | |
| *** muttley has joined #openstack-ironic | 13:26 | |
| openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/ironic-inspector master: No way to unset a property via introspection rules https://review.openstack.org/622898 | 13:27 |
| iurygregory | morning TheJulia | 13:27 |
| *** rcernin has quit IRC | 13:29 | |
| *** muttley has quit IRC | 13:29 | |
| *** rloo has joined #openstack-ironic | 13:29 | |
| *** pcaruana has joined #openstack-ironic | 13:34 | |
| jroll | TheJulia: morning :) I dropped some comments on your clean steps spec too, since I can't be there tomorrow | 13:34 |
| *** pcaruana has quit IRC | 13:39 | |
| *** tssurya has quit IRC | 13:41 | |
| *** tssurya has joined #openstack-ironic | 13:41 | |
| jroll | rloo: this is just waiting for you, if you have time today :) https://review.openstack.org/#/c/616031/ | 13:43 |
| patchbot | patch 616031 - ironic - Allow disabling instance image cache - 3 patch sets | 13:43 |
| *** S4ren has quit IRC | 13:43 | |
| *** pcaruana has joined #openstack-ironic | 13:43 | |
| *** pcaruana has quit IRC | 13:47 | |
| *** arne_wiebalck_ has quit IRC | 13:57 | |
| *** baha has joined #openstack-ironic | 13:57 | |
| *** yolanda has quit IRC | 13:57 | |
| *** iurygregory has quit IRC | 13:59 | |
| *** iurygregory has joined #openstack-ironic | 13:59 | |
| rloo | jroll: thx for pinging me, i looked yesterday? and it needed an update. looking now... | 14:00 |
| jroll | rloo: thanks! | 14:01 |
| rloo | jroll: done. i'll do the other fix. | 14:02 |
| jroll | thanks | 14:02 |
| *** dnuka has quit IRC | 14:08 | |
| *** jcoufal has joined #openstack-ironic | 14:10 | |
| openstackgerrit | Jim Rollenhagen proposed openstack/ironic master: Add a note regarding IPA multidevice fix https://review.openstack.org/599056 | 14:11 |
| *** sthussey has joined #openstack-ironic | 14:13 | |
| *** S4ren has joined #openstack-ironic | 14:18 | |
| *** S4ren has quit IRC | 14:23 | |
| *** baha has quit IRC | 14:30 | |
| *** baha has joined #openstack-ironic | 14:42 | |
| *** baha has quit IRC | 14:44 | |
| *** baha has joined #openstack-ironic | 14:52 | |
| TheJulia | jroll: thanks! | 14:57 |
| TheJulia | Has anyone ever looked at having a mode where we have a single ironic process as opposed to an api and conductor process? | 14:57 |
| TheJulia | (Thinking super standalone-ish cases) | 14:58 |
| dtantsur | TheJulia: tripleo folks keep asking about it. I keep rolling eyes :) | 14:58 |
| *** mjturek has joined #openstack-ironic | 14:58 | |
| TheJulia | (and heading off external factors that have done things like forking project's code bases) | 14:58 |
| dtantsur | this is quite related to their idea of using ironic with k8s, but I'm skeptical | 14:58 |
| TheJulia | dtantsur: I mean, I think it is easy if we had zeromq back :) | 14:59 |
| dtantsur | right | 14:59 |
| dtantsur | or use an RPC solution for RPC | 14:59 |
| TheJulia | yeah | 14:59 |
| dtantsur | I envision problems with load here. I think our conductors can easily saturate the API. | 14:59 |
| TheJulia | I've got some code locally, just my transport_url config is being ignored | 15:00 |
| TheJulia | looks like how we initialize the RPC layer :\ | 15:00 |
| dtantsur | yeah, a can of worms. and I'm not sure why on Earth k8s folks started liking fat containers :) | 15:01 |
| jroll | oh, don't get me started on fat containers | 15:01 |
| TheJulia | they want minimal things that are actually huge under the hood? | 15:01 |
| dtantsur | minimal and huge in one sentence :) | 15:01 |
| TheJulia | If we don't know how the factory works, it can just be a tiny conveyer belt where cute packages rolls out of... ignore the acres of building behind the conveyer belt | 15:02 |
| dtantsur | and to squeeze TFTP in you'll need supervisord or something | 15:02 |
| dtantsur | right | 15:02 |
| dtantsur | TheJulia: if you need inspiration on this path, you can check ironic-inspector's functional tests. they work exactly this way. | 15:02 |
| TheJulia | yeah, there are a whole bunch of problems there :( | 15:02 |
| TheJulia | dtantsur: Oh! really!?!? | 15:02 |
| TheJulia | dtantsur: using the fake oslo_messaging driver? | 15:03 |
| dtantsur | no kidding | 15:03 |
| dtantsur | yes | 15:03 |
| TheJulia | sweet! lets see if they and I are trying to do the exact same thing | 15:03 |
| TheJulia | dtantsur: thanks! | 15:03 |
| dtantsur | np :) | 15:03 |
| iurygregory | dtantsur, since https://review.openstack.org/#/c/622983 failed to merged you want me to change the decorator to the class? | 15:10 |
| patchbot | patch 622983 - ironic - Follow-up Retries and timeout for IPA command - 1 patch set | 15:10 |
| dtantsur | iurygregory: or you can just recheck, I don't insist :) or is it a merge conflict? | 15:10 |
| *** livelace has joined #openstack-ironic | 15:11 | |
| openstackgerrit | Merged openstack/virtualbmc master: Change CI job to use python3 https://review.openstack.org/589296 | 15:12 |
| iurygregory | dtantsur, just ci with some headache, someone gave bad beer to the hosts | 15:12 |
| dtantsur | heh, give them some aspirin and recheck :) | 15:13 |
| iurygregory | yeah =) | 15:13 |
| *** jcoufal_ has joined #openstack-ironic | 15:14 | |
| *** jcoufal_ has quit IRC | 15:15 | |
| *** codyhill has joined #openstack-ironic | 15:15 | |
| dtantsur | speaking of headache, I need to finally record narration for the freaking demo video. will go totally offline for that. | 15:15 |
| iurygregory | good luck | 15:15 |
| *** jcoufal_ has joined #openstack-ironic | 15:16 | |
| *** jcoufal has quit IRC | 15:17 | |
| TheJulia | dtantsur: I think I see what the issue is and why it is working for inspector's functional tests and not for ironic when I try it... it is how our rpc client innitalizes but I think that is just an artifact of early service-ness and is super easy to fix. | 15:18 |
| iurygregory | when its easy everybody is happy \o/ | 15:19 |
| *** gkadam_ has quit IRC | 15:19 | |
| *** jhesketh has quit IRC | 15:34 | |
| *** jhesketh has joined #openstack-ironic | 15:35 | |
| openstackgerrit | Merged openstack/ironic master: Slim down grenade jobs https://review.openstack.org/620970 | 15:35 |
| *** S4ren has joined #openstack-ironic | 15:38 | |
| openstackgerrit | Merged openstack/ironic master: Change multinode jobs to default to local boot https://review.openstack.org/623038 | 15:45 |
| *** jaypipes has quit IRC | 15:56 | |
| *** baha has quit IRC | 15:57 | |
| *** jaypipes has joined #openstack-ironic | 15:58 | |
| *** jcoufal_ has quit IRC | 16:01 | |
| * dtantsur is back, done and exhausted | 16:01 | |
| *** dtrainor has joined #openstack-ironic | 16:05 | |
| dtrainor | hey there, dtantsur. so i'm running in to what I think is profile matching not working in a downstream osp13. ultimately i think it comes down to: ram: -8192MB disk: -81920MB io_ops: 0 instances: 0 does not have 4096 MB usable ram before overcommit, it only has 0 MB. | 16:07 |
| dtrainor | here's a bigger snippet | 16:07 |
| *** robbbe has quit IRC | 16:07 | |
| dtrainor | introspection seems to work fine, enver had a problem with that. node data appears to be accurate. | 16:07 |
| dtantsur | it's undercloud, right? | 16:08 |
| dtrainor | i'll also say that this is on Ravello (i can elaborate if necessary) and there may also be a bug in the vm BIOS that reports inaccurate memory but I don't have any way to prove that. | 16:08 |
| *** codyhill has quit IRC | 16:08 | |
| dtrainor | correct, also using UI to deploy with. it's a very simple deployment with nothing too advanced going on. | 16:08 |
| dtantsur | dtrainor: could you check that your flavors are set up as in https://docs.openstack.org/ironic/latest/install/configure-nova-flavors.html ? | 16:09 |
| dtantsur | especially around the "resources" bits | 16:09 |
| openstackgerrit | Merged openstack/ironic master: Allow disabling instance image cache https://review.openstack.org/616031 | 16:09 |
| openstackgerrit | Merged openstack/ironic master: Expose conductors: api https://review.openstack.org/620243 | 16:10 |
| openstackgerrit | Merged openstack/ironic master: Add a note regarding IPA multidevice fix https://review.openstack.org/599056 | 16:10 |
| *** baha has joined #openstack-ironic | 16:10 | |
| * dtantsur hates his voice, daaaaaaaamn >_< | 16:13 | |
| dtrainor | i've been combing over that doc the last few days and to the best of my knowledge and understanding, the default resources:CUSTOM_BAREMETAL='1' should be ok to use (don't need to specify a custom resource) so long as the flavor property list also contains capabilities:profile='compute' | 16:13 |
| dtrainor | hehe hearing it on recordings? i do the same | 16:13 |
| dtantsur | yeah :( | 16:13 |
| dtrainor | here's some more flavor, node output http://paste.openstack.org/show/736768/ | 16:13 |
| jroll | both of you being blue in my client is very confusing :P | 16:14 |
| dtrainor | haha | 16:14 |
| dtantsur | LOL | 16:14 |
| dtantsur | dtrainor: provision_state=deploying? is it doing something now? | 16:15 |
| dtantsur | jroll: just imagine me talking to myself after narrating 35 mins of demo in one go :D | 16:15 |
| jroll | oh I am :) | 16:15 |
| dtrainor | i think it's just rolling after a failed deployment: | 16:15 |
| dtrainor | Resource CREATE failed: resources.Controller: Resource CREATE failed: ResourceInError: resources[0].resources.Controller: Went to status ERROR due to "Message: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 4f9c8dc1-3e62-4fde-81cd-5ba206686763., Code: 500" | 16:16 |
| dtantsur | dtrainor: what I'm trying to understand is whether your nodes start deploying then fail. Or if they fail before deploying. | 16:16 |
| dtrainor | i'll note that i've managed to reproduce the same kind of deployment in a different environment and that works as expected without modifying any of the resources | 16:16 |
| dtantsur | this looks like at least some nodes to start deploying. can you find a node that does not? | 16:16 |
| dtrainor | oh i see | 16:17 |
| dtrainor | i'm not certain what would constitute "failing" other than the node just being in some weird state after the deployment itself failed | 16:17 |
| dtantsur | two things could happen: 1. some nodes started deploying, but some did not, so heat failed. | 16:18 |
| dtantsur | 2. node start deploying, but fail mid-way. Then nova retries, then eventually fails. | 16:18 |
| dtantsur | both could result in some nodes still deploying when heat bails out | 16:18 |
| dtrainor | understood | 16:19 |
| dtrainor | i've been looking for that in logs but haven't been able to identify any kind of pattern. | 16:19 |
| dtantsur | dtrainor: nothing suspicious in ironic-conductor logs? | 16:19 |
| dtrainor | in the logs when I saw a memory and disk amount of -81920MB it made me wonder if some incorrect or non-value was being reported somehow, somewhere. but my understanding is that there's no pre-deployment introspection or resource matching after the initial introspection, so I don't know where this value comes from | 16:20 |
| *** jcoufal has joined #openstack-ironic | 16:20 | |
| TheJulia | jroll: they are two different colors for me | 16:20 |
| jroll | lucky you :P | 16:20 |
| dtantsur | dtrainor: I've seen nova doing that. I never understood why.. | 16:21 |
| dtrainor | i see nothing suspicious in ironic conductor logs, no | 16:22 |
| dtantsur | dtrainor: what's in $ openstack hypervisor stats show? | 16:23 |
| dtantsur | also try $ openstack compute service list | 16:23 |
| dtrainor | http://paste.openstack.org/show/736769/ | 16:24 |
| dtrainor | i had recently restarted openstack-nova-* to disable, test, and re-enable, the memory and disk filters to test for a possible bug, but same results | 16:25 |
| dtantsur | yeah, RamFilter was not broken in Queens.. | 16:25 |
| * dtantsur starts running out of ideas | 16:25 | |
| * TheJulia reads | 16:26 | |
| dtrainor | leave it to me, to break stuff | 16:26 |
| dtrainor | i'm getting really, really good at this | 16:26 |
| TheJulia | dtrainor: if you ask ironic to run a validate command the nodees that are failing to deploy, what do you get? | 16:27 |
| TheJulia | may be worth checking api/conductor logs to see if that is where it is going sideways | 16:28 |
| TheJulia | in other words, something impossible or incompatible is being requested and because of all the layers of abstraction it is not easy to see | 16:28 |
| dtrainor | so i'm not familiar enough to know the intricate details and differences between iSCSI deploy and Direct deploy, but looking closer I do see an error in ironic-conductor logs about not being able to start the iSCSI target http://paste.openstack.org/show/736770/ | 16:28 |
| dtrainor | TheJulia, that's why i'm enlisting the braintrust, for that exact reason | 16:29 |
| dtantsur | this definitely looks problematic | 16:30 |
| TheJulia | no such file or directory, your ramdisk might be missing the iscsi-initiator-tools | 16:30 |
| dtrainor | I see https://docs.openstack.org/ironic/queens/admin/interfaces/deploy.html that iSCSI deploy is the default, sure | 16:30 |
| *** e0ne has quit IRC | 16:30 | |
| dtrainor | i see. | 16:30 |
| dtrainor | soooo that's interesting | 16:30 |
| dtrainor | let me review something here | 16:30 |
| TheJulia | dtrainor: difference between the two is basically direct uses swift to download the file from a swift backed glance and writes it to disk locally if it can fit it into memory. iscsi offers a disk up via iscsi to the conductor, who connects to it, and writes the image out | 16:30 |
| dtrainor | appreciate it, thanks | 16:31 |
| dtrainor | "the image", are we talking ironic-python-agent.initramfs? | 16:31 |
| TheJulia | no, the image I speak of is what you want written to the disk | 16:32 |
| dtrainor | which image, the overcloud image, e.g. overcloud-fill ? | 16:33 |
| dtrainor | *overcloud-full | 16:33 |
| dtrainor | oh good, my environment shut down | 16:36 |
| dtrainor | TheJulia, re: run a validate command, i'll add that to the list, thank you | 16:37 |
| TheJulia | yeah, it would be overcloud-full that is downloaded | 16:37 |
| TheJulia | validate's output may change with instance information populated because each driver interface evaluates the settings it can see and will raise issues if is anything known | 16:38 |
| dtrainor | understood. all information is good information at this point | 16:40 |
| dtrainor | thank you | 16:40 |
| *** gabys has joined #openstack-ironic | 16:41 | |
| TheJulia | oh, I see why I can't override the rpc url. We fire up the RPC upon config load | 16:41 |
| *** iurygregory is now known as iurygregoy|gone | 16:57 | |
| *** gabys has quit IRC | 17:03 | |
| *** dsneddon has quit IRC | 17:12 | |
| *** dsneddon has joined #openstack-ironic | 17:12 | |
| *** tssurya has quit IRC | 17:18 | |
| *** jtomasek has quit IRC | 17:32 | |
| dtantsur | the demo I've been creating, if somebody is curious: https://youtu.be/R5Uw0_we-p8 | 17:46 |
| *** gabys has joined #openstack-ironic | 17:49 | |
| *** livelace has quit IRC | 17:49 | |
| dougsz | thanks for sharing dtantsur | 17:54 |
| dtantsur | welcome | 17:54 |
| *** gyee has joined #openstack-ironic | 17:55 | |
| *** derekh has quit IRC | 17:59 | |
| *** bdodd has quit IRC | 17:59 | |
| *** dtantsur is now known as dtantsur|afk | 18:01 | |
| dtantsur|afk | see you | 18:01 |
| dtrainor | thanks again for the help today dtantsur|afk | 18:01 |
| *** dougsz has quit IRC | 18:02 | |
| *** e0ne has joined #openstack-ironic | 18:08 | |
| *** e0ne has quit IRC | 18:09 | |
| dtrainor | TheJulia, looks like there's no iSCSI errors any more after using the Direct deploy method, but still seeing that weird: (undercloud.example.com, f9ca17d0-10ae-486d-9ef8-68f268ce9eaf) ram: -8192MB disk: -81920MB io_ops: 0 instances: 0 does not have 4096 MB usable ram before overcommit, it only has 0 MB. | 18:38 |
| *** jcoufal has quit IRC | 18:40 | |
| *** jcoufal has joined #openstack-ironic | 18:40 | |
| *** bdodd has joined #openstack-ironic | 18:42 | |
| TheJulia | dtrainor: can you share your nova-compute.conf ? | 18:43 |
| dtrainor | the validate shows that the ramdisk, kernel, and image_source parameters are missing from both the boot and deploy interfaces. though ramdisk and kernel are available (and the node is configured to use this), however image_source is not. in fact, I don't see it in any ironic or nova config files | 18:43 |
| dtrainor | i don't have a nova-compute.conf | 18:44 |
| dtrainor | this is a downstream queens deployment, i have a nova.conf and a nova-compute.log | 18:48 |
| dtrainor | hrm, there is no driver_info that lists 'ramdisk' for a node, only 'deploy_ramdisk'. The node validate says it's looking for 'ramdisk'. Not sure if that's a red herring; validate looking for 'ramdisk' when it should in fact be looking for 'deploy_ramdisk' | 18:54 |
| *** bfournie has quit IRC | 19:15 | |
| NobodyCam | Good Morning Ironic'ers | 19:36 |
| TheJulia | dtrainor: well, nova.conf's compute parameters I guess | 19:59 |
| TheJulia | so deploy_ramdisk and deploy_kernel are what we expect for ironic-python-agent's ramdisk | 19:59 |
| TheJulia | 'ramdisk' 'kernel' 'image_source... and there are a few other related keys that your noticing are more for a successful deployment | 20:00 |
| TheJulia | dtrainor: since they can be manually populated and you can tell ironic "go deploy now please thanks!" | 20:00 |
| TheJulia | and it will go do if the fields are defined | 20:00 |
| *** gabys has quit IRC | 20:08 | |
| *** gabys has joined #openstack-ironic | 20:09 | |
| *** gabys has quit IRC | 20:09 | |
| *** gabys has joined #openstack-ironic | 20:10 | |
| *** gabys has quit IRC | 20:10 | |
| *** gabys has joined #openstack-ironic | 20:10 | |
| *** gabys has quit IRC | 20:11 | |
| *** gabys has joined #openstack-ironic | 20:11 | |
| *** gabys has quit IRC | 20:11 | |
| *** gabys has joined #openstack-ironic | 20:12 | |
| *** gabys has quit IRC | 20:12 | |
| *** sthussey has quit IRC | 20:12 | |
| *** rcernin has joined #openstack-ironic | 20:12 | |
| *** gabys has joined #openstack-ironic | 20:13 | |
| *** gabys has quit IRC | 20:13 | |
| *** gabys has joined #openstack-ironic | 20:14 | |
| *** gabys has quit IRC | 20:14 | |
| *** gabys has joined #openstack-ironic | 20:14 | |
| *** gabys has quit IRC | 20:14 | |
| dtrainor | sorry, just reading this, let me catch up | 20:15 |
| *** gabys has joined #openstack-ironic | 20:15 | |
| *** gabys has quit IRC | 20:15 | |
| TheJulia | dtrainor: no worries, I've been mentally bouncing between three things all morning. If anything I've just not been looking at IRC constantly | 20:15 |
| dtrainor | ok. so i should in fact have ramdisk, kernel, image_source parameters to the node? | 20:16 |
| TheJulia | well | 20:16 |
| dtrainor | all good, i'm trying to do the same in between testing this :) i'm ok with async comms, i appreciate the help | 20:16 |
| TheJulia | nova populates that information | 20:16 |
| *** gabys has joined #openstack-ironic | 20:16 | |
| dtrainor | sure | 20:16 |
| *** gabys has quit IRC | 20:16 | |
| *** gabys has joined #openstack-ironic | 20:17 | |
| *** gabys has quit IRC | 20:17 | |
| TheJulia | dtrainor: if you want, you can share your nova-compute.log output and I'm happy to sift through it real quick, that might provide insight | 20:17 |
| *** gabys has joined #openstack-ironic | 20:17 | |
| dtrainor | i'm sorry, spaced on that. doing a deployment now to get fresh logs. | 20:18 |
| *** gabys has quit IRC | 20:18 | |
| dtrainor | while also eyeballing this, because it smells familiar https://bugzilla.redhat.com/show_bug.cgi?id=1461889 | 20:18 |
| openstack | bugzilla.redhat.com bug 1461889 in openstack-nova "Nova compute resource_tracker incorrectly sets overcommitted nodes free_mem to 0 affecting nova-scheduler's ram_filter" [Medium,New] - Assigned to nova-maint | 20:18 |
| *** gabys has joined #openstack-ironic | 20:18 | |
| *** gabys has quit IRC | 20:18 | |
| *** gabys has joined #openstack-ironic | 20:19 | |
| *** gabys has quit IRC | 20:19 | |
| *** gabys has joined #openstack-ironic | 20:20 | |
| *** gabys has quit IRC | 20:20 | |
| TheJulia | That sounds a lot like another issue I kind of remember | 20:21 |
| TheJulia | but it feels like that was more recent | 20:21 |
| dtrainor | waiting... | 20:26 |
| TheJulia | launch kerbal space program? ;) | 20:26 |
| dtrainor | that will be the end of me. better pretend i didn't see that. | 20:27 |
| TheJulia | lol | 20:27 |
| dtrainor | waiting for clean-wait before deploying | 20:28 |
| TheJulia | dtrainor: alternative: get a cat? | 20:29 |
| dtrainor | dogs eat cats | 20:29 |
| dtrainor | probably. | 20:29 |
| dtrainor | i have a whole pile of that kinetic sand which is usually good for passing time. brb. | 20:29 |
| TheJulia | but cats learn to interrupt your focus for attention *looks down at her cat who is disregarding the glaring looks* | 20:30 |
| TheJulia | nice | 20:30 |
| dtrainor | ok, it's deploying now | 20:36 |
| dtrainor | what's the best way to get this rather large log to you? | 20:38 |
| TheJulia | compress it and whatever your favorite transfer medium | 20:41 |
| TheJulia | except dcc | 20:42 |
| TheJulia | I'm fairly sure irccloud does not support that | 20:42 |
| dtrainor | heh | 20:49 |
| dtrainor | Resource CREATE failed: resources.CephStorage: Resource CREATE failed: ResourceInError: resources[1].resources.CephStorage: Went to status ERROR due to "Message: Unknown, Code: Unknown" | 20:56 |
| dtrainor | ahh a new error this time. | 20:56 |
| dtrainor | i'll have to try this again, sorry for the delay. | 20:57 |
| *** bfournie has joined #openstack-ironic | 21:04 | |
| TheJulia | dtrainor: no worries :( | 21:08 |
| *** jcoufal has quit IRC | 21:19 | |
| *** baha has quit IRC | 21:48 | |
| *** mjturek has quit IRC | 21:52 | |
| dtrainor | this is seriously messed up... still working at getting a deployment TheJulia | 22:27 |
| TheJulia | dtrainor: don't rush on my account, I have plenty to keep me busy | 22:36 |
| *** mjturek has joined #openstack-ironic | 22:38 | |
| dtrainor | i just want to squash this. badly. | 22:43 |
| TheJulia | was this always on deploying your overcloud? | 22:44 |
| dtrainor | yep. | 22:46 |
| TheJulia | can you just ask the undercloud to create you a server with the same falavor being used for overcloud deployment? | 22:49 |
| *** cristoph_ has joined #openstack-ironic | 22:49 | |
| TheJulia | err, flavor | 22:49 |
| dtrainor | just to test? | 22:50 |
| * TheJulia yeah | 22:50 | |
| dtrainor | i did that when i was playing around with profile matching to see if there were any default non-defaults that were missing | 22:50 |
| dtrainor | i've created this exact same deployment on actual bare metal and can't get it to fail the same way. | 22:51 |
| dtrainor | so there's something different in this environment, I don't know what it is, not sure if it's software or hardware, or the way that software is reacting to hardware, but it's a thing | 22:51 |
| TheJulia | any errors or warnings in your nova-compute.log? | 22:52 |
| TheJulia | tl;dr there should be at least a few warnings starting out | 22:52 |
| dtrainor | i was not able to see any but i had truncated mine to make a smaller log for you to consume, and haven't been able to attempt a deployment since :) | 22:52 |
| *** mjturek has quit IRC | 22:53 | |
| TheJulia | That kentic sand is sounding mighty appealing at the moment :) | 22:53 |
| dtrainor | fact. | 22:54 |
| dtrainor | im going to go read for a bit while this imports / introspects / cleans etc, need a mental break. | 22:54 |
| TheJulia | ++ | 22:54 |
| TheJulia | I'll be around for the next couple hours. I think my dinner plans went sideways so possibly even longer | 22:55 |
| *** cristoph_ has quit IRC | 22:58 | |
| *** mjturek has joined #openstack-ironic | 23:16 | |
| dtrainor | ok. here we go. | 23:41 |
| *** rloo has quit IRC | 23:44 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!