| *** srart has joined #openstack-ironic | 00:12 | |
| *** cjloader has joined #openstack-ironic | 00:20 | |
| *** cjloader has quit IRC | 00:24 | |
| *** bfournie has quit IRC | 00:25 | |
| *** bfournie has joined #openstack-ironic | 00:26 | |
| *** bfournie has quit IRC | 00:30 | |
| *** hshiina has joined #openstack-ironic | 00:36 | |
| *** srart has quit IRC | 00:39 | |
| *** Sukhdev has quit IRC | 00:42 | |
| *** bfournie has joined #openstack-ironic | 00:47 | |
| *** srart has joined #openstack-ironic | 00:55 | |
| *** rloo has quit IRC | 00:58 | |
| *** threestrands has quit IRC | 01:00 | |
| *** zhangfei has joined #openstack-ironic | 01:02 | |
| *** phuongnh has joined #openstack-ironic | 01:08 | |
| *** trungnv has quit IRC | 01:16 | |
| *** trungnv has joined #openstack-ironic | 01:17 | |
| *** gyee has quit IRC | 01:18 | |
| *** baoli has joined #openstack-ironic | 01:23 | |
| *** baoli has quit IRC | 01:28 | |
| *** tiendc has joined #openstack-ironic | 01:34 | |
| *** namnh has joined #openstack-ironic | 01:35 | |
| anupn | Hi I am seeing my devstack installation failing when it is trying to provision the ironic nodes. The setup waits for nodes to become available but none of the nodes become available | 02:02 |
|---|---|---|
| anupn | has anyone seen similar error | 02:02 |
| anupn | I don't see any errors inside ir-cond logs | 02:03 |
| *** hemna_ has quit IRC | 02:12 | |
| *** baoli has joined #openstack-ironic | 02:13 | |
| *** baoli has quit IRC | 02:18 | |
| *** phuongnh has quit IRC | 02:23 | |
| *** phuongnh has joined #openstack-ironic | 02:24 | |
| *** bfournie has quit IRC | 02:33 | |
| *** bfournie has joined #openstack-ironic | 02:34 | |
| *** hemna_ has joined #openstack-ironic | 02:34 | |
| *** bfournie has quit IRC | 02:38 | |
| *** rcernin has quit IRC | 02:46 | |
| *** rcernin has joined #openstack-ironic | 02:50 | |
| *** fragatina has quit IRC | 03:06 | |
| *** cjloader has joined #openstack-ironic | 03:15 | |
| *** cjloader has quit IRC | 03:19 | |
| *** zhangfei has quit IRC | 03:35 | |
| *** rbudden has quit IRC | 03:36 | |
| *** links has joined #openstack-ironic | 03:37 | |
| *** tuanla____ has joined #openstack-ironic | 03:38 | |
| *** zhangfei has joined #openstack-ironic | 03:47 | |
| *** gyankum has joined #openstack-ironic | 03:59 | |
| *** fragatina has joined #openstack-ironic | 04:05 | |
| *** fragatina has quit IRC | 04:06 | |
| *** fragatina has joined #openstack-ironic | 04:06 | |
| *** fragatin_ has joined #openstack-ironic | 04:07 | |
| *** fragatina has quit IRC | 04:10 | |
| *** fragatin_ has quit IRC | 04:13 | |
| *** namnh has quit IRC | 04:18 | |
| *** tuanla____ has quit IRC | 04:18 | |
| *** tiendc has quit IRC | 04:18 | |
| *** tuanla____ has joined #openstack-ironic | 04:18 | |
| *** tiendc has joined #openstack-ironic | 04:18 | |
| *** namnh has joined #openstack-ironic | 04:18 | |
| *** hemna_ has quit IRC | 04:26 | |
| *** pcaruana has joined #openstack-ironic | 04:31 | |
| *** linhnm has joined #openstack-ironic | 04:38 | |
| *** fragatina has joined #openstack-ironic | 04:41 | |
| *** fragatina has quit IRC | 04:58 | |
| *** linhnm has quit IRC | 05:04 | |
| *** marios has joined #openstack-ironic | 05:08 | |
| *** fragatina has joined #openstack-ironic | 05:09 | |
| *** mikal has joined #openstack-ironic | 05:12 | |
| *** mikal_ has quit IRC | 05:15 | |
| *** fragatina has quit IRC | 05:20 | |
| *** fragatina has joined #openstack-ironic | 05:26 | |
| *** linhnm has joined #openstack-ironic | 05:26 | |
| *** zshi has quit IRC | 05:27 | |
| *** zshi has joined #openstack-ironic | 05:28 | |
| *** fragatina has quit IRC | 05:31 | |
| *** mjura has joined #openstack-ironic | 05:41 | |
| *** Chandra has quit IRC | 05:41 | |
| *** jaypipes has quit IRC | 05:41 | |
| *** Chandra has joined #openstack-ironic | 05:41 | |
| *** jaypipes has joined #openstack-ironic | 05:41 | |
| *** masber has joined #openstack-ironic | 05:51 | |
| *** hshiina has quit IRC | 05:54 | |
| *** hshiina has joined #openstack-ironic | 05:55 | |
| *** linhnm has quit IRC | 05:56 | |
| *** fragatina has joined #openstack-ironic | 05:58 | |
| *** liuzz_ has quit IRC | 06:18 | |
| *** liuzz has joined #openstack-ironic | 06:18 | |
| *** linhnm has joined #openstack-ironic | 06:22 | |
| openstackgerrit | ShangXiao proposed openstack/ironic-ui master: Add release notes link to README https://review.openstack.org/568454 | 06:26 |
| *** rbartal has joined #openstack-ironic | 06:30 | |
| openstackgerrit | Pavlo Shchelokovskyy proposed openstack/ironic master: Mark xclarity password as secret https://review.openstack.org/567637 | 06:48 |
| *** trungnv has quit IRC | 06:54 | |
| *** phuongnh has quit IRC | 06:54 | |
| *** phuongnh has joined #openstack-ironic | 06:55 | |
| *** trungnv has joined #openstack-ironic | 06:55 | |
| *** namnh has quit IRC | 07:01 | |
| *** tuanla____ has quit IRC | 07:01 | |
| *** tuanla____ has joined #openstack-ironic | 07:01 | |
| *** namnh has joined #openstack-ironic | 07:01 | |
| *** fragatina has quit IRC | 07:03 | |
| *** ajya has joined #openstack-ironic | 07:06 | |
| *** rcernin has quit IRC | 07:13 | |
| *** tesseract has joined #openstack-ironic | 07:13 | |
| *** pcaruana has quit IRC | 07:17 | |
| *** pcaruana has joined #openstack-ironic | 07:17 | |
| *** pcaruana has quit IRC | 07:18 | |
| *** afazekas|pto is now known as afazekas | 07:18 | |
| *** pmannidi has quit IRC | 07:22 | |
| *** pcaruana has joined #openstack-ironic | 07:23 | |
| *** S4ren has joined #openstack-ironic | 07:31 | |
| openstackgerrit | Harald Jensås proposed openstack/ironic-inspector stable/queens: Call shutdown() on SIGTERM - controlled teardown https://review.openstack.org/563337 | 07:32 |
| *** S4ren has quit IRC | 07:32 | |
| openstackgerrit | Dao Cong Tien proposed openstack/ironic-python-agent master: Rescue bug: tinyipa fails to acquire IP in multitenant env https://review.openstack.org/566279 | 07:32 |
| olivierb- | Good morning everyone, during my investigation in fixing a GPT issue mentioned yesterday in this channel and following TheJulia's pointers and advices, I looked at the code in ironic-lib and ironic-python agent and have some questions for partitioning experts: | 07:34 |
| olivierb- | https://github.com/openstack/ironic-lib/blob/300e171546ed6832106596861bd3ff50fa061dce/ironic_lib/disk_utils.py#L761-L762 | 07:35 |
| olivierb- | https://github.com/openstack/ironic-python-agent/blob/0912a24b384518e402394e63fc1ad94579d38b76/ironic_python_agent/extensions/standby.py#L472-L479 | 07:35 |
| *** linhnm has quit IRC | 07:36 | |
| *** AlexeyAbashkin has joined #openstack-ironic | 07:36 | |
| openstackgerrit | Dao Cong Tien proposed openstack/ironic master: Update CI jobs for rescue mode https://review.openstack.org/528704 | 07:36 |
| olivierb- | doing a basic test on my problematic GPT installation I changed the code accordingly and was able to get a fully working and sound system but with 2 partitions, the second one being added by the fixed code described in the links above | 07:36 |
| openstackgerrit | Dao Cong Tien proposed openstack/ironic master: DNM: check rescue in multi-tenancy env https://review.openstack.org/566286 | 07:37 |
| olivierb- | however if I pnly run sgdisk -e /dev/vda I also get a working and sound GPT but this time with only 1 partition which is what I expect | 07:38 |
| olivierb- | my question is then any reason to really create this second partition ? I personnaly find it very confusing to get extra stuff I did not ask for on my provisioned system but again I am definitely no partitioning/GPT expert | 07:39 |
| hshiina | anupn, hi, it seems that i have hit the same issue now in my environment. | 07:41 |
| *** zhangfei has quit IRC | 07:41 | |
| hshiina | anupn, nodes are stuck in clean wait | 07:42 |
| openstackgerrit | Harald Jensås proposed openstack/ironic-inspector stable/queens: Raise KeyboardInterrupt on SIGTERM - Workaround https://review.openstack.org/563337 | 07:42 |
| openstackgerrit | Tuan Luong-Anh proposed openstack/ironic master: [WIP] Implement iRMC BIOS configuration https://review.openstack.org/534595 | 07:42 |
| hshiina | anupn, a node console displays: No bootable device | 07:43 |
| olivierb- | just realized that the sdgisk -e was called by _fix_gpt_structs so nevermind my questions above, will produce proper fix right now, sorry for the moise | 07:48 |
| olivierb- | noise | 07:48 |
| *** zhangfei has joined #openstack-ironic | 07:58 | |
| openstackgerrit | Vu Cong Tuan proposed openstack/ironic stable/queens: Update auth_uri option to www_authenticate_uri https://review.openstack.org/568501 | 07:59 |
| *** livelace has joined #openstack-ironic | 07:59 | |
| openstackgerrit | zenghui.shi proposed openstack/ironic master: BIOS Settings: add admin doc https://review.openstack.org/568504 | 08:01 |
| *** lucas-afk is now known as lucasagomes | 08:04 | |
| openstackgerrit | Merged openstack/python-ironicclient master: Stop double json decoding API error messages https://review.openstack.org/567836 | 08:04 |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic-inspector master: Add manage_boot parameter to introspection API https://review.openstack.org/316801 | 08:09 |
| *** dougsz has joined #openstack-ironic | 08:09 | |
| *** athomas has joined #openstack-ironic | 08:13 | |
| openstackgerrit | Tuan Luong-Anh proposed openstack/ironic master: [WIP] Implement iRMC BIOS configuration https://review.openstack.org/534595 | 08:18 |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic-inspector master: Add manage_boot parameter to introspection API https://review.openstack.org/316801 | 08:21 |
| *** hshiina has quit IRC | 08:22 | |
| *** mgoddard has joined #openstack-ironic | 08:22 | |
| openstackgerrit | Olivier Bourdon proposed openstack/ironic-lib master: Expose GPT partitioning fixing method https://review.openstack.org/568518 | 08:27 |
| openstackgerrit | Nguyen Van Trung proposed openstack/ironic master: Support raid configuration for BM via irmc driver https://review.openstack.org/512979 | 08:30 |
| *** baoli has joined #openstack-ironic | 08:38 | |
| *** priteau has joined #openstack-ironic | 08:38 | |
| *** MattMan has quit IRC | 08:41 | |
| *** MattMan has joined #openstack-ironic | 08:41 | |
| openstackgerrit | Olivier Bourdon proposed openstack/ironic-python-agent master: Fix GPT partitioning https://review.openstack.org/568524 | 08:42 |
| *** baoli has quit IRC | 08:43 | |
| *** derekh has joined #openstack-ironic | 08:45 | |
| *** e0ne has joined #openstack-ironic | 08:50 | |
| *** trown|outtypewww has quit IRC | 08:52 | |
| *** trown has joined #openstack-ironic | 08:53 | |
| openstackgerrit | Tuan Luong-Anh proposed openstack/ironic master: [WIP] Implement iRMC BIOS configuration https://review.openstack.org/534595 | 09:04 |
| openstackgerrit | Olivier Bourdon proposed openstack/ironic-lib master: Expose GPT partitioning fixing method https://review.openstack.org/568518 | 09:07 |
| *** ajya has quit IRC | 09:09 | |
| openstackgerrit | jiapei proposed openstack/ironic master: Add documentatin for XClarity Driver https://review.openstack.org/559960 | 09:10 |
| *** linhnm has joined #openstack-ironic | 09:11 | |
| *** ajya has joined #openstack-ironic | 09:15 | |
| *** phuongnh has quit IRC | 09:17 | |
| *** serlex has joined #openstack-ironic | 09:26 | |
| *** e0ne has quit IRC | 09:29 | |
| *** e0ne has joined #openstack-ironic | 09:33 | |
| openstackgerrit | Harald Jensås proposed openstack/ironic-inspector stable/pike: Raise KeyboardInterrupt on SIGTERM - Workaround https://review.openstack.org/563338 | 09:37 |
| *** e0ne has quit IRC | 09:38 | |
| *** sambetts|afk is now known as sambetts | 09:40 | |
| sambetts | Morning all | 09:40 |
| *** linhnm has quit IRC | 09:42 | |
| etingof | sambetts, \o | 09:45 |
| sambetts | hey etingof | 09:46 |
| yolanda_ | hi sambetts , we were asked to write some devstack tests for bios automation. But so far, i haven't seen support for cleanup steps in devstack plugin. Have i missed something, or is some feature that should need to be added? | 09:56 |
| *** e0ne has joined #openstack-ironic | 09:59 | |
| vdrok | good morning all | 10:05 |
| sambetts | yolanda_: the bios stuff currently runs as a manual cleaning step, if i remember correctly, that seems to me like a we could write a tempest test to orcastrate moving a node to managable, then trigging the cleaning process with the right steps | 10:05 |
| sambetts | yolanda_: but if there is any configuration needed in ironic.conf to enable it then we'll need to put that into devstack | 10:05 |
| etingof | vdrok, good morning ;) | 10:05 |
| vdrok | morning etingof | 10:06 |
| openstackgerrit | Merged openstack/bifrost master: upgrade setuptools before installing package requirements https://review.openstack.org/568147 | 10:06 |
| yolanda_ | sambetts, i was looking for some sample in devstack tests, to see if soem cleanup step has been tested, but i didn't find anything. So that's why i was wondering, if that's worth to write something more generic | 10:07 |
| yolanda_ | or we just write that specific test? | 10:07 |
| sambetts | yolanda_: I think currently all the cleaning steps get tested via automated cleaning, this is the first case I know of a running a manual cleaning step in the CI, I don't think we test the RAID ones | 10:08 |
| yolanda_ | ok i will take a look at how it can be done | 10:11 |
| *** namnh has quit IRC | 10:15 | |
| *** rh-jelabarre has joined #openstack-ironic | 10:17 | |
| *** tuanla_____ has joined #openstack-ironic | 10:18 | |
| *** tuanla____ has quit IRC | 10:19 | |
| *** tesseract is now known as info | 10:41 | |
| *** info is now known as tesseract | 10:41 | |
| *** tiendc has quit IRC | 10:51 | |
| openstackgerrit | yolanda.robla proposed openstack/ironic-tempest-plugin master: Add the ability to specify bios interface https://review.openstack.org/568551 | 10:57 |
| *** tuanla_____ has quit IRC | 11:00 | |
| openstackgerrit | yolanda.robla proposed openstack/ironic-tempest-plugin master: Add the ability to specify bios interface https://review.openstack.org/568551 | 11:02 |
| *** dtantsur|afk is now known as dtantsur | 11:14 | |
| dtantsur | morning ironic | 11:14 |
| sambetts | hey dtantsur | 11:15 |
| openstackgerrit | yolanda.robla proposed openstack/ironic-tempest-plugin master: Add the ability to specify bios interface https://review.openstack.org/568551 | 11:26 |
| *** jcoufal has joined #openstack-ironic | 11:26 | |
| *** zhangfei has quit IRC | 11:32 | |
| jroll | morning y'all | 11:47 |
| TheJulia | Good morning | 11:49 |
| *** dprince has joined #openstack-ironic | 11:49 | |
| *** ajya has quit IRC | 11:51 | |
| *** ajya has joined #openstack-ironic | 11:51 | |
| *** mjura has quit IRC | 11:51 | |
| sambetts | TheJulia: I'm building a fresh bifrost system atm, just got to enrolling the nodes and I'm getting a "TypeError: __init__() got an unexpected keyword argument 'token'" is this a known issue someones addressing? or should I dig deeper? | 11:55 |
| *** mjura has joined #openstack-ironic | 11:56 | |
| dtantsur | morning jroll, TheJulia, sambetts | 11:57 |
| TheJulia | sambetts: first I'm hearing of it. I think hamzy hit an issue a few days ago with setuptools needing to be the absolute latest version, but that is it | 11:57 |
| sambetts | TheJulia: this seems to be coming from keystoneauth | 11:57 |
| TheJulia | Yeah :( Always is | 11:57 |
| TheJulia | :( | 11:57 |
| sambetts | s/shade/requests/g | 11:58 |
| sambetts | ;) | 11:58 |
| TheJulia | yeah | 11:59 |
| openstackgerrit | Jim Rollenhagen proposed openstack/ironic stable/pike: Tear down console during unprovisioning https://review.openstack.org/568569 | 12:01 |
| openstackgerrit | Jim Rollenhagen proposed openstack/ironic stable/ocata: Tear down console during unprovisioning https://review.openstack.org/568570 | 12:01 |
| jroll | pike/ocata backports for that ^ | 12:01 |
| sambetts | TheJulia: where does the os_ironic ansible module come from, I can't seem to find the source | 12:16 |
| TheJulia | sambetts: it is in the ansible tree | 12:16 |
| sambetts | TheJulia: ah... | 12:16 |
| *** bfournie has joined #openstack-ironic | 12:20 | |
| *** gyankum has quit IRC | 12:25 | |
| *** anton has left #openstack-ironic | 12:35 | |
| *** rbudden has joined #openstack-ironic | 12:44 | |
| olivierb- | Hello everyone, could someone please help me fixing the code change I wrote https://review.openstack.org/568518 which is working properly AFAIK but is failing tests in CI | 12:47 |
| patchbot | patch 568518 - ironic-lib - Expose GPT partitioning fixing method | 12:47 |
| olivierb- | I may have missed some changes in tests to adapt to new exposed code but not very familiar with these tests and all the mocking around them | 12:48 |
| *** rh-jelabarre has quit IRC | 12:52 | |
| *** rh-jelabarre has joined #openstack-ironic | 12:54 | |
| sambetts | TheJulia: welll... this is a weird one, so it turns out you must make sure to unset OS_AUTH_TOKEN from your environment variables when you run the ansible modules now because otherwise keystoneauth will try to pass "token=<OS_AUTH_TOKEN env var>" into the constructor of the keystone auth plugin | 12:55 |
| sambetts | regardless of whether it takes a token or not | 12:55 |
| TheJulia | oh joy | 12:55 |
| sambetts | not really sure how to address that tbh | 12:56 |
| sambetts | unless we make all keystone auth plugins take **kwargs and then just take the ones they care about | 12:57 |
| *** jesusaur has quit IRC | 12:57 | |
| *** rloo has joined #openstack-ironic | 12:57 | |
| sambetts | but the even the error is very unfreindly | 12:57 |
| *** jiapei has quit IRC | 12:59 | |
| TheJulia | olivierb-: so I was thinking that you could just expose _fix_gpt_structs, and not additional logic. | 13:03 |
| rloo | good morning all, TheJulia, sambetts, dtantsur, olivierb-, jroll | 13:04 |
| olivierb- | TheJulia ok my bad will fix this accordingly thanks | 13:04 |
| dtantsur | hi rloo | 13:04 |
| olivierb- | good morning rloo | 13:04 |
| TheJulia | olivierb-: regardless, we likely need to fix gpt regardless if there is a config drive, but one step at a time :) | 13:04 |
| TheJulia | good morning rloo | 13:04 |
| TheJulia | replied in storyboard with my $0.02 | 13:05 |
| *** jesusaur has joined #openstack-ironic | 13:05 | |
| rloo | TheJulia: thanks, looking... | 13:05 |
| *** Goneri has joined #openstack-ironic | 13:06 | |
| rloo | dtantsur, jroll: wrt the power fault recovery work, I forgot to mention yesterday that I commented in the story: https://storyboard.openstack.org/#!/story/1596107#comment-81721 | 13:07 |
| olivierb- | TheJulia agreed however wouldn't the code in ironic-python-agent would also need to require a test to see if device is using GPT before applying _fix_gpt_structs ? in which case I'll also need to expose _is_disk_gpt_partitioned or am I misunderstanding ? | 13:08 |
| TheJulia | it would, but minimally intrusive seems to make sense since it has wrapping logic around gpt | 13:09 |
| rloo | TheJulia: so the faults spec mentioned that the nova-ironic-virt-driver would have to be updated to also take into consideration faults in addition to maintenance. We most likely won't go there, but another reason for adding a fault_type instead of maintenance_type field. | 13:09 |
| dtantsur | rloo: so tl;dr s/maintenance_type/fault_type/ and do not touch maintenance at all? | 13:09 |
| rloo | TheJulia: and then I was thinking later, why 'fault_type' instead of just 'fault'. | 13:09 |
| dtantsur | (for old API versions, I guess, it means maintenance = maintenance or fault_type) | 13:10 |
| TheJulia | rloo: kind of, except we would have to land that change and hold off on fault recovery for a cycle... or just update maintenance | 13:10 |
| rloo | dtantsur: no, would also need to touch maintenance -- as the code does already. we can't change that behaviour. at least not now. | 13:10 |
| TheJulia | as well that is | 13:10 |
| dtantsur | rloo: we can change it in new API versions | 13:10 |
| dtantsur | so, do you suggest to only rename the new field? | 13:10 |
| rloo | sorry, i guess i wasn't clear. i don't know if we're ready/want to talk about how to do more. for now, i am just suggesting renaming the field and NOT saving 'manual' in that field. | 13:11 |
| rloo | i think this 'little' change would make it easier to move forward when/if we want to. | 13:11 |
| olivierb- | TheJulia I am not sure to understand what you meant | 13:11 |
| dtantsur | rloo: I'm cool with that. | 13:11 |
| TheJulia | olivierb-: I'm heating something up to eat, give me a few minutes and I'll pull the link | 13:12 |
| rloo | dtantsur: sweet. also, do you think node.fault is sufficient? i am not sure now, that node.fault_type makes sense. but I'm not an operations person, don't know the lingo. | 13:12 |
| dtantsur | both make equal sense to me, so I guess the shorter the better | 13:12 |
| olivierb- | of course, no pb. Looking back at the ironic-python-agent code I fixed it seems to be pretty much GPT agnostic | 13:13 |
| rloo | dtantsur: ok. i think sort is better in case we want to support multiple faults; then we could add a node.faults :) | 13:13 |
| rloo | s/sort/short/ | 13:13 |
| * sambetts is just lurking, but I like the sounds of this | 13:14 | |
| TheJulia | olivierb-: oh, brain failure | 13:16 |
| olivierb- | nope, too many code eating ;-) | 13:16 |
| olivierb- | your brain is working very well, only early morning for you | 13:17 |
| TheJulia | olivierb-: it is not in ipa yet, but what I was thinking purely the fix structures since it is needed outside of configuration drive use as well | 13:19 |
| TheJulia | I've not had coffee yet either | 13:19 |
| olivierb- | will go for some myself now, take your time | 13:20 |
| rloo | sambetts: thx (I assume you mean the power fault recovery discussion above) | 13:26 |
| sambetts | rloo: yeah, having a .fault initally, and then expanding it to .faults in the future makes a lot of sense to me, it feels a lot like the faults info I see on my BMC if something goes wrong like a PSU failure etc | 13:27 |
| sambetts | rloo: and having it separate from maintenance makes sense to me too | 13:28 |
| rloo | sambetts: good to know! thx. | 13:28 |
| *** gyankum has joined #openstack-ironic | 13:28 | |
| *** mjturek has joined #openstack-ironic | 13:28 | |
| olivierb- | going through the failing tests code, I must admit that I do not understand why it is failing in case of my mods and why it was not before, seems like /dev/fake4 is replaced by UUID therefore the failure or I am misinterpreting the results ? | 13:31 |
| TheJulia | I think we're going to have to sync with maintenance to begin with and then eventually decouple it after a deprecation period since the two concepts are tied together | 13:32 |
| rloo | TheJulia: yes. IF we decide to decouple it etc -- i think that's a separate RFE | 13:32 |
| TheJulia | agreed | 13:32 |
| TheJulia | a separate evolution :) | 13:32 |
| * TheJulia wonders if hacking on the pxe interface and testing should only be done while intoxicated | 13:33 | |
| rloo | :) I added another comment to the story to reflect our discussion today. | 13:33 |
| * TheJulia is hacking on the pxe interface in another window and getting frustrated because of how complex the path can be | 13:34 | |
| TheJulia | olivierb-: thta is exactly what the test output indicates | 13:35 |
| * TheJulia pulls up that code again | 13:35 | |
| olivierb- | ?? | 13:36 |
| dtantsur | TheJulia: yep, at least one shot before even opening the editor | 13:36 |
| TheJulia | dtantsur: note taken! | 13:36 |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: Remove excessive usage of mock_the_extension_manager in unit tests - part 1 https://review.openstack.org/568585 | 13:37 |
| dtantsur | and this requires getting wasted first ^^^ | 13:37 |
| TheJulia | dtantsur: I was going to ask if that was what you were hitting because I decided to try and hack a pxe deploy interface last night and hit it.... | 13:38 |
| dtantsur | TheJulia: hah, yeah. tl;dr if you're not sure you need mock_the_extension_manager - you don't | 13:38 |
| dtantsur | and we cargo-culted it all over the place | 13:39 |
| TheJulia | Yeah, once I realized what it was doing I knew we didn't need it | 13:39 |
| dtantsur | there are very few places where it cannot be replaced by self.config(enabled_drivers=[something]) or removed at all | 13:39 |
| TheJulia | olivierb-: oh, I see what you did, you flipped the node_uuid/device order | 13:40 |
| TheJulia | the caller is device, node_uuid, your new method is node_uuid, device | 13:40 |
| *** milan_ has joined #openstack-ironic | 13:41 | |
| olivierb- | TheJulia thx let me check and correct this | 13:42 |
| openstackgerrit | Olivier Bourdon proposed openstack/ironic-lib master: Expose GPT partitioning fixing method https://review.openstack.org/568518 | 13:45 |
| *** cdearborn has joined #openstack-ironic | 13:45 | |
| *** fragatina has joined #openstack-ironic | 13:46 | |
| *** rh-jelabarre has quit IRC | 13:46 | |
| olivierb- | should be way better now, will wait for tests results many thanks again for catching my stupid mistake, proof is that tests are very usefull as they catched it | 13:46 |
| *** rh-jelabarre has joined #openstack-ironic | 13:49 | |
| *** baoli has joined #openstack-ironic | 13:51 | |
| *** cjloader has joined #openstack-ironic | 13:57 | |
| *** baha has joined #openstack-ironic | 13:58 | |
| TheJulia | :) | 14:14 |
| *** hemna_ has joined #openstack-ironic | 14:15 | |
| openstackgerrit | yolanda.robla proposed openstack/ironic master: Add Node BIOS support - REST API https://review.openstack.org/512579 | 14:16 |
| *** rajinir has joined #openstack-ironic | 14:20 | |
| *** rpioso|afk is now known as rpioso | 14:20 | |
| rpioso | Good morning | 14:21 |
| TheJulia | good morning rpioso | 14:24 |
| rpioso | TheJulia: :) | 14:25 |
| etingof | rpioso, \0 ;) | 14:25 |
| rpioso | etingof: o/ | 14:26 |
| * etingof gets quite frantic tracking down the lurking unittest mocks | 14:28 | |
| *** BernsO has joined #openstack-ironic | 14:35 | |
| *** trown is now known as trown|brb | 14:37 | |
| *** trown|brb is now known as trown | 14:41 | |
| *** ajya has quit IRC | 14:45 | |
| *** ajya has joined #openstack-ironic | 14:46 | |
| jroll | rloo: commented in storyboard, but yeah, I'm good with this too, thanks! | 14:50 |
| rloo | thx jroll! | 14:50 |
| dtantsur | rloo: do you want to update the spec to save some roundtrips between us and the contributor? | 14:51 |
| rloo | dtantsur: sure, I can do that. | 14:51 |
| rloo | dtantsur: will do it today | 14:52 |
| dtantsur | thanks! | 14:52 |
| dtantsur | then we'll just ask the contributor to update the code patches accordingly | 14:53 |
| rloo | dtantsur: makes sense :) | 14:53 |
| *** r-daneel has joined #openstack-ironic | 14:57 | |
| *** mjura has quit IRC | 15:00 | |
| *** bdodd has joined #openstack-ironic | 15:06 | |
| *** rbartal has quit IRC | 15:11 | |
| *** kaifeng has joined #openstack-ironic | 15:12 | |
| kaifeng | morning ironicers | 15:13 |
| openstackgerrit | yolanda.robla proposed openstack/ironic-tempest-plugin master: Add manual clean step ironic standalone test https://review.openstack.org/568616 | 15:13 |
| kaifeng | wow i saw so many messages today for the power fault feature | 15:13 |
| *** serlex has quit IRC | 15:14 | |
| rloo | hey kaifeng! Yeah, I'm updating the spec now. I hope you are OK with it all. | 15:14 |
| kaifeng | hi rloo, i haven't fully read your replies yet, just got back to computer | 15:15 |
| rloo | kaifeng: ok | 15:15 |
| kaifeng | but i feel your concern is related to the naming? | 15:15 |
| rloo | kaifeng: yes, mostly. i want to abstract 'fault' from 'maintenance'; they are two diff concepts. | 15:16 |
| kaifeng | i see you are worried about the manual value | 15:16 |
| rloo | kaifeng: right, because it ties that new field with the maintenance stuff. which means if we ever want to separate them, it will be harder to do. | 15:17 |
| kaifeng | so you want a fault doesn't tie with maintenance, which means we may have a node not in maintenance, but has a fault value there | 15:18 |
| rloo | kaifeng: i have one question for you. just reading/updating the spec now. were you planning on adding a 'dbsync online_data_migration' to set this new field, based on maintenance_reason? the spec sez 'manual', but that is wrong. | 15:18 |
| rloo | kaifeng: no, for your work, when there is a fault, ironic still sets maintenance & maintenance_reason. just like it does now. | 15:19 |
| kaifeng | yes, planned so, let me dig some history | 15:19 |
| rloo | kaifeng: it is about setting things up so that in the future, we *could* change it so it doesn't set maintenance/reason. | 15:19 |
| *** r-daneel has quit IRC | 15:19 | |
| *** r-daneel_ has joined #openstack-ironic | 15:19 | |
| rloo | kaifeng: wrt the db migration -- the only way to get it right is to parse the maintenance_reason string. argh. but we can't do that easily if it is translated. | 15:19 |
| kaifeng | rloo: of course, power fault is tied with maintenace | 15:19 |
| *** links has quit IRC | 15:21 | |
| kaifeng | https://review.openstack.org/#/c/555708/ | 15:21 |
| patchbot | patch 555708 - ironic - Power fault recovery: db and rpc implementation | 15:21 |
| *** r-daneel_ is now known as r-daneel | 15:22 | |
| kaifeng | at patch set 1, dmitry suggests do the migration in alembic script | 15:22 |
| kaifeng | https://review.openstack.org/#/c/555708/1/ironic/db/sqlalchemy/alembic/versions/fb3f10dd262e_add_maintenance_type_to_node_table.py@19 | 15:23 |
| patchbot | patch 555708 - ironic - Power fault recovery: db and rpc implementation | 15:23 |
| kaifeng | and after several updates, Hironori points out that doesn't work for rolling upgrade | 15:24 |
| NobodyCam | Good Morning Ironic'ers | 15:24 |
| NobodyCam | Happy Not Monday | 15:24 |
| kaifeng | so i removed it, the migration is done at rpc layer | 15:24 |
| rloo | kaifeng: 'unknown' works for me; 'manual' doesn't. and we should do these in the dbsync online_data_migration. | 15:24 |
| kaifeng | rloo: i wonder what we can do with 'unknown' type? | 15:25 |
| rloo | kaifeng: so, either 'unknown', None, or we do a best-guess at what it might be by looking at the maintenance_reason value (if it hasn't been translated). | 15:26 |
| rloo | kaifeng: 'unknown' is more accurate that None, for nodes that are already in maintenance. | 15:26 |
| rloo | kaifeng: cuz we don't know | 15:27 |
| rloo | kaifeng: i will suggest something in the spec | 15:27 |
| kaifeng | rloo: ok, i will take look after you updated the spec | 15:28 |
| kaifeng | rloo: thanks for helping | 15:28 |
| rloo | kaifeng: thank YOU for doing the work! | 15:28 |
| dtantsur | maybe just leave it as None? | 15:28 |
| dtantsur | aka NULL | 15:29 |
| kaifeng | ah, we consider maintenance_type is a classification to maintenance field | 15:30 |
| kaifeng | so from the beginning, we assume maintenance_type is None when maintenance is false, right? | 15:30 |
| kaifeng | otherwise, we needn't bother with the compatibility and data migratio | 15:31 |
| kaifeng | migration | 15:31 |
| dtantsur | yeah, I guess we keep them locked together still | 15:31 |
| dtantsur | so maintenance=True and fault=None means "manual maintenance" (or upgraded) | 15:32 |
| dtantsur | maintenance=True and fault!=None means "power/cleaning/etc fault" | 15:32 |
| dtantsur | right rloo ^^^? | 15:32 |
| kaifeng | or unknown? | 15:32 |
| dtantsur | unknown = upgraded | 15:32 |
| rloo | dtantsur: true. although i am open to us adding code to try to guess the fault from the maintenance_reason string. I mention it in the spec. cuz... how will someone get their node 'hooked' into the fault stuff so that they can do power fault recovery. | 15:33 |
| dtantsur | rloo: it's too fragile IMO, esp. if we consider i18n | 15:33 |
| sambetts | why do we need to keep them locked together now? e.g. can't we just have faults be a new thing? and the power sync state code for exmaple do if maintenance or powerfault in node.fault: pass | 15:33 |
| dtantsur | sambetts: to simplify API changes (think about old versions that you have to keep compatible) | 15:34 |
| rloo | dtantsur: i was thinking that we most likely do not have many clusters with i18n. a simple string match or whatever -- if it matches, great, if not, use 'unknown' or None. | 15:34 |
| sambetts | in the old API versions we could just do, if faults != None then maitenance=True | 15:34 |
| dtantsur | rloo: I recall some bugs with Japanese locale popping up | 15:35 |
| dtantsur | so at least we have people using it | 15:35 |
| rloo | sambetts: we could but it is a bigger piece of work, we'd have to agree to a new spec. and even if we did, we'd probably need a deprecation period. we'd need to update nova-ironic-virt driver. | 15:35 |
| rloo | dtantsur: actually, it'd be easy to see which languages have xlations, it is in ironic/locale or whatever directory. | 15:35 |
| dtantsur | right | 15:35 |
| dtantsur | sambetts, rloo, it's a good point, we can do it, but we don't have to do it in the same API version | 15:36 |
| *** gyee has joined #openstack-ironic | 15:39 | |
| mgoddard | we are about to perform a new deployment, and trying to learn lessons from past deployments. In particular, networking between bare metal compute and controllers/API endpoints can get hairy in when there are multiple networks involved for internal API, provisioning and cleaning. Ironic uses a single IP to configure TFTP, HTTP(iPXE) and API hosts. If provisioning net != cleaning net != API net, this implies some IP routing exists between these | 15:46 |
| mgoddard | networks. Could anyone provide information on approaches they've seen used for this in the past, and their relative merits? | 15:46 |
| mgoddard | e.g. do you use neutron routers, or hardware, how do you restrict access between these networks? | 15:48 |
| *** hemna_ has quit IRC | 15:49 | |
| openstackgerrit | Ruby Loo proposed openstack/ironic-specs master: Use node.fault field for power fault recovery work https://review.openstack.org/568627 | 15:49 |
| *** hemna_ has joined #openstack-ironic | 15:50 | |
| rloo | kaifeng, dtantsur: ^^ | 15:50 |
| dtantsur | mgoddard: we're using flat networking but IIRC we set up routing between the provisioning network and the API network | 15:51 |
| dtantsur | bfournie: do you remember details ^^? | 15:51 |
| dtantsur | thanks rloo! | 15:51 |
| rloo | dtantsur: yw. thank you for being patient with me :) | 15:51 |
| dtantsur | heh, it's not too hard :) | 15:52 |
| mgoddard | thanks dtantsur. is that a HW router or neutron? | 15:52 |
| sambetts | mgoddard: we proposed a solution in one of our deployments where we ran three networks, a network for the conductors "conductor net" which is where the TFTP /http server and API server run for internal use, then a cleaning network, and a provisioning network, | 15:52 |
| mgoddard | sambetts: interesting: I had this idea myself this morning, but we already have more networks than I can count so wasn't sure about it | 15:53 |
| dtantsur | mgoddard: let's wait for bfournie, I guess. I realized I don't know | 15:53 |
| mgoddard | interesting to hear someone else has considered it | 15:53 |
| mgoddard | sambetts: when you say proposed, was it ever implemented? | 15:53 |
| kaifeng | rloo: thanks, added to my list :) | 15:54 |
| rloo | kaifeng: thx! | 15:54 |
| sambetts | mgoddard: unforunatly the project was cut short before we could implement it, so at the time we were still running provisioning==cleaning network (API network was separate though) | 15:54 |
| sambetts | mgoddard: we ran an ironic API specifically for the provisioning network | 15:54 |
| sambetts | and hardwired that and the conductors into the network | 15:55 |
| sambetts | the provisioning/cleaning network had no external router | 15:55 |
| sambetts | well no router at all | 15:55 |
| mgoddard | sambetts: and that API listened on this third network too? | 15:55 |
| anupn | Hi Folks, how can I know what network interface the DHCP server setup with Devstack is using for sending out DHCPOFFER packets during PXE boot? is it brbm? | 15:55 |
| TheJulia | anupn: it is a tap that is attached by neutron likely to brbm | 15:56 |
| sambetts | mgoddard: in the proposed implementation yeah, the three networks were completely isolated from each other | 15:57 |
| sambetts | not from each other, from the outside world | 15:57 |
| mgoddard | I see | 15:57 |
| dtantsur | rloo: see inline, I think I know what to do with upgrades | 15:57 |
| anupn | TheJulia: I see. Somehow the VM that I am trying to provision is sending out BOOTP request, and another machine where Devstack is running is receiving BOOTP request from the VM on the underlying physical interface, but not hearing the DHCP offer reply | 15:58 |
| anupn | TheJulia: Do we specify the tap name anywhere in neutron or Ironic conf? | 15:58 |
| mgoddard | sambetts: so you had two two legged neutron routers to get from cleaning/provisioning to network3? | 15:59 |
| anupn | TheJulia: Hmm i see there is a tap named "brbm-tap" | 15:59 |
| rloo | dtantsur: so we need to look at 3 diff faults, power, clean & rescue-abort. | 15:59 |
| sambetts | mgoddard: yeah, and then security group rules to lock down the communication between the the networks to just the traffic types needed by provisioning | 15:59 |
| sambetts | cleaning | 15:59 |
| rloo | dtantsur: but i think what you propose makes sense. | 16:00 |
| mgoddard | ok, got it | 16:00 |
| sambetts | mgoddard: could maybe be done with one neutron router if the rules are written correctly | 16:00 |
| dtantsur | rloo: we're not going to do any automation around rescue-abort and clean, right? so maybe we can ignore them. | 16:00 |
| mgoddard | thanks for explaining :) | 16:01 |
| *** dprince has quit IRC | 16:01 | |
| TheJulia | anupn: no, it is dynamicly handled by neutron from a namespace operated by the neutron-dhcp-agent | 16:01 |
| rloo | dtantsur: nah, we shouldn't ignore, let's do it 'right' if we're going to do it. | 16:01 |
| rloo | dtantsur: just a bit more code :) | 16:01 |
| dtantsur | a bit, hmm :) | 16:01 |
| mgoddard | sambetts: in hindsight, was it a good solution? | 16:01 |
| rloo | ha ha, 'only code'! | 16:01 |
| mgoddard | sambetts: I'd be a bit concerned about the router being a performance bottleneck | 16:03 |
| anupn | TheJulia: umm wait but then the problem I see right now is no connection between the interface where DPCP sends DHCPOFFER packet and the underlying physical interface | 16:03 |
| TheJulia | anupn: so... sounds like misconfiguration, is brbm bound to the physical interface? | 16:03 |
| sambetts | mgoddard: in my opinion yes, it makes the ironic services pretty much as isolated as possible from themselves and from the API network, it is a little over kill though, I think having cleaning==provisioning isn't too bad, as long as that network is isolated from the outside world | 16:04 |
| anupn | TheJulia: No, I haven't manually added brbm to any physical interface, it is as setup by devstack | 16:05 |
| anupn | TheJulia: here is the tcpdump on the underlying physical interface of the host where devstack is running http://paste.openstack.org/show/721022/ | 16:05 |
| anupn | TheJulia: you can see the BOOTP request coming in from the MAC address but on the same interface i don't see the DHCP reply going | 16:06 |
| TheJulia | anupn: without the two being connected, the neutron dhcp server can never get the request | 16:07 |
| bfournie | mgoddard: we also have a setup with provisioning and cleaning network on same network, this was a separate isolated network in tripleo terminology, and created a neutron router to access, with security rules set up. some notes here - http://paste.openstack.org/show/721023/ | 16:07 |
| *** trown is now known as trown|lunch | 16:08 | |
| mgoddard | sambetts: did the design include inspector? separate inspection network? did the inspector DHCP server listen directly on this network? | 16:09 |
| *** lucasagomes is now known as lucas-afk | 16:09 | |
| anupn | TheJulia: So I need to connect brbm to the underlying physical interface, right? | 16:09 |
| anupn | TheJulia: let me do that and see | 16:10 |
| mgoddard | good notes bfournie, thanks for sharing | 16:10 |
| bfournie | mgoddard: no problem, btw our design did not have inspector as we only have inspector running in undercloud | 16:11 |
| sambetts | mgoddard: we actually ran inspection on a separate VLAN completely, one that wasn't managed by neutron at all, we would manually put the inspection network on the switch ports when we racked a new server / have the switch ports default to the inspection network when unconfigured, and then once inspected then node would move into cleaning causing neutron to program the cleaning network onto the | 16:11 |
| sambetts | ports stomping on the inspection VLAN | 16:11 |
| sambetts | mgoddard: we did this because at inspection time we didn't have the local_link_connection infomation avaiable to program the inspection VLAN onto switch | 16:12 |
| sambetts | its a chicken and egg you see | 16:12 |
| mgoddard | sambetts: yeah, we have some automation in kayobe for putting nodes on the inspection network | 16:13 |
| sambetts | mgoddard: we were using full autodiscovery in our deployment, with the idea that you could roll in a rack and inspector would auto discover and enrol all the nodes for us, so we were working on the princple of not know what node was connected to what port | 16:15 |
| sambetts | basically roll in a rack, set the whole switch to the inspection vlan, power on the nodes, watch them pop into ironic | 16:16 |
| sambetts | man... I miss working on that project.. | 16:16 |
| openstackgerrit | Ilya Etingof proposed openstack/sushy-tools master: Add unittests for OpenStack nova driver https://review.openstack.org/568636 | 16:16 |
| mgoddard | sambetts: http://www.stackhpc.com/ironic-idrac-ztp.html | 16:17 |
| mgoddard | yeah that's pretty much our approach. The lack of inspection network integration with neutron gets annoying if you want to reinspect | 16:17 |
| sambetts | yeah reinispect is a probelm, I have talked to dtantsur about splitting discovery and inspection for a while, where discovery would find nodes and their ports etc like it does today, but then inspection would be in a neutron network and be retryable | 16:19 |
| sambetts | but never had the time to work on it | 16:19 |
| mgoddard | how big a feature is it? we have a summer intern joining soon with time allotted for inspector | 16:20 |
| *** tesseract has quit IRC | 16:21 | |
| openstackgerrit | Harald Jensås proposed openstack/ironic-inspector stable/pike: Raise KeyboardInterrupt on SIGTERM - Workaround https://review.openstack.org/563338 | 16:22 |
| sambetts | mgoddard: difficult to tell, but it would definatly involve adding additional logic into the network interfaces for "(dis)connect_inspection_network" functions, its also would involve a fundimental change in how inspection nodes currently PXE boot, i.e. default DHCP, because you would be guarenteed to know the mac address of at least one port, but we already have logic in ironic for that, also | 16:27 |
| sambetts | it ties in with the inspection boot management stuff that dtantsur is/has been working on | 16:27 |
| dtantsur | tl;dr A LOT OF work | 16:28 |
| olivierb- | some tempest tests for https://review.openstack.org/#/c/568518/ have failed, should I relaunch/is this known ? | 16:28 |
| patchbot | patch 568518 - ironic-lib - Expose GPT partitioning fixing method | 16:28 |
| mgoddard | I can imagine that getting out of hand once you start to pick it apart :) | 16:29 |
| mgoddard | probably > 1 intern summer | 16:29 |
| openstackgerrit | Ruby Loo proposed openstack/ironic-specs master: Use node.fault field for power fault recovery work https://review.openstack.org/568627 | 16:29 |
| sambetts | unfortunatly yes, it could probably be somewhat broken down | 16:30 |
| sambetts | e.g. we could add the network interface functions, and call into them from the existing inspector interface to plumb the switches if and only if we know the network info for the ports | 16:31 |
| sambetts | else give up and work as we do today | 16:31 |
| sambetts | and if we state you must turn DHCP off in the neutron inspection network, then it won't even conflict with the inspector DHCP | 16:32 |
| sambetts | the ports will just have an IP address that means nothing in neutron | 16:32 |
| sambetts | (actually ports in neutron can be create with no IP address now, so that could also be a thing) | 16:32 |
| *** dtantsur is now known as dtantsur|afk | 16:33 | |
| dtantsur|afk | see you tomorrow | 16:33 |
| sambetts | o/ dtantsur|afk | 16:33 |
| *** pcaruana has quit IRC | 16:33 | |
| TheJulia | goodnight dtantsur|afk | 16:33 |
| *** BernsO has quit IRC | 16:34 | |
| *** e0ne has quit IRC | 16:35 | |
| *** marios has quit IRC | 16:41 | |
| mgoddard | ok, lots of good context there sambetts. I'll put it on the list of candidates | 16:42 |
| mgoddard | thanks for all the great info you've provided sambetts, bfournie, dtantsur|afk | 16:42 |
| sambetts | mgoddard: no problem :) | 16:44 |
| *** baoli has quit IRC | 16:50 | |
| *** dprince has joined #openstack-ironic | 16:56 | |
| *** dougsz has quit IRC | 16:58 | |
| *** derekh has quit IRC | 17:00 | |
| *** exodusftw has quit IRC | 17:00 | |
| *** kaifeng has quit IRC | 17:02 | |
| *** mgoddard has quit IRC | 17:04 | |
| *** baoli has joined #openstack-ironic | 17:05 | |
| *** yolanda_ has quit IRC | 17:09 | |
| anupn | TheJulia: I connected brbm to the underlying physical interface but don't see DHCP offer still going out Here is the tcpdump on brbm http://paste.openstack.org/show/721030/ | 17:18 |
| anupn | TheJulia: the BOOTP request happens to come in on brbm, but no DHCP offer reply | 17:19 |
| anupn | TheJulia: do i have to explicitly configure DHCP on one of the interface through /etc/network/interfaces? | 17:20 |
| *** milan_ has quit IRC | 17:20 | |
| *** yolanda_ has joined #openstack-ironic | 17:20 | |
| TheJulia | anupn: from with-in the neutron namespace, do you see the dhcp requests? | 17:21 |
| anupn | TheJulia: how to see that? | 17:22 |
| anupn | TheJulia: inside the logs of q-dhcp? | 17:22 |
| *** trown|lunch is now known as trown | 17:22 | |
| TheJulia | anupn: generally your logging should log if a request is received, but if you match the q-dhcp to the network namespace `ip netns list`, you can do `ip netns exec $namespace tcpdump -i namespace_interface -n`... you may want to do `ip netns exec $namespace ip link` to look at the interfaces in the namespace | 17:23 |
| *** mjturek has quit IRC | 17:24 | |
| *** mgoddard has joined #openstack-ironic | 17:25 | |
| * anupn checks the q-dhcp log and netns list | 17:25 | |
| *** AlexeyAbashkin has quit IRC | 17:25 | |
| *** baha has quit IRC | 17:27 | |
| anupn | TheJulia: for 'qdhcp-d2c109c9-ef91-4f28-9ec2-25494f413a22' I see two interfaces; one is loopback and other is tap6ee8763f-71 with status UNKNOWN | 17:30 |
| anupn | TheJulia: should the status be Unknown of that interface? | 17:30 |
| anupn | I will do tcpdump anyways on that interface, and see if DHCP requests are coming in | 17:30 |
| *** akhilaki has joined #openstack-ironic | 17:32 | |
| TheJulia | I think status unknown is fine since it is all in the kernel | 17:33 |
| *** ElCoyote_ has joined #openstack-ironic | 17:35 | |
| *** BernsO has joined #openstack-ironic | 17:35 | |
| *** BernsO has quit IRC | 17:38 | |
| *** e0ne has joined #openstack-ironic | 17:39 | |
| anupn | TheJulia: Oh okay, I don't see anything coming on that tap interface upon running tcpdump | 17:40 |
| anupn | TheJulia: " sudo ip netns exec qdhcp-d2c109c9-ef91-4f28-9ec2-25494f413a22 tcpdump -i tap6ee8763f-71 -vvv -n " | 17:41 |
| *** priteau has quit IRC | 17:52 | |
| *** sambetts is now known as sambetts|afk | 17:54 | |
| sambetts|afk | night all | 17:54 |
| *** gyankum has quit IRC | 17:55 | |
| TheJulia | anupn: so something is not hooked up in ovs, if you work backwards in ovs, that namespace interface should be hooked into ovs ultimately to brbm | 18:00 |
| *** links has joined #openstack-ironic | 18:01 | |
| anupn | TheJulia: Hmm, when i run "ovs-vsctl show", I see the tap6ee8763f-71 under br-int bridge with a type 'internal'. Is that how it should be? | 18:04 |
| anupn | TheJulia: or should that tap interface be under brbm? any idea | 18:04 |
| TheJulia | I really can't tell you off the top of my head since it is so dependent upon neutron configuration, and what network you set/used. https://docs.openstack.org/ironic/latest/contributor/ironic-multitenant-networking.html | 18:07 |
| anupn | TheJulia, hmm yeah true. Btw under the namespaces i.e. "ip netns list" I was seeing qdhcp and qrouter, I should be checking qdhcp right and not qrouter? | 18:10 |
| TheJulia | qrouter just passes packets, qdhcp is what dnsmasq attaches to | 18:11 |
| anupn | TheJulia: alright, thanks. | 18:12 |
| anupn | TheJulia: I am just looking at the link you pasted above and the local.conf used in that is different than mine | 18:13 |
| anupn | one is I haven't specified OVS_PHYSICAL_BRIDGE=brbm option inside local.conf | 18:14 |
| *** baha has joined #openstack-ironic | 18:14 | |
| anupn | but I saw inside devstack/lib/ironic that IRONIC_VM_NETWORK_BRIDGE is brbm so I assumed that is where DHCP offers should come | 18:14 |
| *** e0ne has quit IRC | 18:20 | |
| *** dprince has quit IRC | 18:30 | |
| *** dprince has joined #openstack-ironic | 18:30 | |
| *** e0ne has joined #openstack-ironic | 18:35 | |
| *** mgoddard has quit IRC | 18:40 | |
| anupn | TheJulia: I see the qdhcp namespace is associated with private network and the subnets are in the network 10.0.0.0/24. So DHCP server is likely be using this network, I am wondering will it be able to talk on brbm which is not in that network? | 18:43 |
| *** baoli has quit IRC | 18:43 | |
| *** baoli has joined #openstack-ironic | 18:44 | |
| *** baoli has quit IRC | 18:44 | |
| *** baoli has joined #openstack-ironic | 18:44 | |
| anupn | TheJulia: yes I see the tap interface attached in qdhcp namespace is carrying 10.0.0.2 IP address | 18:46 |
| *** ElCoyote_ has quit IRC | 18:47 | |
| *** e0ne has quit IRC | 19:04 | |
| *** e0ne has joined #openstack-ironic | 19:07 | |
| gyee | TheJulia, I've commented on https://review.openstack.org/#/c/566733. Sorry I haven't had a chance to do any more testing. | 19:07 |
| patchbot | patch 566733 - diskimage-builder - Add simple iscsi-boot element | 19:07 |
| *** r-daneel has quit IRC | 19:18 | |
| *** links has quit IRC | 19:20 | |
| *** milan has joined #openstack-ironic | 19:23 | |
| *** hshiina|afk has quit IRC | 19:23 | |
| *** hshiina has joined #openstack-ironic | 19:23 | |
| *** r-daneel has joined #openstack-ironic | 19:35 | |
| TheJulia | gyee: no worries, everyone is busy :) | 19:39 |
| TheJulia | anupn: brbm is like an ethernet cable in a sense, you can have multiple logical and differently addressed networks coexist and cross over that cable | 19:40 |
| anupn | TheJulia: Yes i agree, but the network where DHCP is running on a network that is not going over brbm cable | 19:49 |
| TheJulia | so the exact same issue you had earlier, brbm is not attached to whatever network your attempting to use | 19:51 |
| *** jtomasek has quit IRC | 19:51 | |
| *** cjloader_ has joined #openstack-ironic | 20:00 | |
| *** cjloader has quit IRC | 20:00 | |
| *** cjloader_ has quit IRC | 20:02 | |
| *** cjloader has joined #openstack-ironic | 20:02 | |
| *** e0ne has quit IRC | 20:02 | |
| *** ajya has quit IRC | 20:07 | |
| *** cjloader has quit IRC | 20:13 | |
| *** cjloader has joined #openstack-ironic | 20:14 | |
| *** fragatina has quit IRC | 20:20 | |
| *** baoli has quit IRC | 20:20 | |
| *** baha has quit IRC | 20:20 | |
| *** fragatina has joined #openstack-ironic | 20:20 | |
| anupn | TheJulia: yes brbm is attached to the underlying host, but somehow the tap interface where dhcp is running is not connecting to the brbm. I am seeing if there are options to tell neutron to run DHCP over brbm | 20:24 |
| anupn | somewhere in configuration files | 20:25 |
| *** e0ne has joined #openstack-ironic | 20:35 | |
| *** baha has joined #openstack-ironic | 20:36 | |
| *** r-daneel_ has joined #openstack-ironic | 20:36 | |
| *** mjturek has joined #openstack-ironic | 20:36 | |
| TheJulia | anupn: have you considered reading through the devstack plugin to see how it is all attached and how settings are used in that regard? | 20:36 |
| *** r-daneel has quit IRC | 20:37 | |
| *** r-daneel_ is now known as r-daneel | 20:37 | |
| anupn | TheJulia: I went a little bit over it to see is brbm specified anywhere or any other network interface name | 20:37 |
| anupn | I see IRONIC_VM_NETWORK_BRIDGE=brbm | 20:38 |
| olivierb- | could someone please have a look at CI tempest tests failure for https://review.openstack.org/#/c/568518/ please ? | 20:38 |
| patchbot | patch 568518 - ironic-lib - Expose GPT partitioning fixing method | 20:38 |
| anupn | and that put me again in another question of does ML2 plugin connects the tap interface where DHCP server is running to brbm, or brbm is just used for internal VM network communication | 20:39 |
| olivierb- | I relaunched the CI tests but it keeps failing and I can not understand why | 20:39 |
| olivierb- | may be it is way too late for my ppor brain | 20:39 |
| *** e0ne has quit IRC | 20:40 | |
| *** baoli has joined #openstack-ironic | 20:41 | |
| TheJulia | anupn: yes, and that is the default for the integration when we perform vm based testing, realistically that should be your physical network bridge if you want this to work. Alternatively it is going to have to be explicitly attached inside ovs via commands. The reason I recommend walking through the devstack plugin is so you can see how it gets assembled in the various scenarios we support with the plugin. | 20:47 |
| TheJulia | anupn: ml2 plugin would in essence attach a segmentation id to a port. That would match the same segmentation id neutron is using for a network | 20:49 |
| *** baha has quit IRC | 20:50 | |
| jroll | olivierb-: I am trying but loading these logs is slowwwwwww | 20:50 |
| *** mjturek has quit IRC | 20:50 | |
| *** cjloader has quit IRC | 20:50 | |
| olivierb- | yep, finally got hold of them after same slowness | 20:50 |
| olivierb- | but could not grab what was wrong | 20:51 |
| anupn | TheJulia: I see good to know. Yes I am reading and understanding the devstack plugin | 20:51 |
| olivierb- | thanks for your time and help jroll | 20:51 |
| jroll | don't thank me yet :) | 20:51 |
| anupn | TheJulia: Also other thing i noticed is I should specify OVS_PHYSICAL_BRIDGE in local.conf | 20:51 |
| olivierb- | even tried the other logs ara-reports which segment failures but did not manage to have them completed | 20:52 |
| jroll | I wonder if it just timed out | 20:52 |
| anupn | as pointed in the document that you pointed https://docs.openstack.org/ironic/pike/contributor/ironic-multitenant-networking.html | 20:53 |
| jroll | must have | 20:53 |
| TheJulia | olivierb-: that looks really broken | 20:53 |
| jroll | all I know is some of these logs are breaking my browser :| | 20:53 |
| TheJulia | heh | 20:53 |
| TheJulia | it looks like in one case tempest failed go grok an api response | 20:54 |
| jroll | only 2.2M, wtf chrome | 20:54 |
| TheJulia | in another, instance_info.image_source was not found and failed a api validation call | 20:54 |
| TheJulia | 2.9MB uncompressed text | 20:54 |
| TheJulia | in gzip... say 10M? | 20:55 |
| TheJulia | :) | 20:55 |
| jroll | ya | 20:55 |
| jroll | well, size warning, but this implies a timeout to me http://logs.openstack.org/18/568518/3/check/ironic-lib-tempest-partition-agent_ipmitool/286695d/logs/tempest.txt.gz | 20:55 |
| jroll | and this mostly looks like a refactor | 20:56 |
| jroll | so idk why it'd be so much slower | 20:56 |
| *** cjloader has joined #openstack-ironic | 20:56 | |
| * jroll calls shenanigans | 20:56 | |
| TheJulia | that log is still loading | 20:58 |
| TheJulia | it has been over two minutes | 20:58 |
| * jroll goes to wget to confirm | 20:58 | |
| jroll | in my browser it cuts off during the tempest run | 20:59 |
| jroll | also your estimate was waaaaay low apparently :) | 20:59 |
| jroll | 100M and counting | 20:59 |
| TheJulia | well, it is dynamic based on content, so lots of repeating tokens.... | 20:59 |
| TheJulia | sweet! | 20:59 |
| jroll | ah, 106M | 20:59 |
| TheJulia | wow | 21:00 |
| jroll | but yeah, appears to confirm ssh works, then a few api calls, and end | 21:00 |
| jroll | appears to be consistently doing this on the same jobs though | 21:01 |
| jroll | (sample size of 2) | 21:01 |
| jroll | have to go for now, will check back later | 21:02 |
| *** trown is now known as trown|outtypewww | 21:05 | |
| TheJulia | olivierb-: it does look like it is hanging in and timing out, but why doesn't make sense :\ I've issued a recheck on another unrelated patch in ironic-lib just to assess the state of CI | 21:23 |
| *** racedo_ has joined #openstack-ironic | 21:41 | |
| *** racedo has quit IRC | 21:42 | |
| *** jcoufal has quit IRC | 21:44 | |
| *** hemna_ has quit IRC | 21:47 | |
| *** cjloader has quit IRC | 21:52 | |
| *** Goneri has quit IRC | 21:53 | |
| *** dprince has quit IRC | 21:55 | |
| *** rcernin has joined #openstack-ironic | 22:00 | |
| *** rbudden has quit IRC | 22:08 | |
| *** hemna_ has joined #openstack-ironic | 22:08 | |
| *** cdearborn has quit IRC | 22:16 | |
| *** livelace has quit IRC | 22:18 | |
| *** rpioso is now known as rpioso|afk | 22:28 | |
| *** baoli has quit IRC | 22:36 | |
| *** baoli has joined #openstack-ironic | 22:37 | |
| *** bfournie has quit IRC | 22:37 | |
| *** pmannidi has joined #openstack-ironic | 22:40 | |
| *** baoli has quit IRC | 22:42 | |
| *** rbudden has joined #openstack-ironic | 22:52 | |
| anupn | TheJulia: I found by researching that there are few options that we can set to tell Neutron to consider a specific network as provisioning network, and then that will run DHCP server on that network | 23:16 |
| anupn | TheJulia: this pointer helped me to understand better https://docs.openstack.org/ironic/pike/contributor/ironic-multitenant-networking.html | 23:17 |
| anupn | TheJulia: thanks for your inputs :) I feel good to debug to this level | 23:17 |
| * anupn thinks he will get few drinks of rum tonight ;) | 23:18 | |
| TheJulia | enjoy! | 23:18 |
| *** rh-jelabarre has quit IRC | 23:33 | |
| *** rbudden has quit IRC | 23:51 | |
| *** bfournie has joined #openstack-ironic | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!