| *** jrist has quit IRC | 00:01 | |
| *** jrist has joined #openstack-ironic | 00:03 | |
| *** jrist has quit IRC | 00:04 | |
| *** betherly has joined #openstack-ironic | 00:06 | |
| *** jrist has joined #openstack-ironic | 00:10 | |
| *** betherly has quit IRC | 00:11 | |
| *** jrist has quit IRC | 00:15 | |
| *** jrist has joined #openstack-ironic | 00:17 | |
| *** gregoryo has joined #openstack-ironic | 00:47 | |
| *** ricolin has joined #openstack-ironic | 01:04 | |
| *** betherly has joined #openstack-ironic | 01:08 | |
| *** betherly has quit IRC | 01:13 | |
| *** igordc has quit IRC | 01:31 | |
| *** betherly has joined #openstack-ironic | 01:40 | |
| *** betherly has quit IRC | 01:44 | |
| *** gyee has quit IRC | 02:00 | |
| *** betherly has joined #openstack-ironic | 02:01 | |
| *** betherly has quit IRC | 02:05 | |
| *** betherly has joined #openstack-ironic | 02:25 | |
| *** betherly has quit IRC | 02:29 | |
| *** rloo has quit IRC | 02:33 | |
| *** gkadam has joined #openstack-ironic | 03:29 | |
| *** gkadam has quit IRC | 03:34 | |
| *** gkadam has joined #openstack-ironic | 03:35 | |
| *** whoami-rajat has joined #openstack-ironic | 03:41 | |
| *** stendulker has joined #openstack-ironic | 03:43 | |
| *** verma-varsha has joined #openstack-ironic | 04:07 | |
| *** mkrai has joined #openstack-ironic | 04:23 | |
| openstackgerrit | Merged openstack/sushy master: Cache message registries https://review.opendev.org/671091 | 04:25 |
|---|---|---|
| *** rachit7 has joined #openstack-ironic | 04:48 | |
| *** diga has joined #openstack-ironic | 04:51 | |
| diga | Good Morning Ironic! | 04:51 |
| diga | stendulker Hi | 04:52 |
| *** verma-varsha1 has joined #openstack-ironic | 04:52 | |
| *** verma-varsha has quit IRC | 04:54 | |
| *** verma-varsha1 has quit IRC | 04:55 | |
| *** stendulker has quit IRC | 05:13 | |
| *** verma-varsha has joined #openstack-ironic | 05:14 | |
| *** stendulker has joined #openstack-ironic | 05:16 | |
| stendulker | Good morning diga | 05:16 |
| diga | Hey | 05:16 |
| *** verma-varsha has quit IRC | 05:16 | |
| diga | I updated my patch https://review.opendev.org/#/c/648360/. It's a cherry for master, can you take a look at it once ? | 05:17 |
| patchbot | patch 648360 - ironic (stable/rocky) - DRAC: Fix OOB introspection to use pxe_enabled fla... - 25 patch sets | 05:17 |
| stendulker | diga, sure. | 05:17 |
| diga | stendulker Thank you! | 05:17 |
| *** ociuhandu has joined #openstack-ironic | 05:22 | |
| mkrai | Hi Ironicers! | 05:26 |
| mkrai | Good morning diga stendulker | 05:26 |
| *** dansmith has quit IRC | 05:26 | |
| *** ociuhandu has quit IRC | 05:27 | |
| mkrai | Can someone please review https://review.opendev.org/#/c/672252/ ? | 05:27 |
| patchbot | patch 672252 - ironic-specs - Add support for manual deploy steps - 3 patch sets | 05:27 |
| *** kaifeng has joined #openstack-ironic | 05:27 | |
| *** dansmith has joined #openstack-ironic | 05:28 | |
| *** henriqueof has joined #openstack-ironic | 06:17 | |
| *** jtomasek has joined #openstack-ironic | 06:21 | |
| *** ricolin_ has joined #openstack-ironic | 06:26 | |
| *** iurygregory has joined #openstack-ironic | 06:29 | |
| *** ricolin has quit IRC | 06:29 | |
| iurygregory | good morning Ironic o/ | 06:29 |
| *** diga has quit IRC | 06:31 | |
| *** rachit7 has quit IRC | 06:36 | |
| stendulker | Good morning mkrai, iurygregory | 06:48 |
| iurygregory | o/ | 06:49 |
| *** verma-varsha has joined #openstack-ironic | 06:53 | |
| *** e0ne has joined #openstack-ironic | 06:57 | |
| *** e0ne has quit IRC | 06:59 | |
| *** diga has joined #openstack-ironic | 07:05 | |
| *** verma-varsha1 has joined #openstack-ironic | 07:10 | |
| *** verma-varsha has quit IRC | 07:10 | |
| *** verma-varsha1 is now known as verma-varsha | 07:10 | |
| *** rachit7 has joined #openstack-ironic | 07:11 | |
| *** pcaruana has quit IRC | 07:12 | |
| *** rpittau|afk is now known as rpittau | 07:13 | |
| rpittau | good morning ironic! o/ | 07:13 |
| *** tesseract has joined #openstack-ironic | 07:20 | |
| rpittau | gate is still KO :/ | 07:21 |
| *** jtomasek has quit IRC | 07:29 | |
| *** tssurya has joined #openstack-ironic | 07:32 | |
| *** Goneri has joined #openstack-ironic | 07:35 | |
| *** dsneddon has quit IRC | 07:35 | |
| kaifeng | rpittau it seems the amazing issue of "/etc/nova is not a directory" is still valid | 07:36 |
| rpittau | kaifeng: yeah, the revert patch has not merged yet | 07:37 |
| *** ociuhandu has joined #openstack-ironic | 07:39 | |
| iurygregory | morning rpittau kaifeng o* | 07:41 |
| kaifeng | so recheck doesn't help | 07:42 |
| iurygregory | also the revert patch will need another recheck =X | 07:42 |
| iurygregory | grenade-py3 (2) failed | 07:42 |
| kaifeng | morning iurygregory | 07:42 |
| iurygregory | different jobs failing each time =( | 07:42 |
| *** pcaruana has joined #openstack-ironic | 07:42 | |
| rpittau | I think it was already triggered | 07:42 |
| rpittau | it'll take time, tempest CI has a lot of jobs to run | 07:43 |
| iurygregory | rpittau, the one in zuul I think is the one that Masayuki Igawa triggered | 07:44 |
| rpittau | aaaaaand it failed | 07:44 |
| iurygregory | yup 1 job | 07:44 |
| *** verma-varsha has quit IRC | 07:46 | |
| mkrai | Hi Ironic team, where does introspection_data come from in call https://github.com/openstack/ironic-inspector/blob/master/ironic_inspector/plugins/standard.py#L99 | 07:46 |
| *** gregoryo has quit IRC | 07:51 | |
| *** jtomasek has joined #openstack-ironic | 07:52 | |
| *** rachit7 has quit IRC | 07:53 | |
| *** e0ne has joined #openstack-ironic | 07:53 | |
| *** ociuhandu has quit IRC | 07:56 | |
| kaifeng | mkrai, the introspection data comes from /v1/continue | 08:07 |
| kaifeng | when a baremetal boots up and IPA collects hardware information, then the IPA will post the collected data to ironic inspector | 08:08 |
| *** jtomasek has quit IRC | 08:09 | |
| *** lucasagomes has joined #openstack-ironic | 08:11 | |
| *** S4ren has joined #openstack-ironic | 08:12 | |
| *** derekh has joined #openstack-ironic | 08:25 | |
| *** dsneddon has joined #openstack-ironic | 08:28 | |
| openstackgerrit | Merged openstack/sushy master: Retry Virtual Media eject action on HTTP 400 response https://review.opendev.org/671306 | 08:46 |
| *** verma-varsha has joined #openstack-ironic | 08:50 | |
| *** priteau has joined #openstack-ironic | 08:53 | |
| openstackgerrit | Merged openstack/ironic-inspector master: CI: clean up the required projects list for grenade https://review.opendev.org/673439 | 08:54 |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic-inspector stable/stein: CI: clean up the required projects list for grenade https://review.opendev.org/673958 | 08:55 |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic-inspector stable/rocky: CI: clean up the required projects list for grenade https://review.opendev.org/673959 | 08:56 |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic-inspector stable/queens: CI: clean up the required projects list for grenade https://review.opendev.org/673960 | 08:56 |
| *** verma-varsha has quit IRC | 09:04 | |
| mkrai | kaifeng, Thanks, how does the ironic-inspector plugins called? | 09:05 |
| mkrai | kaifeng, IPA collects all information at once and then call each plugin? | 09:05 |
| kaifeng | mkrai, IPA only post data to inspector, the data handling is in the inspector side | 09:11 |
| mkrai | kaifeng, Yes I got that. Can you please point me to the inspection call from IPA? | 09:12 |
| kaifeng | you will get an idea if you follow the /v1/continue code flow of inspector | 09:12 |
| mkrai | kaifeng, I am trying to implement inspection for Intel Speed Select Hardware | 09:12 |
| mkrai | I added the plugin in inspector and its working fine | 09:12 |
| kaifeng | the plugin is executed in the code path | 09:12 |
| mkrai | I now want to understand the IPA introspection_data collection, so that I can feed in Intel SST related data in that | 09:13 |
| kaifeng | in a broad view, the IPA collects all the data, inspector processes the data and generate proper node data, then update it to ironic | 09:16 |
| kaifeng | if you want to understand how introspection data is collected, I think you can take look on the ironic-python-agent | 09:17 |
| mkrai | kaifeng, Thanks. I am looking into it. | 09:21 |
| openstackgerrit | pengyuesheng proposed openstack/networking-baremetal master: Blacklist sphinx 2.1.0 (autodoc bug) https://review.opendev.org/673972 | 09:22 |
| openstackgerrit | Kaifeng Wang proposed openstack/ironic-inspector master: Split API and conductor services https://review.opendev.org/673732 | 09:26 |
| openstackgerrit | pengyuesheng proposed openstack/networking-baremetal master: Bump the openstackdocstheme extension to 1.20 https://review.opendev.org/673976 | 09:27 |
| *** verma-varsha has joined #openstack-ironic | 09:28 | |
| erolg | morning all o/ | 09:30 |
| erolg | dtantsur|afk, TheJulia, we tested sw raid with ironic-lib 2.19. We got an error while grub-install proceed. Full journal log: https://gist.github.com/erolg/e8fde9eb67a8cfee19714b2b7fb7108e | 09:35 |
| *** rachit7 has joined #openstack-ironic | 09:35 | |
| *** e0ne_ has joined #openstack-ironic | 09:36 | |
| *** diga has quit IRC | 09:36 | |
| *** e0ne has quit IRC | 09:37 | |
| iurygregory | rpittau, tempest-multinode-full and tempest-multinode-full-py3 failed in the revert patch (╯°□°）╯︵ ┻━┻ | 09:40 |
| rpittau | :( | 09:42 |
| *** priteau has quit IRC | 09:43 | |
| *** mkrai has quit IRC | 09:43 | |
| *** priteau has joined #openstack-ironic | 09:43 | |
| *** diga has joined #openstack-ironic | 09:44 | |
| *** priteau has quit IRC | 09:50 | |
| *** priteau has joined #openstack-ironic | 09:51 | |
| *** ociuhandu has joined #openstack-ironic | 09:57 | |
| *** ociuhandu has quit IRC | 10:01 | |
| rpittau | errands and lunch, BBL | 10:04 |
| *** rpittau is now known as rpittau|bbl | 10:04 | |
| openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/ironic-inspector master: CI documentation https://review.opendev.org/673992 | 10:11 |
| *** ricolin__ has joined #openstack-ironic | 10:28 | |
| *** dtantsur|afk is now known as dtantsur | 10:29 | |
| *** ricolin_ has quit IRC | 10:31 | |
| dtantsur | morning/afternoon | 10:32 |
| *** verma-varsha has quit IRC | 10:34 | |
| dtantsur | erolg: hmm, interesting. I wonder if it's trying to install on an underlying partition | 10:40 |
| iurygregory | dtantsur, morning o/ | 10:41 |
| *** dsneddon has quit IRC | 10:41 | |
| dtantsur | erolg: a wild guess from quick googling: does your user image (not ramdiks) have mdadm utility? | 10:41 |
| dtantsur | hmm, or we may adjust our GRUB installation procedure | 10:50 |
| iurygregory | dtantsur, Im not sure if I got your review in https://review.opendev.org/#/c/673861/, Im not sure if the commit message gave you the wrong idea or if I would need to push another patch only for typo fixes and the change for a differente uuid in the json... =) | 10:51 |
| patchbot | patch 673861 - ironic-prometheus-exporter - Fixes for ironic_prometheus_exporter - 1 patch set | 10:51 |
| dtantsur | iurygregory: the latter please | 10:51 |
| iurygregory | ok o/ | 10:51 |
| dtantsur | it may be annoying for you, but it's quite hard to figure out what to look at when a patch contains many changes not under the same topic. | 10:51 |
| iurygregory | np ^^ | 10:52 |
| iurygregory | fix typo in docstring is ok to keep? | 10:52 |
| iurygregory | or another patch also? | 10:53 |
| dtantsur | iurygregory: you can have one patch with all typos | 10:56 |
| iurygregory | dtantsur, ack | 10:56 |
| *** rcernin has quit IRC | 10:58 | |
| dtantsur | erolg: I've checked a CentOS 7 cloud image, and it doesn't have mdadm. Can it be your case? | 11:01 |
| openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/ironic-prometheus-exporter master: Move metric creation logic from messaging module https://review.opendev.org/673861 | 11:01 |
| erolg | dtantsur, maybe.. But, I see that it can assemble md0 and write deployment image on it. | 11:02 |
| dtantsur | yep! | 11:02 |
| dtantsur | but that uses mdadm from the *ramdisk* while grub (apparently) can try to use mdadm from the *image* | 11:03 |
| erolg | Btw, we are using default ramdisk. It şs debian 9. Am i right? | 11:03 |
| erolg | dtantsur, ah OK. I will check deployment image | 11:04 |
| dtantsur | oh no, the revert failed again :( | 11:16 |
| *** rachit7 has quit IRC | 11:19 | |
| *** ociuhandu has joined #openstack-ironic | 11:20 | |
| *** ociuhandu has quit IRC | 11:21 | |
| *** verma-varsha has joined #openstack-ironic | 11:22 | |
| openstackgerrit | Shivanand Tendulker proposed openstack/ironic master: RAID creation fails with 'ilo5' RAID interface https://review.opendev.org/674007 | 11:26 |
| *** e0ne has joined #openstack-ironic | 11:29 | |
| TheJulia | good morning everyone | 11:30 |
| *** e0ne_ has quit IRC | 11:30 | |
| iurygregory | morning TheJulia | 11:30 |
| kaifeng | morning TheJulia | 11:31 |
| *** stendulker has quit IRC | 11:32 | |
| dtantsur | morning TheJulia | 11:32 |
| kaifeng | I think the service split of inspector is ready for review, it would be nice if anyone has the time to take a look :) | 11:35 |
| dtantsur | \o/ | 11:37 |
| TheJulia | Gentle reminder for everyone: https://etherpad.openstack.org/p/PVG-Ironic-Planning <-- Please indicate attendance so I can hopefully provide the foundation with an accurate estimate of our attendees | 11:40 |
| *** dsneddon has joined #openstack-ironic | 11:41 | |
| kaifeng | TheJulia, sorry for the late status update, I'll not be able to attend the summit due to no topic submitted. | 11:43 |
| TheJulia | :( That is unfortunate, but I understand | 11:43 |
| kaifeng | I was unsure until last week I was told so, so, maybe next time. | 11:44 |
| dtantsur | kaifeng: reviewed, need to check the etcd patch as well now | 11:45 |
| *** priteau has quit IRC | 11:46 | |
| kaifeng | dtantsur, cool thanks! the etcd backend is not working, so I tested with memcached | 11:47 |
| *** dsneddon has quit IRC | 11:47 | |
| dtantsur | kaifeng: should we change the 1st patch then? | 11:47 |
| dtantsur | also, why isn't it working? I don't mind memcached, just wondering. | 11:47 |
| dtantsur | also #sadpanda at you not going to the summit (although I'm not going either) | 11:48 |
| kaifeng | dtantsur, yeah, so close this time.. | 11:49 |
| kaifeng | My suspect on etcd is something wrong within the python-etcd3 | 11:50 |
| kaifeng | 2019-07-27 14:42:35.905 31393 ERROR tooz.drivers.etcd3 grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with: | 11:51 |
| kaifeng | 2019-07-27 14:42:35.905 31393 ERROR tooz.drivers.etcd3 status = StatusCode.DEADLINE_EXCEEDED | 11:51 |
| kaifeng | 2019-07-27 14:42:35.905 31393 ERROR tooz.drivers.etcd3 details = "Deadline Exceeded" | 11:51 |
| kaifeng | 2019-07-27 14:42:35.905 31393 ERROR tooz.drivers.etcd3 debug_error_string = "{"created":"@1564209755.902799515","description":"Error received from peer ipv4:127.0.0.1:2379","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Deadline Exceeded","grpc_status":4}" | 11:51 |
| kaifeng | this is I got from the client, from the etcd side, I can see 2019-07-26 17:44:41.540071 D | etcdserver/api/v3rpc: failed to receive lease keepalive request from gRPC stream ("rpc error: code = Canceled desc = context canceled") | 11:51 |
| dtantsur | hmm | 11:52 |
| dtantsur | I don't know if gRPC is supposed to work with eventlet, to be honest | 11:52 |
| kaifeng | it seems similar with https://github.com/kragniz/python-etcd3/issues/560, but I am not sure, because I am not using grpc proxy | 11:53 |
| kaifeng | Yeah, I suspect that, it works when I am executing in a shell, so I have suspected the concurrency, but I don't have much knowledge on the grpc. | 11:53 |
| *** dsneddon has joined #openstack-ironic | 12:00 | |
| *** ociuhandu has joined #openstack-ironic | 12:01 | |
| kaifeng | dtantsur: the 1st patch still use etcd3 as default value, I think it needs an update, will do it tomorrow. | 12:02 |
| *** verma-varsha has quit IRC | 12:03 | |
| kaifeng | night everyone | 12:03 |
| TheJulia | goodnight kaifeng | 12:03 |
| *** ociuhandu has quit IRC | 12:05 | |
| *** dsneddon has quit IRC | 12:05 | |
| *** dsneddon has joined #openstack-ironic | 12:09 | |
| *** gkadam has quit IRC | 12:10 | |
| erolg | dtantsur, I am happily say that, we successfully deployed baremetal with sw raid using bifrost :) | 12:15 |
| *** ociuhandu has joined #openstack-ironic | 12:15 | |
| dtantsur | \o/ | 12:15 |
| dtantsur | congrats erolg! what solved the last problem? | 12:16 |
| *** dsneddon has quit IRC | 12:16 | |
| *** dsneddon has joined #openstack-ironic | 12:17 | |
| erolg | dtantsur, TheJulia thanks a lot. mdadm is missing in default debian image. we added mdadm in dib_packages variable and build image again. | 12:17 |
| dtantsur | great! I think we should document it | 12:17 |
| dtantsur | erolg: do you have an outline of the steps you took? a blog post would be ideal, but just a list works too :) | 12:18 |
| TheJulia | definitely | 12:21 |
| TheJulia | blog post +++++ | 12:21 |
| * dtantsur update the docs | 12:21 | |
| erolg | dtantsur, Yep. We want to implement sw raid configuration in bifrost playbook. And agree with you also we should note it in bifrost documentation | 12:21 |
| dtantsur | that would be fantastic! | 12:21 |
| erolg | if it is ok for you | 12:21 |
| erolg | :) | 12:22 |
| *** Lucas_Gray has joined #openstack-ironic | 12:22 | |
| *** dsneddon has quit IRC | 12:22 | |
| *** ociuhandu has quit IRC | 12:25 | |
| *** rloo has joined #openstack-ironic | 12:27 | |
| *** ociuhandu has joined #openstack-ironic | 12:27 | |
| *** ociuhandu has quit IRC | 12:31 | |
| *** ociuhandu has joined #openstack-ironic | 12:31 | |
| *** Lucas_Gray has quit IRC | 12:32 | |
| dtantsur | TheJulia: do you think the issue with software RAID in ironic-lib justifies bumping the requirement in IPA to 2.19.0? | 12:33 |
| *** Lucas_Gray has joined #openstack-ironic | 12:35 | |
| TheJulia | dtantsur: I believe it does | 12:35 |
| TheJulia | dtantsur: also that will force a new build to be published which is good++ | 12:36 |
| *** tssurya_ has joined #openstack-ironic | 12:37 | |
| *** tssurya has quit IRC | 12:37 | |
| *** tssurya_ is now known as tssurya | 12:37 | |
| *** Lucas_Gray has quit IRC | 12:42 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: Update software RAID configuration documentation https://review.opendev.org/674019 | 12:43 |
| dtantsur | erolg: could you check if ^^^ matches your experience? | 12:44 |
| * dtantsur wonders why we don't support e.g. RAID 5 and 6 | 12:47 | |
| *** dsneddon has joined #openstack-ironic | 12:52 | |
| dtantsur | and I also wonder if we actually don't support partition images | 12:56 |
| *** rpittau|bbl is now known as rpittau | 12:57 | |
| *** jjohnson42 has joined #openstack-ironic | 13:00 | |
| openstackgerrit | Pradip Kadam proposed openstack/ironic master: DRAC : clear_job_queue clean step to fix pending bios config jobs https://review.opendev.org/674021 | 13:00 |
| TheJulia | dtantsur: ever try to build a software raid5 root? | 13:04 |
| dtantsur | TheJulia: root won't work for sure, I'm wondering about the 2nd disk | 13:05 |
| dtantsur | also note that I'm completely inexperienced with RAID and is learning things right now :) | 13:05 |
| dtantsur | s/is/am/ | 13:05 |
| TheJulia | oh, that... should be viable or possible Or maybe it was the very next step in that code. I just don't remember it now | 13:05 |
| dtantsur | oh, our imagebuild jobs don't work | 13:09 |
| dtantsur | sweet | 13:09 |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic-python-agent master: Repair building tinyipa images after the move to ipa-builder https://review.opendev.org/674026 | 13:16 |
| dtantsur | TheJulia: rpittau: ^^^ | 13:16 |
| dtantsur | unfortunately, this is impossible to test in the CI, so we have to merge and see :( | 13:18 |
| rpittau | yolo merge | 13:19 |
| iurygregory | thanks god that revert exists XD | 13:23 |
| dtantsur | last time I just merged 4 patches one on top of each other :D | 13:25 |
| *** ociuhandu has quit IRC | 13:26 | |
| erolg | dtantsur, I think it is good enough | 13:27 |
| *** ociuhandu has joined #openstack-ironic | 13:27 | |
| *** jcoufal has joined #openstack-ironic | 13:27 | |
| *** priteau has joined #openstack-ironic | 13:28 | |
| *** ociuhandu has quit IRC | 13:31 | |
| *** verma-varsha has joined #openstack-ironic | 13:38 | |
| dtantsur | thx! | 13:41 |
| *** verma-varsha has quit IRC | 13:41 | |
| *** verma-varsha has joined #openstack-ironic | 13:44 | |
| *** verma-varsha has quit IRC | 13:53 | |
| *** priteau has quit IRC | 13:53 | |
| TheJulia | dtantsur: it happens :) | 13:56 |
| *** verma-varsha has joined #openstack-ironic | 13:59 | |
| *** dsneddon has quit IRC | 13:59 | |
| *** priteau has joined #openstack-ironic | 14:01 | |
| dtantsur | the revert of the breaking change keeps failing :tears: | 14:01 |
| *** ociuhandu has joined #openstack-ironic | 14:07 | |
| *** verma-varsha1 has joined #openstack-ironic | 14:15 | |
| *** verma-varsha has quit IRC | 14:16 | |
| *** ociuhandu has quit IRC | 14:16 | |
| *** verma-varsha1 has quit IRC | 14:19 | |
| *** ociuhandu has joined #openstack-ironic | 14:19 | |
| *** jhesketh has quit IRC | 14:23 | |
| *** ociuhandu has quit IRC | 14:23 | |
| *** verma-varsha has joined #openstack-ironic | 14:24 | |
| *** dsneddon has joined #openstack-ironic | 14:29 | |
| *** zbitter is now known as zaneb | 14:32 | |
| *** dsneddon has quit IRC | 14:34 | |
| *** verma-varsha has quit IRC | 14:34 | |
| dtantsur | W00t, it was force-approved. Recheck all the things! | 14:49 |
| *** priteau has quit IRC | 14:56 | |
| *** dsneddon has joined #openstack-ironic | 15:03 | |
| *** mnaser has joined #openstack-ironic | 15:05 | |
| mnaser | is there a way to change the tftpboot path? (i.e. /var/lib/tftpboot instead of /tftpboot ?) | 15:05 |
| dtantsur | mnaser: there certainly is, it's something like tftp_root in the configuration | 15:06 |
| mnaser | i kept looking | 15:06 |
| mnaser | and i kept missing it | 15:06 |
| mnaser | https://github.com/openstack/ironic/blob/68832e83402b241d5745db8a457369ebd743a290/ironic/conf/pxe.py#L70-L79 | 15:06 |
| dtantsur | mnaser: [pxe]tftp_root | 15:06 |
| dtantsur | (I know because tripleo changes it) | 15:06 |
| mnaser | dtantsur: yeah im deploying on centos and looks like tftp uses /var/lib/tftpboot by default | 15:06 |
| mnaser | so similar case here :p | 15:07 |
| dtantsur | :) | 15:07 |
| dtantsur | mnaser: https://github.com/openstack/puppet-ironic/blob/master/manifests/pxe.pp and https://github.com/openstack/puppet-ironic/blob/master/manifests/drivers/pxe.pp can serve as good reference | 15:08 |
| *** ociuhandu has joined #openstack-ironic | 15:10 | |
| *** dsneddon has quit IRC | 15:10 | |
| *** portdirect has joined #openstack-ironic | 15:16 | |
| portdirect | hey - is anyone aware of a vbmc like tool that talks redfish? | 15:16 |
| mnaser | dtantsur: awesome | 15:17 |
| etingof | portdirect, sushy-tools? | 15:19 |
| portdirect | etingof: ask and you shall receive - this looks perfect - thanks! | 15:19 |
| *** Goneri has quit IRC | 15:23 | |
| openstackgerrit | Surya Seetharaman proposed openstack/ironic master: Support power state change callbacks to nova using ksa_adapter https://review.opendev.org/664842 | 15:23 |
| *** gyee has joined #openstack-ironic | 15:27 | |
| *** tssurya has quit IRC | 15:30 | |
| *** dsneddon has joined #openstack-ironic | 15:33 | |
| *** e0ne has quit IRC | 15:37 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: Allow configuring global deploy and rescue kernel/ramdisk https://review.opendev.org/672989 | 15:38 |
| *** sthussey has joined #openstack-ironic | 15:39 | |
| *** dsneddon has quit IRC | 15:42 | |
| *** ociuhandu has quit IRC | 15:47 | |
| *** priteau has joined #openstack-ironic | 15:50 | |
| *** whoami-rajat has quit IRC | 15:51 | |
| *** jtomasek has joined #openstack-ironic | 15:51 | |
| *** lucasagomes has quit IRC | 15:54 | |
| *** gkadam has joined #openstack-ironic | 15:57 | |
| *** gkadam has quit IRC | 16:01 | |
| *** igordc has joined #openstack-ironic | 16:08 | |
| *** yaawang has quit IRC | 16:09 | |
| *** yaawang has joined #openstack-ironic | 16:13 | |
| *** verma-varsha has joined #openstack-ironic | 16:17 | |
| *** dsneddon has joined #openstack-ironic | 16:17 | |
| gregwork | Ironic is beating me up this week. Has anyone seen this kind of error on queens/rdo overcloud deploy: "Failure prepping block device: InternalServerError: Unable to attac | 16:18 |
| gregwork | h VIF 242cb8c5-42c7-44cd-a817-ea53c0ccc61e because Ironic can not update Neutron port 242cb8c5-42c7-44cd-a817-ea53c0ccc61e MAC address to match physical MAC a | 16:18 |
| gregwork | ddress ec:0d:9a:65:e0:dc (HTTP 500)" | 16:18 |
| gregwork | we are getting this on a vanilla undercloud (no previous deploys) so were pretty puzzled by it | 16:18 |
| *** S4ren has quit IRC | 16:19 | |
| TheJulia | gregwork: hmm | 16:19 |
| dtantsur | anything in neutron server logs? | 16:20 |
| TheJulia | gregwork: sounds like ironic doesn't have admin rights..... | 16:20 |
| gregwork | dtantsur: nothing in particular | 16:21 |
| gregwork | TheJulia: thats really odd .. this is essentially a clean openstack undercloud deploy which had no errors | 16:21 |
| TheJulia | gregwork: is the mac it is trying to update to, already in the neutron port list? | 16:22 |
| gregwork | yes | 16:23 |
| gregwork | openstack port show 242cb8c5-42c7-44cd-a817-ea53c0ccc shows it here with the autogenerated mac | 16:23 |
| gregwork | whats kind of strange now is all servers are failing to build | 16:24 |
| gregwork | instead of yesterday where it was just one where we then applied that api timeout fix | 16:24 |
| *** dsneddon has quit IRC | 16:25 | |
| gregwork | nova-scheduler reports: "detail": "There was a conflict when trying to complete your request.\n\n Unable to allocate inventory: Unable t | 16:25 |
| gregwork | o create allocation for 'CUSTOM_BAREMETAL' on resource provider 'cc608be4-4673-4838-9efe-b239f42c2c69'. The requested amount would exceed the capacity. ", "title": "Conflict"}]}) | 16:25 |
| gregwork | for all nodes | 16:25 |
| gregwork | control and osdcompute have the expected resources:CUSTOM_BAREMETAL='1', resources:DISK_GB='0', resources:MEMORY_MB='0', resources:VCPU='0' | 16:26 |
| gregwork | doing a mysql nova_api | 16:27 |
| gregwork | SELECT | 16:27 |
| gregwork | p.uuid, | 16:27 |
| gregwork | p.name, | 16:27 |
| gregwork | CASE | 16:27 |
| gregwork | when i.resource_class_id = 0 then 'VCPU' | 16:27 |
| gregwork | when i.resource_class_id = 1 then 'MEMORY_MB' | 16:27 |
| gregwork | when i.resource_class_id = 2 then 'DISK_GB' | 16:27 |
| gregwork | END as resource_name, | 16:27 |
| gregwork | i.total, | 16:27 |
| gregwork | i.reserved, | 16:27 |
| gregwork | a.used | 16:27 |
| gregwork | i mean its 0 | 16:27 |
| gregwork | heh | 16:27 |
| gregwork | (i dont know how irccloud handled that paste .. sorry if it didnt turn it into a pastebin link) | 16:28 |
| *** yaawang has quit IRC | 16:30 | |
| * TheJulia raises an eyebrow | 16:32 | |
| *** yaawang has joined #openstack-ironic | 16:32 | |
| TheJulia | gregwork: I'd ask in nova regarding placement | 16:35 |
| TheJulia | It sounds like it is completely failing to create the record in placement, which I guess might be why nova is refusing to do anything | 16:36 |
| *** dtantsur is now known as dtantsur|afk | 16:38 | |
| dtantsur|afk | o/ | 16:38 |
| rpittau | bye! o/ | 16:40 |
| *** rpittau is now known as rpittau|afk | 16:40 | |
| *** verma-varsha1 has joined #openstack-ironic | 16:42 | |
| *** verma-varsha has quit IRC | 16:44 | |
| *** verma-varsha1 has quit IRC | 16:46 | |
| TheJulia | gregwork: Was there anything else on the exception? | 16:47 |
| gregwork | yeah 1 sec | 16:47 |
| *** verma-varsha has joined #openstack-ironic | 16:49 | |
| gregwork | actually all the tracebacks in nova-compute are that error | 16:49 |
| *** dougsz has joined #openstack-ironic | 16:50 | |
| gregwork | MacAddressInUseClient: Unable to complete operation for network f861c389-82a2-4dc7-a3cc-ed0fd96346fe. The mac address ec:0d:9a:30:6d:ee is in use. | 16:50 |
| gregwork | in ironic-conductor | 16:50 |
| gregwork | for all the nodes | 16:50 |
| TheJulia | gregwork: yeah, so you need to remove any duplicates in neutron.... | 16:52 |
| TheJulia | the question is how did you orphan duplicates in neutron to begin with... | 16:52 |
| gregwork | yeah, especially since this was the first deploy ever | 16:52 |
| gregwork | we reinstalled director after we got mac address in use errors | 16:52 |
| * TheJulia wonders if the heat port delete patch was... reverted | 16:53 | |
| TheJulia | or dropped downstream | 16:53 |
| TheJulia | gregwork: heat has code that basically goes "okay, forget that, I'll try again" | 16:54 |
| TheJulia | that _was_ one route for this issue, I'm 99.95% sure we backported the fix to delete the port in that case | 16:55 |
| TheJulia | heat cloning now | 16:57 |
| TheJulia | gregwork: reinstalled with the databases preserved? | 16:58 |
| gregwork | no | 16:58 |
| gregwork | full vm scrub | 16:58 |
| TheJulia | okay | 16:58 |
| *** verma-varsha has quit IRC | 16:58 | |
| gregwork | reinstalled from scratch | 16:58 |
| TheJulia | I wonder... | 16:58 |
| TheJulia | gregwork: so what is the first error? I wonder if the placement issue was first which caused the cascade | 16:59 |
| *** dougsz has quit IRC | 17:00 | |
| *** derekh has quit IRC | 17:01 | |
| TheJulia | so yeah, the ports should be deleting if the instance is hitting error state... a06fb942ad785c1c117e9e3f466058e2a2592815 | 17:01 |
| TheJulia | on stable/queens for heat | 17:02 |
| *** igordc has quit IRC | 17:03 | |
| *** priteau has quit IRC | 17:05 | |
| gregwork | is there a way to test this out | 17:07 |
| *** dsneddon has joined #openstack-ironic | 17:07 | |
| gregwork | prove if its been reverted/regressed | 17:07 |
| gregwork | the test here for me is these 6 servers had osp13 deployed on them a year ago shortly after the osp13 ga release | 17:07 |
| gregwork | on the same hardware | 17:07 |
| gregwork | and we didnt run into any of these issues | 17:07 |
| gregwork | a year later we are reinstalling on the same gear from a new director on the same rhv stack | 17:08 |
| gregwork | and everything is blowing up | 17:08 |
| gregwork | whats even more wild is we have done probably 30-40 overcloud deploys in the past 7 days on this hardware where servers come up / boostrap | 17:09 |
| gregwork | but in the last day or so | 17:09 |
| gregwork | we are getting this | 17:09 |
| gregwork | nothing bootstraps, all sorts of strange nova/ironic errors | 17:09 |
| gregwork | we thought maybe something got hosed from an interrupted stack delete of the overcloud (it happened once), so rather than jump into the rabbit hole, we just reinstalled director and copied the templates back over | 17:10 |
| *** dsneddon has quit IRC | 17:12 | |
| *** whoami-rajat has joined #openstack-ironic | 17:15 | |
| *** ashish2307 has joined #openstack-ironic | 17:22 | |
| *** igordc has joined #openstack-ironic | 17:27 | |
| *** jcoufal_ has joined #openstack-ironic | 17:29 | |
| TheJulia | gregwork: umm... hmm | 17:29 |
| *** ricolin__ is now known as ricolin | 17:30 | |
| *** igordc has quit IRC | 17:30 | |
| TheJulia | gregwork, does this look familiar in the file being diffed? https://www.irccloud.com/pastebin/NfHAxi4Q/ | 17:31 |
| *** jcoufal has quit IRC | 17:32 | |
| openstackgerrit | Jim Rollenhagen proposed openstack/python-ironicclient master: Strip prefix when paginating https://review.opendev.org/670600 | 17:32 |
| TheJulia | gregwork: that will only work if it shows as in ERROR state | 17:36 |
| *** dsneddon has joined #openstack-ironic | 17:37 | |
| *** igordc has joined #openstack-ironic | 17:42 | |
| *** dsneddon has quit IRC | 17:42 | |
| *** ociuhandu has joined #openstack-ironic | 17:44 | |
| *** dsneddon has joined #openstack-ironic | 17:45 | |
| gregwork | checking | 17:45 |
| *** ociuhandu has quit IRC | 17:48 | |
| *** betherly has joined #openstack-ironic | 17:49 | |
| *** dsneddon has quit IRC | 17:51 | |
| gregwork | https://usercontent.irccloud-cdn.com/file/Epf9jsU1/diff | 17:51 |
| gregwork | looks like it is in place | 17:52 |
| *** betherly has quit IRC | 17:54 | |
| openstackgerrit | Nisha Brahmankar proposed openstack/ironic-python-agent-builder master: Copies the "ironic-agent" element from DIB https://review.opendev.org/669213 | 17:55 |
| *** dsneddon has joined #openstack-ironic | 17:55 | |
| openstackgerrit | Nisha Brahmankar proposed openstack/ironic master: Uses IPA-B to build insted of CoreOS https://review.opendev.org/673834 | 18:11 |
| openstackgerrit | Nisha Brahmankar proposed openstack/ironic master: Uses IPA-B to build insted of CoreOS https://review.opendev.org/673834 | 18:13 |
| *** jcoufal_ has quit IRC | 18:14 | |
| TheJulia | gregwork: Did the nova/placement folks have any insight on the placement failure your seeing? | 18:21 |
| gregwork | <efried> gregwork: That all looks copacetic. Is the issue reproducible? | 18:21 |
| gregwork | from the things they had me look at | 18:21 |
| *** betherly has joined #openstack-ironic | 18:30 | |
| openstackgerrit | Merged openstack/ironic master: Fixes power-on failure for 'ilo' hardware type https://review.opendev.org/673008 | 18:32 |
| *** jcoufal has joined #openstack-ironic | 18:34 | |
| *** betherly has quit IRC | 18:35 | |
| TheJulia | gregwork: Okay. So what have you been able to identify as the first failure in the series of events? | 18:42 |
| gregwork | i just did a writeup on just that topic for gss.. trying to fpaste it is not working .. | 18:43 |
| gregwork | 1 sec | 18:43 |
| *** diga has quit IRC | 18:44 | |
| gregwork | heh this just wont paste, their portal does something to newlines .. im going to try and screencapture | 18:46 |
| * TheJulia goes back to writing lots of words | 18:48 | |
| gregwork | https://paste.fedoraproject.org/paste/l5uQ8bFQQ3UKLqvJenXp6A | 18:48 |
| gregwork | that seems to work | 18:48 |
| *** jcoufal has quit IRC | 18:49 | |
| TheJulia | whaaaaa... keystone internal error upon cleanup | 18:55 |
| TheJulia | *doubletakes* | 18:55 |
| TheJulia | gregwork: anything in the keystone logs? did the token expire? How long after starting the deploy did it take to get to that point? | 18:56 |
| gregwork | it takes about maybe 10 minutes before things start moving | 18:57 |
| *** betherly has joined #openstack-ironic | 18:58 | |
| *** betherly has quit IRC | 19:03 | |
| gregwork | there are no errors in the keystone logs | 19:04 |
| gregwork | just the get/put | 19:04 |
| *** jtomasek has quit IRC | 19:05 | |
| TheJulia | It all blew up when the config drive was generated... | 19:05 |
| TheJulia | It feels like the token lifetime has been tuned down | 19:05 |
| *** dsneddon has quit IRC | 19:08 | |
| TheJulia | gregwork: that would literally explain everything if it has | 19:10 |
| rpioso | TheJulia: Should you be in a meeting now? | 19:14 |
| TheJulia | rpioso: why yes, I should. DOH! | 19:15 |
| TheJulia | rpioso: thanks, I was heads down in a document | 19:16 |
| rpioso | TheJulia: yw | 19:16 |
| *** tesseract has quit IRC | 19:23 | |
| *** dsneddon has joined #openstack-ironic | 19:40 | |
| *** ash2307 has joined #openstack-ironic | 19:41 | |
| *** jtomasek has joined #openstack-ironic | 19:56 | |
| *** jcoufal has joined #openstack-ironic | 19:57 | |
| *** ash2307 has quit IRC | 19:58 | |
| *** betherly has joined #openstack-ironic | 20:00 | |
| *** ash2307 has joined #openstack-ironic | 20:00 | |
| *** radez has quit IRC | 20:03 | |
| *** betherly has quit IRC | 20:04 | |
| *** jtomasek has quit IRC | 20:06 | |
| *** jcoufal has quit IRC | 20:22 | |
| *** e0ne has joined #openstack-ironic | 20:40 | |
| stevebaker | morning | 20:50 |
| *** betherly has joined #openstack-ironic | 20:59 | |
| *** betherly has quit IRC | 21:03 | |
| *** Lucas_Gray has joined #openstack-ironic | 21:13 | |
| *** betherly has joined #openstack-ironic | 21:19 | |
| *** betherly has quit IRC | 21:24 | |
| *** jjohnson42 has quit IRC | 21:37 | |
| *** jjohnson42 has joined #openstack-ironic | 21:37 | |
| *** jjohnson42 has quit IRC | 21:42 | |
| *** betherly has joined #openstack-ironic | 21:50 | |
| *** betherly has quit IRC | 21:55 | |
| *** henriqueof has quit IRC | 22:07 | |
| *** henriqueof has joined #openstack-ironic | 22:07 | |
| *** betherly has joined #openstack-ironic | 22:11 | |
| *** betherly has quit IRC | 22:15 | |
| *** fyx_ has joined #openstack-ironic | 22:29 | |
| *** dustinc_ has joined #openstack-ironic | 22:29 | |
| *** mnasiadka_ has joined #openstack-ironic | 22:30 | |
| *** dtantsur has joined #openstack-ironic | 22:30 | |
| *** betherly has joined #openstack-ironic | 22:31 | |
| *** betherly has quit IRC | 22:36 | |
| *** dtantsur|afk has quit IRC | 22:37 | |
| *** fyx has quit IRC | 22:37 | |
| *** mnasiadka has quit IRC | 22:37 | |
| *** dustinc has quit IRC | 22:37 | |
| *** mnasiadka_ is now known as mnasiadka | 22:37 | |
| *** fyx_ is now known as fyx | 22:37 | |
| *** dustinc_ is now known as dustinc | 22:37 | |
| *** e0ne_ has joined #openstack-ironic | 22:45 | |
| *** e0ne has quit IRC | 22:46 | |
| *** e0ne has joined #openstack-ironic | 22:47 | |
| *** e0ne_ has quit IRC | 22:50 | |
| *** e0ne has quit IRC | 22:52 | |
| *** whoami-rajat has quit IRC | 22:55 | |
| *** igordc has quit IRC | 23:10 | |
| *** ianychoi has quit IRC | 23:12 | |
| *** betherly has joined #openstack-ironic | 23:12 | |
| *** betherly has quit IRC | 23:18 | |
| *** rcernin has joined #openstack-ironic | 23:21 | |
| openstackgerrit | Merged openstack/ironic master: Creation of UEFI ISO fails with efiboot.img https://review.opendev.org/670698 | 23:27 |
| *** sthussey has quit IRC | 23:49 | |
| *** betherly has joined #openstack-ironic | 23:54 | |
| *** trident has quit IRC | 23:54 | |
| *** betherly has quit IRC | 23:58 | |
| *** trident has joined #openstack-ironic | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!