| *** stendulker has joined #openstack-ironic | 00:04 | |
| *** rbudden has quit IRC | 00:07 | |
| *** stendulker has quit IRC | 00:15 | |
| *** jcoufal_ has joined #openstack-ironic | 00:18 | |
| *** crushil_ has quit IRC | 00:18 | |
| *** jcoufal has quit IRC | 00:22 | |
| TheJulia | networking-generic-switch patch is in the gate | 00:24 |
|---|---|---|
| *** yangyapeng has quit IRC | 00:26 | |
| *** jcoufal_ has quit IRC | 00:28 | |
| *** tzumainn has quit IRC | 00:40 | |
| *** threestrands_ has quit IRC | 00:48 | |
| *** threestrands has joined #openstack-ironic | 00:48 | |
| *** crushil_ has joined #openstack-ironic | 00:49 | |
| *** persia has joined #openstack-ironic | 00:49 | |
| *** tuanla____ has joined #openstack-ironic | 00:53 | |
| *** crushil_ has quit IRC | 00:54 | |
| *** phuongnh has joined #openstack-ironic | 01:08 | |
| *** crushil_ has joined #openstack-ironic | 01:08 | |
| *** tiendc has joined #openstack-ironic | 01:15 | |
| *** tuanla____ has quit IRC | 01:15 | |
| *** yangyapeng has joined #openstack-ironic | 01:15 | |
| *** tuanla____ has joined #openstack-ironic | 01:15 | |
| *** fragatina has quit IRC | 01:34 | |
| *** fragatina has joined #openstack-ironic | 01:35 | |
| *** fragatina has quit IRC | 01:36 | |
| *** fragatina has joined #openstack-ironic | 01:36 | |
| *** fragatin_ has joined #openstack-ironic | 01:38 | |
| *** fragatina has quit IRC | 01:40 | |
| *** fragatin_ has quit IRC | 01:42 | |
| openstackgerrit | Zhijiang Hu proposed openstack/virtualbmc master: Support power reset command https://review.openstack.org/534646 | 01:55 |
| *** openstackstatus has quit IRC | 01:55 | |
| *** harlowja has quit IRC | 01:55 | |
| *** alineb has quit IRC | 01:55 | |
| *** mkrai has quit IRC | 01:55 | |
| *** csmart has quit IRC | 01:55 | |
| *** johnsom has quit IRC | 01:55 | |
| *** robcresswell has quit IRC | 01:55 | |
| *** jmccrory has quit IRC | 01:55 | |
| *** gregwork has quit IRC | 01:55 | |
| *** rajinir has quit IRC | 01:55 | |
| *** jlvillal has quit IRC | 01:55 | |
| *** vdrok has quit IRC | 01:55 | |
| *** fdegir has quit IRC | 01:55 | |
| *** jmccrory has joined #openstack-ironic | 01:55 | |
| *** jlvillal has joined #openstack-ironic | 01:55 | |
| *** rajinir has joined #openstack-ironic | 01:55 | |
| *** vdrok has joined #openstack-ironic | 01:56 | |
| *** johnsom has joined #openstack-ironic | 01:56 | |
| *** openstackstatus has joined #openstack-ironic | 01:56 | |
| *** ChanServ sets mode: +v openstackstatus | 01:56 | |
| *** gregwork has joined #openstack-ironic | 01:56 | |
| *** fdegir has joined #openstack-ironic | 01:56 | |
| *** robcresswell has joined #openstack-ironic | 01:56 | |
| *** csmart has joined #openstack-ironic | 01:57 | |
| *** mkrai has joined #openstack-ironic | 01:57 | |
| *** xnox has quit IRC | 01:58 | |
| *** xnox has joined #openstack-ironic | 02:04 | |
| *** xnox has quit IRC | 02:04 | |
| *** xnox has joined #openstack-ironic | 02:04 | |
| *** crushil has quit IRC | 02:06 | |
| *** milan has joined #openstack-ironic | 02:07 | |
| *** crushil has joined #openstack-ironic | 02:09 | |
| *** alineb has joined #openstack-ironic | 02:11 | |
| *** fragatina has joined #openstack-ironic | 02:13 | |
| *** fragatina has quit IRC | 02:18 | |
| *** hshiina has joined #openstack-ironic | 02:19 | |
| *** jlabarre has quit IRC | 02:20 | |
| *** rbudden has joined #openstack-ironic | 02:37 | |
| *** AlexeyAbashkin has joined #openstack-ironic | 02:46 | |
| *** AlexeyAbashkin has quit IRC | 02:50 | |
| *** rpioso is now known as rpioso|afk | 02:54 | |
| *** dpawar has joined #openstack-ironic | 02:54 | |
| *** dpawar has quit IRC | 02:55 | |
| *** dpawar has joined #openstack-ironic | 02:55 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/networking-baremetal master: Updated from global requirements https://review.openstack.org/533901 | 03:05 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/networking-generic-switch master: Updated from global requirements https://review.openstack.org/535020 | 03:06 |
| *** tzumainn has joined #openstack-ironic | 03:11 | |
| openstackgerrit | wangdequn proposed openstack/ironic master: correct referenced url in comment https://review.openstack.org/534710 | 03:19 |
| *** stendulker has joined #openstack-ironic | 03:22 | |
| *** crushil has quit IRC | 03:25 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-ironic-inspector-client master: Updated from global requirements https://review.openstack.org/534285 | 03:27 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-ironicclient master: Updated from global requirements https://review.openstack.org/535117 | 03:28 |
| *** mariojv has quit IRC | 03:30 | |
| *** bl0m1_ has quit IRC | 03:30 | |
| *** alex_xu has quit IRC | 03:30 | |
| *** retr0h has quit IRC | 03:30 | |
| *** v12aml has quit IRC | 03:30 | |
| *** rpioso|afk has quit IRC | 03:30 | |
| *** ildikov has quit IRC | 03:30 | |
| *** TheJulia has quit IRC | 03:30 | |
| *** samueldmq has quit IRC | 03:30 | |
| *** bl0m1 has joined #openstack-ironic | 03:30 | |
| *** v12aml has joined #openstack-ironic | 03:30 | |
| *** alex_xu has joined #openstack-ironic | 03:30 | |
| *** mariojv has joined #openstack-ironic | 03:30 | |
| *** samueldmq has joined #openstack-ironic | 03:31 | |
| *** rpioso|afk has joined #openstack-ironic | 03:31 | |
| *** ildikov has joined #openstack-ironic | 03:31 | |
| *** TheJulia has joined #openstack-ironic | 03:31 | |
| *** tzumainn has quit IRC | 03:32 | |
| *** retr0h has joined #openstack-ironic | 03:33 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/sushy master: Updated from global requirements https://review.openstack.org/534056 | 03:35 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/sushy-tools master: Updated from global requirements https://review.openstack.org/534058 | 03:35 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/virtualbmc master: Updated from global requirements https://review.openstack.org/534073 | 03:38 |
| *** hshiina has quit IRC | 03:41 | |
| *** AlexeyAbashkin has joined #openstack-ironic | 03:46 | |
| *** AlexeyAbashkin has quit IRC | 03:51 | |
| openstackgerrit | Merged openstack/networking-generic-switch stable/pike: use callbacks from neutron-lib https://review.openstack.org/534955 | 03:52 |
| TheJulia | rechecked pending jobs | 03:56 |
| * TheJulia goes to bed | 03:56 | |
| *** phuongnh has quit IRC | 04:00 | |
| *** tuanla____ has quit IRC | 04:00 | |
| *** phuongnh has joined #openstack-ironic | 04:01 | |
| *** tuanla____ has joined #openstack-ironic | 04:01 | |
| *** hshiina has joined #openstack-ironic | 04:01 | |
| *** hshiina has quit IRC | 04:02 | |
| *** hshiina has joined #openstack-ironic | 04:11 | |
| *** hshiina has quit IRC | 04:11 | |
| openstackgerrit | Rushil Chugh proposed openstack/ironic master: Add XClarity Driver https://review.openstack.org/519427 | 04:12 |
| *** rbudden has quit IRC | 04:17 | |
| *** fragatina has joined #openstack-ironic | 04:18 | |
| *** fragatina has quit IRC | 04:22 | |
| *** dpawar has quit IRC | 04:26 | |
| *** dpawar has joined #openstack-ironic | 04:36 | |
| *** dpawar has quit IRC | 04:39 | |
| *** hshiina has joined #openstack-ironic | 04:41 | |
| *** ccollett has joined #openstack-ironic | 05:00 | |
| *** charcol has quit IRC | 05:02 | |
| *** crushil has joined #openstack-ironic | 05:03 | |
| *** charcol has joined #openstack-ironic | 05:05 | |
| *** hw_wutianwei has quit IRC | 05:06 | |
| *** ccollett has quit IRC | 05:06 | |
| *** gcb has quit IRC | 05:07 | |
| *** hw_wutianwei has joined #openstack-ironic | 05:07 | |
| *** jaianshu has joined #openstack-ironic | 05:12 | |
| *** cjloader has joined #openstack-ironic | 05:14 | |
| *** gcb has joined #openstack-ironic | 05:14 | |
| *** dpawar has joined #openstack-ironic | 05:19 | |
| *** cjloader has quit IRC | 05:19 | |
| *** hieulq has quit IRC | 05:20 | |
| *** links has joined #openstack-ironic | 05:31 | |
| *** sasha2 has quit IRC | 05:34 | |
| *** hieulq has joined #openstack-ironic | 05:35 | |
| openstackgerrit | Dongcan Ye proposed openstack/ironic master: Add meaningful exception in Neutron port show https://review.openstack.org/533640 | 05:46 |
| *** namnh has joined #openstack-ironic | 06:05 | |
| *** cjloader has joined #openstack-ironic | 06:15 | |
| *** cjloader has quit IRC | 06:19 | |
| openstackgerrit | wangdequn proposed openstack/ironic master: correct referenced url in comments https://review.openstack.org/534710 | 06:23 |
| *** jtomasek has quit IRC | 06:30 | |
| openstackgerrit | Merged openstack/networking-generic-switch master: Updated from global requirements https://review.openstack.org/535020 | 06:31 |
| *** fragatina has joined #openstack-ironic | 06:36 | |
| *** jtomasek has joined #openstack-ironic | 06:38 | |
| *** fragatina has quit IRC | 06:41 | |
| *** phuongnh has quit IRC | 06:48 | |
| *** phuongnh has joined #openstack-ironic | 06:49 | |
| *** crushil has quit IRC | 06:51 | |
| *** makowals has joined #openstack-ironic | 06:56 | |
| *** gcb has quit IRC | 06:58 | |
| *** hshiina has quit IRC | 06:58 | |
| *** ianw has quit IRC | 07:02 | |
| *** mjura has joined #openstack-ironic | 07:04 | |
| *** marios has joined #openstack-ironic | 07:05 | |
| *** dims has quit IRC | 07:12 | |
| *** dims has joined #openstack-ironic | 07:14 | |
| *** cjloader has joined #openstack-ironic | 07:14 | |
| *** pcaruana has joined #openstack-ironic | 07:15 | |
| *** cjloader has quit IRC | 07:19 | |
| *** dims has quit IRC | 07:19 | |
| *** dims has joined #openstack-ironic | 07:20 | |
| *** charcol has quit IRC | 07:26 | |
| openstackgerrit | Debayan Ray proposed openstack/sushy master: Add support to accept custom connector object https://review.openstack.org/535200 | 07:37 |
| *** phuongnh has quit IRC | 07:54 | |
| *** tuanla____ has quit IRC | 07:54 | |
| *** namnh has quit IRC | 07:55 | |
| *** tiendc has quit IRC | 07:55 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ironic-ui master: Imported Translations from Zanata https://review.openstack.org/532436 | 07:55 |
| *** rbartal has joined #openstack-ironic | 07:57 | |
| *** phuongnh has joined #openstack-ironic | 08:01 | |
| *** tuanla____ has joined #openstack-ironic | 08:01 | |
| *** namnh has joined #openstack-ironic | 08:01 | |
| *** tiendc has joined #openstack-ironic | 08:01 | |
| *** threestrands has quit IRC | 08:03 | |
| *** tesseract has joined #openstack-ironic | 08:08 | |
| *** phuongnh has quit IRC | 08:16 | |
| *** tiendc has quit IRC | 08:16 | |
| *** namnh has quit IRC | 08:16 | |
| *** tuanla____ has quit IRC | 08:16 | |
| *** tiendc has joined #openstack-ironic | 08:18 | |
| *** namnh has joined #openstack-ironic | 08:18 | |
| *** tuanla____ has joined #openstack-ironic | 08:18 | |
| *** hw_wutianwei has quit IRC | 08:20 | |
| *** hw_wutianwei has joined #openstack-ironic | 08:24 | |
| *** AlexeyAbashkin has joined #openstack-ironic | 08:26 | |
| *** rcernin has quit IRC | 08:28 | |
| *** deray has joined #openstack-ironic | 08:33 | |
| *** tuanla____ has quit IRC | 08:34 | |
| *** namnh has quit IRC | 08:35 | |
| *** tiendc has quit IRC | 08:35 | |
| *** tuanla____ has joined #openstack-ironic | 08:35 | |
| *** namnh has joined #openstack-ironic | 08:35 | |
| *** tiendc has joined #openstack-ironic | 08:35 | |
| deray | Hello Ironic'ers, g'morning | 08:35 |
| deray | Anybody up for this: https://review.openstack.org/#/c/535200/ ? | 08:39 |
| patchbot | patch 535200 - sushy - Add support to accept custom connector object | 08:39 |
| *** milan has quit IRC | 08:40 | |
| *** pcaruana has quit IRC | 08:43 | |
| *** pcaruana has joined #openstack-ironic | 08:43 | |
| *** namnh_ has joined #openstack-ironic | 08:44 | |
| *** namnh has quit IRC | 08:44 | |
| deray | ^^^ this is needs to go in before Sushy gets released (is it today?) | 08:44 |
| *** namnh_ has quit IRC | 08:44 | |
| *** etingof has quit IRC | 08:45 | |
| *** namnh has joined #openstack-ironic | 08:46 | |
| *** namnh has quit IRC | 08:53 | |
| *** tuanla____ has quit IRC | 08:53 | |
| *** tiendc has quit IRC | 08:53 | |
| *** tuanla____ has joined #openstack-ironic | 08:57 | |
| *** tiendc has joined #openstack-ironic | 08:57 | |
| *** namnh has joined #openstack-ironic | 08:57 | |
| *** ianw has joined #openstack-ironic | 08:58 | |
| *** dtantsur|afk is now known as dtantsur | 09:07 | |
| dtantsur | morning ironic | 09:07 |
| dtantsur | deray: yes, it's today. I will check it | 09:07 |
| deray | dtantsur, o/ | 09:08 |
| deray | dtantsur, good morning | 09:09 |
| dtantsur | deray: mind fixing a few things in the release note while you're here? | 09:10 |
| deray | dtantsur, not at all.. tell me | 09:10 |
| dtantsur | deray: see my comment on the patch | 09:10 |
| deray | dtantsur, about s/Add/Adds .. shoudn't this be in imperative mood? I mean shouldn't it be Add instead of Adds? | 09:12 |
| dtantsur | deray: it's a matter of taste, but we picked the "Adds" form for Ironic, so I think we should use it in Sushy | 09:13 |
| dtantsur | I'd prefer using past (Added), but this is what people decided :) | 09:13 |
| deray | dtantsur, okay sure .. going to submit a new one one | 09:13 |
| deray | *now | 09:13 |
| dtantsur | thanks! | 09:14 |
| openstackgerrit | Debayan Ray proposed openstack/sushy master: Add support to accept custom connector object https://review.openstack.org/535200 | 09:16 |
| deray | dtantsur, ^^^ done! | 09:18 |
| dtantsur | cool | 09:18 |
| openstackgerrit | Merged openstack/sushy master: Updated from global requirements https://review.openstack.org/534056 | 09:22 |
| *** e0ne has joined #openstack-ironic | 09:23 | |
| mgoddard | morning pixies | 09:26 |
| openstackgerrit | Merged openstack/networking-baremetal master: Updated from global requirements https://review.openstack.org/533901 | 09:27 |
| openstackgerrit | Merged openstack/ironic-lib master: Updated from global requirements https://review.openstack.org/534932 | 09:29 |
| deray | mgoddard, g'morning o/ | 09:29 |
| dtantsur | morning mgoddard | 09:33 |
| *** etingof has joined #openstack-ironic | 09:33 | |
| *** athomas has joined #openstack-ironic | 09:33 | |
| dtantsur | mmm, is our multinode job completely down now? :-/ | 09:34 |
| *** pmannidi has quit IRC | 09:35 | |
| *** derekh has joined #openstack-ironic | 09:39 | |
| *** dpawar has quit IRC | 09:42 | |
| *** amarao has joined #openstack-ironic | 09:51 | |
| *** sambetts|afk is now known as sambetts | 10:06 | |
| sambetts | o/ Morning all | 10:07 |
| dtantsur | morning sambetts | 10:09 |
| *** dpawar has joined #openstack-ironic | 10:11 | |
| *** cjloader has joined #openstack-ironic | 10:14 | |
| *** tuanla_____ has joined #openstack-ironic | 10:15 | |
| *** tuanla____ has quit IRC | 10:17 | |
| *** cjloader has quit IRC | 10:19 | |
| *** racedo_ has joined #openstack-ironic | 10:23 | |
| *** tesseract has quit IRC | 10:27 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/sushy-tools master: Updated from global requirements https://review.openstack.org/534059 | 10:27 |
| *** tesseract has joined #openstack-ironic | 10:29 | |
| openstackgerrit | Merged openstack/ironic-lib master: Simplify logic in wait_for_disk_to_become_available https://review.openstack.org/534867 | 10:29 |
| *** athomas has quit IRC | 10:30 | |
| *** fragatina has joined #openstack-ironic | 10:37 | |
| *** fragatina has quit IRC | 10:42 | |
| *** milan has joined #openstack-ironic | 10:50 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: Allow setting {provisioning,cleaning}_network in node driver_info https://review.openstack.org/460564 | 10:55 |
| openstackgerrit | Merged openstack/sushy master: Add support to accept custom connector object https://review.openstack.org/535200 | 10:55 |
| *** pmannidi has joined #openstack-ironic | 11:05 | |
| *** pmannidi has quit IRC | 11:08 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: Allow setting {provisioning,cleaning}_network in node driver_info https://review.openstack.org/460564 | 11:12 |
| dtantsur | sambetts, bfournie: wdyt ^^^? | 11:12 |
| *** deray has quit IRC | 11:14 | |
| openstackgerrit | Mark Goddard proposed openstack/ironic master: Add RPC API and conductor manager for traits https://review.openstack.org/535296 | 11:17 |
| openstackgerrit | Shivanand Tendulker proposed openstack/ironic master: Add rescue related methods to network interface https://review.openstack.org/509342 | 11:19 |
| *** stendulker_ has joined #openstack-ironic | 11:23 | |
| openstackgerrit | Harald Jensås proposed openstack/ironic master: Allow setting {provisioning,cleaning}_network in node driver_info https://review.openstack.org/460564 | 11:26 |
| *** stendulker has quit IRC | 11:26 | |
| hjensas | dtantsur: oh crap. I just rebased that same patch. | 11:27 |
| dtantsur | hjensas: could you please revert it? it needs much more than just rebasing | 11:28 |
| hjensas | dtantsur: will do. Sorry. | 11:28 |
| *** stendulker_ has quit IRC | 11:29 | |
| *** namnh has quit IRC | 11:35 | |
| openstackgerrit | Harald Jensås proposed openstack/ironic master: Allow setting {provisioning,cleaning}_network in node driver_info https://review.openstack.org/460564 | 11:36 |
| *** jistr is now known as jistr|mtg | 11:41 | |
| dtantsur | hjensas: no problem. it's actually funny that we thought about the same patch at the same time :) | 11:46 |
| dtantsur | I'd not mind testing and/or reviewing of my version of course ;) | 11:47 |
| hjensas | dtantsur: yes. :) btw, is there a better way to revert than checking out your change and push it for review? | 11:48 |
| dtantsur | hjensas: I don't think so | 11:48 |
| *** jkilpatr has joined #openstack-ironic | 11:53 | |
| *** jkilpatr has quit IRC | 11:53 | |
| *** jkilpatr_ has joined #openstack-ironic | 11:54 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic-lib stable/pike: Use dd conv=sparse when writing images to nodes https://review.openstack.org/535307 | 11:58 |
| *** tuanla____ has joined #openstack-ironic | 12:00 | |
| *** tuanla_____ has quit IRC | 12:02 | |
| *** tuanla____ has quit IRC | 12:03 | |
| *** gmonteiro has joined #openstack-ironic | 12:15 | |
| gmonteiro | Good Morning Ironic | 12:15 |
| *** bfournie has quit IRC | 12:21 | |
| *** dpawar has quit IRC | 12:25 | |
| *** rbartal has quit IRC | 12:28 | |
| dtantsur | morning gmonteiro | 12:40 |
| gmonteiro | o/ dtantsur | 12:41 |
| gmonteiro | dtantsur: you can review this please https://review.openstack.org/#/c/524599/, when you have time :) | 12:43 |
| patchbot | patch 524599 - ironic - Migrate python-oneviewclient validations to onevie... | 12:43 |
| dtantsur | gmonteiro: it's on my radar, but I'm in the middle of a customer escalation downstream, soo.. | 12:44 |
| gmonteiro | ok :) | 12:45 |
| *** jlabarre has joined #openstack-ironic | 12:45 | |
| *** athomas has joined #openstack-ironic | 12:51 | |
| jroll | morning folks | 13:05 |
| *** tiendc has quit IRC | 13:09 | |
| dtantsur | morning jroll | 13:10 |
| *** ianychoi has quit IRC | 13:12 | |
| *** bfournie has joined #openstack-ironic | 13:16 | |
| ameeda | Morning :) | 13:18 |
| ameeda | is there's a considerations about iLO Firmware Version to use server as compute/control node for openstack ? | 13:19 |
| dtantsur | ameeda: hi! I'm not sure, and I don't see our iLO folks online (they're in East Asia) | 13:21 |
| *** MattMan_1 has quit IRC | 13:21 | |
| *** MattMan_1 has joined #openstack-ironic | 13:22 | |
| *** tzumainn has joined #openstack-ironic | 13:27 | |
| *** jaianshu has quit IRC | 13:27 | |
| ameeda | dtantsur: what do you think about this error "During sync_power_state, max retries exceeded for node 885cc81d-6beb-40fe-92ca-498decae02cd, node state None does not match expected state 'power on'. Updating DB state to 'None' Switching node to maintenance mode. Error: IPMI call failed: power status." | 13:29 |
| dtantsur | ameeda: power fault. if you did not change the IPMI credentials (either in ironic or on the node), your BMC might have creashed/hanged | 13:29 |
| dtantsur | sometimes resetting the BMC may help | 13:30 |
| ameeda | when I run introspection, the 2 servers powered on , but keep show "booting... DHCP" | 13:30 |
| ameeda | dtantsur: I am totally lost :( | 13:30 |
| ameeda | I do this "tcpdump -an -i any port 67 or port 68" I see the communication is active | 13:32 |
| dtantsur | ameeda: DHCP may be https://docs.openstack.org/ironic/latest/admin/troubleshooting.html#dhcp-during-pxe-or-ipxe-is-inconsistent-or-unreliable | 13:32 |
| ameeda | also now I can't remove nodes " No conductor service registered which supports driver pxe_ipmitool" | 13:33 |
| dtantsur | and this means ironic-conductor has crashed, check its logs | 13:33 |
| ameeda | ironic.conf -> enabled_drivers=pxe_ipmitool,pxe_ssh,pxe_drac,pxe_ilo,pxe_wol | 13:33 |
| *** amarao1 has joined #openstack-ironic | 13:34 | |
| *** fragatina has joined #openstack-ironic | 13:36 | |
| *** dpawar has joined #openstack-ironic | 13:37 | |
| *** dpawar has quit IRC | 13:37 | |
| openstackgerrit | Stenio Araujo proposed openstack/ironic master: Fix persistent information when getting boot device https://review.openstack.org/524616 | 13:38 |
| *** dpawar has joined #openstack-ironic | 13:38 | |
| *** fragatina has quit IRC | 13:41 | |
| ameeda | yes, I fix it | 13:42 |
| ameeda | so what do you think about my problem ? | 13:42 |
| ameeda | dtantsur: | 13:42 |
| dtantsur | ameeda: you mention 3 different problems, which of them did you manage to fix? | 13:43 |
| ameeda | dtantsur: I am sorry , about unable to introspect nodes, what if I update ilo version ? | 13:44 |
| dtantsur | ameeda: having an up-to-date firmware version never hurts. esp. if you have problems with ironic calling IPMI | 13:44 |
| dtantsur | is anyone looking into the new multinode job failure? sambetts, pas-ha, mgoddard? sorry, I'm buried into an escalation downstream.. | 13:45 |
| ameeda | I update 1 node with ilo 3 version 1.89 Jul 07 2017 | 13:45 |
| ameeda | I will run command for one node . I hope that will help | 13:46 |
| ameeda | dtantsur: how can I be sure about pm_type ? | 13:46 |
| ameeda | is that may cause the issues / | 13:47 |
| dtantsur | sorry, I don't get the question. what is wrong with pm_type? | 13:47 |
| ameeda | I insert pm_type : "pxe_ipmitool" , but I am not sure about it. how can I select correct type ? | 13:47 |
| dtantsur | this looks correct, just make sure to also provide correct credentials | 13:48 |
| ameeda | I am sure about credentials | 13:49 |
| ameeda | let me see, I will run introspect from upgraded ilo version | 13:49 |
| ameeda | I will let you know | 13:49 |
| dtantsur | cool | 13:49 |
| *** weshay is now known as weshay|rover | 13:52 | |
| *** ianychoi has joined #openstack-ironic | 13:55 | |
| *** rloo has joined #openstack-ironic | 13:55 | |
| *** rloo has quit IRC | 13:56 | |
| ameeda | dtantsur: so when I execute " openstack overcloud node introspect --all-manageable" what should I got in server ? | 13:56 |
| dtantsur | this runs introspection for all nodes in MANAGEABLE state | 13:56 |
| ameeda | what I see is trying to boot from DHCP | 13:56 |
| *** rloo has joined #openstack-ironic | 13:56 | |
| dtantsur | this is correct | 13:57 |
| ameeda | then ? | 13:57 |
| dtantsur | then it loads, in case of iPXE, /httpboot/inspector.ipxe, then kernel and ramdisk | 13:57 |
| ameeda | dtantsur: in my case it stuck on booting | 13:58 |
| dtantsur | you mean, after loading the kernel and ramdisk? | 13:58 |
| *** dprince has joined #openstack-ironic | 13:59 | |
| ameeda | I am not sure, but it stuck after power on | 13:59 |
| ameeda | can you please check this log for me ? https://drive.google.com/file/d/1hVk_w-FN2PQzSe8a7LwKy0BlM9hDlxHX/view?usp=sharing | 14:00 |
| *** rbudden has joined #openstack-ironic | 14:00 | |
| dtantsur | "access denied" | 14:02 |
| ameeda | sorry | 14:03 |
| rloo | good morning everyone, dtantsur | 14:04 |
| rloo | dtantsur: has anyone looked into the multinode grenade CI? Looks like it is still failing | 14:04 |
| dtantsur | morning rloo. yes, it is, for a different reason apparently. | 14:05 |
| rloo | dtantsur: ok, i see your note in the whiteboard now | 14:05 |
| dtantsur | ameeda: sorry, what should I see there? that's the log of all openstack services... | 14:06 |
| rloo | dtantsur: well, at least we fixed one issue. i wonder how many issues there may be, given all the projects with changes this week :-( | 14:06 |
| dtantsur | yeah.. | 14:06 |
| dtantsur | ameeda: your problem is likely to not be within openstack, but within hardware and/or networking | 14:07 |
| ameeda | dtantsur: sorry for that, how I can be sure about networking ? | 14:07 |
| dtantsur | ameeda: this may help: https://docs.openstack.org/ironic-inspector/latest/user/troubleshooting.html#troubleshooting-pxe-boot | 14:07 |
| *** jistr|mtg is now known as jistr | 14:08 | |
| ameeda | dtantsur: I will check, Thank you !! | 14:08 |
| jroll | huh, this 'complex priority' section in this post is interesting. https://storyboard-blog.sotk.co.uk/things-that-storyboard-does-differently.html | 14:10 |
| *** Goneri has joined #openstack-ironic | 14:10 | |
| jroll | feels like that could be useful for tracking weekly and cycle priorities for us | 14:10 |
| rloo | jroll: oh, you mean like what other companies etc do? 'project management' ? :) | 14:11 |
| rloo | good morning jroll | 14:11 |
| dtantsur | jroll: yep | 14:12 |
| jroll | rloo: morning! please, no swear words (the PM word) in this channel | 14:12 |
| jroll | :) | 14:12 |
| rloo | jroll: :D | 14:12 |
| dtantsur | TheJulia, rloo, we don't need https://review.openstack.org/#/c/534952/ merged, do we? | 14:14 |
| patchbot | patch 534952 - ironic-lib - Minor follow-up for wait_for_disk tests | 14:14 |
| rloo | dtantsur: there is one thing there, changes a unit test to take < 2 secs. but no, not needed. | 14:15 |
| rloo | dtantsur: we only use the unit tests internally so i am fine if it goes into a sep release | 14:15 |
| dtantsur | rloo: if it's test-only, we can try rebasing it | 14:15 |
| rloo | dtantsur: oh wait | 14:15 |
| rloo | dtantsur: we don't need that. john incorporated that in a diff patch that got merged | 14:16 |
| dtantsur | sweet | 14:16 |
| dtantsur | then shall the release party begin? :) | 14:16 |
| rloo | dtantsur: i had commented but forgot to vote. | 14:16 |
| rloo | dtantsur: release, release, release. YAYYYY!!!! | 14:16 |
| jroll | \o/ | 14:22 |
| * TheJulia sips coffee and tries to wake up | 14:31 | |
| rloo | dtantsur: wrt the ngs error in the logs, that seems normal (I commented in white board). So the question is why nova-compute isn't starting | 14:32 |
| TheJulia | oh crap, is now nova-compute not starting? | 14:32 |
| rloo | good morning TheJulia. and yes. apparently. multinode grenade is still down :-( | 14:32 |
| TheJulia | ugh | 14:33 |
| TheJulia | got a link handy for my coffee sipping? | 14:33 |
| *** cjloader has joined #openstack-ironic | 14:33 | |
| rloo | TheJulia: in whiteboard | 14:33 |
| *** mjturek has joined #openstack-ironic | 14:36 | |
| * dtantsur watches trees flying by his window | 14:38 | |
| TheJulia | wow.... that.... is a fun error | 14:38 |
| * TheJulia thinks dtantsur has the right idea | 14:38 | |
| TheJulia | so nova-compute is failing to start because ironic-api is unreachable.... | 14:41 |
| dtantsur | meh | 14:41 |
| dtantsur | TheJulia, rloo, here is ironic-lib https://review.openstack.org/#/c/535355/ | 14:41 |
| patchbot | patch 535355 - releases - Release ironic-lib 2.12.0 (final) for Queens | 14:41 |
| rloo | dtantsur: thx | 14:42 |
| rloo | TheJulia: ugh | 14:42 |
| *** cjloader has quit IRC | 14:43 | |
| TheJulia | dtantsur: thanks | 14:43 |
| *** cjloader has joined #openstack-ironic | 14:43 | |
| *** dpawar has quit IRC | 14:44 | |
| *** cjloader has quit IRC | 14:44 | |
| *** baha has joined #openstack-ironic | 14:44 | |
| *** cjloader has joined #openstack-ironic | 14:44 | |
| dtantsur | TheJulia, rloo, and here goes sushy https://review.openstack.org/#/c/535357/ | 14:45 |
| patchbot | patch 535357 - releases - Release sushy 1.3.0 (final) for Queens | 14:45 |
| rloo | dtantsur: wheeeee | 14:45 |
| dtantsur | now let's concentrate on ironicclient | 14:45 |
| dtantsur | TheJulia: I've left comments on the version negotiation patches, hope you won't hate me ^_^ | 14:45 |
| TheJulia | dtantsur: ugh, really?!? | 14:46 |
| rloo | umm, let's concentrate on fixing grenade test? :) | 14:47 |
| TheJulia | so, I need to switch laptops because I have a clone of nova on it... but I wonder if our special handle failure/restart because of our client library/nova-compute interaction is still there | 14:47 |
| TheJulia | ugh, where was that... | 14:48 |
| dtantsur | it was in our devstack plugin, no? | 14:49 |
| *** makowals has quit IRC | 14:52 | |
| *** makowals has joined #openstack-ironic | 14:53 | |
| TheJulia | the second restart | 14:54 |
| TheJulia | but, it looks like the general service restart is where things go sideways | 14:54 |
| TheJulia | http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/grenade.sh.txt.gz#_2018-01-18_05_50_50_994 ? | 14:55 |
| ameeda | hi | 14:56 |
| ameeda | what about this log "compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-DNSSEC loop-detect inotify" | 14:56 |
| ameeda | from this "journalctl -u openstack-ironic-inspector-dnsmasq" | 14:56 |
| dtantsur | this is dnsmasq's start up output. what's wrong with it? | 14:57 |
| ameeda | I am not sure about it, just asking | 14:57 |
| *** jaganathan has quit IRC | 14:58 | |
| ameeda | can I describe my status to help me ? | 14:58 |
| TheJulia | my last error log was unrelated it looks like :( Still digging | 15:02 |
| *** jcoufal has joined #openstack-ironic | 15:05 | |
| TheJulia | it kind of looks like the redirect in apache is just not loading | 15:11 |
| TheJulia | ameeda: By all means, some of us are troubleshooting a major CI issue at the moment, but pose your questions/issues and hopefully someone might have seen it before or be able to point you in the right direction | 15:12 |
| *** makowals has quit IRC | 15:12 | |
| TheJulia | rloo: dtantsur: http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/grenade.sh.txt.gz#_2018-01-18_05_50_47_744 | 15:14 |
| sambetts | ameeda: do you see dnsmasq actually receiving the DHCP requests from the baremetal? | 15:15 |
| sambetts | ameeda: dnsmasq will only hand out IPs on a network that it as an IP address in, so does your DNSmasq host have an IP address e.g. x.x.x.2 in your inspection network? | 15:15 |
| rloo | TheJulia: hmm. What changed there? (blue moon maybe?) | 15:16 |
| * dtantsur watches the storm status going to RED in Berlin and thinks he won't get wine today.. | 15:16 | |
| sambetts | dtantsur: woah?! | 15:17 |
| dtantsur | yep :( | 15:17 |
| dtantsur | TheJulia, rloo, maybe we need this apache reload here https://github.com/openstack/ironic/blob/master/devstack/upgrade/upgrade.sh#L102 ? | 15:17 |
| sambetts | dtantsur: it pretty crazy windy here in the south of england too, my garden fence is going crazy and I think is broken | 15:17 |
| sambetts | dtantsur: but I don't think the warnings are Red | 15:17 |
| dtantsur | sambetts: there are videos about flying roofs all over the middle of Germany | 15:18 |
| sambetts | :O | 15:18 |
| dtantsur | check out FB | 15:18 |
| ameeda | TheJulia: ok I will post my issue, Thanks | 15:18 |
| TheJulia | rloo: dtantsur: looking at https://github.com/openstack-dev/devstack/blame/master/lib/apache I'm wondering if the underlying OS changed behavior | 15:18 |
| *** rpioso|afk is now known as rpioso | 15:19 | |
| rpioso | Good morning, all | 15:20 |
| ameeda | sambetts: dnsmasq log : http://paste.openstack.org/show/646916/ | 15:20 |
| ameeda | sambetts: how can I check that inspection network has IP ? | 15:21 |
| TheJulia | looks like keystone is also hitting it on ubuntu, so I'm thinking devstack fix... | 15:21 |
| TheJulia | http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/grenade.sh.txt.gz#_2018-01-18_04_37_17_449 | 15:21 |
| *** hamzy has quit IRC | 15:21 | |
| openstackgerrit | Rushil Chugh proposed openstack/ironic master: Add XClarity Driver https://review.openstack.org/519427 | 15:23 |
| * TheJulia looks at the journal file | 15:25 | |
| rloo | So that 'To activate the new configuration, you need to run...' seems normal. i see that on a multi-grenade test that works. | 15:25 |
| rloo | wondering if there is a check anywhere to see that the redirect works. | 15:25 |
| TheJulia | hmmm... but apache never seems to actually load the config, and it complains in its error log when keystone is down | 15:26 |
| TheJulia | rloo: journal file most likely | 15:26 |
| TheJulia | so it looks like, via the apache log, that it did actually reload 2 seconds after | 15:29 |
| rloo | this is where it is restarted i think: http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/grenade.sh.txt.gz#_2018-01-18_05_50_48_472 | 15:29 |
| TheJulia | http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/screen-n-cpu.txt.gz#_Jan_18_05_45_57_057077 | 15:34 |
| TheJulia | err, unrelated, service down | 15:37 |
| *** MaxPC has joined #openstack-ironic | 15:37 | |
| *** links has quit IRC | 15:38 | |
| TheJulia | so, systemd truly never actually restarts n-cpu | 15:39 |
| TheJulia | 5:48 is the last entry in n-cpu log, 5:50 is when the redirect was loaded | 15:39 |
| TheJulia | it looks like, based on our plugin, while we were not expecting it to not be running | 15:43 |
| TheJulia | so, looks like we restart apache in start_ironic | 15:43 |
| rloo | TheJulia: according to the comments in upgrade.sh, we don't restart n-cpu cuz we didn't upgrade nova. | 15:47 |
| *** hjensas is now known as hjensas|afk | 15:48 | |
| *** zshi has quit IRC | 15:48 | |
| *** alex_xu has quit IRC | 15:50 | |
| *** jerrygb has joined #openstack-ironic | 15:50 | |
| TheJulia | yeah, looks like nova is just dying on its own and being marked as failed | 15:50 |
| TheJulia | although the connection refused bits are still confusing | 15:50 |
| openstackgerrit | Rushil Chugh proposed openstack/ironic master: Add XClarity Driver https://review.openstack.org/519427 | 15:51 |
| *** jerrygb has left #openstack-ironic | 15:51 | |
| *** alex_xu has joined #openstack-ironic | 15:52 | |
| TheJulia | i _suspect_ we might need to just remove the conditional from https://github.com/openstack/ironic/blob/master/devstack/upgrade/upgrade.sh#L109-L116 | 15:52 |
| *** zshi has joined #openstack-ironic | 15:54 | |
| rloo | TheJulia: odd that nova compute would just fail. can we just do a 'if nova-compute-not-running, start_nova_compute' before L122? | 15:55 |
| rloo | TheJulia: oh wait. even though we don't want to upgrade nova, is the upgraded nova code going to be used if we restart nova? | 15:55 |
| TheJulia | I'm trying to verify that it is or is not upgraded at the moment | 15:56 |
| rloo | TheJulia: i wonder if the old nova doesn't work well with some new libraries or something. | 15:56 |
| dtantsur | jroll: maybe you have a useful hint: https://bugzilla.redhat.com/show_bug.cgi?id=1535637 | 15:56 |
| openstack | bugzilla.redhat.com bug 1535637 in openstack-ironic "Free disk space check (for raw image) ignores sparseness" [Unspecified,New] - Assigned to rhos-maint | 15:56 |
| jroll | dtantsur: I think "disk size: 7.0G" is the raw size, no? | 15:57 |
| jroll | or I guess they say it's "close", hrm | 15:58 |
| dtantsur | jroll: I hope so, but I don't know for sure.. | 15:58 |
| jroll | dtantsur: yeah idk this stuff well offhand, though could investigate | 15:59 |
| TheJulia | so we are not upgrading it, it has ironicclient 1.17 installed when the grenade job fails | 15:59 |
| TheJulia | rloo: I truly think systemd is just going "oh, this looks like it is not working, let me kill it for you" | 15:59 |
| dtantsur | I remember a change moving ironicclient initialization from __init__ to init_host | 15:59 |
| *** stendulker has joined #openstack-ironic | 15:59 | |
| *** dpawar has joined #openstack-ironic | 15:59 | |
| TheJulia | except it is already running, and is just spewing errors because ironic is down for the upgrade sequence | 16:00 |
| rloo | TheJulia: i suppose it won't hurt to push up a patch | 16:00 |
| *** dpawar has quit IRC | 16:00 | |
| rloo | why would systemd decide to be helpful? | 16:00 |
| rloo | all of a sudden... | 16:00 |
| TheJulia | rloo: perhaps because we are curesed? | 16:00 |
| TheJulia | i dunno | 16:00 |
| TheJulia | maybe timing | 16:00 |
| rloo | TheJulia: oh, i was thinking 'lucky' | 16:00 |
| TheJulia | we are definitely not lucky | 16:01 |
| TheJulia | anyway, finally switching laptops | 16:01 |
| openstackgerrit | Mark Goddard proposed openstack/ironic master: WIP: API: Node Traits API https://review.openstack.org/532269 | 16:02 |
| TheJulia | woot, network plumbing broken to my dev vm | 16:03 |
| *** dpawar has joined #openstack-ironic | 16:04 | |
| stendulker | Good morning all, TheJulia, rloo, dtantsur, sambetts | 16:04 |
| rloo | morning stendulker | 16:05 |
| *** milan has quit IRC | 16:05 | |
| *** hamzy has joined #openstack-ironic | 16:05 | |
| dtantsur | hey stendulker | 16:05 |
| *** mjura has quit IRC | 16:06 | |
| stendulker | rloo: Wanted to discuss about your comment of whether to add 'mode' argument to all in-tree boot interfaces methods prepare_ramdisk and clean_up_ramdisk | 16:06 |
| stendulker | dtantsur: Hi | 16:06 |
| rloo | stendulker: i haven't had a chance to look at any comments you made since yesterday | 16:07 |
| rloo | stendulker: what do you want to discuss exactly | 16:07 |
| *** hamzy has quit IRC | 16:07 | |
| stendulker | rloo: I think we do not need to do that with new proposed check in validate() | 16:07 |
| *** hamzy has joined #openstack-ironic | 16:07 | |
| rloo | stendulker: should i read something/a comment first, before having this discussion? | 16:07 |
| stendulker | rloo: https://review.openstack.org/#/c/400437/85/ironic/drivers/modules/ilo/boot.py@416 | 16:08 |
| patchbot | patch 400437 - ironic - Agent rescue implementation | 16:08 |
| rloo | stendulker: i don't see any comments there from you | 16:08 |
| *** makowals has joined #openstack-ironic | 16:08 | |
| stendulker | rloo: I have not added | 16:08 |
| rloo | stendulker: ok, it is coming back to me now | 16:08 |
| rloo | stendulker: i am fine if we don't make that change | 16:09 |
| rloo | stendulker: is that what you wanted to know? | 16:09 |
| rloo | stendulker: with the check in validate(), it is probably better not to have that change, so we know what supports/doesn't support | 16:09 |
| stendulker | rloo: ok. also wanted to let you know that we would need 'mode' flag for boot.validate as well | 16:09 |
| rloo | stendulker: oh geez. really? | 16:10 |
| stendulker | we need to call boot.validate() in rescue.validate() | 16:10 |
| stendulker | when manage_boot is true | 16:10 |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Always attempt to restart n-cpu https://review.openstack.org/535391 | 16:10 |
| rloo | stendulker: but does code in boot.validate() need to change depending on deploy or rescue? | 16:10 |
| stendulker | we would be calling prepare_ramdisk | 16:11 |
| TheJulia | folks: If ^^^ works, I'll follow-up with a lp bug and amend the script in a follow-up patch so we can get backup and running asap | 16:11 |
| NobodyCam | Good Morning Ironic'ers | 16:11 |
| stendulker | yes, as when we call boot.validate, node provision state is active | 16:11 |
| rloo | stendulker: would it make sense to check the node state in boot.validate(), to get the mode? | 16:12 |
| *** makowals has quit IRC | 16:12 | |
| stendulker | rloo: we cannot determine how that call has been made | 16:12 |
| mgoddard | rloo, jroll: ^^ now with added RPCs | 16:12 |
| stendulker | we will get state as 'active' and not one of rescuing... | 16:12 |
| rloo | stendulker: and folks thought that the rescue work was done, just needed reviews? :-( | 16:13 |
| rloo | stendulker: sorry. just a bit not optimistic today. | 16:13 |
| stendulker | rloo: I'm sorry, but this agent patch needs bit of work. | 16:13 |
| *** AlexeyAbashkin has quit IRC | 16:13 | |
| rloo | stendulker: off the top of my head, i don't have a solution, but i don't like adding 'mode' to boot.validate(). it makes it inconsistent with all the other Interface.validate()s | 16:14 |
| rloo | stendulker: maybe this is a problem with our code/design/architect, not sure. | 16:14 |
| stendulker | rloo: yes, it would make it incosnsitent | 16:14 |
| rloo | stendulker: yes, i realize the agent patch needs a bit of work. just wish that work had been done sooner :) | 16:14 |
| stendulker | rloo: Another thing is we can live without it... | 16:14 |
| *** hamzy_ has joined #openstack-ironic | 16:14 | |
| stendulker | rloo: We would not be able to skip certain boot.validate() checks based on rescuing state | 16:15 |
| rloo | stendulker: i am fine if we live w/o it, i think anyway. you can add a TODO or something? | 16:15 |
| stendulker | rloo: Also it would un-necessarily check for deploy kernel/ramdisk params | 16:15 |
| rloo | stendulker: i don't know what checks are made, but if we are rescuing, hopefully we have deployed and nothing has changed so that boot.validate() would fail? | 16:15 |
| stendulker | rloo: We do check resccue kernel/ramdisk in rescue.validate, so should be ok, i suppose for the time being | 16:16 |
| stendulker | rloo: Yes, nothing should fail. | 16:16 |
| rloo | stendulker: alternative is to add a new method Boot.validate_rescue() but I don't like that either. | 16:16 |
| rloo | stendulker: i'm good with a TODO for now. | 16:16 |
| stendulker | rloo: Some checks for instance image and BFV | 16:16 |
| rloo | stendulker: a TODO wherever boot.validate() is called in the rescue-related code | 16:17 |
| *** hamzy has quit IRC | 16:17 | |
| stendulker | rloo: Ok, will do that. | 16:17 |
| stendulker | rloo: Thank you. | 16:17 |
| rloo | stendulker: thank you! | 16:17 |
| openstackgerrit | Mark Goddard proposed openstack/ironic master: WIP: API: Node Traits API https://review.openstack.org/532269 | 16:18 |
| *** e0ne has quit IRC | 16:23 | |
| *** crushil has joined #openstack-ironic | 16:23 | |
| *** pcaruana has quit IRC | 16:29 | |
| *** deray has joined #openstack-ironic | 16:33 | |
| deray | dtantsur, hi | 16:33 |
| deray | dtantsur, shouldn't Sushy's major version be updated? | 16:36 |
| dtantsur | deray: why? | 16:36 |
| deray | dtantsur, refresh() API's functionality changes.. | 16:36 |
| deray | dtantsur, Sushy Connector c'tor also changes | 16:37 |
| dtantsur | deray: what's the second change? | 16:37 |
| dtantsur | but yeah, refresh() change may be confusing.. | 16:38 |
| deray | dtantsur, earlier Connector c'tor used to be passed on username/password .. now those are not passed anymore .. those are stored as part of Auth() | 16:38 |
| deray | dtantsur, altho Suhsy root c'tor also changes but its backward compatible anyway | 16:39 |
| jlvillal | Good morning Ironic | 16:39 |
| dtantsur | morning jlvillal | 16:39 |
| deray | jlvillal, g'morning | 16:39 |
| dtantsur | deray: Connector is not a part of the API.. the refresh() change worries me | 16:40 |
| dtantsur | I wonder if we did the right thing with it | 16:40 |
| *** dpawar has quit IRC | 16:40 | |
| *** crushil has quit IRC | 16:40 | |
| jlvillal | dtantsur, sambetts: Regarding the oslo.serialization, I spent some time and think I figured out the issue in Glance. Basically they would try to serialize and Exception and if any Exception was raised then they would return a value. With change no longer raising an exception it broke them. | 16:41 |
| deray | dtantsur, why Connector is not part of public ly exposed classes? | 16:41 |
| jlvillal | The patch: https://review.openstack.org/535166 if interested to look | 16:41 |
| patchbot | patch 535166 - glance - Resolve unit test failures with going to oslo.seri... | 16:41 |
| dtantsur | deray: probably because before your change it was not useful | 16:41 |
| jlvillal | We will see if they approve it or not. | 16:41 |
| deray | dtantsur, but proliantutils were already extending Connector for its own purpose | 16:42 |
| *** fragatina has joined #openstack-ironic | 16:43 | |
| dtantsur | then we probably should not have removed the arguments.. | 16:43 |
| *** fragatina has quit IRC | 16:44 | |
| dtantsur | but I'm more worried about refresh(). with what we changed it's no longer refresh() | 16:44 |
| *** fragatina has joined #openstack-ironic | 16:44 | |
| deray | dtantsur, agreed for Connector.. but for refresh() I think the current behaviour is more apt than the older one. Only issue is it needs to be invalidated first to have the refresh to take effect.. | 16:45 |
| dtantsur | well, right, so refresh is no longer refreshing :) | 16:45 |
| dtantsur | which is.. bad | 16:46 |
| deray | unless invalidated first, yes :) | 16:46 |
| deray | refresh(true) will anyway do .. | 16:46 |
| dtantsur | and it has to be the default | 16:47 |
| dtantsur | I guess we have to flip force to True in https://github.com/openstack/sushy/blob/master/sushy/resources/base.py#L247 | 16:47 |
| dtantsur | TheJulia: ^^^ | 16:47 |
| rloo | good morning NobodyCam, jlvillal | 16:47 |
| rloo | thx mgoddard, hope to look at them soon | 16:48 |
| openstackgerrit | Rushil Chugh proposed openstack/ironic master: Add XClarity Driver https://review.openstack.org/519427 | 16:48 |
| rloo | For anyone not paying attention, the gate is broken, I've updated the whiteboard. Please DO NOT recheck | 16:48 |
| deray | dtantsur, yeah.. it makes me feel so | 16:48 |
| NobodyCam | Good morning rloo :) | 16:48 |
| *** jaganathan has joined #openstack-ironic | 16:49 | |
| deray | dtantsur, TheJulia I can take care of that change .. but alas I can only work on that tomorrow morning my time :( | 16:50 |
| dtantsur | deray: I'll handle it, thanks | 16:50 |
| TheJulia | dtantsur: ugh :( | 16:50 |
| TheJulia | dtantsur: you going to wip -1 your releases repo patch? | 16:50 |
| dtantsur | already | 16:50 |
| TheJulia | awesome | 16:50 |
| deray | dtantsur, thank you! | 16:52 |
| dtantsur | np | 16:53 |
| dtantsur | sorry for not noticing it earlier | 16:53 |
| deray | dtantsur, never mind .. better late than never :) | 16:53 |
| *** amarao1 has quit IRC | 16:54 | |
| openstackgerrit | Michael Turek proposed openstack/ironic-python-agent master: Use lshw in place of dmidecode for the default hardware manager https://review.openstack.org/501799 | 16:54 |
| deray | dtantsur, I also need to take care of the corresponding changes in proliantutils (now that it uses Sushy for all Redfish related stuff) | 16:54 |
| *** marios has quit IRC | 16:56 | |
| deray | dtantsur, TheJulia and others catch you tomorrow then.. good day ahead. bye! | 16:56 |
| TheJulia | goodnight deray | 16:56 |
| deray | dtantsur, TheJulia btw there's another Sushy dependent lib I am aware of: https://github.com/openstack/rsd-lib | 16:58 |
| deray | TheJulia, good day :) | 16:58 |
| *** crushil has joined #openstack-ironic | 16:59 | |
| *** deray has quit IRC | 17:00 | |
| sambetts | jlvillal: I saw! Thanks for pushing on that I really thought it wasn't your patches fault that glance was broken ... I hate that the first response to glances ut breaking was well it can't be our fault... | 17:01 |
| jlvillal | sambetts, heh. True :) | 17:02 |
| openstackgerrit | Dmitry Tantsur proposed openstack/sushy master: Restore the default semantics of refresh() https://review.openstack.org/535406 | 17:03 |
| dtantsur | TheJulia and others ^^^ | 17:03 |
| dtantsur | now lemme address another potentially breaking change | 17:03 |
| *** linkmark has joined #openstack-ironic | 17:03 | |
| openstackgerrit | Ruby Loo proposed openstack/ironic master: Add rescuewait timeout periodic task https://review.openstack.org/353156 | 17:04 |
| *** jcoufal has quit IRC | 17:13 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/sushy master: Restore interface of Connector https://review.openstack.org/535413 | 17:17 |
| dtantsur | TheJulia, rloo, jlvillal, and the second one ^^^ | 17:17 |
| rloo | dtantsur: #@$#% Let me know if you need me to +2. I haven't paid attention to those patches, so it'd be a blind-i-trust-you thing. | 17:18 |
| dtantsur | rloo: they are relatively trivial from Python point of view.. | 17:18 |
| dtantsur | no idea where my eyes were when reviewing all this.. | 17:19 |
| dtantsur | TheJulia: can you take it from here please? We need these two patches merged, then repropose the release? | 17:19 |
| dtantsur | I'll be around, but I'd avoid sitting in front of the computer till night | 17:19 |
| TheJulia | dtantsur: no problem, I've got it | 17:20 |
| dtantsur | awesome, thanks a lot! | 17:20 |
| TheJulia | dtantsur: I'm about to post comments to your latest comments | 17:20 |
| TheJulia | on the clien | 17:20 |
| TheJulia | t | 17:20 |
| dtantsur | awesome! btw the outcome of our today's API-SIG discussion was "Let's stop writing low-level SDKs" :) but that's for the future, bright and shiny | 17:21 |
| rloo | mgoddard: by the way, wrt https://review.openstack.org/#/c/532268/. I have comments in PS5 that weren't addressed yet. PS6 was only a rebase. | 17:22 |
| patchbot | patch 532268 - ironic - Add RPC object for traits | 17:22 |
| dtantsur | rloo: yeah, we need +2 on these, since we're supposed to release today :( | 17:23 |
| dtantsur | I hope these patches are quite straightforward | 17:23 |
| rloo | dtantsur, TheJulia: please ping me. I'll keep an eye out for them. | 17:23 |
| *** crushil has quit IRC | 17:25 | |
| dtantsur | thanks rloo! | 17:25 |
| *** amarao has quit IRC | 17:26 | |
| TheJulia | dtantsur: I need to ponder the one thing, but should be able to update the ironicclient patches today | 17:38 |
| *** sambetts is now known as sambetts|afk | 17:39 | |
| dtantsur | cool! | 17:39 |
| rloo | dtantsur: reviewed and commented on sushy patches, even though i don't know what is going on :) | 17:39 |
| dtantsur | thanks! | 17:39 |
| dtantsur | anything requiring an urgent update? | 17:39 |
| rloo | dtantsur: i don't think urgent or I wouldnt' have +2. I leave it for you to decide. | 17:40 |
| *** athomas has quit IRC | 17:42 | |
| * TheJulia realizes that the podcast recording she will be on later today will be a nice break from the day | 17:43 | |
| *** stendulker_ has joined #openstack-ironic | 17:46 | |
| * dtantsur is semi-offline now | 17:48 | |
| *** stendulker has quit IRC | 17:49 | |
| crushil_ | Hi. I have a doubt regarding py35. I have updated https://review.openstack.org/#/c/519427/32/ironic/tests/unit/drivers/third_party_driver_mocks.py file with the following http://paste.openstack.org/show/ivYEOAWaQNC8hKkLfpZL/ and it still fails with the error TypeError: catching classes that do not inherit from BaseException is not allowed. Any idea what I am missing? | 17:50 |
| patchbot | patch 519427 - ironic - Add XClarity Driver | 17:50 |
| *** stendulker_ has quit IRC | 17:51 | |
| *** penick has joined #openstack-ironic | 17:53 | |
| jroll | crushil_: I think you want sys.modules['xclarity_client.exceptions'] = xclarity_client_exceptions | 17:55 |
| jroll | alternatively, xclarity_client_exceptions.exceptions.XClarityException | 17:56 |
| jroll | er | 17:56 |
| jroll | alternatively, xclarity_client_exceptions.exceptions.XClarityException = Exception | 17:56 |
| openstackgerrit | Dmitry Tantsur proposed openstack/sushy master: Restore the default semantics of refresh() https://review.openstack.org/535406 | 17:57 |
| dtantsur | TheJulia, rloo, I hope that's it now ^^^ | 17:57 |
| rloo | dtantsur: we all hope :) | 17:57 |
| dtantsur | now I really stand up from the computer, will check for pings from time to time | 17:57 |
| *** penick has quit IRC | 17:58 | |
| *** dtantsur is now known as dtantsur|afk | 17:58 | |
| TheJulia | dtantsur|afk: go have a beer or something | 17:58 |
| *** AlexeyAbashkin has joined #openstack-ironic | 17:58 | |
| dtantsur|afk | wine! we have wine for today | 17:59 |
| jroll | \o/ | 17:59 |
| * dtantsur|afk sees potential bikeshedding on "force" argument and runs away | 17:59 | |
| jroll | blue! | 17:59 |
| *** tesseract has quit IRC | 18:00 | |
| *** derekh has quit IRC | 18:00 | |
| crushil_ | jroll, That didn't work. :( | 18:01 |
| jroll | hrm | 18:01 |
| *** AlexeyAbashkin has quit IRC | 18:03 | |
| TheJulia | purple! Purple beer! | 18:04 |
| TheJulia | rloo: looks like 535391 fixes grenade. The jobs are not done yet, but they completed upgrading and are testing | 18:07 |
| rloo | TheJulia: yay. | 18:07 |
| rloo | TheJulia: so we need a 'real' patch etc. that we can approve. | 18:07 |
| TheJulia | if you want do a quick review, I'm going to start writing up a detailed bug entry in launchpad | 18:07 |
| rloo | TheJulia: oh yeah, that. heh. sec. | 18:08 |
| TheJulia | rloo: well, that one has to be rechecked, but that one was intended to be able to be just +a'ed if the universe aligned | 18:08 |
| TheJulia | oh is there already a bug? | 18:08 |
| rloo | TheJulia: doubt it, thought you were going to open one | 18:08 |
| rloo | TheJulia: i meant, 'that' == 'me review' | 18:08 |
| TheJulia | (rechecked due to a post-failure and timeout) :( | 18:08 |
| TheJulia | oh, yeah, take a look, I was just going to update the commit message at this point | 18:09 |
| rloo | jlvillal, sambetts|afk, vdrok, anyone that cares, here's TheJulia's patch to fix multinode grenade: https://review.openstack.org/#/c/535391/ | 18:12 |
| patchbot | patch 535391 - ironic - Always attempt to restart n-cpu | 18:12 |
| *** AlexeyAbashkin has joined #openstack-ironic | 18:14 | |
| openstackgerrit | Adam Kimball proposed openstack/ironic master: Add OpenBMC class for accessing nodes using OpenBMC https://review.openstack.org/429836 | 18:17 |
| *** penick has joined #openstack-ironic | 18:18 | |
| *** penick has quit IRC | 18:18 | |
| openstackgerrit | Michael Turek proposed openstack/ironic-python-agent master: Use lshw in place of dmidecode for the default hardware manager https://review.openstack.org/501799 | 18:18 |
| *** penick has joined #openstack-ironic | 18:18 | |
| *** penick has quit IRC | 18:19 | |
| *** AlexeyAbashkin has quit IRC | 18:19 | |
| *** penick has joined #openstack-ironic | 18:19 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Always attempt to restart n-cpu https://review.openstack.org/535391 | 18:19 |
| *** penick has quit IRC | 18:20 | |
| *** penick has joined #openstack-ironic | 18:20 | |
| *** penick has quit IRC | 18:20 | |
| *** penick has joined #openstack-ironic | 18:21 | |
| *** penick has quit IRC | 18:21 | |
| *** AlexeyAbashkin has joined #openstack-ironic | 18:22 | |
| TheJulia | rloo: ^^^ | 18:23 |
| TheJulia | added the bug | 18:23 |
| rloo | TheJulia: already +2'd. we need another victim. err. core. | 18:24 |
| * TheJulia sets out beer next to the patch | 18:24 | |
| * dtantsur|afk appears from the shadows | 18:24 | |
| TheJulia | I didn't put wine out... | 18:24 |
| TheJulia | :) | 18:24 |
| * jroll gives his max score :( | 18:24 | |
| jroll | it's okay though, I have good beer waiting in the fridge | 18:25 |
| TheJulia | :( and :) | 18:25 |
| TheJulia | smiley only being beer though | 18:25 |
| * dtantsur|afk disappears back in the shadows | 18:26 | |
| jroll | heh | 18:26 |
| *** AlexeyAbashkin has quit IRC | 18:26 | |
| rloo | good bait... | 18:26 |
| TheJulia | one sushy change in the gate, the other started | 18:31 |
| TheJulia | vdrok: good point.... I remember when we ran into that. :\ I ultimately suspect the cleaner/best option would be to start the redirect far earlier, since n-cpu blows up with the client errors... | 18:41 |
| TheJulia | Didn't we talk about at one point, about directly publishing into placement? | 18:41 |
| * jroll wants to | 18:42 | |
| vdrok | yes we did, but i dont think placement supports it yet | 18:42 |
| vdrok | at least ive seen an email regarding that in ml recently | 18:42 |
| jroll | I think the decision between us and nova was to let placement stabilize a bit first, and/or be broken out from nova | 18:42 |
| vdrok | also good morning all :) | 18:42 |
| jroll | morning vdrok :) | 18:43 |
| vdrok | o/ | 18:43 |
| rloo | good morning vdrok | 18:43 |
| TheJulia | good morning vdrok | 18:43 |
| vdrok | or was it about adding ironic nodes into placement aggregates?.. not sure | 18:43 |
| * TheJulia doesn't remember, ETOOMUCHCONTEXTNEEDED | 18:45 | |
| TheJulia | actually, it is more like the context swap runs out of ram | 18:47 |
| jroll | vdrok: without aggregates, nodes are just a resource provider, it definitely supports it | 18:47 |
| TheJulia | I seem to remember mention of some sort of data/structural issue in sydney... :( | 18:50 |
| *** edleafe- has joined #openstack-ironic | 18:55 | |
| *** edleafe has quit IRC | 18:58 | |
| openstackgerrit | Merged openstack/sushy master: Restore interface of Connector https://review.openstack.org/535413 | 18:59 |
| *** hjensas|afk is now known as hjensas | 19:00 | |
| TheJulia | woot, one down, one to go | 19:03 |
| TheJulia | Additional disucssion/comments in the review. Still thinking we need to just go ahead and land, move forward from there. Seems like we're juggling a few distinct issues that are compounding :\ | 19:04 |
| rloo | TheJulia: you mean 535391? yeah, we need to land it if it fixes our gate. Then we better do a better fix so it doesn't break later. | 19:11 |
| *** edleafe- is now known as edleafe | 19:12 | |
| *** dprince has quit IRC | 19:18 | |
| TheJulia | There are many variables :( | 19:18 |
| openstackgerrit | Merged openstack/sushy master: Restore the default semantics of refresh() https://review.openstack.org/535406 | 19:23 |
| *** tzumainn has quit IRC | 19:29 | |
| TheJulia | release for sushy updated | 19:35 |
| TheJulia | https://review.openstack.org/#/c/535357/ | 19:36 |
| patchbot | patch 535357 - releases - Release sushy 1.3.0 (final) for Queens | 19:36 |
| *** harlowja has joined #openstack-ironic | 19:52 | |
| *** dprince has joined #openstack-ironic | 19:54 | |
| rloo | yay, red with pink highlights are gone! long live black! https://docs.openstack.org/ironic/latest/admin/interfaces/boot.html | 20:09 |
| rloo | (compared to pike's red/pink, eg: https://docs.openstack.org/ironic/pike/admin/drivers/ipa.html) | 20:11 |
| *** e0ne has joined #openstack-ironic | 20:15 | |
| *** jlvillal has left #openstack-ironic | 20:23 | |
| *** jlvillal has joined #openstack-ironic | 20:23 | |
| openstackgerrit | OpenStack Release Bot proposed openstack/sushy master: Update reno for stable/queens https://review.openstack.org/535475 | 20:26 |
| *** fragatina has quit IRC | 20:30 | |
| *** hw_wutianwei has quit IRC | 20:31 | |
| *** jlvillal has quit IRC | 20:31 | |
| *** jlvillal has joined #openstack-ironic | 20:31 | |
| *** hw_wutianwei has joined #openstack-ironic | 20:32 | |
| *** jlvillal has left #openstack-ironic | 20:33 | |
| *** jlvillal has joined #openstack-ironic | 20:33 | |
| *** e0ne has quit IRC | 20:39 | |
| *** penick has joined #openstack-ironic | 20:39 | |
| TheJulia | rloo: back to the drawing board http://logs.openstack.org/91/535391/2/check/ironic-grenade-dsvm-multinode-multitenant/09d8568/logs/screen-n-cpu.txt.gz | 20:44 |
| rloo | TheJulia: :-( | 20:44 |
| *** AlexeyAbashkin has joined #openstack-ironic | 20:45 | |
| rloo | TheJulia: looks like what vdrok was warning about | 20:46 |
| * TheJulia spends five minutes of quality time with her cat while thinking about possible options | 20:46 | |
| rloo | why did systemd all of a sudden decide to fail the compute process? | 20:47 |
| rloo | vdrok: ^^ if you have any ideas... | 20:49 |
| *** AlexeyAbashkin has quit IRC | 20:49 | |
| * vdrok looks | 20:49 | |
| TheJulia | Well, that time we stopped nova, but when we went to start it, all of a sudden there was an unxpected parameter because of incompatabilities | 20:51 |
| *** penick has quit IRC | 20:51 | |
| TheJulia | funny thing is, looks like the other job was passing | 20:52 |
| *** penick has joined #openstack-ironic | 20:52 | |
| TheJulia | but, likely compatible libraries since they re differnet scarios if memory serves | 20:52 |
| vdrok | but nova itself should be upgraded already right? | 20:54 |
| rloo | vdrok: no. we don't want nova to be upgraded. | 20:55 |
| rloo | vdrok: that's why we didn't stop/start it in the first place. | 20:55 |
| vdrok | ah, right, we upgrade ironic only. and grenade script seems to have succeeded | 20:55 |
| TheJulia | anyone know why we don't upgrade nova? | 20:56 |
| vdrok | well if we would it would be upgraded before ironic | 20:56 |
| vdrok | which is not the case we want to test :) | 20:57 |
| TheJulia | I suspect we need to consider tossing that out the window | 20:57 |
| vdrok | and yeah here oslo_utils is of newer version it seems | 20:57 |
| vdrok | or not, hrm | 20:58 |
| TheJulia | what breaks if we upgrade.. potentially version pins on master->master... but if we land new logic, then it should be fine.... | 20:58 |
| rloo | vdrok: i think it is oslo.db | 20:58 |
| rloo | vdrok: that is newer verson | 20:58 |
| TheJulia | I bet utils is just passing kwargs or something like that | 20:58 |
| rloo | vdrok: cuz nova error is 'retry_on_request' not there | 20:58 |
| TheJulia | hmm | 20:59 |
| rloo | here it is: https://github.com/openstack/oslo.db/commit/4a727233dc2b7d1021ed7932d2cd27d8053dca31 | 20:59 |
| TheJulia | okay, so just restarting is not viable.... | 21:00 |
| rloo | so we cannot restart an old process if we've upgraded to new libraries | 21:00 |
| rloo | unless we use containers or something but we're not going there. | 21:00 |
| vdrok | right oslodb | 21:00 |
| TheJulia | what if we upgraded nov anad restarted, it would e "out of order" which owuld break pinning for nova (i think) (if we tried to change it without fixing it first) | 21:00 |
| TheJulia | nova and | 21:00 |
| rloo | so again, why did systemd kill nova compute? | 21:00 |
| * vdrok goes to lunch, sry | 21:00 | |
| rloo | vdrok: don't be sorry. i'm sorry! | 21:01 |
| TheJulia | why does systemd suddenly change? | 21:01 |
| rloo | TheJulia: that's my question! :) | 21:02 |
| * TheJulia kind of feels like asking why systemd behavior has changed is sort of in the same class of Star Trek V and "why does god need a starship?" | 21:02 | |
| rloo | TheJulia: no way, i am not giving systemd that much aweness! | 21:02 |
| TheJulia | lol | 21:02 |
| TheJulia | oh, I wasn't suggesting that! I was just thinking that its behavior seems to change suddenly | 21:03 |
| TheJulia | and that seems to be "normal" for systemd | 21:03 |
| rloo | TheJulia: do we know for sure that it is systemd? | 21:03 |
| rloo | TheJulia: next time we try a test, in the patch, we should disable all the tests except the one we're interested in :) | 21:04 |
| TheJulia | rloo: OMG YES! | 21:05 |
| TheJulia | systemd seems to be deciding unilatirally that the service has failed. Nothing in the other logs that I've spotted... | 21:05 |
| TheJulia | hmm, I didn't notice this last time http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/syslog.txt.gz#_Jan_18_05_50_08 | 21:07 |
| openstackgerrit | Mark Goddard proposed openstack/ironic master: Add RPC object for traits https://review.openstack.org/532268 | 21:08 |
| openstackgerrit | Mark Goddard proposed openstack/ironic master: Add RPC API and conductor manager for traits https://review.openstack.org/535296 | 21:08 |
| openstackgerrit | Mark Goddard proposed openstack/ironic master: WIP: API: Node Traits API https://review.openstack.org/532269 | 21:08 |
| openstackgerrit | Mark Goddard proposed openstack/ironic master: Add a version argument to traits DB API https://review.openstack.org/535482 | 21:08 |
| TheJulia | http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/screen-n-cond.txt.gz#_Jan_18_05_51_49_922903 | 21:09 |
| TheJulia | something is seriously going sideways | 21:09 |
| jroll | segfaults :o | 21:11 |
| rloo | TheJulia: i was trying to recall why ironic has to start before nova, and it is the thing you're trying to fix, if we start nova-that-has-ironic-api-version-avail-only-in-new-ironic-api, it'll barf. in our case, i don't think we've up'd the ironic version in nova, so we could upgrade nova first. | 21:11 |
| *** Nisha_Agarwal has joined #openstack-ironic | 21:13 | |
| * TheJulia wonders if it is time to just move grenade to py3 and hope | 21:13 | |
| rloo | nova is using 1.35, a queens release. so it won't work. | 21:13 |
| TheJulia | rloo: but there is logic in grenade to handle the case of n-cpu barfing on initial upgrade start | 21:13 |
| TheJulia | if it is ironic, because of the upgrade order | 21:14 |
| rloo | TheJulia: oh really? | 21:14 |
| TheJulia | oh | 21:14 |
| TheJulia | OH | 21:14 |
| TheJulia | because we never update ironic-api | 21:14 |
| rloo | TheJulia: well, it won't hurt to push up a patch to see what happens. | 21:14 |
| TheJulia | we never start an up to date endpoint | 21:14 |
| rloo | TheJulia: oh. yeah. we want to use the old ironic-api. that's what we're testing. | 21:14 |
| *** mjturek has quit IRC | 21:14 | |
| TheJulia | There is a reason behind that, but does that reason still make sense? | 21:16 |
| rloo | TheJulia: yes it makes sense. cuz we're testing rolling upgrade -- old conductor + new conductor. | 21:16 |
| rloo | TheJulia: in that state, it has to be old api. | 21:17 |
| rloo | TheJulia: step 1.1 in this table: https://docs.openstack.org/ironic/latest/contributor/rolling-upgrades.html#rolling-upgrade-process | 21:18 |
| rloo | TheJulia: I say we don't test rolling upgrades at all! then we should be good. | 21:18 |
| openstackgerrit | Mark Goddard proposed openstack/ironic master: Add RPC object for traits https://review.openstack.org/532268 | 21:18 |
| openstackgerrit | Mark Goddard proposed openstack/ironic master: Add RPC API and conductor manager for traits https://review.openstack.org/535296 | 21:18 |
| openstackgerrit | Mark Goddard proposed openstack/ironic master: WIP: API: Node Traits API https://review.openstack.org/532269 | 21:18 |
| *** mjturek has joined #openstack-ironic | 21:21 | |
| TheJulia | I bet with the conductor going sideways as well, the nova user might actually be hitting a ulimit or some built in limit mechanism, which might also be related to why the nova-compute process goes sideways possibly | 21:22 |
| *** weshay|rover is now known as weshay|rover|den | 21:24 | |
| *** weshay|rover|den is now known as weshay|dentist | 21:24 | |
| rloo | TheJulia: wha do you think systemd is 'watching' to know/think that compute is failing? | 21:26 |
| TheJulia | http://logs.openstack.org/10/534710/1/check/ironic-grenade-dsvm-multinode-multitenant/079fa3e/logs/grenade.sh.txt.gz#_2018-01-17_11_50_39_417 dynamic linking error | 21:26 |
| rloo | TheJulia: that is swift? | 21:28 |
| TheJulia | rloo: don't entirely no, but it looks like the conductor starts going sideways with segfaults before nova-compute is killed off | 21:29 |
| rloo | TheJulia: oh, that is a diff patch/result. interesting. but then it shouldn't be just us, or is ironic the only one that does this sort of grenade test? | 21:32 |
| TheJulia | I think we are the only ones | 21:32 |
| *** mjturek has quit IRC | 21:32 | |
| rloo | TheJulia: I wonder if all these tests are being run on the same set of test nodes/systems. | 21:32 |
| *** openstackgerrit has quit IRC | 21:33 | |
| *** mjturek has joined #openstack-ironic | 21:34 | |
| TheJulia | rax and inap | 21:37 |
| TheJulia | so far | 21:37 |
| *** AlexeyAbashkin has joined #openstack-ironic | 21:45 | |
| *** openstackgerrit has joined #openstack-ironic | 21:48 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: DNM Test if weirdness exists with grande+py3 https://review.openstack.org/535492 | 21:48 |
| TheJulia | highly technical term of weirdness | 21:48 |
| *** AlexeyAbashkin has quit IRC | 21:50 | |
| TheJulia | rloo: ^ maybe a clue in an 1-1.5 hours. I have a podcast recording strating in about 10 minutes, so consider me semi-afk for the next hour | 21:50 |
| rloo | TheJulia: thx. I'll be gone after that. so good luck if you come back, or hopefully someone else continues looking into it | 21:51 |
| TheJulia | the dynamic linker errors are showing up in a successful grenade job | 21:54 |
| TheJulia | so likely unrelated | 21:54 |
| *** hamzy_ has quit IRC | 21:54 | |
| *** linkmark has quit IRC | 21:55 | |
| *** threestrands has joined #openstack-ironic | 21:56 | |
| *** threestrands has quit IRC | 21:56 | |
| *** threestrands has joined #openstack-ironic | 21:56 | |
| *** rcernin has joined #openstack-ironic | 22:06 | |
| *** etingof has quit IRC | 22:08 | |
| *** Goneri has quit IRC | 22:14 | |
| *** makowals has joined #openstack-ironic | 22:15 | |
| *** fragatina has joined #openstack-ironic | 22:17 | |
| *** penick has quit IRC | 22:24 | |
| *** baha has quit IRC | 22:31 | |
| tonyb | In today's session of silly questions from tonyb ... So I have a node that I inspect (openstack baremetal node inspect compute-p) and then deploy ( openstack server create --image ppc64le-img --flavor baremetal test). What see is the deploy images the machine but when it reboots the system is still configured to netboot and there is a pxe config for inspector that it wants to boot | 22:37 |
| tonyb | any idea what I've done wrong? | 22:38 |
| * TheJulia tries to determine... if all questions are silly or no questions are silly... and what is the best stance on that.... | 22:43 | |
| TheJulia | hmmmm | 22:43 |
| TheJulia | tonyb: It sounds like there are multiple things going on there, I don't explicitely remember pxe configs for inspection being created with inspection related flags for inspection... | 22:45 |
| TheJulia | tonyb: if you ask ironic's API with a newer API version, what the node provision_state is, do you get back what you expect? | 22:45 |
| TheJulia | i.e. is it in available after inspection? | 22:45 |
| tonyb | TheJulia: Yup | 22:46 |
| TheJulia | well, your using osc, so... hmm | 22:46 |
| tonyb | TheJulia: if I interupt the boot at that point (and boot the image I just deployed) it's all really happy times | 22:46 |
| TheJulia | tonyb: is your ironic.conf set to default netboot? | 22:46 |
| tonyb | TheJulia: there is a non-zero chance I've messed up my ironic-inspector somehow | 22:47 |
| TheJulia | it sounds like your falling back to inspection | 22:47 |
| *** dprince has quit IRC | 22:47 | |
| tonyb | default_boot_option=local | 22:48 |
| TheJulia | hmm | 22:48 |
| TheJulia | tonyb: ipmi? | 22:48 |
| tonyb | TheJulia: Yup | 22:48 |
| TheJulia | if you manually query the hardware with ipmi, does it reply with a disk and network options? | 22:49 |
| TheJulia | and if that works, what does ipmi say it presently is set to? | 22:49 |
| TheJulia | there is also get current boot device capability.... We added that to OSC in Pike I believe. | 22:49 |
| * tonyb treis to recall the impitool command to do that ... | 22:50 | |
| *** Nisha_Agarwal has quit IRC | 22:51 | |
| TheJulia | ipmitool lanplus arguments for credentials.... and I'm drawing a blank after that :( | 22:51 |
| jroll | tonyb: should be "chassis bootparam get 5" according to our code | 22:51 |
| jroll | https://github.com/openstack/ironic/blob/master/ironic/drivers/modules/ipmitool.py#L994 | 22:52 |
| tonyb | jroll: cool that's what I did now I'll try to decode it | 22:52 |
| jroll | I think you want 'Boot Device Selector' | 22:52 |
| jroll | (for the current one) | 22:52 |
| TheJulia | rloo: totally forgot about it, but swift is not py3 compatible, so the job blew up on py3 | 22:53 |
| TheJulia | s/on py3/on swift/ | 22:53 |
| rloo | TheJulia: %##$#$, right. | 22:53 |
| rloo | TheJulia: too much to remember :-( | 22:53 |
| tonyb | jroll, TheJulia: that line isn't in my ipmitool output | 22:53 |
| jroll | interesting | 22:54 |
| tonyb | No I'm blind | 22:54 |
| TheJulia | tonyb: if you can share, what is the actual hardware your trying to control? | 22:54 |
| tonyb | - Boot Device Selector : Force Boot from default Hard-Drive | 22:54 |
| TheJulia | oh, okay | 22:54 |
| jroll | ok, that seems right, hrm | 22:54 |
| tonyb | TheJulia: IBM POWER-8 mumble mumble | 22:54 |
| * TheJulia had a feeling that it was something like that | 22:54 | |
| TheJulia | mjturek: oh hai! | 22:54 |
| jroll | ^^ | 22:54 |
| tonyb | TheJulia: ;P | 22:54 |
| jroll | tonyb: you said you can manually make it boot from disk fine, right? | 22:55 |
| tonyb | jroll: Yup | 22:55 |
| jroll | well then | 22:55 |
| TheJulia | tonyb: I know mjturek has been doing some stuff recently on power gear with ironic, so maybe he has some insight | 22:55 |
| tonyb | jroll: I just stop the autoboot and manually boot the disk and all good | 22:55 |
| tonyb | but that doesn't really scale | 22:55 |
| jroll | right | 22:55 |
| jroll | it just means I've no clue what could be going wrong then | 22:55 |
| tonyb | TheJulia: Yeah I'm using a coupel of his patches ;P | 22:55 |
| *** MattMan_1 is now known as MattMan | 22:56 | |
| jroll | unless default boot disk order is wonky, maybe | 22:56 |
| *** etingof has joined #openstack-ironic | 22:56 | |
| * tonyb wonders if it's a firmware bug | 22:56 | |
| tonyb | It's a good job the firmware is on github ;P | 22:56 |
| TheJulia | It could be, and then your just falling out to inspector based on configuration possibly | 22:56 |
| jroll | yeah, a firmware rabbit hole sounds fun | 22:56 |
| jroll | :P | 22:56 |
| TheJulia | tonyb: I guess it is using all tftpboot? | 22:57 |
| TheJulia | err | 22:57 |
| TheJulia | tftp? | 22:57 |
| vdrok | rloo: TheJulia so the jobs fail because of sigsegv it seems http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/screen-n-cpu.txt.gz#_Jan_18_05_48_49_185150, which means nova tries to access wrong chunk of memory right? I'm not sure we have something to do with that | 22:57 |
| tonyb | TheJulia: Yeah | 22:57 |
| TheJulia | tonyb: maybe restart the tftp server with verbose logging just to make sure your falling out config wise to inspection | 22:58 |
| TheJulia | it might be they always network boot if offered :\ | 22:58 |
| tonyb | TheJulia: Perhaps that would be mad but not impossible. | 22:59 |
| rloo | vdrok: I think TheJulia looked at another grenade failure and saw a diff failure than nova compute not running. i got stuck on something and haven't looked, but i was thiking about looking at logs for a few other grenade failures. | 23:00 |
| TheJulia | vdrok: I wonder if there are any binary modules that get built/linked/loaded that get upgraded......... | 23:01 |
| vdrok | we dont run multinode grenade on reqs changes? https://review.openstack.org/#/c/533875/ | 23:01 |
| patchbot | patch 533875 - ironic - Updated from global requirements | 23:01 |
| tonyb | TheJulia: I'm not sure I can do that but I can capture all dhcp/tftp packets | 23:02 |
| TheJulia | vdrok: http://logs.openstack.org/36/509336/31/check/ironic-grenade-dsvm-multinode-multitenant/6da9163/logs/screen-n-cond.txt.gz#_Jan_18_05_49_48_642840 | 23:04 |
| vdrok | mhm, and here is the job that was not killed by signal 11 http://logs.openstack.org/16/524616/7/check/ironic-grenade-dsvm-multinode-multitenant/ffca71e/logs/screen-n-cpu.txt.gz#_Jan_18_15_34_25_789836 | 23:05 |
| TheJulia | vdrok: I guess not... :\ | 23:05 |
| tonyb | Hmm I wonder if the power cycle and the setting of the boot device raced somehow | 23:06 |
| * tonyb goes off to test that theory | 23:06 | |
| *** lbragstad has quit IRC | 23:08 | |
| TheJulia | vdrok: except, n-cond still went off into a loop getting sig11'ed | 23:08 |
| TheJulia | err, sigv'ed | 23:08 |
| TheJulia | http://logs.openstack.org/16/524616/7/check/ironic-grenade-dsvm-multinode-multitenant/ffca71e/logs/screen-n-cond.txt.gz#_Jan_18_15_25_12_029694 | 23:08 |
| *** cjloader has quit IRC | 23:11 | |
| TheJulia | hmm... my internet connection just went super-laggy. I wonder if it would speed up if I promised it beer | 23:13 |
| vdrok | looking at logstash, first entry of "killed by signal 11" in our jobs is 17 jan, 16:00 | 23:13 |
| vdrok | sry, 17 jan 02:00 | 23:13 |
| TheJulia | UTC? | 23:14 |
| *** pmannidi has joined #openstack-ironic | 23:14 | |
| vdrok | yeah i think logstash gives utc. at least i see no explicit timezone | 23:15 |
| TheJulia | vdrok: that is erely close to when the other fix was landed :( | 23:16 |
| *** rpioso is now known as rpioso|afk | 23:23 | |
| tonyb | TheJulia, jroll: Thanks for you time it *looks* like a firmware issue, but I'm goign to grvel through the ironic logs to verify that | 23:27 |
| vdrok | TheJulia: nope it was local time, so around 10am utc | 23:28 |
| TheJulia | vdrok: your in pacific right now? | 23:28 |
| vdrok | yup | 23:29 |
| TheJulia | vdrok: so, I think the neutron gate fix landed around 8:30 PM your time last night | 23:29 |
| TheJulia | well, networking-generic-switch | 23:29 |
| TheJulia | tonyb: no problem, happy hunting! | 23:29 |
| vdrok | TheJulia: you mean n-g-s could be the reason? | 23:30 |
| TheJulia | well | 23:31 |
| TheJulia | no | 23:31 |
| TheJulia | https://review.openstack.org/#/c/534955/ merged at 9pm | 23:31 |
| patchbot | patch 534955 - networking-generic-switch (stable/pike) - use callbacks from neutron-lib (MERGED) | 23:31 |
| TheJulia | vdrok: do you have a jobid for the first instance of it occuring? | 23:31 |
| TheJulia | err, revision id | 23:33 |
| *** bfournie has quit IRC | 23:33 | |
| *** bfournie has joined #openstack-ironic | 23:33 | |
| vdrok | i cannot find the very first, as if using the n-cond entry, there are too many of them, logstash cannot show it. but if using n-cpu entry, the first one is http://logs.openstack.org/10/534710/1/check/ironic-grenade-dsvm-multinode-multitenant/079fa3e/logs/screen-n-cpu.txt | 23:35 |
| vdrok | TheJulia: ^^ | 23:35 |
| vdrok | the ones with code 11 in conductor appeared ~an hour earlier | 23:36 |
| TheJulia | so crazy idea, what if we just went ahead and upgraded nova | 23:38 |
| TheJulia | it is a fundamental change in the testing, but... seems we're in a quagmire at this point | 23:39 |
| vdrok | we can try, but we were seeing 11 code even without the restart | 23:44 |
| TheJulia | well, master -> master works | 23:45 |
| TheJulia | which means it is something likely that gets loaded... I'm guessing when the periodic task fires | 23:45 |
| TheJulia | so without containers or venvs... | 23:46 |
| *** penick has joined #openstack-ironic | 23:47 | |
| *** charcol has joined #openstack-ironic | 23:49 | |
| openstackgerrit | Vladyslav Drok proposed openstack/ironic master: Revert "grenade: For multi-node grenade, do not upgrade nova" https://review.openstack.org/535524 | 23:49 |
| vdrok | TheJulia: lets see ^^ | 23:49 |
| vdrok | ah, conflict :) | 23:50 |
| TheJulia | :) | 23:50 |
| *** penick_ has joined #openstack-ironic | 23:51 | |
| *** penick has quit IRC | 23:52 | |
| TheJulia | vdrok: rebasing? | 23:53 |
| vdrok | yeah need couple of mins, current machine is setup with all the different creds | 23:54 |
| TheJulia | k | 23:55 |
| openstackgerrit | Vladyslav Drok proposed openstack/ironic master: Revert "grenade: For multi-node grenade, do not upgrade nova" https://review.openstack.org/535524 | 23:57 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!