| mriedem | the funny thing is, the logstash query is like 99% hits in a non-ironic job | 00:00 |
|---|---|---|
| *** ijw has quit IRC | 00:00 | |
| soliosg | self.free_disk_mb = compute.free_disk_gb * 1024; TypeError: unsupported operand type(s) for *: 'NoneType' and 'int' | 00:00 |
| mriedem | there is 1 hit in an ironic job and that's http://logs.openstack.org/20/407220/2/check/gate-tempest-dsvm-ironic-ipa-wholedisk-agent_ipmitool-tinyipa-multinode-ubuntu-xenial-nv/0779f94/logs/screen-n-sch.txt.gz#_2017-01-04_19_05_16_398 | 00:01 |
| mriedem | which jlvillal poined out | 00:01 |
| soliosg | is happening in latest devstack | 00:01 |
| mriedem | otherwise it's all in the gate-tempest-dsvm-neutron-identity-v3-only-full-ubuntu-xenial-nv job | 00:01 |
| JayF | that seems really weird. | 00:01 |
| JayF | I wonder if there was some subtle timing/ordering change in devstack | 00:02 |
| JayF | that exposed this preexisting race | 00:02 |
| soliosg | I was trying to find out why my devstack started failing after all projects were updated from dec-14-2016 to jan-04-2017 | 00:02 |
| mriedem | ok in this non-ironic job, | 00:07 |
| mriedem | when the n-cpu service starts, we create the compute node record in the db here http://logs.openstack.org/65/416765/1/check/gate-tempest-dsvm-neutron-identity-v3-only-full-ubuntu-xenial-nv/faf1363/logs/screen-n-cpu.txt.gz#_2017-01-04_23_15_04_293 | 00:07 |
| mriedem | when we go to updated it (which would set free_disk_db): | 00:07 |
| mriedem | http://logs.openstack.org/65/416765/1/check/gate-tempest-dsvm-neutron-identity-v3-only-full-ubuntu-xenial-nv/faf1363/logs/screen-n-cpu.txt.gz#_2017-01-04_23_15_06_674 | 00:07 |
| mriedem | kablammo | 00:07 |
| mriedem | *update it | 00:07 |
| mriedem | so that's why the scheduler host manager is getting the compute node record from the db but not getting it's free_disk_gb value | 00:08 |
| JayF | is it a bug in itself that auth failure there can blow it up? | 00:08 |
| JayF | or is that one of those "only in devstack" kind of bugs | 00:08 |
| JayF | it seems like if that blows up, nova should handle it more sanely than the scheduler process bailing | 00:09 |
| mriedem | i wonder if the token is expiring | 00:09 |
| mriedem | well, this is the resource tracker in the n-cpu service | 00:09 |
| mriedem | the scheduler is just pulling garbage from the db | 00:09 |
| mriedem | so, definitely a bug that the scheduler doesn't filter this busted compute node out properly | 00:09 |
| *** jrist_ has joined #openstack-ironic | 00:09 | |
| JayF | well that's what I'm saying, is either scheduler needs to handle the garbage better and/or compute has to guarantee it won't be garbage | 00:09 |
| JayF | right? | 00:09 |
| *** jrist has quit IRC | 00:10 | |
| mriedem | oh you know what this is when the compute is trying to talk to the placement API | 00:10 |
| mriedem | _get_resource_provider | 00:10 |
| mriedem | that's all placement stuff | 00:10 |
| mriedem | i bet it was this that triggered it https://review.openstack.org/#/c/412537/ | 00:12 |
| patchbot_ | patch 412537 - openstack-dev/devstack - Add placement services to default devstack (MERGED) | 00:12 |
| mriedem | merged ~1/3 | 00:12 |
| JayF | ironic jobs have a patch in that disables that, iirc | 00:12 |
| mriedem | yup | 00:12 |
| mriedem | well until https://review.openstack.org/#/c/415523/ | 00:13 |
| patchbot_ | patch 415523 - ironic - Revert "Disable placement-api by default" | 00:13 |
| JayF | but it looks like if we fix the scheduler junk-handling, we kill both bugs with one stone | 00:13 |
| mriedem | which reverts that | 00:13 |
| mriedem | to be clear, what i'm looking at in http://logs.openstack.org/65/416765/1/check/gate-tempest-dsvm-neutron-identity-v3-only-full-ubuntu-xenial-nv/faf1363/logs/screen-n-cpu.txt.gz#_2017-01-04_23_15_06_674 is not an ironic job | 00:13 |
| mriedem | it's a keystone v3 job | 00:13 |
| JayF | yes; but isn't that the same failure that was in an ironic job earlier? | 00:14 |
| JayF | nope, it's not | 00:14 |
| JayF | I read one line and clicked the link from another | 00:14 |
| JayF | wheee | 00:14 |
| mriedem | well, ^ is what is causing the non-ironic host manager to blow up | 00:14 |
| mriedem | i don't know how updated_at is already set, | 00:14 |
| mriedem | for the ironic host manager failure, i can see how there could be a window of failure there | 00:15 |
| mriedem | so probably needs to be fixed too, but lower priority | 00:15 |
| JayF | Makes sense. I'm going to call it a day, but if there's something specifically actionable I can fix or help troubleshoot, feel free to drop me an email or leave me a pm on irc | 00:15 |
| mriedem | yeah my wife is texting me asking when i'm going to be home so i'm already in trouble | 00:18 |
| mriedem | remind me to harass jaypipes in the morning if i don't remember | 00:18 |
| JayF | heh :) have a good evening matt, thanks for digging and being loud about it so I learned something ;) | 00:19 |
| *** absubram has quit IRC | 00:20 | |
| *** vgadiraj has quit IRC | 00:25 | |
| mriedem | AHA https://review.openstack.org/#/c/416599/ | 00:30 |
| patchbot_ | patch 416599 - openstack-dev/devstack - Fix placement service for identity-v3 | 00:30 |
| *** amotoki has quit IRC | 00:34 | |
| *** palmertime has quit IRC | 00:37 | |
| *** hw_wutianwei has joined #openstack-ironic | 00:39 | |
| *** mriedem has quit IRC | 00:41 | |
| *** hoangcx has joined #openstack-ironic | 00:47 | |
| *** rpioso has quit IRC | 00:57 | |
| *** dyasny has joined #openstack-ironic | 00:58 | |
| *** rama_y has quit IRC | 00:59 | |
| *** phuongnh has joined #openstack-ironic | 01:05 | |
| *** awiddersheim has quit IRC | 01:09 | |
| *** tuanluong has joined #openstack-ironic | 01:12 | |
| *** tuanluong_ has joined #openstack-ironic | 01:12 | |
| *** sabeen has left #openstack-ironic | 01:15 | |
| *** ijw has joined #openstack-ironic | 01:17 | |
| *** ijw has quit IRC | 01:20 | |
| *** zhangjl has joined #openstack-ironic | 01:26 | |
| *** awiddersheim has joined #openstack-ironic | 01:31 | |
| *** lindycoder has joined #openstack-ironic | 01:40 | |
| *** lindycoder has quit IRC | 01:41 | |
| *** yufei has joined #openstack-ironic | 01:43 | |
| *** lindycoder has joined #openstack-ironic | 01:45 | |
| *** baoli has joined #openstack-ironic | 01:55 | |
| *** baoli has quit IRC | 02:06 | |
| *** ijw has joined #openstack-ironic | 02:10 | |
| *** ijw has quit IRC | 02:11 | |
| *** MattMan_1 has quit IRC | 02:31 | |
| *** MattMan_1 has joined #openstack-ironic | 02:31 | |
| *** trananhkma has quit IRC | 02:34 | |
| openstackgerrit | Naohiro Tamura proposed openstack/ironic-specs: Add 'abort' support for Soft Power Off and Inject NMI https://review.openstack.org/250783 | 02:38 |
| *** lindycoder has quit IRC | 02:39 | |
| *** baoli has joined #openstack-ironic | 02:42 | |
| *** tiendc has joined #openstack-ironic | 02:48 | |
| openstackgerrit | Naohiro Tamura proposed openstack/ironic-specs: Add 'abort' support for Soft Power Off and Inject NMI https://review.openstack.org/250783 | 02:52 |
| *** baoli has quit IRC | 02:55 | |
| *** baoli has joined #openstack-ironic | 02:58 | |
| *** harlowja has quit IRC | 03:05 | |
| *** tonyb has quit IRC | 03:05 | |
| *** yolanda has quit IRC | 03:05 | |
| *** joemonk has quit IRC | 03:05 | |
| *** yhvh has quit IRC | 03:05 | |
| *** tonyb has joined #openstack-ironic | 03:05 | |
| *** joemonk- has joined #openstack-ironic | 03:05 | |
| *** yolanda has joined #openstack-ironic | 03:05 | |
| *** yhvh has joined #openstack-ironic | 03:06 | |
| *** baoli has quit IRC | 03:18 | |
| *** baoli has joined #openstack-ironic | 03:19 | |
| *** vgadiraj has joined #openstack-ironic | 03:23 | |
| *** amotoki has joined #openstack-ironic | 03:30 | |
| *** baoli has quit IRC | 03:36 | |
| *** links has joined #openstack-ironic | 03:40 | |
| *** vikrant has joined #openstack-ironic | 03:42 | |
| *** nmathew has joined #openstack-ironic | 03:50 | |
| *** Nisha_Agarwal has joined #openstack-ironic | 03:52 | |
| *** ijw has joined #openstack-ironic | 03:53 | |
| *** vgadiraj has quit IRC | 03:56 | |
| *** SerenaFeng has joined #openstack-ironic | 03:57 | |
| *** aparnav has joined #openstack-ironic | 03:59 | |
| *** yufei has quit IRC | 03:59 | |
| *** SerenaFeng has quit IRC | 04:05 | |
| *** SerenaFeng has joined #openstack-ironic | 04:12 | |
| *** SerenaFeng has quit IRC | 04:15 | |
| *** ijw has quit IRC | 04:45 | |
| *** ijw has joined #openstack-ironic | 04:49 | |
| *** zackf is now known as jrcloud | 04:56 | |
| *** jrcloud is now known as zackf | 04:56 | |
| *** sasha2 has quit IRC | 05:05 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ironic-inspector: Updated from global requirements https://review.openstack.org/414284 | 05:06 |
| *** nmathew has quit IRC | 05:11 | |
| *** tiendc has quit IRC | 05:30 | |
| *** SerenaFeng has joined #openstack-ironic | 05:32 | |
| *** tiendc has joined #openstack-ironic | 05:36 | |
| *** SerenaFeng has quit IRC | 05:36 | |
| *** SerenaFeng has joined #openstack-ironic | 05:37 | |
| *** yuikotakadamori has joined #openstack-ironic | 05:54 | |
| *** appu103 has joined #openstack-ironic | 05:55 | |
| *** aparnav has quit IRC | 05:58 | |
| *** Nisha_Agarwal has quit IRC | 06:07 | |
| *** yufei has joined #openstack-ironic | 06:08 | |
| *** aparnav has joined #openstack-ironic | 06:15 | |
| *** appu103 has quit IRC | 06:17 | |
| openstackgerrit | paresh sao proposed openstack/proliantutils: Adds 'UefiTarget' as a boot option https://review.openstack.org/407487 | 06:33 |
| *** hoangcx_ has joined #openstack-ironic | 06:54 | |
| *** hoangcx has quit IRC | 06:56 | |
| *** rbudden has quit IRC | 06:58 | |
| *** moshele has joined #openstack-ironic | 07:03 | |
| *** SerenaFeng has quit IRC | 07:03 | |
| *** pcaruana has joined #openstack-ironic | 07:05 | |
| *** tesseract has joined #openstack-ironic | 07:09 | |
| *** ChubYann has quit IRC | 07:16 | |
| *** rcernin has joined #openstack-ironic | 07:17 | |
| *** yuikotak_ has joined #openstack-ironic | 07:24 | |
| *** yuikotakadamori has quit IRC | 07:24 | |
| *** rbartal has joined #openstack-ironic | 07:25 | |
| *** yuikotak_ is now known as yuikotakadamori | 07:32 | |
| *** Nisha_Agarwal has joined #openstack-ironic | 07:36 | |
| *** e0ne has joined #openstack-ironic | 07:37 | |
| *** openstackgerrit has quit IRC | 07:50 | |
| *** hoangcx has joined #openstack-ironic | 07:53 | |
| *** zhugaoxiao has quit IRC | 07:54 | |
| *** zhugaoxiao has joined #openstack-ironic | 07:54 | |
| *** hoangcx_ has quit IRC | 07:56 | |
| *** SerenaFeng has joined #openstack-ironic | 08:10 | |
| *** ijw has quit IRC | 08:15 | |
| *** SerenaFeng has quit IRC | 08:19 | |
| *** SerenaFeng has joined #openstack-ironic | 08:19 | |
| *** athomas has joined #openstack-ironic | 08:27 | |
| *** openstackgerrit has joined #openstack-ironic | 08:33 | |
| openstackgerrit | yolanda.robla proposed openstack/ironic-specs: Ironic deployment steps https://review.openstack.org/412523 | 08:33 |
| yuikotakadamori | hi, ironic | 08:34 |
| *** zackf has quit IRC | 08:34 | |
| *** pmannidi has quit IRC | 08:57 | |
| *** jtomasek has quit IRC | 08:58 | |
| *** zzzeek has quit IRC | 09:00 | |
| *** ohamada has joined #openstack-ironic | 09:03 | |
| *** zzzeek has joined #openstack-ironic | 09:03 | |
| *** Linkmark has joined #openstack-ironic | 09:03 | |
| *** jtomasek has joined #openstack-ironic | 09:05 | |
| *** pmannidi has joined #openstack-ironic | 09:09 | |
| *** pmannidi has quit IRC | 09:13 | |
| yuikotakadamori | there seems no "graceful shutdown" in ironic, is it? | 09:14 |
| *** milan has joined #openstack-ironic | 09:15 | |
| milan | morning Ironic! :) | 09:15 |
| yuikotakadamori | milan, morning :) | 09:16 |
| milan | morning yuikotakadamori! :) | 09:17 |
| *** pmannidi has joined #openstack-ironic | 09:18 | |
| *** pmannidi has quit IRC | 09:22 | |
| *** lucas-afk is now known as lucasagomes | 09:24 | |
| lucasagomes | morning all | 09:24 |
| milan | morning lucasagomes! :) | 09:25 |
| openstackgerrit | Merged openstack/proliantutils: Adds 'UefiTarget' as a boot option https://review.openstack.org/407487 | 09:31 |
| *** pmannidi has joined #openstack-ironic | 09:33 | |
| *** jtomasek has quit IRC | 09:33 | |
| *** openstackgerrit has quit IRC | 09:33 | |
| *** jtomasek has joined #openstack-ironic | 09:36 | |
| *** pmannidi has quit IRC | 09:37 | |
| *** openstackgerrit has joined #openstack-ironic | 09:43 | |
| openstackgerrit | Merged openstack/bifrost: Fix condition for include ssh_public_key_path https://review.openstack.org/416265 | 09:43 |
| openstackgerrit | Merged openstack/bifrost: Fix uuid and name parameters in inspect task https://review.openstack.org/415741 | 09:44 |
| *** SerenaFeng has quit IRC | 09:44 | |
| *** derekh has joined #openstack-ironic | 09:47 | |
| lucasagomes | gate seems pretty busted !? | 09:47 |
| openstackgerrit | Merged openstack/bifrost: Add missing skip_install conditions https://review.openstack.org/415529 | 09:48 |
| openstackgerrit | Merged openstack/bifrost: Fix stdout not defined https://review.openstack.org/415696 | 09:48 |
| openstackgerrit | Oksana Voshchana proposed openstack/ironic: WIP: TEST GRENADE DEV https://review.openstack.org/397126 | 09:48 |
| *** amotoki has quit IRC | 09:55 | |
| *** hoangcx has quit IRC | 09:57 | |
| vdrok | good morning yuikotakadamori milan lucasagomes and all! | 10:00 |
| openstackgerrit | Galyna Zholtkevych proposed openstack/ironic: ETAGs maintenance for all api node's requests https://review.openstack.org/392213 | 10:08 |
| *** galyna has left #openstack-ironic | 10:09 | |
| dtantsur | morning Ironic | 10:09 |
| dtantsur | morning yuikotakadamori, what do you mean by "graceful shutdown" here? there is in-progress soft power off work | 10:10 |
| dtantsur | lucasagomes, inspector CI just failed with tempest.lib.exceptions.InvalidScope: Invalid Scope None for KeystoneV2AuthProvider | 10:10 |
| vdrok | morning dtantsur | 10:10 |
| vdrok | lucasagomes: dtantsur https://github.com/openstack/tempest/commit/5ac57dad9abbff792105eb675577cde890dc769c | 10:11 |
| lucasagomes | dtantsur, same for ironic at least one patch I've seem (I rechecked and waiting to the return) | 10:11 |
| vdrok | might be that one | 10:11 |
| * lucasagomes looks | 10:11 | |
| *** phuongnh has quit IRC | 10:11 | |
| dtantsur | ugh | 10:11 |
| dtantsur | morning vdrok, thanks for looking into it | 10:11 |
| *** phuongnh has joined #openstack-ironic | 10:11 | |
| lucasagomes | vdrok, hmm yeah could be | 10:11 |
| openstackgerrit | Merged openstack/bifrost: Default keystone to uuid tokens https://review.openstack.org/407231 | 10:12 |
| lucasagomes | vdrok, will you propose a revert ? | 10:12 |
| vdrok | lucasagomes: dunno, will make a patch fixing our tempest tests to begin with | 10:12 |
| vdrok | if it can be fixed in our tree | 10:13 |
| *** openstackstatus has quit IRC | 10:13 | |
| *** openstack has joined #openstack-ironic | 10:15 | |
| *** phuongnh has quit IRC | 10:16 | |
| openstackgerrit | Vladyslav Drok proposed openstack/ironic: Remove service argument from tempest plugin client manager https://review.openstack.org/416926 | 10:23 |
| vdrok | should be it ^ | 10:23 |
| milan | morning dtantsur! :) | 10:24 |
| *** tiendc has quit IRC | 10:26 | |
| *** zhs_ has quit IRC | 10:30 | |
| *** zhs_ has joined #openstack-ironic | 10:30 | |
| lucasagomes | derekh, http://docs.openstack.org/developer/ironic/dev/dev-quickstart.html#deploying-ironic-with-devstack | 10:49 |
| derekh | lucasagomes: thanks | 10:53 |
| *** zhangjl has left #openstack-ironic | 10:55 | |
| *** ashestakov has joined #openstack-ironic | 10:57 | |
| milan | moshele, hi, I'm reading the firewall.py stuff, isn't it possible to use the ibhost to figure out the peers that are EthOIB instead of the regexp? | 10:57 |
| milan | also, could you please paste a sample content of the /sys/class/network... file so I can check the regexp? | 10:58 |
| milan | moshele, I mean https://review.openstack.org/#/c/264257/45/ironic_inspector/firewall.py | 11:01 |
| patchbot_ | patch 264257 - ironic-inspector - Adding InfiniBand Support | 11:01 |
| *** galyna has joined #openstack-ironic | 11:01 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic-inspector: Remove unused "service" argument from tempest client manager https://review.openstack.org/416935 | 11:09 |
| dtantsur | vdrok, milan, ^^^ | 11:09 |
| * milan looks | 11:10 | |
| *** yufei has quit IRC | 11:10 | |
| milan | dtantsur, a link to failing job handy? thx | 11:11 |
| * milan didnt't pay attention to what was failing | 11:13 | |
| dtantsur | milan, e.g. http://logs.openstack.org/26/416926/1/check/gate-tempest-dsvm-ironic-inspector-ubuntu-xenial-nv/5cb8c35/ | 11:14 |
| milan | thx | 11:14 |
| dtantsur | vdrok, your patch failed grenade :( http://logs.openstack.org/26/416926/1/check/gate-grenade-dsvm-ironic-ubuntu-xenial/b67f1cd/ dunno if it's real failure | 11:14 |
| vdrok | dtantsur: yeah, i think grenade is a separate issue | 11:15 |
| vdrok | looking at our whiteboard | 11:15 |
| *** amotoki has joined #openstack-ironic | 11:16 | |
| *** ijw has joined #openstack-ironic | 11:20 | |
| *** sambetts|afk is now known as sambetts | 11:20 | |
| sambetts | Morning all | 11:20 |
| openstackgerrit | Vladyslav Drok proposed openstack/ironic: DNM: testing 416705 https://review.openstack.org/416707 | 11:21 |
| milan | morning sambetts | 11:21 |
| milan | :) | 11:21 |
| sambetts | hey milan | 11:21 |
| vdrok | morning sambetts | 11:21 |
| *** ijw has quit IRC | 11:26 | |
| milan | hmmm dtantsur, vdrok so imo the https://review.openstack.org/416935 looks ok the grenade issue not sure :-/ | 11:29 |
| patchbot_ | patch 416935 - ironic-inspector - Remove unused "service" argument from tempest clie... | 11:29 |
| vdrok | dtantsur: milan https://review.openstack.org/416707 | 11:29 |
| patchbot_ | patch 416707 - ironic - DNM: testing 416705 | 11:29 |
| vdrok | that depends on the grenade fix | 11:29 |
| vdrok | I'll make it depend on mine too, to be completely sure | 11:30 |
| milan | cool | 11:30 |
| * milan -> lunch | 11:30 | |
| *** milan is now known as lunch|milan | 11:31 | |
| openstackgerrit | Vladyslav Drok proposed openstack/ironic: DNM: testing 416705 and 416926 https://review.openstack.org/416707 | 11:31 |
| *** vikrant has quit IRC | 11:38 | |
| lucasagomes | the grenade job seems pretty flaky lately :-/ | 11:40 |
| vdrok | lucasagomes: well, that's devstack^2 :) | 11:41 |
| lucasagomes | :-( | 11:41 |
| vdrok | lucasagomes: ah, I did not get what you mean, rechecks won't help | 11:44 |
| vdrok | lucasagomes: that's separate issue https://etherpad.openstack.org/p/IronicWhiteBoard | 11:44 |
| lucasagomes | vdrok, oh, my bad. I thought it was related | 11:45 |
| vdrok | np :) | 11:45 |
| *** pmannidi has joined #openstack-ironic | 11:46 | |
| moshele | milan: I will check | 11:50 |
| openstackgerrit | Derek Higgins proposed openstack/ironic-python-agent: Advertise the correct address when using IPv6 https://review.openstack.org/411817 | 11:52 |
| openstackgerrit | Derek Higgins proposed openstack/ironic-python-agent: iscsi: Listen for connections on both IPv4 and IPv6 ports https://review.openstack.org/411819 | 11:52 |
| openstackgerrit | Derek Higgins proposed openstack/ironic-python-agent: Agent: Listen for connections on both IPv4 and IPv6 ports https://review.openstack.org/411818 | 11:52 |
| *** pmannidi has quit IRC | 11:54 | |
| *** derekh has quit IRC | 11:55 | |
| *** konetzed has quit IRC | 11:55 | |
| *** tdurakov has quit IRC | 11:55 | |
| *** skazi has quit IRC | 11:55 | |
| *** sambetts has quit IRC | 11:55 | |
| *** mpotdar has quit IRC | 11:55 | |
| *** mtreinish has quit IRC | 11:55 | |
| *** chihhsin has quit IRC | 11:55 | |
| *** afazekas has quit IRC | 11:55 | |
| *** bcornec has quit IRC | 11:55 | |
| *** dansmith has quit IRC | 11:55 | |
| *** devananda has quit IRC | 11:55 | |
| *** afazekas has joined #openstack-ironic | 11:55 | |
| *** konetzed has joined #openstack-ironic | 11:55 | |
| *** sambetts_ has joined #openstack-ironic | 11:55 | |
| *** chihhsin has joined #openstack-ironic | 11:56 | |
| *** devananda has joined #openstack-ironic | 11:56 | |
| *** dansmith has joined #openstack-ironic | 11:56 | |
| *** vikrant has joined #openstack-ironic | 11:56 | |
| *** derekh has joined #openstack-ironic | 11:56 | |
| *** mtreinish has joined #openstack-ironic | 11:56 | |
| *** bcornec has joined #openstack-ironic | 11:56 | |
| *** dansmith is now known as Guest41100 | 11:56 | |
| *** sambetts_ is now known as sambetts | 11:56 | |
| *** tdurakov has joined #openstack-ironic | 11:56 | |
| *** skazi has joined #openstack-ironic | 11:56 | |
| *** serverascode has quit IRC | 11:59 | |
| *** mpotdar has joined #openstack-ironic | 11:59 | |
| *** serverascode has joined #openstack-ironic | 12:00 | |
| *** yufei has joined #openstack-ironic | 12:03 | |
| *** tiendc has joined #openstack-ironic | 12:03 | |
| *** jkilpatr has joined #openstack-ironic | 12:05 | |
| *** lucasagomes is now known as lucas-hungry | 12:05 | |
| openstackgerrit | Eric Xie proposed openstack/ironic: Fix the comma's wrong locations. https://review.openstack.org/416957 | 12:13 |
| *** links has quit IRC | 12:13 | |
| *** tuanluong_ has quit IRC | 12:14 | |
| *** tuanluong has quit IRC | 12:14 | |
| *** g3ek has quit IRC | 12:14 | |
| galyna | Good morning, ironicers :) | 12:16 |
| galyna | There is a question in ironic CLI on how to better implement storing resource (with ETAG) representation in ironic CLI | 12:16 |
| galyna | As it is known user has to have at least some reprsentation at client side to send conditional request. | 12:16 |
| galyna | There are some proposals, I will specify it in the spec: | 12:16 |
| galyna | - During update send get and afterwards update with current representation (I do not like it at all, it descreases performance and changes the general behaviour) | 12:16 |
| galyna | - Force user to do ``ironic node-show <node>`` and get etag in the obtained resource representation | 12:16 |
| galyna | - If a user immediately wants to send conditional request, he may be advised to do sth like node=$(ironic node-show <node>) and then he can send sth like | 12:16 |
| galyna | ``ironic node-update $node`` | 12:16 |
| galyna | and resource will call manager and update itself (doing it through the resource is the similar way how it is done in novaclient for some requests). | 12:16 |
| galyna | But in these two last cases user is forced again to do additional actions to perform a simple command, so I do not like it either | 12:16 |
| galyna | - Cache resource representation at client side using requests-cache, for example | 12:16 |
| galyna | - Cache resource representation at ironic api middleware (or contribute caching to pecan to make it available for other projects) | 12:16 |
| *** ijw has joined #openstack-ironic | 12:21 | |
| *** hwoarang has quit IRC | 12:22 | |
| yolanda | hi, any ironic core around? i'm having some trouble with reading config drive info from a whole disk image | 12:22 |
| *** hwoarang has joined #openstack-ironic | 12:22 | |
| yolanda | i created a partition labeled config-2, but ironic is ignoring it, and creating a new one | 12:22 |
| sambetts | Hi yolanda | 12:22 |
| yolanda | if i look at the code, it seems to do a blkid -t label=xxx but in the image file | 12:23 |
| yolanda | so it's not finding it | 12:23 |
| *** marlinc has quit IRC | 12:24 | |
| *** g3ek has joined #openstack-ironic | 12:24 | |
| yolanda | a blkid -t label=config-2 on overcloud-full.raw image is not finding that, but if you inspect the image with guestfish, list the partitions, and blkid on the right one, it exists | 12:24 |
| mat128 | good morning everyone | 12:25 |
| *** links has joined #openstack-ironic | 12:25 | |
| mat128 | yolanda: maybe it's missing a partprobe / something to reload it? | 12:26 |
| yolanda | mat128, looking at the code, what it seems to be doing, is convert the qcow image to raw, and then doing a blkid on that file | 12:26 |
| *** ijw has quit IRC | 12:27 | |
| mat128 | oh, nvm then :( | 12:27 |
| *** mat128 is now known as mat128|afk | 12:28 | |
| sambetts | yolanda: I don't think you should be storing a config-drive partition in your wholedisk image right? because it should be created using the config-drive created by nova? | 12:28 |
| yolanda | sambetts, but i was told this is supported. I want to create a config-drive partition just empty, because i want to locate it at a different part of the disk | 12:28 |
| *** hwoarang_ has joined #openstack-ironic | 12:28 | |
| sambetts | ah right I see, I thought you meant a populated config-drive in the image | 12:28 |
| yolanda | so i was told to create a partition labeled config-2 on the whole disk image, and ironic will use that. And looking at the code, it seems to be supported, but is failing | 12:29 |
| yolanda | no, just the partition for it | 12:29 |
| yolanda | i'm adding more debug to follow the steps, but seems a problem in listing the partitions on the image | 12:29 |
| *** hwoarang has quit IRC | 12:31 | |
| *** tiendc has quit IRC | 12:34 | |
| yolanda | so, it does a dd from image in /var/lib/ironic/images, to /dev/disk/by-path/xxx... then a partprobe on it, and a blkid -o device /path/to/image. But if you do that on an image file, it's not returning the content of the partitions: blkid -o device /dev/disk/by-path/ip-192.168.24.7:3260-iscsi-iqn.2008-10.org.openstack:e5866d44-6a92-4146-81d9-46fdd46ebe5d-lun-1 -> /dev/disk/by-path/ip-192.168.24.7:3260-iscsi-iqn.2008-10.org.openstack:e5866d44-6a92- | 12:38 |
| yolanda | 4146-81d9-46fdd46ebe5d-lun-1 | 12:38 |
| *** aparnav has quit IRC | 12:38 | |
| *** mat128|afk is now known as mat128 | 12:43 | |
| *** mgould|afk is now known as mgould | 12:46 | |
| mgould | morning Ironic! | 12:46 |
| *** hw_wutianwei has quit IRC | 12:48 | |
| *** vikrant has quit IRC | 12:49 | |
| *** bfournie has quit IRC | 12:50 | |
| *** vinsh_ has quit IRC | 12:51 | |
| *** vikrant has joined #openstack-ironic | 12:54 | |
| *** links has quit IRC | 12:55 | |
| *** vikrant has quit IRC | 12:55 | |
| *** dprince has joined #openstack-ironic | 13:02 | |
| rcernin | sambetts: hi re yesterday com with dtantsur SSL_VERIFY_FAILED, i noted the same with cinder now. Isn't there any change in python env in general? | 13:02 |
| rcernin | sambetts: since I saw it a workaround with https://paste.fedoraproject.org/520239/62135714/ | 13:02 |
| rcernin | for cinder ^^ | 13:02 |
| openstackgerrit | Galyna Zholtkevych proposed openstack/python-ironicclient: [WIP] ETAGs for node in ironicclient https://review.openstack.org/400335 | 13:07 |
| *** links has joined #openstack-ironic | 13:08 | |
| sambetts | rcernin: the ucssdk that is used for the UCSM driver actually has a flag that can be passed in to disable cert verification, its just that there is no ironic config option for triggering passing in that flag | 13:08 |
| *** trown|outtypewww is now known as trown | 13:11 | |
| sambetts | rcernin: although I'm unsure that it was an issue with the driver rather the environment because I don't do anything special in my CI for the UCSM driver and it works out of the box in a clean VM | 13:12 |
| yolanda | i created a bug for it https://bugs.launchpad.net/ironic/+bug/1654269 | 13:14 |
| openstack | Launchpad bug 1654269 in Ironic "Pre-existing config-drive partition not detected on whole disk images" [Undecided,New] | 13:14 |
| rcernin | sambetts: I see two options, either disable verification as you said or setup trust for the certificate that is called from the REST API. | 13:18 |
| rcernin | sambetts: but more than that I tried to understand the bottom problem, like if there was a python change or something that would actually enforce this SSL checks as in Mitaka it didn't have problems. | 13:19 |
| rcernin | so eventually we are maybe moving out of ironic project. | 13:20 |
| *** bfournie has joined #openstack-ironic | 13:22 | |
| *** jjohnson2_ has joined #openstack-ironic | 13:23 | |
| *** ijw has joined #openstack-ironic | 13:23 | |
| jjohnson2_ | Hello, so I come back to do something and I guess Jenkins is no longer content with just the pep8 gate... | 13:23 |
| sambetts | rcernin: hmm yeah thats is it good question, I'm unsure about that :/ | 13:26 |
| *** lunch|milan is now known as milan | 13:26 | |
| *** lucas-hungry is now known as lucasagomes | 13:27 | |
| *** ijw has quit IRC | 13:28 | |
| lucasagomes | jjohnson2, happy new year! You mean, for pyghmi ? | 13:28 |
| jjohnson2_ | lucasagomes: happy new year, yeah... | 13:29 |
| jjohnson2_ | https://review.openstack.org/#/c/416718/ | 13:29 |
| patchbot_ | patch 416718 - pyghmi - Refactor Lenovo OEM support | 13:29 |
| *** jheroux has joined #openstack-ironic | 13:33 | |
| rcernin | sambetts: can you check https://security.openstack.org/guidelines/dg_validate-certificates.html | 13:33 |
| rcernin | sambetts: It is important to note that modules such as httplib within the Python standard library did not verify certificate chains until it was fixed in 2.7.9 release" | 13:33 |
| *** d0ugal has quit IRC | 13:34 | |
| *** d0ugal has joined #openstack-ironic | 13:35 | |
| rcernin | sambetts: okay will try to find out, and let you know if I had any luck. | 13:38 |
| sambetts | rcernin: thanks :) | 13:38 |
| rcernin | sambetts: when you mentioned the verification in ucssdk, do you know how to disable it? | 13:41 |
| sambetts | there is a flag you must pass into the .Login function noSsl=True I think, (its been a while since I worked with the SDK) | 13:44 |
| *** aparnav has joined #openstack-ironic | 13:45 | |
| openstackgerrit | Jarrod Johnson proposed openstack/pyghmi: Add a test for part of SDR https://review.openstack.org/416977 | 13:48 |
| *** baoli has joined #openstack-ironic | 13:51 | |
| openstackgerrit | Jarrod Johnson proposed openstack/pyghmi: Add a test for part of SDR https://review.openstack.org/416977 | 13:51 |
| *** Syed__ has quit IRC | 13:55 | |
| milan | moshele, thx for the comment, found it now (still reviewing) | 13:55 |
| openstackgerrit | Jarrod Johnson proposed openstack/pyghmi: Add a test for part of SDR https://review.openstack.org/416977 | 13:55 |
| *** fritzstauffacher has joined #openstack-ironic | 13:55 | |
| *** lindycoder has joined #openstack-ironic | 13:56 | |
| jroll | morning | 13:58 |
| jroll | dat gate | 13:58 |
| lucasagomes | jroll, morning, yeah :-( | 13:58 |
| *** aparnav has quit IRC | 13:58 | |
| sambetts | hey jroll | 13:59 |
| *** sasha2 has joined #openstack-ironic | 14:01 | |
| *** links has quit IRC | 14:02 | |
| openstackgerrit | Bob Fournier proposed openstack/ironic-inspector: Add plugin to process basic management LLDP TLVs https://review.openstack.org/406496 | 14:05 |
| yolanda | hi lucasagomes , jroll, can you take a look at https://bugs.launchpad.net/ironic/+bug/1654269 ? i'm trying to figure if there is a good alternative to get partition labels for the whole disk image | 14:06 |
| openstack | Launchpad bug 1654269 in Ironic "Pre-existing config-drive partition not detected on whole disk images" [Undecided,New] | 14:06 |
| *** aparnav has joined #openstack-ironic | 14:07 | |
| openstackgerrit | Jarrod Johnson proposed openstack/pyghmi: Add a test for part of SDR https://review.openstack.org/416977 | 14:08 |
| lucasagomes | yolanda, will do | 14:08 |
| jroll | yolanda: the conversations we've add are pretty much the extent of my knowledge on partitioning things, I'm fine with another method though :) | 14:08 |
| yolanda | jroll, i cannot see an easy way without mounting, or with usage of libguestfs | 14:09 |
| jroll | hrm | 14:09 |
| yolanda | i can extract it properly with virt-list-filesystems for example, but this adds a dependency to ironic | 14:09 |
| jroll | can't we check after we write the image? | 14:09 |
| yolanda | it fails on a check to create the config drive partition. If it cannot find it, creates a new one | 14:10 |
| yolanda | so i end with two config drive partitions, the one i created, and the one from ironic | 14:10 |
| jroll | right, but doesn't it check the actual disk after we write the image? | 14:11 |
| jroll | which should show in blkid | 14:11 |
| yolanda | i tested blkid output and does not return the content in the partitions | 14:12 |
| jroll | you tested blkid on the physical disk? not the disk image? | 14:12 |
| yolanda | i added debug output to the ironic commands | 14:13 |
| yolanda | so the output i paste there, is after the dd | 14:13 |
| jroll | hrm | 14:13 |
| * jroll wonders why that wouldn't work | 14:13 | |
| lucasagomes | yolanda, is ./disk the image in the filesystem ? | 14:14 |
| *** pc_m has quit IRC | 14:15 | |
| yolanda | lucasagomes, is the copied image , i added more debug logs to ironic, an see | 14:15 |
| yolanda | and see | 14:15 |
| yolanda | blkid -o device /dev/disk/by-path/ip-192.168.24.7:3260-iscsi-iqn.2008-10.org.openstack: | 14:15 |
| yolanda | 403ce764-2bf1-4e3b-86b7-9fdeafd12f8f-lun-1 | 14:15 |
| yolanda | Command stdout is: "/dev/disk/by-path/ip-192.168.24.7:3260-iscsi-iqn.2008-10.org.openstack:403ce764-2bf1-4e3b-86b7-9fdeafd12f | 14:15 |
| yolanda | 8f-lun-1 | 14:15 |
| patchbot_ | Error: No closing quotation | 14:15 |
| lucasagomes | yolanda, right and is it in raw format ? Or qcow2 ? | 14:15 |
| *** pc_m has joined #openstack-ironic | 14:15 | |
| yolanda | seems it converts to raw | 14:15 |
| jroll | mmmmmmmm iscsi. | 14:15 |
| lucasagomes | cause if it's already in raw format, I suspect you would need to put in it a loop device prior to using blkid | 14:15 |
| lucasagomes | losetup /dev/loop0 ./disk | 14:15 |
| lucasagomes | blkid -o device ... /dev/loop0 | 14:16 |
| yolanda | it's code from ironic... | 14:16 |
| yolanda | shall i update the ironic code and retry? | 14:16 |
| *** baoli has quit IRC | 14:17 | |
| *** baoli has joined #openstack-ironic | 14:17 | |
| *** hoangcx has joined #openstack-ironic | 14:17 | |
| lucasagomes | apparently it will use blkid after the image is being copied, directly on the device so it should be grand | 14:19 |
| lucasagomes | yolanda, you're not using LVM are you ? | 14:19 |
| yolanda | lucasagomes, not for the config-2 partition | 14:20 |
| yolanda | is just a primary one | 14:20 |
| yolanda | see that libguestfs reads it properly | 14:20 |
| lucasagomes | yeah, lemme try to test something locally | 14:24 |
| *** ijw has joined #openstack-ironic | 14:24 | |
| yolanda | i'm going to try updating ironic code and mounting, and get output from blkid then | 14:24 |
| yolanda | not sure what's going on | 14:25 |
| lucasagomes | yolanda, yeah you probably don't need to mount it, maybe update the code to: 1. use lsblk | 14:27 |
| lucasagomes | also check if /dev/disk/by-label/config-2 exists | 14:27 |
| yolanda | ok let me add those checks as well, to capture output | 14:28 |
| lucasagomes | yolanda, http://paste.openstack.org/show/593999/ | 14:28 |
| lucasagomes | I just created a image and partitioned it for testing | 14:29 |
| lucasagomes | attached to a loop device | 14:29 |
| lucasagomes | p5 is a logical parititon and p1 is a primary | 14:29 |
| yolanda | so you mean, in that case i don't have to even use the losetup? | 14:29 |
| *** ijw has quit IRC | 14:30 | |
| lucasagomes | yolanda, no because ironic will run the command against the device where the image was copied onto | 14:30 |
| lucasagomes | so you should be grand | 14:30 |
| yolanda | ok let me test it | 14:30 |
| lucasagomes | that said, running against the file image (e.g "./disk") doesn't work | 14:31 |
| yolanda | then the blkid command is wrong? | 14:31 |
| lucasagomes | apparently | 14:34 |
| lucasagomes | root@ubuntu:/opt/stack/ironic-python-agent# blkid -o device -tLABEL=test-label2 /dev/loop2 | 14:34 |
| lucasagomes | root@ubuntu:/opt/stack/ironic-python-agent# | 14:34 |
| * lucasagomes test on another OS | 14:35 | |
| *** jcoufal has joined #openstack-ironic | 14:35 | |
| sambetts | lucasagomes: glean checks for config-2 like this https://github.com/openstack-infra/glean/blob/master/glean/init/glean.sh#L49 | 14:36 |
| sambetts | lucasagomes: if it helps | 14:36 |
| yolanda | it's how ironic does it as well, and is not working for me | 14:37 |
| lucasagomes | sambetts, thanks | 14:38 |
| lucasagomes | root@ubuntu:/opt/stack/ironic-python-agent# blkid -L test-label /dev/loop2 | 14:38 |
| lucasagomes | /dev/loop2p5 | 14:38 |
| *** Goneri has joined #openstack-ironic | 14:38 | |
| lucasagomes | that actually works locally for me | 14:39 |
| openstackgerrit | Jarrod Johnson proposed openstack/pyghmi: Add a test for part of SDR https://review.openstack.org/416977 | 14:40 |
| yolanda | lucasagomes, maybe i'm creating the label for partition incorrectly? how did you label it? | 14:41 |
| lucasagomes | yolanda, mkfs.ext3 -L <label> <partition> | 14:42 |
| yolanda | same... | 14:42 |
| lucasagomes | yeah try lsblk or the devfs see if it works | 14:43 |
| sambetts | is it possible that the label is being lost somehow in the writing process? | 14:44 |
| yolanda | mm, i'm getting an "unexpected error" when executing the command | 14:45 |
| yolanda | mm, unauthorized command | 14:45 |
| yolanda | output, err = utils.execute('lsblk', '-P', '-o', 'name,label', device, | 14:46 |
| yolanda | check_exit_code=[0, 2], use_standard_locale=True, | 14:46 |
| yolanda | run_as_root=True) | 14:46 |
| yolanda | i'm running as root, but even with that it gets that unauthorized error | 14:46 |
| vdrok | yolanda: maybe it's missing from rootwrap? | 14:47 |
| vdrok | and good morning yolanda and jroll | 14:47 |
| yolanda | looks like... | 14:47 |
| lucasagomes | probably need to add lsblk to the filters | 14:48 |
| lucasagomes | rootwrap filters | 14:48 |
| jlvillal | jroll, FYI: devstack stable/newton appears broken. Well last night it seemed like it. https://review.openstack.org/#/c/416763/ | 14:49 |
| patchbot_ | patch 416763 - openstack-dev/devstack (stable/newton) - WIP/DNM: Check if stable/newton builds | 14:49 |
| milan | moshele, what is the relationship between a port['mac'] and a /sys/network/<eoib_inspector_interface>/eth/neigh:<port[client-id]:last 8bytes> -> MAC? aren't those the same? (supposing the EOIB is a single L2 domain) | 14:49 |
| yolanda | i'm not familiar with that... also i'm adding the check for /dev/disk/by-label, and the lsblk -L commands | 14:50 |
| milan | or sambetts maybe knows? ^^ | 14:50 |
| openstackgerrit | Jarrod Johnson proposed openstack/pyghmi: Add a test for part of SDR https://review.openstack.org/416977 | 14:50 |
| dtantsur | morning jroll | 14:51 |
| jroll | jlvillal: yeah, saw that :| | 14:51 |
| jroll | morning dtantsur, vdrok, jlvillal | 14:51 |
| vdrok | morning jlvillal | 14:51 |
| jlvillal | Hi everyone :) | 14:52 |
| dtantsur | hey jlvillal | 14:52 |
| jlvillal | :) | 14:52 |
| sambetts | milan: I assume the neigh is for neighbor, which would be the switch port not the NIC (just a guess though) | 14:52 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/pyghmi: Add base for unittests https://review.openstack.org/416996 | 14:52 |
| lucasagomes | jjohnson2, ^ | 14:52 |
| *** UForgotten has quit IRC | 14:52 | |
| milan | sambetts, might be yeah | 14:53 |
| * milan unsure how that works, whether it's a next hop or what | 14:53 | |
| milan | esp. w/r fireballing it | 14:53 |
| milan | *firewalling | 14:54 |
| * dtantsur prefers fireballing | 14:54 | |
| mgould | Dragonball-driven development | 14:54 |
| jroll | fireball all the things | 14:54 |
| mat128 | Diablo 1? | 14:54 |
| lucasagomes | yolanda, https://github.com/openstack/ironic/blob/628a2f520cbb21f3fbb44d38875fb29142fe82f7/etc/ironic/rootwrap.d/ironic-lib.filters | 14:55 |
| jjohnson2_ | lucasagomes: thanks, watching the jobs run, I think the doc part is also needed for it to be content, but haven't seen the failure explanation for that yet | 14:56 |
| yolanda | ah lucasagomes, good to know | 14:56 |
| yolanda | going to update it | 14:56 |
| lucasagomes | yolanda, yeah, this is a whitelist of commands ironic can run as root | 14:56 |
| *** palmertime has joined #openstack-ironic | 14:56 | |
| sambetts | milan: “ip neigh show” is the new method to show arp (from IPROUTE2) | 14:56 |
| lucasagomes | jjohnson2, yeah, lemme take a look at the docs too | 14:56 |
| sambetts | milan: not sure if related, but it seems to be arp related :/ | 14:57 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/pyghmi: Add base for unittests https://review.openstack.org/416996 | 14:57 |
| jjohnson2_ | lucasagomes: hmm /usr/local/jenkins/slave_scripts/run-docs.sh: line 31: doc/build/html/.root-marker: No such file or directory | 14:57 |
| *** UForgotten has joined #openstack-ironic | 14:57 | |
| lucasagomes | jjohnson2, yeah basically we have no docs whatsoever heh | 14:58 |
| milan | sambetts, thx, I'm looking into https://review.openstack.org/#/c/264257/45/ironic_inspector/firewall.py | 14:58 |
| patchbot_ | patch 264257 - ironic-inspector - Adding InfiniBand Support | 14:58 |
| *** absubram has joined #openstack-ironic | 14:58 | |
| milan | sambetts, so w/ regular bm nodes this firewall trick w/ mac address works cause those are on the same L2 domain, correct? | 14:59 |
| milan | * L2 domain of the interface the dnsmasq uses | 15:00 |
| milan | sambetts, iirc infiniband is a switched network, so you've got a direct connection from A to B for Ethernet over InfiniBand, i think the MACs should be the same then: B.mac == A.neigh(B).mac | 15:04 |
| milan | but I'm getting lost between what IPA gives in the inventory and what to put to the firewall | 15:05 |
| *** chlong has joined #openstack-ironic | 15:05 | |
| TheJulia | Good morning | 15:05 |
| milan | morning TheJulia! :) | 15:06 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/pyghmi: Add unittest and documentation base https://review.openstack.org/416996 | 15:07 |
| lucasagomes | jjohnson2, ^ that should do it | 15:07 |
| lucasagomes | normally I would separate in 2 patches but, since the tests are failing we need to squash it into one | 15:07 |
| * lucasagomes watches the gate | 15:08 | |
| lucasagomes | oh wait... we already had some docs | 15:08 |
| lucasagomes | urgh... | 15:08 |
| yolanda | lucasagomes, so blkid -L worked, while blkid -t LABEL=config-2, did not | 15:10 |
| yolanda | blkid -L config-2 /dev/disk/by-path/ip-192.168.24.8:3260-iscsi-iqn.2008-10.org.openstack:e09be09e-bfc7-4d18-826f-3a4104ff3837-lun-1" execute /usr/lib/python2.7/site-packages/ironic_lib/utils.py:102 | 15:10 |
| yolanda | 2017-01-05 17:07:04.756 15894 DEBUG ironic_lib.utils [req-6e446711-1e9f-407f-8413-dc0835baecb7 - - - - -] Command stdout is: "/dev/sda2 | 15:10 |
| yolanda | " | 15:10 |
| patchbot_ | Error: No closing quotation | 15:10 |
| patchbot_ | Error: No closing quotation | 15:10 |
| * jlvillal finds it annoying that his UPS package arrived locally last night but still says it won't be delivered until tomorrow :( | 15:11 | |
| dtantsur | morning TheJulia | 15:12 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/pyghmi: Add base for unittests and fix docs creation https://review.openstack.org/416996 | 15:13 |
| yolanda | going to test a full deploy with that, and i'm fine to assign that bug myself and fix it | 15:13 |
| lucasagomes | yolanda, o/ so probably we just need to update that command | 15:13 |
| lucasagomes | yolanda, thanks for doing it btw! | 15:13 |
| yolanda | yep | 15:13 |
| yolanda | just testing a deploy with that | 15:13 |
| * lucasagomes fingers crossed | 15:14 | |
| *** Guest41100 is now known as dansmith | 15:15 | |
| sambetts | milan: as far as I know IPA always gives you the mac address of the NIC, which is the src-mac we'll receive the packets from the BM as long as we're in the same ethernet L2 network, for infiniband I'm really unsure, I would assume ethernet over infini band would be simulating an L2 network but there are weird things with Eth over Infiniaband mac address and different vendors if I remember | 15:15 |
| sambetts | correctly | 15:15 |
| vdrok | good morning mgould mat128 and TheJulia | 15:15 |
| *** dansmith is now known as Guest57355 | 15:15 | |
| *** Guest57355 is now known as dansmith | 15:16 | |
| milan | sambetts, thanks... moshele I wish I could see how this behaves in a lab :-/ | 15:17 |
| TheJulia | Sorry I've been kind of absent this week folks. Working on a presentation for a talk I'm giving in a few weeks. | 15:17 |
| mariojv | JayF: jcook: just fyi, someone else put up the nova patch for ironic rescue: https://review.openstack.org/#/c/416487/ | 15:20 |
| patchbot_ | patch 416487 - nova - virt/ironic: Implement rescue and unrescue | 15:20 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/pyghmi: Add base for unittests and fix docs https://review.openstack.org/416996 | 15:25 |
| *** ijw has joined #openstack-ironic | 15:26 | |
| *** rbartal has quit IRC | 15:27 | |
| *** ijw has quit IRC | 15:31 | |
| lucasagomes | http://logs.openstack.org/96/416996/5/check/gate-pyghmi-docs-ubuntu-xenial/98ff2e3/console.html#_2017-01-05_15_29_07_180009 hmm | 15:31 |
| jjohnson2_ | lucasagomes: yeah I was just looking at that | 15:35 |
| jjohnson2_ | lucasagomes: btw, how did you know 96 was the correct number? | 15:36 |
| lucasagomes | jjohnson2_, which 96 ? | 15:36 |
| yolanda | lucasagomes, seems it worked! | 15:36 |
| jjohnson2_ | lucasagomes: I didn't see in zuul or gerrit where the log files would be (I guessed) | 15:36 |
| lucasagomes | yolanda, w00t!! | 15:36 |
| jjohnson2_ | lucasagomes: the logsopenstack.org url | 15:36 |
| lucasagomes | jjohnson2, oh I was watching the http://status.openstack.org/ | 15:37 |
| jjohnson2_ | lucasagomes: I appreciate the help | 15:37 |
| lucasagomes | for the docs test to run | 15:37 |
| lucasagomes | http://status.openstack.org/zuul I mean | 15:37 |
| jjohnson2_ | lucasagomes: ahh, zuul.openstack.org was what I was looking at, thanks | 15:37 |
| yolanda | [heat-admin@overcloud-novacompute-0 ~]$ lsblk -o name,label | 15:38 |
| yolanda | NAME LABEL | 15:38 |
| yolanda | vda | 15:38 |
| yolanda | ├─vda1 | 15:38 |
| yolanda | ├─vda2 config-2 | 15:38 |
| yolanda | └─vda3 | 15:38 |
| yolanda | ├─vg-var | 15:38 |
| yolanda | ├─vg-tmp | 15:38 |
| yolanda | ├─vg-swap | 15:38 |
| yolanda | ├─vg-home | 15:38 |
| yolanda | │ └─home | 15:38 |
| lucasagomes | yolanda, good stuff. Please ping me when you put up the patch for fixing it | 15:38 |
| yolanda | └─vg-root | 15:38 |
| yolanda | amazing! | 15:38 |
| lucasagomes | yolanda, http://paste.openstack.org/ | 15:38 |
| yolanda | sorry! | 15:38 |
| lucasagomes | good stuff | 15:38 |
| lucasagomes | it's ok :-) | 15:38 |
| *** d0ugal has quit IRC | 15:38 | |
| JayF | lucasagomes: responded to your comment on that rescue spec :) fwiw, that's how we implemented rescue downstream at onmetal (using a crypt'd hash all the way through) | 15:39 |
| lucasagomes | JayF, right... so let's use it upstream too ? | 15:39 |
| lucasagomes | I think it's better than plaintext | 15:39 |
| JayF | lucasagomes: I put a couple of good reasons in the spec why we shouldn't | 15:40 |
| * lucasagomes missed that | 15:40 | |
| lucasagomes | ok, will read it | 15:40 |
| JayF | No, I mean, in comments | 15:40 |
| lucasagomes | ah right | 15:40 |
| JayF | reply to your comment | 15:40 |
| JayF | just making sure the beef of that conversation is captured there :) | 15:40 |
| lucasagomes | ok I will take a look in a min just trying to figure out this pyghmi problem with the docs generation | 15:40 |
| lucasagomes | JayF, perfect yeah | 15:40 |
| NobodyCam | good morning Ironic'ers | 15:41 |
| JayF | lucasagomes: I looked at that message, it's not an error I've seen before | 15:41 |
| *** aparnav has quit IRC | 15:41 | |
| JayF | lucasagomes: the pgyhmi docs used to work, right? This isn't an initial setup? | 15:41 |
| *** d0ugal has joined #openstack-ironic | 15:41 | |
| lucasagomes | yeah, same for me... I don't know why the docs are not being created under pyghmi/doc/build either (right now it's creating a pyghmi/build/ | 15:41 |
| lucasagomes | )* | 15:41 |
| lucasagomes | JayF, apparently so | 15:41 |
| lucasagomes | the conf.py and index.rst were there before | 15:42 |
| JayF | hmm, strange | 15:42 |
| JayF | I'll look at see if anything jumps out at me | 15:42 |
| lucasagomes | NobodyCam, morning | 15:42 |
| NobodyCam | morning lucasagomes :) | 15:43 |
| mariojv | jroll: you may want to reconsider your +2 on https://review.openstack.org/#/c/407236 | 15:43 |
| patchbot_ | patch 407236 - ironic-specs - Update rescue spec | 15:43 |
| mariojv | reasoning is stated in the comment there, but it's in more detail on L216 of https://review.openstack.org/#/c/350831/15/ironic/common/states.py | 15:43 |
| patchbot_ | patch 350831 - ironic - Add API and conductor methods for [un]rescue | 15:43 |
| mariojv | ^ cc: JayF jcook lucasagomes mat128 | 15:44 |
| JayF | mariojv: your suggestion is not something we should do | 15:44 |
| mariojv | why? | 15:44 |
| JayF | mariojv: think about the nova case: if a vm is in rescue, you should be able to delete it | 15:44 |
| openstackgerrit | yolanda.robla proposed openstack/ironic-lib: Fix retrieval of config-2 existing partition https://review.openstack.org/417022 | 15:44 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/pyghmi: Add base for unittests and fix docs https://review.openstack.org/416996 | 15:45 |
| JayF | mariojv: if a vm is in ERROR because it failed to rescue, do you want ot require that user to have $ops-person touch the ironic backend to delete/fix the node? | 15:45 |
| mariojv | JayF: this is about deleting the node itself, not deleting the instance | 15:45 |
| JayF | OH | 15:45 |
| lucasagomes | jjohnson2, ^ fingers crossed (tl;dr to change the doc build dir you have to update setup.cfg) | 15:45 |
| mariojv | deleting instance will go into manage -> provide anyway | 15:45 |
| JayF | mariojv: yeah, I don't know then | 15:45 |
| JayF | mariojv: that's a good catch | 15:46 |
| mariojv | ooh, also, nova delete won't even work right now | 15:46 |
| mariojv | because we don't have rescue -> manage either | 15:46 |
| mariojv | in addition to the *fail states | 15:46 |
| * jroll is lost | 15:46 | |
| JayF | why do you think a nova delete goes to manageable? | 15:46 |
| JayF | it doesn't. | 15:46 |
| jroll | it goes to deleting :) | 15:46 |
| mat128 | but you can't go from RESCUE to deletig | 15:46 |
| jroll | # A rescued node may be deleted. | 15:47 |
| mariojv | ah, you're right | 15:47 |
| TheJulia | mariojv: delete_allowed_states is actual deletion from the database, not state machine related | 15:47 |
| jroll | machine.add_transition(RESCUE, DELETING, 'delete') | 15:47 |
| JayF | yes, you can | 15:47 |
| jroll | so | 15:47 |
| mariojv | deleting -> manageable | 15:47 |
| JayF | mariojv: where do you see that as possible/ | 15:47 |
| mariojv | JayF: "that" == ? | 15:47 |
| * jroll doesn't see any bugs here | 15:47 | |
| mariojv | this should only affect the fail states then | 15:47 |
| JayF | mariojv: 15:47:17 <mariojv> deleting -> manageable | 15:47 |
| mariojv | JayF: http://docs.openstack.org/developer/ironic/_images/states.svg | 15:48 |
| mariojv | oh, nvm | 15:48 |
| mariojv | it goes directly into cleaning | 15:48 |
| JayF | managable, as a state, is only reachable by explicit operator action | 15:48 |
| JayF | none of our automated transitions will drop it there | 15:48 |
| mariojv | jroll: rescue as currently implemented allows someone to delete a node from the DB with tenant data on it | 15:48 |
| mariojv | jroll: that breaks an existing implicit convention that we don't allow someone to do that without putting it in MANAGEABLE first | 15:49 |
| jroll | uh | 15:49 |
| jroll | no? | 15:49 |
| *** vinsh has joined #openstack-ironic | 15:49 | |
| openstackgerrit | Szymon Borkowski proposed openstack/ironic: [PoC] Add fake fields in Node object and RPC https://review.openstack.org/415462 | 15:49 |
| jroll | mariojv: are you talking about DELETE_ALLOWED_STATES | 15:49 |
| mariojv | jroll: yes | 15:49 |
| lucasagomes | jjohnson2, w00t it's green! http://docs-draft.openstack.org/96/416996/6/check/gate-pyghmi-docs-ubuntu-xenial/960528a//doc/build/html/#pyghmi.ipmi.command | 15:49 |
| jroll | mariojv: those are the states in which you CAN delete | 15:49 |
| jroll | not in which you cannot delete | 15:49 |
| jroll | absence of a state there = cannot delete from db in that state | 15:49 |
| *** Syed__ has joined #openstack-ironic | 15:50 | |
| *** rpioso has joined #openstack-ironic | 15:50 | |
| mat128 | mariojv, JayF: any idea why the state machine generated for that review don't contain the new states? http://docs-draft.openstack.org/31/350831/15/check/gate-ironic-docs-ubuntu-xenial/92b8d41//doc/build/html/_images/states.svg | 15:50 |
| mariojv | i think i may be confusing 2 things | 15:50 |
| mariojv | transition to DELETING means put it back into cleaning? | 15:50 |
| jroll | yes | 15:50 |
| mariojv | not "destroy this node" | 15:50 |
| mariojv | ah, that's why i was confused | 15:50 |
| mat128 | mariojv: thats correct | 15:50 |
| jroll | delete states are about deleting the instance | 15:51 |
| mariojv | OK, never mind | 15:51 |
| TheJulia | :) | 15:51 |
| *** rama_y has joined #openstack-ironic | 15:51 | |
| JayF | I don't think node-delete even messes with the state machine | 15:51 |
| jroll | DELETE /v1/nodes/foo -> mysql -c 'delete from nodes where name="foo";' | 15:51 |
| JayF | it just crushinates it | 15:51 |
| * mariojv thinks node-delete ought to be called node-destroy | 15:51 | |
| JayF | yep, that looks like crushination to me | 15:51 |
| mariojv | JayF: that's right | 15:51 |
| JayF | mariojv: yes please yes yes yes | 15:51 |
| mariojv | and you can't crushinate a rescue/unrescuefail node | 15:51 |
| JayF | mariojv: but we can't because it was already wrong, lol | 15:51 |
| TheJulia | Then again, node-destroy might cause people to think that they are nuking the machine and the contents, delete is just removal of the record | 15:52 |
| mariojv | node-remove ? | 15:52 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/pyghmi: Add base for unittests and fix docs https://review.openstack.org/416996 | 15:52 |
| TheJulia | really, delete provision state verb should be 'undeploy' | 15:52 |
| mariojv | idk, i don't want to bikeshed on that. I understand the difference | 15:52 |
| JayF | TheJulia: I can tell you that there's about a once a year OnMetal tradition | 15:52 |
| JayF | TheJulia: of accidentally deleting a node | 15:52 |
| jroll | lol | 15:52 |
| mariojv | hahaha | 15:52 |
| JayF | TheJulia: because of that horribly named cli command | 15:52 |
| TheJulia | lol | 15:52 |
| sambetts | in openstack sdk it is undeploy :) | 15:53 |
| TheJulia | JayF: for a while at HP, I got a number of tickets with similar confusion | 15:53 |
| JayF | like it's legitimately easy to do | 15:53 |
| jroll | sambetts: so we need to kill the ironic cli | 15:54 |
| JayF | it's not even like a dumb mistake, it's just easy to do | 15:54 |
| jroll | :D | 15:54 |
| mat128 | and this, friends, is why semantics are important :) | 15:54 |
| sambetts | I'm trying to bring a number of people up to speed and its confusing as hell :-P | 15:54 |
| sambetts | jroll: hehe yeah | 15:54 |
| TheJulia | mat128: +++ | 15:54 |
| mariojv | i like JayF's suggestion. ironic node-crushinate | 15:54 |
| sambetts | ++ | 15:54 |
| mat128 | reminds me of Xen's "xm destroy" for stopping a VM | 15:55 |
| TheJulia | but what does crushinate do? Does it dispatch a robot to crush the server? Because that would be totally awesome to watch | 15:55 |
| * TheJulia feels like there should be an easter egg in the cli | 15:55 | |
| JayF | heh | 15:55 |
| mat128 | haha | 15:55 |
| JayF | RobotInterface | 15:55 |
| sambetts | mat128: virsh is the same, destory == shutdown domain -_- | 15:56 |
| * mat128 would approve any review doing it | 15:56 | |
| JayF | if you're using the no-robot driver | 15:56 |
| sambetts | mat128: not delete the domain | 15:56 |
| jroll | finally https://github.com/blog/2299-search-commit-messages | 15:56 |
| JayF | the error is "I'm afraid I can't do that, deployer" | 15:56 |
| mat128 | yup | 15:56 |
| sambetts | jroll: :O!! | 15:56 |
| mat128 | sambetts: I think Xen came up with it first | 15:56 |
| TheJulia | and what sufficiently be crushionation.... 20% compression, 50% compression, shredding? | 15:56 |
| mat128 | then virsh/libvirt just followed | 15:56 |
| mariojv | fancy. better than "git log" and doing a search with / | 15:56 |
| sambetts | mat128: ah | 15:56 |
| TheJulia | JayF: <3 the hal style reply. | 15:56 |
| *** dyasny has quit IRC | 15:57 | |
| sambetts | tbh I was thinking the other day if we should have called Ironic nodes domains :-P | 15:57 |
| mat128 | nooooo | 15:57 |
| sambetts | ;) | 15:57 |
| mat128 | first step of getting someone up to speed on openstack baremetal: | 15:57 |
| jroll | mariojv: usually for me it's git log | grep foo; # grrrrr; git log | grep -C 5 foo; # OMG STAHP; git log | grep -C 100 foo | 15:57 |
| mat128 | instance = nova, node = ironic | 15:57 |
| mat128 | once they're past this, they start getting it | 15:57 |
| JayF | mat128: very very yes | 15:57 |
| mat128 | and when they say "server", stop them and ask which one they mean | 15:58 |
| sambetts | jroll: you know about git log -S and -G right? | 15:58 |
| JayF | mat128: especially at Rackspace, we have separate nova/ironic dashboards, so it's like "look at a node here:, and look at a instance here:" | 15:58 |
| mat128 | it also makes much more sense, as in you're leasing a spot on the hardware | 15:58 |
| mariojv | that's probably more efficient for a lot of cases. sometimes i want to see the results 1 by 1 | 15:58 |
| mat128 | to make things worse, we have another software that manages "devices" | 15:59 |
| mat128 | and both nodes and instances are represented as devices, so yeah | 15:59 |
| mariojv | sambetts: can you limit that to just commit messages? | 15:59 |
| mariojv | manpage says it searches full diff | 15:59 |
| jroll | sambetts: I do now :) | 15:59 |
| sambetts | mariojv: one of the options searches commit messages only and one of the options does the diffs I think | 16:00 |
| sambetts | can't remember which way round though | 16:00 |
| sambetts | Ithought that was the case anyway I might be wrong | 16:02 |
| mariojv | i'm sure there's some way to limit by files | 16:02 |
| mariojv | anyway, the new github feature is still neat | 16:02 |
| sambetts | yeah much needed, because I still use github to link other people to a change and being able to find a commit will help | 16:03 |
| mat128 | JayF: vdrok has a good point in https://review.openstack.org/#/c/412523/, write_configdrive isn't exposed and is implicitly called by prepare_image which means our initial impl. will contain a single deploy step, wdyt? | 16:03 |
| patchbot_ | patch 412523 - ironic-specs - Ironic deployment steps | 16:03 |
| *** ijw has joined #openstack-ironic | 16:05 | |
| JayF | mat128: yep, he's right. good catch | 16:06 |
| mat128 | should we (eventually) make it an explicit step? | 16:07 |
| JayF | I think if we did, it should be 1000% separate from all this other work | 16:07 |
| *** rcernin has quit IRC | 16:07 | |
| mat128 | JayF: yes agreed, but would you eventually see this split from imaging? | 16:07 |
| JayF | mat128: The value in that would be if someone wanted to customize the write_configdrive step; for instance, imagine if someone wanted to implement configdrive in virtual media | 16:08 |
| JayF | mat128: but I think it should be a solution to a problem, not done just for the sake of doing it | 16:08 |
| *** dyasny has joined #openstack-ironic | 16:09 | |
| * dtantsur starts https://bugs.launchpad.net/ironic-inspector/+bug/1654318 | 16:09 | |
| openstack | Launchpad bug 1654318 in Ironic Inspector "[RFE] Deprecate setting IPMI credentials" [Wishlist,Triaged] - Assigned to Dmitry Tantsur (divius) | 16:09 |
| mat128 | ok good | 16:09 |
| stevemar | jroll: are the ironic drivers in-tree or out of tree? | 16:09 |
| jroll | stevemar: both! | 16:09 |
| stevemar | jroll: lol, what the... | 16:10 |
| jroll | stevemar: most out of tree drivers are here: https://github.com/openstack/ironic-staging-drivers | 16:10 |
| jroll | but if they have CI they may be in tree | 16:10 |
| stevemar | jroll: i see | 16:10 |
| jroll | (out of tree are not part of the official ironic project) | 16:10 |
| *** ijw has quit IRC | 16:10 | |
| stevemar | where are they in-tree? here? https://github.com/openstack/ironic/tree/master/ironic/drivers/modules | 16:10 |
| dtantsur | jroll, I guess I need a formal rfe-approved for https://bugs.launchpad.net/ironic-python-agent/+bug/1654318 despite the ML discussion, right? | 16:10 |
| openstack | Launchpad bug 1654318 in ironic-python-agent "[RFE] Deprecate setting IPMI credentials during inspection" [Wishlist,Triaged] - Assigned to Dmitry Tantsur (divius) | 16:10 |
| jroll | stevemar: yes, a better registry would be: https://github.com/openstack/ironic/blob/master/setup.cfg#L44 | 16:11 |
| stevemar | gotcha | 16:11 |
| jroll | dtantsur: sure, let me read again quickly and then approve | 16:12 |
| *** moshele has quit IRC | 16:13 | |
| jroll | dtantsur: only question, how do previous API versions work if we remove the code? | 16:13 |
| dtantsur | jroll, they fail with HTTP 400 (the same if the configuration option is not set to True FWIW) | 16:13 |
| jroll | dtantsur: ah right, there's a config, fair enough | 16:14 |
| mariojv | jcook: left 2 small comments on https://review.openstack.org/#/c/350831 (1 line change and running a tox command) | 16:17 |
| patchbot_ | patch 350831 - ironic - Add API and conductor methods for [un]rescue | 16:17 |
| jlvillal | jroll: FYI: on the devstack stable/newton issue. Basically the failing job should not be voting. I will submitting a project-config patch shortly | 16:18 |
| jroll | jlvillal: awesome, thanks | 16:18 |
| jjohnson2_ | lucasagomes: ok, have no idea about that one | 16:18 |
| lucasagomes | JayF, yeah I haven't considered a non-linux rescue image | 16:18 |
| lucasagomes | so your comment makes sense | 16:19 |
| JayF | lucasagomes: cool, ty | 16:19 |
| lucasagomes | JayF, +2+a that patch then | 16:20 |
| JayF | Awesome, thanks | 16:20 |
| JayF | jcook: mariojv ^ rescue spec landing | 16:20 |
| *** tesseract has quit IRC | 16:21 | |
| openstackgerrit | Merged openstack/ironic: Fix the comma's wrong locations. https://review.openstack.org/416957 | 16:22 |
| jroll | sambetts: hey, who's working on the metadata updates for portgroups in nova? (alternatively got a link to the patch?) | 16:23 |
| sambetts | jroll: I think its this https://review.openstack.org/#/c/388756/ | 16:23 |
| patchbot_ | patch 388756 - nova - Generate necessary network metadata for portgroups | 16:23 |
| jroll | sweet, thanks | 16:23 |
| jroll | oh we probably need a client release eh | 16:24 |
| jroll | oh, no, we need to wait for attach/detach | 16:24 |
| jroll | or not wait, but it isn't super useful | 16:24 |
| sambetts | jroll: yeah, i've not seen vsaienk0 to find out what the status is on the patch I left before the holidays | 16:25 |
| openstackgerrit | Merged openstack/ironic-specs: Update rescue spec https://review.openstack.org/407236 | 16:25 |
| jroll | well, RPC APIs merged | 16:25 |
| jroll | so just REST stuff left | 16:25 |
| jroll | \o/ | 16:25 |
| jroll | and docs, and client, and nova | 16:25 |
| jroll | :) | 16:25 |
| sambetts | sweet :D | 16:26 |
| *** dyasny_ has joined #openstack-ironic | 16:26 | |
| *** vgadiraj has joined #openstack-ironic | 16:27 | |
| jroll | sambetts: need to keep pressing, thoughu, nova feature freeze is jan 23-27 | 16:28 |
| *** rama_y has quit IRC | 16:28 | |
| jroll | so 3 weeks to get all that done | 16:28 |
| jroll | I think we can do it | 16:28 |
| *** hoangcx has quit IRC | 16:29 | |
| *** dyasny has quit IRC | 16:30 | |
| *** vgadiraj has quit IRC | 16:30 | |
| *** baoli has quit IRC | 16:31 | |
| sambetts | yeah, we'll have to merge and make a client release before the nova stuff can land | 16:31 |
| vdrok | sambetts: jroll status of the ironic/client patches is - ready for review :) looked mostly good last time I was reviewing it | 16:31 |
| jroll | ++ | 16:31 |
| sambetts | vdrok: cool :D I'll try to review them soon | 16:31 |
| jroll | \o/ for getting portgroups finished over the holidays btw | 16:31 |
| jroll | that's been a long one, thank you all | 16:31 |
| lucasagomes | jjohnson2, http://logs.openstack.org/96/416996/7/check/gate-tempest-dsvm-ironic-pxe_ipmitool-pyghmi-src-ubuntu-xenial/1ac35ff/console.html#_2017-01-05_16_16_59_677546 | 16:34 |
| lucasagomes | this should be fixed when we fix Ironic | 16:34 |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic-inspector: Deprecate setting IPMI credentials https://review.openstack.org/417041 | 16:36 |
| *** hwoarang_ is now known as hwoarang | 16:36 | |
| vdrok | good night everyone! | 16:37 |
| dtantsur | g'night vdrok | 16:37 |
| sambetts | cya vdrok o/ | 16:37 |
| milan | night vdrok! :) | 16:38 |
| *** jvgrant_ has joined #openstack-ironic | 16:48 | |
| *** jvgrant has quit IRC | 16:51 | |
| *** e0ne has quit IRC | 16:52 | |
| *** UForgotten has quit IRC | 16:53 | |
| *** amotoki has quit IRC | 16:55 | |
| *** zackf has joined #openstack-ironic | 16:58 | |
| *** pcaruana has quit IRC | 16:58 | |
| *** ashestakov has quit IRC | 17:00 | |
| *** rbudden has joined #openstack-ironic | 17:01 | |
| *** ijw has joined #openstack-ironic | 17:01 | |
| *** UForgotten has joined #openstack-ironic | 17:05 | |
| *** fritzstauffach-1 has joined #openstack-ironic | 17:06 | |
| *** fritzstauffacher has quit IRC | 17:06 | |
| *** dyasny_ has quit IRC | 17:12 | |
| * sambetts -> meeting then home | 17:12 | |
| sambetts | night all | 17:12 |
| *** dyasny has joined #openstack-ironic | 17:12 | |
| lucasagomes | sambetts, see ya | 17:12 |
| * lucasagomes is heading too | 17:12 | |
| lucasagomes | have a great evening all | 17:12 |
| sambetts | cya lucasagomes | 17:12 |
| *** lucasagomes is now known as lucas-afk | 17:12 | |
| *** derekh has quit IRC | 17:12 | |
| *** sambetts is now known as sambetts|afk | 17:13 | |
| *** athomas has quit IRC | 17:20 | |
| *** LiberalCarrot has joined #openstack-ironic | 17:22 | |
| *** LiberalCarrot has left #openstack-ironic | 17:22 | |
| *** moshele has joined #openstack-ironic | 17:23 | |
| *** trown is now known as trown|lunch | 17:25 | |
| *** shobhar_ has joined #openstack-ironic | 17:27 | |
| *** vgadiraj has joined #openstack-ironic | 17:29 | |
| *** moshele has quit IRC | 17:37 | |
| *** moshele has joined #openstack-ironic | 17:37 | |
| *** moshele has quit IRC | 17:40 | |
| *** ohamada has quit IRC | 17:44 | |
| *** LiberalSquash has joined #openstack-ironic | 17:45 | |
| *** LiberalSquash has left #openstack-ironic | 17:45 | |
| *** dtantsur is now known as dtantsur|afk | 17:50 | |
| *** fritzstauffacher has joined #openstack-ironic | 17:53 | |
| *** fritzstauffach-1 has quit IRC | 17:53 | |
| *** AnxiousGarlic has joined #openstack-ironic | 17:58 | |
| *** AnxiousGarlic has left #openstack-ironic | 17:58 | |
| *** jkilpatr has quit IRC | 18:01 | |
| *** mgould is now known as mgould|afk | 18:11 | |
| *** jkilpatr has joined #openstack-ironic | 18:17 | |
| *** palmertime1 has joined #openstack-ironic | 18:22 | |
| *** ChubYann has joined #openstack-ironic | 18:23 | |
| *** rama_y has joined #openstack-ironic | 18:24 | |
| *** palmertime has quit IRC | 18:24 | |
| *** trown|lunch is now known as trown | 18:24 | |
| *** rbudden has quit IRC | 18:32 | |
| openstackgerrit | Chris Krelle proposed openstack/ironic-specs: Intel Node Manager sensor collection support https://review.openstack.org/413249 | 18:38 |
| *** Nisha_Agarwal has quit IRC | 18:40 | |
| *** vgadiraj has quit IRC | 18:40 | |
| *** baoli has joined #openstack-ironic | 18:43 | |
| *** jkilpatr_ has joined #openstack-ironic | 19:01 | |
| *** vgadiraj has joined #openstack-ironic | 19:02 | |
| *** jkilpatr has quit IRC | 19:05 | |
| *** milan has quit IRC | 19:08 | |
| jlvillal | I feel like the gate issues are an exercise in yak-shaving ... | 19:10 |
| *** rcernin has joined #openstack-ironic | 19:11 | |
| *** harlowja has joined #openstack-ironic | 19:16 | |
| *** dsneddon_afk is now known as dsneddon | 19:16 | |
| *** yufei has quit IRC | 19:18 | |
| palmertime1 | I just deployed a small Newton overcloud and i'm getting 504 errors everywhere. re-starting haproxy on the controller helps for a time then comes back again. I have increased maxconn for the mysql in haproxy.cfg but still will get 504 errors after a while. | 19:26 |
| palmertime1 | Wrong channel … sorry! | 19:27 |
| openstackgerrit | John L. Villalovos proposed openstack/ironic: Remove service argument from tempest plugin client manager https://review.openstack.org/416926 | 19:30 |
| *** jrcloud has joined #openstack-ironic | 19:32 | |
| *** zackf has quit IRC | 19:34 | |
| *** marlinc has joined #openstack-ironic | 19:43 | |
| *** jrcloud is now known as zackf | 19:46 | |
| *** mjturek has quit IRC | 19:50 | |
| *** e0ne has joined #openstack-ironic | 19:56 | |
| *** mtanino has joined #openstack-ironic | 19:57 | |
| *** Sukhdev has joined #openstack-ironic | 20:02 | |
| *** chlong has quit IRC | 20:03 | |
| *** mjturek1 has joined #openstack-ironic | 20:06 | |
| TheJulia | NobodyCam: I left you fresh feedback on 413249... I'm feeling a little nitpicky, but hopefully some if it makes sense | 20:15 |
| NobodyCam | hehehe :) thank you :) | 20:16 |
| *** bfournie has quit IRC | 20:16 | |
| *** zackf has quit IRC | 20:26 | |
| *** e0ne has quit IRC | 20:36 | |
| *** shobhar__ has joined #openstack-ironic | 20:36 | |
| *** shobhar has joined #openstack-ironic | 20:39 | |
| *** shobhar__ has quit IRC | 20:39 | |
| *** shobhar_ has quit IRC | 20:39 | |
| *** shobhar has quit IRC | 20:44 | |
| *** mjturek1 has quit IRC | 20:47 | |
| openstackgerrit | Chris Krelle proposed openstack/ironic-specs: Intel Node Manager sensor collection support https://review.openstack.org/413249 | 20:48 |
| NobodyCam | TheJulia: ^^^ addresses the your comments | 20:49 |
| *** bfournie has joined #openstack-ironic | 20:49 | |
| *** zackf has joined #openstack-ironic | 20:51 | |
| openstackgerrit | Chris Krelle proposed openstack/ironic-specs: Intel Node Manager sensor collection support https://review.openstack.org/413249 | 20:57 |
| TheJulia | NobodyCam: Thank you | 20:59 |
| mrda | Morning Ironic | 21:05 |
| NobodyCam | good morning mrda | 21:09 |
| jjohnson2_ | Hmm, is it the case that no one is using pyghmi in openstack for client side? Just ipmitool? | 21:12 |
| mat128 | jjohnson2_: we tried during our initial implementation but the "ipminative" driver has not followed a contract change in pyghmi | 21:12 |
| mat128 | we fixed it (and also pushed upstream) but encountered other issues | 21:12 |
| mat128 | https://github.com/openstack/ironic/commit/0b927ffc024151a3f24b7097104f8139414524dc | 21:13 |
| mrda | NobodyCam: o/ | 21:15 |
| openstackgerrit | Merged openstack/bifrost: Add dhcp related tasks to inspection role https://review.openstack.org/415240 | 21:16 |
| jjohnson2_ | mat128: hmm, I feel a bit sad that wrapping ipmitool utility is considered preferable to the native python code... | 21:17 |
| mat128 | jjohnson2_: I would much rather have/use native code than wrapping ipmitool believe me :) | 21:17 |
| mat128 | which is why we tried ipminative first and even put effort into making it work | 21:18 |
| *** fritzstauffacher has quit IRC | 21:18 | |
| mat128 | I can always try to switch back | 21:18 |
| JayF | I think not many folks have tested ipminative at scale, and the ones who have (mat128 + onmetal) ran into issues that discouraged us | 21:18 |
| jjohnson2_ | Hmm, would like to hear about scaling issues. Largest one I have tested is 3,000 servers | 21:19 |
| jjohnson2_ | though it wasn't openstack admitively | 21:19 |
| mat128 | I found something interesting by git blaming our internal repo | 21:19 |
| JayF | I suspect a lot of the issues we ran into have been fixed since the fixes for virtualbmc | 21:19 |
| JayF | but we aren't going to proactively change config in onmetal at tihs point | 21:19 |
| mat128 | a commit titled "Enable IPMI power management using ipmitool" | 21:19 |
| mat128 | Pyghmy is using a threading model, locks and package global data storage that are incompatible with green threads used by ironic. | 21:20 |
| *** rbudden has joined #openstack-ironic | 21:20 | |
| * jroll notes ipmitool's initial release was in 2003, 14 year old software almost always beats 3 year old software for reliability (especially when it comes to IPMI, I suspect) | 21:20 | |
| jjohnson2_ | hmm, I use it in an eventlet patched context usually... | 21:20 |
| *** MattMan_1 has quit IRC | 21:21 | |
| mat128 | jjohnson2_: not saying that excerpt from an internal commit is to be believed, but thats about as much as I can find about the issue | 21:21 |
| jjohnson2_ | most of that is for the sake of pooling on filehandles to avoid massive filehandle needs... | 21:21 |
| mat128 | jjohnson2_: the individual who worked on this will be with us next week, if that helps | 21:21 |
| jjohnson2_ | ok, let me know any suggestions | 21:21 |
| mat128 | I'm genuinely surprised to find a summary of the issue in the commit message | 21:22 |
| jjohnson2_ | I've gotten a lot more non-openstack related feedback than I have openstack feedback | 21:22 |
| *** MattMan_1 has joined #openstack-ironic | 21:22 | |
| mat128 | as we arent very good at writing proper commit messages internally, even if we're getting better | 21:22 |
| mat128 | well first step is probably ensure we have a dvsm run using ipminative | 21:22 |
| mat128 | but im sure we have, otherwise that driver would be unsupported | 21:22 |
| mat128 | so that at least means it works at small scale | 21:23 |
| jjohnson2_ | interestingly, a large part of why I did it was to get better performance than fork-execing ipmitool | 21:23 |
| jroll | mat128: https://github.com/openstack/ironic/blob/master/ironic/drivers/agent.py#L87 | 21:23 |
| mat128 | jroll: oh.. | 21:23 |
| jroll | :) | 21:23 |
| jroll | we could easily make one | 21:23 |
| openstackgerrit | Merged openstack/bifrost: Add checks for keystone endpoints https://review.openstack.org/415444 | 21:23 |
| jroll | and I would like to | 21:24 |
| mat128 | would it pass? with the broken contract we have seen I honestly believed nobody was running ipminative anywhere | 21:24 |
| jjohnson2_ | feels weird being in openstack/ but not generally used by openstack | 21:24 |
| mat128 | im all for making it supported/work and switching to it, at scale | 21:24 |
| JayF | jjohnson2_: we use pyghmi, via virtualbmc, in all voting ironic jobs now :) | 21:24 |
| jjohnson2_ | yeah, that's the least fun part though ;) | 21:24 |
| JayF | jjohnson2_: also there is no stackforge/ or similar anymore, openstack/ is wher eanyone using openstack infra has the project live | 21:24 |
| *** ashestakov has joined #openstack-ironic | 21:25 | |
| jroll | we use the server part of pyghmi, yes | 21:25 |
| jroll | not the client | 21:25 |
| jjohnson2_ | well I'm off to pick my daughter up from school | 21:25 |
| jjohnson2_ | good day all | 21:25 |
| openstackgerrit | Mathieu Mitchell proposed openstack/ironic: Attempt replacing ipmitool with ipminative/pyghmi https://review.openstack.org/417196 | 21:26 |
| jroll | \o | 21:26 |
| mat128 | this ^ should tell us | 21:26 |
| jroll | heh | 21:27 |
| jjohnson2_ | well thanks for checking. It's not a terribly practical matter for you all. | 21:27 |
| *** jkilpatr_ has quit IRC | 21:27 | |
| jjohnson2_ | btw when is the tempest gate going to be fixed? | 21:27 |
| JayF | jjohnson2_: working on it :D | 21:27 |
| mat128 | jjohnson2_: it would at least save me from seeing that ipmitool parsing debug output all the time :D | 21:28 |
| jroll | jjohnson2_: don't get me wrong, I'd love to see pyghmi succeed and get used in production, just nobody has taken the time to suss out all the issues | 21:29 |
| jroll | and I don't have a deployment to mess with anymore :( | 21:29 |
| jjohnson2_ | ok, well I'm off for real now | 21:29 |
| jjohnson2_ | see you all later | 21:30 |
| jroll | have a good one | 21:30 |
| jjohnson2_ | hopefully tomorrow morning the gate will let pyghmi reviews pass verify again... | 21:30 |
| * jroll wonders if any inspector folks are around | 21:30 | |
| mat128 | o/ jjohnson2_ | 21:30 |
| jcook | mariojv "allowing RESCUEWAIT -> RESCUEFAIL via API ("abort") to match the cleaning state flow." | 21:33 |
| jcook | I believe you are correct | 21:33 |
| jcook | RESCUEWAIT -> RESCUEFAIL via abort() makes sense for when you go "oh snap! I didn't write dat password down" | 21:34 |
| jcook | abort, rescue | 21:34 |
| *** jjohnson2_ has quit IRC | 21:35 | |
| mat128 | jcook: or the machine is never booting and you are stuck in RESCUEWAIT ;) | 21:37 |
| JayF | imo any *WAIT states should support abort | 21:37 |
| mat128 | ++ | 21:37 |
| jcook | I think it times out in that case? and goes to RESCUEFAIL anyways | 21:37 |
| JayF | or you just go 'oh crap, the DC told me the computer is on fire, abort and crushinate" | 21:37 |
| jcook | maybe not, but yeah | 21:37 |
| mat128 | *ING too, how many times have you tried to issue an action that Ironic just says "nope, cant do that" | 21:38 |
| JayF | yeah but *ING supposedly we're doing something SO IMPORTANT it can't be interrupted | 21:38 |
| jroll | abort is an admin only thing btw | 21:38 |
| mat128 | well, accept the abort and handle it as soon as possible | 21:38 |
| jroll | so the user not writing down the password doesn't help | 21:38 |
| jcook | oh? well then | 21:38 |
| jroll | er, s/help/matter here/ | 21:38 |
| jroll | unless we hook that into nova, which I suppose we could maybe? | 21:39 |
| jroll | ¯\_(ツ)_/¯ | 21:39 |
| jcook | so JayF's use case is it? | 21:39 |
| mat128 | you could change your mind about the rescue? | 21:39 |
| jroll | mat128: the user wouldn't be able to talk to ironic to abort the rescue though | 21:39 |
| mat128 | oh, yeah | 21:39 |
| jroll | well, in some cases, including yours | 21:39 |
| jroll | :P | 21:39 |
| mat128 | yup | 21:39 |
| jcook | CLEANWAIT -> CLEANFAIL via abort() is supported and following the pattern for RESCUEWAIT -> RESCUEFAIL via abort() seems correct | 21:40 |
| jcook | ING via abort not so much | 21:40 |
| mat128 | jcook: make rescue similar to what is already there | 21:40 |
| jcook | ++ | 21:40 |
| mat128 | if there is something to do about the ING states, we'll do it separately | 21:40 |
| jcook | sounds good | 21:41 |
| jcook | I'll make that one fix and make a PR to the spec too | 21:41 |
| *** ashestakov has quit IRC | 21:41 | |
| jroll | I still support abort to be clear | 21:41 |
| jroll | :) | 21:41 |
| jcook | machine.add_transition(RECUEWAIT, RECUEFAIL, 'abort') | 21:41 |
| jcook | :) | 21:41 |
| jroll | BZZZZT | 21:42 |
| jcook | did you just get shocked? | 21:42 |
| * jroll hands jcook a box filled with the letter S | 21:42 | |
| jcook | oh lol | 21:42 |
| jcook | I typed that not copied it so we good | 21:43 |
| jroll | heh | 21:43 |
| mat128 | hahaha | 21:43 |
| jroll | adding a py3 dsvm job if someone would like a look: https://review.openstack.org/#/c/417202/ | 21:48 |
| patchbot_ | patch 417202 - openstack-infra/project-config - Add experimental python3 dsvm job for ironic | 21:48 |
| jroll | jlvillal: ^ given you're familiar | 21:49 |
| jlvillal | jroll: Oh sweet. In a meeting at the moment. | 21:52 |
| jroll | jlvillal: cool, no rush | 21:52 |
| *** mjturek has joined #openstack-ironic | 21:52 | |
| mariojv | woohoo, glad we can use abort with rescue | 21:53 |
| * mat128 is out | 21:53 | |
| mat128 | have a good one everyone | 21:53 |
| mariojv | missed keeping that change after i realized the mistake with DELETING in the state machine that i made | 21:53 |
| mariojv | later mat128 | 21:53 |
| openstackgerrit | Jesse J. Cook proposed openstack/ironic-specs: Abort in RESCUEWAIT should be supported https://review.openstack.org/417205 | 21:54 |
| openstackgerrit | Jesse J. Cook proposed openstack/ironic: Add API and conductor methods for [un]rescue https://review.openstack.org/350831 | 21:54 |
| jcook | mariojv x2 ^ | 21:54 |
| *** trown is now known as trown|outtypewww | 21:57 | |
| *** dprince has quit IRC | 21:58 | |
| *** vgadiraj has quit IRC | 22:10 | |
| *** jkilpatr_ has joined #openstack-ironic | 22:10 | |
| *** mjturek has quit IRC | 22:12 | |
| *** Goneri has quit IRC | 22:21 | |
| *** baoli has quit IRC | 22:26 | |
| jroll | JayF: did you see pabelanger's DIB email by chance? | 22:30 |
| jroll | that might make a perfect IPA image | 22:31 |
| JayF | I did. | 22:31 |
| JayF | And had a very similar thought | 22:31 |
| jroll | :) | 22:31 |
| JayF | but there is a lot of political issues around us getting DIB element in tree and supported | 22:31 |
| JayF | basically we need changes in DIB that for some reason (that I don't understand, and haven't tried to yet, honestly) | 22:31 |
| JayF | they won't merge | 22:31 |
| jroll | wat | 22:31 |
| TheJulia | element deduplication | 22:32 |
| JayF | TheJulia may know more details about it, but that's a lot of the reason I haven't dug with dib hard | 22:32 |
| TheJulia | tl;dr set brain to migraine | 22:32 |
| jroll | :( | 22:32 |
| jroll | TheJulia: I'd love more info | 22:32 |
| *** jheroux has quit IRC | 22:32 | |
| TheJulia | basically support has to land to be able to have duplicate elements and override the existence of one element. Turns out thereis some bug that was attempted to be fixed but it died in -1 land, waiting on something else to be fixed... that I think also died in -1 land | 22:33 |
| jroll | oh, so we can add it in our tree and later remove it there? | 22:33 |
| TheJulia | yup, eventually remove it that is, or make it not really do anything | 22:33 |
| TheJulia | or "Error: go use this element" | 22:34 |
| jroll | I'd be up for adding it now with a different name | 22:34 |
| jroll | and then we could go attack that | 22:34 |
| TheJulia | ++ | 22:34 |
| jroll | or deprecate it | 22:34 |
| jroll | I don't think the name is terribly important, honestly | 22:34 |
| TheJulia | agreed | 22:34 |
| TheJulia | hmmm | 22:34 |
| jroll | okay, so probably not a blocker | 22:35 |
| TheJulia | https://review.openstack.org/#/c/335583/ | 22:35 |
| patchbot_ | patch 335583 - ironic-python-agent - Move diskimage-builder IPA element to IPA tree | 22:35 |
| JayF | ironic-agent-supported | 22:35 |
| JayF | lol | 22:35 |
| TheJulia | https://review.openstack.org/#/c/333597/ | 22:35 |
| patchbot_ | patch 333597 - diskimage-builder - Make svc-map support overridden elements | 22:35 |
| jroll | cool | 22:36 |
| jroll | I will copy this into my todo list for the next couple weeks | 22:36 |
| TheJulia | https://review.openstack.org/#/c/334785/ | 22:36 |
| patchbot_ | patch 334785 - diskimage-builder - Making element overriding explicit (ABANDONED) | 22:36 |
| jroll | and figure out if the above plan is fine with them | 22:36 |
| *** rbudden has quit IRC | 22:36 | |
| jroll | oh he also has a debian-rootfs element in that chain | 22:37 |
| jroll | so we won't need to worry about that little licensing issue we had a while back | 22:37 |
| jroll | nice | 22:38 |
| TheJulia | If it is a separate name, I doub't they would really care right now | 22:38 |
| TheJulia | we would just have to gracefully kill the in-tree element, and eventually just have it spit out an error for our sanity | 22:38 |
| jroll | yeah, still worth a chat now that DIB is much less tripleo-ran | 22:38 |
| TheJulia | yeah | 22:38 |
| jroll | just to make sure everyone is cool | 22:38 |
| JayF | the other nice thing a bout doing it that way | 22:39 |
| TheJulia | I think greg is back sometime next week, so timing then should work | 22:39 |
| JayF | is we could maybe cleanroom it partially with the minimal element talked about on the list | 22:39 |
| jroll | JayF: yeah, I was thinking about that | 22:39 |
| jroll | do it from scratch for ultimate tiny | 22:39 |
| jroll | JayF: TheJulia: thanks for this, good stuff | 22:41 |
| *** Nisha_Agarwal has joined #openstack-ironic | 22:45 | |
| *** jcoufal has quit IRC | 22:56 | |
| *** pmannidi has joined #openstack-ironic | 23:00 | |
| JayF | the devstack stable/newton fix for grenade is in the gate | 23:04 |
| jroll | awesome, thanks | 23:04 |
| * jroll out for the night, have a good one | 23:04 | |
| jlvillal | jroll: left a review for: https://review.openstack.org/#/c/417202/ | 23:04 |
| patchbot_ | patch 417202 - openstack-infra/project-config - Add experimental python3 dsvm job for ironic | 23:04 |
| jroll | thanks | 23:04 |
| NobodyCam | night jroll | 23:05 |
| jroll | night NobodyCam :) | 23:06 |
| NobodyCam | :) | 23:06 |
| jlvillal | Woot, we got the +A on the devstack stable/newton patch to hopefully fix our gate :) | 23:07 |
| jlvillal | JayF: ^^ | 23:07 |
| JayF | 23:04:03 <JayF> the devstack stable/newton fix for grenade is in the gate | 23:07 |
| JayF | you're three minutes behind ;0 | 23:07 |
| JayF | ;) | 23:08 |
| *** Sukhdev has quit IRC | 23:22 | |
| *** absubram has quit IRC | 23:28 | |
| *** lindycoder has quit IRC | 23:31 | |
| *** Linkmark has quit IRC | 23:33 | |
| *** yuikotakadamori has quit IRC | 23:44 | |
| *** Sukhdev has joined #openstack-ironic | 23:53 | |
| *** baoli has joined #openstack-ironic | 23:56 | |
| openstackgerrit | Annie Lezil proposed openstack/ironic-python-agent: [WIP]Reboot and Poweroff fails with coreos IPA image https://review.openstack.org/412594 | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
