| rpioso | TheJulia: Simply renaming the remote fixes it -- git remote rename upstream origin | 00:00 |
|---|---|---|
| rpioso | TheJulia: Yuck! It took me quite some time to narrow it down to that. I thought I still had issues with my dev environment. | 00:01 |
| TheJulia | I have no idea what that could possibly be | 00:01 |
| rpioso | TheJulia: Same here. If you wouldn't mind trying it yourself to see if you get the same result, that would be cool. If not, I would entirely understand. I'd be happy to ask someone else to do that. | 00:03 |
| * rpioso realizes it's time to call it a day | 00:10 | |
| *** rpioso is now known as rpioso|afk | 00:10 | |
| *** cjloader has joined #openstack-ironic | 00:12 | |
| *** cjloader has quit IRC | 00:16 | |
| TheJulia | rpioso|afk: I'm long out of braincells for today, glad to try in the morning | 00:18 |
| *** phuongnh has joined #openstack-ironic | 00:34 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Revert "Switch ironic-tempest-dsvm-ipa-partition-redfish-tinyipa" https://review.openstack.org/582432 | 00:35 |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Revert "Switch ironic-dsvm-standalone to zuulv3 native" https://review.openstack.org/582433 | 00:35 |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Revert "Switch ironic-dsvm-standalone to zuulv3 native" https://review.openstack.org/582433 | 00:36 |
| *** hshiina has joined #openstack-ironic | 00:42 | |
| *** gyee has quit IRC | 00:49 | |
| *** harlowja has quit IRC | 01:05 | |
| *** cjloader has joined #openstack-ironic | 01:13 | |
| *** cjloader has quit IRC | 01:17 | |
| *** tiendc has joined #openstack-ironic | 01:19 | |
| *** cjloader has joined #openstack-ironic | 01:37 | |
| *** cjloader has quit IRC | 01:50 | |
| *** rloo has quit IRC | 02:02 | |
| *** athomas_ has joined #openstack-ironic | 03:21 | |
| *** athomas has quit IRC | 03:23 | |
| *** TxGirlGeek has joined #openstack-ironic | 03:27 | |
| *** boris_42_ has quit IRC | 03:28 | |
| *** jaganathan has joined #openstack-ironic | 03:56 | |
| *** jaganathan has quit IRC | 03:58 | |
| *** jaganathan has joined #openstack-ironic | 03:58 | |
| *** links has joined #openstack-ironic | 04:02 | |
| openstackgerrit | Dao Cong Tien proposed openstack/ironic-tempest-plugin master: Rescue mode tempest tests https://review.openstack.org/528699 | 04:12 |
| *** harlowja has joined #openstack-ironic | 04:25 | |
| *** vmud213 has joined #openstack-ironic | 04:36 | |
| *** harlowja has quit IRC | 04:44 | |
| *** TxGirlGeek has quit IRC | 05:03 | |
| *** e0ne has joined #openstack-ironic | 05:10 | |
| *** ianychoi_ has joined #openstack-ironic | 05:27 | |
| *** ianychoi has quit IRC | 05:30 | |
| *** e0ne has quit IRC | 05:50 | |
| *** mjura has joined #openstack-ironic | 05:53 | |
| *** e0ne has joined #openstack-ironic | 05:58 | |
| *** e0ne has quit IRC | 05:59 | |
| *** e0ne has joined #openstack-ironic | 06:01 | |
| *** e0ne has quit IRC | 06:06 | |
| *** e0ne has joined #openstack-ironic | 06:06 | |
| *** e0ne has quit IRC | 06:09 | |
| *** e0ne has joined #openstack-ironic | 06:17 | |
| *** e0ne_ has joined #openstack-ironic | 06:21 | |
| *** e0ne has quit IRC | 06:22 | |
| *** e0ne has joined #openstack-ironic | 06:22 | |
| *** e0ne_ has quit IRC | 06:26 | |
| *** tiendc has quit IRC | 06:28 | |
| *** tiendc has joined #openstack-ironic | 06:28 | |
| *** phuongnh has quit IRC | 06:28 | |
| *** phuongnh has joined #openstack-ironic | 06:29 | |
| *** tssurya has joined #openstack-ironic | 06:29 | |
| *** e0ne_ has joined #openstack-ironic | 06:33 | |
| *** e0ne_ has quit IRC | 06:33 | |
| *** e0ne has quit IRC | 06:35 | |
| *** adreznec has quit IRC | 06:37 | |
| *** e0ne has joined #openstack-ironic | 06:40 | |
| *** zhangfei has joined #openstack-ironic | 06:41 | |
| *** e0ne has quit IRC | 06:48 | |
| *** lennyb has quit IRC | 06:49 | |
| *** logan- has quit IRC | 06:50 | |
| *** logan- has joined #openstack-ironic | 06:50 | |
| *** serlex has joined #openstack-ironic | 06:51 | |
| *** lennyb has joined #openstack-ironic | 06:52 | |
| *** tssurya has quit IRC | 06:58 | |
| *** tssurya has joined #openstack-ironic | 07:00 | |
| *** adreznec has joined #openstack-ironic | 07:00 | |
| *** tesseract has joined #openstack-ironic | 07:12 | |
| *** masber has joined #openstack-ironic | 07:17 | |
| *** MrBerns has joined #openstack-ironic | 07:21 | |
| *** tiendc_ has joined #openstack-ironic | 07:28 | |
| *** tiendc has quit IRC | 07:30 | |
| openstackgerrit | Madhuri Kumari proposed openstack/ironic master: Implement NoGraphicalConsole Interface https://review.openstack.org/547356 | 07:31 |
| *** phuongnh has quit IRC | 07:33 | |
| *** phuongnh has joined #openstack-ironic | 07:33 | |
| *** zshi has quit IRC | 07:33 | |
| openstackgerrit | Madhuri Kumari proposed openstack/ironic master: Implement basic interfaces for GraphicalConsole Interface https://review.openstack.org/547356 | 07:42 |
| openstackgerrit | Madhuri Kumari proposed openstack/ironic master: Implement basic interfaces for GraphicalConsole Interface https://review.openstack.org/547356 | 07:46 |
| *** hshiina has quit IRC | 07:51 | |
| *** mgoddard has joined #openstack-ironic | 07:54 | |
| *** racedo has joined #openstack-ironic | 07:56 | |
| *** mgoddard has quit IRC | 08:03 | |
| *** e0ne has joined #openstack-ironic | 08:07 | |
| *** zshi has joined #openstack-ironic | 08:09 | |
| *** ajya has joined #openstack-ironic | 08:11 | |
| *** cjloader has joined #openstack-ironic | 08:12 | |
| *** hwoarang has quit IRC | 08:15 | |
| *** cjloader has quit IRC | 08:17 | |
| *** hwoarang has joined #openstack-ironic | 08:23 | |
| *** hwoarang has quit IRC | 08:23 | |
| *** hwoarang has joined #openstack-ironic | 08:23 | |
| *** alexpilotti has quit IRC | 08:26 | |
| *** alexpilotti has joined #openstack-ironic | 08:26 | |
| *** vabada has quit IRC | 08:27 | |
| *** vabada has joined #openstack-ironic | 08:27 | |
| *** vabada has quit IRC | 08:29 | |
| *** vabada has joined #openstack-ironic | 08:30 | |
| openstackgerrit | Madhuri Kumari proposed openstack/ironic master: Implement basic interfaces for GraphicalConsole Interface https://review.openstack.org/547356 | 08:33 |
| *** phuongnh has quit IRC | 08:36 | |
| *** phuongnh has joined #openstack-ironic | 08:37 | |
| *** rcernin has quit IRC | 08:46 | |
| *** milan has joined #openstack-ironic | 08:49 | |
| *** hwoarang has quit IRC | 08:54 | |
| *** hwoarang has joined #openstack-ironic | 08:55 | |
| *** trungnv has quit IRC | 09:01 | |
| *** mgoddard has joined #openstack-ironic | 09:19 | |
| *** pbourke has quit IRC | 09:26 | |
| *** mgoddard has quit IRC | 09:26 | |
| *** pbourke has joined #openstack-ironic | 09:28 | |
| *** dougsz has joined #openstack-ironic | 09:36 | |
| *** MrBerns has quit IRC | 09:58 | |
| *** tiendc_ has quit IRC | 10:37 | |
| *** phuongnh has quit IRC | 10:50 | |
| *** mgoddard has joined #openstack-ironic | 10:57 | |
| *** cjloader has joined #openstack-ironic | 11:12 | |
| *** arxcruz is now known as arxcruz|brb | 11:13 | |
| *** cjloader has quit IRC | 11:17 | |
| *** toabctl has joined #openstack-ironic | 11:18 | |
| *** mgoddard has quit IRC | 11:21 | |
| *** jcoufal has joined #openstack-ironic | 11:30 | |
| *** rh-jelabarre has joined #openstack-ironic | 11:42 | |
| *** MrBerns has joined #openstack-ironic | 11:43 | |
| *** bfournie has quit IRC | 11:44 | |
| MrBerns | Hi guys, Im getting the following error when running the enrollment playbook for a node: http://paste.openstack.org/show/725814/ | 11:48 |
| MrBerns | Has anyone encountered this before? I am using bifrost with keystone deployed | 11:48 |
| openstackgerrit | tianhui proposed openstack/ironic master: Fix bug to doc:configdrive https://review.openstack.org/582514 | 11:52 |
| TheJulia | Good morning everyone | 11:54 |
| TheJulia | MrBerns: no... sounds like your install has some incompatabilities. could you supply `pip freeze` output ? | 11:54 |
| * TheJulia goes and gets coffee and breakfast | 11:56 | |
| TheJulia | mkrai: thanks for the new graphical patches! | 11:56 |
| *** jaganathan has quit IRC | 11:56 | |
| MrBerns | TheJulia, http://paste.openstack.org/show/725816/ | 11:57 |
| *** MattMan has quit IRC | 12:05 | |
| *** MattMan has joined #openstack-ironic | 12:06 | |
| *** bfournie has joined #openstack-ironic | 12:22 | |
| *** skazi has joined #openstack-ironic | 12:23 | |
| jroll | morning y'all | 12:31 |
| *** dprince has joined #openstack-ironic | 12:33 | |
| *** jaypipes is now known as leakypipes | 12:34 | |
| TheJulia | good morning! | 12:35 |
| TheJulia | MrBerns: I would try downgrading keystoneauth1 to something like 3.6.2, although that is just a hunch :\ Credentials are also free-form if memory serves, and I think they removed some parameters in keystoneauth1 | 12:38 |
| MrBerns | Thanks TheJulia, Ill give that a go | 12:39 |
| *** leakypipes is now known as smokypipes | 12:42 | |
| MrBerns | I just tried that, but unfortunately that doesn't seem to work | 12:42 |
| MrBerns | I had a similar problem with the keystoneless install, which complained about 'token' | 12:42 |
| MrBerns | I had to modify loader.py to fix it so Ill try againthere | 12:43 |
| *** hoonetorg has quit IRC | 12:43 | |
| *** trown|outtypewww is now known as trown | 12:47 | |
| *** jcoufal has quit IRC | 12:48 | |
| TheJulia | MrBerns: :( | 12:52 |
| TheJulia | I feel like ironic ci is psychic and knows when I want to merge things that random failures should be super prevelent | 12:52 |
| jroll | the robots are revolting | 12:53 |
| smokypipes | TheJulia, jroll: ok, just a Friday curiosity question. How does Ironic boot-from-volume work, in a couple sentences? does it copy the bootable segment of the volume into memory on the target node and lay the rest of the volume down to local disk? or is there some magic somewhere? | 12:57 |
| *** smokypipes is now known as leakypipes | 12:58 | |
| *** TheJulia is now known as needssleep | 12:58 | |
| *** vmud213 has quit IRC | 12:58 | |
| needssleep | leakypipes: The bootloader (or hardware) reads from the remote volume and inits what is there | 12:58 |
| needssleep | leakypipes: and then continues to use that volume using a software or hardware initiator | 12:59 |
| needssleep | leakypipes: local disk ends up being "ephemeral" :) | 12:59 |
| leakypipes | needssleep: interesting. does the bootloader just assume a particular address in the volume's bytestream for where the boot instructions lie? | 12:59 |
| *** links has quit IRC | 12:59 | |
| leakypipes | needssleep: I'm trying to relate to how a "normal" bootloader would look for the boot instructions at a particular address in firmware | 13:00 |
| leakypipes | firmware/ROM | 13:00 |
| needssleep | leakypipes: ipxe maps the device to the memory address for the endpoint, so for block 0 boot loader booting, it reads it in, the ipxe uefi loader should look for the partition and go from there but I've not tried that. Hardware based should just view it as another disk | 13:01 |
| *** hoonetorg has joined #openstack-ironic | 13:01 | |
| leakypipes | k | 13:01 |
| leakypipes | needssleep: thank you ma'am! :) | 13:02 |
| *** milan has quit IRC | 13:04 | |
| *** milan has joined #openstack-ironic | 13:04 | |
| openstackgerrit | Marc Methot proposed openstack/ironic master: Fixed link to Storyboard instead of launchpad https://review.openstack.org/582534 | 13:07 |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Assert a build timeout for zuul templated CI jobs https://review.openstack.org/582535 | 13:09 |
| *** rloo has joined #openstack-ironic | 13:10 | |
| needssleep | leakypipes: enjoy! | 13:11 |
| *** rloo is now known as waterloo | 13:14 | |
| needssleep | good morning waterloo! | 13:17 |
| waterloo | good morning needssleep! Happy Friday :) | 13:17 |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Add notes for future job migrations https://review.openstack.org/582537 | 13:17 |
| needssleep | waterloo: https://review.openstack.org/#/c/582535/ should pass and allow us to start landing code again. :\ | 13:18 |
| patchbot | patch 582535 - ironic - Assert a build timeout for zuul templated CI jobs | 13:18 |
| *** bnemec has joined #openstack-ironic | 13:18 | |
| waterloo | needssleep: oh geez, thx for that! | 13:19 |
| jroll | oh my | 13:19 |
| jroll | +2 | 13:19 |
| jroll | morning waterloo :) | 13:19 |
| waterloo | good morning jroll! how goes conductor groups? (I haven't yet looked to see) | 13:20 |
| jroll | waterloo: I didn't get to it yesterday - were you going to move the release_mappings thing between patches, or should I go ahead and rebase now? | 13:21 |
| waterloo | jroll: oh, needssleep was fine with it being done in the next patch in the series. | 13:21 |
| jroll | okey | 13:21 |
| waterloo | jroll: let me know if you want to discuss any of my comments | 13:21 |
| *** edleafe is now known as figleaf | 13:21 | |
| waterloo | jroll: we may want to change something... | 13:21 |
| jroll | waterloo: oh! I just saw that you commented on the later ones :) | 13:22 |
| * jroll reads | 13:22 | |
| *** baha has joined #openstack-ironic | 13:22 | |
| waterloo | jroll: i think i am caught up; reviewed them all except unit tests | 13:22 |
| jroll | cool, thank you | 13:22 |
| waterloo | needssleep: I don't understand https://review.openstack.org/#/c/582537/1/zuul.d/legacy-ironic-jobs.yaml (but i also don't remember how all this works) | 13:26 |
| patchbot | patch 582537 - ironic - Add notes for future job migrations | 13:26 |
| waterloo | needssleep: is the NOTE for something to do in the migrated job, when it gets migrated? | 13:26 |
| needssleep | waterloo: that | 13:26 |
| needssleep | just a note so we don't forget those parameters when we migrate them | 13:26 |
| waterloo | needssleep: ok, i'll suggest something else, thx :) | 13:27 |
| needssleep | k | 13:28 |
| jroll | waterloo: so re building the hash ring key without a ':', there's still room for conflicts. for example group "" with hw type "ipmi" makes "ipmi". so does group "ip" with hw type "mi". not sure how many assumptions we want to make about people not shooting themselves in the foot | 13:29 |
| waterloo | jroll: it is ok if we don't do that. ... what about the issue wrt rolling upgrades? | 13:30 |
| jroll | re "I'm good with assuming any conductor, and not making a requirement to have a conductor in the '' conductor group. Just make sure it is random :)" that might be doable, just need some extra code... I'd rather just require some conductors in the default group I think, but will look at what code it takes | 13:30 |
| jroll | right | 13:30 |
| jroll | so I didn't think through the "everything is upgraded and we're rolling through the unpin" | 13:30 |
| jroll | that is definitely a problem :/ | 13:30 |
| jroll | (until you mentioned it that is) | 13:31 |
| waterloo | with the topic needed for drivers. for now, you can assume that there will be a '' conductor-group, but i think we shouldn't assume that there is one or impose it. | 13:31 |
| waterloo | ha ha, it is possible that if i didn't mention it, no one would have noticed when using. but it is also possible, that they would :) | 13:31 |
| waterloo | hence my suggestion to get rid of the ':' so it would all upgrade seamlessly. I think we're talking about non-tree driver types having possible issues? sec, let me see what hw types we have now... | 13:32 |
| jroll | right, it would be out of tree drivers | 13:33 |
| needssleep | I thought we talked about starting without a delimiter and introducing it later once we figured out how we were going to do antiaffinity | 13:33 |
| jroll | we did talk about that, but possible conflicts came up in the spec :/ | 13:33 |
| jroll | it's somewhat improbable that it will happen, but possible | 13:34 |
| waterloo | well, it was my bright idea that caused the conflict, ':'. but i was thinking of classic drivers, which we no longer have to deal with. | 13:34 |
| jroll | then again, ':' is a valid character in entrypoints. we just think it's more improbable to have it there :) | 13:34 |
| * needssleep feels like we will need a small warning in the docs about group naming | 13:35 | |
| jroll | waterloo: your case-sensitivity comments make me realize I also am not validating [a-z0-9] either | 13:35 |
| waterloo | there is a slight possibility with only hwtypes, but that would only be with out-of-tree hw types, AND it would have to be out-of-tree hwtypes WITH some conductor_group value... | 13:35 |
| jroll | work to dooooo | 13:35 |
| jroll | right | 13:35 |
| waterloo | i think if we are good with the discrepancy/out-of-sync during rolling upgrades, i am good with having the ':'. If we are not good with the time that hash_ring will be out-of-sync, then we need to address that. | 13:36 |
| waterloo | during current rolling upgrades, the hash table is out of sync anyway, cuz of old conductors -> new conductors etc. | 13:37 |
| *** med_ has quit IRC | 13:38 | |
| needssleep | Yeah, it would be rolling, the conundrum is API understanding | 13:38 |
| needssleep | since it would be addressing the messages | 13:38 |
| needssleep | so really it is a hard break I think | 13:38 |
| jroll | yeah | 13:39 |
| needssleep | no delimiter would be non-breaking topic selection wise | 13:40 |
| waterloo | needssleep: exactly, which is why i am thinking it might be 'ok' to have no delimiter. | 13:40 |
| jroll | right, it would be worth it to expose the small footgun | 13:40 |
| waterloo | needssleep: although i guess i need to think of future use of this thing and whether the delimiter would be useful. | 13:40 |
| needssleep | jroll: doc notes, and we can de-footgun it next cycle | 13:41 |
| jroll | how do we de-footgun? | 13:41 |
| jroll | that's the issue, adding the delimiter later has the same issue | 13:41 |
| needssleep | plan for the later upgrade | 13:41 |
| waterloo | jroll: can we hash-ring both keys? | 13:41 |
| jroll | waterloo: like two layers of dict keys? | 13:42 |
| waterloo | jroll: what happens if we hash-ring key-with-no-group, key-with-group? | 13:42 |
| jroll | hmm | 13:42 |
| waterloo | no not even two layers. just double the keys. | 13:42 |
| waterloo | the same conductors will be avail in both keys, right? | 13:42 |
| jroll | right, two sets of hash rings | 13:42 |
| jroll | yes | 13:42 |
| waterloo | then we're good. | 13:42 |
| jroll | just trying to think through how we decide which to use | 13:42 |
| waterloo | just do that for this one release...? | 13:42 |
| jroll | or do we only use the one with no delimiter now, then cut over next cycle? | 13:43 |
| waterloo | i suspect we can use your existing code as is. | 13:43 |
| waterloo | thinking... | 13:43 |
| needssleep | also thinking | 13:43 |
| jroll | my mind is wandering to a 2.0 rpc version as we'll need to kill the old ones | 13:44 |
| * jroll puts that aside for a moment | 13:44 | |
| waterloo | i think if we add both keys to the hashring. so eg conductorA handles ipmi. we have keys ':ipmi' and 'ipmi'. the code, based in rpc API version, uses 'ipmi' or ':ipmi'. as long as both have the same values, we're good. | 13:46 |
| waterloo | at some point, we delete the code that adds the 'ipmi' key. i guess the following release cycle. | 13:46 |
| waterloo | jroll: wait. why do we need to kill the old ones? | 13:47 |
| *** mgoddard has joined #openstack-ironic | 13:48 | |
| jroll | I think that will work. trying to think through the removal | 13:48 |
| jroll | waterloo: I need to re-read the pinning code, but it seems to me that someone could pin stein code to queens or something and it would still be using the queens calculation | 13:49 |
| waterloo | after code is using the new keys, it doesn't matter if the old keys stay in the hashring, we won't use it. | 13:49 |
| jroll | (but, we also get the benefit of cleaning up all the RPC backporting) | 13:49 |
| *** dougsz has quit IRC | 13:50 | |
| waterloo | we're at queens now. going to release rocky. when someone does a rolling upgrade from queens to rocky, they'll pin to queens. | 13:50 |
| waterloo | i think that's what we're talking about now, right? that rolling upgrade? | 13:51 |
| jroll | this is somewhat a side topic | 13:51 |
| waterloo | oh wait, you said 'stein' code to queens. | 13:51 |
| waterloo | honestly, i have no idea what happens if you have stein code and pin to queens. | 13:51 |
| waterloo | although -- you can't. | 13:51 |
| jroll | ah, we remove them immediately | 13:51 |
| waterloo | if we're talking about a released 'stein', 'queens' will not be a choice for pinning. | 13:52 |
| jroll | yeah, like I said I need to re-read the code :) | 13:52 |
| jroll | needed to* | 13:52 |
| waterloo | jroll: yes, please re-read, in case i got something wrong. but i think the two keys will work :) | 13:53 |
| jroll | right, the two keys thing is separate from the major rpc version thing | 13:53 |
| waterloo | jroll: right. | 13:53 |
| * jroll has quite a bit of work to do | 13:56 | |
| needssleep | so, we could just fallback if we can't find the topic in a future version, but if we add the logic now and switch later then that will work | 13:57 |
| needssleep | https://github.com/openstack/ironic/blob/a38edcebfa09b0c00c1bf3be6189315abf67bfd5/ironic/conductor/rpcapi.py#L135 | 13:57 |
| *** dougsz has joined #openstack-ironic | 14:00 | |
| *** dprince has quit IRC | 14:00 | |
| *** jcoufal has joined #openstack-ironic | 14:08 | |
| *** efried is now known as fried_rice | 14:11 | |
| needssleep | hmm, we need to double check our excludes for migrated jobs as well | 14:13 |
| *** dprince has joined #openstack-ironic | 14:16 | |
| *** mbeierl has quit IRC | 14:17 | |
| *** dprince has quit IRC | 14:21 | |
| *** dprince has joined #openstack-ironic | 14:21 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Add notes for future job migrations https://review.openstack.org/582537 | 14:23 |
| needssleep | For rocky for networking-generic-switch, https://review.openstack.org/#/c/580904/ could use a quick review | 14:25 |
| patchbot | patch 580904 - networking-generic-switch - Updating required neutron version | 14:25 |
| *** cjloader has joined #openstack-ironic | 14:26 | |
| *** cjloader has quit IRC | 14:26 | |
| *** cjloader has joined #openstack-ironic | 14:27 | |
| *** bnemec has quit IRC | 14:29 | |
| *** bnemec has joined #openstack-ironic | 14:41 | |
| *** lbragstad is now known as elbragstad | 14:43 | |
| openstackgerrit | Julia Kreger proposed openstack/networking-baremetal master: Updating required neutron version https://review.openstack.org/580902 | 14:47 |
| *** skazi has quit IRC | 14:48 | |
| *** cjloader_ has joined #openstack-ironic | 14:52 | |
| *** cjloader has quit IRC | 14:52 | |
| *** cjloader_ has quit IRC | 14:54 | |
| *** cjloader has joined #openstack-ironic | 14:54 | |
| *** cjloader has quit IRC | 14:55 | |
| *** cjloader has joined #openstack-ironic | 14:55 | |
| needssleep | https://review.openstack.org/#/c/580902/ should be another quick review, hopefully it will fix the jobs for networking-baremetal | 14:58 |
| patchbot | patch 580902 - networking-baremetal - Updating required neutron version | 14:58 |
| *** TxGirlGeek has joined #openstack-ironic | 15:00 | |
| needssleep | yet another easy review https://review.openstack.org/#/c/581381/ | 15:00 |
| patchbot | patch 581381 - ironic-inspector - Sync up Flask version with global-requirements | 15:00 |
| *** TxGirlGeek has quit IRC | 15:04 | |
| *** mjura has quit IRC | 15:08 | |
| *** rpittau has quit IRC | 15:12 | |
| *** bnemec has quit IRC | 15:15 | |
| *** zhangfei has quit IRC | 15:15 | |
| *** yolanda__ has joined #openstack-ironic | 15:16 | |
| *** mgoddard has quit IRC | 15:16 | |
| *** bdodd has quit IRC | 15:17 | |
| *** bdodd has joined #openstack-ironic | 15:18 | |
| *** EmilienM is now known as EvilienM | 15:18 | |
| *** yolanda_ has quit IRC | 15:18 | |
| *** bnemec has joined #openstack-ironic | 15:19 | |
| *** serlex has quit IRC | 15:19 | |
| *** tesseract has quit IRC | 15:21 | |
| jroll | hm, needssleep makes a good point with the fallback | 15:24 |
| needssleep | seems like the logical place to catch for now, at least at a high-ish level | 15:24 |
| * jroll plays with it | 15:24 | |
| MrBerns | needssleep, if you have a moment, it seems to me that when I call the enrollment playbook, ansible thinks I am still not using another auth type, when I am in fact using keystone | 15:25 |
| MrBerns | I added some debug tasks to the ironic-enroll-dynamic role | 15:25 |
| *** rpioso|afk is now known as rpioso | 15:25 | |
| rpioso | Good morning | 15:25 |
| *** arxcruz|brb is now known as arxcruz | 15:26 | |
| MrBerns | http://paste.openstack.org/show/725835/ | 15:26 |
| MrBerns | Am I on the right track here or does that look normal? | 15:26 |
| needssleep | jroll: disclaimer: it has been a long time since I've walked through that code completely in terms of hash ring interaction. The ring being one of those things we never really change | 15:28 |
| jroll | needssleep: yeah, I think that goes for all of us :) | 15:28 |
| *** zhangfei has joined #openstack-ironic | 15:28 | |
| needssleep | MrBerns: have you tried updating shade? | 15:28 |
| *** zhangfei has quit IRC | 15:28 | |
| mordred | I didn't do it | 15:28 |
| jroll | narrator: he did. | 15:29 |
| needssleep | lol | 15:29 |
| needssleep | I've not looked at the module in a while, but I suspect we're doing something wrong now with that or the template with config parameters we're writing out that gets picked up for authentication | 15:29 |
| * needssleep seems to remember having to change user_domain_name in ironic's code a couple months ago | 15:30 | |
| MrBerns | shade is at 1.21.1 which seems to be the latest | 15:31 |
| needssleep | hmmmmm | 15:31 |
| needssleep | 1.25 I thought | 15:31 |
| needssleep | 1.28 | 15:31 |
| needssleep | cores: quick pass on python-ironicclient would be appreciated, there are a couple patches sitting out there | 15:32 |
| needssleep | the clouds must be slow today... nearly 1600 seconds to run the multinode tests :\ | 15:33 |
| NobodyCam | Good Morning Ironic'ers ... | 15:35 |
| NobodyCam | and .. | 15:35 |
| NobodyCam | OFC . | 15:35 |
| NobodyCam | TGIF! | 15:35 |
| needssleep | Good morning NobodyCam | 15:37 |
| NobodyCam | Good Morning needssleep ... | 15:38 |
| needssleep | ugh, tree is still on the ground back home | 15:39 |
| NobodyCam | LOL :( | 15:42 |
| needssleep | MrBerns: up for sharing your install config parameters? I'll see if I can reproduce it and pin down the issue on my end | 15:43 |
| MrBerns | needssleep, Yeah can do, what exactly do you need from my end, just the command I used to install bifrost? | 15:45 |
| needssleep | MrBerns: yeah, that would help... and if you set any of the flags by default in one of the settings file | 15:49 |
| needssleep | files | 15:49 |
| needssleep | I'm going to run to the store to get some sandwiches, bbiab | 15:49 |
| MrBerns | needssleep, here are the files from playbooks/inventory/group_vars: http://paste.openstack.org/show/Mq0Q4boCEoRVQPPuUoqq/ | 15:53 |
| *** tssurya has quit IRC | 15:53 | |
| MrBerns | needssleep, and the command I used is: ansible-playbook -vvvv -i inventory/target install.yaml -e enable_keystone=true -e noauth_mode=false | 15:53 |
| *** e0ne has quit IRC | 15:54 | |
| MrBerns | needssleep, Im off to the gym, but let me know if you need me to provide anything else to aid in the investigation | 15:54 |
| MrBerns | needssleep, and thank you :) | 15:54 |
| bfournie | hi, doing a simple backport to stable/queens in python-ironic-inspector-client - https://review.openstack.org/#/c/581875/, I get pep8 failures in a file I didn't change - ironic_inspector_client/test/functional.py:33:1: E402 module level import not at top of file | 15:59 |
| patchbot | patch 581875 - python-ironic-inspector-client (stable/queens) - Provide proper error message if interface name is ... | 15:59 |
| bfournie | is there something different with python-ironic-inspector-client in stable/queens? | 16:00 |
| *** racedo has quit IRC | 16:06 | |
| *** jcoufal has quit IRC | 16:06 | |
| *** mjturek has joined #openstack-ironic | 16:07 | |
| *** harlowja has joined #openstack-ironic | 16:08 | |
| openstackgerrit | Ruby Loo proposed openstack/networking-baremetal master: Updating required neutron version https://review.openstack.org/580902 | 16:09 |
| *** mbeierl has joined #openstack-ironic | 16:10 | |
| *** harlowja has quit IRC | 16:10 | |
| *** jcoufal has joined #openstack-ironic | 16:15 | |
| *** racedo has joined #openstack-ironic | 16:16 | |
| *** racedo has quit IRC | 16:17 | |
| needssleep | bfournie: oh, we likely didn't backport a hacking pin | 16:17 |
| *** fried_rice is now known as fried_rolls | 16:18 | |
| needssleep | fried_rolls: spring, egg, or other? | 16:18 |
| needssleep | stable cores: https://review.openstack.org/#/c/567567/ to fix bfournie's issue above | 16:20 |
| patchbot | patch 567567 - python-ironic-inspector-client (stable/queens) - Gate fix: Cap hacking to avoid gate failure | 16:20 |
| *** baha has quit IRC | 16:20 | |
| bfournie | thanks needssleep | 16:23 |
| *** med_ has joined #openstack-ironic | 16:24 | |
| *** med_ has quit IRC | 16:24 | |
| *** med_ has joined #openstack-ironic | 16:24 | |
| openstackgerrit | Merged openstack/ironic-python-agent master: Remove testrepository https://review.openstack.org/581685 | 16:25 |
| openstackgerrit | Merged openstack/python-ironicclient master: Trivial fix typo of description https://review.openstack.org/581191 | 16:25 |
| openstackgerrit | Merged openstack/python-ironicclient master: Remove testrepository https://review.openstack.org/581666 | 16:25 |
| needssleep | \o/ merging | 16:27 |
| needssleep | another two for ironic | 16:28 |
| *** cjloader has quit IRC | 16:32 | |
| *** cjloader has joined #openstack-ironic | 16:37 | |
| *** bnemec has quit IRC | 16:37 | |
| *** athomas_ has quit IRC | 16:39 | |
| *** mjturek has quit IRC | 16:41 | |
| *** baha has joined #openstack-ironic | 16:42 | |
| *** bnemec has joined #openstack-ironic | 16:45 | |
| *** markguz has joined #openstack-ironic | 16:47 | |
| *** markguz has quit IRC | 16:51 | |
| *** bnemec has quit IRC | 16:53 | |
| openstackgerrit | Merged openstack/networking-generic-switch master: Updating required neutron version https://review.openstack.org/580904 | 16:55 |
| needssleep | yay | 16:56 |
| *** jcoufal has quit IRC | 17:09 | |
| *** med_ has quit IRC | 17:15 | |
| *** milan has quit IRC | 17:17 | |
| *** dougsz has quit IRC | 17:25 | |
| *** bnemec has joined #openstack-ironic | 17:38 | |
| needssleep | MrBerns: So... It worked for me on fedora... I did have to fix two minor things but http://paste.openstack.org/show/725842/ is my pip freeze output | 17:40 |
| openstackgerrit | Julia Kreger proposed openstack/bifrost master: Make sure we have our folders for uwsgi https://review.openstack.org/582653 | 17:44 |
| *** bnemec has quit IRC | 17:47 | |
| openstackgerrit | Merged openstack/ironic master: Assert a build timeout for zuul templated CI jobs https://review.openstack.org/582535 | 17:49 |
| *** mgoddard has joined #openstack-ironic | 17:57 | |
| *** jaganathan has joined #openstack-ironic | 18:00 | |
| openstackgerrit | Julia Kreger proposed openstack/bifrost master: Remove syslinux usage https://review.openstack.org/582655 | 18:03 |
| *** jaganathan has quit IRC | 18:03 | |
| *** trown is now known as trown|lunch | 18:06 | |
| jroll | hmm | 18:10 |
| jroll | if we assume that conductor_group isn't set on any conductors, nor set on any nodes | 18:10 |
| jroll | do we even need to think about rolling upgrades? | 18:10 |
| jroll | the hash ring is local to the process using it | 18:11 |
| *** jistr has quit IRC | 18:11 | |
| *** mgoddard_ has joined #openstack-ironic | 18:11 | |
| jroll | it doesn't need to match the other end | 18:11 |
| *** jistr has joined #openstack-ironic | 18:11 | |
| *** mgoddard has quit IRC | 18:14 | |
| *** mgoddard has joined #openstack-ironic | 18:15 | |
| *** fried_rolls is now known as fried_rice | 18:15 | |
| fried_rice | needssleep: In BJJ, we call sparring "rolling". Not actually a food thing at all :) | 18:15 |
| needssleep | jroll: that is an excellent point | 18:16 |
| jroll | ok, chances I am crazy have dropped | 18:16 |
| jroll | waterloo: thoughts? | 18:16 |
| needssleep | as long as we don't have a delimiter, no, but if we want to we have to plan ahead one cycle if we do want to add a delimiter | 18:16 |
| needssleep | jroll: We work on ironic, of course we are crazy | 18:17 |
| jroll | needssleep: nah, even with a delimiter, I can't think of a case where it breaks | 18:17 |
| waterloo | the hash_ring 'instance' is local to each process (conductor & API) BUT they need to be in sync (read, same) | 18:17 |
| waterloo | if i understand. | 18:17 |
| jroll | I don't believe they do | 18:18 |
| jroll | the hosts in the ring need to be up to date | 18:18 |
| *** mgoddard_ has quit IRC | 18:18 | |
| jroll | but, the lookup results in a (list of) host(s), so as long as that host exists, the message can be sent | 18:18 |
| needssleep | in the form of a topic right? | 18:19 |
| waterloo | thinking.. is it where it rebalances where it is important to update (or cuz conductors disappear)? | 18:19 |
| jroll | the topic is the "hostname" of the conductor, but yes | 18:19 |
| jroll | a rebalance is needed when the list of active conductors changes | 18:20 |
| waterloo | do we randomize which conductor we use if there is more than one conductor that can handle a node? | 18:20 |
| waterloo | i've forgotten how this code works :-( | 18:21 |
| jroll | I believe so | 18:21 |
| jroll | er | 18:21 |
| jroll | there's only one conductor that can handle a node | 18:21 |
| waterloo | if you have 2 conductors that can handle ipmi, and a node has ipmi, then both of those conductors can handle the node, right? | 18:22 |
| jroll | or rather, there are N conductors that handle a node, where N is the 'replicas' argument here: https://github.com/openstack/tooz/blob/master/tooz/hashring.py#L114 | 18:22 |
| jroll | ah we do have replicas config https://github.com/openstack/ironic/blob/master/ironic/conductor/rpcapi.py#L138 | 18:23 |
| waterloo | unfortunate use of node in that tooz code. where node == our conductor. | 18:23 |
| jroll | yes, but (assuming one replica in that config), they are mapped onto exactly one conductor in the hash ring | 18:23 |
| waterloo | heh, that config is still experimental... | 18:24 |
| waterloo | and set to 1 | 18:24 |
| waterloo | so, at the end of the day, what we want is for a node to get the same conductor to handle it, during the entire deployment process, if it happens during a rolling upgrade. | 18:25 |
| jroll | in any case, it will be successfully mapped, as long as the calculation in the hash ring initialization and the calculation in the hash ring fetch, are the same | 18:25 |
| jroll | do we? | 18:26 |
| jroll | take_over should handle it, if it changes | 18:26 |
| waterloo | right, but we want to minimize take_over, no? | 18:26 |
| jroll | yes | 18:27 |
| jroll | but it's going to happen | 18:27 |
| waterloo | w/o a rolling upgrade, we sort of assume the same conductor will handle a node during deployment | 18:27 |
| jroll | just a question of if it happens at unpin time or service restart time | 18:27 |
| jroll | OH. damn. that's why conductors and APIs need to be in sync. sigh. | 18:27 |
| jroll | too much state. | 18:27 |
| waterloo | we should write it up but we should have written it up cuz now we don't have time | 18:28 |
| jroll | sorry for the interruption, but that was helpful in processing this. so thanks for that. :) | 18:28 |
| waterloo | yeah, so even w/o your change. right now. if we do a rolling upgrade, api & conductor hash rings will be out of sync cuz of restarts etc. | 18:28 |
| jroll | ew | 18:29 |
| jroll | I want to say you're wrong there, but you're right :/ | 18:29 |
| waterloo | i thought yesterday, that with your change, the interval got longer wrt out-of-sync. but if it isn't longer (if i'm wrong) or not sig. longer, then.. should be ok..? | 18:29 |
| jroll | that's a fair point | 18:30 |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Ensure we unbind flat network ports and clear BM mac addresses https://review.openstack.org/539981 | 18:30 |
| waterloo | it all depends on when the hashrings get reset in API and conductor, vs when conductors go up/down during upgrade. and API nodes. I *think* the SIGHUP minimizes the hashring sync issue when we unpin. | 18:30 |
| jroll | so maybe using the separator is fine | 18:30 |
| jroll | even if the HUP doesn't trigger it, there's two things that do: | 18:31 |
| jroll | 1) it's calculated in the API on every request | 18:31 |
| jroll | 2) it's reset periodically in the conductor, at a fairly small interval IIRC | 18:31 |
| waterloo | there's a config that specifies the reset interval. used in both api & conductor i thought. | 18:32 |
| jroll | API is always: https://github.com/openstack/ironic/blob/master/ironic/conductor/rpcapi.py#L129 | 18:32 |
| waterloo | oh. hash_ring_reset_interval=180 | 18:32 |
| jroll | ( see also https://github.com/openstack/ironic/commit/d7cc83129966109556df8669aa970ff39fd156ed ) | 18:32 |
| waterloo | ah, so YOU did it! :D | 18:33 |
| waterloo | ok, so the config is used by the conductor | 18:33 |
| jroll | :) | 18:33 |
| waterloo | a conductor (pinned) that comes up with new code, will have a reset hash_ring; when it is unpinned via SIGHUP, the hashring won't be reset. it'll only be reset via the config/reset thing. | 18:34 |
| jroll | also happy to make it reset on SIGHUP, btw | 18:34 |
| waterloo | ^ i think we (you) need to think about whether we need it to reset or not. | 18:35 |
| jroll | (if we have a hook) | 18:35 |
| jroll | I don't think it would be a bad thing | 18:35 |
| jroll | it isn't terribly heavy to reset it | 18:35 |
| waterloo | i am leaning towards not now. but i also am not thinking too hard about this problem. | 18:35 |
| needssleep | hmmm | 18:35 |
| waterloo | i believe it isn't hard to add hooks wrt sighup, etc, but i don't recall offhand. | 18:36 |
| jroll | oh, we don't really have a good hook there right now | 18:36 |
| jroll | probably not hard | 18:36 |
| needssleep | waterloo: so there is code predicated on a conductor owning the node for some tasks, but general things should work afaik | 18:36 |
| waterloo | needssleep: yeah, i think that's correct. | 18:37 |
| *** ajya has quit IRC | 18:37 | |
| waterloo | there's that node.conductor_affinity thing. | 18:37 |
| needssleep | yeah | 18:37 |
| openstackgerrit | Merged openstack/ironic-inspector master: Sync up Flask version with global-requirements https://review.openstack.org/581381 | 18:37 |
| waterloo | i think it is used, to know when to takeover by another conductor | 18:37 |
| needssleep | and is used for periodic tasks | 18:37 |
| * needssleep has been deep in that code recently | 18:39 | |
| waterloo | needssleep: ah yes, i see, _sync_local_state. this is where the hashring is reset too, i think. | 18:39 |
| waterloo | ah, i take that back. a comment led me to believe the ring is reset here but i don't see that it is. | 18:40 |
| MrBerns | Thanks needssleep, Ill check my packages again | 18:41 |
| jroll | I don't see any hash ring resets in the conductor O_o | 18:41 |
| jroll | that isn't good | 18:41 |
| jroll | oh, it does it on fetch, ignore me | 18:41 |
| needssleep | heh | 18:41 |
| * needssleep feels like it is beer-o-clock | 18:42 | |
| jroll | ( https://github.com/openstack/ironic/blob/master/ironic/common/hash_ring.py#L40 for the people in the back) | 18:42 |
| waterloo | hashringmanager is so smart and independent :) | 18:42 |
| jroll | maybe I just need to play with running this code | 18:44 |
| jroll | idk | 18:44 |
| jroll | thanks for the discussion, it was helpful :) | 18:44 |
| *** dprince has quit IRC | 18:45 | |
| openstackgerrit | Chuck Short proposed openstack/ironic-inspector master: Switch to using stestr https://review.openstack.org/578524 | 18:45 |
| *** mgoddard has quit IRC | 18:47 | |
| waterloo | good luck jroll! :) | 18:48 |
| *** markguz has joined #openstack-ironic | 18:50 | |
| *** zzzeek has quit IRC | 18:52 | |
| *** zzzeek has joined #openstack-ironic | 18:57 | |
| *** MrBerns has quit IRC | 19:02 | |
| *** zshi has quit IRC | 19:08 | |
| *** zshi has joined #openstack-ironic | 19:09 | |
| *** zshi has quit IRC | 19:20 | |
| *** zshi has joined #openstack-ironic | 19:21 | |
| *** tssurya has joined #openstack-ironic | 19:21 | |
| openstackgerrit | Aleksandra Bezborodova proposed openstack/sushy master: Add a virtual media resource https://review.openstack.org/570810 | 19:36 |
| *** TxGirlGeek has joined #openstack-ironic | 19:39 | |
| *** markguz_ has joined #openstack-ironic | 19:48 | |
| *** trown|lunch is now known as trown | 19:49 | |
| *** mmethot_ has quit IRC | 19:50 | |
| *** markguz has quit IRC | 19:51 | |
| *** mmethot has joined #openstack-ironic | 19:51 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic-lib master: Fall back to PARTUUID if UUID returns nothing. https://review.openstack.org/569158 | 20:01 |
| *** TxGirlGeek has quit IRC | 20:06 | |
| *** e0ne has joined #openstack-ironic | 20:12 | |
| needssleep | The second deploy step patch is about to enter the gate, maybe ~15 minutes \o/ | 20:12 |
| *** bfournie has quit IRC | 20:14 | |
| *** jungleboyj has joined #openstack-ironic | 20:14 | |
| *** bfournie has joined #openstack-ironic | 20:14 | |
| *** tssurya has quit IRC | 20:22 | |
| *** markguz_ has quit IRC | 20:25 | |
| *** markguz has joined #openstack-ironic | 20:26 | |
| *** boris_42_ has joined #openstack-ironic | 20:28 | |
| *** markguz_ has joined #openstack-ironic | 20:29 | |
| *** markguz__ has joined #openstack-ironic | 20:32 | |
| *** markguz has quit IRC | 20:32 | |
| *** markguz_ has quit IRC | 20:36 | |
| *** markguz has joined #openstack-ironic | 20:37 | |
| *** markguz__ has quit IRC | 20:41 | |
| *** markguz_ has joined #openstack-ironic | 20:41 | |
| *** baha has quit IRC | 20:43 | |
| *** markguz has quit IRC | 20:45 | |
| *** bfournie has joined #openstack-ironic | 20:47 | |
| *** markguz_ has quit IRC | 20:48 | |
| needssleep | I updated the priorities list and added additional details for Monday | 20:57 |
| needssleep | I'm likely going to call it a night shortly | 20:57 |
| *** trown is now known as trown|outtypewww | 21:09 | |
| *** cjloader has quit IRC | 21:11 | |
| *** gyee has joined #openstack-ironic | 21:25 | |
| needssleep | Anyone doing reviews, hold off on approving changes.... static.o.o/logs.o.o is down | 21:27 |
| JayF | I must have missed the snowy weather reports in hell, and/or the flying pigs outside the window! | 21:41 |
| JayF | Change subject: virt/ironic: Implement rescue and unrescue .............................Patch Set 33: Code-Review+2 Workflow+1 | 21:41 |
| JayF | thanks so much to those who finally got this stuff landed :) | 21:41 |
| fried_rice | :* | 21:42 |
| *** e0ne_ has joined #openstack-ironic | 21:42 | |
| *** e0ne has quit IRC | 21:42 | |
| *** ElCoyote_ has left #openstack-ironic | 21:43 | |
| *** ddurst- is now known as ddurst | 21:49 | |
| -openstackstatus- NOTICE: logs.openstack.org is offline, causing POST_FAILURE results from Zuul. Cause and resolution timeframe currently unknown. | 21:50 | |
| *** ChanServ changes topic to "logs.openstack.org is offline, causing POST_FAILURE results from Zuul. Cause and resolution timeframe currently unknown." | 21:50 | |
| *** figleaf is now known as edleafe | 22:05 | |
| *** e0ne_ has quit IRC | 22:12 | |
| *** e0ne has joined #openstack-ironic | 22:13 | |
| *** med_ has joined #openstack-ironic | 22:27 | |
| *** med_ has quit IRC | 22:27 | |
| *** med_ has joined #openstack-ironic | 22:27 | |
| *** ptx0 is now known as kash | 22:29 | |
| *** kash is now known as ptx0 | 22:29 | |
| *** e0ne has quit IRC | 22:33 | |
| *** fried_rice is now known as efried | 22:36 | |
| *** med_ has quit IRC | 22:39 | |
| *** rpioso is now known as rpioso|afk | 23:14 | |
| *** leakypipes is now known as jaypipes | 23:31 | |
| *** ChanServ changes topic to "Bare Metal Provisioning | Status: http://bit.ly/ironic-whiteboard | Docs: http://docs.openstack.org/ironic/ | Bugs: https://storyboard.openstack.org/#!/project_group/75" | 23:34 | |
| -openstackstatus- NOTICE: logs.openstack.org is back on-line. Changes with "POST_FAILURE" job results should be rechecked. | 23:34 | |
| *** TxGirlGeek has joined #openstack-ironic | 23:37 | |
| *** TxGirlGeek has quit IRC | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!