| opendevreview | Julia Kreger proposed openstack/ironic master: Adds service steps https://review.opendev.org/c/openstack/ironic/+/884783 | 00:41 |
|---|---|---|
| opendevreview | Julia Kreger proposed openstack/ironic master: Retool sqlite retries https://review.opendev.org/c/openstack/ironic/+/891333 | 01:45 |
| opendevreview | Merged openstack/ironic master: Add job to test with SQLAlchemy master (2.x) https://review.opendev.org/c/openstack/ironic/+/886020 | 03:13 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/python-ironicclient master: Firmware Interface support https://review.opendev.org/c/openstack/python-ironicclient/+/891560 | 03:25 |
| iurygregory | JayF, TheJulia since I promised last week ^ | 03:25 |
| *** dmellado819181 is now known as dmellado81918 | 04:45 | |
| rpittau | good morning ironic! o/ | 07:04 |
| kubajj | good morning rpittau and Ironic o/ | 08:06 |
| rpittau | hey kubajj :) | 08:07 |
| dtantsur | TheJulia: https://github.com/metal3-io/cluster-api-provider-metal3/issues/1082 is where we're tracking the database locked issue | 08:16 |
| dtantsur | TheJulia: another instance, looks absolutely identical: https://jenkins.nordix.org/view/Metal3%20Periodic/job/metal3_daily_release-1-4_e2e_integration_test_ubuntu/128/ | 08:18 |
| arne_wiebalck | Good morning kubajj rpittau dtantsur and Ironic o/ | 09:40 |
| rpittau | hey arne_wiebalck :) | 09:48 |
| mmalchuk | morning Ironic o/ | 10:30 |
| mmalchuk | rpittau CI still unhappy in stable branches? | 10:30 |
| rpittau | mmalchuk: let me check if the new pkg is there | 10:30 |
| rpittau | new pkg is there, so I'm rechecking the revert for the workaround | 10:31 |
| rpittau | stable branches are probably ok at this point | 10:31 |
| mmalchuk | oh, cool | 10:32 |
| iurygregory | good morning Ironic | 11:31 |
| *** jroll8 is now known as jroll | 12:09 | |
| arne_wiebalck | hey iurygregory o/ | 12:48 |
| iurygregory | hey arne_wiebalck o/ | 12:48 |
| mmalchuk | rpittau https://review.opendev.org/c/openstack/bifrost/+/891127 | 12:53 |
| mmalchuk | rpittau still failed( | 12:53 |
| rpittau | mmalchuk: still isntalling the old pkg systemd-container-252-16.el9.x86_64 | 13:07 |
| rpittau | probably not all mirrors have been synced yet | 13:07 |
| mmalchuk | do you think need couple rechecks? | 13:08 |
| rpittau | mmalchuk: I would wait for this https://review.opendev.org/c/openstack/bifrost/+/891014 to pass first | 13:09 |
| mmalchuk | ok | 13:10 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Adds service steps https://review.opendev.org/c/openstack/ironic/+/884783 | 13:34 |
| JayF | rpittau: PTL nominations are open. It's either you or me buddy. I'm happy to nominate myself and continue another cycle but I'm also happy to step back and let you take the reins :) | 14:05 |
| rpittau | JayF: thanks, I'm still thinking about it mainly based on downstream commitment, but I'm seriously considering it for this cycle :) | 14:08 |
| * TheJulia sighs... ipv6 | 14:10 | |
| dking | IPv6 is the greatest thing that ever got ignored. | 14:10 |
| dking | JayF: I found my issue the other day. It turns out that it had nothing to do with pynetbox. It's an issue in the eventlet module itself. | 14:12 |
| JayF | It wouldn't really be the first time :) | 14:12 |
| dking | JayF: What's worse, it seems to be related to an issue that was submitted in 2017 and has been confirmed across multiple versions, is still open, and looks like it hasn't been touched since early last year: https://github.com/eventlet/eventlet/issues/371 | 14:13 |
| JayF | makes sense | 14:13 |
| JayF | eventlet is amazing and terrible at the same time :-) | 14:13 |
| dking | I don't doubt it. As far as I can tell, the project looks dead. Nearly 200 open issues, almost no activity, and their community link is to a freenode IRC channel, which only has 2 other people in it at the moment. | 14:14 |
| JayF | Oh, it's extremely not dead | 14:15 |
| JayF | it's had 200 issues open since I've used it. It's just a rolling set :) | 14:16 |
| dking | Well, some of them don't roll much, at least. | 14:16 |
| JayF | Use of it has decreased in new projects due to real-async being more easily available (as opposed to green-async) | 14:16 |
| dking | Do you know where to find the community? | 14:16 |
| JayF | I think it's a person or two, I've never tried to reach out for any sync comms | 14:17 |
| JayF | but I have gotten comments/ responses/etc in my issues before | 14:17 |
| dking | Yeah, that's probably the better option. I was wondering why that wasn't used instead, but I don't really know all of the eventlet magic. | 14:17 |
| JayF | The answer to that question, in many openstack contexts, is temporal in nature | 14:17 |
| JayF | there was no real async stuff | 14:17 |
| JayF | there was only green-async via eventlet/greenlet | 14:17 |
| dking | Okay, so what we have now isn't ideal, but at this point, it would just be a beast to update it? | 14:18 |
| JayF | I'd say it slightly different | 14:26 |
| JayF | if openstack was being created today, in python, it'd likely use asyncio | 14:26 |
| dking | Just getting my head around it, would it be at all feasible to consider replacing it in IPA? I believe that there's really only the calls going back to the Ironic API? | 14:30 |
| JayF | In terms of your problem; it'd likely make what you want harder | 14:31 |
| JayF | because you'd have to use async aware libraries for things like pynetbox | 14:31 |
| JayF | which it likely isn't | 14:31 |
| dking | True. It's not the best fix for my one bug. But I'm just wondering if it would be more future proof if it were changed. | 14:31 |
| JayF | and bluntly, with IPA we have all the parallel bits working properly, and it barely changes; so the cost:benefit is way outta whack: big cost, big risk, minimal benefit | 14:31 |
| dking | Yes, that would be a problem on me for my local stuff. | 14:32 |
| dking | That's probably fair. I just worry because if it's using an outdated concept, and that module will likely continue to degrade, it just makes me wonder if we won't start to see more issues in the future. Perhaps, though, it's just best to wait until the problems actually materialize. | 14:34 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Don't yield on power sync at the end of the work https://review.opendev.org/c/openstack/ironic/+/891335 | 14:34 |
| JayF | Yeah, when there are problems we solve them; in the meantime we have too many other problems to generate them ahead of time | 14:37 |
| TheJulia | dtantsur: I'm wondering if https://review.opendev.org/c/openstack/ironic/+/891335 might be worth considering, w/r/t the fact it seems like we end up in the state after the power state check seems to end its work. | 14:39 |
| dtantsur | I'm not even sure how important this yield is... so, everything possible? | 14:41 |
| TheJulia | just thinking it could leave a greenthread out there that doesn't immediately wrap up | 14:41 |
| dtantsur | TheJulia: another thought. there are a few cases where we may be moving a lock across green threads, power sync and inspection are among them. | 14:41 |
| TheJulia | well, yes | 14:42 |
| dtantsur | I highly hope we don't carry an open transaction this way.. but who knows at this point | 14:42 |
| TheJulia | I had thought about that too, but the launcher of the thread is a reader, not a writer in both cases | 14:42 |
| TheJulia | and even then, that originator call *should* be closed out | 14:42 |
| TheJulia | since we extract the data | 14:42 |
| TheJulia | and we don't use the magical orm objects, but there is a possibility (and it seems too close to be a coincidence...) | 14:43 |
| * TheJulia goes back to trying to make a super smart magical decorator that understands when "I can't timeout" so we don't horribly fall over | 14:43 | |
| JayF | I have an MLH intern starting in a couple of months to work with us for 10 weeks upstream. Trying to figure out a good project, and I think I got one, as long as folks are onboard -- how would people feel about adding type hints to Ironic? | 15:12 |
| JayF | oslo.context has them, so there is some precedent for them in openstack, and I think it'd be a good low level project for someone new + would be helpful to folks coming afterwards | 15:13 |
| dtantsur | Oh, and outreachy is about to start once again :) | 15:24 |
| JayF | Yeah, this is driven by G-Research Open Source Software team directly | 15:25 |
| JayF | I'll be mentoring this person, but the project focus will be OpenStack | 15:25 |
| JayF | MLH and Outreachy, afaict, are basically the same thing with different names and slightly changed details, fwiw | 15:25 |
| JayF | dtantsur: wdyt about the meat of that? Type hinting! | 15:25 |
| dtantsur | I have pretty mixed feelings of type hints, to be honest. | 15:28 |
| dtantsur | If we could figure out which parts of Ironic are "library", we could annotate it | 15:28 |
| JayF | What is the harm caused in annotating non-library portions? | 15:29 |
| JayF | I like it when my IDE knows what's going on a little better :D | 15:29 |
| opendevreview | Arne Wiebalck proposed openstack/ironic-python-agent master: Conditional creation of RAIDed ESP for UEFI Software RAID https://review.opendev.org/c/openstack/ironic-python-agent/+/891609 | 15:41 |
| JayF | arne_wiebalck: Lemme DM you the address you can mail in your "Management" card to ;) managers aren't allowed to write code /s | 15:44 |
| JayF | arne_wiebalck: more seriously, nice to see your name come across on gerrit :D | 15:44 |
| dtantsur | JayF: no harm, but to do it properly sometimes requires twisting your brain with the Python's version of generics :) | 16:02 |
| JayF | heh | 16:02 |
| JayF | honestly I was going to suggest, if I wrote it down | 16:02 |
| JayF | to specifically keep things like Callable[] generic to start with | 16:02 |
| JayF | it removes some of the value but makes "get the entire project type hinted at all and mypy enabled" achievable | 16:03 |
| * TheJulia tosses things at python decorators | 16:04 | |
| JayF | python decorators are well-known for how little they complicate type checking <.< >.> | 16:04 |
| JayF | TheJulia: if you need someone to quack with you I'm already in a context switch | 16:04 |
| TheJulia | I'm giving up on trying to dynamically determine the decorated method name | 16:04 |
| * JayF has been playing in the oslo box all morning | 16:04 | |
| TheJulia | since all I can see is the class object as the function with how it is wired in | 16:05 |
| JayF | that is very hard to do especially if you might multiply-decorate | 16:05 |
| JayF | since it can return the name of the decorator below it if stacked | 16:05 |
| TheJulia | because I don't want to raise an error on release or node_update because if they fail, things go sideways horrifically | 16:05 |
| JayF | why not go not magic at all? @retry(timeout=int or None) | 16:05 |
| JayF | and if None it goes kaboom | 16:05 |
| JayF | is that possible? | 16:05 |
| * JayF suspects there's a complication he's unaware of | 16:06 | |
| TheJulia | Can't seem to do it with the style either | 16:06 |
| TheJulia | I've burned close to two hours on it | 16:06 |
| JayF | If you want to sync up about it, just ask. Otherwise I'll let you keep going :) | 16:06 |
| rpittau | good night! o/ | 16:11 |
| JayF | o/ | 16:11 |
| TheJulia | oh noes, I might have typoed it to begin with | 16:16 |
| TheJulia | ugh. | 16:17 |
| TheJulia | I did | 16:17 |
| JayF | don't be annoyed that you typo'd it, be happy that there's not a more annoying technical gremlin under the surface | 16:20 |
| * JayF has gotten very good at consoling himself over silly typos wasting time | 16:20 | |
| JayF | TheJulia: can you paste me up your minimal ironic+nova devstack config again? | 16:38 |
| * JayF prepares to be very angry at devstack again lol | 16:38 | |
| TheJulia | oh jeeze | 16:39 |
| TheJulia | I crashed my window manager and lost the text file | 16:39 |
| TheJulia | but you were missing bits to force it to OVS | 16:39 |
| JayF | you crashed your window manager and lost the text file | 16:40 |
| JayF | I restored a snapshot on my VM and lost it :| | 16:40 |
| TheJulia | heh | 16:40 |
| TheJulia | can you share your config, I'll highlight the bits and send it back | 16:40 |
| JayF | I was using the one from the documentation before I started workign with you | 16:41 |
| JayF | which you replaced with like, a 4 line config | 16:41 |
| TheJulia | it was 4 lines you were missing | 16:42 |
| JayF | lmao | 16:42 |
| JayF | no wonder it didn't work | 16:43 |
| JayF | I removed my whole config and replaced it with those four lines | 16:43 |
| JayF | I was feeling silly for having 30 lines when like 4 were needed, this all makes more sense now | 16:43 |
| TheJulia | well, they are a fairly important set of four, I remember you re-enabled the plugin | 16:43 |
| JayF | I'm going to start with https://docs.openstack.org/ironic/latest/contributor/dev-quickstart.html#ironic-with-ironic-tempest-plugin | 16:44 |
| JayF | and look for OVS-related configuration flags in the .localrc.auto from a CI build, I guess? | 16:44 |
| TheJulia | so, it was something like you needed to enable the following services: q-agt, q-dhcp, q-l3, q-svc, q-meta | 16:45 |
| TheJulia | disable: ovn-controller, ovn-northd | 16:45 |
| TheJulia | set Q_AGENT=openvswitch | 16:45 |
| TheJulia | enable the plugin of course | 16:45 |
| TheJulia | oh, also disable service metadata | 16:46 |
| TheJulia | err | 16:46 |
| TheJulia | disable service q-ovn-metadata-agent | 16:46 |
| arne_wiebalck | JayF: :-D | 16:47 |
| arne_wiebalck | JayF: I was about to prelude that commit with "You know it is quiet summer time when managers start writing code." | 16:48 |
| JayF | thank you TheJulia :D I'll figure it out and when I do that doc is getting updated | 16:48 |
| TheJulia | heh | 16:49 |
| JayF | https://gist.github.com/jayofdoom/b18e86cfe202da1691cdb4f297ff5e48 this is what I'm about to get started with | 16:51 |
| TheJulia | that looks pretty good to me | 16:51 |
| JayF | well I was smart enough to make a new snapshot before stack | 16:51 |
| JayF | so if it doesn't work I'll have that conf still lol | 16:51 |
| * JayF stackin' | 16:51 | |
| JayF | TheJulia: did you have an opinion on 'have MLH interns begin implementing type hinting in Ironic' | 16:52 |
| * JayF wonders if something like that needs to be RFE/spec'd | 16:52 | |
| TheJulia | I find hinting makes it much harder for me to just read the code | 16:52 |
| TheJulia | so not really a fan of it, honestly | 16:52 |
| JayF | I wish it wasn't optional, frankly. It's a straight technical improvment and it's unforunate that aestethics come into play at all :( | 16:53 |
| JayF | I'll write it up though as it seems not to be the slam dunk I assumed it'd be | 16:53 |
| TheJulia | okay | 16:54 |
| JayF | either way I gotta find 10 weeks of low hanging work, probably more, if there are alternate suggestions I'm game :D | 16:54 |
| TheJulia | requirements.... | 16:54 |
| TheJulia | lets see about moving to some newer versions | 16:54 |
| TheJulia | even just in ironic would be a good start :) | 16:55 |
| JayF | I've asked kubajj if he'd be interested in a focus on requirements after he's done with the *_by_arch work | 16:55 |
| TheJulia | k | 16:55 |
| JayF | I thought it'd be wise to keep that kinda work into folks who are likely to be around the community longer | 16:55 |
| JayF | seems like it'd be intimidating for an intern and a lot of ramp up time to throw away the context shortly after | 16:55 |
| TheJulia | on the flip side of it, they should be able to ramp up quickly becuase they don't need to know internals to detect high level issues :) | 16:56 |
| JayF | I think the reason I didn't consider that is because one time work on it is useful; but having the community ramp up so that work can recur or be automated | 16:58 |
| JayF | but it's worth consideration | 16:58 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Retool sqlite retries https://review.opendev.org/c/openstack/ironic/+/891333 | 17:00 |
| TheJulia | yeah, automated would be good in the end | 17:00 |
| clarkb | you can use stub files to keep the actual code clean. But this brings other issues | 17:00 |
| TheJulia | or at least, something better than the current state which can help drive things forward and someone to then champion that moving forward would be key | 17:01 |
| JayF | The least of which is that I have no idea how to do that ;) | 17:01 |
| * TheJulia gets out a box labeled "Magic" and hands it to JayF | 17:01 | |
| JayF | TheJulia: bluntly, I'm not even sure how I'd approach the requirements problem ... but I think you may be right that too much context is paralyzing for some of these problems | 17:01 |
| clarkb | My personal frustration is that you run into corner cases very quickly because python was never intended to be used that way. Things like conditional imports for C implementations vs native implementations of things with identical apis are basically unsupported | 17:01 |
| clarkb | it feels like they took a pedantic appraoch rather than looking at how to apply type checking to existing code bases | 17:02 |
| JayF | the headache I've hit in personal projects is that if you're using libraries that don't do type hinting | 17:02 |
| TheJulia | JayF: oh yes, very yes. And to clarkb's point, an unconventional approach may be needed to at least be a human driver :) | 17:03 |
| JayF | it's really hard to get tools like IDEs to calm down about it | 17:03 |
| TheJulia | dtantsur: revised retries, you are likely going to raise an eyebrow, but the commit message should explain "why". In the morning when you get up, retry it please while CI us under heavy load | 17:04 |
| JayF | I just put a couple of comments on that, too | 17:06 |
| JayF | (the retries) | 17:06 |
| TheJulia | JayF: w/r/t kwarg basing a retry, feel free to give uit a spin | 17:10 |
| JayF | Why do I feel like Daffy Duck right after Bugs has handed him his stick of dynamite \back? | 17:10 |
| JayF | lol | 17:10 |
| * TheJulia whistles | 17:11 | |
| JayF | time for me to do my part | 17:11 |
| JayF | that's right, there's going to be a duck-sized hole in that wall | 17:12 |
| TheJulia | re, the help string, I'll update it once metal3 gives CI results | 17:12 |
| TheJulia | lol | 17:12 |
| JayF | [ERROR] /opt/stack/devstack/functions-common:2339 Neutron did not start | 17:16 |
| JayF | :( going to look at logs after I finish running thru this wall | 17:16 |
| TheJulia | it is likely a silly minor piece of config | 17:17 |
| JayF | I've assumed it's been <10 lines of config the whole time | 17:17 |
| JayF | knowing which lines is the trick ;) | 17:17 |
| JayF | sufferin' succotash | 17:25 |
| JayF | I think I got it | 17:25 |
| JayF | (that was what daffy duck used to spit-exclaim, right?) | 17:25 |
| TheJulia | maybe?! | 17:26 |
| JayF | in true daffy duck fashion I claimed victory too soon | 17:28 |
| TheJulia | At least the 1812 Overture is not playing in the background | 17:29 |
| JayF | my subconsious is cratering my attempts to do this so I don't have to troubleshoot devstack more ;) | 17:29 |
| JayF | OHHH | 17:30 |
| * JayF missed a bunch of () that are needed | 17:31 | |
| JayF | now I got it | 17:32 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Demo: using kwargs instead Retool sqlite retries https://review.opendev.org/c/openstack/ironic/+/891615 | 17:35 |
| JayF | TheJulia: I made a hay of git trying to not cause your change to re-kick CI, so I ended up squashing it all into ^ | 17:35 |
| JayF | TheJulia: if you like it, steal it :) | 17:35 |
| TheJulia | hmm, that might work | 17:36 |
| JayF | it passes unit tests | 17:36 |
| JayF | basically to get arguments into a decorator, you turn it into a decorator factory | 17:36 |
| JayF | which makes sense, and I've followed the pattern, but I'm not sure I fully understood it until today | 17:37 |
| JayF | TheJulia: re: devstack; I think neutron might have just taken too long to come up | 17:40 |
| JayF | the test devstack was running that failed now passes | 17:40 |
| JayF | I'm going to just re-run stack.sh | 17:40 |
| * JayF notes it looks like it got further, perhaps | 17:50 | |
| JayF | nope, died | 17:57 |
| JayF | > .ml2.drivers.ovn.mech_driver.mech_driver [None req-3fa5a9ea-c1ca-4907-a07f-f89506dede20 None None] Geneve max_header_size set too low for OVN (30 vs 38) | 18:01 |
| JayF | is what it's dying on | 18:01 |
| TheJulia | man, why is it trying to load OVN | 18:01 |
| TheJulia | Q_ML2_PLUGIN_MECHANISM_DRIVERS to openvswitch perhaps | 18:01 |
| * JayF added that, restacking | 18:02 | |
| JayF | I'll note that is set in our CI jobs, so that gives me some hope | 18:02 |
| * JayF afk maybe a long lunch | 18:02 | |
| JayF | still failing it looks like, will get a clean run to ensure | 19:25 |
| TheJulia | :( | 19:26 |
| JayF | I think there's a nonzero chance we broke idempotence in Ironic's devstack plugin | 19:26 |
| JayF | well, I shouldn't say it that way | 19:31 |
| JayF | because it's likely *in neutron* if it works reconfigured | 19:31 |
| JayF | TheJulia: I'm going to abandon https://review.opendev.org/c/openstack/ironic/+/891615 since it's mostly copied from you, but it did pass CI | 19:32 |
| JayF | TheJulia: feel free to reuse that (or not, my original comment was a thought not intended to be blocking) | 19:32 |
| TheJulia | ok | 19:33 |
| * JayF saw a ERROR ovsdbapp.backend.ovs_idl.idlutils [-] Unable to open stream to tcp:127.0.0.1:6640 to retrieve schema: Connection refused fly by | 19:42 | |
| JayF | hopefully that's not actually-breaky | 19:42 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Add service steps call to agent logic https://review.opendev.org/c/openstack/ironic/+/891229 | 19:45 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP: Add some service steps https://review.opendev.org/c/openstack/ironic/+/891630 | 19:45 |
| JayF | TheJulia: :( https://gist.github.com/jayofdoom/5b56888bed759840ac31d074679fb8cf | 19:54 |
| JayF | tl;dr | 19:54 |
| JayF | Cannot open network namespace "qdhcp-343ddd44-a6c5-48b3-8e8a-4c2e7d15a8b9": No such file or directory | 19:54 |
| JayF | it's referncing a network namespace that doesn't exist on my machine | 19:54 |
| JayF | in +++/opt/stack/ironic/devstack/lib/ironic:create_ovs_taps:2044 it seems | 19:55 |
| * JayF digs | 19:55 | |
| JayF | Hmm. looks like create_ovs_taps is built with the assumption that qdhcp-{ironic_net_id} will exist as a network namespace | 19:57 |
| JayF | but it doesn't | 19:57 |
| TheJulia | sounds like your somehow still on ovn | 20:00 |
| JayF | How would I validate that? | 20:00 |
| TheJulia | or to be more precise | 20:00 |
| TheJulia | the neutron-dhcp-agent is not running | 20:00 |
| JayF | My reading of the code suggests this might be where I need to add IP_VERSION=4 | 20:00 |
| JayF | to fix it | 20:00 |
| TheJulia | eh, maybe | 20:01 |
| JayF | and also given I got a different/better err; this is more evidence pointing to "we don't do idempotence well in our devstack configs" | 20:01 |
| TheJulia | that is one of the free-bie arguments always in upstream | 20:01 |
| JayF | whoa q-dhcp service is crashing | 20:01 |
| JayF | on this broken devstack | 20:01 |
| TheJulia | enjoy! | 20:01 |
| JayF | hmmm maybe not, just a guru mediation report | 20:02 |
| JayF | oh, when devstack failed | 20:02 |
| JayF | ofc | 20:02 |
| JayF | aight going to try fresh with IP_VERSION=4 | 20:02 |
| clarkb | devstack has never been intended to be idempotent | 20:03 |
| JayF | TIL | 20:03 |
| TheJulia | ... I was going to say something along the lines of "the idea of it being such is laughable" | 20:03 |
| JayF | That's a false assumption I've been carrying for a long, long time | 20:03 |
| JayF | like insanely long time | 20:03 |
| JayF | like might explain some % of why I have always been frustrated with devstack | 20:03 |
| clarkb | it encodes the path from an empty state to the state configured. It knows very little how to configure from one configured state to another | 20:04 |
| clarkb | including "please rerun the exact same config again" it may try to redo unsafe setup steps etc | 20:05 |
| JayF | all this makes 100% sense; I don't know where I got that bad info and am discarding it in favor of the obviously-more-correct info :D | 20:06 |
| opendevreview | Julia Kreger proposed openstack/ironic master: DNM Enable OVN https://review.opendev.org/c/openstack/ironic/+/885087 | 20:21 |
| * TheJulia goes and takes migraine meds | 20:26 | |
| JayF | +++functions-common:test_with_retry:2338 timeout 20 sh -c 'while ! sudo ip netns exec qdhcp-f55cbfba-1879-4c6a-912f-58e84357b813 ip link list | grep '\'' tap'\'' | cut -d'\'':'\'' -f2 | cut -d'\''@'\'' -f1 | cut -b2- | grep '\''^tap'\''; do sleep 1; done' | 20:31 |
| JayF | Cannot open network namespace "qdhcp-f55cbfba-1879-4c6a-912f-58e84357b813": No such file or directory | 20:31 |
| JayF | same err with IP_ADDRESS=4 the whole time :( | 20:31 |
| JayF | I'm going to lightly dig into this but not sure how much capacity I have for this today :| | 20:31 |
| TheJulia | I just took migraine meds | 20:31 |
| TheJulia | capacity is low and banging my head on ovn now | 20:31 |
| JayF | yeah please go rest if you're not feeling well | 20:31 |
| TheJulia | well, more like ~1.25 hours before I have anything I can look at from cI | 20:32 |
| JayF | > Network f55cbfba-1879-4c6a-912f-58e84357b813 is not hosted by any dhcp agent {{(pid=156765) _notify_agents /opt/stack/neutron/neutron/api/rpc/agentnotifi> | 20:32 |
| TheJulia | that is a new one | 20:33 |
| JayF | Do we have any V4-only CI clouds that we set IP_VERSION=4 on? | 20:34 |
| JayF | actually nevermind, going to ignore that, it's probably a red herring | 20:34 |
| JayF | need to dig logs more and see why | 20:34 |
| clarkb | ovh is v4 only the way we deploy it (it can do ipv6 but we don't because of cloud metadata problems that may no longer exist) | 20:34 |
| JayF | so we set IP_ADDRESS=4 in devstack for that stuff? | 20:35 |
| clarkb | the openmetal/inmotion cloud is also v4 only | 20:35 |
| clarkb | you'd have to check job logs. But v4 only clouds are a thing in upstream CI | 20:35 |
| JayF | ack | 20:35 |
| JayF | I'm looking at a .localrc.auto from a successful CI job vs my local.conf and trying to ID anything | 20:38 |
| * JayF sus of Q_ML2_TENANT_NETWORK_TYPE="vxlan" | 20:41 | |
| JayF | Q_USE_SECGROUP="False" | 20:41 |
| JayF | trying with those two pieces set | 20:45 |
| JayF | I'm not sure, but I think I just saw that error code pass by happy with those changes | 21:10 |
| JayF | I'll have to see if it was one or both | 21:10 |
| JayF | FYI: I may have to evacuate due to a brush fire in my area. Right now we're safe but making a go-bag. Three-alarm brush fire at a park about 1.5mi from my house | 21:11 |
| JayF | I better offsite backup this working local.conf LOL | 21:11 |
| JayF | assuming the nodes finish cleaning, but I have no reason to suspect they won't | 21:11 |
| JayF | That blew up, but only because I had ironic tempest enabled but not installing, somehow (?), not a big deal | 21:49 |
| JayF | in any event, I'm unlikely to get useful stuff done b/c I'm on an adrenaline rush from the evac order (we are at level 2; create a go-bag). | 21:50 |
| clarkb | JayF: good luck. I think some people will tell you to clear out your gutters and then water the roof/property if you can too | 21:53 |
| JayF | clarkb: ack; in this case there are two new pieces of info: | 22:08 |
| JayF | 1) we found a video feed from a local news chopper. the fire is dying down and is moving opposite direction of us (wind is so in the opposite direction we have no smoke here) | 22:08 |
| JayF | and 2) They screwed up. Evac notices were supposed to go to *.5* mi away, they went to *5* mi away | 22:08 |
| JayF | we just got the update after packing frantically for 30 minutes | 22:08 |
| JayF | I also have a tile roof? which I think is an advantage but also means spraying with water from beneath is a good way to get unintended water intrusion | 22:09 |
| TheJulia | oh jeeze | 22:10 |
| JayF | I think everyone here in emergency management/fire/etc are a little on edge | 22:10 |
| TheJulia | at least you didn't have a neighbor claiming the HOA spoke with the county and that you were being evacuated | 22:10 |
| TheJulia | yeah | 22:10 |
| JayF | we lost 5 residents last week to a brush fire which quickly went into a trailer park and destroyed 8 houses | 22:10 |
| JayF | so the county gets a 1000x grace from me for missing the decimal point; I'm sure those folks have had a tramatic few days | 22:11 |
| * TheJulia wonders why jobs no longer grab ovn logs | 22:14 | |
| JayF | btw TheJulia in case you missed it | 22:14 |
| JayF | tyvvvvvvvm for helpingwith ironic/nova devstack | 22:14 |
| JayF | it's working | 22:14 |
| TheJulia | woot | 22:14 |
| JayF | well, I should say | 22:14 |
| JayF | it got to trying to do tempest things and failed | 22:14 |
| JayF | which means it's 100x more worky | 22:14 |
| TheJulia | you have to install the plugin for that | 22:14 |
| TheJulia | so... | 22:14 |
| TheJulia | lol | 22:14 |
| JayF | one (or both) of Q_ML2_TENANT_NETWORK_TYPE="vxlan" / Q_USE_SECGROUP="False" | 22:15 |
| JayF | made the difference | 22:15 |
| TheJulia | hmm, zuul_copy_output, why you htae us | 22:18 |
| mmalchuk | rpittau https://review.opendev.org/c/openstack/bifrost/+/891014 | 22:26 |
| mmalchuk | lets merge it | 22:27 |
| TheJulia | button pushed | 22:27 |
| mmalchuk | TheJulia thanks | 22:30 |
| mmalchuk | also I've push recheck all my stable branch changes, CI should pass | 22:30 |
| mmalchuk | lets see | 22:30 |
| opendevreview | Julia Kreger proposed openstack/ironic master: DNM Enable OVN https://review.opendev.org/c/openstack/ironic/+/885087 | 22:37 |
| mmalchuk | TheJulia could you please also push +W here https://review.opendev.org/c/openstack/bifrost/+/891134 | 22:42 |
| mmalchuk | CI is passed already | 22:42 |
| mmalchuk | and I remember about merged but not released) | 22:42 |
| mmalchuk | thanks | 22:42 |
| JayF | by rule we have to land them in order | 22:42 |
| JayF | so we can't land the xena one until it's in yoga, can't land yoga until it's in zed, etc | 22:42 |
| mmalchuk | oh... rules | 22:43 |
| JayF | yeah, we don't ever want someone to upgrade into a bug | 22:43 |
| mmalchuk | thanks JayF | 22:43 |
| opendevreview | Verification of a change to openstack/bifrost stable/xena failed: Create the log file for the disk-image-create command https://review.opendev.org/c/openstack/bifrost/+/891134 | 22:43 |
| TheJulia | yeah, good catch JayF | 22:44 |
| mmalchuk | TheJulia np, lets wait) | 22:44 |
| mmalchuk | also its time to go sleep) | 22:45 |
| mmalchuk | Night brings wisdom) | 22:46 |
| TheJulia | :) | 23:56 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!