| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic-specs master: Firmware Interface https://review.opendev.org/c/openstack/ironic-specs/+/878505 | 02:39 |
|---|---|---|
| rpittau | good morning ironic! o/ | 07:09 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent-builder master: Move ubuntu jobs to jammy https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/879538 | 07:24 |
| opendevreview | Michal Nasiadka proposed openstack/bifrost master: Add logic to enable additional inspect_interfaces https://review.opendev.org/c/openstack/bifrost/+/879730 | 08:39 |
| opendevreview | Michal Nasiadka proposed openstack/bifrost master: Add logic to enable additional inspect_interfaces https://review.opendev.org/c/openstack/bifrost/+/879730 | 10:46 |
| Sandzwerg[m] | Hmm does ironic or Openstack requires a minimum/special version of MySQL/mariaDB? I can't find something at first glance | 10:56 |
| rpittau | seems like ipa-builder is happy with jammy, still need to check the dib job though https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/879538 | 12:17 |
| rpittau | Sandzwerg[m]: we don't recommend a minimum version, but my suggestion is to use mysql/maraidb 5.8 or similar, just because it's the same we use successfully in CI :) | 12:44 |
| rpittau | Sandzwerg[m]: err wrong package, it's mysql-server 8 | 12:47 |
| Sandzwerg[m] | I see, thanks. Is that documented anywhere? | 13:18 |
| TheJulia | A recommended version, as far as I'm aware, is not documented. The easy confusion issue here is we all use a common abstraction library called oslo.db which is built on top of SQLAlchemy. Good starting points are https://docs.openstack.org/oslo.db/latest/user/history.html#id1 and https://docs.sqlalchemy.org/en/14/dialects/mysql.html | 13:29 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: [WIP] Run metal3 integration with ironic-image from source https://review.opendev.org/c/openstack/ironic/+/879162 | 14:07 |
| JayF | Sandzwerg[m]: part of why we have a loose answer to this is that really it should just work with whatever version you pick for your operational needs | 14:17 |
| JayF | Sandzwerg[m]: Ironic even supports use w/sqlite in single conductor scenarios like metal3 | 14:17 |
| TheJulia | https://docs.sqlalchemy.org/en/14/dialects/ <-- Upstream Ironic also has done testing with Postgres as well, but we don't really dive into the non-FOSS databases even though in the wider community some folks have. | 14:22 |
| JayF | I think use of postgres would be nasty for upgrades, I'm pretty sure we have some 'if mysql' code in there | 14:30 |
| TheJulia | the upgrade code gets tested on both | 14:30 |
| JayF | that's pretty slick | 14:30 |
| TheJulia | and yes, there are some conditionals, because we have to approach some things differently | 14:30 |
| TheJulia | Index lookups are one thing | 14:31 |
| JayF | I've been working on ironic for a long time, and I still find easter eggs like this implemented when I wasn't looking :D | 14:31 |
| TheJulia | you can't easily ask one of them (I don't remember which) what indexes exist | 14:31 |
| TheJulia | ) | 14:31 |
| TheJulia | .. We actually got a patch at one point because someone tried running on DB2 | 14:31 |
| JayF | :-O | 14:32 |
| adam_metal3 | hi Ironic, do you know something about the status of Glean ? I see there were no commits merged for 10 months now, and I have reason to think that I have found bugs in it, who should I contact ? | 14:34 |
| JayF | Glean is very actively maintained, it's just mostly finished | 14:34 |
| JayF | if you've found bugs I'm pretty sure there's a bugtracker... | 14:34 |
| TheJulia | the OpenDev folks maintain it for the most part | 14:35 |
| JayF | uses storyboard for bugs, it appears | 14:35 |
| JayF | https://storyboard.openstack.org/ | 14:35 |
| TheJulia | Glean is also super simple, fwiw | 14:35 |
| TheJulia | where as cloud-init supports many different features | 14:36 |
| adam_metal3 | thanks for the info! do you know that in what irc gorup can I find the maintainers if I have questions ? | 14:38 |
| TheJulia | #opendev most likley | 14:38 |
| adam_metal3 | thanks a lot! | 14:38 |
| TheJulia | also just asking in here might be worthwhile as well, some of us have changed/contributed to it | 14:39 |
| adam_metal3 | okay thanks, then will quickly mention it here too, I am experiencing multiple unexpected issues when creating bond interface with glean based on the network_data.json file, I use glean on a centos 9 stream based IPA and the glean services of the bond "slave" interfaces are failing with an error to that always just say the Erro: "<mac of the other slave interface>", I have compared the network_data.json to glean's test | 14:44 |
| adam_metal3 | data, to the network_data schema and I have got one example from Sandzwerg[m] also and my file also follows the same conventions as those | 14:44 |
| adam_metal3 | the other issue is that the bond "master" interface or the bond interface not sure how to call it, has incorrect type, instead of bond it is Ethernet | 14:46 |
| adam_metal3 | clarification the not the bond interface has Ethernet type but the network-script create for said interface | 14:48 |
| Sandzwerg[m] | Thanks for all the mariaDB related answers JayF TheJulia I suspected something like this but wanted to make sure as we're planning on changing our DB setup and three colleagues who do this had some questions regarding this | 14:59 |
| TheJulia | adam_metal3: can you provide a santiized example of what your sending as the metadata? | 15:13 |
| adam_metal3 | yep I am trying to get one in text form from downstream I only have pictures atm | 15:13 |
| TheJulia | text would likely help, I suspect it is something to do with formatting. ... It is not the clearest format unfortunately | 15:14 |
| TheJulia | Sandzwerg[m]: cool, so one other piece of context, the mysql dialect includes use of a galera cluster. | 15:15 |
| Sandzwerg[m] | Ah good to know. Apparently they are thinking about using galera | 15:16 |
| TheJulia | most openstack clusters use galera for cluster replication management... in fact I don't know if anyone has ever used any of the pre-galera replication tools | 15:16 |
| JayF | I totally have | 15:19 |
| JayF | it was GREAT :| | 15:19 |
| TheJulia | what ring repl didn't eventually resolve everything? | 15:23 |
| * TheJulia ducks | 15:23 | |
| TheJulia | Sandzwerg[m]: so... for clarity, the issue with the old style replication, as I remember from doing stuff like this myself, is it was either hard active passive or multi-master, and you basically couldn't rely upon automatic incrementing primary keys | 15:24 |
| TheJulia | so your database schema had to be cognizant of what you were doing and use self-declared/tracking/associated primary keys | 15:25 |
| TheJulia | because the same record would have different table level pkeys | 15:25 |
| TheJulia | at least in multi-master | 15:25 |
| TheJulia | hard active/passive setups I think preserved the primary keys as originally requested | 15:25 |
| clarkb | adam_metal3: it is entirely possible you are the first person to attempt to use bond interfaces with centos 9 or possibly even with network manager all together. Glean was built largely to support OpenDev's test instance network configuration needs (in particular cloud-init didn't support rackspace at one point in time). We've been happy to add features for other use cases | 15:26 |
| clarkb | but our ability to test features like bonding is basically nil | 15:26 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: Run metal3 integration with ironic-image from source https://review.opendev.org/c/openstack/ironic/+/879162 | 15:28 |
| adam_metal3 | clarkb, Thank you for the info! this explains things, | 15:28 |
| TheJulia | NetworkManager based things does change things a bit | 15:29 |
| TheJulia | that could explain things a lot | 15:29 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: Run metal3 integration with ironic-image from source https://review.opendev.org/c/openstack/ironic/+/879162 | 15:29 |
| adam_metal3 | TheJulia, I have pasted the data here: https://pastebin.com/7XPf5bRV the password is:julia01 | 15:29 |
| TheJulia | I'd have to double check the underlying schema, but I *think* that looks correct at a glance | 15:31 |
| TheJulia | hmm, time for a test I think | 15:32 |
| TheJulia | with more coffee | 15:32 |
| adam_metal3 | TheJulia, thanks for taking a look! | 15:36 |
| JayF | iurygregory: around and available, perhaps? | 15:54 |
| * JayF just had all the bifrost nodes disappear from his demo | 15:55 | |
| TheJulia | what did you do?! | 15:56 |
| JayF | nothing? | 15:56 |
| JayF | I have no idea wtf is going on | 15:56 |
| JayF | I'm re-running testenv/install --testenv to try and make it happy | 15:57 |
| JayF | I think I ran deploy without an inventory, like the docs say, and it imght have wiped out my nodes | 15:57 |
| JayF | but I can't be sure b/c I didn't check for them before I started | 15:57 |
| TheJulia | ... your not running in a mode where access rights/auth issues would be an issue | 15:57 |
| JayF | https://docs.openstack.org/bifrost/latest/user/howto.html#deploy-hardware | 15:57 |
| JayF | TheJulia: I'm running all this as my user in a VM I control | 15:58 |
| TheJulia | yeah | 15:58 |
| TheJulia | so... that just triggers playbooks as I understand it | 15:58 |
| * TheJulia wonders | 15:58 | |
| TheJulia | deploy-dynamic.yaml is the playbook | 15:59 |
| TheJulia | that just creates configuration drives and then calls bifrost-deploy-nodes-dynamic | 15:59 |
| JayF | https://gist.github.com/jayofdoom/3f989bd2ea24885ba684eadcf6480171 | 16:00 |
| JayF | this is after I re-ran a testenv and a install | 16:00 |
| JayF | testvm1/2 exist in libvirt, but not in ironic | 16:01 |
| TheJulia | so, did you enroll at any time? | 16:01 |
| JayF | I was under the (flawed?) impression that was done by bifrost-cli install/deploy | 16:02 |
| TheJulia | bifrost-cli has an enroll command | 16:02 |
| TheJulia | which adds the nodes to ironic | 16:02 |
| * JayF checks docs | 16:02 | |
| TheJulia | the deploy takes the nodes and does the request to active state | 16:02 |
| JayF | yeah I think just the docs are not as direct as I expected | 16:02 |
| rpittau | does anyone remember how to increase disk size for ipa-builder jobs? I'm getting a "cpio: write error: No space left on device" on the ubuntu dib job with jammy | 16:04 |
| JayF | enroll did what I needed | 16:05 |
| JayF | TheJulia: what's the tl;dr difference between baremetal-nodes.json, baremetal-inventory.json | 16:06 |
| TheJulia | JayF: I don't think there is a difference | 16:08 |
| TheJulia | there shouldn't be afaik | 16:08 |
| TheJulia | rpittau: where on the disk was it trying to work? | 16:08 |
| rpittau | JayF: AFAIR baremetal-inventory is bifrost specific (ansible), while baremetal-nodes is ironic specific | 16:08 |
| JayF | ah | 16:09 |
| rpittau | TheJulia: it happens in CI when installing and unpacking all the packages for the image, not sure about the target device | 16:11 |
| iurygregory | JayF, finishing lunch (give me 30min) | 16:13 |
| JayF | ack it's not a rush julia set me straight | 16:14 |
| rpittau | alright I can't function properly anymore, going offline | 16:17 |
| rpittau | if anyone is curious the failing job is in this patch https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/879538 | 16:17 |
| rpittau | good night, see you tomorrow! o/ | 16:17 |
| TheJulia | rpittau: ... I guess to grok what is going on , in post-steps we need to get a df command run | 16:43 |
| TheJulia | like we do with logging | 16:43 |
| TheJulia | ... it wouldn't just be a bad thing to have *anyway* | 16:43 |
| TheJulia | adam_metal3: appologies, I'm getting pulled in a number of directions | 16:43 |
| * TheJulia started the morning by getting to write a very long and verbose email | 16:44 | |
| iurygregory | JayF, I'm back | 16:44 |
| JayF | iurygregory: lets just keep our meeting in ~1h:15m | 16:44 |
| iurygregory | JayF, ok o/ | 16:45 |
| JayF | iurygregory: I'm polishing off the last bits, and julia got my issue fixed | 16:45 |
| adam_metal3 | TheJulia: no problem | 16:45 |
| JayF | issue being, I didn't know to run enroll LOL | 16:45 |
| iurygregory | ops =) | 16:45 |
| iurygregory | we have a script for it | 16:45 |
| TheJulia | for the record, we *really* need to just carry a GPIO pin coffee driver or something | 16:45 |
| JayF | I need composable hardware | 16:45 |
| JayF | so I can hook some external storage onto my brain | 16:46 |
| iurygregory | composable coffee | 16:46 |
| TheJulia | do demos in a conference hallway making us coffee | 16:46 |
| TheJulia | "deploy me coffee" | 16:46 |
| JayF | meta conference talk about how all conference demos are contrived | 16:46 |
| JayF | demo like 100 crazy ways to make coffee | 16:46 |
| TheJulia | lolz | 16:46 |
| JayF | the reveal at the end is some person under the table actually mashing buttons to make it work | 16:46 |
| TheJulia | can we have a rack of servers power up and power off through this? | 16:47 |
| JayF | no, but we can have christmas lights in a rack-like-structure flash on and off ;) | 16:47 |
| TheJulia | oh no, we need the lights to be triggered by *at least* relays | 16:47 |
| JayF | once I actually run all the right commands | 17:01 |
| JayF | bifrost is super cool | 17:01 |
| JayF | especially for places that feed nodes provisioned by ironic into ansible anyway | 17:01 |
| JayF | I could see a fully formed workflow all in custom ansible calling out to bifrost for provisioning | 17:02 |
| TheJulia | One of the reason we carry examplish playbooks which can be used/modified | 17:02 |
| TheJulia | :) | 17:02 |
| * TheJulia is having one of those days where forward progress just seems like it is not going to happen today | 17:37 | |
| TheJulia | adam_metal3: I think it would help to get exactly what glean is erroring out with, I am getting a few different things with a mock test I've got half setup, but clean at least seems to think it is doing the right thing | 18:17 |
| TheJulia | adam_metal3: one thing I *am* seeing, is the json being shipped in has a slight linting error, I wouldn't expect that to throw it, but maybe it could be?! | 18:18 |
| TheJulia | tailing commas shouldn't exist if there is not a next element, right after the actual bond link definition | 18:18 |
| adam_metal3 | TheJulia: I linted it with hand as I got it MSTeams and that messed up the linting | 18:19 |
| adam_metal3 | in MSTeams* | 18:19 |
| adam_metal3 | I have not added any coma though, I will check | 18:20 |
| adam_metal3 | ohh you mean the last link, yeah that was me I have removed a link that was not used in later tests | 18:22 |
| adam_metal3 | I have removed it from the pastebin, that is not part of our input (that extra coma) | 18:23 |
| opendevreview | Maksim Malchuk proposed openstack/bifrost master: Remove extra symbols accidentally added https://review.opendev.org/c/openstack/bifrost/+/879547 | 18:23 |
| adam_metal3 | I will type the errors down I only have pictures | 18:26 |
| adam_metal3 | so I theorize that this is where the error is coming from: https://github.com/Nordix/glean/blob/528f7216c64b459c2028a2ba81149548862b58b5/glean/cmd.py#L318 and the error message is just ERROR: 'the mac of the other link' so if I check the mac ending with 11 then it is like ERROR: 'mac ending with 10' and vice versa | 18:29 |
| adam_metal3 | hm I have images of console outputs I will try an image reader to generate some usable text | 18:31 |
| TheJulia | adam_metal3: ack | 18:39 |
| TheJulia | adam_metal3: I seem to encoutner that if the mac being put in doesn't exist on the host | 18:40 |
| adam_metal3 | Dowsntream sent me the output of the ip link command they have the macs | 18:44 |
| TheJulia | glean seems to error there if it can't find the macs when it walks /sys | 18:45 |
| TheJulia | to be precise /sys/class/net/* | 18:45 |
| TheJulia | In part because if memory serves glean collects what it needs on the addresses from there to map to interface names, the interface names in your payload are just for humans to mentally piece it together | 18:46 |
| adam_metal3 | I have extended the pastebin post with an example of glean service output | 18:49 |
| adam_metal3 | it finds the first mac and drops the error for the second | 18:49 |
| opendevreview | Jay Faulkner proposed openstack/ironic-specs master: Add 2023.2 Workitems discussed at Ironic PTG https://review.opendev.org/c/openstack/ironic-specs/+/879381 | 18:49 |
| TheJulia | adam_metal3: I'd ask them to cd /sys/class/net and do something like ; for dev in `ls`; do echo device $dev ; cat $dev/address ; done just to verify the kernel sees it and all | 18:55 |
| TheJulia | (and verify the address matches what is expected) | 18:56 |
| adam_metal3 | sure thanks, I will create a test setup and try it on a vm, my downstream folks won't be available until Tuesday so I will try to reproduce this on a VM, thanks for all the help | 18:59 |
| TheJulia | adam_metal3: I was thinking the same exact thing | 19:05 |
| -opendevstatus- NOTICE: The Gerrit service on review.opendev.org will be offline for extended periods between 22:00 and 00:00 UTC for software upgrades and project renames: https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/thread/VW2O56AXI4OX34CWDNRNZDCWJDZR3QJP/ | 21:04 | |
| -opendevstatus- NOTICE: The Gerrit service on review.opendev.org will be offline for extended periods over the next two hours for software upgrades and project renames: https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/thread/VW2O56AXI4OX34CWDNRNZDCWJDZR3QJP/ | 21:59 | |
| JayF | Reminder: tomorrow is the last day I'll be working, or in IRC, likely for over a calendar week. If you need something from me act quickly :) | 22:50 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!