| opendevreview | Julia Kreger proposed openstack/sushy master: WIP Fix session authentication issues https://review.opendev.org/c/openstack/sushy/+/821324 | 00:14 |
|---|---|---|
| TheJulia | ajya: I've revised https://review.opendev.org/c/openstack/sushy/+/821324 to be more in line with what you were thinking in terms of early session creation. I'm not exactly thrilled with some of it with set_context on direct invocation, but... it was that or re-do like 23 unit tests. Maybe worth still going that route. LMK if the session creation is more in line with what you were thinking and I *think* that will short | 00:16 |
| TheJulia | circuit a number of potential failure cases with re-authing, but we'll still want to guard and fallback I guess. | 00:16 |
| opendevreview | Vanou Ishii proposed openstack/ironic master: [WIP] Deal with iRMC virtual media incompatibility https://review.opendev.org/c/openstack/ironic/+/823790 | 00:52 |
| opendevreview | Verification of a change to openstack/ironic master failed: Use driver_internal_info methods for redfish driver https://review.opendev.org/c/openstack/ironic/+/818508 | 01:01 |
| opendevreview | Verification of a change to openstack/ironic master failed: Use driver_internal_info methods for redfish driver https://review.opendev.org/c/openstack/ironic/+/818508 | 06:22 |
| arne_wiebalck | Good morning, Ironic! | 07:18 |
| janders | good morning arne_wiebalck o/ | 08:28 |
| arne_wiebalck | hey janders, good morning! o/ | 08:32 |
| rpittau | good morning ironic! o/ | 08:35 |
| arne_wiebalck | hey rpittau o/ | 08:37 |
| rpittau | hey arne_wiebalck :) | 08:37 |
| rpittau | new nodeset definition is working as expected, if anyone has a minute please review https://review.opendev.org/c/openstack/bifrost/+/819058 thanks! | 08:42 |
| dtantsur | stevebaker[m]: this is a concern on my side. I'd prefer not to bring in the whole privsep mechanism just for a convenient feature that duplicates the operator's tooling | 09:02 |
| dtantsur | and morning ironic | 09:02 |
| stevebaker[m] | Privsep only runs on startup, I've got it gracefully shutting down when copy operations have finished | 09:04 |
| dtantsur | stevebaker[m]: on start-up or on first usage? this makes a huge difference | 09:06 |
| dtantsur | i.e. if I don't use the copying feature, can I avoid it completely? | 09:06 |
| dtantsur | (also wow, a rare chance to meet you online!) | 09:06 |
| stevebaker[m] | Yes, starts on demand, ends at the end of start-up. | 09:07 |
| stevebaker[m] | (matrix client on my phone :) | 09:07 |
| dtantsur | ha, nice :) | 09:07 |
| dtantsur | stevebaker[m]: I guess this is the part that confuses me: we're making the operator's life easier by copying files, but making it harder by requiring them to configure privsep :) | 09:08 |
| dtantsur | I honestly wonder why openstack doesn't just 'sudo' (and require the operators to configure sudoers) | 09:09 |
| stevebaker[m] | It's just one rootwrap entry, installed by deb/rpm (ok, bifrost needs it's own). Which is better than puppet/ansible/tripleo doing a bunch of distro specific copying | 09:15 |
| dmellado | o/ Happy new year y'all | 09:17 |
| dmellado | so... this was a laptop.. and so | 09:17 |
| dmellado | xD | 09:17 |
| dtantsur | stevebaker[m]: well, bifrost doesn't need the copying feature, so I'd rather keep it without privileged operations | 09:20 |
| dtantsur | happy new year and hi, dmellado | 09:20 |
| dtantsur | dmellado: in case you missed the gerrit notification: I've done a round of updates to the bifrost's "how to" docs | 09:20 |
| dmellado | just checking my email | 09:21 |
| dmellado | let me take a look | 09:21 |
| stevebaker[m] | The bifrost changes are up for review already, resulting in working grub network boot | 09:23 |
| dtantsur | stevebaker[m]: good for testing, but I'm basically -2 to adding privileged operations to bifrost's ironic, given that bifrost is literally playbooks to install ironic | 09:24 |
| dtantsur | (I feel less strongly about ironic itself since I recognize the case of "can I just launch /usr/bin/ironic") | 09:25 |
| stevebaker[m] | Ok, put your thoughts in the reviews when you get the chance. Let's discuss there | 09:28 |
| stevebaker[m] | Night all | 09:28 |
| dtantsur | good night stevebaker[m] | 09:28 |
| dmellado | dtantsur: ++ on the changes | 09:32 |
| dmellado | already merged anyway | 09:32 |
| opendevreview | Verification of a change to openstack/ironic master failed: Ensure 'port' is up2date after binding:host_id https://review.opendev.org/c/openstack/ironic/+/823811 | 09:34 |
| dmellado | rpittau: so, when I was younger and saner, TL;DR, before xmas | 09:34 |
| dmellado | you told me about adding a user to the ipa image, mind remembering me? | 09:34 |
| rpittau | dmellado: hi! happy new year! if you use DIB to create the image you can add ssh access as explained here https://docs.openstack.org/ironic-python-agent-builder/latest/admin/dib.html#ssh-access | 09:40 |
| rpittau | the devuser element in particular allows to login from console using a local user | 09:41 |
| dmellado | o/ checking ;) | 09:43 |
| dmellado | is there any straighforward way to put that on the bifrost install phase? | 09:45 |
| dtantsur | dmellado: I think there is an option for a list of elements | 09:45 |
| dtantsur | (documenting that would be awesome too) | 09:45 |
| dmellado | oh, dib_elements | 09:46 |
| dmellado | just grepped it | 09:46 |
| dmellado | would submit a patch for that later as well | 09:46 |
| dmellado | dib_elements: "ironic-python-agent-ramdisk {{ ipa_extra_dib_elements | default('') }}" | 09:46 |
| dmellado | so adding ipa_extra_dib_elements should work | 09:47 |
| dtantsur | yep | 09:48 |
| rpittau | documenting a thing that I extensively use and it's super useful? yeah, might be a good idea :D | 09:53 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Change the TFTP directory to /var/lib/tftpboot https://review.opendev.org/c/openstack/bifrost/+/823552 | 10:01 |
| dmellado | I assume I would also need create_ipa_image to be set to true | 10:02 |
| dmellado | on target | 10:02 |
| dtantsur | le sigh: https://bugzilla.redhat.com/show_bug.cgi?id=2039224 | 10:04 |
| rpittau | oh that's why... ;/ | 10:05 |
| dmellado | sighh | 10:05 |
| rpittau | dmellado: yes you need to recreate the ipa image | 10:06 |
| opendevreview | Merged openstack/ironic master: Use driver_internal_info methods for redfish driver https://review.opendev.org/c/openstack/ironic/+/818508 | 10:13 |
| opendevreview | chandan kumar proposed openstack/ironic-python-agent-builder stable/wallaby: [DNM] testing dib-centos9 experimental job https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/824136 | 10:22 |
| opendevreview | Merged openstack/ironic master: Enable foreign keys in SQLite https://review.opendev.org/c/openstack/ironic/+/821219 | 10:36 |
| arne_wiebalck | after the upgrade to wallaby, it seems pagination is broken for me: '{"code": 404, "title": "Not Found", "description": ""} (HTTP 404)' ... does that ring a bell for anyone? | 10:37 |
| arne_wiebalck | with '--limit' it breaks exactly at the max_limit, so ''--limit=1000" works, "--limit 1001" breaks | 10:38 |
| dtantsur | Oo | 10:53 |
| dtantsur | a regression in the wsme migration? | 10:54 |
| dmellado | hmmm | 10:59 |
| dmellado | why does it try to crete the ipa image with debian? | 10:59 |
| dmellado | wasn't that centos based? | 10:59 |
| dtantsur | dmellado: that's teh default for bifrost | 10:59 |
| dmellado | "_raw_params": "disk-image-create -o /var/lib/httpboot/ipa debian ironic-python-agent-ramdisk devuser\n" | 11:00 |
| dtantsur | I know, consistency, but bifrost's DIB code pre-dates our published images *by far* | 11:00 |
| dmellado | I see, I'll modify that in the role | 11:00 |
| dtantsur | there is a var for that | 11:00 |
| dmellado | oh, which one' | 11:00 |
| dmellado | I have | 11:01 |
| dmellado | dib_os_element: centos | 11:01 |
| dmellado | 11:01 | |
| dmellado | dib_os_release: 8-stream | 11:01 |
| dmellado | 11:01 | |
| dmellado | on the target | 11:01 |
| dmellado | but doesn't seem to be used for the ipa, maybe an specific one? | 11:01 |
| dtantsur | hmmm, these ones should affect it | 11:02 |
| dtantsur | dmellado: ah, we have ipa_dib_os_element in the playbook | 11:03 |
| dtantsur | yay for confusion | 11:03 |
| rpittau | yep, that | 11:03 |
| dtantsur | but we don't have ipa_dib_os_release because consistency :) | 11:03 |
| rpittau | always overrated | 11:04 |
| dtantsur | this all needs cleaning up and documenting | 11:04 |
| dmellado | +1 to that xD | 11:04 |
| rpittau | I need to review my notes from the debian issue testing I got so far | 11:04 |
| dmellado | I guess I may be the first one to try this let's say 'from scratch' in a while xD | 11:04 |
| dtantsur | it's not impossible. or maybe people just don't report their problem, it's also quite likely. | 11:05 |
| rpittau | I did that before holidays, but I think I lost most of the notes because of VM going foobar | 11:05 |
| dtantsur | this is why we particularly appreciate people who can provide sensible reports :) | 11:05 |
| rpittau | there's an issue with network interfaces not starting when debian boots | 11:06 |
| rpittau | I think the renaming of the interfaces interfere with that | 11:06 |
| dmellado | oh, I've seen that issue on renaming with 'new' kernels in a different scope | 11:06 |
| dmellado | a pain | 11:07 |
| rpittau | yeah | 11:07 |
| rpittau | so dhcp do not activate them | 11:07 |
| arne_wiebalck | dtantsur: pagination, seems like the server on the 2nd chunk gives 'None' instead of 'nodes' in the REQ | 11:10 |
| arne_wiebalck | dtantsur: the client takes that and gets a 404 | 11:10 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Tighted permissions across the board https://review.opendev.org/c/openstack/bifrost/+/824144 | 11:24 |
| dtantsur | arne_wiebalck: sounds like a bug to me | 11:24 |
| arne_wiebalck | dtantsur: https://opendev.org/openstack/ironic/src/branch/stable/wallaby/ironic/api/controllers/v1/node.py#L2074 is missing a resource_url parameter | 11:25 |
| arne_wiebalck | so, it falls back to None | 11:25 |
| iurygregory | good morning Ironic o/ | 11:25 |
| dtantsur | morning iurygregory | 11:26 |
| iurygregory | dtantsur, o/ | 11:26 |
| arne_wiebalck | dtantsur: adding "resource_url='nodes', " to the parameter list fixes it | 11:27 |
| arne_wiebalck | dtantsur: *hot-fixes it | 11:27 |
| arne_wiebalck | dtantsur: not sure this is the correct way to address this | 11:27 |
| dtantsur | doesn't sound immediately wrong to me | 11:31 |
| arne_wiebalck | dtantsur: the only other call to get_nodes_collections sets it before the call | 11:32 |
| janders | See you tomorrow Ironic o/ | 11:32 |
| arne_wiebalck | dtantsur: https://opendev.org/openstack/ironic/src/branch/stable/wallaby/ironic/api/controllers/v1/node.py#L2162 | 11:33 |
| dmellado | rpittau: sigh, forgot dib_env_vars | 11:44 |
| dmellado | lol | 11:44 |
| dmellado | maybe that'd be a good thing to add to the docs as well | 11:44 |
| dmellado | for the dib_dev ssh key and so | 11:44 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Tighted permissions across the board https://review.opendev.org/c/openstack/bifrost/+/824144 | 12:10 |
| dmellado | dtantsur: is there a way to set http_proxy on the ipa? | 12:13 |
| dtantsur | dmellado: when building or when actually running | 12:14 |
| dtantsur | ? | 12:14 |
| dmellado | ideally when building, I know using that dev_user I could just log in and set it up manually | 12:14 |
| dmellado | but I'd rather avoid that | 12:14 |
| dtantsur | I mean, when the proxies should take effect? | 12:15 |
| dmellado | when downloading the deployment image | 12:15 |
| dmellado | I found the issue there xD | 12:15 |
| dtantsur | dmellado: I think #5 of https://docs.openstack.org/ironic/latest/admin/drivers/ipa.html#using-proxies-for-image-download | 12:16 |
| dtantsur | we may need to make it more prominent in the docs | 12:16 |
| dmellado | yeah, 5 looks like the way to go | 12:16 |
| dtantsur | dmellado: if you could copy the parameters to https://docs.openstack.org/ironic/latest/user/deploy.html#populating-instance-information, would be really appreciated | 12:17 |
| dmellado | will do as soon as I try it | 12:17 |
| arne_wiebalck | dtantsur: I am rolling out an updated package to confirm this works, then I'll submit a patch. | 12:22 |
| arne_wiebalck | dtantsur: (for the pagination issue) | 12:22 |
| dmellado | hmm so some misconfiguration or so it seems from my side | 12:28 |
| dmellado | ipa it's trying to get the image from 192.168.122.1 which doesn't work | 12:28 |
| dmellado | but it gets it directly from the ip from the node | 12:28 |
| dmellado | 192.168.28.9 | 12:28 |
| dmellado | 192.168.207.9 | 12:29 |
| dmellado | deployment_image.qcow2, I mean | 12:29 |
| dmellado | Jan 11 12:04:29 localhost.localdomain ironic-python-agent[1641]: 2022-01-11 12:04:29.809 1641 INFO ironic_python_agent.extensions.standby [-] Attempting to download image from http://192.168.122.1:8082/deployment_image.qcow2 | 12:32 |
| dmellado | Jan 11 12:05:29 localhost.localdomain ironic-python-agent[1641]: 2022-01-11 12:05:29.871 1641 WARNING ironic_python_agent.extensions.standby [-] Unable to connect to http://192.168.122.1:8082/deployment_image.qcow2, retrying. Error: HTTPConnectionPool(host='192.168.122.1', port=8082): Max retries exceeded with url: /deployment_image.qcow2 (Caused by | 12:32 |
| dmellado | ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7effa2de> | 12:32 |
| dmellado | changing the ip to the latter one, and I can get it | 12:32 |
| dmellado | probably internal_ip | 12:42 |
| dmellado | https://paste.opendev.org/show/bmaw8ZsWU3GxF3UIRAjL/ | 12:50 |
| dmellado | so, how can I change there | 12:50 |
| dmellado | image_source and config_drive ips | 12:50 |
| dmellado | ? | 12:50 |
| dmellado | image_source: "{{ deploy_url_protocol }}://{{ internal_ip }}:{{ file_url_port }}/{{ deploy_image_filename }}" | 12:51 |
| dmellado | I set internal_ip to the proper one on the group_vars but it doesn't look to be fetched | 12:51 |
| dtantsur | dmellado: group_vars for what? | 12:52 |
| dtantsur | this has to apply to baremetal, not target | 12:53 |
| dmellado | oh, baremetal, got it | 12:53 |
| dmellado | yeah | 12:53 |
| dtantsur | we should probably allow an explicit override to avoid changing internal_ip (which has many meanings) | 12:54 |
| dmellado | ++ | 12:54 |
| dtantsur | actually, we have public_ip that could be used | 12:55 |
| dmellado | but hey I learned on how to ssh into the ipa xD | 12:55 |
| dmellado | If you're up for a shared session on all these hitting the wall things | 12:56 |
| dmellado | for the 'crossing fingers' | 12:56 |
| dmellado | opendev in berlin | 12:56 |
| dmellado | I'll be more than happy xD | 12:56 |
| dtantsur | opendev in berlin, mmmm... | 12:57 |
| dtantsur | assuming travelling is a common thing again | 12:57 |
| dmellado | assumming that, yeah | 12:58 |
| dmellado | maybe it's too much to assume | 12:58 |
| dmellado | given the current status... | 12:58 |
| dtantsur | I still hope :( | 12:58 |
| dmellado | it'd be great, yeah... | 13:02 |
| dmellado | I swear I'm growing tired of the whole covid thing.... | 13:02 |
| dmellado | and I had a massive mess | 13:02 |
| dmellado | as they tested me as a false positive | 13:02 |
| dtantsur | ouch :( | 13:31 |
| dtantsur | well, at least it was false | 13:31 |
| dmellado | yeah, but the dumb helath authorities are calling me nonstop | 13:31 |
| dmellado | seems like they're not in sync with each other | 13:31 |
| dmellado | hmmm now configdrive it's not there? | 13:32 |
| dmellado | d'oh | 13:32 |
| admiyo | dtantsur, I've been chasing the iPXE thing since we chatted last week. I don't get what changed? I'm assuming this is the deploy thing you were talking about https://docs.openstack.org/ironic/latest/admin/interfaces/deploy.html but what happened to "Fetch your image from glance" that we were doing before? If I enable iPXE in kolla, it messes up a bunch of stuff, and I really would rather not replace tftp with http right | 13:36 |
| admiyo | now. | 13:36 |
| admiyo | do we want --deploy-interface ramdisk ? | 13:42 |
| dtantsur | admiyo: Ironic works by serving the image to its ramdisk via HTTP through its HTTP server. If you don't have the HTTP server, well, the image cannot be served. | 13:42 |
| dtantsur | Please don't experiment with the options you don't understand. It will just confuse you more. | 13:42 |
| admiyo | dtantsur, I understand PXE. I don't understand Ironic. | 13:43 |
| admiyo | We had something that worked, and the world moved on. | 13:43 |
| dtantsur | I was referring to --deploy-interface, sorry | 13:43 |
| dtantsur | If enabling the HTTP server in Kolla "messes up a bunch of stuff", please reach out to #openstack-kolla, it should not be happening | 13:43 |
| dtantsur | cc mgoddard | 13:43 |
| admiyo | However, I am at the mercy of the technologies available, and that means Kolla. | 13:43 |
| admiyo | Well, I suspect that what has happened is that we are no longer really testing tftp based deploys that are then set up via Kolla | 13:44 |
| admiyo | I fully expect the Kolla folks to kick me back here. | 13:44 |
| dtantsur | This team is not responsible for anything Kolla delivers. I'm trying my best, but my knowledge is very limited. | 13:44 |
| admiyo | You are saygin that 100% now Ironic needs an internal http server beyond the existing API servers | 13:45 |
| admiyo | existing as of my ancient install | 13:45 |
| dtantsur | yes, absolutely | 13:45 |
| dtantsur | I would also recommend against using TFTP unless you hit issues with iPXE, but that's another topic | 13:45 |
| admiyo | the Kolla way to set that up appears to be en enable ipxe, but that then changes the Ironic boot process | 13:45 |
| admiyo | i.e. if i enable the ipxe server, the cleaning process no longer works. | 13:46 |
| admiyo | Which, to me says that a DHCP delivered option has changed or something | 13:46 |
| admiyo | I still get Grub, so something works, and I can wireshark it to figure out where it is getting that from | 13:46 |
| dmellado | dtantsur: "Deploy step deploy.write_image failed on node ef4bbe4a-3ecd-4bae-888e-52c406707206. Can't download the configdrive content for node ef4bbe4a-3ecd-4bae-888e-52c406707206 from 'http://192.168.207.9:8082/configdrive-ef4bbe4a-3ecd-4bae-888e-52c406707206.iso.gz'. Got status code 404, response body <html>" | 13:47 |
| dmellado | that configdrive shouls show up at /httpboot | 13:47 |
| dmellado | IIUC, isn't it? | 13:47 |
| dtantsur | admiyo: they do, indeed, modify the default boot file to use iPXE: https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/ironic/defaults/main.yml#L257 | 13:47 |
| dtantsur | dmellado: can you curl this literal thing? | 13:48 |
| dmellado | I can't | 13:48 |
| dmellado | that's why I'm asking | 13:48 |
| dmellado | even from the host, not just ipa | 13:49 |
| opendevreview | Verification of a change to openstack/ironic master failed: Ensure 'port' is up2date after binding:host_id https://review.opendev.org/c/openstack/ironic/+/823811 | 13:49 |
| dtantsur | dmellado: wrong port? firewall? | 13:49 |
| dmellado | nope, just not there | 13:49 |
| dtantsur | the file? | 13:50 |
| dmellado | https://paste.opendev.org/show/812027/ | 13:50 |
| dmellado | yeah, that's the content of my httpboot | 13:50 |
| dmellado | so not sure on when that configrive is generated | 13:50 |
| dmellado | maybe I'll just have to reinstall | 13:51 |
| * dtantsur pings #openstack-kolla re testing | 13:51 | |
| dtantsur | dmellado: the deployment playbooks includes a role called bifrost-configdrive-dynamic | 13:52 |
| dtantsur | which is supposed to do that | 13:52 |
| dtantsur | you've changed the httpboot location, haven't you? | 13:52 |
| dmellado | I have | 13:52 |
| dmellado | a-ha | 13:53 |
| dmellado | so it's on the old location | 13:53 |
| dmellado | awesome xD | 13:53 |
| dtantsur | dmellado: you need to pass the same parameter everywhere | 13:53 |
| dmellado | so I have to change httpboot location in baremetal settings | 13:53 |
| dtantsur | dmellado: or just update your bifrost: my change has merged | 13:53 |
| dmellado | rebasing then | 13:55 |
| dmellado | tnx! | 13:55 |
| dtantsur | admiyo: will applying https://opendev.org/openstack/kolla-ansible/commit/411668ea5a05e8182d80be63501e2f7fa1882345 fix your problem? | 14:06 |
| admiyo | Well, no, I think. I enabled that explicitily, and it changes the DHCP options, like you said, so that would just hide what is breaking things on me, I think. I can enable ipxe already, but that does not seem to skip the rest of the work I have to do....let me read further and confirm | 14:08 |
| Dhuldev_Valekar | iurygregory:Hi Gregory, Happy New Year 2022!, While running existing ironic tempest test case "test_introspection_basic" against the dell servers we are facing assertion failure while comparing memory properties of actual servers against the baremetal flavor values post inspection. any suggestions? | 14:08 |
| dtantsur | admiyo: I mean, you can apply this patch and keep ipxe disabled | 14:08 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Follow up to "Run bifrost on CentOS Stream 9" https://review.opendev.org/c/openstack/bifrost/+/824186 | 14:08 |
| iurygregory | Dhuldev_Valekar, do you have a link for the failures? | 14:09 |
| admiyo | Basically, if I read this correctly, the set of changes have made sure that you cannot deploy Ironic with tftp via Kolla. | 14:09 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Follow up to "Run bifrost on CentOS Stream 9" https://review.opendev.org/c/openstack/bifrost/+/824186 | 14:09 |
| admiyo | and, if that is the case, I can deal with it. | 14:09 |
| admiyo | "Note that due to a limitation in Kolla Ansible, PXE and iPXE cannot be used | 14:10 |
| admiyo | together in a single deployment." | 14:10 |
| dtantsur | admiyo: I don't understand how you're making this conclusion. The kolla team has simply missed the fact that Ironic changed its defaults in Wallaby, so they fixed the issue in Xena+. | 14:10 |
| dtantsur | yes, this is correct. but it does not mean you cannot use PXE. | 14:10 |
| dtantsur | the patch I linked will enable the HTTP container without changing the boot process | 14:11 |
| admiyo | dtantsur, it means that I would have to set up my own HTTP server, as the one that Kolla provides does too much | 14:11 |
| admiyo | Not how I read it. | 14:11 |
| dtantsur | I don't understand how you derive "PXE cannot be used" from "PXE and iPXE cannot be used together" | 14:11 |
| mgoddard | admiyo: hi | 14:12 |
| Dhuldev_Valekar | iurygregory: we are running tempest test case on local setup, I can provide log file separately, would that helpful? | 14:12 |
| mgoddard | you are right, that currently the HTTP server is tied to iPXE, which it didn't necessarily have to be, since it's required for the direct deploy interface | 14:12 |
| dtantsur | mgoddard: I think https://opendev.org/openstack/kolla-ansible/commit/411668ea5a05e8182d80be63501e2f7fa1882345 should be in wallaby as well because that's when we changed the default deploy interface | 14:13 |
| mgoddard | dtantsur: that probably makes sense | 14:13 |
| admiyo | Heyo. I am OK with moving to HTTP from tftp if that is the case. I'd rather not apply a Xena patch to Wallaby, when we went with Wallaby in the first place to avoid the chaotic frindge of the bleeding edge! | 14:13 |
| opendevreview | Aija Jauntēva proposed openstack/ironic master: Fix validating input for redfish update_firmware https://review.opendev.org/c/openstack/ironic/+/823701 | 14:14 |
| iurygregory | Dhuldev_Valekar, would be good since the test_introspection_basic has a few tests | 14:14 |
| mgoddard | admiyo: ipxe is the way | 14:15 |
| mgoddard | dtantsur, admiyo: we probably did not see this in kayobe because we (regrettably) provide our own defaults for ironic drivers, and iscsi stayed the default | 14:17 |
| mgoddard | anyhow, here's the backport: https://review.opendev.org/c/openstack/kolla-ansible/+/824155 | 14:17 |
| admiyo | Is iSCSI still there? | 14:18 |
| admiyo | which value is it that changed the default? | 14:20 |
| mgoddard | iscsi is still available in wallaby, just no longer the default | 14:20 |
| admiyo | what value am I looking for? | 14:21 |
| admiyo | ther [iscsi] section does not even exist in the xena ironic default file. | 14:22 |
| mgoddard | default_deploy_interface | 14:22 |
| admiyo | #default_deploy_interface = <None> | 14:22 |
| admiyo | that is from the wallaby default config. | 14:23 |
| admiyo | #default_deploy_interface = iscsi | 14:23 |
| admiyo | we have it commented out, so I should uncomment it | 14:23 |
| * TheJulia wipes sleep from her eyes | 14:25 | |
| TheJulia | Good morning everyone | 14:25 |
| jamesdenton | good morning | 14:25 |
| mgoddard | admiyo: if you use iscsi, you won't need the http server | 14:26 |
| mgoddard | (assuming you continue to use pxe) | 14:26 |
| iurygregory | Dhuldev_Valekar, try checking your tempest conf if the reference for the ONF.compute.flavor_ref matches the flavor you created for your node etc | 14:27 |
| opendevreview | Arne Wiebalck proposed openstack/ironic master: Set resource_url when getting all nodes https://review.opendev.org/c/openstack/ironic/+/824190 | 14:29 |
| arne_wiebalck | dtantsur: ^^ (I did not yet manage to spot the difference to victoria, so some pieces of the picture are missing) | 14:31 |
| Dhuldev_Valekar | iurygregory: https://paste.openstack.org/raw/812032/ | 14:33 |
| arne_wiebalck | dtantsur: there must be some fundamental difference since the API request times have gone down by at least a factor of 5 https://usercontent.irccloud-cdn.com/file/0V0kfRys/wallaby_pagination_speedup.png | 14:33 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Tighten permissions across the board https://review.opendev.org/c/openstack/bifrost/+/824144 | 14:34 |
| dtantsur | arne_wiebalck: well, TheJulia has done a lot of work in this area | 14:34 |
| TheJulia | arne_wiebalck: is that live from your upgrade today? | 14:35 |
| arne_wiebalck | TheJulia: it is | 14:35 |
| TheJulia | arne_wiebalck: that makes me soooo happy | 14:35 |
| TheJulia | arne_wiebalck: blog post? | 14:35 |
| arne_wiebalck | TheJulia: see, we had 3 bands in req times before (1000 nodes, rest of groups, other calls) ... they seem to be gone :) | 14:36 |
| arne_wiebalck | TheJulia: this looks like a massive improvement! | 14:36 |
| arne_wiebalck | TheJulia: thanks for your work! :) | 14:37 |
| * dtantsur dances | 14:37 | |
| TheJulia | arne_wiebalck: has the sporatic grouping after 11:00 and before 12:00 and again around 14:00 been identified yet? | 14:37 |
| arne_wiebalck | TheJulia: the upgrade was much longer than expected due to https://review.opendev.org/c/openstack/ironic/+/824190 | 14:38 |
| iurygregory | Dhuldev_Valekar, in your setup you have nova running ? Can you share the tempest.conf? | 14:38 |
| arne_wiebalck | TheJulia: so, the upgrade did not finish at 11, but only at 2pm or so | 14:38 |
| arne_wiebalck | TheJulia: and then I restarted everything to randomise the startup times | 14:38 |
| dtantsur | arne_wiebalck: could you find a way to unit-test your fix? | 14:39 |
| dtantsur | it's so easy to regress there | 14:39 |
| arne_wiebalck | dtantsur: yeah, I was expecting that question :-D | 14:39 |
| TheJulia | I'm surprised we didn't catch that but I don't remember how I did pagination testing anymore | 14:39 |
| arne_wiebalck | dtantsur: tried to sneak through :-D | 14:39 |
| opendevreview | Aija Jauntēva proposed openstack/ironic master: Add more sources to redfish firmware upgrade https://review.opendev.org/c/openstack/ironic/+/822781 | 14:39 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Add CentOS Stream 9 keystone integration job https://review.opendev.org/c/openstack/bifrost/+/824193 | 14:40 |
| arne_wiebalck | TheJulia: I don't understand the difference to Victoria (and why noone else sees this ... unless we are the only with Nova, a max_limit which is not 0 and on Wallaby+) | 14:40 |
| opendevreview | Aija Jauntēva proposed openstack/ironic master: Add known issue for iDRAC Swift firmware update https://review.opendev.org/c/openstack/ironic/+/822782 | 14:41 |
| TheJulia | you I'm surprised you even found it and I've not completely wrapped my head around it | 14:42 |
| arne_wiebalck | TheJulia: non-working pagination breaks nova immediately | 14:42 |
| TheJulia | but you are likely and early adopter being one of the larger deployments to move faster | 14:43 |
| arne_wiebalck | TheJulia: which is another thing which surprised me | 14:43 |
| TheJulia | I wonder if it is python-ironicclient vs openstacksdk | 14:43 |
| arne_wiebalck | clients have not changed | 14:43 |
| TheJulia | maybe we've got pagination implemented differently | 14:43 |
| TheJulia | did you lookup when they swapped the node list out to openstacksdk? | 14:43 |
| Dhuldev_Valekar | iurygregory : tempest.conf https://paste.openstack.org/raw/812033/ | 14:43 |
| arne_wiebalck | TheJulia: no ... | 14:44 |
| admiyo | mgoddard, I assume I am going to need all of the [iscsi] section now too? | 14:45 |
| iurygregory | Dhuldev_Valekar, if you check the specs for this flavor 83c4f114-ac7c-46a9-af3b-7aa28487a2f4 it will probably show 8192 so this would cause https://github.com/openstack/ironic-tempest-plugin/blob/master/ironic_tempest_plugin/tests/scenario/test_introspection_basic.py#L27-L28 to fail | 14:46 |
| dtantsur | admiyo, mgoddard, note that going back to iscsi pretty much sets you up for the same pain in the next release | 14:47 |
| TheJulia | arne_wiebalck: 3 years ago | 14:47 |
| arne_wiebalck | TheJulia: we are little behind, but not that much :) | 14:48 |
| TheJulia | arne_wiebalck: so, ussuri. I guess I focused on how python-ironicclient did it and didn't look at the sdk | 14:48 |
| mgoddard | admiyo: shouldn't | 14:49 |
| TheJulia | because I likely assumed it to be the same which was bad of me | 14:49 |
| Dhuldev_Valekar | iurygregory : Yes, in the flavor spec it is showing memory as 8192, but post introspection we are getting actual memory value which is 196608 and hence assertion is failing ( 196608 != 8192 ) | 14:50 |
| opendevreview | Aija Jauntēva proposed openstack/ironic master: Fix prepare ramdisk for 'wait' states https://review.opendev.org/c/openstack/ironic/+/823311 | 14:51 |
| iurygregory | the workaround I see would be create a flavor that matches the HW you want to use as a node and set in the tempest.conf | 14:52 |
| TheJulia | arne_wiebalck: commented on the patch with an idea on how to test it | 14:52 |
| arne_wiebalck | TheJulia: not sure I follow ... the sdk is missing some changes to deal with your improvements as you only did them in python-ironicclient? | 14:52 |
| TheJulia | arne_wiebalck: no, your change is because the link was wrong for the next iteration correct? | 14:53 |
| arne_wiebalck | TheJulia: correct | 14:53 |
| arne_wiebalck | TheJulia: the initial REQ is good, next is broken (has 'None') | 14:53 |
| TheJulia | so using the links reference where I don't think python-ironicclient actually uses the link | 14:54 |
| Dhuldev_Valekar | iurygregory : Ok, Thanks! | 14:54 |
| TheJulia | because the pattern is "last returned is the marker for next" | 14:54 |
| TheJulia | at least, that is what is in my brain | 14:54 |
| TheJulia | anyway, it is obvious we're lacking a test to ensure the next is returned properly | 14:56 |
| TheJulia | we need to fix that | 14:56 |
| admiyo | dtantsur, I want to get this working as is, so I have a production server I can modify, and then get a dev server set up that I can experiment with, including much cruft extraction | 14:56 |
| arne_wiebalck | TheJulia: oh, I see | 14:56 |
| admiyo | I'm thinking we go bifrost next, so I'm willing to break things | 14:56 |
| arne_wiebalck | TheJulia: thanks for your comment, will have a look later (meeting now) | 14:57 |
| TheJulia | arne_wiebalck: ack, okay | 14:57 |
| TheJulia | arne_wiebalck: may I tweet that image you shared? | 14:57 |
| dmellado | hmmm error by error xD | 14:58 |
| dmellado | Deploy step deploy.prepare_instance_boot failed: Failed to install a bootloader when deploying node ef4bbe4a-3ecd-4bae-888e-52c406707206. Error: No partition with UUID 0x86d06aeb found on device /dev/sda | 14:58 |
| dmellado | I assume that it failed to partition the disk somehow? | 14:58 |
| arne_wiebalck | TheJulia: we will restart nova-compute as well, just to be sure ... I would wait another couple of hours so we do not have to correct ourselves :) | 14:59 |
| TheJulia | arne_wiebalck: sounds good | 15:00 |
| arne_wiebalck | TheJulia: but, then, yes sure; we can also add this as a comment to your performance blog posts if that fits somewhere | 15:00 |
| TheJulia | I'd just revisit it, tbh | 15:01 |
| dmellado | rpittau: that does ring a bell? I feel like 'almost there' xD | 15:04 |
| opendevreview | Riccardo Pittau proposed openstack/ironic bugfix/18.1: Explicitly pin upper sushy, reduce job count https://review.opendev.org/c/openstack/ironic/+/820927 | 15:05 |
| dtantsur | dmellado: what kind of the image you're using? I wonder if it's a breakage after mgoddard's patch to raise exceptions for whole disk images. | 15:14 |
| mgoddard | dtantsur: it could be | 15:15 |
| mgoddard | or it could be catching a legit error :) | 15:16 |
| mgoddard | e.g. using a non-UEFI image in UEFI mode | 15:16 |
| dtantsur | if so, the error messages leaves a lot to be desired :) | 15:17 |
| dtantsur | but good point re UEFI | 15:17 |
| *** dking is now known as Guest240 | 15:17 | |
| mgoddard | I used guestfish to verify the partition layout of the image | 15:18 |
| dtantsur | or virt-filesystems --long (which is the same thing in one CLI) | 15:18 |
| *** Guest240 is now known as dking | 15:18 | |
| admiyo | 2022-01-11 07:08:30.024 8 ERROR ironic.conductor.base_manager [req-b1634a6d-44bb-4236-bb22-089a567ac315 - - - - -] Failed to register hardware types. Could not find the following interface in the 'ironic.hardware.interfaces.deploy' entrypoint: iscsi. Valid interfaces are ['direct'].: ironic.common.exception.InterfaceNotFoundInEntrypoint: Could not find the following interface in the 'ironic.hardware.interfaces.deploy' | 15:28 |
| admiyo | entrypoint: iscsi. Valid interfaces are ['direct']. | 15:28 |
| admiyo | That is not very friendly | 15:28 |
| dtantsur | admiyo: if you have default_deploy_interface, you need to make sure it's in enabled_deploy_interfaces | 15:30 |
| admiyo | Ah, right... | 15:31 |
| admiyo | enabled_deploy_interface = iscsi,direct | 15:31 |
| admiyo | hrm | 15:31 |
| dmellado | dtantsur: I'm just using a centos one | 15:33 |
| dmellado | but I noticed that the disks are actually within a raid | 15:33 |
| dmellado | which I didn't take into consideration at all | 15:33 |
| dmellado | could that affect? | 15:33 |
| dmellado | I was reading https://docs.openstack.org/ironic/latest/admin/raid.html | 15:33 |
| dtantsur | dmellado: only if you don't have drivers for that in your IPA image | 15:34 |
| admiyo | dtantsur, any other way it could be disabled? | 15:34 |
| dtantsur | dmellado: try reverting https://opendev.org/openstack/ironic/commit/ca6499b62c47b6588b2382f442dbc690140453d5 and seeing if it helps | 15:34 |
| dtantsur | admiyo: are you looking in the resulting ironic.conf or somewhere in kolla? | 15:35 |
| dmellado | dtantsur: ack, so I owuldn't need to define the raid thing | 15:35 |
| opendevreview | Aija Jauntēva proposed openstack/ironic master: Fix Redfish RAID for non-immediate controllers https://review.opendev.org/c/openstack/ironic/+/823312 | 15:35 |
| dtantsur | dmellado: only if you want to actively manager RAID | 15:35 |
| admiyo | both | 15:35 |
| dking | Can anybody think of a reason why Ironic-API can give a 200 for a GET request for a node without any problem, including getting the power state, but when given a PUT for that same node for /states/power, it returns a 404? The node and its power state are in the database. | 15:35 |
| arne_wiebalck | TheJulia: nova for ironic is restarted, API times still look much improved compared to before the upgrade :) | 15:35 |
| dmellado | dtantsur: for the time being I couldn't care less | 15:35 |
| dmellado | dtantsur: how can I revert a commit in bifrost? Is there a way to 'protect' the repos under /opt/stack | 15:36 |
| dmellado | ? | 15:36 |
| dmellado | FWIW I assume it'll be recloned | 15:36 |
| dtantsur | dmellado: for the experiment of yours, just revert it in /opt/stack/ironic and do not re-install, just restart ironic-conductor | 15:37 |
| dmellado | ack | 15:37 |
| dtantsur | dking: sounds puzzling. sometimes 404 means a too low microversion, but likely not for /states/power | 15:37 |
| dtantsur | anything in the logs? | 15:37 |
| admiyo | I bet enabled_deploy_interface should be enabled_deploy_interfaces | 15:37 |
| * admiyo trys that | 15:37 | |
| dtantsur | admiyo: true! good catch | 15:37 |
| opendevreview | Verification of a change to openstack/ironic master failed: Ensure 'port' is up2date after binding:host_id https://review.opendev.org/c/openstack/ironic/+/823811 | 15:37 |
| dking | dtantsur: Thanks for the response. I just noticed that the DB had an error a while ago. I'm going to see if it's responding properly. However, if there were a problem, would the API be able to return information on GET? | 15:40 |
| dtantsur | dking: GET in our API normally works by talking to the DB directly, if that's your question | 15:41 |
| dtantsur | although a failure there should result in HTTP 500, not 404 | 15:41 |
| dking | That's what I thought. And nevermind, that error was unrelated. Which I suppose I could confirm by everything else working fine. | 15:41 |
| dking | Yeah, that's what I would have thought. | 15:42 |
| dtantsur | dking: is the power state problem consistent and applies to all nodes? | 15:43 |
| dking | dtantsur: I suspect so. It's been two so far. Of course, I'm not wanting to turn a bunch of servers off to test, but so far it's been consistent. I could probably find a couple more to test, though. | 15:46 |
| dmellado | dtantsur: same issue | 15:49 |
| dking | dtantsur: Yes, so far it's consistent. I've tried several nodes now. | 15:49 |
| dmellado | should I clean the node before redeploying, or that wouldn't make a change | 15:49 |
| dtantsur | shouldn't change much | 15:49 |
| dtantsur | dmellado: ironic downloads logs from the ramdisk, I think in /var/log/ironic/deploy | 15:50 |
| dtantsur | you need to check that for traces of the root cause. if you have no clue, share it with me. | 15:50 |
| dtantsur | dking: anything in the logs? | 15:50 |
| admiyo | looks like that was it...testing seems to be working. GOtta go through the whole round trip, but if this is it, I will be very happy. First beer for dtantsur and dmellado next face-to-face is on me | 15:51 |
| dtantsur | :) | 15:51 |
| dking | dtantsur: Well, not anything I can see except for the HTTP statuses. However, I just discovered that it's intermittent. | 15:54 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Follow up to "Run bifrost on CentOS Stream 9" https://review.opendev.org/c/openstack/bifrost/+/824186 | 15:54 |
| dking | I tried a bunch of them again, and because I was doing it often enough, I noticed that sometimes, it works. Which is even more puzzling to me. | 15:54 |
| dtantsur | O__o | 15:55 |
| dtantsur | dking: if hacking the code is an option, you can catch the exception and output traceback to see where it's coming from | 15:56 |
| dking | Well, even though, fortunately, this is a development instance, it's still in use, so I'm not too quick to want to do that, but I might step through the code. It seems to me that their would have to be something very off to give a 404 for a node that's in the DB. | 15:58 |
| dmellado | dtantsur: https://paste.openstack.org/show/812035/ | 16:00 |
| dmellado | I see some errors there on finding the partition | 16:00 |
| dmellado | but lack the context | 16:00 |
| dmellado | oh, found something | 16:01 |
| dmellado | Jan 11 15:46:45 localhost.localdomain ironic-python-agent[1649]: 2022-01-11 15:46:45.870 1649 DEBUG ironic_lib.utils [-] Command stdout is: "KNAME="sda" MODEL="PERC H730P Adp " SIZE="479559942144" ROTA="1" TYPE="disk" UUID="" PARTUUID="" | 16:01 |
| dmellado | that has no PARTUUID | 16:01 |
| dmellado | god knows why | 16:01 |
| dtantsur | dmellado: "Boot mode mismatch: target boot mode is uefi, current boot mode is bios" is a comparably large issue IMO | 16:01 |
| dmellado | ouch! | 16:01 |
| dmellado | lol | 16:02 |
| dtantsur | you're using the IPMI driver, right? have you set legacy boot yourself? | 16:02 |
| dmellado | I'm using IPMI | 16:02 |
| dtantsur | if that's not intended, I'd recommend you switch your machines to UEFI | 16:02 |
| dmellado | and I don't recall anything about legacy anything | 16:02 |
| dtantsur | unfortunately, we cannot do it for you when using IPMI | 16:02 |
| dmellado | I'll switch then right away | 16:02 |
| dmellado | should be on IDRAC bios, I assume | 16:02 |
| dtantsur | this will likely make the issue go away, even though the bug seems real | 16:02 |
| dtantsur | dmellado: could you send me (maybe via email if it's too large) the whole journal for investigation? | 16:03 |
| dtantsur | with some context, i.e. which parameters you used for DIB and what command you're running? | 16:03 |
| dtantsur | (better as an email so that I don't forget) | 16:03 |
| dmellado | dtantsur: sure, I'll drop you an email | 16:03 |
| dmellado | parameters, commands, and all disturbing things xD | 16:04 |
| dmellado | thanks in advance | 16:04 |
| dtantsur | dmellado: once you do the initial onboarding, you can try switching the driver to redfish, it will do the boot mode management for you | 16:04 |
| dmellado | ++ | 16:05 |
| dmellado | in any case, just switched to UEFI | 16:05 |
| dmellado | god only knows why it was set to bios... | 16:05 |
| dmellado | dtantsur: sent! | 16:14 |
| dmellado | thanks in advance! | 16:14 |
| dtantsur | thank you! will likely only get to it tomorrow, I hope the switch to UEFI helps | 16:14 |
| dtantsur | okay, thinking about it: your image may not be UEFI compatible | 16:14 |
| dtantsur | dmellado: if you see ^^^, you need to add https://docs.openstack.org/diskimage-builder/latest/elements/block-device-efi/README.html to elements | 16:15 |
| dtantsur | TheJulia: I wonder if we should update bifrost to use ^^ by default | 16:16 |
| dtantsur | given that we use UEFI by default | 16:16 |
| dmellado | dtantsur: same way I added dev-user, isn't it? | 16:18 |
| dtantsur | correct | 16:18 |
| dmellado | ipa_extra_dib_elements: devuser block-device-efi | 16:19 |
| dmellado | gotcha | 16:19 |
| dtantsur | hold on, sorry | 16:19 |
| dtantsur | dmellado: it's about the instance image, not IPA | 16:19 |
| dtantsur | are you building your instance centos image with DIB as well? | 16:20 |
| dmellado | oh, instance, I see | 16:20 |
| dmellado | yeah | 16:20 |
| dtantsur | ... more things to document ... | 16:20 |
| dmellado | lol, sorry to give you more work | 16:20 |
| dmellado | how can I add elements to that? | 16:21 |
| dtantsur | no worries, it's a useful exercise | 16:21 |
| dtantsur | dmellado: just extra_dib_elements | 16:21 |
| dtantsur | if you could post at least a draft documentation patch, would be awesome | 16:21 |
| dmellado | dtantsur: I'll do | 16:22 |
| dtantsur | I'm afraid I'll forget what we've been talking about | 16:22 |
| dmellado | I think we covered quite a few things | 16:22 |
| dmellado | maybe not your easiest start deployment | 16:22 |
| dtantsur | heh | 16:22 |
| dmellado | anyways, gotta run, I'll leave that running and bother you some more tomorrow | 16:23 |
| dmellado | thanks again! | 16:23 |
| dtantsur | sure, see you tomorrow | 16:23 |
| *** sshnaidm is now known as sshnaidm|afk | 16:23 | |
| bkranendonk | hi folks, is there a way to customize the used nameserver in the IPA image? | 16:42 |
| bkranendonk | cant find info about it | 16:42 |
| TheJulia | hmm | 16:42 |
| bkranendonk | cant supply dns using dnsmasq either it seems (im using kolla) | 16:44 |
| TheJulia | it wouldn't be able to be a parameter on the command line for the agent to trigger and act upon as it would be too late, it would need to be in the ramdisk as one would normally customize it. Or DHCP can offer it... | 16:44 |
| TheJulia | why not? | 16:44 |
| TheJulia | just not in their template? | 16:44 |
| bkranendonk | well yeah I can ofc :D, but not without editing the template hehe | 16:45 |
| TheJulia | I'd edit the template and toss a patch up for them | 16:45 |
| TheJulia | personally | 16:45 |
| TheJulia | that seems like a thing that makes sense to be able to set | 16:45 |
| bkranendonk | yep will do | 16:45 |
| bkranendonk | thx! | 16:45 |
| TheJulia | no problem, sorry you ran into that :( | 16:45 |
| TheJulia | ajya: do you happen to be around? | 16:46 |
| TheJulia | nvmd, I see you reviewed the patch | 16:47 |
| arne_wiebalck | bye everyone, see you tomorrow o/ | 17:21 |
| dtantsur | o/ | 17:22 |
| rpittau | good night! o/ | 17:30 |
| iurygregory | bye rpittau o/ | 17:31 |
| rpittau | TheJulia iurygregory one last thing for today, https://review.opendev.org/c/openstack/ironic/+/820927 is now passing CI so we can merge stuff to 18.1 again, it may need some adjustments though, please have a look when you got a chance :) | 18:27 |
| TheJulia | rpittau: thank you for picking that back up | 18:28 |
| rpittau | no problem! :) | 18:28 |
| rpittau | o/ | 18:28 |
| TheJulia | #success https://twitter.com/ArneWiebalck/status/1480937198333853696 | 18:28 |
| opendevstatus | TheJulia: Added success to Success page (https://wiki.openstack.org/wiki/Successes) | 18:28 |
| TheJulia | as much as I would like to see the wiki gone | 18:37 |
| opendevreview | Merged openstack/ironic master: Ensure 'port' is up2date after binding:host_id https://review.opendev.org/c/openstack/ironic/+/823811 | 19:39 |
| stevebaker[m] | good morning ironic | 20:06 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Add CentOS Stream 9 keystone integration job https://review.opendev.org/c/openstack/bifrost/+/824193 | 20:23 |
| TheJulia | stevebaker[m]: https://github.com/openstack/sushy/blob/master/sushy/connector.py | 20:46 |
| opendevreview | Merged openstack/ironic-python-agent-builder stable/xena: Replace genisoimage with xorriso https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/823601 | 21:00 |
| opendevreview | Verification of a change to openstack/ironic bugfix/18.1 failed: Explicitly pin upper sushy, reduce job count https://review.opendev.org/c/openstack/ironic/+/820927 | 21:10 |
| opendevreview | Julia Kreger proposed openstack/python-ironic-inspector-client master: Disable keep-alive https://review.opendev.org/c/openstack/python-ironic-inspector-client/+/824247 | 22:07 |
| opendevreview | Julia Kreger proposed openstack/python-ironic-inspector-client master: Disable keep-alive https://review.opendev.org/c/openstack/python-ironic-inspector-client/+/824247 | 22:08 |
| TheJulia | hjensas: stevebaker[m]: ^^^ | 22:09 |
| stevebaker[m] | yus | 22:14 |
| hjensas | TheJulia: I still se the error with that patch | 22:20 |
| TheJulia | hjensas: can you paste the error real quick | 22:21 |
| TheJulia | its working on paras's env | 22:21 |
| TheJulia | but maybe different things going on | 22:21 |
| TheJulia | inspection is sure not working overall though | 22:21 |
| hjensas | Unexpected exception for https://192.168.24.2:13050/v1/rules/ff0deca3-b954-474e-88f2-5e137909647d: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read)) | 22:21 |
| TheJulia | hjensas: ahh yeah | 22:22 |
| TheJulia | hmm | 22:22 |
| TheJulia | hjensas: any exceptions in the service logs? | 22:22 |
| TheJulia | so paras's environment is also missing br-baremetal | 22:24 |
| hjensas | TheJulia: no errors in ironic-inspector logs - this is a snippet showing the rule delete - https://paste.opendev.org/show/812053/ | 22:25 |
| TheJulia | hjensas: I wonder if inspector is returning a Content-Length field | 22:30 |
| hjensas | TheJulia: I think it does'nt, that would show up when trying to perform the action directly using curl right? | 22:41 |
| hjensas | $ curl -g -i -X delete https://192.168.24.2:13050/v1/rules/67736c21-d09c-488f-b63f-fd0c754141af -H "User-Agent: openstacksdk/0.55.0 keystoneauth1/4.3.1 python-requests/2.25.1 CPython/3.6.8" -H "X-Auth-Token: $TOKEN_ID" -H "X-OpenStack-Ironic-Inspector-API-Version: 1.0" | 22:41 |
| hjensas | HTTP/1.1 204 NO CONTENT | 22:41 |
| hjensas | Content-Type: text/plain; charset=utf-8 | 22:41 |
| hjensas | X-Openstack-Ironic-Inspector-Api-Minimum-Version: 1.0 | 22:41 |
| hjensas | X-Openstack-Ironic-Inspector-Api-Maximum-Version: 1.17 | 22:41 |
| hjensas | Date: Mon, 10 Jan 2022 19:16:18 GMT | 22:41 |
| hjensas | Transfer-Encoding: chunked | 22:41 |
| hjensas | curl: (18) transfer closed with outstanding read data remaining | 22:41 |
| TheJulia | shoot | 22:42 |
| TheJulia | so | 22:42 |
| TheJulia | it *must* reply with a content length if memory serves | 22:42 |
| TheJulia | if transfer encoding gets set | 22:42 |
| TheJulia | if you can get it to dump the entire post body, that migh ebe helpful | 22:42 |
| TheJulia | hjensas: so it is inspector | 22:49 |
| TheJulia | hmm | 22:49 |
| TheJulia | maybe not | 22:49 |
| TheJulia | flask on train | 22:49 |
| TheJulia | deja vu | 22:50 |
| TheJulia | started in 10.3.0 | 22:51 |
| TheJulia | If2e7697e3fde58ab0a4193787e29d3acdca81ebf | 22:51 |
| TheJulia | That is in it | 22:52 |
| TheJulia | because httpd proxies were barfing on the exhcnage | 22:52 |
| stevebaker[m] | TheJulia: I tried your change, it didn't help https://review.opendev.org/c/openstack/python-ironic-inspector-client/+/824247/2/ironic_inspector_client/common/http.py#210 | 22:55 |
| stevebaker[m] | TheJulia: you did a manual curl and even that complained? | 22:56 |
| TheJulia | no, I just re-ran paras's command which was failing | 22:59 |
| TheJulia | but hjensas did | 23:00 |
| stevebaker[m] | I just did it | 23:00 |
| TheJulia | the problem, ultimately is, aiui | 23:00 |
| stevebaker[m] | biab, out to lunch | 23:00 |
| TheJulia | If the client says "give me chunked transfers" it pasts the header on, the rfc if memory serves basically says the server has to honor that and reply with that | 23:01 |
| TheJulia | *but* forcing the request to be octet stream should change the request to not be a chunked transfer | 23:01 |
| stevebaker[m] | how about setting an explicit Content-Length: 0 for 204 responses? | 23:03 |
| TheJulia | That seems like a possibility | 23:03 |
| TheJulia | I don't think 204s require it | 23:03 |
| TheJulia | but why not | 23:03 |
| TheJulia | if you can just hack it in on the reply, I think that would help | 23:03 |
| TheJulia | well, locally where you can test/reproduce it | 23:03 |
| TheJulia | btw, I put the firewall rule change up | 23:04 |
| stevebaker[m] | I'll try it after lunch, if you don't get there first | 23:04 |
| stevebaker[m] | ok | 23:04 |
| TheJulia | I need to go start dinner in a little bit | 23:04 |
| stevebaker[m] | to the food! | 23:04 |
| TheJulia | and call my uncle back east who has tried to call me twice | 23:04 |
| hjensas | https://everything.curl.dev/http/response#chunked-transfer-encoding - It seems there should'nt be a Content-Length if it is Chunked-Transfer ? | 23:04 |
| TheJulia | that is also correct | 23:05 |
| TheJulia | and servers are supposted to honor it | 23:31 |
| TheJulia | so I bet what is happening is we don't have an emtpy carriage return | 23:31 |
| TheJulia | and actually it is new line and carriage return, based upon the standard as the delimiter | 23:36 |
| TheJulia | and I think dmitry is going to consider it a braking api change | 23:36 |
| TheJulia | breaking | 23:36 |
| hjensas | TheJulia: I tried response = flask.make_response('\n\r', code), but curl still gives me the "(18) transfer closed with outstanding read data remaining" error. | 23:50 |
| TheJulia | hjensas: \r\n | 23:56 |
| hjensas | TheJulia: still same error with \r\n | 23:59 |
| TheJulia | hmm | 23:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
