| opendevreview | sean mooney proposed openstack/nova master: add multinode ironic shard job https://review.opendev.org/c/openstack/nova/+/910333 | 00:20 |
|---|---|---|
| opendevreview | Amit Uniyal proposed openstack/nova master: Disconnecting volume from the compute host https://review.opendev.org/c/openstack/nova/+/877446 | 06:27 |
| opendevreview | Amit Uniyal proposed openstack/nova master: Removed explicit call to delete attachment https://review.opendev.org/c/openstack/nova/+/891289 | 06:27 |
| tobias-urdin | sean-k-mooney: when you have time, see if this updated patch is ok now based on your feedback https://review.opendev.org/c/openstack/nova/+/904381 | 10:08 |
| tobias-urdin | also super interested in your ideas on how to solve the bug for this patch https://review.opendev.org/c/openstack/nova/+/875859 that we got hit by some days ago | 10:09 |
| sean-k-mooney[m] | bauzas: can we get https://review.opendev.org/c/openstack/nova/+/896100 in to unblock the uc bump | 10:31 |
| bauzas | sean-k-mooney: I don't have context but I can take a look | 10:32 |
| opendevreview | Merged openstack/nova master: imagebackend: Add support to libvirt_info for LUKS based encryption https://review.opendev.org/c/openstack/nova/+/826755 | 11:22 |
| opendevreview | Merged openstack/nova master: Add encryption support to convert_image https://review.opendev.org/c/openstack/nova/+/870934 | 11:22 |
| ratailor__ | Hi cores, can I get reviews on https://review.opendev.org/q/topic:%22list-requested-az%22 | 11:25 |
| ratailor__ | elodilles, gibi sean-k-mooney[m] dansmith please review ^^ | 11:30 |
| gibi | ratailor__: bauzas: I -1 on the AZ series, the mock in the functional tests feels like a paperover of a bug | 11:47 |
| stephenfin | bauzas: Can we please just merge https://review.opendev.org/c/openstack/nova/+/896100 and move on | 11:48 |
| opendevreview | Amit Uniyal proposed openstack/nova master: Refactor vf profile for PCI device https://review.opendev.org/c/openstack/nova/+/905134 | 11:55 |
| sean-k-mooney | bauzas: tl;dr castalan added 2 new required abstract methods to a basecalse we inherit form | 12:02 |
| sean-k-mooney | bauzas: that got reverted on stable last cycle | 12:02 |
| sean-k-mooney | bauzas: but the castalan version that was intened for stabel only got pulled into master by mistake | 12:02 |
| sean-k-mooney | so we were not seing this for the last cycle | 12:03 |
| sean-k-mooney | bauzas: we should have had this code chagne since bobcat | 12:03 |
| sean-k-mooney | bauzas: i dont really think its reasoable to punt the new castalane version for two cycles in a row just because we didnt accept a patch that addes to uncalled functions that raise not implemetned | 12:05 |
| sean-k-mooney | bauzas: there was a way for castalan to not make this a breaking api change | 12:06 |
| sean-k-mooney | but im not sure how fair it is to say to them rework this now and do another release unless we are going to write that patch for them | 12:07 |
| sean-k-mooney | bauzas: im not particaly happy either but apparenty the non-client release deadline is the deadlien for the code to be merged not the deadline for the requirement repo to be bumped | 12:07 |
| sean-k-mooney | hence reluctant +2 from me | 12:08 |
| opendevreview | Merged openstack/nova master: Fix nova-metadata-api for ovn dhcp native networks https://review.opendev.org/c/openstack/nova/+/910466 | 12:34 |
| gibi | bauzas: would it make sense to fix the AZ series now, (I can try), or the bug I found means we punt it form the release? | 12:40 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Fix for showing AZ without reqspec https://review.opendev.org/c/openstack/nova/+/910728 | 12:59 |
| sean-k-mooney | gibi: technially code feeze was yesterday, but we could accpet it today if we wanted too | 13:10 |
| sean-k-mooney | i would not extend to monday however without going though the excption process | 13:10 |
| sean-k-mooney | gibi: what is the behvior with your fix | 13:11 |
| sean-k-mooney | gibi: i would expect the pinned az to be None | 13:11 |
| sean-k-mooney | if its not in the request spec so in json that would be null not '' | 13:11 |
| sean-k-mooney | gibi: could you add an api sample test to show the end result? | 13:12 |
| sean-k-mooney | we coudl go with empty string i guess instead of null | 13:13 |
| stblatzheim | sean-k-mooney: Fix was merged. Any change i can backport this till stable/zed or at least stable/2023.1 ? | 13:31 |
| stblatzheim | sean-k-mooney: Fix was merged. Any chance i can backport this till stable/zed or at least stable/2023.1 ? | 13:31 |
| sean-k-mooney | yes it shoudl be backportable we have to go newest to oldest but i dont see why it could not go back to zed | 13:32 |
| sean-k-mooney | are you familar with how to cerrypick it | 13:32 |
| sean-k-mooney | i doubt it will have merge confict so fi the patch is merged you can do it via the gerrit ui | 13:33 |
| sean-k-mooney | if its not merged the gerrit ui wont add the cherry-picked form line | 13:33 |
| stblatzheim | I'll give it a try :) It's already merged. | 13:34 |
| sean-k-mooney | that was more a meta comment for the older branches | 13:35 |
| sean-k-mooney | you have ot go n to n-1 then n-1 to n-2 | 13:35 |
| opendevreview | Steven Blatzheim proposed openstack/nova stable/2023.2: Fix nova-metadata-api for ovn dhcp native networks https://review.opendev.org/c/openstack/nova/+/910616 | 13:35 |
| sean-k-mooney | not n to n-2 | 13:35 |
| sean-k-mooney | so the simpelstet way to do that is jsut wait for ^ to merge | 13:35 |
| sean-k-mooney | before creating the next one | 13:35 |
| stblatzheim | Ok, so i'll wait for 2023.2 and when it's merged i'll start 2023.1 and after this merge zed | 13:36 |
| sean-k-mooney | cool | 13:36 |
| stblatzheim | thanks for your help with the whole fix | 13:37 |
| sean-k-mooney | no worries, i have +2'd the first backport feel free to ping me if you need reviews on the others | 13:38 |
| opendevreview | Rajesh Tailor proposed openstack/nova master: Add support for showing requested az in output https://review.opendev.org/c/openstack/nova/+/904568 | 13:44 |
| elodilles | bauzas: when you are around, can you take a second look on this? https://review.opendev.org/c/openstack/nova/+/896100 | 13:58 |
| opendevreview | Amit Uniyal proposed openstack/nova master: Refactor vf profile for PCI device https://review.opendev.org/c/openstack/nova/+/905134 | 14:11 |
| bauzas | gibi: sean-k-mooney: sorry I was doing something else and before I wasn't in front of my screen when you pinged me | 15:11 |
| bauzas | gibi: sean-k-mooney: so I can take a second look on the AZ API | 15:11 |
| bauzas | elodilles: sean-k-mooney: castellan stuff sent to the gate | 15:12 |
| sean-k-mooney | bauzas: i have been dipping in an dout of the az patch | 15:17 |
| sean-k-mooney | i honestly dont have the context fo that loaded enough to review properly | 15:18 |
| sean-k-mooney | gibi seems to have a better grasp of that | 15:18 |
| sean-k-mooney | gibi pushed a followup patch but rajesh then incoperated some of that into there own patch, perhaps they didnt see gibis patch | 15:19 |
| sean-k-mooney | so i dont really know if the comments have been adressed | 15:19 |
| bauzas | I'll take a look before I stop for the weekend | 15:21 |
| gibi | sean-k-mooney: bauzas: there are two comments left open there form me | 15:22 |
| gibi | I can fix them if we still want to land it | 15:22 |
| gibi | but i have limited time left for today | 15:22 |
| opendevreview | Rajesh Tailor proposed openstack/nova master: Add support for showing requested az in output https://review.opendev.org/c/openstack/nova/+/904568 | 15:24 |
| sean-k-mooney | ratailor_: are you actully around ^ | 15:24 |
| ratailor_ | sean-k-mooney, yes | 15:24 |
| sean-k-mooney | ok it woudl be good to check in with bauzas / gibi to make sure everthing that needs to be adress is | 15:25 |
| ratailor_ | sean-k-mooney, sure. Thanks! | 15:25 |
| ratailor_ | just now updated the patch, let's see how it goes. | 15:25 |
| sean-k-mooney | ack i expect most of use will disapare for the weekend in the next 30 mins or so | 15:26 |
| ratailor_ | sean-k-mooney, aha.. then what are the options to get it merged ? | 15:26 |
| gibi | ratailor_: could you drop the changes from https://review.opendev.org/c/openstack/nova/+/904568/16/nova/tests/unit/policies/test_servers.py | 15:26 |
| gibi | they are not needed | 15:26 |
| gibi | then I will be +2 | 15:27 |
| sean-k-mooney | apprently ^ | 15:27 |
| ratailor_ | gibi, sure, i will remove remaining changes there. | 15:27 |
| gibi | cool | 15:28 |
| opendevreview | Amit Uniyal proposed openstack/nova master: Refactor vf profile for PCI device https://review.opendev.org/c/openstack/nova/+/905134 | 15:28 |
| ratailor_ | gibi, how to resolve this https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d83/904568/16/check/openstack-tox-pep8/d83f39e/job-output.txt | 15:35 |
| ratailor_ | nova/api/openstack/compute/views/servers.py:223:5: C901 'ViewBuilder.show' is too complex (41) | 15:35 |
| ratailor_ | gibi, I think, its failing because we have too many ifs | 15:36 |
| ratailor_ | we want to get req_spec during server show call when provided_az is None, but during server list, it will be passed from list_view method, so it might be some-string or '' | 15:39 |
| opendevreview | Fabian Wiesel proposed openstack/nova master: Vmware: Remove uuid parameter from get_vmdk_info call https://review.opendev.org/c/openstack/nova/+/910627 | 15:42 |
| dansmith | sean-k-mooney: was it you that had or was using some log aggregator thing that would collapse logs from multiple services into one? | 15:43 |
| sean-k-mooney | gibi: has one tool for that bu ti also use lnav locally to do something like that | 15:46 |
| dansmith | just thinking that for debugging gate fails, it might be a lot easier if we had a unified journal-style log with all the services from that host so we could open and search for a traceback, error, or instance uuid | 15:46 |
| bauzas | dansmith: I generally use journalctl -f -u devstack@n-* for my controller | 15:47 |
| sean-k-mooney | i think we have a full copy of the journal oputput | 15:47 |
| dansmith | bauzas: right, I mean for gate fails | 15:47 |
| bauzas | ah, aggregating the logfiles | 15:48 |
| dansmith | sean-k-mooney: right but it's not viewable in the browser is it? | 15:48 |
| sean-k-mooney | im on a call but ill see if i can get a link | 15:49 |
| gibi | ratailor_: just pull out the logic to a separate function to reduce complexity | 15:49 |
| ratailor_ | gibi, ack. Thanks! | 15:50 |
| ratailor_ | gibi, inner function ? | 15:50 |
| bauzas | sean-k-mooney: dansmith: yeah we do have the journal https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f81/904568/14/check/tempest-integrated-compute/f81f16a/controller/logs/devstack.journal.README.txt | 15:51 |
| dansmith | yeah, I know, but it's not browseable | 15:51 |
| bauzas | but you need to download the file | 15:51 |
| bauzas | yup | 15:51 |
| dansmith | but yeah, I guess we don't need log merge if we generated the combined log from that | 15:51 |
| dansmith | okay so maybe I should just script doing that from a log url and look at it not in the browser | 15:52 |
| gibi | ratailor_: a private func inte view class is OK too | 15:52 |
| bauzas | honestly, for debugging gate failures, I'm sometimes downloading already | 15:52 |
| ratailor_ | gibi, ack. | 15:52 |
| bauzas | dansmith: we already have download-logs.sh we could add some tooling in that script to do some kind of automatic merge | 15:53 |
| sean-k-mooney | dansmith: ok done. so ya we have the devstack.journal.gz | 16:04 |
| sean-k-mooney | which you cant browse in the web as you said | 16:04 |
| opendevreview | Rajesh Tailor proposed openstack/nova master: Add support for showing requested az in output https://review.opendev.org/c/openstack/nova/+/904568 | 16:09 |
| bauzas | ratailor_: I need to stop now | 16:11 |
| bauzas | gibi: too, I need to stop | 16:11 |
| bauzas | but I can take a look on Monday morning at https://review.opendev.org/c/openstack/nova/+/904568/ | 16:11 |
| ratailor_ | bauzas, isn't today last day for featurefreeze ? | 16:12 |
| sean-k-mooney | it was technially yesterday | 16:12 |
| bauzas | it was yesterday | 16:12 |
| bauzas | but I can give you an exception due to my previous +2 | 16:12 |
| bauzas | gibi found a good point | 16:13 |
| ratailor_ | bauzas, ack. Thanks! | 16:13 |
| ratailor_ | sean-k-mooney, gibi Thanks! | 16:13 |
| bauzas | but if we can't merge it on Monday, then it will be punted to Dalmatian | 16:13 |
| ratailor_ | bauzas, ack. sure. | 16:14 |
| bauzas | sean-k-mooney: dansmith: about the log merging, | 16:14 |
| bauzas | sean-k-mooney: dansmith: could we modify our ansible scripts to create some host logs ? | 16:14 |
| bauzas | 2024-02-28 13:50:19.602722 | TASK [fetch-devstack-log-dir : Collect devstack logs] | 16:14 |
| dansmith | bauzas: I'm cooking something right now, we'll see if it's helpful | 16:15 |
| bauzas | dansmith: I need to go off by now but I could see it on Monday | 16:15 |
| dansmith | o/ | 16:16 |
| dansmith | man, we have got to get brick to stop logging tracebacks any time multipathd is missing | 16:18 |
| sean-k-mooney | but how else would you know... | 16:19 |
| sean-k-mooney | it is a littel excessive | 16:19 |
| bauzas | dansmith: https://opendev.org/openstack/devstack/src/branch/master/roles/export-devstack-journal/tasks/main.yaml#L9-L18 | 16:20 |
| bauzas | dansmith: I think we could change this to create some host log | 16:20 |
| sean-k-mooney | well we do just in binary form | 16:20 |
| dansmith | bauzas: we already have them combined in the journal, and I bet duplicating them in the log dump would not be popular, so just having something I can dump a log url into in the terminal is good enough I think | 16:21 |
| sean-k-mooney | https://opendev.org/openstack/devstack/src/branch/master/roles/export-devstack-journal/tasks/main.yaml#L37-L49 | 16:21 |
| dansmith | less good than in the browser, but the browser struggles anyway | 16:21 |
| bauzas | sean-k-mooney: yeah but as dansmith said, you need to download the whole journal | 16:21 |
| sean-k-mooney | right which woudl be the same... | 16:21 |
| dansmith | https://termbin.com/mqhd | 16:21 |
| bauzas | anyway, /me goes off by now | 16:21 |
| dansmith | if you run that with the url copied from the zuul manifest it drops you right into journalctl with the whole thing merged | 16:22 |
| dansmith | and you can use '-u devstack@n*' to get only nova stuff | 16:22 |
| gibi | bauzas: ratailor_ cool, Im looking at it now | 16:22 |
| sean-k-mooney | cool | 16:22 |
| sean-k-mooney | i was playing iwht something like | 16:22 |
| sean-k-mooney | curl https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f81/904568/14/check/tempest-integrated-compute/f81f16a/controller/logs/devstack.journal.gz | zcat | lnav -q | 16:22 |
| dansmith | can lnav read journal files? | 16:22 |
| sean-k-mooney | kind of but it does not render them well | 16:23 |
| dansmith | well, journalctl can :) | 16:23 |
| sean-k-mooney | so you would proably want to pass them thorugh systemd-joural-remote | 16:23 |
| dansmith | not as nice highlighting and stuff, but.. | 16:23 |
| sean-k-mooney | well i normally pipe journalctl to lnav | 16:24 |
| sean-k-mooney | so ya that script would be useful wehn i need ot do that | 16:24 |
| dansmith | oh yeah just running that | lnav is very nice | 16:25 |
| dansmith | that script I mean | 16:25 |
| sean-k-mooney | ya | 16:25 |
| sean-k-mooney | so we recently moved our new installer to put all the logs into the jorunal | 16:25 |
| dansmith | the comment in there isn't right, let me fix it | 16:25 |
| sean-k-mooney | and in those ci jobs we just are dumping /var/log/messages in to the job logs | 16:26 |
| sean-k-mooney | and i have been using lnave to parst them | 16:26 |
| dansmith | there: https://termbin.com/qpbv | 16:26 |
| dansmith | so I can just right-click copy a url out of a zuul manifest and run that on it, optionally | lnav and it just dumps me into a merged log | 16:26 |
| dansmith | that's nice | 16:26 |
| gibi | bauzas: I'm +2 on the AZ series | 16:27 |
| sean-k-mooney | "/usr/lib/systemd/systemd-journal-remote: No such file or directory" installing that might help | 16:29 |
| dansmith | yeah | 16:30 |
| dansmith | it's also in a different place on ubuntu, so I should make it use either | 16:30 |
| sean-k-mooney | you could just leave it up to the path | 16:30 |
| dansmith | it's not in path on either I think | 16:31 |
| sean-k-mooney | oh ok odd | 16:31 |
| sean-k-mooney | im installing it now but ill let you know | 16:31 |
| sean-k-mooney | /usr/lib/systemd/systemd-journal-remote works on debian testing | 16:32 |
| dansmith | on ubuntu it's /lib/systemd/systemd-journal-remote | 16:32 |
| dansmith | oh is there a link? | 16:32 |
| sean-k-mooney | rwxrwxrwx 1 root root 7 May 30 2023 /lib -> usr/lib | 16:33 |
| sean-k-mooney | so ya /lib links to /usr/lib | 16:33 |
| dansmith | oh heh | 16:33 |
| dansmith | okay perfect then | 16:34 |
| dansmith | I guess I should gist this | 16:34 |
| sean-k-mooney | or we can have the ci also provide the script | 16:34 |
| sean-k-mooney | i like that you dont redownload the file if its there | 16:35 |
| sean-k-mooney | ../log-download.sh https://zuul.opendev.org/t/openstack/build/83d6dc665c2643f88d0025c385987c92/logs | lnav -q | 16:35 |
| sean-k-mooney | that works for me nicely | 16:35 |
| dansmith | https://gist.github.com/kk7ds/9a77939bd314cd4b5a35139adcdcedce | 16:35 |
| sean-k-mooney | osgatelog is a better name | 16:36 |
| dansmith | I also have an osfinger, that attaches to a live running job's console at the terminal | 16:36 |
| sean-k-mooney | oh cool. i never got the finger gateway stuff to work localy | 16:38 |
| sean-k-mooney | i didn know we had that configured for the public zuul | 16:38 |
| dansmith | https://termbin.com/4mx2 | 16:38 |
| dansmith | wait that's not it | 16:39 |
| sean-k-mooney | oh its litrally just conenct with nc or telnet | 16:39 |
| dansmith | that was an earlier bash only one, hang on | 16:39 |
| dansmith | haaaang on | 16:39 |
| sean-k-mooney | why do i feel like you might also have an emacs keybinding for this | 16:40 |
| dansmith | https://github.com/kk7ds/openstack-gerrit-dashboard/blob/master/osfinger.py | 16:40 |
| Uggla | sean-k-mooney[m], question, is the db test_migrations are run during manage db command ? | 18:10 |
| dansmith | Uggla: I'm not sure what that means | 18:34 |
| dansmith | test_migrations is only run when you run functional (I think) not when an admin actually uses nova-manage .. is that what you're asking? | 18:34 |
| dansmith | er, no s/functional/unit/ but.. same answer about nova-manage | 18:36 |
| *** ralonsoh_ is now known as ralonsoh | 19:08 | |
| stblatzheim | https://review.opendev.org/c/openstack/nova/+/910616 | 19:38 |
| stblatzheim | sean-k-mooney: ^ review got stuck after checks, is there a additional review needed? | 19:39 |
| dansmith | stblatzheim: I can hit that but, are we sure that's safe to backport? | 19:42 |
| dansmith | it's ovn-specific in non-ovn-specific code and depends on neutron stuff | 19:42 |
| dansmith | reading through the bug, sean was originally against this and then +2d it so there must be some context I'm missing | 19:47 |
| dansmith | even though it was changed "in victoria" I still don't see the seeming api breakage addressed | 19:47 |
| dansmith | we can circle back on monday where maybe sean can explain why it's the right thing to do | 19:49 |
| stblatzheim | We can wait till monday for sure. Infact the change is about a reliable fix to get the info if dhcp is enabled on the subnet or not. There was infact no real api change but a different device_owner cause of the introduction of ovn. This was never adressed in nova-metadata-api and should be safe to backport. | 19:55 |
| opendevreview | Merged openstack/nova master: Add hw_ephemeral_encryption_secret_uuid image property https://review.opendev.org/c/openstack/nova/+/870935 | 20:05 |
| opendevreview | Merged openstack/nova master: libvirt: make <encryption> a sub element of <source> https://review.opendev.org/c/openstack/nova/+/905515 | 20:05 |
| opendevreview | Merged openstack/nova master: testing: Add ephemeral encryption support to fixtures https://review.opendev.org/c/openstack/nova/+/910571 | 20:05 |
| dansmith | melwitt: I should be able to test the ephemeral encryption stuff by just enabling the barbican devstack plugin, right? assuming I just manually create a flavor after the stack is up for use with it | 20:11 |
| opendevreview | Merged openstack/nova master: Add support for showing requested az in output https://review.opendev.org/c/openstack/nova/+/904568 | 20:39 |
| melwitt | dansmith: pretty much. let me double check if there's any other gotcha in the dnm test patch | 21:19 |
| melwitt | dansmith: yeah, looks like that's all you need | 21:22 |
| melwitt | I think rescue is broken at the moment since I changed things to require the hw_ephemeral_encryption_format if the image is encrypted. slowly working through the stack again | 21:27 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!