| *** irclogbot_3 has quit IRC | 00:16 | |
| *** tetsuro has joined #openstack-nova | 00:19 | |
| *** irclogbot_3 has joined #openstack-nova | 00:20 | |
| *** ociuhandu has joined #openstack-nova | 00:25 | |
| *** ociuhandu has quit IRC | 00:30 | |
| *** tetsuro has quit IRC | 00:39 | |
| *** tetsuro has joined #openstack-nova | 00:41 | |
| openstackgerrit | Keigo Noha proposed openstack/nova master: Change default num_retries for glance to 3 https://review.opendev.org/740389 | 01:34 |
|---|---|---|
| *** Liang__ has joined #openstack-nova | 01:35 | |
| *** markvoelker has joined #openstack-nova | 01:36 | |
| *** markvoelker has quit IRC | 01:41 | |
| *** brinzhang has joined #openstack-nova | 01:47 | |
| *** brinzhang_ has joined #openstack-nova | 01:50 | |
| openstackgerrit | Brin Zhang proposed openstack/nova master: Cyborg shelve/unshelve support https://review.opendev.org/729563 | 01:52 |
| *** brinzhang has quit IRC | 01:54 | |
| *** brinzhang0 has joined #openstack-nova | 02:17 | |
| *** brinzhang_ has quit IRC | 02:20 | |
| *** sapd1 has joined #openstack-nova | 02:34 | |
| *** Liang__ has quit IRC | 02:54 | |
| *** Liang__ has joined #openstack-nova | 02:55 | |
| openstackgerrit | Tony Su proposed openstack/nova master: Provider Config File: YAML file loading and schema validation https://review.opendev.org/673341 | 03:01 |
| openstackgerrit | Tony Su proposed openstack/nova master: Provider Config File: Function to further validate and retrieve configs https://review.opendev.org/676029 | 03:01 |
| openstackgerrit | Tony Su proposed openstack/nova master: Provider Config File: Functions to merge provider configs to provider tree https://review.opendev.org/676522 | 03:01 |
| openstackgerrit | Tony Su proposed openstack/nova master: Provider Config File: Enable loading and merging of provider configs https://review.opendev.org/693460 | 03:01 |
| *** tetsuro has quit IRC | 03:02 | |
| *** rcernin has quit IRC | 03:02 | |
| *** Liang__ has quit IRC | 03:06 | |
| *** yaawang has quit IRC | 03:08 | |
| *** yaawang has joined #openstack-nova | 03:08 | |
| *** dave-mccowan has joined #openstack-nova | 03:11 | |
| *** Liang__ has joined #openstack-nova | 03:18 | |
| *** rcernin has joined #openstack-nova | 03:21 | |
| *** rcernin has quit IRC | 03:25 | |
| *** rcernin has joined #openstack-nova | 03:26 | |
| *** brinzhang_ has joined #openstack-nova | 03:46 | |
| *** brinzhang0 has quit IRC | 03:50 | |
| *** yaawang has quit IRC | 03:54 | |
| *** yaawang has joined #openstack-nova | 03:55 | |
| *** tetsuro has joined #openstack-nova | 03:57 | |
| *** dave-mccowan has quit IRC | 03:57 | |
| *** ociuhandu has joined #openstack-nova | 04:28 | |
| *** ociuhandu has quit IRC | 04:33 | |
| *** ratailor has joined #openstack-nova | 04:39 | |
| *** vishalmanchanda has joined #openstack-nova | 04:51 | |
| *** sapd1 has quit IRC | 05:17 | |
| openstackgerrit | Kevin Zhao proposed openstack/nova master: [WIP] CI: add tempest-integrated-compute-aarch64 job https://review.opendev.org/714439 | 05:30 |
| *** links has joined #openstack-nova | 05:44 | |
| *** udesale has joined #openstack-nova | 05:48 | |
| *** brinzhang0 has joined #openstack-nova | 05:56 | |
| *** brinzhang_ has quit IRC | 05:59 | |
| *** brinzhang_ has joined #openstack-nova | 06:06 | |
| *** brinzhang0 has quit IRC | 06:09 | |
| *** jsuchome has joined #openstack-nova | 06:14 | |
| *** tetsuro has quit IRC | 06:34 | |
| bauzas | good morning Nova | 06:35 |
| *** tetsuro has joined #openstack-nova | 06:36 | |
| brinzhang_ | morning bauzas^ | 06:39 |
| *** yaawang has quit IRC | 06:41 | |
| *** yaawang has joined #openstack-nova | 06:42 | |
| *** maciejjozefczyk has joined #openstack-nova | 06:43 | |
| *** nightmare_unreal has joined #openstack-nova | 06:57 | |
| gibi | good morning folks | 07:00 |
| *** Luzi has joined #openstack-nova | 07:02 | |
| brinzhang_ | gibi: morning ^ | 07:03 |
| brinzhang_ | The nova cyborg interaction patches will be deadline in the runway slot, do we need to ask help from efried? | 07:04 |
| *** tesseract has joined #openstack-nova | 07:05 | |
| gibi | efried is gone so I don't think he will help | 07:07 |
| brinzhang_ | emm..I know, but... | 07:08 |
| *** sapd1 has joined #openstack-nova | 07:11 | |
| *** ociuhandu has joined #openstack-nova | 07:12 | |
| gibi | the last two weeks of mine was busier than I anticipated so I did not have time to review all the things in the slots. I'm wondering that maybe 3 slots is more than what I can cover... maybe more than what we can cover | 07:13 |
| *** tetsuro_ has joined #openstack-nova | 07:15 | |
| *** tetsuro has quit IRC | 07:19 | |
| *** rcernin has quit IRC | 07:24 | |
| *** yaawang has quit IRC | 07:27 | |
| *** dtantsur|afk is now known as dtantsur | 07:27 | |
| *** yaawang has joined #openstack-nova | 07:28 | |
| openstackgerrit | Elod Illes proposed openstack/nova stable/queens: Check cherry-pick hashes in pep8 tox target https://review.opendev.org/741883 | 07:33 |
| *** tosky has joined #openstack-nova | 07:34 | |
| tony_su | gibi: hello gibi, could you spare some time reviewing provider-config-file patches as I saw its expire date is July 23. Today, I rebased them against master tip. I checked zuul failure and it's NOT related to us. | 07:37 |
| *** rcernin has joined #openstack-nova | 07:38 | |
| *** derekh has joined #openstack-nova | 07:39 | |
| gibi | tony_su: I will try but there is simply too much on my plate | 07:39 |
| gibi | sorry | 07:39 |
| *** xek_ has joined #openstack-nova | 07:39 | |
| nightmare_unreal | hello, what's the command for zuul to rebuild the patch. Something we put in patch comments and zuul perform it's checks again | 07:40 |
| tony_su | gibi: got it, Thanks. I will ask alex_xu to make a round of review if he has some bandwidth. https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/provider-config-file | 07:41 |
| *** tetsuro_ has quit IRC | 07:41 | |
| *** tetsuro has joined #openstack-nova | 07:41 | |
| *** rcernin has quit IRC | 07:44 | |
| gibi | tony_su: ack | 07:45 |
| openstackgerrit | Elod Illes proposed openstack/nova stable/rocky: fix scsi disk unit number of the attaching volume when cdrom bus is scsi https://review.opendev.org/730312 | 07:47 |
| nightmare_unreal | found it, it was recheck | 07:48 |
| *** martinkennelly has joined #openstack-nova | 07:51 | |
| *** yaawang has quit IRC | 07:52 | |
| *** yaawang has joined #openstack-nova | 07:52 | |
| alex_xu | tony_su: willl begin to review tomorrow or tonight if possible | 07:54 |
| *** ralonsoh has joined #openstack-nova | 07:55 | |
| *** xiaolin has joined #openstack-nova | 07:55 | |
| *** alex_xu has quit IRC | 08:03 | |
| *** markvoelker has joined #openstack-nova | 08:04 | |
| *** markvoelker has quit IRC | 08:09 | |
| *** fumesover3 has quit IRC | 08:12 | |
| *** markvoelker has joined #openstack-nova | 08:14 | |
| *** links has quit IRC | 08:19 | |
| *** markvoelker has quit IRC | 08:24 | |
| *** markvoelker has joined #openstack-nova | 08:27 | |
| *** yaawang has quit IRC | 08:29 | |
| *** yaawang has joined #openstack-nova | 08:30 | |
| *** markvoelker has quit IRC | 08:32 | |
| *** derekh has quit IRC | 08:34 | |
| *** ralonsoh_ has joined #openstack-nova | 08:53 | |
| *** sapd1 has quit IRC | 08:54 | |
| *** ralonsoh has quit IRC | 08:55 | |
| *** sapd1 has joined #openstack-nova | 09:01 | |
| *** k_mouza has joined #openstack-nova | 09:01 | |
| *** links has joined #openstack-nova | 09:02 | |
| *** rcernin has joined #openstack-nova | 09:10 | |
| *** tetsuro has quit IRC | 09:12 | |
| *** rcernin has quit IRC | 09:15 | |
| *** yaawang has quit IRC | 09:33 | |
| *** yaawang has joined #openstack-nova | 09:34 | |
| *** alex_xu has joined #openstack-nova | 09:38 | |
| *** ratailor has quit IRC | 09:58 | |
| *** ratailor_ has joined #openstack-nova | 09:58 | |
| *** sapd1 has quit IRC | 10:01 | |
| *** fumesover3 has joined #openstack-nova | 10:02 | |
| *** sapd1 has joined #openstack-nova | 10:05 | |
| *** ociuhandu has quit IRC | 10:05 | |
| *** ociuhandu has joined #openstack-nova | 10:08 | |
| *** yaawang has quit IRC | 10:08 | |
| *** k_mouza has quit IRC | 10:08 | |
| *** tosky has quit IRC | 10:08 | |
| *** Luzi has quit IRC | 10:08 | |
| *** irclogbot_3 has quit IRC | 10:08 | |
| *** johnthetubaguy has quit IRC | 10:08 | |
| *** vesper11 has quit IRC | 10:08 | |
| *** jdillaman has quit IRC | 10:09 | |
| *** yaawang has joined #openstack-nova | 10:10 | |
| *** k_mouza has joined #openstack-nova | 10:10 | |
| *** tosky has joined #openstack-nova | 10:10 | |
| *** Luzi has joined #openstack-nova | 10:10 | |
| *** irclogbot_3 has joined #openstack-nova | 10:10 | |
| *** johnthetubaguy has joined #openstack-nova | 10:10 | |
| *** vesper11 has joined #openstack-nova | 10:10 | |
| *** jdillaman has joined #openstack-nova | 10:10 | |
| *** ociuhandu has quit IRC | 10:13 | |
| *** amodi has quit IRC | 10:14 | |
| gibi | brinzhang_: hi! I left comments in https://review.opendev.org/#/c/716186 | 10:14 |
| *** derekh has joined #openstack-nova | 10:20 | |
| *** tkajinam has quit IRC | 10:29 | |
| *** ociuhandu has joined #openstack-nova | 10:32 | |
| *** trident has quit IRC | 10:35 | |
| *** trident has joined #openstack-nova | 10:36 | |
| *** Liang__ has quit IRC | 10:40 | |
| *** brinzhang0 has joined #openstack-nova | 10:56 | |
| *** brinzhang_ has quit IRC | 10:59 | |
| *** brinzhang0 has quit IRC | 11:04 | |
| *** brinzhang_ has joined #openstack-nova | 11:04 | |
| *** jangutter_ is now known as jangutter | 11:12 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Remove two-level loop https://review.opendev.org/741920 | 11:16 |
| sean-k-mooney | just an fyi i wont be around as much this week | 11:22 |
| *** brinzhang_ has quit IRC | 11:31 | |
| *** brinzhang_ has joined #openstack-nova | 11:32 | |
| *** sapd1 has quit IRC | 11:38 | |
| *** raildo has joined #openstack-nova | 11:38 | |
| *** jraju__ has joined #openstack-nova | 11:56 | |
| *** links has quit IRC | 11:57 | |
| *** brinzhang0 has joined #openstack-nova | 11:58 | |
| gibi | sean-k-mooney: ack | 11:58 |
| *** xek_ has quit IRC | 12:00 | |
| *** brinzhang_ has quit IRC | 12:01 | |
| gibi | does somebody knows about the failing novnc tempest test https://c50701f61b79b2f93421-0d0b23b7f62d479822a1b52b605369e3.ssl.cf2.rackcdn.com/673341/50/check/nova-next/5f7dd59/testr_results.html ? | 12:12 |
| gibi | novnc proxy has a stacktrace so I have something to start with | 12:21 |
| gibi | https://zuul.opendev.org/t/openstack/build/1180d2313ade4d5480550313bcd20b79/log/controller/logs/screen-n-novnc-cell1.txt#29 | 12:21 |
| sean-k-mooney | that looks like an interaction issue with eventlet? maybe | 12:25 |
| sean-k-mooney | was there a new releae of either eventlet or websockify | 12:25 |
| sean-k-mooney | not websockify anyway | 12:26 |
| sean-k-mooney | that last release last august | 12:26 |
| sean-k-mooney | eventlet was april | 12:27 |
| openstackgerrit | Merged openstack/nova stable/pike: Clean up allocation if unshelve fails due to neutron https://review.opendev.org/740636 | 12:28 |
| openstackgerrit | Merged openstack/nova stable/rocky: fix scsi disk unit number of the attaching volume when cdrom bus is scsi https://review.opendev.org/730312 | 12:28 |
| openstackgerrit | Merged openstack/nova stable/train: libvirt: Mark e1000e VIF as supported https://review.opendev.org/736962 | 12:29 |
| sean-k-mooney | not obvious changes in the reqruiement repo either | 12:30 |
| *** brinzhang_ has joined #openstack-nova | 12:32 | |
| gibi | if I understand correclty the error coming from the ssl c lib so I cannot really debug it | 12:35 |
| *** brinzhang0 has quit IRC | 12:35 | |
| sean-k-mooney | right so eventlet is monkeypatch the ssl socket | 12:41 |
| sean-k-mooney | but it looks like the version of eventlet we are using is not handelign the wrapping correctly | 12:42 |
| openstackgerrit | Merged openstack/nova stable/train: Correct reported system memory https://review.opendev.org/740667 | 12:42 |
| sean-k-mooney | maybe they broke 3.6 compatiablity but there has not been a releae of eventlet in a few months | 12:42 |
| sean-k-mooney | so im not sure why we would only see it now | 12:43 |
| *** JamesBenson has joined #openstack-nova | 12:43 | |
| sean-k-mooney | they have had this issue in the past https://github.com/eventlet/eventlet/issues/201 | 12:43 |
| sean-k-mooney | and it happened again https://github.com/eventlet/eventlet/issues/311 | 12:45 |
| gibi | those fixes talk about urllib2 but as far as I see we install urllib3 | 12:49 |
| sean-k-mooney | yes my point was more they have missed monkeypatching the context correctly in the past | 12:49 |
| *** jangutter_ has joined #openstack-nova | 12:55 | |
| *** jangutter has quit IRC | 12:58 | |
| sean-k-mooney | ill open an eventlet bug and see what they think | 13:01 |
| gibi | sean-k-mooney: thanks | 13:01 |
| gibi | I'm still trying to figure out what changed now that break us as all the sw releases are couple months old | 13:02 |
| sean-k-mooney | maybe the host openssl version | 13:02 |
| sean-k-mooney | or a recent python 3.6.z release | 13:03 |
| gibi | looking... | 13:03 |
| *** udesale_ has joined #openstack-nova | 13:03 | |
| *** jangutter has joined #openstack-nova | 13:04 | |
| *** udesale has quit IRC | 13:06 | |
| *** jangutter_ has quit IRC | 13:07 | |
| *** rcernin has joined #openstack-nova | 13:07 | |
| gibi | 1.1.1-1ubuntu2.1~18.04.6 was used for the last passing test as well as the failing test | 13:08 |
| gibi | I mean openssl is already the newest version (1.1.1-1ubuntu2.1~18.04.6). | 13:08 |
| *** mvkr has joined #openstack-nova | 13:09 | |
| sean-k-mooney | and the python version? | 13:09 |
| *** k_mouza has quit IRC | 13:09 | |
| gibi | that is also the same python3.6 is already the newest version (3.6.9-1~18.04ubuntu1). | 13:10 |
| gibi | I'm comparing the nova-next run from here (passed) https://review.opendev.org/#/c/739209/ with here (failed) https://review.opendev.org/#/c/728481/ | 13:11 |
| *** rcernin has quit IRC | 13:12 | |
| gibi | libssl-dev is also the same | 13:12 |
| *** nweinber has joined #openstack-nova | 13:14 | |
| *** lbragstad has joined #openstack-nova | 13:16 | |
| *** artom has joined #openstack-nova | 13:18 | |
| *** ratailor_ has quit IRC | 13:19 | |
| *** eharney has joined #openstack-nova | 13:26 | |
| *** xiaolin has quit IRC | 13:26 | |
| gibi | there was some image rebuild during the weekend but it only affected 3.7 and 3.8 images as far as I see https://review.opendev.org/#/c/741789/1 | 13:27 |
| sean-k-mooney | https://github.com/eventlet/eventlet/issues/632 | 13:28 |
| *** ociuhandu_ has joined #openstack-nova | 13:29 | |
| sean-k-mooney | ya but most of the relevent packages seam to be the same between the passing and failing build | 13:29 |
| *** k_mouza has joined #openstack-nova | 13:29 | |
| *** ociuhandu has quit IRC | 13:31 | |
| efried | brinzhang_: Yeah, sorry, I wouldn't be able to make enough time for something the size and scope of nova/cyborg. | 13:32 |
| *** bnemec has joined #openstack-nova | 13:32 | |
| gibi | sean-k-mooney: thanks for reporting that to eventlet | 13:33 |
| sean-k-mooney | there isnt much for them to go on really but maybe that can suggest something | 13:33 |
| sean-k-mooney | gibi: is this repetable by the way | 13:34 |
| *** k_mouza has quit IRC | 13:34 | |
| *** derekh has quit IRC | 13:34 | |
| gibi | sean-k-mooney: I saw it in two independent job run | 13:35 |
| *** efried has quit IRC | 13:36 | |
| gibi | sean-k-mooney: here is another reporduction https://zuul.opendev.org/t/openstack/build/5f7dd591c62843ce9a114276b933a932/log/controller/logs/screen-n-novnc-cell1.txt#68 | 13:36 |
| gibi | from https://review.opendev.org/#/c/673341/50 | 13:36 |
| *** efried has joined #openstack-nova | 13:36 | |
| *** k_mouza has joined #openstack-nova | 13:37 | |
| *** k_mouza has quit IRC | 13:41 | |
| gibi | I don't have SSL/TLS configured for vnc proxy in my devstack | 13:43 |
| *** brinzhang0 has joined #openstack-nova | 13:44 | |
| *** jangutter has quit IRC | 13:47 | |
| *** k_mouza has joined #openstack-nova | 13:47 | |
| *** brinzhang_ has quit IRC | 13:48 | |
| *** jangutter has joined #openstack-nova | 13:49 | |
| *** k_mouza has quit IRC | 13:51 | |
| *** sapd1 has joined #openstack-nova | 13:51 | |
| *** brinzhang_ has joined #openstack-nova | 14:01 | |
| *** k_mouza has joined #openstack-nova | 14:03 | |
| *** brinzhang0 has quit IRC | 14:04 | |
| gibi | so I'm restacking right now | 14:05 |
| *** ociuhandu_ has quit IRC | 14:05 | |
| *** ociuhandu has joined #openstack-nova | 14:06 | |
| *** derekh has joined #openstack-nova | 14:06 | |
| *** k_mouza has quit IRC | 14:06 | |
| sean-k-mooney | im not seeign it in logstash so i can see how often its happens | 14:10 |
| gibi | logstash indexing is behind with about 120 hours | 14:13 |
| sean-k-mooney | ok so 5 days | 14:14 |
| sean-k-mooney | and the failures were 3-4 days ago | 14:14 |
| sean-k-mooney | well in the previous 30 days there are no hits | 14:15 |
| gibi | the last patch merged ~3 days ago | 14:15 |
| gibi | sean-k-mooney: here is the nova-next job history https://zuul.opendev.org/t/openstack/builds?job_name=nova-next&project=openstack%2Fnova&branch=master | 14:22 |
| sean-k-mooney | yep but if it was in logstash i could have added an elasticrecheck query for it | 14:23 |
| *** sapd1 has quit IRC | 14:24 | |
| sean-k-mooney | i would expect this to fail on other jobs too | 14:25 |
| *** xek_ has joined #openstack-nova | 14:26 | |
| gibi | I'm not sure where we run the tempest test test_novnc | 14:27 |
| *** k_mouza has joined #openstack-nova | 14:27 | |
| gibi | hm tempest-integrated-compute also runs that test | 14:27 |
| gibi | and that is green | 14:28 |
| gibi | but that runs without SSL setupo | 14:28 |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: DNM: tesging nova greande job https://review.opendev.org/741956 | 14:28 |
| gibi | https://zuul.opendev.org/t/openstack/build/675ff69ce6624ddb83b7ff7fec1cef9b/log/controller/logs/screen-n-novnc-cell1.txt#7 | 14:28 |
| gmann | gibi: ^^ testing the grenade jobs failure fix ^^ | 14:28 |
| sean-k-mooney | gibi: ah right we might not have tls or run it in other jobs | 14:28 |
| gibi | gmann: thanks for looking into grenade! | 14:29 |
| gmann | legacy jobs were left for glance stabdalone thing | 14:29 |
| gmann | standalone | 14:29 |
| * artom stabs Dalone. | 14:30 | |
| * sean-k-mooney artom is grumpy today | 14:31 | |
| artom | Am I? | 14:32 |
| sean-k-mooney | ^ | 14:32 |
| artom | Maybe Dalone is just a prick. | 14:32 |
| artom | And deserves to be stabbed? | 14:32 |
| sean-k-mooney | maybe | 14:32 |
| * artom checks - OK, no one with "dalone" as their nick here | 14:32 | |
| dansmith | artom: always check *after* the threat | 14:34 |
| dansmith | it's funnier (for the rest of us) that way | 14:34 |
| artom | You mean my usual method of 1. open mouth 2. say dumb thing 3. check 4. regret | 14:34 |
| *** brinzhang0 has joined #openstack-nova | 14:35 | |
| sean-k-mooney | gmann: is standalone glance just without the registry service? | 14:35 |
| sean-k-mooney | or something else | 14:35 |
| *** brinzhang0 has quit IRC | 14:35 | |
| *** brinzhang0 has joined #openstack-nova | 14:36 | |
| gmann | sean-k-mooney: not running glance on uwsgi and rest all service on uwsgi. register service was removed much earlier | 14:37 |
| *** brinzhang_ has quit IRC | 14:38 | |
| gmann | sean-k-mooney: this one https://review.opendev.org/#/c/740322/10 | 14:38 |
| sean-k-mooney | oh ok | 14:38 |
| sean-k-mooney | this is for the multi store stuff so the glance import pipelene stuff works | 14:38 |
| gmann | dansmith: setting standalone as false for legacy grenade jobs too - https://review.opendev.org/#/c/741955/ | 14:38 |
| dansmith | gmann: I saw, thanks | 14:39 |
| dansmith | gmann: lots of tentacles... | 14:39 |
| gmann | yeah, legacy jobs fixes are always 'missed-and-fixed-later' :) | 14:40 |
| *** mgariepy has quit IRC | 14:41 | |
| * gibi is still trying to build a devstack with USE_SSL=True ... | 14:43 | |
| *** mlavalle has joined #openstack-nova | 14:43 | |
| dansmith | gmann: hah | 14:43 |
| *** brinzhang0 has quit IRC | 14:48 | |
| *** brinzhang0 has joined #openstack-nova | 14:49 | |
| *** xek_ has quit IRC | 14:50 | |
| *** mgariepy has joined #openstack-nova | 14:51 | |
| openstackgerrit | Elod Illes proposed openstack/nova stable/queens: fix scsi disk unit number of the attaching volume when cdrom bus is scsi https://review.opendev.org/730313 | 14:53 |
| gibi | sean-k-mooney: opened a nova bug for the novnc breakage https://bugs.launchpad.net/nova/+bug/1888237 | 14:55 |
| openstack | Launchpad bug 1888237 in OpenStack Compute (nova) "novan-next job fails as novnc service fails with TypeError: _wrap_socket() argument 1 must be _socket.socket, not GreenSSLSocket" [Critical,New] | 14:55 |
| *** xek_ has joined #openstack-nova | 14:56 | |
| sean-k-mooney | lyarwood: you might be interested in ^ by the way since we were discussing the need to test new version of it last week | 14:57 |
| lyarwood | sean-k-mooney: yup ack, I'll try to look this evening | 14:58 |
| sean-k-mooney | in this case there has not been a new release but its now failing with tls enabled | 14:58 |
| sean-k-mooney | actully | 14:59 |
| *** dklyle has joined #openstack-nova | 14:59 | |
| sean-k-mooney | there was a new tag of novnc 6 days ago | 14:59 |
| sean-k-mooney | v1.2.0 | 14:59 |
| lyarwood | yeah that did drop a few days ago | 14:59 |
| lyarwood | doesn't nova-next use NOVNC from source? | 15:00 |
| sean-k-mooney | possibly that is what is failing | 15:00 |
| sean-k-mooney | but the error seams to be in websockify | 15:00 |
| gibi | we are pulling 1.1.0 https://zuul.opendev.org/t/openstack/build/1180d2313ade4d5480550313bcd20b79/log/job-output.txt#2154 | 15:00 |
| sean-k-mooney | or eventlet not sure | 15:00 |
| lyarwood | kk then it's not that | 15:00 |
| * lyarwood jumps on a call | 15:00 | |
| openstackgerrit | Sergey Galas' proposed openstack/nova stable/train: fixes broken shared live migration of a vm with a vif https://review.opendev.org/741963 | 15:04 |
| *** xek_ has quit IRC | 15:07 | |
| *** Luzi has quit IRC | 15:09 | |
| *** ralonsoh_ is now known as ralonsoh | 15:10 | |
| *** hamalq has joined #openstack-nova | 15:15 | |
| jgwentworth | lyarwood: you had tried to get nova-next to work to install novnc from source in the past but didn't finish https://review.opendev.org/671490 | 15:18 |
| *** jgwentworth is now known as melwitt | 15:19 | |
| melwitt | I think there were some issues but I don't remember now | 15:19 |
| *** spatel has joined #openstack-nova | 15:29 | |
| *** artom has quit IRC | 15:31 | |
| *** artom has joined #openstack-nova | 15:31 | |
| *** xiaolin has joined #openstack-nova | 15:32 | |
| gibi | sean-k-mooney, lyarwood: I was able to reproduce https://bugs.launchpad.net/nova/+bug/1888237 in a local devstack. I've added the needed local.conf config | 15:34 |
| openstack | Launchpad bug 1888237 in OpenStack Compute (nova) "nova-next job fails as novnc service fails with TypeError: _wrap_socket() argument 1 must be _socket.socket, not GreenSSLSocket" [Critical,New] | 15:34 |
| gibi | I mean I added the local.conf info to the bug | 15:34 |
| sean-k-mooney | ok so enabligh the tls proxy or is there more | 15:35 |
| gibi | tls proxy and forcing devstack to configure ssl for the novncproxy | 15:35 |
| sean-k-mooney | oh you need NOVA_CONSOLE_PROXY_COMPUTE_TLS=True as well | 15:35 |
| gibi | USE_SSL=True | 15:35 |
| gibi | ENABLED_SERVICES+=,tls-proxy | 15:35 |
| gibi | GLANCE_STANDALONE=False | 15:35 |
| gibi | NOVA_CONSOLE_PROXY_COMPUTE_TLS=True | 15:35 |
| sean-k-mooney | yep just read the bug | 15:35 |
| sean-k-mooney | given you have repoduced it i thnk its safe to move to confirmed | 15:37 |
| sean-k-mooney | critical instead of high is because its a gate blocker right | 15:38 |
| gibi | right, it is a gate blocker, so I set it to critical | 15:39 |
| gibi | and now it is in confirmed state too | 15:40 |
| * gibi going backward in the nova repo trying to find a commit where novncproxy still works | 15:46 | |
| *** gyee has joined #openstack-nova | 15:51 | |
| melwitt | gibi: saw your latest comment on the lp bug, fwiw urllib* is mentioned on this old eventlet issue (linked by sean-k-mooney) too. so I wonder if patching it/not patching it is related | 15:54 |
| melwitt | https://github.com/eventlet/eventlet/issues/201 | 15:54 |
| gibi | melwitt: I think something changed the import order and now urllib3 in imported before monkey_patch | 15:54 |
| melwitt | ahh | 15:55 |
| melwitt | I might be remembering it wrong but I want to say that was done intentionally a long time ago.. /me looks | 15:55 |
| sean-k-mooney | dont we have a log message for that | 15:56 |
| gibi | could be that I'm mixing that up | 15:56 |
| gibi | sean-k-mooney: yes, and that is what I find as a difference between the failing and the passing job | 15:56 |
| sean-k-mooney | ah ok | 15:56 |
| melwitt | oh wait sorry I misunderstood the original change | 15:57 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Handle VIR_ERR_DEVICE_MISSING when detaching devices https://review.opendev.org/741561 | 15:57 |
| lyarwood | melwitt: ^ losing my mind with this, hopefully it's slightly better this time around. | 15:57 |
| melwitt | this is what I was looking at the guts of the log message https://github.com/openstack/nova/blob/b7161fe9b92f0045e97c300a80e58d32b6f49be1/nova/monkey_patch.py#L39 | 15:57 |
| melwitt | this made me think we import urllib3 before patching intentionally? | 15:58 |
| gibi | this is the current import order http://paste.openstack.org/show/796133/ | 16:00 |
| melwitt | lyarwood: haha ok. sorry about that, not trying to cause headaches but if all 4 errors are fair game for both live/inactive detach failures then it seemed to me like all possible combos should have test cases. that's why I was asking are we holding to what the code comments say, only some errors are for live and only some are for inactive | 16:00 |
| *** hamalq has quit IRC | 16:01 | |
| gibi | it seems eventlet imports greendns that imports requests that imports urllib3 | 16:01 |
| *** hamalq has joined #openstack-nova | 16:01 | |
| melwitt | I see, I was about to say I didn't see us importing urllib3 directly anywhere | 16:02 |
| lyarwood | melwitt: yeah sorry wasn't a complaint about your review, just a rant :) | 16:02 |
| melwitt | lyarwood: heh ok :) | 16:02 |
| gibi | the passing job uses dnspython===1.16.0 the failing one uses dnspython===2.0.0 | 16:03 |
| gibi | and the dns package is in the import trace aboce | 16:03 |
| gibi | I'm getting close... | 16:04 |
| gibi | yepp pulling dnspython back to 1.16.0 resolves the novnc issue locally | 16:06 |
| gibi | this was the commit bumping that version during the weekned https://review.opendev.org/#/c/741800/ | 16:07 |
| *** hamalq_ has joined #openstack-nova | 16:07 | |
| melwitt | gibi: ahhh nice find! I don't think I ever would have figured it out 😆 | 16:08 |
| gibi | melwitt: it was a shoot in the dark based on the import trace | 16:10 |
| *** hamalq has quit IRC | 16:11 | |
| gibi | should I propose a pin in the global upper-constraints to fix it? | 16:11 |
| sean-k-mooney | ah that was one of the packages that did change between the runs | 16:11 |
| *** yoctozepto has quit IRC | 16:12 | |
| *** yoctozepto has joined #openstack-nova | 16:13 | |
| melwitt | yeah I'm not sure what else we could do, either pin globally or pin nova. I'd ask the requirements peeps: prometheanfire or smcginnis | 16:13 |
| sean-k-mooney | paste.openstack.org/show/796119/ | 16:13 |
| prometheanfire | wat | 16:13 |
| sean-k-mooney | so we went form 1.16.0 to 2.0.0 | 16:13 |
| gibi | prometheanfire: dnspython 1.16.0 -> 2.0.0 bump broke nova novncproxy | 16:14 |
| melwitt | the new dnspython 2.0.0 broke our novnc ssl stuff | 16:14 |
| melwitt | in a way that seems to not be our fault | 16:14 |
| melwitt | https://bugs.launchpad.net/nova/+bug/1888237 | 16:14 |
| openstack | Launchpad bug 1888237 in OpenStack Compute (nova) "nova-next job fails as novnc service fails with TypeError: _wrap_socket() argument 1 must be _socket.socket, not GreenSSLSocket" [Critical,Triaged] | 16:14 |
| prometheanfire | upstream bug in novnc? | 16:14 |
| gibi | prometheanfire: more like a change in behavior (import order) in dnspython | 16:14 |
| gibi | prometheanfire: that breaks eventlet monkey patching for nova | 16:14 |
| *** k_mouza has quit IRC | 16:15 | |
| prometheanfire | ok, upstream bug in dnspython? | 16:15 |
| prometheanfire | I just want to make sure that if/when we mask this we refrence the right bug :D | 16:15 |
| gibi | prometheanfire: I have no idea if dnspython intenationally change things in 2.0.0 that does not work with eventlet | 16:16 |
| gibi | there is a related eventlet issue is https://github.com/eventlet/eventlet/issues/619 | 16:16 |
| gibi | eventlet is incompatible with dnspython 2.0.0rc1 | 16:16 |
| prometheanfire | my main requirement (hehe, get it) is that we refrence when we can remove or relax the masking | 16:17 |
| gibi | prometheanfire: OK, I will reference the above eventlet issue | 16:17 |
| prometheanfire | the eventlet bug looks promising | 16:17 |
| gibi | prometheanfire: so, can I pin dnspython to 1.16.0 in the requirements repo by referencing the eventlet bug? | 16:18 |
| prometheanfire | ya, eventlet deps on dnspython, so the bug is in eventlet | 16:19 |
| prometheanfire | gibi: and mask dnspython 2.0.0 | 16:19 |
| gibi | prometheanfire: ack, will propose a patch soon | 16:20 |
| *** xiaolin has quit IRC | 16:21 | |
| *** xiaolin has joined #openstack-nova | 16:22 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova-specs master: DNM: testing focal https://review.opendev.org/741984 | 16:23 |
| sean-k-mooney | gibi do you want to update https://github.com/eventlet/eventlet/issues/632 | 16:24 |
| gibi | sean-k-mooney: I will do | 16:24 |
| gibi | sean-k-mooney: I duplicate it to 619 | 16:24 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Track blockjob status in DEBUG when polling for completion https://review.opendev.org/729595 | 16:25 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove blockjob.end == 0 workaround resolved in libvirt v2.3.0 https://review.opendev.org/729596 | 16:25 |
| gibi | sean-k-mooney: I think I have no rights to mark it as duplicate but I left a comment stating it | 16:26 |
| sean-k-mooney | i can close it | 16:26 |
| gibi | go ahead | 16:27 |
| sean-k-mooney | just commenting on the other bug is enough to cross link them so ya its closed | 16:27 |
| gibi | cool, thanks | 16:27 |
| *** udesale_ has quit IRC | 16:28 | |
| *** dtantsur is now known as dtantsur|afk | 16:29 | |
| *** ociuhandu_ has joined #openstack-nova | 16:30 | |
| gibi | sean-k-mooney, prometheanfire, melwitt, lyarwood: pushed https://review.opendev.org/#/c/741986/ | 16:32 |
| sean-k-mooney | 27 secods ago | 16:32 |
| sean-k-mooney | oh you pushed | 16:32 |
| sean-k-mooney | i missread | 16:33 |
| sean-k-mooney | i thougt you ment lyarwood pushed it and you noticed and were telling us | 16:33 |
| gibi | sorry :) | 16:33 |
| *** ociuhandu has quit IRC | 16:33 | |
| *** ociuhandu_ has quit IRC | 16:34 | |
| openstackgerrit | Huaqiang Wang proposed openstack/nova master: compute: bump nova-compute version and check in API https://review.opendev.org/740933 | 16:39 |
| openstackgerrit | Huaqiang Wang proposed openstack/nova master: hardware: Enable 'hw:cpu_dedicated_mask' for creating a mixed instance https://review.opendev.org/714706 | 16:39 |
| openstackgerrit | Huaqiang Wang proposed openstack/nova master: hardware: create 'mixed' instance for realtime CPUs https://review.opendev.org/728485 | 16:39 |
| openstackgerrit | Huaqiang Wang proposed openstack/nova master: metadata: export the vCPU IDs that are pinning on the host CPUs https://review.opendev.org/688936 | 16:39 |
| openstackgerrit | Huaqiang Wang proposed openstack/nova master: release note: add new features introduces by use-pcpu-and-vcpu-in-one-instance https://review.opendev.org/736997 | 16:39 |
| openstackgerrit | Huaqiang Wang proposed openstack/nova master: test: add some notes and new tests for mixed instance https://review.opendev.org/741395 | 16:39 |
| *** xiaolin has quit IRC | 16:40 | |
| gibi | OK, I'm ending my workday see you tomorrow | 16:44 |
| melwitt | o/ | 16:47 |
| *** jraju__ has quit IRC | 16:49 | |
| *** jangutter has quit IRC | 16:58 | |
| *** derekh has quit IRC | 17:00 | |
| *** nweinber has quit IRC | 17:02 | |
| *** jangutter has joined #openstack-nova | 17:03 | |
| *** nweinber has joined #openstack-nova | 17:03 | |
| *** brinzhang_ has joined #openstack-nova | 17:12 | |
| *** brinzhang0 has quit IRC | 17:15 | |
| *** dklyle has quit IRC | 17:21 | |
| *** dklyle has joined #openstack-nova | 17:21 | |
| *** jsuchome has quit IRC | 17:32 | |
| *** ociuhandu has joined #openstack-nova | 17:34 | |
| *** brinzhang0 has joined #openstack-nova | 17:37 | |
| *** ociuhandu has quit IRC | 17:38 | |
| *** brinzhang_ has quit IRC | 17:40 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support for resize and cold migration of emulated TPM files https://review.opendev.org/639934 | 17:42 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Don't unset Instance.old_flavor, new_flavor until necessary https://review.opendev.org/741995 | 17:42 |
| *** brinzhang_ has joined #openstack-nova | 17:43 | |
| stephenfin | huaqiang: Have +2d those two patches. However, we're having issues with the gate so I wouldn't recheck yet | 17:45 |
| * stephenfin -> 🏃 | 17:45 | |
| *** brinzhang0 has quit IRC | 17:46 | |
| *** ralonsoh has quit IRC | 17:48 | |
| *** nightmare_unreal has quit IRC | 17:56 | |
| *** brinzhang0 has joined #openstack-nova | 17:56 | |
| *** tesseract has quit IRC | 17:58 | |
| *** brinzhang_ has quit IRC | 17:59 | |
| *** brinzhang_ has joined #openstack-nova | 18:00 | |
| *** brinzhang0 has quit IRC | 18:03 | |
| *** brinzhang0 has joined #openstack-nova | 18:26 | |
| *** brinzhang has joined #openstack-nova | 18:28 | |
| *** brinzhang_ has quit IRC | 18:29 | |
| *** brinzhang0 has quit IRC | 18:31 | |
| *** artom has quit IRC | 18:38 | |
| *** spatel has quit IRC | 18:42 | |
| *** artom has joined #openstack-nova | 18:52 | |
| *** tosky has quit IRC | 18:57 | |
| *** xek_ has joined #openstack-nova | 19:02 | |
| *** maciejjozefczyk has quit IRC | 19:33 | |
| *** ociuhandu has joined #openstack-nova | 19:36 | |
| *** vishalmanchanda has quit IRC | 19:41 | |
| *** brinzhang_ has joined #openstack-nova | 19:59 | |
| *** brinzhang has quit IRC | 20:02 | |
| *** ociuhandu has quit IRC | 20:06 | |
| *** dave-mccowan has joined #openstack-nova | 20:47 | |
| *** bnemec has quit IRC | 20:55 | |
| *** markvoelker has joined #openstack-nova | 20:58 | |
| *** markvoelker has quit IRC | 21:02 | |
| *** nweinber has quit IRC | 21:04 | |
| *** bnemec has joined #openstack-nova | 21:10 | |
| *** nweinber has joined #openstack-nova | 21:11 | |
| *** raildo has quit IRC | 21:25 | |
| *** nweinber has quit IRC | 21:31 | |
| *** brinzhang_ has quit IRC | 21:33 | |
| *** brinzhang0 has joined #openstack-nova | 21:33 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles in FIP policies https://review.opendev.org/742030 | 21:34 |
| *** jdillaman has quit IRC | 21:35 | |
| *** xek_ has quit IRC | 21:40 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles in baremetal nodes policies https://review.opendev.org/741716 | 21:43 |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Pass the actual target in baremetal nodes policy https://review.opendev.org/741717 | 21:43 |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Correct the check_str and pass actual target in FIP pools policy https://review.opendev.org/741797 | 21:53 |
| *** brinzhang_ has joined #openstack-nova | 22:00 | |
| *** brinzhang0 has quit IRC | 22:03 | |
| *** brinzhang0 has joined #openstack-nova | 22:08 | |
| *** brinzhang_ has quit IRC | 22:11 | |
| *** martinkennelly has quit IRC | 22:32 | |
| *** fumesover32 has joined #openstack-nova | 22:39 | |
| *** fumesover3 has quit IRC | 22:39 | |
| *** tkajinam has joined #openstack-nova | 22:53 | |
| *** markvoelker has joined #openstack-nova | 22:53 | |
| *** rcernin has joined #openstack-nova | 22:58 | |
| *** mlavalle has quit IRC | 22:59 | |
| *** rcernin has quit IRC | 22:59 | |
| *** rcernin has joined #openstack-nova | 22:59 | |
| *** markvoelker has quit IRC | 23:03 | |
| *** artom has quit IRC | 23:05 | |
| *** artom has joined #openstack-nova | 23:05 | |
| *** bnemec has quit IRC | 23:20 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!