| johnsom | rm_work if you want me to have a look at the DIB output log, let me know. | 00:00 |
|---|---|---|
| *** ducttape_ has quit IRC | 00:01 | |
| rm_work | this is the ubuntu image | 00:02 |
| rm_work | i'm working on it | 00:02 |
| rm_work | i mean, i see where it installed requests, definitely py3 as it should be... | 00:02 |
| rm_work | and cloud-init is def running in py3 too | 00:02 |
| rm_work | so that should be fine | 00:02 |
| rm_work | uhh | 00:09 |
| rm_work | it's there in /usr/lib/python3.5/site-packages/requests | 00:10 |
| rm_work | but i can't import it | 00:10 |
| rm_work | ah | 00:14 |
| rm_work | that folder isn't being looked at | 00:14 |
| rm_work | it's looking in /usr/local/lib/python3.5/dist-packages, /usr/lib/python3/dist-packages, and /usr/lib/python3.5/dist-packages | 00:15 |
| rm_work | but everything is installed in site-packages | 00:15 |
| rm_work | umm | 00:15 |
| kristianjf | jonsom and xgerman. thanks for all your help. greatly appreciated. | 00:17 |
| johnsom | Sure, hope we have what you need | 00:18 |
| rm_work | so when i build the image on a different box, the stuff goes into the right place (dist-packages) | 00:18 |
| rm_work | when I build the image on this box, it goes into site-packages?? | 00:18 |
| rm_work | last successful build was a couple days ago though so maybe something changed upstream and it'd be broken now on my other buildbox too | 00:19 |
| johnsom | ???? Umm, that is strange | 00:19 |
| johnsom | Well, I know the gate passed at 12:52 | 00:20 |
| rm_work | hmm | 00:21 |
| rm_work | no idea how that could swap... greghaynes ? any ideas? | 00:22 |
| greghaynes | hrm? | 00:25 |
| rm_work | DIB is installing all the python packages into /usr/lib/python3.5/site-packages/ | 00:26 |
| rm_work | but python3 is looking in /usr/local/lib/python3.5/dist-packages, /usr/lib/python3/dist-packages, and /usr/lib/python3.5/dist-packages | 00:26 |
| rm_work | i looked at an image i built previously and everything is installed in dist-packages | 00:26 |
| rm_work | is there a setting I didn't set on this machine? | 00:27 |
| rm_work | the working one (installed in dist-packages) was built via devstack, this one I am doing myself | 00:27 |
| greghaynes | hrmmm, so site-packages should always be looked at. dist-pacakges is a debianism so it just looks there for debian installed things | 00:28 |
| greghaynes | stepping back a bit - what is the fail going on, youre seeing a traceback from cloud-init? | 00:28 |
| rm_work | yeah | 00:28 |
| rm_work | because it can't find `requests`` | 00:28 |
| rm_work | which makes sense, because when i log into the box via console, i run python3, i can't import requests | 00:29 |
| greghaynes | ok | 00:29 |
| rm_work | site.getsitepackages() returns only those dist-package locations | 00:29 |
| greghaynes | and is python2 installed on there? | 00:29 |
| rm_work | i ssh into a previously generated image, and i see the packages all installed in dist-packages | 00:29 |
| greghaynes | oh what | 00:29 |
| rm_work | yes | 00:29 |
| rm_work | (previously generated, on a different machine also) | 00:30 |
| greghaynes | yea so then the next question is - are you sure cloud-init is running under python3 not python2? | 00:30 |
| rm_work | yes | 00:30 |
| rm_work | py3 is clear from the stacktrace | 00:30 |
| greghaynes | thats messed up :p | 00:30 |
| greghaynes | so this should be 100% debian packages | 00:30 |
| greghaynes | since cloud-init is installed from that | 00:30 |
| rm_work | yeah but those libs are installed by DIB for our amphora as well | 00:31 |
| rm_work | and i see ALL of them | 00:31 |
| rm_work | aren't those done via pip? | 00:31 |
| rm_work | they're all there, just in the wrong directory | 00:31 |
| greghaynes | aha | 00:31 |
| rm_work | like, i watched pip uninstall an old `requests` (maybe in dist-packages?) and install a new one | 00:32 |
| greghaynes | so yea, pip uses site-packages and dpkg uses dist-packages. I suspect what goes on is when you pip install -U requests its rm'ing the dist-packages version when it upgrades requests and then is reinstalling in to site-packages | 00:32 |
| johnsom | Yeah, that should be happening here: https://review.openstack.org/#/c/409956/5/elements/amphora-agent/install.d/75-run_setup_install | 00:32 |
| *** ducttape_ has joined #openstack-lbaas | 00:32 | |
| rm_work | yeah so why didn't i see this issue before | 00:32 |
| greghaynes | normally this is fine (although really messy) but no idea why your python isnt looking at site-packages | 00:32 |
| rm_work | i mean, again, I just SSH'd into another amp, and the packages *are* in dist-packages | 00:33 |
| rm_work | all of them. | 00:33 |
| rm_work | maybe PIP on my system is older? | 00:34 |
| rm_work | is it possible that pip changed the way it installs? | 00:34 |
| greghaynes | I just confirmed my python3 does the same thing | 00:34 |
| greghaynes | possible, oh actually tehre was a new setuptools release I think | 00:34 |
| johnsom | DIB installs pip | 00:34 |
| rm_work | AHA | 00:34 |
| rm_work | so I bet that's way out of date | 00:34 |
| rm_work | err | 00:34 |
| rm_work | ok so it doesn't do anything related to my system version of any of those things? | 00:34 |
| rm_work | this is an *older* box | 00:35 |
| rm_work | like, a year older centos release | 00:35 |
| rm_work | early 2015 | 00:35 |
| johnsom | No, it pulls down a fresh one inside the environment | 00:35 |
| rm_work | hmm | 00:35 |
| greghaynes | yea taht all happens within chroot | 00:35 |
| rm_work | working on getting a log | 00:35 |
| greghaynes | you all use the ubuntu element, right> | 00:36 |
| greghaynes | not ubuntu-minimal? | 00:36 |
| johnsom | pip hasn't been update since 11/6 so not likely | 00:36 |
| rm_work | of course my VPN died and interupted it >_ | 00:36 |
| greghaynes | setuptools released last week | 00:36 |
| johnsom | Yeah, ubuntu | 00:36 |
| johnsom | Well, I know our gate built an image ~three hours ago without issue | 00:36 |
| greghaynes | rm_work: are you on this node ATM? | 00:37 |
| rm_work | yeah | 00:37 |
| rm_work | both the old build (working) and the new build (not) | 00:38 |
| greghaynes | can you paste some kind of traceback? | 00:39 |
| *** fnaval_ has quit IRC | 00:39 | |
| greghaynes | also, can you confirm that python3 -c "import site; print(site.getsitepackages())" is the same on both? | 00:39 |
| rm_work | lol i can't even pip because python can't find the pip module | 00:39 |
| rm_work | because it's in the wrong place | 00:39 |
| greghaynes | oh wow | 00:39 |
| greghaynes | so its stupid broken | 00:40 |
| greghaynes | I wonder if ubuntu changed their package | 00:40 |
| rm_work | doubtful? again, johnsom said he sees passing scenarios in the last three hours | 00:40 |
| rm_work | but | 00:40 |
| rm_work | who knows | 00:40 |
| rm_work | those might get cached packages | 00:40 |
| rm_work | and mine definitely doesn't | 00:40 |
| rm_work | what packages would i want to check | 00:40 |
| greghaynes | python3 I believe | 00:41 |
| rm_work | same | 00:42 |
| rm_work | 3.5.1-3 | 00:42 |
| rm_work | it's gotta be something about how DIB chooses to install the packages, right? | 00:43 |
| greghaynes | I mean all dib is doing is pip install | 00:44 |
| johnsom | Yes, it looks like they posted two images today | 00:44 |
| greghaynes | oh right | 00:44 |
| johnsom | http://cloud-images.ubuntu.com/xenial/ | 00:44 |
| greghaynes | so its possible their images are just broken | 00:44 |
| johnsom | I wonder if you were "unlucky" | 00:44 |
| rm_work | urhg | 00:44 |
| rm_work | I get "unlucky" *a lot* | 00:45 |
| johnsom | I have no idea how to tell what changed in those short of pulling them down | 00:45 |
| rm_work | i'll clear the image in my cache | 00:45 |
| greghaynes | ok, I have to run (wifes birthday tonight so really cant skip out on that) but I can poke tomorrow morning if you dont dig anything up | 00:45 |
| rm_work | well, first i'll see which it is | 00:45 |
| rm_work | greghaynes: i'll let you know | 00:45 |
| greghaynes | I do think the dist-packages thing is a red herring though - my system has ['', '/usr/lib/python35.zip', '/usr/lib/python3.5', '/usr/lib/python3.5/plat-x86_64-linux-gnu', '/usr/lib/python3.5/lib-dynload', '/usr/local/lib/python3.5/dist-packages', '/usr/lib/python3/dist-packages'] for sys.path and seems to work fine for stuff I pip install | 00:45 |
| greghaynes | I have no idea how, but it seems to | 00:46 |
| rm_work | maybe my python path is borked? | 00:47 |
| greghaynes | very possible | 00:47 |
| rm_work | yeah i have the "current" image | 00:48 |
| rm_work | hmm | 00:48 |
| *** ducnc has quit IRC | 00:50 | |
| rm_work | ok well | 00:50 |
| *** ducnc has joined #openstack-lbaas | 00:50 | |
| rm_work | I just did `ln -s /usr/lib/python3.5/site-packages /usr/lib/python3.5/dist-packages` and now it seems fine | 00:50 |
| rm_work | ... i'm gonna ... throw that line into one of our post-inst element scripts and just call it good for now until someone else runs into this | 00:51 |
| rm_work | because i'm tired of things not working >_< | 00:51 |
| johnsom | You and me both, this cycle seems like it has been rough on us. | 00:52 |
| *** harlowja has joined #openstack-lbaas | 01:06 | |
| *** ducttape_ has quit IRC | 01:10 | |
| *** ducttape_ has joined #openstack-lbaas | 01:11 | |
| *** gongysh has joined #openstack-lbaas | 01:15 | |
| *** ducttape_ has quit IRC | 01:15 | |
| *** ducttape_ has joined #openstack-lbaas | 01:37 | |
| *** fnaval has joined #openstack-lbaas | 01:40 | |
| *** fnaval_ has joined #openstack-lbaas | 01:41 | |
| *** cody-somerville has quit IRC | 01:42 | |
| *** cody-somerville has joined #openstack-lbaas | 01:42 | |
| *** cody-somerville has quit IRC | 01:42 | |
| *** cody-somerville has joined #openstack-lbaas | 01:42 | |
| *** harlowja has quit IRC | 01:44 | |
| *** fnaval has quit IRC | 01:45 | |
| *** ducnc has quit IRC | 01:47 | |
| *** ducnc has joined #openstack-lbaas | 01:47 | |
| *** ducttape_ has quit IRC | 01:56 | |
| *** ducttape_ has joined #openstack-lbaas | 01:57 | |
| rm_work | ugh and now i don't get networking because the interface is named ens3 instead of eth0 for some reason | 02:12 |
| johnsom | Oh, that is bad/strange. So, I do a bit of "magic" with that | 02:12 |
| johnsom | Since we get the mac addresses, as I move them into the namespace I rename them to the "old" style eth* names | 02:13 |
| rm_work | this is during cloud-init though | 02:13 |
| rm_work | it's trying to bring up eth0 | 02:13 |
| rm_work | i think that's something with this cloud | 02:13 |
| johnsom | So, they should be the new names outside the namespace and get renamed as they move into the namespace. | 02:13 |
| rm_work | I'm going to change the udev rules to name them via the old style | 02:14 |
| johnsom | Oh, ok, yeah, ummmm, not sure on that. Is something injecting it's own cloud-init config???? | 02:14 |
| rm_work | yeah i think so | 02:14 |
| johnsom | Umm, hmm, that could go poorly | 02:15 |
| *** gongysh has quit IRC | 02:15 | |
| rm_work | I think it won't | 02:15 |
| rm_work | when it's spun via octavia | 02:15 |
| johnsom | I know there is some stuff in cloud-init itself that plays with the interface names. But I usually can ignore that for the management lan | 02:15 |
| rm_work | this is me futzing with booting it manually | 02:15 |
| johnsom | Ah, yeah, ok | 02:15 |
| johnsom | Ok, need to run to dinner. Catch you tomorrow | 02:16 |
| rm_work | so maybe i just need to test that way | 02:16 |
| rm_work | yeah, later :) | 02:16 |
| rm_work | crap where did those octavia curl commands go... | 02:17 |
| *** ducttape_ has quit IRC | 02:17 | |
| rm_work | sbalukoff: do you still have that | 02:18 |
| rm_work | found it | 02:19 |
| *** fnaval_ has quit IRC | 02:26 | |
| *** gongysh has joined #openstack-lbaas | 02:29 | |
| rm_work | wow our config in devstack is wonky | 02:42 |
| rm_work | everything is duplicated between [keystone_authtoken] and [service_auth] | 02:42 |
| rm_work | and even within sections there's stuff duplicated | 02:42 |
| greghaynes | heh, so other folks seem to be hitting the same issue you were WRT dist-packages | 02:50 |
| greghaynes | I suspect if we dont root cause it all openstack is likely to be failing tomorrow when images update | 02:50 |
| greghaynes | so, horray you all won the first to fail | 02:50 |
| rm_work | i win that OFTEN | 02:52 |
| rm_work | not kidding | 02:52 |
| rm_work | i wish I knew which section of this config it's actually using | 02:53 |
| rm_work | well whatever, made progress, my eyes are starting to blur. gonna get food | 02:54 |
| rm_work | i forgot to eat today | 02:54 |
| greghaynes | you should eat every day | 02:56 |
| rm_work | AUGH | 03:13 |
| rm_work | the amphora-agent binary also went to /usr/bin instead of /usr/local/bin | 03:13 |
| rm_work | son of a @*()$YRT#F(P*Fc | 03:14 |
| rm_work | why is this happening to me | 03:14 |
| *** ducttape_ has joined #openstack-lbaas | 03:18 | |
| *** kristianjf has quit IRC | 03:19 | |
| *** fnaval has joined #openstack-lbaas | 03:21 | |
| *** ducttape_ has quit IRC | 03:23 | |
| *** cody-somerville has quit IRC | 03:28 | |
| *** links has joined #openstack-lbaas | 03:36 | |
| *** yamamot__ has joined #openstack-lbaas | 03:40 | |
| *** fnaval_ has joined #openstack-lbaas | 04:32 | |
| *** fnaval has quit IRC | 04:35 | |
| *** ankur-gupta-f1 has joined #openstack-lbaas | 04:41 | |
| *** kevo has joined #openstack-lbaas | 04:45 | |
| *** ducttape_ has joined #openstack-lbaas | 04:49 | |
| *** ducttape_ has quit IRC | 04:53 | |
| *** adam_g has joined #openstack-lbaas | 05:47 | |
| *** kevo has quit IRC | 05:57 | |
| *** kevo has joined #openstack-lbaas | 05:57 | |
| *** fnaval_ has quit IRC | 06:14 | |
| *** ducttape_ has joined #openstack-lbaas | 06:19 | |
| *** ducttape_ has quit IRC | 06:24 | |
| *** armax has joined #openstack-lbaas | 06:39 | |
| *** tesseract has joined #openstack-lbaas | 07:08 | |
| *** tesseract is now known as Guest29114 | 07:09 | |
| *** armax has quit IRC | 07:34 | |
| *** pcaruana has joined #openstack-lbaas | 07:34 | |
| *** armax has joined #openstack-lbaas | 07:36 | |
| openstackgerrit | Abed Abu dbai proposed openstack/octavia: ACTIVE-ACTIVE - controller network tasks https://review.openstack.org/323481 | 07:41 |
| *** armax has quit IRC | 07:41 | |
| *** openstackgerrit has quit IRC | 07:48 | |
| *** ducttape_ has joined #openstack-lbaas | 07:50 | |
| *** bana_k has joined #openstack-lbaas | 07:50 | |
| *** nmagnezi has joined #openstack-lbaas | 07:53 | |
| *** ducttape_ has quit IRC | 07:54 | |
| *** rcernin has joined #openstack-lbaas | 08:06 | |
| *** zhouyaguo has quit IRC | 08:08 | |
| *** ihrachys has joined #openstack-lbaas | 08:09 | |
| *** amoralej|off is now known as amoralej | 08:24 | |
| *** anilvenkata has joined #openstack-lbaas | 08:25 | |
| *** yatin is now known as yatin_away | 08:26 | |
| *** ihrachys has quit IRC | 08:36 | |
| greghaynes | rm_work: johnsom https://review.openstack.org/#/c/413487/ | 08:48 |
| *** bana_k has quit IRC | 08:48 | |
| greghaynes | rm_work: ^ this is the root cause of your latest issue and your previous import errors | 08:48 |
| greghaynes | That was super fun to track down | 08:49 |
| *** ducttape_ has joined #openstack-lbaas | 08:50 | |
| *** ducttape_ has quit IRC | 08:55 | |
| *** ducttape_ has joined #openstack-lbaas | 09:00 | |
| *** ducttape_ has quit IRC | 09:05 | |
| *** rcernin has quit IRC | 09:11 | |
| *** chlong has quit IRC | 09:34 | |
| *** ihrachys has joined #openstack-lbaas | 09:51 | |
| *** ducttape_ has joined #openstack-lbaas | 10:02 | |
| *** ducttape_ has quit IRC | 10:07 | |
| *** ducnc has quit IRC | 10:09 | |
| *** ducnc has joined #openstack-lbaas | 10:09 | |
| *** kobis has joined #openstack-lbaas | 10:49 | |
| *** ducttape_ has joined #openstack-lbaas | 11:02 | |
| *** ducttape_ has quit IRC | 11:07 | |
| *** gongysh has quit IRC | 11:13 | |
| *** ducttape_ has joined #openstack-lbaas | 12:03 | |
| *** ducttape_ has quit IRC | 12:08 | |
| *** rcernin has joined #openstack-lbaas | 12:26 | |
| *** pcaruana has quit IRC | 12:49 | |
| *** ducttape_ has joined #openstack-lbaas | 13:04 | |
| *** ducttape_ has quit IRC | 13:06 | |
| *** ducttape_ has joined #openstack-lbaas | 13:06 | |
| *** ducttape_ has quit IRC | 13:07 | |
| *** ducttape_ has joined #openstack-lbaas | 13:23 | |
| *** gongysh has joined #openstack-lbaas | 13:30 | |
| *** pcaruana has joined #openstack-lbaas | 13:32 | |
| *** gongysh has quit IRC | 13:34 | |
| *** armax has joined #openstack-lbaas | 13:36 | |
| *** armax has quit IRC | 13:49 | |
| *** ducttape_ has quit IRC | 13:51 | |
| xgerman | o/ | 13:56 |
| *** benoit has quit IRC | 14:21 | |
| *** benoit has joined #openstack-lbaas | 14:36 | |
| *** pcaruana has quit IRC | 14:38 | |
| *** links has quit IRC | 14:49 | |
| *** pcaruana has joined #openstack-lbaas | 14:51 | |
| *** ihrachys has quit IRC | 15:04 | |
| *** amotoki has quit IRC | 15:14 | |
| *** reedip_ has joined #openstack-lbaas | 15:15 | |
| *** fnaval has joined #openstack-lbaas | 15:22 | |
| *** yamamot__ has quit IRC | 15:23 | |
| *** anilvenkata has quit IRC | 15:26 | |
| *** cody-somerville has joined #openstack-lbaas | 15:26 | |
| *** csomerville has joined #openstack-lbaas | 15:29 | |
| *** cody-somerville has quit IRC | 15:32 | |
| *** yamamoto has joined #openstack-lbaas | 15:41 | |
| *** openstackgerrit_ has joined #openstack-lbaas | 15:48 | |
| *** openstackgerrit_ has quit IRC | 15:49 | |
| *** kobis1 has joined #openstack-lbaas | 15:49 | |
| *** kobis has quit IRC | 15:50 | |
| *** ihrachys has joined #openstack-lbaas | 15:51 | |
| *** chlong has joined #openstack-lbaas | 15:52 | |
| *** nmagnezi_ has joined #openstack-lbaas | 15:56 | |
| *** bana_k has joined #openstack-lbaas | 15:57 | |
| *** nmagnezi has quit IRC | 15:58 | |
| *** openstackgerrit has joined #openstack-lbaas | 16:03 | |
| openstackgerrit | German Eichberger proposed openstack/octavia: Add tempesttest for the N-LBaaS API https://review.openstack.org/413697 | 16:03 |
| *** kobis1 has quit IRC | 16:06 | |
| *** bana_k has quit IRC | 16:08 | |
| *** pcaruana has quit IRC | 16:13 | |
| ankur-gupta-f1 | diltram: you got that etherpad link for the ocata merge priorities? | 16:13 |
| *** Guest29114 has quit IRC | 16:15 | |
| *** benoit has quit IRC | 16:18 | |
| *** benoit has joined #openstack-lbaas | 16:19 | |
| openstackgerrit | German Eichberger proposed openstack/octavia: Introduct Test Base classes for V2 API https://review.openstack.org/405599 | 16:21 |
| diltram | ankur-gupta-f1: yes, you need it? | 16:23 |
| *** ducttape_ has joined #openstack-lbaas | 16:23 | |
| diltram | https://etherpad.openstack.org/p/octavia-ocata-merge-review-priority - this one | 16:23 |
| *** benoit has quit IRC | 16:24 | |
| ankur-gupta-f1 | yup. Just adding German's patchset to it | 16:26 |
| *** ducttape_ has quit IRC | 16:27 | |
| *** yamamoto has quit IRC | 16:41 | |
| *** ankur-gupta-f1 has quit IRC | 16:42 | |
| *** diogogmt has joined #openstack-lbaas | 16:43 | |
| *** ankur-gupta-f1 has joined #openstack-lbaas | 16:45 | |
| johnsom | gerrit is slow today | 16:48 |
| ankur-gupta-f1 | indeed | 16:49 |
| *** reedip_ has quit IRC | 16:49 | |
| openstackgerrit | German Eichberger proposed openstack/octavia: Add tempesttest for the N-LBaaS API https://review.openstack.org/413697 | 16:51 |
| *** Kiall has quit IRC | 16:53 | |
| *** openstack has quit IRC | 17:02 | |
| *** openstack has joined #openstack-lbaas | 17:07 | |
| *** rcernin has quit IRC | 17:10 | |
| *** yamamoto has joined #openstack-lbaas | 17:12 | |
| *** yamamoto has quit IRC | 17:21 | |
| *** ducttape_ has joined #openstack-lbaas | 17:23 | |
| *** ihrachys has quit IRC | 17:31 | |
| rm_work | greghaynes: at least you had a head start :P | 17:31 |
| rm_work | wasn't that a change that someone JUST added yesterday? | 17:32 |
| rm_work | greghaynes: uhh also what the fuck https://review.openstack.org/#/c/413410/ | 17:32 |
| rm_work | err, sorry, language... | 17:32 |
| rm_work | greghaynes: we going to revert that when yours lands? | 17:33 |
| johnsom | What a hairball | 17:34 |
| greghaynes | yes | 17:35 |
| greghaynes | we were a bit concerned that all of openstack was going to break | 17:35 |
| greghaynes | and wanted something quick | 17:35 |
| rm_work | >_< | 17:35 |
| *** ducttape_ has quit IRC | 17:36 | |
| greghaynes | and yes, python2 - 3 is a giant pain | 17:36 |
| rm_work | ok so | 17:36 |
| rm_work | who do I have to bribe/murder/blackmail to make this happen quickly | 17:36 |
| rm_work | whatever i'll just do it myself locally | 17:37 |
| rm_work | sorry I am frustrated because I am *pretty sure* that I am starting to actually believe in curses, and that I have one | 17:42 |
| *** ducttape_ has joined #openstack-lbaas | 17:44 | |
| johnsom | Yeah, in reality the issue is with coordinating major shifts like this. There should be clear statements from OpenStack about what versions and releases are expected to work in each cycle, including a timeline when changes like this are going to occur. | 17:45 |
| xgerman | +1 | 17:45 |
| greghaynes | I'm not sure any statements or anything are really the issue though - you still have to tackle the change? | 17:47 |
| johnsom | Yes, but we could have been ready for it instead of halting work on the project for 2+ weeks! | 17:47 |
| greghaynes | or said differently, I think the statement is you have to py3 next cycle, so we'd jsut be solving these issues in a few weeks | 17:47 |
| rm_work | I might have to agree with greghaynes on this | 17:47 |
| johnsom | reverts == not prepared | 17:48 |
| rm_work | we know py3 has to work... we really wouldn't have fixed it completely until we were forced into getting this working | 17:48 |
| rm_work | this revert is really about some VENV setting change | 17:48 |
| rm_work | not py2->3 | 17:48 |
| greghaynes | I think reverts == bugs, not quite the same. The issue is really a lack of testing coverage which is just the reality when you have a huge support matrix | 17:49 |
| rm_work | reverting that seems like it was a hack | 17:49 |
| greghaynes | I am pretty confident that no matter how long ago I was told I would have made that same mistake because this behavior is just crazy and unexpected | 17:49 |
| johnsom | "reverting progress towards py3 because it breaks the world" | 17:50 |
| greghaynes | so, for you all, if youd like to not deal with the python3 stuff you should be able to pin yourself to python2 very easily | 17:51 |
| greghaynes | the sort of root bug in all of this is we never intended to change the default python on xenial, were supposed to just provide the ubuntu defaults | 17:51 |
| greghaynes | and the ubuntu default changed to python3 | 17:52 |
| johnsom | I understand it is a bug, but it's hugely frustrating on the lack of planning and coordination on the whole transition. I can't point to a wiki page that says, projects should be on this OS release in this cycle, and we will transition gate hosts on this day, then we will transition image builds to default differently on this date. Same with python2 -> 3 | 17:52 |
| rm_work | again, what broke the world is the VENV change | 17:54 |
| greghaynes | ah. I think the way that works is we try to use whatever is the latest supported of the distros, and leave it up to them to decide what dependencies look like. Just for practical reasons it doesnt end up being immediate because it takes time to switch nodes and iron out bugs | 17:54 |
| rm_work | the py3 revert was ... maybe a snap reaction? | 17:54 |
| greghaynes | so its not so much tied to our release cycle as the upstream distro release cycle | 17:54 |
| johnsom | But we didn't. We were still running on py2 after the xenial switch. | 17:55 |
| johnsom | Anyway I am done venting my frustration | 17:55 |
| rm_work | i'm not done venting mine, i'll be doing it the rest of the week probably... it's just slightly less directed | 17:56 |
| johnsom | Ha | 17:56 |
| rm_work | it's seriously almost FUNNY now | 17:56 |
| rm_work | how every time i'm about to make progress on something | 17:56 |
| rm_work | something breaks upstream | 17:56 |
| *** ducttape_ has quit IRC | 17:57 | |
| greghaynes | I think the larger concept is - we dont get to pin all the dependencies in the world. Even if we wanted, if a distro changes a package to something else there is literally 0 we can do except beg them to change it back | 17:57 |
| greghaynes | sometimes we have control in the case of a major release | 17:57 |
| johnsom | I'm not yet to that phase of grieving for the weeks lost in Ocata. | 17:57 |
| johnsom | Yeah, but we have plans and RC's to test with the distros | 17:57 |
| greghaynes | but really we have decided we dont want that - the goal is to let the distros handle it and we should just work on whatever the distros are doing | 17:58 |
| greghaynes | right, which is why we do all this CI - so we dont release in the broken state but we do have to test on latest packages otherwise we end up having requirements other than 'we run on ubuntu' | 17:58 |
| johnsom | At least these changes didn't happen in o-3 | 17:59 |
| greghaynes | yea, and really the correct way to handle this IMO would have been 'were not ready for xenial yet' | 18:00 |
| johnsom | Well, I guess they kind of did | 18:00 |
| *** ducttape_ has joined #openstack-lbaas | 18:00 | |
| greghaynes | it really should have been caught there, but, bugs | 18:00 |
| *** ihrachys has joined #openstack-lbaas | 18:01 | |
| greghaynes | I also forsee next cycle being mostly this :p I think you all are going to be one of the lesser painful service vm transitions | 18:02 |
| greghaynes | not looking forward to some of the other ones | 18:02 |
| *** anilvenkata has joined #openstack-lbaas | 18:03 | |
| greghaynes | rm_work: btw, did you end up merging the symlink site-packages deal? | 18:03 |
| *** ihrachys has quit IRC | 18:03 | |
| greghaynes | if so youll probably want to propose a revert that depends-on https://review.openstack.org/#/c/413487/ | 18:03 |
| rm_work | greghaynes: no i just did it locally | 18:04 |
| greghaynes | ah ok | 18:04 |
| rm_work | but i did revert it and grab your patch just now | 18:04 |
| *** ihrachys has joined #openstack-lbaas | 18:05 | |
| johnsom | Yeah, that is a bit hack-ish to merge | 18:05 |
| greghaynes | at around midnight last night when we were all scratching our heads at what on earth was going on we definitely considered doing it | 18:06 |
| *** ihrachys has quit IRC | 18:06 | |
| rm_work | johnsom: still working on the ens3 issue | 18:11 |
| johnsom | rm_work If you want you can send me your console.log and I can compare to one of mine. See where cloud-init is deviating | 18:12 |
| rm_work | ok well everything else looks good | 18:14 |
| rm_work | besides that | 18:14 |
| *** ducttape_ has quit IRC | 18:18 | |
| *** ducttape_ has joined #openstack-lbaas | 18:30 | |
| *** armax has joined #openstack-lbaas | 18:33 | |
| *** pcaruana has joined #openstack-lbaas | 18:34 | |
| *** bana_k has joined #openstack-lbaas | 18:56 | |
| *** amoralej is now known as amoralej|off | 18:56 | |
| *** armax has quit IRC | 19:00 | |
| *** ducttape_ has quit IRC | 19:00 | |
| *** anilvenkata has quit IRC | 19:06 | |
| *** ihrachys has joined #openstack-lbaas | 19:10 | |
| *** ducttape_ has joined #openstack-lbaas | 19:35 | |
| *** bana_k has quit IRC | 19:47 | |
| *** ducttape_ has quit IRC | 19:48 | |
| *** ducttape_ has joined #openstack-lbaas | 19:52 | |
| *** m-greene has joined #openstack-lbaas | 19:56 | |
| *** yamamoto has joined #openstack-lbaas | 19:59 | |
| *** ihrachys has quit IRC | 20:00 | |
| *** yamamoto has quit IRC | 20:04 | |
| *** pcaruana has quit IRC | 20:11 | |
| *** ducttape_ has quit IRC | 20:12 | |
| *** chlong has quit IRC | 20:36 | |
| *** kevo has quit IRC | 20:38 | |
| *** kevo has joined #openstack-lbaas | 20:38 | |
| *** chlong has joined #openstack-lbaas | 20:52 | |
| *** yamamoto has joined #openstack-lbaas | 21:04 | |
| *** ducttape_ has joined #openstack-lbaas | 21:12 | |
| *** yamamoto has quit IRC | 21:13 | |
| *** ducttape_ has quit IRC | 21:17 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia: Add quota support to Octavia https://review.openstack.org/360794 | 21:21 |
| johnsom | Darn, I guess I missed touching a few files... Ugh, this patch... | 21:22 |
| openstackgerrit | Michael Johnson proposed openstack/octavia: Add quota support to Octavia https://review.openstack.org/360794 | 21:24 |
| rm_work | johnsom: is it possible to boot a VM with TWO of the same network-id bound? :/ | 21:27 |
| johnsom | Ummm, hmmm. I suspect it is possible. | 21:28 |
| blogan | rm_work: so you get 2 interfaces on the same network? | 21:28 |
| johnsom | We make sure that doesn't happen when we manage our amp namespace | 21:28 |
| rm_work | blogan: ^^ yeah | 21:29 |
| blogan | rm_work: do you just need an extra ip? | 21:29 |
| rm_work | no i want to fake out having the mgmt-net also just be public net, but have two interfaces | 21:29 |
| johnsom | Yeah, I can think of some *strange* reasons to do it. | 21:29 |
| blogan | rm_work: i think if you create 2 different ports on teh same network you can do it through nova, i dont know if octavia is setup to use a port to do that htough | 21:30 |
| johnsom | That should work as one will be in the default namespace, the other will be in the amphora-haproxy namespace. | 21:30 |
| johnsom | Though, the check to make sure we don't double up *might* still be in the code | 21:30 |
| blogan | johnsom: i feel like nova fails avlidation for teh same network, but i might be wrong | 21:30 |
| blogan | been a while | 21:30 |
| blogan | i can test it out real quick actually | 21:31 |
| johnsom | hmmm, we hot plug the second one, so maybe that bypasses the nova idiot tst | 21:31 |
| blogan | now that i have everything running in my lxc deploy | 21:31 |
| blogan | yeah a hot plug might bypass it | 21:31 |
| blogan | but i could also be thinking about rackspace's internal nova that gave me that error :) | 21:31 |
| johnsom | I mean you would want to be able to do it if you are moving the ports into different namespaces in the VM | 21:32 |
| johnsom | There are also some creative port load balancing systems that could make use of multiple ports on the same network, but I suspect neutron wouldn't allow those. Certainly not with DVR | 21:33 |
| blogan | yeah nova allowed me todo it | 21:34 |
| blogan | and 2 diff itnerfaces on the amp | 21:36 |
| rm_work | hmm ok | 21:37 |
| rm_work | wait blogan what command did you use to boot that | 21:37 |
| rm_work | ^^ | 21:54 |
| rm_work | blogan: | 21:54 |
| openstackgerrit | Michael Johnson proposed openstack/octavia: Add quota support to Octavia https://review.openstack.org/360794 | 21:59 |
| *** chlong has quit IRC | 22:04 | |
| openstackgerrit | German Eichberger proposed openstack/octavia: Add tempesttest for the N-LBaaS API https://review.openstack.org/413697 | 22:11 |
| *** bana_k has joined #openstack-lbaas | 22:38 | |
| *** ducttape_ has joined #openstack-lbaas | 22:43 | |
| *** diogogmt has quit IRC | 22:44 | |
| *** csomerville has quit IRC | 22:45 | |
| *** ducttape_ has quit IRC | 22:48 | |
| *** fnaval_ has joined #openstack-lbaas | 22:58 | |
| *** fnaval has quit IRC | 23:00 | |
| *** m-greene has quit IRC | 23:08 | |
| *** nmagnezi_ has quit IRC | 23:35 | |
| *** m-greene has joined #openstack-lbaas | 23:47 | |
| *** catintheroof has joined #openstack-lbaas | 23:48 | |
| *** yuanying has quit IRC | 23:50 | |
| rm_work | greghaynes: if I need to add a custom arg to the grub commandline (GRUB_CMDLINE_LINUX in /etc/default/grub) in DIB how should I do that? | 23:51 |
| rm_work | greghaynes: is there an element or something? | 23:51 |
| rm_work | or can I just throw it in one of the other random scripts that runs | 23:51 |
| johnsom | I don't know of any grub magic off my head, you might be adding that to an element or creating a new one | 23:56 |
| johnsom | I know there is a grub2 element in DIB, but we don't use it to my knowledge | 23:57 |
| johnsom | I think it's an ironic thing | 23:57 |
| rm_work | hmm | 23:59 |
| rm_work | yeah i might just ... throw this sed line into a script | 23:59 |
| rm_work | some random post-install. | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!