| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Move tests in to unit specific dir and fix cover https://review.openstack.org/413277 | 00:04 |
|---|---|---|
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Turn on debug tracing by default https://review.openstack.org/413265 | 00:05 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Turn on debug tracing by default https://review.openstack.org/413265 | 00:20 |
| *** dmarlin has quit IRC | 00:20 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Move tests in to unit specific dir and fix cover https://review.openstack.org/413277 | 00:24 |
| ianw | pabelanger: +export LANG=en_US.utf8 | 00:52 |
| ianw | i'm prety sure dib sets LANG=C | 00:52 |
| ianw | i've noticed those warnings too | 00:53 |
| ianw | then dib-run-parts sets it to something else again, so it gets sort ordering consistent when it's finding files | 00:53 |
| ianw | i.e. it's a mess | 00:53 |
| ianw | a problem for another day | 00:53 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Increase func testing for ubuntu-minimal element https://review.openstack.org/411910 | 00:53 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Start func testing on centos-minimal again https://review.openstack.org/411926 | 00:53 |
| pabelanger | ianw: indeed, I want to look at that logic too. but, another day | 00:55 |
| ianw | pabelanger: ^ just made precise non-default, and will add to -nv job, but otherwise gtg | 00:56 |
| pabelanger | ianw: I'd like update our chroot command to use env -i to empty our env vars by default, and explicitly pass in what we need to access | 00:56 |
| pabelanger | ianw: ah, cool | 00:56 |
| openstackgerrit | yunfeng zhou proposed openstack/diskimage-builder: add CONTRIBUTING.rst https://review.openstack.org/412795 | 01:01 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Turn on debug tracing by default https://review.openstack.org/413265 | 01:13 |
| openstackgerrit | yunfeng zhou proposed openstack/diskimage-builder: add CONTRIBUTING.rst https://review.openstack.org/412795 | 01:27 |
| prometheanfire | dirk found a bug in building gentoo on xenial http://sprunge.us/cbhK now with comments | 02:03 |
| prometheanfire | that should fix it | 02:03 |
| prometheanfire | pabelanger: ianw comments before I submit? http://sprunge.us/cbhK | 02:03 |
| greghaynes | ianw: so we might have broken the world on ubuntu | 03:01 |
| greghaynes | Im a bit worried that infras images tomorrow are going to be broken | 03:01 |
| greghaynes | tl;dr is for some reason pip install packages are going in to site-packages and system python is not looking anywhere but dist-packages | 03:01 |
| greghaynes | I have to go out and do wife birthday things tonight, but if I can ill be on later trying to debug | 03:03 |
| greghaynes | I think to repro you just need to make a glean image and boot it and itll fail | 03:03 |
| greghaynes | (or cloud-init with a similar issue) | 03:03 |
| ianw | greghaynes: hmm, let me try | 03:17 |
| ianw | but we make a glean image and boot in tests? | 03:17 |
| greghaynes | we do? | 03:17 |
| greghaynes | oh, the nodepool dsvm does, yes | 03:18 |
| greghaynes | so this may be related to pip-and-virtualenv then | 03:18 |
| greghaynes | if so then not as big of a worry (I think infra doesnt use it yet) | 03:18 |
| ianw | prometheanfire: ^^ set -x should come from the trace cmd | 03:19 |
| prometheanfire | ianw: ya, ignore that part | 03:20 |
| prometheanfire | greghaynes: gentoo looks at site-packages :D maybe infra should use gentoo | 03:20 |
| greghaynes | haha | 03:20 |
| bkero | Wasn't that being discussed for a while? | 03:21 |
| prometheanfire | greghaynes: not stop being here and do wife things | 03:21 |
| prometheanfire | bkero: ? | 03:21 |
| greghaynes | yea, was about to say - I have to run and do that :) | 03:21 |
| ianw | greghaynes: but simple-init brings in pip-and-virtualenv? | 03:21 |
| prometheanfire | bkero: it's stalled because I've been busy with politics and real life things | 03:21 |
| greghaynes | ianw: ah if so then worth looking at if itll be broken | 03:21 |
| prometheanfire | if talking about gentoo images for infra | 03:21 |
| * prometheanfire is now president of gentoo... | 03:22 | |
| ianw | greghaynes: xenial? | 03:22 |
| bkero | prometheanfire: you're president now? | 03:22 |
| bkero | you mean for openstack :P | 03:22 |
| prometheanfire | no, of all of gentoo :P | 03:24 |
| bkero | Oh, so I see | 03:24 |
| prometheanfire | https://wiki.gentoo.org/wiki/Foundation:Main_Page | 03:24 |
| bkero | Haha nice | 03:24 |
| prometheanfire | :P | 03:24 |
| bkero | Right right | 03:25 |
| prometheanfire | it means I had to buy a scanner/printer :( | 03:27 |
| bkero | such management | 03:28 |
| bkero | Next you'll be telling me that you're faxing things. | 03:28 |
| prometheanfire | I dealt with voip faxing a few years ago for an ITSP | 03:39 |
| prometheanfire | that was fun | 03:39 |
| prometheanfire | made a voip-fax to email gateway | 03:39 |
| greghaynes | ianw: yes | 03:41 |
| ianw | greghaynes: ok, building one for a bit of testing onw | 03:42 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: Fix Gentoo builds on Ubuntu 16.04 Xenial hosts https://review.openstack.org/413395 | 03:55 |
| prometheanfire | greghaynes: ianw ^ | 03:55 |
| prometheanfire | we got the bot? | 03:55 |
| bkero | yeah, irc bot got refactored | 03:57 |
| bkero | easier than running 2 of them | 03:57 |
| prometheanfire | ya | 04:03 |
| openstackgerrit | Merged openstack/diskimage-builder: Increase func testing for ubuntu-minimal element https://review.openstack.org/411910 | 04:03 |
| ianw | greghaynes: [[0;1;31mFAILED[0m] Failed to start Glean for interface ens3. | 04:09 |
| ianw | that does seem worrying | 04:09 |
| greghaynes | Yea I bet it'll say can't import something | 04:09 |
| greghaynes | And it'll be installed in site-packages | 04:09 |
| ianw | silly me forgot devuser and i can't get in, rebuilding | 04:09 |
| greghaynes | My money is on the setup tools release | 04:10 |
| prometheanfire | ianw: I've done that a few times, I ended up getting into a host and munging /etc/shadow | 04:11 |
| ianw | prometheanfire: unfortunately, it seems there's no boot time-out | 04:11 |
| greghaynes | Mount; chroot;passwd | 04:12 |
| greghaynes | Surprising the Gentoo person doesn't do this | 04:12 |
| greghaynes | Are stage 1 installs no longer a thing? | 04:12 |
| greghaynes | :p | 04:12 |
| prometheanfire | ianw: ? | 04:13 |
| ianw | yeah, the image is uploaded to a cloud, but i guess i could guestfish in. easier to rebuild | 04:13 |
| prometheanfire | greghaynes: actually, I figured out a better way | 04:13 |
| prometheanfire | passwd takes a chroot option | 04:13 |
| prometheanfire | forgot about that for a sec | 04:13 |
| openstackgerrit | Merged openstack/diskimage-builder: Start func testing on centos-minimal again https://review.openstack.org/411926 | 04:13 |
| prometheanfire | greghaynes: get up to my (gentoo) level | 04:13 |
| greghaynes | Oh wow, TIL | 04:13 |
| ianw | hmm, i wonder if we should put builds on hold | 04:14 |
| prometheanfire | -R, --root CHROOT_DIR directory to chroot into | 04:14 |
| prometheanfire | :D | 04:14 |
| ianw | why wouldn't CI fail on this? | 04:14 |
| greghaynes | ianw: yep, good question | 04:14 |
| ianw | we do at least ssh into it for nodepool | 04:14 |
| greghaynes | ianw: that might be a good idea too re: builds | 04:15 |
| greghaynes | Not crazy about going super firefighter tomorrow | 04:16 |
| greghaynes | Ok, really afk for a few hr | 04:16 |
| ianw | greghaynes: i'll poke a bit, i have about 45 minutes before i have to disappear for a few hours too | 04:16 |
| ianw | ok, it seems the xenial image i've just booted doesn't have python2 | 04:34 |
| prometheanfire | ya, they changed the default iirc | 04:45 |
| ianw | greghaynes: ok, so the problem is definitely in https://review.openstack.org/#/c/408288/ | 05:01 |
| ianw | that used to install "python" there | 05:01 |
| ianw | which meant that dib-python ended up being python2 | 05:01 |
| ianw | which had consequences making everything else work | 05:02 |
| ianw | clearly, things aren't ready for python3 only | 05:02 |
| prometheanfire | not sure why that one non-voting test failed, would appreciate a glance https://review.openstack.org/413395 | 05:03 |
| prometheanfire | ianw: so that's the root of the problem? no py2? | 05:04 |
| ianw | well, trying to use python3 to install glean i guess is the problem, something isn't ready for that | 05:04 |
| ianw | what a mess :( how did this pass CI | 05:04 |
| * prometheanfire has been out of it :( | 05:05 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Revert Xenial to Python 2 https://review.openstack.org/413410 | 05:11 |
| ianw | greghaynes: ok, gotta head out for a bit, but *maybe* this is enough as an emergency measure. building images | 05:11 |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder: Refactor: block-device handling (partitioning) https://review.openstack.org/375261 | 05:43 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Revert Xenial to Python 2 https://review.openstack.org/413410 | 05:45 |
| *** pmannidi_ has quit IRC | 06:13 | |
| *** pmannidi has joined #openstack-dib | 06:14 | |
| greghaynes | ianw: yea, in theory that shouldnt be an issue (no installing python, python3 is py3) since xenial ships without python2 | 07:05 |
| ianw | greghaynes: yeah, but probably not then installing pip from get-pip.py and upstream setuptools etc | 07:05 |
| ianw | that is probably quite untested | 07:05 |
| ianw | i'm just booting now, but the build logs look good | 07:06 |
| greghaynes | right, so I agree - the brokennes is likely new setuptools/pip + xenial python3 | 07:06 |
| ianw | glean is put in dist-config | 07:06 |
| *** adam_g has joined #openstack-dib | 07:07 | |
| adam_g | HI! | 07:07 |
| greghaynes | ohai | 07:07 |
| greghaynes | ianw: btw, did you stop infra images from uploading? | 07:07 |
| adam_g | yeah, so hacking a bit to get a python2 installed /w that patch gets things working | 07:07 |
| adam_g | trying an older ubuntu tarball now | 07:07 |
| ianw | greghaynes: no, i haven't paused them yet | 07:07 |
| greghaynes | adam_g: current theory is new pip/setuptools + python3 + xenial is just plain broken | 07:08 |
| ianw | i think we probably just want an emergency release with this | 07:08 |
| greghaynes | yea, although id like to get a bit better idea of what is going on first | 07:08 |
| adam_g | greghaynes: whats weird is, if i start a fresh xenial manually, use a symlink to the default python3 and follow the workflow everyting works | 07:08 |
| greghaynes | something I hadnt looked in to - its possible we install something (pip maybe) with old setuptools then upgrade setuptools later... | 07:09 |
| greghaynes | adam_g: super wierd | 07:09 |
| ianw | ok, so with that patch it boots and is on the network | 07:10 |
| greghaynes | the revert to python 2? | 07:10 |
| adam_g | ianw: what ends up installing python2 in that case? | 07:11 |
| ianw | adam_g: i dropped a comment, we do that in 75-debian-minimal-baseinstall | 07:11 |
| greghaynes | we have to do python at an odd spot because the tools to do normal package install are written in python | 07:12 |
| greghaynes | so, yea | 07:12 |
| greghaynes | inception | 07:12 |
| ianw | i feel like setting it to "2" is the right thing for now | 07:12 |
| greghaynes | if it works thats fine with me | 07:12 |
| greghaynes | we were effectively doing that before | 07:13 |
| adam_g | ianw: oh, i see.. so its still broken if using ubuntu and not ubuntu-minima | 07:13 |
| greghaynes | eek | 07:13 |
| ianw | http://people.redhat.com/~iwienand/ubuntu-minimal-python3.log | 07:14 |
| ianw | that's a log, it doesn't really show anything | 07:14 |
| ianw | http://nb01.openstack.org/dib.ubuntu-xenial.log | 07:15 |
| ianw | you can compare it to the latest nodepool build | 07:15 |
| greghaynes | btw, the way octavia fixed this is they added a deal to symlink site-packages to dist-pacakges | 07:16 |
| greghaynes | super fugly but it works | 07:16 |
| ianw | ? surely that is wrong | 07:16 |
| greghaynes | indeed | 07:16 |
| greghaynes | although its in /usr/local so in theory it shouldnt break things | 07:17 |
| ianw | adam_g: actually, i'm not sure it's any more broken for ubuntu (!minimal). how was it working previously? | 07:17 |
| adam_g | ianw: xenial comes with python3.5 and thats it | 07:17 |
| adam_g | if you want python2x you need to install it. the ubuntu-miniaml does that via the 75-* you mentioned. | 07:18 |
| greghaynes | octavia uses non-minimal and they were broken with the same issue | 07:18 |
| ianw | right, so before your change, then the ubuntu element would have been finding python3 & using that | 07:18 |
| greghaynes | it manifested differently (they used glean) but it seemed like the same root cause | 07:18 |
| greghaynes | er they dont use gelan | 07:18 |
| ianw | so it must have been broken for a long time? | 07:18 |
| greghaynes | they just broke today | 07:18 |
| greghaynes | ubuntu non-minimal was super wierd, I think we were installing python2 there | 07:19 |
| ianw | was something else installing python2? it's hard to say without logs | 07:19 |
| adam_g | ianw: yeah, it does with master. wiht the patch i commented on, it bails early because its looking for a python2 that does not exist | 07:19 |
| greghaynes | as a transitive dep | 07:19 |
| adam_g | it might be installed later | 07:19 |
| adam_g | actually no | 07:20 |
| ianw | right, if it was installed at some point, the dib-python would find it, right? | 07:20 |
| ianw | the python2 || python3 bit | 07:20 |
| adam_g | ianw: well, the check for it happens pretty earlier on in the build | 07:20 |
| adam_g | anyway | 07:20 |
| adam_g | looking at an image built right now with master and 'ubuntu', python2 doesn't get installed | 07:21 |
| ianw | adam_g: but i mean, that check is done whenever dib-python is called | 07:21 |
| adam_g | (thats without the revert) | 07:21 |
| ianw | adam_g: so does pip work to install anything on that image? | 07:21 |
| ianw | presumably, that image isn't using glean? | 07:21 |
| adam_g | its broken | 07:21 |
| adam_g | it should be using glean but it cant because everything python is fubar | 07:21 |
| greghaynes | ianw: I think when I looked in to this dib-python did python2 || python3, we installed python2 as a transitive dep and so dib-python was *always* python2 | 07:22 |
| greghaynes | which is when I did the explicit DIB_PYTHON thing | 07:22 |
| ianw | right ... if something happened to install python2 *before* glean's install ran, it would seem like it would work | 07:23 |
| adam_g | nothing installs python2 (using 'ubuntu') | 07:23 |
| greghaynes | adam_g: we used to | 07:23 |
| ianw | BUT, it seems like ubuntu (!minimal) has been just as broken & nobody noticed? | 07:23 |
| adam_g | *shrug* | 07:23 |
| greghaynes | ianw: octavia definitely noticed | 07:23 |
| greghaynes | and yes | 07:23 |
| greghaynes | it was just as broken | 07:23 |
| ianw | ok, so I think we need to hack in a python2 install for "ubuntu" | 07:24 |
| ianw | unless anyone has time to dig into setuptools magic at this piont (I don't) | 07:24 |
| adam_g | thats basically what i did, and it ended up getting me a working python | 07:24 |
| greghaynes | ok | 07:24 |
| ianw | we can find the real bug later ... we know how to reproduce | 07:24 |
| greghaynes | I want to spend a few trying to root cause it a bit more but I agree that we should emergency release pretty soon | 07:24 |
| ianw | i feel like we commit my change | 07:24 |
| greghaynes | I just want a tiny bit of time :p | 07:24 |
| ianw | adam_g: can you submit one to install python2 for the ubuntu element? | 07:25 |
| adam_g | ianw: also, fwiw i had to 'apt-get install python' to get a /usr/bin/python2, not 'apt-get install python2.7' | 07:25 |
| adam_g | ianw: yeah, okay | 07:25 |
| ianw | greghaynes: i mean, even if you find the problem in setuptools, you'll still need a new release? | 07:25 |
| adam_g | greghaynes: aren't you in hawaii? | 07:26 |
| greghaynes | adam_g: not yet | 07:26 |
| greghaynes | ianw: yea I just want to see if I can verify were not going to make the problem worse by missing something | 07:26 |
| greghaynes | adam_g: on the 29th | 07:26 |
| ianw | ok, well image builds start at like 13:00UTC. i've got to head to dinner, if we don't have something in a bit, i'll put in the nodepool config to pause the build | 07:27 |
| ianw | i have built and booted the -minimal images with the force to python2, and glean works | 07:28 |
| greghaynes | ok, my pip in chroot is definitely installing stuff in to site-packages | 07:28 |
| greghaynes | BUT | 07:28 |
| greghaynes | things seem to work | 07:28 |
| ianw | what's the sys.path? | 07:28 |
| adam_g | greghaynes: did you stop the build or are you chroot'd into the resulting image? | 07:28 |
| greghaynes | ['', '/usr/lib/python35.zip', '/usr/lib/python3.5', '/usr/lib/python3.5/plat-x86_64-linux-gnu', '/usr/lib/python3.5/lib-dynload', '/usr/lib/python3.5/site-packages', '/usr/local/lib/python3.5/dist-packages', '/usr/lib/python3/dist-packages'] | 07:28 |
| greghaynes | and that is my sys.;path | 07:28 |
| adam_g | greghaynes: thats not what it is post-build | 07:28 |
| greghaynes | adam_g: dropped a bash in part of the image build | 07:28 |
| greghaynes | adam_g: yep | 07:28 |
| adam_g | im thinking thats getting polluted during build | 07:28 |
| greghaynes | so, now the question comes down to - wtf is changing it | 07:29 |
| greghaynes | https://github.com/python/cpython/blob/master/Lib/site.py#L1 has some good info | 07:39 |
| ianw | hmm, pyenv creates a python3 virtualenv on debian with site-packages | 07:43 |
| greghaynes | I think whats supposed to happen is anything pip/python is supposed to use site-packages | 07:44 |
| greghaynes | and really just debian pacakges are suppsoed to use dist-packages | 07:44 |
| greghaynes | I suspect theres a modified ubuntu site.py | 07:44 |
| ianw | that sounds right | 07:44 |
| adam_g | greghaynes: thats the way i thought i remembered it, but its been a while | 07:45 |
| greghaynes | yep, there is | 07:45 |
| adam_g | greghaynes: modified by? | 07:45 |
| greghaynes | adam_g: its modified to search for dist-packages | 07:46 |
| * greghaynes tries to remember how to find the deb which provides this file | 07:46 | |
| greghaynes | should make sure its not recently changed | 07:46 |
| ianw | dpkg -S | 07:46 |
| *** jamielennox has joined #openstack-dib | 07:46 | |
| greghaynes | libpython3.5-minimal:amd64 | 07:46 |
| adam_g | OH | 07:47 |
| adam_g | so i did a quick diff between DIB package manifests from our working images and our broken ones | 07:47 |
| adam_g | http://paste.ubuntu.com/23662868/ | 07:47 |
| adam_g | libpython3.5-dev is new | 07:48 |
| greghaynes | hrm | 07:48 |
| adam_g | oh, wait. -minimal, not -dev. | 07:48 |
| adam_g | dont listen to me | 07:48 |
| *** openstackgerrit has quit IRC | 07:48 | |
| greghaynes | ok, im finishing this build and going to try and chroot in to it after its done | 07:49 |
| jamielennox | so -minimal seemed old, -dev was mid november | 07:49 |
| greghaynes | and see if things are broken with just a chroot | 07:49 |
| greghaynes | jamielennox: ok | 07:50 |
| adam_g | greghaynes: also, theres the issue where things are going into /usr/lib/python3.5/ and not /usr/local/lib/python3.5 | 07:50 |
| greghaynes | adam_g: right | 07:50 |
| greghaynes | adam_g: no idea whats causing that | 07:51 |
| jamielennox | oh, derp, makes sense minimal won't have changed it'll just have a dependency | 07:51 |
| adam_g | i have a hunch | 07:53 |
| adam_g | if 'VIRTUAL_ENV' in os.environ or sys.base_prefix != sys.prefix: | 07:53 |
| adam_g | sitepackages.append(os.path.join(prefix, "lib", | 07:53 |
| adam_g | "python" + sys.version[:3], | 07:53 |
| adam_g | "site-packages")) | 07:53 |
| greghaynes | yea I saw that too | 07:53 |
| greghaynes | oh crap | 07:54 |
| greghaynes | wow | 07:54 |
| adam_g | im building in a venv | 07:54 |
| adam_g | and so are (our nodepoop) | 07:54 |
| ianw | ahhhhhh | 07:54 |
| adam_g | building right now wihtout | 07:54 |
| adam_g | tho i did just hack in an 'unset VIRTUAL_ENV' into the elemant and build with no luck | 07:54 |
| ianw | we should really unset the environment in the chroot | 07:54 |
| ianw | pabelanger was just talking about that | 07:55 |
| greghaynes | agreed | 07:55 |
| greghaynes | although | 07:55 |
| adam_g | ianw: yeah, i dumped an `env` output to see what might be polluting, found the VIRTUAL_ENV as well as tons of other stuff | 07:55 |
| greghaynes | how do we know wnat env vars we unset? | 07:55 |
| greghaynes | since we use env vars to pass in condfig | 07:55 |
| adam_g | greghaynes: ! var.startswith('DIB_') ? | 07:56 |
| greghaynes | yea, that could work | 07:56 |
| greghaynes | I really want to know why ubuntu modifies site.py in that way | 07:57 |
| greghaynes | I get why dist-pacakges | 07:57 |
| greghaynes | but why the venv stuff | 07:57 |
| adam_g | greghaynes: is that an ubuntu delta? | 07:57 |
| greghaynes | yes | 07:57 |
| greghaynes | https://github.com/python/cpython/blob/master/Lib/site.py#L288 | 07:57 |
| greghaynes | ok I just chrooted in and its broken as if it was booted | 07:59 |
| greghaynes | and setting VIRTUAL_ENV=derp fixes it | 07:59 |
| greghaynes | so winner | 07:59 |
| adam_g | greghaynes: what do you mean? | 08:00 |
| greghaynes | http://paste.ubuntu.com/23662905/ | 08:00 |
| adam_g | what | 08:01 |
| greghaynes | yea because I think when you have a venv it adds site-packages | 08:01 |
| adam_g | oh oh oh | 08:01 |
| adam_g | right | 08:01 |
| greghaynes | and so I think this is a two parter | 08:01 |
| adam_g | opposite of what im about to test | 08:01 |
| adam_g | (building with no V_ENV) | 08:01 |
| adam_g | ya | 08:02 |
| adam_g | things work if img built outside of venv | 08:03 |
| greghaynes | adam_g: can you see if stuff is installed in /usr/local/.../site-packges? | 08:03 |
| greghaynes | I suspect whats going on is a combination of /usr/lib + the venv | 08:03 |
| adam_g | yea, one sec | 08:03 |
| adam_g | heres the delta ubuntu/debian carries on site.py | 08:03 |
| adam_g | adam@amebix:~$ sudo mount /dev/nbd3p1 /mnt/qcow/ | 08:03 |
| adam_g | adam@amebix:~$ cd /mnt/qcow/ | 08:04 |
| adam_g | http://paste.ubuntu.com/23662916/ | 08:04 |
| adam_g | adam@amebix:/mnt/qcow$ ls | 08:04 |
| adam_g | er | 08:04 |
| adam_g | greghaynes: /usr/local/lib/python3.5/dist-packages/glean/__init__.py | 08:04 |
| greghaynes | so I need to pdb to confirm, but I think the sys.base_prefix != sys.prefix applies to /usr/local | 08:04 |
| greghaynes | in site.py | 08:04 |
| adam_g | /usr/local/lib/python3.5/dist-packages/ is where i'd expect it to be based on what i saw on a functioning vanilla xenial | 08:05 |
| greghaynes | nope, it should be site- if its installed with pip | 08:05 |
| adam_g | but my IIRC things went to site-packages in older releases BIMBW | 08:05 |
| greghaynes | oh, yea if it changed | 08:05 |
| adam_g | right, i mean just by booting a vanilla (non-dib) xenial, get-pip'in and installing something, things go into /u/l/l/p/dist-packages | 08:06 |
| adam_g | lemme check precise just to be sure | 08:06 |
| greghaynes | so heres what I think is going on - since we have VIRTUAL_ENV set the first path in sys.path ends up being /usr/lib/...site-packages (based on reading of site.py). This means pip installs things in to /usr/lib/...site-packages | 08:07 |
| greghaynes | as a result no site-packages ever gets made in /usr/local | 08:07 |
| jamielennox | what's the diff between upstream site.py and ubuntu site.py? | 08:07 |
| jamielennox | what is ubuntu doing? | 08:08 |
| greghaynes | which means subsequent runs of site.py dont pass on that dir existing and the second part of the VIRTUAL_ENV conditional (sys.base_prefix != sys.prefix) means you will never add site-packages for /usr/lib unless you have VIRTUAL_ENV set | 08:08 |
| greghaynes | since sys.base_prefix = /usr/lib | 08:09 |
| adam_g | jamielennox: http://paste.ubuntu.com/23662916/ | 08:09 |
| greghaynes | and all that to say - the fix is to simply not set VIRTUAL_ENV | 08:09 |
| *** pleia2_ has joined #openstack-dib | 08:09 | |
| greghaynes | and also, why the hell does ubuntu add that conditional | 08:09 |
| greghaynes | jamielennox: IKR | 08:09 |
| adam_g | +For Debian and derivatives, this sys.path is augmented with directories | 08:10 |
| adam_g | +for packages distributed within the distribution. Local addons go | 08:10 |
| adam_g | +into /usr/local/lib/python<version>/dist-packages, Debian addons | 08:10 |
| adam_g | +install into /usr/lib/python3/dist-packages. | 08:10 |
| adam_g | +/usr/lib/python<version>/site-packages is not used. | 08:10 |
| jamielennox | that's weird | 08:11 |
| greghaynes | yea tahts fine, but why the VIRTUAL_ENV :/ | 08:11 |
| greghaynes | I suspect its because they want you to install in to the venv site-pacakges | 08:11 |
| jamielennox | only appended if VIRTUAL_ENV | 08:11 |
| adam_g | actually this is all from debian, not ubuntu | 08:12 |
| greghaynes | nice | 08:12 |
| adam_g | 0 * d/p/distutils-install-layout.diff, d/p/site-locations.diff: Adjust the | 08:12 |
| adam_g | 1 "am I in a virtual environment" tests to include checking | 08:12 |
| adam_g | 2 sys.base_prefix != sys.prefix. This is the definitive such test for | 08:12 |
| adam_g | 3 pyvenv created virtual environments (Barry Warsaw). | 08:12 |
| jamielennox | I can't read debian packages | 08:14 |
| jamielennox | i'm looking at http://www.ubuntuupdates.org/package/core/xenial/main/updates/libpython3.5-dev, i see the change but not the patch | 08:14 |
| greghaynes | jamielennox: I think there isnt a change there, what probably changed is this behavior isnt the same in python2 | 08:15 |
| greghaynes | and we started using python3 | 08:15 |
| greghaynes | although I *thought* we had that working | 08:15 |
| greghaynes | but IMBW | 08:15 |
| jamielennox | i saw lots of messing with that recently in dib history | 08:15 |
| greghaynes | yea, I thought we had a few days of it working with pyhon3 after that merged | 08:15 |
| greghaynes | but now im not trusting my memory | 08:15 |
| adam_g | jamielennox: i usually pull down the package locally and look at it there | 08:15 |
| adam_g | jamielennox: all the web interfaces for it mostly suck unlesls the thing is hosted on launchpad | 08:16 |
| jamielennox | i don't run ubuntu | 08:16 |
| *** pleia2 has quit IRC | 08:16 | |
| adam_g | jamielennox: i mean, pull down the source package not the ubuntu package | 08:16 |
| jamielennox | but i figured there would be a source package download link on that page | 08:16 |
| adam_g | https://launchpad.net/ubuntu/+source/python3.5 | 08:16 |
| adam_g | then grab https://launchpad.net/ubuntu/+archive/primary/+files/python3.5_3.5.2-2ubuntu0~16.04.1.debian.tar.xz | 08:16 |
| adam_g | or if you have 'dget' installed somwhere you can you do: dget -xu https://launchpad.net/ubuntu/+archive/primary/+files/python3.5_3.5.2-2ubuntu0~16.04.1.dsc and it'll pull down the entire source + packaging | 08:17 |
| *** openstackgerrit has joined #openstack-dib | 08:18 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Dont run with VIRTUAL_ENV set https://review.openstack.org/413487 | 08:18 |
| greghaynes | adam_g: can you test with ^ while I write up a commit message essay | 08:18 |
| adam_g | greghaynes: yeah, one sec | 08:18 |
| jamielennox | so is that set there because dib itself is running in a virtualenv? | 08:20 |
| greghaynes | yep | 08:20 |
| jamielennox | you could always just VIRTUAL_ENV= dib-run-parts | 08:20 |
| greghaynes | I think if it checks for exists, not nonzerpo | 08:21 |
| jamielennox | like it may not matter but your return 1 will skip the restore | 08:21 |
| greghaynes | yea, if 'VIRTUAL_ENV' in os.environ | 08:21 |
| greghaynes | so you have to actually unset it I think | 08:21 |
| jamielennox | wouldn't that work for calling a program? | 08:22 |
| jamielennox | oh, dib-run-parts is a function, not a script? | 08:22 |
| greghaynes | its a script | 08:22 |
| jamielennox | oh, lol | 08:23 |
| greghaynes | jamielennox: derp= python -c "import os;print('derp' in os.environ)" ' | 08:23 |
| jamielennox | so it doesn't check that virtualenv is set to a string, just whether it exists in the env at all | 08:23 |
| greghaynes | yep | 08:23 |
| jamielennox | that's terrible | 08:23 |
| greghaynes | its basically a terrible lasagna | 08:24 |
| greghaynes | many layers of terrible | 08:24 |
| ianw | one thing to consider is shipping images to infra with a /usr/bin/python may be an issue. but also, infra-package-needs probably brings in python2 | 08:24 |
| greghaynes | with lots of cheese inbetween | 08:24 |
| ianw | ^^ withOUT a /usr/bin/python | 08:24 |
| greghaynes | ianw: yea, im pretty sure infra installs python2 as a dep | 08:24 |
| greghaynes | although if its an issue itd be a good one to fix IMO | 08:24 |
| adam_g | greghaynes: hrm that doeset seem to work | 08:25 |
| greghaynes | adam_g: w00t | 08:25 |
| adam_g | does dib stuff a build log somewhere by default? | 08:25 |
| greghaynes | nope | 08:25 |
| greghaynes | I mean, it stuffs it in stdout | 08:25 |
| *** pmannidi has quit IRC | 08:27 | |
| jamielennox | what file is the dib-run-parts script? | 08:28 |
| greghaynes | jamielennox: its in dib-utils (the package) | 08:28 |
| greghaynes | in our v2 branch its in tree | 08:28 |
| jamielennox | which is not openstack/diskimage-builder ? | 08:28 |
| greghaynes | http://git.openstack.org/cgit/openstack/dib-utils/tree/bin/dib-run-parts | 08:29 |
| greghaynes | correct | 08:29 |
| jamielennox | more repos! | 08:29 |
| greghaynes | yea | 08:29 |
| greghaynes | its going to die soon (TM) | 08:29 |
| jamielennox | ok, i was going to ask why that's in a seperate repo - but probably not the time | 08:30 |
| adam_g | im going to die soon if i dont get off this ting | 08:30 |
| adam_g | tty tomorrow. it was a great night of bang-head-on-keyboard with you all. | 08:31 |
| jamielennox | README explainsit | 08:31 |
| greghaynes | adam_g: yaya gn | 08:32 |
| greghaynes | jamielennox: its super old, I dont really remember all the history | 08:32 |
| greghaynes | but yea, its turned out to be a pain | 08:33 |
| greghaynes | so, out with it | 08:33 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Dont run with VIRTUAL_ENV set https://review.openstack.org/413487 | 08:35 |
| jamielennox | greghaynes: and so dib-run-parts isn't going to try to run python fragments that need the virtualenv set? | 08:35 |
| greghaynes | jamielennox: they definitely dont, dib-run-parts just runs executables which cant reference code living outside of the chroot | 08:37 |
| greghaynes | (thats not 100% true, sometimes they run outside of the chroot, but even then they cant refer to modules as part of the dib package) | 08:37 |
| jamielennox | ah, ok, you're inside the chroot by that point | 08:37 |
| greghaynes | theres I think 2 phases that are outside, but otherwise its all inside | 08:38 |
| jamielennox | it's not necessarily part of the dib package, requests or other stuff in python | 08:38 |
| ianw | greghaynes: want me to try with 412487? | 08:38 |
| greghaynes | ianw: yesplz | 08:39 |
| jamielennox | but yea, they're scripts so they shouldn't rely on that | 08:39 |
| greghaynes | jamielennox: oh I see what youre getting at - so yea in theory someone could have a script in extra-data.d which depends on some dib dep | 08:39 |
| jamielennox | well yea, i'm just wondering if anything could be relying on VIRTUAL_ENV being set within their elemetn | 08:40 |
| greghaynes | jamielennox: although I dont think this will break taht now that I think of it, I think VIRTUAL_ENV is purely used to tell venv about deactivate/activate and give a prompt | 08:40 |
| greghaynes | not actually for anything search path related | 08:40 |
| jamielennox | oh? didn't know that, i just assumed it had an actual purpose | 08:41 |
| greghaynes | yea I think virtualenv actually modifies sys.path and base_path | 08:41 |
| greghaynes | I really need to stop working though :p | 08:42 |
| greghaynes | gn all | 08:43 |
| * greghaynes out | 08:43 | |
| ianw | ok, well this leaves us with a bit of a timebomb on master | 08:49 |
| greghaynes | Feel free to merge either fix and release btw | 08:52 |
| adam_g | 413487 didnt work for me FWIW | 08:53 |
| greghaynes | Compelling argument | 08:53 |
| greghaynes | Also there's a bug jamielennox pointed out | 08:54 |
| adam_g | okay cool, just making sure theres a handoff | 08:54 |
| adam_g | also updating the other chan | 08:54 |
| * adam_g is gone for real | 08:55 | |
| greghaynes | Ya me too | 08:55 |
| ianw | i think we might need to do the python2 thing, for now | 08:55 |
| greghaynes | Wfm | 08:55 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Revert Xenial to Python 2 https://review.openstack.org/413410 | 09:06 |
| openstackgerrit | Saverio Proto proposed openstack/diskimage-builder: debootstrap: avoid duplicate network configuration https://review.openstack.org/407551 | 09:23 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Revert Xenial to Python 2 https://review.openstack.org/413410 | 09:32 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Revert Xenial to Python 2 https://review.openstack.org/413410 | 09:44 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Revert Xenial to Python 2 https://review.openstack.org/413410 | 09:46 |
| openstackgerrit | Pierre Riteau proposed openstack/diskimage-builder: Update documented default Ubuntu version https://review.openstack.org/413583 | 11:52 |
| openstackgerrit | Pierre Riteau proposed openstack/diskimage-builder: Add Xenial to the list of supported targets https://review.openstack.org/413584 | 11:52 |
| *** hashar has joined #openstack-dib | 12:06 | |
| openstackgerrit | Merged openstack/diskimage-builder: Revert Xenial to Python 2 https://review.openstack.org/413410 | 13:11 |
| ianw | ^ ok 1.26.0 out, http://lists.openstack.org/pipermail/openstack-infra/2016-December/004991.html note sent to infra | 13:21 |
| *** pleia2_ is now known as pleia2 | 13:54 | |
| prometheanfire | ianw: what's with all the releases? | 15:18 |
| prometheanfire | ianw: constant repackaging isn't the most fun | 15:18 |
| *** jamielennox is now known as jamielennox|away | 15:46 | |
| *** openstackgerrit_ has joined #openstack-dib | 15:48 | |
| *** openstackgerrit_ has quit IRC | 15:49 | |
| *** jamielennox|away is now known as jamielennox | 15:54 | |
| *** frontrunner has joined #openstack-dib | 16:39 | |
| greghaynes | ianw: it may have made it past CI because dsvm doesn't use a venv | 16:42 |
| *** openstack has quit IRC | 17:02 | |
| *** openstack has joined #openstack-dib | 17:03 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Dont run with VIRTUAL_ENV set https://review.openstack.org/413487 | 17:20 |
| greghaynes | ianw: adam_g jamielennox https://review.openstack.org/#/c/413487/3 seems to work now | 17:30 |
| greghaynes | cinerama: if youre around to review as well ^ | 17:37 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Dont run with VIRTUAL_ENV set https://review.openstack.org/413487 | 17:40 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Revert "Revert Xenial to Python 2" https://review.openstack.org/413733 | 17:40 |
| cinerama | heyyy | 17:41 |
| *** jesusaur has joined #openstack-dib | 17:41 | |
| greghaynes | ohai | 17:44 |
| *** brault is now known as brault|away | 17:58 | |
| greghaynes | sw8 | 18:04 |
| greghaynes | now to wait for the ianw | 18:04 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Allow disabling of apt gpg checking https://review.openstack.org/409383 | 18:15 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Allow disabling of apt gpg checking https://review.openstack.org/409383 | 18:26 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Add support for using infra's mirrors in tests https://review.openstack.org/408851 | 18:28 |
| *** hogepodge has quit IRC | 18:46 | |
| *** hogepodge has joined #openstack-dib | 19:12 | |
| adam_g | greghaynes: testing now. | 19:45 |
| adam_g | i had a post-DIB puke-y baby all night last night so im just getting out of bed | 19:46 |
| greghaynes | eek | 20:16 |
| greghaynes | sounds like fun | 20:16 |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder: dib-lint: python3 compatibility fixes https://review.openstack.org/413772 | 20:30 |
| *** frontrunner has quit IRC | 20:34 | |
| *** frontrunner has joined #openstack-dib | 20:45 | |
| *** frontrunner has quit IRC | 20:50 | |
| adam_g | greghaynes: works | 21:01 |
| ianw | greghaynes: hmm, that's a good point on ci not using the env | 21:07 |
| ianw | i guess nb.o.o don't eitehr | 21:08 |
| greghaynes | ianw: yea | 21:08 |
| greghaynes | adam_g: woot | 21:08 |
| ianw | greghaynes: but why only python3? | 21:09 |
| greghaynes | ianw: I think Python2 site.py is different | 21:09 |
| greghaynes | Not sure why | 21:09 |
| greghaynes | ianw: you up for merging the other fix and revert revert? | 21:10 |
| ianw | greghaynes: yeah, i'll do a little investigation in a bit to convince myself :) | 21:11 |
| greghaynes | Awesome thanks | 21:11 |
| ianw | i'd like to just double check that the infra builds still get a /usr/bin/python | 21:16 |
| ianw | they *shouldn't* need that, but i would not like to test that premise a day out from xmas | 21:17 |
| greghaynes | ah ok | 21:41 |
| jamielennox | greghaynes: commented, didn't -1 | 21:41 |
| greghaynes | jamielennox: ah, will fix | 21:42 |
| greghaynes | both are valid | 21:42 |
| greghaynes | the check-break isnt an issue in theory but it should be consistent | 21:42 |
| jamielennox | yea, i have no idea if it's a problem or not, i just see it as different to what you did elsewere | 21:43 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Dont run with VIRTUAL_ENV set https://review.openstack.org/413487 | 21:44 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Revert "Revert Xenial to Python 2" https://review.openstack.org/413733 | 21:44 |
| jamielennox | the wrap it in a function seems easiest | 21:44 |
| greghaynes | oh? not sure how you do that easily | 21:45 |
| greghaynes | the easy way I can think of youd use a subshell, but that easy to foot gun | 21:46 |
| jamielennox | nah, | 21:46 |
| jamielennox | function run_dib_parts() { | 21:46 |
| jamielennox | unset | 21:46 |
| jamielennox | dib-parts | 21:46 |
| jamielennox | reset | 21:46 |
| jamielennox | } | 21:46 |
| greghaynes | ah | 21:47 |
| greghaynes | WFM | 21:47 |
| greghaynes | after I finish RMAing this nuc | 21:47 |
| jamielennox | no big deal | 21:47 |
| *** pmannidi has joined #openstack-dib | 21:59 | |
| ianw | can we not env -u ? | 22:06 |
| ianw | also i'm being paranoid about the /usr/bin/python stuff i guess ... since infra builds will already be doing that | 22:06 |
| ianw | so the infra job does run in a virtualenv | 22:11 |
| ianw | devstack job | 22:12 |
| jamielennox | ianw: for infra to work here do you need to propose a requirements bump or something? | 22:15 |
| ianw | jamielennox: sorry, not sure what you mean? | 22:15 |
| jamielennox | we aren't using openstack-infra puppet scripts to deploy and it looks like we won't update disk-image-builder unless nodepool requirements forces it to bump | 22:16 |
| jamielennox | i'm wondering for the nodepool builders and such does infra use constraints? | 22:16 |
| jamielennox | or will it just automatically update all nodepool deps | 22:17 |
| greghaynes | ianw: huh | 22:18 |
| greghaynes | ianw: and the builder is installed in a venv? I thought thats generally not how devstack does things | 22:18 |
| greghaynes | ianw: looks like youre right | 22:19 |
| greghaynes | so, back to having 0 idea why this passed CI | 22:19 |
| ianw | i don't think we a) boot xenial and b) connect ? | 22:20 |
| greghaynes | that would explain it | 22:21 |
| ianw | jamielennox: nodepool builders just use latest release, no constraints etc | 22:21 |
| jamielennox | specifically the latest dib, or just pip install -U nodepool each time? | 22:22 |
| greghaynes | jamielennox: infra doesnt use g-r | 22:22 |
| greghaynes | so, no constraints either | 22:22 |
| ianw | greghaynes: put some comments on using env? seems simpler? | 22:25 |
| greghaynes | ianw: I dont understand what youre saying | 22:26 |
| ianw | just do the calls with "env -u VIRTUAL_ENV ... " to clear it out | 22:27 |
| ianw | we could even put "-u VIRTUAL_ENV" into something like a _DIB_CLEAR_VARS variable, incase we need to add more | 22:28 |
| ianw | (although, we should invert the logic to clear *everything* and just add what we want ... but not today) | 22:28 |
| greghaynes | ah | 22:28 |
| ianw | the chroot already clears tmpdir, which is probably another good one to think about | 22:30 |
| ianw | trying a build with that approach... | 22:34 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Dont run with VIRTUAL_ENV set https://review.openstack.org/413487 | 22:40 |
| jamielennox | cool, i didn't know you could env -u for just a single command | 22:40 |
| greghaynes | turns out that patch is actually a lot more simple than I was making it | 22:41 |
| jamielennox | greghaynes: you skipped a file? | 22:42 |
| greghaynes | so it actually only matters for in-chroot | 22:43 |
| greghaynes | since youre not going to be pip installing anything out of chroot | 22:43 |
| greghaynes | and also your out of chroot python shouldnt be borked | 22:43 |
| ianw | yeah, agree on that | 22:44 |
| *** frontrunner has joined #openstack-dib | 22:57 | |
| ianw | arrggghh, seems my test cloud is broken somehow! does anything ever just work!! | 23:17 |
| jamielennox | not in my experience | 23:19 |
| ianw | it's ok, i just keep trying and eventually the error went away | 23:20 |
| ianw | but then realised i built the test image with the python2 change still in, so it wasn't testing anything anyway :) | 23:21 |
| adam_g | testing the image without a cloud is pretty EZ | 23:22 |
| ianw | yeah, the reliability of our internal cloud is sometimes worrying | 23:28 |
| pabelanger | I have some time tonight to finish off the nodepool dsvm job for diskimage-builder | 23:41 |
| *** hashar has quit IRC | 23:50 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!