| *** ispp has joined #openstack-loci | 07:00 | |
| ispp | SamYaple could this work? after fixing the comments Jean wrote there of course https://review.openstack.org/#/c/584279/ | 07:14 |
|---|---|---|
| *** ispp has quit IRC | 07:27 | |
| *** ispp has joined #openstack-loci | 07:48 | |
| *** e0ne has joined #openstack-loci | 08:22 | |
| *** pbourke has quit IRC | 08:41 | |
| *** pbourke has joined #openstack-loci | 08:42 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/loci master: [WIP] Add OpenSUSE Leap 15 support for Requirements https://review.openstack.org/586295 | 09:42 |
| *** ispp has quit IRC | 10:09 | |
| evrardjp | ispp it should : ) | 10:15 |
| evrardjp | if not we have other issues in zuul :D | 10:15 |
| hrw | evrardjp: nice work on suse stuff | 10:16 |
| evrardjp | hrw: I am just starting : ) | 10:16 |
| evrardjp | I will probably be more vocal when it builds :) | 10:16 |
| hrw | evrardjp: from that moment to building images is ~hour | 10:16 |
| evrardjp | I am doing a map right now, and I found a few bugs in other things | 10:16 |
| evrardjp | hrw: well, we don't have mariadb in leap15 yet, so I think it will be a little harder than that :p | 10:17 |
| hrw | evrardjp: I do not care yet about !loci stuff ;D | 10:17 |
| evrardjp | hahaha | 10:18 |
| evrardjp | which distro are you using? | 10:18 |
| hrw | evrardjp: for me first goalset was to build requirements+other images. | 10:18 |
| hrw | debian:stretch | 10:18 |
| evrardjp | that seems close to build, isn't it? | 10:18 |
| evrardjp | Should we discuss including those in -nv gating or external CI? | 10:18 |
| hrw | evrardjp: https://git.linaro.org/ci/job/configs.git/tree/ldcg-loci can be useful to you too | 10:18 |
| hrw | evrardjp: that build.sh I use to build own images | 10:19 |
| evrardjp | yeah I guessed :) | 10:19 |
| evrardjp | I have my build.sh too it's just far less extensive than that : p | 10:19 |
| hrw | debian images build fine | 10:19 |
| evrardjp | do you have a CI you can use to report on the image building? | 10:20 |
| hrw | evrardjp: once you pass requirements image other just follow | 10:20 |
| evrardjp | on a per commit bassis maybe? | 10:20 |
| evrardjp | hrw: no doubt | 10:20 |
| hrw | evrardjp: there is a work on arm64 ci for openstack but it has some issues due to being in china | 10:20 |
| hrw | evrardjp: so first we need to get it working with kolla before we go for other projects | 10:20 |
| evrardjp | cool | 10:22 |
| evrardjp | the arm64 thingy | 10:22 |
| evrardjp | sad the china issues | 10:22 |
| hrw | our cloud admin works on getting UK and US clouds operational | 10:22 |
| hrw | evrardjp: ImportError: No module named pkg_resources | 10:29 |
| hrw | evrardjp: your patch with opensuse/leap:15 image fails | 10:29 |
| hrw | evrardjp: you may need to add python-setuptools to install list | 10:31 |
| evrardjp | hrw: yeah, it's still WIP | 10:32 |
| evrardjp | hrw: it will not work as is, as there are a few bindep thingies that need to be done | 10:32 |
| evrardjp | hrw: I will inform you when ready | 10:32 |
| *** ispp has joined #openstack-loci | 10:33 | |
| hrw | sure, just wanted to check ;D | 10:33 |
| evrardjp | hehe | 10:34 |
| hrw | next week (when I will be back home) I plan to take a look at scripts/* | 10:34 |
| hrw | evrardjp: python2-setuptools helped | 10:35 |
| hrw | and yep. bindep stuff hurts you | 10:35 |
| hrw | evrardjp: you basically need separate class for opensuse bindep | 10:43 |
| hrw | dpkg for debian, rpm for rhel/centos, zypper for suse | 10:43 |
| hrw | s/rpm/platform:redhat/ s/zypper/platform:suse/ | 10:45 |
| hrw | one step from proper bindeps for you evrardjp | 10:55 |
| hrw | there are some bindep entries which I would rather remove completly ;D | 10:58 |
| hrw | 82 new packages to install. | 10:59 |
| hrw | ;D | 10:59 |
| hrw | evrardjp: https://paste.fedoraproject.org/paste/FHaenZ8fPHp6691N4tQjAQ for you | 11:00 |
| hrw | evrardjp: with this process of building requirements image for opensuse/leap:15 goes | 11:01 |
| hrw | evrardjp: something failed to build still but should get you closer | 11:08 |
| evrardjp | hrw: it's not needed to separate the two, platform:rpm is supported by suse | 11:59 |
| evrardjp | I see you've written libjpeg8 too | 12:00 |
| hrw | evrardjp: but package names are different | 12:05 |
| hrw | evrardjp: so s/platform:redhat platform:suse/platform:rpm/ where are same and done ;D | 12:06 |
| evrardjp | some | 12:06 |
| evrardjp | yeah well I listed them on a paper -- call me old school | 12:06 |
| evrardjp | I arrive to the same list as you | 12:06 |
| hrw | I will have to do some rebuilds to check can some library packages be dropped | 12:07 |
| hrw | as I highly dislike situation when someone install library package instead of a package which depends on it | 12:08 |
| *** ispp has quit IRC | 12:08 | |
| * hrw off | 12:10 | |
| hrw | laptop with broken open/close lid detector suxx. will repair that next week | 12:11 |
| *** ispp has joined #openstack-loci | 12:20 | |
| evrardjp | haha | 12:25 |
| evrardjp | what do you think of building a blacklist for things in loci? | 12:25 |
| evrardjp | I can definitely see a blacklist for arm things, or for some specific cases | 12:25 |
| evrardjp | hrw: you could have used it for scipy | 12:26 |
| evrardjp | SamYaple: opinion? | 12:26 |
| evrardjp | My point is that we can shave off computing minutes for ppl that don't care about x and y | 12:26 |
| evrardjp | *thinking about the planet* | 12:26 |
| *** ispp has quit IRC | 12:36 | |
| evrardjp | pbourke: I am confused about the dockerfiles folder, what is it useful for? | 12:37 |
| evrardjp | I thought the main DOckerfile was used all the time, and needed overrides. | 12:37 |
| evrardjp | it looks like it will get used for the gating: https://github.com/openstack/loci/blob/8a25daae2985cb492c6bd2e7d5338cb0bbac6450/playbooks/loci-builder.yaml#L34 | 12:39 |
| evrardjp | I just wonder if we could make things simpler | 12:39 |
| SamYaple | evrardjp: we could do a blacklist. but to what end? | 12:41 |
| SamYaple | as far as teh dockerfiles, they are there primarily to make the gate move, but also as an example template for a LCOI base image that the user would be maintaining themselves | 12:42 |
| evrardjp | SamYaple: there are wheels I don't need built -- skipping them makes it easy, and fastens the build | 12:42 |
| SamYaple | oh that, yea no to that | 12:42 |
| SamYaple | it kinda kills part of the drive | 12:42 |
| SamYaple | the requirements image *should* contain everything needed to build official openstack projects | 12:42 |
| evrardjp | I agree, project-wise | 12:43 |
| SamYaple | though blacklisting scipy and losing monasca-anlytics is not great, but fine for right now | 12:43 |
| evrardjp | user wise that's kinda meh | 12:43 |
| SamYaple | as far as building *actually* goes, you dont have to use the requirements image at all | 12:43 |
| evrardjp | I like the idea of sharing layer | 12:43 |
| SamYaple | you can push the ocmpiled wheels to a repo and then ship an `--build-args WHEELS=empty.tar.gz` essentiall | 12:44 |
| SamYaple | so you dont even *need* requirements, its already there as a helper piece | 12:44 |
| evrardjp | completely agree -- but why would I setup a new process, while loci works perfectly | 12:44 |
| evrardjp | about dockerfiles: | 12:44 |
| SamYaple | ive been playing with pushing the whls from the requirements build to my local repo | 12:45 |
| SamYaple | so "loci works perfectly" doesn't mean it can't work better in specific envs | 12:45 |
| evrardjp | I think that https://github.com/openstack/loci/blob/8a25daae2985cb492c6bd2e7d5338cb0bbac6450/playbooks/loci-builder.yaml#L34 is limited: for example https://github.com/openstack/loci/blob/fd31a2f0d5d035c4b0171ea773da331ae9428c6c/playbooks/vars.yaml#L29 name ubuntu is limited, what when we want bionic ? | 12:46 |
| evrardjp | except if it's fine to rename "ubuntu" to "xenial" | 12:47 |
| evrardjp | this way bionic has its own name too | 12:47 |
| evrardjp | SamYaple: what's your opinion? | 12:47 |
| SamYaple | evrardjp: i don't believe the plan was to gate xenial and bionic at the same time | 12:50 |
| SamYaple | weve already be through trusty and xenial, granted it was early on | 12:50 |
| SamYaple | but it was a cut over | 12:51 |
| SamYaple | the current image technically should work on trusty, xenial, and bionic | 12:51 |
| SamYaple | you just provide the base | 12:51 |
| evrardjp | I have the feeling if we start to fiddle with the apt sources for bionic we'll have trouble -- like wheels for nova built for the wrong version of libvirt. | 12:54 |
| *** ispp has joined #openstack-loci | 13:03 | |
| SamYaple | evrardjp: we've got a lot of experince between us for building nova and even libvirt in a docker container, i don't believe there are any issue that will cause major touble | 13:07 |
| SamYaple | bionic won't have any cloud-archive stuff, thats probably the *hardest* part | 13:07 |
| *** mandre is now known as mandre_away | 13:18 | |
| evrardjp | ok | 13:18 |
| evrardjp | that explains | 13:19 |
| *** e0ne has quit IRC | 15:16 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/loci master: Add OpenSUSE Leap 15 support for Requirements https://review.openstack.org/586295 | 15:26 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/loci master: Add OpenSUSE Leap 15 support https://review.openstack.org/586295 | 15:26 |
| -openstackstatus- NOTICE: A zuul config error slipped through and caused a pile of job failures with retry_limit - a fix is being applied and should be back up in a few minutes | 15:38 | |
| *** ChanServ changes topic to "A zuul config error slipped through and caused a pile of job failures with retry_limit - a fix is being applied and should be back up in a few minutes" | 15:38 | |
| evrardjp | SamYaple: what do you think of spreading the jobs per distro? | 15:49 |
| evrardjp | per image at least | 15:49 |
| SamYaple | ah yea, forgot to mention that. the "large" jobs i described before would be distro specific | 15:50 |
| evrardjp | SamYaple: I am thinking of yesterday's conversation, if you make the base jobs longer (multiple branches) | 15:50 |
| SamYaple | so we could non-voting the distros | 15:50 |
| evrardjp | good! | 15:50 |
| evrardjp | perfect | 15:50 |
| evrardjp | that's what I thought too | 15:51 |
| evrardjp | ok we are in sync | 15:51 |
| SamYaple | cool. yea thats what we came up with and havent implemented yet :D | 15:51 |
| evrardjp | want help? | 15:51 |
| evrardjp | it's close to week-end for me, but I can do it next week | 15:52 |
| evrardjp | I'd rather not bring opensuse in gates yet if we change this | 15:52 |
| SamYaple | You are welcome to POC this whole idea for opensuse if you want | 15:53 |
| SamYaple | if youre offering | 15:53 |
| SamYaple | we can adapt that work pretty quickly for the other distros i think | 15:53 |
| evrardjp | well if I do the work I will do it directly for all the distros : p | 15:54 |
| evrardjp | I just rather be in sync | 15:54 |
| evrardjp | with how to do it | 15:54 |
| evrardjp | if you have cycles, go ahead, you probably have a better idea of what you want | 15:54 |
| evrardjp | I can review it | 15:54 |
| SamYaple | Ah i meant that was my plan | 15:55 |
| SamYaple | POC one distro, then quickly move the otehrs to it | 15:55 |
| SamYaple | with both old and new jobs running breifly | 15:55 |
| SamYaple | if you POC opensuse, same thing | 15:55 |
| *** openstackgerrit has quit IRC | 16:04 | |
| *** ispp has quit IRC | 16:11 | |
| evrardjp | SamYaple: if I want to build keystone from my previously built requirements on my local machine, shouldn't WHEELS="<docker image>:<docker tag>" work ? | 16:33 |
| evrardjp | or do I absolutely need a registry to listen on localhost? | 16:35 |
| evrardjp | < noob | 16:38 |
| SamYaple | evrardjp: WHEELS=http://some.site/tarball.gz works | 17:29 |
| SamYaple | also what you posted | 17:29 |
| SamYaple | got confused with line split | 17:29 |
| evrardjp | SamYaple: I don't want to publish it yet | 17:31 |
| SamYaple | wait, confused again. WHEELS="<image>:<tag>" | 17:31 |
| SamYaple | that only works if its on dockerhub | 17:31 |
| evrardjp | exactly | 17:31 |
| evrardjp | question: | 17:31 |
| SamYaple | so push to dockerhub if you want to use it that way? | 17:31 |
| evrardjp | in the requirements image, where are the wheels built | 17:31 |
| SamYaple | or WHEELS="<registry>/<image>:<tag>" would work | 17:31 |
| SamYaple | evrardjp: https://github.com/openstack/loci/blob/master/scripts/requirements.sh#L33 | 17:32 |
| evrardjp | SamYaple: I wanted to use the locally built ones | 17:32 |
| SamYaple | there, it builds wheels | 17:32 |
| SamYaple | oh locally built wheels outside of the requirements image? | 17:32 |
| evrardjp | so / | 17:32 |
| evrardjp | wait | 17:32 |
| SamYaple | im confused by what youre asking sorry | 17:32 |
| SamYaple | WHEELS is super flexible though | 17:32 |
| evrardjp | so I want to avoid pushing to a public registry,whether it's dockerhub or anything else | 17:32 |
| evrardjp | so I wanted to have it somewhere local. | 17:33 |
| SamYaple | then your only option is to host it as a tarball | 17:33 |
| SamYaple | only things going into the container are over the network | 17:33 |
| evrardjp | I thought: 1) maybe I could inherit the requirements image, and point to that location | 17:33 |
| SamYaple | no way to inherit or otherwise transfer the wheels without making a layer in the final image (of the size of all the wheels) | 17:34 |
| evrardjp | SamYaple: is there no way to avoid the "over the network" part? | 17:34 |
| SamYaple | specifically no because of the last statement | 17:34 |
| SamYaple | we are avoiding the "layer" part | 17:34 |
| SamYaple | by using the network | 17:34 |
| SamYaple | fwiw, i spin up an apt-cacher, devpi (pypi mirror), and apache file host for these reasons | 17:35 |
| SamYaple | its still 100% possible to build in an isolated environment with access to proper mirrors. I know because i do it :P | 17:36 |
| evrardjp | ok another question then, is there a way to create easily the .tar.gz file from the requirements image? | 17:36 |
| SamYaple | yep | 17:36 |
| SamYaple | its the last layer in the image, its already a tarball | 17:36 |
| SamYaple | you can extract it | 17:36 |
| evrardjp | please explain for docker noobs? | 17:36 |
| SamYaple | `docker export requirements:tag` | 17:36 |
| SamYaple | or save | 17:37 |
| evrardjp | ok | 17:37 |
| SamYaple | i cant ever remember | 17:37 |
| SamYaple | its the one that doesnt flatten | 17:37 |
| evrardjp | yeah but it's a docker command | 17:37 |
| SamYaple | no, there are two commands, export and save | 17:37 |
| SamYaple | the file it produces is a tarball with other tarballs | 17:37 |
| SamYaple | additionally, you can run `WHEELS=<location> scripts/fetch_wheels.py` and itll pump out /opt/loci/wheels.tar.gz | 17:38 |
| SamYaple | if you want to beef up fetch_wheels.py to be a touch mroe generic, thats fine | 17:38 |
| SamYaple | but the logic for layer extraction is there | 17:38 |
| evrardjp | not sure to understand | 17:40 |
| evrardjp | about `WHEELS=<location> scripts/fetch_wheels.py` and itll pump out /opt/loci/wheels.tar.gz | 17:41 |
| SamYaple | https://github.com/openstack/loci/blob/master/scripts/fetch_wheels.py | 17:45 |
| evrardjp | yes I am on it | 17:45 |
| SamYaple | https://github.com/openstack/loci/blob/master/scripts/fetch_wheels.py#L121 | 17:45 |
| evrardjp | so / | 17:46 |
| SamYaple | /tmp not /opt/loci/wheels | 17:46 |
| SamYaple | you can do it by hand, and extract the right tarball with the docker commands, or you can use that script | 17:46 |
| SamYaple | or you can reference that script to see what its doing | 17:46 |
| evrardjp | still not clear. | 17:46 |
| evrardjp | well it's clear if it's external: I can use https://github.com/openstack/loci/blob/5682a103f0588f148b955732a409c76b61c9a59c/scripts/fetch_wheels.py#L105 | 17:47 |
| evrardjp | else registry | 17:47 |
| evrardjp | my point is to point to a wheel location so L102 | 17:47 |
| SamYaple | L102 is not what you think it is | 17:48 |
| SamYaple | thats helper code when run outside the container like im describing | 17:48 |
| SamYaple | there is no wheels file in the container, nothing to f.read() from | 17:48 |
| evrardjp | so your point is I have to extract from requirements a tar containing the wheels -- then copy to docker image -- then point to it using WHEELS= then remove the file | 17:48 |
| SamYaple | if you want to use WHEELS like that, then thats one way you can do it. yes | 17:49 |
| SamYaple | but at the end of the day, WHEELS are fetched over the network | 17:49 |
| SamYaple | lots of ways to provide that network fetching though | 17:49 |
| evrardjp | yeah I could run my own registry that listens on everything | 17:49 |
| evrardjp | and I'd be done | 17:49 |
| SamYaple | docker run -d --net=host registry:2 | 17:50 |
| SamYaple | does that pretty well | 17:50 |
| SamYaple | but we do have the mirror setups in the gate | 17:50 |
| evrardjp | yeah | 17:50 |
| SamYaple | https://github.com/openstack/loci/blob/master/playbooks/setup-gate.yaml#L49 | 17:51 |
| SamYaple | from there to end | 17:51 |
| evrardjp | ok so now I want to learn how to extract those wheels to tar.gz as docker save extract multiple layers | 17:51 |
| SamYaple | using apach as the git mirror | 17:51 |
| evrardjp | SamYaple: yes I read that. I didn't want to follow that route | 17:51 |
| SamYaple | well git cache more accurately | 17:51 |
| evrardjp | I wanted to pass files locally, because of some limitation. | 17:51 |
| SamYaple | evrardjp: last layer in the image is a tarball containing only *whl and upper-constraints.txt from the version it was built from | 17:52 |
| SamYaple | I dont know what to say about the local passing other than we have no plans to ever support it under the current structure which uses docker build | 17:53 |
| SamYaple | or i should rephrase, we dont plan on supporting adding additional layers unneccesarily due to bloating the size | 17:55 |
| SamYaple | if docker build allows us to do this without saving all wheels as a layer, then go for it | 17:55 |
| evrardjp | SamYaple: I will probably need to have some creative thinking | 17:56 |
| evrardjp | as I won't be able to run a shell script either | 17:57 |
| evrardjp | that's gonna be fun! | 17:57 |
| evrardjp | anyway | 17:57 |
| evrardjp | thanks! | 17:57 |
| SamYaple | no problem! but focus on the docker limitation part | 17:57 |
| SamYaple | there is no way to do this from the LOCI side | 17:57 |
| SamYaple | dockers gotta change | 17:58 |
| SamYaple | ah right. i should probably add, we support `docker build <gitrepo>` so anything you do think up has to exist in the Dockerfile and not as an external build tool | 17:58 |
| evrardjp | that was my case anyway | 17:59 |
| evrardjp | I can only use Dockerfiles, and no external things. | 17:59 |
| evrardjp | limits the things. | 18:00 |
| evrardjp | I will try to see what I can find | 18:00 |
| evrardjp | I am off for today | 18:00 |
| evrardjp | thanks for the help! | 18:00 |
| SamYaple | see ya buddy! take care | 18:02 |
| *** e0ne has joined #openstack-loci | 20:06 | |
| *** e0ne has quit IRC | 20:22 | |
| *** pbourke has quit IRC | 21:10 | |
| *** e0ne has joined #openstack-loci | 21:16 | |
| *** pbourke has joined #openstack-loci | 21:27 | |
| *** e0ne has quit IRC | 22:00 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!