| *** openstackgerrit has quit IRC | 05:10 | |
| *** aspiers has quit IRC | 05:16 | |
| *** portdirect has quit IRC | 05:16 | |
| *** aspiers has joined #openstack-loci | 05:35 | |
| *** portdirect has joined #openstack-loci | 05:35 | |
| *** ispp has joined #openstack-loci | 07:00 | |
| *** e0ne has joined #openstack-loci | 08:08 | |
| *** pbourke has quit IRC | 08:52 | |
| *** pbourke has joined #openstack-loci | 08:52 | |
| *** openstackgerrit has joined #openstack-loci | 10:20 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/loci master: [WIP] Add a profile for kolla nova https://review.openstack.org/583542 | 10:20 |
|---|---|---|
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/loci master: [WIP] Add a profile for kolla nova https://review.openstack.org/583542 | 10:26 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/loci master: [WIP] Add a profile for kolla neutron https://review.openstack.org/583545 | 10:32 |
| pbourke | SamYaple: the kolla profiles are working :) | 11:51 |
| pbourke | SamYaple: would you be ok with pushing these images as part of the gate? | 11:52 |
| portdirect | pbourke: how do the above work? | 11:53 |
| pbourke | portdirect: take keystone for example. you build the loci image with the (kolla keystone) profile, which adds the packages kolla-ansible expects to be in the image for a functional deploy. Then I build one more image off of that which adds the kolla start scripts | 11:55 |
| pbourke | portdirect: the code to do that is currently under https://github.com/brk3/kolla-loci | 11:55 |
| hrw | elo | 11:55 |
| hrw | building 'requirements' image on aarch64 is fun | 11:56 |
| hrw | dropped '--no-deps' to check how many packages are arch specific. so far 36 | 11:56 |
| pbourke | if the start scripts were in kolla-ansible the kolla-loci piece wouldn't be required at all. But I met resistence with the former approach so this allows us to do it without touching kolla-ansible at all | 11:56 |
| portdirect | Very nice :) | 11:57 |
| portdirect | I'll check it out when I get into the office | 11:57 |
| pbourke | thanks! | 11:58 |
| pbourke | portdirect: its a little funky to build right now as you have to first build the patched loci base + loci image, then finally the kolla-loci child. But if we can publish them as part of the gate it will be much simpler | 11:59 |
| portdirect | Would be good to get your shim in os-infra, would make me feel a lot more comfortable re merge&publish | 12:02 |
| portdirect | Maybe LOKI? (Loci Optimised Kolla images?) :D | 12:04 |
| pbourke | maybe :) | 13:01 |
| pbourke | I guess I just need to figure out what point I want to get it to before applying for addition | 13:04 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/loci master: Add gate job for rabbitmq https://review.openstack.org/583575 | 13:29 |
| hrw | looks like about 10% of pip packages used by requirements image is architecture specific. so far 42/423 | 13:35 |
| SamYaple | hrw: can you expand on what you mean? | 13:44 |
| SamYaple | im not quite following. are you saying that some of the packaging in upper-constraints.txt can't be built on aarch64? | 13:45 |
| hrw | SamYaple: *-linux_aarch64.whl ones. but they build | 13:45 |
| SamYaple | right, but im not getting why you need to drop `--no-deps` | 13:46 |
| hrw | SamYaple: sorry, was just experimenting | 13:46 |
| SamYaple | oh youre welcome to experiment, im just hoping i can save you some time if i understand what you are working toward | 13:47 |
| hrw | SamYaple: ;) | 13:47 |
| SamYaple | about 10% of packages need to be compiled, the others are saved in thier native whl format (previously we pruned native wheels, but stopped for now, i can provide details of why if desired) | 13:48 |
| SamYaple | but unless a specific package *wont* build on aarch64, the rest should work the same way | 13:48 |
| SamYaple | same number of packages between all arches need to be compiled | 13:48 |
| SamYaple | we don't pull compiled wheels from anywhere for any arch, we build them | 13:49 |
| hrw | SamYaple: we go Debian. there are no fresh enough binary packages for Debian/stable (even with backports) | 13:51 |
| hrw | ops. replied to old message - had backlog moved too much | 13:51 |
| hrw | SamYaple: ok. so the only blocker for me now is that scikit-learn failing without numpy | 13:53 |
| SamYaple | cool. I would suggest, as a test, adding numpy after bindep on this line. https://github.com/openstack/loci/blob/master/scripts/requirements.sh#L22 | 13:54 |
| SamYaple | *if* that works for you, i have some options on how to best solve it :) | 13:54 |
| SamYaple | so make that change and rebuild requirements and report back if it works | 13:54 |
| hrw | SamYaple: thanks | 13:56 |
| hrw | SamYaple: started build. the only changes from origin/master are 'pip install numpy' and '-P1' for clear logs | 13:58 |
| hrw | SamYaple: just moved it after install_packages.sh so toolchain is present ;D | 14:00 |
| SamYaple | heh fair enough :) | 14:03 |
| SamYaple | the idea is to get numpy installed in the build image *before* we start building upper-constraints.txt | 14:03 |
| SamYaple | it looked liek scikit-learn was importing numpy in setup.py | 14:04 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/loci master: Add mariadb to infra profile https://review.openstack.org/583584 | 14:08 |
| hrw | SamYaple: looks like | 14:08 |
| * hrw lunch | 14:22 | |
| ispp | Hi, quick question, why u only create master images? Wouldn't make sense to create master images and at least the last stable? | 15:05 |
| SamYaple | hey ispp, you keep popping in and out before I can give you an answer :) | 15:13 |
| SamYaple | so the main reason for that is we dont have any testing of the deployment of these images yet | 15:13 |
| SamYaple | additionally, we decided on a versioning structure that isn't tied to the stable branches | 15:14 |
| SamYaple | the master code can build newton, ocata, pike, queens, and rocky right now | 15:14 |
| SamYaple | The images we publish are meant for breif testing and not production use. You should always be building your own containers from trusted sources for production | 15:15 |
| ispp | SamYaple thanks for the answer, yep, I have to configure it better to avoid this problems hehe, aham, ok I didn't know you were only using them for a testing environment, sometime is possible that master will be broken and the images too so, would it be possible if I propose a change in loci to include at least stable one or you want to continue basing the images for only testing? | 15:18 |
| SamYaple | You are welcome to propose a patch for anything you think seems good | 15:20 |
| SamYaple | We are pretty accepting of new things | 15:21 |
| ispp | Oh, great, thanks, I will try to find some easy fix for this :) | 15:27 |
| *** ispp has quit IRC | 15:41 | |
| hrw | SamYaple: added scipy as well. same scikit-learn | 16:45 |
| SamYaple | :+1: | 16:47 |
| *** e0ne has quit IRC | 16:51 | |
| *** e0ne has joined #openstack-loci | 17:39 | |
| hrw | ok. scipy requires several dev packages. | 17:44 |
| hrw | https://pastebin.com/GcyZ6BpE | 17:44 |
| *** e0ne has quit IRC | 17:44 | |
| *** e0ne has joined #openstack-loci | 17:48 | |
| *** e0ne has quit IRC | 17:51 | |
| SamYaple | hrw: im sure this is a "loci expects X because all distros have X, but the aarch64 packages arent providing X" | 18:02 |
| SamYaple | weve seen similiar issues, but with distro instead of aarch | 18:02 |
| hrw | possible | 18:05 |
| hrw | ok, added libblas-dev liblapack-dev | 18:10 |
| *** e0ne has joined #openstack-loci | 18:10 | |
| *** e0ne has quit IRC | 18:17 | |
| SamYaple | hopefully when you get it built, youll be able to describe the many rough edges. most of us are just numb to them and dont see them anymore :) | 18:27 |
| portdirect | i find the splinters keep me warm | 18:27 |
| hrw | SamYaple: I lost track how many times I was dealing with blas/lapack ;D | 18:28 |
| SamYaple | portdirect: this assumes you are on fire to being with. so it checks out | 18:29 |
| hrw | Running setup.py bdist_wheel for scipy: still running... | 18:31 |
| hrw | progress was made ;D | 18:31 |
| hrw | should I take care also of centos package names? | 18:31 |
| SamYaple | can you confirm that this is an issue outside of aarch64? | 18:36 |
| SamYaple | i would like to track down why we are seeing different things between the arches | 18:36 |
| hrw | will run build on x86-64 | 18:43 |
| SamYaple | if there are differences in architcture packages needed, bindep supports that | 18:51 |
| SamYaple | otherwise if it fails the same way, throw up a patch and well merge it :) | 18:52 |
| SamYaple | single line patch to a central file is really easy to review :D | 18:52 |
| hrw | yep | 18:58 |
| hrw | hm. my fedora box is on docker 1.13ish and misbehave | 19:00 |
| hrw | 20:59 (1s) hrw@puchatek:loci$ docker build --build-arg PROJECT=requirements --build-arg FROM=debian:stretch --tag requirements:master-debian . | 19:00 |
| hrw | Sending build context to Docker daemon 340 kB | 19:00 |
| hrw | Step 1/23 : ARG FROM=ubuntu:xenial | 19:00 |
| hrw | Please provide a source image with `from` prior to commit | 19:00 |
| hrw | have to update | 19:06 |
| hrw | Successfully tagged requirements:master-debian | 19:37 |
| hrw | on x86-64 | 19:37 |
| hrw | no changes | 19:38 |
| *** e0ne has joined #openstack-loci | 19:39 | |
| SamYaple | then i would suggest adding the deps with the aarch64 bindep profile | 19:43 |
| SamYaple | so as not to add deps to the x86_64 that arent needed | 19:44 |
| SamYaple | hrw: alternatively, `apt-get install python-numpy` should also work, if you want to *not* compile it like that | 19:45 |
| SamYaple | you could include it to the requirements build for aarch64 and it *might* work? | 19:45 |
| *** e0ne has quit IRC | 19:52 | |
| hrw | SamYaple: what is a format for arch specific bindep? | 19:57 |
| hrw | will check | 19:58 |
| hrw | btw - libxml2 should be removed from bindep.txt imho. it is a bad taste to depend on library package instead of something which depends on that library | 20:00 |
| hrw | scikit-learn in progress... | 20:00 |
| SamYaple | hrw: libxml2 is literally needed everywhere since the xml api isn't completely gone | 20:00 |
| SamYaple | i would love to remove it | 20:00 |
| hrw | SamYaple: but you use either binaries using libxml2 or python binary modules which use libxml2 | 20:01 |
| SamYaple | portdirect: did we decide to do a seperate bindep.txt for arch specific things? | 20:01 |
| SamYaple | its been a while since we discussed arch support | 20:02 |
| SamYaple | hrw: while yes, we could specify `libxml2 [keystone nova glance cinder neutron etc etc etc]` it is much easier to do [] | 20:02 |
| portdirect | SamYaple: i cant rememeber, we kicked around a few ideas | 20:03 |
| SamYaple | when openstack as a whole drops libxml2 we can remove it entirely | 20:03 |
| portdirect | but not sure what we ended up thinking | 20:03 |
| portdirect | one thing for sure - we prob need to break out the bindep file in some way | 20:03 |
| SamYaple | ok. lets give it some more thinking | 20:03 |
| portdirect | as its ending up a bit of a monster | 20:04 |
| portdirect | eg: the health-check stuff we want/need in osh and pauls kolla work, along with arch support | 20:04 |
| portdirect | prob mean its time | 20:04 |
| hrw | Successfully tagged requirements:master-debian | 20:08 |
| hrw | on aarch64 | 20:08 |
| SamYaple | hrw: w00t! | 20:08 |
| SamYaple | now build keystone! | 20:08 |
| hrw | now split changes and send for review | 20:10 |
| hrw | --build-arg WHEELS=requirements:master-debian - right? | 20:17 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/loci master: bindep: install blas and lapack headers to build scipy on aarch64 https://review.openstack.org/583726 | 20:18 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/loci master: requirements.sh: install numpy and scipy on AArch64 https://review.openstack.org/583727 | 20:18 |
| SamYaple | hrw: if you are trying to do that, you have to upload the wheel somewhere | 20:20 |
| SamYaple | a local registry or dockerhub works | 20:20 |
| hrw | thanks | 20:20 |
| hrw | https://hub.docker.com/r/haerwu/loci-requirements/ | 20:23 |
| * hrw off | 20:37 | |
| hrw | thanks for help guys and see you tomorrow | 20:37 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!