| jamielennox | yea, so locally if you do lsattr on a symlink you see the failure | 00:02 |
|---|---|---|
| ianw | but why are you the first person to see this? :) | 00:03 |
| jamielennox | but i have tried again to just write 8.8.8.8 in and i'm still seeing errors | 00:03 |
| jamielennox | ianw: i do seem to have that knack :P | 00:03 |
| jamielennox | someone from -infra obviously did notice this but just hacked around it | 00:08 |
| ianw | jamielennox: but ... we do tests without infra elements and it works | 00:09 |
| *** owalsh has joined #openstack-dib | 00:09 | |
| jamielennox | and also it's only a thing here because i'm building images in a substantially different environment to what i want to run them in so resolv.conf is an issue | 00:09 |
| jamielennox | so if that's hitting the errexit we need to make that test succeed? | 00:11 |
| ianw | jamielennox: can you post a log? | 00:13 |
| ianw | https://github.com/openstack/diskimage-builder/blob/master/diskimage_builder/lib/common-functions#L299 | 00:13 |
| ianw | step 1 ... if resolv.conf is a symlink or a file, then move it out of the way, and recreate it pointing to 8.8.8.8 | 00:13 |
| jamielennox | i have a regular nodepool bulid log, it doesn't have DEBUG on | 00:13 |
| ianw | no, not quite, either cat the local resolv.conf into it, or if that doesn't exist (? how) point it to 8.8.8.8 | 00:14 |
| ianw | ergo /etc/resolv.conf should be a file at this point | 00:14 |
| ianw | so ... either something during your build is re-writing resolv.conf | 00:15 |
| ianw | or, maybe the lsattr *does* have issues on aufs | 00:15 |
| ianw | can we determine which of the two above it is? | 00:15 |
| jamielennox | ok, my setup is a bit unusual so need to hack some stuff | 00:17 |
| jamielennox | i'll put in an ls -al /etc/resolv.conf | 00:17 |
| ianw | hmm, i'm not super good with docker. i don't think my local instance is using aufs | 00:18 |
| jamielennox | i'm not actually sure how mine is set up | 00:20 |
| ianw | yeah, this one i've got is debian and using devicemapper it seems | 00:21 |
| jamielennox | i'll need to do like echo $(cat /etc/resolv.conf) right? | 00:22 |
| jamielennox | will cat end up in logs? | 00:22 |
| ianw | yeah, if you're catching stdout | 00:28 |
| jamielennox | 2017-07-06 00:32:13,098 INFO nodepool.image.build.ubuntu-xenial: b'lrwxrwxrwx 1 root root 29 Jul 6 00:29 /tmp/dib_build.irsygmWn/mnt/etc/resolv.conf -> ../run/resolvconf/resolv.conf' | 00:34 |
| jamielennox | 2017-07-06 00:32:13,098 INFO nodepool.image.build.ubuntu-xenial: b'nameserver 8.8.8.8' | 00:34 |
| jamielennox | 2017-07-06 00:32:13,098 INFO nodepool.image.build.ubuntu-xenial: b'nameserver 8.8.4.4' | 00:34 |
| jamielennox | 2017-07-06 00:32:13,102 INFO nodepool.image.build.ubuntu-xenial: b'lsattr: Operation not supported While reading flags on /tmp/dib_build.irsygmWn/mnt/etc/resolv.conf' | 00:34 |
| jamielennox | ah, so i think what's happening there is it's a symlink, but when i cat << EOF > /etc/resolv.conf it's writing through the symlink | 00:35 |
| jamielennox | and i'm pretty sure that the resolvconf package is setting up that link | 00:36 |
| ianw | so that might be the key ... where's that coming from? | 00:37 |
| jamielennox | that's me | 00:37 |
| jamielennox | https://github.com/jamielennox/nodepool-dib-elements/blob/master/bonnyci-nodepool/finalise.d/55-resolv-conf | 00:37 |
| jamielennox | but assume i put an rm /etc/resolv.conf in there first, lsattr will succeed (i think - not sure on aufs), and then it will be replaced with the original resolv.conf? | 00:39 |
| ianw | i mean, the resolvconf package ... i'm not seeing that on our xenial builds, is something different? | 00:39 |
| jamielennox | i'm not sure if i specify that manually or it's part of base | 00:39 |
| jamielennox | it doesn't look like i'm specifying that anywhere specially | 00:41 |
| ianw | listed as "suggested" in http://nb04.openstack.org/dib.ubuntu-xenial.log but not installed | 00:43 |
| jamielennox | so i'm actually fine with having resolvconf installed, it at least appears to be doing the right thing | 00:44 |
| jamielennox | like when the link is in place i get dns | 00:44 |
| ianw | i'm just spinning up an env to test | 00:46 |
| jamielennox | ianw: so i changed the element to delete the old resolv.conf first | 01:00 |
| jamielennox | no error message any more, but my resolv.conf is definitely not what i set it too | 01:01 |
| ianw | lsattr: Operation not supported While reading flags on /opt/tmp/dib_build.symqgeYj/mnt/etc/resolv.conf | 01:04 |
| ianw | Unmount /opt/tmp/dib_build.symqgeYj/mnt/sys | 01:04 |
| ianw | Unmount /opt/tmp/dib_build.symqgeYj/mnt/proc | 01:04 |
| ianw | Unmount /opt/tmp/dib_build.symqgeYj/mnt/dev/pts | 01:04 |
| ianw | Unmount /opt/tmp/dib_build.symqgeYj/mnt/dev | 01:04 |
| ianw | that's with resolvconf installed. but yeah, that *doesn't* stop the build, it still produced the .qcow2 | 01:04 |
| jamielennox | yep, the build was succeeding for me too | 01:07 |
| jamielennox | the question is what should be the right thing there? | 01:08 |
| jamielennox | I'd be happy with a well known file location | 01:10 |
| jamielennox | like if [-e /etc/resolv.conf.GOOD ] move that instead of resolv.conf.ORIG | 01:10 |
| jamielennox | or some test to see if the /etc/resolv.conf that is in place at that point is the same as the one we copied in originally? | 01:12 |
| jamielennox | if it's a symlink i can't set it immutable (which is not a solution i like anyway) | 01:14 |
| ianw | /opt/tmp/dib_build.eAzLKp2u/mnt/etc/resolv.conf -> ../run/resolvconf/resolv.conf | 01:16 |
| jamielennox | yep | 01:18 |
| ianw | jamielennox: well, i built an image with resolvconf and just putting in a "if -L resolv.conf" leave it alone | 01:37 |
| ianw | it boots, /etc/resolv.conf points to /var/run/resolv.conf ... but it's empty | 01:37 |
| ianw | oooohhhh ... minimal image. i haven't put any sort of dhcp resolver in | 01:38 |
| jamielennox | so that's still not ideal because if i write resolv.conf out as a regular file in an element it will still be replaced | 01:40 |
| jamielennox | on the other hand though it's probably not good practice to mess with a resolv.conf in the middle of an element because it can screw up the elements making real network connections | 01:42 |
| jamielennox | (which is why it would be being handled specially originally) | 01:46 |
| ianw | jamielennox: if you write it out as a file, you just have to set it immutable | 01:46 |
| ianw | and do it right at the end ... | 01:46 |
| ianw | alright, i have a working xenial+resolvconf image ... | 01:49 |
| jamielennox | gotta run out for an hour or so | 02:09 |
| ianw | np, just putting up patch now | 02:09 |
| jamielennox | the -L test would solve my problem, but it feels like a very specific solution | 02:09 |
| jamielennox | when i changed the symlink to rm -f /etc/resolv.conf and cat ..8.8.8.8 > it was still deleted | 02:10 |
| jamielennox | because it never tested whether it was changed | 02:10 |
| ianw | jamielennox: if you set it immutable, it wouldn't have been removed though? | 02:14 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add symlink test for resolv.conf restore https://review.openstack.org/480805 | 02:18 |
| jamielennox | ianw: -1 | 03:44 |
| ianw | heh, that's what i get for manually copying | 03:47 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add symlink test for resolv.conf restore https://review.openstack.org/480805 | 03:48 |
| jamielennox | yea, i guess setting it immutable would have worked, i'd prefer not to | 03:55 |
| *** andreas-f has joined #openstack-dib | 04:49 | |
| *** clarkb1 has joined #openstack-dib | 05:20 | |
| *** clarkb has quit IRC | 05:22 | |
| *** andreas-f has quit IRC | 06:07 | |
| *** andreas-f has joined #openstack-dib | 06:41 | |
| *** andreas-f has quit IRC | 07:10 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder master: Docker matrix build https://review.openstack.org/414347 | 07:21 |
| *** isaacb has joined #openstack-dib | 07:24 | |
| hwoarang | ianw: do you have time to talk abuot https://review.openstack.org/#/c/478443/8/diskimage_builder/elements/opensuse/pre-install.d/00-opensuse-setup ? | 08:29 |
| ianw | hwoarang: a little ... it's probably fine as is | 08:48 |
| hwoarang | ianw: yeah i was thinking that it's ok to leave the mirrors inside the image as is. but I can work on a separate patch to decouple that | 08:49 |
| ianw | hwoarang: it's up to you. obviously not important when we just throw images away in func tests | 08:50 |
| hwoarang | but this will also affect the nodepool images right? the produced images will have the openstack mirrors in them | 08:51 |
| hwoarang | but i guess that's fine | 08:51 |
| ianw | we're not setting any of this for nodepool builds | 08:51 |
| hwoarang | ah ok | 08:51 |
| hwoarang | so the nodepool images are using the distro mirrors? | 08:52 |
| hwoarang | interesting. i thought you were using the infra mirrors for all the CI jobs | 08:52 |
| ianw | hwoarang: it's two things ... the actual builds, that happen on nb0X and then get uploaded to the CI providers all happen from upstream mirrors | 08:53 |
| ianw | then when we bring up nodes, we run project-config:nodepool/scripts/configure_mirror.sh which looks at what region it's in and rewrites things to use that regions mirror | 08:54 |
| hwoarang | aha | 08:54 |
| hwoarang | this one seems to know what to do with opensuse | 08:55 |
| hwoarang | so we are good on that end | 08:55 |
| hwoarang | ok then.i think i get it now | 08:55 |
| hwoarang | thank you | 08:55 |
| ianw | yes, so when running devstack on suse, for example, package installs happen from local mirrors | 08:55 |
| ianw | the problem with centos was that running "yum" to create the chroot on ubuntu was going back to default mirrors | 08:55 |
| ianw | which made the CI jobs very unreliable ... often we'd get a bad mirror in the rotation and then that causes havoc failing randomly | 08:56 |
| ianw | so that's what i was referring to with the "building chroot" bit ... making sure that the zypper call running on ubuntu was pulling from the mirror | 08:57 |
| hwoarang | yep i get it now | 08:58 |
| hwoarang | thank you for the explanation | 08:58 |
| ianw | :) | 09:02 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/diskimage-builder master: elements: openstack-ci-mirrors: Use openSUSE mirrors for gating jobs https://review.openstack.org/478443 | 09:14 |
| *** mdbooth has joined #openstack-dib | 11:45 | |
| mdbooth | Can anybody help: 'openstack overcloud image build' fails for me using CentOS/RDO master | 11:46 |
| mdbooth | The symptom is: + systemctl enable os-collect-config | 11:46 |
| mdbooth | Operation failed: No such file or directory | 11:46 |
| mdbooth | The immediate cause is that it wasn't installed, as there are no rdo yum repos in the dib chroot while it builds | 11:47 |
| mdbooth | So all openstack package are actually missing | 11:47 |
| mdbooth | I'm running this on the undercloud, so the repos are obviously present on the host | 11:48 |
| mdbooth | I'm not familiar with dib. Can any point me to some code which constructs the chroot, and may be repsponsible for configuring yum repos? | 11:54 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/diskimage-builder master: elements: openstack-ci-mirrors: Use openSUSE mirrors for gating jobs https://review.openstack.org/478443 | 12:06 |
| *** farosas_ has joined #openstack-dib | 12:08 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/diskimage-builder master: elements: openstack-ci-mirrors: Use openSUSE mirrors for gating jobs https://review.openstack.org/478443 | 12:30 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/diskimage-builder master: elements: openstack-ci-mirrors: Use openSUSE mirrors for gating jobs https://review.openstack.org/478443 | 14:44 |
| *** isaacb has quit IRC | 14:57 | |
| clarkb1 | mdbooth: dib operates in stages with each stage executing scripts included by elements for that stage via run parts | 14:58 |
| clarkb1 | mdbooth: you'll want to find out what elements your image build is using then go from there to see what is configuring yum repos. Dib itself won't do that, the elements should | 14:58 |
| *** farosas_ is now known as farosas | 15:06 | |
| *** clarkb1 is now known as clarkb | 15:54 | |
| mdbooth | clarkb: Thanks. Got it sorted in the end. There's some special sauce to configure yum repos I wasn't aware of. | 15:55 |
| *** andreas-f has joined #openstack-dib | 19:50 | |
| *** andreas-f has quit IRC | 20:49 | |
| *** farosas has quit IRC | 21:09 | |
| *** Sukhdev has joined #openstack-dib | 23:40 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!