| *** mjturek has quit IRC | 00:06 | |
| johnsom | ianw Thanks for working on that mirror patch. Our centos gate has been.... rough | 00:07 |
|---|---|---|
| prometheanfire | ianw: in order to do package actions we need to sync so we need space | 00:22 |
| ianw | johnsom: i had myself convinced that we needed the final image to not refer the to mirror, but looking again i don't think that's the case ... so hopefully that makes things simpler | 00:41 |
| johnsom | Yeah, I think we enable/disable with the ci-mirrors element and call it good | 00:41 |
| ianw | oh, arrgh ... didn't seem to pick it up, but tracing not enabled | 00:52 |
| ianw | ooohhhh; except it's not going to be in the chroot. sigh, always something | 00:53 |
| ianw | or i should say, the mirror config isn't in the chroot | 00:53 |
| prometheanfire | ianw: is there a way to reactivate the cache for finalize.d? | 01:15 |
| prometheanfire | that'd work too | 01:15 |
| ianw | not really, i mean it's supposed to be things that happen inside the final image. it seems like you want post-install.d more maybe | 01:26 |
| *** openstackgerrit has quit IRC | 01:37 | |
| *** tflink has quit IRC | 01:37 | |
| *** odyssey4me has quit IRC | 01:37 | |
| *** johnsom has quit IRC | 01:37 | |
| *** mgagne has quit IRC | 01:37 | |
| *** pleia2 has quit IRC | 01:37 | |
| *** cloudnull has quit IRC | 01:37 | |
| *** hwoarang has quit IRC | 01:37 | |
| *** hogepodge has quit IRC | 01:37 | |
| *** cmurphy has quit IRC | 01:37 | |
| *** tonyb has quit IRC | 01:37 | |
| *** adam_g has quit IRC | 01:37 | |
| *** rm_work has quit IRC | 01:37 | |
| *** fungi has quit IRC | 01:37 | |
| *** TheJulia has quit IRC | 01:37 | |
| *** mordred has quit IRC | 01:37 | |
| *** vabada has quit IRC | 01:37 | |
| *** strigazi has quit IRC | 01:37 | |
| *** zigo has quit IRC | 01:37 | |
| *** prometheanfire has quit IRC | 01:37 | |
| *** jesusaur has quit IRC | 01:37 | |
| *** ianw has quit IRC | 01:37 | |
| *** ChanServ has quit IRC | 01:37 | |
| *** ianw has joined #openstack-dib | 01:43 | |
| *** jesusaur has joined #openstack-dib | 01:43 | |
| *** odyssey4me has joined #openstack-dib | 01:43 | |
| *** tflink has joined #openstack-dib | 01:43 | |
| *** prometheanfire has joined #openstack-dib | 01:43 | |
| *** strigazi has joined #openstack-dib | 01:43 | |
| *** zigo has joined #openstack-dib | 01:43 | |
| *** vabada has joined #openstack-dib | 01:43 | |
| *** rm_work has joined #openstack-dib | 01:44 | |
| *** fungi has joined #openstack-dib | 01:44 | |
| *** pleia2 has joined #openstack-dib | 01:44 | |
| *** cloudnull has joined #openstack-dib | 01:44 | |
| *** openstackgerrit has joined #openstack-dib | 01:44 | |
| *** hwoarang has joined #openstack-dib | 01:45 | |
| *** hogepodge has joined #openstack-dib | 01:45 | |
| *** cmurphy has joined #openstack-dib | 01:45 | |
| *** TheJulia has joined #openstack-dib | 01:46 | |
| *** mordred has joined #openstack-dib | 01:46 | |
| *** adam_g has joined #openstack-dib | 01:46 | |
| *** tonyb has joined #openstack-dib | 01:46 | |
| *** ChanServ has joined #openstack-dib | 01:47 | |
| *** card.freenode.net sets mode: +o ChanServ | 01:47 | |
| *** mgagne has joined #openstack-dib | 01:49 | |
| *** trungnv has joined #openstack-dib | 02:16 | |
| ianw | 2018-10-10 03:21:24.329147 | controller | 2018-10-10 03:21:24.328 | Loading mirror speeds from cached hostfile | 03:27 |
| ianw | 2018-10-10 03:21:27.460514 | controller | 2018-10-10 03:21:27.460 | * epel: mirror.bhs1.ovh.openstack.org | 03:27 |
| prometheanfire | ? | 04:31 |
| *** chhagarw has joined #openstack-dib | 05:01 | |
| *** olivierb has joined #openstack-dib | 06:42 | |
| *** odyssey4me has quit IRC | 07:07 | |
| *** odyssey4me has joined #openstack-dib | 07:08 | |
| *** mgagne has quit IRC | 07:09 | |
| *** tflink has quit IRC | 07:09 | |
| *** mgagne has joined #openstack-dib | 07:10 | |
| *** tflink has joined #openstack-dib | 07:10 | |
| *** mgagne has quit IRC | 07:34 | |
| *** mgagne has joined #openstack-dib | 07:36 | |
| *** chhagarw has quit IRC | 08:43 | |
| *** dhill_ has joined #openstack-dib | 11:38 | |
| openstackgerrit | Ankit Kumar proposed openstack/diskimage-builder master: Adding new dib element https://review.openstack.org/578773 | 12:00 |
| *** jesusaur has quit IRC | 12:10 | |
| *** jesusaur has joined #openstack-dib | 12:16 | |
| *** mjturek has joined #openstack-dib | 13:50 | |
| *** johnsom has joined #openstack-dib | 14:30 | |
| *** clarkb has joined #openstack-dib | 14:43 | |
| prometheanfire | ianw: I'm able to build images just fine with it :| | 16:36 |
| *** olivierb has quit IRC | 19:30 | |
| prometheanfire | ianw: if disk space is an issue then gentoo can't ever do package actions in finalize.d | 20:00 |
| prometheanfire | ianw: is there a way to increase the size available in finalize.d? Is there a way to use caching in finalize.d? | 20:01 |
| prometheanfire | I ask because bootloader's 50-bootloader calls install-packages | 20:02 |
| prometheanfire | even when moving the package stuff to post-installs.d it runs out of space | 20:03 |
| prometheanfire | because of that installs packages | 20:03 |
| prometheanfire | also | 20:03 |
| prometheanfire | that finalize.d was there on purpose, because stuff like bootloader does package actions that need cleaning up | 20:03 |
| prometheanfire | so wtf do I do | 20:04 |
| clarkb | can you install during install.d the run config in finalize? | 20:09 |
| prometheanfire | clarkb: the actions I take in finalize.d are to make sure everything is properly linked and stuff from a packaging perspective | 20:11 |
| prometheanfire | @preserved-rebuild is the big one | 20:12 |
| prometheanfire | clarkb: https://gist.github.com/prometheanfire/1eefd539da9b72f3ae416cab932ec536 | 20:12 |
| prometheanfire | that | 20:12 |
| * prometheanfire doesn't know why the grub package isn't installed in the install phase | 20:14 | |
| clarkb | prometheanfire: on debuntu it is because they configure grub iirc | 20:18 |
| clarkb | why would packages need to be rebuilt? | 20:18 |
| prometheanfire | the preserved-rebuild thing? | 20:19 |
| prometheanfire | https://wiki.gentoo.org/wiki/Preserved-rebuild | 20:19 |
| clarkb | can that go into a 99 step for install? | 20:20 |
| clarkb | basically clean it up at the very end of package installs? | 20:20 |
| prometheanfire | no, because finalize.d does package actions | 20:20 |
| prometheanfire | and that happens after install | 20:20 |
| prometheanfire | it has the be the LAST package action | 20:21 |
| prometheanfire | increasing the disk size seems to 'fix' it | 20:21 |
| clarkb | ya dib assumes a disk size before finalize iirc with some small amount of padding | 20:24 |
| clarkb | you can override that | 20:24 |
| prometheanfire | I don't see a place to do so outside of overriding it for everyone | 20:25 |
| prometheanfire | tests/run_functests.sh defines DIB_CMD | 20:25 |
| prometheanfire | and it's used twice, with the options hard coded (--image-size undefined) | 20:26 |
| clarkb | you'd have to set the image size value as part of the element I think | 20:32 |
| prometheanfire | playbooks/legacy/dib-dsvm-functests-python2-ubuntu-trusty-extras/run.yaml defines running the gentoo test | 20:33 |
| prometheanfire | no clue what gentoo/build-succeeds links to | 20:34 |
| ianw | i really don't think padding the size of the disk in a functional test is a good approach. clearly there is a disconnect here between what the phase is intended to do and what is required | 20:34 |
| prometheanfire | ianw: it's performing package actions, what phase during or after the finalize.d phase can I run my cleanup? | 20:35 |
| prometheanfire | cleanup.d doesn't work because it's not in a chroot | 20:36 |
| prometheanfire | the problem is packages being installed during finalize.d | 20:36 |
| prometheanfire | namely, grub (in bootloader) | 20:36 |
| prometheanfire | if packages stop getting installed there I can stop cleaning up there | 20:36 |
| prometheanfire | https://paste.pound-python.org/show/WITKeUi3m3nZGfL5oTLY/ are the places install-packages are called in finalize.d elements | 20:41 |
| prometheanfire | let me put it this way | 20:43 |
| prometheanfire | 1. install-packages will sync repos if needed | 20:43 |
| prometheanfire | 2. if using dib caching it's always needed as cache is not available in finalize.d | 20:44 |
| prometheanfire | 3. download + unpack is at least 2x the space of just being there to start with | 20:44 |
| prometheanfire | 4. space runs out because download + unpack is > existing | 20:44 |
| prometheanfire | that will happen with any distro, gentoo just happens to have a larger package cache so hit it first | 20:45 |
| prometheanfire | btw, I did try moving it to post-installs.d but still ran out of space | 20:47 |
| prometheanfire | 2018-10-10 19:58:26.992 | rsync: write failed on "/usr/portage/metadata/md5-cache/dev-haskell/extra-1.3.1": No space left on device (28) | 20:47 |
| prometheanfire | that's with a 3G image size | 20:47 |
| clarkb | with other distros you don't do any package things in finalize iirc | 20:48 |
| clarkb | because at that point the disk size is already calculated | 20:48 |
| prometheanfire | they can get away with it because they don't require preserved-rebuild | 20:48 |
| prometheanfire | if an element changes package use flags or keywords in finalize.d that should be propigated out as needed in the final image | 20:49 |
| ianw | prometheanfire: in post-install.d you should not be inside the sized image, you should just be in a chroot on your disk. possibly if that was using a tmpfs you may see something similar, i'm not sure it's related | 20:50 |
| prometheanfire | if an element installs a package in finalize.d without changing stuff like that it should be fine, but if it uninstalls that causes problems too | 20:50 |
| prometheanfire | ianw: finalize.d is where the nospaceleft was found | 20:50 |
| prometheanfire | the stuff I moved to post install worked fine | 20:50 |
| prometheanfire | 2018-10-10 19:56:59.516 | dib-run-parts Running /tmp/in_target.d/finalise.d/50-bootloader | 20:51 |
| prometheanfire | 2018-10-10 19:56:59.667 | INFO:root:Mapping for bootloader : grub-pc | 20:51 |
| prometheanfire | 2018-10-10 19:58:26.992 | rsync: write failed on "/usr/portage/metadata/md5-cache/dev-haskell/extra-1.3.1": No space left on device (28) | 20:51 |
| prometheanfire | 2018-10-10 19:58:26.992 | rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3] | 20:51 |
| ianw | sorry, i must have misunderstood "btw, I did try moving it to post-installs.d but still ran out of space" | 20:51 |
| prometheanfire | btw, I did try moving it to post-installs.d but still ran out of space in finalize.d | 20:51 |
| prometheanfire | :D | 20:51 |
| prometheanfire | again, it's because package actions are taking place in finalize.d, this causes us to sync the repo, which runs us out of space | 20:52 |
| ianw | right, i do get the problem | 20:52 |
| prometheanfire | there's not much I can do about that unless the dib cache can be accessed in finalize.d (is there a reason it's not?) | 20:53 |
| ianw | however, installing the bootloader in this phase has been there since, day 1? essentially forever | 20:53 |
| ianw | well everything is unmounted before we run the "du" to decide on the final image size | 20:53 |
| prometheanfire | I think it's because portage switched to using a tmp dir for syncing so that we could gpg verify it | 20:54 |
| prometheanfire | as far as unmounting, you could unmount, du, remount? | 20:54 |
| ianw | things *could* be re-mounted into the final image, i guess | 20:54 |
| * prometheanfire really does like using dib caching, it's sped up my builds quite a bit | 20:55 | |
| prometheanfire | cut the time by more than half at least | 20:57 |
| ianw | i think that for nodepool we cache the inputs for the -minimal debootstrap/rpm equivalent initial builds | 20:58 |
| ianw | however, i don't think we then cache inside the chroot | 20:58 |
| prometheanfire | so nodepool doesn't use dib cache normally? | 20:59 |
| *** mjturek has quit IRC | 21:00 | |
| ianw | well it uses some caching, but i don't think it is using in-chroot caching like this | 21:01 |
| ianw | IIRC, yum or dnf or something started doing something with it's package caching using some sort of system-specific checksum | 21:02 |
| ianw | it broke the idea that you could share a package cache generically | 21:02 |
| prometheanfire | ok, well, it works here we have metadata base on build options so we can share across most profiles (maybe not musl) | 21:03 |
| prometheanfire | actually, since I'm doing a cache per profile that works too, nevermind | 21:03 |
| prometheanfire | also, distfiles caching is there too (while it doesn't help with compile time it's nice to have stuff fetched already) | 21:03 |
| prometheanfire | we actually made our own tar based format thing to record that metadata | 21:04 |
| prometheanfire | https://dev.gentoo.org/~zmedico/portage/doc/man/xpak.5.html | 21:04 |
| ianw | yeah DIB_APT_LOCAL_CACHE is explicitly turned off in several places | 21:07 |
| prometheanfire | well, it works for gentoo at least :P | 21:08 |
| prometheanfire | looks like the extra package install stuff is run in the final image too (see image-functions) | 21:12 |
| *** dhill_ has quit IRC | 21:13 | |
| ianw | it's certainly not been expressed in the documentation that you *can't* install in these phases, in fact the opposite i guess | 21:15 |
| ianw | https://docs.openstack.org/diskimage-builder/latest/developer/developing_elements.html | 21:15 |
| ianw | so i think we have to consider it as a limitation that we will have to have provision for installs in these phases | 21:16 |
| prometheanfire | yep | 21:17 |
| prometheanfire | would changing the scale here work for you? | 21:17 |
| prometheanfire | du_size=$(tail -n1 ${du_output} | cut -f1 | awk '{print int($1 / 0.6)}') | 21:17 |
| prometheanfire | line 373 of disk-image-create | 21:17 |
| ianw | i don't really think that's much of a solution, we make bigger images for no reason | 21:19 |
| *** dhill_ has joined #openstack-dib | 21:19 | |
| ianw | you could re-mount the cache in 01 finalise.d and unmount it in 99 finalise.d i guess | 21:19 |
| prometheanfire | I can try that too, should work since if cache is not used then it's already a larger image | 21:21 |
| prometheanfire | actually, would the unmount stuff do that automatically? | 21:22 |
| prometheanfire | can root.d run stuff from the bin dir (rather avoid duplicating code) | 21:22 |
| prometheanfire | ianw: what's the equiv of TMP_MOUNT_PATH for finalize.d? | 21:28 |
| ianw | i think $TARGET_ROOT is what you want | 21:43 |
| prometheanfire | k | 21:43 |
| prometheanfire | ianw: it needs to happen outside finalize.d | 21:47 |
| prometheanfire | 2018-10-10 21:46:15.207 | /tmp/in_target.d/finalise.d/01-cache: line 9: TARGET_ROOT: unbound variable | 21:47 |
| ianw | hrm, yes i guess that is all done but run_d_in_target and that's not available | 21:50 |
| ianw | i wonder, /tmp/in_target.d should always be bind mounted in? | 21:52 |
| ianw | could you possibly bind mount it in there, and just fiddle symlinks? | 21:52 |
| prometheanfire | but it's all cleaned up? | 21:53 |
| prometheanfire | not sure how you want me to do that | 21:54 |
| prometheanfire | want me to modify 'function run_d_in_target'? | 21:57 |
| ianw | no i mean you can access /tmp/in_target.d in finalise.d scripts | 21:58 |
| prometheanfire | ok, and it should have access to something? | 21:59 |
| ianw | if you had the cache setup and living in $TMP_HOOKS_PATH/gentoo-cache, then you should be able to symlink /tmp/in_target.d/gentoo-cache in all phases | 22:01 |
| ianw | i think | 22:01 |
| prometheanfire | ah, so change the root.d/50-gentoo-cache to mount there (and bind mount from there) | 22:02 |
| ianw | yes, something along those lines; it is supposed to be the communication channel between outside & inside build | 22:04 |
| prometheanfire | ya | 22:04 |
| prometheanfire | ianw: nope :( | 23:09 |
| prometheanfire | 2018-10-10 23:07:53.788 | mount: /tmp/dib_build.cUGXjxvs/mnt/usr/portage: special device /tmp/in_target.d/usr-portage does not exist. | 23:09 |
| prometheanfire | ianw: cache https://paste.pound-python.org/show/bwngHHqMkgCHoCldkjHZ/ | 23:10 |
| prometheanfire | ianw: finalize https://paste.pound-python.org/show/2NsnuIydxZZERnO8jaeL/ | 23:10 |
| prometheanfire | failed in the finaliz.d part, caching worked til then | 23:11 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!