Thursday, 2018-04-19

*** gouthamr has joined #zuul00:09
openstackgerritMerged openstack-infra/zuul-jobs master: Switch to http://security.debian.org/ for debian  https://review.openstack.org/56234900:32
tristanCcorvus: good idea, i'll remember to do the rebase step in an extra ps.00:53
*** pwhalen_ has joined #zuul01:22
*** pwhalen has quit IRC01:23
*** JasonCL has quit IRC01:31
*** pwhalen_ is now known as pwhalen01:33
*** pwhalen has quit IRC01:34
*** pwhalen has joined #zuul01:34
*** ssbarnea_ has quit IRC01:51
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: Add allowed-triggers and allowed-reporters tenant settings  https://review.openstack.org/55408202:10
*** pwhalen has quit IRC02:15
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Refactor NodeLauncher to be generic  https://review.openstack.org/53555502:23
*** pwhalen has joined #zuul02:43
*** pwhalen has joined #zuul02:43
*** swest has quit IRC02:46
*** swest has joined #zuul02:47
*** harlowja_ has quit IRC02:53
*** toabctl has quit IRC03:00
*** harlowja has joined #zuul03:22
*** harlowja has quit IRC04:52
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: Add allowed-triggers and allowed-reporters tenant settings  https://review.openstack.org/55408205:33
*** yolanda has joined #zuul05:52
*** hashar has joined #zuul05:58
*** toabctl has joined #zuul06:04
*** dims has quit IRC06:54
*** dims has joined #zuul06:56
*** dims has quit IRC07:01
*** dims has joined #zuul07:02
*** bhavik1 has joined #zuul07:05
*** bhavik1 has quit IRC07:07
*** dmellado has joined #zuul07:32
*** jpena|off is now known as jpena07:39
*** electrofelix has joined #zuul07:40
*** ssbarnea_ has joined #zuul08:33
*** rcarrillocruz has quit IRC08:39
*** hashar is now known as hasharAway09:42
*** pbrobinson has quit IRC09:54
*** pbrobinson has joined #zuul09:58
*** xinliang has quit IRC10:39
*** xinliang has joined #zuul10:51
*** jpena is now known as jpena|lunch10:59
*** mugsie has quit IRC11:42
*** mugsie has joined #zuul11:42
*** mugsie has quit IRC11:42
*** mugsie has joined #zuul11:42
*** jpena|lunch is now known as jpena12:02
*** rlandy has joined #zuul12:30
*** ssbarnea_ has quit IRC12:47
*** ssbarnea_ has joined #zuul13:01
*** ssbarnea_ has quit IRC13:03
*** ssbarnea_ has joined #zuul13:09
*** JasonCL has joined #zuul13:09
*** dkranz has joined #zuul13:22
*** swest has quit IRC13:29
Shrewstobiash: have you done any ansible 2.5 testing? if not, i'm going to start on that14:33
tobiashShrews: unfortunately my plan for this week failed and I had to do other things14:40
tobiashSo I haven't started an this yet14:41
openstackgerritDavid Shrewsbury proposed openstack-infra/zuul master: WIP: Upgrade to Ansible 2.5  https://review.openstack.org/56266815:06
*** ssbarnea_ has quit IRC15:09
ShrewsI don't guess anyone has written a script to compare differences in Ansible repo modules with the Zuul copies, have they?15:12
ShrewsYeah, didn't think so.  :(15:12
dmsimard|offShrews: hmm... that sort of reminds me of a tool I've seen in the past that scored forks based on amount of changes in a project.. let me find that15:13
dmsimard|offthe only thing that comes to mind is debtcollector but that's not what I'm looking for15:14
dmsimard|offI'll need to send an email to find it -- let me get back to you on that15:15
*** hasharAway is now known as hashar15:15
Shrewsdmsimard|off: no worries. i'll come up with something15:15
ShrewsI guess it's really just the command.py module I should be concerned about15:19
pabelangerwhere did we land on 2.4 support for zuul? Given that it is likely disruptive to jobs, we'd likely need to have some hands on deck for openstack. But should we start communicating we are looking to bump the ansible version soon?15:21
dmsimard|offThey released Ansible 2.5.1 today. We're late :)15:23
dmsimard|off2.3 is actually EOL15:23
Shrewspabelanger: I suggested a while back we just make the jump to 2.5 since that is released.15:23
Shrewspabelanger: but we need to make sure that works, which is what i'm doing now15:24
Shrewsthat would give us more time to come up with a multi-version solution15:24
pabelangerdmsimard|off: where did you see 2.3 was EOL?15:25
dmsimard|offpabelanger: they only support the two latest releases https://github.com/ansible/ansible/blob/devel/docs/docsite/rst/reference_appendices/release_and_maintenance.rst15:26
dmsimard|off(I linked to github because for some reason http://docs.ansible.com/ansible/latest/reference_appendices/release_and_maintenance.html is not up to date)15:26
corvusanyone else want to review the mqtt driver addition?15:27
pabelangerdmsimard|off: thanks15:27
corvus(it has 2x+2 -- i just didn't want it to slip by anyone interested)15:27
pabelangerShrews: my concern about 2.5, is it might be too new. I'd be curious to see if anybody in openstack community is actually using it yet (as a gauge)15:28
pabelangerbut agree, we need to do something15:28
dmsimard|offpabelanger: there is a 2.5.1 so I would like to think the issues from 2.5.0 (if there was any) have been ironed out15:29
Shrewspabelanger: newness doesn't scare me. backward compatibility does15:29
pabelanger+115:29
pabelangerI haven't had a good upgrade storey with ansible yet15:29
pabelangerstory*15:30
Shrewsbut you know if we were to announce to openstack we intend to upgrade 2.4, we're inevitably going to be asked "why not 2.5?"15:30
pabelangersure, I think we'd also see why 2.5, why not 2.4?15:31
Shrewsbecause 2.5 goes to 1115:32
openstackgerritMerged openstack-infra/zuul-website-media master: Run zuul-website jobs  https://review.openstack.org/56175015:32
*** ssbarnea_ has joined #zuul15:35
pabelangerOSA teams is currently testing ansible 2.5, few issues but seems to be around their connection plugins15:42
openstackgerritJames E. Blair proposed openstack-infra/zuul master: WIP: late bind pipelines  https://review.openstack.org/55361815:45
openstackgerritJames E. Blair proposed openstack-infra/zuul master: WIP: late bind pipelines  https://review.openstack.org/55361815:54
*** electrofelix has quit IRC16:08
*** hashar is now known as hasharAway16:17
dmsimard|offShrews: regarding our earlier discussion: https://softwarefactory-project.io/r/gitweb?p=debtor.git;a=blob;f=README.rst;h=cd43850a5f1721d86731ee968ad4130de82dc6c6;hb=HEAD16:28
dmsimard|off(it's way overkill for what we need but I said I would follow up)16:29
dmsimard|offIt's also made to track /project/ forks (not "module" forks)16:29
clarkbre ansible EOL what does that actually mean? was ansible properly supporting old branches? I don't recall any of my bugfixes ever getting backported for example16:38
clarkbI guess its more a matter of keeping up with what third party modules will support?16:38
*** ssbarnea_ has quit IRC16:53
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Late bind projects  https://review.openstack.org/55361817:04
*** gouthamr is now known as gouthamr|afk17:06
openstackgerritDavid Shrewsbury proposed openstack-infra/zuul master: WIP: Upgrade to Ansible 2.5  https://review.openstack.org/56266817:08
*** jpena is now known as jpena|off17:10
corvusthe late-bind projects change (formerly known as late-bind pipelines) passes all of the non-postgres tests locally (i haven't set up pg yet), so i think it's ready for review17:12
corvusit'll probably be at least an hour before the tests results come back though because of openstack backlog17:12
clarkbthat reminds me I should rereview the secrets change17:12
corvusi'm hoping (i need to regroup and see where we are now) that puts us pretty close to severing all of the links between config objects.  once that's done, we can start individually caching them and re-use them on dynamic reconfiguration.  then hopefully use much less ram.17:14
corvusfbo_: ^ that will conflict with the load-broken-config change17:15
pabelangerShrews: clarkb: shocking testing ansible 2.5.1 with existing playbooks seems to have worked17:18
clarkboh its an entire stack I'll start at bottom of stack17:22
Shrewscorvus: pabelanger: mordred: oh, 2.5 brings us this lovely thing: "Added a configuration file that a site administrator can use to specify modules to exclude from being used."17:25
pabelangeryah, I heard about that. But haven't looked at it yet17:26
Shrewsmaybe we don't have to have zuul carry around those disabled shim modules17:26
clarkbya dmsimard|off pointed it out.17:28
pabelangerShrews: +117:29
Shrewsi am pestering bcoca to point me to docs on it17:32
clarkbit won't help where we conditionally disable modules based on host or source etc17:33
clarkbbut for straight up disabled modules may be useful17:33
Shrewswe always disable the networking modules17:33
Shrewsso at least useful there17:33
Shrewsjimi|ansible: gundalow: is that new feature documented?17:33
Shrewscannot seem to find it in the current docs17:35
*** patriciadomin has quit IRC17:35
dmsimard|offShrews: I have it somewhere in my #ansible-devel logs when I asked about it -- I'll pick it up when I'm not on my phone :p17:42
jimi|ansiblei honestly thought the whitelist/blacklist stuff would have been based on what you all were doing, i'll have to check17:45
jimi|ansiblei'd be surprised if we didn't have docs for that17:45
Shrewsjimi|ansible: yeah, not jumping out at me anywhere17:50
*** ssbarnea_ has joined #zuul17:50
openstackgerritMerged openstack-infra/zuul master: Test base job secrets  https://review.openstack.org/56103017:53
gundalowHi17:55
* gundalow reads 17:55
gundalowShrews: Docs for what?17:56
pabelangerShrews: well, all my tests on playbooks / roles passed using ansible 2.5.1. I'm going to start bumping things, but really good that nothing broke or needed to update playbooks17:56
gundalowpabelanger: \o/17:56
pabelangergundalow: indeed!17:56
Shrewsgundalow: for the new 2.5 config file that "a site administrator can use to specify modules to exclude from being used"17:56
*** gouthamr|afk is now known as gouthamr17:57
Shrewspabelanger: what playbooks are you testing?17:58
pabelangerShrews: roles / playbooks for windmill17:58
Shrewsoh17:58
pabelangerbut still using 2.3 to drive the 2.5.1 ansible-playbook17:58
pabelangerand talking to OSA, they don't seem to have many changes that are needed either17:59
pabelangerhoping that means when we do release 2.5.1 for zuul, minimal changes needed to our jobs17:59
Shrewspabelanger: the porting guides for 2.4 and 2.5 might be useful (http://docs.ansible.com/ansible/latest/porting_guides/porting_guides.html)18:00
Shrewsi didn't see anything when i skimmed 2.5 porting guide. haven't looked at 2.4 one yet18:00
Shrewswell, i guess those docs do not exist and I'll have to go skimming code. i can do that after my walk. biab18:02
openstackgerritMerged openstack-infra/zuul master: Make gearman calls async in ZuulWeb  https://review.openstack.org/56002618:13
*** pabelanger has quit IRC18:17
*** pabelanger has joined #zuul18:17
jlkfolks, we picked bubblewrap for executer, partly BECAUSE it can be used within a container, right?18:29
clarkbjlk: iirc it was picked because it aims to do proper isolation whereas other runtimes do not? But it can be run in a container too (tobiash does this iirc)18:30
*** ssbarnea_ has quit IRC18:30
pabelangerit also could be run as non-root users18:32
pabelangerwhich right kernel18:32
pabelangerwith*18:32
clarkbya my local install is no setuid and I can run it as my normal user18:36
pabelangerI actually wonder how hard it would be to wrap DIB with bwap, since people often complain about the root permissions it needs18:38
clarkbI don't think bwrap would address those (mount in particular)18:39
pabelangeryah, i think more of the extra isolation if DIB breaks out of a chroot, eg: recent iptables issue on nodepool-builder18:40
clarkbsame issue18:40
pabelangeryah?18:40
clarkbpretty sure if running brwap as non root user you don't get access to things that would require root like allowing mounts or getting new network namespace18:41
pabelangeroh, maybe. I thought there was a way to allow that via cgroups18:41
clarkbbut ya isolating networking would be nice to prevent hosing the builder18:47
pabelanger+118:48
pabelangerI mean, I view them as disposable personally, aside from the time it takes to rebuilt / update dns18:49
clarkbmount is likely to be the trickiest bit18:49
clarkbunless you coordinated the mounting outside of bwrap and bind mounted the results into where the elements are running18:50
clarkbbut ya I use disposable vm for dib locally18:50
clarkbpabelanger: from bwrap readme "Bubblewrap could be viewed as setuid implementation of a subset of user namespaces. Emphasis on subset - specifically relevant to the above CVE, bubblewrap does not allow control over iptables."18:55
*** hasharAway is now known as hashar18:57
openstackgerritPaul Belanger proposed openstack-infra/zuul master: Increase unit testing of host / group vars  https://review.openstack.org/55940519:00
openstackgerritPaul Belanger proposed openstack-infra/zuul master: Inventory groups should be under children key  https://review.openstack.org/55940619:00
pabelangerclarkb: so, I parse that as it might of helped with recent iptables issue, if we figured out mounts19:02
clarkbI guess that would be fine in the dib case since the build doesn't want ot update iptables at runtime but when image actually boots19:02
clarkbpabelanger: ya it would've failed the build I think19:02
clarkb(and prevented it from derping the host iptables)19:02
pabelangeryah19:02
pabelangera friday afternoon experiment19:02
openstackgerritDavid Shrewsbury proposed openstack-infra/zuul master: WIP: Update to Ansible 2.5  https://review.openstack.org/56266819:06
openstackgerritDavid Shrewsbury proposed openstack-infra/zuul master: WIP: Update to Ansible 2.5  https://review.openstack.org/56266819:09
tobiashclarkb, pabelanger: yocto (builds custom distros from source) can generate diskimages without root using fakeroot19:11
tobiashmaybe that also would be possible for diskimage-builder19:11
clarkbya there are tools for making docker images now without root too19:11
clarkbI'm sure its possible its just a matter of figuring out what the constraints are19:11
tobiashbut I guess that would be more than a friday afternoon experiment19:11
pabelangeryah, I'm not sure why we don't use fakeroot for DIB. I'm assuming there is a reason19:29
clarkbprobably because you still need root to mount19:34
clarkbso it was seen as unneeded overhead? spamaps may know the history there19:34
clarkbI think if dib itself needs root for mount but the elements don't that would be straightfoward enough19:35
clarkbthen you can isolate build host from the building of iamges and then shuffle bits as necessary19:35
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Late bind projects  https://review.openstack.org/55361819:38
*** harlowja has joined #zuul20:03
pabelangerah, new build dependency for zuul, if not using wheels: re2/re2.h: No such file or directory20:06
corvuspabelanger: oh we forgot to add that to bindep20:08
corvuspabelanger: libre2-dev20:08
pabelangerI believe bindep is right, roles haven't been updated to use bindep.txt yet20:09
clarkbtobiash: because i totally got nerd sniped, I think the way yocoto does it is they don't support partitioning. They write a filesytem to a file and then produce an image with that (so its single fs)20:09
clarkbhttp://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/classes/image_types.bbclass magic seems to happen there20:09
openstackgerritJames E. Blair proposed openstack-infra/zuul master: WIP: Make config objects freezable  https://review.openstack.org/56281620:11
corvusclarkb, tobiash: ^ can you take a quick look at that and tell me if you like that approach?20:11
clarkbcorvus: sure, I should get back to code review now that i have thoroughly been nerd sniped on the how to avoid root/mount problem for image building20:12
corvusclarkb, tobiash: i spotted an error in my last change; that's a response to that.  see my comment on https://review.openstack.org/553618 for more context20:12
clarkbI only see a -1 without a comment on 55361820:13
corvus(that change should actually fail tests because of the error in 562816 -- i mean, it's not an *error* yet, it would only be an error in the future once we start re-using config objects across layouts)20:13
clarkb?20:13
corvusdrat20:13
corvusclarkb: fixed.  was on prev patchset.20:13
clarkbcorvus: off the top of my head iirc subclassing Dict and List has issues but i'd hvae to go reread on what those issues are20:16
corvusclarkb: yeah, my understanding is that this particular usage should be okay, but i can change it if that's wrong20:18
corvus(because, basically, we're just adding an extra check to the underlying methods)20:19
clarkbit has to do with __setitem__ and __getitem__ being implemented in C separate from the python implementation20:19
clarkbor at least getitem is20:19
corvusclarkb: right, so in this case, we've got a python __setitem__ method which performs an extra check, then calls the C __setitem__.  and we're not changing setitem.20:20
corvuser, not changing getitem20:20
clarkbcorvus: I think you may need to handle dict.update() too20:25
corvusclarkb: good call20:25
clarkbbut reading on the issues doing this assuming __setitem__ is used for all updates would be the issue so explicit handling of udpate too should cover the bases?20:25
clarkb__getitem__ and get() being similar this way20:26
pabelangeris anybody else seeing a failure in test_job_list_in_project_template_not_dict_error ?20:26
clarkbcorvus: https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping appears to be the recommened way to do this (you implement the methods for that abc)20:27
pabelangerhttp://logs.openstack.org/05/559405/2/check/tox-py35/728f6a9/testr_results.html.gz20:28
corvusclarkb: i'd like to keep the performance of list/dict as much as possible; if i do that, i'm adding in another layer, right?20:29
clarkbcorvus: ya you'd likely have a dict as a class attribute rather than subclessing dict20:29
corvusit's not like the python docs say "dont't do this": https://docs.python.org/3.5/library/collections.html?highlight=userdict#userdict-objects  "The need for this class has been partially supplanted by the ability to subclass directly from dict"20:30
corvusif we use the abc, we have to implement the getters too, but if we subclass, we get the straight C versions of those20:30
clarkbya20:31
clarkbcorvus: https://docs.python.org/3.5/library/types.html#types.MappingProxyType there is also that20:32
clarkbwhich i think may be implemented as the thing you don't want20:32
clarkb(but not sure of that)20:32
clarkbI don't see a corresponding list proxy20:32
corvusclarkb: interesting; yeah, that seems like it would incur a performance penalty on reads20:33
corvusit's pretty much the interface i want though :)20:33
corvusi'll look at the implementation real quick20:33
clarkboh the docs direct link to the source now thats nice20:34
openstackgerritJames E. Blair proposed openstack-infra/zuul master: WIP: Make config objects freezable  https://review.openstack.org/56281620:35
corvusbiab20:35
corvusback.  i don't understand mappingproxytype.20:38
clarkbhttps://github.com/python/cpython/blob/d019bc8319ea35e93bf4baa38098ff1b57cd3ee5/Objects/descrobject.c#L808 it appears to also be implemented in c20:42
corvusclarkb: yeah, i just found that20:42
corvusclarkb: well, the list version of this would be to just convert it to a tuple20:43
corvusso i could have my freeze method replace lists with tuples, and dicts with mappingproxies before freezing the object itself20:43
clarkbthinking out loud that may have memory benefits because tuples don't need to be able to grow like lists do?20:44
clarkbthe implementation of the mapping proxy doesn't seem to hae that benefit20:45
corvusmaybe?  i'm not sure about that.20:45
clarkbhttps://nedbatchelder.com/blog/201608/lists_vs_tuples.html says tuple are a little more space efficient. Probably not worth worrying too much about20:48
clarkb(which confirmed my suspicion that because lists overallocate and reallocate if you append enough there is a small amount of extra pointer memory floating around)20:48
clarkbI'm going to go and give this cold some rest, I'll try to be back later to catch up on ^^ if you learn anything else20:49
clarkbalso that type(type.__dict__) thing is magic20:49
corvusclarkb: ok.  i'm inclined to go with mappingproxy and tuple, since they're likely easier and more efficient.20:50
corvusi'll work that up in just a bit20:50
corvusclarkb: thanks, and feel better20:50
*** gundalow has quit IRC20:56
openstackgerritJames E. Blair proposed openstack-infra/zuul master: WIP: Make config objects freezable  https://review.openstack.org/56281621:03
corvusclarkb: that's a version with mappingproxy/tuple21:04
clarkbcorvus: left a comment on ^ but its a minor thing21:35
clarkbI like that version its requires a lot less magic21:36
corvusclarkb: yeah; i could drop configurationobject, but i was thinking that it might be nice to indicate *why* it's freezable... i think we might end up with some freezable things which aren't configurationobjects... but i dunno... we'll see how it shakes out.21:44
clarkbthere is a question about override-checkout and handling different branches in different repos on the zuul mailing list. Can the job specify an override-checkout for both repos then when that job runs on branch foo on repo A it does the right thing and when run on branch bar on repo B it does the right thing if both repos are set to override to foo and bar respectively?21:47
corvusclarkb: if i'm following correctly, i *think* https://zuul-ci.org/docs/zuul/user/config.html#attr-pragma.implied-branches may help.  what do you think?21:56
corvusclarkb: (but, also, i think the solution in the email would work too.  that's the explicit version, pragma.implied-branches is the implicit version)21:57
corvusi'll draft a reply and paste it here before i send it out21:58
clarkbah yup, the docs even seem to point out it is this specific case21:58
clarkbjust not "stable" but a mapping between the two none the less21:59
corvusyeah, we wrote this for, i think, openstack <-> ceph branches or something like that.21:59
corvuslike, queens uses emerald or somesuch22:00
*** hashar has quit IRC22:00
corvushttps://etherpad.openstack.org/p/M58BwQJ7l122:24
corvusclarkb: how's that look?22:24
clarkbcorvus: ya I think that is helpful for explaining how to use branches22:28
clarkbin the config I hsould say22:28
corvusi did not set out to answer the question 3 different ways, i just couldn't stop typing :)22:28
clarkbmight even point mriedem to it once its sent as it is realted to the nova stable job experience they had recently22:29
corvussent!22:30
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Late bind projects  https://review.openstack.org/55361822:37
openstackgerritJames E. Blair proposed openstack-infra/zuul master: WIP: Make config objects freezable  https://review.openstack.org/56281622:37
tobiashclarkb: with yocto you can create multiple partitions (using parted)23:36
tobiashe.g. see https://github.com/Freescale/meta-fsl-arm/blob/master/classes/image_types_fsl.bbclass23:36
clarkbtobiash: doesn't that require root?23:37
clarkbtobiash: to partition a device in particular23:37
tobiashNope that works against a local (sparse) file23:37
clarkbhuh ok so ${SDCARD} is a sparse file that gets an mbr/gpt partition table written to it and multiple filesystems? /me looks to see how mkfs targets specific parittions23:38
tobiashclarkb: mkfs probably works on its own files and get dd'ed finally into the partition23:40
clarkbya that would line up with my reading of the ext* mkfs stuff23:41
clarkbit oeprates on its own file for sure23:42
tobiashcorvus: I like the approach of freezing the reusable config options23:42
clarkband then that file can be converted to things like qcow2 in the same .bb23:42
clarkb(so I guess you'd layer in this sdacrd stuff as needed)23:42
tobiashyes23:42
clarkbhttps://github.com/Freescale/meta-fsl-arm/blob/master/classes/image_types_fsl.bbclass#L305 there it is23:43
tobiashby default yocto just generates filesystem images23:43
clarkbthat seems incredibly hacky but I guess it works23:43
clarkbso ya dib could construct a raw disk in this manner too. Then do conversions as a regular user23:44
clarkbI wonder what the performance comparison looks like23:46
tobiashI think both do a final cop,23:47
tobiashcopy23:47
clarkbya they do23:47
tobiashMaybe yocto does an extra one but not sure23:48
clarkbbut one goes through a loopback device (dib) and the other uses dd on file directly23:48
clarkb(in this case I wouldn't be surprised if the dd'ing was faster)23:48
clarkboh except right it writes it to one file (the fs file) that then gets written to the image file23:48
clarkbso that would be an extra copy23:49
tobiashThe dd could benefit more from caching23:49
clarkbits unfortunate all the mkfs's have different flags to do the fs copy but otherwise this is reasonably straightforward23:59
clarkbdoesn't looks like they support xfs?23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!