| anteaya | dstufft: thank you | 00:00 |
|---|---|---|
| anteaya | dstufft: sleep for you yet? | 00:00 |
| mordred | dstufft: yes- thank you! | 00:00 |
| jeblair | clarkb, mordred, fungi: as the current status is: wait for jjb | wait for images; i'm going to afk for a while | 00:00 |
| *** austin81 has left #openstack-infra | 00:00 | |
| mordred | jenkins.o.o is finished jjbing | 00:00 |
| notmorgan | anteaya: uhm sure give me a few moments | 00:01 |
| anteaya | notmorgan: thanks | 00:01 |
| *** sputnik13 has quit IRC | 00:01 | |
| Daviey | Does zuul still create & upload tarballs on tag events for stable/kilo? | 00:01 |
| mordred | Daviey: I expect it to | 00:02 |
| dougwig | fyi, getting broken links and spinnys from gerrit. | 00:02 |
| notmorgan | anteaya: fwiw i'm kindof unhappy to push that through check again for that | 00:02 |
| clarkb | dougwig: what links ar ebroken | 00:02 |
| notmorgan | because someone kicked it from gate and then left it | 00:02 |
| notmorgan | with how backlogged things are | 00:02 |
| *** bpokorny has quit IRC | 00:02 | |
| fungi | mordred: just confirming, you're either running jjb in an interactive root shell or with sudo -H right? otherwise you're not actually updating the jjb cache puppet is wanting updated | 00:02 |
| dougwig | clarkb: the usual server error popup, disappears on retry. | 00:02 |
| clarkb | ok so links ar eworking then? | 00:02 |
| fungi | dougwig: clarkb: midnight utc | 00:02 |
| fungi | database backup time | 00:02 |
| dstufft | anteaya: I fell asleep lat night yea :) on accident but such is life, cough medicine kicking my ass | 00:02 |
| mordred | fungi: nope. thank you. I will start again | 00:02 |
| clarkb | dougwig: I ask because we have had bugs in gerrits url generation in the past | 00:02 |
| dougwig | fungi: ahh. everything hanging atm. ok. | 00:02 |
| clarkb | but server backup time is expected to error on db locks | 00:03 |
| notmorgan | anteaya: ^ see comments. | 00:03 |
| * notmorgan sighs | 00:03 | |
| notmorgan | people need to not leave "removed from gate" comments in if they update the patch | 00:03 |
| dougwig | clarkb: no, normal app errors. poor choice of words. | 00:03 |
| clarkb | dougwig: kk just making sure | 00:03 |
| fungi | mordred: yeah, the jjb cache is relative to $HOME (sorry for not mentioning that before) | 00:03 |
| dougwig | and it's back. | 00:03 |
| anteaya | Daviey: the latest nova stable/kilo tarball is dated Jan 16, 2016: http://tarballs.openstack.org/nova/ bottom of the page | 00:03 |
| mordred | fungi: it's ok - I should have known that | 00:03 |
| mordred | fungi: sudo -H jenkins or sudo -H ? | 00:03 |
| anteaya | notmorgan: I understand, which is why I didn't review for it | 00:04 |
| dstufft | new virtualenv once https://github.com/pypa/virtualenv/pull/849 passes | 00:04 |
| fungi | mordred: root | 00:04 |
| fungi | mordred: i actually run... time sudo -H jenkins-jobs update --delete-old /etc/jenkins_jobs/config | 00:04 |
| notmorgan | i expec tit'll take another 2-3 days to pass check is all. | 00:04 |
| Daviey | anteaya: Oh interesting... I was looking for ceilometer's one... which isnt there | 00:04 |
| notmorgan | because of the rate of transient failures | 00:04 |
| anteaya | dstufft: I hope you get real sleep soon | 00:04 |
| *** sflanigan has joined #openstack-infra | 00:04 | |
| *** sflanigan has joined #openstack-infra | 00:04 | |
| anteaya | dstufft: and feel better | 00:05 |
| fungi | mordred: running under time helps me keep track of how long it's taking on different jenkins masters, which is sometimes useful | 00:05 |
| notmorgan | mtreinish: https://review.openstack.org/#/c/270474/ needs a re +2 because commit message was bad. | 00:05 |
| Daviey | anteaya: It's actually dated 21-Jan-2016 23:16 , which is what i'd expect.. | 00:05 |
| anteaya | Daviey: yay, works as expected! | 00:05 |
| fungi | mordred: jeblair's example also included the config, though from what i can tell that's the default config path anyway | 00:06 |
| anteaya | notmorgan: oh no, I understand | 00:06 |
| anteaya | notmorgan: forget I mentioned it | 00:06 |
| Daviey | anteaya: well.. i've not checked them all.. but not seeing it for ceilometer.. nor in zuul queue? | 00:06 |
| notmorgan | anteaya: too late | 00:06 |
| *** ashleighfarnham has quit IRC | 00:06 | |
| anteaya | notmorgan: sorry | 00:06 |
| anteaya | Daviey: I don't know :/ | 00:07 |
| *** bhunter71 has quit IRC | 00:07 | |
| *** rbrndt has quit IRC | 00:07 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Add helper script to delete Gerrit groups https://review.openstack.org/270537 | 00:07 |
| notmorgan | anteaya: not grumpy at you, grumpy at the whole situation. | 00:07 |
| anteaya | Daviey: so this file lists all the jobs: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml | 00:07 |
| anteaya | notmorgan: lots of grumpy to go around | 00:08 |
| anteaya | Daviey: and in the top there are rules some projects use for what branches run certain jobs | 00:08 |
| fungi | Daviey: you might want to track down the corresponding tarball job logs... it's entirely likely quite a few projects' sdist tarball building is broken with pip 8.0.0 | 00:09 |
| anteaya | Daviey: so any project that runs the python-jobs template for instance: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n292 | 00:09 |
| anteaya | Daviey: runs the tarball job | 00:09 |
| *** sdake has quit IRC | 00:09 | |
| *** bhunter71 has joined #openstack-infra | 00:09 | |
| mordred | notmorgan: you know what fixes the grumpy? | 00:10 |
| *** ayoung has quit IRC | 00:10 | |
| Daviey | anteaya: right.. i get how that bit works. | 00:10 |
| anteaya | Daviey: oh good | 00:10 |
| anteaya | Daviey: well like fungi said the job may be there but broken due to pip 8 | 00:10 |
| *** sdake has joined #openstack-infra | 00:10 | |
| Daviey | fml :) | 00:11 |
| anteaya | or the individual project may have changed their regex for what branches that job runs on, but I doubt it | 00:11 |
| *** markvoelker has quit IRC | 00:11 | |
| anteaya | I'm not a fan of swearing | 00:11 |
| *** ddieterly has quit IRC | 00:11 | |
| anteaya | but I do appreciate the difficulty of your task | 00:11 |
| anteaya | and am grateful you are willing to do it | 00:11 |
| anteaya | thank you | 00:11 |
| anteaya | I don't know an easy way of checking the status of the stable/kilo tarball jobs | 00:12 |
| anteaya | other than checking logs and that file | 00:12 |
| anteaya | sorry | 00:12 |
| *** darrenc_afk is now known as darrenc | 00:12 | |
| notmorgan | anteaya: no idea | 00:12 |
| fungi | Daviey: you're looking for why latest landed ceilometer commit on stable/kilo has no corresponding tarball? | 00:13 |
| fungi | Daviey: or a release tarball for a tag? | 00:13 |
| fungi | i'm guessing the latter. seems you tagged 2015.1.3 an hour or so ago | 00:14 |
| Daviey | fungi: https://jenkins06.openstack.org/job/ceilometer-docs/101/console | 00:14 |
| Daviey | fungi: release tarball for the tag | 00:14 |
| fungi | `git show-ref 2015.1.3` says 02c49c1530aab42044ecddcaf6d5502eff44c961 was the tag object | 00:15 |
| fungi | making http://logs.openstack.org/02/02c49c1530aab42044ecddcaf6d5502eff44c961/ the base url for the logs | 00:15 |
| Daviey | fungi: i linked the jenkins failure ^^ | 00:15 |
| *** gildub has joined #openstack-infra | 00:16 | |
| fungi | Daviey: yeah, http://logs.openstack.org/02/02c49c1530aab42044ecddcaf6d5502eff44c961/release/ceilometer-tarball/d71d4cd/console.html#_2016-01-21_23_08_28_770 looks wrong | 00:16 |
| fungi | creating ceilometer-6.0.0.0b3.dev1 | 00:16 |
| Daviey | hmm | 00:17 |
| *** markvoelker has joined #openstack-infra | 00:18 | |
| fungi | sdague: ^ i suspect the zuul-cloner change has broken all our release jobs | 00:18 |
| fungi | we need to revert | 00:18 |
| fungi | i see z-c being called in there. it's not a change pipeline job | 00:18 |
| *** notnownikki has quit IRC | 00:19 | |
| fungi | as we discussed with dhellmann (yesterday was it?) zuul-cloner is probably only appropriate to change pipeline jobs at the moment | 00:19 |
| clarkb | fungi: really? arg we had existing jobs that used it | 00:19 |
| clarkb | so I figured it was working | 00:19 |
| clarkb | but I guess those like dhellmann's weren't | 00:19 |
| *** ZZelle_ has quit IRC | 00:20 | |
| fungi | INFO:zuul.Cloner:Prepared openstack/ceilometer repo with branch master at commit 1dc9d0d4bddff20e28d2aab39d6df693c83629ae | 00:20 |
| clarkb | so we want it on the one tarball job | 00:20 |
| clarkb | or maybe there isn't one at all in check/ gate | 00:20 |
| clarkb | I'm thinking docs | 00:20 |
| clarkb | there is gate docs and non gate docs | 00:20 |
| *** angdraug has joined #openstack-infra | 00:21 | |
| fungi | INFO:zuul.Cloner:upstream repo is missing branch refs/tags/2015.1.3 | 00:21 |
| fungi | there's what i was looking for | 00:21 |
| fungi | so it fell back to the master branch tip | 00:21 |
| clarkb | and maybe we should move the publish jobs out of the python jobs? | 00:21 |
| Daviey | clarkb: it did fail on the doc job | 00:22 |
| fungi | yeah, we do branch-tarball in post and tarball in pre-release and release | 00:22 |
| fungi | none of those can effectively use zuul-cloner | 00:23 |
| jeblair | i think we should revert the change, and someone should finish zuul-cloner, then we can try again | 00:23 |
| Daviey | Once this is reverted, are one of you able to rekick the job? | 00:24 |
| fungi | which means mordred's jjb updates are for nought | 00:24 |
| jeblair | and i think finishing zuul-cloner would be a great thing for someone who wants to get more involved to do. | 00:25 |
| fungi | Daviey: yeah, do you have a list of all the projects and tag names you did in the past 8.5 hours (just to be on the safe side, as that's when the change merged)? | 00:25 |
| Daviey | fungi: yep.. one moment | 00:25 |
| jeblair | it should be a fairly straightforward python programming task | 00:25 |
| jeblair | and self-contained | 00:26 |
| fungi | yeah, zuul-cloner is fairly well isolated from the rest of the zuul codebase | 00:26 |
| fungi | i agree it's a nice somewhat-low-hanging-fruit introductory project | 00:26 |
| *** zhurong has quit IRC | 00:26 | |
| fungi | most of the learning curve is in understanding zuul so you know what you're interfacing with | 00:26 |
| fungi | from a "what do these parameters mean?" standpoint | 00:27 |
| jeblair | tbh though, i don't know why that change needed to touch the tarball jobs | 00:28 |
| *** gyee has quit IRC | 00:28 | |
| Daviey | fungi: https://gist.github.com/a3f69dc5fce3ad086c5e | 00:28 |
| jeblair | i guess because sdague plans to _only_ test things with constraints | 00:28 |
| clarkb | jeblair: yes that | 00:28 |
| jeblair | so that means that anything that doesn't use constraints will break? | 00:29 |
| clarkb | jeblair: no | 00:29 |
| *** bhunter71 has quit IRC | 00:29 | |
| clarkb | it would still be opt in, but the idea was to allow projects to opt in | 00:29 |
| jeblair | clarkb: well, bitrot? | 00:29 |
| clarkb | they can't do that if the job isn't configured properly | 00:29 |
| jeblair | clarkb: i mean, if we only test nova with constraints, then if we try to build a tarball of it without constraints, someday it will likely break, yeah? | 00:29 |
| clarkb | jeblair: the idea is that it shouldn't, but it is possible that it can. Sort of like how we don't test our lower version boundaries | 00:30 |
| *** gyee has joined #openstack-infra | 00:30 | |
| anteaya | is someone offering the revert? | 00:30 |
| mordred | fungi: shouldI kill them? | 00:30 |
| jeblair | clarkb: right, however, the reason we expect the tarball jobs to work is because we're constantly building the sdist in unit tests | 00:30 |
| clarkb | jeblair: the issue is they break frequently without contraints | 00:30 |
| clarkb | jeblair: we actually almost never build the sdists in unit testes | 00:30 |
| clarkb | jeblair: which has led to several bugs in neutron | 00:31 |
| fungi | mordred: probably, since we're talking about reverting the change you approved now due to collateral damage, which means restarting the jjb updates again once it merges | 00:31 |
| mordred | fungi: kk | 00:31 |
| jeblair | that's like the entire reason we use tox | 00:31 |
| clarkb | jeblair: except that every project configures tox to do a setup develop | 00:31 |
| jeblair | i feel like we had this conversation recently | 00:31 |
| mordred | yeah. actually, we added the setup_develop feature to tox | 00:31 |
| clarkb | which does not make a tarball and does not catch issues like files missing from manifest.in and symlinks | 00:31 |
| jeblair | didn't we switch pep8 or something to do that? | 00:32 |
| anteaya | Daviey: thanks for mentioning the issue | 00:32 |
| anteaya | Daviey: sorry I mis-understood your question earlier | 00:32 |
| clarkb | jeblair: we added a step to the jobs yes but not the tox side | 00:32 |
| mordred | I believe we _did_ add _something_ that did a tarball build | 00:32 |
| clarkb | for pep8 | 00:32 |
| jeblair | okay, so we build a tarball for pep8 great | 00:32 |
| mordred | it's in the pep8 job | 00:32 |
| *** gildub has quit IRC | 00:32 | |
| mordred | not in tox | 00:32 |
| jeblair | will pep8 use constraints? | 00:32 |
| Daviey | anteaya: thanks for trying. :) | 00:32 |
| jeblair | to build that tarball? | 00:32 |
| mordred | just in the pe8 run_pep8.sh | 00:32 |
| anteaya | Daviey: I do the best I can :) | 00:33 |
| clarkb | jeblair: the tox side can | 00:33 |
| NobodyCam | I have a question | 00:33 |
| clarkb | jeblair: the build a tarball does not currently iirc | 00:33 |
| anteaya | NobodyCam: just ask | 00:33 |
| NobodyCam | lol | 00:33 |
| mordred | virtualenv sdist_check && sdist_check/bin/pip install pbr && sdist_check/bin/python setup.py sdist | 00:33 |
| jeblair | clarkb: okay, then we're back to: why does the tarball job need to use constraints? | 00:33 |
| clarkb | jeblair: because we break often without constraints | 00:33 |
| NobodyCam | can someone check my sanity: https://review.openstack.org/#/c/270010 | 00:33 |
| clarkb | jeblair: and we need the output of the tarball jobs to exist | 00:33 |
| jeblair | clarkb: i think we just established that the pep8 job will make sure that we are able to build a tarball without constraints | 00:34 |
| anteaya | NobodyCam: what should I be looking at? | 00:34 |
| clarkb | the whole idea behind constraints is to protect ourselves from broken deps | 00:34 |
| *** piet has quit IRC | 00:34 | |
| clarkb | jeblair: at the time that job runs | 00:34 |
| NobodyCam | I was looking at that patch and my bluetooth mouse went nuts and now I so as a commiter??? | 00:34 |
| clarkb | jeblair: but we should be using constraints there to protect the pep8 jobs too | 00:34 |
| jeblair | okay. then i stick with my original suggestion. revert; someone fixes z-c; retry. | 00:35 |
| NobodyCam | I did not want to submit anything | 00:35 |
| anteaya | NobodyCam: greghaynes is the commiter on that patch you linked | 00:35 |
| jeblair | clarkb, mordred, fungi: anyone second that? | 00:35 |
| anteaya | NobodyCam: I don't see you even as a reviewer on that patch | 00:35 |
| mordred | jeblair: second | 00:35 |
| *** Swami has quit IRC | 00:35 | |
| jeblair | and to anteaya's question -- anyone want to propose the revert? | 00:36 |
| *** otter768 has joined #openstack-infra | 00:36 | |
| fungi | jeblair: yes, i'm seconding by proposing the revert now | 00:36 |
| fungi | just finishing up the commit message | 00:36 |
| mordred | I need to step away from the keyboard for a few minutes - I'm going to go ahead and stop the jjb runs now | 00:36 |
| jeblair | fungi: cool | 00:36 |
| dkehn | can the same setup.cfg be used to build and install a distribution where the target commond goes to /usr/bin instead of /usr/local/bin? | 00:36 |
| mordred | I will re-run them again when I get back to the keyboard | 00:36 |
| mordred | unless someone tells me not to | 00:36 |
| NobodyCam | anteaya: I see this: http://paste.openstack.org/show/HRuPRcOl4fIrzvZDx1hX/ | 00:36 |
| anteaya | NobodyCam: I don't see you as submitting anything, comment or patchset | 00:36 |
| NobodyCam | ya I was just looking | 00:37 |
| clarkb | dkehn: that is controlled by setuptools I think | 00:37 |
| clarkb | dkehn: so depends on your python install and stuff | 00:37 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Revert "always make upper-constraints available to python jobs" https://review.openstack.org/271102 | 00:37 |
| NobodyCam | and I now show the patch as deleteing the whole file | 00:37 |
| fungi | jeblair: mordred: clarkb: sdague: Daviey: ^ | 00:37 |
| clarkb | dkehn: eg virtualenvs vs ubuntu system python vs fedora system python | 00:37 |
| *** sridhar_ram1 has joined #openstack-infra | 00:37 | |
| lifeless | dkehn: 'yes' is the answer | 00:37 |
| anteaya | NobodyCam: I see this: http://paste.openstack.org/show/484632/ | 00:37 |
| lifeless | dkehn: thats an abstract thing, its up to how you invoke pip / setuptools | 00:37 |
| *** gildub has joined #openstack-infra | 00:38 | |
| *** sridhar_ram has quit IRC | 00:38 | |
| NobodyCam | anteaya: thank you for taking the time to look.. I am just going to close teh window and act like nothing ever happeded | 00:38 |
| anteaya | NobodyCam: I think you need to close your bluetooth connection and enjoy your evening | 00:38 |
| anteaya | NobodyCam: check from a laptop when you can | 00:39 |
| NobodyCam | :) | 00:39 |
| dkehn | lifeless: and where the packages are located in the /usr/lib/pythoin2.7 instead of /usr/local/lib/python2.7, is it simple a sdist verses develop | 00:39 |
| openstackgerrit | Merged openstack-infra/release-tools: Optimize announce subject line https://review.openstack.org/270923 | 00:39 |
| anteaya | NobodyCam: thanks for asking :) | 00:39 |
| anteaya | NobodyCam: your bluetooth is making stuff up | 00:39 |
| NobodyCam | have a good night :) | 00:39 |
| anteaya | NobodyCam: thanks you too | 00:39 |
| NobodyCam | lol +++ | 00:39 |
| lifeless | dkehn: not a factor at all | 00:39 |
| openstackgerrit | Merged openstack-infra/release-tools: ignore deleted files when listing deliverable changes https://review.openstack.org/270910 | 00:39 |
| dkehn | lifeless: ok | 00:39 |
| lifeless | dkehn: what are you trying to achieve/debug/analyse? | 00:39 |
| openstackgerrit | Merged openstack-infra/release-tools: make announce.sh more robust for first releases https://review.openstack.org/271011 | 00:40 |
| dkehn | lifeless: someone here is pushing two version of the setup.cfg 1) for development and 2) for production, seems weird | 00:40 |
| *** jaypipes has quit IRC | 00:41 | |
| *** otter768 has quit IRC | 00:41 | |
| fungi | jhesketh: how familiar are you with zuul enqueue-ref? | 00:41 |
| fungi | ;) | 00:41 |
| lifeless | dkehn: its certainly unusual | 00:41 |
| dkehn | lifeless: was think what you had said and I'm resisting the additional setup.cfg | 00:41 |
| lifeless | dkehn: find out what problem they are trying to solve | 00:42 |
| dkehn | lifeless: yep, thx | 00:42 |
| lifeless | dkehn: then you can cross reference the setuptools docs, and/or pip docs | 00:42 |
| lifeless | dkehn: but generally, putting local in the path is not something that python packages have any control over - and if they're influencing that via setup.cfg and pbr --- there's no guarantee that we'd keep supporting it: its really an abstraction layer violation | 00:43 |
| *** salv-orlando has quit IRC | 00:43 | |
| dkehn | lifeless: gotit | 00:44 |
| *** Sukhdev has quit IRC | 00:46 | |
| *** kevinbenton has quit IRC | 00:46 | |
| *** armax has quit IRC | 00:48 | |
| jhesketh | fungi: I haven't used it before sorry | 00:48 |
| jhesketh | I assume it's reasonably straight forward | 00:49 |
| jhesketh | So long as it's the post pipeline jobs | 00:49 |
| fungi | release in this case | 00:49 |
| jhesketh | Yep, I more meant jobs that usually run off refs | 00:49 |
| mordred | back | 00:49 |
| *** Daisy_ has joined #openstack-infra | 00:49 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: WIP: Add initial optional influxdb support for test_runs https://review.openstack.org/271088 | 00:50 |
| jhesketh | (it's likely others would work too) | 00:50 |
| *** kevinbenton has joined #openstack-infra | 00:50 | |
| fungi | but yeah, once my revert of the constraints-everywhere change merges and mordred kicks his jjb updates back off again, after they complete, we have some release tags to reenqueue for Daviey | 00:50 |
| *** kzaitsev_mb has joined #openstack-infra | 00:50 | |
| Daviey | thanks fungi | 00:50 |
| fungi | i'm double-checking now to trim down the list as much as we can safely | 00:50 |
| jhesketh | Yep | 00:50 |
| fungi | jhesketh: the reason i ask is that it's likely i'd be asleep once those conditions are met | 00:50 |
| jhesketh | Sure, happy to do that | 00:51 |
| lifeless | what did constraints everywhere break? | 00:51 |
| dstufft | Ok, 14.0.1 is out now | 00:51 |
| jhesketh | I assume it's just queuing the tags | 00:51 |
| mordred | lifeless: a sequence of things | 00:52 |
| mordred | lifeless: first, I landed the patch without thinking - it updates _all_of the jenkins jobs, and that's a thing that dies horribly due to time it takes | 00:53 |
| mordred | lifeless: second, there was a mismatch it created in how zuul-cloner, it and making tarballs works | 00:53 |
| mordred | lifeless: so we're reverting it so that we can regroup and solve that problem | 00:55 |
| mordred | lifeless: it's not a hard or deep problem - but it is more than can be trivially fixed under duress | 00:55 |
| fungi | Daviey: jhesketh: good news! the only tarballs i don't see are ceilometer, cinder and neutron-vpnaas. the rest listed at https://gist.github.com/Daviey/a3f69dc5fce3ad086c5e seem to exist | 00:55 |
| lifeless | mordred: thanks | 00:55 |
| mordred | lifeless: the ^^ tarballs missing there are the symptom | 00:56 |
| Daviey | fungi: thanks for that! | 00:56 |
| *** kzaitsev_mb has quit IRC | 00:57 | |
| fungi | jhesketh: note you'll want to ignore the git shas in that list as they're the objects being tagged, not the tag objects themselves. as usual use `git show-ref 2015.1.3` to get the tag object | 00:57 |
| jhesketh | fungi: cool | 00:58 |
| jhesketh | Thanks | 00:58 |
| fungi | jhesketh: an example invocation... sudo zuul enqueue-ref --trigger gerrit --pipeline release --project openstack/ceilometer --ref refs/tags/2015.1.3 --newrev 02c49c1530aab42044ecddcaf6d5502eff44c961 | 00:58 |
| *** eil397 has quit IRC | 00:58 | |
| *** jlvillal has quit IRC | 00:58 | |
| *** ccrouch has quit IRC | 00:59 | |
| fungi | jhesketh: if for whatever reason things don't get to the point over the course of your day where it's safe to reenqueue those yet, let me know and i'll pick it back up myself when i wake up | 00:59 |
| *** ybathia has quit IRC | 01:02 | |
| *** jlvillal has joined #openstack-infra | 01:02 | |
| fungi | lifeless: so slightly more detailed summary is that zuul-cloner, while safe for use in zuul change pipelines (like check, gate, experimental, silent), is not yet feature complete for use in zuul's arbitrary git ref pipelines (post, release, pre-release, tag) | 01:03 |
| *** SumitNaiksatam has quit IRC | 01:03 | |
| jhesketh | fungi: will do | 01:04 |
| jesusaurus | fungi: what features are still needed in zuul-cloner for that? | 01:04 |
| *** salv-orlando has joined #openstack-infra | 01:04 | |
| *** dingyichen has quit IRC | 01:04 | |
| openstackgerrit | Merged openstack/diskimage-builder: Force dib-python symlink creation https://review.openstack.org/270994 | 01:04 |
| *** doug-fish has quit IRC | 01:04 | |
| fungi | jesusaurus: awesome of you to ask! | 01:05 |
| *** doug-fish has joined #openstack-infra | 01:05 | |
| *** doug-fish has quit IRC | 01:06 | |
| *** doug-fish has joined #openstack-infra | 01:06 | |
| *** dingyichen has joined #openstack-infra | 01:07 | |
| *** pvaneck has quit IRC | 01:07 | |
| fungi | jesusaurus: short answer is that we need someone to look at the jenkins parameters (seen by z-c as environment variables) set in zuul ref-updated triggered pipeline jobs and see what behaviors z-c needs adapted to deal with that reduced set (compared to patchset-created triggers) | 01:08 |
| fungi | and obviously then propose patches to implement and test that | 01:08 |
| *** markvoelker has quit IRC | 01:09 | |
| *** salv-orlando has quit IRC | 01:09 | |
| *** bpokorny_ has quit IRC | 01:10 | |
| *** sridhar_ram1 has quit IRC | 01:10 | |
| *** bpokorny has joined #openstack-infra | 01:10 | |
| openstackgerrit | Merged openstack-infra/project-config: Revert "always make upper-constraints available to python jobs" https://review.openstack.org/271102 | 01:11 |
| *** sridhar_ram has joined #openstack-infra | 01:11 | |
| jesusaurus | fungi: seems straight-forward, i'll see if i can find some time to work on that | 01:11 |
| *** mriedem has joined #openstack-infra | 01:11 | |
| fungi | jesusaurus: it would be greatly appreciated | 01:11 |
| Daviey | fungi / jhesketh: Here is the tag sha's.. sorry, that would have been more useful - 06/08/2015 | 01:11 |
| Daviey | https://gist.github.com/0a4474cf90e1cc288c6f ^ | 01:12 |
| fungi | Daviey: thanks! | 01:12 |
| anteaya | jesusaurus: I'd like to learn what you are doing by reviewing | 01:12 |
| jhesketh | mordred: the revert has merged.. did you want to kick of jjb? | 01:13 |
| anteaya | jesusaurus: please include be but don't wait for me either | 01:13 |
| mordred | jhesketh: I do - although I believe I need to get latest project-config | 01:13 |
| *** bpokorny has quit IRC | 01:13 | |
| jhesketh | thanks | 01:14 |
| *** bpokorny has joined #openstack-infra | 01:14 | |
| *** sdake has quit IRC | 01:17 | |
| jesusaurus | anteaya: ok | 01:17 |
| *** otter768 has joined #openstack-infra | 01:17 | |
| anteaya | jesusaurus: thanks | 01:18 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool: Add interface to dump dib command for an image https://review.openstack.org/271110 | 01:19 |
| ianw | asselin: ^ | 01:19 |
| * asselin looks | 01:19 | |
| *** Daisy_ has quit IRC | 01:20 | |
| *** Daisy_ has joined #openstack-infra | 01:21 | |
| *** kzaitsev_mb has joined #openstack-infra | 01:23 | |
| anteaya | jhesketh: you have what you need to juggle the broken for your awake hours? | 01:24 |
| jhesketh | I'll try :-) | 01:25 |
| jhesketh | just waiting for jjb to be kicked by mordred | 01:25 |
| clarkb | I should be more around now | 01:25 |
| clarkb | ping if you need more eyes and I may be awake :) | 01:25 |
| * mordred is doing this now | 01:25 | |
| jhesketh | clarkb: appreciate it | 01:25 |
| Sam-I-Am | anteaya: can i get some love on this? https://review.openstack.org/#/c/270444/ | 01:25 |
| anteaya | jhesketh: awesome, happy dancing | 01:25 |
| Sam-I-Am | unless you see a problem, of course :) | 01:26 |
| * anteaya hangs on to click on Sam-I-Am's patch | 01:26 | |
| *** Daisy_ has quit IRC | 01:26 | |
| jhesketh | it looks like the snapshot images are built so new nodes should be coming up on those just now | 01:26 |
| jhesketh | dib still building | 01:26 |
| *** sdake has joined #openstack-infra | 01:28 | |
| jhesketh | something weird looks to be happening with the dib images | 01:28 |
| jeblair | re snapshots -- it does look like most of them have updated successfully -- however, there are some weird errors that i don't understand (that somehow look related to dib)... still digging | 01:29 |
| *** Sukhdev has joined #openstack-infra | 01:29 | |
| jeblair | oh, so image-update will do both dib and snapshot images | 01:30 |
| *** dslev has joined #openstack-infra | 01:30 | |
| jhesketh | jeblair: so yes, it looks like dib is cycling | 01:30 |
| Sam-I-Am | anteaya: thanks@ | 01:30 |
| Sam-I-Am | er ! | 01:30 |
| jhesketh | ie it keeps rebuilding | 01:30 |
| jhesketh | presumably because it fails | 01:31 |
| Sam-I-Am | makes docs updates a lot easier for the project | 01:31 |
| jhesketh | but trying to find the appropriate log part | 01:31 |
| *** rfolco has joined #openstack-infra | 01:31 | |
| dstufft | did things unbreak? | 01:31 |
| jeblair | jhesketh: well, i triggered image-updates after i deleted the database entries, so i would expect at least 2 runs for some of these, but yeah, if we're seeing > 2 then we may have a prob | 01:31 |
| ianw | oh jeez, the f23 build is failing with | 01:31 |
| ianw | At least 269MB more space needed on the / filesystem. | 01:31 |
| ianw | that's a new one | 01:31 |
| Daviey | (i need to go afk. be back tomorrow.. thanks all) | 01:32 |
| *** dslev_ has joined #openstack-infra | 01:32 | |
| jeblair | ianw, jhesketh: ah opt is full | 01:32 |
| clarkb | ya it would restart if a build fails | 01:32 |
| clarkb | jeblair: may need to do that disk cleanup sooner then :) | 01:32 |
| jeblair | probably because we have lots of orphaned images | 01:32 |
| jeblair | clarkb: right :) | 01:32 |
| anteaya | Sam-I-Am: yes, welcome | 01:32 |
| jhesketh | ah good find | 01:32 |
| anteaya | g'night all | 01:32 |
| jeblair | clarkb, jhesketh: i'll clean those up now | 01:32 |
| jeblair | er | 01:32 |
| jeblair | we have lots of directories in /opt/nodepool_dib | 01:33 |
| fungi | dstufft: our unbreak cycle takes a while to unwind | 01:33 |
| jeblair | do they ever get deleted? | 01:33 |
| dstufft | fungi: okay | 01:33 |
| dstufft | let me know | 01:33 |
| clarkb | jeblair: those are a dib leak, apparently intentional? | 01:33 |
| jeblair | most are small... | 01:33 |
| clarkb | jeblair: but they are small and ya | 01:33 |
| fungi | dstufft: we likely won't know if things everywhere are unbroken for up to a few days since there are a few workarounds/pins/caps which need unwinding too | 01:33 |
| dstufft | I'm going to go back to working on warehouse.python.org in the mean time, pip is a recipe for getting angry atm :| | 01:33 |
| jeblair | clarkb, greghaynes: but why is that intentional? | 01:34 |
| clarkb | jeblair: I think for debugging? it wasn't clear but greghaynes thought we may be able to turn it off with a setting set | 01:34 |
| *** sdake has quit IRC | 01:34 | |
| clarkb | but that got lost in the gearman connection died debugging | 01:34 |
| jeblair | cool, that would be nice; we have 1392 entries in that dir | 01:34 |
| fungi | dstufft: it seems like such an overheating spacebar problem that i can't fathom how you have the energy to continue trying to wrestle it into shape | 01:35 |
| *** dslev has quit IRC | 01:35 | |
| openstackgerrit | Merged openstack-infra/project-config: Add logic for networking-ovn jobs https://review.openstack.org/270444 | 01:35 |
| dstufft | fungi: https://caremad.io/s/itZFsGMxDj/ | 01:36 |
| fungi | dstufft: i'm happy we have you pushing boundaries on it wherever it seems reasonable to do so, and am sorry it causes community backlash when you're just trying to make things more sane | 01:36 |
| jeblair | we seem to have files named like "ubuntu-trusty-1450620933.raw" and also "176179.qcow2" | 01:36 |
| jeblair | i guess we cahnged the naming system? | 01:37 |
| clarkb | jeblair: yes it was part of how the workers would know what they should register for | 01:37 |
| jeblair | i'm confused though since there are no entries like 176179.qcow2 in the table | 01:37 |
| clarkb | jeblair: did they get deleted too? | 01:38 |
| jeblair | clarkb: http://paste.openstack.org/show/484630/ | 01:38 |
| jeblair | clarkb: that was the old table before i deleted it | 01:38 |
| jeblair | fedora-22-1448633757.qcow2 exists | 01:38 |
| jeblair | but 173704* does not | 01:38 |
| clarkb | 176179 is there | 01:38 |
| jeblair | so i don't understand which is the current system | 01:38 |
| jeblair | ah yes it is | 01:38 |
| clarkb | current is just the number | 01:38 |
| jeblair | clarkb: ah yep that's starting to make sense... | 01:39 |
| jeblair | most of those are ancient | 01:39 |
| jeblair | er most of the named ones | 01:39 |
| jeblair | i'll delete all of the named ones and all of the ids < 176186 | 01:40 |
| dstufft | fungi: relatively speaking, pip 8 was pretty unenventful tbh, a thing broke, got a few reports, then folks mostly just sat tight while we sorted out. The things mostly making me angry on pip lately are trying to improve the OS <-> pip story and try to integrate with the OS better and the related drama around that :/ | 01:40 |
| clarkb | jeblair: ok | 01:40 |
| *** ayoung has joined #openstack-infra | 01:41 | |
| clarkb | dstufft: would it be useful if we ran a job once a day that grabbed tip of pip master and emailed you results? | 01:41 |
| *** Sukhdev has quit IRC | 01:41 | |
| clarkb | we could set something up like that pretty easily if it would be helpful (and we in theory cover ubuntu, centos and fedora) | 01:41 |
| dstufft | clarkb: pip's test suite? Probably not because the false positive rate in the test suite is super high right now, often takes 2-3 runs to get a pass :( that's a TODO list item for me too, but warehouse is getting the bulk of my focus until we can kill the legacy old code | 01:42 |
| jeblair | jhesketh: i cleaned up 50% of the space; still working but should be enough to unstick | 01:43 |
| jhesketh | yep, saw that | 01:43 |
| clarkb | dstufft: no, our test suite which does things like isntall all the things | 01:43 |
| jhesketh | I'm guessing it shoudl be safe to just let hte images do their thing | 01:43 |
| dstufft | clarkb: oh that, sure | 01:43 |
| clarkb | dstufft: well probably actually not tempest, but the installation part | 01:43 |
| clarkb | dstufft: just as a sanity check of can pip install things in the world | 01:43 |
| *** thorst has joined #openstack-infra | 01:43 | |
| clarkb | with a biased set of almost 400 deps | 01:43 |
| dstufft | at some point I want to get a real regression suite setup | 01:43 |
| ianw | clarkb: i think that would be quite useful even if dstufft isn't "in the loop" ... we certainly could have detected all this earlier and started conversations | 01:44 |
| dstufft | suit | 01:44 |
| dstufft | w/e | 01:44 |
| *** rfolco has quit IRC | 01:44 | |
| dstufft | you can (or should anyways) be able to run pip so that deprecations become failures instead of warnings | 01:44 |
| dstufft | or should be able to* | 01:44 |
| dstufft | not sure I've tested it recently | 01:44 |
| *** Daisy_ has joined #openstack-infra | 01:45 | |
| jeblair | clarkb, jhesketh: since i'm really good at deleting things today, i'm going to delete those .d dirs too | 01:46 |
| clarkb | jeblair: ++ | 01:46 |
| *** apoorvad has quit IRC | 01:46 | |
| jhesketh | jeblair: I was just poking around those to see if we needed them | 01:46 |
| jhesketh | sounds good to me | 01:46 |
| mordred | the jenkinses are all running, btw | 01:46 |
| jeblair | jhesketh: done | 01:47 |
| jeblair | now the directory is empty | 01:47 |
| jeblair | i guess i could have saved myself some time | 01:47 |
| jeblair | (i thought maybe a build was running but i guess not?) | 01:47 |
| *** sshnaidm has quit IRC | 01:48 | |
| jhesketh | jeblair: dib-image-list says it's building 5... I wonder if they'll fail | 01:48 |
| *** otter768 has quit IRC | 01:48 | |
| jeblair | hrm | 01:48 |
| jeblair | disk-image-create -x -t qcow2,vhd --no-tmpfs --qemu-img-options 'compat=0.10' -o /opt/nodepool_di | 01:48 |
| jeblair | b/176197 fedora-minimal vm simple-init openstack-repos nodepool-base node-devstack puppet | 01:48 |
| jeblair | that is not one of the directories i deleted | 01:49 |
| jeblair | i think it's working in dib_tmp right now | 01:50 |
| jeblair | oh wow | 01:50 |
| jeblair | which is another terrifying directory | 01:50 |
| *** sridhar_ram has quit IRC | 01:50 | |
| jeblair | with 56759 entries | 01:50 |
| clarkb | wow | 01:50 |
| jeblair | anyone know if profiledir.idNA7H is still needed? | 01:50 |
| jeblair | or profiledir.mPW2UX ? | 01:50 |
| clarkb | oh I remember greghaynes said there was a bug in dib where the exit handler wasn't being called | 01:50 |
| clarkb | which cleaned that stuff up | 01:51 |
| clarkb | and was recently fixed | 01:51 |
| *** sridhar_ram has joined #openstack-infra | 01:51 | |
| *** sridhar_ram has quit IRC | 01:51 | |
| clarkb | jeblair: I think there may be a profiledir for the current runs, not sure how catastrophic it is if we remove the wrong one | 01:51 |
| jeblair | clarkb: are they single-use dirs? | 01:51 |
| clarkb | I want to say thats whree the data that says eleement X took 5 seconds and Y took 9 seconds goes | 01:52 |
| jeblair | (i don't know where the magic caching happens with dib) | 01:52 |
| jeblair | oh interesting | 01:52 |
| clarkb | jeblair: there is a specific cache dir for dib separate of the tmp dir | 01:52 |
| clarkb | /opt/dib_cache | 01:52 |
| jeblair | i'm not going to look in there yet ;) | 01:52 |
| *** Daisy_ has quit IRC | 01:53 | |
| jeblair | my du on dib_tmp is still running :/ | 01:53 |
| mordred | wow | 01:53 |
| clarkb | jeblair: so if you do a find on type directory and mtime greater than say a day you can delete whatever is in the tmp dir | 01:53 |
| *** Daisy_ has joined #openstack-infra | 01:53 | |
| clarkb | then rerun that tomorrow and we should be pretty clean | 01:53 |
| jeblair | 31G | 01:53 |
| *** Daisy_ has quit IRC | 01:54 | |
| clarkb | the cache should be relatively clean, the only things we may find there are old things cached that don't need to be cached anymore like old cirros images | 01:54 |
| clarkb | http://git.openstack.org/cgit/openstack/diskimage-builder/commit/?id=966f12dc4cd6f48f57a6d170d7fbccc5a28892a9 | 01:54 |
| *** dslev_ has quit IRC | 01:55 | |
| clarkb | I want to say that may be part of the lack of cleanup | 01:55 |
| *** davideagnello has quit IRC | 01:56 | |
| jeblair | clarkb: well, there are things in the image dirs that have mtimes older than 1 day | 01:56 |
| jeblair | clarkb: so i think i'll take the approach of just examining the profiledirs | 01:56 |
| *** _amrith_ is now known as amrith | 01:56 | |
| jeblair | image dirs seem to be getting cleaned up, with only a few exceptions (probably some catastrophic crash) | 01:57 |
| *** angdraug has quit IRC | 01:58 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool: Add interface to dump dib command for an image https://review.openstack.org/271110 | 01:58 |
| *** ddieterly has joined #openstack-infra | 02:01 | |
| jeblair | drwx------ 3 nodepool nodepool 4096 Oct 21 19:48 yum-nodepool-5_cuwM | 02:01 |
| jeblair | i don't know what that is... | 02:01 |
| clarkb | is that in the cache? | 02:02 |
| *** rbrndt has joined #openstack-infra | 02:02 | |
| jeblair | no in dim_tmp | 02:02 |
| jeblair | dib_tmp | 02:02 |
| jeblair | it's the only directory that's not 'profile' or 'image' | 02:02 |
| jeblair | ah | 02:02 |
| jeblair | it's still being used | 02:02 |
| clarkb | if I had to guess it is related to make a centos from scratch with yum | 02:02 |
| clarkb | or fedora | 02:03 |
| jeblair | ianw: ^ do you know about that dir? | 02:03 |
| jeblair | ianw: should it be in dib_tmp as opposed to dib_cache? | 02:03 |
| ianw | jeblair: that does not look familiar | 02:03 |
| ianw | what's in it? | 02:03 |
| jeblair | ianw: http://paste.openstack.org/show/484634/ | 02:03 |
| jeblair | ianw: the f23 files at least were updated today | 02:04 |
| *** dims__ has joined #openstack-infra | 02:04 | |
| ianw | yeah, well that's yum, but i'm not how it's getting that directory... | 02:04 |
| jeblair | /dev/mapper/main-opt 504G 50G 455G 10% /opt | 02:04 |
| jeblair | 455G of cleanup | 02:05 |
| *** thorst has quit IRC | 02:05 | |
| *** dims_ has quit IRC | 02:06 | |
| *** mtanino has quit IRC | 02:07 | |
| *** kzaitsev_mb has quit IRC | 02:07 | |
| mordred | jeblair: nicely done | 02:11 |
| jeblair | mordred: i got to do quite a lot of it by accidentally deleting extra database rows! | 02:11 |
| *** jaosorior has quit IRC | 02:12 | |
| mordred | jeblair: woot! | 02:12 |
| *** rlandy has quit IRC | 02:12 | |
| mordred | jeblair: it's like when you save 50% on that $5000 purchase you weren't intending on making | 02:12 |
| *** mriedem has quit IRC | 02:12 | |
| *** jaosorior has joined #openstack-infra | 02:12 | |
| mordred | it's not like you spent $2500 - it's like you SAVED $2500 | 02:13 |
| jeblair | i totally got a deal on my new dishwasher! | 02:13 |
| *** piet has joined #openstack-infra | 02:13 | |
| jeblair | but yeah, i would rather have not have had to get the new dishwasher. | 02:14 |
| jeblair | i'm not sentencing well so i'm going to afk again. | 02:14 |
| mordred | jeblair: ++ | 02:15 |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/project-config: Adds check-requirements for qa-specs repo https://review.openstack.org/270745 | 02:15 |
| *** MarkAtwood has quit IRC | 02:16 | |
| *** yamamoto_ has joined #openstack-infra | 02:18 | |
| *** doug-fish has quit IRC | 02:20 | |
| *** doug-fish has joined #openstack-infra | 02:21 | |
| *** doug-fish has quit IRC | 02:21 | |
| *** doug-fish has joined #openstack-infra | 02:21 | |
| openstackgerrit | Andre keedy proposed openstack-infra/project-config: Add new project(shovel) to openstack https://review.openstack.org/269411 | 02:27 |
| *** Daisy_ has joined #openstack-infra | 02:27 | |
| *** rossella_s has quit IRC | 02:31 | |
| *** rossella_s has joined #openstack-infra | 02:31 | |
| *** rhallisey has quit IRC | 02:34 | |
| *** [1]Thelo has joined #openstack-infra | 02:34 | |
| *** Thelo has quit IRC | 02:36 | |
| *** [1]Thelo is now known as Thelo | 02:36 | |
| *** sflanigan is now known as sflanigan_afk | 02:37 | |
| *** yamamoto_ has quit IRC | 02:38 | |
| *** bpokorny has quit IRC | 02:39 | |
| jhesketh | mordred: how did jjb stuff go? | 02:39 |
| *** yamamoto_ has joined #openstack-infra | 02:39 | |
| *** Daisy_ has quit IRC | 02:40 | |
| *** Daisy_ has joined #openstack-infra | 02:41 | |
| *** baoli has joined #openstack-infra | 02:41 | |
| *** asalkeld has quit IRC | 02:42 | |
| *** wei-wang has joined #openstack-infra | 02:42 | |
| *** Daisy_ has quit IRC | 02:43 | |
| *** baoli has quit IRC | 02:44 | |
| *** Daisy_ has joined #openstack-infra | 02:44 | |
| *** baoli has joined #openstack-infra | 02:44 | |
| *** maishsk has joined #openstack-infra | 02:46 | |
| *** woodster_ has quit IRC | 02:46 | |
| *** baoli has quit IRC | 02:48 | |
| *** amrith is now known as _amrith_ | 02:48 | |
| *** Daisy_ has quit IRC | 02:48 | |
| *** sflanigan_afk is now known as sflanigan | 02:49 | |
| * fungi strongly suspects it's still running | 02:49 | |
| *** bhunter71 has joined #openstack-infra | 02:49 | |
| *** Daisy_ has joined #openstack-infra | 02:49 | |
| *** moravec has joined #openstack-infra | 02:49 | |
| *** baoli has joined #openstack-infra | 02:49 | |
| fungi | at least one of them will have probably aborted from a transient api error and need starting over | 02:49 |
| *** _amrith_ is now known as amrith | 02:49 | |
| *** doug-fish has quit IRC | 02:50 | |
| *** keedya has joined #openstack-infra | 02:50 | |
| *** Daisy_ has quit IRC | 02:52 | |
| *** Daisy_ has joined #openstack-infra | 02:53 | |
| *** nelsnelson has joined #openstack-infra | 02:54 | |
| *** yamamoto_ has quit IRC | 02:55 | |
| *** otter768 has joined #openstack-infra | 02:58 | |
| *** Daisy_ has quit IRC | 02:58 | |
| *** otter768 has quit IRC | 02:59 | |
| *** Daisy_ has joined #openstack-infra | 02:59 | |
| *** Qiming has joined #openstack-infra | 03:00 | |
| jhesketh | fungi: this is just deploying new job definitions to the masters right? | 03:02 |
| *** baoli has quit IRC | 03:03 | |
| *** bardia has quit IRC | 03:03 | |
| *** Daisy_ has quit IRC | 03:03 | |
| *** kzaitsev_mb has joined #openstack-infra | 03:03 | |
| *** thorst has joined #openstack-infra | 03:06 | |
| clarkb | yes | 03:06 |
| *** sdake has joined #openstack-infra | 03:06 | |
| *** rguillebert has quit IRC | 03:07 | |
| *** achanda has joined #openstack-infra | 03:08 | |
| *** baoli has joined #openstack-infra | 03:09 | |
| fungi | yep | 03:10 |
| fungi | 7k+ of them | 03:10 |
| *** baoli has quit IRC | 03:10 | |
| *** yamahata has quit IRC | 03:11 | |
| mordred | clarkb: on jenkins03 ... | 03:12 |
| *** sdake_ has joined #openstack-infra | 03:12 | |
| mordred | INFO:jenkins_jobs.builder:Not deleting unmanaged jenkins job clarkb-upgrade2-test | 03:12 |
| mordred | that happened after INFO:root:Number of jobs updated: 3645 | 03:12 |
| mordred | I also get this: http://paste.openstack.org/show/484635/ | 03:13 |
| mordred | on all of them | 03:13 |
| mordred | at the end of the run | 03:13 |
| mordred | but they say "cache saved | 03:13 |
| mordred | 1 and 2 are still running | 03:13 |
| *** jpr has joined #openstack-infra | 03:13 | |
| *** amrith is now known as _amrith_ | 03:14 | |
| *** thorst has quit IRC | 03:14 | |
| *** sdake has quit IRC | 03:14 | |
| *** unicell1 has quit IRC | 03:17 | |
| fungi | not surprising, as i said 1 and 2 always seem to take at least 50% longer | 03:17 |
| fungi | probably running on older flavors | 03:18 |
| clarkb | mordred: we can probably delete that job | 03:18 |
| *** baoli has joined #openstack-infra | 03:20 | |
| *** rbrndt has quit IRC | 03:20 | |
| *** yuanying has quit IRC | 03:20 | |
| *** yuanying has joined #openstack-infra | 03:21 | |
| *** links has joined #openstack-infra | 03:26 | |
| *** flwang1 has quit IRC | 03:26 | |
| * greghaynes returns from diknner | 03:27 | |
| greghaynes | Is there dib help needed? | 03:27 |
| greghaynes | as for the .d dirs, those are for manifests and should be extremely small | 03:28 |
| greghaynes | sounds like it was orphaned images taking up space mostly though? That was a known issue (builders wont cleanup existing images) which we decided to just deal with one time | 03:29 |
| *** changzhi has joined #openstack-infra | 03:35 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/project-config: Change service user domain for identity v3 only test https://review.openstack.org/271127 | 03:35 |
| *** jpr has quit IRC | 03:37 | |
| openstackgerrit | Merged openstack/requirements: Pin more broken DIB versions: 1.6.0-1.7.1 https://review.openstack.org/269553 | 03:38 |
| *** changzhi has quit IRC | 03:40 | |
| *** changzhi_ has joined #openstack-infra | 03:40 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/project-config: Run the identity v3 only jobs as part of integrated gate https://review.openstack.org/271128 | 03:40 |
| *** links has quit IRC | 03:40 | |
| *** changzhi_ has quit IRC | 03:41 | |
| *** changzhi has joined #openstack-infra | 03:41 | |
| *** changzhi has quit IRC | 03:42 | |
| *** changzhi has joined #openstack-infra | 03:42 | |
| *** baoli has quit IRC | 03:42 | |
| *** flwang1 has joined #openstack-infra | 03:44 | |
| *** piet has quit IRC | 03:45 | |
| *** kzaitsev_mb has quit IRC | 03:46 | |
| *** yamamoto_ has joined #openstack-infra | 03:47 | |
| *** links has joined #openstack-infra | 03:48 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/project-config: Run the identity v3 only jobs as part of integrated gate https://review.openstack.org/271128 | 03:49 |
| *** doug-fish has joined #openstack-infra | 03:50 | |
| *** tlian has quit IRC | 03:53 | |
| *** baoli has joined #openstack-infra | 03:53 | |
| *** unicell has joined #openstack-infra | 03:53 | |
| Thelo | Hi, I need to take down my CI for maintenance this week end, should I notify someone, or update a page? | 03:54 |
| *** yuanying has quit IRC | 03:55 | |
| *** yuanying has joined #openstack-infra | 03:57 | |
| openstackgerrit | Cody A.W. Somerville proposed openstack/gertty: Add missing import of 'six' to app.py https://review.openstack.org/271130 | 03:59 |
| *** doug-fish has quit IRC | 04:00 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements https://review.openstack.org/268438 | 04:03 |
| *** links has quit IRC | 04:04 | |
| *** bardia has joined #openstack-infra | 04:04 | |
| *** sabeen1 has joined #openstack-infra | 04:07 | |
| *** dims__ has quit IRC | 04:08 | |
| *** piet has joined #openstack-infra | 04:08 | |
| *** bardia has quit IRC | 04:08 | |
| *** vgridnev has joined #openstack-infra | 04:09 | |
| *** flwang1 has quit IRC | 04:09 | |
| *** thorst has joined #openstack-infra | 04:12 | |
| *** daemontool_ has quit IRC | 04:12 | |
| *** daemontool_ has joined #openstack-infra | 04:13 | |
| *** akshai has joined #openstack-infra | 04:13 | |
| *** links has joined #openstack-infra | 04:16 | |
| *** ddieterly has quit IRC | 04:16 | |
| *** akshai has quit IRC | 04:18 | |
| *** daemontool_ has quit IRC | 04:18 | |
| *** baoli has quit IRC | 04:19 | |
| *** thorst has quit IRC | 04:19 | |
| *** coolsvap|away has joined #openstack-infra | 04:21 | |
| *** coolsvap|away is now known as coolsvap | 04:22 | |
| *** Ramanjaneya has joined #openstack-infra | 04:23 | |
| *** links has quit IRC | 04:27 | |
| *** links has joined #openstack-infra | 04:28 | |
| *** Ramanjaneya has quit IRC | 04:29 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool: Add interface to dump dib command for an image https://review.openstack.org/271110 | 04:35 |
| ianw | clarkb: so did the dib images upload? how can we know? | 04:36 |
| *** bardia has joined #openstack-infra | 04:39 | |
| *** Ramanjaneya has joined #openstack-infra | 04:40 | |
| *** kzaitsev_mb has joined #openstack-infra | 04:43 | |
| *** ddieterly has joined #openstack-infra | 04:47 | |
| *** kzaitsev_mb has quit IRC | 04:48 | |
| clarkb | jhesketh can you check? if they are ready in what nodepool image-list reports then they uploaded | 04:49 |
| *** links has quit IRC | 04:50 | |
| jhesketh | doesn't look like they have uploaded | 04:51 |
| jhesketh | ubuntu-trusty is still building, the others are done | 04:51 |
| jhesketh | let me see if I can find out about uploading | 04:52 |
| *** ddieterly has quit IRC | 04:52 | |
| *** kzaitsev_mb has joined #openstack-infra | 04:54 | |
| *** fawadkhaliq has joined #openstack-infra | 04:56 | |
| *** links has joined #openstack-infra | 04:57 | |
| *** piet has quit IRC | 04:58 | |
| jhesketh | it just says 2016-01-22 02:52:05,274 INFO nodepool.builder: Registering image id: 176198 | 04:59 |
| *** kzaitsev_mb has quit IRC | 04:59 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Move hook generation in to python https://review.openstack.org/271139 | 04:59 |
| jhesketh | and then I can't find anything more... ie I can't see it trying to upload | 04:59 |
| jhesketh | maybe I'm looking in the wrong place | 04:59 |
| greghaynes | jhesketh: it can only either build or upload at a given point in time, not buth | 05:00 |
| greghaynes | both | 05:00 |
| jhesketh | greghaynes: right, but once it has built it uploads yes? | 05:00 |
| greghaynes | not necessarially - it asks gearman for the next job | 05:00 |
| greghaynes | which may or may not be an upload | 05:00 |
| *** esker has joined #openstack-infra | 05:01 | |
| greghaynes | I would kind of expect all the builds to happen and then all the uploads | 05:01 |
| jhesketh | ah okay, I see what you mean | 05:01 |
| jhesketh | right, that explains it | 05:01 |
| greghaynes | but not entirely sure that is how it would pan out | 05:01 |
| greghaynes | so, this actually means building/uploading is probably a bit slower now | 05:02 |
| greghaynes | :( | 05:02 |
| greghaynes | until we run more builders | 05:02 |
| jhesketh | ah the last one just finished building while I was looking.. what should I grep for in the logs to see if it's uploading? | 05:02 |
| greghaynes | jhesketh: youll see a got job <something? | 05:02 |
| greghaynes | itll be either image-upload or image-build | 05:03 |
| *** sabeen1 has quit IRC | 05:03 | |
| *** Nikolay_St has quit IRC | 05:05 | |
| *** piet has joined #openstack-infra | 05:05 | |
| jhesketh | hmm, nothing obvious like that... just that the image is ready | 05:06 |
| *** bpokorny has joined #openstack-infra | 05:07 | |
| greghaynes | are debug logs on for the builder? | 05:08 |
| greghaynes | http://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/builder.py#n203 that line | 05:09 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Move hook generation in to python https://review.openstack.org/271139 | 05:10 |
| jhesketh | greghaynes: yep, but all the jobs are just image-build | 05:11 |
| greghaynes | jhesketh: ok, so its probably just doing all the image builds first | 05:12 |
| jhesketh | right, but according to dib-image-list they are all done/ready | 05:12 |
| greghaynes | which makes some sense - nodepool will spam all the image-build jobs to gearman and then all the image-upload's for them once they are done | 05:12 |
| greghaynes | oh, so it is just sitting there not doing anything? | 05:12 |
| jhesketh | well I don't know about it doing nothing, but it doesn't appear to be doing anything | 05:12 |
| jhesketh | greghaynes: does this use zuul's gearman or its own | 05:13 |
| greghaynes | so you probably want ot check the gearman job list to see if any jobs are queued | 05:13 |
| greghaynes | zuul's | 05:13 |
| *** links has quit IRC | 05:13 | |
| *** sdake_ has quit IRC | 05:14 | |
| jhesketh | yep, that's why I asked :-) | 05:14 |
| greghaynes | jhesketh: do you knwo about the disconnect issue we hit last night? | 05:14 |
| greghaynes | I wonder if the same thing happened | 05:14 |
| *** akshai has joined #openstack-infra | 05:14 | |
| jhesketh | I saw something fly by on scrollback, but don't know any details | 05:14 |
| greghaynes | tl;dr is that nodepool disconnected from geard which causes all the queued jobs to get delted | 05:15 |
| greghaynes | and it happened right as it attempted uploading | 05:15 |
| *** thorst has joined #openstack-infra | 05:17 | |
| *** piet has quit IRC | 05:17 | |
| jhesketh | greghaynes: so the upload/delete jobs exist with a worker registered, but no jobs queued | 05:19 |
| jhesketh | (or running) | 05:19 |
| jhesketh | hmm, interesting | 05:19 |
| jhesketh | let me look for disconnects | 05:19 |
| greghaynes | :( so I bet the same thing happened | 05:19 |
| *** akshai has quit IRC | 05:19 | |
| greghaynes | on geard there was an error message | 05:19 |
| greghaynes | not sure what the error was on the nodepool side | 05:20 |
| jhesketh | ah okay, let me go poke geard then | 05:20 |
| jhesketh | greghaynes: do you know what the error message on geard was? | 05:20 |
| greghaynes | jhesketh: not exactly, basically it was "couldnt send data to connection" | 05:20 |
| *** links has joined #openstack-infra | 05:21 | |
| jhesketh | ack | 05:21 |
| *** MarkAtwood has joined #openstack-infra | 05:23 | |
| *** piet has joined #openstack-infra | 05:23 | |
| *** Hal has joined #openstack-infra | 05:24 | |
| *** Hal is now known as Guest72124 | 05:24 | |
| *** vgridnev has quit IRC | 05:24 | |
| *** thorst has quit IRC | 05:25 | |
| *** doug-fish has joined #openstack-infra | 05:32 | |
| *** doug-fish has quit IRC | 05:32 | |
| *** esker has quit IRC | 05:32 | |
| *** links has quit IRC | 05:36 | |
| openstackgerrit | Qiming Teng proposed openstack/requirements: Bump python-senlinclient version to 0.3.0 https://review.openstack.org/263598 | 05:37 |
| *** links has joined #openstack-infra | 05:37 | |
| *** armax has joined #openstack-infra | 05:37 | |
| *** piet has quit IRC | 05:38 | |
| openstackgerrit | Merged openstack/diskimage-builder: Updated from global requirements https://review.openstack.org/268438 | 05:38 |
| *** keedya has quit IRC | 05:39 | |
| *** bhunter71 has quit IRC | 05:40 | |
| *** gildub has quit IRC | 05:43 | |
| *** SlickNik has quit IRC | 05:43 | |
| *** spiffxp has quit IRC | 05:43 | |
| *** markvoelker has joined #openstack-infra | 05:44 | |
| *** tdasilva has quit IRC | 05:45 | |
| *** vipuls has quit IRC | 05:47 | |
| *** vgridnev has joined #openstack-infra | 05:47 | |
| *** ddieterly has joined #openstack-infra | 05:48 | |
| jhesketh | greghaynes: hmm so nothing obvious | 05:50 |
| jhesketh | mordred: is there a way for me to tell when your jjb stuff is done | 05:52 |
| *** ddieterly has quit IRC | 05:52 | |
| *** kushal has quit IRC | 05:57 | |
| *** links has quit IRC | 05:59 | |
| *** Daisy_ has joined #openstack-infra | 05:59 | |
| *** links has joined #openstack-infra | 06:01 | |
| *** kushal has joined #openstack-infra | 06:01 | |
| *** kushal has joined #openstack-infra | 06:01 | |
| *** Daisy_ has quit IRC | 06:03 | |
| *** markvoelker_ has joined #openstack-infra | 06:04 | |
| *** tdasilva has joined #openstack-infra | 06:04 | |
| *** SlickNik has joined #openstack-infra | 06:06 | |
| *** markvoelker has quit IRC | 06:07 | |
| *** spiffxp has joined #openstack-infra | 06:08 | |
| *** vipuls has joined #openstack-infra | 06:08 | |
| *** _nadya_ has joined #openstack-infra | 06:10 | |
| *** hdd has quit IRC | 06:11 | |
| fungi | jhesketh: he was running it under screen, looks like 01 finally finished but 02 crashed out on a jenkins api error so i've restarted it | 06:11 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: add support for package-installs to gentoo https://review.openstack.org/270597 | 06:11 |
| jhesketh | fungi: okay thanks, good to know | 06:11 |
| jhesketh | although what are you doing up still? | 06:11 |
| fungi | on my way to sleep, just checking in | 06:12 |
| fungi | you can attach to his screen session on 02 to monitor is thusly: | 06:13 |
| fungi | sudo su - mordred | 06:13 |
| fungi | script /dev/null | 06:13 |
| fungi | screen -x | 06:13 |
| *** camunoz has quit IRC | 06:13 | |
| fungi | (and when ctrl-a,d to detach rather than exiting the screen session in case he wanted to check the results later too) | 06:14 |
| *** bardia has quit IRC | 06:14 | |
| fungi | anyway, -> sleep | 06:15 |
| *** akshai has joined #openstack-infra | 06:15 | |
| *** aeng has quit IRC | 06:17 | |
| jhesketh | thanks for the tips :-) | 06:17 |
| *** kushal has quit IRC | 06:17 | |
| *** chenli has joined #openstack-infra | 06:19 | |
| *** akshai has quit IRC | 06:20 | |
| chenli | hello, I'm trying to run tempest test by hand, I have followed https://github.com/openstack/tempest | 06:20 |
| chenli | This is my steps and my tempest.conf : http://paste.openstack.org/show/484641/ | 06:21 |
| chenli | Then I try to run command " testr run tempest.api.image.v2.test_images_member_negative.ImagesMemberNegativeTest.test_image_share_invalid_status" | 06:21 |
| chenli | But I get error : http://paste.openstack.org/show/484642/ | 06:21 |
| chenli | anyone can help me ? | 06:21 |
| *** links has quit IRC | 06:22 | |
| *** thorst has joined #openstack-infra | 06:22 | |
| *** links has joined #openstack-infra | 06:23 | |
| *** coolsvap is now known as coolsvap|away | 06:26 | |
| *** exploreshaifali has joined #openstack-infra | 06:27 | |
| *** thorst has quit IRC | 06:29 | |
| *** jaosorior has quit IRC | 06:32 | |
| *** jaosorior has joined #openstack-infra | 06:33 | |
| *** vgridnev has quit IRC | 06:33 | |
| *** bardia has joined #openstack-infra | 06:33 | |
| *** HeOS has quit IRC | 06:33 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/267962 | 06:34 |
| *** pahuang has quit IRC | 06:34 | |
| *** fedexo has joined #openstack-infra | 06:39 | |
| *** bryan_att has quit IRC | 06:39 | |
| *** yamahata has joined #openstack-infra | 06:40 | |
| *** _nadya_ has quit IRC | 06:43 | |
| *** salv-orlando has joined #openstack-infra | 06:44 | |
| *** FallenPegasus has joined #openstack-infra | 06:45 | |
| *** unicell has left #openstack-infra | 06:46 | |
| Ramanjaneya | Hi, How to import a project which is not part of openstack release? | 06:46 |
| *** links has quit IRC | 06:46 | |
| *** achanda has quit IRC | 06:47 | |
| *** scheuran has joined #openstack-infra | 06:48 | |
| *** ddieterly has joined #openstack-infra | 06:49 | |
| Ramanjaneya | any one can help? How to import a project which is not part of openstack release? | 06:49 |
| *** links has joined #openstack-infra | 06:50 | |
| *** unicell1 has joined #openstack-infra | 06:50 | |
| *** hdd has joined #openstack-infra | 06:50 | |
| *** bardia has quit IRC | 06:51 | |
| *** ddieterly has quit IRC | 06:53 | |
| *** vgridnev has joined #openstack-infra | 06:54 | |
| *** elo has joined #openstack-infra | 06:57 | |
| *** k4n0 has joined #openstack-infra | 07:05 | |
| *** links has quit IRC | 07:09 | |
| *** links has joined #openstack-infra | 07:09 | |
| *** salv-orlando has quit IRC | 07:09 | |
| *** hichihara has joined #openstack-infra | 07:11 | |
| *** fedexo has quit IRC | 07:13 | |
| *** hdd has quit IRC | 07:15 | |
| Thelo | chenli: you probably want to ask this question in #openstack-qa | 07:15 |
| chenli | Thelo: sorry... Thanks! | 07:16 |
| *** akshai has joined #openstack-infra | 07:16 | |
| *** armax has quit IRC | 07:18 | |
| *** akshai has quit IRC | 07:21 | |
| *** mpavone has joined #openstack-infra | 07:21 | |
| chenli | This is my steps and my tempest.conf http://paste.openstack.org/show/484646/ | 07:21 |
| chenli | sorry.... | 07:22 |
| *** armax has joined #openstack-infra | 07:24 | |
| *** chenli has quit IRC | 07:24 | |
| *** chenli has joined #openstack-infra | 07:24 | |
| *** thorst has joined #openstack-infra | 07:28 | |
| *** rcernin has joined #openstack-infra | 07:31 | |
| *** links has quit IRC | 07:31 | |
| *** thorst has quit IRC | 07:35 | |
| *** dizquierdo has joined #openstack-infra | 07:39 | |
| *** pavel_bondar has joined #openstack-infra | 07:42 | |
| *** armax has quit IRC | 07:43 | |
| *** ociuhandu has joined #openstack-infra | 07:43 | |
| *** oomichi has joined #openstack-infra | 07:44 | |
| *** links has joined #openstack-infra | 07:48 | |
| *** jed56 has joined #openstack-infra | 07:48 | |
| *** mpavone has quit IRC | 07:48 | |
| *** ddieterly has joined #openstack-infra | 07:50 | |
| *** mpavone has joined #openstack-infra | 07:51 | |
| *** mpavone has quit IRC | 07:52 | |
| *** Daisy_ has joined #openstack-infra | 07:53 | |
| *** links has quit IRC | 07:54 | |
| *** ddieterly has quit IRC | 07:55 | |
| *** Daisy__ has joined #openstack-infra | 07:57 | |
| *** Daisy_ has quit IRC | 07:59 | |
| *** mrmartin has joined #openstack-infra | 08:01 | |
| *** esikachev has joined #openstack-infra | 08:01 | |
| *** changzhi has quit IRC | 08:02 | |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Remove py27-scenario-unit check from sahara https://review.openstack.org/270704 | 08:02 |
| *** psanchez has joined #openstack-infra | 08:02 | |
| openstackgerrit | Evgeny Sikachev proposed openstack-infra/project-config: Separate gate job for sahara and sahara-scenario https://review.openstack.org/270728 | 08:02 |
| *** sflanigan has quit IRC | 08:02 | |
| *** Nikolay_St has joined #openstack-infra | 08:03 | |
| *** markus_z has joined #openstack-infra | 08:04 | |
| *** arxcruz has joined #openstack-infra | 08:07 | |
| AJaeger | Ramanjaneya: is this project related to OpenStack? Then read http://docs.openstack.org/infra/manual/creators.html | 08:07 |
| AJaeger | Reading backscroll I'm a bit confused about the pip status. Do we still use pip 8.0 in the images? I checked one job and it still used 8.0. When should 8.0.2 get used? | 08:10 |
| *** hichihara has quit IRC | 08:11 | |
| *** boris-42 has quit IRC | 08:13 | |
| *** _nadya_ has joined #openstack-infra | 08:14 | |
| *** akshai has joined #openstack-infra | 08:17 | |
| *** achanda has joined #openstack-infra | 08:17 | |
| *** _nadya_ has quit IRC | 08:17 | |
| *** daemontool has joined #openstack-infra | 08:18 | |
| *** andymaier has joined #openstack-infra | 08:19 | |
| *** vgridnev has quit IRC | 08:20 | |
| *** akshai has quit IRC | 08:22 | |
| *** links has joined #openstack-infra | 08:22 | |
| *** achanda has quit IRC | 08:22 | |
| AJaeger | Any project-config core around? We should probably merge https://review.openstack.org/271037 quickly to make internap happy... | 08:22 |
| *** wei-wang has quit IRC | 08:23 | |
| *** matrohon has joined #openstack-infra | 08:24 | |
| jhesketh | AJaeger: thanks, +w | 08:26 |
| *** Daisy__ has quit IRC | 08:28 | |
| *** Daisy_ has joined #openstack-infra | 08:28 | |
| *** thorst has joined #openstack-infra | 08:32 | |
| *** amaretskiy has joined #openstack-infra | 08:32 | |
| *** Daisy_ has quit IRC | 08:33 | |
| openstackgerrit | Merged openstack-infra/project-config: Reduce internap max servers from 72 to 64 https://review.openstack.org/271037 | 08:33 |
| *** links has quit IRC | 08:35 | |
| openstackgerrit | Fausto Marzi proposed openstack/requirements: Add modules for freezer mitaka m2 release https://review.openstack.org/271072 | 08:37 |
| *** zeih has joined #openstack-infra | 08:37 | |
| amaretskiy | hi there! I have a question regarding very frequent error which happens during dsvm workspace setup (logs/devstack-gate-setup-workspace-new.txt.gz) - see http://logs.openstack.org/76/270176/4/check/gate-rally-dsvm-cli/83a07b3/logs/devstack-gate-setup-workspace-new.txt.gz#_2016-01-21_09_01_55_278 | 08:38 |
| *** markvoelker_ has quit IRC | 08:38 | |
| amaretskiy | this "fatal: Couldn't find remote ref" sometimes eats a lot of job's time | 08:39 |
| *** thorst has quit IRC | 08:39 | |
| amaretskiy | in log given above there are 10 minutes lost! | 08:39 |
| amaretskiy | for getting amount of lost time you can use script https://raw.githubusercontent.com/maretskiy/stuff/master/openstack/parse-lost-time.py | 08:40 |
| *** HeOS has joined #openstack-infra | 08:41 | |
| *** oomichi is now known as oomichi_away | 08:45 | |
| AJaeger | amaretskiy: thanks for digging into this. I can't help with this and fear nobody else is around to help either. Best come back during US morning again unless you get an answer. | 08:45 |
| *** flepied has quit IRC | 08:45 | |
| amaretskiy | AJaeger: thank you, I will raise this question later | 08:46 |
| *** daemontool has quit IRC | 08:48 | |
| *** daemontool has joined #openstack-infra | 08:48 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool: Add interface to dump dib command for an image https://review.openstack.org/271110 | 08:49 |
| *** oomichi_away has quit IRC | 08:49 | |
| Thelo | looks like this patch is stucked in the gate https://review.openstack.org/#/c/268348/ , it has already failed because of timeout but is still blocking the next jobs | 08:50 |
| *** k4n0 has quit IRC | 08:52 | |
| *** shardy has joined #openstack-infra | 08:53 | |
| *** ihrachys has joined #openstack-infra | 08:53 | |
| *** yolanda has joined #openstack-infra | 09:00 | |
| yolanda | morning | 09:02 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Fix trailing whitespace https://review.openstack.org/271192 | 09:02 |
| AJaeger | morning, yolanda | 09:03 |
| *** daemontool_ has joined #openstack-infra | 09:03 | |
| *** jordanP has joined #openstack-infra | 09:04 | |
| *** ihrachys has quit IRC | 09:05 | |
| *** daemontool has quit IRC | 09:06 | |
| jhesketh | fungi, mordred: for when you get back: all of the jenkins masters failed. It looks like due to removing an old job, so I think the other jobs should be updated correctly. Unfortunately it looks like the jjb code raises just one error instead of raising all of the errors. jenkins02 failed for a different reason that I'm unsure of. | 09:07 |
| *** ihrachys has joined #openstack-infra | 09:09 | |
| jhesketh | I'm going to queue one of the releases and see how it goes... if it works, I'll do the others | 09:10 |
| *** Daisy_ has joined #openstack-infra | 09:11 | |
| *** Daisy_ has quit IRC | 09:11 | |
| *** Guest16620 is now known as tdurakov | 09:11 | |
| *** Daisy_ has joined #openstack-infra | 09:11 | |
| jhesketh | heh, sigh, of course it goes to jenkins02 | 09:13 |
| *** flepied has joined #openstack-infra | 09:15 | |
| *** jistr has joined #openstack-infra | 09:15 | |
| *** Daisy_ has quit IRC | 09:16 | |
| AJaeger | jhesketh: what's with 268348, see https://jenkins07.openstack.org/job/gate-tempest-dsvm-neutron-full/21777/console | 09:16 |
| AJaeger | build time out 20 mins ago - and still top of the queue in zuul and jenkins is not finished with it. Does that look sane? | 09:17 |
| jhesketh | AJaeger: certainly looks a bit odd | 09:18 |
| *** akshai has joined #openstack-infra | 09:18 | |
| *** hashar has joined #openstack-infra | 09:18 | |
| *** binbincong has quit IRC | 09:18 | |
| *** flepied has quit IRC | 09:20 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Add networking-generic-switch project https://review.openstack.org/267664 | 09:22 |
| *** akshai has quit IRC | 09:23 | |
| *** mhickey has joined #openstack-infra | 09:23 | |
| *** e0ne has joined #openstack-infra | 09:24 | |
| jhesketh | AJaeger: not really sure what's going on.. the worst part is if that job has failed the rest of the pipeline will need redoing :-( | 09:24 |
| *** vgridnev has joined #openstack-infra | 09:25 | |
| *** e0ne has quit IRC | 09:27 | |
| jhesketh | AJaeger: I don't think I can force stop is since I'd need to restart jenkins07 to get the permissions | 09:27 |
| jhesketh | which we may need to anyway | 09:27 |
| *** Daisy_ has joined #openstack-infra | 09:28 | |
| jhesketh | but it's currently building a lot of jobs | 09:28 |
| AJaeger | we can just update it and see whether it leaves the queue ;) | 09:28 |
| AJaeger | jhesketh: no ideas what to do. yolanda, do you have any? | 09:29 |
| *** Daisy_ has quit IRC | 09:29 | |
| *** Daisy_ has joined #openstack-infra | 09:29 | |
| yolanda | let me see if i can stop it, we should with admin perms | 09:30 |
| jhesketh | yolanda: oh cool, you have admin perms? | 09:30 |
| yolanda | i think so | 09:30 |
| yolanda | yep | 09:31 |
| jhesketh | (I never set them up on jenkins) | 09:31 |
| yolanda | fungi gave me some time ago | 09:31 |
| jhesketh | oh cool, that worked | 09:31 |
| jhesketh | thanks yolanda :-) | 09:31 |
| *** Daisy_ has quit IRC | 09:31 | |
| yolanda | jhesketh, i can add you as admin | 09:31 |
| jhesketh | yolanda: while you're t here, would you mind adding me as an admin | 09:31 |
| jhesketh | heh, yes please :-) | 09:31 |
| jhesketh | if you have time to do all the masters that might save some trouble in the future | 09:31 |
| *** Daisy_ has joined #openstack-infra | 09:31 | |
| jhesketh | AJaeger: looks like the gate has recalculated... at least we didn't lose all of the other jobs running on 07 | 09:32 |
| yolanda | mm, jhesketh, actually i'm not sure on how to do it, it works with SSO and is different from what i'm used downstream. Seems you need to belong to a openstack-ci-admins group | 09:33 |
| *** boris-42 has joined #openstack-infra | 09:33 | |
| jhesketh | yolanda: I assume that's a jenkins group? | 09:33 |
| AJaeger | jhesketh: yes, looks like it recovered - and we lost 10 finished gate jobs due to a timeout ;( | 09:33 |
| yolanda | jhesketh, it looks related to launchpad because when you login, you need to check that you belong to that group | 09:34 |
| *** markvoelker has joined #openstack-infra | 09:34 | |
| jhesketh | AJaeger: could have been worse | 09:34 |
| AJaeger | sure ;) | 09:34 |
| AJaeger | thansk for checking | 09:34 |
| *** _nadya_ has joined #openstack-infra | 09:34 | |
| *** Daisy_ has quit IRC | 09:34 | |
| *** Daisy_ has joined #openstack-infra | 09:35 | |
| *** _nadya_ has quit IRC | 09:35 | |
| *** flepied has joined #openstack-infra | 09:35 | |
| jhesketh | yolanda: ah, I can select that group when logging in with ubuntu one. I just never thought to :-( | 09:36 |
| jhesketh | thanks! | 09:36 |
| yolanda | cool | 09:36 |
| *** placko has joined #openstack-infra | 09:38 | |
| *** thorst has joined #openstack-infra | 09:39 | |
| *** markvoelker has quit IRC | 09:39 | |
| pavel_bondar | hi, I have an issue that new version of package infoblox-client>=0.3.3 can not be found in py27 test: http://logs.openstack.org/74/270574/2/check/gate-networking-infoblox-python27/3f04ae3/console.html#_2016-01-22_09_25_22_168 | 09:39 |
| *** derekh has joined #openstack-infra | 09:39 | |
| pavel_bondar | but for the same PS infoblox-client=0.3.5 was installed correctly in pep8 check: http://logs.openstack.org/74/270574/2/check/gate-networking-infoblox-pep8/507ef89/console.html#_2016-01-22_09_25_43_717 | 09:39 |
| pavel_bondar | is it a known issue? | 09:39 |
| *** markvoelker has joined #openstack-infra | 09:39 | |
| *** abregman has joined #openstack-infra | 09:42 | |
| *** exploreshaifali has quit IRC | 09:43 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:43 | |
| *** yaume has joined #openstack-infra | 09:44 | |
| *** markvoelker has quit IRC | 09:44 | |
| *** thorst has quit IRC | 09:45 | |
| *** markvoelker has joined #openstack-infra | 09:45 | |
| hashar | pavel_bondar: dont quote me, but I think the jobs are using a Pypi mirror ( should be http://pypi.openstack.org/openstack/ ) and that mirror might not have received the last infoblox-client updates | 09:45 |
| hashar | url is forged , no idea whether it is the proper one actually :( | 09:45 |
| jhesketh | fungi, mordred: release is failing even if it's not on jenkins02. Since the jjb apply isn't working I might leave it for now sorry | 09:49 |
| Qiming | hi, anyone is looking into the requirements gate? it seems broken | 09:49 |
| *** markvoelker has quit IRC | 09:50 | |
| Qiming | or should I ask this question in a different channel? | 09:52 |
| *** ddieterly has joined #openstack-infra | 09:52 | |
| *** sfinucan has joined #openstack-infra | 09:53 | |
| pavel_bondar | hashar: well, looks like you right, this server doesn't have new versions of infoblox client (0.3.3+) http://pypi.region-b.geo-1.openstack.org/packages/2.7/i/infoblox-client/, but this one have: http://pypi.iad.openstack.org/packages/2.7/i/infoblox-client/ | 09:55 |
| *** ddieterly has quit IRC | 09:57 | |
| pavel_bondar | just curious where should I report issues like that? | 09:57 |
| *** pavel_bondar is now known as pavel_bondar|afk | 09:58 | |
| *** baoli has joined #openstack-infra | 09:59 | |
| *** rguillebert has joined #openstack-infra | 10:01 | |
| *** MarkAtwood has quit IRC | 10:01 | |
| *** _nadya_ has joined #openstack-infra | 10:01 | |
| *** ihrachys has quit IRC | 10:01 | |
| *** baoli has quit IRC | 10:03 | |
| *** jpr has joined #openstack-infra | 10:03 | |
| *** abregman is now known as abregman|afk | 10:05 | |
| *** dtantsur|afk is now known as dtantsur | 10:06 | |
| *** kzaitsev_mb has quit IRC | 10:08 | |
| *** FallenPegasus has quit IRC | 10:10 | |
| *** jaosorior has quit IRC | 10:14 | |
| *** jaosorior has joined #openstack-infra | 10:15 | |
| openstackgerrit | afazekas proposed openstack-infra/tripleo-ci: Add dstat https://review.openstack.org/271218 | 10:18 |
| *** akshai has joined #openstack-infra | 10:18 | |
| *** achanda has joined #openstack-infra | 10:20 | |
| *** Daisy_ has quit IRC | 10:21 | |
| *** Daisy_ has joined #openstack-infra | 10:21 | |
| *** Daisy_ has quit IRC | 10:22 | |
| *** Daisy_ has joined #openstack-infra | 10:22 | |
| *** akshai has quit IRC | 10:23 | |
| openstackgerrit | afazekas proposed openstack-infra/tripleo-ci: Add dstat https://review.openstack.org/271218 | 10:27 |
| *** Daisy_ has quit IRC | 10:27 | |
| *** achanda has quit IRC | 10:28 | |
| *** pblaho has quit IRC | 10:28 | |
| *** bradjones_ has quit IRC | 10:28 | |
| *** bradjones_ has joined #openstack-infra | 10:29 | |
| *** bradjones_ has quit IRC | 10:29 | |
| *** bradjones_ has joined #openstack-infra | 10:29 | |
| *** rossella_s has quit IRC | 10:31 | |
| *** rossella_s has joined #openstack-infra | 10:31 | |
| *** ldnunes has joined #openstack-infra | 10:33 | |
| *** esker has joined #openstack-infra | 10:33 | |
| *** markvoelker has joined #openstack-infra | 10:35 | |
| *** Qiming has quit IRC | 10:37 | |
| openstackgerrit | Merged openstack-infra/project-config: Enable Zaqar in Heat integration jobs https://review.openstack.org/270659 | 10:38 |
| *** esker has quit IRC | 10:38 | |
| openstackgerrit | Merged openstack-infra/project-config: Create puppet-oslo module https://review.openstack.org/270895 | 10:38 |
| *** markvoelker has quit IRC | 10:39 | |
| *** bradjones_ has quit IRC | 10:40 | |
| *** e0ne has joined #openstack-infra | 10:40 | |
| openstackgerrit | Merged openstack-infra/project-config: Separate gate job for sahara and sahara-scenario https://review.openstack.org/270728 | 10:40 |
| *** dingyichen has quit IRC | 10:40 | |
| openstackgerrit | Angus Lees proposed openstack-infra/project-config: Create new tempest-dsvm-full-head job-template https://review.openstack.org/270143 | 10:41 |
| openstackgerrit | Merged openstack-infra/project-config: Remove py27-scenario-unit check from sahara https://review.openstack.org/270704 | 10:41 |
| openstackgerrit | Merged openstack-infra/project-config: puppet/bundle: retry failed network or git requests https://review.openstack.org/271080 | 10:41 |
| *** bradjones_ has joined #openstack-infra | 10:41 | |
| *** bradjones_ has quit IRC | 10:41 | |
| *** bradjones_ has joined #openstack-infra | 10:41 | |
| openstackgerrit | afazekas proposed openstack-infra/tripleo-ci: Add dstat https://review.openstack.org/271218 | 10:42 |
| *** thorst has joined #openstack-infra | 10:43 | |
| *** vgridnev has quit IRC | 10:44 | |
| *** bradjones_ has quit IRC | 10:45 | |
| *** bradjones_ has joined #openstack-infra | 10:47 | |
| *** bradjones_ has quit IRC | 10:47 | |
| *** bradjones_ has joined #openstack-infra | 10:47 | |
| *** sdake has joined #openstack-infra | 10:48 | |
| *** vgridnev has joined #openstack-infra | 10:48 | |
| *** thorst has quit IRC | 10:49 | |
| *** bastelfreak is now known as hrnz | 10:50 | |
| *** hrnz has quit IRC | 10:50 | |
| *** bastelfreak1 has joined #openstack-infra | 10:51 | |
| *** bastelfreak1 is now known as bastelfreak | 10:52 | |
| *** pblaho has joined #openstack-infra | 10:53 | |
| *** ddieterly has joined #openstack-infra | 10:53 | |
| openstackgerrit | Merged openstack-infra/project-config: End Gating for Tuskar Projects https://review.openstack.org/270850 | 10:54 |
| openstackgerrit | Merged openstack-infra/project-config: Make the rally job experimental https://review.openstack.org/183836 | 10:55 |
| hashar | pavel_bondar|afk: sorry, so looks like one of the pypi mirror is not up-to-date :( | 10:55 |
| *** ddieterly has quit IRC | 10:57 | |
| Daviey | fungi/ jhesketh: Did you rekick zuul for my tarball jobs? | 10:58 |
| jhesketh | Daviey: yes, but however it was still failing | 10:58 |
| jhesketh | the jjb updates did not apply (it would seem) | 10:59 |
| Daviey | jhesketh: Are you working on it now? | 10:59 |
| jhesketh | I'm hoping mordred will be able to poke at those some more when he wakes | 10:59 |
| *** sshnaidm has joined #openstack-infra | 10:59 | |
| Daviey | jhesketh: ok, great. Guessing there is nothing i can do? | 10:59 |
| *** dims has joined #openstack-infra | 11:00 | |
| jhesketh | I don't think so | 11:00 |
| Daviey | ok, thanks | 11:00 |
| jhesketh | sorry I'm not able to help:-( | 11:00 |
| openstackgerrit | Merged openstack-infra/project-config: skip most tests on a nova locale import https://review.openstack.org/270987 | 11:01 |
| *** fhubik has joined #openstack-infra | 11:01 | |
| openstackgerrit | Merged openstack-infra/project-config: only run api-guide on tox.ini or api-guide changes https://review.openstack.org/270992 | 11:02 |
| *** pavel_bondar|afk is now known as pavel_bondar | 11:03 | |
| *** mrmartin has quit IRC | 11:03 | |
| openstackgerrit | afazekas proposed openstack-infra/tripleo-ci: scp/ssh do not check for know host keys https://review.openstack.org/269660 | 11:05 |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Add Infra Cloud compute node definition https://review.openstack.org/260022 | 11:09 |
| *** electrofelix has joined #openstack-infra | 11:09 | |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Update Urls for redirects and broken https://review.openstack.org/271243 | 11:10 |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Pass through env proxy variables for docs-linkcheck https://review.openstack.org/271244 | 11:10 |
| esikachev | hi! please, review. we are need proposal bot https://review.openstack.org/#/c/270676/ | 11:10 |
| *** kzaitsev_mb has joined #openstack-infra | 11:11 | |
| *** sdake_ has joined #openstack-infra | 11:12 | |
| *** aysyd has joined #openstack-infra | 11:12 | |
| *** markvoelker has joined #openstack-infra | 11:12 | |
| *** sdake has quit IRC | 11:13 | |
| AJaeger | esikachev: did you read http://docs.openstack.org/developer/requirements/ and especially the section on "Enforcement in Projects"? Is your project setup? | 11:13 |
| * AJaeger does not see check-requirements on the repo but didn't check for open reviews | 11:14 | |
| *** placko has quit IRC | 11:14 | |
| *** Qiming has joined #openstack-infra | 11:15 | |
| esikachev | AJaeger: https://review.openstack.org/#/c/270680/ | 11:15 |
| AJaeger | esikachev: great! | 11:17 |
| *** markvoelker has quit IRC | 11:18 | |
| AJaeger | yeah, integrated gate lag is down to less than 24 hours ... | 11:19 |
| AJaeger | more than 10 jobs merging at once... | 11:20 |
| openstackgerrit | Derek Higgins proposed openstack/diskimage-builder: Replace symlink for dib-python if it exists https://review.openstack.org/271247 | 11:21 |
| *** exploreshaifali has joined #openstack-infra | 11:21 | |
| openstackgerrit | Merged openstack-infra/project-config: Add lib-forward-testing for OSprofiler https://review.openstack.org/269905 | 11:21 |
| *** fawadkhaliq has quit IRC | 11:22 | |
| *** fawadkhaliq has joined #openstack-infra | 11:23 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: skip most tests on a locale import https://review.openstack.org/271014 | 11:23 |
| openstackgerrit | afazekas proposed openstack-infra/tripleo-ci: Add dstat https://review.openstack.org/271218 | 11:23 |
| *** achanda has joined #openstack-infra | 11:24 | |
| *** yamamoto_ has quit IRC | 11:25 | |
| *** ociuhandu has quit IRC | 11:25 | |
| *** rfolco has joined #openstack-infra | 11:25 | |
| *** achanda has quit IRC | 11:28 | |
| *** chenli_ has joined #openstack-infra | 11:28 | |
| *** chenli has quit IRC | 11:30 | |
| *** chenli_ is now known as chenli | 11:30 | |
| *** dizquierdo has quit IRC | 11:32 | |
| *** trown|outttypeww is now known as trown | 11:39 | |
| *** jpr has quit IRC | 11:40 | |
| *** fawadkhaliq has quit IRC | 11:41 | |
| *** jpr has joined #openstack-infra | 11:41 | |
| *** andymaier has quit IRC | 11:43 | |
| *** fhubik is now known as fhubik_brb | 11:46 | |
| *** unicell1 has quit IRC | 11:46 | |
| *** thorst has joined #openstack-infra | 11:47 | |
| *** openstackgerrit has quit IRC | 11:47 | |
| *** openstackgerrit has joined #openstack-infra | 11:47 | |
| *** fawadkhaliq has joined #openstack-infra | 11:48 | |
| openstackgerrit | Sean Dague proposed openstack/requirements: Update to os-brick 0.8.0 https://review.openstack.org/267725 | 11:48 |
| *** fhubik_brb is now known as fhubik | 11:48 | |
| openstackgerrit | Sean Dague proposed openstack/requirements: Bump oslo.i18n to 2.1.0 https://review.openstack.org/257679 | 11:49 |
| *** fawadkhaliq has quit IRC | 11:49 | |
| *** fawadkhaliq has joined #openstack-infra | 11:50 | |
| *** maishsk_ has joined #openstack-infra | 11:50 | |
| *** bradjones_ has quit IRC | 11:52 | |
| *** maishsk has quit IRC | 11:53 | |
| *** maishsk_ is now known as maishsk | 11:53 | |
| *** boris-42 has quit IRC | 11:53 | |
| *** bradjones_ has joined #openstack-infra | 11:53 | |
| *** bradjones_ has quit IRC | 11:53 | |
| *** bradjones_ has joined #openstack-infra | 11:53 | |
| *** fawadkhaliq has quit IRC | 11:54 | |
| *** ddieterly has joined #openstack-infra | 11:54 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Logging in no longer causes page refresh. https://review.openstack.org/270882 | 11:55 |
| openstackgerrit | Merged openstack-infra/project-config: Add check-requirements to sahara-scenario project https://review.openstack.org/270680 | 11:57 |
| openstackgerrit | Merged openstack-infra/project-config: Add project configuration for Vahana. https://review.openstack.org/266549 | 11:57 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Fix width of "no result" and "loading" rows in list tables https://review.openstack.org/269663 | 11:57 |
| *** ddieterly has quit IRC | 11:59 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: Fix zuul cloner macro for requirements repo https://review.openstack.org/271257 | 11:59 |
| sdague | AJaeger: noticed that on requirements fail in the gate this morning | 12:00 |
| *** bradjones has quit IRC | 12:01 | |
| *** bradjones_ is now known as bradjones | 12:01 | |
| *** piet has joined #openstack-infra | 12:02 | |
| *** AndreiB has joined #openstack-infra | 12:02 | |
| *** ociuhandu has joined #openstack-infra | 12:02 | |
| *** maishsk_ has joined #openstack-infra | 12:03 | |
| *** jaosorior has quit IRC | 12:03 | |
| *** maishsk has quit IRC | 12:03 | |
| *** maishsk_ is now known as maishsk | 12:03 | |
| *** jaosorior has joined #openstack-infra | 12:03 | |
| *** maishsk has quit IRC | 12:05 | |
| *** markvoelker has joined #openstack-infra | 12:07 | |
| *** yamamoto has joined #openstack-infra | 12:09 | |
| *** andymaier has joined #openstack-infra | 12:10 | |
| *** baoli has joined #openstack-infra | 12:10 | |
| *** erlon has joined #openstack-infra | 12:10 | |
| *** chenli has quit IRC | 12:10 | |
| *** markvoelker has quit IRC | 12:12 | |
| *** dizquierdo has joined #openstack-infra | 12:13 | |
| *** yamamoto has quit IRC | 12:13 | |
| * krotscheck reads backscroll | 12:15 | |
| *** yamamoto has joined #openstack-infra | 12:15 | |
| krotscheck | Seems like the various pip failures are currently unwinding in the gate, do I have that right? | 12:15 |
| * krotscheck noticed a few of his own builds seem to be passing again. | 12:16 | |
| *** piet has quit IRC | 12:17 | |
| *** moravec has quit IRC | 12:18 | |
| *** akshai has joined #openstack-infra | 12:20 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Remove legacy pypi_trusted_hosts https://review.openstack.org/270930 | 12:22 |
| *** piet has joined #openstack-infra | 12:23 | |
| *** maishsk has joined #openstack-infra | 12:23 | |
| *** akshai has quit IRC | 12:24 | |
| *** mrmartin has joined #openstack-infra | 12:25 | |
| *** mase_x200 has joined #openstack-infra | 12:25 | |
| *** moravec has joined #openstack-infra | 12:25 | |
| *** dprince has joined #openstack-infra | 12:25 | |
| *** sdake_ has quit IRC | 12:27 | |
| *** baoli has quit IRC | 12:28 | |
| mordred | jhesketh: morning! | 12:30 |
| krotscheck | It's a mordred! | 12:31 |
| krotscheck | Hey, how'd things shake out last night with pip things? | 12:31 |
| mordred | dude | 12:31 |
| mordred | last night sucked a lot | 12:31 |
| *** yamamoto has quit IRC | 12:32 | |
| krotscheck | I gathered. | 12:32 |
| krotscheck | Are things back on a happy path, or is there still work I can help with? | 12:32 |
| *** yamamoto has joined #openstack-infra | 12:32 | |
| krotscheck | (Or is the mirror work I'm doing part of the happy path? | 12:32 |
| AJaeger | sdague: looks fine | 12:33 |
| *** yamamoto has quit IRC | 12:33 | |
| *** openstackgerrit has quit IRC | 12:33 | |
| *** openstackgerrit has joined #openstack-infra | 12:33 | |
| *** yamamoto has joined #openstack-infra | 12:33 | |
| AJaeger | I've still seen 5 hours ago some pip failures, testing my usual one now again | 12:33 |
| *** ppai has joined #openstack-infra | 12:33 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Removed manifests for legacy mirrors https://review.openstack.org/253506 | 12:33 |
| mordred | krotscheck: I think the mirror work is part of the happy path - it seems it's maybe unwinding? | 12:33 |
| mordred | krotscheck: also, if we don't watch out, we're going to have awesome new mirrors | 12:34 |
| krotscheck | mordred: That's what I extracted from scrollback. | 12:34 |
| * krotscheck is going to reserve judgement until his rechecks re-check | 12:34 | |
| *** gordc has joined #openstack-infra | 12:34 | |
| *** [1]Thelo has joined #openstack-infra | 12:34 | |
| krotscheck | mordred: NOT NEW MIRRORS | 12:34 |
| * krotscheck gasps | 12:34 | |
| mordred | krotscheck: inorite? | 12:35 |
| krotscheck | mordred: LMK if you have some review bandwidth, there's some things leftover from yesterday that are +2x2 but no +A. | 12:35 |
| AJaeger | sdague: I followed your example with locale imports, see https://review.openstack.org/271014 | 12:35 |
| mordred | krotscheck: I do - but I should coffee first (mmm coffee) | 12:36 |
| sdague | AJaeger: seems good | 12:36 |
| krotscheck | mordred: mmmm... good idea. I should get more. | 12:36 |
| *** Thelo has quit IRC | 12:36 | |
| *** [1]Thelo is now known as Thelo | 12:36 | |
| sdague | honestly, with the locale imports, it might be nice to make a dedicated job for it, and do it like requirements | 12:36 |
| sdague | because it would simplify the zuul logic | 12:37 |
| krotscheck | mordred: Once you're back, I think the plug in the review chain is https://review.openstack.org/#/c/270486/ | 12:37 |
| sdague | to have it opt in instead of opt out | 12:37 |
| *** ildikov has quit IRC | 12:37 | |
| AJaeger | sdague: since we need at least one job to run, we could have a locale-test environment and require it... | 12:38 |
| AJaeger | sdague: let me add this as note for my translation rework and think about it | 12:38 |
| krotscheck | nibalizer: You mention a trivial bug in the hiera data repo, but there wasn't any linking? https://review.openstack.org/#/c/270520/5 | 12:38 |
| mordred | krotscheck: I believe it's that there was a file in hiera that did not have the yamls | 12:40 |
| *** jordanP has quit IRC | 12:40 | |
| mordred | s/the yamls/the .yaml extension/ | 12:40 |
| * mordred checking | 12:40 | |
| AJaeger | sdague: https://review.openstack.org/#/c/271154/ is the fix for liberty nova translations, could you merge that one, please? | 12:41 |
| *** iremizov has joined #openstack-infra | 12:41 | |
| sdague | AJaeger: where are we with an actual i18n full stack test? | 12:41 |
| sdague | because I thought that was going to be in the works for a bit | 12:42 |
| *** markvoelker has joined #openstack-infra | 12:42 | |
| AJaeger | sdague: daisy wanted to work on that but I haven't seen anything of substance ;( | 12:42 |
| sdague | AJaeger: approved | 12:42 |
| *** baoli has joined #openstack-infra | 12:42 | |
| AJaeger | better ask her directly - or on the i18n mailing list. | 12:42 |
| AJaeger | Thanks! | 12:42 |
| *** jaypipes has joined #openstack-infra | 12:45 | |
| *** matrohon has quit IRC | 12:45 | |
| *** jaypipes is now known as leakypipes | 12:45 | |
| *** yamamoto has quit IRC | 12:46 | |
| AJaeger | Just run a recheck - and still got pip 8.0.0 in https://jenkins06.openstack.org/job/gate-openstack-doc-tools-linters/1/console | 12:46 |
| mordred | sdague, AJaeger: I remember someone talking about it recently | 12:46 |
| AJaeger | and a failure due to that ;8 | 12:46 |
| *** yamamoto has joined #openstack-infra | 12:46 | |
| mordred | I do not remember the details, but I believe it's still in works by someone | 12:46 |
| AJaeger | mordred: if you hear it again, please encourage that person ;) | 12:46 |
| AJaeger | there's the spec for a translation test server taking next steps - but that's horizon testing, not really nova etc | 12:47 |
| krotscheck | mordred: Ok, so is this a fixed thing? | 12:47 |
| krotscheck | Also, mordred, for your reading pleasure this morning :https://github.com/nodejs/node/pull/4765 | 12:47 |
| *** ildikov has joined #openstack-infra | 12:49 | |
| *** yamamoto has quit IRC | 12:51 | |
| mordred | krotscheck: wow | 12:51 |
| mordred | krotscheck: that is, in fact, actually interesting | 12:52 |
| krotscheck | mordred: I applaud Microsoft's engagement in the community, but I've been burned too frequently by the browser wars. | 12:53 |
| mordred | krotscheck: I love in the middle where they have the exact discussion we've had in openstack about what to do with third party drivers and other things that we can't test in CI as it relates to how it affects developer's workflow | 12:53 |
| mordred | krotscheck: indeed | 12:53 |
| krotscheck | Imagine if python was a shim on top of multiple engines. | 12:53 |
| mordred | krotscheck: although I hear that they have turned a new leaf and stuff | 12:53 |
| mordred | right? | 12:53 |
| AJaeger | project-config cores, could you review https://review.openstack.org/#/c/269261/ , please? | 12:53 |
| mordred | well | 12:53 |
| mordred | krotscheck: we did try for a while putting pypy into the gate | 12:54 |
| mordred | which is a different implementation of python | 12:54 |
| mordred | krotscheck: it ... didn't fully work as well as you might hope | 12:54 |
| mordred | even though pypy is something that developers _can_ run locally on linux too | 12:54 |
| *** markvoelker has quit IRC | 12:54 | |
| mordred | most of them just simply did not care | 12:54 |
| krotscheck | I heard that worked out really super incredibly well. | 12:54 |
| *** ddieterly has joined #openstack-infra | 12:55 | |
| *** notnownikki has joined #openstack-infra | 12:55 | |
| * krotscheck is debating commenting on the pull request. | 12:55 | |
| mordred | you know what did work out super incredibly well? the veal tartare with duck egg I had about a month ago in montreal. mmmmmmm | 12:55 |
| *** _amrith_ is now known as amrith | 12:56 | |
| krotscheck | That sounds amazing. | 12:56 |
| * Clint perks up. | 12:56 | |
| mordred | Joe Beef | 12:56 |
| mordred | I highly recommend eating at Joe Beef | 12:56 |
| *** abregman|afk has quit IRC | 12:56 | |
| * krotscheck is going to make his coffee mug work out super incredibly well by filling it with coffee. | 12:56 | |
| mordred | I also highly recommend eating at Garde Manger while we're on the subject | 12:56 |
| AJaeger | mordred, thanks for reminding me that the quiche is still not finished for lunch... | 12:57 |
| mordred | AJaeger: seems like it should perhaps be finished | 12:57 |
| krotscheck | mordred: Thanks for +2'ing those | 12:58 |
| *** sdake has joined #openstack-infra | 12:58 | |
| Zara | note to anyone using storyboard: we believe email notifications should work as soon as infra's email config is updated. there are patches in review for that here. https://review.openstack.org/#/c/270331/ and https://review.openstack.org/#/c/270329/ . we think the work is completed on the storyboard side. | 12:58 |
| *** ppai has quit IRC | 12:58 | |
| *** ddieterly has quit IRC | 12:59 | |
| *** dims is now known as dimsum__ | 13:00 | |
| *** vgridnev has quit IRC | 13:01 | |
| *** alaski is now known as lascii | 13:01 | |
| *** claudiub has joined #openstack-infra | 13:01 | |
| krotscheck | mordred: I... _think_... I am in favor of the nodejs patch, because yay open source involvement, and it puts pressure on Apple/Mozilla to contribute with their JS things more. Practically speaking, I think it will be relevant to some microsofties and NOBODY ELSE. | 13:02 |
| *** baoli has quit IRC | 13:03 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Always go to Story Detail Page after adding Story https://review.openstack.org/270337 | 13:04 |
| *** fhubik is now known as fhubik_brb | 13:04 | |
| mordred | Zara: in that first one- it needs to be updated - /path/to/storyboard is not a real path :) | 13:04 |
| *** daemontool_ has quit IRC | 13:04 | |
| mordred | krotscheck: I totally agree | 13:04 |
| mordred | krotscheck: it's another brick in the "what does it mean to do open source on a non-free platform" | 13:05 |
| dstufft | PyPy is great stuff, wish their 3.x support wasn't ancient though :[ | 13:05 |
| mordred | dstufft: ++ | 13:05 |
| mordred | krotscheck: which has come up again in openstack form with the solaris zones based nova driver meets defcore discussion | 13:06 |
| krotscheck | Solaris! I remember yelling at one of their devs. | 13:06 |
| krotscheck | Though I really shouldn't blame him for OpenSolaris going away. | 13:06 |
| openstackgerrit | Merged openstack-infra/puppet-bandersnatch: Make bandersnatch::httpd class https://review.openstack.org/270486 | 13:06 |
| krotscheck | WOOT ^^ | 13:07 |
| mordred | krotscheck: since we cannot test the changes needed to tempest to use non-linux tools inside of the guest to verify the networking has been plumbed through, it makes it very hard to allow them to certify a solaris-running openstack | 13:07 |
| krotscheck | And thus it begins. | 13:07 |
| SotK | mordred: I was hoping that it would just fail to deliver mail to storyboard properly if /path/to/storyboard doesn't exist, especially since we don't have anything to actually call in order to inject mail yet | 13:07 |
| mordred | krotscheck: yeah - opensolaris would have made it a much different discussion | 13:07 |
| dstufft | oh | 13:07 |
| dstufft | heh | 13:07 |
| mordred | SotK: ah. it might | 13:07 |
| *** mrmartin has quit IRC | 13:07 | |
| mordred | SotK: I did not realize that that thing was so far an aspirational thing | 13:07 |
| mordred | dstufft: dude. our new pypi mirror is going to be so awesome | 13:08 |
| mordred | dstufft: or it's going to suck really hard | 13:08 |
| krotscheck | Any non-mordred cores able to look at a doc patch? https://review.openstack.org/#/c/270559/ | 13:08 |
| dstufft | probably when I migrate PyPI to using /packages/<some sha256 hash>/foobar-1.0.tar.gz I should I guess split that up so /packages/ doesn't end up with half a million directories I guess | 13:08 |
| mordred | dstufft: it's tough to say until it exists | 13:08 |
| *** julim has joined #openstack-infra | 13:08 | |
| mordred | dstufft: yes please | 13:08 |
| *** mrmartin has joined #openstack-infra | 13:08 | |
| *** kushal has joined #openstack-infra | 13:08 | |
| krotscheck | I'm a little worried about the initial request lag against our new AFS mirrors. | 13:09 |
| mordred | krotscheck: yah - that's one of the things I'm curious about in practice | 13:09 |
| mordred | krotscheck: I think with warm caches that'll be just fine | 13:09 |
| krotscheck | mordred: Any way we can smoketest that? | 13:09 |
| mordred | krotscheck: but with cold caches ... | 13:09 |
| mordred | krotscheck: yup- we can push a patch to devstack that overwrites the pip.conf before it does anything | 13:09 |
| *** piet has quit IRC | 13:10 | |
| Zara | mordred: sorry, yeah, I should've been clearer that I was using 'completed' to refer to the work sending out notifications (that's our scope so far). we've still got a way to go with replying to them. | 13:10 |
| SotK | indeed, the email notifications are currently pretty basic as a first-pass at them, extra usefulness will come once they at least work a little bit :) | 13:10 |
| *** piet has joined #openstack-infra | 13:10 | |
| Zara | heh snap | 13:10 |
| mordred | Zara, SotK: ++ | 13:10 |
| *** matrohon has joined #openstack-infra | 13:10 | |
| krotscheck | mordred: Yeah, two use cases there. Will it fail when a new thing is requested, and/or will it fail with a completely cold cache. | 13:10 |
| mordred | yup | 13:10 |
| * krotscheck suspects that AFS will handle it reasonably well, but that pip might timeout a lot to begin with | 13:11 | |
| *** dtantsur is now known as dtantsur|brb | 13:11 | |
| dstufft | you can increase pip's timeout | 13:11 |
| *** fhubik_brb is now known as fhubik | 13:11 | |
| mordred | yah. that'll be a thing we just have to learn how to tune and stuff | 13:11 |
| *** rhallisey has joined #openstack-infra | 13:12 | |
| mordred | krotscheck: I also cannot remember how vos release affects blocks that did not change in the new version | 13:12 |
| mordred | I _think_ it does not invalidate them from the cache | 13:12 |
| mordred | but I could be wrong | 13:12 |
| mordred | I suppose that would be fairly easy to test | 13:12 |
| krotscheck | mordred: I recognize that you said words. | 13:12 |
| *** weshay_xchat has joined #openstack-infra | 13:12 | |
| *** weshay_xchat is now known as weshay | 13:13 | |
| mordred | because if it does not invalidate the cache on non-changed blocks, then we're not talking about a huge amount of cold-cache after each release | 13:13 |
| *** yamamoto has joined #openstack-infra | 13:14 | |
| krotscheck | mordred: Given how old AFS is, I suspect they had to not-invalidate non-changed blocks simply by virtue of hardware and network constraints. | 13:14 |
| *** baoli has joined #openstack-infra | 13:14 | |
| *** esker has joined #openstack-infra | 13:15 | |
| *** ccarmack has joined #openstack-infra | 13:16 | |
| *** mrmartin has quit IRC | 13:16 | |
| *** vgridnev has joined #openstack-infra | 13:16 | |
| *** e0ne has quit IRC | 13:16 | |
| *** kgiusti has joined #openstack-infra | 13:17 | |
| *** e0ne has joined #openstack-infra | 13:17 | |
| mordred | krotscheck: yah | 13:18 |
| *** vgridnev has quit IRC | 13:19 | |
| *** Guest72124 has quit IRC | 13:19 | |
| *** edmondsw has joined #openstack-infra | 13:20 | |
| *** esker has quit IRC | 13:20 | |
| *** fhubik is now known as fhubik_brb | 13:20 | |
| *** baoli has quit IRC | 13:20 | |
| *** baoli has joined #openstack-infra | 13:21 | |
| *** baoli has quit IRC | 13:21 | |
| *** vgridnev has joined #openstack-infra | 13:24 | |
| openstackgerrit | Merged openstack-infra/puppet-bandersnatch: Make bandersnatch::cron class https://review.openstack.org/270498 | 13:25 |
| openstackgerrit | Merged openstack-infra/system-config: Add mirror_update.openstack.org https://review.openstack.org/270520 | 13:25 |
| krotscheck | woooo | 13:25 |
| krotscheck | We can create the mirror master! | 13:26 |
| krotscheck | Any infra-root out there able to create mirror_update.openstack.org for us mirror afficionado's? | 13:26 |
| openstackgerrit | Merged openstack-infra/system-config: Add mirror.<region>.openstack.org https://review.openstack.org/238754 | 13:26 |
| krotscheck | (And maybe one of the mirror slaves so we can start testing in earnest? ;) | 13:26 |
| *** bhunter71 has joined #openstack-infra | 13:27 | |
| * krotscheck runs off to rebase all the things | 13:27 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Added extra-index-url to pip optional settings https://review.openstack.org/267751 | 13:28 |
| krotscheck | Question. The wheel mirror slaves will need to be able to write to the AFS share. Should that be a thing that is an inherent thing for the wheel slave - i.e. the job just copies things into the directory and runs vos release - or is that a thing that should be part of the job, i.e. `scp to mirror_update` and then figure out how to run vos release remotely | 13:31 |
| *** malini has joined #openstack-infra | 13:31 | |
| freerunner | Hi folks! Could you help me with review and merge this one https://review.openstack.org/#/c/270770/ , please? | 13:31 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: make releasenotes use zuul cloner with requirements https://review.openstack.org/271284 | 13:31 |
| krotscheck | If the former, I'm going to have to unwind slave.pp, because I don't think we want all of our slaves to be AFS enabled. | 13:31 |
| krotscheck | If the latter, ssh command vos release makes me leery | 13:32 |
| *** jcoufal has joined #openstack-infra | 13:32 | |
| hashar | Zara: storyboard email notification sounds like a nice feature to have :-} | 13:32 |
| *** malini1 has joined #openstack-infra | 13:33 | |
| hashar | mordred: seems you were speaking of pypi mirror, someone complained a few hours ago about one of the mirror missing a package | 13:33 |
| hashar | mordred: do you have any status page for how up-to-date mirrors are? | 13:33 |
| pavel_bondar | I am observing that pypi mirror is out of sync, created story for that (not sure I reported it into right place): https://storyboard.openstack.org/#!/story/2000470 can someone take a look on it? | 13:34 |
| hashar | mordred: ah that was pavel_bondar :-} ^^ | 13:34 |
| pavel_bondar | hashar: yeah, I still see at least one mirror out of sync | 13:34 |
| hashar | Ubuntu has some nice status pages for the repositories mirror. Not random example https://launchpad.net/ubuntu/+mirror/ubuntu.wikimedia.org-archive | 13:35 |
| *** jordanP has joined #openstack-infra | 13:35 | |
| *** malini has quit IRC | 13:36 | |
| *** treaki has joined #openstack-infra | 13:36 | |
| krotscheck | I think we all want a pretty status page. I don't think any of us have the time to build one, but you're welcome to submit patches. | 13:36 |
| krotscheck | As for the out-of-date mirror, that's the HPE region, which is going away in... a week? | 13:37 |
| krotscheck | SO it might just be ok for us to go manually update that one package. | 13:37 |
| krotscheck | Lastly: Soon there will be one mirror to rule them all. | 13:37 |
| *** iremizov has quit IRC | 13:37 | |
| *** markvoelker has joined #openstack-infra | 13:38 | |
| mordred | looking | 13:38 |
| openstackgerrit | Dina Belova proposed openstack-infra/project-config: Add py27, py34 and pep8 jobs for performance-docs https://review.openstack.org/270088 | 13:39 |
| Zara | hashar: I agree! :) it should make it much easier to track things. | 13:40 |
| *** iremizov has joined #openstack-infra | 13:41 | |
| hashar | Zara: I myself wasn't tracking anything :-) but then I only have a couple bugs floating around | 13:41 |
| *** malini1 has left #openstack-infra | 13:41 | |
| *** amrith is now known as _amrith_ | 13:41 | |
| pavel_bondar | krotscheck: updating single infoblox-client package in mirror would work for me, because right now absence of new infoblox-client causes jenkins failure for all new networking-infoblox patches | 13:42 |
| *** dtantsur|brb is now known as dtantsur | 13:42 | |
| krotscheck | pavel_bondar: It's only for nodes in that region though, yes? | 13:42 |
| pavel_bondar | krotscheck: I don't know | 13:43 |
| *** markvoelker has quit IRC | 13:43 | |
| pavel_bondar | krotscheck: where can I get full list of mirrors to check, if the are out of date? | 13:43 |
| pavel_bondar | I have found just two mirrors for now (listed in story), and one was out of date | 13:44 |
| *** ildikov has quit IRC | 13:44 | |
| krotscheck | pavel_bondar: Sec, lemme find 'em for ya. | 13:44 |
| krotscheck | pavel_bondar: http://git.openstack.org/cgit/openstack-infra/system-config/tree/hiera/common.yaml#n275 | 13:45 |
| *** alkhodos_ has joined #openstack-infra | 13:45 | |
| *** tdurakov is now known as Guest62753 | 13:45 | |
| krotscheck | pavel_bondar: The mirror entries you see below the pyp entries will be taking over the jobs, but they don't exist yet. | 13:45 |
| alkhodos_ | Hi, everyone. I have a problem with my CI: it sends gerrit review command with results which can be seen in review comments, but for some reason the results don't appear in the CI section. Example of this can be found here: https://review.openstack.org/#/c/262024/7. (NexentaCI) Any idea what is the issue? | 13:46 |
| *** tlian has joined #openstack-infra | 13:47 | |
| Zara | hashar: heh, I've actually adjusted to using the dashboard recent events, myself! but emails should work better for people who aren't glued to storyboard all day, so I'm glad we're nearly there. :) | 13:47 |
| *** annegentle has joined #openstack-infra | 13:48 | |
| sdague | mordred: tripped over a zuul cloner issue with requirements changes. There is a work around here - https://review.openstack.org/#/c/271257/ that would be nice to land. | 13:48 |
| *** bgaifullin has joined #openstack-infra | 13:48 | |
| bgaifullin | Hi all. Folks please help. I has been created a new project in openstack namespace and I need help with adding me to core reviewers group. | 13:49 |
| bgaifullin | the gerrit group is packetary-core | 13:50 |
| bgaifullin | https://review.openstack.org/#/admin/groups/1251,members | 13:50 |
| *** sdake has quit IRC | 13:51 | |
| *** jswarren has joined #openstack-infra | 13:51 | |
| pavel_bondar | krotscheck: I have checked all starting with 'pypi.', only one of them is out of sync(http://pypi.region-b.geo-1.openstack.org/), it mentioned in story. And pypi.openstack.org look like not a pypi mirror, since it just redirects to http://docs.openstack.org/infra/system-config/ | 13:51 |
| *** zhurong has joined #openstack-infra | 13:52 | |
| krotscheck | Yeah, that one doesn't exist anymore | 13:52 |
| * krotscheck should remove that. | 13:52 | |
| fungi | mordred: did you get a chance to look into why the constraints everywhere revert from yesterday didn't get applied when you were updating job configs on the jenkins masters? | 13:52 |
| fungi | if not, i'll start digging into it now | 13:53 |
| *** fhubik_brb is now known as fhubik | 13:55 | |
| *** doug-fish has joined #openstack-infra | 13:55 | |
| *** ddieterly has joined #openstack-infra | 13:55 | |
| *** baoli has joined #openstack-infra | 13:55 | |
| *** esker has joined #openstack-infra | 13:56 | |
| *** baoli_ has joined #openstack-infra | 13:56 | |
| *** zhurong has quit IRC | 13:57 | |
| fungi | krotscheck: pypi.region-b.geo-1.openstack.org dos still exist. that's the hpcloud pypi mirror (for another week anyway) | 13:58 |
| fungi | er, does | 13:58 |
| krotscheck | fungi: Yeah - I was referring to pypi.openstack.org | 13:58 |
| krotscheck | And its entry in hiera | 13:58 |
| fungi | oh | 13:58 |
| fungi | yeah, that was a vhost on static.o.o | 13:59 |
| *** markvoelker has joined #openstack-infra | 13:59 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Removed pypi.openstack.org from hiera.yaml https://review.openstack.org/271307 | 14:00 |
| fungi | and we dropped it from the apache config there, so it's falling back to the default vhost which is itself now a redirect to docs.openstack.org apparently | 14:00 |
| krotscheck | fungi: ^^ | 14:00 |
| *** baoli has quit IRC | 14:00 | |
| *** ddieterly has quit IRC | 14:00 | |
| *** markvoelker_ has joined #openstack-infra | 14:01 | |
| krotscheck | I need a +A on a 2x+2 -> https://review.openstack.org/#/c/267751/ | 14:01 |
| *** katyafervent is now known as katyafervent_awa | 14:01 | |
| *** erikwilson has joined #openstack-infra | 14:02 | |
| *** erikwilson has quit IRC | 14:03 | |
| *** mase_x200 has quit IRC | 14:03 | |
| *** katyafervent_awa is now known as katyafervent | 14:03 | |
| *** esker has quit IRC | 14:03 | |
| *** markvoelker has quit IRC | 14:04 | |
| fungi | mordred: at least the ceilometer-tarball job (as an example) seems to be reverted back to gerrit-git-prep in xml on all our masters now. i'll dig deeper on what failures jhesketh was encountering | 14:04 |
| *** sriram has joined #openstack-infra | 14:06 | |
| *** esker has joined #openstack-infra | 14:06 | |
| *** dslev_ has joined #openstack-infra | 14:06 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Add output of top to hostinfo https://review.openstack.org/270139 | 14:07 |
| *** bcourt has joined #openstack-infra | 14:07 | |
| fungi | hrm, jhesketh didn't link to specific evidence, just claimed the jobs didn't _seem_ to be updated (which my research contradicts), so i guess i'll just retry and see what happens | 14:07 |
| *** Ramanjaneya has quit IRC | 14:08 | |
| asselin_ | alkhodos_, seems you don't match the necessary regex. looks like you're missing "Build succeeded. " | 14:08 |
| *** daemontool has joined #openstack-infra | 14:08 | |
| asselin_ | alkhodos_, http://docs.openstack.org/infra/system-config/third_party.html#posting-result-to-gerrit | 14:09 |
| *** maishsk has quit IRC | 14:09 | |
| sdague | fungi: requirements patches blocked until - https://review.openstack.org/#/c/271257/ lands | 14:10 |
| fungi | sdague: i didn't realize the requirements tox jobs were still using zuul-cloner | 14:11 |
| fungi | shouldn't that have stopped after we reverted your constraints-everywhere change got reverted? | 14:12 |
| * fungi still isn't sentencing well. needs more coffee | 14:12 | |
| *** dkranz has joined #openstack-infra | 14:12 | |
| *** daemontool_ has joined #openstack-infra | 14:12 | |
| *** daemontool has quit IRC | 14:13 | |
| mordred | fungi: sentences are hard | 14:15 |
| fungi | pavel_bondar: the mirror in question keeps filling up its root filesystem because one day of apache access logs for it is larger than that entire disk. i've cleaned it up enough for the mirror updates to start again for now | 14:15 |
| *** amitgandhinz has joined #openstack-infra | 14:16 | |
| *** amitgandhinz has quit IRC | 14:17 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Point nodepool slaves at new pypi mirror location. https://review.openstack.org/238756 | 14:17 |
| *** amitgandhinz has joined #openstack-infra | 14:17 | |
| *** elmiko has joined #openstack-infra | 14:17 | |
| *** sabeen1 has joined #openstack-infra | 14:18 | |
| elmiko | hi, it appears that our meetings have gotten out of sync with the ICS file at http://eavesdrop.openstack.org/#API_Working_Group , is there a process for correcting this? | 14:18 |
| krotscheck | jeblair: As soon as you come online, the mirror_update and mirror.<region> patches landed. | 14:18 |
| pavel_bondar | fungi: Thanks! | 14:18 |
| sdague | fungi: it got reverted? | 14:18 |
| krotscheck | The next step is to provision nodes :) | 14:18 |
| krotscheck | fungi: Wow. That's.... a lot of logs. | 14:19 |
| *** bmwiedemann has joined #openstack-infra | 14:19 | |
| fungi | sdague: i pinged you a few times while we were working through it | 14:19 |
| *** rlandy has joined #openstack-infra | 14:19 | |
| *** exploreshaifali has quit IRC | 14:19 | |
| sdague | oh, well I was offline, didn't notice | 14:19 |
| sdague | ok | 14:19 |
| fungi | sdague: zuul-cloner is only safe for use in change pipelines. it completely broke the release jobs so we're still trying to fix that | 14:19 |
| sdague | ok | 14:20 |
| fungi | ended up with a bunch of stable point releases built from master branch tips | 14:20 |
| sdague | so, I guess it's just taking a *long* time to propogate these changes out to nodes | 14:20 |
| *** sabeen3 has joined #openstack-infra | 14:20 | |
| elmiko | nvm, found it | 14:21 |
| sdague | because I got a recheck passing this morning on the tox target that changed this | 14:21 |
| sdague | and the requirements jobs are failing now | 14:21 |
| fungi | sdague: yeah, that's what i'm trying to look into now, so we can rerun jobs to fix the broken release artifacts | 14:21 |
| *** akshai has joined #openstack-infra | 14:21 | |
| fungi | it looks to me like jobs did update | 14:21 |
| *** mriedem has joined #openstack-infra | 14:22 | |
| fungi | though also that change altered something like 95% of our 8k jobs, so relying on automated updates of jenkins-job-builder doesn't work. needs manual babysitting because it takes hours | 14:22 |
| *** sabeen1 has quit IRC | 14:22 | |
| fungi | and gets killed by exec safety timeouts or dies on transient jenkins api errors | 14:22 |
| sdague | https://review.openstack.org/#/c/267096/3 just returned with test results | 14:23 |
| sdague | pep8 passing on that means it's still the constraints version out there | 14:23 |
| jhesketh | fungi: just doing a drop by, but the bottom of the screen sessions on the jenkins masters is what made me think that they didn't work | 14:23 |
| jhesketh | fungi: that coupled with the release job still failing | 14:23 |
| *** peristeri has joined #openstack-infra | 14:23 | |
| krotscheck | The AFS mirrors effort has a few low-impact reviews, any takers? - 267751, 271307, 270559 | 14:23 |
| fungi | jhesketh: which release job did you try? ceilometer? it looks like teh ceilometer-tarball job xml on all the masters uses gerrit-git-prep again | 14:24 |
| fungi | jhesketh: was about to try retriggering so i can have some evidence to look at | 14:24 |
| jhesketh | yes, but I didn't look into why it failed | 14:24 |
| jhesketh | the first try it landed on jenkins02 which had a proxy error in the screen | 14:24 |
| fungi | oh, if it was ceilometer then i should be able to find logs | 14:24 |
| jhesketh | the second time it was on different jenkins' | 14:24 |
| fungi | yeah, http://logs.openstack.org/02/02c49c1530aab42044ecddcaf6d5502eff44c961/release/ceilometer-tarball/d505d3f/console.html is still using zuul-cloner | 14:25 |
| fungi | ran on jenkins05 | 14:26 |
| *** baoli_ has quit IRC | 14:26 | |
| fungi | however, last modified time on /var/lib/jenkins/jobs/ceilometer-tarball/config.xml for jenkins05 was 12:16 utc | 14:27 |
| *** kaisers1 has quit IRC | 14:27 | |
| fungi | checking to see when jenkins registered the actual diff. maybe it was applied after that ran (via puppet catching up?) | 14:27 |
| *** kaisers has joined #openstack-infra | 14:27 | |
| openstackgerrit | Dougal Matthews proposed openstack-infra/project-config: Retire the Tuskar project https://review.openstack.org/270869 | 14:28 |
| tonyb | So can anyone explain why http://logs.openstack.org/89/266189/6/check/gate-requirements-pep8/bd0c412/console.html ran zuul-cloner? I though that was fixed with https://review.openstack.org/#/c/271102/ merging ? | 14:28 |
| tonyb | did I misundersatnd something? | 14:28 |
| fungi | tonyb: we're looking into it. the change which introduced that was reverted yesterday but doesn't seem to have propagated | 14:29 |
| *** regXboi has joined #openstack-infra | 14:29 | |
| fungi | or at least hasn't entirely | 14:29 |
| tonyb | fungi: Ahh okay. | 14:29 |
| openstackgerrit | Michael McCune proposed openstack-infra/irc-meetings: Adjust api-wg meeting time to match reality https://review.openstack.org/271321 | 14:30 |
| krotscheck | Clint: Thanks for those reviews :) | 14:30 |
| *** esker has quit IRC | 14:31 | |
| *** rossella_s has quit IRC | 14:31 | |
| fungi | jhesketh: jenkins05 was updated at 11:15 utc to have the needed change to the ceilometer-tarball job, but the last job you triggered ran about 1.5 hours before that | 14:31 |
| tonyb | fungi: can I ask for a ping when you guys have it fixed? to save randomly rechecking? | 14:31 |
| alkhodos_ | asselin_: thanks for your reply! Did something change lately to make it necessary ? Why I'm asking is my CI was running fine and today (or maybe yesterday) stopped posting. Anyway thanks, I will add that regex. | 14:31 |
| *** akshai has quit IRC | 14:31 | |
| fungi | tonyb: sure, hopefully we'll have more people looking into it shortly | 14:31 |
| tonyb | fungi: Thanks. | 14:31 |
| *** rossella_s has joined #openstack-infra | 14:31 | |
| *** fhubik is now known as fhubik_brb | 14:32 | |
| *** fhubik_brb is now known as fhubik | 14:32 | |
| jhesketh | fungi: right, so perhaps it was after puppet did a thing, since I checked all the masters before running the job and they had finished in their screens | 14:32 |
| asselin_ | alkhodos_, not that I'm aware of...that wiki page is under version control if you want to track any changes. | 14:32 |
| fungi | jhesketh: oh yep, looking at mordred's old screen session on jenkins05, it did indeed not complete. it aborted on a jenkins api failure | 14:33 |
| bgaifullin | Folks, could anyone add bgaifullin@mirantis.com to group packetary-core (https://review.openstack.org/#/admin/groups/1251,members) | 14:34 |
| jhesketh | fungi: all the masters looked like that, but I couldn't tell from the output if it was just one job that failed to update or all jobs | 14:34 |
| jhesketh | (I chcked the code and it only raises the first exception) | 14:34 |
| jhesketh | fungi: so I'm not sure what the standard puppet run does/did differently | 14:34 |
| Clint | krotscheck: i like tiny patches | 14:34 |
| jhesketh | probably worth figuring out though | 14:34 |
| fungi | jhesketh: so we're probably just incrementally rolling out the revert a little at a time as puppet runs a bit of it and then gets killed by another timeout | 14:34 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Pass timeout through to floating ip creation https://review.openstack.org/270193 | 14:34 |
| openstackgerrit | Michael McCune proposed openstack-infra/irc-meetings: Adjust api-wg meeting time to match reality https://review.openstack.org/271321 | 14:35 |
| dmsimard | Hi, question about Governance. It was my understanding that http://git.openstack.org/cgit/openstack/governance/plain/reference/projects.yaml was a good reference. I do not see aodh, ceilometer or gnocchi in there but I was told they are in governance regardless. Am I not looking at the right place ? | 14:35 |
| *** xyang1 has joined #openstack-infra | 14:35 | |
| dmsimard | Oh, man, they are under telemetry | 14:36 |
| * dmsimard sighs | 14:36 | |
| *** iremizov has quit IRC | 14:36 | |
| fungi | dmsimard: yes, the ceilometer team changed its name to telemetry recently | 14:36 |
| krotscheck | elmiko: Oooh, yeah, I need to start showing to those meetings . | 14:36 |
| *** sdake has joined #openstack-infra | 14:37 | |
| elmiko | krotscheck: \o/ | 14:37 |
| elmiko | krotscheck: really appreciated your comments on the pagination review, thanks! | 14:37 |
| dmsimard | fungi: *nods* I knew that but I sort of expected all three projects to be at the "top" level like all the others | 14:37 |
| *** daemontool_ has quit IRC | 14:37 | |
| krotscheck | elmiko: You're welcome! It kinda felt like rehashing a can of worms though. | 14:37 |
| elmiko | krotscheck: yea... cdent and i agreed that we need pay more attention to end-users who may be consuming these APIs to create UIs | 14:38 |
| pavel_bondar | fungi: infoblox-client is synced to that mirror, so my jenkins jobs are passing again. Thanks a lot! | 14:39 |
| krotscheck | elmiko: well, I'm trying to convince piet to appoint me API-WG x-project liason for the ux program. | 14:39 |
| *** dslev_ has quit IRC | 14:39 | |
| krotscheck | elmiko: And by "Trying to convince" I really mean "this is the first time I had this idea" | 14:39 |
| elmiko | krotscheck: haha, sweet! | 14:39 |
| alkhodos_ | asselin_: I tried with "Build suceeded. " - still doesn't go through... And another thing is the same command works fine on sandbox. | 14:40 |
| *** zeih has quit IRC | 14:40 | |
| *** ddieterly has joined #openstack-infra | 14:40 | |
| piet | Heard my name... | 14:40 |
| asselin_ | alkhodos_, link to working and not working please | 14:40 |
| fungi | dmsimard: yeah, "project" in the governance sense is short for "project team" and usually has multiple deliverables maintained by that team | 14:40 |
| piet | krotscheck can we chat later? | 14:40 |
| alkhodos_ | https://review.openstack.org/#/c/262024/7 - not working. https://review.openstack.org/#/c/265474 - working sandbox | 14:41 |
| krotscheck | piet: Yep | 14:41 |
| *** iremizov has joined #openstack-infra | 14:41 | |
| krotscheck | piet: Before noon PST, that's when I shift into daycare mode. | 14:42 |
| piet | krotscheck no problem. I just need to get tabs for my truck this morning. Wish me luck... | 14:42 |
| krotscheck | piet: Go forth and battle with the DMV | 14:43 |
| asselin_ | alkhodos_, for https://review.openstack.org/#/c/262024/7 you need to post on patch set 8, the latest patch | 14:45 |
| *** ari has joined #openstack-infra | 14:45 | |
| alkhodos_ | asselin_: so whenever there is a new patchset, the previous one cannot be reviewed anymore? Didn't know that. | 14:46 |
| *** pcaruana has joined #openstack-infra | 14:46 | |
| asselin_ | alkhodos_, they can be reviewed. just that they don't show up in the table at the top. You can see them by scrolling to the bottom of the page and hit the 'toggle ci' button. | 14:47 |
| *** ildikov has joined #openstack-infra | 14:47 | |
| alkhodos_ | asselin_: oh, ok thank you! | 14:47 |
| fungi | jhesketh: mordred: updates on all the jenkins masters except jenkins.o.o and jenkins01 did not complete successfully and crashed out at various places (so 02-07). i've restarted them all in the existing screen sessions | 14:48 |
| fungi | i've got them all open in terminals and will keep tabs on them | 14:48 |
| dmsimard | I am trying to leverage a generic approach to recovering service and diagnostic logs for CI jobs and I understand this might warrant a spec. If you have a few minutes, could you provide your feedback on https://review.openstack.org/#/c/270004 ? (See https://review.openstack.org/#/q/topic:puppet-and-packstack-jobs for a wider context ) | 14:49 |
| fungi | hopefully we'll have this revert propagated in the next few hours | 14:49 |
| dmsimard | AJaeger: ^ | 14:49 |
| vponomaryov | Guys, there is some diff between "rax" nodes and all other, package "kpartx" is not available on "all other". Observed by results in https://review.openstack.org/#/c/271309/ and https://review.openstack.org/#/c/271297/ | 14:50 |
| *** acruz has joined #openstack-infra | 14:50 | |
| krotscheck | Current AFS-mirror patches open (all ready, WIP'd to prevent accidental merging) https://review.openstack.org/#/q/topic:afs-mirror+status:open+branch:master | 14:50 |
| fungi | vponomaryov: this does not surprise me--they're built in slightly different ways in rackspace (at least the devstack images) | 14:50 |
| fungi | vponomaryov: we're trying to work through issues using glance in rackspace so that we can upload the same images there we use in all our other providers | 14:51 |
| vponomaryov | fungi: but rax is the one that works | 14:51 |
| vponomaryov | fungi: all other fails | 14:51 |
| vponomaryov | s/fails/fail/ | 14:52 |
| fungi | vponomaryov: is kpartx used in some job? if so, we need to make sure it gets installed. it's probably preinstalled by rackspace and we use their base images because, as i said, problems with using their glance still | 14:52 |
| *** dslev_ has joined #openstack-infra | 14:52 | |
| *** yamamoto has quit IRC | 14:52 | |
| *** arxcruz has quit IRC | 14:52 | |
| vponomaryov | fungi: it is dependency of other deps | 14:53 |
| vponomaryov | fungi: not direct | 14:53 |
| *** jsavak has joined #openstack-infra | 14:53 | |
| fungi | vponomaryov: i'm guessing whatever change merged to start depending on that package happened to run on rackspace nodes and so snuck through? how long has it been in use? | 14:53 |
| vponomaryov | fungi: day or two | 14:53 |
| vponomaryov | fungi: correct date will be able to provide a little bit later | 14:54 |
| fungi | vponomaryov: what jobs are you seeing this hit? | 14:54 |
| vponomaryov | fungi: "gate-manila-buildimage-generic" | 14:54 |
| vponomaryov | fungi: used by "manila-image-elements" project | 14:54 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Add release note for new get_object() API call https://review.openstack.org/271346 | 14:56 |
| *** markvoelker has joined #openstack-infra | 14:57 | |
| fungi | vponomaryov: looks like you're using dib elements for that. for reusability you should probably add that dependency explicitly? maybe in elements/manila-ubuntu-core/package-installs.yaml | 14:58 |
| fungi | vponomaryov: you say it's an indirect dependency--what's actually using it in the job? | 14:58 |
| vponomaryov | fungi: hard to say, first guess was "sphinx"-deps, but then got know it is not | 14:59 |
| *** zeih has joined #openstack-infra | 14:59 | |
| fungi | ideally whatever uses it should declare appropriate dependencies and make sure they're installed | 14:59 |
| *** kushal has quit IRC | 15:00 | |
| fungi | once we get our glance issues in rackspace sorted out, kpartx won't be preinstalled there either | 15:00 |
| vponomaryov | fungi: ok, I'll what I can do about "dib elements" | 15:00 |
| vponomaryov | fungi: thank you | 15:00 |
| vponomaryov | s/I'll/I'll see/ | 15:00 |
| fungi | so it was just an unlucky break that you ended up growing an unstated dependency on a package one of our service providers thought was neat to include on their custom images | 15:00 |
| *** markvoelker_ has quit IRC | 15:01 | |
| *** mhickey has quit IRC | 15:02 | |
| gordc | hi, was wondering does anyone know why oslo part works but not client part in the following: https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/ceilometer.yaml#L247-L251 | 15:02 |
| *** jsavak has quit IRC | 15:03 | |
| *** jsavak has joined #openstack-infra | 15:03 | |
| gordc | for reference it complains for client gate http://logs.openstack.org/08/235208/3/check/gate-ceilometer-dsvm-integration/a637ac7/logs/devstacklog.txt.gz#_2016-01-22_03_37_41_358 but doesn't in oslo gate. | 15:04 |
| *** esker has joined #openstack-infra | 15:04 | |
| *** burgerk has joined #openstack-infra | 15:04 | |
| *** sdake_ has joined #openstack-infra | 15:04 | |
| *** mrmartin has joined #openstack-infra | 15:04 | |
| *** _amrith_ is now known as amrith | 15:05 | |
| *** sdake has quit IRC | 15:06 | |
| *** ccrouch has joined #openstack-infra | 15:06 | |
| *** dfed717 has joined #openstack-infra | 15:06 | |
| *** dfed718 has joined #openstack-infra | 15:06 | |
| *** abregman has joined #openstack-infra | 15:06 | |
| *** dfed718 has quit IRC | 15:06 | |
| *** jpr has quit IRC | 15:07 | |
| *** dfed717 has quit IRC | 15:09 | |
| *** dfed717 has joined #openstack-infra | 15:09 | |
| *** rbrndt has joined #openstack-infra | 15:09 | |
| *** rbrndt has quit IRC | 15:10 | |
| *** placko_ has joined #openstack-infra | 15:10 | |
| openstackgerrit | Anne Gentle proposed openstack-infra/irc-meetings: Changes odd-weeks doc team meeting to 2000 Wednesday https://review.openstack.org/271361 | 15:10 |
| *** markvoelker has quit IRC | 15:12 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:13 | |
| *** abregman is now known as abregman|afk | 15:13 | |
| *** piet has quit IRC | 15:14 | |
| *** hdd has joined #openstack-infra | 15:14 | |
| *** annegentle has quit IRC | 15:14 | |
| *** mhickey has joined #openstack-infra | 15:14 | |
| AJaeger | dmsimard: top-level is projects, repos are one level lower | 15:14 |
| * dmsimard nods | 15:15 | |
| *** dfed717 has quit IRC | 15:16 | |
| *** abregman|afk is now known as abregman | 15:16 | |
| *** dfed717 has joined #openstack-infra | 15:16 | |
| *** zeih has quit IRC | 15:17 | |
| Daviey | fungi, mordred, jhesketh: Any news on that revert? Still missing 3 tarballs | 15:18 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Add support for command line job names https://review.openstack.org/235421 | 15:18 |
| *** jistr has quit IRC | 15:18 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Add support for periodic jobs https://review.openstack.org/271370 | 15:18 |
| mordred | Daviey: it's still running | 15:19 |
| Daviey | mordred: do we have an eta? | 15:19 |
| Daviey | mordred: i'm blocked on pushing out this point release atm. | 15:20 |
| *** dfed717 has quit IRC | 15:20 | |
| *** dfed717 has joined #openstack-infra | 15:21 | |
| markus_z | sdague: IIUC this was only a temporary workaround for our gates, wasn't it? https://review.openstack.org/#/c/158255/ | 15:21 |
| *** jsavak has quit IRC | 15:22 | |
| sdague | markus_z: maybe, however fc21 is no longer supported, so it could be deleted | 15:22 |
| *** tonytan4ever has joined #openstack-infra | 15:22 | |
| markus_z | sdague: ok, cool, I double-check with the author and push a change, thanks! | 15:22 |
| *** zz_dimtruck is now known as dimtruck | 15:22 | |
| sdague | fungi: any idea how the propogation of the constraints revert is going/ | 15:23 |
| sdague | ? | 15:23 |
| *** jsavak has joined #openstack-infra | 15:23 | |
| markus_z | afazekas: Do you see a reason to keep the rpm introduced in https://review.openstack.org/#/c/158255/ ? | 15:23 |
| *** abregman is now known as abregman|afk | 15:24 | |
| *** dfed717 has quit IRC | 15:24 | |
| openstackgerrit | Merged openstack-infra/shade: add the ability to get an object back from swift https://review.openstack.org/269925 | 15:26 |
| *** fhubik is now known as fhubik_brb | 15:26 | |
| *** baoli has joined #openstack-infra | 15:28 | |
| *** GheRivero has quit IRC | 15:28 | |
| AJaeger | fungi: zuul-git-branch-prep-upper-constraints is using zuul-cloner and thus should not run in the post pipeline, correcT? (on sdague's change https://review.openstack.org/#/c/271284/1/jenkins/jobs/openstack-publish-jobs.yaml ) | 15:29 |
| *** Rivero is now known as GheRivero | 15:29 | |
| *** achanda has joined #openstack-infra | 15:29 | |
| AJaeger | fungi: found it, no need to answer ;) | 15:30 |
| sdake_ | hey folks | 15:30 |
| sdake_ | i am tagging for erlease kolla on my mac | 15:30 |
| sdake_ | i get this error: | 15:30 |
| sdake_ | fatal: 'gerrit' does not appear to be a git repository | 15:30 |
| sdake_ | nm i just figured it out :) | 15:31 |
| sdake_ | thanks | 15:31 |
| AJaeger | thanks, sdake_ for helping sdake_ ! | 15:31 |
| sdake_ | fwiw, git review -s is needed before pushing tags ;) | 15:31 |
| *** abregman|afk is now known as abregman | 15:31 | |
| AJaeger | gordc: still around for your questoin? | 15:32 |
| AJaeger | gordc: see https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/ceilometer.yaml#L237 and read the comment you referenced - you need to add to PROJECTS, don't you? | 15:34 |
| openstackgerrit | gordon chung proposed openstack-infra/project-config: fix ceilometerclient gate https://review.openstack.org/271379 | 15:34 |
| AJaeger | brb | 15:34 |
| openstackgerrit | Ben Nemec proposed openstack/diskimage-builder: Revert "Use pip 7 for ironic" https://review.openstack.org/271380 | 15:34 |
| gordc | AJaeger: yeah, i just saw. i was trying to figure out what oslo.messaing didn't need it but i realised something else is adding it | 15:34 |
| *** achanda has quit IRC | 15:34 | |
| gordc | AJaeger: thanks for following up... patch is ^ | 15:35 |
| fungi | Daviey: sdague: AJaeger: yep, the revert rollout is still in the process of being applied to the jenkins masters which didn't get it last night, hasn't crashed on any of them yet. once i get a majority of them updated i'll put the other masters in prepare for shutdown so they won't run new jobs while i continue to hammer on them | 15:35 |
| fungi | usually takes between 60-90 minutes to complete _if_ it doesn't hit an api error and need to be resumed | 15:36 |
| *** piet has joined #openstack-infra | 15:36 | |
| *** bpokorny has quit IRC | 15:36 | |
| *** baoli has quit IRC | 15:38 | |
| smcginnis | fungi: Is that 60-90 minutes per jenkins master? | 15:38 |
| *** baoli has joined #openstack-infra | 15:39 | |
| mordred | smcginnis: it's in parallel | 15:40 |
| smcginnis | mordred: OK, good. ;) | 15:40 |
| *** mrmartin has quit IRC | 15:40 | |
| fungi | 02 just hit a 502 so it's not going to complete successfully and will need to be resumed in a bit | 15:42 |
| *** mhickey has quit IRC | 15:42 | |
| *** Nikolay_St has quit IRC | 15:43 | |
| fungi | but 02 is usually the really problematic one anyway. it may be best for me to go ahead and take it out of the pool now for the sake of expediency | 15:43 |
| *** markvoelker has joined #openstack-infra | 15:44 | |
| fungi | infra-root: i've put jenkins02 into prepare for shutdown and will cull any ready nodes for it in nodepool | 15:44 |
| *** woodster_ has joined #openstack-infra | 15:44 | |
| *** ddieterly has quit IRC | 15:46 | |
| *** iremizov has quit IRC | 15:47 | |
| *** annegentle has joined #openstack-infra | 15:47 | |
| *** ddieterly has joined #openstack-infra | 15:48 | |
| openstackgerrit | gordon chung proposed openstack-infra/project-config: fix ceilometerclient gate https://review.openstack.org/271379 | 15:48 |
| *** fhubik_brb is now known as fhubik | 15:49 | |
| *** iremizov has joined #openstack-infra | 15:50 | |
| sdague | fungi: so, my initial constraints patch that only did pep8 / unit tests jobs would have been the better one? | 15:50 |
| *** fhubik has quit IRC | 15:50 | |
| *** Nikolay_St has joined #openstack-infra | 15:50 | |
| AJaeger | gordc: we normally use export PROJECTS="openstack/{name} $PROJECTS" - could you use that? See my comment on your first chnage | 15:50 |
| *** yamahata has quit IRC | 15:52 | |
| fungi | sdague: finishing zuul-cloner so it has feature parity with gerrit-git-prep.sh (to be usable in changeless pipelines) would be better still. jesusaurus has already volunteered to look into that, but i'm sure he wouldn't reject help | 15:52 |
| *** esikachev has quit IRC | 15:52 | |
| sdague | well, that's beyond the number of contexts I can add right now | 15:52 |
| *** e0ne has quit IRC | 15:52 | |
| *** yamahata has joined #openstack-infra | 15:52 | |
| openstackgerrit | gordon chung proposed openstack-infra/project-config: fix ceilometerclient gate https://review.openstack.org/271379 | 15:52 |
| gordc | AJaeger: yep. just read it. updated :) ^ | 15:52 |
| fungi | sdague: i completely understand, and sympathize | 15:52 |
| *** e0ne has joined #openstack-infra | 15:53 | |
| fungi | sdague: all in all i think we want to un-revert your patch, we just discovered something we already knew but should have remembered we hadn't implemented yet | 15:53 |
| sdague | ok, well, I guess the question is whether we can do a lesser version of it for only the jobs we know works | 15:53 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: remove nova constraints named jobs https://review.openstack.org/271404 | 15:54 |
| *** EricGonczer_ has joined #openstack-infra | 15:54 | |
| AJaeger | sdague: and then wehther we should do it in steps so that fungi does not need to update jenkins manually since it touches 7000 jobs... | 15:54 |
| sdague | AJaeger: sure | 15:54 |
| *** mrmartin has joined #openstack-infra | 15:54 | |
| *** mhickey has joined #openstack-infra | 15:54 | |
| fungi | well, or just plan to actually be around to babysit the job update so we get it rolled out in a timely fashion | 15:54 |
| sdague | I guess I'll leave it for now, as it's a friday, and I'm not going to be around next week much with the nova midcycle | 15:55 |
| *** Nikolay_St has quit IRC | 15:55 | |
| *** fawadkhaliq has joined #openstack-infra | 15:57 | |
| *** dslev_ has quit IRC | 15:57 | |
| AJaeger | sdague: do you want to unblock gordc and review https://review.openstack.org/#/c/271379, please? | 15:57 |
| *** daemontool has joined #openstack-infra | 15:58 | |
| AJaeger | fungi, so any job that runs in periodic or post pipeline using constraints is broken now, correct? | 15:59 |
| *** tjones1 has joined #openstack-infra | 15:59 | |
| fungi | AJaeger: any job that runs in periodic, post, release, pre-release or tag (at a minimum) are probably making broken assumptions if they're relying on zuul-cloner | 16:00 |
| sdague | AJaeger: though that whole job is kind of wrong | 16:00 |
| sdague | because it links oslo.messaging to other jobs | 16:00 |
| sdague | even though it won't do the right things | 16:00 |
| AJaeger | sdague: could you explain gordc and myself, please? | 16:01 |
| fungi | if you're asking about docs jobs, for example, if a docs job was triggered in post by a merge to stable/liberty it will quite possibly end up with zuul-cloner generating and uploading docs for master as if they were stable/liberty instead | 16:01 |
| gordc | sdague: i believe the reason for adding that job to oslo.messaging was because the oslo team wanted a way to verify their releases before they sent it to pypi. | 16:02 |
| AJaeger | fungi, just reviewing layout.yaml, we have {name}-branch-tarball-constraints as post, we have {name}-docs-constraints as post and release, we have a template periodic-liberty-constraints,... | 16:02 |
| gordc | they had a few broken releases and ceilometer is always the first place to catch them. | 16:02 |
| fungi | AJaeger: yeah, whatever changes added those jobs were likely premature | 16:02 |
| AJaeger | we have {name}-tarball-constraints, ... | 16:03 |
| *** daemontool has quit IRC | 16:03 | |
| fungi | AJaeger: hopefully nothing is running those on anything other than master (in which case they may mostly work as expected) | 16:03 |
| sdague | gordc: that's fine, ithas to be a different job | 16:03 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Move summarize_task_statuses into wmodels.Story https://review.openstack.org/271407 | 16:03 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Resolve Board and Worklist contents on the server-side https://review.openstack.org/271408 | 16:03 |
| AJaeger | fungi, they are run on liberty as well | 16:03 |
| *** abregman is now known as abregman|afk | 16:04 | |
| sdague | gordc: all of "$ZUUL_PROJECT" == "openstack/python-ceilometerclient" ]; then are incorrect | 16:04 |
| fungi | AJaeger: in that case, have a look at one of the logs if you can track one down, and see whether it ended up running with stable/liberty or master refs | 16:04 |
| sdague | libs from git can't be conditional, otherwise you end up with some very weird change queues | 16:05 |
| fungi | AJaeger: or i might be able to help you find a log for one of those for a recent liberty change in a bit | 16:05 |
| sdague | which are basically meaningless | 16:05 |
| *** patient-0-bl0gan is now known as blogan | 16:05 | |
| *** dizquierdo has quit IRC | 16:05 | |
| *** jsavak has quit IRC | 16:05 | |
| jeblair | krotscheck, pavel_bondar: we actually have our list of pypi mirrors documented! http://docs.openstack.org/infra/system-config/bandersnatch.html | 16:06 |
| krotscheck | jeblair: Neat! | 16:06 |
| *** EricGonc_ has joined #openstack-infra | 16:06 | |
| krotscheck | jeblair: Good morning! | 16:06 |
| krotscheck | jeblair: I made the merging happen | 16:06 |
| krotscheck | jeblair: There are things for you to look at :) https://review.openstack.org/#/q/topic:afs-mirror+status:open+branch:master | 16:06 |
| gordc | sdague: the idea is to just test git if it's in that's projects gate... from ceiomter/aodh/gnocchi pov, they are ok with using pip in their gate. | 16:06 |
| *** jsavak has joined #openstack-infra | 16:06 | |
| *** sdake_ has quit IRC | 16:07 | |
| AJaeger | fungi: I have a log and don't like what I see ;( | 16:07 |
| *** vgridnev has quit IRC | 16:07 | |
| AJaeger | wait, wrong log... | 16:07 |
| fungi | AJaeger: yep, we should rip those out | 16:07 |
| AJaeger | let's try again... | 16:07 |
| fungi | switch them back to their non-constraints versions if they're checking out master for stable/liberty changes | 16:07 |
| jeblair | krotscheck: awesome! i just finished reading scrollback, and then need to do whatever it is i do instead of drinking coffee, then i will be ready to finish this! | 16:07 |
| *** salv-orlando has joined #openstack-infra | 16:08 | |
| krotscheck | jeblair: Wait. There are things instead of coffee and mirrors? | 16:08 |
| * krotscheck doesn't understand | 16:08 | |
| fungi | jeblair: have some liquid which is almost but not quite entirely unlike tea? | 16:08 |
| AJaeger | fungi, looks fine, git os-job did the wrong thing for me | 16:08 |
| AJaeger | fungi, I see ":Prepared openstack/neutron repo with branch stable/liberty at commit 65bd02b4a5325e3eaefef61e20160aa15f963d6c" | 16:08 |
| *** katyafervent is now known as katyafervent_awa | 16:08 | |
| AJaeger | and that's the proper hash of neutron liberty | 16:08 |
| jeblair | fungi: like really weak tea. really really weak. sort of homeopathic tea. | 16:09 |
| sdague | gordc: right, but a common job name means they are all linked | 16:09 |
| *** EricGonczer_ has quit IRC | 16:09 | |
| sdague | zuul assumes that a job name means everything is tested exactly the same way every time for all things in that job | 16:09 |
| sdague | that's actually a fundamental construct | 16:09 |
| sdague | doing conditional logic inside a job name based on ZUUL_PROJECT breaks that | 16:10 |
| AJaeger | fungi, so that post job looks fine... | 16:10 |
| gordc | sdague: is the suggestion to either remove conditionals and always test from git (if shared)? or create 3 different similar copies? | 16:10 |
| fungi | grr... jenkins03 hit a 502 proxy error now, so it'll fail out and need resuming soon as well | 16:10 |
| sdague | gordc: 3 different jobs | 16:10 |
| sdague | this is why we have the src jobs | 16:10 |
| sdague | for oslo libraries | 16:11 |
| openstackgerrit | Alexander Maretskiy proposed openstack-infra/project-config: Add timestamps to gate-rally-dsvm-cli output https://review.openstack.org/271412 | 16:11 |
| *** mhickey has quit IRC | 16:11 | |
| dmsimard | AJaeger: so is $PROJECTS a variable that is exposed by defaults in jobs ? | 16:11 |
| gordc | sdague: you have an example of src jobs? | 16:11 |
| *** salv-orlando has quit IRC | 16:12 | |
| AJaeger | dmsimard: really depends on the jobs... | 16:12 |
| *** sdake has joined #openstack-infra | 16:12 | |
| sdague | gordc: https://github.com/openstack-infra/project-config/blob/6594d712fd4d60a2c457fa4bba2905bad9c1bb8f/jenkins/jobs/devstack-gate.yaml#L74-L114 | 16:13 |
| *** bpokorny has joined #openstack-infra | 16:13 | |
| AJaeger | dmsimard: argh, the copy-logs script uses PROJECTS differently than I expected ;( | 16:13 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for keystonemiddleware 4.0.0 / gnocchi bug 1522014 https://review.openstack.org/271414 | 16:13 |
| openstack | bug 1522014 in Gnocchi "keystonemiddleware 4.0.0 breaks gnocchi gate" [Critical,Fix committed] https://launchpad.net/bugs/1522014 | 16:13 |
| gordc | sdague: ack | 16:13 |
| dmsimard | AJaeger: :/ | 16:13 |
| gtmanfred | Shrews: https://review.openstack.org/#/c/270993/ there was the fix to the list roles for v2 | 16:14 |
| *** ddieterly has quit IRC | 16:14 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: remove nova constraints named jobs https://review.openstack.org/271404 | 16:15 |
| sdague | AJaeger: updated | 16:15 |
| dmsimard | AJaeger: I will sleep on things as is and give this some thought over the weekend - don't want to be stuck between a rock and a hard place. I need packstack jobs and I don't want to duplicate the copy-puppet-logs script (modulo ~10 lines), I'm all for a spec for improving things in general at a later time | 16:15 |
| dmsimard | AJaeger: I just want this to land asap in mitaka cycle | 16:15 |
| *** jsavak has quit IRC | 16:15 | |
| Shrews | gtmanfred: saw it. haven't had a chance to go back and re-review yet | 16:15 |
| *** rcernin has quit IRC | 16:15 | |
| *** diazjf has joined #openstack-infra | 16:15 | |
| AJaeger | dmsimard: I understand but I'm investigating other stuff right now. | 16:16 |
| *** jsavak has joined #openstack-infra | 16:16 | |
| gtmanfred | kk :) | 16:16 |
| AJaeger | dmsimard: why don't you make some simple changes as a first step - refactoring the script with exact same functionality. | 16:16 |
| AJaeger | And then we can go from there in smaller steps. | 16:16 |
| *** sputnik13 has joined #openstack-infra | 16:16 | |
| dmsimard | AJaeger: not asking you anything, you've helped a lot already. Just saying I will give this some more thought to see what is the best approach | 16:17 |
| AJaeger | dmsimard: once the problems with our CI our solved, somebody else might be able to discuss the best way forward... | 16:17 |
| AJaeger | dmsimard: thanks | 16:17 |
| dmsimard | <3 | 16:17 |
| *** diazjf1 has joined #openstack-infra | 16:17 | |
| *** iremizov_ has joined #openstack-infra | 16:17 | |
| * dmsimard feels AJaeger doesn't get enough credit for rocking stackalytics | 16:17 | |
| AJaeger | dmsimard: just numbers ;) | 16:18 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Archive all of the delorean logs https://review.openstack.org/271416 | 16:18 |
| AJaeger | fungi, the periodic constraints job looks fine as well "Prepared openstack/neutron repo with commit ba9485d2cceb993c1cc955144366817f229f0036" - and that's on liberty | 16:19 |
| *** mrmartin has quit IRC | 16:19 | |
| *** ociuhandu has quit IRC | 16:19 | |
| *** diazjf has quit IRC | 16:20 | |
| *** zeih has joined #openstack-infra | 16:20 | |
| *** ddieterly has joined #openstack-infra | 16:21 | |
| *** diazjf has joined #openstack-infra | 16:21 | |
| fungi | AJaeger: so it's possible that the main issue is exhibited on tags (where we see it try to do --branch=refs/tags/2015.1.3 and fall back on master when it fails to find that ref) | 16:21 |
| AJaeger | fungi, so that leaves the tarball jobs broken, the rest looks fine... see https://jenkins05.openstack.org/view/All/job/neutron-branch-tarball-constraints/56/ | 16:21 |
| *** Ramanjaneya has joined #openstack-infra | 16:21 | |
| *** Qiming has quit IRC | 16:22 | |
| *** ccarmack has quit IRC | 16:22 | |
| AJaeger | fungi, let me change the tarball jobs for now since those are for sure broken... | 16:22 |
| *** diazjf1 has quit IRC | 16:22 | |
| fungi | AJaeger: that job you just linked was for a master branch commit, not stable/liberty | 16:23 |
| fungi | says "BRANCH=master" there | 16:23 |
| *** tjones1 has quit IRC | 16:23 | |
| *** hdd has quit IRC | 16:24 | |
| AJaeger | fungi: indeed - and https://jenkins05.openstack.org/view/All/job/neutron-branch-tarball-constraints/55/ uses liberty in the name. | 16:25 |
| * AJaeger is confused | 16:25 | |
| *** scheuran has quit IRC | 16:26 | |
| *** rbrndt has joined #openstack-infra | 16:27 | |
| AJaeger | fungi, sorry, I'm confused and can't think straight on this anymore. I don't know why and where it fails exactly... | 16:27 |
| fungi | AJaeger: http://logs.openstack.org/65/65bd02b4a5325e3eaefef61e20160aa15f963d6c/post/neutron-branch-tarball-constraints/26dbdd1/console.html is the log for the latest stable/liberty branch tip | 16:27 |
| fungi | and it does indeed appear that z-c checks out 65bd02b4a5325e3eaefef61e20160aa15f963d6c as it should | 16:27 |
| AJaeger | and 65bd02b4a5325e3eaefef61e20160aa15f963d6c is the tip and zuul-cloner checks it out as it should | 16:27 |
| *** wei-wang has joined #openstack-infra | 16:27 | |
| fungi | INFO:zuul.Cloner:Prepared openstack/neutron repo with branch stable/liberty at commit 65bd02b4a5325e3eaefef61e20160aa15f963d6c | 16:27 |
| fungi | yeah | 16:28 |
| AJaeger | argh - fungi types faster than me ;) | 16:28 |
| *** wei-wang has quit IRC | 16:28 | |
| AJaeger | so, is the release pipeline the problem? | 16:28 |
| *** ccarmack has joined #openstack-infra | 16:28 | |
| fungi | AJaeger: if i were typing in german, i'm sure you'd beat my typing speed easily | 16:28 |
| AJaeger | ;) | 16:28 |
| fungi | AJaeger: yes, so far the two problems i've observed: 1. dhellmann's release notes job which is supposed to checkout master even when the change is for another branch didn't work as intended with zuul-cloner and ended up falling back to the on-disk cached master branch state rather than latest remote; 2. jobs trying to pass a tag to zuul-cloner --branch end up not finding the tag ref (possibly because | 16:31 |
| fungi | it really expects a branch?) and using master instead | 16:31 |
| *** mrmartin has joined #openstack-infra | 16:31 | |
| *** sabeen3 has quit IRC | 16:32 | |
| *** armax has joined #openstack-infra | 16:33 | |
| AJaeger | I see | 16:33 |
| *** zeih has quit IRC | 16:33 | |
| *** sputnik13 has quit IRC | 16:33 | |
| openstackgerrit | Rajini Ram proposed openstack-infra/puppet-openstackci: Updated gearman default port information https://review.openstack.org/267733 | 16:34 |
| AJaeger | but can't think straight to figure out where these conditios are met ;( | 16:34 |
| *** annegentle has quit IRC | 16:35 | |
| *** jaosorior has quit IRC | 16:35 | |
| AJaeger | {name}-docs-constraints' in the release pipeline is one candidate | 16:35 |
| AJaeger | and '{name}-tarball-constraints' is also in release | 16:35 |
| *** iremizov has quit IRC | 16:36 | |
| *** iremizov_ is now known as iremizov | 16:36 | |
| fungi | part of the complication on the release notes job is i think that zuul-cloner is using a zuul merger worker as the remote, and that worker may not have checked out latest master branch tip of the repo in question since the ref triggering the job was for another branch | 16:36 |
| AJaeger | fungi: sorry, no patch coming from me for this... | 16:36 |
| *** iremizov_ has joined #openstack-infra | 16:36 | |
| AJaeger | so, it might work by accident but not by design? | 16:37 |
| fungi | AJaeger: yeah, no worries. it's just useful to have a sounding board to help me think through the issues we're seeing and might see | 16:37 |
| *** annegentle has joined #openstack-infra | 16:38 | |
| openstackgerrit | Rajini Ram proposed openstack-infra/puppet-openstackci: Updated gearman default port information https://review.openstack.org/267733 | 16:38 |
| *** iremizov_ has quit IRC | 16:38 | |
| AJaeger | happy to listen and comment | 16:38 |
| anteaya | Thelo: thanks for asking, if the operational status of your third party ci changes you need to update your system entry on this page: https://wiki.openstack.org/wiki/ThirdPartySystems follow the instructions at the bottom of the page, as well as your system status on your indivdual system page | 16:38 |
| *** zeih has joined #openstack-infra | 16:39 | |
| *** simondodsley has joined #openstack-infra | 16:39 | |
| anteaya | Thelo: you don't need to announce your system status anywhere else, there are too many systems for that to be practical | 16:39 |
| AJaeger | morning, anteaya. Could you review my third keystone linters change some time today, please? https://review.openstack.org/#/c/269261/ | 16:40 |
| anteaya | when I'm finished the other bits on my list, sure | 16:41 |
| openstackgerrit | Dougal Matthews proposed openstack-infra/project-config: Retire the Tuskar project https://review.openstack.org/270869 | 16:41 |
| fungi | infra-root: the resumed jjb updates on jenkins02-07 are running in screen sessions under mordred's account but taking a good deal longer than usual. i've been watching and resuming (restarting) them if they end in a traceback, but i need to step out to do some lunchtime errands if someone else could keep tabs on them until i return | 16:41 |
| fungi | 02 is going much, much, much faster than the others now that it's been in prepare for shutdown for a while | 16:42 |
| AJaeger | anteaya: sure, thanks | 16:42 |
| *** jistr has joined #openstack-infra | 16:43 | |
| *** mtanino has joined #openstack-infra | 16:43 | |
| *** mpavone has joined #openstack-infra | 16:45 | |
| *** HeOS has quit IRC | 16:45 | |
| anteaya | fungi: enjoy lunch | 16:45 |
| anteaya | AJaeger: thank you | 16:45 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for keystonemiddleware 4.0.0 / gnocchi bug 1522014 https://review.openstack.org/271414 | 16:46 |
| openstack | bug 1522014 in Gnocchi "keystonemiddleware 4.0.0 breaks gnocchi gate" [Critical,Fix committed] https://launchpad.net/bugs/1522014 | 16:46 |
| *** dimtruck is now known as zz_dimtruck | 16:46 | |
| fungi | okay, heading out to lunch now, back in an hour or so | 16:47 |
| *** salv-orlando has joined #openstack-infra | 16:47 | |
| openstackgerrit | gordon chung proposed openstack-infra/project-config: fix ceilometerclient gate https://review.openstack.org/271379 | 16:49 |
| *** sabeen has joined #openstack-infra | 16:50 | |
| *** abregman|afk is now known as abregman | 16:50 | |
| *** baoli has quit IRC | 16:50 | |
| *** achanda has joined #openstack-infra | 16:50 | |
| gordc | sdague: i'm not entirely sure what DEVSTACK_PROJECT_FROM_GIT does but ^ | 16:51 |
| *** mrmartin has quit IRC | 16:51 | |
| *** sabeen2 has joined #openstack-infra | 16:52 | |
| *** matrohon has quit IRC | 16:53 | |
| *** mrmartin has joined #openstack-infra | 16:53 | |
| zeih | AJaeger: are you around? | 16:54 |
| *** rcernin has joined #openstack-infra | 16:54 | |
| zeih | hi, I have a question to an merged commit | 16:54 |
| *** sabeen has quit IRC | 16:55 | |
| zeih | to project-config | 16:55 |
| *** e0ne has quit IRC | 16:55 | |
| *** pradk has quit IRC | 16:55 | |
| *** pradk has joined #openstack-infra | 16:56 | |
| nibalizer | mordred: krotscheck yes there was a mis-named file in the hiera git repo | 16:56 |
| nibalizer | that repo doesn't go through gerrit | 16:56 |
| *** dslevin has joined #openstack-infra | 16:57 | |
| *** ashleighfarnham has joined #openstack-infra | 16:57 | |
| *** achanda has quit IRC | 16:57 | |
| *** esker has quit IRC | 16:59 | |
| *** achanda has joined #openstack-infra | 16:59 | |
| *** zz_dimtruck is now known as dimtruck | 17:00 | |
| krotscheck | nibalizer: OK, has that been fixed? | 17:00 |
| *** abregman is now known as abregman|afk | 17:01 | |
| nibalizer | I think so | 17:01 |
| nibalizer | yes it has | 17:02 |
| nibalizer | confirmed | 17:02 |
| *** mpavone has quit IRC | 17:02 | |
| anteaya | alkhodos_: regarding your question: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-01-22.log.html#t2016-01-22T13:46:37 what do you mean by the CI section? I see Nexenta CI commented on patchset 7, the comment includes logs and is listed as a reviewer. What else are you expecting? | 17:02 |
| *** baoli has joined #openstack-infra | 17:02 | |
| *** derekh has quit IRC | 17:02 | |
| *** sfinucan has quit IRC | 17:03 | |
| krotscheck | nibalizer: Cool, thanks! | 17:03 |
| amaretskiy | hi all | 17:03 |
| krotscheck | nibalizer: Also, there's a review I for hiera things that you might be interested in : https://review.openstack.org/#/c/270966/ | 17:03 |
| amaretskiy | i noticed a frequent error "fatal: Couldn't find remote ref" that sometimes eats a lot of job's time | 17:04 |
| zaro | morning | 17:04 |
| *** bardia has joined #openstack-infra | 17:04 | |
| amaretskiy | http://logs.openstack.org/76/270176/4/check/gate-rally-dsvm-cli/83a07b3/logs/devstack-gate-setup-workspace-new.txt.gz#_2016-01-21_09_01_55_278 | 17:04 |
| *** ildikov has quit IRC | 17:04 | |
| anteaya | morning zaro | 17:04 |
| amaretskiy | can someone discuss this error with me? | 17:04 |
| alkhodos_ | anteaya: I meant the section above comments, where all CI results are combined, but asselin_ already answered to that question | 17:05 |
| nibalizer | krotscheck: thanks | 17:06 |
| *** yamahata has quit IRC | 17:06 | |
| *** flepied has quit IRC | 17:06 | |
| openstackgerrit | Daniel Wallace proposed openstack-infra/shade: include keystonev2 role assignments https://review.openstack.org/270993 | 17:06 |
| *** hashar has quit IRC | 17:07 | |
| asselin | amaretskiy, those are expected....the script attempts to find a ref in zuul, it doesn't find it, and tries the next place to look | 17:07 |
| anteaya | alkhodos_: wonderful | 17:07 |
| amaretskiy | asselin: even if this expected, it takes a lot of time periodically | 17:08 |
| *** jistr has quit IRC | 17:08 | |
| amaretskiy | asselin: from 1 seconds to 10 minutes (even more) on sam ejob | 17:08 |
| nibalizer | asselin: pabelanger https://review.openstack.org/#/c/270966/1 | 17:08 |
| amaretskiy | asselin: please look at a small tool i made for this https://raw.githubusercontent.com/maretskiy/stuff/master/openstack/parse-lost-time.py | 17:08 |
| *** nelsnelson has quit IRC | 17:09 | |
| amaretskiy | asselin: http://paste.openstack.org/show/484657/ 604 seconds!!!! | 17:09 |
| *** nelsnelson has joined #openstack-infra | 17:09 | |
| amaretskiy | asselin: these 604 seconds can cause job failed by timeout | 17:09 |
| amaretskiy | asselin: is this normal? | 17:09 |
| *** ccrouch has quit IRC | 17:09 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Wrap long words in the "Items to add" table https://review.openstack.org/271430 | 17:09 |
| *** jistr has joined #openstack-infra | 17:09 | |
| *** baoli has quit IRC | 17:10 | |
| *** ccrouch has joined #openstack-infra | 17:10 | |
| *** baoli has joined #openstack-infra | 17:10 | |
| asselin | amaretskiy, any suggestiong to improve it? | 17:10 |
| clarkb | IO is expensive yes. this is why I have been working yo rrduce the projects list to a smaller set | 17:11 |
| clarkb | *to reduce | 17:11 |
| clarkb | we dont need every project configured from source on every job | 17:11 |
| amaretskiy | asselin: no actual suggestions right now but I can potentially dig into this and maybe I can propose some updates | 17:12 |
| amaretskiy | asselin: just want to discuss this, is thois a topic to improve or a confirmed issue | 17:12 |
| asselin | amaretskiy, clarkb's comments pertain to you question fyi ^^ | 17:12 |
| *** jpr has joined #openstack-infra | 17:12 | |
| *** markus_z has quit IRC | 17:13 | |
| asselin | amaretskiy, I think it's worth to improve. Honestly it would be nice to somehow do these in parallel. | 17:13 |
| amaretskiy | clarkb: but same job loses 1 second one time and 10 minutes next time | 17:13 |
| krotscheck | clarkb, fungi: We've got a few reasonably low-overhead patches for the afs-mirror effort, could one of you take a look? 270966 - 271307 - 270559 | 17:13 |
| clarkb | amaretskiy: it is a function of how busy the servers are | 17:14 |
| *** jistr has quit IRC | 17:14 | |
| amaretskiy | asselin: okay, I will try to research this, maybe I can propose smth | 17:14 |
| asselin | amaretskiy, where do you see 10 minutes? I see 604 seconds...is that the total? | 17:14 |
| amaretskiy | yes, total 604 sec | 17:14 |
| amaretskiy | 10 min 4 sec | 17:14 |
| asselin | ok but no individual one takes that long | 17:14 |
| asselin | what's the max individual one? | 17:15 |
| *** trown is now known as trown|lunch | 17:15 | |
| *** piet has quit IRC | 17:15 | |
| amaretskiy | asselin: total is a sum of all stuck timeouts within the log, some of these timeouts took 1 sec, some several minutes | 17:17 |
| *** jistr has joined #openstack-infra | 17:17 | |
| amaretskiy | asselin: look at http://paste.openstack.org/show/484657/ - here are parsed errors | 17:17 |
| *** mrmartin has quit IRC | 17:18 | |
| clarkb | and the repos involved, repos with more changes are probably slower. neutron nova etc | 17:18 |
| amaretskiy | asselin: the max individual that I saw was about 8 minutes | 17:18 |
| *** yaume has quit IRC | 17:18 | |
| asselin | amaretskiy, I was hoping your script would do the math: min, max, avg, etc | 17:18 |
| clarkb | its possible mergers need ref pruning | 17:19 |
| amaretskiy | asselin: this math is not a problem :) | 17:19 |
| amaretskiy | asselin: if this will help | 17:19 |
| *** sputnik13 has joined #openstack-infra | 17:19 | |
| AndreiB | hi guys, i have a few questions about zuul and the new connections concept recently added | 17:19 |
| asselin | amaretskiy, I think it would. in fact I think it would be nice to include those numbers as part of the job itself.....a bit of self-profiling | 17:20 |
| amaretskiy | asselin: good idea | 17:20 |
| amaretskiy | asselin: I will improve my tool soon | 17:20 |
| *** mrmartin has joined #openstack-infra | 17:20 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate: Set PS4 to provide additional debug info https://review.openstack.org/271434 | 17:21 |
| amaretskiy | asselin: okay, I believe we will return to this question soon :) | 17:21 |
| anteaya | women work on openstack as well, not just guys | 17:21 |
| AndreiB | oh, sorry was not my intention to offend :) | 17:22 |
| AndreiB | can`t seem to figure out a way to separate each project on it`s own account on one zuul server | 17:23 |
| greghaynes | jhesketh: did you figure anything else out about the uploads not happening? | 17:24 |
| greghaynes | clarkb: last night it kind of sounded like the same disconnect issue happened with nodepool but couldn't confirm | 17:25 |
| clarkb | greghaynes: ok we should check geard server log | 17:25 |
| clarkb | I wonder if it is due to the bw used by upload | 17:26 |
| clarkb | since it is fine during build | 17:26 |
| openstackgerrit | Claudiu Belu proposed openstack/requirements: Removes posix-ipc from global-requirements https://review.openstack.org/217824 | 17:26 |
| greghaynes | clarkb: This sounded like no uploads happened. If it did happen again I suspect it is a bug in geard/client when send/recv buffers fill | 17:26 |
| greghaynes | clarkb: but should confirm first | 17:26 |
| *** esker has joined #openstack-infra | 17:27 | |
| *** gyee_ has joined #openstack-infra | 17:27 | |
| *** piet has joined #openstack-infra | 17:28 | |
| krotscheck | AndreiB, anteaya: I usually just refer to all of us as 'muppets'. | 17:28 |
| *** sdake has quit IRC | 17:28 | |
| krotscheck | It makes for way better dance numbers. | 17:28 |
| * krotscheck does a softshoe across the stage. | 17:28 | |
| jpr | asselin, carkb: any hints on where the job record is stored for nodepool/zuul/gearman. I have a disk-image-build that looks like it's stale/broken but can't find a place where I can manually clear it out. | 17:28 |
| anteaya | krotscheck: I like muppets | 17:28 |
| anteaya | sweetums is my favourite | 17:28 |
| krotscheck | He really is adorable. | 17:29 |
| * krotscheck is a fan of Dr. Teeth | 17:29 | |
| *** sputnik13 has quit IRC | 17:29 | |
| krotscheck | And the Swedish Chef, but mostly because I'm jealous of his healthcare system. | 17:29 |
| krotscheck | aaanywa | 17:29 |
| jeblair | krotscheck: hi i'm back | 17:30 |
| *** arxcruz has joined #openstack-infra | 17:30 | |
| anteaya | agreed Swedish Chef is awesome | 17:30 |
| anteaya | jeblair: welcome back | 17:30 |
| krotscheck | jeblair: hiya! | 17:31 |
| jeblair | krotscheck: regarding the wheel builders... | 17:31 |
| anteaya | AndreiB: what are you doing? what are you seeing? and what are you expecting to see? | 17:31 |
| *** acruz has quit IRC | 17:31 | |
| AndreiB | well we`re trying to setup different accounts one for each of our projects | 17:32 |
| krotscheck | jeblair: Yep? | 17:32 |
| jeblair | krotscheck: i think we probably can't trust them to hold the afs admin key needed to vos release... but we _could_ trust them enough to let them write to their own particular volume... | 17:32 |
| anteaya | AndreiB: gerrit accounts? | 17:33 |
| jeblair | krotscheck: so maybe we should plan on making them afs clients | 17:33 |
| krotscheck | Ok, so a volume per wheel mirror volume? That seems a little heavy in the maint whenever a new node comes up. | 17:33 |
| AndreiB | anteaya, yes | 17:33 |
| *** dimtruck is now known as zz_dimtruck | 17:33 | |
| *** dtantsur is now known as dtantsur|afk | 17:33 | |
| jeblair | krotscheck: well, i think the unit of mirroring with wheel mirrors is actuall the wheel architecture... so yeah. | 17:33 |
| AndreiB | there does not seem to be a way to specify the connection name for projects in layout file | 17:33 |
| anteaya | AndreiB: and you are folloiwng this page closely? http://docs.openstack.org/infra/system-config/third_party.html | 17:33 |
| jeblair | krotscheck: but that won't happen that often... like once every 6 months probably :) | 17:34 |
| jeblair | at the most | 17:34 |
| krotscheck | jeblair: True. | 17:34 |
| krotscheck | jeblair: Right, so that's something I'll dig into monday morning. At the moment, we're about ready to start building the regular pypi nodes :) | 17:34 |
| AndreiB | anteaya: we are already running CIs for a few projects, currently each with it`s own zuul | 17:34 |
| jeblair | krotscheck: then i'm thinking we need to write a quick job queue system (hi gearman!) to have a central server (maybe mirror_update.o.o) do vos releases | 17:34 |
| *** jsavak has quit IRC | 17:35 | |
| AndreiB | anteaya: was just wondering if it`s possible to use a single zuul for all of them, now that zuul supports multiple gerrit accounts | 17:35 |
| anteaya | AndreiB: that is the typical way that I know of to do it, one zuul per gerrit account | 17:35 |
| krotscheck | I'm thinking: Let's get the AFS infrastructure up frist, and then layer on the wheels. | 17:35 |
| krotscheck | jeblair: Hrm. I can see that. | 17:35 |
| anteaya | does zuul support multiple gerrit accounts? | 17:35 |
| *** jsavak has joined #openstack-infra | 17:35 | |
| jeblair | krotscheck: so basically the wheel mirror builder does: zuul tells jenkins to tell it to build some wheels; it writes the output into afs directly; it submits a job to vos release; some other host performs the vos release | 17:36 |
| AndreiB | anteaya, yes -> https://github.com/openstack-infra/zuul/commit/352264b3c22b501ae53c1643e31df2ccdfc89388 | 17:36 |
| krotscheck | jeblair: So walk me through this- Wheel slave builds a wheel, then publishes it to its own /afs/.openstack.org/mirror/wheel/DISTRO-VERSION/, and then notifies mirror_update to do a vos release? | 17:36 |
| jeblair | krotscheck: (i agree about the ordering -- just want to finish working through this mentally) | 17:36 |
| jeblair | krotscheck: where is the WIP wheel mirror job patch? | 17:37 |
| jeblair | krotscheck: re sequence ^: yes | 17:37 |
| krotscheck | jeblair: https://review.openstack.org/#/c/164927/ | 17:37 |
| *** burgerk has quit IRC | 17:37 | |
| krotscheck | jeblair: https://review.openstack.org/#/c/164927/19/jenkins/jobs/macros.yaml I think is the pertinent file. | 17:37 |
| jeblair | krotscheck: oh hey we don't need another job queuing system we have one :) | 17:38 |
| krotscheck | So, the distribute-wheel-mirror job builds an AFS-style folder structure from the flat wheel directory. | 17:38 |
| krotscheck | And then it rsyncs that. | 17:38 |
| jeblair | krotscheck: we can make put the afs admin key on a trusted jenkins slave and have a vos release jenkins job | 17:38 |
| *** andymaier has quit IRC | 17:39 | |
| jeblair | and run that job after the wheel build jobs | 17:39 |
| krotscheck | jeblair: Wait, don't we have a trusted slave already? ;) | 17:39 |
| jeblair | krotscheck: i think we do! | 17:39 |
| krotscheck | jeblair: What a coincidence! | 17:39 |
| jeblair | it's amazing | 17:39 |
| anteaya | AndreiB: I believe the the zuul connections feature is actually suspected of causing a zuul bug, I don't know as the feature has been reverted | 17:39 |
| anteaya | AndreiB: but I believe that is the conversation around it | 17:39 |
| krotscheck | jeblair: And it's almost like I _just_ did a job queue dependency thing with some kind of other publishing mechanism, so I know exactly how to make it work. | 17:39 |
| * krotscheck forgets the name. Edge? Graph? Node-something?" | 17:40 | |
| *** jcoufal has quit IRC | 17:40 | |
| anteaya | AndreiB: however as far as documentation of the feature I believe this commit added it: http://git.openstack.org/cgit/openstack-infra/zuul/commit/?id=fe485c609178c9eaa965609e8b74ce6f43d5afdb | 17:40 |
| krotscheck | Ok, so that job that I pointed you at, I can collapse the script into one call - build the wheels, then rsync them to AFS | 17:40 |
| jeblair | krotscheck: cool -- so yeah, that's probably a plan for next week; on to actually doing things now | 17:40 |
| jeblair | krotscheck: ++ | 17:41 |
| krotscheck | And, frankly, I don't really need to rsync, just build the destination path and do a thing if the file isn't there. | 17:41 |
| anteaya | AndreiB: and I believe this is where it is published: http://docs.openstack.org/infra/zuul/connections.html | 17:41 |
| jeblair | mordred: give this plan a once over when you get a sec ^; no rush | 17:41 |
| krotscheck | Or something | 17:41 |
| krotscheck | Anywa | 17:41 |
| krotscheck | jeblair: My current work list https://review.openstack.org/#/q/topic:afs-mirror+status:open+branch:master | 17:41 |
| *** jpr has quit IRC | 17:42 | |
| *** jordanP has quit IRC | 17:42 | |
| krotscheck | jeblair: But getting cores to review those has so far not been successful :/ | 17:42 |
| krotscheck | Ish | 17:42 |
| *** FallenPegasus has joined #openstack-infra | 17:43 | |
| jeblair | krotscheck: looks like 975 and 307 are the most immediate ones | 17:43 |
| *** esikachev has joined #openstack-infra | 17:43 | |
| krotscheck | jeblair: 307 is blocked on 966 I think | 17:44 |
| *** esker has quit IRC | 17:44 | |
| openstackgerrit | gordon chung proposed openstack-infra/project-config: fix ceilometerclient gate https://review.openstack.org/271379 | 17:45 |
| krotscheck | jeblair: Given that the mirror puppet manifests landed, can you bprovision the actual servers now? | 17:45 |
| jeblair | krotscheck: yeah, i'm going to review these 2 then do that | 17:45 |
| AndreiB | anteaya, yes, i looked through all that already, still did not find a way to specify the connection for each project, looks like it`s meant to be used at pipelines | 17:46 |
| *** _nadya_ has quit IRC | 17:46 | |
| *** asselin has quit IRC | 17:46 | |
| jeblair | fungi, mordred: https://review.openstack.org/270979 and parent have a +2 from me | 17:46 |
| jeblair | krotscheck: no servers have been spun up yet, right? | 17:46 |
| *** zz_dimtruck is now known as dimtruck | 17:46 | |
| krotscheck | jeblair: Not that I'm aware of. | 17:47 |
| anteaya | sorry to interrupt, infra-root, zuul used memory is very very high right now: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=392&rra_id=all | 17:47 |
| krotscheck | I think everyone was deferring to your authority | 17:47 |
| jeblair | anteaya: it would be really good to restart it now. | 17:47 |
| anteaya | agreed | 17:47 |
| jeblair | infra-root: i am going to restart zuul | 17:48 |
| nibalizer | jeblair: ok | 17:48 |
| nibalizer | I'm not in a good spot to restart zuul rn | 17:48 |
| *** unicell has joined #openstack-infra | 17:49 | |
| jeblair | #status notice Restarting zuul due to a memory leak | 17:49 |
| openstackstatus | jeblair: sending notice | 17:49 |
| -openstackstatus- NOTICE: Restarting zuul due to a memory leak | 17:50 | |
| jeblair | okay, that's done and things are re-enqueuing | 17:51 |
| *** unicell has quit IRC | 17:51 | |
| anteaya | thank you | 17:51 |
| clarkb | is it a leak or just high overhead for backed up gate? I guess we will know soon enough | 17:51 |
| jeblair | anteaya: thank you | 17:51 |
| *** flepied has joined #openstack-infra | 17:51 | |
| jeblair | clarkb: did you look at the graph? | 17:51 |
| anteaya | glad to be useful | 17:51 |
| clarkb | jeblair: no havent had a chance | 17:51 |
| openstackstatus | jeblair: finished sending notice | 17:52 |
| jeblair | clarkb: this one is telling http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=392&rra_id=3&view_type=&graph_start=1450806441&graph_end=1453484841 | 17:52 |
| anteaya | also when an infra-root has a chance to peek at java melody it might be suggested | 17:52 |
| anteaya | doesn't have to be jeblair | 17:52 |
| *** esker has joined #openstack-infra | 17:52 | |
| anteaya | gerrit cpu usage: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=25&rra_id=all | 17:53 |
| *** asselin has joined #openstack-infra | 17:53 | |
| *** pblaho has quit IRC | 17:53 | |
| jeblair | clarkb: zuul spent a year using 2G of ram, then suddenly in december it starts a 45 degree climb to 30G. | 17:53 |
| *** flepied has quit IRC | 17:53 | |
| *** bgaifullin has quit IRC | 17:53 | |
| AndreiB | anteaya, thanks for your input, sent a message to jhesketh (who added connections to zuul) | 17:53 |
| jeblair | clarkb: i'm fairly certain there's a memory leak introduced by something we merged in the november/december timeframe | 17:54 |
| anteaya | AndreiB: yes that was going to be my suggestion, after we addressed the zuul memory leak in production | 17:54 |
| jeblair | clarkb: which is why i sent that message to -infra. but no one has responded. | 17:54 |
| jeblair | so i guess no one runs zuul | 17:54 |
| *** beisner has quit IRC | 17:54 | |
| *** beisner has joined #openstack-infra | 17:55 | |
| *** zeih has quit IRC | 17:55 | |
| jeblair | krotscheck: https://etherpad.openstack.org/p/new-mirrors | 17:56 |
| krotscheck | Work tracking? | 17:56 |
| jeblair | krotscheck: have the list of hosts handy? | 17:56 |
| greghaynes | hrm, that last part of the graph looks like we hit the leak suddenly | 17:56 |
| *** mrmartin has quit IRC | 17:56 | |
| *** bpokorny_ has joined #openstack-infra | 17:57 | |
| *** jistr has quit IRC | 17:57 | |
| krotscheck | jeblair: yep, just a sec | 17:57 |
| *** jsavak has quit IRC | 17:57 | |
| krotscheck | jeblair: In the etherpad | 17:57 |
| jeblair | greghaynes: yeah, big leak here: http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=392&rra_id=3&view_type=&graph_start=1452199166&graph_end=1452320456&graph_height=120&graph_width=500&title_font_size=10 | 17:57 |
| jeblair | greghaynes: then a little leak here: http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=392&rra_id=3&view_type=&graph_start=1452822155&graph_end=1452913123&graph_height=120&graph_width=500&title_font_size=10 | 17:57 |
| clarkb | I should check if yappi grabs memory details | 17:57 |
| clarkb | but we may be able to use it to help diagnose | 17:57 |
| jeblair | greghaynes: then another big leak here: http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=392&rra_id=3&view_type=&graph_start=1453343151&graph_end=1453478224&graph_height=120&graph_width=500&title_font_size=10 | 17:57 |
| *** ildikov has joined #openstack-infra | 17:58 | |
| *** bpokorny_ has quit IRC | 17:58 | |
| *** jsavak has joined #openstack-infra | 17:58 | |
| greghaynes | Ya, and it was pretty slow before that last one... | 17:58 |
| *** bpokorny_ has joined #openstack-infra | 17:59 | |
| jeblair | yeah, it's probably not a continuous one, it's probably triggered by something specific | 17:59 |
| jeblair | which will make it extra fun to track down. | 17:59 |
| greghaynes | yep | 17:59 |
| *** fawadkhaliq has quit IRC | 17:59 | |
| pabelanger | jeblair: mordred: clarkb: have you seen this shade error before? http://pastebin.com/2y3skAMe | 17:59 |
| *** fawadkhaliq has joined #openstack-infra | 17:59 | |
| mordred | pabelanger: looking | 18:00 |
| *** bpokorny has quit IRC | 18:00 | |
| Shrews | pabelanger: that looks like something we fixed either in shade or occ a while back... or at least similar | 18:01 |
| mordred | pabelanger: what versoins of os-client-config and shade do you have? | 18:01 |
| clarkb | looks like yappi doesn't do memory profiling | 18:01 |
| *** sdake has joined #openstack-infra | 18:01 | |
| clarkb | or at least we don't use it that way? probably worth digging more than my quick check | 18:01 |
| pabelanger | python2-shade.noarch 1.4.0-3.fc24 @fedora | 18:02 |
| pabelanger | python2-os-client-config.noarch 1.7.4-3.fc24 @fedora | 18:02 |
| pabelanger | mordred: Shrews ^ | 18:02 |
| pabelanger | Ya, pretty old | 18:02 |
| *** apoorvad has joined #openstack-infra | 18:02 | |
| pabelanger | let me bump client-config | 18:02 |
| pabelanger | Ya, I don't have caps setup properly | 18:03 |
| pabelanger | fixing now | 18:03 |
| jeblair | clarkb, mordred, krotscheck: we have 4G machines for mirrors now and cpu/ram/net are barely used; i'm thinking of 2G. the only thing we'll lose is some io cache. http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=154 | 18:04 |
| krotscheck | I feel like overall, we'll actually use _less_ disk net. | 18:04 |
| jeblair | clarkb, mordred, krotscheck: but maybe a 4G or 8G machine for mirror_update? | 18:05 |
| clarkb | jeblair: at least in rackspace the network bandwidth scales with the flavor too | 18:05 |
| clarkb | jeblair: so that may be another consideration to look at | 18:05 |
| clarkb | looking at the eth0 graph it is less network io than I expected | 18:05 |
| jeblair | clarkb: yeah, but we're using <16mbit and even a 2G is 400mbit | 18:05 |
| jeblair | right | 18:05 |
| *** esker has quit IRC | 18:05 | |
| clarkb | yup | 18:05 |
| jeblair | we do want a big afs disk cache though | 18:06 |
| jeblair | how big do we want that to be? | 18:06 |
| jeblair | (ie, what's our guess for our working set for mirrors?) | 18:06 |
| *** dfed717 has joined #openstack-infra | 18:07 | |
| clarkb | just pypi is in the 200GB range iirc | 18:07 |
| *** mrmartin has joined #openstack-infra | 18:07 | |
| jeblair | clarkb: we don't use all of pypi though | 18:07 |
| openstackgerrit | Alex Schultz proposed openstack-infra/project-config: Switch fuel-library 3.8 to nonvoting https://review.openstack.org/271452 | 18:07 |
| clarkb | right its a subset, but that is an upper bound | 18:07 |
| clarkb | jeblair: and plan is to put gems/npm/ubuntu/fedora/centos on here too? | 18:08 |
| *** boris-42 has joined #openstack-infra | 18:08 | |
| jeblair | clarkb: yep | 18:08 |
| clarkb | gems is about the same size as pypi last I saw but again we only use a subset | 18:08 |
| jeblair | greghaynes, krotscheck: have you built a wheel mirror from requirements recently? | 18:08 |
| *** hdd has joined #openstack-infra | 18:08 | |
| jeblair | that's pretty close to our working set -- how big was that? | 18:08 |
| krotscheck | jeblair: no, but I can. | 18:10 |
| * krotscheck thinks the last time he built it was 2 weeks ago? No idea if it'll still build things. | 18:10 | |
| jeblair | krotscheck: do you still have the output? | 18:10 |
| jeblair | krotscheck: i'm just sort of looking for order of magnitude :) | 18:10 |
| krotscheck | jeblair: nope. | 18:10 |
| jeblair | ah well, np | 18:11 |
| *** achanda has quit IRC | 18:11 | |
| krotscheck | Wait, maybe | 18:11 |
| jeblair | i'm kind of thinking that we want at least 50G and maybe larger if we end up with the other mirrors | 18:11 |
| krotscheck | Nope. | 18:11 |
| *** achanda has joined #openstack-infra | 18:11 | |
| jeblair | that makes ephemeral disk on rax impractical, so we may want to go with 2G machines and some cinder. | 18:11 |
| krotscheck | Mine didn't actually complete, because there are things not supported on Darwin | 18:12 |
| clarkb | greghaynes: I do not see any tracebacks in the gearman server log | 18:12 |
| *** esikachev has quit IRC | 18:12 | |
| clarkb | greghaynes: so now to check hte nodepool side | 18:12 |
| krotscheck | jeblair: I believe the node archive recommends 400+GB, 1TB for longevity? | 18:12 |
| krotscheck | For a full mirror | 18:12 |
| jeblair | yeah, but i'm just trying to size the afs cache | 18:12 |
| jeblair | which will be hit and miss until we actually see it in action | 18:13 |
| clarkb | we have 5 dib images built, now to check their uploads | 18:13 |
| jeblair | so i'll just start with 2GB ram node and 100GB cinder volume on rax | 18:13 |
| jeblair | krotscheck: is that list of hosts from your patch? if so, there may be a typo in it | 18:14 |
| jeblair | krotscheck: note 'rax' vs 'rackspace' | 18:14 |
| clarkb | greghaynes: no images in the snapshot table are building but a quick skim doesn't show any recent dib uploaded images | 18:14 |
| krotscheck | jeblair: Yes. | 18:14 |
| * krotscheck will update that | 18:14 | |
| jeblair | krotscheck: confirmed the value will be rax | 18:15 |
| krotscheck | ....neeevermind then | 18:15 |
| clarkb | I do not see any upload jobs in the log | 18:15 |
| *** Hal has joined #openstack-infra | 18:15 | |
| clarkb | just jobs for building | 18:15 |
| krotscheck | Wait, no | 18:15 |
| jeblair | krotscheck: well, dfw said 'rackspace' | 18:15 |
| *** mrmartin has quit IRC | 18:15 | |
| clarkb | that sort of makes sense since it was just the dib images on disk that were missing | 18:15 |
| krotscheck | have to update dfw | 18:15 |
| jeblair | krotscheck: right | 18:16 |
| *** Hal is now known as Guest72573 | 18:16 | |
| clarkb | however at ~1400utc today we should've had uploads happen | 18:16 |
| krotscheck | jeblair: Can you perhaps add a +2A to mordred's https://review.openstack.org/#/c/270966/ so I don't have to rebase again later? | 18:16 |
| jeblair | clarkb: should have before then too for the builds i started yesterday | 18:16 |
| *** unicell has joined #openstack-infra | 18:16 | |
| clarkb | jeblair: but those builds were just dib images right? not uploads? | 18:17 |
| clarkb | the images were built successfully from what I can see | 18:17 |
| boris-42 | clarkb: hi this fixes timestampts in rally-dsvm-cli job that fails https://review.openstack.org/#/c/271412/1 | 18:17 |
| jeblair | clarkb: oh, i would have had to trigger an upload separately? | 18:17 |
| boris-42 | clarkb: could you please help to get it in | 18:17 |
| jeblair | i forget the rules | 18:17 |
| *** annegent_ has joined #openstack-infra | 18:17 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Fixed typo in mirrors https://review.openstack.org/271456 | 18:18 |
| clarkb | jeblair: yes build and upload are separate | 18:18 |
| *** Guest72573 has quit IRC | 18:18 | |
| krotscheck | jeblair: ^^ Typo fix | 18:18 |
| jeblair | krotscheck: both approved | 18:18 |
| krotscheck | jeblair: sweet | 18:19 |
| *** annegentle has quit IRC | 18:20 | |
| *** lezbar__ has quit IRC | 18:20 | |
| jlvillal | Where are bugs filed for something against devstack-gate ? I didn't find a devstack-gate project on launchpad. | 18:21 |
| clarkb | according to the logs our cron never fired | 18:21 |
| *** baoli has quit IRC | 18:21 | |
| clarkb | pabelanger: are yo usure apscheduler is working? | 18:22 |
| clarkb | grep 'Updating all image' debug.log has no results in the last 3 log files | 18:22 |
| *** baoli has joined #openstack-infra | 18:22 | |
| * greghaynes is on call so not really here | 18:23 | |
| greghaynes | :( | 18:24 |
| clarkb | so I think there are two separate things here | 18:24 |
| anteaya | jlvillal: https://storyboard.openstack.org/#!/project/712 | 18:24 |
| clarkb | 1) we only did builds which worked but not uploads so didn't get uplaods. 2) cron doesn't seem to have fired | 18:24 |
| jlvillal | anteaya: Thanks. Looking... :) | 18:24 |
| anteaya | jlvillal: welcome | 18:24 |
| jlvillal | anteaya: So not launchpad! | 18:25 |
| anteaya | jlvillal: yes, not launchpad | 18:25 |
| pabelanger | clarkb: I belive so, however somebody else reported some possible apscheduler issue, but never heard back on it | 18:25 |
| clarkb | pabelanger: basically its not running | 18:25 |
| pabelanger | Hmm | 18:25 |
| *** armax has quit IRC | 18:25 | |
| clarkb | pabelanger: 'Updating all images' should show up in the log when we update the images | 18:25 |
| clarkb | pabelanger: does not | 18:26 |
| anteaya | pabelanger: thorst I believe | 18:26 |
| jeblair | krotscheck: okay, i mapped out the flavors for our different providers in the etherpad | 18:26 |
| *** trown|lunch is now known as trown|mostlyafk | 18:26 | |
| jlvillal | anteaya: Do you know how "Closes-bug" works with that? | 18:26 |
| krotscheck | jeblair: Cool. I'm going to pull hpe's instance out. | 18:26 |
| thorst | Yeah - sorry. I needed the apscheduler fix | 18:26 |
| jeblair | krotscheck: i confess, i lack enthusiasm for spinning up an hpcloud mirror... but could be convinced to do it; what do you think? | 18:26 |
| thorst | the PowerVM drivers team is setting up a CI system and we were hit when we were redpeploying | 18:26 |
| *** yolanda has quit IRC | 18:26 | |
| openstackgerrit | Merged openstack-infra/project-config: Add timestamps to gate-rally-dsvm-cli output https://review.openstack.org/271412 | 18:27 |
| krotscheck | jeblair: We can wait a week before doing the swtichover? | 18:27 |
| *** EricGonc_ has quit IRC | 18:27 | |
| pabelanger | clarkb: I see updating all images locally | 18:27 |
| boris-42 | clarkb: thank you | 18:27 |
| krotscheck | jeblair: Or we can use --extra-index-urls | 18:27 |
| pabelanger | clarkb: let me check if they updated | 18:27 |
| pabelanger | and uploaded | 18:27 |
| clarkb | pabelanger: via cron? | 18:27 |
| *** angdraug has joined #openstack-infra | 18:27 | |
| pabelanger | clarkb: should be, I didn't do it | 18:28 |
| jlvillal | anteaya: FYI: I was also told about: https://bugs.launchpad.net/openstack-gate | 18:28 |
| jeblair | krotscheck: meh, maybe it's easiest for me to spin one up. | 18:28 |
| clarkb | pabelanger: well there are two mechanisms | 18:28 |
| krotscheck | jeblair: https://review.openstack.org/#/c/267751/ | 18:28 |
| clarkb | pabelanger: one is the I have started and realize I need to make things right now | 18:28 |
| krotscheck | that'll let me add more-than-one mirror. | 18:28 |
| clarkb | pabelanger: and the other is the cron scheduled daily updates | 18:28 |
| jlvillal | anteaya: But I shouldn't use that one. | 18:28 |
| krotscheck | We can just point it at a mirror that doesn't exist | 18:28 |
| pabelanger | clarkb: right | 18:28 |
| anteaya | jlvillal: I don't even know what openstack-gate might refer to | 18:28 |
| clarkb | pabelanger: tests will pass onthe I have started and realize I have work to do check | 18:28 |
| jeblair | krotscheck: i thought extra-index-url came with some baggage | 18:29 |
| clarkb | pabelanger: we don't have tests for the cron actually happening iirc | 18:29 |
| jlvillal | anteaya: I was told it has to do with elastic recheck stuff. | 18:29 |
| anteaya | jlvillal: and I don't see the use of story in commit messages documented here: https://wiki.openstack.org/wiki/GitCommitMessages#Including_external_references | 18:29 |
| anteaya | jlvillal: oh okay thanks | 18:29 |
| krotscheck | jeblair: We'll have to use it anyway for the wheels? | 18:29 |
| jeblair | oh ok | 18:29 |
| fungi | okay, i'm back now | 18:29 |
| fungi | zuul memory leak, huh? | 18:30 |
| jlvillal | anteaya: Me either on commit messages | 18:30 |
| pabelanger | 2016-01-22 02:14:00,019 INFO nodepool.builder: Creating image centos-7 with filename /opt/nodepool/images/3 | 18:30 |
| jeblair | fungi: still there, had it since december. | 18:30 |
| pabelanger | clarkb: what I see ^ | 18:30 |
| nibalizer | jeblair: mordred with data_binding_terminus = none set in puppet.conf 270966 won't work I think | 18:30 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate: Fix '-f: command not found' error https://review.openstack.org/271463 | 18:30 |
| *** dfed717 has left #openstack-infra | 18:30 | |
| anteaya | jlvillal: https://review.openstack.org/#/c/270337/ | 18:31 |
| *** rossella_s has quit IRC | 18:31 | |
| jeblair | nibalizer: good thing zuul kicked it | 18:31 |
| pabelanger | clarkb: pip freeze: http://pastebin.com/XJTrbCRN | 18:31 |
| fungi | jeblair: yeah, i recall the leak. i guess it got serious | 18:31 |
| anteaya | jlvillal: use Story: and the story number, there is no closes bug | 18:31 |
| *** EricGonczer_ has joined #openstack-infra | 18:31 | |
| clarkb | I am doing a manual upload of fedora-23 to hpcloud-b1 to check tha tthat process generally works | 18:31 |
| *** rossella_s has joined #openstack-infra | 18:31 | |
| jlvillal | anteaya: Ah thanks! | 18:31 |
| clarkb | pabelanger: that is just an image build | 18:31 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Remove databindings terminus setting https://review.openstack.org/271466 | 18:31 |
| clarkb | pabelanger: not an upload | 18:31 |
| nibalizer | https://review.openstack.org/#/c/271466/ would fix that | 18:32 |
| fungi | jenkins02 seems to have updated successfully, i'm going to clean it up and get it restarted | 18:32 |
| clarkb | pabelanger: great for 'Updating all images' | 18:32 |
| clarkb | er s/great/grep/ | 18:32 |
| pabelanger | http://pastebin.com/n0Pp72TQ | 18:32 |
| pabelanger | nodepool.log | 18:32 |
| clarkb | pabelanger: would have to be debug log | 18:32 |
| jeblair | jlvillal: just to set expectations, i'm not sure anyone will see or act on that bug report. if it's something you need fixed, you may be better off working on it yourself. | 18:32 |
| fungi | jenkins06 completed successfully as well. 03, 04 and 05 updates broke and have been resumed again | 18:32 |
| fungi | 07 still running | 18:33 |
| clarkb | pabelanger: I am curious to see if the code that updates all images on a cron scedule is firing. to do that you need to grep for the string I gave you | 18:33 |
| anteaya | jlvillal: welcome | 18:33 |
| pabelanger | 2016-01-22 02:14:00,002 DEBUG nodepool.NodePool: Updating all images. | 18:33 |
| jlvillal | jeblair: I actually had the patch. I was just thinking I would create a bug for it :) | 18:33 |
| jeblair | jlvillal: perfect then :) | 18:33 |
| nibalizer | jeblair: what if I proposed a patch that did the same thing as 270966 functionally but didn't use data bindings | 18:33 |
| jlvillal | jeblair: I ended up just submitting the patch without a story. | 18:33 |
| nibalizer | then we can do the mirrror work and have a data-bindings discussion separately | 18:33 |
| pabelanger | clarkb: I am using nodepool-builder FYI, with nodepool --no-builder flag | 18:34 |
| jeblair | nibalizer: that's probably a win since i don't understand data bindings yet :) | 18:34 |
| pabelanger | or, whatever the toggle is | 18:34 |
| clarkb | pabelanger: we are not, but I don't think that would affect the cron sceduler | 18:34 |
| *** Sukhdev has joined #openstack-infra | 18:34 | |
| nibalizer | jeblair: ok | 18:34 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Remove HP Cloud mirror https://review.openstack.org/271470 | 18:34 |
| anteaya | fungi: thanks for working on getting jjb jobs updated on all the jenkinsii | 18:34 |
| pabelanger | clarkb: let me setup a crontab for 5 mins out and see what happens | 18:34 |
| *** EricGonc_ has joined #openstack-infra | 18:34 | |
| krotscheck | I need one more core for https://review.openstack.org/#/c/270975/ (AFS Pypi mirror) | 18:35 |
| pabelanger | clarkb: but I agree we should add some crontab coverage | 18:35 |
| *** Swami_ has joined #openstack-infra | 18:35 | |
| nibalizer | jeblair: well it actually looks like 270966 is just a refactor to use public_hiera | 18:36 |
| jeblair | krotscheck: i've talked myself into making an hpcloud mirror | 18:36 |
| *** kzaitsev_mb has quit IRC | 18:36 | |
| jeblair | krotscheck: i don't want to make this more complicated | 18:36 |
| nibalizer | as a result you should be able to remove it from your stack | 18:36 |
| krotscheck | jeblair: You got it. | 18:36 |
| krotscheck | I'll abandon that patch | 18:36 |
| jeblair | krotscheck: i'm just going to sigh a lot while i do it. | 18:36 |
| *** jordanP has joined #openstack-infra | 18:36 | |
| jeblair | possibly mutter | 18:36 |
| anteaya | ha ha ha | 18:36 |
| fungi | jenkins02 is cleaned up and starting back up now | 18:36 |
| clarkb | pabelanger: I am going to chcek to see if I can see any of the other crons firing. | 18:36 |
| *** jsavak has quit IRC | 18:37 | |
| *** jsavak has joined #openstack-infra | 18:37 | |
| clarkb | pabelanger: ya so the periodic cleanup is running | 18:37 |
| fungi | jenkins06 completed updates successfully on this last pass. now down to 03, 04, 05, 07 | 18:38 |
| *** EricGonczer_ has quit IRC | 18:38 | |
| openstackgerrit | David Caro proposed openstack-infra/jenkins-job-builder: Added possibility to exclude project combinations https://review.openstack.org/175233 | 18:38 |
| jeblair | shade-launch-node still hasn't landed :( | 18:38 |
| *** EricGonczer_ has joined #openstack-infra | 18:38 | |
| *** electrofelix has quit IRC | 18:38 | |
| clarkb | 2016-01-21 14:14:00,052 DEBUG nodepool.NodePool: Updating all images. was the last time the update image was logged | 18:39 |
| clarkb | so we only skipped today's | 18:39 |
| AJaeger | zeih, now I'm around... | 18:39 |
| pabelanger | clarkb: guess we should add apscheduler.scheduler to logging.conf too since I see the logs with I start nodepoold manually | 18:40 |
| pabelanger | clarkb: yup, crontab fired here | 18:40 |
| pabelanger | cron: | 18:40 |
| pabelanger | image-update: '40 18 * * *' | 18:40 |
| clarkb | this is interesting | 18:40 |
| clarkb | nodepoold has been running since the 20th | 18:40 |
| clarkb | and it triggered the cron on the 21st | 18:41 |
| clarkb | but not the 22nd | 18:41 |
| clarkb | theory time | 18:41 |
| clarkb | that function is still running | 18:41 |
| *** ybathia has joined #openstack-infra | 18:41 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate: Update IRC channel to use #openstack-qa https://review.openstack.org/271474 | 18:41 |
| clarkb | and the geard falling over has it "stuck" | 18:41 |
| *** EricGonc_ has quit IRC | 18:41 | |
| jeblair | clarkb: okay _that_ one should have gotten a work_fail :) | 18:42 |
| *** jpr has joined #openstack-infra | 18:42 | |
| clarkb | jeblair: but the server fell over | 18:42 |
| clarkb | I don't understand how packet can get from worker to client is the server connection does not exist | 18:42 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Move pypi config into public hiera https://review.openstack.org/270966 | 18:42 |
| clarkb | we should definitely handle client disconnects | 18:43 |
| krotscheck | jeblair: Need a +A refresh on https://review.openstack.org/#/c/270966/ | 18:43 |
| krotscheck | Though it won't help until the jobs are registered | 18:43 |
| jeblair | clarkb: oh you're right i got it backwards again; so yeah, implement the disconnect handler | 18:43 |
| jeblair | krotscheck: nibalizer is working on a different version of that patch since apparently that won't actually work with our puppet config | 18:44 |
| krotscheck | clarkb: Do you have time to review https://review.openstack.org/#/c/270975/ ? It's the piece that sets up apache to host afs pypi things | 18:44 |
| krotscheck | jeblair: Oh. | 18:44 |
| clarkb | greghaynes: were you planning to implement client disconnect handling? | 18:45 |
| jeblair | what do we call the 'all of the clouds.yaml' file? | 18:45 |
| * krotscheck wasn't reading backscroll | 18:45 | |
| nibalizer | jeblair: we don't actually need that patch, jeblair as far as I can tell | 18:45 |
| nibalizer | its pureley a refactor | 18:45 |
| *** harlowja has quit IRC | 18:46 | |
| jeblair | nibalizer: yeah, krotscheck wanted to edit the list to fix a typo and wanted to avoid that needing a rebase | 18:46 |
| *** jordanP has quit IRC | 18:46 | |
| * AJaeger does a "pip 8.0.2 works" dance - first time since pip 8.0 release that a change for one repo passed! | 18:46 | |
| *** harlowja has joined #openstack-infra | 18:46 | |
| jeblair | nibalizer: if you want to wait, we can just edit the old list | 18:46 |
| * krotscheck can rebase on master, just needs to know what the path forward is. | 18:46 | |
| * anteaya lunches | 18:47 | |
| openstackgerrit | Rajini Ram proposed openstack-infra/puppet-openstackci: Updated gearman default port information https://review.openstack.org/267733 | 18:47 |
| clarkb | krotscheck: jeblair doesn't the code for https://review.openstack.org/#/c/270975/2/modules/openstack_project/manifests/mirror.pp currently live in http://git.openstack.org/cgit/openstack-infra/puppet-bandersnatch/tree/manifests/httpd.pp ? | 18:47 |
| clarkb | krotscheck: jeblair seems like we are undoing the module split out work | 18:48 |
| krotscheck | clarkb, jeblair: Maybe? The code in that predates things. | 18:48 |
| jeblair | clarkb: we're making a single mirror vhost for lots of mirrors. | 18:48 |
| jeblair | clarkb: it's more than a bandersnatch mirror | 18:48 |
| jeblair | clarkb: so this is openstack-specific | 18:48 |
| clarkb | jeblair: yes I don't dispute that | 18:49 |
| jeblair | i split the bandersnatch apache module out so we could stop using it. | 18:49 |
| jeblair | clarkb: also, just for the record, until yesterday we were not using puppet-bandersnatch *at all* because apparently no one got around to that. | 18:49 |
| clarkb | it looks like that accepts a doc root, configures appache, and writes a robot.txt though | 18:49 |
| clarkb | jeblair: ok | 18:49 |
| openstackgerrit | Andreas Jaeger proposed openstack/diskimage-builder: Remove argparse from requirements https://review.openstack.org/270356 | 18:50 |
| clarkb | I am just trying to wrap my head around why we seem to be copying code in the opposite direction of what we had committed to | 18:50 |
| jeblair | clarkb: right, i don't see how we can have the bandersnatch puppet module set up a vhost that also serves npm out of afs. | 18:50 |
| krotscheck | Would looking at the followup patch help? https://review.openstack.org/#/c/270979/ | 18:50 |
| jeblair | clarkb: i bet there's a puppet way to compose an apache vhost that way, but it will probably involve us making some kind of puppet-mirror class that accepts all of these paths | 18:51 |
| clarkb | jeblair: oh I see, this is thinking ahead to when apache needs to server / and not /pypi | 18:51 |
| jeblair | clarkb: ya krotscheck's 2nd patch will illustrate the problem well | 18:51 |
| jeblair | clarkb: maybe we can make puppet-mirror-things in the future if this all works out | 18:51 |
| * krotscheck is a little worried that after building these awesome mirrors, someone downstream is going to think: Hey, I can have a free caching AFS mirror! | 18:52 | |
| *** esikachev has joined #openstack-infra | 18:52 | |
| *** e0ne has joined #openstack-infra | 18:52 | |
| jeblair | in the mean time, folks can use bandersnatch::mirror + bandersnatch::httpd and get what we have today | 18:52 |
| jeblair | krotscheck: maybe that will be okay? i kind of hope it will? | 18:53 |
| fungi | AJaeger: direct your thanks for pip 8.0.2 to dstufft. he deserves it! | 18:53 |
| krotscheck | jeblair: I... kinda think so too? | 18:53 |
| krotscheck | jeblair: Maybe a thing we can talk to the linux foundation about and say: Hey, how about we do unified mirrors for all the things? | 18:53 |
| krotscheck | Maybe I'm ambitious? | 18:53 |
| fungi | yay! jenkins07 updates complete! now we wait for 03-05 | 18:53 |
| dimsum__ | dstufft : thanks! | 18:53 |
| clarkb | jeblair: krotscheck: last question http://pypi.dfw.openstack.org/ does not yet have a /pypi. Do we need to wait on image rebuilds or a nodepool reconfigure to set the paths on the host sto allow both /simple and /pypi/simple ? | 18:54 |
| jeblair | krotscheck: they'd just change the bylaws | 18:54 |
| krotscheck | jeblair: ouch | 18:54 |
| krotscheck | jeblair: ;) | 18:54 |
| dimsum__ | jeblair : ouch :) | 18:54 |
| clarkb | krotscheck: osuosl sort of already does massive mirroring for kernel.org | 18:54 |
| fungi | if we get down to a couple of jenkins masters still updating, i'll put them in prepare for shutdown and then we can start rechecking/enqueuing things on the assumption they'll work | 18:54 |
| krotscheck | clarkb: We're provisioning brand new hosts. | 18:54 |
| clarkb | krotscheck: right but we configure our test slaves to use /simple today | 18:55 |
| clarkb | krotscheck: so in order to safely transition we need to configure them to use /simple and /pypi/simple | 18:55 |
| jeblair | clarkb: i think we'll change the ready script to use the new path and the new host at the same time? | 18:55 |
| clarkb | jeblair: that will work too | 18:55 |
| krotscheck | clarkb: You mean this? https://review.openstack.org/#/c/238756/ | 18:55 |
| jeblair | ya that :) | 18:56 |
| clarkb | right so the apache config change won't apply to the existing mirrors? | 18:56 |
| krotscheck | (It's WIP'd because merging that might be a problem if we don't have the instances yet) | 18:56 |
| clarkb | as long as we don't change existing mirrors and don't merge 238756 until new mirrors are configured we will be good | 18:56 |
| krotscheck | clarkb: That's the idea. | 18:56 |
| jeblair | aha! it's supposed to be called 'all-clouds.yaml'. it exists in system-config as an erb, but nothing installs it! | 18:57 |
| krotscheck | jeblair: AFK for a shower. Future me will thank me. | 18:57 |
| jeblair | this is why i can't find it. | 18:57 |
| clarkb | ok and current mirrors use a pypi.pp not mirror.pp | 18:57 |
| clarkb | so no cross contamination | 18:58 |
| jeblair | that's the idea | 18:58 |
| clarkb | approved | 18:58 |
| *** eil397 has joined #openstack-infra | 18:58 | |
| *** PsionTheory has joined #openstack-infra | 19:01 | |
| *** Sukhdev has quit IRC | 19:02 | |
| fungi | jenkins03-05 are all going to need to be retried... proxy/api errors again. le sigh | 19:03 |
| clarkb | hpcloud-b1 fedora-23 image upload still in progress | 19:04 |
| fungi | i'm tempted to put all 3 of them in prepare for shutdown, but that leaves us at close to 50% of our usual compliment of masters, which means nearly double the slaves on them | 19:04 |
| fungi | which could create yet more new and different problems for us | 19:04 |
| *** _diana_ has joined #openstack-infra | 19:05 | |
| clarkb | you can run the restart playbook against them and it will do one at a time | 19:06 |
| clarkb | (it will also run bright and early tomorrow utc) | 19:06 |
| *** bpokorny_ has quit IRC | 19:07 | |
| *** bpokorny has joined #openstack-infra | 19:07 | |
| *** _nadya_ has joined #openstack-infra | 19:07 | |
| *** ccarmack has quit IRC | 19:08 | |
| fungi | righht | 19:09 |
| fungi | which is partly why this is getting painful. we're at the edge of where we'd automatically restart them anyway because we expect them to be pretty terribly degraded | 19:10 |
| fungi | bad timing | 19:10 |
| clarkb | and perhaps we should increase the frequency of that cron to twice a week | 19:10 |
| clarkb | or every other day or something | 19:10 |
| * greghaynes returns from phone call | 19:11 | |
| jeblair | um | 19:12 |
| jeblair | do we still need to generate puppet certs? | 19:12 |
| *** rcernin has quit IRC | 19:12 | |
| fungi | i hope not? | 19:12 |
| jeblair | also, has launch-node been updated to run the right kind of puppet? | 19:13 |
| *** esker has joined #openstack-infra | 19:13 | |
| fungi | puppet apply doesn't connect to the puppetmaster at all, right? | 19:13 |
| jeblair | come to think of it, have we worked through any of this yet? | 19:13 |
| krotscheck | jeblair: back | 19:13 |
| krotscheck | clarkb: WOO thanks | 19:14 |
| jeblair | maybe it's good we still have a puppetmaster for a little bit? | 19:14 |
| greghaynes | clarkb: so what was the update on nodepool? | 19:14 |
| greghaynes | lots of backscroll... | 19:14 |
| clarkb | greghaynes: my theory is cron didn't fire because it is still running from before | 19:14 |
| jeblair | mordred: i have some questions for you about puppet and launching nodes when you have a sec :) | 19:14 |
| krotscheck | nibalizer: So, what was the verdict on public hiera things? | 19:14 |
| greghaynes | clarkb: oh, huh. | 19:14 |
| clarkb | greghaynes: basically that function is still on a stack somewhere happily wasting cpu time | 19:14 |
| mordred | jeblair: good morning! | 19:14 |
| clarkb | greghaynes: I am manually performing an upload to confirm that that works | 19:15 |
| clarkb | greghaynes: it is slow | 19:15 |
| jeblair | also, shade-launch-node just bombed on rax with something about security groups | 19:15 |
| greghaynes | clarkb: if so it must run in a separate thread? | 19:15 |
| clarkb | greghaynes: yes I think apscheduler makes a thread to run them in | 19:15 |
| mordred | jeblair: awesome - pastebin? | 19:15 |
| jeblair | clarkb, greghaynes: correct | 19:15 |
| jeblair | mordred: http://paste.openstack.org/show/484734/ | 19:15 |
| greghaynes | clarkb: jeblair hrm, ok. That seems plausible | 19:15 |
| nibalizer | krotscheck: you shouldn't need public hiera | 19:16 |
| jeblair | clarkb: look for self.log.debug("Waiting for images to complete building.") | 19:16 |
| clarkb | greghaynes: so anyways, I think we need to handle the client disconnect before worrying too much about anything else | 19:16 |
| nibalizer | we can have the public hirea databindings conversation separate from the mirrors work you are doing | 19:16 |
| jeblair | clarkb: and then self.log.debug("Done waiting for images to complete building.") | 19:16 |
| nibalizer | so if you could remove that patch from your stack that would be the best | 19:16 |
| krotscheck | nibalizer: Ok, I'll rebase on master then. | 19:16 |
| jeblair | clarkb: if you see the first and not the second, that is very strong support for your theory | 19:16 |
| mordred | jeblair: sigh. that should not be an error at all ... also, I think we may have fixed that as a bug a little while ago | 19:16 |
| nibalizer | krotscheck: thanks | 19:16 |
| jeblair | mordred: brb | 19:16 |
| mordred | jeblair: are we up to date ... I'll look | 19:16 |
| clarkb | jeblair: checking | 19:16 |
| krotscheck | mordred: nibalizer wants to have a conversatoin with you about public hiera things. | 19:16 |
| mordred | krotscheck: I'm excited about that | 19:16 |
| krotscheck | mordred: Specifically about https://review.openstack.org/#/c/270966/ | 19:17 |
| greghaynes | clarkb: Yep, I have been trying to figure out the best way to do that... | 19:17 |
| mordred | nibalizer: sup? | 19:17 |
| nibalizer | mordred: using data bindings | 19:17 |
| clarkb | jeblair: greghaynes yup no done message but there is a waiting message | 19:17 |
| mordred | nibalizer: oh - the databindings is not actually related to mirrors - I just used that as an opportunity to show what it looks like | 19:18 |
| nibalizer | ya | 19:18 |
| mordred | we do not need that patch for the mirror work to progress | 19:18 |
| nibalizer | yes | 19:18 |
| mordred | oh - sorry. I picked up the topic | 19:18 |
| mordred | I'm a bad person | 19:18 |
| greghaynes | clarkb: so question - if nodepool handled the disconnect by trying to 'clean up', should it put the images into error state? | 19:18 |
| clarkb | greghaynes: I think we need to override that method on the client that jeblair linked and have that remove any jobs that had not started yet. Then possibly resubmit them once gearman is back | 19:18 |
| nibalizer | no worries :) | 19:18 |
| *** mrmartin has joined #openstack-infra | 19:18 | |
| greghaynes | clarkb: the problem is that doesn't solve the delete not working issue | 19:18 |
| mordred | nibalizer: are you against databindings? | 19:18 |
| clarkb | greghaynes: well I think we only add the images to the db once the job is picked up | 19:18 |
| mordred | nibalizer: or you just want to have a discussion? | 19:19 |
| clarkb | greghaynes: so we only deal with the subset that had a worker attached to them | 19:19 |
| greghaynes | clarkb: yep, thats what I was thinking too, the question then is what state are images in before then | 19:19 |
| clarkb | greghaynes: I think no state | 19:19 |
| clarkb | unless we want to have it retry | 19:19 |
| nibalizer | mordred: i am wanting a discussion, hopefully with some agreed upon guidelines on where to use them, how, and how to discover/document when data bindings are being used | 19:20 |
| *** Ramanjaneya has quit IRC | 19:20 | |
| openstackgerrit | Andre keedy proposed openstack-infra/project-config: Add new project(shovel) to openstack https://review.openstack.org/269411 | 19:20 |
| mordred | nibalizer: ++ | 19:20 |
| greghaynes | clarkb: nodepool needs to know that it already submitted the job for an image though | 19:20 |
| greghaynes | clarkb: so itll need some internal state tracking | 19:20 |
| clarkb | greghaynes: not if the server connection goes away | 19:21 |
| clarkb | greghaynes: that job will never run | 19:21 |
| greghaynes | clarkb: I mean for while it is waiting | 19:21 |
| *** rbrndt has quit IRC | 19:21 | |
| greghaynes | clarkb: the buildMissingImages method needs to check 'is this image waiting/building?' | 19:21 |
| clarkb | oh I see so you don't over submit | 19:21 |
| greghaynes | ye0p | 19:21 |
| clarkb | I guess a new queued state ? | 19:22 |
| clarkb | queued, building, ready, delete ? | 19:22 |
| greghaynes | Interestingly - it would be valid for that state to be totally in memory | 19:22 |
| greghaynes | since if nodepool dies any queued jobs do too | 19:22 |
| clarkb | yup | 19:22 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/system-config: Fixed typo in mirrors https://review.openstack.org/271456 | 19:22 |
| fungi | we already had that issue with image builds even before workers | 19:23 |
| krotscheck | jeblair, nibalizer ^^ | 19:23 |
| greghaynes | fungi: yep. I think somehow the builders is causing us to hit that case though | 19:23 |
| greghaynes | which is :( | 19:23 |
| mordred | jeblair: https://review.openstack.org/#/c/247099/5/launch/shade-launch-node.py | 19:24 |
| *** johnsom_ has joined #openstack-infra | 19:24 | |
| clarkb | greghaynes: also image uploads "seem" slower | 19:24 |
| mordred | jeblair: that's where I have updates related to puppet apply for launch node | 19:24 |
| clarkb | it is possible that a distinct process solves that problem | 19:24 |
| *** sabeen2 has quit IRC | 19:24 | |
| greghaynes | clarkb: ya, that should be pretty easy for you to test | 19:24 |
| greghaynes | clarkb: just run nodepool-builder | 19:24 |
| clarkb | ya | 19:24 |
| johnsom_ | I'm looking for some devstack plugin.sh help. Not sure if I'm in the right channel. | 19:25 |
| *** chmouel_ is now known as chmouel | 19:25 | |
| clarkb | johnsom_: typically devstack is best handled by #openstack-qa | 19:25 |
| AJaeger | fungi, he does. | 19:25 |
| johnsom_ | clarkb Cool, thanks! | 19:25 |
| krotscheck | is it useful for me to recheck something that has a NOT_REGISTERED job right now? (centos devstack) | 19:25 |
| mordred | jeblair: it needs to be updated - sorry | 19:25 |
| AJaeger | dstufft: thanks for pip 8.0.2 - looks great now! | 19:25 |
| mordred | jeblair: I should have topic'd that puppet-apply | 19:25 |
| *** achanda has quit IRC | 19:26 | |
| fungi | krotscheck: what's the job name and i can check for you | 19:26 |
| *** yamahata has joined #openstack-infra | 19:26 | |
| krotscheck | fungi: gate-openstackci-beaker-centos7-dsvm and gate-infra-puppet-apply-dsvm-centos7 | 19:26 |
| greghaynes | clarkb: ok, I'll play around with waiting until the job is accepted to add images to the db | 19:26 |
| clarkb | greghaynes: cool, and also handle the client disconnect (maybe just by raising an expcetion) so that the cron thread can die | 19:27 |
| fungi | krotscheck: neither of those job names are registered in zuul at all. do you have any reason to believe they may have been added relatively recently? | 19:27 |
| *** sabeen1 has joined #openstack-infra | 19:27 | |
| *** markvoelker has quit IRC | 19:27 | |
| fungi | krotscheck: i have a feeling those are existing jobs and we haven't built/added any new centos7 nodes since the zuul restart a little while ago due to demand ratios | 19:28 |
| Swanson | Something called vahana seems to be choking my nodepool node creation. Couldn't find remote ref master looks to be the error... | 19:28 |
| krotscheck | fungi: No reason, except that the only tiem Iv'e seen those errors is when a new job's been added. | 19:28 |
| krotscheck | fungi: Relevant patch is here https://review.openstack.org/#/c/270975/ | 19:29 |
| fungi | krotscheck: it also happens for rare node types soon after a zuul restart | 19:29 |
| krotscheck | fungi: Good to know :) | 19:29 |
| krotscheck | So I can recheck? | 19:29 |
| *** sabeen3 has joined #openstack-infra | 19:29 | |
| fungi | krotscheck: right now it won't do any good | 19:29 |
| krotscheck | Alright, I can wait :) | 19:30 |
| fungi | clarkb: greghaynes: nodepool list is giving me "IOError: [Errno 32] Broken pipe" | 19:30 |
| * krotscheck starts looking for the appropriate yak | 19:30 | |
| fungi | clarkb: greghaynes: known issue? | 19:30 |
| greghaynes | fungi: uh oh. gearman connection likely | 19:30 |
| fungi | krotscheck: yes, i've just been handed a new yak myself, it seems | 19:30 |
| clarkb | ya Ithink that means we broken the gearman connection | 19:30 |
| * AJaeger wishes everybody a great weekend and signs off... | 19:30 | |
| fungi | can't check to see if we're building the missing node types | 19:30 |
| krotscheck | fungi: It's all fresh and furry! | 19:30 |
| fungi | AJaeger: have a great weekend! | 19:30 |
| AJaeger | thanks, fungi | 19:31 |
| anteaya | AJaeger: bye | 19:31 |
| *** bpokorny has quit IRC | 19:31 | |
| greghaynes | so, we still don't have any great theories on why were getting these gearman disconnects... its a bit concerning | 19:31 |
| clarkb | the gearman server doesn't seem to know yet | 19:31 |
| greghaynes | I wonder if we should colocate another geard on the nodepool host | 19:31 |
| fungi | clarkb: nibalizer: nevermind. i was piping to a failed command. would help if i could type "grep" accurately | 19:31 |
| greghaynes | and use that for builders | 19:31 |
| *** sabeen1 has quit IRC | 19:32 | |
| *** jpr has quit IRC | 19:32 | |
| fungi | broken pipe exception is how nodepool's now reacting to having its stdout piped to a nonexistent process | 19:32 |
| greghaynes | oh, awesome | 19:32 |
| fungi | which makes some sense | 19:32 |
| fungi | just a slight behavior change... i think | 19:32 |
| *** hashar has joined #openstack-infra | 19:32 | |
| greghaynes | I like that a lot better than more gearman connection issues :) | 19:32 |
| *** ari has quit IRC | 19:33 | |
| fungi | so we apparently have no bare-centos7 nodes in existence, not even trying to be built | 19:33 |
| fungi | oh, we have no bare-centos7 images either | 19:34 |
| fungi | which might be why | 19:34 |
| krotscheck | How very zen of us. | 19:34 |
| clarkb | I don't know that we ever had them? | 19:34 |
| clarkb | we had them for centos6 | 19:34 |
| fungi | aha, puppet apply is on dsvm, and so is beaker i guess | 19:34 |
| fungi | yeah, sorry, i'm full of dead-end choices this afternoon | 19:35 |
| openstackgerrit | Merged openstack-infra/project-config: Convert kolla-mesos to linters https://review.openstack.org/270990 | 19:35 |
| Ng | when the new project guide says "ask the infra team to add you to both [gerrit] groups", how would the infra team like to be asked that? :) | 19:35 |
| *** bardia has quit IRC | 19:35 | |
| krotscheck | Ng: I like donuts. I can't speak for anyone else. | 19:35 |
| *** fawadkhaliq has quit IRC | 19:35 | |
| clarkb | greghaynes: ok if you are going to poke at the queued state handling I should go ahead and work on running a separate daemon to see if that addresses performance of uploads | 19:35 |
| greghaynes | clarkb: sounds good | 19:35 |
| fungi | so anyway, we _do_ have devstack-centos7 nodes ready, from what nodepool thinks, but for hours (since before the zuul restart)... so maybe i need to delete and let a replacement get relaunched | 19:35 |
| pabelanger | So, correct me if I am wrong, but I don't see how reno will work outside a git repo. http://pastebin.com/64QdK4jA | 19:35 |
| anteaya | Ng: share the link of the patch that created the groups as well as your gerrit username or email | 19:36 |
| Ng | krotscheck: interesting, I'm not sure how good I'm going to be at writing the usernames on in icing ;) | 19:36 |
| anteaya | Ng: assuming you are the owner of the patch who created the groups | 19:36 |
| Ng | anteaya: thanks :) | 19:36 |
| krotscheck | Ng: I'll eat any failed attempts. :) | 19:36 |
| anteaya | welcome | 19:36 |
| pabelanger | dhellmann: ^ | 19:36 |
| clarkb | pabelanger: if it is running git log as part of docs builds then it won't | 19:36 |
| fungi | Ng: also great to see you suddenly appear again, as if from nowhere! are you around to stay for a change? ;) | 19:36 |
| dhellmann | pabelanger : it does not | 19:37 |
| fungi | Ng: your presence is missed | 19:37 |
| Ng | fungi: I had some distractions for a while with corp bureaucracy, but now I have a new project to work on :) | 19:37 |
| dhellmann | pabelanger : there is an as-yet unimplemented idea to tie reno to setuptools and generate some data files that would substitute for the git log commands when building docs from a tarball or sdist | 19:37 |
| fungi | Ng: having something to work on is a good thing. welcome back to surreality | 19:37 |
| dhellmann | pabelanger : I hope to get to that by the end of this cycle, but if you have time and inclination I'd be happy to go over the plan if you wanted to start exploring implementation | 19:38 |
| *** annegent_ has quit IRC | 19:38 | |
| mordred | dhellmann, pabelanger: there's also bee a different request to run sphinx as part of sdist build by some folks | 19:38 |
| pabelanger | dhellmann: okay, thanks for the info. Running into packaging issues using reno, and wanted to confirm | 19:38 |
| mordred | so that built docs were in the source tarball | 19:38 |
| mordred | but I think that needs design | 19:38 |
| pabelanger | guess I'll drop reno release notes for now | 19:38 |
| *** mrmartin has quit IRC | 19:38 | |
| Ng | https://review.openstack.org/#/c/266549/ is the change that created the project, the gerrit usernames for -core and -release are jclark (ptl) and cmsj (me :) | 19:38 |
| fungi | dhellmann: mordred: if it's my request, it's particularly that i'd love auto-generated manpages to be available in sdist tarballs | 19:39 |
| dhellmann | fungi : that's an interesting idea, but I think it may be orthogonal to the reno question | 19:39 |
| fungi | project's manpage in project's release tarball... seems like goodness | 19:39 |
| mordred | I also do | 19:39 |
| anteaya | Ng: you are the owner of the patch so you get added and you can add whoever you like | 19:39 |
| Ng | anteaya: woot | 19:40 |
| anteaya | Ng: when a gerrit admin has a moment they will add you | 19:40 |
| mordred | just thought I'd mention it because both of them have something to do with altering the current sdist generation concept | 19:40 |
| dhellmann | pabelanger : if you want to do something short term, and you have access to the git repo at any point, you could use the reno report command to generate a flat document | 19:40 |
| fungi | dhellmann: it was more in regards to mordred's sphinx invocation at sdist generation time point | 19:40 |
| dhellmann | pabelanger : though if you don't have the git repo anywhere in the pipeline, that won't help | 19:40 |
| mordred | dhellmann, fungi: so at least considering the whole picture might not be a bad idea as we look to planning impl | 19:40 |
| mordred | I do not thin the impl of both needs to be tied together | 19:40 |
| dhellmann | mordred : sure | 19:40 |
| openstackgerrit | Devananda van der Veen proposed openstack-infra/elastic-recheck: Add query for Ironic bug 1537076 https://review.openstack.org/271496 | 19:40 |
| openstack | bug 1537076 in Ironic "Timed out waiting for Nova hypervisor-stats count >= 1 due to Nova Unable to establish connection to http://127.0.0.1:35357/v2.0/tokens" [High,Confirmed] https://launchpad.net/bugs/1537076 | 19:40 |
| *** krtaylor has quit IRC | 19:41 | |
| dhellmann | mordred : I haven't decided whether to add a uses_reno flag like we have for uses_pbr, or to build the step into pbr itself | 19:41 |
| dhellmann | having pbr rely on reno feels icky, though | 19:41 |
| mordred | dhellmann: we do have things in pbr that use the thing if the thing is there | 19:41 |
| pabelanger | dhellmann: right, I don't have a git repo, packaging tarballs inside mock | 19:41 |
| dhellmann | mordred : ok, so that may be less icky than I imagined | 19:41 |
| fungi | opportunistic imports | 19:41 |
| mordred | dhellmann: so pbr could happily add the feature of doing reno things if reno is importable | 19:41 |
| dhellmann | pabelanger : yeah, that's what I figured | 19:42 |
| mordred | dhellmann: it's how we do the build_sphinx command currently | 19:42 |
| dhellmann | ok, that makes sense, then | 19:42 |
| pabelanger | mordred: Ya, I would be happy with that. | 19:42 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Summit Application API https://review.openstack.org/221964 | 19:42 |
| mordred | I tink it would be a good idea - because we already rely on git being available for pbr sdist tasks - so it's not an undue burden or anything | 19:42 |
| mordred | it would be neat if nova had a flag to mark vms as "ephemeral, don't care" or "important, please care about me during maint events" | 19:44 |
| *** pradk has quit IRC | 19:45 | |
| mordred | just got a maint email from bluebox saying that they need to correct something with the nova volume backing store - and that they'd be suspending and migrating vms ... I want to be able to tell them "srrsly - don't bother migrating anything in the nodepool project" | 19:45 |
| * fungi thinks it would be interesting if nova had support for vax/vms ;) | 19:45 | |
| *** pradk has joined #openstack-infra | 19:45 | |
| mordred | fungi: :) | 19:45 |
| anteaya | mordred: should we shut off nodepool to bluebox for the time being? | 19:45 |
| *** iurygregory has joined #openstack-infra | 19:46 | |
| mordred | anteaya: not yet: Monday (1.25.16) at 1pm PST/ 4pm EST | 19:46 |
| fungi | (i wonder how often the openvms devs have to explain that vms has nothing to do with virtual machines | 19:46 |
| fungi | ) | 19:46 |
| mordred | fungi: did I tell you about the customer call I was on a few years ago where they guy asked if this cloud thing would help him manage his legacy AIX and IRIX things? | 19:46 |
| *** placko_ has quit IRC | 19:46 | |
| anteaya | mordred: okay, that is the scheduled maintainance for the raid correction? | 19:46 |
| mordred | anteaya: I do not think so | 19:46 |
| anteaya | oh something else | 19:47 |
| anteaya | mordred: what did you tell him this cloud thing would manage for him? | 19:47 |
| *** rbrndt has joined #openstack-infra | 19:47 | |
| fungi | mordred: i would have answered "yes, it will save you time elsewhere that you can then spend on your legacy unix systems" | 19:47 |
| *** iurygregory has left #openstack-infra | 19:47 | |
| * fungi spent too much time hearing sleazy tech salespeople over the cube wall | 19:48 | |
| anteaya | I would have figured you for an earphone person | 19:48 |
| mordred | fungi: nicely done | 19:48 |
| *** gordc has quit IRC | 19:48 | |
| mordred | anteaya: I told him "nope. it will not help you with that at all" | 19:49 |
| anteaya | mordred: ha ha ha | 19:49 |
| fungi | anteaya: i was under strict orders to smile and nod in customer meetings, and only answer questions when directly asked by our "sales engineers" | 19:49 |
| openstackgerrit | gordon chung proposed openstack-infra/project-config: fix ceilometerclient gate https://review.openstack.org/271379 | 19:49 |
| anteaya | fungi: ha ha ha | 19:49 |
| anteaya | fungi: that explains so much | 19:49 |
| *** baoli has quit IRC | 19:49 | |
| mordred | fungi: you konw - if tey'd gotten better sales engineers, they would not have had to suck you in to customer meetings | 19:49 |
| mordred | I mean, isn't the point of a sales engineer to be tech enough to answer the questions so that you don't have to bother the actual engineers? | 19:50 |
| fungi | mordred: they no longer have me to suck into customer meetings, so for their sake i hope they got some better sales engineers | 19:50 |
| asselin | clarkb, greghaynes so the gearman port used by nodepool is 4730 same as zuul? if they're both on a single blade they should use a different port, correct? or is nodepool using zuul's gearman server? | 19:50 |
| mordred | fungi: ++ | 19:50 |
| clarkb | asselin: one geard for both | 19:50 |
| mordred | and in the darkness something something | 19:51 |
| asselin | clarkb, which process runs geard? | 19:51 |
| *** ybathia has quit IRC | 19:51 | |
| clarkb | nibalizer: I am reading https://review.openstack.org/#/c/240093/7/manifests/builder.pp and trying to figure out how we would colocate builder and nodepoold on same host, I don't think we can do that with your change | 19:51 |
| *** unicell has left #openstack-infra | 19:51 | |
| clarkb | asselin: by default zuul | 19:51 |
| clarkb | asselin: but you don't have to do it that way | 19:51 |
| fungi | krotscheck: just to keep you updated, still not time to recheck. i deleted the one ready devstack-centos7 node in nodepool but it isn't building a replacement yet. extreme inbalance in demand ratios | 19:52 |
| nibalizer | clarkb: i think i agree with you just a sec | 19:52 |
| fungi | er, imbalance | 19:52 |
| *** achanda has joined #openstack-infra | 19:53 | |
| openstackgerrit | Merged openstack-infra/project-config: Use linters job for keystonemiddleware starting with mitaka https://review.openstack.org/269261 | 19:54 |
| nibalizer | clarkb: yea so what happens if we try to colocate them is the nodepool::install class will conflict | 19:54 |
| *** arxcruz has quit IRC | 19:54 | |
| clarkb | nibalizer: right | 19:55 |
| clarkb | and colocation is desireable | 19:55 |
| nibalizer | but if we don't do it that way it ends up looking kind of nasty | 19:55 |
| clarkb | nibalizer: we could do hacks of if not defined | 19:55 |
| clarkb | in both places | 19:55 |
| nibalizer | ya | 19:55 |
| nibalizer | or pass a 'manage_installation' flag to ::builder | 19:55 |
| *** abregman|afk is now known as abregman | 19:55 | |
| nibalizer | hacks, all | 19:55 |
| mordred | I so love this 'feature' of puppet | 19:56 |
| nibalizer | why not just commit and spin up an external builder? | 19:56 |
| mordred | it's one of my favorites | 19:56 |
| nibalizer | mordred: well so | 19:56 |
| clarkb | nibalizer: because that doesn't help asselin | 19:56 |
| nibalizer | to bring it full circle, databindings fixes this | 19:56 |
| mordred | nibalizer: WOOT | 19:56 |
| mordred | nibalizer: then why don't we do this with databindings? :) | 19:56 |
| nibalizer | becausse you say 'include nodepool::install' everywhere and set parameters with lookups | 19:56 |
| mordred | nibalizer: ++ | 19:56 |
| clarkb | well we don't want two intalls | 19:56 |
| clarkb | we want 1 | 19:57 |
| mordred | clarkb: right | 19:57 |
| *** dims has joined #openstack-infra | 19:57 | |
| clarkb | so I don't think databindings actually help us | 19:57 |
| nibalizer | clarkb: calling 'include nodepool::install' twice will only install it once | 19:57 |
| *** dimsum__ has quit IRC | 19:57 | |
| mordred | clarkb: databindings is a way to take advantage of the fact that two different include statements do not conflict | 19:57 |
| nibalizer | its a way to pass parameters yes | 19:57 |
| mordred | but instead do what we've always wanted to happen everywhere | 19:57 |
| clarkb | wow thats ok | 19:57 |
| mordred | becuase as soon as we move from include to class, we have to do an if-defined | 19:57 |
| mordred | yeah | 19:57 |
| clarkb | thats a language bug :) | 19:58 |
| mordred | clarkb: TOTALLY | 19:58 |
| mordred | however, databindings is the bugfix | 19:58 |
| mordred | and we can use it now | 19:58 |
| mordred | but - as nibz says, we should sketch out some 'here's when and how' ideas | 19:58 |
| nibalizer | also if you look at my comments on the databinds change | 19:58 |
| nibalizer | there is a way to get the functionality without actually using databindings | 19:59 |
| * nibalizer -> lunch | 19:59 | |
| mordred | clarkb: if you havne't seen an example: https://review.openstack.org/#/c/270966/2 | 19:59 |
| clarkb | mordred: ya I am not a fan :) | 19:59 |
| clarkb | for solving this issue at least | 19:59 |
| mordred | k | 19:59 |
| clarkb | the issue is that we want to express what the intsall should look like | 20:00 |
| clarkb | and we can't do that without hiding all of the details in a file somewhere else in a nother language that is implicitly looked up | 20:00 |
| clarkb | which is what I mean about being a language bug | 20:00 |
| mordred | sure. but now you're just expressing dislike for puppet | 20:00 |
| mordred | yeah | 20:00 |
| mordred | it's a bug | 20:00 |
| *** aysyd has quit IRC | 20:00 | |
| mordred | I agree | 20:00 |
| clarkb | the if not defined solution is more readable to me at least | 20:01 |
| mordred | but it's always been a bug - it's puppet exposing internal data model things in to the DSL | 20:01 |
| clarkb | puppet people don't like it but at least you only have to read one file to understand what is going on | 20:01 |
| clarkb | however that said, I think there is a better way to fix this | 20:01 |
| clarkb | builder shouldn't include install | 20:01 |
| clarkb | init should make an install | 20:02 |
| clarkb | then have a nodepool::daemon and nodepool::builder | 20:02 |
| *** ddieterly has quit IRC | 20:02 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: add support for package-installs to gentoo https://review.openstack.org/270597 | 20:02 |
| clarkb | that both require init and configure their respective services assuming an install exists via init | 20:02 |
| clarkb | then you can do include nodepool include nodepool::daemon nodepool::builder | 20:03 |
| *** piet has quit IRC | 20:04 | |
| clarkb | nibalizer: when you get back from lunch what do you think of reorganizing like ^ | 20:04 |
| openstackgerrit | Jay_Clark proposed openstack-infra/project-config: adding project config for openstack-salt-formulas. The openstack-salt team is beginning work on providing Salt forumlas for Openstack, and a git repo is necessary to host our code. https://review.openstack.org/271509 | 20:05 |
| mordred | clarkb: you have the exact same problem | 20:06 |
| mordred | clarkb: the install in init needs parameters | 20:06 |
| clarkb | you don't | 20:06 |
| mordred | you don't need parameters? | 20:06 |
| clarkb | you do but you never call those classes >1 on a single host | 20:07 |
| clarkb | the problem now is we would be running init, daemon, and builer on one host but init daemon and builder all install | 20:07 |
| clarkb | (well right now init and daemon are the same thing) | 20:07 |
| mordred | so - you'd have to have the calling context call nodepool::init and nodepool::daemon and nodepool::builder from the calling context | 20:07 |
| mordred | which is fine | 20:07 |
| clarkb | yes | 20:07 |
| *** e0ne has quit IRC | 20:08 | |
| clarkb | depending on which of the two services you want running on that host | 20:08 |
| mordred | it removes the "class nodepool:builder should get me a working thing" | 20:08 |
| clarkb | yes but thats ok | 20:08 |
| mordred | and move to "I need to call an install class and a service class" | 20:08 |
| mordred | if that seems more readable to you, then sure | 20:08 |
| clarkb | right which seems to be a fairly common thing iwth puppet | 20:08 |
| * mordred does not have the strong opinions on this - thinks all of the options blow | 20:08 | |
| clarkb | the mysql module for example | 20:08 |
| clarkb | or paache even | 20:09 |
| clarkb | you don't get working service with a single include | 20:09 |
| clarkb | you have to construct the install and config and service | 20:10 |
| fungi | krotscheck: feel free to recheck now. nodepool finally completed building a new devstack-centos7 node a few minutes ago, and those jobs are finally showing up registered in gearman now | 20:10 |
| fungi | the jenkins04 jobs update finally failed out on an api error, so i've resumed it | 20:10 |
| openstackgerrit | Jay_Clark proposed openstack-infra/project-config: adding project config for openstack-salt-formulas. https://review.openstack.org/271509 | 20:12 |
| asselin | clarkb, greghaynes is it a known issue that you can't delete a buidling dib image? Also can't seem to see anything about that in the logs. | 20:13 |
| *** ldnunes has quit IRC | 20:14 | |
| clarkb | asselin: yes, if the image doesn't have anything on disk then currently the service doesn't register a delete job for it | 20:14 |
| *** pcaruana has quit IRC | 20:14 | |
| clarkb | needs to be addressed by cleaning up db and stopping any running jobs | 20:14 |
| *** leakypipes has quit IRC | 20:14 | |
| asselin | ok...and it hangs....ok will do that | 20:15 |
| *** achanda has quit IRC | 20:16 | |
| *** esker has quit IRC | 20:16 | |
| *** esker has joined #openstack-infra | 20:17 | |
| *** tonytan4ever has quit IRC | 20:17 | |
| *** dprince has quit IRC | 20:18 | |
| *** dprince has joined #openstack-infra | 20:18 | |
| openstackgerrit | Jay_Clark proposed openstack-infra/project-config: adding project config for openstack-salt-formulas. https://review.openstack.org/271509 | 20:18 |
| fungi | Ng: okay, the several irons i have in the fire now are all heating. has anyone taken care of your initial group member request yet? | 20:19 |
| Ng | fungi: I don't believe so | 20:20 |
| fungi | oh, it looks like they haven't actually been created by manage-projects yet | 20:21 |
| *** edunham has joined #openstack-infra | 20:21 | |
| fungi | and i'm getting an internal server error from puppetboard.openstack.org now | 20:22 |
| Ng | fungi: ah right. I'm pretty much at the end of my day, so we can try again tomorrow :) | 20:22 |
| anteaya | wooooo, more fun | 20:22 |
| anteaya | Ng: no, fungi travels | 20:22 |
| *** sdake_ has joined #openstack-infra | 20:22 | |
| fungi | Ng: thanks, i mainly need to dig into what bit(s) of our automation have gone sideways | 20:22 |
| fungi | anteaya: well, i'm not flying out until sunday | 20:22 |
| anteaya | ah sorry | 20:22 |
| fungi | puppetboard magically is back to working again! | 20:23 |
| fungi | i guess it was just out to lunch for a few minutes while i retried a bunch | 20:23 |
| *** e0ne has joined #openstack-infra | 20:24 | |
| *** EricGonc_ has joined #openstack-infra | 20:24 | |
| fungi | review.o.o last puppeted at 19:46 utc, so ~40 minutes ago | 20:24 |
| jeblair | krotscheck, mordred: sorri i disappeared... and now i need to disappear again for food. but i'll be back after that to continue | 20:25 |
| fungi | jenkins03 job updates failed out on an api error again and have been resumed. still also waiting on 04 and 05 | 20:25 |
| *** sdake has quit IRC | 20:25 | |
| *** nhandler has joined #openstack-infra | 20:26 | |
| *** johnsom_ has quit IRC | 20:26 | |
| *** jpr has joined #openstack-infra | 20:26 | |
| *** apoorvad has quit IRC | 20:28 | |
| *** EricGonczer_ has quit IRC | 20:28 | |
| *** salv-orlando has quit IRC | 20:29 | |
| krotscheck | fungi: Thanks, justdid that | 20:32 |
| krotscheck | jeblair: Happy eating! | 20:32 |
| EmilienM | not sure why but we're having a lot of 'NOT_REGISTRED' jobs in puppet CI | 20:34 |
| krotscheck | fungi: Still not registered? | 20:34 |
| EmilienM | "NOT_REGISTERED" actually | 20:34 |
| *** Sukhdev has joined #openstack-infra | 20:36 | |
| *** ccrouch has quit IRC | 20:38 | |
| fungi | krotscheck: zuul claims it was registered. did you recheck before i said it was clear to do so? really it was not registered until a minute or so before i mentioned it was safe | 20:38 |
| fungi | i'll check the zuul logs to see if anything else is going on | 20:39 |
| fungi | zuul restarts at high-volume times result in potentially hours of NOT_REGISTERED results for jobs running on anything other than bare-trusty or devstack-trusty nodes at the moment | 20:39 |
| fungi | EmilienM: can you give me an example job name? | 20:40 |
| krotscheck | fungi: Ok, I can wait. | 20:40 |
| krotscheck | fungi: Yeah, i rechecked after your comment. | 20:40 |
| EmilienM | fungi: https://review.openstack.org/#/c/271333/ | 20:40 |
| *** ddieterly has joined #openstack-infra | 20:41 | |
| fungi | krotscheck: i'll check the logs. gearman definitely seems to think gate-openstackci-beaker-centos7-dsvm and gate-infra-puppet-apply-dsvm-centos7 are registered now though | 20:41 |
| *** sabeen3 has quit IRC | 20:42 | |
| *** esikachev has quit IRC | 20:42 | |
| dhellmann | I'm having some trouble interpreting AJaeger's comments on https://review.openstack.org/#/c/266515/ about how to publish content to a new site on stack.openstack.org. Does someone have a minute to help me with the changes I need to make? | 20:43 |
| fungi | dhellmann: static.openstack.org? | 20:43 |
| dhellmann | fungi : well, for the releases.openstack.org migration | 20:43 |
| dhellmann | fungi : we've hit an area of infra I don't know enough about to ask questions properly :-/ | 20:43 |
| fungi | dhellmann: just making sure stack.openstack.org was a typo and not some new thing | 20:43 |
| dhellmann | oh, yeah, static | 20:44 |
| dhellmann | I think he just means I need to use the scp publisher instead of the ftp one, but I'm not sure. And I'm not sure what arguments to give to the alternate publisher either. | 20:44 |
| anteaya | dhellmann: to be honest I'm not sure what AJaeger wants there either | 20:44 |
| fungi | krotscheck: i see a SUCCESS response for a gate-infra-puppet-apply-dsvm-centos7 job at 20:39:18 in zuul's debug log. where was your most recent NOT_REGISTERED? | 20:45 |
| krotscheck | fungi looking at zuul's status page | 20:46 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: publish from openstack/releases to releases.openstack.org https://review.openstack.org/266515 | 20:46 |
| krotscheck | fungi: search for 271456 | 20:47 |
| fungi | krotscheck: oh, keep in mind that if those were for changes with other longer-running jobs they may have tried to start a centos7 job before gearman was updated but still be pending | 20:47 |
| dhellmann | anteaya, fungi : I think maybe ^^ is what I need? I'm not sure about the target. I picked that based on the path in https://review.openstack.org/#/c/266510/1/modules/openstack_project/manifests/static.pp | 20:47 |
| fungi | krotscheck: yeah, 271456,2 was uploaded 1.5 hours ago | 20:48 |
| anteaya | dhellmann: I thought it was right the first time | 20:48 |
| Swanson | nodepool issue: git --git-dir=/opt/dib_cache/source-repositories/vahana_16d63c569d46b7301825deb73789dae51c725c69/.git fetch --prune --update-head-ok git://git.openstack.org/openstack/vahana.git +master:master | 20:48 |
| anteaya | dhellmann: I'm sorry I am unable to parse what AJaeger wants here and help you | 20:48 |
| Swanson | is returning an error. | 20:48 |
| Swanson | Is there something up with that project? | 20:48 |
| anteaya | dhellmann: I wish I could be more helpful | 20:48 |
| dhellmann | anteaya : No worries! | 20:48 |
| anteaya | dhellmann: I do look forward to understanding though | 20:49 |
| fungi | krotscheck: so it likely set the result on those jobs to NOT_REGISTERED at or shortly after 19:22 but the node backlog for other jobs which were registered coupled with job runtimes means that it hadn't reported back to the change in gerrit yet | 20:49 |
| dhellmann | anteaya : as do I | 20:49 |
| *** Sukhdev has quit IRC | 20:49 | |
| asselin | is this a known issue? https://git.openstack.org/cgit/openstack/vahana/ "Repository seems to be empty" breaking nodepool image builds | 20:50 |
| fungi | Swanson: i'm looking into it. something has gone wrong with the creation of that repository | 20:50 |
| asselin | oh I guess it is | 20:50 |
| fungi | asselin: same as my answer to Swanson | 20:50 |
| Swanson | fungi, asselin Thanks! | 20:50 |
| *** baoli has joined #openstack-infra | 20:51 | |
| krotscheck | fungi: Ok, cool. So I'll be patient :) | 20:52 |
| asselin | that said, image-build is not stopping still. Swanson is yours stopping? mine is stuck at http://paste.openstack.org/show/484742/ | 20:52 |
| *** baoli_ has joined #openstack-infra | 20:53 | |
| *** achanda has joined #openstack-infra | 20:53 | |
| *** mtanino has quit IRC | 20:55 | |
| fungi | jeblair: mordred: did the "log puppet apply stdout to syslog" fix merge yet? i'm not having luck finding puppet apply output in review.openstack.org's syslog | 20:55 |
| *** dprince has quit IRC | 20:55 | |
| *** tonytan4ever has joined #openstack-infra | 20:56 | |
| *** baoli has quit IRC | 20:56 | |
| fungi | dug it out of the concatenated log on the puppetmaster | 20:57 |
| fungi | 2016-01-22 12:48:24,597 [...] Error: /Stage[main]/Openstack_project::Gerrit/Exec[manage_projects]: Command exceeded timeout | 20:57 |
| fungi | i'll try running manage-projects manually, after checking to make sure there's not already one hung | 20:58 |
| *** e0ne has quit IRC | 20:58 | |
| *** KimJ has joined #openstack-infra | 21:02 | |
| fungi | clarkb: greghaynes: this probably also explains why we didn't get new images today. the change adding that repo to the projects list merged almost two hours before the image update cron would have kicked off and has been referring to a broken repo all this time | 21:03 |
| fungi | so we probably tried to clone it, failed, and the image builds terminated | 21:03 |
| clarkb | fedora is still uploading fwiw | 21:03 |
| * clarkb can update nibalizers change if he is ok with the idea I proposed earlier | 21:03 | |
| clarkb | then will split out the builder daemon | 21:04 |
| mordred | fungi: no, it hasn't | 21:04 |
| Swanson | asselin, mine is restarting about a million times. Well, 9 so far. | 21:04 |
| mordred | fungi: stack is here: https://review.openstack.org/#/c/271086/1 | 21:04 |
| *** jsavak has quit IRC | 21:04 | |
| greghaynes | clarkb: hrm, I don't see anything in gearman client for finding out when a job is starting to be worked on | 21:05 |
| greghaynes | clarkb: did you find something for that? | 21:05 |
| clarkb | greghaynes: I think it is called work status | 21:05 |
| *** roaet- has quit IRC | 21:05 | |
| greghaynes | clarkb: ah, ok | 21:05 |
| greghaynes | I do see that | 21:05 |
| greghaynes | haha, it has a percentage compete argument | 21:06 |
| greghaynes | When are we adding progress bars | 21:06 |
| fungi | mordred: thanks! looks like it's failing tests (or was the last time they were run anyway) | 21:06 |
| mordred | fungi: I rechecked the middle patch just now | 21:06 |
| fungi | k | 21:06 |
| mordred | fungi: and now the last one | 21:07 |
| *** roaet has joined #openstack-infra | 21:07 | |
| *** jsavak has joined #openstack-infra | 21:07 | |
| clarkb | greghaynes: only if we make them sit at 99% for half the time | 21:07 |
| greghaynes | clarkb: It is just a matter of what the builder returns for percentage, so I can go ahead and make them return that from that start | 21:08 |
| clarkb | perfect | 21:09 |
| mordred | fungi: https://review.openstack.org/#/c/271087 this one turns on the feature - it's safe to land before the feature lands | 21:09 |
| *** apoorvad has joined #openstack-infra | 21:10 | |
| fungi | wow, full manage-projects run is at 11 minutes and going | 21:10 |
| fungi | we may need to update our current 15-minute timeout. i have a feeling we quietly edged past it recently | 21:11 |
| mordred | fungi: oh good | 21:11 |
| fungi | which would be the simple explanation for the current issue | 21:11 |
| clarkb | almost 3 hours on this upload so far | 21:12 |
| * clarkb really hopes separate process is happier | 21:12 | |
| fungi | jenkins update, 03-05 are still going. if at least one of those completes successfully i'll put the other two in prepare-for-shutdown and then retrigger release jobs | 21:13 |
| fungi | i just don't feel comfortable downing 3 out of 7 | 21:13 |
| anteaya | go jenkinsii | 21:13 |
| anteaya | makes sense | 21:13 |
| *** ZZelle_ has joined #openstack-infra | 21:13 | |
| cbader_ | I have a question about jenkins Authentication failure to Slave, I have verified the private and public key from jenkins server to slave vm but Jenkins still says failed connection? | 21:16 |
| *** FallenPegasus has quit IRC | 21:16 | |
| *** marcusvrn_ has quit IRC | 21:17 | |
| *** salv-orlando has joined #openstack-infra | 21:18 | |
| cbader_ | I know this is a jenkins issue but have run out of ideas on what might be causing this, I have verifed the jenkins user and it is using the correct key, I used it manually to ssh to vm with jenkins user. | 21:18 |
| *** jtomasek has quit IRC | 21:18 | |
| nibalizer | fungi: for things like manage_projects and jjb, why do we run those from puppet exec and not cron? | 21:18 |
| fungi | nibalizer: better question, why do we run them from puppet and not ci jobs | 21:19 |
| pabelanger | clarkb: haven't read backscroll yet, did you figure out cron issue with nodepool? | 21:19 |
| clarkb | cbader_: check the credentials ocnfigured for use in jenkins under the slave | 21:19 |
| clarkb | pabelanger: ya I think it is the cron hsa started but never completed | 21:20 |
| fungi | nibalizer: though the ansible answer is more compelling than the puppet answer for manage-projects... we need to make updates to the git servers and gerrit server in a dependent order | 21:20 |
| clarkb | pabelanger: so we need to error when we get disconnected from gearman to kill that thread | 21:20 |
| *** abregman is now known as abregman|nb | 21:20 | |
| clarkb | nibalizer: we don't run it every time so it subscribes to config file changes | 21:20 |
| clarkb | nibalizer: we could use inotify to do that too | 21:20 |
| cbader_ | clarkb, I have done that and they look correct and point to same id_rsa key I used to manually connect. | 21:20 |
| pabelanger | clarkb: okay, good to know | 21:20 |
| clarkb | cbader_: file permissions maybe? thats all I can think of | 21:21 |
| cbader_ | clarkb, thanks will verify that also. | 21:21 |
| fungi | okay, manage-projects execution completed aroundthe 20 minute mark (i should have invoked in with time) | 21:21 |
| jlvillal | fungi: Do you know if still want to pin pip below 8? | 21:22 |
| clarkb | nibalizer: so what do you think about having init do the install then have two separate classes for the main daemon and the builder? | 21:22 |
| fungi | the vahana acl seems to gave gotten applied successfully now | 21:22 |
| jlvillal | fungi: In regards to this patch merged: https://review.openstack.org/#/c/270503/ | 21:22 |
| clarkb | nibalizer: then you would include nodepool include nodepool::builder include nodepool::service to coinstll | 21:22 |
| clarkb | jlvillal: pip should be working now with 8.0.2 | 21:22 |
| fungi | jlvillal: no, pip 8.0.2 and virtualenv 14.0.1 should have solved the issues we had | 21:22 |
| jlvillal | Would a revert patch be appropriate then? | 21:23 |
| fungi | jlvillal: yes | 21:23 |
| nibalizer | clarkb: sounds functional, i kinda like the idea of nodepool::install instead of bare nodepool, just to be more clear what its doing | 21:23 |
| jlvillal | Okay, I will spin one up. Thanks | 21:23 |
| jeblair | mordred, krotscheck: back! hopefully for a few uninterrupted hours now | 21:23 |
| fungi | hopefully someone was keeping track of where all the pip/virtualenv pinning workarounds ended up | 21:23 |
| jeblair | fungi: hound was | 21:23 |
| clarkb | nibalizer: ok, I was basing that off of how eg apache works | 21:23 |
| clarkb | nibalizer: the base apache include is just to install right? | 21:24 |
| *** baoli_ has quit IRC | 21:24 | |
| *** bpokorny has joined #openstack-infra | 21:24 | |
| jeblair | mordred: based on your note that you thought you fixed it, i'll reinstall shade and try that launch node again... | 21:25 |
| fungi | Ng: your requested group updates are complete now | 21:25 |
| *** ccrouch has joined #openstack-infra | 21:25 | |
| jeblair | mordred: but i still have the curiosity about puppet apply and launch-node... | 21:25 |
| Ng | fungi: thanks! | 21:25 |
| fungi | Swanson: asselin: your git cloning issues should be resolved now, but let us know if they're not | 21:26 |
| *** kgiusti has left #openstack-infra | 21:26 | |
| mordred | jeblair: yes - did you see my other note linking to the patch that updates shade-launch-node to use puppet-apply? | 21:27 |
| *** sabeen1 has joined #openstack-infra | 21:27 | |
| *** ybathia has joined #openstack-infra | 21:27 | |
| jeblair | mordred: no, can you relink? | 21:27 |
| mordred | jeblair: it needs to be fixed though, but it is mostly done | 21:27 |
| mordred | yes - one sec | 21:27 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/system-config: Revert "Pin pip to less than version 8 in install_puppet.sh" https://review.openstack.org/271526 | 21:27 |
| nibalizer | clarkb: I don't know off the top of my head | 21:28 |
| jlvillal | fungi clarkb mordred ^^^^^ | 21:28 |
| fungi | thanks jlvillal! | 21:28 |
| mordred | jeblair: https://review.openstack.org/#/c/247099/ | 21:28 |
| *** ccrouch has quit IRC | 21:28 | |
| *** salv-orlando has quit IRC | 21:28 | |
| *** ccrouch has joined #openstack-infra | 21:28 | |
| mordred | jeblair: for now it's likely more expedient to just use old launch_node and have it make puppet certs and such | 21:28 |
| jeblair | mordred: well, old *shade*-launch node, yeah? | 21:29 |
| mordred | jeblair: lest we go down an end-of-week yak shaving | 21:29 |
| jeblair | mordred: since we can't launch nodes on half our clouds with *old* launch node | 21:29 |
| mordred | jeblair: yes | 21:29 |
| jeblair | mordred: (and i'm launching mirrors) | 21:29 |
| jeblair | mordred: okay, i'll use what's in https://review.openstack.org/#/c/234975/ | 21:29 |
| mordred | jeblair: ++ | 21:29 |
| jeblair | which is the most up to date shade-launch node without apply | 21:29 |
| mordred | jeblair: if you get that error again, I will appropriately groan | 21:30 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Increase manage-projects timeout to 30 minutes https://review.openstack.org/271527 | 21:30 |
| mordred | jeblair: (largely because I specifically remember that bug) | 21:30 |
| jeblair | mordred: nah, now we get ImportError: No module named v1_1 | 21:31 |
| jeblair | mordred: because my shade update updated novaclient a lot | 21:31 |
| jeblair | mordred: that's because dns.py uses utils.py to call get_client() which uses the old nova stuff | 21:31 |
| jeblair | mordred: oh, i think some of that code is dead, i may be able to hack around it | 21:32 |
| *** amrith is now known as _amrith_ | 21:32 | |
| *** Sukhdev_ has joined #openstack-infra | 21:33 | |
| mordred | jeblair: hrm. I could have sworn I made an update to your patch to kill all that | 21:33 |
| jeblair | mordred: that's probably in your second one which we don't want to yak shave | 21:33 |
| *** eharney has quit IRC | 21:33 | |
| Swanson | fungi, asselin Seems to be off and running. Thanks! | 21:34 |
| mordred | jeblair: sigh. it's in the third one | 21:34 |
| mordred | jeblair: the one that replaces launch-node.py with shade-launch-node.py | 21:34 |
| mordred | jeblair: I may not have structured this work well | 21:34 |
| jeblair | mordred: okay, hacked. | 21:34 |
| jeblair | it's running now | 21:34 |
| mordred | jeblair: https://review.openstack.org/#/c/251063/ fwiw | 21:34 |
| *** eharney has joined #openstack-infra | 21:35 | |
| *** burgerk has joined #openstack-infra | 21:35 | |
| *** sabeen3 has joined #openstack-infra | 21:36 | |
| *** abregman|nb has quit IRC | 21:36 | |
| jeblair | mordred: oh. | 21:37 |
| jeblair | Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find data item bandersnatch_keytab in any Hiera data file and no default supplied at /opt/system-config/production/manifests/site.pp:532 on node mirror_update.openstack.org | 21:37 |
| mordred | uh | 21:38 |
| jeblair | should that be fetchable via puppetmaster? | 21:38 |
| mordred | jeblair: but this is using agent? | 21:38 |
| *** sabeen1 has quit IRC | 21:38 | |
| mordred | it should be .... | 21:38 |
| jeblair | ya | 21:38 |
| jeblair | puppet agent --environment production --server puppetmaster.openstack.org --detailed-exitcodes --no-daemonize --verbose --onetime --pluginsync true --certname mirror_update.openstack.org | 21:38 |
| *** eharney_ has joined #openstack-infra | 21:38 | |
| mordred | it's in fqdn/mirror_update.openstack.org.yaml | 21:39 |
| mordred | so I expect it to be found by puppetmaster via hiera | 21:39 |
| jeblair | it's worth considering that maybe puppetmaster+hiera has a problem with the binary data | 21:39 |
| mordred | it is definitely worth considering | 21:39 |
| jeblair | i will just change that to a simple string for now to eliminate that | 21:39 |
| mordred | jeblair: kk | 21:40 |
| *** markvoelker has joined #openstack-infra | 21:40 | |
| *** markvoelker has quit IRC | 21:40 | |
| mordred | jeblair: other option is that fqdn is not resolving yet | 21:40 |
| *** markvoelker has joined #openstack-infra | 21:41 | |
| *** burgerk has quit IRC | 21:41 | |
| *** burgerk has joined #openstack-infra | 21:41 | |
| jeblair | mordred: you mean, puppet doesn't know what it is (not dns -- since it's not supposed to) | 21:41 |
| mordred | jeblair: we're passing in certname on the command line, which is how node mirror_update.openstack.org works for sure - but in the bootstrap pass, facter may have detected fqdn wrong | 21:41 |
| greghaynes | jeblair: clarkb so for this builder change (for it to properly handle waiting on a builder to begin building an image) we have two options AFAICT - either add back the separate builder id's for images (rather than feeding in the image.id from nodepool nodedb) or we need to add another possible state that images can be in | 21:41 |
| *** bhunter71 has quit IRC | 21:41 | |
| greghaynes | basically nodepool needs to know how to handle an image which has not yet been made versus one that it can ask to delete | 21:42 |
| mordred | jeblair: you could add a facter env var to the front of the puppet agent call | 21:42 |
| *** eharney has quit IRC | 21:42 | |
| mordred | jeblair: FACTER_fqdn=asdf facter fqdn | 21:42 |
| EmilienM | just a dumb question, is the "NOT_REGISTERED" thing for puppet jobs is something you guys are aware? | 21:42 |
| mordred | jeblair: results in facter returning asdf | 21:42 |
| jeblair | greghaynes: which problem are you trying to solve? | 21:42 |
| EmilienM | fungi: you asked me before which patch - should I worry about something? | 21:43 |
| mordred | jeblair: so if you prepend FACTER_fqdn=%(certname)s to the puppet command ... it should workaround that easilyish | 21:43 |
| clarkb | nibalizer: so there is a lot more that actually needs to be done for this change to be correct. We have to make sure we write a config for nodepoold and nodepool-builder, defaults files, apache vhost and sudo things need to only be on the builder manifest, and thats just what I have run into so far | 21:43 |
| jeblair | mordred: i'll try that, since it failed again after the data type change | 21:43 |
| mordred | jeblair: cool. it's my best guess | 21:43 |
| *** mtanino has joined #openstack-infra | 21:43 | |
| clarkb | nibalizer: what do you think of an intermediate change that just colocates nodepoold and nodepool-builder via init.pp? | 21:43 |
| greghaynes | jeblair: having nodepool handle a disconnect from geard gracefully - which means not hanging forever on jobs which wont get run and also deleting things (db rows) on its side | 21:43 |
| fungi | EmilienM: the change you linked looks like all the errors were due to missing devstack-centos7 worker job registrations. this should be resolved as of about an hour ago so rechecking those changes ought to work now | 21:44 |
| EmilienM | fungi: no problem at all, I just wanted to know if you guys are aware | 21:44 |
| *** peristeri has quit IRC | 21:45 | |
| EmilienM | thanks for the follow-up! | 21:45 |
| *** baoli has joined #openstack-infra | 21:45 | |
| fungi | EmilienM: thanks for the details! | 21:45 |
| nibalizer | clarkb: isn't it just a three line patch to 240093 that pulls the class instantiation out of builder.pp and init.pp ? | 21:46 |
| jeblair | greghaynes: i see the problem differently -- when the build job fails (possibly due to a disconnect), nodepool should put the image in the delete state. something should come alond and delete things in the delete state. i think it's okay for whatever deletes images to check that the function exists before calling it. | 21:46 |
| clarkb | nibalizer: no, its much bigger than that | 21:46 |
| jeblair | greghaynes: however, if we want to add an extra state to help us know whether we expect that function to exist, that's cool too | 21:46 |
| nibalizer | clarkb: why? | 21:47 |
| jeblair | greghaynes: though... | 21:47 |
| clarkb | nibalizer: the builder needs its own config, logging config, it hosts the image logs via apache not the service daemon, and the sudo stuff is for the builder not the service daemon | 21:47 |
| clarkb | nibalizer: there is a lot of stuff we hvae to do to support a separate non colocated builder | 21:47 |
| clarkb | which is missing | 21:47 |
| jeblair | greghaynes: i'm not actually sure how that's different than just 'building'. | 21:47 |
| jeblair | greghaynes: if an image is 'building', we can't submit a delete job for it. | 21:47 |
| *** pcaruana has joined #openstack-infra | 21:48 | |
| jeblair | mordred: there's more green now. | 21:48 |
| mordred | jeblair: I like more green | 21:48 |
| nibalizer | clarkb: well sudoers stuff can just move from init.pp to builder.pp | 21:48 |
| nibalizer | thats easy | 21:48 |
| greghaynes | jeblair: Actually I think we can submit a delete job for it - there will only ever be one potential worker who can handle that delete job and it won't return until after building the image | 21:48 |
| clarkb | a lot of it can just move, my point is this change is missing a lot of stuff | 21:49 |
| nibalizer | as for config files, I got the impression that the idea was to use project-config | 21:49 |
| clarkb | and it might be easier to do the halfway step | 21:49 |
| clarkb | then make the jump to separate hosts | 21:49 |
| jeblair | greghaynes: if it never finished building, then there should be nothing to delete, right? ah, i think we're missing a piece here: | 21:49 |
| *** doug-fish has quit IRC | 21:49 | |
| clarkb | nibalizer: not the nodepool.yaml the defaults file for the service | 21:50 |
| jeblair | greghaynes: i think either the builder needs to also handle disconnects and delete the image it just built... or we need nodepool to be able to recover from the case where a builder silently finishes building an image after a disconnect or restart | 21:50 |
| jeblair | greghaynes: the first seems easier to me, and at least more in keeping with nodepool's general error recovery strategy | 21:50 |
| greghaynes | jeblair: The builder does not necessarially disconnect so it does not know about the fact that nodepool submitted jobs and then disconnected | 21:51 |
| jeblair | if not the most efficient allocation of resources | 21:51 |
| *** mriedem has quit IRC | 21:51 | |
| *** burgerk has quit IRC | 21:52 | |
| openstackgerrit | Merged openstack-infra/system-config: Added pypi hosting for AFS mirror slave https://review.openstack.org/270975 | 21:52 |
| nibalizer | clarkb: so we want to support but not use a colocated builder? | 21:52 |
| clarkb | nibalizer: right now what we want is a colocated builder | 21:52 |
| clarkb | eventually we want builders on their own hosts | 21:52 |
| cbader_ | clarkb, I got around my issue by not using the :From a file on Jenkins master" and switched to "Enter directly" for some reason it won't read the file at '/var/lib/jenkins/.ssh/id_rsa' it is set to 600 and jenkins jenkins. go figure | 21:52 |
| clarkb | this change is trying to get all the way to the end goal all at once which is more complicated. So wondering if it makes sense to start off only supporting a colocated builder | 21:53 |
| clarkb | then build on that to allow for separate build hosts | 21:53 |
| *** rcernin has joined #openstack-infra | 21:53 | |
| mordred | also - I imagine openstackci will always want option for colocated builder | 21:53 |
| fungi | also downstream consumers may want to stick with colocated scheduler and builder indefinitely (and even coinstall them with zuul scheduler and merger worker daemons as well) | 21:53 |
| greghaynes | jeblair: so maybe a better idea - if nodepool detects the disconnect it can delete all the corresponding db rows for outstanding image build jobs? | 21:53 |
| mordred | fungi: jinx | 21:53 |
| fungi | yeah, what mordred just said in fewer words | 21:53 |
| nibalizer | mordred: thats why I created this https://review.openstack.org/#/c/269280/1/manifests/nodepool.pp but I'm not very pleased with it | 21:53 |
| greghaynes | jeblair: without asking the builder to delete since the builder would not have seen those jobs yet | 21:54 |
| *** sdake_ has quit IRC | 21:54 | |
| clarkb | nibalizer: but that doesn't work today anyways | 21:54 |
| nibalizer | okay | 21:54 |
| clarkb | we can't colocate beacuse of duplicate resource defs, but we can't separate because nodepool-builder isn't sufficient enough as is | 21:54 |
| jeblair | clarkb: don't we just put a bunch of if's around duplicate resource defs? | 21:55 |
| greghaynes | jeblair: it has the annoying case that a nodepool crash will leave data in the db which is erroneous | 21:55 |
| clarkb | we can fix those problems, I am just wondering if it makes sense to simplify the problem for today in order to make incremental changes | 21:55 |
| clarkb | jeblair: that is one approach yes | 21:55 |
| jeblair | greghaynes: when nodepool starts, we set all building nodes to delete; we should do the same with dib-images | 21:55 |
| greghaynes | jeblair: ah, thatd work | 21:55 |
| clarkb | but I don't want to prsent this api of "run a separate builder" until it has a chance at working | 21:56 |
| clarkb | so trying to figure out if we want to make that change now or make the simpler change now | 21:56 |
| nibalizer | clarkb: I thought we agreed to your plan of telling the user (openstackci in this case) to include install, daemon, and builder explicitly | 21:56 |
| clarkb | nibalizer: yes | 21:56 |
| clarkb | nibalizer: but as I am trying to make that happen I realize its way more complicated | 21:56 |
| clarkb | so i am asking, do we want to solve that problem today | 21:56 |
| nibalizer | i mean the changes you're talking about aren't that hard to make | 21:57 |
| nibalizer | move sudo, create default file | 21:57 |
| nibalizer | create apache file | 21:57 |
| clarkb | its not about the puppet though | 21:57 |
| nibalizer | its largely duplicating existing resources and templates that the nodepool class already has | 21:57 |
| clarkb | its about I want the service deployed today | 21:57 |
| clarkb | and I bet its not going to happen this way | 21:57 |
| clarkb | maybe in a couple weeks when I am not traveling | 21:57 |
| jeblair | greghaynes: but i don't think we've solved the problem of how to reliably get the builder to delete the orphaned image | 21:58 |
| greghaynes | jeblair: Correct | 21:58 |
| jeblair | greghaynes: because if we do what i suggested with having it check for the function, it will see that it isn't registered and simply delete the db row; then the builder will finish the build, and it will just sit there orphaned on disk. | 21:59 |
| nibalizer | clarkb: so minimum work to colocation would be what | 21:59 |
| nibalizer | just setup a config file and a service definition? | 21:59 |
| jeblair | greghaynes: your approach of firing the job into the void would work, but it's a bit fragile, because if geard restarts, that gets lost | 21:59 |
| clarkb | nibalizer: just a service definition | 22:00 |
| clarkb | nibalizer: everything else can be shared if you only allow colocation | 22:00 |
| nibalizer | clarkb: basically line27-40 of this https://review.openstack.org/#/c/240093/7/manifests/builder.pp | 22:00 |
| clarkb | + a defaults file | 22:00 |
| openstackgerrit | gordon chung proposed openstack-infra/project-config: fix ceilometerclient gate https://review.openstack.org/271379 | 22:01 |
| *** gordc has joined #openstack-infra | 22:01 | |
| greghaynes | jeblair: hrm. I wasn't thinking of firing in to the void - nodepool knows what jobs it has submitted and it can detect if a job has been started but there is definitely a potential race there (where the job starting notification happens after a disconnect)... | 22:01 |
| nibalizer | clarkb: kk ill spin that up | 22:02 |
| nibalizer | give me a sec | 22:02 |
| greghaynes | jeblair: So youre thinking set images to delete on a disconnect, then have our cleanup method check that against whether a delete method is registered | 22:02 |
| *** Sukhdev_ has quit IRC | 22:02 | |
| jeblair | greghaynes: well, i'm more thinking about the complete restart case, which is (hopefully) going to happen more often than random disconnects. everything can disappear immediately there and we need to recover from that. | 22:03 |
| jeblair | greghaynes: that's what i was thinking, but it does not handle the case where nodepool restarts but the builder does not | 22:03 |
| greghaynes | jeblair: eek, that gets bad with builder restarting also (just occured to me) | 22:03 |
| greghaynes | we have to make sure the builder fully restarts before nodepool does | 22:03 |
| jeblair | greghaynes: we can make no such guarantees | 22:03 |
| *** jsavak has quit IRC | 22:04 | |
| *** notnownikki has quit IRC | 22:04 | |
| jeblair | any piece needs to be able to restart at any time with no respect to any other piece | 22:04 |
| greghaynes | ya, so if we don't have that then the builder could not be fully up ; nodepool starts and sees no delete jobs registered and deletes db rows; builder comes along and adds jobs | 22:04 |
| jeblair | yep | 22:04 |
| *** eharney_ has quit IRC | 22:05 | |
| greghaynes | If we do it where nodepool keeps tracks of what jobs are waiting to be run / what ones have started then the race is preeeety narro | 22:06 |
| jeblair | greghaynes: maybe we can do something like one of the following 2 ideas: a) nodepool periodically lists functions (hey it does it anyway) and looks for jobs related to unknown images then deletes them | 22:06 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-nodepool: Colocate a builder-worker with the service https://review.openstack.org/271541 | 22:06 |
| jeblair | greghaynes: b) builders periodically tell nodepool what images they have and nodepool tells them to delete ones it doesn't know about | 22:06 |
| nibalizer | clarkb: ^^ | 22:06 |
| *** erlon has quit IRC | 22:06 | |
| *** bcourt has quit IRC | 22:06 | |
| greghaynes | jeblair: ok, I like a) | 22:07 |
| jeblair | greghaynes: a) is actually quite like what we do for detection/cleanup of leaked nodes. | 22:07 |
| greghaynes | ya | 22:07 |
| clarkb | nibalizer: we should include that in init.pp and set --no-builder in templates/nodepool.default.erb | 22:08 |
| *** tlian has quit IRC | 22:08 | |
| greghaynes | ok a) + on disconnect all jobs in building go to deleted and we cleanup best we can | 22:08 |
| clarkb | in the daemon args var | 22:08 |
| jeblair | greghaynes: + delete on startup | 22:08 |
| greghaynes | jeblair: yep | 22:08 |
| jeblair | greghaynes: and yeah, i think that covers it | 22:08 |
| fungi | OKAY! jenkins04 updates finally completed successfully. i'll go ahead and put 03 and 05 in prepare-for-shutdown so we can safely retrigger the impacted release tags while they finish updating their jobs | 22:09 |
| jeblair | fungi: YAY! | 22:09 |
| anteaya | fungi: yay | 22:09 |
| *** salv-orlando has joined #openstack-infra | 22:10 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-nodepool: Colocate a builder-worker with the service https://review.openstack.org/271541 | 22:11 |
| nibalizer | clarkb: https://review.openstack.org/#/c/271541/ | 22:11 |
| jeblair | mordred: how do ansible puppet apply one host? | 22:11 |
| nibalizer | clarkb: i gotta run away again | 22:12 |
| nibalizer | feel free to push a new patch to 271541 if you need to | 22:13 |
| nibalizer | ill be back in like an hour and maybe we can push this out today | 22:13 |
| *** kzaitsev_mb has joined #openstack-infra | 22:13 | |
| jeblair | gah! | 22:14 |
| openstackgerrit | Khai Do proposed openstack-infra/gearman-plugin: Update to Jenkins LTS 1.625.3 and fix function registration https://review.openstack.org/271543 | 22:14 |
| zaro | zxiiro: ^ | 22:14 |
| jeblair | krotscheck: it should be "mirror-update.openstack.org" not 'mirror_update.openstack.org'. dns can't have underscores | 22:14 |
| * greghaynes 's initials are gah, and he likes to think people have hit bugs he wrote when people exclaim that | 22:14 | |
| *** baoli has quit IRC | 22:15 | |
| anteaya | hahaha | 22:15 |
| *** baoli has joined #openstack-infra | 22:15 | |
| anteaya | my initials are almost ack | 22:15 |
| anteaya | but not qite | 22:16 |
| anteaya | quite | 22:16 |
| jeblair | relaunching with mirror-update as the fqdn | 22:16 |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-nodepool: Colocate a builder-worker with the service https://review.openstack.org/271541 | 22:16 |
| clarkb | nibalizer: ^ minor change to move the no builder flag and add a require | 22:17 |
| *** eharney_ has joined #openstack-infra | 22:18 | |
| clarkb | fedora is still uploading | 22:18 |
| jeblair | okay that and all 3 rax mirrors are building | 22:18 |
| *** xyang1 has quit IRC | 22:21 | |
| fungi | jenkins05 updates died on yet another proxy error, restarting. at least this one is out of the line of fire now so should start acting out a little less | 22:24 |
| *** achanda has quit IRC | 22:24 | |
| *** sriram has quit IRC | 22:26 | |
| clarkb | https://review.openstack.org/#/c/271541/3 passed testing, I can give a go of applying it manually to the host prior to merging | 22:27 |
| clarkb | (want fedora upload to finish though) | 22:27 |
| *** bardia has joined #openstack-infra | 22:29 | |
| *** dkranz has quit IRC | 22:30 | |
| clarkb | it just entered hour 4 | 22:30 |
| clarkb | er finished hour 4 | 22:30 |
| *** baoli has quit IRC | 22:32 | |
| fungi | ugh, here's some fresh new broken: https://jenkins06.openstack.org/job/ceilometer-tarball/2/console | 22:32 |
| *** markvoelker has quit IRC | 22:32 | |
| fungi | "no such option: --download-cache" | 22:32 |
| fungi | looks like tox is calling pip with --download-cache=/home/jenkins/workspace/ceilometer | 22:33 |
| fungi | and pip does not (any longer?) consider that a valid option | 22:33 |
| clarkb | I don't see anyting obvious in that tox.ini doing it | 22:34 |
| fungi | however, i triggered cinder at the same time and it worked https://jenkins06.openstack.org/job/cinder-tarball/3/console | 22:34 |
| fungi | clarkb: keep in mind this is a tag on stable/kilo | 22:35 |
| clarkb | oh | 22:35 |
| mordred | jeblair: ansible-playbook -l $hostname playbooks/remote_puppet_else.yaml | 22:35 |
| *** piet has joined #openstack-infra | 22:35 | |
| jeblair | mordred: thanks! | 22:35 |
| clarkb | https://git.openstack.org/cgit/openstack/ceilometer/tree/tox.ini?h=stable/kilo#n15 | 22:35 |
| fungi | http://git.openstack.org/cgit/openstack/ceilometer/tree/tox.ini?h=stable/kilo#n15 | 22:35 |
| fungi | heh | 22:35 |
| *** [1]Thelo has joined #openstack-infra | 22:35 | |
| jeblair | https://git.openstack.org/cgit/openstack/ceilometer/tree/tox.ini?h=stable/kilo#n15 | 22:35 |
| fungi | seems we independently agree | 22:35 |
| jeblair | i just didn't want to be left out | 22:35 |
| fungi | joiner | 22:36 |
| anteaya | ha ha ha | 22:36 |
| jeblair | but yeah, someones been proposing patches to remove that everywhere | 22:36 |
| mordred | https://git.openstack.org/cgit/openstack/ceilometer/tree/tox.ini?h=stable/kilo#n15 ? | 22:36 |
| fungi | mordred: http://git.openstack.org/cgit/openstack/ceilometer/tree/tox.ini?h=stable/kilo#n15 ! | 22:36 |
| jeblair | i think? | 22:36 |
| *** gordc has quit IRC | 22:37 | |
| fungi | hrm... hound only does master branches? | 22:37 |
| *** rlandy has quit IRC | 22:37 | |
| *** PsionTheory has quit IRC | 22:38 | |
| mordred | fungi: yes | 22:38 |
| *** Thelo has quit IRC | 22:38 | |
| *** [1]Thelo is now known as Thelo | 22:38 | |
| mordred | fungi: I think | 22:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-bandersnatch: Make sure mercurial is installed https://review.openstack.org/271548 | 22:39 |
| clarkb | fedora upload completed | 22:39 |
| jeblair | mordred: i need that for bandernatch/update-mirror ^ | 22:39 |
| fungi | 8.0.0 (2016-01-19): BACKWARD INCOMPATIBLE Remove the --download-cache which had been deprecated and no-op'd in 6.0. | 22:39 |
| fungi | i guess we just need to strip that out of tox.ini files and elsewhere when we encounter it now | 22:39 |
| mordred | jeblair: +2 from me | 22:39 |
| clarkb | any opposition to me applying nibalizer's change by hand to start a separate builder process then restart nodepoold to stop running the in process builder? | 22:39 |
| mordred | clarkb: nope - whatever you think is best there is fine with me | 22:40 |
| clarkb | I will be disabling ansible puppet on nodepool.o.o | 22:40 |
| fungi | Daviey: i have good news and bad news... the cinder tarballs should be fine now, ceilometer needs to stop using --download-cache in its tox.ini on stable/kilo (pip behavior change) and neutron-vpnaas switched their release tarball jobs to explicitly use zuul-cloner (a neutron-vpnaas-tarball-constraints job) so that one's still checking out master | 22:40 |
| jeblair | mordred: oh, you rebased my ansible-puppet change on https://review.openstack.org/271085 | 22:42 |
| jeblair | mordred: what do you mean "sync with upstream" ? | 22:42 |
| mordred | jeblair: we upstreamed our puppet module into the ansible repo. since then, it has gotten 2 patches that were not in our copy | 22:43 |
| jeblair | we did? | 22:43 |
| mordred | yup | 22:43 |
| jeblair | oh. um. | 22:43 |
| jeblair | did i know about that? | 22:43 |
| mordred | I believe so, yeah? I mean - we don't consume it from upstream at the moment - but we did write it to be generally useful | 22:44 |
| jeblair | mordred: yeah, i just didn't know that to be useful it had to go somewhere else... | 22:44 |
| jeblair | mordred: where is it? | 22:44 |
| jeblair | mordred: i thought there was a galaxy or something | 22:44 |
| mordred | that's for roles | 22:44 |
| jeblair | oh | 22:44 |
| mordred | this goes into the same place as the "shell" ansible module | 22:44 |
| mordred | or "sycn" | 22:44 |
| jeblair | well, that does sound useful | 22:45 |
| mordred | yah- people seem to like it | 22:45 |
| jeblair | mordred: how does our module get used by us then? | 22:45 |
| mordred | well, right now we stll have a copy | 22:45 |
| mordred | at some point, when we're satisfied that it's solid and we don't need emergency last-minute fixes very often | 22:45 |
| *** Sukhdev has joined #openstack-infra | 22:45 | |
| anteaya | if anyone with gerrit admin permissions is bored, a rackspace instructor onboarding new contributors has requested that all sandbox patches with topic:bug/1234567 be approved: https://review.openstack.org/#/q/topic:bug/1234567+status:open | 22:45 |
| jeblair | yeah, i mean, how does it get used in preference to the one that's presumably built in ansible? | 22:45 |
| mordred | we can just get rid of our copy and use the one we get from "pip install ansible" | 22:45 |
| clarkb | mordred: and to do that I am going to fetch the chagne into /etc/puppet/modules/nodepool then run puppet apply against the local site.pp | 22:46 |
| mordred | jeblair: gotcha | 22:46 |
| mordred | jeblair: right now, beause it's in the library dir of that role, it takes precedence in lookup order | 22:46 |
| fungi | i'm sort of torn on procedure for handling the ceilometer kilo tarball issue. the last change on the branch was successfully built using pip 7. between then and when the tag was pushed, pip 8 was released which dropped an option it was using. now we can't build a tarball from the tag, and we can't update the repo to work with pip 8 without retagging another release. to adhere to our semver scheme, we | 22:46 |
| fungi | have to increment the patch version to 2015.1.4 but there's already a release of that version scheduled later in the year | 22:46 |
| mordred | jeblair: https://github.com/ansible/ansible-modules-extras/blob/devel/system/puppet.py fwiw | 22:46 |
| jeblair | fungi: pin pip 7 in the tarball job and rebuild? | 22:47 |
| clarkb | ugh new gerrit doesn't show you anonymous fetch urls | 22:47 |
| fungi | we could put together a custom job which downgrades to pip 7 when building the tarball, and then retrigger the existing tag with that i guess, though that's funny business | 22:47 |
| anteaya | since sandbox won't allow registered users to do that themselves at present | 22:47 |
| clarkb | anyone know how to make it do that? | 22:47 |
| jeblair | clarkb: i got one | 22:47 |
| jeblair | clarkb: 'download' in the top right | 22:47 |
| fungi | clarkb: use anonymous http rather than http | 22:47 |
| *** hashar has quit IRC | 22:47 | |
| clarkb | oh you have to hit the dropdown | 22:47 |
| fungi | "http" means authenticated | 22:47 |
| clarkb | why is anonymous not default? | 22:47 |
| fungi | in new gerritspeak | 22:47 |
| jeblair | clarkb: was for me | 22:47 |
| clarkb | huh | 22:48 |
| clarkb | maybe it is a user setting | 22:48 |
| jeblair | fungi: why custom job? | 22:48 |
| fungi | jeblair: well, i guess we could make all tarball jobs use pip 7 | 22:48 |
| mordred | jeblair: which, btw, means "/usr/local/lib/python2.7/dist-packages/ansible/modules/extras/system/puppet.py" now shows up on any system that does "pip install ansible" | 22:48 |
| jeblair | fungi: though, i guess you are in the home stretch of just having updated them all | 22:48 |
| jeblair | so maybe that doesn't sound like fun | 22:48 |
| fungi | though that brings up the neutron-vpnaas problem. it needs to stop using a custom "constraints-based" tarball job for now | 22:48 |
| anteaya | dougwig: ^^ | 22:49 |
| jeblair | fungi: yeah, there should be no custom tarball jobs | 22:49 |
| jeblair | nor custom pep8 job | 22:49 |
| jeblair | nor custom py27 job | 22:49 |
| *** piet has quit IRC | 22:49 | |
| jeblair | this is why we have the PTI | 22:49 |
| fungi | i haven't gone spelunking in layout.yaml yet, but i suspect there are other projects with the same which all need to be undone | 22:49 |
| jeblair | we may need to communicate that to project-config reviewers | 22:49 |
| docaedo | Is it generally safe to assume that any PR in openstack-infra will get reviewed in due time without me pestering for reviews? cuz I don't want to be a pest or anything ;) | 22:49 |
| fungi | and we just haven't discovered yet because none of them have done a stable branch release | 22:49 |
| jeblair | docaedo: nope, i think we're drowning now. | 22:50 |
| jeblair | docaedo: we've been putting out fires for pretty much 2 weeks solid. | 22:50 |
| *** rajinir has joined #openstack-infra | 22:50 | |
| anteaya | docaedo: define due time | 22:50 |
| clarkb | ok puppet looked fine, but service failed to start | 22:50 |
| fungi | docaedo: most of our time lately has been spent approving our own changes just to keep things working, and when we have time reverting things which have already been approved | 22:50 |
| fungi | s/keep/get/ | 22:50 |
| docaedo | yeah I figured there's a lot going on, which is why I haven't pinged/pestered | 22:50 |
| jeblair | docaedo: good news is, if we approved it, it probably wouldn't land or work anyway. | 22:51 |
| Daviey | fungi: Okay... considering tags have already been pushed.. what should i do? | 22:51 |
| docaedo | haha, yeah - I'll just sit back and watch you folks figure out and fix all the things so OpenStack can continue it's march towards world domination. | 22:52 |
| jeblair | clarkb: can you approve https://review.openstack.org/271085 ? | 22:52 |
| *** pradk has quit IRC | 22:52 | |
| jeblair | fungi: or you ^ so we can get reasonable ansible puppet output | 22:52 |
| fungi | Daviey: we're debating. i think temporary custom tarball job for ceilometer to be able to get that tarball to build from the existing tag (or temporary change to the main tarball job template), and undoing the addition of constraints-based tarball job for neutron-vpnaas and any others in the same boat | 22:52 |
| anteaya | docaedo: I don't want world domination | 22:52 |
| anteaya | docaedo: world participation is my personal goal | 22:52 |
| fungi | speaking of home stretch, jenkins03 updates completed so i'm putting it back into production now. this just leaves jenkins05 churning | 22:53 |
| mwhahaha | hey openstack-infra folks, could I get some help with https://review.openstack.org/#/c/271452/ as all of our reviews are currently blocked by this failing test. We've got a pending workaround but it's going to take some time to validate. | 22:53 |
| *** rcernin has quit IRC | 22:53 | |
| docaedo | anteaya: haha, I won't rest until AWS concedes defeat and switches their cloud to OpenStack! | 22:54 |
| mordred | jeblair: https://review.openstack.org/#/c/271087/ | 22:54 |
| fungi | docaedo: they might have already and just not told anybody | 22:54 |
| mordred | jeblair: is the change that changes it to go to syslog for us, btw | 22:54 |
| *** krtaylor has joined #openstack-infra | 22:54 | |
| docaedo | fungi: One way to be sure, start a tweet campaign and see if Bezos responds! | 22:54 |
| mordred | jeblair: I didn't depends-on it because it's actually safe to land before the other one | 22:54 |
| dougwig | anteaya, fungi: i told paul about the new way of doing those jobs on wednesday. do you need it patched quickly? | 22:54 |
| jeblair | mordred: right, but the other way around -- | 22:55 |
| mordred | jeblair: although I recognize now that that may be confusing | 22:55 |
| fungi | dougwig: these what jobs? | 22:55 |
| jeblair | mordred: your change to revert back to the default should depend on 271087 | 22:55 |
| mordred | jeblair: ah - you want the default switch to depends-on it? | 22:55 |
| mordred | I can do that | 22:55 |
| * mordred understands jeblair now | 22:55 | |
| dougwig | fungi: "neutron-vpnaas problem. it needs to stop using a custom "constraints-based" tarball job for now" | 22:55 |
| jeblair | mordred: yep, so we don't revert | 22:55 |
| clarkb | isn't json part of stdlib? | 22:55 |
| fungi | dougwig: oh, yep. i was about to find what other repos have the same issues and switch them back at the same time | 22:55 |
| jeblair | fungi, dougwig: ah, it's part of a python-constraints-jobs template | 22:55 |
| anteaya | I belive this is the patch that introduced the neutron vpnaas contstraints tarball job: https://review.openstack.org/#/c/263398/ | 22:56 |
| *** dims_ has joined #openstack-infra | 22:56 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/ansible-puppet: Make puppet logout destination configurable https://review.openstack.org/271086 | 22:56 |
| mordred | clarkb: yes | 22:56 |
| jeblair | fungi, dougwig: oh, wait, maybe openstack-server-release-jobs-constraints | 22:56 |
| mordred | clarkb: but it's not part of stdlib in all of the supported versions of python ansible supports | 22:56 |
| clarkb | mordred: ansible supports 2.5? | 22:57 |
| mordred | clarkb: so they asked we put that in just because | 22:57 |
| *** amitgandhinz has quit IRC | 22:57 | |
| clarkb | thats fair I guess. Surprised that anyone would support python so old | 22:57 |
| jeblair | fungi, dougwig, anteaya: i think neutron and vpnaas may have *2* sets of release jobs | 22:57 |
| fungi | jeblair: yeah, in good news, AJaeger confirmed earlier by digging in stable branch job logs that the branch-tarball-constraints jobs are actually fine, it's just tags where this is causing zuul-cloner to fall back on master branch tip | 22:58 |
| *** dims has quit IRC | 22:58 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Deal with a blank ZUUL_CHANGES later https://review.openstack.org/271559 | 22:58 |
| jeblair | fungi: that may be a coincidence though | 22:58 |
| dougwig | jeblair: is that as not good as it sounds? | 22:58 |
| jeblair | fungi: i'm not sure the behavior is actually defined | 22:58 |
| *** derekh has joined #openstack-infra | 22:58 | |
| anteaya | jeblair: that wouldn't surprise me | 22:58 |
| jeblair | dougwig: it's twice as bad as it sounds? ;) | 22:58 |
| *** diazjf has quit IRC | 22:58 | |
| anteaya | python-jobs has tarball-constraints and branch-tarball-constraints: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/python-jobs.yaml#n492 | 22:59 |
| Daviey | fungi: You've got this.. or should i be doing something? | 22:59 |
| fungi | Daviey: i've got it, though i'll keep you updated | 23:00 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:00 | |
| *** sdake has joined #openstack-infra | 23:00 | |
| jeblair | fungi, anteaya, dougwig: my assumption would be that we should have zero constraints jobs in the release, pre-release, or post pipelines at this point | 23:00 |
| fungi | so anyway, it looks like we're safe to at least swap all uses of openstack-server-release-jobs-constraints back to openstack-server-release-jobs | 23:00 |
| mordred | clarkb: https://github.com/ansible/ansible-modules-extras/pull/1298 | 23:00 |
| jeblair | fungi: it's not a swap | 23:00 |
| jeblair | fungi: it has both | 23:00 |
| fungi | oh, ew | 23:00 |
| jeblair | fungi, anteaya, dougwig: so i think the corrective action would be to remove those jobs from the associated templates, and if those templates are empty, remove the template calls | 23:01 |
| mordred | clarkb: that's wwhere the thing came from - and it has something to do with old centos servers under management | 23:01 |
| fungi | jeblair: strangely, i didn't see it running both sets of jobs | 23:01 |
| *** sdake has quit IRC | 23:01 | |
| mordred | clarkb: I do not think the puppet module per-se was an issue - but they went through and went explicit on all modules importing json | 23:02 |
| jeblair | fungi: hrm. maybe i missed something? i thought i saw both templates being called. | 23:02 |
| fungi | i wonder if the fact that they both declare {name}-merge-release-tags caused the project-templates to get "deduplicated"? | 23:02 |
| clarkb | ok I have a running nodepool-builder process | 23:02 |
| fungi | jeblair: "see it" as in when i looked at the zuul status page it was only running the three from the openstack-server-release-jobs-constraints template | 23:02 |
| fungi | i agree the layout includes both though | 23:02 |
| clarkb | issue was pid file matched main daemons | 23:02 |
| clarkb | I will be pushing a fix for that | 23:03 |
| clarkb | also we proably shouldn't share a log file so that logs go to separate location | 23:03 |
| mordred | clarkb: neat! | 23:03 |
| clarkb | I am going to restart the main nodepoold now so that it will stop running an internal builder | 23:03 |
| mordred | woot | 23:03 |
| openstackgerrit | Merged openstack-infra/ansible-puppet: Sync with upstream ansible https://review.openstack.org/271085 | 23:03 |
| *** ccrouch has quit IRC | 23:03 | |
| openstackgerrit | Merged openstack-infra/ansible-puppet: Log puppet to syslog instead of json https://review.openstack.org/271057 | 23:03 |
| jeblair | mordred: the dns.py file doesn't work with shade :( | 23:04 |
| *** ybathia has quit IRC | 23:04 | |
| anteaya | jeblair: I will see about fixing the templates | 23:04 |
| jeblair | i didn't get the server href i was expecting | 23:04 |
| fungi | jeblair: yeah, we only got logs for those three, confirming my recalled observation http://logs.openstack.org/f8/f899c3dc3423bc990e889232e451aae4b074438b/release/ | 23:04 |
| jeblair | fungi: weird | 23:05 |
| fungi | anyway, i'll get the layout patched up to remove non-change-pipeline constraints jobs for now | 23:05 |
| jeblair | might be worth checking out zuul's understanding of the config... | 23:05 |
| *** ccrouch has joined #openstack-infra | 23:06 | |
| jeblair | fungi: if they're attached to the publish job, it's possible one might be overriding the other (and we got unlucky) | 23:06 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Use different pidfile for builder daemon https://review.openstack.org/271561 | 23:06 |
| clarkb | thats one thing | 23:06 |
| mordred | jeblair: OH! | 23:07 |
| mordred | jeblair: you know what ... yeah, I'll have to do a little work for you for that. so sorry | 23:07 |
| openstackgerrit | Merged openstack-infra/puppet-bandersnatch: Make sure mercurial is installed https://review.openstack.org/271548 | 23:07 |
| mordred | jeblair: it turns out nothing else in the entirety of openstack uses those hrefs | 23:07 |
| *** salv-orlando has quit IRC | 23:07 | |
| clarkb | I am going to attempt uploading fedora-23 to hpcloud-b2 now to compare | 23:08 |
| mordred | jeblair: like, actually nothing except for rax dns | 23:08 |
| jeblair | mordred: yeah, this isn't openstack | 23:08 |
| jeblair | :( | 23:08 |
| mordred | I know | 23:08 |
| mordred | so I did not consider it in shade - I can fix for you though | 23:08 |
| mordred | jeblair: do you still need one currently so you can do the dns? | 23:08 |
| jeblair | mordred: nah, it's only relevant for rax, so i'll run dns.py standalone in the old environment | 23:09 |
| jeblair | so no rush | 23:09 |
| mordred | jeblair: ok | 23:09 |
| jeblair | probably something to make sure we have working before the flip tho | 23:09 |
| fungi | jeblair: presumably we're considering periodic pipelines also unsafe for zuul-cloner for now? | 23:10 |
| jeblair | fungi: no they're fine | 23:10 |
| *** KimJ has quit IRC | 23:10 | |
| fungi | thanks, leaving those untouched then | 23:10 |
| jeblair | fungi: basically, zuul-cloner's test cases came from the devstack-gate test cases | 23:10 |
| jeblair | fungi: so anything that worked in d-g should work with z-c. | 23:10 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Deal with a blank ZUUL_CHANGES later https://review.openstack.org/271559 | 23:10 |
| jeblair | fungi: anything d-g did not do is undefined in z-c. | 23:10 |
| fungi | makes sense | 23:10 |
| jeblair | fungi: there is actually a 1:1 mapping of d-g test cases to python unit tests for z-c | 23:11 |
| fungi | i think i knew that at one point and then conveniently forgot | 23:11 |
| *** dims_ has quit IRC | 23:13 | |
| *** salv-orlando has joined #openstack-infra | 23:14 | |
| *** eharney_ is now known as eharney | 23:15 | |
| *** jamielennox is now known as jamielennox|away | 23:15 | |
| jlvillal | 5 uh-oh 3 | 23:16 |
| *** piet has joined #openstack-infra | 23:16 | |
| clarkb | upload took just under 8 minutes | 23:16 |
| clarkb | so thats a massive improvement over 4 hours | 23:16 |
| clarkb | should I go ahead and manually run uploads across the board? | 23:17 |
| jeblair | clarkb: ++ | 23:17 |
| openstackgerrit | Sven Anderson proposed openstack-dev/pbr: Extend compile_catalog command to compile all .po files https://review.openstack.org/271562 | 23:17 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Don't run constraints jobs in post or release https://review.openstack.org/271563 | 23:17 |
| fungi | jeblair: Daviey: dougwig: ^ there's fixing the neutron-vpnaas situation | 23:17 |
| clarkb | ok so the two outstanding things are getting my nodepool hcnage of the pidfile path merged and we need to split up the logging between the builder and daemon | 23:18 |
| clarkb | the image logging can go in the builders config | 23:18 |
| clarkb | may be able to go back to a non program generated config for the main daemon | 23:18 |
| mordred | clarkb: uhm... how did upload go from 4 hours to 8 minutes? | 23:18 |
| mordred | clarkb: (I'm pleased - but wow) | 23:19 |
| clarkb | mordred: our theory has been cpu starvation. | 23:19 |
| clarkb | that was aagainst hpcloud thoiugh which has typically fast uploads, trying rax-iad now | 23:19 |
| clarkb | since rax tends to be slowest due to large image sizes | 23:19 |
| fungi | lifeless: Nakato: https://review.openstack.org/271563 is another fyi pseudo-revert | 23:19 |
| * jlvillal thinks about how when you put something back together and you have some extra screws. Same concept ;) | 23:20 | |
| clarkb | oh right it serializes all of them | 23:20 |
| clarkb | so rax isn't happening just yet :) | 23:20 |
| jpr | clarkb, asselin: btw, was able to unstuck my config on the build by going into nodepool mysql and removing the stuck job. found this very helpful https://www.mediawiki.org/wiki/Continuous_integration/Architecture/Isolation#Nodepool | 23:20 |
| mordred | clarkb: I'm very excited | 23:21 |
| *** achanda has joined #openstack-infra | 23:21 | |
| jpr | service nodepool start was a better path forward than testing the dib build directly. nodepool launched a job to build and I started seeing artifacts in the cache for the build. having other errors (space for build and old keystone) but the nodepool looks like it's doing what it's supposed to. | 23:22 |
| openstackgerrit | Anita Kuno proposed openstack-infra/project-config: REmove constraints from doc, tarball constraint templates https://review.openstack.org/271564 | 23:22 |
| anteaya | jeblair fungi ^^ | 23:22 |
| Daviey | thanks fungi | 23:23 |
| openstackgerrit | Sven Anderson proposed openstack-dev/pbr: Extend compile_catalog command to compile all .po files https://review.openstack.org/271562 | 23:23 |
| anteaya | jeblair fungi do we still need mine even though fungi offered 271563? | 23:23 |
| *** claudiub has quit IRC | 23:23 | |
| * anteaya reviews 271563 | 23:23 | |
| fungi | anteaya: is that building on 271563? | 23:23 |
| fungi | or a duplicate of? | 23:23 |
| anteaya | fungi: instead of | 23:25 |
| anteaya | I didn't realize you were creating 271563 | 23:25 |
| anteaya | I think yours is better | 23:25 |
| anteaya | I can abandon mine | 23:25 |
| fungi | anteaya: i'll leave it to reviewers to decide. i'm on to solving the ceilometer problem now | 23:25 |
| anteaya | fungi: happy ceilometer | 23:25 |
| anteaya | jeblair: I think fungi's 271563 is better than my 271564, what do you think? | 23:26 |
| *** Swami_ has quit IRC | 23:26 | |
| *** Swami_ has joined #openstack-infra | 23:26 | |
| jeblair | anteaya: yeah, i think 271564 may have the problem that we're leaving the jobs named constraints but changing what they do to be unconstrained, which will be confusing | 23:26 |
| anteaya | yes | 23:27 |
| anteaya | I'll abandon | 23:27 |
| jeblair | so better to just leave those jobs around for now, unused, and then add them back into use once z-c is updated | 23:27 |
| anteaya | yup | 23:27 |
| *** ddieterly has quit IRC | 23:27 | |
| fungi | yeah, i was just trying to stop running them (and remove the project-templates in the layout file since they're an attractive nuisance at the moment) | 23:28 |
| anteaya | yup | 23:28 |
| anteaya | +3 to you | 23:28 |
| anteaya | I'm going to get dinner | 23:28 |
| fungi | also they were duplicative, and possibly exposing some undefined behavior in zuul around two project-templates on a project each containing a job in common | 23:28 |
| anteaya | yeah | 23:29 |
| *** EricGonc_ has quit IRC | 23:29 | |
| jeblair | mordred: so my change to puppet-ansible will somehow land on our hosts (just puppetmaster i guess?) | 23:29 |
| mordred | jeblair: yes. it will get updated in playbooks/update_puppet.yaml | 23:30 |
| jeblair | ok | 23:30 |
| mordred | jeblair: in the line - shell: ansible-galaxy install --force -r roles.yaml | 23:30 |
| mordred | jeblair: if you want to force the issue, you can just run ansible-galaxy install --force -r roles.yaml | 23:30 |
| *** EricGonczer_ has joined #openstack-infra | 23:31 | |
| jeblair | ah, i need to add ssh keys | 23:32 |
| jeblair | should probably have launch-node do that | 23:32 |
| jeblair | Last login: Thu Jan 1 00:00:10 1970 | 23:33 |
| jeblair | so that's what, 600 in epoch time? weird. | 23:33 |
| clarkb | rax upload has been going for about 10 minutes now | 23:34 |
| clarkb | longer uploads are expected because its a bigger image | 23:34 |
| clarkb | will have to see how much longer it is though | 23:34 |
| jeblair | mordred: ansible-playbook -l mirror-update.openstack.org playbooks/remote_puppet_else.yaml | 23:35 |
| jeblair | mordred: fatal: [mirror-update.openstack.org]: FAILED! => {"changed": false, "failed": true, "msg": "Error while linking: [Errno 2] No such file or directory", "path": "/opt/system-config/hieradata", "state": "absent"} | 23:35 |
| *** tonytan4ever has quit IRC | 23:35 | |
| *** EricGonczer_ has quit IRC | 23:35 | |
| jeblair | mordred: that's for: TASK [puppet : ensure hieradata manifest link is present] ********************** | 23:35 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Temporarily pin pip<8 for ceilometer tarballs https://review.openstack.org/271567 | 23:35 |
| fungi | jeblair: Daviey: ^ there's the workaround for the ceilometer tarball | 23:36 |
| *** rbrndt has quit IRC | 23:36 | |
| *** EricGonczer_ has joined #openstack-infra | 23:36 | |
| fungi | i'm working up the fix for ceilometer stable/kilo now so that it can be included in 2015.1.4 | 23:36 |
| *** EricGonczer_ has quit IRC | 23:36 | |
| jeblair | fungi: lgtm | 23:37 |
| openstackgerrit | Merged openstack-infra/project-config: Don't run constraints jobs in post or release https://review.openstack.org/271563 | 23:40 |
| * krotscheck is back for a few minutes | 23:41 | |
| krotscheck | jeblair: Got your note about mirror update. | 23:41 |
| krotscheck | jeblair: Is there already a patch for that rename, or shall I just go do it? | 23:41 |
| krotscheck | jeblair: Ah, I see it's been fixed. | 23:41 |
| *** edmondsw has quit IRC | 23:41 | |
| *** thorst has quit IRC | 23:43 | |
| fungi | Daviey: i've backported cdent's fix from liberty to drop downloadcache in ceilometer in kilo now (for the sake of 2015.1.4 and later): https://review.openstack.org/271570 | 23:43 |
| *** thorst has joined #openstack-infra | 23:43 | |
| fungi | also, jenkins05 finally finished updating successfully, so i'm placing it back into production | 23:43 |
| fungi | that's the last of them! | 23:43 |
| jeblair | krotscheck: yep, all good, just waiting on some initial puppet/ansible runs and possibly some weirdness there | 23:43 |
| jeblair | fungi: yay! that only took 2 days! | 23:44 |
| krotscheck | jeblair: Gotcha. Just rechecked my typo patch | 23:44 |
| fungi | jeblair: yep, that's the blink of an eye on a cosmological timescale | 23:44 |
| fungi | also on a cosmetological one | 23:45 |
| jeblair | fungi: wow. | 23:45 |
| greghaynes | clarkb: oh wow, so separate builder ftw? | 23:45 |
| anteaya | fungi: yay, all the jenkinsii | 23:45 |
| jeblair | i think we should leave the openstack foundation and instead operate under the auspices of the long-now foundation. | 23:45 |
| clarkb | greghaynes: so far yes | 23:45 |
| greghaynes | clarkb: I would be really interested to see what the root cause of the massive slowdown is | 23:46 |
| *** derekh has quit IRC | 23:46 | |
| fungi | jeblair: we should start "the foundation" | 23:46 |
| jeblair | fungi: i'll start editing wikipedia | 23:46 |
| greghaynes | I know python and threads aren't great but I didn't think it would cause that much of a slowdown | 23:46 |
| fungi | itym psychohistremedia? | 23:47 |
| clarkb | greghaynes: we do have a lot of threads that may hurt | 23:47 |
| jeblair | greghaynes: nodepool is cpu bound | 23:47 |
| clarkb | most dont do large io so are fine | 23:47 |
| *** Sukhdev has quit IRC | 23:47 | |
| greghaynes | Ah, CPU bound while taking the GIL | 23:47 |
| *** sdake has joined #openstack-infra | 23:47 | |
| *** ybathia has joined #openstack-infra | 23:48 | |
| jeblair | yeah, and the image uploads are doing a lot of data moving in python (since we're building giant http requests -- in python) | 23:48 |
| *** simondodsley has quit IRC | 23:48 | |
| jeblair | so this isn't like an io thread that can go off in C-land and do its thing | 23:49 |
| fungi | jeblair: oh, right, "wikipedia galactica" | 23:49 |
| greghaynes | Yep, makes sense | 23:49 |
| krotscheck | jeblair: i see error status but cant access the report? | 23:50 |
| mordred | jeblair: well piddle | 23:50 |
| jeblair | greghaynes: so even though nodepool proper spends most of its time just waiting for network io (great for threads), once we have around 1000 of them we actually have enough of them waking up that even that is now consuming the 1 core that the gil gives us | 23:50 |
| jeblair | greghaynes: which is why the next step is nodepool launch workers :) | 23:51 |
| jeblair | but maybe after v3 | 23:51 |
| greghaynes | Haha | 23:51 |
| greghaynes | Ya, and after builders are stableish | 23:51 |
| clarkb | rax still going | 23:51 |
| jeblair | well, that's tuesday | 23:51 |
| clarkb | ~25 minutes so far | 23:51 |
| *** thorst has quit IRC | 23:52 | |
| jeblair | clarkb: i want to say i saw something like 20 mins for a bluebox upload with no contention? | 23:52 |
| clarkb | it was 8 minutes for hpcloud and the hpcloud images are aout 1/3 the size | 23:53 |
| *** ybathia has quit IRC | 23:53 | |
| *** piet has quit IRC | 23:53 | |
| mordred | jeblair: doh. | 23:53 |
| clarkb | 16GB vs 5GB | 23:54 |
| clarkb | I don't see a raw image though | 23:54 |
| clarkb | I thought we had at least one cloud using raw | 23:54 |
| clarkb | oh we must not have fredora 23 on that cloud | 23:54 |
| *** edmondsw has joined #openstack-infra | 23:54 | |
| clarkb | there are 3 images with raw | 23:54 |
| jeblair | mordred: fwiw, the cron run just ran and hit the same errors | 23:55 |
| jeblair | mordred: but i'm guessing from your doh you figured something out? | 23:55 |
| clarkb | 30 minutes for rax | 23:55 |
| mordred | jeblair: yup. logic error on nodes on which we've never run this before :) | 23:56 |
| *** jswarren_ has joined #openstack-infra | 23:56 | |
| jeblair | clarkb: nice! (i guess, i mean, now we return to the world where we think 30 minutes is a long time :) | 23:56 |
| *** hdd_ has joined #openstack-infra | 23:56 | |
| *** hdd has quit IRC | 23:56 | |
| fungi | on plank timescales it's a veritable eternity! | 23:56 |
| fungi | planck | 23:56 |
| *** dims has joined #openstack-infra | 23:56 | |
| *** ashleigh_ has joined #openstack-infra | 23:56 | |
| * jeblair makes fungi walk the planck | 23:57 | |
| * fungi likes bouncing between temporal extremes | 23:57 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/ansible-puppet: Don't try to make a link before the directory https://review.openstack.org/271572 | 23:57 |
| mordred | jeblair: ^^ | 23:57 |
| mordred | jeblair: turns out we need to rsync the puppet modules directory BEFORE writing content in to it | 23:57 |
| clarkb | at a half an hour we can have 48 images max per builder | 23:58 |
| clarkb | roughly | 23:58 |
| clarkb | in a worst case | 23:58 |
| fungi | mmmmmmaths | 23:58 |
| jeblair | clarkb, greghaynes: we could probably multiplex those a little bit -- have a couple of threads doing image uploads. | 23:58 |
| mordred | mmm. matttthhhhhhhs | 23:58 |
| clarkb | if we excluse hpcloud we have 7regions? | 23:59 |
| jeblair | clarkb, greghaynes: like, more than 1 but less than 1000. | 23:59 |
| clarkb | 2 ovh, 3 rax, 1 internap, 1 bluebox | 23:59 |
| mordred | I want to say "docker" or "AFS" in response to the multi-plexing - but I don't think either is right | 23:59 |
| clarkb | that comes down to ~6 images per region | 23:59 |
| jeblair | mordred: if we can get glance to read from afs... | 23:59 |
| *** ashleighfarnham has quit IRC | 23:59 | |
| *** jswarren has quit IRC | 23:59 | |
| *** Sukhdev has joined #openstack-infra | 23:59 | |
| clarkb | trusty, centos7, fedora 23 | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!