| anteaya | I am arriving Saturday yes | 00:02 |
|---|---|---|
| *** gyee has quit IRC | 00:02 | |
| *** nati_ueno has quit IRC | 00:02 | |
| anteaya | would love to have dinner Sunday, clarkb jog0 pleia2 | 00:02 |
| anteaya | pleia2: see you when you are online, enjoy dinner | 00:03 |
| reed | anteaya, in the bay area? | 00:03 |
| *** nhm has joined #openstack-infra | 00:03 | |
| anteaya | seattle | 00:04 |
| anteaya | in SF the weekend after | 00:04 |
| anteaya | at pleia2 's house | 00:04 |
| *** lnxnut has quit IRC | 00:04 | |
| reed | nice! I should be around the weekend after | 00:04 |
| jog0 | clarkb: I will hold off trying things and see what you can put together | 00:04 |
| clarkb | jog0: I just pushed the change to gerrit, going to apply it on the logstash server now | 00:04 |
| clarkb | jog0: applied any logs that go through at this point should have the new ocntent assuming I didn't derp anything | 00:07 |
| jog0 | sweet, so I can use the logurl to filter | 00:08 |
| clarkb | yeah | 00:08 |
| clarkb | or the UUID if you truncate it to 7 characters | 00:08 |
| jog0 | ohh thats what that is | 00:08 |
| jog0 | either way shouldn't be too bad | 00:09 |
| clarkb | right and either will give you a 1:1 mapping | 00:10 |
| jog0 | clarkb: well with 8 jobs running right now I will hold off testing this till Monday | 00:11 |
| clarkb | jog0: ok, I am about done for the day as well | 00:12 |
| locke105 | wow... did you guys know that you can parse JSON with pyYAML... | 00:12 |
| *** grizzled has joined #openstack-infra | 00:12 | |
| clarkb | just making sure that I didn't break anything with my change | 00:12 |
| clarkb | locke105: yeah json is a subset of yaml ish | 00:12 |
| jog0 | clarkb: thanks for the help | 00:12 |
| clarkb | I think there are a couple corner cases but you ignore those and pretend everything is good | 00:12 |
| locke105 | i wonder if i can convert my jenkins job configs from XML to JSON and then run them through jenkins-job-buidler.... | 00:13 |
| jog0 | can't belive this actually works | 00:13 |
| clarkb | jog0: no problem I am really excited about the work you are doing so any thing I can do to help is good | 00:13 |
| jog0 | and how easy it is | 00:13 |
| clarkb | jog0: stuff like this was why I wanted a thing like logstash and elasticsearch | 00:13 |
| jog0 | yeah its pretty boss | 00:13 |
| jog0 | anyway see you sunday | 00:13 |
| clarkb | jog0: yup | 00:14 |
| locke105 | logstash is pretty badass | 00:14 |
| jog0 | locke105: https://github.com/jogo/elasticRecheck | 00:14 |
| locke105 | i have one set up to index nova messages on one of my openstack's internally | 00:14 |
| clarkb | locke105: if you do a 1:1 mapping from xml to json to yaml I don't think you will get yaml that JJB understands | 00:14 |
| clarkb | locke105: one of the nice things about JJB is it simplifies the configs | 00:14 |
| clarkb | locke105: the CI system has one to index our build logs | 00:14 |
| locke105 | yeah | 00:15 |
| fungi | confirmed that using UtahDave's trigger syntax on the salt-trigger slave, the master does in fact instruct all the slaves to fire the puppet agent successfully | 00:15 |
| locke105 | thats whwere i found out about kibana | 00:15 |
| locke105 | this is interesting | 00:15 |
| clarkb | fungi: WOOT | 00:15 |
| locke105 | how do you map a failure to a bug? | 00:16 |
| jog0 | locke105: by hand | 00:16 |
| jog0 | for now | 00:16 |
| *** CaptTofu has joined #openstack-infra | 00:16 | |
| locke105 | oh | 00:16 |
| jog0 | its not that hard it turns out | 00:16 |
| clarkb | ya so a human needs to sort out some identifier that can be searched for | 00:16 |
| clarkb | but once you have that you can check for instances of that bug | 00:16 |
| locke105 | gotcha | 00:16 |
| jog0 | the kibana interface makes it easy as point and click | 00:16 |
| locke105 | yeah | 00:17 |
| clarkb | eventual work will have us doing statistical analysis to find anomolies and potentially submit bugs | 00:17 |
| *** blamar has quit IRC | 00:17 | |
| locke105 | neat | 00:17 |
| jog0 | right, but this is the first baby step to awesomeness | 00:17 |
| clarkb | yup and it is pretty aewsome as is | 00:17 |
| jog0 | once its wired into the workflow better its gonna be really sweet. | 00:18 |
| jog0 | I am hoping to get this moved over to openstack-infra in the near future and then send an ML email out about it inviting people to help with the queries | 00:18 |
| clarkb | ++ | 00:18 |
| jog0 | gonan try out mordred cookiecutter to fix the repo up | 00:19 |
| jog0 | anyway I'm out for real | 00:19 |
| locke105 | so many new cool new things... so little time | 00:20 |
| *** michchap has joined #openstack-infra | 00:23 | |
| clarkb | locke105: right! | 00:25 |
| *** doctorjnupe has quit IRC | 00:25 | |
| clarkb | locke105: re using yaml to parse json the rails vulnerabilites at the beginning of the year were related to that | 00:25 |
| clarkb | locke105: because yaml allows you to serialize native objects and code you could craft special JSON that got parsed and did nasty things | 00:26 |
| clarkb | so use the pyYAML safe parsing method if using it to parse json | 00:26 |
| clarkb | however json support is in python's stdlib so should probably just use that :) | 00:26 |
| clarkb | and now time to AFK have a good weekend everyone | 00:26 |
| *** dcramer__ has joined #openstack-infra | 00:28 | |
| locke105 | thanks for the tip | 00:29 |
| locke105 | most of my jobs share common config so it shouldn't be too bad to pull them into yaml, should save me a ton of work whenever i have new projects to add | 00:30 |
| *** NobodyCam has quit IRC | 00:32 | |
| *** mrodden has joined #openstack-infra | 00:33 | |
| *** NobodyCam has joined #openstack-infra | 00:33 | |
| *** michchap has quit IRC | 00:38 | |
| *** lnxnut has joined #openstack-infra | 00:49 | |
| *** sarob has joined #openstack-infra | 00:54 | |
| portante | hi folks, where can I find swift's code coverage report? | 00:59 |
| portante | the one that is generated after a commit occurs? | 00:59 |
| *** Ryan_Lane has quit IRC | 00:59 | |
| *** sarob has quit IRC | 00:59 | |
| clarkb | portante: http://logs.openstack.org/firsttwocharsofgitsha/gitsha | 01:00 |
| clarkb | for the commit you want to eee | 01:01 |
| *** dcramer__ has quit IRC | 01:03 | |
| portante | k thx | 01:03 |
| anteaya | reed: just got off the phone | 01:04 |
| anteaya | that would be awesome, I am still looking forward to your sailing invitation | 01:04 |
| anteaya | I'm not sure how much I can pack into this trip | 01:05 |
| anteaya | so I will be a good house guest and show up and listen | 01:05 |
| anteaya | :D | 01:05 |
| portante | clarkb: verified it works now, thanks! | 01:07 |
| anteaya | it is 9pm for me and I still have a ton to do plus wake up early, so I'm off | 01:08 |
| anteaya | night | 01:08 |
| *** anteaya has quit IRC | 01:08 | |
| *** atiwari has quit IRC | 01:17 | |
| *** reed has quit IRC | 01:34 | |
| *** adalbas has quit IRC | 01:44 | |
| *** michchap has joined #openstack-infra | 02:03 | |
| *** senk has quit IRC | 02:06 | |
| *** paul-- has quit IRC | 02:08 | |
| *** paul-- has joined #openstack-infra | 02:13 | |
| *** michchap has quit IRC | 02:16 | |
| *** michchap has joined #openstack-infra | 02:16 | |
| *** blamar has joined #openstack-infra | 02:17 | |
| *** Ryan_Lane has joined #openstack-infra | 02:42 | |
| *** dcramer__ has joined #openstack-infra | 02:48 | |
| *** Ryan_Lane has quit IRC | 02:54 | |
| *** Ryan_Lane has joined #openstack-infra | 02:55 | |
| *** thomasm has joined #openstack-infra | 03:02 | |
| *** melwitt has quit IRC | 03:09 | |
| *** doctorjnupe has joined #openstack-infra | 03:10 | |
| *** vipul has quit IRC | 03:10 | |
| *** vipul has joined #openstack-infra | 03:11 | |
| *** dcramer__ has quit IRC | 03:14 | |
| *** Ryan_Lane has quit IRC | 03:25 | |
| *** thomasm has quit IRC | 03:27 | |
| *** doctorjnupe has quit IRC | 03:27 | |
| *** michchap_ has joined #openstack-infra | 03:28 | |
| *** michchap has quit IRC | 03:28 | |
| *** jerryz has quit IRC | 03:36 | |
| *** vipul is now known as vipul-away | 03:48 | |
| *** blamar has quit IRC | 03:55 | |
| *** blamar has joined #openstack-infra | 04:02 | |
| *** UtahDave has joined #openstack-infra | 04:21 | |
| lifeless | openstackgerrit is awol in #tripleo | 04:25 |
| clarkb | lifeless: yeah it died at some point today | 04:25 |
| clarkb | lifeless: I will kick it tomorrow when we do project renaming | 04:26 |
| lifeless | oooh whats being renamed ? | 04:26 |
| clarkb | marconi is moving into openstack/ | 04:27 |
| lifeless | clarkb: speaking of, we can gc the stackforge tripleo repos now. | 04:27 |
| clarkb | lifeless: which ones? (though I think mordred and jeblair like keeping them around as we don't really delete things around here we just shuffle them around) | 04:28 |
| lifeless | we kept them so we didn't break scripts when we moved to openstack/ | 04:28 |
| lifeless | this was a deliberate temporary thing | 04:29 |
| lifeless | oh, looks like they've been cleaned up in the last 48h | 04:29 |
| lifeless | cool. | 04:29 |
| lifeless | (GheRivero was going to get it done, I thought he'd missed everyone :P) | 04:29 |
| *** lnxnut_ has joined #openstack-infra | 04:43 | |
| *** lnxnut has quit IRC | 04:45 | |
| *** thomasm has joined #openstack-infra | 04:53 | |
| *** UtahDave has quit IRC | 04:55 | |
| *** sdake has quit IRC | 04:57 | |
| *** sdake has joined #openstack-infra | 04:57 | |
| *** thomasm has quit IRC | 05:06 | |
| *** reed has joined #openstack-infra | 05:10 | |
| *** reed has quit IRC | 05:12 | |
| *** vipul-away is now known as vipul | 05:13 | |
| *** michchap_ has quit IRC | 05:18 | |
| *** michchap has joined #openstack-infra | 05:18 | |
| *** vogxn has joined #openstack-infra | 05:55 | |
| *** vogxn has quit IRC | 06:03 | |
| *** SergeyLukjanov has joined #openstack-infra | 06:15 | |
| *** dmakogon_ has joined #openstack-infra | 06:22 | |
| *** lnxnut_ has quit IRC | 06:25 | |
| *** lnxnut has joined #openstack-infra | 06:55 | |
| *** Ryan_Lane has joined #openstack-infra | 07:01 | |
| *** Ryan_Lane has quit IRC | 07:02 | |
| *** lnxnut has quit IRC | 07:05 | |
| *** yolanda has joined #openstack-infra | 07:10 | |
| *** doctorjnupe has joined #openstack-infra | 07:16 | |
| *** yolanda has quit IRC | 07:19 | |
| *** doctorjnupe has quit IRC | 07:30 | |
| *** lnxnut has joined #openstack-infra | 07:31 | |
| *** lnxnut has quit IRC | 07:36 | |
| *** yolanda has joined #openstack-infra | 07:45 | |
| *** SergeyLukjanov has quit IRC | 08:03 | |
| *** Ryan_Lane has joined #openstack-infra | 08:22 | |
| *** fbo_away is now known as fbo | 08:22 | |
| *** nicedice has quit IRC | 08:39 | |
| *** mdenny has quit IRC | 08:54 | |
| *** MIDENN_ has joined #openstack-infra | 08:54 | |
| *** MIDENN_ has quit IRC | 08:54 | |
| *** SergeyLukjanov has joined #openstack-infra | 08:55 | |
| *** SergeyLukjanov has quit IRC | 08:55 | |
| *** MIDENN_ has joined #openstack-infra | 08:55 | |
| *** thomasbiege has joined #openstack-infra | 09:02 | |
| *** thomasbiege has quit IRC | 09:08 | |
| *** SergeyLukjanov has joined #openstack-infra | 09:13 | |
| *** grizzled has quit IRC | 09:32 | |
| *** echohead has left #openstack-infra | 09:46 | |
| *** Ryan_Lane has quit IRC | 09:51 | |
| *** SergeyLukjanov has quit IRC | 09:53 | |
| *** alexpilotti has joined #openstack-infra | 09:54 | |
| *** thomasbiege has joined #openstack-infra | 10:04 | |
| *** thomasbiege has quit IRC | 10:17 | |
| *** SergeyLukjanov has joined #openstack-infra | 10:20 | |
| *** lnxnut has joined #openstack-infra | 10:23 | |
| *** lnxnut has quit IRC | 10:30 | |
| *** niska has quit IRC | 11:16 | |
| *** yolanda has quit IRC | 11:18 | |
| *** niska has joined #openstack-infra | 11:19 | |
| *** odyssey4me has joined #openstack-infra | 11:20 | |
| *** jpeeler has quit IRC | 11:51 | |
| *** jpeeler has joined #openstack-infra | 11:51 | |
| *** dmakogon_ has quit IRC | 11:59 | |
| *** nayward has joined #openstack-infra | 11:59 | |
| *** nayward has quit IRC | 12:14 | |
| *** alexpilotti has quit IRC | 12:32 | |
| *** sdake_ has quit IRC | 12:43 | |
| *** nicedice has joined #openstack-infra | 13:00 | |
| *** flaper87|afk is now known as flaper87 | 13:03 | |
| *** nayward has joined #openstack-infra | 13:47 | |
| *** michchap has quit IRC | 13:47 | |
| *** ArxCruz has joined #openstack-infra | 13:55 | |
| *** dkliban has joined #openstack-infra | 14:07 | |
| *** grizzled has joined #openstack-infra | 14:12 | |
| *** boris-42 has joined #openstack-infra | 14:50 | |
| *** rnirmal has joined #openstack-infra | 14:53 | |
| *** dcramer__ has joined #openstack-infra | 15:05 | |
| fungi | lifeless: nah, after he commented in here i diffed what we had in review.o.o and git.o.o against github and then removed the github cruft | 15:18 |
| *** dcramer__ has quit IRC | 15:22 | |
| mordred | morning fungi | 15:22 |
| mordred | I see that the salt thing worked! | 15:22 |
| *** nayward has quit IRC | 15:28 | |
| fungi | mordred: yes, so far... assuming this holds stable the remaining steps i think will be (1) make a jenkins job to fire it on merges for relevant projects, (2) disable the cron jobs for puppet apply on slaves, (3) extend saltiness to remaining servers, (4) disable puppet agent daemon on those | 15:37 |
| mordred | fungi: I thnk those are excellent steps | 15:37 |
| fungi | oh, actually (0) is to expand the trigger to also pull the git repo on the master before sending the update command to the minions | 15:38 |
| mordred | ah yes | 15:38 |
| fungi | which means also making the master a minion | 15:38 |
| fungi | but easy enough to tackle | 15:38 |
| fungi | and i think to avoid replication races we want the master's clone to use gerrit as its remote | 15:39 |
| fungi | the tighter we wrench down the update tolerances here the more we're going to expose other races too. i think the incomplete git clones with missing origin/HEAD on zuul are because in some cases it may start trying to cache them before manage-projects is done running | 15:41 |
| fungi | speculation, i have no concrete evidence to back that up | 15:41 |
| *** UtahDave has joined #openstack-infra | 15:42 | |
| fungi | but it's the sort of race i would expect to expose more of once this is in place | 15:42 |
| mordred | yah. well - also, once we've got this, there are potentially more options to run some actions in sequence | 15:45 |
| mordred | like, if we didn't have puppet run manage projects, but instead made it a salt thing, so that a) we could trigger it manually easily but b) we could construct sequencing for all things that need to happen when projects.yaml gets updated | 15:46 |
| mordred | I think that could be a real win | 15:46 |
| mordred | or it might just be extra complex | 15:46 |
| fungi | it will take some meditating, for sure | 15:47 |
| *** odyssey4me has quit IRC | 16:01 | |
| *** odyssey4me has joined #openstack-infra | 16:01 | |
| mordred | clarkb: if you happen to wake up and feel like battling python3 utf8 things is fun | 16:13 |
| mordred | clarkb: the python3.3 failure on https://review.openstack.org/#/c/42337/ has me 100% baffled | 16:14 |
| fungi | clarkb and i will be working on the marconi rename in about 45 minutes anyway | 16:14 |
| fungi | so i assume he's waking up | 16:14 |
| mordred | yay! | 16:14 |
| mordred | fungi: I might actually be around - sandy's still sleeping, so we're not out doing nola tings yet | 16:15 |
| *** DennyZhang has joined #openstack-infra | 16:18 | |
| fungi | still sleeping? now's the time for a breakfast of greasy gumbo and chilled jager shots | 16:18 |
| fungi | mordred: so i think _run_git_command() is returning a byte type, which you can't .split() on? | 16:20 |
| fungi | i haven't tried recreating it yet | 16:20 |
| fungi | but i think you may want _run_git_command().decode().split() | 16:21 |
| clarkb | morning | 16:22 |
| fungi | howdy | 16:23 |
| clarkb | fungi: mordred https://etherpad.openstack.org/marconi-org-move I am filling that out now | 16:24 |
| fungi | excellent | 16:24 |
| fungi | clarkb: note the slight modification i made to one of the update queries so we preserve timestamp fields | 16:25 |
| mordred | fungi: there's already a decode in the run_shell_command function | 16:32 |
| mordred | fungi: sandy does not think your breakfast suggestion sounds pallatable | 16:33 |
| fungi | aww | 16:33 |
| fungi | i got craving it after i mentioned it | 16:33 |
| clarkb | I agree with sandy | 16:33 |
| mordred | hrm. I wonder if we could script an org move using salt (once we've got the salt stuff up and going) | 16:33 |
| * mordred especially likes salt for "run this chunk of things right now kthxbai" | 16:34 | |
| mordred | would still mean downtime, but potentially very very short downtime | 16:34 |
| mordred | you know what I'd like to do - but would mean every dev in openstack killing me? | 16:37 |
| mordred | rename python-*client to just *client | 16:37 |
| clarkb | +++++++++++++++ | 16:37 |
| mordred | it's a stupid departure | 16:37 |
| clarkb | fungi: did we lose you? | 16:39 |
| fungi | clarkb: lose me where? | 16:40 |
| clarkb | fungi: you dropped off of the etherpad | 16:41 |
| fungi | oh, my wireless hates etherpad | 16:41 |
| fungi | i have to reload the tab fairly frequently | 16:42 |
| clarkb | I will put the jenkins masters in shutdown mode in ~10 minutes | 16:42 |
| fungi | apparently a little bit of packet loss and the ajaxy bits stop working | 16:42 |
| fungi | okay, sounds good | 16:42 |
| *** DennyZhang has quit IRC | 16:42 | |
| mordred | fungi: I have that problem from time to time - we shoudl sort that out at some point | 16:44 |
| clarkb | mordred: fungi: my upgrade etherpad puppet stuff is just about ready I think | 16:44 |
| clarkb | mordred: fungi: I want to spin up a new host from scratch and leave the old behind | 16:44 |
| fungi | also just realized, when we eventually have salt triggering puppet runs everywhere, we're going to need to begin stopping salt-minion instead on systems where we currently stop the puppet agent daemon or where we currently comment out the puppet apply cron jobs | 16:44 |
| fungi | clarkb: yeah, i'm 100% in favor of the new etherpad plan. hopefully i'll get a moment to review that change this weekend | 16:45 |
| clarkb | mordred: fungi: the current patchset redirects to /p/padname and makes import/export work \o/ | 16:45 |
| fungi | clarkb: i saw! very exciting | 16:46 |
| fungi | clarkb: on the marconi move steps, do we not need to shut down jenkins.o.o i guess? (would it run any jobs which would try to use that repo?) | 16:47 |
| clarkb | fungi: probably not, but it won't hurt to make things queue just to be sure | 16:48 |
| fungi | is it safe to delete the local mirror on review.o.o instead of moving the dir? will gerrit autocreate it again? | 16:49 |
| clarkb | fungi: yes gerrit should autocreate it again, I suppose moving it is better | 16:50 |
| fungi | i've only previously mv'd the local mirror repo, so didn't know it would | 16:50 |
| fungi | i thought we also created and init'd it in manage-projects for some reason instead of having gerrit create it | 16:50 |
| clarkb | fungi: jeepyb will actually recreate it, but that will use more cpu cycles | 16:50 |
| clarkb | fungi: so moving is better, good catch | 16:51 |
| clarkb | fungi: also re jenkins.o.o any job that may talk to review.o.o could cause problems | 16:51 |
| fungi | yep, good point, not just that repo | 16:51 |
| fungi | since gerrit will be offline | 16:51 |
| clarkb | jenkinses prepared for shutdown | 16:53 |
| clarkb | we have about 16 minutes until jenkins will be done running jobs | 16:57 |
| fungi | k | 17:01 |
| clarkb | fungi: do you want to do the db update and repo move in review_site and I can do the bookkeeping on our mirrors? | 17:03 |
| fungi | sure thing. logging into review.o.o now | 17:03 |
| clarkb | fungi: no rush we are still waiting on some tests | 17:04 |
| *** paul-- has quit IRC | 17:08 | |
| *** dachary has quit IRC | 17:09 | |
| fungi | zuul says eta ~3 minutes | 17:11 |
| *** dachary has joined #openstack-infra | 17:12 | |
| fungi | finally got the last of the straggler slaves with hung puppet et cetera sync'd up to the salt master | 17:12 |
| clarkb | fungi: nice, that will be helpful when updating the jenkins job workspaces | 17:12 |
| fungi | except for the devstack-launch slaves, which i believe are pending deletion anyway | 17:13 |
| fungi | clarkb: yep, i expected so | 17:13 |
| fungi | which is why i wanted to make sure they were sane before we started | 17:13 |
| clarkb | actuall g-g-p may hanlde this now | 17:14 |
| clarkb | fungi: it does handle it I think I will mark that step as unnecessary | 17:14 |
| fungi | even better! | 17:15 |
| fungi | since it looks like i have about 10% of the slaves which submitted keys for approval not currently responding to salt ping | 17:15 |
| fungi | also, jenkins is quiet now | 17:15 |
| clarkb | looks like zuul is done, I am ready when you are | 17:15 |
| fungi | going ahead | 17:15 |
| fungi | wait | 17:16 |
| fungi | oh, nevermind. that new one is queued | 17:16 |
| fungi | good | 17:16 |
| clarkb | fungi: let me know when gerrit is down | 17:16 |
| fungi | it's down | 17:16 |
| Alex_Gaynor | fungi: eh, sorry, are you guys trying to do a thing and didn't want me to reverify there? You can cancel it and I can re-verify later if that's easier | 17:16 |
| fungi | Alex_Gaynor: nope, it's fine | 17:16 |
| clarkb | Alex_Gaynor: we have protected ourselves from you :) your jobs will start when we are done | 17:17 |
| fungi | it'll start running as soon as we're done | 17:17 |
| clarkb | fungi: is gerrit down? I can start updating the mirrors? | 17:17 |
| * clarkb goes ahead and deletes github repos | 17:18 | |
| fungi | clarkb: it's been down, yes | 17:18 |
| *** paul-- has joined #openstack-infra | 17:19 | |
| fungi | clarkb: once github and gitXX.o.o are done, we should be ready to start gerrit again | 17:21 |
| clarkb | fungi: I think I am done moving things | 17:21 |
| fungi | looks like we're ready? | 17:21 |
| fungi | k | 17:21 |
| clarkb | github repos deleted too | 17:21 |
| clarkb | yup | 17:21 |
| fungi | i can log into gerrit, so we're clear to cancel shutdown on the jenkinses | 17:22 |
| clarkb | ok doing that now | 17:22 |
| portante | fungi: is this scheduled gerrit down time? | 17:23 |
| clarkb | fungi: done | 17:23 |
| fungi | merging that change now | 17:23 |
| clarkb | portante: yes, just not widely announced because we only needed a couple minutes on Saturday | 17:24 |
| portante | great, thanks | 17:24 |
| clarkb | fungi: wait for that change to be on the puppetmaster before starting puppet again on review.o.o | 17:24 |
| clarkb | fungi: that way we should get a puppet run with the correct info immediately | 17:25 |
| fungi | yep | 17:25 |
| clarkb | cgit appears to have cached stackforge/marconi on the front page... that should go away eventually | 17:25 |
| *** SergeyLukjanov has quit IRC | 17:26 | |
| clarkb | hmm | 17:26 |
| fungi | the change is merged. i'll pull on the puppetmaster | 17:26 |
| *** openstackgerrit has joined #openstack-infra | 17:26 | |
| fungi | also, restarted gerritbot ^ | 17:26 |
| clarkb | I can't get https://git.openstack.org/cgit/openstack/marconi | 17:27 |
| fungi | clarkb: we probably need to restart apache on the git servers? | 17:28 |
| fungi | anyway, puppet master has the change we merged now... safe to start puppet again on review.o.o yes? | 17:28 |
| *** thomasbiege1 has joined #openstack-infra | 17:29 | |
| clarkb | fungi: I think I need to edit /etc/cgitrepos as well | 17:29 |
| clarkb | fungi: yes go for it, I will poke at cgit | 17:29 |
| fungi | clarkb: ahh, yes that file may only get refreshed when the project.yaml changes | 17:29 |
| fungi | so probably needs a sed -i before restarting apache on those boxe | 17:30 |
| fungi | s | 17:30 |
| clarkb | fungi: one of them has updated, I think puppet will take care of it for us if we are patient /me tries to be patient | 17:31 |
| fungi | of course, the projects.yaml change we just merged should cause it to update shortly | 17:31 |
| clarkb | yup | 17:31 |
| fungi | yeah, what i was just thinking too | 17:31 |
| fungi | and when salt becomes ubiquitous for puppet synchronicity this should also become a less obvious delay updating | 17:32 |
| fungi | i think a lot of the rename process will, i fact | 17:32 |
| fungi | er, in fact | 17:32 |
| fungi | because we do some things already just to work around/shorten that delay | 17:33 |
| fungi | which should hopefully be obsolete soonish | 17:33 |
| fungi | hmm, we are also missing statusbot | 17:34 |
| fungi | fixing | 17:34 |
| *** openstackstatus has joined #openstack-infra | 17:35 | |
| fungi | there he is | 17:35 |
| clarkb | 2/4 mirror hosts have new cgitrepo config so yes puppet seems to be doing that for us | 17:35 |
| fungi | awesome | 17:36 |
| clarkb | fungi: new repos are not on github yet | 17:37 |
| *** openstackgerrit has quit IRC | 17:37 | |
| clarkb | fungi: do you know if the manage_projects script has run at least once? | 17:37 |
| *** openstackgerrit has joined #openstack-infra | 17:37 | |
| fungi | checking | 17:37 |
| fungi | looks like it's currently running | 17:38 |
| fungi | i probably should have manually triggered a puppet run on review.o.o before starting the agent, just to eliminate the lag there | 17:38 |
| clarkb | fungi: meh | 17:38 |
| clarkb | fungi: cgit is happy now | 17:38 |
| fungi | looks like manage-projects has only been running for 2-3 minutes so far, so may take a little longer | 17:39 |
| clarkb | ok | 17:39 |
| clarkb | more patience :) | 17:39 |
| fungi | jenkinses seem to be working sanely | 17:40 |
| fungi | should i submit a .gitreview change to those two projects, or let them take care of their own? | 17:40 |
| clarkb | fungi: ++ to submitting a couple changes, it will exercise their tests which is a good thing | 17:41 |
| fungi | doing so now | 17:41 |
| *** SergeyLukjanov has joined #openstack-infra | 17:44 | |
| clarkb | fungi: github repos were created, now waiting for replication which your chagnes hsould trigger | 17:44 |
| fungi | hrm... https://review.openstack.org/#/q/status:open+topic:incubation-move,n,z are not happy in jenkinsland | 17:47 |
| fungi | well, one of them anyway | 17:47 |
| fungi | issues installing d2to1 looks like | 17:48 |
| clarkb | hahahahahaha I think it may be due to being forced to use the mirror | 17:48 |
| fungi | poor suckers | 17:49 |
| clarkb | so uh ya I think we can let marconi folks sort that out | 17:49 |
| clarkb | fungi: still no replication to github I am going to manually trigger replication for the two repos | 17:49 |
| openstackgerrit | A change was merged to openstack/requirements: Update sphinxcontrib-pecanwsme to 0.3 https://review.openstack.org/46617 | 17:49 |
| fungi | k | 17:49 |
| clarkb | woot both are in github properly | 17:51 |
| clarkb | I think thats about it | 17:51 |
| clarkb | fungi: anything else you can think of that needs checking? | 17:52 |
| fungi | clarkb: is my comment on https://review.openstack.org/46628 on track, do you think? | 17:52 |
| clarkb | fungi: maybe leave a comment on your marconi change indicating why we think the tests are failing? | 17:52 |
| fungi | heh | 17:52 |
| clarkb | yup that comment looks good | 17:52 |
| fungi | other than that, nothing springs to mind. stuff looks to be working | 17:52 |
| Alex_Gaynor | while everyone is around on a saturday someone should clearly review https://review.openstack.org/#/c/46607/ :P | 17:53 |
| fungi | clearly | 17:53 |
| fungi | nice. javascript uses !== instead of != | 17:54 |
| fungi | oh, i see. includes the type comparison that way | 17:55 |
| fungi | so sayeth http://www.w3schools.com/js/js_comparisons.asp | 17:55 |
| fungi | makes sense when comparing against the null type | 17:56 |
| flaper87 | clarkb: fungi thanks a lot guys! You rock! | 17:57 |
| *** afazekas has quit IRC | 17:57 | |
| fungi | flaper87: note my comment about requirements on https://review.openstack.org/46628 | 17:57 |
| flaper87 | fungi: saw it, will work on that in a bit! | 17:57 |
| flaper87 | thanks for pointing it out! | 17:58 |
| fungi | and let us know if you still need any assistance getting tests to run after trying that | 17:58 |
| UtahDave | fungi: how are things looking with the Salt Jenkins slave? | 18:02 |
| fungi | UtahDave: seems to be working as intended, though i'm making sure salt communication stays healthy before pressing forward adding the job to trigger it automatically | 18:03 |
| fungi | UtahDave: right now i have a handful of minions which submitted keys and which i accepted but which now don't respond to a test.ping | 18:03 |
| UtahDave | fungi: sounds reasonable. Have you tried triggering the puppet runs from the cli on the salt jenkins slave? | 18:03 |
| fungi | UtahDave: yep, that part is working. the reachable minions all fire up their puppet update great | 18:04 |
| UtahDave | nice | 18:04 |
| *** SergeyLukjanov has quit IRC | 18:04 | |
| fungi | anyway, looks like a majority of our centos 6.4 minions have stopped responding to the salt master | 18:04 |
| fungi | er, around half of them at least | 18:05 |
| UtahDave | Salt requires that ports 4505 and 4506 on the master are open and available to the minions. Could there be a firewall issue? | 18:05 |
| * fungi checks iptables | 18:05 | |
| fungi | looks okay... the input chain includes | 18:06 |
| fungi | ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:4505 | 18:06 |
| fungi | ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:4506 | 18:06 |
| fungi | on the master | 18:06 |
| UtahDave | ok, that looks good | 18:06 |
| UtahDave | I think cent6 might have an old version of zmq by default | 18:06 |
| fungi | also, it (so far) only seems to be the centos minions which are dropping out | 18:06 |
| fungi | yeah, that could make some sense then | 18:07 |
| UtahDave | what's the output of salt-call --versions-report on one of the cent vms? | 18:07 |
| fungi | UtahDave: http://paste.openstack.org/show/47085/ | 18:08 |
| UtahDave | Hm. zmq looks good there. | 18:09 |
| UtahDave | Let me go find out if we have any open Cent issues on that. | 18:09 |
| UtahDave | how many of them aren't responding? | 18:09 |
| fungi | 7 out of 16 are unresponsive currently, after less than a day | 18:10 |
| *** SergeyLukjanov has joined #openstack-infra | 18:11 | |
| fungi | UtahDave: also, salt-minion processes are piling up on all the centos minions (whether they're responding or not). some have as many as 20 running concurrently based on my spot checks | 18:13 |
| fungi | note that these are all running the latest security updates for centos 6.4 | 18:14 |
| fungi | so shouldn't be out of date from a centos perspective anyway | 18:14 |
| UtahDave | Hm. I seem to remember a recent email thread about this. I think it was a packaging issue with setting up the service. | 18:14 |
| UtahDave | Are the centos minions with salt-minion processes piling up the same minions that aren't responding to test.pings? | 18:17 |
| fungi | it's piling up on all of the centos minions, whether they're responding or not | 18:18 |
| clarkb | fungi what about the other slaves? | 18:18 |
| fungi | clarkb: the precise slaves i've checked seem to all respond and have only one running minion process | 18:19 |
| *** beagles has quit IRC | 18:21 | |
| *** Ng has quit IRC | 18:22 | |
| *** SergeyLukjanov has quit IRC | 18:23 | |
| *** lnxnut has joined #openstack-infra | 18:24 | |
| fungi | checked them all. tons of processes on the centos slaves, only one each on the precise slaves. so definitely centos-only at this point | 18:26 |
| UtahDave | fungi: looks like the salt mine might be causing this issue. I'm tracking down exactly how to turn it off. | 18:27 |
| fungi | number of salt-minion processes on the centos machines varies widely | 18:27 |
| fungi | UtahDave: great! thanks again | 18:27 |
| fungi | i've also got a problem with salt-minion not starting and puppet agent failing to completion on precise-dev1 but that seems to just be something unrelated. the vm was also unresponsive for no idea how long (weeks?) until i force-restarted it through the rackspace webui | 18:29 |
| fungi | we might want to consider rebuilding that vm at some point | 18:29 |
| UtahDave | did the minion respond correctly after force-restarting the vm? | 18:30 |
| *** nayward has joined #openstack-infra | 18:31 | |
| fungi | UtahDave: on precise-dev1? no, but because the minion daemon dies as soon as i try to start it. haven't bothered to troubleshoot because it's a fairly unimportant system | 18:32 |
| fungi | it's probably missing configuration and other things because of puppet not being able to apply them right now. i don't think the issue there has anything to do with salt | 18:32 |
| UtahDave | ok, good. | 18:34 |
| *** nayward has quit IRC | 18:34 | |
| UtahDave | fungi: ok, on the salt-master can you try setting this option: minion_data_cache: False | 18:38 |
| UtahDave | then restart the salt-master service | 18:38 |
| fungi | trying | 18:38 |
| fungi | that's in place and restarted | 18:40 |
| fungi | and now most of the centos minions are not responding to test.ping | 18:40 |
| UtahDave | can you manually restart the salt-minion service on one of them? | 18:40 |
| fungi | sure thing | 18:40 |
| fungi | probably will need to log in and kill off straggler salt-minion processes. will take a few minutes | 18:41 |
| UtahDave | ok | 18:42 |
| fungi | after restarting salt-minion cleanly on all the centos 6.4 machines, they're responding to test.ping just fine (but that had also cleared up the issue when we saw it in the past) | 18:52 |
| fungi | i'll have a look at them again tomorrow and see if the issue has resurfaced | 18:52 |
| UtahDave | fungi: is your salt master running on an ubuntu machine or a centos 6.4 machine? | 18:52 |
| fungi | since it seems to take a while to get into that state | 18:52 |
| fungi | UtahDave: ubuntu 12.04 lts (precise pangolin) | 18:53 |
| UtahDave | OK, good. I'm going to set up a matching environment right now with 15 or 20 cent vms and let them run overnight to see if I can cause this to happen, too | 18:53 |
| UtahDave | 64bit? | 18:55 |
| fungi | yep, 64-bit on both ubuntu and centos | 18:56 |
| fungi | latest security updates, et cetera | 18:56 |
| fungi | debs and rpms for salt-common, salt-master, salt-minion from saltstack package repositories | 18:57 |
| fungi | latest versions | 18:57 |
| * fungi needs to afk for a walk... bbl | 18:58 | |
| UtahDave | ok, great. | 18:58 |
| UtahDave | same here. talk to you later. | 18:58 |
| fungi | thanks for all your help on this | 18:58 |
| UtahDave | You're | 19:00 |
| UtahDave | welcome. | 19:00 |
| *** lnxnut has quit IRC | 19:04 | |
| *** rnirmal_ has joined #openstack-infra | 19:17 | |
| *** rnirmal has quit IRC | 19:19 | |
| *** rnirmal_ is now known as rnirmal | 19:19 | |
| *** dcramer__ has joined #openstack-infra | 19:22 | |
| *** senk has joined #openstack-infra | 19:35 | |
| *** UtahDave has quit IRC | 19:37 | |
| *** yolanda_ has joined #openstack-infra | 19:38 | |
| clarkb | https://www.taoeffect.com/blog/2013/09/the-apache-contributors-license-agreement-is-very-dangerous/ | 19:43 |
| clarkb | mordred: ^ | 19:43 |
| *** thomasbiege1 has quit IRC | 19:46 | |
| Alex_Gaynor | I am not a lawyer, but the text of the Apache CLA seems pretty straightforward to me, "such license applies only to those patent claims licensable by You that are necessarily infringed by Your Contribution(s) alone or by combination of Your Contribution(s) with the Work to which such Contribution(s) was submitted" | 19:47 |
| lifeless | Alex_Gaynor: sure, their point is that you could do an NSA and read it how you want to. | 19:48 |
| clarkb | Alex_Gaynor: yeah, I haven't taken time to read the linked email thread but I am not sure how that conclusion was reached | 19:48 |
| lifeless | you could say 'you wrote this if statement in the same program' | 19:48 |
| lifeless | and so your code is in combination with the Work | 19:49 |
| clarkb | ah | 19:49 |
| lifeless | engineers make bad backseat lawyers | 19:49 |
| lifeless | because we look for code | 19:50 |
| lifeless | not agreements | 19:50 |
| *** rnirmal has quit IRC | 19:57 | |
| *** boris-42 has quit IRC | 19:57 | |
| *** nhm has quit IRC | 19:59 | |
| *** Ryan_Lane has joined #openstack-infra | 20:02 | |
| *** yolanda_ has quit IRC | 20:06 | |
| *** dcramer__ has quit IRC | 20:38 | |
| *** odyssey4me has quit IRC | 21:27 | |
| *** odyssey4me2 has joined #openstack-infra | 21:27 | |
| *** odyssey4me2 has quit IRC | 21:31 | |
| *** dachary has quit IRC | 21:34 | |
| *** dachary has joined #openstack-infra | 21:35 | |
| *** lifeless has quit IRC | 21:41 | |
| *** senk has quit IRC | 21:56 | |
| *** dripton has quit IRC | 21:57 | |
| *** dcramer__ has joined #openstack-infra | 21:57 | |
| fungi | law is not science, it is art | 22:00 |
| fungi | it doesn't matter how you might choose to interpret that text. what matters is how a judge decides to interpret that text | 22:00 |
| *** dripton has joined #openstack-infra | 22:00 | |
| *** dcramer__ has quit IRC | 22:02 | |
| *** senk has joined #openstack-infra | 22:02 | |
| fungi | though kudos to the author for voicing his concerns and pressing for clarification. hopefully the revisions went through a competent lawyer at that project's end | 22:08 |
| *** pmoosh has joined #openstack-infra | 22:10 | |
| *** sungju has joined #openstack-infra | 22:12 | |
| *** dcramer__ has joined #openstack-infra | 22:15 | |
| *** sungju has quit IRC | 22:20 | |
| *** lifeless has joined #openstack-infra | 22:36 | |
| *** dcramer__ has quit IRC | 23:02 | |
| *** flaper87 is now known as flaper87|afk | 23:05 | |
| *** pmoosh has quit IRC | 23:20 | |
| *** fbo is now known as fbo_away | 23:21 | |
| *** markmcclain has quit IRC | 23:56 | |
| *** pmoosh has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!