| rstarmer | has anyone been noticing very high OVS resource consumption on all-in-one kolla deployments with 3.0.0rc1? | 00:02 |
|---|---|---|
| rstarmer | I just built a system today (and yesterday, and the day before), from scratch, and on a 4c 8GB machine, I see a system load of 4-5 and it appears to all be OVS related. rebuilding with linxubridge, and I have a load of .5 | 00:03 |
| rstarmer | note, there's not a single network created in these cases!?!? | 00:03 |
| kfox1111 | fun stuff: http://vim-adventures.com/ | 00:04 |
| *** haplo37 has quit IRC | 00:16 | |
| *** haplo37 has joined #openstack-kolla | 00:16 | |
| *** dwalsh has joined #openstack-kolla | 00:20 | |
| *** stianaurdal has joined #openstack-kolla | 00:20 | |
| *** stianaurdal has left #openstack-kolla | 00:21 | |
| *** tonanhngo has joined #openstack-kolla | 00:23 | |
| *** jax3242 has joined #openstack-kolla | 00:23 | |
| *** eaguilar has quit IRC | 00:25 | |
| *** tonanhngo has quit IRC | 00:25 | |
| *** rstarmer has quit IRC | 00:27 | |
| *** eaguilar has joined #openstack-kolla | 00:29 | |
| *** dwalsh has quit IRC | 00:33 | |
| *** huikang has joined #openstack-kolla | 00:36 | |
| Jeffrey4l_ | sbezverk, around? | 00:41 |
| Jeffrey4l_ | sbezverk, could u explain this? https://review.openstack.org/#/c/379960/8/ansible/roles/rabbitmq/templates/rabbitmq-env.conf.j2 | 00:41 |
| v1k0d3n | kubectl config set-context $CONTEXT --namespace=<insert-namespace-name-here> | 00:46 |
| kfox1111 | ah. cool. thanks. | 00:46 |
| v1k0d3n | sure | 00:47 |
| *** huikang has quit IRC | 00:53 | |
| *** huikang has joined #openstack-kolla | 00:54 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Download the rabbitmq_clusterer plugins to the correct path https://review.openstack.org/379960 | 00:55 |
| *** huikang has quit IRC | 00:58 | |
| *** eaguilar has quit IRC | 01:03 | |
| *** daneyon has joined #openstack-kolla | 01:04 | |
| *** daneyon has quit IRC | 01:09 | |
| *** eaguilar has joined #openstack-kolla | 01:20 | |
| *** haplo37_ has quit IRC | 01:23 | |
| *** tonanhngo has joined #openstack-kolla | 01:23 | |
| *** tonanhngo has quit IRC | 01:25 | |
| *** haplo37_ has joined #openstack-kolla | 01:25 | |
| *** tonanhngo has joined #openstack-kolla | 01:37 | |
| *** Pavo has quit IRC | 01:48 | |
| *** Pavo has joined #openstack-kolla | 01:49 | |
| *** salv-orlando has joined #openstack-kolla | 02:04 | |
| *** salv-orlando has quit IRC | 02:08 | |
| v1k0d3n | hey sdake i know you're still working things out and been a long nice. noticed that centos/binary for kolla-toolbox is failing on master. going to check into why. i think that's the that's supposed to be working (others aren't building many containers quite yet or still have errors in newton). | 02:09 |
| openstackgerrit | Ken Johnston proposed openstack/kolla: Readability Improvements to Advanced Config Doc https://review.openstack.org/347102 | 02:14 |
| openstackgerrit | Ken Johnston proposed openstack/kolla: Readability Improvements to Advanced Config Doc https://review.openstack.org/347102 | 02:16 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: DO NOT MERGE: TEST MITAKA BRANCH https://review.openstack.org/299182 | 02:20 |
| *** huikang has joined #openstack-kolla | 02:26 | |
| sdake | v1k0d3n roger - i am headed to dinner - will buidl when i get back | 02:27 |
| sdake | which wont be for a few hours | 02:27 |
| sdake | if you get a patch in the queue let someone know if they are around (such as Jeffrey4l_ ) | 02:28 |
| sdake | Jeffrey4l_ gate ffailing on rabbitmq fix | 02:29 |
| sdake | must eat bbl :) | 02:29 |
| *** sdake has quit IRC | 02:29 | |
| Jeffrey4l_ | sdake, it is kolla-toolbox crashed the gate. | 02:29 |
| Jeffrey4l_ | seem like this issue https://github.com/pyca/cryptography/issues/3187 | 02:29 |
| *** huikang has quit IRC | 02:31 | |
| *** duonghq has joined #openstack-kolla | 02:45 | |
| duonghq | morning folks (quite late) | 02:47 |
| *** yuanying has quit IRC | 02:52 | |
| *** yuanying has joined #openstack-kolla | 02:53 | |
| *** g3ek has quit IRC | 02:57 | |
| *** haplo37 has quit IRC | 02:58 | |
| *** haplo37 has joined #openstack-kolla | 02:59 | |
| *** g3ek has joined #openstack-kolla | 03:03 | |
| *** salv-orlando has joined #openstack-kolla | 03:04 | |
| *** salv-orlando has quit IRC | 03:09 | |
| duonghq | zhubingbing: ping | 03:11 |
| *** sdake has joined #openstack-kolla | 03:21 | |
| sdake | v1k0d3n could you define failing for me please | 03:22 |
| sdake | v1k0d3n containers not building on centos is a normal thing - centos rdo has not packaged many services yet | 03:22 |
| *** phuongnh has joined #openstack-kolla | 03:27 | |
| *** sdake_ has joined #openstack-kolla | 03:38 | |
| *** sdake has quit IRC | 03:40 | |
| *** coolsvap has joined #openstack-kolla | 03:41 | |
| *** yuanying has quit IRC | 03:44 | |
| sdake_ | coolsvap you about | 03:47 |
| coolsvap | sdake_: yes | 03:47 |
| sdake_ | how was your weekend | 03:47 |
| *** Pavo has quit IRC | 03:48 | |
| coolsvap | nice | 03:48 |
| coolsvap | how about yours? | 03:48 |
| sdake_ | chaotic as usual | 03:48 |
| sdake_ | but caught up a bit on my house backlog | 03:49 |
| coolsvap | i am currently enjoying my (temporary) bachelor life with wife out of town | 03:49 |
| *** huikang has joined #openstack-kolla | 03:49 | |
| duonghq | sdake_: you keep backlog for everything, don't you? | 03:49 |
| sdake_ | duonghq yup | 03:50 |
| sdake_ | duonghq and prioritize as needed | 03:50 |
| duonghq | sdake_: in your mind or some kind of external storage? | 03:50 |
| *** yuanying has joined #openstack-kolla | 03:51 | |
| *** eaguilar has quit IRC | 03:51 | |
| sdake_ | duonghq external archive | 03:52 |
| sdake_ | https://en.wikipedia.org/wiki/Getting_Things_Done | 03:52 |
| sdake_ | thats what I use | 03:52 |
| duonghq | oh, thank you sdake_ | 03:53 |
| *** Pavo has joined #openstack-kolla | 03:53 | |
| sdake_ | duonghq if ou keep your backlogs in your internal memory - things slip past and thigns dont get done and stress increases | 03:54 |
| sdake_ | GTD = all about living a stress free life | 03:54 |
| sdake_ | interestingly i find with gtd if my backlog (archive) descreases - stress levels increase ;) | 03:55 |
| duonghq | sdake_: tyvm I'm looking for a efficient way to management things | 03:55 |
| sdake_ | well it works for me | 03:55 |
| sdake_ | obviously things throw it off kilter | 03:55 |
| sdake_ | in which case it goes into my backlog | 03:56 |
| sdake_ | somethign to worry about later rather hten nwo :) | 03:56 |
| sdake_ | i combine that with capacity manaement as well | 03:56 |
| sdake_ | my own take on GTD | 03:56 |
| sdake_ | since i manage capacity for alot of other activities | 03:56 |
| sdake_ | composition of two good techniques = a super powerful technique | 03:57 |
| *** huikang has quit IRC | 03:57 | |
| sdake_ | it = being the things that are no longer priority #1 in my profesional or personal life | 03:57 |
| sdake_ | my personal life backlog grows long | 03:57 |
| *** huikang has joined #openstack-kolla | 03:58 | |
| sdake_ | grass needs mowing there | 03:58 |
| duonghq | interesting method | 03:58 |
| sdake_ | i didn't invent gtd | 03:58 |
| sdake_ | i just abuse it | 03:58 |
| duonghq | sure | 03:58 |
| sdake_ | along with compartamentalization, i've got stress pretty well under control :) | 03:58 |
| sdake_ | my work backlog seems to take priority to my personal life backlog | 03:59 |
| sdake_ | i can't say if this is a good hing or a bad thing | 03:59 |
| sdake_ | but if my professional life suffers, my personal life suffers worse | 04:00 |
| *** huikang has quit IRC | 04:02 | |
| *** jmccarthy has quit IRC | 04:02 | |
| coolsvap | sdake_ i think it happens to most people | 04:03 |
| *** jmccarthy has joined #openstack-kolla | 04:04 | |
| sdake_ | yup i rationalize it as above - professional life craters = personal life super craters | 04:04 |
| *** salv-orlando has joined #openstack-kolla | 04:05 | |
| *** haplo37_ has quit IRC | 04:05 | |
| *** haplo37_ has joined #openstack-kolla | 04:08 | |
| *** salv-orlando has quit IRC | 04:10 | |
| sdake_ | duonghq another technique (that doesn't work for me) is called mindfulness | 04:14 |
| sdake_ | https://en.wikipedia.org/wiki/Mindfulness | 04:14 |
| sdake_ | the meditation part i struggle with | 04:15 |
| sdake_ | for me, mindfulness increases stress | 04:15 |
| sdake_ | because is brings to the forefront my backlog in my archives | 04:15 |
| sdake_ | they all jam in my brain at once | 04:16 |
| sdake_ | that damages how compartmentalization works | 04:16 |
| sdake_ | and GTD works | 04:16 |
| sdake_ | and all the various other techniques i rely on | 04:16 |
| duonghq | sure, it's impossible to get something one size fits all | 04:17 |
| sdake_ | lots of people thing mindfulness is the silver bullet | 04:20 |
| sdake_ | maybe for millenials | 04:20 |
| sdake_ | who don't already have too much on their plate to deal with as is | 04:20 |
| duonghq | ya | 04:21 |
| *** bdaca has joined #openstack-kolla | 04:39 | |
| bdaca | hello | 04:39 |
| *** daneyon has joined #openstack-kolla | 04:41 | |
| sdake_ | duonghq i think the turning point for me wrt stress managaement (which is just another form of time management imo) was learning to trust and rely on others | 04:44 |
| duonghq | It's always the hardest part for me :) | 04:44 |
| sdake_ | i take it for granted now | 04:45 |
| sdake_ | in the past i always thought people were out to get me | 04:45 |
| sdake_ | now i don't care if people are out to get me | 04:45 |
| *** daneyon has quit IRC | 04:45 | |
| sdake_ | the people that arn't help my work | 04:45 |
| sdake_ | the people that are, attempt to discredit it | 04:45 |
| sdake_ | easy to identify who to trust who not to trust | 04:46 |
| sdake_ | anyway hope that helped ;) | 04:46 |
| duonghq | sure, that help me very much | 04:47 |
| sdake_ | sup bdaca | 04:48 |
| bdaca | Everything is fine in here :) How are you today sdake_? Still having sleep shortage? I've heard the rumours that you never sleep ;) | 04:55 |
| sdake_ | rumors are just that | 04:55 |
| sdake_ | i get atleast 6 hrs a night | 04:55 |
| sdake_ | but ya during osic testing - got about 1 hr a sleep a day for about a week | 04:56 |
| sdake_ | i was nearly crazy towards the end of that | 04:56 |
| sdake_ | had to be done | 04:56 |
| sdake_ | i did sleep 24 hrs saturday | 04:57 |
| sdake_ | so I guess that was catchup time :) | 04:57 |
| sdake_ | went to bed friday woke up sunday morning | 04:57 |
| sdake_ | first time thats hapepned in awhile :) | 04:57 |
| sdake_ | osic more like 2 days of hibernation | 04:57 |
| sdake_ | or 3 or 4 | 04:57 |
| sdake_ | say folks tc voting is open | 04:58 |
| sdake_ | even if yo udont vote for me, please vote | 04:58 |
| sdake_ | its emberassing that only 600 people out of 2800 people voted in the last tc election | 04:58 |
| sdake_ | that means each vote counts 6x as much | 04:59 |
| *** yuanying has quit IRC | 04:59 | |
| *** yuanying_ has joined #openstack-kolla | 04:59 | |
| sdake_ | only by voting can you select the candidates to shape the future of openstack | 04:59 |
| sdake_ | along with our army of community :) | 05:00 |
| * coolsvap already voted | 05:02 | |
| *** skramaja has joined #openstack-kolla | 05:03 | |
| *** salv-orlando has joined #openstack-kolla | 05:04 | |
| *** bjolo has joined #openstack-kolla | 05:14 | |
| bjolo | morning | 05:15 |
| sdake_ | sup bjolo | 05:17 |
| sdake_ | bjolo warning - mariadb is bust | 05:17 |
| sdake_ | bjolo we are working on it | 05:17 |
| bjolo | ok tnx | 05:17 |
| sdake_ | by we i mean jeffrey4l mostly | 05:17 |
| sdake_ | Jeffrey4l_ that is ^ | 05:17 |
| bjolo | anything i can do to help/test? | 05:17 |
| sdake_ | bjolo nope patches going thru iterations - we know what needs to be fixed and how ti fix it | 05:18 |
| bjolo | ok cool | 05:18 |
| * duonghq already vote, too | 05:18 | |
| sdake_ | its only a z stream version change | 05:18 |
| sdake_ | however it busted our deployment of rabbitmq | 05:18 |
| sdake_ | because we use a plugin called clusterer | 05:18 |
| sdake_ | which isn't packaged with the rest of rabbitmq | 05:19 |
| sdake_ | we have hardcodes on the version numbers | 05:19 |
| sdake_ | easy to fix | 05:19 |
| sdake_ | annoying :) | 05:19 |
| sdake_ | its not like we can dynamically determine it | 05:19 |
| sdake_ | i guess maybe we could | 05:19 |
| *** HyperJohnGraham has quit IRC | 05:19 | |
| sdake_ | possibly in ocata ;) | 05:19 |
| sdake_ | kfox1111 if your still awake - that api thing was the root cause of the ceph not working | 05:20 |
| sdake_ | along with disks not being labeled | 05:20 |
| sdake_ | sbezverk ^^ | 05:20 |
| sdake_ | api thing/api variable | 05:21 |
| sdake_ | if folks will be looking for me tomorrow during us working hours, i'll be out of the office working on my personal backlog | 05:21 |
| sdake_ | hopefully the end of that for awhile | 05:21 |
| *** msimonin has joined #openstack-kolla | 05:36 | |
| openstackgerrit | bjorn lofdahl proposed openstack/kolla: fixed kestone fernet prechecks for multinode deployments https://review.openstack.org/380014 | 05:46 |
| *** Pavo has quit IRC | 05:48 | |
| sdake_ | coolsvap | 05:50 |
| sdake_ | need love from you on the requirements front | 05:50 |
| sdake_ | i know it is totally to late to probably do anything about htis | 05:51 |
| sdake_ | but can you read this | 05:51 |
| sdake_ | https://github.com/pyca/cryptography/issues/3187#issuecomment-251024858https://github.com/pyca/cryptography/issues/3187%23issuecomment-251024858 | 05:51 |
| sdake_ | Jeffrey4l_ workaround in the bug tracker | 05:51 |
| Jeffrey4l_ | yes. --no-binary :all: works. | 05:51 |
| *** Pavo has joined #openstack-kolla | 05:53 | |
| sdake_ | groan on this https://github.com/eliben/pycparser/issues/148 | 05:53 |
| sdake_ | Jeffrey4l_ any problem with that workaround? | 05:53 |
| Jeffrey4l_ | seems no. the kolla-toolbox build successfully. | 05:54 |
| sdake_ | i saw gratuitious warnings all around it | 05:54 |
| sdake_ | in the bug report | 05:54 |
| Jeffrey4l_ | no idea why he say that. | 05:54 |
| sdake_ | what does --no-binary :all: do | 05:54 |
| coolsvap | there is already a mailing list thread on -dev related to it | 05:55 |
| sdake_ | not use wheels? | 05:55 |
| Jeffrey4l_ | correct | 05:55 |
| sdake_ | coolsvap cool - can someone weigh in and say "NEEDS FIXING ASAP" | 05:55 |
| Jeffrey4l_ | not use wheels for all packages. | 05:55 |
| sdake_ | bjolo don't rebuild toolbox either - its bustola | 05:55 |
| bjolo | tnx | 05:57 |
| bjolo | not doing any builds today | 05:57 |
| bjolo | i have builds from friday that i continue to use | 05:57 |
| bjolo | does anyone have any good URL for how to setup keystone v3, domains, horizon and the cloud_admin policy.v3sample.json? | 05:58 |
| bjolo | (in a sense i think this should be the default deploy for kolla, but... ) | 05:59 |
| *** nihilifer has joined #openstack-kolla | 05:59 | |
| *** salv-orlando has quit IRC | 06:00 | |
| sdake_ | Jeffrey4l_ - lets roll wit hthe workaround | 06:01 |
| sdake_ | Jeffrey4l_ if the packagers fix things before 12th, we can revert | 06:01 |
| Jeffrey4l_ | ok | 06:01 |
| sdake_ | sort of shows how fragile pypi is as a dependency management system unfortunately | 06:02 |
| sdake_ | this problem affects source and binary i suspect - so need to fix ahead of rabbitmq I think | 06:02 |
| openstackgerrit | bjorn lofdahl proposed openstack/kolla: optimze prechecks to only run local_actions once https://review.openstack.org/372678 | 06:02 |
| Jeffrey4l_ | yes. in requirements.txt, we normally use xxx>2.1.1 which is unstable. | 06:02 |
| Jeffrey4l_ | openstack/requirements/upper-constrains.txt is a good solution. | 06:03 |
| sdake_ | well if you read the bug log the upper constraint would be 2.1.4 | 06:03 |
| sdake_ | or 2.14 | 06:03 |
| sdake_ | or watever it was | 06:03 |
| sdake_ | but the wheel uploaded was also 2.14 | 06:03 |
| sdake_ | so upper-constraints not sufficient for htis problem unless we upper constraint it at 2.1.3 | 06:04 |
| sdake_ | in this case upper constraints only works after the fact | 06:04 |
| Jeffrey4l_ | you are correct. | 06:04 |
| sdake_ | wht is needed is a check to make sure dates of wheels match pypi apckages | 06:05 |
| sdake_ | and if that isn't the case, then infra can roll back to an older version in their cache and sound the alarm | 06:05 |
| sdake_ | Jeffrey4l_ any problems with ansible 2.1.1.0 you are aware of | 06:06 |
| sdake_ | sbezverk suggested there may be | 06:06 |
| coolsvap | sdake_: Jeffrey4l_ lets go with the workaround, I will follow up with requirements | 06:06 |
| sdake_ | coolsvap wfm | 06:06 |
| sdake_ | we can always revert hte workaround | 06:07 |
| *** egonzalez90 has joined #openstack-kolla | 06:07 | |
| sdake_ | i think the worst case for the workaround is slower performance | 06:07 |
| sdake_ | meh - who cares :) | 06:07 |
| sdake_ | our performance already smokes | 06:07 |
| sdake_ | sup egonzalez90 | 06:07 |
| sdake_ | lots of fires today :) | 06:08 |
| sdake_ | Jeffrey4l_ do you have patches in queue for the workaround atm? | 06:08 |
| sdake_ | if not, can you get em there so we can get em acked | 06:08 |
| Jeffrey4l_ | 1 sec. i am pushing it. | 06:08 |
| sdake_ | if not, i'll start working on it | 06:08 |
| sdake_ | cool thanks :) | 06:09 |
| sdake_ | Jeffrey4l_ real life problem we ran in to today re fact gathering | 06:10 |
| sdake_ | we deployed ceph with -t ceph | 06:10 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix pycparser wheel package issue https://review.openstack.org/380929 | 06:10 |
| sdake_ | and inventory file was empty of nodes for control/compute/monitoring | 06:10 |
| Jeffrey4l_ | sdake_, coolsvap ^^ here is the fix. | 06:10 |
| sdake_ | it only contained a host for storage | 06:10 |
| sdake_ | Jeffrey4l_ and kolla imploded on this scenario | 06:11 |
| Jeffrey4l_ | sdake_, i do not think so. | 06:11 |
| sdake_ | i'd like to see kolla's ceph implementation used in kolla-kubernetes | 06:11 |
| sdake_ | Jeffrey4l_ you dont htink it imploded? | 06:11 |
| sdake_ | I saw it with my own eyes :) | 06:11 |
| *** msimonin has quit IRC | 06:12 | |
| sdake_ | to get from here to there, we need to be able to deploy -t ceph without implosion without having to sspecify an inventory file that contains a bunch of other unrelated stuff | 06:12 |
| sdake_ | thin kof it as an invetory-lite file | 06:12 |
| sdake_ | ceph-only-inventory | 06:12 |
| sdake_ | orsomething like that | 06:12 |
| sdake_ | not something to tackle now or before rc2 | 06:12 |
| sdake_ | but something to think about | 06:12 |
| Jeffrey4l_ | if it works. it is really really kolla. | 06:12 |
| Jeffrey4l_ | kolla/cool | 06:13 |
| sdake_ | the ceph part works | 06:13 |
| sdake_ | with -t it works | 06:13 |
| sdake_ | the ceph integration will work with kubernetes | 06:13 |
| Jeffrey4l_ | so there some issue in current implementation? | 06:13 |
| sdake_ | yes, current implementation is gathering facts (I suspect from our friends in the controller) | 06:13 |
| sdake_ | i could be wrong on specifics | 06:14 |
| Jeffrey4l_ | if there are no any nodes in control groups, it won't gather facts from it. | 06:14 |
| sdake_ | so must put the storage nodes in the controller - or no facts are gathered | 06:14 |
| Jeffrey4l_ | sdake_, is there any bug or log for this? | 06:14 |
| sdake_ | Jeffrey4l_ nah - we found it t-3 hours ago | 06:14 |
| sdake_ | and everyone was working really hard to get a workign system for john | 06:15 |
| sdake_ | itis very easy to replicate | 06:15 |
| sdake_ | i'll file a bug :) | 06:15 |
| Jeffrey4l_ | so the issue is: when using -t ceph, and there are node in control, it will gather facts from it, right? | 06:16 |
| sdake_ | no issue is if no nodes are in control/networking/monitoring/compute, but a node in storage, ansible implodes with some odd error | 06:16 |
| sdake_ | i saw a bunch of errors today so I don't recall exactly what the error was | 06:16 |
| Jeffrey4l_ | could u get the raw error logs? | 06:17 |
| sdake_ | Jeffrey4l_ i will replicate it | 06:18 |
| sdake_ | let me switch vpns | 06:18 |
| Jeffrey4l_ | cool | 06:18 |
| *** sdake has joined #openstack-kolla | 06:22 | |
| *** sdake_ has quit IRC | 06:23 | |
| sdake | gotta say kolla-ansible destroy best feature ever | 06:23 |
| egonzalez90 | sup sdake | 06:26 |
| sdake | egonzalez90 same old :) | 06:27 |
| sdake | more bugs then time :) | 06:27 |
| *** tonanhngo has quit IRC | 06:27 | |
| *** yuanying_ has quit IRC | 06:33 | |
| sdake | speaking of bugs - if folks can start working critical-> down that would be helpful - about 8 days left to tag :) | 06:35 |
| sdake | also if there are bugs that are in progress that can easily be fixed, might as well get those done | 06:36 |
| *** haplo37_ has quit IRC | 06:41 | |
| *** haplo37_ has joined #openstack-kolla | 06:43 | |
| sdake | Jeffrey4l_ more details: https://bugs.launchpad.net/kolla/+bug/1629775 | 06:44 |
| openstack | Launchpad bug 1629775 in kolla "ceph won't deploy alone" [High,Confirmed] | 06:44 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix pycparser wheel package issue https://review.openstack.org/380929 | 06:45 |
| Jeffrey4l_ | roger | 06:45 |
| sdake | Jeffrey4l_ for this can you rebase your rabbitmq on top of it https://review.openstack.org/#/c/380929/2 | 06:47 |
| sdake | (not in it, but as a second patch) | 06:47 |
| sdake | since it takes both of these patches to make the gate operate | 06:47 |
| Jeffrey4l_ | sdake, ceph-mon will be installed on control in your inventory. | 06:48 |
| Jeffrey4l_ | sdake, yes. | 06:48 |
| Jeffrey4l_ | will rebase that. | 06:48 |
| sdake | Jeffrey4l_ so user error | 06:48 |
| Jeffrey4l_ | sdake, i think so. but no sure if install ceph-mon on storage will be successful | 06:49 |
| Jeffrey4l_ | means whether it will raise any other error | 06:49 |
| sdake | solutio nis to change https://github.com/openstack/kolla/blob/master/ansible/inventory/multinode#L111 | 06:49 |
| sdake | ? | 06:49 |
| *** salv-orlando has joined #openstack-kolla | 06:50 | |
| Jeffrey4l_ | sdake, yep. change from control to storage( if you have 3 node ) | 06:50 |
| *** b_bezak has joined #openstack-kolla | 06:50 | |
| sdake | even 1 node ceph works | 06:50 |
| sdake | wh y3 node? | 06:50 |
| Jeffrey4l_ | right. | 06:50 |
| Jeffrey4l_ | you are using 1 node. | 06:50 |
| Jeffrey4l_ | so change to storage will work. | 06:50 |
| Jeffrey4l_ | if you have 10 nodes of storage, simple change control to storage is bad. | 06:51 |
| sdake | i see thanks | 06:51 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Download the rabbitmq_clusterer plugins to the correct path https://review.openstack.org/379960 | 06:51 |
| sdake | ok i'll change that to a wishlist bug | 06:52 |
| sdake | since its a feature request | 06:52 |
| sdake | i wanted to verify somethign wasn't badly broken with site.yaml :) | 06:52 |
| Jeffrey4l_ | sdake, add a new inventory file for standalone ceph, right? | 06:52 |
| sdake | right | 06:52 |
| Jeffrey4l_ | that's cool. | 06:52 |
| *** bjolo has quit IRC | 06:53 | |
| *** ldeptula has joined #openstack-kolla | 06:54 | |
| sdake | mostly to support the kolla-kubernetes case | 06:55 |
| sdake | ok thanks Jeffrey4l_ - thats enough to work with for others or for next cycle | 06:56 |
| sdake | had forgotten mons were on the control nodes | 06:56 |
| Jeffrey4l_ | ;) | 06:58 |
| *** nihilifer has quit IRC | 07:02 | |
| *** tonanhngo has joined #openstack-kolla | 07:02 | |
| *** tonanhngo has quit IRC | 07:04 | |
| *** HyperJohnGraham has joined #openstack-kolla | 07:13 | |
| *** sdake_ has joined #openstack-kolla | 07:14 | |
| *** sdake_ has quit IRC | 07:14 | |
| *** sdake has quit IRC | 07:15 | |
| *** matrohon has joined #openstack-kolla | 07:18 | |
| *** ankush has joined #openstack-kolla | 07:19 | |
| *** msimonin has joined #openstack-kolla | 07:21 | |
| *** daneyon has joined #openstack-kolla | 07:23 | |
| *** sdake has joined #openstack-kolla | 07:26 | |
| *** daneyon has quit IRC | 07:28 | |
| *** egonzalez90 has quit IRC | 07:34 | |
| Jeffrey4l_ | sdake, coolsvap all green https://review.openstack.org/379960 | 07:36 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix pycparser wheel package issue https://review.openstack.org/380957 | 07:37 |
| sdake | coolsvap this needs a workflow https://review.openstack.org/#/c/380929/2 | 07:38 |
| *** tonanhngo has joined #openstack-kolla | 07:38 | |
| coolsvap | sdake: yes was just looking in the failure logs | 07:39 |
| *** tonanhngo has quit IRC | 07:39 | |
| sdake | coolsvap that is to be expected | 07:39 |
| *** zigo has quit IRC | 07:39 | |
| sdake | rabbitmq is busted | 07:39 |
| sdake | the patches could be squashed, but they are seprate issues | 07:40 |
| sdake | with a voting gate we woudl be forced to squash them | 07:40 |
| coolsvap | got it yeah | 07:40 |
| Jeffrey4l_ | the pycparser need revert when it is fixed. | 07:40 |
| sdake | Jeffrey4l_ ack | 07:40 |
| sdake | Jeffrey4l_ yet we need to unblock dev right now ;) | 07:40 |
| Jeffrey4l_ | yes. | 07:41 |
| Jeffrey4l_ | the mitaka branch gate is busted for the same reason. | 07:41 |
| *** zigo has joined #openstack-kolla | 07:42 | |
| *** zigo is now known as Guest24756 | 07:42 | |
| sdake | pls run a recheck on this once those merge: https://review.openstack.org/#/c/379901/2 | 07:42 |
| sdake | i really need ot hit the rack | 07:42 |
| sdake | 1am here | 07:42 |
| sdake | have contractors coming over at 6am | 07:42 |
| openstackgerrit | Merged openstack/kolla: Fix pycparser wheel package issue https://review.openstack.org/380929 | 07:43 |
| coolsvap | Jeffrey4l_: the https://review.openstack.org/#/c/379960/ looks good but we need better handle over the version change | 07:44 |
| *** salv-orl_ has joined #openstack-kolla | 07:44 | |
| Jeffrey4l_ | coolsvap, i'd like to use the repo rabbitmq verison for each distro. lik 3.6.5 for centos and 3.5.7 for ubuntu. | 07:47 |
| Jeffrey4l_ | but seems inc0 like use the same version for both ubuntu and centos. | 07:47 |
| *** salv-orlando has quit IRC | 07:48 | |
| *** Pavo has quit IRC | 07:48 | |
| coolsvap | i think using similar versions might cause issues later | 07:48 |
| coolsvap | if same version is not synced in both at the same time | 07:48 |
| Jeffrey4l_ | coolsvap, i think so. moreover, use the repo packages ,which is well tested, is the best choices. | 07:49 |
| coolsvap | agreed | 07:50 |
| *** bjolo has joined #openstack-kolla | 07:50 | |
| Jeffrey4l_ | we need a strong reason to use the same version for centos and ubuntu. we need sync with inc0 when he come. | 07:51 |
| coolsvap | yes | 07:51 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix ironic failed https://review.openstack.org/379106 | 07:53 |
| *** Pavo has joined #openstack-kolla | 07:53 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Handle the KeyboardInterrunpt properly for build.py script https://review.openstack.org/380824 | 07:53 |
| sdake | Jeffrey4l_ coolsvap just merge what we got | 07:53 |
| sdake | to unblock dev | 07:54 |
| sdake | and sort out the argument with inc0 later | 07:54 |
| sdake | (today or tomorrow) | 07:54 |
| sdake | atm the entire community is blocked on these two patches | 07:54 |
| sdake | the z stream version is not much of matter to me | 07:54 |
| sdake | the erlang deps + y stream version matter the most i think | 07:55 |
| sdake | we have lots of software that is unsynced between distros related to versions | 07:56 |
| sdake | keepalived | 07:56 |
| sdake | i could go on | 07:56 |
| sdake | but i need to go to bed :) | 07:56 |
| sdake | just merge the patches | 07:56 |
| sdake | if michal doesn't like it he can submit a change or ask someone else to do so | 07:56 |
| *** nihilifer has joined #openstack-kolla | 07:57 | |
| sdake | as long as the gate doesn't block, IDC | 07:57 |
| *** shardy has joined #openstack-kolla | 07:58 | |
| sdake | sup shardy | 07:58 |
| sdake | sup nihilifer | 07:58 |
| sdake | and goodnight | 07:58 |
| *** sdake has quit IRC | 07:58 | |
| nihilifer | good night ;) | 07:59 |
| Jeffrey4l_ | good night | 08:05 |
| *** Serlex has joined #openstack-kolla | 08:09 | |
| *** egonzalez90 has joined #openstack-kolla | 08:14 | |
| *** g3ek has quit IRC | 08:15 | |
| *** g3ek has joined #openstack-kolla | 08:16 | |
| *** Guest24756 is now known as zigo | 08:54 | |
| *** yuanying has joined #openstack-kolla | 08:59 | |
| openstackgerrit | Merged openstack/kolla: Download the rabbitmq_clusterer plugins to the correct path https://review.openstack.org/379960 | 09:05 |
| *** pbourke has quit IRC | 09:16 | |
| *** pbourke has joined #openstack-kolla | 09:17 | |
| *** gfidente has joined #openstack-kolla | 09:23 | |
| *** athomas has joined #openstack-kolla | 09:36 | |
| *** bjolo has quit IRC | 09:40 | |
| *** bjolo_ has joined #openstack-kolla | 09:40 | |
| *** haplo37_ has quit IRC | 09:41 | |
| *** haplo37_ has joined #openstack-kolla | 09:43 | |
| *** sdake has joined #openstack-kolla | 09:44 | |
| *** Pavo has quit IRC | 09:48 | |
| sdake | the problem with 3 hour powernaps every 12 hrs | 09:50 |
| sdake | patterns.. | 09:50 |
| sdake | thats the problem | 09:50 |
| sdake | anything busted needing unblocking | 09:51 |
| sdake | seems like we have alot of that going on | 09:51 |
| *** Pavo has joined #openstack-kolla | 09:53 | |
| *** hieulq has quit IRC | 09:54 | |
| sdake | shardy you about | 10:01 |
| shardy | sdake: Hi! | 10:03 |
| sdake | hey shardy - probably out of your area of expertise, but any idea where the SRPMs are for atomic registry? | 10:03 |
| sdake | v1k0d3n showed it to me some months back | 10:03 |
| sdake | and i had a desire to play around with it in more detail | 10:03 |
| sdake | the but there, is "I'll ask anyway" :) | 10:05 |
| *** daneyon has joined #openstack-kolla | 10:05 | |
| shardy | sdake: Hmm, not sure tbh - Slower or rhallisey may know when they come online (or have a better idea who/where to ask) | 10:08 |
| sdake | shardy ya you were a shot in the dark ;) | 10:09 |
| sdake | shardy but only red hatter around at 3am my time :) | 10:09 |
| sdake | that I know may know :) | 10:09 |
| sdake | i dont often see slower on irc | 10:09 |
| shardy | sdake: also probably worth asking in #rdo, a lot of packaging experts hang out there who may know :) | 10:09 |
| sdake | ya i'll check there now - thanks | 10:10 |
| *** daneyon has quit IRC | 10:10 | |
| *** nihilifer has quit IRC | 10:11 | |
| *** nihilifer has joined #openstack-kolla | 10:12 | |
| openstackgerrit | bjorn lofdahl proposed openstack/kolla: fixed kestone fernet prechecks for multinode deployments https://review.openstack.org/380014 | 10:14 |
| *** tonanhngo has joined #openstack-kolla | 10:27 | |
| *** tonanhngo has quit IRC | 10:29 | |
| *** mewald has joined #openstack-kolla | 10:31 | |
| mewald | Guys I have something here. Maybe not Kolla releated but I have never seen it happen in a non Kolla set up: My services (especially) Horizon have insanely high reponse times. Loading Horizon from a controller takes up to 30s and as haproxy is set to a timeout of 10s i get gateway timeout errors. I cannot pin it down. I looked at CPU, memory, swap, full disk but couldn't find any obvious bottlenecks. Any suggestions? Has anyone seen that before? | 10:35 |
| sdake | mewald have not seen | 10:42 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 10:44 |
| kfox1111 | sdake: bad case of insomnia... | 10:45 |
| kfox1111 | had this code running through my head, so figured I might as well try and get it written down, rather then spinning my brain thinking about it over and over again. :/ | 10:46 |
| *** openstackgerrit has quit IRC | 10:48 | |
| *** openstackgerrit has joined #openstack-kolla | 10:48 | |
| bjolo_ | mewald, do you set FQDNs in globals.yml | 10:49 |
| openstackgerrit | Merged openstack/kolla: Add CADF event configurations in Keystone when enabled https://review.openstack.org/380641 | 10:49 |
| bjolo_ | DNS always comes to mind when troubleshooting long strange response times | 10:50 |
| kfox1111 | sdake: I was going to add a few more gate checks to ensure everything was working, then was going to work on this piece... so this is a ilttle accelerated. If we would have tried the standup in a week it would have been ready I think. :/ | 10:51 |
| sdake | kfox1111 pretty much :) | 10:51 |
| sdake | standup? | 10:51 |
| sdake | nah that was to help that cat get rolling | 10:52 |
| sdake | he seemed super dedicated too :) | 10:52 |
| * kfox1111 nods | 10:52 | |
| sdake | hopefully is he back tomorrow/today :) | 10:52 |
| sdake | i am going to ooo for most of day unfortunately | 10:52 |
| kfox1111 | we probably need to kep pushing people towards minikube for testing, rather then full blown multinode | 10:52 |
| kfox1111 | until we have a bit more stuff in place. | 10:53 |
| sdake | ya he wants multinode - has minikube running | 10:53 |
| kfox1111 | ah. | 10:53 |
| sdake | he said minikube was flawless | 10:53 |
| kfox1111 | like I said, it would be much better a week from now. :? | 10:53 |
| sdake | and he really liked it | 10:53 |
| kfox1111 | nice. :) | 10:53 |
| sdake | feel free to push out to wheneveryou think you can make it work for him | 10:53 |
| sdake | i got him into a state of a working ceph system | 10:54 |
| sdake | the problem is when people are working on something - best ot strike while iron is hot ;) | 10:54 |
| sdake | for e.g. i have an interest in atomic registry | 10:54 |
| sdake | and can't find the source code | 10:54 |
| kfox1111 | yeah. if nothing else, we fleshed out a good chunk of missing pieces. | 10:54 |
| sdake | quickly losing interest | 10:54 |
| sdake | it was fantastic for me | 10:54 |
| sdake | i learned all kinds of things | 10:54 |
| sdake | about where are gaps are | 10:54 |
| sdake | where are strengths are | 10:54 |
| sdake | what else needs to be done | 10:55 |
| sdake | etc | 10:55 |
| kfox1111 | the gaps mostly are in the ceph bits, | 10:55 |
| sdake | i gathered that | 10:55 |
| kfox1111 | as they are less then 2 weeeks old, and incomplete. | 10:55 |
| sdake | well we were using ceph from kolla stable | 10:55 |
| sdake | the integration there seems nonexistent | 10:55 |
| kfox1111 | yeah, its not kolla thats incomplete. just the stand up procedure for kolla-kubernetes with its templates. | 10:55 |
| kfox1111 | sbezverk has stood it up manualy a few times, | 10:56 |
| sdake | i think we got that sorted out today | 10:56 |
| kfox1111 | but with no security what so ever. | 10:56 |
| sdake | atleast how to stand up part of the system | 10:56 |
| kfox1111 | no sane op would deploy with an admin key for everything. makes my head spin looking at it. :/ | 10:57 |
| sdake | i'd be more concerned with bigger picture problems | 10:57 |
| sdake | like integrating ceph with kubernetes for e.g. | 10:57 |
| sdake | because that was our first blocker | 10:57 |
| kfox1111 | yeah. thats some of the remaining missing pieces. | 10:58 |
| kfox1111 | the ps should help a lot with that. | 10:58 |
| kfox1111 | part of it is, there's some workflowy bits around creating users/pools and importing keys into secrets in k8s that weren't done yet, and this ps tries to address. | 10:59 |
| kfox1111 | it was left up to an excersize to the reader before, with nothing to read. :/ | 10:59 |
| kfox1111 | uh.... thats weird... Error from server: secrets "ceph-client-admin-keyring" already exists | 11:00 |
| *** duonghq has quit IRC | 11:00 | |
| *** phuongnh has quit IRC | 11:00 | |
| kfox1111 | oh... left a copy still... | 11:00 |
| kfox1111 | OH it was from the mini ceph stuff... ok. | 11:02 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 11:03 |
| kfox1111 | we need a place to store some cached items. | 11:04 |
| kfox1111 | we could probably cut the kolla-kubernetes gate time in half or more if we could stash some stuff and fetch it at the begining of the job. | 11:05 |
| *** tonanhngo has joined #openstack-kolla | 11:08 | |
| *** g3ek has quit IRC | 11:08 | |
| sdake | kfox1111 thats a nogo | 11:09 |
| sdake | kfox1111 i know it would speed up gate times | 11:09 |
| *** tonanhngo has quit IRC | 11:09 | |
| sdake | talking with infra there is no way to make that happen | 11:09 |
| kfox1111 | why not? | 11:09 |
| kfox1111 | we have our own server for apps.openstack.org, which I believe infra is hosting. | 11:10 |
| kfox1111 | we could host our own cache similarly maybe. | 11:10 |
| *** slagle has joined #openstack-kolla | 11:10 | |
| *** haplo37 has quit IRC | 11:11 | |
| *** mliima has joined #openstack-kolla | 11:12 | |
| *** sdake_ has joined #openstack-kolla | 11:12 | |
| sdake_ | kfox1111 we have been unable to come to agreement over what exactly is needed | 11:13 |
| sdake_ | they want to cache our docker built images | 11:13 |
| sdake_ | we want fresly built images each time | 11:13 |
| sdake_ | so caching our docker built images is not good for us | 11:13 |
| sdake_ | but it may be good for you | 11:13 |
| sdake_ | we want to push to a docker registry on infra | 11:13 |
| *** salv-orl_ has quit IRC | 11:14 | |
| sdake_ | infra somehow wants us to not use docker push but some other undefined mechanism | 11:14 |
| *** sdake has quit IRC | 11:14 | |
| *** haplo37 has joined #openstack-kolla | 11:14 | |
| sdake_ | kfox1111 in essence I haven't been able to pin anyone down to do the work | 11:14 |
| sdake_ | and its *new* work | 11:14 |
| sdake_ | kfox1111 not something we can just use that already exists | 11:14 |
| sdake_ | kfox1111 infra is slammed 24/7 | 11:15 |
| sdake_ | and while they are the largest team in openstack (at my last look at teamstats.py) | 11:15 |
| sdake_ | they need more contribs imo :) | 11:15 |
| sdake_ | kfox1111 does that answer your question? | 11:17 |
| *** g3ek has joined #openstack-kolla | 11:18 | |
| kfox1111 | yeah. | 11:18 |
| *** salv-orlando has joined #openstack-kolla | 11:18 | |
| sdake_ | there are a whole bunch of problems wrapped up in the above | 11:18 |
| sdake_ | technical mostly | 11:18 |
| kfox1111 | yeah. for kolla-kubernetes, we can use a cache, as we're not the ones building the images. so testing with cached images is ok. | 11:19 |
| kfox1111 | we can have one job though that bypasses the cache just to double check the images. | 11:19 |
| sdake_ | right - so what builds the cache? | 11:19 |
| kfox1111 | periodic job? | 11:19 |
| kfox1111 | thought I saw something in zuul for that. | 11:19 |
| sdake_ | the cache contains what? | 11:19 |
| kfox1111 | docker images and prebuilt 'kolla genconfig' | 11:20 |
| sdake_ | how often does periodic job run | 11:20 |
| kfox1111 | nightlyish? | 11:20 |
| sdake_ | your going from CI to nightly builds then? :) | 11:20 |
| sdake_ | see - tricky problems here | 11:21 |
| kfox1111 | for keeping a cache up to date. the ci is still there. | 11:21 |
| sdake_ | the cache is part of the ci | 11:21 |
| kfox1111 | it just speeds up the ci. | 11:21 |
| sdake_ | images, configuration etc | 11:21 |
| *** psanchez has joined #openstack-kolla | 11:21 | |
| kfox1111 | sort of. | 11:21 |
| sdake_ | yes but images and config change | 11:21 |
| sdake_ | now if you could trigger on images and config files changing the periodic job | 11:21 |
| kfox1111 | yes/no. less likely to chang ethen the ps's your ci'ing against. | 11:21 |
| sdake_ | (ok so let me play infra here) | 11:21 |
| sdake_ | and ramble on for how we would design the perfect system for this problem | 11:22 |
| sdake_ | and finish up with "but we odn't have anyone to work on that" | 11:22 |
| kfox1111 | sure. | 11:22 |
| sdake_ | its a capacity problem | 11:22 |
| sdake_ | its a "its not really ci unless its built from scratch" problem | 11:23 |
| kfox1111 | but 5 min here, 5 min there, that addes up, and a frustrated dev... (perhaps me) might get time enough to fix it in the spare 5 min's that add up. :) | 11:23 |
| kfox1111 | capacity problem I can buy. | 11:23 |
| kfox1111 | though by having less time in the gate, it may reduce the time taken... depends if cache rebuilding is more common then ci tests. | 11:24 |
| sdake_ | also there was a long discussion of where to store these cached items and how to retrieve them securely | 11:24 |
| sdake_ | when I didn't ask for a cache at all ;-) | 11:24 |
| kfox1111 | the cache would not be for 100% of all things, just the things that don't really maatter. | 11:24 |
| kfox1111 | for example. | 11:24 |
| kfox1111 | I'm testing against the hub containers 2.0.2. | 11:24 |
| kfox1111 | they don't change. ever. | 11:24 |
| kfox1111 | so we can cache the hell out of them. :) | 11:24 |
| sdake_ | yup and the right place to do that is where ? | 11:25 |
| kfox1111 | closeish to the vms. | 11:25 |
| kfox1111 | but even if not, | 11:25 |
| sdake_ | ok cache "HOW" | 11:25 |
| kfox1111 | just tarballing them and doing an import shoudl be faster then pulling them from the hub I think. | 11:25 |
| kfox1111 | we stand up a webserver to host some tarballs. | 11:26 |
| sdake_ | ya just doesn't really meet my definition of quality testing | 11:26 |
| sdake_ | there is already a tarball site | 11:26 |
| sdake_ | we are using a bunch of rarely used mechanism in that model in the gate | 11:26 |
| sdake_ | import export save of docker images | 11:26 |
| sdake_ | super non-standard usage models | 11:26 |
| kfox1111 | it would need a cronjob on the site to docker pull a list of containers and dump them to a tar. | 11:26 |
| sdake_ | yes that could be a periodic job | 11:27 |
| kfox1111 | heh. yeah. | 11:27 |
| sdake_ | the right answer is it needs to be in a docker registry | 11:27 |
| kfox1111 | well, I found alot of weird stuf in their vm's trying to get k8s to run in the gate. :) | 11:27 |
| kfox1111 | their vm's are very nonstandard between the clouds. :/ | 11:27 |
| v1k0d3n | sdake_: holy cow...did you get any sleep at all last night? | 11:27 |
| sdake_ | v1k0d3n only 3 hours | 11:27 |
| kfox1111 | not so sure... | 11:27 |
| v1k0d3n | wow. both of you guys. | 11:28 |
| kfox1111 | v1k0d3n: me too. :/ | 11:28 |
| kfox1111 | insomnia sucks. | 11:28 |
| sdake_ | v1k0d3n its ok I slept 24 hrs saturday | 11:28 |
| v1k0d3n | yeah, no kidding! trust me...i'm there a lot too, i completely get it. | 11:28 |
| v1k0d3n | for me to sleep from 1-6:30 is a huge deal. | 11:28 |
| * kfox1111 nods | 11:28 | |
| v1k0d3n | lol fireman schedule. | 11:28 |
| sdake_ | went to bed friday night woke up sunday morning | 11:28 |
| sdake_ | first time that has happened in awhile | 11:29 |
| * kfox1111 is envious :) | 11:29 | |
| v1k0d3n | at least you got that. | 11:29 |
| sdake_ | so i'm a bit sleeped out | 11:29 |
| kfox1111 | yeah. | 11:29 |
| v1k0d3n | any luck last night? | 11:29 |
| v1k0d3n | with sdsu? | 11:29 |
| sdake_ | we got ceph deployed properly | 11:29 |
| sdake_ | what is sdsu | 11:29 |
| kfox1111 | I've got a ps going to make the ceph/k8s stuff smoother.... still a work in progress though. | 11:30 |
| sdake_ | v1k0d3n any idea where the SRPM files are for the atomic registry | 11:30 |
| kfox1111 | https://review.openstack.org/#/c/381041 | 11:30 |
| v1k0d3n | hmmm i can find out. it's been a while... | 11:30 |
| v1k0d3n | it's default is rpm-ostreee | 11:31 |
| sdake_ | ya thats not going to work for me | 11:31 |
| sdake_ | i get its super-integrated into kubernetes and atomic and whatever stuff red hat hawks | 11:32 |
| sdake_ | i just want a simple SRPM of atomic registry | 11:32 |
| sdake_ | looks like a good building block | 11:32 |
| v1k0d3n | oh...atomic registry...not atomic-os? | 11:33 |
| sdake_ | right | 11:33 |
| v1k0d3n | yeah, get's really messy because openshift is a dependency | 11:35 |
| sdake_ | you gotta be kidding me | 11:35 |
| v1k0d3n | no...in fact i've kind of stopped using it because of that hard dependency...it's too complex for a simple registry. | 11:36 |
| sdake_ | ok - so openshift = kubernetes + some special sauce consisting of mayo and ketchup | 11:36 |
| v1k0d3n | wanna know what i started using? | 11:36 |
| sdake_ | ya lets hear that | 11:36 |
| sdake_ | can't it simply be dconstructed into a non-openshift dependent system? | 11:37 |
| v1k0d3n | this has been so much easier for me...(for something simple w/auth and interface)...if that's what you're after... | 11:37 |
| v1k0d3n | https://hub.docker.com/r/hyper/docker-registry-web/ | 11:37 |
| v1k0d3n | comes with a docker-compose and integrated auth/tls if desired. | 11:37 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 11:38 |
| *** jax3242 has quit IRC | 11:38 | |
| sdake_ | does it come with source code | 11:39 |
| v1k0d3n | yup. one sec... | 11:39 |
| kfox1111 | sdake_: yeah, openshift is kind of half way between a fork of k8s and a distro of k8s + a bunch of stuff on top. | 11:39 |
| kfox1111 | they are activelyl working on undoing the fork and pushing stuff back upstream to k8s, | 11:39 |
| v1k0d3n | https://github.com/mkuchin/docker-registry-web | 11:39 |
| kfox1111 | but it takes time. they have made a lot of enhancements. | 11:39 |
| sdake_ | yup second way better - open core model | 11:40 |
| sdake_ | not ideal but better then a fork | 11:40 |
| *** jrollen has joined #openstack-kolla | 11:40 | |
| kfox1111 | as they pushed quicly into multitenancy, which k8s isn't there yet. | 11:40 |
| sdake_ | k8s has had multitenancy for some time now afaik | 11:40 |
| kfox1111 | I wouldn't call it that. :) | 11:40 |
| sdake_ | anyway i dont care to talk about openshift | 11:40 |
| kfox1111 | you have way too much power without their rbac system, which came from openshift actually, and it only landed in 1.3 and its still alpha. | 11:41 |
| v1k0d3n | kfox1111: yeah, i think that's needed. my biggest issue with atomic/atomic-registry is that i have to run so many extra things (openshift) and it's packaged as "atomic" commands. not sure why it's not a simple docker-compose or kubernetes manifest. | 11:41 |
| sdake_ | because v1k0d3n | 11:41 |
| kfox1111 | v1k0d3n: yeah. | 11:41 |
| sdake_ | STRATEGY ???????? PROFIT | 11:41 |
| kfox1111 | mostly beacause they want you to use all of atomic. | 11:42 |
| *** eaguilar has joined #openstack-kolla | 11:42 | |
| v1k0d3n | at least make it a little less obvious. | 11:42 |
| v1k0d3n | :) haha | 11:42 |
| sdake_ | comon guys dont ruin a perfectly good morning :) | 11:42 |
| kfox1111 | :) | 11:43 |
| sdake_ | i become annoyed when the best tools (such as atomic-registry) can't be used for the job because of someone's personal agenda to measure the worth of their life by their business accomplishments | 11:43 |
| *** tonanhngo has joined #openstack-kolla | 11:43 | |
| sdake_ | that said trying out this kube thing now | 11:44 |
| v1k0d3n | well, back to your point...i've tried so many times to make an atomic registry work in my small lab, but kept running into issues with the dependancies. so i drew back to that repo i set you for the small stuff, and looking at dockyard for some for the bigger stuff. | 11:44 |
| sdake_ | maybe that willbe the best tool for the job | 11:44 |
| *** tonanhngo has quit IRC | 11:45 | |
| sdake_ | Do not use registry as registry container name, it will break REGISTRY_NAME environment variable. | 11:45 |
| sdake_ | groan | 11:45 |
| kfox1111 | oh.... heh. | 11:46 |
| v1k0d3n | i change that in the provided docker-compose file. | 11:46 |
| v1k0d3n | (oh you may be talking about something else...sorry) | 11:47 |
| v1k0d3n | sdake_: i use something like this: https://github.com/mkuchin/docker-registry-web-examples/tree/master/nginx-auth-enabled | 11:47 |
| sbezverk | morning folks, have you fixed last night ceph issue? | 11:47 |
| *** Pavo has quit IRC | 11:48 | |
| sdake_ | sbezverk yup ixed it | 11:48 |
| sdake_ | had to comment out api_server | 11:48 |
| sdake_ | so api_server and orch engine need commenting | 11:48 |
| sdake_ | and now HyperJohnGraham could use some coaching on the next steps of setting up the actual kubernetes cluster | 11:49 |
| sdake_ | i was stuck - not knowing all the various magic incantations | 11:49 |
| kfox1111 | sbezverk: I have a ps in progres to help too. https://review.openstack.org/#/c/381041/3 | 11:49 |
| *** eaguilar has quit IRC | 11:49 | |
| kfox1111 | have a look if you get a sec please. | 11:49 |
| sbezverk | kfox1111: I posted one question | 11:49 |
| kfox1111 | k | 11:50 |
| sdake_ | v1k0d3n if at first you don't succeed, let someone copy your design completely without a completely insane dependency model | 11:50 |
| sdake_ | and fail again :) | 11:50 |
| sbezverk | I do not get why we need exatr mon and admin pods when we use external cluster?? | 11:50 |
| kfox1111 | sbezverk: didn't notice your question. | 11:50 |
| kfox1111 | ok. so, | 11:50 |
| v1k0d3n | failure (and acceptance of failure) is a constant state for everyone these days... | 11:50 |
| kfox1111 | ceph-rbd basically loads a rbd command onto each host if it doesn't already have one, so you don't have to install ceph on the host. | 11:51 |
| kfox1111 | the version will always match whats in kolla too. | 11:51 |
| kfox1111 | ceph-admin is a container with all the privileges needed so that you don't have to go find some host to do the mkfs/etc. | 11:51 |
| sbezverk | kfox1111: here is the problem, it does not have to be always kolla ceph cluster | 11:51 |
| kfox1111 | you can kubectl exec the commands right from the controller. | 11:51 |
| kfox1111 | sbezverk: true. but the client must match what's in the kolal containers. | 11:52 |
| sbezverk | I mean people might already have clusters and with different versions | 11:52 |
| *** sdake_ is now known as sdake | 11:52 | |
| sbezverk | kfox1111: why they have to match wil kolla container? | 11:52 |
| kfox1111 | if external ceph is infernalis, and kolla's hammer, or jewel, you will run into issues. | 11:52 |
| sbezverk | one sec, if there is NO kolla ceph at all | 11:53 |
| sbezverk | there is just operators exisiting cluster | 11:53 |
| kfox1111 | it doesn't much matter about an external ceph cluster if your not going to use it. | 11:53 |
| *** Pavo has joined #openstack-kolla | 11:54 | |
| sbezverk | kfox1111: I am going to use it for kolla-kube volumes | 11:54 |
| sbezverk | but I do not like idea of enforcing some conditions and setting I do not really need | 11:54 |
| kfox1111 | so, only for rbd backing volumes, but not for glance/cinder/nova? | 11:54 |
| sbezverk | yes | 11:55 |
| sbezverk | they might choose to use different | 11:55 |
| kfox1111 | not sure why you would ever do such a thing, but ok. | 11:55 |
| sbezverk | means | 11:55 |
| sbezverk | example I want to use iscsi | 11:55 |
| kfox1111 | then don't use ceph-rbd in that case. | 11:55 |
| sbezverk | and lvm | 11:55 |
| kfox1111 | lvm is still a joke. | 11:55 |
| kfox1111 | its the reference driver in cinder only because cinder doesn't want to tork off all the storage vendors by defaulting to ceph. | 11:56 |
| sbezverk | that is your opinion but nevertheless it is default backend used in cindet | 11:56 |
| kfox1111 | I've yet to see a production cluster use lvm, as its not fault tollerent. | 11:56 |
| sbezverk | and as such is valid and very common option | 11:56 |
| sbezverk | kfox1111: I would suggest just address external cluster without tieing it with any other baggage | 11:58 |
| *** dwalsh has joined #openstack-kolla | 11:58 | |
| sbezverk | and then offer options so people are not forced | 11:58 |
| kfox1111 | then don't use ceph-rbd. | 11:58 |
| kfox1111 | you don't have to. | 11:58 |
| kfox1111 | but if your external ceph version matches your kolla version, then you don't have to install ceph everywhere. just laucnh ceph-rbd. | 11:59 |
| sbezverk | curious how many kolla installed ceph cluster is out there, but I bet non kolla ceph clusters are more. | 12:00 |
| kfox1111 | still though, most people I'vs seen deploy lvm do it because they don't want to learn ceph. | 12:00 |
| *** jax3242 has joined #openstack-kolla | 12:00 | |
| kfox1111 | but if you install ceph anyway, I see very little reason not to use it for cinder too. | 12:00 |
| *** bdaca has quit IRC | 12:00 | |
| kfox1111 | We're using ceph-deploy for our cephs, | 12:00 |
| sbezverk | maybe but it does not have to force you to use kolla ceph cluster | 12:01 |
| kfox1111 | but will be using kolla containers with ceph pointint at it. | 12:01 |
| kfox1111 | this does not tie your hands there. | 12:01 |
| kfox1111 | ceph-deploy is much much more common for deploying ceph then kolla dployed cephs. | 12:03 |
| kfox1111 | we won't be forcing kolla deployed cephs as a requirement. | 12:03 |
| *** b_bezak_ has joined #openstack-kolla | 12:03 | |
| kfox1111 | this is just giving k8s clusters a way not to have to deploy ceph rbd client on the host. | 12:03 |
| sbezverk | but then you need to match this client version with the external cluster and not kolla, no? | 12:05 |
| *** b_bezak has quit IRC | 12:05 | |
| kfox1111 | kolla's clients gota mach the external ceph. yes. | 12:05 |
| kfox1111 | like I said, kolla will break in general if your kolla ceph and your external ceph versions don't match. | 12:05 |
| kfox1111 | the rbd client may be more forgiving, as its just talking to the kernel client. and that's usulaly quite old. | 12:06 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Start testing the deployments https://review.openstack.org/380868 | 12:07 |
| *** tonanhngo has joined #openstack-kolla | 12:13 | |
| *** tonanhngo has quit IRC | 12:14 | |
| *** v1k0d3n has quit IRC | 12:14 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 12:14 |
| *** mewald has quit IRC | 12:17 | |
| *** jrollen has quit IRC | 12:18 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Start testing the deployments https://review.openstack.org/380868 | 12:19 |
| kfox1111 | sbezverk: I just folded in the keepalived stuff into that patch. so we should be able to gate on it soon. :) | 12:21 |
| *** awiddersheim has joined #openstack-kolla | 12:23 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 12:24 |
| *** lamt has quit IRC | 12:25 | |
| Daviey | sdake, coolsvap: rabbitmq still looks broken to me on centos | 12:25 |
| *** eaguilar has joined #openstack-kolla | 12:26 | |
| sdake | sup dudes | 12:26 |
| sdake | kfox1111 i'd like to stick to things our community understands fwiw... | 12:26 |
| *** schwicht has quit IRC | 12:26 | |
| sdake | which is ceph-deploy? | 12:27 |
| sdake | Daviey broken as in not building? | 12:27 |
| sdake | gate was green | 12:27 |
| Daviey | sdake: check the logs | 12:28 |
| Daviey | sdake: the logs for source centos are pretty much contentless | 12:29 |
| sdake | in the gate job? | 12:29 |
| sdake | code looked correct - gate was green | 12:29 |
| sdake | not sure we can expect more from our core reviewers | 12:29 |
| sdake | i dont want us to start distrusting our tools | 12:29 |
| sdake | thats the worst possible thing that can happen | 12:30 |
| Daviey | sdake: no, not saying that | 12:30 |
| kfox1111 | sdake: yeah, ceph deploy is used almost everywhere. | 12:30 |
| Daviey | sdake: I'm saying, that the issue looks unresolved | 12:30 |
| sdake | Daviey so the gate was incorrect then? | 12:30 |
| sdake | Daviey are you deploying multinode or aio? | 12:30 |
| Daviey | sdake: looks like it | 12:30 |
| Daviey | sdake: no, this is BUILDING | 12:30 |
| kfox1111 | we won't be switching away from it for quite some time. so trust me, the work flow's going to work with it too if I ahve anything to say about it. :) | 12:31 |
| Daviey | sdake: https://review.openstack.org/#/c/379960/ | 12:31 |
| Daviey | sdake: https://review.openstack.org/#/c/379960/gate-kolla-dsvm-build-centos-source-centos-7-nv | 12:31 |
| sdake | kfox1111 well we need to decide then if we are to jettison our ceph implementation | 12:31 |
| sdake | because i'd like things to be common - whatever common is | 12:31 |
| kfox1111 | I think its fine, as a batteries included thing. | 12:31 |
| sdake | i dont particularly care | 12:31 |
| *** Serlex has quit IRC | 12:32 | |
| sdake | yes i think we should be documented the batteries included parts not some other project ;) | 12:32 |
| sdake | i am in no way locked in or tied to the ceph implementation we have | 12:32 |
| kfox1111 | what the kolla-kubernetes worflow does though by default, not sure. ceph-deploy is more common, but, /me shurgs. | 12:32 |
| sdake | if there is something better that does the ob we need lets use that | 12:32 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Fixing neutron-openvswitch missing mount and variable https://review.openstack.org/380836 | 12:32 |
| sdake | lets be consistent in tools, logging, monitoring etc | 12:33 |
| sdake | so we don't end up with two wildly divergent things ;) | 12:33 |
| kfox1111 | +1 | 12:33 |
| sdake | if ceph-deploy is the answer wfm, i dont really care as long as it works and provides one button deploy :) | 12:33 |
| sdake | if it doesn't do those things then we need to document the one button deploy method | 12:34 |
| kfox1111 | well, right now, deploying ceph isn't my main concern. | 12:34 |
| sdake | i know your a long ways away rom that with kolla-kubernetes | 12:34 |
| kfox1111 | as that's well documented. | 12:34 |
| kfox1111 | my concern's hooking it into kolla-kubernetes. | 12:34 |
| sdake | how does ceph-deploy make that easier? | 12:35 |
| kfox1111 | the workflow around, creating the users/volumes, formatting them, hooking them into k8s via pv's, etc. | 12:35 |
| kfox1111 | it doesnt. | 12:35 |
| *** dave-mccowan has joined #openstack-kolla | 12:35 | |
| sdake | ok so why document two divergent techniques? | 12:36 |
| Daviey | ceph-deploy, being upstreams own tool.. means we *might* need to maintain less inhouse. | 12:36 |
| sdake | ok its totally on the table to remove ceph from kolla | 12:36 |
| sdake | i dont care as i said - not tied to the implementation | 12:36 |
| sdake | but want one implementation not two ;) | 12:36 |
| kfox1111 | I think there's a bit of confusion here... | 12:36 |
| kfox1111 | there's 2 parts here.. | 12:36 |
| sdake | if someone uses the external ceph feature, they use that at their own risk :) | 12:37 |
| kfox1111 | ceph the server, and ceph the client. | 12:37 |
| kfox1111 | ceph the server, is commonly deployed with ceph-deploy. | 12:37 |
| kfox1111 | ceph the client is baked into the kolla containers, and that must still be there. | 12:37 |
| kfox1111 | I'm focused on the client bits. | 12:37 |
| sdake | ceph the server in kolla land is almsot always deployed with the kolla ceph implementation | 12:37 |
| sdake | unless someone wants an external ceph | 12:37 |
| sdake | in which case I don't know how they deploy | 12:38 |
| kfox1111 | I need my openstacks to be backed with ceph for glance/cinder volumes, and all the state for the k8s containers that needs to be kept. | 12:38 |
| sdake | totally get that, you get that with an ansible deploy of ceph in containers using the techniques we used tonight/yesterday :) | 12:38 |
| kfox1111 | so that's all going to be kolla ceph for sure. | 12:38 |
| kfox1111 | we have existing cephs deployed with ceph-deploy though, so those are unlekly to move to kolla containers. | 12:39 |
| sdake | totallly get that too - thats why external ceph model exists in kolla | 12:39 |
| kfox1111 | yeah. | 12:39 |
| kfox1111 | so I'm going to make sure that use case works, as its my primary use case. :) | 12:39 |
| sdake | just like ops may not want to run a db in kolla and want to run their own deploy of that | 12:39 |
| kfox1111 | right. | 12:39 |
| kfox1111 | I may run that way for a while too. | 12:40 |
| kfox1111 | as our existing db cluster works pretty well. | 12:40 |
| *** fguillot has joined #openstack-kolla | 12:40 | |
| sdake | i understand all of this, what i struggle with is documeting two competing methods of deploying ceph in the same project | 12:40 |
| kfox1111 | ah. | 12:40 |
| kfox1111 | well, I thin kthe docs should basically say, stand up a ceph. look here or here. | 12:40 |
| sdake | either a) lets make a decision to eject the ceph implementation or b) eject the documentation for deploying via ceph-deploy | 12:41 |
| kfox1111 | then, once you do that, then the steps to hook it in start. | 12:41 |
| kfox1111 | they shoudl be pretty similar at that point. | 12:41 |
| kfox1111 | I don't think we shoudl recreate the docs for either deploying with ceph-deploy or with kolla-ansible. they already have docs. | 12:41 |
| *** jrollen has joined #openstack-kolla | 12:41 | |
| kfox1111 | the kolla-kube docs should be: you have an external ceph, good, now here's how you hook it in... | 12:42 |
| sdake | ok - so we need to document how to create an external ceph with kolla itself | 12:42 |
| sdake | because that is undocumented | 12:42 |
| sdake | and that stops people from adopting the project | 12:42 |
| kfox1111 | mmm... I think those docs will be different for kolla-ansible then with kolla-kubernetes? | 12:43 |
| sdake | the longer it takes me to get from steps a to z the more squirrels that pop up | 12:43 |
| sdake | kfox1111 right | 12:43 |
| sdake | kfox1111 kolla-ansible needs no such thing | 12:43 |
| kfox1111 | anyway, I am starting to document it here: https://review.openstack.org/#/c/381041/5 | 12:43 |
| kfox1111 | still a big work in progress. lots of gaps. but less gaps then a few hours ago. :) | 12:43 |
| kfox1111 | plus building tooling to help so the user doesn't have to do as much. | 12:44 |
| sdake | kfox1111 squrrel hunting atm - will take a look at your doc changes when done | 12:45 |
| kfox1111 | k | 12:45 |
| *** jrollen has quit IRC | 12:45 | |
| kfox1111 | anyway... my goal with it is the same as minikube. | 12:45 |
| *** jrollen has joined #openstack-kolla | 12:45 | |
| kfox1111 | before minikube/the minikube docs, it was increadilby hard to test kout kolla-kubernetes, | 12:46 |
| kfox1111 | as there were large swaths of stuff undcoumented. | 12:46 |
| kfox1111 | now someone can pick it up and get a working system in half an hour. | 12:46 |
| kfox1111 | I'm going to try to do the same with the ceph docs and gate. | 12:47 |
| kfox1111 | it just takes time. | 12:47 |
| sdake | cool | 12:47 |
| sdake | one set of tools , plz :) | 12:47 |
| sdake | pick one or another - and eject the other | 12:47 |
| coolsvap | the change for pycparser downgrade has landed in requirements for all branches, will be merged in 30-25 mins | 12:48 |
| sdake | coolsvap thats good news | 12:48 |
| coolsvap | sdake: Jeffrey4l_ Daviey ^^ | 12:48 |
| kfox1111 | the tools on the kolla-kubernetes will be the same either way. | 12:48 |
| coolsvap | 30-45* | 12:48 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 12:49 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Start testing the deployments https://review.openstack.org/380868 | 12:52 |
| *** jroll has quit IRC | 12:53 | |
| *** schwicht has joined #openstack-kolla | 12:54 | |
| sdake | anyone care to see the registry gui? | 12:55 |
| *** jrollen is now known as jroll | 12:55 | |
| kfox1111 | registry gui? | 12:55 |
| sdake | i threw it together in 10 or 15 minutes | 12:55 |
| sdake | ya let me open a port on my firewall | 12:55 |
| Daviey | coolsvap: erm | 12:56 |
| Daviey | coolsvap: the issue with rabbitmq on centos is that, deb was updated to be rabbitmq-server_3.6.5 from curl+.deb... but centos is still using yum at version 3.6.2 | 12:56 |
| Daviey | And worse, our gate is lieing to us | 12:57 |
| sdake | kfox1111 broked.selfip.net:8080 | 12:57 |
| sdake | Daviey how could dthat possibly be, rdo has moved to 3.6.5 ... | 12:58 |
| kfox1111 | sdake: nice. :) | 12:58 |
| sdake | its not actually my work on the code | 12:58 |
| sdake | v1k0d3n turned me onto it | 12:59 |
| sdake | i can finally tidy up my registry | 12:59 |
| Daviey | sdake: hmm | 12:59 |
| Daviey | sdake: well, can someone else reproduce it locally.. looks like someone else did | 12:59 |
| sdake | gpl2.0 looks like a winner | 13:00 |
| sdake | https://github.com/mkuchin/docker-registry-web | 13:00 |
| Daviey | i don't think bug 1629596 really is a duplcate | 13:01 |
| openstack | bug 1611655 in kolla "duplicate for #1629596 kolla.image.build.rabbitmq curl No such directory" [Critical,Fix released] https://launchpad.net/bugs/1611655 | 13:01 |
| Daviey | sdake: Are you able to reproduce the issue there? | 13:02 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 13:02 |
| kfox1111 | there.s the issue... | 13:02 |
| sdake | Daviey i was squirrel hunting | 13:02 |
| sdake | daviey now i am buildling | 13:02 |
| kfox1111 | /etc/ceph/ceph.client.admin.keyring != /etc/ceph.client.admin.keyring | 13:02 |
| kfox1111 | bleh. | 13:02 |
| kfox1111 | :) | 13:03 |
| *** tonanhngo has joined #openstack-kolla | 13:04 | |
| *** tonanhngo has quit IRC | 13:05 | |
| openstackgerrit | Dave Walker proposed openstack/kolla: Install RabbitMQ from curl/rpm https://review.openstack.org/381104 | 13:07 |
| sdake | Daviey master wfm: INFO:kolla.image.build.rabbitmq:Step 4 : RUN rm -rf /var/lib/rabbitmq/* && curl -o /usr/lib/rabbitmq/lib/rabbitmq_server-3.6.5/plugins/rabbitmq_clusterer-3.6.x-667f92b0.ez http://www.rabbitmq.com/community-plugins/v3.6.x/rabbitmq_clusterer-3.6.x-667f92b0.ez && /usr/lib/rabbitmq/bin/rabbitmq-plugins enable --offline rabbitmq_management rabbitmq_clusterer && /bin/true | 13:07 |
| *** athomas has quit IRC | 13:08 | |
| *** athomas has joined #openstack-kolla | 13:08 | |
| sdake | Daviey [sdake@minime-03 tools]$ git pull --rebase | 13:09 |
| sdake | Current branch master is up to date. | 13:09 |
| Daviey | sdake: it worked for you? | 13:09 |
| sdake | Daviey run the command pip show kolla for me pls | 13:09 |
| sdake | and paste in irc | 13:09 |
| sdake | yes wfm | 13:09 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Start testing the deployments https://review.openstack.org/380868 | 13:10 |
| Daviey | wtf | 13:10 |
| Daviey | sdake: Version: 3.0.0.0rc2.dev241 | 13:10 |
| sdake | Daviey therein lies your problem | 13:10 |
| sdake | RTD :) | 13:10 |
| sdake | pip remove kolla | 13:10 |
| *** Serlex has joined #openstack-kolla | 13:11 | |
| sdake | i even emailed the mailing list about this issue with the kolla tag... (that the docs were updated) | 13:11 |
| Daviey | sdake: what version do you have? | 13:11 |
| sdake | no idea i dont pip install | 13:11 |
| sdake | because thats not how kolla works for devs | 13:11 |
| Daviey | sdake: what mail are you talking about? | 13:12 |
| Daviey | sdake: pip should work for devs! | 13:12 |
| sdake | http://docs.openstack.org/developer/kolla/quickstart.html | 13:12 |
| sdake | quoted: | 13:13 |
| sdake | Warning Kolla uses PBR in its implementation. PBR provides version information to Kolla about the package in use. This information is later used when building images to specify the Docker tag used in the image built. When installing the Kolla package via pip, PBR will always use the PBR version information. When obtaining a copy of the software via git, PBR will use the git version information, but ONLY if Kolla has not been pip installed via the | 13:13 |
| sdake | pip package manager. This is why there is an operator workflow and a developer workflow. | 13:13 |
| sdake | use git blame to find the line - and yo uwill find the long irc conversation with doug in the commit log about the problem with pbr | 13:13 |
| sdake | i trust doug knows what he is talking aobut when it comes to pbr :) | 13:13 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 13:13 |
| Daviey | sdake: I'm pretty confident in syaing that pbr and pip should be able to work for developer workflow... but i'll happily read the backlog | 13:14 |
| sdake | please do | 13:14 |
| sdake | if you find a better solution i'm open to it | 13:14 |
| sdake | this was what we came up with after 15-20 hours of collaboration | 13:14 |
| *** athomas has joined #openstack-kolla | 13:15 | |
| sdake | it is clearly suboptimal | 13:15 |
| sdake | we are fixing it by having a dev workflow and an installation guide on the main docs.oo website | 13:15 |
| sdake | (the confusion in the docs aroudn multiple workflows) | 13:15 |
| kfox1111 | well, the nice thing about having a 15 min gate, is you can run two at the same time and still continue to work. :) | 13:16 |
| sdake | Daviey here is the deal - for months i've been telling peopele to pip remove kolla | 13:16 |
| sdake | because it results in inconsistent bbehavior with pbr and versioning of containers | 13:16 |
| sdake | that has been codified into docs | 13:16 |
| sdake | now i dont get asked 5 times a day why kolla doesn't work | 13:16 |
| sdake | i call that progress | 13:16 |
| *** inc0 has joined #openstack-kolla | 13:17 | |
| Daviey | well sure... but i've been pip installing from git for months... and it has only broken this morning | 13:17 |
| sdake | try pip remove | 13:17 |
| sdake | and build | 13:17 |
| sdake | and see what happens | 13:17 |
| sdake | (with master) | 13:17 |
| sdake | 1 USD on the line :) | 13:17 |
| Daviey | (if this works, i will literally eat my hat) | 13:18 |
| kfox1111 | Daviey: be carefull with that... I have a friend that said he'd eat his shoe if gas prices ever droped be low 3 dollars... he ows us like 3 pairs by now. :) | 13:18 |
| inc0 | good morning | 13:18 |
| kfox1111 | morning | 13:19 |
| sdake | well to be honest i have no idea how pbr works | 13:19 |
| sdake | i am just going off doug's guidance | 13:19 |
| sdake | and his last word on the subject was "if the docs work that is the true test" | 13:19 |
| Daviey | Thankfuly, i don't need to eat my hat | 13:19 |
| Daviey | (it still failed) | 13:19 |
| sdake | Daviey throw me yoru build line | 13:19 |
| Daviey | sudo -E kolla-build --registry 127.0.0.1:4000 --push -n local -t source --threads 1 --keep rabbitmq | 13:19 |
| *** srwilkers has joined #openstack-kolla | 13:21 | |
| sdake | Daviey how is that working if you pip removed kolla? | 13:21 |
| sdake | you should have no kolla-build tool anymore | 13:21 |
| Daviey | sdake: well i pip removed and pip installed from pwd | 13:21 |
| sdake | ok well thats not what i said :) | 13:21 |
| sdake | pip -e doesn't work either | 13:22 |
| sdake | but trying yoru build operation without the pip -e | 13:22 |
| *** absubram has quit IRC | 13:22 | |
| sdake | INFO:kolla.image.build.rabbitmq:Step 4 : RUN rm -rf /var/lib/rabbitmq/* && curl -o /usr/lib/rabbitmq/lib/rabbitmq_server-3.6.5/plugins/rabbitmq_clusterer-3.6.x-667f92b0.ez http://www.rabbitmq.com/community-plugins/v3.6.x/rabbitmq_clusterer-3.6.x-667f92b0.ez && /usr/lib/rabbitmq/bin/rabbitmq-plugins enable --offline rabbitmq_management rabbitmq_clusterer && /bin/true | 13:22 |
| sdake | wfm | 13:22 |
| sdake | with your build line | 13:23 |
| sdake | changing kolla-build to ./build.py | 13:23 |
| sdake | daviey what your missing is --tag... | 13:23 |
| sdake | because pbr is BDWNF | 13:23 |
| kfox1111 | there we go... ceph-admin container seems to work... | 13:23 |
| sdake | and doug says its impossible to fix | 13:23 |
| kfox1111 | now to plumb in the rbd creation bits... | 13:23 |
| sdake | or atleast that was my interpretation thereof | 13:24 |
| sdake | admittedly i am a pbr rookie | 13:24 |
| sdake | as stated above | 13:24 |
| sdake | I know absolutely nothing about it | 13:25 |
| sdake | other then its caused me numerous headaches over the last 3+ months | 13:25 |
| sdake | and now its causing you one too :) | 13:25 |
| Daviey | yeah, i had pbr issues previously... had to dig into the internals of it and patch it | 13:25 |
| Daviey | sdake: ok, using tools/build.py, seeing the same issue | 13:26 |
| sdake | with kolla not pip installed in any way? | 13:26 |
| Daviey | sdake: indeed... | 13:26 |
| sdake | if you have a modified pbr (?) could be the cause | 13:26 |
| sdake | not sure it definately works for me | 13:27 |
| Daviey | sdake: No.. i mean, a year ago i had to fix br upstream | 13:27 |
| Daviey | pbr* | 13:27 |
| sdake | and it works for the gate | 13:27 |
| sdake | maybe your mirrors are pulling from unsynced locations | 13:27 |
| Daviey | i bet that is what it is | 13:27 |
| sdake | rather unsynced CDNs | 13:28 |
| Daviey | the issue is, i've got an old version of rabbitmq... i bet we are using different mirrors | 13:28 |
| *** jrist has quit IRC | 13:28 | |
| sdake | no idea which mirror i am using | 13:28 |
| inc0 | kfox1111, what's quickest way to purge kolla-k8s? | 13:28 |
| inc0 | (apart from rebootstrapping whole vms | 13:28 |
| inc0 | ) | 13:28 |
| kfox1111 | minikube delete. | 13:29 |
| kfox1111 | :) | 13:29 |
| kfox1111 | apart from that, not a quick way. as thats a workflow thing. :/ | 13:29 |
| sdake | INFO:kolla.image.build.rabbitmq: * base: mirror.hmc.edu | 13:29 |
| sdake | INFO:kolla.image.build.rabbitmq: * epel: mirror.chpc.utah.edu | 13:29 |
| sdake | INFO:kolla.image.build.rabbitmq: * extras: mirror.pac-12.org | 13:29 |
| sdake | INFO:kolla.image.build.rabbitmq: * updates: mirror.pac-12.org | 13:29 |
| *** pbourke has quit IRC | 13:29 | |
| *** eaguilar has quit IRC | 13:29 | |
| Daviey | sdake: rabbitmq is coming from SIG or RDO? | 13:29 |
| sdake | inc0 we had a great webex session with HyperJohnGraham around kolla-kubernetes last night | 13:30 |
| sdake | we found a whole slew of things new | 13:30 |
| sdake | inc0 you might benefit from such a sessio naggain - because the job is not done on his end | 13:30 |
| sdake | or ours :) | 13:30 |
| sdake | new as in new problems | 13:30 |
| inc0 | what was the session about? | 13:30 |
| sdake | or maybe they were just old problems | 13:30 |
| sdake | it was a "deploy k8s on multinode" | 13:31 |
| sdake | 4+ hours of debug | 13:31 |
| sdake | very enlightening | 13:31 |
| sdake | i learned a whole slew of new things | 13:31 |
| kfox1111 | the docs/workflow for multinode are .... a work in progress. :) | 13:31 |
| inc0 | well, things is we don't have good way to do it yet ourselves;) | 13:31 |
| sdake | kfox1111 roger | 13:31 |
| *** eaguilar has joined #openstack-kolla | 13:31 | |
| awiddersheim | anyone able to finish reviewing https://review.openstack.org/#/c/376524/ | 13:32 |
| kfox1111 | we need a gate really. | 13:32 |
| awiddersheim | seems like it is nearly ready | 13:32 |
| inc0 | that being said, I'm close to make ansible playbook for that working | 13:32 |
| sdake | inc0 i know, that is what we LEARNED yesterday | 13:32 |
| kfox1111 | if its not tested, its broken. :) | 13:32 |
| *** pbourke has joined #openstack-kolla | 13:32 | |
| kfox1111 | inc0: have a look at what ryan's done too. there are a few reviews. | 13:32 |
| inc0 | done awiddersheim | 13:32 |
| *** schwicht has quit IRC | 13:33 | |
| sdake | not passing gate!! | 13:33 |
| sdake | dont workflow that | 13:33 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 13:33 |
| sdake | i put in a recheck on it for you awiddersheim | 13:33 |
| awiddersheim | ok thanks | 13:33 |
| sdake | awiddersheim our gate has been broken since about friday | 13:34 |
| sdake | until early this morning | 13:34 |
| sdake | because of two issues | 13:34 |
| *** imcsk8 has joined #openstack-kolla | 13:34 | |
| sdake | i wont get into details unless you really want to know but they are fixed | 13:34 |
| inc0 | sdake, it's rabbitmq | 13:34 |
| awiddersheim | how does the gate get modified? | 13:34 |
| awiddersheim | seems to be outside of the project | 13:34 |
| sdake | iinc0 that is 1 of the 2 issues | 13:34 |
| openstackgerrit | Merged openstack/kolla: Remove redundant common play and add haproxy hosts https://review.openstack.org/376524 | 13:34 |
| inc0 | awiddersheim, it's not yours, it's delorean removing rabbitmq | 13:34 |
| sdake | no delorean didn't remove rabbitmq | 13:34 |
| inc0 | well bumped version | 13:35 |
| sdake | that is 1 of 2 problems | 13:35 |
| inc0 | what's the second one? | 13:35 |
| sdake | the other problem being someone uploaded a wheel which takes precedence over tar.gz packaging in pypi | 13:35 |
| sdake | and the wheel is bust | 13:35 |
| sdake | its on dev mailing list | 13:36 |
| inc0 | none of these are problems with this patch, anyway we need to fix gates asap | 13:36 |
| sdake | we should not merge patches with red gates, even if we think its something else | 13:36 |
| sdake | rabbitmq fails very early | 13:36 |
| Daviey | inc0: which gate log are you looking ay? | 13:36 |
| Daviey | at? | 13:36 |
| inc0 | Daviey, http://logs.openstack.org/24/376524/4/check/gate-kolla-dsvm-build-centos-binary-centos-7-nv/6a4c6e6/console.html#_2016-09-30_16_33_25_128382 | 13:37 |
| sdake | threfore if awiddersheim 's patch breaks the gate, we wont know specifically his patch was the cause | 13:37 |
| sdake | because it has been workflowed | 13:37 |
| Daviey | sdake: if that is the rule, then we should make the gates voting | 13:37 |
| sdake | Daviey if only we could | 13:37 |
| sdake | Daviey i would | 13:37 |
| Daviey | inc0: RIGHT! You are seeing the same thing on that ate that i am locally | 13:38 |
| *** jrist has joined #openstack-kolla | 13:38 | |
| sdake | Daviey yo uconvince infra to that mirrors need creation | 13:38 |
| inc0 | too lat enow, we'll have to fix if it breaks, but it won't | 13:38 |
| Daviey | inc0: the version number of rabbitmq is wrong, so the path is wrong for the plugin | 13:38 |
| sdake | and then i'll submit the project config repo change for the change from non-voting to voting | 13:38 |
| inc0 | it's simple ansible and if problem would be with ansible ubuntu woyld be red too | 13:38 |
| inc0 | yeah Daviey | 13:38 |
| sdake | Daviey thus far, I've been unsuccessful | 13:38 |
| Daviey | inc0: i pushed up a change.. i don't like it | 13:38 |
| sdake | my patch for adding ust one mirror has been sitting in infra's review queue going on 2 months | 13:39 |
| inc0 | maybe quick shellscript to figure out latest version based on ls? | 13:39 |
| sdake | inc0 pretty sure Jeffrey4l_ tried that | 13:39 |
| inc0 | sdake, I don't want mirror there | 13:39 |
| inc0 | because if we make infra mirror | 13:39 |
| inc0 | gates will be green but nobody will be able to build images | 13:39 |
| inc0 | so pretty much gates will cause more hurt than help | 13:39 |
| Daviey | inc0: it is nasty, https://review.openstack.org/#/c/381104/ | 13:39 |
| sdake | i dont get the logic there | 13:39 |
| Daviey | (but no worse than deb) | 13:40 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Start testing the deployments https://review.openstack.org/380868 | 13:40 |
| sdake | but whatever, not my problem ;) | 13:40 |
| sdake | you want voting gates, you need mirrors | 13:40 |
| *** pbourke has quit IRC | 13:40 | |
| Daviey | sdake: You leaving the project? | 13:40 |
| sdake | its as simple as that | 13:40 |
| sdake | Daviey no not at all | 13:40 |
| *** srwilkers has quit IRC | 13:40 | |
| sdake | Daviey i mean sorting out mirrors is not my problem ;) | 13:40 |
| Daviey | Oh | 13:40 |
| sdake | because i can't get infra tom move on the patches | 13:40 |
| inc0 | Daviey, yeah, deb is similar | 13:40 |
| sdake | can lead horse to water | 13:40 |
| inc0 | all due to clusterer plugin | 13:41 |
| sdake | and all that | 13:41 |
| *** huikang has joined #openstack-kolla | 13:41 | |
| *** huikang has quit IRC | 13:41 | |
| Daviey | I *hate* pinning to versions.. it makes us responsible for security patching | 13:41 |
| *** huikang has joined #openstack-kolla | 13:41 | |
| sdake | indeed it is suboptimal | 13:41 |
| sdake | but it unblocks the gate | 13:41 |
| sdake | direciton to Jeffrey4l_ was a) unblock the gate | 13:41 |
| sdake | b) solve problem correctly | 13:42 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 13:42 |
| sdake | i think b is harder then people think | 13:42 |
| sdake | but never underestimate the power of persistence ;) | 13:42 |
| inc0 | unless we make logic to get correct clusterer version automatically | 13:42 |
| inc0 | we're bound to pin versions | 13:42 |
| sdake | its not so easy because the plugin has some crazy hash at the end | 13:43 |
| sdake | its not like you can curl a wildcard... | 13:43 |
| Daviey | sdake: curl | grep | 13:43 |
| sdake | atleast not that I know how | 13:43 |
| Daviey | determine version, if it is deterministic :) | 13:43 |
| sdake | it is deterministic except the hash is wack | 13:43 |
| sdake | and the y stream is incompatible | 13:43 |
| inc0 | Daviey, +1, would solve immediate issue | 13:43 |
| inc0 | alternatively we can try to ditch clusterer | 13:43 |
| sdake | the immediate issue is solved | 13:43 |
| inc0 | I know it's cool, | 13:44 |
| *** salv-orl_ has joined #openstack-kolla | 13:44 | |
| sdake | the harder more correct way is to get deb and rdo to package clusterer | 13:44 |
| Daviey | why is that hard? | 13:44 |
| *** LamT__ has joined #openstack-kolla | 13:45 | |
| sdake | then *THEY* are responsible for it | 13:45 |
| Daviey | Ubuntu did package clusterer | 13:45 |
| sdake | sweet | 13:45 |
| sdake | rdo hasn't | 13:45 |
| sdake | perhaps we should be using the ubuntu clusterer packages | 13:45 |
| kfox1111 | I'v egotten one package into rdo... stuff only goes in if you really want it to. | 13:45 |
| sdake | we need to rid ourselves of pulls from github and the like | 13:45 |
| Daviey | Yes! | 13:46 |
| Daviey | I can certainly move things in Ubuntu if needed... i'm core there | 13:46 |
| *** bjolo has joined #openstack-kolla | 13:46 | |
| sdake | i think where we are stuck is rdo | 13:46 |
| *** schwicht has joined #openstack-kolla | 13:47 | |
| *** salv-orlando has quit IRC | 13:47 | |
| sdake | and i don't have time to package a plugin unfortunately | 13:47 |
| kfox1111 | it won't get through rdo unless someone actively works on pushing it through. | 13:47 |
| *** srwilkers has joined #openstack-kolla | 13:47 | |
| kfox1111 | I don't know the process enough to do it. Never did figure it out. had to beg rdo folks for weeks to get something going. | 13:47 |
| sdake | Daviey the other angle on this problem is inc0 wants same version #s of rabbitmq for both distros | 13:48 |
| *** Pavo has quit IRC | 13:48 | |
| sdake | Daviey which means ?? not sure how you pull that one off in distro packaging | 13:48 |
| sdake | we have different distro versioned packages all over the place | 13:48 |
| sdake | i dont see why rabbitmq is all that special, minus the fact we have to deal with the plugin | 13:48 |
| sdake | kfox1111 the process is documented | 13:49 |
| sdake | however - i think we have bigger fish to fry at this point | 13:50 |
| kfox1111 | sdake: I tried folliwing it several times... the documentation is... inconsistant with reality. :) | 13:50 |
| sdake | like releasing 3.0.0 rather then bickering about details ;) | 13:50 |
| sdake | kfox1111 yup docs usually are ;) | 13:50 |
| * kfox1111 nods | 13:50 | |
| sdake | since the gate is unblocked, from my pov, the immediate pain is solved | 13:51 |
| sdake | will more pain come? yes | 13:51 |
| Daviey | I don't like it... but: | 13:51 |
| kfox1111 | I can uaually make headway through lack of technical documentation. | 13:51 |
| sdake | does someone have time to make the pain go away? don't know | 13:51 |
| Daviey | $ curl https://www.rabbitmq.com/releases/rabbitmq-server/current/ 2>/dev/null | sed 's/<[^>]*>/ /g' | grep noarch.rpm | grep -v -e suse -e asc | awk '{ print $1 }' | 13:51 |
| Daviey | rabbitmq-server-3.6.5-1.noarch.rpm | 13:51 |
| kfox1111 | political/structural documentations much harder for me to parse when its incomplete. | 13:51 |
| sdake | Daviey you mean a curl | 13:52 |
| sdake | no thats a nonstarter, that curl pulls in the wrong dep chain for errlang | 13:52 |
| Daviey | sdake: it is a curl of dog poo | 13:52 |
| sdake | whih is where all the problems are | 13:52 |
| sdake | erlang is a big pile of steaming poo - especially EPMD | 13:52 |
| kfox1111 | rbd: extraneous parameter --image-feature | 13:52 |
| sdake | i've looked at the C code for EPMD | 13:52 |
| sdake | trust me when I say its untidy | 13:52 |
| kfox1111 | is that hammer vs jewel? | 13:52 |
| sdake | and I use that word constructively :) | 13:53 |
| *** Pavo has joined #openstack-kolla | 13:53 | |
| sdake | kfox1111 unknown - google may know :) | 13:53 |
| inc0 | sdake, if we don't have same version between ubuntu and centos, we need to make grep of package version to figure out clusterer curl | 13:53 |
| inc0 | same shit | 13:53 |
| sdake | inc0 you mean y version of z version? | 13:53 |
| inc0 | both | 13:53 |
| Daviey | yeah.. i just used the wrong src url.. but that is what i was thinking | 13:53 |
| sdake | wrong | 13:53 |
| sdake | just z version | 13:53 |
| kfox1111 | inc0: even if you an make them match for a time, is probably tehy will skew at times. | 13:53 |
| sdake | rather just y version | 13:54 |
| sdake | y is a stable abi | 13:54 |
| sdake | 3.6 = any 3.6 plugin will work | 13:54 |
| inc0 | and some undeterministic hash | 13:54 |
| sdake | 3.5 = any 3.5 plugin will work | 13:54 |
| sdake | 3.5 with 3.6 plugin = hell breaks loose | 13:54 |
| sdake | the z version is irrelevant | 13:54 |
| Daviey | lets just flippin' base64 encode the plugin and put it in the Dockerfile :D | 13:54 |
| sdake | inc0 its probably determisitic, just not sure how | 13:55 |
| kfox1111 | Daviey: there is an add command. ;) | 13:55 |
| sdake | Daviey sounds feasible - opens us up to license contamination | 13:55 |
| inc0 | https://github.com/openstack/kolla/blob/master/docker/rabbitmq/Dockerfile.j2#L43 you have .z version there | 13:55 |
| inc0 | no matter whether or not plugin works, you still need to figure out path to cp it into | 13:55 |
| sdake | inc0 are you talking abou t this part: && curl -o /usr/lib/rabbitmq/lib/rabbitmq_server-3.6.5 | 13:56 |
| inc0 | yup | 13:56 |
| Daviey | kfox1111: yeah, adding base64 binaries looks less bad than adding binaries to git :) | 13:56 |
| sdake | if so, that version number is required because that is a PATH | 13:56 |
| sdake | that has nothing to do with whether hte plugin will actually work once loaded | 13:57 |
| sdake | other hten "can it be found" | 13:57 |
| inc0 | all I'm saying is that we need both, y and z | 13:57 |
| *** tonanhngo has joined #openstack-kolla | 13:57 | |
| sdake | to determine the path - yes | 13:57 |
| sdake | how to detemrien x.y.z - dunno | 13:57 |
| inc0 | can we curl -o rabbitmq binary for Newton please? | 13:58 |
| sdake | absolutely not for rdo | 13:58 |
| inc0 | not ideas, we'll need to care for security updates | 13:58 |
| sdake | do whaty ou like with mariadb ;) | 13:58 |
| sdake | sorry with ubuntu | 13:58 |
| inc0 | duh | 13:59 |
| inc0 | woes of multidistro | 13:59 |
| sdake | again the hard solution to this problem is to get rdo to package clusterer | 13:59 |
| sdake | then it will land in the natural place | 13:59 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 13:59 |
| inc0 | not in our control | 14:00 |
| sdake | maybe there is some yum magic you can do during build to encode the environment | 14:00 |
| inc0 | which means not gonna happen in N | 14:00 |
| sdake | inc0 it is in our control, we can submit such changes to rdo | 14:00 |
| sdake | ya wont happen for n for sure | 14:00 |
| sdake | but ocata is a possibility | 14:00 |
| inc0 | how about installing it from apt/you | 14:00 |
| inc0 | yum | 14:00 |
| inc0 | and checking installed version | 14:00 |
| sdake | how about we use rdo | 14:00 |
| sdake | its a system dependency | 14:00 |
| sdake | just like apache | 14:00 |
| inc0 | and shell scripts with lots of greps and curls to download/install correct clusterer? | 14:00 |
| sdake | which has different versions | 14:00 |
| sdake | inc0 i dunno just spitballing if you want ot make it futureproof | 14:01 |
| *** huikang has quit IRC | 14:01 | |
| sdake | the best way to handle it imo is just to release on the normal 45 day zstream any updates that happen | 14:01 |
| *** huikang has joined #openstack-kolla | 14:01 | |
| sdake | i doubt updates will happen in rdo for that package | 14:01 |
| sdake | dmsimard any confirmation on above speculation line? | 14:01 |
| dmsimard | sdake: I'm fighting fires right now, please be explicit don't have time to read context | 14:02 |
| *** pbourke has joined #openstack-kolla | 14:02 | |
| sdake | dmsimard will rabbitmq move from 3.6.5 to 3.6.6 | 14:02 |
| sdake | in rdo newton | 14:02 |
| sdake | at any point in the newton lifecycle | 14:02 |
| dmsimard | I don't know, why would that happen ? | 14:02 |
| dmsimard | At any point in the lifecycle? Maybe. | 14:03 |
| sdake | dmsimard it moved from 3.6.2 to 3.6.5 | 14:03 |
| dmsimard | We update components throughout stable lifecycles | 14:03 |
| sdake | dmsimard we have a plugin which depends on a path | 14:03 |
| sdake | and that path changes depending on the version of rabbitmq in use | 14:03 |
| sdake | therefore version changes are very disruptive | 14:03 |
| dmsimard | sdake: well hopefully that path no longer changes >= 3.6.5 :P | 14:03 |
| sdake | (to inc0) | 14:04 |
| sdake | dmsimard yup my speculation as well | 14:04 |
| inc0 | why wouldn't it changE? | 14:04 |
| *** eaguilar has quit IRC | 14:06 | |
| sdake | thanks for the clarification dmsimard - appreciate your time :) | 14:06 |
| *** eaguilar has joined #openstack-kolla | 14:06 | |
| dmsimard | sdake: what I would say is that before pushing updates to stable releases, the new versions are staged in a testing repository | 14:07 |
| dmsimard | so I would encourage you to use that to see those changes coming | 14:07 |
| *** eaguilar_ has joined #openstack-kolla | 14:08 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Start testing the deployments https://review.openstack.org/380868 | 14:08 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla: Updated from global requirements https://review.openstack.org/375989 | 14:10 |
| *** eaguilar has quit IRC | 14:11 | |
| dmsimard | the testing repositories are exposed exactly to give time for people to test it and report (or fix) any issues | 14:13 |
| dmsimard | That said, if centos+rdo kolla gate jobs become voting and stable sometime throughout ocata, we can pull them in RDO CI | 14:14 |
| dmsimard | so you'll have some coverage there | 14:14 |
| *** huikang has quit IRC | 14:18 | |
| *** huikang has joined #openstack-kolla | 14:18 | |
| *** zhubingbing has joined #openstack-kolla | 14:19 | |
| zhubingbing | hello guys | 14:19 |
| *** absubram has joined #openstack-kolla | 14:21 | |
| *** mark-casey1 has joined #openstack-kolla | 14:22 | |
| *** huikang has quit IRC | 14:23 | |
| sdake | sup zhubingbing | 14:26 |
| zhubingbing | hi | 14:28 |
| zhubingbing | i have rest a few days | 14:29 |
| zhubingbing | － ） | 14:29 |
| zhubingbing | sdake nice meet to you | 14:29 |
| zhubingbing | I can put in a fight tomorrow. | 14:30 |
| zhubingbing | －） | 14:30 |
| *** zhubingbing has quit IRC | 14:30 | |
| *** Marcellin__ has joined #openstack-kolla | 14:31 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla: Allow service_checks to run independently of kolla-ansible https://review.openstack.org/381161 | 14:31 |
| *** DanyC has joined #openstack-kolla | 14:33 | |
| *** lamt has joined #openstack-kolla | 14:34 | |
| sdake | say folks gotta jet for about an hour, but recall i will out of the office most of the day | 14:36 |
| sdake | bbi1hr | 14:36 |
| *** MarMat has joined #openstack-kolla | 14:36 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Start testing the deployments https://review.openstack.org/380868 | 14:43 |
| *** ldeptula has quit IRC | 14:44 | |
| openstackgerrit | Mark Duggan proposed openstack/kolla: Iscsid container fails to start on storage node when cinder is enabled https://review.openstack.org/381166 | 14:45 |
| *** ChanServ sets mode: +o inc0 | 14:46 | |
| inc0 | RC2 DEADLINE - OCT 12, 2016; Please read the documentation here: http://docs.openstack.org/developer/kolla/; Kolla IRC meetngs on Wednesdays @ 16:00 UTC - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b (old logs from #kolla http://goo.gl/VKpPzA); Barcelona summit schedule is up https://www.openstack.org/summit/barcelona-2016/summit-schedule/global-search?t=Kolla%3A | 14:47 |
| *** inc0 changes topic to "RC2 DEADLINE - OCT 12, 2016; Please read the documentation here: http://docs.openstack.org/developer/kolla/; Kolla IRC meetngs on Wednesdays @ 16:00 UTC - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b (old logs from #kolla http://goo.gl/VKpPzA); Barcelona summit schedule is up https://www.openstack.org/summit/barcelona-2016/summit-schedule/global-se" | 14:47 | |
| *** ChanServ sets mode: +o inc0 | 14:47 | |
| inc0 | hmm topic is too long | 14:47 |
| *** inc0 changes topic to "RC2 DEADLINE - OCT 12, 2016; Please read the documentation here: http://docs.openstack.org/developer/kolla/; Kolla IRC meetngs on Wednesdays @ 16:00 UTC - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b (old logs from #kolla http://goo.gl/VKpPzA); Summit schedule https://www.openstack.org/summit/barcelona-2016/summit-schedule/global-search?t=Kolla%3A" | 14:48 | |
| *** inc0 sets mode: -o inc0 | 14:49 | |
| *** diogogmt has joined #openstack-kolla | 14:50 | |
| *** lamt has quit IRC | 14:51 | |
| coolsvap | inc0: you might want to use tiny urls :) | 14:51 |
| inc0 | that is too high tech for me | 14:52 |
| * coolsvap :D | 14:52 | |
| *** DanyC has quit IRC | 14:55 | |
| openstackgerrit | Mark Duggan proposed openstack/kolla: Iscsid container fails to start on storage node when cinder is enabled https://review.openstack.org/381166 | 14:57 |
| Daviey | inc0: So how are we fixing rabbitmq TODAY? | 14:58 |
| inc0 | Daviey, gates are green now | 14:58 |
| inc0 | Jeffrey bumped ubuntu version | 14:58 |
| Daviey | inc0: hmm | 14:59 |
| inc0 | however it's a ticking bomb | 14:59 |
| Daviey | inc0: This has *broken* centos for me, https://github.com/openstack/kolla/commit/bbf9b90b061d1c53633471d1a332eaad433e9ef7 | 14:59 |
| inc0 | all it takes is another z stream update | 14:59 |
| inc0 | Daviey, duh, I'll need more caffeine for this shit | 15:00 |
| inc0 | so best idea I can think of currently - ls /usr/lib/rabbitmq/lib to get dir for curl -o | 15:00 |
| inc0 | and put clusterer there | 15:01 |
| Daviey | that works, providing clusterer is compatible | 15:01 |
| inc0 | we'll need to pin versions of rabbitmq to 3.6.z | 15:01 |
| inc0 | it will survive z-stream updates, y stream will be uglier | 15:01 |
| Daviey | (I'm *really* confused why this is working for sdake tho) | 15:02 |
| inc0 | I'm cleaning my env now, will try to build fresh in a moment | 15:03 |
| Jeffrey4l_ | hi inc0, do we have a conclusion about the rabbitmq version on different distro? | 15:03 |
| inc0 | we're just discussing this | 15:04 |
| inc0 | well, we need y to be the same | 15:04 |
| inc0 | I think we can handle z version to differ | 15:04 |
| inc0 | but y version has to be the same due to clusterer thingy | 15:04 |
| *** fguillot has quit IRC | 15:04 | |
| Jeffrey4l_ | i am on installing rabbitmq from distro repo. | 15:05 |
| Jeffrey4l_ | what's the clusterer thing? | 15:05 |
| Daviey | Jeffrey4l_: yeah, nobody likes the curl approach | 15:05 |
| Daviey | Jeffrey4l_: the plugin | 15:05 |
| inc0 | Jeffrey4l_, it's a plugin we use to build rabbitmq cluster | 15:05 |
| inc0 | it has to be curl-ed down and put in correct dir | 15:05 |
| inc0 | which is rabbitmq version specific | 15:05 |
| Jeffrey4l_ | we can install the differect culsterer for different rabbitmq y version. | 15:06 |
| Daviey | I think.. we should land my tech debt patch: https://review.openstack.org/#/c/381104/ | 15:06 |
| inc0 | well, getting *this* coded down will be signigicantly harder | 15:06 |
| Daviey | and fix it by getting clusterer in RDO | 15:06 |
| Jeffrey4l_ | inc0, Daviey could u check the PS6 in the rabbitmq fix? https://review.openstack.org/#/c/379960/6/docker/rabbitmq/Dockerfile.j2 | 15:07 |
| inc0 | well we won't get it to rdo soon and we won't get it to uca soon | 15:07 |
| Daviey | Any time we need to install via curl, it is kolla tech debt | 15:07 |
| Daviey | Jeffrey4l_: ah! | 15:08 |
| inc0 | I don't like that at all | 15:08 |
| inc0 | this just add to complexity | 15:08 |
| inc0 | (2 different y versions( | 15:08 |
| Jeffrey4l_ | but it can handle z stream change, right? | 15:08 |
| Daviey | wtf | 15:09 |
| inc0 | yeah, I'd be ok with that, but we need to have some detection code here: https://github.com/openstack/kolla/blob/master/docker/rabbitmq/Dockerfile.j2#L43 | 15:09 |
| Jeffrey4l_ | install rabbitmq from url is really bad, imo. | 15:09 |
| inc0 | even simple ls would suffice | 15:09 |
| Daviey | +1 | 15:09 |
| inc0 | and we need to pin y version | 15:09 |
| inc0 | to be future-proof | 15:09 |
| *** egonzalez90 has quit IRC | 15:09 | |
| Jeffrey4l_ | curl -o $(echo /usr/lib/rabbitmq/lib/rabbitmq_server-3.6.*)/plugins/ | 15:10 |
| Jeffrey4l_ | this will work. | 15:10 |
| Jeffrey4l_ | inc0, Daviey patch set 2 use such solution https://review.openstack.org/#/c/379960/3/docker/rabbitmq/Dockerfile.j2 | 15:11 |
| Daviey | I am really confused... | 15:13 |
| inc0 | Jeffrey4l_, are we 100% positive that it will always return single dir? | 15:13 |
| Daviey | https://review.openstack.org/#/c/379960/ landed 6 hours ago | 15:13 |
| Daviey | but it is absent from, https://github.com/openstack/kolla/blob/master/docker/rabbitmq/Dockerfile.j2 ?? | 15:13 |
| *** Serlex has quit IRC | 15:14 | |
| Jeffrey4l_ | Daviey, yes. that PS just fix the gate. we need some talk about the rabbitmq-server version. then we can take the further steps. | 15:14 |
| Daviey | oh | 15:14 |
| Daviey | Jeffrey4l_: well... i don't see why the gate is working! | 15:14 |
| Daviey | Jeffrey4l_: curent master is failing for me, due to installing older rabbitmq and the directory not existing for the plugin | 15:15 |
| Daviey | https://review.openstack.org/#/c/379960/6/docker/rabbitmq/Dockerfile.j2 Would fix it! | 15:15 |
| Jeffrey4l_ | really? what version are u installing? | 15:15 |
| Daviey | Jeffrey4l_: Master | 15:16 |
| Daviey | right now | 15:16 |
| Daviey | Jeffrey4l_: centos source | 15:16 |
| Jeffrey4l_ | are u using a old, locally repo? | 15:16 |
| Daviey | Jeffrey4l_: nope... public repos.. untouched by me | 15:16 |
| Daviey | Jeffrey4l_: i unmarked bug 1629596 as a dupe | 15:17 |
| openstack | bug 1629596 in kolla "rabbitmq image build fails with kolla master" [Undecided,New] https://launchpad.net/bugs/1629596 | 15:17 |
| Jeffrey4l_ | Daviey, could u check the build log and get which rabbitmq-version are u installing? | 15:18 |
| Daviey | Jeffrey4l_: I am still getting v3.6.2.. | 15:18 |
| Daviey | Jeffrey4l_: and the path isn't there as it is now looking for .5 | 15:18 |
| Daviey | (for the plugin) | 15:19 |
| Jeffrey4l_ | weird | 15:19 |
| Daviey | ie, https://review.openstack.org/#/c/379960/10/docker/rabbitmq/Dockerfile.j2 broke me | 15:19 |
| Jeffrey4l_ | Daviey, from your log, http://paste.openstack.org/show/583796/ | 15:20 |
| Jeffrey4l_ | you are download clusterer to folder rabbitmq_server-3.6.2 | 15:20 |
| Daviey | Jeffrey4l_: right | 15:20 |
| Jeffrey4l_ | if you still installing rabbitm 3.6.2, you won't get the error. | 15:20 |
| Daviey | (That isn't my paste) | 15:20 |
| Jeffrey4l_ | ok. | 15:21 |
| Daviey | Jeffrey4l_: my log, https://bugs.launchpad.net/kolla/+bug/1611655/comments/5 | 15:21 |
| openstack | Launchpad bug 1611655 in kolla "kolla.image.build.rabbitmq curl No such directory" [Critical,Fix released] | 15:21 |
| Jeffrey4l_ | anyway, ping the rabbitmq versio in the distro is not a good idea. we hit rabbitmq server issue on centos when we force installing rabbitmq 3.5.7 . | 15:22 |
| Daviey | Jeffrey4l_: i entered the docker container, and /usr/lib/rabbitmq/lib/rabbitmq_server-3.6.5 did NOT exist.. but .2 did | 15:22 |
| Daviey | Jeffrey4l_: I am totally against pinnng.. it makes us responsible for security patching. That alone is enough | 15:23 |
| Jeffrey4l_ | Daviey, in centos, the repo is using mirrorurl, you may get a old repo url when centos determine the fast repo. | 15:24 |
| Jeffrey4l_ | i will push a PS to handle the z stream change for rabbitmq, which can fix you issue also. | 15:25 |
| Daviey | Jeffrey4l_: okay, if you want me to try it.. can you ping me a changeset url | 15:26 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla: Fix sahara endpoint url https://review.openstack.org/381188 | 15:26 |
| Jeffrey4l_ | Daviey, np. | 15:27 |
| *** haplo37__ has joined #openstack-kolla | 15:29 | |
| HyperJohnGraham | good morning all | 15:29 |
| HyperJohnGraham | And thanks for all the help yesterday | 15:29 |
| inc0 | HyperJohnGraham, kolla-k8s things?;) | 15:31 |
| *** huikang has joined #openstack-kolla | 15:32 | |
| HyperJohnGraham | yes we had a productive webex | 15:32 |
| HyperJohnGraham | in the end we wiped the config and tweaked the ceph config until it worked | 15:33 |
| HyperJohnGraham | now need to re-run the kolla-kubernetes bits | 15:33 |
| HyperJohnGraham | setting up a local ceph instance outside kk was a critical missing step | 15:34 |
| *** vhosakot has joined #openstack-kolla | 15:34 | |
| inc0 | HyperJohnGraham, yeah, well, kolla-ansible knows how to do it:) | 15:35 |
| Jeffrey4l_ | Daviey, fix is here https://review.openstack.org/#/c/381197/ | 15:38 |
| inc0 | that actually looks good Jeffrey4l_ | 15:40 |
| inc0 | that'll lose any requirement on .z version pinning | 15:41 |
| Jeffrey4l_ | yes. but i still do not like install rabbitmq-server from url for ubuntu. and this also pin the rabbitmq-server version. | 15:43 |
| Jeffrey4l_ | inc0, could u review this https://review.openstack.org/379106 | 15:48 |
| *** Pavo has quit IRC | 15:48 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Start testing the deployments https://review.openstack.org/380868 | 15:51 |
| *** b_bezak_ has quit IRC | 15:52 | |
| *** harlowja_at_home has joined #openstack-kolla | 15:53 | |
| *** Pavo has joined #openstack-kolla | 15:54 | |
| inc0 | Jeffrey4l_, done | 15:54 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 15:57 |
| openstackgerrit | Merged openstack/kolla: Fix ironic failed https://review.openstack.org/379106 | 15:58 |
| *** huikang has quit IRC | 15:58 | |
| *** huikang has joined #openstack-kolla | 15:59 | |
| *** tonanhngo has quit IRC | 15:59 | |
| *** shardy has quit IRC | 16:00 | |
| *** fguillot has joined #openstack-kolla | 16:00 | |
| sdake | e3qSUP sup | 16:03 |
| Jeffrey4l_ | got it. thanks. | 16:03 |
| sdake | sup folks | 16:03 |
| *** huikang_ has joined #openstack-kolla | 16:03 | |
| *** sean-k-mooney has joined #openstack-kolla | 16:03 | |
| *** huikang has quit IRC | 16:03 | |
| sdake | looks good Jeffrey4l_ | 16:04 |
| sdake | i doubt rdo will release 3.7 midstream | 16:04 |
| sdake | if that happens we will just have to suck it up | 16:04 |
| sdake | but this keeps us from playing catchup on the zstream (security+bugfix releasees) | 16:04 |
| MarMat | Hi guys, I'm wondering how will kolla handle actual pycparser break https://github.com/pyca/cryptography/issues/3187 ? Is a pin of pycparser to 2.13 needed while actually installed 2.14 from pip is broken? | 16:05 |
| sdake | MarMat we have fixed that | 16:05 |
| sdake | MarMat also global requirements has made a change should hit our queue in about 1 hr or so | 16:05 |
| Jeffrey4l_ | MarMat, check this https://review.openstack.org/380929 | 16:06 |
| MarMat | sdake oh, ok, thanks, seems you had a busy weekend. | 16:06 |
| Jeffrey4l_ | sdake, i do not think rdo will release 3.7 in newton. if so, he may break lots of thing ( deploy tool, upgrade progress etc) | 16:07 |
| sdake | Jeffrey4l_ agree | 16:07 |
| *** matrohon has quit IRC | 16:09 | |
| *** eaguilar_ has quit IRC | 16:12 | |
| *** msimonin has quit IRC | 16:15 | |
| openstackgerrit | Nikita Gerasimov proposed openstack/kolla: Change mysql-check in HAProxy to post MySQL 4.1 https://review.openstack.org/381218 | 16:18 |
| sean-k-mooney | MarMat: regarding the pycparser issue i think the broken wheel package has also been removed so without any workaround the issue should now be resolved. | 16:20 |
| MarMat | sean-k-mooney fine, I saw that error in my nightly run... | 16:21 |
| sean-k-mooney | MarMat: well i think the package was just pulled down in the last 30mins or so so you might hit it if you are behind a cahcing proxy or pypi mirior so it was a valid question | 16:22 |
| *** huikang_ has quit IRC | 16:26 | |
| *** huikang has joined #openstack-kolla | 16:26 | |
| *** sdake has quit IRC | 16:30 | |
| *** huikang has quit IRC | 16:31 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 16:31 |
| *** sdake has joined #openstack-kolla | 16:34 | |
| *** athomas has quit IRC | 16:36 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla: Fix sahara endpoint url https://review.openstack.org/381188 | 16:37 |
| inc0 | kfox1111, kolla-kubernetes command will use /etc/kolla/k8s.yml config? | 16:38 |
| inc0 | nvm | 16:39 |
| inc0 | my bad;) | 16:39 |
| sdake | Jeffrey4l_ that pyc patch can be reverted | 16:39 |
| sdake | the cryptography one | 16:39 |
| Jeffrey4l_ | the wheel is removed? | 16:39 |
| sdake | yup | 16:39 |
| sdake | see mailing list for confirmation | 16:40 |
| Jeffrey4l_ | cool. | 16:40 |
| * Jeffrey4l_ is checking | 16:40 | |
| sdake | its also removed in the cache infra maintains (which are used for ubuntu builds) | 16:40 |
| *** msimonin has joined #openstack-kolla | 16:40 | |
| *** harlowja_still_a has joined #openstack-kolla | 16:40 | |
| *** sbezverk has quit IRC | 16:42 | |
| *** harlowja_at_home has quit IRC | 16:44 | |
| *** DanyC has joined #openstack-kolla | 16:44 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Revert "Fix pycparser wheel package issue" https://review.openstack.org/381237 | 16:44 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 16:44 |
| *** gfhellma has joined #openstack-kolla | 16:45 | |
| openstackgerrit | Merged openstack/kolla: Handle z stream change for rabbitmq-server https://review.openstack.org/381197 | 16:46 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Start testing the deployments https://review.openstack.org/380868 | 16:46 |
| *** harlowja_still_a has quit IRC | 16:46 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Add hexdump package for ironic-conductor container https://review.openstack.org/381238 | 16:46 |
| *** tonanhngo has joined #openstack-kolla | 16:48 | |
| *** DanyC has quit IRC | 16:48 | |
| *** tonanhngo has quit IRC | 16:50 | |
| *** tonanhngo has joined #openstack-kolla | 16:50 | |
| *** HyperJohnGraham has quit IRC | 16:51 | |
| *** sbezverk has joined #openstack-kolla | 16:52 | |
| *** msimonin has quit IRC | 16:52 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: [wip] fix ironic inspector https://review.openstack.org/380774 | 16:53 |
| sdake | Jeffrey4l_ you left a -2 on https://review.openstack.org/#/c/380774/ without rationale | 16:54 |
| Jeffrey4l_ | sdake, it is still wip. | 16:55 |
| Jeffrey4l_ | will fix some issue later | 16:55 |
| sdake | is it because your the author and waiting for it not to be wip? | 16:55 |
| *** msimonin has joined #openstack-kolla | 16:55 | |
| *** sbezverk_ has joined #openstack-kolla | 17:01 | |
| *** sbezverk has quit IRC | 17:04 | |
| *** unicell has quit IRC | 17:05 | |
| *** eaguilar has joined #openstack-kolla | 17:06 | |
| sdake | rbergeron you about | 17:10 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: DO_NOT_MERGE: Use better URL for China https://review.openstack.org/329060 | 17:11 |
| *** salv-orl_ has quit IRC | 17:12 | |
| rbergeron | sdake:sorta | 17:12 |
| *** ankush has quit IRC | 17:17 | |
| *** salv-orlando has joined #openstack-kolla | 17:20 | |
| sdake | burnin down the house! | 17:23 |
| *** lrensing has joined #openstack-kolla | 17:24 | |
| openstackgerrit | Merged openstack/kolla: Iscsid container fails to start on storage node when cinder is enabled https://review.openstack.org/381166 | 17:35 |
| sean-k-mooney | pbourke: regarding https://bugs.launchpad.net/kolla/+bug/1629237 removing dump-init has no effect for me. i still have the same issue with the ceph_mons hanging | 17:35 |
| openstack | Launchpad bug 1629237 in kolla "Ceph monitors not responding (Error connecting to cluster: TimedOut)" [High,Confirmed] - Assigned to Paul Bourke (pauldbourke) | 17:35 |
| g3ek | who g3ek | 17:35 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Add hexdump package for ironic-conductor container https://review.openstack.org/381238 | 17:36 |
| sean-k-mooney | pbourke: did you make any other changes? | 17:36 |
| *** huikang has joined #openstack-kolla | 17:37 | |
| *** eaguilar_ has joined #openstack-kolla | 17:37 | |
| inc0 | sean-k-mooney, ad ceph | 17:38 |
| inc0 | make sure it's not networking | 17:38 |
| *** sdake has quit IRC | 17:39 | |
| inc0 | maybe after upgrade to jewel something bad have happened in this front? | 17:39 |
| sean-k-mooney | inc0: its not a netwoking issue as i can use netcat to connect to the port from insde the boostrap container and if i exec into the mon container ceph -s still hangs | 17:39 |
| inc0 | ceph -s is talking over net even from whithin container | 17:40 |
| sean-k-mooney | inc0: i could revert back form jewel. this is a clean install e.g. not doing an upgrade. is ceph tested in the gate? | 17:40 |
| *** Pavo has quit IRC | 17:40 | |
| *** unicell has joined #openstack-kolla | 17:40 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: install pxelinux for ironic_pxe https://review.openstack.org/380350 | 17:40 |
| inc0 | it's not, but we did test several scenerios of ceph jewel on all major distros | 17:41 |
| *** eaguilar has quit IRC | 17:41 | |
| sean-k-mooney | inc0: using the nodes local ip hangs ceph -s -m 192.168.11.1 using ceph -s -m 127.0.0.1 different output but still does not work | 17:42 |
| sean-k-mooney | ceph -s -m 127.0.0.1 | 17:43 |
| sean-k-mooney | 2016-10-03 18:42:31.758336 7f20e0104700 0 -- :/3647168411 >> 127.0.0.1:6789/0 pipe(0x7f20dc05a410 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f20dc05f2d0).fault | 17:43 |
| sean-k-mooney | 2016-10-03 18:42:34.758515 7f20cbfff700 0 -- :/3647168411 >> 127.0.0.1:6789/0 pipe(0x7f20d0000cc0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f20d0002000).fault | 17:43 |
| inc0 | ceph -s from OSD container works? | 17:44 |
| *** sdake has joined #openstack-kolla | 17:44 | |
| sean-k-mooney | inc0: its failing bootstrapping the osds do there are none to check from | 17:44 |
| inc0 | what's in your ceph.conf then? | 17:45 |
| sean-k-mooney | inc0: ceph.conf http://paste.openstack.org/show/584055/ | 17:46 |
| inc0 | you have 6 mons? | 17:47 |
| sean-k-mooney | yes | 17:47 |
| sean-k-mooney | all are failing | 17:47 |
| inc0 | try to set up odd number plz | 17:47 |
| sean-k-mooney | ok i will have 9 in the end so ill drop down to 3 and test | 17:48 |
| sean-k-mooney | would an even number of mons break it | 17:48 |
| inc0 | not sure | 17:48 |
| inc0 | I know mons have some sort of quorum mechanism | 17:48 |
| sean-k-mooney | yes it is acatully the line that check the quorum status that is failing int the osd bootstrap | 17:49 |
| gmmaha | inc0: sean-k-mooney: I havent heard of even no. of mons causing a cluster to fail.. will be curious if that is the case. | 17:53 |
| gmmaha | and would like to know if that is truly the problem here | 17:53 |
| *** salv-orlando has quit IRC | 17:53 | |
| *** strigazi is now known as strigazi_AFK | 17:53 | |
| *** huikang has quit IRC | 17:54 | |
| sean-k-mooney | gmmaha: im just rerunning the deploy now so ill let you know shortly how that turns out | 17:54 |
| gmmaha | thanks sean-k-mooney | 17:54 |
| sean-k-mooney | gmmaha: inc0 yes if i reduce for 6 to 3 it passes the failing step. | 17:56 |
| *** Pavo has joined #openstack-kolla | 17:56 | |
| gmmaha | ohh.. thats interesting. | 17:56 |
| inc0 | :) | 17:56 |
| gmmaha | sean-k-mooney: can you point me to the step taht is failing in the deploy? | 17:56 |
| gmmaha | inc0: ^ | 17:57 |
| inc0 | gmmaha, osd bootstrapping | 17:57 |
| sean-k-mooney | yes too second ill pull it up on github | 17:57 |
| inc0 | but basically it was ceph-mon issue | 17:57 |
| inc0 | steps to reproduce: deploy even number of ceph mons, docker exec -it ceph_mon ceph -s | 17:57 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Start testing the deployments https://review.openstack.org/380868 | 17:57 |
| *** Marcellin__ has quit IRC | 17:57 | |
| sean-k-mooney | inc0: i also have one local change which was disableing dumb-init | 17:58 |
| sean-k-mooney | when the deploy finishes ill rebuild the ceph images without that change and test it also to see if that is needed. | 17:59 |
| inc0 | sean-k-mooney, try without it, I haven't seen any issues with dumb init | 17:59 |
| inc0 | k | 17:59 |
| *** Pavo has quit IRC | 17:59 | |
| * gmmaha goes to try building a ceph cluster with even mons | 18:01 | |
| sean-k-mooney | gmmaha: it was this line that was failing in the osd bootstrap https://github.com/openstack/kolla/blob/master/docker/ceph/ceph-osd/extend_start.sh#L18 | 18:01 |
| *** sdake has quit IRC | 18:01 | |
| gmmaha | sean-k-mooney: thanks.. checking it now | 18:01 |
| sean-k-mooney | gmmaha: as inc0 said the issue is related to the fact that the mons did not reach a quorum with an even number of mons | 18:02 |
| gmmaha | sean-k-mooney: right, but i didn't know that the cluster would actaully fail. the quorum is supposed to happen with even no. of mons (theoretically, per the docs) | 18:02 |
| inc0 | btw sean-k-mooney ... 9 mons? how many OSDs are you going to deploy? | 18:03 |
| sean-k-mooney | inc0: 9. im setting up 9 node each running all of openstack so that we can should down indevigual nodes for maintenense without downtime | 18:03 |
| sean-k-mooney | inc0: its proably overkill | 18:04 |
| inc0 | yeah... | 18:04 |
| inc0 | might cause problems | 18:04 |
| inc0 | especially on galera side | 18:04 |
| inc0 | too big of a cluster isn't always good | 18:04 |
| inc0 | but interesting experiment | 18:04 |
| sean-k-mooney | how many controlser was tested in osic? | 18:05 |
| inc0 | well, 3 controllers successfully run 300+ node openstack | 18:05 |
| inc0 | dedicated database nodes tho | 18:05 |
| sean-k-mooney | im putting together this cluser for our team to use internanly for openstack dev | 18:05 |
| *** Pavo has joined #openstack-kolla | 18:06 | |
| sean-k-mooney | we have about 40-50 servers over all. the initall 9 nodes will be effectivly all in one nodes but in a cluster. i will be adding oters over time as compute nodes | 18:06 |
| inc0 | I think 3 aio nodes would still be ok | 18:07 |
| inc0 | just limit ram/cpu allocation for "controllers" | 18:07 |
| inc0 | I'm also playing with idea of colocated ceph osd and compute | 18:07 |
| *** eaguilar_ has quit IRC | 18:09 | |
| sean-k-mooney | inc0: if i had 3 controler could i also have the osds on the computes by just adjusting the inventory file so that the rados gateways and mons were on the contoler but osds on all nodes? | 18:09 |
| sean-k-mooney | i need to spread out the osd to be able to disable any one node with out issue using erasure code pools but i really dont need to have 9 controlers 3 woudl be fine. | 18:11 |
| inc0 | yeah | 18:11 |
| inc0 | however if you'll use erasure coding, keep eye for CPU consumption | 18:11 |
| inc0 | it's cool stuff, but cpu hungry | 18:11 |
| sean-k-mooney | is it muti threaded well? | 18:12 |
| inc0 | I'd assume | 18:12 |
| inc0 | I don't know this space very well | 18:12 |
| sean-k-mooney | im using earsure coding at home but i have only 2 cores on my ceph box and yes it is not very happy on the cpu front | 18:12 |
| sean-k-mooney | these systems have 48 cores so it should work alittle better | 18:13 |
| *** daneyon has joined #openstack-kolla | 18:13 | |
| *** eaguilar has joined #openstack-kolla | 18:14 | |
| sean-k-mooney | inc0: actully is there any benifit to have mons coloated with osds or would i be better seperating the control nodes entirely? | 18:14 |
| *** HyperJohnGraham has joined #openstack-kolla | 18:15 | |
| *** msimonin has quit IRC | 18:15 | |
| gmmaha | sean-k-mooney: inc0: my test setup worked with 4 mons. http://paste.openstack.org/show/584062/ Granted i am ruunning master code, but essentially it should work | 18:15 |
| inc0 | well, benefit would be that you can use storage in control nodes too | 18:15 |
| inc0 | interesting | 18:15 |
| inc0 | so 4 works, 3 works but 6 doesnt | 18:16 |
| inc0 | ? | 18:16 |
| sean-k-mooney | that was me just being bad at english i ment to type for mons not 4 that was a froidien slip of typeing | 18:16 |
| sean-k-mooney | oh wait i read half of my last messange and half of gmmaha';s | 18:17 |
| sean-k-mooney | ignore me | 18:17 |
| *** daneyon has quit IRC | 18:18 | |
| inc0 | it's late and cold in Ireland, you're excused | 18:18 |
| inc0 | drink beer, or tea, or beer with tea | 18:18 |
| sean-k-mooney | yes actully i have been siping camamile tea but that probably isnt helping | 18:18 |
| *** gfidente has quit IRC | 18:19 | |
| inc0 | well, it's good for sleep | 18:19 |
| *** Pavo has quit IRC | 18:19 | |
| inc0 | again, late, cold... | 18:19 |
| sean-k-mooney | true am stacking failed on heat. ill kick off the depoy again tomorow and test with master conatienrs also | 18:20 |
| sean-k-mooney | inc0: thanks i totally did not think about the fact i was using 6 mons. | 18:21 |
| *** sean-k-mooney is now known as sean-k-mooneyAFK | 18:21 | |
| *** msimonin has joined #openstack-kolla | 18:22 | |
| *** bmace_ has quit IRC | 18:22 | |
| gmmaha | have a nice night sean-k-mooneyAFK | 18:23 |
| *** Pavo has joined #openstack-kolla | 18:23 | |
| *** Pavo has quit IRC | 18:24 | |
| *** mewald has joined #openstack-kolla | 18:27 | |
| *** Pavo has joined #openstack-kolla | 18:28 | |
| *** mewald1 has joined #openstack-kolla | 18:30 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Start testing the deployments https://review.openstack.org/380868 | 18:31 |
| *** mewald has quit IRC | 18:31 | |
| *** harlowja has quit IRC | 18:38 | |
| *** harlowja has joined #openstack-kolla | 18:39 | |
| *** eaguilar has quit IRC | 18:40 | |
| *** eaguilar has joined #openstack-kolla | 18:42 | |
| *** HyperJohnGraham has quit IRC | 18:45 | |
| inc0 | hmm I wonder why bootstrapping mariadb takes this long for kolla-k8s | 18:46 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 18:48 |
| *** HyperJohnGraham_ has joined #openstack-kolla | 18:49 | |
| *** jax3242 has quit IRC | 18:49 | |
| *** HyperJohnGraham_ is now known as HyperJohnGraham | 18:49 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Start testing the deployments https://review.openstack.org/380868 | 18:57 |
| *** msimonin has quit IRC | 18:59 | |
| *** msimonin has joined #openstack-kolla | 18:59 | |
| inc0 | sbezverk_, ping | 19:00 |
| kfox1111 | inc0: shouldn't take very long. | 19:01 |
| inc0 | [ERROR] mysqld: Can't lock aria control file '/var/lib/mysql/aria_log_control' for exclusive use, error: 37. Will retry for 30 seconds << anyone seen anything like this? | 19:01 |
| kfox1111 | some number of seconds. | 19:01 |
| kfox1111 | weird. yeah, never seen that before. | 19:01 |
| inc0 | checking nfs stuf | 19:01 |
| inc0 | f | 19:01 |
| kfox1111 | ah. could be. | 19:02 |
| kfox1111 | never tried mysql on top of nfs before. | 19:02 |
| kfox1111 | may need an nfs lock daemon, or to turn off some locking in the config file. | 19:03 |
| *** msimonin has quit IRC | 19:03 | |
| inc0 | well thing is, nothing else is supposed to access it right? | 19:05 |
| kfox1111 | yeah. | 19:06 |
| *** haplo37__ has quit IRC | 19:06 | |
| *** msimonin has joined #openstack-kolla | 19:18 | |
| *** awidders_ has joined #openstack-kolla | 19:22 | |
| *** awiddersheim has quit IRC | 19:22 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Start testing the deployments https://review.openstack.org/380868 | 19:28 |
| *** salv-orlando has joined #openstack-kolla | 19:30 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 19:34 |
| *** salv-orl_ has joined #openstack-kolla | 19:44 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Start testing the deployments https://review.openstack.org/380868 | 19:45 |
| *** HyperJohnGraham_ has joined #openstack-kolla | 19:46 | |
| *** david-lyle has joined #openstack-kolla | 19:47 | |
| *** salv-orlando has quit IRC | 19:47 | |
| *** mewald1 has left #openstack-kolla | 19:47 | |
| *** eaguilar has quit IRC | 19:49 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 19:49 |
| *** msimonin has quit IRC | 19:49 | |
| *** msimonin1 has joined #openstack-kolla | 19:50 | |
| *** bjolo has quit IRC | 19:51 | |
| *** eaguilar has joined #openstack-kolla | 19:54 | |
| HyperJohnGraham | sbezverk_: did you get a chance to look at things this morning we are planing to reconvene ~5pm pst | 19:54 |
| *** tonanhngo has quit IRC | 19:57 | |
| *** HyperJohnGraham has quit IRC | 20:02 | |
| *** rodrigo_pereira has joined #openstack-kolla | 20:05 | |
| *** sdake has joined #openstack-kolla | 20:05 | |
| sdake | anyone blocked? | 20:06 |
| sdake | i am here for a short while | 20:06 |
| *** b_bezak has joined #openstack-kolla | 20:08 | |
| HyperJohnGraham_ | sdake my brain hurts ... can you unblock that ? | 20:09 |
| sdake | HyperJohnGraham_ - i'd like to help wit hthat; we have a dealdine of aug 12 for our 3.0.0 | 20:10 |
| sdake | and if I don't deliver, i'm fired | 20:10 |
| HyperJohnGraham_ | :) | 20:10 |
| sdake | well not really i already elected not to run :) | 20:10 |
| sdake | but still want to finish the job here | 20:10 |
| sdake | then we can get back to you | 20:10 |
| sdake | you have my word on that | 20:10 |
| sdake | assuming i can drag sbezverk_ and kfox1111 into another 1-2 hr webex after they sort out the problems we found yesterday | 20:11 |
| HyperJohnGraham_ | I am heading off to meet Vernor Vinge :) | 20:11 |
| sdake | btw, yesterday = super high value for me | 20:11 |
| sdake | i appreciate it :) | 20:11 |
| HyperJohnGraham_ | me to ! | 20:11 |
| sdake | i owe you one | 20:11 |
| HyperJohnGraham_ | ok ill be back later ... | 20:11 |
| sdake | i bet vernor is an interesting cat | 20:11 |
| sdake | you will hae to tell us about it :) | 20:11 |
| HyperJohnGraham_ | will do | 20:11 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla: Updated from global requirements https://review.openstack.org/375989 | 20:11 |
| * HyperJohnGraham_ gathers his AI about him | 20:12 | |
| sdake | inc0 are you about | 20:12 |
| *** MartinsG has joined #openstack-kolla | 20:13 | |
| inc0 | o7 | 20:13 |
| sdake | nice | 20:13 |
| sdake | inc0 are we in agreement or disagreement re deprecation of heka | 20:13 |
| inc0 | I think we deprecate it | 20:14 |
| inc0 | afaik | 20:14 |
| sdake | so disagreement | 20:14 |
| inc0 | you don't want to deprecate it? | 20:14 |
| sdake | i do | 20:14 |
| sdake | but i dont think we have enough time to present a migration path for operators on the oeprator mailing list prior to he 12th | 20:14 |
| inc0 | sdake, migration path == upgrade kolla-n to kolla-o | 20:15 |
| sdake | if you can deliver a migration apth to the operators list and get their buyin (or a we don't care works) then lets do it | 20:15 |
| inc0 | n is heka. | 20:15 |
| sdake | inc0 can you take that on | 20:15 |
| inc0 | if we can't deliver migration path we can't deprecate it | 20:15 |
| inc0 | but it should be seamless | 20:15 |
| sdake | the operator list discussion | 20:15 |
| MartinsG | Hi, guys! Need some help. Trying to deploy kolla in multinode environment. Having a task failed: "Starting heka container" with error: msg: TypeError("'NoneType' object is not iterable",). Do you have any idea how to fix it or what should I check? Thanks a lot! | 20:15 |
| sdake | MartinsG yes | 20:15 |
| sdake | MartinsG are you using stable/mitaka? | 20:16 |
| MartinsG | Yes, 2.0.2 | 20:16 |
| sdake | marting run pip show ansible | 20:16 |
| sdake | MartinsG | 20:16 |
| inc0 | sdake, well, I'm ok to add warning that heka will disapear in Ocata | 20:16 |
| MartinsG | Name: ansible Version: 1.9.6 | 20:16 |
| sdake | run pip show kolla as well | 20:16 |
| sdake | the operator list must buy in as well | 20:16 |
| MartinsG | Name: kolla Version: 2.0.2 | 20:17 |
| sdake | that is written right in the requirements | 20:17 |
| sdake | sudo pip remove kolla | 20:17 |
| sdake | MartinsG please follow the documentation for "eval workflow" | 20:17 |
| sdake | oh wait | 20:17 |
| sdake | did you follow that workflow or do you have a git checkout? | 20:17 |
| sdake | inc0 i am protecting you and the project from ahving to wait two cycles for that PMT | 20:18 |
| MartinsG | Hmm, that's strange, I followed the workflow in the quick start guide and in the multinode guide as well. Installed kolla from pip and ansible from pip as well (with version <2.0 specified). | 20:18 |
| *** tonanhngo has joined #openstack-kolla | 20:18 | |
| sdake | so you ran sudo pip install kolla? | 20:18 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla: heka deprecation https://review.openstack.org/381293 | 20:18 |
| *** b_bezak has quit IRC | 20:18 | |
| MartinsG | Yes, I ran pip install kolla | 20:18 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla: Heka deprecation https://review.openstack.org/381293 | 20:19 |
| sdake | marting wierd error you got there | 20:19 |
| inc0 | sdake, none of operators replied on mailing list | 20:19 |
| sdake | MartinsG not what i expected to see - i expected you hd ansible 2.z installed | 20:19 |
| inc0 | either way | 20:19 |
| sdake | inc0 link to mailing list link? | 20:19 |
| sdake | it must be on the operator list | 20:19 |
| sdake | its written right i n the requirements | 20:20 |
| sdake | read the requirements | 20:20 |
| sdake | operators dont always follow openstack-dev, they follow openstack-operators (which is a list all ptls should be onimo :) | 20:20 |
| sdake | would you prefer i write the message to operator list? | 20:21 |
| sdake | or form one for you to write and defend ;) | 20:21 |
| inc0 | sdake, I can write it, issue is, it's 3 months window right? | 20:21 |
| sdake | nope | 20:21 |
| MartinsG | sdake: kolla-ansible prechecks required me to have ansible <2.0, so that's what I did. http://docs.openstack.org/developer/kolla/quickstart.html confirms, that stable branch needs <2.0.0. Am I missing anything? | 20:21 |
| sdake | right if we deprecate now, 3 month window | 20:21 |
| sdake | if we depercate in ocata, one cycle | 20:21 |
| sdake | i.e. pike is where it can be removed | 20:22 |
| sdake | deprecating now would be better | 20:22 |
| sdake | but we need to follow that operator requirement | 20:22 |
| *** tonanhngo has quit IRC | 20:23 | |
| *** tonanhngo has joined #openstack-kolla | 20:27 | |
| *** jax3242 has joined #openstack-kolla | 20:27 | |
| *** Pavo has quit IRC | 20:27 | |
| *** HyperJohnGraham has joined #openstack-kolla | 20:28 | |
| *** david-lyle has quit IRC | 20:30 | |
| *** Pavo has joined #openstack-kolla | 20:32 | |
| *** david-lyle has joined #openstack-kolla | 20:35 | |
| *** Jeffrey4l__ has joined #openstack-kolla | 20:35 | |
| *** Jeffrey4l_ has quit IRC | 20:35 | |
| rodrigo_pereira | hi, i'm trying run the kolla on virtual machine using the ansible 1.9.6 and the branch stable/mitaka of the kolla, and the deploy fail on the horizon task. What can it be? | 20:38 |
| inc0 | rodrigo_pereira, what is the error? | 20:42 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla: Heka deprecation https://review.openstack.org/381293 | 20:43 |
| *** david-lyle has quit IRC | 20:44 | |
| sdake | MartinsG how did you install ansible 1.9? | 20:45 |
| sdake | MartinsG the way to install ansible1.9 is sudo yum remove ansible | 20:45 |
| sdake | martinsG sudo pip install ansible==1.9.4 | 20:46 |
| *** signed8bit has joined #openstack-kolla | 20:46 | |
| *** signed8bit has quit IRC | 20:46 | |
| rodrigo_pereira | inc0, the task is the creation of the _member_ role, and the error message is "The module os_keystone_role was not found in configured module paths. ..." | 20:46 |
| *** inc0 has quit IRC | 20:46 | |
| MartinsG | sdake: that's exactly what I did | 20:46 |
| sdake | MartinsG i'm at a bit of a loss | 20:47 |
| sdake | MartinsG how about this | 20:47 |
| sdake | what distro are you on? | 20:47 |
| MartinsG | CentOS | 20:48 |
| sdake | ok - sudo pip remove ansible | 20:48 |
| sdake | sudo yum uninstall ansible | 20:48 |
| sdake | sudo yum remove ansible | 20:48 |
| sdake | sudo pip uninstall ansible | 20:48 |
| *** ccesario has quit IRC | 20:48 | |
| sdake | so lets start with nothign installed :) | 20:49 |
| sdake | then once that is done | 20:49 |
| sdake | sudo rm -rf /usr/lib/python2.7/site-packages/ansible* | 20:49 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Start testing the deployments https://review.openstack.org/380868 | 20:52 |
| sdake | MartinsG when you get done with all that run sudo yum install ansible1.9 | 20:53 |
| MartinsG | sdake: Oh, thanks, didn't know there was a 1.9 version of ansible available in repos. I will do it soon. | 20:54 |
| sdake | make sure ot remove the cruft first | 20:54 |
| sdake | or you end up with pyc/pyo files (precompiled stuff) which mucks up the execution | 20:54 |
| *** daneyon has joined #openstack-kolla | 20:55 | |
| *** lrensing has quit IRC | 20:56 | |
| *** bmace has joined #openstack-kolla | 20:58 | |
| *** vinkman has joined #openstack-kolla | 20:59 | |
| *** daneyon has quit IRC | 21:00 | |
| *** dwalsh has quit IRC | 21:03 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 21:10 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Start testing the deployments https://review.openstack.org/380868 | 21:15 |
| *** openstackgerrit has quit IRC | 21:19 | |
| *** openstackgerrit has joined #openstack-kolla | 21:19 | |
| *** david-lyle has joined #openstack-kolla | 21:19 | |
| *** schwicht has quit IRC | 21:20 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 21:28 |
| *** eaguilar has quit IRC | 21:29 | |
| *** diogogmt has quit IRC | 21:32 | |
| kfox1111 | sdake: I'm doing it together with the gate change for 2 reasons. 1, to ensure I get the right thing committed. It took several tries to get the container bits right, and it didn't show up until it was tested. and 2, to prevent regressions. not sure we want to accept much untested code. | 21:36 |
| *** tonanhngo has quit IRC | 21:36 | |
| sdake | wfm | 21:36 |
| sdake | say as much in the review | 21:37 |
| sdake | fix the nitpick | 21:37 |
| *** msimonin1 has quit IRC | 21:37 | |
| sdake | and i'll +2 | 21:37 |
| sdake | no need to respond in irc, i stay on top of reviews i involve myself in | 21:37 |
| sdake | unless its a pressing emergency | 21:37 |
| kfox1111 | k. yeah. still trying to get the thing to work. its close I think, but its fighting.... | 21:37 |
| kfox1111 | the docker hub being on the fritz isn't helping. :/ | 21:37 |
| kfox1111 | (need that cache... :) | 21:37 |
| sdake | would like answers to my questions, although that isn't mandatory - just for my learning experience | 21:37 |
| kfox1111 | yup. will do | 21:38 |
| sdake | the more i learn the better i can review | 21:38 |
| sdake | going forward after my ptl responsibilities complete, i want to review kolla-kubernetes and kolla-ansible equally | 21:38 |
| sdake | or as equally as feasible ;) | 21:38 |
| sdake | i stay awa yfrom reviews i dont understand :) | 21:39 |
| kfox1111 | nice. :) | 21:39 |
| sdake | and kolla-kubernetes is a bit unknown to me | 21:39 |
| sdake | as you could tell rom last nights call with john | 21:39 |
| *** inc0 has joined #openstack-kolla | 21:41 | |
| kfox1111 | its in a lot of flux right now too. hard to keep up with it. | 21:42 |
| *** tonanhngo has joined #openstack-kolla | 21:43 | |
| *** dave-mccowan has quit IRC | 21:45 | |
| sdake | right | 21:46 |
| sdake | 3 months in dude | 21:46 |
| sdake | not expecting miracles :) | 21:46 |
| sdake | I think the miracle has already happened tbh :) | 21:46 |
| kfox1111 | if you still want the gate change broken out, I'll break it out when it works. that way I can test what I'm testing. :) | 21:46 |
| *** b_bezak has joined #openstack-kolla | 21:48 | |
| *** tonanhngo has quit IRC | 21:48 | |
| kfox1111 | hub still broken... | 21:49 |
| * kfox1111 is sad | 21:49 | |
| *** vhosakot has quit IRC | 21:52 | |
| *** schwicht has joined #openstack-kolla | 21:53 | |
| *** b_bezak has quit IRC | 21:56 | |
| kfox1111 | this hub thing down really hurts. :/ | 22:02 |
| *** rodrigo_pereira has quit IRC | 22:02 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 22:04 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Start testing the deployments https://review.openstack.org/380868 | 22:06 |
| *** schwicht has quit IRC | 22:06 | |
| *** david-lyle has quit IRC | 22:09 | |
| *** schwicht has joined #openstack-kolla | 22:13 | |
| *** sdake has quit IRC | 22:16 | |
| *** mark-casey1 has quit IRC | 22:16 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 22:24 |
| kfox1111 | sdake: like, that one ^-- the tests finally showed an issue with the template. | 22:26 |
| *** Pavo has quit IRC | 22:27 | |
| *** salv-orlando has joined #openstack-kolla | 22:28 | |
| *** salv-orl_ has quit IRC | 22:28 | |
| *** Pavo has joined #openstack-kolla | 22:33 | |
| *** sdake has joined #openstack-kolla | 22:33 | |
| *** schwicht has quit IRC | 22:34 | |
| *** david-lyle has joined #openstack-kolla | 22:36 | |
| *** absubram has quit IRC | 22:38 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Start testing the deployments https://review.openstack.org/380868 | 22:40 |
| *** MarMat has quit IRC | 22:44 | |
| *** MarMat has joined #openstack-kolla | 22:44 | |
| *** salv-orlando has quit IRC | 22:55 | |
| *** gfhellma has quit IRC | 23:10 | |
| sdake | kfox1111 docker hub still down? | 23:13 |
| *** tonanhngo has joined #openstack-kolla | 23:15 | |
| *** schwicht has joined #openstack-kolla | 23:15 | |
| *** tonanhngo has quit IRC | 23:18 | |
| *** HyperJohnGraham has quit IRC | 23:19 | |
| sdake | kfox1111 one thing that helps with gate patches | 23:21 |
| sdake | kfox1111 is to put a [wip] in front of the commit until you think its ready | 23:21 |
| sdake | your submission indicates an implicit +2 on your part | 23:21 |
| sdake | kfox1111 but gate patches are sort of special because they require like 40 or 50 iterations to get to a working state. | 23:21 |
| *** tonanhngo has joined #openstack-kolla | 23:27 | |
| *** tonanhngo has quit IRC | 23:28 | |
| *** fguillot has quit IRC | 23:31 | |
| *** tonanhngo has joined #openstack-kolla | 23:35 | |
| *** tonanhngo has quit IRC | 23:40 | |
| *** eaguilar has joined #openstack-kolla | 23:40 | |
| kfox1111 | sdake: yeah, the one patch should be done by now... grr. | 23:42 |
| kfox1111 | I'll add wip back to it until I get a run that works. | 23:42 |
| sdake | it takes me 50 goes at a gate patch | 23:43 |
| sdake | so dont feel behidn :) | 23:43 |
| kfox1111 | the last one was like 100. :/ | 23:43 |
| sdake | this might be something minor | 23:43 |
| kfox1111 | yeah. | 23:43 |
| sdake | for 50 patches | 23:43 |
| kfox1111 | the isci check is stubbed out right now. | 23:43 |
| sdake | gate work is really hard | 23:43 |
| sdake | glad your learning it | 23:43 |
| kfox1111 | as there are no 3.0 containers released yet, and iscsi only works with 3. :/ | 23:43 |
| kfox1111 | learning, and hating it. :) | 23:44 |
| *** david-lyle has quit IRC | 23:44 | |
| sdake | you could always build it ;-) | 23:44 |
| kfox1111 | there really needs to be some docs about the differences between node types, cause its been horible to try to devine it via failed jobs. :) | 23:44 |
| sdake | kfox1111 ya its rough | 23:44 |
| kfox1111 | yeah, i walked into that one didn't I? ;) | 23:44 |
| sdake | kfox1111 i'd recommend looking at hte kolla gate jobs | 23:45 |
| sdake | we have it sorted out thtere for everything but bluebox | 23:45 |
| sdake | and possibly rax.ord | 23:45 |
| kfox1111 | I'm just puzzled by some of these failures. | 23:45 |
| *** v1k0d3n has joined #openstack-kolla | 23:45 | |
| kfox1111 | in minikube, ceph works every time. | 23:45 |
| kfox1111 | in the gate, we're seeing stuck pgs at random times. | 23:45 |
| kfox1111 | our ceph guru here's even a little puzzzled by it. | 23:46 |
| kfox1111 | adding yet more logging to see if I can catch it. | 23:46 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 23:46 |
| *** tonanhngo has joined #openstack-kolla | 23:47 | |
| kfox1111 | I just need it to get a little bit further, to test the ceph-rbd stuff out. :/ | 23:47 |
| kfox1111 | it worked a few times. :/ | 23:47 |
| *** tonanhngo has quit IRC | 23:47 | |
| kfox1111 | hmm... I wonder if this is a multinic thing.... | 23:48 |
| kfox1111 | maybe its the rackspace nodes... | 23:48 |
| *** v1k0d3n has quit IRC | 23:50 | |
| kfox1111 | so frustrating... | 23:52 |
| sdake | rax.ord? | 23:52 |
| sdake | those don't work well | 23:52 |
| sdake | the second line tells you the cloud provider from the logs | 23:52 |
| kfox1111 | not sure. some nodes have two nics and I was told they were racspace nodes. | 23:52 |
| sdake | i'm not sure if its how we use networking or they use networking | 23:52 |
| sdake | check the logs | 23:52 |
| sdake | click through | 23:52 |
| kfox1111 | could be both. | 23:53 |
| sdake | you can tell where it runs | 23:53 |
| sdake | we have a bug for it kicking around in our massive db | 23:53 |
| sdake | but fortunately rax.ord is not scheduled often | 23:53 |
| sbezverk_ | sdake: new multinode installation with kubeadm is not that straight forward | 23:53 |
| sdake | osic doing most of the work | 23:53 |
| sdake | sbezverk_ clearly from our experience yesterday ;) | 23:54 |
| sbezverk_ | sdake: took whole day to clean it up. I was just rebuilding my cluster | 23:54 |
| sbezverk_ | at least now I have a baseline if John has questions | 23:54 |
| sdake | osic doing most of the gate work that is | 23:54 |
| sdake | baseline = great | 23:54 |
| sdake | HyperJohnGraham_ sbezverk_ pings you :) | 23:55 |
| sbezverk_ | sdake: oh no, not today, ;-) it is late | 23:56 |
| sbezverk_ | sdake: kube people keep changing the way they deploy components, before everything was running in docker, not by some reason they decided kubelet to run as a regular service and the rest in docker. I wish they share reasoning behind all these | 23:58 |
| *** lamt has joined #openstack-kolla | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!