| kfox1111 | sbezverk: no. don't have any. | 00:01 |
|---|---|---|
| kfox1111 | I've tried ipmitool with lots of different bmc's though over the years. | 00:02 |
| kfox1111 | they mostly just work when enabled. | 00:02 |
| kfox1111 | but they are often flaky/insecure. so best put on their own dedicated network. | 00:02 |
| kfox1111 | I had one cluster that they would lock up randomly if more then ~100 were on the same subnet. | 00:02 |
| sbezverk | kfox1111: I am about to test fencing in the test bed for summit demo | 00:03 |
| sbezverk | but when I try manually get info from cisco CIMC on the server using ipmitool I am getting errors.. | 00:03 |
| kfox1111 | sbezverk: some bmc's need you to explicitly enable ipmi support. | 00:05 |
| kfox1111 | they disable it by default. | 00:05 |
| sbezverk | oh, hmmm let me check.. | 00:05 |
| *** salv-orlando has joined #openstack-kolla | 00:09 | |
| sbezverk | kfox1111: you were right, there is a section in CIMC setup for ipmi and it was not enabled by default | 00:09 |
| Pavo | kfox1111 ok on my master node I have 500Gb HDD space 1 HDD btw and Centos 7 as the OS, on the Controller I have 6 250Gb HDDs non raided with Centos 7 as OS and on the Compute I have 28 1.5Tb SAS SSDs in RAID 5 with Centos 7 as OS | 00:10 |
| Pavo | what would you recommend, | 00:10 |
| Pavo | and the master node is a VM btw | 00:10 |
| kfox1111 | sbezverk: cool. | 00:11 |
| *** haplo37_ has quit IRC | 00:12 | |
| kfox1111 | is the master node on one of the other nodes? | 00:12 |
| Pavo | kfox1111 this is current setup | 00:12 |
| Pavo | http://pastebin.com/72A5snhF | 00:12 |
| Pavo | no, master node is a VM on my mac pro | 00:12 |
| kfox1111 | ah. ok. | 00:13 |
| kfox1111 | hmm... | 00:13 |
| kfox1111 | what are you planning on useing the cloud for? | 00:13 |
| sdake_ | kfox1111 i had cherrypicked that patch at 8:45 am this morning | 00:13 |
| Pavo | Heat development and docker developmet | 00:13 |
| kfox1111 | sdake_: to newton? | 00:13 |
| sdake_ | yup | 00:13 |
| sdake_ | see log in review | 00:14 |
| *** haplo37_ has joined #openstack-kolla | 00:14 | |
| *** salv-orlando has quit IRC | 00:14 | |
| kfox1111 | sdake_: ok. just saw on the bug report it was listed as open still. but if you've already taken care of it, I'll just ignore it. :) | 00:14 |
| Pavo | and to document image building for our tools | 00:14 |
| kfox1111 | Pavo: so, you probably won't be using all that storage? | 00:15 |
| sdake_ | kfox1111 sometimes launchpad doesn't keep up :( | 00:15 |
| Pavo | well no, not all of it | 00:15 |
| sdake_ | kfox1111 the reason it is still open is here: https://review.openstack.org/#/c/387078/ | 00:15 |
| Pavo | kfox1111 I can move around the SAS SSDs between the servers without issues | 00:16 |
| *** aernhart has joined #openstack-kolla | 00:17 | |
| kfox1111 | Pavo: you could take the storage and rebalance it 50/50. so put 3 of the 250g drives into the compute and 14 of the 1.5 t into the controller. | 00:17 |
| Pavo | but in order to use any of it in RAID config I have to be able to install the OS as UEFI | 00:17 |
| *** aernhart has quit IRC | 00:17 | |
| Pavo | well I could but then I would have to pull one of the backplanes form the compute and add it to controller because currently controller only has 1 backplane | 00:17 |
| Pavo | which houses 6 SAS drives | 00:17 |
| Pavo | but its doable yes | 00:17 |
| kfox1111 | maybe use the 3 250g drives for journal drives, and use 3 1.5 drives for osd's per node. | 00:18 |
| kfox1111 | and then use the rest in a raid for the vm's. | 00:18 |
| Pavo | can I do that during OS install? | 00:18 |
| Pavo | or would auto configure hdds work during OS install | 00:18 |
| kfox1111 | I've done centos on uefi. works ok. | 00:18 |
| Pavo | yeah installing from USB isn't an issue right now | 00:19 |
| Pavo | but to achieve this ceph looks like I need to make some changes to hdds volumes either during OS install or after | 00:20 |
| Pavo | and want to get the HDDs correct before I start doing it | 00:20 |
| Pavo | so you recommend 50/50 on the drives for both servers | 00:21 |
| kfox1111 | ceph will format them during the ceph deployment. | 00:21 |
| Pavo | so I take it I only use 1 drive for the OS then? | 00:21 |
| kfox1111 | Pavo: yeah, those are some pretty heavy servers. you should be able to run compute on both of them. | 00:21 |
| kfox1111 | you've got a ton of drives there, and for the use case you listed, probably don't have much use of that much storage. | 00:22 |
| kfox1111 | so raid1'ing a pare of them for a root drive is probably fine? | 00:22 |
| *** tonanhngo has joined #openstack-kolla | 00:22 | |
| Pavo | well not right now I don't but when doing full packet capture that storage is eatten up quick | 00:22 |
| *** tonanhngo has quit IRC | 00:23 | |
| kfox1111 | you planning on doing that on this cloud? | 00:23 |
| Pavo | yes sir | 00:23 |
| kfox1111 | ah. | 00:23 |
| Pavo | have already done it | 00:23 |
| Pavo | works perfectly fine | 00:23 |
| Pavo | hence the ton of storage lol | 00:23 |
| kfox1111 | doing traffic inspection in/out of the cloud, or between vm's? | 00:24 |
| kfox1111 | makes sense. | 00:24 |
| Pavo | capture from outside into inside | 00:24 |
| kfox1111 | so probably better to have most of the storage on the network node then? | 00:24 |
| Pavo | passive alaysis with a Gigaview tap | 00:24 |
| Pavo | exactly the way I have it setup now | 00:25 |
| Pavo | compute, network and storage on same node | 00:25 |
| kfox1111 | oh. you put the nn on the compute node. | 00:25 |
| Pavo | yeah | 00:25 |
| kfox1111 | k. | 00:25 |
| kfox1111 | so what do you think you will put in cinder? | 00:26 |
| Pavo | instances of different tools we use to do verious stuff, sorry can not go into detail on those | 00:27 |
| kfox1111 | heh. k. let me rephrase that then.... how much space do you think you will consume in cinder? :) | 00:28 |
| kfox1111 | and how valuable is the data? | 00:28 |
| Pavo | well with the current way we are using openstack, btw we are using packstack currently we use about 23Tb of storage in about a week with full packet capture | 00:29 |
| Pavo | the data gets pushed and purged off the kit after capture | 00:29 |
| kfox1111 | are you putting the data in cinder though? | 00:29 |
| Pavo | so being on the cloud for short amount of time | 00:29 |
| Pavo | yes | 00:29 |
| Pavo | but yes using Cinder as a backend for storage currently | 00:30 |
| Pavo | but the way packstack does that is completely different then kolla from what I can see | 00:30 |
| kfox1111 | so your capturing packets in the vm's, not the hosts? | 00:30 |
| Pavo | yes the instances | 00:30 |
| Pavo | in different areas of the network | 00:31 |
| Pavo | using ERSPAn and RSPAN and normal taps | 00:31 |
| kfox1111 | if you dont care about loosing some data if you loose a single drive, lvm still might be a good option. | 00:31 |
| kfox1111 | you will pay some storage overhead with ceph. most of the time its worth it for the increased availability, | 00:32 |
| kfox1111 | but if you don't care about loosing data occationally, and have a huge amount of data you want to churn through, lvm still might be a viable option then. | 00:32 |
| kfox1111 | with lvm there's not really an option to not do iscsi still, but you still may get some benifit from running cinder-volume on the compute node in lvm/iscsi. | 00:33 |
| kfox1111 | then you don't have any networking in the way between your storgage and vm's. | 00:34 |
| Pavo | ok I understand about half of that lol | 00:35 |
| Pavo | I am not storage strong | 00:35 |
| Pavo | lol | 00:35 |
| kfox1111 | k. for your use case, I'm thinking you leave the storage as is, | 00:35 |
| Pavo | what do I need to change before deploying with kolla on the nodes? | 00:35 |
| kfox1111 | then run cinder-volume on the compute node. | 00:36 |
| Pavo | would I do that in globals or inventory? | 00:36 |
| Pavo | starting up master node now | 00:37 |
| kfox1111 | and use the lvm backend. | 00:37 |
| kfox1111 | I havent deployed kolla-ansible before, so don't know the details. | 00:38 |
| kfox1111 | I do know the lvm bits were added only in newton, so you will need to use that release. | 00:38 |
| Pavo | what do you use? | 00:38 |
| Pavo | newtron fully working? | 00:38 |
| kfox1111 | right now, manually built cloud. initial 1 node packstack deploy, then scaled up. | 00:38 |
| *** yingjun has joined #openstack-kolla | 00:38 | |
| kfox1111 | working on kolla-kubernetes though, to make it production ready, then will switch to that. | 00:38 |
| Pavo | scaled to how far, because the max we can scale with packstack is 5 nodes | 00:38 |
| *** aernhart has joined #openstack-kolla | 00:39 | |
| *** duonghq has joined #openstack-kolla | 00:39 | |
| duonghq | morning | 00:39 |
| Pavo | morning | 00:40 |
| kfox1111 | I have only done single node deploys, then manually build out cobbler based kickstarts to add hundreds of compute nodes. | 00:40 |
| Pavo | ah | 00:40 |
| *** aernhart has quit IRC | 00:40 | |
| Pavo | sounds like a pain in the ass | 00:40 |
| kfox1111 | yeah, not the most pleasent install. | 00:41 |
| kfox1111 | but there's worse things. :) | 00:41 |
| *** spsingh has joined #openstack-kolla | 00:41 | |
| sbezverk | kfox1111: planning to test fencing do you want to watch? for now without ipmi as it is still failing to communicate with cimc | 00:44 |
| kfox1111 | sbezverk: fencing without shutting down the node isn't really fencing. :/ | 00:45 |
| kfox1111 | did you just stub out the ipmi call then? | 00:46 |
| *** spsingh has quit IRC | 00:46 | |
| kfox1111 | do you need some help debugging the cimc? | 00:46 |
| sbezverk | kfox1111: I think it is config issue on the server side | 00:47 |
| sbezverk | I just sent email to our ucs servers support people | 00:48 |
| sbezverk | for now it will just timeout without doing anything | 00:48 |
| kfox1111 | ah. k. | 00:48 |
| kfox1111 | the fence script should be written to fail to unlock the rbd volume in that case. | 00:49 |
| britthouser3 | What ipmi module are you using sbezverk? lan or lanplus? | 00:49 |
| sbezverk | lanplus | 00:49 |
| britthouser3 | That should be right for newer UCS firmwares | 00:49 |
| sbezverk | sdake's server has not the latest but one right before the latest | 00:50 |
| kfox1111 | sbezverk: do you have the username/password set right too? ipmitool doesn't aways print the most useful error messages if they are wrong. | 00:50 |
| sbezverk | britthouser3: I am using falcor cluster | 00:50 |
| britthouser3 | oh that should be recent enough then. | 00:50 |
| sbezverk | kfox1111: yes I think ipmi username and password is the same as cimc | 00:51 |
| *** tovin07 has joined #openstack-kolla | 00:52 | |
| sbezverk | britthouser3: I am using 5 boxes from that cluster for summit demo | 00:52 |
| kfox1111 | sbezverk: might double check though. easy to get off by a character depending on the bmc. some don't ask for multiple passwords to verify you type it in right. | 00:52 |
| *** yingjun has quit IRC | 00:52 | |
| britthouser3 | which one isn’t powering off? | 00:52 |
| sbezverk | here is webex link https://cisco.webex.com/join/sbezverk | 00:53 |
| sbezverk | I am planning to test fencing now so whoever wants to see welcome to join | 00:53 |
| britthouser3 | is it 10.23.223.78? | 00:53 |
| sbezverk | 78,79,80,81,82 | 00:54 |
| sbezverk | these 5 boxes I rebuild everything on them | 00:54 |
| sbezverk | now they are running kolla-kubernetes | 00:54 |
| sbezverk | here is the error I am getting from ipmitool | 00:55 |
| sbezverk | http://paste.openstack.org/show/585869/ | 00:55 |
| *** spsingh has joined #openstack-kolla | 00:57 | |
| kfox1111 | sbezverk: hmm... that seems to imply its doing more then just basic username/password auth. | 00:58 |
| kfox1111 | or trying at least. | 00:58 |
| britthouser3 | yeah you’ve got an encryption key set for IPMI | 00:59 |
| sbezverk | kfox1111: that is what I am afraid I saw some notes about vulnerability discovered | 00:59 |
| sbezverk | before and I suspect they beefed up security | 00:59 |
| britthouser3 | YOu need to specify the same encryption key also | 00:59 |
| britthouser3 | or set key to all zeros | 01:00 |
| sbezverk | britthouser3: it does not let me | 01:00 |
| *** yingjun has joined #openstack-kolla | 01:00 | |
| britthouser3 | let you set it to zeros? | 01:00 |
| sbezverk | britthouser3: if it set to all 0 I get XML parse error when try to save | 01:00 |
| sbezverk | so I selected randomized, then it allowed to save | 01:01 |
| kfox1111 | probably need to specify it on both sides then? | 01:01 |
| sbezverk | kfox1111: tried that, ipmitool complains that the key is too long | 01:01 |
| kfox1111 | can you set it manually to something that isn't too long? | 01:02 |
| britthouser3 | try .79 again | 01:02 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Modular gate https://review.openstack.org/386966 | 01:05 |
| sbezverk | same thing | 01:05 |
| britthouser3 | =( | 01:06 |
| britthouser3 | setting code to 40 zeros should be none | 01:06 |
| sbezverk | britthouser3: I sent email to cisco ucs support people maybe they could suggest somehting | 01:06 |
| sbezverk | > RAKP 2 HMAC is invalid | 01:06 |
| sbezverk | Error: Unable to establish IPMI v2 / RMCP+ session | 01:06 |
| sbezverk | Running Get PICMG Properties my_addr 0x20, transit 0, target 0x20 | 01:06 |
| sbezverk | No Response from Get PICMG Properties | 01:06 |
| sbezverk | No PICMG Extenstion discovered | 01:06 |
| sbezverk | Error sending Chassis Status command | 01:06 |
| kfox1111 | can you change it to simple auth? | 01:07 |
| sbezverk | kfox1111: Do not see any option for that :-( | 01:09 |
| britthouser3 | I set both .78 and .79 to 0000000000000000000000000000000000000000 | 01:09 |
| britthouser3 | that should be ‘none' | 01:09 |
| *** salv-orlando has joined #openstack-kolla | 01:10 | |
| sbezverk | britthouser3: it worked now | 01:10 |
| sbezverk | !! | 01:10 |
| openstack | sbezverk: Error: "!" is not a valid command. | 01:10 |
| kfox1111 | nice. :) | 01:11 |
| sbezverk | it looks like the issue is '!' in password | 01:11 |
| *** zhubingbing has joined #openstack-kolla | 01:11 | |
| sbezverk | I tried to backspace it but it does not like it | 01:11 |
| Jeffrey4l | sup kfox1111 | 01:12 |
| kfox1111 | Jeffrey4l: hey. :) | 01:12 |
| kfox1111 | working off a cold. :/ | 01:12 |
| Jeffrey4l | did you see the same ceph issue on centos? | 01:13 |
| Jeffrey4l | kfox1111, ^^ | 01:13 |
| britthouser3 | w00t! sbezverk | 01:14 |
| britthouser3 | I’ll let you set the rest to the same | 01:14 |
| *** zhurong has joined #openstack-kolla | 01:14 | |
| sbezverk | britthouser3: done, thank you for your help!! | 01:14 |
| *** salv-orlando has quit IRC | 01:15 | |
| kfox1111 | Jeffrey4l: don't remember the context for the question. which issue? | 01:15 |
| Jeffrey4l | kfox1111, <kfox1111> Jeffrey4l: I ran into a similar issue before with centos. giant had preference to hammer and caused a huge mess when trying to upgrade. :/ | 01:15 |
| kfox1111 | Jeffrey4l: ah. that was a couple years ago. | 01:16 |
| sbezverk | kfox1111: the test plan is to stop kubelet service on an node with mariadb and then it should be powered off and mariadb move to another controller. Anything else? | 01:16 |
| Jeffrey4l | hmm. OK | 01:16 |
| kfox1111 | I'm still running infernalis. will try jewel here in a few days when my new test hardware comes in. | 01:16 |
| Jeffrey4l | cool. | 01:16 |
| kfox1111 | sbezverk: turning off kubelet doesn't nessisarily kill the mariadb pod. | 01:17 |
| kfox1111 | so the cluster might notice kubelet go away and then unlock the rbd volume prematurely. | 01:17 |
| sbezverk | kfox1111: hmm, I thought by stopping kubelet, controller process on apiserver will stop receiving keepalives and set that node notready | 01:18 |
| sbezverk | which will trigger fencing pod to act | 01:19 |
| kfox1111 | sbezverk: it should. | 01:19 |
| kfox1111 | just saying, the fence agent needs to ensure the ipmipower off before the unlock. which the latest version of the code in revew doesnt do. | 01:19 |
| kfox1111 | if you have a new version that isn't posted yet though, then that should be ok. | 01:19 |
| bjolo | sdake_: just sahara as far as I could tell | 01:21 |
| sdake_ | bjolo did you get any sleep :) | 01:22 |
| sbezverk | kfox1111: sounds good, I will swap ceph lock and ipmi and try that if you strongly feel it is right way | 01:22 |
| kfox1111 | yeah. I think its very important. | 01:22 |
| bjolo | My kids just woke me up :p | 01:22 |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix cinder upgrade permission issue https://review.openstack.org/386930 | 01:23 |
| sdake_ | bjolo tell me about it - my kids do that to me all the time | 01:23 |
| sdake_ | bjolo my wife wants to go shopping - i'll bbin30 mins or so ok? | 01:23 |
| sdake_ | bjolo in meantime sync up with Jeffrey4l | 01:23 |
| sdake_ | he is a cr, he can make the same decisions i can | 01:23 |
| bjolo | No worries. I go back to sleep for a few more | 01:24 |
| sdake_ | bjolo the fact that sahara is doa isn't super concernign to me, but if its an easy fix might as well merge it | 01:24 |
| sdake_ | bjolo link the fix to Jeffrey4l so he can have a look plz | 01:24 |
| bjolo | Ok take it with Jeffrey4l | 01:24 |
| Jeffrey4l | np | 01:24 |
| britthouser3 | anytime sbezverk ! | 01:24 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: DO NOT MERGE: test eth1 https://review.openstack.org/385238 | 01:25 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Fencing pod https://review.openstack.org/383922 | 01:26 |
| bjolo | https://review.openstack.org/387092 | 01:26 |
| Pavo | ok so.... still kinda confused on this cinder issue, do I only use 1 SAS SSD drive for the OS and the rest in a RAID 5 and attach the RAID for Cinder or put the OS on the RAID and use Cinder on it? | 01:26 |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix sahara upgrade permission issue https://review.openstack.org/386928 | 01:27 |
| Pavo | using CentOS 7 as host OS and base for kolla btw | 01:28 |
| *** eaguilar has joined #openstack-kolla | 01:28 | |
| bjolo | Jeffrey4l: enable sahara breaks cron container. Gets stuck in restarting | 01:28 |
| Jeffrey4l | got any logs info? bjolo | 01:28 |
| bjolo | Yes check the bug on launchpad | 01:29 |
| Jeffrey4l | ok | 01:29 |
| Pavo | any suggestions? | 01:29 |
| bjolo | I'm on my phone irc client atm | 01:30 |
| Jeffrey4l | roger. so your patch fix this issue? the patch lgtm | 01:30 |
| bjolo | Yes | 01:30 |
| bjolo | Tested | 01:30 |
| Jeffrey4l | +2 | 01:31 |
| bjolo | Sweet | 01:31 |
| Jeffrey4l | nice jobs ;) | 01:31 |
| bjolo | Ok bok to sleep for me | 01:31 |
| bjolo | Cya in a few hours :) | 01:32 |
| Jeffrey4l | good night | 01:32 |
| kfox1111 | Pavo: with lvm, you create an lvm pool for it to create volumes out of. | 01:32 |
| sbezverk | kfox1111: ipmi did not work but complete recovery took 2 minutes and 3 seconds | 01:32 |
| Pavo | would I do that on the OS drive or the RAID? | 01:32 |
| kfox1111 | Pavo: whichever you create the vg on. probabaly best on the raid set. | 01:33 |
| Pavo | ok thats what I was thinking, I will redo the drives tomorrow, will 50/50 on the servers with the drives and then only use 2x 250Gb SAS SSDs for OS and the rest for storage and and create the lvm pool on the RAID for cinder | 01:35 |
| Pavo | does that sound about right? | 01:35 |
| kfox1111 | sbezverk: I'm going to try and add insteon support to the fence agent at some point in the near future too. as my home system doesn't have ipmi, but I do have some insteon equipment. | 01:35 |
| kfox1111 | Pavo: if your doing lvm, not ceph, I'd leve all the storage where it is. | 01:35 |
| Pavo | oh ok | 01:35 |
| kfox1111 | and run cinder-volume on the compute node, not the controller. so it has access to all the raid storage. | 01:36 |
| Pavo | well still gonna switch 1 drive for a 250Gb drive to use only for OS | 01:36 |
| kfox1111 | yeah. that should work. | 01:36 |
| Pavo | anything special I have to do wehn installing CentOS 7 on that type of setup, I have never done secondary storage for linux before | 01:37 |
| Pavo | have only used the entire RAID for OS during install | 01:37 |
| *** rhallisey has quit IRC | 01:38 | |
| kfox1111 | just avoid having it touch the raid, and then do a pvcreate/vgcreate manuallly on it after its instealled for cindr to use. | 01:38 |
| Pavo | ok basically just follow the directions on http://docs.openstack.org/developer/kolla/cinder-guide.html | 01:39 |
| kfox1111 | yeah. that looks about right. | 01:40 |
| kfox1111 | skip the loopback stuff. | 01:40 |
| Pavo | k | 01:40 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Fencing pod https://review.openstack.org/383922 | 01:43 |
| Pavo | kfox1111 so I would only set compute in globals under storage? | 01:46 |
| kfox1111 | I would guess so. never done it before. so your milage may varry. :) | 01:48 |
| Pavo | lol ok | 01:48 |
| kfox1111 | be back in a few. the boy wants some attention. | 01:50 |
| *** haplo37 has quit IRC | 01:52 | |
| *** g3ek has quit IRC | 01:52 | |
| mtaylor22 | Jeffrey4l: ping | 01:53 |
| *** g3ek has joined #openstack-kolla | 01:53 | |
| *** spsingh has quit IRC | 01:54 | |
| *** haplo37 has joined #openstack-kolla | 01:54 | |
| Jeffrey4l | mtaylor22, sup | 01:54 |
| mtaylor22 | re https://review.openstack.org/#/c/385559/ | 01:54 |
| mtaylor22 | do you plan on going down this route? | 01:54 |
| mtaylor22 | or using /run:/run:shared as per docker bug repor | 01:54 |
| mtaylor22 | t | 01:54 |
| Jeffrey4l | going down router mean? | 01:55 |
| Jeffrey4l | /run:/run:shared solve the issue. | 01:55 |
| mtaylor22 | on all containers using /run ? | 01:56 |
| Jeffrey4l | i think it should a kernel issue(or this is how it works when using mix private mount and shared mount ) | 01:56 |
| mtaylor22 | yeah | 01:56 |
| Jeffrey4l | mtaylor22, yep. when using /run, just use /run:shared. | 01:56 |
| mtaylor22 | hmm ok cool. | 01:56 |
| Jeffrey4l | use specified /run/xxx is hard. | 01:56 |
| mtaylor22 | yep yep, gotcha. | 01:56 |
| Jeffrey4l | i debugged several service. | 01:56 |
| mtaylor22 | are you going to update https://review.openstack.org/#/c/385559/ ? | 01:56 |
| Jeffrey4l | tgt use /run/tgt.bla.bla.sock file, which is hard to change. | 01:57 |
| mtaylor22 | yep | 01:57 |
| Jeffrey4l | the patch is OK now. | 01:57 |
| Jeffrey4l | there is nothing to change afaik. | 01:57 |
| Jeffrey4l | have u tried it? did u see the issue when using that patch? mtaylor22 | 01:58 |
| mtaylor22 | about to test now | 01:58 |
| mtaylor22 | hence why i wanted to ping you first | 01:58 |
| Jeffrey4l | cool. | 01:58 |
| Jeffrey4l | it works fine in my test. | 01:58 |
| Jeffrey4l | waiting for you test result ;) | 01:58 |
| *** spsingh has joined #openstack-kolla | 01:59 | |
| mtaylor22 | heh | 01:59 |
| mtaylor22 | which one do you want me to test though | 01:59 |
| mtaylor22 | your commit (385559), or just using /run:/run:shared | 01:59 |
| Jeffrey4l | https://review.openstack.org/#/c/385559/ is using /run:/run:shared solutions. | 02:00 |
| mtaylor22 | oh! i didn't see your last update | 02:00 |
| mtaylor22 | awesome, ok thanks. I will test now under Mitaka. | 02:00 |
| Jeffrey4l | mtaylor22, ^^ so just use the patch ;) | 02:00 |
| mtaylor22 | yup :) | 02:00 |
| Jeffrey4l | OK. | 02:01 |
| sbezverk | kfox1111: one scenario needs to be discussed | 02:08 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Fix docs formatting error https://review.openstack.org/380161 | 02:09 |
| *** salv-orlando has joined #openstack-kolla | 02:11 | |
| *** spsingh has quit IRC | 02:14 | |
| *** asalkeld has joined #openstack-kolla | 02:16 | |
| kfox1111 | back. | 02:16 |
| *** tonanhngo has joined #openstack-kolla | 02:16 | |
| *** salv-orlando has quit IRC | 02:16 | |
| kfox1111 | sbezverk: whats up? | 02:16 |
| *** tonanhngo has quit IRC | 02:17 | |
| *** sdake_ has quit IRC | 02:18 | |
| Pavo | which is better kolla-ansible or kolla-kubernetes to deploy openstack on 2 nodes? | 02:20 |
| kfox1111 | Pavo: depends who you ask. ;) | 02:21 |
| kfox1111 | I'm not unbiased. | 02:21 |
| Pavo | well I have never used kubernetes before but heard its pretty amazing | 02:21 |
| kfox1111 | but I'd say if you need something production ready now, ansible is probably a better fit. | 02:21 |
| kfox1111 | kolla-kubernetes is still in development. I'm going to try and put it in production soonish though. | 02:22 |
| Pavo | especially the self healing feature | 02:22 |
| kfox1111 | you can do self healing with other tech too, such as pacemaker/keepalive. | 02:22 |
| kfox1111 | not sure how much of that kolla-ansible sets up though. | 02:23 |
| duonghq | kolla-ansible already setup haproxy | 02:23 |
| duonghq | and it's work fine | 02:23 |
| duonghq | (for me) | 02:23 |
| duonghq | haproxy/keepalived stack, indeed | 02:24 |
| Pavo | so no advantages using kolla-kubernetes over kolla-ansible? | 02:24 |
| kfox1111 | haproxy isn't quite the same thing as ha/self healing. | 02:24 |
| kfox1111 | Pavo: there are some big advantages I think. | 02:24 |
| Pavo | like? | 02:24 |
| kfox1111 | scaling will work smoother/more automatic. | 02:25 |
| kfox1111 | minor upgrades will be easier. | 02:25 |
| duonghq | scheduling of k8s is much better than haproxy/keepalived stack one | 02:25 |
| kfox1111 | you can create simpler ha setups with it. wont failover as fast, but will be easier to debug. | 02:25 |
| duonghq | (re | 02:26 |
| duonghq | (re-scheduling) | 02:26 |
| kfox1111 | single pane of glass stuff, so easier debugging/mgmt, etc. | 02:26 |
| Pavo | guess kolla-ansible would be a better solution for my use case then | 02:27 |
| duonghq | but it's not quite mature as ansible one, quite sad | 02:27 |
| duonghq | ah, sorry, miss reading | 02:28 |
| kfox1111 | Pavo: for now, yeah. it will be easier to stand up. | 02:29 |
| Pavo | ok | 02:29 |
| *** phuongnh has joined #openstack-kolla | 02:29 | |
| kfox1111 | I'm working on getting gating working for kolla-kubernetes though, | 02:29 |
| kfox1111 | so I think before long, the kolla-kubernetes stuff will be better tested if no one steps up to write corrisponding kolla-ansible tests. | 02:29 |
| Pavo | right now it takes around 2 hours to do a from ground up deployment from baremetel to full cloud with the script we use with packstack | 02:29 |
| kfox1111 | yeah. kolla will be much faster then that. | 02:30 |
| Pavo | trying to lean towards more of a docker approach so we can leverage some of our tools with the cloud | 02:30 |
| kfox1111 | I'm doing a full kolla-kubernetes deployment in a vm, in the gate including launching two vm's, setting up a virtual network between them, creating a cindre volume, putting data in it, and moving it from vm to vm, | 02:30 |
| kfox1111 | all in about 20-24 min. | 02:30 |
| Pavo | wow | 02:31 |
| Pavo | thats WAY faster | 02:31 |
| Pavo | lol | 02:31 |
| kfox1111 | and the vm's dont even have hardware accleration. :) | 02:31 |
| kfox1111 | and we're not caching any of the images. :) | 02:31 |
| kfox1111 | I'm guessing we can get that down to 15 min if we tried. | 02:31 |
| Pavo | kfox1111 when you go back to work can you get me that cobbler instructions for UEFI | 02:31 |
| duonghq | do not understand why we use haproxy/keepalived stack, it's easier for setup but I heard from a guys from clusterlab that ldirectord is better than haproxy in LB manner, cannot find evidence | 02:31 |
| kfox1111 | Pavo: I'll see what I can do. we never wrote it down. plus I have an outage to do next week. :/ | 02:32 |
| kfox1111 | you really should do a tcpdump and look at the files its trying to download and verify they are on disk. | 02:32 |
| Pavo | or maybe someone could help me with this bifrost thing | 02:32 |
| Pavo | rofl | 02:33 |
| Pavo | http://docs.openstack.org/developer/kolla/bifrost.html | 02:33 |
| *** mtaylor22 has quit IRC | 02:33 | |
| Pavo | doesn't even say what bifrost is or used for | 02:33 |
| kfox1111 | yeah, I think that is very very new. | 02:34 |
| Pavo | question | 02:34 |
| kfox1111 | maybe answer | 02:35 |
| Pavo | do you think if you guys built a container and put all the repos that kolla uses when building images with kolla-build would be alot faster for building the images? | 02:35 |
| Pavo | I know it would mean a shit ton of hard coded changes in dockerfiles but I think it would be extremely benifcal | 02:36 |
| kfox1111 | yes. but it would then produce images that would probably be out of date/contain security issues. | 02:36 |
| kfox1111 | I've been pushing that we should keep prebuilt/up to date images on the hub. | 02:36 |
| Pavo | not if the repo container was the only one that pulled newest files on build time | 02:36 |
| Pavo | it can do a rsync or something like a git pull | 02:37 |
| Pavo | before building the rest of the images | 02:37 |
| Pavo | I would think | 02:37 |
| Pavo | that would allow the control of using only known working rpms | 02:38 |
| Pavo | instead of relying on public repos to pull packages | 02:38 |
| Pavo | I have done the normal centos repos as a docker container before | 02:39 |
| Pavo | but I know there are a ton of single repo files that are pulled during build process like rabbitmq for example | 02:39 |
| kfox1111 | yeah. | 02:40 |
| kfox1111 | but would still have to rebuild all the containers, and it would be faster still just to be able to pull aready up to date ones. | 02:40 |
| kfox1111 | most people do't have to customaize the containers themselves at all. | 02:40 |
| Pavo | only disadvantage I can see is your project would have to setup a public server to house those repos to pull from | 02:40 |
| Pavo | true or that would work also | 02:41 |
| kfox1111 | hub.docker.com. :) | 02:41 |
| openstackgerrit | caoyuan proposed openstack/kolla: Fix watcher upgrade permission issue https://review.openstack.org/386923 | 02:41 |
| Pavo | just seeing the build image process is the only thing that takes FOREVER | 02:42 |
| Pavo | :P | 02:42 |
| * kfox1111 nods. | 02:42 | |
| kfox1111 | I'm using the upstream ones for testing against. | 02:42 |
| kfox1111 | thats why the gate goes so fask. | 02:43 |
| kfox1111 | fast. | 02:43 |
| Pavo | but deploying after on a 10Gb network is fast | 02:43 |
| * kfox1111 nods | 02:43 | |
| openstackgerrit | caoyuan proposed openstack/kolla: Fix watcher upgrade permission issue https://review.openstack.org/386923 | 02:43 |
| Pavo | once I figure out how to do DDNS on a Cisco 4431 ISR router I will setup my stack to have public access if you guys want some test beds to play with | 02:44 |
| kfox1111 | cool. | 02:44 |
| *** portdirect has quit IRC | 02:46 | |
| Pavo | don't think would be a bad speeds to have that would you? http://www.speedtest.net/my-result/5720735407 | 02:46 |
| kfox1111 | nice. :) | 02:46 |
| openstackgerrit | caoyuan proposed openstack/kolla: Fix rally upgrade permission issue https://review.openstack.org/386927 | 02:47 |
| *** mtaylor22 has joined #openstack-kolla | 02:48 | |
| *** tonanhngo has joined #openstack-kolla | 02:58 | |
| Pavo | ok heading to bed gotta get up for PT @0530 | 02:59 |
| Pavo | see ya tomorrow | 02:59 |
| *** tonanhngo has quit IRC | 02:59 | |
| kfox1111 | l8r. | 03:05 |
| *** sdake has joined #openstack-kolla | 03:12 | |
| sdake | sup peeps | 03:15 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Add validate no new line at end of file https://review.openstack.org/380340 | 03:18 |
| openstackgerrit | Merged openstack/kolla: fix cron logrotate for sahara https://review.openstack.org/387092 | 03:27 |
| mtaylor22 | sup sdake | 03:31 |
| mtaylor22 | Jeffrey4l: sorry, had to deal with internal stuff .. deploying now on mitaka for testing | 03:31 |
| sdake | sup mtaylor22 | 03:31 |
| Jeffrey4l | cool. | 03:31 |
| mtaylor22 | sdake: just out of curiosity, where are you located? | 03:31 |
| sdake | scottsdale, az | 03:31 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Modular gate https://review.openstack.org/386966 | 03:31 |
| mtaylor22 | roger. 8.32pm there. | 03:32 |
| sdake | ya - where abouts are you located | 03:32 |
| mtaylor22 | Sydney | 03:32 |
| sdake | cool | 03:33 |
| mtaylor22 | 14:33 here | 03:33 |
| sdake | ya - a fella named angus used to be on my team in aus | 03:34 |
| sdake | asakeld | 03:34 |
| mtaylor22 | at cisco? | 03:34 |
| sdake | nah when i was a sith lord at red hat | 03:34 |
| mtaylor22 | hahaha | 03:34 |
| mtaylor22 | yeah their office is fairly close to where i am | 03:34 |
| mtaylor22 | RedHat is in North Sydney | 03:34 |
| mtaylor22 | im on the other side of the harbour bridge ;) | 03:35 |
| mtaylor22 | i went there once to sit an exam | 03:35 |
| sdake | even though he was in syndney he worked from home | 03:35 |
| mtaylor22 | interesting | 03:35 |
| sdake | ya we pretty much opened up aus for red hat for angus | 03:36 |
| sdake | and a few other peole | 03:36 |
| sdake | opening a country is very expensive | 03:36 |
| sdake | requires vps and stuff to get involved | 03:38 |
| sdake | major pita | 03:38 |
| sdake | global mobility in general is a pita | 03:38 |
| sdake | which co you at mytalor | 03:39 |
| sdake | mtaylor22 that is | 03:40 |
| mtaylor22 | Digital Pacific | 03:40 |
| mtaylor22 | www.digitalpacific.com.au | 03:40 |
| mtaylor22 | can imagine | 03:44 |
| *** neilus has joined #openstack-kolla | 03:46 | |
| *** eaguilar has quit IRC | 03:48 | |
| *** neilus has quit IRC | 03:50 | |
| *** dave-mccowan has quit IRC | 03:51 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Modular gate https://review.openstack.org/386966 | 04:01 |
| *** salv-orlando has joined #openstack-kolla | 04:13 | |
| *** salv-orlando has quit IRC | 04:17 | |
| *** ssurana has joined #openstack-kolla | 04:20 | |
| mtaylor22 | Jeffrey4l: looks good on mitaka | 04:23 |
| mtaylor22 | (so far) | 04:23 |
| Jeffrey4l | good news :) | 04:29 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Fencing pod https://review.openstack.org/383922 | 04:31 |
| asalkeld | :wave: | 04:34 |
| *** ssurana has quit IRC | 04:43 | |
| *** salv-orlando has joined #openstack-kolla | 05:02 | |
| mtaylor22 | Jeffrey4l: nm having issues with vxlan now | 05:05 |
| mtaylor22 | and vlan provider networks | 05:05 |
| mtaylor22 | hmm ok, will troubleshoot | 05:05 |
| *** msimonin has joined #openstack-kolla | 05:15 | |
| *** msimonin has quit IRC | 05:15 | |
| *** tonanhngo has joined #openstack-kolla | 05:20 | |
| *** msimonin has joined #openstack-kolla | 05:21 | |
| *** sdake has quit IRC | 05:23 | |
| *** msimonin has quit IRC | 05:23 | |
| *** msimonin has joined #openstack-kolla | 05:24 | |
| *** msimonin has quit IRC | 05:31 | |
| *** sdake has joined #openstack-kolla | 05:33 | |
| *** neilus has joined #openstack-kolla | 05:34 | |
| bjolo__ | morning | 05:35 |
| *** neilus has quit IRC | 05:38 | |
| mtaylor22 | w | 05:40 |
| *** sdake has quit IRC | 05:44 | |
| *** msimonin has joined #openstack-kolla | 05:45 | |
| *** ssurana has joined #openstack-kolla | 05:45 | |
| *** msimonin has quit IRC | 05:45 | |
| *** Serlex has joined #openstack-kolla | 05:47 | |
| *** salv-orlando has quit IRC | 05:51 | |
| Jeffrey4l | mtaylor22, still have issue? | 05:55 |
| mtaylor22 | yeah local though | 05:55 |
| Jeffrey4l | any details? | 05:55 |
| Jeffrey4l | mtaylor22, do u mean vxlan do not work with namespace patch? | 05:57 |
| mtaylor22 | i will confirm shortly | 05:58 |
| *** msimonin has joined #openstack-kolla | 05:58 | |
| *** ssurana has quit IRC | 06:00 | |
| *** msimonin has quit IRC | 06:00 | |
| *** egonzalez90 has joined #openstack-kolla | 06:01 | |
| *** ssurana has joined #openstack-kolla | 06:05 | |
| *** ssurana1 has joined #openstack-kolla | 06:09 | |
| *** ssurana2 has joined #openstack-kolla | 06:11 | |
| mtaylor22 | Jeffrey4l: looks good mate | 06:12 |
| mtaylor22 | was an issue local to my environment ;) | 06:12 |
| Jeffrey4l | cool. finally this issue is gone ;). thanks for the test | 06:12 |
| *** ssurana has quit IRC | 06:13 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix common role upgrade permission issue https://review.openstack.org/387171 | 06:14 |
| mtaylor22 | thanks for the help with it so f ar | 06:14 |
| mtaylor22 | far * | 06:14 |
| mtaylor22 | i think all of us put a lot of work into that damn bug | 06:14 |
| Jeffrey4l | yes | 06:15 |
| *** ssurana1 has quit IRC | 06:15 | |
| mtaylor22 | ok cya tomorrow. | 06:17 |
| Jeffrey4l | see you ;) | 06:18 |
| *** mtaylor22 has quit IRC | 06:19 | |
| *** coolsvap has joined #openstack-kolla | 06:20 | |
| *** tonanhngo has quit IRC | 06:20 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix ironic upgrade permission issue https://review.openstack.org/387180 | 06:28 |
| duonghq | did Kolla support upgrade of OpenStack service, include db upgrade...? | 06:32 |
| *** huhaoran has joined #openstack-kolla | 06:33 | |
| Jeffrey4l | duonghq, yes | 06:34 |
| duonghq | Jeffrey4l: include zero downtime of keystone? | 06:35 |
| Jeffrey4l | duonghq, kolla upgrades service by 1/3 nodes by 1/3 nodes. so yes for keystone. | 06:37 |
| duonghq | Jeffrey4l: does it generalize for all other service? | 06:38 |
| Jeffrey4l | duonghq, yes. | 06:38 |
| *** Jeffrey4l has quit IRC | 06:45 | |
| *** ssurana2 has quit IRC | 06:51 | |
| *** salv-orlando has joined #openstack-kolla | 06:52 | |
| *** salv-orlando has quit IRC | 06:56 | |
| duonghq | Jeffrey4l: for Neutron: Neutron doesn't support zero-downtime upgrade | 06:58 |
| duonghq | so at some time, the db vesion and code version is out of sync | 06:58 |
| duonghq | and the neutron service is unstable for some node (nodes haven't upgrade yet) | 06:58 |
| duonghq | those nodes can write dirty value into db | 06:59 |
| *** Jeffrey4l has joined #openstack-kolla | 07:00 | |
| duonghq | Jeffrey4l, ping | 07:01 |
| Jeffrey4l | duonghq, pong | 07:01 |
| duonghq | for Neutron: Neutron doesn't support zero-downtime upgrade | 07:01 |
| duonghq | so at some time, the db vesion and code version is out of sync | 07:01 |
| duonghq | and the neutron service is unstable for some node (nodes haven't upgrade yet) | 07:01 |
| duonghq | those nodes can write dirty value into db | 07:01 |
| duonghq | Kolla cannot handle this | 07:01 |
| duonghq | so zero downtime upgrade is not applicable for Neutron? | 07:01 |
| Jeffrey4l | hmm. ok so kolla handle this in wrong. | 07:02 |
| Jeffrey4l | you are correct. | 07:02 |
| Jeffrey4l | so the best way to upgrade neutron is: stop all neutron related service/container, then upgrade, correct? | 07:03 |
| *** tonanhngo has joined #openstack-kolla | 07:03 | |
| duonghq | Jeffrey4l, atm, I think so | 07:03 |
| Jeffrey4l | OK. so this should be well documented in kolla: stop all neutron when upgrading. | 07:03 |
| *** tonanhngo has quit IRC | 07:04 | |
| duonghq | but, do we have any plan for add zero-downtime upgrade capability for OpenStack service, I mean the service doesn't need to aware it is in upgrade process | 07:04 |
| duonghq | hmm, not only Neutron, many service except Keystone doesn't support zero downtime yet | 07:05 |
| Jeffrey4l | first of all, the services should support this. for the neutron case, there is nothing kolla can do ;( | 07:05 |
| Jeffrey4l | nova is using version objects which support upgrade online and upgrade partially. | 07:05 |
| duonghq | yet, it's minimal downtime? | 07:06 |
| Jeffrey4l | nova? | 07:06 |
| duonghq | yes | 07:06 |
| duonghq | minimal downtime is not zero downtime | 07:06 |
| Jeffrey4l | nova should support 0 downtime upgrade in theory. But no idea for the real test. | 07:07 |
| Jeffrey4l | duonghq, any idea for decrease the downtime during upgrade? | 07:07 |
| duonghq | Jeffrey4l, imo, it's heavily depend on service's architecture and logic, do you think so? | 07:09 |
| Jeffrey4l | yes. | 07:09 |
| duonghq | did you think about this issue Jeffrey4l ? | 07:09 |
| Jeffrey4l | service should handle the incompatible data structure in db and rpc when just upgraded part of services. | 07:10 |
| Jeffrey4l | which one? | 07:10 |
| duonghq | about support zero downtime upgrade by Kolla itself? | 07:10 |
| Jeffrey4l | hmm. there are not much kolla can do for this, unless the service support this. | 07:11 |
| duonghq | hmm, seems that | 07:12 |
| duonghq | Jeffrey4l, one more issue: | 07:13 |
| Jeffrey4l | yep | 07:13 |
| duonghq | atm, Kolla doesn't invoke service's specific db migration | 07:13 |
| *** imcsk8 is now known as imcsk8_PTO | 07:14 | |
| Jeffrey4l | duonghq, no, it invoke during first deploy and upgrade. | 07:14 |
| Jeffrey4l | https://github.com/openstack/kolla/blob/stable/mitaka/docker/nova/nova-api/extend_start.sh#L5 | 07:15 |
| Jeffrey4l | https://github.com/openstack/kolla/blob/stable/mitaka/ansible/roles/nova/tasks/bootstrap_service.yml#L8 | 07:15 |
| Jeffrey4l | check this two links ^^ | 07:15 |
| *** salv-orlando has joined #openstack-kolla | 07:15 | |
| Jeffrey4l | duonghq, ^^ | 07:16 |
| duonghq | but, it's done on all controller node? | 07:16 |
| Jeffrey4l | duonghq, no. only the first node of nova-api | 07:16 |
| duonghq | ah, seen | 07:16 |
| Jeffrey4l | https://github.com/openstack/kolla/blob/stable/mitaka/ansible/roles/nova/tasks/bootstrap_service.yml#L20 | 07:16 |
| Jeffrey4l | delete_to | 07:16 |
| duonghq | Jeffrey4l, understodod | 07:16 |
| Jeffrey4l | ;) | 07:17 |
| duonghq | Jeffrey4l, seem that we haven't had logic for keystone yet :) | 07:18 |
| Jeffrey4l | duonghq, we have here https://github.com/openstack/kolla/blob/stable/mitaka/ansible/roles/keystone/tasks/bootstrap_service.yml | 07:19 |
| duonghq | Jeffrey4l, I mean specific logic here | 07:19 |
| duonghq | http://docs.openstack.org/developer/keystone/upgrading.html | 07:19 |
| duonghq | you can see for db, it take at least 3 command | 07:19 |
| *** b_bezak has joined #openstack-kolla | 07:20 | |
| duonghq | db_sync (expand migrate contract) | 07:20 |
| duonghq | so, the upgrading logic is dedicated for each service, we should take care of that | 07:21 |
| Jeffrey4l | i only found `keystone-manage db_sync ` command. where is your said 3 commands | 07:22 |
| duonghq | 1 command with 3 option for 3 phase, | 07:23 |
| duonghq | sorry for confusing | 07:23 |
| *** tonanhngo has joined #openstack-kolla | 07:23 | |
| Jeffrey4l | i still can not get your point? 3 phase is? | 07:24 |
| *** tonanhngo has quit IRC | 07:24 | |
| *** matrohon has joined #openstack-kolla | 07:24 | |
| duonghq | Jeffrey4l, yes | 07:25 |
| *** zhurong has quit IRC | 07:26 | |
| Jeffrey4l | could u explain more? | 07:29 |
| duonghq | Jeffrey4l: keystone need 3 phase for db upgrade, but in Kolla, we have not implemented it | 07:31 |
| Serlex | Hello, cheeky help with murano ansible? | 07:32 |
| duonghq | https://github.com/openstack/kolla/blob/stable/mitaka/ansible/roles/keystone/tasks/bootstrap_service.yml | 07:32 |
| duonghq | https://github.com/openstack/kolla/blob/stable/mitaka/docker/keystone/extend_start.sh | 07:32 |
| Jeffrey4l | got it. you are saying db_sync --expand db_sync --migrate and db_sync --contract | 07:32 |
| Jeffrey4l | right? | 07:33 |
| duonghq | right, | 07:33 |
| Jeffrey4l | yes. kolla doesn't implement such. | 07:33 |
| duonghq | Keystone needs 9 step execute in this order | 07:34 |
| *** openstackgerrit has quit IRC | 07:34 | |
| duonghq | correct order | 07:34 |
| *** openstackgerrit has joined #openstack-kolla | 07:34 | |
| Jeffrey4l | yes. kolla should support this. I will file a bug and hope we can support keystone ugprade without downtime. | 07:34 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix neutron upgrade permission issue https://review.openstack.org/387212 | 07:35 |
| Jeffrey4l | fyi duonghq bug is created https://bugs.launchpad.net/kolla/+bug/1634016 | 07:37 |
| openstack | Launchpad bug 1634016 in kolla "kolla do not support keystone upgrade without downtime" [High,New] | 07:37 |
| duonghq | roger Jeffrey4l | 07:38 |
| *** mnasiadka has joined #openstack-kolla | 07:40 | |
| bjolo__ | have something changed in the buildscripts in the last few weeks? my buildtimes have gone up from about 40 min to 80min | 07:41 |
| bjolo__ | could be my build env as well (VMs on vmware), hence asking before going on goose chase | 07:42 |
| *** berendt has joined #openstack-kolla | 07:42 | |
| bjolo__ | anyone else seen an increase in build times? | 07:43 |
| *** steve-noyes has joined #openstack-kolla | 07:44 | |
| *** shardy has joined #openstack-kolla | 07:46 | |
| duonghq | Jeffrey4l, may I take care this bug? | 07:47 |
| Jeffrey4l | duonghq, yes. of curse. | 07:48 |
| duonghq | thank Jeffrey4l | 07:48 |
| Jeffrey4l | bjolo__, the ci build time seems OK. | 07:48 |
| Jeffrey4l | maybe network issue? | 07:48 |
| bjolo__ | tnx Jeffrey4l | 07:49 |
| Jeffrey4l | np. | 07:49 |
| bjolo__ | i need to get more serious about my build env | 07:49 |
| Jeffrey4l | kolla is using more repo mirror. so it is unstable sometime. | 07:51 |
| bjolo__ | started small with only one type manually, now im doing ubuntu/centos/source/binary in serial. ubuntu binary seems to be the slowest | 07:53 |
| *** g3ek has quit IRC | 07:56 | |
| *** tonanhngo has joined #openstack-kolla | 07:57 | |
| openstackgerrit | Merged openstack/kolla: Fix watcher upgrade permission issue https://review.openstack.org/386923 | 07:57 |
| *** tonanhngo has quit IRC | 07:57 | |
| openstackgerrit | Merged openstack/kolla: Fix rally upgrade permission issue https://review.openstack.org/386927 | 08:00 |
| *** haplo37 has quit IRC | 08:01 | |
| *** haplo37 has joined #openstack-kolla | 08:02 | |
| *** papacz has joined #openstack-kolla | 08:03 | |
| *** neilus has joined #openstack-kolla | 08:04 | |
| openstackgerrit | Merged openstack/kolla: Fix mariadb upgrade permission issue https://review.openstack.org/386515 | 08:05 |
| openstackgerrit | Merged openstack/kolla: Delete the unnecessary space https://review.openstack.org/386887 | 08:05 |
| *** g3ek has joined #openstack-kolla | 08:05 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: add panko dockerfile https://review.openstack.org/387232 | 08:09 |
| *** neilus has quit IRC | 08:09 | |
| *** tonanhngo has joined #openstack-kolla | 08:17 | |
| *** papacz has quit IRC | 08:17 | |
| *** tonanhngo has quit IRC | 08:18 | |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Improve VIP existence check https://review.openstack.org/381589 | 08:25 |
| *** athomas has joined #openstack-kolla | 08:25 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix heat upgrade permission issue https://review.openstack.org/387252 | 08:27 |
| duonghq | coolsvap, can you review this psanchez https://review.openstack.org/#/c/377154/ | 08:28 |
| duonghq | *patchset | 08:28 |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix mariadb upgrade permission issue https://review.openstack.org/387253 | 08:30 |
| *** nradojevic has joined #openstack-kolla | 08:33 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix elasticsearch upgrade permission issue https://review.openstack.org/387255 | 08:36 |
| *** tonanhngo has joined #openstack-kolla | 08:37 | |
| *** portdirect has joined #openstack-kolla | 08:38 | |
| *** tonanhngo has quit IRC | 08:38 | |
| *** papacz has joined #openstack-kolla | 08:44 | |
| openstackgerrit | Merged openstack/kolla: Cinder endpoint wrong fix https://review.openstack.org/387078 | 08:48 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Use correct Kibana version on the gates https://review.openstack.org/379815 | 08:50 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Fix log directory of Elasticsearch https://review.openstack.org/387268 | 08:51 |
| *** tonanhngo has joined #openstack-kolla | 08:54 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Merge run commands in elasticsearch container https://review.openstack.org/387272 | 08:55 |
| *** tonanhngo has quit IRC | 08:55 | |
| *** portdirect has quit IRC | 09:02 | |
| *** msimonin has joined #openstack-kolla | 09:02 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix mariadb upgrade permission issue https://review.openstack.org/387279 | 09:03 |
| *** ankush has joined #openstack-kolla | 09:03 | |
| openstackgerrit | Merged openstack/kolla: Fix elasticsearch upgrade permission issue https://review.openstack.org/386523 | 09:04 |
| *** awebber has joined #openstack-kolla | 09:05 | |
| *** rmart04 has joined #openstack-kolla | 09:08 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix elasticsearch upgrade permission issue https://review.openstack.org/387283 | 09:10 |
| duonghq | why it's reported as FAILED? http://logs.openstack.org/77/386977/1/check/gate-kolla-dsvm-build-centos-source-centos-7-nv/b3dbbc8/console.html | 09:17 |
| duonghq | just wonder about failure reasons | 09:17 |
| msimonin | Hi Kolla team | 09:21 |
| msimonin | Steve proposed help for https://bugs.launchpad.net/kolla/+bug/1617334 | 09:21 |
| openstack | Launchpad bug 1617334 in kolla "reconfigure action fails on [neutron | Restart the neutron_openvswitch_agent container] " [Critical,Incomplete] | 09:21 |
| *** tonanhngo has joined #openstack-kolla | 09:22 | |
| *** tonanhngo has quit IRC | 09:23 | |
| *** gfidente has joined #openstack-kolla | 09:23 | |
| *** gfidente has joined #openstack-kolla | 09:23 | |
| *** salv-orlando has quit IRC | 09:26 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix senlin upgrade permission issue https://review.openstack.org/387296 | 09:28 |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix rally upgrade permission issue https://review.openstack.org/387297 | 09:30 |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix watcher upgrade permission issue https://review.openstack.org/387300 | 09:32 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Fix log directory of Elasticsearch https://review.openstack.org/387268 | 09:33 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Fix log directory of Elasticsearch https://review.openstack.org/387268 | 09:34 |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix mongodb upgrade permission issue https://review.openstack.org/387305 | 09:41 |
| *** yingjun has quit IRC | 09:42 | |
| *** yingjun has joined #openstack-kolla | 09:42 | |
| *** yingjun has quit IRC | 09:46 | |
| openstackgerrit | Praveen N proposed openstack/kolla: Corrected directory for Elasticsearch https://review.openstack.org/387310 | 09:48 |
| *** tovin07 has quit IRC | 09:51 | |
| *** ssurana has joined #openstack-kolla | 09:51 | |
| *** duonghq has quit IRC | 09:56 | |
| *** ssurana has quit IRC | 09:56 | |
| *** zhangyufei has joined #openstack-kolla | 09:58 | |
| openstackgerrit | Vladislav Belogrudov proposed openstack/kolla: Ceilometer cannot create standalone mysql database https://review.openstack.org/387319 | 10:02 |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix influxdb upgrade permission issue https://review.openstack.org/387325 | 10:07 |
| *** mnasiadka has quit IRC | 10:11 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: add panko dockerfile https://review.openstack.org/387232 | 10:16 |
| openstackgerrit | Merged openstack/kolla: Fix senlin upgrade permission issue https://review.openstack.org/387296 | 10:17 |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix senlin upgrade permission issue https://review.openstack.org/387333 | 10:18 |
| *** tonanhngo has joined #openstack-kolla | 10:19 | |
| *** tonanhngo has quit IRC | 10:20 | |
| openstackgerrit | Merged openstack/kolla: Fix manila upgrade permission issue https://review.openstack.org/386920 | 10:27 |
| *** salv-orlando has joined #openstack-kolla | 10:27 | |
| *** yingjun has joined #openstack-kolla | 10:28 | |
| openstackgerrit | Vladislav Belogrudov proposed openstack/kolla: Ceilometer cannot create standalone mysql database https://review.openstack.org/387319 | 10:28 |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix manila upgrade permission issue https://review.openstack.org/387337 | 10:29 |
| *** huhaoran has quit IRC | 10:29 | |
| *** salv-orlando has quit IRC | 10:31 | |
| *** tonanhngo has joined #openstack-kolla | 10:34 | |
| *** yingjun has quit IRC | 10:35 | |
| *** yingjun has joined #openstack-kolla | 10:35 | |
| *** tonanhngo has quit IRC | 10:35 | |
| *** gfidente has quit IRC | 10:36 | |
| *** yingjun has quit IRC | 10:39 | |
| *** sdake has joined #openstack-kolla | 10:40 | |
| *** sdake has quit IRC | 10:40 | |
| *** berendt has quit IRC | 10:40 | |
| *** janem has joined #openstack-kolla | 10:43 | |
| *** haplo37_ has quit IRC | 10:44 | |
| *** haplo37_ has joined #openstack-kolla | 10:46 | |
| *** rhallisey has joined #openstack-kolla | 10:47 | |
| *** msimonin has quit IRC | 10:48 | |
| *** gfidente has joined #openstack-kolla | 10:50 | |
| *** gfidente has joined #openstack-kolla | 10:50 | |
| *** rhallisey has quit IRC | 10:51 | |
| *** mnasiadka has joined #openstack-kolla | 10:51 | |
| *** zhangyufei has quit IRC | 10:52 | |
| *** rhallisey has joined #openstack-kolla | 10:53 | |
| bjolo__ | cinder_api not building on ubuntu binary? | 10:56 |
| *** salv-orlando has joined #openstack-kolla | 10:57 | |
| *** phuongnh has quit IRC | 10:59 | |
| *** tonanhngo has joined #openstack-kolla | 10:59 | |
| *** tonanhngo has quit IRC | 10:59 | |
| *** salv-orlando has quit IRC | 11:02 | |
| *** kproskurin has joined #openstack-kolla | 11:06 | |
| *** awebber has quit IRC | 11:08 | |
| *** derekjhyang has quit IRC | 11:10 | |
| *** gfidente has quit IRC | 11:12 | |
| *** mliima has joined #openstack-kolla | 11:14 | |
| mnasiadka | bjolo__: I've built it successfuly on Friday | 11:16 |
| bjolo__ | looks like whole ubuntu cinder is broken today | 11:17 |
| bjolo__ | both binary and source builts fails for me today | 11:18 |
| *** janem has quit IRC | 11:18 | |
| *** janem has joined #openstack-kolla | 11:20 | |
| bjolo__ | cinder-base fails | 11:20 |
| bjolo__ | ceph-common : Depends: python-cephfs (= 10.2.2-0ubuntu0.16.04.2) but it is not going to be installed | 11:20 |
| *** tonanhngo has joined #openstack-kolla | 11:20 | |
| *** huhaoran has joined #openstack-kolla | 11:20 | |
| *** salv-orlando has joined #openstack-kolla | 11:21 | |
| *** tonanhngo has quit IRC | 11:21 | |
| *** yingjun has joined #openstack-kolla | 11:22 | |
| *** salv-orl_ has joined #openstack-kolla | 11:25 | |
| *** zhubingbing has quit IRC | 11:26 | |
| *** salv-orlando has quit IRC | 11:28 | |
| *** salv-orl_ has quit IRC | 11:29 | |
| *** dave-mccowan has joined #openstack-kolla | 11:30 | |
| openstackgerrit | Merged openstack/kolla: add etcd to precheck https://review.openstack.org/378210 | 11:31 |
| *** berendt has joined #openstack-kolla | 11:33 | |
| *** ppalacios has joined #openstack-kolla | 11:33 | |
| sbezverk | pbourke: wrt https://review.openstack.org/#/c/375615/ , do you have second to chat? | 11:34 |
| mnasiadka | bjolo__: maybe that's an effect of dumping ceph.com repos? | 11:38 |
| bjolo__ | could be. looks like no ceph related containers build on ubuntu atm | 11:39 |
| *** salv-orlando has joined #openstack-kolla | 11:39 | |
| mnasiadka | not yet merged - https://review.openstack.org/#/c/387081/ | 11:40 |
| mnasiadka | so it's not the case :) | 11:41 |
| *** salv-orlando has quit IRC | 11:44 | |
| *** gfidente has joined #openstack-kolla | 11:47 | |
| bjolo__ | mnasiadka, can you try and do a build? | 11:51 |
| bjolo__ | i just created a bug https://bugs.launchpad.net/kolla/+bug/1634107 | 11:51 |
| openstack | Launchpad bug 1634107 in kolla "ubuntu builds fail on ceph packages" [Undecided,New] | 11:51 |
| mnasiadka | let me try to rebuild cinder | 11:52 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Update OpenStack service names https://review.openstack.org/387387 | 11:53 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Update OpenStack service names https://review.openstack.org/387387 | 11:53 |
| *** tonanhngo has joined #openstack-kolla | 11:55 | |
| *** tonanhngo has quit IRC | 11:56 | |
| *** pece has joined #openstack-kolla | 12:02 | |
| *** neilus has joined #openstack-kolla | 12:06 | |
| *** msimonin has joined #openstack-kolla | 12:06 | |
| *** neilus has quit IRC | 12:10 | |
| mnasiadka | bjolo__: cinder-api built successfuly with latest stable/newton | 12:10 |
| bjolo__ | what?! | 12:10 |
| bjolo__ | hmmmm | 12:11 |
| bjolo__ | ./build.py --registry dockreg.mydomain.net:4000 --push --nocache --threads 64 --base ubuntu --type binary --tag 3.0.0.161017b --logs-dir /tmp/kolla-ubuntu-binary/ | 12:11 |
| openstackgerrit | Merged openstack/kolla: Fix heat upgrade permission issue https://review.openstack.org/386512 | 12:13 |
| *** tonanhngo has joined #openstack-kolla | 12:14 | |
| openstackgerrit | Merged openstack/kolla: Fix congress upgrade permission issue https://review.openstack.org/386900 | 12:14 |
| openstackgerrit | Merged openstack/kolla: Fix mistral upgrade permission issue https://review.openstack.org/386917 | 12:14 |
| *** tonanhngo has quit IRC | 12:14 | |
| *** coolsvap has quit IRC | 12:18 | |
| *** yingjun has quit IRC | 12:19 | |
| *** yingjun has joined #openstack-kolla | 12:19 | |
| *** v1k0d3n has quit IRC | 12:20 | |
| *** Guest23990 is now known as zigo | 12:21 | |
| *** yingjun_ has joined #openstack-kolla | 12:22 | |
| *** yingjun has quit IRC | 12:23 | |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Add guide for Hitachi NAS Platform Driver for Manila https://review.openstack.org/386552 | 12:30 |
| bjolo__ | mnasiadka, strange that it works for you. ive tried on 3 different build hosts with fresh git clones. fails on all of them | 12:30 |
| bjolo__ | what build string do you have? | 12:31 |
| mnasiadka | --base ubuntu --type binary --nocache | 12:32 |
| *** zhurong has joined #openstack-kolla | 12:35 | |
| mnasiadka | bjolo__: tried on another host, and now it fails | 12:35 |
| bjolo__ | thank you! :) | 12:35 |
| *** steve-noyes has left #openstack-kolla | 12:36 | |
| bjolo__ | best failure ever :) | 12:36 |
| *** yingjun_ has quit IRC | 12:37 | |
| *** yingjun has joined #openstack-kolla | 12:38 | |
| *** gfidente has quit IRC | 12:39 | |
| *** haplo37 has quit IRC | 12:39 | |
| *** g3ek has quit IRC | 12:40 | |
| *** denaitre has joined #openstack-kolla | 12:41 | |
| *** haplo37 has joined #openstack-kolla | 12:41 | |
| *** g3ek has joined #openstack-kolla | 12:42 | |
| *** eaguilar has joined #openstack-kolla | 12:43 | |
| mnasiadka | bjolo__: well, the gate tests (checked on some recent stable/newton gerrit topic) for ubuntu also are failing... | 12:46 |
| *** steve-noyes has joined #openstack-kolla | 12:46 | |
| bjolo__ | ok bug confirmed | 12:46 |
| bjolo__ | i was going nuts for a while here | 12:47 |
| *** tonanhngo has joined #openstack-kolla | 12:48 | |
| bjolo__ | Jeffrey4l, around? | 12:54 |
| *** zhurong has quit IRC | 12:55 | |
| *** zhurong has joined #openstack-kolla | 12:55 | |
| *** strigazi1AFK has joined #openstack-kolla | 12:55 | |
| *** strigazi_AFK has quit IRC | 12:58 | |
| mnasiadka | bjolo__: but that bug should affect binary, but not source | 12:59 |
| bjolo__ | let me verify with a new source build | 13:01 |
| sbezverk | rhallisey: ping | 13:03 |
| rhallisey | sbezverk, hey | 13:03 |
| sbezverk | rhallisey: I would like to propose to use inc0 nucs as a plan B, but use sdake's 5 servers as a plan A especially since I have them completly prepared and tested multiple times failover. | 13:04 |
| rhallisey | sbezverk, ok | 13:05 |
| rhallisey | sbezverk, that works better | 13:05 |
| rhallisey | show a shutdown of one on the screen works | 13:05 |
| sbezverk | rhallisey: it is just I am really really concerned to build inc0 nucs in Barcelona | 13:05 |
| rhallisey | also less setup when we arrive | 13:05 |
| sbezverk | rhallisey: the only thing to make sure is connectivity | 13:06 |
| sbezverk | to the internet | 13:06 |
| bjolo__ | source is only openstack source. ceph and related packages are always binary right | 13:06 |
| mnasiadka | ah ok, sorry | 13:06 |
| bjolo__ | i presume | 13:07 |
| rhallisey | sbezverk, should be ok in the conference | 13:07 |
| bjolo__ | build is sway | 13:07 |
| bjolo__ | away | 13:07 |
| rhallisey | sbezverk, let's still setup the nucs | 13:07 |
| sbezverk | rhallisey: I see failover time between 3 to 3.5 minutes | 13:07 |
| rhallisey | gotcha | 13:07 |
| rhallisey | nice | 13:07 |
| *** jtriley has joined #openstack-kolla | 13:08 | |
| *** eaguilar_ has joined #openstack-kolla | 13:08 | |
| sbezverk | if we have hangouts this week I can demonstrate | 13:08 |
| *** nradojevic1 has joined #openstack-kolla | 13:08 | |
| *** eaguilar has quit IRC | 13:09 | |
| *** nradojevic has quit IRC | 13:09 | |
| rhallisey | sbezverk, ok | 13:09 |
| *** spsingh has joined #openstack-kolla | 13:09 | |
| rhallisey | maybe on web | 13:09 |
| rhallisey | wed* | 13:09 |
| sbezverk | rhallisey: please check this bug: https://bugs.launchpad.net/neutron/+bug/1634123 I hit on a cluster reboot | 13:10 |
| openstack | Launchpad bug 1634123 in neutron "neutron openvswitch agent exists if unix:/var/run/openvswitch/db.sock is not yet created" [High,New] | 13:10 |
| rhallisey | sbezverk, agree that's important | 13:11 |
| rhallisey | workflow will help a bit | 13:11 |
| rhallisey | but it's best not to rely on it | 13:11 |
| sbezverk | rhallisey: interestingly neutron people marked it as high!! wow.. did not expect that :-) | 13:12 |
| rhallisey | ya :) | 13:12 |
| *** ankush has quit IRC | 13:14 | |
| sbezverk | rhallisey: in this specific case "cluster reboot" workflow unfortunately will not help.. as all pods are scheduled by kube-scheduler | 13:14 |
| rhallisey | because vswitchd and db are in the same pod | 13:15 |
| rhallisey | ? | 13:16 |
| rhallisey | oh this is only for reboot | 13:16 |
| rhallisey | right I see | 13:17 |
| *** ankush has joined #openstack-kolla | 13:17 | |
| rhallisey | sbezverk, this case is an argument for why there shouldn't be an external workflow | 13:17 |
| *** huhaoran has quit IRC | 13:19 | |
| *** gfidente has joined #openstack-kolla | 13:20 | |
| *** mgiles has joined #openstack-kolla | 13:20 | |
| sbezverk | rhallisey: :-) | 13:21 |
| *** schwicht has joined #openstack-kolla | 13:21 | |
| *** mgiles has quit IRC | 13:27 | |
| sbezverk | rhallisey: if you could review this PS would be great: https://review.openstack.org/#/c/375615 if it is in master it will be easier to deploy | 13:28 |
| rhallisey | sure | 13:28 |
| *** dwalsh has joined #openstack-kolla | 13:35 | |
| *** yingjun has quit IRC | 13:40 | |
| *** yingjun has joined #openstack-kolla | 13:41 | |
| *** wirehead_ has quit IRC | 13:42 | |
| *** yingjun has quit IRC | 13:45 | |
| *** wirehead_ has joined #openstack-kolla | 13:48 | |
| *** dwalsh has quit IRC | 13:51 | |
| *** inc0 has joined #openstack-kolla | 13:54 | |
| inc0 | good mornign | 13:55 |
| *** portdirect has joined #openstack-kolla | 13:58 | |
| *** eaguilar_ has quit IRC | 14:00 | |
| *** eaguilar has joined #openstack-kolla | 14:01 | |
| *** haplo37_ has quit IRC | 14:02 | |
| *** hogepodge has quit IRC | 14:04 | |
| *** haplo37_ has joined #openstack-kolla | 14:05 | |
| *** vks100 has joined #openstack-kolla | 14:06 | |
| britthouser3 | morning inc0! | 14:06 |
| *** tonanhngo has quit IRC | 14:06 | |
| *** tonanhngo has joined #openstack-kolla | 14:07 | |
| inc0 | wassup britthouser3 ? | 14:08 |
| portdirect | hey kfox1111, are you using planning to use init containers for kolla-kubernetes? | 14:08 |
| *** zhubingbing has joined #openstack-kolla | 14:08 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Fix memcache configuration of ceilometer https://review.openstack.org/387460 | 14:09 |
| *** jtriley has quit IRC | 14:12 | |
| *** jtriley has joined #openstack-kolla | 14:14 | |
| awiddersheim | Could use some help figuring out how cinder things work | 14:15 |
| awiddersheim | in the kolla world | 14:15 |
| awiddersheim | specifically things like iscsi or nfs | 14:15 |
| awiddersheim | trying to get NFS working now, cinder-volume mount the NFS share and create the volume | 14:16 |
| awiddersheim | then it looks like nova-compute can also mount the NFS share | 14:16 |
| sbezverk | inc0: morning.. remember you said you would check this ps ;-) https://review.openstack.org/375615 | 14:16 |
| awiddersheim | but then in the libvirtd logs it complains about not being able to access the volume | 14:16 |
| awiddersheim | I'm not sure if libvirtd is supposed to mount the NFS storage or if like | 14:17 |
| inc0 | ehh | 14:17 |
| awiddersheim | nova-compute does it for it and somehow libvirtd is supposed ot see it | 14:17 |
| inc0 | do we really have to do this now sbezverk ? | 14:17 |
| inc0 | can we wait till summit at least? | 14:17 |
| inc0 | so we can discuss | 14:17 |
| inc0 | what will be replacement of heka? | 14:18 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Fencing pod https://review.openstack.org/383922 | 14:18 |
| inc0 | but ok, I'll review it later | 14:18 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Fix configuration templates used for Ceilometer https://review.openstack.org/387477 | 14:18 |
| inc0 | Jeffrey4l, run:shared will fix stale namespace issue? | 14:19 |
| inc0 | that seems...safe | 14:19 |
| inc0 | ;) | 14:19 |
| sbezverk | inc0: it is for kube, does not impact kolla at all | 14:20 |
| *** vks100 has quit IRC | 14:20 | |
| sbezverk | it is just to bring new image, since kolla supplies us with images | 14:20 |
| inc0 | sbezverk, but we want to have same logging arch between kube and kolla | 14:20 |
| inc0 | ansible | 14:21 |
| inc0 | and since we already have this discussion | 14:21 |
| sbezverk | inc0: sure thing, in kube we desinged that is supports multiple collectors, fluentd is first one | 14:21 |
| inc0 | we could wait a bit and make it properly from day one | 14:21 |
| sbezverk | if later we decide to use different it is perfectly fine | 14:21 |
| inc0 | I'm just afraid that if it will work we'll never really change it | 14:22 |
| sbezverk | inc0: we need logging now | 14:22 |
| inc0 | and we'll have dirty state | 14:22 |
| *** dotplus_ is now known as dotplus | 14:22 | |
| sbezverk | inc0: also my kubetoolbox image depends on it so I cannot merge it :-( | 14:23 |
| sbezverk | inc0: again it is just another image, nothing prevents it to replace when kolla decides whihc tool to use | 14:23 |
| sbezverk | but to wait when it will be decided is not fair for kube as we need logging | 14:24 |
| *** coolsvap has joined #openstack-kolla | 14:25 | |
| inc0 | ok you've made your point, but when we move on with kolla-ansible, we'll replace fluentd with whatever we decide on | 14:26 |
| inc0 | so try to minimize your dependency on fluentd plz | 14:27 |
| sbezverk | inc0: sure thing, as I said nothing on our side prevents from using another collector it is very pluggable and does not require any massive changes for pods | 14:28 |
| *** eaguilar_ has joined #openstack-kolla | 14:28 | |
| *** ankush has quit IRC | 14:29 | |
| *** eaguilar has quit IRC | 14:29 | |
| *** michauds has joined #openstack-kolla | 14:31 | |
| sbezverk | inc0: rhallisey: kfox1111: I just updated kube to version 1.4.3 and with fencing pod I got 1m36s complete failover time :-) | 14:34 |
| rhallisey | wo | 14:34 |
| rhallisey | fast | 14:34 |
| sbezverk | rhallisey: I saw there were lots of issues fixed between 14.0 and 1.4.3 | 14:35 |
| *** srwilkers has joined #openstack-kolla | 14:41 | |
| *** dmsimard is now known as dmsimard|afk | 14:49 | |
| *** shasha_t_ has joined #openstack-kolla | 15:00 | |
| *** shasha_t_ is now known as shasha_tavil | 15:00 | |
| *** neilus has joined #openstack-kolla | 15:00 | |
| *** nradojevic1 has quit IRC | 15:01 | |
| *** neilus has quit IRC | 15:09 | |
| *** srwilkers has quit IRC | 15:11 | |
| *** sdake has joined #openstack-kolla | 15:15 | |
| *** nradojevic has joined #openstack-kolla | 15:19 | |
| *** nradojevic has quit IRC | 15:19 | |
| *** zhurong has quit IRC | 15:22 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Fencing pod https://review.openstack.org/383922 | 15:22 |
| *** mnasiadka has quit IRC | 15:27 | |
| kfox1111 | sbezverk: cool. | 15:29 |
| kfox1111 | how much of that was the timeout and how much pulling the containers? | 15:29 |
| kfox1111 | if that is a significant part, you could use a daemon set with a command: sleep loop to precache the images to the controller nodes. | 15:31 |
| *** awebber has joined #openstack-kolla | 15:31 | |
| *** macsz_ has quit IRC | 15:31 | |
| *** macsz_ has joined #openstack-kolla | 15:32 | |
| *** b_bezak has quit IRC | 15:32 | |
| *** macsz_ is now known as macsz | 15:32 | |
| *** macsz has quit IRC | 15:33 | |
| *** macsz has joined #openstack-kolla | 15:33 | |
| *** Serlex has quit IRC | 15:35 | |
| *** lamt has joined #openstack-kolla | 15:39 | |
| *** harlowja_at_home has joined #openstack-kolla | 15:41 | |
| *** steve-noyes has quit IRC | 15:45 | |
| *** jistr is now known as jistr|biab | 15:46 | |
| *** steve-noyes has joined #openstack-kolla | 15:47 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Adding customer interface for keepalived https://review.openstack.org/387538 | 15:49 |
| kfox1111 | if that is a significant part, you could use a daemon set with a command: sleep loop to precache the images to the controller nodes.f | 15:50 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Adding custom api interface for keepalived https://review.openstack.org/387538 | 15:50 |
| *** athomas has quit IRC | 15:51 | |
| kfox1111 | sbezverk: is it tested? | 15:51 |
| *** berendt has quit IRC | 15:52 | |
| *** david-lyle_ has joined #openstack-kolla | 15:52 | |
| *** david-lyle has quit IRC | 15:53 | |
| *** hogepodge has joined #openstack-kolla | 15:53 | |
| *** eaguilar_ has quit IRC | 15:54 | |
| *** sdake_ has joined #openstack-kolla | 15:55 | |
| *** eaguilar has joined #openstack-kolla | 15:56 | |
| *** athomas has joined #openstack-kolla | 15:56 | |
| *** sdake has quit IRC | 15:58 | |
| *** mandre has quit IRC | 15:59 | |
| sdake_ | morning peeps | 16:02 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Adding custom api interface for keepalived https://review.openstack.org/387538 | 16:02 |
| sdake_ | anyone got an update on how rc3 is looking? | 16:03 |
| *** pritesh has joined #openstack-kolla | 16:05 | |
| inc0 | sdake_, https://review.openstack.org/#/q/topic:bug/1631503 | 16:06 |
| *** janem has quit IRC | 16:07 | |
| *** HyperJohnGraham_ has quit IRC | 16:07 | |
| sdake_ | this needs an ack: https://review.openstack.org/#/c/387081/ | 16:08 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Adding custom api interface for keepalived https://review.openstack.org/387538 | 16:08 |
| *** srwilkers has joined #openstack-kolla | 16:08 | |
| *** rmart04 has quit IRC | 16:11 | |
| sbezverk | kfox1111: just tested it | 16:12 |
| *** eaguilar has quit IRC | 16:12 | |
| sbezverk | it takes value from the file now | 16:12 |
| sbezverk | kfox1111: the logest part now in failover is PodInitialization and PodCreation phases.. | 16:13 |
| *** mandre has joined #openstack-kolla | 16:15 | |
| *** awebber has quit IRC | 16:17 | |
| *** vhosakot has joined #openstack-kolla | 16:20 | |
| openstackgerrit | Merged openstack/kolla: Remove ceph repo for Ubuntu install https://review.openstack.org/387081 | 16:20 |
| *** jistr|biab is now known as jistr | 16:22 | |
| *** janem has joined #openstack-kolla | 16:22 | |
| *** matrohon has quit IRC | 16:22 | |
| *** david-lyle_ is now known as david-lyle | 16:24 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Remove ceph repo for Ubuntu install https://review.openstack.org/387556 | 16:24 |
| *** diogogmt has joined #openstack-kolla | 16:28 | |
| *** dmsimard|afk is now known as dmsimard | 16:28 | |
| sdake_ | inc0 looks lik ea mountain of upgrade patches | 16:29 |
| sdake_ | rhallisey inc0 can you ack https://review.openstack.org/#/c/387556/ | 16:30 |
| sdake_ | or pbourke_ ^ | 16:30 |
| *** tonanhngo has quit IRC | 16:31 | |
| *** unicell1 has quit IRC | 16:31 | |
| openstackgerrit | Merged openstack/kolla: fix cron logrotate for sahara https://review.openstack.org/387130 | 16:37 |
| *** diogogmt has quit IRC | 16:37 | |
| *** diogogmt has joined #openstack-kolla | 16:38 | |
| *** bmace has joined #openstack-kolla | 16:40 | |
| sdake_ | inc0 or other cores, this needs an ack https://review.openstack.org/#/c/385559/ | 16:42 |
| sdake_ | folks - i'd like to get all patches merged and backported minus the upgrade patches | 16:44 |
| sdake_ | so we have 2 full days to work on the upgrade stuff | 16:44 |
| sdake_ | (reviewing thereof) | 16:44 |
| sdake_ | there are patches in queue for every bug in newton3 | 16:44 |
| *** jheroux has joined #openstack-kolla | 16:47 | |
| pbourke_ | sdake_: busy day today, will get across reviews tomorrow. Can ack that but better wait for the gate? | 16:47 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix stale namespace removal issue https://review.openstack.org/387563 | 16:47 |
| *** zhubingbing has quit IRC | 16:47 | |
| openstackgerrit | Merged openstack/kolla: Fix stale namespace removal issue https://review.openstack.org/385559 | 16:48 |
| sdake_ | pbourke_ if you +2 i'll +w later when gate wraps up | 16:48 |
| *** dwalsh has joined #openstack-kolla | 16:48 | |
| kfox1111 | sbezverk: so precaching the image might help then. | 16:48 |
| pbourke_ | done | 16:48 |
| sdake_ | this one too https://review.openstack.org/387563 | 16:48 |
| kfox1111 | a daemonset with sleep loop should help. | 16:48 |
| sbezverk | kfox1111: they are already sitting in docker | 16:48 |
| sdake_ | pbourke_ ^ :) | 16:48 |
| kfox1111 | ah. ok. | 16:48 |
| sdake_ | we have about 30 reviews for upgrades.. | 16:48 |
| * sdake_ groans | 16:48 | |
| sbezverk | cause I am bouncing between two controllers which I kill in turns | 16:49 |
| sbezverk | kfox1111: I tested keepalived change, it works | 16:49 |
| *** jtriley has quit IRC | 16:50 | |
| kfox1111 | sbezverk: k. ill workflow it then. | 16:50 |
| *** schwicht has quit IRC | 16:50 | |
| *** msimonin has quit IRC | 16:50 | |
| *** schwicht has joined #openstack-kolla | 16:51 | |
| *** msimonin1 has joined #openstack-kolla | 16:51 | |
| *** msimonin1 has quit IRC | 16:51 | |
| sdake_ | 150 people in channel - looks like new record ;) | 16:51 |
| *** kproskurin has quit IRC | 16:51 | |
| kfox1111 | sbezverk: currently keepalived is not in the gate, so testing against it doesnt' help. | 16:51 |
| kfox1111 | I tried enabling it, but it broke things. | 16:52 |
| kfox1111 | if you get a bit of time, you could try and re'enable it. | 16:52 |
| *** eaguilar has joined #openstack-kolla | 16:52 | |
| sdake_ | Daviey you about? | 16:52 |
| sdake_ | or egonzalez90 ? | 16:52 |
| sbezverk | kfox1111: hmmm I tested this change in the test bed I am playing with now, with different interfaces looks good to me. not sure what is with gate.. I guess at one point we will need to investigate.. | 16:53 |
| sdake_ | vhosakot ping | 16:54 |
| vhosakot | sdake_: pong | 16:54 |
| sdake_ | vhosakot need some reviews if you hav etime | 16:54 |
| vhosakot | sdake_: yes, will review today... sure thing | 16:55 |
| sbezverk | kfox1111: interesting observation, when fencing pod on the smae server as mariadb then when this node dies it takes longer to recover about 3 minutes, if on another live controller then about 1m30s | 16:55 |
| *** harlowja_at_home has quit IRC | 16:55 | |
| sdake_ | vhosakot i have selected reviews | 16:55 |
| sdake_ | we are 2 days away from releasing 3.0.0 | 16:55 |
| sdake_ | vhosakot i dont need master stuff reviewed necessarily, need specific things reviewed | 16:55 |
| vhosakot | sdake_: yep | 16:55 |
| sdake_ | vhosakot do you have time to do it now - i can walk you throug hthe changes needing review | 16:55 |
| vhosakot | sure send me the links.. or is there a topic or a gerrit search string? | 16:55 |
| sdake_ | vhosakot this needs a review https://review.openstack.org/#/c/387563/ | 16:56 |
| sdake_ | note its a backport | 16:56 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Adding custom api interface for keepalived https://review.openstack.org/387538 | 16:56 |
| sbezverk | kfox1111: stepping out for a couple of hours.. | 16:57 |
| vhosakot | sdake_: yep, reviewwing now | 16:57 |
| kfox1111 | sbezverk: probably because it turns itself off, then the other fencing agent has to go through a loop before the rbd lock is deleted. | 16:58 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Remove ceph repo for Ubuntu install https://review.openstack.org/387556 | 16:58 |
| *** lamt has quit IRC | 17:00 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla: Ceilometer cannot create standalone mysql database https://review.openstack.org/387319 | 17:01 |
| *** srwilkers has quit IRC | 17:01 | |
| sdake_ | vhosakot https://review.openstack.org/#/c/387563/ | 17:03 |
| sdake_ | vhosakot note also a backport | 17:03 |
| openstackgerrit | Merged openstack/kolla: Fix configuration templates used for Ceilometer https://review.openstack.org/387477 | 17:03 |
| sdake_ | note all - if you have a bug that is rc3 worthy and you already ahve a fix, please take care to bring it into the release asap | 17:04 |
| vhosakot | sdake_: done | 17:04 |
| sdake_ | rc3 worthy = fubared | 17:04 |
| sdake_ | kfox1111 sbezverk - you wanted to know when we are doing the final tag - oct 18th - recommend testing with latest master to make sure nothign is busted | 17:05 |
| *** unicell has joined #openstack-kolla | 17:06 | |
| pbourke_ | sdake_: are most of the permissions related upgrade patches are tested and essentially rubber stamp at this point? | 17:07 |
| pbourke_ | s/are tested/tested | 17:07 |
| kfox1111 | sdake_: k. will kick off a new gate run. | 17:07 |
| sdake_ | pbourke_ i dont think tested | 17:08 |
| sdake_ | pbourke_ the other patchess are rubber stamp at this point | 17:08 |
| sdake_ | (the ones not related to upgrade) | 17:08 |
| pbourke_ | :/ | 17:08 |
| sdake_ | right | 17:08 |
| sdake_ | huge mess huh | 17:08 |
| pbourke_ | upgrade gates should be priority #1 for O | 17:09 |
| sdake_ | pbourke_ we are just going to have to use our best judgement with the upgrade aptches or alternatively someone will have to pull em down one by one and make a repo | 17:09 |
| sdake_ | pbourke_ agree 100% | 17:09 |
| sdake_ | pbourke_ finding this problem at the last minute = not idela | 17:09 |
| pbourke_ | I'll try test some of them tomorrow | 17:09 |
| pbourke_ | just to ease my mind before approving | 17:09 |
| sdake_ | i htink the issue is they need to be tested ias a group | 17:09 |
| pbourke_ | I can test a subset at least | 17:10 |
| pbourke_ | the core ones | 17:10 |
| *** b_bezak has joined #openstack-kolla | 17:10 | |
| kfox1111 | sdake_: tests running. should know in about 20min if all goes well. | 17:11 |
| sdake_ | kfox1111 are you running against 2.0.0 images? | 17:11 |
| sdake_ | the 3.0.0 images haven't been pushed yet | 17:11 |
| kfox1111 | sdake_: yeah. no 3.0.0 images to test with. | 17:11 |
| rhallisey | sdake_, I was going to tag today | 17:11 |
| *** b_bezak has quit IRC | 17:12 | |
| kfox1111 | the images don't look to change much though. most of the breakage we've seen is in the ansible bits. | 17:12 |
| rhallisey | you're talking about kolla right? | 17:12 |
| kfox1111 | yeah. | 17:12 |
| rhallisey | kk | 17:12 |
| *** pece has quit IRC | 17:12 | |
| *** b_bezak has joined #openstack-kolla | 17:12 | |
| rhallisey | gotcha | 17:12 |
| *** sdake has joined #openstack-kolla | 17:14 | |
| *** ssurana has joined #openstack-kolla | 17:16 | |
| *** b_bezak has quit IRC | 17:17 | |
| *** sdake_ has quit IRC | 17:17 | |
| *** tonanhngo has joined #openstack-kolla | 17:18 | |
| *** jtriley has joined #openstack-kolla | 17:21 | |
| sdake | pbourke_ is this critical for 3.0.0 https://bugs.launchpad.net/kolla/+bug/1634083 | 17:24 |
| openstack | Launchpad bug 1634083 in kolla "ceilometer cannot use standalone host" [Undecided,In progress] - Assigned to Vladislav Belogrudov (vlad-belogrudov) | 17:24 |
| openstackgerrit | Merged openstack/kolla: Fix docs formatting error https://review.openstack.org/380161 | 17:24 |
| openstackgerrit | Merged openstack/kolla: Update OpenStack service names https://review.openstack.org/387387 | 17:25 |
| *** jtriley has quit IRC | 17:26 | |
| *** ssurana has quit IRC | 17:26 | |
| sdake | Jeffrey4l ping re upgrade patches | 17:27 |
| kfox1111 | sdake: can you ping me once patches stop merging? it would be good to test right before the release. | 17:27 |
| sdake | kfox1111 that will probably be 23:45 tomorrow | 17:27 |
| sdake | god willing | 17:28 |
| kfox1111 | heh. k. | 17:28 |
| sdake | and i wont have much time to ping between tagging and releasing - we have over 30 patches related to upgrade | 17:28 |
| sdake | that all need backports | 17:28 |
| sdake | kfox1111 i think what would be wise for you is to test prior to backport of the upgrade patchset | 17:29 |
| kfox1111 | its, fine, so long as they don't touch the config/templates. but if they do, we should test. | 17:29 |
| sdake | once everything else is merged | 17:29 |
| sdake | they touch the json... | 17:29 |
| kfox1111 | I don't really have a way to test unmerged kolla bits. :/ | 17:29 |
| kfox1111 | we probably should come up with a gate for that though. | 17:29 |
| sdake | i could produce a patch stream out of them | 17:29 |
| sdake | then it would be one git cherrypick operation from gerrit | 17:29 |
| kfox1111 | as the kolla-kubernetes gate has picked up at least one ceph related bug in kolla already. :) | 17:30 |
| kfox1111 | yeah, that might work. | 17:30 |
| kfox1111 | maybe we do that tomorrow afternoon? | 17:30 |
| sdake | kfox1111 i'll let you knwo when i get to that - it needs to happen today | 17:30 |
| kfox1111 | I'm in the middle of an outage today. | 17:30 |
| sdake | tomororw is tag day | 17:30 |
| kfox1111 | (tenhcchnically tomorrow too, b ut should get most of the stuff done today) | 17:31 |
| sdake | the cherrypick part | 17:31 |
| sdake | i guess the testing can wait until tomorrow | 17:31 |
| sdake | but owuld prefer not to wait | 17:31 |
| kfox1111 | I'd like to do the test as close to release as possible, (or at least one of the tests) just to ensure the release is sane. | 17:31 |
| openstackgerrit | Merged openstack/kolla: Fix stale namespace removal issue https://review.openstack.org/387563 | 17:32 |
| kfox1111 | it only takes about 20 min, so multiple is ok. | 17:32 |
| kfox1111 | so far so good. most recent test passed. | 17:32 |
| sdake | did matt simonin join today to work on https://bugs.launchpad.net/kolla/+bug/1617334 | 17:32 |
| openstack | Launchpad bug 1617334 in kolla "reconfigure action fails on [neutron | Restart the neutron_openvswitch_agent container] " [Critical,Incomplete] | 17:32 |
| kfox1111 | well, at least with ubuntu. still having that very strange issue with centos libvirt and rbd volumes... | 17:33 |
| kfox1111 | bbiab | 17:33 |
| sdake | kfox1111 expand - not sure which wierd issue your talking about | 17:33 |
| *** ssurana has joined #openstack-kolla | 17:36 | |
| *** HyperJohnGraham_ has joined #openstack-kolla | 17:40 | |
| *** haplo37_ has quit IRC | 17:40 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Remove ceph repo for Ubuntu install https://review.openstack.org/387556 | 17:40 |
| *** bachp has quit IRC | 17:42 | |
| *** dave-mccowan has quit IRC | 17:42 | |
| *** haplo37_ has joined #openstack-kolla | 17:42 | |
| sdake | Jeffrey4l are you syaing this bug isn't critical: https://bugs.launchpad.net/kolla/+bug/1632811/comments/2 | 17:43 |
| openstack | Launchpad bug 1632811 in kolla "Keystone token table filling up" [Critical,In progress] - Assigned to Mathias Ewald (mewald) | 17:43 |
| sdake | Jeffrey4l or some other solution must be found? | 17:43 |
| *** dave-mccowan has joined #openstack-kolla | 17:43 | |
| sdake | inc0 what is the etherpad for upgrade bug | 17:44 |
| *** ayoung has joined #openstack-kolla | 17:45 | |
| *** berendt has joined #openstack-kolla | 17:46 | |
| sdake | berendt morning fine sir | 17:46 |
| sdake | berendt have any extra hp to spare today - tomorrow is tag day and upgrades = busted | 17:46 |
| sdake | berendt we have a mountain of patches in the review queue related to it, but nobody can work on it until tomorrow it appears.... | 17:46 |
| sdake | (when we tag.) | 17:46 |
| berendt | good evening fits better for me ;) | 17:47 |
| sdake | yar - morning for me :) | 17:47 |
| *** kproskurin has joined #openstack-kolla | 17:47 | |
| *** neilus has joined #openstack-kolla | 17:47 | |
| berendt | i will have a look on some more reviews now | 17:48 |
| *** HyperJohnGraham_ has quit IRC | 17:49 | |
| jascott1 | brew install --HEAD tsocks | 17:52 |
| *** sdake has quit IRC | 17:52 | |
| *** sdake has joined #openstack-kolla | 17:52 | |
| *** ChanServ sets mode: +o sdake | 17:53 | |
| *** kproskurin has quit IRC | 17:54 | |
| *** sdake changes topic to "FINAL NEWTON DEADLINE - OCT 18, 2016; Work on this bug -> https://etherpad.openstack.org/p/kolla-bug-1631503 ; Please read the documentation here: http://docs.openstack.org/developer/kolla/; Kolla IRC meetngs on Wednesdays @ 16:00 UTC - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b (old logs from #kolla http://goo.gl/VKpPzA)" | 17:54 | |
| *** bachp has joined #openstack-kolla | 17:56 | |
| *** HyperJohnGraham_ has joined #openstack-kolla | 17:57 | |
| openstackgerrit | Merged openstack/kolla: Ceilometer cannot create standalone mysql database https://review.openstack.org/387319 | 17:59 |
| openstackgerrit | Merged openstack/kolla: Fix common role upgrade permission issue https://review.openstack.org/387171 | 18:01 |
| *** lamt has joined #openstack-kolla | 18:01 | |
| sdake | pbourke_ re rubber stamp patches - anything that is a stable/newton backport is a rubber stamp | 18:02 |
| *** lamt has quit IRC | 18:03 | |
| sdake | pbourke_ so if you want to stamp those - that owuld help :) | 18:03 |
| sdake | berendt ^^ | 18:03 |
| sdake | inc0 ^^ | 18:03 |
| openstackgerrit | Merged openstack/kolla: Fix watcher upgrade permission issue https://review.openstack.org/387300 | 18:03 |
| sdake | even if the patch is broken, its already in master | 18:03 |
| sdake | we need to fix it in master if we see defects in teh backports | 18:03 |
| sdake | and backport a new patch | 18:03 |
| sdake | any Q? :) | 18:03 |
| berendt | it will be a lot of work :( | 18:04 |
| *** srwilkers has joined #openstack-kolla | 18:04 | |
| openstackgerrit | Merged openstack/kolla: Fix rally upgrade permission issue https://review.openstack.org/387297 | 18:06 |
| sdake | berendt ya and we got 1 day to do it in | 18:07 |
| openstackgerrit | Merged openstack/kolla: Fix elasticsearch upgrade permission issue https://review.openstack.org/387283 | 18:07 |
| *** ChanServ sets mode: -o sdake | 18:07 | |
| *** mliima_ has joined #openstack-kolla | 18:08 | |
| *** HyperJohnGraham_ has quit IRC | 18:09 | |
| *** vhosakot has quit IRC | 18:11 | |
| *** mliima has quit IRC | 18:11 | |
| *** lamt has joined #openstack-kolla | 18:12 | |
| *** jtriley has joined #openstack-kolla | 18:12 | |
| *** HyperJohnGraham_ has joined #openstack-kolla | 18:13 | |
| *** rhallisey has quit IRC | 18:16 | |
| sdake | looks like inc0 picked the hard one ;) https://review.openstack.org/#/c/386709/1 | 18:16 |
| sdake | inc0 Jeffrey4l did provide reviews there - not sure if you saw | 18:16 |
| inc0 | yeah, looking at it now | 18:17 |
| *** mliima__ has joined #openstack-kolla | 18:17 | |
| inc0 | I need to test one thing as I don't think it's correct | 18:17 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix cinder upgrade permission issue https://review.openstack.org/387597 | 18:18 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix sahara upgrade permission issue https://review.openstack.org/387600 | 18:18 |
| *** shardy is now known as shardy_afk | 18:19 | |
| inc0 | I need to reset env to test out things | 18:19 |
| inc0 | anyway, that's on me | 18:19 |
| inc0 | will be done today | 18:19 |
| openstackgerrit | Merged openstack/kolla: Fix cinder upgrade permission issue https://review.openstack.org/386930 | 18:19 |
| *** mliima_ has quit IRC | 18:20 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix magnum upgrade permission issue https://review.openstack.org/387603 | 18:20 |
| openstackgerrit | Merged openstack/kolla: Fix sahara upgrade permission issue https://review.openstack.org/386928 | 18:20 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix telegraf upgrade permission issue https://review.openstack.org/387604 | 18:20 |
| sdake | inc0 wfm | 18:21 |
| sdake | lets work on the quue folks and get thigns merged or fixed | 18:22 |
| sdake | related to the above bug | 18:22 |
| sdake | i think almost everything is in good shape | 18:22 |
| sdake | minus a few patches with -1s | 18:22 |
| sdake | stable/newton - please make sure passing gate -then ack it :) | 18:22 |
| *** MarMat has joined #openstack-kolla | 18:23 | |
| *** mliima_ has joined #openstack-kolla | 18:23 | |
| sdake | mliima_ - need help revieiwng bug 1631503 | 18:23 |
| openstack | bug 1631503 in kolla "inconsistent UID in named volume results in inability to upgrade" [Critical,In progress] https://launchpad.net/bugs/1631503 - Assigned to Steven Dake (sdake) | 18:23 |
| sdake | 17 patche sin queue | 18:24 |
| *** mliima__ has quit IRC | 18:25 | |
| *** steve-noyes1 has joined #openstack-kolla | 18:26 | |
| kfox1111 | back for a few. | 18:27 |
| openstackgerrit | Merged openstack/kolla: Fix mariadb upgrade permission issue https://review.openstack.org/387279 | 18:28 |
| kfox1111 | sdake: rbd backed cinder volumes are failing to attach to the vm's when the centos image is used: | 18:28 |
| openstackgerrit | Merged openstack/kolla: Fix magnum upgrade permission issue https://review.openstack.org/386934 | 18:29 |
| *** steve-noyes has quit IRC | 18:29 | |
| openstackgerrit | Merged openstack/kolla: Fix telegraf upgrade permission issue https://review.openstack.org/386935 | 18:29 |
| kfox1111 | sdake: http://logs.openstack.org/66/386966/18/experimental/gate-kolla-kubernetes-ansible-workflow-ceph-nv/0ce0d9a/logs/openstack/kolla-nova-compute-hu3fg.txt | 18:29 |
| kfox1111 | the config is the same as with the ubuntu image, and it works there. | 18:30 |
| kfox1111 | wait... that isn't a good example..... | 18:30 |
| kfox1111 | sec.. | 18:30 |
| *** steve-noyes has joined #openstack-kolla | 18:30 | |
| *** steve-noyes1 has quit IRC | 18:30 | |
| sdake | kfox1111 ya - you fixed that - in the 3.0.0 images | 18:32 |
| openstackgerrit | Merged openstack/kolla: Fix manila upgrade permission issue https://review.openstack.org/387337 | 18:32 |
| kfox1111 | sdake: I did? | 18:33 |
| sdake | kfox1111 someone fixed it | 18:33 |
| sdake | kfox1111 pretty sure | 18:33 |
| kfox1111 | no, I fixed an issue in the kolla tree about the cinder endpoint being wrong, but that affected both ubuntu and centos. | 18:33 |
| kfox1111 | there's one more I'm fighting where it only affects centos. | 18:33 |
| sdake | kfox1111 make it harder :) | 18:34 |
| kfox1111 | not sure where the error lies, as the config between ubuntu and centos are the same. | 18:34 |
| openstackgerrit | Merged openstack/kolla: Fix glance upgrade permission issue https://review.openstack.org/386978 | 18:34 |
| openstackgerrit | Merged openstack/kolla: Fix mongodb upgrade permission issue https://review.openstack.org/387305 | 18:34 |
| *** steve-noyes has quit IRC | 18:35 | |
| *** HyperJohnGraham_ has quit IRC | 18:35 | |
| openstackgerrit | Merged openstack/kolla: Fix log directory of Elasticsearch https://review.openstack.org/387268 | 18:35 |
| openstackgerrit | Merged openstack/kolla: Merge run commands in elasticsearch container https://review.openstack.org/387272 | 18:35 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Fix mongodb upgrade permission issue https://review.openstack.org/387615 | 18:36 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Fix log directory of Elasticsearch https://review.openstack.org/387616 | 18:36 |
| openstackgerrit | Merged openstack/kolla: Fix keystone upgrade permission issue https://review.openstack.org/386977 | 18:40 |
| sdake | berendt re https://review.openstack.org/#/c/387616/1 - this patch is not totally correct, the log files are lost in an upgrade | 18:40 |
| sdake | folks - if it isn't critical, don't cherry-pick it please | 18:43 |
| berendt | sdake yes, this only fixes the logpath | 18:43 |
| berendt | elasticsearch in heka is missing at the moment, i think this is an other bug | 18:43 |
| openstackgerrit | Merged openstack/kolla: Fix senlin upgrade permission issue https://review.openstack.org/387333 | 18:43 |
| berendt | it is the same for the kibana logpath (https://review.openstack.org/#/c/386980/) | 18:44 |
| *** pbourke_ has quit IRC | 18:46 | |
| *** pbourke_ has joined #openstack-kolla | 18:47 | |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Fix murano upgrade permission issue https://review.openstack.org/386922 | 18:48 |
| *** rhallisey has joined #openstack-kolla | 18:49 | |
| *** msimonin has joined #openstack-kolla | 18:50 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla: Updated from global requirements https://review.openstack.org/387628 | 18:50 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Add logrotate configuration for elasticsearch https://review.openstack.org/387630 | 18:51 |
| *** mliima__ has joined #openstack-kolla | 18:53 | |
| sdake | berendt if you see issue iwth https://review.openstack.org/#/c/387604/1 | 18:55 |
| *** mliima_ has quit IRC | 18:55 | |
| berendt | sdake i do not have a running telegraf service, i only found this volume in the tasks | 18:55 |
| sdake | ya - well -1 the review then :) | 18:55 |
| sdake | i did for ya | 18:56 |
| sdake | if its a master review - 1 em | 18:56 |
| berendt | was not sure if it is really required.. | 18:56 |
| sdake | if its a stable/newton review, rubber stamp and fix with followon patches | 18:56 |
| sdake | berendt right me either | 18:56 |
| sdake | i'm not sure why telegraph would have haproxy | 18:56 |
| sdake | a link to its socket i mean | 18:57 |
| berendt | maybe a copy & paste issue | 18:57 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Modular gate https://review.openstack.org/386966 | 18:58 |
| berendt | according to the config the stats URL is used to access haproxy stats | 18:58 |
| berendt | i think haproxy socket is not required | 18:58 |
| sdake | bjolo - wake up from keyboard :) | 18:58 |
| *** jmccarthy has quit IRC | 18:58 | |
| sdake | berendt performance monitoring may not be on the same host as the controllers (i.e. haproxy) | 18:59 |
| kfox1111 | berendt: I don't have nay context, but I use the socket all the time with a script called hatop on some of my other haproxy systems. | 18:59 |
| kfox1111 | its kind of nice. | 18:59 |
| berendt | sdake corret, because of the the input for haproxy inside telegraf is the stats URL and not the socket | 18:59 |
| sdake | kfox1111 ya - i get the value of the socket access | 18:59 |
| *** jmccarthy has joined #openstack-kolla | 19:00 | |
| berendt | kfox1111 is it not possible to use an URL with hatop? | 19:00 |
| sdake | berendt could you expand on that with a link to code? | 19:00 |
| berendt | sdake https://github.com/openstack/kolla/blob/master/ansible/roles/telegraf/templates/telegraf.conf.j2#L36-L39 | 19:00 |
| kfox1111 | berendt: doesn't look like it. | 19:01 |
| sdake | bjolo wake up :) | 19:01 |
| kfox1111 | I think the url is more static for gathering metrics. the hatop tool is more dynamic. | 19:01 |
| sdake | bjolo__ :) | 19:01 |
| berendt | kfox1111 you have to hit ctrl+r.. | 19:01 |
| sdake | bjolo__ hopefully our computer is buzzing :) | 19:01 |
| kfox1111 | berendt: in hatop? | 19:02 |
| berendt | kfox1111 no when using the webinterface | 19:02 |
| kfox1111 | berendt: ah. | 19:02 |
| kfox1111 | no, hatop is more interactive then that. | 19:02 |
| berendt | sdake i will open a bug report + review for this haproxy_socket volume | 19:02 |
| berendt | mewald should check this tomorrow | 19:03 |
| kfox1111 | it would be nice to ship hatop in the haproxy container at some point. | 19:03 |
| *** HyperJohnGraham_ has joined #openstack-kolla | 19:03 | |
| sdake | meeting - afk for 30 mins | 19:03 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Remove haproxy_socket volume from telegraf container https://review.openstack.org/387639 | 19:05 |
| berendt | i am also not sure why the docker.sock volume is required inside telegraf | 19:07 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Testing trunk... https://review.openstack.org/387641 | 19:08 |
| inc0 | Jeffrey4l, still up? | 19:08 |
| inc0 | I guess not | 19:08 |
| inc0 | but fuck, nova is going to be hard | 19:08 |
| inc0 | in terms of permissions | 19:08 |
| *** janem has quit IRC | 19:11 | |
| *** srwilkers has quit IRC | 19:11 | |
| *** msimonin has quit IRC | 19:13 | |
| *** spsingh has quit IRC | 19:15 | |
| *** dwalsh has quit IRC | 19:15 | |
| *** HyperJohnGraham_ has quit IRC | 19:17 | |
| berendt | sleeping time, see you tomorrow | 19:17 |
| *** berendt has quit IRC | 19:17 | |
| *** msimonin has joined #openstack-kolla | 19:18 | |
| sdake | msimonin are ou the cat that reported the bug with linuxbridge? | 19:19 |
| openstackgerrit | Merged openstack/kolla: Remove ceph repo for Ubuntu install https://review.openstack.org/387556 | 19:19 |
| *** mliima__ has quit IRC | 19:20 | |
| sdake | inc0 one option with nova if the current technique doesn't work is extend_start.sh...... | 19:20 |
| sdake | i thought nova would be problematic for the approach we are using :( | 19:20 |
| inc0 | sdake, so I'm testing it in and out | 19:20 |
| sdake | in and out? | 19:20 |
| inc0 | and for now it seems if we chown to nova | 19:20 |
| inc0 | it works | 19:21 |
| inc0 | cuz libvirt runs as root | 19:21 |
| sdake | on ubuntu it runs as root | 19:21 |
| sdake | ? | 19:21 |
| sdake | not sure that is the case on centos | 19:21 |
| *** HyperJohnGraham_ has joined #openstack-kolla | 19:21 | |
| sdake | but may be | 19:21 |
| inc0 | I'm pretty sure it is | 19:21 |
| inc0 | it's priviledged | 19:21 |
| openstackgerrit | Merged openstack/kolla: Fix log directory of Elasticsearch https://review.openstack.org/387616 | 19:21 |
| inc0 | anyway, console log was owned by libvirt user | 19:21 |
| inc0 | I chowned it and all worked wel | 19:21 |
| inc0 | l | 19:22 |
| inc0 | now I'm deploying without ceph to see if that's the case for local ephemerals as well | 19:22 |
| sdake | ya local ephemerals could be problematic | 19:22 |
| inc0 | or not | 19:22 |
| sdake | depends on what uid libvirt runs on in centos | 19:22 |
| inc0 | if libvirt runs as root | 19:22 |
| sdake | right - i'll let you knwo now | 19:22 |
| sdake | moment | 19:23 |
| *** DanyC has joined #openstack-kolla | 19:24 | |
| *** sdake_ has joined #openstack-kolla | 19:26 | |
| sdake_ | inc0 - looking into libvirt on centos - bbi10 | 19:27 |
| inc0 | yeah I'm redeploying without ceph now | 19:27 |
| *** sdake has quit IRC | 19:27 | |
| sdake_ | inc0 [sdake@minime-03 tools]$ docker exec nova_libvirt ps -ef | grep libvirt | 19:28 |
| sdake_ | root 9137 8730 0 15:25 ? 00:00:00 /usr/sbin/libvirtd --listen | 19:28 |
| sdake_ | inc0 not sure if libvirt drops permissions to something else or not | 19:29 |
| kfox1111 | hmm... cinder-manage-db is fairly consistently crashing now in the gate... will have to track that down soon. | 19:30 |
| kfox1111 | did any cinder related stuff merge in the last few hours? | 19:30 |
| *** g3ek has quit IRC | 19:30 | |
| sdake_ | kfox1111 yes | 19:30 |
| kfox1111 | awesome. :/ | 19:30 |
| sdake_ | kfox1111 let me look at the patch | 19:30 |
| *** haplo37 has quit IRC | 19:31 | |
| kfox1111 | I'm seeing; cinder-manage-db-5m3cn 0/1 CrashLoopBackOff 4 2m | 19:31 |
| *** HyperJohnGraham_ has quit IRC | 19:31 | |
| *** HyperJohnGraham_ has joined #openstack-kolla | 19:31 | |
| sdake_ | kfox1111 Change-Id: If778c981cfcf020fcf0be6bf7e38a3243e63b85c | 19:32 |
| sdake_ | kfox1111 is cinder using the json code? | 19:32 |
| inc0 | sdake_, so testing on ubuntu seems to show a-ok | 19:32 |
| sdake_ | the patch looks good to me | 19:32 |
| inc0 | when I chowned disk to nova (disk had different owner) | 19:33 |
| *** haplo37 has joined #openstack-kolla | 19:33 | |
| sdake_ | inc0 with your version of the patch? | 19:33 |
| inc0 | vm didnt notice it | 19:33 |
| inc0 | yes | 19:33 |
| sdake_ | how about starting a new vm? | 19:33 |
| sdake_ | not the upgrade case | 19:33 |
| inc0 | after reboot of vm perms came back to original | 19:33 |
| inc0 | all seems to be ok | 19:33 |
| *** g3ek has joined #openstack-kolla | 19:33 | |
| inc0 | so all the dirs were owned by nova originally | 19:33 |
| inc0 | only non-nova owned stuff were disk file and console-log file | 19:34 |
| inc0 | and both seems to survive chowning without any issue | 19:34 |
| sdake_ | try stopping the vm and starting it | 19:34 |
| sdake_ | if the permissions are wrong on the restart, want to make sure that works | 19:34 |
| kfox1111 | sdake_: yeah. | 19:34 |
| inc0 | disk gets it's permissions back, it's all good | 19:35 |
| sdake_ | is there no way to use /var/lib/nova/*disk* | 19:35 |
| sdake_ | pretty sure that code uses glob - so that should work | 19:35 |
| inc0 | I chowned disk to nova -> stop vm -> start vm -> disk gets its perms back | 19:35 |
| inc0 | all good | 19:36 |
| inc0 | so indeed my version of patch should work | 19:36 |
| sdake_ | i guess - but not entirely correct ;) | 19:36 |
| inc0 | according to every test I can think of:) | 19:36 |
| kfox1111 | sdake_: looks like a big ps. :/ | 19:36 |
| inc0 | sdake_, I don't think we can figure anything better | 19:36 |
| sdake_ | kfox1111 we have about 30 of these patches that have hti the repo in teh last 4 hours | 19:36 |
| kfox1111 | ouch. | 19:37 |
| sdake_ | inc0 ok - link | 19:37 |
| kfox1111 | thats hazardous... :/ | 19:37 |
| inc0 | https://review.openstack.org/#/c/386709/ | 19:37 |
| sdake_ | kfox1111 i hear what your syaing - however - upgrade is broken as is | 19:37 |
| sdake_ | thats more hazerdous | 19:37 |
| inc0 | kfox1111, we found project-wide issue | 19:37 |
| *** HyperJohnGraham_ has quit IRC | 19:37 | |
| kfox1111 | yeah. just bad to find it so late. | 19:38 |
| inc0 | yeah | 19:38 |
| kfox1111 | I think we should stop adding any new feature in o, and work on the gate. | 19:38 |
| kfox1111 | if we can't test this kind of stuff as it goes in, we're just going to keep running into this kind of thing. | 19:38 |
| inc0 | kfox1111, +1 to that, but I'd also like to finish up features we already have | 19:38 |
| *** HyperJohnGraham_ has joined #openstack-kolla | 19:38 | |
| inc0 | like make plugins great again | 19:39 |
| kfox1111 | and its going to bhe the ops that suffer. :/ | 19:39 |
| sdake_ | inc0 we can remove https://review.openstack.org/#/c/386709/1/ansible/roles/nova/templates/nova-ssh.json.j2 | 19:39 |
| kfox1111 | +1. | 19:39 |
| kfox1111 | sdake_: I don't see anything ovious in that ps that would cause the issue. | 19:39 |
| sdake_ | kfox1111 me either | 19:39 |
| kfox1111 | I've gota head back to the outage, but will try and look at it further asap. | 19:39 |
| kfox1111 | on top of it all, I'm fighting a cold too. :/ | 19:39 |
| kfox1111 | bbiab | 19:40 |
| sdake_ | kfox1111 we can release 3.0.1 to make kubernetes work if needed early | 19:40 |
| kfox1111 | k. | 19:40 |
| sdake_ | kfox1111 i've managed to stay cold free in the runup to summit :) | 19:40 |
| *** DanyC has left #openstack-kolla | 19:40 | |
| inc0 | sdake_, k8s uses master, let's keep it this way | 19:40 |
| kfox1111 | yes/no. | 19:40 |
| inc0 | I don't think we should make release based on k8s tbh | 19:40 |
| sdake_ | inc0 agree with keeping k8s on master of kolla | 19:41 |
| kfox1111 | if we released a fixed 3.0.1 before we do the ansible split in trunk, then I think we'd be good. | 19:41 |
| kfox1111 | the ansible split's going to make kolla-kubernetes unusable with trunk for a while. | 19:41 |
| inc0 | kfox1111, fixed with what? | 19:41 |
| inc0 | why? | 19:41 |
| sdake_ | inc0 cinder is not working for k8s | 19:41 |
| sdake_ | inc0 i dont know if cinder is cratered entirely in kolla or not | 19:41 |
| kfox1111 | the amount of rekergiggering of stuff between repos is likely to be very messy for a while. | 19:41 |
| sdake_ | my take is the repo split plus getting the gates operational will tke the whole cycle :) | 19:42 |
| sdake_ | but what do I know :) | 19:42 |
| *** portdirect has quit IRC | 19:42 | |
| inc0 | kfox1111, but you'll build images just the same | 19:42 |
| sdake_ | inc0 he isn't building image in the k8s gate | 19:42 |
| kfox1111 | inc0: I don't build images at all. ;) | 19:42 |
| sdake_ | inc0 although he should be ;-) | 19:42 |
| kfox1111 | its the ansible config bits that kolla-kubernetes pulls in that are at issue. | 19:42 |
| kfox1111 | they are too intimately tied together right now. | 19:43 |
| inc0 | oh please don't tell me you need 3.0.1 to be able to download them from dockerhum :( | 19:43 |
| inc0 | also tbh we can have some cronjob to create kolla-latest;) | 19:43 |
| sdake_ | kfox1111 agree that the issue you see is probably related to the ansible config templates | 19:43 |
| kfox1111 | inc0: we might need 3.0.1 to fix bugs with genconfig that were interoduced at the last moment. :/ | 19:43 |
| inc0 | daily builds from master uploaded to gates?;) | 19:43 |
| kfox1111 | sicne so much code is being rushed in at the last moment. :/ | 19:43 |
| sdake_ | inc0 good luck on that nut | 19:43 |
| sdake_ | kfox1111 these are critical bug fixes only kfox1111 | 19:44 |
| sdake_ | kfox1111 things that make kolla-ansible unusable at all | 19:44 |
| kfox1111 | sdake_: trunk is failing for sure: http://logs.openstack.org/41/387641/1/experimental/gate-kolla-kubernetes-deploy-ubuntu-binary-ceph-nv/d5ddc7d/console.html | 19:44 |
| sdake_ | kfox1111 and being rushed because our deadlines | 19:44 |
| kfox1111 | same issue each time. cinder-manage-db crashes. | 19:44 |
| kfox1111 | I'll look at it asap, but really do have to go now. bbiab. | 19:44 |
| sdake_ | kfox1111 c anyo uget a strace inthere | 19:44 |
| sdake_ | from there i can fix | 19:44 |
| sdake_ | kfox1111 by tomorrow morning or later today is fine on the strace if you can spare the time :) | 19:45 |
| kfox1111 | its hard to strace... I may have to fire up another minikube and launch it with debug. | 19:45 |
| kfox1111 | bbiab | 19:45 |
| sdake_ | need an strace tot see what the process is doing | 19:45 |
| sdake_ | then its eas yto fix permissions issues | 19:46 |
| sdake_ | whic hthis most likely is | 19:46 |
| *** neilus has quit IRC | 19:46 | |
| sdake_ | kfox1111 to strace - just change the extend_start and rebuild | 19:48 |
| *** HyperJohnGraham_ has quit IRC | 19:48 | |
| *** neilus has joined #openstack-kolla | 19:50 | |
| *** neilus has quit IRC | 19:57 | |
| *** srwilkers has joined #openstack-kolla | 19:58 | |
| *** brad[] has quit IRC | 20:00 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla: Fix nova upgrade permission issue https://review.openstack.org/386709 | 20:00 |
| inc0 | sdake_, ^ | 20:01 |
| sdake_ | so wife bought a laptop from lenovo to install linux on | 20:05 |
| sdake_ | and guess what- it doesn't work with linux! | 20:05 |
| jascott1 | secureboot or what? | 20:06 |
| sdake_ | something to do with raid and powersavings | 20:06 |
| sdake_ | and intel drivers | 20:06 |
| inc0 | which linux? | 20:07 |
| sdake_ | any linux | 20:07 |
| sdake_ | lenovo added raid to their laptops for no good reason other then to make power savings work with the intel driver for the storage device | 20:08 |
| sdake_ | there is no wayt o shut it off | 20:08 |
| sdake_ | and no linux board bringup has been done on the raid stuff | 20:08 |
| *** ntpttr_ is now known as ntpttr | 20:10 | |
| inc0 | yeah but linux kernel has some raid stuff | 20:12 |
| inc0 | I mean, servers uses raids... | 20:13 |
| inc0 | nvm, shame, I liked lenovo | 20:13 |
| inc0 | now I'm not going to buy lenovo | 20:13 |
| sdake_ | this needs an ack | 20:15 |
| sdake_ | https://review.openstack.org/#/c/386478/1 | 20:15 |
| sdake_ | inc0 i guess lenovo is working on a bios fix | 20:15 |
| sdake_ | its their latest released laptop - like 3 weeks old | 20:16 |
| sdake_ | but you would think the ycould ddo kernel.org bringup before releasing a product | 20:16 |
| *** matrohon has joined #openstack-kolla | 20:18 | |
| *** eaguilar_ has joined #openstack-kolla | 20:19 | |
| *** ppalacios has quit IRC | 20:19 | |
| *** eaguilar has quit IRC | 20:20 | |
| sdake_ | pbourke_ if your about | 20:21 |
| sdake_ | it looks like possibly the ol gate is busted | 20:21 |
| *** brad[] has joined #openstack-kolla | 20:22 | |
| sdake_ | pbourke_ ^ | 20:22 |
| sdake_ | https://review.openstack.org/#/c/386478/1 | 20:22 |
| sdake_ | pbourke_ looks like ol may have some type of outage: http://logs.openstack.org/97/387597/1/check/gate-kolla-dsvm-build-oraclelinux-binary-centos-7-nv/8ee2b06/console.html#_2016-10-17_18_56_19_224459 | 20:26 |
| sdake_ | pbourke_ but i'm not entirel ysure | 20:27 |
| sdake_ | this is ready for merge: https://review.openstack.org/#/c/386922/3 | 20:35 |
| *** dwalsh has joined #openstack-kolla | 20:38 | |
| *** schwicht has quit IRC | 20:45 | |
| *** schwicht has joined #openstack-kolla | 20:46 | |
| sdake_ | rhallisey can you ack this pls: https://review.openstack.org/#/c/386922/3 | 20:46 |
| sdake_ | and this one: https://review.openstack.org/#/c/386478/1 | 20:47 |
| *** matrohon has quit IRC | 20:50 | |
| openstackgerrit | Merged openstack/kolla: Fix murano upgrade permission issue https://review.openstack.org/386922 | 20:50 |
| *** schwicht has quit IRC | 20:51 | |
| *** schwicht has joined #openstack-kolla | 20:51 | |
| openstackgerrit | Merged openstack/kolla: move sys.exit out of try finally block in set_config.py file https://review.openstack.org/386478 | 20:53 |
| *** schwicht has quit IRC | 20:58 | |
| *** haplo37_ has quit IRC | 21:01 | |
| *** haplo37_ has joined #openstack-kolla | 21:03 | |
| *** jtriley has quit IRC | 21:04 | |
| openstackgerrit | Waldemar Znoinski proposed openstack/kolla: copy iscsid logrotate config https://review.openstack.org/387683 | 21:06 |
| openstackgerrit | Waldemar Znoinski proposed openstack/kolla: copy ldlinux.c32 to /tftpboot in ironic_pxe https://review.openstack.org/387684 | 21:06 |
| *** janem has joined #openstack-kolla | 21:07 | |
| *** portdirect has joined #openstack-kolla | 21:07 | |
| openstackgerrit | Waldemar Znoinski proposed openstack/kolla: set basic kolla structures for iscsid https://review.openstack.org/387685 | 21:11 |
| *** janem has quit IRC | 21:15 | |
| sbezverk | kfox1111: ping | 21:17 |
| *** eaguilar_ has quit IRC | 21:17 | |
| *** neilus has joined #openstack-kolla | 21:18 | |
| Pavo | evening | 21:23 |
| *** eaguilar has joined #openstack-kolla | 21:23 | |
| *** athomas has quit IRC | 21:23 | |
| *** msimonin has quit IRC | 21:24 | |
| inc0 | hey Pavo | 21:25 |
| Pavo | hows it been | 21:25 |
| inc0 | busy:) you're going to Barcelona? | 21:26 |
| awiddersheim | inc0: working on some NFS changes and need some help on how to best approach some changes | 21:26 |
| awiddersheim | not sure what to do in some of these circumstances | 21:26 |
| inc0 | awiddersheim, shoot | 21:26 |
| awiddersheim | alright so my colleague and I with the help of rhallisey have been working on getting NFS working | 21:26 |
| inc0 | nfs as cinder driver? | 21:27 |
| awiddersheim | yes | 21:27 |
| inc0 | k | 21:27 |
| awiddersheim | using this PS as the basis | 21:27 |
| awiddersheim | https://review.openstack.org/#/c/279913 | 21:27 |
| awiddersheim | however the PS has a few problems with it that we have come across and we finally worked through most of them | 21:27 |
| awiddersheim | the big sticking point right now | 21:27 |
| *** eaguilar has quit IRC | 21:27 | |
| inc0 | rhallisey, mind removing -2 | 21:28 |
| inc0 | ? | 21:28 |
| awiddersheim | is when attaching a volume to an instance | 21:28 |
| awiddersheim | the nova-compute container will mount the NFS share | 21:28 |
| awiddersheim | in /var/lib/nova/mnt or whatever | 21:28 |
| kfox1111 | sbezverk: hi. whats up? | 21:28 |
| awiddersheim | and then nova-libvirt tries to read it but can't | 21:28 |
| rhallisey | inc0, sure | 21:28 |
| inc0 | hmm so /var/lib/nova is shared between both | 21:29 |
| kfox1111 | sdake_: I'm not building the containers, so that makes it harder. I think I can just slip in the strace into the k8s template though. | 21:29 |
| rhallisey | inc0, ya I think we need a shared vol there | 21:29 |
| inc0 | we have it | 21:29 |
| inc0 | its named volume | 21:29 |
| kfox1111 | its a little painful though to actually get the logs out of it though, as it crashes. | 21:29 |
| inc0 | in reality this is directory /var/lib/docker/volumes/nova_compute/_data | 21:29 |
| kfox1111 | I guess I can put in a long sleep, and then have it wait for the job to timeout. | 21:30 |
| rhallisey | inc0, when you create mount it's not visible to the host or nova_compute | 21:30 |
| inc0 | hmm | 21:31 |
| *** eaguilar has joined #openstack-kolla | 21:31 | |
| inc0 | probably because no nfs in containers? | 21:31 |
| inc0 | I dunno, we need to disect how nfs is mounted | 21:32 |
| awiddersheim | inc0: sorry | 21:33 |
| sbezverk | kfox1111: trying to decypher jq syntax, could you check what I do wrong? | 21:33 |
| awiddersheim | put all my chat in wrong windo | 21:33 |
| awiddersheim | wait one second | 21:33 |
| awiddersheim | so yeah | 21:33 |
| awiddersheim | nova-compute does the NFS mount | 21:33 |
| awiddersheim | and nova-libvirt tries to read it | 21:33 |
| awiddersheim | but can't | 21:33 |
| kfox1111 | sbezverk: sure. what are you trying to do? | 21:34 |
| awiddersheim | and the solution seems to make that a :shared directory | 21:34 |
| sbezverk | kfox1111: take a lot at this when you have a chance: | 21:34 |
| sbezverk | kubectl get pods -o json | jq -r '.items[] | select(.spec.nodeName=="falkor-control-03.k8s.cisco.com" | select(.status.conditions[] | select(.type=="Ready" and .status!="True"))) | .metadata.name' | 21:34 |
| awiddersheim | so mounts in nova-compute are visible in nova-libvirst | 21:34 |
| awiddersheim | that make sense? | 21:34 |
| awiddersheim | so in other words | 21:34 |
| inc0 | awiddersheim, you might try to add :shared to ansible | 21:34 |
| inc0 | it shouldn't be problem | 21:35 |
| awiddersheim | right | 21:35 |
| awiddersheim | so we did that | 21:35 |
| kfox1111 | sbezverk: can you pastebin me a 'kubectl get pods -o json' too? | 21:35 |
| awiddersheim | and it worked | 21:35 |
| awiddersheim | but the problem is you can't do :shared | 21:35 |
| awiddersheim | with named volumes | 21:35 |
| awiddersheim | docker doesn't allow it | 21:35 |
| awiddersheim | so we changed | 21:35 |
| sbezverk | kfox1111: sure one sec | 21:35 |
| awiddersheim | https://github.com/openstack/kolla/blob/master/ansible/roles/nova/tasks/start_compute.yml#L19 | 21:35 |
| rhallisey | brb | 21:35 |
| awiddersheim | to "/var/lib/nova:/var/lib/nova/:shared" | 21:35 |
| awiddersheim | and it worked | 21:35 |
| inc0 | awiddersheim, then make this change to ansible | 21:36 |
| inc0 | I'm ok with it | 21:36 |
| awiddersheim | nova-compute's mounted the NFS share and nova-libvirt could see it and use the volume | 21:36 |
| awiddersheim | ok | 21:36 |
| awiddersheim | one more thing | 21:36 |
| awiddersheim | permissions :( | 21:36 |
| awiddersheim | when we made that change we started to run into permissions problems | 21:36 |
| awiddersheim | and this is the oen I'm not really sure about | 21:36 |
| sbezverk | kfox1111: http://paste.openstack.org/show/586042/ | 21:36 |
| awiddersheim | nova-compute tries to create /var/lib/nova/instances in it's extended start | 21:36 |
| awiddersheim | but can't because it's root:root | 21:36 |
| awiddersheim | two options i can think of is have nova-libvirt's extended_start fix the permissions or create local nova user in kolla's playbooks? | 21:36 |
| awiddersheim | I dont' know... they all seem ugly to me | 21:37 |
| inc0 | awiddersheim, can we mount nfs volume in specified location? | 21:37 |
| awiddersheim | not sure | 21:37 |
| inc0 | https://review.openstack.org/#/c/386709/ look at this | 21:37 |
| awiddersheim | so far I've only let nova do it's thing and mount it where it wants it | 21:37 |
| inc0 | we will chown stuff in /var/lib/nova/instances | 21:38 |
| inc0 | maybe it will fix your permission issue | 21:38 |
| inc0 | not sure how it will affect nfs tho | 21:38 |
| awiddersheim | it'd only work if you did it for /var/lib/nova | 21:39 |
| awiddersheim | but that may work | 21:39 |
| awiddersheim | I'm not seeing where that is in this PS though | 21:39 |
| awiddersheim | yes, I'd think that would work but only if /var/lib/nova was the directory | 21:40 |
| kfox1111 | sbezverk: seemed to have truncated it. | 21:40 |
| kfox1111 | email maybe | 21:40 |
| kfox1111 | ? | 21:40 |
| awiddersheim | oh it is | 21:40 |
| awiddersheim | https://review.openstack.org/#/c/386709/2/ansible/roles/nova/templates/nova-compute.json.j2 | 21:40 |
| awiddersheim | here | 21:40 |
| awiddersheim | that _should_ work nicely | 21:41 |
| awiddersheim | I can try this | 21:41 |
| *** srwilkers has quit IRC | 21:41 | |
| awiddersheim | this runs before the command? | 21:41 |
| awiddersheim | like where does this stuff happen from an order standpoint when the container starts? | 21:42 |
| *** jheroux has quit IRC | 21:42 | |
| awiddersheim | before extended_start or after? | 21:42 |
| *** dwalsh has quit IRC | 21:42 | |
| sbezverk | kfox1111: I partially figured it out.. | 21:43 |
| sbezverk | I am thinking that there is no need to check status of the pod on notready node.. | 21:44 |
| *** shardy_afk has quit IRC | 21:44 | |
| sbezverk | I think all pods which were bound to failed node should be killed unconditionally | 21:44 |
| kfox1111 | yeah, if the node is not ready, and your going to shoot it, you can probably skip that part. | 21:44 |
| kfox1111 | just match all pods on the node. | 21:45 |
| sbezverk | kfox1111: perfect | 21:45 |
| awiddersheim | inc0: looks like before so I think that will help | 21:49 |
| awiddersheim | great! thanks! | 21:49 |
| inc0 | np | 21:50 |
| inc0 | awiddersheim, make sure that it wont break permissions inside share | 21:54 |
| inc0 | I'm not sure how nfs would work if we chown -r parent folder | 21:54 |
| *** neilus has quit IRC | 22:00 | |
| *** inc0 has quit IRC | 22:01 | |
| *** HyperJohnGraham_ has joined #openstack-kolla | 22:04 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Fencing pod https://review.openstack.org/383922 | 22:05 |
| sbezverk | kfox1111: latest fencing PS with a couple of changes from your wishlist ;-) | 22:06 |
| *** HyperJohnGraham_ has quit IRC | 22:10 | |
| kfox1111 | sbezverk: looking better. | 22:10 |
| *** spsingh has joined #openstack-kolla | 22:11 | |
| kfox1111 | hmm.... maybe abug... | 22:11 |
| *** HyperJohnGraham_ has joined #openstack-kolla | 22:11 | |
| sbezverk | kfox1111: ? | 22:11 |
| kfox1111 | sec. commenting on it inline. | 22:12 |
| sbezverk | kfox1111: k | 22:12 |
| *** spsingh has quit IRC | 22:15 | |
| awiddersheim | inc0: the NFS mount only happens after nova-compute is up and running | 22:17 |
| *** HyperJohnGraham_ has quit IRC | 22:17 | |
| *** HyperJohnGraham_ has joined #openstack-kolla | 22:18 | |
| awiddersheim | so that should just be chowning non-NFS files | 22:18 |
| awiddersheim | at that point I believe | 22:18 |
| *** HyperJohnGraham_ has quit IRC | 22:23 | |
| *** HyperJohnGraham_ has joined #openstack-kolla | 22:23 | |
| jascott1 | inc0 sierra now officially supported. haha | 22:25 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Fencing pod https://review.openstack.org/383922 | 22:26 |
| *** sdake_ has quit IRC | 22:27 | |
| *** sdake has joined #openstack-kolla | 22:30 | |
| sdake | back | 22:31 |
| sdake | sorry had an errand to run | 22:31 |
| kfox1111 | sbezverk: better. :) | 22:31 |
| kfox1111 | what happens if you hit the mismatch between failed node name case or volume not locked by any node case? | 22:31 |
| kfox1111 | it still will delete pods in that case. | 22:31 |
| kfox1111 | that may be ok. haven't had much time to think about it. | 22:32 |
| kfox1111 | hmm... let me propose a thought experiment... | 22:33 |
| kfox1111 | say the cloud has a noisy vm. the user's hamering the vm hard enough the kubelet is slow enough to respond the node goes notready. | 22:33 |
| kfox1111 | the operator specifically seperated computes from controllers. | 22:33 |
| kfox1111 | with it outside of the lock held check, the code will kill the node/vm. | 22:34 |
| *** sbezverk has quit IRC | 22:34 | |
| kfox1111 | so, its probably better to kill the node only in the case where an rbd lock is held. | 22:34 |
| *** schwicht has joined #openstack-kolla | 22:39 | |
| *** sdake_ has joined #openstack-kolla | 22:40 | |
| *** lamt has quit IRC | 22:41 | |
| *** neilus has joined #openstack-kolla | 22:43 | |
| *** schwicht has quit IRC | 22:44 | |
| *** sdake has quit IRC | 22:44 | |
| *** janem has joined #openstack-kolla | 22:47 | |
| *** HyperJohnGraham_ has quit IRC | 22:48 | |
| kfox1111 | sdake_: do you know of any other changes to cinder? maybe the config files? | 22:48 |
| *** neilus has quit IRC | 22:49 | |
| sdake_ | kfox1111 git log is your friend | 22:49 |
| sdake_ | but not sure | 22:49 |
| kfox1111 | k. I may just have to manually bisect it. :/ | 22:51 |
| *** MarMat_ has joined #openstack-kolla | 22:56 | |
| *** MarMat has quit IRC | 22:56 | |
| *** daneyon has joined #openstack-kolla | 22:56 | |
| sdake_ | rather then bisecting, might be faster to use git log :) | 22:56 |
| sdake_ | but a git bisect would be hlepful to identify the offender | 22:56 |
| kfox1111 | arg. all the dates are in local time... | 22:57 |
| sdake_ | kfox1111 yes - that was a desire of jeffrey's | 22:57 |
| sdake_ | kfox1111 what is the issue with local time? | 22:57 |
| kfox1111 | well, I knew it worked at 9:00 am my time. | 22:57 |
| kfox1111 | so its one of the commits after. | 22:57 |
| sdake_ | oh i thought you meant inside the container | 22:58 |
| sdake_ | kfox1111 we merged about 30 patches at 9:00 am yoru time | 22:59 |
| sdake_ | or more | 22:59 |
| *** HyperJohnGraham_ has joined #openstack-kolla | 22:59 | |
| *** michauds has quit IRC | 23:00 | |
| kfox1111 | yeah. hence the thinking I might bisect it. don't want to try them one at a time. :) | 23:01 |
| kfox1111 | going to try thisn one first though: e29a09df239e36991f5210da3934ee291a47a4ec | 23:02 |
| kfox1111 | as there's that cinder patch right after. | 23:02 |
| *** eaguilar has quit IRC | 23:02 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Testing trunk... https://review.openstack.org/387641 | 23:03 |
| Jeffrey4l | sdake_, pong | 23:08 |
| *** HyperJohnGraham_ has quit IRC | 23:12 | |
| *** srwilkers has joined #openstack-kolla | 23:12 | |
| *** srwilkers has quit IRC | 23:12 | |
| kfox1111 | is kolla going to be at the ptg? | 23:20 |
| Jeffrey4l | kfox1111, yes | 23:21 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: move sys.exit out of try finally block in set_config.py file https://review.openstack.org/387715 | 23:22 |
| kfox1111 | ok. cool. | 23:22 |
| kfox1111 | sdake_: e29a09df239e36991f5210da3934ee291a47a4ec did work... going to try the one right after. if that works, then I'll bisect. | 23:23 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Testing trunk... https://review.openstack.org/387641 | 23:24 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix common role upgrade permission issue https://review.openstack.org/387717 | 23:26 |
| *** haplo37_ has quit IRC | 23:29 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Modular gate https://review.openstack.org/386966 | 23:30 |
| *** papacz has quit IRC | 23:30 | |
| *** haplo37_ has joined #openstack-kolla | 23:30 | |
| sdake_ | rhallisey need a merge on https://review.openstack.org/#/c/387715/1 | 23:30 |
| sdake_ | hey Jeffrey4l | 23:30 |
| sdake_ | Jeffrey4l -- i'm here all night to sort out rc3 | 23:30 |
| Jeffrey4l | sup sdake_ | 23:30 |
| sdake_ | Jeffrey4l if there is anything you need, let me know | 23:30 |
| sdake_ | Jeffrey4l we think we have a regression in the merging of the upgrade madness | 23:30 |
| Jeffrey4l | cool. | 23:31 |
| Jeffrey4l | what's it? | 23:31 |
| sdake_ | Jeffrey4l kfox1111 is bisecting - so we will knwo soon | 23:31 |
| *** papacz has joined #openstack-kolla | 23:31 | |
| sdake_ | cinder no longer works apparently | 23:31 |
| sdake_ | i haven' ttried it on my gear | 23:31 |
| Jeffrey4l | kfox1111, any detail logs? | 23:31 |
| sdake_ | been busy getting patches merged that could be merged | 23:31 |
| kfox1111 | good greef that's a lot of patches... | 23:31 |
| Jeffrey4l | cinder + lvm or cinder + ceph? | 23:31 |
| sdake_ | Jeffrey4l not sure | 23:31 |
| Jeffrey4l | i can test this locally and try to find the root cause. | 23:31 |
| *** lamt has joined #openstack-kolla | 23:31 | |
| Jeffrey4l | kfox1111, ^^ | 23:31 |
| kfox1111 | Jeffrey4l: cinder & ceph. | 23:31 |
| kfox1111 | don't have a gate test for cinder + lvm. | 23:32 |
| Jeffrey4l | yes. | 23:32 |
| kfox1111 | thoughi ts failing on a manage-db though, so probably effects both. | 23:32 |
| sdake_ | Jeffrey4l cinder + ceph is running with kolla-kubernetes - so if we can be sure its not in ansible proper | 23:32 |
| Jeffrey4l | kfox1111, did u catch any logs? | 23:32 |
| sdake_ | thats probably good enuf | 23:32 |
| sdake_ | Jeffrey4l no logs - no strace of the binary | 23:32 |
| sdake_ | Jeffrey4l if you reproduce it, an strace of the db manage would be emmensly helpful | 23:33 |
| Jeffrey4l | OK. i will try to deploy cinder w/ ceph | 23:33 |
| sdake_ | Jeffrey4l looking at all the patches that merged in backports | 23:33 |
| sdake_ | Jeffrey4l i noticed /var/lib/kolla was used laot | 23:33 |
| sdake_ | should it not be /var/lib/kolla/keystone for example? | 23:33 |
| kfox1111 | Jeffrey4l: the container just crashies. no easy way to get logs at the moment. :/ | 23:33 |
| sdake_ | kfox1111 a bisect would help :) | 23:33 |
| kfox1111 | yup. its in the gate right now. | 23:34 |
| sdake_ | your bisecting in the gate? | 23:34 |
| * sdake_ whooshes | 23:34 | |
| kfox1111 | see zuul 387641,3 | 23:34 |
| *** sbezverk has joined #openstack-kolla | 23:34 | |
| kfox1111 | the ubuntu-binary-ceph one's the one to watch. | 23:34 |
| Jeffrey4l | sdake_, re /var/lib/kolla hmm. hold on, i will check this. it should be correct. | 23:34 |
| Jeffrey4l | kfox1111, we haven't gate for ceph and ceph+cinder | 23:35 |
| sdake_ | kfox1111 dont have time to watch individual gate jobs, tr yto look at the outstanding work holistically :) | 23:35 |
| sdake_ | Jeffrey4l - kfox1111 has one in kolla-kubernetes... | 23:35 |
| Jeffrey4l | ok. | 23:35 |
| kfox1111 | Jeffrey4l: not for ansible. :) | 23:35 |
| Jeffrey4l | roger | 23:35 |
| sdake_ | what we do have is one more dude trained on the 100+ patchset frustration of adding a gate job ;) | 23:35 |
| kfox1111 | sdake_: yeah. and was crazy enough to do it again right after wards. ;) | 23:36 |
| kfox1111 | and working on a multinode variant. :) | 23:36 |
| sdake_ | kfox1111 actually adding new jobs is pretty esy | 23:37 |
| sdake_ | although multinode wont be easy at all | 23:37 |
| sdake_ | but dding new gate jobs to new repos is hard | 23:37 |
| kfox1111 | yeah. | 23:37 |
| *** awiddersheim has quit IRC | 23:37 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix stale namespace removal issue https://review.openstack.org/387721 | 23:37 |
| kfox1111 | heh. most of the gate job issue was the lack of workflow. | 23:37 |
| kfox1111 | I had to write the whole thing from scratch. | 23:37 |
| sdake_ | need an ack on https://review.openstack.org/#/c/387721/1 | 23:38 |
| sdake_ | kfox1111 yup - hard isn't it :) | 23:38 |
| kfox1111 | a 726 line shell script isn't the easist thing to write. ;) | 23:38 |
| sdake_ | yup | 23:39 |
| Jeffrey4l | sdake_, re /var/lib/kolla it is used only for two case: 1. hake_socket, which is a syslog socket actually, full path is: /var/lib/kolla/heka/log 2, haproxy.sock | 23:39 |
| *** awiddersheim has joined #openstack-kolla | 23:39 | |
| sdake_ | to be brutually frank, i wasn't sure yo ucould pull it off kfox1111 :) | 23:39 |
| sdake_ | its super hard | 23:39 |
| sdake_ | so - NICE JOB :) | 23:39 |
| kfox1111 | sdake_: ok, so the next one failed. | 23:41 |
| kfox1111 | so it is that commit. | 23:41 |
| kfox1111 | thx. :) | 23:41 |
| kfox1111 | 032234bad63fa4b9f8c04d31d1be26d97086a0fa | 23:41 |
| kfox1111 | hmm.... | 23:42 |
| kfox1111 | did cinder use to run as root? | 23:42 |
| *** sbezverk has quit IRC | 23:43 | |
| sdake_ | kfox1111 got a review to go with that commit | 23:43 |
| kfox1111 | and or was the 'permissions' section added in the newton container agent? | 23:43 |
| Jeffrey4l | kfox1111, which container is not started? | 23:43 |
| kfox1111 | 2016-10-17 23:37:26.632644 | cinder-manage-db-liacb 0/1 CrashLoopBackOff 4 2m | 23:43 |
| kfox1111 | sdake_: not off hand. its the same one you pointed at earlier. | 23:43 |
| sdake_ | Jeffrey4l i think that would be bootstrap in kolla-ansible | 23:43 |
| sdake_ | kfox1111 thats a bummer | 23:44 |
| Jeffrey4l | yes. | 23:44 |
| sdake_ | Jeffrey4l - if you can reproduce, its the cinder upgrade patch | 23:44 |
| Jeffrey4l | yep. i am tying. | 23:44 |
| kfox1111 | this may be a mitaka/newton incompatabiliy if the permission feature was added in newton. | 23:44 |
| sdake_ | kfox1111 i makes sense cinder wouldl write volumes as the non-cinder user | 23:45 |
| sdake_ | since nova would be communicating with it possibly | 23:45 |
| sdake_ | what that patch does is make everything owned by cinder | 23:45 |
| sdake_ | moment switching vpns | 23:45 |
| Jeffrey4l | kfox1111, it is compatible. 1. in mitaka, 'permissions' is never used. 2. in mitaka, there is no permission section, too. | 23:45 |
| *** jtriley has joined #openstack-kolla | 23:45 | |
| kfox1111 | but if the permissions section is there, will a mitaka container freak out? | 23:45 |
| Jeffrey4l | kfox1111, no. | 23:46 |
| *** sdake has joined #openstack-kolla | 23:46 | |
| kfox1111 | k. | 23:46 |
| sdake | here is the review: https://review.openstack.org/#/c/386930/ | 23:46 |
| Jeffrey4l | now, there are lots of container do not have permissions section. | 23:46 |
| kfox1111 | yup. thats it. | 23:46 |
| sdake | patch still looks good | 23:47 |
| Jeffrey4l | yes. | 23:47 |
| sdake | obviously it may not be | 23:47 |
| sdake | but i dont see any glaring problems iwth it | 23:47 |
| Jeffrey4l | kfox1111, did u got anything by using `docker logs` | 23:48 |
| sdake | There were 16869 failed login attempts since the last successful login. | 23:48 |
| kfox1111 | Jeffrey4l: Its a gate job, I don't have easy access to run tthings in it. :/ I've added a metric ton of logging so far and haven't gotten a good way to get logs from crashed containers yet. | 23:48 |
| kfox1111 | sdake_: hehe. internet facing box? :) | 23:48 |
| Jeffrey4l | sdake, lol | 23:48 |
| sdake | yup | 23:48 |
| Jeffrey4l | sdake, changed the default 22 to anything else ;) | 23:49 |
| sdake | Jeffrey4l ya i know i should do that | 23:49 |
| kfox1111 | I'm going to try and revert just the one patch and see if it works. | 23:49 |
| sdake | kfox1111 great idea | 23:49 |
| Jeffrey4l | kfox1111, add `docker logs` before ci exit | 23:49 |
| kfox1111 | cause if its a greater issue with permission sections, then it will still fail. | 23:49 |
| kfox1111 | Jeffrey4l: not an easy thing to just do. the container names fluctuate. | 23:49 |
| Jeffrey4l | kfox1111, docker logs all. | 23:50 |
| *** sdake_ has quit IRC | 23:50 | |
| Jeffrey4l | docker logs $(docker ps -qa) | 23:50 |
| kfox1111 | k | 23:50 |
| Jeffrey4l | wait. | 23:50 |
| Jeffrey4l | this is wrong. | 23:50 |
| Jeffrey4l | need a for | 23:50 |
| kfox1111 | yeah. its hard to get right. haven't had time to do so yet. :/ | 23:50 |
| sdake | ()[cinder@cb85e5445546 lib]$ ls -ld cinder | 23:51 |
| sdake | drwxr-xr-x. 1 cinder cinder 0 Oct 13 00:53 cinder | 23:51 |
| Jeffrey4l | sdake, this one docker ps -a | awk '/cinder/{print $1}' | xargs -n1 docker logs | 23:52 |
| Jeffrey4l | kfox1111, ^^ | 23:52 |
| Jeffrey4l | ` docker ps -a | awk '/cinder/{print $1}' | xargs -n1 docker logs ` | 23:52 |
| sdake | Jeffrey4l i dont have cinder deployed nor ceph on it | 23:52 |
| kfox1111 | k | 23:52 |
| Jeffrey4l | sorry, pinged wrong person. ;) | 23:52 |
| sdake | what we need is a strace in the container on the db manage ;) | 23:53 |
| sdake | i can guarantee you i can fix it with that info | 23:53 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Testing trunk... https://review.openstack.org/387641 | 23:53 |
| kfox1111 | sdake: k. I'll try and get a strace next. | 23:55 |
| kfox1111 | we'll see if the patch revert works. | 23:55 |
| sdake | kfox1111 if it does, we hvae a serious problem on our hands | 23:55 |
| sdake | we have about 30 patches that look a whole lot like that one that hit the repo | 23:56 |
| kfox1111 | sdake: I'm not sure which is worse. :) this one patch having an issue, or potentially 30'sh of them. | 23:56 |
| kfox1111 | yeah. | 23:56 |
| Jeffrey4l | sdake, why rc3 will be tagged? utc time? | 23:56 |
| sdake | 23:45 oct 18 | 23:56 |
| sdake | may slip a little depending on how screwed we are | 23:57 |
| Jeffrey4l | fyi: deploying cinder + ceph on master. | 23:57 |
| sdake | but can't slip past 19th midday us time | 23:57 |
| *** Pavo has quit IRC | 23:57 | |
| Jeffrey4l | roger. | 23:57 |
| sdake | which is about oct 19th 1800 utc | 23:58 |
| *** mtaylor22 has joined #openstack-kolla | 23:58 | |
| sdake | at this point since we have roughly 24 hours left | 23:59 |
| sdake | our options are to proceed with getting upgrade working | 23:59 |
| sdake | i think we are close here | 23:59 |
| sdake | or mark 3.0.0 upgrade as doa and tag that | 23:59 |
| sdake | and release a 3.0.1 asap | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!