| *** shardy has quit IRC | 00:01 | |
| kfox1111 | arg.. no, its just not working at all... | 00:03 |
|---|---|---|
| *** schwicht has joined #openstack-kolla | 00:04 | |
| *** pramodrj07 has quit IRC | 00:07 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Local registry in gate for trunk builds https://review.openstack.org/418550 | 00:10 |
| *** mattmceuen has quit IRC | 00:13 | |
| *** lrensing has quit IRC | 00:17 | |
| *** lamt has quit IRC | 00:17 | |
| *** schwicht has quit IRC | 00:20 | |
| sbezverk | kfox1111: if you want to check the vswitchd fix: https://review.openstack.org/424902 | 00:20 |
| *** eaguilar has joined #openstack-kolla | 00:26 | |
| jascott1 | portdirect ping | 00:26 |
| *** TxGirlGeek has quit IRC | 00:26 | |
| *** sdake has joined #openstack-kolla | 00:26 | |
| *** inc0 has joined #openstack-kolla | 00:31 | |
| inc0 | hey again | 00:31 |
| inc0 | so, gates | 00:31 |
| inc0 | red | 00:31 |
| inc0 | I'm worried, any centos people aroudn? | 00:31 |
| *** zhubingbing has joined #openstack-kolla | 00:35 | |
| duonghq | hi inc0 | 00:35 |
| duonghq | morning guys | 00:35 |
| jascott1 | good morning | 00:36 |
| duonghq | hi jascott1 | 00:37 |
| Pavo | sup inc0 I'm a centos user | 00:37 |
| inc0 | hey duonghq | 00:38 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Local registry in gate for trunk builds https://review.openstack.org/418550 | 00:38 |
| inc0 | Pavo, our gates are broken | 00:38 |
| inc0 | centos | 00:38 |
| inc0 | https://review.openstack.org/#/c/424784/ look at gate status | 00:38 |
| Pavo | ah I have no clue on those, never looked at them but will take a look | 00:39 |
| inc0 | yeah gates requires special kind of touch | 00:39 |
| inc0 | but let me know if you run into any issues, they seem consistently broken | 00:39 |
| inc0 | Jeffrey4l_, https://review.openstack.org/#/c/424784/ <- when you wake up please take a look | 00:39 |
| jascott1 | new to this party but this is the error right? Unable to open /dev/sr0 read-write (Read-only file system). | 00:41 |
| jascott1 | Error: /dev/sr0: unrecognised disk label | 00:41 |
| jascott1 | weird chars in the file? before that error there is this | 00:42 |
| jascott1 | cp: cannot stat ‘/var/lib/docker/volumes/kolla_logs/_data/*’: No such file or directory | 00:42 |
| inc0 | chars were there for a while now | 00:43 |
| inc0 | bit this /dev/sr0 looks suspicious | 00:43 |
| *** adrian_otto has joined #openstack-kolla | 00:43 | |
| *** tonanhngo has quit IRC | 00:44 | |
| *** tonanhngo has joined #openstack-kolla | 00:44 | |
| *** zhurong has joined #openstack-kolla | 00:45 | |
| kfox1111 | inc0: you there? | 00:53 |
| inc0 | yup | 00:53 |
| inc0 | wassup? | 00:53 |
| kfox1111 | to your mailing list question, | 00:53 |
| inc0 | which one?:) | 00:53 |
| kfox1111 | no, I think he wants to run arm based debian containers, on top of an arm based distro. | 00:53 |
| inc0 | ah, debian | 00:53 |
| kfox1111 | no other distro's really support arm very well. | 00:54 |
| inc0 | but we would need debian support, gates and such | 00:54 |
| kfox1111 | so no arm-centos for kolla. | 00:54 |
| kfox1111 | right. | 00:54 |
| kfox1111 | so if he's voluneering, | 00:54 |
| kfox1111 | which it sounds like he's doing? he needs to do that stuff. | 00:54 |
| *** cuongnv has joined #openstack-kolla | 00:54 | |
| inc0 | well, my point is - you want us to keep it, you take care of it | 00:55 |
| kfox1111 | It sounded like hew was saying that too. | 00:55 |
| inc0 | and first step would be write gate jobs | 00:55 |
| inc0 | yeah, we'll talk about it tomorrow, we can just ask:) | 00:55 |
| kfox1111 | oh. you responded to a second thread I think. | 00:55 |
| kfox1111 | there was an arm one too at somepoint.... | 00:55 |
| inc0 | yeah, one on openstack dev and another on operators list | 00:56 |
| inc0 | I saw it | 00:56 |
| kfox1111 | yeah. sorry. got a bit confused. | 00:56 |
| kfox1111 | the arm one makes a lot of sence. | 00:56 |
| inc0 | no worries | 00:56 |
| inc0 | well, yeah, it is use case | 00:56 |
| inc0 | as long as we have someone to take care of it | 00:56 |
| *** adrian_otto has quit IRC | 00:57 | |
| inc0 | we can keep it | 00:57 |
| inc0 | but really... | 00:57 |
| kfox1111 | the arm guy sounded like he might try post ptg. | 00:57 |
| inc0 | I'd put requirement - you want debian, start with gates | 00:57 |
| kfox1111 | I'd say, lets not kick it out and see if he follows through. if not, we drop it? | 00:58 |
| kfox1111 | +1 | 00:58 |
| inc0 | if he volunteers, we shouldn't kick it out just yet, we can always do it | 00:58 |
| inc0 | but I want him saying clearly that "yes, I, the arm guy, hereby volunteer to write debian gates" | 00:58 |
| kfox1111 | might double check with him to verify his intent was to volunteer. | 00:58 |
| kfox1111 | yeah. | 00:58 |
| kfox1111 | and spell out explicitly whats needed ahead of time. | 00:59 |
| kfox1111 | "this is the work you need to do to keep it." "do you volunteer to do that?" | 00:59 |
| inc0 | it's a bit rude to call him arm guy tho, I can't remember his nick here | 00:59 |
| inc0 | yup agree | 01:00 |
| inc0 | once we have gates, it's going to be whole lot easier, so that's a first step | 01:00 |
| *** adrian_otto has joined #openstack-kolla | 01:02 | |
| Pavo | so.... hows was everyone's day | 01:02 |
| Pavo | mine was boring as f*** | 01:03 |
| *** schwicht has joined #openstack-kolla | 01:03 | |
| inc0 | Pavo, meetings 9am-2pm | 01:03 |
| inc0 | mind is fried | 01:03 |
| Pavo | inc0 class over paloalto from 8am-4pm | 01:03 |
| Pavo | boring............. | 01:03 |
| Pavo | lol | 01:03 |
| Pavo | firewall is a firewall is a firewall | 01:04 |
| Pavo | lol | 01:04 |
| inc0 | well, at least you're in sunny California;) | 01:04 |
| Pavo | no matter the vendor | 01:04 |
| Pavo | hell I wished | 01:04 |
| Pavo | in cold as hell Baltimore, MD | 01:04 |
| inc0 | ahh....paloalto not as Palo Alto, CA | 01:04 |
| Pavo | Paloalto firewalls lol | 01:04 |
| inc0 | yeah, here in OR it's actually pretty nive | 01:05 |
| Pavo | its like 30 here | 01:05 |
| jascott1 | https://www.youtube.com/watch?v=h9iwI3dbxqs | 01:05 |
| inc0 | I think it's in 40s today here | 01:05 |
| *** msimonin1 has quit IRC | 01:05 | |
| kfox1111 | inc0: sorry. no rudeness intended. just didn't have the email handy. | 01:06 |
| inc0 | what this guy is doing to that poor guitar!? | 01:06 |
| Pavo | lol | 01:06 |
| kfox1111 | paloalto firewall. | 01:06 |
| jascott1 | and the poor xylophone | 01:06 |
| inc0 | kfox1111, let's hope he wont read this and hold it against us;) if he does: sorry :( | 01:06 |
| kfox1111 | +1. sorry. no offence intended. | 01:07 |
| *** eaguilar has quit IRC | 01:08 | |
| *** goldyfruit has quit IRC | 01:09 | |
| *** bpattewar has quit IRC | 01:10 | |
| *** adrian_otto has quit IRC | 01:14 | |
| *** adrian_otto has joined #openstack-kolla | 01:15 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Local registry in gate for trunk builds https://review.openstack.org/418550 | 01:16 |
| *** adrian_otto has quit IRC | 01:16 | |
| *** sdake has quit IRC | 01:22 | |
| *** adrian_otto has joined #openstack-kolla | 01:26 | |
| *** inc0 has quit IRC | 01:26 | |
| kfox1111 | gota get this local repo patch done. :/ | 01:26 |
| *** lamt has joined #openstack-kolla | 01:31 | |
| *** adrian_otto has quit IRC | 01:32 | |
| *** sdake has joined #openstack-kolla | 01:35 | |
| *** sayantani01 has joined #openstack-kolla | 01:37 | |
| *** sayantani01 has quit IRC | 01:37 | |
| *** sayantani01 has joined #openstack-kolla | 01:37 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Capture lists of containers by type. https://review.openstack.org/424921 | 01:38 |
| kfox1111 | and, we're goign to need a kubernetes rpm/deb repo mirror at some point too. :/ | 01:41 |
| *** tovin07_ has joined #openstack-kolla | 01:43 | |
| *** goldyfruit has joined #openstack-kolla | 01:43 | |
| *** schwicht has quit IRC | 01:45 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Replace heka with fluentd in all profiles https://review.openstack.org/424928 | 01:55 |
| *** trey_ has quit IRC | 02:01 | |
| *** jrobinson has quit IRC | 02:01 | |
| *** jrobinson has joined #openstack-kolla | 02:06 | |
| sbezverk | kfox1111: could you ack this fix we talked about: https://review.openstack.org/#/c/424902/1 | 02:06 |
| *** sdake has quit IRC | 02:06 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Change kolla_freezer_sudoers 640 to 440 https://review.openstack.org/424934 | 02:07 |
| jascott1 | kfox1111 qwang is going to work on local repo patch | 02:11 |
| jascott1 | oh sorry you meant registry | 02:12 |
| jascott1 | i was talking about helm local path charts | 02:12 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Capture lists of containers by type. https://review.openstack.org/424921 | 02:15 |
| *** sacharya has joined #openstack-kolla | 02:15 | |
| *** zhubingbing has quit IRC | 02:15 | |
| kfox1111 | jascott1: awesome. :) | 02:15 |
| kfox1111 | sbezverk: done. thanks. | 02:16 |
| *** zhubingbing has joined #openstack-kolla | 02:20 | |
| *** schwicht has joined #openstack-kolla | 02:26 | |
| *** haplo37 has quit IRC | 02:34 | |
| *** eaguilar has joined #openstack-kolla | 02:39 | |
| *** haplo37 has joined #openstack-kolla | 02:42 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Capture lists of containers by type. https://review.openstack.org/424921 | 02:43 |
| *** zhubingbing has quit IRC | 02:45 | |
| *** sudipto has joined #openstack-kolla | 02:47 | |
| *** sudipto_ has joined #openstack-kolla | 02:47 | |
| *** l4yerffeJ has quit IRC | 02:48 | |
| *** l4yerffeJ has joined #openstack-kolla | 02:49 | |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-ansible: Specify 'become' to neccesary tasks (general roles) https://review.openstack.org/398682 | 02:53 |
| *** prameswar has joined #openstack-kolla | 03:00 | |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-ansible: Specify 'become' for only neccesary tasks (default roles) https://review.openstack.org/398684 | 03:06 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-ansible: Specify 'become' for only neccesary tasks (all other roles) https://review.openstack.org/398685 | 03:08 |
| openstackgerrit | sayantani proposed openstack/kolla-ansible: Modify config.yml playbook for bifrost role https://review.openstack.org/419196 | 03:12 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Local registry in gate for trunk builds https://review.openstack.org/418550 | 03:13 |
| *** jrobinson_ has joined #openstack-kolla | 03:14 | |
| *** zhubingbing has joined #openstack-kolla | 03:14 | |
| *** jrobinson_ has quit IRC | 03:16 | |
| *** jrobinson has quit IRC | 03:17 | |
| *** jrobinson has joined #openstack-kolla | 03:17 | |
| *** sdake has joined #openstack-kolla | 03:20 | |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla: [WIP] Apply neutron database migration https://review.openstack.org/407840 | 03:22 |
| *** sp___ has joined #openstack-kolla | 03:22 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Capture lists of containers by type. https://review.openstack.org/424921 | 03:23 |
| *** zhubingbing has quit IRC | 03:29 | |
| *** hfu has joined #openstack-kolla | 03:32 | |
| sp___ | mandre: ping... | 03:34 |
| sp___ | mandre: If I am right you are *Martin André* | 03:34 |
| sp___ | ? | 03:34 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Apply neutron database migration https://review.openstack.org/407840 | 03:36 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-ansible: Apply neutron database migration https://review.openstack.org/407922 | 03:36 |
| SamYaple | sp___: that is who mandre is, yes | 03:39 |
| duonghq | sp___, last few days you pinged me? | 03:42 |
| sp___ | duonghq: yes that was regarding review for coverage BP, but that has been done by Jeffrey4l_ , pbourke and mandre. | 03:44 |
| sp___ | SamYaple: thanks | 03:44 |
| duonghq | sp___, ack | 03:44 |
| duonghq | is it merged? | 03:44 |
| *** eaguilar has quit IRC | 03:44 | |
| sp___ | duonghq: tx for reply | 03:44 |
| duonghq | I busied for awhile | 03:44 |
| duonghq | sorry | 03:44 |
| sp___ | duonghq: no, it is under progress after review | 03:45 |
| sp___ | duonghq: np | 03:45 |
| duonghq | nice | 03:45 |
| sp___ | duonghq: is there any chance of extention of release date ? | 03:46 |
| sp___ | duonghq: today is the last date right ? | 03:46 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Local registry in gate for trunk builds https://review.openstack.org/418550 | 03:47 |
| duonghq | sp___, for kolla and kolla-ansible: today is the last day | 03:47 |
| openstackgerrit | Zeyu Zhu proposed openstack/kolla-ansible: Modify the version_info in pbr https://review.openstack.org/424965 | 03:47 |
| duonghq | no chance for deadline extension | 03:47 |
| sp___ | duonghq: thanks | 03:47 |
| duonghq | for o-3 | 03:47 |
| duonghq | np | 03:48 |
| *** portdirect_away is now known as portdirect | 03:55 | |
| *** mdnadeem has joined #openstack-kolla | 03:59 | |
| *** l4yerffeJ has quit IRC | 04:01 | |
| *** l4yerffeJ has joined #openstack-kolla | 04:01 | |
| *** tovin07 has quit IRC | 04:02 | |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-ansible: Specify 'become' to neccesary tasks (general roles) https://review.openstack.org/398682 | 04:03 |
| sdake | sp___ last day for what? | 04:04 |
| sdake | sp___ for which deliverable? | 04:04 |
| sp___ | sdake: for kolla | 04:04 |
| sdake | sp___ kolla, kolla-ansible follow the release:trialing-cycle schedule | 04:05 |
| sdake | sp__ kolla-kubernetes follows the release:independent schedule (meaning we release when we want) | 04:05 |
| sdake | sp__ if your developing a bug fix, o3 is not the final deadline | 04:05 |
| sdake | sp__ if your developing a blueprint, o3 is the final deadline for blueprints | 04:05 |
| sdake | duonghq there actually is a mechanism for extension - its called an FFE | 04:06 |
| sdake | duonghq for release:trailing-cycle deliverables | 04:06 |
| sdake | feature-freeze extension | 04:06 |
| sdake | you can send a mail to the mailing list and the PTL can decide to approve it or not | 04:06 |
| sdake | if the community disagrees, then i'm not really sure what the process is as we typically bleed into the rcs | 04:07 |
| sdake | for features | 04:07 |
| sdake | although I think that is bad practice especially with the PTG / summit schedules going forward | 04:07 |
| sp___ | sdake: ok, seem it would be completed today. I am working on coverage BP of kollla some more cases to add. | 04:07 |
| sp___ | sdake: thanks for info | 04:07 |
| sdake | sp___ happy to help | 04:07 |
| sdake | inc0 he did indicate he would do that | 04:08 |
| sdake | oh no inc0 | 04:08 |
| sdake | ok - well gotta jet to take kids home bbl | 04:08 |
| *** sdake has quit IRC | 04:08 | |
| *** jrobinson is now known as jrobinson-afk | 04:15 | |
| *** adrian_otto has joined #openstack-kolla | 04:16 | |
| *** adrian_otto has quit IRC | 04:16 | |
| *** skramaja has joined #openstack-kolla | 04:17 | |
| portdirect | o/ morning kolla | 04:18 |
| portdirect | jascott1: delayed pong | 04:19 |
| portdirect | cheers for the head up about plugins | 04:19 |
| duonghq | morning portdirect | 04:20 |
| sbezverk | portdirect: can you look at these two please: https://review.openstack.org/#/c/424697/ and https://review.openstack.org/#/c/424902/1 | 04:23 |
| kfox1111 | There we go... trunk got past neutron now. | 04:28 |
| kfox1111 | now looks like just ceph.... | 04:28 |
| kfox1111 | well, cinder. | 04:28 |
| kfox1111 | last day for o3 kolla stuff? | 04:29 |
| kfox1111 | can we please get https://review.openstack.org/#/c/422950/ in then? | 04:29 |
| *** dave-mccowan has quit IRC | 04:30 | |
| *** lamt has quit IRC | 04:32 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Add cinder control package with kubernetes entrypoint https://review.openstack.org/424697 | 04:34 |
| kfox1111 | ugg... | 04:35 |
| kfox1111 | that means two more rebases. :/ | 04:35 |
| sbezverk | kfox1111: will be done soon with services ;) no more rebases because of that.. | 04:37 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Fixing location of dependency for vswitchd https://review.openstack.org/424902 | 04:38 |
| kfox1111 | yeah. but am trying to get the gate more stable. :/ | 04:38 |
| kfox1111 | which is hard to do with checks failing and rebasing. :/ | 04:38 |
| kfox1111 | not your fault. just frustrating. | 04:38 |
| *** jrobinson-afk is now known as jrobinson | 04:39 | |
| *** sdake_ has joined #openstack-kolla | 04:40 | |
| *** sdake_ has quit IRC | 04:40 | |
| *** sdake_ has joined #openstack-kolla | 04:40 | |
| kfox1111 | cute... | 04:43 |
| kfox1111 | 2017-01-25 04:12:22.975 7 ERROR oslo_messaging.rpc.server libvirtError: internal error: unable to execute QEMU command '__com.redhat_drive_add': Device 'drive-virtio-disk1' could not be initialized | 04:43 |
| SamYaple | kfox1111: im pretty sure the static uid patch wasnt tested before it merged, fyi. so i think it might be related to that | 04:46 |
| *** goldyfruit has quit IRC | 04:47 | |
| kfox1111 | SamYaple: oh. for 4.x containers? | 04:47 |
| kfox1111 | interesting. | 04:47 |
| SamYaple | kfox1111: i tested a version of it on ubuntu. and i got it to pass the gate, but im 95% sure no one tested centos family beyond "it passed the gate" | 04:48 |
| SamYaple | but i ran out of time to maintain the patch and abandonded it. someone else finished it | 04:48 |
| kfox1111 | ah. ok. | 04:49 |
| kfox1111 | sbezverk: yeah, I think I've seen several cases of gate failure due to openvswitch-agent coming up before vswitchd now. | 05:02 |
| kfox1111 | http://logs.openstack.org/50/418550/55/check/gate-kolla-kubernetes-deploy-centos-binary-3-ceph-multi-nv/caf246b/console.html | 05:02 |
| kfox1111 | SamYaple: libvirt logs here: http://logs.openstack.org/50/418550/55/check/gate-kolla-kubernetes-deploy-centos-binary-t-ceph-multi-nv/245adef/logs/openstack/kolla-nova-libvirt-p9376.txt they are rather large, so b ecareful. | 05:09 |
| SamYaple | kfox1111: oh thats ceph related | 05:10 |
| SamYaple | kfox1111: ceph probably isnt operational or it crashed | 05:11 |
| kfox1111 | ceph's happy. | 05:11 |
| kfox1111 | the vm was launched via glance. | 05:12 |
| kfox1111 | the full ceph logs are here: http://logs.openstack.org/50/418550/55/check/gate-kolla-kubernetes-deploy-centos-binary-t-ceph-multi-nv/245adef/logs/ceph.log | 05:12 |
| kfox1111 | I can't find anything in particular wrong in the logs. :/ | 05:12 |
| SamYaple | dontknow.jpg | 05:13 |
| SamYaple | but im pretty sure thats ceph related | 05:13 |
| SamYaple | possibly permissions related to ceph | 05:13 |
| kfox1111 | last time I saw something like that, it was a bad rpm installed in the container. :/ | 05:13 |
| SamYaple | sounds right | 05:13 |
| SamYaple | kraken just related btw | 05:14 |
| SamYaple | hopefully you arent accidentally pulling that in | 05:14 |
| *** lamt has joined #openstack-kolla | 05:14 | |
| kfox1111 | SamYaple: cool. | 05:15 |
| kfox1111 | looks like its pinned in trunk to jewel. | 05:15 |
| SamYaple | yea. but _still_ | 05:16 |
| SamYaple | double check :P | 05:16 |
| kfox1111 | not easy... the tarball its pulling from is in docker repo format. :/ | 05:16 |
| kfox1111 | oh.. yeah. the ones portdirect is posting should be similar. | 05:19 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: Optimize reconfigure action for nova https://review.openstack.org/407416 | 05:20 |
| kfox1111 | gotta head out. nite all. | 05:23 |
| kfox1111 | it is jewel btw. | 05:24 |
| kfox1111 | just pulled and verified. | 05:24 |
| *** huzhengchuan has quit IRC | 05:36 | |
| *** tonanhngo has quit IRC | 05:36 | |
| *** tonanhngo has joined #openstack-kolla | 05:37 | |
| *** tonanhngo has quit IRC | 05:41 | |
| openstackgerrit | Surya Prakash Singh proposed openstack/kolla: Code coverage improvement for kolla https://review.openstack.org/422444 | 05:44 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Add cinder control package with kubernetes entrypoint https://review.openstack.org/424697 | 05:46 |
| *** sudswas__ has joined #openstack-kolla | 05:47 | |
| *** sudipto has quit IRC | 05:50 | |
| *** sudipto_ has quit IRC | 05:50 | |
| *** sudipto has joined #openstack-kolla | 05:50 | |
| *** sacharya has quit IRC | 05:51 | |
| jascott1 | portdirect top of the morning to ye | 06:00 |
| portdirect | word jascott1, hows it going dude? | 06:04 |
| jascott1 | all good. will be seeing you in ATL krom willing | 06:04 |
| jascott1 | portdirect how are things on the east side (of the atlantic)? | 06:06 |
| *** zhubingbing has joined #openstack-kolla | 06:07 | |
| portdirect | damp | 06:08 |
| *** skramaja has quit IRC | 06:09 | |
| jascott1 | damp here as well :/ | 06:10 |
| jascott1 | then again it is portland http://isitraininginportland.com/ | 06:10 |
| sdake_ | portdirect any chance you got the dev env to go | 06:10 |
| jascott1 | hey sdake | 06:11 |
| sdake_ | portdirect i got to the point of iscsi complaints | 06:11 |
| sdake_ | sup jascott1 | 06:11 |
| *** slagle has quit IRC | 06:15 | |
| *** prameswar has quit IRC | 06:15 | |
| *** jmontleon has quit IRC | 06:16 | |
| *** slagle has joined #openstack-kolla | 06:17 | |
| *** jmontleon has joined #openstack-kolla | 06:19 | |
| *** bjolo has quit IRC | 06:27 | |
| *** bmace_ has quit IRC | 06:30 | |
| *** zhubingbing has quit IRC | 06:34 | |
| *** Jeffrey4l_ has quit IRC | 06:48 | |
| *** zhubingbing has joined #openstack-kolla | 06:50 | |
| *** Jeffrey4l has joined #openstack-kolla | 06:53 | |
| *** trinaths has joined #openstack-kolla | 06:53 | |
| *** jrobinson has quit IRC | 06:54 | |
| *** tovin07 has joined #openstack-kolla | 06:54 | |
| openstackgerrit | Anh Tran proposed openstack/kolla-kubernetes: Typo fix: folowing => following https://review.openstack.org/425008 | 06:59 |
| *** sdake_ has quit IRC | 07:01 | |
| *** zhubingbing has quit IRC | 07:05 | |
| *** bmace_ has joined #openstack-kolla | 07:07 | |
| *** zhubingbing has joined #openstack-kolla | 07:08 | |
| *** sudipto_ has joined #openstack-kolla | 07:13 | |
| *** sudipto has quit IRC | 07:14 | |
| *** sudswas__ has quit IRC | 07:14 | |
| *** sudipto has joined #openstack-kolla | 07:14 | |
| *** msimonin has joined #openstack-kolla | 07:16 | |
| *** matrohon has joined #openstack-kolla | 07:16 | |
| *** msimonin has quit IRC | 07:17 | |
| *** msimonin has joined #openstack-kolla | 07:17 | |
| *** msimonin has quit IRC | 07:18 | |
| *** msimonin has joined #openstack-kolla | 07:18 | |
| *** msimonin has quit IRC | 07:18 | |
| *** msimonin has joined #openstack-kolla | 07:19 | |
| *** msimonin has quit IRC | 07:19 | |
| *** msimonin has joined #openstack-kolla | 07:19 | |
| *** msimonin has quit IRC | 07:20 | |
| *** matrohon has quit IRC | 07:23 | |
| *** pcaruana has joined #openstack-kolla | 07:28 | |
| *** hfu has quit IRC | 07:29 | |
| *** tovin07 has quit IRC | 07:32 | |
| *** britthouser7 has joined #openstack-kolla | 07:34 | |
| *** zhubingbing has quit IRC | 07:35 | |
| *** klindgren_ has joined #openstack-kolla | 07:35 | |
| *** sayantan_ has joined #openstack-kolla | 07:36 | |
| *** klindgren has quit IRC | 07:38 | |
| *** sayantani01 has quit IRC | 07:38 | |
| *** britthouser has quit IRC | 07:38 | |
| *** mgoddard has joined #openstack-kolla | 07:39 | |
| *** prameswar has joined #openstack-kolla | 07:45 | |
| *** tovin07 has joined #openstack-kolla | 07:45 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for elasticsearch https://review.openstack.org/424139 | 07:48 |
| openstackgerrit | Surya Prakash Singh proposed openstack/kolla: Docs updated to get coverage report https://review.openstack.org/425032 | 07:52 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: Optimize reconfigure action for horizon https://review.openstack.org/422719 | 07:57 |
| *** ingmar23 has joined #openstack-kolla | 07:59 | |
| *** bjolo has joined #openstack-kolla | 08:00 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for solum https://review.openstack.org/424167 | 08:04 |
| *** saneax-_-|AFK is now known as saneax | 08:06 | |
| *** tovin07 has quit IRC | 08:13 | |
| *** shardy has joined #openstack-kolla | 08:19 | |
| *** huzhengchuan has joined #openstack-kolla | 08:23 | |
| *** tovin07 has joined #openstack-kolla | 08:29 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 08:30 | |
| *** l4yerffeJ_ has joined #openstack-kolla | 08:30 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: Run detect openstack_release variable play always https://review.openstack.org/425050 | 08:33 |
| *** l4yerffeJ has quit IRC | 08:34 | |
| *** egonzalez has joined #openstack-kolla | 08:34 | |
| *** Jeffrey4l has quit IRC | 08:34 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfigure action for manila https://review.openstack.org/421365 | 08:35 |
| *** msimonin has joined #openstack-kolla | 08:37 | |
| ingmar23 | exit | 08:37 |
| *** ingmar23 has quit IRC | 08:37 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible: Custom init-runonce values https://review.openstack.org/424682 | 08:37 |
| *** zhubingbing has joined #openstack-kolla | 08:37 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for aodh https://review.openstack.org/423670 | 08:41 |
| *** hfu has joined #openstack-kolla | 08:42 | |
| openstackgerrit | Pete Birley proposed openstack/kolla-kubernetes: Update docs to pin k8s version to 1.5.1 and remove helm pull workaround https://review.openstack.org/425059 | 08:44 |
| openstackgerrit | Pete Birley proposed openstack/kolla-kubernetes: Update docs to pin k8s version to 1.5.1 and remove helm pull workaround https://review.openstack.org/425059 | 08:47 |
| openstackgerrit | Pete Birley proposed openstack/kolla-kubernetes: Update docs to pin K8s at v1.5.1 and remove helm pull workaround https://review.openstack.org/425059 | 08:48 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible: Integrate designate with neutron https://review.openstack.org/424679 | 08:52 |
| egonzalez | portdirect: ping | 08:52 |
| trinaths | hi, while setting up the kolla-kubernetes environment, the was struck at Ansible tasks TASK [kube-join : join workers to the initialized kubernetes cluster] ********** | 08:56 |
| trinaths | from a hour its struck there. Any clue to debug it ? | 08:56 |
| sp_ | egonzalez: if you have sometime in this crunchy situation, would be great if you review these https://review.openstack.org/#/c/425000/ https://review.openstack.org/#/c/425032/ https://review.openstack.org/#/c/422444/ | 08:58 |
| sp_ | egonzalez: I think won't take much time. Please review | 08:59 |
| *** hfu has quit IRC | 08:59 | |
| *** tovin07 has quit IRC | 09:00 | |
| openstackgerrit | sayantani proposed openstack/kolla: [WIP] Update Multinode deployment of kolla guide https://review.openstack.org/408364 | 09:01 |
| egonzalez | sp_: added a comment https://review.openstack.org/#/c/425000/2 | 09:02 |
| sp_ | egonzalez: done | 09:04 |
| sp_ | was a typo | 09:04 |
| sp_ | thanks | 09:04 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for kuryr https://review.openstack.org/424161 | 09:05 |
| *** sayantan_ has quit IRC | 09:07 | |
| *** newmember has joined #openstack-kolla | 09:08 | |
| *** hfu has joined #openstack-kolla | 09:08 | |
| *** Serlex has joined #openstack-kolla | 09:10 | |
| portdirect | egonzalez: pong | 09:10 |
| egonzalez | is this bp ready to mark as implemented? https://blueprints.launchpad.net/kolla/+spec/add-kubernetes-images | 09:11 |
| egonzalez | portdirect: ^^ | 09:11 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for octavia https://review.openstack.org/423846 | 09:11 |
| portdirect | egonzalez: yeah the kolla side of that's done | 09:12 |
| egonzalez | portdirect: thanks | 09:12 |
| portdirect | egonzalez: so its ok to mark as complete - cheers | 09:12 |
| portdirect | trinaths: ping | 09:12 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for octavia https://review.openstack.org/423846 | 09:14 |
| *** zhurong has quit IRC | 09:14 | |
| *** tovin07 has joined #openstack-kolla | 09:15 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible: Integrate designate with neutron https://review.openstack.org/424679 | 09:18 |
| n0isyn0ise | hi kolla-team, i'm using kolla from the stable/newton branch, I am able to build images with kolla build but I am now trying to use the --build-args option with http_proxy and no_proxy but the no_proxy seems to not be handled, or i use it badly - Maybe someone has an advice about its use ? | 09:18 |
| *** portdirect is now known as portdirect_brb | 09:22 | |
| *** Mech422 has quit IRC | 09:23 | |
| *** Mech422 has joined #openstack-kolla | 09:24 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible: Custom init-runonce values https://review.openstack.org/424682 | 09:24 |
| *** hfu_ has joined #openstack-kolla | 09:26 | |
| *** hfu has quit IRC | 09:29 | |
| trinaths | portdirect_brb: pong | 09:30 |
| trinaths | portdirect_brb: i was struck at task, TASK [kube-join : join workers to the initialized kubernetes cluster] | 09:31 |
| trinaths | portdirect_brb: log here, http://paste.openstack.org/show/596394/ | 09:32 |
| *** portdirect_brb is now known as portdirect | 09:32 | |
| portdirect | yeah i hit the same thing this morning | 09:32 |
| portdirect | got a ps in to fix the docs and resovle the tiller issue with pulling images from gcr.io | 09:33 |
| portdirect | can we destory that cluster? | 09:33 |
| portdirect | and then run the ./setup-halcyon.sh script again | 09:33 |
| portdirect | but this time specify 'v1.5.1' as the k8s version | 09:34 |
| portdirect | can then we can make a change to the ansible to man you dont need the image pull workaround | 09:34 |
| portdirect | in kube-deploy/roles/deploy-kube/tasks/centos.yml | 09:35 |
| portdirect | can you make the changes from here: | 09:35 |
| portdirect | https://github.com/att-comdev/halcyon-kubernetes/pull/43/files | 09:35 |
| *** mdnadeem has quit IRC | 09:37 | |
| *** mdnadeem has joined #openstack-kolla | 09:39 | |
| *** tovin07 has quit IRC | 09:41 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible: Optimize reconfiguration for tacker https://review.openstack.org/422036 | 09:41 |
| *** sp___ has quit IRC | 09:42 | |
| trinaths | portdirect: doing. | 09:45 |
| portdirect | if you can i'd also make another change | 09:45 |
| portdirect | in the config.rb file | 09:45 |
| portdirect | I'd strongly recommend upping the ram and vcpu given to each node | 09:46 |
| trinaths | portdirect: okay. please tell me. just destroyed the cluster | 09:46 |
| trinaths | oh i have workstation with 16GB RAM | 09:46 |
| *** msimonin has quit IRC | 09:46 | |
| portdirect | if you can up it to 3072 per node that would be great | 09:47 |
| portdirect | but obviously the hardware you have available limits the scope there | 09:47 |
| portdirect | I'm running with 6GB ram and 4vcpu per node - but then have a bit more to play with on my workstation | 09:48 |
| *** strigazi_AFK is now known as strigazi | 09:50 | |
| trinaths | portdirect: ok | 09:51 |
| *** athomas has joined #openstack-kolla | 09:53 | |
| *** tovin07_ has quit IRC | 09:55 | |
| *** mgoddard has quit IRC | 10:03 | |
| *** cuongnv has quit IRC | 10:03 | |
| *** duonghq has quit IRC | 10:05 | |
| Serlex | thanks for the info portdirect. Have you come across " TASK [kube-init : check for bootstrap]" "The connection to the server localhost:8080 was refused" | 10:06 |
| *** msimonin has joined #openstack-kolla | 10:12 | |
| *** sambetts|afk is now known as sambetts | 10:13 | |
| *** simon-AS559 has joined #openstack-kolla | 10:15 | |
| *** hfu_ has quit IRC | 10:16 | |
| *** openstackgerrit has quit IRC | 10:17 | |
| *** skramaja has joined #openstack-kolla | 10:18 | |
| *** mgoddard has joined #openstack-kolla | 10:19 | |
| *** egonzalez has quit IRC | 10:20 | |
| *** mdnadeem has quit IRC | 10:21 | |
| portdirect | Serlex: :( | 10:23 |
| portdirect | i think i know what may be up there | 10:23 |
| Serlex | I missed a setup or a proxy issue | 10:24 |
| portdirect | can you run the ./setup-halcyon.sh specifying 'v1.5.1' as the k8s version to use | 10:24 |
| portdirect | if you have a proxy then that could well be a cause | 10:24 |
| Serlex | not setup...step* | 10:24 |
| Serlex | ok sure, I will check | 10:24 |
| portdirect | and unfortunately my knowledge of that are is very small :( | 10:24 |
| portdirect | *area | 10:24 |
| *** portdirect is now known as portdirect_away | 10:27 | |
| *** mdnadeem has joined #openstack-kolla | 10:29 | |
| *** pbourke has quit IRC | 10:29 | |
| *** pbourke has joined #openstack-kolla | 10:30 | |
| *** msimonin has quit IRC | 10:36 | |
| *** gfidente has joined #openstack-kolla | 10:40 | |
| *** gfidente has quit IRC | 10:40 | |
| *** gfidente has joined #openstack-kolla | 10:40 | |
| *** nrado has joined #openstack-kolla | 10:49 | |
| *** newmember has quit IRC | 11:00 | |
| *** egonzalez has joined #openstack-kolla | 11:02 | |
| *** nrado has quit IRC | 11:05 | |
| *** nrado has joined #openstack-kolla | 11:05 | |
| egonzalez | zhubingbing: ping | 11:07 |
| *** nrado has quit IRC | 11:08 | |
| *** nrado has joined #openstack-kolla | 11:08 | |
| *** hfu has joined #openstack-kolla | 11:15 | |
| *** papacz has joined #openstack-kolla | 11:16 | |
| *** openstackgerrit has joined #openstack-kolla | 11:20 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for solum https://review.openstack.org/424167 | 11:20 |
| openstackgerrit | Merged openstack/kolla-ansible: Optimize reconfiguration for mistral https://review.openstack.org/423787 | 11:25 |
| *** eaguilar has joined #openstack-kolla | 11:28 | |
| mliima | morning guys | 11:28 |
| egonzalez | morning mliima , how things going? | 11:29 |
| mliima | fine egonzalez :) | 11:30 |
| mliima | and you? | 11:30 |
| openstackgerrit | Merged openstack/kolla-ansible: Optimize reconfiguration for watcher https://review.openstack.org/424105 | 11:38 |
| openstackgerrit | Merged openstack/kolla-ansible: Optimize reconfiguration for sahara https://review.openstack.org/424165 | 11:38 |
| *** schwicht has quit IRC | 11:39 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla: Reduce size of images built from git repos https://review.openstack.org/421252 | 11:43 |
| *** mgoddard has quit IRC | 11:45 | |
| openstackgerrit | Mauricio Lima proposed openstack/kolla-ansible: Optimize reconfiguration for grafana https://review.openstack.org/424158 | 11:48 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla-ansible: Optimize reconfiguration for grafana https://review.openstack.org/424158 | 11:51 |
| openstackgerrit | serlex proposed openstack/kolla-kubernetes: Added option to use --provide with vagrant https://review.openstack.org/425148 | 11:57 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla-ansible: Optimize reconfiguration for kuryr https://review.openstack.org/424161 | 11:59 |
| openstackgerrit | Nenad Radojevic proposed openstack/kolla-ansible: Use neutron user in neutron_lbaas.conf and overwrite defaults https://review.openstack.org/425150 | 11:59 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for kuryr https://review.openstack.org/424161 | 12:00 |
| *** hfu has quit IRC | 12:00 | |
| *** hfu has joined #openstack-kolla | 12:00 | |
| *** hfu has quit IRC | 12:01 | |
| *** hfu has joined #openstack-kolla | 12:01 | |
| *** hfu has quit IRC | 12:01 | |
| openstackgerrit | Mauricio Lima proposed openstack/kolla-ansible: Optimize reconfiguration for collectd https://review.openstack.org/424133 | 12:02 |
| *** portdirect_away is now known as portdirect | 12:02 | |
| *** hfu has joined #openstack-kolla | 12:02 | |
| *** hfu has quit IRC | 12:02 | |
| *** hfu has joined #openstack-kolla | 12:03 | |
| *** tonanhngo has joined #openstack-kolla | 12:03 | |
| *** hfu has quit IRC | 12:03 | |
| *** hfu has joined #openstack-kolla | 12:03 | |
| *** hfu has quit IRC | 12:04 | |
| openstackgerrit | Mauricio Lima proposed openstack/kolla-ansible: Optimize reconfiguration for influxdb https://review.openstack.org/424159 | 12:04 |
| *** tonanhngo has quit IRC | 12:08 | |
| *** haplo37_ has quit IRC | 12:08 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for kuryr https://review.openstack.org/424161 | 12:09 |
| *** haplo37_ has joined #openstack-kolla | 12:10 | |
| openstackgerrit | Mauricio Lima proposed openstack/kolla-ansible: Optimize reconfiguration for designate https://review.openstack.org/424155 | 12:12 |
| *** trinaths has left #openstack-kolla | 12:14 | |
| portdirect | Serlex: regarding https://review.openstack.org/425148, did you need to use this option or is it just for info? | 12:17 |
| Serlex | Hi, I had to use it and also a good info? | 12:19 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla-ansible: Optimize reconfigure action for cinder https://review.openstack.org/422384 | 12:19 |
| Serlex | is that ok? | 12:19 |
| *** shardy is now known as shardy_lunch | 12:20 | |
| portdirect | yeah thats great | 12:20 |
| portdirect | good to know - I've never needed to specify it manually myself | 12:20 |
| Serlex | phew...firrst commit :D | 12:21 |
| Serlex | probably last | 12:21 |
| portdirect | lol - docs are the most important thing as well! | 12:21 |
| portdirect | dont say that :( | 12:21 |
| Serlex | I was been critical of myself :D | 12:22 |
| portdirect | always better to undersell and over-deliver :) | 12:24 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for karbor https://review.openstack.org/424160 | 12:24 |
| egonzalez | Serlex: I would use a lower case in the You ;) | 12:24 |
| wznoinsk | hi all | 12:27 |
| egonzalez | wznoinsk: sup | 12:27 |
| *** mgiles has joined #openstack-kolla | 12:27 | |
| wznoinsk | hi egonzalez , I'm wondering whether the problem I have a trivial one or not... I have nova_compute container that qemu-nbd mounts /dev/nbdS under /var/lib/nova/instances/UUID/rootfs, /var/lib/nova is a docker volume from the host but because nbd is mounted on top of it in nova_compute container the nova_libvirt container doesn't see it, anyone seen/fix that? | 12:29 |
| *** eaguilar has quit IRC | 12:29 | |
| wznoinsk | a minor fact: i'm using lxc virt_type of libvirt, but I guess the same problem would happen with any other virt_type if you're using raw type of image (lxc converts qcow2 -> raw before the above mounting as it needs that format) | 12:32 |
| Jeffrey4l_ | egonzalez, could u review this https://review.openstack.org/424928 which unblock kolla's gate. | 12:33 |
| *** rmart04 has joined #openstack-kolla | 12:33 | |
| egonzalez | Jeffrey4l_: done | 12:34 |
| Jeffrey4l_ | thanks. | 12:35 |
| Jeffrey4l_ | egonzalez, another big one https://review.openstack.org/416046 ;) | 12:35 |
| *** eaguilar has joined #openstack-kolla | 12:36 | |
| egonzalez | wznoinsk: never saw that befero, never tried lxc as virt_type, maybe need to install some dependency and other configuration. Can you try with qemu/kvm? | 12:36 |
| Serlex | egonzalez: it was suppose to be a new line | 12:37 |
| Serlex | But I will change it now | 12:37 |
| egonzalez | Jeffrey4l_: doing, gerrit is running so slow today :/ | 12:38 |
| *** sudipto has quit IRC | 12:39 | |
| *** sudipto_ has quit IRC | 12:39 | |
| Jeffrey4l_ | egonzalez, recommend to use gertty https://github.com/openstack/gertty | 12:39 |
| openstackgerrit | Mark Giles proposed openstack/kolla-kubernetes: Nova-control service deployment https://review.openstack.org/421457 | 12:39 |
| *** sdake has joined #openstack-kolla | 12:45 | |
| sdake | morning peeps | 12:45 |
| -openstackstatus- NOTICE: Gerrit is going to be restarted due to slow performance | 12:46 | |
| wznoinsk | egonzalez, qemu/kvm works fine as it's ok with using qcow2 file which is "dropped" to /var/lib/nova/instances/UUID , qemu-system-x86_64 has this as device -drive file=/var/lib/nova/instances/62bbb295-2df9-4f28-819e-b99df661384f/disk,format=qcow2 | 12:47 |
| wznoinsk | egonzalez, lxc needs a filesystem 'unpacked' from the image, it does it via binding an image to /dev/nbdX device and mount that devices as /var/lib/nova/instances/UUID/rootfs in nova_compute container (not seen by nova_libvirt) - hope the explanation is clear enough | 12:48 |
| egonzalez | wznoinsk: IIRC lxc need a nova-compute-lxc package installed | 12:49 |
| -openstackstatus- NOTICE: Gerrit has been successfully restarted | 12:49 | |
| wznoinsk | egonzalez, it has, from nova point of view everything works fine | 12:50 |
| egonzalez | wznoinsk: as per your explanation, maybe need to add some mount points to the container | 12:50 |
| openstackgerrit | serlex proposed openstack/kolla-kubernetes: Added option to use --provide with vagrant. https://review.openstack.org/425148 | 12:50 |
| wznoinsk | egonzalez, yes, something around that, there's one mount per lxc instance spawned so not sure how to achieve that so the mount is seen by both containers | 12:51 |
| wznoinsk | egonzalez, in short: I'm mounting something in containerA and I wanted containerB to see that as well | 12:52 |
| wznoinsk | this issue happens only when using Kolla, normally nova_compute/nova_libvirt have access to the same directory and mountpoint on the host hence not being a problem | 12:53 |
| *** sp___ has joined #openstack-kolla | 12:53 | |
| *** nrado has quit IRC | 12:55 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for karbor https://review.openstack.org/424160 | 12:56 |
| egonzalez | wznoinsk: only this mount points are bind in libvirt and not in nova /sys/fs/cgroup /etc/libvirt/qemu | 12:59 |
| wznoinsk | I see the same /var/lib/nova/instances in both containers, that's correct | 13:00 |
| wznoinsk | qemu-nbd mounts lxc instance rootfs over a subdirectory of /var/lib/nova/instances in nova_compute container tho | 13:00 |
| egonzalez | wznoinsk: what changes you made in docker images to install lxc driver? | 13:02 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for karbor https://review.openstack.org/424160 | 13:04 |
| *** hfu has joined #openstack-kolla | 13:05 | |
| sdake | egonzalez how are ou liking ibm these dyas | 13:07 |
| *** eaguilar has quit IRC | 13:10 | |
| wznoinsk | egonzalez, very few, two additional mount points at the container start task and install of nova-compute-lxc | 13:10 |
| *** hfu has quit IRC | 13:10 | |
| wznoinsk | hi sdake | 13:10 |
| sp___ | hi... sdake | 13:11 |
| *** zioproto has joined #openstack-kolla | 13:13 | |
| *** eaguilar has joined #openstack-kolla | 13:15 | |
| egonzalez | sdake: all fine, today I had to give a presentation talk and evengelized kolla around IBMers hehe | 13:16 |
| sdake | egonzalez nice! | 13:17 |
| sdake | sup wznoinsk sp___ | 13:17 |
| sdake | I have a 2 hour seminar on all things kolla next months | 13:17 |
| sdake | next month in february | 13:17 |
| sdake | 2 hours - goran - that will take some time | 13:17 |
| sdake | egonzalez do you hae any material you prepped you can share that i can crib from | 13:17 |
| sp___ | sdake: there are some review items here | 13:18 |
| sp___ | sdake: if you got some free time please review these | 13:18 |
| sp___ | https://review.openstack.org/#/c/425000/ https://review.openstack.org/#/c/425032/ https://review.openstack.org/#/c/422444/16 | 13:19 |
| sdake | sp___ actually now is the school hour | 13:19 |
| sdake | sp___ from around 6-9am every day | 13:19 |
| mgiles | Are the gates completely broken this morning? | 13:19 |
| sdake | sp___ (wife out of town so have no coverage) | 13:19 |
| mgiles | this doesn't look normal: http://status.openstack.org/zuul/ | 13:19 |
| sdake | mgiles - i will check that - which deliverable | 13:19 |
| mgiles | all of them | 13:19 |
| sdake | mgiles gerrit was restarted earlier | 13:19 |
| sdake | mgiles try a recheck | 13:19 |
| sp___ | sdake: thank, np | 13:19 |
| mgiles | it's not that they're failing, they just aren't running | 13:20 |
| mgiles | I don't know if a recheck will fix that? I can try | 13:20 |
| sdake | mgiles - a recheck should fix it | 13:21 |
| sdake | mgiles if gerrit restarts, it causes all kinds of chaos with the gates | 13:21 |
| mgiles | ok. thanks sdake | 13:21 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for karbor https://review.openstack.org/424160 | 13:25 |
| mgiles | sdake Still not seeing tests running | 13:25 |
| *** shardy_lunch is now known as shardy | 13:26 | |
| *** v1k0d3n has quit IRC | 13:27 | |
| sdake | mgiles openstack-infra is the place to go :) | 13:28 |
| sdake | mgiles join the channel and i can get hte conversation started | 13:28 |
| openstackgerrit | Merged openstack/kolla: Replace heka with fluentd in all profiles https://review.openstack.org/424928 | 13:29 |
| *** v1k0d3n has joined #openstack-kolla | 13:29 | |
| sdake | mgiles I mean #openstack-infra on irc | 13:29 |
| sdake | mgiles note the gate jobs take a few minutes to actually show up in zuul | 13:30 |
| mgiles | yeah, just seeing a lot of stuff queued form a long time ago, but maybe it's all related to the gerrit reboot | 13:30 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla-ansible: Optimize reconfiguration for grafana https://review.openstack.org/424158 | 13:35 |
| sdake | mgiles nice you got he conversation started on your own :) | 13:35 |
| *** williamcaban has joined #openstack-kolla | 13:37 | |
| *** lamt has quit IRC | 13:37 | |
| *** lamt has joined #openstack-kolla | 13:38 | |
| *** rhallisey has joined #openstack-kolla | 13:38 | |
| *** lamt has quit IRC | 13:42 | |
| openstackgerrit | Mauricio Lima proposed openstack/kolla-ansible: Optimize reconfiguration for influxdb https://review.openstack.org/424159 | 13:45 |
| *** zhubingbing has quit IRC | 13:49 | |
| *** zhubingbing has joined #openstack-kolla | 13:52 | |
| *** mdnadeem has quit IRC | 13:53 | |
| *** ipsecguy has quit IRC | 13:53 | |
| *** mdnadeem has joined #openstack-kolla | 13:55 | |
| *** sp___ has quit IRC | 13:55 | |
| *** goldyfruit has joined #openstack-kolla | 13:59 | |
| *** ipsecguy has joined #openstack-kolla | 14:01 | |
| openstackgerrit | Merged openstack/kolla-ansible: Optimize reconfiguration for barbican https://review.openstack.org/423728 | 14:02 |
| openstackgerrit | Saverio Proto proposed openstack/kolla: Use become: True when updating /etc/hosts https://review.openstack.org/425191 | 14:02 |
| *** dmsimard|off is now known as dmsimard | 14:02 | |
| zioproto | Bugfix for stable/newton are still accepted I hope ? | 14:02 |
| egonzalez | zioproto: yup, but need to be a backport from a patch from master | 14:05 |
| zioproto | egonzalez: it is a cherry pick | 14:05 |
| *** tonanhngo has joined #openstack-kolla | 14:07 | |
| egonzalez | hrm, become line is in different order than master PS, order in this case does not matter but is not cherry-picked | 14:09 |
| *** matrohon has joined #openstack-kolla | 14:15 | |
| *** dave-mccowan has joined #openstack-kolla | 14:16 | |
| *** jpstnh has joined #openstack-kolla | 14:16 | |
| *** skramaja has quit IRC | 14:19 | |
| *** hfu has joined #openstack-kolla | 14:20 | |
| *** zhubingbing has quit IRC | 14:21 | |
| *** zhubingbing has joined #openstack-kolla | 14:23 | |
| *** williamcaban has quit IRC | 14:23 | |
| *** jtriley has joined #openstack-kolla | 14:24 | |
| *** duonghq has joined #openstack-kolla | 14:25 | |
| *** huzhengchuan has quit IRC | 14:25 | |
| zioproto | egonzalez: cherry pick had a conflict that I had to resolve | 14:26 |
| zioproto | I have a beginner question. I face yet another Become: true problem | 14:27 |
| zioproto | do docker pull do I always need sudo ? | 14:27 |
| zioproto | This is not working for me: https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/common/tasks/pull.yml#L2 | 14:28 |
| zioproto | without become True | 14:28 |
| zioproto | I run the command | 14:29 |
| zioproto | kolla-ansible -i multinode pull | 14:29 |
| zioproto | and I fail because of permission error | 14:29 |
| *** japestinho_ has joined #openstack-kolla | 14:29 | |
| *** egonzalez has quit IRC | 14:30 | |
| zioproto | so anywhere I have a pull | 14:30 |
| zioproto | I mean a docker pull | 14:31 |
| zioproto | I have a permission error | 14:31 |
| zioproto | but I installed docker with Kolla as well | 14:31 |
| zioproto | using the bootstrap-servers kolla tool | 14:31 |
| *** jpstnh has quit IRC | 14:33 | |
| *** fguillot has joined #openstack-kolla | 14:33 | |
| *** williamcaban has joined #openstack-kolla | 14:38 | |
| *** huzhengchuan has joined #openstack-kolla | 14:40 | |
| zioproto | anyone ? | 14:42 |
| zioproto | should I fill a bug ? | 14:42 |
| rhallisey | what's the error? | 14:42 |
| rhallisey | docker giving you a perm error? | 14:43 |
| *** williamcaban has quit IRC | 14:43 | |
| zioproto | http://paste.openstack.org/show/596459/ | 14:43 |
| zioproto | It is always a permission error | 14:43 |
| zioproto | I can Become: true | 14:43 |
| zioproto | and go ahead | 14:43 |
| zioproto | but | 14:43 |
| zioproto | I would like to understand if I got something wrong with the permissions | 14:43 |
| zioproto | I am just following the Newton documentation | 14:44 |
| zioproto | is 'docker pull' a command that generally needs sudo rights ? | 14:44 |
| rhallisey | ya you generally need sudo | 14:45 |
| rhallisey | test it by hand | 14:45 |
| zioproto | of course | 14:45 |
| zioproto | then I need to patch here | 14:45 |
| zioproto | https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/common/tasks/pull.yml#L2 | 14:45 |
| zioproto | all these pull actions | 14:45 |
| zioproto | they should require Become: true | 14:46 |
| zioproto | am I wrong ? | 14:46 |
| zioproto | probably who wrote this code was using only the root user ? is this possible ? | 14:46 |
| *** TxGirlGeek has joined #openstack-kolla | 14:47 | |
| rhallisey | ya I think you'll need a become: True | 14:47 |
| rhallisey | kolla points to docker docs that ask you to setup docker this way | 14:48 |
| rhallisey | so if you don't do that, this task will fail without root | 14:48 |
| openstackgerrit | Saverio Proto proposed openstack/kolla-ansible: Use become: True when pulling docker images https://review.openstack.org/425215 | 14:49 |
| rhallisey | zioproto, I think you'll need that for all images | 14:50 |
| rhallisey | I would also file a bug | 14:50 |
| *** matrohon_ has joined #openstack-kolla | 14:50 | |
| sdake | hey folks, I have a meeting conflict during our team meeting today - so won't be able to make it | 14:52 |
| sdake | I'll be with you in spirit :) | 14:52 |
| sdake | someone woot for me plz :) | 14:52 |
| sdake | (and I'll read the logs today) | 14:52 |
| *** matrohon has quit IRC | 14:53 | |
| zioproto | rhallisey: https://bugs.launchpad.net/kolla-ansible/+bug/1659318 | 14:54 |
| openstack | Launchpad bug 1659318 in kolla-ansible "Docker pull needs become: True" [Undecided,New] | 14:54 |
| *** matrohon__ has joined #openstack-kolla | 14:55 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes: WIP: Add Development Setup Script for use with Kubernetes Helm https://review.openstack.org/425219 | 14:55 |
| sdake | groan | 14:56 |
| *** matrohon_ has quit IRC | 14:57 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes: WIP: Add Development Setup Script for use with Kubernetes Helm https://review.openstack.org/405720 | 14:57 |
| sean-k-mooney | rhallisey: do you allways want to assume that we have not added the current use to the docker group though by adding become=true | 14:58 |
| sdake | mgiles bmace_ ^^ that last rev is where I get to with the iscsi error | 14:58 |
| *** TxGirlGeek has quit IRC | 14:58 | |
| sdake | sean-k-mooney i want to speak with you about how you setup outlook when I get a chance, but that wont be for a few hours | 14:58 |
| sdake | sean-k-mooney will you be around then today - or will it be beer-o-clock for ya | 14:58 |
| sean-k-mooney | sdake: sure if i can help i will | 14:58 |
| rhallisey | sean-k-mooney, I mentioned that the kolla links to docs that tell you to do so | 14:58 |
| *** TxGirlGeek has joined #openstack-kolla | 14:58 | |
| sdake | sean-k-mooney there is some config option you are using that seems to allow inline responses | 14:59 |
| sdake | my outlook 2016 doesn't break the blue bar in html posts | 14:59 |
| rhallisey | sean-k-mooney, I wasn't sure if they are still there | 14:59 |
| sean-k-mooney | rhallisey: yes im pretty sure i implmented it in he kolla-host playbook too | 14:59 |
| sdake | sean-k-mooney I think perhaps you changed your outlook to post in non-html | 14:59 |
| sdake | sean-k-mooney I can't find a proper workaround via google | 14:59 |
| sdake | sean-k-mooney and microsoft has known about the problem for going on 2 years and not fixed it | 14:59 |
| sean-k-mooney | sdake: yes on my laptop not on my desktop | 14:59 |
| rhallisey | zioproto, ^ | 14:59 |
| sean-k-mooney | mailly becasue i was lazy | 15:00 |
| *** williamcaban has joined #openstack-kolla | 15:02 | |
| sean-k-mooney | sdake: ill take a few screen shots of my settings but yes i set it to add the > and also add a [Mooney,Sean K] tag automatically | 15:03 |
| sean-k-mooney | i have outlook set to default to html for new mails but for replives it uses the same format as the message being replied too so rich/plain text for the mailing list | 15:04 |
| *** sayantani01 has joined #openstack-kolla | 15:07 | |
| *** TxGirlGeek has quit IRC | 15:08 | |
| *** tonanhngo has quit IRC | 15:08 | |
| *** TxGirlGeek has joined #openstack-kolla | 15:09 | |
| *** tonanhngo has joined #openstack-kolla | 15:09 | |
| *** hfu has quit IRC | 15:11 | |
| *** williamcaban has quit IRC | 15:11 | |
| *** lrensing has joined #openstack-kolla | 15:11 | |
| *** TxGirlGeek has quit IRC | 15:13 | |
| *** tonanhngo has quit IRC | 15:13 | |
| openstackgerrit | Saverio Proto proposed openstack/kolla-ansible: Use become: True when pulling docker images https://review.openstack.org/425215 | 15:15 |
| *** saneax is now known as saneax-_-|AFK | 15:15 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: Refactor register.yml files https://review.openstack.org/416046 | 15:17 |
| openstackgerrit | Saverio Proto proposed openstack/kolla-ansible: Use become: True when pulling docker images https://review.openstack.org/425215 | 15:17 |
| *** zioproto has left #openstack-kolla | 15:17 | |
| *** zioproto has joined #openstack-kolla | 15:17 | |
| zioproto | So I should add the ubuntu user to the docker group ? | 15:19 |
| zioproto | this task should be done in the bootstrap-servers playbook ? we should add the ubuntu user to the docker group ? | 15:19 |
| *** lamt has joined #openstack-kolla | 15:19 | |
| *** Mech422 has quit IRC | 15:21 | |
| *** zhubingbing has quit IRC | 15:25 | |
| *** williamcaban has joined #openstack-kolla | 15:27 | |
| *** zhubingbing has joined #openstack-kolla | 15:27 | |
| *** TxGirlGeek has joined #openstack-kolla | 15:28 | |
| openstackgerrit | Merged openstack/kolla-ansible: Optimize reconfiguration for ceilometer https://review.openstack.org/423606 | 15:30 |
| openstackgerrit | Merged openstack/kolla-ansible: Optimize reconfigure action for gnocchi https://review.openstack.org/423133 | 15:30 |
| *** srwilkers has joined #openstack-kolla | 15:31 | |
| *** srwilkers has quit IRC | 15:31 | |
| *** srwilkers has joined #openstack-kolla | 15:32 | |
| *** SridharP has joined #openstack-kolla | 15:32 | |
| *** williamcaban has quit IRC | 15:34 | |
| berendt | I will join the team meeting a few minutes later. Hanging around on the highway... | 15:35 |
| *** japestinho__ has joined #openstack-kolla | 15:35 | |
| zioproto | the meeting is on this channel ? | 15:36 |
| *** rmart04 has quit IRC | 15:38 | |
| *** japestinho_ has quit IRC | 15:38 | |
| *** TxGirlGeek has quit IRC | 15:41 | |
| zioproto | ok I figured out that using the 'kolla' user you dont need sudo to make docker operations | 15:42 |
| zioproto | but who is actually using this kolla user ? | 15:42 |
| *** egonzalez has joined #openstack-kolla | 15:44 | |
| zioproto | ok I see now that the kolla user ssh keys are generated and put in /etc/kolla | 15:47 |
| *** bjolo has quit IRC | 15:50 | |
| *** SridharP1 has joined #openstack-kolla | 15:50 | |
| zioproto | how can I use kolla_ssh_key when I run the kolla commands ? | 15:50 |
| *** inc0 has joined #openstack-kolla | 15:51 | |
| *** SridharP has quit IRC | 15:52 | |
| inc0 | good morning | 15:52 |
| SridharP1 | Hi | 15:52 |
| *** SridharP1 has left #openstack-kolla | 15:52 | |
| *** SridharP1 has joined #openstack-kolla | 15:53 | |
| srwilkers | morning inc0 | 15:53 |
| egonzalez | Jeffrey4l_: i'm testing gertty, have you ever saw this error "mapping values are not allowed here" | 15:53 |
| *** sdake_ has joined #openstack-kolla | 15:53 | |
| Jeffrey4l_ | egonzalez, no.. | 15:53 |
| inc0 | Jeffrey4l_, did you get my message about gates? | 15:53 |
| Jeffrey4l_ | inc0, yep. it is fixed by https://review.openstack.org/424928 | 15:54 |
| *** msimonin has joined #openstack-kolla | 15:55 | |
| *** sdake has quit IRC | 15:56 | |
| zhubingbing | hey guys | 15:57 |
| inc0 | meeting in 3 | 15:58 |
| *** eaguilar has quit IRC | 15:58 | |
| zhubingbing | kolla-meeting-3？ | 15:58 |
| inc0 | noo...openstack-meeting-4;) in 3 as in 3 minutes | 15:59 |
| inc0 | 1 minute now | 15:59 |
| zhubingbing | ok | 15:59 |
| zhubingbing | ；） | 15:59 |
| *** srwilkers has quit IRC | 16:05 | |
| *** srwilkers has joined #openstack-kolla | 16:05 | |
| openstackgerrit | Saverio Proto proposed openstack/kolla: Use become: True when updating /etc/hosts https://review.openstack.org/425191 | 16:06 |
| *** mdnadeem has quit IRC | 16:13 | |
| *** RichWellum has joined #openstack-kolla | 16:13 | |
| openstackgerrit | Saverio Proto proposed openstack/kolla-ansible: use become: True to set sysctl values https://review.openstack.org/425263 | 16:17 |
| * portdirect stands in the silence, and reflects on the serenity of a empty #kolla, birds start to sing in the distance, and a cool breeze blows. | 16:22 | |
| *** sacharya has joined #openstack-kolla | 16:24 | |
| *** sacharya has quit IRC | 16:24 | |
| *** TxGirlGeek has joined #openstack-kolla | 16:25 | |
| openstackgerrit | Farid Da Encarnacao proposed openstack/kolla-ansible: Fix typo heat.conf https://review.openstack.org/425270 | 16:26 |
| *** williamcaban has joined #openstack-kolla | 16:27 | |
| *** RichWellum has quit IRC | 16:28 | |
| *** williamcaban has quit IRC | 16:33 | |
| *** SridharP has joined #openstack-kolla | 16:33 | |
| *** prameswar has quit IRC | 16:34 | |
| kfox1111 | morning. | 16:34 |
| *** SridharP2 has joined #openstack-kolla | 16:35 | |
| *** SridharP1 has quit IRC | 16:35 | |
| kfox1111 | and.. rebase time.. :/ | 16:35 |
| *** SridharP has quit IRC | 16:38 | |
| * portdirect the silence is broken, a rabbit runs into a bush, clouds appear. somewhere a tractor is working the feilds | 16:40 | |
| *** williamcaban has joined #openstack-kolla | 16:43 | |
| *** nrado has joined #openstack-kolla | 16:44 | |
| *** zioproto has quit IRC | 16:45 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Local registry in gate for trunk builds https://review.openstack.org/418550 | 16:46 |
| *** mdorman has joined #openstack-kolla | 16:49 | |
| mdorman | does the meeting start in 10 minutes, or did it start 50 minutes ago? | 16:50 |
| *** retr0h has joined #openstack-kolla | 16:50 | |
| openstackgerrit | Merged openstack/kolla-ansible: Fix cleanup-containers script https://review.openstack.org/420775 | 16:53 |
| egonzalez | mdorman: 50 ago | 16:53 |
| openstackgerrit | Merged openstack/kolla-ansible: Optimize reconfigure action for nova https://review.openstack.org/407416 | 16:54 |
| *** zhubingbing has quit IRC | 16:55 | |
| mdorman | thanks | 16:56 |
| gema | inc0: I am also adamant about gates | 16:59 |
| *** athomas has quit IRC | 16:59 | |
| gema | that's why the 3rd party CI is high on my list so our images can be tested | 16:59 |
| gema | and show up in the voting | 17:00 |
| duonghq | inc0, egonzalez I'll be back in about 8hr, and available for next | 17:00 |
| inc0 | cool | 17:00 |
| inc0 | thanks duonghq | 17:00 |
| inc0 | gema, I think we should write regular gatejobs for infra too | 17:00 |
| *** TxGirlGeek has quit IRC | 17:00 | |
| Jeffrey4l_ | sean-k-mooney, re qemu, no. in the gate, we are already in kvm. it will be very slow to use nested qemu. | 17:00 |
| gema | inc0: we don't have ARM64 on those gates yet | 17:01 |
| inc0 | sean-k-mooney, we don't have control over how vms in infra are spawned | 17:01 |
| duonghq | next 6-7hrs, and will "disappear" for couple of days due to "Tet holiday" in Vietnam (Lunar new year) | 17:01 |
| gema | also on my TODO but that is a bigger undertaking | 17:01 |
| *** TxGirlGeek has joined #openstack-kolla | 17:01 | |
| duonghq | night guys | 17:01 |
| SridharP2 | hello | 17:01 |
| Jeffrey4l_ | aha, duonghq me too. :) | 17:01 |
| *** jpstnh has joined #openstack-kolla | 17:01 | |
| SridharP2 | I am trying setup the kolla environment using QSG | 17:01 |
| gema | berendt, inc0: thanks for your help | 17:02 |
| inc0 | gema, so what we could do is you could write script like https://github.com/openstack/kolla/blob/master/tools/setup_gate.sh | 17:02 |
| duonghq | Jeffrey4l_, nice | 17:02 |
| inc0 | for arm | 17:02 |
| *** duonghq has quit IRC | 17:02 | |
| *** Serlex has left #openstack-kolla | 17:02 | |
| sdake_ | hey peeps | 17:02 |
| inc0 | or no, that doesn't make any sense | 17:02 |
| inc0 | gema, we'll figure it out | 17:02 |
| SridharP2 | It gets strucked @TASK [kube-join : join workers to the initialized kubernetes cluster] ********** | 17:02 |
| SridharP2 | skipping: [kube1] | 17:02 |
| gema | inc0: ack, thanks | 17:02 |
| inc0 | gmorning sdake | 17:02 |
| sdake_ | bmace_ i've nearly got the deployment script working (with help from friends) | 17:02 |
| inc0 | ok afk, moving to office | 17:03 |
| sdake_ | bmace_ icsid resets | 17:03 |
| gema | night | 17:03 |
| *** inc0 has quit IRC | 17:03 | |
| sdake_ | bmace_ rather enters a backoff crsh | 17:03 |
| sean-k-mooney | Jeffrey4l_: i was suggesting useing the image properties to request the kvm on the host to emulate an arm cpu | 17:03 |
| sean-k-mooney | Jeffrey4l_: well i was thinking that i did not say it till now | 17:03 |
| gema | sean-k-mooney: I haven't tried that, but it sounds slow :D | 17:04 |
| *** japestinho__ has quit IRC | 17:04 | |
| gema | but it may work for testing, will think about it | 17:04 |
| sean-k-mooney | gema: well its not insanly slow but if it was supported it would mean your thridparty ci could be first party | 17:05 |
| sean-k-mooney | but it would require a custom image from infra with the image properties set and they may not be able to provide that | 17:05 |
| *** Jeffrey4l_ has quit IRC | 17:06 | |
| *** sp___ has joined #openstack-kolla | 17:07 | |
| *** tonanhngo has joined #openstack-kolla | 17:07 | |
| sbezverk | SridharP2: are you trying to use kubeadm for bringing up your cluster? | 17:10 |
| *** jemcevoy has quit IRC | 17:11 | |
| portdirect | SridharP2: The issue you are having (if using halcyon) is that the docs refer to a k8s version that we cannot (yet) provide a controller manager image that supports ceph | 17:11 |
| portdirect | if you run the setup script with v1.5.1 instead of 1.5.2 and recreate the cluster you should be good to go | 17:12 |
| SridharP2 | Quit | 17:12 |
| *** SridharP2 has left #openstack-kolla | 17:12 | |
| *** newmember has joined #openstack-kolla | 17:13 | |
| sean-k-mooney | portdirect: is there a plan to provide a production way of deploying k8s for pike. be that in kolla-k8s or a recommedation that is gated on? | 17:13 |
| berendt | gema: you are welcome :-) | 17:14 |
| portdirect | sean-k-mooney: not that I'm aware of, deploying k8s for production is quite an involved activity and one I'd personally prefer to keep out of scope for kolla-k8s | 17:15 |
| sean-k-mooney | it would be nice to have a tested recomendation for someone why is new to k8 that wants to use kolla-k8 to stand up openstack | 17:15 |
| portdirect | though we will need to address quite a few issues on that front - especially regarding the kubernetes-entrypoint container and the distribution of access tokens | 17:16 |
| sean-k-mooney | personally there are several options discribed for dev but im not sure how to take the next step to production with kolla-k8 | 17:16 |
| *** sp__ has joined #openstack-kolla | 17:16 | |
| * sean-k-mooney admitly im not sure how to take the first step to getting a dev setup with kolla-k8s working yet either | 17:17 | |
| portdirect | sean-k-mooney: are you familiar with k8s? | 17:17 |
| sean-k-mooney | portdirect: not partically | 17:17 |
| * portdirect laughs and crys at the same time | 17:17 | |
| sean-k-mooney | i have been reading up on it in my spare time but have not spent enough time with it to be comfortable | 17:18 |
| portdirect | cool - in many ways you can view it (and I'll get shot for saying this) as a mini openstack | 17:18 |
| sean-k-mooney | portdirect: well i am aware of what is should do and some of its concepts | 17:18 |
| portdirect | and so when it comes to a production cluster you have a similar number of viarbles to consider | 17:18 |
| sean-k-mooney | i just have no hands on experince using it really yet | 17:18 |
| *** williamcaban has quit IRC | 17:19 | |
| portdirect | we should definalty document some of them, ie PKI for api servers and etcd, and CNI networking | 17:19 |
| sean-k-mooney | hehe ya i have noticed that k8 is perhaps the only tech stack i have found with = or more installers then openstack | 17:19 |
| *** sp___ has quit IRC | 17:19 | |
| portdirect | as well as ABAC for the api - which no openstack-on-k8s project has addressed to date | 17:19 |
| sean-k-mooney | ABAC account based acess control? | 17:20 |
| sean-k-mooney | atribute? | 17:20 |
| portdirect | attribute | 17:20 |
| *** srwilkers has quit IRC | 17:20 | |
| sean-k-mooney | ah ok | 17:20 |
| portdirect | so you can understand why I'm a bit hesitant to go down that rabbit hole - your point above was right on the money, considering its age i think k8s has more installers than openstack | 17:21 |
| *** l4yerffeJ_ has quit IRC | 17:21 | |
| *** l4yerffeJ_ has joined #openstack-kolla | 17:21 | |
| portdirect | and that can turn into a higly opinionated area | 17:21 |
| sean-k-mooney | it would be good to have a guide or a blog that explained how to transition form kolla-ansible to kolla-k8s. e.g. if you are already comfortable with kolla-ansibel what do you need to know about kolla-k8s to get started with it | 17:22 |
| portdirect | I personally have used both ansible and puppet (via foreman) for production clusters | 17:22 |
| sean-k-mooney | however i am aware that kolla-k8s is changing alot with helm and charts and operators currently | 17:23 |
| *** jheroux has joined #openstack-kolla | 17:24 | |
| portdirect | I'd need to defer to someone more familiar with kolla-ansible on that front - I would like to see us move away from any dependency on kolla-ansible and so by that stage i think we will have two totally separate deliverables - and moving seamlessly from one to the other would be as involved as moving from say OSA to kolla-ansible | 17:24 |
| *** Jeffrey4l has joined #openstack-kolla | 17:25 | |
| *** TxGirlGeek has quit IRC | 17:25 | |
| *** rushil has joined #openstack-kolla | 17:25 | |
| sean-k-mooney | oh yes i understand. i was thinking for developers as i would like to get to the point where i can contribute to feature in both | 17:26 |
| sean-k-mooney | e.g. i add a contaienr to kolla. i would like to know how to extend kolla-k8s the same way i would add an ansibel role. | 17:26 |
| sean-k-mooney | i just dont know enough about how kolla-k8s works with k8s to make that conenction yet | 17:27 |
| *** TxGirlGeek has joined #openstack-kolla | 17:27 | |
| bmace_ | sdake: that is great about iscsid. i just found out what was wrong with my k8s cluster access that was failing so i am going to try to run again now and see how it goes. i'm still working off a branch after my genconfig fixes rather than head since the fluentd changes seemed to have broken genconfig yesterday (at least from this centos container environment) | 17:27 |
| portdirect | oh - yeah - that makes sense - and i think that the work sdake_'s doing is the first step in that direction | 17:27 |
| sdake_ | bmace_ - i'll push an update as soon as possible - ther are many more problems in the script | 17:29 |
| sdake_ | bmace_ i have a 30 minute meeting - then need to schlep the changes from a dev machine to my git repo | 17:29 |
| *** Jeffrey4l has quit IRC | 17:30 | |
| openstackgerrit | Merged openstack/kolla-kubernetes: Typo fix: folowing => following https://review.openstack.org/425008 | 17:30 |
| *** msimonin has quit IRC | 17:30 | |
| *** sp__ has quit IRC | 17:30 | |
| portdirect | sdake_: are you running 1.5.2 on your machines? | 17:31 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible: Implement Keystone zero-downtime upgrade https://review.openstack.org/405215 | 17:31 |
| sdake_ | portdirect think so | 17:31 |
| portdirect | sdake_: att-comdev have not published a ceph enable controller manager image for that version which is whats caused a few people to get stuck during deployment | 17:32 |
| portdirect | any chance you could run kubectl version just to double check? | 17:32 |
| sbezverk | portdirect: is your setuo script uses dynamic pvc? | 17:34 |
| sdake_ | yup | 17:34 |
| sdake_ | portdirect server is 1.5.2, client is 1.5.1 | 17:34 |
| portdirect | sbezverk: no but it pulls in the image required to enable that | 17:34 |
| sbezverk | portdirect: for a bit I wa running 1.5.2 with 1.5.1 att controller manager | 17:35 |
| sbezverk | have not noticed any issues | 17:35 |
| portdirect | sbezverk: yes - but the setup sets the version to be the same | 17:35 |
| sbezverk | so no really need to rollback to 1.5.1 for everything | 17:36 |
| portdirect | so doint it manually is fine - but is not the process we have documented | 17:36 |
| sbezverk | portdirect: sure sure | 17:36 |
| portdirect | sbezverk: https://github.com/att-comdev/halcyon-kubernetes/blob/master/kube-deploy/roles/kube-prep/tasks/patch-kube.yml#L18 | 17:38 |
| portdirect | without a pullable version of that image the playbook will hang - as has been reported by three people on this channel today | 17:38 |
| *** jmontleon has quit IRC | 17:39 | |
| mgiles | kfox1111 sbezverk portdirect When you have a chance, can you review nova-control service: https://review.openstack.org/#/c/421457 | 17:39 |
| mgiles | No hurry, but it's been sitting almost a week with no changes other than rebases and I wouldn't mind getting it closed out | 17:39 |
| sbezverk | mgiles: man this is 3rd time I +2 you ;-) | 17:40 |
| mgiles | I know - I don't think anybody else has looked at it to give it the second +2 | 17:40 |
| sbezverk | mgiles: please twist somebody's arm to workflow it | 17:40 |
| mgiles | Thanks | 17:40 |
| portdirect | mgiles: even has reno | 17:40 |
| * portdirect seraches for reason to be vindictive | 17:40 | |
| *** jmontleon has joined #openstack-kolla | 17:41 | |
| mgiles | real nice portdirect | 17:41 |
| portdirect | mgiles: i came up blank :( - nice work :) | 17:42 |
| mgiles | thanks portdirect. I'll try to give you a reason next time. :) | 17:43 |
| *** rushil has quit IRC | 17:45 | |
| *** RichWellum has joined #openstack-kolla | 17:47 | |
| *** jmontleon has quit IRC | 17:49 | |
| portdirect | kfox1111/sbezverk : is this still required https://blueprints.launchpad.net/kolla/+spec/api-interface-bind-address-override | 17:50 |
| kfox1111 | back | 17:53 |
| *** tonanhngo has quit IRC | 17:53 | |
| kfox1111 | portdirect: yes/no? | 17:53 |
| kfox1111 | portdirect: all of the existing services are complete. I'd expect other non tested services not to work though. | 17:54 |
| kfox1111 | portdirect: but it may not matter depending on what we do with confimap generation going forward. | 17:54 |
| *** jmontleon has joined #openstack-kolla | 17:54 | |
| kfox1111 | so, its either ongoing, or complete. | 17:54 |
| portdirect | so wa can mark it as complete? inc0 wanted me to get an answer | 17:56 |
| * kfox1111 shrugs | 17:57 | |
| *** simon-AS559 has quit IRC | 17:57 | |
| kfox1111 | if we get to helm generated configmaps before we add any more services, yes. | 17:57 |
| kfox1111 | if someone wants more services before that, maybe not. | 17:58 |
| *** klindgren__ has joined #openstack-kolla | 17:58 | |
| kfox1111 | theres a third option, to split the bp and close it for ocata, and make anew one for pike | 17:58 |
| portdirect | right lets close it then - i don't think we should add any more services until we can actually run stuff :) | 17:58 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Nova-control service deployment https://review.openstack.org/421457 | 17:58 |
| kfox1111 | mgiles: theres kubernetes_entrypoint: true set on a lot of things explicitly. this makes it hard to turn on/off as a set. | 17:59 |
| kfox1111 | Though I know other ps's still have that in there too. | 17:59 |
| mgiles | kfox1111 Do we have an alternative? | 18:00 |
| kfox1111 | could you make another ps that removes the lower layer kubernetes_entrypoint: true stuff from all the helm/services/*/values.yaml? | 18:00 |
| mgiles | And specify it globally? | 18:01 |
| kfox1111 | mgiles: a global.kolla.all.kubernetes_entrypoint=true is enough to cover them all. | 18:01 |
| *** klindgren_ has quit IRC | 18:01 | |
| kfox1111 | yeah. | 18:01 |
| *** gfidente is now known as gfidente|afk | 18:01 | |
| mgiles | That means it will get run even for containers that don't need entrypoint? | 18:01 |
| mgiles | like the svcs | 18:01 |
| kfox1111 | false should be needed only if you really dont want it. | 18:01 |
| sbezverk | portdirect: couple of services are not done | 18:01 |
| kfox1111 | svc's don't look at the var. | 18:01 |
| mgiles | ah ok | 18:02 |
| mgiles | makes sense | 18:02 |
| kfox1111 | portdirect: "actually running stuff" means on bare metal? | 18:02 |
| kfox1111 | that should happen in the next week or two. | 18:02 |
| *** egonzalez has quit IRC | 18:02 | |
| kfox1111 | I think we're close. | 18:02 |
| kfox1111 | so adding more services may come realatively soon. | 18:02 |
| *** TxGirlGeek has quit IRC | 18:03 | |
| sbezverk | kfox1111: what about the horizon check fix? did it help? | 18:04 |
| *** TxGirlGeek has joined #openstack-kolla | 18:04 | |
| sbezverk | kfox1111: it seems horizon service ps is still hanging inprogress state | 18:05 |
| sbezverk | kfox1111: nvm | 18:06 |
| kfox1111 | there are a few gate fixes still in review. :/ | 18:07 |
| kfox1111 | and I'm trying to get a cache going to help with the network/hub flakyness. | 18:08 |
| kfox1111 | between the two, I'm hoping to get it stable again. | 18:08 |
| sbezverk | kfox1111: neutron in services bp shows as todo, but I think you completed it, right? | 18:09 |
| openstackgerrit | sayantani proposed openstack/kolla: [WIP] Update Multinode deployment of kolla guide https://review.openstack.org/408364 | 18:09 |
| *** TxGirlGeek has quit IRC | 18:10 | |
| *** matrohon__ has quit IRC | 18:10 | |
| bmace_ | sdake_: think i made it to your iscsid error. you said you had a fix for it going up in not too long? | 18:10 |
| *** TxGirlGeek has joined #openstack-kolla | 18:12 | |
| sbezverk | bmace_: I worked a little bit with iscsi, could you share what exactly the problem with it? | 18:13 |
| sayantani01 | Hi guys, can I have some eyes on https://review.openstack.org/#/c/424271/ ? :) | 18:14 |
| bmace_ | sbezverk: i'm just running the dev_workflow.sh script near the end of the containerized dev environment setup and it is blowing up with: Unable to locate iscsid directory in search_paths=[/etc/kolla, /etc/kolla-kubernetes, /opt/kolla/etc/kolla, /opt/kolla-kubernetes/etc/kolla-kubernetes] | 18:16 |
| bmace_ | sdake was hitting iscsi issues also and said he had something working | 18:16 |
| bmace_ | i haven't dug too much into the script or why it might be failing yet. if it isn't the same issue he was hitting I will. | 18:16 |
| openstackgerrit | Mark Giles proposed openstack/kolla-kubernetes: Improve handling of kubernetes_entrypoint variable in nova services https://review.openstack.org/425324 | 18:17 |
| openstackgerrit | sayantani proposed openstack/kolla-ansible: Fix RabbitMQ server error in bifrost https://review.openstack.org/424384 | 18:17 |
| sdake_ | i am grabbingbmyup - working on it | 18:17 |
| sbezverk | bmace_: ok got it, so it is not real iscsi issue :-), it looks more like a packaging issue | 18:17 |
| sdake_ | bmace_ yup working on it | 18:17 |
| sdake_ | bmace_ i have several fixes | 18:17 |
| sdake_ | bmace_ you can diff the env and kee pthe ones you want or just pull a fresh patch | 18:17 |
| sdake_ | note iscsi-tgtd now fails, but everything deploys propery | 18:18 |
| sdake_ | iscsi-tgdt enters a backoff crash loop | 18:18 |
| sdake_ | sbezverk ^^ so that may be a legitimate issue | 18:18 |
| sdake_ | sbezverk i can provide you access if you like to take a quick peek | 18:18 |
| sdake_ | sbezverk any diagnostics you could provide would be helpful - even if we are doing it wrong just stating that owuld be helfpul :) | 18:19 |
| *** unicell has joined #openstack-kolla | 18:20 | |
| sbezverk | sdake_: most likley you hit a known bug, when host has iscsi-initiator-utils already installed, in the container iscsd fails to start | 18:20 |
| sdake_ | sbezverk i am not running in a container | 18:20 |
| sbezverk | remvoing iscsi-utils from the host will resolve it | 18:20 |
| sdake_ | sbezverk i am running on bare metal | 18:20 |
| sbezverk | sdake_: then what goes into a crash loop? | 18:21 |
| sdake_ | sbezverk i am sorting out workflow for real ops, not a dev workflow | 18:21 |
| sdake_ | sbezverk its best if yu look at the machine i think :) | 18:21 |
| sbezverk | ok, how I can access it? | 18:21 |
| sdake_ | i'll sen you a pm with creds | 18:21 |
| sdake_ | http://paste.fedoraproject.org/536506/14853686/ -> http://paste.fedoraproject.org/536506/14853686 | 18:25 |
| *** rushil has joined #openstack-kolla | 18:25 | |
| *** inc0 has joined #openstack-kolla | 18:29 | |
| *** sayantani01 has quit IRC | 18:30 | |
| kfox1111 | sbezverk: yeah. | 18:31 |
| sbezverk | kfox1111: so we can change it to done right? | 18:33 |
| *** sambetts is now known as sambetts|afk | 18:34 | |
| *** inc0 has quit IRC | 18:34 | |
| sdake_ | i'll be pushing a patch in short order | 18:35 |
| *** inc0 has joined #openstack-kolla | 18:35 | |
| sdake_ | openstack endpoint list seems busted | 18:35 |
| sdake_ | bmace_ to get past the iscsi problem uninstall the iscsi initiator | 18:35 |
| sdake_ | bmace_ on whatever node is running iscsi | 18:35 |
| sdake_ | iscsi-inititaor is a default packgae in centos | 18:35 |
| sbezverk | sdake_: not really, it gets installed only by command | 18:36 |
| sdake_ | sbezverk ok - ill check int o that | 18:36 |
| sdake_ | sbezverk what about this openstack endpoint list freezing up? | 18:36 |
| sdake_ | sbezverk shouldn't that owrk? | 18:36 |
| *** jmccarthy has joined #openstack-kolla | 18:36 | |
| sbezverk | sdake_: most likely something is not right with services | 18:36 |
| bmace_ | sdake_: that is to fix the iscsi-tgtd issue you are hitting right, not the earlier issue that you have changes for? | 18:37 |
| *** sdake_ has quit IRC | 18:39 | |
| *** sdake has joined #openstack-kolla | 18:40 | |
| *** krtaylor has quit IRC | 18:40 | |
| *** Jeffrey4l has joined #openstack-kolla | 18:41 | |
| sdake | this is like reliving a bad nightmare of kolla-ansible startup :) | 18:42 |
| bmace_ | lol | 18:43 |
| kfox1111 | sbezverk: yeah. | 18:44 |
| *** williamcaban has joined #openstack-kolla | 18:44 | |
| kfox1111 | sdake: does catalog list work? | 18:44 |
| bmace_ | sdake: using the halycon-vagrant stuff i don't have iscsi-utils on those nodes so i might be ok once i get past the earlier "unable to locate" iscsid error? | 18:44 |
| kfox1111 | sdake: I found some of the keystone admin stuff doesn't work outside of k8s as part of the local repo patchset. I fix it there. | 18:45 |
| sdake | kfox1111 sbezverk is debuggin with me | 18:45 |
| kfox1111 | sdake: openstack catalog list should work though. if not, its keystone nto working. | 18:45 |
| *** Jeffrey4l has quit IRC | 18:46 | |
| *** rwellum has joined #openstack-kolla | 18:46 | |
| sdake | kfox1111 ya - OS_AUTH_CONFIG is not correct at present - we are on it :) | 18:46 |
| bmace_ | sdake: did you upload your changes? i see some from this morning around setup_dev_env and dev_workflow but the dev_workflow.sh change is size 0. | 18:46 |
| *** sean-k-mooney has quit IRC | 18:47 | |
| kfox1111 | sdake: ok. cool. | 18:47 |
| *** sean-k-mooney has joined #openstack-kolla | 18:48 | |
| sdake | bmace_ i've got a lunch appointment and i will upload it after lucnh - after i debug this stuff a bit more | 18:48 |
| kfox1111 | sdake: just wanted to mention the admin related keystone stuff not working outside of stuff running inside k8s in case you trip over that. | 18:48 |
| sdake | bmace_ i have a hard appointment in 20 mins so dont' have time to update the patc hyet | 18:48 |
| bmace_ | sdake: kk | 18:48 |
| sdake | bmace_ after i get back from lucnh its first order of buviness | 18:49 |
| sdake | the 0 byte change was to fix the problem with a script not being executable | 18:49 |
| bmace_ | sdake: sounds great. ah, yup, i fixed the same in my container. maybe we can get this all working today! nice start anyway. lots of improvements we can make to it. | 18:50 |
| sdake | bmace_ it will be working today i am sure of i t | 18:50 |
| bmace_ | i had to make a lot of local changes for proxy stuff which needs to be supported in a cleaner way in general. | 18:50 |
| bmace_ | i suppose it can be just via build / run args also, so could even be just a doc thing | 18:51 |
| *** rushil has quit IRC | 18:51 | |
| *** tonanhngo has joined #openstack-kolla | 18:53 | |
| *** pcaruana has quit IRC | 18:53 | |
| *** britthouser has joined #openstack-kolla | 18:54 | |
| *** inc0 has quit IRC | 18:54 | |
| kfox1111 | sbezverk: fyi, https://review.openstack.org/#/c/425331/ | 18:55 |
| *** inc0 has joined #openstack-kolla | 18:56 | |
| sbezverk | kfox1111: what is it for? never had to deal with tha before | 18:57 |
| *** tonanhngo has quit IRC | 18:57 | |
| sdake | bmace_ - ok dude i think we are in good shape | 18:58 |
| sdake | bmace_ but i really gotta jet - or going to be late | 18:58 |
| sdake | bmace_ i have to tear down my env since i didn't setup kubeadm properly apparently | 18:58 |
| sdake | bmace_ however i'll push my changes up in a new review and you can use what you want | 18:58 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla: Create deliverables.yaml https://review.openstack.org/424784 | 18:58 |
| bmace_ | sdake: kk, fantastic, thanks | 18:58 |
| *** shardy is now known as shardy_afk | 19:01 | |
| *** williamcaban has quit IRC | 19:01 | |
| *** rushil has joined #openstack-kolla | 19:04 | |
| britthouser | Sorry I missed meeting today. My slack-irc server seems to have died. | 19:04 |
| openstackgerrit | Mark Giles proposed openstack/kolla-kubernetes: Improve handling of kubernetes_entrypoint variable https://review.openstack.org/425350 | 19:04 |
| *** esmiurium has quit IRC | 19:05 | |
| mgiles | kfox1111 fyi, created another patch set to clean up the same issue with the entrypoint variable in the other services. So now it should be consistent throughout | 19:05 |
| kfox1111 | cool. thanks. | 19:06 |
| kfox1111 | sbezverk: going to merge that with the work to gather docker container lists, so I can push them to tarballs.o.o and pull them back into the gate. | 19:07 |
| kfox1111 | sbezverk: that way, we will see less breakages due to network connectivity and hub issues. | 19:07 |
| *** MasterOfBugs has joined #openstack-kolla | 19:08 | |
| *** MasterOfBugs has quit IRC | 19:08 | |
| *** RichWellum has quit IRC | 19:09 | |
| *** MasterOfBugs has joined #openstack-kolla | 19:09 | |
| *** pramodrj07 has joined #openstack-kolla | 19:09 | |
| *** Jeffrey4l has joined #openstack-kolla | 19:10 | |
| *** britthouser7 has quit IRC | 19:10 | |
| openstackgerrit | Merged openstack/kolla-ansible: Fix typo heat.conf https://review.openstack.org/425270 | 19:11 |
| *** esmiurium has joined #openstack-kolla | 19:12 | |
| sbezverk | kfox1111: so docker hub's hip hops will not affect us :) I like it!! | 19:13 |
| kfox1111 | and the same for google's container registry. and I think there was a third one in the list too.... | 19:14 |
| *** jmccarthy has quit IRC | 19:15 | |
| *** Jeffrey4l has quit IRC | 19:15 | |
| kfox1111 | http://logs.openstack.org/21/424921/4/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-nv/1636f33/logs/docker_images.txt | 19:15 |
| kfox1111 | yeah. quay too. :/ | 19:15 |
| kfox1111 | docker.io for calico. quay for calico. gcr for the rest of k8s, and docker.io for kolla. | 19:15 |
| *** britthouser has quit IRC | 19:16 | |
| kfox1111 | so only the periodic jobs will pull from there, unless something changes between a periodic and a gate run. | 19:16 |
| *** britthouser has joined #openstack-kolla | 19:16 | |
| kfox1111 | so, the gate shouldn't break in that case. will just switch to remove for those containers. | 19:17 |
| kfox1111 | and then the next periodic run will fix it. | 19:17 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla: Implement Keystone zero-downtime upgrade https://review.openstack.org/388544 | 19:17 |
| *** williamcaban has joined #openstack-kolla | 19:18 | |
| *** krtaylor has joined #openstack-kolla | 19:19 | |
| *** williamcaban has quit IRC | 19:22 | |
| *** msimonin has joined #openstack-kolla | 19:28 | |
| nrado | can someone explain to me why I got -1 from jenkins https://review.openstack.org/#/c/422566/ ? How can i check it? | 19:29 |
| *** mgoddard has joined #openstack-kolla | 19:29 | |
| qwang | nrado: it's weird. try to recheck and see how it goes | 19:41 |
| kfox1111 | and... another case of vswitchd out of order... | 19:41 |
| kfox1111 | we really need a fix for that.. | 19:41 |
| *** sayantani01 has joined #openstack-kolla | 19:43 | |
| kfox1111 | sbezverk: uh, heres one that loks like a vswitchd issue, | 19:45 |
| sbezverk | kfox1111: do you mean neutron ovs agent getting ahaed of vswitchd? | 19:45 |
| kfox1111 | but is with helm entrypoint: http://logs.openstack.org/24/425324/1/check/gate-kolla-kubernetes-deploy-centos-binary-2-helm-entrypoint-nv/7464604/console.html | 19:45 |
| *** britthouser has quit IRC | 19:45 | |
| kfox1111 | sbezverk: yeah. | 19:45 |
| *** britthouser has joined #openstack-kolla | 19:45 | |
| kfox1111 | so for some reason, that race is hitting us a lot now. | 19:46 |
| kfox1111 | we need to figure out a fix. :/ | 19:46 |
| kfox1111 | the problem is ovs isn't responsive yet, right? | 19:46 |
| sbezverk | kfox1111: I can try to workaround it by loading module in both containers ovsdb and vswitchd butthis is not a good solution | 19:46 |
| kfox1111 | before neutron-openvswitch-agent starts, I mean. | 19:46 |
| *** williamcaban has joined #openstack-kolla | 19:47 | |
| kfox1111 | what if we added an init container to neutron-openvswitch-agent that blocked until it could do an appropriate openvswitch command? | 19:47 |
| sbezverk | kfox1111: no vswitch has not loaded ovs kernel module yet | 19:47 |
| kfox1111 | ovs-vsctl show or something. | 19:47 |
| sbezverk | kfox1111: lsmod | grep ovs | 19:47 |
| *** rushil has quit IRC | 19:48 | |
| *** srwilkers has joined #openstack-kolla | 19:48 | |
| kfox1111 | are you sure its actually the kernel module its loking for? | 19:48 |
| *** williamcaban has quit IRC | 19:48 | |
| kfox1111 | I would expect openvswitch-agentn not to talk to the kernel directly, | 19:48 |
| kfox1111 | but to vswitchd. | 19:48 |
| *** rushil has joined #openstack-kolla | 19:48 | |
| sbezverk | kfox1111: right it does not | 19:49 |
| kfox1111 | k. | 19:49 |
| sbezverk | kfox1111: but if the module is not loaded, then when it tries to get br-int datapath id, it comes back blank | 19:49 |
| sbezverk | kfox1111: I do not know exactly which system call it uses to get that info | 19:50 |
| kfox1111 | can that happen before ovs-vsctl show works? | 19:50 |
| kfox1111 | I think its talking through the unix socket to the db? | 19:50 |
| sbezverk | kfox1111: yes because for that ovsdb has already socket up and running | 19:50 |
| kfox1111 | but the db's blocking waiting to talk to vswitchd and then neutron times out the connection? | 19:51 |
| kfox1111 | this is going to be a hard one for me to test at the moment. | 19:51 |
| sbezverk | kfox1111: hard to confirm, do not know how neutron ovs agent requests bridge's dataplane id | 19:51 |
| sbezverk | kfox1111: easy solution at least for now | 19:52 |
| sbezverk | moving openvswitch at the top of workflow | 19:52 |
| kfox1111 | oh... just put in a barior... | 19:52 |
| sbezverk | it will give sufficient time for ovs to wake up | 19:52 |
| kfox1111 | yeah. that would fix the gate. wouldn't fix the reboot issue. | 19:52 |
| sbezverk | kfox1111: we can use lsmod | grep openvswitch as a parameter for barrier | 19:53 |
| *** williamcaban has joined #openstack-kolla | 19:53 | |
| kfox1111 | yeah, but not sure that will actually work or not, or if vswithc has to be up too. | 19:53 |
| kfox1111 | theres a slight race there. | 19:53 |
| kfox1111 | it would help for sure. | 19:53 |
| kfox1111 | or, mayube we just do both. | 19:54 |
| kfox1111 | check for kernel module. check for ovs-vsctl show working. | 19:54 |
| kfox1111 | if both work, we're good to go either way. | 19:54 |
| sbezverk | kfox1111: we do not have ovs bits installed on the host | 19:54 |
| kfox1111 | init-container. | 19:54 |
| sbezverk | so running ovs commands from host is problematic | 19:54 |
| sbezverk | ah in init container | 19:54 |
| sbezverk | ok then.. | 19:54 |
| kfox1111 | yeah. we just block neutron-openvswitch-agent entirely until ready. | 19:55 |
| kfox1111 | I think I can take a quick stab at it... | 19:55 |
| sbezverk | kfox1111: sounds good | 19:55 |
| sbezverk | kfox1111: you will need a sample of output when module is not loaded | 19:56 |
| sbezverk | if I rememebr right it generates something, not an error though | 19:56 |
| kfox1111 | lsmod | grep openvswitch | wc -l | 19:56 |
| kfox1111 | should work. | 19:57 |
| kfox1111 | if not 0, somethings loaded? | 19:57 |
| *** Jeffrey4l has joined #openstack-kolla | 19:58 | |
| *** esmiurium has quit IRC | 19:59 | |
| *** zioproto has joined #openstack-kolla | 19:59 | |
| *** williamcaban has quit IRC | 20:00 | |
| openstackgerrit | Merged openstack/kolla-kubernetes: Improve handling of kubernetes_entrypoint variable https://review.openstack.org/425350 | 20:01 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Add dep for neutron-openvswitch-agent on a working openvswitch. https://review.openstack.org/425376 | 20:02 |
| kfox1111 | sbezverk: have a look at that one. | 20:02 |
| kfox1111 | ok. we have a gate change incoming. https://review.openstack.org/#/c/425331/ | 20:03 |
| kfox1111 | shouldn't have any effect, but just a heads up. | 20:03 |
| sbezverk | kfox1111: I think second check is redundant as ovs will always returns something | 20:04 |
| *** williamcaban has joined #openstack-kolla | 20:04 | |
| sbezverk | unless you analyze the exact output | 20:04 |
| kfox1111 | sbezverk: you think the vsctl command only talks to the db, not the actual vswitchd? | 20:04 |
| *** Jeffrey4l has quit IRC | 20:04 | |
| *** williamcaban has quit IRC | 20:04 | |
| nrado | can i undo a recheck | 20:04 |
| nrado | ?? | 20:05 |
| nrado | was in the wrong window… | 20:05 |
| sbezverk | kfox1111: I think so, but not 100% sure | 20:06 |
| sbezverk | kfox1111: I think vswitchd is just needed to execute commands of neutron ovs agent | 20:07 |
| kfox1111 | sbezverk: lets be extra sure then, to stablize the gate. we can always reevaluate later and remove it if we can prove it doesnt add value? | 20:07 |
| sbezverk | but the actual ovs-vs commands talk to ovsdb directly | 20:07 |
| kfox1111 | well, we may have race still then... | 20:07 |
| *** esmiurium has joined #openstack-kolla | 20:07 | |
| kfox1111 | how do we ensure vswitchd is actually up then? | 20:07 |
| sbezverk | kfox1111: sure thing I was not against it just a heads up ;) | 20:08 |
| kfox1111 | yeah. no, its good. we need to figure out all the ways it can break. | 20:08 |
| sbezverk | kfox1111: the most relable way is to use entrypoint | 20:08 |
| kfox1111 | is there a test command nwe can come up with that actually talks to the vswitchd. | 20:08 |
| kfox1111 | we are using nentrypoint. but it fails on this case. :/ | 20:08 |
| sbezverk | kfox1111: but we are not checking on POD | 20:09 |
| sbezverk | kfox1111: or daemonset | 20:09 |
| sbezverk | neutron ovs agent has no dependency confiugred for vswitchd | 20:09 |
| kfox1111 | two isssues. I don't think you can check to see if a pod is running on the same host? | 20:09 |
| kfox1111 | and 2, it makes for much worse scaling. :/ | 20:10 |
| kfox1111 | as its cheep to talk to something on the same server. | 20:10 |
| kfox1111 | its much more expensive for all the things to talk back to the central server. | 20:10 |
| kfox1111 | one other random note. | 20:10 |
| kfox1111 | in the reboot tests, I have seen kubernetes state thigns are ready when they arnt'. | 20:10 |
| kfox1111 | ie, if you pull the rug out from under it, | 20:11 |
| kfox1111 | then start everythign back up. | 20:11 |
| kfox1111 | etcd still has a record of all the pods being "ready", but they arent'. | 20:11 |
| kfox1111 | then it goes and tries to relaunch everything, and updates the status as it goes. | 20:11 |
| kfox1111 | kubernetes-entrypoint may malfunction in this state. not sure. :/ | 20:11 |
| sbezverk | kfox1111: damonset can be queried for a state if opd | 20:11 |
| sbezverk | that is what they need to use for this check | 20:12 |
| kfox1111 | does it check locally only, or does it call back to the server? | 20:12 |
| sbezverk | kfox1111: it cannot check locally as locally no info exist.. I am positive they do it via api server | 20:13 |
| sbezverk | well at least that is how I would approach it, to use single point of truth | 20:13 |
| kfox1111 | yeah. and I really don't like that for scaling/security reasons. :/ | 20:13 |
| kfox1111 | theres got to be a better way for this one use case. | 20:14 |
| kfox1111 | in another project, I just touch a file in one, and look for it in another pod. simple/scalable, if a little hacky looking. | 20:14 |
| sbezverk | kfox1111: what if that file gets removed accidently? | 20:15 |
| sbezverk | or will be left from the previous run? | 20:16 |
| kfox1111 | it woudl only affect restarts on the container. so not too bad, but don't remove a file if needed... | 20:16 |
| kfox1111 | hard reboot case could be handled by placing it in /dev/shm. | 20:16 |
| kfox1111 | it will get whiped after reboot then, so won't exist. | 20:16 |
| sbezverk | I think api should be the central point of info requests | 20:16 |
| kfox1111 | that requires A, extra security granted to containers. adn B, extra load on the api server. | 20:17 |
| kfox1111 | both as an op make me nervious when you might have hundreds of nodes, that run user code. | 20:17 |
| sbezverk | kfox1111: container automatically gets token to request api server so no need any additional | 20:18 |
| sbezverk | info | 20:18 |
| kfox1111 | sbezverk: yeah, a privileged one. and that really scares me. there is a ps in to disable that horible behavior. :/ | 20:19 |
| sbezverk | kfox1111: well then it will break operator | 20:19 |
| kfox1111 | the flag can be used selectively per pod/container. so only enabled where strictly needed. | 20:20 |
| kfox1111 | so won't break operators. | 20:20 |
| sbezverk | as it uses this token to authenticate against api server for its quereis | 20:20 |
| kfox1111 | yup. the idea is to lessen the attack surface to only where needed, not to disable it entirely everywhere. | 20:20 |
| sbezverk | kfox1111: ok if selective flag is available then should be ok | 20:20 |
| kfox1111 | I just object to it being on 1000 pods when 10 pods would be sufficient. :) | 20:21 |
| kfox1111 | I'm really glad they have the token feature though. very important. | 20:21 |
| kfox1111 | still wish nova would figure that out. :/ | 20:22 |
| *** nrado has quit IRC | 20:22 | |
| sbezverk | kfox1111: yep, really usefull, makes life much easier when talking to api from the container | 20:26 |
| kfox1111 | sbezverk: uh... this looks a bit weird: http://logs.openstack.org/50/418550/56/check/gate-kolla-kubernetes-deploy-centos-binary-2-helm-entrypoint-nv/aeed5cb/console.html | 20:28 |
| kfox1111 | weird... | 20:30 |
| kfox1111 | http://logs.openstack.org/50/418550/56/check/gate-kolla-kubernetes-deploy-centos-binary-2-helm-entrypoint-nv/aeed5cb/logs/pods/kolla-cinder-create-keystone-service-9nr89-main.txt | 20:30 |
| kfox1111 | its trying to contact 35357 on keystone-public? | 20:30 |
| sbezverk | hmm it should be 5000 on public | 20:32 |
| kfox1111 | oh... because of this: | 20:32 |
| kfox1111 | {{- $defaultPublicHost := printf "%s-public" $elementName }} | 20:32 |
| kfox1111 | {{- $publicHost := .Values.public_host | default $defaultPublicHost }} | 20:32 |
| kfox1111 | hmm... | 20:32 |
| kfox1111 | a, this isn't using the global api. we missed it somehow... | 20:33 |
| kfox1111 | and b, its registering a k8s internal dns name for external use.... | 20:33 |
| sbezverk | kfox1111: how it was wokring before ?!?!? | 20:33 |
| kfox1111 | no idea.... I'm pretty confused by that. | 20:33 |
| sbezverk | kfox1111: even today I saw sucessfull runs of entrypoint job | 20:34 |
| kfox1111 | maybe we never ran any commands that relooked up the keystone endpoint from the endpoitns list. | 20:34 |
| kfox1111 | its affecting this ps, as I'm trying to make admin endpoint external too. | 20:34 |
| sbezverk | kfox1111: it should be straight forward, you just need to external port for keystone-admin svc | 20:35 |
| kfox1111 | not quite. | 20:36 |
| sbezverk | then it will allocate cluster ip | 20:36 |
| kfox1111 | the entrypoint in keystone has to point to the external address as well. | 20:36 |
| *** Mech422 has joined #openstack-kolla | 20:36 | |
| kfox1111 | thats the problem I'm running into. | 20:36 |
| sbezverk | ok | 20:36 |
| kfox1111 | it was registered as "http://keystone-admin:...." | 20:36 |
| kfox1111 | which doesn't work outside of k8s. | 20:36 |
| sbezverk | ok right our approach does not allow overriding keystone-admin | 20:39 |
| sbezverk | by anything including ip address | 20:39 |
| kfox1111 | I think I've got a ps kergiggered around. | 20:39 |
| kfox1111 | added global api support, and renamed the var a little bit. | 20:39 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Local registry in gate for trunk builds https://review.openstack.org/418550 | 20:42 |
| kfox1111 | sbezverk: the openvswitch folks recommended ovs-appctl version | 20:43 |
| *** rwellum has quit IRC | 20:44 | |
| *** williamcaban has joined #openstack-kolla | 20:45 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Add dep for neutron-openvswitch-agent on a working openvswitch. https://review.openstack.org/425376 | 20:45 |
| *** Jeffrey4l has joined #openstack-kolla | 20:45 | |
| *** saneax-_-|AFK has quit IRC | 20:46 | |
| sbezverk | kfox1111: cool | 20:47 |
| *** williamcaban has quit IRC | 20:49 | |
| *** mgiles has quit IRC | 20:58 | |
| *** TxGirlGeek has quit IRC | 21:02 | |
| *** jtriley has quit IRC | 21:02 | |
| *** shardy_afk has quit IRC | 21:03 | |
| openstackgerrit | sayantani proposed openstack/kolla-ansible: Modify config.yml playbook for bifrost role https://review.openstack.org/419196 | 21:03 |
| *** shardy_afk has joined #openstack-kolla | 21:05 | |
| *** TxGirlGeek has joined #openstack-kolla | 21:05 | |
| *** shardy_afk is now known as shardy | 21:07 | |
| *** zioproto has quit IRC | 21:11 | |
| openstackgerrit | sayantani proposed openstack/kolla: Update Multinode deployment of kolla guide https://review.openstack.org/408364 | 21:16 |
| *** bpattewar has joined #openstack-kolla | 21:16 | |
| openstackgerrit | sayantani proposed openstack/kolla: Update Multinode deployment of kolla guide https://review.openstack.org/408364 | 21:18 |
| *** rushil has quit IRC | 21:19 | |
| *** erlon has quit IRC | 21:20 | |
| *** erlon has joined #openstack-kolla | 21:23 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Capture lists of containers by type and cache containers. https://review.openstack.org/424921 | 21:23 |
| *** sayantani01 has quit IRC | 21:23 | |
| kfox1111 | sbezverk: theres part 2 of 3. :) | 21:24 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Support rolling upgrades of horizon https://review.openstack.org/333996 | 21:32 |
| kfox1111 | inc0: https://review.openstack.org/#/c/422950/ ? :) | 21:32 |
| *** l4yerffeJ has joined #openstack-kolla | 21:34 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 21:34 | |
| *** l4yerffeJ_ has quit IRC | 21:35 | |
| sbezverk | kfox1111: are you sure bzip exists in the base image used for vm? | 21:35 |
| inc0 | there ya go kfox1111 | 21:35 |
| kfox1111 | inc0: thanks. | 21:35 |
| *** Jeffrey4l has quit IRC | 21:35 | |
| kfox1111 | sbezverk: oh. yeah. was going to force it in. but forgot. | 21:35 |
| kfox1111 | guess we'll see if it is or not. :/ | 21:35 |
| *** sayantani01 has joined #openstack-kolla | 21:36 | |
| kfox1111 | interesting... the openvswitch commend is just blocking.... | 21:36 |
| kfox1111 | same behavior as neutron-openvswitch-agent it seems. | 21:37 |
| kfox1111 | maybe just needs a timeout... | 21:37 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Add dep for neutron-openvswitch-agent on a working openvswitch. https://review.openstack.org/425376 | 21:38 |
| sbezverk | kfox1111: does it get stuck on second check? | 21:40 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Improve handling of kubernetes_entrypoint variable in nova services https://review.openstack.org/425324 | 21:40 |
| kfox1111 | sbezverk: yeah. it seemed to. for 3 min. really weird. | 21:40 |
| *** rushil has joined #openstack-kolla | 21:41 | |
| kfox1111 | rarg... | 21:42 |
| sbezverk | kfox1111: one thing could help us is timestapms when each container starts and finishes for the case of init containers.. then it would be easier to see possible racing things.. | 21:42 |
| kfox1111 | those should be in the pod/containername.txt file. | 21:43 |
| kfox1111 | at the bottom. | 21:43 |
| kfox1111 | its a kubectl describe pod <x> | 21:43 |
| kfox1111 | which has the events. | 21:43 |
| sbezverk | ah ok those.. | 21:43 |
| kfox1111 | the one thing I don't like, is its in minutes ago from now. :/ | 21:43 |
| *** shardy has quit IRC | 21:45 | |
| jascott1 | kfox1111 when you get time, what do you think about logical condition format? https://github.com/kubernetes/helm/issues/1837 | 21:47 |
| sbezverk | kfox1111: yeah we need msec timestamps | 21:47 |
| sbezverk | jascott1: would it be more yaml like, conditions: and then under list of charts with : enabled/disabled | 21:49 |
| sbezverk | like this conditions: | 21:49 |
| sbezverk | subchart1: enabled | 21:49 |
| sbezverk | subchart2: disabled | 21:49 |
| *** gfidente|afk has quit IRC | 21:49 | |
| jascott1 | sbezverk yes like that | 21:49 |
| jascott1 | but also we are talking about AND/OR/NOT | 21:49 |
| jascott1 | sub1,sub2&sub3 | 21:50 |
| jascott1 | sub1 OR sub2 AND sub3 | 21:50 |
| jascott1 | sub1^sub2 | 21:51 |
| jascott1 | sub1 AND NOT sub2 | 21:51 |
| sbezverk | jascott1: I see, that might be interesting task to parse all these | 21:51 |
| jascott1 | sbezverk lol yeah Ive been thinking the same since I will probably be doing it :) | 21:51 |
| jascott1 | there is also this idea of the format k8 labels use: {key: environment, operator: NotIn, values: [dev]} | 21:53 |
| jascott1 | which is maybe more complex but already exists | 21:54 |
| inc0 | sadpanda...my galera is broken | 21:54 |
| inc0 | it doesn't get to listen on 3306 and I have no idea why | 21:54 |
| *** v1k0d3n has quit IRC | 21:59 | |
| *** breitz has quit IRC | 22:00 | |
| openstackgerrit | sayantani proposed openstack/kolla-ansible: Update Bifrost playbooks https://review.openstack.org/425435 | 22:00 |
| kfox1111 | jascott1: theres one weird thing there... | 22:00 |
| *** breitz has joined #openstack-kolla | 22:00 | |
| *** v1k0d3n has joined #openstack-kolla | 22:01 | |
| kfox1111 | jascott1: , should probablyl be scoped lower then all the operators. | 22:01 |
| kfox1111 | so, foo & bar, | 22:01 |
| kfox1111 | foo could be "global.a,global.b" | 22:01 |
| kfox1111 | whichever is first defined. | 22:01 |
| jascott1 | so you are saying comma is only delimiter and not operator | 22:03 |
| jascott1 | ? | 22:03 |
| *** Syffs has joined #openstack-kolla | 22:04 | |
| *** fguillot has quit IRC | 22:04 | |
| kfox1111 | well, its an operator of sorts, | 22:04 |
| kfox1111 | but it happens before anythign else. | 22:04 |
| sbezverk | jascott1: imho you should use already existing format for something similar you try to achieve and not invent yet another option | 22:04 |
| kfox1111 | or else ()'s are needed for grouping. | 22:04 |
| kfox1111 | don't know I guess. | 22:05 |
| kfox1111 | just thinkg, you might want: | 22:05 |
| kfox1111 | (global.a,global.b)&(global.c) | 22:05 |
| kfox1111 | first defined of a/b's boolean, | 22:05 |
| kfox1111 | anded with value c. | 22:05 |
| kfox1111 | this is kind of why I was suggesting just to use gotl. | 22:07 |
| kfox1111 | it gets complicated fast. :/ | 22:07 |
| kfox1111 | though gotls lang has its own uglies. :/ | 22:07 |
| kfox1111 | but they are known and accepted already. | 22:08 |
| *** lrensing has quit IRC | 22:09 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes: WIP: Add Development Setup Script for use with Kubernetes Helm https://review.openstack.org/405720 | 22:09 |
| sdake | bmace_ give thtat a go | 22:09 |
| bmace_ | kk, i'll take a look | 22:10 |
| kfox1111 | sdake: the kolla namespace has to be explicitly created. | 22:11 |
| sdake | kfox1111 gotta pick up kids from school brb | 22:11 |
| *** srwilkers has quit IRC | 22:13 | |
| bmace_ | sdake: not clear how anything in that change set would fix anything.. nothing changed in dev_workflow.sh where the failure is? | 22:13 |
| *** goldyfruit has quit IRC | 22:16 | |
| *** rhallisey has quit IRC | 22:18 | |
| sbezverk | kfox1111: can you print what gets returned by ovs command you used? | 22:19 |
| kfox1111 | yeah. | 22:19 |
| kfox1111 | wow. all green, first time. its been a while. :/ | 22:21 |
| kfox1111 | really need to get back there asap. | 22:22 |
| sdake | bmace_ which failure do you get | 22:22 |
| kfox1111 | sdake: https://review.openstack.org/#/c/424921/ please. | 22:22 |
| bmace_ | sdake: from dev_workflow.sh run: Unable to locate iscsid directory in search_paths=[/etc/kolla, /etc/kolla-kubernetes, /opt/kolla/etc/kolla, /opt/kolla-kubernetes/etc/kolla-kubernetes] | 22:22 |
| sdake | bmace_ ya - that is because >> is coiming from ceph_all_in_one instead of iscsi_all_in_one | 22:23 |
| sdake | pretty sur thats in that path but maybe i fat fingered it | 22:23 |
| kfox1111 | sdake: didn't see it in the ps. | 22:24 |
| sdake | ok will look - can't do 2 things at the exact same itme | 22:24 |
| sdake | so pick which one you gys want first :) | 22:24 |
| sdake | let me have a look at the dev workflow since bmace has been waiting on it all day | 22:24 |
| kfox1111 | of corse, I will vote for my review. ;) | 22:24 |
| kfox1111 | trying hard to get the gate stable again. | 22:25 |
| kfox1111 | but dev env is important too. | 22:25 |
| kfox1111 | k | 22:25 |
| sbezverk | kfox1111: please add this command for me | 22:28 |
| sbezverk | ovs-dpctl show | 22:28 |
| *** rwellum has joined #openstack-kolla | 22:29 | |
| sbezverk | just output no check or anything | 22:29 |
| kfox1111 | in another loop? at the end? | 22:29 |
| sbezverk | kfox1111: before the loop | 22:29 |
| sbezverk | nothing else | 22:29 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes: WIP: Add Development Setup Script for use with Kubernetes Helm https://review.openstack.org/405720 | 22:30 |
| bmace_ | kk, i see the change. i'll make it in my container and try it out | 22:30 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes: WIP: Add Development Setup Script for use with Kubernetes Helm https://review.openstack.org/405720 | 22:31 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Add dep for neutron-openvswitch-agent on a working openvswitch. https://review.openstack.org/425376 | 22:31 |
| sdake | bmace_ there is our final work | 22:31 |
| sdake | bmace_ or semi-final - that got us running | 22:31 |
| kfox1111 | sdake: looks good to me. | 22:31 |
| kfox1111 | sdake: did you get it fully working? | 22:31 |
| sdake | kfox1111 our env was misconfigured from kubeadm | 22:32 |
| sdake | kfox1111 so not certian although all containers were in an operational state | 22:32 |
| bmace_ | sdake: i'm betting line 17 should be iscsi and not ceph also? | 22:32 |
| sdake | bmace_ i'm not sure the script needs more work | 22:32 |
| sdake | it needs to be made idempotent for one | 22:32 |
| kfox1111 | heh. | 22:32 |
| kfox1111 | not sure thats a good idea. | 22:33 |
| kfox1111 | well, idempotent is a good idea, | 22:33 |
| * sdake has flashbacks of kolla pre compoe days | 22:33 | |
| sdake | compose days | 22:33 |
| *** jheroux has quit IRC | 22:33 | |
| kfox1111 | but not trying to make the current workflow scripts that way. :/ | 22:33 |
| kfox1111 | yeah. it was intended just to get it workign once from scratch to working and give up if it fails. | 22:33 |
| sbezverk | kfox1111: I think line 89 is missing ';' | 22:33 |
| kfox1111 | fault tolerance would be a huge pain. | 22:33 |
| sdake | kfox1111 i get it | 22:33 |
| kfox1111 | sbezverk: doh. your right. | 22:34 |
| sdake | kfox1111 ok ie had a bit of a rough day so I gotta jet for a bit | 22:34 |
| sdake | bmace_ remember the key ther eis to uninstall iscsi-initiatoior on your host | 22:34 |
| kfox1111 | k. | 22:34 |
| kfox1111 | sdake: hope you didn't use my name in vain too often. ;) | 22:34 |
| sdake | not a tall | 22:34 |
| sdake | thanks for getting it started ;) | 22:34 |
| sdake | there was alot of confusion about why _ vs - in command namees | 22:35 |
| kfox1111 | thanks for working on it. :) | 22:35 |
| sdake | and why some things are in test and some in tools | 22:35 |
| sdake | anyway - I'll be documenting how to do it without a script | 22:35 |
| sdake | and we can tune from there | 22:35 |
| kfox1111 | 'why' implies perhaps some concious decision. | 22:35 |
| bmace_ | sdake: see my comment about line 17? i can make it in the review also, but almost sure that should be iscsi rather than ceph now | 22:35 |
| kfox1111 | the idea behind tests/tools was tests were scripts really intended just for the gate, | 22:35 |
| * sdake wants to ptfo | 22:35 | |
| kfox1111 | while tools were more generally useful. | 22:36 |
| kfox1111 | not sure it fell out that way in reality though. | 22:36 |
| sdake | bmace_ which file | 22:36 |
| sdake | bmace_ i'll look | 22:36 |
| jascott1 | sbezverk +1 for reusing existing format (and pkg) | 22:36 |
| bmace_ | the one you recently uploaded. setup_config.sh | 22:36 |
| kfox1111 | oh... | 22:37 |
| openstackgerrit | Merged openstack/kolla-ansible: Fix bootstrap.yml in Bifrost https://review.openstack.org/424271 | 22:37 |
| sdake | good call bmace | 22:37 |
| kfox1111 | sbezverk: I think the ; thing might be why its not starting too. | 22:37 |
| sdake | although it works with ceph version :) | 22:37 |
| kfox1111 | another missing ';' | 22:37 |
| sdake | bmace_ double check that file is in the repo | 22:37 |
| openstackgerrit | Merged openstack/kolla: Add ceph fs containers https://review.openstack.org/422950 | 22:37 |
| kfox1111 | inc0: thanks. :) | 22:38 |
| sbezverk | kfox1111: two more | 22:38 |
| kfox1111 | yeah. | 22:38 |
| sdake | bmace_ indeed it is | 22:38 |
| bmace_ | tests/bin/setup_config.sh? it was part of your last change | 22:38 |
| inc0 | np kfox1111 | 22:38 |
| sdake | bmace_ i didn't test with it that way | 22:38 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Add dep for neutron-openvswitch-agent on a working openvswitch. https://review.openstack.org/425376 | 22:38 |
| bmace_ | sdake: well, it would be really odd to use the ceph all in one kubernetes config when using iscsid | 22:38 |
| bmace_ | i'm assuming it is wrong, but i'll try it my way and make sure | 22:39 |
| sdake | bmace_ well it should be iscsi, what i'm getting at it I am out of steam to test it | 22:40 |
| kfox1111 | yeah. I think iscsi's config shoudl be used. | 22:40 |
| sdake | but I'll push an update if you test it that way | 22:40 |
| bmace_ | kk, will take me a few to get my kubeX env back in shape, then i'll try it | 22:40 |
| *** sayantani01 has quit IRC | 22:40 | |
| *** sayantani01 has joined #openstack-kolla | 22:41 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes: WIP: Add Development Setup Script for use with Kubernetes Helm https://review.openstack.org/405720 | 22:41 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible: Implement Keystone zero-downtime upgrade https://review.openstack.org/425446 | 22:46 |
| inc0 | hey, can someone deploy ansible-ubuntu-source? | 22:46 |
| inc0 | my mariadb is acting up on fresh deploy and I'm scared by it | 22:46 |
| kfox1111 | inc0: trigger a gate job test? | 22:48 |
| inc0 | gates are fine, but gates aren't really multinode | 22:48 |
| inc0 | not in ansible | 22:48 |
| *** mgoddard has quit IRC | 22:51 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible: Implement Keystone zero-downtime upgrade https://review.openstack.org/425446 | 22:51 |
| kfox1111 | ah. ok. | 22:52 |
| *** dave-mcc_ has joined #openstack-kolla | 22:52 | |
| *** dave-mccowan has quit IRC | 22:54 | |
| inc0 | ok...that's probably local to me | 22:54 |
| inc0 | vms aren't behaving consistently | 22:55 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla: Implement Keystone zero-downtime upgrade https://review.openstack.org/388544 | 22:55 |
| *** vhosakot has joined #openstack-kolla | 22:57 | |
| *** huzhengchuan has quit IRC | 22:59 | |
| kfox1111 | oh my gosh yes! | 22:59 |
| kfox1111 | https://blueprints.launchpad.net/horizon/+spec/reorganise-access-and-security | 22:59 |
| kfox1111 | that crazy ui is finally going away | 22:59 |
| * kfox1111 jumps up and down and dances | 22:59 | |
| *** l4yerffeJ_ has joined #openstack-kolla | 23:03 | |
| *** l4yerffeJ has quit IRC | 23:03 | |
| *** dave-mccowan has joined #openstack-kolla | 23:05 | |
| *** dave-mcc_ has quit IRC | 23:05 | |
| *** rushil has quit IRC | 23:07 | |
| *** esmiurium has quit IRC | 23:09 | |
| sdake | gates are fine | 23:13 |
| sdake | finedog ftw | 23:13 |
| *** TxGirlGeek has quit IRC | 23:17 | |
| *** esmiurium has joined #openstack-kolla | 23:18 | |
| *** krtaylor has quit IRC | 23:18 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Local registry in gate for trunk builds https://review.openstack.org/418550 | 23:18 |
| *** rushil has joined #openstack-kolla | 23:19 | |
| kfox1111 | sbezverk: http://logs.openstack.org/76/425376/5/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-nv/77a2ba7/logs/pods/kolla-neutron-openvswitch-agent-network-f4h0q-deps-and-update-permissions.txt | 23:19 |
| kfox1111 | interesting... | 23:20 |
| sbezverk | kfox1111: yeah it looks like it does not see pid file | 23:21 |
| kfox1111 | /var/run is mounted though. | 23:21 |
| kfox1111 | looks like it may need privilege too. | 23:21 |
| sbezverk | kfox1111: we need to confirm if pid file gets created | 23:22 |
| kfox1111 | do you have a running container we can poke at? | 23:22 |
| sbezverk | one sec, I recently killed my ovs, let me try to deploy service | 23:23 |
| kfox1111 | on my bare metal host runnig ovs on the host, it does. | 23:23 |
| kfox1111 | I'm guessing if its looking for the pid file, it will be looking up the pid too. | 23:24 |
| kfox1111 | maybe we need to set the pid namespace on it to host too. | 23:24 |
| kfox1111 | though thats a pod level thing. :/ | 23:25 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Add dep for neutron-openvswitch-agent on a working openvswitch. https://review.openstack.org/425376 | 23:25 |
| *** newmember has quit IRC | 23:25 | |
| sbezverk | kfox1111: give me 5 minutes I think I will get ovs up and running | 23:25 |
| kfox1111 | k | 23:25 |
| *** newmember has joined #openstack-kolla | 23:25 | |
| kfox1111 | oh... interesting... | 23:26 |
| kfox1111 | its getting the pid out of the pid file, | 23:26 |
| kfox1111 | then opening /var/runn/openvswitch/ovs-vswitchd.<PID>.ctl socket. | 23:27 |
| kfox1111 | so we shouldn't need pid namespace in that case.... | 23:28 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Add dep for neutron-openvswitch-agent on a working openvswitch. https://review.openstack.org/425376 | 23:28 |
| sdake | rwellum http://docs.openstack.org/developer/kolla-kubernetes/development-environment.html | 23:28 |
| *** vhosakot has quit IRC | 23:29 | |
| *** rushil has quit IRC | 23:33 | |
| rwellum | sdake, thanks for the link! | 23:35 |
| sdake | rwellum np :) | 23:38 |
| sdake | service layer looking pretty good :) | 23:40 |
| kfox1111 | sbezverk: https://review.openstack.org/#/c/424368/ should be ready | 23:41 |
| kfox1111 | sdake: wait till you see it wrapped up in a computekit package. :) | 23:42 |
| * inc0 is scared | 23:42 | |
| sbezverk | kfox1111: docker exec -tu root -ti f0e12b3057b3 ls /var/run/openvswitch | 23:42 |
| sbezverk | br-ex.mgmt br-ex.snoop db.sock ovs-vswitchd.14.ctl ovsdb-server.1.ctl | 23:42 |
| *** lamt has quit IRC | 23:42 | |
| kfox1111 | sbezverk: weird. no pid file... | 23:43 |
| sdake | inc0 the workflow is pretty good, although super pokey | 23:43 |
| kfox1111 | maybe there is a missign arg? | 23:43 |
| sdake | although nobody cares about deploy speed, its operational performance that matters | 23:43 |
| inc0 | sdake, scared because my kolla-ansible deploy is broken and I have no idea whats happening | 23:43 |
| sbezverk | kfox1111: 424368 ever succeeded for entrypoint? | 23:43 |
| sdake | inc0 oh gotcha | 23:43 |
| inc0 | can you try deploying master | 23:43 |
| kfox1111 | I suspect the shell workflow will be faster then the helm one. | 23:43 |
| inc0 | ? | 23:43 |
| sdake | inc0 i am leaving for dinner with my parents in 10 minutes - so not really | 23:44 |
| sdake | inc0 possibly when i get back | 23:44 |
| kfox1111 | as it won't bother with loading stuff that isn't needed at the time until later. we'll see though. | 23:44 |
| inc0 | I'd appreciate it | 23:44 |
| sdake | inc0 can you be more specific what is busted | 23:44 |
| inc0 | mariadb doesn't bind to a port | 23:44 |
| sdake | inc0 deploy - reconfigure - ugprade ? | 23:44 |
| kfox1111 | sbezverk: yeah. | 23:44 |
| sdake | inc0 3 nodes or aio? | 23:44 |
| inc0 | and first error I'm getting is "connection reused" | 23:44 |
| inc0 | 3 nodes | 23:44 |
| sdake | iptables? | 23:44 |
| sdake | refused? | 23:45 |
| sdake | or reused :) | 23:45 |
| inc0 | well it should bind to port | 23:45 |
| inc0 | refused | 23:45 |
| *** sayantani01 has quit IRC | 23:45 | |
| inc0 | ss -plant doesn't show it | 23:45 |
| sdake | i'v seen that in days past | 23:45 |
| inc0 | or rather, not consistently | 23:45 |
| *** sayantani01 has joined #openstack-kolla | 23:45 | |
| kfox1111 | uh... weird... | 23:46 |
| kfox1111 | grep vswitchd /usr/lib/systemd/ -r #returns nothing. | 23:46 |
| kfox1111 | oh.. they are doing something quite different... | 23:47 |
| kfox1111 | ovs-ctl. | 23:47 |
| kfox1111 | so, we may need --pidfile=/var/run/openvswitch/ovs-vswitchd.pid | 23:52 |
| kfox1111 | it says its default, but doesn't seem to be adding it. | 23:52 |
| kfox1111 | and its explicit in redhat's running version. | 23:52 |
| *** rushil has joined #openstack-kolla | 23:53 | |
| sbezverk | kfox1111: I see another issue on my cluster :( entrypoint doesn ot see socker file | 23:55 |
| kfox1111 | hmm? | 23:55 |
| kfox1111 | the ovs one? | 23:55 |
| kfox1111 | that is really strange, cause that seems to be working in the gate. and there's no conditional around it. | 23:56 |
| sbezverk | kfox1111: I can see the file exist | 23:56 |
| kfox1111 | its blocking on it though? | 23:57 |
| *** lamt has joined #openstack-kolla | 23:58 | |
| kfox1111 | hmm.. do we have reqdeps and initialize-ovs-vswitchd reversed in the vswitchd template? | 23:58 |
| kfox1111 | shouldn't the socket exist before it tries to tweak the db? | 23:58 |
| kfox1111 | maybe its just fail/retrying until it works. but less then ideal... | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!