| *** genek has quit IRC | 00:00 | |
| *** mnasiadka has quit IRC | 00:01 | |
| *** mnasiadka has joined #openstack-kolla | 00:02 | |
| *** danardelean has quit IRC | 00:04 | |
| *** bastafidli has quit IRC | 00:06 | |
| *** mnasiadka has quit IRC | 00:07 | |
| *** MarginHu has joined #openstack-kolla | 00:10 | |
| *** mnasiadka has joined #openstack-kolla | 00:12 | |
| *** vhosakot has quit IRC | 00:14 | |
| *** mnasiadka has quit IRC | 00:18 | |
| *** yingjun has joined #openstack-kolla | 00:19 | |
| *** mnasiadka has joined #openstack-kolla | 00:22 | |
| *** mnasiadka has quit IRC | 00:27 | |
| -openstackstatus- NOTICE: due to unscheduled restart of zuulv3.o.o you will need to 'recheck' your jobs that were last running. Sorry for the inconvenience. | 00:32 | |
| *** mnasiadka has joined #openstack-kolla | 00:33 | |
| *** xinliang has quit IRC | 00:37 | |
| *** genek has joined #openstack-kolla | 00:38 | |
| *** mnasiadka has quit IRC | 00:39 | |
| *** tovin07_ has joined #openstack-kolla | 00:40 | |
| *** mnasiadka has joined #openstack-kolla | 00:43 | |
| *** genek has quit IRC | 00:44 | |
| *** mnasiadka has quit IRC | 00:47 | |
| *** salv-orlando has joined #openstack-kolla | 00:48 | |
| *** goldyfruit1 has joined #openstack-kolla | 00:48 | |
| *** goldyfruit has quit IRC | 00:48 | |
| *** xinliang has joined #openstack-kolla | 00:50 | |
| *** xinliang has joined #openstack-kolla | 00:50 | |
| *** yangyapeng has quit IRC | 00:51 | |
| *** yangyapeng has joined #openstack-kolla | 00:51 | |
| *** salv-orlando has quit IRC | 00:52 | |
| *** mnasiadka has joined #openstack-kolla | 00:53 | |
| *** yangyapeng has quit IRC | 00:56 | |
| *** mnasiadka has quit IRC | 00:57 | |
| *** huanxie has joined #openstack-kolla | 01:05 | |
| openstackgerrit | Heetae Ahn proposed openstack/kolla master: Add RHEL subscription registration https://review.openstack.org/495148 | 01:18 |
|---|---|---|
| Sean-md | hi all, when I run "helm install ..cinder-control..." to deploy kolla-kubernetes | 01:21 |
| *** kevzha01 has joined #openstack-kolla | 01:22 | |
| Sean-md | "/usr/local/bin/dumb-init /bin/bash /usr/local/bin/kolla_start" is executed, before running "kolla-start" by using the command of cinder-api pod | 01:22 |
| lxdong | how to join openstack-kolla slack? | 01:22 |
| Sean-md | So, cinder-api pod is crashing, because port 8776 is reused. | 01:24 |
| Sean-md | lxdong, openstack-kolla channel is in kubernetes(kubernetes.slack.com) | 01:26 |
| lxdong | ok | 01:30 |
| Sean-md | please let me know how not to run "/usr/local/bin/dumb-init /bin/bash /usr/local/bin/kolla_start" when deploying cinder-control using helm | 01:31 |
| kolla-slack | <lxdong.007> nice | 01:32 |
| *** naichuans has quit IRC | 01:34 | |
| *** genek has joined #openstack-kolla | 01:34 | |
| kolla-slack | <lxdong.007> ```bash /usr/libexec/qemu-kvm -name guest=instance-00000034,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-2-instance-00000034/master-key.aes -machine pc-i440fx-rhel7.4.0,accel=tcg,usb=off,dump-guest-core=off -cpu | 01:38 |
| kolla-slack | Opteron_G4,acpi=on,ss=on,monitor=on,movbe=on,hypervisor=on,arat=on,fsgsbase=on,bmi1=on,smep=on,bmi2=on,erms=on,mpx=on,adx=on,smap=on,clflushopt=on,pku=on,ospke=on,xsaveopt=on,xgetbv1=on,mmxext=on,3dnowext=on,3dnow=on,cr8legacy=on,avx=off,misalignsse=off,3dnowprefetch=off,xop=off,fma4=off -m 4096 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid a5ad08ca-e0eb-4c77-abea-04291743ad5c -smbios type=1,manufacturer=OpenStack | 01:38 |
| kolla-slack | Foundation,product=OpenStack Nova,version=16.0.0,serial=4c4c4544-004e-3210-8048-c4c04f5a3258,uuid=a5ad08ca-e0eb-4c77-abea-04291743ad5c,family=Virtual Machine -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-2-instance-00000034/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -de | 01:38 |
| kolla-slack | virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -object secret,id=virtio-disk0-secret0,data=eMczfVZF7mk1iEmpRigZS1F6p3n3hTtbwjGS3PosENU=,keyid=masterKey0,iv=ZEcx6nyM1EYm8S6h0bPJ7g==,format=base64 -drive | 01:38 |
| kolla-slack | file=rbd:volumes/volume-fea9b99e-686d-495f-bcf0-9afe9ed151c3:id=cinder:auth_supported=cephx\;none:mon_host=192.168.4.150\:6789\;192.168.4.151\:6789,file.password-secret=virtio-disk0-secret0,format=raw,if=none,id=drive-virtio-disk0,serial=fea9b99e-686d-495f-bcf0-9afe9ed151c3,cache=writeback,discard=unmap -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=26,id=hostnet0 -device | 01:38 |
| kolla-slack | virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:a4:33:76,bus=pci.0,addr=0x3 -chardev pty,id=charserial0,logfile=/var/lib/nova/instances/a5ad08ca-e0eb-4c77-abea-04291743ad5c/console.log,logappend=off -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -chardev | 01:38 |
| kolla-slack | socket,id=charchannel1,path=/var/lib/libvirt/qemu/org.qemu.guest_agent.0.instance-00000034.sock,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -spice port=5900,addr=192.168.4.150,disable-ticketing,seamless-migration=on -k en-us -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device | 01:38 |
| kolla-slack | virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on ``` | 01:38 |
| *** genek has quit IRC | 01:39 | |
| kolla-slack | <lxdong.007> It's an issue about windows instance mouse drifting .I have use_usb_tablet = True and pointer_model = usbtablet in nova.conf with spice enabled and [spice].agent_enabled = True , instance still created with usb=off and doesn't take effect. | 01:41 |
| *** yangyapeng has joined #openstack-kolla | 01:48 | |
| *** huzhengchuan has joined #openstack-kolla | 01:48 | |
| *** salv-orlando has joined #openstack-kolla | 01:48 | |
| *** yangyape_ has joined #openstack-kolla | 01:49 | |
| *** yangyapeng has quit IRC | 01:52 | |
| *** genek has joined #openstack-kolla | 01:53 | |
| *** salv-orlando has quit IRC | 01:53 | |
| *** jtriley has joined #openstack-kolla | 01:53 | |
| *** lukepatrick has joined #openstack-kolla | 01:55 | |
| *** genek has quit IRC | 01:58 | |
| *** genek has joined #openstack-kolla | 01:59 | |
| *** genek has quit IRC | 02:03 | |
| *** zhubingbing has joined #openstack-kolla | 02:03 | |
| *** huanxie has quit IRC | 02:09 | |
| *** lxdong has quit IRC | 02:10 | |
| *** huanxie has joined #openstack-kolla | 02:10 | |
| *** lukepatrick has quit IRC | 02:12 | |
| *** lukepatrick has joined #openstack-kolla | 02:14 | |
| *** genek has joined #openstack-kolla | 02:17 | |
| *** jtriley has quit IRC | 02:18 | |
| *** jtriley has joined #openstack-kolla | 02:27 | |
| *** pramodrj07 has quit IRC | 02:29 | |
| *** MasterOfBugs has quit IRC | 02:29 | |
| *** huanxie has quit IRC | 02:42 | |
| *** dave-mccowan has quit IRC | 02:44 | |
| *** duonghq has quit IRC | 02:47 | |
| *** hieulq has quit IRC | 02:47 | |
| *** duonghq has joined #openstack-kolla | 02:48 | |
| *** hieulq has joined #openstack-kolla | 02:48 | |
| *** dave-mccowan has joined #openstack-kolla | 02:48 | |
| *** zhangfei has joined #openstack-kolla | 02:49 | |
| *** salv-orlando has joined #openstack-kolla | 02:50 | |
| *** unicell has joined #openstack-kolla | 02:54 | |
| *** salv-orlando has quit IRC | 02:54 | |
| kolla-slack | <lxdong.007> Did anyone meet the issue about mouse drifting of windows instance runing in kvm? | 02:55 |
| *** hieulq has quit IRC | 02:56 | |
| *** duonghq has quit IRC | 02:56 | |
| *** duonghq has joined #openstack-kolla | 02:57 | |
| *** hieulq has joined #openstack-kolla | 02:57 | |
| *** unicell has quit IRC | 02:58 | |
| *** lukepatrick has quit IRC | 03:00 | |
| *** lukepatrick has joined #openstack-kolla | 03:01 | |
| *** unicell has joined #openstack-kolla | 03:06 | |
| *** jtriley has quit IRC | 03:06 | |
| *** huzhengchuan has quit IRC | 03:09 | |
| *** unicell has quit IRC | 03:10 | |
| *** openstackgerrit has quit IRC | 03:22 | |
| *** jtriley has joined #openstack-kolla | 03:23 | |
| *** huanxie has joined #openstack-kolla | 03:27 | |
| *** unicell has joined #openstack-kolla | 03:30 | |
| *** goldyfruit1 has quit IRC | 03:31 | |
| *** pratapagoutham has quit IRC | 03:31 | |
| *** lukepatrick has quit IRC | 03:31 | |
| *** lukepatrick has joined #openstack-kolla | 03:32 | |
| *** unicell has quit IRC | 03:34 | |
| *** mdnadeem has joined #openstack-kolla | 03:36 | |
| *** jtriley has quit IRC | 03:42 | |
| *** genek has quit IRC | 03:42 | |
| *** unicell has joined #openstack-kolla | 03:43 | |
| *** unicell has quit IRC | 03:47 | |
| *** salv-orlando has joined #openstack-kolla | 03:50 | |
| duonghq | anybody see gerrit is getting trouble? | 03:52 |
| *** unicell has joined #openstack-kolla | 03:54 | |
| *** lukepatrick has quit IRC | 03:54 | |
| *** jtriley has joined #openstack-kolla | 03:55 | |
| *** gkadam has joined #openstack-kolla | 03:55 | |
| *** salv-orlando has quit IRC | 03:55 | |
| *** kiennt26 has quit IRC | 03:57 | |
| *** genek has joined #openstack-kolla | 03:57 | |
| *** kiennt26 has joined #openstack-kolla | 03:57 | |
| *** huanxie has quit IRC | 03:58 | |
| *** unicell has quit IRC | 03:59 | |
| *** genek has quit IRC | 04:01 | |
| *** Jeffrey4l has quit IRC | 04:02 | |
| *** Jeffrey4l has joined #openstack-kolla | 04:03 | |
| *** unicell has joined #openstack-kolla | 04:07 | |
| *** genek has joined #openstack-kolla | 04:11 | |
| *** unicell has quit IRC | 04:11 | |
| zhubingbing | duonghq | 04:12 |
| zhubingbing | yeah | 04:13 |
| zhubingbing | I can't open gerrit | 04:13 |
| duonghq | zhubingbing, ack | 04:14 |
| *** genek has quit IRC | 04:15 | |
| *** unicell has joined #openstack-kolla | 04:19 | |
| *** genek has joined #openstack-kolla | 04:20 | |
| *** unicell has quit IRC | 04:23 | |
| *** huanxie has joined #openstack-kolla | 04:26 | |
| *** Pavo has quit IRC | 04:27 | |
| *** jinke has joined #openstack-kolla | 04:29 | |
| *** unicell has joined #openstack-kolla | 04:31 | |
| *** jinke has left #openstack-kolla | 04:32 | |
| *** jinke has joined #openstack-kolla | 04:34 | |
| *** unicell has quit IRC | 04:36 | |
| *** janki has joined #openstack-kolla | 04:36 | |
| *** unicell has joined #openstack-kolla | 04:39 | |
| *** zhangfei has quit IRC | 04:42 | |
| *** kiennt26 has quit IRC | 04:47 | |
| *** dave-mccowan has quit IRC | 04:47 | |
| *** signed8bit has joined #openstack-kolla | 04:50 | |
| *** unicell1 has joined #openstack-kolla | 04:50 | |
| *** salv-orlando has joined #openstack-kolla | 04:51 | |
| *** fguillot has joined #openstack-kolla | 04:51 | |
| *** unicell has quit IRC | 04:52 | |
| *** fguillot has left #openstack-kolla | 04:52 | |
| *** salv-orlando has quit IRC | 04:57 | |
| *** zhangfei has joined #openstack-kolla | 04:58 | |
| *** unicell has joined #openstack-kolla | 04:58 | |
| *** unicell1 has quit IRC | 04:59 | |
| *** genek has quit IRC | 05:00 | |
| *** genek has joined #openstack-kolla | 05:00 | |
| masber | good afternoon, I was wondering if someone could help me troubleshooting kolla-ansible deployment? I am getting this error http://paste.openstack.org/raw/623913/ | 05:01 |
| masber | I am really desperate, can't understand what is the issue | 05:04 |
| *** huanxie has quit IRC | 05:05 | |
| *** genek has quit IRC | 05:05 | |
| *** hyakuhei has quit IRC | 05:05 | |
| *** coolsvap has joined #openstack-kolla | 05:07 | |
| *** huanxie has joined #openstack-kolla | 05:07 | |
| *** jtriley has quit IRC | 05:08 | |
| *** genek has joined #openstack-kolla | 05:13 | |
| *** salv-orlando has joined #openstack-kolla | 05:15 | |
| *** genek has quit IRC | 05:17 | |
| *** shardy has quit IRC | 05:23 | |
| *** unicell1 has joined #openstack-kolla | 05:24 | |
| *** shardy has joined #openstack-kolla | 05:25 | |
| *** janki has quit IRC | 05:25 | |
| *** unicell has quit IRC | 05:27 | |
| *** unicell2 has joined #openstack-kolla | 05:27 | |
| *** unicell1 has quit IRC | 05:29 | |
| *** unicell has joined #openstack-kolla | 05:30 | |
| *** unicell2 has quit IRC | 05:31 | |
| *** unicell1 has joined #openstack-kolla | 05:33 | |
| *** unicell has quit IRC | 05:34 | |
| inc0 | masber: check docker logs nova-compute on test-openstack-controller | 05:38 |
| inc0 | also /var/lib/docker/volmes/kolla_logs/_data/nova | 05:38 |
| inc0 | Jeffrey4l: around? | 05:38 |
| Jeffrey4l | inc0, yep | 05:38 |
| inc0 | https://review.openstack.org/#/c/508759/ <- this seems ready | 05:38 |
| inc0 | https://review.openstack.org/#/c/512779/ <- I started to fiddle around multinode | 05:39 |
| Jeffrey4l | yes. should we add other binary jobs? | 05:39 |
| inc0 | no | 05:40 |
| inc0 | I think we shouldn't have binary jobs in ansible | 05:40 |
| inc0 | no difference in deployment logic and I'd rather have more scenarios | 05:40 |
| Jeffrey4l | iirc, we should add binary jobs to periodic jobs pipeline? | 05:40 |
| inc0 | yeah, but that's for later, first we need basic deploy gates for kolla and dockerhub pipeline for source images | 05:41 |
| Jeffrey4l | or use "experimetal" to trigger the jobs to check whether it is OK. | 05:41 |
| Jeffrey4l | ok. | 05:42 |
| *** bmace has quit IRC | 05:42 | |
| inc0 | nah, I'd rather have daily checks | 05:42 |
| *** bmace has joined #openstack-kolla | 05:42 | |
| inc0 | also experimental doesnt make sense any more | 05:42 |
| Jeffrey4l | daily checks is hard to fixed issue. | 05:42 |
| Jeffrey4l | i.e. if only centos+binary jobs failed, how can i push a patch and test it? | 05:42 |
| Jeffrey4l | then periodic jobs + experimetal jobs should be better. | 05:43 |
| inc0 | not sure how experimental will woork now, but you're right | 05:44 |
| inc0 | that'd make total sens | 05:44 |
| inc0 | ee | 05:44 |
| inc0 | ehh, I can't write;) | 05:44 |
| Jeffrey4l | anyway, that will be future tasks. | 05:44 |
| inc0 | right, we need our gates up and running first | 05:44 |
| inc0 | also I'd love to move more of setup_gate to ansible | 05:44 |
| Jeffrey4l | let us focus the source jobs right now | 05:45 |
| inc0 | agree | 05:45 |
| inc0 | we need to make deployment gates work in kolla | 05:45 |
| inc0 | I was thinking of using zuul cloner to pull patchset from kolla-ansible (since it works) | 05:46 |
| inc0 | and call kolla-ansible's run.yml from kolla job | 05:46 |
| inc0 | with additional env that will assume images are already built | 05:47 |
| Jeffrey4l | inc0, seem there is a way to use other's project jobs in zuulv3. | 05:47 |
| Jeffrey4l | i saw these once. | 05:47 |
| Jeffrey4l | maybe it works. | 05:47 |
| inc0 | take a look at it if you have time | 05:47 |
| Jeffrey4l | i will. | 05:48 |
| inc0 | I'll ask reviewers tomorrow on meeting to merge our gate patch asap | 05:48 |
| inc0 | for ansible | 05:48 |
| Jeffrey4l | btw, https://review.openstack.org/508661 kolla build jobs is OK now. | 05:48 |
| Jeffrey4l | we can fix the deploy jobs later. | 05:48 |
| inc0 | yeah I agree | 05:48 |
| Jeffrey4l | so this is ready to review too. | 05:48 |
| inc0 | better to have failing deploy gates than nothing at all | 05:48 |
| Jeffrey4l | deploy jobs is added with failing in kolla | 05:49 |
| inc0 | yes, I figured that we need to solve zuulv3 deploy gates in kolla ansible first | 05:49 |
| inc0 | and now to figure out how to call these plays correctly | 05:50 |
| Jeffrey4l | yes. i saw | 05:50 |
| Jeffrey4l | another thing, i think it is time to release next z stream for ocata and pike branch. | 05:51 |
| Jeffrey4l | centos 7.4 breaks all release tags | 05:51 |
| SamYaple | Jeffrey4l: why is that? | 05:51 |
| Jeffrey4l | SamYaple, cenots 7.4? | 05:51 |
| SamYaple | yes | 05:51 |
| SamYaple | like what specifically so im aware | 05:51 |
| Jeffrey4l | libvirt is upgraded and need change the configuration. | 05:52 |
| Jeffrey4l | let me find the patch. | 05:52 |
| SamYaple | ah yes. im familiar with that issue | 05:52 |
| inc0 | yeah, we had it in ubuntu because it got newer libvirt first | 05:52 |
| SamYaple | yea a while back | 05:52 |
| Jeffrey4l | https://review.openstack.org/#/c/506515/ | 05:53 |
| Jeffrey4l | here is the patch. | 05:53 |
| Jeffrey4l | https://review.openstack.org/#/c/506515/5/ansible/roles/nova/templates/qemu.conf.j2 | 05:53 |
| Jeffrey4l | here it he key fixed for centos 7.4 SamYaple | 05:53 |
| *** huanxie has quit IRC | 05:53 | |
| SamYaple | thanks Jeffrey4l | 05:57 |
| Jeffrey4l | np | 05:57 |
| SamYaple | inc0: im going to be putting in a OSH gate in loci (non-voting) i would like to do the same for kolla-ansible | 05:58 |
| SamYaple | do you see any issue with that? | 05:58 |
| masber | inc0, nova-compute should run on the control node or the compute? | 05:58 |
| SamYaple | rather ill be using the OSH gate job in LOCI | 05:58 |
| Jeffrey4l | what is "OSH" gate? | 05:59 |
| SamYaple | Jeffrey4l: openstack-helm | 05:59 |
| inc0 | SamYaple: if you can handle extended starts | 05:59 |
| inc0 | then sure | 05:59 |
| inc0 | masber: compute | 05:59 |
| SamYaple | inc0: yes those will be handled. we are still looking at the best way to do it | 05:59 |
| SamYaple | whether a base image of kolla or post layers with kolla bits | 06:00 |
| inc0 | cool | 06:00 |
| SamYaple | so LOCI will be cross-gating, so your gate work will get reconsumed (and ill be more able to help out with kolla-ansible gate) | 06:01 |
| SamYaple | it doesnt affect you directly in kolla, but jsut so youre aware | 06:01 |
| inc0 | good to hear, should be fairly easy to do | 06:02 |
| inc0 | you'll just need a lot of image name overrides;) | 06:02 |
| SamYaple | yea got some experince with the cross project gating, its super simple | 06:02 |
| masber | inc0, ok test-openstack-controller is my control and network node and compute-21 is for computing and storage. Going back to my error, nova-compute is not running on test-openstack-controller. http://paste.openstack.org/raw/623915/ | 06:02 |
| SamYaple | inc0: thats not so hard actually! its pretty straight forward | 06:02 |
| SamYaple | kolla-ansible was well thought out | 06:03 |
| inc0 | I know it's not hard, just a lot of it | 06:03 |
| inc0 | ;) | 06:03 |
| SamYaple | inc0: and in the case of cross gating we can just kolla-name the images so no overrides needed in gate | 06:03 |
| inc0 | nova_compute_image_full: ... | 06:03 |
| *** openstackgerrit has joined #openstack-kolla | 06:03 | |
| openstackgerrit | jinke proposed openstack/kolla-ansible master: Skydive cannot recognize namespace created by neutron https://review.openstack.org/508399 | 06:03 |
| inc0 | yeah whichever works, but I'm pretty sure you can override your globals | 06:04 |
| SamYaple | indeed thats of course doable | 06:04 |
| inc0 | I would be ok if you'd want to have it in our gate globals | 06:04 |
| SamYaple | well ill have a better grasp of whats needed once the kolla-ansible gates are at 100% | 06:04 |
| inc0 | it's templated so whatever, you can do sth like {% if loci_gate %}{% include 'loci_globals.yml' %} | 06:05 |
| inc0 | aio are ok already | 06:05 |
| SamYaple | i mean like fully converted from bash scripts | 06:05 |
| inc0 | https://review.openstack.org/#/c/508759/ | 06:05 |
| SamYaple | which i want to help with | 06:05 |
| inc0 | well global.yml generation is already in ansible | 06:05 |
| inc0 | so while I agree - we want to move out of bash in favor of ansible - that shouldn't block you | 06:06 |
| SamYaple | it honestly should be as simple as conditionalizing the image import, tarball vs loci gen'd | 06:06 |
| inc0 | your call, your help will ofc be appreciated | 06:06 |
| inc0 | it is really | 06:06 |
| SamYaple | ive always like gate work, now it *really* like it | 06:06 |
| SamYaple | with zuulv3 | 06:06 |
| inc0 | if only you'd get feedback in less than hour or so, I'd love it too | 06:07 |
| SamYaple | what do you mean? | 06:07 |
| inc0 | for now you can mimic what we did in kolla build gates | 06:07 |
| inc0 | gates takes ~20min each | 06:07 |
| SamYaple | yea that should be fine. what was the issue with that? | 06:07 |
| inc0 | I've seen them queued for 40 min sometimes | 06:08 |
| SamYaple | oh you mean for testing the gates | 06:08 |
| inc0 | yeah | 06:08 |
| SamYaple | yea that is what it is | 06:08 |
| SamYaple | i do alot of work saturday/sunday for that reason | 06:08 |
| SamYaple | insta-quee | 06:08 |
| inc0 | yeah it was awesome;) | 06:08 |
| inc0 | I did a bunch oo | 06:08 |
| inc0 | too | 06:08 |
| inc0 | anyway, I think it makes sense to move image preparation for k-ans to ansible too | 06:09 |
| inc0 | as next step | 06:09 |
| inc0 | move it out of bash | 06:09 |
| inc0 | if you're up to it SamYaple, I'd be happy to review, otherwise I'll do it as soon as we restore our basic gating infra (get coverage en par with previous) | 06:10 |
| SamYaple | yea as soon as you merge that patch im going to push up some updated bits for you | 06:10 |
| SamYaple | scrap a good chunk of the bash stuff | 06:10 |
| inc0 | cool | 06:11 |
| SamYaple | easier to iterate from "working" :) | 06:11 |
| inc0 | Jeffrey4l: can you ask pbourke and other tomorrow their morning to review gates asap? assuming Paul wasn't blown away by hurricane | 06:11 |
| Jeffrey4l | inc0, sure. | 06:12 |
| inc0 | cool, I'm siging off for now, talk to you all tomorrow | 06:12 |
| SamYaple | if you say his name three times he appears | 06:12 |
| SamYaple | pbourke pbourke pbourke | 06:12 |
| SamYaple | (eventually) | 06:12 |
| Jeffrey4l | lol | 06:12 |
| SamYaple | inc0: take care! | 06:12 |
| inc0 | actually it's 7am there | 06:12 |
| SamYaple | yup | 06:12 |
| inc0 | so he should be up pretty soon | 06:12 |
| SamYaple | this is about the time he pops onto IRC | 06:12 |
| SamYaple | well in about an hour | 06:13 |
| inc0 | yeah I won't be here in an hour, I'm half asleep already, gym killed me today | 06:13 |
| inc0 | gnite | 06:13 |
| * inc0 off | 06:13 | |
| SamYaple | nite | 06:13 |
| Jeffrey4l | nite | 06:13 |
| masber | which images do you use? | 06:14 |
| masber | I am trying centos binary 4.0.0, 4.0.3, 5.0.0 and 5.0.1 and none of them are working for me | 06:15 |
| SamYaple | masber: somethign that isnt ready for consumption, loci. i was tlaking to michal about having a job in loci repo to run kolla-ansible with loci images | 06:15 |
| SamYaple | kolla is the whole shebang, images and deploy. loci just does images | 06:16 |
| SamYaple | never deploy | 06:16 |
| *** jinke has quit IRC | 06:17 | |
| *** kiennt26 has joined #openstack-kolla | 06:17 | |
| *** xinliang has quit IRC | 06:17 | |
| *** mnasiadka has joined #openstack-kolla | 06:21 | |
| *** janki has joined #openstack-kolla | 06:24 | |
| Sean-md | hi all, the container of "centos-source-cinder-api:5.0.0" automatically runs "/usr/local/bin/dumb-init /bin/bash /usr/local/bin/kolla_start" on startup | 06:24 |
| Sean-md | it is not in /etc/rc.d.. | 06:24 |
| Sean-md | where can I remove "/usr/local/bin/dumb-init /bin/bash /usr/local/bin/kolla_start"??? | 06:25 |
| Sean-md | this is kolla-kubernetes | 06:26 |
| *** xinliang has joined #openstack-kolla | 06:26 | |
| *** jinke has joined #openstack-kolla | 06:29 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Move to zuul v3 in project jobs https://review.openstack.org/508759 | 06:30 |
| masber | can I use kolla-ansible 5.0.0 to deploy 4.0.0 images? | 06:31 |
| *** salv-orlando has quit IRC | 06:33 | |
| *** salv-orlando has joined #openstack-kolla | 06:35 | |
| *** danardelean has joined #openstack-kolla | 06:35 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Move to zuul v3 in project jobs https://review.openstack.org/508759 | 06:37 |
| *** danardelean has quit IRC | 06:37 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Move to zuul v3 in project jobs https://review.openstack.org/508759 | 06:38 |
| *** zhangfei has quit IRC | 06:43 | |
| *** pcaruana has joined #openstack-kolla | 06:45 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Move to zuul v3 in project jobs https://review.openstack.org/508759 | 06:48 |
| *** signed8b_ has joined #openstack-kolla | 06:50 | |
| *** signed8bit has quit IRC | 06:53 | |
| *** zhangfei has joined #openstack-kolla | 06:56 | |
| *** signed8b_ has quit IRC | 06:56 | |
| *** unicell1 has quit IRC | 06:59 | |
| *** genek has joined #openstack-kolla | 07:02 | |
| *** dardelean has joined #openstack-kolla | 07:08 | |
| *** danardelean has joined #openstack-kolla | 07:12 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Move to zuul v3 in project jobs https://review.openstack.org/508759 | 07:15 |
| *** danardelean has quit IRC | 07:16 | |
| *** b_bezak has joined #openstack-kolla | 07:21 | |
| *** genek has quit IRC | 07:23 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Move to zuul v3 in project jobs https://review.openstack.org/508759 | 07:24 |
| *** mewald has joined #openstack-kolla | 07:26 | |
| *** shardy is now known as shardy_afk | 07:26 | |
| mewald | Morning, how are we going to proceed with https://review.openstack.org/#/c/485150/ ? | 07:26 |
| *** tovin07_ has quit IRC | 07:28 | |
| *** ipsecguy_ has joined #openstack-kolla | 07:28 | |
| *** caowei has joined #openstack-kolla | 07:29 | |
| *** ipsecguy has quit IRC | 07:31 | |
| *** genek has joined #openstack-kolla | 07:32 | |
| *** danardelean has joined #openstack-kolla | 07:34 | |
| mewald | I just ran a recheck, now I am seeing only success from zuul. So if that's ok, I would like to see this merged soon :) | 07:35 |
| *** zhubingbing has quit IRC | 07:35 | |
| *** zhubingbing has joined #openstack-kolla | 07:36 | |
| *** yingjun has quit IRC | 07:36 | |
| *** zhubingbing has quit IRC | 07:39 | |
| *** zhubingbing has joined #openstack-kolla | 07:41 | |
| kolla-slack | <lxdong.007> To make use_usb_table=True take effect has to make [spice].agent_enabled=False which causes regression of guest console experience, Is there any tradeoff method? | 07:45 |
| *** egonzalez has joined #openstack-kolla | 07:46 | |
| *** huanxie has joined #openstack-kolla | 07:48 | |
| *** matrohon_ has joined #openstack-kolla | 07:51 | |
| *** paramite has joined #openstack-kolla | 07:52 | |
| *** shardy_afk is now known as shardy | 08:00 | |
| *** caowei has quit IRC | 08:04 | |
| *** caowei has joined #openstack-kolla | 08:05 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Move to zuul v3 in project jobs https://review.openstack.org/508759 | 08:06 |
| *** dougsz has joined #openstack-kolla | 08:07 | |
| *** lebauce_ has joined #openstack-kolla | 08:07 | |
| *** vkhanna has quit IRC | 08:07 | |
| *** ansiwen has quit IRC | 08:08 | |
| *** vkhanna has joined #openstack-kolla | 08:09 | |
| *** lebauce_ has quit IRC | 08:10 | |
| *** ansiwen has joined #openstack-kolla | 08:10 | |
| *** genek has quit IRC | 08:12 | |
| *** mgoddard has joined #openstack-kolla | 08:14 | |
| *** lebauce has quit IRC | 08:15 | |
| *** genek has joined #openstack-kolla | 08:18 | |
| *** lebauce has joined #openstack-kolla | 08:19 | |
| *** cah_link_ has joined #openstack-kolla | 08:20 | |
| *** genek has quit IRC | 08:23 | |
| *** genek has joined #openstack-kolla | 08:24 | |
| *** ipsecguy has joined #openstack-kolla | 08:24 | |
| *** clayton_ has joined #openstack-kolla | 08:27 | |
| *** mgoddard_ has joined #openstack-kolla | 08:27 | |
| *** ipsecguy_ has quit IRC | 08:28 | |
| *** clayton has quit IRC | 08:28 | |
| *** dciabrin has quit IRC | 08:28 | |
| *** mgoddard has quit IRC | 08:28 | |
| *** dciabrin has joined #openstack-kolla | 08:28 | |
| *** clayton_ is now known as clayton | 08:28 | |
| *** genek has quit IRC | 08:29 | |
| *** jrist has quit IRC | 08:30 | |
| *** mnasiadk1 has joined #openstack-kolla | 08:33 | |
| *** mnasiadka has quit IRC | 08:35 | |
| *** cah_link__ has joined #openstack-kolla | 08:44 | |
| mewald | The sensu images also build successfully https://review.openstack.org/#/c/486379/12 Please review :) | 08:45 |
| *** cah_link_ has quit IRC | 08:46 | |
| egonzalez | mewald, we have no CI now, there is code freeze at this moment until zuulv3 is working fine | 08:47 |
| mewald | egonzalez: Ahh that's still going on :( I ran recheck and Zuul came back with success to I was happy it's working now :D | 08:48 |
| egonzalez | only pep and docs tests | 08:49 |
| mewald | Do we still now have an ETA on when this is going to be fixed? | 08:49 |
| mewald | *not | 08:50 |
| *** dangtrinhnt has joined #openstack-kolla | 08:51 | |
| *** david-lyle has quit IRC | 08:53 | |
| dangtrinhnt | Hi, Trying to run the deploy-tacker-demo-sfc but looks like my public network does not work because I set neutron_external_interface to a non-IP interface. So my public network now use the default network range (10.0.2.0/24) | 08:54 |
| egonzalez | dangtrinhnt, the interface should not have any IP address, but the link connected to the interface should have a valid range configured to it | 08:57 |
| egonzalez | that range is the one you set in the public network , example for init-runonce https://github.com/openstack/kolla-ansible/blob/master/tools/init-runonce#L12 | 08:58 |
| dangtrinhnt | egonzalez, ah ok, I think I get it now. Many thanks. | 09:01 |
| *** Trinh_ has joined #openstack-kolla | 09:07 | |
| *** dangtrinhnt has left #openstack-kolla | 09:07 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Move to zuul v3 in project jobs https://review.openstack.org/508759 | 09:10 |
| *** danardelean has quit IRC | 09:21 | |
| *** cah_link__ has quit IRC | 09:26 | |
| openstackgerrit | caoyuan proposed openstack/kolla-kubernetes master: Update the link for Big Tent Governance https://review.openstack.org/512982 | 09:26 |
| *** cah_link__ has joined #openstack-kolla | 09:26 | |
| openstackgerrit | Chen proposed openstack/kolla-ansible master: Associate text to link https://review.openstack.org/512983 | 09:27 |
| *** caowei has quit IRC | 09:28 | |
| *** egonzalez has quit IRC | 09:31 | |
| *** pratapagoutham has joined #openstack-kolla | 09:45 | |
| *** pmisiak has joined #openstack-kolla | 09:49 | |
| pmisiak | hi guys | 09:50 |
| *** egonzalez has joined #openstack-kolla | 09:50 | |
| pmisiak | do you plan to backport this bug to stable/ocata? https://bugs.launchpad.net/kolla-ansible/+bug/1712767 | 09:51 |
| openstack | Launchpad bug 1712767 in kolla-ansible "openvswitch_db shouldn't be connect from remote host" [Undecided,Fix released] - Assigned to Kuo-tung Kao (jelly) (coding1314) | 09:51 |
| pmisiak | LBaaS doesn't work without this commit in stable/ocata because neutron-lbaas-agent try to connect to 127.0.0.1 insted of api_interface IP address | 09:52 |
| *** danardelean has joined #openstack-kolla | 09:54 | |
| *** rhallisey has joined #openstack-kolla | 09:54 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Move to zuul v3 in project jobs https://review.openstack.org/508759 | 09:55 |
| bjolo | morning | 09:56 |
| bjolo | is there a way to list the exakt command used to run a container? | 09:56 |
| bjolo | for ex | 09:56 |
| bjolo | i want to see the exact command used to fire up the neutron_dhcp_agent container | 09:57 |
| bjolo | egonzalez: | 09:57 |
| bjolo | found a tool called runlike | 09:57 |
| bjolo | pip install runlike | 09:57 |
| egonzalez | bjolo, in /etc/kolla/neutron-dhcp-agent/config.json | 09:57 |
| *** danardelean has quit IRC | 09:58 | |
| bjolo | sweet | 09:58 |
| bjolo | we have found a bug in old newton cloud | 09:59 |
| bjolo | built a new dhcp container that i wanna try out manually | 09:59 |
| bjolo | on one node only | 09:59 |
| bjolo | so my idea is to stop and remove the current container | 09:59 |
| bjolo | fire up the new container using the same start string, just bumping the container name | 10:00 |
| bjolo | should work? | 10:00 |
| pmisiak | bjolo: I remember in mitaka there was 'start' action in ansible, this action recreates all containers | 10:00 |
| pmisiak | maybe its still in newton? | 10:01 |
| bjolo | start action where? | 10:01 |
| bjolo | ouch not there on newton | 10:03 |
| bjolo | http://paste.openstack.org/show/623935/ | 10:03 |
| bjolo | thats how it looks in newton | 10:03 |
| pmisiak | when you run kolla-ansile you will see command line with ansible | 10:03 |
| pmisiak | and there is action=deploy for example | 10:03 |
| pmisiak | you can hack it and replace deploy with start :P | 10:04 |
| pmisiak | I used this many times | 10:04 |
| openstackgerrit | Merged openstack/kolla-ansible stable/ocata: Fix wrong condition for iscsid container https://review.openstack.org/505081 | 10:05 |
| bjolo | here is the runlike output | 10:05 |
| bjolo | http://paste.openstack.org/show/623936/ | 10:05 |
| bjolo | pmisiak: not sure i follow | 10:05 |
| *** kiennt26 has quit IRC | 10:05 | |
| pmisiak | bjolo | 10:06 |
| pmisiak | bjolo: https://github.com/openstack/kolla/blob/stable/newton/ansible/roles/neutron/tasks/start.yml | 10:06 |
| pmisiak | im talking about this file :) | 10:06 |
| bjolo | i wanna do this outside of kolla. just try a new binary directly from docker | 10:06 |
| pmisiak | aa ok :) | 10:06 |
| pmisiak | i thought you want to just recreate a container using kolla | 10:06 |
| bjolo | i wanna se if the bug patch in the new container fix our issue | 10:06 |
| bjolo | i dont want to reconfigure or upgrade the whole cloud | 10:07 |
| bjolo | we got 150 nodes in it | 10:07 |
| *** Trinh_ is now known as dangtrinhnt | 10:07 | |
| *** matrohon_ has quit IRC | 10:08 | |
| bjolo | pmisiak: about your hack | 10:08 |
| pmisiak | but kolla will recreaate only missing containers | 10:08 |
| pmisiak | all others wont be touched | 10:08 |
| *** yangyape_ has quit IRC | 10:09 | |
| bjolo | so if i delete a container on a node, i can recreate it by replacing action=start | 10:09 |
| pmisiak | eys | 10:10 |
| pmisiak | yes | 10:10 |
| bjolo | but a deploy would do the same thing right? | 10:10 |
| pmisiak | you can also add tags to pin this to particular role | 10:10 |
| pmisiak | for example -e tags=neutron | 10:10 |
| bjolo | yes, but that is kinda broken i newton afair | 10:11 |
| pmisiak | sr, -t neutron | 10:11 |
| pmisiak | --tags neutron :) | 10:11 |
| bjolo | anytime i tried --limit or --tag it broke somewhere | 10:11 |
| bjolo | inventory and set_facts issues | 10:11 |
| *** danardelean has joined #openstack-kolla | 10:11 | |
| pmisiak | but for start it should work | 10:12 |
| bjolo | yes, but what is the difference between action=start and action=deploy in this use case | 10:12 |
| bjolo | deleted container on target node gets recreated and started | 10:12 |
| pmisiak | yes | 10:12 |
| bjolo | so .... difference? | 10:13 |
| pmisiak | look at this file: https://github.com/openstack/kolla/blob/stable/newton/ansible/roles/neutron/tasks/deploy.yml | 10:14 |
| pmisiak | its for deploy action | 10:14 |
| pmisiak | it includes start.yml | 10:14 |
| pmisiak | when you run action=start you will run only start.yml | 10:15 |
| pmisiak | no config.yml and all others actions | 10:15 |
| bjolo | yes deploy takes longer but all things equal, those steps should be green. i.e. nothing happens | 10:16 |
| bjolo | just a speed thing | 10:16 |
| pmisiak | but when you run deploy with --tags | 10:16 |
| pmisiak | then you can have issues with variables | 10:17 |
| pmisiak | its only a hack to be sure that other things wont be touched by kolla | 10:17 |
| bjolo | ok gotcha | 10:17 |
| pmisiak | no reconfiguration etc. | 10:18 |
| bjolo | so how do i run a "new" dhcp container on one node manually? | 10:18 |
| bjolo | new binary | 10:18 |
| bjolo | looking at this again https://github.com/openstack/kolla/blob/stable/newton/ansible/roles/neutron/tasks/start.yml | 10:19 |
| bjolo | anyway i can dump out the resulting docker run command from that? | 10:19 |
| *** kevzha01 has quit IRC | 10:22 | |
| *** mewald has quit IRC | 10:25 | |
| bjolo | http://paste.openstack.org/show/623941/ | 10:26 |
| bjolo | just change the container name? | 10:26 |
| *** pbourke has quit IRC | 10:27 | |
| bjolo | stop and rm current neutron_dhcp_agent first of course | 10:28 |
| *** pbourke has joined #openstack-kolla | 10:28 | |
| bjolo | egonzalez: pbourke yey or nay :) | 10:29 |
| pbourke | bjolo: what are we talking about? | 10:29 |
| bjolo | we have issues with neutron on a kolla newton cloud | 10:30 |
| bjolo | believe it is a bugg that has been fixed so we have built new neutron_dhcp_agent container | 10:30 |
| bjolo | now i wanna try that container out manually on just one node | 10:31 |
| bjolo | simple idea is to stop and rm current container | 10:31 |
| bjolo | then docker run the new container | 10:31 |
| pbourke | ok | 10:31 |
| bjolo | but i need to have the exact start string for docker | 10:31 |
| pbourke | i have the perfect solution for you :) | 10:31 |
| pbourke | https://github.com/nexdrew/rekcod | 10:31 |
| bjolo | nice :) | 10:32 |
| bjolo | i found a tool called runlike | 10:32 |
| bjolo | http://paste.openstack.org/show/623941/ | 10:32 |
| pbourke | ah, same thing | 10:33 |
| pbourke | does it not work? | 10:33 |
| bjolo | gonna try rekcod and see if there is any difference in the output. if not -> verified | 10:33 |
| bjolo | have not tried yet | 10:33 |
| bjolo | wanted to bounce the idea here first | 10:33 |
| pbourke | ive used it often in the past for debugging | 10:34 |
| bjolo | i have another thing i like to discuss with you later thou :) | 10:34 |
| pbourke | sure | 10:34 |
| *** duonghq has quit IRC | 10:34 | |
| pbourke | btw, I think we really need your inventory-group-as-config-target bp | 10:34 |
| bjolo | its about moving the state into the container bootscripts instead of reconfigure/deploy | 10:34 |
| pbourke | inc0 indicated it was already possible but I dont think so | 10:35 |
| pbourke | which state? | 10:35 |
| bjolo | for ex neutron multiple external networks that you coded for newton release | 10:35 |
| bjolo | if i want to add a new external network, i need to run reconfigure to actually get the new bridge configured up | 10:36 |
| bjolo | correct? | 10:37 |
| bjolo | i add the interface and bridge to globals | 10:38 |
| bjolo | run reconfigure which actually goes into the container and run ovs-ovctl commands | 10:39 |
| *** egonzalez has quit IRC | 10:39 | |
| pbourke | yes | 10:39 |
| bjolo | could that code be moved or duped into the container startup sequence? | 10:40 |
| bjolo | im working on the new granular config workflow (mostly in my head) | 10:40 |
| bjolo | the genereal idea and unix wow is that a reboot should set the correct start | 10:41 |
| bjolo | any system is only binaries and config files right | 10:41 |
| pbourke | so you'd have to reboot to reconfig? | 10:41 |
| bjolo | booting the binary with the config file should set the correct state | 10:41 |
| bjolo | yes | 10:41 |
| bjolo | restart container in this case | 10:42 |
| pbourke | whats the benefit over just an ansible reconfigure | 10:42 |
| pbourke | also how do you get the new config files laid down | 10:42 |
| bjolo | granuality | 10:42 |
| bjolo | the workflow on high level | 10:42 |
| *** pratapagoutham has quit IRC | 10:43 | |
| bjolo | generate new config, push config to target nodes, restart containers that have new config | 10:43 |
| bjolo | simple | 10:43 |
| bjolo | no hidden steps or magic happens here | 10:43 |
| bjolo | also very git friendly to track changes | 10:44 |
| bjolo | now we have situation that new config file is in place but the state might not be correct | 10:44 |
| bjolo | i.e. new ovs bridge is not up | 10:45 |
| *** mrunge has quit IRC | 10:45 | |
| bjolo | to many magic happens here and config is instantiated with external commands leads to complex state machine | 10:46 |
| bjolo | approach above is very simple 1,2,3 approach | 10:47 |
| pbourke | im just not sure I see how moving into the container startup makes it any different | 10:47 |
| pbourke | sorry maybe its obvious :) | 10:47 |
| bjolo | for an admin it is. at least to me :) | 10:47 |
| pbourke | whether the steps happen in container start or ansible run, they're still the same steps | 10:47 |
| *** pratapagoutham has joined #openstack-kolla | 10:47 | |
| bjolo | yes, but the logic and workflow is simpler | 10:48 |
| bjolo | 1,2,3 | 10:48 |
| bjolo | if i restart a container, it runs according to its config | 10:48 |
| bjolo | no mismatch | 10:48 |
| *** mrunge has joined #openstack-kolla | 10:48 | |
| pbourke | if I re-run ansible, it runs according to its config | 10:49 |
| pbourke | same thing :p | 10:49 |
| bjolo | reconfigure as it is today is a dead end | 10:49 |
| bjolo | no admin will ever use it | 10:49 |
| pbourke | why | 10:49 |
| bjolo | read my BP about it | 10:49 |
| bjolo | tries todo too much | 10:50 |
| bjolo | to many places shit can go wrong | 10:50 |
| bjolo | i have no way to verify or inspect the new config before it is pushed out | 10:50 |
| bjolo | admins wants granularity | 10:50 |
| bjolo | multistep | 10:51 |
| bjolo | take my use case above | 10:51 |
| bjolo | yes, i should try this in a lab i know :) | 10:51 |
| bjolo | i want to try a new container on one node just to se if that fix the issue | 10:51 |
| bjolo | i dont want to run reconfigure on the whole damn cloud for that | 10:52 |
| *** caowei has joined #openstack-kolla | 10:52 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Move to zuul v3 in project jobs https://review.openstack.org/508759 | 10:52 |
| bjolo | every change that goes into a production cloud is done very incremental and controlled | 10:52 |
| bjolo | downtime costs millions per hour | 10:53 |
| pbourke | ok the bp makes a little more sense | 10:53 |
| pbourke | I dont think Id seen this one | 10:53 |
| bjolo | so in liue with the bp | 10:53 |
| pbourke | yeah so it seems we want to make the reconfig process more granular | 10:54 |
| pbourke | more than just moving where things happen | 10:54 |
| bjolo | 1,2,3 approach to reconfigure where the container start set the state | 10:54 |
| pbourke | that would probably be fine | 10:54 |
| bjolo | im actually rewriting our internal playbooks in this fashion atm | 10:55 |
| bjolo | all file copy and templating of config files dumps them out locally first | 10:55 |
| bjolo | i.e. the final files are dumped out locally | 10:55 |
| bjolo | then we get verify and review what was changed and check into git | 10:55 |
| pbourke | sounds nice | 10:56 |
| bjolo | then we push the files and restart services as needed | 10:56 |
| bjolo | same 1,2,3 logic | 10:56 |
| pbourke | makes sense | 10:56 |
| hrw | Does someone know how to build image used to setup Docker registry? It is available only for x86-64 architecture ;( | 10:57 |
| pbourke | bjolo: I think if you could prototype one service in kolla to follow that pattern it would be very useful for people to see | 10:57 |
| bjolo | i gotta go for lunch and workout | 10:57 |
| bjolo | pbourke: i will give it a try | 10:57 |
| pbourke | bjolo: cool thanks for explaining | 10:58 |
| bjolo | i can otherwise demo a normal OS file | 10:58 |
| bjolo | the code i work on right now is actually just /etc/resolv.conf | 10:58 |
| bjolo | for the os | 10:58 |
| openstackgerrit | zhangfei gao proposed openstack/kolla master: ironic-pxe: generate aarch64 bootfile grubaa64.efi https://review.openstack.org/512540 | 10:59 |
| hrw | pbourke: do you know who I should ask to mark https://review.openstack.org/#/c/423239/ (add support for ppc64le arch) as abandoned? it was done in whole set of my patches. | 10:59 |
| bjolo | have a template file with some jinja and variables. different hosts sets different variables | 10:59 |
| pbourke | hrw: can you not abandon yourself? | 10:59 |
| hrw | pbourke: not a core dev | 10:59 |
| pbourke | ah I thought you were the author | 11:00 |
| bjolo | resulting file is dumped out in a structure git/<hostname>/etc/resolv.conf | 11:00 |
| hrw | pbourke: it is not mine and original author vanished | 11:00 |
| pbourke | hrw: I can do it | 11:00 |
| hrw | pbourke: thanks! | 11:00 |
| pbourke | hrw: ping me any others if needed | 11:01 |
| hrw | pbourke: thanks. that was the only one | 11:02 |
| *** gfidente has joined #openstack-kolla | 11:03 | |
| *** gfidente has quit IRC | 11:03 | |
| *** gfidente has joined #openstack-kolla | 11:03 | |
| *** danardelean has quit IRC | 11:04 | |
| *** danardelean has joined #openstack-kolla | 11:07 | |
| *** jrist has joined #openstack-kolla | 11:09 | |
| *** zhangfei has quit IRC | 11:12 | |
| *** huanxie has quit IRC | 11:13 | |
| *** danardelean has quit IRC | 11:20 | |
| *** danardelean has joined #openstack-kolla | 11:22 | |
| *** rwallner has joined #openstack-kolla | 11:27 | |
| *** danardelean has quit IRC | 11:29 | |
| *** caowei has quit IRC | 11:31 | |
| *** dave-mccowan has joined #openstack-kolla | 11:31 | |
| *** danardelean has joined #openstack-kolla | 11:32 | |
| *** dave-mcc_ has joined #openstack-kolla | 11:35 | |
| *** danardelean has quit IRC | 11:36 | |
| *** dave-mccowan has quit IRC | 11:37 | |
| openstackgerrit | Martin André proposed openstack/kolla master: Moving jobs to kolla repo https://review.openstack.org/508661 | 11:37 |
| *** b_bezak has quit IRC | 11:38 | |
| *** danardelean has joined #openstack-kolla | 11:40 | |
| *** danardelean has quit IRC | 11:40 | |
| *** danardelean has joined #openstack-kolla | 11:40 | |
| *** danardelean has quit IRC | 11:41 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Update the manila shares link https://review.openstack.org/513004 | 11:41 |
| *** danardelean has joined #openstack-kolla | 11:43 | |
| *** yangyapeng has joined #openstack-kolla | 11:45 | |
| *** huanxie has joined #openstack-kolla | 11:46 | |
| *** danardelean has quit IRC | 11:46 | |
| *** ansmith has quit IRC | 11:46 | |
| *** danardelean has joined #openstack-kolla | 11:46 | |
| *** solomon_ezhra has joined #openstack-kolla | 11:51 | |
| *** solomon_ezhra has quit IRC | 12:00 | |
| *** Pavo has joined #openstack-kolla | 12:02 | |
| *** dangtrinhnt has quit IRC | 12:11 | |
| *** huanxie has quit IRC | 12:16 | |
| openstackgerrit | Daniel Alvarez proposed openstack/kolla master: Add Dockerfile for networking-ovn-metadata-agent https://review.openstack.org/511225 | 12:17 |
| *** goldyfruit has joined #openstack-kolla | 12:18 | |
| openstackgerrit | Daniel Alvarez proposed openstack/kolla master: Add Dockerfile for networking-ovn-metadata-agent https://review.openstack.org/511225 | 12:21 |
| *** goldyfruit has quit IRC | 12:22 | |
| *** dangtrinhnt has joined #openstack-kolla | 12:22 | |
| *** salv-orlando has quit IRC | 12:26 | |
| *** salv-orlando has joined #openstack-kolla | 12:28 | |
| *** dave-mcc_ is now known as dave-mccowan | 12:29 | |
| *** huanxie has joined #openstack-kolla | 12:30 | |
| *** gkadam has quit IRC | 12:34 | |
| *** nguyentrihai has joined #openstack-kolla | 12:37 | |
| *** ansmith has joined #openstack-kolla | 12:41 | |
| bjolo | pbourke: have you used rekcod on neutron_openvswitch_agent containers? | 12:42 |
| bjolo | pbourke: runlike does not catch --net=host | 12:42 |
| bjolo | for ex | 12:42 |
| pbourke | bjolo: not sure, probably? | 12:42 |
| *** kbaegis has joined #openstack-kolla | 12:45 | |
| *** clayton has quit IRC | 12:45 | |
| *** mnasiadk1 has quit IRC | 12:47 | |
| *** clayton has joined #openstack-kolla | 12:49 | |
| *** rhallisey has quit IRC | 12:52 | |
| *** rhallisey has joined #openstack-kolla | 12:53 | |
| bjolo | rekcod does not seem to work either | 12:54 |
| bjolo | it catches net=host but the start string does not bring up the container properly | 12:54 |
| bjolo | openvswitch_vswitchd and neutron_openvswitch_agent have issues | 12:55 |
| bjolo | docker and permissions | 12:55 |
| *** shardy has quit IRC | 13:00 | |
| *** shardy has joined #openstack-kolla | 13:00 | |
| *** huanxie has quit IRC | 13:01 | |
| *** signed8bit has joined #openstack-kolla | 13:01 | |
| *** alanmeadows has quit IRC | 13:02 | |
| *** alanmeadows has joined #openstack-kolla | 13:02 | |
| *** goldyfruit has joined #openstack-kolla | 13:04 | |
| *** bastafidli has joined #openstack-kolla | 13:10 | |
| *** danardelean has quit IRC | 13:11 | |
| *** shardy has quit IRC | 13:11 | |
| pbourke | hmm | 13:12 |
| *** shardy has joined #openstack-kolla | 13:13 | |
| *** matrohon_ has joined #openstack-kolla | 13:20 | |
| *** danardelean has joined #openstack-kolla | 13:23 | |
| *** cah_link__ has quit IRC | 13:23 | |
| bjolo | to start with, the openvswitch kernel module does not get loaded | 13:23 |
| *** danardelean has quit IRC | 13:23 | |
| *** danardelean has joined #openstack-kolla | 13:23 | |
| pomac | bjolo, pbourke: and if we load the module, it will complain about no permisisons to attach to brigdes etc - so something is really broken... | 13:31 |
| pbourke | bjolo: pomac: in most cases the command rekcod gives should be what ansible is running | 13:37 |
| pbourke | what you can do is replace kolla_start with 'bash' | 13:37 |
| pbourke | jump in and try running the start scripts by hand | 13:37 |
| pbourke | that way you can have a closer look | 13:38 |
| *** bmace has quit IRC | 13:39 | |
| *** signed8bit has quit IRC | 13:42 | |
| bjolo | pbourke: | 13:44 |
| bjolo | will try | 13:44 |
| bjolo | good idea | 13:44 |
| bjolo | when is kolla_extend_start copied into the container? | 13:44 |
| *** signed8b_ has joined #openstack-kolla | 13:44 | |
| bjolo | i would assume insmod openvswitch happens there for ex | 13:45 |
| *** jtriley has joined #openstack-kolla | 13:47 | |
| pbourke | at build time | 13:47 |
| bjolo | really | 13:57 |
| bjolo | how does that work with kolla_kuberneetes for ex | 13:58 |
| bjolo | i presumed the containers were generic binaries. only preloaded with kolla_start | 13:58 |
| bjolo | then each orch tool injected their own "boot" scripts | 13:58 |
| *** hrw has quit IRC | 13:59 | |
| pbourke | that may be well how kolla-k8s does it | 14:00 |
| pbourke | im not sure | 14:00 |
| pbourke | but for standard ansible they're built in | 14:00 |
| *** hrw has joined #openstack-kolla | 14:00 | |
| bjolo | hmmm not sure i like that | 14:00 |
| pbourke | yeah there are plenty who agree | 14:00 |
| bjolo | boot scripts are config files | 14:00 |
| pbourke | SamYaple: ^ | 14:01 |
| bjolo | as we discussed previously | 14:01 |
| bjolo | 1,2,3 pattern. generate config. push config, restart containers | 14:01 |
| pbourke | im not opposed to kolla moving in this direction | 14:01 |
| bjolo | been toying around and with this pattern, all type of actions would be blazingly fast | 14:02 |
| bjolo | besides the initial genconfig which needs to inventory and set_facts for all nodes | 14:03 |
| bjolo | but even that would not take long | 14:03 |
| pbourke | I think the original idea was that containers would be fully imutable once created | 14:04 |
| pbourke | but that never really was practical | 14:04 |
| bjolo | yea, ive heard that story | 14:04 |
| bjolo | but define immutable? | 14:05 |
| bjolo | i would say what is wanted is trackable | 14:05 |
| bjolo | i.e. every change is visible and tracked. which in a sense is immutable | 14:05 |
| bjolo | the binary container is immutable by definition | 14:06 |
| bjolo | a cm workflow supported by git makes changes trackable | 14:06 |
| bjolo | for ex we have a tool called etckeeper installed on every server/vm | 14:07 |
| bjolo | basically puts whole /etc in git | 14:07 |
| *** aagate has joined #openstack-kolla | 14:08 | |
| bjolo | also triggered by yum/apt so the whole list of packages are also tracked as a text file | 14:08 |
| bjolo | automatic snapshots of /etc | 14:08 |
| bjolo | commits | 14:08 |
| bjolo | the way we operate kolla today is with genconfig and etckeeper | 14:09 |
| bjolo | never with reconfigure | 14:09 |
| bjolo | we generate the configs, then we look on the nodes what has changed with git status | 14:09 |
| *** mnasiadka has joined #openstack-kolla | 14:10 | |
| bjolo | sounds not scalable but it is not that bad actually | 14:10 |
| *** salv-orlando has quit IRC | 14:11 | |
| *** salv-orlando has joined #openstack-kolla | 14:11 | |
| bjolo | we only have a few node types right, and with tmux sync panes and ansible one can handle that pretty easy | 14:11 |
| *** huanxie has joined #openstack-kolla | 14:12 | |
| *** sambetts|afk is now known as sambetts | 14:14 | |
| *** salv-orlando has quit IRC | 14:16 | |
| *** janki has quit IRC | 14:25 | |
| *** jamesbenson has joined #openstack-kolla | 14:32 | |
| *** erlon has joined #openstack-kolla | 14:32 | |
| *** lukepatrick has joined #openstack-kolla | 14:36 | |
| *** david-lyle has joined #openstack-kolla | 14:38 | |
| *** huanxie has quit IRC | 14:42 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Move to zuul v3 in project jobs https://review.openstack.org/508759 | 14:44 |
| *** mnasiadka has quit IRC | 14:52 | |
| *** pmisiak has quit IRC | 14:54 | |
| *** duonghq has joined #openstack-kolla | 14:54 | |
| *** MarginHu has quit IRC | 15:03 | |
| *** mdnadeem has quit IRC | 15:03 | |
| duonghq | evening guys | 15:06 |
| *** coolsvap has quit IRC | 15:06 | |
| *** bastafidli has quit IRC | 15:07 | |
| *** bastafidli has joined #openstack-kolla | 15:09 | |
| *** lrensing has joined #openstack-kolla | 15:13 | |
| *** lpetrut has joined #openstack-kolla | 15:14 | |
| *** pcaruana has quit IRC | 15:19 | |
| *** hongbin has joined #openstack-kolla | 15:27 | |
| *** lvdombrkr has quit IRC | 15:36 | |
| inc0 | good morning | 15:42 |
| *** lrensing has quit IRC | 15:44 | |
| duonghq | morning inc0 | 15:44 |
| inc0 | Jeffrey4l: what was the issue with https://review.openstack.org/#/c/508759/ ? what are you working on? | 15:45 |
| *** lukepatr_ has joined #openstack-kolla | 15:50 | |
| *** lukepatrick has quit IRC | 15:53 | |
| kfox1111 | good morning. | 15:54 |
| *** vhosakot has joined #openstack-kolla | 15:55 | |
| *** danardelean has quit IRC | 15:55 | |
| chason | o/ | 15:55 |
| inc0 | meeting | 15:58 |
| Jeffrey4l | inc0, i am working on it. i made some refator. but upgrade rabbitmq failed right, no idea why. | 15:59 |
| inc0 | kk | 15:59 |
| inc0 | llet's talk about this on meeting | 15:59 |
| *** gkadam has joined #openstack-kolla | 16:02 | |
| *** britthouser has joined #openstack-kolla | 16:02 | |
| *** genek has joined #openstack-kolla | 16:09 | |
| *** lrensing has joined #openstack-kolla | 16:10 | |
| *** salv-orlando has joined #openstack-kolla | 16:12 | |
| *** mmehan has joined #openstack-kolla | 16:15 | |
| *** salv-orlando has quit IRC | 16:17 | |
| openstackgerrit | Martin André proposed openstack/kolla master: Add Apache packages to mistral-api container https://review.openstack.org/513089 | 16:21 |
| *** gkadam has quit IRC | 16:22 | |
| *** signed8b_ has quit IRC | 16:23 | |
| *** salv-orlando has joined #openstack-kolla | 16:25 | |
| *** ntpttr_laptop has joined #openstack-kolla | 16:26 | |
| *** ntpttr_laptop has quit IRC | 16:28 | |
| *** matrohon_ has quit IRC | 16:33 | |
| *** vhosakot has quit IRC | 16:34 | |
| *** vhosakot has joined #openstack-kolla | 16:35 | |
| *** danardelean has joined #openstack-kolla | 16:35 | |
| *** vhosakot has quit IRC | 16:39 | |
| *** danardelean has quit IRC | 16:40 | |
| *** vhosakot has joined #openstack-kolla | 16:40 | |
| SamYaple | bjolo: immutable as in config and everything was baked into the image. this was not praticle | 16:40 |
| SamYaple | bjolo: we evetually settled on config out of the container, scripts in the container | 16:40 |
| SamYaple | bjolo: i am personalyl in the camp that images are binary blobs and any starting scripts should be bound in at runtime, but some people disagree with taht | 16:41 |
| *** vhosakot has quit IRC | 16:46 | |
| *** vhosakot has joined #openstack-kolla | 16:46 | |
| openstackgerrit | Martin André proposed openstack/kolla master: Moving jobs to kolla repo https://review.openstack.org/508661 | 16:50 |
| *** mgoddard_ has quit IRC | 16:51 | |
| *** jamesbenson has quit IRC | 16:52 | |
| *** dougsz has quit IRC | 16:55 | |
| *** jmccarthy has joined #openstack-kolla | 16:58 | |
| jmccarthy | Trying to figure out about using horizon to view instance console (instance on xen compute) - the console.log works, but not the console, any ideas ? | 16:58 |
| Jeffrey4l | about https://review.openstack.org/508759 | 16:58 |
| Jeffrey4l | i made some refactor. now upgrade rabbitmq failed. ready no idea why. | 16:58 |
| Jeffrey4l | inc0, ^^ and patch 83 is the old one which is green | 16:59 |
| kolla-slack | <inc0> kk I'll work on this today | 16:59 |
| Jeffrey4l | thx. | 17:00 |
| *** jamesbenson has joined #openstack-kolla | 17:07 | |
| *** jamesbenson has quit IRC | 17:12 | |
| *** jamesbenson has joined #openstack-kolla | 17:12 | |
| *** jamesbenson has quit IRC | 17:14 | |
| *** jamesbenson has joined #openstack-kolla | 17:21 | |
| *** huanxie has joined #openstack-kolla | 17:22 | |
| *** jamesbenson has quit IRC | 17:22 | |
| duonghq | Jeffrey4l, can you review this? https://review.openstack.org/#/c/425446/ | 17:24 |
| Jeffrey4l | will review it tomorrow ;) | 17:25 |
| *** lpetrut has quit IRC | 17:25 | |
| *** jamesbenson has joined #openstack-kolla | 17:25 | |
| *** jamesbenson has quit IRC | 17:27 | |
| duonghq | thanks Jeffrey4l | 17:27 |
| *** duonghq has quit IRC | 17:28 | |
| *** jmccarthy has left #openstack-kolla | 17:28 | |
| Jeffrey4l | np | 17:28 |
| *** pbourke-home has joined #openstack-kolla | 17:30 | |
| *** vhosakot has quit IRC | 17:34 | |
| *** pbourke-home has quit IRC | 17:37 | |
| *** jamesbenson has joined #openstack-kolla | 17:37 | |
| *** jamesbenson has quit IRC | 17:38 | |
| *** jamesbenson has joined #openstack-kolla | 17:40 | |
| *** lukepatrick has joined #openstack-kolla | 17:41 | |
| *** jamesbenson has quit IRC | 17:42 | |
| *** lukepatr_ has quit IRC | 17:43 | |
| *** jamesbenson has joined #openstack-kolla | 17:46 | |
| *** sambetts is now known as sambetts|afk | 17:46 | |
| *** jamesbenson has quit IRC | 17:49 | |
| *** jamesbenson has joined #openstack-kolla | 17:49 | |
| *** britthouser has quit IRC | 17:55 | |
| *** jamesbenson has quit IRC | 17:56 | |
| *** mwynne has joined #openstack-kolla | 17:59 | |
| *** huanxie has quit IRC | 18:03 | |
| *** jamesbenson has joined #openstack-kolla | 18:06 | |
| openstackgerrit | Merged openstack/kolla-kubernetes master: Update the link for Big Tent Governance https://review.openstack.org/512982 | 18:09 |
| inc0 | kfox1111: around? | 18:18 |
| kolla-slack | <kfox1111> Kind of. What's up? | 18:18 |
| inc0 | I don't know if you've seen our meeting today, but at the end we have one action for you;) or us rather | 18:19 |
| *** lpetrut has joined #openstack-kolla | 18:19 | |
| inc0 | let's do few hrs meeting (hangouts? recording?) | 18:19 |
| inc0 | to walk through full installation of kolla-k8s | 18:19 |
| inc0 | and general q&a and training | 18:20 |
| inc0 | we need to create pipeline to get others to your level of understanding of kolla-k8s | 18:20 |
| inc0 | because I'm not even close for example | 18:20 |
| inc0 | and obviously you can't handle the thing yourself | 18:20 |
| inc0 | are you willing to do sth like this? later this week or next week? | 18:21 |
| kolla-slack | <kfox1111> +1 | 18:21 |
| inc0 | cool, what time would be best for you? | 18:21 |
| *** rwsu has quit IRC | 18:25 | |
| *** paramite has quit IRC | 18:25 | |
| *** newmember has joined #openstack-kolla | 18:30 | |
| *** bjolo_ has joined #openstack-kolla | 18:34 | |
| kolla-slack | <kfox1111> That is a darned good question. | 18:35 |
| rwellum | Plus debugging help would be great. It's a big question though: even k8s takes a lot of expertise. | 18:36 |
| bjolo | SamYaple: tnx for clarifying | 18:43 |
| *** matrohon_ has joined #openstack-kolla | 18:45 | |
| bjolo | SamYaple: how does kolla-k8s start its containers? i.e. i presume the ansible start scripts not always make sense in k8s | 18:59 |
| SamYaple | bjolo: kolla-k8s modifies the scripts in the kolla repo to work with both ansible and kubernetes. additionally they run scripts like https://github.com/openstack/kolla-kubernetes/blob/9ca5ae155c484f594dd26479c59dd53eb8381853/helm/microservice/nova-placement-deployment/templates/nova-placement.yaml#L26 | 19:03 |
| *** huanxie has joined #openstack-kolla | 19:19 | |
| inc0 | SamYaple: that;s a negative | 19:20 |
| inc0 | as it's stand today ansible code is forked and lives in kolla-k8s repo | 19:20 |
| inc0 | \ahh, start scripts actually are similar | 19:20 |
| inc0 | sorry just got online;) kolla_start is the same regardless of orch tool | 19:21 |
| *** dgedia has joined #openstack-kolla | 19:25 | |
| *** lukepatr_ has joined #openstack-kolla | 19:28 | |
| dgedia | I am trying to install openstack using kolla-ansible on a multi-node environment. However, I am facing an error when I deploy. Error is " API Error: 500 Server Error http: server gave HTTP response to HTTPS client". Am I missing any configuration? Any help would be really appreciated | 19:28 |
| inc0 | dgedia: what caused this error? | 19:29 |
| inc0 | which command and so on | 19:29 |
| *** lukepatrick has quit IRC | 19:30 | |
| *** salv-orlando has quit IRC | 19:31 | |
| dgedia | @inc0, I did "kolla-ansible deploy -i kolla-ansible/ansible/inventory/multinode" | 19:31 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Move to zuul v3 in project jobs https://review.openstack.org/508759 | 19:32 |
| inc0 | and full logs plz then:) | 19:32 |
| inc0 | or at least task which threw this error | 19:32 |
| *** salv-orlando has joined #openstack-kolla | 19:35 | |
| *** rwsu has joined #openstack-kolla | 19:37 | |
| *** dgedia_ has joined #openstack-kolla | 19:44 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode scenarios in zuulv3 https://review.openstack.org/512779 | 19:44 |
| dgedia_ | @inc0, Please find error details here : https://pastebin.com/BBG7ak1C | 19:45 |
| *** dgedia has quit IRC | 19:45 | |
| inc0 | dgedia_: it's multinode right? | 19:46 |
| dgedia_ | yup | 19:46 |
| inc0 | did you install docker manually or used bootstrap servers? | 19:46 |
| inc0 | also you are using local registry right? | 19:46 |
| *** csuttles_ has quit IRC | 19:47 | |
| *** huanxie has quit IRC | 19:50 | |
| dgedia_ | @inc0, installed docker manually. Yes, I am using local registry | 19:52 |
| inc0 | right, I bet you didn't add --insecure-registry :) | 19:52 |
| inc0 | https://docs.docker.com/registry/insecure/ <- make sure to configure insecure-registries pointing to node with your private registry on every docker service | 19:53 |
| inc0 | or, alternatively, if you can quickly cleanup your env | 19:54 |
| inc0 | you can just run kolla-ansible bootstrap-servers and this command will install docker, configure insecure registry and do all that | 19:54 |
| dgedia_ | I edited the /etc/default/docker with DOCKER_OPTS="--insecure-registry 192.168.1.100:5000" | 19:54 |
| inc0 | I think newer versions of docker doesn't use this file | 19:55 |
| inc0 | which version of docker you're running? | 19:55 |
| inc0 | and which base distro? | 19:55 |
| dgedia_ | Docker version 17.09.0-ce, build afdb6d4 | 19:57 |
| inc0 | right, so try method described in docker docs | 19:57 |
| inc0 | disclaimer - best tested version of docker is 1.12 | 19:57 |
| dgedia_ | as well as I use Ubuntu 16.04 | 19:57 |
| pomac | dgedia_: check your systemd unit file - it might use different configuration ways | 19:58 |
| pomac | dgedia_: (we had a similar issue with centos 7) | 19:58 |
| inc0 | pomac is right - systemd conf files is another way to do it | 19:58 |
| inc0 | to confirm if your docker registry works, run docker info | 19:58 |
| pomac | inc0: some of them move things around, but i actually think that this is the deamon.json or something like that file | 19:59 |
| inc0 | yeah https://docs.docker.com/registry/insecure/#deploy-a-plain-http-registry deamon.json is described here | 19:59 |
| pomac | so edit /etc/docker/daemon.json | 19:59 |
| inc0 | but I tink it's fairly new | 19:59 |
| pomac | inc0: thanks | 19:59 |
| pomac | inc0: it is - and it's a bit annoying when things chnage | 20:00 |
| *** lpetrut has quit IRC | 20:00 | |
| inc0 | that's one of reasons why docker 1.12 is best tested and we kinda stick to it | 20:00 |
| dgedia_ | docker info : https://pastebin.com/TQmMj89P | 20:00 |
| *** lpetrut has joined #openstack-kolla | 20:00 | |
| inc0 | oh so it does have insecure registry configured correctly | 20:01 |
| dgedia_ | I'll get docker 1.12 installed in the meanshile since it is best tested | 20:01 |
| inc0 | are you sure thats the case for all the nodes? | 20:01 |
| dgedia_ | Appreciate inc0 and pomac for promt suggestions :) | 20:01 |
| *** salv-orlando has quit IRC | 20:02 | |
| inc0 | no worries, let me know if you have further questions | 20:02 |
| *** salv-orlando has joined #openstack-kolla | 20:02 | |
| inc0 | also if you want to once and for all test if your local registry works well | 20:02 |
| inc0 | try docker pull 192.168.100.13:5000/ubuntu-source-base:5.0.0 | 20:03 |
| inc0 | (image name may differ depending on you distro/type/version of kolla) | 20:03 |
| *** ipsecguy_ has joined #openstack-kolla | 20:04 | |
| dgedia_ | how could I verify distro/type/version of kolla ? | 20:05 |
| *** gema has quit IRC | 20:06 | |
| *** salv-orlando has quit IRC | 20:07 | |
| *** ipsecguy has quit IRC | 20:08 | |
| *** gema has joined #openstack-kolla | 20:08 | |
| *** gema has quit IRC | 20:08 | |
| *** gema has joined #openstack-kolla | 20:08 | |
| *** dgedia_ has quit IRC | 20:14 | |
| *** ansmith has quit IRC | 20:17 | |
| *** openstackgerrit has quit IRC | 20:17 | |
| *** lpetrut has quit IRC | 20:17 | |
| *** salv-orlando has joined #openstack-kolla | 20:18 | |
| *** dgedia has joined #openstack-kolla | 20:31 | |
| *** rwallner has quit IRC | 20:31 | |
| *** rwallner has joined #openstack-kolla | 20:33 | |
| *** rwallner_ has joined #openstack-kolla | 20:34 | |
| *** kbaegis1 has joined #openstack-kolla | 20:34 | |
| *** kbaegis has quit IRC | 20:34 | |
| *** kbaegis1 has quit IRC | 20:36 | |
| *** rwallner has quit IRC | 20:37 | |
| *** rwallner_ has quit IRC | 20:38 | |
| *** kolla-slack has quit IRC | 20:39 | |
| *** kolla-slack has joined #openstack-kolla | 20:39 | |
| bjolo | speaking of docker 1.12 and baremetal | 20:41 |
| bjolo | on redhat/centos it is hardcoded to 1.12.0 | 20:42 |
| bjolo | but for debian it says docker-engine=1.12.* | 20:42 |
| SamYaple | bjolo: the version of docker on debian based distros is appended with additional stuff | 20:51 |
| SamYaple | easier to just do a splat | 20:52 |
| *** sbezverk has quit IRC | 20:54 | |
| bjolo | yea i get that, but dont we get 1.12.0 on RH and 1.12.6 on debian now? | 20:57 |
| *** matrohon_ has quit IRC | 20:58 | |
| SamYaple | bjolo: i use 17.09 and im pretty sure kolla gates with 17.09 | 20:58 |
| SamYaple | so i dont think it matters that much | 20:58 |
| bjolo | we have had some issues with different docker version mismatch in our env | 20:59 |
| bjolo | compute nodes installed at different times got different versions of docker | 20:59 |
| *** jtriley has quit IRC | 20:59 | |
| bjolo | actually now rewriting our playbook and hardcode it to 1.12.0 since that is what is probably most tested | 21:00 |
| SamYaple | bjolo: i doubt that personally | 21:00 |
| SamYaple | again, the gates dont use that | 21:00 |
| bjolo | we have two clouds in prod. cloud-1 is newtonw and various docker 1.12 and 1.13 | 21:02 |
| bjolo | cloud-2 is ocata and docker 17.05 | 21:03 |
| bjolo | on cloud-2 we can not do a docker restart libvirt. well we can, but after that we lose contacts with all running vms | 21:04 |
| bjolo | everything else equals | 21:04 |
| bjolo | cloud-1 does not have this issue | 21:04 |
| SamYaple | bjolo: you need to enable live-restore | 21:04 |
| bjolo | done that | 21:04 |
| SamYaple | hmmmmmm | 21:04 |
| SamYaple | are the qmeu processes dying? | 21:05 |
| SamYaple | or just libvirt cant reconnect | 21:05 |
| inc0 | ps aux on host shows vms running? | 21:06 |
| bjolo | http://paste.openstack.org/show/624026/ | 21:06 |
| bjolo | we lose the sockets to the vms | 21:07 |
| bjolo | libvirt can communicate | 21:07 |
| SamYaple | bjolo: oh i believe you when you say things, you dont have to post proof | 21:07 |
| bjolo | pomac: has the details | 21:07 |
| bjolo | i know. did that to get fresh eyes | 21:07 |
| bjolo | maybe somehing wrong | 21:08 |
| SamYaple | that looks fine, but the lose the sockets to the vm is strange... | 21:08 |
| SamYaple | would need more details about that | 21:08 |
| pomac | Yep, thats what happens to us | 21:08 |
| pomac | It's only when libvirt is restarted, it's like when the filesystem is remounted the actual socket quality is dropped (and they become dumb, unconnected files again) | 21:09 |
| SamYaple | pomac: sounds like an issue with shared mounts | 21:10 |
| bjolo | the other thing observered is that the qemu-kvm processes are moved out from libvirt container | 21:10 |
| SamYaple | can either of you post your /etc/systemd/system/docker* stuff? | 21:10 |
| pomac | SamYaple: yes | 21:10 |
| SamYaple | bjolo: thats libvirt moving them to another cgroup | 21:10 |
| pomac | bjolo: I'm not on vpn or anything | 21:10 |
| SamYaple | coincidentally, thats the only reason vms survive a libvirt restart | 21:10 |
| bjolo | ok | 21:10 |
| SamYaple | non-kvm vms dont survive libvirt container restart | 21:11 |
| *** bastafidli has quit IRC | 21:11 | |
| pomac | If the shared mount is actually a mount, then the sockets shouldn't survive the remount that happens - if the mount was kept - it would be a differnet storry | 21:12 |
| pomac | story | 21:12 |
| SamYaple | pomac: the socket lives in /var/lib/nova/* which is a volume so it should be shared | 21:12 |
| pomac | SamYaple: unfortunatelty it seems to be mounted by docker, so you have two VFS layers - thus the space where the socket lived is gone on the restart | 21:13 |
| inc0 | also it works on 1.12...docker has nasty habit of breaking things so 17.05 - no diea if they broke anything | 21:13 |
| bjolo | http://paste.openstack.org/show/624029/ | 21:14 |
| pomac | inc0: we upgraded due to other odd problems with btrfs | 21:14 |
| SamYaple | bjolo: i belivee the way you have that configured means live-restore is not actually taking place because daemon.json isn't being read | 21:14 |
| bjolo | well it picks up insecure registry somehow | 21:15 |
| bjolo | http://paste.openstack.org/show/624030/ | 21:15 |
| SamYaple | ah then it must default to reading it | 21:16 |
| *** salv-orlando has quit IRC | 21:16 | |
| SamYaple | for future reference, *all* daemon options can go in daemon.json and you can leave the Exec part empty | 21:16 |
| SamYaple | back to the issue at hand | 21:16 |
| SamYaple | i would have to duplicate this, but it sounds like it shouldn't be hard... | 21:16 |
| SamYaple | can i get your host os | 21:16 |
| pomac | SamYaple: centos 7 | 21:17 |
| pomac | SamYaple: with some more recent kernels - in some ways (due to rcu issues with namespaces) | 21:18 |
| *** salv-orlando has joined #openstack-kolla | 21:18 | |
| SamYaple | backing filesystem | 21:18 |
| pomac | SamYaple: (oh and centos frankenkernel is quite unknown) | 21:18 |
| bjolo | btrfs | 21:18 |
| pomac | SamYaple: btrfs | 21:18 |
| SamYaple | oh. well thats the issue | 21:18 |
| bjolo | elrepo kernel | 21:18 |
| pomac | SamYaple: since it actually "mounts" the filesystem | 21:18 |
| SamYaple | they changed the way the driver works | 21:18 |
| SamYaple | it does some weird stuff, yea | 21:19 |
| SamYaple | overlayfs[2] (ext4 only i believe) works better | 21:19 |
| SamYaple | its been a while since i played with that. i can test it though | 21:19 |
| SamYaple | give me a few | 21:19 |
| SamYaple | i can probably get you a docker commit id | 21:19 |
| *** salv-orlando has quit IRC | 21:21 | |
| pomac | SamYaple: it's annoying when it doesn't work the same for all things - it sounds like a implementation fault really... | 21:21 |
| pomac | Also, using socket files like that is a bit odd... =) | 21:21 |
| SamYaple | pomac: to be clear here, i belive teh issue is docker changed a thing and centos btrfs is old and dumb (regardless of the backport) | 21:22 |
| SamYaple | im going to verify it happens/doesnt happen on ubuntu first | 21:22 |
| bjolo | cool | 21:22 |
| bjolo | tnx SamYaple | 21:22 |
| *** salv-orlando has joined #openstack-kolla | 21:23 | |
| SamYaple | if at all possible, consider using overlayfs2 as teh backend though, save you lots of headache | 21:23 |
| pomac | SamYaple: we run a 4.9 mainline kernel atm | 21:23 |
| bjolo | SamYaple: you must have a running vm before the libvirt restart | 21:26 |
| bjolo | pomac: refresh my mind. we could migrate a "lost" vm to another node right | 21:26 |
| pomac | SamYaple: can one run one backend for one container and others for others? That would be one solution | 21:27 |
| *** newmember has quit IRC | 21:27 | |
| SamYaple | bjolo: im going to reproduce it from a pure socket breakage for simplicity | 21:27 |
| pomac | bjolo: I think so | 21:27 |
| SamYaple | outside of kolla | 21:27 |
| bjolo | ok | 21:27 |
| *** newmember has joined #openstack-kolla | 21:27 | |
| bjolo | once migrated it was back online again inside the libvirt container | 21:28 |
| bjolo | we noticed this after we gone live so we ended up migrate around all vms on the cloud :) | 21:28 |
| SamYaple | nice... | 21:29 |
| bjolo | only about 100 so not too bad | 21:29 |
| pomac | the interesting part is that what happens is that libvirt can't START new kvm instances | 21:29 |
| pomac | until you have migrated all old ones out of the way | 21:29 |
| bjolo | and then restart libvirt again right | 21:29 |
| pomac | <- tired | 21:29 |
| dgedia | @inc0, I successfully bootstrapped, but while doing kolla deploy, I am getting an error "The requested image does not exist: 192.168.100.13:5000/kolla/ubuntu-source-fluentd:5.0.0" | 21:29 |
| bjolo | me too | 21:29 |
| pomac | bjolo: my alarm clock is set for 6:30 - ;) | 21:30 |
| bjolo | tomorrow it is openstack nordic days in copenhagen denmark | 21:30 |
| bjolo | a bunch of us are going | 21:30 |
| bjolo | SamYaple: really tnx for testing out. i have to drop out for tonight. 1130PM mytime | 21:31 |
| pomac | hummm.. need to restart to get the damned irssi notifier a chance of actually working... | 21:31 |
| pomac | brb | 21:32 |
| *** pomac has quit IRC | 21:32 | |
| inc0 | dgedia: did you do kolla-build with --push? | 21:32 |
| dgedia | haven't build it | 21:33 |
| dgedia | how should I approach to build ? | 21:33 |
| *** pomac has joined #openstack-kolla | 21:34 | |
| dgedia | kolla-build --registry 192.168.100.13:5000 --push -b ubuntu -t binary | 21:35 |
| dgedia | is that the right way to do? | 21:35 |
| pomac | SamYaple: if you use my nick, i shold hopefully get proper notifications from irssinotifier now | 21:35 |
| pomac | And it works too - basically, i can provide bjolo with info tomorrow if you want to relay something | 21:36 |
| bjolo | goodnight humanoids | 21:37 |
| pomac | night bjolo, see you in a few hours | 21:37 |
| bjolo | yup | 21:37 |
| dgedia | inc0 ? | 21:39 |
| bjolo | dgedia: that should work | 21:39 |
| inc0 | dgedia: yeah, but don't use ubuntu binary | 21:39 |
| inc0 | use ubuntu source | 21:39 |
| inc0 | its much more stable | 21:39 |
| dgedia | cool :) | 21:39 |
| inc0 | and make sure globals.yml has same configs | 21:40 |
| *** paramite has joined #openstack-kolla | 21:44 | |
| *** jamesbenson has quit IRC | 21:49 | |
| *** jamesbenson has joined #openstack-kolla | 21:52 | |
| dgedia | thanks inc0 for your suggestion, appreciate it :) | 21:55 |
| inc0 | no worries, I'm working as we speak to provide images in dockerhub so it should be much easier later on | 21:56 |
| *** zhubingbing_ has joined #openstack-kolla | 21:56 | |
| *** dardelean_ has joined #openstack-kolla | 21:56 | |
| *** jamesbenson has quit IRC | 21:56 | |
| dgedia | That would be really great! | 21:56 |
| *** zhubingbing has quit IRC | 21:58 | |
| *** dardelean has quit IRC | 22:00 | |
| dgedia | @inc0, there isn't any issue right if I build binary and source images both (However specify only source in globals.yml) ? I happened to build binary earlier. | 22:02 |
| *** MarginHu has joined #openstack-kolla | 22:05 | |
| *** rwallner has joined #openstack-kolla | 22:05 | |
| *** rwsu has quit IRC | 22:05 | |
| *** ipsecguy_ has quit IRC | 22:06 | |
| *** dardelean__ has joined #openstack-kolla | 22:08 | |
| *** rwallner has quit IRC | 22:09 | |
| *** dardelean_ has quit IRC | 22:11 | |
| *** dave-mccowan has quit IRC | 22:11 | |
| *** rhallisey has quit IRC | 22:11 | |
| inc0 | dgedia: no, only thing abut ubuntu binary is that it was breaking more often than source | 22:18 |
| inc0 | but you deploy what you specify in globals | 22:18 |
| *** brad[] has quit IRC | 22:23 | |
| *** brad[] has joined #openstack-kolla | 22:23 | |
| *** ipsecguy has joined #openstack-kolla | 22:29 | |
| *** lukepatr_ has quit IRC | 22:29 | |
| dgedia | inc0, sounds good. How long does it take the build to complete? | 22:33 |
| *** salv-orlando has quit IRC | 22:33 | |
| inc0 | depends on stor/net | 22:33 |
| *** bjolo_ has quit IRC | 22:33 | |
| inc0 | on ssd+good net it was ~20min | 22:33 |
| *** ansmith has joined #openstack-kolla | 22:35 | |
| dgedia | inc0 how should I go about with building images that failed with kolla-build? | 22:42 |
| dgedia | I have a bunch of images that failed in build process | 22:43 |
| *** rwallner has joined #openstack-kolla | 22:44 | |
| *** rwallner has quit IRC | 22:44 | |
| dgedia | inc0, failed images: https://pastebin.com/4LW1CtaT | 22:45 |
| *** rwallner has joined #openstack-kolla | 22:45 | |
| *** rwallner has quit IRC | 22:48 | |
| *** rwsu has joined #openstack-kolla | 23:01 | |
| inc0 | dgedia: error will be somewhere above | 23:02 |
| *** dave-mccowan has joined #openstack-kolla | 23:06 | |
| *** lrensing has quit IRC | 23:07 | |
| *** hongbin has quit IRC | 23:21 | |
| *** salv-orlando has joined #openstack-kolla | 23:34 | |
| *** salv-orlando has quit IRC | 23:39 | |
| *** ekimshi has quit IRC | 23:44 | |
| *** goldyfruit has quit IRC | 23:46 | |
| *** dardelean__ has quit IRC | 23:48 | |
| *** danardelean has joined #openstack-kolla | 23:50 | |
| *** dgedia has quit IRC | 23:52 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!