Wednesday, 2017-10-18

*** genek has quit IRC00:00
*** mnasiadka has quit IRC00:01
*** mnasiadka has joined #openstack-kolla00:02
*** danardelean has quit IRC00:04
*** bastafidli has quit IRC00:06
*** mnasiadka has quit IRC00:07
*** MarginHu has joined #openstack-kolla00:10
*** mnasiadka has joined #openstack-kolla00:12
*** vhosakot has quit IRC00:14
*** mnasiadka has quit IRC00:18
*** yingjun has joined #openstack-kolla00:19
*** mnasiadka has joined #openstack-kolla00:22
*** mnasiadka has quit IRC00:27
-openstackstatus- NOTICE: due to unscheduled restart of zuulv3.o.o you will need to 'recheck' your jobs that were last running. Sorry for the inconvenience.00:32
*** mnasiadka has joined #openstack-kolla00:33
*** xinliang has quit IRC00:37
*** genek has joined #openstack-kolla00:38
*** mnasiadka has quit IRC00:39
*** tovin07_ has joined #openstack-kolla00:40
*** mnasiadka has joined #openstack-kolla00:43
*** genek has quit IRC00:44
*** mnasiadka has quit IRC00:47
*** salv-orlando has joined #openstack-kolla00:48
*** goldyfruit1 has joined #openstack-kolla00:48
*** goldyfruit has quit IRC00:48
*** xinliang has joined #openstack-kolla00:50
*** xinliang has joined #openstack-kolla00:50
*** yangyapeng has quit IRC00:51
*** yangyapeng has joined #openstack-kolla00:51
*** salv-orlando has quit IRC00:52
*** mnasiadka has joined #openstack-kolla00:53
*** yangyapeng has quit IRC00:56
*** mnasiadka has quit IRC00:57
*** huanxie has joined #openstack-kolla01:05
openstackgerritHeetae Ahn proposed openstack/kolla master: Add RHEL subscription registration  https://review.openstack.org/49514801:18
Sean-mdhi all, when I run "helm install ..cinder-control..." to deploy kolla-kubernetes01:21
*** kevzha01 has joined #openstack-kolla01:22
Sean-md"/usr/local/bin/dumb-init /bin/bash /usr/local/bin/kolla_start" is executed, before running "kolla-start" by using the command of cinder-api pod01:22
lxdonghow to join openstack-kolla slack?01:22
Sean-mdSo, cinder-api pod is crashing, because port 8776 is reused.01:24
Sean-mdlxdong, openstack-kolla channel is in kubernetes(kubernetes.slack.com)01:26
lxdongok01:30
Sean-mdplease let me know how not to run "/usr/local/bin/dumb-init /bin/bash /usr/local/bin/kolla_start" when deploying cinder-control using helm01:31
kolla-slack<lxdong.007> nice01:32
*** naichuans has quit IRC01:34
*** genek has joined #openstack-kolla01:34
kolla-slack<lxdong.007> ```bash /usr/libexec/qemu-kvm -name guest=instance-00000034,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-2-instance-00000034/master-key.aes -machine pc-i440fx-rhel7.4.0,accel=tcg,usb=off,dump-guest-core=off -cpu01:38
kolla-slackOpteron_G4,acpi=on,ss=on,monitor=on,movbe=on,hypervisor=on,arat=on,fsgsbase=on,bmi1=on,smep=on,bmi2=on,erms=on,mpx=on,adx=on,smap=on,clflushopt=on,pku=on,ospke=on,xsaveopt=on,xgetbv1=on,mmxext=on,3dnowext=on,3dnow=on,cr8legacy=on,avx=off,misalignsse=off,3dnowprefetch=off,xop=off,fma4=off -m 4096 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid a5ad08ca-e0eb-4c77-abea-04291743ad5c -smbios type=1,manufacturer=OpenStack01:38
kolla-slackFoundation,product=OpenStack Nova,version=16.0.0,serial=4c4c4544-004e-3210-8048-c4c04f5a3258,uuid=a5ad08ca-e0eb-4c77-abea-04291743ad5c,family=Virtual Machine -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-2-instance-00000034/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -de01:38
kolla-slackvirtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -object secret,id=virtio-disk0-secret0,data=eMczfVZF7mk1iEmpRigZS1F6p3n3hTtbwjGS3PosENU=,keyid=masterKey0,iv=ZEcx6nyM1EYm8S6h0bPJ7g==,format=base64 -drive01:38
kolla-slackfile=rbd:volumes/volume-fea9b99e-686d-495f-bcf0-9afe9ed151c3:id=cinder:auth_supported=cephx\;none:mon_host=192.168.4.150\:6789\;192.168.4.151\:6789,file.password-secret=virtio-disk0-secret0,format=raw,if=none,id=drive-virtio-disk0,serial=fea9b99e-686d-495f-bcf0-9afe9ed151c3,cache=writeback,discard=unmap -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=26,id=hostnet0 -device01:38
kolla-slackvirtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:a4:33:76,bus=pci.0,addr=0x3 -chardev pty,id=charserial0,logfile=/var/lib/nova/instances/a5ad08ca-e0eb-4c77-abea-04291743ad5c/console.log,logappend=off -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -chardev01:38
kolla-slacksocket,id=charchannel1,path=/var/lib/libvirt/qemu/org.qemu.guest_agent.0.instance-00000034.sock,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -spice port=5900,addr=192.168.4.150,disable-ticketing,seamless-migration=on -k en-us -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device01:38
kolla-slackvirtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on ```01:38
*** genek has quit IRC01:39
kolla-slack<lxdong.007> It's an issue about windows instance mouse drifting .I have use_usb_tablet = True and pointer_model = usbtablet in nova.conf with spice enabled and [spice].agent_enabled = True , instance still created with usb=off and doesn't take effect.01:41
*** yangyapeng has joined #openstack-kolla01:48
*** huzhengchuan has joined #openstack-kolla01:48
*** salv-orlando has joined #openstack-kolla01:48
*** yangyape_ has joined #openstack-kolla01:49
*** yangyapeng has quit IRC01:52
*** genek has joined #openstack-kolla01:53
*** salv-orlando has quit IRC01:53
*** jtriley has joined #openstack-kolla01:53
*** lukepatrick has joined #openstack-kolla01:55
*** genek has quit IRC01:58
*** genek has joined #openstack-kolla01:59
*** genek has quit IRC02:03
*** zhubingbing has joined #openstack-kolla02:03
*** huanxie has quit IRC02:09
*** lxdong has quit IRC02:10
*** huanxie has joined #openstack-kolla02:10
*** lukepatrick has quit IRC02:12
*** lukepatrick has joined #openstack-kolla02:14
*** genek has joined #openstack-kolla02:17
*** jtriley has quit IRC02:18
*** jtriley has joined #openstack-kolla02:27
*** pramodrj07 has quit IRC02:29
*** MasterOfBugs has quit IRC02:29
*** huanxie has quit IRC02:42
*** dave-mccowan has quit IRC02:44
*** duonghq has quit IRC02:47
*** hieulq has quit IRC02:47
*** duonghq has joined #openstack-kolla02:48
*** hieulq has joined #openstack-kolla02:48
*** dave-mccowan has joined #openstack-kolla02:48
*** zhangfei has joined #openstack-kolla02:49
*** salv-orlando has joined #openstack-kolla02:50
*** unicell has joined #openstack-kolla02:54
*** salv-orlando has quit IRC02:54
kolla-slack<lxdong.007> Did anyone meet the issue about mouse drifting of windows instance runing in kvm?02:55
*** hieulq has quit IRC02:56
*** duonghq has quit IRC02:56
*** duonghq has joined #openstack-kolla02:57
*** hieulq has joined #openstack-kolla02:57
*** unicell has quit IRC02:58
*** lukepatrick has quit IRC03:00
*** lukepatrick has joined #openstack-kolla03:01
*** unicell has joined #openstack-kolla03:06
*** jtriley has quit IRC03:06
*** huzhengchuan has quit IRC03:09
*** unicell has quit IRC03:10
*** openstackgerrit has quit IRC03:22
*** jtriley has joined #openstack-kolla03:23
*** huanxie has joined #openstack-kolla03:27
*** unicell has joined #openstack-kolla03:30
*** goldyfruit1 has quit IRC03:31
*** pratapagoutham has quit IRC03:31
*** lukepatrick has quit IRC03:31
*** lukepatrick has joined #openstack-kolla03:32
*** unicell has quit IRC03:34
*** mdnadeem has joined #openstack-kolla03:36
*** jtriley has quit IRC03:42
*** genek has quit IRC03:42
*** unicell has joined #openstack-kolla03:43
*** unicell has quit IRC03:47
*** salv-orlando has joined #openstack-kolla03:50
duonghqanybody see gerrit is getting trouble?03:52
*** unicell has joined #openstack-kolla03:54
*** lukepatrick has quit IRC03:54
*** jtriley has joined #openstack-kolla03:55
*** gkadam has joined #openstack-kolla03:55
*** salv-orlando has quit IRC03:55
*** kiennt26 has quit IRC03:57
*** genek has joined #openstack-kolla03:57
*** kiennt26 has joined #openstack-kolla03:57
*** huanxie has quit IRC03:58
*** unicell has quit IRC03:59
*** genek has quit IRC04:01
*** Jeffrey4l has quit IRC04:02
*** Jeffrey4l has joined #openstack-kolla04:03
*** unicell has joined #openstack-kolla04:07
*** genek has joined #openstack-kolla04:11
*** unicell has quit IRC04:11
zhubingbingduonghq04:12
zhubingbingyeah04:13
zhubingbingI can't open gerrit04:13
duonghqzhubingbing, ack04:14
*** genek has quit IRC04:15
*** unicell has joined #openstack-kolla04:19
*** genek has joined #openstack-kolla04:20
*** unicell has quit IRC04:23
*** huanxie has joined #openstack-kolla04:26
*** Pavo has quit IRC04:27
*** jinke has joined #openstack-kolla04:29
*** unicell has joined #openstack-kolla04:31
*** jinke has left #openstack-kolla04:32
*** jinke has joined #openstack-kolla04:34
*** unicell has quit IRC04:36
*** janki has joined #openstack-kolla04:36
*** unicell has joined #openstack-kolla04:39
*** zhangfei has quit IRC04:42
*** kiennt26 has quit IRC04:47
*** dave-mccowan has quit IRC04:47
*** signed8bit has joined #openstack-kolla04:50
*** unicell1 has joined #openstack-kolla04:50
*** salv-orlando has joined #openstack-kolla04:51
*** fguillot has joined #openstack-kolla04:51
*** unicell has quit IRC04:52
*** fguillot has left #openstack-kolla04:52
*** salv-orlando has quit IRC04:57
*** zhangfei has joined #openstack-kolla04:58
*** unicell has joined #openstack-kolla04:58
*** unicell1 has quit IRC04:59
*** genek has quit IRC05:00
*** genek has joined #openstack-kolla05:00
masbergood afternoon, I was wondering if someone could help me troubleshooting kolla-ansible deployment? I am getting this error http://paste.openstack.org/raw/623913/05:01
masberI am really desperate, can't understand what is the issue05:04
*** huanxie has quit IRC05:05
*** genek has quit IRC05:05
*** hyakuhei has quit IRC05:05
*** coolsvap has joined #openstack-kolla05:07
*** huanxie has joined #openstack-kolla05:07
*** jtriley has quit IRC05:08
*** genek has joined #openstack-kolla05:13
*** salv-orlando has joined #openstack-kolla05:15
*** genek has quit IRC05:17
*** shardy has quit IRC05:23
*** unicell1 has joined #openstack-kolla05:24
*** shardy has joined #openstack-kolla05:25
*** janki has quit IRC05:25
*** unicell has quit IRC05:27
*** unicell2 has joined #openstack-kolla05:27
*** unicell1 has quit IRC05:29
*** unicell has joined #openstack-kolla05:30
*** unicell2 has quit IRC05:31
*** unicell1 has joined #openstack-kolla05:33
*** unicell has quit IRC05:34
inc0masber: check docker logs nova-compute on test-openstack-controller05:38
inc0also /var/lib/docker/volmes/kolla_logs/_data/nova05:38
inc0Jeffrey4l: around?05:38
Jeffrey4linc0, yep05:38
inc0https://review.openstack.org/#/c/508759/ <- this seems ready05:38
inc0https://review.openstack.org/#/c/512779/ <- I started to fiddle around multinode05:39
Jeffrey4lyes. should we add other binary jobs?05:39
inc0no05:40
inc0I think we shouldn't have binary jobs in ansible05:40
inc0no difference in deployment logic and I'd rather have more scenarios05:40
Jeffrey4liirc, we should add binary jobs to periodic jobs pipeline?05:40
inc0yeah, but that's for later, first we need basic deploy gates for kolla and dockerhub pipeline for source images05:41
Jeffrey4lor use "experimetal" to trigger the jobs to check whether it is OK.05:41
Jeffrey4lok.05:42
*** bmace has quit IRC05:42
inc0nah, I'd rather have daily checks05:42
*** bmace has joined #openstack-kolla05:42
inc0also experimental doesnt make sense any more05:42
Jeffrey4ldaily checks is hard to fixed issue.05:42
Jeffrey4li.e. if only centos+binary jobs failed, how can i push a patch and test it?05:42
Jeffrey4lthen periodic jobs + experimetal jobs should be better.05:43
inc0not sure how experimental will woork now, but you're right05:44
inc0that'd make total sens05:44
inc0ee05:44
inc0ehh, I can't write;)05:44
Jeffrey4lanyway, that will be future tasks.05:44
inc0right, we need our gates up and running first05:44
inc0also I'd love to move more of setup_gate to ansible05:44
Jeffrey4llet us focus the source jobs right now05:45
inc0agree05:45
inc0we need to make deployment gates work in kolla05:45
inc0I was thinking of using zuul cloner to pull patchset from kolla-ansible (since it works)05:46
inc0and call kolla-ansible's run.yml  from kolla job05:46
inc0with additional env that will assume images are already built05:47
Jeffrey4linc0, seem there is a way to use other's project jobs in zuulv3.05:47
Jeffrey4li saw these once.05:47
Jeffrey4lmaybe it works.05:47
inc0take a look at it if you have time05:47
Jeffrey4li will.05:48
inc0I'll ask reviewers tomorrow on meeting to merge our gate patch asap05:48
inc0for ansible05:48
Jeffrey4lbtw, https://review.openstack.org/508661  kolla build jobs is OK now.05:48
Jeffrey4lwe can fix the deploy jobs later.05:48
inc0yeah I agree05:48
Jeffrey4lso this is ready to review too.05:48
inc0better to have failing deploy gates than nothing at all05:48
Jeffrey4ldeploy jobs is added with failing in kolla05:49
inc0yes, I figured that we need to solve zuulv3 deploy gates in kolla ansible first05:49
inc0and now to figure out how to call these plays correctly05:50
Jeffrey4lyes. i saw05:50
Jeffrey4lanother thing, i think it is time to release next z stream for ocata and pike branch.05:51
Jeffrey4lcentos 7.4 breaks all release tags05:51
SamYapleJeffrey4l: why is that?05:51
Jeffrey4lSamYaple, cenots 7.4?05:51
SamYapleyes05:51
SamYaplelike what specifically so im aware05:51
Jeffrey4llibvirt is upgraded and need change the configuration.05:52
Jeffrey4llet me find the patch.05:52
SamYapleah yes. im familiar with that issue05:52
inc0yeah, we had it in ubuntu because it got newer libvirt first05:52
SamYapleyea a while back05:52
Jeffrey4lhttps://review.openstack.org/#/c/506515/05:53
Jeffrey4lhere is the patch.05:53
Jeffrey4lhttps://review.openstack.org/#/c/506515/5/ansible/roles/nova/templates/qemu.conf.j205:53
Jeffrey4lhere it he key fixed for centos 7.4 SamYaple05:53
*** huanxie has quit IRC05:53
SamYaplethanks Jeffrey4l05:57
Jeffrey4lnp05:57
SamYapleinc0: im going to be putting in a OSH gate in loci (non-voting) i would like to do the same for kolla-ansible05:58
SamYapledo you see any issue with that?05:58
masberinc0, nova-compute should run on the control node or the compute?05:58
SamYaplerather ill be using the OSH gate job in LOCI05:58
Jeffrey4lwhat is "OSH" gate?05:59
SamYapleJeffrey4l: openstack-helm05:59
inc0SamYaple: if you can handle extended starts05:59
inc0then sure05:59
inc0masber: compute05:59
SamYapleinc0: yes those will be handled. we are still looking at the best way to do it05:59
SamYaplewhether a base image of kolla or post layers with kolla bits06:00
inc0cool06:00
SamYapleso LOCI will be cross-gating, so your gate work will get reconsumed (and ill be more able to help out with kolla-ansible gate)06:01
SamYapleit doesnt affect you directly in kolla, but jsut so youre aware06:01
inc0good to hear, should be fairly easy to do06:02
inc0you'll just need a lot of image name overrides;)06:02
SamYapleyea got some experince with the cross project gating, its super simple06:02
masberinc0, ok test-openstack-controller is my control and network node and compute-21 is for computing and storage. Going back to my error, nova-compute is not running on test-openstack-controller. http://paste.openstack.org/raw/623915/06:02
SamYapleinc0: thats not so hard actually! its pretty straight forward06:02
SamYaplekolla-ansible was well thought out06:03
inc0I know it's not hard, just a lot of it06:03
inc0;)06:03
SamYapleinc0: and in the case of cross gating we can just kolla-name the images so no overrides needed in gate06:03
inc0nova_compute_image_full: ...06:03
*** openstackgerrit has joined #openstack-kolla06:03
openstackgerritjinke proposed openstack/kolla-ansible master: Skydive cannot recognize namespace created by neutron  https://review.openstack.org/50839906:03
inc0yeah whichever works, but I'm pretty sure you can override your globals06:04
SamYapleindeed thats of course doable06:04
inc0I would be ok if you'd want to have it in our gate globals06:04
SamYaplewell ill have a better grasp of whats needed once the kolla-ansible gates are at 100%06:04
inc0it's templated so whatever, you can do sth like {% if loci_gate %}{% include 'loci_globals.yml' %}06:05
inc0aio are ok already06:05
SamYaplei mean like fully converted from bash scripts06:05
inc0https://review.openstack.org/#/c/508759/06:05
SamYaplewhich i want to help with06:05
inc0well global.yml generation is already in ansible06:05
inc0so while I agree - we want to move out of bash in favor of ansible - that shouldn't block you06:06
SamYapleit honestly should be as simple as conditionalizing the image import, tarball vs loci gen'd06:06
inc0your call, your help will ofc be appreciated06:06
inc0it is really06:06
SamYapleive always like gate work, now it *really* like it06:06
SamYaplewith zuulv306:06
inc0if only you'd get feedback in less than hour or so, I'd love it too06:07
SamYaplewhat do you mean?06:07
inc0for now you can mimic what we did in kolla build gates06:07
inc0gates takes ~20min each06:07
SamYapleyea that should be fine. what was the issue with that?06:07
inc0I've seen them queued for 40 min sometimes06:08
SamYapleoh you mean for testing the gates06:08
inc0yeah06:08
SamYapleyea that is what it is06:08
SamYaplei do alot of work saturday/sunday for that reason06:08
SamYapleinsta-quee06:08
inc0yeah it was awesome;)06:08
inc0I did a bunch oo06:08
inc0too06:08
inc0anyway, I think it makes sense to move image preparation for k-ans to ansible too06:09
inc0as next step06:09
inc0move it out of bash06:09
inc0if you're up to it SamYaple, I'd be happy to review, otherwise I'll do it as soon as we restore our basic gating infra (get coverage en par with previous)06:10
SamYapleyea as soon as you merge that patch im going to push up some updated bits for you06:10
SamYaplescrap a good chunk of the bash stuff06:10
inc0cool06:11
SamYapleeasier to iterate from "working" :)06:11
inc0Jeffrey4l: can you ask pbourke and other tomorrow their morning to review gates asap? assuming Paul wasn't blown away by hurricane06:11
Jeffrey4linc0, sure.06:12
inc0cool, I'm siging off for now, talk to you all tomorrow06:12
SamYapleif you say his name three times he appears06:12
SamYaplepbourke pbourke pbourke06:12
SamYaple(eventually)06:12
Jeffrey4llol06:12
SamYapleinc0: take care!06:12
inc0actually it's 7am there06:12
SamYapleyup06:12
inc0so he should be up pretty soon06:12
SamYaplethis is about the time he pops onto IRC06:12
SamYaplewell in about an hour06:13
inc0yeah I won't be here in an hour, I'm half asleep already, gym killed me today06:13
inc0gnite06:13
* inc0 off06:13
SamYaplenite06:13
Jeffrey4lnite06:13
masberwhich images do you use?06:14
masberI am trying centos binary 4.0.0, 4.0.3, 5.0.0 and 5.0.1 and none of them are working for me06:15
SamYaplemasber: somethign that isnt ready for consumption, loci. i was tlaking to michal about having a job in loci repo to run kolla-ansible with loci images06:15
SamYaplekolla is the whole shebang, images and deploy. loci just does images06:16
SamYaplenever deploy06:16
*** jinke has quit IRC06:17
*** kiennt26 has joined #openstack-kolla06:17
*** xinliang has quit IRC06:17
*** mnasiadka has joined #openstack-kolla06:21
*** janki has joined #openstack-kolla06:24
Sean-mdhi all, the container of "centos-source-cinder-api:5.0.0" automatically runs "/usr/local/bin/dumb-init /bin/bash /usr/local/bin/kolla_start" on startup06:24
Sean-mdit is not in /etc/rc.d..06:24
Sean-mdwhere can I remove "/usr/local/bin/dumb-init /bin/bash /usr/local/bin/kolla_start"???06:25
Sean-mdthis is kolla-kubernetes06:26
*** xinliang has joined #openstack-kolla06:26
*** jinke has joined #openstack-kolla06:29
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: Move to zuul v3 in project jobs  https://review.openstack.org/50875906:30
masbercan I use kolla-ansible 5.0.0 to deploy 4.0.0 images?06:31
*** salv-orlando has quit IRC06:33
*** salv-orlando has joined #openstack-kolla06:35
*** danardelean has joined #openstack-kolla06:35
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: Move to zuul v3 in project jobs  https://review.openstack.org/50875906:37
*** danardelean has quit IRC06:37
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: Move to zuul v3 in project jobs  https://review.openstack.org/50875906:38
*** zhangfei has quit IRC06:43
*** pcaruana has joined #openstack-kolla06:45
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: Move to zuul v3 in project jobs  https://review.openstack.org/50875906:48
*** signed8b_ has joined #openstack-kolla06:50
*** signed8bit has quit IRC06:53
*** zhangfei has joined #openstack-kolla06:56
*** signed8b_ has quit IRC06:56
*** unicell1 has quit IRC06:59
*** genek has joined #openstack-kolla07:02
*** dardelean has joined #openstack-kolla07:08
*** danardelean has joined #openstack-kolla07:12
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: Move to zuul v3 in project jobs  https://review.openstack.org/50875907:15
*** danardelean has quit IRC07:16
*** b_bezak has joined #openstack-kolla07:21
*** genek has quit IRC07:23
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: Move to zuul v3 in project jobs  https://review.openstack.org/50875907:24
*** mewald has joined #openstack-kolla07:26
*** shardy is now known as shardy_afk07:26
mewaldMorning, how are we going to proceed with https://review.openstack.org/#/c/485150/ ?07:26
*** tovin07_ has quit IRC07:28
*** ipsecguy_ has joined #openstack-kolla07:28
*** caowei has joined #openstack-kolla07:29
*** ipsecguy has quit IRC07:31
*** genek has joined #openstack-kolla07:32
*** danardelean has joined #openstack-kolla07:34
mewaldI just ran a recheck, now I am seeing only success from zuul. So if that's ok, I would like to see this merged soon :)07:35
*** zhubingbing has quit IRC07:35
*** zhubingbing has joined #openstack-kolla07:36
*** yingjun has quit IRC07:36
*** zhubingbing has quit IRC07:39
*** zhubingbing has joined #openstack-kolla07:41
kolla-slack<lxdong.007> To make use_usb_table=True take effect has to make [spice].agent_enabled=False which causes regression of guest console experience, Is there any tradeoff method?07:45
*** egonzalez has joined #openstack-kolla07:46
*** huanxie has joined #openstack-kolla07:48
*** matrohon_ has joined #openstack-kolla07:51
*** paramite has joined #openstack-kolla07:52
*** shardy_afk is now known as shardy08:00
*** caowei has quit IRC08:04
*** caowei has joined #openstack-kolla08:05
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: Move to zuul v3 in project jobs  https://review.openstack.org/50875908:06
*** dougsz has joined #openstack-kolla08:07
*** lebauce_ has joined #openstack-kolla08:07
*** vkhanna has quit IRC08:07
*** ansiwen has quit IRC08:08
*** vkhanna has joined #openstack-kolla08:09
*** lebauce_ has quit IRC08:10
*** ansiwen has joined #openstack-kolla08:10
*** genek has quit IRC08:12
*** mgoddard has joined #openstack-kolla08:14
*** lebauce has quit IRC08:15
*** genek has joined #openstack-kolla08:18
*** lebauce has joined #openstack-kolla08:19
*** cah_link_ has joined #openstack-kolla08:20
*** genek has quit IRC08:23
*** genek has joined #openstack-kolla08:24
*** ipsecguy has joined #openstack-kolla08:24
*** clayton_ has joined #openstack-kolla08:27
*** mgoddard_ has joined #openstack-kolla08:27
*** ipsecguy_ has quit IRC08:28
*** clayton has quit IRC08:28
*** dciabrin has quit IRC08:28
*** mgoddard has quit IRC08:28
*** dciabrin has joined #openstack-kolla08:28
*** clayton_ is now known as clayton08:28
*** genek has quit IRC08:29
*** jrist has quit IRC08:30
*** mnasiadk1 has joined #openstack-kolla08:33
*** mnasiadka has quit IRC08:35
*** cah_link__ has joined #openstack-kolla08:44
mewaldThe sensu images also build successfully https://review.openstack.org/#/c/486379/12 Please review :)08:45
*** cah_link_ has quit IRC08:46
egonzalezmewald, we have no CI now, there is code freeze at this moment until zuulv3 is working fine08:47
mewaldegonzalez: Ahh that's still going on :( I ran recheck and Zuul came back with success to I was happy it's working now :D08:48
egonzalezonly pep and docs tests08:49
mewaldDo we still now have an ETA on when this is going to be fixed?08:49
mewald*not08:50
*** dangtrinhnt has joined #openstack-kolla08:51
*** david-lyle has quit IRC08:53
dangtrinhntHi, Trying to run the deploy-tacker-demo-sfc but looks like my public network does not work because I set neutron_external_interface to a non-IP interface. So my public network now use the default network range (10.0.2.0/24)08:54
egonzalezdangtrinhnt, the interface should not have any IP address, but the link connected to the interface should have a valid range configured to it08:57
egonzalezthat range is the one you set in the public network , example for init-runonce https://github.com/openstack/kolla-ansible/blob/master/tools/init-runonce#L1208:58
dangtrinhntegonzalez, ah ok, I think I get it now. Many thanks.09:01
*** Trinh_ has joined #openstack-kolla09:07
*** dangtrinhnt has left #openstack-kolla09:07
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: Move to zuul v3 in project jobs  https://review.openstack.org/50875909:10
*** danardelean has quit IRC09:21
*** cah_link__ has quit IRC09:26
openstackgerritcaoyuan proposed openstack/kolla-kubernetes master: Update the link for Big Tent Governance  https://review.openstack.org/51298209:26
*** cah_link__ has joined #openstack-kolla09:26
openstackgerritChen proposed openstack/kolla-ansible master: Associate text to link  https://review.openstack.org/51298309:27
*** caowei has quit IRC09:28
*** egonzalez has quit IRC09:31
*** pratapagoutham has joined #openstack-kolla09:45
*** pmisiak has joined #openstack-kolla09:49
pmisiakhi guys09:50
*** egonzalez has joined #openstack-kolla09:50
pmisiakdo you plan to backport this bug to stable/ocata? https://bugs.launchpad.net/kolla-ansible/+bug/171276709:51
openstackLaunchpad bug 1712767 in kolla-ansible "openvswitch_db shouldn't be connect from remote host" [Undecided,Fix released] - Assigned to Kuo-tung Kao (jelly) (coding1314)09:51
pmisiakLBaaS doesn't work without this commit in stable/ocata because neutron-lbaas-agent try to connect to 127.0.0.1 insted of api_interface IP address09:52
*** danardelean has joined #openstack-kolla09:54
*** rhallisey has joined #openstack-kolla09:54
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: Move to zuul v3 in project jobs  https://review.openstack.org/50875909:55
bjolomorning09:56
bjolois there a way to list the exakt command used to run a container?09:56
bjolofor ex09:56
bjoloi want to see the exact command used to fire up the neutron_dhcp_agent container09:57
bjoloegonzalez:09:57
bjolofound a tool called runlike09:57
bjolopip install runlike09:57
egonzalezbjolo, in /etc/kolla/neutron-dhcp-agent/config.json09:57
*** danardelean has quit IRC09:58
bjolosweet09:58
bjolowe have found a bug in old newton cloud09:59
bjolobuilt a new dhcp container that i wanna try out manually09:59
bjoloon one node only09:59
bjoloso my idea is to stop and remove the current container09:59
bjolofire up the new container using the same start string, just bumping the container name10:00
bjoloshould work?10:00
pmisiakbjolo: I remember in mitaka there was 'start' action in ansible, this action recreates all containers10:00
pmisiakmaybe its still in newton?10:01
bjolostart action where?10:01
bjoloouch not there on newton10:03
bjolohttp://paste.openstack.org/show/623935/10:03
bjolothats how it looks in newton10:03
pmisiakwhen you run kolla-ansile you will see command line with ansible10:03
pmisiakand there is action=deploy for example10:03
pmisiakyou can hack it and replace deploy with start :P10:04
pmisiakI used this many times10:04
openstackgerritMerged openstack/kolla-ansible stable/ocata: Fix wrong condition for iscsid container  https://review.openstack.org/50508110:05
bjolohere is the runlike output10:05
bjolohttp://paste.openstack.org/show/623936/10:05
bjolopmisiak: not sure i follow10:05
*** kiennt26 has quit IRC10:05
pmisiakbjolo10:06
pmisiakbjolo: https://github.com/openstack/kolla/blob/stable/newton/ansible/roles/neutron/tasks/start.yml10:06
pmisiakim talking about this file :)10:06
bjoloi wanna do this outside of kolla. just try a new binary directly from docker10:06
pmisiakaa ok :)10:06
pmisiaki thought you want to just recreate a container using kolla10:06
bjoloi wanna se if the bug patch in the new container fix our issue10:06
bjoloi dont want to reconfigure or upgrade the whole cloud10:07
bjolowe got 150 nodes in it10:07
*** Trinh_ is now known as dangtrinhnt10:07
*** matrohon_ has quit IRC10:08
bjolopmisiak: about your hack10:08
pmisiakbut kolla will recreaate only missing containers10:08
pmisiakall others wont be touched10:08
*** yangyape_ has quit IRC10:09
bjoloso if i delete a container on a node, i can recreate it by replacing action=start10:09
pmisiakeys10:10
pmisiakyes10:10
bjolobut a deploy would do the same thing right?10:10
pmisiakyou can also add tags to pin this to particular role10:10
pmisiakfor example -e tags=neutron10:10
bjoloyes, but that is kinda broken i newton afair10:11
pmisiaksr, -t neutron10:11
pmisiak--tags neutron :)10:11
bjoloanytime i tried --limit or --tag it broke somewhere10:11
bjoloinventory and set_facts issues10:11
*** danardelean has joined #openstack-kolla10:11
pmisiakbut for start it should work10:12
bjoloyes, but what is the difference between action=start and action=deploy in this use case10:12
bjolodeleted container on target node gets recreated and started10:12
pmisiakyes10:12
bjoloso .... difference?10:13
pmisiaklook at this file: https://github.com/openstack/kolla/blob/stable/newton/ansible/roles/neutron/tasks/deploy.yml10:14
pmisiakits for deploy action10:14
pmisiakit includes start.yml10:14
pmisiakwhen you run action=start you will run only start.yml10:15
pmisiakno config.yml and all others actions10:15
bjoloyes deploy takes longer but all things equal, those steps should be green. i.e. nothing happens10:16
bjolojust a speed thing10:16
pmisiakbut when you run deploy with --tags10:16
pmisiakthen you can have issues with variables10:17
pmisiakits only a hack to be sure that other things wont be touched by kolla10:17
bjolook gotcha10:17
pmisiakno reconfiguration etc.10:18
bjoloso how do i run a "new" dhcp container on one node manually?10:18
bjolonew binary10:18
bjololooking at this again https://github.com/openstack/kolla/blob/stable/newton/ansible/roles/neutron/tasks/start.yml10:19
bjoloanyway i can dump out the resulting docker run command from that?10:19
*** kevzha01 has quit IRC10:22
*** mewald has quit IRC10:25
bjolohttp://paste.openstack.org/show/623941/10:26
bjolojust change the container name?10:26
*** pbourke has quit IRC10:27
bjolostop and rm current neutron_dhcp_agent first of course10:28
*** pbourke has joined #openstack-kolla10:28
bjoloegonzalez: pbourke yey or nay :)10:29
pbourkebjolo: what are we talking about?10:29
bjolowe have issues with neutron on a kolla newton cloud10:30
bjolobelieve it is a bugg that has been fixed so we have built new neutron_dhcp_agent container10:30
bjolonow i wanna try that container out manually on just one node10:31
bjolosimple idea is to stop and rm current container10:31
bjolothen docker run the new container10:31
pbourkeok10:31
bjolobut i need to have the exact start string for docker10:31
pbourkei have the perfect solution for you :)10:31
pbourkehttps://github.com/nexdrew/rekcod10:31
bjolonice :)10:32
bjoloi found a tool called runlike10:32
bjolohttp://paste.openstack.org/show/623941/10:32
pbourkeah, same thing10:33
pbourkedoes it not work?10:33
bjologonna try rekcod and see if there is any difference in the output. if not -> verified10:33
bjolohave not tried yet10:33
bjolowanted to bounce the idea here first10:33
pbourkeive used it often in the past for debugging10:34
bjoloi have another thing i like to discuss with you later thou :)10:34
pbourkesure10:34
*** duonghq has quit IRC10:34
pbourkebtw, I think we really need your inventory-group-as-config-target bp10:34
bjoloits about moving the state into the container bootscripts instead of reconfigure/deploy10:34
pbourkeinc0 indicated it was already possible but I dont think so10:35
pbourkewhich state?10:35
bjolofor ex neutron multiple external networks that you coded for newton release10:35
bjoloif i want to add a new external network, i need to run reconfigure to actually get the new bridge configured up10:36
bjolocorrect?10:37
bjoloi add the interface and bridge to globals10:38
bjolorun reconfigure which actually goes into the container and run ovs-ovctl commands10:39
*** egonzalez has quit IRC10:39
pbourkeyes10:39
bjolocould that code be moved or duped into the container startup sequence?10:40
bjoloim working on the new granular config workflow (mostly in my head)10:40
bjolothe genereal idea and unix wow is that a reboot should set the correct start10:41
bjoloany system is only binaries and config files right10:41
pbourkeso you'd have to reboot to reconfig?10:41
bjolobooting the binary with the config file should set the correct state10:41
bjoloyes10:41
bjolorestart container in this case10:42
pbourkewhats the benefit over just an ansible reconfigure10:42
pbourkealso how do you get the new config files laid down10:42
bjologranuality10:42
bjolothe workflow on high level10:42
*** pratapagoutham has quit IRC10:43
bjologenerate new config, push config to target nodes, restart containers that have new config10:43
bjolosimple10:43
bjolono hidden steps or magic happens here10:43
bjoloalso very git friendly to track changes10:44
bjolonow we have situation that new config file is in place but the state might not be correct10:44
bjoloi.e. new ovs bridge is not up10:45
*** mrunge has quit IRC10:45
bjoloto many magic happens here and config is instantiated with external commands leads to complex state machine10:46
bjoloapproach above is very simple 1,2,3 approach10:47
pbourkeim just not sure I see how moving into the container startup makes it any different10:47
pbourkesorry maybe its obvious :)10:47
bjolofor an admin it is. at least to me :)10:47
pbourkewhether the steps happen in container start or ansible run, they're still the same steps10:47
*** pratapagoutham has joined #openstack-kolla10:47
bjoloyes, but the logic and workflow is simpler10:48
bjolo1,2,310:48
bjoloif i restart a container, it runs according to its config10:48
bjolono mismatch10:48
*** mrunge has joined #openstack-kolla10:48
pbourkeif I re-run ansible, it runs according to its config10:49
pbourkesame thing :p10:49
bjoloreconfigure as it is today is a dead end10:49
bjolono admin will ever use it10:49
pbourkewhy10:49
bjoloread my BP about it10:49
bjolotries todo too much10:50
bjoloto many places shit can go wrong10:50
bjoloi have no way to verify or inspect the new config before it is pushed out10:50
bjoloadmins wants granularity10:50
bjolomultistep10:51
bjolotake my use case above10:51
bjoloyes, i should try this in a lab i know :)10:51
bjoloi want to try a new container on one node just to se if that fix the issue10:51
bjoloi dont want to run reconfigure on the whole damn cloud for that10:52
*** caowei has joined #openstack-kolla10:52
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: Move to zuul v3 in project jobs  https://review.openstack.org/50875910:52
bjoloevery change that goes into a production cloud is done very incremental and controlled10:52
bjolodowntime costs millions per hour10:53
pbourkeok the bp makes a little more sense10:53
pbourkeI dont think Id seen this one10:53
bjoloso in liue with the bp10:53
pbourkeyeah so it seems we want to make the reconfig process more granular10:54
pbourkemore than just moving where things happen10:54
bjolo1,2,3 approach to reconfigure where the container start set the state10:54
pbourkethat would probably be fine10:54
bjoloim actually rewriting our internal playbooks in this fashion atm10:55
bjoloall file copy and templating of config files dumps them out locally first10:55
bjoloi.e. the final files are dumped out locally10:55
bjolothen we get verify and review what was changed and check into git10:55
pbourkesounds nice10:56
bjolothen we push the files and restart services as needed10:56
bjolosame 1,2,3 logic10:56
pbourkemakes sense10:56
hrwDoes someone know how to build image used to setup Docker registry? It is available only for x86-64 architecture ;(10:57
pbourkebjolo: I think if you could prototype one service in kolla to follow that pattern it would be very useful for people to see10:57
bjoloi gotta go for lunch and workout10:57
bjolopbourke: i will give it a try10:57
pbourkebjolo: cool thanks for explaining10:58
bjoloi can otherwise demo a normal OS file10:58
bjolothe code i work on right now is actually just /etc/resolv.conf10:58
bjolofor the os10:58
openstackgerritzhangfei gao proposed openstack/kolla master: ironic-pxe: generate aarch64 bootfile grubaa64.efi  https://review.openstack.org/51254010:59
hrwpbourke: do you know who I should ask to mark https://review.openstack.org/#/c/423239/ (add support for ppc64le arch) as abandoned? it was done in whole set of my patches.10:59
bjolohave a template file with some jinja and variables. different hosts sets different variables10:59
pbourkehrw: can you not abandon yourself?10:59
hrwpbourke: not a core dev10:59
pbourkeah I thought you were the author11:00
bjoloresulting file is dumped out in a structure git/<hostname>/etc/resolv.conf11:00
hrwpbourke: it is not mine and original author vanished11:00
pbourkehrw: I can do it11:00
hrwpbourke: thanks!11:00
pbourkehrw: ping me any others if needed11:01
hrwpbourke: thanks. that was the only one11:02
*** gfidente has joined #openstack-kolla11:03
*** gfidente has quit IRC11:03
*** gfidente has joined #openstack-kolla11:03
*** danardelean has quit IRC11:04
*** danardelean has joined #openstack-kolla11:07
*** jrist has joined #openstack-kolla11:09
*** zhangfei has quit IRC11:12
*** huanxie has quit IRC11:13
*** danardelean has quit IRC11:20
*** danardelean has joined #openstack-kolla11:22
*** rwallner has joined #openstack-kolla11:27
*** danardelean has quit IRC11:29
*** caowei has quit IRC11:31
*** dave-mccowan has joined #openstack-kolla11:31
*** danardelean has joined #openstack-kolla11:32
*** dave-mcc_ has joined #openstack-kolla11:35
*** danardelean has quit IRC11:36
*** dave-mccowan has quit IRC11:37
openstackgerritMartin André proposed openstack/kolla master: Moving jobs to kolla repo  https://review.openstack.org/50866111:37
*** b_bezak has quit IRC11:38
*** danardelean has joined #openstack-kolla11:40
*** danardelean has quit IRC11:40
*** danardelean has joined #openstack-kolla11:40
*** danardelean has quit IRC11:41
openstackgerritcaoyuan proposed openstack/kolla-ansible master: Update the manila shares link  https://review.openstack.org/51300411:41
*** danardelean has joined #openstack-kolla11:43
*** yangyapeng has joined #openstack-kolla11:45
*** huanxie has joined #openstack-kolla11:46
*** danardelean has quit IRC11:46
*** ansmith has quit IRC11:46
*** danardelean has joined #openstack-kolla11:46
*** solomon_ezhra has joined #openstack-kolla11:51
*** solomon_ezhra has quit IRC12:00
*** Pavo has joined #openstack-kolla12:02
*** dangtrinhnt has quit IRC12:11
*** huanxie has quit IRC12:16
openstackgerritDaniel Alvarez proposed openstack/kolla master: Add Dockerfile for networking-ovn-metadata-agent  https://review.openstack.org/51122512:17
*** goldyfruit has joined #openstack-kolla12:18
openstackgerritDaniel Alvarez proposed openstack/kolla master: Add Dockerfile for networking-ovn-metadata-agent  https://review.openstack.org/51122512:21
*** goldyfruit has quit IRC12:22
*** dangtrinhnt has joined #openstack-kolla12:22
*** salv-orlando has quit IRC12:26
*** salv-orlando has joined #openstack-kolla12:28
*** dave-mcc_ is now known as dave-mccowan12:29
*** huanxie has joined #openstack-kolla12:30
*** gkadam has quit IRC12:34
*** nguyentrihai has joined #openstack-kolla12:37
*** ansmith has joined #openstack-kolla12:41
bjolopbourke: have you used rekcod on neutron_openvswitch_agent containers?12:42
bjolopbourke: runlike does not catch --net=host12:42
bjolofor ex12:42
pbourkebjolo: not sure, probably?12:42
*** kbaegis has joined #openstack-kolla12:45
*** clayton has quit IRC12:45
*** mnasiadk1 has quit IRC12:47
*** clayton has joined #openstack-kolla12:49
*** rhallisey has quit IRC12:52
*** rhallisey has joined #openstack-kolla12:53
bjolorekcod does not seem to work either12:54
bjoloit catches net=host but the start string does not bring up the container properly12:54
bjoloopenvswitch_vswitchd and neutron_openvswitch_agent have issues12:55
bjolodocker and permissions12:55
*** shardy has quit IRC13:00
*** shardy has joined #openstack-kolla13:00
*** huanxie has quit IRC13:01
*** signed8bit has joined #openstack-kolla13:01
*** alanmeadows has quit IRC13:02
*** alanmeadows has joined #openstack-kolla13:02
*** goldyfruit has joined #openstack-kolla13:04
*** bastafidli has joined #openstack-kolla13:10
*** danardelean has quit IRC13:11
*** shardy has quit IRC13:11
pbourkehmm13:12
*** shardy has joined #openstack-kolla13:13
*** matrohon_ has joined #openstack-kolla13:20
*** danardelean has joined #openstack-kolla13:23
*** cah_link__ has quit IRC13:23
bjoloto start with, the openvswitch kernel module does not get loaded13:23
*** danardelean has quit IRC13:23
*** danardelean has joined #openstack-kolla13:23
pomacbjolo, pbourke: and if we load the module, it will complain about no permisisons to attach to brigdes etc - so something is really broken...13:31
pbourkebjolo: pomac: in most cases the command rekcod gives should be what ansible is running13:37
pbourkewhat you can do is replace kolla_start with 'bash'13:37
pbourkejump in and try running the start scripts by hand13:37
pbourkethat way you can have a closer look13:38
*** bmace has quit IRC13:39
*** signed8bit has quit IRC13:42
bjolopbourke:13:44
bjolowill try13:44
bjologood idea13:44
bjolowhen is kolla_extend_start copied into the container?13:44
*** signed8b_ has joined #openstack-kolla13:44
bjoloi would assume insmod openvswitch happens there for ex13:45
*** jtriley has joined #openstack-kolla13:47
pbourkeat build time13:47
bjoloreally13:57
bjolohow does that work with kolla_kuberneetes for ex13:58
bjoloi presumed the containers were generic binaries. only preloaded with kolla_start13:58
bjolothen each orch tool injected their own "boot" scripts13:58
*** hrw has quit IRC13:59
pbourkethat may be well how kolla-k8s does it14:00
pbourkeim not sure14:00
pbourkebut for standard ansible they're built in14:00
*** hrw has joined #openstack-kolla14:00
bjolohmmm not sure i like that14:00
pbourkeyeah there are plenty who agree14:00
bjoloboot scripts are config files14:00
pbourkeSamYaple: ^14:01
bjoloas we discussed previously14:01
bjolo1,2,3 pattern. generate config. push config, restart containers14:01
pbourkeim not opposed to kolla moving in this direction14:01
bjolobeen toying around and with this pattern, all type of actions would be blazingly fast14:02
bjolobesides the initial genconfig which needs to inventory and set_facts for all nodes14:03
bjolobut even that would not take long14:03
pbourkeI think the original idea was that containers would be fully imutable once created14:04
pbourkebut that never really was practical14:04
bjoloyea, ive heard that story14:04
bjolobut define immutable?14:05
bjoloi would say what is wanted is trackable14:05
bjoloi.e. every change is visible and tracked. which in a sense is immutable14:05
bjolothe binary container is immutable by definition14:06
bjoloa cm workflow supported by git makes changes trackable14:06
bjolofor ex we have a tool called etckeeper installed on every server/vm14:07
bjolobasically puts whole /etc in git14:07
*** aagate has joined #openstack-kolla14:08
bjoloalso triggered by yum/apt so the whole list of packages are also tracked as a text file14:08
bjoloautomatic snapshots of /etc14:08
bjolocommits14:08
bjolothe way we operate kolla today is with genconfig and etckeeper14:09
bjolonever with reconfigure14:09
bjolowe generate the configs, then we look on the nodes what has changed with git status14:09
*** mnasiadka has joined #openstack-kolla14:10
bjolosounds not scalable but it is not that bad actually14:10
*** salv-orlando has quit IRC14:11
*** salv-orlando has joined #openstack-kolla14:11
bjolowe only have a few node types right, and with tmux sync panes and ansible one can handle that pretty easy14:11
*** huanxie has joined #openstack-kolla14:12
*** sambetts|afk is now known as sambetts14:14
*** salv-orlando has quit IRC14:16
*** janki has quit IRC14:25
*** jamesbenson has joined #openstack-kolla14:32
*** erlon has joined #openstack-kolla14:32
*** lukepatrick has joined #openstack-kolla14:36
*** david-lyle has joined #openstack-kolla14:38
*** huanxie has quit IRC14:42
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: Move to zuul v3 in project jobs  https://review.openstack.org/50875914:44
*** mnasiadka has quit IRC14:52
*** pmisiak has quit IRC14:54
*** duonghq has joined #openstack-kolla14:54
*** MarginHu has quit IRC15:03
*** mdnadeem has quit IRC15:03
duonghqevening guys15:06
*** coolsvap has quit IRC15:06
*** bastafidli has quit IRC15:07
*** bastafidli has joined #openstack-kolla15:09
*** lrensing has joined #openstack-kolla15:13
*** lpetrut has joined #openstack-kolla15:14
*** pcaruana has quit IRC15:19
*** hongbin has joined #openstack-kolla15:27
*** lvdombrkr has quit IRC15:36
inc0good morning15:42
*** lrensing has quit IRC15:44
duonghqmorning inc015:44
inc0Jeffrey4l: what was the issue with https://review.openstack.org/#/c/508759/ ? what are you working on?15:45
*** lukepatr_ has joined #openstack-kolla15:50
*** lukepatrick has quit IRC15:53
kfox1111good morning.15:54
*** vhosakot has joined #openstack-kolla15:55
*** danardelean has quit IRC15:55
chasono/15:55
inc0meeting15:58
Jeffrey4linc0, i am working on it. i made some refator. but upgrade rabbitmq failed right, no idea why.15:59
inc0kk15:59
inc0llet's talk about this on meeting15:59
*** gkadam has joined #openstack-kolla16:02
*** britthouser has joined #openstack-kolla16:02
*** genek has joined #openstack-kolla16:09
*** lrensing has joined #openstack-kolla16:10
*** salv-orlando has joined #openstack-kolla16:12
*** mmehan has joined #openstack-kolla16:15
*** salv-orlando has quit IRC16:17
openstackgerritMartin André proposed openstack/kolla master: Add Apache packages to mistral-api container  https://review.openstack.org/51308916:21
*** gkadam has quit IRC16:22
*** signed8b_ has quit IRC16:23
*** salv-orlando has joined #openstack-kolla16:25
*** ntpttr_laptop has joined #openstack-kolla16:26
*** ntpttr_laptop has quit IRC16:28
*** matrohon_ has quit IRC16:33
*** vhosakot has quit IRC16:34
*** vhosakot has joined #openstack-kolla16:35
*** danardelean has joined #openstack-kolla16:35
*** vhosakot has quit IRC16:39
*** danardelean has quit IRC16:40
*** vhosakot has joined #openstack-kolla16:40
SamYaplebjolo: immutable as in config and everything was baked into the image. this was not praticle16:40
SamYaplebjolo: we evetually settled on config out of the container, scripts in the container16:40
SamYaplebjolo: i am personalyl in the camp that images are binary blobs and any starting scripts should be bound in at runtime, but some people disagree with taht16:41
*** vhosakot has quit IRC16:46
*** vhosakot has joined #openstack-kolla16:46
openstackgerritMartin André proposed openstack/kolla master: Moving jobs to kolla repo  https://review.openstack.org/50866116:50
*** mgoddard_ has quit IRC16:51
*** jamesbenson has quit IRC16:52
*** dougsz has quit IRC16:55
*** jmccarthy has joined #openstack-kolla16:58
jmccarthyTrying to figure out about using horizon to view instance console (instance on xen compute) - the console.log works, but not the console, any ideas ?16:58
Jeffrey4labout https://review.openstack.org/50875916:58
Jeffrey4li made some refactor. now upgrade rabbitmq failed. ready no idea why.16:58
Jeffrey4linc0, ^^  and patch 83 is the old one which is green16:59
kolla-slack<inc0> kk I'll work on this today16:59
Jeffrey4lthx.17:00
*** jamesbenson has joined #openstack-kolla17:07
*** jamesbenson has quit IRC17:12
*** jamesbenson has joined #openstack-kolla17:12
*** jamesbenson has quit IRC17:14
*** jamesbenson has joined #openstack-kolla17:21
*** huanxie has joined #openstack-kolla17:22
*** jamesbenson has quit IRC17:22
duonghqJeffrey4l, can you review this? https://review.openstack.org/#/c/425446/17:24
Jeffrey4lwill review it tomorrow ;)17:25
*** lpetrut has quit IRC17:25
*** jamesbenson has joined #openstack-kolla17:25
*** jamesbenson has quit IRC17:27
duonghqthanks Jeffrey4l17:27
*** duonghq has quit IRC17:28
*** jmccarthy has left #openstack-kolla17:28
Jeffrey4lnp17:28
*** pbourke-home has joined #openstack-kolla17:30
*** vhosakot has quit IRC17:34
*** pbourke-home has quit IRC17:37
*** jamesbenson has joined #openstack-kolla17:37
*** jamesbenson has quit IRC17:38
*** jamesbenson has joined #openstack-kolla17:40
*** lukepatrick has joined #openstack-kolla17:41
*** jamesbenson has quit IRC17:42
*** lukepatr_ has quit IRC17:43
*** jamesbenson has joined #openstack-kolla17:46
*** sambetts is now known as sambetts|afk17:46
*** jamesbenson has quit IRC17:49
*** jamesbenson has joined #openstack-kolla17:49
*** britthouser has quit IRC17:55
*** jamesbenson has quit IRC17:56
*** mwynne has joined #openstack-kolla17:59
*** huanxie has quit IRC18:03
*** jamesbenson has joined #openstack-kolla18:06
openstackgerritMerged openstack/kolla-kubernetes master: Update the link for Big Tent Governance  https://review.openstack.org/51298218:09
inc0kfox1111: around?18:18
kolla-slack<kfox1111> Kind of. What's up?18:18
inc0I don't know if you've seen our meeting today, but at the end we have one action for you;) or us rather18:19
*** lpetrut has joined #openstack-kolla18:19
inc0let's do few hrs meeting (hangouts? recording?)18:19
inc0to walk through full installation of kolla-k8s18:19
inc0and general q&a and training18:20
inc0we need to create pipeline to get others to your level of understanding of kolla-k8s18:20
inc0because I'm not even close for example18:20
inc0and obviously you can't handle the thing yourself18:20
inc0are you willing to do sth like this? later this week or next week?18:21
kolla-slack<kfox1111> +118:21
inc0cool, what time would be best for you?18:21
*** rwsu has quit IRC18:25
*** paramite has quit IRC18:25
*** newmember has joined #openstack-kolla18:30
*** bjolo_ has joined #openstack-kolla18:34
kolla-slack<kfox1111> That is a darned good question.18:35
rwellumPlus debugging help would be great. It's a big question though: even k8s takes a lot of expertise.18:36
bjoloSamYaple: tnx for clarifying18:43
*** matrohon_ has joined #openstack-kolla18:45
bjoloSamYaple: how does kolla-k8s start its containers? i.e. i presume the ansible start scripts not always make sense in k8s18:59
SamYaplebjolo: kolla-k8s modifies the scripts in the kolla repo to work with both ansible and kubernetes. additionally they run scripts like https://github.com/openstack/kolla-kubernetes/blob/9ca5ae155c484f594dd26479c59dd53eb8381853/helm/microservice/nova-placement-deployment/templates/nova-placement.yaml#L2619:03
*** huanxie has joined #openstack-kolla19:19
inc0SamYaple: that;s a negative19:20
inc0as it's stand today ansible code is forked and lives in kolla-k8s repo19:20
inc0\ahh, start scripts actually are similar19:20
inc0sorry just got online;) kolla_start is the same regardless of orch tool19:21
*** dgedia has joined #openstack-kolla19:25
*** lukepatr_ has joined #openstack-kolla19:28
dgediaI am trying to install openstack using kolla-ansible on a multi-node environment. However, I am facing an error when I deploy. Error is " API Error: 500 Server Error http: server gave HTTP response to HTTPS client". Am I missing any configuration? Any help would be really appreciated19:28
inc0dgedia: what caused this error?19:29
inc0which command and so on19:29
*** lukepatrick has quit IRC19:30
*** salv-orlando has quit IRC19:31
dgedia@inc0, I did "kolla-ansible deploy -i kolla-ansible/ansible/inventory/multinode"19:31
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Move to zuul v3 in project jobs  https://review.openstack.org/50875919:32
inc0and full logs plz then:)19:32
inc0or at least task which threw this error19:32
*** salv-orlando has joined #openstack-kolla19:35
*** rwsu has joined #openstack-kolla19:37
*** dgedia_ has joined #openstack-kolla19:44
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode scenarios in zuulv3  https://review.openstack.org/51277919:44
dgedia_@inc0, Please find error details here : https://pastebin.com/BBG7ak1C19:45
*** dgedia has quit IRC19:45
inc0dgedia_: it's multinode right?19:46
dgedia_yup19:46
inc0did you install docker manually or used bootstrap servers?19:46
inc0also you are using local registry right?19:46
*** csuttles_ has quit IRC19:47
*** huanxie has quit IRC19:50
dgedia_@inc0, installed docker manually. Yes, I am using local registry19:52
inc0right, I bet you didn't add --insecure-registry :)19:52
inc0https://docs.docker.com/registry/insecure/ <- make sure to configure insecure-registries pointing to node with your private registry on every docker service19:53
inc0or, alternatively, if you can quickly cleanup your env19:54
inc0you can just run kolla-ansible bootstrap-servers and this command will install docker, configure insecure registry and do all that19:54
dgedia_I edited the /etc/default/docker with DOCKER_OPTS="--insecure-registry 192.168.1.100:5000"19:54
inc0I think newer versions of docker doesn't use this file19:55
inc0which version of docker you're running?19:55
inc0and which base distro?19:55
dgedia_Docker version 17.09.0-ce, build afdb6d419:57
inc0right, so try method described in docker docs19:57
inc0disclaimer - best tested version of docker is 1.1219:57
dgedia_as well as I use Ubuntu 16.0419:57
pomacdgedia_: check your systemd unit file - it might use different configuration ways19:58
pomacdgedia_: (we had a similar issue with centos 7)19:58
inc0pomac is right - systemd conf files is another way to do it19:58
inc0to confirm if your docker registry works, run docker info19:58
pomacinc0: some of them move things around, but i actually think that this is the deamon.json or something like that file19:59
inc0yeah https://docs.docker.com/registry/insecure/#deploy-a-plain-http-registry deamon.json is described here19:59
pomacso edit /etc/docker/daemon.json19:59
inc0but I tink it's fairly new19:59
pomacinc0: thanks19:59
pomacinc0: it is - and it's a bit annoying when things chnage20:00
*** lpetrut has quit IRC20:00
inc0that's one of reasons why docker 1.12 is best tested and we kinda stick to it20:00
dgedia_docker info : https://pastebin.com/TQmMj89P20:00
*** lpetrut has joined #openstack-kolla20:00
inc0oh so it does have insecure registry configured correctly20:01
dgedia_I'll get docker 1.12 installed in the meanshile since it is best tested20:01
inc0are you sure thats the case for all the nodes?20:01
dgedia_Appreciate inc0 and pomac for promt suggestions :)20:01
*** salv-orlando has quit IRC20:02
inc0no worries, let me know if you have further questions20:02
*** salv-orlando has joined #openstack-kolla20:02
inc0also if you want to once and for all test if your local registry works well20:02
inc0try docker pull 192.168.100.13:5000/ubuntu-source-base:5.0.020:03
inc0(image name may differ depending on you distro/type/version of kolla)20:03
*** ipsecguy_ has joined #openstack-kolla20:04
dgedia_how could I verify distro/type/version of kolla ?20:05
*** gema has quit IRC20:06
*** salv-orlando has quit IRC20:07
*** ipsecguy has quit IRC20:08
*** gema has joined #openstack-kolla20:08
*** gema has quit IRC20:08
*** gema has joined #openstack-kolla20:08
*** dgedia_ has quit IRC20:14
*** ansmith has quit IRC20:17
*** openstackgerrit has quit IRC20:17
*** lpetrut has quit IRC20:17
*** salv-orlando has joined #openstack-kolla20:18
*** dgedia has joined #openstack-kolla20:31
*** rwallner has quit IRC20:31
*** rwallner has joined #openstack-kolla20:33
*** rwallner_ has joined #openstack-kolla20:34
*** kbaegis1 has joined #openstack-kolla20:34
*** kbaegis has quit IRC20:34
*** kbaegis1 has quit IRC20:36
*** rwallner has quit IRC20:37
*** rwallner_ has quit IRC20:38
*** kolla-slack has quit IRC20:39
*** kolla-slack has joined #openstack-kolla20:39
bjolospeaking of docker 1.12 and baremetal20:41
bjoloon redhat/centos it is hardcoded to 1.12.020:42
bjolobut for debian it says docker-engine=1.12.*20:42
SamYaplebjolo: the version of docker on debian based distros is appended with additional stuff20:51
SamYapleeasier to just do a splat20:52
*** sbezverk has quit IRC20:54
bjoloyea i get that, but dont we get 1.12.0 on RH and 1.12.6 on debian now?20:57
*** matrohon_ has quit IRC20:58
SamYaplebjolo: i use 17.09 and im pretty sure kolla gates with 17.0920:58
SamYapleso i dont think it matters that much20:58
bjolowe have had some issues with different docker version mismatch in our env20:59
bjolocompute nodes installed at different times got different versions of docker20:59
*** jtriley has quit IRC20:59
bjoloactually now rewriting our playbook and hardcode it to 1.12.0 since that is what is probably most tested21:00
SamYaplebjolo: i doubt that personally21:00
SamYapleagain, the gates dont use that21:00
bjolowe have two clouds in prod. cloud-1 is newtonw and various docker 1.12 and 1.1321:02
bjolocloud-2 is ocata and docker 17.0521:03
bjoloon cloud-2 we can not do a docker restart libvirt. well we can, but after that we lose contacts with all running vms21:04
bjoloeverything else equals21:04
bjolocloud-1 does not have this issue21:04
SamYaplebjolo: you need to enable live-restore21:04
bjolodone that21:04
SamYaplehmmmmmm21:04
SamYapleare the qmeu processes dying?21:05
SamYapleor just libvirt cant reconnect21:05
inc0ps aux on host shows vms running?21:06
bjolohttp://paste.openstack.org/show/624026/21:06
bjolowe lose the sockets to the vms21:07
bjololibvirt can communicate21:07
SamYaplebjolo: oh i believe you when you say things, you dont have to post proof21:07
bjolopomac: has the details21:07
bjoloi know. did that to get fresh eyes21:07
bjolomaybe somehing wrong21:08
SamYaplethat looks fine, but the lose the sockets to the vm is strange...21:08
SamYaplewould need more details about that21:08
pomacYep, thats what happens to us21:08
pomacIt's only when libvirt is restarted, it's like when the filesystem is remounted the actual socket quality is dropped (and they become dumb, unconnected files again)21:09
SamYaplepomac: sounds like an issue with shared mounts21:10
bjolothe other thing observered is that the qemu-kvm processes are moved out from libvirt container21:10
SamYaplecan either of you post your /etc/systemd/system/docker* stuff?21:10
pomacSamYaple: yes21:10
SamYaplebjolo: thats libvirt moving them to another cgroup21:10
pomacbjolo: I'm not on vpn or anything21:10
SamYaplecoincidentally, thats the only reason vms survive a libvirt restart21:10
bjolook21:10
SamYaplenon-kvm vms dont survive libvirt container restart21:11
*** bastafidli has quit IRC21:11
pomacIf the shared mount is actually a mount, then the sockets shouldn't survive the remount that happens - if the mount was kept - it would be a differnet storry21:12
pomacstory21:12
SamYaplepomac: the socket lives in /var/lib/nova/* which is a volume so it should be shared21:12
pomacSamYaple: unfortunatelty it seems to be mounted by docker, so you have two VFS layers - thus the space where the socket lived is gone on the restart21:13
inc0also it works on 1.12...docker has nasty habit of breaking things so 17.05 - no diea if they broke anything21:13
bjolohttp://paste.openstack.org/show/624029/21:14
pomacinc0: we upgraded due to other odd problems with btrfs21:14
SamYaplebjolo: i belivee the way you have that configured means live-restore is not actually taking place because daemon.json isn't being read21:14
bjolowell it picks up insecure registry somehow21:15
bjolohttp://paste.openstack.org/show/624030/21:15
SamYapleah then it must default to reading it21:16
*** salv-orlando has quit IRC21:16
SamYaplefor future reference, *all* daemon options can go in daemon.json and you can leave the Exec part empty21:16
SamYapleback to the issue at hand21:16
SamYaplei would have to duplicate this, but it sounds like it shouldn't be hard...21:16
SamYaplecan i get your host os21:16
pomacSamYaple: centos 721:17
pomacSamYaple: with some more recent kernels - in some ways (due to rcu issues with namespaces)21:18
*** salv-orlando has joined #openstack-kolla21:18
SamYaplebacking filesystem21:18
pomacSamYaple: (oh and centos frankenkernel is quite unknown)21:18
bjolobtrfs21:18
pomacSamYaple: btrfs21:18
SamYapleoh. well thats the issue21:18
bjoloelrepo kernel21:18
pomacSamYaple: since it actually "mounts" the filesystem21:18
SamYaplethey changed the way the driver works21:18
SamYapleit does some weird stuff, yea21:19
SamYapleoverlayfs[2] (ext4 only i believe) works better21:19
SamYapleits been a while since i played with that. i can test it though21:19
SamYaplegive me a few21:19
SamYaplei can probably get you a docker commit id21:19
*** salv-orlando has quit IRC21:21
pomacSamYaple: it's annoying when it doesn't work the same for all things - it sounds like a implementation fault really...21:21
pomacAlso, using socket files like that is a bit odd... =)21:21
SamYaplepomac: to be clear here, i belive teh issue is docker changed a thing and centos btrfs is old and dumb (regardless of the backport)21:22
SamYapleim going to verify it happens/doesnt happen on ubuntu first21:22
bjolocool21:22
bjolotnx SamYaple21:22
*** salv-orlando has joined #openstack-kolla21:23
SamYapleif at all possible, consider using overlayfs2 as teh backend though, save you lots of headache21:23
pomacSamYaple: we run a 4.9 mainline kernel atm21:23
bjoloSamYaple: you must have a running vm before the libvirt restart21:26
bjolopomac: refresh my mind. we could migrate a "lost" vm to another node right21:26
pomacSamYaple: can one run one backend for one container and others for others? That would be one solution21:27
*** newmember has quit IRC21:27
SamYaplebjolo: im going to reproduce it from a pure socket breakage for simplicity21:27
pomacbjolo: I think so21:27
SamYapleoutside of kolla21:27
bjolook21:27
*** newmember has joined #openstack-kolla21:27
bjoloonce migrated it was back online again inside the libvirt container21:28
bjolowe noticed this after we gone live so we ended up migrate around all vms on the cloud :)21:28
SamYaplenice...21:29
bjoloonly about 100 so not too bad21:29
pomacthe interesting part is that what happens is that libvirt can't START new kvm instances21:29
pomacuntil you have migrated all old ones out of the way21:29
bjoloand then restart libvirt again right21:29
pomac<- tired21:29
dgedia@inc0, I successfully bootstrapped, but while doing kolla deploy, I am getting an error "The requested image does not exist: 192.168.100.13:5000/kolla/ubuntu-source-fluentd:5.0.0"21:29
bjolome too21:29
pomacbjolo: my alarm clock is set for 6:30 - ;)21:30
bjolotomorrow it is openstack nordic days in copenhagen denmark21:30
bjoloa bunch of us are going21:30
bjoloSamYaple: really tnx for testing out. i have to drop out for tonight. 1130PM mytime21:31
pomachummm.. need to restart to get the damned irssi notifier a chance of actually working...21:31
pomacbrb21:32
*** pomac has quit IRC21:32
inc0dgedia: did you do kolla-build with --push?21:32
dgediahaven't build it21:33
dgediahow should I approach to build ?21:33
*** pomac has joined #openstack-kolla21:34
dgediakolla-build --registry 192.168.100.13:5000 --push -b ubuntu -t binary21:35
dgediais that the right way to do?21:35
pomacSamYaple: if you use my nick, i shold hopefully get proper notifications from irssinotifier now21:35
pomacAnd it works too - basically, i can provide bjolo with info tomorrow if you want to relay something21:36
bjologoodnight humanoids21:37
pomacnight bjolo, see you in a few hours21:37
bjoloyup21:37
dgediainc0 ?21:39
bjolodgedia: that should work21:39
inc0dgedia: yeah, but don't use ubuntu binary21:39
inc0use ubuntu source21:39
inc0its much more stable21:39
dgediacool :)21:39
inc0and make sure globals.yml has same configs21:40
*** paramite has joined #openstack-kolla21:44
*** jamesbenson has quit IRC21:49
*** jamesbenson has joined #openstack-kolla21:52
dgediathanks inc0 for your suggestion, appreciate it :)21:55
inc0no worries, I'm working as we speak to provide images in dockerhub so it should be much easier later on21:56
*** zhubingbing_ has joined #openstack-kolla21:56
*** dardelean_ has joined #openstack-kolla21:56
*** jamesbenson has quit IRC21:56
dgediaThat would be really great!21:56
*** zhubingbing has quit IRC21:58
*** dardelean has quit IRC22:00
dgedia@inc0, there isn't any issue right if I build binary and source images both (However specify only source in globals.yml) ? I happened to build binary earlier.22:02
*** MarginHu has joined #openstack-kolla22:05
*** rwallner has joined #openstack-kolla22:05
*** rwsu has quit IRC22:05
*** ipsecguy_ has quit IRC22:06
*** dardelean__ has joined #openstack-kolla22:08
*** rwallner has quit IRC22:09
*** dardelean_ has quit IRC22:11
*** dave-mccowan has quit IRC22:11
*** rhallisey has quit IRC22:11
inc0dgedia: no, only thing abut ubuntu binary is that it was breaking more often than source22:18
inc0but you deploy what you specify in globals22:18
*** brad[] has quit IRC22:23
*** brad[] has joined #openstack-kolla22:23
*** ipsecguy has joined #openstack-kolla22:29
*** lukepatr_ has quit IRC22:29
dgediainc0, sounds good. How long does it take the build to complete?22:33
*** salv-orlando has quit IRC22:33
inc0depends on stor/net22:33
*** bjolo_ has quit IRC22:33
inc0on ssd+good net it was ~20min22:33
*** ansmith has joined #openstack-kolla22:35
dgediainc0 how should I go about with building images that failed with kolla-build?22:42
dgediaI have a bunch of images that failed in build process22:43
*** rwallner has joined #openstack-kolla22:44
*** rwallner has quit IRC22:44
dgediainc0, failed images: https://pastebin.com/4LW1CtaT22:45
*** rwallner has joined #openstack-kolla22:45
*** rwallner has quit IRC22:48
*** rwsu has joined #openstack-kolla23:01
inc0dgedia: error will be somewhere above23:02
*** dave-mccowan has joined #openstack-kolla23:06
*** lrensing has quit IRC23:07
*** hongbin has quit IRC23:21
*** salv-orlando has joined #openstack-kolla23:34
*** salv-orlando has quit IRC23:39
*** ekimshi has quit IRC23:44
*** goldyfruit has quit IRC23:46
*** dardelean__ has quit IRC23:48
*** danardelean has joined #openstack-kolla23:50
*** dgedia has quit IRC23:52

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!