Tuesday, 2019-10-08

*** BjoernT has joined #openstack-kolla01:40
*** BjoernT has quit IRC01:43
*** igordc has quit IRC02:52
*** negronjl has quit IRC02:59
*** goldyfruit_ has quit IRC03:05
*** goldyfruit has joined #openstack-kolla03:06
*** negronjl has joined #openstack-kolla03:07
*** goldyfruit has quit IRC03:08
*** goldyfruit has joined #openstack-kolla03:11
*** goldyfruit has quit IRC03:12
*** igordc has joined #openstack-kolla03:19
*** radeks_ has joined #openstack-kolla04:21
*** dave-mccowan has quit IRC04:41
*** nawar has joined #openstack-kolla05:48
openstackgerritMerged openstack/kolla stable/stein: build: enable Octavia for Ubuntu/binary  https://review.opendev.org/68712505:54
osmanlicilegimorning06:12
*** negronjl has quit IRC06:14
*** negronjl has joined #openstack-kolla06:17
*** cah_link has joined #openstack-kolla06:18
yoctozeptomorning06:36
*** nawar has left #openstack-kolla07:11
mnasiadkamorning07:13
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Implement IPv6 support in the control plane  https://review.opendev.org/68157307:24
*** lklimin has joined #openstack-kolla07:27
openstackgerritDincer Celik proposed openstack/kolla-ansible master: Remove /etc/hosts entries pointing hostname to localhost  https://review.opendev.org/68523307:45
hrwelo07:55
mgoddardmorning08:11
*** k_mouza has joined #openstack-kolla08:13
osmanlicilegiguys, what's the proper way to set a variable to every build of kolla? I'd like to use apt proxy for every build and I've tried setting http_proxy in base_header but didn't work. do I need to set http_proxy for every image?08:20
hrwosmanlicilegi: in base. all are based on base08:22
osmanlicilegiso I'm not wrong but it didn't work :|08:22
hrwosmanlicilegi: easiest way? replaces sources.list file in docker/base/08:24
hrwosmanlicilegi: http://apt-cacher-ng-ip:port/deb.debian.org/ way08:24
hrwshould work but not tried08:24
osmanlicilegiI'll give it a try08:25
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Docs: improve Nova documentation  https://review.opendev.org/68699508:30
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/stein: Fix swift-proxy-server memcached configuration  https://review.opendev.org/68723608:31
hrwosmanlicilegi: https://paste.debian.net/1105318/08:31
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/rocky: Fix swift-proxy-server memcached configuration  https://review.opendev.org/68723708:31
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/queens: Fix swift-proxy-server memcached configuration  https://review.opendev.org/68723808:31
*** dswebb has quit IRC08:31
openstackgerritIsaac Prior proposed openstack/kayobe master: Add ansible tag header to Host Configuration doc page [WIP]  https://review.opendev.org/68724008:34
*** dswebb has joined #openstack-kolla08:34
*** rpittau|afk is now known as rpittau08:38
*** k_mouza has quit IRC08:41
mnasiadkaosmanlicilegi: Kolla-build picks up HTTP(S)_PROXY var from env and passes that to containers - that doesn't work?08:42
mnasiadkaosmanlicilegi: https://docs.openstack.org/kolla/latest/admin/image-building.html#building-behind-a-proxy08:42
*** k_mouza has joined #openstack-kolla08:44
osmanlicilegimnasiadka: unfortunately it does not work for me. when I set the proxy at {% block base_header %} and unset them at {% block base_footer %}, only the base build goes to proxy.08:47
*** priteau has joined #openstack-kolla08:47
hrwosmanlicilegi: so do not unset it08:49
mnasiadkaosmanlicilegi: tried "export HTTP_PROXY=bla:port" and then kolla-build ?08:49
mnasiadkaor use kolla-build --build-args HTTP_PROXY=blah:port08:51
mgoddardosmanlicilegi: if you want to unset, do it in 'block footer'08:56
mgoddardthat will always be in the last image08:56
hrwthat way it will be set for all images (due to base_header) and unset for all images (due to footer being in each image)08:56
mgoddardyou can also just use 'header' rather than 'base_header'09:00
osmanlicilegigot it. guys you're great! you'll all have a cold beer whenever I see you \m/ u_u \m/09:01
openstackgerritMerged openstack/kolla-ansible master: Use internal API for heat -> heat communication  https://review.opendev.org/67671609:03
mgoddardosmanlicilegi: if you think the docs could be clearer (always possible), feel free to send a patch :)09:09
openstackgerritAlfredo Moralejo proposed openstack/kolla master: Remove designate-pool-manager container  https://review.opendev.org/68725309:09
*** ondrejduchon has joined #openstack-kolla09:17
*** ondrejduchon_ has joined #openstack-kolla09:19
*** ondrejduchon has quit IRC09:20
*** ondrejduchon_ has quit IRC09:21
stingrayzahappy tuesday folks o/09:21
yoctozeptostingrayza: is there some holiday somewhere?09:22
stingrayzaperhaps, not here though ;)09:24
stingrayzajust trying to brighten up your morning yoctozepto :D09:24
yoctozeptowell then, happy Tuesday09:24
openstackgerritMerged openstack/kolla master: Deprecate Almanach images  https://review.opendev.org/68661609:26
*** ondrejduchon has joined #openstack-kolla09:27
ondrejduchonmgoddard: Hello, could you please check modified logging.conf feature https://review.opendev.org/#/c/679812/5. Thank you09:28
mgoddardhi ondrejduchon, we're in feature freeze now, so it will have to wait for the Ussuri cycle. I have quite a busy few days ahead09:29
*** pcaruana has joined #openstack-kolla09:30
hrwtwo weeks ago tuesday meant taco tuesday ;D09:33
*** dswebb has quit IRC09:36
yoctozeptohrw: you fancy tacos?09:39
*** Blinkiz has joined #openstack-kolla09:39
hrwyoctozepto: I was in San Diego. they have tacos every 50m09:39
yoctozeptooh my09:41
hrwin selected locations of course09:43
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Implement IPv6 support in the control plane  https://review.opendev.org/68157310:13
*** k_mouza has quit IRC10:16
openstackgerritIsaac Prior proposed openstack/kayobe master: Add ansible tag headers to Host Configuration doc page  https://review.opendev.org/68724010:16
openstackgerritIsaac Prior proposed openstack/kayobe master: Add ansible tag headers to Host Configuration doc page  https://review.opendev.org/68724010:19
*** k_mouza has joined #openstack-kolla10:20
openstackgerritIsaac Prior proposed openstack/kayobe master: Add ansible tag headers to Host Configuration doc page  https://review.opendev.org/68724010:22
openstackgerritJan Vondra proposed openstack/kolla-ansible master: Added RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS option to rabbitmq-env.conf  https://review.opendev.org/68636910:35
*** k_mouza has quit IRC10:36
*** k_mouza has joined #openstack-kolla10:37
*** k_mouza has quit IRC11:01
*** k_mouza has joined #openstack-kolla11:02
*** vmixor has joined #openstack-kolla11:06
*** gfidente has quit IRC11:09
primeministerpHi Everyone, having a problem when trying to deploy with Ubuntu images built from source on 18.04 arm64.  I've been able to reproduce w/ Debian images built from source on Buster arm64 also.11:11
primeministerpRUNNING HANDLER [rabbitmq : Waiting for rabbitmq to start on first node] ******************************************************************************************************************************************11:11
primeministerpfatal: [localhost]: FAILED! => {"changed": true, "cmd": "docker exec rabbitmq rabbitmqctl wait /var/lib/rabbitmq/mnesia/rabbitmq.pid", "delta": "0:00:00.648651", "end": "2019-10-07 19:13:00.734522", "msg": "non-zero return code", "rc": 137, "start": "2019-10-07 19:13:00.085871", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}11:11
*** gfidente has joined #openstack-kolla11:11
hrwprimeministerp: and you are on ipv4, right?11:12
primeministerpyes11:13
primeministerpseems that it connect to the rabbit ip11:13
primeministerpERROR: epmd error for host ampere01: address (cannot connect to host/port)11:13
primeministerpthat's in the startup_log11:13
primeministerpufw maybe/11:13
yoctozeptoprimeministerp: does hostname resolve to the api address?11:16
primeministerpyes11:17
primeministerpgoing to check /etc/hosts11:17
hrw"host ampere01:"11:18
hrwcan any node resolve 'ampere01' to proper IP?11:19
hrwI workarounded it with creating 10.101.*.* entries in one of my domains and using resulting names11:19
primeministerpit's only one node11:20
primeministerpI think it was a 127.0.1.1 address that was causing it the issue in /etc/hosts11:20
hrwstill matters as containers need to resolve name11:20
primeministerpI'll know i a sec11:20
*** kplant has joined #openstack-kolla11:21
hrwprimeministerp: 127.x.x.x == localhost anyway so you can not connect outside of host. and container != host11:21
primeministerphrw, yes I know.  It was a residual from the installation.  Looking much better now.11:22
primeministerphrw, fwiw, the issue w/ the libvirt not recognizing aarch64 didn't raise it's head after I reinstalled.  I must have corrupted something somewhere before.11:23
hrwok11:27
*** dave-mccowan has joined #openstack-kolla11:28
yoctozeptoprimeministerp, hrw, BaptisteGer: the hostname issues in Ubuntu is what osmanlicilegi is working on atm11:35
yoctozeptothe 127.0.1.1 addressing11:35
yoctozeptocausing troubles for rabbitmq11:36
hrwcool11:37
BaptisteGergreat!11:38
hrwopenstackgerrit: thanks to osmanlicilegi for working on rabbitmq hostname issue in kolla11:38
yoctozeptoxD11:38
yoctozeptowell, the workaround is to just manually remove the offending entry in /etc/hosts11:39
yoctozeptoand disable any cloud-init stuff that would overwrite it11:39
yoctozeptoBaptisteGer: you should just check your hostname if it resolves to the correct address11:41
yoctozeptoand adjust any source which makes it wrong11:41
yoctozeptoeh buggy ansible https://github.com/ansible/ansible/issues/6322711:41
mnasiadkayoctozepto: use 2.8.5 :)11:42
yoctozeptomnasiadka: yeah, I reported that...11:43
* yoctozepto sends mnasiadka coffee11:43
mnasiadkaWell, this issue will be fixed one year later - by Ansible dev standards11:44
mnasiadkaSo that's why I said, what I said :)11:44
yoctozeptothat's why we shall upgrade minimum in Ussuri11:45
BaptisteGeryoctozepto: It does resolve to the ipv6 address (since it is the topmost entry). Will look into it !11:46
yoctozeptoand for train just in ipv6 docs11:46
yoctozeptoBaptisteGer: make sure it does not resolve to an IPv4 address11:46
primeministerpyoctozepto, thanks for the info11:46
yoctozeptoor it will be preferred11:46
BaptisteGeryoctozepto: yup!11:47
*** k_mouza_ has joined #openstack-kolla11:51
*** k_mouza has quit IRC11:54
*** k_mouza_ has quit IRC12:01
*** k_mouza has joined #openstack-kolla12:03
*** k_mouza has quit IRC12:06
*** k_mouza has joined #openstack-kolla12:07
*** k_mouza has quit IRC12:07
*** k_mouza has joined #openstack-kolla12:08
primeministerpGuys, thanks for the help, everything seems to deployed now.12:15
*** jdwidari has joined #openstack-kolla12:22
*** k_mouza has quit IRC12:28
*** kplant has quit IRC12:28
*** k_mouza has joined #openstack-kolla12:30
primeministerpSo is there some specific metadata needed at the flavor and image level to get vms, to launch properly on aarch64?12:40
*** pcaruana has quit IRC12:53
*** vmixor has quit IRC13:02
*** vmixor has joined #openstack-kolla13:04
*** vmixor has quit IRC13:24
*** mixor has joined #openstack-kolla13:24
*** goldyfruit has joined #openstack-kolla13:28
*** cah_link has quit IRC13:34
primeministerpyoctozepto, so I'm still having issues launching images on aarch6413:36
*** goldyfruit_ has joined #openstack-kolla13:36
primeministerpyoctozepto, virCapabilitiesDomainDataLookupInternal:735 : invalid argument: could not find capabilities for domaintype=kvm13:37
primeministerpyoctozepto, any ideas?13:38
*** goldyfruit has quit IRC13:39
hrwprimeministerp: two things:13:40
hrwprimeministerp: 1. master or stable/stein or older?13:40
hrwprimeministerp: 2. pastebin nova-compute/nova-libvirt logs when you create instance13:41
*** pbing19 has joined #openstack-kolla13:42
primeministerpmaster13:47
*** k_mouza has quit IRC13:48
*** k_mouza_ has joined #openstack-kolla13:48
primeministerphrw, nova compute log: http://paste.openstack.org/show/782100/13:48
hrwthx13:50
primeministerphrw, libvirt log: http://paste.openstack.org/show/782101/13:50
hrw2019-10-08 12:49:58.096+0000: 28852: error : virStorageFileReportBrokenChain:4780 : Cannot access storage file './debian-10.1.0-arm64-netinst.iso': No such file or directory13:51
primeministerphrw, the bit at the end is me trying to use virtinstall against the containerized libvirt.  It was successful against the local instance13:51
hrwthis is not from openstack13:51
hrwprimeministerp: ls -l /dev/kvm ?13:51
primeministerproot@ampere01:/var/log/kolla/libvirt# kvm-ok13:52
primeministerpINFO: /dev/kvm exists13:52
primeministerpKVM acceleration can be used13:52
hrwin nova-libvirt container?13:52
primeministerp(nova-libvirt)[root@ampere01 /]# ls /dev/kvm13:52
primeministerp(nova-lib13:52
primeministerpit's ther13:53
primeministerpe13:53
primeministerpcrw-rw---- 1 root 113 10, 232 Oct  8 08:44 /dev/kvm13:53
primeministerpthat's from in the container13:53
hrw(nova-libvirt)[root@c8n1 /]# ls -l /dev/kvm13:54
hrwcrw-rw---- 1 root qemu 10, 232 Oct  2 15:11 /dev/kvm13:54
primeministerp(nova-libvirt)[root@ampere01 images]# ls -l /dev/kvm13:54
primeministerpcrw-rw---- 1 root 113 10, 232 Oct  8 08:44 /dev/kvm13:54
primeministerp(nova-libvirt)[root@ampere01 images]#13:54
hrwweird13:55
primeministerpyou think it's metadata?13:55
primeministerpdo i need a architecture=aarch64 on the image?13:55
hrwno need for any metadata since queens13:55
*** pbing19 has quit IRC13:55
hrwI rather wonder why /dev/kvm has wrong permissions13:56
*** pbing19 has joined #openstack-kolla13:56
primeministerpwhat should the permissions be?13:57
primeministerpthat's the same as the underlying host13:57
hrwespecially when /usr/local/bin/kolla_extend_start takes care of permissions. 660 root:qemu /dev/kvm13:57
hrwprimeministerp: that maybe also a reason why xml for your VM instance looks so basic13:58
hrwhttps://paste.debian.net/1105368/ is my standard cirros one13:58
hrwprimeministerp: pastebin globals.yml please13:59
primeministerphrw, http://paste.openstack.org/show/782102/14:00
hrwthx14:01
*** pbing19 has quit IRC14:05
*** pbing19 has joined #openstack-kolla14:13
*** k_mouza_ has quit IRC14:14
openstackgerritMerged openstack/kolla master: Remove designate-pool-manager container  https://review.opendev.org/68725314:16
*** k_mouza has joined #openstack-kolla14:18
*** k_mouza has quit IRC14:19
*** k_mouza has joined #openstack-kolla14:20
yoctozeptoprimeministerp, hrw: where are permissions wrong? looking good14:34
yoctozeptolooks like arm issue, hrw must help you, I can't really, only some rpi laying around ;p14:35
hrwrebuilding images for deploy...14:44
*** pcaruana has joined #openstack-kolla14:46
hrwERROR:kolla.common.utils.kolla-toolbox:error creating overlay mount to /var/lib/docker/overlay2/24a013caae532b0f020b0942346bb1693508c4e2e74d8067dfb3ddcd4e4740d2-init/merged: device or resource busy14:49
hrwfscking docker shit14:49
openstackgerritIsaac Prior proposed openstack/kayobe master: Add ansible tag headers to Host Configuration doc page  https://review.opendev.org/68724014:50
*** k_mouza has quit IRC14:52
*** k_mouza_ has joined #openstack-kolla14:52
yoctozeptohrw: no, it's just some busy resource15:01
yoctozeptowe are busy, resources are too15:01
*** ivve has joined #openstack-kolla15:03
hrwyoctozepto: it is a bug in docker. trust me ;d15:03
hrwjust do not want to go through browser history to find link15:03
yoctozeptoxD15:03
yoctozeptook15:04
hrwoverlay2 abuses overlayfs15:04
yoctozeptoa-busy-es :D15:07
hrwyoctozepto: [18395.398800] overlayfs: lowerdir is in-use as upperdir/workdir15:30
*** ivve has quit IRC15:36
yoctozeptohrw: ah, that looks like a bug indeed15:40
hrwI call it a day15:42
hrwmy build machine crashed nth time15:43
hrwtoday15:43
*** rpittau is now known as rpittau|afk15:47
yoctozeptohrw: :<15:48
*** goldyfruit___ has joined #openstack-kolla16:02
*** goldyfruit_ has quit IRC16:05
*** ondrejduchon has quit IRC16:09
*** pbing19 has quit IRC16:12
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Implement IPv6 support in the control plane  https://review.opendev.org/68157316:15
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Implement IPv6 support in the control plane  https://review.opendev.org/68157316:18
*** pbing19 has joined #openstack-kolla16:25
*** igordc has quit IRC16:25
*** k_mouza_ has quit IRC16:32
*** k_mouza has joined #openstack-kolla16:32
*** k_mouza_ has joined #openstack-kolla16:33
*** mixor has quit IRC16:37
*** k_mouza has quit IRC16:37
*** k_mouza_ has quit IRC16:37
*** vmixor has joined #openstack-kolla16:40
*** factor has quit IRC16:43
*** k_mouza has joined #openstack-kolla16:53
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Implement IPv6 support in the control plane  https://review.opendev.org/68157316:56
*** ivve has joined #openstack-kolla16:57
*** k_mouza has quit IRC16:58
*** priteau has quit IRC17:07
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Implement IPv6 support in the control plane  https://review.opendev.org/68157317:17
*** kplant has joined #openstack-kolla17:23
kplantis it still a possibility that the prometheus roles are being removed from k-a?17:25
*** dave-mccowan has quit IRC17:28
blineanyone seen this error from magnum: https://pastebin.com/vFJMAciM17:31
blinefresh install; creating swarm mode cluster17:32
blinekolla from latest git checkout, stein binary centos17:35
*** igordc has joined #openstack-kolla17:46
*** dave-mccowan has joined #openstack-kolla17:48
*** negronjl has quit IRC17:53
*** negronjl has joined #openstack-kolla17:55
yoctozeptokplant: I doubt18:09
*** gfidente is now known as gfidente|afk18:10
yoctozeptobline: nope, and no magnum experts around, better get more info at #openstack-containers18:11
yoctozeptobline: last git checkout would be train though, so make sure you are on the stable/stein branch in kolla-ansible too18:12
yoctozeptonot sure if there are any relevant changes between releases18:13
yoctozeptoprobably not18:13
*** vmixor has quit IRC18:28
*** ianw_pto is now known as ianw18:59
*** dave-mccowan has quit IRC19:10
*** kplant has quit IRC19:44
*** pbing19 has quit IRC19:52
*** pbing19 has joined #openstack-kolla19:54
*** dave-mccowan has joined #openstack-kolla19:59
*** jdwidari has quit IRC20:06
*** pcaruana has quit IRC20:35
*** gfidente|afk is now known as gfidente20:50
*** igordc has quit IRC20:59
*** igordc has joined #openstack-kolla21:00
*** jbadiapa has quit IRC21:03
*** gfidente is now known as gfidente|afk21:19
*** ivve has quit IRC22:03
*** goldyfruit_ has joined #openstack-kolla22:05
*** pbing19 has quit IRC22:07
*** goldyfruit___ has quit IRC22:07
blinethanks yoctozepto, I was using master, going to try with the stable/stein branch and see if that fixes it22:13
*** crindi has joined #openstack-kolla22:23
*** goldyfruit_ has quit IRC22:39
*** k_mouza has joined #openstack-kolla23:05
*** k_mouza has quit IRC23:09

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!