Friday, 2019-12-06

*** schwicht has quit IRC00:02
*** xinliang has joined #openstack-kolla00:02
*** schwicht has joined #openstack-kolla00:35
*** jcmdln has joined #openstack-kolla00:38
*** zhanglong has joined #openstack-kolla00:45
*** zhanglong has quit IRC01:02
*** zhanglong has joined #openstack-kolla01:09
*** ricolin has joined #openstack-kolla01:10
*** schwicht has quit IRC01:26
*** zhanglong has quit IRC01:41
*** zhanglong has joined #openstack-kolla01:53
openstackgerritYU CHENGDE proposed openstack/kolla master: Let nova-libvirt restarting be independent of libvirtd.log  https://review.opendev.org/69760501:54
*** schwicht has joined #openstack-kolla01:59
*** schwicht has quit IRC02:22
*** schwicht has joined #openstack-kolla02:25
*** rphillips has quit IRC03:20
*** rphillips has joined #openstack-kolla03:32
*** schwicht has quit IRC03:34
*** openstackstatus has joined #openstack-kolla03:41
*** ChanServ sets mode: +v openstackstatus03:41
*** dave-mccowan has quit IRC04:04
*** zhanglong has quit IRC04:04
*** xinliang has quit IRC04:09
*** ricolin has quit IRC04:16
*** ricolin has joined #openstack-kolla04:18
*** schwicht has joined #openstack-kolla04:30
*** zhanglong has joined #openstack-kolla04:56
*** raukadah is now known as chkumar|ruck05:20
*** skramaja has joined #openstack-kolla05:31
*** obre has quit IRC05:34
*** obre has joined #openstack-kolla05:41
*** zhanglong has quit IRC06:03
yoctozeptomorning06:42
*** zhanglong has joined #openstack-kolla06:50
*** sri_ has joined #openstack-kolla06:51
sri_Morning06:53
*** zhanglong has quit IRC06:59
mnasiadkaMorning07:01
*** zhanglong has joined #openstack-kolla07:04
*** xinliang has joined #openstack-kolla07:15
*** cah_link has joined #openstack-kolla07:22
*** dpawlik has joined #openstack-kolla08:02
*** pcaruana has joined #openstack-kolla08:11
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637608:13
*** rpittau|afk is now known as rpittau08:21
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637608:37
hrwyoł09:00
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637609:10
*** gfidente|afk is now known as gfidente09:11
*** dswebb has quit IRC09:19
openstackgerritMerged openstack/kolla stable/stein: Install etcd3gw to fix Ubuntu binary tooz coordination  https://review.opendev.org/69742709:27
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: base: use upstream dumb-init on all architectures  https://review.opendev.org/69637009:30
hrwpep8/yaml fix09:30
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: add more tests for enable_repos template method  https://review.opendev.org/69735409:31
hrwrebased09:31
hrwtrivial one anyone? https://review.opendev.org/#/c/697351/09:31
patchbotpatch 697351 - kolla - releasenotes: fix file location - 1 patch set09:31
*** tonythomas has joined #openstack-kolla09:31
mnasiadkaCRITICAL neutron [-] Unhandled error: AttributeError: 'module' object has no attribute 'abc'09:36
mnasiadkatodays winner09:36
mnasiadkahrw, yoctozepto: so, we have another python2 vs 3 issue - https://8838e06c389b32ad359d-4ca7e044918ce86ea788bdbe0e5e3d01.ssl.cf1.rackcdn.com/676128/18/check/kolla-ansible-centos-source/8a2a622/primary/logs/kolla/neutron/neutron-dhcp-agent.txt.gz09:41
mnasiadkathis is not easily solvable09:44
mnasiadka...09:44
*** xinliang has quit IRC09:45
*** k_mouza has joined #openstack-kolla09:45
hrwmnasiadka: add neutron-dhcp-agent into UNBUILDABLE_IMAGES?09:46
hrw:D09:46
mnasiadkahrw: well, we can then stop running kolla-ansible-centos-source I guess09:46
mnasiadka:)09:46
hrwor define centos7 unsupportable09:47
mnasiadkayup09:49
mnasiadkamgoddard: ^^09:49
hrwCI blocker as well09:54
mnasiadkayup09:54
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637609:54
yoctozeptomgoddard, mnasiadka, hrw: you have my +2 for stopping centos jobs on ussuri for now - no point in this pain09:59
openstackgerritMark Goddard proposed openstack/kayobe stable/stein: Task to generate nginx.conf in inspection-store missing `become: True`  https://review.opendev.org/69764110:04
openstackgerritMark Goddard proposed openstack/kayobe stable/rocky: Task to generate nginx.conf in inspection-store missing `become: True`  https://review.opendev.org/69764210:04
mgoddardmorning10:05
hrwmgoddard: morning10:05
hrwmgoddard: morning discussion: to drop c7 or not10:05
mgoddardhrw: just catching up10:05
hrw;d10:05
mgoddardseems we are busted10:07
mgoddardI'd like to see how far we can get with centos8+source10:07
mgoddardI'll spend some time on it today10:08
mgoddardhrw: did you have a patch that uses copr?10:08
hrwmgoddard: moment10:08
*** born2bake has joined #openstack-kolla10:09
hrwmgoddard: apply https://paste.centos.org/view/9c6a7731 on top of https://review.opendev.org/692368 one10:12
patchbotpatch 692368 - kolla - WIP: move to CentOS 8 in containers - 22 patch sets10:12
mgoddardhrw: thanks10:13
hrwmgoddard: centos8 does not give nova/neutron/cinder/glance etc due to lack of ceph10:13
mgoddardhrw: so we'll remove ceph :)10:14
hrw;D10:14
mgoddardbetter than no coverage10:14
hrwsure10:14
mgoddardremoving CI could get messy for bug fixes that we want to backport10:14
hrwmgoddard: 692368 requires rebase and adapting repos.yaml10:15
mgoddardk10:15
hrwmgoddard: for start you can even skip repos.yaml if 'dnf config-manager' will not complain about enabling unknown repos10:16
*** vmixor has joined #openstack-kolla10:16
yoctozeptomgoddard: going for c8 and removing ceph is similarly messy for backports :-)10:19
yoctozeptono way to address this nicely10:19
hrwif rpm and python3 then c8 else c710:20
yoctozeptomgoddard, hrw: what about pinning c7 ci to train branches instead?10:22
mgoddardyoctozepto: then it would not be CI :)10:22
yoctozeptomgoddard: for c7 which you stop at train?10:22
yoctozeptoit seems CI all right10:22
yoctozeptoand then you have nice backports :D10:22
yoctozeptoall solutions are bad10:23
yoctozeptojust giving another one10:23
*** vmixor has quit IRC10:24
hrwI like the idea of 'let get c8 at whatever is there' 🙃10:25
mgoddardyoctozepto: I guess we could deploy train containers using master k-a10:25
mgoddardI'd like to see if we can move forward before going backward though10:25
yoctozeptomgoddard: +1 for c8 already10:26
yoctozeptoc7 can be separate jobs10:26
mgoddardhrw: seems dnf config-manager disable does not seem to ignore errors :/10:26
yoctozeptojust food for thought10:26
yoctozeptoI'm a bit busy with other things but will review if you ping10:27
hrwmgoddard: you rebased?10:27
mgoddardhrw: yes10:27
hrwmgoddard: please push to gerrit10:27
mgoddardhrw: not pushed yet10:27
mgoddardstill tinkering10:27
hrwok10:27
*** k_mouza has quit IRC10:27
hrwmgoddard: https://review.opendev.org/#/c/613815/ would be nice to have probably.10:28
patchbotpatch 613815 - kolla - Remove tgtd and scsi-target-utils support in CentO... - 12 patch sets10:28
hrwmgoddard: bridge-utils and btrfs stuff is also not present in rhel/c810:28
hrwrebases fine on master10:29
mgoddardhrw: need something on the k-a side before removing tgtd10:30
hrwright10:30
mgoddardhrw: https://review.opendev.org/63147910:30
patchbotpatch 631479 - kolla-ansible - Configure Cinder to use lioadm on RPM-based distros - 10 patch sets10:30
mgoddardhrw: I think safest option might be to make this depend on c7 vs c810:30
hrwsi10:31
mgoddardbecause the problem was always migration from tgtd to lio10:31
mgoddardnow is our chance to switch10:31
hrwhttps://twitter.com/haerwu/status/120289846962052710410:32
mgoddardhrw: :)10:32
*** zhanglong has quit IRC10:34
*** k_mouza has joined #openstack-kolla10:38
hrwmgoddard: in worst case we can make kolla copr and put there ceph from https://koji.mbox.centos.org/koji/buildinfo?buildID=58210:40
born2bakeHi, I am wondering if octavia image was updated and fixed https://bugs.launchpad.net/kolla-ansible/+bug/1847905 ; Is there possibility to update images so I can make sure that latest fixes are applied?10:43
openstackLaunchpad bug 1847905 in kolla-ansible "Octavia loadbalancer pending create state" [Undecided,New]10:43
hrwmgoddard: https://copr.fedorainfracloud.org/coprs/hrw/kolla-extras/packages/10:45
mnasiadkahrw: last time I checked there were some deps missing to build ceph for el810:48
mnasiadkahrw: https://trello.com/b/ZAMLb0pT/ceph-el810:51
mnasiadkahrw: and essentially https://copr.fedorainfracloud.org/coprs/ktdreyer/ceph-el8/10:53
hrwmnasiadka: great!10:53
hrwdropped kolla-extras then10:55
hrwit's centos. to build one out-of-distro package you first need to add gazillion of deps10:56
mgoddardhrw: 40411:00
hrwmgoddard: dropped11:00
yoctozeptomgoddard: look at mnasiadka's11:10
* hrw out11:31
*** zbr has quit IRC11:47
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637611:47
*** zbr has joined #openstack-kolla11:48
*** k_mouza has quit IRC11:54
*** k_mouza has joined #openstack-kolla11:57
*** negronjl has quit IRC12:14
*** negronjl has joined #openstack-kolla12:16
*** k_mouza has quit IRC12:24
mnasiadkagfidente: around?12:26
gfidentemnasiadka i12:27
mnasiadkagfidente: any advice how to obtain ceph packages for centos8? any fedora builds or something like that?12:27
gfidentemnasiadka yeah we don't have them yet12:27
gfidentewe haven't been able to start building them yet :(12:28
openstackgerritMark Goddard proposed openstack/kolla master: Switch to stable/train for source images on CentOS/RHEL  https://review.opendev.org/69765812:36
mgoddardyoctozepto, hrw, mnasiadka: Trying ^ as an alternative to c8. Switch source tarballs to stable/train12:38
hrwmgoddard: if python2 then stable/train?12:38
mgoddardhrw: if centos/rhel then stable/train12:39
mgoddardto keep us sane :)12:39
hrwmgoddard: you just broke tripleo812:39
mgoddardhrw: how so? source only12:39
hrwok12:39
mgoddardtripleo doesn't do source12:39
hrwlet's see how it pass zuul12:39
hrw+112:40
yoctozeptomgoddard: glad my solution wins12:40
mgoddardyoctozepto: I didn't realise that's what your solution was12:41
mgoddardI thought you meant pin kolla & k-a to train12:41
mgoddardor use train images from dockerhub12:41
mgoddardanyways12:41
yoctozeptowith pinning I meant this, no problem, sir12:41
*** schwicht has joined #openstack-kolla12:46
dirtwashso we trying to deploy our test cluster again but now after cleanup it fails every time with : Waiting for nova-compute services to register themselves , the container logs dont say anything but some python package warning, any ideas what to look for? i pruned docker volumes and rebooted hosts12:51
dirtwashpretty much default config, just the most basic test setup12:52
*** schwicht has quit IRC12:52
hrwshow logs?12:52
dirtwashhrw: http://paste.openstack.org/show/da5z13K9JJBLUG1Fzbuv/12:55
dirtwashthe docker logs all show the same for all the nova containers12:55
dirtwashubuntu 180412:55
yoctozeptodirtwash: /var/log/kolla on the host12:57
yoctozeptocontainer logs are usually very limited12:57
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637612:57
dirtwashyoctozepto: which one to look at though, there's plenty12:58
yoctozeptonova/nova-compute.log12:58
dirtwashhm libvirtd error12:59
dirtwashyoctozepto: thanks, I'll figure it out from here13:00
hrwdirtwash: x86?13:00
*** kplant has joined #openstack-kolla13:00
dirtwashhrw: yea13:00
dirtwashbut libvirtd is running fine hm13:00
dirtwasherror is  libvirtError: unable to connect to server at '10.3.0.197:16509': Connection refused13:01
openstackgerritPierre Riteau proposed openstack/kayobe stable/stein: Task to generate nginx.conf in inspection-store missing `become: True`  https://review.opendev.org/69764113:10
openstackgerritPierre Riteau proposed openstack/kayobe stable/stein: Fix kolla-ansible role tests  https://review.opendev.org/69766213:10
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Improve Apache logging  https://review.opendev.org/69726213:11
*** yan0s has joined #openstack-kolla13:12
*** schwicht has joined #openstack-kolla13:21
*** schwicht has quit IRC13:26
*** sri_ has quit IRC13:30
openstackgerritWill Szumski proposed openstack/kayobe master: Use OpenStack Train release  https://review.opendev.org/69461613:35
*** skramaja has quit IRC13:36
openstackgerritMark Goddard proposed openstack/kolla master: Switch to stable/train for source images on CentOS/RHEL  https://review.opendev.org/69765813:37
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637613:38
*** mgoddard has quit IRC13:44
*** mgoddard has joined #openstack-kolla13:45
openstackgerritWill Szumski proposed openstack/kayobe master: Improve detection of base path  https://review.opendev.org/65586413:47
*** mgoddard has quit IRC13:58
*** mgoddard has joined #openstack-kolla14:05
*** cah_link has quit IRC14:06
*** cah_link has joined #openstack-kolla14:15
*** schwicht has joined #openstack-kolla14:22
*** cah_link has quit IRC14:25
hrwhave a nice weekend14:27
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637614:28
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637614:28
mgoddardyou too hrw14:29
*** graysonh has joined #openstack-kolla14:40
*** goldyfruit___ has joined #openstack-kolla14:41
dirtwashcan cinder used without backup pool?14:47
*** negronjl has quit IRC14:51
dirtwashman the openstack docs are utterly useless14:52
*** negronjl has joined #openstack-kolla14:53
*** chkumar|ruck is now known as raukadah15:01
openstackgerritMark Goddard proposed openstack/kayobe stable/rocky: Fix kolla-ansible role tests  https://review.opendev.org/69767315:03
*** dave-mccowan has joined #openstack-kolla15:05
*** goldyfruit___ is now known as goldyfruit15:07
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637615:10
*** dave-mccowan has quit IRC15:10
yoctozeptodirtwash: docs have varying quality15:13
yoctozeptocinder-backup can be disabled15:13
yoctozeptoif you don't want it :-)15:13
openstackgerritMark Goddard proposed openstack/kolla master: WIP: move to CentOS 8 in containers  https://review.opendev.org/69236815:15
mgoddardLet's see how it goes ^15:16
*** KeithMnemonic has joined #openstack-kolla15:29
mgoddardmnasiadka: https://review.opendev.org/#/c/69765815:35
patchbotpatch 697658 - kolla - Switch to stable/train for source images on CentOS... - 2 patch sets15:35
mnasiadkamgoddard: off we go15:36
openstackgerritRadosÅ‚aw Piliszek proposed openstack/kolla master: ovn-base: Create /run/ovn in extend_start.sh  https://review.opendev.org/69714815:38
openstackgerritRadosÅ‚aw Piliszek proposed openstack/kolla master: Disable building binary images for senlin-conductor and senlin-health-manager on debian/ubuntu  https://review.opendev.org/69744515:38
yoctozepto^ rebased15:38
openstackgerritMark Goddard proposed openstack/kolla master: Revert "Introduce macro to remove Py3 only plugins on CentOS 7"  https://review.opendev.org/69768015:43
* mgoddard cleans up15:44
yoctozeptovery good mgoddard15:46
yoctozeptoI'm gonna fix https://review.opendev.org/697478 if you don't mind :D15:46
patchbotpatch 697478 - kolla-ansible - CI: Don't fail on exepected critical log messages - 2 patch sets15:46
openstackgerritMerged openstack/kayobe master: Remove workaround for kolla-ansible post-deploy  https://review.opendev.org/69611015:50
openstackgerritRadosÅ‚aw Piliszek proposed openstack/kolla-ansible master: CI: Don't fail on expected critical log messages  https://review.opendev.org/69747815:53
yoctozeptomgoddard: ^ I think this will do the job15:55
mgoddardyoctozepto: thx15:55
*** yan0s has quit IRC15:59
openstackgerritMark Goddard proposed openstack/kayobe master: Improve docs on kolla-ansible  https://review.opendev.org/69768215:59
openstackgerritPierre Riteau proposed openstack/kayobe master: Use {{ openstack_branch }} as version of kolla-ansible in ansible tests  https://review.opendev.org/69768416:06
openstackgerritPierre Riteau proposed openstack/kayobe stable/rocky: Task to generate nginx.conf in inspection-store missing `become: True`  https://review.opendev.org/69764216:12
*** dpawlik has quit IRC16:14
*** graysonh has quit IRC16:20
*** graysonh has joined #openstack-kolla16:20
openstackgerritMark Goddard proposed openstack/kolla-ansible master: WIP: CI: Start testing CentOS 8 hosts  https://review.opendev.org/69354416:25
openstackgerritMark Goddard proposed openstack/kolla-ansible master: WIP: Disable fluentd in CI for CentOS 8  https://review.opendev.org/69769016:27
openstackgerritMark Goddard proposed openstack/kolla master: WIP: move to CentOS 8 in containers  https://review.opendev.org/69236816:27
openstackgerritPierre Riteau proposed openstack/kayobe master: Use {{ openstack_branch }} as version of kolla-ansible in ansible tests  https://review.opendev.org/69768416:30
*** graysonh has quit IRC16:36
*** graysonh has joined #openstack-kolla16:36
*** k_mouza has joined #openstack-kolla16:44
*** k_mouza has quit IRC16:48
*** rpittau is now known as rpittau|afk16:51
openstackgerritMerged openstack/kayobe master: Fix kayobe overcloud provision without seed in limit  https://review.opendev.org/69352617:05
openstackgerritMerged openstack/kayobe master: Add docker_registry_mirrors to docker.yml  https://review.opendev.org/69520617:06
openstackgerritMerged openstack/kayobe master: Fix use of IPA images in Bifrost without a checksum  https://review.opendev.org/69373917:06
openstackgerritMerged openstack/kayobe master: Stop running kolla-target-venv in seed hypervisor host upgrade  https://review.opendev.org/69298317:11
openstackgerritMerged openstack/kayobe master: Stop gathering facts for host command run  https://review.opendev.org/69298117:11
*** graysonh has quit IRC17:18
openstackgerritRadosÅ‚aw Piliszek proposed openstack/kolla master: Added python3 packages for mellanox agent container  https://review.opendev.org/69723617:19
*** KeithMnemonic has quit IRC17:22
*** pcaruana has quit IRC17:23
openstackgerritMark Goddard proposed openstack/kolla-ansible master: WIP: CI: Start testing CentOS 8 hosts  https://review.opendev.org/69354417:33
openstackgerritMark Goddard proposed openstack/kolla-ansible master: WIP: Disable fluentd in CI for CentOS 8  https://review.opendev.org/69769017:33
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Use local python interpreter for keystone cron generator  https://review.opendev.org/69770017:33
openstackgerritMark Goddard proposed openstack/kolla master: WIP: move to CentOS 8 in containers  https://review.opendev.org/69236817:34
openstackgerritWill Szumski proposed openstack/kayobe master: WIP: versioned links for release notes  https://review.opendev.org/69770117:35
openstackgerritMerged openstack/kolla master: Switch to stable/train for source images on CentOS/RHEL  https://review.opendev.org/69765817:38
*** negronjl has quit IRC17:39
yoctozeptomgoddard, mnasiadka, hrw: ^ yay, merged17:39
mgoddardwoop17:39
openstackgerritMark Goddard proposed openstack/kolla master: WIP: move to CentOS 8 in containers  https://review.opendev.org/69236817:41
openstackgerritWill Szumski proposed openstack/kayobe master: Adds support for Ansible 2.8, bump mimimum to 2.6  https://review.opendev.org/69770217:41
yoctozeptomnasiadka: https://review.opendev.org/676128 merge conflicts :<17:41
patchbotpatch 676128 - kolla - Build images with healthcheck scripts for core ser... - 18 patch sets17:41
mgoddardmnasiadka: https://review.opendev.org/#/c/69447617:44
patchbotpatch 694476 - kolla-ansible - Default to etcd3gw driver for etcd-based coordination - 5 patch sets17:44
*** negronjl has joined #openstack-kolla17:44
mnasiadkaMg17:45
mnasiadkaUps17:45
mnasiadkamgoddard: done17:45
yoctozeptomnasiadka: did you mean Magnesium?17:45
mgoddardsweet17:45
mnasiadkayoctozepto: aye, will fix on Mon17:45
yoctozeptomnasiadka: ack17:45
yoctozeptomgoddard: are not images in dockerhub b0rken atm?17:47
yoctozeptoas in w+1 from mnasiadka will break at gate17:47
mgoddardyoctozepto: you may be right17:48
yoctozeptoI'll add a depends-on17:48
mgoddardit won't work as the other patch has merged17:48
yoctozeptomgoddard: to a merging one17:48
mgoddardok17:48
openstackgerritRadosÅ‚aw Piliszek proposed openstack/kolla-ansible master: Default to etcd3gw driver for etcd-based coordination  https://review.opendev.org/69447617:49
yoctozeptomgoddard, mnasiadka: feel bad to w+1 myself https://review.opendev.org/69447617:49
patchbotpatch 694476 - kolla-ansible - Default to etcd3gw driver for etcd-based coordination - 6 patch sets17:49
mgoddardyoctozepto: I imagine it won't work as that one is already in the gate17:50
yoctozeptomgoddard: depends on timing, yeah17:51
yoctozeptowell, gate behaves weirdly, let's find out17:51
*** ricolin has quit IRC17:55
*** dcapone2004 has joined #openstack-kolla17:58
dcapone2004I have been testing kolla-ansible in an AIO environment for the past few months and I am having an issue relating to updates...I am unsure if this is an issue with AIO deployments only or if I am doing something wrong, or what...but essentially anytime there is an update to docker, I lose network connectivity to my VMs until the server is18:00
dcapone2004completely restarted18:00
openstackgerritGowrishankar proposed openstack/kolla master: Provision building netcontrold container  https://review.opendev.org/68909018:00
kplantdcapone2004: i have the same problem in my production environment18:00
dcapone2004after the update, the docker containers appear to come back up, status shows up X minutes ago, but no network to the containers18:00
kplantyou can also move the vip to another control none (in non aio)18:00
dcapone2004is your production AIO or multi-server?  I was hoping this was some type of AIO issue only18:01
kplantmulti-node18:01
yoctozeptokplant: I think it does not apply to vip18:02
kplantrestarting the docker daemon hoses all vm connectivity until i move to another control node18:02
yoctozeptobut guests losing their ovs instance18:02
dcapone2004so when you perform the update, do you only have a problem when you update the servers hosting the neutron containers?18:02
kplanti think it's ovs related18:03
yoctozeptoI believe this is due to ovs and/or its agent going down18:03
kplantif i restart the docker daemons on the network nodes18:03
kplantovs doesn't appear to be a fan of that18:03
yoctozeptohmm18:03
yoctozeptodo we have this issue filed?18:04
openstackgerritMerged openstack/kayobe stable/stein: Fix kolla-ansible role tests  https://review.opendev.org/69766218:04
openstackgerritMerged openstack/kayobe stable/stein: Task to generate nginx.conf in inspection-store missing `become: True`  https://review.opendev.org/69764118:04
kplanti remember talking with a core about it a few weeks ago i think18:04
openstackgerritMerged openstack/kayobe master: Revert to yatesr.timezone role  https://review.opendev.org/69230718:04
openstackgerritMerged openstack/kayobe master: Remove sphinx from test-requirements.txt  https://review.opendev.org/68749918:04
dcapone2004ok, but if you restart docker on the compute or cinder nodes, that doesn't break anything?18:04
openstackgerritMerged openstack/kayobe master: Add new variables for customising the deployment image in Bifrost  https://review.opendev.org/69055918:04
openstackgerritMerged openstack/kayobe master: Add documentation for Bifrost and IPA configuration  https://review.opendev.org/69351518:04
openstackgerritMerged openstack/kayobe master: Add cross references from deployment to config docs  https://review.opendev.org/69352218:04
kplantdcapone2004: that's right18:04
openstackgerritMerged openstack/kayobe master: Support installing packages in IPA images  https://review.opendev.org/69431018:04
openstackgerritMerged openstack/kayobe master: Don't list tasks for config dump  https://review.opendev.org/68155818:04
dcapone2004ok, at least I understand the scope of the issue as we progress to a multi-node test18:05
dcapone2004so, in your multi-mode, you update one neutron host, move the VIP to a different neutron host running a redundant container to restore connectivity?18:05
kplantyep18:06
dcapone2004then the opposite when updating the other node?18:06
kplantbetter practice is moving the vip first18:06
dcapone2004well yes :-)18:06
noxoidi assume you are not using HA neutron routers?18:06
kplanti am18:06
dcapone2004but I was trying to look at it from a "what fixes it" perspective18:06
kplanti've never tried just restarting the ovs containers18:07
kplantlet me break my lab and give that a shot18:07
dcapone2004I've tried that :-)… didn't work for me in AIO18:08
kplantrip in peace18:08
kplanthmmm18:08
dcapone2004I tried restarting all of the neutron containers18:08
kplanti wonder if it's a docker thing18:08
kplanti've had problems with odkcer and libvirt fighting over networking before18:08
kplantdepending on who was restarted last18:08
yoctozeptoI doubt it's docker in particular, probably it breaks just due to services going down and guests not18:08
dcapone2004but a complete reboot magically fixed everything....and I haven't been able to figure out what exactly the "magic" of a reboot does18:08
yoctozeptoyes, because guests come later18:09
yoctozeptorestart guests themselves and you are probably fine18:09
yoctozepto(full restart, down guest, up guest)18:09
noxoidmy multinode setup is able to tolerate individual ovs container restarts as well as docker daemon restarts without issue (at least from neutrons point of view)18:09
kplantyoctozepto: i don't do anything to the guests and rebooting the control node fixes the issue18:09
kplanti've had to restart the docker daemon to update trusted CAs (because it sucks)18:09
yoctozeptodcapone2004: which connectivity is lost?18:09
kplantnot even because of updates18:10
yoctozeptoguest<->guest or guest<->external?18:10
kplantit's guest<->ex for me18:10
dcapone2004haven't checked guest to guest18:10
kplantthe vxlans themselves are fine18:10
dcapone2004guest to external is definitely lost18:10
yoctozeptoOK, IGNORE ALL I SAID :D18:10
yoctozeptoif guest<->guest work fine then ignore my mumblings above18:10
dcapone2004I also think I tried rebooting a guest node one time and that didn't fix it either...but let me do that now18:11
kplanti'm breaking my lab right now18:11
dcapone2004since I just applied updates to my lab and my VMs are presently down18:11
yoctozeptodcapone2004: no, it won't, yeah18:11
kplantugh18:11
kplantkeepalived was smart and moved the vip18:11
kplanti can't even break it when i want to18:11
noxoidcreate a standalone (non-ha) router?18:11
yoctozeptokplant: just moving the vip helps there?18:12
yoctozeptofeels weird18:12
kplantyes beacuse the other control nodes aren't restarted18:12
yoctozeptovip is for api access18:12
yoctozeptoyeah but apis should not be called after the networking is established in the first place :D18:12
yoctozeptootherwise it would be very low on performance18:13
kplanti wonder if this was fixed in the branch i'm running?18:13
kplantit doesn't break anymore18:13
yoctozeptodo logs of nova/neutron show anything on the affected nodes (Control/compute)18:13
kplanti definitely remember talking to someone about this exploding instance networking and rmq18:13
yoctozeptoI don't remember18:13
kplantwell this is great18:13
kplantit works now lol18:13
yoctozeptoxD18:14
kplant(openstack) [root@chrnc-dev-os-build-01 kolla]# git describe18:14
kplant8.0.1-50-gc0f40d818:14
yoctozeptoI have 3 control+network nodes18:14
kplant(openstack) [root@chrnc-dev-os-build-01 kolla-ansible]# git describe18:14
yoctozeptoI update one at a time18:14
kplant8.0.1-89-g22dc15618:14
yoctozeptoand never experienced this issue18:14
kplantdcapone2004: are you on code older than that?18:14
dcapone2004so might be a non-HA neutron router issue only being I have AIO18:14
dcapone2004let me check18:14
kplantwell i have ha routers and i had that problem as well18:15
yoctozeptoif you can reproduce, please raise a bug18:15
yoctozeptothen we can work18:15
yoctozeptootherwise no problem :D18:15
kplantyeah i think this was fixed18:15
kplantgj whoever that was18:15
kplanthey is train ready for the show?18:16
dcapone2004do you use git clone versions in production?  I was testing with pip packages of kolla-ansible18:18
kplantyes i pull from git18:18
dcapone2004even in production?18:18
kplantpypi is usually far behind and missing some good features and fixes18:18
kplantyep, production18:18
kplanti also have some modifications that i build that most people wouldn't care about18:19
kplantso i don't submit them and just build locally18:19
openstackgerritMerged openstack/kayobe master: Use docker_custom_config variable  https://review.opendev.org/69096518:19
dcapone2004ok, probably need to change my strategy a bit then...I usually think of pulling from git repos as dev stuff not production ready18:19
kplantas long as you pull from stable/<release> you should be fine18:19
kplantmaster is roulette18:20
dcapone2004ok thanks....I will do that for our multi-node test18:20
noxoid^18:20
noxoidi run stable/stein as well from git. has worked nicely18:21
kplantyeah i rebuild containers from stable/stein more than once and month18:21
kplantand drop them in production, nobody has any idea18:21
dcapone2004out of curiosity, do you separately deploy ceph with ceph-ansible and attach to it with kolla-ansible or do you use kolla-ansible to deploy ceph?18:22
kplanti use ceph-ansible18:22
kplanti feel k-a is really good at openstack and ceph-ansible is really good at ceph18:23
kplantso i like using them both18:23
kplantalthough mnasiadka, i believe, is working hard on adding ceph-ansible to kolla-anisble18:23
dcapone2004yeah that was my plan18:23
kplantkind of how openstack-ansible does it18:23
yoctozeptodcapone2004: (kolla core here) I also run production off a stable/ branch18:26
yoctozeptowe care to keep them really stable18:26
kplant^ can confirm. they've yelled at me for submitting non-fixes to stable branches :-)18:26
yoctozeptoand tbh, we are bad at releasing18:26
yoctozepto;p18:26
yoctozeptofwiw, the images on dockerhub most ppl use are built off stable/18:29
mnasiadkakplant: currently the plan is to have a CI with ceph-ansible, have you seen my mail to ML?18:33
kplanti have not18:34
dcapone2004thanks, I can now get back to some testing18:34
*** graysonh has joined #openstack-kolla18:35
*** dcapone2004 has quit IRC18:35
mnasiadkakplant: time to read mails with [kolla] tag :)18:35
yoctozeptoreliable ways to get info/support on kolla include this very channel, the openstack-discuss mailing list and our launchpad project page (bugs section)18:37
yoctozeptowe don't really do any ask/stackoverflow18:37
openstackgerritMerged openstack/kolla master: ovn-base: Create /run/ovn in extend_start.sh  https://review.opendev.org/69714818:52
*** dcapone2004 has joined #openstack-kolla18:53
openstackgerritMerged openstack/kayobe stable/rocky: Fix kolla-ansible role tests  https://review.opendev.org/69767318:56
dcapone2004ok, so I used git and it is giving me version 8.0.2.dev52 is this correct?  The "dev" part appears different than the gcXX kplant referenced, so I am cloning the wrong branch18:56
yoctozeptodcapone2004: kplant probably did not pull recently18:58
yoctozepto:D18:58
dcapone2004ok, so that is correct version number and is acceptable for production?18:59
yoctozeptoas long as you run off stable/ you are fine for production18:59
dcapone2004I trust ya18:59
openstackgerritMerged openstack/kolla master: Disable building binary images for senlin-conductor and senlin-health-manager on debian/ubuntu  https://review.opendev.org/69744518:59
*** dcapone2004 has quit IRC19:01
*** schwicht has quit IRC19:15
*** bjoernt has quit IRC19:23
*** bjoernt has joined #openstack-kolla19:28
openstackgerritMerged openstack/kolla master: Remove skydive rpm installation  https://review.opendev.org/69670919:37
openstackgerritGaëtan Trellu proposed openstack/kolla-ansible master: Use internal API for masakari-monitor  https://review.opendev.org/69771219:51
*** schwicht has joined #openstack-kolla20:12
*** kplant has quit IRC20:14
*** k_mouza has joined #openstack-kolla20:44
mlozaDid anyone got zun containers to work with cinder volumes in stable/stein branch?20:46
mloza The container creation stuck at "Creating". The volume was created in Cinder but doesn't seem to be attached propertly.20:47
*** k_mouza has quit IRC20:49
openstackgerritMerged openstack/kolla master: Added python3 packages for mellanox agent container  https://review.opendev.org/69723621:05
openstackgerritMerged openstack/kolla master: Vitrage: build images for Debian  https://review.opendev.org/69593121:05
*** rouk has quit IRC21:44
*** irclogbot_3 has quit IRC21:45
openstackgerritMerged openstack/kayobe master: Pass package list to Ansible package modules  https://review.opendev.org/69139222:12
*** goldyfruit has quit IRC22:12
openstackgerritMark Goddard proposed openstack/kolla master: WIP: move to CentOS 8 in containers  https://review.opendev.org/69236822:14
*** schwicht has quit IRC22:23
*** schwicht has joined #openstack-kolla22:54
*** schwicht has quit IRC22:58
*** jbadiapa has quit IRC23:51

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!