| *** schwicht has quit IRC | 00:02 | |
| *** xinliang has joined #openstack-kolla | 00:02 | |
| *** schwicht has joined #openstack-kolla | 00:35 | |
| *** jcmdln has joined #openstack-kolla | 00:38 | |
| *** zhanglong has joined #openstack-kolla | 00:45 | |
| *** zhanglong has quit IRC | 01:02 | |
| *** zhanglong has joined #openstack-kolla | 01:09 | |
| *** ricolin has joined #openstack-kolla | 01:10 | |
| *** schwicht has quit IRC | 01:26 | |
| *** zhanglong has quit IRC | 01:41 | |
| *** zhanglong has joined #openstack-kolla | 01:53 | |
| openstackgerrit | YU CHENGDE proposed openstack/kolla master: Let nova-libvirt restarting be independent of libvirtd.log https://review.opendev.org/697605 | 01:54 |
|---|---|---|
| *** schwicht has joined #openstack-kolla | 01:59 | |
| *** schwicht has quit IRC | 02:22 | |
| *** schwicht has joined #openstack-kolla | 02:25 | |
| *** rphillips has quit IRC | 03:20 | |
| *** rphillips has joined #openstack-kolla | 03:32 | |
| *** schwicht has quit IRC | 03:34 | |
| *** openstackstatus has joined #openstack-kolla | 03:41 | |
| *** ChanServ sets mode: +v openstackstatus | 03:41 | |
| *** dave-mccowan has quit IRC | 04:04 | |
| *** zhanglong has quit IRC | 04:04 | |
| *** xinliang has quit IRC | 04:09 | |
| *** ricolin has quit IRC | 04:16 | |
| *** ricolin has joined #openstack-kolla | 04:18 | |
| *** schwicht has joined #openstack-kolla | 04:30 | |
| *** zhanglong has joined #openstack-kolla | 04:56 | |
| *** raukadah is now known as chkumar|ruck | 05:20 | |
| *** skramaja has joined #openstack-kolla | 05:31 | |
| *** obre has quit IRC | 05:34 | |
| *** obre has joined #openstack-kolla | 05:41 | |
| *** zhanglong has quit IRC | 06:03 | |
| yoctozepto | morning | 06:42 |
| *** zhanglong has joined #openstack-kolla | 06:50 | |
| *** sri_ has joined #openstack-kolla | 06:51 | |
| sri_ | Morning | 06:53 |
| *** zhanglong has quit IRC | 06:59 | |
| mnasiadka | Morning | 07:01 |
| *** zhanglong has joined #openstack-kolla | 07:04 | |
| *** xinliang has joined #openstack-kolla | 07:15 | |
| *** cah_link has joined #openstack-kolla | 07:22 | |
| *** dpawlik has joined #openstack-kolla | 08:02 | |
| *** pcaruana has joined #openstack-kolla | 08:11 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 08:13 |
| *** rpittau|afk is now known as rpittau | 08:21 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 08:37 |
| hrw | yoł | 09:00 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 09:10 |
| *** gfidente|afk is now known as gfidente | 09:11 | |
| *** dswebb has quit IRC | 09:19 | |
| openstackgerrit | Merged openstack/kolla stable/stein: Install etcd3gw to fix Ubuntu binary tooz coordination https://review.opendev.org/697427 | 09:27 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: base: use upstream dumb-init on all architectures https://review.opendev.org/696370 | 09:30 |
| hrw | pep8/yaml fix | 09:30 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: add more tests for enable_repos template method https://review.opendev.org/697354 | 09:31 |
| hrw | rebased | 09:31 |
| hrw | trivial one anyone? https://review.opendev.org/#/c/697351/ | 09:31 |
| patchbot | patch 697351 - kolla - releasenotes: fix file location - 1 patch set | 09:31 |
| *** tonythomas has joined #openstack-kolla | 09:31 | |
| mnasiadka | CRITICAL neutron [-] Unhandled error: AttributeError: 'module' object has no attribute 'abc' | 09:36 |
| mnasiadka | todays winner | 09:36 |
| mnasiadka | hrw, yoctozepto: so, we have another python2 vs 3 issue - https://8838e06c389b32ad359d-4ca7e044918ce86ea788bdbe0e5e3d01.ssl.cf1.rackcdn.com/676128/18/check/kolla-ansible-centos-source/8a2a622/primary/logs/kolla/neutron/neutron-dhcp-agent.txt.gz | 09:41 |
| mnasiadka | this is not easily solvable | 09:44 |
| mnasiadka | ... | 09:44 |
| *** xinliang has quit IRC | 09:45 | |
| *** k_mouza has joined #openstack-kolla | 09:45 | |
| hrw | mnasiadka: add neutron-dhcp-agent into UNBUILDABLE_IMAGES? | 09:46 |
| hrw | :D | 09:46 |
| mnasiadka | hrw: well, we can then stop running kolla-ansible-centos-source I guess | 09:46 |
| mnasiadka | :) | 09:46 |
| hrw | or define centos7 unsupportable | 09:47 |
| mnasiadka | yup | 09:49 |
| mnasiadka | mgoddard: ^^ | 09:49 |
| hrw | CI blocker as well | 09:54 |
| mnasiadka | yup | 09:54 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 09:54 |
| yoctozepto | mgoddard, mnasiadka, hrw: you have my +2 for stopping centos jobs on ussuri for now - no point in this pain | 09:59 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe stable/stein: Task to generate nginx.conf in inspection-store missing `become: True` https://review.opendev.org/697641 | 10:04 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe stable/rocky: Task to generate nginx.conf in inspection-store missing `become: True` https://review.opendev.org/697642 | 10:04 |
| mgoddard | morning | 10:05 |
| hrw | mgoddard: morning | 10:05 |
| hrw | mgoddard: morning discussion: to drop c7 or not | 10:05 |
| mgoddard | hrw: just catching up | 10:05 |
| hrw | ;d | 10:05 |
| mgoddard | seems we are busted | 10:07 |
| mgoddard | I'd like to see how far we can get with centos8+source | 10:07 |
| mgoddard | I'll spend some time on it today | 10:08 |
| mgoddard | hrw: did you have a patch that uses copr? | 10:08 |
| hrw | mgoddard: moment | 10:08 |
| *** born2bake has joined #openstack-kolla | 10:09 | |
| hrw | mgoddard: apply https://paste.centos.org/view/9c6a7731 on top of https://review.opendev.org/692368 one | 10:12 |
| patchbot | patch 692368 - kolla - WIP: move to CentOS 8 in containers - 22 patch sets | 10:12 |
| mgoddard | hrw: thanks | 10:13 |
| hrw | mgoddard: centos8 does not give nova/neutron/cinder/glance etc due to lack of ceph | 10:13 |
| mgoddard | hrw: so we'll remove ceph :) | 10:14 |
| hrw | ;D | 10:14 |
| mgoddard | better than no coverage | 10:14 |
| hrw | sure | 10:14 |
| mgoddard | removing CI could get messy for bug fixes that we want to backport | 10:14 |
| hrw | mgoddard: 692368 requires rebase and adapting repos.yaml | 10:15 |
| mgoddard | k | 10:15 |
| hrw | mgoddard: for start you can even skip repos.yaml if 'dnf config-manager' will not complain about enabling unknown repos | 10:16 |
| *** vmixor has joined #openstack-kolla | 10:16 | |
| yoctozepto | mgoddard: going for c8 and removing ceph is similarly messy for backports :-) | 10:19 |
| yoctozepto | no way to address this nicely | 10:19 |
| hrw | if rpm and python3 then c8 else c7 | 10:20 |
| yoctozepto | mgoddard, hrw: what about pinning c7 ci to train branches instead? | 10:22 |
| mgoddard | yoctozepto: then it would not be CI :) | 10:22 |
| yoctozepto | mgoddard: for c7 which you stop at train? | 10:22 |
| yoctozepto | it seems CI all right | 10:22 |
| yoctozepto | and then you have nice backports :D | 10:22 |
| yoctozepto | all solutions are bad | 10:23 |
| yoctozepto | just giving another one | 10:23 |
| *** vmixor has quit IRC | 10:24 | |
| hrw | I like the idea of 'let get c8 at whatever is there' 🙃 | 10:25 |
| mgoddard | yoctozepto: I guess we could deploy train containers using master k-a | 10:25 |
| mgoddard | I'd like to see if we can move forward before going backward though | 10:25 |
| yoctozepto | mgoddard: +1 for c8 already | 10:26 |
| yoctozepto | c7 can be separate jobs | 10:26 |
| mgoddard | hrw: seems dnf config-manager disable does not seem to ignore errors :/ | 10:26 |
| yoctozepto | just food for thought | 10:26 |
| yoctozepto | I'm a bit busy with other things but will review if you ping | 10:27 |
| hrw | mgoddard: you rebased? | 10:27 |
| mgoddard | hrw: yes | 10:27 |
| hrw | mgoddard: please push to gerrit | 10:27 |
| mgoddard | hrw: not pushed yet | 10:27 |
| mgoddard | still tinkering | 10:27 |
| hrw | ok | 10:27 |
| *** k_mouza has quit IRC | 10:27 | |
| hrw | mgoddard: https://review.opendev.org/#/c/613815/ would be nice to have probably. | 10:28 |
| patchbot | patch 613815 - kolla - Remove tgtd and scsi-target-utils support in CentO... - 12 patch sets | 10:28 |
| hrw | mgoddard: bridge-utils and btrfs stuff is also not present in rhel/c8 | 10:28 |
| hrw | rebases fine on master | 10:29 |
| mgoddard | hrw: need something on the k-a side before removing tgtd | 10:30 |
| hrw | right | 10:30 |
| mgoddard | hrw: https://review.opendev.org/631479 | 10:30 |
| patchbot | patch 631479 - kolla-ansible - Configure Cinder to use lioadm on RPM-based distros - 10 patch sets | 10:30 |
| mgoddard | hrw: I think safest option might be to make this depend on c7 vs c8 | 10:30 |
| hrw | si | 10:31 |
| mgoddard | because the problem was always migration from tgtd to lio | 10:31 |
| mgoddard | now is our chance to switch | 10:31 |
| hrw | https://twitter.com/haerwu/status/1202898469620527104 | 10:32 |
| mgoddard | hrw: :) | 10:32 |
| *** zhanglong has quit IRC | 10:34 | |
| *** k_mouza has joined #openstack-kolla | 10:38 | |
| hrw | mgoddard: in worst case we can make kolla copr and put there ceph from https://koji.mbox.centos.org/koji/buildinfo?buildID=582 | 10:40 |
| born2bake | Hi, I am wondering if octavia image was updated and fixed https://bugs.launchpad.net/kolla-ansible/+bug/1847905 ; Is there possibility to update images so I can make sure that latest fixes are applied? | 10:43 |
| openstack | Launchpad bug 1847905 in kolla-ansible "Octavia loadbalancer pending create state" [Undecided,New] | 10:43 |
| hrw | mgoddard: https://copr.fedorainfracloud.org/coprs/hrw/kolla-extras/packages/ | 10:45 |
| mnasiadka | hrw: last time I checked there were some deps missing to build ceph for el8 | 10:48 |
| mnasiadka | hrw: https://trello.com/b/ZAMLb0pT/ceph-el8 | 10:51 |
| mnasiadka | hrw: and essentially https://copr.fedorainfracloud.org/coprs/ktdreyer/ceph-el8/ | 10:53 |
| hrw | mnasiadka: great! | 10:53 |
| hrw | dropped kolla-extras then | 10:55 |
| hrw | it's centos. to build one out-of-distro package you first need to add gazillion of deps | 10:56 |
| mgoddard | hrw: 404 | 11:00 |
| hrw | mgoddard: dropped | 11:00 |
| yoctozepto | mgoddard: look at mnasiadka's | 11:10 |
| * hrw out | 11:31 | |
| *** zbr has quit IRC | 11:47 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 11:47 |
| *** zbr has joined #openstack-kolla | 11:48 | |
| *** k_mouza has quit IRC | 11:54 | |
| *** k_mouza has joined #openstack-kolla | 11:57 | |
| *** negronjl has quit IRC | 12:14 | |
| *** negronjl has joined #openstack-kolla | 12:16 | |
| *** k_mouza has quit IRC | 12:24 | |
| mnasiadka | gfidente: around? | 12:26 |
| gfidente | mnasiadka i | 12:27 |
| mnasiadka | gfidente: any advice how to obtain ceph packages for centos8? any fedora builds or something like that? | 12:27 |
| gfidente | mnasiadka yeah we don't have them yet | 12:27 |
| gfidente | we haven't been able to start building them yet :( | 12:28 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: Switch to stable/train for source images on CentOS/RHEL https://review.opendev.org/697658 | 12:36 |
| mgoddard | yoctozepto, hrw, mnasiadka: Trying ^ as an alternative to c8. Switch source tarballs to stable/train | 12:38 |
| hrw | mgoddard: if python2 then stable/train? | 12:38 |
| mgoddard | hrw: if centos/rhel then stable/train | 12:39 |
| mgoddard | to keep us sane :) | 12:39 |
| hrw | mgoddard: you just broke tripleo8 | 12:39 |
| mgoddard | hrw: how so? source only | 12:39 |
| hrw | ok | 12:39 |
| mgoddard | tripleo doesn't do source | 12:39 |
| hrw | let's see how it pass zuul | 12:39 |
| hrw | +1 | 12:40 |
| yoctozepto | mgoddard: glad my solution wins | 12:40 |
| mgoddard | yoctozepto: I didn't realise that's what your solution was | 12:41 |
| mgoddard | I thought you meant pin kolla & k-a to train | 12:41 |
| mgoddard | or use train images from dockerhub | 12:41 |
| mgoddard | anyways | 12:41 |
| yoctozepto | with pinning I meant this, no problem, sir | 12:41 |
| *** schwicht has joined #openstack-kolla | 12:46 | |
| dirtwash | so we trying to deploy our test cluster again but now after cleanup it fails every time with : Waiting for nova-compute services to register themselves , the container logs dont say anything but some python package warning, any ideas what to look for? i pruned docker volumes and rebooted hosts | 12:51 |
| dirtwash | pretty much default config, just the most basic test setup | 12:52 |
| *** schwicht has quit IRC | 12:52 | |
| hrw | show logs? | 12:52 |
| dirtwash | hrw: http://paste.openstack.org/show/da5z13K9JJBLUG1Fzbuv/ | 12:55 |
| dirtwash | the docker logs all show the same for all the nova containers | 12:55 |
| dirtwash | ubuntu 1804 | 12:55 |
| yoctozepto | dirtwash: /var/log/kolla on the host | 12:57 |
| yoctozepto | container logs are usually very limited | 12:57 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 12:57 |
| dirtwash | yoctozepto: which one to look at though, there's plenty | 12:58 |
| yoctozepto | nova/nova-compute.log | 12:58 |
| dirtwash | hm libvirtd error | 12:59 |
| dirtwash | yoctozepto: thanks, I'll figure it out from here | 13:00 |
| hrw | dirtwash: x86? | 13:00 |
| *** kplant has joined #openstack-kolla | 13:00 | |
| dirtwash | hrw: yea | 13:00 |
| dirtwash | but libvirtd is running fine hm | 13:00 |
| dirtwash | error is libvirtError: unable to connect to server at '10.3.0.197:16509': Connection refused | 13:01 |
| openstackgerrit | Pierre Riteau proposed openstack/kayobe stable/stein: Task to generate nginx.conf in inspection-store missing `become: True` https://review.opendev.org/697641 | 13:10 |
| openstackgerrit | Pierre Riteau proposed openstack/kayobe stable/stein: Fix kolla-ansible role tests https://review.opendev.org/697662 | 13:10 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Improve Apache logging https://review.opendev.org/697262 | 13:11 |
| *** yan0s has joined #openstack-kolla | 13:12 | |
| *** schwicht has joined #openstack-kolla | 13:21 | |
| *** schwicht has quit IRC | 13:26 | |
| *** sri_ has quit IRC | 13:30 | |
| openstackgerrit | Will Szumski proposed openstack/kayobe master: Use OpenStack Train release https://review.opendev.org/694616 | 13:35 |
| *** skramaja has quit IRC | 13:36 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: Switch to stable/train for source images on CentOS/RHEL https://review.opendev.org/697658 | 13:37 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 13:38 |
| *** mgoddard has quit IRC | 13:44 | |
| *** mgoddard has joined #openstack-kolla | 13:45 | |
| openstackgerrit | Will Szumski proposed openstack/kayobe master: Improve detection of base path https://review.opendev.org/655864 | 13:47 |
| *** mgoddard has quit IRC | 13:58 | |
| *** mgoddard has joined #openstack-kolla | 14:05 | |
| *** cah_link has quit IRC | 14:06 | |
| *** cah_link has joined #openstack-kolla | 14:15 | |
| *** schwicht has joined #openstack-kolla | 14:22 | |
| *** cah_link has quit IRC | 14:25 | |
| hrw | have a nice weekend | 14:27 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 14:28 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 14:28 |
| mgoddard | you too hrw | 14:29 |
| *** graysonh has joined #openstack-kolla | 14:40 | |
| *** goldyfruit___ has joined #openstack-kolla | 14:41 | |
| dirtwash | can cinder used without backup pool? | 14:47 |
| *** negronjl has quit IRC | 14:51 | |
| dirtwash | man the openstack docs are utterly useless | 14:52 |
| *** negronjl has joined #openstack-kolla | 14:53 | |
| *** chkumar|ruck is now known as raukadah | 15:01 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe stable/rocky: Fix kolla-ansible role tests https://review.opendev.org/697673 | 15:03 |
| *** dave-mccowan has joined #openstack-kolla | 15:05 | |
| *** goldyfruit___ is now known as goldyfruit | 15:07 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI https://review.opendev.org/676376 | 15:10 |
| *** dave-mccowan has quit IRC | 15:10 | |
| yoctozepto | dirtwash: docs have varying quality | 15:13 |
| yoctozepto | cinder-backup can be disabled | 15:13 |
| yoctozepto | if you don't want it :-) | 15:13 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: WIP: move to CentOS 8 in containers https://review.opendev.org/692368 | 15:15 |
| mgoddard | Let's see how it goes ^ | 15:16 |
| *** KeithMnemonic has joined #openstack-kolla | 15:29 | |
| mgoddard | mnasiadka: https://review.opendev.org/#/c/697658 | 15:35 |
| patchbot | patch 697658 - kolla - Switch to stable/train for source images on CentOS... - 2 patch sets | 15:35 |
| mnasiadka | mgoddard: off we go | 15:36 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla master: ovn-base: Create /run/ovn in extend_start.sh https://review.opendev.org/697148 | 15:38 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla master: Disable building binary images for senlin-conductor and senlin-health-manager on debian/ubuntu https://review.opendev.org/697445 | 15:38 |
| yoctozepto | ^ rebased | 15:38 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: Revert "Introduce macro to remove Py3 only plugins on CentOS 7" https://review.opendev.org/697680 | 15:43 |
| * mgoddard cleans up | 15:44 | |
| yoctozepto | very good mgoddard | 15:46 |
| yoctozepto | I'm gonna fix https://review.opendev.org/697478 if you don't mind :D | 15:46 |
| patchbot | patch 697478 - kolla-ansible - CI: Don't fail on exepected critical log messages - 2 patch sets | 15:46 |
| openstackgerrit | Merged openstack/kayobe master: Remove workaround for kolla-ansible post-deploy https://review.opendev.org/696110 | 15:50 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: CI: Don't fail on expected critical log messages https://review.opendev.org/697478 | 15:53 |
| yoctozepto | mgoddard: ^ I think this will do the job | 15:55 |
| mgoddard | yoctozepto: thx | 15:55 |
| *** yan0s has quit IRC | 15:59 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Improve docs on kolla-ansible https://review.opendev.org/697682 | 15:59 |
| openstackgerrit | Pierre Riteau proposed openstack/kayobe master: Use {{ openstack_branch }} as version of kolla-ansible in ansible tests https://review.opendev.org/697684 | 16:06 |
| openstackgerrit | Pierre Riteau proposed openstack/kayobe stable/rocky: Task to generate nginx.conf in inspection-store missing `become: True` https://review.opendev.org/697642 | 16:12 |
| *** dpawlik has quit IRC | 16:14 | |
| *** graysonh has quit IRC | 16:20 | |
| *** graysonh has joined #openstack-kolla | 16:20 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: WIP: CI: Start testing CentOS 8 hosts https://review.opendev.org/693544 | 16:25 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: WIP: Disable fluentd in CI for CentOS 8 https://review.opendev.org/697690 | 16:27 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: WIP: move to CentOS 8 in containers https://review.opendev.org/692368 | 16:27 |
| openstackgerrit | Pierre Riteau proposed openstack/kayobe master: Use {{ openstack_branch }} as version of kolla-ansible in ansible tests https://review.opendev.org/697684 | 16:30 |
| *** graysonh has quit IRC | 16:36 | |
| *** graysonh has joined #openstack-kolla | 16:36 | |
| *** k_mouza has joined #openstack-kolla | 16:44 | |
| *** k_mouza has quit IRC | 16:48 | |
| *** rpittau is now known as rpittau|afk | 16:51 | |
| openstackgerrit | Merged openstack/kayobe master: Fix kayobe overcloud provision without seed in limit https://review.opendev.org/693526 | 17:05 |
| openstackgerrit | Merged openstack/kayobe master: Add docker_registry_mirrors to docker.yml https://review.opendev.org/695206 | 17:06 |
| openstackgerrit | Merged openstack/kayobe master: Fix use of IPA images in Bifrost without a checksum https://review.opendev.org/693739 | 17:06 |
| openstackgerrit | Merged openstack/kayobe master: Stop running kolla-target-venv in seed hypervisor host upgrade https://review.opendev.org/692983 | 17:11 |
| openstackgerrit | Merged openstack/kayobe master: Stop gathering facts for host command run https://review.opendev.org/692981 | 17:11 |
| *** graysonh has quit IRC | 17:18 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla master: Added python3 packages for mellanox agent container https://review.opendev.org/697236 | 17:19 |
| *** KeithMnemonic has quit IRC | 17:22 | |
| *** pcaruana has quit IRC | 17:23 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: WIP: CI: Start testing CentOS 8 hosts https://review.opendev.org/693544 | 17:33 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: WIP: Disable fluentd in CI for CentOS 8 https://review.opendev.org/697690 | 17:33 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Use local python interpreter for keystone cron generator https://review.opendev.org/697700 | 17:33 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: WIP: move to CentOS 8 in containers https://review.opendev.org/692368 | 17:34 |
| openstackgerrit | Will Szumski proposed openstack/kayobe master: WIP: versioned links for release notes https://review.opendev.org/697701 | 17:35 |
| openstackgerrit | Merged openstack/kolla master: Switch to stable/train for source images on CentOS/RHEL https://review.opendev.org/697658 | 17:38 |
| *** negronjl has quit IRC | 17:39 | |
| yoctozepto | mgoddard, mnasiadka, hrw: ^ yay, merged | 17:39 |
| mgoddard | woop | 17:39 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: WIP: move to CentOS 8 in containers https://review.opendev.org/692368 | 17:41 |
| openstackgerrit | Will Szumski proposed openstack/kayobe master: Adds support for Ansible 2.8, bump mimimum to 2.6 https://review.opendev.org/697702 | 17:41 |
| yoctozepto | mnasiadka: https://review.opendev.org/676128 merge conflicts :< | 17:41 |
| patchbot | patch 676128 - kolla - Build images with healthcheck scripts for core ser... - 18 patch sets | 17:41 |
| mgoddard | mnasiadka: https://review.opendev.org/#/c/694476 | 17:44 |
| patchbot | patch 694476 - kolla-ansible - Default to etcd3gw driver for etcd-based coordination - 5 patch sets | 17:44 |
| *** negronjl has joined #openstack-kolla | 17:44 | |
| mnasiadka | Mg | 17:45 |
| mnasiadka | Ups | 17:45 |
| mnasiadka | mgoddard: done | 17:45 |
| yoctozepto | mnasiadka: did you mean Magnesium? | 17:45 |
| mgoddard | sweet | 17:45 |
| mnasiadka | yoctozepto: aye, will fix on Mon | 17:45 |
| yoctozepto | mnasiadka: ack | 17:45 |
| yoctozepto | mgoddard: are not images in dockerhub b0rken atm? | 17:47 |
| yoctozepto | as in w+1 from mnasiadka will break at gate | 17:47 |
| mgoddard | yoctozepto: you may be right | 17:48 |
| yoctozepto | I'll add a depends-on | 17:48 |
| mgoddard | it won't work as the other patch has merged | 17:48 |
| yoctozepto | mgoddard: to a merging one | 17:48 |
| mgoddard | ok | 17:48 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Default to etcd3gw driver for etcd-based coordination https://review.opendev.org/694476 | 17:49 |
| yoctozepto | mgoddard, mnasiadka: feel bad to w+1 myself https://review.opendev.org/694476 | 17:49 |
| patchbot | patch 694476 - kolla-ansible - Default to etcd3gw driver for etcd-based coordination - 6 patch sets | 17:49 |
| mgoddard | yoctozepto: I imagine it won't work as that one is already in the gate | 17:50 |
| yoctozepto | mgoddard: depends on timing, yeah | 17:51 |
| yoctozepto | well, gate behaves weirdly, let's find out | 17:51 |
| *** ricolin has quit IRC | 17:55 | |
| *** dcapone2004 has joined #openstack-kolla | 17:58 | |
| dcapone2004 | I have been testing kolla-ansible in an AIO environment for the past few months and I am having an issue relating to updates...I am unsure if this is an issue with AIO deployments only or if I am doing something wrong, or what...but essentially anytime there is an update to docker, I lose network connectivity to my VMs until the server is | 18:00 |
| dcapone2004 | completely restarted | 18:00 |
| openstackgerrit | Gowrishankar proposed openstack/kolla master: Provision building netcontrold container https://review.opendev.org/689090 | 18:00 |
| kplant | dcapone2004: i have the same problem in my production environment | 18:00 |
| dcapone2004 | after the update, the docker containers appear to come back up, status shows up X minutes ago, but no network to the containers | 18:00 |
| kplant | you can also move the vip to another control none (in non aio) | 18:00 |
| dcapone2004 | is your production AIO or multi-server? I was hoping this was some type of AIO issue only | 18:01 |
| kplant | multi-node | 18:01 |
| yoctozepto | kplant: I think it does not apply to vip | 18:02 |
| kplant | restarting the docker daemon hoses all vm connectivity until i move to another control node | 18:02 |
| yoctozepto | but guests losing their ovs instance | 18:02 |
| dcapone2004 | so when you perform the update, do you only have a problem when you update the servers hosting the neutron containers? | 18:02 |
| kplant | i think it's ovs related | 18:03 |
| yoctozepto | I believe this is due to ovs and/or its agent going down | 18:03 |
| kplant | if i restart the docker daemons on the network nodes | 18:03 |
| kplant | ovs doesn't appear to be a fan of that | 18:03 |
| yoctozepto | hmm | 18:03 |
| yoctozepto | do we have this issue filed? | 18:04 |
| openstackgerrit | Merged openstack/kayobe stable/stein: Fix kolla-ansible role tests https://review.opendev.org/697662 | 18:04 |
| openstackgerrit | Merged openstack/kayobe stable/stein: Task to generate nginx.conf in inspection-store missing `become: True` https://review.opendev.org/697641 | 18:04 |
| kplant | i remember talking with a core about it a few weeks ago i think | 18:04 |
| openstackgerrit | Merged openstack/kayobe master: Revert to yatesr.timezone role https://review.opendev.org/692307 | 18:04 |
| openstackgerrit | Merged openstack/kayobe master: Remove sphinx from test-requirements.txt https://review.opendev.org/687499 | 18:04 |
| dcapone2004 | ok, but if you restart docker on the compute or cinder nodes, that doesn't break anything? | 18:04 |
| openstackgerrit | Merged openstack/kayobe master: Add new variables for customising the deployment image in Bifrost https://review.opendev.org/690559 | 18:04 |
| openstackgerrit | Merged openstack/kayobe master: Add documentation for Bifrost and IPA configuration https://review.opendev.org/693515 | 18:04 |
| openstackgerrit | Merged openstack/kayobe master: Add cross references from deployment to config docs https://review.opendev.org/693522 | 18:04 |
| kplant | dcapone2004: that's right | 18:04 |
| openstackgerrit | Merged openstack/kayobe master: Support installing packages in IPA images https://review.opendev.org/694310 | 18:04 |
| openstackgerrit | Merged openstack/kayobe master: Don't list tasks for config dump https://review.opendev.org/681558 | 18:04 |
| dcapone2004 | ok, at least I understand the scope of the issue as we progress to a multi-node test | 18:05 |
| dcapone2004 | so, in your multi-mode, you update one neutron host, move the VIP to a different neutron host running a redundant container to restore connectivity? | 18:05 |
| kplant | yep | 18:06 |
| dcapone2004 | then the opposite when updating the other node? | 18:06 |
| kplant | better practice is moving the vip first | 18:06 |
| dcapone2004 | well yes :-) | 18:06 |
| noxoid | i assume you are not using HA neutron routers? | 18:06 |
| kplant | i am | 18:06 |
| dcapone2004 | but I was trying to look at it from a "what fixes it" perspective | 18:06 |
| kplant | i've never tried just restarting the ovs containers | 18:07 |
| kplant | let me break my lab and give that a shot | 18:07 |
| dcapone2004 | I've tried that :-)… didn't work for me in AIO | 18:08 |
| kplant | rip in peace | 18:08 |
| kplant | hmmm | 18:08 |
| dcapone2004 | I tried restarting all of the neutron containers | 18:08 |
| kplant | i wonder if it's a docker thing | 18:08 |
| kplant | i've had problems with odkcer and libvirt fighting over networking before | 18:08 |
| kplant | depending on who was restarted last | 18:08 |
| yoctozepto | I doubt it's docker in particular, probably it breaks just due to services going down and guests not | 18:08 |
| dcapone2004 | but a complete reboot magically fixed everything....and I haven't been able to figure out what exactly the "magic" of a reboot does | 18:08 |
| yoctozepto | yes, because guests come later | 18:09 |
| yoctozepto | restart guests themselves and you are probably fine | 18:09 |
| yoctozepto | (full restart, down guest, up guest) | 18:09 |
| noxoid | my multinode setup is able to tolerate individual ovs container restarts as well as docker daemon restarts without issue (at least from neutrons point of view) | 18:09 |
| kplant | yoctozepto: i don't do anything to the guests and rebooting the control node fixes the issue | 18:09 |
| kplant | i've had to restart the docker daemon to update trusted CAs (because it sucks) | 18:09 |
| yoctozepto | dcapone2004: which connectivity is lost? | 18:09 |
| kplant | not even because of updates | 18:10 |
| yoctozepto | guest<->guest or guest<->external? | 18:10 |
| kplant | it's guest<->ex for me | 18:10 |
| dcapone2004 | haven't checked guest to guest | 18:10 |
| kplant | the vxlans themselves are fine | 18:10 |
| dcapone2004 | guest to external is definitely lost | 18:10 |
| yoctozepto | OK, IGNORE ALL I SAID :D | 18:10 |
| yoctozepto | if guest<->guest work fine then ignore my mumblings above | 18:10 |
| dcapone2004 | I also think I tried rebooting a guest node one time and that didn't fix it either...but let me do that now | 18:11 |
| kplant | i'm breaking my lab right now | 18:11 |
| dcapone2004 | since I just applied updates to my lab and my VMs are presently down | 18:11 |
| yoctozepto | dcapone2004: no, it won't, yeah | 18:11 |
| kplant | ugh | 18:11 |
| kplant | keepalived was smart and moved the vip | 18:11 |
| kplant | i can't even break it when i want to | 18:11 |
| noxoid | create a standalone (non-ha) router? | 18:11 |
| yoctozepto | kplant: just moving the vip helps there? | 18:12 |
| yoctozepto | feels weird | 18:12 |
| kplant | yes beacuse the other control nodes aren't restarted | 18:12 |
| yoctozepto | vip is for api access | 18:12 |
| yoctozepto | yeah but apis should not be called after the networking is established in the first place :D | 18:12 |
| yoctozepto | otherwise it would be very low on performance | 18:13 |
| kplant | i wonder if this was fixed in the branch i'm running? | 18:13 |
| kplant | it doesn't break anymore | 18:13 |
| yoctozepto | do logs of nova/neutron show anything on the affected nodes (Control/compute) | 18:13 |
| kplant | i definitely remember talking to someone about this exploding instance networking and rmq | 18:13 |
| yoctozepto | I don't remember | 18:13 |
| kplant | well this is great | 18:13 |
| kplant | it works now lol | 18:13 |
| yoctozepto | xD | 18:14 |
| kplant | (openstack) [root@chrnc-dev-os-build-01 kolla]# git describe | 18:14 |
| kplant | 8.0.1-50-gc0f40d8 | 18:14 |
| yoctozepto | I have 3 control+network nodes | 18:14 |
| kplant | (openstack) [root@chrnc-dev-os-build-01 kolla-ansible]# git describe | 18:14 |
| yoctozepto | I update one at a time | 18:14 |
| kplant | 8.0.1-89-g22dc156 | 18:14 |
| yoctozepto | and never experienced this issue | 18:14 |
| kplant | dcapone2004: are you on code older than that? | 18:14 |
| dcapone2004 | so might be a non-HA neutron router issue only being I have AIO | 18:14 |
| dcapone2004 | let me check | 18:14 |
| kplant | well i have ha routers and i had that problem as well | 18:15 |
| yoctozepto | if you can reproduce, please raise a bug | 18:15 |
| yoctozepto | then we can work | 18:15 |
| yoctozepto | otherwise no problem :D | 18:15 |
| kplant | yeah i think this was fixed | 18:15 |
| kplant | gj whoever that was | 18:15 |
| kplant | hey is train ready for the show? | 18:16 |
| dcapone2004 | do you use git clone versions in production? I was testing with pip packages of kolla-ansible | 18:18 |
| kplant | yes i pull from git | 18:18 |
| dcapone2004 | even in production? | 18:18 |
| kplant | pypi is usually far behind and missing some good features and fixes | 18:18 |
| kplant | yep, production | 18:18 |
| kplant | i also have some modifications that i build that most people wouldn't care about | 18:19 |
| kplant | so i don't submit them and just build locally | 18:19 |
| openstackgerrit | Merged openstack/kayobe master: Use docker_custom_config variable https://review.opendev.org/690965 | 18:19 |
| dcapone2004 | ok, probably need to change my strategy a bit then...I usually think of pulling from git repos as dev stuff not production ready | 18:19 |
| kplant | as long as you pull from stable/<release> you should be fine | 18:19 |
| kplant | master is roulette | 18:20 |
| dcapone2004 | ok thanks....I will do that for our multi-node test | 18:20 |
| noxoid | ^ | 18:20 |
| noxoid | i run stable/stein as well from git. has worked nicely | 18:21 |
| kplant | yeah i rebuild containers from stable/stein more than once and month | 18:21 |
| kplant | and drop them in production, nobody has any idea | 18:21 |
| dcapone2004 | out of curiosity, do you separately deploy ceph with ceph-ansible and attach to it with kolla-ansible or do you use kolla-ansible to deploy ceph? | 18:22 |
| kplant | i use ceph-ansible | 18:22 |
| kplant | i feel k-a is really good at openstack and ceph-ansible is really good at ceph | 18:23 |
| kplant | so i like using them both | 18:23 |
| kplant | although mnasiadka, i believe, is working hard on adding ceph-ansible to kolla-anisble | 18:23 |
| dcapone2004 | yeah that was my plan | 18:23 |
| kplant | kind of how openstack-ansible does it | 18:23 |
| yoctozepto | dcapone2004: (kolla core here) I also run production off a stable/ branch | 18:26 |
| yoctozepto | we care to keep them really stable | 18:26 |
| kplant | ^ can confirm. they've yelled at me for submitting non-fixes to stable branches :-) | 18:26 |
| yoctozepto | and tbh, we are bad at releasing | 18:26 |
| yoctozepto | ;p | 18:26 |
| yoctozepto | fwiw, the images on dockerhub most ppl use are built off stable/ | 18:29 |
| mnasiadka | kplant: currently the plan is to have a CI with ceph-ansible, have you seen my mail to ML? | 18:33 |
| kplant | i have not | 18:34 |
| dcapone2004 | thanks, I can now get back to some testing | 18:34 |
| *** graysonh has joined #openstack-kolla | 18:35 | |
| *** dcapone2004 has quit IRC | 18:35 | |
| mnasiadka | kplant: time to read mails with [kolla] tag :) | 18:35 |
| yoctozepto | reliable ways to get info/support on kolla include this very channel, the openstack-discuss mailing list and our launchpad project page (bugs section) | 18:37 |
| yoctozepto | we don't really do any ask/stackoverflow | 18:37 |
| openstackgerrit | Merged openstack/kolla master: ovn-base: Create /run/ovn in extend_start.sh https://review.opendev.org/697148 | 18:52 |
| *** dcapone2004 has joined #openstack-kolla | 18:53 | |
| openstackgerrit | Merged openstack/kayobe stable/rocky: Fix kolla-ansible role tests https://review.opendev.org/697673 | 18:56 |
| dcapone2004 | ok, so I used git and it is giving me version 8.0.2.dev52 is this correct? The "dev" part appears different than the gcXX kplant referenced, so I am cloning the wrong branch | 18:56 |
| yoctozepto | dcapone2004: kplant probably did not pull recently | 18:58 |
| yoctozepto | :D | 18:58 |
| dcapone2004 | ok, so that is correct version number and is acceptable for production? | 18:59 |
| yoctozepto | as long as you run off stable/ you are fine for production | 18:59 |
| dcapone2004 | I trust ya | 18:59 |
| openstackgerrit | Merged openstack/kolla master: Disable building binary images for senlin-conductor and senlin-health-manager on debian/ubuntu https://review.opendev.org/697445 | 18:59 |
| *** dcapone2004 has quit IRC | 19:01 | |
| *** schwicht has quit IRC | 19:15 | |
| *** bjoernt has quit IRC | 19:23 | |
| *** bjoernt has joined #openstack-kolla | 19:28 | |
| openstackgerrit | Merged openstack/kolla master: Remove skydive rpm installation https://review.opendev.org/696709 | 19:37 |
| openstackgerrit | Gaëtan Trellu proposed openstack/kolla-ansible master: Use internal API for masakari-monitor https://review.opendev.org/697712 | 19:51 |
| *** schwicht has joined #openstack-kolla | 20:12 | |
| *** kplant has quit IRC | 20:14 | |
| *** k_mouza has joined #openstack-kolla | 20:44 | |
| mloza | Did anyone got zun containers to work with cinder volumes in stable/stein branch? | 20:46 |
| mloza | The container creation stuck at "Creating". The volume was created in Cinder but doesn't seem to be attached propertly. | 20:47 |
| *** k_mouza has quit IRC | 20:49 | |
| openstackgerrit | Merged openstack/kolla master: Added python3 packages for mellanox agent container https://review.opendev.org/697236 | 21:05 |
| openstackgerrit | Merged openstack/kolla master: Vitrage: build images for Debian https://review.opendev.org/695931 | 21:05 |
| *** rouk has quit IRC | 21:44 | |
| *** irclogbot_3 has quit IRC | 21:45 | |
| openstackgerrit | Merged openstack/kayobe master: Pass package list to Ansible package modules https://review.opendev.org/691392 | 22:12 |
| *** goldyfruit has quit IRC | 22:12 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: WIP: move to CentOS 8 in containers https://review.opendev.org/692368 | 22:14 |
| *** schwicht has quit IRC | 22:23 | |
| *** schwicht has joined #openstack-kolla | 22:54 | |
| *** schwicht has quit IRC | 22:58 | |
| *** jbadiapa has quit IRC | 23:51 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!