Thursday, 2020-03-12

*** openstackstatus has joined #openstack-kolla00:44
*** ChanServ sets mode: +v openstackstatus00:44
*** lile has quit IRC00:55
*** hongbin has joined #openstack-kolla01:12
*** diurnalist has joined #openstack-kolla01:33
*** diurnalist has quit IRC01:38
*** Tony31 has joined #openstack-kolla01:41
*** vishalmanchanda has joined #openstack-kolla01:51
*** zhanglong has joined #openstack-kolla02:23
*** xinliang has joined #openstack-kolla02:47
*** xinliang has quit IRC02:57
*** hongbin has quit IRC03:03
*** KeithMnemonic1 has joined #openstack-kolla03:28
*** KeithMnemonic has quit IRC03:31
*** ricolin_ has joined #openstack-kolla03:31
*** ricolin_ has quit IRC03:33
*** ricolin_ has joined #openstack-kolla03:34
*** ykarel|away is now known as ykarel03:48
*** dave-mccowan has quit IRC03:58
*** KeithMnemonic1 has quit IRC04:06
*** zhanglong has quit IRC04:10
*** factor has joined #openstack-kolla04:34
*** skramaja has joined #openstack-kolla04:54
*** zhanglong has joined #openstack-kolla05:31
*** evrardjp has quit IRC05:35
*** evrardjp has joined #openstack-kolla05:36
*** diurnalist has joined #openstack-kolla05:47
*** strigazi has quit IRC05:49
*** sri_ has joined #openstack-kolla05:51
*** strigazi has joined #openstack-kolla05:51
sri_morning05:52
*** diurnalist has quit IRC05:52
sri_Tony31, did you catch my comment yesterday05:59
Tony31Good morning sri_ - how are you doing?06:00
Tony31About cinder-backup ? Thanks a lot for the reply. I saw it this morning. I searched for the variable you mentioned and it's set in the kayobe source. I've installed the variable in my variables files now and will see if a deployment excludes it from install.06:01
*** Luzi has joined #openstack-kolla06:02
sri_Tony31, I am doing well, thanks for asking06:07
*** cah_link has joined #openstack-kolla06:11
*** abdysn has joined #openstack-kolla06:14
openstackgerritGauvain Pocentek proposed openstack/kolla master: Fix invalid test in cinder data migration script  https://review.opendev.org/71205506:32
cosmicsoundgreets06:55
Tony31hi cosmic06:58
*** strigazi has quit IRC07:03
cosmicsoundhey Tony3107:09
*** rpittau|afk is now known as rpittau07:17
cosmicsoundswift is not working with ceph backend?07:21
sri_cosmicsound, are you running rgw ? and set enable_ceph_rgw_keystone: "yes" ?07:29
sri_are you using external ceph ? if yes you have follow this https://docs.ceph.com/docs/mimic/radosgw/keystone/07:31
*** suff has joined #openstack-kolla07:35
Tony31is swift working now?07:35
cosmicsoundsri_ , we use external ceph indeed07:38
sri_cosmicsound, you can follow this  https://docs.ceph.com/docs/mimic/radosgw/keystone/07:39
cosmicsoundthanks sri_ checking it07:40
sri_Tony31, If your not using ceph, you can try this http://paste.openstack.org/show/790575/07:41
Tony31@sri_ I was running into this https://storyboard.openstack.org/#!/story/200729707:44
sri_Tony31, hmm, I think i was using stein when i tested swift, maybe something is broke in train i am not sure ;(07:46
Tony31I'm not sure if it's related to Kayobe or affects Kolla07:47
sri_hmm07:54
*** Fl1nt has joined #openstack-kolla08:02
*** suff has quit IRC08:04
Fl1nt@cosmicsound, look at your volumes and prune them when doing a destroy as sometimes that destroy just forget about them. There is also a chance that you need to perform the bootstrap step again on some edge case where destroy clean up even the kolla host folders.08:05
*** suff has joined #openstack-kolla08:05
Fl1ntI had the issue earlier today.08:05
*** bengates has joined #openstack-kolla08:09
hrwhttps://review.opendev.org/#/c/712485/ yoctozepto mnasiadka08:22
patchbotpatch 712485 - kolla - openstack-base: pin setuptools < 46 to get horizon... - 1 patch set08:22
mnasiadkahrw: well yoctozepto is complaining on the language :D08:23
*** tonythomas has joined #openstack-kolla08:24
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: openstack-base: pin setuptools < 46 to get horizon working  https://review.opendev.org/71248508:25
hrwchanged08:25
hrwrephrased instead of complaining on his complaints08:25
openstackgerritMatthias Runge proposed openstack/kolla master: Put back previously removed plugins  https://review.opendev.org/71262408:33
*** jbadiapa has joined #openstack-kolla08:36
*** zhanglong has quit IRC08:44
*** k_mouza has joined #openstack-kolla08:44
*** k_mouza has quit IRC08:49
*** rpittau is now known as rpittau|bbl08:52
Fl1ntok, folks, how much is kolla-ansible in sync with/dependent of kolla-build images versioning? I mean, let say I want to use kolla-ansible 9.x series playbooks/roles over a 7.1.0 installation, is that a no no?08:52
Fl1ntI'm having a hard time to find out something within the main (vanilla) roles that could prohibit it.08:52
hrwFl1nt: same release you have to use08:53
hrwimages 7, deployment 708:53
hrwetc08:53
Fl1ntSame major you mean08:53
hrwyes08:53
hrwunless you want to have issues no one encounter before08:54
Fl1ntI'll take a better exemple08:54
Fl1ntI had the same feeling, and actually enforce that for now08:54
Fl1ntBUT08:54
Fl1ntI checked the swift role08:54
Fl1ntand didn't found anything prohibiting a 7.1 platform to be deployed with the 9.0.1 playbooks as it add few usefull mechanisms without changing options that could broke the swift install.08:55
hrwFl1nt: there are roles which may not get changed much between releases08:56
hrwFl1nt: at one moment deploying X images with X+1 kolla-ansible == non working database08:56
hrwwhile X with X or X+1 with X+1 worked fine08:57
Fl1ntok, that may not be true for all, especially with deprecated options that became EOL/No longer available.08:57
hrwFl1nt: if you see good stuff in swift/9 you would like to have in swift/7 then consider checking gerrit and marking patches for backporting08:58
Fl1ntyep, notice that, ok, I'll stick with X/X and X+1/X+108:58
*** sluna has quit IRC08:58
*** sluna has joined #openstack-kolla08:58
Fl1ntI'll definitely have to check for gerrit and all that build system on Openstack as I plan to contribute more now that I'm no longer dependent of an employer contribution policy.08:59
hrwhttps://review.opendev.org/#/c/712485 got RP+209:00
patchbotpatch 712485 - kolla - openstack-base: pin setuptools < 46 to get horizon... - 2 patch sets09:00
*** hrw has quit IRC09:03
*** Tony31 has quit IRC09:11
*** ktibi has joined #openstack-kolla09:11
*** hrw has joined #openstack-kolla09:17
*** reph has joined #openstack-kolla09:17
*** ykarel is now known as ykarel|lunch09:19
*** priteau has joined #openstack-kolla09:25
mnasiadkahrw: off you go ;)09:27
hrwgood09:28
*** reph has quit IRC09:29
*** reph has joined #openstack-kolla09:30
*** k_mouza has joined #openstack-kolla09:34
yoctozeptomgoddard, mnasiadka, hrw: I was negative mostly due to "why not just horizon"09:35
yoctozeptospelling is irrelevant, just marked because of my grammar nazi soul09:35
yoctozepto;p09:35
mnasiadkayoctozepto: but you only expressed your grammar nazi soul09:36
mnasiadka:)09:36
yoctozeptomnasiadka: nope, please read more carefully09:36
yoctozepto"maybe downgrade in horizon only, hmm?"09:37
mgoddardyoctozepto: fix, then follow up09:37
mgoddardpriority is green CI09:37
yoctozeptomgoddard: hence no -1, just no +2, that was a question :-)09:37
mnasiadkayoctozepto: well, and do what in horizon? downgrade something that is installed? I think that would be more dirty.09:37
mgoddardmnasiadka: nah, it's horizon's problem. Better limit the weirdness09:38
yoctozeptomgoddard, mnasiadka: it's already installed anyway09:38
yoctozeptoit's setuptools09:38
mnasiadkalooking at that, wouldn't it make sense to have some local upper constrains (per image if required) in some better form, than in Dockerfiles? :)09:39
mnasiadka(I might be just picky)09:39
hrwmnasiadka: virtualenv installs setuptools. we can 1. downgrade in next step or 2. do not install setuptools with virtualenv and install it in next step09:40
mgoddardFl1nt: about mixing versions: it might work, but we make no guarantees. Things change between releases and we only track current. That said, we've had a lot of success running magnum a release or two ahead of other things, mostly to keep up with k8s09:41
cosmicsoundFl1nt , i did solved most of issues by fresh pull and bootstrap and several cleanings09:41
mnasiadkahrw: yeah well, both solutions are similar - difference is just no uninstallation on 2. :)09:42
Fl1nt@mgoddard, ok, cool, I'll test that in a lab env ^^09:43
openstackgerritYongjun Bai proposed openstack/kolla-ansible master: Fix:copy ca certificates to horizon container  https://review.opendev.org/71263009:43
mgoddardFl1nt: please make it clear in bug reports if you are mixing :)09:43
Fl1ntbtw, I looked at your kayobe swift roles, definitely worthy to be backported to kolla ^^09:44
mgoddardFl1nt: sure. Is that something you would be able to handle?09:44
Fl1ntI will for sure, but honestly, I'm not gonna do a bug report until I validated an error isn't coming from a mixin actually.09:45
mgoddardok09:45
Fl1ntI'll have to look at how all the plumbing works (Gerrit/CICD) but yep.09:45
openstackgerritYongjun Bai proposed openstack/kolla-ansible master: Fix:copy ca certificates to horizon container  https://review.opendev.org/71263009:48
openstackgerritYongjun Bai proposed openstack/kolla-ansible master: Fix:copy ca certificates to horizon container  https://review.opendev.org/71263009:49
hrwwhen "docker prune -af" ends with "Total reclaimed space: 132.6GB" it shows that too much kolla was done09:52
Fl1nthe he he that's actually the power of kolla, it really bloomed the Openstack enterprise grade lifecycle management.09:54
Fl1ntBefore kolla, companies that I worked with where all trying to implement their own lifecycle management solution with more or less success in the process, kolla changed that situation from one day to another.09:55
Fl1ntcompletely erased years if not decades of tech debts at some point for one of my previous job.09:55
Fl1ntand a guy that I worked with even find a sweet term for our platforms during the process, called them frankeinClouds ^^09:56
Fl1ntfrankenClouds ^^09:56
hrwFl1nt: do not ask what I maintained before kolla.09:57
openstackgerritMerged openstack/kolla-ansible master: Host OS prechecks follow up  https://review.opendev.org/71250209:57
hrwI can point you to git repo if you want nightmares09:57
Fl1ntWhat did you maintained @hrw ? :D09:57
hrwnice try09:57
Fl1ntsend the git repo, I'm always curious to look at what other engineered ^^09:58
hrwFl1nt: https://git.linaro.org/leg/sdi/openstack-ref-architecture.git/09:58
Fl1ntI worked with TripleO for a bit too... it was a nightmare, I love the way kayobe simplified all that ^^09:59
hrwFl1nt: if you ask me about anything about that repo I will not confirm that I ever worked on it.09:59
Fl1ntdude, that git solution is nothing compared to the crappy solution that I had to (recently <1 year) worked with, I can't link it the last one without risking any legal action, but trust me, at least yours is based on ansible usual structure.10:01
hrwFl1nt: to make it funnier - all that crap would be avoided if we did copy of liberty debian packages :D10:03
Fl1ntall I can say is, it was called cloud-management, and handled everything on an over-engineered structure that abused the ansible vars override and a crappy yarn system in order to try to streamline the external role dependencies etc.10:03
Fl1ntIt's funny to see how much crap can be produced by company only because they fear to collaborate each other on something similar that could benefit them.10:04
Fl1ntfrom what I can see, almost every company based their lifecycle tools on ansible, which is what kolla do, but all those initiative became crappy legacy because they didn't shared it. Until Kolla.10:05
hrwFl1nt: I think that adding aarch64 support, reviving Debian support and getting to point when we were able to deploy something took less workhours than dealing with previous10:05
hrwbbiab10:05
Fl1ntI didn't used a debian install since a while ^^10:10
Fl1ntEverything on RH or CentOS since around 12/13 years now10:11
*** zhanglong has joined #openstack-kolla10:17
*** ykarel|lunch is now known as ykarel10:19
*** reph has quit IRC10:22
yoctozeptohrw, mgoddard: hmm, we are using delorean on Train CentOS 8?10:24
mgoddardyoctozepto: yes10:26
*** rohit02 has joined #openstack-kolla10:26
rohit02can we use virtual network interface for neutron external in kolla ansible?10:27
hrwyoctozepto: centos8-train delorean to be exact10:28
yoctozeptohrw, mgoddard: ack, thanks10:28
hrwyw10:31
openstackgerritMarcin Juszkiewicz proposed openstack/kolla stable/train: openstack-base: pin setuptools < 46 to get horizon working  https://review.opendev.org/71264410:36
*** GeraldS has joined #openstack-kolla10:36
openstackgerritMerged openstack/kolla-ansible stable/train: Use internal API for masakari-monitor  https://review.opendev.org/71243110:36
openstackgerritMerged openstack/kolla-ansible stable/train: Swift: remove meta field from rsync command  https://review.opendev.org/71242810:36
openstackgerritMerged openstack/kolla-ansible stable/rocky: Swift: remove meta field from rsync command  https://review.opendev.org/71243010:36
openstackgerritMerged openstack/kolla-ansible stable/stein: Swift: remove meta field from rsync command  https://review.opendev.org/71242910:36
hrwalso marked RP+210:37
*** reph has joined #openstack-kolla10:38
Fl1ntIs there a reason why kolla fluentd container is using a more than deprecated version with rocky/stein ?10:52
mgoddardFl1nt: yes - nobody upgraded it10:53
mgoddardit is upgraded in train10:53
Fl1ntok ^^, I had that on my todo so ^^10:54
*** reph has quit IRC10:54
hrwFl1nt: fluentd/td-agent is nightmare10:55
Fl1ntyeah ^^ But I'm discussing with an operator, and he raised me a legitimate question, so I'll answer it and work to fix that ^^10:56
*** vishalmanchanda has quit IRC11:00
*** reph has joined #openstack-kolla11:01
*** iniazi has quit IRC11:07
*** kplant has joined #openstack-kolla11:09
*** ktibi has quit IRC11:10
*** ktibi has joined #openstack-kolla11:11
yoctozeptoFl1nt: go train :D11:15
yoctozeptohrw: https://review.opendev.org/712644 - needs action (tripleo)11:16
patchbotpatch 712644 - kolla (stable/train) - openstack-base: pin setuptools < 46 to get horizon... - 1 patch set11:16
Fl1ntI can't ^^ that's not on my call ^^11:16
Fl1ntbut I will on next month.11:17
Fl1nthum... are LABEL directive ever used within the kolla-build for packages version pining for instance ?11:20
Fl1ntbecause, td repo is pointing to: 3.x version: https://opendev.org/openstack/kolla/src/tag/9.0.1/docker/base/td.repo11:21
Fl1ntbut dockerfile label state11:21
Fl1ntfluentd_version=0.1211:22
Fl1nthttps://opendev.org/openstack/kolla/src/tag/9.0.1/docker/fluentd/Dockerfile.j2#L411:22
mgoddardFl1nt: that is confusingly named. We basically check for <=0.12 or >0.12.11:23
hrwFl1nt: depends on distro and arch we use either td-agent or fluentd and either 0.12 or 0.1411:23
mgoddardFl1nt: and fluentd version != td-agent version...11:23
hrwor sth like that11:23
hrwit is mess11:23
mgoddardnot entirely of our own making :)11:23
hrwArm Treasuredata is a .... ..... ..... unable to do releases11:23
Fl1ntDistro CentOS RPM arch x86_6411:23
Fl1ntso the packages installed should be either: 2.3.6-0 for rocky/stein and 3.6.0-0 for train, however, it seems it's 0.12 which is installed. so, how is macro.j2 using LABEL in order to pin the package? at yum install or within the repo?11:25
openstackgerritMerged openstack/kayobe master: CentOS 8: Add seed and overcloud CI jobs, drop CentOS 7 jobs  https://review.opendev.org/70978711:29
hrwyoctozepto: needs tripleo team as psutil/_psutil_common.c:9:20: fatal error: Python.h: No such file or directory11:30
yoctozeptohrw: argh11:30
mgoddardFl1nt: the label is just indicative. It's used in kolla-ansible to control logic11:32
Fl1ntok, so the td-agent package in my case will use 2.3.6-0 rpm version and LABEL will be 0.14 that kolla-ansible will use for ???11:33
*** rohit02 has quit IRC11:33
Fl1ntok, got it11:35
Fl1ntso11:35
Fl1ntwith rocky/stein, on a CentOS x86_64 install, fluentd role use td-agent 2.3.6-0 rpm package version which so use a fluentd 0.12 and with train it install 3.6.0-0 which use a fluentd 1.011:36
Fl1ntmeaning that rocky/stein fluentd configs are 0.12 and train are 1.0 ok cool11:37
openstackgerritMerged openstack/kolla master: openstack-base: pin setuptools < 46 to get horizon working  https://review.opendev.org/71248511:44
*** rgogunskiy has joined #openstack-kolla11:46
*** cah_link has quit IRC12:04
*** reph has quit IRC12:10
*** Luzi has quit IRC12:10
*** reph has joined #openstack-kolla12:11
*** zhanglong has quit IRC12:15
*** reph has quit IRC12:16
*** gfidente has joined #openstack-kolla12:21
openstackgerritMark Goddard proposed openstack/kayobe stable/train: Make local kolla-ansible Python executable configurable  https://review.opendev.org/71106012:28
*** zhanglong has joined #openstack-kolla12:30
*** Fl1nt has quit IRC12:35
mgoddardhrw: you mentioned this earlier: fatal error: Python.h: No such file or directory12:36
mgoddardI am seeing that on tenks12:36
mgoddardalso seeing breakage on kayobe12:36
hrwyay12:37
mgoddardI think that virtualenv command now uses python3 by default on CentOS 712:37
*** Fl1nt has joined #openstack-kolla12:37
mgoddardso python3-devel missing12:37
hrwpatch it with -ppython2 if centos7?12:37
mgoddardyes12:37
hrwwe agreed no py3 for c712:38
mgoddardbut that's a bad default for CentOS 712:38
yoctozeptoping centos-devel12:38
hrwmgoddard: py3 or py2 as bad?12:39
mgoddardpy3 on centos is half-assed12:39
hrwindeed12:43
Fl1ntexcept on C812:45
hrwFl1nt: there is no py2 on c8.12:45
Fl1ntno I know, I talked about PY312:45
hrwI know12:45
*** dciabrin_ is now known as dciabrin12:45
Fl1nt^^12:46
Fl1ntbtw, don't get why they didn't used dnf as default replacement for yum yet on C812:47
kplantthey did12:48
kplantyum is a symlink to dnf12:48
*** zhanglong has quit IRC12:49
yoctozeptomgoddard, hrw: it's mostly missing selinux bindings for source builds I guess? (py3)12:51
yoctozeptoanyways, that venv thingy sounds breaking12:51
hrwFl1nt: yum v4 is dnf12:52
hrwFl1nt: yum 'binary' is there for those used to yum12:52
yoctozeptohrw, mgoddard: is centos channel logged?12:53
mgoddardnot sure12:53
Fl1nt@kplant, wait what?? So someone lied to me that whole time??12:53
* Fl1nt go check that12:53
Fl1ntWoooh you're right @kplant nice one ^^ I didn't checked earlier but it's sweet ^^12:56
Fl1nt@hrw, yeah, just didn't knew that they've made that link, just always thought that centos would have switch to dnf without looking at those used to the yum command as fedora did ^^12:58
Fl1nthabits are hard to loose I guess ^^12:58
Fl1ntexit12:58
Fl1ntwhoops, wrong window ^^12:59
yoctozeptomgoddard, hrw: so - are we hitting red hat folks?12:59
yoctozeptohrw <- red hat folk12:59
mgoddardshould probably raise it with them13:00
mgoddardwhere to start?13:00
yoctozeptowith hrw if he knows the quickest path13:00
yoctozeptootherwise bugzilla13:00
hrwyoctozepto: ?13:00
hrwFl1nt: note that on c7 you can enable epel and install dnf13:00
yoctozeptohrw: centos claims it's upstream13:00
yoctozeptohrw: the virtualenv switching defaults13:00
yoctozeptomgoddard: https://bugzilla.redhat.com/13:01
hrw13:37 < hrw> patch it with -ppython2 if centos7?13:01
yoctozeptoif no better way13:01
hrwno, you ...13:01
hrwif you want py2 then add '-p python2' to virtualenv call13:01
hrwdone13:01
hrwif you want something done for sure, then do not rely on defaults13:01
yoctozeptohrw: yeah, but like 99% of python world expects virtualenv to run python, not python3?13:02
hrwand?13:02
*** rpittau|bbl is now known as rpittau13:02
yoctozeptohrw: surprised you are not surprised13:02
hrwyou want it fixed in kolla or want to spend unknown amount of time for change somewhere?13:02
yoctozeptohrw: fair point13:03
hrw14:02 (0s) hrw@puchatek:~$ python --version13:03
hrwPython 3.7.613:03
hrwno one sane want to run 'python'13:03
hrwyou either want 'python3' or 'python2'13:03
hrwif your work is multidistro/os/arch/whatever13:03
yoctozeptohrw: it's centos/rhel 7, they normally don't do such switches over night, that's what we expect from them and that's why we choose them13:04
hrw[root@4eb21ecff0ab /]# virtualenv /tmp/v13:05
hrwNew python executable in /tmp/v/bin/python13:05
hrwInstalling setuptools, pip, wheel...done.13:05
hrw[root@4eb21ecff0ab /]# . /tmp/v/bin/activate13:05
hrw(v) [root@4eb21ecff0ab /]# python --version13:05
hrwPython 2.7.513:05
hrwcentos:7 container13:05
yoctozeptohrw: weird, just checked myself and had the same13:05
Fl1ntfor that one I do exactly the same as hrw, my CICD worker host is using virtualenv with a VIRTUALENV_PYTHON=blabla and then I'm mkproject with -ppython3 whenever I want it, like with kolla-ansible for instance.13:05
yoctozeptohrw, mgoddard: dafu ^13:05
* hrw -> stable/train13:06
yoctozepto(I mean hrw, not Fl1nt message)13:06
hrw "stdout": "created virtual environment CPython3.6.8.final.0-64 in 442ms13:07
yoctozeptomgoddard, hrw: in case of tripleo job, they run pip3 to install virtualenv so no wonder it gets python3 inside13:09
yoctozeptomgoddard, hrw: not sure if that was done deliberately or ansible actually changed defaults13:09
mgoddardspinning up a VM to test13:09
yoctozeptomgoddard, hrw: https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/base/pre.yaml#L43-L49 partially deliberately13:10
yoctozeptomgoddard, hrw: I'm betting on infra images to have received python3 blessings13:11
hrwargh13:12
yoctozeptoonto tripleo13:12
openstackgerritMark Goddard proposed openstack/kayobe stable/train: WIP: Force python 2 for virtualenvs  https://review.opendev.org/71267313:15
yoctozeptomgoddard: any idea what hit kayobe there?13:17
mgoddardyoctozepto: nope13:17
yoctozeptomgoddard: well, tripleo kinda hit themselves13:18
yoctozeptomgoddard: aren't you doing pip installs in kayobe?13:18
yoctozeptoon bare I mean13:18
yoctozeptoand detect pip3 and prefer13:19
yoctozeptoto install virtualenv13:19
hrwyoctozepto: I may not care did tripleo hit themselves or not. but that fact hits us13:19
yoctozeptohrw: disable voting? ;p13:20
yoctozeptohrw: kolla itself not affected13:20
hrwyoctozepto: kolla is affected as long tripleo CI is voting13:20
*** k_mouza has quit IRC13:21
yoctozeptohrw: you must be in bad mood today13:22
mnasiadkaoh boy, just set it as non voting and case closed13:22
*** cah_link has joined #openstack-kolla13:24
hrwyoctozepto: maybe13:27
yoctozeptomnasiadka: easier said than done13:27
mnasiadkayoctozepto: why?13:27
yoctozeptomnasiadka: it's not friendly towards tripleo ;-)13:29
hrwyoctozepto: did you contacted them? discussed issue?13:30
yoctozeptocloudnull: emergency tripleo gate situation blocking kolla train, please help - we are considering non-voting soon13:30
yoctozeptohrw: yeah, on tripleo13:30
yoctozeptohrw: ignored me13:30
yoctozeptohrw: last hope in our liaison13:30
cloudnullyoctozepto - please state the nature of our tripleo emergency.13:31
yoctozeptocloudnull: hi, thanks for coming13:31
yoctozeptocloudnull: https://zuul.opendev.org/t/openstack/build/bf343e1ad1684bfb902fb0703085e92213:31
yoctozeptofatal error: Python.h: No such file or directory13:31
yoctozeptocloudnull: debugged down to this13:31
yoctozeptohttps://opendev.org/opendev/base-jobs/src/branch/master/playbooks/base/pre.yaml#L43-L4913:32
cloudnullhum, was python-devel removed ?13:32
yoctozeptocloudnull: no idea, but it must have gone one way or another13:32
mgoddardcloudnull: it appears that python3 is being used by default by virtualenv13:32
yoctozeptocloudnull: it seems to be python3-devel though13:32
yoctozepto^13:32
yoctozeptocloudnull: see the lines I highlighted13:32
mgoddardwe are seeing similar in kayobe & tenks13:32
yoctozeptocloudnull: you detect pip3 and install it for py313:32
yoctozeptocloudnull: betting on change in infra image13:33
yoctozeptomgoddard: ^13:33
mgoddardyoctozepto: are you sure those lines are relevant? they went in some time ago13:33
cloudnull++13:33
hrwcloudnull: your CI job does py3 virtualenv13:33
yoctozeptomgoddard: maybe pip3 was not there at the time13:33
cloudnullthat's a base job for all of infra13:33
yoctozeptoit does pip3 virtualenv13:33
mgoddardI can't reproduce locally13:33
yoctozeptoso no wonder it gets python venv13:33
* cloudnull py3 has venv built in, not sure why anyone would be virtualenv ...13:34
yoctozeptomgoddard: try installing virtualenv with pip313:34
yoctozeptothere is some mess and something has changed13:34
mgoddardah, that could be it13:34
yoctozeptothe part from pip3 to virtualenv to python3 is clear13:34
* cloudnull going to go ask some folks whats what 13:35
yoctozeptonow, kayobe must be either using pip3 implicitly or semi-explicitly (if you are surprised)13:35
yoctozeptoand tripleo does it plainly13:35
mgoddardyoctozepto: I think you have it13:36
mgoddardI did a pip3 install virtualenv13:36
mgoddardnow my virtualenvs are py3 (no surprise)13:36
cloudnulltripleo would only use py3 on cent8+13:36
yoctozeptocloudnull: we too13:37
cloudnullthis job is running cent7, unless i'm mistaken ?13:37
yoctozeptocloudnull: but it happens otherwise - as I said - betting on infra change13:37
yoctozeptocloudnull: c713:37
yoctozeptocloudnull: I confirm13:37
cloudnullthe job name says c7 but the output shows py3.613:38
cloudnullso idk ...13:38
cloudnullto fix this I suspect all we need is to have python{2,3}-devel installed.13:39
*** skramaja has quit IRC13:40
openstackgerritMerged openstack/kolla-ansible stable/rocky: Fix HAProxy monitor VIP precheck  https://review.opendev.org/71244113:40
yoctozeptocloudnull: for the first try, running off py3 in c7 could still be surprising in the long run13:42
cloudnull++13:43
cloudnullin in the #infra channel, it seems that py3 is being installed by default there .13:44
cloudnullwhich is surprising13:44
yoctozeptomgoddard: mhm, you are using image-provided virtualenv in kayobe which ends up being py313:46
yoctozeptocloudnull: ^13:46
yoctozeptomgoddard, cloudnull: so both the issues are the same mess13:47
yoctozeptomgoddard, cloudnull: moving to infra then13:47
stingrayzahey all o/ been a while. (been far away from my openstack for some time, but recently back...)13:51
stingrayzajust upgraded my local k-a deployment to train - what an easy experience (and give yourselves a pat on the back for making it so)13:52
yoctozeptostingrayza: thanks :-) I did it back in the day and was also happy; there was no end to patting :-)13:53
*** bengates has quit IRC13:57
*** dave-mccowan has joined #openstack-kolla14:02
openstackgerritRadosław Piliszek proposed openstack/kolla stable/train: openstack-base: pin setuptools < 46 to get horizon working  https://review.opendev.org/71264414:10
*** bengates has joined #openstack-kolla14:12
openstackgerritMark Goddard proposed openstack/kayobe stable/train: WIP: Force python 2 for virtualenvs  https://review.opendev.org/71267314:20
openstackgerritRadosław Piliszek proposed openstack/kayobe stable/train: DNM: use base-test for kayobe-base to test drop of bad workaround  https://review.opendev.org/71269414:22
yoctozeptomgoddard: ^14:22
mgoddardyoctozepto: interesting14:23
yoctozeptomgoddard: going to become infra some day this way...14:23
mgoddardwhy not14:23
yoctozeptomgoddard: yeah, why, kolla ptl dreaming of first-class service? :-)14:24
emccormickHey, did the simple logo customization ever land?14:24
yoctozepto(not that I would mind much)14:24
*** jimcrowleyibm has joined #openstack-kolla14:24
emccormickNot seeing a doc for it from my google fu14:24
yoctozeptoemccormick: probably not, you mean something horizon in kolla-ansible?14:25
emccormickyes, sorry. Horizon14:25
*** dave-mccowan has quit IRC14:25
emccormickwe had a blueprint back in 2017 to allow for replacing the horizon logos without having to hack them into the container image14:26
yoctozeptoemccormick: I guess it got stuck at kolla :/14:27
emccormick*sniffle*14:28
openstackgerritMark Goddard proposed openstack/kayobe stable/train: WIP: Force python 2 for virtualenvs  https://review.opendev.org/71267314:29
*** dave-mccowan has joined #openstack-kolla14:30
hrwemccormick: do your theme, use overlays to add it into image and then list own theme first on deployment?14:30
emccormickYeah I tried doing that a couple releases ago but never got it to work14:31
*** abdysn has quit IRC14:31
hrwwe did that way at Linaro14:31
emccormickI don't really have a full blown theme replacement though14:31
hrwemccormick: your theme can be copy of original one + own logo14:31
emccormickI just did a little CSS to tweak colors, and then replaced the logo files14:31
*** Fl1nt has quit IRC14:32
emccormickI'm honestly not very good with all that front end stuff ;)14:32
emccormickdo you know exactly what to replace in the container image? Everything I ever replaced never seemed to change it14:33
hrwemccormick: https://git.linaro.org/leg/sdi/kolla/ldc-overlay.git/tree/linaro-override-train.j214:33
hrwemccormick: we pass it via --template-override14:34
emccormickok cool. thanks. Will give it a shot14:36
*** rgogunskiy has quit IRC14:36
openstackgerritMark Goddard proposed openstack/kayobe stable/train: WIP: Force python 2 for virtualenvs  https://review.opendev.org/71267314:39
*** ricolin_ has quit IRC14:41
*** ricolin_ has joined #openstack-kolla14:41
mgoddardyoctozepto: your kayobe patch looks quite happy so far14:43
yoctozeptomgoddard: yup14:46
yoctozeptomgoddard: so train revert is possible14:46
yoctozeptomgoddard: also testing rocky for ubuntu py 3.514:46
yoctozeptomgoddard: (devstack)14:46
mgoddardwell this is just kayobe14:46
mgoddardbut it's a good sign14:46
weshay|ruckmgoddard, catching up14:46
yoctozeptomgoddard: if virtualenv was broken, you would see it14:46
weshay|rucklooking for patches where tripleo train and earlier are failing14:47
yoctozeptoweshay|ruck: let me summarize this for you14:47
weshay|ruckk14:47
yoctozeptoweshay|ruck: infra installed python3+pip3 in centos7 images14:47
yoctozepto(well, all for that matter)14:47
*** zbr has joined #openstack-kolla14:47
yoctozeptoweshay|ruck: and base job does detection of py version based on pip3 availability14:47
weshay|ruckah14:47
yoctozeptoweshay|ruck: and replaced virtualenv happily14:48
yoctozeptoweshay|ruck: ta-da14:48
*** ricolin_ has quit IRC14:48
yoctozeptoweshay|ruck: working with infra on this14:48
weshay|ruckk.. thank you.. /me catching up in there as well14:48
*** diurnalist has joined #openstack-kolla14:53
weshay|ruckk..14:54
weshay|rucksee the issue14:54
weshay|ruck2020-03-12 01:51:36.567684 | LOOP [build-containers : Install deps from upper-constraints]14:54
weshay|ruck2020-03-12 01:51:53.406550 | primary | changed: "kolla"14:54
weshay|ruck2020-03-12 01:52:27.887975 | primary | changed: "tripleo-common"14:54
weshay|ruck2020-03-12 01:52:37.301868 | primary | ERROR: Item: python-tripleoclient14:54
weshay|ruck2020-03-12 01:52:37.302375 | primary | {14:54
weshay|ruck2020-03-12 01:52:37.302512 | primary |   "ansible_loop_var": "item",14:54
*** iniazi has joined #openstack-kolla14:59
weshay|ruckyoctozepto, mgoddard ok.. we're discussing how to resolve now15:01
weshay|ruckthanks for the info15:01
iniazion a failed deployment, can I re-run deployment again after fixing it or should I destroy and then deploy again?15:02
*** GeraldS has quit IRC15:02
iniaziits been a while since I did a k-a deployment15:03
*** suff has quit IRC15:04
mgoddardweshay|ruck: FWIW, I think we have two candidate solutions for kayobe at least15:06
mgoddardweshay|ruck: bodge: uninstall virtualenv3, reinstall virtualenv2 https://review.opendev.org/71267315:06
patchbotpatch 712673 - kayobe (stable/train) - WIP: Force python 2 for virtualenvs - 4 patch sets15:06
mgoddardweshay|ruck: better: fix in base zuul job: https://review.opendev.org/71269415:07
patchbotpatch 712694 - kayobe (stable/train) - DNM: use base-test for kayobe-base to test drop of... - 1 patch set15:07
*** sri_ has quit IRC15:08
*** arxcruz|rover has joined #openstack-kolla15:09
arxcruz|roverweshay|ruck: p/15:09
*** reph has joined #openstack-kolla15:11
yoctozeptomgoddard, hrw: infra fix going in 8-)15:19
mgoddardyoctozepto: sweet!15:20
mgoddardthanks for pushing on it15:20
yoctozeptoweshay|ruck, cloudnull, arxcruz|rover: ^^15:20
yoctozeptomgoddard: felt brave enough today15:20
weshay|ruckk.. fyi https://review.opendev.org/#/c/712706/15:23
patchbotpatch 712706 - tripleo-ci - Install python headers - 1 patch set15:23
hrwyoctozepto: coolio!15:25
yoctozeptofor future reference: https://review.opendev.org/712107 <- this is the patch going in15:27
patchbotpatch 712107 - opendev/base-jobs - Cleanup workaround for importlib-resources bad ver... - 1 patch set15:27
yoctozeptoweshay|ruck: k, makes sense to possibly delay failure15:28
yoctozeptoaaand it merged :D15:29
openstackgerritRadosław Piliszek proposed openstack/kolla stable/train: openstack-base: pin setuptools < 46 to get horizon working  https://review.opendev.org/71264415:30
yoctozeptohrw, mgoddard: ^15:30
mgoddardniiice15:31
hrwuf15:32
openstackgerritRadosław Piliszek proposed openstack/kolla stable/train: Add ssh clients to Ironic Conductor container  https://review.opendev.org/71242215:34
openstackgerritRadosław Piliszek proposed openstack/kolla stable/train: Clean up collectd dockerfile  https://review.opendev.org/71189115:34
*** ykarel is now known as ykarel|away15:34
openstackgerritRadosław Piliszek proposed openstack/kolla stable/train: kibana: enable for non-x86 on Debian/Ubuntu  https://review.opendev.org/70778915:34
yoctozeptorebased w+115:35
*** TrevorV has joined #openstack-kolla15:39
hrw 70 files changed, 170 insertions(+), 626 deletions(-)15:44
hrwcleaning will be ugly15:44
openstackgerritMark Goddard proposed openstack/kolla master: CI: CentOS 8: Enable ceph-ansible jobs  https://review.opendev.org/71271915:50
*** cah_link has quit IRC15:50
*** cah_link has joined #openstack-kolla15:50
openstackgerritMark Goddard proposed openstack/kayobe master: CentOS 8: Enable upgrade jobs  https://review.opendev.org/71272315:58
*** xinliang has joined #openstack-kolla16:11
openstackgerritMark Goddard proposed openstack/kolla stable/rocky: Bump manila version for Rocky  https://review.opendev.org/71272916:15
iniazihmm, trying to deploy stable/train (i.e. git), and getting error on undefined zookeeper_port for telegraf; which sounds strange, i.e. i'm assuming its zookeeper_client_port it wants?16:18
iniazimy question is, am I missing something else that its giving me an undefined on it... or should i just configure that variable in globals.yml?16:19
*** xinliang has quit IRC16:20
iniaziwell anyway, i'm going to just set that variable (wonder if i can do zookeeper_port: "{{ zookeeper_client_port }}" in globals.yml" ) and i guess redeploy16:22
openstackgerritRadosław Piliszek proposed openstack/kolla master: CI: Ignore .zuul.d  https://review.opendev.org/71273016:25
yoctozeptomgoddard: ^16:25
mgoddardyoctozepto: classic16:26
yoctozeptomgoddard: k-a is happy16:26
*** TrevorV has quit IRC16:27
yoctozeptoiniazi: this part is unfortunately untested; if you think this is a bug, please report16:27
*** TrevorV has joined #openstack-kolla16:28
*** Blinkiz has quit IRC16:30
*** Blinkiz has joined #openstack-kolla16:31
*** TrevorV has quit IRC16:31
*** TrevorV has joined #openstack-kolla16:32
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Separate per-service host configuration tasks  https://review.opendev.org/70312716:33
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Separate per-service host configuration tasks  https://review.opendev.org/70312716:36
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Improve Ansible version check error handling  https://review.opendev.org/70526116:40
openstackgerritMark Goddard proposed openstack/kayobe master: Avoid writing out requirements.txt in kolla-ansible role  https://review.opendev.org/70785316:42
openstackgerritMark Goddard proposed openstack/kayobe master: CI: Add overcloud host configure job  https://review.opendev.org/71027716:44
openstackgerritMark Goddard proposed openstack/kayobe master: CentOS 8: Support DNF  https://review.opendev.org/71104416:49
openstackgerritMark Goddard proposed openstack/kayobe master: CI: Add overcloud host configure job  https://review.opendev.org/71027716:49
*** bengates has quit IRC16:50
*** chrizl has joined #openstack-kolla16:50
*** bengates has joined #openstack-kolla16:51
*** bengates has quit IRC16:56
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Copy ca certificates also to horizon container  https://review.opendev.org/71263016:59
*** priteau has quit IRC17:01
kplantwould any of you smart people have a guess as to why i can create new instances with flavor X but cannot resize instances to flavor X17:04
iniaziyoctozepto, hmm, is there specific tags I should attach to the bugreport?17:04
kplanti have plenty of available resources17:04
yoctozeptoiniazi: mostly irrelevant17:04
yoctozeptokplant: any specific error message? probably question to nova either way17:05
kplantjust the classic: "No valid host was found. No valid host found for resize (HTTP 400)"17:05
iniazisubmitted https://bugs.launchpad.net/kolla-ansible/+bug/186717917:05
openstackLaunchpad bug 1867179 in kolla-ansible "Deploy fails on zookeeper_port undefined in telegraf.conf.j2" [Undecided,New]17:05
hrwkplant: looks like not enough resources17:05
kplantit does, but i have plenty of resources17:05
kplantand i can create new instances with the same flavor17:05
hrwkplant: go debug level of logs on nova?17:06
kplantactually am in the middle of the reconfigure pushing out debug=True now17:07
openstackgerritJames Kirsch proposed openstack/kolla master: Install uwsgi for Keystone  https://review.opendev.org/71087917:07
kplantoh that's _super_ annoying17:17
kplantit wasn't working because i renamed the AZ17:17
openstackgerritMark Goddard proposed openstack/kayobe stable/train: Switch from shade to openstacksdk  https://review.opendev.org/71106117:18
openstackgerritMark Goddard proposed openstack/kayobe stable/train: Remove activate-virtualenv and deactivate-virtualenv roles  https://review.opendev.org/71106217:18
openstackgerritMark Goddard proposed openstack/kayobe stable/train: CentOS 8: Use ansible_playbook_python for localhost dependencies  https://review.opendev.org/71106317:18
openstackgerritMark Goddard proposed openstack/kayobe stable/train: Make Kayobe code compatible with Python 3  https://review.opendev.org/71106417:18
openstackgerritMark Goddard proposed openstack/kayobe stable/train: CentOS 8: Bump MichaelRigart.interfaces to 1.4.0  https://review.opendev.org/71106517:18
openstackgerritMark Goddard proposed openstack/kayobe stable/train: CentOS 8: Disable ntpd, enable chrony container  https://review.opendev.org/71106617:18
openstackgerritMark Goddard proposed openstack/kayobe stable/train: CentOS 8: Use same local python version for kolla-ansible  https://review.opendev.org/71121917:18
openstackgerritMark Goddard proposed openstack/kayobe stable/train: CentOS 8: Add seed and overcloud CI jobs  https://review.opendev.org/71106717:18
openstackgerritMark Goddard proposed openstack/kolla-ansible master: DNM: test ansible group prechecks  https://review.opendev.org/71274317:21
*** rpittau is now known as rpittau|afk17:27
*** cah_link has quit IRC17:29
*** evrardjp has quit IRC17:35
*** evrardjp has joined #openstack-kolla17:36
iniazihmm getting an error on kibana to register in es.  its trying to match a hostname (showing as an ip) to the kolla_internal_fqdn, i'm guessing: (this is on deploy on stable/train):17:43
iniazifatal: [controller01]: FAILED! => {"attempts": 20, "changed": false, "msg": "hostname '10.x.x.x' doesn't match either of 'control.DOMAIN', 'controller.DOMAIN'", "status": -1, "url": "https://10.x.x.x:9200/.kibana"}17:44
openstackgerritMark Goddard proposed openstack/kolla master: CentOS 8: Enable hacluster and masakari images  https://review.opendev.org/70270617:44
iniazishould kolla_internal_vip_address be set to the dnsname and not an IP?17:45
*** kozhukalov has joined #openstack-kolla17:46
iniazii'll just set it to the hostname?17:47
hrwcan anyone build ubuntu/binary neutron-base image?17:50
yoctozeptohrw: which branch? (and what's to be expected?)17:56
hrwmaster17:58
hrwyoctozepto: I expect it to build17:59
hrwof course17:59
yoctozeptohrw: hah, ok, running18:04
*** Trevor_V has joined #openstack-kolla18:07
*** TrevorV has quit IRC18:10
*** jimcrowleyibm is now known as jimcrowleyibm[aw18:16
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Fix telegraf with zookeeper (wrong port variable reference)  https://review.opendev.org/71275818:16
yoctozeptohrw: Failed to fetch http://piotrkosoft.net/pub/mirrors/ubuntu/dists/bionic-security/main/binary-amd64/Packages.gz  File has unexpected size (834437 != 779543). Mirror sync in progress?18:18
yoctozeptoI guess it was random? ;D18:18
yoctozeptoretrying18:18
yoctozeptohrw: INFO:kolla.common.utils.neutron-base:E: Unable to locate package python3-networking-ovn18:19
yoctozeptohrw: kolla RED again? ;o18:20
*** reph has quit IRC18:21
hrwthank you18:22
hrwyoctozepto: not RED. it is Polish mirror issue18:23
yoctozeptohrw: nah, it got past mirror sync, I mean lack of ovn18:24
hrwwill check on server18:24
hrwlooks like package got dropped from UCA18:26
hrwyes. train had 7.0.0 and it is the latest version of that package in UCA archive18:27
yoctozeptohrw: argh18:29
yoctozeptohrw: and we want ovn in ussuri...18:29
yoctozeptomnasiadka: ^18:29
hrwhttps://paste.centos.org/view/26a6caa5 ;(18:30
mnasiadkayoctozepto: it’s now in neutron code, no worries18:30
mnasiadkaI can raise a change in master tomorrow to sort out this mess18:30
mnasiadkaNetworking-ovn should cease to exist from Ussuri18:31
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: neutron-base: drop python3-networking-ovn on Ubuntu  https://review.opendev.org/71276218:31
hrwmnasiadka: o! even easier!18:32
mnasiadkahrw: you’re welcome :)18:32
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: neutron-base: drop python3-networking-ovn  https://review.opendev.org/71276218:32
hrwmnasiadka: please improve commit message18:32
* hrw off18:33
mnasiadkahrw: can I do it tomorrow? Mind blown from a week worth of fight with rabbitmq and neutron/queens :)18:33
yoctozeptohrw, mnasiadka: thanks for handling! I forgot it ships by default from Ussuri18:34
hrwmnasiadka: no problemo18:36
*** gfidente is now known as gfidente|afk18:41
openstackgerritRadosław Piliszek proposed openstack/kolla master: CI: Enable Docker IPv6  https://review.opendev.org/71276518:44
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: CI: IPv6 external network  https://review.opendev.org/71276818:50
*** Trevor_V has quit IRC18:51
*** jimcrowleyibm has joined #openstack-kolla18:51
*** jimcrowleyibm is now known as jimcrowleyibm[aw18:55
*** gmann is now known as gmann_lunch19:00
*** gfidente|afk has quit IRC19:05
*** jimcrowleyibm has joined #openstack-kolla19:08
openstackgerritMerged openstack/kolla stable/train: openstack-base: pin setuptools < 46 to get horizon working  https://review.opendev.org/71264419:12
*** kozhukalov has quit IRC19:13
*** jimcrowleyibm is now known as jimcrowleyibm[aw19:14
*** diurnalist has quit IRC19:14
*** reph has joined #openstack-kolla19:22
openstackgerritMerged openstack/kolla stable/train: kibana: enable for non-x86 on Debian/Ubuntu  https://review.opendev.org/70778919:25
openstackgerritMerged openstack/kolla stable/train: Clean up collectd dockerfile  https://review.opendev.org/71189119:25
openstackgerritMerged openstack/kolla stable/train: Add ssh clients to Ironic Conductor container  https://review.opendev.org/71242219:25
*** reph has quit IRC19:49
*** kplant has quit IRC19:50
*** gmann_lunch is now known as gmann20:01
*** diurnalist has joined #openstack-kolla20:33
*** rgogunskiy has joined #openstack-kolla20:38
*** rgogunskiy has quit IRC20:42
*** ktibi has quit IRC20:45
iniazihmm, anyone using kibana (and es i suppose) and possibly caused by some logic (haven't looked through yet) of having kolla_internal_fqdn.21:04
iniazisomewhere it looks for the kolla_internal_vip_address to match one of the two kolla_internal_fqdn and kolla_external_fqdn , as part of the kolla config and fails on:21:04
iniazi{"attempts": 20, "changed": false, "msg": "hostname '10.x.x.x' doesn't match either of 'control.DOMAIN', 'controller.DOMAIN'", "status": -1, "url": "https://10.x.x.x:9200/.kibana"}21:05
iniazioh i asked the question earlier.  i tried to set the kolla_internal_vip_address as hostname instead of IP.   but then it breaks chrony, so maybe something i'm missing.  or is kibana not used much and possibly a bug21:06
openstackgerritMerged openstack/kolla-ansible master: Copy ca certificates also to horizon container  https://review.opendev.org/71263021:11
*** jimcrowleyibm has joined #openstack-kolla21:17
openstackgerritJames Kirsch proposed openstack/kolla master: Install uwsgi for Keystone  https://review.opendev.org/71087921:27
*** jimcrowleyibm is now known as jimcrowleyibm[aw21:29
iniazihmm in stable/stein, url is constructed with kolla_internal_vip_address url: "{{ internal_protocol }}://{{ kolla_internal_vip_address }}:{{ elasticsearch_port }}/.kibana"21:34
iniazibut in stable/train its via kolla_internal_fqdn: url: "{{ internal_protocol }}://{{ kolla_internal_fqdn | put_address_in_context('url') }}:{{ elasticsearch_port }}/.kibana"21:34
iniazioops, the last one, i quoted master, not stable/train.  both use kolla_internal_vip_address, but only add 'put_address_in_context'.  sorry for the noise.  i'll try to change it to whats in master21:38
*** jbadiapa has quit IRC22:06
*** jbadiapa has joined #openstack-kolla22:09
*** etp has quit IRC22:15
*** etp has joined #openstack-kolla22:16
*** factor has quit IRC22:27
*** diurnalist has quit IRC22:28
*** diurnalist has joined #openstack-kolla22:36
*** jbadiapa has quit IRC22:37
*** diurnalist has quit IRC22:43
*** diurnalist has joined #openstack-kolla22:44
*** dave-mccowan has quit IRC22:50
*** dave-mccowan has joined #openstack-kolla23:10
*** threestrands has joined #openstack-kolla23:50

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!