Friday, 2020-03-13

*** dychen has joined #openstack-infra00:02
*** rlandy has quit IRC00:17
*** tosky has quit IRC00:18
openstackgerritIan Wienand proposed zuul/zuul master: [dnm] unit test to see if jobs run when nodes are updated  https://review.opendev.org/71282100:25
*** jamesmcarthur has joined #openstack-infra00:27
openstackgerritMerged opendev/system-config master: pip3: Add python3-distutils  https://review.opendev.org/71281800:29
*** tetsuro has joined #openstack-infra00:36
*** admcleod has quit IRC00:37
*** armax has quit IRC00:40
*** jamesmcarthur has quit IRC00:44
*** jamesmcarthur has joined #openstack-infra00:47
*** dchen has quit IRC00:50
*** dychen has quit IRC00:52
*** matt_kosut has joined #openstack-infra01:00
*** tetsuro_ has joined #openstack-infra01:01
*** matt_kosut has quit IRC01:04
*** tetsuro has quit IRC01:05
*** gyee has quit IRC01:05
*** admcleod has joined #openstack-infra01:11
*** Goneri has quit IRC01:15
*** jamesmcarthur has quit IRC01:17
*** diablo_rojo has quit IRC01:22
*** dave-mccowan has joined #openstack-infra01:32
*** armax has joined #openstack-infra01:40
openstackgerritIan Wienand proposed opendev/system-config master: [dnm] test with plain nodes  https://review.opendev.org/71281901:40
*** yamamoto has quit IRC01:41
*** yamamoto has joined #openstack-infra01:42
openstackgerritIan Wienand proposed opendev/system-config master: [dnm] test with plain nodes  https://review.opendev.org/71281901:43
openstackgerritIan Wienand proposed opendev/system-config master: [dnm] test with plain nodes  https://review.opendev.org/71281901:44
ianwoh damn, should have distributed a config file to the new nb to stop it trying to build everything ...01:49
*** yamamoto has quit IRC01:49
openstackgerritIan Wienand proposed openstack/project-config master: Move fedora-30 builds to nb01.opendev.org  https://review.opendev.org/69312001:55
ianwinfra-root: ^ i'll probably merge that despite probably not too many reviewers, because currently nb01 is trying to take build jobs and failing and i don't think that's good02:01
*** yamamoto has joined #openstack-infra02:04
*** yamamoto has quit IRC02:05
clarkbI can take a quick look02:08
clarkbseems fine +202:09
fungiapproved it02:10
ianwthanks; i'll watch the container, and restart it if it needs it02:11
fungiawesome02:18
openstackgerritMerged openstack/project-config master: Move fedora-30 builds to nb01.opendev.org  https://review.opendev.org/69312002:22
openstackgerritIan Wienand proposed opendev/system-config master: nodepool-builder: add /opt/dib_cache  https://review.opendev.org/71282402:22
*** jamesmcarthur has joined #openstack-infra02:25
ianwoh, that's one i didn't think of; we hard-code the path to "cacert: /home/nodepool/.config/openstack/limestone_cacert.pem" in the config file02:28
ianwand rax is giving : Max retries exceeded with url: /v2/images/f62dc67c-d867-46f8-80e8-6e666ac77b8d (Caused by SSLError(SSLError(1, '[SSL: WRONG_SIGNATURE_TYPE] wrong signature type (_ssl.c:1076)')))02:30
ianwi wonder if we don't install ca-certs maybe ...?02:30
ianwno, it seems to be there02:35
*** psachin has joined #openstack-infra02:43
*** ociuhandu has joined #openstack-infra02:43
ianwOOHHHH i don't think we want that02:43
*** stevebaker has quit IRC02:44
ianwdamn .. i think the container builder thinks it owns all the regions now and is trying to delete anything that is *not* fedora02:44
ianwi'm going to stop it, emergency it and think about it02:44
ianwthis works for the arm builder, because it's the only thing with that region02:46
*** jamesmcarthur has quit IRC02:47
*** ociuhandu has quit IRC02:48
*** diablo_rojo has joined #openstack-infra02:50
ianwit looks like buster (container os) dropped tls1.1 .. so either rax isn't supporting that or something else is wrong02:54
*** stevebaker has joined #openstack-infra03:06
*** psachin has quit IRC03:30
*** diablo_rojo has quit IRC03:35
clarkboh wow that was unexpected03:38
clarkbdid it succeed?03:38
clarkb(we'll eventually return to normal if it did but wondering if we need to figure out a workaround to speed things up03:38
clarkbianw: I think the correct way to do what you want with nodepools expectations is to pause the other images but list them all03:42
*** dave-mccowan has quit IRC03:42
clarkbthen it will know about them and not delete them but also not buuld them03:42
*** jamesmcarthur has joined #openstack-infra03:46
*** tetsuro_ has quit IRC03:54
*** stevebaker has quit IRC03:57
*** tetsuro has joined #openstack-infra04:00
*** apetrich has quit IRC04:00
*** jamesmcarthur has quit IRC04:00
ianwyeah, that might work.  i'm running out of time to get it all merged today, i'll write up some stories04:02
ianwclarkb: btw the next thing to debug is the large amount of failures @ https://review.opendev.org/#/c/712819/ ... ensure-tox i think is the place to fixup for the plain nodes04:26
*** ijw has quit IRC04:33
*** ykarel|away is now known as ykarel04:34
*** udesale has joined #openstack-infra04:40
*** Martouf has quit IRC04:41
*** ijw has joined #openstack-infra04:49
*** ijw has quit IRC04:54
*** ricolin has quit IRC05:03
*** ricolin has joined #openstack-infra05:17
*** bnemec has quit IRC05:19
*** ijw has joined #openstack-infra05:20
*** ijw has quit IRC05:25
openstackgerritMerged openstack/diskimage-builder master: Remove hacking from requirements  https://review.opendev.org/71277805:27
openstackgerritIan Wienand proposed openstack/project-config master: Revert "Move fedora-30 builds to nb01.opendev.org"  https://review.opendev.org/71283605:28
*** bnemec has joined #openstack-infra05:28
*** evrardjp has quit IRC05:35
*** evrardjp has joined #openstack-infra05:36
ianwmordred: 2.34.1 dib released with hacking removed05:39
*** ijw has joined #openstack-infra05:52
*** ijw has quit IRC05:56
*** ijw has joined #openstack-infra06:24
*** ijw has quit IRC06:29
*** lmiccini has quit IRC06:33
openstackgerritMerged openstack/project-config master: Revert "Move fedora-30 builds to nb01.opendev.org"  https://review.opendev.org/71283606:33
*** lmiccini has joined #openstack-infra06:33
*** dpawlik has quit IRC06:37
*** ahosam has joined #openstack-infra06:51
*** dpawlik has joined #openstack-infra06:55
*** ijw has joined #openstack-infra06:55
*** ijw has quit IRC07:00
*** roman_g has quit IRC07:01
*** matt_kosut has joined #openstack-infra07:01
*** matt_kosut has quit IRC07:06
*** dpawlik has quit IRC07:10
*** dpawlik has joined #openstack-infra07:16
*** rpittau|afk is now known as rpittau07:24
*** pgaxatte has joined #openstack-infra07:26
*** ijw has joined #openstack-infra07:27
*** yamamoto has joined #openstack-infra07:31
*** ijw has quit IRC07:31
*** matt_kosut has joined #openstack-infra07:32
*** factor has joined #openstack-infra07:50
*** slaweq has joined #openstack-infra07:56
*** ijw has joined #openstack-infra07:58
*** jcapitao has joined #openstack-infra07:59
*** ociuhandu has joined #openstack-infra08:02
*** ijw has quit IRC08:03
*** ociuhandu has quit IRC08:07
kevinzHi infra, could you help to check why the test initial on linaro us are failed: https://zuul.opendev.org/t/openstack/build/fbea5d9b0a574a6ca5005769eb2c1219/console08:09
kevinzfungi: ^^ pls help to check when you convenient :D08:11
*** tesseract has joined #openstack-infra08:12
*** amoralej|off is now known as amoralej08:17
*** ociuhandu has joined #openstack-infra08:20
*** ociuhandu has quit IRC08:24
*** rcernin has quit IRC08:27
*** hashar has joined #openstack-infra08:31
*** tkajinam has quit IRC08:31
*** ociuhandu has joined #openstack-infra08:40
*** tosky has joined #openstack-infra08:46
*** ociuhandu has quit IRC08:49
*** ociuhandu has joined #openstack-infra08:50
*** xek has joined #openstack-infra08:50
*** pkopec has joined #openstack-infra08:51
*** jpena|off is now known as jpena08:53
*** ralonsoh has joined #openstack-infra08:59
*** ijw has joined #openstack-infra09:01
*** ociuhandu has quit IRC09:02
*** ijw has quit IRC09:06
*** ykarel is now known as ykarel|lunch09:07
*** ScottMC has quit IRC09:07
*** ociuhandu has joined #openstack-infra09:14
*** ociuhandu has quit IRC09:15
*** ociuhandu has joined #openstack-infra09:16
*** ociuhandu has quit IRC09:25
*** ociuhandu has joined #openstack-infra09:26
*** derekh has joined #openstack-infra09:33
*** ijw has joined #openstack-infra09:33
*** ijw has quit IRC09:37
*** gfidente has joined #openstack-infra09:44
*** yamamoto has quit IRC09:49
*** apetrich has joined #openstack-infra09:50
*** ociuhandu has quit IRC09:53
*** ykarel|lunch is now known as ykarel09:56
*** ijw has joined #openstack-infra10:04
*** sgw has quit IRC10:04
*** yamamoto has joined #openstack-infra10:05
*** ijw has quit IRC10:08
*** ociuhandu has joined #openstack-infra10:10
*** lpetrut has joined #openstack-infra10:12
*** ociuhandu has quit IRC10:12
*** ociuhandu has joined #openstack-infra10:12
*** tetsuro has quit IRC10:19
*** ociuhandu has quit IRC10:22
*** ociuhandu has joined #openstack-infra10:24
*** nightmare_unreal has joined #openstack-infra10:28
*** ociuhandu has quit IRC10:28
*** ijw has joined #openstack-infra10:35
*** ahosam has quit IRC10:38
*** ahosam has joined #openstack-infra10:39
*** ijw has quit IRC10:40
*** yamamoto has quit IRC10:40
*** ahosam has quit IRC10:51
*** ociuhandu has joined #openstack-infra11:00
*** ijw has joined #openstack-infra11:07
*** ScottMC has joined #openstack-infra11:07
*** udesale_ has joined #openstack-infra11:09
*** ijw has quit IRC11:11
*** udesale has quit IRC11:12
donnydianw: I am hoping to get some help with DIB - I am trying to figure out why my scripts don't actually run11:21
donnydDIB loads the enviroment.d files - but not the install and post-install file11:22
*** nicolasbock has joined #openstack-infra11:25
*** rpittau is now known as rpittau|bbl11:28
*** ijw has joined #openstack-infra11:38
*** jcapitao is now known as jcapitao_lunch11:38
*** ijw has quit IRC11:42
*** ociuhandu has quit IRC11:48
*** nicolasbock has quit IRC12:07
*** nicolasbock has joined #openstack-infra12:07
*** zxiiro has joined #openstack-infra12:08
*** ociuhandu has joined #openstack-infra12:09
*** ijw has joined #openstack-infra12:09
*** rlandy has joined #openstack-infra12:11
*** panda has joined #openstack-infra12:13
*** ijw has quit IRC12:14
*** udesale_ is now known as udesale12:16
*** smarcet has joined #openstack-infra12:16
nightmare_unrealhow can i specify on which cell to launch an instance12:18
AJaegernightmare_unreal: I'm missing context on your question. could you elaborate a bit more, please? is this a usage of OpenStack question? Then better ask on #openstack. This is the channel for running the Infrastructure that OpenStack is developed on.12:20
nightmare_unrealoh okay12:22
nightmare_unreali was trying to add new cell to my existing devstack from another devstack. like attaach cell to devstack1 from devstack212:23
nightmare_unrealthus this question maybe not right channel12:23
AJaegernightmare_unreal: for devstack, better ask on #openstack-qa. In general, developer discussion is on #openstack-dev.12:25
nightmare_unrealthanks12:26
openstackgerritMerged zuul/zuul master: Be explicit about source of base images  https://review.opendev.org/71277212:26
*** jpena is now known as jpena|lunch12:30
*** yamamoto has joined #openstack-infra12:35
openstackgerritMerged zuul/zuul master: Remove stretch-backports from docker build  https://review.opendev.org/71273712:35
*** ykarel is now known as ykarel|afk12:44
*** rh-jelabarre has quit IRC12:44
*** rh-jelabarre has joined #openstack-infra12:44
openstackgerritTobias Henkel proposed zuul/zuul master: Defer setting build pause to event queue  https://review.opendev.org/71293912:49
*** smarcet has quit IRC12:50
*** smarcet has joined #openstack-infra12:52
*** Lucas_Gray has joined #openstack-infra12:53
*** Lucas_Gray has quit IRC12:55
*** Lucas_Gray has joined #openstack-infra12:59
*** yamamoto has quit IRC13:04
*** rpittau|bbl is now known as rpittau13:04
*** jcapitao_lunch is now known as jcapitao13:05
*** smarcet has quit IRC13:09
*** Lucas_Gray has quit IRC13:09
*** Lucas_Gray has joined #openstack-infra13:11
*** yamamoto has joined #openstack-infra13:14
*** smarcet has joined #openstack-infra13:19
*** Lucas_Gray has quit IRC13:23
*** yamamoto has quit IRC13:23
*** pgaxatte has quit IRC13:25
*** pgaxatte has joined #openstack-infra13:26
*** Lucas_Gray has joined #openstack-infra13:27
*** lbragstad__ has quit IRC13:29
*** cdearborn has joined #openstack-infra13:31
*** lbragstad has joined #openstack-infra13:31
*** Goneri has joined #openstack-infra13:33
*** jpena|lunch is now known as jpena13:35
*** rcernin has joined #openstack-infra13:41
*** yamamoto has joined #openstack-infra13:41
*** ijw has joined #openstack-infra13:44
*** ijw has quit IRC13:48
*** smarcet has quit IRC13:52
fungiianw: clarkb: still catching up on scrollback, but you can enable tls 1.1 by tweaking MinProtocol in the system_default_sect section of /etc/ssl/openssl.cnf, it's just set to 1.2 by default for improved safety13:54
fungii've had to lower it in recent past to get python 3.5 to build13:55
*** ykarel|afk is now known as ykarel13:55
*** rh-jlabarre has joined #openstack-infra13:55
*** amoralej is now known as amoralej|lunch13:55
*** rh-jelabarre has quit IRC13:56
*** ociuhandu has quit IRC13:59
*** factor has quit IRC14:00
*** factor has joined #openstack-infra14:00
lbragstadmnaser frickler o/ i'm curious if either of you would like to give https://review.opendev.org/#/c/711114/4 a look whenever you have a minute?14:01
*** factor has quit IRC14:03
*** factor has joined #openstack-infra14:04
clarkbfungi: want to propose that update to https://opendev.org/zuul/nodepool/src/branch/master/Dockerfile#L5314:04
zbrlast night some changes made to constraints broke most of py27 users,  this is how i discovered that we no longer have a py27 job on requirements14:04
clarkbfungi: you can add it as another RUN directive under that heading14:04
zbrbut we do have lots of conditions related to py27 in them.14:04
openstackgerritJan Kubovy proposed zuul/zuul master: WIP: Separate connection registries in tests  https://review.opendev.org/71295814:06
AJaegerconfig-core, a few open reviews, please consider them: https://review.opendev.org/711114 https://review.opendev.org/711878 https://review.opendev.org/712586 https://review.opendev.org/712048 https://review.opendev.org/#/c/708642/14:06
*** ociuhandu has joined #openstack-infra14:07
clarkbAJaeger: noted, will try to get at those when at a computer14:07
AJaegerthanks, clarkb14:07
*** ociuhandu has quit IRC14:07
*** ociuhandu has joined #openstack-infra14:07
clarkbzbr: I guess that is related to removing python2.7 support14:07
clarkbzbr there will be growing pains14:07
zbrclarkb: i think that constraints are the last place to frop py27 from.14:08
zbralso makes no sense to drop the job and keep the conditions in.14:09
zbris like "disabling the testing"14:09
zbrclarkb: example of bug I had to fix due to this https://review.opendev.org/#/c/712942/1/upper-constraints.txt14:09
clarkbzbr I agree but iirc others were trying to use that as the carrot/stick to move people along14:09
*** bnemec is now known as beekneemech14:10
*** rh-jlabarre has quit IRC14:10
zbri asked on reqs channel but got no answer.14:10
clarkbI've suggested similar splits to that and been rehected14:10
clarkb*rejected14:10
*** rh-jelabarre has joined #openstack-infra14:11
fungizbr: stable branches of requirements no longer run py27 jobs?14:11
fungior you mean master branch changes of requirements broke projects trying to use it with python 2.7?14:12
zbrwe have functional testing jobs that are using reqs, because they use tox.14:12
zbrfungi: yep14:12
fungiftr, we said all official openstack projects are free to drop python 2.7 support in master between ussuri milestones 1 and 214:13
*** factor has quit IRC14:14
*** factor has joined #openstack-infra14:14
fungior phrased another way, projects which want to keep using python 2.7 after milestone 1 were on their own to locally reimplement support for it because they should not rely on other projects continuing to do so14:14
zbri doubt anyone can explain dropping the testing of upper-constrains for py27 while the file looks like https://opendev.org/openstack/requirements/src/branch/master/upper-constraints.txt14:15
zbr~30% of is full of conditions related to py2714:15
zbrand testing it with py27 is not really an issue preventing openstack from moving forward.14:15
*** factor has quit IRC14:15
fungiwell, py27 testing had to be dropped from it before it could remove those entries14:16
fungiotherwise the change to clean up the file won't be able to merge14:16
zbri would argue that this should be done, but even if done, it could have being proposed in a single change14:17
fungiis your concern that they haven't finished cleaning that file up yet, or that you really didn't want openstack to stop supporting python 2.7 in ussuri?14:17
zbrcurrent status allows introduction of breakages14:17
zbri do want to keep the file py27 compatible.14:18
zbrwe are still working to port jobs, but that recent change broke most of our functional testing (ansible molecule)14:19
zbrobviously that I could disable use of constraints for this specific tox env, but there are tens of repos, not to mentions that doing this may introduce other issues.14:20
*** ociuhandu has quit IRC14:23
clarkbI'm not sure why ansible-molecule is using openstack constraints? but ya the assertion is no more new python2.7 fixes/support14:24
clarkbthis is what I was told alittle while back when I suggested that as a fix until the second milestone drop (and now we are past that right so even less likely to go in now)14:25
zbrclarkb: molecule is testing ansible roles, these roles may have dependencies specific to openstack, so it makes sense to use the same constraints, or you risk testing them with deps that are different than what is used in production.14:26
clarkbthen I guess you should drop python2.7 testing there14:27
fungiyeah, i mean, openstack has agreed since years that this would be the cycle where they officially drop support for python 2.714:28
*** ociuhandu has joined #openstack-infra14:29
fungiand the plan passed around more than a cycle ago was that projects which want to individually retain python 2.7 support should prepare for their dependencies to likely stop supporting python 2.7 after milestone 114:29
*** Lucas_Gray has quit IRC14:29
zbrfungi:  in that case, can you explain me why the py27 job was removed without removing the conditions at the same time?14:31
fungitesting ansible roles on python 2.7 with "the same constraints" as openstack ussuri makes no sense if openstack ussuri doesn't support python 2.714:31
fungizbr: yes, because dropping the jobs means less stuff to arbitrarily break requirements changes, while dropping the python 2.8 lines from files in the requirements repository is mere cleanup14:32
fungier, the 2.7 lines14:32
*** Lucas_Gray has joined #openstack-infra14:33
*** yamamoto has quit IRC14:33
zbri wonder if we can put something inside that file to force it to break on any py27, otherwise the entire file is source of misinformation14:33
fungimost likely the job was removed because something else it relied on dropped python 2.7 support and it stopped working, but i'm not really following the goings-on in the requirements project, you'd have to ask one of them (not sure why this is even on topic for #openstack-infra to be honest)14:33
zbrthat is correct, sadly #openstack-requirements seens to be in quarantine, not much activity around.14:34
*** mhu has joined #openstack-infra14:34
openstackgerritMerged opendev/glean master: Fix a handful of bugs in config-drive processing  https://review.opendev.org/70362314:42
*** Liang__ has quit IRC14:43
fungizbr: you might try the mailing list with [requirements] in the subject, in that case14:45
*** ociuhandu has quit IRC14:49
*** ijw has joined #openstack-infra14:51
fungidigging through master branch history in requirements and project-config, i don't see that they took any direct steps to drop python 2.7 testing recently. they had to stop running 2.7 cross-testing jobs against other projects which had stopped supporting 2.7 but that happened a while back14:52
fungithey're still running the cross-swift-py27 job on requirements changes because swift is still supporting python 2.714:54
fungipresumably they can't remove at least some of the python_version=='2.7' lines right now because of that14:55
*** amoralej|lunch is now known as amoralej14:55
*** ijw has quit IRC14:55
fungirequirements-tox-py27-check-uc is configured to run in check and gate but is being skipped for some reason not immediately apparent, looking to see if i can figure out why15:02
fungiyeah, http://zuul.opendev.org/t/openstack/job/requirements-tox-py27-check-uc isn't run for master, just stable branches through train15:03
*** nightmare_unreal has quit IRC15:05
fungithe last change it gated on master was https://review.opendev.org/70227715:06
fungizbr: yeah, so the requirements-tox-py27-check-uc job was removed in https://review.opendev.org/702478 (as suspected it was in response to a library project dropping support for 2.7)15:10
fungii thought it was still in the configuration because `git remote update` was not working in my local clone for some reason (complaining about origin/master being an ambiguous refname)15:12
*** pkopec has quit IRC15:13
*** yamamoto has joined #openstack-infra15:15
*** ociuhandu has joined #openstack-infra15:19
*** eharney has quit IRC15:19
*** ociuhandu has quit IRC15:20
*** ociuhandu has joined #openstack-infra15:20
*** ijw has joined #openstack-infra15:22
*** gyee has joined #openstack-infra15:23
*** ijw has quit IRC15:27
*** ociuhandu_ has joined #openstack-infra15:28
*** yamamoto has quit IRC15:28
*** jamesmcarthur has joined #openstack-infra15:29
*** ociuhandu has quit IRC15:32
*** ccamacho has quit IRC15:32
*** ociuhandu_ has quit IRC15:33
*** ijw has joined #openstack-infra15:44
openstackgerritMerged openstack/project-config master: Add a new project and repository for tripleo-ipa  https://review.opendev.org/71111415:44
*** rh-jelabarre has quit IRC15:45
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: DNM: rebase unittests to base-minimal-test  https://review.opendev.org/71298515:45
clarkblooking at the ensure-tox errors that ianw has generated on our new test "plain" node type the issue is that we don't have a pip installed at all https://zuul.opendev.org/t/openstack/build/7ed04083ad92443fb1e95445033d62d0/console#1/0/0/bionic15:48
*** ijw has quit IRC15:48
*** ykarel is now known as ykarel|away15:48
*** lpetrut has quit IRC15:48
clarkbmy initial thoughts on that are maybe we should add an ensure-pip role, it can even use our cached get-pip.py script to install a pip15:48
clarkbor use distro packaging15:49
*** ijw has joined #openstack-infra15:49
openstackgerritMerged openstack/project-config master: New repo: devstack-plugin-open-cas  https://review.opendev.org/71187815:51
clarkbthat probably deserves a bit more thought before diving into implementation. mordred ^ that probably interests you as well15:51
fungitox --version || ( python3 -m venv /usr/env/tox && /usr/env/tox/bin/pip install tox && ln -s /usr/env/tox/bin/tox /usr/local/bin/ )15:52
mordredclarkb: ugh15:52
clarkbfungi: oh ya we could bootstrap it from python3 -mvenv15:52
openstackgerritMerged openstack/project-config master: Add OpenInfra Labs IRC channels to bots  https://review.opendev.org/71258615:52
openstackgerritMerged openstack/project-config master: Remove neutron-tempest-dvr job from Neutron's dashboard  https://review.opendev.org/71204815:52
fungithe python3-venv package embeds a copy of pip for its own internal use15:52
clarkbactually this is all the case where we could preinstall tox on our test nodes in a venv15:53
fungiso that it can provide it in any venv you create15:53
clarkbthen set the tox path as an ansible var15:53
mordredwe ca15:53
mordredcan15:53
clarkbfungi: for tox specifically I think doing what you describe on the image builds may be the best option for now15:53
clarkbthen update the job vars15:53
mordredbut why don't we update ensure-tox to do what fungi said so that it's more resilient15:53
clarkbmordred: ya we can do that too15:53
mordredyeah - as an "also"15:53
fungiftr, that's basically how i install tox on my systems15:53
fungiexcept i symlink it in ~bin/ instead of /usr/local/bin/15:54
fungier, in ~/bin/ instead of /usr/local/bin/15:54
*** smarcet has joined #openstack-infra15:58
openstackgerritClark Boylan proposed zuul/nodepool master: Switch min TLS version to 1.1 in the docker images  https://review.opendev.org/71299715:59
fungid'oh, i was halfway through editing a change for that. at least i know what to review now ;)15:59
clarkbfungi: oh you never responded and I started digging into ianws backlog from last night. Sorry16:00
*** lajoskatona has joined #openstack-infra16:00
fungino worries, and yeah i was juggling a bunch of stuff anyway16:00
*** jamesmcarthur has quit IRC16:00
lajoskatonaHi, could you please help me with https://review.opendev.org/712777 ?16:01
lajoskatonaIt seems that the setuptools issue breaks py37/py38 jobs in some networking projects, and it seems to be persistent16:01
*** pgaxatte has quit IRC16:02
*** jamesmcarthur has joined #openstack-infra16:03
*** yamamoto has joined #openstack-infra16:04
lajoskatonaone example could be: https://daebb073eb40170317c6-5a97607e4573b334b582a5ecc4c19d48.ssl.cf2.rackcdn.com/712855/2/check/openstack-tox-py37/e3dadb5/job-output.txt16:06
openstackgerritClark Boylan proposed zuul/nodepool master: Add visual dividers for each image in Dockerfile  https://review.opendev.org/71300216:07
*** yamamoto has quit IRC16:07
clarkblajoskatona: this is a known problem. Basically setuptools deprecated a feature (called Feature I think for extra confusion) a while back then finally removed it and pushed a release with that removal16:09
clarkblajoskatona: this means that python packages using that feature now fail to build using a current setuptools. Unfortunately, you can' really control the version of setuptools in the wild effectively, which means we can't just pin setuptools16:10
clarkblajoskatona: ideally that means we fix occurences of this in the broken python packages16:10
clarkbI'm not sure if the horizon or requirements teams have made any progress on figuring out a fix going forward though16:10
lajoskatonaclarkb: Hi, so this means we have to add some similar caping to all packages that has the issue?16:12
clarkblajoskatona: capping probably won't fix this bceause older versions of pyScss are likely using the same removed feature16:13
mordredyeah- we need to work with upstreams to fix themselves - because they are, in fact, currently broken16:13
lajoskatonaclarkb: ahhha,ok, so let's wait and try to find horizon & requirements team on let's say Monday...16:13
*** lpetrut has joined #openstack-infra16:14
clarkbmy hunch is that pyScss is no longer maintained and we'll either need to start maintaining it, fork it, or use a different tool16:14
mordreduse a different tool16:15
mordredwebpack does a great job for javascript16:15
mordredbut - you know - whatever16:16
lajoskatonasound good16:16
lajoskatonasounds -^16:16
lajoskatonaAnyway, thanks, on Monday then I check with amotoki, as he is horizon PTL, and perhaps with fresh mind it will be easier to find an optimal solution16:17
clarkbmordred: fungi: there is one gotcha with the python3 -m venv idea that just occured to me. On centos7 currently tox is installed under python2 and will default to python2. If we switch to python3 -m venv && venv/bin/pip install tox that will switch to a python3 default tox16:21
clarkbits a relatively minor thing, but I half expect it to trip tripleo up (especially given zbr's constraints concerns)16:21
clarkbThis may be a case of "sorry" and move forward via update of tox.ini files though16:21
fungidetect centos-7 and install old python-tox rpm?16:21
*** rh-jelabarre has joined #openstack-infra16:21
clarkbfungi: the problem there is that tox is ancient and likely to break on the tox.ini files in projects (since they use newer features)16:22
fungiafter all, it's only a matter of time before the tox maintainers decide to stop supporting python 2.7 in new releases anyway16:22
zbri think i can take this risk16:22
clarkbzbr: which risk, python3 default or old tox?16:22
zbri prefer new tox, the old one from rpm is pretty much useless afaik16:23
zbris 1.4.3 or somthing like thi16:23
fungibut keep in mind, since we're no longer preinstalling tox in the images, and doing it at job runtime instead, jobs can be adjusted to bring their own tox16:23
zbrwhich is a good thing16:23
fungiso if tripelo needs to adjust their jobs to install tox via python 2.7 by default, then they can do so16:24
clarkbfungi: yup, I think that is the messaging we can use "We'll preinstall tox in a virtualenv under python3. This will be a current tox. If you need something else you are free to install it however you like"16:24
clarkbok I'll get a change going for the preinstall on our images bit16:24
zbrin fact instead of forcing the version, I would prefer to manually specify python version when creating venv for tools that need specific one.16:25
clarkbthen take a look at ensure tox though thats gone through so many recent changes we may need to have a longer conversation with the zuul maintainers about that16:25
clarkbzbr: right we are basically getting away from forcing anything by removing globally installed tools. We will still preinstall some things that you can use if they work for you but otherwise bootstrap what you need specifically16:25
*** jcapitao is now known as jcapitao_afk16:25
amotokipyscss is not maintained over 4 years. e0ne and I explored a way to avoid the current failure and it looks like forking pyscss would be easiest.16:25
mordredclarkb, fungi: how about detect centos-7 and no-tox, yum install python-virtualenv and install the tox into that16:26
mordredamotoki: nod16:26
zbrclarkb: i like the move. we can have one generic "venv" where we can install all our tools that we need, bindep, tox, ....16:26
amotokie0ne forked the original pyscss and django-pyscss into his repos16:26
fungifor tox, could we just add the env for it in a role in the tox parent job and not bother even putting a convenience copy on the images?16:26
amotokiand proposed https://review.opendev.org/#/c/712976/ and https://review.opendev.org/#/c/712978/16:26
mordredamotoki: I would LOVE to say "I'll help you convert to webpack" ... but I'm about 10 tasks behind on my already too long list- so I support you forking if that's the best bet16:27
zbrthe old story about using --user, proved to be a permanent source of pain. i am no longer using it anywhere.16:27
fungizbr: putting all tools in a common env is problematic as well (especially if they have conflicting requirements). better to just put each in a separate envc16:27
fungis/envc/env/16:27
mordredamotoki: https://review.opendev.org/#/c/712976/ is +2 from me once it's updated16:27
mordredfungi: we could - but I think we should put a convenience copy in the images16:28
mordredfungi: because otherwise 75% of the jobs in our system are going to start with "install tox"16:28
openstackgerritClark Boylan proposed openstack/project-config master: Install tox into a virtualenv on our images  https://review.opendev.org/71301716:29
zbrfungi: tricky because you cannot use them at the same time, thus is why i prefer one venv. if somoene wants isolated ones, they should have an option to define a custom venv for that.16:29
mordredwhich just seems wasteful16:29
clarkbmordred: I'd like to avoid having diverging paths as much as possible (just makes debugging difficult)16:29
zbryep, a pre-loaded venv with bindep, tox, linters? would be highly appreciated.16:29
mordredzbr: I disagree. there's no need to put bindep and tox into the same venv16:29
fungimordred: yeah, i guess it's a question of where we draw the line on what we should cache convenience copies of on our images16:30
clarkbmordred: I think what we (opendev) can do is something similar to my change above. Then jobs can use that by default, but if they need something different then they can easily install what you suggest or install from the distro package or whatever and we haven't polluted things making that difficult16:30
mordredclarkb: I agree - but python2 on centos7 is a vanishingly small use case - this is a backwards compat nightmare no matter what16:30
clarkbmordred: right tahts why I want us to punt on it :)16:30
mordredclarkb: ya - totally16:30
mordredkk. I'm on board with that16:30
clarkbmordred: and if anyone has that corner case we've not prevented them from addressing it16:30
fungibecause anything which goes wrong with the convenience copy means a round-trip through the image update process and isn't a self-testable change16:30
clarkb(the current situation does sort of prevent them in some cases and we are getting away from that)16:30
clarkbfungi: yes, but pypi flakyness has similar job relaibility issues as we do more and more runtime installs16:31
mordredfungi: I agree about it being a line question16:31
*** openstackgerrit has quit IRC16:31
mordredbut I think things where we are not expecting people to have a common case of caring about the version - pip and tox fall in to this category16:32
mordredbasically - things where people need it to be there but it's an entry point into the system that lets them express teh project-specific needs16:32
*** openstackgerrit has joined #openstack-infra16:32
openstackgerritSean McGinnis proposed openstack/reviewstats master: Use unittest.mock instead of third party mock  https://review.opendev.org/71301916:32
clarkbmordred: ++16:33
mordredfor openstack, _really_ the most imporant thing is tox because it's the most standardized entrypoint16:33
clarkbthey are bootstrap tools16:33
mordredyup16:33
mordredeverything else can be safely expressed in repo16:33
mordredbut a lack of tox makes everything a nightmare16:33
*** lajoskatona has left #openstack-infra16:33
*** lajoskatona has quit IRC16:33
clarkbin any case my proposed change doesn't affect the global installs of tox so is safe to land as is I believe16:33
clarkbthen we can continue to iterate with the test plain image to see if we like that setup or not16:33
openstackgerritLance Bragstad proposed openstack/project-config master: Add queues for tripleo-ipa project  https://review.opendev.org/71111516:34
clarkb(we don't have to consider this final, but I think being able to actually run it with jobs may be useful)16:34
clarkblbragstad: I think you can make those changes in repo16:34
* lbragstad nods16:35
lbragstadsounds like i should abandon that patch then16:35
clarkblbragstad: there are some exceptions like setting up release jobs that centralized config is useful for, but I don't think the queue names need that16:35
*** hashar has quit IRC16:35
clarkb(zuul just grabs the first queue name it gets from configs iirc so as long as you don't diverge it will be fine)16:36
lbragstadclarkb ok - i'll abandon16:36
lbragstadclarkb thanks for the information, i appreciate it16:36
*** sgw has joined #openstack-infra16:37
*** yamamoto has joined #openstack-infra16:42
lbragstadclarkb picking up here - https://docs.openstack.org/infra/manual/creators.html#update-the-gerrit-group-members - outside of asking here, is there anything you need from me process wise for that step?16:49
clarkblbragstad: it helps to have a link to the original creating change (just for context), but I can find that myself quickly enough16:50
lbragstadclarkb https://review.opendev.org/#/c/711114/16:50
lbragstadspecifically - https://review.opendev.org/#/c/711114/4/gerrit/acls/x/tripleo-ipa.config16:50
lbragstadmy gerrit account id is 504616:51
clarkblbragstad: done16:51
lbragstadclarkb thank you, sir16:52
clarkbyou are welcome16:52
fungii usually just copy/paste from the committer e-mail address on the introducing change and that should unambiguously resolve to the correct gerrit account16:52
fungi(and when it doesn't, that's a signal that we need to clean up a duplicate account in the db)16:52
clarkbfungi: ya, the two reasons for pulling up the original chagne are to get the user details and to make sure there aren't multiple groups that need editing16:52
fungiright, i also use the acl from the change to get the list of groups it may have added16:53
*** jcapitao_afk is now known as jcapitao16:53
fungi(though if the group already has members, i take that as a signal that it's a preexisting group i shouldn't manually add anyone to)16:53
fungieventually we could probably automate the process following that set of steps by default when there's no namespace-specific override16:54
* lbragstad nods16:54
fungithough first we really need to namespace our gerrit groups to make them more collision resistant16:56
*** rpittau is now known as rpittau|afk16:58
*** ahosam has joined #openstack-infra17:00
openstackgerritJames E. Blair proposed zuul/zuul master: Add TLS support for ZooKeeper  https://review.opendev.org/71253117:00
openstackgerritJames E. Blair proposed zuul/zuul master: Use ZK TLS in quickstart  https://review.opendev.org/71281717:00
*** ahosam has quit IRC17:01
*** lmiccini has quit IRC17:02
*** hashar has joined #openstack-infra17:05
*** tesseract has quit IRC17:07
*** Evg_Inb has joined #openstack-infra17:09
mordredfungi: that woudl be a fun migration17:09
Evg_Inbcan anyone say if there is a manual trigger to re-generate libvirt.xml files for all instances running on a hypervisor? "/var/lib/nova/instances/<ID>/libvirt.xml"17:10
fungiEvg_Inb: i think you're looking for a channel where openstack software developers and operators hang out. maybe try #openstack or #openstack-dev. here in the -infra channel we're the folks running the community's collaboration systems and services17:11
Evg_Inbgot it, thanks !17:12
clarkbmordred: https://review.opendev.org/#/c/713017/ is ready for review (tox venv change) if you have a moment17:16
clarkbI can retrigger builds of the plain images once that lands17:16
*** lbragstad has quit IRC17:23
*** lbragstad has joined #openstack-infra17:26
*** diablo_rojo has joined #openstack-infra17:27
*** dpawlik has quit IRC17:30
*** iurygregory has quit IRC17:32
openstackgerritMerged openstack/project-config master: Install tox into a virtualenv on our images  https://review.opendev.org/71301717:32
*** evrardjp has quit IRC17:35
*** evrardjp has joined #openstack-infra17:36
*** sshnaidm|afk has quit IRC17:37
clarkbthanks! ^17:37
*** hamzy has joined #openstack-infra17:45
*** Evg_Inb has quit IRC17:46
openstackgerritJames E. Blair proposed zuul/zuul-operator master: Use explicit provides/requires for container jobs  https://review.opendev.org/71281617:49
openstackgerritJames E. Blair proposed zuul/zuul-operator master: Update to dhall lang v14  https://review.opendev.org/71064917:49
openstackgerritJames E. Blair proposed zuul/zuul-operator master: Add TLS configuration to ZooKeeper service  https://review.opendev.org/71275917:49
openstackgerritJames E. Blair proposed zuul/zuul-operator master: Use explicit provides/requires for container jobs  https://review.opendev.org/71281617:51
openstackgerritJames E. Blair proposed zuul/zuul-operator master: Update to dhall lang v14  https://review.opendev.org/71064917:51
openstackgerritJames E. Blair proposed zuul/zuul-operator master: Add TLS configuration to ZooKeeper service  https://review.opendev.org/71275917:51
*** udesale_ has joined #openstack-infra17:52
*** eharney has joined #openstack-infra17:52
*** jamesmcarthur has quit IRC17:53
*** udesale has quit IRC17:54
*** jamesmcarthur has joined #openstack-infra17:57
*** derekh has quit IRC18:01
*** jamesmcarthur has quit IRC18:05
*** jamesmcarthur has joined #openstack-infra18:06
*** jpena is now known as jpena|off18:07
*** jcapitao has quit IRC18:12
*** hashar has quit IRC18:14
*** chandankumar is now known as raukadah18:14
*** smarcet has quit IRC18:16
*** jamesmcarthur has quit IRC18:22
*** jamesmcarthur has joined #openstack-infra18:22
*** smarcet has joined #openstack-infra18:27
*** gfidente has quit IRC18:32
*** yamamoto has quit IRC18:33
openstackgerritJames E. Blair proposed zuul/nodepool master: WIP test functional-openshift on centos8  https://review.opendev.org/71304618:33
*** ralonsoh has quit IRC18:36
*** lbragstad_ has joined #openstack-infra18:37
melwittclarkb: do you happen to know if OVH restricts both read and write disk iops or only one of them?18:38
*** udesale_ has quit IRC18:39
clarkbI think its a combined limit18:39
fungiso the harder you read, the less you can write and vice versa18:39
melwittI'm chatting with zzzeek in -nova about how we could potentially deal with https://bugs.launchpad.net/nova/+bug/184492918:39
openstackLaunchpad bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed]18:39
*** lbragstad has quit IRC18:39
openstackgerritMonty Taylor proposed openstack/project-config master: Add fedora-30 to nb01.opendev.org  https://review.opendev.org/71304718:40
clarkbmelwitt: amorin would be the best resource on that though (as we don't know specifics just observations on behavior)18:40
melwittthanks18:46
clarkbfungi: and ya that18:47
fungibut it's friday night for him i suspect18:48
openstackgerritMonty Taylor proposed opendev/system-config master: Add a nodepool helper script into /usr/local  https://review.opendev.org/71305018:52
*** smarcet has quit IRC18:53
melwittyeah, realizing though that we have seen the fail on other node providers too, just much more rarely18:53
melwittthanks18:54
* melwitt bbl18:55
openstackgerritMonty Taylor proposed opendev/system-config master: Add a nodepool helper script into /usr/local  https://review.opendev.org/71305018:56
openstackgerritMonty Taylor proposed opendev/system-config master: Add a nodepool helper script into /usr/local  https://review.opendev.org/71305018:57
*** smarcet has joined #openstack-infra19:02
*** jbryce is now known as airship-irc-bot19:03
*** Lucas_Gray has quit IRC19:05
*** lpetrut has quit IRC19:05
*** amoralej is now known as amoralej|off19:06
*** airship-irc-bot is now known as jbryce19:07
*** smarcet has quit IRC19:10
*** smarcet has joined #openstack-infra19:11
*** smarcet has quit IRC19:13
*** smarcet has joined #openstack-infra19:15
*** smarcet has quit IRC19:15
openstackgerritAndreas Jaeger proposed zuul/zuul-jobs master: Use a zuul_* and add an .ansible-lint file  https://review.opendev.org/71254719:27
AJaegermordred: I updated your change, this passes locally now ^19:27
*** ociuhandu has joined #openstack-infra19:32
mordredAJaeger: oh whoops - sorry about that19:33
mordredand thanks19:33
AJaegeryou're welcome!19:36
AJaegermordred: oops, still saw an error locally that puzzles me; I'm not going to fix it today, so if it fails in Zuul, back to you ;)19:38
openstackgerritMerged openstack/project-config master: Add fedora-30 to nb01.opendev.org  https://review.opendev.org/71304719:47
*** jamesmcarthur has quit IRC19:50
*** jamesmcarthur has joined #openstack-infra19:51
*** jamesmcarthur has quit IRC19:57
*** jamesmcarthur has joined #openstack-infra19:58
openstackgerritDavid Shrewsbury proposed zuul/nodepool master: Stop comparing hostnames to determine ownership  https://review.opendev.org/71305719:59
openstackgerritJames E. Blair proposed openstack/project-config master: Revert "Add fedora-30 to nb01.opendev.org"  https://review.opendev.org/71305819:59
openstackgerritMonty Taylor proposed opendev/system-config master: Add a nodepool helper script into /usr/local  https://review.opendev.org/71305020:03
openstackgerritMonty Taylor proposed zuul/nodepool master: Revert "Dockerfile: add support for arbritary uid"  https://review.opendev.org/71306020:10
openstackgerritDavid Shrewsbury proposed zuul/nodepool master: Stop comparing hostnames to determine ownership  https://review.opendev.org/71305720:13
*** diablo_rojo has quit IRC20:19
*** smarcet has joined #openstack-infra20:19
openstackgerritMerged openstack/project-config master: Revert "Add fedora-30 to nb01.opendev.org"  https://review.opendev.org/71305820:21
*** ociuhandu has quit IRC20:23
*** hashar has joined #openstack-infra20:25
*** diablo_rojo has joined #openstack-infra20:34
prometheanfireheh, almost marked the seo mail (openstack-discuss) as spam20:37
mordredprometheanfire: Do you need SEO services????!!!?20:38
*** jamesmcarthur has quit IRC20:40
*** jamesmcarthur has joined #openstack-infra20:41
prometheanfireI obviously don't, since they seem to be able to find me so easily20:43
*** jamesmcarthur has quit IRC20:45
*** jamesmcarthur has joined #openstack-infra20:45
openstackgerritJames E. Blair proposed opendev/system-config master: nodepool-builder: add /opt/dib_cache  https://review.opendev.org/71282420:53
*** ijw has quit IRC20:54
*** ociuhandu has joined #openstack-infra20:55
*** slaweq has quit IRC21:04
*** jamesmcarthur has quit IRC21:05
*** jamesmcarthur has joined #openstack-infra21:06
*** ociuhandu has quit IRC21:06
*** ociuhandu has joined #openstack-infra21:07
*** jamesmcarthur has quit IRC21:12
*** jamesmcarthur has joined #openstack-infra21:13
*** jamesmcarthur has quit IRC21:15
*** ociuhandu has quit IRC21:16
*** jamesmcarthur has joined #openstack-infra21:18
*** rlandy has quit IRC21:21
prometheanfireis etherpad the official way to propose community goals?21:25
*** smarcet has quit IRC21:25
*** jamesmcarthur has quit IRC21:29
amorinhey all, I am just catching back the log here, I saw my name highlighted about a bug in OVH gra1?21:30
*** smarcet has joined #openstack-infra21:31
amorinI wont have time to take a look tonight, but I may tomorrow21:31
amorinI'll let you know21:31
amorincc melwitt clarkb21:31
clarkbamorin: they thing iops limits may be impacted database transactions21:32
clarkbmelwitt would have much more info than me though21:32
amorinack, feel free to give me info21:32
amorinif I remember well we already had iops issue, no?21:33
clarkbamorin: ya I think those were largely resolved, but this may be a corner case issue (it apparently happesn on other clouds too (noisy neighbors maybe?) but more frequently on ovh)21:34
amorinack, I need to go for now, but I'l try to take a look tomorrow21:35
*** jamesmcarthur has joined #openstack-infra21:36
melwittamorin: hey, this is regarding https://bugs.launchpad.net/nova/+bug/1844929 where occasionally a db call does not get back results in the gate. it seems related to load and happens most commonly on OVH and INAP node providers. and I was looking for some ideas on how we could potentially tune mysqld config to help it do better in such env21:41
openstackLaunchpad bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed]21:41
*** stevebaker has joined #openstack-infra21:42
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Add CONTRIBUTE file  https://review.opendev.org/70553521:43
fungiprometheanfire: for openstack? i think proposing a gerrit change to https://opendev.org/openstack/governance/src/branch/master/goals and discussing on the ml is the expected way21:43
prometheanfirethanks for the link, couldn't find it when I searched21:44
fungiprometheanfire: i would probably start with an ml thread to gauge interest anyway21:44
fungiprometheanfire: process is published at https://governance.openstack.org/tc/goals/index.html21:45
prometheanfireI kinda did already with the request to switch from mock to unittests.mock21:45
prometheanfireI didn't see openstack/governance mentioned there :P21:46
fungiseems like there's been some changes flying around already to implement that, so it's a good start21:47
prometheanfireyep21:48
prometheanfirethere was recently a mock change that brock some people, I pointed them at that (nova/neutron iirc)21:48
*** ijw has joined #openstack-infra21:51
*** sgw has quit IRC21:54
*** ijw has quit IRC21:56
openstackgerritMerged zuul/zuul-operator master: Add CONTRIBUTE file  https://review.opendev.org/70553522:03
*** ociuhandu has joined #openstack-infra22:07
*** ociuhandu has quit IRC22:20
*** ijw has joined #openstack-infra22:23
fungibrock by mock22:24
*** jamesmcarthur has quit IRC22:25
*** ijw has quit IRC22:27
*** jamesmcarthur has joined #openstack-infra22:33
*** jamesmcarthur has quit IRC22:42
*** jamesmcarthur has joined #openstack-infra22:43
*** jamesmcarthur has quit IRC22:43
*** ijw has joined #openstack-infra22:49
prometheanfirewat22:49
*** aedc has joined #openstack-infra22:52
*** rcernin has quit IRC22:58
*** rcernin has joined #openstack-infra22:59
*** aedc has quit IRC22:59
*** hashar has quit IRC23:01
openstackgerritIan Wienand proposed zuul/nodepool master: Dockerfile: add lsb-release  https://review.opendev.org/71307923:18
*** zxiiro has quit IRC23:19
*** sgw has joined #openstack-infra23:19
*** gyee has quit IRC23:22
*** xek has quit IRC23:38
openstackgerritMohammed Naser proposed openstack/project-config master: add vexxhost/openstack-operator  https://review.opendev.org/71308023:48
*** lbragstad_ has quit IRC23:49
openstackgerritClark Boylan proposed openstack/project-config master: Statically cache devstack images and packages  https://review.opendev.org/71308123:55

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!