| *** rpioso is now known as rpioso|afk | 00:01 | |
| *** eernst has quit IRC | 00:02 | |
| *** newstem has joined #openstack-infra | 00:04 | |
| Jeffrey4l | tonyb, re EOL openstack/kolla for newton today, sure | 00:06 |
|---|---|---|
| *** liusheng has quit IRC | 00:07 | |
| *** gouthamr has joined #openstack-infra | 00:09 | |
| pabelanger | ianw: centos-7 still looks broken | 00:10 |
| pabelanger | I think because we set DIB_EPEL_DISABLED in nodepool.o.o | 00:10 |
| clarkb | pabelanger: its explicitly enabling it for that one install though right? | 00:11 |
| *** dingyichen has joined #openstack-infra | 00:11 | |
| pabelanger | clarkb: yah, but the issue is: https://git.openstack.org/cgit/openstack/diskimage-builder/tree/diskimage_builder/elements/epel/pre-install.d/05-rpm-epel-release#n36 | 00:12 |
| pabelanger | https://review.openstack.org/#/c/561479/10/diskimage_builder/elements/epel/pkg-map | 00:12 |
| pabelanger | has missed centos | 00:12 |
| pabelanger | centos7 is for the wrong element | 00:12 |
| pabelanger | well, the other centos7 element | 00:13 |
| pabelanger | centos-minimal is centos | 00:13 |
| pabelanger | so, we'll need to fix and tag 2.14.1 | 00:13 |
| clarkb | the distro is still centos7 if using centos-minimal isn't it? | 00:14 |
| ianw | oh i'm constantly confused over this | 00:15 |
| clarkb | if svc map doesn't work that way it probably should | 00:15 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder master: Fix epel element for centos-minimal https://review.openstack.org/562432 | 00:15 |
| clarkb | regardless of what element you use to install centos7 the distro is stiill centos7 | 00:15 |
| pabelanger | clarkb: ianw: ^ | 00:15 |
| pabelanger | no, it is just centos for centos-minimal | 00:16 |
| ianw | yeah, it's messed up | 00:16 |
| clarkb | that seems like a bug | 00:16 |
| pabelanger | 2018-04-18 12:27:29.435 | ++ /opt/dib/tmp/dib_build.aHtWvnMf/hooks/environment.d/10-centos-distro-name.bash:source:1 : DISTRO_NAME=centos | 00:16 |
| *** edmondsw has joined #openstack-infra | 00:16 | |
| clarkb | either it should always just be centos or it should alawys be centos7 | 00:16 |
| clarkb | not one or the other :/ | 00:16 |
| *** eernst has joined #openstack-infra | 00:16 | |
| pabelanger | yah, I am unsure why they are different | 00:16 |
| ianw | clarkb: yeah, it should be, all we need is a time-machine :) at any rate i think we can learn from this for centos8 era | 00:17 |
| clarkb | ++ | 00:17 |
| ianw | pabelanger: i guess this didn't fail CI because we don't pip-and-virtualenv in the nodepool tests | 00:17 |
| ianw | we probably should | 00:17 |
| pabelanger | DIB_EPEL_DISABLED: '1' | 00:18 |
| pabelanger | we need to set that in testing | 00:18 |
| pabelanger | or should | 00:18 |
| ianw | i'm not sure that would have highlighted this though? we need to be trying to use epel to notice it's not working | 00:19 |
| ianw | but, i don't think it hurts | 00:19 |
| *** dingyichen has quit IRC | 00:19 | |
| pabelanger | well, we cannot disable EPEL | 00:20 |
| *** dingyichen has joined #openstack-infra | 00:20 | |
| pabelanger | that is why nb01.o.o is failing | 00:20 |
| pabelanger | since yum-utils didn't get installed | 00:20 |
| pabelanger | http://nb01.openstack.org/centos-7-0000008996.log | 00:20 |
| *** dingyichen has quit IRC | 00:21 | |
| *** dingyichen has joined #openstack-infra | 00:21 | |
| ianw | pableanger: right, "yum-config-manager: command not found" because of the missing match | 00:22 |
| *** edmondsw has quit IRC | 00:23 | |
| ianw | this started because i noticed that python2-pip was actually from epel, and pip-and-virtualenv *thought* it was installing it and overwriting it, but it wasn't really | 00:23 |
| ianw | so i included epel in pip-and-virtualenv | 00:24 |
| *** edmondsw has joined #openstack-infra | 00:24 | |
| ianw | but clearly none of our CI includes epel or pip-and-virtualenv? | 00:24 |
| *** dingyichen has quit IRC | 00:24 | |
| pabelanger | yah, there was a break in tripleo too, but we fixed that this afternoon | 00:24 |
| pabelanger | ianw: no, we have pip-and-virtualenv | 00:25 |
| *** dingyichen has joined #openstack-infra | 00:25 | |
| ianw | pabelanger: hmm, from source? how did we miss then then? | 00:25 |
| pabelanger | it is only a failure with DIB_EPEL_DISABLED: '1' | 00:26 |
| pabelanger | and we don't set it in our CI | 00:26 |
| pabelanger | so we don't call yum-config-manager | 00:26 |
| pabelanger | http://logs.openstack.org/79/561479/10/gate/nodepool-functional-py35-redhat-src/a0dbc04/controller/logs/builds/centos-7-0000000001_log.txt.gz#_2018-04-18_12_29_58_322 | 00:27 |
| *** zhurong has joined #openstack-infra | 00:27 | |
| pabelanger | is where we install get-pip.py | 00:27 |
| ianw | oohhh, right; ok we don't try to disable it so never noticed the missing package, i see now, sorry | 00:28 |
| *** edmondsw has quit IRC | 00:28 | |
| pabelanger | yah | 00:29 |
| pabelanger | http://logs.openstack.org/79/561479/10/gate/nodepool-functional-py35-redhat-src/a0dbc04/controller/logs/builds/centos-7-0000000001_log.txt.gz#_2018-04-18_12_28_50_280 | 00:29 |
| pabelanger | is where it would try to disable | 00:29 |
| *** edmondsw has joined #openstack-infra | 00:29 | |
| ianw | and where i did update it to do yum install --enablerepo=epel python2-pip doesn't care | 00:29 |
| pabelanger | right | 00:29 |
| ianw | ok, yay continuing the long tradition of closely spaced dib point releases | 00:30 |
| pabelanger | we likely should set DIB_EPEL_DISABLED in nodepool dsvm, since we don't actually want to test EPEL packages | 00:30 |
| *** edmondsw has quit IRC | 00:32 | |
| ianw | ++ | 00:32 |
| *** edmondsw has joined #openstack-infra | 00:32 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Switch to http://security.debian.org/ for debian https://review.openstack.org/562349 | 00:32 |
| *** caphrim007_ has quit IRC | 00:37 | |
| *** edmondsw has quit IRC | 00:37 | |
| *** caphrim007 has joined #openstack-infra | 00:38 | |
| *** eernst has quit IRC | 00:39 | |
| *** caphrim007_ has joined #openstack-infra | 00:40 | |
| *** felipemonteiro_ has joined #openstack-infra | 00:40 | |
| *** caphrim007 has quit IRC | 00:43 | |
| *** felipemonteiro_ has quit IRC | 00:46 | |
| *** eernst has joined #openstack-infra | 00:46 | |
| *** eernst has quit IRC | 00:51 | |
| *** tobberydberg has quit IRC | 00:52 | |
| *** anteaya has quit IRC | 01:02 | |
| *** wolverin_ has quit IRC | 01:04 | |
| *** wolverineav has joined #openstack-infra | 01:05 | |
| *** eernst has joined #openstack-infra | 01:08 | |
| *** wolverineav has quit IRC | 01:12 | |
| *** Kevin_Zheng has joined #openstack-infra | 01:19 | |
| *** yamahata has quit IRC | 01:19 | |
| *** cshastri has joined #openstack-infra | 01:19 | |
| *** slaweq has joined #openstack-infra | 01:20 | |
| *** slaweq has quit IRC | 01:25 | |
| *** zhurong has quit IRC | 01:29 | |
| *** germs has quit IRC | 01:30 | |
| *** germs has joined #openstack-infra | 01:30 | |
| *** germs has quit IRC | 01:30 | |
| *** germs has joined #openstack-infra | 01:30 | |
| *** dave-mccowan has joined #openstack-infra | 01:37 | |
| *** tobberydberg has joined #openstack-infra | 01:37 | |
| *** tobberydberg has quit IRC | 01:42 | |
| *** tobberydberg has joined #openstack-infra | 01:44 | |
| ianw | ok, i'm going to try deleting the 2848 images we've leaked into rax-ord | 01:46 |
| ianw | i am pretty sure the shade magic that stores the link to the swift container inside the image is not working in RAX | 01:46 |
| ianw | hence shade doesn't know how to remove the object side of things. but let's clean up the existing images and then we can worry about that | 01:47 |
| *** ssbarnea_ has quit IRC | 01:51 | |
| *** markvoelker_ has joined #openstack-infra | 01:52 | |
| *** markvoelker has quit IRC | 01:53 | |
| *** zhangfei has joined #openstack-infra | 01:55 | |
| *** salv-orl_ has joined #openstack-infra | 01:59 | |
| *** dayou has quit IRC | 02:00 | |
| *** salv-orlando has quit IRC | 02:02 | |
| *** jamesmcarthur has joined #openstack-infra | 02:05 | |
| *** dayou has joined #openstack-infra | 02:05 | |
| *** janki has joined #openstack-infra | 02:07 | |
| *** hongbin_ has joined #openstack-infra | 02:10 | |
| *** jchhatbar has joined #openstack-infra | 02:10 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Add allowed-triggers and allowed-reporters tenant settings https://review.openstack.org/554082 | 02:10 |
| *** salv-orl_ has quit IRC | 02:11 | |
| *** janki has quit IRC | 02:12 | |
| *** gcb has joined #openstack-infra | 02:13 | |
| *** salv-orlando has joined #openstack-infra | 02:13 | |
| *** ramishra has joined #openstack-infra | 02:17 | |
| *** salv-orlando has quit IRC | 02:20 | |
| *** salv-orlando has joined #openstack-infra | 02:22 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Refactor NodeLauncher to be generic https://review.openstack.org/535555 | 02:23 |
| *** masuberu has joined #openstack-infra | 02:27 | |
| *** masber has quit IRC | 02:31 | |
| *** kiennt2609 has joined #openstack-infra | 02:33 | |
| *** kiennt2609 has quit IRC | 02:34 | |
| *** kiennt2637 has joined #openstack-infra | 02:34 | |
| *** kiennt2637 has quit IRC | 02:35 | |
| *** kiennt2609 has joined #openstack-infra | 02:35 | |
| *** stakeda has joined #openstack-infra | 02:39 | |
| *** psachin has joined #openstack-infra | 02:41 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Fix epel element for centos-minimal https://review.openstack.org/562432 | 02:42 |
| *** zhurong has joined #openstack-infra | 02:50 | |
| xinliang | ianw/pabelanger: Now arm64 ubuntu/debian node can be accessed. thanks | 02:50 |
| xinliang | But still has some issues need to login ubuntu/debian to debug. | 02:50 |
| ianw | xinliang: so the debian issue i guess we know, with the scsi drivers not accessing the config-drive? | 02:51 |
| ianw | ubuntu should be ok? | 02:51 |
| xinliang | Debian node also can be accessed , but the previous issue: package can be installed still happend. | 02:52 |
| xinliang | yes, ubuntu node can access now. But I found one strange issue: http://logs.openstack.org/59/557659/14/experimental/kolla-build-debian-source-arm64/4c888fd/job-output.txt.gz#_2018-04-18_11_27_57_643130 | 02:52 |
| ianw | xinliang: oh, what was that? i know that pabelanger has been working on the debian mirrors which might be related | 02:52 |
| *** harlowja_ has quit IRC | 02:53 | |
| xinliang | ianw: this is the debian package installing issue: http://logs.openstack.org/59/557659/15/experimental/kolla-build-debian-source-arm64/7d377d9/job-output.txt.gz#_2018-04-19_01_52_57_150114 | 02:54 |
| ianw | xinliang: hmm ... interesting. i had to switch the download of a file during build from http to https ... it seemed there was a bad cache or something in between | 02:54 |
| xinliang | ianw: yes, ubuntu node seems has https fetching issue. | 02:54 |
| *** rfolco|off has quit IRC | 02:55 | |
| pabelanger | xinliang: you likely want to stop fetching the key from the network and load it from local filesystem | 02:55 |
| *** rfolco|off has joined #openstack-infra | 02:55 | |
| xinliang | ianw: For the debian node package installing issue, maybe there is some broken package installed during image building stage | 02:56 |
| ianw | xinliang: to be clear, i think what pabelanger is saying is that we can cache that key into something like /opt/files during the image build, if it is required constantly, to avoid pulling over the network | 02:56 |
| *** vivsoni has quit IRC | 02:56 | |
| pabelanger | ianw: or cached in the kolla project. I'm not sure what the apt-key is for | 02:57 |
| xinliang | pabelanger:but Fetching remote key is control by the building process of kolla | 02:57 |
| *** vivsoni has joined #openstack-infra | 02:58 | |
| xinliang | pabelanger: kolla might add some repos whose keys either can from key server or a link | 02:58 |
| pabelanger | xinliang: right, I am suggesting move those keys in tree | 02:59 |
| ianw | "Setting up collectd (5.7.1-1.1) ...", | 02:59 |
| ianw | "Job for collectd.service failed because the control process exited with error code.", | 02:59 |
| pabelanger | the release.key you are using hasn't changed is 2017-08-23 12:08 | 02:59 |
| ianw | xinliang: ^ so that's the debian error that causes things to fail | 02:59 |
| ianw | that seems like a debian bug if collectd doesn't install correctly | 02:59 |
| *** toabctl has quit IRC | 03:00 | |
| xinliang | pabelanger: yes, caching might be better, but if a node can't fetching https things, that might not reasonable | 03:00 |
| ianw | https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=779483 ... fails if no FQDN | 03:00 |
| openstack | Debian bug 779483 in collectd "collectd: Fails to install if no FQDN domain name" [Wishlist,Open] | 03:00 |
| pabelanger | networking in linaro-cn1 isn't the best | 03:00 |
| pabelanger | I'm not sure how well jobs are going to work there in general | 03:01 |
| xinliang | ianw: hmm, let me try if i can reproduce the debian issue on my local machine | 03:02 |
| ianw | xinliang: ooohh, that's right, so zuul can log into debian, but none of us roots can | 03:03 |
| *** aeng has quit IRC | 03:04 | |
| xinliang | pabelanger: I've tried it can fetch https key at other linaro-cn1 vm. No idea why it failed on ubuntu node. | 03:04 |
| *** udesale has joined #openstack-infra | 03:08 | |
| *** jamesmcarthur has quit IRC | 03:09 | |
| *** kiennt2609 has quit IRC | 03:10 | |
| xinliang | ianw: sorry, the debian package issue is not the same as previos one. This time is collectd, last time are libffi-dev and libssl-dev | 03:13 |
| xinliang | but, strangely , i can install collectd on my local machine. I will try the linaro-cn1 mirror | 03:14 |
| ianw | xinliang: if it's related to hostnames, etc, that might be different | 03:14 |
| xinliang | ianw, the bug you just show is old and fired for jessie | 03:15 |
| ianw | but still open :) | 03:15 |
| *** aeng has joined #openstack-infra | 03:16 | |
| *** esberglu has joined #openstack-infra | 03:18 | |
| ianw | xinliang: do you have a easily accessible host where you could build the scsi driver we need? we could just manually drop it in until either the cloud upgrades or debian includes it? | 03:19 |
| *** dhajare has joined #openstack-infra | 03:21 | |
| *** slaweq has joined #openstack-infra | 03:21 | |
| xinliang | ianw: which scsi driver? we can fetching the debian kernel source code package and build it | 03:21 |
| *** harlowja has joined #openstack-infra | 03:22 | |
| xinliang | I think we can build it on the linaro-cn1 mirror host on a debian container. what do you think of this? | 03:22 |
| *** masuberu has quit IRC | 03:23 | |
| ianw | xinliang: hmm, that probably would work | 03:24 |
| *** slaweq has quit IRC | 03:26 | |
| ianw | i guess it's a bit fragile, if the kernel updates | 03:27 |
| ianw | in that case i guess the module just doesn't load | 03:27 |
| *** masuberu has joined #openstack-infra | 03:33 | |
| *** nicolasbock has quit IRC | 03:34 | |
| *** masber has joined #openstack-infra | 03:35 | |
| *** cshastri has quit IRC | 03:35 | |
| *** masuberu has quit IRC | 03:38 | |
| *** dhajare has quit IRC | 03:40 | |
| ianw | pabelanger: yay, centos images converting | 03:41 |
| *** dhajare has joined #openstack-infra | 03:44 | |
| prometheanfire | requirements jobs are stuck because we can't build newer libvirt-python on xenial (old libvirt is messing it up), we have two choices I can see, somehow install a newer libvirt or cap libvirt-python to <4.2.0 | 03:48 |
| prometheanfire | http://logs.openstack.org/51/562351/1/check/requirements-tox-py27-check-uc/8b78b71/job-output.txt.gz#_2018-04-18_21_13_19_022587 | 03:49 |
| *** jamesmcarthur has joined #openstack-infra | 03:49 | |
| *** dhajare has quit IRC | 03:53 | |
| xinliang | ianw: yes, when kernel updated we need rebuild again. | 03:57 |
| ianw | promethanfire: hmm, do we only install UCA in devstack jobs? | 03:57 |
| *** cshastri has joined #openstack-infra | 03:58 | |
| ianw | prometheanfire: why are we building it too? we usually have a wheel | 03:58 |
| prometheanfire | ianw: it's part of our generate-constraints job | 03:58 |
| prometheanfire | I'm not sure why a wheel isn't used | 03:58 |
| prometheanfire | ah, only a tar is published | 03:59 |
| prometheanfire | which needs building | 03:59 |
| ianw | did it just update? | 03:59 |
| *** dhajare has joined #openstack-infra | 03:59 | |
| ianw | sometimes a wheel is published and this just sorts itself out | 03:59 |
| prometheanfire | no, 4.2.0 has been out for a while | 04:00 |
| prometheanfire | this has been failing since the 4th | 04:00 |
| prometheanfire | we just noticed in the last couple days | 04:00 |
| prometheanfire | 4.2.0 released april 3 | 04:00 |
| prometheanfire | 4.1.0 is just a tar.gz | 04:01 |
| *** kiennt2609 has joined #openstack-infra | 04:02 | |
| *** kiennt2609 has quit IRC | 04:03 | |
| *** pbourke has quit IRC | 04:07 | |
| *** pbourke has joined #openstack-infra | 04:08 | |
| *** hongbin_ has quit IRC | 04:11 | |
| *** yamahata has joined #openstack-infra | 04:11 | |
| *** germs has quit IRC | 04:12 | |
| clarkb | libvirt python is supposed to be backward compat with older lobvirt | 04:15 |
| clarkb | should check if upstream knows | 04:15 |
| prometheanfire | ya, I'll do that, but I suspect it only goes so far | 04:16 |
| *** bobh has joined #openstack-infra | 04:17 | |
| *** jchhatba_ has joined #openstack-infra | 04:18 | |
| *** jchhatba_ has quit IRC | 04:19 | |
| *** jchhatba_ has joined #openstack-infra | 04:19 | |
| *** jchhatbar has quit IRC | 04:21 | |
| *** zhurong has quit IRC | 04:21 | |
| *** bobh has quit IRC | 04:36 | |
| *** rajinir has quit IRC | 04:50 | |
| *** jchhatbar has joined #openstack-infra | 04:50 | |
| *** eernst has quit IRC | 04:50 | |
| *** eernst has joined #openstack-infra | 04:51 | |
| *** harlowja has quit IRC | 04:52 | |
| *** jchhatba_ has quit IRC | 04:53 | |
| *** Qiming has quit IRC | 04:59 | |
| *** Qiming has joined #openstack-infra | 05:02 | |
| *** bhujay has joined #openstack-infra | 05:05 | |
| *** claudiub|2 has joined #openstack-infra | 05:10 | |
| *** pcichy has joined #openstack-infra | 05:12 | |
| *** Qiming_ has joined #openstack-infra | 05:13 | |
| mordred | ianw: awesome | 05:15 |
| *** annp has quit IRC | 05:15 | |
| mordred | ianw: (re images and objects in rax) | 05:15 |
| *** annp has joined #openstack-infra | 05:15 | |
| ianw | mordred: delete_object('images', 'opensuse-423-1516489083') recurses right? | 05:17 |
| *** mwarad has joined #openstack-infra | 05:17 | |
| *** links has joined #openstack-infra | 05:18 | |
| *** mwarad has quit IRC | 05:18 | |
| *** udesale_ has joined #openstack-infra | 05:19 | |
| *** udesale has quit IRC | 05:19 | |
| *** mwarad has joined #openstack-infra | 05:19 | |
| *** _mwarad_ has joined #openstack-infra | 05:19 | |
| *** _mwarad_ has quit IRC | 05:19 | |
| *** armaan has joined #openstack-infra | 05:20 | |
| *** pgadiya has joined #openstack-infra | 05:21 | |
| *** pgadiya has quit IRC | 05:21 | |
| *** slaweq has joined #openstack-infra | 05:22 | |
| *** e0ne has joined #openstack-infra | 05:27 | |
| *** slaweq has quit IRC | 05:27 | |
| *** pcichy has quit IRC | 05:32 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Add allowed-triggers and allowed-reporters tenant settings https://review.openstack.org/554082 | 05:33 |
| *** larainema has quit IRC | 05:34 | |
| mordred | ianw: yes | 05:37 |
| *** zhurong has joined #openstack-infra | 05:37 | |
| *** quiquell|off is now known as quiquell|ruck | 05:43 | |
| *** armaan has quit IRC | 05:51 | |
| *** yolanda has joined #openstack-infra | 05:52 | |
| *** hashar has joined #openstack-infra | 05:57 | |
| *** TobbeCN has joined #openstack-infra | 05:57 | |
| prometheanfire | pabelanger clarkb: I know it's late and all, but I'm guessing no progress on the devstack patches? | 05:57 |
| *** aeng has quit IRC | 05:59 | |
| *** _mwarad_ has joined #openstack-infra | 06:03 | |
| *** mwarad has quit IRC | 06:03 | |
| *** bhujay has quit IRC | 06:04 | |
| *** bhujay has joined #openstack-infra | 06:04 | |
| *** toabctl has joined #openstack-infra | 06:04 | |
| xinliang | ianw: i can reproduce that debian collectd installing bug. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=779483 | 06:11 |
| openstack | Debian bug 779483 in collectd "collectd: Fails to install if no FQDN domain name" [Wishlist,Open] | 06:11 |
| xinliang | ianw: it seems if the hostname is not a dns name it will failed to install collectd pkg. Which means the host name should be pinged in the node. | 06:15 |
| xinliang | Or collectd installing will be failed | 06:15 |
| *** armaan has joined #openstack-infra | 06:16 | |
| *** _mwarad_ has quit IRC | 06:16 | |
| *** jamesmcarthur has quit IRC | 06:17 | |
| *** bhujay has quit IRC | 06:20 | |
| ianw | xinliang: hmm, so basically if "ping $(hostname)" doesn't work it fails? | 06:20 |
| *** bhujay has joined #openstack-infra | 06:20 | |
| xinliang | ianw: yes | 06:20 |
| *** pcaruana has joined #openstack-infra | 06:21 | |
| xinliang | ianw: I also reproduce this issue on my local ubuntu machine | 06:21 |
| xinliang | but the linaro-cn1 ubuntu node which can install collectd | 06:21 |
| xinliang | so the hostname might not be setting properly on linaro-cn1 debian node, i think | 06:22 |
| *** stakeda has quit IRC | 06:26 | |
| *** dhill__ has joined #openstack-infra | 06:30 | |
| *** dsariel has joined #openstack-infra | 06:31 | |
| xinliang | ianw: from uname -a | 06:31 |
| xinliang | ubuntu: http://logs.openstack.org/59/557659/14/experimental/kolla-build-debian-source-arm64/4c888fd/zuul-info/zuul-info.primary.txt | 06:31 |
| xinliang | debian: http://logs.openstack.org/59/557659/15/experimental/kolla-build-debian-source-arm64/7d377d9/zuul-info/zuul-info.primary.txt | 06:32 |
| xinliang | we can see ubuntu node set hostname to ubuntu-xenial-arm64-linaro-cn1-0003579987 | 06:32 |
| *** dhill_ has quit IRC | 06:32 | |
| xinliang | but debian node's hostname still debian which might not be pinged | 06:33 |
| *** zhangfei has quit IRC | 06:36 | |
| *** slaweq has joined #openstack-infra | 06:36 | |
| *** slaweq_ has joined #openstack-infra | 06:39 | |
| *** florianf has joined #openstack-infra | 06:40 | |
| *** slaweq has quit IRC | 06:41 | |
| *** dklyle has joined #openstack-infra | 06:41 | |
| *** david-lyle has joined #openstack-infra | 06:41 | |
| *** zhangfei has joined #openstack-infra | 06:48 | |
| *** dsariel has quit IRC | 06:49 | |
| *** dims has quit IRC | 06:54 | |
| *** quiquell|ruck is now known as quiquell|ruck|af | 06:54 | |
| *** quiquell|ruck|af is now known as quique|ruck|afk | 06:54 | |
| *** dims has joined #openstack-infra | 06:56 | |
| *** dims has quit IRC | 07:01 | |
| *** alexchadin has joined #openstack-infra | 07:02 | |
| *** dims has joined #openstack-infra | 07:02 | |
| *** hemna_ has quit IRC | 07:03 | |
| *** namnh has joined #openstack-infra | 07:05 | |
| *** bhavik1 has joined #openstack-infra | 07:05 | |
| *** bhavik1 has quit IRC | 07:07 | |
| *** tesseract has joined #openstack-infra | 07:10 | |
| *** shardy has joined #openstack-infra | 07:10 | |
| *** dsariel has joined #openstack-infra | 07:11 | |
| *** dhajare has quit IRC | 07:16 | |
| *** ociuhandu has joined #openstack-infra | 07:16 | |
| *** ociuhandu has quit IRC | 07:17 | |
| *** zhurong has quit IRC | 07:18 | |
| *** dhajare has joined #openstack-infra | 07:21 | |
| *** salv-orlando has quit IRC | 07:23 | |
| *** AL34N1X has joined #openstack-infra | 07:24 | |
| *** salv-orlando has joined #openstack-infra | 07:24 | |
| *** AL34N1X has quit IRC | 07:25 | |
| *** kiennt2609 has joined #openstack-infra | 07:26 | |
| *** salv-orlando has quit IRC | 07:28 | |
| ianw | xinliang: hmm, sorry i'm not familiar if we setup the hostname specifically; worth looking through zuul-jobs as we might do something in there. clarkb may also know | 07:31 |
| *** amoralej|off is now known as amoralej | 07:32 | |
| *** dmellado has joined #openstack-infra | 07:32 | |
| xinliang | ianw: never mind | 07:32 |
| *** rcernin has quit IRC | 07:33 | |
| *** quique|ruck|afk is now known as quiquell|ruck | 07:34 | |
| *** zoli|gone is now known as zoli|wfh | 07:35 | |
| *** dsariel has quit IRC | 07:35 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Unpause Xenial builds https://review.openstack.org/562508 | 07:36 |
| *** zoli|wfh is now known as zoliXXL | 07:36 | |
| *** salv-orlando has joined #openstack-infra | 07:37 | |
| *** jpena|off is now known as jpena | 07:39 | |
| *** electrofelix has joined #openstack-infra | 07:40 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add RAX image cleanup script https://review.openstack.org/562510 | 07:42 |
| ianw | #status log 7000+ leaked images and ~200TB of leaked images and objects cleaned up from our 3 RAX regions. See https://review.openstack.org/#/c/562510/ for more details | 07:43 |
| openstackstatus | ianw: finished logging | 07:44 |
| *** pcaruana has quit IRC | 07:45 | |
| *** pcaruana has joined #openstack-infra | 07:46 | |
| *** jpich has joined #openstack-infra | 07:48 | |
| *** yamahata has quit IRC | 07:49 | |
| *** Qiming has quit IRC | 07:53 | |
| xinliang | clarkb: do you know how nodepool set the hostname of a node? i see that the hostname is the same as the node instance name. | 07:54 |
| xinliang | clarkb: now we found arm64 debian node is not setting the hostname properly | 07:55 |
| frickler | xinliang: is the issue with config-drive on debian fixed? iiuc glean sets the hostname based on data from config-drive | 07:56 |
| frickler | xinliang: looks like it is still missing in ansible_devices here http://logs.openstack.org/59/557659/15/experimental/kolla-build-debian-source-arm64/7d377d9/zuul-info/host-info.primary.yaml | 07:58 |
| *** salv-orlando has quit IRC | 08:00 | |
| *** salv-orlando has joined #openstack-infra | 08:01 | |
| xinliang | frickler: the issue with config-drive is not fixed. So you the incorrect hostname is due to config-drive issue? | 08:02 |
| *** tosky has joined #openstack-infra | 08:02 | |
| frickler | xinliang: I'm pretty sure it is, yes. | 08:03 |
| xinliang | frickler: hmm, thanks , then we need to fix config-drive issue first. | 08:03 |
| *** salv-orlando has quit IRC | 08:05 | |
| *** bhujay has quit IRC | 08:10 | |
| *** lucas-afk is now known as lucasagomes | 08:11 | |
| *** alexchadin has quit IRC | 08:13 | |
| *** dayou has quit IRC | 08:14 | |
| *** dayou has joined #openstack-infra | 08:15 | |
| *** owalsh_afk is now known as owalsh | 08:16 | |
| *** larainema has joined #openstack-infra | 08:19 | |
| *** david-lyle has quit IRC | 08:23 | |
| *** dklyle has quit IRC | 08:23 | |
| *** HeOS has joined #openstack-infra | 08:26 | |
| *** Qiming_ is now known as Qiming | 08:27 | |
| *** gfidente has joined #openstack-infra | 08:30 | |
| *** gfidente has joined #openstack-infra | 08:30 | |
| *** ssbarnea_ has joined #openstack-infra | 08:33 | |
| *** dingyichen has quit IRC | 08:36 | |
| *** vivsoni has quit IRC | 08:38 | |
| *** rcarrillocruz has quit IRC | 08:39 | |
| xinliang | ianw: trying to fix debian config-drive issue by using linaro kernel before upstream fix that issue. linaro kernel will keep updating | 08:40 |
| *** derekh has joined #openstack-infra | 08:40 | |
| xinliang | fire a bug to traking enable CONFIG_SCSI_SYM53C8XX_2 : https://bugs.linaro.org/show_bug.cgi?id=3755 | 08:41 |
| openstack | bugs.linaro.org bug 3755 in Enterprise "RPK: Enable CONFIG_SCSI_SYM53C8XX_2" [Enhancement,Unconfirmed] - Assigned to graeme.gregory | 08:41 |
| *** dhajare has quit IRC | 08:42 | |
| *** efoley has joined #openstack-infra | 08:44 | |
| *** eernst has quit IRC | 08:51 | |
| *** eernst has joined #openstack-infra | 08:51 | |
| *** newstem has quit IRC | 08:54 | |
| *** dhajare has joined #openstack-infra | 08:54 | |
| *** vivsoni has joined #openstack-infra | 08:55 | |
| *** salv-orlando has joined #openstack-infra | 08:58 | |
| *** bhujay has joined #openstack-infra | 09:00 | |
| *** newstem has joined #openstack-infra | 09:07 | |
| *** dizquierdo has joined #openstack-infra | 09:15 | |
| *** eernst has quit IRC | 09:20 | |
| *** eernst has joined #openstack-infra | 09:20 | |
| dmellado | o/ frickler | 09:27 |
| dmellado | we've been observing some issues on our gates that we're totally unable to reproduce outside of the upstream infra | 09:27 |
| dmellado | could you please freeze this machine (http://zuul.openstack.org/stream.html?uuid=4a08a594fe1444cab85e71bfee500e79&logfile=console.log) | 09:27 |
| dmellado | and grant us access for a while? | 09:28 |
| dmellado | This is the issue http://logs.openstack.org/64/561364/2/check/kuryr-kubernetes-tempest-octavia/f028a60/controller/logs/screen-o-api.txt.gz#_Apr_19_08_25_22_230839 | 09:28 |
| dmellado | and we suspect it has something to be with the way the infra is configured | 09:28 |
| dmellado | yolanda: ^^ | 09:28 |
| dmellado | clarkb: pabelanger dmsimard|off ^^ | 09:29 |
| dmellado | AJaeger: ^^ | 09:32 |
| *** panda|rover|off is now known as panda|rover | 09:32 | |
| *** efoley has quit IRC | 09:35 | |
| *** zhangfei has quit IRC | 09:35 | |
| *** caphrim007 has joined #openstack-infra | 09:36 | |
| *** caphrim007_ has quit IRC | 09:38 | |
| *** agopi has quit IRC | 09:40 | |
| dmellado | ubuntu-xenial-rax-dfw-0003608416 <= this is the host I'd like to freeze for a while | 09:40 |
| *** hashar is now known as hasharAway | 09:42 | |
| *** salv-orl_ has joined #openstack-infra | 09:43 | |
| frickler | dmellado: which job is that node running and for which project+branch? I need that for the hold | 09:47 |
| *** salv-orlando has quit IRC | 09:47 | |
| dmellado | frickler: kuryr-kubernetes-tempest-octavia | 09:48 |
| dmellado | kuryr-tempest-plugin master | 09:48 |
| dmellado | https://review.openstack.org/#/c/561364/ | 09:48 |
| frickler | dmellado: o.k., it should be held when/if the job fails, but I can also give you access now already, if you let me know your ssh public key | 09:50 |
| dmellado | frickler: sure, | 09:50 |
| dmellado | github.com/danielmellado.keys | 09:51 |
| dmellado | in any case I'll wait for tempest to go and fail | 09:51 |
| dmellado | thanks! | 09:51 |
| dmellado | I'll try to finish asap | 09:51 |
| dmellado | dulek: ^^ I'll ping yo when this is done | 09:51 |
| dulek | dmellado: Sure, thanks. | 09:52 |
| frickler | dmellado: node ip is 104.239.135.58, login as root should work for you now | 09:52 |
| frickler | dmellado: please ping infra-root once you don't need that node anymore | 09:53 |
| ianw | xinliang: good idea, can you propose something to install that in dib / project-config? | 09:54 |
| *** armaan has quit IRC | 09:55 | |
| *** armaan has joined #openstack-infra | 09:55 | |
| *** eernst has quit IRC | 09:55 | |
| *** eernst has joined #openstack-infra | 09:55 | |
| *** namnh has quit IRC | 10:05 | |
| *** kjackal has quit IRC | 10:08 | |
| *** kjackal has joined #openstack-infra | 10:08 | |
| *** sambetts|afk is now known as sambetts | 10:09 | |
| dmellado | frickler: will do, thanks again! | 10:12 |
| openstackgerrit | Olivier Bourdon proposed openstack/diskimage-builder master: Fix /etc/network/interfaces file contents https://review.openstack.org/562535 | 10:13 |
| *** efoley has joined #openstack-infra | 10:20 | |
| *** kiennt2609 has quit IRC | 10:24 | |
| *** bhujay has quit IRC | 10:33 | |
| *** xinliang has quit IRC | 10:39 | |
| *** boden has joined #openstack-infra | 10:42 | |
| *** zoliXXL is now known as zoli|lunch | 10:47 | |
| *** xinliang has joined #openstack-infra | 10:51 | |
| *** nicolasbock has joined #openstack-infra | 10:57 | |
| *** jpena is now known as jpena|lunch | 10:59 | |
| *** armaan has quit IRC | 10:59 | |
| *** armaan has joined #openstack-infra | 10:59 | |
| *** efoley has quit IRC | 11:08 | |
| *** vivsoni has quit IRC | 11:14 | |
| *** jamesmcarthur has joined #openstack-infra | 11:15 | |
| *** vivsoni has joined #openstack-infra | 11:19 | |
| *** jamesmcarthur has quit IRC | 11:19 | |
| *** zhurong has joined #openstack-infra | 11:22 | |
| *** dhajare has quit IRC | 11:31 | |
| *** lucasagomes is now known as lucas-hungry | 11:35 | |
| *** armaan has quit IRC | 11:37 | |
| *** vivsoni has quit IRC | 11:42 | |
| *** mugsie has quit IRC | 11:42 | |
| *** mugsie has joined #openstack-infra | 11:42 | |
| *** mugsie has quit IRC | 11:42 | |
| *** mugsie has joined #openstack-infra | 11:42 | |
| *** ldnunes has joined #openstack-infra | 11:52 | |
| *** psachin has quit IRC | 11:52 | |
| *** zoli|lunch is now known as zoli | 11:56 | |
| *** zoli is now known as zoli|wfh | 11:56 | |
| *** zoli|wfh is now known as zoli | 11:56 | |
| *** isviridov_away has quit IRC | 11:57 | |
| *** vaidy has quit IRC | 11:57 | |
| *** greghaynes has quit IRC | 11:58 | |
| *** rh-jelabarre has joined #openstack-infra | 11:58 | |
| *** dizquierdo has quit IRC | 11:58 | |
| *** armaan has joined #openstack-infra | 12:00 | |
| *** mriedem has joined #openstack-infra | 12:00 | |
| *** psachin has joined #openstack-infra | 12:01 | |
| *** greghaynes has joined #openstack-infra | 12:01 | |
| *** jpena|lunch is now known as jpena | 12:02 | |
| *** amoralej is now known as amoralej|lunch | 12:06 | |
| *** panda|rover is now known as panda|rover|lnc | 12:10 | |
| *** armaan has quit IRC | 12:11 | |
| *** quiquell|ruck is now known as quique|ruck|food | 12:11 | |
| *** jcoufal has joined #openstack-infra | 12:11 | |
| *** vaidy has joined #openstack-infra | 12:12 | |
| *** isviridov_away has joined #openstack-infra | 12:13 | |
| *** jamesmcarthur has joined #openstack-infra | 12:16 | |
| *** armaan has joined #openstack-infra | 12:17 | |
| *** yamamoto_ has quit IRC | 12:21 | |
| *** armaan has quit IRC | 12:22 | |
| *** jamesmcarthur has quit IRC | 12:26 | |
| *** rfolco|off is now known as rfolco | 12:26 | |
| *** jamesmcarthur has joined #openstack-infra | 12:26 | |
| *** yamamoto has joined #openstack-infra | 12:27 | |
| *** agopi has joined #openstack-infra | 12:28 | |
| *** thiagolib_ has joined #openstack-infra | 12:30 | |
| *** rlandy has joined #openstack-infra | 12:30 | |
| *** krenczewski has quit IRC | 12:30 | |
| *** quique|ruck|food is now known as quiquell|ruck | 12:31 | |
| *** trown|outtypewww is now known as trown | 12:32 | |
| *** cshastri has quit IRC | 12:32 | |
| *** jamesmcarthur has quit IRC | 12:32 | |
| *** lucas-hungry is now known as lucasagomes | 12:33 | |
| *** tpsilva has joined #openstack-infra | 12:37 | |
| *** krenczewski has joined #openstack-infra | 12:37 | |
| *** edmondsw has joined #openstack-infra | 12:38 | |
| *** agopi has quit IRC | 12:38 | |
| *** agopi has joined #openstack-infra | 12:39 | |
| openstackgerrit | Andrey Kurilin proposed openstack-infra/project-config master: [rally] Remove old rally-cinder jobs https://review.openstack.org/562568 | 12:39 |
| openstackgerrit | Andrey Kurilin proposed openstack-infra/openstack-zuul-jobs master: [rally] Remove old rally-cinder jobs https://review.openstack.org/562569 | 12:39 |
| *** agopi_ has joined #openstack-infra | 12:40 | |
| *** e0ne has quit IRC | 12:40 | |
| openstackgerrit | Andrey Kurilin proposed openstack-infra/project-config master: [rally] Remove old rally-cinder jobs https://review.openstack.org/562568 | 12:43 |
| *** zhurong has quit IRC | 12:43 | |
| dhellmann | clarkb & infra folks: as a heads-up, today is the first milestone deadline, so I expect to be tagging a bunch of repositories | 12:43 |
| *** ssbarnea_ has quit IRC | 12:47 | |
| *** neiloy has joined #openstack-infra | 12:50 | |
| *** neiloy has quit IRC | 12:50 | |
| *** neiloy has joined #openstack-infra | 12:51 | |
| *** eernst has quit IRC | 12:51 | |
| *** eernst has joined #openstack-infra | 12:51 | |
| *** agopi__ has joined #openstack-infra | 12:53 | |
| *** newstem has quit IRC | 12:55 | |
| *** agopi has quit IRC | 12:55 | |
| *** agopi_ has quit IRC | 12:55 | |
| *** agopi__ has quit IRC | 12:55 | |
| *** agopi has joined #openstack-infra | 12:56 | |
| *** agopi has quit IRC | 12:56 | |
| *** e0ne has joined #openstack-infra | 12:59 | |
| *** efoley has joined #openstack-infra | 13:01 | |
| *** ssbarnea_ has joined #openstack-infra | 13:01 | |
| *** kgiusti has joined #openstack-infra | 13:01 | |
| *** agopi has joined #openstack-infra | 13:02 | |
| *** armaan has joined #openstack-infra | 13:02 | |
| *** ihar has quit IRC | 13:02 | |
| *** ssbarnea_ has quit IRC | 13:03 | |
| *** esberglu has quit IRC | 13:05 | |
| *** dbecker has joined #openstack-infra | 13:05 | |
| *** neiloy has quit IRC | 13:06 | |
| *** neiloy has joined #openstack-infra | 13:07 | |
| *** panda|rover|lnc is now known as panda|rover | 13:07 | |
| *** Goneri has joined #openstack-infra | 13:07 | |
| *** ssbarnea_ has joined #openstack-infra | 13:09 | |
| *** jistr is now known as jistr|mtg | 13:14 | |
| *** markvoelker_ has quit IRC | 13:16 | |
| *** e0ne_ has joined #openstack-infra | 13:17 | |
| *** markvoelker has joined #openstack-infra | 13:18 | |
| *** e0ne has quit IRC | 13:20 | |
| *** jcoufal has quit IRC | 13:21 | |
| *** jcoufal has joined #openstack-infra | 13:23 | |
| weshay | question: I would like to have the same job voting on some repos and not voting on other repos. Should the zuul def of the job be job.abstract to facilitate that? /me working on https://review.openstack.org/#/c/562353/ | 13:23 |
| *** cshastri has joined #openstack-infra | 13:24 | |
| *** esberglu has joined #openstack-infra | 13:24 | |
| *** zhangfei has joined #openstack-infra | 13:25 | |
| *** VW has joined #openstack-infra | 13:26 | |
| *** salv-orl_ has quit IRC | 13:27 | |
| *** eharney has joined #openstack-infra | 13:30 | |
| *** wolverineav has joined #openstack-infra | 13:33 | |
| *** psachin has quit IRC | 13:33 | |
| *** TobbeCN has quit IRC | 13:35 | |
| corvus | weshay: no, just set the voting attribute on the job in the project-pipeline | 13:36 |
| *** TobbeCN has joined #openstack-infra | 13:36 | |
| *** wolverineav has quit IRC | 13:37 | |
| corvus | weshay: left comment for clarity. though i didn't address the actual zuul error (which looks like the job is already defined in another repo) | 13:38 |
| weshay | corvus, ya.. I have it defined twice just based on the change in voting | 13:38 |
| weshay | so .. I think you are saying I can use a template, but override the voting attribute | 13:39 |
| weshay | I was told I could not do that | 13:39 |
| weshay | the other is defined here https://review.openstack.org/#/c/562347/ | 13:40 |
| *** TobbeCN has quit IRC | 13:40 | |
| corvus | weshay: not a template. the change you linked doesn't have a project-template involved. | 13:40 |
| dhellmann | we had a pre-release job failure caused by a failure to install libvirt during the sdist build. I feel like someone said something in the last week or so about not being able to install libvirt on xenial any more? I don't know if this is the same error, though. Has anyone encountered: http://logs.openstack.org/c9/c9263cde360d37654c4298c496cd9af251f23ce7/pre-release/release-openstack-python/541ad7d/job-output.txt.gz#_201 | 13:41 |
| dhellmann | 8-04-19_13_21_50_210978 | 13:41 |
| dhellmann | that link again, since it wrapped: http://logs.openstack.org/c9/c9263cde360d37654c4298c496cd9af251f23ce7/pre-release/release-openstack-python/541ad7d/job-output.txt.gz#_2018-04-19_13_21_50_210978 | 13:41 |
| weshay | corvus, right.. atm it's just two jobs defined w/ the same name.. I had started w/ a template | 13:41 |
| weshay | earlier | 13:41 |
| *** cshastri has quit IRC | 13:41 | |
| weshay | corvus, ok.. I'll try to just override the voting variable | 13:41 |
| *** caphrim007 has quit IRC | 13:44 | |
| *** amoralej|lunch is now known as amoralej | 13:45 | |
| weshay | corvus, thanks.. this makes a lot more sense now | 13:46 |
| *** hongbin_ has joined #openstack-infra | 13:46 | |
| *** ihar has joined #openstack-infra | 13:47 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Revert "Pause ubuntu-xenial DIBs" https://review.openstack.org/562594 | 13:49 |
| openstackgerrit | Merged openstack-infra/project-config master: Unpause Xenial builds https://review.openstack.org/562508 | 13:55 |
| *** annp_ has joined #openstack-infra | 13:58 | |
| *** krenczewski has quit IRC | 13:58 | |
| pabelanger | clarkb: ^I've unpinned xenial for jobs, FYI | 14:01 |
| *** rajinir has joined #openstack-infra | 14:02 | |
| *** newstem has joined #openstack-infra | 14:03 | |
| *** newstem has quit IRC | 14:04 | |
| dmsimard|off | pabelanger: does that debian-jessie error ring you a bell ? http://logs.openstack.org/45/547245/4/check/ara-integration-debian-py27-2.4.3.0/9cb4bfe/ara-report/result/5bec8917-d339-4e71-86aa-5af2ad296b1f/ | 14:10 |
| *** yamahata has joined #openstack-infra | 14:10 | |
| *** jistr|mtg is now known as jistr | 14:10 | |
| *** nicolasbock has quit IRC | 14:10 | |
| *** Adri2000 has quit IRC | 14:13 | |
| pabelanger | dmsimard|off: yah, just fixing up reprepro for debian now | 14:13 |
| pabelanger | dmsimard|off: hopefully another hour and it should be working | 14:13 |
| dmsimard|off | Cool, thanks | 14:14 |
| pabelanger | dmsimard|off: but you can try a recheck and see if that was just an old image | 14:14 |
| *** bdodd has joined #openstack-infra | 14:14 | |
| dmsimard|off | I tried one this morning, I'll wait a little bit before retrying | 14:14 |
| *** bobh has joined #openstack-infra | 14:14 | |
| *** kiennt26_ has joined #openstack-infra | 14:16 | |
| mnaser | hey, i'm not sure whats going on but i guess there's leaking volumes or something | 14:18 |
| mnaser | and nodepool is just create->fail->create cycle | 14:19 |
| openstackgerrit | Merged openstack-infra/system-config master: Remove debian-security from reprepro https://review.openstack.org/562348 | 14:19 |
| *** rpioso|afk is now known as rpioso | 14:20 | |
| pabelanger | mnaser: yah, we've been leaking them since switching to boot-from-volume. Have you been able to see anything on openstack side to why that is? | 14:20 |
| mnaser | pabelanger: honestly i've been a bit busy to be tracking those down, we have made a lot of improvements to prevent it so it would be healthy to just delete them and "try again" | 14:21 |
| pabelanger | sure, I can clean them up and see if we leak again | 14:21 |
| frickler | dhellmann: just guessing, that may be caused by the recent-ish release of python-libvirt==4.2.0. we get 4.0.0 from queens UCA and 4.1.0 is blocked. may need to block 4.2.0, too. | 14:22 |
| *** efoley has quit IRC | 14:23 | |
| *** quiquell|ruck is now known as quiquell|off | 14:25 | |
| pabelanger | mnaser: we also need to update nodepool to add quota support for volumes, I'll see if I can do that for today | 14:25 |
| frickler | clarkb: running reproduce.sh seems utterly broken by now, maybe we should drop it if we don't intend to fix it. | 14:27 |
| pabelanger | mnaser: volume 139fe4e8-b911-4484-a3d7-3d48ca3d01da looks to be stuck in creating, do you mind looking to see why? | 14:27 |
| frickler | clarkb: first issue is zuul needs py3 now, then zuul-cloner blows up: http://paste.openstack.org/show/719564/ | 14:27 |
| mnaser | reset and deleted pabelanger, anything else you're seeing? | 14:28 |
| *** armaan has quit IRC | 14:28 | |
| pabelanger | mnaser: no, everything else was cleaned out. now we wait to see if we leak again | 14:29 |
| *** krenczewski has joined #openstack-infra | 14:30 | |
| dhellmann | frickler : yeah, thanks. prometheanfire is looking into it in #openstack-requirements | 14:30 |
| pabelanger | mnaser: actually | 14:31 |
| prometheanfire | frickler: in #virt on oftc really | 14:31 |
| prometheanfire | upstream is looking at it, but we should probably still mask 4.2.0 | 14:31 |
| pabelanger | mnaser: any idea what is happening with volume 6e425fa4-66cc-4b55-b8eb-4fb4db5ff96f / f428e504-2ffe-4027-b0a0-0963c6de43ba | 14:32 |
| *** eharney_ has joined #openstack-infra | 14:32 | |
| *** jamesmcarthur has joined #openstack-infra | 14:33 | |
| pabelanger | clarkb: fungi: any objections to tagging a new release of gear today? | 14:33 |
| *** eharney has quit IRC | 14:35 | |
| corvus | frickler: it may be a matter of installing zuul v2 instead of v3 | 14:35 |
| corvus | frickler: though... does it use zuul-cloner to fetch from mergers? that's not going to work anymore | 14:36 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/project-config master: Import os_masakari to openstack-ansible https://review.openstack.org/562615 | 14:37 |
| *** jamesmcarthur has quit IRC | 14:37 | |
| *** efoley has joined #openstack-infra | 14:40 | |
| prometheanfire | frickler: https://www.redhat.com/archives/libvir-list/2018-April/msg01916.html | 14:41 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/project-config master: Import os_masakari to openstack-ansible https://review.openstack.org/562615 | 14:42 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: added endpoint to retrieve track groups metadata https://review.openstack.org/562618 | 14:43 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack-infra/project-config master: Add os_masakari repo base jobs https://review.openstack.org/562619 | 14:44 |
| dhellmann | are there any issues with the mailing list server? I've sent a couple of messages to the list but not received the usual copies, haven't received any other messages for over an hour, and the last message on http://lists.openstack.org/pipermail/openstack-dev/2018-April/date.html is about that old too | 14:44 |
| dhellmann | that's for openstack-dev, if it makes any difference | 14:44 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: added endpoint to retrieve track groups metadata https://review.openstack.org/562618 | 14:45 |
| corvus | dhellmann: i'll take a look | 14:46 |
| dhellmann | corvus : thanks | 14:46 |
| *** markvoelker has quit IRC | 14:47 | |
| pabelanger | ianw: clarkb: just noticed, we are running diskimage-builder on nodepool-builders as python2. I thought we changed that out to be python3 | 14:47 |
| pabelanger | ianw: clarkb: http://paste.openstack.org/show/719566/ | 14:48 |
| corvus | dhellmann: have you sent messages later than your 'reminder for rocky-1' msg? | 14:48 |
| pabelanger | guess we have an issue some place | 14:48 |
| dhellmann | corvus : yes, there should be a couple about a freezer-dr release failure | 14:48 |
| *** markvoelker has joined #openstack-infra | 14:48 | |
| dhellmann | "a couple" == exactly 2 in this case | 14:49 |
| *** salv-orlando has joined #openstack-infra | 14:51 | |
| corvus | we do seem to be accumulating entries in the incoming mailman queue | 14:51 |
| corvus | something has /srv/mailman/openstack/locks/openstack-dev.lock locked | 14:53 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Updated filter/order param https://review.openstack.org/562623 | 14:53 |
| *** Goneri has quit IRC | 14:54 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Updated filter/order param https://review.openstack.org/562623 | 14:56 |
| *** Goneri has joined #openstack-infra | 14:57 | |
| *** kjackal has quit IRC | 14:57 | |
| corvus | i believe an apache process holding the lock for openstack-dev has died. i will remove the lock file manually. | 14:58 |
| corvus | dhellmann: email flood inbound :) | 14:59 |
| * dhellmann braces his mua | 14:59 | |
| *** kiennt26_ has quit IRC | 15:01 | |
| *** TobbeCN has joined #openstack-infra | 15:04 | |
| *** zhangfei has quit IRC | 15:06 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: WIP: Upgrade to Ansible 2.5 https://review.openstack.org/562668 | 15:06 |
| *** TobbeCN has quit IRC | 15:08 | |
| *** ssbarnea_ has quit IRC | 15:09 | |
| *** boden has quit IRC | 15:09 | |
| *** VW_ has joined #openstack-infra | 15:09 | |
| dhellmann | corvus : I think I've received the emails I expected and the summary page seems to have updated | 15:09 |
| dhellmann | corvus : thanks! | 15:09 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add docs on mailman lock files https://review.openstack.org/562670 | 15:10 |
| corvus | infra-root: ^ fyi | 15:10 |
| *** zerick has joined #openstack-infra | 15:10 | |
| corvus | dhellmann: thanks for noticing :) | 15:10 |
| corvus | i think there is a timeout, so it should be self-correcting, but i think the timeout is very generous. | 15:11 |
| corvus | like, hours | 15:11 |
| dhellmann | corvus : human-based service monitoring ftw | 15:11 |
| *** Nil_ has joined #openstack-infra | 15:12 | |
| *** VW has quit IRC | 15:12 | |
| *** yamamoto has quit IRC | 15:13 | |
| *** yamamoto has joined #openstack-infra | 15:14 | |
| *** hasharAway is now known as hashar | 15:15 | |
| *** cshastri has joined #openstack-infra | 15:19 | |
| *** yamamoto has quit IRC | 15:19 | |
| *** annp_ has quit IRC | 15:23 | |
| *** yamahata has quit IRC | 15:23 | |
| *** germs has joined #openstack-infra | 15:24 | |
| *** germs has quit IRC | 15:24 | |
| *** germs has joined #openstack-infra | 15:24 | |
| openstackgerrit | Merged openstack-infra/zuul-website-media master: Run zuul-website jobs https://review.openstack.org/561750 | 15:32 |
| *** ramishra_ has joined #openstack-infra | 15:33 | |
| *** yamamoto has joined #openstack-infra | 15:34 | |
| *** ssbarnea_ has joined #openstack-infra | 15:35 | |
| *** ramishra has quit IRC | 15:37 | |
| *** PsionTheory has joined #openstack-infra | 15:38 | |
| *** camunoz has joined #openstack-infra | 15:39 | |
| pabelanger | okay, new xenial images are online with pip10 by default | 15:40 |
| *** pcichy has joined #openstack-infra | 15:41 | |
| *** armaan has joined #openstack-infra | 15:42 | |
| *** hemna_ has joined #openstack-infra | 15:43 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: late bind pipelines https://review.openstack.org/553618 | 15:45 |
| *** pcichy has quit IRC | 15:46 | |
| *** armaan has quit IRC | 15:47 | |
| *** jmorgan1 has quit IRC | 15:48 | |
| *** armaan has joined #openstack-infra | 15:48 | |
| clarkb | pabelanger: puppet uses openstack-pip provider to install diskimage_builder so its going to be python2 | 15:51 |
| clarkb | frickler: for reproduce.sh zuul-cloner needs to be installed from the last zuul 2.x release, however since we don't publish zuul refs anymore I'm not sure that is super helpful either | 15:52 |
| *** dklyle has joined #openstack-infra | 15:52 | |
| *** david-lyle has joined #openstack-infra | 15:52 | |
| *** caphrim007 has joined #openstack-infra | 15:53 | |
| *** dklyle_ has joined #openstack-infra | 15:53 | |
| *** caphrim007 has quit IRC | 15:53 | |
| *** caphrim007 has joined #openstack-infra | 15:54 | |
| *** caphrim007 has quit IRC | 15:54 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: late bind pipelines https://review.openstack.org/553618 | 15:54 |
| *** jamesmcarthur has joined #openstack-infra | 15:54 | |
| clarkb | I'm having an incredibly slow start today. Putting the kids in "school" means I get to suffer the wrath of the little plague carriers even more now I Guess | 15:54 |
| *** HeOS has quit IRC | 15:56 | |
| *** yamahata has joined #openstack-infra | 15:56 | |
| *** boden has joined #openstack-infra | 15:56 | |
| *** caphrim007 has joined #openstack-infra | 15:57 | |
| corvus | clarkb: send them to the dentist too! | 15:57 |
| corvus | oh, plague, not plaque. sorry. | 15:57 |
| *** slaweq_ has quit IRC | 15:59 | |
| *** slaweq has joined #openstack-infra | 15:59 | |
| *** slaweq has quit IRC | 16:00 | |
| *** slaweq has joined #openstack-infra | 16:00 | |
| pabelanger | must be milestone week, backlong of 1.1k node requests :) | 16:00 |
| *** links has quit IRC | 16:01 | |
| *** armaan has quit IRC | 16:02 | |
| clarkb | I don't think I've ever had a run of time where I've been sick more often than the last 6 months. Eventually it has to get better right? | 16:02 |
| *** nicolasbock has joined #openstack-infra | 16:02 | |
| clarkb | pabelanger: did you see my comment about installing disk image builder on the builder nodes under pip2? that is why it runs under python2 | 16:02 |
| pabelanger | clarkb: wife is constantly sick, I've assumes when kids are 18 and out of house, it gets better | 16:02 |
| clarkb | pabelanger: as for gear release I don't have any objections to it. Gear has largely been a corvus effort so maybe check with corvus too | 16:03 |
| *** pcaruana has quit IRC | 16:03 | |
| pabelanger | clarkb: okay, will check pip provider, I thought we updated it to be pip3 | 16:03 |
| frickler | prometheanfire: thanks for the pointer. so it indeed seems reasonable to block 4.2.0 and wait for a release with that patch in it | 16:03 |
| corvus | clarkb, pabelanger: wfm. there's a gear behavior change that will end up in this release, and we may want to update zuul to account for it. | 16:04 |
| prometheanfire | frickler: see https://review.openstack.org/562613 | 16:04 |
| clarkb | pabelanger: puppet-diskimage_provider is what does it | 16:04 |
| frickler | clarkb: o.k., so it looks indeed like it would make sense to drop reproduce.sh and invest more effort into building a replacement for v3 jobs | 16:04 |
| clarkb | frickler: ya I think so | 16:05 |
| *** felipemonteiro has joined #openstack-infra | 16:05 | |
| corvus | clarkb, pabelanger: i say go for it. when it's released, we can bump zuul's version req and remove line 1935 of executor/server.py | 16:05 |
| corvus | frickler: there are stories about what's needed in storyboard | 16:05 |
| *** slaweq has quit IRC | 16:05 | |
| pabelanger | corvus: clarkb: 0.12.0? since we have new features | 16:06 |
| pabelanger | 0.11.1 is current | 16:06 |
| pabelanger | c00ca944db0d6dc6ef90859b0b9b7f3a58196fb0 as 0.12.0 for gear | 16:08 |
| pabelanger | will tag in a moment | 16:08 |
| *** eernst has quit IRC | 16:08 | |
| *** electrofelix has quit IRC | 16:08 | |
| corvus | pabelanger: wfm | 16:09 |
| pabelanger | 0.12.0 pushed | 16:11 |
| *** jamesmcarthur has quit IRC | 16:12 | |
| pabelanger | dmsimard|off: did you want try debian again? I've updated reprepro to remove debian-security packages | 16:12 |
| *** jpich has quit IRC | 16:15 | |
| *** hashar is now known as hasharAway | 16:17 | |
| pabelanger | dmsimard|off: you also have 2 held nodes in nodepool, looks like 16 days old. Can we delete them? | 16:17 |
| *** HeOS has joined #openstack-infra | 16:20 | |
| dmsimard|off | pabelanger: yes and yes (I still need to troubleshoot that failure but it's something I'll need to circle back to) | 16:23 |
| *** HeOS has quit IRC | 16:27 | |
| pabelanger | clarkb: I wonder if we call pip install -u nodepool, if that is bumping and installing DIB as python | 16:28 |
| pabelanger | python3* | 16:28 |
| pabelanger | dmsimard|off: k, was looking to see what could be freed up for backlog of nodepool | 16:29 |
| pabelanger | mgagne: we're seeing a spike in nodes in deleting state: http://grafana.openstack.org/dashboard/db/nodepool-inap | 16:31 |
| pabelanger | as well as launch failures | 16:31 |
| clarkb | pabelanger: ya if it is listed as a dependency then likely | 16:32 |
| clarkb | so we may flip back and forth in that case | 16:33 |
| pabelanger | yah | 16:33 |
| *** ramishra_ has quit IRC | 16:33 | |
| pabelanger | I'll update puppet-diskimage_builder for pip3 | 16:33 |
| clarkb | pabelanger: in that case I'd probably remove the nomral dib pip install from the puppet manifest | 16:33 |
| clarkb | pabelanger: and just keep the from source option if people opt into that | 16:33 |
| *** jamesmcarthur has joined #openstack-infra | 16:33 | |
| clarkb | (otherwise rely on nodepool install to pull it in) | 16:33 |
| pabelanger | clarkb: only issue with that, is if we need to do a new DIB release for some reason, we'll need to wait for nodepool release to pick it up | 16:34 |
| *** jchhatbar has quit IRC | 16:35 | |
| *** florianf has quit IRC | 16:35 | |
| clarkb | pabelanger: oh right | 16:36 |
| clarkb | I guess if they both use the same python then nodepool update will see the existing isntall and not touch it | 16:36 |
| clarkb | ++ to your original idea then | 16:36 |
| *** lucasagomes is now known as lucas-afk | 16:40 | |
| *** wolverineav has joined #openstack-infra | 16:43 | |
| *** HeOS has joined #openstack-infra | 16:53 | |
| *** ssbarnea_ has quit IRC | 16:53 | |
| *** jamesmcarthur has quit IRC | 16:53 | |
| *** bobh has quit IRC | 16:54 | |
| mgagne | pabelanger: thanks, will be looking into it | 16:54 |
| *** VW_ has quit IRC | 17:00 | |
| *** VW has joined #openstack-infra | 17:00 | |
| *** derekh has quit IRC | 17:01 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Late bind projects https://review.openstack.org/553618 | 17:04 |
| *** trown is now known as trown|lunch | 17:05 | |
| *** VW has quit IRC | 17:05 | |
| *** kjackal has joined #openstack-infra | 17:05 | |
| *** wolverineav has quit IRC | 17:05 | |
| *** wolverineav has joined #openstack-infra | 17:06 | |
| *** pblaho has quit IRC | 17:06 | |
| corvus | we have a significant backlog today, already an 1k node requests | 17:06 |
| *** cshastri has quit IRC | 17:06 | |
| *** gouthamr is now known as gouthamr|afk | 17:06 | |
| pabelanger | yah | 17:07 |
| *** kjackal has quit IRC | 17:07 | |
| *** jamesmcarthur has joined #openstack-infra | 17:07 | |
| pabelanger | few errors in ovh, but think that is just because we are at capacity | 17:07 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: WIP: Upgrade to Ansible 2.5 https://review.openstack.org/562668 | 17:08 |
| *** zoli is now known as zoli|gone | 17:08 | |
| *** zoli|gone is now known as zoli | 17:08 | |
| corvus | pabelanger, Shrews: we don't have a 'time to delete' metric do we? | 17:09 |
| corvus | like, from the moment that nodepool switches a node to delete state to the time the znode is removed | 17:09 |
| *** jpena is now known as jpena|off | 17:10 | |
| corvus | i'm looking at inap, and it has a lot of nodes in delete, but i can't tell if they are cycling through, or if they are stuck. obviously, looking at the individual nodes would indicate that, but that metric might show the same thing in aggregate | 17:10 |
| *** wolverineav has quit IRC | 17:10 | |
| Shrews | corvus: i want to say "no" | 17:10 |
| pabelanger | corvus: yah, mgagne is looking into inap, I think there is an issue going on ATM | 17:10 |
| pabelanger | but would be also nice to have that metric | 17:11 |
| mgagne | pabelanger: I think it's the same mysterious issue as last times | 17:11 |
| corvus | it'd sort of be the mirror of 'time to ready' which tells us if high nodes in building are okay or not | 17:11 |
| pabelanger | mgagne: I wonder if related to us uploading images again to glance, ubuntu-xenial would have just been uploaded about the time the delete starting to lag. | 17:12 |
| pabelanger | mgagne: are we still running with force raw as false? | 17:12 |
| mgagne | pabelanger: afaik yes | 17:12 |
| *** efoley has quit IRC | 17:12 | |
| *** e0ne has joined #openstack-infra | 17:12 | |
| mgagne | pabelanger: but nodes are in BUILD state, so something timed out and nodepool started to delete. but delete never happens for unknown reasons. | 17:13 |
| *** bobh has joined #openstack-infra | 17:13 | |
| pabelanger | mgagne: yah, I wonder if that is because compute nodes is busying coverting qcow2 to raw | 17:13 |
| mgagne | and now it's stuck because: "[instance: 51612309-ba1a-4e69-9c95-b156453e8e13] Instance is already in deleting state, ignoring this request" in nova-cells. and I need to update database. | 17:14 |
| *** rpioso is now known as rpioso|eat | 17:14 | |
| mgagne | somehow the delete task doesn't get through or is cancelled and nova still think the delete process in going on while it's not. | 17:14 |
| pabelanger | k | 17:14 |
| *** e0ne_ has quit IRC | 17:15 | |
| *** wolverineav has joined #openstack-infra | 17:15 | |
| mgagne | it's difficult to debug and pin point the root cause. even if I update database, I think something cached the state and reupdated the database with previous value. it's a mess. | 17:16 |
| mgagne | oh, now one is gone | 17:16 |
| pabelanger | dmsimard|off: how is the version of ansible dectected in http://logs.openstack.org/18/562718/1/check/windmill-ubuntu-xenial/f90f64f/ara-report/ (top right)? Is that to be the running version of ansible? | 17:16 |
| mgagne | magic | 17:16 |
| pabelanger | mgagne: I wonder if nodepool called delete again on that instance | 17:17 |
| mgagne | ok, instances are getting deleted now | 17:17 |
| mgagne | I think so | 17:17 |
| mgagne | because I'm getting the above error multiple times | 17:17 |
| pabelanger | so, we try to delete, but compute doesn't respond | 17:17 |
| pabelanger | then eventually it does | 17:17 |
| pabelanger | would be interesting to see what CPU usage looks like on those compute nodes | 17:18 |
| mgagne | well, nova-cells doesn't cooperate because it thinks the delete process is still happening | 17:18 |
| pabelanger | yah | 17:18 |
| mgagne | so you can't reinit the process without updating the database | 17:18 |
| mgagne | which is... frustrating =) | 17:18 |
| pabelanger | yah, we seem to be launching nodes now | 17:19 |
| mgagne | some are getting deleted, I'm not sure how long it will take to complete | 17:19 |
| pabelanger | my gut is telling me, this might be a result of uploading the latest ubuntu-xenial dib | 17:19 |
| pabelanger | thundering hurd | 17:19 |
| mgagne | and I seem to be fighting a cache somewhere because I feel like I need to update the database multiple times | 17:19 |
| *** yamamoto has quit IRC | 17:20 | |
| mgagne | well, it's not the first time you upload a new image. why today | 17:20 |
| pabelanger | we just unpaused ubuntu-xenial, it was the first time in 7 days we rotated that image | 17:20 |
| pabelanger | and it is our most popular image, so I imaging we trying to use it on all the nodes at once | 17:21 |
| mgagne | ok but... no issue last week or for last couple weeks | 17:21 |
| pabelanger | yah, possible we haven't use this much capacity in a while | 17:21 |
| pabelanger | would need to check logs | 17:21 |
| *** armaan has joined #openstack-infra | 17:24 | |
| *** udesale_ has quit IRC | 17:24 | |
| Shrews | corvus: pabelanger: mordred: oh, 2.5 brings us this lovely thing: "Added a configuration file that a site administrator can use to specify modules to exclude from being used." | 17:25 |
| mgagne | what I feel is missing is the list of running tasks on nova-compute so I can see if it's really happening or not. | 17:25 |
| Shrews | oop, not #zuul | 17:25 |
| *** efried has quit IRC | 17:26 | |
| openstackgerrit | Michael Johnson proposed openstack/diskimage-builder master: Add pip cache cleanup to pip-and-virtualenv https://review.openstack.org/562055 | 17:27 |
| mgagne | pabelanger: no more instances stuck in deleting state, it's a miracle. | 17:29 |
| pabelanger | mgagne: yay | 17:29 |
| *** pcichy has joined #openstack-infra | 17:29 | |
| * mgagne goes back to my ops cave | 17:30 | |
| *** SumitNaiksatam has joined #openstack-infra | 17:30 | |
| pabelanger | mgagne: do you have any metric for load on those compute nodes? maybe to see if nova process if CPU pinned? | 17:30 |
| openstackgerrit | Lance Bragstad proposed openstack-infra/project-config master: Add oslo.limit to zuul project list https://review.openstack.org/550591 | 17:30 |
| mgagne | I can look into it, I'm not the expert in graphs and metrics :P | 17:30 |
| *** VW has joined #openstack-infra | 17:31 | |
| *** david-lyle has quit IRC | 17:32 | |
| *** dklyle_ has quit IRC | 17:33 | |
| *** dklyle has quit IRC | 17:33 | |
| *** shardy has quit IRC | 17:35 | |
| *** efried has joined #openstack-infra | 17:35 | |
| mgagne | pabelanger: ok, the graphs are really a piece of art. but on average, CPU looks to stay between 20-40% with spikes between 40-60%. we do have holes in the graphs but at this point, I don't know if it's caused by a real issue or some "known issues" with our monitoring tools. | 17:35 |
| *** patriciadomin has quit IRC | 17:35 | |
| *** VW has quit IRC | 17:35 | |
| *** wolverineav has quit IRC | 17:39 | |
| *** jamesmcarthur has quit IRC | 17:39 | |
| *** wolverineav has joined #openstack-infra | 17:39 | |
| *** jamesmcarthur has joined #openstack-infra | 17:39 | |
| dmsimard|off | pabelanger: that's the version of Ansible running on the webserver portion. The version of Ansible for the specific playbook is inside the parameters pane | 17:40 |
| *** wolverineav has quit IRC | 17:44 | |
| *** efried has quit IRC | 17:44 | |
| pabelanger | dmsimard|off: ah, was confused. But now see the version I expecty | 17:45 |
| pabelanger | expect* | 17:45 |
| pabelanger | mgagne: holes when we had the delete issue? | 17:45 |
| mgagne | yea, very recently and this night. but we have holes all the time so... | 17:46 |
| pabelanger | mgagne: how are the metrics collected? We've seen gaps in cacti.o.o when there was an issue on the remote node side | 17:47 |
| *** panda|rover is now known as panda|rover|off | 17:47 | |
| mgagne | through snmp so udp I guess | 17:47 |
| pabelanger | yah, same for us | 17:47 |
| mgagne | our monitoring system has some issues (I don't want to use the word garbage) | 17:48 |
| melwitt | can anyone give a hint about what this graph means when it shows a drop in "accepting?" does that mean jobs aren't being accepted by nodes? http://grafana.openstack.org/dashboard/db/zuul-status?panelId=19&fullscreen | 17:48 |
| *** thiagolib_ has quit IRC | 17:49 | |
| *** eharney_ is now known as eharney | 17:50 | |
| pabelanger | melwitt: drop usually mean the executor has a queue of jobs it is trying to process, you can see that in executor queue panel and starting builds panel | 17:50 |
| *** ssbarnea_ has joined #openstack-infra | 17:51 | |
| melwitt | thanks pabelanger | 17:51 |
| pabelanger | it could be because an executor has hit the limit of jobs it runs (memory / load usage) | 17:51 |
| pabelanger | but things are just busy today | 17:52 |
| *** sambetts is now known as sambetts|afk | 17:52 | |
| openstackgerrit | Merged openstack-infra/zuul master: Test base job secrets https://review.openstack.org/561030 | 17:53 |
| *** efried has joined #openstack-infra | 17:53 | |
| *** e0ne has quit IRC | 17:54 | |
| *** felipemonteiro_ has joined #openstack-infra | 17:55 | |
| *** armaan has quit IRC | 17:56 | |
| *** gouthamr|afk is now known as gouthamr | 17:57 | |
| *** jamesmcarthur has quit IRC | 17:58 | |
| *** jamesmcarthur has joined #openstack-infra | 17:58 | |
| *** VW has joined #openstack-infra | 17:58 | |
| *** felipemonteiro has quit IRC | 17:59 | |
| *** slaweq has joined #openstack-infra | 18:01 | |
| *** amoralej is now known as amoralej|off | 18:03 | |
| *** VW has quit IRC | 18:03 | |
| *** cshastri has joined #openstack-infra | 18:05 | |
| *** slaweq has quit IRC | 18:06 | |
| *** felipemonteiro__ has joined #openstack-infra | 18:08 | |
| *** felipemonteiro_ has quit IRC | 18:08 | |
| *** SumitNaiksatam has quit IRC | 18:09 | |
| pabelanger | clarkb: seems ubuntu-xenial DIBs are working well, I haven't seen any fallout yet of pip10 on the images | 18:10 |
| pabelanger | same for centos-7 | 18:10 |
| clarkb | nice, that means the tox siblings fix must be working | 18:11 |
| pabelanger | yah | 18:12 |
| openstackgerrit | Merged openstack-infra/zuul master: Make gearman calls async in ZuulWeb https://review.openstack.org/560026 | 18:13 |
| clarkb | pabelanger: probably worth status logging that? | 18:14 |
| pabelanger | clarkb: sure | 18:17 |
| *** pabelanger has quit IRC | 18:17 | |
| *** pabelanger has joined #openstack-infra | 18:17 | |
| *** cshastri has quit IRC | 18:19 | |
| pabelanger | #status log all DIB images (minus gentoo) have been unpaused for nodepool-builder. Latest release of diskimage-builder fixed our issues related to pip10 and glean failing to boot. | 18:19 |
| openstackstatus | pabelanger: finished logging | 18:19 |
| *** yamamoto has joined #openstack-infra | 18:20 | |
| *** trown|lunch is now known as trown | 18:21 | |
| *** jamesmcarthur has quit IRC | 18:23 | |
| *** rwsu has quit IRC | 18:27 | |
| *** VW has joined #openstack-infra | 18:28 | |
| *** rpioso|eat is now known as rpioso | 18:28 | |
| *** VW has quit IRC | 18:29 | |
| *** VW has joined #openstack-infra | 18:29 | |
| *** pcichy has quit IRC | 18:30 | |
| *** yamamoto has quit IRC | 18:30 | |
| *** pcichy has joined #openstack-infra | 18:30 | |
| *** ssbarnea_ has quit IRC | 18:30 | |
| *** jamesmcarthur has joined #openstack-infra | 18:31 | |
| imacdonn | any known issues with the stable/ocata branch of devstack? Seems to be failing at: | 18:34 |
| imacdonn | + lib/keystone:configure_keystone:205 : cp -p /opt/stack/keystone/etc/policy.json /etc/keystone | 18:34 |
| imacdonn | cp: cannot stat '/opt/stack/keystone/etc/policy.json': No such file or directory | 18:34 |
| clarkb | imacdonn: looks like master doesn't have that file. If I had to guess you are using master keystone | 18:36 |
| imacdonn | hmm, I switched to the stable/ocata branch before running ./stack.sh | 18:36 |
| *** TobbeCN has joined #openstack-infra | 18:36 | |
| imacdonn | looking at https://bugs.launchpad.net/devstack/+bug/1633986 - should have googled before asking ;) | 18:37 |
| openstack | Launchpad bug 1633986 in devstack "Got "no policy json found "when run newest devstack" [Undecided,Invalid] - Assigned to Kevin Zhao (kevin-zhao) | 18:37 |
| clarkb | if the repo already exists it may not checkout the older branch to avoid losing data? | 18:37 |
| rm_work | did gerrit just die? | 18:38 |
| rm_work | or just me | 18:38 |
| imacdonn | the git repo was a fresh clone | 18:38 |
| rm_work | ah just me | 18:38 |
| *** wolverineav has joined #openstack-infra | 18:38 | |
| clarkb | imacdonn: did you clone it or did devstack? | 18:38 |
| clarkb | imacdonn: if you cloned it then devstack may be leaving it alone? unsure just a hunch | 18:38 |
| imacdonn | I cloned it, then switched to the stable/ocata branch, then ran stack.sh | 18:39 |
| imacdonn | 'git log' in /opt/stack/keystone looks right for ocata | 18:39 |
| clarkb | does that file exist? | 18:40 |
| imacdonn | (actually I need to look a bit closer to verify that it's really ocata) ... but no, the policy.json file does not exist | 18:40 |
| *** TobbeCN has quit IRC | 18:41 | |
| clarkb | git describe should tell you what you are checked out on | 18:41 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Fix puppet config for puppet 4 https://review.openstack.org/561746 | 18:44 |
| imacdonn | yeah, that's not right .. it's keystone 13.0.0.0rc1-129-g780f971 | 18:45 |
| imacdonn | I still don't really understand how RECLONE works | 18:46 |
| imacdonn | I turned that off, because it was faiing with: | 18:46 |
| imacdonn | + functions-common:git_update_remote_branch:631 : git checkout -b stable/ocata -t origin/stable/ocata | 18:46 |
| imacdonn | error: Your local changes to the following files would be overwritten by checkout: | 18:46 |
| imacdonn | upper-constraints.txt | 18:46 |
| imacdonn | Please, commit your changes or stash them before you can switch branches. | 18:46 |
| imacdonn | Aborting | 18:46 |
| *** camunoz has quit IRC | 18:47 | |
| *** e0ne has joined #openstack-infra | 18:47 | |
| clarkb | ya its avoiding losing your data there | 18:48 |
| imacdonn | Trying again after 'rm -rf /opt/stack/*' | 18:48 |
| imacdonn | arg .. now it's failing because some pacakges are too new :/ | 18:51 |
| imacdonn | perhaps this is futile ... I'm trying to get a Cinder 3rd Party CI to use the matching devstack branch when testing backports | 18:52 |
| clarkb | the newer packages may be due to the earlier run? | 18:53 |
| imacdonn | yeah | 18:53 |
| clarkb | the way devstack-gate and zuulv3 native jobs handle this is to effectively clone and checkout all the repos before devstack runs then tell devstack to use the repos as is | 18:53 |
| imacdonn | problem here is that the too-new stuff is coming from dist-packages | 18:55 |
| *** tesseract has quit IRC | 18:55 | |
| *** hasharAway is now known as hashar | 18:57 | |
| *** thiagolib_ has joined #openstack-infra | 18:57 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Increase unit testing of host / group vars https://review.openstack.org/559405 | 19:00 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Inventory groups should be under children key https://review.openstack.org/559406 | 19:00 |
| *** rockyg has joined #openstack-infra | 19:01 | |
| *** camunoz has joined #openstack-infra | 19:02 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: WIP: Update to Ansible 2.5 https://review.openstack.org/562668 | 19:06 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: WIP: Update to Ansible 2.5 https://review.openstack.org/562668 | 19:09 |
| *** felipemonteiro_ has joined #openstack-infra | 19:09 | |
| *** felipemonteiro__ has quit IRC | 19:13 | |
| *** jamesmcarthur has quit IRC | 19:16 | |
| *** jamesmcarthur has joined #openstack-infra | 19:17 | |
| *** wolverineav has quit IRC | 19:18 | |
| *** wolverineav has joined #openstack-infra | 19:18 | |
| *** rockyg has quit IRC | 19:21 | |
| *** eernst has joined #openstack-infra | 19:23 | |
| *** wolverineav has quit IRC | 19:25 | |
| *** wolverineav has joined #openstack-infra | 19:25 | |
| *** slaweq has joined #openstack-infra | 19:30 | |
| *** slaweq has quit IRC | 19:30 | |
| *** imacdonn has quit IRC | 19:36 | |
| *** imacdonn has joined #openstack-infra | 19:36 | |
| *** tpsilva has quit IRC | 19:36 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Late bind projects https://review.openstack.org/553618 | 19:38 |
| *** eernst has quit IRC | 19:39 | |
| *** eharney has quit IRC | 19:39 | |
| pabelanger | clarkb: just reading up on ubuntu-bionic, looks like next week (April 26th) is still release day. Bionic DIBs seems to have been working well since we started building beta images. | 19:45 |
| *** niska has quit IRC | 19:52 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Fix puppet config for puppet 4 https://review.openstack.org/561746 | 19:52 |
| *** slaweq has joined #openstack-infra | 19:54 | |
| zigo | pabelanger: Hi there! Did you have time to investigate the issue with the stretch-security mirror? The file at http://mirror.dfw.rax.openstack.org/debian/dists/stretch-security/main/binary-amd64/Packages still holds the jessie-security. | 19:57 |
| *** niska has joined #openstack-infra | 19:57 | |
| zigo | (I'm not pushing, I just want to know if the status changed, or if I may help or something...) | 19:57 |
| pabelanger | zigo: Oh, hmm. They should be deleted | 19:58 |
| pabelanger | let me check | 19:58 |
| zigo | pabelanger: Well, the Packages file still contains Jessie stuff. | 19:58 |
| zigo | pabelanger: Search for Package: libicu-dev in that file. | 19:58 |
| pabelanger | zigo: yah, we purged it from reprepro, I assumed those directories would also be deleted | 19:58 |
| pabelanger | might have to manually delete it | 19:59 |
| pabelanger | let me check something | 19:59 |
| zigo | If you see "deb8u7", then that's the wrong one. It must contain deb9u2. | 19:59 |
| zigo | And it's still the old jessie-security update for libicu-dev, not the stretch one ATM... :( | 20:00 |
| pabelanger | zigo: yah, the plan is not to mirror debian-secuirty into the mirror.debian repo, now we go directly upstream. If we do want to mirror it, we'll create mirror.debian-security to hold them | 20:00 |
| zigo | pabelanger: I very much agree with the plan ! :) | 20:00 |
| zigo | pabelanger: security.debian.org aren't like other repositories, they are maintained by the Debian System Administrator (ie: DSA) team, not by any random person that wants to mirror Debian. | 20:01 |
| zigo | So it's a much more reliable network of mirrors. | 20:01 |
| pabelanger | zigo: yah, i think our mistake was trying to merge the 2 pools into a single pool in AFS | 20:02 |
| pabelanger | that created checksum issues for packages | 20:02 |
| zigo | ok | 20:02 |
| zigo | pabelanger: So, shall I do a recheck of the puppet-openstack patch that adds Debian checks? Or shall I wait? | 20:03 |
| *** harlowja has joined #openstack-infra | 20:03 | |
| pabelanger | zigo: yah, hold on moment, I deleted folders and running reprepro again to confirm it is proper | 20:04 |
| zigo | Thanks. | 20:04 |
| pabelanger | then AFS will release the volumes | 20:04 |
| pabelanger | then we can recheck, and if that still fails, I'll kick off new image builds, as we get sources from AFS | 20:04 |
| *** wolverineav has quit IRC | 20:05 | |
| *** Adri2000 has joined #openstack-infra | 20:06 | |
| *** e0ne has quit IRC | 20:07 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: Make config objects freezable https://review.openstack.org/562816 | 20:11 |
| *** caphrim007_ has joined #openstack-infra | 20:12 | |
| *** rockyg has joined #openstack-infra | 20:12 | |
| *** tosky has quit IRC | 20:15 | |
| pabelanger | zigo: okay, try now | 20:15 |
| *** tosky has joined #openstack-infra | 20:16 | |
| zigo | pabelanger: Cheers, trying ! | 20:16 |
| pabelanger | http://mirror.dfw.rax.openstack.org/debian/dists/ now updated to remove -security | 20:16 |
| pabelanger | and jobs are already configured to use upstream | 20:16 |
| *** caphrim007 has quit IRC | 20:16 | |
| zigo | pabelanger: Though isn't this something to change in the Debian image? | 20:17 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Fix puppet config for puppet 4 https://review.openstack.org/561746 | 20:18 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/ansible-role-puppet master: Don't hardcode puppet-3-specific config paths https://review.openstack.org/562068 | 20:18 |
| pabelanger | zigo: possible, if we are still seeing a failure then we'll need to kick off a new build now that AFS mirrors don't have -security repos. | 20:18 |
| pabelanger | let me check build logs | 20:18 |
| *** e0ne has joined #openstack-infra | 20:19 | |
| *** camunoz has quit IRC | 20:21 | |
| *** jamesmcarthur has quit IRC | 20:24 | |
| *** kgiusti has left #openstack-infra | 20:31 | |
| *** gyee has joined #openstack-infra | 20:32 | |
| *** rockyg has quit IRC | 20:33 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: Make config objects freezable https://review.openstack.org/562816 | 20:35 |
| *** gfidente has quit IRC | 20:36 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: update the release tools to mark newton as closed https://review.openstack.org/562826 | 20:36 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: update the release tools to mark newton as closed https://review.openstack.org/562826 | 20:38 |
| *** jamesmcarthur has joined #openstack-infra | 20:39 | |
| *** jamesmcarthur has quit IRC | 20:43 | |
| *** jmorgan1 has joined #openstack-infra | 20:45 | |
| *** trown is now known as trown|outtypewww | 20:49 | |
| *** eharney has joined #openstack-infra | 20:50 | |
| *** slaweq has quit IRC | 20:54 | |
| *** felipemonteiro_ has quit IRC | 20:54 | |
| *** slaweq has joined #openstack-infra | 20:54 | |
| *** slaweq has quit IRC | 20:59 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: Make config objects freezable https://review.openstack.org/562816 | 21:03 |
| *** rfolco is now known as rfolco|off | 21:06 | |
| *** pcichy has quit IRC | 21:08 | |
| *** dbecker has quit IRC | 21:09 | |
| *** e0ne has quit IRC | 21:10 | |
| *** dbecker has joined #openstack-infra | 21:13 | |
| *** armaan has joined #openstack-infra | 21:14 | |
| *** rwsu has joined #openstack-infra | 21:15 | |
| *** ldnunes has quit IRC | 21:15 | |
| *** Goneri has quit IRC | 21:17 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-mysql_backup master: Use instance variables in puppet template https://review.openstack.org/562832 | 21:17 |
| *** dklyle has joined #openstack-infra | 21:17 | |
| ianw | pabelanger: no calamities with dib 2.14.1 ? | 21:18 |
| *** jtomasek has quit IRC | 21:28 | |
| *** jtomasek has joined #openstack-infra | 21:29 | |
| *** wolverineav has joined #openstack-infra | 21:29 | |
| *** esberglu has quit IRC | 21:30 | |
| *** vtapia has quit IRC | 21:32 | |
| clarkb | ianw: I don't think we've noticed any | 21:36 |
| clarkb | ianw: we are using new images according to pabelanger and jobs haven't gone off the deep end | 21:36 |
| *** thiagolib_ has quit IRC | 21:37 | |
| pabelanger | ianw: clarkb: no, I think things are working fine | 21:37 |
| pabelanger | ianw: clarkb: only outstanding issue is stale packages in debian, that should be fix, waiting to hear from zigo to confirm | 21:38 |
| zigo | pabelanger: It looks like working, see this: http://logs.openstack.org/85/561085/3/check/puppet-openstack-integration-4-scenario001-tempest-debian-stable/448abcd/job-output.txt.gz | 21:40 |
| zigo | pabelanger: Yes, the job is failing, but that's expected ... :P | 21:40 |
| zigo | pabelanger: At least, now, the job is *RUNNING* ! :) | 21:41 |
| pabelanger | great | 21:41 |
| *** agopi has quit IRC | 21:41 | |
| zigo | pabelanger: What's now working though is this: http://logs.openstack.org/85/561085/3/check/puppet-openstack-integration-4-scenario001-tempest-debian-stable/448abcd/job-output.txt.gz#_2018-04-19_21_34_29_522843 | 21:42 |
| zigo | pabelanger: I didn't know it was using the AFS mirror for puppet lab stuff... | 21:42 |
| zigo | pabelanger: Could you add stretch in the mix for that one ? | 21:42 |
| *** jcoufal has quit IRC | 21:43 | |
| zigo | FYI, it does work here on my virtualbox using upstream puppetlabs repo. | 21:43 |
| pabelanger | zigo: we'll need to update : http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/mirror_update.pp#n331 | 21:43 |
| pabelanger | releases, will need to include stretch | 21:43 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Fix puppet config for puppet 4 https://review.openstack.org/561746 | 21:44 |
| zigo | pabelanger: Yeah, exactly ! | 21:44 |
| zigo | pabelanger: Should I just do a patch for this? | 21:45 |
| pabelanger | zigo: yah, then I can review | 21:45 |
| pabelanger | and ianw / clarkb | 21:45 |
| zigo | pabelanger: ie, should: releases => ['xenial'], | 21:45 |
| zigo | become: releases => ['xenial', 'stretch'], | 21:45 |
| zigo | ? | 21:45 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Fix puppet config for puppet 4 https://review.openstack.org/561746 | 21:45 |
| pabelanger | zigo: yah, i think so | 21:45 |
| pabelanger | https://apt.puppetlabs.com/dists/ does list stretch | 21:46 |
| zigo | Thanks, doing it. | 21:46 |
| zigo | Yeah, I know, and it does work in my virtualbox VM here on my laptop with that one. :) | 21:46 |
| *** yamamoto has joined #openstack-infra | 21:49 | |
| *** boden has quit IRC | 21:49 | |
| openstackgerrit | Thomas Goirand proposed openstack-infra/system-config master: Also mirror stretch for puppetlabs https://review.openstack.org/562839 | 21:49 |
| zigo | pabelanger: There you go ! | 21:49 |
| zigo | clarkb: Can you also review https://review.openstack.org/562839 please ? | 21:50 |
| pabelanger | +2 | 21:52 |
| clarkb | pabelanger: looking at file sizes we should be good to just have reprepro update and vos release happen automatically ya? | 21:54 |
| clarkb | (eg we don't need to hold the lock and go through htat process for bigger updatess) | 21:54 |
| clarkb | oh ianw noted that as well | 21:54 |
| clarkb | (I just looked at hte Packages list file and its small in total | 21:54 |
| *** neiloy has quit IRC | 21:55 | |
| pabelanger | yah | 21:56 |
| ianw | oh good https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=895532 merged | 21:59 |
| openstack | Debian bug 895532 in src:linux "linux-image-4.14.0-0.bpo.3-arm64: Enable CONFIG_SCSI_SYM53C8XX_2 for ARM64" [Normal,Fixed] | 21:59 |
| ianw | if we get that kernel, it means we can log in on arm64 debian. currently it doesn't have the drivers for the older (non-virtio) kvm scsi devices presented by the current cloud, so no config drive and no keys deployed | 22:00 |
| *** hashar has quit IRC | 22:00 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/bindep master: Add debian-stable support https://review.openstack.org/562845 | 22:00 |
| openstackgerrit | Paul Belanger proposed openstack-infra/bindep master: Remove debian-jessie support https://review.openstack.org/562846 | 22:00 |
| pabelanger | ianw: wow, that was fast | 22:01 |
| clarkb | ianw: that would explain why the hostnames are not real hostnames on those nodes too | 22:01 |
| clarkb | seems like someone had problems installing some software due to that | 22:01 |
| clarkb | ping $hostname not working and such | 22:01 |
| ianw | clarkb: ahhh, yeah that could be right. yep that was an issue with collectd install where it tries to ping itself we were discussing yesterday | 22:02 |
| *** bobh has quit IRC | 22:02 | |
| *** caphrim007_ has quit IRC | 22:03 | |
| clarkb | ianw: I believe that glean would do the right thing based on hostname info from config drive if that is working (write the correct config to /etc/hosts) | 22:03 |
| pabelanger | okay, https://review.openstack.org/#/q/topic:debian-stable should be all that is needed to remove debian-jessie from nodepool | 22:03 |
| *** caphrim007 has joined #openstack-infra | 22:03 | |
| pabelanger | I'm going to start another thread on ML to see if there are objections | 22:03 |
| pabelanger | basically ara, and ansible-hardening | 22:03 |
| *** dklyle has quit IRC | 22:08 | |
| *** caphrim007_ has joined #openstack-infra | 22:10 | |
| pabelanger | okay, sent to ML | 22:12 |
| *** caphrim007 has quit IRC | 22:13 | |
| clarkb | alright I need to follow up on neutron job indexing | 22:18 |
| clarkb | *log indexing | 22:18 |
| ianw | out of the huge amount of stuff i deleted, we seem to have *one* stuck object in rax | 22:27 |
| ianw | oh, weird. it was there, i tried to delete it via the webui, it gave an error, but now it seems gone anyway | 22:28 |
| clarkb | ihar: to confirm it appears that my fix to os-loganalyze got things indexed for neutron jobs again | 22:29 |
| clarkb | ihar: build_short_uuid:"91dbaba" AND filename:"job-output.txt" is a logstash query that should confirm this for | 22:29 |
| clarkb | *for you | 22:29 |
| clarkb | ianw: those are the best "please do this" "no sorry I can't" "let me try again" "oh I've already done it" | 22:30 |
| clarkb | ianw: dmsimard|off now I guess we just watch it and see fi we leak going forward | 22:30 |
| *** rcernin has joined #openstack-infra | 22:30 | |
| dmsimard|off | \o/ | 22:31 |
| dmsimard|off | Thanks for taking care of that | 22:31 |
| pabelanger | dmsimard|off: mind looking at https://review.openstack.org/562844/ for ARA | 22:32 |
| *** stevebaker has quit IRC | 22:32 | |
| dmsimard|off | +3 | 22:33 |
| ianw | clarkb / dmsimard|off: one thing i noticed was that we were not falling into https://git.openstack.org/cgit/openstack-infra/shade/tree/shade/openstackcloud.py#n4560 as i was deleting, which suggests to me the link between image->object storage maybe isn't setup right | 22:33 |
| ianw | but, we don't appear to have leaked objects on the latest upload either, so maybe it is working and i was just looking at really old images | 22:34 |
| ianw | if i get a sec, i'll upload with an instrumented shade and see if i can double check | 22:34 |
| clarkb | ianw: I want to say mordred said current shade will remove the swift objects once the glance import is done? | 22:34 |
| clarkb | which if that is true you won't need to execute that code on delete typically (I think you'd only need to do it if the glance import failed) | 22:35 |
| *** stevebaker has joined #openstack-infra | 22:35 | |
| clarkb | ianw: https://git.openstack.org/cgit/openstack-infra/shade/tree/shade/openstackcloud.py#n4962 seems to do that | 22:36 |
| pabelanger | dmsimard|off: thanks | 22:36 |
| *** TobbeCN has joined #openstack-infra | 22:37 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Late bind projects https://review.openstack.org/553618 | 22:37 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: Make config objects freezable https://review.openstack.org/562816 | 22:37 |
| *** mriedem is now known as mriedem_away | 22:38 | |
| ianw | clarkb: yeah, i didn't see where IMAGE_OBJECT_KEY was removed from the image though? | 22:38 |
| ianw | clarkb: anyway, https://review.openstack.org/#/c/562510/ did the job -- i think it's the api equivalent of just "rm *" anyway, despite the checks | 22:39 |
| clarkb | ianw: https://git.openstack.org/cgit/openstack-infra/shade/tree/shade/openstackcloud.py#n4955 seems to be where we attempt to set the k/v pair at least | 22:41 |
| *** TobbeCN has quit IRC | 22:41 | |
| openstackgerrit | Merged openstack-infra/system-config master: Also mirror stretch for puppetlabs https://review.openstack.org/562839 | 22:45 |
| *** rwsu has quit IRC | 22:46 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/bindep master: Add openstack-infra/project-config as required project https://review.openstack.org/562862 | 22:46 |
| pabelanger | AJaeger: I think we should start pushing on removing bindep-fallback.txt. Maybe we can come up with a plan and send it out to the ML for comments. | 22:47 |
| pabelanger | AJaeger: eg: any project missing bindep.txt in their project repo, we copy bindep-fallback.txt into it, then delete it from our images | 22:48 |
| clarkb | pabelanger: won't that affect your change above? (it assumes the nodepool elements location for the content) | 22:49 |
| *** edmondsw has quit IRC | 22:49 | |
| *** edmondsw has joined #openstack-infra | 22:49 | |
| *** Sukhdev has joined #openstack-infra | 22:49 | |
| *** armaan has quit IRC | 22:50 | |
| pabelanger | clarkb: yah, eventually. I think we'd keep testing it until all projects are off it. Then we can likely just remove that file and use the intree version to test distro specific things | 22:50 |
| pabelanger | maybe we don't even need that job any more, since we are just testing we can install packages from bindep-fallback.txt | 22:50 |
| *** armaan has joined #openstack-infra | 22:50 | |
| pabelanger | I'm not really keen on adding debian-stretch packages into it | 22:51 |
| pabelanger | clarkb: we could also just say, bionic, stretch, fedora-28 are no longer supported, lock the file and if projects needs things, copy it intree an modify | 22:52 |
| pabelanger | I actually might like that approach better | 22:52 |
| pabelanger | as it forces projects to react | 22:52 |
| *** edmondsw has quit IRC | 22:53 | |
| clarkb | ianw: reading https://git.openstack.org/cgit/openstack-infra/shade/tree/shade/_normalize.py#n314 I think the issue is we need to check if that k is in image.properties not in image itself | 22:54 |
| clarkb | pabelanger: ya the only downside to it is package names tend to not chagne much over time | 22:55 |
| clarkb | so may still be slow going for some projects but that seems like a reasonable stance, going forward you have to update in tree | 22:55 |
| *** slaweq has joined #openstack-infra | 22:55 | |
| pabelanger | yah, going to send a quick ML post to infra about it | 22:55 |
| *** hongbin_ has quit IRC | 22:57 | |
| *** slaweq has quit IRC | 23:00 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/bindep master: DNM - bionic https://review.openstack.org/562866 | 23:01 |
| *** rpioso is now known as rpioso|afk | 23:01 | |
| *** tosky has quit IRC | 23:10 | |
| pabelanger | clarkb: yah, it does look like bionic would pass using bindep-fallback.txt. Maybe we could prime it with a faulty package deliberately to force the issue | 23:11 |
| clarkb | the downside to that approach is it will make it harder to flip everyone over to bionic by default whether they want it or not | 23:12 |
| clarkb | (because we'd be beraking many) | 23:12 |
| pabelanger | right | 23:12 |
| clarkb | apparently it is already time to decide if we want to attend ptg as a team | 23:13 |
| clarkb | (I say yes, it will be excellent time to make progress on modernizing config management and updating base server OSes etc) | 23:13 |
| *** salv-orlando has quit IRC | 23:13 | |
| pabelanger | +1 | 23:13 |
| *** salv-orlando has joined #openstack-infra | 23:13 | |
| clarkb | if you anticipate attending the PTG let me know as I think part of the survey is giving a rough headcount | 23:15 |
| clarkb | (I'll send a more formal mention to the infra list and bring it up in the meeting too) | 23:15 |
| pabelanger | okay, email sent to ML about bindep-fallback | 23:15 |
| *** rockyg has joined #openstack-infra | 23:15 | |
| *** hemna_ has quit IRC | 23:20 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy debian-jessie / fedora-26 https://review.openstack.org/562870 | 23:20 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy debian-jessie / fedora-26 https://review.openstack.org/562870 | 23:22 |
| *** caphrim007_ has quit IRC | 23:23 | |
| *** caphrim007 has joined #openstack-infra | 23:24 | |
| *** caphrim007 has quit IRC | 23:25 | |
| *** caphrim007_ has joined #openstack-infra | 23:25 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy debian-jessie https://review.openstack.org/562870 | 23:27 |
| *** caphrim007_ has quit IRC | 23:30 | |
| clarkb | ianw: before you call it a weekend it would be nice if you had time to go over topic:venv_support. I'd be curious what your feedback is on those changes | 23:31 |
| ianw | clarkb: ok | 23:33 |
| ianw | clarkb: as a very first thought, what i've always thought, is that probably this breaks every plugin | 23:34 |
| ianw | which is maybe just something we need to do | 23:34 |
| clarkb | ianw: it mostly worked with ironic except for one assumption there | 23:35 |
| clarkb | but yes PATH in particular is likely ot be a problem | 23:35 |
| pabelanger | and another ML post about ubuntu-bionic and legacy nodesets | 23:37 |
| *** Goneri has joined #openstack-infra | 23:40 | |
| *** agopi has joined #openstack-infra | 23:41 | |
| *** jamesmcarthur has joined #openstack-infra | 23:43 | |
| *** rockyg has quit IRC | 23:44 | |
| prometheanfire | diablo_rojo: how many train emails so far? | 23:47 |
| pabelanger | conig-core: https://review.openstack.org/562870/ easy review to remove unused nodeset from ozj | 23:51 |
| clarkb | pabelanger: re legacy nodesets they don't really seem to be all that different from the non legacy nodesets | 23:51 |
| clarkb | legacy-ubuntu-xenial-2-node is identical to ubuntu-xenial-2-node for example | 23:52 |
| clarkb | (so I think we shouldn't really need to worry too much about bionic) | 23:52 |
| clarkb | (the non legacy nodeset should just work right) | 23:52 |
| pabelanger | clarkb: main one is legacy-ubuntu-xenial vs ubuntu-xenial, but yah you are right. | 23:54 |
| pabelanger | it means jobs would still switch their nodesets for legacy jobs | 23:55 |
| pabelanger | but again, likely you said last week, we should push back on that happening in project-config in favor of intree changes | 23:55 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!