Tuesday, 2018-11-27

*** _alastor_ has quit IRC00:02
*** jesusaur has quit IRC00:02
*** wolverineav has quit IRC00:04
*** wolverineav has joined #openstack-infra00:08
*** slaweq has joined #openstack-infra00:16
*** tosky has quit IRC00:16
mriedem_awaywell this doesn't look good http://logs.openstack.org/19/619319/2/check/devstack-multinode/196ebc3/controller/logs/devstacklog.txt.gz#_2018-11-26_23_19_41_32300:19
*** jesusaur has joined #openstack-infra00:23
mriedem_awaynvm, maybe a fluke00:23
*** mriedem_away has quit IRC00:23
*** slaweq has quit IRC00:24
*** ijw has quit IRC00:28
*** ijw has joined #openstack-infra00:29
*** armstrong has quit IRC00:34
*** longkb has joined #openstack-infra00:39
*** longkb has quit IRC00:39
*** fresta has joined #openstack-infra00:49
*** fresta_ has quit IRC00:53
*** _alastor_ has joined #openstack-infra00:54
*** _alastor_ has quit IRC00:59
openstackgerritIan Wienand proposed openstack-infra/zuul master: Add gearman stats reference  https://review.openstack.org/62019201:04
*** jamesmcarthur has joined #openstack-infra01:04
*** jamesmcarthur has quit IRC01:08
*** sthussey has quit IRC01:11
*** slaweq has joined #openstack-infra01:16
*** slaweq has quit IRC01:24
*** eernst has joined #openstack-infra01:25
*** eernst has quit IRC01:30
*** wolverin_ has joined #openstack-infra01:41
*** wolverin_ has quit IRC01:42
*** wolverineav has quit IRC01:42
*** wolverineav has joined #openstack-infra01:42
*** ijw has quit IRC01:47
*** ijw has joined #openstack-infra01:48
*** ijw has quit IRC01:52
*** hongbin has joined #openstack-infra01:55
ianwmriedem_away: hrm, i think that walks /proc/pid so is going to be racy01:55
ianwhowever, we handle that on the open but not subsequent read01:56
*** slaweq has joined #openstack-infra02:13
*** wolverineav has quit IRC02:16
*** wolverineav has joined #openstack-infra02:16
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Capture ramdisk test run logs  https://review.openstack.org/61940002:16
*** wolverineav has quit IRC02:16
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Fix unit tests for elements  https://review.openstack.org/61938702:16
openstackgerritIan Wienand proposed openstack/diskimage-builder master: package-installs: provide for skip from env var  https://review.openstack.org/61911902:16
openstackgerritIan Wienand proposed openstack/diskimage-builder master: simple-init: allow for NetworkManager support  https://review.openstack.org/61912002:16
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Make tripleo-buildimage-overcloud-full-centos-7 non-voting  https://review.openstack.org/62020002:16
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Revert "Make tripleo-buildimage-overcloud-full-centos-7 non-voting"  https://review.openstack.org/62020102:16
*** wolverineav has joined #openstack-infra02:17
*** wolverineav has quit IRC02:21
*** ramishra has joined #openstack-infra02:22
*** mrsoul has joined #openstack-infra02:23
*** slaweq has quit IRC02:24
*** psachin has joined #openstack-infra02:40
*** agopi has joined #openstack-infra02:49
*** bhavikdbavishi has joined #openstack-infra02:55
*** ijw has joined #openstack-infra02:56
*** ijw has quit IRC03:01
*** dtroyer has quit IRC03:10
*** dtroyer has joined #openstack-infra03:11
*** slaweq has joined #openstack-infra03:16
*** ddurst has quit IRC03:19
*** ddurst has joined #openstack-infra03:21
*** calebb has joined #openstack-infra03:22
*** slaweq has quit IRC03:24
*** ddurst has quit IRC03:30
*** ddurst has joined #openstack-infra03:31
*** ijw has joined #openstack-infra03:34
*** ijw has quit IRC03:39
*** diablo_rojo has quit IRC03:42
*** udesale has joined #openstack-infra03:54
*** ykarel|away has joined #openstack-infra03:57
*** hongbin has quit IRC04:06
*** ykarel|away is now known as ykarel04:06
*** dtroyer has quit IRC04:09
*** slaweq has joined #openstack-infra04:11
*** rh-jelabarre has quit IRC04:17
*** rh-jelabarre has joined #openstack-infra04:20
*** slaweq has quit IRC04:24
*** udesale has quit IRC04:31
*** dtroyer has joined #openstack-infra04:38
*** ykarel has quit IRC04:46
*** janki has joined #openstack-infra04:49
*** ijw has joined #openstack-infra04:50
*** ijw has quit IRC04:55
*** ykarel has joined #openstack-infra05:03
*** ijw has joined #openstack-infra05:10
*** ijw has quit IRC05:14
*** chandan_kumar has joined #openstack-infra05:26
*** chandan_kumar is now known as chkumar|ruck05:28
*** ijw has joined #openstack-infra05:28
*** ijw has quit IRC05:33
openstackgerritMerged openstack-infra/zuul master: Add support for Gerrit v2.16's change URL schema  https://review.openstack.org/61953305:45
openstackgerritMerged openstack-infra/nodepool master: Update node during lockNode  https://review.openstack.org/61645005:47
*** ijw has joined #openstack-infra05:47
*** ijw has quit IRC05:51
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: Add allowed-labels tenant setting  https://review.openstack.org/61774006:00
*** jamesmcarthur has joined #openstack-infra06:04
*** ijw has joined #openstack-infra06:06
*** jamesmcarthur has quit IRC06:09
*** ijw has quit IRC06:10
*** slaweq has joined #openstack-infra06:11
*** bhavikdbavishi has quit IRC06:16
*** slaweq has quit IRC06:24
*** ccamacho has quit IRC06:30
*** graphene has joined #openstack-infra06:30
*** bhavikdbavishi has joined #openstack-infra06:35
*** _alastor_ has joined #openstack-infra06:35
*** _alastor_ has quit IRC06:40
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Fix test race in test_hold_expiration_higher_than_default  https://review.openstack.org/62022206:44
*** dpawlik has joined #openstack-infra06:47
*** dpawlik has quit IRC06:54
*** rcernin has quit IRC06:58
*** ahosam has joined #openstack-infra07:02
*** ahosam has quit IRC07:16
*** aojea has joined #openstack-infra07:21
*** dpawlik has joined #openstack-infra07:23
*** pcaruana has joined #openstack-infra07:23
*** quiquell has joined #openstack-infra07:33
*** hjensas has joined #openstack-infra07:34
*** ykarel is now known as ykarel|lunch07:35
*** dpawlik has quit IRC07:38
*** slaweq has joined #openstack-infra07:42
*** dpawlik has joined #openstack-infra07:44
*** simon-AS559 has joined #openstack-infra07:47
openstackgerritMerged openstack-infra/nodepool master: Add extra safety belt when reusing a node  https://review.openstack.org/61646507:49
*** ccamacho has joined #openstack-infra07:56
*** longkb has joined #openstack-infra08:00
*** ginopc has joined #openstack-infra08:04
*** yboaron has joined #openstack-infra08:06
*** kjackal has quit IRC08:11
*** florianf|afk is now known as florianf08:13
*** graphene has quit IRC08:14
*** graphene has joined #openstack-infra08:16
*** lpetrut has joined #openstack-infra08:21
*** jtomasek has joined #openstack-infra08:22
*** priteau has joined #openstack-infra08:22
*** shardy has joined #openstack-infra08:25
*** kjackal has joined #openstack-infra08:29
*** ralonsoh has joined #openstack-infra08:29
*** ykarel|lunch is now known as ykarel08:32
*** ijw has joined #openstack-infra08:36
*** tosky has joined #openstack-infra08:38
*** ijw has quit IRC08:41
*** irclogbot_1 has quit IRC08:44
*** irclogbot_1 has joined #openstack-infra08:46
*** yboaron_ has joined #openstack-infra08:47
*** yboaron has quit IRC08:51
*** jpena|off is now known as jpena08:51
*** irclogbot_1 has quit IRC08:53
*** irclogbot_1 has joined #openstack-infra08:55
*** ijw has joined #openstack-infra08:57
*** ijw has quit IRC09:01
*** xek has joined #openstack-infra09:01
*** rossella_s has joined #openstack-infra09:04
*** jpich has joined #openstack-infra09:05
*** gfidente has joined #openstack-infra09:05
*** udesale has joined #openstack-infra09:09
*** ahosam has joined #openstack-infra09:15
*** yboaron_ has quit IRC09:16
*** yboaron_ has joined #openstack-infra09:17
openstackgerritIldiko Vancsa proposed openstack-infra/project-config master: Adding repo and Sotryboard for the Edge Comptuing Group  https://review.openstack.org/62015109:18
*** ccamacho has quit IRC09:20
*** ccamacho has joined #openstack-infra09:30
*** derekh has joined #openstack-infra09:40
*** xek has quit IRC09:44
*** alexchadin has joined #openstack-infra09:46
*** xek has joined #openstack-infra09:48
*** longkb has quit IRC09:52
*** roman_g has quit IRC09:58
*** roman_g has joined #openstack-infra10:00
openstackgerritIldiko Vancsa proposed openstack-infra/project-config master: Adding repo and Sotryboard for the Edge Comptuing Group  https://review.openstack.org/62015110:12
*** xek has quit IRC10:18
openstackgerritIan Wienand proposed openstack-infra/zuul master: Add gearman stats reference  https://review.openstack.org/62019210:19
openstackgerritIan Wienand proposed openstack-infra/zuul master: Remove STATE_PENDING  https://review.openstack.org/62028410:19
openstackgerritIan Wienand proposed openstack-infra/zuul master: [wip] Rework zuul nodepool stats reporting  https://review.openstack.org/62028510:19
*** bhavikdbavishi has quit IRC10:22
*** xek has joined #openstack-infra10:23
*** ramishra has quit IRC10:26
*** udesale has quit IRC10:33
*** udesale has joined #openstack-infra10:34
*** _alastor_ has joined #openstack-infra10:37
*** udesale has quit IRC10:39
*** _alastor_ has quit IRC10:41
*** ramishra has joined #openstack-infra10:49
*** alexchadin has quit IRC10:54
openstackgerritSorin Sbarnea proposed openstack-infra/zuul master: Bump ansible from 2.5 to 2.6  https://review.openstack.org/62029110:56
*** simon-AS559 has quit IRC10:59
*** sean-k-mooney has quit IRC11:04
*** udesale has joined #openstack-infra11:06
*** electrofelix has joined #openstack-infra11:17
*** roman_g has quit IRC11:18
*** bhavikdbavishi has joined #openstack-infra11:19
openstackgerritJose Luis Franco proposed openstack-infra/project-config master: Remove ansible-role-redhat-subscription from central repo  https://review.openstack.org/61797411:21
*** dosaboy has quit IRC11:24
*** roman_g has joined #openstack-infra11:26
*** roman_g has quit IRC11:26
*** roman_g has joined #openstack-infra11:26
*** dosaboy has joined #openstack-infra11:31
*** ahosam has quit IRC11:31
*** sambetts_ has joined #openstack-infra11:32
*** sambetts_ is now known as sambetts|afk11:32
*** chkumar|ruck is now known as chkumar|away11:32
*** mgutehall has quit IRC11:38
*** udesale has quit IRC11:38
*** mgutehall has joined #openstack-infra11:39
*** xek has quit IRC11:40
*** ahosam has joined #openstack-infra11:41
*** armstrong has joined #openstack-infra11:43
*** udesale has joined #openstack-infra11:45
*** rfolco is now known as rfolco_doctor11:45
*** udesale has quit IRC11:45
*** udesale has joined #openstack-infra11:46
*** kjackal has quit IRC11:46
*** gfidente has quit IRC11:47
*** udesale has quit IRC11:52
*** udesale has joined #openstack-infra11:53
*** bhavikdbavishi has quit IRC11:54
*** gfidente has joined #openstack-infra11:56
*** janki has quit IRC11:59
*** kjackal has joined #openstack-infra12:04
*** ahosam has quit IRC12:05
*** ahosam has joined #openstack-infra12:05
*** xek has joined #openstack-infra12:11
*** ccamacho has quit IRC12:12
*** yamamoto has quit IRC12:16
*** yamamoto has joined #openstack-infra12:16
*** udesale has quit IRC12:20
*** ahosam has quit IRC12:20
*** ahosam has joined #openstack-infra12:20
*** udesale has joined #openstack-infra12:32
*** udesale has quit IRC12:32
*** kjackal has quit IRC12:32
*** kjackal has joined #openstack-infra12:33
*** udesale has joined #openstack-infra12:34
*** jpena is now known as jpena|lunch12:36
*** kjackal has quit IRC12:37
*** kjackal_v2 has joined #openstack-infra12:37
*** ccamacho has joined #openstack-infra12:41
*** ykarel is now known as ykarel|away12:42
*** trown has quit IRC12:51
*** trown|brb has joined #openstack-infra12:53
*** udesale has quit IRC12:54
*** ykarel|away has quit IRC12:56
*** takamatsu has quit IRC12:58
*** arxcruz is now known as arxcruz|brb13:00
*** boden has joined #openstack-infra13:01
*** dave-mccowan has joined #openstack-infra13:01
*** graphene has quit IRC13:03
*** fuentess_ has joined #openstack-infra13:04
*** ccamacho has quit IRC13:04
*** graphene has joined #openstack-infra13:04
*** ccamacho has joined #openstack-infra13:04
openstackgerritBenoĆ®t Bayszczak proposed openstack-infra/zuul master: add fetch_vault_secrets Ansible module  https://review.openstack.org/62031113:05
*** dave-mccowan has quit IRC13:06
*** aojeagarcia has joined #openstack-infra13:09
*** aojea has quit IRC13:09
*** ahosam has quit IRC13:17
*** takamatsu has joined #openstack-infra13:21
*** gfidente has quit IRC13:28
*** rlandy has joined #openstack-infra13:28
*** ykarel|away has joined #openstack-infra13:28
*** gfidente has joined #openstack-infra13:29
*** _alastor_ has joined #openstack-infra13:31
*** takamatsu has quit IRC13:37
*** jpena|lunch is now known as jpena13:38
*** takamatsu has joined #openstack-infra13:43
*** jhesketh_ has joined #openstack-infra13:44
*** trown|brb is now known as trown13:45
*** jhesketh has quit IRC13:50
*** hughsaunders has quit IRC13:50
*** armstrong has quit IRC13:53
*** udesale has joined #openstack-infra13:56
*** kgiusti has joined #openstack-infra13:58
*** sthussey has joined #openstack-infra13:58
*** udesale has quit IRC14:05
*** udesale has joined #openstack-infra14:05
*** pbourke has quit IRC14:07
openstackgerritIldiko Vancsa proposed openstack-infra/project-config master: Adding repo and Sotryboard for the Edge Comptuing Group  https://review.openstack.org/62015114:09
*** kjackal_v2 has quit IRC14:12
*** kjackal has joined #openstack-infra14:13
*** bobh has joined #openstack-infra14:14
*** ykarel|away has quit IRC14:16
*** mriedem has joined #openstack-infra14:18
*** eharney has joined #openstack-infra14:19
*** portdirect has joined #openstack-infra14:26
portdirectfungi: o/ hey I seem to have two accounts in gerrit14:30
openstackgerritAntoine Musso proposed openstack-infra/zuul master: gerrit: use change: when querying changes  https://review.openstack.org/62032814:30
portdirectcan we remove the inactive one?14:30
*** xek has quit IRC14:30
fungiportdirect: well, they're both "active" from gerrit's perspective, which is the crux of the problem. we need to figure out which can be marked inactive14:32
fungilemme pull up the details on them real quick14:32
*** _alastor_ has quit IRC14:33
fungiboth accounds were created in 2016, less than a week apart14:33
fungiaccounts 23928 and 2404914:34
*** xek has joined #openstack-infra14:34
fungi23928 has username portdirect while 24049 has no username14:35
fungi24049 also has no ssh keys14:35
fungiso i think that's the one to deactivate as long as it's not the one tied to your in-use openid14:36
portdirectI think so as well14:36
*** mordred has joined #openstack-infra14:36
fungiportdirect: if you log into the gerrit webui and go to https://review.openstack.org/#/settings/ what account id does it show?14:37
* mordred waves to all the nice people14:37
portdirectfungi: 2392814:38
mordredinfra-root: heads up - I have successfully diagnosed that something is broken on my laptop (filesystem corruption perhaps?) so I need to reinstall it - I'm backing up important files that seem mostly fine, but then will be doing a re-install - so I'm likely still not really here14:39
fungiportdirect: simple then. i'll mark 24049 inactive in just a moment14:40
fungiportdirect: and done. now remind me, what was the core review group to which you needed to be added while you were out?14:41
*** xek_ has joined #openstack-infra14:43
portdirectoh - we have two14:44
portdirectopenstack-helm-images, and openstack-helm-docs14:44
*** jento has quit IRC14:45
portdirectif you could add me there, it would be great and I'll get the other cores up and running14:45
*** xek has quit IRC14:46
*** rfolco_doctor is now known as rfolco14:53
*** jento has joined #openstack-infra14:55
*** edmondsw has joined #openstack-infra14:59
*** dpawlik has quit IRC15:04
*** ianychoi_ is now known as ianychoi15:06
*** roman_g has quit IRC15:15
*** lpetrut has quit IRC15:18
*** eharney has quit IRC15:19
mriedemcould i get one more core for this relatively simple change that's holding up testing of placement changes with tempest/grenade on py3 https://review.openstack.org/#/c/617662/15:23
mriedemi think once ^ is in we can start moving off the legacy integrated-gate template as well15:24
mriedemnot sure what dhellmann has for plans about that15:24
fungiportdirect: done and done15:24
*** eharney has joined #openstack-infra15:26
*** shardy has quit IRC15:26
*** rkukura has quit IRC15:27
*** lewo has joined #openstack-infra15:27
*** bhavikdbavishi has joined #openstack-infra15:31
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Add grenade-py3 job to integrated-gate-py35 template  https://review.openstack.org/61766215:35
*** ykarel|away has joined #openstack-infra15:36
*** pbourke has joined #openstack-infra15:37
*** shardy has joined #openstack-infra15:44
*** jcoufal has joined #openstack-infra15:47
*** dpawlik has joined #openstack-infra15:50
*** jcoufal has quit IRC15:52
*** quiquell is now known as quiquell|off15:54
*** gfidente has quit IRC15:54
*** gfidente has joined #openstack-infra15:54
*** udesale has quit IRC15:56
*** udesale has joined #openstack-infra15:57
*** udesale has quit IRC15:58
*** dansmith has quit IRC16:02
*** dansmith has joined #openstack-infra16:02
*** diablo_rojo has joined #openstack-infra16:02
*** _alastor_ has joined #openstack-infra16:11
*** jamesmcarthur has joined #openstack-infra16:11
* mordred has cobbled together a makeshift system for use while working on the laptop - so can be at least partially helpful16:18
clarkbI'm technically back today, but have come down with extra plague crud after spending weekend with some small children. This to say I'm going to be slow and brain not working well so please be paitent :)16:20
*** eharney has quit IRC16:23
*** armax has quit IRC16:23
*** jcoufal has joined #openstack-infra16:28
*** apetrich has quit IRC16:28
*** lpetrut has joined #openstack-infra16:30
*** yboaron_ has quit IRC16:35
*** roman_g has joined #openstack-infra16:37
*** tpsilva has joined #openstack-infra16:41
*** apetrich has joined #openstack-infra16:42
*** jamesmcarthur has quit IRC16:48
*** rkukura has joined #openstack-infra16:49
*** jamesmcarthur has joined #openstack-infra16:49
*** agopi is now known as agopi|food_run16:49
*** arxcruz|brb is now known as arxcruz16:54
clarkbmordred: one thing I/we have run into reviewing some of ianw's docker related changes is that docker manages iptables and knows nothing about iptables persistent and vice versa16:58
clarkbmordred: as is anytime we update firewall rules we'll need to restart the docker daemon which restarts containers (I think)16:58
clarkbmordred: I'm thinking we should probably consider if a switch to the DOCKER_USER (I think that is what it is called) chain is worthwhile and update our spec with whatever we decide16:59
mordredclarkb: oh joy. I'll try to wrap my head around that16:59
*** gyee has joined #openstack-infra16:59
clarkbI think restarting docker when we restart iptables persistent is workable and we can even hack a unit modification file to ensure that relationship. But that may not work great once gerrit runs in docker? though firewall rules for gerrit also basically never change17:00
*** jpich has quit IRC17:01
*** ginopc has quit IRC17:01
mordredclarkb: yah - seems workable for a start, but also like we might want to ponder more too17:01
clarkbfungi: also you brought up nftables as another thing to be concerned about. At least on bionic the iptables persistent service is actually nftables that reads iptables rules so I expect htat isn't a major issue17:01
*** lpetrut has quit IRC17:01
fungiclarkb: yeah, it's an _eventual_ concern when distros we care about finally decide to drop the (deprecated) "iptables driver" for nftables17:02
fungibut then it's mostly (i think?) just a matter of learning the new rule syntax17:02
clarkbya, and likely won't happen until things like neutron and docker and so on all speak nftables17:03
clarkbwe likely have quite some time there17:03
*** sshnaidm|afk is now known as sshnaidm17:08
mordredclarkb, fungi, ianw: it might be worth consider if we can ignore the docker iptables rules completely ... we don't need/want network isolation from the docker containers we're running (or will run)17:08
mordredso launching them with network namespacing turned off should potentially obviate the need for the docker nat layer17:09
clarkbya I had forgotten that was part of the spec, but I think that is correct. Docker has a bunch of iptables rules in place to do nat and allow contaainers to speak from container namespace out through your default route17:09
clarkbbut if we don't network namespace then the containers won't need that and if our iptables rules overwrite docker's we should be fine17:09
mordred++17:09
*** lpetrut has joined #openstack-infra17:10
fungiso basically ignoring the isolation use case and focusing on containers mostly as a service packaging construct?17:12
fungimakes sense if we're sticking them on separate virtual machines already17:12
clarkbfungi: ya it also simplifies things like management tooling (think zuul command line commands)17:12
clarkband it was totally in the spec I just didn't draw the relationship between no network namespacing and iptables concerns until mordred said something about it17:13
*** dpawlik has quit IRC17:13
*** dpawlik has joined #openstack-infra17:14
mordredyup. we already run our services without a network isolation layer and we're designed for that to work properly. adding network isolation doesn't improve anythng for our services, and it turns out actually makes some bits worse :)17:14
clarkbthe one thing it would give us is the ability to pack services on arbitrary ports17:15
clarkbbut I don't think that is a need right now17:15
*** dpawlik has quit IRC17:18
*** diablo_rojo has quit IRC17:19
*** psachin has quit IRC17:21
*** roman_g has quit IRC17:21
*** derekh has quit IRC17:23
*** roman_g has joined #openstack-infra17:23
clarkbcorvus: fungi google says we have opendev.org dns pointed to our dns servers now17:25
clarkbbut looks like we are still waiting on the dnssec glue17:25
corvusclarkb: https://dnssec-debugger.verisignlabs.com/opendev.org  agrees we're waiting on ds records in .org17:26
fungiclarkb: indeed! i checked whois not a couple hours ago and it still had the old nameservers but now it's correct17:26
clarkbreading email sounds like registrar wanted ns/soa records to propogate before the ds went in? I don't get it but at least we have a rough timeline for when that should be done now17:27
*** ykarel|away has quit IRC17:28
corvuswhatever floats their boat :)17:28
corvusthat is nice that all the other checkboxes are green17:29
*** dpawlik has joined #openstack-infra17:29
*** apetrich has quit IRC17:30
*** jcoufal has quit IRC17:31
*** simon-AS559 has joined #openstack-infra17:32
*** dpawlik has quit IRC17:34
*** diablo_rojo has joined #openstack-infra17:35
mordred\o/17:37
*** simon-AS559 has quit IRC17:39
*** agopi|food_run is now known as agopi17:41
*** trown is now known as trown|lunch17:42
*** apetrich has joined #openstack-infra17:50
fungistill no mass complaints of gerrit slowness since i blocked stackalytics-bot-2 with iptables17:52
funginot sure if it was coincidence, tempted to remove the firewall rule and see what happens17:52
clarkbfungi: does stackalytics git log indicate any changes that might expalin it?17:52
*** florianf is now known as florianf|afk17:53
funginot that i can spot at least17:55
*** ijw has joined #openstack-infra17:56
fungii have a feeling they restarted it or it's in a crash-and-restart loop or something which was causing it to re-query the entire change history of every project in review.o.o17:56
clarkbfrickler: gmann: the other item on my list to check on after the long weekend was how the bionic transition with devstack/tempest is going. I think things were looking good after corvus' suggestions were implemented. Anything else we can help review/look at?17:59
clarkbinfra meeting in an hour. I did update the agenda last week :)18:01
*** ijw has quit IRC18:02
*** ahosam has joined #openstack-infra18:03
fricklerclarkb: most things are looking pretty good for bionic now. the situation for the various neutron jobs is still a bit unclear, I'm hoping that slaweq can take a closer look soon18:06
*** bnemec has quit IRC18:06
*** jcoufal has joined #openstack-infra18:06
*** bnemec has joined #openstack-infra18:06
fricklerclarkb: I did find an issue with this keystoneclient job, though, albeit unrelated to bionic. this list shows it is always failing since 11/06 except for one single run (remove the result filter to see the failures) http://zuul.openstack.org/builds?job_name=keystoneclient-devstack-functional&result=SUCCESS18:07
clarkbfrickler: interesting that it is always a retry limit18:08
fricklerclarkb: and I haven't found a change yet that may have triggered the failures18:08
fricklerclarkb: yes, it is failing in pre18:08
fricklerclarkb: first devstack runs and then some additional mysql setup fails18:08
*** bhavikdbavishi has quit IRC18:08
fricklerclarkb: because devstack mangles mysql root account18:08
fricklerclarkb: but I don't know why it passed earlier18:09
*** jpena is now known as jpena|off18:09
clarkbfrickler: looks like the error is table=None in the column update?18:11
fricklerclarkb: oh, wait, for the passing jobs, "18:11
fricklertest-setup : Run tools/test-setup.sh" task is skipped18:11
*** jtomasek has quit IRC18:12
*** graphene has quit IRC18:13
openstackgerritMerged openstack-infra/project-config master: Adding repo and Sotryboard for the Edge Comptuing Group  https://review.openstack.org/62015118:13
fricklerclarkb: no, the typical failure is this one, some look different probably due to errored patches http://logs.openstack.org/39/605539/24/check/keystoneclient-devstack-functional/9fff540/job-output.txt.gz#_2018-11-27_04_39_26_93904118:13
*** graphene has joined #openstack-infra18:14
clarkbfrickler: ah ok. I think the issue there is test-setup.sh is in conflict with devstack. That script is meant for setting things up for tox to run when there is no system like devstack deploying an entire system18:14
clarkbhttp://logs.openstack.org/39/605539/24/check/keystoneclient-devstack-functional/9fff540/ara-report/file/3bbc84cb-a4b0-4753-8bad-78207066cb8d/ is where it comes from18:15
fricklerclarkb: right, but it seems to have been working until three weeks ago18:16
clarkbhttps://git.openstack.org/cgit/openstack-dev/devstack/tree/.zuul.yaml#n455 is the job that pulls it in18:17
clarkbwhich points out you can't interfere with the devstack deployment18:17
*** gary_perkins has quit IRC18:17
fricklerclarkb: ah, the job definition was changed here, I missed that patch earlier https://review.openstack.org/61338518:18
fricklerclarkb: so I'll talk to the keystone folks about that18:18
*** gary_perkins has joined #openstack-infra18:18
clarkbfrickler: the issue seems to be when that job is run against the keystone repo. keystoneclient doesn't have a tools/test-setup.sh but keystone does18:20
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/zuul-jobs master: Add a markdownlint job and role  https://review.openstack.org/60769118:20
clarkbfrickler: this is why the tests passed on the change you found above, but then job fails when run against keyston18:20
fricklerclarkb: exactly, this is why it wasn't noticed by gating18:20
fricklerand this explains why there was still a single job passing later on 19th, which was very confusing for me18:21
*** wolverineav has joined #openstack-infra18:24
fricklerclarkb: the question is: can we pin that role to use tools/test-setup.sh from the project which defines the job instead from the project the job is being run against? that might be more in line with the intended use18:25
*** ykarel|away has joined #openstack-infra18:27
*** simon-AS559 has joined #openstack-infra18:28
clarkbfrickler: yes I agree test-setup.sh seems coupled to the repo the test is in? I believe that role comes from zuul jobs so we may need to make that optional behavior or provide an alternative role18:28
clarkbcorvus: ^ fyi18:28
*** rlandy is now known as rlandy|brb18:28
fricklerclarkb: it uses zuul_work_dir as reference, so that is the project the job is being run against18:29
fricklerclarkb: not sure how one would reference the project where the job is defined, is that even possible currently?18:30
*** shoogz has quit IRC18:30
*** bobh has quit IRC18:32
*** simon-AS559 has quit IRC18:32
clarkbfrickler: I think the role itself may know what its path is? though on the remote node that path is different18:32
*** simon-AS559 has joined #openstack-infra18:32
*** lpetrut has quit IRC18:33
*** dpawlik has joined #openstack-infra18:37
*** armax has joined #openstack-infra18:37
*** simon-AS559 has quit IRC18:39
fricklerclarkb: actually one would need to dissect the last item in zuul._inheritance_path . not sure whether that's feasible though. probably easier to add an option to explicitly specify a fixed project to use per job18:40
corvusdon't use _inheritance path18:41
corvusi'm confused -- can someone succinctly state the problem?18:41
clarkbcorvus: the tox job paths invoke tools/test-setup.sh if that script exists in the current project repo. The problem is when a job that invokes tox with test-setup.sh is used as an integration job and invokes test-setup.sh on the wrong project18:42
*** dpawlik has quit IRC18:42
clarkbcorvus: specifically this job works against python-keystoneclient (where the job is defined) because there is no test-setup.sh but fails when run against keystone because teh script exists there and conflicts with devstack18:43
corvusclarkb: it sounds like the job should not use the test-setup role then18:43
corvusit should just use the tox role alone18:44
clarkbya that may be the fix for this particular case since keystoneclient doesn't have a test-setup.sh18:44
clarkbhowever if it did, how would we address that? presumably the test-setup.sh should run from where the job is defined? I guess that isn't necessarily always going to be the case18:44
corvusclarkb: no i think this is the general solution :)18:44
clarkb(and maybe that is an indication that that setup should be in ansible roles/playbooks and not in the ease of use script)18:45
corvusclarkb: if you don't want to run test-setup, don't have the job run test-setup :)18:45
corvusi know i sound like the doctor that says "don't do that if it hurts" but... i think that's the answer :)18:45
clarkbright, but in the case where you do want to run it, how do you select the right test-setup script. And It hink the answer is your job is complex enough now that it should use ansible for that18:45
clarkbrather than use the ease of use feature that is largely intended for unittesting and not integration testing18:46
corvusclarkb: ah, yes, if that were hypothetically the case, yes.18:46
corvusclarkb: probably this job inherits from a tox job, but it shouldn't.  that job does too much.  but we made recomposable roles, so it's easy to make a job which just uses the tox role and omits the test-setup role.18:46
corvusfrickler: ^18:47
clarkbcorvus: it inherits from a special devstack tox job that mordred wrote for testing shade I think18:47
clarkbI think we may want a different special devstack tox job that drops the test-setup.sh role18:47
corvusclarkb: or maybe that job can drop test-setup?  i wonder :)18:48
clarkbI'm guessing mordred is using it for something, but that could also be the case. <-- mordred fyi18:48
fungii concur, tools/test-setup.sh exists mainly so that a developer running tests locally can run that (and bindep) first to work out the correct dev environment. exercising it in unit test jobs makes sense. running it in a multi-repo integration scenario doesn't really18:48
fungithe only other solution which _might_ make sense is for integration test jobs to run the tools/test-setup.sh of *every* project being installed from source (as a workaround for lack of system-level package maintainer scripts)18:50
fungibut this should really be considered as in the same situation as the bindep role18:50
fungieither you want to run it for all projects being installed from source, or none of them (and provide your own mechanism for getting the system package dependencies of your integration test satisfied)18:51
clarkbya I think the none of them and have job figure it out more directly is the ticket for more complex testing18:51
fungime too18:52
*** dpawlik has joined #openstack-infra18:53
fricklero.k., that all sounds reasonable. will look at a solution for the job with lbragstad tomorrow18:53
*** irdr has quit IRC18:54
*** mordred[m] has quit IRC18:54
*** eharney has joined #openstack-infra18:54
*** ykarel|away has quit IRC18:56
*** dpawlik has quit IRC18:57
*** irdr has joined #openstack-infra18:57
*** trown|lunch is now known as trown18:58
*** wolverin_ has joined #openstack-infra19:03
*** wolverineav has quit IRC19:05
*** hongbin has joined #openstack-infra19:05
hongbinhi, i wonder how i can use the !join: tag in zuul job definition. i tried this https://review.openstack.org/#/c/619642/8/.zuul.yaml@19 , but it didn't work19:06
*** ralonsoh has quit IRC19:07
clarkbhongbin: we are in our weekly meeting right now but we can take a look after19:08
hongbinclarkb: ack19:08
*** ahosam has quit IRC19:08
*** rlandy|brb is now known as rlandy19:09
*** roman_g has quit IRC19:12
mordredclarkb, fungi, corvus: I agree with the words said before about the devstack job19:13
*** electrofelix has quit IRC19:25
*** gfidente has quit IRC19:28
*** hughsaunders has joined #openstack-infra19:35
*** shardy has quit IRC19:35
*** dpawlik has joined #openstack-infra19:38
*** wolverin_ has quit IRC19:41
*** wolverineav has joined #openstack-infra19:41
*** wolverineav has quit IRC19:42
*** wolverineav has joined #openstack-infra19:42
openstackgerritMerged openstack-infra/nodepool master: Update devstack test to Fedora 28  https://review.openstack.org/61437519:44
*** bawz has quit IRC19:47
*** bawz has joined #openstack-infra19:49
*** wolverineav has quit IRC19:53
*** bawz has quit IRC19:54
logan-hongbin: http://paste.openstack.org/raw/736134/ might work19:54
*** aojeagarcia has quit IRC19:55
hongbinlogan-: cool, will try that, thx19:55
openstackgerritMerged openstack-infra/puppet-ansible master: Fix template variables for puppet 4  https://review.openstack.org/58569219:56
*** irdr has quit IRC19:56
*** irdr has joined #openstack-infra19:59
clarkbianw: my hunch is that glean itself guards against this somewhat so falling through is mostly safe20:00
clarkbianw: but we should probably add another ConditionPathExists for the debuntu path as well20:00
ianwclarkb: yeah, my original intuition that it runs every time on xenial->onwards may have been correct20:00
clarkbhongbin: logan-: fwiw google seems to think that !join is a cloudforms thing? I don't think that serialization method is valid in zuul or ansible (we use safe load) so you'll have to use teh ansible jinja constructs instead. But I'm not sure if those will be passed through to ansible in a way that makes sense from the zuul config20:02
*** simon-AS559 has joined #openstack-infra20:02
clarkbhongbin: logan- certainly worth a try, but that config is zuul side and passed to ansible as variables and I'm not sure ansible will jinja at that point or not20:03
*** simon-AS5591 has joined #openstack-infra20:04
*** bawz has joined #openstack-infra20:04
ianwclarkb: the other thing is, and i'm not sure i'm brave enough to change it, but i think that "Before=network-pre.target" is wrong20:05
clarkbya I'm seeing that in the commit message now. You moved the glean-nm stuff ot happen after local-fs20:06
ianwhttps://cloudinit.readthedocs.io/en/latest/topics/boot.html is pretty good, and we're the equivalent of "local".  that's why i changed it to local-fs20:07
*** simon-AS559 has quit IRC20:07
clarkbshould it be before network-manager isntead? I worry that saying after some early unit can still result in running very late20:07
clarkbthe cloud init docs assert it runs as soon as local-fs happens is that due to the wants?20:07
clarkbbasically what ensures that we don't run after local-fs and also after network manager20:07
*** jcoufal has quit IRC20:09
ianwyeah i think so ... i spent us thanksgiving day with a tunneled VNC connection to a held node trying to figure it all out; it was super frustrating because after i'd break into the devstack dib built vm and reboot it, it would work20:09
*** bawz has quit IRC20:10
ianwbut it consistently works in CI pulled in at that stage, and matches cloud-init which i presume works :)  so that was how i ended up there20:10
clarkbianw: in the case of config drive glean manages the mounting of the config drive anyway right? so we aren't relying on systemd units external to glean to do that for us20:11
ianwthat's correct, glean does that.  there's a change in the glean queue actually to make it work with busybox mount20:11
ianwwhich, as busybox is want to do, just different enough that it doesn't work :)20:12
ianwi wonder if we need the defaultdepenedencies=no too20:13
ianwistr we put that in after some other issue20:13
*** jamesmcarthur has quit IRC20:13
clarkbianw: also comment at https://review.openstack.org/#/c/619120/7/diskimage_builder/elements/simple-init/post-install.d/80-simple-init doesn't seem to have been answered20:15
clarkbiirc red hat distros don't enable services by default, is network manager special there?20:15
*** jtomasek has joined #openstack-infra20:17
*** lennyb_ has joined #openstack-infra20:17
ianwyeah, empirically from CI testing I guess that it is enabled sufficiently.  i think it's really network.service that was special due to it's non-default nature20:18
clarkbianw: this is probably bike shedding but reading the skip_when tests I almostly feel a when: instaed of skip_when: will read better20:20
*** lennyb has quit IRC20:20
clarkbbasically the positive assertion match avoids the double negative of not when != some value20:21
clarkbwhen = some value and when != some value seems easier to reason about? its not terrible though so I don't think its necessary to change20:21
*** jrist has quit IRC20:22
ianwi'm open to that, my thought was that it would be most common to just not install a given package when some flag ... but i think it's probably more useful as a toggle20:24
*** wolverineav has joined #openstack-infra20:24
clarkbI +2'd but left a comment with ^ in it as a suggestion20:24
ianwi'll loop back; i'll see how it looks as a "when", i'll put some comments in the service file, add a follow-on to skip deb interfaces & try moving the phase of the existing service20:26
ianwfor the arch stuff, we have "arch:" and "not-arch:" ... not sure that was a great choice in hindsight.  didn't want to replicate that with the skipping20:27
*** wolverineav has quit IRC20:33
*** wolverineav has joined #openstack-infra20:34
*** slaweq_ has joined #openstack-infra20:44
*** eharney has quit IRC20:45
*** jtomasek has quit IRC20:45
*** rossella_s has quit IRC20:57
*** slaweq_ has quit IRC21:03
*** ahosam has joined #openstack-infra21:04
mordredianw, clarkb: left a comment about the network mode on https://review.openstack.org/#/c/60602821:06
*** xarses has joined #openstack-infra21:12
*** andreww has quit IRC21:14
*** jrist has joined #openstack-infra21:15
*** kgiusti has left #openstack-infra21:23
*** jcoufal has joined #openstack-infra21:28
*** eharney has joined #openstack-infra21:28
*** yamamoto has quit IRC21:29
*** tpsilva has quit IRC21:31
*** wolverineav has quit IRC21:32
*** wolverineav has joined #openstack-infra21:34
*** xek_ has quit IRC21:35
clarkbmordred: did we arrive at a new passwords file process that works with new gpg?21:35
clarkbmordred: and does letting emacs do it for me break it?21:35
* clarkb uses emacs and can be instructed on cleanup if necessary21:36
*** wolverineav has quit IRC21:38
*** wolverineav has joined #openstack-infra21:38
AJaeger_clarkb: there was some gpg-agent -d emacs invocation proposed - hope you find it in backscroll...21:39
clarkbfungi: I just accepted a moderated email to the infra list. For the spam messages like those from qq.com are you rejecting or discarding? And do you add those addresses to the ban list?21:39
AJaeger_clarkb: not sure whether that was the final word on it21:39
mordredclarkb: yes - what AJaeger_ said was the latest thing I know about it21:40
clarkbat least I think I accepted it, not seeing it in the archive or in my inbox yet ...21:40
clarkbah terhe it is21:40
*** pcaruana has quit IRC21:46
*** priteau has quit IRC21:47
openstackgerritMerged openstack-infra/system-config master: Turn on the future parser for eavesdrop.o.o  https://review.openstack.org/59004821:47
*** kjackal has quit IRC21:48
*** trown is now known as trown|outtypewww21:49
*** dpawlik has quit IRC21:54
*** dpawlik has joined #openstack-infra21:55
*** dpawlik has quit IRC22:00
*** slaweq has quit IRC22:03
fungiclarkb: discarding22:03
fungiclarkb: in short, i skim the queue, set any legit-looking messages to approve and then hit the "discard all messages marked as deferred" checkbox and submit22:03
clarkbfungi: thanks22:04
*** xarses_ has joined #openstack-infra22:05
fungii've done it a couple times already today, so hopefully it wasn't overflowing22:06
*** yboaron_ has joined #openstack-infra22:06
*** yamamoto has joined #openstack-infra22:07
*** xarses has quit IRC22:08
clarkbit wasn't, about 10 entries I think22:08
clarkbI'll work on cleaning out the spam now22:08
*** yboaron_ has quit IRC22:10
fungicool, thanks!22:11
*** yamamoto has quit IRC22:18
*** slaweq has joined #openstack-infra22:19
*** slaweq has quit IRC22:24
*** ahosam has quit IRC22:24
openstackgerritMerged openstack-infra/zuul master: Improve resource usage with semaphores  https://review.openstack.org/61900422:34
openstackgerritMerged openstack-infra/zuul master: Display ref instead of NA for time triggered items  https://review.openstack.org/61925222:34
*** boden has quit IRC22:34
openstackgerritMerged openstack-infra/zuul master: Fix manual dequeue of github items  https://review.openstack.org/61927222:35
openstackgerritMerged openstack-infra/zuul master: Retry queries for commits  https://review.openstack.org/57513722:35
*** wolverineav has quit IRC22:36
openstackgerritIan Wienand proposed openstack-infra/glean master: Add NetworkManager distro plugin support  https://review.openstack.org/61896422:37
*** wolverineav has joined #openstack-infra22:37
*** simon-AS5591 has quit IRC22:38
*** wolverineav has quit IRC22:39
*** wolverin_ has joined #openstack-infra22:39
*** rossella_s has joined #openstack-infra22:46
*** bawz has joined #openstack-infra22:48
*** wolverin_ has quit IRC22:49
*** dhineshb has joined #openstack-infra22:49
*** wolverineav has joined #openstack-infra22:50
ianwclarkb: hrm, now i'm looking at a different cloud-init service file22:53
*** eharney has quit IRC22:54
*** priteau has joined #openstack-infra22:54
*** jcoufal has quit IRC22:56
*** rcernin has joined #openstack-infra22:57
*** priteau has quit IRC22:58
*** wolverineav has quit IRC22:59
*** wolverineav has joined #openstack-infra23:02
*** adriant has quit IRC23:07
*** wolverin_ has joined #openstack-infra23:12
*** jamesmcarthur has joined #openstack-infra23:14
*** adriant has joined #openstack-infra23:15
*** wolverin_ has quit IRC23:15
*** wolverin_ has joined #openstack-infra23:15
*** wolverineav has quit IRC23:15
*** jamesmcarthur has quit IRC23:18
*** rossella_s has quit IRC23:23
clarkbre gpg on bridge there is a gpg agent running23:24
clarkbshould I kill the process?23:24
clarkbmordred: fungi ^23:24
ianwyeah, that's what i do23:25
clarkbunrelated to gpg it seems like the tripleo node usage may be reaching steady state around 46.7% node usage.23:25
clarkbI've watched it go over and under that value today23:25
clarkbtotally not scientific, but may indicate we've hit a wall on the node usage improvements there?23:25
clarkbok /me kills the gpg agent on bridge23:25
openstackgerritIan Wienand proposed openstack-infra/glean master: Add NetworkManager distro plugin support  https://review.openstack.org/61896423:26
openstackgerritIan Wienand proposed openstack-infra/glean master: A systemd skip for Debuntu systems  https://review.openstack.org/62042023:26
fungiclarkb: gpg2 basically auto-spawns a persistent agent process. i think there's a configuration option we can add to tell it to only run the agent on demand and destroy it once the command completes23:26
fungibut that's not its defauly23:26
fungidefault23:26
clarkbfungi: ya  Ithink my gpg2 stopped doing it but I didn't change any config I wonder if suse flipped that option23:26
fungiclarkb: so tripleo is no longer >50% of our node usage, but isn't much under so far either?23:27
*** rlandy has quit IRC23:27
clarkbfungi: correct, granted this value includes the original >50% start as well. I'm looking at a month right now and right about a month ago was when we started logging this stuff23:28
clarkblet me see if I can get it to look at the last week23:28
fungiwell, last week is certain to be anomalous anyway23:28
*** slaweq has joined #openstack-infra23:29
clarkbya though should give us a sense of how far the old data is skewing things? or maybe only that turkey week skews things too?23:29
fungihard to say, really23:30
openstackgerritMerged openstack/diskimage-builder master: Make tripleo-buildimage-overcloud-full-centos-7 non-voting  https://review.openstack.org/62020023:32
clarkbfor the last week tripleo is 42%23:33
*** slaweq has quit IRC23:33
clarkbI think its clear their improvements have helped. Mostly interested to see if we've stopped improving though23:34
*** yamamoto has joined #openstack-infra23:34
clarkbthe last two weeks it is 47%23:34
clarkbI'm guessing this week will be least anomolous of the recent data though because no ptg and no US holiday23:35
fungiyeah, same23:37
fungithough we're getting into holidays for europe now too23:37
clarkbthe good news is there is definitely improvement from when we started measuring this23:37
clarkbso thank you for all who helped with that23:38
clarkbnow I'm just going to make sure we don't forget after the initial push :)23:38
*** fuentess_ has quit IRC23:45
*** mriedem has quit IRC23:46
*** armax has quit IRC23:49
*** jhesketh_ is now known as jhesketh23:51
*** armax has joined #openstack-infra23:52
*** diablo_rojo has quit IRC23:57
*** andreww has joined #openstack-infra23:58
*** andreww has joined #openstack-infra23:59
*** hongbin has quit IRC23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!