Wednesday, 2018-04-25

hubbotFAILING CHECK JOBS: gate-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master, gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP.00:21
*** yolanda_ has joined #oooq00:24
*** yolanda__ has quit IRC00:27
*** skramaja has joined #oooq02:04
*** agopi|zzz has quit IRC02:18
hubbotFAILING CHECK JOBS: gate-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master, gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP.02:21
*** jaganathan has joined #oooq02:23
*** agopi|zzz has joined #oooq02:38
*** jaganathan has quit IRC03:47
*** jaganathan has joined #oooq03:47
*** ykarel|away has joined #oooq04:06
*** ykarel|away is now known as ykarel04:10
hubbotFAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, gate-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP.04:21
*** links has joined #oooq04:29
*** pgadiya has joined #oooq04:38
*** pgadiya has quit IRC04:38
*** skramaja has quit IRC04:49
*** ratailor has joined #oooq05:00
*** marios has joined #oooq05:18
quiquell|off!gatestatus05:46
openstackquiquell|off: Error: "gatestatus" is not a valid command.05:46
hubbotFAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP.05:46
*** jfrancoa has joined #oooq05:55
*** quiquell|off is now known as quiquell|rover06:01
quiquell|roverWe have a blue ball at pikes container build job06:01
quiquell|roverLet's wait for the promotion06:01
*** jtomasek has joined #oooq06:05
ykarelquiquell|rover, there are other issues as well in pike06:05
quiquell|roverdamn06:07
quiquell|roverAt least this is verifying already https://review.openstack.org/#/c/563850/06:08
*** jaganathan has quit IRC06:12
hubbotFAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP.06:21
*** jaganathan has joined #oooq06:27
*** kopecmartin has joined #oooq06:39
*** ccamacho has quit IRC06:42
*** ccamacho has joined #oooq06:43
*** quiquell|rover is now known as quique|rover|afk06:49
*** udesale has joined #oooq06:51
*** tesseract has joined #oooq07:01
ykarelcan we get https://review.openstack.org/#/c/563843/ merged, to fix ocata promotion07:03
*** bogdando has joined #oooq07:08
*** udesale has quit IRC07:09
ykarelpanda|ruck|off, quique|rover|afk i have started openstack-periodic for master and queens07:11
ykarelfs035 in master is expected to fail, which would be fixed with revert:- https://review.openstack.org/#/c/564096/07:12
*** holser__ has joined #oooq07:14
*** amoralej|off is now known as amoralej07:20
*** udesale has joined #oooq07:26
*** florianf has joined #oooq07:30
*** quique|rover|afk is now known as quiquell|rover07:30
*** udesale has quit IRC07:33
quiquell|roverykarel:  https://review.openstack.org/#/c/563850/ is still not merged07:33
quiquell|roverperiodic-tripleo-centos-7-master-containers-build will fail07:33
ykarelquiquell|rover, i have added a temp patch for it:- https://review.rdoproject.org/r/#/c/13534/07:34
*** apetrich has quit IRC07:34
*** apetrich has joined #oooq07:35
quiquell|roverykarel: We have to remember to revert that, thats a hacky hack :-)07:38
ykarelquiquell|rover, yup i will take care07:38
ykarelas soon as upstream merges07:39
quiquell|roverykarel: Let's move https://review.openstack.org/#/c/563843/07:41
quiquell|roverOk found No package matching 'python-stestr'07:44
ykarelack07:47
quiquell|roverykarel: Thanks07:47
*** ykarel is now known as ykarel|lunch07:48
quiquell|roverykarel|lunch: https://bugs.launchpad.net/tripleo/+bug/176682207:52
openstackLaunchpad bug 1766822 in tripleo "At periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-ocata No package matching 'python-stestr'" [Critical,In progress] - Assigned to Quique Llorente (quiquell)07:52
quiquell|roverOcata promotion blocker: https://review.openstack.org/#/c/563843/ needs +2 +1w07:54
*** skramaja has joined #oooq08:02
arxcruzadarazs: can you +2 +w https://review.openstack.org/#/c/563843/ ?08:10
*** zoli is now known as zoli|wfh08:14
*** zoli|wfh is now known as zoli08:14
*** lucas-afk is now known as lucasagomes08:16
*** jaosorior has joined #oooq08:16
hubbotFAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP.08:21
*** ykarel|lunch is now known as ykarel08:24
ykarelquiquell|rover, ack08:24
ykarelquiquell|rover, amoralej master containers-build passed with kolla Temp patch08:24
quiquell|roverykarel: We have all the other fixes in place08:25
quiquell|roverLet's see...08:25
amoralejykarel, good08:25
ykarelquiquell|rover, fs035 will fail08:25
amoralejwe should have the patch merged in an hour or so08:25
amoralejhopefully08:25
ykarellet's see others08:25
ykarelamoralej, yeah, we need to revert temp patch post that08:25
quiquell|roverykarel: Do you have around the issue with fs035 ?08:26
ykarelquiquell|rover, yes https://review.openstack.org/#/c/564096/08:26
quiquell|roverykarel: Thanks08:28
*** gkadam has joined #oooq08:32
quiquell|roverykarel: We can close this ?08:49
quiquell|roverhttps://bugs.launchpad.net/tripleo/+bug/176627108:49
openstackLaunchpad bug 1766271 in tripleo "Installing packages: gperftools-libs-2.4.91-1.el7.x86_64.rpm not found" [Critical,Triaged] - Assigned to Alfredo Moralejo (amoralej)08:49
ykarelquiquell|rover, yes08:50
quiquell|roverI think the mirroring is ok now08:50
*** panda|ruck|off is now known as panda|ruck08:54
panda|ruckmorning08:54
panda|ruckwhat did I miss ?08:54
quiquell|roverpanda|ruck: Etherpad updated08:57
quiquell|roverAll news are there08:57
*** zoli is now known as zoli|brb09:03
panda|ruckykarel: you started master and queens manually ?09:04
ykarelpanda|ruck, rescheduled:- https://review.rdoproject.org/r/#/c/13535/09:05
panda|ruckykarel: because of the outage yesterday or to test the latest patches merged ?09:06
ykarelpanda|ruck, to test the latest merged patches09:07
quiquell|roverpanda|ruck: I am going to wait to test overlay2 until promotions are fixed09:10
quiquell|roverSo we have some09:10
quiquell|roverPromoter is doing the job now09:11
*** moguimar has joined #oooq09:18
quiquell|roverpanda|ruck: Can you do some reviewing at the ocata promotion blocker ? https://review.openstack.org/#/c/563843/09:19
*** moguimar has quit IRC09:23
panda|ruckquiquell|rover: approved09:23
*** dtantsur|afk is now known as dtantsur09:28
quiquell|roverpanda|ruck: Thanks09:28
*** moguimar has joined #oooq09:32
*** zoli|brb is now known as zoli09:36
*** zoli is now known as zoli|wfh09:36
quiquell|roverpanda|ruck: Could you also do some reviewing here ? https://review.openstack.org/#/c/558369/09:38
quiquell|roverTo facilitate rover duties09:38
*** tcw has quit IRC09:39
*** tcw has joined #oooq09:41
ykarelchandankumar, any idea why fs27 failing: https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-singlenode-featureset027-master/048baba/undercloud/home/jenkins/tempest_output.log.txt.gz09:59
panda|ruckadarazs: added a comment in https://review.rdoproject.org/api/13460 , also it seems you have some conflics on your patches10:02
adarazspanda|ruck: checking.10:02
chandankumarykarel: it is failing while listing tests https://github.com/openstack/heat-tempest-plugin/commit/b7011c73e69e3ad8ff3ad23dd911e09f8fce86c110:02
chandankumarmight be the probable fix10:03
chandankumarykarel: it should be available in heat-tempest-plugin rpm10:04
chandankumarykarel: what about storing the rpm -qa output in tempest.log?10:05
chandankumarfor tempest plugins10:05
chandankumarykarel: this commit is picked at this time 2018-04-25 03:35:1610:06
chandankumarykarel: we need to rebuild the containers i think to pick the changes?10:06
hubbotFAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP.10:21
ykarelchandankumar, python2-heat-tests-tempest.noarch 0:0.1.1-0.20180424213819.b7011c7.el7.centos was in the build container10:23
ykarelchandankumar, i can see the same failures in queens as well10:32
ykarelmay be the mentioned commit is the problem10:33
ykarelhttps://github.com/openstack/heat-tempest-plugin/commit/b7011c73e69e3ad8ff3ad23dd911e09f8fce86c110:33
*** jaganathan has quit IRC10:34
panda|ruckadarazs: about to approve https://review.rdoproject.org/api/13356, but it conflicts with https://review.rdoproject.org/api/13445 which is also approved at this point10:38
adarazspanda|ruck: these 5 changes are based on each other.10:39
adarazspanda|ruck: they shouldn't be conflicting.10:39
adarazspanda|ruck: it's one long chain of changes.10:39
panda|ruckadarazs: gerrit disagrees, but anyway, everything approved, except for the doc10:40
ykarelchandankumar, or old gabbi is the problem10:40
*** ykarel is now known as ykarel|away10:42
adarazspanda|ruck: I fixed the docs typo in the last change instead: https://review.rdoproject.org/r/1348310:42
adarazspanda|ruck: indeed gerrit is doing something weird, I'm trying to untangle things with rebases.10:45
*** jaganathan has joined #oooq10:47
quiquell|roverarxcruz: What's the repository of the config_tempest.py ?10:47
quiquell|roverchandankumar: ^10:47
adarazspanda|ruck: can you +2 this if there's no other problem than the s/gatecheck/gatestatus/? https://review.rdoproject.org/r/1346010:49
*** udesale has joined #oooq10:51
arxcruzquiquell|rover: github.com/openstack/python-tempestconf10:52
chandankumarykarel|away: might be10:53
*** udesale has quit IRC10:59
*** ykarel|away is now known as ykarel10:59
panda|ruckadarazs: do you remember why we are requesting a node here https://ci.centos.org/job/tripleo-quickstart-promote-queens-rdo_trunk-minimal/29/console even if the quickstart is a noop ?10:59
chandankumarykarel: how can i make sure that heat-tempest-plugin commit is there in container?11:00
ykarelchandankumar, from container build jonb11:00
adarazspanda|ruck: isn't the promotion step with installing dlrnapi runs on that node?11:01
ykarelchandankumar, https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-centos-7-master-containers-build/d02cd70/kolla/logs/tempest.log11:01
panda|ruckadarazs: as far as I see it install dlrnapu, quickstart, and test it, but doesn't seem to need a node11:01
adarazspanda|ruck: then probably we just reused the previous templates for the shell part that requests an ode.11:02
ykarelpanda|ruck, in phase1 we test minimal deployment on libvirt host(requested from cico)11:02
chandankumarykarel: that commit is there in container11:03
chandankumarbuild log11:03
ykarelyes11:03
panda|ruckykarel: yeah ok, it's a full deploy11:07
*** lucasagomes is now known as lucas-hungry11:14
quiquell|rover!gatestatus11:15
openstackquiquell|rover: Error: "gatestatus" is not a valid command.11:15
hubbotFAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal | check logs @ https://review.openstack.org/472607 and fix them ASAP.11:15
*** moguimar has quit IRC11:15
*** atoth has joined #oooq11:18
*** zoli|wfh is now known as zoli|lunch11:19
quiquell|roverarxcruz: I am debugging jenkins-tripleo-quickstart-extras-gate-newton-delorean-full-minimal11:22
quiquell|rovertempest.conf.gz and tempest-deployer-input.conf.gz have no identity.uri11:23
chandankumarquiquell|rover: is the question is for newton job?11:26
quiquell|roverchandankumar: Yep, trying to debug them11:27
chandankumarquiquell|rover: then config_tempest.py is coming from https://github.com/redhat-openstack/tempest/tree/newton/tools11:27
*** moguimar has joined #oooq11:28
chandankumarquiquell|rover: here is the bug https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/templates/configure-tempest.sh.j2#L12311:28
quiquell|roverOk, I will change and add a review11:29
quiquell|roverSo we can get cleaner getstatus11:29
quiquell|roverThanks a lot chandankumar11:29
chandankumarquiquell|rover: review coming11:31
quiquell|roverUps was pushing it11:31
quiquell|roverpanda|ruck: Give a review to this so we clean the getstatus a little11:36
quiquell|roverhttps://review.openstack.org/#/c/56416711:36
arxcruzquiquell|rover: yes, i'm aware, i know how to fix it11:37
quiquell|roverarxcruz: chandankumar pushed a fix already, let's get a review11:38
arxcruzand chandankumar fix it11:38
arxcruzand the configure-tempest is again getting full of 'if release in'11:39
arxcruz:(11:39
quiquell|roverit need per release includes11:39
quiquell|roveror similar11:40
arxcruzchandankumar: python-tempestconf complains if there's a identity.uri in master?11:40
arxcruzor something like that ?11:40
chandankumararxcruz: nope11:40
arxcruzif we are using queens and pass identity.uri will it fail ?11:40
chandankumarmaster tempestconf will work only with master11:40
chandankumararxcruz: for queens if we are sourceing overcloudrc v3 it will not complain11:41
chandankumartill ocata it complains11:41
chandankumarsorry11:41
arxcruzso, might get warnings in tempestconf instead of a lot of ifs here11:41
chandankumararxcruz: if we pass identity.uri to queens it will not complain11:41
arxcruzchandankumar: what i mean is, passing or not identity.uri, we should check in python-tempestconf and set this properly11:42
*** udesale has joined #oooq11:42
chandankumararxcruz: https://github.com/openstack/python-tempestconf/blob/master/config_tempest/main.py#L92 this one got cleaner11:42
arxcruzinstead of have to pass this manually11:42
chandankumararxcruz: yup, we are planning for the same11:42
chandankumararxcruz: a good chunk of code was refactored11:43
chandankumararxcruz: we will be getting rid of most of the items https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/templates/configure-tempest.sh.j2#L12211:43
chandankumarin new tempestconf release11:43
chandankumarand may be next sprint we will be working on that11:44
arxcruzchandankumar: great11:44
arxcruzchandankumar: yay, python code! :D11:45
*** amoralej is now known as amoralej|lunch11:48
*** quiquell|rover is now known as quique|rover|lch11:50
*** udesale has quit IRC11:51
*** udesale has joined #oooq11:52
*** ratailor has quit IRC11:52
panda|ruckI see this in master OVB https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/f87c10b/undercloud/home/jenkins/overcloud_prep_images.log.txt.gz#_2018-04-25_09_11_1911:54
panda|ruck2018-04-25 09:11:19 | cp: cannot create regular file ‘/var/lib/ironic/httpboot/agent.kernel’: No such file or directory11:54
panda|ruck2018-04-25 09:11:19 | Command 'sudo cp -f "./ironic-python-agent.kernel" "/var/lib/ironic/httpboot/agent.kernel"' returned non-zero exit status 111:54
panda|ruckis it known ?11:54
panda|ruckykarel: have you seen this ?11:54
panda|ruckif not I'll open a bug after program meeting11:55
ykarelpanda|ruck, yes11:55
panda|ruckykarel: is there a bug already ?11:55
ykarelpanda|ruck, https://review.openstack.org/#/c/564096/11:56
ykarelabout to merge11:56
ykarelIt's affecting all places where undercloud is not containerized and overcloud_prep_images is executed11:57
panda|ruckok, no bug attached11:57
panda|ruckI'll creat e it anyway to keep track11:57
ykarelOkk11:57
panda|ruckykarel: when did you rescheduled promotion run ?11:57
ykarelit's in next 1 hour11:58
ykarel13:00 UTC11:58
ykarelif tripleoclient and heat-tempest-plugin fixes do not make in next 1 hour there is no benefit of running promotion jobs11:59
ykarelpanda|ruck, create bug for heat-tempest-plugin as well12:01
ykarelohh no, gate reset again12:03
ykarelpanda|ruck, is there a way to check what made gate jobs reset12:04
panda|ruckykarel: usually it's when one of the patches fail12:05
panda|ruckykarel: everything after that patch in zull is reset12:06
ykarelyes, can we find that patch12:06
panda|ruckykarel: I have it, but I'm not sure it has been the same all the time, also, zull removes it on the reset12:06
panda|ruckthat's why it resets it, to retest without it12:06
ykarelOkk12:06
panda|ruckykarel: it's a patch in tripleo-validation12:06
ykarelOk checking12:07
ykarelif that's a real issue or not12:07
panda|ruckykarel: http://logs.openstack.org/82/562282/8/gate/tripleo-ci-centos-7-containers-multinode/07ab872/job-output.txt.gz#_2018-04-25_09_28_03_83339612:08
panda|ruckykarel: this is what I saw in last zuul run12:08
panda|ruckykarel: this is the change https://review.openstack.org/#/c/562282/12:08
*** tosky has joined #oooq12:09
ykarelsome networking issue12:10
ykarel2018-04-25 09:28:03 | ssh: connect to host 158.69.87.210 port 22: Connection timed out12:10
ykarelbut is not related to patch12:11
panda|ruckykarel: http://status.openstack.org/openstack-health/#/?searchProject=tripleo&groupKey=build_name12:12
panda|ruckykarel: performance are dropping12:12
panda|ruckykarel: scenario007 seems to be having soem problem12:14
*** fuzzball81 has joined #oooq12:19
hubbotFAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master | check logs @ https://review.openstack.org/472607 and fix them ASAP.12:21
*** jjoyce has quit IRC12:23
*** atoth has quit IRC12:23
*** tcw has quit IRC12:23
*** jtomasek has quit IRC12:23
*** ykarel has quit IRC12:23
*** yolanda_ has quit IRC12:23
*** jrist has quit IRC12:23
*** chem has quit IRC12:23
*** gchamoul has quit IRC12:23
*** atoth has joined #oooq12:26
*** tcw has joined #oooq12:26
*** jtomasek has joined #oooq12:26
*** ykarel has joined #oooq12:26
*** yolanda_ has joined #oooq12:26
*** chem has joined #oooq12:26
*** jrist has joined #oooq12:26
*** gchamoul has joined #oooq12:26
*** bogdando has quit IRC12:26
*** ykarel is now known as ykarel|away12:27
*** zoli|lunch is now known as zoli12:31
*** zoli is now known as zoli|wfh12:31
*** zoli|wfh is now known as zoli12:31
*** quique|rover|lch is now known as quiquell|rover12:31
*** lucas-hungry is now known as lucasagomes12:33
*** panda|ruck is now known as panda|ruck|lunch12:38
*** amoralej|lunch is now known as amoralej12:40
*** ykarel|away has quit IRC12:42
quiquell|roverSomeone has problems with TASK [undercloud-setup : Test ping across vxlan tunnel (and optionally restart ovs)]12:49
quiquell|roverRunning a reproducer in RDO ?12:49
panda|ruck|lunchquiquell|rover: not with the reproducer, but some upstream job is hitting problems in the same group of tasks12:50
quiquell|roverdamn12:50
quiquell|roverDo you have the job ?12:51
myoungquiquell|rover: I think trown|outtypewww has been tracking that issue as well, specifically reproducing multinode jobs on RDO cloud, with issues with vxlan tunnels (afaik)12:51
quiquell|roverFeels like ingress problems at undercloud12:51
quiquell|roverwith ICMP at least12:51
myoungquiquell|rover, panda|ruck|lunch: this was a persistant issue (I had thought) with upgrade iterative development, using RDO cloud reproducers for multinode jobs12:51
myoungthat motivated the current sprint around libvirt reproducers (in part)12:52
quiquell|rovermyoung: Thanks will take a look12:52
*** Goneri has joined #oooq12:55
*** ykarel|away has joined #oooq13:05
*** ykarel|away is now known as ykarel13:06
*** trown|outtypewww is now known as trown13:11
trownquiquell|rover: it is only on keystone-only13:12
trownquiquell|rover: https://review.openstack.org/#/c/564050/ is fix13:12
quiquell|rovertrown: Running a reproducer on a not keystone only qucikstar bug13:13
quiquell|roverGet the same13:13
quiquell|rovertrown: It's a multinode-1ctlr-featureset03613:15
quiquell|roverGoing to test the patch though13:15
trownquiquell|rover: ya might be same root cause13:18
*** udesale has quit IRC13:19
quiquell|roverPatched let'se see, thanks13:19
*** panda|ruck|lunch is now known as panda|ruck13:20
*** zoli is now known as zli|afk13:21
*** zli|afk is now known as zoli|afk13:21
quiquell|roverWhat happends if you have refspec in the zuul_changes for the same repo but different branches ?13:26
quiquell|roverIt tries to merge all of them ?13:26
trownI doubt good things will happen13:27
quiquell|rovertrown: Feels like it13:28
*** skramaja has quit IRC13:43
*** dtantsur is now known as dtantsur|brb13:53
*** trown is now known as trown|brb14:03
panda|ruckquiquell|rover: just opened https://bugs.launchpad.net/tripleo/+bug/1766884 but really not sure what to do, seems completely random14:05
openstackLaunchpad bug 1766884 in tripleo "Gate jobs are failing in different ways related to network issues" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)14:05
panda|ruckquiquell|rover: gates are going to reset again, and queue will increase14:06
quiquell|roverpanda|ruck: maintenance ?14:06
panda|ruckquiquell|rover: also, the patch to fix promotion has not been merged probably, and will take a lot of time14:06
panda|ruckquiquell|rover: maintenance what ?14:06
quiquell|rover"panda|ruck: maintenance ?14:07
quiquell|roverDamn sorry14:07
quiquell|rover" panda|ruck | quiquell|rover: gates are going to reset again"14:07
quiquell|roverThat's it14:07
quiquell|roverWhy are they going to be reset ?14:07
*** trown|brb is now known as trown14:07
panda|ruckquiquell|rover: are you familiar with how zuul works ? I can offer a brief explanation14:08
quiquell|roverpanda|ruck: read the stuff about it14:09
quiquell|roverI have found this week that the gate queue was huge14:09
quiquell|roverpanda|ruck: I am having the same problem at one reproducer14:10
panda|ruckquiquell|rover: bj ? I have escalation in 20 minutes14:11
quiquell|roverhttps://bluejeans.com/789106523214:13
quiquell|roverpanda|ruck: ^14:14
hubbotFAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master | check logs @ https://review.openstack.org/472607 and fix them ASAP.14:21
quiquell|roverlsfa14:29
panda|rucklzh14:30
quiquell|roverkeyboard fart14:30
*** zoli|afk is now known as zli|afk14:30
*** zli|afk is now known as zoli|wfh14:30
myoung%gatestatus14:31
hubbot`All check jobs are working fine on stable/queens, master.14:31
myoung!gatestatus14:32
openstackmyoung: Error: "gatestatus" is not a valid command.14:32
hubbotFAILING CHECK JOBS: gate-tripleo-ci-centos-7-container-to-container-upgrades-master-nv, tripleo-quickstart-extras-gate-newton-delorean-full-minimal, gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master | check logs @ https://review.openstack.org/472607 and fix them ASAP.14:32
quiquell|roverpanda|ruck: The vxlan problem is different from mine14:35
quiquell|roverThe upstream infra14:36
quiquell|rover2018-04-25 09:19:31 | + sudo ovs-vsctl --may-exist add-port br-ex br-ex_158.69.87.210 -- set interface br-ex_158.69.87.210 type=vxlan options:remote_ip=158.69.87.210 options:key=1000003 options:local_ip=158.69.87.23514:36
quiquell|rover2018-04-25 09:19:31 | + remote_command 158.69.87.210 'source /opt/stack/new/devstack/functions-common;                             install_package openvswitch openstack-selinux;                             restart_service openvswitch'14:36
quiquell|rover2018-04-25 09:19:31 | + local 'ssh_opts=-tt -o PasswordAuthentication=no -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o ConnectionAttempts=4'14:36
quiquell|rover2018-04-25 09:19:31 | + local dest_host=158.69.87.21014:36
quiquell|rover2018-04-25 09:19:31 | + shift14:36
quiquell|rover2018-04-25 09:19:31 | + ssh -tt -o PasswordAuthentication=no -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o ConnectionAttempts=4 158.69.87.210 'source /opt/stack/new/devstack/functions-common;                             install_package openvswitch openstack-selinux;                             restart_service openvswitch'14:36
quiquell|rover2018-04-25 09:28:03 | ssh: connect to host 158.69.87.210 port 22: Connection timed out14:36
*** dtantsur|brb is now known as dtantsur14:51
myoungadarazs, trown: regarding the stable branch patches for the bot, yesterday I had captured that they were for THT:{pike, ocata}, was that supposed to be (instead) TQE?14:57
adarazsmyoung: we don't have TQE stable branches14:57
adarazsmyoung: so yeah, THT is a good target.14:57
* myoung has upstream and internal CI wires crossed14:57
adarazsmyoung: https://review.openstack.org/560452 -- here's the one on queens.14:58
panda|ruckmyoung: you should start using 802.1114:58
adarazssome thing should work.14:58
*** atoth has quit IRC15:01
*** tcw has quit IRC15:01
*** jtomasek has quit IRC15:01
*** yolanda_ has quit IRC15:01
*** jrist has quit IRC15:01
*** chem has quit IRC15:01
*** gchamoul has quit IRC15:01
*** tosky has quit IRC15:01
*** kopecmartin has quit IRC15:01
*** jbadiapa has quit IRC15:01
*** sshnaidm|off has quit IRC15:01
*** matbu has quit IRC15:01
*** panda|ruck has quit IRC15:01
*** quiquell|rover has quit IRC15:01
*** adarazs has quit IRC15:01
*** ChanServ has quit IRC15:01
*** quiquell|rover has joined #oooq15:01
*** adarazs has joined #oooq15:01
*** ChanServ has joined #oooq15:01
*** barjavel.freenode.net sets mode: +o ChanServ15:01
*** atoth has joined #oooq15:02
*** tcw has joined #oooq15:02
*** jtomasek has joined #oooq15:02
*** yolanda_ has joined #oooq15:02
*** chem has joined #oooq15:02
*** jrist has joined #oooq15:02
*** gchamoul has joined #oooq15:02
*** bandini has quit IRC15:03
*** tosky has joined #oooq15:04
*** kopecmartin has joined #oooq15:04
*** jbadiapa has joined #oooq15:04
*** sshnaidm|off has joined #oooq15:04
*** matbu has joined #oooq15:04
*** panda|ruck has joined #oooq15:04
adarazshuh, what a netsplit15:04
*** weshay has quit IRC15:04
*** dtrainor has joined #oooq15:05
*** bandini has joined #oooq15:05
*** weshay has joined #oooq15:06
panda|ruckadarazs: ?15:08
adarazspanda|ruck: half the people on the channel left and rejoined as far as I saw with netsplit messages15:11
panda|ruckok, repeating question15:12
panda|ruckadarazs: are you going to work on https://trello.com/c/iWAz4ONC/718-hubbot-bot-trailing-misc-items-stemming-from-discussion ?15:12
adarazspanda|ruck: no, myoung is taking that.15:12
adarazsgoing to help if necessary :)15:12
panda|ruckmyoung: do you have time for a chat ?15:14
myoungpanda|ruck: i do, in a few, juggling atm and about to have an item in #rdo meeting15:15
*** quiquell|rover is now known as quiquell|off15:19
panda|ruckarxcruz: pping15:19
adarazspanda|ruck, myoung: I will move over the "Karma" plugin from the old hubbot to the new hubbot`... I hacked in a bit of more cleverness about trying to catch karma changes inline like this: myoung++15:20
hubbotadarazs: myoung's karma is now 915:20
adarazsdidn't publish it anywhere, but at least it will keep working :)15:20
* myoung isn't sure he deserves any karma this morning15:20
panda|ruckarxcruz: in  https://review.openstack.org/560006, it's ok if test_white_regex is an empty string ?15:21
panda|ruckarxcruz: in roles/validate-tempest/defaults/main.yml line 815:21
panda|rucksorry, this is from chandankumar15:22
chandankumarpanda|ruck: if it is empty no test will run15:23
chandankumarhttps://review.openstack.org/#/c/560006/6/roles/validate-tempest/defaults/main.yml@815:23
arxcruzpanda|ruck: pong15:24
chandankumarby default it will trigger smoke tests15:24
*** rlandy has joined #oooq15:25
panda|ruckchandankumar: with this syntax, the default will be used only when test_regex is not defined. If it's defined by empty, the default is not applied15:25
chandankumarif test_white_regex is empty we need to pass test names in tempest_whitelist15:25
panda|ruckchandankumar: also, in https://review.openstack.org/560374, how do you tell what to skip for undercloud, and whot for overcloud ? or the list is the same for both ?15:27
chandankumarpanda|ruck: we need to use tempest_blacklist option15:27
chandankumarin the skip file15:27
chandankumarhttps://review.openstack.org/#/c/560374/15:27
*** hubbot has quit IRC15:29
*** hubbot` is now known as hubbot15:29
adarazshubbot is dead, long live hubbot :P :)15:29
hubbotadarazs: Error: "is" is not a valid command.15:29
adarazsmyoung: I shut down the instance on the virtblade machine./15:29
chandankumarpanda|ruck: in skip file there we need to put undercloud to true15:29
adarazs%repeat gate-nag 7200 gatestatus15:31
hubbotAll check jobs are working fine on stable/queens, master.15:31
adarazs%printusers15:31
hubbotCurrent userFilter: []; all users and comments within the timeLimit: zuul (6), rdo-ci (7), rdothirdparty (4)15:31
adarazs%config plugins.GateStatus.userFilter zuul rdo-ci rdothirdparty15:31
hubbotadarazs: The operation succeeded.15:31
adarazs%gatestatus15:31
hubbotFAILING CHECK JOBS on stable/queens: openstack-tox-pep8, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/560452, master: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master @ https://review.openstack.org/56044515:31
panda|ruckchandankumar: ok, one change at a time, starting with the first question15:32
myoungpanda|ruck: I have time now, or anytime today until ~9pm your time15:43
panda|ruckmyoung: ok joining your channel15:45
rlandytrown: your snapshot review/instructions worked for me - restored the nodes from snapshot after full deploy.15:46
rlandywrt README, I was looking for the snapshots using virsh15:47
rlandyto prevent others making the same mistake, we should doc t just look in the images dir15:47
*** kopecmartin has quit IRC16:01
*** moguimar has quit IRC16:01
rlandyarxcruz; look ok to you? https://review.openstack.org/#/c/563589/2?16:03
arxcruzrlandy: yup16:03
rlandyarxcruz; ok - I +1 voted waiting on your expertise16:04
*** holser__ has quit IRC16:05
panda|ruckrlandy: do you have 5 minutes for me today ?16:06
*** atoth_ has joined #oooq16:06
rlandypanda|ruck: yep - I retested with supersede - seemed ok afterwards16:07
rlandypossibly a mirror issue16:07
rlandysomething else to discuss?16:07
*** florianf has quit IRC16:07
*** marios has quit IRC16:07
*** atoth has quit IRC16:09
*** links has quit IRC16:09
panda|ruckrlandy: oh, ok, it was that, also wanted to check on load and status of the integration, if you thinkwe are on track to start iterative testing on friday.16:09
*** holser__ has joined #oooq16:10
rlandypanda|ruck: yes ... referring to cards ...16:10
rlandyhttps://trello.com/c/CF0NQn9t/703-create-a-different-path-for-the-nodepool-setup-role-to-generate-a-generic-clean-nodepool-environment16:10
rlandyhas no QE yet16:10
rlandycould sign up if nobody else will take it16:11
rlandytested the shapshot stuff last night16:11
rlandyreproducer review is in16:11
rlandyfs003 works16:11
rlandyfs010 is tested all the time16:11
rlandyDNS stuff was merged16:11
rlandyand trown tested fs03716:11
*** holser__ has quit IRC16:11
rlandyI am retesting that now with his review16:11
panda|ruckrlandy: that card has a change that is awaiting review, so go for it.16:12
rlandypanda|ruck: ok - will look at ter I confirm fs037 review16:12
panda|ruckrlandy: so you're testing the integration with also the snapshotting on the side ?16:12
rlandypanda|ruck: can you define integration?16:12
rlandyI am testing with the reproducer16:12
*** bandini has quit IRC16:12
panda|ruckrlandy: just using the first word on the card https://trello.com/c/UnueoxJM/688-integrate-libvirt-into-the-existing-reproducer-scripts-so-we-have-just-one-if-workable16:13
rlandypanda|ruck: ack - the review is in16:13
rlandyand I am using that code16:13
panda|ruckrlandy: perfect, thanks.16:13
rlandyif someone els ewants to test16:13
rlandythat would be good since I am developer on that16:13
rlandypanda|ruck: wrt testing cards, trown and I are using those fs all the time, we just left the cards unassigned in case someone else wanted to test16:15
*** bandini has joined #oooq16:15
rlandyif they don't have owners by the end of sprint you can assign us16:15
rlandywe worked on them16:15
*** moguimar has joined #oooq16:17
panda|ruckrlandy: ok, I think friday I'll move them in progress with all the members on the sprint, so we make sure everyone is getting consistent results16:18
rlandypanda|ruck: sounds good16:18
rlandypanda|ruck: one thing that is worrying me about reproducer is that ZUUL_CHANGES are only incorporated in the last step16:21
rlandyeven though we define ZUUL_CHANGES upfront for tq/tqe/tripleo-ci16:22
rlandyI don;t see those changes incorporated until we run toci-gatetest16:22
rlandyin the playbooks we run,I don't see zuul changes picked up16:23
panda|ruckrlandy: the playbooks we run should not need to look at ZUUL_CHANGES16:25
rlandypanda|ruck: they do when the changes are in the playbook and roles we run to set up16:25
*** jaosorior has quit IRC16:25
rlandyie: I am adding those changes through git fetch16:26
panda|ruckok, to test ourselves16:26
rlandypanda|ruck: see comment ...16:26
rlandy# Export ZUUL_CHANGES in the Ansible host if there are changes in16:26
rlandy# tripleo-quickstart, tripleo-quickstart-extras or tripleo-ci repos16:26
rlandy# before running playbooks16:26
rlandyso - if you do, nothing picks those changes up16:26
rlandyside point for our sprint16:27
rlandybut we need to fix that16:27
*** lucasagomes is now known as lucas-afk16:31
panda|ruckyeah, that is a problem for the reproducer itself, not only for the libvirt reproducer, we just clone tq and tqe16:31
panda|ruckwho tests the tester ?16:31
rlandymaybe I can put up a quicks review16:35
panda|ruckrlandy: added the card https://trello.com/c/caCmSItG/720-add-tq-and-tqe-as-changes-for-the-reproducer-to-test. Out of scope for now.16:36
*** trown is now known as trown|lunch16:36
*** panda|ruck is now known as panda|ruck|off16:39
rlandymyoung: FYI ... https://review.openstack.org/#/c/564282/ - I know this is part of your cleanup the work but these things are driving me crazy in my own testing16:50
rlandywe can kill that review when you have your patches up16:50
*** amoralej is now known as amoralej|off16:51
*** zoli|wfh is now known as zoli|gone16:51
*** zoli|gone is now known as zoli16:51
myoungrlandy cool i'm working on that card this afternoon16:54
*** ykarel has quit IRC17:00
*** links has joined #oooq17:08
*** tcw has quit IRC17:09
*** links has quit IRC17:09
*** gkadam has quit IRC17:10
*** jfrancoa has quit IRC17:10
*** tosky has quit IRC17:25
*** tosky has joined #oooq17:26
hubbotFAILING CHECK JOBS on stable/queens: openstack-tox-pep8, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/560452, master: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master @ https://review.openstack.org/56044517:31
*** ykarel has joined #oooq17:36
*** tcw has joined #oooq17:41
*** ykarel has quit IRC17:42
*** dtantsur is now known as dtantsur|afk17:43
*** atoth_ has quit IRC17:46
*** hubbot has quit IRC17:55
*** hubbot has joined #oooq17:56
*** atoth_ has joined #oooq17:58
myoungpanda|ruck|off, rlandy, weshay, trown|lunch: I'm not enabled for the tripleo-infra project...could one of you please ping me?18:04
myoung(or anyone with rdo cloud :: tripleo-infra access) - no hubbot at the moment18:07
*** Goneri has quit IRC18:18
rlandymyoung: ping18:30
rlandywhat do you need?18:30
myoungrlandy: 2 things...one urgent.  urgent = reboot of hubbot VM 192.168.100.4318:37
myoungit's on the rdocloud tripleo-infra tenant18:37
rlandymyoung: you have no access to that tenant?18:38
myoungi do not18:39
myoungthat's the other thing i need :)18:39
*** trown|lunch is now known as trown18:39
rlandymyoung: hard reboot? soft reboot?18:39
myoungi attempted soft by running reboot as root, it's hung so guessing a hard / power on18:40
rlandyhard rebooting now18:40
*** hubbot` has joined #oooq18:41
rlandymyoung: wrt getting access, you need to ask rdo-infra18:41
rlandymyoung: hubbot should be ready for login18:41
myoung192.168.100.43 still isn't responding to pings from jumphost18:42
rlandylet me check18:43
*** hubbot has quit IRC18:44
rlandyhmm - no response18:44
rlandysame as before?18:44
myoungaye18:44
myoungI'm on the jumphost now18:44
myoungoh i'm silly18:45
myoungit's back...ICMP must be blocked18:45
myoungthx18:45
myoung%gatestatus18:45
hubbot`FAILING CHECK JOBS on stable/queens: openstack-tox-pep8, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/560452, master: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master @ https://review.openstack.org/56044518:45
rlandymyoung:ping is still not responding for me18:47
myoungrlandy: aye but can ssh into it18:47
myoungping port blocked guessing?  (checking)18:47
rlandylol18:47
rlandycheck sec groups :)18:47
*** myoung is now known as myoung|dr19:03
*** atoth_ has quit IRC19:03
rlandytrown: for the snapshot option in the reproducer, what is the requirement>19:07
rlandyjust to create a snapshot before running deploy?19:07
rlandyor the toci-gatetest step19:08
rookweshay myoung|dr - trying to get parity between browbeat collectd and upstream collectd https://review.openstack.org/#/c/562813/19:08
rooknot sure if you guys want to weigh in.19:08
trownrlandy: I think running it before we ssh in and run toci19:08
rookthis initial commit is to just get the right plugins in... step two is to setup collectd on the overcloud.19:08
rlandytrown: yes - trying to define it :)19:08
rlandywhat exactly do we run?19:08
trownrlandy: it would be cool to later add the ability to snapshot before deploy, but that is pretty tricky19:08
trownbecause that is mid CI run19:09
rlandytrown: so I understand ..19:09
rlandywe need to add an option to snapshot before ssh'ing to subnode-0?19:09
rlandythat is all?19:09
rlandyno requirement to restore the snapshot after toci-gate-test step?19:10
trownrlandy: ya I think so19:10
rlandytrown: ok - got it - adding that19:10
rlandyin separate review19:10
rlandytrown: btw +2 on the fs037 review - worked for me19:11
trownrlandy: nice19:13
*** Goneri has joined #oooq19:36
rlandytrown: question about libvirt_volume_path - why did you choose to override this default? https://github.com/openstack/tripleo-quickstart/blob/master/roles/common/defaults/main.yml#L20819:40
rlandyLIBVIRT_VOLUME_PATH="{{ libvirt_volume_path }}"19:41
rlandy^^ if I set that in the reproducer and consider https://review.openstack.org/#/c/564282/1/playbooks/libvirt-fake-nodepool.yml19:42
rlandyit will use the default19:42
rlandytrown: ^^ commented lines in https://review.openstack.org/564332 WIP Add a snapshot option to the reproducer script19:53
trownrlandy: hmm I think I was just copying what was in sagi's poc... the default seems fine for me... but it will be in /tmp in that script which is probably small in your default config box20:11
rlandyhmm ... /opt has the same problem20:12
rlandymy default config is not a consideration20:12
rlandybut because it's a default20:12
rlandyso then we should allow people to specify a dir20:13
trownrlandy: right, 2 different things... I will not override default in my patch, but we also want ability to override default in the script20:13
rlandyso we leave the default as is but give the reproducer script the option to override it20:14
trownya I think that is right20:14
rlandyfine - fixing20:14
*** tesseract has quit IRC20:40
hubbot`FAILING CHECK JOBS on stable/queens: openstack-tox-pep8, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/560452, master: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master @ https://review.openstack.org/56044520:40
*** myoung|dr is now known as myoung20:41
myoung%gatestatus20:44
hubbot`FAILING CHECK JOBS on stable/queens: openstack-tox-pep8, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/560452, master: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master @ https://review.openstack.org/56044520:44
*** agopi|zzz has quit IRC20:45
*** agopi|zzz has joined #oooq20:52
*** trown is now known as trown|outtypewww20:54
*** jtomasek has quit IRC21:28
*** agopi|zzz has quit IRC21:35
*** Goneri has quit IRC22:04
*** rlandy has quit IRC22:11
*** chem has quit IRC22:37
hubbot`FAILING CHECK JOBS on stable/queens: openstack-tox-pep8, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/560452, master: gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master @ https://review.openstack.org/56044522:40
*** agopi|zzz has joined #oooq23:35
*** agopi|zzz has quit IRC23:46
*** tosky has quit IRC23:50

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!