Monday, 2019-04-08

hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248,  (2 more messages)00:33
*** irclogbot_0 has quit IRC01:44
*** vinaykns has joined #oooq02:12
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248,  (2 more messages)02:33
*** aakarsh has quit IRC02:59
*** vinaykns has quit IRC03:01
*** udesale has joined #oooq03:04
*** raukadah is now known as chandankumar03:14
*** ykarel has joined #oooq04:12
*** aakarsh has joined #oooq04:22
*** ratailor has joined #oooq04:33
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248,  (2 more messages)04:34
*** ykarel has quit IRC04:58
*** jfrancoa has joined #oooq05:13
*** ykarel has joined #oooq05:16
*** marios has joined #oooq05:30
*** jbadiapa has joined #oooq05:55
*** jaosorior has joined #oooq06:05
*** quiquell|off is now known as quiquell|rover06:05
mariosquiquell|rover: how green http://rhos-release.virt.bos.redhat.com:3030/rhosp :D happy monday06:17
quiquell|rovermarios: well I think there is not much stuff merged :-) but I am ok with green06:17
mariosquiquell|rover: :D06:19
quiquell|rover08:17:49          marios | quiquell|rover: very nice :D 09:17 < marios> quiquell|rover: how green http://rhos-release.virt.bos.redhat.com:3030/rhosp :D happy monday06:19
quiquell|roverThis is very TripleO :-)06:20
*** jtomasek has joined #oooq06:20
*** jtomasek has quit IRC06:22
*** quiquell|rover is now known as quique|rover|brb06:22
*** jtomasek has joined #oooq06:23
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248,  (2 more messages)06:34
*** jtomasek has quit IRC06:49
*** jtomasek has joined #oooq06:49
*** kopecmartin|off is now known as kopecmartin06:50
*** jtomasek has quit IRC06:51
*** jtomasek has joined #oooq06:51
*** sshnaidm|off is now known as sshnaidm06:53
*** apetrich has joined #oooq06:59
*** jtomasek has quit IRC06:59
*** jtomasek has joined #oooq07:00
*** quique|rover|brb is now known as quiquell|rover07:01
*** bogdando has joined #oooq07:16
*** ykarel is now known as ykarel|lunch07:17
*** tosky has joined #oooq07:19
*** amoralej|off is now known as amoralej07:39
*** holser_ has joined #oooq07:41
*** skramaja has joined #oooq07:46
*** yolanda_ has joined #oooq07:52
*** dsneddon has joined #oooq07:52
*** jpena|off is now known as jpena07:54
*** chem has joined #oooq07:54
sshnaidmquiquell|rover, hi, where do you plan to use them? https://review.rdoproject.org/r/#/c/20016/07:56
*** dsneddon has quit IRC07:57
quiquell|roversshnaidm: https://review.rdoproject.org/r/1889207:59
quiquell|roversshnaidm: I am testing https://review.rdoproject.org/r/1889208:00
quiquell|roverykarel|lunch: stein centosci still not working ?08:02
quiquell|roverhas fail todayu08:04
*** ykarel|lunch is now known as ykarel08:06
ykarelquiquell|rover, it started working and i can see phase 1 stein promotions08:06
ykarelnot checked todays08:07
quiquell|roverykarel: I see it failing https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-stein-current-tripleo-delorean-minimal-11/console.txt.gz08:07
quiquell|roverykarel: maybe there is nothing to promote ?08:07
ykarelquiquell|rover, https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-stein-current-tripleo-delorean-minimal-11/undercloud/home/stack/overcloud_deploy.log.gz08:08
ykarelquiquell|rover, good to see next run, or rerun, i have seen such intermittent failures before,08:08
ykarelfailed once ?08:08
quiquell|roveryep once08:09
quiquell|roverOk let's wait more re-runs08:09
quiquell|roverat least CI part is ok now08:09
quiquell|roversshnaidm: have increase the delay between retries at the standalone ci review you have08:09
sshnaidmquiquell|rover, yeah.. that's some problem there, logging stops when creating this swap file, but job continues08:10
quiquell|roversshnaidm: Ahh I forgot08:10
quiquell|roversshnaidm: swapping and nested would be a problem ?08:11
sshnaidmquiquell|rover, I saw logging stops in my local env too sometimes, but don't know what's a problem08:11
*** yolanda_ has quit IRC08:11
sshnaidmquiquell|rover, maybe it breaks when long time no response from host or kind of08:11
*** yolanda_ has joined #oooq08:11
sshnaidmquiquell|rover, who knows this finger..08:11
sshnaidmquiquell|rover, I'll try to increase jobs time to 3 hours too, I think it's limited to 2 now. And let's see just jobs logs, without console log08:13
*** panda has joined #oooq08:14
quiquell|roversshnaidm: ack08:14
*** saneax has joined #oooq08:15
zbrpanda: morning!!08:17
*** saneax has quit IRC08:19
*** sanjayu_ has joined #oooq08:19
zbrsshnaidm: may you can tell me what is the purpose of config/release/centosci/master-consistent.yml release file?08:20
sshnaidmzbr, its purpose is to confuse people08:21
zbrit does a realyl good job at that.08:21
sshnaidmzbr, it does its work08:21
zbrcan it be removed?08:22
sshnaidmzbr, while working in some company before, I had one lab network engineer that checked cables in that way: trying to pull them out and listening on which floor shouting and yelling08:23
*** sanjayu__ has joined #oooq08:24
zbrsshnaidm: i know the technique, usually it does produce results. not always what you want but.... :)08:24
quiquell|roverthey will not forget you for sure08:24
sshnaidmzbr, centos ci is blackbox with a lot of legacy, cleaning one won08:25
sshnaidmt help08:25
sshnaidmzbr, I wouldn't touch it without a good plan what to clean and what not08:25
*** sanjayu_ has quit IRC08:27
*** ccamacho has joined #oooq08:28
zbrsshnaidm: i do not fave any desires to touch but that release file preventing me from adding some release file validations because it was failing them. this is why i asked, not because I have some desire about centosci08:28
zbrsshnaidm: if you are curious about the failure, see http://logs.openstack.org/65/649965/16/check/tripleo-ci-validations/d2bdec9/tox/reports.html08:29
sshnaidmzbr, just don't validate it08:29
zbri hate to add excludes, they are prone to become future bugs.08:29
zbr2nd question: how can I programatelly detect if a release happened or not? (it should return true for stein at this moment). i am trying to avoid writing code that would need to be fixed very soon.08:31
*** sanjayu__ has quit IRC08:31
*** sanjayu_ has joined #oooq08:31
quiquell|roverzbr: promotions are done calling DLRN08:31
sshnaidmzbr, we have these ones: http://artifacts.ci.centos.org/artifacts/rdo/images/master/delorean/consistent/stable/08:31
quiquell|roverzbr: you can use dlrn python client08:31
sshnaidmzbr, from 2017 year08:31
sshnaidmzbr, I think you can delete it (but don't catch me on word!)08:32
zbrsshnaidm: i was considering replacing testing with stable, thanks. seems on the safe side.08:32
*** dsneddon has joined #oooq08:32
*** derekh has joined #oooq08:33
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248,  (2 more messages)08:34
*** dtantsur|afk is now known as dtantsur08:35
quiquell|roversshnaidm: do we need extra +2 to merge https://review.rdoproject.org/r/#/q/topic:periodic-reproducer+(status:open+OR+status:merged)  ?08:56
*** d0ugal has quit IRC08:56
sshnaidmquiquell|rover, you can merge it09:00
sshnaidmquiquell|rover, in our ci repos there is no such policy09:00
quiquell|roverack09:03
quiquell|roversshnaidm: +w09:03
sshnaidmquiquell|rover, thanks09:04
sshnaidmquiquell|rover, did you try to reproduce something from periodics?09:04
quiquell|roversshnaidm: nope09:05
sshnaidmquiquell|rover, I successfully reproduced containers build, but need a few tweaks: http://paste.openstack.org/show/748991/09:06
sshnaidmquiquell|rover, I think need some docs to do about it..09:06
sshnaidmquiquell|rover, or maybe to do all this logins conditional09:06
quiquell|roversshnaidm: why do you have gate too ?09:06
sshnaidmquiquell|rover, just used to, can ignore it :)09:07
quiquell|roverhumm with push_containers false you don't need rdo registry09:07
quiquell|roverwas one of your reviews09:07
sshnaidmquiquell|rover, it still tried09:07
quiquell|roverhummm ok09:07
quiquell|roverWill try to test the buildah stuff with it09:07
quiquell|roversshnaidm: I am checking the upgrades jobs to see if they work or not though09:08
sshnaidmquiquell|rover, there are a few logins, to dlrn api and registry09:08
sshnaidmquiquell|rover, ok, tell me if something doesn't work, will continue to fix09:08
quiquell|roverwill do09:08
sshnaidmquiquell|rover, I'll try to write some docs about testing config/ patches09:08
*** yolanda_ has quit IRC09:09
zbrsshnaidm: can you help me with an early review on https://review.openstack.org/#/c/649965/ ? i made it pass, ignore the -2 from wes, is only because he wants to double check it, still we will need another reviewer anyway.09:09
*** d0ugal has joined #oooq09:10
sshnaidmzbr, will look09:11
zbrsshnaidm: thanks.09:11
sshnaidmzbr, commented09:28
quiquell|roverzbr: Can you rebase https://review.rdoproject.org/r/#/c/19844/ I will merge it after09:36
zbrquiquell|rover: doing now...09:36
arxcruzkopecmartin: https://review.openstack.org/#/c/638617/09:39
arxcruzkopecmartin: do yu know why the demo job fail ? I'm not finding the logs09:39
*** holser_ is now known as holser|lunch09:40
kopecmartinarxcruz, it failed to list the roles09:40
kopecmartinarxcruz, on this line https://review.openstack.org/#/c/638617/1/config_tempest/main.py@51009:40
arxcruzkopecmartin: ok, so, i'll wait your next patch for review :)09:41
arxcruzkopecmartin: you can use exceptions.Forbidden09:42
arxcruzkopecmartin: https://github.com/openstack/python-tempestconf/blob/master/config_tempest/services/object_storage.py#L4009:43
arxcruzquiquell|rover: https://bugs.launchpad.net/tripleo/+bug/182169009:45
openstackLaunchpad bug 1821690 in tripleo "periodic-ovb-1ctlr_1comp-featureset020-master fails tempest test_auto_allocate_network tempest.api.compute.images.test_images" [High,Fix released] - Assigned to Arx Cruz (arxcruz)09:45
arxcruzthere's no fix release AFAIK09:45
arxcruzthere's a workaround to disable the test09:45
arxcruzI don't know if we should just close it or not, maybe ask weshay|ruck when he's available09:45
quiquell|roverarxcruz: just blind closed it sorry09:45
quiquell|roverok09:45
quiquell|roverlet's just ask09:45
arxcruzkopecmartin: https://review.openstack.org/648121  will you be angry with me if i ask for func tests? and a dummy job testing it ?09:51
kopecmartinarxcruz, what about trying it manually? :D09:51
zbrquiquell|rover: sshnaidm : fixed https://review.rdoproject.org/r/#/c/19844/09:51
quiquell|roverzbr: +w09:52
arxcruzchandankumar: https://review.openstack.org/#/c/650613/ is this the new collect-logs repo ?09:54
chandankumararxcruz: yes09:54
quiquell|roversshnaidm: cockpit grafana upgrade https://review.rdoproject.org/r/#/c/20019/09:55
*** d0ugal has quit IRC09:58
*** d0ugal has joined #oooq09:59
*** d0ugal has quit IRC10:02
zbrquiquell|rover: sshnaidm : another one fixed https://review.rdoproject.org/r/#/c/19540/ -- feeling like a productive monday10:02
*** d0ugal has joined #oooq10:02
quiquell|roverzbr: we cannot just use $@ ?10:05
quiquell|roverinstead of EOF ?10:05
zbrquiquell|rover: not safe, it chokes on quotes and other similar stuff.10:05
quiquell|roverzbr: ack10:06
quiquell|rover+210:06
zbrthe redirection hack is know to be much safer. (not as nice, but avoids...)10:06
zbrwhen it comes to bash, usually the resilient solution is the ugly one, not the beautiful/simple one.10:07
*** yolanda_ has joined #oooq10:11
quiquell|roverarxcruz: is same issue here ? https://bugs.launchpad.net/tripleo/+bug/182334710:29
openstackLaunchpad bug 1823347 in tripleo "periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset021-stein is failing overcloud deploy - ERROR configuring ovn_controller" [Critical,Fix released] - Assigned to wes hayutin (weshayutin)10:29
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248,  (2 more messages)10:34
*** yolanda_ has quit IRC10:38
arxcruzquiquell|rover: checking10:43
arxcruzquiquell|rover: nope, this is fixed for real10:43
quiquell|roverok10:43
*** yolanda_ has joined #oooq10:48
*** holser|lunch is now known as holser_10:54
chandankumarsshnaidm: zbr quiquell|rover https://review.openstack.org/#/c/650613/ please have a look when free!11:00
*** panda is now known as panda|lunch11:06
*** udesale has quit IRC11:08
*** udesale has joined #oooq11:09
*** d0ugal has quit IRC11:10
*** d0ugal has joined #oooq11:10
sshnaidmquiquell|rover, standalone executed till tempest \o/ http://logs.rdoproject.org/17/19917/7/check/tripleo-ci-reproducer-fedora-28-libvirt-standalone/fad7942/tripleo-ci-reproducer/logs/01/1001/1/check/tripleo-ci-centos-7-standalone-dlrn-hash-tag/817a810/job-output.txt.gz11:12
sshnaidmquiquell|rover, no idea what helped..11:12
sshnaidmquiquell|rover, failed on no tempest image and this is legit11:12
quiquell|roversshnaidm: maybe it just take a lot of time for dd11:13
quiquell|rover?11:13
quiquell|roverhumm took around 40 minuts to create swap file11:14
quiquell|roverholy...11:14
quiquell|roverdo we need swap still ?11:14
sshnaidmquiquell|rover, maybe, we'll see11:15
zbrsshnaidm: https://review.rdoproject.org/r/#/c/19540/  please.11:15
sshnaidmquiquell|rover, not sure if we need it for standalone, need to check dstat or other stats11:15
sshnaidmquiquell|rover, if not, need some options to skip it as it's in base job11:16
quiquell|roversshnaidm: humm ifwe skip it in base job we have to skip it here too11:16
sshnaidmzbr, +w11:17
zbrsshnaidm: thanks. re swap creationg, I wonder why we do not attempt to create it without dd, based on https://linuxize.com/post/create-a-linux-swap-file/ we should try without dd first.11:19
sshnaidmzbr, I think I tried with fallocate and it didn't work in ci, don't remember why..11:19
sshnaidmzbr, maybe we don't have it in repos  or kind of11:20
zbrdepends on filesystem, but the best approach is to try first and use fallback.11:20
*** dtantsur is now known as dtantsur|brb11:20
zbrit depends on lots of factors.11:20
sshnaidmquiquell|rover, seems like close to 4G: http://logs.rdoproject.org/17/19917/7/check/tripleo-ci-reproducer-fedora-28-libvirt-standalone/fad7942/tripleo-ci-reproducer/logs/01/1001/1/check/tripleo-ci-centos-7-standalone-dlrn-hash-tag/817a810/logs/undercloud/var/log/extra/dstat.html.gz11:20
zbranyway we need to find a way to avoid the 40min delay, makes no sense to me. i am sure there are alternatives, we only need to find them.11:21
quiquell|roverholy sh... let's bypass that 40 minuts extra time for CI is a pain11:21
quiquell|roverzbr: agree11:21
zbr... also I have seeing a dd that obliterates 8GB of my ssd each time I run... I should probably expense it ;)11:22
sshnaidmquiquell|rover, we definitely can make 2G: http://logs.rdoproject.org/17/19917/7/check/tripleo-ci-reproducer-fedora-28-libvirt-standalone/fad7942/tripleo-ci-reproducer/logs/01/1001/1/check/tripleo-ci-centos-7-standalone-dlrn-hash-tag/817a810/logs/undercloud/var/log/extra/swaps.txt.gz11:24
sshnaidmquiquell|rover, zbr it's only in libvirt job, don't forget :)11:24
quiquell|roversshnaidm: better try first to bypass it11:24
quiquell|roversshnaidm: what do you think ?11:24
sshnaidmquiquell|rover, yeah, worth to try11:25
sshnaidmquiquell|rover, anyway, need to get tempest cirros image as well11:25
quiquell|roversshnaidm: we don't run tempest from a container ?11:26
sshnaidmquiquell|rover, it's testing image11:26
sshnaidmquiquell|rover, used in tempest tests11:26
quiquell|roversshnaidm: ack, so this is libvirt issue or reproducer ?11:26
sshnaidmquiquell|rover, it's prepared in upstream images, but we don't prepare it in our libvirt image11:27
*** jpena is now known as jpena|lunch11:27
quiquell|roverAhh get it now ok11:27
quiquell|roverat least now we have libvirt CI :-)11:27
quiquell|roversort of11:27
sshnaidmquiquell|rover, creating swap take 40 seconds, not minutes :)11:29
sshnaidmbetter to leave it then11:30
quiquell|roverWhat did I check ?11:30
quiquell|roverNot my day11:30
*** rfolco has joined #oooq11:33
quiquell|roverykarel, sshnaidm: Now the check at build-test-package is working as expected http://logs.rdoproject.org/91/18891/5/check/tripleo-ci-centos-7-standalone-rdo/b2f6880/job-output.txt.gz11:37
quiquell|rover"Unable to build gnocchixyz/gnocchi. No pkg found."11:38
quiquell|roverbut working fine with skipped projects https://review.rdoproject.org/r/#/c/18892/11:38
quiquell|roverthis is the review https://review.openstack.org/#/c/636588/11:39
quiquell|roverzbr, panda|lunch, marios: ^11:39
sshnaidmgreat11:39
quiquell|roverLet's merge the fix11:40
ykarelquiquell|rover, ack but to get it working you need to add testproject to skip list right?11:42
ykarelso we don't know how many such projects are there11:42
quiquell|roverykarel: have added in the review11:42
ykarelwhich we need to add to skip list11:42
quiquell|roverykarel: did it https://review.openstack.org/#/c/636588/11:43
quiquell|roverykarel: that's why the test reviews work11:43
ykarelif this patch will merge without adding all the projects to skip list, we will have a situation where we need to add project to skip list whenever needed11:43
ykarelyup that's what my point was earlier how are you handling testproject11:43
ykareland it's skipped project11:44
quiquell|roveris now at skipped project11:44
quiquell|roverso it continues11:44
quiquell|roverwe don't fail there11:44
quiquell|rovererror is if projec tis not at RDO and is not at skipped list11:45
quiquell|rovertest project is not at RDO but it's at skipped list11:45
ykarelquiquell|rover, what is meant by: test project is not at RDO?11:47
ykarelyou mean package11:47
ykarelso that list is very large11:47
ykarelif you mean package then ^^11:47
quiquell|roverI mean project name11:47
quiquell|roverIf we have more test projects we will have to enlarge that list11:48
quiquell|roverykarel: there are that much of projects ?11:53
quiquell|roverykarel: from wich we don't have dlrn mapping and we want to test ?11:54
ykarelquiquell|rover, one example: rdo-jobs,11:54
ykarelgating_scripts, rdoinfo etc11:54
ykarelthere can be many11:54
ykarelin upstream too may be11:54
quiquell|rovermaybe we can ignore the launching project ?11:54
quiquell|roverI mean at the end what we want it to prevent stuff from Depends-On11:55
quiquell|roverbut if the launching project has zuul config to run the job then it's suppose to run it11:55
quiquell|roverwhat do you think ?11:55
arxcruzkopecmartin: ?11:57
kopecmartinarxcruz, yes?11:57
*** rlandy has joined #oooq11:57
arxcruzkopecmartin: https://review.openstack.org/648121  will you be angry with me if i ask for func tests? and a dummy job testing it ?11:57
*** quiquell|rover is now known as quique|rover|eat11:57
*** derekh has quit IRC11:58
kopecmartinarxcruz, ah, I can create a unit test, although, the code is really simple ..11:59
*** rlandy is now known as rlandy|ruck12:02
rlandy|ruckweshay|ruck: I can go to the escalation meeting today12:02
weshay|ruckk thanks12:02
sshnaidmweshay|ruck, hi12:04
weshay|ruckhey o12:04
sshnaidmweshay|ruck, do you have 5 mins to talk about rhel7 on rdo?12:05
weshay|rucksure12:05
weshay|rucksshnaidm I'm in my blue12:06
rlandy|ruckquique|rover|eat: arxcruz: hi - what's the story with https://bugs.launchpad.net/tripleo/+bug/1821690?12:06
openstackLaunchpad bug 1821690 in tripleo "periodic-ovb-1ctlr_1comp-featureset020-master fails tempest test_auto_allocate_network tempest.api.compute.images.test_images" [High,In progress] - Assigned to Arx Cruz (arxcruz)12:06
rlandy|ruckback to 'in progress'?12:06
quique|rover|eatrlandy|ruck: I closed it but looks like is still open12:07
quique|rover|eatrlandy|ruck: the only stuff we did was add the test to skip list12:07
quique|rover|eatrlandy|ruck: arx corrected me12:07
quique|rover|eatrlandy|ruck: btw we have to look at master noop for tripleo-ci-centos-7-containers-multinode12:07
quique|rover|eatrlandy|ruck: it has fail at port binding at keyston12:07
rlandy|ruckquique|rover|eat: k - just so I know what to report at escalation meeting12:09
quique|rover|eatIs in the etherpad12:09
sshnaidmrfolco, rlandy|ruck where is config of our internal zuul? is it connected to upshift?12:15
rlandy|rucksshnaidm: http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/doc/oooq-upstream-baremetal.rst12:16
rlandy|ruck^^ links to all downstream stuff are included12:16
rlandy|ruckSoftware Factory config repo: http://git.app.eng.bos.redhat.com/git/openstack/sf-config.git/tree/12:16
rlandy|ruck- tripleo-ci-internal-config: http://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-config.git/tree/12:17
*** Vorrtex has joined #oooq12:17
rlandy|ruck^^ looking for12:17
sshnaidmrlandy|ruck, do you know who built these rhel7 nodes? https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/local/nodes12:17
rlandy|rucksshnaidm: admins - fbo maybe12:18
rlandy|rucksshnaidm: look at http://git.app.eng.bos.redhat.com/git/openstack/sf-config.git/tree/nodepool/nodepool.yaml12:19
rlandy|ruckI added the nodeset off that node12:19
rlandy|ruckfor the rhos-14 work12:19
*** jpena|lunch is now known as jpena12:22
arxcruzchandankumar: where's your scenario001 card in https://tree.taiga.io/project/tripleo-ci-board/us/972 ?12:26
*** amoralej is now known as amoralej|lunch12:28
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248,  (2 more messages)12:34
zbrweshay|ruck: can you please help me with https://review.rdoproject.org/r/#/c/19966/ ?12:37
zbrexample of execution https://seashells.io/v/TZT6DMaC12:37
*** Vorrtex has quit IRC12:38
*** quique|rover|eat is now known as quiquell|rover12:40
*** dtantsur|brb is now known as dtantsur12:47
chandankumararxcruz: tracked here https://tree.taiga.io/project/tripleo-ci-board/task/90612:48
chandankumararxcruz: need to move it to another user story12:49
arxcruzchandankumar: i'll work on the scenario00212:49
chandankumararxcruz: cool!12:49
arxcruzi'll create the card, but just to you be aware12:49
*** derekh has joined #oooq12:50
chandankumarack!12:50
*** ratailor has quit IRC12:52
ykarelquiquell|rover, have you got what was the issue for network outage on 5th of April that you created rdo cloud ticket12:52
quiquell|roverykarel:12:53
quiquell|roverykarel: nope12:53
ykarelweshay|ruck, ^^ what i mentioned in call, outage was around for couple of minutes, but issue not known12:53
ykarelquiquell|rover, ack12:53
weshay|ruckk12:53
weshay|ruckthanks ykarel :)12:53
quiquell|roverwas fast issue12:54
*** quiquell|rover is now known as quique|rover|bbl12:57
*** mjturek has joined #oooq13:00
*** amoralej|lunch is now known as amoralej13:02
zbrweshay|ruck: re https://tree.taiga.io/project/tripleo-ci-board/epic/774 -- i have no idea why 804 ended up in current sprint even if it was parked, but 805 which was in progress ended up not in it. I see no option to change that myself.13:02
*** mjturek has quit IRC13:04
weshay|ruckzbr the thought was that we had to kick out the molecule work due to other priorities.  We addressed this in planning a bit..  I could see the release molecule work being part of this / folding into this13:05
weshay|ruckzbr essentially this is to give you space to work on an 1. molecule testing a role, 2. testing config 3.  testing a simple script13:06
weshay|ruckzbr how you organize that work I think is largely up to you13:06
zbrweshay|ruck: sure. my only concern was that we kick the one that was ready for review, and left the one that was not worked on. ok.13:06
weshay|ruckzbr so I'm happy to get back into this with you a bit to give you more clarity, but just note.. we added this to give you time/space on molecule13:07
zbrweshay|ruck: sure. let me know when you have some time to talk.13:10
zbrgoing for lunch now...13:10
*** Vorrtex has joined #oooq13:11
*** mjturek has joined #oooq13:14
*** panda|lunch is now known as panda13:20
*** vinaykns has joined #oooq13:37
*** jjoyce has joined #oooq13:40
*** Goneri has joined #oooq13:49
*** quique|rover|bbl is now known as quiquell|rover13:51
*** dtrainor has joined #oooq13:52
zbrsshnaidm: re sftp, it does not allow "-p" on mkdir, i tested.13:58
sshnaidmzbr, ok13:58
sshnaidmzbr, but then it will fail if exists13:59
zbrsshnaidm: yep, but I managed to find a way to make more resilient, please check https://review.rdoproject.org/r/#/c/19966/8/ci-scripts/promote-images.sh13:59
zbrthat one is also dealing with the distros, but if you want I can isolated the "resilient" part in another review.14:00
rfolcoping scrum marios, quiquell, sshnaidm, weshay, panda, rlandy, arxcruz, mwhahaha, rfolco, chkumar, zbr, kopecmartin14:01
quiquell|roverweshay|ruck: can you remove -1 from here ? https://review.rdoproject.org/r/#/c/18502/14:01
zbrthe "-" suffix is key for ignoring errors (works like || true).14:01
rfolcorfolco@radio: stick together team14:02
sshnaidmzbr, I think better to fix the images part firstly14:02
rfolcoSCRUM time14:02
quiquell|roverykarel: Have removed testproject from skip list, it now also cheks zuul.project.name https://review.openstack.org/#/c/636588/14:03
ykarelquiquell|rover, but looks like it will not work with Depends-ON, or it will work?14:04
weshay|ruckquiquell|rover done14:05
quiquell|roverykarel: what do you mean ?14:05
ykarel - artg_change.project != zuul.project.name14:05
ykarelso mainly you did ^^, right?14:05
quiquell|roveryep14:05
quiquell|roverI mean like limiting the project where do you throw zuul does not make much sense14:05
quiquell|roverAt the end this is about controlling Depends-On that we don't know about14:06
quiquell|roverThey have to be a zuul project though14:06
ykarelquiquell|rover, i think that condition will just cover the project against which job is fired14:07
quiquell|roverykarel: artg_change.project is the project of the Depends-On this part of the role is iterated on them14:08
quiquell|roverthe two test reviews confirm that14:08
ykarelithat i understood, mainly talking about: - artg_change.project != zuul.project.name14:09
ykareli just rechecked one of the testreview14:09
*** bogdando has quit IRC14:19
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248,  (2 more messages)14:34
mariosarxcruz: auf wiedersehen!14:40
arxcruzmarios: tchuss :D14:41
sshnaidmrlandy|ruck, we have nodesets config in tripleo-ci-internal-jobs, shouldn't it be in tripleo-ci-internal-config? Or it doesn't matter much14:47
rlandy|ruckquiquell|rover: objected to having stuff not requiring secrets in a repo we can't test14:48
rlandy|ruckdoesn't really matter14:48
quiquell|rovereasier to check in the same review of changes in those cases14:48
sshnaidmrlandy|ruck, it's not base repos14:52
rlandy|rucksshnaidm: ping me after meeting - we can go through it14:53
sshnaidmrlandy|ruck, ack14:53
ykarelquiquell|rover, can you review https://review.openstack.org/#/c/622037/14:54
quiquell|roverykarel: do yo have a test review ?14:56
ykarelquiquell|rover, yes see Alfredo comment14:57
ykarelit mainly adds a blank line when combining repo for delorean14:57
ykarelbecause we can't assume repos will have blank lines in them14:57
quiquell|rover+w14:58
ykarelquiquell|rover, Thanks14:58
rlandy|ruckmarios: let's talk after meeting re: osp-1414:59
quiquell|roversshnaidm: reproducer calling autohold before job finish -> https://review.rdoproject.org/r/#/c/20020/15:00
sshnaidmquiquell|rover, great, +215:01
quiquell|roverweshay|ruck: can you merge it ? https://review.rdoproject.org/r/#/c/20020/15:02
quiquell|roverI have also merge panda work on post failure to fail on success jobs15:02
kopecmartinarxcruz, i've added a unit test here https://review.openstack.org/#/c/648121 , I've tried to edit our jobs to test it  https://review.openstack.org/#/c/650874/, but it turns out it's not that easy and i really don't want to waste my time on DNM patch .. please try it manually if you wanna see it's working15:03
*** ykarel is now known as ykarel|away15:06
mariosrlandy|ruck: sure gime few mins15:10
arxcruzkopecmartin: well, we can discuss this with tosky and chandankumar but you're adding a functionality without test... at least the unit test is being executed?15:12
kopecmartinarxcruz, sure .. yes of course, the unit test is executed .. ok, let's see what chandankumar and tosky say about the change, thanks15:13
rfolcomarios, are you happy or you need more? https://etherpad.openstack.org/p/tripleo-ci-squad-meeting L3915:14
*** ykarel|away has quit IRC15:15
mariosrlandy|ruck: talk in 15 mins ok for you?15:15
mariosrfolco: looking15:16
rlandy|ruckmarios: sure15:16
mariosrfolco: fine thanks15:16
rfolcomarios, ack("cool bro")15:17
marios;)15:17
pandamarios: can't you use HTML5 ?15:19
mariospanda: haha.. pacman in comments or it didn't happen15:20
marios(don't know(15:20
rlandy|ruckquiquell|rover: hi - iiuc, you did this work already? https://tree.taiga.io/project/tripleo-ci-board/task/938?kanban-status=144727515:20
rfolcomarios, I don't have flash plugin either15:26
*** rfolco is now known as rfolco|lunch15:29
rlandy|ruckmarios: http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/config/release/rhos-14.yml#n6715:41
weshay|ruckrlandy|ruck quiquell|rover fyi.. board is updated http://rhos-release.virt.bos.redhat.com:3030/rhosp15:44
quiquell|roverrlandy|ruck: It's done let me know if we miss anything else15:48
quiquell|roverRead you tomorrow team15:49
*** jfrancoa has quit IRC15:50
*** quiquell|rover is now known as quiquell|off15:50
rlandy|ruckweshay|ruck: board looks good - http://rhos-release.virt.bos.redhat.com:3030/rhospImport master in first line doesn't have any days 'empty d'15:53
marioso/ ttyl folks /me shutdown sequqnce15:56
*** rfolco|lunch is now known as rfolco15:58
*** mjturek has quit IRC16:00
weshay|ruckrlandy|ruck ya, burns will fix that16:02
rlandy|ruckcool - then it's good16:02
weshay|ruckand green16:02
*** mjturek has joined #oooq16:02
rlandy|ruckweshay|ruck: un'rucking for a bit - working on ha change sand standaloen rhos-14 change/hw updates16:03
weshay|ruckrlandy|ruck quiquell|off found another use case for the reproducer that I think can only be done w/ zuul via the reproducer :)16:03
weshay|ruckrlandy|ruck ack16:03
*** rlandy|ruck is now known as rlandy16:03
rlandyweshay|ruck: great - thanks for the cover16:03
weshay|rucknp16:05
weshay|ruckzbr I can chat now if you want.. I'm debugging w/ Emilien a little on the side but have the space and time16:05
*** ccamacho has quit IRC16:11
sshnaidmrlandy, weshay|ruck can we chat about rhel repos?16:15
rlandysshnaidm: yep16:16
sshnaidmrlandy, weshay|ruck https://bluejeans.com/u/sshnaidm/16:16
*** ykarel|away has joined #oooq16:16
weshay|ruckhttps://code.engineering.redhat.com/gerrit/gitweb?p=tripleo-environments.git;a=tree;f=config/release;h=b02256a3b9ade8c154c5a86ac40e4031e552114f;hb=HEAD16:17
weshay|rucksshnaidm ^16:17
zbrweshay|ruck: perfect.16:18
*** marios has quit IRC16:25
*** dtantsur is now known as dtantsur|afk16:27
weshay|ruckzbr ok.. 2 min16:32
weshay|ruckwill be in my blue16:32
rlandysshnaidm: https://code.engineering.redhat.com/gerrit/#/c/166374/16:32
weshay|rucksshnaidm, rlandy /me drops.. good chat16:33
sshnaidmrlandy, https://code.engineering.redhat.com/gerrit/#/c/167241/16:33
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248,  (2 more messages)16:34
rlandysshnaidm: ack - give me a few - I'll add a mergeable review so we don't duplicate16:34
rlandyweshay|ruck: simple change discussed on friday: https://review.openstack.org/#/c/650959/16:45
*** jpena is now known as jpena|off16:45
rlandyweshay|ruck: notice rdocloud mirror down?16:50
rlandyasking on rhos-ops16:51
weshay|ruckzbr https://review.openstack.org/#/c/649965/16:53
weshay|ruckrlandy I did just see some 3rd party jobs blow up16:53
*** derekh has quit IRC16:58
rfolcoping bug triage panda weshay|ruck17:01
rfolcoweshay|ruck, you need to skip it ?17:03
weshay|ruckI'll be there few minutes17:03
*** panda has quit IRC17:04
*** udesale has quit IRC17:05
*** udesale has joined #oooq17:09
*** kopecmartin is now known as kopecmartin|off17:16
rfolcoweshay|ruck, https://review.openstack.org/#/c/641455/4/roles/validate-tempest/vars/tempest_skip_master.yml17:17
zbrsshnaidm: still here? I want to wf https://review.openstack.org/#/c/649965/17:32
*** udesale has quit IRC17:41
zbroops, posted wrong link, correct one: https://review.rdoproject.org/r/#/c/20024/17:42
zbrrlandy: maybe you can help with ^?17:42
rlandyzbr: ready to merge: https://review.rdoproject.org/r/#/c/20024/?18:03
zbrrlandy: yes.18:03
*** skramaja has quit IRC18:04
*** ykarel|away has quit IRC18:13
zbrrlandy: thanks, now we only need to review https://review.rdoproject.org/r/#/c/19966/ with weshay|ruck and we can close the promoter card.18:14
zbri tested it, thus is why I am now removing few "foo" folders from the image server.18:15
zbrit will be fun as I suppose that I suppose we will have to add multi-arch support in addition to multi distro but that's for another time.18:17
chandankumarweshay|ruck: rlandy https://review.openstack.org/#/c/649553/ and https://review.rdoproject.org/r/#/c/19916/ to complete scenario 01 to os_tempest18:19
*** weshay|ruck is now known as weshay18:24
weshaychandankumar /me looks18:24
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248,  (2 more messages)18:34
weshayrlandy you want to kill the ci.centos jobs this week?18:35
rlandyweshay: sure - if we are not using them18:35
rlandyI can do that tomorroq18:35
rlandytomorrow18:35
weshaycool18:35
*** sshnaidm is now known as sshnaidm|off18:38
zbrrlandy: see https://seashells.io/v/BdjX7kuu for example on https://review.rdoproject.org/r/#/c/19966/ -- I also fixed the typo.18:54
rlandyweshay: can you explain this ... http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/config/release/rhos-14.yml#n68 ... docker_registry_host: docker-registry.engineering.redhat.com - but looking at  update registry if needed in container-prepare-parameters.yaml ... adds INSECURE_REGISTRY='--insecure-registry 192.168.24.1:8787' to the /etc/sysconfig/docker18:59
weshayya.. that is the registry on the undercloud18:59
weshayneeds to be set insecure18:59
rlandyyes - but the var should resolve to docker-registry.engineering.redhat.com19:00
weshayrlandy no19:00
rlandyhow so?19:00
weshayyou set insecure on the undercloud registry19:00
weshayit's for the overcloud deployment19:00
rlandyroles/standalone/tasks/main.yml19:00
rlandyweshay: in a standalone deployment where is the overcloud set?19:01
weshayoh19:03
weshayrlandy I think it still holds true19:04
rlandyweshay: ok - so pls point that out to me19:04
weshaybecause the standalone first pulls all the containers into a local registry19:04
weshayok.. sec /me looks at tc19:04
weshayrlandy I'm on a tmate w/ Emilien debugging podman19:05
weshayrlandy our work on friday paid off :))19:05
rlandygood19:05
rlandyweshay: k - pls ping when you have a minute19:05
weshayrlandy19:07
weshayDockerInsecureRegistryAddress:19:07
weshay    - $IP:878719:07
weshayhttps://docs.openstack.org/tripleo-docs/latest/install/containers_deployment/standalone.html19:07
weshayrlandy see it in the doc?19:08
* rlandy is stubbon - still does not explain how {{docker_registry_host}} resolves to local ip19:09
weshayrlandy it shows it using the local ip there19:10
weshayexport IP=192.168.24.219:10
weshayrlandy see?19:10
rlandyI know - will explain more when you have a minute to bj19:10
weshayI can blue19:10
rlandyfine19:10
rlandyjoining your blue19:10
weshaymaybe Emilien would join too19:10
*** holser_ has quit IRC19:10
*** amoralej is now known as amoralej|off19:15
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248,  (2 more messages)20:34
jdennisIs there a reason why tripleo-quickstart/requirements.txt pins the version of ansible at exactly version 2.5.7? We are developing a new tripleo-qucikstart-extras role that requires ansible>=2.7 (because of new openssl _pkcs12 module). Is updating the ansible version going to be a problem?20:44
*** jbadiapa has quit IRC20:48
rlandyweshay: check it out ... success - rhos-14 standalone https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/stream/ec6d821534a140d3bac7c7f65f9f239c?logfile=console.log20:51
rlandyjdennis: that was probably the last confirmed tested ansible version - there are review out there to update ansible20:53
rlandyzbr: ^^20:53
*** mjturek has quit IRC20:57
*** Goneri has quit IRC21:14
*** Vorrtex has quit IRC21:29
rlandyugh - picky, picky, picky linters21:37
weshayrlandy WHA21:53
weshaythat's awesome21:53
* weshay looks21:53
weshaysorry.. been heads down21:53
rlandyweshay: fighting with linters again now21:53
weshaylolz21:53
rlandyhad to hack up or rhos-release routine because it didn't suit the linter21:53
rlandyugh21:53
* rlandy has a love/hate relationship with linters21:54
weshayrlandy which one? https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds21:54
weshayI missed the link21:54
weshayhttps://code.engineering.redhat.com/gerrit/#/c/166374/ ?21:54
rlandyweshay: https://sf.hosted.upshift.rdu2.redhat.com/logs/74/166374/12/check/tripleo-ci-rhel-7-standalone-rhos-14/ec6d821/21:54
rlandyhttps://code.engineering.redhat.com/gerrit/#/c/166374 change21:55
rlandynow I am juts modifying the playbook to get linters to pass21:55
rlandyit won't let us run yum in shell21:55
rlandyreally, really? why>21:55
weshayrlandy you installing the rhos-release rpm?21:59
rlandyweshay: ack21:59
rlandyat least I want to - if the linters wouldn't mind21:59
rlandyweshay: see the cert piece - you may not like that21:59
rlandybut the fail will be fast22:00
weshayrlandy if the linter is messin w/ you.. using rpm -Uvh htl22:00
weshayurl22:00
weshayworks just as well :)22:00
weshayrlandy or use yum and use failed_when22:00
rlandytried that trick already - no dice22:01
weshayhrm22:01
weshayrlandy why is it linting config?22:01
rlandyweshay: I think current patch is good22:01
rlandylinters approved22:01
rlandyno failures so far22:01
weshayah nevr mind22:01
rlandyletting this run - then will merge so that sagi gets the nodesets etc. without duplication22:02
rlandywelcome back OSP!!!22:02
rlandymissed you, old friend22:02
* rlandy need to move on and get HA reviews in now22:03
*** vinaykns has quit IRC22:18
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 @ https://review.openstack.org/602248,  (2 more messages)22:34
*** aakarsh has quit IRC22:54
rlandyand we're green :)23:51
*** rlandy has quit IRC23:52
*** tosky has quit IRC23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!